2023-10-16T12:15:08.718 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T12:15:08.719 DEBUG:teuthology.run:Teuthology command: teuthology --owner mchangir --name mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --archive /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986 --verbose -- /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/orig.config.yaml 2023-10-16T12:15:08.765 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T12:15:08.851 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986 branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: mchangir@redhat.com first_in_suite: false job_id: '7428986' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: c7471068426370a3754c2250e7729280233d552b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7471068426370a3754c2250e7729280233d552b kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-snap_schedule-support-subvol-and-group-arguments sha1: 9359c5b40091bccbce966736dc1a3c5bcc76ab0d owner: mchangir priority: 25 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7945 sha1: c7471068426370a3754c2250e7729280233d552b sleep_before_teardown: 0 suite: fs suite_branch: mgr-snap_schedule-support-subvol-and-group-arguments suite_path: /home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa suite_relpath: qa suite_repo: https://github.com/mchangir/ceph.git suite_sha1: 9359c5b40091bccbce966736dc1a3c5bcc76ab0d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH9fQ0gtr9jZfHA6EZvk7I44etDngq/AlmCWvEkCDjt0JPtwM4iUwHk9OZCL+D1oK6jGFM14qRQXlIlT4mXQIDs= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfNhnIlKlzJjnyFNggcHy/KuWjP3yzVYBJoBRaR4nIeuYgvBKYWmC5RzfBA9XWR9KibzaLap60SCOgnwgtX6Bg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_11:33:41 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-16T12:15:08.851 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa; will attempt to use it 2023-10-16T12:15:08.852 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks 2023-10-16T12:15:08.852 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-16T12:15:08.854 INFO:teuthology.task.internal:Checking packages... 2023-10-16T12:15:08.876 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7471068426370a3754c2250e7729280233d552b' 2023-10-16T12:15:08.876 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-16T12:15:08.877 INFO:teuthology.packaging:ref: None 2023-10-16T12:15:08.877 INFO:teuthology.packaging:tag: None 2023-10-16T12:15:08.877 INFO:teuthology.packaging:branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-16T12:15:08.877 INFO:teuthology.packaging:sha1: c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:15:08.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-16T12:15:09.157 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6675.gc7471068 2023-10-16T12:15:09.158 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-16T12:15:09.165 INFO:teuthology.task.internal:no buildpackages task found 2023-10-16T12:15:09.165 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-16T12:15:09.202 INFO:teuthology.task.internal:Saving configuration 2023-10-16T12:15:09.217 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-16T12:15:09.227 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T12:15:09.254 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 12:10:19.113882', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH9fQ0gtr9jZfHA6EZvk7I44etDngq/AlmCWvEkCDjt0JPtwM4iUwHk9OZCL+D1oK6jGFM14qRQXlIlT4mXQIDs='} 2023-10-16T12:15:09.276 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 12:10:19.112006', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfNhnIlKlzJjnyFNggcHy/KuWjP3yzVYBJoBRaR4nIeuYgvBKYWmC5RzfBA9XWR9KibzaLap60SCOgnwgtX6Bg='} 2023-10-16T12:15:09.277 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-16T12:15:09.283 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-16T12:15:09.283 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-16T12:15:09.283 INFO:teuthology.run_tasks:Running task console_log... 2023-10-16T12:15:09.363 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc5140d99d0>, signals=[15]) 2023-10-16T12:15:09.364 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-16T12:15:09.452 INFO:teuthology.task.internal:Opening connections... 2023-10-16T12:15:09.453 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:15:09.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:15:09.552 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:15:09.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:15:09.636 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-16T12:15:09.643 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2023-10-16T12:15:09.661 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2023-10-16T12:15:09.661 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2023-10-16T12:15:09.717 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2023-10-16T12:15:09.717 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2023-10-16T12:15:09.718 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2023-10-16T12:15:09.718 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2023-10-16T12:15:09.718 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2023-10-16T12:15:09.718 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2023-10-16T12:15:09.718 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-16T12:15:09.719 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2023-10-16T12:15:09.719 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2023-10-16T12:15:09.719 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-16T12:15:09.719 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2023-10-16T12:15:09.719 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T12:15:09.720 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-16T12:15:09.720 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-16T12:15:09.720 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2023-10-16T12:15:09.749 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2023-10-16T12:15:09.767 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2023-10-16T12:15:09.768 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2023-10-16T12:15:09.825 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2023-10-16T12:15:09.826 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="9" 2023-10-16T12:15:09.826 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2023-10-16T12:15:09.826 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2023-10-16T12:15:09.826 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="9" 2023-10-16T12:15:09.827 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el9" 2023-10-16T12:15:09.827 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-16T12:15:09.827 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2023-10-16T12:15:09.827 INFO:teuthology.orchestra.run.smithi144.stdout:LOGO="fedora-logo-icon" 2023-10-16T12:15:09.827 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-16T12:15:09.827 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2023-10-16T12:15:09.828 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T12:15:09.828 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-16T12:15:09.828 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-16T12:15:09.828 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2023-10-16T12:15:09.849 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-16T12:15:09.858 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-16T12:15:09.877 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-16T12:15:09.877 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2023-10-16T12:15:09.880 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2023-10-16T12:15:09.896 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-16T12:15:09.902 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-16T12:15:09.902 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2023-10-16T12:15:09.936 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2023-10-16T12:15:10.182 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-16T12:15:10.237 INFO:teuthology.run_tasks:Running task kernel... 2023-10-16T12:15:10.251 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-16T12:15:10.252 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2023-10-16T12:15:10.252 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2023-10-16T12:15:10.253 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T12:15:10.253 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T12:15:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:15:10.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-10-16T12:15:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:15:10.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-10-16T12:15:10.437 DEBUG:teuthology.task.kernel:sha1 for client.1 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T12:15:10.437 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-16T12:15:10.437 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2023-10-16T12:15:10.455 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-361.el9.x86_64 2023-10-16T12:15:10.455 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T12:15:10.456 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-16T12:15:10.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum upgrade -y linux-firmware 2023-10-16T12:15:10.644 DEBUG:teuthology.task.kernel:sha1 for client.0 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T12:15:10.645 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-16T12:15:10.645 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2023-10-16T12:15:10.662 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2023-10-16T12:15:10.663 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T12:15:10.663 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-16T12:15:10.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum upgrade -y linux-firmware 2023-10-16T12:15:13.130 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - AppStream 6.0 MB/s | 6.3 MB 00:01 2023-10-16T12:15:13.186 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.3 MB 00:00 2023-10-16T12:15:14.635 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2023-10-16T12:15:14.913 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2023-10-16T12:15:15.826 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 7.9 MB 00:00 2023-10-16T12:15:18.586 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 7.9 MB 00:03 2023-10-16T12:15:24.154 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 18 MB 00:04 2023-10-16T12:15:28.196 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 5.7 MB 00:00 2023-10-16T12:15:29.638 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 14 kB 00:00 2023-10-16T12:15:31.598 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 10 MB/s | 19 MB 00:01 2023-10-16T12:15:36.148 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-10-16T12:15:38.285 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-10-16T12:15:38.292 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-10-16T12:15:38.293 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2023-10-16T12:15:38.293 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-10-16T12:15:38.293 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2023-10-16T12:15:38.293 INFO:teuthology.orchestra.run.smithi144.stdout: linux-firmware noarch 20230926-140.el9 CentOS-BaseOS 326 M 2023-10-16T12:15:38.294 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:15:38.294 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-10-16T12:15:38.294 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-10-16T12:15:38.294 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 1 Package 2023-10-16T12:15:38.294 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:15:38.295 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 326 M 2023-10-16T12:15:38.295 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2023-10-16T12:15:47.356 INFO:teuthology.orchestra.run.smithi144.stdout:linux-firmware-20230926-140.el9.noarch.rpm 36 MB/s | 326 MB 00:09 2023-10-16T12:15:47.357 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2023-10-16T12:15:47.358 INFO:teuthology.orchestra.run.smithi144.stdout:Total 36 MB/s | 326 MB 00:09 2023-10-16T12:15:47.358 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2023-10-16T12:15:47.456 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2023-10-16T12:15:47.456 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2023-10-16T12:15:49.540 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2023-10-16T12:15:49.540 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2023-10-16T12:15:55.043 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2023-10-16T12:15:55.499 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T12:15:55.587 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T12:15:56.564 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T12:15:56.565 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T12:15:56.910 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T12:15:56.911 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:15:56.911 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2023-10-16T12:15:56.911 INFO:teuthology.orchestra.run.smithi144.stdout: linux-firmware-20230926-140.el9.noarch 2023-10-16T12:15:56.911 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:15:56.912 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-10-16T12:15:57.191 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.1... 2023-10-16T12:15:57.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T12:15:57.462 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ 2023-10-16T12:15:57.463 DEBUG:teuthology.orchestra.run.smithi144:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ --input-file=- 2023-10-16T12:15:57.465 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi144.front.sepia.ceph.com to complete... 2023-10-16T12:15:58.489 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 438 kB/s | 18 MB 00:41 2023-10-16T12:16:03.341 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.7 MB 00:01 2023-10-16T12:16:16.838 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16 12:16:16 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [320683515/320683515] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-16T12:16:16.838 INFO:teuthology.orchestra.run.smithi144.stderr:FINISHED --2023-10-16 12:16:16-- 2023-10-16T12:16:16.838 INFO:teuthology.orchestra.run.smithi144.stderr:Total wall clock time: 19s 2023-10-16T12:16:16.838 INFO:teuthology.orchestra.run.smithi144.stderr:Downloaded: 1 files, 306M in 19s (16.2 MB/s) 2023-10-16T12:16:16.840 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.1... 2023-10-16T12:16:16.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-16T12:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout:Verifying... ######################################## 2023-10-16T12:16:18.767 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing... ######################################## 2023-10-16T12:16:18.846 INFO:teuthology.orchestra.run.smithi144.stdout:Updating / installing... 2023-10-16T12:16:26.027 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-16T12:16:41.685 INFO:teuthology.orchestra.run.smithi144.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-16T12:16:46.386 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 352 B/s | 14 kB 00:41 2023-10-16T12:16:48.490 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 19 MB 00:01 2023-10-16T12:16:53.063 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2023-10-16T12:16:55.312 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-10-16T12:16:55.324 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-10-16T12:16:55.324 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-10-16T12:16:55.324 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-10-16T12:16:55.324 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2023-10-16T12:16:55.324 INFO:teuthology.orchestra.run.smithi060.stdout: linux-firmware noarch 20230926-140.el9 CentOS-BaseOS 326 M 2023-10-16T12:16:55.325 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:16:55.325 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-10-16T12:16:55.325 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-10-16T12:16:55.325 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 1 Package 2023-10-16T12:16:55.326 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:16:55.326 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 326 M 2023-10-16T12:16:55.326 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-10-16T12:17:02.906 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-16T12:17:02.907 DEBUG:teuthology.orchestra.run.smithi144:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-16T12:17:03.012 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.012 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.013 INFO:teuthology.orchestra.run.smithi144.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-16T12:17:03.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-16T12:17:03.013 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-16T12:17:03.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-16T12:17:03.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-16T12:17:03.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-16T12:17:03.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-16T12:17:03.014 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-16T12:17:03.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-16T12:17:03.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-16T12:17:03.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-16T12:17:03.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-16T12:17:03.015 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-16T12:17:03.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-16T12:17:03.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-16T12:17:03.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-16T12:17:03.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-16T12:17:03.016 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-16T12:17:03.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-16T12:17:03.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-16T12:17:03.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-16T12:17:03.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-16T12:17:03.017 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-16T12:17:03.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-16T12:17:03.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-16T12:17:03.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-16T12:17:03.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-16T12:17:03.018 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-16T12:17:03.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-16T12:17:03.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-16T12:17:03.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-16T12:17:03.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-16T12:17:03.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-16T12:17:03.019 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-16T12:17:03.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-16T12:17:03.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-16T12:17:03.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-16T12:17:03.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-16T12:17:03.020 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-16T12:17:03.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-16T12:17:03.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-16T12:17:03.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-16T12:17:03.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-16T12:17:03.021 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-16T12:17:03.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-16T12:17:03.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-16T12:17:03.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-16T12:17:03.022 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-16T12:17:03.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-16T12:17:03.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-16T12:17:03.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-16T12:17:03.023 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-16T12:17:03.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-16T12:17:03.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-16T12:17:03.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-16T12:17:03.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-16T12:17:03.024 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-16T12:17:03.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-16T12:17:03.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-16T12:17:03.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-16T12:17:03.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-16T12:17:03.025 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-16T12:17:03.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-16T12:17:03.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-16T12:17:03.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-16T12:17:03.026 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-16T12:17:03.027 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-16T12:17:03.027 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-16T12:17:03.027 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-16T12:17:03.027 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-16T12:17:03.028 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-16T12:17:03.028 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-16T12:17:03.028 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-16T12:17:03.028 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-16T12:17:03.028 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-16T12:17:03.028 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-16T12:17:03.029 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-16T12:17:03.029 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-16T12:17:03.029 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-16T12:17:03.029 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-16T12:17:03.029 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-16T12:17:03.030 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-16T12:17:03.030 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-16T12:17:03.030 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-16T12:17:03.030 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-16T12:17:03.030 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-16T12:17:03.031 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-16T12:17:03.031 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-16T12:17:03.031 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-16T12:17:03.031 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-16T12:17:03.031 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-16T12:17:03.031 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-16T12:17:03.032 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-16T12:17:03.032 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-16T12:17:03.032 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-16T12:17:03.032 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-16T12:17:03.032 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-16T12:17:03.033 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-16T12:17:03.033 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-16T12:17:03.033 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-16T12:17:03.033 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-16T12:17:03.033 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-16T12:17:03.034 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-16T12:17:03.034 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-16T12:17:03.034 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-16T12:17:03.034 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-16T12:17:03.034 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-16T12:17:03.035 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-16T12:17:03.035 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-16T12:17:03.035 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-16T12:17:03.035 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-16T12:17:03.035 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-16T12:17:03.036 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-16T12:17:03.036 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-16T12:17:03.036 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-16T12:17:03.036 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-16T12:17:03.036 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-16T12:17:03.037 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-16T12:17:03.037 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-16T12:17:03.037 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-16T12:17:03.037 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-16T12:17:03.037 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-16T12:17:03.038 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-16T12:17:03.038 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-16T12:17:03.038 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-16T12:17:03.038 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-16T12:17:03.038 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-16T12:17:03.039 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-16T12:17:03.039 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-16T12:17:03.039 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-16T12:17:03.039 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-16T12:17:03.039 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-16T12:17:03.040 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-16T12:17:03.040 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-16T12:17:03.040 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-16T12:17:03.040 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-16T12:17:03.040 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-16T12:17:03.040 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-16T12:17:03.041 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-16T12:17:03.041 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-16T12:17:03.041 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-16T12:17:03.041 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-16T12:17:03.041 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-16T12:17:03.042 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-16T12:17:03.042 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-16T12:17:03.042 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-16T12:17:03.042 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-16T12:17:03.042 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-16T12:17:03.043 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-16T12:17:03.043 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-16T12:17:03.043 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-16T12:17:03.043 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-16T12:17:03.043 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-16T12:17:03.044 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-16T12:17:03.044 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-16T12:17:03.044 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-16T12:17:03.044 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-16T12:17:03.044 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-16T12:17:03.044 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-16T12:17:03.045 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-16T12:17:03.045 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-16T12:17:03.045 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-16T12:17:03.045 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-16T12:17:03.045 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-16T12:17:03.045 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-16T12:17:03.046 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-16T12:17:03.046 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-16T12:17:03.046 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-16T12:17:03.046 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-16T12:17:03.046 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-16T12:17:03.047 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-16T12:17:03.047 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-16T12:17:03.047 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-16T12:17:03.047 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-16T12:17:03.047 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-16T12:17:03.047 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-16T12:17:03.048 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-16T12:17:03.048 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-16T12:17:03.048 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-16T12:17:03.048 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-16T12:17:03.048 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-16T12:17:03.049 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-16T12:17:03.049 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-16T12:17:03.049 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-16T12:17:03.049 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-16T12:17:03.049 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-16T12:17:03.050 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-16T12:17:03.050 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-16T12:17:03.050 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-16T12:17:03.050 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-16T12:17:03.051 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-16T12:17:03.051 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-16T12:17:03.051 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-16T12:17:03.051 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-16T12:17:03.051 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-16T12:17:03.052 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-16T12:17:03.052 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-16T12:17:03.052 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-16T12:17:03.052 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-16T12:17:03.052 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-16T12:17:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-16T12:17:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-16T12:17:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-16T12:17:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-16T12:17:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-16T12:17:03.053 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-16T12:17:03.054 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-16T12:17:03.054 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-16T12:17:03.054 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-16T12:17:03.054 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-16T12:17:03.055 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-16T12:17:03.055 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-16T12:17:03.055 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-16T12:17:03.055 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-16T12:17:03.055 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-16T12:17:03.056 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-16T12:17:03.056 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-16T12:17:03.056 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-16T12:17:03.056 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-16T12:17:03.056 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-16T12:17:03.057 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-16T12:17:03.057 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-16T12:17:03.057 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-16T12:17:03.057 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-16T12:17:03.057 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-16T12:17:03.058 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-16T12:17:03.058 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-16T12:17:03.058 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-16T12:17:03.058 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-16T12:17:03.058 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-16T12:17:03.059 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-16T12:17:03.059 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-16T12:17:03.059 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-16T12:17:03.059 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-16T12:17:03.059 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-16T12:17:03.060 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-16T12:17:03.060 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-16T12:17:03.060 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-16T12:17:03.060 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-16T12:17:03.060 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-16T12:17:03.061 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-16T12:17:03.061 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-16T12:17:03.061 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-16T12:17:03.061 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-16T12:17:03.061 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-16T12:17:03.061 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-16T12:17:03.062 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-16T12:17:03.062 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-16T12:17:03.062 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-16T12:17:03.062 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-16T12:17:03.062 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-16T12:17:03.063 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-16T12:17:03.063 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-16T12:17:03.063 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-16T12:17:03.063 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-16T12:17:03.063 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-16T12:17:03.064 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-16T12:17:03.064 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-16T12:17:03.064 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-16T12:17:03.064 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-16T12:17:03.064 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-16T12:17:03.064 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-16T12:17:03.065 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-16T12:17:03.065 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-16T12:17:03.065 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-16T12:17:03.065 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-16T12:17:03.065 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-16T12:17:03.066 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-16T12:17:03.066 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-16T12:17:03.066 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-16T12:17:03.066 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-16T12:17:03.066 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-16T12:17:03.067 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-16T12:17:03.067 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-16T12:17:03.067 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-16T12:17:03.067 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-16T12:17:03.067 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-16T12:17:03.068 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-16T12:17:03.068 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-16T12:17:03.068 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-16T12:17:03.068 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-16T12:17:03.068 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-16T12:17:03.069 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-16T12:17:03.069 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-16T12:17:03.069 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-16T12:17:03.069 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-16T12:17:03.069 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-16T12:17:03.070 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-16T12:17:03.070 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-16T12:17:03.070 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-16T12:17:03.070 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-16T12:17:03.071 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-16T12:17:03.071 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-16T12:17:03.071 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-16T12:17:03.071 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-16T12:17:03.071 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-16T12:17:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-16T12:17:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-16T12:17:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-16T12:17:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-16T12:17:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-16T12:17:03.072 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-16T12:17:03.073 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-16T12:17:03.073 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-16T12:17:03.073 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-16T12:17:03.073 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-16T12:17:03.073 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-16T12:17:03.074 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-16T12:17:03.074 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-16T12:17:03.074 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-16T12:17:03.074 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-16T12:17:03.074 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-16T12:17:03.075 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-16T12:17:03.075 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-16T12:17:03.075 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-16T12:17:03.075 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-16T12:17:03.075 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-16T12:17:03.076 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-16T12:17:03.076 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-16T12:17:03.076 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-16T12:17:03.076 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-16T12:17:03.076 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-16T12:17:03.077 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-16T12:17:03.077 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-16T12:17:03.077 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-16T12:17:03.077 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-16T12:17:03.077 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-16T12:17:03.078 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-16T12:17:03.078 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-16T12:17:03.078 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-16T12:17:03.078 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-16T12:17:03.078 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-16T12:17:03.078 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-16T12:17:03.079 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-16T12:17:03.079 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-16T12:17:03.079 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-16T12:17:03.079 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-16T12:17:03.079 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-16T12:17:03.080 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-16T12:17:03.080 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-16T12:17:03.080 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-16T12:17:03.080 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-16T12:17:03.080 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-16T12:17:03.081 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-16T12:17:03.081 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-16T12:17:03.081 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-16T12:17:03.081 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-16T12:17:03.081 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-16T12:17:03.082 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-16T12:17:03.082 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-16T12:17:03.082 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-16T12:17:03.082 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-16T12:17:03.082 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-16T12:17:03.083 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-16T12:17:03.083 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-16T12:17:03.083 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-16T12:17:03.083 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-16T12:17:03.083 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-16T12:17:03.083 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-16T12:17:03.084 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-16T12:17:03.084 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-16T12:17:03.084 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-16T12:17:03.084 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-16T12:17:03.084 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-16T12:17:03.085 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-16T12:17:03.085 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-16T12:17:03.085 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-16T12:17:03.085 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-16T12:17:03.085 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-16T12:17:03.086 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-16T12:17:03.086 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-16T12:17:03.086 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-16T12:17:03.086 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-16T12:17:03.086 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-16T12:17:03.087 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-16T12:17:03.087 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-16T12:17:03.087 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-16T12:17:03.087 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-16T12:17:03.087 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-16T12:17:03.087 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-16T12:17:03.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-16T12:17:03.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-16T12:17:03.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-16T12:17:03.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-16T12:17:03.088 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-16T12:17:03.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-16T12:17:03.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-16T12:17:03.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-16T12:17:03.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-16T12:17:03.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-16T12:17:03.089 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-16T12:17:03.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-16T12:17:03.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-16T12:17:03.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-16T12:17:03.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-16T12:17:03.090 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-16T12:17:03.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-16T12:17:03.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-16T12:17:03.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-16T12:17:03.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-16T12:17:03.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-16T12:17:03.091 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-16T12:17:03.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-16T12:17:03.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-16T12:17:03.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-16T12:17:03.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-16T12:17:03.092 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-16T12:17:03.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-16T12:17:03.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-16T12:17:03.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-16T12:17:03.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-16T12:17:03.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-16T12:17:03.093 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-16T12:17:03.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-16T12:17:03.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-16T12:17:03.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-16T12:17:03.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-16T12:17:03.094 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-16T12:17:03.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-16T12:17:03.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-16T12:17:03.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-16T12:17:03.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-16T12:17:03.095 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-16T12:17:03.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-16T12:17:03.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-16T12:17:03.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-16T12:17:03.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-16T12:17:03.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-16T12:17:03.096 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-16T12:17:03.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-16T12:17:03.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-16T12:17:03.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-16T12:17:03.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-16T12:17:03.097 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-16T12:17:03.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-16T12:17:03.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-16T12:17:03.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-16T12:17:03.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-16T12:17:03.098 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-16T12:17:03.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-16T12:17:03.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-16T12:17:03.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-16T12:17:03.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-16T12:17:03.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-16T12:17:03.099 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-16T12:17:03.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-16T12:17:03.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-16T12:17:03.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-16T12:17:03.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-16T12:17:03.100 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-16T12:17:03.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-16T12:17:03.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-16T12:17:03.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-16T12:17:03.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-16T12:17:03.101 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-16T12:17:03.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-16T12:17:03.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-16T12:17:03.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-16T12:17:03.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-16T12:17:03.102 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-16T12:17:03.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-16T12:17:03.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-16T12:17:03.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-16T12:17:03.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-16T12:17:03.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-16T12:17:03.103 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-16T12:17:03.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-16T12:17:03.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-16T12:17:03.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-16T12:17:03.104 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-16T12:17:03.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-16T12:17:03.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-16T12:17:03.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-16T12:17:03.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-16T12:17:03.105 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-16T12:17:03.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-16T12:17:03.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-16T12:17:03.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-16T12:17:03.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-16T12:17:03.106 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-16T12:17:03.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-16T12:17:03.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-16T12:17:03.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-16T12:17:03.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-16T12:17:03.107 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-16T12:17:03.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-16T12:17:03.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-16T12:17:03.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-16T12:17:03.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-16T12:17:03.108 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-16T12:17:03.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-16T12:17:03.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-16T12:17:03.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-16T12:17:03.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-16T12:17:03.109 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-16T12:17:03.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-16T12:17:03.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-16T12:17:03.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-16T12:17:03.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-16T12:17:03.110 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-16T12:17:03.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-16T12:17:03.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-16T12:17:03.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-16T12:17:03.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-16T12:17:03.111 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-16T12:17:03.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-16T12:17:03.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-16T12:17:03.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-16T12:17:03.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-16T12:17:03.112 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-16T12:17:03.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-16T12:17:03.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-16T12:17:03.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-16T12:17:03.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-16T12:17:03.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-16T12:17:03.113 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-16T12:17:03.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-16T12:17:03.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-16T12:17:03.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-16T12:17:03.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-16T12:17:03.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-16T12:17:03.114 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-16T12:17:03.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-16T12:17:03.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-16T12:17:03.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-16T12:17:03.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-16T12:17:03.115 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-16T12:17:03.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-16T12:17:03.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-16T12:17:03.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-16T12:17:03.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-16T12:17:03.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-16T12:17:03.116 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-16T12:17:03.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-16T12:17:03.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-16T12:17:03.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-16T12:17:03.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-16T12:17:03.117 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-16T12:17:03.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-16T12:17:03.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-16T12:17:03.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-16T12:17:03.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-16T12:17:03.118 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-16T12:17:03.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-16T12:17:03.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-16T12:17:03.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-16T12:17:03.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-16T12:17:03.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-16T12:17:03.119 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-16T12:17:03.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-16T12:17:03.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-16T12:17:03.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-16T12:17:03.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-16T12:17:03.120 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-16T12:17:03.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-16T12:17:03.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-16T12:17:03.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-16T12:17:03.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-16T12:17:03.121 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-16T12:17:03.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-16T12:17:03.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-16T12:17:03.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-16T12:17:03.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-16T12:17:03.122 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-16T12:17:03.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-16T12:17:03.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-16T12:17:03.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-16T12:17:03.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-16T12:17:03.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-16T12:17:03.123 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-16T12:17:03.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-16T12:17:03.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-16T12:17:03.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-16T12:17:03.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-16T12:17:03.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-16T12:17:03.124 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-16T12:17:03.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-16T12:17:03.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-16T12:17:03.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-16T12:17:03.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-16T12:17:03.125 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-16T12:17:03.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-16T12:17:03.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-16T12:17:03.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-16T12:17:03.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-16T12:17:03.126 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-16T12:17:03.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-16T12:17:03.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-16T12:17:03.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-16T12:17:03.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-16T12:17:03.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-16T12:17:03.127 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-16T12:17:03.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-16T12:17:03.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-16T12:17:03.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-16T12:17:03.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-16T12:17:03.128 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-16T12:17:03.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-16T12:17:03.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-16T12:17:03.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-16T12:17:03.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-16T12:17:03.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-16T12:17:03.129 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-16T12:17:03.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-16T12:17:03.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-16T12:17:03.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-16T12:17:03.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-16T12:17:03.130 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-16T12:17:03.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-16T12:17:03.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-16T12:17:03.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-16T12:17:03.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-16T12:17:03.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-16T12:17:03.131 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-16T12:17:03.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-16T12:17:03.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-16T12:17:03.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-16T12:17:03.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-16T12:17:03.132 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-16T12:17:03.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-16T12:17:03.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-16T12:17:03.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-16T12:17:03.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-16T12:17:03.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-16T12:17:03.133 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-16T12:17:03.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-16T12:17:03.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-16T12:17:03.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-16T12:17:03.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-16T12:17:03.134 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-16T12:17:03.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-16T12:17:03.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-16T12:17:03.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-16T12:17:03.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-16T12:17:03.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-16T12:17:03.135 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-16T12:17:03.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-16T12:17:03.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-16T12:17:03.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-16T12:17:03.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-16T12:17:03.136 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-16T12:17:03.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-16T12:17:03.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-16T12:17:03.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-16T12:17:03.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-16T12:17:03.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-16T12:17:03.137 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-16T12:17:03.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-16T12:17:03.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-16T12:17:03.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-16T12:17:03.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-16T12:17:03.138 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-16T12:17:03.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-16T12:17:03.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-16T12:17:03.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-16T12:17:03.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-16T12:17:03.139 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-16T12:17:03.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-16T12:17:03.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-16T12:17:03.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-16T12:17:03.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-16T12:17:03.140 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-16T12:17:03.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-16T12:17:03.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-16T12:17:03.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-16T12:17:03.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-16T12:17:03.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-16T12:17:03.141 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-16T12:17:03.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-16T12:17:03.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-16T12:17:03.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-16T12:17:03.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-16T12:17:03.142 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-16T12:17:03.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-16T12:17:03.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-16T12:17:03.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-16T12:17:03.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-16T12:17:03.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-16T12:17:03.143 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-16T12:17:03.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-16T12:17:03.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-16T12:17:03.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-16T12:17:03.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-16T12:17:03.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-16T12:17:03.144 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-16T12:17:03.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-16T12:17:03.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-16T12:17:03.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-16T12:17:03.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-16T12:17:03.145 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-16T12:17:03.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-16T12:17:03.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-16T12:17:03.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-16T12:17:03.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-16T12:17:03.146 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-16T12:17:03.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-16T12:17:03.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-16T12:17:03.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-16T12:17:03.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-16T12:17:03.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-16T12:17:03.147 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-16T12:17:03.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-16T12:17:03.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-16T12:17:03.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-16T12:17:03.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-16T12:17:03.148 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-16T12:17:03.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-16T12:17:03.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-16T12:17:03.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-16T12:17:03.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-16T12:17:03.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-16T12:17:03.149 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-16T12:17:03.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-16T12:17:03.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-16T12:17:03.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-16T12:17:03.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-16T12:17:03.150 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-16T12:17:03.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-16T12:17:03.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-16T12:17:03.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-16T12:17:03.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-16T12:17:03.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-16T12:17:03.151 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-16T12:17:03.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-16T12:17:03.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-16T12:17:03.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-16T12:17:03.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-16T12:17:03.152 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-16T12:17:03.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-16T12:17:03.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-16T12:17:03.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-16T12:17:03.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-16T12:17:03.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-16T12:17:03.153 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-16T12:17:03.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-16T12:17:03.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-16T12:17:03.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-16T12:17:03.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-16T12:17:03.154 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-16T12:17:03.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-16T12:17:03.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-16T12:17:03.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-16T12:17:03.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-16T12:17:03.155 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-16T12:17:03.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-16T12:17:03.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-16T12:17:03.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-16T12:17:03.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-16T12:17:03.156 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-16T12:17:03.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-16T12:17:03.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-16T12:17:03.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-16T12:17:03.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-16T12:17:03.157 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-16T12:17:03.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-16T12:17:03.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-16T12:17:03.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-16T12:17:03.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-16T12:17:03.158 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-16T12:17:03.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-16T12:17:03.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-16T12:17:03.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-16T12:17:03.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-16T12:17:03.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-16T12:17:03.159 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-16T12:17:03.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-16T12:17:03.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-16T12:17:03.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-16T12:17:03.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-16T12:17:03.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-16T12:17:03.160 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-16T12:17:03.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-16T12:17:03.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-16T12:17:03.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-16T12:17:03.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-16T12:17:03.161 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-16T12:17:03.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-16T12:17:03.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-16T12:17:03.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-16T12:17:03.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-16T12:17:03.162 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-16T12:17:03.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-16T12:17:03.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-16T12:17:03.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-16T12:17:03.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-16T12:17:03.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-16T12:17:03.163 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-16T12:17:03.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-16T12:17:03.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-16T12:17:03.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-16T12:17:03.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-16T12:17:03.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-16T12:17:03.164 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-16T12:17:03.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-16T12:17:03.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-16T12:17:03.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-16T12:17:03.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-16T12:17:03.165 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-16T12:17:03.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-16T12:17:03.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-16T12:17:03.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-16T12:17:03.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-16T12:17:03.166 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-16T12:17:03.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-16T12:17:03.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-16T12:17:03.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-16T12:17:03.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-16T12:17:03.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-16T12:17:03.167 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-16T12:17:03.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-16T12:17:03.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-16T12:17:03.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-16T12:17:03.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-16T12:17:03.168 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-16T12:17:03.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-16T12:17:03.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-16T12:17:03.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-16T12:17:03.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-16T12:17:03.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-16T12:17:03.169 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-16T12:17:03.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-16T12:17:03.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-16T12:17:03.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-16T12:17:03.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-16T12:17:03.170 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-16T12:17:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-16T12:17:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-16T12:17:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-16T12:17:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-16T12:17:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-16T12:17:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-16T12:17:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-16T12:17:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-16T12:17:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-16T12:17:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-16T12:17:03.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-16T12:17:03.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-16T12:17:03.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-16T12:17:03.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-16T12:17:03.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-16T12:17:03.173 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-16T12:17:03.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-16T12:17:03.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-16T12:17:03.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-16T12:17:03.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-16T12:17:03.174 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-16T12:17:03.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-16T12:17:03.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-16T12:17:03.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-16T12:17:03.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-16T12:17:03.175 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-16T12:17:03.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-16T12:17:03.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-16T12:17:03.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-16T12:17:03.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-16T12:17:03.176 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-16T12:17:03.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-16T12:17:03.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-16T12:17:03.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-16T12:17:03.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-16T12:17:03.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-16T12:17:03.177 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-16T12:17:03.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-16T12:17:03.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-16T12:17:03.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-16T12:17:03.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-16T12:17:03.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-16T12:17:03.178 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-16T12:17:03.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-16T12:17:03.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-16T12:17:03.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-16T12:17:03.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-16T12:17:03.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-16T12:17:03.179 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-16T12:17:03.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-16T12:17:03.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-16T12:17:03.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-16T12:17:03.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-16T12:17:03.180 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-16T12:17:03.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-16T12:17:03.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-16T12:17:03.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-16T12:17:03.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-16T12:17:03.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-16T12:17:03.181 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-16T12:17:03.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-16T12:17:03.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-16T12:17:03.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-16T12:17:03.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-16T12:17:03.182 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-16T12:17:03.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-16T12:17:03.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-16T12:17:03.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-16T12:17:03.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-16T12:17:03.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-16T12:17:03.183 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-16T12:17:03.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-16T12:17:03.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-16T12:17:03.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-16T12:17:03.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-16T12:17:03.184 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-16T12:17:03.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-16T12:17:03.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-16T12:17:03.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-16T12:17:03.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-16T12:17:03.185 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-16T12:17:03.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-16T12:17:03.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-16T12:17:03.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-16T12:17:03.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-16T12:17:03.186 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-16T12:17:03.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-16T12:17:03.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-16T12:17:03.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-16T12:17:03.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-16T12:17:03.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-16T12:17:03.187 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-16T12:17:03.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-16T12:17:03.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-16T12:17:03.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-16T12:17:03.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-16T12:17:03.188 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-16T12:17:03.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-16T12:17:03.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-16T12:17:03.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-16T12:17:03.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-16T12:17:03.189 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-16T12:17:03.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-16T12:17:03.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-16T12:17:03.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-16T12:17:03.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-16T12:17:03.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-16T12:17:03.190 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-16T12:17:03.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-16T12:17:03.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-16T12:17:03.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-16T12:17:03.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-16T12:17:03.191 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-16T12:17:03.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-16T12:17:03.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-16T12:17:03.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-16T12:17:03.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-16T12:17:03.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-16T12:17:03.192 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-16T12:17:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-16T12:17:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-16T12:17:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-16T12:17:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-16T12:17:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-16T12:17:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-16T12:17:03.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-16T12:17:03.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-16T12:17:03.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-16T12:17:03.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-16T12:17:03.194 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-16T12:17:03.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-16T12:17:03.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-16T12:17:03.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-16T12:17:03.195 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-16T12:17:03.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-16T12:17:03.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-16T12:17:03.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-16T12:17:03.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-16T12:17:03.196 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-16T12:17:03.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-16T12:17:03.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-16T12:17:03.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-16T12:17:03.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-16T12:17:03.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-16T12:17:03.197 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-16T12:17:03.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-16T12:17:03.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-16T12:17:03.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-16T12:17:03.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-16T12:17:03.198 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-16T12:17:03.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-16T12:17:03.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-16T12:17:03.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-16T12:17:03.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-16T12:17:03.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-16T12:17:03.199 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-16T12:17:03.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-16T12:17:03.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-16T12:17:03.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-16T12:17:03.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-16T12:17:03.200 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-16T12:17:03.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-16T12:17:03.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-16T12:17:03.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-16T12:17:03.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-16T12:17:03.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-16T12:17:03.201 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-16T12:17:03.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-16T12:17:03.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-16T12:17:03.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-16T12:17:03.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-16T12:17:03.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-16T12:17:03.202 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-16T12:17:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-16T12:17:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-16T12:17:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-16T12:17:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-16T12:17:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-16T12:17:03.203 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-16T12:17:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-16T12:17:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-16T12:17:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-16T12:17:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-16T12:17:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-16T12:17:03.204 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-16T12:17:03.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-16T12:17:03.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-16T12:17:03.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-16T12:17:03.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-16T12:17:03.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-16T12:17:03.205 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-16T12:17:03.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-16T12:17:03.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-16T12:17:03.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-16T12:17:03.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-16T12:17:03.206 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-16T12:17:03.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-16T12:17:03.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-16T12:17:03.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-16T12:17:03.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-16T12:17:03.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-16T12:17:03.207 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-16T12:17:03.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-16T12:17:03.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-16T12:17:03.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-16T12:17:03.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-16T12:17:03.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-16T12:17:03.208 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-16T12:17:03.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-16T12:17:03.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-16T12:17:03.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-16T12:17:03.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-16T12:17:03.209 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-16T12:17:03.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-16T12:17:03.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-16T12:17:03.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-16T12:17:03.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-16T12:17:03.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-16T12:17:03.210 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-16T12:17:03.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-16T12:17:03.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-16T12:17:03.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-16T12:17:03.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-16T12:17:03.211 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-16T12:17:03.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-16T12:17:03.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-16T12:17:03.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-16T12:17:03.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-16T12:17:03.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-16T12:17:03.212 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-16T12:17:03.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-16T12:17:03.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-16T12:17:03.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-16T12:17:03.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-16T12:17:03.213 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-16T12:17:03.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-16T12:17:03.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-16T12:17:03.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-16T12:17:03.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-16T12:17:03.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-16T12:17:03.214 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-16T12:17:03.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-16T12:17:03.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-16T12:17:03.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-16T12:17:03.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-16T12:17:03.215 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-16T12:17:03.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-16T12:17:03.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-16T12:17:03.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-16T12:17:03.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-16T12:17:03.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-16T12:17:03.216 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-16T12:17:03.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-16T12:17:03.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-16T12:17:03.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-16T12:17:03.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-16T12:17:03.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-16T12:17:03.217 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-16T12:17:03.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-16T12:17:03.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-16T12:17:03.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-16T12:17:03.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-16T12:17:03.218 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-16T12:17:03.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-16T12:17:03.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-16T12:17:03.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-16T12:17:03.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-16T12:17:03.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-16T12:17:03.219 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-16T12:17:03.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-16T12:17:03.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-16T12:17:03.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-16T12:17:03.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-16T12:17:03.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-16T12:17:03.220 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-16T12:17:03.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-16T12:17:03.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-16T12:17:03.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-16T12:17:03.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-16T12:17:03.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-16T12:17:03.221 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-16T12:17:03.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-16T12:17:03.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-16T12:17:03.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-16T12:17:03.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-16T12:17:03.222 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-16T12:17:03.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-16T12:17:03.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-16T12:17:03.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-16T12:17:03.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-16T12:17:03.223 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-16T12:17:03.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-16T12:17:03.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-16T12:17:03.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-16T12:17:03.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-16T12:17:03.224 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-16T12:17:03.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-16T12:17:03.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-16T12:17:03.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-16T12:17:03.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-16T12:17:03.225 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-16T12:17:03.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-16T12:17:03.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-16T12:17:03.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-16T12:17:03.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-16T12:17:03.226 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-16T12:17:03.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-16T12:17:03.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-16T12:17:03.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-16T12:17:03.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-16T12:17:03.227 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-16T12:17:03.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-16T12:17:03.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-16T12:17:03.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-16T12:17:03.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-16T12:17:03.228 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-16T12:17:03.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-16T12:17:03.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-16T12:17:03.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-16T12:17:03.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-16T12:17:03.229 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-16T12:17:03.230 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-16T12:17:03.230 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-16T12:17:03.230 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-16T12:17:03.230 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-16T12:17:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-16T12:17:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-16T12:17:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-16T12:17:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-16T12:17:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-16T12:17:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-16T12:17:03.232 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-16T12:17:03.232 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-16T12:17:03.232 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-16T12:17:03.232 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-16T12:17:03.232 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-16T12:17:03.233 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-16T12:17:03.233 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-16T12:17:03.233 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-16T12:17:03.233 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-16T12:17:03.233 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-16T12:17:03.234 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-16T12:17:03.234 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-16T12:17:03.234 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-16T12:17:03.235 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-16T12:17:03.235 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-16T12:17:03.235 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-16T12:17:03.236 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-16T12:17:03.236 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-16T12:17:03.236 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-16T12:17:03.236 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-16T12:17:03.236 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-16T12:17:03.237 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-16T12:17:03.237 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-16T12:17:03.237 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-16T12:17:03.237 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-16T12:17:03.237 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-16T12:17:03.237 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-16T12:17:03.238 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-16T12:17:03.238 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-16T12:17:03.238 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-16T12:17:03.238 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-16T12:17:03.238 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-16T12:17:03.239 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-16T12:17:03.239 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-16T12:17:03.239 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-16T12:17:03.239 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-16T12:17:03.239 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-16T12:17:03.240 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-16T12:17:03.240 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-16T12:17:03.240 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-16T12:17:03.240 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-16T12:17:03.240 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-16T12:17:03.240 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-16T12:17:03.241 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-16T12:17:03.241 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-16T12:17:03.241 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-16T12:17:03.241 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-16T12:17:03.241 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-16T12:17:03.242 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-16T12:17:03.242 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-16T12:17:03.242 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-16T12:17:03.242 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-16T12:17:03.242 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-16T12:17:03.242 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-16T12:17:03.243 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-16T12:17:03.243 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-16T12:17:03.243 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-16T12:17:03.243 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-16T12:17:03.243 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-16T12:17:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-16T12:17:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-16T12:17:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-16T12:17:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-16T12:17:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-16T12:17:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-16T12:17:03.245 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-16T12:17:03.245 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-16T12:17:03.245 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-16T12:17:03.245 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-16T12:17:03.245 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-16T12:17:03.246 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-16T12:17:03.246 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-16T12:17:03.246 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-16T12:17:03.246 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-16T12:17:03.246 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-16T12:17:03.247 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-16T12:17:03.247 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-16T12:17:03.247 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-16T12:17:03.247 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-16T12:17:03.247 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-16T12:17:03.248 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-16T12:17:03.248 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-16T12:17:03.248 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-16T12:17:03.248 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-16T12:17:03.248 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-16T12:17:03.248 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-16T12:17:03.249 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-16T12:17:03.249 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-16T12:17:03.249 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-16T12:17:03.249 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-16T12:17:03.249 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-16T12:17:03.250 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-16T12:17:03.250 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-16T12:17:03.250 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-16T12:17:03.250 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-16T12:17:03.250 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-16T12:17:03.250 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-16T12:17:03.251 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-16T12:17:03.251 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-16T12:17:03.251 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-16T12:17:03.251 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-16T12:17:03.251 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-16T12:17:03.251 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-16T12:17:03.252 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-16T12:17:03.252 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-16T12:17:03.252 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-16T12:17:03.252 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-16T12:17:03.252 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-16T12:17:03.253 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-16T12:17:03.253 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-16T12:17:03.253 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-16T12:17:03.253 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-16T12:17:03.253 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-16T12:17:03.254 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-16T12:17:03.254 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-16T12:17:03.254 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-16T12:17:03.254 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-16T12:17:03.254 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-16T12:17:03.254 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-16T12:17:03.255 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-16T12:17:03.255 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-16T12:17:03.255 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-16T12:17:03.255 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-16T12:17:03.255 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-16T12:17:03.256 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-16T12:17:03.256 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-16T12:17:03.256 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-16T12:17:03.256 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-16T12:17:03.256 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-16T12:17:03.257 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-16T12:17:03.257 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-16T12:17:03.257 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-16T12:17:03.257 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-16T12:17:03.257 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-16T12:17:03.257 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-16T12:17:03.258 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-16T12:17:03.258 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-16T12:17:03.258 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-16T12:17:03.258 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-16T12:17:03.258 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-16T12:17:03.259 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-16T12:17:03.259 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-16T12:17:03.259 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-16T12:17:03.259 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-16T12:17:03.259 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-16T12:17:03.260 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-16T12:17:03.260 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-16T12:17:03.260 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-16T12:17:03.260 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-16T12:17:03.260 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-16T12:17:03.261 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-16T12:17:03.261 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-16T12:17:03.261 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-16T12:17:03.261 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-16T12:17:03.261 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-16T12:17:03.262 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-16T12:17:03.262 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-16T12:17:03.262 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-16T12:17:03.262 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-16T12:17:03.262 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-16T12:17:03.263 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-16T12:17:03.263 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-16T12:17:03.263 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-16T12:17:03.263 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-16T12:17:03.263 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-16T12:17:03.263 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-16T12:17:03.264 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-16T12:17:03.264 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-16T12:17:03.264 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-16T12:17:03.264 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-16T12:17:03.264 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-16T12:17:03.265 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-16T12:17:03.265 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-16T12:17:03.265 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-16T12:17:03.265 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-16T12:17:03.265 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-16T12:17:03.265 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-16T12:17:03.266 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-16T12:17:03.266 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-16T12:17:03.266 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-16T12:17:03.266 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-16T12:17:03.266 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-16T12:17:03.267 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-16T12:17:03.267 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-16T12:17:03.267 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-16T12:17:03.267 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-16T12:17:03.267 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-16T12:17:03.268 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-16T12:17:03.268 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-16T12:17:03.268 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-16T12:17:03.268 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-16T12:17:03.268 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-16T12:17:03.269 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-16T12:17:03.269 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-16T12:17:03.269 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-16T12:17:03.269 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-16T12:17:03.269 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-16T12:17:03.270 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-16T12:17:03.270 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-16T12:17:03.270 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-16T12:17:03.270 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-16T12:17:03.270 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-16T12:17:03.270 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-16T12:17:03.271 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-16T12:17:03.271 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-16T12:17:03.271 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-16T12:17:03.271 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-16T12:17:03.271 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-16T12:17:03.272 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-16T12:17:03.272 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-16T12:17:03.272 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-16T12:17:03.272 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-16T12:17:03.272 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-16T12:17:03.273 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-16T12:17:03.273 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-16T12:17:03.273 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-16T12:17:03.273 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-16T12:17:03.273 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-16T12:17:03.273 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-16T12:17:03.274 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-16T12:17:03.274 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-16T12:17:03.274 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-16T12:17:03.274 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-16T12:17:03.274 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-16T12:17:03.275 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-16T12:17:03.275 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-16T12:17:03.275 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-16T12:17:03.275 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-16T12:17:03.275 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-16T12:17:03.276 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-16T12:17:03.276 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-16T12:17:03.276 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-16T12:17:03.276 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-16T12:17:03.276 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-16T12:17:03.277 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-16T12:17:03.277 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-16T12:17:03.277 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-16T12:17:03.277 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-16T12:17:03.277 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-16T12:17:03.277 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-16T12:17:03.278 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-16T12:17:03.278 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-16T12:17:03.278 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-16T12:17:03.278 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-16T12:17:03.279 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-16T12:17:03.279 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-16T12:17:03.279 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-16T12:17:03.279 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-16T12:17:03.279 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-16T12:17:03.280 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-16T12:17:03.280 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-16T12:17:03.280 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-16T12:17:03.280 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-16T12:17:03.280 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-16T12:17:03.281 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-16T12:17:03.281 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-16T12:17:03.281 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-16T12:17:03.281 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-16T12:17:03.281 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-16T12:17:03.282 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-16T12:17:03.282 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-16T12:17:03.282 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-16T12:17:03.282 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-16T12:17:03.282 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-16T12:17:03.282 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-16T12:17:03.283 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-16T12:17:03.283 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-16T12:17:03.283 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-16T12:17:03.283 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-16T12:17:03.284 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-16T12:17:03.284 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-16T12:17:03.284 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-16T12:17:03.284 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-16T12:17:03.284 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-16T12:17:03.284 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-16T12:17:03.285 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-16T12:17:03.285 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-16T12:17:03.285 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-16T12:17:03.285 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-16T12:17:03.285 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-16T12:17:03.286 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-16T12:17:03.286 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-16T12:17:03.286 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-16T12:17:03.286 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-16T12:17:03.286 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-16T12:17:03.287 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-16T12:17:03.287 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-16T12:17:03.287 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-16T12:17:03.287 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-16T12:17:03.287 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-16T12:17:03.288 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-16T12:17:03.288 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-16T12:17:03.288 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-16T12:17:03.288 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-16T12:17:03.288 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-16T12:17:03.289 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-16T12:17:03.289 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-16T12:17:03.289 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-16T12:17:03.289 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-16T12:17:03.289 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-16T12:17:03.289 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-16T12:17:03.290 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-16T12:17:03.290 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-16T12:17:03.290 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-16T12:17:03.290 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-16T12:17:03.290 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-16T12:17:03.291 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-16T12:17:03.291 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-16T12:17:03.291 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-16T12:17:03.291 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-16T12:17:03.291 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-16T12:17:03.291 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-16T12:17:03.292 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-16T12:17:03.292 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-16T12:17:03.292 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-16T12:17:03.292 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-16T12:17:03.292 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-16T12:17:03.293 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-16T12:17:03.293 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-16T12:17:03.293 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-16T12:17:03.293 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-16T12:17:03.293 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-16T12:17:03.293 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-16T12:17:03.294 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-16T12:17:03.294 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-16T12:17:03.294 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-16T12:17:03.294 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-16T12:17:03.294 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-16T12:17:03.295 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-16T12:17:03.295 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-16T12:17:03.295 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-16T12:17:03.295 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-16T12:17:03.295 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-16T12:17:03.296 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-16T12:17:03.296 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-16T12:17:03.296 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-16T12:17:03.296 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-16T12:17:03.296 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-16T12:17:03.296 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-16T12:17:03.297 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-16T12:17:03.297 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-16T12:17:03.297 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-16T12:17:03.297 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-16T12:17:03.298 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-16T12:17:03.298 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-16T12:17:03.298 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-16T12:17:03.298 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-16T12:17:03.298 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-16T12:17:03.299 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-16T12:17:03.299 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-16T12:17:03.299 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-16T12:17:03.299 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-16T12:17:03.299 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-16T12:17:03.300 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-16T12:17:03.300 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-16T12:17:03.300 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-16T12:17:03.300 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-16T12:17:03.300 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-16T12:17:03.300 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-16T12:17:03.301 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-16T12:17:03.301 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-16T12:17:03.301 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-16T12:17:03.301 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-16T12:17:03.301 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-16T12:17:03.302 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-16T12:17:03.302 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-16T12:17:03.302 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-16T12:17:03.302 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-16T12:17:03.302 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-16T12:17:03.303 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-16T12:17:03.303 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-16T12:17:03.303 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-16T12:17:03.303 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-16T12:17:03.303 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-16T12:17:03.303 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-16T12:17:03.304 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-16T12:17:03.304 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-16T12:17:03.304 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-16T12:17:03.304 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-16T12:17:03.304 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-16T12:17:03.305 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-16T12:17:03.305 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-16T12:17:03.305 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-16T12:17:03.305 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-16T12:17:03.305 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-16T12:17:03.305 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-16T12:17:03.306 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-16T12:17:03.306 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-16T12:17:03.306 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-16T12:17:03.306 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-16T12:17:03.306 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-16T12:17:03.307 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-16T12:17:03.307 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-16T12:17:03.307 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-16T12:17:03.307 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-16T12:17:03.307 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-16T12:17:03.307 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-16T12:17:03.308 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-16T12:17:03.308 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-16T12:17:03.308 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-16T12:17:03.308 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-16T12:17:03.308 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-16T12:17:03.309 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-16T12:17:03.309 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-16T12:17:03.309 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-16T12:17:03.309 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-16T12:17:03.309 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-16T12:17:03.310 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-16T12:17:03.310 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-16T12:17:03.310 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-16T12:17:03.310 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-16T12:17:03.310 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-16T12:17:03.311 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-16T12:17:03.311 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-16T12:17:03.311 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-16T12:17:03.311 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-16T12:17:03.311 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-16T12:17:03.311 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-16T12:17:03.312 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-16T12:17:03.312 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-16T12:17:03.312 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-16T12:17:03.312 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-16T12:17:03.312 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-16T12:17:03.313 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-16T12:17:03.313 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-16T12:17:03.313 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-16T12:17:03.313 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-16T12:17:03.313 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-16T12:17:03.313 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-16T12:17:03.314 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-16T12:17:03.314 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-16T12:17:03.314 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-16T12:17:03.314 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-16T12:17:03.314 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-16T12:17:03.315 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-16T12:17:03.315 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-16T12:17:03.315 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-16T12:17:03.315 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-16T12:17:03.315 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-16T12:17:03.316 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-16T12:17:03.316 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-16T12:17:03.316 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-16T12:17:03.316 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-16T12:17:03.316 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-16T12:17:03.316 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-16T12:17:03.317 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-16T12:17:03.317 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-16T12:17:03.317 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-16T12:17:03.317 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-16T12:17:03.317 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-16T12:17:03.318 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-16T12:17:03.318 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-16T12:17:03.318 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-16T12:17:03.318 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-16T12:17:03.318 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-16T12:17:03.319 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-16T12:17:03.319 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-16T12:17:03.319 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-16T12:17:03.319 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-16T12:17:03.319 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-16T12:17:03.319 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-16T12:17:03.320 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-16T12:17:03.320 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-16T12:17:03.320 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-16T12:17:03.320 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-16T12:17:03.320 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-16T12:17:03.321 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-16T12:17:03.321 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-16T12:17:03.321 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-16T12:17:03.321 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-16T12:17:03.321 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-16T12:17:03.322 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-16T12:17:03.322 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-16T12:17:03.322 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-16T12:17:03.322 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-16T12:17:03.322 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-16T12:17:03.323 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-16T12:17:03.323 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-16T12:17:03.323 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-16T12:17:03.323 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-16T12:17:03.323 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-16T12:17:03.324 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-16T12:17:03.324 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-16T12:17:03.324 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-16T12:17:03.324 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-16T12:17:03.324 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-16T12:17:03.324 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-16T12:17:03.325 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-16T12:17:03.325 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-16T12:17:03.325 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-16T12:17:03.325 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-16T12:17:03.325 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-16T12:17:03.326 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-16T12:17:03.326 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-16T12:17:03.326 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-16T12:17:03.326 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-16T12:17:03.326 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-16T12:17:03.327 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-16T12:17:03.327 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-16T12:17:03.327 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-16T12:17:03.327 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-16T12:17:03.327 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-16T12:17:03.328 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-16T12:17:03.328 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-16T12:17:03.328 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-16T12:17:03.328 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-16T12:17:03.328 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-16T12:17:03.328 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-16T12:17:03.329 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-16T12:17:03.329 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-16T12:17:03.329 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-16T12:17:03.329 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-16T12:17:03.329 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-16T12:17:03.330 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-16T12:17:03.330 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-16T12:17:03.330 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-16T12:17:03.330 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-16T12:17:03.330 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-16T12:17:03.331 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-16T12:17:03.331 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-16T12:17:03.331 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-16T12:17:03.331 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-16T12:17:03.331 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-16T12:17:03.332 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-16T12:17:03.332 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-16T12:17:03.332 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-16T12:17:03.332 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-16T12:17:03.332 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-16T12:17:03.332 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-16T12:17:03.333 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-16T12:17:03.333 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-16T12:17:03.333 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-16T12:17:03.333 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-16T12:17:03.333 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-16T12:17:03.334 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-16T12:17:03.334 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-16T12:17:03.334 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-16T12:17:03.334 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-16T12:17:03.334 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-16T12:17:03.334 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-16T12:17:03.335 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-16T12:17:03.335 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-16T12:17:03.335 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-16T12:17:03.335 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-16T12:17:03.335 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-16T12:17:03.336 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-16T12:17:03.336 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-16T12:17:03.336 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-16T12:17:03.336 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-16T12:17:03.336 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-16T12:17:03.336 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-16T12:17:03.337 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-16T12:17:03.337 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-16T12:17:03.337 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-16T12:17:03.337 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-16T12:17:03.337 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-16T12:17:03.337 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-16T12:17:03.338 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-16T12:17:03.338 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-16T12:17:03.338 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-16T12:17:03.338 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-16T12:17:03.338 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-16T12:17:03.339 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-16T12:17:03.339 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-16T12:17:03.339 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-16T12:17:03.339 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-16T12:17:03.339 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-16T12:17:03.339 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-16T12:17:03.340 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-16T12:17:03.340 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-16T12:17:03.340 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-16T12:17:03.340 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-16T12:17:03.340 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-16T12:17:03.341 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-16T12:17:03.341 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-16T12:17:03.341 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-16T12:17:03.341 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-16T12:17:03.341 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-16T12:17:03.341 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-16T12:17:03.342 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-16T12:17:03.342 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-16T12:17:03.342 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-16T12:17:03.342 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-16T12:17:03.342 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-16T12:17:03.343 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-16T12:17:03.343 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-16T12:17:03.343 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-16T12:17:03.343 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-16T12:17:03.343 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-16T12:17:03.344 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-16T12:17:03.344 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-16T12:17:03.344 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-16T12:17:03.344 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-16T12:17:03.344 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-16T12:17:03.345 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-16T12:17:03.345 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-16T12:17:03.345 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-16T12:17:03.345 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-16T12:17:03.345 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-16T12:17:03.346 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-16T12:17:03.346 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-16T12:17:03.346 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-16T12:17:03.346 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-16T12:17:03.347 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-16T12:17:03.347 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-16T12:17:03.347 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-16T12:17:03.347 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-16T12:17:03.347 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-16T12:17:03.348 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-16T12:17:03.348 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-16T12:17:03.348 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-16T12:17:03.348 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-16T12:17:03.348 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-16T12:17:03.349 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-16T12:17:03.349 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-16T12:17:03.349 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-16T12:17:03.349 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-16T12:17:03.350 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-16T12:17:03.350 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-16T12:17:03.350 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-16T12:17:03.350 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-16T12:17:03.350 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-16T12:17:03.351 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-16T12:17:03.351 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-16T12:17:03.351 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-16T12:17:03.351 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-16T12:17:03.352 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-16T12:17:03.352 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-16T12:17:03.352 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-16T12:17:03.352 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-16T12:17:03.352 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-16T12:17:03.352 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-16T12:17:03.353 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-16T12:17:03.353 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-16T12:17:03.353 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-16T12:17:03.353 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-16T12:17:03.353 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-16T12:17:03.354 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-16T12:17:03.354 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-16T12:17:03.354 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-16T12:17:03.354 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-16T12:17:03.354 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-16T12:17:03.355 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-16T12:17:03.355 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-16T12:17:03.355 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-16T12:17:03.355 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-16T12:17:03.355 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-16T12:17:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-16T12:17:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-16T12:17:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-16T12:17:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-16T12:17:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-16T12:17:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-16T12:17:03.357 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-16T12:17:03.357 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-16T12:17:03.357 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-16T12:17:03.357 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-16T12:17:03.357 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-16T12:17:03.357 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-16T12:17:03.358 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-16T12:17:03.358 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-16T12:17:03.358 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-16T12:17:03.358 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-16T12:17:03.358 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-16T12:17:03.359 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-16T12:17:03.359 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-16T12:17:03.359 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-16T12:17:03.359 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-16T12:17:03.359 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-16T12:17:03.360 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-16T12:17:03.360 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-16T12:17:03.360 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-16T12:17:03.360 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-16T12:17:03.360 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-16T12:17:03.360 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-16T12:17:03.361 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-16T12:17:03.361 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-16T12:17:03.361 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-16T12:17:03.361 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-16T12:17:03.361 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-16T12:17:03.362 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-16T12:17:03.362 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-16T12:17:03.362 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-16T12:17:03.362 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-16T12:17:03.362 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-16T12:17:03.363 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-16T12:17:03.363 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-16T12:17:03.363 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-16T12:17:03.363 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-16T12:17:03.363 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-16T12:17:03.363 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-16T12:17:03.364 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-16T12:17:03.364 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-16T12:17:03.364 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-16T12:17:03.364 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-16T12:17:03.364 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-16T12:17:03.365 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-16T12:17:03.365 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-16T12:17:03.365 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-16T12:17:03.365 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-16T12:17:03.365 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-16T12:17:03.366 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-16T12:17:03.366 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-16T12:17:03.366 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-16T12:17:03.366 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-16T12:17:03.366 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-16T12:17:03.367 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-16T12:17:03.367 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-16T12:17:03.367 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-16T12:17:03.367 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-16T12:17:03.367 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-16T12:17:03.367 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-16T12:17:03.368 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-16T12:17:03.368 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-16T12:17:03.368 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-16T12:17:03.368 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-16T12:17:03.368 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-16T12:17:03.369 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-16T12:17:03.369 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-16T12:17:03.369 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-16T12:17:03.369 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-16T12:17:03.369 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-16T12:17:03.369 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-16T12:17:03.370 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-16T12:17:03.370 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-16T12:17:03.370 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-16T12:17:03.370 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-16T12:17:03.370 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-16T12:17:03.371 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-16T12:17:03.371 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-16T12:17:03.371 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-16T12:17:03.371 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-16T12:17:03.371 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-16T12:17:03.372 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-16T12:17:03.372 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-16T12:17:03.372 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-16T12:17:03.372 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-16T12:17:03.372 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-16T12:17:03.372 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-16T12:17:03.373 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-16T12:17:03.373 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-16T12:17:03.373 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-16T12:17:03.373 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-16T12:17:03.373 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-16T12:17:03.374 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-16T12:17:03.374 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-16T12:17:03.374 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-16T12:17:03.374 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-16T12:17:03.374 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-16T12:17:03.375 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-16T12:17:03.375 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-16T12:17:03.375 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-16T12:17:03.375 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-16T12:17:03.375 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-16T12:17:03.376 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-16T12:17:03.376 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-16T12:17:03.376 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-16T12:17:03.376 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-16T12:17:03.376 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-16T12:17:03.376 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-16T12:17:03.377 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-16T12:17:03.377 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-16T12:17:03.377 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-16T12:17:03.377 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-16T12:17:03.377 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-16T12:17:03.378 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-16T12:17:03.378 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-16T12:17:03.378 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-16T12:17:03.378 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-16T12:17:03.378 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-16T12:17:03.379 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-16T12:17:03.379 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-16T12:17:03.379 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-16T12:17:03.379 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-16T12:17:03.379 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-16T12:17:03.379 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-16T12:17:03.380 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-16T12:17:03.380 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-16T12:17:03.380 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-16T12:17:03.380 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-16T12:17:03.380 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-16T12:17:03.380 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-16T12:17:03.381 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-16T12:17:03.381 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-16T12:17:03.381 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-16T12:17:03.381 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-16T12:17:03.381 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-16T12:17:03.382 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-16T12:17:03.382 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-16T12:17:03.382 INFO:teuthology.orchestra.run.smithi144.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-16T12:17:03.383 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.383 DEBUG:teuthology.orchestra.run.smithi144:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-16T12:17:03.407 INFO:teuthology.orchestra.run.smithi144.stdout:postinstall scriptlet (using /bin/sh): 2023-10-16T12:17:03.407 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-16T12:17:03.407 INFO:teuthology.orchestra.run.smithi144.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:17:03.407 INFO:teuthology.orchestra.run.smithi144.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:17:03.408 INFO:teuthology.orchestra.run.smithi144.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.408 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:17:03.408 INFO:teuthology.orchestra.run.smithi144.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:17:03.408 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2023-10-16T12:17:03.408 INFO:teuthology.orchestra.run.smithi144.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-16T12:17:03.409 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-16T12:17:03.409 INFO:teuthology.orchestra.run.smithi144.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-16T12:17:03.409 INFO:teuthology.orchestra.run.smithi144.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-16T12:17:03.409 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.409 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2023-10-16T12:17:03.410 INFO:teuthology.orchestra.run.smithi144.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-16T12:17:03.410 INFO:teuthology.orchestra.run.smithi144.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-16T12:17:03.410 INFO:teuthology.orchestra.run.smithi144.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.410 INFO:teuthology.orchestra.run.smithi144.stdout:fi 2023-10-16T12:17:03.411 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2023-10-16T12:17:03.488 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2023-10-16T12:17:03.488 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2023-10-16T12:17:03.488 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.06 2023-10-16T12:17:03.489 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 68.el9 2023-10-16T12:17:03.489 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2023-10-16T12:17:03.489 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-16T12:17:03.489 INFO:teuthology.orchestra.run.smithi144.stdout:Group : Unspecified 2023-10-16T12:17:03.489 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 8264254 2023-10-16T12:17:03.490 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2023-10-16T12:17:03.490 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-16T12:17:03.490 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-16T12:17:03.490 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-16T12:17:03.490 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-16T12:17:03.491 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : builder@centos.org 2023-10-16T12:17:03.491 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2023-10-16T12:17:03.491 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-16T12:17:03.491 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2023-10-16T12:17:03.491 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2023-10-16T12:17:03.492 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:17:03.492 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-16T12:17:03.492 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-16T12:17:03.492 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-16T12:17:03.492 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2023-10-16T12:17:03.493 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:17:03.493 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2023-10-16T12:17:03.494 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-16T12:17:03.494 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 6.6.0-rc5-g47672119149b 2023-10-16T12:17:03.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-16T12:17:03.672 INFO:teuthology.orchestra.run.smithi060.stdout:linux-firmware-20230926-140.el9.noarch.rpm 39 MB/s | 326 MB 00:08 2023-10-16T12:17:03.673 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-10-16T12:17:03.674 INFO:teuthology.orchestra.run.smithi060.stdout:Total 39 MB/s | 326 MB 00:08 2023-10-16T12:17:03.674 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-10-16T12:17:03.794 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-10-16T12:17:03.795 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-10-16T12:17:04.146 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2023-10-16T12:17:05.464 INFO:teuthology.orchestra.run.smithi144.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-16T12:17:05.511 INFO:teuthology.orchestra.run.smithi144.stderr:done 2023-10-16T12:17:05.513 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2023-10-16T12:17:05.528 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.HgGdKmqSaY 2023-10-16T12:17:05.529 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HgGdKmqSaY 2023-10-16T12:17:05.597 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.HgGdKmqSaY 2023-10-16T12:17:05.737 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.HgGdKmqSaY is 6KB 2023-10-16T12:17:05.786 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.HgGdKmqSaY 2023-10-16T12:17:05.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2023-10-16T12:17:05.865 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-16T12:17:05.865 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-16T12:17:05.866 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-16T12:17:05.866 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b.conf 2023-10-16T12:17:05.867 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b 2023-10-16T12:17:05.977 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-10-16T12:17:05.978 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-10-16T12:17:06.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2023-10-16T12:17:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-10-16T12:17:12.194 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T12:17:12.297 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T12:17:13.034 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T12:17:13.034 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-16T12:17:13.422 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-16T12:17:13.423 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:17:13.423 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2023-10-16T12:17:13.423 INFO:teuthology.orchestra.run.smithi060.stdout: linux-firmware-20230926-140.el9.noarch 2023-10-16T12:17:13.423 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:17:13.423 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-10-16T12:17:13.693 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-16T12:17:13.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T12:17:13.914 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ 2023-10-16T12:17:13.914 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ --input-file=- 2023-10-16T12:17:13.917 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi060.front.sepia.ceph.com to complete... 2023-10-16T12:17:24.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16 12:17:24 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [320683515/320683515] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-16T12:17:24.817 INFO:teuthology.orchestra.run.smithi060.stderr:FINISHED --2023-10-16 12:17:24-- 2023-10-16T12:17:24.817 INFO:teuthology.orchestra.run.smithi060.stderr:Total wall clock time: 11s 2023-10-16T12:17:24.817 INFO:teuthology.orchestra.run.smithi060.stderr:Downloaded: 1 files, 306M in 11s (28.9 MB/s) 2023-10-16T12:17:24.819 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-16T12:17:24.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-16T12:17:26.762 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying... ######################################## 2023-10-16T12:17:26.841 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing... ######################################## 2023-10-16T12:17:26.925 INFO:teuthology.orchestra.run.smithi060.stdout:Updating / installing... 2023-10-16T12:17:34.330 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-16T12:17:36.015 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-16T12:17:36.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-16T12:17:36.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:17:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-16T12:18:11.722 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-16T12:18:11.723 DEBUG:teuthology.orchestra.run.smithi060:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-16T12:18:11.791 INFO:teuthology.orchestra.run.smithi060.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T12:18:11.791 INFO:teuthology.orchestra.run.smithi060.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-16T12:18:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T12:18:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-16T12:18:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-16T12:18:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-16T12:18:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-16T12:18:11.793 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-16T12:18:11.793 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-16T12:18:11.793 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-16T12:18:11.793 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-16T12:18:11.793 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-16T12:18:11.794 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-16T12:18:11.794 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-16T12:18:11.794 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-16T12:18:11.794 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-16T12:18:11.794 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-16T12:18:11.795 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-16T12:18:11.795 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-16T12:18:11.795 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-16T12:18:11.795 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-16T12:18:11.795 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-16T12:18:11.796 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-16T12:18:11.796 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-16T12:18:11.796 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-16T12:18:11.796 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-16T12:18:11.796 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-16T12:18:11.797 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-16T12:18:11.797 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-16T12:18:11.797 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-16T12:18:11.797 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-16T12:18:11.797 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-16T12:18:11.798 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-16T12:18:11.798 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-16T12:18:11.798 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-16T12:18:11.798 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-16T12:18:11.798 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-16T12:18:11.799 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-16T12:18:11.799 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-16T12:18:11.799 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-16T12:18:11.799 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-16T12:18:11.799 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-16T12:18:11.800 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-16T12:18:11.800 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-16T12:18:11.800 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-16T12:18:11.800 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-16T12:18:11.800 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-16T12:18:11.801 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-16T12:18:11.801 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-16T12:18:11.801 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-16T12:18:11.801 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-16T12:18:11.801 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-16T12:18:11.801 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-16T12:18:11.802 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-16T12:18:11.802 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-16T12:18:11.802 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-16T12:18:11.802 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-16T12:18:11.802 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-16T12:18:11.803 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-16T12:18:11.803 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-16T12:18:11.803 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-16T12:18:11.803 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-16T12:18:11.803 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-16T12:18:11.803 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-16T12:18:11.804 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-16T12:18:11.804 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-16T12:18:11.804 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-16T12:18:11.804 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-16T12:18:11.804 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-16T12:18:11.805 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-16T12:18:11.805 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-16T12:18:11.805 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-16T12:18:11.805 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-16T12:18:11.805 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-16T12:18:11.805 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-16T12:18:11.806 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-16T12:18:11.806 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-16T12:18:11.806 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-16T12:18:11.806 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-16T12:18:11.806 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-16T12:18:11.807 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-16T12:18:11.807 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-16T12:18:11.807 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-16T12:18:11.807 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-16T12:18:11.807 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-16T12:18:11.808 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-16T12:18:11.808 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-16T12:18:11.808 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-16T12:18:11.808 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-16T12:18:11.808 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-16T12:18:11.808 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-16T12:18:11.809 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-16T12:18:11.809 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-16T12:18:11.809 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-16T12:18:11.809 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-16T12:18:11.809 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-16T12:18:11.810 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-16T12:18:11.810 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-16T12:18:11.810 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-16T12:18:11.810 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-16T12:18:11.810 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-16T12:18:11.811 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-16T12:18:11.811 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-16T12:18:11.811 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-16T12:18:11.811 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-16T12:18:11.811 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-16T12:18:11.811 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-16T12:18:11.812 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-16T12:18:11.812 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-16T12:18:11.812 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-16T12:18:11.812 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-16T12:18:11.812 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-16T12:18:11.813 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-16T12:18:11.813 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-16T12:18:11.813 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-16T12:18:11.813 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-16T12:18:11.813 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-16T12:18:11.813 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-16T12:18:11.814 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-16T12:18:11.814 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-16T12:18:11.814 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-16T12:18:11.814 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-16T12:18:11.814 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-16T12:18:11.815 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-16T12:18:11.815 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-16T12:18:11.815 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-16T12:18:11.815 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-16T12:18:11.815 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-16T12:18:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-16T12:18:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-16T12:18:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-16T12:18:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-16T12:18:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-16T12:18:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-16T12:18:11.817 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-16T12:18:11.817 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-16T12:18:11.817 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-16T12:18:11.817 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-16T12:18:11.817 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-16T12:18:11.818 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-16T12:18:11.818 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-16T12:18:11.818 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-16T12:18:11.818 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-16T12:18:11.818 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-16T12:18:11.818 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-16T12:18:11.819 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-16T12:18:11.819 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-16T12:18:11.819 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-16T12:18:11.819 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-16T12:18:11.819 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-16T12:18:11.820 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-16T12:18:11.820 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-16T12:18:11.820 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-16T12:18:11.820 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-16T12:18:11.820 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-16T12:18:11.821 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-16T12:18:11.821 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-16T12:18:11.821 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-16T12:18:11.821 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-16T12:18:11.821 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-16T12:18:11.822 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-16T12:18:11.822 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-16T12:18:11.822 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-16T12:18:11.822 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-16T12:18:11.822 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-16T12:18:11.823 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-16T12:18:11.823 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-16T12:18:11.823 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-16T12:18:11.823 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-16T12:18:11.823 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-16T12:18:11.823 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-16T12:18:11.824 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-16T12:18:11.824 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-16T12:18:11.824 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-16T12:18:11.824 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-16T12:18:11.824 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-16T12:18:11.825 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-16T12:18:11.825 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-16T12:18:11.825 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-16T12:18:11.825 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-16T12:18:11.825 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-16T12:18:11.825 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-16T12:18:11.826 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-16T12:18:11.826 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-16T12:18:11.826 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-16T12:18:11.826 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-16T12:18:11.826 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-16T12:18:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-16T12:18:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-16T12:18:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-16T12:18:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-16T12:18:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-16T12:18:11.828 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-16T12:18:11.828 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-16T12:18:11.828 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-16T12:18:11.828 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-16T12:18:11.828 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-16T12:18:11.828 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-16T12:18:11.829 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-16T12:18:11.829 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-16T12:18:11.829 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-16T12:18:11.829 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-16T12:18:11.829 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-16T12:18:11.830 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-16T12:18:11.830 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-16T12:18:11.830 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-16T12:18:11.830 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-16T12:18:11.830 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-16T12:18:11.831 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-16T12:18:11.831 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-16T12:18:11.831 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-16T12:18:11.831 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-16T12:18:11.831 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-16T12:18:11.831 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-16T12:18:11.832 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-16T12:18:11.832 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-16T12:18:11.832 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-16T12:18:11.832 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-16T12:18:11.832 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-16T12:18:11.832 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-16T12:18:11.833 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-16T12:18:11.833 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-16T12:18:11.833 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-16T12:18:11.833 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-16T12:18:11.833 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-16T12:18:11.834 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-16T12:18:11.834 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-16T12:18:11.834 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-16T12:18:11.834 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-16T12:18:11.834 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-16T12:18:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-16T12:18:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-16T12:18:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-16T12:18:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-16T12:18:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-16T12:18:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-16T12:18:11.836 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-16T12:18:11.836 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-16T12:18:11.836 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-16T12:18:11.836 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-16T12:18:11.836 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-16T12:18:11.837 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-16T12:18:11.837 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-16T12:18:11.837 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-16T12:18:11.837 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-16T12:18:11.837 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-16T12:18:11.837 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-16T12:18:11.838 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-16T12:18:11.838 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-16T12:18:11.838 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-16T12:18:11.838 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-16T12:18:11.838 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-16T12:18:11.839 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-16T12:18:11.839 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-16T12:18:11.839 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-16T12:18:11.839 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-16T12:18:11.839 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-16T12:18:11.840 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-16T12:18:11.840 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-16T12:18:11.840 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-16T12:18:11.840 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-16T12:18:11.840 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-16T12:18:11.841 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-16T12:18:11.841 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-16T12:18:11.841 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-16T12:18:11.841 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-16T12:18:11.841 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-16T12:18:11.841 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-16T12:18:11.842 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-16T12:18:11.842 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-16T12:18:11.842 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-16T12:18:11.842 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-16T12:18:11.842 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-16T12:18:11.843 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-16T12:18:11.843 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-16T12:18:11.843 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-16T12:18:11.843 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-16T12:18:11.843 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-16T12:18:11.843 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-16T12:18:11.844 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-16T12:18:11.844 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-16T12:18:11.844 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-16T12:18:11.844 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-16T12:18:11.844 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-16T12:18:11.845 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-16T12:18:11.845 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-16T12:18:11.845 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-16T12:18:11.845 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-16T12:18:11.845 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-16T12:18:11.845 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-16T12:18:11.846 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-16T12:18:11.846 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-16T12:18:11.846 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-16T12:18:11.846 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-16T12:18:11.846 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-16T12:18:11.847 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-16T12:18:11.847 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-16T12:18:11.847 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-16T12:18:11.847 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-16T12:18:11.847 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-16T12:18:11.848 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-16T12:18:11.848 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-16T12:18:11.848 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-16T12:18:11.848 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-16T12:18:11.848 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-16T12:18:11.848 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-16T12:18:11.849 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-16T12:18:11.849 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-16T12:18:11.849 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-16T12:18:11.849 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-16T12:18:11.849 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-16T12:18:11.850 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-16T12:18:11.850 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-16T12:18:11.850 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-16T12:18:11.850 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-16T12:18:11.850 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-16T12:18:11.851 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-16T12:18:11.851 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-16T12:18:11.851 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-16T12:18:11.851 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-16T12:18:11.851 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-16T12:18:11.852 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-16T12:18:11.852 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-16T12:18:11.852 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-16T12:18:11.852 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-16T12:18:11.852 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-16T12:18:11.852 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-16T12:18:11.853 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-16T12:18:11.853 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-16T12:18:11.853 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-16T12:18:11.853 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-16T12:18:11.853 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-16T12:18:11.854 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-16T12:18:11.854 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-16T12:18:11.854 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-16T12:18:11.854 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-16T12:18:11.854 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-16T12:18:11.854 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-16T12:18:11.855 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-16T12:18:11.855 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-16T12:18:11.855 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-16T12:18:11.855 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-16T12:18:11.855 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-16T12:18:11.856 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-16T12:18:11.856 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-16T12:18:11.856 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-16T12:18:11.856 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-16T12:18:11.856 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-16T12:18:11.856 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-16T12:18:11.857 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-16T12:18:11.857 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-16T12:18:11.857 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-16T12:18:11.857 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-16T12:18:11.857 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-16T12:18:11.858 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-16T12:18:11.858 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-16T12:18:11.858 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-16T12:18:11.858 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-16T12:18:11.858 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-16T12:18:11.859 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-16T12:18:11.859 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-16T12:18:11.859 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-16T12:18:11.859 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-16T12:18:11.859 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-16T12:18:11.860 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-16T12:18:11.860 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-16T12:18:11.860 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-16T12:18:11.860 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-16T12:18:11.860 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-16T12:18:11.860 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-16T12:18:11.861 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-16T12:18:11.861 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-16T12:18:11.861 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-16T12:18:11.861 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-16T12:18:11.861 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-16T12:18:11.862 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-16T12:18:11.862 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-16T12:18:11.862 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-16T12:18:11.862 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-16T12:18:11.862 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-16T12:18:11.863 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-16T12:18:11.863 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-16T12:18:11.863 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-16T12:18:11.863 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-16T12:18:11.863 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-16T12:18:11.864 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-16T12:18:11.864 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-16T12:18:11.864 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-16T12:18:11.864 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-16T12:18:11.864 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-16T12:18:11.865 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-16T12:18:11.865 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-16T12:18:11.865 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-16T12:18:11.865 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-16T12:18:11.865 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-16T12:18:11.865 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-16T12:18:11.866 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-16T12:18:11.866 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-16T12:18:11.866 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-16T12:18:11.866 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-16T12:18:11.866 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-16T12:18:11.867 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-16T12:18:11.867 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-16T12:18:11.867 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-16T12:18:11.867 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-16T12:18:11.867 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-16T12:18:11.867 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-16T12:18:11.868 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-16T12:18:11.868 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-16T12:18:11.868 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-16T12:18:11.868 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-16T12:18:11.868 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-16T12:18:11.869 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-16T12:18:11.869 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-16T12:18:11.869 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-16T12:18:11.869 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-16T12:18:11.869 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-16T12:18:11.869 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-16T12:18:11.870 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-16T12:18:11.870 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-16T12:18:11.870 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-16T12:18:11.870 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-16T12:18:11.870 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-16T12:18:11.871 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-16T12:18:11.871 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-16T12:18:11.871 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-16T12:18:11.871 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-16T12:18:11.871 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-16T12:18:11.872 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-16T12:18:11.872 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-16T12:18:11.872 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-16T12:18:11.872 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-16T12:18:11.872 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-16T12:18:11.872 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-16T12:18:11.873 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-16T12:18:11.873 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-16T12:18:11.873 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-16T12:18:11.873 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-16T12:18:11.873 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-16T12:18:11.873 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-16T12:18:11.874 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-16T12:18:11.874 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-16T12:18:11.874 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-16T12:18:11.874 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-16T12:18:11.874 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-16T12:18:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-16T12:18:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-16T12:18:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-16T12:18:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-16T12:18:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-16T12:18:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-16T12:18:11.876 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-16T12:18:11.876 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-16T12:18:11.876 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-16T12:18:11.876 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-16T12:18:11.876 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-16T12:18:11.876 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-16T12:18:11.877 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-16T12:18:11.877 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-16T12:18:11.877 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-16T12:18:11.877 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-16T12:18:11.877 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-16T12:18:11.878 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-16T12:18:11.878 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-16T12:18:11.878 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-16T12:18:11.878 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-16T12:18:11.878 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-16T12:18:11.879 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-16T12:18:11.879 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-16T12:18:11.879 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-16T12:18:11.879 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-16T12:18:11.879 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-16T12:18:11.879 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-16T12:18:11.880 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-16T12:18:11.880 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-16T12:18:11.880 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-16T12:18:11.880 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-16T12:18:11.880 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-16T12:18:11.881 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-16T12:18:11.881 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-16T12:18:11.881 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-16T12:18:11.881 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-16T12:18:11.881 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-16T12:18:11.881 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-16T12:18:11.882 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-16T12:18:11.882 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-16T12:18:11.882 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-16T12:18:11.882 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-16T12:18:11.882 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-16T12:18:11.883 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-16T12:18:11.883 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-16T12:18:11.883 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-16T12:18:11.883 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-16T12:18:11.883 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-16T12:18:11.884 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-16T12:18:11.884 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-16T12:18:11.884 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-16T12:18:11.884 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-16T12:18:11.884 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-16T12:18:11.885 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-16T12:18:11.885 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-16T12:18:11.885 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-16T12:18:11.885 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-16T12:18:11.885 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-16T12:18:11.885 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-16T12:18:11.886 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-16T12:18:11.886 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-16T12:18:11.886 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-16T12:18:11.886 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-16T12:18:11.886 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-16T12:18:11.887 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-16T12:18:11.887 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-16T12:18:11.887 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-16T12:18:11.887 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-16T12:18:11.887 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-16T12:18:11.887 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-16T12:18:11.888 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-16T12:18:11.888 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-16T12:18:11.888 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-16T12:18:11.888 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-16T12:18:11.888 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-16T12:18:11.889 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-16T12:18:11.889 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-16T12:18:11.889 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-16T12:18:11.889 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-16T12:18:11.889 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-16T12:18:11.889 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-16T12:18:11.890 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-16T12:18:11.890 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-16T12:18:11.890 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-16T12:18:11.890 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-16T12:18:11.890 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-16T12:18:11.891 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-16T12:18:11.891 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-16T12:18:11.891 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-16T12:18:11.891 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-16T12:18:11.891 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-16T12:18:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-16T12:18:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-16T12:18:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-16T12:18:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-16T12:18:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-16T12:18:11.892 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-16T12:18:11.893 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-16T12:18:11.893 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-16T12:18:11.893 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-16T12:18:11.893 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-16T12:18:11.893 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-16T12:18:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-16T12:18:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-16T12:18:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-16T12:18:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-16T12:18:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-16T12:18:11.895 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-16T12:18:11.895 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-16T12:18:11.895 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-16T12:18:11.895 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-16T12:18:11.895 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-16T12:18:11.896 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-16T12:18:11.896 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-16T12:18:11.896 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-16T12:18:11.896 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-16T12:18:11.896 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-16T12:18:11.896 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-16T12:18:11.897 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-16T12:18:11.897 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-16T12:18:11.897 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-16T12:18:11.897 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-16T12:18:11.897 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-16T12:18:11.898 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-16T12:18:11.898 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-16T12:18:11.898 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-16T12:18:11.898 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-16T12:18:11.898 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-16T12:18:11.898 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-16T12:18:11.899 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-16T12:18:11.899 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-16T12:18:11.899 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-16T12:18:11.899 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-16T12:18:11.899 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-16T12:18:11.900 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-16T12:18:11.900 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-16T12:18:11.900 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-16T12:18:11.900 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-16T12:18:11.900 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-16T12:18:11.901 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-16T12:18:11.901 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-16T12:18:11.901 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-16T12:18:11.901 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-16T12:18:11.901 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-16T12:18:11.901 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-16T12:18:11.902 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-16T12:18:11.902 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-16T12:18:11.902 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-16T12:18:11.902 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-16T12:18:11.902 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-16T12:18:11.903 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-16T12:18:11.903 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-16T12:18:11.903 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-16T12:18:11.903 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-16T12:18:11.903 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-16T12:18:11.903 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-16T12:18:11.904 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-16T12:18:11.904 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-16T12:18:11.904 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-16T12:18:11.904 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-16T12:18:11.904 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-16T12:18:11.904 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-16T12:18:11.905 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-16T12:18:11.905 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-16T12:18:11.905 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-16T12:18:11.905 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-16T12:18:11.905 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-16T12:18:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-16T12:18:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-16T12:18:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-16T12:18:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-16T12:18:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-16T12:18:11.907 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-16T12:18:11.907 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-16T12:18:11.907 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-16T12:18:11.907 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-16T12:18:11.907 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-16T12:18:11.907 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-16T12:18:11.908 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-16T12:18:11.908 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-16T12:18:11.908 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-16T12:18:11.908 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-16T12:18:11.908 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-16T12:18:11.909 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-16T12:18:11.909 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-16T12:18:11.909 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-16T12:18:11.909 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-16T12:18:11.909 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-16T12:18:11.910 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-16T12:18:11.910 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-16T12:18:11.910 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-16T12:18:11.910 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-16T12:18:11.910 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-16T12:18:11.911 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-16T12:18:11.911 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-16T12:18:11.911 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-16T12:18:11.911 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-16T12:18:11.911 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-16T12:18:11.911 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-16T12:18:11.912 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-16T12:18:11.912 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-16T12:18:11.912 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-16T12:18:11.912 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-16T12:18:11.912 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-16T12:18:11.912 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-16T12:18:11.913 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-16T12:18:11.913 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-16T12:18:11.913 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-16T12:18:11.913 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-16T12:18:11.913 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-16T12:18:11.914 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-16T12:18:11.914 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-16T12:18:11.914 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-16T12:18:11.914 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-16T12:18:11.914 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-16T12:18:11.914 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-16T12:18:11.915 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-16T12:18:11.915 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-16T12:18:11.915 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-16T12:18:11.915 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-16T12:18:11.915 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-16T12:18:11.916 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-16T12:18:11.916 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-16T12:18:11.916 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-16T12:18:11.916 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-16T12:18:11.916 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-16T12:18:11.916 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-16T12:18:11.917 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-16T12:18:11.917 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-16T12:18:11.917 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-16T12:18:11.917 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-16T12:18:11.917 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-16T12:18:11.917 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-16T12:18:11.918 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-16T12:18:11.918 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-16T12:18:11.918 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-16T12:18:11.918 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-16T12:18:11.918 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-16T12:18:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-16T12:18:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-16T12:18:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-16T12:18:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-16T12:18:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-16T12:18:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-16T12:18:11.920 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-16T12:18:11.920 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-16T12:18:11.920 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-16T12:18:11.920 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-16T12:18:11.920 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-16T12:18:11.921 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-16T12:18:11.921 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-16T12:18:11.921 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-16T12:18:11.921 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-16T12:18:11.921 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-16T12:18:11.921 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-16T12:18:11.922 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-16T12:18:11.922 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-16T12:18:11.922 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-16T12:18:11.922 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-16T12:18:11.922 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-16T12:18:11.923 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-16T12:18:11.923 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-16T12:18:11.923 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-16T12:18:11.923 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-16T12:18:11.923 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-16T12:18:11.924 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-16T12:18:11.924 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-16T12:18:11.924 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-16T12:18:11.924 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-16T12:18:11.924 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-16T12:18:11.924 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-16T12:18:11.925 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-16T12:18:11.925 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-16T12:18:11.925 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-16T12:18:11.925 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-16T12:18:11.925 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-16T12:18:11.926 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-16T12:18:11.926 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-16T12:18:11.926 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-16T12:18:11.926 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-16T12:18:11.926 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-16T12:18:11.927 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-16T12:18:11.927 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-16T12:18:11.927 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-16T12:18:11.927 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-16T12:18:11.927 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-16T12:18:11.927 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-16T12:18:11.928 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-16T12:18:11.928 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-16T12:18:11.928 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-16T12:18:11.928 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-16T12:18:11.928 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-16T12:18:11.929 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-16T12:18:11.929 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-16T12:18:11.929 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-16T12:18:11.929 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-16T12:18:11.929 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-16T12:18:11.930 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-16T12:18:11.930 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-16T12:18:11.930 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-16T12:18:11.930 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-16T12:18:11.930 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-16T12:18:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-16T12:18:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-16T12:18:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-16T12:18:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-16T12:18:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-16T12:18:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-16T12:18:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-16T12:18:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-16T12:18:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-16T12:18:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-16T12:18:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-16T12:18:11.933 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-16T12:18:11.933 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-16T12:18:11.933 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-16T12:18:11.933 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-16T12:18:11.933 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-16T12:18:11.934 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-16T12:18:11.934 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-16T12:18:11.934 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-16T12:18:11.934 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-16T12:18:11.934 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-16T12:18:11.935 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-16T12:18:11.935 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-16T12:18:11.935 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-16T12:18:11.935 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-16T12:18:11.935 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-16T12:18:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-16T12:18:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-16T12:18:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-16T12:18:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-16T12:18:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-16T12:18:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-16T12:18:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-16T12:18:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-16T12:18:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-16T12:18:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-16T12:18:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-16T12:18:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-16T12:18:11.938 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-16T12:18:11.938 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-16T12:18:11.938 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-16T12:18:11.938 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-16T12:18:11.938 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-16T12:18:11.939 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-16T12:18:11.939 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-16T12:18:11.939 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-16T12:18:11.939 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-16T12:18:11.939 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-16T12:18:11.940 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-16T12:18:11.940 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-16T12:18:11.940 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-16T12:18:11.940 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-16T12:18:11.940 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-16T12:18:11.940 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-16T12:18:11.941 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-16T12:18:11.941 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-16T12:18:11.941 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-16T12:18:11.941 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-16T12:18:11.941 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-16T12:18:11.942 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-16T12:18:11.942 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-16T12:18:11.942 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-16T12:18:11.942 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-16T12:18:11.942 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-16T12:18:11.943 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-16T12:18:11.943 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-16T12:18:11.943 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-16T12:18:11.943 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-16T12:18:11.943 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-16T12:18:11.943 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-16T12:18:11.944 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-16T12:18:11.944 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-16T12:18:11.944 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-16T12:18:11.944 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-16T12:18:11.944 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-16T12:18:11.945 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-16T12:18:11.945 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-16T12:18:11.945 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-16T12:18:11.945 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-16T12:18:11.945 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-16T12:18:11.945 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-16T12:18:11.946 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-16T12:18:11.946 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-16T12:18:11.946 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-16T12:18:11.946 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-16T12:18:11.946 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-16T12:18:11.947 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-16T12:18:11.947 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-16T12:18:11.947 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-16T12:18:11.947 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-16T12:18:11.947 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-16T12:18:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-16T12:18:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-16T12:18:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-16T12:18:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-16T12:18:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-16T12:18:11.948 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-16T12:18:11.949 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-16T12:18:11.949 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-16T12:18:11.949 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-16T12:18:11.949 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-16T12:18:11.949 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-16T12:18:11.950 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-16T12:18:11.950 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-16T12:18:11.950 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-16T12:18:11.950 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-16T12:18:11.950 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-16T12:18:11.951 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-16T12:18:11.951 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-16T12:18:11.951 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-16T12:18:11.951 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-16T12:18:11.951 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-16T12:18:11.952 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-16T12:18:11.952 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-16T12:18:11.952 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-16T12:18:11.952 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-16T12:18:11.952 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-16T12:18:11.952 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-16T12:18:11.953 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-16T12:18:11.953 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-16T12:18:11.953 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-16T12:18:11.953 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-16T12:18:11.953 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-16T12:18:11.954 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-16T12:18:11.954 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-16T12:18:11.954 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-16T12:18:11.954 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-16T12:18:11.954 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-16T12:18:11.954 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-16T12:18:11.955 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-16T12:18:11.955 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-16T12:18:11.955 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-16T12:18:11.955 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-16T12:18:11.955 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-16T12:18:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-16T12:18:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-16T12:18:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-16T12:18:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-16T12:18:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-16T12:18:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-16T12:18:11.957 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-16T12:18:11.957 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-16T12:18:11.957 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-16T12:18:11.957 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-16T12:18:11.957 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-16T12:18:11.958 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-16T12:18:11.958 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-16T12:18:11.958 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-16T12:18:11.958 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-16T12:18:11.958 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-16T12:18:11.959 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-16T12:18:11.959 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-16T12:18:11.959 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-16T12:18:11.959 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-16T12:18:11.959 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-16T12:18:11.959 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-16T12:18:11.960 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-16T12:18:11.960 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-16T12:18:11.960 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-16T12:18:11.960 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-16T12:18:11.960 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-16T12:18:11.961 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-16T12:18:11.961 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-16T12:18:11.961 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-16T12:18:11.961 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-16T12:18:11.961 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-16T12:18:11.962 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-16T12:18:11.962 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-16T12:18:11.962 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-16T12:18:11.962 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-16T12:18:11.962 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-16T12:18:11.963 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-16T12:18:11.963 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-16T12:18:11.963 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-16T12:18:11.963 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-16T12:18:11.963 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-16T12:18:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-16T12:18:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-16T12:18:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-16T12:18:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-16T12:18:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-16T12:18:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-16T12:18:11.965 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-16T12:18:11.965 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-16T12:18:11.965 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-16T12:18:11.965 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-16T12:18:11.965 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-16T12:18:11.966 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-16T12:18:11.966 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-16T12:18:11.966 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-16T12:18:11.966 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-16T12:18:11.966 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-16T12:18:11.967 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-16T12:18:11.967 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-16T12:18:11.967 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-16T12:18:11.967 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-16T12:18:11.967 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-16T12:18:11.968 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-16T12:18:11.968 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-16T12:18:11.968 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-16T12:18:11.968 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-16T12:18:11.968 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-16T12:18:11.968 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-16T12:18:11.969 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-16T12:18:11.969 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-16T12:18:11.969 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-16T12:18:11.969 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-16T12:18:11.969 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-16T12:18:11.970 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-16T12:18:11.970 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-16T12:18:11.970 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-16T12:18:11.970 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-16T12:18:11.970 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-16T12:18:11.970 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-16T12:18:11.971 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-16T12:18:11.971 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-16T12:18:11.971 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-16T12:18:11.971 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-16T12:18:11.971 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-16T12:18:11.972 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-16T12:18:11.972 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-16T12:18:11.972 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-16T12:18:11.972 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-16T12:18:11.972 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-16T12:18:11.972 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-16T12:18:11.973 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-16T12:18:11.973 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-16T12:18:11.973 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-16T12:18:11.973 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-16T12:18:11.973 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-16T12:18:11.974 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-16T12:18:11.974 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-16T12:18:11.974 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-16T12:18:11.974 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-16T12:18:11.974 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-16T12:18:11.974 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-16T12:18:11.975 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-16T12:18:11.975 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-16T12:18:11.975 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-16T12:18:11.975 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-16T12:18:11.975 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-16T12:18:11.976 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-16T12:18:11.976 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-16T12:18:11.976 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-16T12:18:11.976 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-16T12:18:11.976 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-16T12:18:11.977 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-16T12:18:11.977 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-16T12:18:11.977 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-16T12:18:11.977 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-16T12:18:11.977 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-16T12:18:11.977 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-16T12:18:11.978 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-16T12:18:11.978 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-16T12:18:11.978 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-16T12:18:11.978 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-16T12:18:11.978 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-16T12:18:11.979 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-16T12:18:11.979 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-16T12:18:11.979 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-16T12:18:11.979 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-16T12:18:11.979 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-16T12:18:11.980 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-16T12:18:11.980 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-16T12:18:11.980 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-16T12:18:11.980 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-16T12:18:11.980 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-16T12:18:11.980 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-16T12:18:11.981 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-16T12:18:11.981 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-16T12:18:11.981 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-16T12:18:11.981 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-16T12:18:11.981 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-16T12:18:11.982 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-16T12:18:11.982 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-16T12:18:11.982 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-16T12:18:11.982 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-16T12:18:11.982 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-16T12:18:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-16T12:18:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-16T12:18:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-16T12:18:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-16T12:18:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-16T12:18:11.983 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-16T12:18:11.984 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-16T12:18:11.984 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-16T12:18:11.984 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-16T12:18:11.984 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-16T12:18:11.984 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-16T12:18:11.984 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-16T12:18:11.985 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-16T12:18:11.985 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-16T12:18:11.985 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-16T12:18:11.985 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-16T12:18:11.985 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-16T12:18:11.986 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-16T12:18:11.986 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-16T12:18:11.986 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-16T12:18:11.986 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-16T12:18:11.986 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-16T12:18:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-16T12:18:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-16T12:18:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-16T12:18:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-16T12:18:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-16T12:18:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-16T12:18:11.988 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-16T12:18:11.988 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-16T12:18:11.988 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-16T12:18:11.988 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-16T12:18:11.988 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-16T12:18:11.988 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-16T12:18:11.989 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-16T12:18:11.989 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-16T12:18:11.989 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-16T12:18:11.989 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-16T12:18:11.989 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-16T12:18:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-16T12:18:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-16T12:18:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-16T12:18:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-16T12:18:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-16T12:18:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-16T12:18:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-16T12:18:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-16T12:18:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-16T12:18:11.991 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-16T12:18:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-16T12:18:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-16T12:18:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-16T12:18:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-16T12:18:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-16T12:18:11.992 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-16T12:18:11.993 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-16T12:18:11.993 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-16T12:18:11.993 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-16T12:18:11.993 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-16T12:18:11.993 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-16T12:18:11.994 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-16T12:18:11.994 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-16T12:18:11.994 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-16T12:18:11.994 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-16T12:18:11.994 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-16T12:18:11.995 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-16T12:18:11.995 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-16T12:18:11.995 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-16T12:18:11.995 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-16T12:18:11.995 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-16T12:18:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-16T12:18:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-16T12:18:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-16T12:18:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-16T12:18:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-16T12:18:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-16T12:18:11.997 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-16T12:18:11.997 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-16T12:18:11.997 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-16T12:18:11.997 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-16T12:18:11.997 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-16T12:18:11.998 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-16T12:18:11.998 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-16T12:18:11.998 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-16T12:18:11.998 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-16T12:18:11.998 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-16T12:18:11.999 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-16T12:18:11.999 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-16T12:18:11.999 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-16T12:18:11.999 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-16T12:18:11.999 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-16T12:18:11.999 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-16T12:18:12.000 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-16T12:18:12.000 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-16T12:18:12.000 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-16T12:18:12.000 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-16T12:18:12.000 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-16T12:18:12.001 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-16T12:18:12.001 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-16T12:18:12.001 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-16T12:18:12.001 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-16T12:18:12.001 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-16T12:18:12.001 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-16T12:18:12.002 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-16T12:18:12.002 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-16T12:18:12.002 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-16T12:18:12.002 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-16T12:18:12.002 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-16T12:18:12.003 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-16T12:18:12.003 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-16T12:18:12.003 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-16T12:18:12.003 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-16T12:18:12.003 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-16T12:18:12.004 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-16T12:18:12.004 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-16T12:18:12.004 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-16T12:18:12.004 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-16T12:18:12.004 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-16T12:18:12.005 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-16T12:18:12.005 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-16T12:18:12.005 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-16T12:18:12.005 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-16T12:18:12.005 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-16T12:18:12.005 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-16T12:18:12.006 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-16T12:18:12.006 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-16T12:18:12.006 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-16T12:18:12.006 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-16T12:18:12.006 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-16T12:18:12.007 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-16T12:18:12.007 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-16T12:18:12.007 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-16T12:18:12.007 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-16T12:18:12.007 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-16T12:18:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-16T12:18:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-16T12:18:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-16T12:18:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-16T12:18:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-16T12:18:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-16T12:18:12.009 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-16T12:18:12.009 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-16T12:18:12.009 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-16T12:18:12.009 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-16T12:18:12.009 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-16T12:18:12.010 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-16T12:18:12.010 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-16T12:18:12.010 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-16T12:18:12.010 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-16T12:18:12.010 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-16T12:18:12.011 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-16T12:18:12.011 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-16T12:18:12.011 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-16T12:18:12.011 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-16T12:18:12.011 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-16T12:18:12.012 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-16T12:18:12.012 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-16T12:18:12.012 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-16T12:18:12.012 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-16T12:18:12.012 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-16T12:18:12.012 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-16T12:18:12.013 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-16T12:18:12.013 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-16T12:18:12.013 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-16T12:18:12.013 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-16T12:18:12.013 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-16T12:18:12.013 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-16T12:18:12.014 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-16T12:18:12.014 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-16T12:18:12.014 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-16T12:18:12.014 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-16T12:18:12.014 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-16T12:18:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-16T12:18:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-16T12:18:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-16T12:18:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-16T12:18:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-16T12:18:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-16T12:18:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-16T12:18:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-16T12:18:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-16T12:18:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-16T12:18:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-16T12:18:12.016 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-16T12:18:12.017 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-16T12:18:12.017 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-16T12:18:12.017 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-16T12:18:12.017 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-16T12:18:12.017 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-16T12:18:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-16T12:18:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-16T12:18:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-16T12:18:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-16T12:18:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-16T12:18:12.018 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-16T12:18:12.019 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-16T12:18:12.019 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-16T12:18:12.019 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-16T12:18:12.019 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-16T12:18:12.019 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-16T12:18:12.020 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-16T12:18:12.020 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-16T12:18:12.020 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-16T12:18:12.020 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-16T12:18:12.020 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-16T12:18:12.020 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-16T12:18:12.021 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-16T12:18:12.021 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-16T12:18:12.021 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-16T12:18:12.021 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-16T12:18:12.021 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-16T12:18:12.022 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-16T12:18:12.022 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-16T12:18:12.022 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-16T12:18:12.022 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-16T12:18:12.022 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-16T12:18:12.022 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-16T12:18:12.023 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-16T12:18:12.023 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-16T12:18:12.023 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-16T12:18:12.023 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-16T12:18:12.023 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-16T12:18:12.024 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-16T12:18:12.024 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-16T12:18:12.024 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-16T12:18:12.024 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-16T12:18:12.024 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-16T12:18:12.024 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-16T12:18:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-16T12:18:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-16T12:18:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-16T12:18:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-16T12:18:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-16T12:18:12.026 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-16T12:18:12.026 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-16T12:18:12.026 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-16T12:18:12.026 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-16T12:18:12.026 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-16T12:18:12.027 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-16T12:18:12.027 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-16T12:18:12.027 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-16T12:18:12.027 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-16T12:18:12.027 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-16T12:18:12.027 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-16T12:18:12.028 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-16T12:18:12.028 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-16T12:18:12.028 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-16T12:18:12.028 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-16T12:18:12.028 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-16T12:18:12.029 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-16T12:18:12.029 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-16T12:18:12.029 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-16T12:18:12.029 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-16T12:18:12.029 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-16T12:18:12.030 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-16T12:18:12.030 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-16T12:18:12.030 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-16T12:18:12.030 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-16T12:18:12.030 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-16T12:18:12.030 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-16T12:18:12.031 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-16T12:18:12.031 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-16T12:18:12.031 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-16T12:18:12.031 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-16T12:18:12.031 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-16T12:18:12.032 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-16T12:18:12.032 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-16T12:18:12.032 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-16T12:18:12.032 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-16T12:18:12.032 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-16T12:18:12.033 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-16T12:18:12.033 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-16T12:18:12.033 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-16T12:18:12.033 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-16T12:18:12.033 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-16T12:18:12.033 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-16T12:18:12.034 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-16T12:18:12.034 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-16T12:18:12.034 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-16T12:18:12.034 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-16T12:18:12.034 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-16T12:18:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-16T12:18:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-16T12:18:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-16T12:18:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-16T12:18:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-16T12:18:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-16T12:18:12.036 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-16T12:18:12.036 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-16T12:18:12.036 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-16T12:18:12.036 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-16T12:18:12.036 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-16T12:18:12.036 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-16T12:18:12.037 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-16T12:18:12.037 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-16T12:18:12.037 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-16T12:18:12.037 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-16T12:18:12.037 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-16T12:18:12.038 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-16T12:18:12.038 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-16T12:18:12.038 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-16T12:18:12.038 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-16T12:18:12.038 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-16T12:18:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-16T12:18:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-16T12:18:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-16T12:18:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-16T12:18:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-16T12:18:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-16T12:18:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-16T12:18:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-16T12:18:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-16T12:18:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-16T12:18:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-16T12:18:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-16T12:18:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-16T12:18:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-16T12:18:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-16T12:18:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-16T12:18:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-16T12:18:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-16T12:18:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-16T12:18:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-16T12:18:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-16T12:18:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-16T12:18:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-16T12:18:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-16T12:18:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-16T12:18:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-16T12:18:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-16T12:18:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-16T12:18:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-16T12:18:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-16T12:18:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-16T12:18:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-16T12:18:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-16T12:18:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-16T12:18:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-16T12:18:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-16T12:18:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-16T12:18:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-16T12:18:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-16T12:18:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-16T12:18:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-16T12:18:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-16T12:18:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-16T12:18:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-16T12:18:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-16T12:18:12.047 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-16T12:18:12.047 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-16T12:18:12.047 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-16T12:18:12.047 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-16T12:18:12.047 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-16T12:18:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-16T12:18:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-16T12:18:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-16T12:18:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-16T12:18:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-16T12:18:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-16T12:18:12.049 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-16T12:18:12.049 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-16T12:18:12.049 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-16T12:18:12.049 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-16T12:18:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-16T12:18:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-16T12:18:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-16T12:18:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-16T12:18:12.050 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-16T12:18:12.051 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-16T12:18:12.051 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-16T12:18:12.051 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-16T12:18:12.051 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-16T12:18:12.051 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-16T12:18:12.051 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-16T12:18:12.052 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-16T12:18:12.052 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-16T12:18:12.052 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-16T12:18:12.052 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-16T12:18:12.052 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-16T12:18:12.052 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-16T12:18:12.053 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-16T12:18:12.053 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-16T12:18:12.053 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-16T12:18:12.053 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-16T12:18:12.053 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-16T12:18:12.054 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-16T12:18:12.054 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-16T12:18:12.054 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-16T12:18:12.054 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-16T12:18:12.054 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-16T12:18:12.054 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-16T12:18:12.055 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-16T12:18:12.055 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-16T12:18:12.055 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-16T12:18:12.057 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-16T12:18:12.058 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-16T12:18:12.058 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-16T12:18:12.058 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-16T12:18:12.058 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-16T12:18:12.058 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-16T12:18:12.059 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-16T12:18:12.059 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-16T12:18:12.059 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-16T12:18:12.059 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-16T12:18:12.059 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-16T12:18:12.060 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-16T12:18:12.060 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-16T12:18:12.060 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-16T12:18:12.060 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-16T12:18:12.060 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-16T12:18:12.060 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-16T12:18:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-16T12:18:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-16T12:18:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-16T12:18:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-16T12:18:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-16T12:18:12.062 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-16T12:18:12.062 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-16T12:18:12.062 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-16T12:18:12.062 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-16T12:18:12.062 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-16T12:18:12.063 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-16T12:18:12.063 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-16T12:18:12.063 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-16T12:18:12.063 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-16T12:18:12.063 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-16T12:18:12.063 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-16T12:18:12.064 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-16T12:18:12.064 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-16T12:18:12.064 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-16T12:18:12.064 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-16T12:18:12.064 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-16T12:18:12.065 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-16T12:18:12.065 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-16T12:18:12.065 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-16T12:18:12.065 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-16T12:18:12.065 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-16T12:18:12.065 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-16T12:18:12.066 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-16T12:18:12.066 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-16T12:18:12.066 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-16T12:18:12.066 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-16T12:18:12.066 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-16T12:18:12.067 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-16T12:18:12.067 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-16T12:18:12.067 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-16T12:18:12.067 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-16T12:18:12.067 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-16T12:18:12.068 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-16T12:18:12.068 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-16T12:18:12.068 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-16T12:18:12.068 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-16T12:18:12.068 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-16T12:18:12.069 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-16T12:18:12.069 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-16T12:18:12.069 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-16T12:18:12.069 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-16T12:18:12.069 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-16T12:18:12.070 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-16T12:18:12.070 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-16T12:18:12.070 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-16T12:18:12.070 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-16T12:18:12.070 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-16T12:18:12.071 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-16T12:18:12.071 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-16T12:18:12.071 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-16T12:18:12.071 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-16T12:18:12.071 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-16T12:18:12.071 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-16T12:18:12.072 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-16T12:18:12.072 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-16T12:18:12.072 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-16T12:18:12.072 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-16T12:18:12.072 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-16T12:18:12.073 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-16T12:18:12.073 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-16T12:18:12.073 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-16T12:18:12.073 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-16T12:18:12.073 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-16T12:18:12.074 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-16T12:18:12.074 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-16T12:18:12.074 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-16T12:18:12.074 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-16T12:18:12.074 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-16T12:18:12.075 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-16T12:18:12.075 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-16T12:18:12.075 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-16T12:18:12.075 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-16T12:18:12.075 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-16T12:18:12.075 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-16T12:18:12.076 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-16T12:18:12.076 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-16T12:18:12.076 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-16T12:18:12.076 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-16T12:18:12.076 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-16T12:18:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-16T12:18:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-16T12:18:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-16T12:18:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-16T12:18:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-16T12:18:12.077 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-16T12:18:12.078 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-16T12:18:12.078 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-16T12:18:12.078 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-16T12:18:12.078 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-16T12:18:12.078 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-16T12:18:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-16T12:18:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-16T12:18:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-16T12:18:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-16T12:18:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-16T12:18:12.079 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-16T12:18:12.080 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-16T12:18:12.080 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-16T12:18:12.080 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-16T12:18:12.080 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-16T12:18:12.080 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-16T12:18:12.081 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-16T12:18:12.081 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-16T12:18:12.081 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-16T12:18:12.081 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-16T12:18:12.081 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-16T12:18:12.082 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-16T12:18:12.082 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-16T12:18:12.082 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-16T12:18:12.082 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-16T12:18:12.082 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-16T12:18:12.083 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-16T12:18:12.083 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-16T12:18:12.083 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-16T12:18:12.083 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-16T12:18:12.083 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-16T12:18:12.084 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-16T12:18:12.084 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-16T12:18:12.084 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-16T12:18:12.084 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-16T12:18:12.084 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-16T12:18:12.084 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-16T12:18:12.085 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-16T12:18:12.085 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-16T12:18:12.085 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-16T12:18:12.085 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-16T12:18:12.085 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-16T12:18:12.086 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-16T12:18:12.086 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-16T12:18:12.086 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-16T12:18:12.086 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-16T12:18:12.086 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-16T12:18:12.087 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-16T12:18:12.087 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-16T12:18:12.087 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-16T12:18:12.087 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-16T12:18:12.087 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-16T12:18:12.087 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-16T12:18:12.088 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-16T12:18:12.088 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-16T12:18:12.088 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-16T12:18:12.088 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-16T12:18:12.088 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-16T12:18:12.089 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-16T12:18:12.089 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-16T12:18:12.089 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-16T12:18:12.089 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-16T12:18:12.089 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-16T12:18:12.089 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-16T12:18:12.090 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-16T12:18:12.090 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-16T12:18:12.090 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-16T12:18:12.090 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-16T12:18:12.090 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-16T12:18:12.091 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-16T12:18:12.091 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-16T12:18:12.091 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-16T12:18:12.091 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-16T12:18:12.091 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-16T12:18:12.092 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-16T12:18:12.092 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-16T12:18:12.092 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-16T12:18:12.092 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-16T12:18:12.092 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-16T12:18:12.092 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-16T12:18:12.093 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-16T12:18:12.093 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-16T12:18:12.093 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-16T12:18:12.093 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-16T12:18:12.093 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-16T12:18:12.094 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-16T12:18:12.094 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-16T12:18:12.094 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-16T12:18:12.094 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-16T12:18:12.094 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-16T12:18:12.095 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-16T12:18:12.095 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-16T12:18:12.095 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-16T12:18:12.095 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-16T12:18:12.095 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-16T12:18:12.095 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-16T12:18:12.096 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-16T12:18:12.096 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-16T12:18:12.096 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-16T12:18:12.096 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-16T12:18:12.096 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-16T12:18:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-16T12:18:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-16T12:18:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-16T12:18:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-16T12:18:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-16T12:18:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-16T12:18:12.098 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-16T12:18:12.098 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-16T12:18:12.098 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-16T12:18:12.098 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-16T12:18:12.098 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-16T12:18:12.099 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-16T12:18:12.099 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-16T12:18:12.099 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-16T12:18:12.099 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-16T12:18:12.099 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-16T12:18:12.100 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-16T12:18:12.100 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-16T12:18:12.100 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-16T12:18:12.100 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-16T12:18:12.100 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-16T12:18:12.100 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-16T12:18:12.101 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-16T12:18:12.101 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-16T12:18:12.101 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-16T12:18:12.101 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-16T12:18:12.101 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-16T12:18:12.102 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-16T12:18:12.102 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-16T12:18:12.102 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-16T12:18:12.102 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-16T12:18:12.102 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-16T12:18:12.103 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-16T12:18:12.103 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-16T12:18:12.103 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-16T12:18:12.103 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-16T12:18:12.103 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-16T12:18:12.103 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-16T12:18:12.104 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-16T12:18:12.104 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-16T12:18:12.104 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-16T12:18:12.104 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-16T12:18:12.104 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-16T12:18:12.104 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-16T12:18:12.105 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-16T12:18:12.105 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-16T12:18:12.105 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-16T12:18:12.105 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-16T12:18:12.105 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-16T12:18:12.106 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-16T12:18:12.106 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-16T12:18:12.106 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-16T12:18:12.106 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-16T12:18:12.106 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-16T12:18:12.107 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-16T12:18:12.107 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-16T12:18:12.107 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-16T12:18:12.107 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-16T12:18:12.107 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-16T12:18:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-16T12:18:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-16T12:18:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-16T12:18:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-16T12:18:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-16T12:18:12.108 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-16T12:18:12.109 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-16T12:18:12.109 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-16T12:18:12.109 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-16T12:18:12.109 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-16T12:18:12.109 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-16T12:18:12.110 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-16T12:18:12.110 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-16T12:18:12.110 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-16T12:18:12.110 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-16T12:18:12.110 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-16T12:18:12.111 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-16T12:18:12.111 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-16T12:18:12.111 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-16T12:18:12.111 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-16T12:18:12.111 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-16T12:18:12.112 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-16T12:18:12.112 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-16T12:18:12.112 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-16T12:18:12.112 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-16T12:18:12.112 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-16T12:18:12.112 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-16T12:18:12.113 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-16T12:18:12.113 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-16T12:18:12.113 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-16T12:18:12.113 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-16T12:18:12.113 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-16T12:18:12.114 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-16T12:18:12.114 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-16T12:18:12.114 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-16T12:18:12.114 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-16T12:18:12.114 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-16T12:18:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-16T12:18:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-16T12:18:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-16T12:18:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-16T12:18:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-16T12:18:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-16T12:18:12.116 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-16T12:18:12.116 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-16T12:18:12.116 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-16T12:18:12.116 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-16T12:18:12.116 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-16T12:18:12.117 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-16T12:18:12.117 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-16T12:18:12.117 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-16T12:18:12.117 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-16T12:18:12.117 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-16T12:18:12.117 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-16T12:18:12.118 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-16T12:18:12.118 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-16T12:18:12.118 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-16T12:18:12.118 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-16T12:18:12.118 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-16T12:18:12.119 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-16T12:18:12.119 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-16T12:18:12.119 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-16T12:18:12.119 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-16T12:18:12.119 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-16T12:18:12.120 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-16T12:18:12.120 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-16T12:18:12.120 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-16T12:18:12.120 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-16T12:18:12.120 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-16T12:18:12.120 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-16T12:18:12.121 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-16T12:18:12.121 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-16T12:18:12.121 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-16T12:18:12.121 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-16T12:18:12.121 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-16T12:18:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-16T12:18:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-16T12:18:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-16T12:18:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-16T12:18:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-16T12:18:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-16T12:18:12.123 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-16T12:18:12.123 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-16T12:18:12.123 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-16T12:18:12.123 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-16T12:18:12.123 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-16T12:18:12.124 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-16T12:18:12.124 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-16T12:18:12.124 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-16T12:18:12.124 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-16T12:18:12.124 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-16T12:18:12.124 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-16T12:18:12.125 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-16T12:18:12.125 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-16T12:18:12.125 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-16T12:18:12.125 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-16T12:18:12.125 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-16T12:18:12.126 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-16T12:18:12.126 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-16T12:18:12.126 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-16T12:18:12.126 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-16T12:18:12.126 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-16T12:18:12.126 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-16T12:18:12.127 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-16T12:18:12.127 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-16T12:18:12.127 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-16T12:18:12.127 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-16T12:18:12.127 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-16T12:18:12.128 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-16T12:18:12.128 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-16T12:18:12.128 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-16T12:18:12.128 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-16T12:18:12.128 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-16T12:18:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-16T12:18:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-16T12:18:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-16T12:18:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-16T12:18:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-16T12:18:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-16T12:18:12.130 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-16T12:18:12.130 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-16T12:18:12.130 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-16T12:18:12.130 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-16T12:18:12.130 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-16T12:18:12.131 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-16T12:18:12.131 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-16T12:18:12.131 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-16T12:18:12.131 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-16T12:18:12.131 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-16T12:18:12.132 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-16T12:18:12.132 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-16T12:18:12.132 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-16T12:18:12.132 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-16T12:18:12.132 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-16T12:18:12.132 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-16T12:18:12.133 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-16T12:18:12.133 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-16T12:18:12.133 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-16T12:18:12.133 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-16T12:18:12.133 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-16T12:18:12.134 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-16T12:18:12.134 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-16T12:18:12.134 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-16T12:18:12.134 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-16T12:18:12.134 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-16T12:18:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-16T12:18:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-16T12:18:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-16T12:18:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-16T12:18:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-16T12:18:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-16T12:18:12.136 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-16T12:18:12.136 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-16T12:18:12.136 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-16T12:18:12.136 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-16T12:18:12.136 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-16T12:18:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-16T12:18:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-16T12:18:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-16T12:18:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-16T12:18:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-16T12:18:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-16T12:18:12.138 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-16T12:18:12.138 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-16T12:18:12.138 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-16T12:18:12.138 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-16T12:18:12.138 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-16T12:18:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-16T12:18:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-16T12:18:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-16T12:18:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-16T12:18:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-16T12:18:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-16T12:18:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-16T12:18:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-16T12:18:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-16T12:18:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-16T12:18:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-16T12:18:12.141 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-16T12:18:12.141 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-16T12:18:12.141 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-16T12:18:12.141 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-16T12:18:12.141 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-16T12:18:12.141 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-16T12:18:12.142 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-16T12:18:12.142 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-16T12:18:12.142 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-16T12:18:12.142 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-16T12:18:12.142 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-16T12:18:12.143 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-16T12:18:12.143 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-16T12:18:12.143 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-16T12:18:12.143 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-16T12:18:12.143 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-16T12:18:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-16T12:18:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-16T12:18:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-16T12:18:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-16T12:18:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-16T12:18:12.144 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-16T12:18:12.145 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-16T12:18:12.145 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-16T12:18:12.145 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-16T12:18:12.145 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-16T12:18:12.145 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-16T12:18:12.146 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-16T12:18:12.146 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-16T12:18:12.146 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-16T12:18:12.146 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-16T12:18:12.146 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-16T12:18:12.147 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-16T12:18:12.147 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-16T12:18:12.147 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-16T12:18:12.147 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-16T12:18:12.147 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-16T12:18:12.148 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-16T12:18:12.148 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-16T12:18:12.148 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-16T12:18:12.148 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-16T12:18:12.148 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-16T12:18:12.148 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-16T12:18:12.149 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-16T12:18:12.149 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-16T12:18:12.149 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-16T12:18:12.149 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-16T12:18:12.149 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-16T12:18:12.150 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-16T12:18:12.150 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-16T12:18:12.150 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-16T12:18:12.150 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-16T12:18:12.150 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-16T12:18:12.151 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-16T12:18:12.151 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-16T12:18:12.151 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-16T12:18:12.151 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-16T12:18:12.151 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-16T12:18:12.152 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-16T12:18:12.152 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-16T12:18:12.152 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-16T12:18:12.152 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-16T12:18:12.152 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-16T12:18:12.152 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-16T12:18:12.153 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-16T12:18:12.153 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-16T12:18:12.153 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-16T12:18:12.153 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-16T12:18:12.153 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-16T12:18:12.154 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-16T12:18:12.154 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-16T12:18:12.154 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-16T12:18:12.154 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-16T12:18:12.154 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-16T12:18:12.155 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-16T12:18:12.155 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-16T12:18:12.155 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-16T12:18:12.155 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-16T12:18:12.155 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-16T12:18:12.155 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-16T12:18:12.156 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-16T12:18:12.156 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-16T12:18:12.156 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-16T12:18:12.156 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-16T12:18:12.156 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-16T12:18:12.157 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-16T12:18:12.158 DEBUG:teuthology.orchestra.run.smithi060:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-16T12:18:12.183 INFO:teuthology.orchestra.run.smithi060.stdout:postinstall scriptlet (using /bin/sh): 2023-10-16T12:18:12.184 INFO:teuthology.orchestra.run.smithi060.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-16T12:18:12.184 INFO:teuthology.orchestra.run.smithi060.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:18:12.184 INFO:teuthology.orchestra.run.smithi060.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:18:12.184 INFO:teuthology.orchestra.run.smithi060.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T12:18:12.184 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:18:12.185 INFO:teuthology.orchestra.run.smithi060.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-16T12:18:12.185 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2023-10-16T12:18:12.185 INFO:teuthology.orchestra.run.smithi060.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-16T12:18:12.185 INFO:teuthology.orchestra.run.smithi060.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-16T12:18:12.185 INFO:teuthology.orchestra.run.smithi060.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-16T12:18:12.186 INFO:teuthology.orchestra.run.smithi060.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-16T12:18:12.186 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-16T12:18:12.186 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2023-10-16T12:18:12.186 INFO:teuthology.orchestra.run.smithi060.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-16T12:18:12.186 INFO:teuthology.orchestra.run.smithi060.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-16T12:18:12.187 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-16T12:18:12.187 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2023-10-16T12:18:12.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2023-10-16T12:18:12.271 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2023-10-16T12:18:12.272 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2023-10-16T12:18:12.272 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2023-10-16T12:18:12.272 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2023-10-16T12:18:12.272 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2023-10-16T12:18:12.272 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-16T12:18:12.273 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2023-10-16T12:18:12.273 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2023-10-16T12:18:12.273 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2023-10-16T12:18:12.273 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-16T12:18:12.273 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-16T12:18:12.274 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-16T12:18:12.274 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-16T12:18:12.274 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2023-10-16T12:18:12.274 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2023-10-16T12:18:12.274 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-16T12:18:12.275 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2023-10-16T12:18:12.275 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2023-10-16T12:18:12.275 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:18:12.275 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-16T12:18:12.275 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-16T12:18:12.275 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-16T12:18:12.276 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2023-10-16T12:18:12.276 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:18:12.276 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2023-10-16T12:18:12.277 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-16T12:18:12.277 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 6.6.0-rc5-g47672119149b 2023-10-16T12:18:12.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-16T12:18:12.868 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2023-10-16T12:18:14.256 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-16T12:18:14.306 INFO:teuthology.orchestra.run.smithi060.stderr:done 2023-10-16T12:18:14.307 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2023-10-16T12:18:14.322 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.UZfPSIb7fF 2023-10-16T12:18:14.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UZfPSIb7fF 2023-10-16T12:18:14.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.UZfPSIb7fF 2023-10-16T12:18:14.543 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.UZfPSIb7fF is 6KB 2023-10-16T12:18:14.593 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.UZfPSIb7fF 2023-10-16T12:18:14.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2023-10-16T12:18:14.676 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-16T12:18:14.676 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-16T12:18:14.677 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-16T12:18:14.677 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b.conf 2023-10-16T12:18:14.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b 2023-10-16T12:18:14.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2023-10-16T12:18:36.022 DEBUG:teuthology.orchestra.remote:timed out 2023-10-16T12:18:44.827 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-16T12:18:44.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:18:44.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:18:45.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-16T12:18:45.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:18:48.519 DEBUG:teuthology.orchestra.run.smithi144:> true 2023-10-16T12:18:49.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-16T12:18:49.151 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.6.0-rc5-g47672119149b"... 2023-10-16T12:18:49.151 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2023-10-16T12:18:49.192 INFO:teuthology.orchestra.run.smithi144.stdout:6.6.0-rc5-g47672119149b 2023-10-16T12:18:49.192 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-16T12:18:49.192 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T12:18:49.193 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-16T12:18:50.195 DEBUG:teuthology.parallel:result is None 2023-10-16T12:19:03.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2023-10-16T12:19:12.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:19:12.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:19:30.555 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2023-10-16T12:19:42.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:19:42.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:19:45.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2023-10-16T12:20:00.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:20:00.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:20:01.003 DEBUG:teuthology.orchestra.run.smithi060:> true 2023-10-16T12:20:01.505 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:20:01.506 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g47672119149b"... 2023-10-16T12:20:01.506 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2023-10-16T12:20:01.547 INFO:teuthology.orchestra.run.smithi060.stdout:6.6.0-rc5-g47672119149b 2023-10-16T12:20:01.548 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-16T12:20:01.548 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T12:20:01.548 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-16T12:20:02.549 DEBUG:teuthology.parallel:result is None 2023-10-16T12:20:02.550 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-16T12:20:02.557 INFO:teuthology.task.internal:Creating test directory... 2023-10-16T12:20:02.558 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T12:20:02.567 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T12:20:02.630 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-16T12:20:02.636 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-16T12:20:02.667 INFO:teuthology.task.internal:Creating archive directory... 2023-10-16T12:20:02.667 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T12:20:02.675 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T12:20:02.752 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-16T12:20:02.759 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-16T12:20:02.760 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T12:20:02.765 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T12:20:02.926 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T12:20:02.970 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T12:20:02.994 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T12:20:03.090 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T12:20:03.094 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-16T12:20:03.103 INFO:teuthology.task.internal:Configuring sudo... 2023-10-16T12:20:03.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T12:20:03.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T12:20:03.210 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-16T12:20:03.220 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-16T12:20:03.220 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T12:20:03.226 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T12:20:03.284 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T12:20:03.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T12:20:03.469 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T12:20:03.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T12:20:03.626 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:20:03.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T12:20:03.698 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T12:20:03.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T12:20:03.914 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T12:20:03.999 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T12:20:04.092 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:20:04.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T12:20:04.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2023-10-16T12:20:04.199 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2023-10-16T12:20:04.356 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T12:20:04.873 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-16T12:20:04.885 INFO:teuthology.task.internal:Starting timer... 2023-10-16T12:20:04.885 INFO:teuthology.run_tasks:Running task pcp... 2023-10-16T12:20:04.912 INFO:teuthology.run_tasks:Running task selinux... 2023-10-16T12:20:04.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2023-10-16T12:20:05.167 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2023-10-16T12:20:05.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2023-10-16T12:20:05.320 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2023-10-16T12:20:05.324 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-16T12:20:05.324 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2023-10-16T12:20:05.410 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2023-10-16T12:20:05.410 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2023-10-16T12:20:05.469 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2023-10-16T12:20:05.469 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2023-10-16T12:20:05.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T12:20:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:20:05.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T12:20:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:20:05.663 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-16T12:20:05.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2023-10-16T12:20:05.755 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2023-10-16T12:20:05.865 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-16T12:20:05.875 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-16T12:20:05.880 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-16T12:20:05.984 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-16T12:20:05.999 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-16T12:20:06.001 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-16T12:30:16.846 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2023-10-16T12:30:16.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:30:16.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:30:16.992 DEBUG:teuthology.orchestra.run.smithi060:> true 2023-10-16T12:30:17.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-10-16T12:30:17.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-16T12:30:17.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:30:17.283 DEBUG:teuthology.orchestra.run.smithi144:> true 2023-10-16T12:30:17.435 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-10-16T12:30:17.436 INFO:teuthology.run_tasks:Running task clock... 2023-10-16T12:30:17.447 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-16T12:30:17.447 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T12:30:17.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T12:30:17.457 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T12:30:17.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T12:30:17.560 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T12:30:17.562 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T12:30:17.610 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T12:30:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T12:30:17.706 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2023-10-16T12:30:17.723 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2023-10-16T12:30:17.749 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2023-10-16T12:30:17.768 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2023-10-16T12:30:17.800 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T12:30:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T12:30:17.853 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T12:30:17.882 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T12:30:17.977 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2023-10-16T12:30:17.986 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T12:30:17.986 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2023-10-16T12:30:17.987 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:17.987 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:17.987 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:17.987 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2023-10-16T12:30:18.024 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T12:30:18.024 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2023-10-16T12:30:18.024 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:18.024 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:18.025 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:18.025 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:30:18.026 INFO:teuthology.run_tasks:Running task install... 2023-10-16T12:30:18.035 DEBUG:teuthology.task.install:project ceph 2023-10-16T12:30:18.035 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}} 2023-10-16T12:30:18.036 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'} 2023-10-16T12:30:18.036 INFO:teuthology.task.install:Using flavor: default 2023-10-16T12:30:18.042 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-16T12:30:18.042 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-16T12:30:18.043 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-16T12:30:18.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:30:18.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-16T12:30:18.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:30:18.192 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/ 2023-10-16T12:30:18.192 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-16T12:30:18.272 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/ 2023-10-16T12:30:18.272 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-16T12:30:18.306 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T12:30:18.307 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:30:18.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T12:30:18.375 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T12:30:18.376 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:30:18.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T12:30:18.418 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T12:30:18.419 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T12:30:18.474 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T12:30:18.475 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T12:30:18.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T12:30:18.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T12:30:18.714 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T12:30:18.749 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T12:30:18.814 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2023-10-16T12:30:18.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2023-10-16T12:30:18.842 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2023-10-16T12:30:18.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2023-10-16T12:30:19.186 INFO:teuthology.orchestra.run.smithi144.stdout:72 files removed 2023-10-16T12:30:19.190 INFO:teuthology.orchestra.run.smithi060.stdout:72 files removed 2023-10-16T12:30:19.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T12:30:19.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T12:30:20.201 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 166 kB/s | 78 kB 00:00 2023-10-16T12:30:20.365 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 130 kB/s | 78 kB 00:00 2023-10-16T12:30:20.701 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 23 kB/s | 9.5 kB 00:00 2023-10-16T12:30:20.965 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 19 kB/s | 9.5 kB 00:00 2023-10-16T12:30:21.160 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2023-10-16T12:30:21.523 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2023-10-16T12:30:21.896 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.3 MB 00:00 2023-10-16T12:30:22.302 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.3 MB 00:00 2023-10-16T12:30:23.480 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2023-10-16T12:30:23.953 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2023-10-16T12:30:25.468 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.9 MB 00:01 2023-10-16T12:30:28.315 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 7.9 MB 00:03 2023-10-16T12:30:30.143 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 18 MB 00:03 2023-10-16T12:30:33.116 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 18 MB 00:03 2023-10-16T12:30:35.611 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.7 MB 00:01 2023-10-16T12:30:37.445 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 14 kB 00:00 2023-10-16T12:30:38.179 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.7 MB 00:01 2023-10-16T12:30:39.452 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 10 MB/s | 19 MB 00:01 2023-10-16T12:30:39.822 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 14 kB 00:00 2023-10-16T12:30:41.749 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-10-16T12:30:44.546 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-10-16T12:30:47.050 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T12:30:47.052 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T12:30:47.073 INFO:teuthology.orchestra.run.smithi144.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-16T12:30:47.073 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2023-10-16T12:30:47.074 INFO:teuthology.orchestra.run.smithi144.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-16T12:30:47.076 INFO:teuthology.orchestra.run.smithi144.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-16T12:30:47.077 INFO:teuthology.orchestra.run.smithi144.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-16T12:30:47.080 INFO:teuthology.orchestra.run.smithi144.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-16T12:30:47.083 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-16T12:30:47.085 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-16T12:30:47.161 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-10-16T12:30:47.186 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================ 2023-10-16T12:30:47.186 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2023-10-16T12:30:47.186 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================ 2023-10-16T12:30:47.186 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2023-10-16T12:30:47.187 INFO:teuthology.orchestra.run.smithi144.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-16T12:30:47.187 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el9 ceph 6.5 k 2023-10-16T12:30:47.187 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el9 ceph 5.2 M 2023-10-16T12:30:47.187 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 855 k 2023-10-16T12:30:47.187 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el9 ceph 141 k 2023-10-16T12:30:47.187 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el9 ceph 1.5 M 2023-10-16T12:30:47.188 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 132 k 2023-10-16T12:30:47.188 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 1.8 M 2023-10-16T12:30:47.188 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 7.4 M 2023-10-16T12:30:47.188 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 47 k 2023-10-16T12:30:47.189 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 8.7 M 2023-10-16T12:30:47.189 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el9 ceph 41 M 2023-10-16T12:30:47.189 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 262 k 2023-10-16T12:30:47.189 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 269 k 2023-10-16T12:30:47.190 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 218 k 2023-10-16T12:30:47.190 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 18 k 2023-10-16T12:30:47.190 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-16T12:30:47.190 INFO:teuthology.orchestra.run.smithi144.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-16T12:30:47.190 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-16T12:30:47.191 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 31 k 2023-10-16T12:30:47.191 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 715 k 2023-10-16T12:30:47.191 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-16T12:30:47.191 INFO:teuthology.orchestra.run.smithi144.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-16T12:30:47.191 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-16T12:30:47.192 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-16T12:30:47.192 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-16T12:30:47.192 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-16T12:30:47.192 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-16T12:30:47.192 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el9 ceph 162 k 2023-10-16T12:30:47.192 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el9 ceph 320 k 2023-10-16T12:30:47.193 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 297 k 2023-10-16T12:30:47.193 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 99 k 2023-10-16T12:30:47.193 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 85 k 2023-10-16T12:30:47.193 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-16T12:30:47.194 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 168 k 2023-10-16T12:30:47.194 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-16T12:30:47.194 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2023-10-16T12:30:47.194 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-16T12:30:47.194 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-16T12:30:47.195 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-16T12:30:47.195 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-16T12:30:47.195 INFO:teuthology.orchestra.run.smithi144.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-16T12:30:47.195 INFO:teuthology.orchestra.run.smithi144.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-16T12:30:47.195 INFO:teuthology.orchestra.run.smithi144.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-16T12:30:47.196 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.3 M 2023-10-16T12:30:47.196 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-16T12:30:47.196 INFO:teuthology.orchestra.run.smithi144.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-16T12:30:47.196 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2023-10-16T12:30:47.196 INFO:teuthology.orchestra.run.smithi144.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-16T12:30:47.197 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-16T12:30:47.197 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 19 M 2023-10-16T12:30:47.197 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 23 k 2023-10-16T12:30:47.197 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el9 ceph 2.1 M 2023-10-16T12:30:47.197 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 243 k 2023-10-16T12:30:47.197 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.4 M 2023-10-16T12:30:47.198 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 18 M 2023-10-16T12:30:47.198 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 15 k 2023-10-16T12:30:47.198 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el9 ceph 24 k 2023-10-16T12:30:47.198 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-16T12:30:47.198 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-16T12:30:47.199 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-16T12:30:47.199 INFO:teuthology.orchestra.run.smithi144.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-16T12:30:47.199 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-16T12:30:47.199 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-16T12:30:47.199 INFO:teuthology.orchestra.run.smithi144.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-16T12:30:47.200 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-16T12:30:47.200 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-16T12:30:47.200 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el9 ceph 163 k 2023-10-16T12:30:47.200 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-16T12:30:47.200 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-16T12:30:47.200 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-16T12:30:47.201 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-16T12:30:47.201 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-16T12:30:47.201 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 473 k 2023-10-16T12:30:47.201 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-16T12:30:47.201 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.8 M 2023-10-16T12:30:47.202 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-16T12:30:47.202 INFO:teuthology.orchestra.run.smithi144.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-16T12:30:47.202 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-16T12:30:47.202 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-16T12:30:47.202 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-16T12:30:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-16T12:30:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-16T12:30:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-16T12:30:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-16T12:30:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-16T12:30:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-16T12:30:47.204 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-16T12:30:47.204 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 45 k 2023-10-16T12:30:47.204 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-16T12:30:47.204 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-16T12:30:47.204 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-16T12:30:47.205 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-16T12:30:47.205 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-16T12:30:47.205 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-16T12:30:47.205 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-16T12:30:47.205 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-16T12:30:47.206 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-16T12:30:47.206 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-16T12:30:47.206 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-16T12:30:47.206 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-16T12:30:47.206 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-16T12:30:47.207 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-16T12:30:47.207 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-16T12:30:47.207 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-16T12:30:47.207 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-16T12:30:47.207 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-16T12:30:47.208 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-16T12:30:47.208 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-16T12:30:47.208 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-16T12:30:47.208 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-16T12:30:47.208 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-16T12:30:47.209 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-16T12:30:47.209 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-16T12:30:47.209 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-16T12:30:47.209 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-16T12:30:47.209 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-16T12:30:47.210 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-16T12:30:47.210 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-16T12:30:47.210 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-16T12:30:47.210 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-16T12:30:47.210 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-16T12:30:47.211 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-16T12:30:47.211 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-16T12:30:47.211 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-16T12:30:47.211 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-16T12:30:47.211 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-16T12:30:47.212 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-16T12:30:47.212 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-16T12:30:47.212 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-16T12:30:47.212 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-16T12:30:47.212 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-16T12:30:47.213 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-16T12:30:47.213 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-16T12:30:47.213 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-16T12:30:47.213 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-16T12:30:47.213 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-16T12:30:47.213 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-16T12:30:47.214 INFO:teuthology.orchestra.run.smithi144.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-16T12:30:47.214 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-16T12:30:47.214 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-16T12:30:47.214 INFO:teuthology.orchestra.run.smithi144.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-16T12:30:47.214 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-16T12:30:47.215 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-16T12:30:47.215 INFO:teuthology.orchestra.run.smithi144.stdout:Installing weak dependencies: 2023-10-16T12:30:47.215 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-16T12:30:47.215 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:30:47.215 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-10-16T12:30:47.215 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================ 2023-10-16T12:30:47.216 INFO:teuthology.orchestra.run.smithi144.stdout:Install 133 Packages 2023-10-16T12:30:47.216 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 10 Packages 2023-10-16T12:30:47.216 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:30:47.216 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 200 M 2023-10-16T12:30:47.217 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2023-10-16T12:30:48.569 INFO:teuthology.orchestra.run.smithi144.stdout:(1/143): ceph-18.0.0-6675.gc7471068.el9.x86_64. 11 kB/s | 6.5 kB 00:00 2023-10-16T12:30:48.986 INFO:teuthology.orchestra.run.smithi144.stdout:(2/143): ceph-base-18.0.0-6675.gc7471068.el9.x8 5.1 MB/s | 5.2 MB 00:01 2023-10-16T12:30:49.036 INFO:teuthology.orchestra.run.smithi144.stdout:(3/143): ceph-immutable-object-cache-18.0.0-667 2.8 MB/s | 141 kB 00:00 2023-10-16T12:30:49.161 INFO:teuthology.orchestra.run.smithi144.stdout:(4/143): ceph-fuse-18.0.0-6675.gc7471068.el9.x8 1.4 MB/s | 855 kB 00:00 2023-10-16T12:30:49.253 INFO:teuthology.orchestra.run.smithi144.stdout:(5/143): ceph-mds-18.0.0-6675.gc7471068.el9.x86 9.6 MB/s | 2.1 MB 00:00 2023-10-16T12:30:49.562 INFO:teuthology.orchestra.run.smithi144.stdout:(6/143): ceph-mon-18.0.0-6675.gc7471068.el9.x86 14 MB/s | 4.4 MB 00:00 2023-10-16T12:30:49.804 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T12:30:49.806 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-16T12:30:49.827 INFO:teuthology.orchestra.run.smithi060.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-16T12:30:49.828 INFO:teuthology.orchestra.run.smithi060.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-16T12:30:49.830 INFO:teuthology.orchestra.run.smithi060.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-16T12:30:49.831 INFO:teuthology.orchestra.run.smithi060.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-16T12:30:49.834 INFO:teuthology.orchestra.run.smithi060.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-16T12:30:49.837 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-16T12:30:49.839 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-16T12:30:49.879 INFO:teuthology.orchestra.run.smithi144.stdout:(7/143): ceph-mgr-18.0.0-6675.gc7471068.el9.x86 2.0 MB/s | 1.5 MB 00:00 2023-10-16T12:30:49.926 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-10-16T12:30:49.953 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2023-10-16T12:30:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-10-16T12:30:49.953 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2023-10-16T12:30:49.954 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-10-16T12:30:49.954 INFO:teuthology.orchestra.run.smithi060.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-16T12:30:49.954 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el9 ceph 6.5 k 2023-10-16T12:30:49.954 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el9 ceph 5.2 M 2023-10-16T12:30:49.955 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 855 k 2023-10-16T12:30:49.955 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el9 ceph 141 k 2023-10-16T12:30:49.955 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el9 ceph 1.5 M 2023-10-16T12:30:49.955 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 132 k 2023-10-16T12:30:49.955 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 1.8 M 2023-10-16T12:30:49.956 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 7.4 M 2023-10-16T12:30:49.956 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 47 k 2023-10-16T12:30:49.956 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 8.7 M 2023-10-16T12:30:49.956 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el9 ceph 41 M 2023-10-16T12:30:49.956 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 262 k 2023-10-16T12:30:49.957 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 269 k 2023-10-16T12:30:49.957 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 218 k 2023-10-16T12:30:49.957 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 18 k 2023-10-16T12:30:49.957 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-16T12:30:49.957 INFO:teuthology.orchestra.run.smithi060.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-16T12:30:49.958 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-16T12:30:49.958 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 31 k 2023-10-16T12:30:49.958 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 715 k 2023-10-16T12:30:49.958 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-16T12:30:49.958 INFO:teuthology.orchestra.run.smithi060.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-16T12:30:49.959 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-16T12:30:49.959 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-16T12:30:49.959 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-16T12:30:49.959 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-16T12:30:49.959 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-16T12:30:49.960 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el9 ceph 162 k 2023-10-16T12:30:49.960 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el9 ceph 320 k 2023-10-16T12:30:49.960 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 297 k 2023-10-16T12:30:49.960 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 99 k 2023-10-16T12:30:49.960 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 85 k 2023-10-16T12:30:49.961 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-16T12:30:49.961 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 168 k 2023-10-16T12:30:49.961 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-16T12:30:49.961 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2023-10-16T12:30:49.961 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-16T12:30:49.962 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-16T12:30:49.962 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-16T12:30:49.962 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-16T12:30:49.962 INFO:teuthology.orchestra.run.smithi060.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-16T12:30:49.962 INFO:teuthology.orchestra.run.smithi060.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-16T12:30:49.963 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-16T12:30:49.963 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.3 M 2023-10-16T12:30:49.963 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-16T12:30:49.963 INFO:teuthology.orchestra.run.smithi060.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-16T12:30:49.963 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-10-16T12:30:49.964 INFO:teuthology.orchestra.run.smithi060.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-16T12:30:49.964 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-16T12:30:49.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 19 M 2023-10-16T12:30:49.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 23 k 2023-10-16T12:30:49.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el9 ceph 2.1 M 2023-10-16T12:30:49.964 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 243 k 2023-10-16T12:30:49.965 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.4 M 2023-10-16T12:30:49.965 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 18 M 2023-10-16T12:30:49.965 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 15 k 2023-10-16T12:30:49.965 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el9 ceph 24 k 2023-10-16T12:30:49.965 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-16T12:30:49.966 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-16T12:30:49.966 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-16T12:30:49.966 INFO:teuthology.orchestra.run.smithi060.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-16T12:30:49.966 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-16T12:30:49.966 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-16T12:30:49.967 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-16T12:30:49.967 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-16T12:30:49.967 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-16T12:30:49.967 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el9 ceph 163 k 2023-10-16T12:30:49.967 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-16T12:30:49.967 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-16T12:30:49.968 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-16T12:30:49.968 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-16T12:30:49.968 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-16T12:30:49.968 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 473 k 2023-10-16T12:30:49.968 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-16T12:30:49.969 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.8 M 2023-10-16T12:30:49.969 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-16T12:30:49.969 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-16T12:30:49.969 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-16T12:30:49.969 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-16T12:30:49.969 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-16T12:30:49.970 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-16T12:30:49.970 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-16T12:30:49.970 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-16T12:30:49.970 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-16T12:30:49.970 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-16T12:30:49.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-16T12:30:49.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-16T12:30:49.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 45 k 2023-10-16T12:30:49.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-16T12:30:49.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-16T12:30:49.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-16T12:30:49.972 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-16T12:30:49.972 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-16T12:30:49.972 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-16T12:30:49.972 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-16T12:30:49.972 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-16T12:30:49.972 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-16T12:30:49.973 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-16T12:30:49.973 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-16T12:30:49.973 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-16T12:30:49.973 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-16T12:30:49.973 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-16T12:30:49.973 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-16T12:30:49.974 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-16T12:30:49.974 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-16T12:30:49.974 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-16T12:30:49.974 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-16T12:30:49.974 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-16T12:30:49.975 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-16T12:30:49.975 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-16T12:30:49.975 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-16T12:30:49.975 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-16T12:30:49.975 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-16T12:30:49.975 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-16T12:30:49.976 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-16T12:30:49.976 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-16T12:30:49.976 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-16T12:30:49.976 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-16T12:30:49.976 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-16T12:30:49.976 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-16T12:30:49.977 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-16T12:30:49.977 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-16T12:30:49.977 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-16T12:30:49.977 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-16T12:30:49.977 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-16T12:30:49.978 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-16T12:30:49.978 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-16T12:30:49.978 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-16T12:30:49.978 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-16T12:30:49.978 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-16T12:30:49.979 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-16T12:30:49.979 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-16T12:30:49.979 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-16T12:30:49.979 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-16T12:30:49.979 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-16T12:30:49.979 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-16T12:30:49.980 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-16T12:30:49.980 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-16T12:30:49.980 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-16T12:30:49.980 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-16T12:30:49.980 INFO:teuthology.orchestra.run.smithi060.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-16T12:30:49.981 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-16T12:30:49.981 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-16T12:30:49.981 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2023-10-16T12:30:49.981 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-16T12:30:49.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:30:49.982 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-10-16T12:30:49.982 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2023-10-16T12:30:49.982 INFO:teuthology.orchestra.run.smithi060.stdout:Install 133 Packages 2023-10-16T12:30:49.982 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 10 Packages 2023-10-16T12:30:49.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:30:49.983 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 200 M 2023-10-16T12:30:49.983 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-10-16T12:30:50.529 INFO:teuthology.orchestra.run.smithi144.stdout:(8/143): ceph-osd-18.0.0-6675.gc7471068.el9.x86 18 MB/s | 18 MB 00:00 2023-10-16T12:30:50.580 INFO:teuthology.orchestra.run.smithi144.stdout:(9/143): ceph-selinux-18.0.0-6675.gc7471068.el9 493 kB/s | 24 kB 00:00 2023-10-16T12:30:51.600 INFO:teuthology.orchestra.run.smithi060.stdout:(1/143): ceph-18.0.0-6675.gc7471068.el9.x86_64. 11 kB/s | 6.5 kB 00:00 2023-10-16T12:30:52.250 INFO:teuthology.orchestra.run.smithi060.stdout:(2/143): ceph-fuse-18.0.0-6675.gc7471068.el9.x8 1.3 MB/s | 855 kB 00:00 2023-10-16T12:30:52.359 INFO:teuthology.orchestra.run.smithi060.stdout:(3/143): ceph-immutable-object-cache-18.0.0-667 1.3 MB/s | 141 kB 00:00 2023-10-16T12:30:52.792 INFO:teuthology.orchestra.run.smithi060.stdout:(4/143): ceph-mds-18.0.0-6675.gc7471068.el9.x86 4.8 MB/s | 2.1 MB 00:00 2023-10-16T12:30:52.896 INFO:teuthology.orchestra.run.smithi144.stdout:(10/143): ceph-test-18.0.0-6675.gc7471068.el9.x 18 MB/s | 41 MB 00:02 2023-10-16T12:30:52.901 INFO:teuthology.orchestra.run.smithi060.stdout:(5/143): ceph-base-18.0.0-6675.gc7471068.el9.x8 2.8 MB/s | 5.2 MB 00:01 2023-10-16T12:30:52.963 INFO:teuthology.orchestra.run.smithi144.stdout:(11/143): cephfs-mirror-18.0.0-6675.gc7471068.e 3.3 MB/s | 218 kB 00:00 2023-10-16T12:30:53.021 INFO:teuthology.orchestra.run.smithi144.stdout:(12/143): libcephfs-devel-18.0.0-6675.gc7471068 543 kB/s | 31 kB 00:00 2023-10-16T12:30:53.093 INFO:teuthology.orchestra.run.smithi060.stdout:(6/143): ceph-mgr-18.0.0-6675.gc7471068.el9.x86 4.9 MB/s | 1.5 MB 00:00 2023-10-16T12:30:53.113 INFO:teuthology.orchestra.run.smithi144.stdout:(13/143): libcephfs2-18.0.0-6675.gc7471068.el9. 7.7 MB/s | 715 kB 00:00 2023-10-16T12:30:53.180 INFO:teuthology.orchestra.run.smithi144.stdout:(14/143): libcephsqlite-18.0.0-6675.gc7471068.e 2.4 MB/s | 163 kB 00:00 2023-10-16T12:30:53.239 INFO:teuthology.orchestra.run.smithi144.stdout:(15/143): librados-devel-18.0.0-6675.gc7471068. 2.2 MB/s | 127 kB 00:00 2023-10-16T12:30:53.323 INFO:teuthology.orchestra.run.smithi144.stdout:(16/143): libradosstriper1-18.0.0-6675.gc747106 5.6 MB/s | 473 kB 00:00 2023-10-16T12:30:53.523 INFO:teuthology.orchestra.run.smithi144.stdout:(17/143): ceph-radosgw-18.0.0-6675.gc7471068.el 2.4 MB/s | 8.7 MB 00:03 2023-10-16T12:30:53.640 INFO:teuthology.orchestra.run.smithi144.stdout:(18/143): python3-ceph-argparse-18.0.0-6675.gc7 390 kB/s | 45 kB 00:00 2023-10-16T12:30:53.756 INFO:teuthology.orchestra.run.smithi144.stdout:(19/143): librgw2-18.0.0-6675.gc7471068.el9.x86 11 MB/s | 4.8 MB 00:00 2023-10-16T12:30:53.790 INFO:teuthology.orchestra.run.smithi144.stdout:(20/143): python3-ceph-common-18.0.0-6675.gc747 849 kB/s | 127 kB 00:00 2023-10-16T12:30:53.810 INFO:teuthology.orchestra.run.smithi060.stdout:(7/143): ceph-mon-18.0.0-6675.gc7471068.el9.x86 4.8 MB/s | 4.4 MB 00:00 2023-10-16T12:30:53.832 INFO:teuthology.orchestra.run.smithi144.stdout:(21/143): python3-cephfs-18.0.0-6675.gc7471068. 2.1 MB/s | 162 kB 00:00 2023-10-16T12:30:53.908 INFO:teuthology.orchestra.run.smithi144.stdout:(22/143): python3-rbd-18.0.0-6675.gc7471068.el9 3.9 MB/s | 297 kB 00:00 2023-10-16T12:30:53.966 INFO:teuthology.orchestra.run.smithi144.stdout:(23/143): python3-rados-18.0.0-6675.gc7471068.e 1.8 MB/s | 320 kB 00:00 2023-10-16T12:30:53.992 INFO:teuthology.orchestra.run.smithi144.stdout:(24/143): python3-rgw-18.0.0-6675.gc7471068.el9 1.2 MB/s | 99 kB 00:00 2023-10-16T12:30:54.050 INFO:teuthology.orchestra.run.smithi144.stdout:(25/143): rbd-fuse-18.0.0-6675.gc7471068.el9.x8 1.0 MB/s | 85 kB 00:00 2023-10-16T12:30:54.351 INFO:teuthology.orchestra.run.smithi144.stdout:(26/143): rbd-mirror-18.0.0-6675.gc7471068.el9. 8.4 MB/s | 3.0 MB 00:00 2023-10-16T12:30:54.411 INFO:teuthology.orchestra.run.smithi144.stdout:(27/143): ceph-grafana-dashboards-18.0.0-6675.g 403 kB/s | 23 kB 00:00 2023-10-16T12:30:54.476 INFO:teuthology.orchestra.run.smithi144.stdout:(28/143): ceph-mgr-cephadm-18.0.0-6675.gc747106 2.0 MB/s | 132 kB 00:00 2023-10-16T12:30:54.752 INFO:teuthology.orchestra.run.smithi144.stdout:(29/143): ceph-mgr-dashboard-18.0.0-6675.gc7471 6.4 MB/s | 1.8 MB 00:00 2023-10-16T12:30:54.902 INFO:teuthology.orchestra.run.smithi144.stdout:(30/143): rbd-nbd-18.0.0-6675.gc7471068.el9.x86 198 kB/s | 168 kB 00:00 2023-10-16T12:30:55.044 INFO:teuthology.orchestra.run.smithi144.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6675.gc7 1.7 MB/s | 243 kB 00:00 2023-10-16T12:30:55.069 INFO:teuthology.orchestra.run.smithi060.stdout:(8/143): ceph-osd-18.0.0-6675.gc7471068.el9.x86 9.0 MB/s | 18 MB 00:01 2023-10-16T12:30:55.128 INFO:teuthology.orchestra.run.smithi144.stdout:(32/143): ceph-mgr-rook-18.0.0-6675.gc7471068.e 578 kB/s | 47 kB 00:00 2023-10-16T12:30:55.169 INFO:teuthology.orchestra.run.smithi060.stdout:(9/143): ceph-selinux-18.0.0-6675.gc7471068.el9 244 kB/s | 24 kB 00:00 2023-10-16T12:30:55.211 INFO:teuthology.orchestra.run.smithi144.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6675.gc 177 kB/s | 15 kB 00:00 2023-10-16T12:30:55.387 INFO:teuthology.orchestra.run.smithi144.stdout:(34/143): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2023-10-16T12:30:55.420 INFO:teuthology.orchestra.run.smithi144.stdout:(35/143): ceph-volume-18.0.0-6675.gc7471068.el9 1.2 MB/s | 262 kB 00:00 2023-10-16T12:30:55.462 INFO:teuthology.orchestra.run.smithi144.stdout:(36/143): cephadm-18.0.0-6675.gc7471068.el9.noa 3.5 MB/s | 269 kB 00:00 2023-10-16T12:30:55.504 INFO:teuthology.orchestra.run.smithi144.stdout:(37/143): cephfs-top-18.0.0-6675.gc7471068.el9. 219 kB/s | 18 kB 00:00 2023-10-16T12:30:55.754 INFO:teuthology.orchestra.run.smithi144.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 587 kB/s | 106 kB 00:00 2023-10-16T12:30:55.811 INFO:teuthology.orchestra.run.smithi060.stdout:(10/143): ceph-radosgw-18.0.0-6675.gc7471068.el 4.3 MB/s | 8.7 MB 00:01 2023-10-16T12:30:55.868 INFO:teuthology.orchestra.run.smithi144.stdout:(39/143): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 920 kB 00:00 2023-10-16T12:30:55.905 INFO:teuthology.orchestra.run.smithi144.stdout:(40/143): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 307 kB 00:00 2023-10-16T12:30:55.919 INFO:teuthology.orchestra.run.smithi060.stdout:(11/143): cephfs-mirror-18.0.0-6675.gc7471068.e 2.0 MB/s | 218 kB 00:00 2023-10-16T12:30:55.930 INFO:teuthology.orchestra.run.smithi144.stdout:(41/143): flexiblas-3.0.4-8.el9.x86_64.rpm 544 kB/s | 32 kB 00:00 2023-10-16T12:30:56.006 INFO:teuthology.orchestra.run.smithi144.stdout:(42/143): flexiblas-openblas-openmp-3.0.4-8.el9 231 kB/s | 17 kB 00:00 2023-10-16T12:30:56.139 INFO:teuthology.orchestra.run.smithi144.stdout:(43/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2023-10-16T12:30:56.144 INFO:teuthology.orchestra.run.smithi060.stdout:(12/143): ceph-common-18.0.0-6675.gc7471068.el9 3.7 MB/s | 19 MB 00:05 2023-10-16T12:30:56.164 INFO:teuthology.orchestra.run.smithi144.stdout:(44/143): libaio-devel-0.3.111-13.el9.x86_64.rp 67 kB/s | 10 kB 00:00 2023-10-16T12:30:56.187 INFO:teuthology.orchestra.run.smithi060.stdout:(13/143): libcephfs-devel-18.0.0-6675.gc7471068 117 kB/s | 31 kB 00:00 2023-10-16T12:30:56.240 INFO:teuthology.orchestra.run.smithi144.stdout:(45/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2023-10-16T12:30:56.281 INFO:teuthology.orchestra.run.smithi144.stdout:(46/143): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 661 kB 00:00 2023-10-16T12:30:56.320 INFO:teuthology.orchestra.run.smithi060.stdout:(14/143): libcephsqlite-18.0.0-6675.gc7471068.e 1.2 MB/s | 163 kB 00:00 2023-10-16T12:30:56.332 INFO:teuthology.orchestra.run.smithi144.stdout:(47/143): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2023-10-16T12:30:56.357 INFO:teuthology.orchestra.run.smithi144.stdout:(48/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 622 kB/s | 46 kB 00:00 2023-10-16T12:30:56.370 INFO:teuthology.orchestra.run.smithi060.stdout:(15/143): libcephfs2-18.0.0-6675.gc7471068.el9. 3.1 MB/s | 715 kB 00:00 2023-10-16T12:30:56.404 INFO:teuthology.orchestra.run.smithi060.stdout:(16/143): librados-devel-18.0.0-6675.gc7471068. 1.5 MB/s | 127 kB 00:00 2023-10-16T12:30:56.416 INFO:teuthology.orchestra.run.smithi144.stdout:(49/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 282 kB 00:00 2023-10-16T12:30:56.496 INFO:teuthology.orchestra.run.smithi060.stdout:(17/143): libradosstriper1-18.0.0-6675.gc747106 3.7 MB/s | 473 kB 00:00 2023-10-16T12:30:56.517 INFO:teuthology.orchestra.run.smithi144.stdout:(50/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2023-10-16T12:30:56.542 INFO:teuthology.orchestra.run.smithi144.stdout:(51/143): openblas-0.3.21-2.el9.x86_64.rpm 274 kB/s | 34 kB 00:00 2023-10-16T12:30:56.563 INFO:teuthology.orchestra.run.smithi060.stdout:(18/143): python3-ceph-argparse-18.0.0-6675.gc7 684 kB/s | 45 kB 00:00 2023-10-16T12:30:56.621 INFO:teuthology.orchestra.run.smithi060.stdout:(19/143): python3-ceph-common-18.0.0-6675.gc747 2.2 MB/s | 127 kB 00:00 2023-10-16T12:30:56.671 INFO:teuthology.orchestra.run.smithi060.stdout:(20/143): python3-cephfs-18.0.0-6675.gc7471068. 3.2 MB/s | 162 kB 00:00 2023-10-16T12:30:56.767 INFO:teuthology.orchestra.run.smithi144.stdout:(52/143): postgresql-13.11-1.el9.x86_64.rpm 6.8 MB/s | 1.5 MB 00:00 2023-10-16T12:30:56.889 INFO:teuthology.orchestra.run.smithi060.stdout:(21/143): python3-rados-18.0.0-6675.gc7471068.e 1.4 MB/s | 320 kB 00:00 2023-10-16T12:30:56.937 INFO:teuthology.orchestra.run.smithi144.stdout:(53/143): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2023-10-16T12:30:56.972 INFO:teuthology.orchestra.run.smithi060.stdout:(22/143): python3-rbd-18.0.0-6675.gc7471068.el9 3.5 MB/s | 297 kB 00:00 2023-10-16T12:30:56.987 INFO:teuthology.orchestra.run.smithi144.stdout:(54/143): postgresql-contrib-13.11-1.el9.x86_64 3.7 MB/s | 811 kB 00:00 2023-10-16T12:30:57.012 INFO:teuthology.orchestra.run.smithi144.stdout:(55/143): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2023-10-16T12:30:57.040 INFO:teuthology.orchestra.run.smithi060.stdout:(23/143): python3-rgw-18.0.0-6675.gc7471068.el9 1.5 MB/s | 99 kB 00:00 2023-10-16T12:30:57.098 INFO:teuthology.orchestra.run.smithi060.stdout:(24/143): rbd-fuse-18.0.0-6675.gc7471068.el9.x8 1.5 MB/s | 85 kB 00:00 2023-10-16T12:30:57.298 INFO:teuthology.orchestra.run.smithi060.stdout:(25/143): librgw2-18.0.0-6675.gc7471068.el9.x86 5.4 MB/s | 4.8 MB 00:00 2023-10-16T12:30:57.357 INFO:teuthology.orchestra.run.smithi060.stdout:(26/143): rbd-nbd-18.0.0-6675.gc7471068.el9.x86 2.9 MB/s | 168 kB 00:00 2023-10-16T12:30:57.388 INFO:teuthology.orchestra.run.smithi144.stdout:(56/143): postgresql-server-13.11-1.el9.x86_64. 14 MB/s | 5.7 MB 00:00 2023-10-16T12:30:57.424 INFO:teuthology.orchestra.run.smithi060.stdout:(27/143): ceph-grafana-dashboards-18.0.0-6675.g 353 kB/s | 23 kB 00:00 2023-10-16T12:30:57.491 INFO:teuthology.orchestra.run.smithi060.stdout:(28/143): ceph-mgr-cephadm-18.0.0-6675.gc747106 2.0 MB/s | 132 kB 00:00 2023-10-16T12:30:57.666 INFO:teuthology.orchestra.run.smithi060.stdout:(29/143): rbd-mirror-18.0.0-6675.gc7471068.el9. 5.3 MB/s | 3.0 MB 00:00 2023-10-16T12:30:57.680 INFO:teuthology.orchestra.run.smithi144.stdout:(57/143): python3-babel-2.9.1-2.el9.noarch.rpm 8.7 MB/s | 5.8 MB 00:00 2023-10-16T12:30:57.705 INFO:teuthology.orchestra.run.smithi144.stdout:(58/143): python3-devel-3.9.17-2.el9.x86_64.rpm 677 kB/s | 205 kB 00:00 2023-10-16T12:30:57.764 INFO:teuthology.orchestra.run.smithi144.stdout:(59/143): python3-jsonpatch-1.21-16.el9.noarch. 429 kB/s | 24 kB 00:00 2023-10-16T12:30:57.797 INFO:teuthology.orchestra.run.smithi144.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 2.5 MB/s | 228 kB 00:00 2023-10-16T12:30:57.808 INFO:teuthology.orchestra.run.smithi060.stdout:(30/143): ceph-mgr-dashboard-18.0.0-6675.gc7471 5.6 MB/s | 1.8 MB 00:00 2023-10-16T12:30:57.814 INFO:teuthology.orchestra.run.smithi144.stdout:(61/143): python3-jsonpointer-2.0-4.el9.noarch. 379 kB/s | 18 kB 00:00 2023-10-16T12:30:57.856 INFO:teuthology.orchestra.run.smithi144.stdout:(62/143): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2023-10-16T12:30:57.873 INFO:teuthology.orchestra.run.smithi144.stdout:(63/143): python3-markupsafe-1.1.1-12.el9.x86_6 545 kB/s | 32 kB 00:00 2023-10-16T12:30:57.883 INFO:teuthology.orchestra.run.smithi060.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6675.gc7 3.2 MB/s | 243 kB 00:00 2023-10-16T12:30:57.959 INFO:teuthology.orchestra.run.smithi060.stdout:(32/143): ceph-mgr-rook-18.0.0-6675.gc7471068.e 642 kB/s | 47 kB 00:00 2023-10-16T12:30:58.042 INFO:teuthology.orchestra.run.smithi060.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6675.gc 179 kB/s | 15 kB 00:00 2023-10-16T12:30:58.194 INFO:teuthology.orchestra.run.smithi060.stdout:(34/143): ceph-volume-18.0.0-6675.gc7471068.el9 1.7 MB/s | 262 kB 00:00 2023-10-16T12:30:58.207 INFO:teuthology.orchestra.run.smithi144.stdout:(64/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2023-10-16T12:30:58.265 INFO:teuthology.orchestra.run.smithi144.stdout:(65/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 556 kB/s | 217 kB 00:00 2023-10-16T12:30:58.285 INFO:teuthology.orchestra.run.smithi060.stdout:(35/143): cephadm-18.0.0-6675.gc7471068.el9.noa 2.9 MB/s | 269 kB 00:00 2023-10-16T12:30:58.332 INFO:teuthology.orchestra.run.smithi144.stdout:(66/143): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2023-10-16T12:30:58.357 INFO:teuthology.orchestra.run.smithi144.stdout:(67/143): python3-prettytable-0.7.2-27.el9.noar 460 kB/s | 41 kB 00:00 2023-10-16T12:30:58.368 INFO:teuthology.orchestra.run.smithi060.stdout:(36/143): cephfs-top-18.0.0-6675.gc7471068.el9. 222 kB/s | 18 kB 00:00 2023-10-16T12:30:58.408 INFO:teuthology.orchestra.run.smithi144.stdout:(68/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-16T12:30:58.458 INFO:teuthology.orchestra.run.smithi144.stdout:(69/143): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2023-10-16T12:30:58.483 INFO:teuthology.orchestra.run.smithi144.stdout:(70/143): python3-requests-oauthlib-1.3.0-12.el 587 kB/s | 43 kB 00:00 2023-10-16T12:30:58.727 INFO:teuthology.orchestra.run.smithi060.stdout:(37/143): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 920 kB 00:00 2023-10-16T12:30:58.775 INFO:teuthology.orchestra.run.smithi144.stdout:(71/143): python3-toml-0.10.2-6.el9.noarch.rpm 133 kB/s | 39 kB 00:00 2023-10-16T12:30:58.786 INFO:teuthology.orchestra.run.smithi060.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2023-10-16T12:30:58.928 INFO:teuthology.orchestra.run.smithi060.stdout:(39/143): ceph-mgr-diskprediction-local-18.0.0- 5.9 MB/s | 7.4 MB 00:01 2023-10-16T12:30:58.969 INFO:teuthology.orchestra.run.smithi060.stdout:(40/143): flex-2.6.4-9.el9.x86_64.rpm 1.6 MB/s | 307 kB 00:00 2023-10-16T12:30:59.011 INFO:teuthology.orchestra.run.smithi060.stdout:(41/143): flexiblas-3.0.4-8.el9.x86_64.rpm 398 kB/s | 32 kB 00:00 2023-10-16T12:30:59.212 INFO:teuthology.orchestra.run.smithi060.stdout:(42/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2023-10-16T12:30:59.254 INFO:teuthology.orchestra.run.smithi060.stdout:(43/143): flexiblas-openblas-openmp-3.0.4-8.el9 72 kB/s | 17 kB 00:00 2023-10-16T12:30:59.259 INFO:teuthology.orchestra.run.smithi144.stdout:(72/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2023-10-16T12:30:59.292 INFO:teuthology.orchestra.run.smithi144.stdout:(73/143): socat-1.7.4.1-5.el9.x86_64.rpm 590 kB/s | 300 kB 00:00 2023-10-16T12:30:59.296 INFO:teuthology.orchestra.run.smithi060.stdout:(44/143): libaio-devel-0.3.111-13.el9.x86_64.rp 122 kB/s | 10 kB 00:00 2023-10-16T12:30:59.317 INFO:teuthology.orchestra.run.smithi144.stdout:(74/143): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2023-10-16T12:30:59.321 INFO:teuthology.orchestra.run.smithi060.stdout:(45/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2023-10-16T12:30:59.343 INFO:teuthology.orchestra.run.smithi144.stdout:(75/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-10-16T12:30:59.363 INFO:teuthology.orchestra.run.smithi060.stdout:(46/143): librdkafka-1.6.1-102.el9.x86_64.rpm 10 MB/s | 661 kB 00:00 2023-10-16T12:30:59.368 INFO:teuthology.orchestra.run.smithi144.stdout:(76/143): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2023-10-16T12:30:59.459 INFO:teuthology.orchestra.run.smithi060.stdout:(47/143): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 578 kB 00:00 2023-10-16T12:30:59.497 INFO:teuthology.orchestra.run.smithi060.stdout:(48/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 363 kB/s | 46 kB 00:00 2023-10-16T12:30:59.531 INFO:teuthology.orchestra.run.smithi060.stdout:(49/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.9 MB/s | 282 kB 00:00 2023-10-16T12:30:59.564 INFO:teuthology.orchestra.run.smithi060.stdout:(50/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-10-16T12:30:59.606 INFO:teuthology.orchestra.run.smithi060.stdout:(51/143): openblas-0.3.21-2.el9.x86_64.rpm 472 kB/s | 34 kB 00:00 2023-10-16T12:30:59.644 INFO:teuthology.orchestra.run.smithi144.stdout:(77/143): ceph-common-18.0.0-6675.gc7471068.el9 1.6 MB/s | 19 MB 00:11 2023-10-16T12:30:59.727 INFO:teuthology.orchestra.run.smithi144.stdout:(78/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2023-10-16T12:30:59.760 INFO:teuthology.orchestra.run.smithi144.stdout:(79/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 505 kB/s | 195 kB 00:00 2023-10-16T12:30:59.782 INFO:teuthology.orchestra.run.smithi060.stdout:(52/143): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2023-10-16T12:30:59.794 INFO:teuthology.orchestra.run.smithi144.stdout:(80/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 241 kB 00:00 2023-10-16T12:30:59.820 INFO:teuthology.orchestra.run.smithi144.stdout:(81/143): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2023-10-16T12:30:59.840 INFO:teuthology.orchestra.run.smithi060.stdout:(53/143): postgresql-13.11-1.el9.x86_64.rpm 7.0 MB/s | 1.5 MB 00:00 2023-10-16T12:30:59.845 INFO:teuthology.orchestra.run.smithi144.stdout:(82/143): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2023-10-16T12:30:59.870 INFO:teuthology.orchestra.run.smithi144.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-10-16T12:30:59.882 INFO:teuthology.orchestra.run.smithi060.stdout:(54/143): postgresql-contrib-13.11-1.el9.x86_64 8.9 MB/s | 811 kB 00:00 2023-10-16T12:30:59.904 INFO:teuthology.orchestra.run.smithi144.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 3.3 MB/s | 188 kB 00:00 2023-10-16T12:30:59.908 INFO:teuthology.orchestra.run.smithi060.stdout:(55/143): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2023-10-16T12:31:00.075 INFO:teuthology.orchestra.run.smithi060.stdout:(56/143): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2023-10-16T12:31:00.213 INFO:teuthology.orchestra.run.smithi144.stdout:(85/143): elfutils-libelf-devel-0.189-3.el9.x86 72 kB/s | 22 kB 00:00 2023-10-16T12:31:00.225 INFO:teuthology.orchestra.run.smithi060.stdout:(57/143): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-10-16T12:31:00.250 INFO:teuthology.orchestra.run.smithi060.stdout:(58/143): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 205 kB 00:00 2023-10-16T12:31:00.309 INFO:teuthology.orchestra.run.smithi060.stdout:(59/143): python3-jsonpatch-1.21-16.el9.noarch. 430 kB/s | 24 kB 00:00 2023-10-16T12:31:00.342 INFO:teuthology.orchestra.run.smithi060.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 2.5 MB/s | 228 kB 00:00 2023-10-16T12:31:00.384 INFO:teuthology.orchestra.run.smithi060.stdout:(61/143): python3-jsonpointer-2.0-4.el9.noarch. 249 kB/s | 18 kB 00:00 2023-10-16T12:31:00.410 INFO:teuthology.orchestra.run.smithi060.stdout:(62/143): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2023-10-16T12:31:00.452 INFO:teuthology.orchestra.run.smithi060.stdout:(63/143): python3-markupsafe-1.1.1-12.el9.x86_6 485 kB/s | 32 kB 00:00 2023-10-16T12:31:00.613 INFO:teuthology.orchestra.run.smithi144.stdout:(86/143): automake-1.16.2-8.el9.noarch.rpm 931 kB/s | 689 kB 00:00 2023-10-16T12:31:00.627 INFO:teuthology.orchestra.run.smithi060.stdout:(64/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2023-10-16T12:31:00.638 INFO:teuthology.orchestra.run.smithi144.stdout:(87/143): jq-1.6-15.el9.x86_64.rpm 440 kB/s | 187 kB 00:00 2023-10-16T12:31:00.652 INFO:teuthology.orchestra.run.smithi060.stdout:(65/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2023-10-16T12:31:00.677 INFO:teuthology.orchestra.run.smithi060.stdout:(66/143): python3-oauthlib-3.1.1-5.el9.noarch.r 3.6 MB/s | 174 kB 00:00 2023-10-16T12:31:00.720 INFO:teuthology.orchestra.run.smithi144.stdout:(88/143): python3-cryptography-36.0.1-4.el9.x86 1.4 MB/s | 1.2 MB 00:00 2023-10-16T12:31:00.728 INFO:teuthology.orchestra.run.smithi060.stdout:(67/143): python3-prettytable-0.7.2-27.el9.noar 552 kB/s | 41 kB 00:00 2023-10-16T12:31:00.747 INFO:teuthology.orchestra.run.smithi144.stdout:(89/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 340 kB/s | 45 kB 00:00 2023-10-16T12:31:00.753 INFO:teuthology.orchestra.run.smithi060.stdout:(68/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-16T12:31:00.772 INFO:teuthology.orchestra.run.smithi144.stdout:(90/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 192 kB/s | 26 kB 00:00 2023-10-16T12:31:00.787 INFO:teuthology.orchestra.run.smithi060.stdout:(69/143): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2023-10-16T12:31:00.837 INFO:teuthology.orchestra.run.smithi060.stdout:(70/143): python3-requests-oauthlib-1.3.0-12.el 528 kB/s | 43 kB 00:00 2023-10-16T12:31:00.864 INFO:teuthology.orchestra.run.smithi144.stdout:(91/143): python3-pytz-2021.1-5.el9.noarch.rpm 438 kB/s | 51 kB 00:00 2023-10-16T12:31:00.931 INFO:teuthology.orchestra.run.smithi144.stdout:(92/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 545 kB/s | 86 kB 00:00 2023-10-16T12:31:00.981 INFO:teuthology.orchestra.run.smithi144.stdout:(93/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 389 kB/s | 45 kB 00:00 2023-10-16T12:31:01.098 INFO:teuthology.orchestra.run.smithi144.stdout:(94/143): openssl-devel-3.0.7-24.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2023-10-16T12:31:01.157 INFO:teuthology.orchestra.run.smithi144.stdout:(95/143): fmt-8.1.1-5.el9.x86_64.rpm 494 kB/s | 111 kB 00:00 2023-10-16T12:31:01.224 INFO:teuthology.orchestra.run.smithi144.stdout:(96/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.2 MB/s | 309 kB 00:00 2023-10-16T12:31:01.249 INFO:teuthology.orchestra.run.smithi144.stdout:(97/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 267 kB/s | 24 kB 00:00 2023-10-16T12:31:01.333 INFO:teuthology.orchestra.run.smithi144.stdout:(98/143): liboath-2.6.7-2.el9.x86_64.rpm 450 kB/s | 49 kB 00:00 2023-10-16T12:31:01.337 INFO:teuthology.orchestra.run.smithi060.stdout:(71/143): ceph-test-18.0.0-6675.gc7471068.el9.x 6.6 MB/s | 41 MB 00:06 2023-10-16T12:31:01.366 INFO:teuthology.orchestra.run.smithi144.stdout:(99/143): libunwind-1.6.2-1.el9.x86_64.rpm 579 kB/s | 67 kB 00:00 2023-10-16T12:31:01.508 INFO:teuthology.orchestra.run.smithi144.stdout:(100/143): libarrow-9.0.0-9.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2023-10-16T12:31:01.567 INFO:teuthology.orchestra.run.smithi144.stdout:(101/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 757 kB/s | 43 kB 00:00 2023-10-16T12:31:01.579 INFO:teuthology.orchestra.run.smithi060.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 53 kB/s | 39 kB 00:00 2023-10-16T12:31:01.617 INFO:teuthology.orchestra.run.smithi144.stdout:(102/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.9 MB/s | 837 kB 00:00 2023-10-16T12:31:01.667 INFO:teuthology.orchestra.run.smithi144.stdout:(103/143): python3-certifi-2021.10.8-2.el9.noar 303 kB/s | 15 kB 00:00 2023-10-16T12:31:01.692 INFO:teuthology.orchestra.run.smithi144.stdout:(104/143): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2023-10-16T12:31:01.751 INFO:teuthology.orchestra.run.smithi144.stdout:(105/143): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2023-10-16T12:31:01.785 INFO:teuthology.orchestra.run.smithi144.stdout:(106/143): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2023-10-16T12:31:01.788 INFO:teuthology.orchestra.run.smithi060.stdout:(73/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2023-10-16T12:31:01.813 INFO:teuthology.orchestra.run.smithi060.stdout:(74/143): socat-1.7.4.1-5.el9.x86_64.rpm 651 kB/s | 300 kB 00:00 2023-10-16T12:31:01.818 INFO:teuthology.orchestra.run.smithi144.stdout:(107/143): python3-asyncssh-2.13.2-3.el9.noarch 1.2 MB/s | 548 kB 00:00 2023-10-16T12:31:01.844 INFO:teuthology.orchestra.run.smithi144.stdout:(108/143): python3-jaraco-8.2.1-3.el9.noarch.rp 186 kB/s | 11 kB 00:00 2023-10-16T12:31:01.863 INFO:teuthology.orchestra.run.smithi060.stdout:(75/143): uuid-1.6.2-55.el9.x86_64.rpm 205 kB/s | 56 kB 00:00 2023-10-16T12:31:01.877 INFO:teuthology.orchestra.run.smithi144.stdout:(109/143): python3-google-auth-2.23.2-1.el9.noa 1.5 MB/s | 196 kB 00:00 2023-10-16T12:31:01.903 INFO:teuthology.orchestra.run.smithi144.stdout:(110/143): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2023-10-16T12:31:01.914 INFO:teuthology.orchestra.run.smithi060.stdout:(76/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 635 kB/s | 63 kB 00:00 2023-10-16T12:31:01.939 INFO:teuthology.orchestra.run.smithi060.stdout:(77/143): fuse-2.9.9-15.el9.x86_64.rpm 635 kB/s | 79 kB 00:00 2023-10-16T12:31:01.953 INFO:teuthology.orchestra.run.smithi144.stdout:(111/143): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2023-10-16T12:31:01.978 INFO:teuthology.orchestra.run.smithi144.stdout:(112/143): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2023-10-16T12:31:01.989 INFO:teuthology.orchestra.run.smithi060.stdout:(78/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2023-10-16T12:31:02.003 INFO:teuthology.orchestra.run.smithi144.stdout:(113/143): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2023-10-16T12:31:02.029 INFO:teuthology.orchestra.run.smithi144.stdout:(114/143): python3-jwt+crypto-2.4.0-1.el9.noarc 121 kB/s | 9.0 kB 00:00 2023-10-16T12:31:02.048 INFO:teuthology.orchestra.run.smithi060.stdout:(79/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2023-10-16T12:31:02.073 INFO:teuthology.orchestra.run.smithi060.stdout:(80/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 241 kB 00:00 2023-10-16T12:31:02.079 INFO:teuthology.orchestra.run.smithi144.stdout:(115/143): python3-jwt-2.4.0-1.el9.noarch.rpm 552 kB/s | 41 kB 00:00 2023-10-16T12:31:02.098 INFO:teuthology.orchestra.run.smithi060.stdout:(81/143): python3-ply-3.11-14.el9.noarch.rpm 958 kB/s | 103 kB 00:00 2023-10-16T12:31:02.104 INFO:teuthology.orchestra.run.smithi144.stdout:(116/143): python3-logutils-0.3.5-21.el9.noarch 621 kB/s | 46 kB 00:00 2023-10-16T12:31:02.132 INFO:teuthology.orchestra.run.smithi060.stdout:(82/143): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2023-10-16T12:31:02.157 INFO:teuthology.orchestra.run.smithi060.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 1.4 MB/s | 113 kB 00:00 2023-10-16T12:31:02.163 INFO:teuthology.orchestra.run.smithi144.stdout:(117/143): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2023-10-16T12:31:02.188 INFO:teuthology.orchestra.run.smithi144.stdout:(118/143): python3-more-itertools-8.12.0-2.el9. 729 kB/s | 79 kB 00:00 2023-10-16T12:31:02.191 INFO:teuthology.orchestra.run.smithi060.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 2.0 MB/s | 188 kB 00:00 2023-10-16T12:31:02.230 INFO:teuthology.orchestra.run.smithi144.stdout:(119/143): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2023-10-16T12:31:02.255 INFO:teuthology.orchestra.run.smithi144.stdout:(120/143): python3-portend-3.1.0-2.el9.noarch.r 248 kB/s | 16 kB 00:00 2023-10-16T12:31:02.297 INFO:teuthology.orchestra.run.smithi144.stdout:(121/143): python3-kubernetes-26.1.0-2.el9.noar 3.5 MB/s | 1.0 MB 00:00 2023-10-16T12:31:02.314 INFO:teuthology.orchestra.run.smithi144.stdout:(122/143): python3-repoze-lru-0.7-16.el9.noarch 531 kB/s | 31 kB 00:00 2023-10-16T12:31:02.331 INFO:teuthology.orchestra.run.smithi144.stdout:(123/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 901 kB/s | 90 kB 00:00 2023-10-16T12:31:02.382 INFO:teuthology.orchestra.run.smithi144.stdout:(124/143): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2023-10-16T12:31:02.398 INFO:teuthology.orchestra.run.smithi144.stdout:(125/143): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2023-10-16T12:31:02.416 INFO:teuthology.orchestra.run.smithi060.stdout:(85/143): python3-cryptography-36.0.1-4.el9.x86 4.8 MB/s | 1.2 MB 00:00 2023-10-16T12:31:02.432 INFO:teuthology.orchestra.run.smithi144.stdout:(126/143): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2023-10-16T12:31:02.449 INFO:teuthology.orchestra.run.smithi144.stdout:(127/143): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2023-10-16T12:31:02.507 INFO:teuthology.orchestra.run.smithi144.stdout:(128/143): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2023-10-16T12:31:02.509 INFO:teuthology.orchestra.run.smithi060.stdout:(86/143): elfutils-libelf-devel-0.189-3.el9.x86 70 kB/s | 22 kB 00:00 2023-10-16T12:31:02.566 INFO:teuthology.orchestra.run.smithi144.stdout:(129/143): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2023-10-16T12:31:02.600 INFO:teuthology.orchestra.run.smithi144.stdout:(130/143): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2023-10-16T12:31:02.625 INFO:teuthology.orchestra.run.smithi144.stdout:(131/143): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2023-10-16T12:31:02.675 INFO:teuthology.orchestra.run.smithi144.stdout:(132/143): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-10-16T12:31:02.808 INFO:teuthology.orchestra.run.smithi060.stdout:(87/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 152 kB/s | 45 kB 00:00 2023-10-16T12:31:02.925 INFO:teuthology.orchestra.run.smithi060.stdout:(88/143): automake-1.16.2-8.el9.noarch.rpm 901 kB/s | 689 kB 00:00 2023-10-16T12:31:02.975 INFO:teuthology.orchestra.run.smithi144.stdout:(133/143): librbd1-18.0.0-6675.gc7471068.el9.x8 10 MB/s | 3.0 MB 00:00 2023-10-16T12:31:02.978 INFO:teuthology.orchestra.run.smithi060.stdout:(89/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 151 kB/s | 26 kB 00:00 2023-10-16T12:31:03.025 INFO:teuthology.orchestra.run.smithi144.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 68 kB 00:00 2023-10-16T12:31:03.042 INFO:teuthology.orchestra.run.smithi060.stdout:(90/143): jq-1.6-15.el9.x86_64.rpm 299 kB/s | 187 kB 00:00 2023-10-16T12:31:03.126 INFO:teuthology.orchestra.run.smithi060.stdout:(91/143): python3-pytz-2021.1-5.el9.noarch.rpm 348 kB/s | 51 kB 00:00 2023-10-16T12:31:03.185 INFO:teuthology.orchestra.run.smithi060.stdout:(92/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 611 kB/s | 86 kB 00:00 2023-10-16T12:31:03.193 INFO:teuthology.orchestra.run.smithi144.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 13 MB/s | 2.2 MB 00:00 2023-10-16T12:31:03.310 INFO:teuthology.orchestra.run.smithi060.stdout:(93/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 247 kB/s | 45 kB 00:00 2023-10-16T12:31:03.326 INFO:teuthology.orchestra.run.smithi144.stdout:(136/143): NetworkManager-libnm-1.45.4-1.el9.x8 13 MB/s | 1.8 MB 00:00 2023-10-16T12:31:03.393 INFO:teuthology.orchestra.run.smithi144.stdout:(137/143): NetworkManager-team-1.45.4-1.el9.x86 628 kB/s | 41 kB 00:00 2023-10-16T12:31:03.402 INFO:teuthology.orchestra.run.smithi060.stdout:(94/143): openssl-devel-3.0.7-24.el9.x86_64.rpm 8.7 MB/s | 4.1 MB 00:00 2023-10-16T12:31:03.444 INFO:teuthology.orchestra.run.smithi144.stdout:(138/143): NetworkManager-tui-1.45.4-1.el9.x86_ 4.9 MB/s | 244 kB 00:00 2023-10-16T12:31:03.477 INFO:teuthology.orchestra.run.smithi144.stdout:(139/143): acl-2.3.1-4.el9.x86_64.rpm 2.1 MB/s | 69 kB 00:00 2023-10-16T12:31:03.528 INFO:teuthology.orchestra.run.smithi144.stdout:(140/143): thrift-0.15.0-2.el9.x86_64.rpm 1.7 MB/s | 1.6 MB 00:00 2023-10-16T12:31:03.577 INFO:teuthology.orchestra.run.smithi060.stdout:(95/143): fmt-8.1.1-5.el9.x86_64.rpm 283 kB/s | 111 kB 00:00 2023-10-16T12:31:03.578 INFO:teuthology.orchestra.run.smithi144.stdout:(141/143): libacl-2.3.1-4.el9.x86_64.rpm 220 kB/s | 22 kB 00:00 2023-10-16T12:31:03.612 INFO:teuthology.orchestra.run.smithi144.stdout:(142/143): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 321 kB 00:00 2023-10-16T12:31:03.627 INFO:teuthology.orchestra.run.smithi060.stdout:(96/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 516 kB/s | 24 kB 00:00 2023-10-16T12:31:03.669 INFO:teuthology.orchestra.run.smithi060.stdout:(97/143): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2023-10-16T12:31:03.736 INFO:teuthology.orchestra.run.smithi060.stdout:(98/143): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2023-10-16T12:31:03.770 INFO:teuthology.orchestra.run.smithi060.stdout:(99/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 676 kB/s | 309 kB 00:00 2023-10-16T12:31:03.803 INFO:teuthology.orchestra.run.smithi144.stdout:(143/143): librados2-18.0.0-6675.gc7471068.el9. 2.8 MB/s | 3.3 MB 00:01 2023-10-16T12:31:03.812 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2023-10-16T12:31:03.812 INFO:teuthology.orchestra.run.smithi144.stdout:Total 12 MB/s | 200 MB 00:16 2023-10-16T12:31:04.145 INFO:teuthology.orchestra.run.smithi060.stdout:(100/143): python3-asyncssh-2.13.2-3.el9.noarch 1.4 MB/s | 548 kB 00:00 2023-10-16T12:31:04.212 INFO:teuthology.orchestra.run.smithi060.stdout:(101/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 662 kB/s | 43 kB 00:00 2023-10-16T12:31:04.254 INFO:teuthology.orchestra.run.smithi060.stdout:(102/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.6 MB/s | 837 kB 00:00 2023-10-16T12:31:04.255 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2023-10-16T12:31:04.279 INFO:teuthology.orchestra.run.smithi060.stdout:(103/143): python3-cachetools-4.2.4-1.el9.noarc 487 kB/s | 32 kB 00:00 2023-10-16T12:31:04.313 INFO:teuthology.orchestra.run.smithi060.stdout:(104/143): python3-certifi-2021.10.8-2.el9.noar 260 kB/s | 15 kB 00:00 2023-10-16T12:31:04.413 INFO:teuthology.orchestra.run.smithi060.stdout:(105/143): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2023-10-16T12:31:04.457 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2023-10-16T12:31:04.458 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2023-10-16T12:31:04.472 INFO:teuthology.orchestra.run.smithi060.stdout:(106/143): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2023-10-16T12:31:04.522 INFO:teuthology.orchestra.run.smithi060.stdout:(107/143): python3-jaraco-8.2.1-3.el9.noarch.rp 218 kB/s | 11 kB 00:00 2023-10-16T12:31:04.631 INFO:teuthology.orchestra.run.smithi060.stdout:(108/143): python3-google-auth-2.23.2-1.el9.noa 907 kB/s | 196 kB 00:00 2023-10-16T12:31:04.657 INFO:teuthology.orchestra.run.smithi060.stdout:(109/143): python3-jaraco-classes-3.2.1-5.el9.n 133 kB/s | 18 kB 00:00 2023-10-16T12:31:04.682 INFO:teuthology.orchestra.run.smithi060.stdout:(110/143): python3-jaraco-collections-3.0.0-8.e 470 kB/s | 23 kB 00:00 2023-10-16T12:31:04.707 INFO:teuthology.orchestra.run.smithi060.stdout:(111/143): python3-jaraco-functools-3.5.0-2.el9 394 kB/s | 19 kB 00:00 2023-10-16T12:31:04.732 INFO:teuthology.orchestra.run.smithi060.stdout:(112/143): python3-jaraco-text-3.2.0-6.el9.noar 399 kB/s | 20 kB 00:00 2023-10-16T12:31:04.758 INFO:teuthology.orchestra.run.smithi060.stdout:(113/143): python3-jwt+crypto-2.4.0-1.el9.noarc 183 kB/s | 9.0 kB 00:00 2023-10-16T12:31:04.825 INFO:teuthology.orchestra.run.smithi060.stdout:(114/143): python3-jwt-2.4.0-1.el9.noarch.rpm 450 kB/s | 41 kB 00:00 2023-10-16T12:31:04.925 INFO:teuthology.orchestra.run.smithi060.stdout:(115/143): python3-logutils-0.3.5-21.el9.noarch 467 kB/s | 46 kB 00:00 2023-10-16T12:31:05.033 INFO:teuthology.orchestra.run.smithi060.stdout:(116/143): python3-more-itertools-8.12.0-2.el9. 734 kB/s | 79 kB 00:00 2023-10-16T12:31:05.134 INFO:teuthology.orchestra.run.smithi060.stdout:(117/143): python3-natsort-7.1.1-5.el9.noarch.r 583 kB/s | 58 kB 00:00 2023-10-16T12:31:05.226 INFO:teuthology.orchestra.run.smithi060.stdout:(118/143): python3-kubernetes-26.1.0-2.el9.noar 2.2 MB/s | 1.0 MB 00:00 2023-10-16T12:31:05.259 INFO:teuthology.orchestra.run.smithi060.stdout:(119/143): python3-portend-3.1.0-2.el9.noarch.r 507 kB/s | 16 kB 00:00 2023-10-16T12:31:05.368 INFO:teuthology.orchestra.run.smithi060.stdout:(120/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 831 kB/s | 90 kB 00:00 2023-10-16T12:31:05.418 INFO:teuthology.orchestra.run.smithi060.stdout:(121/143): python3-repoze-lru-0.7-16.el9.noarch 631 kB/s | 31 kB 00:00 2023-10-16T12:31:05.543 INFO:teuthology.orchestra.run.smithi060.stdout:(122/143): python3-pecan-1.4.2-3.el9.noarch.rpm 666 kB/s | 272 kB 00:00 2023-10-16T12:31:05.569 INFO:teuthology.orchestra.run.smithi060.stdout:(123/143): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2023-10-16T12:31:05.627 INFO:teuthology.orchestra.run.smithi060.stdout:(124/143): python3-tempora-5.0.0-2.el9.noarch.r 620 kB/s | 36 kB 00:00 2023-10-16T12:31:05.686 INFO:teuthology.orchestra.run.smithi060.stdout:(125/143): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2023-10-16T12:31:05.720 INFO:teuthology.orchestra.run.smithi060.stdout:(126/143): python3-typing-extensions-4.4.0-2.el 560 kB/s | 51 kB 00:00 2023-10-16T12:31:05.878 INFO:teuthology.orchestra.run.smithi060.stdout:(127/143): libarrow-9.0.0-9.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2023-10-16T12:31:05.903 INFO:teuthology.orchestra.run.smithi060.stdout:(128/143): python3-websocket-client-1.2.3-2.el9 491 kB/s | 90 kB 00:00 2023-10-16T12:31:05.962 INFO:teuthology.orchestra.run.smithi060.stdout:(129/143): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2023-10-16T12:31:06.113 INFO:teuthology.orchestra.run.smithi060.stdout:(130/143): python3-webob-1.8.7-6.el9.noarch.rpm 540 kB/s | 230 kB 00:00 2023-10-16T12:31:06.146 INFO:teuthology.orchestra.run.smithi060.stdout:(131/143): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2023-10-16T12:31:06.180 INFO:teuthology.orchestra.run.smithi060.stdout:(132/143): python3-werkzeug-2.0.3-3.el9.1.noarc 1.4 MB/s | 427 kB 00:00 2023-10-16T12:31:06.562 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2023-10-16T12:31:06.563 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2023-10-16T12:31:06.622 INFO:teuthology.orchestra.run.smithi060.stdout:(133/143): librbd1-18.0.0-6675.gc7471068.el9.x8 6.9 MB/s | 3.0 MB 00:00 2023-10-16T12:31:06.689 INFO:teuthology.orchestra.run.smithi060.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 68 kB 00:00 2023-10-16T12:31:06.906 INFO:teuthology.orchestra.run.smithi060.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 10 MB/s | 2.2 MB 00:00 2023-10-16T12:31:06.998 INFO:teuthology.orchestra.run.smithi060.stdout:(136/143): librados2-18.0.0-6675.gc7471068.el9. 3.9 MB/s | 3.3 MB 00:00 2023-10-16T12:31:07.081 INFO:teuthology.orchestra.run.smithi060.stdout:(137/143): NetworkManager-libnm-1.45.4-1.el9.x8 10 MB/s | 1.8 MB 00:00 2023-10-16T12:31:07.107 INFO:teuthology.orchestra.run.smithi060.stdout:(138/143): NetworkManager-team-1.45.4-1.el9.x86 446 kB/s | 41 kB 00:00 2023-10-16T12:31:07.140 INFO:teuthology.orchestra.run.smithi060.stdout:(139/143): NetworkManager-tui-1.45.4-1.el9.x86_ 4.2 MB/s | 244 kB 00:00 2023-10-16T12:31:07.165 INFO:teuthology.orchestra.run.smithi060.stdout:(140/143): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2023-10-16T12:31:07.191 INFO:teuthology.orchestra.run.smithi060.stdout:(141/143): libacl-2.3.1-4.el9.x86_64.rpm 453 kB/s | 22 kB 00:00 2023-10-16T12:31:07.224 INFO:teuthology.orchestra.run.smithi060.stdout:(142/143): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2023-10-16T12:31:07.874 INFO:teuthology.orchestra.run.smithi060.stdout:(143/143): thrift-0.15.0-2.el9.x86_64.rpm 920 kB/s | 1.6 MB 00:01 2023-10-16T12:31:07.888 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-10-16T12:31:07.888 INFO:teuthology.orchestra.run.smithi060.stdout:Total 11 MB/s | 200 MB 00:17 2023-10-16T12:31:08.336 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-10-16T12:31:08.554 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-10-16T12:31:08.554 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-10-16T12:31:10.819 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-10-16T12:31:10.819 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-10-16T12:31:12.463 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2023-10-16T12:31:12.580 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-16T12:31:12.601 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-16T12:31:12.820 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-16T12:31:12.932 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-16T12:31:13.354 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-16T12:31:13.391 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-16T12:31:13.537 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-16T12:31:13.575 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-16T12:31:13.696 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-16T12:31:13.761 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-16T12:31:13.810 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-16T12:31:14.058 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-16T12:31:14.138 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-16T12:31:14.163 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-16T12:31:14.314 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-16T12:31:14.365 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-16T12:31:14.448 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-16T12:31:14.852 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-16T12:31:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T12:31:15.083 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T12:31:15.333 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T12:31:15.423 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T12:31:15.865 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T12:31:15.903 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-16T12:31:15.957 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-16T12:31:16.068 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-16T12:31:16.138 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-16T12:31:16.174 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-16T12:31:16.229 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-16T12:31:16.333 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-16T12:31:16.376 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-16T12:31:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-16T12:31:16.506 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-16T12:31:16.556 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 27/153 2023-10-16T12:31:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 28/153 2023-10-16T12:31:16.715 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 29/153 2023-10-16T12:31:16.890 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-16T12:31:16.958 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-16T12:31:16.977 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-10-16T12:31:17.054 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-16T12:31:17.081 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-16T12:31:17.107 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-16T12:31:17.215 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-16T12:31:17.634 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-16T12:31:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-16T12:31:17.730 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-16T12:31:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-16T12:31:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 35/153 2023-10-16T12:31:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-16T12:31:17.925 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-16T12:31:18.011 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-16T12:31:18.056 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-16T12:31:18.126 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-16T12:31:18.299 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-16T12:31:18.301 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-16T12:31:18.342 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-16T12:31:18.355 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-16T12:31:18.382 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-16T12:31:18.456 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-16T12:31:18.513 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-16T12:31:18.521 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-16T12:31:18.573 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-16T12:31:18.595 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-16T12:31:18.643 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-16T12:31:18.762 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-16T12:31:18.814 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-16T12:31:18.860 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-16T12:31:18.943 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-16T12:31:18.990 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-16T12:31:19.002 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-16T12:31:19.026 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-16T12:31:19.031 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 50/153 2023-10-16T12:31:19.078 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-16T12:31:19.112 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-16T12:31:19.151 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-16T12:31:19.193 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-16T12:31:19.239 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-16T12:31:19.240 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-16T12:31:19.269 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-16T12:31:19.403 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-16T12:31:19.413 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-16T12:31:19.464 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-16T12:31:19.537 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-16T12:31:19.614 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-16T12:31:19.803 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-16T12:31:19.884 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-16T12:31:19.933 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T12:31:19.941 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-16T12:31:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-16T12:31:20.120 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-16T12:31:20.205 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-16T12:31:20.234 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-16T12:31:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-16T12:31:20.285 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T12:31:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-16T12:31:20.352 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T12:31:20.370 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-16T12:31:20.424 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-16T12:31:20.507 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-16T12:31:20.565 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-16T12:31:20.606 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-16T12:31:20.731 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-16T12:31:20.777 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-16T12:31:20.811 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-16T12:31:20.873 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-16T12:31:20.877 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-16T12:31:20.915 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-16T12:31:20.924 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-16T12:31:20.983 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-16T12:31:21.060 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-16T12:31:21.104 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-16T12:31:21.150 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-16T12:31:21.228 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-16T12:31:21.248 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-16T12:31:21.263 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-16T12:31:21.311 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-16T12:31:21.321 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-16T12:31:21.432 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-16T12:31:21.481 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-16T12:31:21.539 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-16T12:31:21.651 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-16T12:31:21.687 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 27/153 2023-10-16T12:31:21.739 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 28/153 2023-10-16T12:31:21.862 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 29/153 2023-10-16T12:31:22.050 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-16T12:31:22.156 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-16T12:31:22.329 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-16T12:31:22.384 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-16T12:31:22.417 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-16T12:31:22.545 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-16T12:31:22.584 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-16T12:31:22.833 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-16T12:31:22.865 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-16T12:31:22.911 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-16T12:31:22.957 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-16T12:31:23.155 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-16T12:31:23.215 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-16T12:31:23.256 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 35/153 2023-10-16T12:31:23.297 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-16T12:31:23.466 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-16T12:31:23.513 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-16T12:31:23.591 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-16T12:31:23.685 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-16T12:31:23.763 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-16T12:31:23.839 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-16T12:31:23.914 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-16T12:31:23.926 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-16T12:31:24.001 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-16T12:31:24.076 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-16T12:31:24.143 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-16T12:31:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-16T12:31:24.496 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-16T12:31:24.567 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-16T12:31:24.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-16T12:31:24.762 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 50/153 2023-10-16T12:31:24.820 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-16T12:31:24.860 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-16T12:31:24.900 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-16T12:31:24.943 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-16T12:31:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-16T12:31:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-16T12:31:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-16T12:31:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-16T12:31:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-16T12:31:25.489 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-16T12:31:25.569 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-16T12:31:25.606 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-16T12:31:25.637 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-16T12:31:25.670 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-16T12:31:25.677 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-16T12:31:25.738 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-16T12:31:25.946 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-16T12:31:26.008 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-16T12:31:26.035 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-16T12:31:26.039 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-16T12:31:26.059 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-16T12:31:26.079 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-16T12:31:26.123 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-16T12:31:26.175 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-16T12:31:26.179 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-16T12:31:26.214 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 98/153 2023-10-16T12:31:26.270 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-16T12:31:26.286 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-16T12:31:26.384 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-16T12:31:26.448 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-16T12:31:26.494 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-16T12:31:26.632 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-16T12:31:26.689 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-16T12:31:26.741 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-16T12:31:26.746 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-16T12:31:26.778 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-16T12:31:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-16T12:31:26.852 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-16T12:31:26.853 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-16T12:31:26.880 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 102/153 2023-10-16T12:31:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-16T12:31:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-16T12:31:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-16T12:31:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-16T12:31:28.334 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-16T12:31:28.387 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-16T12:31:28.421 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-16T12:31:28.557 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-16T12:31:28.630 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-16T12:31:28.889 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-16T12:31:28.926 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-16T12:31:29.045 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-16T12:31:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-16T12:31:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-16T12:31:29.933 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-16T12:31:30.080 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-16T12:31:30.157 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-16T12:31:30.214 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-16T12:31:30.421 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-16T12:31:30.421 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-16T12:31:30.422 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-16T12:31:30.422 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:30.465 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-16T12:31:31.743 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-16T12:31:31.831 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-16T12:31:31.880 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-16T12:31:31.916 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-16T12:31:31.973 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-16T12:31:32.361 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-16T12:31:32.414 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-16T12:31:32.603 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-16T12:31:32.635 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 98/153 2023-10-16T12:31:32.696 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-16T12:31:33.228 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-16T12:31:33.556 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-16T12:31:33.667 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-16T12:31:33.701 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 102/153 2023-10-16T12:31:36.703 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-16T12:31:36.746 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-16T12:31:37.417 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-16T12:31:37.467 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-16T12:31:37.668 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-16T12:31:37.668 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-16T12:31:37.669 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-16T12:31:37.669 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:37.718 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-16T12:31:43.143 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-16T12:31:43.143 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2023-10-16T12:31:43.143 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2023-10-16T12:31:43.144 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2023-10-16T12:31:43.144 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2023-10-16T12:31:43.144 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2023-10-16T12:31:43.144 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /root 2023-10-16T12:31:43.144 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2023-10-16T12:31:43.145 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:43.360 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-16T12:31:43.407 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-16T12:31:43.407 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:43.407 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-16T12:31:43.407 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:45.096 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-16T12:31:45.177 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-16T12:31:45.177 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:45.177 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-16T12:31:45.178 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T12:31:45.178 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T12:31:45.178 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:45.405 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-16T12:31:45.478 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-16T12:31:45.479 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:45.479 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-16T12:31:45.479 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T12:31:45.479 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T12:31:45.480 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:45.955 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-16T12:31:46.027 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-16T12:31:46.027 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:46.027 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-16T12:31:46.028 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T12:31:46.028 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T12:31:46.028 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:46.964 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-16T12:31:47.000 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-16T12:31:47.055 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 111/153 2023-10-16T12:31:47.170 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 112/153 2023-10-16T12:31:47.203 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-16T12:31:47.510 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-16T12:31:47.584 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-16T12:31:48.679 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-16T12:31:48.704 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-16T12:31:48.845 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-16T12:31:49.022 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 116/153 2023-10-16T12:31:49.048 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-16T12:31:49.121 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-16T12:31:49.121 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:49.122 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-16T12:31:49.122 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T12:31:49.122 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T12:31:49.122 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:49.174 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-16T12:31:49.212 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-16T12:31:50.061 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 119/153 2023-10-16T12:31:50.091 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-16T12:31:50.173 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-16T12:31:50.173 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:50.173 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-16T12:31:50.173 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T12:31:50.174 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T12:31:50.174 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:50.225 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-16T12:31:50.322 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-16T12:31:50.322 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:50.322 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-16T12:31:50.323 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:50.393 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-16T12:31:50.494 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-16T12:31:50.494 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:50.494 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-16T12:31:50.494 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T12:31:50.495 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T12:31:50.495 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-16T12:31:50.817 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2023-10-16T12:31:50.817 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2023-10-16T12:31:50.818 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2023-10-16T12:31:50.818 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2023-10-16T12:31:50.818 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2023-10-16T12:31:50.818 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2023-10-16T12:31:50.818 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2023-10-16T12:31:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:50.860 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-16T12:31:50.941 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-16T12:31:50.942 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:50.942 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-16T12:31:50.942 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T12:31:50.942 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T12:31:50.943 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:31:51.031 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-16T12:31:51.083 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-16T12:31:51.084 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:51.084 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-16T12:31:51.084 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:52.832 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-16T12:31:52.935 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-16T12:31:52.935 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:52.936 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-16T12:31:52.936 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T12:31:52.936 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T12:31:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:53.166 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-16T12:31:53.246 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-16T12:31:53.247 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:53.247 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-16T12:31:53.248 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T12:31:53.248 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T12:31:53.248 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:53.758 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-16T12:31:53.845 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-16T12:31:53.845 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:53.846 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-16T12:31:53.846 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T12:31:53.846 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T12:31:53.846 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-16T12:31:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-16T12:31:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 111/153 2023-10-16T12:31:54.962 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 112/153 2023-10-16T12:31:55.006 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-16T12:31:55.109 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 124/153 2023-10-16T12:31:55.164 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 125/153 2023-10-16T12:31:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 126/153 2023-10-16T12:31:55.352 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-16T12:31:55.400 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-16T12:31:55.486 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-16T12:31:55.607 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-16T12:31:55.670 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 129/153 2023-10-16T12:31:55.716 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-16T12:31:55.747 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-16T12:31:55.776 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 132/153 2023-10-16T12:31:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-16T12:31:55.946 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-16T12:31:56.032 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-16T12:31:56.123 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-16T12:31:56.614 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-16T12:31:56.651 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-16T12:31:56.802 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-16T12:31:56.983 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 116/153 2023-10-16T12:31:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-16T12:31:57.049 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T12:31:57.087 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-16T12:31:57.088 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:57.088 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-16T12:31:57.088 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T12:31:57.088 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T12:31:57.089 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:57.104 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T12:31:57.145 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-16T12:31:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T12:31:57.188 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-16T12:31:57.219 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 138/153 2023-10-16T12:31:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 119/153 2023-10-16T12:31:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-16T12:31:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-16T12:31:58.276 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:58.277 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-16T12:31:58.277 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T12:31:58.277 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T12:31:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-16T12:31:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-16T12:31:58.409 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:58.409 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-16T12:31:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-16T12:31:58.575 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-16T12:31:58.575 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:58.576 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-16T12:31:58.576 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T12:31:58.576 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-16T12:31:58.576 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:58.615 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-16T12:31:58.710 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-16T12:31:58.753 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-16T12:31:58.872 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-16T12:31:58.912 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-16T12:31:58.949 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-16T12:31:58.949 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T12:31:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-16T12:31:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-16T12:31:59.051 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T12:31:59.051 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-16T12:31:59.051 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T12:31:59.051 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T12:31:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:31:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T12:31:59.109 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-16T12:31:59.141 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-16T12:31:59.174 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-16T12:31:59.212 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T12:31:59.213 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T12:31:59.524 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T12:31:59.525 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T12:31:59.671 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T12:31:59.702 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T12:31:59.702 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T12:31:59.857 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T12:31:59.981 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T12:32:00.229 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T12:32:00.229 INFO:teuthology.orchestra.run.smithi144.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-16T12:32:00.230 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:03.438 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 124/153 2023-10-16T12:32:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 125/153 2023-10-16T12:32:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 126/153 2023-10-16T12:32:03.832 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-16T12:32:03.965 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-16T12:32:04.039 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 129/153 2023-10-16T12:32:04.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-16T12:32:04.125 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-16T12:32:04.155 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 132/153 2023-10-16T12:32:04.202 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-16T12:32:04.302 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-16T12:32:04.403 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-16T12:32:04.450 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-16T12:32:04.743 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 1/153 2023-10-16T12:32:04.744 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2/153 2023-10-16T12:32:04.744 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 3/153 2023-10-16T12:32:04.744 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 4/153 2023-10-16T12:32:04.744 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/153 2023-10-16T12:32:04.744 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-16T12:32:04.745 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-16T12:32:04.745 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-16T12:32:04.745 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 9/153 2023-10-16T12:32:04.745 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 10/153 2023-10-16T12:32:04.745 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 11/153 2023-10-16T12:32:04.746 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-16T12:32:04.746 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 13/153 2023-10-16T12:32:04.746 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 14/153 2023-10-16T12:32:04.746 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 15/153 2023-10-16T12:32:04.746 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 16/153 2023-10-16T12:32:04.747 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 17/153 2023-10-16T12:32:04.747 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 18/153 2023-10-16T12:32:04.747 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 19/153 2023-10-16T12:32:04.747 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/153 2023-10-16T12:32:04.747 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 21/153 2023-10-16T12:32:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 22/153 2023-10-16T12:32:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 23/153 2023-10-16T12:32:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 24/153 2023-10-16T12:32:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 25/153 2023-10-16T12:32:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 26/153 2023-10-16T12:32:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 27/153 2023-10-16T12:32:04.749 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 28/153 2023-10-16T12:32:04.760 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/153 2023-10-16T12:32:04.760 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 30/153 2023-10-16T12:32:04.760 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 31/153 2023-10-16T12:32:04.760 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/153 2023-10-16T12:32:04.760 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/153 2023-10-16T12:32:04.761 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 34/153 2023-10-16T12:32:04.761 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/153 2023-10-16T12:32:04.761 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 36/153 2023-10-16T12:32:04.761 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 37/153 2023-10-16T12:32:04.761 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 38/153 2023-10-16T12:32:04.762 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-16T12:32:04.762 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-16T12:32:04.762 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-16T12:32:04.762 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-16T12:32:04.762 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-16T12:32:04.763 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-16T12:32:04.763 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-16T12:32:04.763 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-16T12:32:04.763 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-16T12:32:04.763 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-16T12:32:04.764 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-16T12:32:04.764 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-16T12:32:04.764 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-16T12:32:04.764 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-16T12:32:04.764 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-16T12:32:04.765 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-16T12:32:04.765 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-16T12:32:04.765 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-16T12:32:04.765 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-16T12:32:04.765 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-16T12:32:04.766 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-16T12:32:04.766 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-16T12:32:04.766 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-16T12:32:04.767 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-16T12:32:04.767 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-16T12:32:04.767 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-16T12:32:04.767 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-16T12:32:04.768 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-16T12:32:04.768 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-16T12:32:04.768 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-16T12:32:04.768 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-16T12:32:04.768 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-16T12:32:04.769 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-16T12:32:04.769 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-16T12:32:04.769 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-16T12:32:04.769 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-16T12:32:04.769 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-16T12:32:04.769 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-16T12:32:04.770 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-16T12:32:04.770 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-16T12:32:04.770 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-16T12:32:04.770 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-16T12:32:04.770 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-16T12:32:04.771 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-16T12:32:04.771 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-16T12:32:04.771 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-16T12:32:04.771 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-16T12:32:04.771 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-16T12:32:04.772 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-16T12:32:04.772 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-16T12:32:04.772 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-16T12:32:04.772 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-16T12:32:04.772 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-16T12:32:04.773 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-16T12:32:04.773 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-16T12:32:04.773 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-16T12:32:04.773 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-16T12:32:04.774 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-16T12:32:04.774 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-16T12:32:04.774 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-16T12:32:04.774 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-16T12:32:04.774 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-16T12:32:04.775 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-16T12:32:04.775 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-16T12:32:04.775 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-16T12:32:04.775 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-16T12:32:04.775 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-16T12:32:04.776 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-16T12:32:04.776 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-16T12:32:04.776 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 108/153 2023-10-16T12:32:04.776 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-16T12:32:04.776 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-16T12:32:04.777 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-16T12:32:04.777 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-16T12:32:04.777 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-16T12:32:04.777 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-16T12:32:04.777 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-16T12:32:04.777 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-16T12:32:04.778 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-16T12:32:04.778 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-16T12:32:04.778 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-16T12:32:04.778 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-16T12:32:04.779 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-16T12:32:04.779 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-16T12:32:04.779 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-16T12:32:04.779 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-16T12:32:04.779 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-16T12:32:04.780 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-16T12:32:04.780 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-16T12:32:04.780 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-16T12:32:04.780 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-16T12:32:04.780 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-16T12:32:04.781 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-16T12:32:04.781 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-16T12:32:04.781 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-16T12:32:04.781 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 134/153 2023-10-16T12:32:04.781 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-16T12:32:04.781 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 136/153 2023-10-16T12:32:04.782 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-16T12:32:04.782 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-16T12:32:04.782 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-16T12:32:04.782 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-16T12:32:04.783 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-16T12:32:04.783 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-16T12:32:04.783 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-16T12:32:04.783 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-16T12:32:04.784 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-16T12:32:04.784 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-16T12:32:04.784 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-16T12:32:04.784 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-16T12:32:04.784 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-16T12:32:04.785 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-16T12:32:04.785 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T12:32:04.785 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-16T12:32:05.460 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T12:32:05.507 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T12:32:05.566 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-16T12:32:05.629 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 138/153 2023-10-16T12:32:06.856 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T12:32:06.857 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:06.857 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2023-10-16T12:32:06.857 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:06.857 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:06.858 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:06.858 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:06.858 INFO:teuthology.orchestra.run.smithi144.stdout: acl-2.3.1-4.el9.x86_64 2023-10-16T12:32:06.858 INFO:teuthology.orchestra.run.smithi144.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-16T12:32:06.859 INFO:teuthology.orchestra.run.smithi144.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-16T12:32:06.859 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.859 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.859 INFO:teuthology.orchestra.run.smithi144.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-16T12:32:06.859 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2023-10-16T12:32:06.860 INFO:teuthology.orchestra.run.smithi144.stdout: automake-1.16.2-8.el9.noarch 2023-10-16T12:32:06.860 INFO:teuthology.orchestra.run.smithi144.stdout: bison-3.7.4-5.el9.x86_64 2023-10-16T12:32:06.860 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-16T12:32:06.860 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.860 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.861 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.861 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.861 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.861 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.861 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.862 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.862 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.862 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.862 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.862 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.863 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.863 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.863 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.863 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.864 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.864 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.864 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.864 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.864 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.865 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.865 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:06.865 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-16T12:32:06.865 INFO:teuthology.orchestra.run.smithi144.stdout: flex-2.6.4-9.el9.x86_64 2023-10-16T12:32:06.865 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-16T12:32:06.866 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-16T12:32:06.866 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-16T12:32:06.866 INFO:teuthology.orchestra.run.smithi144.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-16T12:32:06.866 INFO:teuthology.orchestra.run.smithi144.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-16T12:32:06.866 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-16T12:32:06.867 INFO:teuthology.orchestra.run.smithi144.stdout: jq-1.6-15.el9.x86_64 2023-10-16T12:32:06.867 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-16T12:32:06.867 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-16T12:32:06.867 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-16T12:32:06.867 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.868 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.868 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.868 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-16T12:32:06.868 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-16T12:32:06.868 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-16T12:32:06.869 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-16T12:32:06.869 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-16T12:32:06.869 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.869 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.869 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-16T12:32:06.870 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.870 INFO:teuthology.orchestra.run.smithi144.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-16T12:32:06.870 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-16T12:32:06.870 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-16T12:32:06.870 INFO:teuthology.orchestra.run.smithi144.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-16T12:32:06.871 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-16T12:32:06.871 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-16T12:32:06.871 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-16T12:32:06.871 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-16T12:32:06.871 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-16T12:32:06.872 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-16T12:32:06.872 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-13.11-1.el9.x86_64 2023-10-16T12:32:06.872 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-16T12:32:06.872 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-16T12:32:06.873 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-16T12:32:06.873 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-16T12:32:06.873 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-16T12:32:06.873 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-16T12:32:06.874 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-16T12:32:06.874 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.874 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.874 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.874 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-16T12:32:06.875 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-16T12:32:06.875 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-16T12:32:06.875 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-16T12:32:06.875 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-16T12:32:06.875 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-16T12:32:06.876 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-16T12:32:06.876 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-16T12:32:06.876 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-16T12:32:06.876 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-16T12:32:06.876 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-16T12:32:06.877 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-16T12:32:06.877 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-16T12:32:06.877 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-16T12:32:06.877 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-16T12:32:06.877 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-16T12:32:06.878 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-16T12:32:06.878 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-16T12:32:06.878 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-16T12:32:06.878 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-16T12:32:06.879 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-16T12:32:06.879 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-16T12:32:06.879 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-16T12:32:06.879 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-16T12:32:06.879 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-16T12:32:06.880 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-16T12:32:06.881 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-16T12:32:06.881 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply-3.11-14.el9.noarch 2023-10-16T12:32:06.881 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-16T12:32:06.881 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-16T12:32:06.882 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-16T12:32:06.882 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-16T12:32:06.882 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-16T12:32:06.882 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-16T12:32:06.882 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-16T12:32:06.883 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.883 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.883 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-16T12:32:06.883 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-16T12:32:06.883 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-16T12:32:06.884 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.884 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-16T12:32:06.884 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-16T12:32:06.884 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-16T12:32:06.884 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-16T12:32:06.885 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-16T12:32:06.885 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-16T12:32:06.885 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-16T12:32:06.885 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-16T12:32:06.885 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-16T12:32:06.886 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-16T12:32:06.886 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-16T12:32:06.886 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.886 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.886 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:06.887 INFO:teuthology.orchestra.run.smithi144.stdout: re2-1:20211101-3.el9.x86_64 2023-10-16T12:32:06.887 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-16T12:32:06.887 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-16T12:32:06.887 INFO:teuthology.orchestra.run.smithi144.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-16T12:32:06.887 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-16T12:32:06.887 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-16T12:32:06.888 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-16T12:32:06.888 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:06.889 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-10-16T12:32:07.090 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-16T12:32:07.213 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-16T12:32:07.264 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-16T12:32:07.401 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-16T12:32:07.409 DEBUG:teuthology.parallel:result is None 2023-10-16T12:32:07.440 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-16T12:32:07.477 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-16T12:32:07.478 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T12:32:07.606 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-16T12:32:07.668 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-16T12:32:07.702 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-16T12:32:07.735 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-16T12:32:07.776 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T12:32:07.776 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T12:32:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-16T12:32:08.144 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T12:32:08.316 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-16T12:32:08.347 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T12:32:08.347 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T12:32:08.469 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-16T12:32:08.596 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T12:32:08.853 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T12:32:08.853 INFO:teuthology.orchestra.run.smithi060.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-16T12:32:08.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:13.586 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 1/153 2023-10-16T12:32:13.586 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2/153 2023-10-16T12:32:13.587 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 3/153 2023-10-16T12:32:13.587 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 4/153 2023-10-16T12:32:13.587 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/153 2023-10-16T12:32:13.587 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-16T12:32:13.587 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-16T12:32:13.588 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-16T12:32:13.588 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 9/153 2023-10-16T12:32:13.588 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 10/153 2023-10-16T12:32:13.588 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 11/153 2023-10-16T12:32:13.588 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-16T12:32:13.589 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 13/153 2023-10-16T12:32:13.589 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 14/153 2023-10-16T12:32:13.589 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 15/153 2023-10-16T12:32:13.589 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 16/153 2023-10-16T12:32:13.589 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 17/153 2023-10-16T12:32:13.589 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 18/153 2023-10-16T12:32:13.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 19/153 2023-10-16T12:32:13.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/153 2023-10-16T12:32:13.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 21/153 2023-10-16T12:32:13.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 22/153 2023-10-16T12:32:13.590 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 23/153 2023-10-16T12:32:13.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 24/153 2023-10-16T12:32:13.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 25/153 2023-10-16T12:32:13.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 26/153 2023-10-16T12:32:13.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 27/153 2023-10-16T12:32:13.591 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 28/153 2023-10-16T12:32:13.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/153 2023-10-16T12:32:13.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 30/153 2023-10-16T12:32:13.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 31/153 2023-10-16T12:32:13.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/153 2023-10-16T12:32:13.592 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/153 2023-10-16T12:32:13.593 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 34/153 2023-10-16T12:32:13.593 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/153 2023-10-16T12:32:13.593 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 36/153 2023-10-16T12:32:13.593 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 37/153 2023-10-16T12:32:13.593 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 38/153 2023-10-16T12:32:13.593 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-16T12:32:13.594 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-16T12:32:13.594 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-16T12:32:13.594 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-16T12:32:13.594 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-16T12:32:13.594 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-16T12:32:13.595 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-16T12:32:13.595 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-16T12:32:13.595 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-16T12:32:13.595 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-16T12:32:13.595 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-16T12:32:13.596 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-16T12:32:13.596 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-16T12:32:13.596 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-16T12:32:13.596 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-16T12:32:13.596 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-16T12:32:13.596 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-16T12:32:13.597 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-16T12:32:13.597 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-16T12:32:13.597 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-16T12:32:13.597 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-16T12:32:13.597 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-16T12:32:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-16T12:32:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-16T12:32:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-16T12:32:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-16T12:32:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-16T12:32:13.598 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-16T12:32:13.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-16T12:32:13.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-16T12:32:13.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-16T12:32:13.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-16T12:32:13.599 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-16T12:32:13.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-16T12:32:13.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-16T12:32:13.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-16T12:32:13.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-16T12:32:13.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-16T12:32:13.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-16T12:32:13.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-16T12:32:13.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-16T12:32:13.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-16T12:32:13.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-16T12:32:13.601 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-16T12:32:13.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-16T12:32:13.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-16T12:32:13.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-16T12:32:13.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-16T12:32:13.602 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-16T12:32:13.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-16T12:32:13.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-16T12:32:13.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-16T12:32:13.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-16T12:32:13.603 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-16T12:32:13.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-16T12:32:13.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-16T12:32:13.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-16T12:32:13.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-16T12:32:13.604 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-16T12:32:13.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-16T12:32:13.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-16T12:32:13.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-16T12:32:13.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-16T12:32:13.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-16T12:32:13.605 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-16T12:32:13.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-16T12:32:13.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-16T12:32:13.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-16T12:32:13.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-16T12:32:13.606 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 108/153 2023-10-16T12:32:13.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-16T12:32:13.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-16T12:32:13.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-16T12:32:13.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-16T12:32:13.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-16T12:32:13.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-16T12:32:13.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-16T12:32:13.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-16T12:32:13.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-16T12:32:13.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-16T12:32:13.608 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-16T12:32:13.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-16T12:32:13.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-16T12:32:13.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-16T12:32:13.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-16T12:32:13.609 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-16T12:32:13.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-16T12:32:13.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-16T12:32:13.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-16T12:32:13.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-16T12:32:13.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-16T12:32:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-16T12:32:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-16T12:32:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-16T12:32:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-16T12:32:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 134/153 2023-10-16T12:32:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-16T12:32:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 136/153 2023-10-16T12:32:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-16T12:32:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-16T12:32:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-16T12:32:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-16T12:32:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-16T12:32:13.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-16T12:32:13.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-16T12:32:13.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-16T12:32:13.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-16T12:32:13.613 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-16T12:32:13.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-16T12:32:13.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-16T12:32:13.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-16T12:32:13.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-16T12:32:13.614 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-16T12:32:13.615 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-16T12:32:15.881 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-16T12:32:15.881 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:15.881 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2023-10-16T12:32:15.882 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:15.882 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:15.882 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:15.882 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-16T12:32:15.882 INFO:teuthology.orchestra.run.smithi060.stdout: acl-2.3.1-4.el9.x86_64 2023-10-16T12:32:15.883 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-16T12:32:15.883 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-16T12:32:15.883 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.883 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.884 INFO:teuthology.orchestra.run.smithi060.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-16T12:32:15.884 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-10-16T12:32:15.884 INFO:teuthology.orchestra.run.smithi060.stdout: automake-1.16.2-8.el9.noarch 2023-10-16T12:32:15.884 INFO:teuthology.orchestra.run.smithi060.stdout: bison-3.7.4-5.el9.x86_64 2023-10-16T12:32:15.884 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-16T12:32:15.885 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.885 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.885 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.885 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.886 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.886 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.886 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.886 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.886 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.887 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.887 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.887 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.887 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.888 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.888 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.888 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.888 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.888 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.889 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.889 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.889 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.889 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.889 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-16T12:32:15.890 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-16T12:32:15.890 INFO:teuthology.orchestra.run.smithi060.stdout: flex-2.6.4-9.el9.x86_64 2023-10-16T12:32:15.890 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-16T12:32:15.890 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-16T12:32:15.891 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-16T12:32:15.891 INFO:teuthology.orchestra.run.smithi060.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-16T12:32:15.891 INFO:teuthology.orchestra.run.smithi060.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-16T12:32:15.891 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-16T12:32:15.891 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-15.el9.x86_64 2023-10-16T12:32:15.892 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-16T12:32:15.892 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-16T12:32:15.892 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-16T12:32:15.892 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.892 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.893 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.893 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-16T12:32:15.893 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-16T12:32:15.893 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-16T12:32:15.894 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-16T12:32:15.894 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-16T12:32:15.894 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.894 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.894 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-16T12:32:15.895 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.895 INFO:teuthology.orchestra.run.smithi060.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-16T12:32:15.895 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-16T12:32:15.895 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-16T12:32:15.896 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-16T12:32:15.896 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-16T12:32:15.896 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-16T12:32:15.896 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-16T12:32:15.896 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-16T12:32:15.897 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-16T12:32:15.897 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-16T12:32:15.897 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-13.11-1.el9.x86_64 2023-10-16T12:32:15.897 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-16T12:32:15.897 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-16T12:32:15.898 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-16T12:32:15.898 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-16T12:32:15.898 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-16T12:32:15.898 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-16T12:32:15.899 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-16T12:32:15.899 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.899 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.899 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.900 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-16T12:32:15.900 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-16T12:32:15.900 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-16T12:32:15.900 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-16T12:32:15.901 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-16T12:32:15.901 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-16T12:32:15.901 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-16T12:32:15.901 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-16T12:32:15.901 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-16T12:32:15.902 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-16T12:32:15.902 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-16T12:32:15.902 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-16T12:32:15.902 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-16T12:32:15.902 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-16T12:32:15.903 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-16T12:32:15.903 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-16T12:32:15.903 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-16T12:32:15.903 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-16T12:32:15.904 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-16T12:32:15.904 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-16T12:32:15.904 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-16T12:32:15.904 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-16T12:32:15.904 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-16T12:32:15.905 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-16T12:32:15.905 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-16T12:32:15.905 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-16T12:32:15.905 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-16T12:32:15.905 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2023-10-16T12:32:15.906 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-16T12:32:15.906 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-16T12:32:15.906 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-16T12:32:15.906 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-16T12:32:15.906 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-16T12:32:15.907 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-16T12:32:15.907 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-16T12:32:15.907 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.907 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.907 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-16T12:32:15.908 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-16T12:32:15.908 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-16T12:32:15.908 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.908 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-16T12:32:15.908 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-16T12:32:15.909 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-16T12:32:15.909 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-16T12:32:15.909 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-16T12:32:15.909 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-16T12:32:15.909 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-16T12:32:15.910 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-16T12:32:15.910 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-16T12:32:15.910 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-16T12:32:15.910 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-16T12:32:15.911 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.911 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.911 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-16T12:32:15.911 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2023-10-16T12:32:15.911 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-16T12:32:15.912 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-16T12:32:15.912 INFO:teuthology.orchestra.run.smithi060.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-16T12:32:15.912 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-16T12:32:15.912 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-16T12:32:15.913 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-16T12:32:15.913 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:15.913 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-10-16T12:32:16.450 DEBUG:teuthology.parallel:result is None 2023-10-16T12:32:16.450 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:32:16.815 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T12:32:16.876 INFO:teuthology.orchestra.run.smithi060.stdout:18.0.0-6675.gc7471068.el9 2023-10-16T12:32:16.877 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el9 2023-10-16T12:32:16.878 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-16T12:32:16.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:32:17.030 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T12:32:17.106 INFO:teuthology.orchestra.run.smithi144.stdout:18.0.0-6675.gc7471068.el9 2023-10-16T12:32:17.106 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el9 2023-10-16T12:32:17.107 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-16T12:32:17.107 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-16T12:32:17.108 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:17.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T12:32:17.190 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:17.190 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T12:32:17.301 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-16T12:32:17.301 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:17.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T12:32:17.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T12:32:17.506 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:17.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T12:32:17.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T12:32:17.758 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-16T12:32:17.758 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:17.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T12:32:17.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T12:32:17.962 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:17.963 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T12:32:18.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T12:32:18.188 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-16T12:32:18.189 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:18.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T12:32:18.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T12:32:18.402 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:18.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T12:32:18.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T12:32:18.578 INFO:teuthology.run_tasks:Running task ceph... 2023-10-16T12:32:18.645 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-16T12:32:18.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2023-10-16T12:32:18.652 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2023-10-16T12:32:18.737 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-16T12:32:18.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T12:32:18.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T12:32:18.823 INFO:tasks.ceph:Creating extra log directories... 2023-10-16T12:32:18.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T12:32:18.828 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T12:32:18.924 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-16T12:32:18.924 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-16T12:32:18.924 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-16T12:32:18.925 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:18.925 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T12:32:18.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:19.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:19.253 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:19.254 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T12:32:19.310 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:19.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:19.685 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-16T12:32:19.685 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-16T12:32:19.686 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986', 'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7428986', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-snap_schedule-support-subvol-and-group-arguments', 'sha1': '9359c5b40091bccbce966736dc1a3c5bcc76ab0d'}}, 'owner': 'mchangir', 'priority': 25, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7945, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'mgr-snap_schedule-support-subvol-and-group-arguments', 'suite_path': '/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mchangir/ceph.git', 'suite_sha1': '9359c5b40091bccbce966736dc1a3c5bcc76ab0d', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH9fQ0gtr9jZfHA6EZvk7I44etDngq/AlmCWvEkCDjt0JPtwM4iUwHk9OZCL+D1oK6jGFM14qRQXlIlT4mXQIDs=', 'smithi144.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfNhnIlKlzJjnyFNggcHy/KuWjP3yzVYBJoBRaR4nIeuYgvBKYWmC5RzfBA9XWR9KibzaLap60SCOgnwgtX6Bg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_11:33:41', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-16T12:32:19.687 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T12:32:19.696 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T12:32:19.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T12:32:19.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T12:32:19.855 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:19.856 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T12:32:19.900 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T12:32:19.900 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2023-10-16T12:32:19.991 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T12:32:19.992 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:19.992 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 632 Links: 1 2023-10-16T12:32:19.992 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:19.992 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:19.992 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-10-16 12:32:09.311643563 +0000 2023-10-16T12:32:19.993 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-10-16 12:29:33.070114767 +0000 2023-10-16T12:32:19.993 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-10-16 12:29:33.071114758 +0000 2023-10-16T12:32:19.993 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-10-16 12:29:33.070114767 +0000 2023-10-16T12:32:19.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T12:32:20.090 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-10-16T12:32:20.091 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-10-16T12:32:20.091 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000819005 s, 625 kB/s 2023-10-16T12:32:20.094 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T12:32:20.147 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2023-10-16T12:32:20.193 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T12:32:20.194 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:20.194 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 652 Links: 1 2023-10-16T12:32:20.194 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:20.194 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:20.195 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-10-16 12:32:09.311643563 +0000 2023-10-16T12:32:20.195 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-10-16 12:29:34.158104862 +0000 2023-10-16T12:32:20.195 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-10-16 12:29:34.158104862 +0000 2023-10-16T12:32:20.195 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-10-16 12:29:34.158104862 +0000 2023-10-16T12:32:20.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T12:32:20.276 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-10-16T12:32:20.276 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-10-16T12:32:20.277 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000206028 s, 2.5 MB/s 2023-10-16T12:32:20.279 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T12:32:20.332 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2023-10-16T12:32:20.378 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T12:32:20.379 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:20.379 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 671 Links: 1 2023-10-16T12:32:20.379 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:20.379 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:20.379 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-10-16 12:32:09.312643554 +0000 2023-10-16T12:32:20.380 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-10-16 12:29:35.205095329 +0000 2023-10-16T12:32:20.380 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-10-16 12:29:35.205095329 +0000 2023-10-16T12:32:20.380 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-10-16 12:29:35.205095329 +0000 2023-10-16T12:32:20.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T12:32:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-10-16T12:32:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-10-16T12:32:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000233737 s, 2.2 MB/s 2023-10-16T12:32:20.516 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T12:32:20.571 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2023-10-16T12:32:20.617 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T12:32:20.617 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:20.617 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 690 Links: 1 2023-10-16T12:32:20.617 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:20.617 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:20.618 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-10-16 12:32:09.313643545 +0000 2023-10-16T12:32:20.618 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-10-16 12:29:36.264085686 +0000 2023-10-16T12:32:20.618 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-10-16 12:29:36.264085686 +0000 2023-10-16T12:32:20.618 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2023-10-16 12:29:36.264085686 +0000 2023-10-16T12:32:20.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T12:32:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-10-16T12:32:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-10-16T12:32:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000208242 s, 2.5 MB/s 2023-10-16T12:32:20.761 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T12:32:20.806 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:20.807 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:20.807 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T12:32:20.865 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T12:32:20.866 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2023-10-16T12:32:20.918 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T12:32:20.918 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:20.918 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 632 Links: 1 2023-10-16T12:32:20.919 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:20.919 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:20.919 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-16 12:32:00.668847189 +0000 2023-10-16T12:32:20.919 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-16 12:29:07.428410764 +0000 2023-10-16T12:32:20.919 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-16 12:29:07.428410764 +0000 2023-10-16T12:32:20.919 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-16 12:29:07.428410764 +0000 2023-10-16T12:32:20.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T12:32:21.006 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-16T12:32:21.007 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-16T12:32:21.007 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000330346 s, 1.5 MB/s 2023-10-16T12:32:21.010 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T12:32:21.067 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2023-10-16T12:32:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T12:32:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:21.118 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 653 Links: 1 2023-10-16T12:32:21.118 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:21.119 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:21.119 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-16 12:32:00.669847172 +0000 2023-10-16T12:32:21.119 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-16 12:29:08.439393630 +0000 2023-10-16T12:32:21.119 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-16 12:29:08.439393630 +0000 2023-10-16T12:32:21.119 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-16 12:29:08.439393630 +0000 2023-10-16T12:32:21.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T12:32:21.210 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-16T12:32:21.210 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-16T12:32:21.211 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000337145 s, 1.5 MB/s 2023-10-16T12:32:21.213 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T12:32:21.273 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2023-10-16T12:32:21.326 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T12:32:21.326 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:21.326 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 670 Links: 1 2023-10-16T12:32:21.327 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:21.327 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:21.327 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-16 12:32:00.670847155 +0000 2023-10-16T12:32:21.327 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-16 12:29:09.465376242 +0000 2023-10-16T12:32:21.327 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-16 12:29:09.465376242 +0000 2023-10-16T12:32:21.327 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-16 12:29:09.465376242 +0000 2023-10-16T12:32:21.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T12:32:21.419 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-16T12:32:21.420 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-16T12:32:21.420 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000285563 s, 1.8 MB/s 2023-10-16T12:32:21.423 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T12:32:21.479 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2023-10-16T12:32:21.530 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T12:32:21.530 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:32:21.530 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 689 Links: 1 2023-10-16T12:32:21.530 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:32:21.531 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:32:21.531 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-10-16 12:32:00.671847138 +0000 2023-10-16T12:32:21.531 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-10-16 12:29:10.475359125 +0000 2023-10-16T12:32:21.531 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-10-16 12:29:10.475359125 +0000 2023-10-16T12:32:21.531 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2023-10-16 12:29:10.475359125 +0000 2023-10-16T12:32:21.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T12:32:21.614 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-10-16T12:32:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-10-16T12:32:21.615 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000324336 s, 1.6 MB/s 2023-10-16T12:32:21.617 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T12:32:21.673 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:21.674 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T12:32:21.674 INFO:tasks.ceph:Generating config... 2023-10-16T12:32:21.676 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-16T12:32:21.676 INFO:tasks.ceph:[client] debug client = 20 2023-10-16T12:32:21.676 INFO:tasks.ceph:[client] debug ms = 1 2023-10-16T12:32:21.677 INFO:tasks.ceph:[client] pid file = 2023-10-16T12:32:21.677 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-16T12:32:21.677 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-16T12:32:21.678 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-16T12:32:21.678 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-16T12:32:21.678 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-16T12:32:21.679 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-16T12:32:21.679 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-16T12:32:21.679 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-16T12:32:21.680 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-16T12:32:21.680 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-16T12:32:21.680 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-16T12:32:21.680 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-16T12:32:21.681 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-16T12:32:21.681 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-16T12:32:21.681 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-16T12:32:21.681 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-16T12:32:21.682 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-16T12:32:21.682 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-16T12:32:21.682 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-16T12:32:21.682 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-16T12:32:21.683 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-16T12:32:21.683 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-16T12:32:21.683 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-16T12:32:21.683 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-16T12:32:21.684 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-16T12:32:21.684 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-16T12:32:21.684 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-16T12:32:21.684 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-16T12:32:21.685 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-16T12:32:21.685 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-16T12:32:21.685 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-16T12:32:21.685 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-16T12:32:21.686 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-16T12:32:21.686 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-16T12:32:21.686 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-16T12:32:21.686 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-16T12:32:21.687 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-16T12:32:21.687 INFO:tasks.ceph:Setting up mon.a... 2023-10-16T12:32:21.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-16T12:32:21.840 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2023-10-16T12:32:21.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-16T12:32:21.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T12:32:22.089 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60'), ('mon.b', '172.21.15.144'), ('mon.c', '[v2:172.21.15.144:3301,v1:172.21.15.144:6790]')] 2023-10-16T12:32:22.090 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60,172.21.15.144,[v2:172.21.15.144:3301,v1:172.21.15.144:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-16T12:32:22.091 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:22.092 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T12:32:22.151 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --add b 172.21.15.144 --addv c '[v2:172.21.15.144:3301,v1:172.21.15.144:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:22.316 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:22.316 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid d49fcbeb-3f16-4e9a-bc57-c0252aab0f68 2023-10-16T12:32:22.317 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = pacific 2023-10-16T12:32:22.317 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2023-10-16T12:32:22.317 INFO:teuthology.orchestra.run.smithi060.stdout:fsid d49fcbeb-3f16-4e9a-bc57-c0252aab0f68 2023-10-16T12:32:22.317 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2023-10-16T12:32:22.315383+0000 2023-10-16T12:32:22.318 INFO:teuthology.orchestra.run.smithi060.stdout:created 2023-10-16T12:32:22.315383+0000 2023-10-16T12:32:22.318 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 16 (pacific) 2023-10-16T12:32:22.318 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2023-10-16T12:32:22.318 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2023-10-16T12:32:22.318 INFO:teuthology.orchestra.run.smithi060.stdout:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2023-10-16T12:32:22.319 INFO:teuthology.orchestra.run.smithi060.stdout:2: [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] mon.c 2023-10-16T12:32:22.319 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-16T12:32:22.320 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T12:32:22.373 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d49fcbeb-3f16-4e9a-bc57-c0252aab0f68... 2023-10-16T12:32:22.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T12:32:22.381 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T12:32:22.556 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2023-10-16T12:32:22.556 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2023-10-16T12:32:22.557 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T12:32:22.557 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2023-10-16T12:32:22.557 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.557 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2023-10-16T12:32:22.557 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.558 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2023-10-16T12:32:22.558 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.558 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2023-10-16T12:32:22.558 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2023-10-16T12:32:22.559 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.559 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2023-10-16T12:32:22.559 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2023-10-16T12:32:22.559 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.559 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T12:32:22.560 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2023-10-16T12:32:22.560 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.560 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T12:32:22.560 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T12:32:22.561 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.561 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2023-10-16T12:32:22.561 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.561 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2023-10-16T12:32:22.561 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2023-10-16T12:32:22.562 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2023-10-16T12:32:22.562 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2023-10-16T12:32:22.562 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2023-10-16T12:32:22.562 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2023-10-16T12:32:22.563 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2023-10-16T12:32:22.563 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T12:32:22.563 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T12:32:22.563 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2023-10-16T12:32:22.563 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.564 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T12:32:22.564 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.564 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2023-10-16T12:32:22.564 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.565 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2023-10-16T12:32:22.565 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.565 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2023-10-16T12:32:22.565 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2023-10-16T12:32:22.566 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2023-10-16T12:32:22.566 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,172.21.15.144,[v2:172.21.15.144:3301,v1:172.21.15.144:6790]" 2023-10-16T12:32:22.566 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on skipped message = False 2023-10-16T12:32:22.566 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = d49fcbeb-3f16-4e9a-bc57-c0252aab0f68 2023-10-16T12:32:22.566 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.567 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2023-10-16T12:32:22.567 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2023-10-16T12:32:22.567 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.567 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2023-10-16T12:32:22.568 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2023-10-16T12:32:22.568 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2023-10-16T12:32:22.568 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.568 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2023-10-16T12:32:22.569 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2023-10-16T12:32:22.569 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.569 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2023-10-16T12:32:22.569 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2023-10-16T12:32:22.569 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2023-10-16T12:32:22.570 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.570 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2023-10-16T12:32:22.570 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2023-10-16T12:32:22.570 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.571 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2023-10-16T12:32:22.571 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.571 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2023-10-16T12:32:22.571 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.571 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2023-10-16T12:32:22.572 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.572 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2023-10-16T12:32:22.572 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2023-10-16T12:32:22.572 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.573 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2023-10-16T12:32:22.573 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2023-10-16T12:32:22.573 INFO:teuthology.orchestra.run.smithi060.stdout: bdev async discard = True 2023-10-16T12:32:22.573 INFO:teuthology.orchestra.run.smithi060.stdout: bdev enable discard = True 2023-10-16T12:32:22.573 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore allocator = bitmap 2023-10-16T12:32:22.574 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2023-10-16T12:32:22.574 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2023-10-16T12:32:22.574 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1/20 2023-10-16T12:32:22.574 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1/20 2023-10-16T12:32:22.574 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-10-16T12:32:22.575 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2023-10-16T12:32:22.575 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 4/10 2023-10-16T12:32:22.575 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T12:32:22.575 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2023-10-16T12:32:22.576 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2023-10-16T12:32:22.576 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2023-10-16T12:32:22.576 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2023-10-16T12:32:22.576 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2023-10-16T12:32:22.576 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.577 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2023-10-16T12:32:22.577 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-10-16T12:32:22.577 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2023-10-16T12:32:22.577 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2023-10-16T12:32:22.577 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2023-10-16T12:32:22.578 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:32:22.578 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:32:22.578 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2023-10-16T12:32:22.578 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2023-10-16T12:32:22.579 INFO:teuthology.orchestra.run.smithi060.stdout: debug finisher = 20 2023-10-16T12:32:22.579 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.579 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2023-10-16T12:32:22.579 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-10-16T12:32:22.579 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2023-10-16T12:32:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2023-10-16T12:32:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2023-10-16T12:32:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2023-10-16T12:32:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2023-10-16T12:32:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:32:22.581 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2023-10-16T12:32:22.581 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2023-10-16T12:32:22.581 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:32:22.581 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.581 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T12:32:22.582 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T12:32:22.582 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T12:32:22.582 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.582 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T12:32:22.583 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T12:32:22.583 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T12:32:22.583 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.583 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2023-10-16T12:32:22.584 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2023-10-16T12:32:22.584 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.584 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2023-10-16T12:32:22.584 INFO:teuthology.orchestra.run.smithi060.stdout: mon op complaint time = 120 2023-10-16T12:32:22.584 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:32:22.585 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2023-10-16T12:32:22.585 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2023-10-16T12:32:22.585 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2023-10-16T12:32:22.586 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2023-10-16T12:32:22.586 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T12:32:22.586 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T12:32:22.586 INFO:teuthology.orchestra.run.smithi060.stdout: client mount timeout = 600 2023-10-16T12:32:22.586 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2023-10-16T12:32:22.587 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-10-16T12:32:22.587 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = "" 2023-10-16T12:32:22.587 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 900 2023-10-16T12:32:22.588 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 900 2023-10-16T12:32:22.588 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2023-10-16T12:32:22.588 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2023-10-16T12:32:22.588 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2023-10-16T12:32:22.589 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2023-10-16T12:32:22.589 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c] 2023-10-16T12:32:22.589 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2023-10-16T12:32:22.589 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d] 2023-10-16T12:32:22.590 INFO:teuthology.orchestra.run.smithi060.stdout:[mds] 2023-10-16T12:32:22.590 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 20 2023-10-16T12:32:22.590 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds balancer = 20 2023-10-16T12:32:22.590 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2023-10-16T12:32:22.591 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug frag = True 2023-10-16T12:32:22.591 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug scatterstat = True 2023-10-16T12:32:22.591 INFO:teuthology.orchestra.run.smithi060.stdout: mds op complaint time = 180 2023-10-16T12:32:22.591 INFO:teuthology.orchestra.run.smithi060.stdout: mds verify scatter = True 2023-10-16T12:32:22.592 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2023-10-16T12:32:22.592 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 900 2023-10-16T12:32:22.592 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 900 2023-10-16T12:32:22.593 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2023-10-16T12:32:22.593 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2023-10-16T12:32:22.593 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T12:32:22.594 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2023-10-16T12:32:22.594 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.594 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2023-10-16T12:32:22.595 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.595 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2023-10-16T12:32:22.595 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.595 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2023-10-16T12:32:22.596 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2023-10-16T12:32:22.596 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.596 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2023-10-16T12:32:22.596 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2023-10-16T12:32:22.596 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.597 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T12:32:22.597 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2023-10-16T12:32:22.597 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.597 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T12:32:22.597 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T12:32:22.598 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.598 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2023-10-16T12:32:22.598 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.599 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2023-10-16T12:32:22.599 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2023-10-16T12:32:22.599 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2023-10-16T12:32:22.599 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2023-10-16T12:32:22.600 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2023-10-16T12:32:22.600 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2023-10-16T12:32:22.601 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2023-10-16T12:32:22.601 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T12:32:22.601 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T12:32:22.601 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2023-10-16T12:32:22.601 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.602 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T12:32:22.602 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.602 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2023-10-16T12:32:22.602 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.603 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2023-10-16T12:32:22.603 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.603 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2023-10-16T12:32:22.603 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2023-10-16T12:32:22.603 INFO:teuthology.orchestra.run.smithi144.stdout: mon health detail to clog = false 2023-10-16T12:32:22.604 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = "172.21.15.60,172.21.15.144,[v2:172.21.15.144:3301,v1:172.21.15.144:6790]" 2023-10-16T12:32:22.604 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on skipped message = False 2023-10-16T12:32:22.604 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = d49fcbeb-3f16-4e9a-bc57-c0252aab0f68 2023-10-16T12:32:22.604 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.604 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2023-10-16T12:32:22.605 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2023-10-16T12:32:22.605 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.605 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2023-10-16T12:32:22.605 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2023-10-16T12:32:22.606 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock profile = high_recovery_ops 2023-10-16T12:32:22.606 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.606 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2023-10-16T12:32:22.606 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2023-10-16T12:32:22.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.607 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2023-10-16T12:32:22.607 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2023-10-16T12:32:22.607 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2023-10-16T12:32:22.607 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.608 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2023-10-16T12:32:22.608 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2023-10-16T12:32:22.608 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.608 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2023-10-16T12:32:22.609 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.609 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2023-10-16T12:32:22.609 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.609 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2023-10-16T12:32:22.609 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.610 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2023-10-16T12:32:22.610 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2023-10-16T12:32:22.610 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.610 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2023-10-16T12:32:22.611 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2023-10-16T12:32:22.611 INFO:teuthology.orchestra.run.smithi144.stdout: bdev async discard = True 2023-10-16T12:32:22.611 INFO:teuthology.orchestra.run.smithi144.stdout: bdev enable discard = True 2023-10-16T12:32:22.611 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore allocator = bitmap 2023-10-16T12:32:22.611 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore block size = 96636764160 2023-10-16T12:32:22.612 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore fsck on mount = True 2023-10-16T12:32:22.612 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluefs = 1/20 2023-10-16T12:32:22.612 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluestore = 1/20 2023-10-16T12:32:22.612 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-16T12:32:22.612 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 20 2023-10-16T12:32:22.613 INFO:teuthology.orchestra.run.smithi144.stdout: debug rocksdb = 4/10 2023-10-16T12:32:22.613 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T12:32:22.613 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd full ratio = 0.9 2023-10-16T12:32:22.613 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd nearfull ratio = 0.8 2023-10-16T12:32:22.614 INFO:teuthology.orchestra.run.smithi144.stdout: osd failsafe full ratio = 0.95 2023-10-16T12:32:22.614 INFO:teuthology.orchestra.run.smithi144.stdout: osd objectstore = bluestore 2023-10-16T12:32:22.614 INFO:teuthology.orchestra.run.smithi144.stdout: osd op complaint time = 180 2023-10-16T12:32:22.614 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.614 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2023-10-16T12:32:22.615 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-16T12:32:22.615 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2023-10-16T12:32:22.615 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2023-10-16T12:32:22.615 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2023-10-16T12:32:22.615 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:32:22.616 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:32:22.616 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2023-10-16T12:32:22.616 INFO:teuthology.orchestra.run.smithi144.stdout: debug client = 20 2023-10-16T12:32:22.616 INFO:teuthology.orchestra.run.smithi144.stdout: debug finisher = 20 2023-10-16T12:32:22.616 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.617 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2023-10-16T12:32:22.617 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-16T12:32:22.617 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2023-10-16T12:32:22.617 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2023-10-16T12:32:22.618 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2023-10-16T12:32:22.618 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2023-10-16T12:32:22.618 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2023-10-16T12:32:22.618 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:32:22.618 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2023-10-16T12:32:22.619 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2023-10-16T12:32:22.619 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:32:22.619 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.619 INFO:teuthology.orchestra.run.smithi144.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T12:32:22.620 INFO:teuthology.orchestra.run.smithi144.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T12:32:22.620 INFO:teuthology.orchestra.run.smithi144.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T12:32:22.620 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.620 INFO:teuthology.orchestra.run.smithi144.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T12:32:22.620 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T12:32:22.621 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T12:32:22.621 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.621 INFO:teuthology.orchestra.run.smithi144.stdout: # 1m isn't quite enough 2023-10-16T12:32:22.621 INFO:teuthology.orchestra.run.smithi144.stdout: mon_down_mkfs_grace = 2m 2023-10-16T12:32:22.622 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.622 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_filestore_osds = false 2023-10-16T12:32:22.622 INFO:teuthology.orchestra.run.smithi144.stdout: mon op complaint time = 120 2023-10-16T12:32:22.622 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-10-16T12:32:22.623 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2023-10-16T12:32:22.623 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2023-10-16T12:32:22.623 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2023-10-16T12:32:22.623 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2023-10-16T12:32:22.624 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T12:32:22.624 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T12:32:22.624 INFO:teuthology.orchestra.run.smithi144.stdout: client mount timeout = 600 2023-10-16T12:32:22.625 INFO:teuthology.orchestra.run.smithi144.stdout: debug client = 20 2023-10-16T12:32:22.625 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-16T12:32:22.625 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = "" 2023-10-16T12:32:22.625 INFO:teuthology.orchestra.run.smithi144.stdout: rados mon op timeout = 900 2023-10-16T12:32:22.625 INFO:teuthology.orchestra.run.smithi144.stdout: rados osd op timeout = 900 2023-10-16T12:32:22.626 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2023-10-16T12:32:22.626 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.b] 2023-10-16T12:32:22.626 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.c] 2023-10-16T12:32:22.626 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.a] 2023-10-16T12:32:22.627 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.c] 2023-10-16T12:32:22.627 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.b] 2023-10-16T12:32:22.627 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.d] 2023-10-16T12:32:22.627 INFO:teuthology.orchestra.run.smithi144.stdout:[mds] 2023-10-16T12:32:22.627 INFO:teuthology.orchestra.run.smithi144.stdout: debug mds = 20 2023-10-16T12:32:22.628 INFO:teuthology.orchestra.run.smithi144.stdout: debug mds balancer = 20 2023-10-16T12:32:22.628 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2023-10-16T12:32:22.628 INFO:teuthology.orchestra.run.smithi144.stdout: mds debug frag = True 2023-10-16T12:32:22.629 INFO:teuthology.orchestra.run.smithi144.stdout: mds debug scatterstat = True 2023-10-16T12:32:22.629 INFO:teuthology.orchestra.run.smithi144.stdout: mds op complaint time = 180 2023-10-16T12:32:22.629 INFO:teuthology.orchestra.run.smithi144.stdout: mds verify scatter = True 2023-10-16T12:32:22.629 INFO:teuthology.orchestra.run.smithi144.stdout: osd op complaint time = 180 2023-10-16T12:32:22.630 INFO:teuthology.orchestra.run.smithi144.stdout: rados mon op timeout = 900 2023-10-16T12:32:22.630 INFO:teuthology.orchestra.run.smithi144.stdout: rados osd op timeout = 900 2023-10-16T12:32:22.632 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-16T12:32:22.633 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-16T12:32:22.794 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-16T12:32:22.795 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:22.795 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-16T12:32:22.850 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:22.851 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-16T12:32:22.901 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:32:22.902 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:22.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T12:32:22.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T12:32:23.047 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:23.048 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:23.146 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:32:23.146 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:23.147 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T12:32:23.148 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T12:32:23.304 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:23.304 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:23.404 INFO:tasks.ceph:Setting up mon nodes... 2023-10-16T12:32:23.405 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-16T12:32:23.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T12:32:23.608 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T12:32:23.619 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T12:32:23.814 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T12:32:23.825 INFO:tasks.ceph:Setting up mds nodes... 2023-10-16T12:32:23.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T12:32:24.026 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T12:32:24.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-16T12:32:24.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T12:32:24.328 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T12:32:24.337 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-16T12:32:24.441 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T12:32:24.632 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T12:32:24.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-16T12:32:24.733 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T12:32:24.918 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T12:32:24.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-16T12:32:25.016 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-16T12:32:25.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-16T12:32:25.155 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-16T12:32:25.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-16T12:32:25.355 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-16T12:32:25.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-16T12:32:25.559 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-16T12:32:25.611 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-16T12:32:25.737 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-16T12:32:25.798 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-16T12:32:25.799 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T12:32:25.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-16T12:32:25.913 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:25.914 INFO:tasks.ceph:role: osd.0 2023-10-16T12:32:25.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:32:25.914 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T12:32:26.018 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:26.018 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:26.018 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:26.019 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:26.019 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:26.019 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:26.019 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:26.020 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:26.020 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:26.020 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:26.052 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2023-10-16T12:32:26.059 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2023-10-16T12:32:26.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-16T12:32:26.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-16T12:32:26.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-16T12:32:26.367 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:26.367 INFO:tasks.ceph:role: osd.1 2023-10-16T12:32:26.368 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:32:26.368 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T12:32:26.467 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:26.467 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:26.468 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:26.468 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:26.468 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:26.468 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:26.468 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:26.469 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:26.469 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:26.469 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:26.502 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2023-10-16T12:32:26.506 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2023-10-16T12:32:26.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-16T12:32:26.617 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-16T12:32:26.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-16T12:32:26.796 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:26.797 INFO:tasks.ceph:role: osd.2 2023-10-16T12:32:26.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:32:26.797 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T12:32:26.892 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:26.892 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:26.893 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:26.893 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:26.893 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:26.925 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2023-10-16T12:32:26.930 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2023-10-16T12:32:26.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-16T12:32:27.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-16T12:32:27.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-16T12:32:27.245 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:27.245 INFO:tasks.ceph:role: osd.3 2023-10-16T12:32:27.246 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:32:27.246 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T12:32:27.343 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:27.344 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:27.345 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:27.345 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:27.345 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:27.376 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2023-10-16T12:32:27.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2023-10-16T12:32:27.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-16T12:32:27.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-16T12:32:27.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:27.836+0000 7fa3f5c14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-16T12:32:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:27.836+0000 7fa3f5c14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-16T12:32:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:27.836+0000 7fa3f5c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T12:32:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:27.836+0000 7fa3f5c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T12:32:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:27.836+0000 7fa3f5c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T12:32:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:27.840+0000 7fa3f5c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-16T12:32:31.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-16T12:32:31.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:31.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:31.273+0000 7f3f76016500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-16T12:32:31.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:31.274+0000 7f3f76016500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-16T12:32:31.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:31.274+0000 7f3f76016500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T12:32:31.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:31.274+0000 7f3f76016500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T12:32:31.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:31.274+0000 7f3f76016500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T12:32:31.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:31.276+0000 7f3f76016500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-16T12:32:34.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-16T12:32:34.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:34.449+0000 7fad3b317500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-16T12:32:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:34.449+0000 7fad3b317500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-16T12:32:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:34.450+0000 7fad3b317500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T12:32:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:34.450+0000 7fad3b317500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T12:32:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:34.450+0000 7fad3b317500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T12:32:34.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:34.452+0000 7fad3b317500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-16T12:32:37.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-16T12:32:37.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:37.892+0000 7fac8b425500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-16T12:32:37.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:37.893+0000 7fac8b425500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-16T12:32:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:37.893+0000 7fac8b425500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T12:32:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:37.893+0000 7fac8b425500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T12:32:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:37.893+0000 7fac8b425500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T12:32:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:32:37.895+0000 7fac8b425500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-16T12:32:41.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-16T12:32:41.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-16T12:32:41.272 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:41.272 INFO:tasks.ceph:role: osd.4 2023-10-16T12:32:41.272 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:32:41.272 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T12:32:41.376 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:41.376 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:41.377 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:41.377 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:41.377 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:41.377 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:41.377 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:41.377 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:41.378 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:41.378 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:41.880 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-16T12:32:41.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-16T12:32:41.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-16T12:32:42.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-16T12:32:42.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-16T12:32:42.279 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:42.279 INFO:tasks.ceph:role: osd.5 2023-10-16T12:32:42.279 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:32:42.279 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T12:32:42.378 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:42.378 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:42.379 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:42.379 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:42.379 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:42.379 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:42.379 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:42.379 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:42.380 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:42.380 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:42.857 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-16T12:32:42.864 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-16T12:32:42.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-16T12:32:42.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-16T12:32:43.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-16T12:32:43.166 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:43.167 INFO:tasks.ceph:role: osd.6 2023-10-16T12:32:43.167 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:32:43.167 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T12:32:43.260 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:43.260 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:43.261 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:43.261 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:43.261 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:43.261 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:43.261 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:43.262 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:43.262 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:43.262 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:43.783 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-16T12:32:43.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-16T12:32:43.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-16T12:32:43.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-16T12:32:44.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-16T12:32:44.153 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:32:44.154 INFO:tasks.ceph:role: osd.7 2023-10-16T12:32:44.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:32:44.154 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T12:32:44.256 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:32:44.257 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:32:44.257 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:32:44.257 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T12:32:44.258 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:32:44.258 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:32:44.258 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:32:44.258 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T12:32:44.258 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:32:44.259 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:32:44.714 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2023-10-16T12:32:44.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2023-10-16T12:32:44.720 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-16T12:32:44.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-16T12:32:44.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:45.109 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:45.107+0000 7fb9ce20c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-16T12:32:45.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:45.108+0000 7fb9ce20c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-16T12:32:45.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:45.108+0000 7fb9ce20c500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T12:32:45.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:45.108+0000 7fb9ce20c500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T12:32:45.111 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:45.108+0000 7fb9ce20c500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T12:32:45.112 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:45.111+0000 7fb9ce20c500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-16T12:32:48.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-16T12:32:48.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:48.543 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:48.541+0000 7f9e6380d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-16T12:32:48.544 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:48.542+0000 7f9e6380d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-16T12:32:48.544 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:48.542+0000 7f9e6380d500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T12:32:48.544 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:48.542+0000 7f9e6380d500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T12:32:48.544 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:48.542+0000 7f9e6380d500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T12:32:48.546 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:48.544+0000 7f9e6380d500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-16T12:32:49.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:49.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:51.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-16T12:32:51.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:51.707 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:51.704+0000 7f82a30c3500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-16T12:32:51.707 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:51.704+0000 7f82a30c3500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-16T12:32:51.708 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:51.705+0000 7f82a30c3500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T12:32:51.708 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:51.705+0000 7f82a30c3500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T12:32:51.708 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:51.705+0000 7f82a30c3500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T12:32:51.709 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:51.707+0000 7f82a30c3500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-16T12:32:54.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-16T12:32:54.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:32:55.117 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:55.114+0000 7f0688392500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-16T12:32:55.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:55.115+0000 7f0688392500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-16T12:32:55.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:55.115+0000 7f0688392500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T12:32:55.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:55.115+0000 7f0688392500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T12:32:55.119 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:55.115+0000 7f0688392500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T12:32:55.120 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:32:55.117+0000 7f0688392500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-16T12:32:58.290 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-16T12:32:58.391 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-16T12:32:58.392 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:58.392 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-16T12:32:58.504 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:58.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-16T12:32:58.612 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:58.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-16T12:32:58.719 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:58.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-16T12:32:58.821 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:58.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-16T12:32:58.926 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:58.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-16T12:32:59.034 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:59.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-16T12:32:59.137 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-16T12:32:59.241 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-16T12:32:59.336 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-16T12:32:59.430 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.430 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-16T12:32:59.518 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-16T12:32:59.608 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.609 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-16T12:32:59.749 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-16T12:32:59.841 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:59.842 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-16T12:32:59.905 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:32:59.905 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-16T12:32:59.963 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:32:59.964 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-16T12:33:00.023 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:00.024 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-16T12:33:00.077 INFO:tasks.ceph:Adding keys to all mons... 2023-10-16T12:33:00.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T12:33:00.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T12:33:00.179 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.y] 2023-10-16T12:33:00.180 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA== 2023-10-16T12:33:00.180 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.a] 2023-10-16T12:33:00.180 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A== 2023-10-16T12:33:00.180 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.c] 2023-10-16T12:33:00.181 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA== 2023-10-16T12:33:00.181 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2023-10-16T12:33:00.181 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ== 2023-10-16T12:33:00.181 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2023-10-16T12:33:00.182 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw== 2023-10-16T12:33:00.182 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2023-10-16T12:33:00.182 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow== 2023-10-16T12:33:00.183 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.3] 2023-10-16T12:33:00.183 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ== 2023-10-16T12:33:00.183 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2023-10-16T12:33:00.183 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA== 2023-10-16T12:33:00.184 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.b] 2023-10-16T12:33:00.184 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA== 2023-10-16T12:33:00.184 INFO:teuthology.orchestra.run.smithi144.stdout:[mds.d] 2023-10-16T12:33:00.184 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ== 2023-10-16T12:33:00.184 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.4] 2023-10-16T12:33:00.185 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig== 2023-10-16T12:33:00.185 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.5] 2023-10-16T12:33:00.185 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ== 2023-10-16T12:33:00.185 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.6] 2023-10-16T12:33:00.186 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng== 2023-10-16T12:33:00.186 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.7] 2023-10-16T12:33:00.187 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA== 2023-10-16T12:33:00.187 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2023-10-16T12:33:00.187 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:33:00.187 INFO:teuthology.orchestra.run.smithi144.stdout:[client.2] 2023-10-16T12:33:00.187 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg== 2023-10-16T12:33:00.188 INFO:teuthology.orchestra.run.smithi144.stdout:[client.1] 2023-10-16T12:33:00.188 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ== 2023-10-16T12:33:00.188 INFO:teuthology.orchestra.run.smithi144.stdout:[client.3] 2023-10-16T12:33:00.188 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA== 2023-10-16T12:33:00.190 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2023-10-16T12:33:00.191 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA== 2023-10-16T12:33:00.191 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2023-10-16T12:33:00.191 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A== 2023-10-16T12:33:00.191 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c] 2023-10-16T12:33:00.192 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA== 2023-10-16T12:33:00.192 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2023-10-16T12:33:00.192 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ== 2023-10-16T12:33:00.192 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2023-10-16T12:33:00.192 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw== 2023-10-16T12:33:00.193 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2023-10-16T12:33:00.193 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow== 2023-10-16T12:33:00.193 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2023-10-16T12:33:00.193 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ== 2023-10-16T12:33:00.193 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2023-10-16T12:33:00.194 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA== 2023-10-16T12:33:00.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2023-10-16T12:33:00.194 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA== 2023-10-16T12:33:00.195 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d] 2023-10-16T12:33:00.195 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ== 2023-10-16T12:33:00.195 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2023-10-16T12:33:00.195 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig== 2023-10-16T12:33:00.195 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2023-10-16T12:33:00.196 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ== 2023-10-16T12:33:00.196 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2023-10-16T12:33:00.196 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng== 2023-10-16T12:33:00.196 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2023-10-16T12:33:00.197 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA== 2023-10-16T12:33:00.197 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:33:00.197 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:33:00.198 INFO:teuthology.orchestra.run.smithi060.stdout:[client.2] 2023-10-16T12:33:00.198 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg== 2023-10-16T12:33:00.198 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2023-10-16T12:33:00.198 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ== 2023-10-16T12:33:00.199 INFO:teuthology.orchestra.run.smithi060.stdout:[client.3] 2023-10-16T12:33:00.199 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA== 2023-10-16T12:33:00.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:33:00.209 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:33:00.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:00.368 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:00.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:00.526 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:00.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:00.686 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:00.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:00.842 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:00.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:00.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:01.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:01.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:01.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:33:01.306 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:33:01.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:01.459 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:01.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:01.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:33:01.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:01.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:01.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:01.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:02.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:02.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:02.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:02.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:33:02.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.561 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.718 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:02.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:33:03.035 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-16T12:33:03.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-16T12:33:03.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T12:33:03.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-16T12:33:03.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-16T12:33:03.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T12:33:03.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-16T12:33:03.751 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-16T12:33:03.851 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T12:33:04.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-16T12:33:04.100 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:33:04.109 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:33:04.164 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-16T12:33:04.165 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-16T12:33:04.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-16T12:33:04.172 INFO:tasks.ceph.mon.a:Started 2023-10-16T12:33:04.172 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-16T12:33:04.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-16T12:33:04.181 INFO:tasks.ceph.mon.b:Started 2023-10-16T12:33:04.182 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-16T12:33:04.182 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-16T12:33:04.191 INFO:tasks.ceph.mon.c:Started 2023-10-16T12:33:04.191 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-16T12:33:04.191 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T12:33:04.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T12:33:04.200 INFO:tasks.ceph.mgr.y:Started 2023-10-16T12:33:04.201 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T12:33:04.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T12:33:04.206 INFO:tasks.ceph.mgr.x:Started 2023-10-16T12:33:04.206 INFO:tasks.ceph:Setting crush tunables to default 2023-10-16T12:33:04.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2023-10-16T12:33:04.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.390+0000 7f65009cb640 1 Processor -- start 2023-10-16T12:33:04.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.390+0000 7f65009cb640 1 -- start start 2023-10-16T12:33:04.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.391+0000 7f65009cb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.391+0000 7f65009cb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.391+0000 7f65009cb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0d0970 con 0x7f64fc0ce960 2023-10-16T12:33:04.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.391+0000 7f65009cb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64fc0d0ea0 con 0x7f64fc0b2080 2023-10-16T12:33:04.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f65009cb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0d0bc0 con 0x7f64fc0b1ca0 2023-10-16T12:33:04.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64fb59e640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f64fc0ce960 0x7f64fc0b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:57904/0 (socket says 172.21.15.60:57904) 2023-10-16T12:33:04.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64fb59e640 1 -- 172.21.15.60:0/587712161 learned_addr learned my addr 172.21.15.60:0/587712161 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:04.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64f259c640 1 -- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 msgr2=0x7f64fc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T12:33:04.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64f259c640 1 -- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 msgr2=0x7f64fc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T12:33:04.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.392+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T12:33:04.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.393+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 msgr2=0x7f64fc0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T12:33:04.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.393+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 msgr2=0x7f64fc0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T12:33:04.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.393+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T12:33:04.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.393+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:33:04.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.393+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:33:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.594+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.594+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.594+0000 7f64f259c640 1 -- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 msgr2=0x7f64fc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T12:33:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.594+0000 7f64f259c640 1 -- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 msgr2=0x7f64fc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T12:33:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.594+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T12:33:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.595+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 msgr2=0x7f64fc0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T12:33:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.595+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 msgr2=0x7f64fc0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T12:33:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.595+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T12:33:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.595+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:33:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.595+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:33:04.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.629+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248443820 0 0) 0x7f64fc0d0970 con 0x7f64fc0ce960 2023-10-16T12:33:04.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.629+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8003520 con 0x7f64fc0ce960 2023-10-16T12:33:04.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412422433 0 0) 0x7f64e8003520 con 0x7f64fc0ce960 2023-10-16T12:33:04.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 msgr2=0x7f64fc0d36b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:33:04.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64fc0d36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 msgr2=0x7f64fc0ce420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:33:04.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 --2- 172.21.15.60:0/587712161 >> v2:172.21.15.60:3300/0 conn(0x7f64fc0b2080 0x7f64fc0ce420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0cd940 con 0x7f64fc0ce960 2023-10-16T12:33:04.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.635+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f64f40019f0 con 0x7f64fc0ce960 2023-10-16T12:33:04.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 >> v1:172.21.15.144:6789/0 conn(0x7f64fc0ce960 legacy=0x7f64fc0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:04.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e8003970 0x7f64e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e8004d00 0x7f64e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0d0ea0 con 0x7f64e8003970 2023-10-16T12:33:04.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0d0bc0 con 0x7f64e8004d00 2023-10-16T12:33:04.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e800bab0 con 0x7f64fc0b1ca0 2023-10-16T12:33:04.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64f259c640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e8003970 0x7f64e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.636+0000 7f64fb59e640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e8004d00 0x7f64e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.637+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 msgr2=0x7f64e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:04.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.637+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0b1ca0 0x7f64e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.637+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e8004d00 msgr2=0x7f64e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:04.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.637+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64e8004d00 0x7f64e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.637+0000 7f64fad9d640 1 -- 172.21.15.60:0/587712161 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0cd940 con 0x7f64e8003970 2023-10-16T12:33:04.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.639+0000 7f64fad9d640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e8003970 0x7f64e8003d90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00da90 tx=0x7f64e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a3fd0fe1e8baa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:04.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.642+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02c810 con 0x7f64e8003970 2023-10-16T12:33:04.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.642+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec030070 con 0x7f64e8003970 2023-10-16T12:33:04.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.642+0000 7f64fa59c640 1 -- 172.21.15.60:0/587712161 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02cbb0 con 0x7f64e8003970 2023-10-16T12:33:04.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.643+0000 7f65009cb640 1 -- 172.21.15.60:0/587712161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e8003970 msgr2=0x7f64e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:04.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.643+0000 7f65009cb640 1 --2- 172.21.15.60:0/587712161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e8003970 0x7f64e8003d90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00da90 tx=0x7f64e80057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.643+0000 7f64fb59e640 1 -- 172.21.15.60:0/587712161 reap_dead start 2023-10-16T12:33:04.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.643+0000 7f65009cb640 1 -- 172.21.15.60:0/587712161 shutdown_connections 2023-10-16T12:33:04.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.643+0000 7f65009cb640 1 -- 172.21.15.60:0/587712161 >> 172.21.15.60:0/587712161 conn(0x7f64fc0d2b90 msgr2=0x7f64fc0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:04.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.644+0000 7f65009cb640 1 -- 172.21.15.60:0/587712161 shutdown_connections 2023-10-16T12:33:04.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.644+0000 7f65009cb640 1 -- 172.21.15.60:0/587712161 wait complete. 2023-10-16T12:33:04.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.646+0000 7f65009cb640 1 Processor -- start 2023-10-16T12:33:04.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.646+0000 7f65009cb640 1 -- start start 2023-10-16T12:33:04.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.647+0000 7f65009cb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0c5750 0x7f64fc0c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.647+0000 7f65009cb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0ca1a0 0x7f64fc0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.647+0000 7f65009cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc155700 0x7f64fc0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:04.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.647+0000 7f65009cb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0c7b50 con 0x7f64fc155700 2023-10-16T12:33:04.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.647+0000 7f65009cb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0cc940 con 0x7f64fc0c5750 2023-10-16T12:33:04.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.647+0000 7f65009cb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0ca620 con 0x7f64fc0ca1a0 2023-10-16T12:33:04.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fb59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc155700 0x7f64fc0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0c5750 0x7f64fc0c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64f259c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0ca1a0 0x7f64fc0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:04.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0c5750 0x7f64fc0c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:50596/0 (socket says 172.21.15.60:50596) 2023-10-16T12:33:04.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64f259c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0ca1a0 0x7f64fc0cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:57252/0 (socket says 172.21.15.60:57252) 2023-10-16T12:33:04.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fad9d640 1 -- 172.21.15.60:0/3748797435 learned_addr learned my addr 172.21.15.60:0/3748797435 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:04.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fb59e640 1 -- 172.21.15.60:0/3748797435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0ca1a0 msgr2=0x7f64fc0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fb59e640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0ca1a0 0x7f64fc0cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fb59e640 1 -- 172.21.15.60:0/3748797435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0c5750 msgr2=0x7f64fc0c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fb59e640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0c5750 0x7f64fc0c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.648+0000 7f64fb59e640 1 -- 172.21.15.60:0/3748797435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0c7490 con 0x7f64fc155700 2023-10-16T12:33:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.649+0000 7f64fb59e640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc155700 0x7f64fc0cc400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0c64c0 tx=0x7f64f400a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.649+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4013600 con 0x7f64fc155700 2023-10-16T12:33:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.649+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f4013760 con 0x7f64fc155700 2023-10-16T12:33:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.649+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc0c7770 con 0x7f64fc155700 2023-10-16T12:33:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.649+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4013b10 con 0x7f64fc155700 2023-10-16T12:33:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.649+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc0ca900 con 0x7f64fc155700 2023-10-16T12:33:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.650+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f64f401e460 con 0x7f64fc155700 2023-10-16T12:33:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.650+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f64f401d3c0 con 0x7f64fc155700 2023-10-16T12:33:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.651+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc0c7c90 con 0x7f64fc155700 2023-10-16T12:33:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.655+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f64fc0c7c90 con 0x7f64fc155700 2023-10-16T12:33:04.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.730+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f64fc04e9b0 con 0x7f64fc155700 2023-10-16T12:33:04.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.733+0000 7f64f3fff640 1 -- 172.21.15.60:0/3748797435 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f64f401e8a0 con 0x7f64fc155700 2023-10-16T12:33:04.735 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2023-10-16T12:33:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.734+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc155700 msgr2=0x7f64fc0cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.734+0000 7f65009cb640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc155700 0x7f64fc0cc400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0c64c0 tx=0x7f64f400a730 comp rx=0 tx=0).stop 2023-10-16T12:33:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 shutdown_connections 2023-10-16T12:33:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc0ca1a0 0x7f64fc0cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc155700 0x7f64fc0cc400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 --2- 172.21.15.60:0/3748797435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0c5750 0x7f64fc0c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 >> 172.21.15.60:0/3748797435 conn(0x7f64fc0d2b90 msgr2=0x7f64fc1603b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 shutdown_connections 2023-10-16T12:33:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:04.735+0000 7f65009cb640 1 -- 172.21.15.60:0/3748797435 wait complete. 2023-10-16T12:33:04.755 INFO:tasks.ceph:check_enable_crimson: False 2023-10-16T12:33:04.755 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-16T12:33:04.756 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:33:04.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-16T12:33:04.837 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:33:04.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-16T12:33:04.962 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:33:04.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-16T12:33:05.099 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:33:05.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-16T12:33:05.187 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:05.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-16T12:33:05.255 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:05.252+0000 7ffb7eeb6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:33:05.277 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:05.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-16T12:33:05.322 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:05.321+0000 7f487782c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:33:05.323 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:05.320+0000 7ffb7eeb6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:33:05.364 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:05.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-16T12:33:05.390 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:05.389+0000 7f487782c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:33:05.494 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:05.495 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-16T12:33:05.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 95d94d30-9650-41d5-b171-abf1ef7481ee 0 2023-10-16T12:33:05.817 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.814+0000 7f4b26527640 1 Processor -- start 2023-10-16T12:33:05.817 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.815+0000 7f4b26527640 1 -- start start 2023-10-16T12:33:05.820 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b26527640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b2015cd70 0x7f4b200d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:05.820 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b26527640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4b200cded0 0x7f4b200b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:05.820 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b26527640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2015d150 con 0x7f4b200b8480 2023-10-16T12:33:05.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b26527640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b200d1180 con 0x7f4b200cded0 2023-10-16T12:33:05.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b26527640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b200d1460 con 0x7f4b2015cd70 2023-10-16T12:33:05.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b25525640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b2015cd70 0x7f4b200d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:05.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b25525640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b2015cd70 0x7f4b200d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43528/0 (socket says 172.21.15.144:43528) 2023-10-16T12:33:05.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.816+0000 7f4b25525640 1 -- 172.21.15.144:0/2199581582 learned_addr learned my addr 172.21.15.144:0/2199581582 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:05.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.817+0000 7f4b25d26640 1 --2- 172.21.15.144:0/2199581582 >> v2:172.21.15.60:3300/0 conn(0x7f4b200cded0 0x7f4b200b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:05.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.817+0000 7f4b25525640 1 -- 172.21.15.144:0/2199581582 >> v2:172.21.15.60:3300/0 conn(0x7f4b200cded0 msgr2=0x7f4b200b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:05.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.817+0000 7f4b25525640 1 --2- 172.21.15.144:0/2199581582 >> v2:172.21.15.60:3300/0 conn(0x7f4b200cded0 0x7f4b200b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.817+0000 7f4b25525640 1 -- 172.21.15.144:0/2199581582 >> v1:172.21.15.60:6789/0 conn(0x7f4b200b8480 legacy=0x7f4b200cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:05.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.817+0000 7f4b25525640 1 -- 172.21.15.144:0/2199581582 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200d1740 con 0x7f4b2015cd70 2023-10-16T12:33:05.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.817+0000 7f4b25d26640 1 --2- 172.21.15.144:0/2199581582 >> v2:172.21.15.60:3300/0 conn(0x7f4b200cded0 0x7f4b200b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:05.827 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.818+0000 7f4b25525640 1 --2- 172.21.15.144:0/2199581582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b2015cd70 0x7f4b200d2220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4b200b56a0 tx=0x7f4b1c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83fec0b38441e153 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:05.828 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.818+0000 7f4b17fff640 1 -- 172.21.15.144:0/2199581582 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c031870 con 0x7f4b2015cd70 2023-10-16T12:33:05.828 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.818+0000 7f4b17fff640 1 -- 172.21.15.144:0/2199581582 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c0319d0 con 0x7f4b2015cd70 2023-10-16T12:33:05.828 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.818+0000 7f4b17fff640 1 -- 172.21.15.144:0/2199581582 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c031d60 con 0x7f4b2015cd70 2023-10-16T12:33:05.828 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 -- 172.21.15.144:0/2199581582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b2015cd70 msgr2=0x7f4b200d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:05.828 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 --2- 172.21.15.144:0/2199581582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b2015cd70 0x7f4b200d2220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4b200b56a0 tx=0x7f4b1c02b910 comp rx=0 tx=0).stop 2023-10-16T12:33:05.829 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 -- 172.21.15.144:0/2199581582 shutdown_connections 2023-10-16T12:33:05.829 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 --2- 172.21.15.144:0/2199581582 >> v2:172.21.15.60:3300/0 conn(0x7f4b200cded0 0x7f4b200b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.829 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 --2- 172.21.15.144:0/2199581582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b2015cd70 0x7f4b200d2220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.829 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 -- 172.21.15.144:0/2199581582 >> 172.21.15.144:0/2199581582 conn(0x7f4b200cf570 msgr2=0x7f4b200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:05.830 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 -- 172.21.15.144:0/2199581582 shutdown_connections 2023-10-16T12:33:05.830 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.819+0000 7f4b26527640 1 -- 172.21.15.144:0/2199581582 wait complete. 2023-10-16T12:33:05.830 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.821+0000 7f4b26527640 1 Processor -- start 2023-10-16T12:33:05.830 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.821+0000 7f4b26527640 1 -- start start 2023-10-16T12:33:05.831 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b26527640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 0x7f4b2014fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:05.831 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b26527640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b200cded0 0x7f4b201f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:05.831 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b26527640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b2015cd70 0x7f4b200a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:05.831 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b26527640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2015e740 con 0x7f4b200cded0 2023-10-16T12:33:05.832 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b26527640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20160240 con 0x7f4b200b8480 2023-10-16T12:33:05.832 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b26527640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200b5ea0 con 0x7f4b2015cd70 2023-10-16T12:33:05.832 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 0x7f4b2014fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:05.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 0x7f4b2014fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43542/0 (socket says 172.21.15.144:43542) 2023-10-16T12:33:05.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 -- 172.21.15.144:0/822263733 learned_addr learned my addr 172.21.15.144:0/822263733 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:05.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25d26640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b2015cd70 0x7f4b200a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b24d24640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b200cded0 0x7f4b201f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 -- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b2015cd70 msgr2=0x7f4b200a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b2015cd70 0x7f4b200a9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 -- 172.21.15.144:0/822263733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b200cded0 msgr2=0x7f4b201f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b200cded0 0x7f4b201f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.822+0000 7f4b25525640 1 -- 172.21.15.144:0/822263733 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c0094d0 con 0x7f4b200b8480 2023-10-16T12:33:05.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b25525640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 0x7f4b2014fef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c000c00 tx=0x7f4b1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:05.836 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c031840 con 0x7f4b200b8480 2023-10-16T12:33:05.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c0443f0 con 0x7f4b200b8480 2023-10-16T12:33:05.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0447a0 con 0x7f4b200b8480 2023-10-16T12:33:05.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b200b6180 con 0x7f4b200b8480 2023-10-16T12:33:05.837 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b200b6740 con 0x7f4b200b8480 2023-10-16T12:33:05.838 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.823+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4b1c044c00 con 0x7f4b200b8480 2023-10-16T12:33:05.838 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.824+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b1c02d040 con 0x7f4b200b8480 2023-10-16T12:33:05.839 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.824+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2004e9b0 con 0x7f4b200b8480 2023-10-16T12:33:05.839 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.827+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4b1c035020 con 0x7f4b200b8480 2023-10-16T12:33:05.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.905+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "95d94d30-9650-41d5-b171-abf1ef7481ee", "id": 0} v 0) v1 -- 0x7f4b200d0ea0 con 0x7f4b200b8480 2023-10-16T12:33:05.914 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.911+0000 7f4b15ffb640 1 -- 172.21.15.144:0/822263733 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "95d94d30-9650-41d5-b171-abf1ef7481ee", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4b1c04dcf0 con 0x7f4b200b8480 2023-10-16T12:33:05.914 INFO:teuthology.orchestra.run.smithi144.stdout:0 2023-10-16T12:33:05.915 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.912+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 msgr2=0x7f4b2014fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:05.916 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.912+0000 7f4b26527640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 0x7f4b2014fef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c000c00 tx=0x7f4b1c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.916 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 shutdown_connections 2023-10-16T12:33:05.916 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b2015cd70 0x7f4b200a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.916 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b200cded0 0x7f4b201f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.916 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 --2- 172.21.15.144:0/822263733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b200b8480 0x7f4b2014fef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:05.917 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 >> 172.21.15.144:0/822263733 conn(0x7f4b200cf570 msgr2=0x7f4b200b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:05.917 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 shutdown_connections 2023-10-16T12:33:05.917 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:05.913+0000 7f4b26527640 1 -- 172.21.15.144:0/822263733 wait complete. 2023-10-16T12:33:05.931 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 573b73e3-922c-4699-a735-2278cc9fa140 1 2023-10-16T12:33:05.956 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:05.953+0000 7ffb7eeb6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:33:06.023 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.020+0000 7ffb7eeb6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:33:06.038 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.036+0000 7f487782c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:33:06.077 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.074+0000 7ffb7eeb6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:33:06.101 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.100+0000 7f487782c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:33:06.116 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.113+0000 7fb3b0802640 1 Processor -- start 2023-10-16T12:33:06.116 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.113+0000 7fb3b0802640 1 -- start start 2023-10-16T12:33:06.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.114+0000 7fb3b0802640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac15cd70 0x7fb3ac0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.114+0000 7fb3b0802640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3ac0cded0 0x7fb3ac0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.114+0000 7fb3b0802640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac15d150 con 0x7fb3ac0b8480 2023-10-16T12:33:06.118 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.114+0000 7fb3b0802640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ac0d1180 con 0x7fb3ac0cded0 2023-10-16T12:33:06.119 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.114+0000 7fb3b0802640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0d1460 con 0x7fb3ac15cd70 2023-10-16T12:33:06.119 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3ab7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac15cd70 0x7fb3ac0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.119 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3a3fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb3ac0b8480 0x7fb3ac0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34796/0 (socket says 172.21.15.144:34796) 2023-10-16T12:33:06.119 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3a3fff640 1 -- 172.21.15.144:0/2822348456 learned_addr learned my addr 172.21.15.144:0/2822348456 (peer_addr_for_me v1:172.21.15.144:0/0) 2023-10-16T12:33:06.119 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3abfff640 1 --2- 172.21.15.144:0/2822348456 >> v2:172.21.15.60:3300/0 conn(0x7fb3ac0cded0 0x7fb3ac0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.120 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2822348456 >> v2:172.21.15.60:3300/0 conn(0x7fb3ac0cded0 msgr2=0x7fb3ac0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:06.120 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3aaffd640 1 -- 172.21.15.144:0/2822348456 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827366521 0 0) 0x7fb3ac15d150 con 0x7fb3ac0b8480 2023-10-16T12:33:06.120 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3ab7fe640 1 --2- 172.21.15.144:0/2822348456 >> v2:172.21.15.60:3300/0 conn(0x7fb3ac0cded0 0x7fb3ac0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.120 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2822348456 >> v1:172.21.15.144:6789/0 conn(0x7fb3ac0b8480 legacy=0x7fb3ac0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.121 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.115+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2822348456 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0d1740 con 0x7fb3ac15cd70 2023-10-16T12:33:06.123 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.116+0000 7fb3ab7fe640 1 --2- 172.21.15.144:0/2822348456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac15cd70 0x7fb3ac0d2220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0097a0 tx=0x7fb39c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9442f45d9e68df26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:06.124 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3aaffd640 1 -- 172.21.15.144:0/2822348456 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c02d040 con 0x7fb3ac15cd70 2023-10-16T12:33:06.124 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3aaffd640 1 -- 172.21.15.144:0/2822348456 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c031810 con 0x7fb3ac15cd70 2023-10-16T12:33:06.124 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3aaffd640 1 -- 172.21.15.144:0/2822348456 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c031ba0 con 0x7fb3ac15cd70 2023-10-16T12:33:06.124 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 -- 172.21.15.144:0/2822348456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac15cd70 msgr2=0x7fb3ac0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.124 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2822348456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac15cd70 0x7fb3ac0d2220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb39c0097a0 tx=0x7fb39c02b8b0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 -- 172.21.15.144:0/2822348456 shutdown_connections 2023-10-16T12:33:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2822348456 >> v2:172.21.15.60:3300/0 conn(0x7fb3ac0cded0 0x7fb3ac0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2822348456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac15cd70 0x7fb3ac0d2220 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 -- 172.21.15.144:0/2822348456 >> 172.21.15.144:0/2822348456 conn(0x7fb3ac0cf570 msgr2=0x7fb3ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 -- 172.21.15.144:0/2822348456 shutdown_connections 2023-10-16T12:33:06.126 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.117+0000 7fb3b0802640 1 -- 172.21.15.144:0/2822348456 wait complete. 2023-10-16T12:33:06.126 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.119+0000 7fb3b0802640 1 Processor -- start 2023-10-16T12:33:06.126 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.119+0000 7fb3b0802640 1 -- start start 2023-10-16T12:33:06.126 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3b0802640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 0x7fb3ac14bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.127 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3b0802640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac0cded0 0x7fb3ac0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.127 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3b0802640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3ac15cd70 0x7fb3ac149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.127 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3b0802640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac15a3d0 con 0x7fb3ac0cded0 2023-10-16T12:33:06.127 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3b0802640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac149790 con 0x7fb3ac15cd70 2023-10-16T12:33:06.127 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3b0802640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac148b80 con 0x7fb3ac0b8480 2023-10-16T12:33:06.127 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3ab7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 0x7fb3ac14bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.128 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3ab7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 0x7fb3ac14bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:40778/0 (socket says 172.21.15.144:40778) 2023-10-16T12:33:06.128 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2905221976 learned_addr learned my addr 172.21.15.144:0/2905221976 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:06.128 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.120+0000 7fb3abfff640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3ac15cd70 0x7fb3ac149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.128 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3a3fff640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac0cded0 0x7fb3ac0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.128 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3ac15cd70 msgr2=0x7fb3ac149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.129 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3ab7fe640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3ac15cd70 0x7fb3ac149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.129 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2905221976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac0cded0 msgr2=0x7fb3ac0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.129 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3ab7fe640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac0cded0 0x7fb3ac0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.129 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3ab7fe640 1 -- 172.21.15.144:0/2905221976 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c009400 con 0x7fb3ac0b8480 2023-10-16T12:33:06.129 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3ab7fe640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 0x7fb3ac14bf40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb39c000c00 tx=0x7fb39c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:06.131 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c03ad60 con 0x7fb3ac0b8480 2023-10-16T12:33:06.131 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c0443f0 con 0x7fb3ac0b8480 2023-10-16T12:33:06.131 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c0447a0 con 0x7fb3ac0b8480 2023-10-16T12:33:06.131 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.121+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac148e60 con 0x7fb3ac0b8480 2023-10-16T12:33:06.131 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.122+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac149420 con 0x7fb3ac0b8480 2023-10-16T12:33:06.132 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.122+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb39c044c00 con 0x7fb3ac0b8480 2023-10-16T12:33:06.132 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.122+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fb39c04c3b0 con 0x7fb3ac0b8480 2023-10-16T12:33:06.132 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.123+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac04e9b0 con 0x7fb3ac0b8480 2023-10-16T12:33:06.132 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.128+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb39c035020 con 0x7fb3ac0b8480 2023-10-16T12:33:06.153 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.151+0000 7ffb7eeb6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:33:06.155 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.153+0000 7f487782c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:33:06.205 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.202+0000 7ffb7eeb6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:33:06.218 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.215+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "573b73e3-922c-4699-a735-2278cc9fa140", "id": 1} v 0) v1 -- 0x7fb3ac0ced70 con 0x7fb3ac0b8480 2023-10-16T12:33:06.220 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.219+0000 7f487782c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:33:06.225 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.222+0000 7fb3a8ff9640 1 -- 172.21.15.144:0/2905221976 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "573b73e3-922c-4699-a735-2278cc9fa140", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb39c04d450 con 0x7fb3ac0b8480 2023-10-16T12:33:06.225 INFO:teuthology.orchestra.run.smithi144.stdout:1 2023-10-16T12:33:06.226 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 msgr2=0x7fb3ac14bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.227 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 0x7fb3ac14bf40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb39c000c00 tx=0x7fb39c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.227 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 shutdown_connections 2023-10-16T12:33:06.227 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3ac15cd70 0x7fb3ac149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.227 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac0cded0 0x7fb3ac0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.227 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 --2- 172.21.15.144:0/2905221976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3ac0b8480 0x7fb3ac14bf40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.228 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 >> 172.21.15.144:0/2905221976 conn(0x7fb3ac0cf570 msgr2=0x7fb3ac15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:06.228 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 shutdown_connections 2023-10-16T12:33:06.228 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.224+0000 7fb3b0802640 1 -- 172.21.15.144:0/2905221976 wait complete. 2023-10-16T12:33:06.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 67f1d1ee-c766-4cf0-a83d-60d628c75c8e 2 2023-10-16T12:33:06.272 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.271+0000 7f487782c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:33:06.325 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.323+0000 7ffb7eeb6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:33:06.390 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.389+0000 7f487782c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:33:06.410 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.407+0000 7ffb7eeb6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:33:06.466 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.463+0000 7ffb7eeb6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:33:06.473 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.472+0000 7f487782c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:33:06.474 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.471+0000 7efe9feb4640 1 Processor -- start 2023-10-16T12:33:06.474 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.471+0000 7efe9feb4640 1 -- start start 2023-10-16T12:33:06.475 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.472+0000 7efe9feb4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe980b3420 0x7efe980b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.476 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.472+0000 7efe9feb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cfaf0 0x7efe980b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.476 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.472+0000 7efe9feb4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe980d3980 con 0x7efe980cf710 2023-10-16T12:33:06.476 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.472+0000 7efe9feb4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe980d3c60 con 0x7efe980b3420 2023-10-16T12:33:06.476 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.472+0000 7efe9feb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe980ce630 con 0x7efe980cfaf0 2023-10-16T12:33:06.476 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.472+0000 7efe9d44f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe980b3420 0x7efe980b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.477 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9dc50640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7efe980cf710 0x7efe980b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34804/0 (socket says 172.21.15.144:34804) 2023-10-16T12:33:06.477 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9dc50640 1 -- 172.21.15.144:0/2092728714 learned_addr learned my addr 172.21.15.144:0/2092728714 (peer_addr_for_me v1:172.21.15.144:0/0) 2023-10-16T12:33:06.477 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9e451640 1 --2- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cfaf0 0x7efe980b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.477 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9cc4e640 1 -- 172.21.15.144:0/2092728714 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716584783 0 0) 0x7efe980d3980 con 0x7efe980cf710 2023-10-16T12:33:06.477 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9d44f640 1 -- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cfaf0 msgr2=0x7efe980b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:06.478 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9d44f640 1 --2- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cfaf0 0x7efe980b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.478 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9d44f640 1 -- 172.21.15.144:0/2092728714 >> v1:172.21.15.144:6789/0 conn(0x7efe980cf710 legacy=0x7efe980b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.478 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.473+0000 7efe9d44f640 1 -- 172.21.15.144:0/2092728714 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe980ce910 con 0x7efe980b3420 2023-10-16T12:33:06.480 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.474+0000 7efe9d44f640 1 --2- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980b3420 0x7efe980b5cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efe880097a0 tx=0x7efe8802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0768137666426dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:06.480 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.474+0000 7efe9cc4e640 1 -- 172.21.15.144:0/2092728714 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8802e070 con 0x7efe980b3420 2023-10-16T12:33:06.480 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.474+0000 7efe9cc4e640 1 -- 172.21.15.144:0/2092728714 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe880315a0 con 0x7efe980b3420 2023-10-16T12:33:06.480 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.474+0000 7efe9cc4e640 1 -- 172.21.15.144:0/2092728714 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88031930 con 0x7efe980b3420 2023-10-16T12:33:06.480 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 -- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980b3420 msgr2=0x7efe980b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.481 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 --2- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980b3420 0x7efe980b5cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efe880097a0 tx=0x7efe8802b840 comp rx=0 tx=0).stop 2023-10-16T12:33:06.481 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 -- 172.21.15.144:0/2092728714 shutdown_connections 2023-10-16T12:33:06.481 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 --2- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cfaf0 0x7efe980b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.481 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 --2- 172.21.15.144:0/2092728714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980b3420 0x7efe980b5cc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.481 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 -- 172.21.15.144:0/2092728714 >> 172.21.15.144:0/2092728714 conn(0x7efe980d2cc0 msgr2=0x7efe980d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:06.482 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 -- 172.21.15.144:0/2092728714 shutdown_connections 2023-10-16T12:33:06.482 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.475+0000 7efe9feb4640 1 -- 172.21.15.144:0/2092728714 wait complete. 2023-10-16T12:33:06.482 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.477+0000 7efe9feb4640 1 Processor -- start 2023-10-16T12:33:06.482 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.477+0000 7efe9feb4640 1 -- start start 2023-10-16T12:33:06.484 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9feb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe980b3420 0x7efe980cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.484 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9feb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cf710 0x7efe981521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9feb4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 0x7efe980c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9feb4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98152720 con 0x7efe980b3420 2023-10-16T12:33:06.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9feb4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe980af930 con 0x7efe980cfaf0 2023-10-16T12:33:06.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9feb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe980afc10 con 0x7efe980cf710 2023-10-16T12:33:06.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9e451640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 0x7efe980c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9d44f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cf710 0x7efe981521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9e451640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 0x7efe980c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43584/0 (socket says 172.21.15.144:43584) 2023-10-16T12:33:06.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.478+0000 7efe9e451640 1 -- 172.21.15.144:0/1449842688 learned_addr learned my addr 172.21.15.144:0/1449842688 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:06.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9dc50640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe980b3420 0x7efe980cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9e451640 1 -- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cf710 msgr2=0x7efe981521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.487 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9e451640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cf710 0x7efe981521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.487 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9e451640 1 -- 172.21.15.144:0/1449842688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe980b3420 msgr2=0x7efe980cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.487 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9e451640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe980b3420 0x7efe980cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.487 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9e451640 1 -- 172.21.15.144:0/1449842688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe88009400 con 0x7efe980cfaf0 2023-10-16T12:33:06.487 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9dc50640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe980b3420 0x7efe980cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:06.488 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9e451640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 0x7efe980c9d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe9000efc0 tx=0x7efe9000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:06.488 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90009ba0 con 0x7efe980cfaf0 2023-10-16T12:33:06.488 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9000f040 con 0x7efe980cfaf0 2023-10-16T12:33:06.488 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe900146c0 con 0x7efe980cfaf0 2023-10-16T12:33:06.488 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe980afef0 con 0x7efe980cfaf0 2023-10-16T12:33:06.489 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.479+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9815e380 con 0x7efe980cfaf0 2023-10-16T12:33:06.489 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.480+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efe90014b20 con 0x7efe980cfaf0 2023-10-16T12:33:06.489 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.480+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7efe9001d820 con 0x7efe980cfaf0 2023-10-16T12:33:06.489 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.481+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe980b0410 con 0x7efe980cfaf0 2023-10-16T12:33:06.490 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.483+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7efe980b0410 con 0x7efe980cfaf0 2023-10-16T12:33:06.529 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.527+0000 7f487782c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:33:06.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.527+0000 7ffb7eeb6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:33:06.563 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.560+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "67f1d1ee-c766-4cf0-a83d-60d628c75c8e", "id": 2} v 0) v1 -- 0x7efe9804e160 con 0x7efe980cfaf0 2023-10-16T12:33:06.570 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.567+0000 7efe8e7fc640 1 -- 172.21.15.144:0/1449842688 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67f1d1ee-c766-4cf0-a83d-60d628c75c8e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7efe90018020 con 0x7efe980cfaf0 2023-10-16T12:33:06.570 INFO:teuthology.orchestra.run.smithi144.stdout:2 2023-10-16T12:33:06.572 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 msgr2=0x7efe980c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.572 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 0x7efe980c9d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe9000efc0 tx=0x7efe9000c490 comp rx=0 tx=0).stop 2023-10-16T12:33:06.572 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 shutdown_connections 2023-10-16T12:33:06.573 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe980cf710 0x7efe981521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.573 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe980cfaf0 0x7efe980c9d90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.573 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 --2- 172.21.15.144:0/1449842688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe980b3420 0x7efe980cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.573 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 >> 172.21.15.144:0/1449842688 conn(0x7efe980d2cc0 msgr2=0x7efe980b5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:06.573 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 shutdown_connections 2023-10-16T12:33:06.574 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.569+0000 7efe9feb4640 1 -- 172.21.15.144:0/1449842688 wait complete. 2023-10-16T12:33:06.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new b4006289-df86-422f-889e-1221600060e9 3 2023-10-16T12:33:06.590 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:06.587+0000 7ffb7eeb6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:33:06.592 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.591+0000 7f487782c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:33:06.650 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:06.648+0000 7f487782c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:33:06.776 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.773+0000 7fed769fb640 1 Processor -- start 2023-10-16T12:33:06.776 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.773+0000 7fed769fb640 1 -- start start 2023-10-16T12:33:06.777 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.774+0000 7fed769fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 0x7fed700cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.778 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.774+0000 7fed769fb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed700d0be0 0x7fed700cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.778 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed769fb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7015d150 con 0x7fed7015cd70 2023-10-16T12:33:06.778 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed769fb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700d1180 con 0x7fed700d0be0 2023-10-16T12:33:06.778 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed769fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed700d1460 con 0x7fed700b8480 2023-10-16T12:33:06.779 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 0x7fed700cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.779 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 0x7fed700cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:34364/0 (socket says 172.21.15.144:34364) 2023-10-16T12:33:06.779 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 -- 172.21.15.144:0/3937584869 learned_addr learned my addr 172.21.15.144:0/3937584869 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:06.779 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed761fa640 1 --2- 172.21.15.144:0/3937584869 >> v2:172.21.15.144:3300/0 conn(0x7fed700d0be0 0x7fed700cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.780 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 -- 172.21.15.144:0/3937584869 >> v2:172.21.15.144:3300/0 conn(0x7fed700d0be0 msgr2=0x7fed700cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:06.781 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 --2- 172.21.15.144:0/3937584869 >> v2:172.21.15.144:3300/0 conn(0x7fed700d0be0 0x7fed700cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.781 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 -- 172.21.15.144:0/3937584869 >> v1:172.21.15.144:6789/0 conn(0x7fed7015cd70 legacy=0x7fed700d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:06.781 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.775+0000 7fed751f8640 1 -- 172.21.15.144:0/3937584869 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700d1740 con 0x7fed700b8480 2023-10-16T12:33:06.781 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.776+0000 7fed751f8640 1 --2- 172.21.15.144:0/3937584869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 0x7fed700cd7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fed60009870 tx=0x7fed6002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=915c48d2bdf366a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:06.781 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed749f7640 1 -- 172.21.15.144:0/3937584869 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60031640 con 0x7fed700b8480 2023-10-16T12:33:06.782 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed749f7640 1 -- 172.21.15.144:0/3937584869 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed600317a0 con 0x7fed700b8480 2023-10-16T12:33:06.782 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed749f7640 1 -- 172.21.15.144:0/3937584869 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60031b30 con 0x7fed700b8480 2023-10-16T12:33:06.782 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed769fb640 1 -- 172.21.15.144:0/3937584869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 msgr2=0x7fed700cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.782 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed769fb640 1 --2- 172.21.15.144:0/3937584869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 0x7fed700cd7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fed60009870 tx=0x7fed6002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.782 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed769fb640 1 -- 172.21.15.144:0/3937584869 shutdown_connections 2023-10-16T12:33:06.783 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed769fb640 1 --2- 172.21.15.144:0/3937584869 >> v2:172.21.15.144:3300/0 conn(0x7fed700d0be0 0x7fed700cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.783 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed769fb640 1 --2- 172.21.15.144:0/3937584869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700b8480 0x7fed700cd7c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.783 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.777+0000 7fed769fb640 1 -- 172.21.15.144:0/3937584869 >> 172.21.15.144:0/3937584869 conn(0x7fed700cf570 msgr2=0x7fed700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:06.783 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.778+0000 7fed769fb640 1 -- 172.21.15.144:0/3937584869 shutdown_connections 2023-10-16T12:33:06.783 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.778+0000 7fed769fb640 1 -- 172.21.15.144:0/3937584869 wait complete. 2023-10-16T12:33:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.779+0000 7fed769fb640 1 Processor -- start 2023-10-16T12:33:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.779+0000 7fed769fb640 1 -- start start 2023-10-16T12:33:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed769fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 0x7fed701f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.786 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed769fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700d0be0 0x7fed700b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed769fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed7015cd70 0x7fed700b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed769fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70145cc0 con 0x7fed7015cd70 2023-10-16T12:33:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed769fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed700c8a20 con 0x7fed700b8480 2023-10-16T12:33:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed769fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed700c8d00 con 0x7fed700d0be0 2023-10-16T12:33:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed759f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 0x7fed701f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.787 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed759f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 0x7fed701f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44370/0 (socket says 172.21.15.144:44370) 2023-10-16T12:33:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed759f9640 1 -- 172.21.15.144:0/448421308 learned_addr learned my addr 172.21.15.144:0/448421308 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.780+0000 7fed751f8640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700d0be0 0x7fed700b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed761fa640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed7015cd70 0x7fed700b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed759f9640 1 -- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700d0be0 msgr2=0x7fed700b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.788 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed759f9640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700d0be0 0x7fed700b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed759f9640 1 -- 172.21.15.144:0/448421308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed7015cd70 msgr2=0x7fed700b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed759f9640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed7015cd70 0x7fed700b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed759f9640 1 -- 172.21.15.144:0/448421308 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed600094d0 con 0x7fed700b8480 2023-10-16T12:33:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed761fa640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed7015cd70 0x7fed700b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:06.789 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed759f9640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 0x7fed701f1c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c430 tx=0x7fed6c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:06.790 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c011790 con 0x7fed700b8480 2023-10-16T12:33:06.790 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c0118f0 con 0x7fed700b8480 2023-10-16T12:33:06.790 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.781+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c011ce0 con 0x7fed700b8480 2023-10-16T12:33:06.790 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.782+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed700c8fe0 con 0x7fed700b8480 2023-10-16T12:33:06.790 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.782+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701f21c0 con 0x7fed700b8480 2023-10-16T12:33:06.791 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.782+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fed6c01c760 con 0x7fed700b8480 2023-10-16T12:33:06.791 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.782+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fed6c011e40 con 0x7fed700b8480 2023-10-16T12:33:06.791 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.783+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed700d28f0 con 0x7fed700b8480 2023-10-16T12:33:06.791 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.787+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fed6c012790 con 0x7fed700b8480 2023-10-16T12:33:06.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.864+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b4006289-df86-422f-889e-1221600060e9", "id": 3} v 0) v1 -- 0x7fed7004e9b0 con 0x7fed700b8480 2023-10-16T12:33:06.873 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.871+0000 7fed667fc640 1 -- 172.21.15.144:0/448421308 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b4006289-df86-422f-889e-1221600060e9", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fed6c01ca10 con 0x7fed700b8480 2023-10-16T12:33:06.874 INFO:teuthology.orchestra.run.smithi144.stdout:3 2023-10-16T12:33:06.875 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 msgr2=0x7fed701f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:06.875 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 0x7fed701f1c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c430 tx=0x7fed6c00c8f0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.875 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 shutdown_connections 2023-10-16T12:33:06.876 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed700d0be0 0x7fed700b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.876 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed7015cd70 0x7fed700b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.876 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 --2- 172.21.15.144:0/448421308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed700b8480 0x7fed701f1c80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:06.876 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.872+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 >> 172.21.15.144:0/448421308 conn(0x7fed700cf570 msgr2=0x7fed700b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:06.877 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.873+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 shutdown_connections 2023-10-16T12:33:06.877 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:06.873+0000 7fed769fb640 1 -- 172.21.15.144:0/448421308 wait complete. 2023-10-16T12:33:06.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new e11a5f19-1b14-45dd-8749-b082b4ddb0c5 4 2023-10-16T12:33:07.070 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.067+0000 7ffb7eeb6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:33:07.084 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.082+0000 7f5764415640 1 Processor -- start 2023-10-16T12:33:07.085 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.082+0000 7f5764415640 1 -- start start 2023-10-16T12:33:07.087 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.083+0000 7f5764415640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 0x7f575c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.087 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.083+0000 7f5764415640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f575c0c9000 0x7f575c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.087 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.083+0000 7f5764415640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f575c06a2e0 0x7f575c0cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.087 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.083+0000 7f5764415640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c161a50 con 0x7f575c0c9000 2023-10-16T12:33:07.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f5764415640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c06a750 con 0x7f575c06a2e0 2023-10-16T12:33:07.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f5764415640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0cb140 con 0x7f575c0addb0 2023-10-16T12:33:07.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 0x7f575c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57629b2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f575c06a2e0 0x7f575c0cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.088 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 0x7f575c0caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:34390/0 (socket says 172.21.15.144:34390) 2023-10-16T12:33:07.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 -- 172.21.15.144:0/3185130710 learned_addr learned my addr 172.21.15.144:0/3185130710 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:07.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57619b0640 1 --2- 172.21.15.144:0/3185130710 >> v2:172.21.15.60:3300/0 conn(0x7f575c0c9000 0x7f575c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 -- 172.21.15.144:0/3185130710 >> v2:172.21.15.144:3300/0 conn(0x7f575c06a2e0 msgr2=0x7f575c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:07.089 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 --2- 172.21.15.144:0/3185130710 >> v2:172.21.15.144:3300/0 conn(0x7f575c06a2e0 0x7f575c0cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.090 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 -- 172.21.15.144:0/3185130710 >> v2:172.21.15.60:3300/0 conn(0x7f575c0c9000 msgr2=0x7f575c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:07.090 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 --2- 172.21.15.144:0/3185130710 >> v2:172.21.15.60:3300/0 conn(0x7f575c0c9000 0x7f575c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.090 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.084+0000 7f57621b1640 1 -- 172.21.15.144:0/3185130710 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c06b030 con 0x7f575c0addb0 2023-10-16T12:33:07.091 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.085+0000 7f57621b1640 1 --2- 172.21.15.144:0/3185130710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 0x7f575c0caa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f575c0cf850 tx=0x7f575802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6487479948bfac96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:07.091 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f57611af640 1 -- 172.21.15.144:0/3185130710 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758031640 con 0x7f575c0addb0 2023-10-16T12:33:07.092 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f57611af640 1 -- 172.21.15.144:0/3185130710 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580317a0 con 0x7f575c0addb0 2023-10-16T12:33:07.092 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f57611af640 1 -- 172.21.15.144:0/3185130710 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758031b30 con 0x7f575c0addb0 2023-10-16T12:33:07.092 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 -- 172.21.15.144:0/3185130710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 msgr2=0x7f575c0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.092 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 --2- 172.21.15.144:0/3185130710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 0x7f575c0caa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f575c0cf850 tx=0x7f575802b8f0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.092 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 -- 172.21.15.144:0/3185130710 shutdown_connections 2023-10-16T12:33:07.093 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 --2- 172.21.15.144:0/3185130710 >> v2:172.21.15.144:3300/0 conn(0x7f575c06a2e0 0x7f575c0cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.093 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 --2- 172.21.15.144:0/3185130710 >> v2:172.21.15.60:3300/0 conn(0x7f575c0c9000 0x7f575c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.093 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 --2- 172.21.15.144:0/3185130710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0addb0 0x7f575c0caa80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.093 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.086+0000 7f5764415640 1 -- 172.21.15.144:0/3185130710 >> 172.21.15.144:0/3185130710 conn(0x7f575c0ce9b0 msgr2=0x7f575c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:07.093 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.087+0000 7f5764415640 1 -- 172.21.15.144:0/3185130710 shutdown_connections 2023-10-16T12:33:07.094 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.087+0000 7f5764415640 1 -- 172.21.15.144:0/3185130710 wait complete. 2023-10-16T12:33:07.096 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.089+0000 7f5764415640 1 Processor -- start 2023-10-16T12:33:07.096 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.089+0000 7f5764415640 1 -- start start 2023-10-16T12:33:07.096 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f5764415640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 0x7f575c0c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f5764415640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575c0addb0 0x7f575c1fa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f5764415640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0c9000 0x7f575c1f7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f5764415640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0c80d0 con 0x7f575c0addb0 2023-10-16T12:33:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f5764415640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1f9fb0 con 0x7f575c06a2e0 2023-10-16T12:33:07.097 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f5764415640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1f7940 con 0x7f575c0c9000 2023-10-16T12:33:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f57621b1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 0x7f575c0c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f57621b1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 0x7f575c0c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44382/0 (socket says 172.21.15.144:44382) 2023-10-16T12:33:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f57621b1640 1 -- 172.21.15.144:0/61772610 learned_addr learned my addr 172.21.15.144:0/61772610 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f57629b2640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0c9000 0x7f575c1f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.098 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.091+0000 7f57619b0640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575c0addb0 0x7f575c1fa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.099 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f57621b1640 1 -- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0c9000 msgr2=0x7f575c1f7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.099 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f57621b1640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0c9000 0x7f575c1f7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.099 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f57621b1640 1 -- 172.21.15.144:0/61772610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575c0addb0 msgr2=0x7f575c1fa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.099 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f57621b1640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575c0addb0 0x7f575c1fa9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.099 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f57621b1640 1 -- 172.21.15.144:0/61772610 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580094d0 con 0x7f575c06a2e0 2023-10-16T12:33:07.100 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f57621b1640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 0x7f575c0c7450 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5758000c00 tx=0x7f57580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:07.100 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575803a930 con 0x7f575c06a2e0 2023-10-16T12:33:07.100 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575803aa90 con 0x7f575c06a2e0 2023-10-16T12:33:07.100 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575803ae40 con 0x7f575c06a2e0 2023-10-16T12:33:07.100 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0c7b10 con 0x7f575c06a2e0 2023-10-16T12:33:07.101 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.092+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1fa1c0 con 0x7f575c06a2e0 2023-10-16T12:33:07.101 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.093+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f575802f050 con 0x7f575c06a2e0 2023-10-16T12:33:07.101 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.093+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f575804dc60 con 0x7f575c06a2e0 2023-10-16T12:33:07.101 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.094+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c04e160 con 0x7f575c06a2e0 2023-10-16T12:33:07.102 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.098+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5758035020 con 0x7f575c06a2e0 2023-10-16T12:33:07.122 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.121+0000 7f487782c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:33:07.177 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.174+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e11a5f19-1b14-45dd-8749-b082b4ddb0c5", "id": 4} v 0) v1 -- 0x7f575c06c5b0 con 0x7f575c06a2e0 2023-10-16T12:33:07.184 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.181+0000 7f5752ffd640 1 -- 172.21.15.144:0/61772610 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e11a5f19-1b14-45dd-8749-b082b4ddb0c5", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f575802d040 con 0x7f575c06a2e0 2023-10-16T12:33:07.184 INFO:teuthology.orchestra.run.smithi144.stdout:4 2023-10-16T12:33:07.185 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 msgr2=0x7f575c0c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.186 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 0x7f575c0c7450 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5758000c00 tx=0x7f57580046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.186 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 shutdown_connections 2023-10-16T12:33:07.186 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f575c0c9000 0x7f575c1f7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.187 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f575c0addb0 0x7f575c1fa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.187 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 --2- 172.21.15.144:0/61772610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f575c06a2e0 0x7f575c0c7450 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.187 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 >> 172.21.15.144:0/61772610 conn(0x7f575c0ce9b0 msgr2=0x7f575c0aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:07.187 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 shutdown_connections 2023-10-16T12:33:07.187 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.183+0000 7f5764415640 1 -- 172.21.15.144:0/61772610 wait complete. 2023-10-16T12:33:07.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new e19e3b6a-c82e-47b4-bbc1-2542f699dd39 5 2023-10-16T12:33:07.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.241+0000 7ffb7eeb6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:33:07.291 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.289+0000 7f487782c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:33:07.388 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.385+0000 7fd94246d640 1 Processor -- start 2023-10-16T12:33:07.388 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.385+0000 7fd94246d640 1 -- start start 2023-10-16T12:33:07.390 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94246d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd93c15cd70 0x7fd93c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.390 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94246d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd93c0b8480 0x7fd93c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.390 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94246d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c15d150 con 0x7fd93c0d0be0 2023-10-16T12:33:07.390 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94246d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0d1180 con 0x7fd93c15cd70 2023-10-16T12:33:07.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94246d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0d1460 con 0x7fd93c0b8480 2023-10-16T12:33:07.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd941c6c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd93c0d0be0 0x7fd93c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57580/0 (socket says 172.21.15.144:57580) 2023-10-16T12:33:07.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd941c6c640 1 -- 172.21.15.144:0/2957579261 learned_addr learned my addr 172.21.15.144:0/2957579261 (peer_addr_for_me v1:172.21.15.144:0/0) 2023-10-16T12:33:07.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd940c6a640 1 --2- 172.21.15.144:0/2957579261 >> v2:172.21.15.144:3300/0 conn(0x7fd93c0b8480 0x7fd93c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.391 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd940c6a640 1 -- 172.21.15.144:0/2957579261 >> v2:172.21.15.60:3300/0 conn(0x7fd93c15cd70 msgr2=0x7fd93c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:07.392 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94146b640 1 --2- 172.21.15.144:0/2957579261 >> v2:172.21.15.60:3300/0 conn(0x7fd93c15cd70 0x7fd93c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.392 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd940c6a640 1 --2- 172.21.15.144:0/2957579261 >> v2:172.21.15.60:3300/0 conn(0x7fd93c15cd70 0x7fd93c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.392 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd940c6a640 1 -- 172.21.15.144:0/2957579261 >> v1:172.21.15.144:6789/0 conn(0x7fd93c0d0be0 legacy=0x7fd93c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.392 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd933fff640 1 -- 172.21.15.144:0/2957579261 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783925303 0 0) 0x7fd93c15d150 con 0x7fd93c0d0be0 2023-10-16T12:33:07.393 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd940c6a640 1 -- 172.21.15.144:0/2957579261 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0d1740 con 0x7fd93c0b8480 2023-10-16T12:33:07.393 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.387+0000 7fd94146b640 1 --2- 172.21.15.144:0/2957579261 >> v2:172.21.15.60:3300/0 conn(0x7fd93c15cd70 0x7fd93c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:07.396 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.388+0000 7fd940c6a640 1 --2- 172.21.15.144:0/2957579261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c0b8480 0x7fd93c0cd7c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd93c066a00 tx=0x7fd92c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=217e87376dc4e08d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:07.396 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd933fff640 1 -- 172.21.15.144:0/2957579261 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c02e070 con 0x7fd93c0b8480 2023-10-16T12:33:07.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd933fff640 1 -- 172.21.15.144:0/2957579261 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c0315a0 con 0x7fd93c0b8480 2023-10-16T12:33:07.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd933fff640 1 -- 172.21.15.144:0/2957579261 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c031930 con 0x7fd93c0b8480 2023-10-16T12:33:07.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 -- 172.21.15.144:0/2957579261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c0b8480 msgr2=0x7fd93c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.397 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 --2- 172.21.15.144:0/2957579261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c0b8480 0x7fd93c0cd7c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd93c066a00 tx=0x7fd92c02b840 comp rx=0 tx=0).stop 2023-10-16T12:33:07.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 -- 172.21.15.144:0/2957579261 shutdown_connections 2023-10-16T12:33:07.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 --2- 172.21.15.144:0/2957579261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c0b8480 0x7fd93c0cd7c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 --2- 172.21.15.144:0/2957579261 >> v2:172.21.15.60:3300/0 conn(0x7fd93c15cd70 0x7fd93c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 -- 172.21.15.144:0/2957579261 >> 172.21.15.144:0/2957579261 conn(0x7fd93c0cf570 msgr2=0x7fd93c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:07.398 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 -- 172.21.15.144:0/2957579261 shutdown_connections 2023-10-16T12:33:07.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.389+0000 7fd94246d640 1 -- 172.21.15.144:0/2957579261 wait complete. 2023-10-16T12:33:07.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.391+0000 7fd94246d640 1 Processor -- start 2023-10-16T12:33:07.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.391+0000 7fd94246d640 1 -- start start 2023-10-16T12:33:07.399 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94246d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd93c0b8480 0x7fd93c0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94246d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd93c0d0be0 0x7fd93c15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94246d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 0x7fd93c15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94246d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c1f2bc0 con 0x7fd93c0b8480 2023-10-16T12:33:07.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94246d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c1f2850 con 0x7fd93c15cd70 2023-10-16T12:33:07.400 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94246d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0cae90 con 0x7fd93c0d0be0 2023-10-16T12:33:07.401 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd941c6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 0x7fd93c15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.401 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd940c6a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd93c0d0be0 0x7fd93c15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.401 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd941c6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 0x7fd93c15dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44394/0 (socket says 172.21.15.144:44394) 2023-10-16T12:33:07.401 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd941c6c640 1 -- 172.21.15.144:0/340241524 learned_addr learned my addr 172.21.15.144:0/340241524 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:07.401 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.392+0000 7fd94146b640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd93c0b8480 0x7fd93c0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.402 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd941c6c640 1 -- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd93c0d0be0 msgr2=0x7fd93c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.402 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd941c6c640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd93c0d0be0 0x7fd93c15d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.402 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd941c6c640 1 -- 172.21.15.144:0/340241524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd93c0b8480 msgr2=0x7fd93c0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.402 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd941c6c640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd93c0b8480 0x7fd93c0ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.402 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd941c6c640 1 -- 172.21.15.144:0/340241524 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c009400 con 0x7fd93c15cd70 2023-10-16T12:33:07.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd941c6c640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 0x7fd93c15dd50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd93400c750 tx=0x7fd93400cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:07.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400ce60 con 0x7fd93c15cd70 2023-10-16T12:33:07.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93400a640 con 0x7fd93c15cd70 2023-10-16T12:33:07.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400a9d0 con 0x7fd93c15cd70 2023-10-16T12:33:07.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c0cb170 con 0x7fd93c15cd70 2023-10-16T12:33:07.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c0caae0 con 0x7fd93c15cd70 2023-10-16T12:33:07.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.393+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd9340070d0 con 0x7fd93c15cd70 2023-10-16T12:33:07.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.394+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fd934007900 con 0x7fd93c15cd70 2023-10-16T12:33:07.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.394+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c0d28f0 con 0x7fd93c15cd70 2023-10-16T12:33:07.405 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.396+0000 7ffb7eeb6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:33:07.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.397+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd934021060 con 0x7fd93c15cd70 2023-10-16T12:33:07.438 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.437+0000 7f487782c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:33:07.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.451+0000 7ffb7eeb6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:33:07.477 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.474+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e19e3b6a-c82e-47b4-bbc1-2542f699dd39", "id": 5} v 0) v1 -- 0x7fd93c1f2d00 con 0x7fd93c15cd70 2023-10-16T12:33:07.483 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.480+0000 7fd931ffb640 1 -- 172.21.15.144:0/340241524 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e19e3b6a-c82e-47b4-bbc1-2542f699dd39", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd93c1f2d00 con 0x7fd93c15cd70 2023-10-16T12:33:07.483 INFO:teuthology.orchestra.run.smithi144.stdout:5 2023-10-16T12:33:07.484 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 msgr2=0x7fd93c15dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 0x7fd93c15dd50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd93400c750 tx=0x7fd93400cc10 comp rx=0 tx=0).stop 2023-10-16T12:33:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 shutdown_connections 2023-10-16T12:33:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd93c0d0be0 0x7fd93c15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.485 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd93c15cd70 0x7fd93c15dd50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 --2- 172.21.15.144:0/340241524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd93c0b8480 0x7fd93c0ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 >> 172.21.15.144:0/340241524 conn(0x7fd93c0cf570 msgr2=0x7fd93c06bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:07.486 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 shutdown_connections 2023-10-16T12:33:07.487 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.482+0000 7fd94246d640 1 -- 172.21.15.144:0/340241524 wait complete. 2023-10-16T12:33:07.494 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.493+0000 7f487782c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:33:07.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new b23bca8c-e831-4949-b3da-3e333307e029 6 2023-10-16T12:33:07.569 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.566+0000 7ffb7eeb6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:33:07.608 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.607+0000 7f487782c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:33:07.688 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.685+0000 7ffb7eeb6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:33:07.724 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.723+0000 7f487782c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:33:07.735 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.732+0000 7fc869947640 1 Processor -- start 2023-10-16T12:33:07.735 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.732+0000 7fc869947640 1 -- start start 2023-10-16T12:33:07.737 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.734+0000 7fc869947640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 0x7fc8640b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.737 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.734+0000 7fc869947640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640d18c0 con 0x7fc8640cfaf0 2023-10-16T12:33:07.737 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.734+0000 7fc869947640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640d3980 con 0x7fc8640b6cd0 2023-10-16T12:33:07.737 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.734+0000 7fc869947640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640d3c60 con 0x7fc8640cf710 2023-10-16T12:33:07.738 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.734+0000 7fc862ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 0x7fc8640b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.738 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.735+0000 7fc8637fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc8640b6cd0 0x7fc8640d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57590/0 (socket says 172.21.15.144:57590) 2023-10-16T12:33:07.738 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.735+0000 7fc862ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 0x7fc8640b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:34410/0 (socket says 172.21.15.144:34410) 2023-10-16T12:33:07.739 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.735+0000 7fc8637fe640 1 -- 172.21.15.144:0/1580124681 learned_addr learned my addr 172.21.15.144:0/1580124681 (peer_addr_for_me v1:172.21.15.144:0/0) 2023-10-16T12:33:07.739 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.735+0000 7fc862ffd640 1 -- 172.21.15.144:0/1580124681 >> v1:172.21.15.144:6789/0 conn(0x7fc8640b6cd0 legacy=0x7fc8640d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.739 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.735+0000 7fc862ffd640 1 -- 172.21.15.144:0/1580124681 >> v1:172.21.15.60:6789/0 conn(0x7fc8640cfaf0 legacy=0x7fc8640d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:07.739 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.735+0000 7fc862ffd640 1 -- 172.21.15.144:0/1580124681 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640ce630 con 0x7fc8640cf710 2023-10-16T12:33:07.740 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.736+0000 7fc862ffd640 1 --2- 172.21.15.144:0/1580124681 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 0x7fc8640b2d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097c0 tx=0x7fc85402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2352268cbf79b1a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:07.740 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.736+0000 7fc8627fc640 1 -- 172.21.15.144:0/1580124681 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854031800 con 0x7fc8640cf710 2023-10-16T12:33:07.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.736+0000 7fc8627fc640 1 -- 172.21.15.144:0/1580124681 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854031960 con 0x7fc8640cf710 2023-10-16T12:33:07.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.736+0000 7fc8627fc640 1 -- 172.21.15.144:0/1580124681 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854031cf0 con 0x7fc8640cf710 2023-10-16T12:33:07.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 -- 172.21.15.144:0/1580124681 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 msgr2=0x7fc8640b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 --2- 172.21.15.144:0/1580124681 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 0x7fc8640b2d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097c0 tx=0x7fc85402b8a0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 -- 172.21.15.144:0/1580124681 shutdown_connections 2023-10-16T12:33:07.742 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 --2- 172.21.15.144:0/1580124681 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640cf710 0x7fc8640b2d10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.742 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 -- 172.21.15.144:0/1580124681 >> 172.21.15.144:0/1580124681 conn(0x7fc8640d2cc0 msgr2=0x7fc8640d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:07.742 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 -- 172.21.15.144:0/1580124681 shutdown_connections 2023-10-16T12:33:07.742 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.737+0000 7fc869947640 1 -- 172.21.15.144:0/1580124681 wait complete. 2023-10-16T12:33:07.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.739+0000 7fc869947640 1 Processor -- start 2023-10-16T12:33:07.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.739+0000 7fc869947640 1 -- start start 2023-10-16T12:33:07.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc869947640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 0x7fc8640cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc869947640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640cf710 0x7fc86406ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc869947640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8640cfaf0 0x7fc86406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:07.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc869947640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8641f27e0 con 0x7fc8640cfaf0 2023-10-16T12:33:07.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc869947640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8641608a0 con 0x7fc8640cf710 2023-10-16T12:33:07.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc869947640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864158e90 con 0x7fc8640b6cd0 2023-10-16T12:33:07.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 0x7fc8640cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc85bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640cf710 0x7fc86406ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 0x7fc8640cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:34412/0 (socket says 172.21.15.144:34412) 2023-10-16T12:33:07.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 -- 172.21.15.144:0/920224943 learned_addr learned my addr 172.21.15.144:0/920224943 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:07.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc8637fe640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8640cfaf0 0x7fc86406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:07.747 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 -- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640cf710 msgr2=0x7fc86406ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.747 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640cf710 0x7fc86406ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.747 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 -- 172.21.15.144:0/920224943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8640cfaf0 msgr2=0x7fc86406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.747 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.740+0000 7fc862ffd640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8640cfaf0 0x7fc86406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.747 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc862ffd640 1 -- 172.21.15.144:0/920224943 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540093a0 con 0x7fc8640b6cd0 2023-10-16T12:33:07.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc862ffd640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 0x7fc8640cc940 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc85402beb0 tx=0x7fc8540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:07.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85403bcc0 con 0x7fc8640b6cd0 2023-10-16T12:33:07.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85403be20 con 0x7fc8640b6cd0 2023-10-16T12:33:07.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8641f2a00 con 0x7fc8640b6cd0 2023-10-16T12:33:07.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854045630 con 0x7fc8640b6cd0 2023-10-16T12:33:07.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.741+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8641f2fc0 con 0x7fc8640b6cd0 2023-10-16T12:33:07.749 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.742+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc854045a90 con 0x7fc8640b6cd0 2023-10-16T12:33:07.749 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.742+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fc85402e070 con 0x7fc8640b6cd0 2023-10-16T12:33:07.749 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.742+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86404e160 con 0x7fc8640b6cd0 2023-10-16T12:33:07.750 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.745+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc854035020 con 0x7fc8640b6cd0 2023-10-16T12:33:07.824 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.822+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b23bca8c-e831-4949-b3da-3e333307e029", "id": 6} v 0) v1 -- 0x7fc8640b2fd0 con 0x7fc8640b6cd0 2023-10-16T12:33:07.831 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.828+0000 7fc860ff9640 1 -- 172.21.15.144:0/920224943 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b23bca8c-e831-4949-b3da-3e333307e029", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc85404e450 con 0x7fc8640b6cd0 2023-10-16T12:33:07.831 INFO:teuthology.orchestra.run.smithi144.stdout:6 2023-10-16T12:33:07.832 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 msgr2=0x7fc8640cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:07.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 0x7fc8640cc940 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc85402beb0 tx=0x7fc8540046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 shutdown_connections 2023-10-16T12:33:07.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8640cfaf0 0x7fc86406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8640cf710 0x7fc86406ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 --2- 172.21.15.144:0/920224943 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8640b6cd0 0x7fc8640cc940 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:07.833 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 >> 172.21.15.144:0/920224943 conn(0x7fc8640d2cc0 msgr2=0x7fc8640b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 shutdown_connections 2023-10-16T12:33:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:07.830+0000 7fc869947640 1 -- 172.21.15.144:0/920224943 wait complete. 2023-10-16T12:33:07.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 4062aa4b-4de1-479c-ad22-40a6624130b5 7 2023-10-16T12:33:07.857 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.854+0000 7ffb7eeb6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:33:07.891 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.889+0000 7f487782c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:33:07.909 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:07.907+0000 7ffb7eeb6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:33:07.942 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:07.941+0000 7f487782c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:33:08.046 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.043+0000 7faf44d81640 1 Processor -- start 2023-10-16T12:33:08.046 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.044+0000 7faf44d81640 1 -- start start 2023-10-16T12:33:08.047 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf44d81640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 0x7faf400d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.048 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf44d81640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4015d150 con 0x7faf400b8480 2023-10-16T12:33:08.048 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf44d81640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf400d1180 con 0x7faf400cded0 2023-10-16T12:33:08.048 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf44d81640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400d1460 con 0x7faf4015cd70 2023-10-16T12:33:08.048 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf3f7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 0x7faf400d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.049 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf3f7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 0x7faf400d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:34424/0 (socket says 172.21.15.144:34424) 2023-10-16T12:33:08.049 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.045+0000 7faf3f7fe640 1 -- 172.21.15.144:0/2554113345 learned_addr learned my addr 172.21.15.144:0/2554113345 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:08.050 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.046+0000 7faf3f7fe640 1 -- 172.21.15.144:0/2554113345 >> v1:172.21.15.144:6789/0 conn(0x7faf400cded0 legacy=0x7faf400b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.050 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.046+0000 7faf3f7fe640 1 -- 172.21.15.144:0/2554113345 >> v1:172.21.15.60:6789/0 conn(0x7faf400b8480 legacy=0x7faf400cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:08.051 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.046+0000 7faf3f7fe640 1 -- 172.21.15.144:0/2554113345 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf400d1740 con 0x7faf4015cd70 2023-10-16T12:33:08.051 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.047+0000 7faf3f7fe640 1 --2- 172.21.15.144:0/2554113345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 0x7faf400d2220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faf300098a0 tx=0x7faf3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a11189c00891c3e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:08.051 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.047+0000 7faf3e7fc640 1 -- 172.21.15.144:0/2554113345 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30030840 con 0x7faf4015cd70 2023-10-16T12:33:08.051 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.047+0000 7faf3e7fc640 1 -- 172.21.15.144:0/2554113345 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf300309a0 con 0x7faf4015cd70 2023-10-16T12:33:08.051 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.047+0000 7faf3e7fc640 1 -- 172.21.15.144:0/2554113345 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30030d30 con 0x7faf4015cd70 2023-10-16T12:33:08.052 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 -- 172.21.15.144:0/2554113345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 msgr2=0x7faf400d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.052 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 --2- 172.21.15.144:0/2554113345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 0x7faf400d2220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faf300098a0 tx=0x7faf3002b980 comp rx=0 tx=0).stop 2023-10-16T12:33:08.052 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 -- 172.21.15.144:0/2554113345 shutdown_connections 2023-10-16T12:33:08.052 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 --2- 172.21.15.144:0/2554113345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf4015cd70 0x7faf400d2220 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.052 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 -- 172.21.15.144:0/2554113345 >> 172.21.15.144:0/2554113345 conn(0x7faf400cf570 msgr2=0x7faf400cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:08.053 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 -- 172.21.15.144:0/2554113345 shutdown_connections 2023-10-16T12:33:08.053 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.048+0000 7faf44d81640 1 -- 172.21.15.144:0/2554113345 wait complete. 2023-10-16T12:33:08.055 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.049+0000 7faf44d81640 1 Processor -- start 2023-10-16T12:33:08.055 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.050+0000 7faf44d81640 1 -- start start 2023-10-16T12:33:08.055 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.050+0000 7faf44d81640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf400b8480 0x7faf401f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.056 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf44d81640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 0x7faf401f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.056 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf44d81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4015cd70 0x7faf400caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.056 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf44d81640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf400c9f10 con 0x7faf4015cd70 2023-10-16T12:33:08.056 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf44d81640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf401f02c0 con 0x7faf400b8480 2023-10-16T12:33:08.056 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf44d81640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf401f3840 con 0x7faf400cded0 2023-10-16T12:33:08.057 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf400b8480 0x7faf401f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.057 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 0x7faf401f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.057 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 0x7faf401f61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.144:34432/0 (socket says 172.21.15.144:34432) 2023-10-16T12:33:08.057 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 -- 172.21.15.144:0/940011845 learned_addr learned my addr 172.21.15.144:0/940011845 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-10-16T12:33:08.057 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3ffff640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4015cd70 0x7faf400caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.058 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 -- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf400b8480 msgr2=0x7faf401f5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.058 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf400b8480 0x7faf401f5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.058 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 -- 172.21.15.144:0/940011845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4015cd70 msgr2=0x7faf400caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.058 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4015cd70 0x7faf400caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.058 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3effd640 1 -- 172.21.15.144:0/940011845 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300094d0 con 0x7faf400cded0 2023-10-16T12:33:08.059 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.051+0000 7faf3ffff640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4015cd70 0x7faf400caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:08.059 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf3effd640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 0x7faf401f61b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faf3400c430 tx=0x7faf3400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:08.059 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34011790 con 0x7faf400cded0 2023-10-16T12:33:08.059 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf340118f0 con 0x7faf400cded0 2023-10-16T12:33:08.059 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34011ce0 con 0x7faf400cded0 2023-10-16T12:33:08.060 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf401f3b20 con 0x7faf400cded0 2023-10-16T12:33:08.060 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf401f40e0 con 0x7faf400cded0 2023-10-16T12:33:08.060 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7faf3401c760 con 0x7faf400cded0 2023-10-16T12:33:08.060 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.052+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7faf34011e40 con 0x7faf400cded0 2023-10-16T12:33:08.061 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.053+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf400d1ce0 con 0x7faf400cded0 2023-10-16T12:33:08.061 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.056+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7faf400d1ce0 con 0x7faf400cded0 2023-10-16T12:33:08.076 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:08.074+0000 7ffb7eeb6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:33:08.108 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:08.106+0000 7f487782c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:33:08.136 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.133+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4062aa4b-4de1-479c-ad22-40a6624130b5", "id": 7} v 0) v1 -- 0x7faf4004e9b0 con 0x7faf400cded0 2023-10-16T12:33:08.145 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.142+0000 7faf27fff640 1 -- 172.21.15.144:0/940011845 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4062aa4b-4de1-479c-ad22-40a6624130b5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7faf3401ca10 con 0x7faf400cded0 2023-10-16T12:33:08.145 INFO:teuthology.orchestra.run.smithi144.stdout:7 2023-10-16T12:33:08.146 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.143+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 msgr2=0x7faf401f61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.143+0000 7faf44d81640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 0x7faf401f61b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faf3400c430 tx=0x7faf3400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 shutdown_connections 2023-10-16T12:33:08.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf400cded0 0x7faf401f61b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4015cd70 0x7faf400caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.147 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 --2- 172.21.15.144:0/940011845 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf400b8480 0x7faf401f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 >> 172.21.15.144:0/940011845 conn(0x7faf400cf570 msgr2=0x7faf4015d700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:08.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 shutdown_connections 2023-10-16T12:33:08.148 INFO:teuthology.orchestra.run.smithi144.stderr:2023-10-16T12:33:08.144+0000 7faf44d81640 1 -- 172.21.15.144:0/940011845 wait complete. 2023-10-16T12:33:08.162 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-16T12:33:08.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-16T12:33:08.172 INFO:tasks.ceph.osd.0:Started 2023-10-16T12:33:08.172 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-16T12:33:08.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-16T12:33:08.181 INFO:tasks.ceph.osd.1:Started 2023-10-16T12:33:08.181 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-16T12:33:08.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-16T12:33:08.189 INFO:tasks.ceph.osd.2:Started 2023-10-16T12:33:08.189 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-16T12:33:08.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-16T12:33:08.198 INFO:tasks.ceph.osd.3:Started 2023-10-16T12:33:08.198 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-16T12:33:08.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-16T12:33:08.207 INFO:tasks.ceph.osd.4:Started 2023-10-16T12:33:08.207 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-16T12:33:08.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-16T12:33:08.216 INFO:tasks.ceph.osd.5:Started 2023-10-16T12:33:08.216 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-16T12:33:08.216 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-16T12:33:08.222 INFO:tasks.ceph.osd.6:Started 2023-10-16T12:33:08.223 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-16T12:33:08.223 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-16T12:33:08.228 INFO:tasks.ceph.osd.7:Started 2023-10-16T12:33:08.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:08.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.446+0000 7f9364101640 1 Processor -- start 2023-10-16T12:33:08.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.447+0000 7f9364101640 1 -- start start 2023-10-16T12:33:08.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9364101640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f935c0adb50 0x7f935c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9364101640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c070fd0 0x7f935c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9364101640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c071450 con 0x7f935c0adf30 2023-10-16T12:33:08.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9364101640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0cc600 con 0x7f935c0adb50 2023-10-16T12:33:08.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9364101640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0cc8e0 con 0x7f935c070fd0 2023-10-16T12:33:08.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f936169c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c070fd0 0x7f935c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9361e9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f935c0adb50 0x7f935c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f936269e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f935c0adf30 0x7f935c164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:36142/0 (socket says 172.21.15.60:36142) 2023-10-16T12:33:08.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f9361e9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f935c0adb50 0x7f935c06f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:50638/0 (socket says 172.21.15.60:50638) 2023-10-16T12:33:08.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.448+0000 7f936269e640 1 -- 172.21.15.60:0/3759994360 learned_addr learned my addr 172.21.15.60:0/3759994360 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:08.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.449+0000 7f9361e9d640 1 -- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c070fd0 msgr2=0x7f935c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:08.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.449+0000 7f9361e9d640 1 --2- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c070fd0 0x7f935c0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.449+0000 7f9361e9d640 1 -- 172.21.15.60:0/3759994360 >> v1:172.21.15.60:6789/0 conn(0x7f935c0adf30 legacy=0x7f935c164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.449+0000 7f9361e9d640 1 -- 172.21.15.60:0/3759994360 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0ccbc0 con 0x7f935c0adb50 2023-10-16T12:33:08.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.449+0000 7f936169c640 1 --2- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c070fd0 0x7f935c0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:08.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.449+0000 7f9360e9b640 1 -- 172.21.15.60:0/3759994360 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092735835 0 0) 0x7f935c071450 con 0x7f935c0adf30 2023-10-16T12:33:08.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.450+0000 7f9361e9d640 1 --2- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c0adb50 0x7f935c06f550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f93580097a0 tx=0x7f935802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e43ee93fab8ea71d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:08.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.451+0000 7f9360e9b640 1 -- 172.21.15.60:0/3759994360 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935802d040 con 0x7f935c0adb50 2023-10-16T12:33:08.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.451+0000 7f9360e9b640 1 -- 172.21.15.60:0/3759994360 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9358031810 con 0x7f935c0adb50 2023-10-16T12:33:08.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.451+0000 7f9360e9b640 1 -- 172.21.15.60:0/3759994360 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358031ba0 con 0x7f935c0adb50 2023-10-16T12:33:08.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.451+0000 7f9364101640 1 -- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c0adb50 msgr2=0x7f935c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.451+0000 7f9364101640 1 --2- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c0adb50 0x7f935c06f550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f93580097a0 tx=0x7f935802b8b0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.452+0000 7f9364101640 1 -- 172.21.15.60:0/3759994360 shutdown_connections 2023-10-16T12:33:08.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.452+0000 7f9364101640 1 --2- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c070fd0 0x7f935c0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.452+0000 7f9364101640 1 --2- 172.21.15.60:0/3759994360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c0adb50 0x7f935c06f550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.452+0000 7f9364101640 1 -- 172.21.15.60:0/3759994360 >> 172.21.15.60:0/3759994360 conn(0x7f935c0cb410 msgr2=0x7f935c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:08.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.452+0000 7f9364101640 1 -- 172.21.15.60:0/3759994360 shutdown_connections 2023-10-16T12:33:08.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.452+0000 7f9364101640 1 -- 172.21.15.60:0/3759994360 wait complete. 2023-10-16T12:33:08.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.453+0000 7f9364101640 1 Processor -- start 2023-10-16T12:33:08.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.454+0000 7f9364101640 1 -- start start 2023-10-16T12:33:08.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.454+0000 7f9364101640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c070fd0 0x7f935c152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.454+0000 7f9364101640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 0x7f935c0c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f9364101640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c0adf30 0x7f935c0b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:08.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f9364101640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c9890 con 0x7f935c0adb50 2023-10-16T12:33:08.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f9364101640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c92d0 con 0x7f935c070fd0 2023-10-16T12:33:08.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f9364101640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c95b0 con 0x7f935c0adf30 2023-10-16T12:33:08.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 0x7f935c0c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936269e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c0adf30 0x7f935c0b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 0x7f935c0c8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36620/0 (socket says 172.21.15.60:36620) 2023-10-16T12:33:08.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 -- 172.21.15.60:0/3805435283 learned_addr learned my addr 172.21.15.60:0/3805435283 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:08.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f9361e9d640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c070fd0 0x7f935c152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:08.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 -- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c0adf30 msgr2=0x7f935c0b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c0adf30 0x7f935c0b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 -- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c070fd0 msgr2=0x7f935c152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c070fd0 0x7f935c152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.455+0000 7f936169c640 1 -- 172.21.15.60:0/3805435283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c009400 con 0x7f935c0adb50 2023-10-16T12:33:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f936169c640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 0x7f935c0c8a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f934c00e960 tx=0x7f934c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c00ca00 con 0x7f935c0adb50 2023-10-16T12:33:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c00cb60 con 0x7f935c0adb50 2023-10-16T12:33:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c013470 con 0x7f935c0adb50 2023-10-16T12:33:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9358009400 con 0x7f935c0adb50 2023-10-16T12:33:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c0c39e0 con 0x7f935c0adb50 2023-10-16T12:33:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f934c005c90 con 0x7f935c0adb50 2023-10-16T12:33:08.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.456+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f934c0067f0 con 0x7f935c0adb50 2023-10-16T12:33:08.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.457+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c04e160 con 0x7f935c0adb50 2023-10-16T12:33:08.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.462+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f934c021060 con 0x7f935c0adb50 2023-10-16T12:33:08.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.525+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f935c0b4910 con 0x7f935c0adb50 2023-10-16T12:33:08.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.526+0000 7f93527fc640 1 -- 172.21.15.60:0/3805435283 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f934c029020 con 0x7f935c0adb50 2023-10-16T12:33:08.528 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:08.528 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:08.137168+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:08.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 msgr2=0x7f935c0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:08.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 0x7f935c0c8a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f934c00e960 tx=0x7f934c00ee20 comp rx=0 tx=0).stop 2023-10-16T12:33:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 shutdown_connections 2023-10-16T12:33:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f935c0adf30 0x7f935c0b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f935c0adb50 0x7f935c0c8a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 --2- 172.21.15.60:0/3805435283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f935c070fd0 0x7f935c152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 >> 172.21.15.60:0/3805435283 conn(0x7f935c0cb410 msgr2=0x7f935c06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 shutdown_connections 2023-10-16T12:33:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:08.528+0000 7f9364101640 1 -- 172.21.15.60:0/3805435283 wait complete. 2023-10-16T12:33:08.546 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-16T12:33:08.546 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-16T12:33:08.660 INFO:tasks.ceph.osd.2.smithi060.stderr:2023-10-16T12:33:08.659+0000 7fc872414500 -1 Falling back to public interface 2023-10-16T12:33:08.912 INFO:tasks.ceph.osd.3.smithi060.stderr:2023-10-16T12:33:08.911+0000 7f4dde425500 -1 Falling back to public interface 2023-10-16T12:33:08.916 INFO:tasks.ceph.osd.1.smithi060.stderr:2023-10-16T12:33:08.915+0000 7fedf8c0c500 -1 Falling back to public interface 2023-10-16T12:33:08.919 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-16T12:33:08.917+0000 7f549048a500 -1 Falling back to public interface 2023-10-16T12:33:08.925 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-16T12:33:08.922+0000 7fa82e638500 -1 Falling back to public interface 2023-10-16T12:33:08.928 INFO:tasks.ceph.osd.0.smithi060.stderr:2023-10-16T12:33:08.927+0000 7fed4440e500 -1 Falling back to public interface 2023-10-16T12:33:08.929 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-16T12:33:08.926+0000 7fd81a21d500 -1 Falling back to public interface 2023-10-16T12:33:08.929 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-16T12:33:08.926+0000 7f1749614500 -1 Falling back to public interface 2023-10-16T12:33:09.221 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:09.218+0000 7ffb7eeb6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:33:09.290 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:09.288+0000 7f487782c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:33:09.353 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:09.523 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.522+0000 7f5f9ce00640 1 Processor -- start 2023-10-16T12:33:09.524 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.522+0000 7f5f9ce00640 1 -- start start 2023-10-16T12:33:09.525 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.523+0000 7f5f9ce00640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f980ce390 0x7f5f9815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:09.525 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.523+0000 7f5f9ce00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9815ed60 con 0x7f5f9815aaa0 2023-10-16T12:33:09.525 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.523+0000 7f5f9ce00640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980707e0 con 0x7f5f9815ae80 2023-10-16T12:33:09.525 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.523+0000 7f5f9ce00640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9806b810 con 0x7f5f980ce390 2023-10-16T12:33:09.526 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5f9815aaa0 0x7f5f980cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41716/0 (socket says 172.21.15.60:41716) 2023-10-16T12:33:09.526 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9659c640 1 -- 172.21.15.60:0/4035983790 learned_addr learned my addr 172.21.15.60:0/4035983790 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:09.526 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f96d9d640 1 --2- 172.21.15.60:0/4035983790 >> v2:172.21.15.144:3300/0 conn(0x7f5f980ce390 0x7f5f9815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:09.527 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096822655 0 0) 0x7f5f9815ed60 con 0x7f5f9815aaa0 2023-10-16T12:33:09.527 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f80003520 con 0x7f5f9815aaa0 2023-10-16T12:33:09.527 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089953975 0 0) 0x7f5f980707e0 con 0x7f5f9815ae80 2023-10-16T12:33:09.527 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9815ed60 con 0x7f5f9815ae80 2023-10-16T12:33:09.528 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f96d9d640 1 -- 172.21.15.60:0/4035983790 >> v1:172.21.15.144:6789/0 conn(0x7f5f9815ae80 legacy=0x7f5f98070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:09.528 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3182532616 0 0) 0x7f5f9815ed60 con 0x7f5f9815ae80 2023-10-16T12:33:09.528 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.524+0000 7f5f96d9d640 1 -- 172.21.15.60:0/4035983790 >> v1:172.21.15.60:6789/0 conn(0x7f5f9815aaa0 legacy=0x7f5f980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:09.528 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.525+0000 7f5f96d9d640 1 -- 172.21.15.60:0/4035983790 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9815ef90 con 0x7f5f980ce390 2023-10-16T12:33:09.529 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.526+0000 7f5f96d9d640 1 --2- 172.21.15.60:0/4035983790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f9815e820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009d90 tx=0x7f5f9002bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965d3340a82d7992 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:09.529 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.527+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f880015f0 con 0x7f5f980ce390 2023-10-16T12:33:09.530 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.527+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9002c830 con 0x7f5f980ce390 2023-10-16T12:33:09.530 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.527+0000 7f5f9559a640 1 -- 172.21.15.60:0/4035983790 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c001670 con 0x7f5f980ce390 2023-10-16T12:33:09.530 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.527+0000 7f5f9ce00640 1 -- 172.21.15.60:0/4035983790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 msgr2=0x7f5f9815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:09.530 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.527+0000 7f5f9ce00640 1 --2- 172.21.15.60:0/4035983790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f9815e820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009d90 tx=0x7f5f9002bd00 comp rx=0 tx=0).stop 2023-10-16T12:33:09.531 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.528+0000 7f5f9ce00640 1 -- 172.21.15.60:0/4035983790 shutdown_connections 2023-10-16T12:33:09.531 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.528+0000 7f5f9ce00640 1 --2- 172.21.15.60:0/4035983790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f9815e820 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:09.531 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.528+0000 7f5f9ce00640 1 -- 172.21.15.60:0/4035983790 >> 172.21.15.60:0/4035983790 conn(0x7f5f980cb410 msgr2=0x7f5f980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:09.531 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.528+0000 7f5f9ce00640 1 -- 172.21.15.60:0/4035983790 shutdown_connections 2023-10-16T12:33:09.531 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.528+0000 7f5f9ce00640 1 -- 172.21.15.60:0/4035983790 wait complete. 2023-10-16T12:33:09.532 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:09.527+0000 7ffb7eeb6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:33:09.534 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.529+0000 7f5f9ce00640 1 Processor -- start 2023-10-16T12:33:09.534 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.529+0000 7f5f9ce00640 1 -- start start 2023-10-16T12:33:09.534 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.530+0000 7f5f9ce00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f980c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:09.535 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.530+0000 7f5f9ce00640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f9815aaa0 0x7f5f980b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:09.535 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.530+0000 7f5f9ce00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 0x7f5f980b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:09.535 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f9ce00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9814f770 con 0x7f5f9815ae80 2023-10-16T12:33:09.535 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f9ce00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980c8890 con 0x7f5f980ce390 2023-10-16T12:33:09.535 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f9ce00640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980c7830 con 0x7f5f9815aaa0 2023-10-16T12:33:09.535 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 0x7f5f980b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:09.536 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f9659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f980c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:09.536 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f9659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f980c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:43042/0 (socket says 172.21.15.60:43042) 2023-10-16T12:33:09.536 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 0x7f5f980b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45194/0 (socket says 172.21.15.60:45194) 2023-10-16T12:33:09.536 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f9659c640 1 -- 172.21.15.60:0/3251979003 learned_addr learned my addr 172.21.15.60:0/3251979003 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:09.536 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f95d9b640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f9815aaa0 0x7f5f980b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:09.536 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 -- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f9815aaa0 msgr2=0x7f5f980b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:09.537 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f9815aaa0 0x7f5f980b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:09.537 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 -- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 msgr2=0x7f5f980c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:09.537 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f980c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:09.537 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.531+0000 7f5f96d9d640 1 -- 172.21.15.60:0/3251979003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90009a70 con 0x7f5f9815ae80 2023-10-16T12:33:09.537 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.532+0000 7f5f96d9d640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 0x7f5f980b0600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f980cc370 tx=0x7f5f90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:09.537 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.532+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90034e50 con 0x7f5f9815ae80 2023-10-16T12:33:09.538 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.532+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9002ce80 con 0x7f5f9815ae80 2023-10-16T12:33:09.538 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.532+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90003bd0 con 0x7f5f9815ae80 2023-10-16T12:33:09.538 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.532+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f980c7ad0 con 0x7f5f9815ae80 2023-10-16T12:33:09.538 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.532+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f980c8090 con 0x7f5f9815ae80 2023-10-16T12:33:09.538 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.533+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5f9003b050 con 0x7f5f9815ae80 2023-10-16T12:33:09.539 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.533+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5f9003eae0 con 0x7f5f9815ae80 2023-10-16T12:33:09.539 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.533+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9804e160 con 0x7f5f9815ae80 2023-10-16T12:33:09.539 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.536+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5f90039070 con 0x7f5f9815ae80 2023-10-16T12:33:09.591 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:09.588+0000 7ffb7eeb6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:33:09.602 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.600+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f980cdc60 con 0x7f5f9815ae80 2023-10-16T12:33:09.603 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.601+0000 7f5f86ffd640 1 -- 172.21.15.60:0/3251979003 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5f9003ed40 con 0x7f5f9815ae80 2023-10-16T12:33:09.603 INFO:teuthology.misc.health.smithi060.stdout: 2023-10-16T12:33:09.603 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:08.137168+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:09.604 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 msgr2=0x7f5f980b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:09.605 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 0x7f5f980b0600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f980cc370 tx=0x7f5f90004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:09.605 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 shutdown_connections 2023-10-16T12:33:09.605 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f9815aaa0 0x7f5f980b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:09.605 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f9815ae80 0x7f5f980b0600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:09.605 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 --2- 172.21.15.60:0/3251979003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f980ce390 0x7f5f980c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:09.606 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 >> 172.21.15.60:0/3251979003 conn(0x7f5f980cb410 msgr2=0x7f5f9806ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:09.606 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 shutdown_connections 2023-10-16T12:33:09.606 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:09.603+0000 7f5f9ce00640 1 -- 172.21.15.60:0/3251979003 wait complete. 2023-10-16T12:33:09.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-16T12:33:09.635 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:09.633+0000 7f487782c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:33:09.697 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:09.696+0000 7f487782c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:33:09.703 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:09.700+0000 7ffb7eeb6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:33:09.821 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:33:09.819+0000 7f487782c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:33:10.753 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:10.750+0000 7ffb209e2640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-16T12:33:10.755 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:33:10.752+0000 7ffb209e2640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-16T12:33:12.210 INFO:tasks.ceph.osd.2.smithi060.stderr:2023-10-16T12:33:12.209+0000 7fc872414500 -1 osd.2 0 log_to_monitors true 2023-10-16T12:33:12.460 INFO:tasks.ceph.osd.1.smithi060.stderr:2023-10-16T12:33:12.458+0000 7fedf8c0c500 -1 osd.1 0 log_to_monitors true 2023-10-16T12:33:12.463 INFO:tasks.ceph.osd.3.smithi060.stderr:2023-10-16T12:33:12.462+0000 7f4dde425500 -1 osd.3 0 log_to_monitors true 2023-10-16T12:33:12.484 INFO:tasks.ceph.osd.0.smithi060.stderr:2023-10-16T12:33:12.483+0000 7fed4440e500 -1 osd.0 0 log_to_monitors true 2023-10-16T12:33:12.488 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-16T12:33:12.486+0000 7fa82e638500 -1 osd.7 0 log_to_monitors true 2023-10-16T12:33:12.516 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-16T12:33:12.514+0000 7f1749614500 -1 osd.5 0 log_to_monitors true 2023-10-16T12:33:12.700 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-16T12:33:12.698+0000 7f549048a500 -1 osd.4 0 log_to_monitors true 2023-10-16T12:33:12.780 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-16T12:33:12.778+0000 7fd81a21d500 -1 osd.6 0 log_to_monitors true 2023-10-16T12:33:15.508 INFO:tasks.ceph.osd.2.smithi060.stderr:2023-10-16T12:33:15.506+0000 7fc86edbe640 -1 osd.2 0 waiting for initial osdmap 2023-10-16T12:33:15.526 INFO:tasks.ceph.osd.2.smithi060.stderr:2023-10-16T12:33:15.525+0000 7fc865bb7640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:15.537 INFO:tasks.ceph.osd.1.smithi060.stderr:2023-10-16T12:33:15.536+0000 7fedf54e1640 -1 osd.1 0 waiting for initial osdmap 2023-10-16T12:33:15.553 INFO:tasks.ceph.osd.3.smithi060.stderr:2023-10-16T12:33:15.552+0000 7f4dd95ee640 -1 osd.3 0 waiting for initial osdmap 2023-10-16T12:33:15.555 INFO:tasks.ceph.osd.1.smithi060.stderr:2023-10-16T12:33:15.554+0000 7fedec2da640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:15.559 INFO:tasks.ceph.osd.0.smithi060.stderr:2023-10-16T12:33:15.558+0000 7fed40dcf640 -1 osd.0 0 waiting for initial osdmap 2023-10-16T12:33:15.567 INFO:tasks.ceph.osd.3.smithi060.stderr:2023-10-16T12:33:15.566+0000 7f4dd1c21640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:15.575 INFO:tasks.ceph.osd.0.smithi060.stderr:2023-10-16T12:33:15.574+0000 7fed37bc8640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:15.702 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-16T12:33:15.700+0000 7fa829da3640 -1 osd.7 0 waiting for initial osdmap 2023-10-16T12:33:15.723 INFO:tasks.ceph.osd.7.smithi144.stderr:2023-10-16T12:33:15.720+0000 7fa8223d6640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:15.820 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-16T12:33:15.818+0000 7f548bbf3640 -1 osd.4 0 waiting for initial osdmap 2023-10-16T12:33:15.838 INFO:tasks.ceph.osd.4.smithi144.stderr:2023-10-16T12:33:15.836+0000 7f5484226640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:15.846 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-16T12:33:15.844+0000 7f17446ba640 -1 osd.5 0 waiting for initial osdmap 2023-10-16T12:33:15.858 INFO:tasks.ceph.osd.5.smithi144.stderr:2023-10-16T12:33:15.855+0000 7f173cced640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:16.187 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-16T12:33:16.185+0000 7fd81533c640 -1 osd.6 0 waiting for initial osdmap 2023-10-16T12:33:16.206 INFO:tasks.ceph.osd.6.smithi144.stderr:2023-10-16T12:33:16.204+0000 7fd80d96f640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:33:16.425 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:16.628 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.626+0000 7ff056c32640 1 Processor -- start 2023-10-16T12:33:16.628 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.627+0000 7ff056c32640 1 -- start start 2023-10-16T12:33:16.630 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.628+0000 7ff056c32640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0500c9fc0 0x7ff0500715b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:16.630 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.628+0000 7ff056c32640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff050070d80 0x7ff0500c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:16.631 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.628+0000 7ff056c32640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05006c940 con 0x7ff050159cf0 2023-10-16T12:33:16.631 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.628+0000 7ff056c32640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05006d150 con 0x7ff0500c9fc0 2023-10-16T12:33:16.631 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.628+0000 7ff056c32640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff050071190 con 0x7ff050070d80 2023-10-16T12:33:16.631 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.628+0000 7ff05542f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0500c9fc0 0x7ff0500715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:16.632 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff05542f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0500c9fc0 0x7ff0500715b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45246/0 (socket says 172.21.15.60:45246) 2023-10-16T12:33:16.632 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff05542f640 1 -- 172.21.15.60:0/2389158592 learned_addr learned my addr 172.21.15.60:0/2389158592 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:16.632 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff056431640 1 --2- 172.21.15.60:0/2389158592 >> v2:172.21.15.144:3300/0 conn(0x7ff050070d80 0x7ff0500c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:16.632 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff05542f640 1 -- 172.21.15.60:0/2389158592 >> v2:172.21.15.144:3300/0 conn(0x7ff050070d80 msgr2=0x7ff0500c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:16.632 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff05542f640 1 --2- 172.21.15.60:0/2389158592 >> v2:172.21.15.144:3300/0 conn(0x7ff050070d80 0x7ff0500c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.633 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff05542f640 1 -- 172.21.15.60:0/2389158592 >> v1:172.21.15.144:6789/0 conn(0x7ff050159cf0 legacy=0x7ff0500c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:16.633 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.629+0000 7ff05542f640 1 -- 172.21.15.60:0/2389158592 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05015a0d0 con 0x7ff0500c9fc0 2023-10-16T12:33:16.634 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.630+0000 7ff05542f640 1 --2- 172.21.15.60:0/2389158592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff0500715b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff0500cb530 tx=0x7ff04002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b5e90e9edb8460 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:16.635 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.630+0000 7ff054c2e640 1 -- 172.21.15.60:0/2389158592 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04002f850 con 0x7ff0500c9fc0 2023-10-16T12:33:16.635 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.630+0000 7ff054c2e640 1 -- 172.21.15.60:0/2389158592 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04002f9b0 con 0x7ff0500c9fc0 2023-10-16T12:33:16.635 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.630+0000 7ff054c2e640 1 -- 172.21.15.60:0/2389158592 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04002fd40 con 0x7ff0500c9fc0 2023-10-16T12:33:16.635 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 -- 172.21.15.60:0/2389158592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 msgr2=0x7ff0500715b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:16.636 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 --2- 172.21.15.60:0/2389158592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff0500715b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff0500cb530 tx=0x7ff04002ba30 comp rx=0 tx=0).stop 2023-10-16T12:33:16.636 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 -- 172.21.15.60:0/2389158592 shutdown_connections 2023-10-16T12:33:16.636 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 --2- 172.21.15.60:0/2389158592 >> v2:172.21.15.144:3300/0 conn(0x7ff050070d80 0x7ff0500c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.636 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 --2- 172.21.15.60:0/2389158592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff0500715b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.637 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 -- 172.21.15.60:0/2389158592 >> 172.21.15.60:0/2389158592 conn(0x7ff0500ceb80 msgr2=0x7ff0500cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:16.637 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 -- 172.21.15.60:0/2389158592 shutdown_connections 2023-10-16T12:33:16.637 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.631+0000 7ff056c32640 1 -- 172.21.15.60:0/2389158592 wait complete. 2023-10-16T12:33:16.637 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.633+0000 7ff056c32640 1 Processor -- start 2023-10-16T12:33:16.637 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.633+0000 7ff056c32640 1 -- start start 2023-10-16T12:33:16.639 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056c32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff050070d80 0x7ff0500c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:16.640 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056c32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff050155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:16.640 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056c32640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff050159cf0 0x7ff050151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:16.640 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056c32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff050156390 con 0x7ff0500c9fc0 2023-10-16T12:33:16.640 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056c32640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff050151a80 con 0x7ff050159cf0 2023-10-16T12:33:16.640 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056c32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff050151d20 con 0x7ff050070d80 2023-10-16T12:33:16.641 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff050155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:16.641 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff050155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45262/0 (socket says 172.21.15.60:45262) 2023-10-16T12:33:16.641 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 -- 172.21.15.60:0/846011232 learned_addr learned my addr 172.21.15.60:0/846011232 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:16.641 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff056431640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff050159cf0 0x7ff050151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:16.641 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff055c30640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff050070d80 0x7ff0500c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:16.642 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 -- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff050070d80 msgr2=0x7ff0500c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:16.642 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff050070d80 0x7ff0500c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.642 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 -- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff050159cf0 msgr2=0x7ff050151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:16.642 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff050159cf0 0x7ff050151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.642 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.634+0000 7ff05542f640 1 -- 172.21.15.60:0/846011232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400094d0 con 0x7ff0500c9fc0 2023-10-16T12:33:16.643 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.635+0000 7ff05542f640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff050155e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff04002ba70 tx=0x7ff040004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:16.643 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.635+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040037d10 con 0x7ff0500c9fc0 2023-10-16T12:33:16.643 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.635+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040037e90 con 0x7ff0500c9fc0 2023-10-16T12:33:16.643 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.635+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff050152000 con 0x7ff0500c9fc0 2023-10-16T12:33:16.643 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.635+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040041640 con 0x7ff0500c9fc0 2023-10-16T12:33:16.643 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.635+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0501525c0 con 0x7ff0500c9fc0 2023-10-16T12:33:16.644 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.636+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff050154140 con 0x7ff0500c9fc0 2023-10-16T12:33:16.644 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.637+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7ff0400417a0 con 0x7ff0500c9fc0 2023-10-16T12:33:16.644 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.638+0000 7ff0467fc640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff03406ade0 0x7ff03406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:16.644 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.638+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7ff04003c070 con 0x7ff0500c9fc0 2023-10-16T12:33:16.645 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.639+0000 7ff055c30640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff03406ade0 0x7ff03406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:16.645 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.640+0000 7ff055c30640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff03406ade0 0x7ff03406d2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff04c000f60 tx=0x7ff04c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:16.645 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.643+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff050154140 con 0x7ff0500c9fc0 2023-10-16T12:33:16.836 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.835+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff050154140 con 0x7ff0500c9fc0 2023-10-16T12:33:16.838 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.836+0000 7ff0467fc640 1 -- 172.21.15.60:0/846011232 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10896 (secure 0 0 0) 0x7ff050154140 con 0x7ff0500c9fc0 2023-10-16T12:33:16.838 INFO:teuthology.misc.health.smithi060.stdout: 2023-10-16T12:33:16.838 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":14,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:16.511383+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:16.840 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff03406ade0 msgr2=0x7ff03406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:16.841 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff03406ade0 0x7ff03406d2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff04c000f60 tx=0x7ff04c009040 comp rx=0 tx=0).stop 2023-10-16T12:33:16.841 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 msgr2=0x7ff050155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:16.841 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff050155e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff04002ba70 tx=0x7ff040004730 comp rx=0 tx=0).stop 2023-10-16T12:33:16.841 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 shutdown_connections 2023-10-16T12:33:16.842 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff03406ade0 0x7ff03406d2a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.842 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff050159cf0 0x7ff050151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.842 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0500c9fc0 0x7ff050155e50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.842 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 --2- 172.21.15.60:0/846011232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff050070d80 0x7ff0500c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:16.842 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 >> 172.21.15.60:0/846011232 conn(0x7ff0500ceb80 msgr2=0x7ff0500cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:16.843 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 shutdown_connections 2023-10-16T12:33:16.843 INFO:teuthology.misc.health.smithi060.stderr:2023-10-16T12:33:16.839+0000 7ff056c32640 1 -- 172.21.15.60:0/846011232 wait complete. 2023-10-16T12:33:16.854 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-16T12:33:16.855 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-16T12:33:16.855 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:33:16.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:33:16.864 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:33:16.864 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:33:16.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:33:16.873 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:33:16.874 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:33:16.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:33:16.883 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:33:16.884 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:33:16.884 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:33:16.894 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:33:16.894 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-16T12:33:16.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:17.015 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:33:17.030 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:33:17.047 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:33:17.051 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:33:17.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.075+0000 7fed50178640 1 Processor -- start 2023-10-16T12:33:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.075+0000 7fed50178640 1 -- start start 2023-10-16T12:33:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.076+0000 7fed50178640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed480cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.076+0000 7fed50178640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fed480ce7f0 0x7fed480cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.076+0000 7fed50178640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed480cc940 con 0x7fed4815f220 2023-10-16T12:33:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.076+0000 7fed50178640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed480d1560 con 0x7fed480ce7f0 2023-10-16T12:33:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.076+0000 7fed50178640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480cbd30 con 0x7fed480cd510 2023-10-16T12:33:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fed480ce7f0 0x7fed480cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fed480ce7f0 0x7fed480cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45304/0 (socket says 172.21.15.60:45304) 2023-10-16T12:33:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 -- 172.21.15.60:0/3865997928 learned_addr learned my addr 172.21.15.60:0/3865997928 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4df14640 1 --2- 172.21.15.60:0/3865997928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed480cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 -- 172.21.15.60:0/3865997928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 msgr2=0x7fed480cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 --2- 172.21.15.60:0/3865997928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed480cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 -- 172.21.15.60:0/3865997928 >> v1:172.21.15.144:6789/0 conn(0x7fed4815f220 legacy=0x7fed480cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.077+0000 7fed4e715640 1 -- 172.21.15.60:0/3865997928 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed480cc010 con 0x7fed480ce7f0 2023-10-16T12:33:17.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.078+0000 7fed4e715640 1 --2- 172.21.15.60:0/3865997928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed480ce7f0 0x7fed480cec80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fed40009e50 tx=0x7fed4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6e52a0962fc993 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:17.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed4cf12640 1 -- 172.21.15.60:0/3865997928 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4002cb00 con 0x7fed480ce7f0 2023-10-16T12:33:17.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed4cf12640 1 -- 172.21.15.60:0/3865997928 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4002cc60 con 0x7fed480ce7f0 2023-10-16T12:33:17.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed4cf12640 1 -- 172.21.15.60:0/3865997928 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400343f0 con 0x7fed480ce7f0 2023-10-16T12:33:17.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 -- 172.21.15.60:0/3865997928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed480ce7f0 msgr2=0x7fed480cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 --2- 172.21.15.60:0/3865997928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed480ce7f0 0x7fed480cec80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fed40009e50 tx=0x7fed4002bf60 comp rx=0 tx=0).stop 2023-10-16T12:33:17.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 -- 172.21.15.60:0/3865997928 shutdown_connections 2023-10-16T12:33:17.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 --2- 172.21.15.60:0/3865997928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed480ce7f0 0x7fed480cec80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 --2- 172.21.15.60:0/3865997928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed480cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 -- 172.21.15.60:0/3865997928 >> 172.21.15.60:0/3865997928 conn(0x7fed48099ea0 msgr2=0x7fed480d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:17.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 -- 172.21.15.60:0/3865997928 shutdown_connections 2023-10-16T12:33:17.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.079+0000 7fed50178640 1 -- 172.21.15.60:0/3865997928 wait complete. 2023-10-16T12:33:17.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.081+0000 7fed50178640 1 Processor -- start 2023-10-16T12:33:17.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.081+0000 7fed50178640 1 -- start start 2023-10-16T12:33:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed50178640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed48149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed50178640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed480ce7f0 0x7fed4806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed50178640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 0x7fed4806b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed50178640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48149880 con 0x7fed4815f220 2023-10-16T12:33:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed50178640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481f3680 con 0x7fed480ce7f0 2023-10-16T12:33:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed50178640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed48149ce0 con 0x7fed480cd510 2023-10-16T12:33:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed4df14640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed48149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed4e715640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 0x7fed4806b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed4d713640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed480ce7f0 0x7fed4806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed4df14640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed48149340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:33820/0 (socket says 172.21.15.60:33820) 2023-10-16T12:33:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed4e715640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 0x7fed4806b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45312/0 (socket says 172.21.15.60:45312) 2023-10-16T12:33:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.082+0000 7fed4df14640 1 -- 172.21.15.60:0/4229146032 learned_addr learned my addr 172.21.15.60:0/4229146032 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4e715640 1 -- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 msgr2=0x7fed48149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4e715640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed48149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4e715640 1 -- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed480ce7f0 msgr2=0x7fed4806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4e715640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed480ce7f0 0x7fed4806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4e715640 1 -- 172.21.15.60:0/4229146032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40009b30 con 0x7fed4815f220 2023-10-16T12:33:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4df14640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed48149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed4e715640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 0x7fed4806b3f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fed40032000 tx=0x7fed40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40009510 con 0x7fed4815f220 2023-10-16T12:33:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4003e450 con 0x7fed4815f220 2023-10-16T12:33:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed48149fc0 con 0x7fed4815f220 2023-10-16T12:33:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4003e800 con 0x7fed4815f220 2023-10-16T12:33:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.083+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed481f28e0 con 0x7fed4815f220 2023-10-16T12:33:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.084+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4814a4e0 con 0x7fed4815f220 2023-10-16T12:33:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.085+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fed4003e960 con 0x7fed4815f220 2023-10-16T12:33:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.086+0000 7fed3e7fc640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fed2c06ade0 0x7fed2c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.086+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fed400abbf0 con 0x7fed4815f220 2023-10-16T12:33:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.086+0000 7fed4df14640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fed2c06ade0 0x7fed2c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.087+0000 7fed4df14640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fed2c06ade0 0x7fed2c06d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed480cda30 tx=0x7fed44009110 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.091+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed4814a4e0 con 0x7fed4815f220 2023-10-16T12:33:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.282+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed4814a4e0 con 0x7fed4815f220 2023-10-16T12:33:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.283+0000 7fed3e7fc640 1 -- 172.21.15.60:0/4229146032 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10896 (secure 0 0 0) 0x7fed4814a4e0 con 0x7fed4815f220 2023-10-16T12:33:17.285 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:17.285 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:16.511383+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fed2c06ade0 msgr2=0x7fed2c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fed2c06ade0 0x7fed2c06d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed480cda30 tx=0x7fed44009110 comp rx=0 tx=0).stop 2023-10-16T12:33:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 msgr2=0x7fed4806b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 0x7fed4806b3f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fed40032000 tx=0x7fed40004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 shutdown_connections 2023-10-16T12:33:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fed2c06ade0 0x7fed2c06d2a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed4815f220 0x7fed4806b3f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed480ce7f0 0x7fed4806aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 --2- 172.21.15.60:0/4229146032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed480cd510 0x7fed48149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 >> 172.21.15.60:0/4229146032 conn(0x7fed48099ea0 msgr2=0x7fed48159300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:17.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 shutdown_connections 2023-10-16T12:33:17.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.286+0000 7fed50178640 1 -- 172.21.15.60:0/4229146032 wait complete. 2023-10-16T12:33:17.305 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-10-16T12:33:17.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:33:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.522+0000 7f0ffaf01640 1 Processor -- start 2023-10-16T12:33:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.522+0000 7f0ffaf01640 1 -- start start 2023-10-16T12:33:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ffaf01640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0ff415d000 0x7f0ff406e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ffaf01640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40ae4f0 con 0x7f0ff4071d40 2023-10-16T12:33:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ffaf01640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff406e6f0 con 0x7f0ff406b4c0 2023-10-16T12:33:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ffaf01640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff4070f80 con 0x7f0ff415d000 2023-10-16T12:33:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ff3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0ff4071d40 0x7f0ff406adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41754/0 (socket says 172.21.15.60:41754) 2023-10-16T12:33:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ff3fff640 1 -- 172.21.15.60:0/3759354315 learned_addr learned my addr 172.21.15.60:0/3759354315 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.523+0000 7f0ff8c9d640 1 --2- 172.21.15.60:0/3759354315 >> v2:172.21.15.144:3300/0 conn(0x7f0ff415d000 0x7f0ff406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985858543 0 0) 0x7f0ff40ae4f0 con 0x7f0ff4071d40 2023-10-16T12:33:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe4003520 con 0x7f0ff4071d40 2023-10-16T12:33:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515711234 0 0) 0x7f0fe4003520 con 0x7f0ff4071d40 2023-10-16T12:33:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 >> v2:172.21.15.144:3300/0 conn(0x7f0ff415d000 msgr2=0x7f0ff406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 --2- 172.21.15.60:0/3759354315 >> v2:172.21.15.144:3300/0 conn(0x7f0ff415d000 0x7f0ff406e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 >> v1:172.21.15.144:6789/0 conn(0x7f0ff406b4c0 legacy=0x7f0ff406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:17.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4071260 con 0x7f0ff4071d40 2023-10-16T12:33:17.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0fe0001670 con 0x7f0ff4071d40 2023-10-16T12:33:17.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 >> v1:172.21.15.60:6789/0 conn(0x7f0ff4071d40 legacy=0x7f0ff406adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4003970 0x7f0fe4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fe40047c0 0x7f0fe4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe4005180 0x7f0fe400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4070f80 con 0x7f0fe4005180 2023-10-16T12:33:17.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe400c020 con 0x7f0fe40047c0 2023-10-16T12:33:17.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe400c1e0 con 0x7f0fe4003970 2023-10-16T12:33:17.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.524+0000 7f0ff949e640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe4005180 0x7f0fe400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.525+0000 7f0ff949e640 1 -- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4003970 msgr2=0x7f0fe4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:17.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.525+0000 7f0ff949e640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4003970 0x7f0fe4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.525+0000 7f0ff949e640 1 -- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fe40047c0 msgr2=0x7f0fe4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:17.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.525+0000 7f0ff3fff640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fe40047c0 0x7f0fe4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.525+0000 7f0ff949e640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fe40047c0 0x7f0fe4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.525+0000 7f0ff949e640 1 -- 172.21.15.60:0/3759354315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4003520 con 0x7f0fe4005180 2023-10-16T12:33:17.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.526+0000 7f0ff949e640 1 -- 172.21.15.60:0/3759354315 reap_dead start 2023-10-16T12:33:17.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.526+0000 7f0ff949e640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe4005180 0x7f0fe400b980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0fec004480 tx=0x7f0fe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b936e158a2ae6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:17.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.526+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec006b50 con 0x7f0fe4005180 2023-10-16T12:33:17.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.526+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec006cd0 con 0x7f0fe4005180 2023-10-16T12:33:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.526+0000 7f0ff37fe640 1 -- 172.21.15.60:0/3759354315 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec007260 con 0x7f0fe4005180 2023-10-16T12:33:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 -- 172.21.15.60:0/3759354315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe4005180 msgr2=0x7f0fe400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe4005180 0x7f0fe400b980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0fec004480 tx=0x7f0fe40057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 -- 172.21.15.60:0/3759354315 shutdown_connections 2023-10-16T12:33:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/3759354315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fe4005180 0x7f0fe400b980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 -- 172.21.15.60:0/3759354315 >> 172.21.15.60:0/3759354315 conn(0x7f0ff40b0c80 msgr2=0x7f0ff40b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:17.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 -- 172.21.15.60:0/3759354315 shutdown_connections 2023-10-16T12:33:17.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.527+0000 7f0ffaf01640 1 -- 172.21.15.60:0/3759354315 wait complete. 2023-10-16T12:33:17.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.529+0000 7f0ffaf01640 1 Processor -- start 2023-10-16T12:33:17.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.529+0000 7f0ffaf01640 1 -- start start 2023-10-16T12:33:17.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ffaf01640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4005180 0x7f0ff409fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ffaf01640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 0x7f0ff415b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ffaf01640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ff415b750 0x7f0ff415ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ffaf01640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff409fa30 con 0x7f0ff409e8d0 2023-10-16T12:33:17.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ffaf01640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff409ee20 con 0x7f0ff415b750 2023-10-16T12:33:17.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ffaf01640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff409f100 con 0x7f0fe4005180 2023-10-16T12:33:17.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 0x7f0ff415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 0x7f0ff415b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45328/0 (socket says 172.21.15.60:45328) 2023-10-16T12:33:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ff3fff640 1 -- 172.21.15.60:0/2928307787 learned_addr learned my addr 172.21.15.60:0/2928307787 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ff8c9d640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4005180 0x7f0ff409fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.530+0000 7f0ff949e640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ff415b750 0x7f0ff415ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff3fff640 1 -- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4005180 msgr2=0x7f0ff409fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff3fff640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4005180 0x7f0ff409fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff3fff640 1 -- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ff415b750 msgr2=0x7f0ff415ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff3fff640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ff415b750 0x7f0ff415ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff3fff640 1 -- 172.21.15.60:0/2928307787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff409f3e0 con 0x7f0ff409e8d0 2023-10-16T12:33:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff3fff640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 0x7f0ff415b100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000d770 tx=0x7f0fe000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00074e0 con 0x7f0ff409e8d0 2023-10-16T12:33:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0007640 con 0x7f0ff409e8d0 2023-10-16T12:33:17.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004a00 con 0x7f0ff409e8d0 2023-10-16T12:33:17.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff409f6c0 con 0x7f0ff409e8d0 2023-10-16T12:33:17.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.531+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff415f3b0 con 0x7f0ff409e8d0 2023-10-16T12:33:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.533+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff409cfc0 con 0x7f0ff409e8d0 2023-10-16T12:33:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.533+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f0fe0004b60 con 0x7f0ff409e8d0 2023-10-16T12:33:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.534+0000 7f0ff17fa640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fd806ae30 0x7f0fd806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.534+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f0fe0088c20 con 0x7f0ff409e8d0 2023-10-16T12:33:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.535+0000 7f0ff8c9d640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fd806ae30 0x7f0fd806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.536+0000 7f0ff8c9d640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fd806ae30 0x7f0fd806d2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009a50 tx=0x7f0fe8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.539+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fe0054310 con 0x7f0ff409e8d0 2023-10-16T12:33:17.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.778+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff404e160 con 0x7f0ff409e8d0 2023-10-16T12:33:17.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.808+0000 7f0ff17fa640 1 -- 172.21.15.60:0/2928307787 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7f0fe0019090 con 0x7f0ff409e8d0 2023-10-16T12:33:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:17.810 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-16T12:33:17.811 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 2 2023-10-16T12:33:17.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fd806ae30 msgr2=0x7f0fd806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fd806ae30 0x7f0fd806d2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009a50 tx=0x7f0fe8008040 comp rx=0 tx=0).stop 2023-10-16T12:33:17.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 msgr2=0x7f0ff415b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:17.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 0x7f0ff415b100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000d770 tx=0x7f0fe000dc30 comp rx=0 tx=0).stop 2023-10-16T12:33:17.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 shutdown_connections 2023-10-16T12:33:17.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fd806ae30 0x7f0fd806d2f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ff415b750 0x7f0ff415ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ff409e8d0 0x7f0ff415b100 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 --2- 172.21.15.60:0/2928307787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fe4005180 0x7f0ff409fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:17.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 >> 172.21.15.60:0/2928307787 conn(0x7f0ff40b0c80 msgr2=0x7f0ff40af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:17.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.811+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 shutdown_connections 2023-10-16T12:33:17.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:17.812+0000 7f0ffaf01640 1 -- 172.21.15.60:0/2928307787 wait complete. 2023-10-16T12:33:19.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:33:19.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:19.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:20.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.058+0000 7fd31bfa9640 1 Processor -- start 2023-10-16T12:33:20.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.058+0000 7fd31bfa9640 1 -- start start 2023-10-16T12:33:20.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd31bfa9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd31415aaa0 0x7fd31415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd31bfa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3140ce390 0x7fd314070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd31bfa9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31415ed60 con 0x7fd31415ae80 2023-10-16T12:33:20.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd31bfa9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31406b810 con 0x7fd31415aaa0 2023-10-16T12:33:20.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd31bfa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140707e0 con 0x7fd3140ce390 2023-10-16T12:33:20.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd319d45640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd31415aaa0 0x7fd31415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd319544640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd31415ae80 0x7fd3140cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55116/0 (socket says 172.21.15.60:55116) 2023-10-16T12:33:20.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.059+0000 7fd319544640 1 -- 172.21.15.60:0/4246339655 learned_addr learned my addr 172.21.15.60:0/4246339655 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:20.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.060+0000 7fd31a546640 1 --2- 172.21.15.60:0/4246339655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3140ce390 0x7fd314070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.060+0000 7fd319d45640 1 -- 172.21.15.60:0/4246339655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3140ce390 msgr2=0x7fd314070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:20.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.060+0000 7fd319d45640 1 --2- 172.21.15.60:0/4246339655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3140ce390 0x7fd314070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.060+0000 7fd319d45640 1 -- 172.21.15.60:0/4246339655 >> v1:172.21.15.60:6789/0 conn(0x7fd31415ae80 legacy=0x7fd3140cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.060+0000 7fd319d45640 1 -- 172.21.15.60:0/4246339655 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31415ef90 con 0x7fd31415aaa0 2023-10-16T12:33:20.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.061+0000 7fd319d45640 1 --2- 172.21.15.60:0/4246339655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd31415e820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd304009d90 tx=0x7fd30402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82394163f1052261 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:20.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.061+0000 7fd318d43640 1 -- 172.21.15.60:0/4246339655 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30402cc60 con 0x7fd31415aaa0 2023-10-16T12:33:20.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.061+0000 7fd318d43640 1 -- 172.21.15.60:0/4246339655 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30402cdc0 con 0x7fd31415aaa0 2023-10-16T12:33:20.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.061+0000 7fd318d43640 1 -- 172.21.15.60:0/4246339655 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd304034660 con 0x7fd31415aaa0 2023-10-16T12:33:20.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.062+0000 7fd31bfa9640 1 -- 172.21.15.60:0/4246339655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 msgr2=0x7fd31415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.062+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/4246339655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd31415e820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd304009d90 tx=0x7fd30402beb0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.063+0000 7fd31bfa9640 1 -- 172.21.15.60:0/4246339655 shutdown_connections 2023-10-16T12:33:20.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.063+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/4246339655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3140ce390 0x7fd314070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.063+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/4246339655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd31415e820 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.063+0000 7fd31bfa9640 1 -- 172.21.15.60:0/4246339655 >> 172.21.15.60:0/4246339655 conn(0x7fd3140cb410 msgr2=0x7fd3140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:20.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.063+0000 7fd31bfa9640 1 -- 172.21.15.60:0/4246339655 shutdown_connections 2023-10-16T12:33:20.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.063+0000 7fd31bfa9640 1 -- 172.21.15.60:0/4246339655 wait complete. 2023-10-16T12:33:20.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.065+0000 7fd31bfa9640 1 Processor -- start 2023-10-16T12:33:20.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.065+0000 7fd31bfa9640 1 -- start start 2023-10-16T12:33:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.066+0000 7fd31bfa9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3140ce390 0x7fd314159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.066+0000 7fd31bfa9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd314158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.066+0000 7fd31bfa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31415ae80 0x7fd3140c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.066+0000 7fd31bfa9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140c8ea0 con 0x7fd31415aaa0 2023-10-16T12:33:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.066+0000 7fd31bfa9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141592c0 con 0x7fd3140ce390 2023-10-16T12:33:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.066+0000 7fd31bfa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140c63c0 con 0x7fd31415ae80 2023-10-16T12:33:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd314158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd314158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52990/0 (socket says 172.21.15.60:52990) 2023-10-16T12:33:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 -- 172.21.15.60:0/2407283341 learned_addr learned my addr 172.21.15.60:0/2407283341 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319d45640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3140ce390 0x7fd314159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd31a546640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31415ae80 0x7fd3140c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 -- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31415ae80 msgr2=0x7fd3140c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31415ae80 0x7fd3140c1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 -- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3140ce390 msgr2=0x7fd314159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3140ce390 0x7fd314159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 -- 172.21.15.60:0/2407283341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd304009a70 con 0x7fd31415aaa0 2023-10-16T12:33:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319d45640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3140ce390 0x7fd314159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.067+0000 7fd319544640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd314158d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd30800c390 tx=0x7fd30800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.068+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308005ce0 con 0x7fd31415aaa0 2023-10-16T12:33:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.068+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308005e40 con 0x7fd31415aaa0 2023-10-16T12:33:20.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.068+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3140c66a0 con 0x7fd31415aaa0 2023-10-16T12:33:20.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.068+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308007310 con 0x7fd31415aaa0 2023-10-16T12:33:20.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.068+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3140c8ac0 con 0x7fd31415aaa0 2023-10-16T12:33:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.069+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3140c6bc0 con 0x7fd31415aaa0 2023-10-16T12:33:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.070+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fd308007470 con 0x7fd31415aaa0 2023-10-16T12:33:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.072+0000 7fd3127fc640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd30006ade0 0x7fd30006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.072+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd3080886a0 con 0x7fd31415aaa0 2023-10-16T12:33:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.072+0000 7fd319d45640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd30006ade0 0x7fd30006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.074+0000 7fd319d45640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd30006ade0 0x7fd30006d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd304036000 tx=0x7fd304030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:20.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.077+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd308053b70 con 0x7fd31415aaa0 2023-10-16T12:33:20.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.328+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3140c8fe0 con 0x7fd31415aaa0 2023-10-16T12:33:20.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.329+0000 7fd3127fc640 1 -- 172.21.15.60:0/2407283341 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7fd3140c8fe0 con 0x7fd31415aaa0 2023-10-16T12:33:20.332 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:20.332 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-16T12:33:20.332 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 3 2023-10-16T12:33:20.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd30006ade0 msgr2=0x7fd30006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd30006ade0 0x7fd30006d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd304036000 tx=0x7fd304030000 comp rx=0 tx=0).stop 2023-10-16T12:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 msgr2=0x7fd314158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd314158d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd30800c390 tx=0x7fd30800c850 comp rx=0 tx=0).stop 2023-10-16T12:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 shutdown_connections 2023-10-16T12:33:20.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd30006ade0 0x7fd30006d2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd31415ae80 0x7fd3140c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd31415aaa0 0x7fd314158d80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 --2- 172.21.15.60:0/2407283341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3140ce390 0x7fd314159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.332+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 >> 172.21.15.60:0/2407283341 conn(0x7fd3140cb410 msgr2=0x7fd3140b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:20.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.333+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 shutdown_connections 2023-10-16T12:33:20.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.333+0000 7fd31bfa9640 1 -- 172.21.15.60:0/2407283341 wait complete. 2023-10-16T12:33:20.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:20.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.553+0000 7f0bb9875640 1 Processor -- start 2023-10-16T12:33:20.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.553+0000 7f0bb9875640 1 -- start start 2023-10-16T12:33:20.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.554+0000 7f0bb9875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415aaa0 0x7f0bb415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.554+0000 7f0bb9875640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bb415ae80 0x7f0bb40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.554+0000 7f0bb9875640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb415ed60 con 0x7f0bb40ccaa0 2023-10-16T12:33:20.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.554+0000 7f0bb9875640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb406b810 con 0x7f0bb415ae80 2023-10-16T12:33:20.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.554+0000 7f0bb9875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40707e0 con 0x7f0bb415aaa0 2023-10-16T12:33:20.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bb415ae80 0x7f0bb40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bb415ae80 0x7f0bb40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52992/0 (socket says 172.21.15.60:52992) 2023-10-16T12:33:20.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 -- 172.21.15.60:0/3491445725 learned_addr learned my addr 172.21.15.60:0/3491445725 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:20.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 -- 172.21.15.60:0/3491445725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415aaa0 msgr2=0x7f0bb415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:20.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb2ffd640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415aaa0 0x7f0bb415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415aaa0 0x7f0bb415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 -- 172.21.15.60:0/3491445725 >> v1:172.21.15.144:6789/0 conn(0x7f0bb40ccaa0 legacy=0x7f0bb4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:20.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb27fc640 1 -- 172.21.15.60:0/3491445725 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb415ef90 con 0x7f0bb415ae80 2023-10-16T12:33:20.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.555+0000 7f0bb2ffd640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415aaa0 0x7f0bb415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:20.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.556+0000 7f0bb27fc640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415ae80 0x7f0bb40cc560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009870 tx=0x7f0ba802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9419d9ca0c65d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:20.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb1ffb640 1 -- 172.21.15.60:0/3491445725 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8031640 con 0x7f0bb415ae80 2023-10-16T12:33:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb1ffb640 1 -- 172.21.15.60:0/3491445725 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba80317a0 con 0x7f0bb415ae80 2023-10-16T12:33:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb1ffb640 1 -- 172.21.15.60:0/3491445725 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8031b30 con 0x7f0bb415ae80 2023-10-16T12:33:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 -- 172.21.15.60:0/3491445725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415ae80 msgr2=0x7f0bb40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415ae80 0x7f0bb40cc560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009870 tx=0x7f0ba802b8f0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 -- 172.21.15.60:0/3491445725 shutdown_connections 2023-10-16T12:33:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415ae80 0x7f0bb40cc560 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 --2- 172.21.15.60:0/3491445725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415aaa0 0x7f0bb415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 -- 172.21.15.60:0/3491445725 >> 172.21.15.60:0/3491445725 conn(0x7f0bb40cb410 msgr2=0x7f0bb40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 -- 172.21.15.60:0/3491445725 shutdown_connections 2023-10-16T12:33:20.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.557+0000 7f0bb9875640 1 -- 172.21.15.60:0/3491445725 wait complete. 2023-10-16T12:33:20.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.559+0000 7f0bb9875640 1 Processor -- start 2023-10-16T12:33:20.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.559+0000 7f0bb9875640 1 -- start start 2023-10-16T12:33:20.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb9875640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb40ccaa0 0x7f0bb40c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb9875640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 0x7f0bb4159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb9875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415ae80 0x7f0bb40c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb9875640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40c41e0 con 0x7f0bb415aaa0 2023-10-16T12:33:20.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb9875640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb40c89a0 con 0x7f0bb40ccaa0 2023-10-16T12:33:20.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb9875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41597a0 con 0x7f0bb415ae80 2023-10-16T12:33:20.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 0x7f0bb4159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 0x7f0bb4159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52998/0 (socket says 172.21.15.60:52998) 2023-10-16T12:33:20.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.560+0000 7f0bb27fc640 1 -- 172.21.15.60:0/1928080648 learned_addr learned my addr 172.21.15.60:0/1928080648 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:20.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb27fc640 1 -- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415ae80 msgr2=0x7f0bb40c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb27fc640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415ae80 0x7f0bb40c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb27fc640 1 -- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb40ccaa0 msgr2=0x7f0bb40c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb27fc640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb40ccaa0 0x7f0bb40c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb27fc640 1 -- 172.21.15.60:0/1928080648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80094d0 con 0x7f0bb415aaa0 2023-10-16T12:33:20.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb27fc640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 0x7f0bb4159260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80090b0 tx=0x7f0ba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:20.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba803bc70 con 0x7f0bb415aaa0 2023-10-16T12:33:20.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba803bdd0 con 0x7f0bb415aaa0 2023-10-16T12:33:20.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8045630 con 0x7f0bb415aaa0 2023-10-16T12:33:20.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.561+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb40c3840 con 0x7f0bb415aaa0 2023-10-16T12:33:20.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.562+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb40c3e00 con 0x7f0bb415aaa0 2023-10-16T12:33:20.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.563+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb40c4320 con 0x7f0bb415aaa0 2023-10-16T12:33:20.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.564+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f0ba802f050 con 0x7f0bb415aaa0 2023-10-16T12:33:20.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.564+0000 7f0b9bfff640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0ba006ade0 0x7f0ba006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:20.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.565+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0ba80ab2d0 con 0x7f0bb415aaa0 2023-10-16T12:33:20.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.565+0000 7f0bb2ffd640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0ba006ade0 0x7f0ba006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:20.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.566+0000 7f0bb2ffd640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0ba006ade0 0x7f0ba006d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006fd0 tx=0x7f0ba4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:20.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.569+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bb40c4320 con 0x7f0bb415aaa0 2023-10-16T12:33:20.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.771+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bb40c4320 con 0x7f0bb415aaa0 2023-10-16T12:33:20.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.773+0000 7f0b9bfff640 1 -- 172.21.15.60:0/1928080648 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f0bb40c4320 con 0x7f0bb415aaa0 2023-10-16T12:33:20.776 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:20.776 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:20.531656+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:20.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.775+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0ba006ade0 msgr2=0x7f0ba006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.775+0000 7f0bb9875640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0ba006ade0 0x7f0ba006d2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006fd0 tx=0x7f0ba4008040 comp rx=0 tx=0).stop 2023-10-16T12:33:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.775+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 msgr2=0x7f0bb4159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.775+0000 7f0bb9875640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 0x7f0bb4159260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80090b0 tx=0x7f0ba80046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 shutdown_connections 2023-10-16T12:33:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0ba006ade0 0x7f0ba006d2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb415ae80 0x7f0bb40c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb415aaa0 0x7f0bb4159260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 --2- 172.21.15.60:0/1928080648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb40ccaa0 0x7f0bb40c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 >> 172.21.15.60:0/1928080648 conn(0x7f0bb40cb410 msgr2=0x7f0bb40b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 shutdown_connections 2023-10-16T12:33:20.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:20.776+0000 7f0bb9875640 1 -- 172.21.15.60:0/1928080648 wait complete. 2023-10-16T12:33:20.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:33:20.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:33:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.005+0000 7f7cd50a2640 1 Processor -- start 2023-10-16T12:33:21.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.005+0000 7f7cd50a2640 1 -- start start 2023-10-16T12:33:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cd50a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7cd00cf710 0x7f7cd00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cd50a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cd00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cd50a2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd00d3980 con 0x7f7cd00b38e0 2023-10-16T12:33:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cd50a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd00d3c60 con 0x7f7cd00cf710 2023-10-16T12:33:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cd50a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00ce630 con 0x7f7cd00cfaf0 2023-10-16T12:33:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7ccf59e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7cd00b38e0 0x7f7cd00d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55120/0 (socket says 172.21.15.60:55120) 2023-10-16T12:33:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7ccf59e640 1 -- 172.21.15.60:0/1471521170 learned_addr learned my addr 172.21.15.60:0/1471521170 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cce59c640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cd00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.007+0000 7f7cced9d640 1 --2- 172.21.15.60:0/1471521170 >> v2:172.21.15.144:3300/0 conn(0x7f7cd00cf710 0x7f7cd00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267247384 0 0) 0x7f7cd00d3980 con 0x7f7cd00b38e0 2023-10-16T12:33:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb8003520 con 0x7f7cd00b38e0 2023-10-16T12:33:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 371151293 0 0) 0x7f7cb8003520 con 0x7f7cd00b38e0 2023-10-16T12:33:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 msgr2=0x7f7cd00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cd00b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 >> v2:172.21.15.144:3300/0 conn(0x7f7cd00cf710 msgr2=0x7f7cd00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 --2- 172.21.15.60:0/1471521170 >> v2:172.21.15.144:3300/0 conn(0x7f7cd00cf710 0x7f7cd00b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd00ce910 con 0x7f7cd00b38e0 2023-10-16T12:33:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7cc80019f0 con 0x7f7cd00b38e0 2023-10-16T12:33:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 >> v1:172.21.15.60:6789/0 conn(0x7f7cd00b38e0 legacy=0x7f7cd00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cb8003970 0x7f7cb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00d3c60 con 0x7f7cb8004d00 2023-10-16T12:33:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00ce630 con 0x7f7cb8003970 2023-10-16T12:33:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.008+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb800b980 con 0x7f7cd00cfaf0 2023-10-16T12:33:21.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7ccf59e640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7cced9d640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cb8003970 0x7f7cb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7ccf59e640 1 -- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 msgr2=0x7f7cb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7cce59c640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7ccf59e640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cb80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7ccf59e640 1 -- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cb8003970 msgr2=0x7f7cb8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7ccf59e640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cb8003970 0x7f7cb8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7ccf59e640 1 -- 172.21.15.60:0/1471521170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8003520 con 0x7f7cb8004d00 2023-10-16T12:33:21.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7cced9d640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cb8003970 0x7f7cb8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:21.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.009+0000 7f7cce59c640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00cfaf0 0x7f7cb80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.010+0000 7f7ccf59e640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cb8005180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800a830 tx=0x7f7cb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e339e7d050602d56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.010+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8034630 con 0x7f7cb8004d00 2023-10-16T12:33:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.010+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc802f070 con 0x7f7cb8004d00 2023-10-16T12:33:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.010+0000 7f7ccdd9b640 1 -- 172.21.15.60:0/1471521170 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc80349f0 con 0x7f7cb8004d00 2023-10-16T12:33:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7cd50a2640 1 -- 172.21.15.60:0/1471521170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 msgr2=0x7f7cb8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/1471521170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cb8005180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800a830 tx=0x7f7cb80057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7ccf59e640 1 -- 172.21.15.60:0/1471521170 reap_dead start 2023-10-16T12:33:21.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7cd50a2640 1 -- 172.21.15.60:0/1471521170 shutdown_connections 2023-10-16T12:33:21.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7cd50a2640 1 -- 172.21.15.60:0/1471521170 >> 172.21.15.60:0/1471521170 conn(0x7f7cd00d2cc0 msgr2=0x7f7cd00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:21.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7cd50a2640 1 -- 172.21.15.60:0/1471521170 shutdown_connections 2023-10-16T12:33:21.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.011+0000 7f7cd50a2640 1 -- 172.21.15.60:0/1471521170 wait complete. 2023-10-16T12:33:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.013+0000 7f7cd50a2640 1 Processor -- start 2023-10-16T12:33:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.013+0000 7f7cd50a2640 1 -- start start 2023-10-16T12:33:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cd50a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cd01f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cd50a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd0150fa0 0x7f7cd00b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cd50a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00b4cf0 0x7f7cd00b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cd50a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd015d660 con 0x7f7cb8004d00 2023-10-16T12:33:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cd50a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00b5170 con 0x7f7cd0150fa0 2023-10-16T12:33:21.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cd50a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00b0620 con 0x7f7cd00b4cf0 2023-10-16T12:33:21.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cced9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cd01f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cced9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cd01f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53006/0 (socket says 172.21.15.60:53006) 2023-10-16T12:33:21.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.014+0000 7f7cced9d640 1 -- 172.21.15.60:0/517608692 learned_addr learned my addr 172.21.15.60:0/517608692 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:21.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7ccf59e640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00b4cf0 0x7f7cd00b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cce59c640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd0150fa0 0x7f7cd00b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cced9d640 1 -- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00b4cf0 msgr2=0x7f7cd00b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cced9d640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00b4cf0 0x7f7cd00b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cced9d640 1 -- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd0150fa0 msgr2=0x7f7cd00b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cced9d640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd0150fa0 0x7f7cd00b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cced9d640 1 -- 172.21.15.60:0/517608692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd00afa10 con 0x7f7cb8004d00 2023-10-16T12:33:21.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cce59c640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd0150fa0 0x7f7cd00b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:21.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cced9d640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cd01f2370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0010950 tx=0x7f7cc0010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.015+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000ec20 con 0x7f7cb8004d00 2023-10-16T12:33:21.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.016+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc000ed80 con 0x7f7cb8004d00 2023-10-16T12:33:21.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.016+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0004220 con 0x7f7cb8004d00 2023-10-16T12:33:21.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.016+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd00afcf0 con 0x7f7cb8004d00 2023-10-16T12:33:21.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.016+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd00b02b0 con 0x7f7cb8004d00 2023-10-16T12:33:21.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.017+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd00b0470 con 0x7f7cb8004d00 2023-10-16T12:33:21.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.018+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f7cc0004380 con 0x7f7cb8004d00 2023-10-16T12:33:21.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.019+0000 7f7cb77fe640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cbc06ae10 0x7f7cbc06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.019+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7cc00895e0 con 0x7f7cb8004d00 2023-10-16T12:33:21.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.020+0000 7f7cce59c640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cbc06ae10 0x7f7cbc06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.021+0000 7f7cce59c640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cbc06ae10 0x7f7cbc06d2d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7cd006c230 tx=0x7f7cc4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.024+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7cc0054990 con 0x7f7cb8004d00 2023-10-16T12:33:21.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.221+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7cd004e160 con 0x7f7cb8004d00 2023-10-16T12:33:21.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.221+0000 7f7cb77fe640 1 -- 172.21.15.60:0/517608692 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7cc00210a0 con 0x7f7cb8004d00 2023-10-16T12:33:21.224 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:33:21.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cbc06ae10 msgr2=0x7f7cbc06d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cbc06ae10 0x7f7cbc06d2d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7cd006c230 tx=0x7f7cc4008040 comp rx=0 tx=0).stop 2023-10-16T12:33:21.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 msgr2=0x7f7cd01f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cd01f2370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0010950 tx=0x7f7cc0010e10 comp rx=0 tx=0).stop 2023-10-16T12:33:21.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 shutdown_connections 2023-10-16T12:33:21.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cbc06ae10 0x7f7cbc06d2d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7cd00b4cf0 0x7f7cd00b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd0150fa0 0x7f7cd00b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 --2- 172.21.15.60:0/517608692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb8004d00 0x7f7cd01f2370 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 >> 172.21.15.60:0/517608692 conn(0x7f7cd00d2cc0 msgr2=0x7f7cd00b7300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:21.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 shutdown_connections 2023-10-16T12:33:21.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.224+0000 7f7cd50a2640 1 -- 172.21.15.60:0/517608692 wait complete. 2023-10-16T12:33:21.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:33:21.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.447+0000 7f701015a640 1 Processor -- start 2023-10-16T12:33:21.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.447+0000 7f701015a640 1 -- start start 2023-10-16T12:33:21.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.448+0000 7f701015a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 0x7f70081615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.448+0000 7f701015a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f700815db00 0x7f70080b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.448+0000 7f701015a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70080b0db0 con 0x7f70080b0410 2023-10-16T12:33:21.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.448+0000 7f701015a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700815e3e0 con 0x7f700815db00 2023-10-16T12:33:21.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.448+0000 7f701015a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080cd4a0 con 0x7f70080cd0c0 2023-10-16T12:33:21.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700d6f5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f70080b0410 0x7f700815d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55136/0 (socket says 172.21.15.60:55136) 2023-10-16T12:33:21.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700d6f5640 1 -- 172.21.15.60:0/1862626928 learned_addr learned my addr 172.21.15.60:0/1862626928 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:21.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700def6640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 0x7f70081615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700e6f7640 1 --2- 172.21.15.60:0/1862626928 >> v2:172.21.15.144:3300/0 conn(0x7f700815db00 0x7f70080b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451968457 0 0) 0x7f70080b0db0 con 0x7f70080b0410 2023-10-16T12:33:21.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff0003520 con 0x7f70080b0410 2023-10-16T12:33:21.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 109672464 0 0) 0x7f6ff0003520 con 0x7f70080b0410 2023-10-16T12:33:21.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.449+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 msgr2=0x7f70081615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 0x7f70081615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 >> v2:172.21.15.144:3300/0 conn(0x7f700815db00 msgr2=0x7f70080b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 --2- 172.21.15.60:0/1862626928 >> v2:172.21.15.144:3300/0 conn(0x7f700815db00 0x7f70080b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700815e100 con 0x7f70080b0410 2023-10-16T12:33:21.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f6ffc001670 con 0x7f70080b0410 2023-10-16T12:33:21.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 >> v1:172.21.15.60:6789/0 conn(0x7f70080b0410 legacy=0x7f700815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 0x7f6ff0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff0003f40 0x7f6ff00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0004d00 0x7f6ff0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700815e3e0 con 0x7f6ff0003f40 2023-10-16T12:33:21.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080cd4a0 con 0x7f6ff0004d00 2023-10-16T12:33:21.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000ba30 con 0x7f70080cd0c0 2023-10-16T12:33:21.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700d6f5640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff0003f40 0x7f6ff00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700def6640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 0x7f6ff0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.450+0000 7f700e6f7640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0004d00 0x7f6ff0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.451+0000 7f700d6f5640 1 -- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 msgr2=0x7f6ff0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.451+0000 7f700d6f5640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080cd0c0 0x7f6ff0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.451+0000 7f700d6f5640 1 -- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0004d00 msgr2=0x7f6ff0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.451+0000 7f700d6f5640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0004d00 0x7f6ff0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.451+0000 7f700d6f5640 1 -- 172.21.15.60:0/1862626928 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0003520 con 0x7f6ff0003f40 2023-10-16T12:33:21.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.452+0000 7f700d6f5640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff0003f40 0x7f6ff00047c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00c160 tx=0x7f6ff00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463d0c430aed2303 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.452+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc005ad0 con 0x7f6ff0003f40 2023-10-16T12:33:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.452+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc005c30 con 0x7f6ff0003f40 2023-10-16T12:33:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.452+0000 7f700cef4640 1 -- 172.21.15.60:0/1862626928 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc02c810 con 0x7f6ff0003f40 2023-10-16T12:33:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f701015a640 1 -- 172.21.15.60:0/1862626928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff0003f40 msgr2=0x7f6ff00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f701015a640 1 --2- 172.21.15.60:0/1862626928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff0003f40 0x7f6ff00047c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00c160 tx=0x7f6ff00057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f700e6f7640 1 -- 172.21.15.60:0/1862626928 reap_dead start 2023-10-16T12:33:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f701015a640 1 -- 172.21.15.60:0/1862626928 shutdown_connections 2023-10-16T12:33:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f701015a640 1 -- 172.21.15.60:0/1862626928 >> 172.21.15.60:0/1862626928 conn(0x7f70080cb410 msgr2=0x7f70080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f701015a640 1 -- 172.21.15.60:0/1862626928 shutdown_connections 2023-10-16T12:33:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.453+0000 7f701015a640 1 -- 172.21.15.60:0/1862626928 wait complete. 2023-10-16T12:33:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.455+0000 7f701015a640 1 Processor -- start 2023-10-16T12:33:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.455+0000 7f701015a640 1 -- start start 2023-10-16T12:33:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f701015a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0003f40 0x7f700814ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f701015a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080aebb0 0x7f70080af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f701015a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 0x7f70080b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f701015a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700815c6f0 con 0x7f70080b3630 2023-10-16T12:33:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f701015a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080b3ff0 con 0x7f6ff0003f40 2023-10-16T12:33:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f701015a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080ae720 con 0x7f70080aebb0 2023-10-16T12:33:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f700e6f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 0x7f70080b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f700e6f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 0x7f70080b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53032/0 (socket says 172.21.15.60:53032) 2023-10-16T12:33:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f700e6f7640 1 -- 172.21.15.60:0/2754273085 learned_addr learned my addr 172.21.15.60:0/2754273085 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f700d6f5640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080aebb0 0x7f70080af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.456+0000 7f700def6640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0003f40 0x7f700814ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f700e6f7640 1 -- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080aebb0 msgr2=0x7f70080af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f700e6f7640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080aebb0 0x7f70080af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f700e6f7640 1 -- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0003f40 msgr2=0x7f700814ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f700e6f7640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0003f40 0x7f700814ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f700e6f7640 1 -- 172.21.15.60:0/2754273085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70080adb50 con 0x7f70080b3630 2023-10-16T12:33:21.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f700e6f7640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 0x7f70080b3ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f700815afc0 tx=0x7f7000011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000004500 con 0x7f70080b3630 2023-10-16T12:33:21.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70080addf0 con 0x7f70080b3630 2023-10-16T12:33:21.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.457+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70080ae3b0 con 0x7f70080b3630 2023-10-16T12:33:21.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.458+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000003610 con 0x7f70080b3630 2023-10-16T12:33:21.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.458+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000039c0 con 0x7f70080b3630 2023-10-16T12:33:21.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.459+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe8005250 con 0x7f70080b3630 2023-10-16T12:33:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.460+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f7000003b20 con 0x7f70080b3630 2023-10-16T12:33:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.461+0000 7f70067fc640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ff406ade0 0x7f6ff406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.461+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f70000893e0 con 0x7f70080b3630 2023-10-16T12:33:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.461+0000 7f700def6640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ff406ade0 0x7f6ff406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.462+0000 7f700def6640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ff406ade0 0x7f6ff406d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8007440 tx=0x7f6ff800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.465+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70000548c0 con 0x7f70080b3630 2023-10-16T12:33:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.668+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6fe8005500 con 0x7f70080b3630 2023-10-16T12:33:21.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.669+0000 7f70067fc640 1 -- 172.21.15.60:0/2754273085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7000057ee0 con 0x7f70080b3630 2023-10-16T12:33:21.671 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:21.672 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:33:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:33:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:33:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:33:21.672 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:33:21.673 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:33:21.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.671+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ff406ade0 msgr2=0x7f6ff406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.671+0000 7f701015a640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ff406ade0 0x7f6ff406d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8007440 tx=0x7f6ff800b040 comp rx=0 tx=0).stop 2023-10-16T12:33:21.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.671+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 msgr2=0x7f70080b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.671+0000 7f701015a640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 0x7f70080b3ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f700815afc0 tx=0x7f7000011920 comp rx=0 tx=0).stop 2023-10-16T12:33:21.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.671+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 shutdown_connections 2023-10-16T12:33:21.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ff406ade0 0x7f6ff406d2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f70080aebb0 0x7f70080af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70080b3630 0x7f70080b3ab0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 --2- 172.21.15.60:0/2754273085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ff0003f40 0x7f700814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 >> 172.21.15.60:0/2754273085 conn(0x7f70080cb410 msgr2=0x7f70080ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:21.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 shutdown_connections 2023-10-16T12:33:21.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.672+0000 7f701015a640 1 -- 172.21.15.60:0/2754273085 wait complete. 2023-10-16T12:33:21.694 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:33:21.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:33:21.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.927+0000 7f0b4c98e640 1 Processor -- start 2023-10-16T12:33:21.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.927+0000 7f0b4c98e640 1 -- start start 2023-10-16T12:33:21.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.928+0000 7f0b4c98e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b48159bd0 0x7f0b48159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.928+0000 7f0b4c98e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806b3e0 0x7f0b4806cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.928+0000 7f0b4c98e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b4806d2c0 0x7f0b4815d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.928+0000 7f0b4c98e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b480c5740 con 0x7f0b4806d2c0 2023-10-16T12:33:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.928+0000 7f0b4c98e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4806d750 con 0x7f0b48159bd0 2023-10-16T12:33:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.928+0000 7f0b4c98e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4815a810 con 0x7f0b4806b3e0 2023-10-16T12:33:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b4806d2c0 0x7f0b4815d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b477fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b48159bd0 0x7f0b48159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b4806d2c0 0x7f0b4815d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53036/0 (socket says 172.21.15.60:53036) 2023-10-16T12:33:21.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 -- 172.21.15.60:0/3110457076 learned_addr learned my addr 172.21.15.60:0/3110457076 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:21.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b3ffff640 1 --2- 172.21.15.60:0/3110457076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806b3e0 0x7f0b4806cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 -- 172.21.15.60:0/3110457076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806b3e0 msgr2=0x7f0b4806cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 --2- 172.21.15.60:0/3110457076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806b3e0 0x7f0b4806cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 -- 172.21.15.60:0/3110457076 >> v2:172.21.15.144:3300/0 conn(0x7f0b48159bd0 msgr2=0x7f0b48159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:21.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.929+0000 7f0b47fff640 1 --2- 172.21.15.60:0/3110457076 >> v2:172.21.15.144:3300/0 conn(0x7f0b48159bd0 0x7f0b48159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.930+0000 7f0b47fff640 1 -- 172.21.15.60:0/3110457076 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4815aaf0 con 0x7f0b4806d2c0 2023-10-16T12:33:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.931+0000 7f0b47fff640 1 --2- 172.21.15.60:0/3110457076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b4806d2c0 0x7f0b4815d5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b480743c0 tx=0x7f0b4002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342394f947857567 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.931+0000 7f0b46ffd640 1 -- 172.21.15.60:0/3110457076 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4002c9e0 con 0x7f0b4806d2c0 2023-10-16T12:33:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.931+0000 7f0b46ffd640 1 -- 172.21.15.60:0/3110457076 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4002cb40 con 0x7f0b4806d2c0 2023-10-16T12:33:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.931+0000 7f0b46ffd640 1 -- 172.21.15.60:0/3110457076 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400343f0 con 0x7f0b4806d2c0 2023-10-16T12:33:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.931+0000 7f0b4c98e640 1 -- 172.21.15.60:0/3110457076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b4806d2c0 msgr2=0x7f0b4815d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.931+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/3110457076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b4806d2c0 0x7f0b4815d5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b480743c0 tx=0x7f0b4002be40 comp rx=0 tx=0).stop 2023-10-16T12:33:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 -- 172.21.15.60:0/3110457076 shutdown_connections 2023-10-16T12:33:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/3110457076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b4806d2c0 0x7f0b4815d5a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/3110457076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806b3e0 0x7f0b4806cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/3110457076 >> v2:172.21.15.144:3300/0 conn(0x7f0b48159bd0 0x7f0b48159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 -- 172.21.15.60:0/3110457076 >> 172.21.15.60:0/3110457076 conn(0x7f0b480ce820 msgr2=0x7f0b480cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 -- 172.21.15.60:0/3110457076 shutdown_connections 2023-10-16T12:33:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.932+0000 7f0b4c98e640 1 -- 172.21.15.60:0/3110457076 wait complete. 2023-10-16T12:33:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.934+0000 7f0b4c98e640 1 Processor -- start 2023-10-16T12:33:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.934+0000 7f0b4c98e640 1 -- start start 2023-10-16T12:33:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b4c98e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b4806b3e0 0x7f0b48150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b4c98e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806d2c0 0x7f0b48158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b4c98e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 0x7f0b48156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b4c98e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4806cf90 con 0x7f0b48159bd0 2023-10-16T12:33:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b4c98e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b48151320 con 0x7f0b4806b3e0 2023-10-16T12:33:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b4c98e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4806bc50 con 0x7f0b4806d2c0 2023-10-16T12:33:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b47fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 0x7f0b48156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b47fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 0x7f0b48156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53038/0 (socket says 172.21.15.60:53038) 2023-10-16T12:33:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b47fff640 1 -- 172.21.15.60:0/2233293617 learned_addr learned my addr 172.21.15.60:0/2233293617 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b477fe640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b4806b3e0 0x7f0b48150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.935+0000 7f0b3ffff640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806d2c0 0x7f0b48158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b47fff640 1 -- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806d2c0 msgr2=0x7f0b48158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b47fff640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806d2c0 0x7f0b48158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b47fff640 1 -- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b4806b3e0 msgr2=0x7f0b48150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b47fff640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b4806b3e0 0x7f0b48150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b47fff640 1 -- 172.21.15.60:0/2233293617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b40009b30 con 0x7f0b48159bd0 2023-10-16T12:33:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b47fff640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 0x7f0b48156ef0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b40036040 tx=0x7f0b40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40009510 con 0x7f0b48159bd0 2023-10-16T12:33:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4003e450 con 0x7f0b48159bd0 2023-10-16T12:33:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.936+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4806bef0 con 0x7f0b48159bd0 2023-10-16T12:33:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.937+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4003e800 con 0x7f0b48159bd0 2023-10-16T12:33:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.937+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4806c4b0 con 0x7f0b48159bd0 2023-10-16T12:33:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.938+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4804e9b0 con 0x7f0b48159bd0 2023-10-16T12:33:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.939+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f0b4003e960 con 0x7f0b48159bd0 2023-10-16T12:33:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.940+0000 7f0b44ff9640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0b2806ade0 0x7f0b2806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.940+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0b400abd30 con 0x7f0b48159bd0 2023-10-16T12:33:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.941+0000 7f0b477fe640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0b2806ade0 0x7f0b2806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.941+0000 7f0b477fe640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0b2806ade0 0x7f0b2806d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009ac0 tx=0x7f0b38009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:21.945+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b400772b0 con 0x7f0b48159bd0 2023-10-16T12:33:22.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.144+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0b480cf730 con 0x7f0b48159bd0 2023-10-16T12:33:22.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.579+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0b4007fd50 con 0x7f0b48159bd0 2023-10-16T12:33:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.937+0000 7f0b44ff9640 1 -- 172.21.15.60:0/2233293617 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0b40092400 con 0x7f0b48159bd0 2023-10-16T12:33:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:33:22.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0b2806ade0 msgr2=0x7f0b2806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:22.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0b2806ade0 0x7f0b2806d2a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009ac0 tx=0x7f0b38009190 comp rx=0 tx=0).stop 2023-10-16T12:33:22.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 msgr2=0x7f0b48156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:22.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 0x7f0b48156ef0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b40036040 tx=0x7f0b40004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:22.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 shutdown_connections 2023-10-16T12:33:22.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0b2806ade0 0x7f0b2806d2a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:22.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0b4806d2c0 0x7f0b48158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:22.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b48159bd0 0x7f0b48156ef0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:22.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 --2- 172.21.15.60:0/2233293617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b4806b3e0 0x7f0b48150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:22.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.941+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 >> 172.21.15.60:0/2233293617 conn(0x7f0b480ce820 msgr2=0x7f0b480af330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:22.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.942+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 shutdown_connections 2023-10-16T12:33:22.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:22.942+0000 7f0b4c98e640 1 -- 172.21.15.60:0/2233293617 wait complete. 2023-10-16T12:33:22.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:33:23.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.186+0000 7fa29759e640 1 Processor -- start 2023-10-16T12:33:23.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.186+0000 7fa29759e640 1 -- start start 2023-10-16T12:33:23.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.187+0000 7fa29759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa29815ac70 0x7fa2980cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.187+0000 7fa29759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2980cc850 con 0x7fa29806a4b0 2023-10-16T12:33:23.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.187+0000 7fa29759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2980ccb30 con 0x7fa2980b4160 2023-10-16T12:33:23.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.187+0000 7fa29759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2980cce10 con 0x7fa29815ac70 2023-10-16T12:33:23.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.187+0000 7fa296d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa29806a4b0 0x7fa29806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55138/0 (socket says 172.21.15.60:55138) 2023-10-16T12:33:23.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.187+0000 7fa296d9d640 1 -- 172.21.15.60:0/4068070084 learned_addr learned my addr 172.21.15.60:0/4068070084 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177316459 0 0) 0x7fa2980cc850 con 0x7fa29806a4b0 2023-10-16T12:33:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa280003520 con 0x7fa29806a4b0 2023-10-16T12:33:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686023939 0 0) 0x7fa280003520 con 0x7fa29806a4b0 2023-10-16T12:33:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 >> v2:172.21.15.144:3300/0 conn(0x7fa29815ac70 msgr2=0x7fa2980cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 --2- 172.21.15.60:0/4068070084 >> v2:172.21.15.144:3300/0 conn(0x7fa29815ac70 0x7fa2980cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 >> v1:172.21.15.144:6789/0 conn(0x7fa2980b4160 legacy=0x7fa298069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2980cd0f0 con 0x7fa29806a4b0 2023-10-16T12:33:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa2900019f0 con 0x7fa29806a4b0 2023-10-16T12:33:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 >> v1:172.21.15.60:6789/0 conn(0x7fa29806a4b0 legacy=0x7fa29806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa280003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2800047c0 0x7fa280004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2800052a0 0x7fa28000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980cce10 con 0x7fa2800039c0 2023-10-16T12:33:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28000c1f0 con 0x7fa2800052a0 2023-10-16T12:33:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.188+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28000c3b0 con 0x7fa2800047c0 2023-10-16T12:33:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa29659c640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa280003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa296d9d640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2800052a0 0x7fa28000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa295d9b640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2800047c0 0x7fa280004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa29659c640 1 -- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2800047c0 msgr2=0x7fa280004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa29659c640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2800047c0 0x7fa280004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa29659c640 1 -- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2800052a0 msgr2=0x7fa28000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa29659c640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2800052a0 0x7fa28000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa29659c640 1 -- 172.21.15.60:0/4068070084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280003520 con 0x7fa2800039c0 2023-10-16T12:33:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa296d9d640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2800052a0 0x7fa28000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.189+0000 7fa296d9d640 1 -- 172.21.15.60:0/4068070084 reap_dead start 2023-10-16T12:33:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.190+0000 7fa29659c640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa280003dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa288006fd0 tx=0x7fa2800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22c3a6a02afd42e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.190+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288032600 con 0x7fa2800039c0 2023-10-16T12:33:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.190+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28802e050 con 0x7fa2800039c0 2023-10-16T12:33:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.190+0000 7fa29559a640 1 -- 172.21.15.60:0/4068070084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2880329c0 con 0x7fa2800039c0 2023-10-16T12:33:23.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 -- 172.21.15.60:0/4068070084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 msgr2=0x7fa280003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:23.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa280003dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa288006fd0 tx=0x7fa2800057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 -- 172.21.15.60:0/4068070084 shutdown_connections 2023-10-16T12:33:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 --2- 172.21.15.60:0/4068070084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa280003dc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 -- 172.21.15.60:0/4068070084 >> 172.21.15.60:0/4068070084 conn(0x7fa2980ce6a0 msgr2=0x7fa2980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 -- 172.21.15.60:0/4068070084 shutdown_connections 2023-10-16T12:33:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.191+0000 7fa29759e640 1 -- 172.21.15.60:0/4068070084 wait complete. 2023-10-16T12:33:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.193+0000 7fa29759e640 1 Processor -- start 2023-10-16T12:33:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.193+0000 7fa29759e640 1 -- start start 2023-10-16T12:33:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa2981ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2980c2190 0x7fa2981ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29815b050 0x7fa29815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980c4350 con 0x7fa2800039c0 2023-10-16T12:33:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2981edee0 con 0x7fa2980c2190 2023-10-16T12:33:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980c3cd0 con 0x7fa29815b050 2023-10-16T12:33:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa2981ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa2981ed9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53066/0 (socket says 172.21.15.60:53066) 2023-10-16T12:33:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.194+0000 7fa29659c640 1 -- 172.21.15.60:0/426739478 learned_addr learned my addr 172.21.15.60:0/426739478 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa296d9d640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29815b050 0x7fa29815b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa295d9b640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2980c2190 0x7fa2981ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29659c640 1 -- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29815b050 msgr2=0x7fa29815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29659c640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29815b050 0x7fa29815b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29659c640 1 -- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2980c2190 msgr2=0x7fa2981ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29659c640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2980c2190 0x7fa2981ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29659c640 1 -- 172.21.15.60:0/426739478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29815da50 con 0x7fa2800039c0 2023-10-16T12:33:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa296d9d640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29815b050 0x7fa29815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29659c640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa2981ed9a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa28802bb50 tx=0x7fa2880047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28803c880 con 0x7fa2800039c0 2023-10-16T12:33:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28802be90 con 0x7fa2800039c0 2023-10-16T12:33:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28803cbe0 con 0x7fa2800039c0 2023-10-16T12:33:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.195+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29815dd30 con 0x7fa2800039c0 2023-10-16T12:33:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.196+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2981eeef0 con 0x7fa2800039c0 2023-10-16T12:33:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.197+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29815df70 con 0x7fa2800039c0 2023-10-16T12:33:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.198+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa288039040 con 0x7fa2800039c0 2023-10-16T12:33:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.199+0000 7fa286ffd640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa27806aea0 0x7fa27806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.200+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa2880aab20 con 0x7fa2800039c0 2023-10-16T12:33:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.200+0000 7fa295d9b640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa27806aea0 0x7fa27806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.201+0000 7fa295d9b640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa27806aea0 0x7fa27806d360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa2980c1fb0 tx=0x7fa28c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.205+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa29815df70 con 0x7fa2800039c0 2023-10-16T12:33:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.402+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa29815df70 con 0x7fa2800039c0 2023-10-16T12:33:23.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.944+0000 7fa286ffd640 1 -- 172.21.15.60:0/426739478 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa29815df70 con 0x7fa2800039c0 2023-10-16T12:33:23.946 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:33:23.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.947+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa27806aea0 msgr2=0x7fa27806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:23.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.947+0000 7fa29759e640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa27806aea0 0x7fa27806d360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa2980c1fb0 tx=0x7fa28c009040 comp rx=0 tx=0).stop 2023-10-16T12:33:23.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.947+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 msgr2=0x7fa2981ed9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.947+0000 7fa29759e640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa2981ed9a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa28802bb50 tx=0x7fa2880047d0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 shutdown_connections 2023-10-16T12:33:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa27806aea0 0x7fa27806d360 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29815b050 0x7fa29815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2980c2190 0x7fa2981ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 --2- 172.21.15.60:0/426739478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2800039c0 0x7fa2981ed9a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 >> 172.21.15.60:0/426739478 conn(0x7fa2980ce6a0 msgr2=0x7fa2980c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 shutdown_connections 2023-10-16T12:33:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:23.948+0000 7fa29759e640 1 -- 172.21.15.60:0/426739478 wait complete. 2023-10-16T12:33:23.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:33:24.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.176+0000 7fa050e1a640 1 Processor -- start 2023-10-16T12:33:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.176+0000 7fa050e1a640 1 -- start start 2023-10-16T12:33:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa050e1a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa04c15c9f0 0x7fa04c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa050e1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 0x7fa04c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa050e1a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c160a90 con 0x7fa04c0c9230 2023-10-16T12:33:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa050e1a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c0cdec0 con 0x7fa04c15c9f0 2023-10-16T12:33:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa050e1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0cd900 con 0x7fa04c15dde0 2023-10-16T12:33:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa042ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa04c0c9230 0x7fa04c0709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55142/0 (socket says 172.21.15.60:55142) 2023-10-16T12:33:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa042ffd640 1 -- 172.21.15.60:0/4169811902 learned_addr learned my addr 172.21.15.60:0/4169811902 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa04b7fe640 1 --2- 172.21.15.60:0/4169811902 >> v2:172.21.15.144:3300/0 conn(0x7fa04c15c9f0 0x7fa04c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.178+0000 7fa04bfff640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 0x7fa04c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16467921 0 0) 0x7fa04c160a90 con 0x7fa04c0c9230 2023-10-16T12:33:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa038003520 con 0x7fa04c0c9230 2023-10-16T12:33:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 842456161 0 0) 0x7fa038003520 con 0x7fa04c0c9230 2023-10-16T12:33:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 msgr2=0x7fa04c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 0x7fa04c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 >> v2:172.21.15.144:3300/0 conn(0x7fa04c15c9f0 msgr2=0x7fa04c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 --2- 172.21.15.60:0/4169811902 >> v2:172.21.15.144:3300/0 conn(0x7fa04c15c9f0 0x7fa04c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c0cdbe0 con 0x7fa04c0c9230 2023-10-16T12:33:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa034001670 con 0x7fa04c0c9230 2023-10-16T12:33:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 >> v1:172.21.15.60:6789/0 conn(0x7fa04c0c9230 legacy=0x7fa04c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa038003970 0x7fa038003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 0x7fa0380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa038005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0cd900 con 0x7fa038004d00 2023-10-16T12:33:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03800b980 con 0x7fa038003970 2023-10-16T12:33:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03800bac0 con 0x7fa04c15dde0 2023-10-16T12:33:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.179+0000 7fa04bfff640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa038005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa04b7fe640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa038003970 0x7fa038003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa042ffd640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 0x7fa0380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa04bfff640 1 -- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 msgr2=0x7fa0380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa04bfff640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15dde0 0x7fa0380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa04bfff640 1 -- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa038003970 msgr2=0x7fa038003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa04bfff640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa038003970 0x7fa038003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.180+0000 7fa04bfff640 1 -- 172.21.15.60:0/4169811902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa038003520 con 0x7fa038004d00 2023-10-16T12:33:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.181+0000 7fa04bfff640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa038005180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa044009a90 tx=0x7fa0380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f926a6eecef537 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.181+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04402c680 con 0x7fa038004d00 2023-10-16T12:33:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.181+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04402c7e0 con 0x7fa038004d00 2023-10-16T12:33:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.181+0000 7fa04affd640 1 -- 172.21.15.60:0/4169811902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04402cb70 con 0x7fa038004d00 2023-10-16T12:33:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa050e1a640 1 -- 172.21.15.60:0/4169811902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 msgr2=0x7fa038005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4169811902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa038005180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa044009a90 tx=0x7fa0380057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa04bfff640 1 -- 172.21.15.60:0/4169811902 reap_dead start 2023-10-16T12:33:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa050e1a640 1 -- 172.21.15.60:0/4169811902 shutdown_connections 2023-10-16T12:33:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa050e1a640 1 -- 172.21.15.60:0/4169811902 >> 172.21.15.60:0/4169811902 conn(0x7fa04c0ce6a0 msgr2=0x7fa04c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa050e1a640 1 -- 172.21.15.60:0/4169811902 shutdown_connections 2023-10-16T12:33:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.182+0000 7fa050e1a640 1 -- 172.21.15.60:0/4169811902 wait complete. 2023-10-16T12:33:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.184+0000 7fa050e1a640 1 Processor -- start 2023-10-16T12:33:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.184+0000 7fa050e1a640 1 -- start start 2023-10-16T12:33:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.185+0000 7fa050e1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa04c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.185+0000 7fa050e1a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa04c0c4ff0 0x7fa04c15f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.185+0000 7fa050e1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15f810 0x7fa04c154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.185+0000 7fa050e1a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c15fca0 con 0x7fa038004d00 2023-10-16T12:33:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.185+0000 7fa050e1a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c06af00 con 0x7fa04c0c4ff0 2023-10-16T12:33:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.185+0000 7fa050e1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c06a380 con 0x7fa04c15f810 2023-10-16T12:33:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa04c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa04c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53076/0 (socket says 172.21.15.60:53076) 2023-10-16T12:33:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 -- 172.21.15.60:0/4206106978 learned_addr learned my addr 172.21.15.60:0/4206106978 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 -- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15f810 msgr2=0x7fa04c154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15f810 0x7fa04c154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa042ffd640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa04c0c4ff0 0x7fa04c15f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 -- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa04c0c4ff0 msgr2=0x7fa04c15f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa04c0c4ff0 0x7fa04c15f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 -- 172.21.15.60:0/4206106978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c06a660 con 0x7fa038004d00 2023-10-16T12:33:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa042ffd640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa04c0c4ff0 0x7fa04c15f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.186+0000 7fa04b7fe640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa04c069da0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa03c009fb0 tx=0x7fa03c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.187+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00e030 con 0x7fa038004d00 2023-10-16T12:33:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.187+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c016470 con 0x7fa038004d00 2023-10-16T12:33:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.187+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c016820 con 0x7fa038004d00 2023-10-16T12:33:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.187+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c06a940 con 0x7fa038004d00 2023-10-16T12:33:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.187+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c0c5470 con 0x7fa038004d00 2023-10-16T12:33:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.188+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c04e9b0 con 0x7fa038004d00 2023-10-16T12:33:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.189+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa03c016980 con 0x7fa038004d00 2023-10-16T12:33:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.190+0000 7fa048ff9640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa02c06ae70 0x7fa02c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.190+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa03c00f070 con 0x7fa038004d00 2023-10-16T12:33:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.191+0000 7fa042ffd640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa02c06ae70 0x7fa02c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.192+0000 7fa042ffd640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa02c06ae70 0x7fa02c06d330 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa034005990 tx=0x7fa034000f50 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.195+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa03c055650 con 0x7fa038004d00 2023-10-16T12:33:24.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:24.431+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa04c0adac0 con 0x7fa038004d00 2023-10-16T12:33:25.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.370+0000 7fa048ff9640 1 -- 172.21.15.60:0/4206106978 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa03c088e60 con 0x7fa038004d00 2023-10-16T12:33:25.372 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 2 and data pool 3 2023-10-16T12:33:25.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa02c06ae70 msgr2=0x7fa02c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa02c06ae70 0x7fa02c06d330 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa034005990 tx=0x7fa034000f50 comp rx=0 tx=0).stop 2023-10-16T12:33:25.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 msgr2=0x7fa04c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa04c069da0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa03c009fb0 tx=0x7fa03c0115a0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 shutdown_connections 2023-10-16T12:33:25.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa02c06ae70 0x7fa02c06d330 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa04c15f810 0x7fa04c154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa04c0c4ff0 0x7fa04c15f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 --2- 172.21.15.60:0/4206106978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa038004d00 0x7fa04c069da0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 >> 172.21.15.60:0/4206106978 conn(0x7fa04c0ce6a0 msgr2=0x7fa04c0ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:25.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 shutdown_connections 2023-10-16T12:33:25.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.373+0000 7fa050e1a640 1 -- 172.21.15.60:0/4206106978 wait complete. 2023-10-16T12:33:25.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.597+0000 7f08a4202640 1 Processor -- start 2023-10-16T12:33:25.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.597+0000 7f08a4202640 1 -- start start 2023-10-16T12:33:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a4202640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f089c159b20 0x7f089c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a4202640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f089c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a4202640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c06f7f0 con 0x7f089c159f00 2023-10-16T12:33:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a4202640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089c06c5e0 con 0x7f089c159b20 2023-10-16T12:33:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a4202640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c06c8c0 con 0x7f089c0b0b20 2023-10-16T12:33:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a179d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f089c159f00 0x7f089c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55144/0 (socket says 172.21.15.60:55144) 2023-10-16T12:33:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a179d640 1 -- 172.21.15.60:0/4294283370 learned_addr learned my addr 172.21.15.60:0/4294283370 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/4294283370 >> v2:172.21.15.144:3300/0 conn(0x7f089c159b20 0x7f089c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a279f640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f089c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.599+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978840617 0 0) 0x7f089c06f7f0 con 0x7f089c159f00 2023-10-16T12:33:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0888003520 con 0x7f089c159f00 2023-10-16T12:33:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2877071208 0 0) 0x7f0888003520 con 0x7f089c159f00 2023-10-16T12:33:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 msgr2=0x7f089c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f089c06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 >> v2:172.21.15.144:3300/0 conn(0x7f089c159b20 msgr2=0x7f089c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 --2- 172.21.15.60:0/4294283370 >> v2:172.21.15.144:3300/0 conn(0x7f089c159b20 0x7f089c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c06cba0 con 0x7f089c159f00 2023-10-16T12:33:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f088c001670 con 0x7f089c159f00 2023-10-16T12:33:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 >> v1:172.21.15.60:6789/0 conn(0x7f089c159f00 legacy=0x7f089c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0888003970 0x7f0888003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f08880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888004d00 0x7f0888005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c06c5e0 con 0x7f0888003970 2023-10-16T12:33:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c06c8c0 con 0x7f0888004d00 2023-10-16T12:33:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088800db90 con 0x7f089c0b0b20 2023-10-16T12:33:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.600+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0888003970 0x7f0888003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a279f640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888004d00 0x7f0888005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a179d640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f08880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a1f9e640 1 -- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 msgr2=0x7f08880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f08880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a1f9e640 1 -- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888004d00 msgr2=0x7f0888005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888004d00 0x7f0888005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a1f9e640 1 -- 172.21.15.60:0/4294283370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888003520 con 0x7f0888003970 2023-10-16T12:33:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.601+0000 7f08a179d640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c0b0b20 0x7f08880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:25.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.602+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0888003970 0x7f0888003d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0898009af0 tx=0x7f08880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200a1824ed464c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.602+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089802c710 con 0x7f0888003970 2023-10-16T12:33:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.602+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089802c870 con 0x7f0888003970 2023-10-16T12:33:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.602+0000 7f08a0f9c640 1 -- 172.21.15.60:0/4294283370 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089802cc00 con 0x7f0888003970 2023-10-16T12:33:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a4202640 1 -- 172.21.15.60:0/4294283370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0888003970 msgr2=0x7f0888003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a4202640 1 --2- 172.21.15.60:0/4294283370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0888003970 0x7f0888003d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0898009af0 tx=0x7f08880057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a279f640 1 -- 172.21.15.60:0/4294283370 reap_dead start 2023-10-16T12:33:25.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a4202640 1 -- 172.21.15.60:0/4294283370 shutdown_connections 2023-10-16T12:33:25.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a4202640 1 -- 172.21.15.60:0/4294283370 >> 172.21.15.60:0/4294283370 conn(0x7f089c0cb410 msgr2=0x7f089c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:25.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a4202640 1 -- 172.21.15.60:0/4294283370 shutdown_connections 2023-10-16T12:33:25.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.603+0000 7f08a4202640 1 -- 172.21.15.60:0/4294283370 wait complete. 2023-10-16T12:33:25.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.605+0000 7f08a4202640 1 Processor -- start 2023-10-16T12:33:25.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.605+0000 7f08a4202640 1 -- start start 2023-10-16T12:33:25.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a4202640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888003970 0x7f089c150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a4202640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 0x7f089c0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a4202640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c150ae0 0x7f089c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a4202640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c1525f0 con 0x7f089c154bc0 2023-10-16T12:33:25.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a4202640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c15c1f0 con 0x7f0888003970 2023-10-16T12:33:25.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a4202640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c15c4d0 con 0x7f089c150ae0 2023-10-16T12:33:25.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 0x7f089c0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 0x7f089c0c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53088/0 (socket says 172.21.15.60:53088) 2023-10-16T12:33:25.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 -- 172.21.15.60:0/731820579 learned_addr learned my addr 172.21.15.60:0/731820579 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:25.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888003970 0x7f089c150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a279f640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c150ae0 0x7f089c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 -- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c150ae0 msgr2=0x7f089c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c150ae0 0x7f089c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 -- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888003970 msgr2=0x7f089c150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888003970 0x7f089c150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a179d640 1 -- 172.21.15.60:0/731820579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c151fa0 con 0x7f089c154bc0 2023-10-16T12:33:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.606+0000 7f08a279f640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c150ae0 0x7f089c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.607+0000 7f08a179d640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 0x7f089c0c5a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f088c001860 tx=0x7f088c00dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.607+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0074e0 con 0x7f089c154bc0 2023-10-16T12:33:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.607+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c007640 con 0x7f089c154bc0 2023-10-16T12:33:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.607+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0049e0 con 0x7f089c154bc0 2023-10-16T12:33:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.607+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c152280 con 0x7f089c154bc0 2023-10-16T12:33:25.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.607+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c0c6180 con 0x7f089c154bc0 2023-10-16T12:33:25.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.608+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c04e160 con 0x7f089c154bc0 2023-10-16T12:33:25.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.609+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f088c004b40 con 0x7f089c154bc0 2023-10-16T12:33:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.610+0000 7f08927fc640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f088006ae70 0x7f088006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.610+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f088c088fd0 con 0x7f089c154bc0 2023-10-16T12:33:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.611+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f088006ae70 0x7f088006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:25.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.612+0000 7f08a1f9e640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f088006ae70 0x7f088006d330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0898036000 tx=0x7f0898030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:25.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.615+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f088c054080 con 0x7f089c154bc0 2023-10-16T12:33:25.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.810+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f089c0b10b0 con 0x7f089c154bc0 2023-10-16T12:33:25.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.813+0000 7f08927fc640 1 -- 172.21.15.60:0/731820579 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f088c0576a0 con 0x7f089c154bc0 2023-10-16T12:33:25.816 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:25.816 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:25.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.815+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f088006ae70 msgr2=0x7f088006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.815+0000 7f08a4202640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f088006ae70 0x7f088006d330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0898036000 tx=0x7f0898030000 comp rx=0 tx=0).stop 2023-10-16T12:33:25.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.815+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 msgr2=0x7f089c0c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:25.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.815+0000 7f08a4202640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 0x7f089c0c5a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f088c001860 tx=0x7f088c00dc70 comp rx=0 tx=0).stop 2023-10-16T12:33:25.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 shutdown_connections 2023-10-16T12:33:25.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f088006ae70 0x7f088006d330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f089c150ae0 0x7f089c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f089c154bc0 0x7f089c0c5a60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 --2- 172.21.15.60:0/731820579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0888003970 0x7f089c150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:25.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 >> 172.21.15.60:0/731820579 conn(0x7f089c0cb410 msgr2=0x7f089c0cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:25.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 shutdown_connections 2023-10-16T12:33:25.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:25.816+0000 7f08a4202640 1 -- 172.21.15.60:0/731820579 wait complete. 2023-10-16T12:33:25.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:26.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.055+0000 7fd9634bb640 1 Processor -- start 2023-10-16T12:33:26.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.056+0000 7fd9634bb640 1 -- start start 2023-10-16T12:33:26.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.057+0000 7fd9634bb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd95c0b3070 0x7fd95c0cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.057+0000 7fd9634bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd95c0b3e70 0x7fd95c0cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.057+0000 7fd9634bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0b3940 con 0x7fd95c0b3450 2023-10-16T12:33:26.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.057+0000 7fd9634bb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0b45e0 con 0x7fd95c0b3070 2023-10-16T12:33:26.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.057+0000 7fd9634bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0b3b90 con 0x7fd95c0b3e70 2023-10-16T12:33:26.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.057+0000 7fd961257640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd95c0b3070 0x7fd95c0cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd961257640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd95c0b3070 0x7fd95c0cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53104/0 (socket says 172.21.15.60:53104) 2023-10-16T12:33:26.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd961257640 1 -- 172.21.15.60:0/3623259928 learned_addr learned my addr 172.21.15.60:0/3623259928 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:26.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd960a56640 1 --2- 172.21.15.60:0/3623259928 >> v2:172.21.15.144:3300/0 conn(0x7fd95c0b3e70 0x7fd95c0cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd961257640 1 -- 172.21.15.60:0/3623259928 >> v2:172.21.15.144:3300/0 conn(0x7fd95c0b3e70 msgr2=0x7fd95c0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:26.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd961257640 1 --2- 172.21.15.60:0/3623259928 >> v2:172.21.15.144:3300/0 conn(0x7fd95c0b3e70 0x7fd95c0cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd961257640 1 -- 172.21.15.60:0/3623259928 >> v1:172.21.15.60:6789/0 conn(0x7fd95c0b3450 legacy=0x7fd95c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:26.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.058+0000 7fd961257640 1 -- 172.21.15.60:0/3623259928 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0d3840 con 0x7fd95c0b3070 2023-10-16T12:33:26.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.059+0000 7fd961257640 1 --2- 172.21.15.60:0/3623259928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3070 0x7fd95c0cf500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0b58e0 tx=0x7fd95802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51714a3e83f4e79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:26.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.059+0000 7fd953fff640 1 -- 172.21.15.60:0/3623259928 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958004230 con 0x7fd95c0b3070 2023-10-16T12:33:26.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.059+0000 7fd953fff640 1 -- 172.21.15.60:0/3623259928 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd958004390 con 0x7fd95c0b3070 2023-10-16T12:33:26.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.059+0000 7fd953fff640 1 -- 172.21.15.60:0/3623259928 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958030840 con 0x7fd95c0b3070 2023-10-16T12:33:26.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 -- 172.21.15.60:0/3623259928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3070 msgr2=0x7fd95c0cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 --2- 172.21.15.60:0/3623259928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3070 0x7fd95c0cf500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0b58e0 tx=0x7fd95802b980 comp rx=0 tx=0).stop 2023-10-16T12:33:26.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 -- 172.21.15.60:0/3623259928 shutdown_connections 2023-10-16T12:33:26.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 --2- 172.21.15.60:0/3623259928 >> v2:172.21.15.144:3300/0 conn(0x7fd95c0b3e70 0x7fd95c0cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 --2- 172.21.15.60:0/3623259928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3070 0x7fd95c0cf500 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 -- 172.21.15.60:0/3623259928 >> 172.21.15.60:0/3623259928 conn(0x7fd95c0d2d20 msgr2=0x7fd95c0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:26.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 -- 172.21.15.60:0/3623259928 shutdown_connections 2023-10-16T12:33:26.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.060+0000 7fd9634bb640 1 -- 172.21.15.60:0/3623259928 wait complete. 2023-10-16T12:33:26.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.062+0000 7fd9634bb640 1 Processor -- start 2023-10-16T12:33:26.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.062+0000 7fd9634bb640 1 -- start start 2023-10-16T12:33:26.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd9634bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd95c0b3070 0x7fd95c09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd9634bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 0x7fd95c09f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd9634bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95c0b3e70 0x7fd95c1f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd9634bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0cb980 con 0x7fd95c0b3450 2023-10-16T12:33:26.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd9634bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0cbbd0 con 0x7fd95c0b3e70 2023-10-16T12:33:26.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd9634bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0cbeb0 con 0x7fd95c0b3070 2023-10-16T12:33:26.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.063+0000 7fd960a56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 0x7fd95c09f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 0x7fd95c09f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53112/0 (socket says 172.21.15.60:53112) 2023-10-16T12:33:26.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 -- 172.21.15.60:0/4206304277 learned_addr learned my addr 172.21.15.60:0/4206304277 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:26.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd961257640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd95c0b3070 0x7fd95c09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd961a58640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95c0b3e70 0x7fd95c1f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 -- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd95c0b3070 msgr2=0x7fd95c09ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd95c0b3070 0x7fd95c09ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 -- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95c0b3e70 msgr2=0x7fd95c1f5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95c0b3e70 0x7fd95c1f5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 -- 172.21.15.60:0/4206304277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958009460 con 0x7fd95c0b3450 2023-10-16T12:33:26.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.064+0000 7fd960a56640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 0x7fd95c09f190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00c7f0 tx=0x7fd94c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:26.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.065+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c004220 con 0x7fd95c0b3450 2023-10-16T12:33:26.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.065+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c004380 con 0x7fd95c0b3450 2023-10-16T12:33:26.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.065+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c007840 con 0x7fd95c0b3450 2023-10-16T12:33:26.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.065+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c0cc190 con 0x7fd95c0b3450 2023-10-16T12:33:26.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.065+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c160460 con 0x7fd95c0b3450 2023-10-16T12:33:26.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.066+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c0cc6b0 con 0x7fd95c0b3450 2023-10-16T12:33:26.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.067+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd94c007bb0 con 0x7fd95c0b3450 2023-10-16T12:33:26.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.068+0000 7fd951ffb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd94006ae70 0x7fd94006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.068+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd94c0895a0 con 0x7fd95c0b3450 2023-10-16T12:33:26.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.068+0000 7fd961257640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd94006ae70 0x7fd94006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.070+0000 7fd961257640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd94006ae70 0x7fd94006d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd958035000 tx=0x7fd958008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:26.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.073+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd95c0cc6b0 con 0x7fd95c0b3450 2023-10-16T12:33:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.270+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd95c0cc6b0 con 0x7fd95c0b3450 2023-10-16T12:33:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.272+0000 7fd951ffb640 1 -- 172.21.15.60:0/4206304277 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fd95c0cc6b0 con 0x7fd95c0b3450 2023-10-16T12:33:26.275 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:26.275 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.274+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd94006ae70 msgr2=0x7fd94006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.274+0000 7fd9634bb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd94006ae70 0x7fd94006d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd958035000 tx=0x7fd958008030 comp rx=0 tx=0).stop 2023-10-16T12:33:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.274+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 msgr2=0x7fd95c09f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 0x7fd95c09f190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00c7f0 tx=0x7fd94c00ccb0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 shutdown_connections 2023-10-16T12:33:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd94006ae70 0x7fd94006d330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95c0b3e70 0x7fd95c1f5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95c0b3450 0x7fd95c09f190 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 --2- 172.21.15.60:0/4206304277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd95c0b3070 0x7fd95c09ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 >> 172.21.15.60:0/4206304277 conn(0x7fd95c0d2d20 msgr2=0x7fd95c1e5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 shutdown_connections 2023-10-16T12:33:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.275+0000 7fd9634bb640 1 -- 172.21.15.60:0/4206304277 wait complete. 2023-10-16T12:33:26.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:33:26.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.522+0000 7f1f0759e640 1 Processor -- start 2023-10-16T12:33:26.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.523+0000 7f1f0759e640 1 -- start start 2023-10-16T12:33:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.524+0000 7f1f0759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0806c040 0x7f1f0806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.524+0000 7f1f0759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f080ad970 0x7f1f0815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.524+0000 7f1f0759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0806c960 con 0x7f1f0815ebc0 2023-10-16T12:33:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.524+0000 7f1f0759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0815b9d0 con 0x7f1f080ad970 2023-10-16T12:33:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.524+0000 7f1f0759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0815bcb0 con 0x7f1f0806c040 2023-10-16T12:33:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f05d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f080ad970 0x7f1f0815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f06d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1f0815ebc0 0x7f1f080cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55168/0 (socket says 172.21.15.60:55168) 2023-10-16T12:33:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f06d9d640 1 -- 172.21.15.60:0/1403866418 learned_addr learned my addr 172.21.15.60:0/1403866418 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:26.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f0659c640 1 --2- 172.21.15.60:0/1403866418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0806c040 0x7f1f0806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f05d9b640 1 -- 172.21.15.60:0/1403866418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0806c040 msgr2=0x7f1f0806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f05d9b640 1 --2- 172.21.15.60:0/1403866418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0806c040 0x7f1f0806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f05d9b640 1 -- 172.21.15.60:0/1403866418 >> v1:172.21.15.60:6789/0 conn(0x7f1f0815ebc0 legacy=0x7f1f080cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f05d9b640 1 -- 172.21.15.60:0/1403866418 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0815bf90 con 0x7f1f080ad970 2023-10-16T12:33:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.525+0000 7f1f0559a640 1 -- 172.21.15.60:0/1403866418 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106261609 0 0) 0x7f1f0806c960 con 0x7f1f0815ebc0 2023-10-16T12:33:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.527+0000 7f1f05d9b640 1 --2- 172.21.15.60:0/1403866418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f080ad970 0x7f1f0815e680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009dc0 tx=0x7f1ef002be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d422f14f33b1363 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.527+0000 7f1f0559a640 1 -- 172.21.15.60:0/1403866418 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef002cc30 con 0x7f1f080ad970 2023-10-16T12:33:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.527+0000 7f1f0559a640 1 -- 172.21.15.60:0/1403866418 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef002cd90 con 0x7f1f080ad970 2023-10-16T12:33:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.527+0000 7f1f0559a640 1 -- 172.21.15.60:0/1403866418 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0034660 con 0x7f1f080ad970 2023-10-16T12:33:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.527+0000 7f1f0759e640 1 -- 172.21.15.60:0/1403866418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f080ad970 msgr2=0x7f1f0815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.527+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1403866418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f080ad970 0x7f1f0815e680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009dc0 tx=0x7f1ef002be80 comp rx=0 tx=0).stop 2023-10-16T12:33:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.528+0000 7f1f0759e640 1 -- 172.21.15.60:0/1403866418 shutdown_connections 2023-10-16T12:33:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.528+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1403866418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f080ad970 0x7f1f0815e680 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.528+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1403866418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0806c040 0x7f1f0806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.528+0000 7f1f0759e640 1 -- 172.21.15.60:0/1403866418 >> 172.21.15.60:0/1403866418 conn(0x7f1f080ce6a0 msgr2=0x7f1f080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.528+0000 7f1f0759e640 1 -- 172.21.15.60:0/1403866418 shutdown_connections 2023-10-16T12:33:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.528+0000 7f1f0759e640 1 -- 172.21.15.60:0/1403866418 wait complete. 2023-10-16T12:33:26.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.530+0000 7f1f0759e640 1 Processor -- start 2023-10-16T12:33:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.530+0000 7f1f0759e640 1 -- start start 2023-10-16T12:33:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 0x7f1f08149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f080ad970 0x7f1f080c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0815ebc0 0x7f1f080c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080c5b20 con 0x7f1f0806c040 2023-10-16T12:33:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f081e4ac0 con 0x7f1f080ad970 2023-10-16T12:33:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080af200 con 0x7f1f0815ebc0 2023-10-16T12:33:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 0x7f1f08149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 0x7f1f08149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53140/0 (socket says 172.21.15.60:53140) 2023-10-16T12:33:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 -- 172.21.15.60:0/1723226652 learned_addr learned my addr 172.21.15.60:0/1723226652 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 -- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0815ebc0 msgr2=0x7f1f080c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0815ebc0 0x7f1f080c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f05d9b640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f080ad970 0x7f1f080c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 -- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f080ad970 msgr2=0x7f1f080c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f080ad970 0x7f1f080c5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.531+0000 7f1f0659c640 1 -- 172.21.15.60:0/1723226652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef0009a70 con 0x7f1f0806c040 2023-10-16T12:33:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.532+0000 7f1f0659c640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 0x7f1f08149c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800d5f0 tx=0x7f1ef800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.532+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef80125f0 con 0x7f1f0806c040 2023-10-16T12:33:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.532+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8012750 con 0x7f1f0806c040 2023-10-16T12:33:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.532+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8012b00 con 0x7f1f0806c040 2023-10-16T12:33:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.532+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f080ae5f0 con 0x7f1f0806c040 2023-10-16T12:33:26.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.532+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f080aebb0 con 0x7f1f0806c040 2023-10-16T12:33:26.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.533+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0806b330 con 0x7f1f0806c040 2023-10-16T12:33:26.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.534+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1ef8012c60 con 0x7f1f0806c040 2023-10-16T12:33:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.535+0000 7f1efeffd640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1ee806ae70 0x7f1ee806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.535+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ef8089c10 con 0x7f1f0806c040 2023-10-16T12:33:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.536+0000 7f1f05d9b640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1ee806ae70 0x7f1ee806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.537+0000 7f1f05d9b640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1ee806ae70 0x7f1ee806d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0036000 tx=0x7f1ef0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.539+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ef8054b90 con 0x7f1f0806c040 2023-10-16T12:33:26.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:26.776+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1f0804e9b0 con 0x7f1f0806c040 2023-10-16T12:33:27.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.583+0000 7f1efeffd640 1 -- 172.21.15.60:0/1723226652 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1ef80581b0 con 0x7f1f0806c040 2023-10-16T12:33:27.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1ee806ae70 msgr2=0x7f1ee806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1ee806ae70 0x7f1ee806d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0036000 tx=0x7f1ef0030000 comp rx=0 tx=0).stop 2023-10-16T12:33:27.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 msgr2=0x7f1f08149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 0x7f1f08149c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800d5f0 tx=0x7f1ef800dab0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 shutdown_connections 2023-10-16T12:33:27.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1ee806ae70 0x7f1ee806d330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f0815ebc0 0x7f1f080c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f080ad970 0x7f1f080c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 --2- 172.21.15.60:0/1723226652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f0806c040 0x7f1f08149c80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.586+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 >> 172.21.15.60:0/1723226652 conn(0x7f1f080ce6a0 msgr2=0x7f1f080c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:27.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.587+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 shutdown_connections 2023-10-16T12:33:27.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.587+0000 7f1f0759e640 1 -- 172.21.15.60:0/1723226652 wait complete. 2023-10-16T12:33:27.611 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:33:27.611 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:33:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.832+0000 7f5661127640 1 Processor -- start 2023-10-16T12:33:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.832+0000 7f5661127640 1 -- start start 2023-10-16T12:33:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f5661127640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 0x7f565c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f5661127640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c15b9d0 con 0x7f565c06c040 2023-10-16T12:33:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f5661127640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c15bcb0 con 0x7f565c0ccfb0 2023-10-16T12:33:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f5661127640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c15bf90 con 0x7f565c0ad970 2023-10-16T12:33:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f565bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f565c06c040 0x7f565c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55174/0 (socket says 172.21.15.60:55174) 2023-10-16T12:33:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f565bfff640 1 -- 172.21.15.60:0/1425515158 learned_addr learned my addr 172.21.15.60:0/1425515158 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.834+0000 7f565b7fe640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 0x7f565c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737246246 0 0) 0x7f565c15b9d0 con 0x7f565c06c040 2023-10-16T12:33:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5644003520 con 0x7f565c06c040 2023-10-16T12:33:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245364334 0 0) 0x7f5644003520 con 0x7f565c06c040 2023-10-16T12:33:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 msgr2=0x7f565c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 0x7f565c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 >> v1:172.21.15.144:6789/0 conn(0x7f565c0ccfb0 legacy=0x7f565c0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0cc270 con 0x7f565c06c040 2023-10-16T12:33:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f564c001670 con 0x7f565c06c040 2023-10-16T12:33:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 >> v1:172.21.15.60:6789/0 conn(0x7f565c06c040 legacy=0x7f565c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f5644003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 0x7f56440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5644004d00 0x7f5644005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c15bf90 con 0x7f56440039a0 2023-10-16T12:33:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564400bac0 con 0x7f5644004d00 2023-10-16T12:33:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564400bc80 con 0x7f565c0ad970 2023-10-16T12:33:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.835+0000 7f565bfff640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f5644003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f565b7fe640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 0x7f56440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f5660926640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5644004d00 0x7f5644005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f565bfff640 1 -- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 msgr2=0x7f56440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f565bfff640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c0ad970 0x7f56440047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f565bfff640 1 -- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5644004d00 msgr2=0x7f5644005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f565bfff640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5644004d00 0x7f5644005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f565bfff640 1 -- 172.21.15.60:0/1425515158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644003520 con 0x7f56440039a0 2023-10-16T12:33:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.836+0000 7f5660926640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5644004d00 0x7f5644005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.837+0000 7f565bfff640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f5644003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f564c00c0e0 tx=0x7f56440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb967495b72942f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.837+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c005ad0 con 0x7f56440039a0 2023-10-16T12:33:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.837+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c005c50 con 0x7f56440039a0 2023-10-16T12:33:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.837+0000 7f565affd640 1 -- 172.21.15.60:0/1425515158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c007070 con 0x7f56440039a0 2023-10-16T12:33:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5661127640 1 -- 172.21.15.60:0/1425515158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 msgr2=0x7f5644003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5661127640 1 --2- 172.21.15.60:0/1425515158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f5644003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f564c00c0e0 tx=0x7f56440057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5660926640 1 -- 172.21.15.60:0/1425515158 reap_dead start 2023-10-16T12:33:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5661127640 1 -- 172.21.15.60:0/1425515158 shutdown_connections 2023-10-16T12:33:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5661127640 1 -- 172.21.15.60:0/1425515158 >> 172.21.15.60:0/1425515158 conn(0x7f565c0ce6a0 msgr2=0x7f565c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5661127640 1 -- 172.21.15.60:0/1425515158 shutdown_connections 2023-10-16T12:33:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.838+0000 7f5661127640 1 -- 172.21.15.60:0/1425515158 wait complete. 2023-10-16T12:33:27.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.840+0000 7f5661127640 1 Processor -- start 2023-10-16T12:33:27.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.840+0000 7f5661127640 1 -- start start 2023-10-16T12:33:27.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5661127640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f565c0c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5661127640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c1f3110 0x7f565c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5661127640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f565c1f1c70 0x7f565c0b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5661127640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0af650 con 0x7f56440039a0 2023-10-16T12:33:27.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5661127640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c58a0 con 0x7f565c1f1c70 2023-10-16T12:33:27.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5661127640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c5b80 con 0x7f565c1f3110 2023-10-16T12:33:27.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f565c0c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f565c0c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53158/0 (socket says 172.21.15.60:53158) 2023-10-16T12:33:27.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 -- 172.21.15.60:0/4268505438 learned_addr learned my addr 172.21.15.60:0/4268505438 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:27.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565b7fe640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c1f3110 0x7f565c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 -- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c1f3110 msgr2=0x7f565c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c1f3110 0x7f565c0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 -- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f565c1f1c70 msgr2=0x7f565c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:27.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5660926640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f565c1f1c70 0x7f565c0b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f565c1f1c70 0x7f565c0b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:27.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f565bfff640 1 -- 172.21.15.60:0/4268505438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0cdcd0 con 0x7f56440039a0 2023-10-16T12:33:27.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.841+0000 7f5660926640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f565c1f1c70 0x7f565c0b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:27.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.842+0000 7f565bfff640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f565c0c6380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f564c00c0e0 tx=0x7f564c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:27.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.842+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c039070 con 0x7f56440039a0 2023-10-16T12:33:27.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.842+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c007580 con 0x7f56440039a0 2023-10-16T12:33:27.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.842+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c03c040 con 0x7f56440039a0 2023-10-16T12:33:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.842+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c0c5e60 con 0x7f56440039a0 2023-10-16T12:33:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.842+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c0af2e0 con 0x7f56440039a0 2023-10-16T12:33:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.843+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c0c6020 con 0x7f56440039a0 2023-10-16T12:33:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.844+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f564c032070 con 0x7f56440039a0 2023-10-16T12:33:27.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.845+0000 7f5658ff9640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f564806ae70 0x7f564806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:27.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.845+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f564c0aa830 con 0x7f56440039a0 2023-10-16T12:33:27.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.846+0000 7f565b7fe640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f564806ae70 0x7f564806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:27.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.847+0000 7f565b7fe640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f564806ae70 0x7f564806d330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f565c0b1230 tx=0x7f5650008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:27.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:27.850+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f564c075950 con 0x7f56440039a0 2023-10-16T12:33:28.041 INFO:tasks.ceph.mds.d.smithi144.stderr:2023-10-16T12:33:28.040+0000 7fcaacd3a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:33:28.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.084+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f565c06b330 con 0x7f56440039a0 2023-10-16T12:33:28.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.597+0000 7f5658ff9640 1 -- 172.21.15.60:0/4268505438 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f564c078f70 con 0x7f56440039a0 2023-10-16T12:33:28.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f564806ae70 msgr2=0x7f564806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:28.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f564806ae70 0x7f564806d330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f565c0b1230 tx=0x7f5650008040 comp rx=0 tx=0).stop 2023-10-16T12:33:28.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 msgr2=0x7f565c0c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:28.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f565c0c6380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f564c00c0e0 tx=0x7f564c001930 comp rx=0 tx=0).stop 2023-10-16T12:33:28.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 shutdown_connections 2023-10-16T12:33:28.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f564806ae70 0x7f564806d330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f565c1f3110 0x7f565c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f565c1f1c70 0x7f565c0b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 --2- 172.21.15.60:0/4268505438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56440039a0 0x7f565c0c6380 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.600+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 >> 172.21.15.60:0/4268505438 conn(0x7f565c0ce6a0 msgr2=0x7f565c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:28.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.601+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 shutdown_connections 2023-10-16T12:33:28.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.601+0000 7f5661127640 1 -- 172.21.15.60:0/4268505438 wait complete. 2023-10-16T12:33:28.625 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:33:28.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:33:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.846+0000 7ff77d96c640 1 Processor -- start 2023-10-16T12:33:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.846+0000 7ff77d96c640 1 -- start start 2023-10-16T12:33:28.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff77d96c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff7780c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:28.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff77d96c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7780c9230 0x7ff7780709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:28.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff77d96c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff778160a90 con 0x7ff778070f20 2023-10-16T12:33:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff77d96c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7780cdec0 con 0x7ff7780c9230 2023-10-16T12:33:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff77d96c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7780cd900 con 0x7ff77815c9f0 2023-10-16T12:33:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff777fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7780c9230 0x7ff7780709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff777fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7780c9230 0x7ff7780709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53174/0 (socket says 172.21.15.60:53174) 2023-10-16T12:33:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.848+0000 7ff777fff640 1 -- 172.21.15.60:0/79275298 learned_addr learned my addr 172.21.15.60:0/79275298 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.849+0000 7ff77c96a640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff7780c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.849+0000 7ff777fff640 1 -- 172.21.15.60:0/79275298 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 msgr2=0x7ff7780c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.849+0000 7ff777fff640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff7780c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.849+0000 7ff777fff640 1 -- 172.21.15.60:0/79275298 >> v1:172.21.15.144:6789/0 conn(0x7ff778070f20 legacy=0x7ff7780ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.849+0000 7ff777fff640 1 -- 172.21.15.60:0/79275298 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780cdbe0 con 0x7ff7780c9230 2023-10-16T12:33:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.849+0000 7ff77c96a640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff7780c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:28.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.850+0000 7ff777fff640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7780c9230 0x7ff7780709e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff764009870 tx=0x7ff76402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3d0853e2e91ea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:28.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff7777fe640 1 -- 172.21.15.60:0/79275298 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764030840 con 0x7ff7780c9230 2023-10-16T12:33:28.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff7777fe640 1 -- 172.21.15.60:0/79275298 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7640309a0 con 0x7ff7780c9230 2023-10-16T12:33:28.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff7777fe640 1 -- 172.21.15.60:0/79275298 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764030d30 con 0x7ff7780c9230 2023-10-16T12:33:28.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff77d96c640 1 -- 172.21.15.60:0/79275298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7780c9230 msgr2=0x7ff7780709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:28.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff77d96c640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7780c9230 0x7ff7780709e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff764009870 tx=0x7ff76402b980 comp rx=0 tx=0).stop 2023-10-16T12:33:28.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff77d96c640 1 -- 172.21.15.60:0/79275298 shutdown_connections 2023-10-16T12:33:28.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff77d96c640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7780c9230 0x7ff7780709e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff77d96c640 1 --2- 172.21.15.60:0/79275298 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff7780c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.851+0000 7ff77d96c640 1 -- 172.21.15.60:0/79275298 >> 172.21.15.60:0/79275298 conn(0x7ff7780ce6a0 msgr2=0x7ff7780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:28.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.852+0000 7ff77d96c640 1 -- 172.21.15.60:0/79275298 shutdown_connections 2023-10-16T12:33:28.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.852+0000 7ff77d96c640 1 -- 172.21.15.60:0/79275298 wait complete. 2023-10-16T12:33:28.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.853+0000 7ff77d96c640 1 Processor -- start 2023-10-16T12:33:28.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.854+0000 7ff77d96c640 1 -- start start 2023-10-16T12:33:28.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.854+0000 7ff77d96c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 0x7ff778158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.854+0000 7ff77d96c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7780c9230 0x7ff77806c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77d96c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff778148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77d96c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7780c5ac0 con 0x7ff778070f20 2023-10-16T12:33:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77d96c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778158120 con 0x7ff7780c9230 2023-10-16T12:33:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77d96c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff778158400 con 0x7ff77815c9f0 2023-10-16T12:33:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 0x7ff778158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 0x7ff778158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53180/0 (socket says 172.21.15.60:53180) 2023-10-16T12:33:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 -- 172.21.15.60:0/3288536323 learned_addr learned my addr 172.21.15.60:0/3288536323 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff777fff640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7780c9230 0x7ff77806c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 -- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 msgr2=0x7ff778148a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff778148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 -- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7780c9230 msgr2=0x7ff77806c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7780c9230 0x7ff77806c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff77c96a640 1 -- 172.21.15.60:0/3288536323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7640094d0 con 0x7ff778070f20 2023-10-16T12:33:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.855+0000 7ff777fff640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7780c9230 0x7ff77806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:28.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.856+0000 7ff77c96a640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 0x7ff778158c00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00e730 tx=0x7ff76c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.856+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c004230 con 0x7ff778070f20 2023-10-16T12:33:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.856+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c004390 con 0x7ff778070f20 2023-10-16T12:33:28.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.856+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c00c8f0 con 0x7ff778070f20 2023-10-16T12:33:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.856+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7781586e0 con 0x7ff778070f20 2023-10-16T12:33:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.856+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77806cbc0 con 0x7ff778070f20 2023-10-16T12:33:28.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.857+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77804e9b0 con 0x7ff778070f20 2023-10-16T12:33:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.858+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff76c00ca50 con 0x7ff778070f20 2023-10-16T12:33:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.859+0000 7ff7757fa640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff75c06ae70 0x7ff75c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.859+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff76c0892b0 con 0x7ff778070f20 2023-10-16T12:33:28.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.860+0000 7ff777fff640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff75c06ae70 0x7ff75c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.861+0000 7ff777fff640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff75c06ae70 0x7ff75c06d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff764036000 tx=0x7ff76403b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:28.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:28.863+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff76c0542c0 con 0x7ff778070f20 2023-10-16T12:33:29.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.099+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7780adac0 con 0x7ff778070f20 2023-10-16T12:33:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.101+0000 7ff7757fa640 1 -- 172.21.15.60:0/3288536323 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7ff76c0578e0 con 0x7ff778070f20 2023-10-16T12:33:29.103 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:29.103 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:33:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:33:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff75c06ae70 msgr2=0x7ff75c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff75c06ae70 0x7ff75c06d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff764036000 tx=0x7ff76403b040 comp rx=0 tx=0).stop 2023-10-16T12:33:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 msgr2=0x7ff778158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 0x7ff778158c00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00e730 tx=0x7ff76c00ebf0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 shutdown_connections 2023-10-16T12:33:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff75c06ae70 0x7ff75c06d330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff77815c9f0 0x7ff778148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7780c9230 0x7ff77806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 --2- 172.21.15.60:0/3288536323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff778070f20 0x7ff778158c00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 >> 172.21.15.60:0/3288536323 conn(0x7ff7780ce6a0 msgr2=0x7ff77815ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 shutdown_connections 2023-10-16T12:33:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.103+0000 7ff77d96c640 1 -- 172.21.15.60:0/3288536323 wait complete. 2023-10-16T12:33:29.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:29.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.350+0000 7f7673147640 1 Processor -- start 2023-10-16T12:33:29.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.350+0000 7f7673147640 1 -- start start 2023-10-16T12:33:29.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.352+0000 7f7673147640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f766c0cd0c0 0x7f766c1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.352+0000 7f7673147640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f766c15db00 0x7f766c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.352+0000 7f7673147640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c15e3e0 con 0x7f766c0b0410 2023-10-16T12:33:29.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.352+0000 7f7673147640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c0b0de0 con 0x7f766c15db00 2023-10-16T12:33:29.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.352+0000 7f7673147640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c0cd4a0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f766bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f766c0b0410 0x7f766c15d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54148/0 (socket says 172.21.15.60:54148) 2023-10-16T12:33:29.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f766bfff640 1 -- 172.21.15.60:0/1138709948 learned_addr learned my addr 172.21.15.60:0/1138709948 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:29.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f76716e4640 1 --2- 172.21.15.60:0/1138709948 >> v2:172.21.15.60:3300/0 conn(0x7f766c15db00 0x7f766c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f7670ee3640 1 --2- 172.21.15.60:0/1138709948 >> v2:172.21.15.144:3300/0 conn(0x7f766c0cd0c0 0x7f766c1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f766b7fe640 1 -- 172.21.15.60:0/1138709948 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251093083 0 0) 0x7f766c15e3e0 con 0x7f766c0b0410 2023-10-16T12:33:29.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f766b7fe640 1 -- 172.21.15.60:0/1138709948 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f765c003520 con 0x7f766c0b0410 2023-10-16T12:33:29.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f76716e4640 1 -- 172.21.15.60:0/1138709948 >> v2:172.21.15.144:3300/0 conn(0x7f766c0cd0c0 msgr2=0x7f766c1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:29.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f76716e4640 1 --2- 172.21.15.60:0/1138709948 >> v2:172.21.15.144:3300/0 conn(0x7f766c0cd0c0 0x7f766c1615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f76716e4640 1 -- 172.21.15.60:0/1138709948 >> v1:172.21.15.60:6789/0 conn(0x7f766c0b0410 legacy=0x7f766c15d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.353+0000 7f76716e4640 1 -- 172.21.15.60:0/1138709948 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c15e100 con 0x7f766c15db00 2023-10-16T12:33:29.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.354+0000 7f76716e4640 1 --2- 172.21.15.60:0/1138709948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c15db00 0x7f766c0b08a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f766c0cc4e0 tx=0x7f766402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b65844b9e0ce8ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:29.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f766b7fe640 1 -- 172.21.15.60:0/1138709948 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766402cbf0 con 0x7f766c15db00 2023-10-16T12:33:29.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f766b7fe640 1 -- 172.21.15.60:0/1138709948 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766402cd50 con 0x7f766c15db00 2023-10-16T12:33:29.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f766b7fe640 1 -- 172.21.15.60:0/1138709948 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664034660 con 0x7f766c15db00 2023-10-16T12:33:29.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 -- 172.21.15.60:0/1138709948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c15db00 msgr2=0x7f766c0b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 --2- 172.21.15.60:0/1138709948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c15db00 0x7f766c0b08a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f766c0cc4e0 tx=0x7f766402be40 comp rx=0 tx=0).stop 2023-10-16T12:33:29.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 -- 172.21.15.60:0/1138709948 shutdown_connections 2023-10-16T12:33:29.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 --2- 172.21.15.60:0/1138709948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c15db00 0x7f766c0b08a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 --2- 172.21.15.60:0/1138709948 >> v2:172.21.15.144:3300/0 conn(0x7f766c0cd0c0 0x7f766c1615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 -- 172.21.15.60:0/1138709948 >> 172.21.15.60:0/1138709948 conn(0x7f766c0cb410 msgr2=0x7f766c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:29.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 -- 172.21.15.60:0/1138709948 shutdown_connections 2023-10-16T12:33:29.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.355+0000 7f7673147640 1 -- 172.21.15.60:0/1138709948 wait complete. 2023-10-16T12:33:29.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.357+0000 7f7673147640 1 Processor -- start 2023-10-16T12:33:29.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.357+0000 7f7673147640 1 -- start start 2023-10-16T12:33:29.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.358+0000 7f7673147640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f766c0b0410 0x7f766c06f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.358+0000 7f7673147640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 0x7f766c15f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.358+0000 7f7673147640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f766c15db00 0x7f766c0b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.358+0000 7f7673147640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c1606f0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.358+0000 7f7673147640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c15fae0 con 0x7f766c15db00 2023-10-16T12:33:29.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.358+0000 7f7673147640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c15fdc0 con 0x7f766c0b0410 2023-10-16T12:33:29.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 0x7f766c15f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 0x7f766c15f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57718/0 (socket says 172.21.15.60:57718) 2023-10-16T12:33:29.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 -- 172.21.15.60:0/3179014988 learned_addr learned my addr 172.21.15.60:0/3179014988 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:29.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f7670ee3640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f766c0b0410 0x7f766c06f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f76716e4640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f766c15db00 0x7f766c0b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 -- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f766c0b0410 msgr2=0x7f766c06f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f766c0b0410 0x7f766c06f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 -- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f766c15db00 msgr2=0x7f766c0b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f766c15db00 0x7f766c0b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 -- 172.21.15.60:0/3179014988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664009a70 con 0x7f766c0cd0c0 2023-10-16T12:33:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.359+0000 7f766bfff640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 0x7f766c15f040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f765800c410 tx=0x7f765800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.360+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658005ce0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.360+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658005e40 con 0x7f766c0cd0c0 2023-10-16T12:33:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.360+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76580072f0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.360+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c1600a0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.360+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c0b17c0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.361+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c04e160 con 0x7f766c0cd0c0 2023-10-16T12:33:29.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.362+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7658007510 con 0x7f766c0cd0c0 2023-10-16T12:33:29.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.363+0000 7f76697fa640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f765006ae70 0x7f765006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.363+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7658088cd0 con 0x7f766c0cd0c0 2023-10-16T12:33:29.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.364+0000 7f7670ee3640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f765006ae70 0x7f765006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.365+0000 7f7670ee3640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f765006ae70 0x7f765006d330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7660009ad0 tx=0x7f7660008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:29.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.367+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7658053e30 con 0x7f766c0cd0c0 2023-10-16T12:33:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.564+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f766c0b4910 con 0x7f766c0cd0c0 2023-10-16T12:33:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.566+0000 7f76697fa640 1 -- 172.21.15.60:0/3179014988 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f7658017090 con 0x7f766c0cd0c0 2023-10-16T12:33:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:29.569 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f765006ae70 msgr2=0x7f765006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f765006ae70 0x7f765006d330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7660009ad0 tx=0x7f7660008040 comp rx=0 tx=0).stop 2023-10-16T12:33:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 msgr2=0x7f766c15f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 0x7f766c15f040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f765800c410 tx=0x7f765800c8d0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 shutdown_connections 2023-10-16T12:33:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f765006ae70 0x7f765006d330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f766c15db00 0x7f766c0b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f766c0cd0c0 0x7f766c15f040 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 --2- 172.21.15.60:0/3179014988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f766c0b0410 0x7f766c06f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 >> 172.21.15.60:0/3179014988 conn(0x7f766c0cb410 msgr2=0x7f766c0aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.568+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 shutdown_connections 2023-10-16T12:33:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.569+0000 7f7673147640 1 -- 172.21.15.60:0/3179014988 wait complete. 2023-10-16T12:33:29.592 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:33:29.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:33:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.816+0000 7fd01a261640 1 Processor -- start 2023-10-16T12:33:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.816+0000 7fd01a261640 1 -- start start 2023-10-16T12:33:29.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd01a261640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd014159f00 0x7fd0140b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd01a261640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd01415d880 0x7fd0140b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd01a261640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01406f7f0 con 0x7fd014159b20 2023-10-16T12:33:29.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd01a261640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01406c5e0 con 0x7fd01415d880 2023-10-16T12:33:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd01a261640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01406c8c0 con 0x7fd014159f00 2023-10-16T12:33:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd013fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd01415d880 0x7fd0140b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd013fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd01415d880 0x7fd0140b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57722/0 (socket says 172.21.15.60:57722) 2023-10-16T12:33:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.818+0000 7fd013fff640 1 -- 172.21.15.60:0/113009417 learned_addr learned my addr 172.21.15.60:0/113009417 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:29.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.819+0000 7fd012ffd640 1 --2- 172.21.15.60:0/113009417 >> v2:172.21.15.144:3300/0 conn(0x7fd014159f00 0x7fd0140b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.819+0000 7fd013fff640 1 -- 172.21.15.60:0/113009417 >> v2:172.21.15.144:3300/0 conn(0x7fd014159f00 msgr2=0x7fd0140b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:29.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.819+0000 7fd013fff640 1 --2- 172.21.15.60:0/113009417 >> v2:172.21.15.144:3300/0 conn(0x7fd014159f00 0x7fd0140b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.819+0000 7fd013fff640 1 -- 172.21.15.60:0/113009417 >> v1:172.21.15.144:6789/0 conn(0x7fd014159b20 legacy=0x7fd01406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:29.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.819+0000 7fd013fff640 1 -- 172.21.15.60:0/113009417 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01406cba0 con 0x7fd01415d880 2023-10-16T12:33:29.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.820+0000 7fd013fff640 1 --2- 172.21.15.60:0/113009417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140b0950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd01415d770 tx=0x7fd00c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df658680873059ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:29.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.820+0000 7fd0127fc640 1 -- 172.21.15.60:0/113009417 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c02ca80 con 0x7fd01415d880 2023-10-16T12:33:29.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.820+0000 7fd0127fc640 1 -- 172.21.15.60:0/113009417 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c02cbe0 con 0x7fd01415d880 2023-10-16T12:33:29.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.820+0000 7fd0127fc640 1 -- 172.21.15.60:0/113009417 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0343f0 con 0x7fd01415d880 2023-10-16T12:33:29.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 -- 172.21.15.60:0/113009417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 msgr2=0x7fd0140b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 --2- 172.21.15.60:0/113009417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140b0950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd01415d770 tx=0x7fd00c02bee0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 -- 172.21.15.60:0/113009417 shutdown_connections 2023-10-16T12:33:29.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 --2- 172.21.15.60:0/113009417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140b0950 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 --2- 172.21.15.60:0/113009417 >> v2:172.21.15.144:3300/0 conn(0x7fd014159f00 0x7fd0140b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 -- 172.21.15.60:0/113009417 >> 172.21.15.60:0/113009417 conn(0x7fd0140cb410 msgr2=0x7fd0140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:29.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 -- 172.21.15.60:0/113009417 shutdown_connections 2023-10-16T12:33:29.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.821+0000 7fd01a261640 1 -- 172.21.15.60:0/113009417 wait complete. 2023-10-16T12:33:29.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.823+0000 7fd01a261640 1 Processor -- start 2023-10-16T12:33:29.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.823+0000 7fd01a261640 1 -- start start 2023-10-16T12:33:29.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.824+0000 7fd01a261640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd014159b20 0x7fd0141664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.824+0000 7fd01a261640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd014159f00 0x7fd01414c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.824+0000 7fd01a261640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.824+0000 7fd01a261640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01414d280 con 0x7fd01415d880 2023-10-16T12:33:29.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.824+0000 7fd01a261640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0140c43b0 con 0x7fd014159b20 2023-10-16T12:33:29.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.824+0000 7fd01a261640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0140c4690 con 0x7fd014159f00 2023-10-16T12:33:29.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57732/0 (socket says 172.21.15.60:57732) 2023-10-16T12:33:29.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 -- 172.21.15.60:0/2406231117 learned_addr learned my addr 172.21.15.60:0/2406231117 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:29.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd0137fe640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd014159b20 0x7fd0141664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd012ffd640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd014159f00 0x7fd01414c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 -- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd014159f00 msgr2=0x7fd01414c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd014159f00 0x7fd01414c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 -- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd014159b20 msgr2=0x7fd0141664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:29.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd014159b20 0x7fd0141664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:29.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.825+0000 7fd013fff640 1 -- 172.21.15.60:0/2406231117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c009b30 con 0x7fd01415d880 2023-10-16T12:33:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.826+0000 7fd013fff640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140cd020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd00c036040 tx=0x7fd00c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.826+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c009510 con 0x7fd01415d880 2023-10-16T12:33:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.826+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c03e450 con 0x7fd01415d880 2023-10-16T12:33:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.826+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c03e7e0 con 0x7fd01415d880 2023-10-16T12:33:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.826+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0140c5be0 con 0x7fd01415d880 2023-10-16T12:33:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.826+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd014166a10 con 0x7fd01415d880 2023-10-16T12:33:29.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.827+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0140c6100 con 0x7fd01415d880 2023-10-16T12:33:29.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.828+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd00c02cd50 con 0x7fd01415d880 2023-10-16T12:33:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.829+0000 7fcffbfff640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd00006aea0 0x7fd00006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.829+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd00c0abed0 con 0x7fd01415d880 2023-10-16T12:33:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.830+0000 7fd0137fe640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd00006aea0 0x7fd00006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.831+0000 7fd0137fe640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd00006aea0 0x7fd00006d360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd004006fd0 tx=0x7fd004008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:29.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:29.834+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd0140c6100 con 0x7fd01415d880 2023-10-16T12:33:30.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.025+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0140c5a60 con 0x7fd00006aea0 2023-10-16T12:33:30.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.050+0000 7fcffbfff640 1 -- 172.21.15.60:0/2406231117 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264513 (secure 0 0 0) 0x7fd00400be30 con 0x7fd00006aea0 2023-10-16T12:33:30.056 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:30.058 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:33:30.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd00006aea0 msgr2=0x7fd00006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd00006aea0 0x7fd00006d360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd004006fd0 tx=0x7fd004008040 comp rx=0 tx=0).stop 2023-10-16T12:33:30.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 msgr2=0x7fd0140cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140cd020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd00c036040 tx=0x7fd00c004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:30.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 shutdown_connections 2023-10-16T12:33:30.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd00006aea0 0x7fd00006d360 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd014159f00 0x7fd01414c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd01415d880 0x7fd0140cd020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 --2- 172.21.15.60:0/2406231117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd014159b20 0x7fd0141664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 >> 172.21.15.60:0/2406231117 conn(0x7fd0140cb410 msgr2=0x7fd0140ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:30.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 shutdown_connections 2023-10-16T12:33:30.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.056+0000 7fd01a261640 1 -- 172.21.15.60:0/2406231117 wait complete. 2023-10-16T12:33:30.080 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-16T12:33:29.733092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236216,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738504,"statfs":{"total":773094113280,"available":772852228096,"internally_reserved":0,"allocated":3055616,"data_stored":1816468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":80,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002238"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930864+0000","last_change":"2023-10-16T12:33:24.620784+0000","last_active":"2023-10-16T12:33:25.930864+0000","last_peered":"2023-10-16T12:33:25.930864+0000","last_clean":"2023-10-16T12:33:25.930864+0000","last_became_active":"2023-10-16T12:33:24.620205+0000","last_became_peered":"2023-10-16T12:33:24.620205+0000","last_unstale":"2023-10-16T12:33:25.930864+0000","last_undegraded":"2023-10-16T12:33:25.930864+0000","last_fullsized":"2023-10-16T12:33:25.930864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585097+0000","last_change":"2023-10-16T12:33:24.608395+0000","last_active":"2023-10-16T12:33:25.585097+0000","last_peered":"2023-10-16T12:33:25.585097+0000","last_clean":"2023-10-16T12:33:25.585097+0000","last_became_active":"2023-10-16T12:33:24.608062+0000","last_became_peered":"2023-10-16T12:33:24.608062+0000","last_unstale":"2023-10-16T12:33:25.585097+0000","last_undegraded":"2023-10-16T12:33:25.585097+0000","last_fullsized":"2023-10-16T12:33:25.585097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928883+0000","last_change":"2023-10-16T12:33:24.607434+0000","last_active":"2023-10-16T12:33:25.928883+0000","last_peered":"2023-10-16T12:33:25.928883+0000","last_clean":"2023-10-16T12:33:25.928883+0000","last_became_active":"2023-10-16T12:33:24.607119+0000","last_became_peered":"2023-10-16T12:33:24.607119+0000","last_unstale":"2023-10-16T12:33:25.928883+0000","last_undegraded":"2023-10-16T12:33:25.928883+0000","last_fullsized":"2023-10-16T12:33:25.928883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.592697+0000","last_change":"2023-10-16T12:33:24.611057+0000","last_active":"2023-10-16T12:33:25.592697+0000","last_peered":"2023-10-16T12:33:25.592697+0000","last_clean":"2023-10-16T12:33:25.592697+0000","last_became_active":"2023-10-16T12:33:24.610357+0000","last_became_peered":"2023-10-16T12:33:24.610357+0000","last_unstale":"2023-10-16T12:33:25.592697+0000","last_undegraded":"2023-10-16T12:33:25.592697+0000","last_fullsized":"2023-10-16T12:33:25.592697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930042+0000","last_change":"2023-10-16T12:33:24.617198+0000","last_active":"2023-10-16T12:33:25.930042+0000","last_peered":"2023-10-16T12:33:25.930042+0000","last_clean":"2023-10-16T12:33:25.930042+0000","last_became_active":"2023-10-16T12:33:24.616710+0000","last_became_peered":"2023-10-16T12:33:24.616710+0000","last_unstale":"2023-10-16T12:33:25.930042+0000","last_undegraded":"2023-10-16T12:33:25.930042+0000","last_fullsized":"2023-10-16T12:33:25.930042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585757+0000","last_change":"2023-10-16T12:33:24.608373+0000","last_active":"2023-10-16T12:33:25.585757+0000","last_peered":"2023-10-16T12:33:25.585757+0000","last_clean":"2023-10-16T12:33:25.585757+0000","last_became_active":"2023-10-16T12:33:24.608083+0000","last_became_peered":"2023-10-16T12:33:24.608083+0000","last_unstale":"2023-10-16T12:33:25.585757+0000","last_undegraded":"2023-10-16T12:33:25.585757+0000","last_fullsized":"2023-10-16T12:33:25.585757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928757+0000","last_change":"2023-10-16T12:33:24.620178+0000","last_active":"2023-10-16T12:33:25.928757+0000","last_peered":"2023-10-16T12:33:25.928757+0000","last_clean":"2023-10-16T12:33:25.928757+0000","last_became_active":"2023-10-16T12:33:24.619871+0000","last_became_peered":"2023-10-16T12:33:24.619871+0000","last_unstale":"2023-10-16T12:33:25.928757+0000","last_undegraded":"2023-10-16T12:33:25.928757+0000","last_fullsized":"2023-10-16T12:33:25.928757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929201+0000","last_change":"2023-10-16T12:33:24.610760+0000","last_active":"2023-10-16T12:33:25.929201+0000","last_peered":"2023-10-16T12:33:25.929201+0000","last_clean":"2023-10-16T12:33:25.929201+0000","last_became_active":"2023-10-16T12:33:24.609947+0000","last_became_peered":"2023-10-16T12:33:24.609947+0000","last_unstale":"2023-10-16T12:33:25.929201+0000","last_undegraded":"2023-10-16T12:33:25.929201+0000","last_fullsized":"2023-10-16T12:33:25.929201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930940+0000","last_change":"2023-10-16T12:33:24.626800+0000","last_active":"2023-10-16T12:33:25.930940+0000","last_peered":"2023-10-16T12:33:25.930940+0000","last_clean":"2023-10-16T12:33:25.930940+0000","last_became_active":"2023-10-16T12:33:24.626566+0000","last_became_peered":"2023-10-16T12:33:24.626566+0000","last_unstale":"2023-10-16T12:33:25.930940+0000","last_undegraded":"2023-10-16T12:33:25.930940+0000","last_fullsized":"2023-10-16T12:33:25.930940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:08.131810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928952+0000","last_change":"2023-10-16T12:33:24.610841+0000","last_active":"2023-10-16T12:33:25.928952+0000","last_peered":"2023-10-16T12:33:25.928952+0000","last_clean":"2023-10-16T12:33:25.928952+0000","last_became_active":"2023-10-16T12:33:24.610533+0000","last_became_peered":"2023-10-16T12:33:24.610533+0000","last_unstale":"2023-10-16T12:33:25.928952+0000","last_undegraded":"2023-10-16T12:33:25.928952+0000","last_fullsized":"2023-10-16T12:33:25.928952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:39.121403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928420+0000","last_change":"2023-10-16T12:33:24.609014+0000","last_active":"2023-10-16T12:33:25.928420+0000","last_peered":"2023-10-16T12:33:25.928420+0000","last_clean":"2023-10-16T12:33:25.928420+0000","last_became_active":"2023-10-16T12:33:24.607622+0000","last_became_peered":"2023-10-16T12:33:24.607622+0000","last_unstale":"2023-10-16T12:33:25.928420+0000","last_undegraded":"2023-10-16T12:33:25.928420+0000","last_fullsized":"2023-10-16T12:33:25.928420+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:00.436524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588652+0000","last_change":"2023-10-16T12:33:24.609916+0000","last_active":"2023-10-16T12:33:25.588652+0000","last_peered":"2023-10-16T12:33:25.588652+0000","last_clean":"2023-10-16T12:33:25.588652+0000","last_became_active":"2023-10-16T12:33:24.609319+0000","last_became_peered":"2023-10-16T12:33:24.609319+0000","last_unstale":"2023-10-16T12:33:25.588652+0000","last_undegraded":"2023-10-16T12:33:25.588652+0000","last_fullsized":"2023-10-16T12:33:25.588652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:04.453258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929179+0000","last_change":"2023-10-16T12:33:24.622611+0000","last_active":"2023-10-16T12:33:25.929179+0000","last_peered":"2023-10-16T12:33:25.929179+0000","last_clean":"2023-10-16T12:33:25.929179+0000","last_became_active":"2023-10-16T12:33:24.622083+0000","last_became_peered":"2023-10-16T12:33:24.622083+0000","last_unstale":"2023-10-16T12:33:25.929179+0000","last_undegraded":"2023-10-16T12:33:25.929179+0000","last_fullsized":"2023-10-16T12:33:25.929179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927071+0000","last_change":"2023-10-16T12:33:24.605142+0000","last_active":"2023-10-16T12:33:25.927071+0000","last_peered":"2023-10-16T12:33:25.927071+0000","last_clean":"2023-10-16T12:33:25.927071+0000","last_became_active":"2023-10-16T12:33:24.604653+0000","last_became_peered":"2023-10-16T12:33:24.604653+0000","last_unstale":"2023-10-16T12:33:25.927071+0000","last_undegraded":"2023-10-16T12:33:25.927071+0000","last_fullsized":"2023-10-16T12:33:25.927071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:42.877240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584708+0000","last_change":"2023-10-16T12:33:24.606078+0000","last_active":"2023-10-16T12:33:25.584708+0000","last_peered":"2023-10-16T12:33:25.584708+0000","last_clean":"2023-10-16T12:33:25.584708+0000","last_became_active":"2023-10-16T12:33:24.605741+0000","last_became_peered":"2023-10-16T12:33:24.605741+0000","last_unstale":"2023-10-16T12:33:25.584708+0000","last_undegraded":"2023-10-16T12:33:25.584708+0000","last_fullsized":"2023-10-16T12:33:25.584708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584951+0000","last_change":"2023-10-16T12:33:24.606094+0000","last_active":"2023-10-16T12:33:25.584951+0000","last_peered":"2023-10-16T12:33:25.584951+0000","last_clean":"2023-10-16T12:33:25.584951+0000","last_became_active":"2023-10-16T12:33:24.605770+0000","last_became_peered":"2023-10-16T12:33:24.605770+0000","last_unstale":"2023-10-16T12:33:25.584951+0000","last_undegraded":"2023-10-16T12:33:25.584951+0000","last_fullsized":"2023-10-16T12:33:25.584951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590756+0000","last_change":"2023-10-16T12:33:24.618354+0000","last_active":"2023-10-16T12:33:25.590756+0000","last_peered":"2023-10-16T12:33:25.590756+0000","last_clean":"2023-10-16T12:33:25.590756+0000","last_became_active":"2023-10-16T12:33:24.617040+0000","last_became_peered":"2023-10-16T12:33:24.617040+0000","last_unstale":"2023-10-16T12:33:25.590756+0000","last_undegraded":"2023-10-16T12:33:25.590756+0000","last_fullsized":"2023-10-16T12:33:25.590756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927146+0000","last_change":"2023-10-16T12:33:24.612736+0000","last_active":"2023-10-16T12:33:25.927146+0000","last_peered":"2023-10-16T12:33:25.927146+0000","last_clean":"2023-10-16T12:33:25.927146+0000","last_became_active":"2023-10-16T12:33:24.612452+0000","last_became_peered":"2023-10-16T12:33:24.612452+0000","last_unstale":"2023-10-16T12:33:25.927146+0000","last_undegraded":"2023-10-16T12:33:25.927146+0000","last_fullsized":"2023-10-16T12:33:25.927146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930486+0000","last_change":"2023-10-16T12:33:24.621402+0000","last_active":"2023-10-16T12:33:25.930486+0000","last_peered":"2023-10-16T12:33:25.930486+0000","last_clean":"2023-10-16T12:33:25.930486+0000","last_became_active":"2023-10-16T12:33:24.621020+0000","last_became_peered":"2023-10-16T12:33:24.621020+0000","last_unstale":"2023-10-16T12:33:25.930486+0000","last_undegraded":"2023-10-16T12:33:25.930486+0000","last_fullsized":"2023-10-16T12:33:25.930486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928170+0000","last_change":"2023-10-16T12:33:24.613689+0000","last_active":"2023-10-16T12:33:25.928170+0000","last_peered":"2023-10-16T12:33:25.928170+0000","last_clean":"2023-10-16T12:33:25.928170+0000","last_became_active":"2023-10-16T12:33:24.613366+0000","last_became_peered":"2023-10-16T12:33:24.613366+0000","last_unstale":"2023-10-16T12:33:25.928170+0000","last_undegraded":"2023-10-16T12:33:25.928170+0000","last_fullsized":"2023-10-16T12:33:25.928170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:46.931083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928066+0000","last_change":"2023-10-16T12:33:24.615117+0000","last_active":"2023-10-16T12:33:25.928066+0000","last_peered":"2023-10-16T12:33:25.928066+0000","last_clean":"2023-10-16T12:33:25.928066+0000","last_became_active":"2023-10-16T12:33:24.614794+0000","last_became_peered":"2023-10-16T12:33:24.614794+0000","last_unstale":"2023-10-16T12:33:25.928066+0000","last_undegraded":"2023-10-16T12:33:25.928066+0000","last_fullsized":"2023-10-16T12:33:25.928066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586082+0000","last_change":"2023-10-16T12:33:24.616251+0000","last_active":"2023-10-16T12:33:25.586082+0000","last_peered":"2023-10-16T12:33:25.586082+0000","last_clean":"2023-10-16T12:33:25.586082+0000","last_became_active":"2023-10-16T12:33:24.615946+0000","last_became_peered":"2023-10-16T12:33:24.615946+0000","last_unstale":"2023-10-16T12:33:25.586082+0000","last_undegraded":"2023-10-16T12:33:25.586082+0000","last_fullsized":"2023-10-16T12:33:25.586082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929157+0000","last_change":"2023-10-16T12:33:24.613346+0000","last_active":"2023-10-16T12:33:25.929157+0000","last_peered":"2023-10-16T12:33:25.929157+0000","last_clean":"2023-10-16T12:33:25.929157+0000","last_became_active":"2023-10-16T12:33:24.612974+0000","last_became_peered":"2023-10-16T12:33:24.612974+0000","last_unstale":"2023-10-16T12:33:25.929157+0000","last_undegraded":"2023-10-16T12:33:25.929157+0000","last_fullsized":"2023-10-16T12:33:25.929157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585129+0000","last_change":"2023-10-16T12:33:24.624851+0000","last_active":"2023-10-16T12:33:25.585129+0000","last_peered":"2023-10-16T12:33:25.585129+0000","last_clean":"2023-10-16T12:33:25.585129+0000","last_became_active":"2023-10-16T12:33:24.624550+0000","last_became_peered":"2023-10-16T12:33:24.624550+0000","last_unstale":"2023-10-16T12:33:25.585129+0000","last_undegraded":"2023-10-16T12:33:25.585129+0000","last_fullsized":"2023-10-16T12:33:25.585129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930566+0000","last_change":"2023-10-16T12:33:24.614362+0000","last_active":"2023-10-16T12:33:25.930566+0000","last_peered":"2023-10-16T12:33:25.930566+0000","last_clean":"2023-10-16T12:33:25.930566+0000","last_became_active":"2023-10-16T12:33:24.613880+0000","last_became_peered":"2023-10-16T12:33:24.613880+0000","last_unstale":"2023-10-16T12:33:25.930566+0000","last_undegraded":"2023-10-16T12:33:25.930566+0000","last_fullsized":"2023-10-16T12:33:25.930566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929120+0000","last_change":"2023-10-16T12:33:24.618147+0000","last_active":"2023-10-16T12:33:25.929120+0000","last_peered":"2023-10-16T12:33:25.929120+0000","last_clean":"2023-10-16T12:33:25.929120+0000","last_became_active":"2023-10-16T12:33:24.617724+0000","last_became_peered":"2023-10-16T12:33:24.617724+0000","last_unstale":"2023-10-16T12:33:25.929120+0000","last_undegraded":"2023-10-16T12:33:25.929120+0000","last_fullsized":"2023-10-16T12:33:25.929120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:52.314433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585125+0000","last_change":"2023-10-16T12:33:24.605102+0000","last_active":"2023-10-16T12:33:25.585125+0000","last_peered":"2023-10-16T12:33:25.585125+0000","last_clean":"2023-10-16T12:33:25.585125+0000","last_became_active":"2023-10-16T12:33:24.604498+0000","last_became_peered":"2023-10-16T12:33:24.604498+0000","last_unstale":"2023-10-16T12:33:25.585125+0000","last_undegraded":"2023-10-16T12:33:25.585125+0000","last_fullsized":"2023-10-16T12:33:25.585125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930199+0000","last_change":"2023-10-16T12:33:24.613950+0000","last_active":"2023-10-16T12:33:25.930199+0000","last_peered":"2023-10-16T12:33:25.930199+0000","last_clean":"2023-10-16T12:33:25.930199+0000","last_became_active":"2023-10-16T12:33:24.613512+0000","last_became_peered":"2023-10-16T12:33:24.613512+0000","last_unstale":"2023-10-16T12:33:25.930199+0000","last_undegraded":"2023-10-16T12:33:25.930199+0000","last_fullsized":"2023-10-16T12:33:25.930199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928442+0000","last_change":"2023-10-16T12:33:24.612300+0000","last_active":"2023-10-16T12:33:25.928442+0000","last_peered":"2023-10-16T12:33:25.928442+0000","last_clean":"2023-10-16T12:33:25.928442+0000","last_became_active":"2023-10-16T12:33:24.611773+0000","last_became_peered":"2023-10-16T12:33:24.611773+0000","last_unstale":"2023-10-16T12:33:25.928442+0000","last_undegraded":"2023-10-16T12:33:25.928442+0000","last_fullsized":"2023-10-16T12:33:25.928442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590561+0000","last_change":"2023-10-16T12:33:24.615637+0000","last_active":"2023-10-16T12:33:25.590561+0000","last_peered":"2023-10-16T12:33:25.590561+0000","last_clean":"2023-10-16T12:33:25.590561+0000","last_became_active":"2023-10-16T12:33:24.615341+0000","last_became_peered":"2023-10-16T12:33:24.615341+0000","last_unstale":"2023-10-16T12:33:25.590561+0000","last_undegraded":"2023-10-16T12:33:25.590561+0000","last_fullsized":"2023-10-16T12:33:25.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928067+0000","last_change":"2023-10-16T12:33:24.603765+0000","last_active":"2023-10-16T12:33:25.928067+0000","last_peered":"2023-10-16T12:33:25.928067+0000","last_clean":"2023-10-16T12:33:25.928067+0000","last_became_active":"2023-10-16T12:33:24.603469+0000","last_became_peered":"2023-10-16T12:33:24.603469+0000","last_unstale":"2023-10-16T12:33:25.928067+0000","last_undegraded":"2023-10-16T12:33:25.928067+0000","last_fullsized":"2023-10-16T12:33:25.928067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927844+0000","last_change":"2023-10-16T12:33:24.617779+0000","last_active":"2023-10-16T12:33:25.927844+0000","last_peered":"2023-10-16T12:33:25.927844+0000","last_clean":"2023-10-16T12:33:25.927844+0000","last_became_active":"2023-10-16T12:33:24.617328+0000","last_became_peered":"2023-10-16T12:33:24.617328+0000","last_unstale":"2023-10-16T12:33:25.927844+0000","last_undegraded":"2023-10-16T12:33:25.927844+0000","last_fullsized":"2023-10-16T12:33:25.927844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:26.089073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.587918+0000","last_change":"2023-10-16T12:33:24.605171+0000","last_active":"2023-10-16T12:33:25.587918+0000","last_peered":"2023-10-16T12:33:25.587918+0000","last_clean":"2023-10-16T12:33:25.587918+0000","last_became_active":"2023-10-16T12:33:24.604706+0000","last_became_peered":"2023-10-16T12:33:24.604706+0000","last_unstale":"2023-10-16T12:33:25.587918+0000","last_undegraded":"2023-10-16T12:33:25.587918+0000","last_fullsized":"2023-10-16T12:33:25.587918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:45.448070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929923+0000","last_change":"2023-10-16T12:33:23.983081+0000","last_active":"2023-10-16T12:33:25.929923+0000","last_peered":"2023-10-16T12:33:25.929923+0000","last_clean":"2023-10-16T12:33:25.929923+0000","last_became_active":"2023-10-16T12:33:23.979378+0000","last_became_peered":"2023-10-16T12:33:23.979378+0000","last_unstale":"2023-10-16T12:33:25.929923+0000","last_undegraded":"2023-10-16T12:33:25.929923+0000","last_fullsized":"2023-10-16T12:33:25.929923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928531+0000","last_change":"2023-10-16T12:33:24.612688+0000","last_active":"2023-10-16T12:33:25.928531+0000","last_peered":"2023-10-16T12:33:25.928531+0000","last_clean":"2023-10-16T12:33:25.928531+0000","last_became_active":"2023-10-16T12:33:24.612202+0000","last_became_peered":"2023-10-16T12:33:24.612202+0000","last_unstale":"2023-10-16T12:33:25.928531+0000","last_undegraded":"2023-10-16T12:33:25.928531+0000","last_fullsized":"2023-10-16T12:33:25.928531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929373+0000","last_change":"2023-10-16T12:33:23.983502+0000","last_active":"2023-10-16T12:33:25.929373+0000","last_peered":"2023-10-16T12:33:25.929373+0000","last_clean":"2023-10-16T12:33:25.929373+0000","last_became_active":"2023-10-16T12:33:23.983189+0000","last_became_peered":"2023-10-16T12:33:23.983189+0000","last_unstale":"2023-10-16T12:33:25.929373+0000","last_undegraded":"2023-10-16T12:33:25.929373+0000","last_fullsized":"2023-10-16T12:33:25.929373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.912908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929270+0000","last_change":"2023-10-16T12:33:23.984326+0000","last_active":"2023-10-16T12:33:25.929270+0000","last_peered":"2023-10-16T12:33:25.929270+0000","last_clean":"2023-10-16T12:33:25.929270+0000","last_became_active":"2023-10-16T12:33:23.984015+0000","last_became_peered":"2023-10-16T12:33:23.984015+0000","last_unstale":"2023-10-16T12:33:25.929270+0000","last_undegraded":"2023-10-16T12:33:25.929270+0000","last_fullsized":"2023-10-16T12:33:25.929270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929604+0000","last_change":"2023-10-16T12:33:24.615262+0000","last_active":"2023-10-16T12:33:25.929604+0000","last_peered":"2023-10-16T12:33:25.929604+0000","last_clean":"2023-10-16T12:33:25.929604+0000","last_became_active":"2023-10-16T12:33:24.610418+0000","last_became_peered":"2023-10-16T12:33:24.610418+0000","last_unstale":"2023-10-16T12:33:25.929604+0000","last_undegraded":"2023-10-16T12:33:25.929604+0000","last_fullsized":"2023-10-16T12:33:25.929604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585175+0000","last_change":"2023-10-16T12:33:23.976361+0000","last_active":"2023-10-16T12:33:25.585175+0000","last_peered":"2023-10-16T12:33:25.585175+0000","last_clean":"2023-10-16T12:33:25.585175+0000","last_became_active":"2023-10-16T12:33:23.975488+0000","last_became_peered":"2023-10-16T12:33:23.975488+0000","last_unstale":"2023-10-16T12:33:25.585175+0000","last_undegraded":"2023-10-16T12:33:25.585175+0000","last_fullsized":"2023-10-16T12:33:25.585175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584519+0000","last_change":"2023-10-16T12:33:24.603552+0000","last_active":"2023-10-16T12:33:25.584519+0000","last_peered":"2023-10-16T12:33:25.584519+0000","last_clean":"2023-10-16T12:33:25.584519+0000","last_became_active":"2023-10-16T12:33:24.602661+0000","last_became_peered":"2023-10-16T12:33:24.602661+0000","last_unstale":"2023-10-16T12:33:25.584519+0000","last_undegraded":"2023-10-16T12:33:25.584519+0000","last_fullsized":"2023-10-16T12:33:25.584519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929244+0000","last_change":"2023-10-16T12:33:23.983406+0000","last_active":"2023-10-16T12:33:25.929244+0000","last_peered":"2023-10-16T12:33:25.929244+0000","last_clean":"2023-10-16T12:33:25.929244+0000","last_became_active":"2023-10-16T12:33:23.982570+0000","last_became_peered":"2023-10-16T12:33:23.982570+0000","last_unstale":"2023-10-16T12:33:25.929244+0000","last_undegraded":"2023-10-16T12:33:25.929244+0000","last_fullsized":"2023-10-16T12:33:25.929244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:17.276818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591933+0000","last_change":"2023-10-16T12:33:24.616611+0000","last_active":"2023-10-16T12:33:25.591933+0000","last_peered":"2023-10-16T12:33:25.591933+0000","last_clean":"2023-10-16T12:33:25.591933+0000","last_became_active":"2023-10-16T12:33:24.616318+0000","last_became_peered":"2023-10-16T12:33:24.616318+0000","last_unstale":"2023-10-16T12:33:25.591933+0000","last_undegraded":"2023-10-16T12:33:25.591933+0000","last_fullsized":"2023-10-16T12:33:25.591933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591819+0000","last_change":"2023-10-16T12:33:23.982425+0000","last_active":"2023-10-16T12:33:25.591819+0000","last_peered":"2023-10-16T12:33:25.591819+0000","last_clean":"2023-10-16T12:33:25.591819+0000","last_became_active":"2023-10-16T12:33:23.982159+0000","last_became_peered":"2023-10-16T12:33:23.982159+0000","last_unstale":"2023-10-16T12:33:25.591819+0000","last_undegraded":"2023-10-16T12:33:25.591819+0000","last_fullsized":"2023-10-16T12:33:25.591819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:00.917480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929362+0000","last_change":"2023-10-16T12:33:24.607789+0000","last_active":"2023-10-16T12:33:25.929362+0000","last_peered":"2023-10-16T12:33:25.929362+0000","last_clean":"2023-10-16T12:33:25.929362+0000","last_became_active":"2023-10-16T12:33:24.606558+0000","last_became_peered":"2023-10-16T12:33:24.606558+0000","last_unstale":"2023-10-16T12:33:25.929362+0000","last_undegraded":"2023-10-16T12:33:25.929362+0000","last_fullsized":"2023-10-16T12:33:25.929362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584911+0000","last_change":"2023-10-16T12:33:23.994926+0000","last_active":"2023-10-16T12:33:25.584911+0000","last_peered":"2023-10-16T12:33:25.584911+0000","last_clean":"2023-10-16T12:33:25.584911+0000","last_became_active":"2023-10-16T12:33:23.994616+0000","last_became_peered":"2023-10-16T12:33:23.994616+0000","last_unstale":"2023-10-16T12:33:25.584911+0000","last_undegraded":"2023-10-16T12:33:25.584911+0000","last_fullsized":"2023-10-16T12:33:25.584911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.615708+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.615262+0000","last_became_peered":"2023-10-16T12:33:24.615262+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928417+0000","last_change":"2023-10-16T12:33:23.990728+0000","last_active":"2023-10-16T12:33:25.928417+0000","last_peered":"2023-10-16T12:33:25.928417+0000","last_clean":"2023-10-16T12:33:25.928417+0000","last_became_active":"2023-10-16T12:33:23.988939+0000","last_became_peered":"2023-10-16T12:33:23.988939+0000","last_unstale":"2023-10-16T12:33:25.928417+0000","last_undegraded":"2023-10-16T12:33:25.928417+0000","last_fullsized":"2023-10-16T12:33:25.928417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:58.091012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930079+0000","last_change":"2023-10-16T12:33:24.620766+0000","last_active":"2023-10-16T12:33:25.930079+0000","last_peered":"2023-10-16T12:33:25.930079+0000","last_clean":"2023-10-16T12:33:25.930079+0000","last_became_active":"2023-10-16T12:33:24.620453+0000","last_became_peered":"2023-10-16T12:33:24.620453+0000","last_unstale":"2023-10-16T12:33:25.930079+0000","last_undegraded":"2023-10-16T12:33:25.930079+0000","last_fullsized":"2023-10-16T12:33:25.930079+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928702+0000","last_change":"2023-10-16T12:33:23.968343+0000","last_active":"2023-10-16T12:33:25.928702+0000","last_peered":"2023-10-16T12:33:25.928702+0000","last_clean":"2023-10-16T12:33:25.928702+0000","last_became_active":"2023-10-16T12:33:23.967874+0000","last_became_peered":"2023-10-16T12:33:23.967874+0000","last_unstale":"2023-10-16T12:33:25.928702+0000","last_undegraded":"2023-10-16T12:33:25.928702+0000","last_fullsized":"2023-10-16T12:33:25.928702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927246+0000","last_change":"2023-10-16T12:33:24.615841+0000","last_active":"2023-10-16T12:33:25.927246+0000","last_peered":"2023-10-16T12:33:25.927246+0000","last_clean":"2023-10-16T12:33:25.927246+0000","last_became_active":"2023-10-16T12:33:24.615333+0000","last_became_peered":"2023-10-16T12:33:24.615333+0000","last_unstale":"2023-10-16T12:33:25.927246+0000","last_undegraded":"2023-10-16T12:33:25.927246+0000","last_fullsized":"2023-10-16T12:33:25.927246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:23.519029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590504+0000","last_change":"2023-10-16T12:33:23.984016+0000","last_active":"2023-10-16T12:33:25.590504+0000","last_peered":"2023-10-16T12:33:25.590504+0000","last_clean":"2023-10-16T12:33:25.590504+0000","last_became_active":"2023-10-16T12:33:23.983727+0000","last_became_peered":"2023-10-16T12:33:23.983727+0000","last_unstale":"2023-10-16T12:33:25.590504+0000","last_undegraded":"2023-10-16T12:33:25.590504+0000","last_fullsized":"2023-10-16T12:33:25.590504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930468+0000","last_change":"2023-10-16T12:33:24.623341+0000","last_active":"2023-10-16T12:33:25.930468+0000","last_peered":"2023-10-16T12:33:25.930468+0000","last_clean":"2023-10-16T12:33:25.930468+0000","last_became_active":"2023-10-16T12:33:24.622356+0000","last_became_peered":"2023-10-16T12:33:24.622356+0000","last_unstale":"2023-10-16T12:33:25.930468+0000","last_undegraded":"2023-10-16T12:33:25.930468+0000","last_fullsized":"2023-10-16T12:33:25.930468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927892+0000","last_change":"2023-10-16T12:33:23.982823+0000","last_active":"2023-10-16T12:33:25.927892+0000","last_peered":"2023-10-16T12:33:25.927892+0000","last_clean":"2023-10-16T12:33:25.927892+0000","last_became_active":"2023-10-16T12:33:23.981883+0000","last_became_peered":"2023-10-16T12:33:23.981883+0000","last_unstale":"2023-10-16T12:33:25.927892+0000","last_undegraded":"2023-10-16T12:33:25.927892+0000","last_fullsized":"2023-10-16T12:33:25.927892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:09.201117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927851+0000","last_change":"2023-10-16T12:33:24.617583+0000","last_active":"2023-10-16T12:33:25.927851+0000","last_peered":"2023-10-16T12:33:25.927851+0000","last_clean":"2023-10-16T12:33:25.927851+0000","last_became_active":"2023-10-16T12:33:24.616992+0000","last_became_peered":"2023-10-16T12:33:24.616992+0000","last_unstale":"2023-10-16T12:33:25.927851+0000","last_undegraded":"2023-10-16T12:33:25.927851+0000","last_fullsized":"2023-10-16T12:33:25.927851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:38.188085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.926869+0000","last_change":"2023-10-16T12:33:23.988300+0000","last_active":"2023-10-16T12:33:25.926869+0000","last_peered":"2023-10-16T12:33:25.926869+0000","last_clean":"2023-10-16T12:33:25.926869+0000","last_became_active":"2023-10-16T12:33:23.987831+0000","last_became_peered":"2023-10-16T12:33:23.987831+0000","last_unstale":"2023-10-16T12:33:25.926869+0000","last_undegraded":"2023-10-16T12:33:25.926869+0000","last_fullsized":"2023-10-16T12:33:25.926869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928930+0000","last_change":"2023-10-16T12:33:24.613800+0000","last_active":"2023-10-16T12:33:25.928930+0000","last_peered":"2023-10-16T12:33:25.928930+0000","last_clean":"2023-10-16T12:33:25.928930+0000","last_became_active":"2023-10-16T12:33:24.613268+0000","last_became_peered":"2023-10-16T12:33:24.613268+0000","last_unstale":"2023-10-16T12:33:25.928930+0000","last_undegraded":"2023-10-16T12:33:25.928930+0000","last_fullsized":"2023-10-16T12:33:25.928930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588350+0000","last_change":"2023-10-16T12:33:23.970854+0000","last_active":"2023-10-16T12:33:25.588350+0000","last_peered":"2023-10-16T12:33:25.588350+0000","last_clean":"2023-10-16T12:33:25.588350+0000","last_became_active":"2023-10-16T12:33:23.970073+0000","last_became_peered":"2023-10-16T12:33:23.970073+0000","last_unstale":"2023-10-16T12:33:25.588350+0000","last_undegraded":"2023-10-16T12:33:25.588350+0000","last_fullsized":"2023-10-16T12:33:25.588350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930633+0000","last_change":"2023-10-16T12:33:24.624669+0000","last_active":"2023-10-16T12:33:25.930633+0000","last_peered":"2023-10-16T12:33:25.930633+0000","last_clean":"2023-10-16T12:33:25.930633+0000","last_became_active":"2023-10-16T12:33:24.624315+0000","last_became_peered":"2023-10-16T12:33:24.624315+0000","last_unstale":"2023-10-16T12:33:25.930633+0000","last_undegraded":"2023-10-16T12:33:25.930633+0000","last_fullsized":"2023-10-16T12:33:25.930633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927431+0000","last_change":"2023-10-16T12:33:23.980566+0000","last_active":"2023-10-16T12:33:25.927431+0000","last_peered":"2023-10-16T12:33:25.927431+0000","last_clean":"2023-10-16T12:33:25.927431+0000","last_became_active":"2023-10-16T12:33:23.980200+0000","last_became_peered":"2023-10-16T12:33:23.980200+0000","last_unstale":"2023-10-16T12:33:25.927431+0000","last_undegraded":"2023-10-16T12:33:25.927431+0000","last_fullsized":"2023-10-16T12:33:25.927431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.931065+0000","last_change":"2023-10-16T12:33:24.624902+0000","last_active":"2023-10-16T12:33:25.931065+0000","last_peered":"2023-10-16T12:33:25.931065+0000","last_clean":"2023-10-16T12:33:25.931065+0000","last_became_active":"2023-10-16T12:33:24.624646+0000","last_became_peered":"2023-10-16T12:33:24.624646+0000","last_unstale":"2023-10-16T12:33:25.931065+0000","last_undegraded":"2023-10-16T12:33:25.931065+0000","last_fullsized":"2023-10-16T12:33:25.931065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:21.560247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585130+0000","last_change":"2023-10-16T12:33:23.984004+0000","last_active":"2023-10-16T12:33:25.585130+0000","last_peered":"2023-10-16T12:33:25.585130+0000","last_clean":"2023-10-16T12:33:25.585130+0000","last_became_active":"2023-10-16T12:33:23.982385+0000","last_became_peered":"2023-10-16T12:33:23.982385+0000","last_unstale":"2023-10-16T12:33:25.585130+0000","last_undegraded":"2023-10-16T12:33:25.585130+0000","last_fullsized":"2023-10-16T12:33:25.585130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586032+0000","last_change":"2023-10-16T12:33:24.611253+0000","last_active":"2023-10-16T12:33:25.586032+0000","last_peered":"2023-10-16T12:33:25.586032+0000","last_clean":"2023-10-16T12:33:25.586032+0000","last_became_active":"2023-10-16T12:33:24.610712+0000","last_became_peered":"2023-10-16T12:33:24.610712+0000","last_unstale":"2023-10-16T12:33:25.586032+0000","last_undegraded":"2023-10-16T12:33:25.586032+0000","last_fullsized":"2023-10-16T12:33:25.586032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927465+0000","last_change":"2023-10-16T12:33:23.985969+0000","last_active":"2023-10-16T12:33:25.927465+0000","last_peered":"2023-10-16T12:33:25.927465+0000","last_clean":"2023-10-16T12:33:25.927465+0000","last_became_active":"2023-10-16T12:33:23.985467+0000","last_became_peered":"2023-10-16T12:33:23.985467+0000","last_unstale":"2023-10-16T12:33:25.927465+0000","last_undegraded":"2023-10-16T12:33:25.927465+0000","last_fullsized":"2023-10-16T12:33:25.927465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930068+0000","last_change":"2023-10-16T12:33:24.627563+0000","last_active":"2023-10-16T12:33:25.930068+0000","last_peered":"2023-10-16T12:33:25.930068+0000","last_clean":"2023-10-16T12:33:25.930068+0000","last_became_active":"2023-10-16T12:33:24.627241+0000","last_became_peered":"2023-10-16T12:33:24.627241+0000","last_unstale":"2023-10-16T12:33:25.930068+0000","last_undegraded":"2023-10-16T12:33:25.930068+0000","last_fullsized":"2023-10-16T12:33:25.930068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-16T12:33:24.598233+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:33:24.598233+0000","last_peered":"2023-10-16T12:33:24.598233+0000","last_clean":"2023-10-16T12:33:24.598233+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:33:24.598233+0000","last_undegraded":"2023-10-16T12:33:24.598233+0000","last_fullsized":"2023-10-16T12:33:24.598233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585104+0000","last_change":"2023-10-16T12:33:23.972294+0000","last_active":"2023-10-16T12:33:25.585104+0000","last_peered":"2023-10-16T12:33:25.585104+0000","last_clean":"2023-10-16T12:33:25.585104+0000","last_became_active":"2023-10-16T12:33:23.971886+0000","last_became_peered":"2023-10-16T12:33:23.971886+0000","last_unstale":"2023-10-16T12:33:25.585104+0000","last_undegraded":"2023-10-16T12:33:25.585104+0000","last_fullsized":"2023-10-16T12:33:25.585104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.874107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929522+0000","last_change":"2023-10-16T12:33:24.604564+0000","last_active":"2023-10-16T12:33:25.929522+0000","last_peered":"2023-10-16T12:33:25.929522+0000","last_clean":"2023-10-16T12:33:25.929522+0000","last_became_active":"2023-10-16T12:33:24.604240+0000","last_became_peered":"2023-10-16T12:33:24.604240+0000","last_unstale":"2023-10-16T12:33:25.929522+0000","last_undegraded":"2023-10-16T12:33:25.929522+0000","last_fullsized":"2023-10-16T12:33:25.929522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927720+0000","last_change":"2023-10-16T12:33:23.988943+0000","last_active":"2023-10-16T12:33:25.927720+0000","last_peered":"2023-10-16T12:33:25.927720+0000","last_clean":"2023-10-16T12:33:25.927720+0000","last_became_active":"2023-10-16T12:33:23.988631+0000","last_became_peered":"2023-10-16T12:33:23.988631+0000","last_unstale":"2023-10-16T12:33:25.927720+0000","last_undegraded":"2023-10-16T12:33:25.927720+0000","last_fullsized":"2023-10-16T12:33:25.927720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927155+0000","last_change":"2023-10-16T12:33:24.615033+0000","last_active":"2023-10-16T12:33:25.927155+0000","last_peered":"2023-10-16T12:33:25.927155+0000","last_clean":"2023-10-16T12:33:25.927155+0000","last_became_active":"2023-10-16T12:33:24.614433+0000","last_became_peered":"2023-10-16T12:33:24.614433+0000","last_unstale":"2023-10-16T12:33:25.927155+0000","last_undegraded":"2023-10-16T12:33:25.927155+0000","last_fullsized":"2023-10-16T12:33:25.927155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:38.208712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928316+0000","last_change":"2023-10-16T12:33:23.978410+0000","last_active":"2023-10-16T12:33:25.928316+0000","last_peered":"2023-10-16T12:33:25.928316+0000","last_clean":"2023-10-16T12:33:25.928316+0000","last_became_active":"2023-10-16T12:33:23.978050+0000","last_became_peered":"2023-10-16T12:33:23.978050+0000","last_unstale":"2023-10-16T12:33:25.928316+0000","last_undegraded":"2023-10-16T12:33:25.928316+0000","last_fullsized":"2023-10-16T12:33:25.928316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584693+0000","last_change":"2023-10-16T12:33:24.600057+0000","last_active":"2023-10-16T12:33:25.584693+0000","last_peered":"2023-10-16T12:33:25.584693+0000","last_clean":"2023-10-16T12:33:25.584693+0000","last_became_active":"2023-10-16T12:33:24.599294+0000","last_became_peered":"2023-10-16T12:33:24.599294+0000","last_unstale":"2023-10-16T12:33:25.584693+0000","last_undegraded":"2023-10-16T12:33:25.584693+0000","last_fullsized":"2023-10-16T12:33:25.584693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590705+0000","last_change":"2023-10-16T12:33:23.990485+0000","last_active":"2023-10-16T12:33:25.590705+0000","last_peered":"2023-10-16T12:33:25.590705+0000","last_clean":"2023-10-16T12:33:25.590705+0000","last_became_active":"2023-10-16T12:33:23.989836+0000","last_became_peered":"2023-10-16T12:33:23.989836+0000","last_unstale":"2023-10-16T12:33:25.590705+0000","last_undegraded":"2023-10-16T12:33:25.590705+0000","last_fullsized":"2023-10-16T12:33:25.590705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.589289+0000","last_change":"2023-10-16T12:33:24.619525+0000","last_active":"2023-10-16T12:33:25.589289+0000","last_peered":"2023-10-16T12:33:25.589289+0000","last_clean":"2023-10-16T12:33:25.589289+0000","last_became_active":"2023-10-16T12:33:24.619242+0000","last_became_peered":"2023-10-16T12:33:24.619242+0000","last_unstale":"2023-10-16T12:33:25.589289+0000","last_undegraded":"2023-10-16T12:33:25.589289+0000","last_fullsized":"2023-10-16T12:33:25.589289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:12.969746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584815+0000","last_change":"2023-10-16T12:33:23.976427+0000","last_active":"2023-10-16T12:33:25.584815+0000","last_peered":"2023-10-16T12:33:25.584815+0000","last_clean":"2023-10-16T12:33:25.584815+0000","last_became_active":"2023-10-16T12:33:23.975234+0000","last_became_peered":"2023-10-16T12:33:23.975234+0000","last_unstale":"2023-10-16T12:33:25.584815+0000","last_undegraded":"2023-10-16T12:33:25.584815+0000","last_fullsized":"2023-10-16T12:33:25.584815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590492+0000","last_change":"2023-10-16T12:33:24.614482+0000","last_active":"2023-10-16T12:33:25.590492+0000","last_peered":"2023-10-16T12:33:25.590492+0000","last_clean":"2023-10-16T12:33:25.590492+0000","last_became_active":"2023-10-16T12:33:24.614144+0000","last_became_peered":"2023-10-16T12:33:24.614144+0000","last_unstale":"2023-10-16T12:33:25.590492+0000","last_undegraded":"2023-10-16T12:33:25.590492+0000","last_fullsized":"2023-10-16T12:33:25.590492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930752+0000","last_change":"2023-10-16T12:33:23.986466+0000","last_active":"2023-10-16T12:33:25.930752+0000","last_peered":"2023-10-16T12:33:25.930752+0000","last_clean":"2023-10-16T12:33:25.930752+0000","last_became_active":"2023-10-16T12:33:23.985508+0000","last_became_peered":"2023-10-16T12:33:23.985508+0000","last_unstale":"2023-10-16T12:33:25.930752+0000","last_undegraded":"2023-10-16T12:33:25.930752+0000","last_fullsized":"2023-10-16T12:33:25.930752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:08.057398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584833+0000","last_change":"2023-10-16T12:33:24.605662+0000","last_active":"2023-10-16T12:33:25.584833+0000","last_peered":"2023-10-16T12:33:25.584833+0000","last_clean":"2023-10-16T12:33:25.584833+0000","last_became_active":"2023-10-16T12:33:24.605319+0000","last_became_peered":"2023-10-16T12:33:24.605319+0000","last_unstale":"2023-10-16T12:33:25.584833+0000","last_undegraded":"2023-10-16T12:33:25.584833+0000","last_fullsized":"2023-10-16T12:33:25.584833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585096+0000","last_change":"2023-10-16T12:33:23.986752+0000","last_active":"2023-10-16T12:33:25.585096+0000","last_peered":"2023-10-16T12:33:25.585096+0000","last_clean":"2023-10-16T12:33:25.585096+0000","last_became_active":"2023-10-16T12:33:23.986189+0000","last_became_peered":"2023-10-16T12:33:23.986189+0000","last_unstale":"2023-10-16T12:33:25.585096+0000","last_undegraded":"2023-10-16T12:33:25.585096+0000","last_fullsized":"2023-10-16T12:33:25.585096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930740+0000","last_change":"2023-10-16T12:33:24.616668+0000","last_active":"2023-10-16T12:33:25.930740+0000","last_peered":"2023-10-16T12:33:25.930740+0000","last_clean":"2023-10-16T12:33:25.930740+0000","last_became_active":"2023-10-16T12:33:24.616308+0000","last_became_peered":"2023-10-16T12:33:24.616308+0000","last_unstale":"2023-10-16T12:33:25.930740+0000","last_undegraded":"2023-10-16T12:33:25.930740+0000","last_fullsized":"2023-10-16T12:33:25.930740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929376+0000","last_change":"2023-10-16T12:33:23.991338+0000","last_active":"2023-10-16T12:33:25.929376+0000","last_peered":"2023-10-16T12:33:25.929376+0000","last_clean":"2023-10-16T12:33:25.929376+0000","last_became_active":"2023-10-16T12:33:23.986162+0000","last_became_peered":"2023-10-16T12:33:23.986162+0000","last_unstale":"2023-10-16T12:33:25.929376+0000","last_undegraded":"2023-10-16T12:33:25.929376+0000","last_fullsized":"2023-10-16T12:33:25.929376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:32.667813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928205+0000","last_change":"2023-10-16T12:33:24.612178+0000","last_active":"2023-10-16T12:33:25.928205+0000","last_peered":"2023-10-16T12:33:25.928205+0000","last_clean":"2023-10-16T12:33:25.928205+0000","last_became_active":"2023-10-16T12:33:24.608876+0000","last_became_peered":"2023-10-16T12:33:24.608876+0000","last_unstale":"2023-10-16T12:33:25.928205+0000","last_undegraded":"2023-10-16T12:33:25.928205+0000","last_fullsized":"2023-10-16T12:33:25.928205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927671+0000","last_change":"2023-10-16T12:33:23.983928+0000","last_active":"2023-10-16T12:33:25.927671+0000","last_peered":"2023-10-16T12:33:25.927671+0000","last_clean":"2023-10-16T12:33:25.927671+0000","last_became_active":"2023-10-16T12:33:23.983266+0000","last_became_peered":"2023-10-16T12:33:23.983266+0000","last_unstale":"2023-10-16T12:33:25.927671+0000","last_undegraded":"2023-10-16T12:33:25.927671+0000","last_fullsized":"2023-10-16T12:33:25.927671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.605248+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.604915+0000","last_became_peered":"2023-10-16T12:33:24.604915+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584607+0000","last_change":"2023-10-16T12:33:23.987636+0000","last_active":"2023-10-16T12:33:25.584607+0000","last_peered":"2023-10-16T12:33:25.584607+0000","last_clean":"2023-10-16T12:33:25.584607+0000","last_became_active":"2023-10-16T12:33:23.987422+0000","last_became_peered":"2023-10-16T12:33:23.987422+0000","last_unstale":"2023-10-16T12:33:25.584607+0000","last_undegraded":"2023-10-16T12:33:25.584607+0000","last_fullsized":"2023-10-16T12:33:25.584607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930014+0000","last_change":"2023-10-16T12:33:24.607092+0000","last_active":"2023-10-16T12:33:25.930014+0000","last_peered":"2023-10-16T12:33:25.930014+0000","last_clean":"2023-10-16T12:33:25.930014+0000","last_became_active":"2023-10-16T12:33:24.606816+0000","last_became_peered":"2023-10-16T12:33:24.606816+0000","last_unstale":"2023-10-16T12:33:25.930014+0000","last_undegraded":"2023-10-16T12:33:25.930014+0000","last_fullsized":"2023-10-16T12:33:25.930014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584703+0000","last_change":"2023-10-16T12:33:23.982671+0000","last_active":"2023-10-16T12:33:25.584703+0000","last_peered":"2023-10-16T12:33:25.584703+0000","last_clean":"2023-10-16T12:33:25.584703+0000","last_became_active":"2023-10-16T12:33:23.982314+0000","last_became_peered":"2023-10-16T12:33:23.982314+0000","last_unstale":"2023-10-16T12:33:25.584703+0000","last_undegraded":"2023-10-16T12:33:25.584703+0000","last_fullsized":"2023-10-16T12:33:25.584703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930259+0000","last_change":"2023-10-16T12:33:24.614159+0000","last_active":"2023-10-16T12:33:25.930259+0000","last_peered":"2023-10-16T12:33:25.930259+0000","last_clean":"2023-10-16T12:33:25.930259+0000","last_became_active":"2023-10-16T12:33:24.613857+0000","last_became_peered":"2023-10-16T12:33:24.613857+0000","last_unstale":"2023-10-16T12:33:25.930259+0000","last_undegraded":"2023-10-16T12:33:25.930259+0000","last_fullsized":"2023-10-16T12:33:25.930259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930386+0000","last_change":"2023-10-16T12:33:23.987798+0000","last_active":"2023-10-16T12:33:25.930386+0000","last_peered":"2023-10-16T12:33:25.930386+0000","last_clean":"2023-10-16T12:33:25.930386+0000","last_became_active":"2023-10-16T12:33:23.987460+0000","last_became_peered":"2023-10-16T12:33:23.987460+0000","last_unstale":"2023-10-16T12:33:25.930386+0000","last_undegraded":"2023-10-16T12:33:25.930386+0000","last_fullsized":"2023-10-16T12:33:25.930386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929124+0000","last_change":"2023-10-16T12:33:24.615457+0000","last_active":"2023-10-16T12:33:25.929124+0000","last_peered":"2023-10-16T12:33:25.929124+0000","last_clean":"2023-10-16T12:33:25.929124+0000","last_became_active":"2023-10-16T12:33:24.615166+0000","last_became_peered":"2023-10-16T12:33:24.615166+0000","last_unstale":"2023-10-16T12:33:25.929124+0000","last_undegraded":"2023-10-16T12:33:25.929124+0000","last_fullsized":"2023-10-16T12:33:25.929124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:24.606678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586074+0000","last_change":"2023-10-16T12:33:24.604607+0000","last_active":"2023-10-16T12:33:25.586074+0000","last_peered":"2023-10-16T12:33:25.586074+0000","last_clean":"2023-10-16T12:33:25.586074+0000","last_became_active":"2023-10-16T12:33:24.604183+0000","last_became_peered":"2023-10-16T12:33:24.604183+0000","last_unstale":"2023-10-16T12:33:25.586074+0000","last_undegraded":"2023-10-16T12:33:25.586074+0000","last_fullsized":"2023-10-16T12:33:25.586074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929969+0000","last_change":"2023-10-16T12:33:23.990373+0000","last_active":"2023-10-16T12:33:25.929969+0000","last_peered":"2023-10-16T12:33:25.929969+0000","last_clean":"2023-10-16T12:33:25.929969+0000","last_became_active":"2023-10-16T12:33:23.990049+0000","last_became_peered":"2023-10-16T12:33:23.990049+0000","last_unstale":"2023-10-16T12:33:25.929969+0000","last_undegraded":"2023-10-16T12:33:25.929969+0000","last_fullsized":"2023-10-16T12:33:25.929969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585658+0000","last_change":"2023-10-16T12:33:24.612037+0000","last_active":"2023-10-16T12:33:25.585658+0000","last_peered":"2023-10-16T12:33:25.585658+0000","last_clean":"2023-10-16T12:33:25.585658+0000","last_became_active":"2023-10-16T12:33:24.611609+0000","last_became_peered":"2023-10-16T12:33:24.611609+0000","last_unstale":"2023-10-16T12:33:25.585658+0000","last_undegraded":"2023-10-16T12:33:25.585658+0000","last_fullsized":"2023-10-16T12:33:25.585658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930569+0000","last_change":"2023-10-16T12:33:23.985742+0000","last_active":"2023-10-16T12:33:25.930569+0000","last_peered":"2023-10-16T12:33:25.930569+0000","last_clean":"2023-10-16T12:33:25.930569+0000","last_became_active":"2023-10-16T12:33:23.985053+0000","last_became_peered":"2023-10-16T12:33:23.985053+0000","last_unstale":"2023-10-16T12:33:25.930569+0000","last_undegraded":"2023-10-16T12:33:25.930569+0000","last_fullsized":"2023-10-16T12:33:25.930569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:43.191029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927970+0000","last_change":"2023-10-16T12:33:24.610651+0000","last_active":"2023-10-16T12:33:25.927970+0000","last_peered":"2023-10-16T12:33:25.927970+0000","last_clean":"2023-10-16T12:33:25.927970+0000","last_became_active":"2023-10-16T12:33:24.605512+0000","last_became_peered":"2023-10-16T12:33:24.605512+0000","last_unstale":"2023-10-16T12:33:25.927970+0000","last_undegraded":"2023-10-16T12:33:25.927970+0000","last_fullsized":"2023-10-16T12:33:25.927970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930559+0000","last_change":"2023-10-16T12:33:23.981596+0000","last_active":"2023-10-16T12:33:25.930559+0000","last_peered":"2023-10-16T12:33:25.930559+0000","last_clean":"2023-10-16T12:33:25.930559+0000","last_became_active":"2023-10-16T12:33:23.981314+0000","last_became_peered":"2023-10-16T12:33:23.981314+0000","last_unstale":"2023-10-16T12:33:25.930559+0000","last_undegraded":"2023-10-16T12:33:25.930559+0000","last_fullsized":"2023-10-16T12:33:25.930559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930671+0000","last_change":"2023-10-16T12:33:24.613513+0000","last_active":"2023-10-16T12:33:25.930671+0000","last_peered":"2023-10-16T12:33:25.930671+0000","last_clean":"2023-10-16T12:33:25.930671+0000","last_became_active":"2023-10-16T12:33:24.613158+0000","last_became_peered":"2023-10-16T12:33:24.613158+0000","last_unstale":"2023-10-16T12:33:25.930671+0000","last_undegraded":"2023-10-16T12:33:25.930671+0000","last_fullsized":"2023-10-16T12:33:25.930671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929580+0000","last_change":"2023-10-16T12:33:23.978618+0000","last_active":"2023-10-16T12:33:25.929580+0000","last_peered":"2023-10-16T12:33:25.929580+0000","last_clean":"2023-10-16T12:33:25.929580+0000","last_became_active":"2023-10-16T12:33:23.978325+0000","last_became_peered":"2023-10-16T12:33:23.978325+0000","last_unstale":"2023-10-16T12:33:25.929580+0000","last_undegraded":"2023-10-16T12:33:25.929580+0000","last_fullsized":"2023-10-16T12:33:25.929580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.544}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9379999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3340000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.077}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":831488,"data_stored":669870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:33:30.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:33:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.352+0000 7f803f661640 1 Processor -- start 2023-10-16T12:33:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.353+0000 7f803f661640 1 -- start start 2023-10-16T12:33:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.354+0000 7f803f661640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f803815b010 0x7f80380ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.354+0000 7f803f661640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80380afee0 0x7f80380b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.354+0000 7f803f661640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80380cf020 0x7f80380cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.354+0000 7f803f661640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80380cf9f0 con 0x7f80380cf020 2023-10-16T12:33:30.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803f661640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80380cee00 con 0x7f80380afee0 2023-10-16T12:33:30.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803f661640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80380aeba0 con 0x7f803815b010 2023-10-16T12:33:30.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80380cf020 0x7f80380cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80380cf020 0x7f80380cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57740/0 (socket says 172.21.15.60:57740) 2023-10-16T12:33:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 -- 172.21.15.60:0/333734950 learned_addr learned my addr 172.21.15.60:0/333734950 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803d3fd640 1 --2- 172.21.15.60:0/333734950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f803815b010 0x7f80380ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803cbfc640 1 --2- 172.21.15.60:0/333734950 >> v2:172.21.15.144:3300/0 conn(0x7f80380afee0 0x7f80380b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 -- 172.21.15.60:0/333734950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f803815b010 msgr2=0x7f80380ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 --2- 172.21.15.60:0/333734950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f803815b010 0x7f80380ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 -- 172.21.15.60:0/333734950 >> v2:172.21.15.144:3300/0 conn(0x7f80380afee0 msgr2=0x7f80380b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 --2- 172.21.15.60:0/333734950 >> v2:172.21.15.144:3300/0 conn(0x7f80380afee0 0x7f80380b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.355+0000 7f803dbfe640 1 -- 172.21.15.60:0/333734950 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80380ae270 con 0x7f80380cf020 2023-10-16T12:33:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.357+0000 7f803dbfe640 1 --2- 172.21.15.60:0/333734950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380cf020 0x7f80380cf4b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8038070170 tx=0x7f803002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9145ed3b469e6642 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.357+0000 7f802ffff640 1 -- 172.21.15.60:0/333734950 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803002cb00 con 0x7f80380cf020 2023-10-16T12:33:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.357+0000 7f802ffff640 1 -- 172.21.15.60:0/333734950 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803002cc60 con 0x7f80380cf020 2023-10-16T12:33:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.357+0000 7f802ffff640 1 -- 172.21.15.60:0/333734950 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80300343f0 con 0x7f80380cf020 2023-10-16T12:33:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.357+0000 7f803f661640 1 -- 172.21.15.60:0/333734950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380cf020 msgr2=0x7f80380cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.357+0000 7f803f661640 1 --2- 172.21.15.60:0/333734950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380cf020 0x7f80380cf4b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8038070170 tx=0x7f803002bf60 comp rx=0 tx=0).stop 2023-10-16T12:33:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 -- 172.21.15.60:0/333734950 shutdown_connections 2023-10-16T12:33:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 --2- 172.21.15.60:0/333734950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380cf020 0x7f80380cf4b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 --2- 172.21.15.60:0/333734950 >> v2:172.21.15.144:3300/0 conn(0x7f80380afee0 0x7f80380b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 --2- 172.21.15.60:0/333734950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f803815b010 0x7f80380ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 -- 172.21.15.60:0/333734950 >> 172.21.15.60:0/333734950 conn(0x7f80380d0ad0 msgr2=0x7f80380d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 -- 172.21.15.60:0/333734950 shutdown_connections 2023-10-16T12:33:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.358+0000 7f803f661640 1 -- 172.21.15.60:0/333734950 wait complete. 2023-10-16T12:33:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.360+0000 7f803f661640 1 Processor -- start 2023-10-16T12:33:30.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.360+0000 7f803f661640 1 -- start start 2023-10-16T12:33:30.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803f661640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 0x7f80381f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803f661640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80380cf020 0x7f80380c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803f661640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803815b010 0x7f80380c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803f661640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380c7fa0 con 0x7f80380afee0 2023-10-16T12:33:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803f661640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380c8280 con 0x7f803815b010 2023-10-16T12:33:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803f661640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80380c8560 con 0x7f80380cf020 2023-10-16T12:33:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803d3fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 0x7f80381f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803d3fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 0x7f80381f42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57744/0 (socket says 172.21.15.60:57744) 2023-10-16T12:33:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803d3fd640 1 -- 172.21.15.60:0/1860665910 learned_addr learned my addr 172.21.15.60:0/1860665910 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.361+0000 7f803cbfc640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80380cf020 0x7f80380c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803dbfe640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803815b010 0x7f80380c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803d3fd640 1 -- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80380cf020 msgr2=0x7f80380c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803d3fd640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80380cf020 0x7f80380c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803d3fd640 1 -- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803815b010 msgr2=0x7f80380c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803d3fd640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803815b010 0x7f80380c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803d3fd640 1 -- 172.21.15.60:0/1860665910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8030009b30 con 0x7f80380afee0 2023-10-16T12:33:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803dbfe640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803815b010 0x7f80380c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803d3fd640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 0x7f80381f42d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f803400d7d0 tx=0x7f803400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:30.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80340096c0 con 0x7f80380afee0 2023-10-16T12:33:30.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8034009820 con 0x7f80380afee0 2023-10-16T12:33:30.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034009bd0 con 0x7f80380afee0 2023-10-16T12:33:30.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.362+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80380c8840 con 0x7f80380afee0 2023-10-16T12:33:30.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.363+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80380b6150 con 0x7f80380afee0 2023-10-16T12:33:30.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.364+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803804e160 con 0x7f80380afee0 2023-10-16T12:33:30.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.365+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8034009d30 con 0x7f80380afee0 2023-10-16T12:33:30.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.366+0000 7f802dffb640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f801c06ae70 0x7f801c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.366+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8034089d60 con 0x7f80380afee0 2023-10-16T12:33:30.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.366+0000 7f803cbfc640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f801c06ae70 0x7f801c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.367+0000 7f803cbfc640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f801c06ae70 0x7f801c06d330 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8028006fd0 tx=0x7f8028008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:30.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.370+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8034054e10 con 0x7f80380afee0 2023-10-16T12:33:30.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.561+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80380ae830 con 0x7f801c06ae70 2023-10-16T12:33:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.582+0000 7f802dffb640 1 -- 172.21.15.60:0/1860665910 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264513 (secure 0 0 0) 0x7f802800f830 con 0x7f801c06ae70 2023-10-16T12:33:30.587 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:30.589 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:33:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f801c06ae70 msgr2=0x7f801c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f801c06ae70 0x7f801c06d330 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8028006fd0 tx=0x7f8028008040 comp rx=0 tx=0).stop 2023-10-16T12:33:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 msgr2=0x7f80381f42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 0x7f80381f42d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f803400d7d0 tx=0x7f803400dc90 comp rx=0 tx=0).stop 2023-10-16T12:33:30.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 shutdown_connections 2023-10-16T12:33:30.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f801c06ae70 0x7f801c06d330 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80380cf020 0x7f80380c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803815b010 0x7f80380c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 --2- 172.21.15.60:0/1860665910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80380afee0 0x7f80381f42d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.587+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 >> 172.21.15.60:0/1860665910 conn(0x7f80380d0ad0 msgr2=0x7f803815b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.588+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 shutdown_connections 2023-10-16T12:33:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.588+0000 7f803f661640 1 -- 172.21.15.60:0/1860665910 wait complete. 2023-10-16T12:33:30.610 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-16T12:33:29.733092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236216,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738504,"statfs":{"total":773094113280,"available":772852228096,"internally_reserved":0,"allocated":3055616,"data_stored":1816468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":80,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002238"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930864+0000","last_change":"2023-10-16T12:33:24.620784+0000","last_active":"2023-10-16T12:33:25.930864+0000","last_peered":"2023-10-16T12:33:25.930864+0000","last_clean":"2023-10-16T12:33:25.930864+0000","last_became_active":"2023-10-16T12:33:24.620205+0000","last_became_peered":"2023-10-16T12:33:24.620205+0000","last_unstale":"2023-10-16T12:33:25.930864+0000","last_undegraded":"2023-10-16T12:33:25.930864+0000","last_fullsized":"2023-10-16T12:33:25.930864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585097+0000","last_change":"2023-10-16T12:33:24.608395+0000","last_active":"2023-10-16T12:33:25.585097+0000","last_peered":"2023-10-16T12:33:25.585097+0000","last_clean":"2023-10-16T12:33:25.585097+0000","last_became_active":"2023-10-16T12:33:24.608062+0000","last_became_peered":"2023-10-16T12:33:24.608062+0000","last_unstale":"2023-10-16T12:33:25.585097+0000","last_undegraded":"2023-10-16T12:33:25.585097+0000","last_fullsized":"2023-10-16T12:33:25.585097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928883+0000","last_change":"2023-10-16T12:33:24.607434+0000","last_active":"2023-10-16T12:33:25.928883+0000","last_peered":"2023-10-16T12:33:25.928883+0000","last_clean":"2023-10-16T12:33:25.928883+0000","last_became_active":"2023-10-16T12:33:24.607119+0000","last_became_peered":"2023-10-16T12:33:24.607119+0000","last_unstale":"2023-10-16T12:33:25.928883+0000","last_undegraded":"2023-10-16T12:33:25.928883+0000","last_fullsized":"2023-10-16T12:33:25.928883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.592697+0000","last_change":"2023-10-16T12:33:24.611057+0000","last_active":"2023-10-16T12:33:25.592697+0000","last_peered":"2023-10-16T12:33:25.592697+0000","last_clean":"2023-10-16T12:33:25.592697+0000","last_became_active":"2023-10-16T12:33:24.610357+0000","last_became_peered":"2023-10-16T12:33:24.610357+0000","last_unstale":"2023-10-16T12:33:25.592697+0000","last_undegraded":"2023-10-16T12:33:25.592697+0000","last_fullsized":"2023-10-16T12:33:25.592697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930042+0000","last_change":"2023-10-16T12:33:24.617198+0000","last_active":"2023-10-16T12:33:25.930042+0000","last_peered":"2023-10-16T12:33:25.930042+0000","last_clean":"2023-10-16T12:33:25.930042+0000","last_became_active":"2023-10-16T12:33:24.616710+0000","last_became_peered":"2023-10-16T12:33:24.616710+0000","last_unstale":"2023-10-16T12:33:25.930042+0000","last_undegraded":"2023-10-16T12:33:25.930042+0000","last_fullsized":"2023-10-16T12:33:25.930042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585757+0000","last_change":"2023-10-16T12:33:24.608373+0000","last_active":"2023-10-16T12:33:25.585757+0000","last_peered":"2023-10-16T12:33:25.585757+0000","last_clean":"2023-10-16T12:33:25.585757+0000","last_became_active":"2023-10-16T12:33:24.608083+0000","last_became_peered":"2023-10-16T12:33:24.608083+0000","last_unstale":"2023-10-16T12:33:25.585757+0000","last_undegraded":"2023-10-16T12:33:25.585757+0000","last_fullsized":"2023-10-16T12:33:25.585757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928757+0000","last_change":"2023-10-16T12:33:24.620178+0000","last_active":"2023-10-16T12:33:25.928757+0000","last_peered":"2023-10-16T12:33:25.928757+0000","last_clean":"2023-10-16T12:33:25.928757+0000","last_became_active":"2023-10-16T12:33:24.619871+0000","last_became_peered":"2023-10-16T12:33:24.619871+0000","last_unstale":"2023-10-16T12:33:25.928757+0000","last_undegraded":"2023-10-16T12:33:25.928757+0000","last_fullsized":"2023-10-16T12:33:25.928757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929201+0000","last_change":"2023-10-16T12:33:24.610760+0000","last_active":"2023-10-16T12:33:25.929201+0000","last_peered":"2023-10-16T12:33:25.929201+0000","last_clean":"2023-10-16T12:33:25.929201+0000","last_became_active":"2023-10-16T12:33:24.609947+0000","last_became_peered":"2023-10-16T12:33:24.609947+0000","last_unstale":"2023-10-16T12:33:25.929201+0000","last_undegraded":"2023-10-16T12:33:25.929201+0000","last_fullsized":"2023-10-16T12:33:25.929201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930940+0000","last_change":"2023-10-16T12:33:24.626800+0000","last_active":"2023-10-16T12:33:25.930940+0000","last_peered":"2023-10-16T12:33:25.930940+0000","last_clean":"2023-10-16T12:33:25.930940+0000","last_became_active":"2023-10-16T12:33:24.626566+0000","last_became_peered":"2023-10-16T12:33:24.626566+0000","last_unstale":"2023-10-16T12:33:25.930940+0000","last_undegraded":"2023-10-16T12:33:25.930940+0000","last_fullsized":"2023-10-16T12:33:25.930940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:08.131810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928952+0000","last_change":"2023-10-16T12:33:24.610841+0000","last_active":"2023-10-16T12:33:25.928952+0000","last_peered":"2023-10-16T12:33:25.928952+0000","last_clean":"2023-10-16T12:33:25.928952+0000","last_became_active":"2023-10-16T12:33:24.610533+0000","last_became_peered":"2023-10-16T12:33:24.610533+0000","last_unstale":"2023-10-16T12:33:25.928952+0000","last_undegraded":"2023-10-16T12:33:25.928952+0000","last_fullsized":"2023-10-16T12:33:25.928952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:39.121403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928420+0000","last_change":"2023-10-16T12:33:24.609014+0000","last_active":"2023-10-16T12:33:25.928420+0000","last_peered":"2023-10-16T12:33:25.928420+0000","last_clean":"2023-10-16T12:33:25.928420+0000","last_became_active":"2023-10-16T12:33:24.607622+0000","last_became_peered":"2023-10-16T12:33:24.607622+0000","last_unstale":"2023-10-16T12:33:25.928420+0000","last_undegraded":"2023-10-16T12:33:25.928420+0000","last_fullsized":"2023-10-16T12:33:25.928420+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:00.436524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588652+0000","last_change":"2023-10-16T12:33:24.609916+0000","last_active":"2023-10-16T12:33:25.588652+0000","last_peered":"2023-10-16T12:33:25.588652+0000","last_clean":"2023-10-16T12:33:25.588652+0000","last_became_active":"2023-10-16T12:33:24.609319+0000","last_became_peered":"2023-10-16T12:33:24.609319+0000","last_unstale":"2023-10-16T12:33:25.588652+0000","last_undegraded":"2023-10-16T12:33:25.588652+0000","last_fullsized":"2023-10-16T12:33:25.588652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:04.453258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929179+0000","last_change":"2023-10-16T12:33:24.622611+0000","last_active":"2023-10-16T12:33:25.929179+0000","last_peered":"2023-10-16T12:33:25.929179+0000","last_clean":"2023-10-16T12:33:25.929179+0000","last_became_active":"2023-10-16T12:33:24.622083+0000","last_became_peered":"2023-10-16T12:33:24.622083+0000","last_unstale":"2023-10-16T12:33:25.929179+0000","last_undegraded":"2023-10-16T12:33:25.929179+0000","last_fullsized":"2023-10-16T12:33:25.929179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927071+0000","last_change":"2023-10-16T12:33:24.605142+0000","last_active":"2023-10-16T12:33:25.927071+0000","last_peered":"2023-10-16T12:33:25.927071+0000","last_clean":"2023-10-16T12:33:25.927071+0000","last_became_active":"2023-10-16T12:33:24.604653+0000","last_became_peered":"2023-10-16T12:33:24.604653+0000","last_unstale":"2023-10-16T12:33:25.927071+0000","last_undegraded":"2023-10-16T12:33:25.927071+0000","last_fullsized":"2023-10-16T12:33:25.927071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:42.877240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584708+0000","last_change":"2023-10-16T12:33:24.606078+0000","last_active":"2023-10-16T12:33:25.584708+0000","last_peered":"2023-10-16T12:33:25.584708+0000","last_clean":"2023-10-16T12:33:25.584708+0000","last_became_active":"2023-10-16T12:33:24.605741+0000","last_became_peered":"2023-10-16T12:33:24.605741+0000","last_unstale":"2023-10-16T12:33:25.584708+0000","last_undegraded":"2023-10-16T12:33:25.584708+0000","last_fullsized":"2023-10-16T12:33:25.584708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584951+0000","last_change":"2023-10-16T12:33:24.606094+0000","last_active":"2023-10-16T12:33:25.584951+0000","last_peered":"2023-10-16T12:33:25.584951+0000","last_clean":"2023-10-16T12:33:25.584951+0000","last_became_active":"2023-10-16T12:33:24.605770+0000","last_became_peered":"2023-10-16T12:33:24.605770+0000","last_unstale":"2023-10-16T12:33:25.584951+0000","last_undegraded":"2023-10-16T12:33:25.584951+0000","last_fullsized":"2023-10-16T12:33:25.584951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590756+0000","last_change":"2023-10-16T12:33:24.618354+0000","last_active":"2023-10-16T12:33:25.590756+0000","last_peered":"2023-10-16T12:33:25.590756+0000","last_clean":"2023-10-16T12:33:25.590756+0000","last_became_active":"2023-10-16T12:33:24.617040+0000","last_became_peered":"2023-10-16T12:33:24.617040+0000","last_unstale":"2023-10-16T12:33:25.590756+0000","last_undegraded":"2023-10-16T12:33:25.590756+0000","last_fullsized":"2023-10-16T12:33:25.590756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927146+0000","last_change":"2023-10-16T12:33:24.612736+0000","last_active":"2023-10-16T12:33:25.927146+0000","last_peered":"2023-10-16T12:33:25.927146+0000","last_clean":"2023-10-16T12:33:25.927146+0000","last_became_active":"2023-10-16T12:33:24.612452+0000","last_became_peered":"2023-10-16T12:33:24.612452+0000","last_unstale":"2023-10-16T12:33:25.927146+0000","last_undegraded":"2023-10-16T12:33:25.927146+0000","last_fullsized":"2023-10-16T12:33:25.927146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930486+0000","last_change":"2023-10-16T12:33:24.621402+0000","last_active":"2023-10-16T12:33:25.930486+0000","last_peered":"2023-10-16T12:33:25.930486+0000","last_clean":"2023-10-16T12:33:25.930486+0000","last_became_active":"2023-10-16T12:33:24.621020+0000","last_became_peered":"2023-10-16T12:33:24.621020+0000","last_unstale":"2023-10-16T12:33:25.930486+0000","last_undegraded":"2023-10-16T12:33:25.930486+0000","last_fullsized":"2023-10-16T12:33:25.930486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928170+0000","last_change":"2023-10-16T12:33:24.613689+0000","last_active":"2023-10-16T12:33:25.928170+0000","last_peered":"2023-10-16T12:33:25.928170+0000","last_clean":"2023-10-16T12:33:25.928170+0000","last_became_active":"2023-10-16T12:33:24.613366+0000","last_became_peered":"2023-10-16T12:33:24.613366+0000","last_unstale":"2023-10-16T12:33:25.928170+0000","last_undegraded":"2023-10-16T12:33:25.928170+0000","last_fullsized":"2023-10-16T12:33:25.928170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:46.931083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928066+0000","last_change":"2023-10-16T12:33:24.615117+0000","last_active":"2023-10-16T12:33:25.928066+0000","last_peered":"2023-10-16T12:33:25.928066+0000","last_clean":"2023-10-16T12:33:25.928066+0000","last_became_active":"2023-10-16T12:33:24.614794+0000","last_became_peered":"2023-10-16T12:33:24.614794+0000","last_unstale":"2023-10-16T12:33:25.928066+0000","last_undegraded":"2023-10-16T12:33:25.928066+0000","last_fullsized":"2023-10-16T12:33:25.928066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586082+0000","last_change":"2023-10-16T12:33:24.616251+0000","last_active":"2023-10-16T12:33:25.586082+0000","last_peered":"2023-10-16T12:33:25.586082+0000","last_clean":"2023-10-16T12:33:25.586082+0000","last_became_active":"2023-10-16T12:33:24.615946+0000","last_became_peered":"2023-10-16T12:33:24.615946+0000","last_unstale":"2023-10-16T12:33:25.586082+0000","last_undegraded":"2023-10-16T12:33:25.586082+0000","last_fullsized":"2023-10-16T12:33:25.586082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929157+0000","last_change":"2023-10-16T12:33:24.613346+0000","last_active":"2023-10-16T12:33:25.929157+0000","last_peered":"2023-10-16T12:33:25.929157+0000","last_clean":"2023-10-16T12:33:25.929157+0000","last_became_active":"2023-10-16T12:33:24.612974+0000","last_became_peered":"2023-10-16T12:33:24.612974+0000","last_unstale":"2023-10-16T12:33:25.929157+0000","last_undegraded":"2023-10-16T12:33:25.929157+0000","last_fullsized":"2023-10-16T12:33:25.929157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585129+0000","last_change":"2023-10-16T12:33:24.624851+0000","last_active":"2023-10-16T12:33:25.585129+0000","last_peered":"2023-10-16T12:33:25.585129+0000","last_clean":"2023-10-16T12:33:25.585129+0000","last_became_active":"2023-10-16T12:33:24.624550+0000","last_became_peered":"2023-10-16T12:33:24.624550+0000","last_unstale":"2023-10-16T12:33:25.585129+0000","last_undegraded":"2023-10-16T12:33:25.585129+0000","last_fullsized":"2023-10-16T12:33:25.585129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930566+0000","last_change":"2023-10-16T12:33:24.614362+0000","last_active":"2023-10-16T12:33:25.930566+0000","last_peered":"2023-10-16T12:33:25.930566+0000","last_clean":"2023-10-16T12:33:25.930566+0000","last_became_active":"2023-10-16T12:33:24.613880+0000","last_became_peered":"2023-10-16T12:33:24.613880+0000","last_unstale":"2023-10-16T12:33:25.930566+0000","last_undegraded":"2023-10-16T12:33:25.930566+0000","last_fullsized":"2023-10-16T12:33:25.930566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929120+0000","last_change":"2023-10-16T12:33:24.618147+0000","last_active":"2023-10-16T12:33:25.929120+0000","last_peered":"2023-10-16T12:33:25.929120+0000","last_clean":"2023-10-16T12:33:25.929120+0000","last_became_active":"2023-10-16T12:33:24.617724+0000","last_became_peered":"2023-10-16T12:33:24.617724+0000","last_unstale":"2023-10-16T12:33:25.929120+0000","last_undegraded":"2023-10-16T12:33:25.929120+0000","last_fullsized":"2023-10-16T12:33:25.929120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:52.314433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585125+0000","last_change":"2023-10-16T12:33:24.605102+0000","last_active":"2023-10-16T12:33:25.585125+0000","last_peered":"2023-10-16T12:33:25.585125+0000","last_clean":"2023-10-16T12:33:25.585125+0000","last_became_active":"2023-10-16T12:33:24.604498+0000","last_became_peered":"2023-10-16T12:33:24.604498+0000","last_unstale":"2023-10-16T12:33:25.585125+0000","last_undegraded":"2023-10-16T12:33:25.585125+0000","last_fullsized":"2023-10-16T12:33:25.585125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930199+0000","last_change":"2023-10-16T12:33:24.613950+0000","last_active":"2023-10-16T12:33:25.930199+0000","last_peered":"2023-10-16T12:33:25.930199+0000","last_clean":"2023-10-16T12:33:25.930199+0000","last_became_active":"2023-10-16T12:33:24.613512+0000","last_became_peered":"2023-10-16T12:33:24.613512+0000","last_unstale":"2023-10-16T12:33:25.930199+0000","last_undegraded":"2023-10-16T12:33:25.930199+0000","last_fullsized":"2023-10-16T12:33:25.930199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928442+0000","last_change":"2023-10-16T12:33:24.612300+0000","last_active":"2023-10-16T12:33:25.928442+0000","last_peered":"2023-10-16T12:33:25.928442+0000","last_clean":"2023-10-16T12:33:25.928442+0000","last_became_active":"2023-10-16T12:33:24.611773+0000","last_became_peered":"2023-10-16T12:33:24.611773+0000","last_unstale":"2023-10-16T12:33:25.928442+0000","last_undegraded":"2023-10-16T12:33:25.928442+0000","last_fullsized":"2023-10-16T12:33:25.928442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590561+0000","last_change":"2023-10-16T12:33:24.615637+0000","last_active":"2023-10-16T12:33:25.590561+0000","last_peered":"2023-10-16T12:33:25.590561+0000","last_clean":"2023-10-16T12:33:25.590561+0000","last_became_active":"2023-10-16T12:33:24.615341+0000","last_became_peered":"2023-10-16T12:33:24.615341+0000","last_unstale":"2023-10-16T12:33:25.590561+0000","last_undegraded":"2023-10-16T12:33:25.590561+0000","last_fullsized":"2023-10-16T12:33:25.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928067+0000","last_change":"2023-10-16T12:33:24.603765+0000","last_active":"2023-10-16T12:33:25.928067+0000","last_peered":"2023-10-16T12:33:25.928067+0000","last_clean":"2023-10-16T12:33:25.928067+0000","last_became_active":"2023-10-16T12:33:24.603469+0000","last_became_peered":"2023-10-16T12:33:24.603469+0000","last_unstale":"2023-10-16T12:33:25.928067+0000","last_undegraded":"2023-10-16T12:33:25.928067+0000","last_fullsized":"2023-10-16T12:33:25.928067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927844+0000","last_change":"2023-10-16T12:33:24.617779+0000","last_active":"2023-10-16T12:33:25.927844+0000","last_peered":"2023-10-16T12:33:25.927844+0000","last_clean":"2023-10-16T12:33:25.927844+0000","last_became_active":"2023-10-16T12:33:24.617328+0000","last_became_peered":"2023-10-16T12:33:24.617328+0000","last_unstale":"2023-10-16T12:33:25.927844+0000","last_undegraded":"2023-10-16T12:33:25.927844+0000","last_fullsized":"2023-10-16T12:33:25.927844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:26.089073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.587918+0000","last_change":"2023-10-16T12:33:24.605171+0000","last_active":"2023-10-16T12:33:25.587918+0000","last_peered":"2023-10-16T12:33:25.587918+0000","last_clean":"2023-10-16T12:33:25.587918+0000","last_became_active":"2023-10-16T12:33:24.604706+0000","last_became_peered":"2023-10-16T12:33:24.604706+0000","last_unstale":"2023-10-16T12:33:25.587918+0000","last_undegraded":"2023-10-16T12:33:25.587918+0000","last_fullsized":"2023-10-16T12:33:25.587918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:45.448070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929923+0000","last_change":"2023-10-16T12:33:23.983081+0000","last_active":"2023-10-16T12:33:25.929923+0000","last_peered":"2023-10-16T12:33:25.929923+0000","last_clean":"2023-10-16T12:33:25.929923+0000","last_became_active":"2023-10-16T12:33:23.979378+0000","last_became_peered":"2023-10-16T12:33:23.979378+0000","last_unstale":"2023-10-16T12:33:25.929923+0000","last_undegraded":"2023-10-16T12:33:25.929923+0000","last_fullsized":"2023-10-16T12:33:25.929923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928531+0000","last_change":"2023-10-16T12:33:24.612688+0000","last_active":"2023-10-16T12:33:25.928531+0000","last_peered":"2023-10-16T12:33:25.928531+0000","last_clean":"2023-10-16T12:33:25.928531+0000","last_became_active":"2023-10-16T12:33:24.612202+0000","last_became_peered":"2023-10-16T12:33:24.612202+0000","last_unstale":"2023-10-16T12:33:25.928531+0000","last_undegraded":"2023-10-16T12:33:25.928531+0000","last_fullsized":"2023-10-16T12:33:25.928531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929373+0000","last_change":"2023-10-16T12:33:23.983502+0000","last_active":"2023-10-16T12:33:25.929373+0000","last_peered":"2023-10-16T12:33:25.929373+0000","last_clean":"2023-10-16T12:33:25.929373+0000","last_became_active":"2023-10-16T12:33:23.983189+0000","last_became_peered":"2023-10-16T12:33:23.983189+0000","last_unstale":"2023-10-16T12:33:25.929373+0000","last_undegraded":"2023-10-16T12:33:25.929373+0000","last_fullsized":"2023-10-16T12:33:25.929373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.912908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929270+0000","last_change":"2023-10-16T12:33:23.984326+0000","last_active":"2023-10-16T12:33:25.929270+0000","last_peered":"2023-10-16T12:33:25.929270+0000","last_clean":"2023-10-16T12:33:25.929270+0000","last_became_active":"2023-10-16T12:33:23.984015+0000","last_became_peered":"2023-10-16T12:33:23.984015+0000","last_unstale":"2023-10-16T12:33:25.929270+0000","last_undegraded":"2023-10-16T12:33:25.929270+0000","last_fullsized":"2023-10-16T12:33:25.929270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929604+0000","last_change":"2023-10-16T12:33:24.615262+0000","last_active":"2023-10-16T12:33:25.929604+0000","last_peered":"2023-10-16T12:33:25.929604+0000","last_clean":"2023-10-16T12:33:25.929604+0000","last_became_active":"2023-10-16T12:33:24.610418+0000","last_became_peered":"2023-10-16T12:33:24.610418+0000","last_unstale":"2023-10-16T12:33:25.929604+0000","last_undegraded":"2023-10-16T12:33:25.929604+0000","last_fullsized":"2023-10-16T12:33:25.929604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585175+0000","last_change":"2023-10-16T12:33:23.976361+0000","last_active":"2023-10-16T12:33:25.585175+0000","last_peered":"2023-10-16T12:33:25.585175+0000","last_clean":"2023-10-16T12:33:25.585175+0000","last_became_active":"2023-10-16T12:33:23.975488+0000","last_became_peered":"2023-10-16T12:33:23.975488+0000","last_unstale":"2023-10-16T12:33:25.585175+0000","last_undegraded":"2023-10-16T12:33:25.585175+0000","last_fullsized":"2023-10-16T12:33:25.585175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584519+0000","last_change":"2023-10-16T12:33:24.603552+0000","last_active":"2023-10-16T12:33:25.584519+0000","last_peered":"2023-10-16T12:33:25.584519+0000","last_clean":"2023-10-16T12:33:25.584519+0000","last_became_active":"2023-10-16T12:33:24.602661+0000","last_became_peered":"2023-10-16T12:33:24.602661+0000","last_unstale":"2023-10-16T12:33:25.584519+0000","last_undegraded":"2023-10-16T12:33:25.584519+0000","last_fullsized":"2023-10-16T12:33:25.584519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929244+0000","last_change":"2023-10-16T12:33:23.983406+0000","last_active":"2023-10-16T12:33:25.929244+0000","last_peered":"2023-10-16T12:33:25.929244+0000","last_clean":"2023-10-16T12:33:25.929244+0000","last_became_active":"2023-10-16T12:33:23.982570+0000","last_became_peered":"2023-10-16T12:33:23.982570+0000","last_unstale":"2023-10-16T12:33:25.929244+0000","last_undegraded":"2023-10-16T12:33:25.929244+0000","last_fullsized":"2023-10-16T12:33:25.929244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:17.276818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591933+0000","last_change":"2023-10-16T12:33:24.616611+0000","last_active":"2023-10-16T12:33:25.591933+0000","last_peered":"2023-10-16T12:33:25.591933+0000","last_clean":"2023-10-16T12:33:25.591933+0000","last_became_active":"2023-10-16T12:33:24.616318+0000","last_became_peered":"2023-10-16T12:33:24.616318+0000","last_unstale":"2023-10-16T12:33:25.591933+0000","last_undegraded":"2023-10-16T12:33:25.591933+0000","last_fullsized":"2023-10-16T12:33:25.591933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591819+0000","last_change":"2023-10-16T12:33:23.982425+0000","last_active":"2023-10-16T12:33:25.591819+0000","last_peered":"2023-10-16T12:33:25.591819+0000","last_clean":"2023-10-16T12:33:25.591819+0000","last_became_active":"2023-10-16T12:33:23.982159+0000","last_became_peered":"2023-10-16T12:33:23.982159+0000","last_unstale":"2023-10-16T12:33:25.591819+0000","last_undegraded":"2023-10-16T12:33:25.591819+0000","last_fullsized":"2023-10-16T12:33:25.591819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:00.917480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929362+0000","last_change":"2023-10-16T12:33:24.607789+0000","last_active":"2023-10-16T12:33:25.929362+0000","last_peered":"2023-10-16T12:33:25.929362+0000","last_clean":"2023-10-16T12:33:25.929362+0000","last_became_active":"2023-10-16T12:33:24.606558+0000","last_became_peered":"2023-10-16T12:33:24.606558+0000","last_unstale":"2023-10-16T12:33:25.929362+0000","last_undegraded":"2023-10-16T12:33:25.929362+0000","last_fullsized":"2023-10-16T12:33:25.929362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584911+0000","last_change":"2023-10-16T12:33:23.994926+0000","last_active":"2023-10-16T12:33:25.584911+0000","last_peered":"2023-10-16T12:33:25.584911+0000","last_clean":"2023-10-16T12:33:25.584911+0000","last_became_active":"2023-10-16T12:33:23.994616+0000","last_became_peered":"2023-10-16T12:33:23.994616+0000","last_unstale":"2023-10-16T12:33:25.584911+0000","last_undegraded":"2023-10-16T12:33:25.584911+0000","last_fullsized":"2023-10-16T12:33:25.584911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.615708+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.615262+0000","last_became_peered":"2023-10-16T12:33:24.615262+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928417+0000","last_change":"2023-10-16T12:33:23.990728+0000","last_active":"2023-10-16T12:33:25.928417+0000","last_peered":"2023-10-16T12:33:25.928417+0000","last_clean":"2023-10-16T12:33:25.928417+0000","last_became_active":"2023-10-16T12:33:23.988939+0000","last_became_peered":"2023-10-16T12:33:23.988939+0000","last_unstale":"2023-10-16T12:33:25.928417+0000","last_undegraded":"2023-10-16T12:33:25.928417+0000","last_fullsized":"2023-10-16T12:33:25.928417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:58.091012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930079+0000","last_change":"2023-10-16T12:33:24.620766+0000","last_active":"2023-10-16T12:33:25.930079+0000","last_peered":"2023-10-16T12:33:25.930079+0000","last_clean":"2023-10-16T12:33:25.930079+0000","last_became_active":"2023-10-16T12:33:24.620453+0000","last_became_peered":"2023-10-16T12:33:24.620453+0000","last_unstale":"2023-10-16T12:33:25.930079+0000","last_undegraded":"2023-10-16T12:33:25.930079+0000","last_fullsized":"2023-10-16T12:33:25.930079+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928702+0000","last_change":"2023-10-16T12:33:23.968343+0000","last_active":"2023-10-16T12:33:25.928702+0000","last_peered":"2023-10-16T12:33:25.928702+0000","last_clean":"2023-10-16T12:33:25.928702+0000","last_became_active":"2023-10-16T12:33:23.967874+0000","last_became_peered":"2023-10-16T12:33:23.967874+0000","last_unstale":"2023-10-16T12:33:25.928702+0000","last_undegraded":"2023-10-16T12:33:25.928702+0000","last_fullsized":"2023-10-16T12:33:25.928702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927246+0000","last_change":"2023-10-16T12:33:24.615841+0000","last_active":"2023-10-16T12:33:25.927246+0000","last_peered":"2023-10-16T12:33:25.927246+0000","last_clean":"2023-10-16T12:33:25.927246+0000","last_became_active":"2023-10-16T12:33:24.615333+0000","last_became_peered":"2023-10-16T12:33:24.615333+0000","last_unstale":"2023-10-16T12:33:25.927246+0000","last_undegraded":"2023-10-16T12:33:25.927246+0000","last_fullsized":"2023-10-16T12:33:25.927246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:23.519029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590504+0000","last_change":"2023-10-16T12:33:23.984016+0000","last_active":"2023-10-16T12:33:25.590504+0000","last_peered":"2023-10-16T12:33:25.590504+0000","last_clean":"2023-10-16T12:33:25.590504+0000","last_became_active":"2023-10-16T12:33:23.983727+0000","last_became_peered":"2023-10-16T12:33:23.983727+0000","last_unstale":"2023-10-16T12:33:25.590504+0000","last_undegraded":"2023-10-16T12:33:25.590504+0000","last_fullsized":"2023-10-16T12:33:25.590504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930468+0000","last_change":"2023-10-16T12:33:24.623341+0000","last_active":"2023-10-16T12:33:25.930468+0000","last_peered":"2023-10-16T12:33:25.930468+0000","last_clean":"2023-10-16T12:33:25.930468+0000","last_became_active":"2023-10-16T12:33:24.622356+0000","last_became_peered":"2023-10-16T12:33:24.622356+0000","last_unstale":"2023-10-16T12:33:25.930468+0000","last_undegraded":"2023-10-16T12:33:25.930468+0000","last_fullsized":"2023-10-16T12:33:25.930468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927892+0000","last_change":"2023-10-16T12:33:23.982823+0000","last_active":"2023-10-16T12:33:25.927892+0000","last_peered":"2023-10-16T12:33:25.927892+0000","last_clean":"2023-10-16T12:33:25.927892+0000","last_became_active":"2023-10-16T12:33:23.981883+0000","last_became_peered":"2023-10-16T12:33:23.981883+0000","last_unstale":"2023-10-16T12:33:25.927892+0000","last_undegraded":"2023-10-16T12:33:25.927892+0000","last_fullsized":"2023-10-16T12:33:25.927892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:09.201117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927851+0000","last_change":"2023-10-16T12:33:24.617583+0000","last_active":"2023-10-16T12:33:25.927851+0000","last_peered":"2023-10-16T12:33:25.927851+0000","last_clean":"2023-10-16T12:33:25.927851+0000","last_became_active":"2023-10-16T12:33:24.616992+0000","last_became_peered":"2023-10-16T12:33:24.616992+0000","last_unstale":"2023-10-16T12:33:25.927851+0000","last_undegraded":"2023-10-16T12:33:25.927851+0000","last_fullsized":"2023-10-16T12:33:25.927851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:38.188085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.926869+0000","last_change":"2023-10-16T12:33:23.988300+0000","last_active":"2023-10-16T12:33:25.926869+0000","last_peered":"2023-10-16T12:33:25.926869+0000","last_clean":"2023-10-16T12:33:25.926869+0000","last_became_active":"2023-10-16T12:33:23.987831+0000","last_became_peered":"2023-10-16T12:33:23.987831+0000","last_unstale":"2023-10-16T12:33:25.926869+0000","last_undegraded":"2023-10-16T12:33:25.926869+0000","last_fullsized":"2023-10-16T12:33:25.926869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928930+0000","last_change":"2023-10-16T12:33:24.613800+0000","last_active":"2023-10-16T12:33:25.928930+0000","last_peered":"2023-10-16T12:33:25.928930+0000","last_clean":"2023-10-16T12:33:25.928930+0000","last_became_active":"2023-10-16T12:33:24.613268+0000","last_became_peered":"2023-10-16T12:33:24.613268+0000","last_unstale":"2023-10-16T12:33:25.928930+0000","last_undegraded":"2023-10-16T12:33:25.928930+0000","last_fullsized":"2023-10-16T12:33:25.928930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588350+0000","last_change":"2023-10-16T12:33:23.970854+0000","last_active":"2023-10-16T12:33:25.588350+0000","last_peered":"2023-10-16T12:33:25.588350+0000","last_clean":"2023-10-16T12:33:25.588350+0000","last_became_active":"2023-10-16T12:33:23.970073+0000","last_became_peered":"2023-10-16T12:33:23.970073+0000","last_unstale":"2023-10-16T12:33:25.588350+0000","last_undegraded":"2023-10-16T12:33:25.588350+0000","last_fullsized":"2023-10-16T12:33:25.588350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930633+0000","last_change":"2023-10-16T12:33:24.624669+0000","last_active":"2023-10-16T12:33:25.930633+0000","last_peered":"2023-10-16T12:33:25.930633+0000","last_clean":"2023-10-16T12:33:25.930633+0000","last_became_active":"2023-10-16T12:33:24.624315+0000","last_became_peered":"2023-10-16T12:33:24.624315+0000","last_unstale":"2023-10-16T12:33:25.930633+0000","last_undegraded":"2023-10-16T12:33:25.930633+0000","last_fullsized":"2023-10-16T12:33:25.930633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927431+0000","last_change":"2023-10-16T12:33:23.980566+0000","last_active":"2023-10-16T12:33:25.927431+0000","last_peered":"2023-10-16T12:33:25.927431+0000","last_clean":"2023-10-16T12:33:25.927431+0000","last_became_active":"2023-10-16T12:33:23.980200+0000","last_became_peered":"2023-10-16T12:33:23.980200+0000","last_unstale":"2023-10-16T12:33:25.927431+0000","last_undegraded":"2023-10-16T12:33:25.927431+0000","last_fullsized":"2023-10-16T12:33:25.927431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.931065+0000","last_change":"2023-10-16T12:33:24.624902+0000","last_active":"2023-10-16T12:33:25.931065+0000","last_peered":"2023-10-16T12:33:25.931065+0000","last_clean":"2023-10-16T12:33:25.931065+0000","last_became_active":"2023-10-16T12:33:24.624646+0000","last_became_peered":"2023-10-16T12:33:24.624646+0000","last_unstale":"2023-10-16T12:33:25.931065+0000","last_undegraded":"2023-10-16T12:33:25.931065+0000","last_fullsized":"2023-10-16T12:33:25.931065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:21.560247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585130+0000","last_change":"2023-10-16T12:33:23.984004+0000","last_active":"2023-10-16T12:33:25.585130+0000","last_peered":"2023-10-16T12:33:25.585130+0000","last_clean":"2023-10-16T12:33:25.585130+0000","last_became_active":"2023-10-16T12:33:23.982385+0000","last_became_peered":"2023-10-16T12:33:23.982385+0000","last_unstale":"2023-10-16T12:33:25.585130+0000","last_undegraded":"2023-10-16T12:33:25.585130+0000","last_fullsized":"2023-10-16T12:33:25.585130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586032+0000","last_change":"2023-10-16T12:33:24.611253+0000","last_active":"2023-10-16T12:33:25.586032+0000","last_peered":"2023-10-16T12:33:25.586032+0000","last_clean":"2023-10-16T12:33:25.586032+0000","last_became_active":"2023-10-16T12:33:24.610712+0000","last_became_peered":"2023-10-16T12:33:24.610712+0000","last_unstale":"2023-10-16T12:33:25.586032+0000","last_undegraded":"2023-10-16T12:33:25.586032+0000","last_fullsized":"2023-10-16T12:33:25.586032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927465+0000","last_change":"2023-10-16T12:33:23.985969+0000","last_active":"2023-10-16T12:33:25.927465+0000","last_peered":"2023-10-16T12:33:25.927465+0000","last_clean":"2023-10-16T12:33:25.927465+0000","last_became_active":"2023-10-16T12:33:23.985467+0000","last_became_peered":"2023-10-16T12:33:23.985467+0000","last_unstale":"2023-10-16T12:33:25.927465+0000","last_undegraded":"2023-10-16T12:33:25.927465+0000","last_fullsized":"2023-10-16T12:33:25.927465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930068+0000","last_change":"2023-10-16T12:33:24.627563+0000","last_active":"2023-10-16T12:33:25.930068+0000","last_peered":"2023-10-16T12:33:25.930068+0000","last_clean":"2023-10-16T12:33:25.930068+0000","last_became_active":"2023-10-16T12:33:24.627241+0000","last_became_peered":"2023-10-16T12:33:24.627241+0000","last_unstale":"2023-10-16T12:33:25.930068+0000","last_undegraded":"2023-10-16T12:33:25.930068+0000","last_fullsized":"2023-10-16T12:33:25.930068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-16T12:33:24.598233+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:33:24.598233+0000","last_peered":"2023-10-16T12:33:24.598233+0000","last_clean":"2023-10-16T12:33:24.598233+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:33:24.598233+0000","last_undegraded":"2023-10-16T12:33:24.598233+0000","last_fullsized":"2023-10-16T12:33:24.598233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585104+0000","last_change":"2023-10-16T12:33:23.972294+0000","last_active":"2023-10-16T12:33:25.585104+0000","last_peered":"2023-10-16T12:33:25.585104+0000","last_clean":"2023-10-16T12:33:25.585104+0000","last_became_active":"2023-10-16T12:33:23.971886+0000","last_became_peered":"2023-10-16T12:33:23.971886+0000","last_unstale":"2023-10-16T12:33:25.585104+0000","last_undegraded":"2023-10-16T12:33:25.585104+0000","last_fullsized":"2023-10-16T12:33:25.585104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.874107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929522+0000","last_change":"2023-10-16T12:33:24.604564+0000","last_active":"2023-10-16T12:33:25.929522+0000","last_peered":"2023-10-16T12:33:25.929522+0000","last_clean":"2023-10-16T12:33:25.929522+0000","last_became_active":"2023-10-16T12:33:24.604240+0000","last_became_peered":"2023-10-16T12:33:24.604240+0000","last_unstale":"2023-10-16T12:33:25.929522+0000","last_undegraded":"2023-10-16T12:33:25.929522+0000","last_fullsized":"2023-10-16T12:33:25.929522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927720+0000","last_change":"2023-10-16T12:33:23.988943+0000","last_active":"2023-10-16T12:33:25.927720+0000","last_peered":"2023-10-16T12:33:25.927720+0000","last_clean":"2023-10-16T12:33:25.927720+0000","last_became_active":"2023-10-16T12:33:23.988631+0000","last_became_peered":"2023-10-16T12:33:23.988631+0000","last_unstale":"2023-10-16T12:33:25.927720+0000","last_undegraded":"2023-10-16T12:33:25.927720+0000","last_fullsized":"2023-10-16T12:33:25.927720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927155+0000","last_change":"2023-10-16T12:33:24.615033+0000","last_active":"2023-10-16T12:33:25.927155+0000","last_peered":"2023-10-16T12:33:25.927155+0000","last_clean":"2023-10-16T12:33:25.927155+0000","last_became_active":"2023-10-16T12:33:24.614433+0000","last_became_peered":"2023-10-16T12:33:24.614433+0000","last_unstale":"2023-10-16T12:33:25.927155+0000","last_undegraded":"2023-10-16T12:33:25.927155+0000","last_fullsized":"2023-10-16T12:33:25.927155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:38.208712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928316+0000","last_change":"2023-10-16T12:33:23.978410+0000","last_active":"2023-10-16T12:33:25.928316+0000","last_peered":"2023-10-16T12:33:25.928316+0000","last_clean":"2023-10-16T12:33:25.928316+0000","last_became_active":"2023-10-16T12:33:23.978050+0000","last_became_peered":"2023-10-16T12:33:23.978050+0000","last_unstale":"2023-10-16T12:33:25.928316+0000","last_undegraded":"2023-10-16T12:33:25.928316+0000","last_fullsized":"2023-10-16T12:33:25.928316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584693+0000","last_change":"2023-10-16T12:33:24.600057+0000","last_active":"2023-10-16T12:33:25.584693+0000","last_peered":"2023-10-16T12:33:25.584693+0000","last_clean":"2023-10-16T12:33:25.584693+0000","last_became_active":"2023-10-16T12:33:24.599294+0000","last_became_peered":"2023-10-16T12:33:24.599294+0000","last_unstale":"2023-10-16T12:33:25.584693+0000","last_undegraded":"2023-10-16T12:33:25.584693+0000","last_fullsized":"2023-10-16T12:33:25.584693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590705+0000","last_change":"2023-10-16T12:33:23.990485+0000","last_active":"2023-10-16T12:33:25.590705+0000","last_peered":"2023-10-16T12:33:25.590705+0000","last_clean":"2023-10-16T12:33:25.590705+0000","last_became_active":"2023-10-16T12:33:23.989836+0000","last_became_peered":"2023-10-16T12:33:23.989836+0000","last_unstale":"2023-10-16T12:33:25.590705+0000","last_undegraded":"2023-10-16T12:33:25.590705+0000","last_fullsized":"2023-10-16T12:33:25.590705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.589289+0000","last_change":"2023-10-16T12:33:24.619525+0000","last_active":"2023-10-16T12:33:25.589289+0000","last_peered":"2023-10-16T12:33:25.589289+0000","last_clean":"2023-10-16T12:33:25.589289+0000","last_became_active":"2023-10-16T12:33:24.619242+0000","last_became_peered":"2023-10-16T12:33:24.619242+0000","last_unstale":"2023-10-16T12:33:25.589289+0000","last_undegraded":"2023-10-16T12:33:25.589289+0000","last_fullsized":"2023-10-16T12:33:25.589289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:12.969746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584815+0000","last_change":"2023-10-16T12:33:23.976427+0000","last_active":"2023-10-16T12:33:25.584815+0000","last_peered":"2023-10-16T12:33:25.584815+0000","last_clean":"2023-10-16T12:33:25.584815+0000","last_became_active":"2023-10-16T12:33:23.975234+0000","last_became_peered":"2023-10-16T12:33:23.975234+0000","last_unstale":"2023-10-16T12:33:25.584815+0000","last_undegraded":"2023-10-16T12:33:25.584815+0000","last_fullsized":"2023-10-16T12:33:25.584815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590492+0000","last_change":"2023-10-16T12:33:24.614482+0000","last_active":"2023-10-16T12:33:25.590492+0000","last_peered":"2023-10-16T12:33:25.590492+0000","last_clean":"2023-10-16T12:33:25.590492+0000","last_became_active":"2023-10-16T12:33:24.614144+0000","last_became_peered":"2023-10-16T12:33:24.614144+0000","last_unstale":"2023-10-16T12:33:25.590492+0000","last_undegraded":"2023-10-16T12:33:25.590492+0000","last_fullsized":"2023-10-16T12:33:25.590492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930752+0000","last_change":"2023-10-16T12:33:23.986466+0000","last_active":"2023-10-16T12:33:25.930752+0000","last_peered":"2023-10-16T12:33:25.930752+0000","last_clean":"2023-10-16T12:33:25.930752+0000","last_became_active":"2023-10-16T12:33:23.985508+0000","last_became_peered":"2023-10-16T12:33:23.985508+0000","last_unstale":"2023-10-16T12:33:25.930752+0000","last_undegraded":"2023-10-16T12:33:25.930752+0000","last_fullsized":"2023-10-16T12:33:25.930752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:08.057398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584833+0000","last_change":"2023-10-16T12:33:24.605662+0000","last_active":"2023-10-16T12:33:25.584833+0000","last_peered":"2023-10-16T12:33:25.584833+0000","last_clean":"2023-10-16T12:33:25.584833+0000","last_became_active":"2023-10-16T12:33:24.605319+0000","last_became_peered":"2023-10-16T12:33:24.605319+0000","last_unstale":"2023-10-16T12:33:25.584833+0000","last_undegraded":"2023-10-16T12:33:25.584833+0000","last_fullsized":"2023-10-16T12:33:25.584833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585096+0000","last_change":"2023-10-16T12:33:23.986752+0000","last_active":"2023-10-16T12:33:25.585096+0000","last_peered":"2023-10-16T12:33:25.585096+0000","last_clean":"2023-10-16T12:33:25.585096+0000","last_became_active":"2023-10-16T12:33:23.986189+0000","last_became_peered":"2023-10-16T12:33:23.986189+0000","last_unstale":"2023-10-16T12:33:25.585096+0000","last_undegraded":"2023-10-16T12:33:25.585096+0000","last_fullsized":"2023-10-16T12:33:25.585096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930740+0000","last_change":"2023-10-16T12:33:24.616668+0000","last_active":"2023-10-16T12:33:25.930740+0000","last_peered":"2023-10-16T12:33:25.930740+0000","last_clean":"2023-10-16T12:33:25.930740+0000","last_became_active":"2023-10-16T12:33:24.616308+0000","last_became_peered":"2023-10-16T12:33:24.616308+0000","last_unstale":"2023-10-16T12:33:25.930740+0000","last_undegraded":"2023-10-16T12:33:25.930740+0000","last_fullsized":"2023-10-16T12:33:25.930740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929376+0000","last_change":"2023-10-16T12:33:23.991338+0000","last_active":"2023-10-16T12:33:25.929376+0000","last_peered":"2023-10-16T12:33:25.929376+0000","last_clean":"2023-10-16T12:33:25.929376+0000","last_became_active":"2023-10-16T12:33:23.986162+0000","last_became_peered":"2023-10-16T12:33:23.986162+0000","last_unstale":"2023-10-16T12:33:25.929376+0000","last_undegraded":"2023-10-16T12:33:25.929376+0000","last_fullsized":"2023-10-16T12:33:25.929376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:32.667813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928205+0000","last_change":"2023-10-16T12:33:24.612178+0000","last_active":"2023-10-16T12:33:25.928205+0000","last_peered":"2023-10-16T12:33:25.928205+0000","last_clean":"2023-10-16T12:33:25.928205+0000","last_became_active":"2023-10-16T12:33:24.608876+0000","last_became_peered":"2023-10-16T12:33:24.608876+0000","last_unstale":"2023-10-16T12:33:25.928205+0000","last_undegraded":"2023-10-16T12:33:25.928205+0000","last_fullsized":"2023-10-16T12:33:25.928205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927671+0000","last_change":"2023-10-16T12:33:23.983928+0000","last_active":"2023-10-16T12:33:25.927671+0000","last_peered":"2023-10-16T12:33:25.927671+0000","last_clean":"2023-10-16T12:33:25.927671+0000","last_became_active":"2023-10-16T12:33:23.983266+0000","last_became_peered":"2023-10-16T12:33:23.983266+0000","last_unstale":"2023-10-16T12:33:25.927671+0000","last_undegraded":"2023-10-16T12:33:25.927671+0000","last_fullsized":"2023-10-16T12:33:25.927671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.605248+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.604915+0000","last_became_peered":"2023-10-16T12:33:24.604915+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584607+0000","last_change":"2023-10-16T12:33:23.987636+0000","last_active":"2023-10-16T12:33:25.584607+0000","last_peered":"2023-10-16T12:33:25.584607+0000","last_clean":"2023-10-16T12:33:25.584607+0000","last_became_active":"2023-10-16T12:33:23.987422+0000","last_became_peered":"2023-10-16T12:33:23.987422+0000","last_unstale":"2023-10-16T12:33:25.584607+0000","last_undegraded":"2023-10-16T12:33:25.584607+0000","last_fullsized":"2023-10-16T12:33:25.584607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930014+0000","last_change":"2023-10-16T12:33:24.607092+0000","last_active":"2023-10-16T12:33:25.930014+0000","last_peered":"2023-10-16T12:33:25.930014+0000","last_clean":"2023-10-16T12:33:25.930014+0000","last_became_active":"2023-10-16T12:33:24.606816+0000","last_became_peered":"2023-10-16T12:33:24.606816+0000","last_unstale":"2023-10-16T12:33:25.930014+0000","last_undegraded":"2023-10-16T12:33:25.930014+0000","last_fullsized":"2023-10-16T12:33:25.930014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584703+0000","last_change":"2023-10-16T12:33:23.982671+0000","last_active":"2023-10-16T12:33:25.584703+0000","last_peered":"2023-10-16T12:33:25.584703+0000","last_clean":"2023-10-16T12:33:25.584703+0000","last_became_active":"2023-10-16T12:33:23.982314+0000","last_became_peered":"2023-10-16T12:33:23.982314+0000","last_unstale":"2023-10-16T12:33:25.584703+0000","last_undegraded":"2023-10-16T12:33:25.584703+0000","last_fullsized":"2023-10-16T12:33:25.584703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930259+0000","last_change":"2023-10-16T12:33:24.614159+0000","last_active":"2023-10-16T12:33:25.930259+0000","last_peered":"2023-10-16T12:33:25.930259+0000","last_clean":"2023-10-16T12:33:25.930259+0000","last_became_active":"2023-10-16T12:33:24.613857+0000","last_became_peered":"2023-10-16T12:33:24.613857+0000","last_unstale":"2023-10-16T12:33:25.930259+0000","last_undegraded":"2023-10-16T12:33:25.930259+0000","last_fullsized":"2023-10-16T12:33:25.930259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930386+0000","last_change":"2023-10-16T12:33:23.987798+0000","last_active":"2023-10-16T12:33:25.930386+0000","last_peered":"2023-10-16T12:33:25.930386+0000","last_clean":"2023-10-16T12:33:25.930386+0000","last_became_active":"2023-10-16T12:33:23.987460+0000","last_became_peered":"2023-10-16T12:33:23.987460+0000","last_unstale":"2023-10-16T12:33:25.930386+0000","last_undegraded":"2023-10-16T12:33:25.930386+0000","last_fullsized":"2023-10-16T12:33:25.930386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929124+0000","last_change":"2023-10-16T12:33:24.615457+0000","last_active":"2023-10-16T12:33:25.929124+0000","last_peered":"2023-10-16T12:33:25.929124+0000","last_clean":"2023-10-16T12:33:25.929124+0000","last_became_active":"2023-10-16T12:33:24.615166+0000","last_became_peered":"2023-10-16T12:33:24.615166+0000","last_unstale":"2023-10-16T12:33:25.929124+0000","last_undegraded":"2023-10-16T12:33:25.929124+0000","last_fullsized":"2023-10-16T12:33:25.929124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:24.606678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586074+0000","last_change":"2023-10-16T12:33:24.604607+0000","last_active":"2023-10-16T12:33:25.586074+0000","last_peered":"2023-10-16T12:33:25.586074+0000","last_clean":"2023-10-16T12:33:25.586074+0000","last_became_active":"2023-10-16T12:33:24.604183+0000","last_became_peered":"2023-10-16T12:33:24.604183+0000","last_unstale":"2023-10-16T12:33:25.586074+0000","last_undegraded":"2023-10-16T12:33:25.586074+0000","last_fullsized":"2023-10-16T12:33:25.586074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929969+0000","last_change":"2023-10-16T12:33:23.990373+0000","last_active":"2023-10-16T12:33:25.929969+0000","last_peered":"2023-10-16T12:33:25.929969+0000","last_clean":"2023-10-16T12:33:25.929969+0000","last_became_active":"2023-10-16T12:33:23.990049+0000","last_became_peered":"2023-10-16T12:33:23.990049+0000","last_unstale":"2023-10-16T12:33:25.929969+0000","last_undegraded":"2023-10-16T12:33:25.929969+0000","last_fullsized":"2023-10-16T12:33:25.929969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585658+0000","last_change":"2023-10-16T12:33:24.612037+0000","last_active":"2023-10-16T12:33:25.585658+0000","last_peered":"2023-10-16T12:33:25.585658+0000","last_clean":"2023-10-16T12:33:25.585658+0000","last_became_active":"2023-10-16T12:33:24.611609+0000","last_became_peered":"2023-10-16T12:33:24.611609+0000","last_unstale":"2023-10-16T12:33:25.585658+0000","last_undegraded":"2023-10-16T12:33:25.585658+0000","last_fullsized":"2023-10-16T12:33:25.585658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930569+0000","last_change":"2023-10-16T12:33:23.985742+0000","last_active":"2023-10-16T12:33:25.930569+0000","last_peered":"2023-10-16T12:33:25.930569+0000","last_clean":"2023-10-16T12:33:25.930569+0000","last_became_active":"2023-10-16T12:33:23.985053+0000","last_became_peered":"2023-10-16T12:33:23.985053+0000","last_unstale":"2023-10-16T12:33:25.930569+0000","last_undegraded":"2023-10-16T12:33:25.930569+0000","last_fullsized":"2023-10-16T12:33:25.930569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:43.191029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927970+0000","last_change":"2023-10-16T12:33:24.610651+0000","last_active":"2023-10-16T12:33:25.927970+0000","last_peered":"2023-10-16T12:33:25.927970+0000","last_clean":"2023-10-16T12:33:25.927970+0000","last_became_active":"2023-10-16T12:33:24.605512+0000","last_became_peered":"2023-10-16T12:33:24.605512+0000","last_unstale":"2023-10-16T12:33:25.927970+0000","last_undegraded":"2023-10-16T12:33:25.927970+0000","last_fullsized":"2023-10-16T12:33:25.927970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930559+0000","last_change":"2023-10-16T12:33:23.981596+0000","last_active":"2023-10-16T12:33:25.930559+0000","last_peered":"2023-10-16T12:33:25.930559+0000","last_clean":"2023-10-16T12:33:25.930559+0000","last_became_active":"2023-10-16T12:33:23.981314+0000","last_became_peered":"2023-10-16T12:33:23.981314+0000","last_unstale":"2023-10-16T12:33:25.930559+0000","last_undegraded":"2023-10-16T12:33:25.930559+0000","last_fullsized":"2023-10-16T12:33:25.930559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930671+0000","last_change":"2023-10-16T12:33:24.613513+0000","last_active":"2023-10-16T12:33:25.930671+0000","last_peered":"2023-10-16T12:33:25.930671+0000","last_clean":"2023-10-16T12:33:25.930671+0000","last_became_active":"2023-10-16T12:33:24.613158+0000","last_became_peered":"2023-10-16T12:33:24.613158+0000","last_unstale":"2023-10-16T12:33:25.930671+0000","last_undegraded":"2023-10-16T12:33:25.930671+0000","last_fullsized":"2023-10-16T12:33:25.930671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929580+0000","last_change":"2023-10-16T12:33:23.978618+0000","last_active":"2023-10-16T12:33:25.929580+0000","last_peered":"2023-10-16T12:33:25.929580+0000","last_clean":"2023-10-16T12:33:25.929580+0000","last_became_active":"2023-10-16T12:33:23.978325+0000","last_became_peered":"2023-10-16T12:33:23.978325+0000","last_unstale":"2023-10-16T12:33:25.929580+0000","last_undegraded":"2023-10-16T12:33:25.929580+0000","last_fullsized":"2023-10-16T12:33:25.929580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.544}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9379999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":2,"up_from":14,"seq":60129542147,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3340000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.077}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542147,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":831488,"data_stored":669870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:33:30.616 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:33:30.616 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-16T12:33:30.617 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-16T12:33:30.617 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T12:33:30.618 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-16T12:33:30.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.883+0000 7f479c87c640 1 Processor -- start 2023-10-16T12:33:30.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.883+0000 7f479c87c640 1 -- start start 2023-10-16T12:33:30.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.885+0000 7f479c87c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4798161490 0x7f47980ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.885+0000 7f479c87c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479806b5c0 0x7f47980ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.885+0000 7f479c87c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980ce610 con 0x7f47980afc70 2023-10-16T12:33:30.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.885+0000 7f479c87c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47980ad760 con 0x7f4798161490 2023-10-16T12:33:30.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.885+0000 7f479c87c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47980ada40 con 0x7f479806b5c0 2023-10-16T12:33:30.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.885+0000 7f478dd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f47980afc70 0x7f479806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54154/0 (socket says 172.21.15.60:54154) 2023-10-16T12:33:30.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f478dd9b640 1 -- 172.21.15.60:0/1832746550 learned_addr learned my addr 172.21.15.60:0/1832746550 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:30.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f479659c640 1 --2- 172.21.15.60:0/1832746550 >> v2:172.21.15.60:3300/0 conn(0x7f4798161490 0x7f47980ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f4796d9d640 1 --2- 172.21.15.60:0/1832746550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479806b5c0 0x7f47980ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f4795d9b640 1 -- 172.21.15.60:0/1832746550 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125289026 0 0) 0x7f47980ce610 con 0x7f47980afc70 2023-10-16T12:33:30.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f4795d9b640 1 -- 172.21.15.60:0/1832746550 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4784003520 con 0x7f47980afc70 2023-10-16T12:33:30.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f479659c640 1 -- 172.21.15.60:0/1832746550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479806b5c0 msgr2=0x7f47980ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:30.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f479659c640 1 --2- 172.21.15.60:0/1832746550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479806b5c0 0x7f47980ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f479659c640 1 -- 172.21.15.60:0/1832746550 >> v1:172.21.15.60:6789/0 conn(0x7f47980afc70 legacy=0x7f479806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.886+0000 7f479659c640 1 -- 172.21.15.60:0/1832746550 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980add20 con 0x7f4798161490 2023-10-16T12:33:30.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.887+0000 7f479659c640 1 --2- 172.21.15.60:0/1832746550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4798161490 0x7f47980ccab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4788009790 tx=0x7f478802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6418c4379b92cd76 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:30.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f4795d9b640 1 -- 172.21.15.60:0/1832746550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788031800 con 0x7f4798161490 2023-10-16T12:33:30.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f4795d9b640 1 -- 172.21.15.60:0/1832746550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788031960 con 0x7f4798161490 2023-10-16T12:33:30.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f4795d9b640 1 -- 172.21.15.60:0/1832746550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788031cf0 con 0x7f4798161490 2023-10-16T12:33:30.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 -- 172.21.15.60:0/1832746550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4798161490 msgr2=0x7f47980ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 --2- 172.21.15.60:0/1832746550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4798161490 0x7f47980ccab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4788009790 tx=0x7f478802b8a0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 -- 172.21.15.60:0/1832746550 shutdown_connections 2023-10-16T12:33:30.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 --2- 172.21.15.60:0/1832746550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479806b5c0 0x7f47980ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 --2- 172.21.15.60:0/1832746550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4798161490 0x7f47980ccab0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 -- 172.21.15.60:0/1832746550 >> 172.21.15.60:0/1832746550 conn(0x7f47980d0ad0 msgr2=0x7f47980d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:30.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.888+0000 7f479c87c640 1 -- 172.21.15.60:0/1832746550 shutdown_connections 2023-10-16T12:33:30.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.889+0000 7f479c87c640 1 -- 172.21.15.60:0/1832746550 wait complete. 2023-10-16T12:33:30.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.890+0000 7f479c87c640 1 Processor -- start 2023-10-16T12:33:30.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.890+0000 7f479c87c640 1 -- start start 2023-10-16T12:33:30.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.891+0000 7f479c87c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 0x7f47980c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.891+0000 7f479c87c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47980afc70 0x7f47980b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.891+0000 7f479c87c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4798161490 0x7f479815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.891+0000 7f479c87c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47980ca4b0 con 0x7f479806b5c0 2023-10-16T12:33:30.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.891+0000 7f479c87c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47980b0050 con 0x7f47980afc70 2023-10-16T12:33:30.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.891+0000 7f479c87c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47980c9ef0 con 0x7f4798161490 2023-10-16T12:33:30.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 0x7f47980c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 0x7f47980c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57772/0 (socket says 172.21.15.60:57772) 2023-10-16T12:33:30.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 -- 172.21.15.60:0/4228576287 learned_addr learned my addr 172.21.15.60:0/4228576287 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:30.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 -- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4798161490 msgr2=0x7f479815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f4796d9d640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4798161490 0x7f479815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4798161490 0x7f479815b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 -- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47980afc70 msgr2=0x7f47980b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:30.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f478dd9b640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47980afc70 0x7f47980b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47980afc70 0x7f47980b6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:30.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 -- 172.21.15.60:0/4228576287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880093a0 con 0x7f479806b5c0 2023-10-16T12:33:30.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f478dd9b640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47980afc70 0x7f47980b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:30.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.892+0000 7f479659c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 0x7f47980c97f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4788034040 tx=0x7f47880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:30.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.893+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788005da0 con 0x7f479806b5c0 2023-10-16T12:33:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.893+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478803a850 con 0x7f479806b5c0 2023-10-16T12:33:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.893+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478803ac00 con 0x7f479806b5c0 2023-10-16T12:33:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.893+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47980ca1d0 con 0x7f479806b5c0 2023-10-16T12:33:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.893+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47980b0540 con 0x7f479806b5c0 2023-10-16T12:33:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.894+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479804e160 con 0x7f479806b5c0 2023-10-16T12:33:30.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.895+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4788006030 con 0x7f479806b5c0 2023-10-16T12:33:30.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.896+0000 7f478f7fe640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f477806aea0 0x7f477806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:30.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.896+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f478802e070 con 0x7f479806b5c0 2023-10-16T12:33:30.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.897+0000 7f478dd9b640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f477806aea0 0x7f477806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:30.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.898+0000 7f478dd9b640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f477806aea0 0x7f477806d360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47980cca50 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:30.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:30.901+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4788076d10 con 0x7f479806b5c0 2023-10-16T12:33:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.147+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f47980b2ef0 con 0x7f479806b5c0 2023-10-16T12:33:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.152+0000 7f478f7fe640 1 -- 172.21.15.60:0/4228576287 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171037 (secure 0 0 0) 0x7f478804a020 con 0x7f479806b5c0 2023-10-16T12:33:31.157 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f477806aea0 msgr2=0x7f477806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f477806aea0 0x7f477806d360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47980cca50 tx=0x7f4780009040 comp rx=0 tx=0).stop 2023-10-16T12:33:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 msgr2=0x7f47980c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 0x7f47980c97f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4788034040 tx=0x7f47880046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 shutdown_connections 2023-10-16T12:33:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f477806aea0 0x7f477806d360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4798161490 0x7f479815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47980afc70 0x7f47980b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 --2- 172.21.15.60:0/4228576287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f479806b5c0 0x7f47980c97f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 >> 172.21.15.60:0/4228576287 conn(0x7f47980d0ad0 msgr2=0x7f4798160510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 shutdown_connections 2023-10-16T12:33:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.157+0000 7f479c87c640 1 -- 172.21.15.60:0/4228576287 wait complete. 2023-10-16T12:33:31.180 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":1973903962},{"type":"v1","addr":"172.21.15.144:6833","nonce":1973903962}]},"active_addr":"172.21.15.144:6833/1973903962","active_change":"2023-10-16T12:33:09.706176+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4098,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3946760462}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":2619332853}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1199236332}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1714573694}]}]} 2023-10-16T12:33:31.184 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-16T12:33:31.185 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-16T12:33:31.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.407+0000 7f539a494640 1 Processor -- start 2023-10-16T12:33:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.407+0000 7f539a494640 1 -- start start 2023-10-16T12:33:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.409+0000 7f539a494640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 0x7f539415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.409+0000 7f539a494640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f53940ce390 0x7f5394070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.409+0000 7f539a494640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539415ed60 con 0x7f539415aaa0 2023-10-16T12:33:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.409+0000 7f539a494640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539406b810 con 0x7f53940ce390 2023-10-16T12:33:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.409+0000 7f539a494640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53940707e0 con 0x7f539415ae80 2023-10-16T12:33:31.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.410+0000 7f53937fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 0x7f539415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.410+0000 7f5398a31640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f53940ce390 0x7f5394070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.410+0000 7f5398a31640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f53940ce390 0x7f5394070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:48856/0 (socket says 172.21.15.60:48856) 2023-10-16T12:33:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.410+0000 7f5398a31640 1 -- 172.21.15.60:0/2002931233 learned_addr learned my addr 172.21.15.60:0/2002931233 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.410+0000 7f53937fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 0x7f539415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53072/0 (socket says 172.21.15.60:53072) 2023-10-16T12:33:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.411+0000 7f5398a31640 1 -- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 msgr2=0x7f539415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.411+0000 7f5398a31640 1 --2- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 0x7f539415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.411+0000 7f5398a31640 1 -- 172.21.15.60:0/2002931233 >> v1:172.21.15.144:6789/0 conn(0x7f539415aaa0 legacy=0x7f53940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.411+0000 7f5398a31640 1 -- 172.21.15.60:0/2002931233 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539415ef90 con 0x7f53940ce390 2023-10-16T12:33:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.411+0000 7f53937fe640 1 --2- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 0x7f539415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:31.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.412+0000 7f5398a31640 1 --2- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53940ce390 0x7f5394070f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f538c009e50 tx=0x7f538c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25f7c907c57f9ce9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:31.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f5392ffd640 1 -- 172.21.15.60:0/2002931233 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c02cb00 con 0x7f53940ce390 2023-10-16T12:33:31.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f5392ffd640 1 -- 172.21.15.60:0/2002931233 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c02cc60 con 0x7f53940ce390 2023-10-16T12:33:31.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f5392ffd640 1 -- 172.21.15.60:0/2002931233 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0343f0 con 0x7f53940ce390 2023-10-16T12:33:31.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 -- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53940ce390 msgr2=0x7f5394070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 --2- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53940ce390 0x7f5394070f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f538c009e50 tx=0x7f538c02bf60 comp rx=0 tx=0).stop 2023-10-16T12:33:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 -- 172.21.15.60:0/2002931233 shutdown_connections 2023-10-16T12:33:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 --2- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53940ce390 0x7f5394070f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 --2- 172.21.15.60:0/2002931233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415ae80 0x7f539415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 -- 172.21.15.60:0/2002931233 >> 172.21.15.60:0/2002931233 conn(0x7f53940cb410 msgr2=0x7f53940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 -- 172.21.15.60:0/2002931233 shutdown_connections 2023-10-16T12:33:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.413+0000 7f539a494640 1 -- 172.21.15.60:0/2002931233 wait complete. 2023-10-16T12:33:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.415+0000 7f539a494640 1 Processor -- start 2023-10-16T12:33:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.415+0000 7f539a494640 1 -- start start 2023-10-16T12:33:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.416+0000 7f539a494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 0x7f53940c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.416+0000 7f539a494640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415aaa0 0x7f539414bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.416+0000 7f539a494640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f539415ae80 0x7f539415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.416+0000 7f539a494640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539415d480 con 0x7f53940ce390 2023-10-16T12:33:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.416+0000 7f539a494640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394158140 con 0x7f539415ae80 2023-10-16T12:33:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.416+0000 7f539a494640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539415c800 con 0x7f539415aaa0 2023-10-16T12:33:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 0x7f53940c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 0x7f53940c4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57780/0 (socket says 172.21.15.60:57780) 2023-10-16T12:33:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 -- 172.21.15.60:0/2912466530 learned_addr learned my addr 172.21.15.60:0/2912466530 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5398a31640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f539415ae80 0x7f539415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f53937fe640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415aaa0 0x7f539414bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 -- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415aaa0 msgr2=0x7f539414bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415aaa0 0x7f539414bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 -- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f539415ae80 msgr2=0x7f539415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f539415ae80 0x7f539415d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5393fff640 1 -- 172.21.15.60:0/2912466530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388009400 con 0x7f53940ce390 2023-10-16T12:33:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.417+0000 7f5398a31640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f539415ae80 0x7f539415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.418+0000 7f5393fff640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 0x7f53940c4ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f538400c450 tx=0x7f538400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.418+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53840052d0 con 0x7f53940ce390 2023-10-16T12:33:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.418+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5384005430 con 0x7f53940ce390 2023-10-16T12:33:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.418+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c009b30 con 0x7f53940ce390 2023-10-16T12:33:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.418+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384012870 con 0x7f53940ce390 2023-10-16T12:33:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.418+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539415cdc0 con 0x7f53940ce390 2023-10-16T12:33:31.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.419+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539415d5c0 con 0x7f53940ce390 2023-10-16T12:33:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.420+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f53840129d0 con 0x7f53940ce390 2023-10-16T12:33:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.421+0000 7f5390ff9640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f538006aec0 0x7f538006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.421+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5384089020 con 0x7f53940ce390 2023-10-16T12:33:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.422+0000 7f53937fe640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f538006aec0 0x7f538006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.423+0000 7f53937fe640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f538006aec0 0x7f538006d380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53940c2430 tx=0x7f5388008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:31.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.426+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f539415d5c0 con 0x7f53940ce390 2023-10-16T12:33:31.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.622+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f539415d5c0 con 0x7f53940ce390 2023-10-16T12:33:31.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.625+0000 7f5390ff9640 1 -- 172.21.15.60:0/2912466530 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f5384054240 con 0x7f53940ce390 2023-10-16T12:33:31.628 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:31.629 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:31.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f538006aec0 msgr2=0x7f538006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f538006aec0 0x7f538006d380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53940c2430 tx=0x7f5388008040 comp rx=0 tx=0).stop 2023-10-16T12:33:31.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 msgr2=0x7f53940c4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 0x7f53940c4ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f538400c450 tx=0x7f538400c910 comp rx=0 tx=0).stop 2023-10-16T12:33:31.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 shutdown_connections 2023-10-16T12:33:31.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f538006aec0 0x7f538006d380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f539415aaa0 0x7f539414bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f539415ae80 0x7f539415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 --2- 172.21.15.60:0/2912466530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53940ce390 0x7f53940c4ed0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.628+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 >> 172.21.15.60:0/2912466530 conn(0x7f53940cb410 msgr2=0x7f539406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:31.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.629+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 shutdown_connections 2023-10-16T12:33:31.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.629+0000 7f539a494640 1 -- 172.21.15.60:0/2912466530 wait complete. 2023-10-16T12:33:31.650 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-16T12:33:31.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:31.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.874+0000 7f8803d78640 1 Processor -- start 2023-10-16T12:33:31.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.875+0000 7f8803d78640 1 -- start start 2023-10-16T12:33:31.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.876+0000 7f8803d78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 0x7f87fc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.876+0000 7f8803d78640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc15ed60 con 0x7f87fc15ae80 2023-10-16T12:33:31.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.876+0000 7f8803d78640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0707e0 con 0x7f87fc15aaa0 2023-10-16T12:33:31.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.876+0000 7f8803d78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc06b810 con 0x7f87fc0ce390 2023-10-16T12:33:31.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.876+0000 7f8801313640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f87fc15ae80 0x7f87fc070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54166/0 (socket says 172.21.15.60:54166) 2023-10-16T12:33:31.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.876+0000 7f8801313640 1 -- 172.21.15.60:0/1799716492 learned_addr learned my addr 172.21.15.60:0/1799716492 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8802315640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 0x7f87fc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895561722 0 0) 0x7f87fc15ed60 con 0x7f87fc15ae80 2023-10-16T12:33:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e8003520 con 0x7f87fc15ae80 2023-10-16T12:33:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491728133 0 0) 0x7f87e8003520 con 0x7f87fc15ae80 2023-10-16T12:33:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 msgr2=0x7f87fc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 0x7f87fc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 >> v1:172.21.15.144:6789/0 conn(0x7f87fc15aaa0 legacy=0x7f87fc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc15ef90 con 0x7f87fc15ae80 2023-10-16T12:33:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f87ec001690 con 0x7f87fc15ae80 2023-10-16T12:33:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.877+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 >> v1:172.21.15.60:6789/0 conn(0x7f87fc15ae80 legacy=0x7f87fc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8800b12640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e80039a0 0x7f87e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8800b12640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8800b12640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 0x7f87e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc06b810 con 0x7f87e80047c0 2023-10-16T12:33:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e800ba30 con 0x7f87e80039a0 2023-10-16T12:33:31.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e800bbf0 con 0x7f87fc0ce390 2023-10-16T12:33:31.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801313640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801b14640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e80039a0 0x7f87e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801313640 1 -- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 msgr2=0x7f87e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:31.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801313640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0ce390 0x7f87e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801313640 1 -- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e80039a0 msgr2=0x7f87e8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:31.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801313640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e80039a0 0x7f87e8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801313640 1 -- 172.21.15.60:0/1799716492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e8003520 con 0x7f87e80047c0 2023-10-16T12:33:31.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.878+0000 7f8801b14640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e80039a0 0x7f87e8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:31.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.879+0000 7f8801313640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87e8004c40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f87ec001eb0 tx=0x7f87e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5231bae9376d8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:31.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec005ad0 con 0x7f87e80047c0 2023-10-16T12:33:31.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec005c30 con 0x7f87e80047c0 2023-10-16T12:33:31.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8800b12640 1 -- 172.21.15.60:0/1799716492 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec02c830 con 0x7f87e80047c0 2023-10-16T12:33:31.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8803d78640 1 -- 172.21.15.60:0/1799716492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 msgr2=0x7f87e8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8803d78640 1 --2- 172.21.15.60:0/1799716492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87e8004c40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f87ec001eb0 tx=0x7f87e80057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8802315640 1 -- 172.21.15.60:0/1799716492 reap_dead start 2023-10-16T12:33:31.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8803d78640 1 -- 172.21.15.60:0/1799716492 shutdown_connections 2023-10-16T12:33:31.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8803d78640 1 -- 172.21.15.60:0/1799716492 >> 172.21.15.60:0/1799716492 conn(0x7f87fc0cb410 msgr2=0x7f87fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:31.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8803d78640 1 -- 172.21.15.60:0/1799716492 shutdown_connections 2023-10-16T12:33:31.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.880+0000 7f8803d78640 1 -- 172.21.15.60:0/1799716492 wait complete. 2023-10-16T12:33:31.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.882+0000 7f8803d78640 1 Processor -- start 2023-10-16T12:33:31.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.882+0000 7f8803d78640 1 -- start start 2023-10-16T12:33:31.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8803d78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87fc0c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8803d78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0c5070 0x7f87fc14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8803d78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87fc15d660 0x7f87fc0af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8803d78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc15c690 con 0x7f87e80047c0 2023-10-16T12:33:31.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8803d78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc15c8a0 con 0x7f87fc15d660 2023-10-16T12:33:31.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8803d78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc15cb80 con 0x7f87fc0c5070 2023-10-16T12:33:31.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.883+0000 7f8801b14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87fc0c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87fc0c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57806/0 (socket says 172.21.15.60:57806) 2023-10-16T12:33:31.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 -- 172.21.15.60:0/2665718105 learned_addr learned my addr 172.21.15.60:0/2665718105 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:31.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8802315640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87fc15d660 0x7f87fc0af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 -- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0c5070 msgr2=0x7f87fc14bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0c5070 0x7f87fc14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 -- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87fc15d660 msgr2=0x7f87fc0af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:31.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87fc15d660 0x7f87fc0af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:31.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 -- 172.21.15.60:0/2665718105 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc15ce60 con 0x7f87e80047c0 2023-10-16T12:33:31.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f8801b14640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87fc0c24c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87f800d770 tx=0x7f87f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:31.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8005270 con 0x7f87e80047c0 2023-10-16T12:33:31.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f80053d0 con 0x7f87e80047c0 2023-10-16T12:33:31.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.884+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f800b8f0 con 0x7f87e80047c0 2023-10-16T12:33:31.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.885+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc15d140 con 0x7f87e80047c0 2023-10-16T12:33:31.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.885+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc14c240 con 0x7f87e80047c0 2023-10-16T12:33:31.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.886+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc04e160 con 0x7f87e80047c0 2023-10-16T12:33:31.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.887+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f87f800ba50 con 0x7f87e80047c0 2023-10-16T12:33:31.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.888+0000 7f87f27fc640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f87e006aea0 0x7f87e006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:31.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.888+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87f8089570 con 0x7f87e80047c0 2023-10-16T12:33:31.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.888+0000 7f8801313640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f87e006aea0 0x7f87e006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:31.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.889+0000 7f8801313640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f87e006aea0 0x7f87e006d360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f87ec034000 tx=0x7f87ec007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:31.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:31.892+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87f8054610 con 0x7f87e80047c0 2023-10-16T12:33:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.094+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87fc15e350 con 0x7f87e80047c0 2023-10-16T12:33:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.097+0000 7f87f27fc640 1 -- 172.21.15.60:0/2665718105 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f87f8019310 con 0x7f87e80047c0 2023-10-16T12:33:32.101 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:32.101 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.100+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f87e006aea0 msgr2=0x7f87e006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.100+0000 7f8803d78640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f87e006aea0 0x7f87e006d360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f87ec034000 tx=0x7f87ec007000 comp rx=0 tx=0).stop 2023-10-16T12:33:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.100+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 msgr2=0x7f87fc0c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.100+0000 7f8803d78640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87fc0c24c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87f800d770 tx=0x7f87f800dc30 comp rx=0 tx=0).stop 2023-10-16T12:33:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.100+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 shutdown_connections 2023-10-16T12:33:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f87e006aea0 0x7f87e006d360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87fc0c5070 0x7f87fc14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87fc15d660 0x7f87fc0af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 --2- 172.21.15.60:0/2665718105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80047c0 0x7f87fc0c24c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 >> 172.21.15.60:0/2665718105 conn(0x7f87fc0cb410 msgr2=0x7f87fc0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 shutdown_connections 2023-10-16T12:33:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.101+0000 7f8803d78640 1 -- 172.21.15.60:0/2665718105 wait complete. 2023-10-16T12:33:32.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-16T12:33:32.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-16T12:33:32.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-16T12:33:32.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-16T12:33:32.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-16T12:33:32.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-16T12:33:32.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-16T12:33:32.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-16T12:33:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.432+0000 7faa3149f640 1 Processor -- start 2023-10-16T12:33:32.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.432+0000 7faa3149f640 1 -- start start 2023-10-16T12:33:32.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.435+0000 7faa3149f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa24053d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.436+0000 7faa2bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7faa240e3190 0x7faa2403e920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60230/0 (socket says 172.21.15.60:60230) 2023-10-16T12:33:32.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.436+0000 7faa2bfff640 1 -- 172.21.15.60:0/1949061120 learned_addr learned my addr 172.21.15.60:0/1949061120 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:32.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.436+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa24006e80 con 0x7faa240542d0 2023-10-16T12:33:32.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.436+0000 7faa2b7fe640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa24053d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.436+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa24007160 con 0x7faa240e3190 2023-10-16T12:33:32.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.437+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2403f100 con 0x7faa24038180 2023-10-16T12:33:32.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.437+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113443665 0 0) 0x7faa24006e80 con 0x7faa240542d0 2023-10-16T12:33:32.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.437+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa1c003520 con 0x7faa240542d0 2023-10-16T12:33:32.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.437+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625944662 0 0) 0x7faa24007160 con 0x7faa240e3190 2023-10-16T12:33:32.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.437+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa1c0048a0 con 0x7faa240e3190 2023-10-16T12:33:32.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.438+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1537065168 0 0) 0x7faa1c003520 con 0x7faa240542d0 2023-10-16T12:33:32.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.438+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 msgr2=0x7faa24053d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.438+0000 7faa2affd640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa24053d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.438+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 >> v1:172.21.15.144:6789/0 conn(0x7faa240e3190 legacy=0x7faa2403e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.438+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24007440 con 0x7faa240542d0 2023-10-16T12:33:32.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.438+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7faa2c04e9b0 con 0x7faa240542d0 2023-10-16T12:33:32.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.439+0000 7faa2b7fe640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa24053d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:32.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.439+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 >> v1:172.21.15.60:6789/0 conn(0x7faa240542d0 legacy=0x7faa240573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.439+0000 7faa2affd640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa1c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.440+0000 7faa2affd640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa1c003760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.440+0000 7faa2affd640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa1c003ca0 0x7faa1c004120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.440+0000 7faa2bfff640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa1c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.440+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2403f100 con 0x7faa1c004d40 2023-10-16T12:33:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.440+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c00bb10 con 0x7faa1c003ca0 2023-10-16T12:33:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.441+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c00bca0 con 0x7faa24038180 2023-10-16T12:33:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.441+0000 7faa2b7fe640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa1c003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.441+0000 7faa30c9e640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa1c003ca0 0x7faa1c004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.443+0000 7faa2bfff640 1 -- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 msgr2=0x7faa1c003760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.443+0000 7faa2bfff640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa24038180 0x7faa1c003760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.443+0000 7faa2bfff640 1 -- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa1c003ca0 msgr2=0x7faa1c004120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.443+0000 7faa2bfff640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa1c003ca0 0x7faa1c004120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.443+0000 7faa2bfff640 1 -- 172.21.15.60:0/1949061120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa20009400 con 0x7faa1c004d40 2023-10-16T12:33:32.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.445+0000 7faa2bfff640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa1c005160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faa180019f0 tx=0x7faa1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876f0264f452751 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.445+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18035070 con 0x7faa1c004d40 2023-10-16T12:33:32.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.445+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa180048a0 con 0x7faa1c004d40 2023-10-16T12:33:32.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.446+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 msgr2=0x7faa1c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.446+0000 7faa2affd640 1 -- 172.21.15.60:0/1949061120 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18035070 con 0x7faa1c004d40 2023-10-16T12:33:32.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.446+0000 7faa3149f640 1 --2- 172.21.15.60:0/1949061120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa1c005160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faa180019f0 tx=0x7faa1c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.446+0000 7faa30c9e640 1 -- 172.21.15.60:0/1949061120 reap_dead start 2023-10-16T12:33:32.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.447+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 shutdown_connections 2023-10-16T12:33:32.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.447+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 >> 172.21.15.60:0/1949061120 conn(0x7faa240590c0 msgr2=0x7faa240594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.447+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 shutdown_connections 2023-10-16T12:33:32.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.447+0000 7faa3149f640 1 -- 172.21.15.60:0/1949061120 wait complete. 2023-10-16T12:33:32.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.450+0000 7faa3149f640 1 Processor -- start 2023-10-16T12:33:32.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.450+0000 7faa3149f640 1 -- start start 2023-10-16T12:33:32.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.450+0000 7f4dd1ebb640 1 Processor -- start 2023-10-16T12:33:32.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.450+0000 7f4dd1ebb640 1 -- start start 2023-10-16T12:33:32.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.452+0000 7faa3149f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa2404edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.452+0000 7faa3149f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa2416d4c0 0x7faa2416d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.452+0000 7faa3149f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa24168f20 0x7faa241693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.452+0000 7faa3149f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2416de80 con 0x7faa1c004d40 2023-10-16T12:33:32.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.452+0000 7f4dd1ebb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc06a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7f4dd1ebb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc0cc2b0 con 0x7f4dcc15cf90 2023-10-16T12:33:32.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7f4dd1ebb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc0cc590 con 0x7f4dcc06a670 2023-10-16T12:33:32.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7f4dd1ebb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0cc870 con 0x7f4dcc0cf200 2023-10-16T12:33:32.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa2404edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7f4dcbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4dcc15cf90 0x7f4dcc0cd350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54172/0 (socket says 172.21.15.60:54172) 2023-10-16T12:33:32.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7f4dcbfff640 1 -- 172.21.15.60:0/1573099578 learned_addr learned my addr 172.21.15.60:0/1573099578 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:32.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa2404edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57826/0 (socket says 172.21.15.60:57826) 2023-10-16T12:33:32.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa2bfff640 1 -- 172.21.15.60:0/2719090253 learned_addr learned my addr 172.21.15.60:0/2719090253 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa3149f640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa24169f70 con 0x7faa24168f20 2023-10-16T12:33:32.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa3149f640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa24169c90 con 0x7faa2416d4c0 2023-10-16T12:33:32.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa2b7fe640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa2416d4c0 0x7faa2416d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.453+0000 7faa30c9e640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa24168f20 0x7faa241693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.454+0000 7f4dcb7fe640 1 --2- 172.21.15.60:0/1573099578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc06a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.454+0000 7faa30c9e640 1 -- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa2416d4c0 msgr2=0x7faa2416d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.454+0000 7faa30c9e640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faa2416d4c0 0x7faa2416d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.454+0000 7faa30c9e640 1 -- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 msgr2=0x7faa2404edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.454+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072576129 0 0) 0x7f4dcc0cc2b0 con 0x7f4dcc15cf90 2023-10-16T12:33:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db8003520 con 0x7f4dcc15cf90 2023-10-16T12:33:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675915914 0 0) 0x7f4dcc0cc590 con 0x7f4dcc06a670 2023-10-16T12:33:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dcc0cc2b0 con 0x7f4dcc06a670 2023-10-16T12:33:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7fc4730d7640 1 Processor -- start 2023-10-16T12:33:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7faa30c9e640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa2404edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/1573099578 >> v1:172.21.15.144:6789/0 conn(0x7f4dcc06a670 legacy=0x7f4dcc15c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/1573099578 >> v1:172.21.15.60:6789/0 conn(0x7f4dcc15cf90 legacy=0x7f4dcc0cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/1573099578 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0ccb50 con 0x7f4dcc0cf200 2023-10-16T12:33:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7faa30c9e640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa240ce020 con 0x7faa24168f20 2023-10-16T12:33:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.455+0000 7faa2bfff640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa1c004d40 0x7faa2404edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.456+0000 7faa30c9e640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa24168f20 0x7faa241693a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faa2c069740 tx=0x7faa2c068000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.456+0000 7fc4730d7640 1 -- start start 2023-10-16T12:33:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.456+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c067330 con 0x7faa24168f20 2023-10-16T12:33:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.456+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c093470 con 0x7faa24168f20 2023-10-16T12:33:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.456+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c093860 con 0x7faa24168f20 2023-10-16T12:33:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.456+0000 7faa3149f640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa240ce300 con 0x7faa24168f20 2023-10-16T12:33:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.457+0000 7f4dcb7fe640 1 --2- 172.21.15.60:0/1573099578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc06a130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0097d0 tx=0x7f4dbc02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f1cac250a631009 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.457+0000 7faa3149f640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2404f310 con 0x7faa24168f20 2023-10-16T12:33:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc030800 con 0x7f4dcc0cf200 2023-10-16T12:33:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc030960 con 0x7f4dcc0cf200 2023-10-16T12:33:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7f4dcaffd640 1 -- 172.21.15.60:0/1573099578 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc030cf0 con 0x7f4dcc0cf200 2023-10-16T12:33:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7fc4730d7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc46c0addb0 0x7fc46c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7fc4730d7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc46c0c9000 0x7fc46c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7fc4730d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c06a2e0 0x7fc46c0cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7fc4730d7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46c161a50 con 0x7fc46c0addb0 2023-10-16T12:33:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7fc4730d7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46c06a730 con 0x7fc46c0c9000 2023-10-16T12:33:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.458+0000 7fc4730d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c0cb140 con 0x7fc46c06a2e0 2023-10-16T12:33:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.459+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faa04000fc0 con 0x7faa24168f20 2023-10-16T12:33:32.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.459+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/1573099578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 msgr2=0x7f4dcc06a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.459+0000 7f4dd1ebb640 1 --2- 172.21.15.60:0/1573099578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc06a130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0097d0 tx=0x7f4dbc02b940 comp rx=0 tx=0).stop 2023-10-16T12:33:32.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.459+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/1573099578 shutdown_connections 2023-10-16T12:33:32.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.459+0000 7f4dd1ebb640 1 --2- 172.21.15.60:0/1573099578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc06a130 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.459+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/1573099578 >> 172.21.15.60:0/1573099578 conn(0x7f4dcc0ce6e0 msgr2=0x7f4dcc0ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7fc471674640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c06a2e0 0x7fc46c0cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7fc463fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc46c0c9000 0x7fc46c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7fc463fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc46c0c9000 0x7fc46c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:48904/0 (socket says 172.21.15.60:48904) 2023-10-16T12:33:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/1573099578 shutdown_connections 2023-10-16T12:33:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7f16e29f0640 1 Processor -- start 2023-10-16T12:33:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7fc463fff640 1 -- 172.21.15.60:0/86282864 learned_addr learned my addr 172.21.15.60:0/86282864 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7fc470e73640 1 --2- 172.21.15.60:0/86282864 >> v2:172.21.15.60:3300/0 conn(0x7fc46c0addb0 0x7fc46c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.460+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/1573099578 wait complete. 2023-10-16T12:33:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7fc470e73640 1 -- 172.21.15.60:0/86282864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c06a2e0 msgr2=0x7fc46c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7fc470e73640 1 --2- 172.21.15.60:0/86282864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c06a2e0 0x7fc46c0cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7fc470e73640 1 -- 172.21.15.60:0/86282864 >> v2:172.21.15.144:3300/0 conn(0x7fc46c0c9000 msgr2=0x7fc46c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7fc470e73640 1 --2- 172.21.15.60:0/86282864 >> v2:172.21.15.144:3300/0 conn(0x7fc46c0c9000 0x7fc46c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7fc470e73640 1 -- 172.21.15.60:0/86282864 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c06b030 con 0x7fc46c0addb0 2023-10-16T12:33:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.461+0000 7f16e29f0640 1 -- start start 2023-10-16T12:33:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.463+0000 7f16e29f0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0caa80 0x7f16dc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.463+0000 7f16e29f0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16dc0caec0 con 0x7f16dc06a2e0 2023-10-16T12:33:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.463+0000 7f16e29f0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16dc0cb140 con 0x7f16dc0addb0 2023-10-16T12:33:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.463+0000 7f16e29f0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc161a50 con 0x7f16dc0caa80 2023-10-16T12:33:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.463+0000 7fc470e73640 1 --2- 172.21.15.60:0/86282864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c0addb0 0x7fc46c0caa80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc46c0cdbe0 tx=0x7fc45c02c0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39dcb40529eb98be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faa2c090070 con 0x7faa24168f20 2023-10-16T12:33:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7f16dbfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f16dc0addb0 0x7f16dc0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60256/0 (socket says 172.21.15.60:60256) 2023-10-16T12:33:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7f16dbfff640 1 -- 172.21.15.60:0/577517372 learned_addr learned my addr 172.21.15.60:0/577517372 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:32.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7faed2809640 1 Processor -- start 2023-10-16T12:33:32.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7f4dd1ebb640 1 Processor -- start 2023-10-16T12:33:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7fc46bfff640 1 -- 172.21.15.60:0/86282864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c02cc40 con 0x7fc46c0addb0 2023-10-16T12:33:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7fc46bfff640 1 -- 172.21.15.60:0/86282864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c02cda0 con 0x7fc46c0addb0 2023-10-16T12:33:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7fc46bfff640 1 -- 172.21.15.60:0/86282864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c034660 con 0x7fc46c0addb0 2023-10-16T12:33:32.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7faed2809640 1 -- start start 2023-10-16T12:33:32.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.464+0000 7f16d3fff640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0caa80 0x7f16dc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104196140 0 0) 0x7f16dc0caec0 con 0x7f16dc06a2e0 2023-10-16T12:33:32.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c8003520 con 0x7f16dc06a2e0 2023-10-16T12:33:32.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7faa28ff9640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faa1006aea0 0x7faa1006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934270724 0 0) 0x7f16dc0cb140 con 0x7f16dc0addb0 2023-10-16T12:33:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc0caec0 con 0x7f16dc0addb0 2023-10-16T12:33:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 473187551 0 0) 0x7f16c8003520 con 0x7f16dc06a2e0 2023-10-16T12:33:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0caa80 msgr2=0x7f16dc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0caa80 0x7f16dc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 >> v1:172.21.15.144:6789/0 conn(0x7f16dc0addb0 legacy=0x7f16dc0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc06b030 con 0x7f16dc06a2e0 2023-10-16T12:33:32.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.465+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa2c102da0 con 0x7faa24168f20 2023-10-16T12:33:32.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7faa28ff9640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7faa100730c0 0x7faa100754e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa10075b90 con 0x7faa100730c0 2023-10-16T12:33:32.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa2c1053c0 con 0x7faa24168f20 2023-10-16T12:33:32.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 -- 172.21.15.60:0/86282864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c0addb0 msgr2=0x7fc46c0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 --2- 172.21.15.60:0/86282864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c0addb0 0x7fc46c0caa80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc46c0cdbe0 tx=0x7fc45c02c0a0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 -- 172.21.15.60:0/86282864 shutdown_connections 2023-10-16T12:33:32.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 --2- 172.21.15.60:0/86282864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c06a2e0 0x7fc46c0cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 --2- 172.21.15.60:0/86282864 >> v2:172.21.15.144:3300/0 conn(0x7fc46c0c9000 0x7fc46c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 --2- 172.21.15.60:0/86282864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c0addb0 0x7fc46c0caa80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7fc4730d7640 1 -- 172.21.15.60:0/86282864 >> 172.21.15.60:0/86282864 conn(0x7fc46c0ce9b0 msgr2=0x7fc46c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f16d40019f0 con 0x7f16dc06a2e0 2023-10-16T12:33:32.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 >> v1:172.21.15.60:6789/0 conn(0x7f16dc06a2e0 legacy=0x7f16dc1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7f16db7fe640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16c8003b80 0x7f16c8003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.466+0000 7faa2b7fe640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7faa100730c0 0x7faa100754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16db7fe640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c80047c0 0x7f16c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16db7fe640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16c80051d0 0x7f16c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc161a50 con 0x7f16c80047c0 2023-10-16T12:33:32.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c800c0d0 con 0x7f16c80051d0 2023-10-16T12:33:32.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c800c290 con 0x7f16c8003b80 2023-10-16T12:33:32.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faa2bfff640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faa1006aea0 0x7faa1006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16dbfff640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16c8003b80 0x7f16c8003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faed2809640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faecc0b4d50 0x7faecc167950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faed2809640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc06c230 con 0x7faecc15cec0 2023-10-16T12:33:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faed2809640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc06c510 con 0x7faecc0d1b60 2023-10-16T12:33:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faed2809640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0d0700 con 0x7faecc0b4d50 2023-10-16T12:33:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faecbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faecc15cec0 0x7faecc0b1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54196/0 (socket says 172.21.15.60:54196) 2023-10-16T12:33:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7faecbfff640 1 -- 172.21.15.60:0/2942159510 learned_addr learned my addr 172.21.15.60:0/2942159510 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7f16d3fff640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c80047c0 0x7f16c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7faa2b7fe640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7faa100730c0 0x7faa100754e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7fc4730d7640 1 -- 172.21.15.60:0/86282864 shutdown_connections 2023-10-16T12:33:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.467+0000 7fc4730d7640 1 -- 172.21.15.60:0/86282864 wait complete. 2023-10-16T12:33:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7f16e0f8d640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16c80051d0 0x7f16c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7f4dd1ebb640 1 -- start start 2023-10-16T12:33:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7f16d3fff640 1 -- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16c8003b80 msgr2=0x7f16c8003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7f16d3fff640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16c8003b80 0x7f16c8003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7f16d3fff640 1 -- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16c80051d0 msgr2=0x7f16c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7f16d3fff640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16c80051d0 0x7f16c800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7f16d3fff640 1 -- 172.21.15.60:0/577517372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8003520 con 0x7f16c80047c0 2023-10-16T12:33:32.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7faed15a7640 1 --2- 172.21.15.60:0/2942159510 >> v2:172.21.15.144:3300/0 conn(0x7faecc0b4d50 0x7faecc167950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120153355 0 0) 0x7faecc06c230 con 0x7faecc15cec0 2023-10-16T12:33:32.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.468+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb4003520 con 0x7faecc15cec0 2023-10-16T12:33:32.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278628570 0 0) 0x7faeb4003520 con 0x7faecc15cec0 2023-10-16T12:33:32.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 >> v2:172.21.15.144:3300/0 conn(0x7faecc0b4d50 msgr2=0x7faecc167950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 --2- 172.21.15.60:0/2942159510 >> v2:172.21.15.144:3300/0 conn(0x7faecc0b4d50 0x7faecc167950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 >> v1:172.21.15.144:6789/0 conn(0x7faecc0d1b60 legacy=0x7faecc0b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc06c7f0 con 0x7faecc15cec0 2023-10-16T12:33:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faed15a7640 1 --2- 172.21.15.60:0/2942159510 >> v2:172.21.15.144:3300/0 conn(0x7faecc0b4d50 0x7faecc167950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7faebc001670 con 0x7faecc15cec0 2023-10-16T12:33:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 >> v1:172.21.15.60:6789/0 conn(0x7faecc15cec0 legacy=0x7faecc0b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faeb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40047c0 0x7faeb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faeb4005180 0x7faeb400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb400bff0 con 0x7faeb4003970 2023-10-16T12:33:32.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb400c160 con 0x7faeb40047c0 2023-10-16T12:33:32.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.469+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb400c320 con 0x7faeb4005180 2023-10-16T12:33:32.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.470+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== osd.0 v2:172.21.15.60:6823/132166300 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7faa10075b90 con 0x7faa100730c0 2023-10-16T12:33:32.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.470+0000 7faecbfff640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faeb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.470+0000 7faecb7fe640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40047c0 0x7faeb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.470+0000 7faa2bfff640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faa1006aea0 0x7faa1006d360 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa240e1db0 tx=0x7faa18042040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.470+0000 7fc4730d7640 1 Processor -- start 2023-10-16T12:33:32.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.470+0000 7fc4730d7640 1 -- start start 2023-10-16T12:33:32.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f16d3fff640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c80047c0 0x7f16c8004c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16c400b8e0 tx=0x7f16c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140d79c3b134d4e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f4dd1ebb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dcc06a670 0x7f4dcc157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f4dd1ebb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f4dd1ebb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dcc15cf90 0x7f4dcc0c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f4dd1ebb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc158290 con 0x7f4dcc15cf90 2023-10-16T12:33:32.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f4dd1ebb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0afbb0 con 0x7f4dcc06a670 2023-10-16T12:33:32.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f4dd1ebb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0afe90 con 0x7f4dcc0cf200 2023-10-16T12:33:32.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7faecbfff640 1 -- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faeb4005180 msgr2=0x7faeb400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7faecbfff640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faeb4005180 0x7faeb400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7faecbfff640 1 -- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40047c0 msgr2=0x7faeb4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7faecbfff640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40047c0 0x7faeb4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7faecbfff640 1 -- 172.21.15.60:0/2942159510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb4003520 con 0x7faeb4003970 2023-10-16T12:33:32.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7faed15a7640 1 -- 172.21.15.60:0/2942159510 reap_dead start 2023-10-16T12:33:32.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c40343f0 con 0x7f16c80047c0 2023-10-16T12:33:32.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.471+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c4034550 con 0x7f16c80047c0 2023-10-16T12:33:32.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f4dc2ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc4730d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c06a2e0 0x7fc46c15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc4730d7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46c0addb0 0x7fc46c0c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc4730d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c0c9000 0x7fc46c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc4730d7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c0c5a80 con 0x7fc46c06a2e0 2023-10-16T12:33:32.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f4dcb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dcc06a670 0x7f4dcc157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc4730d7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c0c6320 con 0x7fc46c0addb0 2023-10-16T12:33:32.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc4730d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c0c56f0 con 0x7fc46c0c9000 2023-10-16T12:33:32.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f4dcbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dcc15cf90 0x7f4dcc0c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f16db7fe640 1 -- 172.21.15.60:0/577517372 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c40343f0 con 0x7f16c80047c0 2023-10-16T12:33:32.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc470e73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c06a2e0 0x7fc46c15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f16e29f0640 1 -- 172.21.15.60:0/577517372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c80047c0 msgr2=0x7f16c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f16e29f0640 1 --2- 172.21.15.60:0/577517372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c80047c0 0x7f16c8004c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16c400b8e0 tx=0x7f16c80057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f16e0f8d640 1 -- 172.21.15.60:0/577517372 reap_dead start 2023-10-16T12:33:32.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7faecbfff640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faeb4003d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c240 tx=0x7faeb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e141051765c652 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f16e29f0640 1 -- 172.21.15.60:0/577517372 shutdown_connections 2023-10-16T12:33:32.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f16e29f0640 1 -- 172.21.15.60:0/577517372 >> 172.21.15.60:0/577517372 conn(0x7f16dc0ce9b0 msgr2=0x7f16dc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dcc06a670 0x7f4dcc157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:48956/0 (socket says 172.21.15.60:48956) 2023-10-16T12:33:32.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/2556200368 learned_addr learned my addr 172.21.15.60:0/2556200368 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7f4dcbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dcc15cf90 0x7f4dcc0c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57852/0 (socket says 172.21.15.60:57852) 2023-10-16T12:33:32.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f16e29f0640 1 -- 172.21.15.60:0/577517372 shutdown_connections 2023-10-16T12:33:32.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc463fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46c0addb0 0x7fc46c0c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc470e73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c06a2e0 0x7fc46c15d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57864/0 (socket says 172.21.15.60:57864) 2023-10-16T12:33:32.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.472+0000 7fc470e73640 1 -- 172.21.15.60:0/3643371400 learned_addr learned my addr 172.21.15.60:0/3643371400 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7fc471674640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c0c9000 0x7fc46c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 msgr2=0x7f4dcc0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc0c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dcc15cf90 msgr2=0x7f4dcc0c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dcc15cf90 0x7f4dcc0c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f4dcb7fe640 1 -- 172.21.15.60:0/2556200368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009400 con 0x7f4dcc06a670 2023-10-16T12:33:32.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.474+0000 7f4dc2ffd640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dcc0cf200 0x7f4dcc0c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:32.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7fc470e73640 1 -- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c0c9000 msgr2=0x7fc46c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7fc470e73640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46c0c9000 0x7fc46c0c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.474+0000 7ff556015640 1 Processor -- start 2023-10-16T12:33:32.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.474+0000 7f4dcb7fe640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dcc06a670 0x7f4dcc157d50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0098f0 tx=0x7f4dbc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7f16e29f0640 1 -- 172.21.15.60:0/577517372 wait complete. 2023-10-16T12:33:32.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7fc470e73640 1 -- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46c0addb0 msgr2=0x7fc46c0c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7fc470e73640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46c0addb0 0x7fc46c0c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.473+0000 7fc470e73640 1 -- 172.21.15.60:0/3643371400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c009a70 con 0x7fc46c06a2e0 2023-10-16T12:33:32.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc005ad0 con 0x7faeb4003970 2023-10-16T12:33:32.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc005c30 con 0x7faeb4003970 2023-10-16T12:33:32.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7faecaffd640 1 -- 172.21.15.60:0/2942159510 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc02ca90 con 0x7faeb4003970 2023-10-16T12:33:32.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7ff556015640 1 -- start start 2023-10-16T12:33:32.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.474+0000 7fc470e73640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c06a2e0 0x7fc46c15d7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00b140 tx=0x7fc45c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc03e040 con 0x7f4dcc06a670 2023-10-16T12:33:32.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.474+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c004880 con 0x7fc46c06a2e0 2023-10-16T12:33:32.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c003e10 con 0x7fc46c06a2e0 2023-10-16T12:33:32.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c0c5cd0 con 0x7fc46c06a2e0 2023-10-16T12:33:32.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c15e4a0 con 0x7fc46c06a2e0 2023-10-16T12:33:32.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faed2809640 1 -- 172.21.15.60:0/2942159510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 msgr2=0x7faeb4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faed2809640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faeb4003d90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c240 tx=0x7faeb40057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc039c10 con 0x7f4dcc06a670 2023-10-16T12:33:32.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0443f0 con 0x7f4dcc06a670 2023-10-16T12:33:32.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faed2809640 1 -- 172.21.15.60:0/2942159510 shutdown_connections 2023-10-16T12:33:32.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faed2809640 1 --2- 172.21.15.60:0/2942159510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faeb4003d90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faed2809640 1 -- 172.21.15.60:0/2942159510 >> 172.21.15.60:0/2942159510 conn(0x7faecc0d2bc0 msgr2=0x7faecc0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/2556200368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dcc0b0170 con 0x7f4dcc06a670 2023-10-16T12:33:32.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7f4dd1ebb640 1 -- 172.21.15.60:0/2556200368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc0c2680 con 0x7f4dcc06a670 2023-10-16T12:33:32.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.477+0000 7f16e29f0640 1 Processor -- start 2023-10-16T12:33:32.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faf9bb6c640 1 Processor -- start 2023-10-16T12:33:32.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.476+0000 7faf9bb6c640 1 -- start start 2023-10-16T12:33:32.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.477+0000 7faed2809640 1 -- 172.21.15.60:0/2942159510 shutdown_connections 2023-10-16T12:33:32.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.477+0000 7faed2809640 1 -- 172.21.15.60:0/2942159510 wait complete. 2023-10-16T12:33:32.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.475+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c034680 con 0x7fc46c06a2e0 2023-10-16T12:33:32.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.477+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc46c06b5f0 con 0x7fc46c06a2e0 2023-10-16T12:33:32.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.477+0000 7f16e29f0640 1 -- start start 2023-10-16T12:33:32.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.477+0000 7ff556015640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff5500cf5f0 0x7ff55006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.478+0000 7ff556015640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 0x7ff5500cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.478+0000 7ff556015640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff550161740 con 0x7ff550070830 2023-10-16T12:33:32.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.478+0000 7ff556015640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5500cdbd0 con 0x7ff5500cf5f0 2023-10-16T12:33:32.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.478+0000 7ff556015640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55006b0f0 con 0x7ff550070c10 2023-10-16T12:33:32.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.478+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4da4000fc0 con 0x7f4dcc06a670 2023-10-16T12:33:32.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7ff54ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 0x7ff5500cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7ff54ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 0x7ff5500cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53204/0 (socket says 172.21.15.60:53204) 2023-10-16T12:33:32.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7ff54ffff640 1 -- 172.21.15.60:0/774109541 learned_addr learned my addr 172.21.15.60:0/774109541 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7faf9bb6c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf940cf890 0x7faf940c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7f16e29f0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0c4ed0 0x7f16dc070640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7ff547fff640 1 --2- 172.21.15.60:0/774109541 >> v2:172.21.15.144:3300/0 conn(0x7ff5500cf5f0 0x7ff55006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7f16e29f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16dc0b4530 0x7f16dc0af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7f16e29f0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16dc0af810 0x7f16dc15d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7f16dbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0c4ed0 0x7f16dc070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7faf9bb6c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf940cfc70 0x7faf940ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7faf9bb6c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940b3e80 con 0x7faf940c5cf0 2023-10-16T12:33:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc45c03b070 con 0x7fc46c06a2e0 2023-10-16T12:33:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7faf9bb6c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf940b4630 con 0x7faf940cfc70 2023-10-16T12:33:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.479+0000 7faf9bb6c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf940cf000 con 0x7faf940cf890 2023-10-16T12:33:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7faf99107640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faf940c5cf0 0x7faf940b5b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54210/0 (socket says 172.21.15.60:54210) 2023-10-16T12:33:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7faf99107640 1 -- 172.21.15.60:0/1146184630 learned_addr learned my addr 172.21.15.60:0/1146184630 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7f16dbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0c4ed0 0x7f16dc070640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53206/0 (socket says 172.21.15.60:53206) 2023-10-16T12:33:32.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7f16dbfff640 1 -- 172.21.15.60:0/2979065860 learned_addr learned my addr 172.21.15.60:0/2979065860 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f3eec644640 1 Processor -- start 2023-10-16T12:33:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f16e0f8d640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16dc0af810 0x7f16dc15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f16d3fff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16dc0b4530 0x7f16dc0af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7ff54ffff640 1 -- 172.21.15.60:0/774109541 >> v2:172.21.15.144:3300/0 conn(0x7ff5500cf5f0 msgr2=0x7ff55006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7ff54ffff640 1 --2- 172.21.15.60:0/774109541 >> v2:172.21.15.144:3300/0 conn(0x7ff5500cf5f0 0x7ff55006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7ff54ffff640 1 -- 172.21.15.60:0/774109541 >> v1:172.21.15.144:6789/0 conn(0x7ff550070830 legacy=0x7ff55006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7ff54ffff640 1 -- 172.21.15.60:0/774109541 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5501619b0 con 0x7ff550070c10 2023-10-16T12:33:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc070b80 con 0x7f16dc0b4530 2023-10-16T12:33:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7faed2809640 1 Processor -- start 2023-10-16T12:33:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.480+0000 7faed2809640 1 -- start start 2023-10-16T12:33:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7faf99908640 1 --2- 172.21.15.60:0/1146184630 >> v2:172.21.15.144:3300/0 conn(0x7faf940cf890 0x7faf940c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0afca0 con 0x7f16dc0af810 2023-10-16T12:33:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0b16d0 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faed2809640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faecc0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faed2809640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faecc0b57e0 0x7faecc15f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faed2809640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faecc0b5c40 0x7faecc15f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faf9a109640 1 --2- 172.21.15.60:0/1146184630 >> v2:172.21.15.60:3300/0 conn(0x7faf940cfc70 0x7faf940ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faf98906640 1 -- 172.21.15.60:0/1146184630 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643694653 0 0) 0x7faf940b3e80 con 0x7faf940c5cf0 2023-10-16T12:33:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7fc469ffb640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44c06af40 0x7fc44c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7ff54ffff640 1 --2- 172.21.15.60:0/774109541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 0x7ff5500cd690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff548009870 tx=0x7ff54802b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216ddc9e61a74542 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7f16dbfff640 1 -- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16dc0af810 msgr2=0x7f16dc15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7f16dbfff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16dc0af810 0x7f16dc15d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7f16dbfff640 1 -- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16dc0b4530 msgr2=0x7f16dc0af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc45c0ac1b0 con 0x7fc46c06a2e0 2023-10-16T12:33:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faecbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faecc0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faed2809640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc15e500 con 0x7faeb4003970 2023-10-16T12:33:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faecbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faecc0cb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57882/0 (socket says 172.21.15.60:57882) 2023-10-16T12:33:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faecbfff640 1 -- 172.21.15.60:0/3034493040 learned_addr learned my addr 172.21.15.60:0/3034493040 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7faf98906640 1 -- 172.21.15.60:0/1146184630 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf80003520 con 0x7faf940c5cf0 2023-10-16T12:33:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7f16dbfff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16dc0b4530 0x7f16dc0af2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f16dbfff640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc164170 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc15e7e0 con 0x7faecc0b57e0 2023-10-16T12:33:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc15eac0 con 0x7faecc0b5c40 2023-10-16T12:33:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faecb7fe640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faecc0b57e0 0x7faecc15f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7fc469ffb640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc44c073160 0x7fc44c075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faed15a7640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faecc0b5c40 0x7faecc15f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 --> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c075c30 con 0x7fc44c073160 2023-10-16T12:33:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7fc463fff640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44c06af40 0x7fc44c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc45c0ac560 con 0x7fc46c06a2e0 2023-10-16T12:33:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f16d3fff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16dc0b4530 0x7f16dc0af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faf99908640 1 -- 172.21.15.60:0/1146184630 >> v2:172.21.15.60:3300/0 conn(0x7faf940cfc70 msgr2=0x7faf940ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faf99908640 1 --2- 172.21.15.60:0/1146184630 >> v2:172.21.15.60:3300/0 conn(0x7faf940cfc70 0x7faf940ceac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faf99908640 1 -- 172.21.15.60:0/1146184630 >> v1:172.21.15.60:6789/0 conn(0x7faf940c5cf0 legacy=0x7faf940b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7faf99908640 1 -- 172.21.15.60:0/1146184630 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940ce630 con 0x7faf940cf890 2023-10-16T12:33:32.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.481+0000 7f3eec644640 1 -- start start 2023-10-16T12:33:32.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f3eec644640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ee40caa80 0x7f3ee4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f16dbfff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0c4ed0 0x7f16dc070640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f16cc001630 tx=0x7f16cc001af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f3eec644640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ee406a2e0 0x7f3ee41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f3eec644640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40caef0 con 0x7f3ee40addb0 2023-10-16T12:33:32.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f3eec644640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee4161a50 con 0x7f3ee40caa80 2023-10-16T12:33:32.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f3eec644640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee406a6f0 con 0x7f3ee406a2e0 2023-10-16T12:33:32.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7faed15a7640 1 -- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faecc0b57e0 msgr2=0x7faecc15f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7faed15a7640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faecc0b57e0 0x7faecc15f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7faed15a7640 1 -- 172.21.15.60:0/3034493040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 msgr2=0x7faecc0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7faed15a7640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faeb4003970 0x7faecc0cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7ff54effd640 1 -- 172.21.15.60:0/774109541 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5480317e0 con 0x7ff550070c10 2023-10-16T12:33:32.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7ff54effd640 1 -- 172.21.15.60:0/774109541 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548031940 con 0x7ff550070c10 2023-10-16T12:33:32.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7ff54effd640 1 -- 172.21.15.60:0/774109541 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548031d10 con 0x7ff550070c10 2023-10-16T12:33:32.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f3eea3e0640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3ee40addb0 0x7f3ee40c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60268/0 (socket says 172.21.15.60:60268) 2023-10-16T12:33:32.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f3eea3e0640 1 -- 172.21.15.60:0/2055750305 learned_addr learned my addr 172.21.15.60:0/2055750305 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:32.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.482+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4dbc043390 con 0x7f4dcc06a670 2023-10-16T12:33:32.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f4dc8ff9640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4dac06ae70 0x7f4dac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.483+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dbc0abe30 con 0x7f4dcc06a670 2023-10-16T12:33:32.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f4dc8ff9640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7f4dac073010 0x7f4dac075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 --> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4dac075ae0 con 0x7f4dac073010 2023-10-16T12:33:32.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4dbc0ac280 con 0x7f4dcc06a670 2023-10-16T12:33:32.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7faed15a7640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc15eda0 con 0x7faecc0b5c40 2023-10-16T12:33:32.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f3ee9bdf640 1 --2- 172.21.15.60:0/2055750305 >> v2:172.21.15.60:3300/0 conn(0x7f3ee40caa80 0x7f3ee4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7fc471674640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc44c073160 0x7fc44c075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f4dcbfff640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7f4dac073010 0x7f4dac075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc001ce0 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc00f9c0 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc00fd50 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc164450 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.484+0000 7faf9a109640 1 --2- 172.21.15.60:0/1146184630 >> v2:172.21.15.60:3300/0 conn(0x7faf940cfc70 0x7faf940ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:32.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f3eeabe1640 1 --2- 172.21.15.60:0/2055750305 >> v2:172.21.15.144:3300/0 conn(0x7f3ee406a2e0 0x7f3ee41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f3ee93de640 1 -- 172.21.15.60:0/2055750305 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939301579 0 0) 0x7f3ee40caef0 con 0x7f3ee40addb0 2023-10-16T12:33:32.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7faf99908640 1 --2- 172.21.15.60:0/1146184630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cf890 0x7faf940c6220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf940d2c20 tx=0x7faf8c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c9720d1997e23e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7ff556015640 1 -- 172.21.15.60:0/774109541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 msgr2=0x7ff5500cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7ff556015640 1 --2- 172.21.15.60:0/774109541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 0x7ff5500cd690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff548009870 tx=0x7ff54802b880 comp rx=0 tx=0).stop 2023-10-16T12:33:32.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7ff556015640 1 -- 172.21.15.60:0/774109541 shutdown_connections 2023-10-16T12:33:32.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7ff556015640 1 --2- 172.21.15.60:0/774109541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff550070c10 0x7ff5500cd690 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7ff556015640 1 --2- 172.21.15.60:0/774109541 >> v2:172.21.15.144:3300/0 conn(0x7ff5500cf5f0 0x7ff55006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7ff556015640 1 -- 172.21.15.60:0/774109541 >> 172.21.15.60:0/774109541 conn(0x7ff5500cfac0 msgr2=0x7ff5500cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f3ee93de640 1 -- 172.21.15.60:0/2055750305 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed0003520 con 0x7f3ee40addb0 2023-10-16T12:33:32.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2055750305 >> v2:172.21.15.144:3300/0 conn(0x7f3ee406a2e0 msgr2=0x7f3ee41650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:32.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faf98906640 1 -- 172.21.15.60:0/1146184630 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c02cc60 con 0x7faf940cf890 2023-10-16T12:33:32.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7f3ee9bdf640 1 --2- 172.21.15.60:0/2055750305 >> v2:172.21.15.144:3300/0 conn(0x7f3ee406a2e0 0x7f3ee41650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2055750305 >> v1:172.21.15.144:6789/0 conn(0x7f3ee40addb0 legacy=0x7f3ee40c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7f3ee93de640 1 -- 172.21.15.60:0/2055750305 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602204138 0 0) 0x7f3ed0003520 con 0x7f3ee40addb0 2023-10-16T12:33:32.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7fc463fff640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44c06af40 0x7fc44c06d400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc46c1611b0 tx=0x7fc454008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7faed15a7640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faecc0b5c40 0x7faecc15f800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faecc074530 tx=0x7faec400c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400b5e0 con 0x7faecc0b5c40 2023-10-16T12:33:32.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec400b740 con 0x7faecc0b5c40 2023-10-16T12:33:32.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400bad0 con 0x7faecc0b5c40 2023-10-16T12:33:32.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc0cbac0 con 0x7faecc0b5c40 2023-10-16T12:33:32.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc150e00 con 0x7faecc0b5c40 2023-10-16T12:33:32.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc164a10 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2055750305 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40cb140 con 0x7f3ee40caa80 2023-10-16T12:33:32.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faf98906640 1 -- 172.21.15.60:0/1146184630 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c02cdc0 con 0x7faf940cf890 2023-10-16T12:33:32.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7faf98906640 1 -- 172.21.15.60:0/1146184630 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c034640 con 0x7faf940cf890 2023-10-16T12:33:32.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7ff556015640 1 -- 172.21.15.60:0/774109541 shutdown_connections 2023-10-16T12:33:32.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7ff556015640 1 -- 172.21.15.60:0/774109541 wait complete. 2023-10-16T12:33:32.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16dc06b310 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7faf9bb6c640 1 -- 172.21.15.60:0/1146184630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cf890 msgr2=0x7faf940c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7faf9bb6c640 1 --2- 172.21.15.60:0/1146184630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cf890 0x7faf940c6220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf940d2c20 tx=0x7faf8c02beb0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.485+0000 7f4dc2ffd640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4dac06ae70 0x7f4dac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.486+0000 7f4dcbfff640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7f4dac073010 0x7f4dac075430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7faf9bb6c640 1 -- 172.21.15.60:0/1146184630 shutdown_connections 2023-10-16T12:33:32.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7faf9bb6c640 1 --2- 172.21.15.60:0/1146184630 >> v2:172.21.15.60:3300/0 conn(0x7faf940cfc70 0x7faf940ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7faf9bb6c640 1 --2- 172.21.15.60:0/1146184630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cf890 0x7faf940c6220 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.487+0000 7faf9bb6c640 1 -- 172.21.15.60:0/1146184630 >> 172.21.15.60:0/1146184630 conn(0x7faf940d2cc0 msgr2=0x7faf940d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7f3ee9bdf640 1 --2- 172.21.15.60:0/2055750305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40caa80 0x7f3ee4069da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40cdd20 tx=0x7f3ed402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55d494c297bdb31 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7faf9bb6c640 1 -- 172.21.15.60:0/1146184630 shutdown_connections 2023-10-16T12:33:32.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7f4dc2ffd640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4dac06ae70 0x7f4dac06d330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc14bdd0 tx=0x7f4db4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== osd.2 v2:172.21.15.60:6800/1201910913 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f4dac075ae0 con 0x7f4dac073010 2023-10-16T12:33:32.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7f3ee93de640 1 -- 172.21.15.60:0/2055750305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0001670 con 0x7f3ee40caa80 2023-10-16T12:33:32.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faecc15e670 con 0x7faecc0b5c40 2023-10-16T12:33:32.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.489+0000 7f3ee93de640 1 -- 172.21.15.60:0/2055750305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed4033440 con 0x7f3ee40caa80 2023-10-16T12:33:32.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.489+0000 7f3ee93de640 1 -- 172.21.15.60:0/2055750305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40337b0 con 0x7f3ee40caa80 2023-10-16T12:33:32.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7f3eec644640 1 -- 172.21.15.60:0/2055750305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40caa80 msgr2=0x7f3ee4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7f3eec644640 1 --2- 172.21.15.60:0/2055750305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40caa80 0x7f3ee4069da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40cdd20 tx=0x7f3ed402b7a0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7f3eec644640 1 -- 172.21.15.60:0/2055750305 shutdown_connections 2023-10-16T12:33:32.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7f3eec644640 1 --2- 172.21.15.60:0/2055750305 >> v2:172.21.15.144:3300/0 conn(0x7f3ee406a2e0 0x7f3ee41650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7f3eec644640 1 --2- 172.21.15.60:0/2055750305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40caa80 0x7f3ee4069da0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7f3eec644640 1 -- 172.21.15.60:0/2055750305 >> 172.21.15.60:0/2055750305 conn(0x7f3ee40ce9b0 msgr2=0x7f3ee40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.490+0000 7ff556015640 1 Processor -- start 2023-10-16T12:33:32.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7ff556015640 1 -- start start 2023-10-16T12:33:32.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.488+0000 7faf9bb6c640 1 -- 172.21.15.60:0/1146184630 wait complete. 2023-10-16T12:33:32.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7faf9bb6c640 1 Processor -- start 2023-10-16T12:33:32.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7f3eec644640 1 -- 172.21.15.60:0/2055750305 shutdown_connections 2023-10-16T12:33:32.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7f3eec644640 1 -- 172.21.15.60:0/2055750305 wait complete. 2023-10-16T12:33:32.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7faf9bb6c640 1 -- start start 2023-10-16T12:33:32.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faec400bc30 con 0x7faecc0b5c40 2023-10-16T12:33:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.492+0000 7fc471674640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc44c073160 0x7fc44c075580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.491+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f16cc01d460 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.492+0000 7f16d97fa640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16bc06af40 0x7f16bc06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.492+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16cc089f90 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7f16d97fa640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f16bc073160 0x7f16bc075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16bc075c30 con 0x7f16bc073160 2023-10-16T12:33:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16cc08a340 con 0x7f16dc0c4ed0 2023-10-16T12:33:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7faec8ff9640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faeb806af70 0x7faeb806d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faec408a590 con 0x7faecc0b5c40 2023-10-16T12:33:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.492+0000 7ff556015640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff550070830 0x7ff550157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.492+0000 7ff556015640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff550070c10 0x7ff550155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7faec8ff9640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7faeb8073190 0x7faeb80755b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7faf9bb6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf940c5cf0 0x7faf940c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== osd.1 v2:172.21.15.60:6813/4168261604 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fc44c075c30 con 0x7fc44c073160 2023-10-16T12:33:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faeb8075c60 con 0x7faeb8073190 2023-10-16T12:33:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7faec408a940 con 0x7faecc0b5c40 2023-10-16T12:33:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.493+0000 7ff547fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff550070c10 0x7ff550155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7faecbfff640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faeb806af70 0x7faeb806d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7faf9bb6c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940cf890 0x7faf94160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7f3eec644640 1 Processor -- start 2023-10-16T12:33:32.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7f16d3fff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16bc06af40 0x7f16bc06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7faf9bb6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cfc70 0x7faf940b78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff550070830 0x7ff550157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff550070830 0x7ff550157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49016/0 (socket says 172.21.15.60:49016) 2023-10-16T12:33:32.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7ff54f7fe640 1 -- 172.21.15.60:0/3501136311 learned_addr learned my addr 172.21.15.60:0/3501136311 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7ff547fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff550070c10 0x7ff550155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57894/0 (socket says 172.21.15.60:57894) 2023-10-16T12:33:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7ff556015640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5500cf5f0 0x7ff5501fa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7faf99908640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf940c5cf0 0x7faf940c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55015bea0 con 0x7ff550070c10 2023-10-16T12:33:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7f16e0f8d640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f16bc073160 0x7f16bc075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55015b710 con 0x7ff550070830 2023-10-16T12:33:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5500c8b00 con 0x7ff5500cf5f0 2023-10-16T12:33:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7f3eec644640 1 -- start start 2023-10-16T12:33:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7faf9bb6c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940cc9b0 con 0x7faf940c5cf0 2023-10-16T12:33:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7faf9bb6c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940b8990 con 0x7faf940cfc70 2023-10-16T12:33:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7faf9bb6c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf940cc2f0 con 0x7faf940cf890 2023-10-16T12:33:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7faf99908640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf940c5cf0 0x7faf940c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57902/0 (socket says 172.21.15.60:57902) 2023-10-16T12:33:32.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7ff54ffff640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5500cf5f0 0x7ff5501fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7faf99908640 1 -- 172.21.15.60:0/2437252870 learned_addr learned my addr 172.21.15.60:0/2437252870 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7faf99107640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940cf890 0x7faf94160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.495+0000 7faf9a109640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cfc70 0x7faf940b78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7f16d3fff640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16bc06af40 0x7f16bc06d400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16dc165030 tx=0x7f16c4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7ff547fff640 1 -- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5500cf5f0 msgr2=0x7ff5501fa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7ff547fff640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5500cf5f0 0x7ff5501fa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7ff547fff640 1 -- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff550070830 msgr2=0x7ff550157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7ff547fff640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff550070830 0x7ff550157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7ff547fff640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480094d0 con 0x7ff550070c10 2023-10-16T12:33:32.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faf9a109640 1 -- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940cf890 msgr2=0x7faf94160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faf9a109640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf940cf890 0x7faf94160660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faf9a109640 1 -- 172.21.15.60:0/2437252870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf940c5cf0 msgr2=0x7faf940c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faf9a109640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf940c5cf0 0x7faf940c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faf9a109640 1 -- 172.21.15.60:0/2437252870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c009a70 con 0x7faf940cfc70 2023-10-16T12:33:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7ff547fff640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff550070c10 0x7ff550155dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff540007d80 tx=0x7ff54000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7f16e0f8d640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f16bc073160 0x7f16bc075580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540014420 con 0x7ff550070c10 2023-10-16T12:33:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7faf9a109640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cfc70 0x7faf940b78c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faf8800e560 tx=0x7faf8800ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55015b920 con 0x7ff550070c10 2023-10-16T12:33:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5500c8790 con 0x7ff550070c10 2023-10-16T12:33:32.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff540014580 con 0x7ff550070c10 2023-10-16T12:33:32.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540014910 con 0x7ff550070c10 2023-10-16T12:33:32.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.494+0000 7faecb7fe640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7faeb8073190 0x7faeb80755b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faecbfff640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faeb806af70 0x7faeb806d430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faebc000f60 tx=0x7faebc009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.496+0000 7faecb7fe640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7faeb8073190 0x7faeb80755b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.497+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88004230 con 0x7faf940cfc70 2023-10-16T12:33:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88004390 con 0x7faf940cfc70 2023-10-16T12:33:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88003810 con 0x7faf940cfc70 2023-10-16T12:33:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7f3eec644640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee406a2e0 0x7f3ee4154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7f3eec644640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40addb0 0x7f3ee40c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7f3eec644640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee40caa80 0x7f3ee40c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7f3eec644640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b25e0 con 0x7f3ee40addb0 2023-10-16T12:33:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7f3eec644640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee4161570 con 0x7f3ee40caa80 2023-10-16T12:33:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7f3eec644640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b15d0 con 0x7f3ee406a2e0 2023-10-16T12:33:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 --> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faa04002cb0 con 0x7faa100730c0 2023-10-16T12:33:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== osd.4 v2:172.21.15.144:6800/2038056415 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7faeb8075c60 con 0x7faeb8073190 2023-10-16T12:33:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.498+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf940cc5d0 con 0x7faf940cfc70 2023-10-16T12:33:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf94160ba0 con 0x7faf940cfc70 2023-10-16T12:33:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5500cf060 con 0x7ff550070c10 2023-10-16T12:33:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7faa28ff9640 1 -- 172.21.15.60:0/2719090253 <== osd.0 v2:172.21.15.60:6823/132166300 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faa04002cb0 con 0x7faa100730c0 2023-10-16T12:33:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf940d3980 con 0x7faf940cfc70 2023-10-16T12:33:32.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== osd.3 v2:172.21.15.60:6808/302373198 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f16bc075c30 con 0x7f16bc073160 2023-10-16T12:33:32.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7faa100730c0 msgr2=0x7faa100754e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7faa167fc640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7faa100730c0 0x7faa100754e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7f3eea3e0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee406a2e0 0x7f3ee4154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7f3ee9bdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40addb0 0x7f3ee40c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7f3ee9bdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40addb0 0x7f3ee40c4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57916/0 (socket says 172.21.15.60:57916) 2023-10-16T12:33:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.499+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2307908461 learned_addr learned my addr 172.21.15.60:0/2307908461 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f3eeabe1640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee40caa80 0x7f3ee40c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee406a2e0 msgr2=0x7f3ee4154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f3ee9bdf640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee406a2e0 0x7f3ee4154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee40caa80 msgr2=0x7f3ee40c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f3ee9bdf640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee40caa80 0x7f3ee40c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.500+0000 7f3ee9bdf640 1 -- 172.21.15.60:0/2307908461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4009400 con 0x7f3ee40addb0 2023-10-16T12:33:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7f3ee9bdf640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40addb0 0x7f3ee40c4690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3ed402b7e0 tx=0x7f3ed40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faa1006aea0 msgr2=0x7faa1006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7faa167fc640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faa1006aea0 0x7faa1006d360 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa240e1db0 tx=0x7faa18042040 comp rx=0 tx=0).stop 2023-10-16T12:33:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa24168f20 msgr2=0x7faa241693a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7faa167fc640 1 --2- 172.21.15.60:0/2719090253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa24168f20 0x7faa241693a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faa2c069740 tx=0x7faa2c068000 comp rx=0 tx=0).stop 2023-10-16T12:33:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7faa30c9e640 1 -- 172.21.15.60:0/2719090253 reap_dead start 2023-10-16T12:33:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 shutdown_connections 2023-10-16T12:33:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 >> 172.21.15.60:0/2719090253 conn(0x7faa240590c0 msgr2=0x7faa24004d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.501+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40337b0 con 0x7f3ee40addb0 2023-10-16T12:33:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed4033930 con 0x7f3ee40addb0 2023-10-16T12:33:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee414b9d0 con 0x7f3ee40addb0 2023-10-16T12:33:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 shutdown_connections 2023-10-16T12:33:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7faa167fc640 1 -- 172.21.15.60:0/2719090253 wait complete. 2023-10-16T12:33:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee40b1b00 con 0x7f3ee40addb0 2023-10-16T12:33:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff540014a70 con 0x7ff550070c10 2023-10-16T12:33:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.502+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4033ce0 con 0x7f3ee40addb0 2023-10-16T12:33:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.503+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ecc000fc0 con 0x7f3ee40addb0 2023-10-16T12:33:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.504+0000 7ff54cff9640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff53006aea0 0x7ff53006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff54008a4b0 con 0x7ff550070c10 2023-10-16T12:33:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.503+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faf88003970 con 0x7faf940cfc70 2023-10-16T12:33:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7faf867fc640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faf7806af40 0x7faf7806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf88089450 con 0x7faf940cfc70 2023-10-16T12:33:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7faf867fc640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7faf78073160 0x7faf78075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 --> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf78075c30 con 0x7faf78073160 2023-10-16T12:33:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf88089800 con 0x7faf940cfc70 2023-10-16T12:33:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.506+0000 7faf99107640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7faf78073160 0x7faf78075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.506+0000 7faf99908640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faf7806af40 0x7faf7806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7ff54cff9640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7ff5300730c0 0x7ff5300754e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.506+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3ed40304b0 con 0x7f3ee40addb0 2023-10-16T12:33:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.507+0000 7faf99107640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7faf78073160 0x7faf78075580 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.507+0000 7f3edaffd640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806af40 0x7f3ec806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.508+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ed40abba0 con 0x7f3ee40addb0 2023-10-16T12:33:32.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.508+0000 7f3edaffd640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f3ec8073160 0x7f3ec8075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:32.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.508+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 --> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ec8075c30 con 0x7f3ec8073160 2023-10-16T12:33:32.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.508+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed40abf50 con 0x7f3ee40addb0 2023-10-16T12:33:32.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.505+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff530075b90 con 0x7ff5300730c0 2023-10-16T12:33:32.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.508+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54008a860 con 0x7ff550070c10 2023-10-16T12:33:32.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.506+0000 7ff54ffff640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7ff5300730c0 0x7ff5300754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.506+0000 7ff54f7fe640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff53006aea0 0x7ff53006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.508+0000 7f3eea3e0640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806af40 0x7f3ec806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.509+0000 7f3eeabe1640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f3ec8073160 0x7f3ec8075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:32.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.509+0000 7faf99908640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faf7806af40 0x7faf7806d400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf8c02bef0 tx=0x7faf8c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.510+0000 7ff54f7fe640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff53006aea0 0x7ff53006d360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff538001760 tx=0x7ff538009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.510+0000 7f3eeabe1640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f3ec8073160 0x7f3ec8075580 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.510+0000 7f3eea3e0640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806af40 0x7f3ec806d400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00017a0 tx=0x7f3ee0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.510+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== osd.5 v2:172.21.15.144:6814/1961528470 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7faf78075c30 con 0x7faf78073160 2023-10-16T12:33:32.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.511+0000 7ff54ffff640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7ff5300730c0 0x7ff5300754e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.513+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== osd.6 v2:172.21.15.144:6813/3196639449 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff530075b90 con 0x7ff5300730c0 2023-10-16T12:33:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.514+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== osd.7 v2:172.21.15.144:6808/3315484258 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f3ec8075c30 con 0x7f3ec8073160 2023-10-16T12:33:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.522+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 --> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faecc06cad0 con 0x7faeb8073190 2023-10-16T12:33:32.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.522+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 --> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4da4002cb0 con 0x7f4dac073010 2023-10-16T12:33:32.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.523+0000 7f4dc8ff9640 1 -- 172.21.15.60:0/2556200368 <== osd.2 v2:172.21.15.60:6800/1201910913 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4da4002cb0 con 0x7f4dac073010 2023-10-16T12:33:32.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.523+0000 7faec8ff9640 1 -- 172.21.15.60:0/3034493040 <== osd.4 v2:172.21.15.144:6800/2038056415 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faec00013f0 con 0x7faeb8073190 2023-10-16T12:33:32.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7f4dac073010 msgr2=0x7f4dac075430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7f4dac073010 0x7f4dac075430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4dac06ae70 msgr2=0x7f4dac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4dac06ae70 0x7f4dac06d330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc14bdd0 tx=0x7f4db4009040 comp rx=0 tx=0).stop 2023-10-16T12:33:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dcc06a670 msgr2=0x7f4dcc157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 --2- 172.21.15.60:0/2556200368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dcc06a670 0x7f4dcc157d50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0098f0 tx=0x7f4dbc004730 comp rx=0 tx=0).stop 2023-10-16T12:33:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dcbfff640 1 -- 172.21.15.60:0/2556200368 reap_dead start 2023-10-16T12:33:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 shutdown_connections 2023-10-16T12:33:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.524+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 >> 172.21.15.60:0/2556200368 conn(0x7f4dcc0ce6e0 msgr2=0x7f4dcc1593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.525+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7faeb8073190 msgr2=0x7faeb80755b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.525+0000 7faed2809640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7faeb8073190 0x7faeb80755b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.525+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faeb806af70 msgr2=0x7faeb806d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.525+0000 7faed2809640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faeb806af70 0x7faeb806d430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faebc000f60 tx=0x7faebc009030 comp rx=0 tx=0).stop 2023-10-16T12:33:32.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.527+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 shutdown_connections 2023-10-16T12:33:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.525+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faecc0b5c40 msgr2=0x7faecc15f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.525+0000 7faed2809640 1 --2- 172.21.15.60:0/3034493040 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faecc0b5c40 0x7faecc15f800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faecc074530 tx=0x7faec400c7b0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.527+0000 7faed15a7640 1 -- 172.21.15.60:0/3034493040 reap_dead start 2023-10-16T12:33:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.527+0000 7f4dc1ffb640 1 -- 172.21.15.60:0/2556200368 wait complete. 2023-10-16T12:33:32.639 INFO:teuthology.orchestra.run.smithi060.stdout:60129542150 2023-10-16T12:33:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.528+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 --> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc46c06b310 con 0x7fc44c073160 2023-10-16T12:33:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.528+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 shutdown_connections 2023-10-16T12:33:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.529+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 >> 172.21.15.60:0/3034493040 conn(0x7faecc0d2bc0 msgr2=0x7faecc15bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T12:33:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.529+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 shutdown_connections 2023-10-16T12:33:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.529+0000 7faed2809640 1 -- 172.21.15.60:0/3034493040 wait complete. 2023-10-16T12:33:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.529+0000 7fc469ffb640 1 -- 172.21.15.60:0/3643371400 <== osd.1 v2:172.21.15.60:6813/4168261604 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc44c075c30 con 0x7fc44c073160 2023-10-16T12:33:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc44c073160 msgr2=0x7fc44c075580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc4730d7640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc44c073160 0x7fc44c075580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44c06af40 msgr2=0x7fc44c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc4730d7640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44c06af40 0x7fc44c06d400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc46c1611b0 tx=0x7fc454008040 comp rx=0 tx=0).stop 2023-10-16T12:33:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c06a2e0 msgr2=0x7fc46c15d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc4730d7640 1 --2- 172.21.15.60:0/3643371400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46c06a2e0 0x7fc46c15d7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00b140 tx=0x7fc45c004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.531+0000 7fc471674640 1 -- 172.21.15.60:0/3643371400 reap_dead start 2023-10-16T12:33:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.532+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 shutdown_connections 2023-10-16T12:33:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.532+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 >> 172.21.15.60:0/3643371400 conn(0x7fc46c0ce9b0 msgr2=0x7fc46c072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.532+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 shutdown_connections 2023-10-16T12:33:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.532+0000 7fc4730d7640 1 -- 172.21.15.60:0/3643371400 wait complete. 2023-10-16T12:33:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.533+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 --> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f16dc06a750 con 0x7f16bc073160 2023-10-16T12:33:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.534+0000 7f16d97fa640 1 -- 172.21.15.60:0/2979065860 <== osd.3 v2:172.21.15.60:6808/302373198 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f16d40019b0 con 0x7f16bc073160 2023-10-16T12:33:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.535+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f16bc073160 msgr2=0x7f16bc075580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.535+0000 7f16e29f0640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f16bc073160 0x7f16bc075580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.535+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16bc06af40 msgr2=0x7f16bc06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.535+0000 7f16e29f0640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16bc06af40 0x7f16bc06d400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16dc165030 tx=0x7f16c4030000 comp rx=0 tx=0).stop 2023-10-16T12:33:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.535+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0c4ed0 msgr2=0x7f16dc070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.535+0000 7f16e29f0640 1 --2- 172.21.15.60:0/2979065860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16dc0c4ed0 0x7f16dc070640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f16cc001630 tx=0x7f16cc001af0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.536+0000 7f16e0f8d640 1 -- 172.21.15.60:0/2979065860 reap_dead start 2023-10-16T12:33:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.537+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 shutdown_connections 2023-10-16T12:33:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.537+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 >> 172.21.15.60:0/2979065860 conn(0x7f16dc0ce9b0 msgr2=0x7f16dc06bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.537+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 shutdown_connections 2023-10-16T12:33:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.537+0000 7f16e29f0640 1 -- 172.21.15.60:0/2979065860 wait complete. 2023-10-16T12:33:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.540+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 --> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff55006b3d0 con 0x7ff5300730c0 2023-10-16T12:33:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.541+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 --> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faf940d3c20 con 0x7faf78073160 2023-10-16T12:33:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.541+0000 7ff54cff9640 1 -- 172.21.15.60:0/3501136311 <== osd.6 v2:172.21.15.144:6813/3196639449 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff548031ab0 con 0x7ff5300730c0 2023-10-16T12:33:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.542+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7ff5300730c0 msgr2=0x7ff5300754e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.542+0000 7faf867fc640 1 -- 172.21.15.60:0/2437252870 <== osd.5 v2:172.21.15.144:6814/1961528470 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faf90001350 con 0x7faf78073160 2023-10-16T12:33:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.542+0000 7ff556015640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7ff5300730c0 0x7ff5300754e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff53006aea0 msgr2=0x7ff53006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff556015640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff53006aea0 0x7ff53006d360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff538001760 tx=0x7ff538009040 comp rx=0 tx=0).stop 2023-10-16T12:33:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff550070c10 msgr2=0x7ff550155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff556015640 1 --2- 172.21.15.60:0/3501136311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff550070c10 0x7ff550155dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff540007d80 tx=0x7ff54000e4b0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff54ffff640 1 -- 172.21.15.60:0/3501136311 reap_dead start 2023-10-16T12:33:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 shutdown_connections 2023-10-16T12:33:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 >> 172.21.15.60:0/3501136311 conn(0x7ff5500cfac0 msgr2=0x7ff5500ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7faf78073160 msgr2=0x7faf78075580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7faf9bb6c640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7faf78073160 0x7faf78075580 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faf7806af40 msgr2=0x7faf7806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7faf9bb6c640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7faf7806af40 0x7faf7806d400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf8c02bef0 tx=0x7faf8c030000 comp rx=0 tx=0).stop 2023-10-16T12:33:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cfc70 msgr2=0x7faf940b78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.543+0000 7faf9bb6c640 1 --2- 172.21.15.60:0/2437252870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf940cfc70 0x7faf940b78c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faf8800e560 tx=0x7faf8800ea20 comp rx=0 tx=0).stop 2023-10-16T12:33:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7faf9a109640 1 -- 172.21.15.60:0/2437252870 reap_dead start 2023-10-16T12:33:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 shutdown_connections 2023-10-16T12:33:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 >> 172.21.15.60:0/2437252870 conn(0x7faf940d2cc0 msgr2=0x7faf94164a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 shutdown_connections 2023-10-16T12:33:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7ff556015640 1 -- 172.21.15.60:0/3501136311 wait complete. 2023-10-16T12:33:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 shutdown_connections 2023-10-16T12:33:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.544+0000 7faf9bb6c640 1 -- 172.21.15.60:0/2437252870 wait complete. 2023-10-16T12:33:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.545+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 --> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ecc002cb0 con 0x7f3ec8073160 2023-10-16T12:33:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.547+0000 7f3edaffd640 1 -- 172.21.15.60:0/2307908461 <== osd.7 v2:172.21.15.144:6808/3315484258 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3ecc002cb0 con 0x7f3ec8073160 2023-10-16T12:33:32.659 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:32.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T12:33:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.550+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f3ec8073160 msgr2=0x7f3ec8075580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.550+0000 7f3eec644640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f3ec8073160 0x7f3ec8075580 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.550+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806af40 msgr2=0x7f3ec806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.550+0000 7f3eec644640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806af40 0x7f3ec806d400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00017a0 tx=0x7f3ee0009040 comp rx=0 tx=0).stop 2023-10-16T12:33:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40addb0 msgr2=0x7f3ee40c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eec644640 1 --2- 172.21.15.60:0/2307908461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40addb0 0x7f3ee40c4690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3ed402b7e0 tx=0x7f3ed40046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eeabe1640 1 -- 172.21.15.60:0/2307908461 reap_dead start 2023-10-16T12:33:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 shutdown_connections 2023-10-16T12:33:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 >> 172.21.15.60:0/2307908461 conn(0x7f3ee40ce9b0 msgr2=0x7f3ee40cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:32.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 shutdown_connections 2023-10-16T12:33:32.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:32.551+0000 7f3eec644640 1 -- 172.21.15.60:0/2307908461 wait complete. 2023-10-16T12:33:32.663 INFO:teuthology.orchestra.run.smithi060.stdout:60129542151 2023-10-16T12:33:32.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T12:33:32.665 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:32.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T12:33:32.666 INFO:teuthology.orchestra.run.smithi060.stdout:60129542151 2023-10-16T12:33:32.667 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T12:33:32.668 INFO:teuthology.orchestra.run.smithi060.stdout:60129542151 2023-10-16T12:33:32.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T12:33:32.669 INFO:teuthology.orchestra.run.smithi060.stdout:60129542150 2023-10-16T12:33:32.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T12:33:32.670 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:32.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T12:33:33.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.005+0000 7f4b3dd53640 1 Processor -- start 2023-10-16T12:33:33.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.006+0000 7f4b3dd53640 1 -- start start 2023-10-16T12:33:33.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.008+0000 7f4b3dd53640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b300e64b0 0x7f4b3003e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.009+0000 7f4b3dd53640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b30053cf0 0x7f4b30054160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.009+0000 7f4b3dd53640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3003c8d0 con 0x7f4b300382c0 2023-10-16T12:33:33.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.009+0000 7f4b377fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b300e64b0 0x7f4b3003e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.010+0000 7f4b377fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b300e64b0 0x7f4b3003e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49050/0 (socket says 172.21.15.60:49050) 2023-10-16T12:33:33.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.010+0000 7f4b3dd53640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b300546a0 con 0x7f4b300e64b0 2023-10-16T12:33:33.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.010+0000 7f4b3dd53640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b30057200 con 0x7f4b30053cf0 2023-10-16T12:33:33.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.010+0000 7f4b377fe640 1 -- 172.21.15.60:0/863574809 learned_addr learned my addr 172.21.15.60:0/863574809 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.010+0000 7f4b37fff640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b30053cf0 0x7f4b30054160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.012+0000 7f4b377fe640 1 -- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b30053cf0 msgr2=0x7f4b30054160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.012+0000 7f4b377fe640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b30053cf0 0x7f4b30054160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.012+0000 7f4b367fc640 1 -- 172.21.15.60:0/863574809 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491299748 0 0) 0x7f4b24001110 con 0x7f4b300382c0 2023-10-16T12:33:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.012+0000 7f4b377fe640 1 -- 172.21.15.60:0/863574809 >> v1:172.21.15.60:6789/0 conn(0x7f4b300382c0 legacy=0x7f4b3003c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.012+0000 7f4b377fe640 1 -- 172.21.15.60:0/863574809 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300574a0 con 0x7f4b300e64b0 2023-10-16T12:33:33.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.013+0000 7f4b37fff640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b30053cf0 0x7f4b30054160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.015+0000 7f4b377fe640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b300e64b0 0x7f4b3003e940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009870 tx=0x7f4b2c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9255cf3935da731f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.015+0000 7f4b367fc640 1 -- 172.21.15.60:0/863574809 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c030610 con 0x7f4b300e64b0 2023-10-16T12:33:33.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.015+0000 7ff710c15640 1 Processor -- start 2023-10-16T12:33:33.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.016+0000 7f4b367fc640 1 -- 172.21.15.60:0/863574809 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c030770 con 0x7f4b300e64b0 2023-10-16T12:33:33.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.016+0000 7f4b367fc640 1 -- 172.21.15.60:0/863574809 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c030610 con 0x7f4b300e64b0 2023-10-16T12:33:33.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.016+0000 7ff710c15640 1 -- start start 2023-10-16T12:33:33.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.016+0000 7f4b3dd53640 1 -- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b300e64b0 msgr2=0x7f4b3003e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.016+0000 7f4b3dd53640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b300e64b0 0x7f4b3003e940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009870 tx=0x7f4b2c02b960 comp rx=0 tx=0).stop 2023-10-16T12:33:33.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.016+0000 7fb578373640 1 Processor -- start 2023-10-16T12:33:33.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7fb578373640 1 -- start start 2023-10-16T12:33:33.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7f4b3dd53640 1 -- 172.21.15.60:0/863574809 shutdown_connections 2023-10-16T12:33:33.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7f4b3dd53640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b30053cf0 0x7f4b30054160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7f4b3dd53640 1 --2- 172.21.15.60:0/863574809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b300e64b0 0x7f4b3003e940 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7f4b3dd53640 1 -- 172.21.15.60:0/863574809 >> 172.21.15.60:0/863574809 conn(0x7f4b30055a30 msgr2=0x7f4b30055e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7f4b3dd53640 1 -- 172.21.15.60:0/863574809 shutdown_connections 2023-10-16T12:33:33.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.017+0000 7f4b3dd53640 1 -- 172.21.15.60:0/863574809 wait complete. 2023-10-16T12:33:33.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.018+0000 7ff710c15640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff70c0adc30 0x7ff70c0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.018+0000 7ff710c15640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 0x7ff70c0cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.018+0000 7ff710c15640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70c0c9c50 con 0x7ff70c06f250 2023-10-16T12:33:33.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.018+0000 7ff710c15640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c0701d0 con 0x7ff70c0adc30 2023-10-16T12:33:33.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.018+0000 7ff710c15640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0cc870 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.019+0000 7fb578373640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb568053dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.019+0000 7fb578373640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb568054300 0x7fb568057410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.019+0000 7fb578373640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb568006eb0 con 0x7fb5680e31c0 2023-10-16T12:33:33.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.019+0000 7fb578373640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56803f130 con 0x7fb568054300 2023-10-16T12:33:33.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.019+0000 7fb578373640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb568007c70 con 0x7fb5680381b0 2023-10-16T12:33:33.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.019+0000 7fb57590e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb568053dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7fb57590e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb568053dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53248/0 (socket says 172.21.15.60:53248) 2023-10-16T12:33:33.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7f4b3dd53640 1 Processor -- start 2023-10-16T12:33:33.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7ff70ad9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 0x7ff70c0cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7ff70ad9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 0x7ff70c0cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53252/0 (socket says 172.21.15.60:53252) 2023-10-16T12:33:33.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7ff70ad9d640 1 -- 172.21.15.60:0/4145539928 learned_addr learned my addr 172.21.15.60:0/4145539928 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7fb57590e640 1 -- 172.21.15.60:0/774349361 learned_addr learned my addr 172.21.15.60:0/774349361 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7f4b3dd53640 1 -- start start 2023-10-16T12:33:33.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7ff709d9b640 1 --2- 172.21.15.60:0/4145539928 >> v2:172.21.15.144:3300/0 conn(0x7ff70c0adc30 0x7ff70c0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7fb57590e640 1 -- 172.21.15.60:0/774349361 >> v2:172.21.15.144:3300/0 conn(0x7fb568054300 msgr2=0x7fb568057410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7fb57590e640 1 --2- 172.21.15.60:0/774349361 >> v2:172.21.15.144:3300/0 conn(0x7fb568054300 0x7fb568057410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.020+0000 7fb57590e640 1 -- 172.21.15.60:0/774349361 >> v1:172.21.15.60:6789/0 conn(0x7fb5680e31c0 legacy=0x7fb56803e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7fb57590e640 1 -- 172.21.15.60:0/774349361 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568007190 con 0x7fb5680381b0 2023-10-16T12:33:33.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7fdb9a397640 1 Processor -- start 2023-10-16T12:33:33.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7fdb9a397640 1 -- start start 2023-10-16T12:33:33.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.022+0000 7f4b3dd53640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 0x7f4b3016d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7ff70ad9d640 1 -- 172.21.15.60:0/4145539928 >> v2:172.21.15.144:3300/0 conn(0x7ff70c0adc30 msgr2=0x7ff70c0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7ff70ad9d640 1 --2- 172.21.15.60:0/4145539928 >> v2:172.21.15.144:3300/0 conn(0x7ff70c0adc30 0x7ff70c0c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7ff70ad9d640 1 -- 172.21.15.60:0/4145539928 >> v1:172.21.15.144:6789/0 conn(0x7ff70c06f250 legacy=0x7ff70c06f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.021+0000 7ff70ad9d640 1 -- 172.21.15.60:0/4145539928 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0c9f20 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7ff70ad9d640 1 --2- 172.21.15.60:0/4145539928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 0x7ff70c0cd430 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009870 tx=0x7ff6f402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bbd35bc373210a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b3dd53640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b30053cf0 0x7f4b300e4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b3dd53640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b300e64b0 0x7f4b30058160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b3dd53640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3016daa0 con 0x7f4b300382c0 2023-10-16T12:33:33.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b3dd53640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3016c4c0 con 0x7f4b30053cf0 2023-10-16T12:33:33.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b3dd53640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3016c7a0 con 0x7f4b300e64b0 2023-10-16T12:33:33.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7fb57590e640 1 --2- 172.21.15.60:0/774349361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb568053dc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb570063e40 tx=0x7fb5700a5db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d2a2684e00cfe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7fdb9a397640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c033eb0 0x7fdb8c053040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7fb57510d640 1 -- 172.21.15.60:0/774349361 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700a6a10 con 0x7fb5680381b0 2023-10-16T12:33:33.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7fdb9a397640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 0x7fdb8c0e2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb9a397640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb8c00c0e0 0x7fdb8c006080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 0x7f4b3016d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 0x7f4b3016d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57926/0 (socket says 172.21.15.60:57926) 2023-10-16T12:33:33.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b377fe640 1 -- 172.21.15.60:0/510899378 learned_addr learned my addr 172.21.15.60:0/510899378 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.023+0000 7f4b36ffd640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b30053cf0 0x7f4b300e4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7f4b37fff640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b300e64b0 0x7f4b30058160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 0x7fdb8c0e2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 0x7fdb8c0e2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57940/0 (socket says 172.21.15.60:57940) 2023-10-16T12:33:33.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb937fe640 1 -- 172.21.15.60:0/1777549184 learned_addr learned my addr 172.21.15.60:0/1777549184 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c0542e0 con 0x7fdb8c050120 2023-10-16T12:33:33.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c034290 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0536d0 con 0x7fdb8c033eb0 2023-10-16T12:33:33.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7ff70959a640 1 -- 172.21.15.60:0/4145539928 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4030860 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7ff70959a640 1 -- 172.21.15.60:0/4145539928 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f40309c0 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7ff70959a640 1 -- 172.21.15.60:0/4145539928 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4030d90 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb93fff640 1 --2- 172.21.15.60:0/1777549184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c033eb0 0x7fdb8c053040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7fdb98934640 1 --2- 172.21.15.60:0/1777549184 >> v2:172.21.15.144:3300/0 conn(0x7fdb8c00c0e0 0x7fdb8c006080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.025+0000 7fb57510d640 1 -- 172.21.15.60:0/774349361 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5700a6bb0 con 0x7fb5680381b0 2023-10-16T12:33:33.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.025+0000 7fb57510d640 1 -- 172.21.15.60:0/774349361 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700ae440 con 0x7fb5680381b0 2023-10-16T12:33:33.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7f4b377fe640 1 -- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b300e64b0 msgr2=0x7f4b30058160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7f4b377fe640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b300e64b0 0x7f4b30058160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7f4b377fe640 1 -- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b30053cf0 msgr2=0x7f4b300e4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7f4b377fe640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b30053cf0 0x7f4b300e4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.024+0000 7f4b377fe640 1 -- 172.21.15.60:0/510899378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3804e6d0 con 0x7f4b300382c0 2023-10-16T12:33:33.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.025+0000 7ff710c15640 1 -- 172.21.15.60:0/4145539928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 msgr2=0x7ff70c0cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.025+0000 7f4b377fe640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 0x7f4b3016d560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c000c00 tx=0x7f4b2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.025+0000 7ff710c15640 1 --2- 172.21.15.60:0/4145539928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 0x7ff70c0cd430 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009870 tx=0x7ff6f402b9a0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7ff710c15640 1 -- 172.21.15.60:0/4145539928 shutdown_connections 2023-10-16T12:33:33.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7ff710c15640 1 --2- 172.21.15.60:0/4145539928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c0ccfa0 0x7ff70c0cd430 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7ff710c15640 1 --2- 172.21.15.60:0/4145539928 >> v2:172.21.15.144:3300/0 conn(0x7ff70c0adc30 0x7ff70c0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7ff710c15640 1 -- 172.21.15.60:0/4145539928 >> 172.21.15.60:0/4145539928 conn(0x7ff70c0ce6e0 msgr2=0x7ff70c0ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 -- 172.21.15.60:0/774349361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 msgr2=0x7fb568053dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 --2- 172.21.15.60:0/774349361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb568053dc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb570063e40 tx=0x7fb5700a5db0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fdb93fff640 1 -- 172.21.15.60:0/1777549184 >> v2:172.21.15.144:3300/0 conn(0x7fdb8c00c0e0 msgr2=0x7fdb8c006080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fdb93fff640 1 --2- 172.21.15.60:0/1777549184 >> v2:172.21.15.144:3300/0 conn(0x7fdb8c00c0e0 0x7fdb8c006080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fdb93fff640 1 -- 172.21.15.60:0/1777549184 >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 msgr2=0x7fdb8c0e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fdb93fff640 1 --2- 172.21.15.60:0/1777549184 >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 0x7fdb8c0e2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fdb93fff640 1 -- 172.21.15.60:0/1777549184 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c0539b0 con 0x7fdb8c033eb0 2023-10-16T12:33:33.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fdb937fe640 1 --2- 172.21.15.60:0/1777549184 >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 0x7fdb8c0e2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:33.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 -- 172.21.15.60:0/774349361 shutdown_connections 2023-10-16T12:33:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 --2- 172.21.15.60:0/774349361 >> v2:172.21.15.144:3300/0 conn(0x7fb568054300 0x7fb568057410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 --2- 172.21.15.60:0/774349361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb568053dc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 -- 172.21.15.60:0/774349361 >> 172.21.15.60:0/774349361 conn(0x7fb5680590f0 msgr2=0x7fb568059500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.026+0000 7fb578373640 1 -- 172.21.15.60:0/774349361 shutdown_connections 2023-10-16T12:33:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c03e040 con 0x7f4b300382c0 2023-10-16T12:33:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c030e50 con 0x7f4b300382c0 2023-10-16T12:33:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c030610 con 0x7f4b300382c0 2023-10-16T12:33:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7f4b3dd53640 1 -- 172.21.15.60:0/510899378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c0094d0 con 0x7f4b300382c0 2023-10-16T12:33:33.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7fb578373640 1 -- 172.21.15.60:0/774349361 wait complete. 2023-10-16T12:33:33.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7f4b3dd53640 1 -- 172.21.15.60:0/510899378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3016cd60 con 0x7f4b300382c0 2023-10-16T12:33:33.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.028+0000 7fdb93fff640 1 --2- 172.21.15.60:0/1777549184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c033eb0 0x7fdb8c053040 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00df80 tx=0x7fdb8802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70704bb177337163 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.027+0000 7ff710c15640 1 -- 172.21.15.60:0/4145539928 shutdown_connections 2023-10-16T12:33:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.029+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b10005250 con 0x7f4b300382c0 2023-10-16T12:33:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.028+0000 7fdb92ffd640 1 -- 172.21.15.60:0/1777549184 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb880316d0 con 0x7fdb8c033eb0 2023-10-16T12:33:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.028+0000 7fdb92ffd640 1 -- 172.21.15.60:0/1777549184 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb88031830 con 0x7fdb8c033eb0 2023-10-16T12:33:33.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.029+0000 7fb578373640 1 Processor -- start 2023-10-16T12:33:33.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.028+0000 7fdb92ffd640 1 -- 172.21.15.60:0/1777549184 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88031bc0 con 0x7fdb8c033eb0 2023-10-16T12:33:33.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.029+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c033eb0 msgr2=0x7fdb8c053040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.030+0000 7fb578373640 1 -- start start 2023-10-16T12:33:33.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.029+0000 7fdb9a397640 1 --2- 172.21.15.60:0/1777549184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c033eb0 0x7fdb8c053040 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00df80 tx=0x7fdb8802b770 comp rx=0 tx=0).stop 2023-10-16T12:33:33.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.028+0000 7ff710c15640 1 -- 172.21.15.60:0/4145539928 wait complete. 2023-10-16T12:33:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.030+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 shutdown_connections 2023-10-16T12:33:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7fdb9a397640 1 --2- 172.21.15.60:0/1777549184 >> v2:172.21.15.144:3300/0 conn(0x7fdb8c00c0e0 0x7fdb8c006080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7fdb9a397640 1 --2- 172.21.15.60:0/1777549184 >> v2:172.21.15.60:3300/0 conn(0x7fdb8c050120 0x7fdb8c0e2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7fdb9a397640 1 --2- 172.21.15.60:0/1777549184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c033eb0 0x7fdb8c053040 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 >> 172.21.15.60:0/1777549184 conn(0x7fdb8c054b00 msgr2=0x7fdb8c054ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 shutdown_connections 2023-10-16T12:33:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7fdb9a397640 1 -- 172.21.15.60:0/1777549184 wait complete. 2023-10-16T12:33:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.032+0000 7fb578373640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb56800ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.032+0000 7fb578373640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 0x7fb568186aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.032+0000 7fb578373640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680e31c0 0x7fb568185970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.031+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4b2c004030 con 0x7f4b300382c0 2023-10-16T12:33:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb57590e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 0x7fb568186aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb57590e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 0x7fb568186aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57954/0 (socket says 172.21.15.60:57954) 2023-10-16T12:33:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb57610f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb56800ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7f1468d90640 1 Processor -- start 2023-10-16T12:33:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.032+0000 7f6eec589640 1 Processor -- start 2023-10-16T12:33:33.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.032+0000 7f6eec589640 1 -- start start 2023-10-16T12:33:33.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7f4b23fff640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4b1c06af40 0x7f4b1c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb57590e640 1 -- 172.21.15.60:0/403753440 learned_addr learned my addr 172.21.15.60:0/403753440 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680531c0 con 0x7fb568054300 2023-10-16T12:33:33.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb568186fe0 con 0x7fb5680e31c0 2023-10-16T12:33:33.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.033+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56800be30 con 0x7fb5680381b0 2023-10-16T12:33:33.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f1468d90640 1 -- start start 2023-10-16T12:33:33.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f6eec589640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ee4159110 0x7f6ee415cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f6eec589640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee406a4b0 0x7f6ee406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b2c0abb20 con 0x7f4b300382c0 2023-10-16T12:33:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f6eec589640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee41594f0 con 0x7f6ee40ade50 2023-10-16T12:33:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f6eec589640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee415d130 con 0x7f6ee4159110 2023-10-16T12:33:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f6eec589640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee415d400 con 0x7f6ee406a4b0 2023-10-16T12:33:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7fb57590e640 1 -- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 msgr2=0x7fb56800ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7fb57590e640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb56800ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7fb57590e640 1 -- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680e31c0 msgr2=0x7fb568185970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7fb576910640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680e31c0 0x7fb568185970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.034+0000 7f4b36ffd640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4b1c06af40 0x7f4b1c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7f6ee9b24640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6ee40ade50 0x7f6ee4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54240/0 (socket says 172.21.15.60:54240) 2023-10-16T12:33:33.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7f6ee9b24640 1 -- 172.21.15.60:0/4062518568 learned_addr learned my addr 172.21.15.60:0/4062518568 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:33.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7f6eea325640 1 --2- 172.21.15.60:0/4062518568 >> v2:172.21.15.60:3300/0 conn(0x7f6ee4159110 0x7f6ee415cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7fb57590e640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680e31c0 0x7fb568185970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7fdb9a397640 1 Processor -- start 2023-10-16T12:33:33.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7fb57590e640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57004f050 con 0x7fb568054300 2023-10-16T12:33:33.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7f6eeab26640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee406a4b0 0x7f6ee406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7fb576910640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680e31c0 0x7fb568185970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:33.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7f6eea325640 1 -- 172.21.15.60:0/4062518568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee406a4b0 msgr2=0x7f6ee406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f6eea325640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee406a4b0 0x7f6ee406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f6eea325640 1 -- 172.21.15.60:0/4062518568 >> v1:172.21.15.60:6789/0 conn(0x7f6ee40ade50 legacy=0x7f6ee4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f6eea325640 1 -- 172.21.15.60:0/4062518568 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee415d6e0 con 0x7f6ee4159110 2023-10-16T12:33:33.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f6eeab26640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee406a4b0 0x7f6ee406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:33.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7ff710c15640 1 Processor -- start 2023-10-16T12:33:33.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f1468d90640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14640cf710 0x7f14640b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f1468d90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f14640b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f1468d90640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640d3980 con 0x7f14640b38e0 2023-10-16T12:33:33.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f1468d90640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640d3c60 con 0x7f14640cf710 2023-10-16T12:33:33.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7f1468d90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14640ce630 con 0x7f14640cfaf0 2023-10-16T12:33:33.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7fb57590e640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 0x7fb568186aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb57004f940 tx=0x7fb570067b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.036+0000 7ff710c15640 1 -- start start 2023-10-16T12:33:33.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7f146259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14640cf710 0x7f14640b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7f1462d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f14640b38e0 0x7f14640d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60286/0 (socket says 172.21.15.60:60286) 2023-10-16T12:33:33.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7f1462d9d640 1 -- 172.21.15.60:0/1603780281 learned_addr learned my addr 172.21.15.60:0/1603780281 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:33.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.035+0000 7fdb9a397640 1 -- start start 2023-10-16T12:33:33.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1603780281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f14640b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7fdb9a397640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 0x7fdb8c0d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7fdb9a397640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c033eb0 0x7fdb8c184980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7fdb9a397640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c050120 0x7fdb8c184ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7fdb9a397640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c185400 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7fdb9a397640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c189520 con 0x7fdb8c033eb0 2023-10-16T12:33:33.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7fdb9a397640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c188c30 con 0x7fdb8c050120 2023-10-16T12:33:33.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7f4b36ffd640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4b1c06af40 0x7f4b1c06d400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4b30058cc0 tx=0x7f4b24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7f146159a640 1 -- 172.21.15.60:0/1603780281 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392613254 0 0) 0x7f14640d3980 con 0x7f14640b38e0 2023-10-16T12:33:33.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 0x7fdb8c0d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.037+0000 7f6eea325640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee415cbf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000a940 tx=0x7f6ee002de90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc2ce2149ce2e25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7ff710c15640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c06f250 0x7ff70c0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7ff710c15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff70c0adc30 0x7ff70c1601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7f146159a640 1 -- 172.21.15.60:0/1603780281 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c003520 con 0x7f14640b38e0 2023-10-16T12:33:33.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7f1461d9b640 1 -- 172.21.15.60:0/1603780281 >> v2:172.21.15.144:3300/0 conn(0x7f14640cf710 msgr2=0x7f14640b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1603780281 >> v2:172.21.15.144:3300/0 conn(0x7f14640cf710 0x7f14640b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7f1461d9b640 1 -- 172.21.15.60:0/1603780281 >> v1:172.21.15.144:6789/0 conn(0x7f14640b38e0 legacy=0x7f14640d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7f1461d9b640 1 -- 172.21.15.60:0/1603780281 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640ce910 con 0x7f14640cfaf0 2023-10-16T12:33:33.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7ff710c15640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70c0ccfa0 0x7ff70c0c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7ff710c15640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b3cd0 con 0x7ff70c0adc30 2023-10-16T12:33:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7ff710c15640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c160720 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7ff710c15640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b1410 con 0x7ff70c06f250 2023-10-16T12:33:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb937fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c033eb0 0x7fdb8c184980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 0x7fdb8c0d84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57962/0 (socket says 172.21.15.60:57962) 2023-10-16T12:33:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 -- 172.21.15.60:0/677238805 learned_addr learned my addr 172.21.15.60:0/677238805 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb98934640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c050120 0x7fdb8c184ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7ff70a59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c06f250 0x7ff70c0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7ff70a59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c06f250 0x7ff70c0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53310/0 (socket says 172.21.15.60:53310) 2023-10-16T12:33:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7ff70a59c640 1 -- 172.21.15.60:0/3606865425 learned_addr learned my addr 172.21.15.60:0/3606865425 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7ff70ad9d640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70c0ccfa0 0x7ff70c0c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 -- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c050120 msgr2=0x7fdb8c184ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c050120 0x7fdb8c184ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 -- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c033eb0 msgr2=0x7fdb8c184980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c033eb0 0x7fdb8c184980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.038+0000 7fdb93fff640 1 -- 172.21.15.60:0/677238805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9404e6d0 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7ff709d9b640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff70c0adc30 0x7ff70c1601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7fdb93fff640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 0x7fdb8c0d84a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdb880097d0 tx=0x7fdb880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.039+0000 7f00e2683640 1 Processor -- start 2023-10-16T12:33:33.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7ff70ad9d640 1 -- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c06f250 msgr2=0x7ff70c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7ff70ad9d640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c06f250 0x7ff70c0b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7ff70ad9d640 1 -- 172.21.15.60:0/3606865425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff70c0adc30 msgr2=0x7ff70c1601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7ff70ad9d640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff70c0adc30 0x7ff70c1601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7ff70ad9d640 1 -- 172.21.15.60:0/3606865425 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40094d0 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1603780281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f14640b3250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14580097a0 tx=0x7f145802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6bb28be17f5c98c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7f00e2683640 1 -- start start 2023-10-16T12:33:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8803e940 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7f146159a640 1 -- 172.21.15.60:0/1603780281 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458031780 con 0x7f14640cfaf0 2023-10-16T12:33:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7f146159a640 1 -- 172.21.15.60:0/1603780281 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14580318e0 con 0x7f14640cfaf0 2023-10-16T12:33:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7f146159a640 1 -- 172.21.15.60:0/1603780281 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458031c70 con 0x7f14640cfaf0 2023-10-16T12:33:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7ff70ad9d640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70c0ccfa0 0x7ff70c0c7c70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4033000 tx=0x7ff6f4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7f6ee9323640 1 -- 172.21.15.60:0/4062518568 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee002ec40 con 0x7f6ee4159110 2023-10-16T12:33:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7f6ee9323640 1 -- 172.21.15.60:0/4062518568 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee002eda0 con 0x7f6ee4159110 2023-10-16T12:33:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7f6ee9323640 1 -- 172.21.15.60:0/4062518568 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee002ec40 con 0x7f6ee4159110 2023-10-16T12:33:33.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.040+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700be070 con 0x7fb568054300 2023-10-16T12:33:33.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f403e040 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56800ddc0 con 0x7fb568054300 2023-10-16T12:33:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56800b220 con 0x7fb568054300 2023-10-16T12:33:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7f1468d90640 1 -- 172.21.15.60:0/1603780281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 msgr2=0x7f14640b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f1468d90640 1 --2- 172.21.15.60:0/1603780281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f14640b3250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14580097a0 tx=0x7f145802b820 comp rx=0 tx=0).stop 2023-10-16T12:33:33.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c0b16f0 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f6eec589640 1 -- 172.21.15.60:0/4062518568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 msgr2=0x7f6ee415cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f6eec589640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee415cbf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000a940 tx=0x7f6ee002de90 comp rx=0 tx=0).stop 2023-10-16T12:33:33.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb88009400 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8803eae0 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8803ee90 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f1468d90640 1 -- 172.21.15.60:0/1603780281 shutdown_connections 2023-10-16T12:33:33.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f00e2683640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00dc0cd970 0x7f00dc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f1468d90640 1 --2- 172.21.15.60:0/1603780281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f14640b3250 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f1468d90640 1 --2- 172.21.15.60:0/1603780281 >> v2:172.21.15.144:3300/0 conn(0x7f14640cf710 0x7f14640b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f1468d90640 1 -- 172.21.15.60:0/1603780281 >> 172.21.15.60:0/1603780281 conn(0x7f14640d2cc0 msgr2=0x7f14640d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4b2c076bd0 con 0x7f4b300382c0 2023-10-16T12:33:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5700a6a30 con 0x7fb568054300 2023-10-16T12:33:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700b7820 con 0x7fb568054300 2023-10-16T12:33:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c0cf800 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f00e2683640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc0b5d90 con 0x7f00dc15d410 2023-10-16T12:33:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f00e2683640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc160fc0 con 0x7f00dc15d030 2023-10-16T12:33:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.042+0000 7f00e2683640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc0d2b50 con 0x7f00dc0cd970 2023-10-16T12:33:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f00e0c20640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00dc0cd970 0x7f00dc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f00dbfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f00dc15d030 0x7f00dc161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60288/0 (socket says 172.21.15.60:60288) 2023-10-16T12:33:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f00dbfff640 1 -- 172.21.15.60:0/1827327846 learned_addr learned my addr 172.21.15.60:0/1827327846 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f1468d90640 1 -- 172.21.15.60:0/1603780281 shutdown_connections 2023-10-16T12:33:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f6eec589640 1 -- 172.21.15.60:0/4062518568 shutdown_connections 2023-10-16T12:33:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f6eec589640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee406a4b0 0x7f6ee406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f6eec589640 1 --2- 172.21.15.60:0/4062518568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee415cbf0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f6eec589640 1 -- 172.21.15.60:0/4062518568 >> 172.21.15.60:0/4062518568 conn(0x7f6ee40ce9b0 msgr2=0x7f6ee40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f6eec589640 1 -- 172.21.15.60:0/4062518568 shutdown_connections 2023-10-16T12:33:33.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f6eec589640 1 -- 172.21.15.60:0/4062518568 wait complete. 2023-10-16T12:33:33.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7fcc66fc4640 1 Processor -- start 2023-10-16T12:33:33.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.043+0000 7f1468d90640 1 -- 172.21.15.60:0/1603780281 wait complete. 2023-10-16T12:33:33.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.041+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c0b1cb0 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4039c10 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7f00e0c20640 1 -- 172.21.15.60:0/1827327846 >> v1:172.21.15.144:6789/0 conn(0x7f00dc15d030 legacy=0x7f00dc161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7f00e0c20640 1 -- 172.21.15.60:0/1827327846 >> v1:172.21.15.60:6789/0 conn(0x7f00dc15d410 legacy=0x7f00dc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558005250 con 0x7fb568054300 2023-10-16T12:33:33.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.045+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8c005340 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7fcc66fc4640 1 -- start start 2023-10-16T12:33:33.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.044+0000 7f00e0c20640 1 -- 172.21.15.60:0/1827327846 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc0d25d0 con 0x7f00dc0cd970 2023-10-16T12:33:33.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.045+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f0005250 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f403e040 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7f1468d90640 1 Processor -- start 2023-10-16T12:33:33.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7f1468d90640 1 -- start start 2023-10-16T12:33:33.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7f00e0c20640 1 --2- 172.21.15.60:0/1827327846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc0cd970 0x7f00dc0d1630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009d90 tx=0x7f00d402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a2b5f5082507dbc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7f6eec589640 1 Processor -- start 2023-10-16T12:33:33.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7fcc66fc4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcc600cf1c0 0x7fcc6006c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7f6eec589640 1 -- start start 2023-10-16T12:33:33.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7fcc66fc4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc6006a910 0x7fcc601582f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7fcc66fc4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc600cc8c0 con 0x7fcc6006f440 2023-10-16T12:33:33.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7fcc66fc4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6006adc0 con 0x7fcc600cf1c0 2023-10-16T12:33:33.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7fcc66fc4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600ccba0 con 0x7fcc6006a910 2023-10-16T12:33:33.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7fcc657c1640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fcc6006f440 0x7fcc600cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60294/0 (socket says 172.21.15.60:60294) 2023-10-16T12:33:33.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7fcc657c1640 1 -- 172.21.15.60:0/3948131439 learned_addr learned my addr 172.21.15.60:0/3948131439 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:33.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.047+0000 7f1468d90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 0x7f14641600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f1468d90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14640cf710 0x7f14640c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f1468d90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f1464070a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f1468d90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640cb720 con 0x7f14640cf710 2023-10-16T12:33:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f1468d90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640b28e0 con 0x7f14640b38e0 2023-10-16T12:33:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f1468d90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146415fde0 con 0x7f14640cfaf0 2023-10-16T12:33:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7fcc667c3640 1 --2- 172.21.15.60:0/3948131439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc6006a910 0x7fcc601582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7fcc64fc0640 1 -- 172.21.15.60:0/3948131439 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035159402 0 0) 0x7fcc600cc8c0 con 0x7fcc6006f440 2023-10-16T12:33:33.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7fcc64fc0640 1 -- 172.21.15.60:0/3948131439 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c003520 con 0x7fcc6006f440 2023-10-16T12:33:33.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f00db7fe640 1 -- 172.21.15.60:0/1827327846 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d402cb70 con 0x7f00dc0cd970 2023-10-16T12:33:33.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f146259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 0x7f14641600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f00db7fe640 1 -- 172.21.15.60:0/1827327846 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d402ccd0 con 0x7f00dc0cd970 2023-10-16T12:33:33.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eec589640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 0x7f6ee41f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f00db7fe640 1 -- 172.21.15.60:0/1827327846 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4034660 con 0x7f00dc0cd970 2023-10-16T12:33:33.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.046+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdb8803b040 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7fdb90ff9640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdb7806ae70 0x7fdb7806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb880ab220 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f1462d9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f1464070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f146259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 0x7f14641600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49144/0 (socket says 172.21.15.60:49144) 2023-10-16T12:33:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.048+0000 7f146259c640 1 -- 172.21.15.60:0/1477962585 learned_addr learned my addr 172.21.15.60:0/1477962585 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14640cf710 0x7f14640c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eec589640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee40ade50 0x7f6ee40ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eec589640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee40af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eec589640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41550e0 con 0x7f6ee4159110 2023-10-16T12:33:33.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eec589640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee4150710 con 0x7f6ee406a4b0 2023-10-16T12:33:33.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eec589640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41f6850 con 0x7f6ee40ade50 2023-10-16T12:33:33.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eeab26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee40af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eea325640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 0x7f6ee41f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7fcc667c3640 1 -- 172.21.15.60:0/3948131439 >> v2:172.21.15.60:3300/0 conn(0x7fcc600cf1c0 msgr2=0x7fcc6006c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:33.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7fcc667c3640 1 --2- 172.21.15.60:0/3948131439 >> v2:172.21.15.60:3300/0 conn(0x7fcc600cf1c0 0x7fcc6006c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7fcc667c3640 1 -- 172.21.15.60:0/3948131439 >> v1:172.21.15.144:6789/0 conn(0x7fcc6006f440 legacy=0x7fcc600cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7fcc667c3640 1 -- 172.21.15.60:0/3948131439 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc600cce80 con 0x7fcc6006a910 2023-10-16T12:33:33.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f00e2683640 1 -- 172.21.15.60:0/1827327846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc0cd970 msgr2=0x7f00dc0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f00e2683640 1 --2- 172.21.15.60:0/1827327846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc0cd970 0x7f00dc0d1630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009d90 tx=0x7f00d402be40 comp rx=0 tx=0).stop 2023-10-16T12:33:33.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f146259c640 1 -- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 msgr2=0x7f1464070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eea325640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 0x7f6ee41f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49152/0 (socket says 172.21.15.60:49152) 2023-10-16T12:33:33.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f6eea325640 1 -- 172.21.15.60:0/2593532649 learned_addr learned my addr 172.21.15.60:0/2593532649 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.049+0000 7f146259c640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f1464070a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f6ee9b24640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee40ade50 0x7f6ee40ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f00e2683640 1 -- 172.21.15.60:0/1827327846 shutdown_connections 2023-10-16T12:33:33.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f00e2683640 1 --2- 172.21.15.60:0/1827327846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc0cd970 0x7f00dc0d1630 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f00e2683640 1 -- 172.21.15.60:0/1827327846 >> 172.21.15.60:0/1827327846 conn(0x7f00dc0cf720 msgr2=0x7f00dc0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f146259c640 1 -- 172.21.15.60:0/1477962585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14640cf710 msgr2=0x7f14640c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f146259c640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14640cf710 0x7f14640c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f146259c640 1 -- 172.21.15.60:0/1477962585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458009400 con 0x7f14640b38e0 2023-10-16T12:33:33.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f00e2683640 1 -- 172.21.15.60:0/1827327846 shutdown_connections 2023-10-16T12:33:33.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f00e2683640 1 -- 172.21.15.60:0/1827327846 wait complete. 2023-10-16T12:33:33.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f6eea325640 1 -- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee40ade50 msgr2=0x7f6ee40ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f6eea325640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee40ade50 0x7f6ee40ab9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f6eea325640 1 -- 172.21.15.60:0/2593532649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 msgr2=0x7f6ee40af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f6eea325640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee40af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f6eea325640 1 -- 172.21.15.60:0/2593532649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee000a5a0 con 0x7f6ee406a4b0 2023-10-16T12:33:33.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14640cf710 0x7f14640c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:33.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.050+0000 7f146259c640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 0x7f14641600c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f145c00efc0 tx=0x7f145c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.051+0000 7fcc667c3640 1 --2- 172.21.15.60:0/3948131439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc6006a910 0x7fcc601582f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800aea0 tx=0x7fcc5802e170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea87bddf4169c22a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.051+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c009bd0 con 0x7f14640b38e0 2023-10-16T12:33:33.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.051+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c00f040 con 0x7f14640b38e0 2023-10-16T12:33:33.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.051+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff6f40443f0 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c0146a0 con 0x7f14640b38e0 2023-10-16T12:33:33.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14640cb930 con 0x7f14640b38e0 2023-10-16T12:33:33.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7fcc64fc0640 1 -- 172.21.15.60:0/3948131439 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5802ed10 con 0x7fcc6006a910 2023-10-16T12:33:33.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7fcc64fc0640 1 -- 172.21.15.60:0/3948131439 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5800a2e0 con 0x7fcc6006a910 2023-10-16T12:33:33.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7fcc64fc0640 1 -- 172.21.15.60:0/3948131439 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58008660 con 0x7fcc6006a910 2023-10-16T12:33:33.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14640cbef0 con 0x7f14640b38e0 2023-10-16T12:33:33.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7f00e2683640 1 Processor -- start 2023-10-16T12:33:33.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7f6eea325640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 0x7f6ee41f6310 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000aa60 tx=0x7f6ee00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.051+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb5700b5040 con 0x7fb568054300 2023-10-16T12:33:33.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.052+0000 7fb566ffd640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb55406ae70 0x7fb55406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7f00e2683640 1 -- start start 2023-10-16T12:33:33.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5700f3fd0 con 0x7fb568054300 2023-10-16T12:33:33.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7ff702ffd640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff6ec06ae90 0x7ff6ec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3948131439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc6006a910 msgr2=0x7fcc601582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3948131439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc6006a910 0x7fcc601582f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800aea0 tx=0x7fcc5802e170 comp rx=0 tx=0).stop 2023-10-16T12:33:33.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fb57610f640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb55406ae70 0x7fb55406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3948131439 shutdown_connections 2023-10-16T12:33:33.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3948131439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc6006a910 0x7fcc601582f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3948131439 >> v2:172.21.15.60:3300/0 conn(0x7fcc600cf1c0 0x7fcc6006c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3948131439 >> 172.21.15.60:0/3948131439 conn(0x7fcc600ce6a0 msgr2=0x7fcc600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6f40ab920 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3948131439 shutdown_connections 2023-10-16T12:33:33.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.053+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3948131439 wait complete. 2023-10-16T12:33:33.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.054+0000 7fdb937fe640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdb7806ae70 0x7fdb7806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.054+0000 7ff70a59c640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff6ec06ae90 0x7ff6ec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.054+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14640d2890 con 0x7f14640b38e0 2023-10-16T12:33:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.054+0000 7f00e2683640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 0x7f00dc0c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.054+0000 7f00e2683640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc15d410 0x7f00dc15a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00e2683640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00dc15ae80 0x7f00dc1ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00e2683640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc15b710 con 0x7f00dc15d410 2023-10-16T12:33:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00e2683640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc0c8dd0 con 0x7f00dc15d030 2023-10-16T12:33:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00e2683640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc15d930 con 0x7f00dc15ae80 2023-10-16T12:33:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7fb57610f640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb55406ae70 0x7fb55406d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb56c000f60 tx=0x7fb56c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 0x7f00dc0c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 0x7f00dc0c8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49162/0 (socket says 172.21.15.60:49162) 2023-10-16T12:33:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00dbfff640 1 -- 172.21.15.60:0/165744376 learned_addr learned my addr 172.21.15.60:0/165744376 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb88076350 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00d3fff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc15d410 0x7f00dc15a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7f00e0c20640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00dc15ae80 0x7f00dc1ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7ff70a59c640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff6ec06ae90 0x7ff6ec06d350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff704001ef0 tx=0x7ff704009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.055+0000 7fdb937fe640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdb7806ae70 0x7fdb7806d330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb94068d00 tx=0x7fdb9406c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7f00dbfff640 1 -- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00dc15ae80 msgr2=0x7f00dc1ffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7f00dbfff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00dc15ae80 0x7f00dc1ffd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7f00dbfff640 1 -- 172.21.15.60:0/165744376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc15d410 msgr2=0x7f00dc15a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7fcc66fc4640 1 Processor -- start 2023-10-16T12:33:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.058+0000 7fcc66fc4640 1 -- start start 2023-10-16T12:33:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7f00dbfff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc15d410 0x7f00dc15a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7f00dbfff640 1 -- 172.21.15.60:0/165744376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d4009a70 con 0x7f00dc15d030 2023-10-16T12:33:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.056+0000 7f00d3fff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc15d410 0x7f00dc15a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:33.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.057+0000 7f00dbfff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 0x7f00dc0c8890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00c780 tx=0x7f00cc00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc66fc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 0x7fcc6015a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc66fc4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6006f440 0x7fcc600c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc66fc4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc600cf1c0 0x7fcc600c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc66fc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600c65c0 con 0x7fcc6006a910 2023-10-16T12:33:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc66fc4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600c7450 con 0x7fcc6006f440 2023-10-16T12:33:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc66fc4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600c7730 con 0x7fcc600cf1c0 2023-10-16T12:33:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc65fc2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 0x7fcc6015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc65fc2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 0x7fcc6015a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58012/0 (socket says 172.21.15.60:58012) 2023-10-16T12:33:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc65fc2640 1 -- 172.21.15.60:0/3660716034 learned_addr learned my addr 172.21.15.60:0/3660716034 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:33.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5700f03a0 con 0x7fb568054300 2023-10-16T12:33:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.060+0000 7fcc667c3640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc600cf1c0 0x7fcc600c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc65fc2640 1 -- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc600cf1c0 msgr2=0x7fcc600c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc65fc2640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc600cf1c0 0x7fcc600c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc65fc2640 1 -- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6006f440 msgr2=0x7fcc600c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc657c1640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6006f440 0x7fcc600c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc65fc2640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6006f440 0x7fcc600c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc65fc2640 1 -- 172.21.15.60:0/3660716034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5800ab80 con 0x7fcc6006a910 2023-10-16T12:33:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc657c1640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6006f440 0x7fcc600c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc65fc2640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 0x7fcc6015a220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00b410 tx=0x7fcc5c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c011830 con 0x7fcc6006a910 2023-10-16T12:33:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c0119b0 con 0x7fcc6006a910 2023-10-16T12:33:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.061+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c011d60 con 0x7fcc6006a910 2023-10-16T12:33:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.062+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc600c7a10 con 0x7fcc6006a910 2023-10-16T12:33:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.062+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6015a760 con 0x7fcc6006a910 2023-10-16T12:33:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.063+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc600c7f30 con 0x7fcc6006a910 2023-10-16T12:33:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.064+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc5c01a400 con 0x7fcc6006a910 2023-10-16T12:33:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.065+0000 7fcc527fc640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcc4406ae70 0x7fcc4406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.065+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc5c089850 con 0x7fcc6006a910 2023-10-16T12:33:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.066+0000 7fcc657c1640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcc4406ae70 0x7fcc4406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.067+0000 7fcc657c1640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcc4406ae70 0x7fcc4406d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcc540011e0 tx=0x7fcc54009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.068+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc600c7f30 con 0x7fcc6006a910 2023-10-16T12:33:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.071+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6f40769e0 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.071+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc0072d0 con 0x7f00dc15d030 2023-10-16T12:33:33.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.071+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc007430 con 0x7f00dc15d030 2023-10-16T12:33:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.071+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc0c9000 con 0x7f00dc15d030 2023-10-16T12:33:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.071+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc0047e0 con 0x7f00dc15d030 2023-10-16T12:33:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.071+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc15dba0 con 0x7f00dc15d030 2023-10-16T12:33:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.072+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0003e50 con 0x7f6ee406a4b0 2023-10-16T12:33:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.072+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00dc0ce9a0 con 0x7f00dc15d030 2023-10-16T12:33:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.072+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee0006e10 con 0x7f6ee406a4b0 2023-10-16T12:33:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.072+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0036640 con 0x7f6ee406a4b0 2023-10-16T12:33:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.072+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee41544d0 con 0x7f6ee406a4b0 2023-10-16T12:33:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.073+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee4154a90 con 0x7f6ee406a4b0 2023-10-16T12:33:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.074+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f145c014800 con 0x7f14640b38e0 2023-10-16T12:33:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.074+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee404e160 con 0x7f6ee406a4b0 2023-10-16T12:33:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.076+0000 7f1452ffd640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f144406ae70 0x7f144406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.076+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f145c08a6f0 con 0x7f14640b38e0 2023-10-16T12:33:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.077+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f144406ae70 0x7f144406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.078+0000 7f1461d9b640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f144406ae70 0x7f144406d330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1458036000 tx=0x7f145803d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.086+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f145c055700 con 0x7f14640b38e0 2023-10-16T12:33:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.090+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6ee003d040 con 0x7f6ee406a4b0 2023-10-16T12:33:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.089+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f00cc004940 con 0x7f00dc15d030 2023-10-16T12:33:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.091+0000 7f00d97fa640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f00bc06ae70 0x7f00bc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.091+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00cc088de0 con 0x7f00dc15d030 2023-10-16T12:33:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.091+0000 7f6edaffd640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ec806af40 0x7f6ec806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.091+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ee00ac360 con 0x7f6ee406a4b0 2023-10-16T12:33:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.092+0000 7f00d3fff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f00bc06ae70 0x7f00bc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.092+0000 7f6ee9b24640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ec806af40 0x7f6ec806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:33.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.093+0000 7f00d3fff640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f00bc06ae70 0x7f00bc06d330 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0cd2b0 tx=0x7f00c4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.094+0000 7f6ee9b24640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ec806af40 0x7f6ec806d400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400a480 tx=0x7f6ed400a410 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:33.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.101+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00cc053e10 con 0x7f00dc15d030 2023-10-16T12:33:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.114+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ee00775c0 con 0x7f6ee406a4b0 2023-10-16T12:33:33.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.426+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdb8c052bf0 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.430 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2023-10-16T12:33:33.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.427+0000 7fdb90ff9640 1 -- 172.21.15.60:0/677238805 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb88079970 con 0x7fdb8c00c0e0 2023-10-16T12:33:33.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.430+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdb7806ae70 msgr2=0x7fdb7806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.431+0000 7fdb9a397640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdb7806ae70 0x7fdb7806d330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb94068d00 tx=0x7fdb9406c040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.431+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 msgr2=0x7fdb8c0d84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.431+0000 7fdb9a397640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 0x7fdb8c0d84a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdb880097d0 tx=0x7fdb880046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 shutdown_connections 2023-10-16T12:33:33.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdb7806ae70 0x7fdb7806d330 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c050120 0x7fdb8c184ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c033eb0 0x7fdb8c184980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 --2- 172.21.15.60:0/677238805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c00c0e0 0x7fdb8c0d84a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 >> 172.21.15.60:0/677238805 conn(0x7fdb8c054b00 msgr2=0x7fdb8c0e1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 shutdown_connections 2023-10-16T12:33:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.432+0000 7fdb9a397640 1 -- 172.21.15.60:0/677238805 wait complete. 2023-10-16T12:33:33.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.1 2023-10-16T12:33:33.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.477+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4b10005500 con 0x7f4b300382c0 2023-10-16T12:33:33.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.477+0000 7f4b23fff640 1 -- 172.21.15.60:0/510899378 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4b2c049090 con 0x7f4b300382c0 2023-10-16T12:33:33.480 INFO:teuthology.orchestra.run.smithi060.stdout:60129542148 2023-10-16T12:33:33.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4b1c06af40 msgr2=0x7f4b1c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4b1c06af40 0x7f4b1c06d400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4b30058cc0 tx=0x7f4b24009040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 msgr2=0x7f4b3016d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 0x7f4b3016d560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c000c00 tx=0x7f4b2c004730 comp rx=0 tx=0).stop 2023-10-16T12:33:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 shutdown_connections 2023-10-16T12:33:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4b1c06af40 0x7f4b1c06d400 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4b300e64b0 0x7f4b30058160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b30053cf0 0x7f4b300e4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 --2- 172.21.15.60:0/510899378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b300382c0 0x7f4b3016d560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 >> 172.21.15.60:0/510899378 conn(0x7f4b30055a30 msgr2=0x7f4b30051940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.481+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb5580057d0 con 0x7fb568054300 2023-10-16T12:33:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.482+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 shutdown_connections 2023-10-16T12:33:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.482+0000 7f4b21ffb640 1 -- 172.21.15.60:0/510899378 wait complete. 2023-10-16T12:33:33.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.482+0000 7fb566ffd640 1 -- 172.21.15.60:0/403753440 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb5700b3070 con 0x7fb568054300 2023-10-16T12:33:33.487 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2023-10-16T12:33:33.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb55406ae70 msgr2=0x7fb55406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb55406ae70 0x7fb55406d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb56c000f60 tx=0x7fb56c009040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 msgr2=0x7fb568186aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 0x7fb568186aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb57004f940 tx=0x7fb570067b80 comp rx=0 tx=0).stop 2023-10-16T12:33:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 shutdown_connections 2023-10-16T12:33:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb55406ae70 0x7fb55406d330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680e31c0 0x7fb568185970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb568054300 0x7fb568186aa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 --2- 172.21.15.60:0/403753440 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680381b0 0x7fb56800ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.485+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 >> 172.21.15.60:0/403753440 conn(0x7fb5680590f0 msgr2=0x7fb56803da90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.486+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 shutdown_connections 2023-10-16T12:33:33.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.486+0000 7fb578373640 1 -- 172.21.15.60:0/403753440 wait complete. 2023-10-16T12:33:33.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.498+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6ee406ca00 con 0x7f6ee406a4b0 2023-10-16T12:33:33.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.499+0000 7f6edaffd640 1 -- 172.21.15.60:0/2593532649 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ee007abe0 con 0x7f6ee406a4b0 2023-10-16T12:33:33.502 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ec806af40 msgr2=0x7f6ec806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ec806af40 0x7f6ec806d400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400a480 tx=0x7f6ed400a410 comp rx=0 tx=0).stop 2023-10-16T12:33:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 msgr2=0x7f6ee41f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 0x7f6ee41f6310 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000aa60 tx=0x7f6ee00046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 shutdown_connections 2023-10-16T12:33:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6ec806af40 0x7f6ec806d400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ee40ade50 0x7f6ee40ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ee4159110 0x7f6ee40af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 --2- 172.21.15.60:0/2593532649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ee406a4b0 0x7f6ee41f6310 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 >> 172.21.15.60:0/2593532649 conn(0x7f6ee40ce9b0 msgr2=0x7f6ee40ce080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 shutdown_connections 2023-10-16T12:33:33.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.502+0000 7f6eec589640 1 -- 172.21.15.60:0/2593532649 wait complete. 2023-10-16T12:33:33.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.0 2023-10-16T12:33:33.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.510+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fcc6004e9b0 con 0x7fcc6006a910 2023-10-16T12:33:33.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.511+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff6f0005500 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.512+0000 7fcc527fc640 1 -- 172.21.15.60:0/3660716034 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc5c015090 con 0x7fcc6006a910 2023-10-16T12:33:33.515 INFO:teuthology.orchestra.run.smithi060.stdout:60129542148 2023-10-16T12:33:33.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.512+0000 7ff702ffd640 1 -- 172.21.15.60:0/3606865425 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff6f407a000 con 0x7ff70c0ccfa0 2023-10-16T12:33:33.516 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:33.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.512+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f146404e160 con 0x7f14640b38e0 2023-10-16T12:33:33.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.2 2023-10-16T12:33:33.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.513+0000 7f1452ffd640 1 -- 172.21.15.60:0/1477962585 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f145c058d20 con 0x7f14640b38e0 2023-10-16T12:33:33.518 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:33.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff6ec06ae90 msgr2=0x7ff6ec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7ff710c15640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff6ec06ae90 0x7ff6ec06d350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff704001ef0 tx=0x7ff704009040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70c0ccfa0 msgr2=0x7ff70c0c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7ff710c15640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70c0ccfa0 0x7ff70c0c7c70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4033000 tx=0x7ff6f4004730 comp rx=0 tx=0).stop 2023-10-16T12:33:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcc4406ae70 msgr2=0x7fcc4406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcc4406ae70 0x7fcc4406d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcc540011e0 tx=0x7fcc54009040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 msgr2=0x7fcc6015a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.515+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 0x7fcc6015a220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00b410 tx=0x7fcc5c00b8d0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 shutdown_connections 2023-10-16T12:33:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcc4406ae70 0x7fcc4406d330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc600cf1c0 0x7fcc600c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6006f440 0x7fcc600c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 --2- 172.21.15.60:0/3660716034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6006a910 0x7fcc6015a220 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 >> 172.21.15.60:0/3660716034 conn(0x7fcc600ce6a0 msgr2=0x7fcc600cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 shutdown_connections 2023-10-16T12:33:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff6ec06ae90 0x7ff6ec06d350 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70c0ccfa0 0x7ff70c0c7c70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff70c0adc30 0x7ff70c1601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 --2- 172.21.15.60:0/3606865425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff70c06f250 0x7ff70c0b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 >> 172.21.15.60:0/3606865425 conn(0x7ff70c0ce6e0 msgr2=0x7ff70c0cdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 shutdown_connections 2023-10-16T12:33:33.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7ff710c15640 1 -- 172.21.15.60:0/3606865425 wait complete. 2023-10-16T12:33:33.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f144406ae70 msgr2=0x7f144406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f144406ae70 0x7f144406d330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1458036000 tx=0x7f145803d040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 msgr2=0x7f14641600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 0x7f14641600c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f145c00efc0 tx=0x7f145c00c490 comp rx=0 tx=0).stop 2023-10-16T12:33:33.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 shutdown_connections 2023-10-16T12:33:33.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 shutdown_connections 2023-10-16T12:33:33.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7fcc66fc4640 1 -- 172.21.15.60:0/3660716034 wait complete. 2023-10-16T12:33:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f144406ae70 0x7f144406d330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14640cfaf0 0x7f1464070a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14640cf710 0x7f14640c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 --2- 172.21.15.60:0/1477962585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14640b38e0 0x7f14641600c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.516+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 >> 172.21.15.60:0/1477962585 conn(0x7f14640d2cc0 msgr2=0x7f14640b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.517+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 shutdown_connections 2023-10-16T12:33:33.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.517+0000 7f1468d90640 1 -- 172.21.15.60:0/1477962585 wait complete. 2023-10-16T12:33:33.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.519+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f00dc0cec20 con 0x7f00dc15d030 2023-10-16T12:33:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.520+0000 7f00d97fa640 1 -- 172.21.15.60:0/165744376 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f00cc018070 con 0x7f00dc15d030 2023-10-16T12:33:33.529 INFO:teuthology.orchestra.run.smithi060.stdout:60129542147 2023-10-16T12:33:33.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.5 2023-10-16T12:33:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f00bc06ae70 msgr2=0x7f00bc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f00bc06ae70 0x7f00bc06d330 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0cd2b0 tx=0x7f00c4009040 comp rx=0 tx=0).stop 2023-10-16T12:33:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 msgr2=0x7f00dc0c8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 0x7f00dc0c8890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00c780 tx=0x7f00cc00cc40 comp rx=0 tx=0).stop 2023-10-16T12:33:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 shutdown_connections 2023-10-16T12:33:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f00bc06ae70 0x7f00bc06d330 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00dc15ae80 0x7f00dc1ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc15d410 0x7f00dc15a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 --2- 172.21.15.60:0/165744376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00dc15d030 0x7f00dc0c8890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:33.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 >> 172.21.15.60:0/165744376 conn(0x7f00dc0cf720 msgr2=0x7f00dc06c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:33.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 shutdown_connections 2023-10-16T12:33:33.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:33.523+0000 7f00e2683640 1 -- 172.21.15.60:0/165744376 wait complete. 2023-10-16T12:33:33.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.4 2023-10-16T12:33:33.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.7 2023-10-16T12:33:33.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.3 2023-10-16T12:33:33.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.6 2023-10-16T12:33:34.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T12:33:34.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T12:33:34.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T12:33:34.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T12:33:34.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T12:33:34.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T12:33:34.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T12:33:34.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T12:33:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.755+0000 7f4540de9640 1 Processor -- start 2023-10-16T12:33:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.756+0000 7f4540de9640 1 -- start start 2023-10-16T12:33:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.757+0000 7f4540de9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45340df3f0 0x7f4534007470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.758+0000 7f4540de9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4534050270 0x7f4534054150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.758+0000 7f4540de9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45340063d0 con 0x7f4534005ed0 2023-10-16T12:33:34.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.758+0000 7f4540de9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45340df7d0 con 0x7f4534050270 2023-10-16T12:33:34.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.758+0000 7f4540de9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45340066b0 con 0x7f45340df3f0 2023-10-16T12:33:34.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.758+0000 7f453affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4534005ed0 0x7f453404fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54268/0 (socket says 172.21.15.60:54268) 2023-10-16T12:33:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.758+0000 7f453affd640 1 -- 172.21.15.60:0/3413526661 learned_addr learned my addr 172.21.15.60:0/3413526661 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.759+0000 7f453bfff640 1 --2- 172.21.15.60:0/3413526661 >> v2:172.21.15.144:3300/0 conn(0x7f4534050270 0x7f4534054150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.759+0000 7f453b7fe640 1 --2- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45340df3f0 0x7f4534007470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.760+0000 7f453bfff640 1 -- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45340df3f0 msgr2=0x7f4534007470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.760+0000 7f453bfff640 1 --2- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45340df3f0 0x7f4534007470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.760+0000 7f453bfff640 1 -- 172.21.15.60:0/3413526661 >> v1:172.21.15.60:6789/0 conn(0x7f4534005ed0 legacy=0x7f453404fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.760+0000 7f453bfff640 1 -- 172.21.15.60:0/3413526661 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534006990 con 0x7f4534050270 2023-10-16T12:33:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.762+0000 7f453bfff640 1 --2- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534054150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f453c04f940 tx=0x7f453c0a80f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff929acee59c8520 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.762+0000 7f453a7fc640 1 -- 172.21.15.60:0/3413526661 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0b2400 con 0x7f4534050270 2023-10-16T12:33:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.763+0000 7f453a7fc640 1 -- 172.21.15.60:0/3413526661 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c0ac070 con 0x7f4534050270 2023-10-16T12:33:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.763+0000 7f453a7fc640 1 -- 172.21.15.60:0/3413526661 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0b2860 con 0x7f4534050270 2023-10-16T12:33:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.764+0000 7f4540de9640 1 -- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 msgr2=0x7f4534054150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.764+0000 7f4540de9640 1 --2- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534054150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f453c04f940 tx=0x7f453c0a80f0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.764+0000 7f4540de9640 1 -- 172.21.15.60:0/3413526661 shutdown_connections 2023-10-16T12:33:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.764+0000 7f4540de9640 1 --2- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534054150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.764+0000 7f4540de9640 1 --2- 172.21.15.60:0/3413526661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45340df3f0 0x7f4534007470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.765+0000 7f4540de9640 1 -- 172.21.15.60:0/3413526661 >> 172.21.15.60:0/3413526661 conn(0x7f4534054b00 msgr2=0x7f4534054f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.765+0000 7f4540de9640 1 -- 172.21.15.60:0/3413526661 shutdown_connections 2023-10-16T12:33:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.765+0000 7f4540de9640 1 -- 172.21.15.60:0/3413526661 wait complete. 2023-10-16T12:33:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.769+0000 7f4540de9640 1 Processor -- start 2023-10-16T12:33:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.769+0000 7f4540de9640 1 -- start start 2023-10-16T12:33:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.771+0000 7f4540de9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4534005ed0 0x7f453404df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.771+0000 7f4540de9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534174460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.772+0000 7f4540de9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 0x7f453404c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.772+0000 7f4540de9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453402b670 con 0x7f45340df3f0 2023-10-16T12:33:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.772+0000 7f4540de9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453404ce70 con 0x7f4534050270 2023-10-16T12:33:34.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.772+0000 7f4540de9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4534048740 con 0x7f4534005ed0 2023-10-16T12:33:34.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 0x7f453404c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 0x7f453404c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58022/0 (socket says 172.21.15.60:58022) 2023-10-16T12:33:34.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 -- 172.21.15.60:0/369713216 learned_addr learned my addr 172.21.15.60:0/369713216 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453b7fe640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4534005ed0 0x7f453404df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453affd640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534174460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 -- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4534005ed0 msgr2=0x7f453404df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4534005ed0 0x7f453404df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 -- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 msgr2=0x7f4534174460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534174460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.773+0000 7f453bfff640 1 -- 172.21.15.60:0/369713216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c04e6d0 con 0x7f45340df3f0 2023-10-16T12:33:34.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.774+0000 7f453bfff640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 0x7f453404c5e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f453c0687d0 tx=0x7f453c067b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.774+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0bd940 con 0x7f45340df3f0 2023-10-16T12:33:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.775+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c0bdaa0 con 0x7f45340df3f0 2023-10-16T12:33:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.775+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0bd940 con 0x7f45340df3f0 2023-10-16T12:33:34.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.777+0000 7f4540de9640 1 -- 172.21.15.60:0/369713216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4534048a20 con 0x7f45340df3f0 2023-10-16T12:33:34.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.778+0000 7f4540de9640 1 -- 172.21.15.60:0/369713216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4534048fe0 con 0x7f45340df3f0 2023-10-16T12:33:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.781+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f453c0bdc50 con 0x7f45340df3f0 2023-10-16T12:33:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.781+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4534050c60 con 0x7f45340df3f0 2023-10-16T12:33:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.783+0000 7f4527fff640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f452006aed0 0x7f452006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.783+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f453c1278e0 con 0x7f45340df3f0 2023-10-16T12:33:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.784+0000 7f453b7fe640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f452006aed0 0x7f452006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.786+0000 7f453b7fe640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f452006aed0 0x7f452006d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4530006fd0 tx=0x7f4530008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.793+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f453c0f6420 con 0x7f45340df3f0 2023-10-16T12:33:34.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.801+0000 7f0dc029b640 1 Processor -- start 2023-10-16T12:33:34.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.801+0000 7f0dc029b640 1 -- start start 2023-10-16T12:33:34.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.803+0000 7f0dc029b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0db80cf710 0x7f0db80b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.803+0000 7f0dc029b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0db80b6cd0 0x7f0db80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.804+0000 7f0dc029b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db80d3980 con 0x7f0db80cfaf0 2023-10-16T12:33:34.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.804+0000 7f0dc029b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80d3c60 con 0x7f0db80cf710 2023-10-16T12:33:34.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.804+0000 7f0dc029b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80ce630 con 0x7f0db80b6cd0 2023-10-16T12:33:34.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.804+0000 7f0dbe037640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0db80cf710 0x7f0db80b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.804+0000 7f0dbe037640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0db80cf710 0x7f0db80b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58034/0 (socket says 172.21.15.60:58034) 2023-10-16T12:33:34.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.805+0000 7f0dbe037640 1 -- 172.21.15.60:0/1482863495 learned_addr learned my addr 172.21.15.60:0/1482863495 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.805+0000 7f0dbe838640 1 --2- 172.21.15.60:0/1482863495 >> v2:172.21.15.144:3300/0 conn(0x7f0db80b6cd0 0x7f0db80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.805+0000 7f0dbe037640 1 -- 172.21.15.60:0/1482863495 >> v2:172.21.15.144:3300/0 conn(0x7f0db80b6cd0 msgr2=0x7f0db80b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.805+0000 7f0dbe037640 1 --2- 172.21.15.60:0/1482863495 >> v2:172.21.15.144:3300/0 conn(0x7f0db80b6cd0 0x7f0db80b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.805+0000 7f0dbe037640 1 -- 172.21.15.60:0/1482863495 >> v1:172.21.15.60:6789/0 conn(0x7f0db80cfaf0 legacy=0x7f0db80d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.805+0000 7f0dbe037640 1 -- 172.21.15.60:0/1482863495 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80ce910 con 0x7f0db80cf710 2023-10-16T12:33:34.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.807+0000 7f0dbe037640 1 --2- 172.21.15.60:0/1482863495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db80b2d10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0db80b5890 tx=0x7f0db002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61b2cdf6bd51546 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.808+0000 7f0dbd035640 1 -- 172.21.15.60:0/1482863495 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db002ca80 con 0x7f0db80cf710 2023-10-16T12:33:34.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.808+0000 7f0dbd035640 1 -- 172.21.15.60:0/1482863495 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db002cbe0 con 0x7f0db80cf710 2023-10-16T12:33:34.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.808+0000 7f0dbd035640 1 -- 172.21.15.60:0/1482863495 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db002ca80 con 0x7f0db80cf710 2023-10-16T12:33:34.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.809+0000 7f0dc029b640 1 -- 172.21.15.60:0/1482863495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 msgr2=0x7f0db80b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.809+0000 7f0dc029b640 1 --2- 172.21.15.60:0/1482863495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db80b2d10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0db80b5890 tx=0x7f0db002bee0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.810+0000 7f0dc029b640 1 -- 172.21.15.60:0/1482863495 shutdown_connections 2023-10-16T12:33:34.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.810+0000 7f0dc029b640 1 --2- 172.21.15.60:0/1482863495 >> v2:172.21.15.144:3300/0 conn(0x7f0db80b6cd0 0x7f0db80b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.810+0000 7f0dc029b640 1 --2- 172.21.15.60:0/1482863495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db80b2d10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.810+0000 7f0dc029b640 1 -- 172.21.15.60:0/1482863495 >> 172.21.15.60:0/1482863495 conn(0x7f0db80d2cc0 msgr2=0x7f0db80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.810+0000 7f0dc029b640 1 -- 172.21.15.60:0/1482863495 shutdown_connections 2023-10-16T12:33:34.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.810+0000 7f0dc029b640 1 -- 172.21.15.60:0/1482863495 wait complete. 2023-10-16T12:33:34.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.813+0000 7f0dc029b640 1 Processor -- start 2023-10-16T12:33:34.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.814+0000 7f0dc029b640 1 -- start start 2023-10-16T12:33:34.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.815+0000 7f0dc029b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0db80b6cd0 0x7f0db81e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.815+0000 7f0dc029b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db81fbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.815+0000 7f0dc029b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0db80cfaf0 0x7f0db81fc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.816+0000 7f0dc029b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db81fedc0 con 0x7f0db80cf710 2023-10-16T12:33:34.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.816+0000 7f0dbd836640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db81fbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.816+0000 7f0dbe037640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0db80b6cd0 0x7f0db81e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.816+0000 7f0dbd836640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db81fbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58048/0 (socket says 172.21.15.60:58048) 2023-10-16T12:33:34.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.816+0000 7f0dbd836640 1 -- 172.21.15.60:0/2204752359 learned_addr learned my addr 172.21.15.60:0/2204752359 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.816+0000 7f0dbe838640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0db80cfaf0 0x7f0db81fc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.817+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db81ff0a0 con 0x7f0db80b6cd0 2023-10-16T12:33:34.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.817+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db81ff380 con 0x7f0db80cfaf0 2023-10-16T12:33:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.818+0000 7f0dbd836640 1 -- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0db80cfaf0 msgr2=0x7f0db81fc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.818+0000 7f0dbd836640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0db80cfaf0 0x7f0db81fc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.818+0000 7f0dbd836640 1 -- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0db80b6cd0 msgr2=0x7f0db81e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.818+0000 7f0dbd836640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0db80b6cd0 0x7f0db81e6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.819+0000 7f0dbd836640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0009ad0 con 0x7f0db80cf710 2023-10-16T12:33:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.819+0000 7f0dbd836640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db81fbda0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c750 tx=0x7f0da800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.820+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da80050d0 con 0x7f0db80cf710 2023-10-16T12:33:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.820+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8005230 con 0x7f0db80cf710 2023-10-16T12:33:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.820+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800a640 con 0x7f0db80cf710 2023-10-16T12:33:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.821+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db81ff660 con 0x7f0db80cf710 2023-10-16T12:33:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.821+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db81e7150 con 0x7f0db80cf710 2023-10-16T12:33:34.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.824+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db81ff7f0 con 0x7f0db80cf710 2023-10-16T12:33:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.834+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0da800a7a0 con 0x7f0db80cf710 2023-10-16T12:33:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.836+0000 7f0daeffd640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d9c06aec0 0x7f0d9c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.837+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0da8054fc0 con 0x7f0db80cf710 2023-10-16T12:33:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.837+0000 7f0dbe037640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d9c06aec0 0x7f0d9c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.838+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0da8054bf0 con 0x7f0db80cf710 2023-10-16T12:33:34.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.839+0000 7f0dbe037640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d9c06aec0 0x7f0d9c06d380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0db0035000 tx=0x7f0db0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.866+0000 7f6b33402640 1 Processor -- start 2023-10-16T12:33:34.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.866+0000 7f6b33402640 1 -- start start 2023-10-16T12:33:34.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.868+0000 7f6b33402640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b2c0ad920 0x7f6b2c0add00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.868+0000 7f6b33402640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b2c06a4b0 0x7f6b2c0cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.868+0000 7f6b33402640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c06ba40 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.868+0000 7f6b33402640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2c06c000 con 0x7f6b2c06a4b0 2023-10-16T12:33:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.868+0000 7f6b33402640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2c06bd20 con 0x7f6b2c0ad920 2023-10-16T12:33:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.869+0000 7f6b3119e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b2c0ad920 0x7f6b2c0add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.869+0000 7f6b3119e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b2c0ad920 0x7f6b2c0add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49212/0 (socket says 172.21.15.60:49212) 2023-10-16T12:33:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.869+0000 7f6b3119e640 1 -- 172.21.15.60:0/3556547329 learned_addr learned my addr 172.21.15.60:0/3556547329 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.869+0000 7f6b3199f640 1 --2- 172.21.15.60:0/3556547329 >> v2:172.21.15.60:3300/0 conn(0x7f6b2c06a4b0 0x7f6b2c0cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.870+0000 7f6b3199f640 1 -- 172.21.15.60:0/3556547329 >> v2:172.21.15.144:3300/0 conn(0x7f6b2c0ad920 msgr2=0x7f6b2c0add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.870+0000 7f6b3199f640 1 --2- 172.21.15.60:0/3556547329 >> v2:172.21.15.144:3300/0 conn(0x7f6b2c0ad920 0x7f6b2c0add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.870+0000 7f6b3199f640 1 -- 172.21.15.60:0/3556547329 >> v1:172.21.15.144:6789/0 conn(0x7f6b2c0ce2b0 legacy=0x7f6b2c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.870+0000 7f6b3199f640 1 -- 172.21.15.60:0/3556547329 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0c91a0 con 0x7f6b2c06a4b0 2023-10-16T12:33:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.872+0000 7f6b3199f640 1 --2- 172.21.15.60:0/3556547329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c06a4b0 0x7f6b2c0cc580 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0cf4f0 tx=0x7f6b2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6ca6b84eba555f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.873+0000 7f6b23fff640 1 -- 172.21.15.60:0/3556547329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28031830 con 0x7f6b2c06a4b0 2023-10-16T12:33:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.873+0000 7f6b23fff640 1 -- 172.21.15.60:0/3556547329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b28031990 con 0x7f6b2c06a4b0 2023-10-16T12:33:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.873+0000 7f6b23fff640 1 -- 172.21.15.60:0/3556547329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28031d20 con 0x7f6b2c06a4b0 2023-10-16T12:33:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.874+0000 7f6b33402640 1 -- 172.21.15.60:0/3556547329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c06a4b0 msgr2=0x7f6b2c0cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.874+0000 7f6b33402640 1 --2- 172.21.15.60:0/3556547329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c06a4b0 0x7f6b2c0cc580 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0cf4f0 tx=0x7f6b2802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f6b33402640 1 -- 172.21.15.60:0/3556547329 shutdown_connections 2023-10-16T12:33:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f6b33402640 1 --2- 172.21.15.60:0/3556547329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c06a4b0 0x7f6b2c0cc580 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f6b33402640 1 --2- 172.21.15.60:0/3556547329 >> v2:172.21.15.144:3300/0 conn(0x7f6b2c0ad920 0x7f6b2c0add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f6b33402640 1 -- 172.21.15.60:0/3556547329 >> 172.21.15.60:0/3556547329 conn(0x7f6b2c0ce9b0 msgr2=0x7f6b2c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f4d31815640 1 Processor -- start 2023-10-16T12:33:34.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f6b33402640 1 -- 172.21.15.60:0/3556547329 shutdown_connections 2023-10-16T12:33:34.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f4d31815640 1 -- start start 2023-10-16T12:33:34.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.875+0000 7f6b33402640 1 -- 172.21.15.60:0/3556547329 wait complete. 2023-10-16T12:33:34.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.877+0000 7f4d31815640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c069da0 0x7f4d2c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.877+0000 7f4d31815640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4d2c160b90 0x7f4d2c161020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.877+0000 7f4d31815640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c15d8e0 con 0x7f4d2c0cac60 2023-10-16T12:33:34.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.877+0000 7f4d31815640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2c15d650 con 0x7f4d2c160b90 2023-10-16T12:33:34.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.877+0000 7f4d31815640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c15dbc0 con 0x7f4d2c069da0 2023-10-16T12:33:34.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.878+0000 7f6b33402640 1 Processor -- start 2023-10-16T12:33:34.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.878+0000 7f4d2a7fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f4d2c0cac60 0x7f4d2c06a180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60320/0 (socket says 172.21.15.60:60320) 2023-10-16T12:33:34.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.878+0000 7f4d2a7fc640 1 -- 172.21.15.60:0/2711366564 learned_addr learned my addr 172.21.15.60:0/2711366564 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:34.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.878+0000 7f4d2affd640 1 --2- 172.21.15.60:0/2711366564 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c069da0 0x7f4d2c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.878+0000 7f6b33402640 1 -- start start 2023-10-16T12:33:34.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.878+0000 7f4d2b7fe640 1 --2- 172.21.15.60:0/2711366564 >> v2:172.21.15.60:3300/0 conn(0x7f4d2c160b90 0x7f4d2c161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d29ffb640 1 -- 172.21.15.60:0/2711366564 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051873520 0 0) 0x7f4d2c15d8e0 con 0x7f4d2c0cac60 2023-10-16T12:33:34.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d29ffb640 1 -- 172.21.15.60:0/2711366564 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14003520 con 0x7f4d2c0cac60 2023-10-16T12:33:34.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d2b7fe640 1 -- 172.21.15.60:0/2711366564 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c069da0 msgr2=0x7f4d2c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d2b7fe640 1 --2- 172.21.15.60:0/2711366564 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c069da0 0x7f4d2c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.880+0000 7f6b33402640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 0x7f6b2c14b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d2b7fe640 1 -- 172.21.15.60:0/2711366564 >> v1:172.21.15.144:6789/0 conn(0x7f4d2c0cac60 legacy=0x7f4d2c06a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d29ffb640 1 -- 172.21.15.60:0/2711366564 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829813699 0 0) 0x7f4d14003520 con 0x7f4d2c0cac60 2023-10-16T12:33:34.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.880+0000 7f6b33402640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b2c0ad920 0x7f6b2c14ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.879+0000 7f4d2b7fe640 1 -- 172.21.15.60:0/2711366564 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c15dea0 con 0x7f4d2c160b90 2023-10-16T12:33:34.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.880+0000 7f0330ae0640 1 Processor -- start 2023-10-16T12:33:34.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.880+0000 7f0330ae0640 1 -- start start 2023-10-16T12:33:34.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b33402640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 0x7f6b2c06f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f4d2b7fe640 1 --2- 172.21.15.60:0/2711366564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c160b90 0x7f4d2c161020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0097d0 tx=0x7f4d1c02b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e6c2b24858077b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f4d29ffb640 1 -- 172.21.15.60:0/2711366564 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240019f0 con 0x7f4d2c160b90 2023-10-16T12:33:34.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f4d29ffb640 1 -- 172.21.15.60:0/2711366564 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c02e050 con 0x7f4d2c160b90 2023-10-16T12:33:34.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f4d29ffb640 1 -- 172.21.15.60:0/2711366564 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240019f0 con 0x7f4d2c160b90 2023-10-16T12:33:34.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f0330ae0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f032c0ade50 0x7f032c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f0330ae0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c06a2e0 0x7f032c06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f0330ae0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c1594f0 con 0x7f032c159110 2023-10-16T12:33:34.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f0330ae0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032c06a770 con 0x7f032c0ade50 2023-10-16T12:33:34.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f0330ae0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c15d400 con 0x7f032c06a2e0 2023-10-16T12:33:34.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 -- 172.21.15.60:0/2711366564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c160b90 msgr2=0x7f4d2c161020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 --2- 172.21.15.60:0/2711366564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c160b90 0x7f4d2c161020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0097d0 tx=0x7f4d1c02b800 comp rx=0 tx=0).stop 2023-10-16T12:33:34.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 -- 172.21.15.60:0/2711366564 shutdown_connections 2023-10-16T12:33:34.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f032a59c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f032c159110 0x7f032c15cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60326/0 (socket says 172.21.15.60:60326) 2023-10-16T12:33:34.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f032a59c640 1 -- 172.21.15.60:0/1068700301 learned_addr learned my addr 172.21.15.60:0/1068700301 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:34.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 --2- 172.21.15.60:0/2711366564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c160b90 0x7f4d2c161020 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 --2- 172.21.15.60:0/2711366564 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c069da0 0x7f4d2c06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 -- 172.21.15.60:0/2711366564 >> 172.21.15.60:0/2711366564 conn(0x7f4d2c0ce9b0 msgr2=0x7f4d2c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f4d31815640 1 -- 172.21.15.60:0/2711366564 shutdown_connections 2023-10-16T12:33:34.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f0329d9b640 1 --2- 172.21.15.60:0/1068700301 >> v2:172.21.15.60:3300/0 conn(0x7f032c0ade50 0x7f032c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f4d31815640 1 -- 172.21.15.60:0/2711366564 wait complete. 2023-10-16T12:33:34.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f032ad9d640 1 --2- 172.21.15.60:0/1068700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c06a2e0 0x7f032c06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f032959a640 1 -- 172.21.15.60:0/1068700301 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006782159 0 0) 0x7f032c1594f0 con 0x7f032c159110 2023-10-16T12:33:34.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f032959a640 1 -- 172.21.15.60:0/1068700301 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314003520 con 0x7f032c159110 2023-10-16T12:33:34.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.885+0000 7f0329d9b640 1 -- 172.21.15.60:0/1068700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c06a2e0 msgr2=0x7f032c06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.885+0000 7f0329d9b640 1 --2- 172.21.15.60:0/1068700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c06a2e0 0x7f032c06fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.885+0000 7f0329d9b640 1 -- 172.21.15.60:0/1068700301 >> v1:172.21.15.144:6789/0 conn(0x7f032c159110 legacy=0x7f032c15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.885+0000 7f0329d9b640 1 -- 172.21.15.60:0/1068700301 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c15d6e0 con 0x7f032c0ade50 2023-10-16T12:33:34.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.886+0000 7f4d31815640 1 Processor -- start 2023-10-16T12:33:34.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.887+0000 7f4d31815640 1 -- start start 2023-10-16T12:33:34.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.887+0000 7f0329d9b640 1 --2- 172.21.15.60:0/1068700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c0ade50 0x7f032c069da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f031c00a940 tx=0x7f031c030090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7759295b201eabc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b3199f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 0x7f6b2c06f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b3199f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 0x7f6b2c06f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58084/0 (socket says 172.21.15.60:58084) 2023-10-16T12:33:34.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b3119e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 0x7f6b2c14b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b3119e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 0x7f6b2c14b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53400/0 (socket says 172.21.15.60:53400) 2023-10-16T12:33:34.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b3199f640 1 -- 172.21.15.60:0/2928450395 learned_addr learned my addr 172.21.15.60:0/2928450395 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b3099d640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b2c0ad920 0x7f6b2c14ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c147ae0 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c14b590 con 0x7f6b2c0ad920 2023-10-16T12:33:34.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.881+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0c7960 con 0x7f6b2c06a4b0 2023-10-16T12:33:34.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f6b3199f640 1 -- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 msgr2=0x7f6b2c14b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f6b3199f640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 0x7f6b2c14b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f6b3199f640 1 -- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b2c0ad920 msgr2=0x7f6b2c14ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f6b3199f640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b2c0ad920 0x7f6b2c14ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.882+0000 7f6b3199f640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b28009400 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.883+0000 7f6b3199f640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 0x7f6b2c06f860 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6b28034040 tx=0x7f6b280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b3119e640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 0x7f6b2c14b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b3099d640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b2c0ad920 0x7f6b2c14ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2803a830 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2803a990 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2803ad20 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c0c7c40 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.884+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c0c8200 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.887+0000 7f032959a640 1 -- 172.21.15.60:0/1068700301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031c030e40 con 0x7f032c0ade50 2023-10-16T12:33:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.887+0000 7f032959a640 1 -- 172.21.15.60:0/1068700301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031c038440 con 0x7f032c0ade50 2023-10-16T12:33:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.887+0000 7f032959a640 1 -- 172.21.15.60:0/1068700301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031c0387d0 con 0x7f032c0ade50 2023-10-16T12:33:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.889+0000 7f0330ae0640 1 -- 172.21.15.60:0/1068700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c0ade50 msgr2=0x7f032c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.889+0000 7f0330ae0640 1 --2- 172.21.15.60:0/1068700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c0ade50 0x7f032c069da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f031c00a940 tx=0x7f031c030090 comp rx=0 tx=0).stop 2023-10-16T12:33:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.889+0000 7f4d31815640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 0x7f4d2c0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.889+0000 7f0330ae0640 1 -- 172.21.15.60:0/1068700301 shutdown_connections 2023-10-16T12:33:34.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.889+0000 7f0330ae0640 1 --2- 172.21.15.60:0/1068700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c06a2e0 0x7f032c06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.889+0000 7f0330ae0640 1 --2- 172.21.15.60:0/1068700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c0ade50 0x7f032c069da0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d31815640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d2c0cac60 0x7f4d2c0ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f0330ae0640 1 -- 172.21.15.60:0/1068700301 >> 172.21.15.60:0/1068700301 conn(0x7f032c0ce9b0 msgr2=0x7f032c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d2affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 0x7f4d2c0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f95c2875640 1 Processor -- start 2023-10-16T12:33:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d31815640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c160b90 0x7f4d2c0ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d31815640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c1557f0 con 0x7f4d2c069da0 2023-10-16T12:33:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d31815640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0aeeb0 con 0x7f4d2c0cac60 2023-10-16T12:33:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d31815640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0c6f70 con 0x7f4d2c160b90 2023-10-16T12:33:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2c04e160 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f0330ae0640 1 -- 172.21.15.60:0/1068700301 shutdown_connections 2023-10-16T12:33:34.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f95c2875640 1 -- start start 2023-10-16T12:33:34.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d2affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 0x7f4d2c0b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58088/0 (socket says 172.21.15.60:58088) 2023-10-16T12:33:34.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f4d2affd640 1 -- 172.21.15.60:0/115031895 learned_addr learned my addr 172.21.15.60:0/115031895 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6b280443f0 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f7a3bfff640 1 Processor -- start 2023-10-16T12:33:34.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f4d2affd640 1 -- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c160b90 msgr2=0x7f4d2c0ca030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:33:34.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f4d2affd640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c160b90 0x7f4d2c0ca030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f4d2affd640 1 -- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d2c0cac60 msgr2=0x7f4d2c0ae3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:33:34.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f4d2affd640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d2c0cac60 0x7f4d2c0ae3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f4d2affd640 1 -- 172.21.15.60:0/115031895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c009400 con 0x7f4d2c069da0 2023-10-16T12:33:34.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.891+0000 7f7a3bfff640 1 -- start start 2023-10-16T12:33:34.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.890+0000 7f0330ae0640 1 -- 172.21.15.60:0/1068700301 wait complete. 2023-10-16T12:33:34.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f95c2875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f0330ae0640 1 Processor -- start 2023-10-16T12:33:34.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f6b21ffb640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6b1006aef0 0x7f6b1006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b280ab550 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f0330ae0640 1 -- start start 2023-10-16T12:33:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f95c2875640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc06cf50 con 0x7f95bc158f60 2023-10-16T12:33:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.893+0000 7f95c2875640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc15ccf0 con 0x7f95bc06a4b0 2023-10-16T12:33:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.894+0000 7f95c2875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc15d970 con 0x7f95bc0ce1c0 2023-10-16T12:33:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.894+0000 7f6b3119e640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6b1006aef0 0x7f6b1006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.894+0000 7f95bbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f95bc158f60 0x7f95bc069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54280/0 (socket says 172.21.15.60:54280) 2023-10-16T12:33:34.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.894+0000 7f95bbfff640 1 -- 172.21.15.60:0/2428126978 learned_addr learned my addr 172.21.15.60:0/2428126978 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:34.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.894+0000 7f7a3bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.894+0000 7f95c0e12640 1 --2- 172.21.15.60:0/2428126978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f7a327fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7a3c0b8480 0x7f7a3c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54284/0 (socket says 172.21.15.60:54284) 2023-10-16T12:33:34.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f7a327fc640 1 -- 172.21.15.60:0/3088779650 learned_addr learned my addr 172.21.15.60:0/3088779650 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:34.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f7a3affd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f6b3119e640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6b1006aef0 0x7f6b1006d3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009d70 tx=0x7f6b2400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f95baffd640 1 -- 172.21.15.60:0/2428126978 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299840299 0 0) 0x7f95bc15ccf0 con 0x7f95bc06a4b0 2023-10-16T12:33:34.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f95baffd640 1 -- 172.21.15.60:0/2428126978 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a8003520 con 0x7f95bc06a4b0 2023-10-16T12:33:34.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c15d150 con 0x7f7a3c0b8480 2023-10-16T12:33:34.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c0d1180 con 0x7f7a3c0cded0 2023-10-16T12:33:34.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0d1460 con 0x7f7a3c15cd70 2023-10-16T12:33:34.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f4d2affd640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 0x7f4d2c0b15a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000d5f0 tx=0x7f4d2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f95c0e12640 1 -- 172.21.15.60:0/2428126978 >> v1:172.21.15.144:6789/0 conn(0x7f95bc06a4b0 legacy=0x7f95bc0cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f95c0e12640 1 -- 172.21.15.60:0/2428126978 >> v1:172.21.15.60:6789/0 conn(0x7f95bc158f60 legacy=0x7f95bc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f95c0e12640 1 -- 172.21.15.60:0/2428126978 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc15cfd0 con 0x7f95bc0ce1c0 2023-10-16T12:33:34.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f0330ae0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 0x7f032c0c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.895+0000 7f0330ae0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c0ade50 0x7f032c0c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f0330ae0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f032c159110 0x7f032c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f0330ae0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0c8f90 con 0x7f032c06a2e0 2023-10-16T12:33:34.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f0330ae0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0b43d0 con 0x7f032c159110 2023-10-16T12:33:34.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f0330ae0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0c8310 con 0x7f032c0ade50 2023-10-16T12:33:34.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f032a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 0x7f032c0c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f7a3a7fc640 1 -- 172.21.15.60:0/3088779650 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146749476 0 0) 0x7f7a3c0d1180 con 0x7f7a3c0cded0 2023-10-16T12:33:34.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f7a3a7fc640 1 -- 172.21.15.60:0/3088779650 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a28003520 con 0x7f7a3c0cded0 2023-10-16T12:33:34.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f032a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 0x7f032c0c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58096/0 (socket says 172.21.15.60:58096) 2023-10-16T12:33:34.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.896+0000 7f032a59c640 1 -- 172.21.15.60:0/4281685424 learned_addr learned my addr 172.21.15.60:0/4281685424 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f0329d9b640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c0ade50 0x7f032c0c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f032a59c640 1 -- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c0ade50 msgr2=0x7f032c0c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f032a59c640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c0ade50 0x7f032c0c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f032a59c640 1 -- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f032c159110 msgr2=0x7f032c0b3e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:33:34.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f032a59c640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f032c159110 0x7f032c0b3e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f032a59c640 1 -- 172.21.15.60:0/4281685424 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c00a5a0 con 0x7f032c06a2e0 2023-10-16T12:33:34.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f7a3affd640 1 -- 172.21.15.60:0/3088779650 >> v1:172.21.15.144:6789/0 conn(0x7f7a3c0cded0 legacy=0x7f7a3c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f7a3affd640 1 -- 172.21.15.60:0/3088779650 >> v1:172.21.15.60:6789/0 conn(0x7f7a3c0b8480 legacy=0x7f7a3c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.897+0000 7f7a3affd640 1 -- 172.21.15.60:0/3088779650 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c0d1740 con 0x7f7a3c15cd70 2023-10-16T12:33:34.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.898+0000 7f032a59c640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 0x7f032c0c7d10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0320001460 tx=0x7f0320001920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.898+0000 7f95c0e12640 1 --2- 172.21.15.60:0/2428126978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0ade50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95a40090e0 tx=0x7f95a402cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80784958304f8ac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.898+0000 7f95baffd640 1 -- 172.21.15.60:0/2428126978 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a402d940 con 0x7f95bc0ce1c0 2023-10-16T12:33:34.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.898+0000 7f95baffd640 1 -- 172.21.15.60:0/2428126978 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a402daa0 con 0x7f95bc0ce1c0 2023-10-16T12:33:34.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.898+0000 7f95baffd640 1 -- 172.21.15.60:0/2428126978 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a402de30 con 0x7f95bc0ce1c0 2023-10-16T12:33:34.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.899+0000 7f7a3affd640 1 --2- 172.21.15.60:0/3088779650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c0d2220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009dc0 tx=0x7f7a2c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3528272269a2b14c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.899+0000 7f95c2875640 1 -- 172.21.15.60:0/2428126978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 msgr2=0x7f95bc0ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.899+0000 7f95c2875640 1 --2- 172.21.15.60:0/2428126978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0ade50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95a40090e0 tx=0x7f95a402cb90 comp rx=0 tx=0).stop 2023-10-16T12:33:34.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f7a3a7fc640 1 -- 172.21.15.60:0/3088779650 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c02ccc0 con 0x7f7a3c15cd70 2023-10-16T12:33:34.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f7a3a7fc640 1 -- 172.21.15.60:0/3088779650 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c02ce20 con 0x7f7a3c15cd70 2023-10-16T12:33:34.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f7a3a7fc640 1 -- 172.21.15.60:0/3088779650 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c034660 con 0x7f7a3c15cd70 2023-10-16T12:33:34.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7fe7bf690640 1 Processor -- start 2023-10-16T12:33:34.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f95c2875640 1 -- 172.21.15.60:0/2428126978 shutdown_connections 2023-10-16T12:33:34.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f95c2875640 1 --2- 172.21.15.60:0/2428126978 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0ade50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f95c2875640 1 -- 172.21.15.60:0/2428126978 >> 172.21.15.60:0/2428126978 conn(0x7f95bc0ce9b0 msgr2=0x7f95bc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f95c2875640 1 -- 172.21.15.60:0/2428126978 shutdown_connections 2023-10-16T12:33:34.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 msgr2=0x7f7a3c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/3088779650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c0d2220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009dc0 tx=0x7f7a2c02bf10 comp rx=0 tx=0).stop 2023-10-16T12:33:34.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d200125f0 con 0x7f4d2c069da0 2023-10-16T12:33:34.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d20012750 con 0x7f4d2c069da0 2023-10-16T12:33:34.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20012b00 con 0x7f4d2c069da0 2023-10-16T12:33:34.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 shutdown_connections 2023-10-16T12:33:34.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/3088779650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c0d2220 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 >> 172.21.15.60:0/3088779650 conn(0x7f7a3c0cf570 msgr2=0x7f7a3c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b28076800 con 0x7f6b2c0ce2b0 2023-10-16T12:33:34.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.900+0000 7f95c2875640 1 -- 172.21.15.60:0/2428126978 wait complete. 2023-10-16T12:33:34.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.901+0000 7fe7bf690640 1 -- start start 2023-10-16T12:33:34.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f4d31815640 1 -- 172.21.15.60:0/115031895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c0c7250 con 0x7f4d2c069da0 2023-10-16T12:33:34.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f4d31815640 1 -- 172.21.15.60:0/115031895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c0c7810 con 0x7f4d2c069da0 2023-10-16T12:33:34.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032000f640 con 0x7f032c06a2e0 2023-10-16T12:33:34.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 shutdown_connections 2023-10-16T12:33:34.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032000f7a0 con 0x7f032c06a2e0 2023-10-16T12:33:34.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032000fb30 con 0x7f032c06a2e0 2023-10-16T12:33:34.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.903+0000 7f95c2875640 1 Processor -- start 2023-10-16T12:33:34.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.903+0000 7fe7bf690640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b815af80 0x7fe7b80cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.903+0000 7fe7bf690640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7b806faf0 0x7fe7b815f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.903+0000 7fe7bf690640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b815eff0 con 0x7fe7b8065ab0 2023-10-16T12:33:34.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.903+0000 7fe7bf690640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b80d0930 con 0x7fe7b806faf0 2023-10-16T12:33:34.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.903+0000 7fe7bf690640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815ecc0 con 0x7fe7b815af80 2023-10-16T12:33:34.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.904+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c0c85f0 con 0x7f032c06a2e0 2023-10-16T12:33:34.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.902+0000 7f7a3bfff640 1 -- 172.21.15.60:0/3088779650 wait complete. 2023-10-16T12:33:34.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.904+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c0c8bb0 con 0x7f032c06a2e0 2023-10-16T12:33:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.904+0000 7fe7bcc2b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7b806faf0 0x7fe7b815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.904+0000 7fe7bcc2b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7b806faf0 0x7fe7b815f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58100/0 (socket says 172.21.15.60:58100) 2023-10-16T12:33:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.904+0000 7fe7bcc2b640 1 -- 172.21.15.60:0/3223493721 learned_addr learned my addr 172.21.15.60:0/3223493721 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.905+0000 7f7a3bfff640 1 Processor -- start 2023-10-16T12:33:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.905+0000 7fe7bcc2b640 1 -- 172.21.15.60:0/3223493721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b815af80 msgr2=0x7fe7b80cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.905+0000 7fe7bcc2b640 1 --2- 172.21.15.60:0/3223493721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b815af80 0x7fe7b80cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.905+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c04e160 con 0x7f4d2c069da0 2023-10-16T12:33:34.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.906+0000 7fe7bcc2b640 1 -- 172.21.15.60:0/3223493721 >> v1:172.21.15.144:6789/0 conn(0x7fe7b8065ab0 legacy=0x7fe7b80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:34.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.906+0000 7f7a3bfff640 1 -- start start 2023-10-16T12:33:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.906+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c0c8d40 con 0x7f032c06a2e0 2023-10-16T12:33:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.906+0000 7fe7bcc2b640 1 -- 172.21.15.60:0/3223493721 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b806cec0 con 0x7fe7b806faf0 2023-10-16T12:33:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f7a3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 0x7f7a3c1e6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f7a3bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a3c0cded0 0x7f7a3c1e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f7a3bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c1e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f7a3bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c1e5c80 con 0x7f7a3c0b8480 2023-10-16T12:33:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f7a3bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c15acb0 con 0x7f7a3c0cded0 2023-10-16T12:33:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f7a3bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c15a9d0 con 0x7f7a3c15cd70 2023-10-16T12:33:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4d20012c60 con 0x7f4d2c069da0 2023-10-16T12:33:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.904+0000 7f95c2875640 1 -- start start 2023-10-16T12:33:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f95c2875640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06a4b0 0x7f95bc0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f95c2875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f95c2875640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 0x7f95bc0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f95c2875640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0c9280 con 0x7f95bc158f60 2023-10-16T12:33:34.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f95c2875640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0c6920 con 0x7f95bc06a4b0 2023-10-16T12:33:34.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.907+0000 7f95c2875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0c8cc0 con 0x7f95bc0ce1c0 2023-10-16T12:33:34.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7fe7bcc2b640 1 --2- 172.21.15.60:0/3223493721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b806faf0 0x7fe7b815f480 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009870 tx=0x7fe7a802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d391832d1e335b83 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f7a3affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 0x7f7a3c1e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f95c0e12640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 0x7f95bc0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f7a3affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 0x7f7a3c1e6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58112/0 (socket says 172.21.15.60:58112) 2023-10-16T12:33:34.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f7a3affd640 1 -- 172.21.15.60:0/2826203210 learned_addr learned my addr 172.21.15.60:0/2826203210 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f7a3b7fe640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c1e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f4d13fff640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4d1806af40 0x7f4d1806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f032000d050 con 0x7f032c06a2e0 2023-10-16T12:33:34.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f95c0e12640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 0x7f95bc0c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58124/0 (socket says 172.21.15.60:58124) 2023-10-16T12:33:34.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f95c0e12640 1 -- 172.21.15.60:0/1116092504 learned_addr learned my addr 172.21.15.60:0/1116092504 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.908+0000 7f95bb7fe640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f95bbfff640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06a4b0 0x7f95bc0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f7a3affd640 1 -- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 msgr2=0x7f7a3c1e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d20089ac0 con 0x7f4d2c069da0 2023-10-16T12:33:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f95c0e12640 1 -- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 msgr2=0x7f95bc0c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f95c0e12640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0c63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f95c0e12640 1 -- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06a4b0 msgr2=0x7f95bc0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f95c0e12640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06a4b0 0x7f95bc0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.909+0000 7f95c0e12640 1 -- 172.21.15.60:0/1116092504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4009c90 con 0x7f95bc158f60 2023-10-16T12:33:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f4d2a7fc640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4d1806af40 0x7f4d1806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f95c0e12640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 0x7f95bc0c6d70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f95a40090e0 tx=0x7f95a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f7a327fc640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a3c0cded0 0x7f7a3c1e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f7a3affd640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c1e7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f7a3affd640 1 -- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a3c0cded0 msgr2=0x7f7a3c1e6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f7a3affd640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a3c0cded0 0x7f7a3c1e6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f7a3affd640 1 -- 172.21.15.60:0/2826203210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c009a70 con 0x7f7a3c0b8480 2023-10-16T12:33:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.910+0000 7f7a3b7fe640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c1e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:34.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.911+0000 7f7a327fc640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a3c0cded0 0x7f7a3c1e6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:34.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.911+0000 7f7a3affd640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 0x7f7a3c1e6110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c000f80 tx=0x7f7a2c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.911+0000 7f4d2a7fc640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4d1806af40 0x7f4d1806d400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4d24001700 tx=0x7f4d24001360 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.911+0000 7f031affd640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f030c06aec0 0x7f030c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.911+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03200899e0 con 0x7f032c06a2e0 2023-10-16T12:33:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.912+0000 7f0329d9b640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f030c06aec0 0x7f030c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.913+0000 7f0329d9b640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f030c06aec0 0x7f030c06d380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f031c03a000 tx=0x7f031c034000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.914+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4040670 con 0x7f95bc158f60 2023-10-16T12:33:34.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.914+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc0c8fa0 con 0x7f95bc158f60 2023-10-16T12:33:34.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.914+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc157930 con 0x7f95bc158f60 2023-10-16T12:33:34.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.914+0000 7fe7affff640 1 -- 172.21.15.60:0/3223493721 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8030860 con 0x7fe7b806faf0 2023-10-16T12:33:34.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.915+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d2008be60 con 0x7f4d2c069da0 2023-10-16T12:33:34.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.915+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c040650 con 0x7f7a3c0b8480 2023-10-16T12:33:34.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.915+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c1e5630 con 0x7f7a3c0b8480 2023-10-16T12:33:34.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.915+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c15a5f0 con 0x7f7a3c0b8480 2023-10-16T12:33:34.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.916+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc157b50 con 0x7f95bc158f60 2023-10-16T12:33:34.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.916+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c04e9b0 con 0x7f7a3c0b8480 2023-10-16T12:33:34.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.921+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03200585b0 con 0x7f032c06a2e0 2023-10-16T12:33:34.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.921+0000 7fe7affff640 1 -- 172.21.15.60:0/3223493721 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a80309c0 con 0x7fe7b806faf0 2023-10-16T12:33:34.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.921+0000 7fe7affff640 1 -- 172.21.15.60:0/3223493721 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8030d30 con 0x7fe7b806faf0 2023-10-16T12:33:34.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.921+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a40407d0 con 0x7f95bc158f60 2023-10-16T12:33:34.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.921+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4040b40 con 0x7f95bc158f60 2023-10-16T12:33:34.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.922+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0407b0 con 0x7f7a3c0b8480 2023-10-16T12:33:34.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.922+0000 7fe7bf690640 1 -- 172.21.15.60:0/3223493721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b806faf0 msgr2=0x7fe7b815f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.922+0000 7fe7bf690640 1 --2- 172.21.15.60:0/3223493721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b806faf0 0x7fe7b815f480 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009870 tx=0x7fe7a802b9a0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.924+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c040b20 con 0x7f7a3c0b8480 2023-10-16T12:33:34.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.924+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f95a403c070 con 0x7f95bc158f60 2023-10-16T12:33:34.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.926+0000 7f95b8ff9640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f959c06ae70 0x7f959c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.926+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95a40ac8d0 con 0x7f95bc158f60 2023-10-16T12:33:34.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.924+0000 7fe7bf690640 1 -- 172.21.15.60:0/3223493721 shutdown_connections 2023-10-16T12:33:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.924+0000 7fe7bf690640 1 --2- 172.21.15.60:0/3223493721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b806faf0 0x7fe7b815f480 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.924+0000 7fe7bf690640 1 --2- 172.21.15.60:0/3223493721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b815af80 0x7fe7b80cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.924+0000 7fe7bf690640 1 -- 172.21.15.60:0/3223493721 >> 172.21.15.60:0/3223493721 conn(0x7fe7b80d0ab0 msgr2=0x7fe7b80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.926+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7a2c040c80 con 0x7f7a3c0b8480 2023-10-16T12:33:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.927+0000 7f7a38ff9640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7a1c06af00 0x7f7a1c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.927+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a2c0ac660 con 0x7f7a3c0b8480 2023-10-16T12:33:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.927+0000 7fe7bf690640 1 -- 172.21.15.60:0/3223493721 shutdown_connections 2023-10-16T12:33:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.927+0000 7fe7bf690640 1 -- 172.21.15.60:0/3223493721 wait complete. 2023-10-16T12:33:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.928+0000 7f95bbfff640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f959c06ae70 0x7f959c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.928+0000 7f7a327fc640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7a1c06af00 0x7f7a1c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.929+0000 7fe7bf690640 1 Processor -- start 2023-10-16T12:33:34.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.930+0000 7f7a327fc640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7a1c06af00 0x7f7a1c06d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0714a0 tx=0x7f7a24009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.930+0000 7f95bbfff640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f959c06ae70 0x7f959c06d330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95b4005c00 tx=0x7f95b4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.930+0000 7fe7bf690640 1 -- start start 2023-10-16T12:33:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bf690640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 0x7fe7b815a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bf690640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b806faf0 0x7fe7b80b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bf690640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7b815af80 0x7fe7b8150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bf690640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815a220 con 0x7fe7b8065ab0 2023-10-16T12:33:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bf690640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80b65e0 con 0x7fe7b815af80 2023-10-16T12:33:34.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bf690640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80b17c0 con 0x7fe7b806faf0 2023-10-16T12:33:34.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bd42c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 0x7fe7b815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bd42c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 0x7fe7b815a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58126/0 (socket says 172.21.15.60:58126) 2023-10-16T12:33:34.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.931+0000 7fe7bd42c640 1 -- 172.21.15.60:0/2825641187 learned_addr learned my addr 172.21.15.60:0/2825641187 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:34.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bcc2b640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b806faf0 0x7fe7b80b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bdc2d640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7b815af80 0x7fe7b8150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bd42c640 1 -- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b806faf0 msgr2=0x7fe7b80b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bd42c640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b806faf0 0x7fe7b80b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bd42c640 1 -- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7b815af80 msgr2=0x7fe7b8150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:34.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bd42c640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7b815af80 0x7fe7b8150d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.932+0000 7fe7bd42c640 1 -- 172.21.15.60:0/2825641187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a80094d0 con 0x7fe7b8065ab0 2023-10-16T12:33:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.933+0000 7fe7bd42c640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 0x7fe7b815a440 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400b4a0 tx=0x7fe7b400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.933+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a2c0777a0 con 0x7f7a3c0b8480 2023-10-16T12:33:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.942+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95a4077aa0 con 0x7f95bc158f60 2023-10-16T12:33:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.942+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4010610 con 0x7fe7b8065ab0 2023-10-16T12:33:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.943+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b80b5d40 con 0x7fe7b8065ab0 2023-10-16T12:33:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.943+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b80b6300 con 0x7fe7b8065ab0 2023-10-16T12:33:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.943+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b4010770 con 0x7fe7b8065ab0 2023-10-16T12:33:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.943+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4010b20 con 0x7fe7b8065ab0 2023-10-16T12:33:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.944+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe798005250 con 0x7fe7b8065ab0 2023-10-16T12:33:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.955+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe7b4010c80 con 0x7fe7b8065ab0 2023-10-16T12:33:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.957+0000 7fe7adffb640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe79c06ae70 0x7fe79c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.957+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7b4089610 con 0x7fe7b8065ab0 2023-10-16T12:33:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.958+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7b40899f0 con 0x7fe7b8065ab0 2023-10-16T12:33:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.960+0000 7fe7bcc2b640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe79c06ae70 0x7fe79c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:34.961+0000 7fe7bcc2b640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe79c06ae70 0x7fe79c06d330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8036000 tx=0x7fe7a803b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:35.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.210+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4534008a10 con 0x7f45340df3f0 2023-10-16T12:33:35.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.211+0000 7f4527fff640 1 -- 172.21.15.60:0/369713216 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f453c0c2090 con 0x7f45340df3f0 2023-10-16T12:33:35.214 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:35.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.216+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f452006aed0 msgr2=0x7f452006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.216+0000 7f4525ffb640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f452006aed0 0x7f452006d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4530006fd0 tx=0x7f4530008040 comp rx=0 tx=0).stop 2023-10-16T12:33:35.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.217+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 msgr2=0x7f453404c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.217+0000 7f4525ffb640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 0x7f453404c5e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f453c0687d0 tx=0x7f453c067b80 comp rx=0 tx=0).stop 2023-10-16T12:33:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 shutdown_connections 2023-10-16T12:33:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f452006aed0 0x7f452006d390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45340df3f0 0x7f453404c5e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4534050270 0x7f4534174460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 --2- 172.21.15.60:0/369713216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4534005ed0 0x7f453404df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 >> 172.21.15.60:0/369713216 conn(0x7f4534054b00 msgr2=0x7f45340de500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 shutdown_connections 2023-10-16T12:33:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.219+0000 7f4525ffb640 1 -- 172.21.15.60:0/369713216 wait complete. 2023-10-16T12:33:35.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.1 2023-10-16T12:33:35.251 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.266+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0db804e160 con 0x7f0db80cf710 2023-10-16T12:33:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.267+0000 7f0daeffd640 1 -- 172.21.15.60:0/2204752359 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0da8058210 con 0x7f0db80cf710 2023-10-16T12:33:35.270 INFO:teuthology.orchestra.run.smithi060.stdout:60129542150 2023-10-16T12:33:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.270+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d9c06aec0 msgr2=0x7f0d9c06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.270+0000 7f0dc029b640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d9c06aec0 0x7f0d9c06d380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0db0035000 tx=0x7f0db0030000 comp rx=0 tx=0).stop 2023-10-16T12:33:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 msgr2=0x7f0db81fbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db81fbda0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c750 tx=0x7f0da800cc10 comp rx=0 tx=0).stop 2023-10-16T12:33:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 shutdown_connections 2023-10-16T12:33:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d9c06aec0 0x7f0d9c06d380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0db80cfaf0 0x7f0db81fc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0db80cf710 0x7f0db81fbda0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 --2- 172.21.15.60:0/2204752359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0db80b6cd0 0x7f0db81e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.271+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 >> 172.21.15.60:0/2204752359 conn(0x7f0db80d2cc0 msgr2=0x7f0db80b7170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.272+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 shutdown_connections 2023-10-16T12:33:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.272+0000 7f0dc029b640 1 -- 172.21.15.60:0/2204752359 wait complete. 2023-10-16T12:33:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.290+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f032c0ca810 con 0x7f032c06a2e0 2023-10-16T12:33:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.291+0000 7f031affd640 1 -- 172.21.15.60:0/4281685424 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0320014020 con 0x7f032c06a2e0 2023-10-16T12:33:35.293 INFO:teuthology.orchestra.run.smithi060.stdout:60129542151 2023-10-16T12:33:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f030c06aec0 msgr2=0x7f030c06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f030c06aec0 0x7f030c06d380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f031c03a000 tx=0x7f031c034000 comp rx=0 tx=0).stop 2023-10-16T12:33:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 msgr2=0x7f032c0c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 0x7f032c0c7d10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0320001460 tx=0x7f0320001920 comp rx=0 tx=0).stop 2023-10-16T12:33:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 shutdown_connections 2023-10-16T12:33:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f030c06aec0 0x7f030c06d380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f032c0ade50 0x7f032c0c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f032c159110 0x7f032c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 --2- 172.21.15.60:0/4281685424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f032c06a2e0 0x7f032c0c7d10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.295+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 >> 172.21.15.60:0/4281685424 conn(0x7f032c0ce9b0 msgr2=0x7f032c158350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.296+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 shutdown_connections 2023-10-16T12:33:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.296+0000 7f0330ae0640 1 -- 172.21.15.60:0/4281685424 wait complete. 2023-10-16T12:33:35.300 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.0 2023-10-16T12:33:35.301 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.315 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2023-10-16T12:33:35.315 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.316+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4d2c0c1c20 con 0x7f4d2c069da0 2023-10-16T12:33:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.316+0000 7f4d13fff640 1 -- 172.21.15.60:0/115031895 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d20054b50 con 0x7f4d2c069da0 2023-10-16T12:33:35.319 INFO:teuthology.orchestra.run.smithi060.stdout:60129542151 2023-10-16T12:33:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.318+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4d1806af40 msgr2=0x7f4d1806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.318+0000 7f4d11ffb640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4d1806af40 0x7f4d1806d400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4d24001700 tx=0x7f4d24001360 comp rx=0 tx=0).stop 2023-10-16T12:33:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.318+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 msgr2=0x7f4d2c0b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.318+0000 7f4d11ffb640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 0x7f4d2c0b15a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000d5f0 tx=0x7f4d2000dab0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 shutdown_connections 2023-10-16T12:33:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4d1806af40 0x7f4d1806d400 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d2c160b90 0x7f4d2c0ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d2c0cac60 0x7f4d2c0ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 --2- 172.21.15.60:0/115031895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d2c069da0 0x7f4d2c0b15a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 >> 172.21.15.60:0/115031895 conn(0x7f4d2c0ce9b0 msgr2=0x7f4d2c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 shutdown_connections 2023-10-16T12:33:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.319+0000 7f4d11ffb640 1 -- 172.21.15.60:0/115031895 wait complete. 2023-10-16T12:33:35.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.332+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6b2c0712b0 con 0x7f6b2c0ce2b0 2023-10-16T12:33:35.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.332+0000 7f6b21ffb640 1 -- 172.21.15.60:0/2928450395 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b28079e20 con 0x7f6b2c0ce2b0 2023-10-16T12:33:35.335 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:35.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6b1006aef0 msgr2=0x7f6b1006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6b1006aef0 0x7f6b1006d3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009d70 tx=0x7f6b2400b040 comp rx=0 tx=0).stop 2023-10-16T12:33:35.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 msgr2=0x7f6b2c06f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 0x7f6b2c06f860 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6b28034040 tx=0x7f6b280046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 shutdown_connections 2023-10-16T12:33:35.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6b1006aef0 0x7f6b1006d3b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b2c0ce2b0 0x7f6b2c06f860 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b2c0ad920 0x7f6b2c14ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 --2- 172.21.15.60:0/2928450395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b2c06a4b0 0x7f6b2c14b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 >> 172.21.15.60:0/2928450395 conn(0x7f6b2c0ce9b0 msgr2=0x7f6b2c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 shutdown_connections 2023-10-16T12:33:35.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.335+0000 7f6b33402640 1 -- 172.21.15.60:0/2928450395 wait complete. 2023-10-16T12:33:35.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2023-10-16T12:33:35.341 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.2 2023-10-16T12:33:35.358 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.359+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7a3c0b6e70 con 0x7f7a3c0b8480 2023-10-16T12:33:35.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.360+0000 7f7a38ff9640 1 -- 172.21.15.60:0/2826203210 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a2c07adc0 con 0x7f7a3c0b8480 2023-10-16T12:33:35.362 INFO:teuthology.orchestra.run.smithi060.stdout:60129542151 2023-10-16T12:33:35.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7a1c06af00 msgr2=0x7f7a1c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7a1c06af00 0x7f7a1c06d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0714a0 tx=0x7f7a24009040 comp rx=0 tx=0).stop 2023-10-16T12:33:35.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 msgr2=0x7f7a3c1e6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 0x7f7a3c1e6110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c000f80 tx=0x7f7a2c004ab0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 shutdown_connections 2023-10-16T12:33:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7a1c06af00 0x7f7a1c06d3c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a3c15cd70 0x7f7a3c1e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a3c0cded0 0x7f7a3c1e6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 --2- 172.21.15.60:0/2826203210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a3c0b8480 0x7f7a3c1e6110 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 >> 172.21.15.60:0/2826203210 conn(0x7f7a3c0cf570 msgr2=0x7f7a3c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 shutdown_connections 2023-10-16T12:33:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.362+0000 7f7a3bfff640 1 -- 172.21.15.60:0/2826203210 wait complete. 2023-10-16T12:33:35.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.3 2023-10-16T12:33:35.380 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.384+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f95bc04e160 con 0x7f95bc158f60 2023-10-16T12:33:35.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.385+0000 7f95b8ff9640 1 -- 172.21.15.60:0/1116092504 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95a407b0c0 con 0x7f95bc158f60 2023-10-16T12:33:35.387 INFO:teuthology.orchestra.run.smithi060.stdout:60129542149 2023-10-16T12:33:35.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f959c06ae70 msgr2=0x7f959c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f959c06ae70 0x7f959c06d330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95b4005c00 tx=0x7f95b4009040 comp rx=0 tx=0).stop 2023-10-16T12:33:35.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 msgr2=0x7f95bc0c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 0x7f95bc0c6d70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f95a40090e0 tx=0x7f95a40046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 shutdown_connections 2023-10-16T12:33:35.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f959c06ae70 0x7f959c06d330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ce1c0 0x7f95bc0c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc158f60 0x7f95bc0c6d70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 --2- 172.21.15.60:0/1116092504 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06a4b0 0x7f95bc0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 >> 172.21.15.60:0/1116092504 conn(0x7f95bc0ce9b0 msgr2=0x7f95bc159c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 shutdown_connections 2023-10-16T12:33:35.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.387+0000 7f95c2875640 1 -- 172.21.15.60:0/1116092504 wait complete. 2023-10-16T12:33:35.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.391+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe7980057d0 con 0x7fe7b8065ab0 2023-10-16T12:33:35.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.392+0000 7fe7adffb640 1 -- 172.21.15.60:0/2825641187 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7b4014090 con 0x7fe7b8065ab0 2023-10-16T12:33:35.394 INFO:teuthology.orchestra.run.smithi060.stdout:60129542150 2023-10-16T12:33:35.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.395+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe79c06ae70 msgr2=0x7fe79c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.395+0000 7fe7bf690640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe79c06ae70 0x7fe79c06d330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8036000 tx=0x7fe7a803b040 comp rx=0 tx=0).stop 2023-10-16T12:33:35.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.395+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 msgr2=0x7fe7b815a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.395+0000 7fe7bf690640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 0x7fe7b815a440 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400b4a0 tx=0x7fe7b400b960 comp rx=0 tx=0).stop 2023-10-16T12:33:35.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 shutdown_connections 2023-10-16T12:33:35.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe79c06ae70 0x7fe79c06d330 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7b806faf0 0x7fe7b80b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe7b815af80 0x7fe7b8150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 --2- 172.21.15.60:0/2825641187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b8065ab0 0x7fe7b815a440 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 >> 172.21.15.60:0/2825641187 conn(0x7fe7b80d0ab0 msgr2=0x7fe7b815bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 shutdown_connections 2023-10-16T12:33:35.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.396+0000 7fe7bf690640 1 -- 172.21.15.60:0/2825641187 wait complete. 2023-10-16T12:33:35.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.7 2023-10-16T12:33:35.413 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2023-10-16T12:33:35.416 DEBUG:teuthology.parallel:result is None 2023-10-16T12:33:35.417 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T12:33:35.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:33:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.674+0000 7f6fee9b0640 1 Processor -- start 2023-10-16T12:33:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.675+0000 7f6fee9b0640 1 -- start start 2023-10-16T12:33:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.676+0000 7f6fee9b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fe815aaa0 0x7f6fe815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:35.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.676+0000 7f6fee9b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:35.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.676+0000 7f6fee9b0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe815ed60 con 0x7f6fe815ae80 2023-10-16T12:33:35.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.676+0000 7f6fee9b0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe806b810 con 0x7f6fe815aaa0 2023-10-16T12:33:35.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.676+0000 7f6fee9b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80707e0 con 0x7f6fe80ce390 2023-10-16T12:33:35.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.677+0000 7f6fecf4d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:35.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.677+0000 7f6fecf4d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53470/0 (socket says 172.21.15.60:53470) 2023-10-16T12:33:35.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.677+0000 7f6fecf4d640 1 -- 172.21.15.60:0/1549976322 learned_addr learned my addr 172.21.15.60:0/1549976322 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:35.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.677+0000 7f6fe7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fe815aaa0 0x7f6fe815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:35.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.678+0000 7f6fe77fe640 1 -- 172.21.15.60:0/1549976322 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577840187 0 0) 0x7f6fe815ed60 con 0x7f6fe815ae80 2023-10-16T12:33:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.678+0000 7f6fe77fe640 1 -- 172.21.15.60:0/1549976322 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4003520 con 0x7f6fe815ae80 2023-10-16T12:33:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.678+0000 7f6fecf4d640 1 -- 172.21.15.60:0/1549976322 >> v2:172.21.15.144:3300/0 conn(0x7f6fe815aaa0 msgr2=0x7f6fe815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.678+0000 7f6fecf4d640 1 --2- 172.21.15.60:0/1549976322 >> v2:172.21.15.144:3300/0 conn(0x7f6fe815aaa0 0x7f6fe815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.678+0000 7f6fecf4d640 1 -- 172.21.15.60:0/1549976322 >> v1:172.21.15.144:6789/0 conn(0x7f6fe815ae80 legacy=0x7f6fe80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.678+0000 7f6fecf4d640 1 -- 172.21.15.60:0/1549976322 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe815ef90 con 0x7f6fe80ce390 2023-10-16T12:33:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.679+0000 7f6fecf4d640 1 --2- 172.21.15.60:0/1549976322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe8070f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009e50 tx=0x7f6fe002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480676baa0a111e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fe77fe640 1 -- 172.21.15.60:0/1549976322 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe002cb00 con 0x7f6fe80ce390 2023-10-16T12:33:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fe77fe640 1 -- 172.21.15.60:0/1549976322 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe002cc60 con 0x7f6fe80ce390 2023-10-16T12:33:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fe77fe640 1 -- 172.21.15.60:0/1549976322 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe00343f0 con 0x7f6fe80ce390 2023-10-16T12:33:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 -- 172.21.15.60:0/1549976322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 msgr2=0x7f6fe8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/1549976322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe8070f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009e50 tx=0x7f6fe002bf60 comp rx=0 tx=0).stop 2023-10-16T12:33:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 -- 172.21.15.60:0/1549976322 shutdown_connections 2023-10-16T12:33:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/1549976322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe8070f50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/1549976322 >> v2:172.21.15.144:3300/0 conn(0x7f6fe815aaa0 0x7f6fe815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 -- 172.21.15.60:0/1549976322 >> 172.21.15.60:0/1549976322 conn(0x7f6fe80cb410 msgr2=0x7f6fe80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 -- 172.21.15.60:0/1549976322 shutdown_connections 2023-10-16T12:33:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.680+0000 7f6fee9b0640 1 -- 172.21.15.60:0/1549976322 wait complete. 2023-10-16T12:33:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.682+0000 7f6fee9b0640 1 Processor -- start 2023-10-16T12:33:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.682+0000 7f6fee9b0640 1 -- start start 2023-10-16T12:33:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.683+0000 7f6fee9b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe80c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.683+0000 7f6fee9b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe815aaa0 0x7f6fe814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.683+0000 7f6fee9b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 0x7f6fe81f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.683+0000 7f6fee9b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80c45e0 con 0x7f6fe815ae80 2023-10-16T12:33:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.683+0000 7f6fee9b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe814aa40 con 0x7f6fe815aaa0 2023-10-16T12:33:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.683+0000 7f6fee9b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80c39d0 con 0x7f6fe80ce390 2023-10-16T12:33:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 0x7f6fe81f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 0x7f6fe81f5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58136/0 (socket says 172.21.15.60:58136) 2023-10-16T12:33:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 -- 172.21.15.60:0/4059270901 learned_addr learned my addr 172.21.15.60:0/4059270901 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fe7fff640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe80c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fdffff640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe815aaa0 0x7f6fe814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 -- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 msgr2=0x7f6fe80c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe80c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 -- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe815aaa0 msgr2=0x7f6fe814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe815aaa0 0x7f6fe814a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.684+0000 7f6fecf4d640 1 -- 172.21.15.60:0/4059270901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0009b30 con 0x7f6fe815ae80 2023-10-16T12:33:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.685+0000 7f6fecf4d640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 0x7f6fe81f5e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009e50 tx=0x7f6fe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.685+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0009510 con 0x7f6fe815ae80 2023-10-16T12:33:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.685+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe0003e10 con 0x7f6fe815ae80 2023-10-16T12:33:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.685+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0004450 con 0x7f6fe815ae80 2023-10-16T12:33:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.685+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe80c3cb0 con 0x7f6fe815ae80 2023-10-16T12:33:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.685+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe80c4270 con 0x7f6fe815ae80 2023-10-16T12:33:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.686+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe80c4750 con 0x7f6fe815ae80 2023-10-16T12:33:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.687+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6fe00045b0 con 0x7f6fe815ae80 2023-10-16T12:33:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.688+0000 7f6fe57fa640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6fc806ae70 0x7f6fc806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.688+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe00abf20 con 0x7f6fe815ae80 2023-10-16T12:33:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.688+0000 7f6fe7fff640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6fc806ae70 0x7f6fc806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.690+0000 7f6fe7fff640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6fc806ae70 0x7f6fc806d330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80c54e0 tx=0x7f6fd8009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.692+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fe0077100 con 0x7f6fe815ae80 2023-10-16T12:33:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.874+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fe815f550 con 0x7f6fc806ae70 2023-10-16T12:33:35.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.896+0000 7f6fe57fa640 1 -- 172.21.15.60:0/4059270901 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264381 (secure 0 0 0) 0x7f6fd8011a30 con 0x7f6fc806ae70 2023-10-16T12:33:35.903 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:33:35.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.902+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6fc806ae70 msgr2=0x7f6fc806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.902+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6fc806ae70 0x7f6fc806d330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80c54e0 tx=0x7f6fd8009190 comp rx=0 tx=0).stop 2023-10-16T12:33:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.902+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 msgr2=0x7f6fe81f5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.902+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 0x7f6fe81f5e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009e50 tx=0x7f6fe0004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 shutdown_connections 2023-10-16T12:33:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6fc806ae70 0x7f6fc806d330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe815ae80 0x7f6fe81f5e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe815aaa0 0x7f6fe814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 --2- 172.21.15.60:0/4059270901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80ce390 0x7f6fe80c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 >> 172.21.15.60:0/4059270901 conn(0x7f6fe80cb410 msgr2=0x7f6fe80b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 shutdown_connections 2023-10-16T12:33:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:35.903+0000 7f6fee9b0640 1 -- 172.21.15.60:0/4059270901 wait complete. 2023-10-16T12:33:35.924 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:33:35.735333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236216,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738504,"statfs":{"total":773094113280,"available":772852228096,"internally_reserved":0,"allocated":3055616,"data_stored":1816468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":81,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.154574"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930864+0000","last_change":"2023-10-16T12:33:24.620784+0000","last_active":"2023-10-16T12:33:25.930864+0000","last_peered":"2023-10-16T12:33:25.930864+0000","last_clean":"2023-10-16T12:33:25.930864+0000","last_became_active":"2023-10-16T12:33:24.620205+0000","last_became_peered":"2023-10-16T12:33:24.620205+0000","last_unstale":"2023-10-16T12:33:25.930864+0000","last_undegraded":"2023-10-16T12:33:25.930864+0000","last_fullsized":"2023-10-16T12:33:25.930864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585097+0000","last_change":"2023-10-16T12:33:24.608395+0000","last_active":"2023-10-16T12:33:25.585097+0000","last_peered":"2023-10-16T12:33:25.585097+0000","last_clean":"2023-10-16T12:33:25.585097+0000","last_became_active":"2023-10-16T12:33:24.608062+0000","last_became_peered":"2023-10-16T12:33:24.608062+0000","last_unstale":"2023-10-16T12:33:25.585097+0000","last_undegraded":"2023-10-16T12:33:25.585097+0000","last_fullsized":"2023-10-16T12:33:25.585097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928883+0000","last_change":"2023-10-16T12:33:24.607434+0000","last_active":"2023-10-16T12:33:25.928883+0000","last_peered":"2023-10-16T12:33:25.928883+0000","last_clean":"2023-10-16T12:33:25.928883+0000","last_became_active":"2023-10-16T12:33:24.607119+0000","last_became_peered":"2023-10-16T12:33:24.607119+0000","last_unstale":"2023-10-16T12:33:25.928883+0000","last_undegraded":"2023-10-16T12:33:25.928883+0000","last_fullsized":"2023-10-16T12:33:25.928883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.592697+0000","last_change":"2023-10-16T12:33:24.611057+0000","last_active":"2023-10-16T12:33:25.592697+0000","last_peered":"2023-10-16T12:33:25.592697+0000","last_clean":"2023-10-16T12:33:25.592697+0000","last_became_active":"2023-10-16T12:33:24.610357+0000","last_became_peered":"2023-10-16T12:33:24.610357+0000","last_unstale":"2023-10-16T12:33:25.592697+0000","last_undegraded":"2023-10-16T12:33:25.592697+0000","last_fullsized":"2023-10-16T12:33:25.592697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930042+0000","last_change":"2023-10-16T12:33:24.617198+0000","last_active":"2023-10-16T12:33:25.930042+0000","last_peered":"2023-10-16T12:33:25.930042+0000","last_clean":"2023-10-16T12:33:25.930042+0000","last_became_active":"2023-10-16T12:33:24.616710+0000","last_became_peered":"2023-10-16T12:33:24.616710+0000","last_unstale":"2023-10-16T12:33:25.930042+0000","last_undegraded":"2023-10-16T12:33:25.930042+0000","last_fullsized":"2023-10-16T12:33:25.930042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585757+0000","last_change":"2023-10-16T12:33:24.608373+0000","last_active":"2023-10-16T12:33:25.585757+0000","last_peered":"2023-10-16T12:33:25.585757+0000","last_clean":"2023-10-16T12:33:25.585757+0000","last_became_active":"2023-10-16T12:33:24.608083+0000","last_became_peered":"2023-10-16T12:33:24.608083+0000","last_unstale":"2023-10-16T12:33:25.585757+0000","last_undegraded":"2023-10-16T12:33:25.585757+0000","last_fullsized":"2023-10-16T12:33:25.585757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928757+0000","last_change":"2023-10-16T12:33:24.620178+0000","last_active":"2023-10-16T12:33:25.928757+0000","last_peered":"2023-10-16T12:33:25.928757+0000","last_clean":"2023-10-16T12:33:25.928757+0000","last_became_active":"2023-10-16T12:33:24.619871+0000","last_became_peered":"2023-10-16T12:33:24.619871+0000","last_unstale":"2023-10-16T12:33:25.928757+0000","last_undegraded":"2023-10-16T12:33:25.928757+0000","last_fullsized":"2023-10-16T12:33:25.928757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929201+0000","last_change":"2023-10-16T12:33:24.610760+0000","last_active":"2023-10-16T12:33:25.929201+0000","last_peered":"2023-10-16T12:33:25.929201+0000","last_clean":"2023-10-16T12:33:25.929201+0000","last_became_active":"2023-10-16T12:33:24.609947+0000","last_became_peered":"2023-10-16T12:33:24.609947+0000","last_unstale":"2023-10-16T12:33:25.929201+0000","last_undegraded":"2023-10-16T12:33:25.929201+0000","last_fullsized":"2023-10-16T12:33:25.929201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930940+0000","last_change":"2023-10-16T12:33:24.626800+0000","last_active":"2023-10-16T12:33:25.930940+0000","last_peered":"2023-10-16T12:33:25.930940+0000","last_clean":"2023-10-16T12:33:25.930940+0000","last_became_active":"2023-10-16T12:33:24.626566+0000","last_became_peered":"2023-10-16T12:33:24.626566+0000","last_unstale":"2023-10-16T12:33:25.930940+0000","last_undegraded":"2023-10-16T12:33:25.930940+0000","last_fullsized":"2023-10-16T12:33:25.930940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:08.131810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928952+0000","last_change":"2023-10-16T12:33:24.610841+0000","last_active":"2023-10-16T12:33:25.928952+0000","last_peered":"2023-10-16T12:33:25.928952+0000","last_clean":"2023-10-16T12:33:25.928952+0000","last_became_active":"2023-10-16T12:33:24.610533+0000","last_became_peered":"2023-10-16T12:33:24.610533+0000","last_unstale":"2023-10-16T12:33:25.928952+0000","last_undegraded":"2023-10-16T12:33:25.928952+0000","last_fullsized":"2023-10-16T12:33:25.928952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:39.121403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928420+0000","last_change":"2023-10-16T12:33:24.609014+0000","last_active":"2023-10-16T12:33:25.928420+0000","last_peered":"2023-10-16T12:33:25.928420+0000","last_clean":"2023-10-16T12:33:25.928420+0000","last_became_active":"2023-10-16T12:33:24.607622+0000","last_became_peered":"2023-10-16T12:33:24.607622+0000","last_unstale":"2023-10-16T12:33:25.928420+0000","last_undegraded":"2023-10-16T12:33:25.928420+0000","last_fullsized":"2023-10-16T12:33:25.928420+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:00.436524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588652+0000","last_change":"2023-10-16T12:33:24.609916+0000","last_active":"2023-10-16T12:33:25.588652+0000","last_peered":"2023-10-16T12:33:25.588652+0000","last_clean":"2023-10-16T12:33:25.588652+0000","last_became_active":"2023-10-16T12:33:24.609319+0000","last_became_peered":"2023-10-16T12:33:24.609319+0000","last_unstale":"2023-10-16T12:33:25.588652+0000","last_undegraded":"2023-10-16T12:33:25.588652+0000","last_fullsized":"2023-10-16T12:33:25.588652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:04.453258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929179+0000","last_change":"2023-10-16T12:33:24.622611+0000","last_active":"2023-10-16T12:33:25.929179+0000","last_peered":"2023-10-16T12:33:25.929179+0000","last_clean":"2023-10-16T12:33:25.929179+0000","last_became_active":"2023-10-16T12:33:24.622083+0000","last_became_peered":"2023-10-16T12:33:24.622083+0000","last_unstale":"2023-10-16T12:33:25.929179+0000","last_undegraded":"2023-10-16T12:33:25.929179+0000","last_fullsized":"2023-10-16T12:33:25.929179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927071+0000","last_change":"2023-10-16T12:33:24.605142+0000","last_active":"2023-10-16T12:33:25.927071+0000","last_peered":"2023-10-16T12:33:25.927071+0000","last_clean":"2023-10-16T12:33:25.927071+0000","last_became_active":"2023-10-16T12:33:24.604653+0000","last_became_peered":"2023-10-16T12:33:24.604653+0000","last_unstale":"2023-10-16T12:33:25.927071+0000","last_undegraded":"2023-10-16T12:33:25.927071+0000","last_fullsized":"2023-10-16T12:33:25.927071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:42.877240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584708+0000","last_change":"2023-10-16T12:33:24.606078+0000","last_active":"2023-10-16T12:33:25.584708+0000","last_peered":"2023-10-16T12:33:25.584708+0000","last_clean":"2023-10-16T12:33:25.584708+0000","last_became_active":"2023-10-16T12:33:24.605741+0000","last_became_peered":"2023-10-16T12:33:24.605741+0000","last_unstale":"2023-10-16T12:33:25.584708+0000","last_undegraded":"2023-10-16T12:33:25.584708+0000","last_fullsized":"2023-10-16T12:33:25.584708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584951+0000","last_change":"2023-10-16T12:33:24.606094+0000","last_active":"2023-10-16T12:33:25.584951+0000","last_peered":"2023-10-16T12:33:25.584951+0000","last_clean":"2023-10-16T12:33:25.584951+0000","last_became_active":"2023-10-16T12:33:24.605770+0000","last_became_peered":"2023-10-16T12:33:24.605770+0000","last_unstale":"2023-10-16T12:33:25.584951+0000","last_undegraded":"2023-10-16T12:33:25.584951+0000","last_fullsized":"2023-10-16T12:33:25.584951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590756+0000","last_change":"2023-10-16T12:33:24.618354+0000","last_active":"2023-10-16T12:33:25.590756+0000","last_peered":"2023-10-16T12:33:25.590756+0000","last_clean":"2023-10-16T12:33:25.590756+0000","last_became_active":"2023-10-16T12:33:24.617040+0000","last_became_peered":"2023-10-16T12:33:24.617040+0000","last_unstale":"2023-10-16T12:33:25.590756+0000","last_undegraded":"2023-10-16T12:33:25.590756+0000","last_fullsized":"2023-10-16T12:33:25.590756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927146+0000","last_change":"2023-10-16T12:33:24.612736+0000","last_active":"2023-10-16T12:33:25.927146+0000","last_peered":"2023-10-16T12:33:25.927146+0000","last_clean":"2023-10-16T12:33:25.927146+0000","last_became_active":"2023-10-16T12:33:24.612452+0000","last_became_peered":"2023-10-16T12:33:24.612452+0000","last_unstale":"2023-10-16T12:33:25.927146+0000","last_undegraded":"2023-10-16T12:33:25.927146+0000","last_fullsized":"2023-10-16T12:33:25.927146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930486+0000","last_change":"2023-10-16T12:33:24.621402+0000","last_active":"2023-10-16T12:33:25.930486+0000","last_peered":"2023-10-16T12:33:25.930486+0000","last_clean":"2023-10-16T12:33:25.930486+0000","last_became_active":"2023-10-16T12:33:24.621020+0000","last_became_peered":"2023-10-16T12:33:24.621020+0000","last_unstale":"2023-10-16T12:33:25.930486+0000","last_undegraded":"2023-10-16T12:33:25.930486+0000","last_fullsized":"2023-10-16T12:33:25.930486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928170+0000","last_change":"2023-10-16T12:33:24.613689+0000","last_active":"2023-10-16T12:33:25.928170+0000","last_peered":"2023-10-16T12:33:25.928170+0000","last_clean":"2023-10-16T12:33:25.928170+0000","last_became_active":"2023-10-16T12:33:24.613366+0000","last_became_peered":"2023-10-16T12:33:24.613366+0000","last_unstale":"2023-10-16T12:33:25.928170+0000","last_undegraded":"2023-10-16T12:33:25.928170+0000","last_fullsized":"2023-10-16T12:33:25.928170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:46.931083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928066+0000","last_change":"2023-10-16T12:33:24.615117+0000","last_active":"2023-10-16T12:33:25.928066+0000","last_peered":"2023-10-16T12:33:25.928066+0000","last_clean":"2023-10-16T12:33:25.928066+0000","last_became_active":"2023-10-16T12:33:24.614794+0000","last_became_peered":"2023-10-16T12:33:24.614794+0000","last_unstale":"2023-10-16T12:33:25.928066+0000","last_undegraded":"2023-10-16T12:33:25.928066+0000","last_fullsized":"2023-10-16T12:33:25.928066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586082+0000","last_change":"2023-10-16T12:33:24.616251+0000","last_active":"2023-10-16T12:33:25.586082+0000","last_peered":"2023-10-16T12:33:25.586082+0000","last_clean":"2023-10-16T12:33:25.586082+0000","last_became_active":"2023-10-16T12:33:24.615946+0000","last_became_peered":"2023-10-16T12:33:24.615946+0000","last_unstale":"2023-10-16T12:33:25.586082+0000","last_undegraded":"2023-10-16T12:33:25.586082+0000","last_fullsized":"2023-10-16T12:33:25.586082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929157+0000","last_change":"2023-10-16T12:33:24.613346+0000","last_active":"2023-10-16T12:33:25.929157+0000","last_peered":"2023-10-16T12:33:25.929157+0000","last_clean":"2023-10-16T12:33:25.929157+0000","last_became_active":"2023-10-16T12:33:24.612974+0000","last_became_peered":"2023-10-16T12:33:24.612974+0000","last_unstale":"2023-10-16T12:33:25.929157+0000","last_undegraded":"2023-10-16T12:33:25.929157+0000","last_fullsized":"2023-10-16T12:33:25.929157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585129+0000","last_change":"2023-10-16T12:33:24.624851+0000","last_active":"2023-10-16T12:33:25.585129+0000","last_peered":"2023-10-16T12:33:25.585129+0000","last_clean":"2023-10-16T12:33:25.585129+0000","last_became_active":"2023-10-16T12:33:24.624550+0000","last_became_peered":"2023-10-16T12:33:24.624550+0000","last_unstale":"2023-10-16T12:33:25.585129+0000","last_undegraded":"2023-10-16T12:33:25.585129+0000","last_fullsized":"2023-10-16T12:33:25.585129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930566+0000","last_change":"2023-10-16T12:33:24.614362+0000","last_active":"2023-10-16T12:33:25.930566+0000","last_peered":"2023-10-16T12:33:25.930566+0000","last_clean":"2023-10-16T12:33:25.930566+0000","last_became_active":"2023-10-16T12:33:24.613880+0000","last_became_peered":"2023-10-16T12:33:24.613880+0000","last_unstale":"2023-10-16T12:33:25.930566+0000","last_undegraded":"2023-10-16T12:33:25.930566+0000","last_fullsized":"2023-10-16T12:33:25.930566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929120+0000","last_change":"2023-10-16T12:33:24.618147+0000","last_active":"2023-10-16T12:33:25.929120+0000","last_peered":"2023-10-16T12:33:25.929120+0000","last_clean":"2023-10-16T12:33:25.929120+0000","last_became_active":"2023-10-16T12:33:24.617724+0000","last_became_peered":"2023-10-16T12:33:24.617724+0000","last_unstale":"2023-10-16T12:33:25.929120+0000","last_undegraded":"2023-10-16T12:33:25.929120+0000","last_fullsized":"2023-10-16T12:33:25.929120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:52.314433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585125+0000","last_change":"2023-10-16T12:33:24.605102+0000","last_active":"2023-10-16T12:33:25.585125+0000","last_peered":"2023-10-16T12:33:25.585125+0000","last_clean":"2023-10-16T12:33:25.585125+0000","last_became_active":"2023-10-16T12:33:24.604498+0000","last_became_peered":"2023-10-16T12:33:24.604498+0000","last_unstale":"2023-10-16T12:33:25.585125+0000","last_undegraded":"2023-10-16T12:33:25.585125+0000","last_fullsized":"2023-10-16T12:33:25.585125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930199+0000","last_change":"2023-10-16T12:33:24.613950+0000","last_active":"2023-10-16T12:33:25.930199+0000","last_peered":"2023-10-16T12:33:25.930199+0000","last_clean":"2023-10-16T12:33:25.930199+0000","last_became_active":"2023-10-16T12:33:24.613512+0000","last_became_peered":"2023-10-16T12:33:24.613512+0000","last_unstale":"2023-10-16T12:33:25.930199+0000","last_undegraded":"2023-10-16T12:33:25.930199+0000","last_fullsized":"2023-10-16T12:33:25.930199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928442+0000","last_change":"2023-10-16T12:33:24.612300+0000","last_active":"2023-10-16T12:33:25.928442+0000","last_peered":"2023-10-16T12:33:25.928442+0000","last_clean":"2023-10-16T12:33:25.928442+0000","last_became_active":"2023-10-16T12:33:24.611773+0000","last_became_peered":"2023-10-16T12:33:24.611773+0000","last_unstale":"2023-10-16T12:33:25.928442+0000","last_undegraded":"2023-10-16T12:33:25.928442+0000","last_fullsized":"2023-10-16T12:33:25.928442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590561+0000","last_change":"2023-10-16T12:33:24.615637+0000","last_active":"2023-10-16T12:33:25.590561+0000","last_peered":"2023-10-16T12:33:25.590561+0000","last_clean":"2023-10-16T12:33:25.590561+0000","last_became_active":"2023-10-16T12:33:24.615341+0000","last_became_peered":"2023-10-16T12:33:24.615341+0000","last_unstale":"2023-10-16T12:33:25.590561+0000","last_undegraded":"2023-10-16T12:33:25.590561+0000","last_fullsized":"2023-10-16T12:33:25.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928067+0000","last_change":"2023-10-16T12:33:24.603765+0000","last_active":"2023-10-16T12:33:25.928067+0000","last_peered":"2023-10-16T12:33:25.928067+0000","last_clean":"2023-10-16T12:33:25.928067+0000","last_became_active":"2023-10-16T12:33:24.603469+0000","last_became_peered":"2023-10-16T12:33:24.603469+0000","last_unstale":"2023-10-16T12:33:25.928067+0000","last_undegraded":"2023-10-16T12:33:25.928067+0000","last_fullsized":"2023-10-16T12:33:25.928067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927844+0000","last_change":"2023-10-16T12:33:24.617779+0000","last_active":"2023-10-16T12:33:25.927844+0000","last_peered":"2023-10-16T12:33:25.927844+0000","last_clean":"2023-10-16T12:33:25.927844+0000","last_became_active":"2023-10-16T12:33:24.617328+0000","last_became_peered":"2023-10-16T12:33:24.617328+0000","last_unstale":"2023-10-16T12:33:25.927844+0000","last_undegraded":"2023-10-16T12:33:25.927844+0000","last_fullsized":"2023-10-16T12:33:25.927844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:26.089073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.587918+0000","last_change":"2023-10-16T12:33:24.605171+0000","last_active":"2023-10-16T12:33:25.587918+0000","last_peered":"2023-10-16T12:33:25.587918+0000","last_clean":"2023-10-16T12:33:25.587918+0000","last_became_active":"2023-10-16T12:33:24.604706+0000","last_became_peered":"2023-10-16T12:33:24.604706+0000","last_unstale":"2023-10-16T12:33:25.587918+0000","last_undegraded":"2023-10-16T12:33:25.587918+0000","last_fullsized":"2023-10-16T12:33:25.587918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:45.448070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929923+0000","last_change":"2023-10-16T12:33:23.983081+0000","last_active":"2023-10-16T12:33:25.929923+0000","last_peered":"2023-10-16T12:33:25.929923+0000","last_clean":"2023-10-16T12:33:25.929923+0000","last_became_active":"2023-10-16T12:33:23.979378+0000","last_became_peered":"2023-10-16T12:33:23.979378+0000","last_unstale":"2023-10-16T12:33:25.929923+0000","last_undegraded":"2023-10-16T12:33:25.929923+0000","last_fullsized":"2023-10-16T12:33:25.929923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928531+0000","last_change":"2023-10-16T12:33:24.612688+0000","last_active":"2023-10-16T12:33:25.928531+0000","last_peered":"2023-10-16T12:33:25.928531+0000","last_clean":"2023-10-16T12:33:25.928531+0000","last_became_active":"2023-10-16T12:33:24.612202+0000","last_became_peered":"2023-10-16T12:33:24.612202+0000","last_unstale":"2023-10-16T12:33:25.928531+0000","last_undegraded":"2023-10-16T12:33:25.928531+0000","last_fullsized":"2023-10-16T12:33:25.928531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929373+0000","last_change":"2023-10-16T12:33:23.983502+0000","last_active":"2023-10-16T12:33:25.929373+0000","last_peered":"2023-10-16T12:33:25.929373+0000","last_clean":"2023-10-16T12:33:25.929373+0000","last_became_active":"2023-10-16T12:33:23.983189+0000","last_became_peered":"2023-10-16T12:33:23.983189+0000","last_unstale":"2023-10-16T12:33:25.929373+0000","last_undegraded":"2023-10-16T12:33:25.929373+0000","last_fullsized":"2023-10-16T12:33:25.929373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.912908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929270+0000","last_change":"2023-10-16T12:33:23.984326+0000","last_active":"2023-10-16T12:33:25.929270+0000","last_peered":"2023-10-16T12:33:25.929270+0000","last_clean":"2023-10-16T12:33:25.929270+0000","last_became_active":"2023-10-16T12:33:23.984015+0000","last_became_peered":"2023-10-16T12:33:23.984015+0000","last_unstale":"2023-10-16T12:33:25.929270+0000","last_undegraded":"2023-10-16T12:33:25.929270+0000","last_fullsized":"2023-10-16T12:33:25.929270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929604+0000","last_change":"2023-10-16T12:33:24.615262+0000","last_active":"2023-10-16T12:33:25.929604+0000","last_peered":"2023-10-16T12:33:25.929604+0000","last_clean":"2023-10-16T12:33:25.929604+0000","last_became_active":"2023-10-16T12:33:24.610418+0000","last_became_peered":"2023-10-16T12:33:24.610418+0000","last_unstale":"2023-10-16T12:33:25.929604+0000","last_undegraded":"2023-10-16T12:33:25.929604+0000","last_fullsized":"2023-10-16T12:33:25.929604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585175+0000","last_change":"2023-10-16T12:33:23.976361+0000","last_active":"2023-10-16T12:33:25.585175+0000","last_peered":"2023-10-16T12:33:25.585175+0000","last_clean":"2023-10-16T12:33:25.585175+0000","last_became_active":"2023-10-16T12:33:23.975488+0000","last_became_peered":"2023-10-16T12:33:23.975488+0000","last_unstale":"2023-10-16T12:33:25.585175+0000","last_undegraded":"2023-10-16T12:33:25.585175+0000","last_fullsized":"2023-10-16T12:33:25.585175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584519+0000","last_change":"2023-10-16T12:33:24.603552+0000","last_active":"2023-10-16T12:33:25.584519+0000","last_peered":"2023-10-16T12:33:25.584519+0000","last_clean":"2023-10-16T12:33:25.584519+0000","last_became_active":"2023-10-16T12:33:24.602661+0000","last_became_peered":"2023-10-16T12:33:24.602661+0000","last_unstale":"2023-10-16T12:33:25.584519+0000","last_undegraded":"2023-10-16T12:33:25.584519+0000","last_fullsized":"2023-10-16T12:33:25.584519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929244+0000","last_change":"2023-10-16T12:33:23.983406+0000","last_active":"2023-10-16T12:33:25.929244+0000","last_peered":"2023-10-16T12:33:25.929244+0000","last_clean":"2023-10-16T12:33:25.929244+0000","last_became_active":"2023-10-16T12:33:23.982570+0000","last_became_peered":"2023-10-16T12:33:23.982570+0000","last_unstale":"2023-10-16T12:33:25.929244+0000","last_undegraded":"2023-10-16T12:33:25.929244+0000","last_fullsized":"2023-10-16T12:33:25.929244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:17.276818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591933+0000","last_change":"2023-10-16T12:33:24.616611+0000","last_active":"2023-10-16T12:33:25.591933+0000","last_peered":"2023-10-16T12:33:25.591933+0000","last_clean":"2023-10-16T12:33:25.591933+0000","last_became_active":"2023-10-16T12:33:24.616318+0000","last_became_peered":"2023-10-16T12:33:24.616318+0000","last_unstale":"2023-10-16T12:33:25.591933+0000","last_undegraded":"2023-10-16T12:33:25.591933+0000","last_fullsized":"2023-10-16T12:33:25.591933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591819+0000","last_change":"2023-10-16T12:33:23.982425+0000","last_active":"2023-10-16T12:33:25.591819+0000","last_peered":"2023-10-16T12:33:25.591819+0000","last_clean":"2023-10-16T12:33:25.591819+0000","last_became_active":"2023-10-16T12:33:23.982159+0000","last_became_peered":"2023-10-16T12:33:23.982159+0000","last_unstale":"2023-10-16T12:33:25.591819+0000","last_undegraded":"2023-10-16T12:33:25.591819+0000","last_fullsized":"2023-10-16T12:33:25.591819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:00.917480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929362+0000","last_change":"2023-10-16T12:33:24.607789+0000","last_active":"2023-10-16T12:33:25.929362+0000","last_peered":"2023-10-16T12:33:25.929362+0000","last_clean":"2023-10-16T12:33:25.929362+0000","last_became_active":"2023-10-16T12:33:24.606558+0000","last_became_peered":"2023-10-16T12:33:24.606558+0000","last_unstale":"2023-10-16T12:33:25.929362+0000","last_undegraded":"2023-10-16T12:33:25.929362+0000","last_fullsized":"2023-10-16T12:33:25.929362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584911+0000","last_change":"2023-10-16T12:33:23.994926+0000","last_active":"2023-10-16T12:33:25.584911+0000","last_peered":"2023-10-16T12:33:25.584911+0000","last_clean":"2023-10-16T12:33:25.584911+0000","last_became_active":"2023-10-16T12:33:23.994616+0000","last_became_peered":"2023-10-16T12:33:23.994616+0000","last_unstale":"2023-10-16T12:33:25.584911+0000","last_undegraded":"2023-10-16T12:33:25.584911+0000","last_fullsized":"2023-10-16T12:33:25.584911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.615708+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.615262+0000","last_became_peered":"2023-10-16T12:33:24.615262+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928417+0000","last_change":"2023-10-16T12:33:23.990728+0000","last_active":"2023-10-16T12:33:25.928417+0000","last_peered":"2023-10-16T12:33:25.928417+0000","last_clean":"2023-10-16T12:33:25.928417+0000","last_became_active":"2023-10-16T12:33:23.988939+0000","last_became_peered":"2023-10-16T12:33:23.988939+0000","last_unstale":"2023-10-16T12:33:25.928417+0000","last_undegraded":"2023-10-16T12:33:25.928417+0000","last_fullsized":"2023-10-16T12:33:25.928417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:58.091012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930079+0000","last_change":"2023-10-16T12:33:24.620766+0000","last_active":"2023-10-16T12:33:25.930079+0000","last_peered":"2023-10-16T12:33:25.930079+0000","last_clean":"2023-10-16T12:33:25.930079+0000","last_became_active":"2023-10-16T12:33:24.620453+0000","last_became_peered":"2023-10-16T12:33:24.620453+0000","last_unstale":"2023-10-16T12:33:25.930079+0000","last_undegraded":"2023-10-16T12:33:25.930079+0000","last_fullsized":"2023-10-16T12:33:25.930079+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928702+0000","last_change":"2023-10-16T12:33:23.968343+0000","last_active":"2023-10-16T12:33:25.928702+0000","last_peered":"2023-10-16T12:33:25.928702+0000","last_clean":"2023-10-16T12:33:25.928702+0000","last_became_active":"2023-10-16T12:33:23.967874+0000","last_became_peered":"2023-10-16T12:33:23.967874+0000","last_unstale":"2023-10-16T12:33:25.928702+0000","last_undegraded":"2023-10-16T12:33:25.928702+0000","last_fullsized":"2023-10-16T12:33:25.928702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927246+0000","last_change":"2023-10-16T12:33:24.615841+0000","last_active":"2023-10-16T12:33:25.927246+0000","last_peered":"2023-10-16T12:33:25.927246+0000","last_clean":"2023-10-16T12:33:25.927246+0000","last_became_active":"2023-10-16T12:33:24.615333+0000","last_became_peered":"2023-10-16T12:33:24.615333+0000","last_unstale":"2023-10-16T12:33:25.927246+0000","last_undegraded":"2023-10-16T12:33:25.927246+0000","last_fullsized":"2023-10-16T12:33:25.927246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:23.519029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590504+0000","last_change":"2023-10-16T12:33:23.984016+0000","last_active":"2023-10-16T12:33:25.590504+0000","last_peered":"2023-10-16T12:33:25.590504+0000","last_clean":"2023-10-16T12:33:25.590504+0000","last_became_active":"2023-10-16T12:33:23.983727+0000","last_became_peered":"2023-10-16T12:33:23.983727+0000","last_unstale":"2023-10-16T12:33:25.590504+0000","last_undegraded":"2023-10-16T12:33:25.590504+0000","last_fullsized":"2023-10-16T12:33:25.590504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930468+0000","last_change":"2023-10-16T12:33:24.623341+0000","last_active":"2023-10-16T12:33:25.930468+0000","last_peered":"2023-10-16T12:33:25.930468+0000","last_clean":"2023-10-16T12:33:25.930468+0000","last_became_active":"2023-10-16T12:33:24.622356+0000","last_became_peered":"2023-10-16T12:33:24.622356+0000","last_unstale":"2023-10-16T12:33:25.930468+0000","last_undegraded":"2023-10-16T12:33:25.930468+0000","last_fullsized":"2023-10-16T12:33:25.930468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927892+0000","last_change":"2023-10-16T12:33:23.982823+0000","last_active":"2023-10-16T12:33:25.927892+0000","last_peered":"2023-10-16T12:33:25.927892+0000","last_clean":"2023-10-16T12:33:25.927892+0000","last_became_active":"2023-10-16T12:33:23.981883+0000","last_became_peered":"2023-10-16T12:33:23.981883+0000","last_unstale":"2023-10-16T12:33:25.927892+0000","last_undegraded":"2023-10-16T12:33:25.927892+0000","last_fullsized":"2023-10-16T12:33:25.927892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:09.201117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927851+0000","last_change":"2023-10-16T12:33:24.617583+0000","last_active":"2023-10-16T12:33:25.927851+0000","last_peered":"2023-10-16T12:33:25.927851+0000","last_clean":"2023-10-16T12:33:25.927851+0000","last_became_active":"2023-10-16T12:33:24.616992+0000","last_became_peered":"2023-10-16T12:33:24.616992+0000","last_unstale":"2023-10-16T12:33:25.927851+0000","last_undegraded":"2023-10-16T12:33:25.927851+0000","last_fullsized":"2023-10-16T12:33:25.927851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:38.188085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.926869+0000","last_change":"2023-10-16T12:33:23.988300+0000","last_active":"2023-10-16T12:33:25.926869+0000","last_peered":"2023-10-16T12:33:25.926869+0000","last_clean":"2023-10-16T12:33:25.926869+0000","last_became_active":"2023-10-16T12:33:23.987831+0000","last_became_peered":"2023-10-16T12:33:23.987831+0000","last_unstale":"2023-10-16T12:33:25.926869+0000","last_undegraded":"2023-10-16T12:33:25.926869+0000","last_fullsized":"2023-10-16T12:33:25.926869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928930+0000","last_change":"2023-10-16T12:33:24.613800+0000","last_active":"2023-10-16T12:33:25.928930+0000","last_peered":"2023-10-16T12:33:25.928930+0000","last_clean":"2023-10-16T12:33:25.928930+0000","last_became_active":"2023-10-16T12:33:24.613268+0000","last_became_peered":"2023-10-16T12:33:24.613268+0000","last_unstale":"2023-10-16T12:33:25.928930+0000","last_undegraded":"2023-10-16T12:33:25.928930+0000","last_fullsized":"2023-10-16T12:33:25.928930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588350+0000","last_change":"2023-10-16T12:33:23.970854+0000","last_active":"2023-10-16T12:33:25.588350+0000","last_peered":"2023-10-16T12:33:25.588350+0000","last_clean":"2023-10-16T12:33:25.588350+0000","last_became_active":"2023-10-16T12:33:23.970073+0000","last_became_peered":"2023-10-16T12:33:23.970073+0000","last_unstale":"2023-10-16T12:33:25.588350+0000","last_undegraded":"2023-10-16T12:33:25.588350+0000","last_fullsized":"2023-10-16T12:33:25.588350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930633+0000","last_change":"2023-10-16T12:33:24.624669+0000","last_active":"2023-10-16T12:33:25.930633+0000","last_peered":"2023-10-16T12:33:25.930633+0000","last_clean":"2023-10-16T12:33:25.930633+0000","last_became_active":"2023-10-16T12:33:24.624315+0000","last_became_peered":"2023-10-16T12:33:24.624315+0000","last_unstale":"2023-10-16T12:33:25.930633+0000","last_undegraded":"2023-10-16T12:33:25.930633+0000","last_fullsized":"2023-10-16T12:33:25.930633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927431+0000","last_change":"2023-10-16T12:33:23.980566+0000","last_active":"2023-10-16T12:33:25.927431+0000","last_peered":"2023-10-16T12:33:25.927431+0000","last_clean":"2023-10-16T12:33:25.927431+0000","last_became_active":"2023-10-16T12:33:23.980200+0000","last_became_peered":"2023-10-16T12:33:23.980200+0000","last_unstale":"2023-10-16T12:33:25.927431+0000","last_undegraded":"2023-10-16T12:33:25.927431+0000","last_fullsized":"2023-10-16T12:33:25.927431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.931065+0000","last_change":"2023-10-16T12:33:24.624902+0000","last_active":"2023-10-16T12:33:25.931065+0000","last_peered":"2023-10-16T12:33:25.931065+0000","last_clean":"2023-10-16T12:33:25.931065+0000","last_became_active":"2023-10-16T12:33:24.624646+0000","last_became_peered":"2023-10-16T12:33:24.624646+0000","last_unstale":"2023-10-16T12:33:25.931065+0000","last_undegraded":"2023-10-16T12:33:25.931065+0000","last_fullsized":"2023-10-16T12:33:25.931065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:21.560247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585130+0000","last_change":"2023-10-16T12:33:23.984004+0000","last_active":"2023-10-16T12:33:25.585130+0000","last_peered":"2023-10-16T12:33:25.585130+0000","last_clean":"2023-10-16T12:33:25.585130+0000","last_became_active":"2023-10-16T12:33:23.982385+0000","last_became_peered":"2023-10-16T12:33:23.982385+0000","last_unstale":"2023-10-16T12:33:25.585130+0000","last_undegraded":"2023-10-16T12:33:25.585130+0000","last_fullsized":"2023-10-16T12:33:25.585130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586032+0000","last_change":"2023-10-16T12:33:24.611253+0000","last_active":"2023-10-16T12:33:25.586032+0000","last_peered":"2023-10-16T12:33:25.586032+0000","last_clean":"2023-10-16T12:33:25.586032+0000","last_became_active":"2023-10-16T12:33:24.610712+0000","last_became_peered":"2023-10-16T12:33:24.610712+0000","last_unstale":"2023-10-16T12:33:25.586032+0000","last_undegraded":"2023-10-16T12:33:25.586032+0000","last_fullsized":"2023-10-16T12:33:25.586032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927465+0000","last_change":"2023-10-16T12:33:23.985969+0000","last_active":"2023-10-16T12:33:25.927465+0000","last_peered":"2023-10-16T12:33:25.927465+0000","last_clean":"2023-10-16T12:33:25.927465+0000","last_became_active":"2023-10-16T12:33:23.985467+0000","last_became_peered":"2023-10-16T12:33:23.985467+0000","last_unstale":"2023-10-16T12:33:25.927465+0000","last_undegraded":"2023-10-16T12:33:25.927465+0000","last_fullsized":"2023-10-16T12:33:25.927465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930068+0000","last_change":"2023-10-16T12:33:24.627563+0000","last_active":"2023-10-16T12:33:25.930068+0000","last_peered":"2023-10-16T12:33:25.930068+0000","last_clean":"2023-10-16T12:33:25.930068+0000","last_became_active":"2023-10-16T12:33:24.627241+0000","last_became_peered":"2023-10-16T12:33:24.627241+0000","last_unstale":"2023-10-16T12:33:25.930068+0000","last_undegraded":"2023-10-16T12:33:25.930068+0000","last_fullsized":"2023-10-16T12:33:25.930068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-16T12:33:24.598233+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:33:24.598233+0000","last_peered":"2023-10-16T12:33:24.598233+0000","last_clean":"2023-10-16T12:33:24.598233+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:33:24.598233+0000","last_undegraded":"2023-10-16T12:33:24.598233+0000","last_fullsized":"2023-10-16T12:33:24.598233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585104+0000","last_change":"2023-10-16T12:33:23.972294+0000","last_active":"2023-10-16T12:33:25.585104+0000","last_peered":"2023-10-16T12:33:25.585104+0000","last_clean":"2023-10-16T12:33:25.585104+0000","last_became_active":"2023-10-16T12:33:23.971886+0000","last_became_peered":"2023-10-16T12:33:23.971886+0000","last_unstale":"2023-10-16T12:33:25.585104+0000","last_undegraded":"2023-10-16T12:33:25.585104+0000","last_fullsized":"2023-10-16T12:33:25.585104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.874107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929522+0000","last_change":"2023-10-16T12:33:24.604564+0000","last_active":"2023-10-16T12:33:25.929522+0000","last_peered":"2023-10-16T12:33:25.929522+0000","last_clean":"2023-10-16T12:33:25.929522+0000","last_became_active":"2023-10-16T12:33:24.604240+0000","last_became_peered":"2023-10-16T12:33:24.604240+0000","last_unstale":"2023-10-16T12:33:25.929522+0000","last_undegraded":"2023-10-16T12:33:25.929522+0000","last_fullsized":"2023-10-16T12:33:25.929522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927720+0000","last_change":"2023-10-16T12:33:23.988943+0000","last_active":"2023-10-16T12:33:25.927720+0000","last_peered":"2023-10-16T12:33:25.927720+0000","last_clean":"2023-10-16T12:33:25.927720+0000","last_became_active":"2023-10-16T12:33:23.988631+0000","last_became_peered":"2023-10-16T12:33:23.988631+0000","last_unstale":"2023-10-16T12:33:25.927720+0000","last_undegraded":"2023-10-16T12:33:25.927720+0000","last_fullsized":"2023-10-16T12:33:25.927720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927155+0000","last_change":"2023-10-16T12:33:24.615033+0000","last_active":"2023-10-16T12:33:25.927155+0000","last_peered":"2023-10-16T12:33:25.927155+0000","last_clean":"2023-10-16T12:33:25.927155+0000","last_became_active":"2023-10-16T12:33:24.614433+0000","last_became_peered":"2023-10-16T12:33:24.614433+0000","last_unstale":"2023-10-16T12:33:25.927155+0000","last_undegraded":"2023-10-16T12:33:25.927155+0000","last_fullsized":"2023-10-16T12:33:25.927155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:38.208712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928316+0000","last_change":"2023-10-16T12:33:23.978410+0000","last_active":"2023-10-16T12:33:25.928316+0000","last_peered":"2023-10-16T12:33:25.928316+0000","last_clean":"2023-10-16T12:33:25.928316+0000","last_became_active":"2023-10-16T12:33:23.978050+0000","last_became_peered":"2023-10-16T12:33:23.978050+0000","last_unstale":"2023-10-16T12:33:25.928316+0000","last_undegraded":"2023-10-16T12:33:25.928316+0000","last_fullsized":"2023-10-16T12:33:25.928316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584693+0000","last_change":"2023-10-16T12:33:24.600057+0000","last_active":"2023-10-16T12:33:25.584693+0000","last_peered":"2023-10-16T12:33:25.584693+0000","last_clean":"2023-10-16T12:33:25.584693+0000","last_became_active":"2023-10-16T12:33:24.599294+0000","last_became_peered":"2023-10-16T12:33:24.599294+0000","last_unstale":"2023-10-16T12:33:25.584693+0000","last_undegraded":"2023-10-16T12:33:25.584693+0000","last_fullsized":"2023-10-16T12:33:25.584693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590705+0000","last_change":"2023-10-16T12:33:23.990485+0000","last_active":"2023-10-16T12:33:25.590705+0000","last_peered":"2023-10-16T12:33:25.590705+0000","last_clean":"2023-10-16T12:33:25.590705+0000","last_became_active":"2023-10-16T12:33:23.989836+0000","last_became_peered":"2023-10-16T12:33:23.989836+0000","last_unstale":"2023-10-16T12:33:25.590705+0000","last_undegraded":"2023-10-16T12:33:25.590705+0000","last_fullsized":"2023-10-16T12:33:25.590705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.589289+0000","last_change":"2023-10-16T12:33:24.619525+0000","last_active":"2023-10-16T12:33:25.589289+0000","last_peered":"2023-10-16T12:33:25.589289+0000","last_clean":"2023-10-16T12:33:25.589289+0000","last_became_active":"2023-10-16T12:33:24.619242+0000","last_became_peered":"2023-10-16T12:33:24.619242+0000","last_unstale":"2023-10-16T12:33:25.589289+0000","last_undegraded":"2023-10-16T12:33:25.589289+0000","last_fullsized":"2023-10-16T12:33:25.589289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:12.969746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584815+0000","last_change":"2023-10-16T12:33:23.976427+0000","last_active":"2023-10-16T12:33:25.584815+0000","last_peered":"2023-10-16T12:33:25.584815+0000","last_clean":"2023-10-16T12:33:25.584815+0000","last_became_active":"2023-10-16T12:33:23.975234+0000","last_became_peered":"2023-10-16T12:33:23.975234+0000","last_unstale":"2023-10-16T12:33:25.584815+0000","last_undegraded":"2023-10-16T12:33:25.584815+0000","last_fullsized":"2023-10-16T12:33:25.584815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590492+0000","last_change":"2023-10-16T12:33:24.614482+0000","last_active":"2023-10-16T12:33:25.590492+0000","last_peered":"2023-10-16T12:33:25.590492+0000","last_clean":"2023-10-16T12:33:25.590492+0000","last_became_active":"2023-10-16T12:33:24.614144+0000","last_became_peered":"2023-10-16T12:33:24.614144+0000","last_unstale":"2023-10-16T12:33:25.590492+0000","last_undegraded":"2023-10-16T12:33:25.590492+0000","last_fullsized":"2023-10-16T12:33:25.590492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930752+0000","last_change":"2023-10-16T12:33:23.986466+0000","last_active":"2023-10-16T12:33:25.930752+0000","last_peered":"2023-10-16T12:33:25.930752+0000","last_clean":"2023-10-16T12:33:25.930752+0000","last_became_active":"2023-10-16T12:33:23.985508+0000","last_became_peered":"2023-10-16T12:33:23.985508+0000","last_unstale":"2023-10-16T12:33:25.930752+0000","last_undegraded":"2023-10-16T12:33:25.930752+0000","last_fullsized":"2023-10-16T12:33:25.930752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:08.057398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584833+0000","last_change":"2023-10-16T12:33:24.605662+0000","last_active":"2023-10-16T12:33:25.584833+0000","last_peered":"2023-10-16T12:33:25.584833+0000","last_clean":"2023-10-16T12:33:25.584833+0000","last_became_active":"2023-10-16T12:33:24.605319+0000","last_became_peered":"2023-10-16T12:33:24.605319+0000","last_unstale":"2023-10-16T12:33:25.584833+0000","last_undegraded":"2023-10-16T12:33:25.584833+0000","last_fullsized":"2023-10-16T12:33:25.584833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:26.775691+0000","last_change":"2023-10-16T12:33:23.986752+0000","last_active":"2023-10-16T12:33:26.775691+0000","last_peered":"2023-10-16T12:33:26.775691+0000","last_clean":"2023-10-16T12:33:26.775691+0000","last_became_active":"2023-10-16T12:33:23.986189+0000","last_became_peered":"2023-10-16T12:33:23.986189+0000","last_unstale":"2023-10-16T12:33:26.775691+0000","last_undegraded":"2023-10-16T12:33:26.775691+0000","last_fullsized":"2023-10-16T12:33:26.775691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930740+0000","last_change":"2023-10-16T12:33:24.616668+0000","last_active":"2023-10-16T12:33:25.930740+0000","last_peered":"2023-10-16T12:33:25.930740+0000","last_clean":"2023-10-16T12:33:25.930740+0000","last_became_active":"2023-10-16T12:33:24.616308+0000","last_became_peered":"2023-10-16T12:33:24.616308+0000","last_unstale":"2023-10-16T12:33:25.930740+0000","last_undegraded":"2023-10-16T12:33:25.930740+0000","last_fullsized":"2023-10-16T12:33:25.930740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929376+0000","last_change":"2023-10-16T12:33:23.991338+0000","last_active":"2023-10-16T12:33:25.929376+0000","last_peered":"2023-10-16T12:33:25.929376+0000","last_clean":"2023-10-16T12:33:25.929376+0000","last_became_active":"2023-10-16T12:33:23.986162+0000","last_became_peered":"2023-10-16T12:33:23.986162+0000","last_unstale":"2023-10-16T12:33:25.929376+0000","last_undegraded":"2023-10-16T12:33:25.929376+0000","last_fullsized":"2023-10-16T12:33:25.929376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:32.667813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928205+0000","last_change":"2023-10-16T12:33:24.612178+0000","last_active":"2023-10-16T12:33:25.928205+0000","last_peered":"2023-10-16T12:33:25.928205+0000","last_clean":"2023-10-16T12:33:25.928205+0000","last_became_active":"2023-10-16T12:33:24.608876+0000","last_became_peered":"2023-10-16T12:33:24.608876+0000","last_unstale":"2023-10-16T12:33:25.928205+0000","last_undegraded":"2023-10-16T12:33:25.928205+0000","last_fullsized":"2023-10-16T12:33:25.928205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927671+0000","last_change":"2023-10-16T12:33:23.983928+0000","last_active":"2023-10-16T12:33:25.927671+0000","last_peered":"2023-10-16T12:33:25.927671+0000","last_clean":"2023-10-16T12:33:25.927671+0000","last_became_active":"2023-10-16T12:33:23.983266+0000","last_became_peered":"2023-10-16T12:33:23.983266+0000","last_unstale":"2023-10-16T12:33:25.927671+0000","last_undegraded":"2023-10-16T12:33:25.927671+0000","last_fullsized":"2023-10-16T12:33:25.927671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.605248+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.604915+0000","last_became_peered":"2023-10-16T12:33:24.604915+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584607+0000","last_change":"2023-10-16T12:33:23.987636+0000","last_active":"2023-10-16T12:33:25.584607+0000","last_peered":"2023-10-16T12:33:25.584607+0000","last_clean":"2023-10-16T12:33:25.584607+0000","last_became_active":"2023-10-16T12:33:23.987422+0000","last_became_peered":"2023-10-16T12:33:23.987422+0000","last_unstale":"2023-10-16T12:33:25.584607+0000","last_undegraded":"2023-10-16T12:33:25.584607+0000","last_fullsized":"2023-10-16T12:33:25.584607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930014+0000","last_change":"2023-10-16T12:33:24.607092+0000","last_active":"2023-10-16T12:33:25.930014+0000","last_peered":"2023-10-16T12:33:25.930014+0000","last_clean":"2023-10-16T12:33:25.930014+0000","last_became_active":"2023-10-16T12:33:24.606816+0000","last_became_peered":"2023-10-16T12:33:24.606816+0000","last_unstale":"2023-10-16T12:33:25.930014+0000","last_undegraded":"2023-10-16T12:33:25.930014+0000","last_fullsized":"2023-10-16T12:33:25.930014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584703+0000","last_change":"2023-10-16T12:33:23.982671+0000","last_active":"2023-10-16T12:33:25.584703+0000","last_peered":"2023-10-16T12:33:25.584703+0000","last_clean":"2023-10-16T12:33:25.584703+0000","last_became_active":"2023-10-16T12:33:23.982314+0000","last_became_peered":"2023-10-16T12:33:23.982314+0000","last_unstale":"2023-10-16T12:33:25.584703+0000","last_undegraded":"2023-10-16T12:33:25.584703+0000","last_fullsized":"2023-10-16T12:33:25.584703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930259+0000","last_change":"2023-10-16T12:33:24.614159+0000","last_active":"2023-10-16T12:33:25.930259+0000","last_peered":"2023-10-16T12:33:25.930259+0000","last_clean":"2023-10-16T12:33:25.930259+0000","last_became_active":"2023-10-16T12:33:24.613857+0000","last_became_peered":"2023-10-16T12:33:24.613857+0000","last_unstale":"2023-10-16T12:33:25.930259+0000","last_undegraded":"2023-10-16T12:33:25.930259+0000","last_fullsized":"2023-10-16T12:33:25.930259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930386+0000","last_change":"2023-10-16T12:33:23.987798+0000","last_active":"2023-10-16T12:33:25.930386+0000","last_peered":"2023-10-16T12:33:25.930386+0000","last_clean":"2023-10-16T12:33:25.930386+0000","last_became_active":"2023-10-16T12:33:23.987460+0000","last_became_peered":"2023-10-16T12:33:23.987460+0000","last_unstale":"2023-10-16T12:33:25.930386+0000","last_undegraded":"2023-10-16T12:33:25.930386+0000","last_fullsized":"2023-10-16T12:33:25.930386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929124+0000","last_change":"2023-10-16T12:33:24.615457+0000","last_active":"2023-10-16T12:33:25.929124+0000","last_peered":"2023-10-16T12:33:25.929124+0000","last_clean":"2023-10-16T12:33:25.929124+0000","last_became_active":"2023-10-16T12:33:24.615166+0000","last_became_peered":"2023-10-16T12:33:24.615166+0000","last_unstale":"2023-10-16T12:33:25.929124+0000","last_undegraded":"2023-10-16T12:33:25.929124+0000","last_fullsized":"2023-10-16T12:33:25.929124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:24.606678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586074+0000","last_change":"2023-10-16T12:33:24.604607+0000","last_active":"2023-10-16T12:33:25.586074+0000","last_peered":"2023-10-16T12:33:25.586074+0000","last_clean":"2023-10-16T12:33:25.586074+0000","last_became_active":"2023-10-16T12:33:24.604183+0000","last_became_peered":"2023-10-16T12:33:24.604183+0000","last_unstale":"2023-10-16T12:33:25.586074+0000","last_undegraded":"2023-10-16T12:33:25.586074+0000","last_fullsized":"2023-10-16T12:33:25.586074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929969+0000","last_change":"2023-10-16T12:33:23.990373+0000","last_active":"2023-10-16T12:33:25.929969+0000","last_peered":"2023-10-16T12:33:25.929969+0000","last_clean":"2023-10-16T12:33:25.929969+0000","last_became_active":"2023-10-16T12:33:23.990049+0000","last_became_peered":"2023-10-16T12:33:23.990049+0000","last_unstale":"2023-10-16T12:33:25.929969+0000","last_undegraded":"2023-10-16T12:33:25.929969+0000","last_fullsized":"2023-10-16T12:33:25.929969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585658+0000","last_change":"2023-10-16T12:33:24.612037+0000","last_active":"2023-10-16T12:33:25.585658+0000","last_peered":"2023-10-16T12:33:25.585658+0000","last_clean":"2023-10-16T12:33:25.585658+0000","last_became_active":"2023-10-16T12:33:24.611609+0000","last_became_peered":"2023-10-16T12:33:24.611609+0000","last_unstale":"2023-10-16T12:33:25.585658+0000","last_undegraded":"2023-10-16T12:33:25.585658+0000","last_fullsized":"2023-10-16T12:33:25.585658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930569+0000","last_change":"2023-10-16T12:33:23.985742+0000","last_active":"2023-10-16T12:33:25.930569+0000","last_peered":"2023-10-16T12:33:25.930569+0000","last_clean":"2023-10-16T12:33:25.930569+0000","last_became_active":"2023-10-16T12:33:23.985053+0000","last_became_peered":"2023-10-16T12:33:23.985053+0000","last_unstale":"2023-10-16T12:33:25.930569+0000","last_undegraded":"2023-10-16T12:33:25.930569+0000","last_fullsized":"2023-10-16T12:33:25.930569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:43.191029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927970+0000","last_change":"2023-10-16T12:33:24.610651+0000","last_active":"2023-10-16T12:33:25.927970+0000","last_peered":"2023-10-16T12:33:25.927970+0000","last_clean":"2023-10-16T12:33:25.927970+0000","last_became_active":"2023-10-16T12:33:24.605512+0000","last_became_peered":"2023-10-16T12:33:24.605512+0000","last_unstale":"2023-10-16T12:33:25.927970+0000","last_undegraded":"2023-10-16T12:33:25.927970+0000","last_fullsized":"2023-10-16T12:33:25.927970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930559+0000","last_change":"2023-10-16T12:33:23.981596+0000","last_active":"2023-10-16T12:33:25.930559+0000","last_peered":"2023-10-16T12:33:25.930559+0000","last_clean":"2023-10-16T12:33:25.930559+0000","last_became_active":"2023-10-16T12:33:23.981314+0000","last_became_peered":"2023-10-16T12:33:23.981314+0000","last_unstale":"2023-10-16T12:33:25.930559+0000","last_undegraded":"2023-10-16T12:33:25.930559+0000","last_fullsized":"2023-10-16T12:33:25.930559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930671+0000","last_change":"2023-10-16T12:33:24.613513+0000","last_active":"2023-10-16T12:33:25.930671+0000","last_peered":"2023-10-16T12:33:25.930671+0000","last_clean":"2023-10-16T12:33:25.930671+0000","last_became_active":"2023-10-16T12:33:24.613158+0000","last_became_peered":"2023-10-16T12:33:24.613158+0000","last_unstale":"2023-10-16T12:33:25.930671+0000","last_undegraded":"2023-10-16T12:33:25.930671+0000","last_fullsized":"2023-10-16T12:33:25.930671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929580+0000","last_change":"2023-10-16T12:33:23.978618+0000","last_active":"2023-10-16T12:33:25.929580+0000","last_peered":"2023-10-16T12:33:25.929580+0000","last_clean":"2023-10-16T12:33:25.929580+0000","last_became_active":"2023-10-16T12:33:23.978325+0000","last_became_peered":"2023-10-16T12:33:23.978325+0000","last_unstale":"2023-10-16T12:33:25.929580+0000","last_undegraded":"2023-10-16T12:33:25.929580+0000","last_fullsized":"2023-10-16T12:33:25.929580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2130000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1139999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":831488,"data_stored":669870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.758}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:33:35.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:33:36.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.186+0000 7fc2864df640 1 Processor -- start 2023-10-16T12:33:36.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.186+0000 7fc2864df640 1 -- start start 2023-10-16T12:33:36.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2864df640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2800ad970 0x7fc2800cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2864df640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2800ccfb0 0x7fc2800cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2864df640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28015b9d0 con 0x7fc28006c040 2023-10-16T12:33:36.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2864df640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28015bcb0 con 0x7fc2800ad970 2023-10-16T12:33:36.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2864df640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28015bf90 con 0x7fc2800ccfb0 2023-10-16T12:33:36.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2854dd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc28006c040 0x7fc28006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54294/0 (socket says 172.21.15.60:54294) 2023-10-16T12:33:36.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc2854dd640 1 -- 172.21.15.60:0/2591609966 learned_addr learned my addr 172.21.15.60:0/2591609966 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:36.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.188+0000 7fc284cdc640 1 --2- 172.21.15.60:0/2591609966 >> v2:172.21.15.60:3300/0 conn(0x7fc2800ad970 0x7fc2800cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.189+0000 7fc285cde640 1 --2- 172.21.15.60:0/2591609966 >> v2:172.21.15.144:3300/0 conn(0x7fc2800ccfb0 0x7fc2800cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.189+0000 7fc284cdc640 1 -- 172.21.15.60:0/2591609966 >> v2:172.21.15.144:3300/0 conn(0x7fc2800ccfb0 msgr2=0x7fc2800cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:36.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.189+0000 7fc284cdc640 1 --2- 172.21.15.60:0/2591609966 >> v2:172.21.15.144:3300/0 conn(0x7fc2800ccfb0 0x7fc2800cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.189+0000 7fc284cdc640 1 -- 172.21.15.60:0/2591609966 >> v1:172.21.15.60:6789/0 conn(0x7fc28006c040 legacy=0x7fc28006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.189+0000 7fc284cdc640 1 -- 172.21.15.60:0/2591609966 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2800cc270 con 0x7fc2800ad970 2023-10-16T12:33:36.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.190+0000 7fc284cdc640 1 --2- 172.21.15.60:0/2591609966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ad970 0x7fc2800cca70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2700097a0 tx=0x7fc27002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2287e3bb04d12bc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:36.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.190+0000 7fc277fff640 1 -- 172.21.15.60:0/2591609966 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270031830 con 0x7fc2800ad970 2023-10-16T12:33:36.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.190+0000 7fc277fff640 1 -- 172.21.15.60:0/2591609966 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc270031990 con 0x7fc2800ad970 2023-10-16T12:33:36.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.190+0000 7fc277fff640 1 -- 172.21.15.60:0/2591609966 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270031d20 con 0x7fc2800ad970 2023-10-16T12:33:36.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 -- 172.21.15.60:0/2591609966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ad970 msgr2=0x7fc2800cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 --2- 172.21.15.60:0/2591609966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ad970 0x7fc2800cca70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2700097a0 tx=0x7fc27002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 -- 172.21.15.60:0/2591609966 shutdown_connections 2023-10-16T12:33:36.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 --2- 172.21.15.60:0/2591609966 >> v2:172.21.15.144:3300/0 conn(0x7fc2800ccfb0 0x7fc2800cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 --2- 172.21.15.60:0/2591609966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ad970 0x7fc2800cca70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 -- 172.21.15.60:0/2591609966 >> 172.21.15.60:0/2591609966 conn(0x7fc2800ce6a0 msgr2=0x7fc2800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:36.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 -- 172.21.15.60:0/2591609966 shutdown_connections 2023-10-16T12:33:36.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.191+0000 7fc2864df640 1 -- 172.21.15.60:0/2591609966 wait complete. 2023-10-16T12:33:36.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.193+0000 7fc2864df640 1 Processor -- start 2023-10-16T12:33:36.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.193+0000 7fc2864df640 1 -- start start 2023-10-16T12:33:36.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.193+0000 7fc2864df640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc28006c040 0x7fc2801f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc2864df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2800ad970 0x7fc2801f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc2864df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 0x7fc280152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc2864df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2801f6bb0 con 0x7fc2800ccfb0 2023-10-16T12:33:36.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc2864df640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2801f2a90 con 0x7fc28006c040 2023-10-16T12:33:36.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc2864df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2801f2d70 con 0x7fc2800ad970 2023-10-16T12:33:36.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 0x7fc280152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 0x7fc280152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58146/0 (socket says 172.21.15.60:58146) 2023-10-16T12:33:36.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 -- 172.21.15.60:0/110875792 learned_addr learned my addr 172.21.15.60:0/110875792 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:36.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 -- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2800ad970 msgr2=0x7fc2801f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2800ad970 0x7fc2801f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 -- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc28006c040 msgr2=0x7fc2801f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc28006c040 0x7fc2801f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 -- 172.21.15.60:0/110875792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270009400 con 0x7fc2800ccfb0 2023-10-16T12:33:36.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.194+0000 7fc285cde640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 0x7fc280152290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc278007e60 tx=0x7fc27800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:36.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.195+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc278014420 con 0x7fc2800ccfb0 2023-10-16T12:33:36.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.195+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2780145a0 con 0x7fc2800ccfb0 2023-10-16T12:33:36.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.195+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc278014950 con 0x7fc2800ccfb0 2023-10-16T12:33:36.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.195+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2801f3050 con 0x7fc2800ccfb0 2023-10-16T12:33:36.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.195+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2800b0df0 con 0x7fc2800ccfb0 2023-10-16T12:33:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.196+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28006b330 con 0x7fc2800ccfb0 2023-10-16T12:33:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.197+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc278014ab0 con 0x7fc2800ccfb0 2023-10-16T12:33:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.198+0000 7fc275ffb640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc26406ae70 0x7fc26406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.198+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc278089fa0 con 0x7fc2800ccfb0 2023-10-16T12:33:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.199+0000 7fc2854dd640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc26406ae70 0x7fc26406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.200+0000 7fc2854dd640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc26406ae70 0x7fc26406d330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0015f0 tx=0x7fc27c007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.202+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc278055040 con 0x7fc2800ccfb0 2023-10-16T12:33:36.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.389+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2800cc550 con 0x7fc26406ae70 2023-10-16T12:33:36.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.411+0000 7fc275ffb640 1 -- 172.21.15.60:0/110875792 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264381 (secure 0 0 0) 0x7fc27c0058b0 con 0x7fc26406ae70 2023-10-16T12:33:36.415 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:33:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc26406ae70 msgr2=0x7fc26406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc26406ae70 0x7fc26406d330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0015f0 tx=0x7fc27c007a40 comp rx=0 tx=0).stop 2023-10-16T12:33:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 msgr2=0x7fc280152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 0x7fc280152290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc278007e60 tx=0x7fc27800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 shutdown_connections 2023-10-16T12:33:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc26406ae70 0x7fc26406d330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2800ad970 0x7fc2801f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2800ccfb0 0x7fc280152290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 --2- 172.21.15.60:0/110875792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc28006c040 0x7fc2801f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.417+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 >> 172.21.15.60:0/110875792 conn(0x7fc2800ce6a0 msgr2=0x7fc2800cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:36.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.418+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 shutdown_connections 2023-10-16T12:33:36.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.418+0000 7fc2864df640 1 -- 172.21.15.60:0/110875792 wait complete. 2023-10-16T12:33:36.440 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:33:35.735333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236216,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738504,"statfs":{"total":773094113280,"available":772852228096,"internally_reserved":0,"allocated":3055616,"data_stored":1816468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":81,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.154574"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930864+0000","last_change":"2023-10-16T12:33:24.620784+0000","last_active":"2023-10-16T12:33:25.930864+0000","last_peered":"2023-10-16T12:33:25.930864+0000","last_clean":"2023-10-16T12:33:25.930864+0000","last_became_active":"2023-10-16T12:33:24.620205+0000","last_became_peered":"2023-10-16T12:33:24.620205+0000","last_unstale":"2023-10-16T12:33:25.930864+0000","last_undegraded":"2023-10-16T12:33:25.930864+0000","last_fullsized":"2023-10-16T12:33:25.930864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585097+0000","last_change":"2023-10-16T12:33:24.608395+0000","last_active":"2023-10-16T12:33:25.585097+0000","last_peered":"2023-10-16T12:33:25.585097+0000","last_clean":"2023-10-16T12:33:25.585097+0000","last_became_active":"2023-10-16T12:33:24.608062+0000","last_became_peered":"2023-10-16T12:33:24.608062+0000","last_unstale":"2023-10-16T12:33:25.585097+0000","last_undegraded":"2023-10-16T12:33:25.585097+0000","last_fullsized":"2023-10-16T12:33:25.585097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928883+0000","last_change":"2023-10-16T12:33:24.607434+0000","last_active":"2023-10-16T12:33:25.928883+0000","last_peered":"2023-10-16T12:33:25.928883+0000","last_clean":"2023-10-16T12:33:25.928883+0000","last_became_active":"2023-10-16T12:33:24.607119+0000","last_became_peered":"2023-10-16T12:33:24.607119+0000","last_unstale":"2023-10-16T12:33:25.928883+0000","last_undegraded":"2023-10-16T12:33:25.928883+0000","last_fullsized":"2023-10-16T12:33:25.928883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.592697+0000","last_change":"2023-10-16T12:33:24.611057+0000","last_active":"2023-10-16T12:33:25.592697+0000","last_peered":"2023-10-16T12:33:25.592697+0000","last_clean":"2023-10-16T12:33:25.592697+0000","last_became_active":"2023-10-16T12:33:24.610357+0000","last_became_peered":"2023-10-16T12:33:24.610357+0000","last_unstale":"2023-10-16T12:33:25.592697+0000","last_undegraded":"2023-10-16T12:33:25.592697+0000","last_fullsized":"2023-10-16T12:33:25.592697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:03.927987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930042+0000","last_change":"2023-10-16T12:33:24.617198+0000","last_active":"2023-10-16T12:33:25.930042+0000","last_peered":"2023-10-16T12:33:25.930042+0000","last_clean":"2023-10-16T12:33:25.930042+0000","last_became_active":"2023-10-16T12:33:24.616710+0000","last_became_peered":"2023-10-16T12:33:24.616710+0000","last_unstale":"2023-10-16T12:33:25.930042+0000","last_undegraded":"2023-10-16T12:33:25.930042+0000","last_fullsized":"2023-10-16T12:33:25.930042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585757+0000","last_change":"2023-10-16T12:33:24.608373+0000","last_active":"2023-10-16T12:33:25.585757+0000","last_peered":"2023-10-16T12:33:25.585757+0000","last_clean":"2023-10-16T12:33:25.585757+0000","last_became_active":"2023-10-16T12:33:24.608083+0000","last_became_peered":"2023-10-16T12:33:24.608083+0000","last_unstale":"2023-10-16T12:33:25.585757+0000","last_undegraded":"2023-10-16T12:33:25.585757+0000","last_fullsized":"2023-10-16T12:33:25.585757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928757+0000","last_change":"2023-10-16T12:33:24.620178+0000","last_active":"2023-10-16T12:33:25.928757+0000","last_peered":"2023-10-16T12:33:25.928757+0000","last_clean":"2023-10-16T12:33:25.928757+0000","last_became_active":"2023-10-16T12:33:24.619871+0000","last_became_peered":"2023-10-16T12:33:24.619871+0000","last_unstale":"2023-10-16T12:33:25.928757+0000","last_undegraded":"2023-10-16T12:33:25.928757+0000","last_fullsized":"2023-10-16T12:33:25.928757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929201+0000","last_change":"2023-10-16T12:33:24.610760+0000","last_active":"2023-10-16T12:33:25.929201+0000","last_peered":"2023-10-16T12:33:25.929201+0000","last_clean":"2023-10-16T12:33:25.929201+0000","last_became_active":"2023-10-16T12:33:24.609947+0000","last_became_peered":"2023-10-16T12:33:24.609947+0000","last_unstale":"2023-10-16T12:33:25.929201+0000","last_undegraded":"2023-10-16T12:33:25.929201+0000","last_fullsized":"2023-10-16T12:33:25.929201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930940+0000","last_change":"2023-10-16T12:33:24.626800+0000","last_active":"2023-10-16T12:33:25.930940+0000","last_peered":"2023-10-16T12:33:25.930940+0000","last_clean":"2023-10-16T12:33:25.930940+0000","last_became_active":"2023-10-16T12:33:24.626566+0000","last_became_peered":"2023-10-16T12:33:24.626566+0000","last_unstale":"2023-10-16T12:33:25.930940+0000","last_undegraded":"2023-10-16T12:33:25.930940+0000","last_fullsized":"2023-10-16T12:33:25.930940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:08.131810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928952+0000","last_change":"2023-10-16T12:33:24.610841+0000","last_active":"2023-10-16T12:33:25.928952+0000","last_peered":"2023-10-16T12:33:25.928952+0000","last_clean":"2023-10-16T12:33:25.928952+0000","last_became_active":"2023-10-16T12:33:24.610533+0000","last_became_peered":"2023-10-16T12:33:24.610533+0000","last_unstale":"2023-10-16T12:33:25.928952+0000","last_undegraded":"2023-10-16T12:33:25.928952+0000","last_fullsized":"2023-10-16T12:33:25.928952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:39.121403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928420+0000","last_change":"2023-10-16T12:33:24.609014+0000","last_active":"2023-10-16T12:33:25.928420+0000","last_peered":"2023-10-16T12:33:25.928420+0000","last_clean":"2023-10-16T12:33:25.928420+0000","last_became_active":"2023-10-16T12:33:24.607622+0000","last_became_peered":"2023-10-16T12:33:24.607622+0000","last_unstale":"2023-10-16T12:33:25.928420+0000","last_undegraded":"2023-10-16T12:33:25.928420+0000","last_fullsized":"2023-10-16T12:33:25.928420+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:00.436524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588652+0000","last_change":"2023-10-16T12:33:24.609916+0000","last_active":"2023-10-16T12:33:25.588652+0000","last_peered":"2023-10-16T12:33:25.588652+0000","last_clean":"2023-10-16T12:33:25.588652+0000","last_became_active":"2023-10-16T12:33:24.609319+0000","last_became_peered":"2023-10-16T12:33:24.609319+0000","last_unstale":"2023-10-16T12:33:25.588652+0000","last_undegraded":"2023-10-16T12:33:25.588652+0000","last_fullsized":"2023-10-16T12:33:25.588652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:04.453258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929179+0000","last_change":"2023-10-16T12:33:24.622611+0000","last_active":"2023-10-16T12:33:25.929179+0000","last_peered":"2023-10-16T12:33:25.929179+0000","last_clean":"2023-10-16T12:33:25.929179+0000","last_became_active":"2023-10-16T12:33:24.622083+0000","last_became_peered":"2023-10-16T12:33:24.622083+0000","last_unstale":"2023-10-16T12:33:25.929179+0000","last_undegraded":"2023-10-16T12:33:25.929179+0000","last_fullsized":"2023-10-16T12:33:25.929179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927071+0000","last_change":"2023-10-16T12:33:24.605142+0000","last_active":"2023-10-16T12:33:25.927071+0000","last_peered":"2023-10-16T12:33:25.927071+0000","last_clean":"2023-10-16T12:33:25.927071+0000","last_became_active":"2023-10-16T12:33:24.604653+0000","last_became_peered":"2023-10-16T12:33:24.604653+0000","last_unstale":"2023-10-16T12:33:25.927071+0000","last_undegraded":"2023-10-16T12:33:25.927071+0000","last_fullsized":"2023-10-16T12:33:25.927071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:42.877240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584708+0000","last_change":"2023-10-16T12:33:24.606078+0000","last_active":"2023-10-16T12:33:25.584708+0000","last_peered":"2023-10-16T12:33:25.584708+0000","last_clean":"2023-10-16T12:33:25.584708+0000","last_became_active":"2023-10-16T12:33:24.605741+0000","last_became_peered":"2023-10-16T12:33:24.605741+0000","last_unstale":"2023-10-16T12:33:25.584708+0000","last_undegraded":"2023-10-16T12:33:25.584708+0000","last_fullsized":"2023-10-16T12:33:25.584708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584951+0000","last_change":"2023-10-16T12:33:24.606094+0000","last_active":"2023-10-16T12:33:25.584951+0000","last_peered":"2023-10-16T12:33:25.584951+0000","last_clean":"2023-10-16T12:33:25.584951+0000","last_became_active":"2023-10-16T12:33:24.605770+0000","last_became_peered":"2023-10-16T12:33:24.605770+0000","last_unstale":"2023-10-16T12:33:25.584951+0000","last_undegraded":"2023-10-16T12:33:25.584951+0000","last_fullsized":"2023-10-16T12:33:25.584951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590756+0000","last_change":"2023-10-16T12:33:24.618354+0000","last_active":"2023-10-16T12:33:25.590756+0000","last_peered":"2023-10-16T12:33:25.590756+0000","last_clean":"2023-10-16T12:33:25.590756+0000","last_became_active":"2023-10-16T12:33:24.617040+0000","last_became_peered":"2023-10-16T12:33:24.617040+0000","last_unstale":"2023-10-16T12:33:25.590756+0000","last_undegraded":"2023-10-16T12:33:25.590756+0000","last_fullsized":"2023-10-16T12:33:25.590756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927146+0000","last_change":"2023-10-16T12:33:24.612736+0000","last_active":"2023-10-16T12:33:25.927146+0000","last_peered":"2023-10-16T12:33:25.927146+0000","last_clean":"2023-10-16T12:33:25.927146+0000","last_became_active":"2023-10-16T12:33:24.612452+0000","last_became_peered":"2023-10-16T12:33:24.612452+0000","last_unstale":"2023-10-16T12:33:25.927146+0000","last_undegraded":"2023-10-16T12:33:25.927146+0000","last_fullsized":"2023-10-16T12:33:25.927146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930486+0000","last_change":"2023-10-16T12:33:24.621402+0000","last_active":"2023-10-16T12:33:25.930486+0000","last_peered":"2023-10-16T12:33:25.930486+0000","last_clean":"2023-10-16T12:33:25.930486+0000","last_became_active":"2023-10-16T12:33:24.621020+0000","last_became_peered":"2023-10-16T12:33:24.621020+0000","last_unstale":"2023-10-16T12:33:25.930486+0000","last_undegraded":"2023-10-16T12:33:25.930486+0000","last_fullsized":"2023-10-16T12:33:25.930486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:13.324213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928170+0000","last_change":"2023-10-16T12:33:24.613689+0000","last_active":"2023-10-16T12:33:25.928170+0000","last_peered":"2023-10-16T12:33:25.928170+0000","last_clean":"2023-10-16T12:33:25.928170+0000","last_became_active":"2023-10-16T12:33:24.613366+0000","last_became_peered":"2023-10-16T12:33:24.613366+0000","last_unstale":"2023-10-16T12:33:25.928170+0000","last_undegraded":"2023-10-16T12:33:25.928170+0000","last_fullsized":"2023-10-16T12:33:25.928170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:46.931083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928066+0000","last_change":"2023-10-16T12:33:24.615117+0000","last_active":"2023-10-16T12:33:25.928066+0000","last_peered":"2023-10-16T12:33:25.928066+0000","last_clean":"2023-10-16T12:33:25.928066+0000","last_became_active":"2023-10-16T12:33:24.614794+0000","last_became_peered":"2023-10-16T12:33:24.614794+0000","last_unstale":"2023-10-16T12:33:25.928066+0000","last_undegraded":"2023-10-16T12:33:25.928066+0000","last_fullsized":"2023-10-16T12:33:25.928066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586082+0000","last_change":"2023-10-16T12:33:24.616251+0000","last_active":"2023-10-16T12:33:25.586082+0000","last_peered":"2023-10-16T12:33:25.586082+0000","last_clean":"2023-10-16T12:33:25.586082+0000","last_became_active":"2023-10-16T12:33:24.615946+0000","last_became_peered":"2023-10-16T12:33:24.615946+0000","last_unstale":"2023-10-16T12:33:25.586082+0000","last_undegraded":"2023-10-16T12:33:25.586082+0000","last_fullsized":"2023-10-16T12:33:25.586082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929157+0000","last_change":"2023-10-16T12:33:24.613346+0000","last_active":"2023-10-16T12:33:25.929157+0000","last_peered":"2023-10-16T12:33:25.929157+0000","last_clean":"2023-10-16T12:33:25.929157+0000","last_became_active":"2023-10-16T12:33:24.612974+0000","last_became_peered":"2023-10-16T12:33:24.612974+0000","last_unstale":"2023-10-16T12:33:25.929157+0000","last_undegraded":"2023-10-16T12:33:25.929157+0000","last_fullsized":"2023-10-16T12:33:25.929157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585129+0000","last_change":"2023-10-16T12:33:24.624851+0000","last_active":"2023-10-16T12:33:25.585129+0000","last_peered":"2023-10-16T12:33:25.585129+0000","last_clean":"2023-10-16T12:33:25.585129+0000","last_became_active":"2023-10-16T12:33:24.624550+0000","last_became_peered":"2023-10-16T12:33:24.624550+0000","last_unstale":"2023-10-16T12:33:25.585129+0000","last_undegraded":"2023-10-16T12:33:25.585129+0000","last_fullsized":"2023-10-16T12:33:25.585129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930566+0000","last_change":"2023-10-16T12:33:24.614362+0000","last_active":"2023-10-16T12:33:25.930566+0000","last_peered":"2023-10-16T12:33:25.930566+0000","last_clean":"2023-10-16T12:33:25.930566+0000","last_became_active":"2023-10-16T12:33:24.613880+0000","last_became_peered":"2023-10-16T12:33:24.613880+0000","last_unstale":"2023-10-16T12:33:25.930566+0000","last_undegraded":"2023-10-16T12:33:25.930566+0000","last_fullsized":"2023-10-16T12:33:25.930566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929120+0000","last_change":"2023-10-16T12:33:24.618147+0000","last_active":"2023-10-16T12:33:25.929120+0000","last_peered":"2023-10-16T12:33:25.929120+0000","last_clean":"2023-10-16T12:33:25.929120+0000","last_became_active":"2023-10-16T12:33:24.617724+0000","last_became_peered":"2023-10-16T12:33:24.617724+0000","last_unstale":"2023-10-16T12:33:25.929120+0000","last_undegraded":"2023-10-16T12:33:25.929120+0000","last_fullsized":"2023-10-16T12:33:25.929120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:52.314433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585125+0000","last_change":"2023-10-16T12:33:24.605102+0000","last_active":"2023-10-16T12:33:25.585125+0000","last_peered":"2023-10-16T12:33:25.585125+0000","last_clean":"2023-10-16T12:33:25.585125+0000","last_became_active":"2023-10-16T12:33:24.604498+0000","last_became_peered":"2023-10-16T12:33:24.604498+0000","last_unstale":"2023-10-16T12:33:25.585125+0000","last_undegraded":"2023-10-16T12:33:25.585125+0000","last_fullsized":"2023-10-16T12:33:25.585125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930199+0000","last_change":"2023-10-16T12:33:24.613950+0000","last_active":"2023-10-16T12:33:25.930199+0000","last_peered":"2023-10-16T12:33:25.930199+0000","last_clean":"2023-10-16T12:33:25.930199+0000","last_became_active":"2023-10-16T12:33:24.613512+0000","last_became_peered":"2023-10-16T12:33:24.613512+0000","last_unstale":"2023-10-16T12:33:25.930199+0000","last_undegraded":"2023-10-16T12:33:25.930199+0000","last_fullsized":"2023-10-16T12:33:25.930199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928442+0000","last_change":"2023-10-16T12:33:24.612300+0000","last_active":"2023-10-16T12:33:25.928442+0000","last_peered":"2023-10-16T12:33:25.928442+0000","last_clean":"2023-10-16T12:33:25.928442+0000","last_became_active":"2023-10-16T12:33:24.611773+0000","last_became_peered":"2023-10-16T12:33:24.611773+0000","last_unstale":"2023-10-16T12:33:25.928442+0000","last_undegraded":"2023-10-16T12:33:25.928442+0000","last_fullsized":"2023-10-16T12:33:25.928442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590561+0000","last_change":"2023-10-16T12:33:24.615637+0000","last_active":"2023-10-16T12:33:25.590561+0000","last_peered":"2023-10-16T12:33:25.590561+0000","last_clean":"2023-10-16T12:33:25.590561+0000","last_became_active":"2023-10-16T12:33:24.615341+0000","last_became_peered":"2023-10-16T12:33:24.615341+0000","last_unstale":"2023-10-16T12:33:25.590561+0000","last_undegraded":"2023-10-16T12:33:25.590561+0000","last_fullsized":"2023-10-16T12:33:25.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928067+0000","last_change":"2023-10-16T12:33:24.603765+0000","last_active":"2023-10-16T12:33:25.928067+0000","last_peered":"2023-10-16T12:33:25.928067+0000","last_clean":"2023-10-16T12:33:25.928067+0000","last_became_active":"2023-10-16T12:33:24.603469+0000","last_became_peered":"2023-10-16T12:33:24.603469+0000","last_unstale":"2023-10-16T12:33:25.928067+0000","last_undegraded":"2023-10-16T12:33:25.928067+0000","last_fullsized":"2023-10-16T12:33:25.928067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927844+0000","last_change":"2023-10-16T12:33:24.617779+0000","last_active":"2023-10-16T12:33:25.927844+0000","last_peered":"2023-10-16T12:33:25.927844+0000","last_clean":"2023-10-16T12:33:25.927844+0000","last_became_active":"2023-10-16T12:33:24.617328+0000","last_became_peered":"2023-10-16T12:33:24.617328+0000","last_unstale":"2023-10-16T12:33:25.927844+0000","last_undegraded":"2023-10-16T12:33:25.927844+0000","last_fullsized":"2023-10-16T12:33:25.927844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:26.089073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.587918+0000","last_change":"2023-10-16T12:33:24.605171+0000","last_active":"2023-10-16T12:33:25.587918+0000","last_peered":"2023-10-16T12:33:25.587918+0000","last_clean":"2023-10-16T12:33:25.587918+0000","last_became_active":"2023-10-16T12:33:24.604706+0000","last_became_peered":"2023-10-16T12:33:24.604706+0000","last_unstale":"2023-10-16T12:33:25.587918+0000","last_undegraded":"2023-10-16T12:33:25.587918+0000","last_fullsized":"2023-10-16T12:33:25.587918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:45.448070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929923+0000","last_change":"2023-10-16T12:33:23.983081+0000","last_active":"2023-10-16T12:33:25.929923+0000","last_peered":"2023-10-16T12:33:25.929923+0000","last_clean":"2023-10-16T12:33:25.929923+0000","last_became_active":"2023-10-16T12:33:23.979378+0000","last_became_peered":"2023-10-16T12:33:23.979378+0000","last_unstale":"2023-10-16T12:33:25.929923+0000","last_undegraded":"2023-10-16T12:33:25.929923+0000","last_fullsized":"2023-10-16T12:33:25.929923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928531+0000","last_change":"2023-10-16T12:33:24.612688+0000","last_active":"2023-10-16T12:33:25.928531+0000","last_peered":"2023-10-16T12:33:25.928531+0000","last_clean":"2023-10-16T12:33:25.928531+0000","last_became_active":"2023-10-16T12:33:24.612202+0000","last_became_peered":"2023-10-16T12:33:24.612202+0000","last_unstale":"2023-10-16T12:33:25.928531+0000","last_undegraded":"2023-10-16T12:33:25.928531+0000","last_fullsized":"2023-10-16T12:33:25.928531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929373+0000","last_change":"2023-10-16T12:33:23.983502+0000","last_active":"2023-10-16T12:33:25.929373+0000","last_peered":"2023-10-16T12:33:25.929373+0000","last_clean":"2023-10-16T12:33:25.929373+0000","last_became_active":"2023-10-16T12:33:23.983189+0000","last_became_peered":"2023-10-16T12:33:23.983189+0000","last_unstale":"2023-10-16T12:33:25.929373+0000","last_undegraded":"2023-10-16T12:33:25.929373+0000","last_fullsized":"2023-10-16T12:33:25.929373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:34.912908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929270+0000","last_change":"2023-10-16T12:33:23.984326+0000","last_active":"2023-10-16T12:33:25.929270+0000","last_peered":"2023-10-16T12:33:25.929270+0000","last_clean":"2023-10-16T12:33:25.929270+0000","last_became_active":"2023-10-16T12:33:23.984015+0000","last_became_peered":"2023-10-16T12:33:23.984015+0000","last_unstale":"2023-10-16T12:33:25.929270+0000","last_undegraded":"2023-10-16T12:33:25.929270+0000","last_fullsized":"2023-10-16T12:33:25.929270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929604+0000","last_change":"2023-10-16T12:33:24.615262+0000","last_active":"2023-10-16T12:33:25.929604+0000","last_peered":"2023-10-16T12:33:25.929604+0000","last_clean":"2023-10-16T12:33:25.929604+0000","last_became_active":"2023-10-16T12:33:24.610418+0000","last_became_peered":"2023-10-16T12:33:24.610418+0000","last_unstale":"2023-10-16T12:33:25.929604+0000","last_undegraded":"2023-10-16T12:33:25.929604+0000","last_fullsized":"2023-10-16T12:33:25.929604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:45.603374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585175+0000","last_change":"2023-10-16T12:33:23.976361+0000","last_active":"2023-10-16T12:33:25.585175+0000","last_peered":"2023-10-16T12:33:25.585175+0000","last_clean":"2023-10-16T12:33:25.585175+0000","last_became_active":"2023-10-16T12:33:23.975488+0000","last_became_peered":"2023-10-16T12:33:23.975488+0000","last_unstale":"2023-10-16T12:33:25.585175+0000","last_undegraded":"2023-10-16T12:33:25.585175+0000","last_fullsized":"2023-10-16T12:33:25.585175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584519+0000","last_change":"2023-10-16T12:33:24.603552+0000","last_active":"2023-10-16T12:33:25.584519+0000","last_peered":"2023-10-16T12:33:25.584519+0000","last_clean":"2023-10-16T12:33:25.584519+0000","last_became_active":"2023-10-16T12:33:24.602661+0000","last_became_peered":"2023-10-16T12:33:24.602661+0000","last_unstale":"2023-10-16T12:33:25.584519+0000","last_undegraded":"2023-10-16T12:33:25.584519+0000","last_fullsized":"2023-10-16T12:33:25.584519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929244+0000","last_change":"2023-10-16T12:33:23.983406+0000","last_active":"2023-10-16T12:33:25.929244+0000","last_peered":"2023-10-16T12:33:25.929244+0000","last_clean":"2023-10-16T12:33:25.929244+0000","last_became_active":"2023-10-16T12:33:23.982570+0000","last_became_peered":"2023-10-16T12:33:23.982570+0000","last_unstale":"2023-10-16T12:33:25.929244+0000","last_undegraded":"2023-10-16T12:33:25.929244+0000","last_fullsized":"2023-10-16T12:33:25.929244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:17.276818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591933+0000","last_change":"2023-10-16T12:33:24.616611+0000","last_active":"2023-10-16T12:33:25.591933+0000","last_peered":"2023-10-16T12:33:25.591933+0000","last_clean":"2023-10-16T12:33:25.591933+0000","last_became_active":"2023-10-16T12:33:24.616318+0000","last_became_peered":"2023-10-16T12:33:24.616318+0000","last_unstale":"2023-10-16T12:33:25.591933+0000","last_undegraded":"2023-10-16T12:33:25.591933+0000","last_fullsized":"2023-10-16T12:33:25.591933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.591819+0000","last_change":"2023-10-16T12:33:23.982425+0000","last_active":"2023-10-16T12:33:25.591819+0000","last_peered":"2023-10-16T12:33:25.591819+0000","last_clean":"2023-10-16T12:33:25.591819+0000","last_became_active":"2023-10-16T12:33:23.982159+0000","last_became_peered":"2023-10-16T12:33:23.982159+0000","last_unstale":"2023-10-16T12:33:25.591819+0000","last_undegraded":"2023-10-16T12:33:25.591819+0000","last_fullsized":"2023-10-16T12:33:25.591819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:00.917480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929362+0000","last_change":"2023-10-16T12:33:24.607789+0000","last_active":"2023-10-16T12:33:25.929362+0000","last_peered":"2023-10-16T12:33:25.929362+0000","last_clean":"2023-10-16T12:33:25.929362+0000","last_became_active":"2023-10-16T12:33:24.606558+0000","last_became_peered":"2023-10-16T12:33:24.606558+0000","last_unstale":"2023-10-16T12:33:25.929362+0000","last_undegraded":"2023-10-16T12:33:25.929362+0000","last_fullsized":"2023-10-16T12:33:25.929362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584911+0000","last_change":"2023-10-16T12:33:23.994926+0000","last_active":"2023-10-16T12:33:25.584911+0000","last_peered":"2023-10-16T12:33:25.584911+0000","last_clean":"2023-10-16T12:33:25.584911+0000","last_became_active":"2023-10-16T12:33:23.994616+0000","last_became_peered":"2023-10-16T12:33:23.994616+0000","last_unstale":"2023-10-16T12:33:25.584911+0000","last_undegraded":"2023-10-16T12:33:25.584911+0000","last_fullsized":"2023-10-16T12:33:25.584911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.615708+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.615262+0000","last_became_peered":"2023-10-16T12:33:24.615262+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:42.947040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928417+0000","last_change":"2023-10-16T12:33:23.990728+0000","last_active":"2023-10-16T12:33:25.928417+0000","last_peered":"2023-10-16T12:33:25.928417+0000","last_clean":"2023-10-16T12:33:25.928417+0000","last_became_active":"2023-10-16T12:33:23.988939+0000","last_became_peered":"2023-10-16T12:33:23.988939+0000","last_unstale":"2023-10-16T12:33:25.928417+0000","last_undegraded":"2023-10-16T12:33:25.928417+0000","last_fullsized":"2023-10-16T12:33:25.928417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:58.091012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930079+0000","last_change":"2023-10-16T12:33:24.620766+0000","last_active":"2023-10-16T12:33:25.930079+0000","last_peered":"2023-10-16T12:33:25.930079+0000","last_clean":"2023-10-16T12:33:25.930079+0000","last_became_active":"2023-10-16T12:33:24.620453+0000","last_became_peered":"2023-10-16T12:33:24.620453+0000","last_unstale":"2023-10-16T12:33:25.930079+0000","last_undegraded":"2023-10-16T12:33:25.930079+0000","last_fullsized":"2023-10-16T12:33:25.930079+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928702+0000","last_change":"2023-10-16T12:33:23.968343+0000","last_active":"2023-10-16T12:33:25.928702+0000","last_peered":"2023-10-16T12:33:25.928702+0000","last_clean":"2023-10-16T12:33:25.928702+0000","last_became_active":"2023-10-16T12:33:23.967874+0000","last_became_peered":"2023-10-16T12:33:23.967874+0000","last_unstale":"2023-10-16T12:33:25.928702+0000","last_undegraded":"2023-10-16T12:33:25.928702+0000","last_fullsized":"2023-10-16T12:33:25.928702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927246+0000","last_change":"2023-10-16T12:33:24.615841+0000","last_active":"2023-10-16T12:33:25.927246+0000","last_peered":"2023-10-16T12:33:25.927246+0000","last_clean":"2023-10-16T12:33:25.927246+0000","last_became_active":"2023-10-16T12:33:24.615333+0000","last_became_peered":"2023-10-16T12:33:24.615333+0000","last_unstale":"2023-10-16T12:33:25.927246+0000","last_undegraded":"2023-10-16T12:33:25.927246+0000","last_fullsized":"2023-10-16T12:33:25.927246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:23.519029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590504+0000","last_change":"2023-10-16T12:33:23.984016+0000","last_active":"2023-10-16T12:33:25.590504+0000","last_peered":"2023-10-16T12:33:25.590504+0000","last_clean":"2023-10-16T12:33:25.590504+0000","last_became_active":"2023-10-16T12:33:23.983727+0000","last_became_peered":"2023-10-16T12:33:23.983727+0000","last_unstale":"2023-10-16T12:33:25.590504+0000","last_undegraded":"2023-10-16T12:33:25.590504+0000","last_fullsized":"2023-10-16T12:33:25.590504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930468+0000","last_change":"2023-10-16T12:33:24.623341+0000","last_active":"2023-10-16T12:33:25.930468+0000","last_peered":"2023-10-16T12:33:25.930468+0000","last_clean":"2023-10-16T12:33:25.930468+0000","last_became_active":"2023-10-16T12:33:24.622356+0000","last_became_peered":"2023-10-16T12:33:24.622356+0000","last_unstale":"2023-10-16T12:33:25.930468+0000","last_undegraded":"2023-10-16T12:33:25.930468+0000","last_fullsized":"2023-10-16T12:33:25.930468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:58.917472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927892+0000","last_change":"2023-10-16T12:33:23.982823+0000","last_active":"2023-10-16T12:33:25.927892+0000","last_peered":"2023-10-16T12:33:25.927892+0000","last_clean":"2023-10-16T12:33:25.927892+0000","last_became_active":"2023-10-16T12:33:23.981883+0000","last_became_peered":"2023-10-16T12:33:23.981883+0000","last_unstale":"2023-10-16T12:33:25.927892+0000","last_undegraded":"2023-10-16T12:33:25.927892+0000","last_fullsized":"2023-10-16T12:33:25.927892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:09.201117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927851+0000","last_change":"2023-10-16T12:33:24.617583+0000","last_active":"2023-10-16T12:33:25.927851+0000","last_peered":"2023-10-16T12:33:25.927851+0000","last_clean":"2023-10-16T12:33:25.927851+0000","last_became_active":"2023-10-16T12:33:24.616992+0000","last_became_peered":"2023-10-16T12:33:24.616992+0000","last_unstale":"2023-10-16T12:33:25.927851+0000","last_undegraded":"2023-10-16T12:33:25.927851+0000","last_fullsized":"2023-10-16T12:33:25.927851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:38.188085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.926869+0000","last_change":"2023-10-16T12:33:23.988300+0000","last_active":"2023-10-16T12:33:25.926869+0000","last_peered":"2023-10-16T12:33:25.926869+0000","last_clean":"2023-10-16T12:33:25.926869+0000","last_became_active":"2023-10-16T12:33:23.987831+0000","last_became_peered":"2023-10-16T12:33:23.987831+0000","last_unstale":"2023-10-16T12:33:25.926869+0000","last_undegraded":"2023-10-16T12:33:25.926869+0000","last_fullsized":"2023-10-16T12:33:25.926869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928930+0000","last_change":"2023-10-16T12:33:24.613800+0000","last_active":"2023-10-16T12:33:25.928930+0000","last_peered":"2023-10-16T12:33:25.928930+0000","last_clean":"2023-10-16T12:33:25.928930+0000","last_became_active":"2023-10-16T12:33:24.613268+0000","last_became_peered":"2023-10-16T12:33:24.613268+0000","last_unstale":"2023-10-16T12:33:25.928930+0000","last_undegraded":"2023-10-16T12:33:25.928930+0000","last_fullsized":"2023-10-16T12:33:25.928930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588350+0000","last_change":"2023-10-16T12:33:23.970854+0000","last_active":"2023-10-16T12:33:25.588350+0000","last_peered":"2023-10-16T12:33:25.588350+0000","last_clean":"2023-10-16T12:33:25.588350+0000","last_became_active":"2023-10-16T12:33:23.970073+0000","last_became_peered":"2023-10-16T12:33:23.970073+0000","last_unstale":"2023-10-16T12:33:25.588350+0000","last_undegraded":"2023-10-16T12:33:25.588350+0000","last_fullsized":"2023-10-16T12:33:25.588350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930633+0000","last_change":"2023-10-16T12:33:24.624669+0000","last_active":"2023-10-16T12:33:25.930633+0000","last_peered":"2023-10-16T12:33:25.930633+0000","last_clean":"2023-10-16T12:33:25.930633+0000","last_became_active":"2023-10-16T12:33:24.624315+0000","last_became_peered":"2023-10-16T12:33:24.624315+0000","last_unstale":"2023-10-16T12:33:25.930633+0000","last_undegraded":"2023-10-16T12:33:25.930633+0000","last_fullsized":"2023-10-16T12:33:25.930633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:26.649183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927431+0000","last_change":"2023-10-16T12:33:23.980566+0000","last_active":"2023-10-16T12:33:25.927431+0000","last_peered":"2023-10-16T12:33:25.927431+0000","last_clean":"2023-10-16T12:33:25.927431+0000","last_became_active":"2023-10-16T12:33:23.980200+0000","last_became_peered":"2023-10-16T12:33:23.980200+0000","last_unstale":"2023-10-16T12:33:25.927431+0000","last_undegraded":"2023-10-16T12:33:25.927431+0000","last_fullsized":"2023-10-16T12:33:25.927431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:07.785639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.931065+0000","last_change":"2023-10-16T12:33:24.624902+0000","last_active":"2023-10-16T12:33:25.931065+0000","last_peered":"2023-10-16T12:33:25.931065+0000","last_clean":"2023-10-16T12:33:25.931065+0000","last_became_active":"2023-10-16T12:33:24.624646+0000","last_became_peered":"2023-10-16T12:33:24.624646+0000","last_unstale":"2023-10-16T12:33:25.931065+0000","last_undegraded":"2023-10-16T12:33:25.931065+0000","last_fullsized":"2023-10-16T12:33:25.931065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:21.560247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585130+0000","last_change":"2023-10-16T12:33:23.984004+0000","last_active":"2023-10-16T12:33:25.585130+0000","last_peered":"2023-10-16T12:33:25.585130+0000","last_clean":"2023-10-16T12:33:25.585130+0000","last_became_active":"2023-10-16T12:33:23.982385+0000","last_became_peered":"2023-10-16T12:33:23.982385+0000","last_unstale":"2023-10-16T12:33:25.585130+0000","last_undegraded":"2023-10-16T12:33:25.585130+0000","last_fullsized":"2023-10-16T12:33:25.585130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586032+0000","last_change":"2023-10-16T12:33:24.611253+0000","last_active":"2023-10-16T12:33:25.586032+0000","last_peered":"2023-10-16T12:33:25.586032+0000","last_clean":"2023-10-16T12:33:25.586032+0000","last_became_active":"2023-10-16T12:33:24.610712+0000","last_became_peered":"2023-10-16T12:33:24.610712+0000","last_unstale":"2023-10-16T12:33:25.586032+0000","last_undegraded":"2023-10-16T12:33:25.586032+0000","last_fullsized":"2023-10-16T12:33:25.586032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:45.149939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927465+0000","last_change":"2023-10-16T12:33:23.985969+0000","last_active":"2023-10-16T12:33:25.927465+0000","last_peered":"2023-10-16T12:33:25.927465+0000","last_clean":"2023-10-16T12:33:25.927465+0000","last_became_active":"2023-10-16T12:33:23.985467+0000","last_became_peered":"2023-10-16T12:33:23.985467+0000","last_unstale":"2023-10-16T12:33:25.927465+0000","last_undegraded":"2023-10-16T12:33:25.927465+0000","last_fullsized":"2023-10-16T12:33:25.927465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930068+0000","last_change":"2023-10-16T12:33:24.627563+0000","last_active":"2023-10-16T12:33:25.930068+0000","last_peered":"2023-10-16T12:33:25.930068+0000","last_clean":"2023-10-16T12:33:25.930068+0000","last_became_active":"2023-10-16T12:33:24.627241+0000","last_became_peered":"2023-10-16T12:33:24.627241+0000","last_unstale":"2023-10-16T12:33:25.930068+0000","last_undegraded":"2023-10-16T12:33:25.930068+0000","last_fullsized":"2023-10-16T12:33:25.930068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-16T12:33:24.598233+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:33:24.598233+0000","last_peered":"2023-10-16T12:33:24.598233+0000","last_clean":"2023-10-16T12:33:24.598233+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:33:24.598233+0000","last_undegraded":"2023-10-16T12:33:24.598233+0000","last_fullsized":"2023-10-16T12:33:24.598233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585104+0000","last_change":"2023-10-16T12:33:23.972294+0000","last_active":"2023-10-16T12:33:25.585104+0000","last_peered":"2023-10-16T12:33:25.585104+0000","last_clean":"2023-10-16T12:33:25.585104+0000","last_became_active":"2023-10-16T12:33:23.971886+0000","last_became_peered":"2023-10-16T12:33:23.971886+0000","last_unstale":"2023-10-16T12:33:25.585104+0000","last_undegraded":"2023-10-16T12:33:25.585104+0000","last_fullsized":"2023-10-16T12:33:25.585104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:41.874107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929522+0000","last_change":"2023-10-16T12:33:24.604564+0000","last_active":"2023-10-16T12:33:25.929522+0000","last_peered":"2023-10-16T12:33:25.929522+0000","last_clean":"2023-10-16T12:33:25.929522+0000","last_became_active":"2023-10-16T12:33:24.604240+0000","last_became_peered":"2023-10-16T12:33:24.604240+0000","last_unstale":"2023-10-16T12:33:25.929522+0000","last_undegraded":"2023-10-16T12:33:25.929522+0000","last_fullsized":"2023-10-16T12:33:25.929522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927720+0000","last_change":"2023-10-16T12:33:23.988943+0000","last_active":"2023-10-16T12:33:25.927720+0000","last_peered":"2023-10-16T12:33:25.927720+0000","last_clean":"2023-10-16T12:33:25.927720+0000","last_became_active":"2023-10-16T12:33:23.988631+0000","last_became_peered":"2023-10-16T12:33:23.988631+0000","last_unstale":"2023-10-16T12:33:25.927720+0000","last_undegraded":"2023-10-16T12:33:25.927720+0000","last_fullsized":"2023-10-16T12:33:25.927720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927155+0000","last_change":"2023-10-16T12:33:24.615033+0000","last_active":"2023-10-16T12:33:25.927155+0000","last_peered":"2023-10-16T12:33:25.927155+0000","last_clean":"2023-10-16T12:33:25.927155+0000","last_became_active":"2023-10-16T12:33:24.614433+0000","last_became_peered":"2023-10-16T12:33:24.614433+0000","last_unstale":"2023-10-16T12:33:25.927155+0000","last_undegraded":"2023-10-16T12:33:25.927155+0000","last_fullsized":"2023-10-16T12:33:25.927155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:38.208712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928316+0000","last_change":"2023-10-16T12:33:23.978410+0000","last_active":"2023-10-16T12:33:25.928316+0000","last_peered":"2023-10-16T12:33:25.928316+0000","last_clean":"2023-10-16T12:33:25.928316+0000","last_became_active":"2023-10-16T12:33:23.978050+0000","last_became_peered":"2023-10-16T12:33:23.978050+0000","last_unstale":"2023-10-16T12:33:25.928316+0000","last_undegraded":"2023-10-16T12:33:25.928316+0000","last_fullsized":"2023-10-16T12:33:25.928316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584693+0000","last_change":"2023-10-16T12:33:24.600057+0000","last_active":"2023-10-16T12:33:25.584693+0000","last_peered":"2023-10-16T12:33:25.584693+0000","last_clean":"2023-10-16T12:33:25.584693+0000","last_became_active":"2023-10-16T12:33:24.599294+0000","last_became_peered":"2023-10-16T12:33:24.599294+0000","last_unstale":"2023-10-16T12:33:25.584693+0000","last_undegraded":"2023-10-16T12:33:25.584693+0000","last_fullsized":"2023-10-16T12:33:25.584693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:34.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590705+0000","last_change":"2023-10-16T12:33:23.990485+0000","last_active":"2023-10-16T12:33:25.590705+0000","last_peered":"2023-10-16T12:33:25.590705+0000","last_clean":"2023-10-16T12:33:25.590705+0000","last_became_active":"2023-10-16T12:33:23.989836+0000","last_became_peered":"2023-10-16T12:33:23.989836+0000","last_unstale":"2023-10-16T12:33:25.590705+0000","last_undegraded":"2023-10-16T12:33:25.590705+0000","last_fullsized":"2023-10-16T12:33:25.590705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:55:37.205578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.589289+0000","last_change":"2023-10-16T12:33:24.619525+0000","last_active":"2023-10-16T12:33:25.589289+0000","last_peered":"2023-10-16T12:33:25.589289+0000","last_clean":"2023-10-16T12:33:25.589289+0000","last_became_active":"2023-10-16T12:33:24.619242+0000","last_became_peered":"2023-10-16T12:33:24.619242+0000","last_unstale":"2023-10-16T12:33:25.589289+0000","last_undegraded":"2023-10-16T12:33:25.589289+0000","last_fullsized":"2023-10-16T12:33:25.589289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:12.969746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584815+0000","last_change":"2023-10-16T12:33:23.976427+0000","last_active":"2023-10-16T12:33:25.584815+0000","last_peered":"2023-10-16T12:33:25.584815+0000","last_clean":"2023-10-16T12:33:25.584815+0000","last_became_active":"2023-10-16T12:33:23.975234+0000","last_became_peered":"2023-10-16T12:33:23.975234+0000","last_unstale":"2023-10-16T12:33:25.584815+0000","last_undegraded":"2023-10-16T12:33:25.584815+0000","last_fullsized":"2023-10-16T12:33:25.584815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:03.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.590492+0000","last_change":"2023-10-16T12:33:24.614482+0000","last_active":"2023-10-16T12:33:25.590492+0000","last_peered":"2023-10-16T12:33:25.590492+0000","last_clean":"2023-10-16T12:33:25.590492+0000","last_became_active":"2023-10-16T12:33:24.614144+0000","last_became_peered":"2023-10-16T12:33:24.614144+0000","last_unstale":"2023-10-16T12:33:25.590492+0000","last_undegraded":"2023-10-16T12:33:25.590492+0000","last_fullsized":"2023-10-16T12:33:25.590492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:05.934302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930752+0000","last_change":"2023-10-16T12:33:23.986466+0000","last_active":"2023-10-16T12:33:25.930752+0000","last_peered":"2023-10-16T12:33:25.930752+0000","last_clean":"2023-10-16T12:33:25.930752+0000","last_became_active":"2023-10-16T12:33:23.985508+0000","last_became_peered":"2023-10-16T12:33:23.985508+0000","last_unstale":"2023-10-16T12:33:25.930752+0000","last_undegraded":"2023-10-16T12:33:25.930752+0000","last_fullsized":"2023-10-16T12:33:25.930752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:08.057398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584833+0000","last_change":"2023-10-16T12:33:24.605662+0000","last_active":"2023-10-16T12:33:25.584833+0000","last_peered":"2023-10-16T12:33:25.584833+0000","last_clean":"2023-10-16T12:33:25.584833+0000","last_became_active":"2023-10-16T12:33:24.605319+0000","last_became_peered":"2023-10-16T12:33:24.605319+0000","last_unstale":"2023-10-16T12:33:25.584833+0000","last_undegraded":"2023-10-16T12:33:25.584833+0000","last_fullsized":"2023-10-16T12:33:25.584833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:01.920614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:26.775691+0000","last_change":"2023-10-16T12:33:23.986752+0000","last_active":"2023-10-16T12:33:26.775691+0000","last_peered":"2023-10-16T12:33:26.775691+0000","last_clean":"2023-10-16T12:33:26.775691+0000","last_became_active":"2023-10-16T12:33:23.986189+0000","last_became_peered":"2023-10-16T12:33:23.986189+0000","last_unstale":"2023-10-16T12:33:26.775691+0000","last_undegraded":"2023-10-16T12:33:26.775691+0000","last_fullsized":"2023-10-16T12:33:26.775691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930740+0000","last_change":"2023-10-16T12:33:24.616668+0000","last_active":"2023-10-16T12:33:25.930740+0000","last_peered":"2023-10-16T12:33:25.930740+0000","last_clean":"2023-10-16T12:33:25.930740+0000","last_became_active":"2023-10-16T12:33:24.616308+0000","last_became_peered":"2023-10-16T12:33:24.616308+0000","last_unstale":"2023-10-16T12:33:25.930740+0000","last_undegraded":"2023-10-16T12:33:25.930740+0000","last_fullsized":"2023-10-16T12:33:25.930740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:50.114419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929376+0000","last_change":"2023-10-16T12:33:23.991338+0000","last_active":"2023-10-16T12:33:25.929376+0000","last_peered":"2023-10-16T12:33:25.929376+0000","last_clean":"2023-10-16T12:33:25.929376+0000","last_became_active":"2023-10-16T12:33:23.986162+0000","last_became_peered":"2023-10-16T12:33:23.986162+0000","last_unstale":"2023-10-16T12:33:25.929376+0000","last_undegraded":"2023-10-16T12:33:25.929376+0000","last_fullsized":"2023-10-16T12:33:25.929376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:32.667813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.928205+0000","last_change":"2023-10-16T12:33:24.612178+0000","last_active":"2023-10-16T12:33:25.928205+0000","last_peered":"2023-10-16T12:33:25.928205+0000","last_clean":"2023-10-16T12:33:25.928205+0000","last_became_active":"2023-10-16T12:33:24.608876+0000","last_became_peered":"2023-10-16T12:33:24.608876+0000","last_unstale":"2023-10-16T12:33:25.928205+0000","last_undegraded":"2023-10-16T12:33:25.928205+0000","last_fullsized":"2023-10-16T12:33:25.928205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:49.254518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927671+0000","last_change":"2023-10-16T12:33:23.983928+0000","last_active":"2023-10-16T12:33:25.927671+0000","last_peered":"2023-10-16T12:33:25.927671+0000","last_clean":"2023-10-16T12:33:25.927671+0000","last_became_active":"2023-10-16T12:33:23.983266+0000","last_became_peered":"2023-10-16T12:33:23.983266+0000","last_unstale":"2023-10-16T12:33:25.927671+0000","last_undegraded":"2023-10-16T12:33:25.927671+0000","last_fullsized":"2023-10-16T12:33:25.927671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:44.444937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.588965+0000","last_change":"2023-10-16T12:33:24.605248+0000","last_active":"2023-10-16T12:33:25.588965+0000","last_peered":"2023-10-16T12:33:25.588965+0000","last_clean":"2023-10-16T12:33:25.588965+0000","last_became_active":"2023-10-16T12:33:24.604915+0000","last_became_peered":"2023-10-16T12:33:24.604915+0000","last_unstale":"2023-10-16T12:33:25.588965+0000","last_undegraded":"2023-10-16T12:33:25.588965+0000","last_fullsized":"2023-10-16T12:33:25.588965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:10.203500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584607+0000","last_change":"2023-10-16T12:33:23.987636+0000","last_active":"2023-10-16T12:33:25.584607+0000","last_peered":"2023-10-16T12:33:25.584607+0000","last_clean":"2023-10-16T12:33:25.584607+0000","last_became_active":"2023-10-16T12:33:23.987422+0000","last_became_peered":"2023-10-16T12:33:23.987422+0000","last_unstale":"2023-10-16T12:33:25.584607+0000","last_undegraded":"2023-10-16T12:33:25.584607+0000","last_fullsized":"2023-10-16T12:33:25.584607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930014+0000","last_change":"2023-10-16T12:33:24.607092+0000","last_active":"2023-10-16T12:33:25.930014+0000","last_peered":"2023-10-16T12:33:25.930014+0000","last_clean":"2023-10-16T12:33:25.930014+0000","last_became_active":"2023-10-16T12:33:24.606816+0000","last_became_peered":"2023-10-16T12:33:24.606816+0000","last_unstale":"2023-10-16T12:33:25.930014+0000","last_undegraded":"2023-10-16T12:33:25.930014+0000","last_fullsized":"2023-10-16T12:33:25.930014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:09.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.584703+0000","last_change":"2023-10-16T12:33:23.982671+0000","last_active":"2023-10-16T12:33:25.584703+0000","last_peered":"2023-10-16T12:33:25.584703+0000","last_clean":"2023-10-16T12:33:25.584703+0000","last_became_active":"2023-10-16T12:33:23.982314+0000","last_became_peered":"2023-10-16T12:33:23.982314+0000","last_unstale":"2023-10-16T12:33:25.584703+0000","last_undegraded":"2023-10-16T12:33:25.584703+0000","last_fullsized":"2023-10-16T12:33:25.584703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:19.174532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930259+0000","last_change":"2023-10-16T12:33:24.614159+0000","last_active":"2023-10-16T12:33:25.930259+0000","last_peered":"2023-10-16T12:33:25.930259+0000","last_clean":"2023-10-16T12:33:25.930259+0000","last_became_active":"2023-10-16T12:33:24.613857+0000","last_became_peered":"2023-10-16T12:33:24.613857+0000","last_unstale":"2023-10-16T12:33:25.930259+0000","last_undegraded":"2023-10-16T12:33:25.930259+0000","last_fullsized":"2023-10-16T12:33:25.930259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:31.318689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930386+0000","last_change":"2023-10-16T12:33:23.987798+0000","last_active":"2023-10-16T12:33:25.930386+0000","last_peered":"2023-10-16T12:33:25.930386+0000","last_clean":"2023-10-16T12:33:25.930386+0000","last_became_active":"2023-10-16T12:33:23.987460+0000","last_became_peered":"2023-10-16T12:33:23.987460+0000","last_unstale":"2023-10-16T12:33:25.930386+0000","last_undegraded":"2023-10-16T12:33:25.930386+0000","last_fullsized":"2023-10-16T12:33:25.930386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:15.423009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929124+0000","last_change":"2023-10-16T12:33:24.615457+0000","last_active":"2023-10-16T12:33:25.929124+0000","last_peered":"2023-10-16T12:33:25.929124+0000","last_clean":"2023-10-16T12:33:25.929124+0000","last_became_active":"2023-10-16T12:33:24.615166+0000","last_became_peered":"2023-10-16T12:33:24.615166+0000","last_unstale":"2023-10-16T12:33:25.929124+0000","last_undegraded":"2023-10-16T12:33:25.929124+0000","last_fullsized":"2023-10-16T12:33:25.929124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:24.606678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.586074+0000","last_change":"2023-10-16T12:33:24.604607+0000","last_active":"2023-10-16T12:33:25.586074+0000","last_peered":"2023-10-16T12:33:25.586074+0000","last_clean":"2023-10-16T12:33:25.586074+0000","last_became_active":"2023-10-16T12:33:24.604183+0000","last_became_peered":"2023-10-16T12:33:24.604183+0000","last_unstale":"2023-10-16T12:33:25.586074+0000","last_undegraded":"2023-10-16T12:33:25.586074+0000","last_fullsized":"2023-10-16T12:33:25.586074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929969+0000","last_change":"2023-10-16T12:33:23.990373+0000","last_active":"2023-10-16T12:33:25.929969+0000","last_peered":"2023-10-16T12:33:25.929969+0000","last_clean":"2023-10-16T12:33:25.929969+0000","last_became_active":"2023-10-16T12:33:23.990049+0000","last_became_peered":"2023-10-16T12:33:23.990049+0000","last_unstale":"2023-10-16T12:33:25.929969+0000","last_undegraded":"2023-10-16T12:33:25.929969+0000","last_fullsized":"2023-10-16T12:33:25.929969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:36.472040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.585658+0000","last_change":"2023-10-16T12:33:24.612037+0000","last_active":"2023-10-16T12:33:25.585658+0000","last_peered":"2023-10-16T12:33:25.585658+0000","last_clean":"2023-10-16T12:33:25.585658+0000","last_became_active":"2023-10-16T12:33:24.611609+0000","last_became_peered":"2023-10-16T12:33:24.611609+0000","last_unstale":"2023-10-16T12:33:25.585658+0000","last_undegraded":"2023-10-16T12:33:25.585658+0000","last_fullsized":"2023-10-16T12:33:25.585658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:44.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930569+0000","last_change":"2023-10-16T12:33:23.985742+0000","last_active":"2023-10-16T12:33:25.930569+0000","last_peered":"2023-10-16T12:33:25.930569+0000","last_clean":"2023-10-16T12:33:25.930569+0000","last_became_active":"2023-10-16T12:33:23.985053+0000","last_became_peered":"2023-10-16T12:33:23.985053+0000","last_unstale":"2023-10-16T12:33:25.930569+0000","last_undegraded":"2023-10-16T12:33:25.930569+0000","last_fullsized":"2023-10-16T12:33:25.930569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:43.191029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.927970+0000","last_change":"2023-10-16T12:33:24.610651+0000","last_active":"2023-10-16T12:33:25.927970+0000","last_peered":"2023-10-16T12:33:25.927970+0000","last_clean":"2023-10-16T12:33:25.927970+0000","last_became_active":"2023-10-16T12:33:24.605512+0000","last_became_peered":"2023-10-16T12:33:24.605512+0000","last_unstale":"2023-10-16T12:33:25.927970+0000","last_undegraded":"2023-10-16T12:33:25.927970+0000","last_fullsized":"2023-10-16T12:33:25.927970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:50.839660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930559+0000","last_change":"2023-10-16T12:33:23.981596+0000","last_active":"2023-10-16T12:33:25.930559+0000","last_peered":"2023-10-16T12:33:25.930559+0000","last_clean":"2023-10-16T12:33:25.930559+0000","last_became_active":"2023-10-16T12:33:23.981314+0000","last_became_peered":"2023-10-16T12:33:23.981314+0000","last_unstale":"2023-10-16T12:33:25.930559+0000","last_undegraded":"2023-10-16T12:33:25.930559+0000","last_fullsized":"2023-10-16T12:33:25.930559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:22.515896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.930671+0000","last_change":"2023-10-16T12:33:24.613513+0000","last_active":"2023-10-16T12:33:25.930671+0000","last_peered":"2023-10-16T12:33:25.930671+0000","last_clean":"2023-10-16T12:33:25.930671+0000","last_became_active":"2023-10-16T12:33:24.613158+0000","last_became_peered":"2023-10-16T12:33:24.613158+0000","last_unstale":"2023-10-16T12:33:25.930671+0000","last_undegraded":"2023-10-16T12:33:25.930671+0000","last_fullsized":"2023-10-16T12:33:25.930671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:23.938821+0000","last_clean_scrub_stamp":"2023-10-16T12:33:23.938821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:22.211586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-16T12:33:25.929580+0000","last_change":"2023-10-16T12:33:23.978618+0000","last_active":"2023-10-16T12:33:25.929580+0000","last_peered":"2023-10-16T12:33:25.929580+0000","last_clean":"2023-10-16T12:33:25.929580+0000","last_became_active":"2023-10-16T12:33:23.978325+0000","last_became_peered":"2023-10-16T12:33:23.978325+0000","last_unstale":"2023-10-16T12:33:25.929580+0000","last_undegraded":"2023-10-16T12:33:25.929580+0000","last_fullsized":"2023-10-16T12:33:25.929580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:22.935687+0000","last_clean_scrub_stamp":"2023-10-16T12:33:22.935687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:11.966613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2130000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":80476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1139999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":831488,"data_stored":669870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.758}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:33:36.445 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T12:33:36.445 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-16T12:33:36.446 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-16T12:33:36.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T12:33:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.709+0000 7f892bfff640 1 Processor -- start 2023-10-16T12:33:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.710+0000 7f892bfff640 1 -- start start 2023-10-16T12:33:36.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.711+0000 7f892bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f892c15ce40 0x7f892c0d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.711+0000 7f892bfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0d12b0 con 0x7f892c0cd7c0 2023-10-16T12:33:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.711+0000 7f892bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0d14f0 con 0x7f892c0b8560 2023-10-16T12:33:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.711+0000 7f892bfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c0d17d0 con 0x7f892c15ce40 2023-10-16T12:33:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f892b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f892c0cd7c0 0x7f892c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54306/0 (socket says 172.21.15.60:54306) 2023-10-16T12:33:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f892b7fe640 1 -- 172.21.15.60:0/1089357636 learned_addr learned my addr 172.21.15.60:0/1089357636 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f892affd640 1 --2- 172.21.15.60:0/1089357636 >> v2:172.21.15.144:3300/0 conn(0x7f892c15ce40 0x7f892c0d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047222367 0 0) 0x7f8924001110 con 0x7f892c0cd7c0 2023-10-16T12:33:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918003520 con 0x7f892c0cd7c0 2023-10-16T12:33:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1329330837 0 0) 0x7f8918003520 con 0x7f892c0cd7c0 2023-10-16T12:33:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 >> v2:172.21.15.144:3300/0 conn(0x7f892c15ce40 msgr2=0x7f892c0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 --2- 172.21.15.60:0/1089357636 >> v2:172.21.15.144:3300/0 conn(0x7f892c15ce40 0x7f892c0d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 >> v1:172.21.15.144:6789/0 conn(0x7f892c0b8560 legacy=0x7f892c0d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0d1ab0 con 0x7f892c0cd7c0 2023-10-16T12:33:36.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f8924001800 con 0x7f892c0cd7c0 2023-10-16T12:33:36.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.712+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 >> v1:172.21.15.60:6789/0 conn(0x7f892c0cd7c0 legacy=0x7f892c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f8929ffb640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f8918003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f8929ffb640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89180047c0 0x7f8918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f8929ffb640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89180051d0 0x7f891800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c0d17d0 con 0x7f89180039c0 2023-10-16T12:33:36.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891800c120 con 0x7f89180051d0 2023-10-16T12:33:36.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891800c2e0 con 0x7f89180047c0 2023-10-16T12:33:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892affd640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f8918003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892affd640 1 -- 172.21.15.60:0/1089357636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89180047c0 msgr2=0x7f8918004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892affd640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89180047c0 0x7f8918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892affd640 1 -- 172.21.15.60:0/1089357636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89180051d0 msgr2=0x7f891800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892affd640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89180051d0 0x7f891800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892affd640 1 -- 172.21.15.60:0/1089357636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918003520 con 0x7f89180039c0 2023-10-16T12:33:36.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.713+0000 7f892b7fe640 1 -- 172.21.15.60:0/1089357636 reap_dead start 2023-10-16T12:33:36.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.714+0000 7f892affd640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f8918003de0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f891c009f90 tx=0x7f89180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d32cbde6f033003 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:36.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c02cbf0 con 0x7f89180039c0 2023-10-16T12:33:36.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00f040 con 0x7f89180039c0 2023-10-16T12:33:36.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f8929ffb640 1 -- 172.21.15.60:0/1089357636 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c034440 con 0x7f89180039c0 2023-10-16T12:33:36.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 -- 172.21.15.60:0/1089357636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 msgr2=0x7f8918003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f8918003de0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f891c009f90 tx=0x7f89180057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 -- 172.21.15.60:0/1089357636 shutdown_connections 2023-10-16T12:33:36.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 --2- 172.21.15.60:0/1089357636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f8918003de0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 -- 172.21.15.60:0/1089357636 >> 172.21.15.60:0/1089357636 conn(0x7f892c0cf570 msgr2=0x7f892c0cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:36.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 -- 172.21.15.60:0/1089357636 shutdown_connections 2023-10-16T12:33:36.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.715+0000 7f892bfff640 1 -- 172.21.15.60:0/1089357636 wait complete. 2023-10-16T12:33:36.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.717+0000 7f892bfff640 1 Processor -- start 2023-10-16T12:33:36.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.717+0000 7f892bfff640 1 -- start start 2023-10-16T12:33:36.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.718+0000 7f892bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f892c155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.718+0000 7f892bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f892c15ec10 0x7f892c15f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.718+0000 7f892bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f892c15f090 0x7f892c15fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.718+0000 7f892bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c1f3540 con 0x7f89180039c0 2023-10-16T12:33:36.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.718+0000 7f892bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c1f3260 con 0x7f892c15ec10 2023-10-16T12:33:36.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.718+0000 7f892bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c155380 con 0x7f892c15f090 2023-10-16T12:33:36.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f892c155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f892c155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58172/0 (socket says 172.21.15.60:58172) 2023-10-16T12:33:36.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 -- 172.21.15.60:0/2724262892 learned_addr learned my addr 172.21.15.60:0/2724262892 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:36.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892a7fc640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f892c15ec10 0x7f892c15f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892b7fe640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f892c15f090 0x7f892c15fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 -- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f892c15f090 msgr2=0x7f892c15fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f892c15f090 0x7f892c15fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 -- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f892c15ec10 msgr2=0x7f892c15f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f892c15ec10 0x7f892c15f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 -- 172.21.15.60:0/2724262892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c155660 con 0x7f89180039c0 2023-10-16T12:33:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892a7fc640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f892c15ec10 0x7f892c15f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.719+0000 7f892affd640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f892c155e60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f891c02c590 tx=0x7f891c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:36.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.720+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c034d80 con 0x7f89180039c0 2023-10-16T12:33:36.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.720+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c040450 con 0x7f89180039c0 2023-10-16T12:33:36.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.720+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c040800 con 0x7f89180039c0 2023-10-16T12:33:36.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.720+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c155940 con 0x7f89180039c0 2023-10-16T12:33:36.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.720+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c1f26e0 con 0x7f89180039c0 2023-10-16T12:33:36.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.721+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c0674f0 con 0x7f89180039c0 2023-10-16T12:33:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.722+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f891c040960 con 0x7f89180039c0 2023-10-16T12:33:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.723+0000 7f89237fe640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f890c06ae70 0x7f890c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.723+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f891c0abe10 con 0x7f89180039c0 2023-10-16T12:33:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.723+0000 7f892a7fc640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f890c06ae70 0x7f890c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.724+0000 7f892a7fc640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f890c06ae70 0x7f890c06d330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f892c1f00f0 tx=0x7f8914009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:36.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.727+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f891c076ec0 con 0x7f89180039c0 2023-10-16T12:33:36.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.958+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f892c1551e0 con 0x7f89180039c0 2023-10-16T12:33:36.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.959+0000 7f89237fe640 1 -- 172.21.15.60:0/2724262892 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f891c039040 con 0x7f89180039c0 2023-10-16T12:33:36.962 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:36.962 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-16T12:33:36.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f890c06ae70 msgr2=0x7f890c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f890c06ae70 0x7f890c06d330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f892c1f00f0 tx=0x7f8914009040 comp rx=0 tx=0).stop 2023-10-16T12:33:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 msgr2=0x7f892c155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f892c155e60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f891c02c590 tx=0x7f891c009750 comp rx=0 tx=0).stop 2023-10-16T12:33:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 shutdown_connections 2023-10-16T12:33:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f890c06ae70 0x7f890c06d330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f892c15f090 0x7f892c15fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f892c15ec10 0x7f892c15f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 --2- 172.21.15.60:0/2724262892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89180039c0 0x7f892c155e60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:36.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 >> 172.21.15.60:0/2724262892 conn(0x7f892c0cf570 msgr2=0x7f892c0b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:36.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 shutdown_connections 2023-10-16T12:33:36.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:36.962+0000 7f892bfff640 1 -- 172.21.15.60:0/2724262892 wait complete. 2023-10-16T12:33:36.985 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-16T12:33:36.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.209+0000 7f1c3ab05640 1 Processor -- start 2023-10-16T12:33:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.209+0000 7f1c3ab05640 1 -- start start 2023-10-16T12:33:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c3ab05640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c3415aaa0 0x7f1c3415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c3ab05640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c34070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c3ab05640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3415ed60 con 0x7f1c3415ae80 2023-10-16T12:33:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c3ab05640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3406b810 con 0x7f1c3415aaa0 2023-10-16T12:33:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c3ab05640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c340707e0 con 0x7f1c340ce390 2023-10-16T12:33:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c388a1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c3415aaa0 0x7f1c3415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c33fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1c3415ae80 0x7f1c340cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54310/0 (socket says 172.21.15.60:54310) 2023-10-16T12:33:37.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c33fff640 1 -- 172.21.15.60:0/2001910423 learned_addr learned my addr 172.21.15.60:0/2001910423 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.211+0000 7f1c390a2640 1 --2- 172.21.15.60:0/2001910423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c34070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.212+0000 7f1c388a1640 1 -- 172.21.15.60:0/2001910423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 msgr2=0x7f1c34070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.212+0000 7f1c388a1640 1 --2- 172.21.15.60:0/2001910423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c34070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.212+0000 7f1c388a1640 1 -- 172.21.15.60:0/2001910423 >> v1:172.21.15.60:6789/0 conn(0x7f1c3415ae80 legacy=0x7f1c340cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.212+0000 7f1c388a1640 1 -- 172.21.15.60:0/2001910423 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3415ef90 con 0x7f1c3415aaa0 2023-10-16T12:33:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.212+0000 7f1c337fe640 1 -- 172.21.15.60:0/2001910423 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053843108 0 0) 0x7f1c3415ed60 con 0x7f1c3415ae80 2023-10-16T12:33:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.213+0000 7f1c388a1640 1 --2- 172.21.15.60:0/2001910423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c3415e820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1c280097a0 tx=0x7f1c2802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45bcd463c978223 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.213+0000 7f1c337fe640 1 -- 172.21.15.60:0/2001910423 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2802d040 con 0x7f1c3415aaa0 2023-10-16T12:33:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.213+0000 7f1c337fe640 1 -- 172.21.15.60:0/2001910423 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c28031810 con 0x7f1c3415aaa0 2023-10-16T12:33:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.213+0000 7f1c337fe640 1 -- 172.21.15.60:0/2001910423 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28031ba0 con 0x7f1c3415aaa0 2023-10-16T12:33:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 -- 172.21.15.60:0/2001910423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 msgr2=0x7f1c3415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/2001910423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c3415e820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1c280097a0 tx=0x7f1c2802b8b0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 -- 172.21.15.60:0/2001910423 shutdown_connections 2023-10-16T12:33:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/2001910423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c34070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/2001910423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c3415e820 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 -- 172.21.15.60:0/2001910423 >> 172.21.15.60:0/2001910423 conn(0x7f1c340cb410 msgr2=0x7f1c340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 -- 172.21.15.60:0/2001910423 shutdown_connections 2023-10-16T12:33:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.214+0000 7f1c3ab05640 1 -- 172.21.15.60:0/2001910423 wait complete. 2023-10-16T12:33:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.216+0000 7f1c3ab05640 1 Processor -- start 2023-10-16T12:33:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.216+0000 7f1c3ab05640 1 -- start start 2023-10-16T12:33:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c3ab05640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c340c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c3ab05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c34158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c3ab05640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c3415ae80 0x7f1c340ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c3ab05640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34159e20 con 0x7f1c3415aaa0 2023-10-16T12:33:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c3ab05640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34159030 con 0x7f1c3415ae80 2023-10-16T12:33:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c3ab05640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34159580 con 0x7f1c340ce390 2023-10-16T12:33:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c34158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c34158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58194/0 (socket says 172.21.15.60:58194) 2023-10-16T12:33:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c33fff640 1 -- 172.21.15.60:0/1809105568 learned_addr learned my addr 172.21.15.60:0/1809105568 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c388a1640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c340c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c390a2640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c3415ae80 0x7f1c340ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c33fff640 1 -- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 msgr2=0x7f1c340c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.217+0000 7f1c33fff640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c340c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c33fff640 1 -- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c3415ae80 msgr2=0x7f1c340ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c33fff640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c3415ae80 0x7f1c340ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c33fff640 1 -- 172.21.15.60:0/1809105568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28009400 con 0x7f1c3415aaa0 2023-10-16T12:33:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c388a1640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c340c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c390a2640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c3415ae80 0x7f1c340ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c33fff640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c34158af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000c3e0 tx=0x7f1c2000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20005ce0 con 0x7f1c3415aaa0 2023-10-16T12:33:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20005e40 con 0x7f1c3415aaa0 2023-10-16T12:33:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20007310 con 0x7f1c3415aaa0 2023-10-16T12:33:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c34159860 con 0x7f1c3415aaa0 2023-10-16T12:33:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.218+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c340c32b0 con 0x7f1c3415aaa0 2023-10-16T12:33:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.219+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c340c3110 con 0x7f1c3415aaa0 2023-10-16T12:33:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.221+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c20007530 con 0x7f1c3415aaa0 2023-10-16T12:33:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.221+0000 7f1c317fa640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c1806aec0 0x7f1c1806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.222+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c20088ca0 con 0x7f1c3415aaa0 2023-10-16T12:33:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.222+0000 7f1c388a1640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c1806aec0 0x7f1c1806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.223+0000 7f1c388a1640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c1806aec0 0x7f1c1806d380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1c28036000 tx=0x7f1c28008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.226+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c340c3110 con 0x7f1c3415aaa0 2023-10-16T12:33:37.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.421+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c340c3110 con 0x7f1c3415aaa0 2023-10-16T12:33:37.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.424+0000 7f1c317fa640 1 -- 172.21.15.60:0/1809105568 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f1c340c3110 con 0x7f1c3415aaa0 2023-10-16T12:33:37.427 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:37.427 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:37.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c1806aec0 msgr2=0x7f1c1806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c1806aec0 0x7f1c1806d380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1c28036000 tx=0x7f1c28008030 comp rx=0 tx=0).stop 2023-10-16T12:33:37.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 msgr2=0x7f1c34158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c34158af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000c3e0 tx=0x7f1c2000c8a0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 shutdown_connections 2023-10-16T12:33:37.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c1806aec0 0x7f1c1806d380 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c3415ae80 0x7f1c340ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c3415aaa0 0x7f1c34158af0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 --2- 172.21.15.60:0/1809105568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c340ce390 0x7f1c340c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.426+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 >> 172.21.15.60:0/1809105568 conn(0x7f1c340cb410 msgr2=0x7f1c3406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.427+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 shutdown_connections 2023-10-16T12:33:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.427+0000 7f1c3ab05640 1 -- 172.21.15.60:0/1809105568 wait complete. 2023-10-16T12:33:37.449 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:33:37.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:33:37.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.681+0000 7f8e3f47e640 1 Processor -- start 2023-10-16T12:33:37.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.681+0000 7f8e3f47e640 1 -- start start 2023-10-16T12:33:37.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3f47e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e380ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3f47e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 0x7f8e380ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3f47e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e380ce480 con 0x7f8e380ad670 2023-10-16T12:33:37.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3f47e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e380ca950 con 0x7f8e380cd260 2023-10-16T12:33:37.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3f47e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380cd7b0 con 0x7f8e3815cc00 2023-10-16T12:33:37.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3e47c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e380ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3dc7b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 0x7f8e380ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3dc7b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 0x7f8e380ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49352/0 (socket says 172.21.15.60:49352) 2023-10-16T12:33:37.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.683+0000 7f8e3dc7b640 1 -- 172.21.15.60:0/2359158258 learned_addr learned my addr 172.21.15.60:0/2359158258 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:37.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.684+0000 7f8e3e47c640 1 -- 172.21.15.60:0/2359158258 >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 msgr2=0x7f8e380ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:37.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.684+0000 7f8e3e47c640 1 --2- 172.21.15.60:0/2359158258 >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 0x7f8e380ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.684+0000 7f8e3e47c640 1 -- 172.21.15.60:0/2359158258 >> v1:172.21.15.144:6789/0 conn(0x7f8e380ad670 legacy=0x7f8e380c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.684+0000 7f8e3e47c640 1 -- 172.21.15.60:0/2359158258 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e380cda90 con 0x7f8e3815cc00 2023-10-16T12:33:37.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.684+0000 7f8e3dc7b640 1 --2- 172.21.15.60:0/2359158258 >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 0x7f8e380ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:33:37.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.685+0000 7f8e3e47c640 1 --2- 172.21.15.60:0/2359158258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e380ca410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8e340097a0 tx=0x7f8e3402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea34b376ab49967 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:37.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.686+0000 7f8e3d47a640 1 -- 172.21.15.60:0/2359158258 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3402d040 con 0x7f8e3815cc00 2023-10-16T12:33:37.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.686+0000 7f8e3d47a640 1 -- 172.21.15.60:0/2359158258 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e34031810 con 0x7f8e3815cc00 2023-10-16T12:33:37.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.686+0000 7f8e3d47a640 1 -- 172.21.15.60:0/2359158258 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34031ba0 con 0x7f8e3815cc00 2023-10-16T12:33:37.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 -- 172.21.15.60:0/2359158258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 msgr2=0x7f8e380ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/2359158258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e380ca410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8e340097a0 tx=0x7f8e3402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 -- 172.21.15.60:0/2359158258 shutdown_connections 2023-10-16T12:33:37.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/2359158258 >> v2:172.21.15.144:3300/0 conn(0x7f8e380cd260 0x7f8e380ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/2359158258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e380ca410 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 -- 172.21.15.60:0/2359158258 >> 172.21.15.60:0/2359158258 conn(0x7f8e380b0170 msgr2=0x7f8e380b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:37.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 -- 172.21.15.60:0/2359158258 shutdown_connections 2023-10-16T12:33:37.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.687+0000 7f8e3f47e640 1 -- 172.21.15.60:0/2359158258 wait complete. 2023-10-16T12:33:37.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.689+0000 7f8e3f47e640 1 Processor -- start 2023-10-16T12:33:37.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.689+0000 7f8e3f47e640 1 -- start start 2023-10-16T12:33:37.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3f47e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e380ad670 0x7f8e380c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3f47e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 0x7f8e380adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3f47e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e38149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3f47e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380c8a20 con 0x7f8e380cd260 2023-10-16T12:33:37.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3f47e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380c7980 con 0x7f8e380ad670 2023-10-16T12:33:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3f47e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380c7c60 con 0x7f8e3815cc00 2023-10-16T12:33:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3dc7b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 0x7f8e380adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3dc7b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 0x7f8e380adfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58200/0 (socket says 172.21.15.60:58200) 2023-10-16T12:33:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3dc7b640 1 -- 172.21.15.60:0/668046713 learned_addr learned my addr 172.21.15.60:0/668046713 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.690+0000 7f8e3e47c640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e380ad670 0x7f8e380c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3dc7b640 1 -- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 msgr2=0x7f8e38149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3ec7d640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e38149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3dc7b640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e38149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3dc7b640 1 -- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e380ad670 msgr2=0x7f8e380c7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3dc7b640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e380ad670 0x7f8e380c7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3dc7b640 1 -- 172.21.15.60:0/668046713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c0094d0 con 0x7f8e380cd260 2023-10-16T12:33:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3ec7d640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e38149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3dc7b640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 0x7f8e380adfa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00e9d0 tx=0x7f8e2c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c00ca60 con 0x7f8e380cd260 2023-10-16T12:33:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c00cbc0 con 0x7f8e380cd260 2023-10-16T12:33:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e34009400 con 0x7f8e380cd260 2023-10-16T12:33:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.691+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c013470 con 0x7f8e380cd260 2023-10-16T12:33:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.692+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e380c7ff0 con 0x7f8e380cd260 2023-10-16T12:33:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.693+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3804e9b0 con 0x7f8e380cd260 2023-10-16T12:33:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.694+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8e2c0135d0 con 0x7f8e380cd260 2023-10-16T12:33:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.696+0000 7f8e2affd640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8e1c06ae70 0x7f8e1c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.696+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e2c089950 con 0x7f8e380cd260 2023-10-16T12:33:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.697+0000 7f8e3e47c640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8e1c06ae70 0x7f8e1c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.698+0000 7f8e3e47c640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8e1c06ae70 0x7f8e1c06d330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e34036000 tx=0x7f8e3403c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.701+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e2c054a00 con 0x7f8e380cd260 2023-10-16T12:33:37.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.906+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e3806d320 con 0x7f8e380cd260 2023-10-16T12:33:37.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.907+0000 7f8e2affd640 1 -- 172.21.15.60:0/668046713 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e2c0170a0 con 0x7f8e380cd260 2023-10-16T12:33:37.910 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:33:37.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8e1c06ae70 msgr2=0x7f8e1c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8e1c06ae70 0x7f8e1c06d330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e34036000 tx=0x7f8e3403c040 comp rx=0 tx=0).stop 2023-10-16T12:33:37.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 msgr2=0x7f8e380adfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:37.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 0x7f8e380adfa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00e9d0 tx=0x7f8e2c00ee90 comp rx=0 tx=0).stop 2023-10-16T12:33:37.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 shutdown_connections 2023-10-16T12:33:37.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8e1c06ae70 0x7f8e1c06d330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e3815cc00 0x7f8e38149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e380cd260 0x7f8e380adfa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 --2- 172.21.15.60:0/668046713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e380ad670 0x7f8e380c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:37.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.911+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 >> 172.21.15.60:0/668046713 conn(0x7f8e380b0170 msgr2=0x7f8e380af250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:37.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.912+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 shutdown_connections 2023-10-16T12:33:37.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:37.912+0000 7f8e3f47e640 1 -- 172.21.15.60:0/668046713 wait complete. 2023-10-16T12:33:37.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:33:38.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.161+0000 7f0fd1d27640 1 Processor -- start 2023-10-16T12:33:38.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.162+0000 7f0fd1d27640 1 -- start start 2023-10-16T12:33:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.163+0000 7f0fd1d27640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.163+0000 7f0fd1d27640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0fcc06c810 0x7f0fcc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1d27640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc070df0 con 0x7f0fcc070450 2023-10-16T12:33:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1d27640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fcc15e490 con 0x7f0fcc06c810 2023-10-16T12:33:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1d27640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc15aef0 con 0x7f0fcc159950 2023-10-16T12:33:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fcbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0fcc070450 0x7f0fcc06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54318/0 (socket says 172.21.15.60:54318) 2023-10-16T12:33:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1526640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0fcc06c810 0x7f0fcc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fcbfff640 1 -- 172.21.15.60:0/4248231659 learned_addr learned my addr 172.21.15.60:0/4248231659 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:38.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd0d25640 1 --2- 172.21.15.60:0/4248231659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1526640 1 -- 172.21.15.60:0/4248231659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 msgr2=0x7f0fcc15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1526640 1 --2- 172.21.15.60:0/4248231659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fd1526640 1 -- 172.21.15.60:0/4248231659 >> v1:172.21.15.60:6789/0 conn(0x7f0fcc070450 legacy=0x7f0fcc06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.164+0000 7f0fcb7fe640 1 -- 172.21.15.60:0/4248231659 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697952736 0 0) 0x7f0fcc070df0 con 0x7f0fcc070450 2023-10-16T12:33:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.165+0000 7f0fd1526640 1 -- 172.21.15.60:0/4248231659 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc06cc80 con 0x7f0fcc06c810 2023-10-16T12:33:38.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.166+0000 7f0fd1526640 1 --2- 172.21.15.60:0/4248231659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0708b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009d90 tx=0x7f0fc402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f9556178c4e529 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.166+0000 7f0fcb7fe640 1 -- 172.21.15.60:0/4248231659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc402cbc0 con 0x7f0fcc06c810 2023-10-16T12:33:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.166+0000 7f0fcb7fe640 1 -- 172.21.15.60:0/4248231659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc402cd20 con 0x7f0fcc06c810 2023-10-16T12:33:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.166+0000 7f0fcb7fe640 1 -- 172.21.15.60:0/4248231659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4034660 con 0x7f0fcc06c810 2023-10-16T12:33:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 -- 172.21.15.60:0/4248231659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc06c810 msgr2=0x7f0fcc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/4248231659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0708b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009d90 tx=0x7f0fc402be10 comp rx=0 tx=0).stop 2023-10-16T12:33:38.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 -- 172.21.15.60:0/4248231659 shutdown_connections 2023-10-16T12:33:38.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/4248231659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0708b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/4248231659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 -- 172.21.15.60:0/4248231659 >> 172.21.15.60:0/4248231659 conn(0x7f0fcc0ce6a0 msgr2=0x7f0fcc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:38.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 -- 172.21.15.60:0/4248231659 shutdown_connections 2023-10-16T12:33:38.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.167+0000 7f0fd1d27640 1 -- 172.21.15.60:0/4248231659 wait complete. 2023-10-16T12:33:38.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.168+0000 7f0fd1d27640 1 Processor -- start 2023-10-16T12:33:38.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.169+0000 7f0fd1d27640 1 -- start start 2023-10-16T12:33:38.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1d27640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1d27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 0x7f0fcc0ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1d27640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc0c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1d27640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0c50b0 con 0x7f0fcc070450 2023-10-16T12:33:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1d27640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0c8360 con 0x7f0fcc06c810 2023-10-16T12:33:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1d27640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0ae5b0 con 0x7f0fcc159950 2023-10-16T12:33:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 0x7f0fcc0ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 0x7f0fcc0ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58226/0 (socket says 172.21.15.60:58226) 2023-10-16T12:33:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 -- 172.21.15.60:0/886895618 learned_addr learned my addr 172.21.15.60:0/886895618 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd0d25640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1526640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc0c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 -- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 msgr2=0x7f0fcc0c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc0c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 -- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fcc06c810 msgr2=0x7f0fcc0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fcbfff640 1 -- 172.21.15.60:0/886895618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4009a70 con 0x7f0fcc070450 2023-10-16T12:33:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.170+0000 7f0fd1526640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc0c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.171+0000 7f0fcbfff640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 0x7f0fcc0ae070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800c500 tx=0x7f0fb800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.171+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8007560 con 0x7f0fcc070450 2023-10-16T12:33:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.171+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb80047e0 con 0x7f0fcc070450 2023-10-16T12:33:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.171+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc0ae7e0 con 0x7f0fcc070450 2023-10-16T12:33:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.171+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8004b90 con 0x7f0fcc070450 2023-10-16T12:33:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.171+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc0aeda0 con 0x7f0fcc070450 2023-10-16T12:33:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.172+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc04e9b0 con 0x7f0fcc070450 2023-10-16T12:33:38.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.173+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0fb8004cf0 con 0x7f0fcc070450 2023-10-16T12:33:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.175+0000 7f0fc97fa640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fb006ae70 0x7f0fb006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.175+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fb8088a90 con 0x7f0fcc070450 2023-10-16T12:33:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.176+0000 7f0fd0d25640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fb006ae70 0x7f0fb006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.177+0000 7f0fd0d25640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fb006ae70 0x7f0fb006d330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0006fd0 tx=0x7f0fc0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.179+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fb8053bd0 con 0x7f0fcc070450 2023-10-16T12:33:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.374+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0fcc0b4620 con 0x7f0fcc070450 2023-10-16T12:33:38.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.375+0000 7f0fc97fa640 1 -- 172.21.15.60:0/886895618 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0fb80180a0 con 0x7f0fcc070450 2023-10-16T12:33:38.378 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:33:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fb006ae70 msgr2=0x7f0fb006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fb006ae70 0x7f0fb006d330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0006fd0 tx=0x7f0fc0008040 comp rx=0 tx=0).stop 2023-10-16T12:33:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 msgr2=0x7f0fcc0ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 0x7f0fcc0ae070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800c500 tx=0x7f0fb800c9c0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 shutdown_connections 2023-10-16T12:33:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0fb006ae70 0x7f0fb006d330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc159950 0x7f0fcc0c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc070450 0x7f0fcc0ae070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 --2- 172.21.15.60:0/886895618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fcc06c810 0x7f0fcc0c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 >> 172.21.15.60:0/886895618 conn(0x7f0fcc0ce6a0 msgr2=0x7f0fcc06bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 shutdown_connections 2023-10-16T12:33:38.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.378+0000 7f0fd1d27640 1 -- 172.21.15.60:0/886895618 wait complete. 2023-10-16T12:33:38.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:33:38.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.631+0000 7fb742ada640 1 Processor -- start 2023-10-16T12:33:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.631+0000 7fb742ada640 1 -- start start 2023-10-16T12:33:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb742ada640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb73c15ad70 0x7fb73c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb742ada640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb73c0c9ca0 0x7fb73c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb742ada640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c0cb030 con 0x7fb73c0ca3b0 2023-10-16T12:33:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb742ada640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c0c91c0 con 0x7fb73c0c9ca0 2023-10-16T12:33:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb742ada640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c0c94a0 con 0x7fb73c15ad70 2023-10-16T12:33:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb73bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb73c0c9ca0 0x7fb73c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb73bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb73c0c9ca0 0x7fb73c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58234/0 (socket says 172.21.15.60:58234) 2023-10-16T12:33:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb73bfff640 1 -- 172.21.15.60:0/2012275520 learned_addr learned my addr 172.21.15.60:0/2012275520 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.633+0000 7fb740876640 1 --2- 172.21.15.60:0/2012275520 >> v2:172.21.15.144:3300/0 conn(0x7fb73c15ad70 0x7fb73c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.634+0000 7fb73bfff640 1 -- 172.21.15.60:0/2012275520 >> v2:172.21.15.144:3300/0 conn(0x7fb73c15ad70 msgr2=0x7fb73c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.634+0000 7fb73bfff640 1 --2- 172.21.15.60:0/2012275520 >> v2:172.21.15.144:3300/0 conn(0x7fb73c15ad70 0x7fb73c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.634+0000 7fb73bfff640 1 -- 172.21.15.60:0/2012275520 >> v1:172.21.15.144:6789/0 conn(0x7fb73c0ca3b0 legacy=0x7fb73c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.634+0000 7fb73bfff640 1 -- 172.21.15.60:0/2012275520 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c0c9780 con 0x7fb73c0c9ca0 2023-10-16T12:33:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.635+0000 7fb73bfff640 1 --2- 172.21.15.60:0/2012275520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c06a2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0cf780 tx=0x7fb72c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b634fa2747554e02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.636+0000 7fb73b7fe640 1 -- 172.21.15.60:0/2012275520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c030860 con 0x7fb73c0c9ca0 2023-10-16T12:33:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.636+0000 7fb73b7fe640 1 -- 172.21.15.60:0/2012275520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c0309c0 con 0x7fb73c0c9ca0 2023-10-16T12:33:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.636+0000 7fb73b7fe640 1 -- 172.21.15.60:0/2012275520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c030d50 con 0x7fb73c0c9ca0 2023-10-16T12:33:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.636+0000 7fb742ada640 1 -- 172.21.15.60:0/2012275520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c0c9ca0 msgr2=0x7fb73c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.636+0000 7fb742ada640 1 --2- 172.21.15.60:0/2012275520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c06a2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0cf780 tx=0x7fb72c02b9a0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.637+0000 7fb742ada640 1 -- 172.21.15.60:0/2012275520 shutdown_connections 2023-10-16T12:33:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.637+0000 7fb742ada640 1 --2- 172.21.15.60:0/2012275520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c06a2e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.637+0000 7fb742ada640 1 --2- 172.21.15.60:0/2012275520 >> v2:172.21.15.144:3300/0 conn(0x7fb73c15ad70 0x7fb73c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.637+0000 7fb742ada640 1 -- 172.21.15.60:0/2012275520 >> 172.21.15.60:0/2012275520 conn(0x7fb73c0ce9b0 msgr2=0x7fb73c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.637+0000 7fb742ada640 1 -- 172.21.15.60:0/2012275520 shutdown_connections 2023-10-16T12:33:38.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.637+0000 7fb742ada640 1 -- 172.21.15.60:0/2012275520 wait complete. 2023-10-16T12:33:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.638+0000 7fb742ada640 1 Processor -- start 2023-10-16T12:33:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.639+0000 7fb742ada640 1 -- start start 2023-10-16T12:33:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.639+0000 7fb742ada640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c0c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb742ada640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb73c0ca3b0 0x7fb73c1f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb742ada640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 0x7fb73c1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb742ada640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0a9090 con 0x7fb73c15ad70 2023-10-16T12:33:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb742ada640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0c7510 con 0x7fb73c0c9ca0 2023-10-16T12:33:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb742ada640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0c77f0 con 0x7fb73c0ca3b0 2023-10-16T12:33:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 0x7fb73c1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 0x7fb73c1f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58248/0 (socket says 172.21.15.60:58248) 2023-10-16T12:33:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 -- 172.21.15.60:0/2793228120 learned_addr learned my addr 172.21.15.60:0/2793228120 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb740876640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c0c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb73bfff640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb73c0ca3b0 0x7fb73c1f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 -- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb73c0ca3b0 msgr2=0x7fb73c1f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb73c0ca3b0 0x7fb73c1f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 -- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb73c0c9ca0 msgr2=0x7fb73c0c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c0c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb741077640 1 -- 172.21.15.60:0/2793228120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0094d0 con 0x7fb73c15ad70 2023-10-16T12:33:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.640+0000 7fb73bfff640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb73c0ca3b0 0x7fb73c1f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.641+0000 7fb741077640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 0x7fb73c1f3990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c7a0 tx=0x7fb73400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.641+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734003b70 con 0x7fb73c15ad70 2023-10-16T12:33:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.641+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb734003cf0 con 0x7fb73c15ad70 2023-10-16T12:33:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.641+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734003ea0 con 0x7fb73c15ad70 2023-10-16T12:33:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.641+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c0c7ad0 con 0x7fb73c15ad70 2023-10-16T12:33:38.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.641+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c0c8090 con 0x7fb73c15ad70 2023-10-16T12:33:38.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.642+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73c04e160 con 0x7fb73c15ad70 2023-10-16T12:33:38.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.643+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb73400aa90 con 0x7fb73c15ad70 2023-10-16T12:33:38.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.644+0000 7fb7397fa640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb72806ae70 0x7fb72806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:38.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.644+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb734089470 con 0x7fb73c15ad70 2023-10-16T12:33:38.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.645+0000 7fb740876640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb72806ae70 0x7fb72806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:38.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.646+0000 7fb740876640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb72806ae70 0x7fb72806d330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb730009a30 tx=0x7fb730008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.651+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb734054520 con 0x7fb73c15ad70 2023-10-16T12:33:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.846+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb73c0c55d0 con 0x7fb73c15ad70 2023-10-16T12:33:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.847+0000 7fb7397fa640 1 -- 172.21.15.60:0/2793228120 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb734057b40 con 0x7fb73c15ad70 2023-10-16T12:33:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:33:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb72806ae70 msgr2=0x7fb72806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb72806ae70 0x7fb72806d330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb730009a30 tx=0x7fb730008040 comp rx=0 tx=0).stop 2023-10-16T12:33:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 msgr2=0x7fb73c1f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 0x7fb73c1f3990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c7a0 tx=0x7fb73400cc60 comp rx=0 tx=0).stop 2023-10-16T12:33:38.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 shutdown_connections 2023-10-16T12:33:38.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb72806ae70 0x7fb72806d330 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb73c0ca3b0 0x7fb73c1f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb73c15ad70 0x7fb73c1f3990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 --2- 172.21.15.60:0/2793228120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb73c0c9ca0 0x7fb73c0c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:38.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 >> 172.21.15.60:0/2793228120 conn(0x7fb73c0ce9b0 msgr2=0x7fb73c15a040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:38.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.850+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 shutdown_connections 2023-10-16T12:33:38.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:38.851+0000 7fb742ada640 1 -- 172.21.15.60:0/2793228120 wait complete. 2023-10-16T12:33:38.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:33:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.090+0000 7f8481c46640 1 Processor -- start 2023-10-16T12:33:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.090+0000 7f8481c46640 1 -- start start 2023-10-16T12:33:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.091+0000 7f8481c46640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 0x7f847c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.091+0000 7f8481c46640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c159790 con 0x7f847c0cc920 2023-10-16T12:33:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.091+0000 7f8481c46640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0c9b20 con 0x7f847c0caa10 2023-10-16T12:33:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.091+0000 7f8481c46640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0c9e00 con 0x7f847c1593b0 2023-10-16T12:33:39.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.092+0000 7f847bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f847c0cc920 0x7f847c0ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49182/0 (socket says 172.21.15.60:49182) 2023-10-16T12:33:39.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.092+0000 7f847bfff640 1 -- 172.21.15.60:0/3301098329 learned_addr learned my addr 172.21.15.60:0/3301098329 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.092+0000 7f847b7fe640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 0x7f847c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.092+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237722889 0 0) 0x7f847c159790 con 0x7f847c0cc920 2023-10-16T12:33:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.092+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8468003520 con 0x7f847c0cc920 2023-10-16T12:33:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625778890 0 0) 0x7f8468003520 con 0x7f847c0cc920 2023-10-16T12:33:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 msgr2=0x7f847c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 0x7f847c06f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 >> v1:172.21.15.144:6789/0 conn(0x7f847c0caa10 legacy=0x7f847c06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0ca0e0 con 0x7f847c0cc920 2023-10-16T12:33:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f84740019f0 con 0x7f847c0cc920 2023-10-16T12:33:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 >> v1:172.21.15.60:6789/0 conn(0x7f847c0cc920 legacy=0x7f847c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 0x7f84680039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8468003f70 0x7f84680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f8468005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0c9e00 con 0x7f8468004d00 2023-10-16T12:33:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846800de20 con 0x7f8468003f70 2023-10-16T12:33:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846800dfe0 con 0x7f847c1593b0 2023-10-16T12:33:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847bfff640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f8468005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f847b7fe640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 0x7f84680039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.093+0000 7f8473fff640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8468003f70 0x7f84680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.094+0000 7f847bfff640 1 -- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 msgr2=0x7f84680039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.094+0000 7f847bfff640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c1593b0 0x7f84680039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.094+0000 7f847bfff640 1 -- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8468003f70 msgr2=0x7f84680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.094+0000 7f847bfff640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8468003f70 0x7f84680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.094+0000 7f847bfff640 1 -- 172.21.15.60:0/3301098329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8468003520 con 0x7f8468004d00 2023-10-16T12:33:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.094+0000 7f8473fff640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8468003f70 0x7f84680047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f847bfff640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f8468005160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84740013d0 tx=0x7f84680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41ab3e599ca2cf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474034630 con 0x7f8468004d00 2023-10-16T12:33:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847402f070 con 0x7f8468004d00 2023-10-16T12:33:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f847affd640 1 -- 172.21.15.60:0/3301098329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474034a10 con 0x7f8468004d00 2023-10-16T12:33:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f8481c46640 1 -- 172.21.15.60:0/3301098329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 msgr2=0x7f8468005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f8481c46640 1 --2- 172.21.15.60:0/3301098329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f8468005160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84740013d0 tx=0x7f84680057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.095+0000 7f847bfff640 1 -- 172.21.15.60:0/3301098329 reap_dead start 2023-10-16T12:33:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.096+0000 7f8481c46640 1 -- 172.21.15.60:0/3301098329 shutdown_connections 2023-10-16T12:33:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.096+0000 7f8481c46640 1 -- 172.21.15.60:0/3301098329 >> 172.21.15.60:0/3301098329 conn(0x7f847c0cb410 msgr2=0x7f847c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.096+0000 7f8481c46640 1 -- 172.21.15.60:0/3301098329 shutdown_connections 2023-10-16T12:33:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.096+0000 7f8481c46640 1 -- 172.21.15.60:0/3301098329 wait complete. 2023-10-16T12:33:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.097+0000 7f8481c46640 1 Processor -- start 2023-10-16T12:33:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.098+0000 7f8481c46640 1 -- start start 2023-10-16T12:33:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.098+0000 7f8481c46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f847c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.098+0000 7f8481c46640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f847c15aac0 0x7f847c15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.098+0000 7f8481c46640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c0b1c00 0x7f847c0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f8481c46640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c15b510 con 0x7f8468004d00 2023-10-16T12:33:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f8481c46640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c15b7f0 con 0x7f847c15aac0 2023-10-16T12:33:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f8481c46640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c15bad0 con 0x7f847c0b1c00 2023-10-16T12:33:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f847c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f847c06b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45824/0 (socket says 172.21.15.60:45824) 2023-10-16T12:33:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 -- 172.21.15.60:0/3133413814 learned_addr learned my addr 172.21.15.60:0/3133413814 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f8473fff640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f847c15aac0 0x7f847c15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847bfff640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c0b1c00 0x7f847c0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 -- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c0b1c00 msgr2=0x7f847c0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c0b1c00 0x7f847c0b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 -- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f847c15aac0 msgr2=0x7f847c15af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f847c15aac0 0x7f847c15af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f847b7fe640 1 -- 172.21.15.60:0/3133413814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c15bdb0 con 0x7f8468004d00 2023-10-16T12:33:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.099+0000 7f8473fff640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f847c15aac0 0x7f847c15af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.100+0000 7f847b7fe640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f847c06b750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f846c011600 tx=0x7f846c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.100+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00f920 con 0x7f8468004d00 2023-10-16T12:33:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.100+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c00fa80 con 0x7f8468004d00 2023-10-16T12:33:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.100+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00fdf0 con 0x7f8468004d00 2023-10-16T12:33:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.100+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c15c090 con 0x7f8468004d00 2023-10-16T12:33:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.100+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c06bc90 con 0x7f8468004d00 2023-10-16T12:33:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.101+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c04e160 con 0x7f8468004d00 2023-10-16T12:33:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.102+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f846c015460 con 0x7f8468004d00 2023-10-16T12:33:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.103+0000 7f8478ff9640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f845c06ae70 0x7f845c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.103+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f846c089b00 con 0x7f8468004d00 2023-10-16T12:33:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.104+0000 7f8473fff640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f845c06ae70 0x7f845c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.105+0000 7f8473fff640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f845c06ae70 0x7f845c06d330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8464000c00 tx=0x7f8464009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:39.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.107+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f846c08be60 con 0x7f8468004d00 2023-10-16T12:33:39.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.342+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f847c0b4910 con 0x7f8468004d00 2023-10-16T12:33:39.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.343+0000 7f8478ff9640 1 -- 172.21.15.60:0/3133413814 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f846c022090 con 0x7f8468004d00 2023-10-16T12:33:39.346 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:39.346 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:33:39.346 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:33:39.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f845c06ae70 msgr2=0x7f845c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f845c06ae70 0x7f845c06d330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8464000c00 tx=0x7f8464009040 comp rx=0 tx=0).stop 2023-10-16T12:33:39.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 msgr2=0x7f847c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f847c06b750 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f846c011600 tx=0x7f846c011ac0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 shutdown_connections 2023-10-16T12:33:39.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f845c06ae70 0x7f845c06d330 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f847c0b1c00 0x7f847c0b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f847c15aac0 0x7f847c15af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 --2- 172.21.15.60:0/3133413814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8468004d00 0x7f847c06b750 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.345+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 >> 172.21.15.60:0/3133413814 conn(0x7f847c0cb410 msgr2=0x7f847c0c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:39.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.346+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 shutdown_connections 2023-10-16T12:33:39.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.346+0000 7f8481c46640 1 -- 172.21.15.60:0/3133413814 wait complete. 2023-10-16T12:33:39.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.585+0000 7f236614e640 1 Processor -- start 2023-10-16T12:33:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.585+0000 7f236614e640 1 -- start start 2023-10-16T12:33:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236614e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2360159950 0x7f236015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236614e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360070450 0x7f23600708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236614e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2360070df0 con 0x7f236006c100 2023-10-16T12:33:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236614e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236015e490 con 0x7f2360159950 2023-10-16T12:33:39.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236614e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236015aef0 con 0x7f2360070450 2023-10-16T12:33:39.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236514c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2360159950 0x7f236015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236594d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f236006c100 0x7f236006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49192/0 (socket says 172.21.15.60:49192) 2023-10-16T12:33:39.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236594d640 1 -- 172.21.15.60:0/385345896 learned_addr learned my addr 172.21.15.60:0/385345896 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.587+0000 7f236494b640 1 --2- 172.21.15.60:0/385345896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360070450 0x7f23600708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.588+0000 7f236514c640 1 -- 172.21.15.60:0/385345896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360070450 msgr2=0x7f23600708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.588+0000 7f236514c640 1 --2- 172.21.15.60:0/385345896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360070450 0x7f23600708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.588+0000 7f236514c640 1 -- 172.21.15.60:0/385345896 >> v1:172.21.15.60:6789/0 conn(0x7f236006c100 legacy=0x7f236006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.588+0000 7f236514c640 1 -- 172.21.15.60:0/385345896 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236006cc80 con 0x7f2360159950 2023-10-16T12:33:39.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.588+0000 7f2357fff640 1 -- 172.21.15.60:0/385345896 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134973335 0 0) 0x7f2360070df0 con 0x7f236006c100 2023-10-16T12:33:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.589+0000 7f236514c640 1 --2- 172.21.15.60:0/385345896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360159950 0x7f236015a9b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f235c009d90 tx=0x7f235c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd8ec1ead8a99c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.589+0000 7f2357fff640 1 -- 172.21.15.60:0/385345896 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c02cbc0 con 0x7f2360159950 2023-10-16T12:33:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.589+0000 7f2357fff640 1 -- 172.21.15.60:0/385345896 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c02cd20 con 0x7f2360159950 2023-10-16T12:33:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.589+0000 7f2357fff640 1 -- 172.21.15.60:0/385345896 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c034660 con 0x7f2360159950 2023-10-16T12:33:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 -- 172.21.15.60:0/385345896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360159950 msgr2=0x7f236015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 --2- 172.21.15.60:0/385345896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360159950 0x7f236015a9b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f235c009d90 tx=0x7f235c02be10 comp rx=0 tx=0).stop 2023-10-16T12:33:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 -- 172.21.15.60:0/385345896 shutdown_connections 2023-10-16T12:33:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 --2- 172.21.15.60:0/385345896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360070450 0x7f23600708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 --2- 172.21.15.60:0/385345896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360159950 0x7f236015a9b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 -- 172.21.15.60:0/385345896 >> 172.21.15.60:0/385345896 conn(0x7f23600ce6a0 msgr2=0x7f23600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 -- 172.21.15.60:0/385345896 shutdown_connections 2023-10-16T12:33:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.590+0000 7f236614e640 1 -- 172.21.15.60:0/385345896 wait complete. 2023-10-16T12:33:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.591+0000 7f236614e640 1 Processor -- start 2023-10-16T12:33:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.592+0000 7f236614e640 1 -- start start 2023-10-16T12:33:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236614e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f236006c100 0x7f23601f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236614e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 0x7f23600c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236614e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360159950 0x7f236006ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236614e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601f20c0 con 0x7f2360070450 2023-10-16T12:33:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236614e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23600b3320 con 0x7f236006c100 2023-10-16T12:33:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236614e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23600b2310 con 0x7f2360159950 2023-10-16T12:33:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 0x7f23600c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 0x7f23600c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45830/0 (socket says 172.21.15.60:45830) 2023-10-16T12:33:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 -- 172.21.15.60:0/698231247 learned_addr learned my addr 172.21.15.60:0/698231247 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:39.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236594d640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360159950 0x7f236006ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 -- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360159950 msgr2=0x7f236006ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360159950 0x7f236006ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 -- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f236006c100 msgr2=0x7f23601f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236514c640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f236006c100 0x7f23601f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f236006c100 0x7f23601f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236494b640 1 -- 172.21.15.60:0/698231247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c009a70 con 0x7f2360070450 2023-10-16T12:33:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.593+0000 7f236514c640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f236006c100 0x7f23601f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.594+0000 7f236494b640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 0x7f23600c21b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f235000d800 tx=0x7f235000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.594+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000b660 con 0x7f2360070450 2023-10-16T12:33:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.594+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000b7c0 con 0x7f2360070450 2023-10-16T12:33:39.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.594+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23600c26f0 con 0x7f2360070450 2023-10-16T12:33:39.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.594+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000bb50 con 0x7f2360070450 2023-10-16T12:33:39.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.594+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23600b2840 con 0x7f2360070450 2023-10-16T12:33:39.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.595+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236004e9b0 con 0x7f2360070450 2023-10-16T12:33:39.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.596+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f235000bcb0 con 0x7f2360070450 2023-10-16T12:33:39.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.597+0000 7f2355ffb640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f234406ae70 0x7f234406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:39.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.597+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2350089bf0 con 0x7f2360070450 2023-10-16T12:33:39.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.598+0000 7f236514c640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f234406ae70 0x7f234406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:39.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.599+0000 7f236514c640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f234406ae70 0x7f234406d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f235c036000 tx=0x7f235c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:39.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.602+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2350054c00 con 0x7f2360070450 2023-10-16T12:33:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.801+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23600b4620 con 0x7f2360070450 2023-10-16T12:33:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.804+0000 7f2355ffb640 1 -- 172.21.15.60:0/698231247 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f2350058220 con 0x7f2360070450 2023-10-16T12:33:39.807 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:39.807 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:39.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f234406ae70 msgr2=0x7f234406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f234406ae70 0x7f234406d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f235c036000 tx=0x7f235c030000 comp rx=0 tx=0).stop 2023-10-16T12:33:39.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 msgr2=0x7f23600c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 0x7f23600c21b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f235000d800 tx=0x7f235000dcc0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 shutdown_connections 2023-10-16T12:33:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f234406ae70 0x7f234406d330 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2360159950 0x7f236006ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2360070450 0x7f23600c21b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.807+0000 7f236614e640 1 --2- 172.21.15.60:0/698231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f236006c100 0x7f23601f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.808+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 >> 172.21.15.60:0/698231247 conn(0x7f23600ce6a0 msgr2=0x7f2360159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.808+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 shutdown_connections 2023-10-16T12:33:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:39.808+0000 7f236614e640 1 -- 172.21.15.60:0/698231247 wait complete. 2023-10-16T12:33:39.830 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:33:39.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:33:40.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.051+0000 7fdfc9409640 1 Processor -- start 2023-10-16T12:33:40.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.051+0000 7fdfc9409640 1 -- start start 2023-10-16T12:33:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc9409640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdfc406c850 0x7fdfc40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc9409640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4070360 con 0x7fdfc4159ab0 2023-10-16T12:33:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc9409640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40cd1f0 con 0x7fdfc4159e90 2023-10-16T12:33:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc9409640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc415d890 con 0x7fdfc406c850 2023-10-16T12:33:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdfc4159ab0 0x7fdfc406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49204/0 (socket says 172.21.15.60:49204) 2023-10-16T12:33:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc2ffd640 1 -- 172.21.15.60:0/1460688126 learned_addr learned my addr 172.21.15.60:0/1460688126 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc37fe640 1 --2- 172.21.15.60:0/1460688126 >> v2:172.21.15.60:3300/0 conn(0x7fdfc406c850 0x7fdfc40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc27fc640 1 -- 172.21.15.60:0/1460688126 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535542188 0 0) 0x7fdfc4070360 con 0x7fdfc4159ab0 2023-10-16T12:33:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.053+0000 7fdfc27fc640 1 -- 172.21.15.60:0/1460688126 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb0003520 con 0x7fdfc4159ab0 2023-10-16T12:33:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.054+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1460688126 >> v1:172.21.15.144:6789/0 conn(0x7fdfc4159e90 legacy=0x7fdfc406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.054+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1460688126 >> v1:172.21.15.60:6789/0 conn(0x7fdfc4159ab0 legacy=0x7fdfc406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.054+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1460688126 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40cd4d0 con 0x7fdfc406c850 2023-10-16T12:33:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.055+0000 7fdfc37fe640 1 --2- 172.21.15.60:0/1460688126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc406c850 0x7fdfc40ce370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009d90 tx=0x7fdfbc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796cb16db838170 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.055+0000 7fdfc27fc640 1 -- 172.21.15.60:0/1460688126 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc02cbd0 con 0x7fdfc406c850 2023-10-16T12:33:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.055+0000 7fdfc27fc640 1 -- 172.21.15.60:0/1460688126 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc02cd30 con 0x7fdfc406c850 2023-10-16T12:33:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.055+0000 7fdfc27fc640 1 -- 172.21.15.60:0/1460688126 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc02cea0 con 0x7fdfc406c850 2023-10-16T12:33:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 -- 172.21.15.60:0/1460688126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc406c850 msgr2=0x7fdfc40ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1460688126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc406c850 0x7fdfc40ce370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009d90 tx=0x7fdfbc02be20 comp rx=0 tx=0).stop 2023-10-16T12:33:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 -- 172.21.15.60:0/1460688126 shutdown_connections 2023-10-16T12:33:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1460688126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc406c850 0x7fdfc40ce370 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 -- 172.21.15.60:0/1460688126 >> 172.21.15.60:0/1460688126 conn(0x7fdfc40ce9b0 msgr2=0x7fdfc40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 -- 172.21.15.60:0/1460688126 shutdown_connections 2023-10-16T12:33:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.056+0000 7fdfc9409640 1 -- 172.21.15.60:0/1460688126 wait complete. 2023-10-16T12:33:40.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.058+0000 7fdfc9409640 1 Processor -- start 2023-10-16T12:33:40.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.058+0000 7fdfc9409640 1 -- start start 2023-10-16T12:33:40.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc9409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfc406c850 0x7fdfc40c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc9409640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfc4159ab0 0x7fdfc40c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc9409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 0x7fdfc40af700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc9409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40c5440 con 0x7fdfc4159e90 2023-10-16T12:33:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc9409640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40c89a0 con 0x7fdfc4159ab0 2023-10-16T12:33:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc9409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40c86c0 con 0x7fdfc406c850 2023-10-16T12:33:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 0x7fdfc40af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 0x7fdfc40af700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45842/0 (socket says 172.21.15.60:45842) 2023-10-16T12:33:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1061318169 learned_addr learned my addr 172.21.15.60:0/1061318169 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc2ffd640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfc406c850 0x7fdfc40c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfbbfff640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfc4159ab0 0x7fdfc40c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfc406c850 msgr2=0x7fdfc40c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfc406c850 0x7fdfc40c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfc4159ab0 msgr2=0x7fdfc40c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfc4159ab0 0x7fdfc40c4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.059+0000 7fdfc37fe640 1 -- 172.21.15.60:0/1061318169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc009a70 con 0x7fdfc4159e90 2023-10-16T12:33:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.060+0000 7fdfc37fe640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 0x7fdfc40af700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009d90 tx=0x7fdfbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.060+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc02cea0 con 0x7fdfc4159e90 2023-10-16T12:33:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.060+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc040650 con 0x7fdfc4159e90 2023-10-16T12:33:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.060+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc040a00 con 0x7fdfc4159e90 2023-10-16T12:33:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.060+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc414db20 con 0x7fdfc4159e90 2023-10-16T12:33:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.060+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc414e0e0 con 0x7fdfc4159e90 2023-10-16T12:33:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.062+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc414d980 con 0x7fdfc4159e90 2023-10-16T12:33:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.062+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdfbc040b60 con 0x7fdfc4159e90 2023-10-16T12:33:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.063+0000 7fdfbb7fe640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdfa406ae70 0x7fdfa406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.063+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfbc0ac390 con 0x7fdfc4159e90 2023-10-16T12:33:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.064+0000 7fdfc2ffd640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdfa406ae70 0x7fdfa406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.065+0000 7fdfc2ffd640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdfa406ae70 0x7fdfa406d330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4005880 tx=0x7fdfb4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.068+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdfc414d980 con 0x7fdfc4159e90 2023-10-16T12:33:40.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.255+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdfc414d980 con 0x7fdfc4159e90 2023-10-16T12:33:40.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.256+0000 7fdfbb7fe640 1 -- 172.21.15.60:0/1061318169 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdfc414d980 con 0x7fdfc4159e90 2023-10-16T12:33:40.259 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:33:40.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdfa406ae70 msgr2=0x7fdfa406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdfa406ae70 0x7fdfa406d330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4005880 tx=0x7fdfb4009040 comp rx=0 tx=0).stop 2023-10-16T12:33:40.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 msgr2=0x7fdfc40af700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 0x7fdfc40af700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009d90 tx=0x7fdfbc004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:40.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 shutdown_connections 2023-10-16T12:33:40.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fdfa406ae70 0x7fdfa406d330 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfc4159e90 0x7fdfc40af700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfc4159ab0 0x7fdfc40c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 --2- 172.21.15.60:0/1061318169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfc406c850 0x7fdfc40c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 >> 172.21.15.60:0/1061318169 conn(0x7fdfc40ce9b0 msgr2=0x7fdfc406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:40.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 shutdown_connections 2023-10-16T12:33:40.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.259+0000 7fdfc9409640 1 -- 172.21.15.60:0/1061318169 wait complete. 2023-10-16T12:33:40.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:33:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.497+0000 7fc4affff640 1 Processor -- start 2023-10-16T12:33:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.497+0000 7fc4affff640 1 -- start start 2023-10-16T12:33:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.499+0000 7fc4affff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4b0160c80 0x7fc4b0161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.499+0000 7fc4affff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b00ae1a0 con 0x7fc4b00ad7a0 2023-10-16T12:33:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4affff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b00ae480 con 0x7fc4b0070950 2023-10-16T12:33:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4affff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b00ae760 con 0x7fc4b0160c80 2023-10-16T12:33:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4a7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc4b00ad7a0 0x7fc4b0070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49210/0 (socket says 172.21.15.60:49210) 2023-10-16T12:33:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4a7fff640 1 -- 172.21.15.60:0/657474648 learned_addr learned my addr 172.21.15.60:0/657474648 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4aeffd640 1 --2- 172.21.15.60:0/657474648 >> v2:172.21.15.60:3300/0 conn(0x7fc4b0160c80 0x7fc4b0161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4ae7fc640 1 -- 172.21.15.60:0/657474648 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711807570 0 0) 0x7fc4b00ae1a0 con 0x7fc4b00ad7a0 2023-10-16T12:33:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4ae7fc640 1 -- 172.21.15.60:0/657474648 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c003520 con 0x7fc4b00ad7a0 2023-10-16T12:33:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4aeffd640 1 -- 172.21.15.60:0/657474648 >> v1:172.21.15.144:6789/0 conn(0x7fc4b0070950 legacy=0x7fc4b0140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.500+0000 7fc4aeffd640 1 -- 172.21.15.60:0/657474648 >> v1:172.21.15.60:6789/0 conn(0x7fc4b00ad7a0 legacy=0x7fc4b0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.501+0000 7fc4aeffd640 1 -- 172.21.15.60:0/657474648 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b00aea40 con 0x7fc4b0160c80 2023-10-16T12:33:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.502+0000 7fc4aeffd640 1 --2- 172.21.15.60:0/657474648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0160c80 0x7fc4b0161060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00097a0 tx=0x7fc4a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d97c4a53791588 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.502+0000 7fc4ae7fc640 1 -- 172.21.15.60:0/657474648 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0030610 con 0x7fc4b0160c80 2023-10-16T12:33:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.502+0000 7fc4ae7fc640 1 -- 172.21.15.60:0/657474648 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a0030770 con 0x7fc4b0160c80 2023-10-16T12:33:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.502+0000 7fc4ae7fc640 1 -- 172.21.15.60:0/657474648 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0030ae0 con 0x7fc4b0160c80 2023-10-16T12:33:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 -- 172.21.15.60:0/657474648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0160c80 msgr2=0x7fc4b0161060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 --2- 172.21.15.60:0/657474648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0160c80 0x7fc4b0161060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00097a0 tx=0x7fc4a002b960 comp rx=0 tx=0).stop 2023-10-16T12:33:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 -- 172.21.15.60:0/657474648 shutdown_connections 2023-10-16T12:33:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 --2- 172.21.15.60:0/657474648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0160c80 0x7fc4b0161060 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 -- 172.21.15.60:0/657474648 >> 172.21.15.60:0/657474648 conn(0x7fc4b00cc230 msgr2=0x7fc4b00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 -- 172.21.15.60:0/657474648 shutdown_connections 2023-10-16T12:33:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.503+0000 7fc4affff640 1 -- 172.21.15.60:0/657474648 wait complete. 2023-10-16T12:33:40.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.505+0000 7fc4affff640 1 Processor -- start 2023-10-16T12:33:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.505+0000 7fc4affff640 1 -- start start 2023-10-16T12:33:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 0x7fc4b00ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4affff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b00ad7a0 0x7fc4b015b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4affff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b0160c80 0x7fc4b006aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4affff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00ac2e0 con 0x7fc4b0070950 2023-10-16T12:33:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4affff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b015bc00 con 0x7fc4b00ad7a0 2023-10-16T12:33:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4affff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00aba40 con 0x7fc4b0160c80 2023-10-16T12:33:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 0x7fc4b00ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 0x7fc4b00ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45852/0 (socket says 172.21.15.60:45852) 2023-10-16T12:33:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 -- 172.21.15.60:0/2083682108 learned_addr learned my addr 172.21.15.60:0/2083682108 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4a7fff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b00ad7a0 0x7fc4b015b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 -- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b0160c80 msgr2=0x7fc4b006aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4af7fe640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b0160c80 0x7fc4b006aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b0160c80 0x7fc4b006aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 -- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b00ad7a0 msgr2=0x7fc4b015b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b00ad7a0 0x7fc4b015b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4aeffd640 1 -- 172.21.15.60:0/2083682108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a0009400 con 0x7fc4b0070950 2023-10-16T12:33:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.506+0000 7fc4a7fff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b00ad7a0 0x7fc4b015b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.507+0000 7fc4aeffd640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 0x7fc4b00ab430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4a002bf70 tx=0x7fc4a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.507+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a003e070 con 0x7fc4b0070950 2023-10-16T12:33:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.507+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a0030e10 con 0x7fc4b0070950 2023-10-16T12:33:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.507+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0039c50 con 0x7fc4b0070950 2023-10-16T12:33:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.507+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b00abd20 con 0x7fc4b0070950 2023-10-16T12:33:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.507+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b00a85b0 con 0x7fc4b0070950 2023-10-16T12:33:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.508+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b004e9b0 con 0x7fc4b0070950 2023-10-16T12:33:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.509+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc4a004b400 con 0x7fc4b0070950 2023-10-16T12:33:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.510+0000 7fc4a77fe640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49006aec0 0x7fc49006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.510+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4a00ab830 con 0x7fc4b0070950 2023-10-16T12:33:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.511+0000 7fc4a7fff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49006aec0 0x7fc49006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.512+0000 7fc4a7fff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49006aec0 0x7fc49006d380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00ab970 tx=0x7fc498009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.515+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4a0076b00 con 0x7fc4b0070950 2023-10-16T12:33:40.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.716+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4b00cf230 con 0x7fc4b0070950 2023-10-16T12:33:40.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.717+0000 7fc4a77fe640 1 -- 172.21.15.60:0/2083682108 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4a0039db0 con 0x7fc4b0070950 2023-10-16T12:33:40.719 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:33:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49006aec0 msgr2=0x7fc49006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49006aec0 0x7fc49006d380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00ab970 tx=0x7fc498009040 comp rx=0 tx=0).stop 2023-10-16T12:33:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 msgr2=0x7fc4b00ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 0x7fc4b00ab430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4a002bf70 tx=0x7fc4a00046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 shutdown_connections 2023-10-16T12:33:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49006aec0 0x7fc49006d380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b0160c80 0x7fc4b006aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b00ad7a0 0x7fc4b015b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 --2- 172.21.15.60:0/2083682108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b0070950 0x7fc4b00ab430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 >> 172.21.15.60:0/2083682108 conn(0x7fc4b00cc230 msgr2=0x7fc4b015f340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:40.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 shutdown_connections 2023-10-16T12:33:40.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.720+0000 7fc4affff640 1 -- 172.21.15.60:0/2083682108 wait complete. 2023-10-16T12:33:40.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:33:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.960+0000 7f4194c15640 1 Processor -- start 2023-10-16T12:33:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.961+0000 7f4194c15640 1 -- start start 2023-10-16T12:33:40.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.962+0000 7f4194c15640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f419015af80 0x7f41900cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.962+0000 7f4194c15640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419015eff0 con 0x7f419006faf0 2023-10-16T12:33:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.962+0000 7f4194c15640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419015ecc0 con 0x7f4190065ab0 2023-10-16T12:33:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.962+0000 7f4194c15640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41900d0930 con 0x7f419015af80 2023-10-16T12:33:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418dd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f419006faf0 0x7f419015f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49224/0 (socket says 172.21.15.60:49224) 2023-10-16T12:33:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418dd9b640 1 -- 172.21.15.60:0/222572937 learned_addr learned my addr 172.21.15.60:0/222572937 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:40.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418e59c640 1 --2- 172.21.15.60:0/222572937 >> v2:172.21.15.144:3300/0 conn(0x7f419015af80 0x7f41900cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205584428 0 0) 0x7f419015eff0 con 0x7f419006faf0 2023-10-16T12:33:40.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f417c003520 con 0x7f419006faf0 2023-10-16T12:33:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 860690869 0 0) 0x7f417c003520 con 0x7f419006faf0 2023-10-16T12:33:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 >> v2:172.21.15.144:3300/0 conn(0x7f419015af80 msgr2=0x7f41900cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.963+0000 7f418d59a640 1 --2- 172.21.15.60:0/222572937 >> v2:172.21.15.144:3300/0 conn(0x7f419015af80 0x7f41900cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 >> v1:172.21.15.144:6789/0 conn(0x7f4190065ab0 legacy=0x7f41900ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419015b360 con 0x7f419006faf0 2023-10-16T12:33:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4178001670 con 0x7f419006faf0 2023-10-16T12:33:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 >> v1:172.21.15.60:6789/0 conn(0x7f419006faf0 legacy=0x7f419015f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f417c003970 0x7f417c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417c004390 0x7f417c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c005100 0x7f417c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41900d0930 con 0x7f417c004390 2023-10-16T12:33:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c00c050 con 0x7f417c005100 2023-10-16T12:33:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c00c210 con 0x7f417c003970 2023-10-16T12:33:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418dd9b640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417c004390 0x7f417c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418e59c640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f417c003970 0x7f417c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418dd9b640 1 -- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f417c003970 msgr2=0x7f417c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418dd9b640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f417c003970 0x7f417c003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418dd9b640 1 -- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c005100 msgr2=0x7f417c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418dd9b640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c005100 0x7f417c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418dd9b640 1 -- 172.21.15.60:0/222572937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c003520 con 0x7f417c004390 2023-10-16T12:33:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.964+0000 7f418ed9d640 1 -- 172.21.15.60:0/222572937 reap_dead start 2023-10-16T12:33:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.966+0000 7f418dd9b640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417c004390 0x7f417c004860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f417800c190 tx=0x7f417c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7f22ec57a2e089 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.966+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178005ad0 con 0x7f417c004390 2023-10-16T12:33:40.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.966+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4178005c30 con 0x7f417c004390 2023-10-16T12:33:40.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.966+0000 7f418d59a640 1 -- 172.21.15.60:0/222572937 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417802c7c0 con 0x7f417c004390 2023-10-16T12:33:40.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.966+0000 7f4194c15640 1 -- 172.21.15.60:0/222572937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417c004390 msgr2=0x7f417c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.966+0000 7f4194c15640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417c004390 0x7f417c004860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f417800c190 tx=0x7f417c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.967+0000 7f4194c15640 1 -- 172.21.15.60:0/222572937 shutdown_connections 2023-10-16T12:33:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.967+0000 7f4194c15640 1 --2- 172.21.15.60:0/222572937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417c004390 0x7f417c004860 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.967+0000 7f4194c15640 1 -- 172.21.15.60:0/222572937 >> 172.21.15.60:0/222572937 conn(0x7f41900d0ab0 msgr2=0x7f41900d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.967+0000 7f4194c15640 1 -- 172.21.15.60:0/222572937 shutdown_connections 2023-10-16T12:33:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.967+0000 7f4194c15640 1 -- 172.21.15.60:0/222572937 wait complete. 2023-10-16T12:33:40.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.969+0000 7f4194c15640 1 Processor -- start 2023-10-16T12:33:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.969+0000 7f4194c15640 1 -- start start 2023-10-16T12:33:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f4194c15640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c004390 0x7f419015a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f4194c15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 0x7f4190149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f4194c15640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41900b5a10 0x7f41900ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f4194c15640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4190149a60 con 0x7f4190157d80 2023-10-16T12:33:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f4194c15640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41900b4720 con 0x7f417c004390 2023-10-16T12:33:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f4194c15640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41900b5580 con 0x7f41900b5a10 2023-10-16T12:33:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f418dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 0x7f4190149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f418dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 0x7f4190149520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45874/0 (socket says 172.21.15.60:45874) 2023-10-16T12:33:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f418dd9b640 1 -- 172.21.15.60:0/419414998 learned_addr learned my addr 172.21.15.60:0/419414998 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f418e59c640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c004390 0x7f419015a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f418dd9b640 1 -- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41900b5a10 msgr2=0x7f41900ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.970+0000 7f418dd9b640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41900b5a10 0x7f41900ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f418dd9b640 1 -- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c004390 msgr2=0x7f419015a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:40.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f418dd9b640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c004390 0x7f419015a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:40.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f418dd9b640 1 -- 172.21.15.60:0/419414998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41900b4970 con 0x7f4190157d80 2023-10-16T12:33:40.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f418dd9b640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 0x7f4190149520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4178000f60 tx=0x7f417802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417803c040 con 0x7f4190157d80 2023-10-16T12:33:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417802cd90 con 0x7f4190157d80 2023-10-16T12:33:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41780072b0 con 0x7f4190157d80 2023-10-16T12:33:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41900b4c50 con 0x7f4190157d80 2023-10-16T12:33:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.971+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41900b5210 con 0x7f4190157d80 2023-10-16T12:33:40.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.972+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419004e160 con 0x7f4190157d80 2023-10-16T12:33:40.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.973+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4178032070 con 0x7f4190157d80 2023-10-16T12:33:40.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.974+0000 7f4186ffd640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f417006ae70 0x7f417006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:40.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.974+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41780aaa50 con 0x7f4190157d80 2023-10-16T12:33:40.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.975+0000 7f418e59c640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f417006ae70 0x7f417006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:40.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.976+0000 7f418e59c640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f417006ae70 0x7f417006d330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4180007520 tx=0x7f418000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:40.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:40.979+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4178075af0 con 0x7f4190157d80 2023-10-16T12:33:41.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.174+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41900b56c0 con 0x7f4190157d80 2023-10-16T12:33:41.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.175+0000 7f4186ffd640 1 -- 172.21.15.60:0/419414998 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4178079110 con 0x7f4190157d80 2023-10-16T12:33:41.177 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:33:41.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.177+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f417006ae70 msgr2=0x7f417006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.177+0000 7f4194c15640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f417006ae70 0x7f417006d330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4180007520 tx=0x7f418000b040 comp rx=0 tx=0).stop 2023-10-16T12:33:41.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.177+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 msgr2=0x7f4190149520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.177+0000 7f4194c15640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 0x7f4190149520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4178000f60 tx=0x7f417802e070 comp rx=0 tx=0).stop 2023-10-16T12:33:41.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 shutdown_connections 2023-10-16T12:33:41.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f417006ae70 0x7f417006d330 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41900b5a10 0x7f41900ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4190157d80 0x7f4190149520 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 --2- 172.21.15.60:0/419414998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f417c004390 0x7f419015a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 >> 172.21.15.60:0/419414998 conn(0x7f41900d0ab0 msgr2=0x7f41900c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:41.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 shutdown_connections 2023-10-16T12:33:41.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.178+0000 7f4194c15640 1 -- 172.21.15.60:0/419414998 wait complete. 2023-10-16T12:33:41.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:33:41.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.403+0000 7fd8af599640 1 Processor -- start 2023-10-16T12:33:41.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.403+0000 7fd8af599640 1 -- start start 2023-10-16T12:33:41.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8af599640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8a80ad970 0x7fd8a80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8af599640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a815b9d0 con 0x7fd8a80ccfb0 2023-10-16T12:33:41.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8af599640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a815bcb0 con 0x7fd8a806c040 2023-10-16T12:33:41.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8af599640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a815bf90 con 0x7fd8a80ad970 2023-10-16T12:33:41.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8aed98640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd8a80ccfb0 0x7fd8a80cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49228/0 (socket says 172.21.15.60:49228) 2023-10-16T12:33:41.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8aed98640 1 -- 172.21.15.60:0/2712970959 learned_addr learned my addr 172.21.15.60:0/2712970959 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:41.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8add96640 1 --2- 172.21.15.60:0/2712970959 >> v2:172.21.15.144:3300/0 conn(0x7fd8a80ad970 0x7fd8a80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.405+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275466574 0 0) 0x7fd8a815b9d0 con 0x7fd8a80ccfb0 2023-10-16T12:33:41.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd894003520 con 0x7fd8a80ccfb0 2023-10-16T12:33:41.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102600491 0 0) 0x7fd894003520 con 0x7fd8a80ccfb0 2023-10-16T12:33:41.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 >> v2:172.21.15.144:3300/0 conn(0x7fd8a80ad970 msgr2=0x7fd8a80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:41.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 --2- 172.21.15.60:0/2712970959 >> v2:172.21.15.144:3300/0 conn(0x7fd8a80ad970 0x7fd8a80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 >> v1:172.21.15.144:6789/0 conn(0x7fd8a806c040 legacy=0x7fd8a806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a80cc270 con 0x7fd8a80ccfb0 2023-10-16T12:33:41.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd8a0001670 con 0x7fd8a80ccfb0 2023-10-16T12:33:41.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 >> v1:172.21.15.60:6789/0 conn(0x7fd8a80ccfb0 legacy=0x7fd8a80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd894003970 0x7fd894003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940047c0 0x7fd894004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd894005180 0x7fd89400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a815bf90 con 0x7fd8940047c0 2023-10-16T12:33:41.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89400c020 con 0x7fd894005180 2023-10-16T12:33:41.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89400c1e0 con 0x7fd894003970 2023-10-16T12:33:41.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.406+0000 7fd8add96640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940047c0 0x7fd894004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8aed98640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd894005180 0x7fd89400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8add96640 1 -- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd894003970 msgr2=0x7fd894003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:41.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8ae597640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd894003970 0x7fd894003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8add96640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd894003970 0x7fd894003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8add96640 1 -- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd894005180 msgr2=0x7fd89400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:41.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8add96640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd894005180 0x7fd89400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8add96640 1 -- 172.21.15.60:0/2712970959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894003520 con 0x7fd8940047c0 2023-10-16T12:33:41.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8aed98640 1 -- 172.21.15.60:0/2712970959 reap_dead start 2023-10-16T12:33:41.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.407+0000 7fd8ae597640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd894003970 0x7fd894003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:41.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.408+0000 7fd8add96640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940047c0 0x7fd894004c40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd898009a90 tx=0x7fd8940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc058c6c70798f0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:41.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.408+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89802c680 con 0x7fd8940047c0 2023-10-16T12:33:41.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.408+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89802c7e0 con 0x7fd8940047c0 2023-10-16T12:33:41.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.408+0000 7fd8ad595640 1 -- 172.21.15.60:0/2712970959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89802cb90 con 0x7fd8940047c0 2023-10-16T12:33:41.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 -- 172.21.15.60:0/2712970959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940047c0 msgr2=0x7fd894004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940047c0 0x7fd894004c40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd898009a90 tx=0x7fd8940057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 -- 172.21.15.60:0/2712970959 shutdown_connections 2023-10-16T12:33:41.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 --2- 172.21.15.60:0/2712970959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940047c0 0x7fd894004c40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 -- 172.21.15.60:0/2712970959 >> 172.21.15.60:0/2712970959 conn(0x7fd8a80ce6a0 msgr2=0x7fd8a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 -- 172.21.15.60:0/2712970959 shutdown_connections 2023-10-16T12:33:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.409+0000 7fd8af599640 1 -- 172.21.15.60:0/2712970959 wait complete. 2023-10-16T12:33:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.411+0000 7fd8af599640 1 Processor -- start 2023-10-16T12:33:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.411+0000 7fd8af599640 1 -- start start 2023-10-16T12:33:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8af599640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8940047c0 0x7fd8a80ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8af599640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 0x7fd8a80c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8af599640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8a80c6c90 0x7fd8a8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8af599640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a81e4db0 con 0x7fd8a80aff90 2023-10-16T12:33:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8af599640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a80c8570 con 0x7fd8940047c0 2023-10-16T12:33:41.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8af599640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a80c7200 con 0x7fd8a80c6c90 2023-10-16T12:33:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 0x7fd8a80c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 0x7fd8a80c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45886/0 (socket says 172.21.15.60:45886) 2023-10-16T12:33:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 -- 172.21.15.60:0/2459800351 learned_addr learned my addr 172.21.15.60:0/2459800351 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8aed98640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8a80c6c90 0x7fd8a8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8ae597640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8940047c0 0x7fd8a80ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 -- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8a80c6c90 msgr2=0x7fd8a8158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8a80c6c90 0x7fd8a8158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 -- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8940047c0 msgr2=0x7fd8a80ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8940047c0 0x7fd8a80ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.412+0000 7fd8add96640 1 -- 172.21.15.60:0/2459800351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a81e4480 con 0x7fd8a80aff90 2023-10-16T12:33:41.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd8ae597640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8940047c0 0x7fd8a80ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd8add96640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 0x7fd8a80c8030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd898009bb0 tx=0x7fd898004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898034a90 con 0x7fd8a80aff90 2023-10-16T12:33:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898034c10 con 0x7fd8a80aff90 2023-10-16T12:33:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89803e400 con 0x7fd8a80aff90 2023-10-16T12:33:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a81e4760 con 0x7fd8a80aff90 2023-10-16T12:33:41.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.413+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a8158980 con 0x7fd8a80aff90 2023-10-16T12:33:41.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.415+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a81e4f20 con 0x7fd8a80aff90 2023-10-16T12:33:41.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.415+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd89803e560 con 0x7fd8a80aff90 2023-10-16T12:33:41.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.417+0000 7fd89effd640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd88c06ae70 0x7fd88c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.417+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8980ab650 con 0x7fd8a80aff90 2023-10-16T12:33:41.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.418+0000 7fd8ae597640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd88c06ae70 0x7fd88c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.419+0000 7fd8ae597640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd88c06ae70 0x7fd88c06d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4000f80 tx=0x7fd8a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:41.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.422+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd898076710 con 0x7fd8a80aff90 2023-10-16T12:33:41.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.659+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8a804e9b0 con 0x7fd8a80aff90 2023-10-16T12:33:41.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.660+0000 7fd89effd640 1 -- 172.21.15.60:0/2459800351 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fd898079d30 con 0x7fd8a80aff90 2023-10-16T12:33:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:41.663 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:33:41.664 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:33:41.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd88c06ae70 msgr2=0x7fd88c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd88c06ae70 0x7fd88c06d330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4000f80 tx=0x7fd8a4009040 comp rx=0 tx=0).stop 2023-10-16T12:33:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 msgr2=0x7fd8a80c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 0x7fd8a80c8030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd898009bb0 tx=0x7fd898004730 comp rx=0 tx=0).stop 2023-10-16T12:33:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 shutdown_connections 2023-10-16T12:33:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd88c06ae70 0x7fd88c06d330 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8a80c6c90 0x7fd8a8158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8a80aff90 0x7fd8a80c8030 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 --2- 172.21.15.60:0/2459800351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8940047c0 0x7fd8a80ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 >> 172.21.15.60:0/2459800351 conn(0x7fd8a80ce6a0 msgr2=0x7fd8a80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 shutdown_connections 2023-10-16T12:33:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.663+0000 7fd8af599640 1 -- 172.21.15.60:0/2459800351 wait complete. 2023-10-16T12:33:41.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.905+0000 7f8cadec2640 1 Processor -- start 2023-10-16T12:33:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.905+0000 7f8cadec2640 1 -- start start 2023-10-16T12:33:41.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.906+0000 7f8cadec2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8ca815aaa0 0x7f8ca815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8cadec2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca80ce390 0x7f8ca8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8cadec2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca815ed60 con 0x7f8ca815ae80 2023-10-16T12:33:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8cadec2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca806b810 con 0x7f8ca80ce390 2023-10-16T12:33:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8cadec2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca80707e0 con 0x7f8ca815aaa0 2023-10-16T12:33:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca80ce390 0x7f8ca8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca80ce390 0x7f8ca8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45894/0 (socket says 172.21.15.60:45894) 2023-10-16T12:33:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 -- 172.21.15.60:0/377018782 learned_addr learned my addr 172.21.15.60:0/377018782 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca77fe640 1 --2- 172.21.15.60:0/377018782 >> v2:172.21.15.144:3300/0 conn(0x7f8ca815aaa0 0x7f8ca815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 -- 172.21.15.60:0/377018782 >> v2:172.21.15.144:3300/0 conn(0x7f8ca815aaa0 msgr2=0x7f8ca815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 --2- 172.21.15.60:0/377018782 >> v2:172.21.15.144:3300/0 conn(0x7f8ca815aaa0 0x7f8ca815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 -- 172.21.15.60:0/377018782 >> v1:172.21.15.144:6789/0 conn(0x7f8ca815ae80 legacy=0x7f8ca80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.907+0000 7f8ca7fff640 1 -- 172.21.15.60:0/377018782 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca815ef90 con 0x7f8ca80ce390 2023-10-16T12:33:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.909+0000 7f8ca7fff640 1 --2- 172.21.15.60:0/377018782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80ce390 0x7f8ca8070f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80cd990 tx=0x7f8ca002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcec685c15c04f38 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.909+0000 7f8ca6ffd640 1 -- 172.21.15.60:0/377018782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca002ca80 con 0x7f8ca80ce390 2023-10-16T12:33:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.909+0000 7f8ca6ffd640 1 -- 172.21.15.60:0/377018782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca002cbe0 con 0x7f8ca80ce390 2023-10-16T12:33:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.909+0000 7f8ca6ffd640 1 -- 172.21.15.60:0/377018782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca00343f0 con 0x7f8ca80ce390 2023-10-16T12:33:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.909+0000 7f8cadec2640 1 -- 172.21.15.60:0/377018782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80ce390 msgr2=0x7f8ca8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.909+0000 7f8cadec2640 1 --2- 172.21.15.60:0/377018782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80ce390 0x7f8ca8070f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80cd990 tx=0x7f8ca002bee0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.910+0000 7f8cadec2640 1 -- 172.21.15.60:0/377018782 shutdown_connections 2023-10-16T12:33:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.910+0000 7f8cadec2640 1 --2- 172.21.15.60:0/377018782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80ce390 0x7f8ca8070f50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.910+0000 7f8cadec2640 1 --2- 172.21.15.60:0/377018782 >> v2:172.21.15.144:3300/0 conn(0x7f8ca815aaa0 0x7f8ca815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.910+0000 7f8cadec2640 1 -- 172.21.15.60:0/377018782 >> 172.21.15.60:0/377018782 conn(0x7f8ca80cb410 msgr2=0x7f8ca80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.910+0000 7f8cadec2640 1 -- 172.21.15.60:0/377018782 shutdown_connections 2023-10-16T12:33:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.910+0000 7f8cadec2640 1 -- 172.21.15.60:0/377018782 wait complete. 2023-10-16T12:33:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.912+0000 7f8cadec2640 1 Processor -- start 2023-10-16T12:33:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.912+0000 7f8cadec2640 1 -- start start 2023-10-16T12:33:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8cadec2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80ce390 0x7f8ca814ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8cadec2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 0x7f8ca815b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8cadec2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca815ae80 0x7f8ca80c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8cadec2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80c9020 con 0x7f8ca815aaa0 2023-10-16T12:33:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8cadec2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80c9270 con 0x7f8ca80ce390 2023-10-16T12:33:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8cadec2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80c9550 con 0x7f8ca815ae80 2023-10-16T12:33:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8c9effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 0x7f8ca815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8c9effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 0x7f8ca815b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45902/0 (socket says 172.21.15.60:45902) 2023-10-16T12:33:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8c9effd640 1 -- 172.21.15.60:0/3202077804 learned_addr learned my addr 172.21.15.60:0/3202077804 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8ca77fe640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80ce390 0x7f8ca814ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.913+0000 7f8ca7fff640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca815ae80 0x7f8ca80c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8c9effd640 1 -- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca815ae80 msgr2=0x7f8ca80c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8c9effd640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca815ae80 0x7f8ca80c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8c9effd640 1 -- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80ce390 msgr2=0x7f8ca814ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8c9effd640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80ce390 0x7f8ca814ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8c9effd640 1 -- 172.21.15.60:0/3202077804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0009b30 con 0x7f8ca815aaa0 2023-10-16T12:33:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8ca77fe640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80ce390 0x7f8ca814ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8c9effd640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 0x7f8ca815b320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000b3c0 tx=0x7f8c9000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90011830 con 0x7f8ca815aaa0 2023-10-16T12:33:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c90011990 con 0x7f8ca815aaa0 2023-10-16T12:33:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90011d80 con 0x7f8ca815aaa0 2023-10-16T12:33:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca80c9830 con 0x7f8ca815aaa0 2023-10-16T12:33:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.914+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca814f470 con 0x7f8ca815aaa0 2023-10-16T12:33:41.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.915+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca804e160 con 0x7f8ca815aaa0 2023-10-16T12:33:41.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.917+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c9001b460 con 0x7f8ca815aaa0 2023-10-16T12:33:41.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.917+0000 7f8ca4ff9640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c8806ae70 0x7f8c8806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:41.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.918+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c90089170 con 0x7f8ca815aaa0 2023-10-16T12:33:41.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.918+0000 7f8ca77fe640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c8806ae70 0x7f8c8806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:41.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.919+0000 7f8ca77fe640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c8806ae70 0x7f8c8806d330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8c98006fd0 tx=0x7f8c98008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:41.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:41.922+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c90054210 con 0x7f8ca815aaa0 2023-10-16T12:33:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.117+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ca80cdc60 con 0x7f8ca815aaa0 2023-10-16T12:33:42.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.120+0000 7f8ca4ff9640 1 -- 172.21.15.60:0/3202077804 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f8c90057830 con 0x7f8ca815aaa0 2023-10-16T12:33:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:42.123 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.122+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c8806ae70 msgr2=0x7f8c8806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.122+0000 7f8cadec2640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c8806ae70 0x7f8c8806d330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8c98006fd0 tx=0x7f8c98008040 comp rx=0 tx=0).stop 2023-10-16T12:33:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.122+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 msgr2=0x7f8ca815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.122+0000 7f8cadec2640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 0x7f8ca815b320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000b3c0 tx=0x7f8c9000b880 comp rx=0 tx=0).stop 2023-10-16T12:33:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 shutdown_connections 2023-10-16T12:33:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c8806ae70 0x7f8c8806d330 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca815ae80 0x7f8ca80c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca815aaa0 0x7f8ca815b320 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 --2- 172.21.15.60:0/3202077804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80ce390 0x7f8ca814ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 >> 172.21.15.60:0/3202077804 conn(0x7f8ca80cb410 msgr2=0x7f8ca8158c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 shutdown_connections 2023-10-16T12:33:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.123+0000 7f8cadec2640 1 -- 172.21.15.60:0/3202077804 wait complete. 2023-10-16T12:33:42.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:33:42.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.372+0000 7f130a38e640 1 Processor -- start 2023-10-16T12:33:42.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.372+0000 7f130a38e640 1 -- start start 2023-10-16T12:33:42.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.374+0000 7f130a38e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1304158ed0 0x7f13040ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:42.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.374+0000 7f130a38e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 0x7f13040b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.374+0000 7f130a38e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130415ce30 con 0x7f13040ce1a0 2023-10-16T12:33:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.374+0000 7f130a38e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130415d9d0 con 0x7f1304158ed0 2023-10-16T12:33:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.374+0000 7f130a38e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040b0d50 con 0x7f13040b0410 2023-10-16T12:33:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 0x7f13040b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f1303fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1304158ed0 0x7f13040ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 0x7f13040b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:50832/0 (socket says 172.21.15.60:50832) 2023-10-16T12:33:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 -- 172.21.15.60:0/1607437953 learned_addr learned my addr 172.21.15.60:0/1607437953 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 -- 172.21.15.60:0/1607437953 >> v2:172.21.15.144:3300/0 conn(0x7f1304158ed0 msgr2=0x7f13040ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 --2- 172.21.15.60:0/1607437953 >> v2:172.21.15.144:3300/0 conn(0x7f1304158ed0 0x7f13040ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 -- 172.21.15.60:0/1607437953 >> v1:172.21.15.144:6789/0 conn(0x7f13040ce1a0 legacy=0x7f130415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.375+0000 7f130892b640 1 -- 172.21.15.60:0/1607437953 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040c9080 con 0x7f13040b0410 2023-10-16T12:33:42.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.377+0000 7f130892b640 1 --2- 172.21.15.60:0/1607437953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 0x7f13040b0810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009e50 tx=0x7f12fc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f37f07f663d8780 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:42.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.377+0000 7f1302ffd640 1 -- 172.21.15.60:0/1607437953 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc02ca70 con 0x7f13040b0410 2023-10-16T12:33:42.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.377+0000 7f1302ffd640 1 -- 172.21.15.60:0/1607437953 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc02cbd0 con 0x7f13040b0410 2023-10-16T12:33:42.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.377+0000 7f1302ffd640 1 -- 172.21.15.60:0/1607437953 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0343f0 con 0x7f13040b0410 2023-10-16T12:33:42.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 -- 172.21.15.60:0/1607437953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 msgr2=0x7f13040b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 --2- 172.21.15.60:0/1607437953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 0x7f13040b0810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009e50 tx=0x7f12fc02bed0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 -- 172.21.15.60:0/1607437953 shutdown_connections 2023-10-16T12:33:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 --2- 172.21.15.60:0/1607437953 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040b0410 0x7f13040b0810 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 --2- 172.21.15.60:0/1607437953 >> v2:172.21.15.144:3300/0 conn(0x7f1304158ed0 0x7f13040ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 -- 172.21.15.60:0/1607437953 >> 172.21.15.60:0/1607437953 conn(0x7f13040cb410 msgr2=0x7f13040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 -- 172.21.15.60:0/1607437953 shutdown_connections 2023-10-16T12:33:42.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.378+0000 7f130a38e640 1 -- 172.21.15.60:0/1607437953 wait complete. 2023-10-16T12:33:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.380+0000 7f130a38e640 1 Processor -- start 2023-10-16T12:33:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.380+0000 7f130a38e640 1 -- start start 2023-10-16T12:33:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f130a38e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 0x7f13041e7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f130a38e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040ce1a0 0x7f1304156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f130a38e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1304158ed0 0x7f1304156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:42.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f130a38e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040c3990 con 0x7f13040b0410 2023-10-16T12:33:42.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f130a38e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13041e79a0 con 0x7f1304158ed0 2023-10-16T12:33:42.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f130a38e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040c2d10 con 0x7f13040ce1a0 2023-10-16T12:33:42.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f1303fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 0x7f13041e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:42.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f1303fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 0x7f13041e7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45912/0 (socket says 172.21.15.60:45912) 2023-10-16T12:33:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f1303fff640 1 -- 172.21.15.60:0/3927825885 learned_addr learned my addr 172.21.15.60:0/3927825885 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f1303fff640 1 -- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040ce1a0 msgr2=0x7f1304156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f13037fe640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040ce1a0 0x7f1304156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.381+0000 7f1303fff640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040ce1a0 0x7f1304156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1303fff640 1 -- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1304158ed0 msgr2=0x7f1304156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1303fff640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1304158ed0 0x7f1304156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1303fff640 1 -- 172.21.15.60:0/3927825885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc009b30 con 0x7f13040b0410 2023-10-16T12:33:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1303fff640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 0x7f13041e7460 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f400e690 tx=0x7f12f400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f400ca30 con 0x7f13040b0410 2023-10-16T12:33:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f400cb90 con 0x7f13040b0410 2023-10-16T12:33:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13040c2ff0 con 0x7f13040b0410 2023-10-16T12:33:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f40113f0 con 0x7f13040b0410 2023-10-16T12:33:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.382+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13040c35b0 con 0x7f13040b0410 2023-10-16T12:33:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.384+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130404e160 con 0x7f13040b0410 2023-10-16T12:33:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.384+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f12f4011550 con 0x7f13040b0410 2023-10-16T12:33:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.385+0000 7f1300ff9640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f12f006ae70 0x7f12f006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.385+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12f40892e0 con 0x7f13040b0410 2023-10-16T12:33:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.386+0000 7f13037fe640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f12f006ae70 0x7f12f006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.387+0000 7f13037fe640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f12f006ae70 0x7f12f006d330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f13040c54a0 tx=0x7f12f8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:42.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.390+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12f4054390 con 0x7f13040b0410 2023-10-16T12:33:42.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.627+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13040b4910 con 0x7f13040b0410 2023-10-16T12:33:42.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.627+0000 7f1300ff9640 1 -- 172.21.15.60:0/3927825885 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f12f40579b0 con 0x7f13040b0410 2023-10-16T12:33:42.630 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:42.631 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:33:42.631 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:33:42.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f12f006ae70 msgr2=0x7f12f006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f12f006ae70 0x7f12f006d330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f13040c54a0 tx=0x7f12f8009040 comp rx=0 tx=0).stop 2023-10-16T12:33:42.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 msgr2=0x7f13041e7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:42.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 0x7f13041e7460 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f400e690 tx=0x7f12f400eb50 comp rx=0 tx=0).stop 2023-10-16T12:33:42.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 shutdown_connections 2023-10-16T12:33:42.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f12f006ae70 0x7f12f006d330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13040ce1a0 0x7f1304156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1304158ed0 0x7f1304156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 --2- 172.21.15.60:0/3927825885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13040b0410 0x7f13041e7460 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:42.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 >> 172.21.15.60:0/3927825885 conn(0x7f13040cb410 msgr2=0x7f130406c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:42.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 shutdown_connections 2023-10-16T12:33:42.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:42.630+0000 7f130a38e640 1 -- 172.21.15.60:0/3927825885 wait complete. 2023-10-16T12:33:43.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:33:24.434648+0000', 'modified': '2023-10-16T12:33:28.085691+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.144:6837/872963903', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6836', 'nonce': 872963903}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 872963903}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:33:43.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:33:43.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T12:33:43.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.290+0000 7f9707e0e640 1 Processor -- start 2023-10-16T12:33:43.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.290+0000 7f9707e0e640 1 -- start start 2023-10-16T12:33:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.291+0000 7f9707e0e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f97000cf710 0x7f97000b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.291+0000 7f9707e0e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f97000b6cd0 0x7f97000b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.291+0000 7f9707e0e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97000d3980 con 0x7f97000cfaf0 2023-10-16T12:33:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.291+0000 7f9707e0e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97000d3c60 con 0x7f97000cf710 2023-10-16T12:33:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.291+0000 7f9707e0e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97000ce630 con 0x7f97000b6cd0 2023-10-16T12:33:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f9705baa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f97000cf710 0x7f97000b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f97053a9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f97000cfaf0 0x7f97000d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49238/0 (socket says 172.21.15.60:49238) 2023-10-16T12:33:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f97053a9640 1 -- 172.21.15.60:0/1179403903 learned_addr learned my addr 172.21.15.60:0/1179403903 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f97063ab640 1 --2- 172.21.15.60:0/1179403903 >> v2:172.21.15.144:3300/0 conn(0x7f97000b6cd0 0x7f97000b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f9705baa640 1 -- 172.21.15.60:0/1179403903 >> v2:172.21.15.144:3300/0 conn(0x7f97000b6cd0 msgr2=0x7f97000b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f9705baa640 1 --2- 172.21.15.60:0/1179403903 >> v2:172.21.15.144:3300/0 conn(0x7f97000b6cd0 0x7f97000b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f9705baa640 1 -- 172.21.15.60:0/1179403903 >> v1:172.21.15.60:6789/0 conn(0x7f97000cfaf0 legacy=0x7f97000d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f9705baa640 1 -- 172.21.15.60:0/1179403903 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97000ce910 con 0x7f97000cf710 2023-10-16T12:33:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f97063ab640 1 --2- 172.21.15.60:0/1179403903 >> v2:172.21.15.144:3300/0 conn(0x7f97000b6cd0 0x7f97000b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.292+0000 7f9704ba8640 1 -- 172.21.15.60:0/1179403903 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547489580 0 0) 0x7f97000d3980 con 0x7f97000cfaf0 2023-10-16T12:33:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.294+0000 7f9705baa640 1 --2- 172.21.15.60:0/1179403903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000b2d10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97000b5890 tx=0x7f96fc02be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae421fda356c82f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.294+0000 7f9704ba8640 1 -- 172.21.15.60:0/1179403903 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc02cc20 con 0x7f97000cf710 2023-10-16T12:33:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.294+0000 7f9704ba8640 1 -- 172.21.15.60:0/1179403903 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc02cd80 con 0x7f97000cf710 2023-10-16T12:33:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.294+0000 7f9704ba8640 1 -- 172.21.15.60:0/1179403903 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc034660 con 0x7f97000cf710 2023-10-16T12:33:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 -- 172.21.15.60:0/1179403903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 msgr2=0x7f97000b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 --2- 172.21.15.60:0/1179403903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000b2d10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97000b5890 tx=0x7f96fc02be70 comp rx=0 tx=0).stop 2023-10-16T12:33:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 -- 172.21.15.60:0/1179403903 shutdown_connections 2023-10-16T12:33:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 --2- 172.21.15.60:0/1179403903 >> v2:172.21.15.144:3300/0 conn(0x7f97000b6cd0 0x7f97000b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 --2- 172.21.15.60:0/1179403903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000b2d10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 -- 172.21.15.60:0/1179403903 >> 172.21.15.60:0/1179403903 conn(0x7f97000d2cc0 msgr2=0x7f97000d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:43.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 -- 172.21.15.60:0/1179403903 shutdown_connections 2023-10-16T12:33:43.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.295+0000 7f9707e0e640 1 -- 172.21.15.60:0/1179403903 wait complete. 2023-10-16T12:33:43.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.297+0000 7f9707e0e640 1 Processor -- start 2023-10-16T12:33:43.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.297+0000 7f9707e0e640 1 -- start start 2023-10-16T12:33:43.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9707e0e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97000b6cd0 0x7f97000c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9707e0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9707e0e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97000cfaf0 0x7f97000c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9707e0e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97001e5b80 con 0x7f97000cf710 2023-10-16T12:33:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9707e0e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97000c91a0 con 0x7f97000cfaf0 2023-10-16T12:33:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9707e0e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97000ccef0 con 0x7f97000b6cd0 2023-10-16T12:33:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f97053a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f97053a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45942/0 (socket says 172.21.15.60:45942) 2023-10-16T12:33:43.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f97053a9640 1 -- 172.21.15.60:0/3568415084 learned_addr learned my addr 172.21.15.60:0/3568415084 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:43.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.298+0000 7f9705baa640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97000b6cd0 0x7f97000c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97063ab640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97000cfaf0 0x7f97000c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97053a9640 1 -- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97000b6cd0 msgr2=0x7f97000c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97053a9640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97000b6cd0 0x7f97000c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97053a9640 1 -- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97000cfaf0 msgr2=0x7f97000c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97053a9640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97000cfaf0 0x7f97000c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97053a9640 1 -- 172.21.15.60:0/3568415084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc009a70 con 0x7f97000cf710 2023-10-16T12:33:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f97053a9640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000cbc00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f96f000c4d0 tx=0x7f96f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f96f67fc640 1 -- 172.21.15.60:0/3568415084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f00074e0 con 0x7f97000cf710 2023-10-16T12:33:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f96f67fc640 1 -- 172.21.15.60:0/3568415084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f0007660 con 0x7f97000cf710 2023-10-16T12:33:43.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97000cc650 con 0x7f97000cf710 2023-10-16T12:33:43.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f96f67fc640 1 -- 172.21.15.60:0/3568415084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0004a20 con 0x7f97000cf710 2023-10-16T12:33:43.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.299+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97000ccc10 con 0x7f97000cf710 2023-10-16T12:33:43.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.302+0000 7f96f67fc640 1 -- 172.21.15.60:0/3568415084 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f96f0004b80 con 0x7f97000cf710 2023-10-16T12:33:43.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.302+0000 7f96f67fc640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f96e406ae70 0x7f96e406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.303+0000 7f96f67fc640 1 -- 172.21.15.60:0/3568415084 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96f00887b0 con 0x7f97000cf710 2023-10-16T12:33:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.303+0000 7f9705baa640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f96e406ae70 0x7f96e406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.303+0000 7f970a9a2740 1 Processor -- start 2023-10-16T12:33:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.303+0000 7f970a9a2740 1 -- start start 2023-10-16T12:33:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c2f80 0x555f44495e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c3360 0x555f444c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 0x555f444c6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f970a9a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f443b67a0 con 0x555f444963c0 2023-10-16T12:33:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f44408740 con 0x555f444c3360 2023-10-16T12:33:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f44405e30 con 0x555f444c2f80 2023-10-16T12:33:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.304+0000 7f9705baa640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f96e406ae70 0x7f96e406d330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f96fc036000 tx=0x7f96fc030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f9705baa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c2f80 0x555f44495e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 0x555f444c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 0x555f444c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45956/0 (socket says 172.21.15.60:45956) 2023-10-16T12:33:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 -- 172.21.15.60:0/3853093692 learned_addr learned my addr 172.21.15.60:0/3853093692 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f9705baa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c2f80 0x555f44495e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:50862/0 (socket says 172.21.15.60:50862) 2023-10-16T12:33:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97053a9640 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c3360 0x555f444c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 -- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c2f80 msgr2=0x555f44495e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c2f80 0x555f44495e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 -- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c3360 msgr2=0x555f444c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c3360 0x555f444c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.305+0000 7f97063ab640 1 -- 172.21.15.60:0/3853093692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f443c5910 con 0x555f444963c0 2023-10-16T12:33:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f97063ab640 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 0x555f444c6250 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f96f800ca20 tx=0x7f96f800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f7efc3cf5fb38a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f96e37fe640 1 -- 172.21.15.60:0/3853093692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f800eb40 con 0x555f444963c0 2023-10-16T12:33:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f96e37fe640 1 -- 172.21.15.60:0/3853093692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f800eca0 con 0x555f444963c0 2023-10-16T12:33:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f96e37fe640 1 -- 172.21.15.60:0/3853093692 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8013450 con 0x555f444963c0 2023-10-16T12:33:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f970a9a2740 1 -- 172.21.15.60:0/3853093692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 msgr2=0x555f444c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 0x555f444c6250 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f96f800ca20 tx=0x7f96f800cee0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f970a9a2740 1 -- 172.21.15.60:0/3853093692 shutdown_connections 2023-10-16T12:33:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.306+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444963c0 0x555f444c6250 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.307+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c3360 0x555f444c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.307+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3853093692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c2f80 0x555f44495e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.307+0000 7f970a9a2740 1 -- 172.21.15.60:0/3853093692 >> 172.21.15.60:0/3853093692 conn(0x555f44498190 msgr2=0x555f444e0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.307+0000 7f970a9a2740 1 -- 172.21.15.60:0/3853093692 shutdown_connections 2023-10-16T12:33:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.307+0000 7f970a9a2740 1 -- 172.21.15.60:0/3853093692 wait complete. 2023-10-16T12:33:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.307+0000 7f970a9a2740 1 -- start start 2023-10-16T12:33:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.308+0000 7f970a9a2740 1 Processor -- start 2023-10-16T12:33:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 0x555f444be2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444beef0 0x555f444be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f44401720 con 0x555f444c3360 2023-10-16T12:33:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4439f970 con 0x555f444c2f80 2023-10-16T12:33:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f4439f610 con 0x555f444beef0 2023-10-16T12:33:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f970a9a2740 10 client.0 _ll_register_callbacks cb 0x555f443c19d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:33:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 0x555f444be2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 0x555f444be2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45968/0 (socket says 172.21.15.60:45968) 2023-10-16T12:33:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 -- 172.21.15.60:0/3321013768 learned_addr learned my addr 172.21.15.60:0/3321013768 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f9705baa640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 -- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444beef0 msgr2=0x555f444be7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97063ab640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444beef0 0x555f444be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444beef0 0x555f444be7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 -- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 msgr2=0x555f444c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f97053a9640 1 -- 172.21.15.60:0/3321013768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f800c620 con 0x555f444c3360 2023-10-16T12:33:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.309+0000 7f9705baa640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f97053a9640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 0x555f444be2a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f96f00011e0 tx=0x7f96f0057d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91aa0eeca0cdb6d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f96e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:33:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f96e0ff9640 1 -- 172.21.15.60:0/3321013768 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f005f580 con 0x555f444c3360 2023-10-16T12:33:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f96e0ff9640 1 -- 172.21.15.60:0/3321013768 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96f005f810 con 0x555f444c3360 2023-10-16T12:33:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f44428160 con 0x555f444c3360 2023-10-16T12:33:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f96e0ff9640 1 -- 172.21.15.60:0/3321013768 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96f005b150 con 0x555f444c3360 2023-10-16T12:33:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f970a9a2740 10 client.4605 fetch_fsmap learned FSMap version 8 2023-10-16T12:33:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x555f443ffda0 con 0x555f444c3360 2023-10-16T12:33:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f96e0ff9640 1 -- 172.21.15.60:0/3321013768 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555f44428160 con 0x555f444c3360 2023-10-16T12:33:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f96e0ff9640 1 -- 172.21.15.60:0/3321013768 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f96f005b2b0 con 0x555f444c3360 2023-10-16T12:33:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.310+0000 7f970a9a2740 10 client.4605 fetch_fsmap finished waiting for FSMap version 8 2023-10-16T12:33:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.311+0000 7f970a9a2740 10 client.4605 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:33:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.311+0000 7f970a9a2740 20 client.4605 populate_metadata read hostname 'smithi060' 2023-10-16T12:33:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.311+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444fc310 0x555f444fe700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.311+0000 7f970a9a2740 4 client.4605 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:33:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.311+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 --> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f443d3cc0 con 0x555f444fc310 2023-10-16T12:33:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.311+0000 7f9705baa640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444fc310 0x555f444fe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.312+0000 7f9705baa640 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444fc310 0x555f444fe700 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e027bd5421626e58 server_cookie=ada8945a76a7f0fb in_seq=0 out_seq=0 2023-10-16T12:33:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.312+0000 7f96e0ff9640 10 client.4605 ms_handle_connect on v2:172.21.15.144:6836/872963903 2023-10-16T12:33:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f96e0ff9640 1 -- 172.21.15.60:0/3321013768 <== mds.0 v2:172.21.15.144:6836/872963903 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555f443d3cc0 con 0x555f444fc310 2023-10-16T12:33:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f96e0ff9640 10 client.4605 handle_command_reply: tid=0 2023-10-16T12:33:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 client.4605 shutdown 2023-10-16T12:33:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 msgr2=0x555f444be2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 0x555f444be2a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f96f00011e0 tx=0x7f96f0057d50 comp rx=0 tx=0).stop 2023-10-16T12:33:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 shutdown_connections 2023-10-16T12:33:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444fc310 0x555f444fe700 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444beef0 0x555f444be7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c3360 0x555f444be2a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 --2- 172.21.15.60:0/3321013768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f96e0ff9640 0 client.4605 ms_handle_reset on v2:172.21.15.144:6836/872963903 2023-10-16T12:33:43.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 >> 172.21.15.60:0/3321013768 conn(0x555f444963c0 msgr2=0x555f444e4220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:43.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 shutdown_connections 2023-10-16T12:33:43.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 1 -- 172.21.15.60:0/3321013768 wait complete. 2023-10-16T12:33:43.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.314+0000 7f970a9a2740 20 client.4605 trim_cache size 0 max 16384 2023-10-16T12:33:43.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.331+0000 7f970a9a2740 1 Processor -- start 2023-10-16T12:33:43.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.331+0000 7f970a9a2740 1 -- start start 2023-10-16T12:33:43.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 0x555f444f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444f3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 0x555f444e4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f970a9a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f44165f20 con 0x555f444beef0 2023-10-16T12:33:43.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f44499b20 con 0x555f444c2f80 2023-10-16T12:33:43.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f444c7610 con 0x555f444c3360 2023-10-16T12:33:43.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f9705baa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 0x555f444f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f9705baa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 0x555f444f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45982/0 (socket says 172.21.15.60:45982) 2023-10-16T12:33:43.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.332+0000 7f9705baa640 1 -- 172.21.15.60:0/1921018779 learned_addr learned my addr 172.21.15.60:0/1921018779 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:43.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f9705baa640 1 -- 172.21.15.60:0/1921018779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 msgr2=0x555f444e4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f9705baa640 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 0x555f444e4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f9705baa640 1 -- 172.21.15.60:0/1921018779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 msgr2=0x555f444f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f9705baa640 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f9705baa640 1 -- 172.21.15.60:0/1921018779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f4439eb80 con 0x555f444beef0 2023-10-16T12:33:43.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f9705baa640 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 0x555f444f3450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009ed0 tx=0x7f96fc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1751501dd980dc3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f96e3fff640 1 -- 172.21.15.60:0/1921018779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc034ea0 con 0x555f444beef0 2023-10-16T12:33:43.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f96e3fff640 1 -- 172.21.15.60:0/1921018779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc053040 con 0x555f444beef0 2023-10-16T12:33:43.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.333+0000 7f96e3fff640 1 -- 172.21.15.60:0/1921018779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc054450 con 0x555f444beef0 2023-10-16T12:33:43.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 -- 172.21.15.60:0/1921018779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 msgr2=0x555f444f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 0x555f444f3450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009ed0 tx=0x7f96fc02e000 comp rx=0 tx=0).stop 2023-10-16T12:33:43.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 -- 172.21.15.60:0/1921018779 shutdown_connections 2023-10-16T12:33:43.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 0x555f444e4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444c2f80 0x555f444f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 --2- 172.21.15.60:0/1921018779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444beef0 0x555f444f3450 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 -- 172.21.15.60:0/1921018779 >> 172.21.15.60:0/1921018779 conn(0x555f444963c0 msgr2=0x555f44499f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 -- 172.21.15.60:0/1921018779 shutdown_connections 2023-10-16T12:33:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.334+0000 7f970a9a2740 1 -- 172.21.15.60:0/1921018779 wait complete. 2023-10-16T12:33:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.335+0000 7f970a9a2740 1 -- start start 2023-10-16T12:33:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.335+0000 7f970a9a2740 1 Processor -- start 2023-10-16T12:33:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444beef0 0x555f44548a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 0x555f44494e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 0x555f444f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f444d6840 con 0x555f444c2f80 2023-10-16T12:33:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f444c19e0 con 0x555f444beef0 2023-10-16T12:33:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f444bc9a0 con 0x555f444c3360 2023-10-16T12:33:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f970a9a2740 10 client.0 _ll_register_callbacks cb 0x7f96f00538d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:33:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f97053a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 0x555f44494e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f97053a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 0x555f44494e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45990/0 (socket says 172.21.15.60:45990) 2023-10-16T12:33:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f97053a9640 1 -- 172.21.15.60:0/4065081552 learned_addr learned my addr 172.21.15.60:0/4065081552 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.336+0000 7f9705baa640 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444beef0 0x555f44548a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f97053a9640 1 -- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 msgr2=0x555f444f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f97053a9640 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 0x555f444f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f97053a9640 1 -- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444beef0 msgr2=0x555f44548a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:43.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f97053a9640 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444beef0 0x555f44548a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f97053a9640 1 -- 172.21.15.60:0/4065081552 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97000ce910 con 0x555f444c2f80 2023-10-16T12:33:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f97053a9640 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 0x555f44494e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f96f0007ec0 tx=0x7f96f0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d429c08eb54be6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f96e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:33:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f96e0ff9640 1 -- 172.21.15.60:0/4065081552 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f000e040 con 0x555f444c2f80 2023-10-16T12:33:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f96e0ff9640 1 -- 172.21.15.60:0/4065081552 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96f005d740 con 0x555f444c2f80 2023-10-16T12:33:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96f800eca0 con 0x555f444c2f80 2023-10-16T12:33:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f96e0ff9640 1 -- 172.21.15.60:0/4065081552 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96f005f810 con 0x555f444c2f80 2023-10-16T12:33:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f970a9a2740 10 client.4611 fetch_fsmap learned FSMap version 8 2023-10-16T12:33:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x555f443c4170 con 0x555f444c2f80 2023-10-16T12:33:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.337+0000 7f96e0ff9640 1 -- 172.21.15.60:0/4065081552 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f800eca0 con 0x555f444c2f80 2023-10-16T12:33:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f96e0ff9640 1 -- 172.21.15.60:0/4065081552 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f96f0075070 con 0x555f444c2f80 2023-10-16T12:33:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f970a9a2740 10 client.4611 fetch_fsmap finished waiting for FSMap version 8 2023-10-16T12:33:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f970a9a2740 10 client.4611 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:33:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f970a9a2740 20 client.4611 populate_metadata read hostname 'smithi060' 2023-10-16T12:33:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f970a9a2740 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444d6c60 0x555f444d7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f970a9a2740 4 client.4611 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:33:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 --> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555f443fa700 con 0x555f444d6c60 2023-10-16T12:33:43.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.338+0000 7f9705baa640 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444d6c60 0x555f444d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.339+0000 7f9705baa640 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444d6c60 0x555f444d7040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d2d59adc400664a server_cookie=fa5a62df9ebd07b0 in_seq=0 out_seq=0 2023-10-16T12:33:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.339+0000 7f96e0ff9640 10 client.4611 ms_handle_connect on v2:172.21.15.144:6836/872963903 2023-10-16T12:33:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.339+0000 7f96e0ff9640 1 -- 172.21.15.60:0/4065081552 <== mds.0 v2:172.21.15.144:6836/872963903 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x555f443fa700 con 0x555f444d6c60 2023-10-16T12:33:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.339+0000 7f96e0ff9640 10 client.4611 handle_command_reply: tid=0 2023-10-16T12:33:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.339+0000 7f970a9a2740 1 client.4611 shutdown 2023-10-16T12:33:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 msgr2=0x555f44494e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 0x555f44494e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f96f0007ec0 tx=0x7f96f0000ed0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 shutdown_connections 2023-10-16T12:33:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:6836/872963903,v1:172.21.15.144:6837/872963903] conn(0x555f444d6c60 0x555f444d7040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x555f444c3360 0x555f444f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555f444c2f80 0x555f44494e40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 --2- 172.21.15.60:0/4065081552 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x555f444beef0 0x555f44548a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f96e0ff9640 0 client.4611 ms_handle_reset on v2:172.21.15.144:6836/872963903 2023-10-16T12:33:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 >> 172.21.15.60:0/4065081552 conn(0x555f444963c0 msgr2=0x555f44491560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 shutdown_connections 2023-10-16T12:33:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 1 -- 172.21.15.60:0/4065081552 wait complete. 2023-10-16T12:33:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.340+0000 7f970a9a2740 20 client.4611 trim_cache size 0 max 16384 2023-10-16T12:33:43.354 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:33:43.354 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:33:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:33:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4279, 2023-10-16T12:33:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:33:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:33:43.355 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:33:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 17.966659776, 2023-10-16T12:33:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 8, 2023-10-16T12:33:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 22, 2023-10-16T12:33:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 22, 2023-10-16T12:33:43.356 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 26.286887280999998 2023-10-16T12:33:43.357 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:33:43.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f96e406ae70 msgr2=0x7f96e406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f96e406ae70 0x7f96e406d330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f96fc036000 tx=0x7f96fc030000 comp rx=0 tx=0).stop 2023-10-16T12:33:43.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 msgr2=0x7f97000cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:43.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000cbc00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f96f000c4d0 tx=0x7f96f000c990 comp rx=0 tx=0).stop 2023-10-16T12:33:43.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 shutdown_connections 2023-10-16T12:33:43.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f96e406ae70 0x7f96e406d330 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97000cfaf0 0x7f97000c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97000cf710 0x7f97000cbc00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 --2- 172.21.15.60:0/3568415084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97000b6cd0 0x7f97000c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 >> 172.21.15.60:0/3568415084 conn(0x7f97000d2cc0 msgr2=0x7f97000c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 shutdown_connections 2023-10-16T12:33:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:43.341+0000 7f9707e0e640 1 -- 172.21.15.60:0/3568415084 wait complete. 2023-10-16T12:33:43.361 INFO:teuthology.run_tasks:Running task kclient... 2023-10-16T12:33:43.385 INFO:tasks.kclient:Mounting kernel clients... 2023-10-16T12:33:43.385 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-16T12:33:43.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.386 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:33:43.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2023-10-16T12:33:43.559 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2023-10-16T12:33:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:33:43.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.564 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:33:43.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.619 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2023-10-16T12:33:43.724 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2023-10-16T12:33:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:33:43.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.729 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-16T12:33:43.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-16T12:33:43.894 INFO:teuthology.orchestra.run.smithi144.stderr:Cannot find device "ceph-brx" 2023-10-16T12:33:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:33:43.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.898 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-16T12:33:43.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:43.952 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-16T12:33:44.049 INFO:teuthology.orchestra.run.smithi144.stderr:Cannot find device "ceph-brx" 2023-10-16T12:33:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:33:44.053 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-10-16T12:33:44.053 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:33:44.054 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:33:44.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:33:44.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.279+0000 7fa8c5b67640 1 Processor -- start 2023-10-16T12:33:44.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.279+0000 7fa8c5b67640 1 -- start start 2023-10-16T12:33:44.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5b67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c006f8d0 0x7fa8c0153680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5b67640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa8c00c9b70 0x7fa8c00ca000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5b67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c0070ae0 con 0x7fa8c01532a0 2023-10-16T12:33:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5b67640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c00caa60 con 0x7fa8c00c9b70 2023-10-16T12:33:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5b67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c00ca540 con 0x7fa8c006f8d0 2023-10-16T12:33:44.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5366640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa8c00c9b70 0x7fa8c00ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:44.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5366640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa8c00c9b70 0x7fa8c00ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46004/0 (socket says 172.21.15.60:46004) 2023-10-16T12:33:44.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.281+0000 7fa8c5366640 1 -- 172.21.15.60:0/1254631804 learned_addr learned my addr 172.21.15.60:0/1254631804 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:44.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.282+0000 7fa8bffff640 1 --2- 172.21.15.60:0/1254631804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c006f8d0 0x7fa8c0153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.282+0000 7fa8c5366640 1 -- 172.21.15.60:0/1254631804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c006f8d0 msgr2=0x7fa8c0153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.282+0000 7fa8c5366640 1 --2- 172.21.15.60:0/1254631804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c006f8d0 0x7fa8c0153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.282+0000 7fa8c5366640 1 -- 172.21.15.60:0/1254631804 >> v1:172.21.15.144:6789/0 conn(0x7fa8c01532a0 legacy=0x7fa8c015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.282+0000 7fa8c5366640 1 -- 172.21.15.60:0/1254631804 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c006ff60 con 0x7fa8c00c9b70 2023-10-16T12:33:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.283+0000 7fa8c5366640 1 --2- 172.21.15.60:0/1254631804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c00c9b70 0x7fa8c00ca000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e50 tx=0x7fa8b802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3499de05f14fe06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.283+0000 7fa8bf7fe640 1 -- 172.21.15.60:0/1254631804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b802cb00 con 0x7fa8c00c9b70 2023-10-16T12:33:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.283+0000 7fa8bf7fe640 1 -- 172.21.15.60:0/1254631804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b802cc60 con 0x7fa8c00c9b70 2023-10-16T12:33:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.283+0000 7fa8bf7fe640 1 -- 172.21.15.60:0/1254631804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80343f0 con 0x7fa8c00c9b70 2023-10-16T12:33:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 -- 172.21.15.60:0/1254631804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c00c9b70 msgr2=0x7fa8c00ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/1254631804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c00c9b70 0x7fa8c00ca000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e50 tx=0x7fa8b802bf60 comp rx=0 tx=0).stop 2023-10-16T12:33:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 -- 172.21.15.60:0/1254631804 shutdown_connections 2023-10-16T12:33:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/1254631804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c00c9b70 0x7fa8c00ca000 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/1254631804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c006f8d0 0x7fa8c0153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 -- 172.21.15.60:0/1254631804 >> 172.21.15.60:0/1254631804 conn(0x7fa8c00b0a90 msgr2=0x7fa8c00ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:44.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 -- 172.21.15.60:0/1254631804 shutdown_connections 2023-10-16T12:33:44.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.284+0000 7fa8c5b67640 1 -- 172.21.15.60:0/1254631804 wait complete. 2023-10-16T12:33:44.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.286+0000 7fa8c5b67640 1 Processor -- start 2023-10-16T12:33:44.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.286+0000 7fa8c5b67640 1 -- start start 2023-10-16T12:33:44.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5b67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c006f8d0 0x7fa8c00a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:44.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5b67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c00c9b70 0x7fa8c00a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:44.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5b67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 0x7fa8c00a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:44.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5b67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c00c7a00 con 0x7fa8c01532a0 2023-10-16T12:33:44.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5b67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c014f0b0 con 0x7fa8c006f8d0 2023-10-16T12:33:44.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5b67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c00a8fd0 con 0x7fa8c00c9b70 2023-10-16T12:33:44.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5366640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 0x7fa8c00a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:44.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5366640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 0x7fa8c00a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46016/0 (socket says 172.21.15.60:46016) 2023-10-16T12:33:44.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.287+0000 7fa8c5366640 1 -- 172.21.15.60:0/2118160046 learned_addr learned my addr 172.21.15.60:0/2118160046 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:44.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c4b65640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c006f8d0 0x7fa8c00a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:44.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8bffff640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c00c9b70 0x7fa8c00a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:44.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c5366640 1 -- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c00c9b70 msgr2=0x7fa8c00a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:44.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c5366640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c00c9b70 0x7fa8c00a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c5366640 1 -- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c006f8d0 msgr2=0x7fa8c00a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:44.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c5366640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c006f8d0 0x7fa8c00a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c5366640 1 -- 172.21.15.60:0/2118160046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8009b30 con 0x7fa8c01532a0 2023-10-16T12:33:44.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8bffff640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c00c9b70 0x7fa8c00a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:44.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.288+0000 7fa8c5366640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 0x7fa8c00a89c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e50 tx=0x7fa8b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:44.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.289+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8009510 con 0x7fa8c01532a0 2023-10-16T12:33:44.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.289+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b803e450 con 0x7fa8c01532a0 2023-10-16T12:33:44.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.289+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b803e800 con 0x7fa8c01532a0 2023-10-16T12:33:44.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.289+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c00a92b0 con 0x7fa8c01532a0 2023-10-16T12:33:44.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.289+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c00c7620 con 0x7fa8c01532a0 2023-10-16T12:33:44.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.290+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c00c7b40 con 0x7fa8c01532a0 2023-10-16T12:33:44.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.291+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa8b803e960 con 0x7fa8c01532a0 2023-10-16T12:33:44.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.292+0000 7fa8bd7fa640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa8a406ae70 0x7fa8a406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:44.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.292+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8b80ac300 con 0x7fa8c01532a0 2023-10-16T12:33:44.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.292+0000 7fa8c4b65640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa8a406ae70 0x7fa8a406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:44.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.294+0000 7fa8c4b65640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa8a406ae70 0x7fa8a406d330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4000f60 tx=0x7fa8b4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:44.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.296+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8c00c7b40 con 0x7fa8c01532a0 2023-10-16T12:33:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.531+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa8c00c7b40 con 0x7fa8c01532a0 2023-10-16T12:33:44.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.532+0000 7fa8bd7fa640 1 -- 172.21.15.60:0/2118160046 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa8c00c7b40 con 0x7fa8c01532a0 2023-10-16T12:33:44.535 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:33:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.534+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa8a406ae70 msgr2=0x7fa8a406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.534+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa8a406ae70 0x7fa8a406d330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4000f60 tx=0x7fa8b4009040 comp rx=0 tx=0).stop 2023-10-16T12:33:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.534+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 msgr2=0x7fa8c00a89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.534+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 0x7fa8c00a89c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e50 tx=0x7fa8b8004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 shutdown_connections 2023-10-16T12:33:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa8a406ae70 0x7fa8a406d330 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c00c9b70 0x7fa8c00a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c01532a0 0x7fa8c00a89c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 --2- 172.21.15.60:0/2118160046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c006f8d0 0x7fa8c00a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 >> 172.21.15.60:0/2118160046 conn(0x7fa8c00b0a90 msgr2=0x7fa8c00cb890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 shutdown_connections 2023-10-16T12:33:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:44.535+0000 7fa8c5b67640 1 -- 172.21.15.60:0/2118160046 wait complete. 2023-10-16T12:33:44.556 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:33:44.557 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:33:44.557 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:33:44.557 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:33:44.557 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:33:44.558 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:33:44.558 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:33:44.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:33:44.558 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:33:44.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:44.559 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:33:44.614 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:44.614 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:44.615 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:33:44.615 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:33:44.616 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:33:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:44.616 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:33:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:33:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42362sec preferred_lft 42362sec 2023-10-16T12:33:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:33:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:44.617 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:44.617 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:44.618 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:33:44.619 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T12:33:44.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:44.619 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:44.620 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:44.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add name ceph-brx type bridge 2023-10-16T12:33:44.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr flush dev ceph-brx 2023-10-16T12:33:44.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set ceph-brx up 2023-10-16T12:33:44.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T12:33:44.621 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:45.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:45.098 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:33:45.179 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:33:45.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:45.183 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:33:45.238 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:33:45.238 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:33:45.239 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:33:45.239 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:33:45.239 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:33:45.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:45.240 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:45.240 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:45.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:33:45.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:33:45.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:33:45.242 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:45.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:45.982 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:33:46.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:46.043 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2023-10-16T12:33:46.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:46.103 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:46.103 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:46.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:33:46.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-16T12:33:46.104 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:46.264 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-16T12:33:46.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:46.265 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:46.265 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:46.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-16T12:33:46.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T12:33:46.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-16T12:33:46.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-16T12:33:46.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-16T12:33:46.267 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:46.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:46.563 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:46.563 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:46.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:33:46.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T12:33:46.564 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:46.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:46.739 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:33:46.793 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:33:46.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:46.794 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:33:46.845 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:33:46.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:46.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:46.989 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:46.989 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:47.490 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:47.490 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:47.491 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:33:47.491 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:33:47.491 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:33:47.496 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:33:47.496 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:33:47.496 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:33:47.496 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:33:47.497 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:33:47.501 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:33:47.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:47.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:33:47.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:47.605 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:47.690 INFO:teuthology.orchestra.run.smithi060.stdout:client4576 2023-10-16T12:33:47.691 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:33:47.691 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:33:47.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:47.752 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:47.832 INFO:teuthology.orchestra.run.smithi060.stdout:client4576 2023-10-16T12:33:47.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:47.834 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:47.920 INFO:teuthology.orchestra.run.smithi060.stdout:client4576 2023-10-16T12:33:47.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:47.921 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.client4576/status') 2023-10-16T12:33:48.043 INFO:teuthology.orchestra.run.smithi060.stdout:instance: client.4576 (3)172.21.15.60:0/764640546 2023-10-16T12:33:48.043 INFO:teuthology.orchestra.run.smithi060.stdout:blocklisted: false 2023-10-16T12:33:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2023-10-16T12:33:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2023-10-16T12:33:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:69 bytes copied, 7.7365e-05 s, 892 kB/s 2023-10-16T12:33:48.048 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2023-10-16T12:33:48.048 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:33:48.048 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T12:33:48.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:33:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.271+0000 7fc6b1b1a640 1 Processor -- start 2023-10-16T12:33:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.271+0000 7fc6b1b1a640 1 -- start start 2023-10-16T12:33:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.273+0000 7fc6b1b1a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc6ac070d70 0x7fc6ac06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.273+0000 7fc6b1b1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 0x7fc6ac06c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.273+0000 7fc6b1b1a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ac06c9c0 con 0x7fc6ac15aad0 2023-10-16T12:33:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.273+0000 7fc6b1b1a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac0b4620 con 0x7fc6ac070d70 2023-10-16T12:33:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.273+0000 7fc6b1b1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0b4340 con 0x7fc6ac0712c0 2023-10-16T12:33:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 0x7fc6ac06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 0x7fc6ac06c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:50920/0 (socket says 172.21.15.60:50920) 2023-10-16T12:33:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 -- 172.21.15.60:0/1620735944 learned_addr learned my addr 172.21.15.60:0/1620735944 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6abfff640 1 --2- 172.21.15.60:0/1620735944 >> v2:172.21.15.144:3300/0 conn(0x7fc6ac070d70 0x7fc6ac06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 -- 172.21.15.60:0/1620735944 >> v2:172.21.15.144:3300/0 conn(0x7fc6ac070d70 msgr2=0x7fc6ac06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 --2- 172.21.15.60:0/1620735944 >> v2:172.21.15.144:3300/0 conn(0x7fc6ac070d70 0x7fc6ac06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 -- 172.21.15.60:0/1620735944 >> v1:172.21.15.144:6789/0 conn(0x7fc6ac15aad0 legacy=0x7fc6ac0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6b1319640 1 -- 172.21.15.60:0/1620735944 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac06f520 con 0x7fc6ac0712c0 2023-10-16T12:33:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6abfff640 1 --2- 172.21.15.60:0/1620735944 >> v2:172.21.15.144:3300/0 conn(0x7fc6ac070d70 0x7fc6ac06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:33:48.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.274+0000 7fc6ab7fe640 1 -- 172.21.15.60:0/1620735944 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822413953 0 0) 0x7fc6ac06c9c0 con 0x7fc6ac15aad0 2023-10-16T12:33:48.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.276+0000 7fc6b1319640 1 --2- 172.21.15.60:0/1620735944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 0x7fc6ac06c480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009e50 tx=0x7fc6a402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726b997fa2a85af7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.276+0000 7fc6ab7fe640 1 -- 172.21.15.60:0/1620735944 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a402cb90 con 0x7fc6ac0712c0 2023-10-16T12:33:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.276+0000 7fc6ab7fe640 1 -- 172.21.15.60:0/1620735944 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a402ccf0 con 0x7fc6ac0712c0 2023-10-16T12:33:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.276+0000 7fc6ab7fe640 1 -- 172.21.15.60:0/1620735944 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4034610 con 0x7fc6ac0712c0 2023-10-16T12:33:48.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1620735944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 msgr2=0x7fc6ac06c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1620735944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 0x7fc6ac06c480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009e50 tx=0x7fc6a402bff0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1620735944 shutdown_connections 2023-10-16T12:33:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1620735944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac0712c0 0x7fc6ac06c480 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1620735944 >> v2:172.21.15.144:3300/0 conn(0x7fc6ac070d70 0x7fc6ac06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1620735944 >> 172.21.15.60:0/1620735944 conn(0x7fc6ac0b0a90 msgr2=0x7fc6ac0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1620735944 shutdown_connections 2023-10-16T12:33:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.277+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1620735944 wait complete. 2023-10-16T12:33:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.279+0000 7fc6b1b1a640 1 Processor -- start 2023-10-16T12:33:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.279+0000 7fc6b1b1a640 1 -- start start 2023-10-16T12:33:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.280+0000 7fc6b1b1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac070d70 0x7fc6ac158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.280+0000 7fc6b1b1a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0712c0 0x7fc6ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.280+0000 7fc6b1b1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 0x7fc6ac0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1b1a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c9a10 con 0x7fc6ac15aad0 2023-10-16T12:33:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1b1a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac15b0b0 con 0x7fc6ac0712c0 2023-10-16T12:33:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1b1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c8d90 con 0x7fc6ac070d70 2023-10-16T12:33:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 0x7fc6ac0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 0x7fc6ac0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46032/0 (socket says 172.21.15.60:46032) 2023-10-16T12:33:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 -- 172.21.15.60:0/1247079879 learned_addr learned my addr 172.21.15.60:0/1247079879 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b0b18640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac070d70 0x7fc6ac158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6abfff640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0712c0 0x7fc6ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 -- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac070d70 msgr2=0x7fc6ac158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac070d70 0x7fc6ac158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 -- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0712c0 msgr2=0x7fc6ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0712c0 0x7fc6ac0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.281+0000 7fc6b1319640 1 -- 172.21.15.60:0/1247079879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a4009b30 con 0x7fc6ac15aad0 2023-10-16T12:33:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.282+0000 7fc6b1319640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 0x7fc6ac0acdd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009e50 tx=0x7fc6a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.282+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4009510 con 0x7fc6ac15aad0 2023-10-16T12:33:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.282+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a403e450 con 0x7fc6ac15aad0 2023-10-16T12:33:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.282+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac0c9070 con 0x7fc6ac15aad0 2023-10-16T12:33:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.282+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a403e800 con 0x7fc6ac15aad0 2023-10-16T12:33:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.282+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac0c9630 con 0x7fc6ac15aad0 2023-10-16T12:33:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.283+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac04e9b0 con 0x7fc6ac15aad0 2023-10-16T12:33:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.285+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc6a403e960 con 0x7fc6ac15aad0 2023-10-16T12:33:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.285+0000 7fc6a97fa640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc69006ae70 0x7fc69006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.286+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a40ac240 con 0x7fc6ac15aad0 2023-10-16T12:33:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.286+0000 7fc6b0b18640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc69006ae70 0x7fc69006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.287+0000 7fc6b0b18640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc69006ae70 0x7fc69006d330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0000f60 tx=0x7fc6a0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.290+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6a40772e0 con 0x7fc6ac15aad0 2023-10-16T12:33:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.529+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc6ac0cf520 con 0x7fc6ac15aad0 2023-10-16T12:33:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.530+0000 7fc6a97fa640 1 -- 172.21.15.60:0/1247079879 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc6a402ce60 con 0x7fc6ac15aad0 2023-10-16T12:33:48.533 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:33:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc69006ae70 msgr2=0x7fc69006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc69006ae70 0x7fc69006d330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0000f60 tx=0x7fc6a0009040 comp rx=0 tx=0).stop 2023-10-16T12:33:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 msgr2=0x7fc6ac0acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 0x7fc6ac0acdd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009e50 tx=0x7fc6a4004a40 comp rx=0 tx=0).stop 2023-10-16T12:33:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 shutdown_connections 2023-10-16T12:33:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc69006ae70 0x7fc69006d330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6ac15aad0 0x7fc6ac0acdd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0712c0 0x7fc6ac0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 --2- 172.21.15.60:0/1247079879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6ac070d70 0x7fc6ac158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 >> 172.21.15.60:0/1247079879 conn(0x7fc6ac0b0a90 msgr2=0x7fc6ac0ce680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 shutdown_connections 2023-10-16T12:33:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:48.533+0000 7fc6b1b1a640 1 -- 172.21.15.60:0/1247079879 wait complete. 2023-10-16T12:33:48.555 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:33:48.556 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:33:48.556 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:33:48.556 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-16T12:33:48.556 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T12:33:48.557 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:33:48.557 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:33:48.557 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:33:48.557 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-16T12:33:48.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:48.558 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:33:48.619 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:48.619 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:48.620 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:48.620 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:48.620 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:48.620 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:48.621 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:48.621 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:48.621 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:33:48.621 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:48.621 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:48.622 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:33:48.622 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:33:48.622 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:48.622 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:33:48.623 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:33:48.623 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42358sec preferred_lft 42358sec 2023-10-16T12:33:48.623 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:33:48.623 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:48.623 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:48.624 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:48.624 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:33:48.624 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:33:48.624 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:48.624 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:33:48.625 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:48.625 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:33:48.625 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:48.625 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:33:48.626 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:33:48.626 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:33:48.626 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:48.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:48.627 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:33:48.721 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:33:48.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:48.726 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:33:48.783 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:33:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:33:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:33:48.784 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:33:48.785 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:33:48.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:48.786 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:48.787 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:48.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:33:48.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:33:48.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:33:48.789 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:48.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:48.987 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:33:49.044 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:33:49.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:49.046 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2023-10-16T12:33:49.104 INFO:teuthology.orchestra.run.smithi060.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-16T12:33:49.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:49.106 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:49.107 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:49.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:33:49.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-16T12:33:49.108 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:49.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:49.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T12:33:49.354 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:49.355 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:49.355 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:49.355 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.356 INFO:teuthology.orchestra.run.smithi060.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:33:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 2e:05:dd:00:ff:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:33:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:33:49.357 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.357 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::2c05:ddff:fe00:fff9/64 scope link 2023-10-16T12:33:49.357 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:49.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T12:33:49.464 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.466 INFO:teuthology.orchestra.run.smithi060.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:33:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 2e:05:dd:00:ff:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:33:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:33:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::2c05:ddff:fe00:fff9/64 scope link 2023-10-16T12:33:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:49.469 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-16T12:33:49.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:49.470 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:49.470 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:49.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-16T12:33:49.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T12:33:49.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-16T12:33:49.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-16T12:33:49.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-16T12:33:49.472 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:49.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:49.765 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:33:49.766 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:33:49.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.1 up 2023-10-16T12:33:49.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T12:33:49.767 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:33:49.950 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:49.951 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-16T12:33:50.005 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-16T12:33:50.005 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:50.006 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-16T12:33:50.059 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2023-10-16T12:33:50.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:50.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:50.185 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:50.186 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:50.220 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:50.221 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:50.221 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:33:50.221 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 2@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:33:50.221 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:33:50.223 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2023-10-16T12:33:50.224 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:33:50.224 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:33:50.224 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:33:50.224 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:33:50.227 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:33:50.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:50.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-16T12:33:50.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:50.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:50.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:50.303 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:50.388 INFO:teuthology.orchestra.run.smithi060.stdout:client4484 2023-10-16T12:33:50.389 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:33:50.389 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:33:50.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:50.433 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:50.532 INFO:teuthology.orchestra.run.smithi060.stdout:client4484 2023-10-16T12:33:50.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:50.533 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:50.578 INFO:teuthology.orchestra.run.smithi060.stdout:client4484 2023-10-16T12:33:50.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:50.579 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.client4484/status') 2023-10-16T12:33:50.707 INFO:teuthology.orchestra.run.smithi060.stdout:instance: client.4484 (3)172.21.15.60:0/2652155013 2023-10-16T12:33:50.707 INFO:teuthology.orchestra.run.smithi060.stdout:blocklisted: false 2023-10-16T12:33:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2023-10-16T12:33:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2023-10-16T12:33:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:70 bytes copied, 7.1004e-05 s, 986 kB/s 2023-10-16T12:33:50.710 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-10-16T12:33:50.711 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:33:50.711 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T12:33:50.711 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:33:50.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.906+0000 7f324c71d640 1 Processor -- start 2023-10-16T12:33:50.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.906+0000 7f324c71d640 1 -- start start 2023-10-16T12:33:50.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f324c71d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f324415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f324c71d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f324415ae80 0x7f32440cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f324c71d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324415ed60 con 0x7f32440ccaa0 2023-10-16T12:33:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f324c71d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f324406b810 con 0x7f324415ae80 2023-10-16T12:33:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f324c71d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32440707e0 con 0x7f324415aaa0 2023-10-16T12:33:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f3249cb8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f324415ae80 0x7f32440cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f3249cb8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f324415ae80 0x7f32440cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47886/0 (socket says 172.21.15.60:47886) 2023-10-16T12:33:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f3249cb8640 1 -- 172.21.15.60:0/1940390920 learned_addr learned my addr 172.21.15.60:0/1940390920 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.908+0000 7f324a4b9640 1 --2- 172.21.15.60:0/1940390920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f324415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:50.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.909+0000 7f3249cb8640 1 -- 172.21.15.60:0/1940390920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 msgr2=0x7f324415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.909+0000 7f3249cb8640 1 --2- 172.21.15.60:0/1940390920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f324415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.909+0000 7f3249cb8640 1 -- 172.21.15.60:0/1940390920 >> v1:172.21.15.144:6789/0 conn(0x7f32440ccaa0 legacy=0x7f3244070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.909+0000 7f3249cb8640 1 -- 172.21.15.60:0/1940390920 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324415ef90 con 0x7f324415ae80 2023-10-16T12:33:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.910+0000 7f3249cb8640 1 --2- 172.21.15.60:0/1940390920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440cc560 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3234009870 tx=0x7f323402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22bf2b89debea7e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.910+0000 7f32494b7640 1 -- 172.21.15.60:0/1940390920 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3234030840 con 0x7f324415ae80 2023-10-16T12:33:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.910+0000 7f32494b7640 1 -- 172.21.15.60:0/1940390920 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32340309a0 con 0x7f324415ae80 2023-10-16T12:33:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.910+0000 7f32494b7640 1 -- 172.21.15.60:0/1940390920 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3234030d30 con 0x7f324415ae80 2023-10-16T12:33:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 -- 172.21.15.60:0/1940390920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 msgr2=0x7f32440cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 --2- 172.21.15.60:0/1940390920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440cc560 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3234009870 tx=0x7f323402b980 comp rx=0 tx=0).stop 2023-10-16T12:33:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 -- 172.21.15.60:0/1940390920 shutdown_connections 2023-10-16T12:33:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 --2- 172.21.15.60:0/1940390920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440cc560 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 --2- 172.21.15.60:0/1940390920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f324415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 -- 172.21.15.60:0/1940390920 >> 172.21.15.60:0/1940390920 conn(0x7f32440cb410 msgr2=0x7f32440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 -- 172.21.15.60:0/1940390920 shutdown_connections 2023-10-16T12:33:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.911+0000 7f324c71d640 1 -- 172.21.15.60:0/1940390920 wait complete. 2023-10-16T12:33:50.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.913+0000 7f324c71d640 1 Processor -- start 2023-10-16T12:33:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.913+0000 7f324c71d640 1 -- start start 2023-10-16T12:33:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324c71d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32440ccaa0 0x7f32441edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324c71d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f32441eeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324c71d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324c71d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32440c33d0 con 0x7f324415ae80 2023-10-16T12:33:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324c71d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32440c2e10 con 0x7f32440ccaa0 2023-10-16T12:33:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324c71d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32440c30f0 con 0x7f324415aaa0 2023-10-16T12:33:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324acba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324acba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47896/0 (socket says 172.21.15.60:47896) 2023-10-16T12:33:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.914+0000 7f324acba640 1 -- 172.21.15.60:0/2151090755 learned_addr learned my addr 172.21.15.60:0/2151090755 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324a4b9640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32440ccaa0 0x7f32441edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f3249cb8640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f32441eeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324acba640 1 -- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 msgr2=0x7f32441eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324acba640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f32441eeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324acba640 1 -- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32440ccaa0 msgr2=0x7f32441edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324acba640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32440ccaa0 0x7f32441edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324acba640 1 -- 172.21.15.60:0/2151090755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32340094d0 con 0x7f324415ae80 2023-10-16T12:33:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f324acba640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440c24a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c7a0 tx=0x7f323c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.915+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c003b70 con 0x7f324415ae80 2023-10-16T12:33:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.916+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c003cf0 con 0x7f324415ae80 2023-10-16T12:33:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.916+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c003ea0 con 0x7f324415ae80 2023-10-16T12:33:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.916+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32441ee6c0 con 0x7f324415ae80 2023-10-16T12:33:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.916+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32441ee2e0 con 0x7f324415ae80 2023-10-16T12:33:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.917+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32441eebe0 con 0x7f324415ae80 2023-10-16T12:33:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.918+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f323c00aa90 con 0x7f324415ae80 2023-10-16T12:33:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.919+0000 7f323affd640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f322806ae70 0x7f322806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.919+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f323c0895c0 con 0x7f324415ae80 2023-10-16T12:33:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.920+0000 7f324a4b9640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f322806ae70 0x7f322806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.921+0000 7f324a4b9640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f322806ae70 0x7f322806d330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3240006fd0 tx=0x7f3240008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:50.924+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32441eebe0 con 0x7f324415ae80 2023-10-16T12:33:51.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.163+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f32441eebe0 con 0x7f324415ae80 2023-10-16T12:33:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.164+0000 7f323affd640 1 -- 172.21.15.60:0/2151090755 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f32441eebe0 con 0x7f324415ae80 2023-10-16T12:33:51.167 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:33:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f322806ae70 msgr2=0x7f322806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f322806ae70 0x7f322806d330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3240006fd0 tx=0x7f3240008040 comp rx=0 tx=0).stop 2023-10-16T12:33:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 msgr2=0x7f32440c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440c24a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c7a0 tx=0x7f323c00cc60 comp rx=0 tx=0).stop 2023-10-16T12:33:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 shutdown_connections 2023-10-16T12:33:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f322806ae70 0x7f322806d330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f324415aaa0 0x7f32441eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f324415ae80 0x7f32440c24a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 --2- 172.21.15.60:0/2151090755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32440ccaa0 0x7f32441edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 >> 172.21.15.60:0/2151090755 conn(0x7f32440cb410 msgr2=0x7f324415a000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 shutdown_connections 2023-10-16T12:33:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:51.167+0000 7f324c71d640 1 -- 172.21.15.60:0/2151090755 wait complete. 2023-10-16T12:33:51.189 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:33:51.189 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:33:51.189 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi144.front.sepia.ceph.com 2023-10-16T12:33:51.190 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-16T12:33:51.190 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T12:33:51.190 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:33:51.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:33:51.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:33:51.191 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-16T12:33:51.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:51.192 DEBUG:teuthology.orchestra.run.smithi144:> ip addr 2023-10-16T12:33:51.238 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:51.239 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:51.239 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:51.239 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:51.239 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:51.240 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:51.240 INFO:teuthology.orchestra.run.smithi144.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:51.240 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:22 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:51.240 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f0 2023-10-16T12:33:51.240 INFO:teuthology.orchestra.run.smithi144.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:51.241 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:23 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:51.241 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f1 2023-10-16T12:33:51.241 INFO:teuthology.orchestra.run.smithi144.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:51.241 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5e brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:51.241 INFO:teuthology.orchestra.run.smithi144.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:33:51.242 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5f brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:51.242 INFO:teuthology.orchestra.run.smithi144.stdout: inet 172.21.15.144/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T12:33:51.242 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft 42296sec preferred_lft 42296sec 2023-10-16T12:33:51.242 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2023-10-16T12:33:51.243 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:51.243 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T12:33:51.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:51.244 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:51.244 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:51.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add name ceph-brx type bridge 2023-10-16T12:33:51.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr flush dev ceph-brx 2023-10-16T12:33:51.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set ceph-brx up 2023-10-16T12:33:51.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T12:33:51.245 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:51.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:51.720 DEBUG:teuthology.orchestra.run.smithi144:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:33:51.794 INFO:teuthology.orchestra.run.smithi144.stdout:1 2023-10-16T12:33:51.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:51.798 DEBUG:teuthology.orchestra.run.smithi144:> route 2023-10-16T12:33:51.844 INFO:teuthology.orchestra.run.smithi144.stdout:Kernel IP routing table 2023-10-16T12:33:51.845 INFO:teuthology.orchestra.run.smithi144.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:33:51.845 INFO:teuthology.orchestra.run.smithi144.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T12:33:51.845 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T12:33:51.845 INFO:teuthology.orchestra.run.smithi144.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:33:51.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:51.846 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:51.846 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:51.847 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T12:33:51.847 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T12:33:51.847 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T12:33:51.848 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:52.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:52.574 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-16T12:33:52.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:52.630 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list-id 2023-10-16T12:33:52.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:52.684 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:52.684 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:52.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T12:33:52.685 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-16T12:33:52.685 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:52.831 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-16T12:33:52.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:52.832 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:52.832 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:52.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-16T12:33:52.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T12:33:52.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-16T12:33:52.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-16T12:33:52.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-16T12:33:52.834 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:53.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:53.105 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:53.105 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:53.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set brx.0 up 2023-10-16T12:33:53.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T12:33:53.106 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:53.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:53.259 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-16T12:33:53.310 INFO:teuthology.orchestra.run.smithi144.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-16T12:33:53.310 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:53.311 DEBUG:teuthology.orchestra.run.smithi144:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-16T12:33:53.360 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-10-16T12:33:53.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:53.361 DEBUG:teuthology.orchestra.run.smithi144:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:53.501 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:53.501 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:53.943 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:53.943 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:53.943 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:33:53.944 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: trying mount with new device syntax: 1@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:33:53.944 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:33:53.948 INFO:teuthology.orchestra.run.smithi144.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-10-16T12:33:53.949 INFO:teuthology.orchestra.run.smithi144.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:33:53.949 INFO:teuthology.orchestra.run.smithi144.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:33:53.949 INFO:teuthology.orchestra.run.smithi144.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:33:53.949 INFO:teuthology.orchestra.run.smithi144.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:33:53.953 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:33:53.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:53.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-16T12:33:54.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:54.049 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:54.104 INFO:teuthology.orchestra.run.smithi144.stdout:client4493 2023-10-16T12:33:54.104 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:54.105 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:33:54.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:54.158 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:54.261 INFO:teuthology.orchestra.run.smithi144.stdout:client4493 2023-10-16T12:33:54.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:54.262 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:54.320 INFO:teuthology.orchestra.run.smithi144.stdout:client4493 2023-10-16T12:33:54.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:54.321 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.client4493/status') 2023-10-16T12:33:54.417 INFO:teuthology.orchestra.run.smithi144.stdout:instance: client.4493 (3)192.168.0.1:0/181180511 2023-10-16T12:33:54.417 INFO:teuthology.orchestra.run.smithi144.stdout:blocklisted: false 2023-10-16T12:33:54.418 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2023-10-16T12:33:54.418 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2023-10-16T12:33:54.418 INFO:teuthology.orchestra.run.smithi144.stderr:68 bytes copied, 6.289e-05 s, 1.1 MB/s 2023-10-16T12:33:54.421 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2023-10-16T12:33:54.421 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:33:54.422 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T12:33:54.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:33:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.654+0000 7f1efe013640 1 Processor -- start 2023-10-16T12:33:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.654+0000 7f1efe013640 1 -- start start 2023-10-16T12:33:54.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.655+0000 7f1efe013640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ef81532a0 0x7f1ef806f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:54.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.655+0000 7f1efe013640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef80c9b70 0x7f1ef80c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:54.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.655+0000 7f1efe013640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef8070ae0 con 0x7f1ef8153680 2023-10-16T12:33:54.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.655+0000 7f1efe013640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef80caa60 con 0x7f1ef81532a0 2023-10-16T12:33:54.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.655+0000 7f1efe013640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80ca520 con 0x7f1ef80c9b70 2023-10-16T12:33:54.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efd011640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ef81532a0 0x7f1ef806f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:54.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efc810640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1ef8153680 0x7f1ef815a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54484/0 (socket says 172.21.15.60:54484) 2023-10-16T12:33:54.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efc810640 1 -- 172.21.15.60:0/1026679617 learned_addr learned my addr 172.21.15.60:0/1026679617 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efd812640 1 --2- 172.21.15.60:0/1026679617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef80c9b70 0x7f1ef80c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efd011640 1 -- 172.21.15.60:0/1026679617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef80c9b70 msgr2=0x7f1ef80c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efd011640 1 --2- 172.21.15.60:0/1026679617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef80c9b70 0x7f1ef80c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efd011640 1 -- 172.21.15.60:0/1026679617 >> v1:172.21.15.60:6789/0 conn(0x7f1ef8153680 legacy=0x7f1ef815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1efd011640 1 -- 172.21.15.60:0/1026679617 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef8153b30 con 0x7f1ef81532a0 2023-10-16T12:33:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.656+0000 7f1eeffff640 1 -- 172.21.15.60:0/1026679617 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389435841 0 0) 0x7f1ee8001110 con 0x7f1ef8153680 2023-10-16T12:33:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.657+0000 7f1efd011640 1 --2- 172.21.15.60:0/1026679617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef81532a0 0x7f1ef806f8d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40097a0 tx=0x7f1ef402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b55e2af79ef756 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1eeffff640 1 -- 172.21.15.60:0/1026679617 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4031830 con 0x7f1ef81532a0 2023-10-16T12:33:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1eeffff640 1 -- 172.21.15.60:0/1026679617 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef4031990 con 0x7f1ef81532a0 2023-10-16T12:33:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1eeffff640 1 -- 172.21.15.60:0/1026679617 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4031d20 con 0x7f1ef81532a0 2023-10-16T12:33:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1efe013640 1 -- 172.21.15.60:0/1026679617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef81532a0 msgr2=0x7f1ef806f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1efe013640 1 --2- 172.21.15.60:0/1026679617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef81532a0 0x7f1ef806f8d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40097a0 tx=0x7f1ef402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1efe013640 1 -- 172.21.15.60:0/1026679617 shutdown_connections 2023-10-16T12:33:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1efe013640 1 --2- 172.21.15.60:0/1026679617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef80c9b70 0x7f1ef80c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1efe013640 1 --2- 172.21.15.60:0/1026679617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef81532a0 0x7f1ef806f8d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.658+0000 7f1efe013640 1 -- 172.21.15.60:0/1026679617 >> 172.21.15.60:0/1026679617 conn(0x7f1ef80b0a90 msgr2=0x7f1ef80ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.659+0000 7f1efe013640 1 -- 172.21.15.60:0/1026679617 shutdown_connections 2023-10-16T12:33:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.659+0000 7f1efe013640 1 -- 172.21.15.60:0/1026679617 wait complete. 2023-10-16T12:33:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.660+0000 7f1efe013640 1 Processor -- start 2023-10-16T12:33:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.661+0000 7f1efe013640 1 -- start start 2023-10-16T12:33:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.661+0000 7f1efe013640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 0x7f1ef813d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.661+0000 7f1efe013640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ef81532a0 0x7f1ef80c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efe013640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef8153680 0x7f1ef80cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efe013640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80c6ae0 con 0x7f1ef80c9b70 2023-10-16T12:33:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efe013640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80c5f10 con 0x7f1ef81532a0 2023-10-16T12:33:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efe013640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef813dde0 con 0x7f1ef8153680 2023-10-16T12:33:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 0x7f1ef813d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 0x7f1ef813d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47914/0 (socket says 172.21.15.60:47914) 2023-10-16T12:33:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 -- 172.21.15.60:0/2755538646 learned_addr learned my addr 172.21.15.60:0/2755538646 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 -- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef8153680 msgr2=0x7f1ef80cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef8153680 0x7f1ef80cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efc810640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ef81532a0 0x7f1ef80c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 -- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ef81532a0 msgr2=0x7f1ef80c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:54.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ef81532a0 0x7f1ef80c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efd011640 1 -- 172.21.15.60:0/2755538646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef4009400 con 0x7f1ef80c9b70 2023-10-16T12:33:54.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.662+0000 7f1efc810640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ef81532a0 0x7f1ef80c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:54.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.663+0000 7f1efd011640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 0x7f1ef813d8a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402bee0 tx=0x7f1ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:54.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.663+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef403a6f0 con 0x7f1ef80c9b70 2023-10-16T12:33:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.663+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef403a870 con 0x7f1ef80c9b70 2023-10-16T12:33:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.663+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef403ac20 con 0x7f1ef80c9b70 2023-10-16T12:33:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.663+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef80c61b0 con 0x7f1ef80c9b70 2023-10-16T12:33:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.663+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef80c6770 con 0x7f1ef80c9b70 2023-10-16T12:33:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.664+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef80c6900 con 0x7f1ef80c9b70 2023-10-16T12:33:54.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.665+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1ef4030380 con 0x7f1ef80c9b70 2023-10-16T12:33:54.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.666+0000 7f1eedffb640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1edc06ae70 0x7f1edc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.666+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ef40ab100 con 0x7f1ef80c9b70 2023-10-16T12:33:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.667+0000 7f1efc810640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1edc06ae70 0x7f1edc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.668+0000 7f1efc810640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1edc06ae70 0x7f1edc06d330 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80c5b40 tx=0x7f1ee8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.671+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ef4076120 con 0x7f1ef80c9b70 2023-10-16T12:33:54.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.906+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1ef804e9b0 con 0x7f1ef80c9b70 2023-10-16T12:33:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.908+0000 7f1eedffb640 1 -- 172.21.15.60:0/2755538646 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1ef4035060 con 0x7f1ef80c9b70 2023-10-16T12:33:54.910 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:33:54.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.910+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1edc06ae70 msgr2=0x7f1edc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:54.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.910+0000 7f1efe013640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1edc06ae70 0x7f1edc06d330 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80c5b40 tx=0x7f1ee8009040 comp rx=0 tx=0).stop 2023-10-16T12:33:54.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.910+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 msgr2=0x7f1ef813d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:54.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.910+0000 7f1efe013640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 0x7f1ef813d8a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402bee0 tx=0x7f1ef40046c0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.910+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 shutdown_connections 2023-10-16T12:33:54.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.910+0000 7f1efe013640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1edc06ae70 0x7f1edc06d330 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.911+0000 7f1efe013640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ef8153680 0x7f1ef80cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.911+0000 7f1efe013640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ef81532a0 0x7f1ef80c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.911+0000 7f1efe013640 1 --2- 172.21.15.60:0/2755538646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ef80c9b70 0x7f1ef813d8a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:54.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.911+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 >> 172.21.15.60:0/2755538646 conn(0x7f1ef80b0a90 msgr2=0x7f1ef8158130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:54.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.911+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 shutdown_connections 2023-10-16T12:33:54.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:54.911+0000 7f1efe013640 1 -- 172.21.15.60:0/2755538646 wait complete. 2023-10-16T12:33:54.932 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:33:54.932 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:33:54.933 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi144.front.sepia.ceph.com 2023-10-16T12:33:54.933 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-16T12:33:54.933 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T12:33:54.933 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:33:54.934 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:33:54.934 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:33:54.934 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-16T12:33:54.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:54.935 DEBUG:teuthology.orchestra.run.smithi144:> ip addr 2023-10-16T12:33:54.995 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:54.995 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:54.995 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:54.996 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:54.996 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:54.996 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:54.996 INFO:teuthology.orchestra.run.smithi144.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:54.996 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:22 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:54.996 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f0 2023-10-16T12:33:54.997 INFO:teuthology.orchestra.run.smithi144.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:54.997 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:d9:38:23 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:54.997 INFO:teuthology.orchestra.run.smithi144.stdout: altname enp7s0f1 2023-10-16T12:33:54.997 INFO:teuthology.orchestra.run.smithi144.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:33:54.997 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5e brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:54.998 INFO:teuthology.orchestra.run.smithi144.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:33:54.998 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 0c:c4:7a:88:7f:5f brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:54.998 INFO:teuthology.orchestra.run.smithi144.stdout: inet 172.21.15.144/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-16T12:33:54.998 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft 42292sec preferred_lft 42292sec 2023-10-16T12:33:54.998 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::ec4:7aff:fe88:7f5f/64 scope link 2023-10-16T12:33:54.999 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:54.999 INFO:teuthology.orchestra.run.smithi144.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:33:54.999 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 46:02:13:6d:00:9f brd ff:ff:ff:ff:ff:ff 2023-10-16T12:33:54.999 INFO:teuthology.orchestra.run.smithi144.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:33:54.999 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.000 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::74b0:77ff:fe68:785e/64 scope link 2023-10-16T12:33:55.000 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.000 INFO:teuthology.orchestra.run.smithi144.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:33:55.000 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 46:02:13:6d:00:9f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T12:33:55.000 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::4402:13ff:fe6d:9f/64 scope link 2023-10-16T12:33:55.001 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.002 DEBUG:teuthology.orchestra.run.smithi144:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:33:55.098 INFO:teuthology.orchestra.run.smithi144.stdout:1 2023-10-16T12:33:55.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.103 DEBUG:teuthology.orchestra.run.smithi144:> route 2023-10-16T12:33:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:Kernel IP routing table 2023-10-16T12:33:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:33:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T12:33:55.155 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T12:33:55.156 INFO:teuthology.orchestra.run.smithi144.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:33:55.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.156 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:55.157 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:55.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T12:33:55.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T12:33:55.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T12:33:55.158 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:55.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.342 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-16T12:33:55.393 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T12:33:55.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.395 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list-id 2023-10-16T12:33:55.448 INFO:teuthology.orchestra.run.smithi144.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-16T12:33:55.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.450 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:55.450 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:55.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T12:33:55.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-16T12:33:55.451 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:55.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T12:33:55.665 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:55.665 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:55.665 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:55.666 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.666 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:55.666 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.666 INFO:teuthology.orchestra.run.smithi144.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:33:55.666 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 7e:dd:6f:51:b9:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:33:55.667 INFO:teuthology.orchestra.run.smithi144.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:33:55.667 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.667 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::7cdd:6fff:fe51:b958/64 scope link 2023-10-16T12:33:55.667 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T12:33:55.762 INFO:teuthology.orchestra.run.smithi144.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:33:55.763 INFO:teuthology.orchestra.run.smithi144.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:33:55.763 INFO:teuthology.orchestra.run.smithi144.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:33:55.763 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.763 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 ::1/128 scope host 2023-10-16T12:33:55.764 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.764 INFO:teuthology.orchestra.run.smithi144.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:33:55.764 INFO:teuthology.orchestra.run.smithi144.stdout: link/ether 7e:dd:6f:51:b9:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:33:55.764 INFO:teuthology.orchestra.run.smithi144.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:33:55.764 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.765 INFO:teuthology.orchestra.run.smithi144.stdout: inet6 fe80::7cdd:6fff:fe51:b958/64 scope link 2023-10-16T12:33:55.765 INFO:teuthology.orchestra.run.smithi144.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:33:55.766 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-16T12:33:55.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:55.767 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:55.768 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:55.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-16T12:33:55.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T12:33:55.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-16T12:33:55.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-16T12:33:55.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-16T12:33:55.769 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:56.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.039 DEBUG:teuthology.orchestra.run.smithi144:> (cd / && exec bash -c ' 2023-10-16T12:33:56.039 DEBUG:teuthology.orchestra.run.smithi144:> set -e 2023-10-16T12:33:56.039 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set brx.1 up 2023-10-16T12:33:56.039 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T12:33:56.040 DEBUG:teuthology.orchestra.run.smithi144:> ') 2023-10-16T12:33:56.208 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:56.208 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-16T12:33:56.259 INFO:teuthology.orchestra.run.smithi144.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-16T12:33:56.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:33:56.260 DEBUG:teuthology.orchestra.run.smithi144:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-16T12:33:56.308 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2023-10-16T12:33:56.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:56.498 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:56.498 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:56.534 INFO:teuthology.orchestra.run.smithi144.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:33:56.535 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:33:56.535 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:33:56.535 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: trying mount with new device syntax: 3@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:33:56.535 INFO:teuthology.orchestra.run.smithi144.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:33:56.539 INFO:teuthology.orchestra.run.smithi144.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2023-10-16T12:33:56.539 INFO:teuthology.orchestra.run.smithi144.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:33:56.539 INFO:teuthology.orchestra.run.smithi144.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:33:56.540 INFO:teuthology.orchestra.run.smithi144.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:33:56.540 INFO:teuthology.orchestra.run.smithi144.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:33:56.544 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:33:56.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-16T12:33:56.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.645 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:56.702 INFO:teuthology.orchestra.run.smithi144.stdout:client4638 2023-10-16T12:33:56.703 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:33:56.703 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:33:56.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.752 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:56.855 INFO:teuthology.orchestra.run.smithi144.stdout:client4638 2023-10-16T12:33:56.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.857 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:33:56.917 INFO:teuthology.orchestra.run.smithi144.stdout:client4638 2023-10-16T12:33:56.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:33:56.918 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.client4638/status') 2023-10-16T12:33:57.010 INFO:teuthology.orchestra.run.smithi144.stdout:instance: client.4638 (3)172.21.15.144:0/2094203377 2023-10-16T12:33:57.011 INFO:teuthology.orchestra.run.smithi144.stdout:blocklisted: false 2023-10-16T12:33:57.011 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2023-10-16T12:33:57.011 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2023-10-16T12:33:57.011 INFO:teuthology.orchestra.run.smithi144.stderr:71 bytes copied, 8.3579e-05 s, 849 kB/s 2023-10-16T12:33:57.014 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-16T12:33:57.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.265+0000 7fb9533ff640 1 Processor -- start 2023-10-16T12:33:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.265+0000 7fb9533ff640 1 -- start start 2023-10-16T12:33:57.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9533ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb94c06c040 0x7fb94c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9533ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9533ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c06c960 con 0x7fb94c15ebc0 2023-10-16T12:33:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9533ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c15b9d0 con 0x7fb94c06c040 2023-10-16T12:33:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9533ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c15bcb0 con 0x7fb94c0ad970 2023-10-16T12:33:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9523fd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb94c06c040 0x7fb94c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9523fd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb94c06c040 0x7fb94c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47926/0 (socket says 172.21.15.60:47926) 2023-10-16T12:33:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.267+0000 7fb9523fd640 1 -- 172.21.15.60:0/629786158 learned_addr learned my addr 172.21.15.60:0/629786158 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.268+0000 7fb951bfc640 1 --2- 172.21.15.60:0/629786158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.268+0000 7fb9523fd640 1 -- 172.21.15.60:0/629786158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 msgr2=0x7fb94c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.268+0000 7fb9523fd640 1 --2- 172.21.15.60:0/629786158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.268+0000 7fb9523fd640 1 -- 172.21.15.60:0/629786158 >> v1:172.21.15.144:6789/0 conn(0x7fb94c15ebc0 legacy=0x7fb94c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.268+0000 7fb9523fd640 1 -- 172.21.15.60:0/629786158 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c15bf90 con 0x7fb94c06c040 2023-10-16T12:33:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.269+0000 7fb9523fd640 1 --2- 172.21.15.60:0/629786158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c06c420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb9480098a0 tx=0x7fb94802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4862d0177d10d3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.269+0000 7fb9513fb640 1 -- 172.21.15.60:0/629786158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948030840 con 0x7fb94c06c040 2023-10-16T12:33:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.269+0000 7fb9513fb640 1 -- 172.21.15.60:0/629786158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9480309a0 con 0x7fb94c06c040 2023-10-16T12:33:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.269+0000 7fb9513fb640 1 -- 172.21.15.60:0/629786158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948030d30 con 0x7fb94c06c040 2023-10-16T12:33:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.270+0000 7fb9533ff640 1 -- 172.21.15.60:0/629786158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 msgr2=0x7fb94c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.270+0000 7fb9533ff640 1 --2- 172.21.15.60:0/629786158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c06c420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb9480098a0 tx=0x7fb94802b980 comp rx=0 tx=0).stop 2023-10-16T12:33:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.270+0000 7fb9533ff640 1 -- 172.21.15.60:0/629786158 shutdown_connections 2023-10-16T12:33:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.270+0000 7fb9533ff640 1 --2- 172.21.15.60:0/629786158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.270+0000 7fb9533ff640 1 --2- 172.21.15.60:0/629786158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c06c420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.270+0000 7fb9533ff640 1 -- 172.21.15.60:0/629786158 >> 172.21.15.60:0/629786158 conn(0x7fb94c0ce6a0 msgr2=0x7fb94c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.271+0000 7fb9533ff640 1 -- 172.21.15.60:0/629786158 shutdown_connections 2023-10-16T12:33:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.271+0000 7fb9533ff640 1 -- 172.21.15.60:0/629786158 wait complete. 2023-10-16T12:33:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.273+0000 7fb9533ff640 1 Processor -- start 2023-10-16T12:33:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.273+0000 7fb9533ff640 1 -- start start 2023-10-16T12:33:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.274+0000 7fb9533ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.274+0000 7fb9533ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.274+0000 7fb9533ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb94c15ebc0 0x7fb94c1f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.274+0000 7fb9533ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0b3e90 con 0x7fb94c06c040 2023-10-16T12:33:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.274+0000 7fb9533ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c14e3e0 con 0x7fb94c15ebc0 2023-10-16T12:33:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.274+0000 7fb9533ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0c8180 con 0x7fb94c0ad970 2023-10-16T12:33:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.275+0000 7fb9523fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.275+0000 7fb9523fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c0b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47938/0 (socket says 172.21.15.60:47938) 2023-10-16T12:33:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.275+0000 7fb9523fd640 1 -- 172.21.15.60:0/683654557 learned_addr learned my addr 172.21.15.60:0/683654557 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.275+0000 7fb952bfe640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb94c15ebc0 0x7fb94c1f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.275+0000 7fb951bfc640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9523fd640 1 -- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 msgr2=0x7fb94c0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9523fd640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9523fd640 1 -- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb94c15ebc0 msgr2=0x7fb94c1f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9523fd640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb94c15ebc0 0x7fb94c1f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9523fd640 1 -- 172.21.15.60:0/683654557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9480094d0 con 0x7fb94c06c040 2023-10-16T12:33:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9523fd640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c0b3140 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb948000c00 tx=0x7fb948004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948039a90 con 0x7fb94c06c040 2023-10-16T12:33:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb948039bf0 con 0x7fb94c06c040 2023-10-16T12:33:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948043410 con 0x7fb94c06c040 2023-10-16T12:33:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.276+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c0c7640 con 0x7fb94c06c040 2023-10-16T12:33:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.277+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c0c7bc0 con 0x7fb94c06c040 2023-10-16T12:33:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.278+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c06b330 con 0x7fb94c06c040 2023-10-16T12:33:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.279+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb94802f390 con 0x7fb94c06c040 2023-10-16T12:33:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.280+0000 7fb942ffd640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb93006aea0 0x7fb93006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.280+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9480ab710 con 0x7fb94c06c040 2023-10-16T12:33:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.281+0000 7fb951bfc640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb93006aea0 0x7fb93006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.282+0000 7fb951bfc640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb93006aea0 0x7fb93006d360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb94c06a760 tx=0x7fb93c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.284+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9480767d0 con 0x7fb94c06c040 2023-10-16T12:33:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.482+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb94c04e9b0 con 0x7fb94c06c040 2023-10-16T12:33:57.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.484+0000 7fb942ffd640 1 -- 172.21.15.60:0/683654557 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fb948079df0 con 0x7fb94c06c040 2023-10-16T12:33:57.487 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:57.487 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb93006aea0 msgr2=0x7fb93006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb93006aea0 0x7fb93006d360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb94c06a760 tx=0x7fb93c008040 comp rx=0 tx=0).stop 2023-10-16T12:33:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 msgr2=0x7fb94c0b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c0b3140 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb948000c00 tx=0x7fb948004730 comp rx=0 tx=0).stop 2023-10-16T12:33:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 shutdown_connections 2023-10-16T12:33:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb93006aea0 0x7fb93006d360 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb94c0ad970 0x7fb94c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb94c15ebc0 0x7fb94c1f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 --2- 172.21.15.60:0/683654557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c06c040 0x7fb94c0b3140 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 >> 172.21.15.60:0/683654557 conn(0x7fb94c0ce6a0 msgr2=0x7fb94c15d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 shutdown_connections 2023-10-16T12:33:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.487+0000 7fb9533ff640 1 -- 172.21.15.60:0/683654557 wait complete. 2023-10-16T12:33:57.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:33:57.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:33:57.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.741+0000 7fde51e61640 1 Processor -- start 2023-10-16T12:33:57.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.741+0000 7fde51e61640 1 -- start start 2023-10-16T12:33:57.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde51e61640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fde4c15ae10 0x7fde4c0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde51e61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 0x7fde4c0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde51e61640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c15b1f0 con 0x7fde4c0cf330 2023-10-16T12:33:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde51e61640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c06f650 con 0x7fde4c15ae10 2023-10-16T12:33:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde51e61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c06bb70 con 0x7fde4c06d570 2023-10-16T12:33:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde4bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fde4c0cf330 0x7fde4c06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54496/0 (socket says 172.21.15.60:54496) 2023-10-16T12:33:57.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.743+0000 7fde4bfff640 1 -- 172.21.15.60:0/1789916889 learned_addr learned my addr 172.21.15.60:0/1789916889 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:57.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4b7fe640 1 --2- 172.21.15.60:0/1789916889 >> v2:172.21.15.144:3300/0 conn(0x7fde4c15ae10 0x7fde4c0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4affd640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 0x7fde4c0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373580656 0 0) 0x7fde44001490 con 0x7fde4c0cf330 2023-10-16T12:33:57.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde34003520 con 0x7fde4c0cf330 2023-10-16T12:33:57.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2116135649 0 0) 0x7fde34003520 con 0x7fde4c0cf330 2023-10-16T12:33:57.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 msgr2=0x7fde4c0caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 0x7fde4c0caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 >> v2:172.21.15.144:3300/0 conn(0x7fde4c15ae10 msgr2=0x7fde4c0ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 --2- 172.21.15.60:0/1789916889 >> v2:172.21.15.144:3300/0 conn(0x7fde4c15ae10 0x7fde4c0ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0b4190 con 0x7fde4c0cf330 2023-10-16T12:33:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fde44001b80 con 0x7fde4c0cf330 2023-10-16T12:33:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 >> v1:172.21.15.60:6789/0 conn(0x7fde4c0cf330 legacy=0x7fde4c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.744+0000 7fde4a7fc640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 0x7fde34003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4a7fc640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4a7fc640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde34004d00 0x7fde34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c06bb70 con 0x7fde34003ef0 2023-10-16T12:33:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3400b980 con 0x7fde34004d00 2023-10-16T12:33:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3400bac0 con 0x7fde4c06d570 2023-10-16T12:33:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4affd640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4b7fe640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 0x7fde34003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4affd640 1 -- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 msgr2=0x7fde34003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:57.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4affd640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c06d570 0x7fde34003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4affd640 1 -- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde34004d00 msgr2=0x7fde34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:57.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4affd640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde34004d00 0x7fde34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.745+0000 7fde4affd640 1 -- 172.21.15.60:0/1789916889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34003520 con 0x7fde34003ef0 2023-10-16T12:33:57.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.746+0000 7fde4affd640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde340047c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fde40009ac0 tx=0x7fde340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32735c7b9de5d4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4002c6b0 con 0x7fde34003ef0 2023-10-16T12:33:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4002c810 con 0x7fde34003ef0 2023-10-16T12:33:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde4a7fc640 1 -- 172.21.15.60:0/1789916889 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4002cba0 con 0x7fde34003ef0 2023-10-16T12:33:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde51e61640 1 -- 172.21.15.60:0/1789916889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 msgr2=0x7fde340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde51e61640 1 --2- 172.21.15.60:0/1789916889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde340047c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fde40009ac0 tx=0x7fde340057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde4bfff640 1 -- 172.21.15.60:0/1789916889 reap_dead start 2023-10-16T12:33:57.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde51e61640 1 -- 172.21.15.60:0/1789916889 shutdown_connections 2023-10-16T12:33:57.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.747+0000 7fde51e61640 1 -- 172.21.15.60:0/1789916889 >> 172.21.15.60:0/1789916889 conn(0x7fde4c0ce840 msgr2=0x7fde4c0cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:57.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.748+0000 7fde51e61640 1 -- 172.21.15.60:0/1789916889 shutdown_connections 2023-10-16T12:33:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.748+0000 7fde51e61640 1 -- 172.21.15.60:0/1789916889 wait complete. 2023-10-16T12:33:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.749+0000 7fde51e61640 1 Processor -- start 2023-10-16T12:33:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.749+0000 7fde51e61640 1 -- start start 2023-10-16T12:33:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.750+0000 7fde51e61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde4c0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.750+0000 7fde51e61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c0c2230 0x7fde4c0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.750+0000 7fde51e61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde4c151520 0x7fde4c0c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde51e61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0c2a10 con 0x7fde34003ef0 2023-10-16T12:33:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde51e61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0c2cf0 con 0x7fde4c151520 2023-10-16T12:33:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde51e61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0c2fd0 con 0x7fde4c0c2230 2023-10-16T12:33:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde4c0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde4c0ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47958/0 (socket says 172.21.15.60:47958) 2023-10-16T12:33:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 -- 172.21.15.60:0/3050594325 learned_addr learned my addr 172.21.15.60:0/3050594325 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 -- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c0c2230 msgr2=0x7fde4c0b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c0c2230 0x7fde4c0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 -- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde4c151520 msgr2=0x7fde4c0c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde4c151520 0x7fde4c0c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 -- 172.21.15.60:0/3050594325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0c32b0 con 0x7fde34003ef0 2023-10-16T12:33:57.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.751+0000 7fde4b7fe640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde4c0ce090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0108e0 tx=0x7fde3c010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:57.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.752+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00eb60 con 0x7fde34003ef0 2023-10-16T12:33:57.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.752+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c00ecc0 con 0x7fde34003ef0 2023-10-16T12:33:57.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.752+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c01c400 con 0x7fde34003ef0 2023-10-16T12:33:57.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.752+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c0c3590 con 0x7fde34003ef0 2023-10-16T12:33:57.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.752+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c0b1d80 con 0x7fde34003ef0 2023-10-16T12:33:57.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.753+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c04e160 con 0x7fde34003ef0 2023-10-16T12:33:57.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.754+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fde3c01c560 con 0x7fde34003ef0 2023-10-16T12:33:57.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.755+0000 7fde33fff640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fde3806aea0 0x7fde3806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:57.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.755+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde3c089d70 con 0x7fde34003ef0 2023-10-16T12:33:57.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.756+0000 7fde4affd640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fde3806aea0 0x7fde3806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:57.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.757+0000 7fde4affd640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fde3806aea0 0x7fde3806d360 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fde40036000 tx=0x7fde4002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:57.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.760+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fde3c054e30 con 0x7fde34003ef0 2023-10-16T12:33:57.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.955+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde4c0b39a0 con 0x7fde34003ef0 2023-10-16T12:33:57.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.956+0000 7fde33fff640 1 -- 172.21.15.60:0/3050594325 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde3c058450 con 0x7fde34003ef0 2023-10-16T12:33:57.958 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:33:57.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.958+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fde3806aea0 msgr2=0x7fde3806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.958+0000 7fde51e61640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fde3806aea0 0x7fde3806d360 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fde40036000 tx=0x7fde4002e000 comp rx=0 tx=0).stop 2023-10-16T12:33:57.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 msgr2=0x7fde4c0ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:57.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde4c0ce090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0108e0 tx=0x7fde3c010da0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 shutdown_connections 2023-10-16T12:33:57.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fde3806aea0 0x7fde3806d360 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde4c0c2230 0x7fde4c0b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde4c151520 0x7fde4c0c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 --2- 172.21.15.60:0/3050594325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde34003ef0 0x7fde4c0ce090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:57.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 >> 172.21.15.60:0/3050594325 conn(0x7fde4c0ce840 msgr2=0x7fde4c0c1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:57.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 shutdown_connections 2023-10-16T12:33:57.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:57.959+0000 7fde51e61640 1 -- 172.21.15.60:0/3050594325 wait complete. 2023-10-16T12:33:57.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:33:58.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.213+0000 7f4b12e75640 1 Processor -- start 2023-10-16T12:33:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.213+0000 7f4b12e75640 1 -- start start 2023-10-16T12:33:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.215+0000 7f4b12e75640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b0c0cded0 0x7f4b0c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.215+0000 7f4b12e75640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c15d150 con 0x7f4b0c0b8480 2023-10-16T12:33:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.215+0000 7f4b12e75640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0d1180 con 0x7f4b0c15cd70 2023-10-16T12:33:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.215+0000 7f4b12e75640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0c0d1460 con 0x7f4b0c0cded0 2023-10-16T12:33:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.215+0000 7f4b11672640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4b0c0b8480 0x7f4b0c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54512/0 (socket says 172.21.15.60:54512) 2023-10-16T12:33:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.215+0000 7f4b11672640 1 -- 172.21.15.60:0/2424909852 learned_addr learned my addr 172.21.15.60:0/2424909852 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b12674640 1 --2- 172.21.15.60:0/2424909852 >> v2:172.21.15.144:3300/0 conn(0x7f4b0c0cded0 0x7f4b0c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193730374 0 0) 0x7f4b0c15d150 con 0x7f4b0c0b8480 2023-10-16T12:33:58.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af8003520 con 0x7f4b0c0b8480 2023-10-16T12:33:58.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 361914926 0 0) 0x7f4af8003520 con 0x7f4b0c0b8480 2023-10-16T12:33:58.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 >> v2:172.21.15.144:3300/0 conn(0x7f4b0c0cded0 msgr2=0x7f4b0c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:58.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 --2- 172.21.15.60:0/2424909852 >> v2:172.21.15.144:3300/0 conn(0x7f4b0c0cded0 0x7f4b0c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 >> v1:172.21.15.144:6789/0 conn(0x7f4b0c15cd70 legacy=0x7f4b0c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c0d1740 con 0x7f4b0c0b8480 2023-10-16T12:33:58.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4afc001670 con 0x7f4b0c0b8480 2023-10-16T12:33:58.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 >> v1:172.21.15.60:6789/0 conn(0x7f4b0c0b8480 legacy=0x7f4b0c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af80039c0 0x7f4af8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4af80047c0 0x7f4af8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8005180 0x7f4af800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0d1460 con 0x7f4af80047c0 2023-10-16T12:33:58.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af800c080 con 0x7f4af80039c0 2023-10-16T12:33:58.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.216+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af800c240 con 0x7f4af8005180 2023-10-16T12:33:58.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11672640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4af80047c0 0x7f4af8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11e73640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af80039c0 0x7f4af8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11672640 1 -- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8005180 msgr2=0x7f4af800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:58.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11672640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af8005180 0x7f4af800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11672640 1 -- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af80039c0 msgr2=0x7f4af8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:58.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11672640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4af80039c0 0x7f4af8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b11672640 1 -- 172.21.15.60:0/2424909852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af8003520 con 0x7f4af80047c0 2023-10-16T12:33:58.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.217+0000 7f4b12674640 1 -- 172.21.15.60:0/2424909852 reap_dead start 2023-10-16T12:33:58.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.218+0000 7f4b11672640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4af80047c0 0x7f4af8004c40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00c0e0 tx=0x7f4af80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aed2441821c597b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:58.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc005ad0 con 0x7f4af80047c0 2023-10-16T12:33:58.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc005c30 con 0x7f4af80047c0 2023-10-16T12:33:58.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b10e71640 1 -- 172.21.15.60:0/2424909852 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc02ca00 con 0x7f4af80047c0 2023-10-16T12:33:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b12e75640 1 -- 172.21.15.60:0/2424909852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4af80047c0 msgr2=0x7f4af8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b12e75640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4af80047c0 0x7f4af8004c40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00c0e0 tx=0x7f4af80057f0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b12e75640 1 -- 172.21.15.60:0/2424909852 shutdown_connections 2023-10-16T12:33:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b12e75640 1 --2- 172.21.15.60:0/2424909852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4af80047c0 0x7f4af8004c40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.219+0000 7f4b12e75640 1 -- 172.21.15.60:0/2424909852 >> 172.21.15.60:0/2424909852 conn(0x7f4b0c0cf570 msgr2=0x7f4b0c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.220+0000 7f4b12e75640 1 -- 172.21.15.60:0/2424909852 shutdown_connections 2023-10-16T12:33:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.220+0000 7f4b12e75640 1 -- 172.21.15.60:0/2424909852 wait complete. 2023-10-16T12:33:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.221+0000 7f4b12e75640 1 Processor -- start 2023-10-16T12:33:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.222+0000 7f4b12e75640 1 -- start start 2023-10-16T12:33:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.222+0000 7f4b12e75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80047c0 0x7f4b0c1f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12e75640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b0c0ab5e0 0x7f4b0c0aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12e75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 0x7f4b0c0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12e75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0aa530 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12e75640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0b1410 con 0x7f4b0c0ab5e0 2023-10-16T12:33:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12e75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c0a98b0 con 0x7f4af80047c0 2023-10-16T12:33:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 0x7f4b0c0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 0x7f4b0c0b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47972/0 (socket says 172.21.15.60:47972) 2023-10-16T12:33:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 -- 172.21.15.60:0/3228010869 learned_addr learned my addr 172.21.15.60:0/3228010869 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:58.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b11672640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b0c0ab5e0 0x7f4b0c0aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b11e73640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80047c0 0x7f4b0c1f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 -- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80047c0 msgr2=0x7f4b0c1f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80047c0 0x7f4b0c1f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 -- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b0c0ab5e0 msgr2=0x7f4b0c0aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b0c0ab5e0 0x7f4b0c0aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b12674640 1 -- 172.21.15.60:0/3228010869 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c0a9b90 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b11672640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b0c0ab5e0 0x7f4b0c0aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:58.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.223+0000 7f4b11e73640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80047c0 0x7f4b0c1f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:58.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.224+0000 7f4b12674640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 0x7f4b0c0b0ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0cd0f0 tx=0x7f4b0400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:58.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.224+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0400c620 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.224+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0400c780 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.224+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c0a9e70 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.224+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0400cb30 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.224+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c1f2130 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.226+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c04e9b0 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.227+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4b0400cc90 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.228+0000 7f4b027fc640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4af006ae70 0x7f4af006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.228+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b04089420 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.228+0000 7f4b11e73640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4af006ae70 0x7f4af006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.229+0000 7f4b11e73640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4af006ae70 0x7f4af006d330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007fd0 tx=0x7f4b08009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:58.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.232+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4b040544b0 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.429+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4b0c0b2b10 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.430+0000 7f4b027fc640 1 -- 172.21.15.60:0/3228010869 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4b04057ad0 con 0x7f4b0c0b0a50 2023-10-16T12:33:58.433 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:33:58.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4af006ae70 msgr2=0x7f4af006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4af006ae70 0x7f4af006d330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007fd0 tx=0x7f4b08009040 comp rx=0 tx=0).stop 2023-10-16T12:33:58.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 msgr2=0x7f4b0c0b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 0x7f4b0c0b0ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0cd0f0 tx=0x7f4b0400ec00 comp rx=0 tx=0).stop 2023-10-16T12:33:58.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 shutdown_connections 2023-10-16T12:33:58.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4af006ae70 0x7f4af006d330 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b0c0b0a50 0x7f4b0c0b0ed0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b0c0ab5e0 0x7f4b0c0aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 --2- 172.21.15.60:0/3228010869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4af80047c0 0x7f4b0c1f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 >> 172.21.15.60:0/3228010869 conn(0x7f4b0c0cf570 msgr2=0x7f4b0c0b7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:58.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.433+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 shutdown_connections 2023-10-16T12:33:58.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.434+0000 7f4b12e75640 1 -- 172.21.15.60:0/3228010869 wait complete. 2023-10-16T12:33:58.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:33:58.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.683+0000 7fc5651ef640 1 Processor -- start 2023-10-16T12:33:58.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.684+0000 7fc5651ef640 1 -- start start 2023-10-16T12:33:58.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc5651ef640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc56006c040 0x7fc56006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc5651ef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5600cda80 0x7fc56015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc5651ef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56006c960 con 0x7fc5600ad970 2023-10-16T12:33:58.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc5651ef640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56015b9d0 con 0x7fc56006c040 2023-10-16T12:33:58.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc5651ef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56015bcb0 con 0x7fc5600cda80 2023-10-16T12:33:58.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc55ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc56006c040 0x7fc56006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc55ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc56006c040 0x7fc56006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47974/0 (socket says 172.21.15.60:47974) 2023-10-16T12:33:58.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.685+0000 7fc55ffff640 1 -- 172.21.15.60:0/1330444527 learned_addr learned my addr 172.21.15.60:0/1330444527 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:58.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.686+0000 7fc5649ee640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5600cda80 0x7fc56015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.686+0000 7fc55ffff640 1 -- 172.21.15.60:0/1330444527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5600cda80 msgr2=0x7fc56015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.686+0000 7fc55ffff640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5600cda80 0x7fc56015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.686+0000 7fc55ffff640 1 -- 172.21.15.60:0/1330444527 >> v1:172.21.15.144:6789/0 conn(0x7fc5600ad970 legacy=0x7fc56015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.686+0000 7fc55ffff640 1 -- 172.21.15.60:0/1330444527 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56015bf90 con 0x7fc56006c040 2023-10-16T12:33:58.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.686+0000 7fc5649ee640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5600cda80 0x7fc56015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:58.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.687+0000 7fc55ffff640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56006c040 0x7fc56006c420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc550009870 tx=0x7fc55002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a71d1a0232c7925 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.687+0000 7fc55f7fe640 1 -- 172.21.15.60:0/1330444527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550031640 con 0x7fc56006c040 2023-10-16T12:33:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.687+0000 7fc55f7fe640 1 -- 172.21.15.60:0/1330444527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5500317a0 con 0x7fc56006c040 2023-10-16T12:33:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.687+0000 7fc55f7fe640 1 -- 172.21.15.60:0/1330444527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550031b30 con 0x7fc56006c040 2023-10-16T12:33:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 -- 172.21.15.60:0/1330444527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56006c040 msgr2=0x7fc56006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56006c040 0x7fc56006c420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc550009870 tx=0x7fc55002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 -- 172.21.15.60:0/1330444527 shutdown_connections 2023-10-16T12:33:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5600cda80 0x7fc56015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 --2- 172.21.15.60:0/1330444527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56006c040 0x7fc56006c420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 -- 172.21.15.60:0/1330444527 >> 172.21.15.60:0/1330444527 conn(0x7fc5600ce6a0 msgr2=0x7fc5600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 -- 172.21.15.60:0/1330444527 shutdown_connections 2023-10-16T12:33:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.688+0000 7fc5651ef640 1 -- 172.21.15.60:0/1330444527 wait complete. 2023-10-16T12:33:58.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.690+0000 7fc5651ef640 1 Processor -- start 2023-10-16T12:33:58.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.690+0000 7fc5651ef640 1 -- start start 2023-10-16T12:33:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5651ef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc56006c040 0x7fc56015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5651ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5600ad970 0x7fc5600c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5651ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 0x7fc5600c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5651ef640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600ae130 con 0x7fc5600cda80 2023-10-16T12:33:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5651ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600b2aa0 con 0x7fc5600ad970 2023-10-16T12:33:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5651ef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600b2d80 con 0x7fc56006c040 2023-10-16T12:33:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5649ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 0x7fc5600c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5649ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 0x7fc5600c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47988/0 (socket says 172.21.15.60:47988) 2023-10-16T12:33:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.691+0000 7fc5649ee640 1 -- 172.21.15.60:0/826834996 learned_addr learned my addr 172.21.15.60:0/826834996 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc55ffff640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc56006c040 0x7fc56015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc557fff640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5600ad970 0x7fc5600c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc5649ee640 1 -- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc56006c040 msgr2=0x7fc56015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc5649ee640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc56006c040 0x7fc56015a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc5649ee640 1 -- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5600ad970 msgr2=0x7fc5600c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc5649ee640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5600ad970 0x7fc5600c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc5649ee640 1 -- 172.21.15.60:0/826834996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500094d0 con 0x7fc5600cda80 2023-10-16T12:33:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc557fff640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5600ad970 0x7fc5600c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:33:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc5649ee640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 0x7fc5600c5770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc55800cd50 tx=0x7fc55800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.692+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800f680 con 0x7fc5600cda80 2023-10-16T12:33:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.693+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55800f800 con 0x7fc5600cda80 2023-10-16T12:33:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.693+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800fbb0 con 0x7fc5600cda80 2023-10-16T12:33:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.693+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5600b3060 con 0x7fc5600cda80 2023-10-16T12:33:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.693+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56015a6c0 con 0x7fc5600cda80 2023-10-16T12:33:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.694+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56006b330 con 0x7fc5600cda80 2023-10-16T12:33:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.695+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc55800b040 con 0x7fc5600cda80 2023-10-16T12:33:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.696+0000 7fc55d7fa640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc54006ae70 0x7fc54006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.696+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc55808a0b0 con 0x7fc5600cda80 2023-10-16T12:33:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.696+0000 7fc55ffff640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc54006ae70 0x7fc54006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:58.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.697+0000 7fc55ffff640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc54006ae70 0x7fc54006d330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc550036000 tx=0x7fc55003c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:58.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.700+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5580550c0 con 0x7fc5600cda80 2023-10-16T12:33:58.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.896+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc56015a8e0 con 0x7fc5600cda80 2023-10-16T12:33:58.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.897+0000 7fc55d7fa640 1 -- 172.21.15.60:0/826834996 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5580586e0 con 0x7fc5600cda80 2023-10-16T12:33:58.900 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:33:58.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.900+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc54006ae70 msgr2=0x7fc54006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.900+0000 7fc5651ef640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc54006ae70 0x7fc54006d330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc550036000 tx=0x7fc55003c040 comp rx=0 tx=0).stop 2023-10-16T12:33:58.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.900+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 msgr2=0x7fc5600c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:58.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.900+0000 7fc5651ef640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 0x7fc5600c5770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc55800cd50 tx=0x7fc55800a650 comp rx=0 tx=0).stop 2023-10-16T12:33:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.900+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 shutdown_connections 2023-10-16T12:33:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.900+0000 7fc5651ef640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc54006ae70 0x7fc54006d330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.901+0000 7fc5651ef640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600cda80 0x7fc5600c5770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.901+0000 7fc5651ef640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5600ad970 0x7fc5600c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.901+0000 7fc5651ef640 1 --2- 172.21.15.60:0/826834996 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc56006c040 0x7fc56015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.901+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 >> 172.21.15.60:0/826834996 conn(0x7fc5600ce6a0 msgr2=0x7fc56015db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.901+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 shutdown_connections 2023-10-16T12:33:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:58.901+0000 7fc5651ef640 1 -- 172.21.15.60:0/826834996 wait complete. 2023-10-16T12:33:58.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:33:59.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.144+0000 7ff2bb54e640 1 Processor -- start 2023-10-16T12:33:59.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.145+0000 7ff2bb54e640 1 -- start start 2023-10-16T12:33:59.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.146+0000 7ff2bb54e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.146+0000 7ff2bb54e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2b40ce390 0x7ff2b4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.146+0000 7ff2bb54e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b415ed60 con 0x7ff2b415aaa0 2023-10-16T12:33:59.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.146+0000 7ff2bb54e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b406b810 con 0x7ff2b40ce390 2023-10-16T12:33:59.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.146+0000 7ff2bb54e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b40707e0 con 0x7ff2b415ae80 2023-10-16T12:33:59.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b9aeb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2b40ce390 0x7ff2b4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b92ea640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff2b415aaa0 0x7ff2b40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44102/0 (socket says 172.21.15.60:44102) 2023-10-16T12:33:59.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b92ea640 1 -- 172.21.15.60:0/2455304317 learned_addr learned my addr 172.21.15.60:0/2455304317 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:59.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b8ae9640 1 --2- 172.21.15.60:0/2455304317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b9aeb640 1 -- 172.21.15.60:0/2455304317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 msgr2=0x7ff2b415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b9aeb640 1 --2- 172.21.15.60:0/2455304317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b9aeb640 1 -- 172.21.15.60:0/2455304317 >> v1:172.21.15.60:6789/0 conn(0x7ff2b415aaa0 legacy=0x7ff2b40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.147+0000 7ff2b9aeb640 1 -- 172.21.15.60:0/2455304317 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b415ef90 con 0x7ff2b40ce390 2023-10-16T12:33:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.148+0000 7ff2b9aeb640 1 --2- 172.21.15.60:0/2455304317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b40ce390 0x7ff2b4070f50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac009d90 tx=0x7ff2ac02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667744f5a6c540d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2abfff640 1 -- 172.21.15.60:0/2455304317 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac02cc50 con 0x7ff2b40ce390 2023-10-16T12:33:59.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2abfff640 1 -- 172.21.15.60:0/2455304317 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ac02cdb0 con 0x7ff2b40ce390 2023-10-16T12:33:59.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2abfff640 1 -- 172.21.15.60:0/2455304317 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac034660 con 0x7ff2b40ce390 2023-10-16T12:33:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2455304317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b40ce390 msgr2=0x7ff2b4070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2455304317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b40ce390 0x7ff2b4070f50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac009d90 tx=0x7ff2ac02bea0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2455304317 shutdown_connections 2023-10-16T12:33:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2455304317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b40ce390 0x7ff2b4070f50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2455304317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2455304317 >> 172.21.15.60:0/2455304317 conn(0x7ff2b40cb410 msgr2=0x7ff2b40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.149+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2455304317 shutdown_connections 2023-10-16T12:33:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.150+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2455304317 wait complete. 2023-10-16T12:33:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.151+0000 7ff2bb54e640 1 Processor -- start 2023-10-16T12:33:59.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.151+0000 7ff2bb54e640 1 -- start start 2023-10-16T12:33:59.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.152+0000 7ff2bb54e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2b40ce390 0x7ff2b415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.152+0000 7ff2bb54e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 0x7ff2b415b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.152+0000 7ff2bb54e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b4151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.152+0000 7ff2bb54e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b41521f0 con 0x7ff2b415aaa0 2023-10-16T12:33:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.152+0000 7ff2bb54e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b414ca70 con 0x7ff2b40ce390 2023-10-16T12:33:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.152+0000 7ff2bb54e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b4151f10 con 0x7ff2b415ae80 2023-10-16T12:33:59.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 0x7ff2b415b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 0x7ff2b415b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48196/0 (socket says 172.21.15.60:48196) 2023-10-16T12:33:59.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 -- 172.21.15.60:0/2576037997 learned_addr learned my addr 172.21.15.60:0/2576037997 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:59.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b92ea640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2b40ce390 0x7ff2b415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 -- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 msgr2=0x7ff2b4151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b4151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 -- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2b40ce390 msgr2=0x7ff2b415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2b40ce390 0x7ff2b415b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 -- 172.21.15.60:0/2576037997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac009a70 con 0x7ff2b415aaa0 2023-10-16T12:33:59.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.153+0000 7ff2b8ae9640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 0x7ff2b415b800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400d5f0 tx=0x7ff2a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.154+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a40125f0 con 0x7ff2b415aaa0 2023-10-16T12:33:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.154+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a4012750 con 0x7ff2b415aaa0 2023-10-16T12:33:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.154+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a4012b00 con 0x7ff2b415aaa0 2023-10-16T12:33:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.154+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b414cc60 con 0x7ff2b415aaa0 2023-10-16T12:33:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.154+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b414d220 con 0x7ff2b415aaa0 2023-10-16T12:33:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.155+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b4066de0 con 0x7ff2b415aaa0 2023-10-16T12:33:59.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.156+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff2a4012c60 con 0x7ff2b415aaa0 2023-10-16T12:33:59.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.157+0000 7ff2a9ffb640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff29806ae70 0x7ff29806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.157+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2a4089b80 con 0x7ff2b415aaa0 2023-10-16T12:33:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.157+0000 7ff2b92ea640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff29806ae70 0x7ff29806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.158+0000 7ff2b92ea640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff29806ae70 0x7ff29806d330 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0005880 tx=0x7ff2b0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:59.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.161+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2a4054b90 con 0x7ff2b415aaa0 2023-10-16T12:33:59.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.360+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2b404e160 con 0x7ff2b415aaa0 2023-10-16T12:33:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.362+0000 7ff2a9ffb640 1 -- 172.21.15.60:0/2576037997 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7ff2a4089e60 con 0x7ff2b415aaa0 2023-10-16T12:33:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:33:59.366 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:33:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.365+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff29806ae70 msgr2=0x7ff29806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.365+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff29806ae70 0x7ff29806d330 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0005880 tx=0x7ff2b0009040 comp rx=0 tx=0).stop 2023-10-16T12:33:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.365+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 msgr2=0x7ff2b415b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.365+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 0x7ff2b415b800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400d5f0 tx=0x7ff2a400dab0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 shutdown_connections 2023-10-16T12:33:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff29806ae70 0x7ff29806d330 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2b415ae80 0x7ff2b4151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b415aaa0 0x7ff2b415b800 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 --2- 172.21.15.60:0/2576037997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2b40ce390 0x7ff2b415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 >> 172.21.15.60:0/2576037997 conn(0x7ff2b40cb410 msgr2=0x7ff2b406ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 shutdown_connections 2023-10-16T12:33:59.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.366+0000 7ff2bb54e640 1 -- 172.21.15.60:0/2576037997 wait complete. 2023-10-16T12:33:59.386 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:33:59.386 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:33:59.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.611+0000 7ff11c8bb640 1 Processor -- start 2023-10-16T12:33:59.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.611+0000 7ff11c8bb640 1 -- start start 2023-10-16T12:33:59.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11c8bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff1181428c0 0x7ff1180c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11c8bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1180c9690 0x7ff1180ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11c8bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180cc6b0 con 0x7ff11806b660 2023-10-16T12:33:59.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11c8bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1180c9b20 con 0x7ff1181428c0 2023-10-16T12:33:59.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11c8bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1181599b0 con 0x7ff1180c9690 2023-10-16T12:33:59.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff11806b660 0x7ff11806ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44110/0 (socket says 172.21.15.60:44110) 2023-10-16T12:33:59.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff11659c640 1 -- 172.21.15.60:0/497591508 learned_addr learned my addr 172.21.15.60:0/497591508 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:33:59.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff115d9b640 1 --2- 172.21.15.60:0/497591508 >> v2:172.21.15.144:3300/0 conn(0x7ff1181428c0 0x7ff1180c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.613+0000 7ff116d9d640 1 --2- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1180c9690 0x7ff1180ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.614+0000 7ff11559a640 1 -- 172.21.15.60:0/497591508 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575432208 0 0) 0x7ff1180cc6b0 con 0x7ff11806b660 2023-10-16T12:33:59.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.614+0000 7ff11559a640 1 -- 172.21.15.60:0/497591508 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff104003520 con 0x7ff11806b660 2023-10-16T12:33:59.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.614+0000 7ff115d9b640 1 -- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1180c9690 msgr2=0x7ff1180ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:33:59.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.614+0000 7ff115d9b640 1 --2- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1180c9690 0x7ff1180ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.614+0000 7ff115d9b640 1 -- 172.21.15.60:0/497591508 >> v1:172.21.15.60:6789/0 conn(0x7ff11806b660 legacy=0x7ff11806ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.614+0000 7ff115d9b640 1 -- 172.21.15.60:0/497591508 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11806c290 con 0x7ff1181428c0 2023-10-16T12:33:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.616+0000 7ff115d9b640 1 --2- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1181428c0 0x7ff1180c9150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff100009d90 tx=0x7ff10002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d5e8c29ff0fba92 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.617+0000 7ff11559a640 1 -- 172.21.15.60:0/497591508 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10002cc50 con 0x7ff1181428c0 2023-10-16T12:33:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.617+0000 7ff11559a640 1 -- 172.21.15.60:0/497591508 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10002cdb0 con 0x7ff1181428c0 2023-10-16T12:33:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.617+0000 7ff11559a640 1 -- 172.21.15.60:0/497591508 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100034640 con 0x7ff1181428c0 2023-10-16T12:33:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.617+0000 7ff11c8bb640 1 -- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1181428c0 msgr2=0x7ff1180c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.617+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1181428c0 0x7ff1180c9150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff100009d90 tx=0x7ff10002bea0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.618+0000 7ff11c8bb640 1 -- 172.21.15.60:0/497591508 shutdown_connections 2023-10-16T12:33:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.618+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1180c9690 0x7ff1180ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.618+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/497591508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1181428c0 0x7ff1180c9150 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.618+0000 7ff11c8bb640 1 -- 172.21.15.60:0/497591508 >> 172.21.15.60:0/497591508 conn(0x7ff1180ce9b0 msgr2=0x7ff1180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.618+0000 7ff11c8bb640 1 -- 172.21.15.60:0/497591508 shutdown_connections 2023-10-16T12:33:59.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.618+0000 7ff11c8bb640 1 -- 172.21.15.60:0/497591508 wait complete. 2023-10-16T12:33:59.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.620+0000 7ff11c8bb640 1 Processor -- start 2023-10-16T12:33:59.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.620+0000 7ff11c8bb640 1 -- start start 2023-10-16T12:33:59.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11c8bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff11806b660 0x7ff11815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11c8bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1180c9690 0x7ff1180c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11c8bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 0x7ff1180c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11c8bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11815a700 con 0x7ff1181428c0 2023-10-16T12:33:59.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11c8bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11815a9e0 con 0x7ff1180c9690 2023-10-16T12:33:59.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11c8bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11815acc0 con 0x7ff11806b660 2023-10-16T12:33:59.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff116d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 0x7ff1180c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff116d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 0x7ff1180c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48202/0 (socket says 172.21.15.60:48202) 2023-10-16T12:33:59.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff116d9d640 1 -- 172.21.15.60:0/887080645 learned_addr learned my addr 172.21.15.60:0/887080645 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:33:59.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff115d9b640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1180c9690 0x7ff1180c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.622+0000 7ff11659c640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff11806b660 0x7ff11815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff116d9d640 1 -- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff11806b660 msgr2=0x7ff11815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff116d9d640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff11806b660 0x7ff11815b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff116d9d640 1 -- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1180c9690 msgr2=0x7ff1180c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff116d9d640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1180c9690 0x7ff1180c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff116d9d640 1 -- 172.21.15.60:0/887080645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff100009a70 con 0x7ff1181428c0 2023-10-16T12:33:59.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff116d9d640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 0x7ff1180c8230 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e7f0 tx=0x7ff11000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:59.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110009510 con 0x7ff1181428c0 2023-10-16T12:33:59.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11000ee40 con 0x7ff1181428c0 2023-10-16T12:33:59.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11815afa0 con 0x7ff1181428c0 2023-10-16T12:33:59.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.623+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11000cb80 con 0x7ff1181428c0 2023-10-16T12:33:59.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.624+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1180c5f20 con 0x7ff1181428c0 2023-10-16T12:33:59.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.625+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11815b4c0 con 0x7ff1181428c0 2023-10-16T12:33:59.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.626+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff11000cce0 con 0x7ff1181428c0 2023-10-16T12:33:59.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.627+0000 7ff10effd640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff0f806ae70 0x7ff0f806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:33:59.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.627+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff110089790 con 0x7ff1181428c0 2023-10-16T12:33:59.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.627+0000 7ff11659c640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff0f806ae70 0x7ff0f806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:33:59.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.628+0000 7ff11659c640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff0f806ae70 0x7ff0f806d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff108005880 tx=0x7ff108009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:33:59.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.632+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff11815b4c0 con 0x7ff1181428c0 2023-10-16T12:33:59.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.829+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff11815b4c0 con 0x7ff1181428c0 2023-10-16T12:33:59.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.831+0000 7ff10effd640 1 -- 172.21.15.60:0/887080645 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff11815b4c0 con 0x7ff1181428c0 2023-10-16T12:33:59.833 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:33:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.833+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff0f806ae70 msgr2=0x7ff0f806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.833+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff0f806ae70 0x7ff0f806d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff108005880 tx=0x7ff108009040 comp rx=0 tx=0).stop 2023-10-16T12:33:59.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.833+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 msgr2=0x7ff1180c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:33:59.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.833+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 0x7ff1180c8230 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e7f0 tx=0x7ff11000ebf0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 shutdown_connections 2023-10-16T12:33:59.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff0f806ae70 0x7ff0f806d330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1181428c0 0x7ff1180c8230 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1180c9690 0x7ff1180c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 --2- 172.21.15.60:0/887080645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff11806b660 0x7ff11815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:33:59.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 >> 172.21.15.60:0/887080645 conn(0x7ff1180ce9b0 msgr2=0x7ff1180c4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:33:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 shutdown_connections 2023-10-16T12:33:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:33:59.834+0000 7ff11c8bb640 1 -- 172.21.15.60:0/887080645 wait complete. 2023-10-16T12:33:59.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:00.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.075+0000 7fb63a11d640 1 Processor -- start 2023-10-16T12:34:00.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.076+0000 7fb63a11d640 1 -- start start 2023-10-16T12:34:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.077+0000 7fb63a11d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb63415d330 0x7fb6341618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.077+0000 7fb63a11d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb634065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.077+0000 7fb63a11d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb63406cc80 0x7fb634065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.077+0000 7fb63a11d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63406b1e0 con 0x7fb63406cc80 2023-10-16T12:34:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.077+0000 7fb63a11d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340d0890 con 0x7fb63415d330 2023-10-16T12:34:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.077+0000 7fb63a11d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340d05b0 con 0x7fb63406c7d0 2023-10-16T12:34:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb63406cc80 0x7fb634065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb63406cc80 0x7fb634065ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48214/0 (socket says 172.21.15.60:48214) 2023-10-16T12:34:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 -- 172.21.15.60:0/474437680 learned_addr learned my addr 172.21.15.60:0/474437680 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb6337fe640 1 --2- 172.21.15.60:0/474437680 >> v2:172.21.15.144:3300/0 conn(0x7fb63415d330 0x7fb6341618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb632ffd640 1 --2- 172.21.15.60:0/474437680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb634065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 -- 172.21.15.60:0/474437680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 msgr2=0x7fb634065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 --2- 172.21.15.60:0/474437680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb634065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 -- 172.21.15.60:0/474437680 >> v2:172.21.15.144:3300/0 conn(0x7fb63415d330 msgr2=0x7fb6341618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 --2- 172.21.15.60:0/474437680 >> v2:172.21.15.144:3300/0 conn(0x7fb63415d330 0x7fb6341618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.078+0000 7fb633fff640 1 -- 172.21.15.60:0/474437680 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634160dd0 con 0x7fb63406cc80 2023-10-16T12:34:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.079+0000 7fb633fff640 1 --2- 172.21.15.60:0/474437680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63406cc80 0x7fb634065ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb6340744f0 tx=0x7fb62c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d2b4bbd767dd23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb6327fc640 1 -- 172.21.15.60:0/474437680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0317e0 con 0x7fb63406cc80 2023-10-16T12:34:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb6327fc640 1 -- 172.21.15.60:0/474437680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c031940 con 0x7fb63406cc80 2023-10-16T12:34:00.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb6327fc640 1 -- 172.21.15.60:0/474437680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c031cd0 con 0x7fb63406cc80 2023-10-16T12:34:00.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 -- 172.21.15.60:0/474437680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63406cc80 msgr2=0x7fb634065ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 --2- 172.21.15.60:0/474437680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63406cc80 0x7fb634065ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb6340744f0 tx=0x7fb62c02b880 comp rx=0 tx=0).stop 2023-10-16T12:34:00.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 -- 172.21.15.60:0/474437680 shutdown_connections 2023-10-16T12:34:00.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 --2- 172.21.15.60:0/474437680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63406cc80 0x7fb634065ff0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 --2- 172.21.15.60:0/474437680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb634065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 --2- 172.21.15.60:0/474437680 >> v2:172.21.15.144:3300/0 conn(0x7fb63415d330 0x7fb6341618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 -- 172.21.15.60:0/474437680 >> 172.21.15.60:0/474437680 conn(0x7fb6340d0ab0 msgr2=0x7fb6340d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.080+0000 7fb63a11d640 1 -- 172.21.15.60:0/474437680 shutdown_connections 2023-10-16T12:34:00.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.081+0000 7fb63a11d640 1 -- 172.21.15.60:0/474437680 wait complete. 2023-10-16T12:34:00.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.082+0000 7fb63a11d640 1 Processor -- start 2023-10-16T12:34:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.082+0000 7fb63a11d640 1 -- start start 2023-10-16T12:34:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.083+0000 7fb63a11d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb6340cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.083+0000 7fb63a11d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63406cc80 0x7fb6340cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.083+0000 7fb63a11d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 0x7fb634152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.083+0000 7fb63a11d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341f6300 con 0x7fb63415d330 2023-10-16T12:34:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.083+0000 7fb63a11d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341f65a0 con 0x7fb63406cc80 2023-10-16T12:34:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.083+0000 7fb63a11d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6341f6880 con 0x7fb63406c7d0 2023-10-16T12:34:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 0x7fb634152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 0x7fb634152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48226/0 (socket says 172.21.15.60:48226) 2023-10-16T12:34:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 -- 172.21.15.60:0/2873479377 learned_addr learned my addr 172.21.15.60:0/2873479377 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb632ffd640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63406cc80 0x7fb6340cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb6337fe640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb6340cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 -- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 msgr2=0x7fb6340cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb6340cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 -- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63406cc80 msgr2=0x7fb6340cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63406cc80 0x7fb6340cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb633fff640 1 -- 172.21.15.60:0/2873479377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0094d0 con 0x7fb63415d330 2023-10-16T12:34:00.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb6337fe640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb6340cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:00.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.084+0000 7fb632ffd640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63406cc80 0x7fb6340cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:00.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.085+0000 7fb633fff640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 0x7fb634152ad0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0090b0 tx=0x7fb62c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:00.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.085+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c032880 con 0x7fb63415d330 2023-10-16T12:34:00.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.085+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c032a20 con 0x7fb63415d330 2023-10-16T12:34:00.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.085+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c032dd0 con 0x7fb63415d330 2023-10-16T12:34:00.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.085+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6341f6b60 con 0x7fb63415d330 2023-10-16T12:34:00.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.085+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6341f7080 con 0x7fb63415d330 2023-10-16T12:34:00.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.086+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6341f6d50 con 0x7fb63415d330 2023-10-16T12:34:00.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.087+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb62c04e450 con 0x7fb63415d330 2023-10-16T12:34:00.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.088+0000 7fb62bfff640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb61406aec0 0x7fb61406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.088+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb62c0ab7c0 con 0x7fb63415d330 2023-10-16T12:34:00.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.089+0000 7fb6337fe640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb61406aec0 0x7fb61406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.090+0000 7fb6337fe640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb61406aec0 0x7fb61406d380 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb624009ed0 tx=0x7fb62400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:00.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.093+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb62c076850 con 0x7fb63415d330 2023-10-16T12:34:00.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.292+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb63404e160 con 0x7fb63415d330 2023-10-16T12:34:00.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.292+0000 7fb62bfff640 1 -- 172.21.15.60:0/2873479377 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb62c079e70 con 0x7fb63415d330 2023-10-16T12:34:00.295 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb61406aec0 msgr2=0x7fb61406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb61406aec0 0x7fb61406d380 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb624009ed0 tx=0x7fb62400b040 comp rx=0 tx=0).stop 2023-10-16T12:34:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 msgr2=0x7fb634152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 0x7fb634152ad0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0090b0 tx=0x7fb62c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 shutdown_connections 2023-10-16T12:34:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb61406aec0 0x7fb61406d380 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb63415d330 0x7fb634152ad0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63406cc80 0x7fb6340cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 --2- 172.21.15.60:0/2873479377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb63406c7d0 0x7fb6340cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 >> 172.21.15.60:0/2873479377 conn(0x7fb6340d0ab0 msgr2=0x7fb6341579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.295+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 shutdown_connections 2023-10-16T12:34:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.296+0000 7fb63a11d640 1 -- 172.21.15.60:0/2873479377 wait complete. 2023-10-16T12:34:00.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:00.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.547+0000 7f067f59e640 1 Processor -- start 2023-10-16T12:34:00.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.548+0000 7f067f59e640 1 -- start start 2023-10-16T12:34:00.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.549+0000 7f067f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 0x7f06800d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.549+0000 7f067f59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f06800b8480 0x7f06800cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.549+0000 7f067f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068015d150 con 0x7f06800d0be0 2023-10-16T12:34:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.549+0000 7f067f59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06800d1180 con 0x7f06800b8480 2023-10-16T12:34:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.549+0000 7f067f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06800d1460 con 0x7f068015cd70 2023-10-16T12:34:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.550+0000 7f067e59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 0x7f06800d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.550+0000 7f067e59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 0x7f06800d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:38148/0 (socket says 172.21.15.60:38148) 2023-10-16T12:34:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.550+0000 7f067e59c640 1 -- 172.21.15.60:0/2420243789 learned_addr learned my addr 172.21.15.60:0/2420243789 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.550+0000 7f067ed9d640 1 --1- 172.21.15.60:0/2420243789 >> v1:172.21.15.144:6789/0 conn(0x7f06800d0be0 0x7f06800b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60318/0 (socket says 172.21.15.60:60318) 2023-10-16T12:34:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.550+0000 7f067dd9b640 1 --2- 172.21.15.60:0/2420243789 >> v2:172.21.15.144:3300/0 conn(0x7f06800b8480 0x7f06800cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.551+0000 7f067e59c640 1 -- 172.21.15.60:0/2420243789 >> v2:172.21.15.144:3300/0 conn(0x7f06800b8480 msgr2=0x7f06800cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.551+0000 7f067e59c640 1 --2- 172.21.15.60:0/2420243789 >> v2:172.21.15.144:3300/0 conn(0x7f06800b8480 0x7f06800cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.551+0000 7f067e59c640 1 -- 172.21.15.60:0/2420243789 >> v1:172.21.15.144:6789/0 conn(0x7f06800d0be0 legacy=0x7f06800b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.551+0000 7f067e59c640 1 -- 172.21.15.60:0/2420243789 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800d1740 con 0x7f068015cd70 2023-10-16T12:34:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.552+0000 7f067e59c640 1 --2- 172.21.15.60:0/2420243789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 0x7f06800d2220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0670009e50 tx=0x7f067002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4b2f3054e18d6a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.553+0000 7f067d59a640 1 -- 172.21.15.60:0/2420243789 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067002cb00 con 0x7f068015cd70 2023-10-16T12:34:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.553+0000 7f067d59a640 1 -- 172.21.15.60:0/2420243789 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067002cc60 con 0x7f068015cd70 2023-10-16T12:34:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.553+0000 7f067d59a640 1 -- 172.21.15.60:0/2420243789 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700343f0 con 0x7f068015cd70 2023-10-16T12:34:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.553+0000 7f067f59e640 1 -- 172.21.15.60:0/2420243789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 msgr2=0x7f06800d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.553+0000 7f067f59e640 1 --2- 172.21.15.60:0/2420243789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 0x7f06800d2220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0670009e50 tx=0x7f067002bf60 comp rx=0 tx=0).stop 2023-10-16T12:34:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.554+0000 7f067f59e640 1 -- 172.21.15.60:0/2420243789 shutdown_connections 2023-10-16T12:34:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.554+0000 7f067f59e640 1 --2- 172.21.15.60:0/2420243789 >> v2:172.21.15.144:3300/0 conn(0x7f06800b8480 0x7f06800cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.554+0000 7f067f59e640 1 --2- 172.21.15.60:0/2420243789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f068015cd70 0x7f06800d2220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.554+0000 7f067f59e640 1 -- 172.21.15.60:0/2420243789 >> 172.21.15.60:0/2420243789 conn(0x7f06800cf570 msgr2=0x7f06800cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.554+0000 7f067f59e640 1 -- 172.21.15.60:0/2420243789 shutdown_connections 2023-10-16T12:34:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.554+0000 7f067f59e640 1 -- 172.21.15.60:0/2420243789 wait complete. 2023-10-16T12:34:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.556+0000 7f067f59e640 1 Processor -- start 2023-10-16T12:34:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.556+0000 7f067f59e640 1 -- start start 2023-10-16T12:34:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 0x7f068014c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06800d0be0 0x7f068014d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f068015cd70 0x7f068015e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06801f1f60 con 0x7f06800b8480 2023-10-16T12:34:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06800c7830 con 0x7f068015cd70 2023-10-16T12:34:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06800c7b10 con 0x7f06800d0be0 2023-10-16T12:34:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 0x7f068014c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 0x7f068014c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48230/0 (socket says 172.21.15.60:48230) 2023-10-16T12:34:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067e59c640 1 -- 172.21.15.60:0/3005185415 learned_addr learned my addr 172.21.15.60:0/3005185415 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.557+0000 7f067ed9d640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f068015cd70 0x7f068015e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067dd9b640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06800d0be0 0x7f068014d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067e59c640 1 -- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06800d0be0 msgr2=0x7f068014d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067e59c640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06800d0be0 0x7f068014d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067e59c640 1 -- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f068015cd70 msgr2=0x7f068015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067e59c640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f068015cd70 0x7f068015e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067e59c640 1 -- 172.21.15.60:0/3005185415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670009b30 con 0x7f06800b8480 2023-10-16T12:34:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067e59c640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 0x7f068014c8b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0670000f80 tx=0x7f0670004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670040670 con 0x7f06800b8480 2023-10-16T12:34:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06700407d0 con 0x7f06800b8480 2023-10-16T12:34:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670040b80 con 0x7f06800b8480 2023-10-16T12:34:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.558+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06800c7df0 con 0x7f06800b8480 2023-10-16T12:34:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.559+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06800c83b0 con 0x7f06800b8480 2023-10-16T12:34:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.560+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068004e9b0 con 0x7f06800b8480 2023-10-16T12:34:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.560+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0670040ce0 con 0x7f06800b8480 2023-10-16T12:34:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.561+0000 7f0676ffd640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f066006ae70 0x7f066006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.562+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06700ac580 con 0x7f06800b8480 2023-10-16T12:34:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.562+0000 7f067dd9b640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f066006ae70 0x7f066006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.563+0000 7f067dd9b640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f066006ae70 0x7f066006d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06801f5530 tx=0x7f0668008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.566+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0670077640 con 0x7f06800b8480 2023-10-16T12:34:00.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.763+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f068014cdf0 con 0x7f06800b8480 2023-10-16T12:34:00.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.764+0000 7f0676ffd640 1 -- 172.21.15.60:0/3005185415 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0670009510 con 0x7f06800b8480 2023-10-16T12:34:00.767 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f066006ae70 msgr2=0x7f066006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f066006ae70 0x7f066006d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06801f5530 tx=0x7f0668008040 comp rx=0 tx=0).stop 2023-10-16T12:34:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 msgr2=0x7f068014c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 0x7f068014c8b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0670000f80 tx=0x7f0670004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 shutdown_connections 2023-10-16T12:34:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f066006ae70 0x7f066006d330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f06800d0be0 0x7f068014d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f068015cd70 0x7f068015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 --2- 172.21.15.60:0/3005185415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06800b8480 0x7f068014c8b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:00.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 >> 172.21.15.60:0/3005185415 conn(0x7f06800cf570 msgr2=0x7f06800b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:00.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 shutdown_connections 2023-10-16T12:34:00.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:00.767+0000 7f067f59e640 1 -- 172.21.15.60:0/3005185415 wait complete. 2023-10-16T12:34:00.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.008+0000 7f696ac6d640 1 Processor -- start 2023-10-16T12:34:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.009+0000 7f696ac6d640 1 -- start start 2023-10-16T12:34:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.010+0000 7f696ac6d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69640ad970 0x7f69640cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.010+0000 7f696ac6d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696415b9d0 con 0x7f69640ccfb0 2023-10-16T12:34:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.010+0000 7f696ac6d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696415bcb0 con 0x7f696406c040 2023-10-16T12:34:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.010+0000 7f696ac6d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696415bf90 con 0x7f69640ad970 2023-10-16T12:34:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f696946a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69640ad970 0x7f69640cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f696946a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69640ad970 0x7f69640cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48238/0 (socket says 172.21.15.60:48238) 2023-10-16T12:34:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f696946a640 1 -- 172.21.15.60:0/652069064 learned_addr learned my addr 172.21.15.60:0/652069064 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f696946a640 1 -- 172.21.15.60:0/652069064 >> v1:172.21.15.144:6789/0 conn(0x7f696406c040 legacy=0x7f696406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f696946a640 1 -- 172.21.15.60:0/652069064 >> v1:172.21.15.60:6789/0 conn(0x7f69640ccfb0 legacy=0x7f69640cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f696946a640 1 -- 172.21.15.60:0/652069064 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69640cc270 con 0x7f69640ad970 2023-10-16T12:34:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.011+0000 7f6968c69640 1 -- 172.21.15.60:0/652069064 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860149744 0 0) 0x7f696415b9d0 con 0x7f69640ccfb0 2023-10-16T12:34:01.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.012+0000 7f696946a640 1 --2- 172.21.15.60:0/652069064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69640cca70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f69540098a0 tx=0x7f695402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb14a72d85921d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f6968c69640 1 -- 172.21.15.60:0/652069064 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695402f840 con 0x7f69640ad970 2023-10-16T12:34:01.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f6968c69640 1 -- 172.21.15.60:0/652069064 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695402f9a0 con 0x7f69640ad970 2023-10-16T12:34:01.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f6968c69640 1 -- 172.21.15.60:0/652069064 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695402fd30 con 0x7f69640ad970 2023-10-16T12:34:01.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 -- 172.21.15.60:0/652069064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 msgr2=0x7f69640cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 --2- 172.21.15.60:0/652069064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69640cca70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f69540098a0 tx=0x7f695402ba20 comp rx=0 tx=0).stop 2023-10-16T12:34:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 -- 172.21.15.60:0/652069064 shutdown_connections 2023-10-16T12:34:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 --2- 172.21.15.60:0/652069064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69640cca70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 -- 172.21.15.60:0/652069064 >> 172.21.15.60:0/652069064 conn(0x7f69640ce6a0 msgr2=0x7f69640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 -- 172.21.15.60:0/652069064 shutdown_connections 2023-10-16T12:34:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.013+0000 7f696ac6d640 1 -- 172.21.15.60:0/652069064 wait complete. 2023-10-16T12:34:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.015+0000 7f696ac6d640 1 Processor -- start 2023-10-16T12:34:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.015+0000 7f696ac6d640 1 -- start start 2023-10-16T12:34:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696ac6d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f696406c040 0x7f69641f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696ac6d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69641e8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696ac6d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69640ccfb0 0x7f69641e9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696ac6d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696414a6d0 con 0x7f69640ad970 2023-10-16T12:34:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696ac6d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69640b0650 con 0x7f69640ccfb0 2023-10-16T12:34:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696ac6d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696414a970 con 0x7f696406c040 2023-10-16T12:34:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696946a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69641e8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.016+0000 7f696946a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69641e8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48242/0 (socket says 172.21.15.60:48242) 2023-10-16T12:34:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 -- 172.21.15.60:0/948175068 learned_addr learned my addr 172.21.15.60:0/948175068 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 -- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f696406c040 msgr2=0x7f69641f5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f696406c040 0x7f69641f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 -- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69640ccfb0 msgr2=0x7f69641e9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69640ccfb0 0x7f69641e9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 -- 172.21.15.60:0/948175068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69540094d0 con 0x7f69640ad970 2023-10-16T12:34:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f696946a640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69641e8c20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f69540098a0 tx=0x7f6954004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.017+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695402f7f0 con 0x7f69640ad970 2023-10-16T12:34:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.018+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6954041400 con 0x7f69640ad970 2023-10-16T12:34:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.018+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696414ac50 con 0x7f69640ad970 2023-10-16T12:34:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.018+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69540417b0 con 0x7f69640ad970 2023-10-16T12:34:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.018+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696414d460 con 0x7f69640ad970 2023-10-16T12:34:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.019+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696404e9b0 con 0x7f69640ad970 2023-10-16T12:34:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.020+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6954041910 con 0x7f69640ad970 2023-10-16T12:34:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.022+0000 7f695a7fc640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f694806ae70 0x7f694806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.022+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f695403c070 con 0x7f69640ad970 2023-10-16T12:34:01.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.022+0000 7f6969c6b640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f694806ae70 0x7f694806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.023+0000 7f6969c6b640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f694806ae70 0x7f694806d330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6960000f60 tx=0x7f69600079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.026+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6954077640 con 0x7f69640ad970 2023-10-16T12:34:01.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.238+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f696406b2b0 con 0x7f69640ad970 2023-10-16T12:34:01.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.242+0000 7f695a7fc640 1 -- 172.21.15.60:0/948175068 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f695407ac60 con 0x7f69640ad970 2023-10-16T12:34:01.245 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:01.245 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f694806ae70 msgr2=0x7f694806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f694806ae70 0x7f694806d330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6960000f60 tx=0x7f69600079b0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 msgr2=0x7f69641e8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69641e8c20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f69540098a0 tx=0x7f6954004730 comp rx=0 tx=0).stop 2023-10-16T12:34:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 shutdown_connections 2023-10-16T12:34:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f694806ae70 0x7f694806d330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69640ccfb0 0x7f69641e9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640ad970 0x7f69641e8c20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 --2- 172.21.15.60:0/948175068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f696406c040 0x7f69641f5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 >> 172.21.15.60:0/948175068 conn(0x7f69640ce6a0 msgr2=0x7f69640cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 shutdown_connections 2023-10-16T12:34:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.244+0000 7f696ac6d640 1 -- 172.21.15.60:0/948175068 wait complete. 2023-10-16T12:34:01.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:01.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.496+0000 7f2237fff640 1 Processor -- start 2023-10-16T12:34:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.496+0000 7f2237fff640 1 -- start start 2023-10-16T12:34:01.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.497+0000 7f2237fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f223815cd00 0x7f22380cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.497+0000 7f2237fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22380ce520 0x7f22380cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.497+0000 7f2237fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22380cb540 0x7f2238070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.497+0000 7f2237fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223815d0e0 con 0x7f22380ce520 2023-10-16T12:34:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.497+0000 7f2237fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22380cbd80 con 0x7f223815cd00 2023-10-16T12:34:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.497+0000 7f2237fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22380cbaa0 con 0x7f22380cb540 2023-10-16T12:34:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22380ce520 0x7f22380cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22380ce520 0x7f22380cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48254/0 (socket says 172.21.15.60:48254) 2023-10-16T12:34:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 -- 172.21.15.60:0/1360266647 learned_addr learned my addr 172.21.15.60:0/1360266647 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22377fe640 1 --2- 172.21.15.60:0/1360266647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22380cb540 0x7f2238070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f2236ffd640 1 --2- 172.21.15.60:0/1360266647 >> v2:172.21.15.144:3300/0 conn(0x7f223815cd00 0x7f22380cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 -- 172.21.15.60:0/1360266647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22380cb540 msgr2=0x7f2238070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 --2- 172.21.15.60:0/1360266647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22380cb540 0x7f2238070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 -- 172.21.15.60:0/1360266647 >> v2:172.21.15.144:3300/0 conn(0x7f223815cd00 msgr2=0x7f22380cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 --2- 172.21.15.60:0/1360266647 >> v2:172.21.15.144:3300/0 conn(0x7f223815cd00 0x7f22380cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.498+0000 7f22367fc640 1 -- 172.21.15.60:0/1360266647 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22380cd500 con 0x7f22380ce520 2023-10-16T12:34:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.500+0000 7f22367fc640 1 --2- 172.21.15.60:0/1360266647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f22380cb000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2238071c10 tx=0x7f222002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa12f236db99763 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.500+0000 7f2235ffb640 1 -- 172.21.15.60:0/1360266647 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220031860 con 0x7f22380ce520 2023-10-16T12:34:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.500+0000 7f2235ffb640 1 -- 172.21.15.60:0/1360266647 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22200319c0 con 0x7f22380ce520 2023-10-16T12:34:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.500+0000 7f2235ffb640 1 -- 172.21.15.60:0/1360266647 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220031d50 con 0x7f22380ce520 2023-10-16T12:34:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 -- 172.21.15.60:0/1360266647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 msgr2=0x7f22380cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 --2- 172.21.15.60:0/1360266647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f22380cb000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2238071c10 tx=0x7f222002b900 comp rx=0 tx=0).stop 2023-10-16T12:34:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 -- 172.21.15.60:0/1360266647 shutdown_connections 2023-10-16T12:34:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 --2- 172.21.15.60:0/1360266647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22380cb540 0x7f2238070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 --2- 172.21.15.60:0/1360266647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f22380cb000 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 --2- 172.21.15.60:0/1360266647 >> v2:172.21.15.144:3300/0 conn(0x7f223815cd00 0x7f22380cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 -- 172.21.15.60:0/1360266647 >> 172.21.15.60:0/1360266647 conn(0x7f22380cf790 msgr2=0x7f22380cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 -- 172.21.15.60:0/1360266647 shutdown_connections 2023-10-16T12:34:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.501+0000 7f2237fff640 1 -- 172.21.15.60:0/1360266647 wait complete. 2023-10-16T12:34:01.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.503+0000 7f2237fff640 1 Processor -- start 2023-10-16T12:34:01.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.503+0000 7f2237fff640 1 -- start start 2023-10-16T12:34:01.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f2237fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22380cb540 0x7f22380c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f2237fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f223806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f2237fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f223815cd00 0x7f223806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f2237fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2238148be0 con 0x7f22380ce520 2023-10-16T12:34:01.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f2237fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22380c58b0 con 0x7f22380cb540 2023-10-16T12:34:01.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f2237fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22380c4db0 con 0x7f223815cd00 2023-10-16T12:34:01.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f223806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f223806d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48256/0 (socket says 172.21.15.60:48256) 2023-10-16T12:34:01.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 -- 172.21.15.60:0/3363779494 learned_addr learned my addr 172.21.15.60:0/3363779494 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:01.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 -- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f223815cd00 msgr2=0x7f223806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f223815cd00 0x7f223806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 -- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22380cb540 msgr2=0x7f22380c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22380cb540 0x7f22380c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.504+0000 7f22367fc640 1 -- 172.21.15.60:0/3363779494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22200094d0 con 0x7f22380ce520 2023-10-16T12:34:01.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.505+0000 7f22367fc640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f223806d510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f222002be90 tx=0x7f22200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.505+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222003ad30 con 0x7f22380ce520 2023-10-16T12:34:01.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.505+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222003ae90 con 0x7f22380ce520 2023-10-16T12:34:01.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.505+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220044610 con 0x7f22380ce520 2023-10-16T12:34:01.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.505+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22380c5010 con 0x7f22380ce520 2023-10-16T12:34:01.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.505+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22380c55d0 con 0x7f22380ce520 2023-10-16T12:34:01.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.507+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2238148940 con 0x7f22380ce520 2023-10-16T12:34:01.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.507+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f222004d450 con 0x7f22380ce520 2023-10-16T12:34:01.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.508+0000 7f222f7fe640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f221806ae70 0x7f221806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.508+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22200abb50 con 0x7f22380ce520 2023-10-16T12:34:01.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.509+0000 7f2236ffd640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f221806ae70 0x7f221806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.510+0000 7f2236ffd640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f221806ae70 0x7f221806d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2228009dd0 tx=0x7f2228009450 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.513+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2238148940 con 0x7f22380ce520 2023-10-16T12:34:01.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.730+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2238148d20 con 0x7f22380ce520 2023-10-16T12:34:01.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.731+0000 7f222f7fe640 1 -- 172.21.15.60:0/3363779494 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2238148d20 con 0x7f22380ce520 2023-10-16T12:34:01.734 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:01.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f221806ae70 msgr2=0x7f221806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f221806ae70 0x7f221806d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2228009dd0 tx=0x7f2228009450 comp rx=0 tx=0).stop 2023-10-16T12:34:01.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 msgr2=0x7f223806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f223806d510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f222002be90 tx=0x7f22200046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 shutdown_connections 2023-10-16T12:34:01.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f221806ae70 0x7f221806d330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f223815cd00 0x7f223806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22380ce520 0x7f223806d510 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 --2- 172.21.15.60:0/3363779494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22380cb540 0x7f22380c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 >> 172.21.15.60:0/3363779494 conn(0x7f22380cf790 msgr2=0x7f22380b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:01.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 shutdown_connections 2023-10-16T12:34:01.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.734+0000 7f2237fff640 1 -- 172.21.15.60:0/3363779494 wait complete. 2023-10-16T12:34:01.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.960+0000 7f3d0675e640 1 Processor -- start 2023-10-16T12:34:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.960+0000 7f3d0675e640 1 -- start start 2023-10-16T12:34:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0675e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d0006c040 0x7f3d0006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0675e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3d000ad970 0x7f3d0015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0675e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0006c960 con 0x7f3d0015ebc0 2023-10-16T12:34:01.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0675e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0015b9d0 con 0x7f3d0006c040 2023-10-16T12:34:01.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0675e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0015bcb0 con 0x7f3d000ad970 2023-10-16T12:34:01.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0575c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d0006c040 0x7f3d0006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d05f5d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3d0015ebc0 0x7f3d000cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44118/0 (socket says 172.21.15.60:44118) 2023-10-16T12:34:01.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d0575c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d0006c040 0x7f3d0006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48266/0 (socket says 172.21.15.60:48266) 2023-10-16T12:34:01.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.962+0000 7f3d05f5d640 1 -- 172.21.15.60:0/2893967511 learned_addr learned my addr 172.21.15.60:0/2893967511 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.963+0000 7f3d04f5b640 1 --2- 172.21.15.60:0/2893967511 >> v2:172.21.15.144:3300/0 conn(0x7f3d000ad970 0x7f3d0015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.963+0000 7f3d0575c640 1 -- 172.21.15.60:0/2893967511 >> v2:172.21.15.144:3300/0 conn(0x7f3d000ad970 msgr2=0x7f3d0015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.963+0000 7f3d0575c640 1 --2- 172.21.15.60:0/2893967511 >> v2:172.21.15.144:3300/0 conn(0x7f3d000ad970 0x7f3d0015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.963+0000 7f3d0575c640 1 -- 172.21.15.60:0/2893967511 >> v1:172.21.15.60:6789/0 conn(0x7f3d0015ebc0 legacy=0x7f3d000cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.963+0000 7f3d0575c640 1 -- 172.21.15.60:0/2893967511 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0015bf90 con 0x7f3d0006c040 2023-10-16T12:34:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.963+0000 7f3cf7fff640 1 -- 172.21.15.60:0/2893967511 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994550906 0 0) 0x7f3d0006c960 con 0x7f3d0015ebc0 2023-10-16T12:34:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.965+0000 7f3d0575c640 1 --2- 172.21.15.60:0/2893967511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d0006c420 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d000cb060 tx=0x7f3cfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc8ba2367486834 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.965+0000 7f3cf7fff640 1 -- 172.21.15.60:0/2893967511 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc02cc80 con 0x7f3d0006c040 2023-10-16T12:34:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.965+0000 7f3cf7fff640 1 -- 172.21.15.60:0/2893967511 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc02cde0 con 0x7f3d0006c040 2023-10-16T12:34:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.965+0000 7f3cf7fff640 1 -- 172.21.15.60:0/2893967511 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc034660 con 0x7f3d0006c040 2023-10-16T12:34:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 -- 172.21.15.60:0/2893967511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 msgr2=0x7f3d0006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 --2- 172.21.15.60:0/2893967511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d0006c420 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d000cb060 tx=0x7f3cfc02bed0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 -- 172.21.15.60:0/2893967511 shutdown_connections 2023-10-16T12:34:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 --2- 172.21.15.60:0/2893967511 >> v2:172.21.15.144:3300/0 conn(0x7f3d000ad970 0x7f3d0015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 --2- 172.21.15.60:0/2893967511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d0006c420 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 -- 172.21.15.60:0/2893967511 >> 172.21.15.60:0/2893967511 conn(0x7f3d000ce6a0 msgr2=0x7f3d000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 -- 172.21.15.60:0/2893967511 shutdown_connections 2023-10-16T12:34:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.966+0000 7f3d0675e640 1 -- 172.21.15.60:0/2893967511 wait complete. 2023-10-16T12:34:01.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.968+0000 7f3d0675e640 1 Processor -- start 2023-10-16T12:34:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.968+0000 7f3d0675e640 1 -- start start 2023-10-16T12:34:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0675e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d001f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0675e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d000ad970 0x7f3d000acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0675e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0015ebc0 0x7f3d000aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0675e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000af960 con 0x7f3d0006c040 2023-10-16T12:34:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0675e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000afc40 con 0x7f3d0015ebc0 2023-10-16T12:34:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0675e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000aff20 con 0x7f3d000ad970 2023-10-16T12:34:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d001f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d001f2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48276/0 (socket says 172.21.15.60:48276) 2023-10-16T12:34:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 -- 172.21.15.60:0/1130157920 learned_addr learned my addr 172.21.15.60:0/1130157920 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 -- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d000ad970 msgr2=0x7f3d000acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d000ad970 0x7f3d000acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d05f5d640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0015ebc0 0x7f3d000aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 -- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0015ebc0 msgr2=0x7f3d000aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.969+0000 7f3d0575c640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0015ebc0 0x7f3d000aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3d0575c640 1 -- 172.21.15.60:0/1130157920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc009a70 con 0x7f3d0006c040 2023-10-16T12:34:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3d05f5d640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0015ebc0 0x7f3d000aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3d0575c640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d001f2990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc037040 tx=0x7f3cfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc040650 con 0x7f3d0006c040 2023-10-16T12:34:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc0407b0 con 0x7f3d0006c040 2023-10-16T12:34:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc040b60 con 0x7f3d0006c040 2023-10-16T12:34:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d000c8cf0 con 0x7f3d0006c040 2023-10-16T12:34:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.970+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d000b0bd0 con 0x7f3d0006c040 2023-10-16T12:34:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.971+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0006b330 con 0x7f3d0006c040 2023-10-16T12:34:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.972+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3cfc040cc0 con 0x7f3d0006c040 2023-10-16T12:34:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.973+0000 7f3cf5ffb640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ce406ae70 0x7f3ce406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.973+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cfc0ac500 con 0x7f3d0006c040 2023-10-16T12:34:01.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.974+0000 7f3d04f5b640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ce406ae70 0x7f3ce406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:01.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.975+0000 7f3d04f5b640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ce406ae70 0x7f3ce406d330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006fd0 tx=0x7f3cf0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:01.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:01.978+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3cfc0775a0 con 0x7f3d0006c040 2023-10-16T12:34:02.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.172+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d0004e9b0 con 0x7f3d0006c040 2023-10-16T12:34:02.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.172+0000 7f3cf5ffb640 1 -- 172.21.15.60:0/1130157920 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3cfc07abc0 con 0x7f3d0006c040 2023-10-16T12:34:02.175 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:02.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ce406ae70 msgr2=0x7f3ce406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ce406ae70 0x7f3ce406d330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006fd0 tx=0x7f3cf0008040 comp rx=0 tx=0).stop 2023-10-16T12:34:02.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 msgr2=0x7f3d001f2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d001f2990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc037040 tx=0x7f3cfc004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:02.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 shutdown_connections 2023-10-16T12:34:02.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ce406ae70 0x7f3ce406d330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3d000ad970 0x7f3d000acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d0015ebc0 0x7f3d000aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 --2- 172.21.15.60:0/1130157920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d0006c040 0x7f3d001f2990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.175+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 >> 172.21.15.60:0/1130157920 conn(0x7f3d000ce6a0 msgr2=0x7f3d000cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:02.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.176+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 shutdown_connections 2023-10-16T12:34:02.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.176+0000 7f3d0675e640 1 -- 172.21.15.60:0/1130157920 wait complete. 2023-10-16T12:34:02.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:02.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.427+0000 7f2a428ff640 1 Processor -- start 2023-10-16T12:34:02.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.428+0000 7f2a428ff640 1 -- start start 2023-10-16T12:34:02.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.429+0000 7f2a428ff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2a3c0b8480 0x7f2a3c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.429+0000 7f2a428ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3c15d150 con 0x7f2a3c0d0be0 2023-10-16T12:34:02.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.429+0000 7f2a428ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3c0d1180 con 0x7f2a3c15cd70 2023-10-16T12:34:02.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.429+0000 7f2a428ff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3c0d1460 con 0x7f2a3c0b8480 2023-10-16T12:34:02.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.430+0000 7f2a420fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2a3c0d0be0 0x7f2a3c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44130/0 (socket says 172.21.15.60:44130) 2023-10-16T12:34:02.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.430+0000 7f2a420fe640 1 -- 172.21.15.60:0/2773154408 learned_addr learned my addr 172.21.15.60:0/2773154408 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:02.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.430+0000 7f2a410fc640 1 --2- 172.21.15.60:0/2773154408 >> v2:172.21.15.144:3300/0 conn(0x7f2a3c0b8480 0x7f2a3c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.430+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894451054 0 0) 0x7f2a3c15d150 con 0x7f2a3c0d0be0 2023-10-16T12:34:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.430+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a28003520 con 0x7f2a3c0d0be0 2023-10-16T12:34:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2723320785 0 0) 0x7f2a28003520 con 0x7f2a3c0d0be0 2023-10-16T12:34:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 >> v2:172.21.15.144:3300/0 conn(0x7f2a3c0b8480 msgr2=0x7f2a3c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 --2- 172.21.15.60:0/2773154408 >> v2:172.21.15.144:3300/0 conn(0x7f2a3c0b8480 0x7f2a3c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 >> v1:172.21.15.144:6789/0 conn(0x7f2a3c15cd70 legacy=0x7f2a3c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c0d1740 con 0x7f2a3c0d0be0 2023-10-16T12:34:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2a340019f0 con 0x7f2a3c0d0be0 2023-10-16T12:34:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 >> v1:172.21.15.60:6789/0 conn(0x7f2a3c0d0be0 legacy=0x7f2a3c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a28003970 0x7f2a28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a280047c0 0x7f2a28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a280052a0 0x7f2a2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0d1460 con 0x7f2a28003970 2023-10-16T12:34:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2800c170 con 0x7f2a280052a0 2023-10-16T12:34:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2800c330 con 0x7f2a280047c0 2023-10-16T12:34:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a410fc640 1 --2- 172.21.15.60:0/2773154408 >> v2:172.21.15.144:3300/0 conn(0x7f2a3c0b8480 0x7f2a3c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a418fd640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a28003970 0x7f2a28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a420fe640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a280052a0 0x7f2a2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.431+0000 7f2a418fd640 1 -- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a280047c0 msgr2=0x7f2a28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.432+0000 7f2a418fd640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a280047c0 0x7f2a28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.432+0000 7f2a418fd640 1 -- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a280052a0 msgr2=0x7f2a2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.432+0000 7f2a418fd640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a280052a0 0x7f2a2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.432+0000 7f2a418fd640 1 -- 172.21.15.60:0/2773154408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28003520 con 0x7f2a28003970 2023-10-16T12:34:02.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.432+0000 7f2a420fe640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a280052a0 0x7f2a2800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:02.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.432+0000 7f2a420fe640 1 -- 172.21.15.60:0/2773154408 reap_dead start 2023-10-16T12:34:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.433+0000 7f2a418fd640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a28003970 0x7f2a28003d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a38001720 tx=0x7f2a280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c609573c065a46f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.433+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a380052d0 con 0x7f2a28003970 2023-10-16T12:34:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.433+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3802e050 con 0x7f2a28003970 2023-10-16T12:34:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.433+0000 7f2a408fb640 1 -- 172.21.15.60:0/2773154408 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a38003830 con 0x7f2a28003970 2023-10-16T12:34:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 -- 172.21.15.60:0/2773154408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a28003970 msgr2=0x7f2a28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a28003970 0x7f2a28003d90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2a38001720 tx=0x7f2a280057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 -- 172.21.15.60:0/2773154408 shutdown_connections 2023-10-16T12:34:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 --2- 172.21.15.60:0/2773154408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a28003970 0x7f2a28003d90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 -- 172.21.15.60:0/2773154408 >> 172.21.15.60:0/2773154408 conn(0x7f2a3c0cf570 msgr2=0x7f2a3c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 -- 172.21.15.60:0/2773154408 shutdown_connections 2023-10-16T12:34:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.434+0000 7f2a428ff640 1 -- 172.21.15.60:0/2773154408 wait complete. 2023-10-16T12:34:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.436+0000 7f2a428ff640 1 Processor -- start 2023-10-16T12:34:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.436+0000 7f2a428ff640 1 -- start start 2023-10-16T12:34:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a428ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a28003970 0x7f2a3c151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a428ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3c153de0 0x7f2a3c0cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a428ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 0x7f2a3c0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a428ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c1524c0 con 0x7f2a3c154260 2023-10-16T12:34:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a428ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0c9bc0 con 0x7f2a28003970 2023-10-16T12:34:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a428ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0ca180 con 0x7f2a3c153de0 2023-10-16T12:34:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a420fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 0x7f2a3c0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a420fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 0x7f2a3c0cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48292/0 (socket says 172.21.15.60:48292) 2023-10-16T12:34:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.437+0000 7f2a420fe640 1 -- 172.21.15.60:0/3477499305 learned_addr learned my addr 172.21.15.60:0/3477499305 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:02.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a418fd640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a28003970 0x7f2a3c151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a410fc640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3c153de0 0x7f2a3c0cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a420fe640 1 -- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3c153de0 msgr2=0x7f2a3c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a420fe640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3c153de0 0x7f2a3c0cbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a420fe640 1 -- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a28003970 msgr2=0x7f2a3c151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a420fe640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a28003970 0x7f2a3c151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a420fe640 1 -- 172.21.15.60:0/3477499305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c0cc7d0 con 0x7f2a3c154260 2023-10-16T12:34:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a420fe640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 0x7f2a3c0cc1a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0cd0f0 tx=0x7f2a3400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.438+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34004d60 con 0x7f2a3c154260 2023-10-16T12:34:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.439+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3400bb20 con 0x7f2a3c154260 2023-10-16T12:34:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.439+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c0ccab0 con 0x7f2a3c154260 2023-10-16T12:34:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.439+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400be90 con 0x7f2a3c154260 2023-10-16T12:34:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.439+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c1f2690 con 0x7f2a3c154260 2023-10-16T12:34:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.440+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c04e9b0 con 0x7f2a3c154260 2023-10-16T12:34:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.441+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2a34007070 con 0x7f2a3c154260 2023-10-16T12:34:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.442+0000 7f2a327fc640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2a2006ae70 0x7f2a2006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.442+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a340899e0 con 0x7f2a3c154260 2023-10-16T12:34:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.443+0000 7f2a418fd640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2a2006ae70 0x7f2a2006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.444+0000 7f2a418fd640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2a2006ae70 0x7f2a2006d330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a38033000 tx=0x7f2a38009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.447+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a340549f0 con 0x7f2a3c154260 2023-10-16T12:34:02.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.645+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a3c0b4e50 con 0x7f2a3c154260 2023-10-16T12:34:02.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.646+0000 7f2a327fc640 1 -- 172.21.15.60:0/3477499305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2a34058010 con 0x7f2a3c154260 2023-10-16T12:34:02.649 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:02.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.649+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2a2006ae70 msgr2=0x7f2a2006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.649+0000 7f2a428ff640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2a2006ae70 0x7f2a2006d330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a38033000 tx=0x7f2a38009030 comp rx=0 tx=0).stop 2023-10-16T12:34:02.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.649+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 msgr2=0x7f2a3c0cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.649+0000 7f2a428ff640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 0x7f2a3c0cc1a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0cd0f0 tx=0x7f2a3400b4a0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 shutdown_connections 2023-10-16T12:34:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2a2006ae70 0x7f2a2006d330 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3c153de0 0x7f2a3c0cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3c154260 0x7f2a3c0cc1a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 --2- 172.21.15.60:0/3477499305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a28003970 0x7f2a3c151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 >> 172.21.15.60:0/3477499305 conn(0x7f2a3c0cf570 msgr2=0x7f2a3c0d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:02.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 shutdown_connections 2023-10-16T12:34:02.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.650+0000 7f2a428ff640 1 -- 172.21.15.60:0/3477499305 wait complete. 2023-10-16T12:34:02.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:02.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.888+0000 7f75f69a0640 1 Processor -- start 2023-10-16T12:34:02.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.888+0000 7f75f69a0640 1 -- start start 2023-10-16T12:34:02.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.890+0000 7f75f69a0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 0x7f75f00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.890+0000 7f75f69a0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f015b9d0 con 0x7f75f006c040 2023-10-16T12:34:02.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.890+0000 7f75f69a0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f015bcb0 con 0x7f75f00ad970 2023-10-16T12:34:02.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.890+0000 7f75f69a0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f015b7d0 con 0x7f75f015b320 2023-10-16T12:34:02.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.890+0000 7f75f599e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75f006c040 0x7f75f006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44134/0 (socket says 172.21.15.60:44134) 2023-10-16T12:34:02.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.890+0000 7f75f599e640 1 -- 172.21.15.60:0/950169956 learned_addr learned my addr 172.21.15.60:0/950169956 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:02.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f619f640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 0x7f75f00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282898886 0 0) 0x7f75f015b9d0 con 0x7f75f006c040 2023-10-16T12:34:02.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc003520 con 0x7f75f006c040 2023-10-16T12:34:02.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3474839195 0 0) 0x7f75dc003520 con 0x7f75f006c040 2023-10-16T12:34:02.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 msgr2=0x7f75f00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:02.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 0x7f75f00cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 >> v1:172.21.15.144:6789/0 conn(0x7f75f00ad970 legacy=0x7f75f00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f015bf90 con 0x7f75f006c040 2023-10-16T12:34:02.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f75ec0019f0 con 0x7f75f006c040 2023-10-16T12:34:02.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 >> v1:172.21.15.60:6789/0 conn(0x7f75f006c040 legacy=0x7f75f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75dc0047c0 0x7f75dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 0x7f75dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f015b7d0 con 0x7f75dc0039a0 2023-10-16T12:34:02.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc00ba90 con 0x7f75dc0047c0 2023-10-16T12:34:02.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.891+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc00bc50 con 0x7f75f015b320 2023-10-16T12:34:02.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f599e640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f619f640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 0x7f75dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f519d640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75dc0047c0 0x7f75dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f599e640 1 -- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 msgr2=0x7f75dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:02.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f599e640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f015b320 0x7f75dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f599e640 1 -- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75dc0047c0 msgr2=0x7f75dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:02.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f599e640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75dc0047c0 0x7f75dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f599e640 1 -- 172.21.15.60:0/950169956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc003520 con 0x7f75dc0039a0 2023-10-16T12:34:02.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.892+0000 7f75f519d640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75dc0047c0 0x7f75dc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:02.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.893+0000 7f75f599e640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75dc003dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f75ec002230 tx=0x7f75dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0daa75e43014c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:02.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec034630 con 0x7f75dc0039a0 2023-10-16T12:34:02.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec02f070 con 0x7f75dc0039a0 2023-10-16T12:34:02.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f499c640 1 -- 172.21.15.60:0/950169956 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0349f0 con 0x7f75dc0039a0 2023-10-16T12:34:02.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f69a0640 1 -- 172.21.15.60:0/950169956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 msgr2=0x7f75dc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f69a0640 1 --2- 172.21.15.60:0/950169956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75dc003dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f75ec002230 tx=0x7f75dc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f619f640 1 -- 172.21.15.60:0/950169956 reap_dead start 2023-10-16T12:34:02.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f69a0640 1 -- 172.21.15.60:0/950169956 shutdown_connections 2023-10-16T12:34:02.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f69a0640 1 -- 172.21.15.60:0/950169956 >> 172.21.15.60:0/950169956 conn(0x7f75f00ce6a0 msgr2=0x7f75f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:02.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f69a0640 1 -- 172.21.15.60:0/950169956 shutdown_connections 2023-10-16T12:34:02.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.894+0000 7f75f69a0640 1 -- 172.21.15.60:0/950169956 wait complete. 2023-10-16T12:34:02.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.896+0000 7f75f69a0640 1 Processor -- start 2023-10-16T12:34:02.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.896+0000 7f75f69a0640 1 -- start start 2023-10-16T12:34:02.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.897+0000 7f75f69a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75f00c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.897+0000 7f75f69a0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f00c3730 0x7f75f00c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.897+0000 7f75f69a0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f01569c0 0x7f75f0156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.897+0000 7f75f69a0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f0157390 con 0x7f75dc0039a0 2023-10-16T12:34:02.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.897+0000 7f75f69a0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00c3bf0 con 0x7f75f00c3730 2023-10-16T12:34:02.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.897+0000 7f75f69a0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00c3ed0 con 0x7f75f01569c0 2023-10-16T12:34:02.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75f00c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75f00c2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48308/0 (socket says 172.21.15.60:48308) 2023-10-16T12:34:02.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 -- 172.21.15.60:0/1438218713 learned_addr learned my addr 172.21.15.60:0/1438218713 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:02.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f519d640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f00c3730 0x7f75f00c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f619f640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f01569c0 0x7f75f0156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 -- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f01569c0 msgr2=0x7f75f0156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f01569c0 0x7f75f0156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 -- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f00c3730 msgr2=0x7f75f00c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:02.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f00c3730 0x7f75f00c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:02.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f599e640 1 -- 172.21.15.60:0/1438218713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f00c41b0 con 0x7f75dc0039a0 2023-10-16T12:34:02.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.898+0000 7f75f619f640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f01569c0 0x7f75f0156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:02.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.899+0000 7f75f599e640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75f00c2be0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f75ec002230 tx=0x7f75ec001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:02.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.899+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec043070 con 0x7f75dc0039a0 2023-10-16T12:34:02.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.899+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec034640 con 0x7f75dc0039a0 2023-10-16T12:34:02.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.899+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0349b0 con 0x7f75dc0039a0 2023-10-16T12:34:02.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.899+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f00c67c0 con 0x7f75dc0039a0 2023-10-16T12:34:02.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.899+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f00c3120 con 0x7f75dc0039a0 2023-10-16T12:34:02.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.900+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f006b330 con 0x7f75dc0039a0 2023-10-16T12:34:02.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.901+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f75ec031030 con 0x7f75dc0039a0 2023-10-16T12:34:02.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.903+0000 7f75e67fc640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f75d406ae70 0x7f75d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:02.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.903+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75ec0ab660 con 0x7f75dc0039a0 2023-10-16T12:34:02.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.903+0000 7f75f519d640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f75d406ae70 0x7f75d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:02.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.904+0000 7f75f519d640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f75d406ae70 0x7f75d406d330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f75f006b8e0 tx=0x7f75e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:02.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:02.907+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75ec076670 con 0x7f75dc0039a0 2023-10-16T12:34:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.108+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75f006b330 con 0x7f75dc0039a0 2023-10-16T12:34:03.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.111+0000 7f75e67fc640 1 -- 172.21.15.60:0/1438218713 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f75ec079c90 con 0x7f75dc0039a0 2023-10-16T12:34:03.114 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:03.114 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f75d406ae70 msgr2=0x7f75d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f75d406ae70 0x7f75d406d330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f75f006b8e0 tx=0x7f75e0009040 comp rx=0 tx=0).stop 2023-10-16T12:34:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 msgr2=0x7f75f00c2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75f00c2be0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f75ec002230 tx=0x7f75ec001cb0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 shutdown_connections 2023-10-16T12:34:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f75d406ae70 0x7f75d406d330 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f01569c0 0x7f75f0156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f00c3730 0x7f75f00c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 --2- 172.21.15.60:0/1438218713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75dc0039a0 0x7f75f00c2be0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 >> 172.21.15.60:0/1438218713 conn(0x7f75f00ce6a0 msgr2=0x7f75f00cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 shutdown_connections 2023-10-16T12:34:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.114+0000 7f75f69a0640 1 -- 172.21.15.60:0/1438218713 wait complete. 2023-10-16T12:34:03.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:03.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:03.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.358+0000 7f7726e2d640 1 Processor -- start 2023-10-16T12:34:03.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.359+0000 7f7726e2d640 1 -- start start 2023-10-16T12:34:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.360+0000 7f7726e2d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77201618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.360+0000 7f7726e2d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f772006c7d0 0x7f7720065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.360+0000 7f7726e2d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f772006cc80 0x7f7720065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.360+0000 7f7726e2d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772006b1e0 con 0x7f772006cc80 2023-10-16T12:34:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f7726e2d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77200d0890 con 0x7f772006c7d0 2023-10-16T12:34:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f7726e2d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200d05b0 con 0x7f772015d330 2023-10-16T12:34:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f77253ca640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f772006cc80 0x7f7720065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f77253ca640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f772006cc80 0x7f7720065ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48310/0 (socket says 172.21.15.60:48310) 2023-10-16T12:34:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f77253ca640 1 -- 172.21.15.60:0/1379111500 learned_addr learned my addr 172.21.15.60:0/1379111500 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f7717fff640 1 --2- 172.21.15.60:0/1379111500 >> v2:172.21.15.144:3300/0 conn(0x7f772006c7d0 0x7f7720065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.361+0000 7f7724bc9640 1 --2- 172.21.15.60:0/1379111500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77201618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.362+0000 7f77253ca640 1 -- 172.21.15.60:0/1379111500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 msgr2=0x7f77201618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:03.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.362+0000 7f77253ca640 1 --2- 172.21.15.60:0/1379111500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77201618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.362+0000 7f77253ca640 1 -- 172.21.15.60:0/1379111500 >> v2:172.21.15.144:3300/0 conn(0x7f772006c7d0 msgr2=0x7f7720065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.362+0000 7f77253ca640 1 --2- 172.21.15.60:0/1379111500 >> v2:172.21.15.144:3300/0 conn(0x7f772006c7d0 0x7f7720065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.362+0000 7f77253ca640 1 -- 172.21.15.60:0/1379111500 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7720160dd0 con 0x7f772006cc80 2023-10-16T12:34:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.363+0000 7f77253ca640 1 --2- 172.21.15.60:0/1379111500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006cc80 0x7f7720065ff0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77200b35b0 tx=0x7f771802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d3345b30db7c77 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.363+0000 7f77177fe640 1 -- 172.21.15.60:0/1379111500 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771802c9e0 con 0x7f772006cc80 2023-10-16T12:34:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.363+0000 7f77177fe640 1 -- 172.21.15.60:0/1379111500 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771802cb40 con 0x7f772006cc80 2023-10-16T12:34:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.363+0000 7f77177fe640 1 -- 172.21.15.60:0/1379111500 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77180343f0 con 0x7f772006cc80 2023-10-16T12:34:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 -- 172.21.15.60:0/1379111500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006cc80 msgr2=0x7f7720065ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 --2- 172.21.15.60:0/1379111500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006cc80 0x7f7720065ff0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77200b35b0 tx=0x7f771802be40 comp rx=0 tx=0).stop 2023-10-16T12:34:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 -- 172.21.15.60:0/1379111500 shutdown_connections 2023-10-16T12:34:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 --2- 172.21.15.60:0/1379111500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006cc80 0x7f7720065ff0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 --2- 172.21.15.60:0/1379111500 >> v2:172.21.15.144:3300/0 conn(0x7f772006c7d0 0x7f7720065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 --2- 172.21.15.60:0/1379111500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77201618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 -- 172.21.15.60:0/1379111500 >> 172.21.15.60:0/1379111500 conn(0x7f77200d0ab0 msgr2=0x7f77200d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 -- 172.21.15.60:0/1379111500 shutdown_connections 2023-10-16T12:34:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.364+0000 7f7726e2d640 1 -- 172.21.15.60:0/1379111500 wait complete. 2023-10-16T12:34:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.366+0000 7f7726e2d640 1 Processor -- start 2023-10-16T12:34:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.366+0000 7f7726e2d640 1 -- start start 2023-10-16T12:34:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7726e2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 0x7f7720156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7726e2d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772006cc80 0x7f77200af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7726e2d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77200ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7726e2d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200aed80 con 0x7f772006c7d0 2023-10-16T12:34:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7726e2d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200ade20 con 0x7f772006cc80 2023-10-16T12:34:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7726e2d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200ae100 con 0x7f772015d330 2023-10-16T12:34:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7724bc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 0x7f7720156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.367+0000 7f7724bc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 0x7f7720156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48320/0 (socket says 172.21.15.60:48320) 2023-10-16T12:34:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 -- 172.21.15.60:0/23919346 learned_addr learned my addr 172.21.15.60:0/23919346 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7717fff640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772006cc80 0x7f77200af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f77253ca640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77200ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 -- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 msgr2=0x7f77200ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77200ca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 -- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772006cc80 msgr2=0x7f77200af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772006cc80 0x7f77200af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 -- 172.21.15.60:0/23919346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718009b30 con 0x7f772006c7d0 2023-10-16T12:34:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7717fff640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772006cc80 0x7f77200af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.368+0000 7f7724bc9640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 0x7f7720156d10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f771c00d800 tx=0x7f771c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.369+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00b6c0 con 0x7f772006c7d0 2023-10-16T12:34:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.369+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c00b820 con 0x7f772006c7d0 2023-10-16T12:34:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.369+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00bbb0 con 0x7f772006c7d0 2023-10-16T12:34:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.369+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77200ae3e0 con 0x7f772006c7d0 2023-10-16T12:34:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.369+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77200ae9a0 con 0x7f772006c7d0 2023-10-16T12:34:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.370+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772004e160 con 0x7f772006c7d0 2023-10-16T12:34:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.371+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f771c00bd10 con 0x7f772006c7d0 2023-10-16T12:34:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.372+0000 7f77157fa640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f770c06ae70 0x7f770c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.372+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f771c089b10 con 0x7f772006c7d0 2023-10-16T12:34:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.372+0000 7f7717fff640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f770c06ae70 0x7f770c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.373+0000 7f7717fff640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f770c06ae70 0x7f770c06d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77200d16a0 tx=0x7f7710008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.376+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f771c054bc0 con 0x7f772006c7d0 2023-10-16T12:34:03.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.576+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7720161080 con 0x7f772006c7d0 2023-10-16T12:34:03.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.576+0000 7f77157fa640 1 -- 172.21.15.60:0/23919346 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f771c0581e0 con 0x7f772006c7d0 2023-10-16T12:34:03.579 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:03.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.579+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f770c06ae70 msgr2=0x7f770c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f770c06ae70 0x7f770c06d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77200d16a0 tx=0x7f7710008040 comp rx=0 tx=0).stop 2023-10-16T12:34:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 msgr2=0x7f7720156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 0x7f7720156d10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f771c00d800 tx=0x7f771c00dcc0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 shutdown_connections 2023-10-16T12:34:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f770c06ae70 0x7f770c06d330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d330 0x7f77200ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f772006cc80 0x7f77200af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 --2- 172.21.15.60:0/23919346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f772006c7d0 0x7f7720156d10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 >> 172.21.15.60:0/23919346 conn(0x7f77200d0ab0 msgr2=0x7f77200ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 shutdown_connections 2023-10-16T12:34:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.580+0000 7f7726e2d640 1 -- 172.21.15.60:0/23919346 wait complete. 2023-10-16T12:34:03.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:03.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.832+0000 7fad4a16a640 1 Processor -- start 2023-10-16T12:34:03.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.832+0000 7fad4a16a640 1 -- start start 2023-10-16T12:34:03.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.833+0000 7fad4a16a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad4415dde0 0x7fad440709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.833+0000 7fad4a16a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad44070f20 0x7fad440ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.833+0000 7fad4a16a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad44160a90 con 0x7fad4415c9f0 2023-10-16T12:34:03.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.833+0000 7fad4a16a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad44071350 con 0x7fad44070f20 2023-10-16T12:34:03.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.833+0000 7fad4a16a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440cdec0 con 0x7fad4415dde0 2023-10-16T12:34:03.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad44070f20 0x7fad440ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad44070f20 0x7fad440ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48326/0 (socket says 172.21.15.60:48326) 2023-10-16T12:34:03.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 -- 172.21.15.60:0/3018247454 learned_addr learned my addr 172.21.15.60:0/3018247454 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:03.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad48967640 1 --2- 172.21.15.60:0/3018247454 >> v2:172.21.15.144:3300/0 conn(0x7fad4415dde0 0x7fad440709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 -- 172.21.15.60:0/3018247454 >> v2:172.21.15.144:3300/0 conn(0x7fad4415dde0 msgr2=0x7fad440709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:03.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 --2- 172.21.15.60:0/3018247454 >> v2:172.21.15.144:3300/0 conn(0x7fad4415dde0 0x7fad440709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 -- 172.21.15.60:0/3018247454 >> v1:172.21.15.60:6789/0 conn(0x7fad4415c9f0 legacy=0x7fad440c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad49969640 1 -- 172.21.15.60:0/3018247454 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440cd900 con 0x7fad44070f20 2023-10-16T12:34:03.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.834+0000 7fad48967640 1 --2- 172.21.15.60:0/3018247454 >> v2:172.21.15.144:3300/0 conn(0x7fad4415dde0 0x7fad440709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:03.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.835+0000 7fad49969640 1 --2- 172.21.15.60:0/3018247454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad44070f20 0x7fad440ac890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad440b1260 tx=0x7fad3c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5127921cb573759 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:03.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad3bfff640 1 -- 172.21.15.60:0/3018247454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c02cb10 con 0x7fad44070f20 2023-10-16T12:34:03.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad3bfff640 1 -- 172.21.15.60:0/3018247454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c02cc70 con 0x7fad44070f20 2023-10-16T12:34:03.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad3bfff640 1 -- 172.21.15.60:0/3018247454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c0343f0 con 0x7fad44070f20 2023-10-16T12:34:03.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 -- 172.21.15.60:0/3018247454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad44070f20 msgr2=0x7fad440ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 --2- 172.21.15.60:0/3018247454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad44070f20 0x7fad440ac890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad440b1260 tx=0x7fad3c02bf70 comp rx=0 tx=0).stop 2023-10-16T12:34:03.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 -- 172.21.15.60:0/3018247454 shutdown_connections 2023-10-16T12:34:03.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 --2- 172.21.15.60:0/3018247454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad44070f20 0x7fad440ac890 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 --2- 172.21.15.60:0/3018247454 >> v2:172.21.15.144:3300/0 conn(0x7fad4415dde0 0x7fad440709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 -- 172.21.15.60:0/3018247454 >> 172.21.15.60:0/3018247454 conn(0x7fad440ce6a0 msgr2=0x7fad440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 -- 172.21.15.60:0/3018247454 shutdown_connections 2023-10-16T12:34:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.836+0000 7fad4a16a640 1 -- 172.21.15.60:0/3018247454 wait complete. 2023-10-16T12:34:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.838+0000 7fad4a16a640 1 Processor -- start 2023-10-16T12:34:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.838+0000 7fad4a16a640 1 -- start start 2023-10-16T12:34:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.839+0000 7fad4a16a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad44070f20 0x7fad44150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.839+0000 7fad4a16a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 0x7fad441502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.839+0000 7fad4a16a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad4415dde0 0x7fad4406ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.839+0000 7fad4a16a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4406a8e0 con 0x7fad4415c9f0 2023-10-16T12:34:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.839+0000 7fad4a16a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4406c270 con 0x7fad4415dde0 2023-10-16T12:34:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.839+0000 7fad4a16a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad44069f40 con 0x7fad44070f20 2023-10-16T12:34:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 0x7fad441502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 0x7fad441502c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48328/0 (socket says 172.21.15.60:48328) 2023-10-16T12:34:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 -- 172.21.15.60:0/929426813 learned_addr learned my addr 172.21.15.60:0/929426813 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad49969640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad4415dde0 0x7fad4406ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 -- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad44070f20 msgr2=0x7fad44150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad49168640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad44070f20 0x7fad44150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad44070f20 0x7fad44150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 -- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad4415dde0 msgr2=0x7fad4406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:03.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad4415dde0 0x7fad4406ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:03.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad48967640 1 -- 172.21.15.60:0/929426813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c009b30 con 0x7fad4415c9f0 2023-10-16T12:34:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.840+0000 7fad49969640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad4415dde0 0x7fad4406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.841+0000 7fad48967640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 0x7fad441502c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fad3400c7f0 tx=0x7fad3400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.841+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34007a30 con 0x7fad4415c9f0 2023-10-16T12:34:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.841+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad34007b90 con 0x7fad4415c9f0 2023-10-16T12:34:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.841+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400ce40 con 0x7fad4415c9f0 2023-10-16T12:34:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.841+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4406a220 con 0x7fad4415c9f0 2023-10-16T12:34:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.841+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4406bf80 con 0x7fad4415c9f0 2023-10-16T12:34:03.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.842+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4406aa20 con 0x7fad4415c9f0 2023-10-16T12:34:03.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.843+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fad340113f0 con 0x7fad4415c9f0 2023-10-16T12:34:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.844+0000 7fad39ffb640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2806ae70 0x7fad2806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.844+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad340897c0 con 0x7fad4415c9f0 2023-10-16T12:34:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.845+0000 7fad49168640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2806ae70 0x7fad2806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.846+0000 7fad49168640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2806ae70 0x7fad2806d330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fad40000f60 tx=0x7fad40009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:03.849+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad4406aa20 con 0x7fad4415c9f0 2023-10-16T12:34:04.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.047+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad4406aa20 con 0x7fad4415c9f0 2023-10-16T12:34:04.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.048+0000 7fad39ffb640 1 -- 172.21.15.60:0/929426813 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad4406aa20 con 0x7fad4415c9f0 2023-10-16T12:34:04.050 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:04.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2806ae70 msgr2=0x7fad2806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2806ae70 0x7fad2806d330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fad40000f60 tx=0x7fad40009040 comp rx=0 tx=0).stop 2023-10-16T12:34:04.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 msgr2=0x7fad441502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 0x7fad441502c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fad3400c7f0 tx=0x7fad3400cbf0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 shutdown_connections 2023-10-16T12:34:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2806ae70 0x7fad2806d330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad4415dde0 0x7fad4406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad4415c9f0 0x7fad441502c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 --2- 172.21.15.60:0/929426813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad44070f20 0x7fad44150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.050+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 >> 172.21.15.60:0/929426813 conn(0x7fad440ce6a0 msgr2=0x7fad440cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.051+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 shutdown_connections 2023-10-16T12:34:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.051+0000 7fad4a16a640 1 -- 172.21.15.60:0/929426813 wait complete. 2023-10-16T12:34:04.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:04.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.306+0000 7f0d6bfff640 1 Processor -- start 2023-10-16T12:34:04.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.306+0000 7f0d6bfff640 1 -- start start 2023-10-16T12:34:04.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.308+0000 7f0d6bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0d6c15cd70 0x7f0d6c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.308+0000 7f0d6bfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c15d150 con 0x7f0d6c0b8480 2023-10-16T12:34:04.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.308+0000 7f0d6bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0d1180 con 0x7f0d6c0cded0 2023-10-16T12:34:04.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.308+0000 7f0d6bfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c0d1460 con 0x7f0d6c15cd70 2023-10-16T12:34:04.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.308+0000 7f0d627fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0d6c0b8480 0x7f0d6c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44144/0 (socket says 172.21.15.60:44144) 2023-10-16T12:34:04.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.308+0000 7f0d627fc640 1 -- 172.21.15.60:0/4250528609 learned_addr learned my addr 172.21.15.60:0/4250528609 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6affd640 1 --2- 172.21.15.60:0/4250528609 >> v2:172.21.15.144:3300/0 conn(0x7f0d6c15cd70 0x7f0d6c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949037693 0 0) 0x7f0d6c15d150 con 0x7f0d6c0b8480 2023-10-16T12:34:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d58003520 con 0x7f0d6c0b8480 2023-10-16T12:34:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984144701 0 0) 0x7f0d58003520 con 0x7f0d6c0b8480 2023-10-16T12:34:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 >> v2:172.21.15.144:3300/0 conn(0x7f0d6c15cd70 msgr2=0x7f0d6c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 --2- 172.21.15.60:0/4250528609 >> v2:172.21.15.144:3300/0 conn(0x7f0d6c15cd70 0x7f0d6c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 >> v1:172.21.15.144:6789/0 conn(0x7f0d6c0cded0 legacy=0x7f0d6c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0d1740 con 0x7f0d6c0b8480 2023-10-16T12:34:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0d54001670 con 0x7f0d6c0b8480 2023-10-16T12:34:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 >> v1:172.21.15.60:6789/0 conn(0x7f0d6c0b8480 legacy=0x7f0d6c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d58003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d580047c0 0x7f0d58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d580051d0 0x7f0d5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0d1460 con 0x7f0d580039c0 2023-10-16T12:34:04.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5800c120 con 0x7f0d580051d0 2023-10-16T12:34:04.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.309+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5800c2e0 con 0x7f0d580047c0 2023-10-16T12:34:04.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6affd640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d58003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d627fc640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d580047c0 0x7f0d58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6b7fe640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d580051d0 0x7f0d5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6affd640 1 -- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d580047c0 msgr2=0x7f0d58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:04.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6affd640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d580047c0 0x7f0d58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6affd640 1 -- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d580051d0 msgr2=0x7f0d5800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:04.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6affd640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d580051d0 0x7f0d5800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6affd640 1 -- 172.21.15.60:0/4250528609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d58003520 con 0x7f0d580039c0 2023-10-16T12:34:04.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.310+0000 7f0d6b7fe640 1 -- 172.21.15.60:0/4250528609 reap_dead start 2023-10-16T12:34:04.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.311+0000 7f0d6affd640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d58003de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00ba30 tx=0x7f0d580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10dfba595d3d544 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.311+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c032640 con 0x7f0d580039c0 2023-10-16T12:34:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c0327a0 con 0x7f0d580039c0 2023-10-16T12:34:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6a7fc640 1 -- 172.21.15.60:0/4250528609 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c032b30 con 0x7f0d580039c0 2023-10-16T12:34:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 -- 172.21.15.60:0/4250528609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 msgr2=0x7f0d58003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d58003de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00ba30 tx=0x7f0d580057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 -- 172.21.15.60:0/4250528609 shutdown_connections 2023-10-16T12:34:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/4250528609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d58003de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 -- 172.21.15.60:0/4250528609 >> 172.21.15.60:0/4250528609 conn(0x7f0d6c0cf570 msgr2=0x7f0d6c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 -- 172.21.15.60:0/4250528609 shutdown_connections 2023-10-16T12:34:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.312+0000 7f0d6bfff640 1 -- 172.21.15.60:0/4250528609 wait complete. 2023-10-16T12:34:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.314+0000 7f0d6bfff640 1 Processor -- start 2023-10-16T12:34:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.314+0000 7f0d6bfff640 1 -- start start 2023-10-16T12:34:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.315+0000 7f0d6bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d6c0ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.315+0000 7f0d6bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d6c0ab440 0x7f0d6c150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.315+0000 7f0d6bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d6c0ab8c0 0x7f0d6c151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.315+0000 7f0d6bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c15d340 con 0x7f0d580039c0 2023-10-16T12:34:04.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.315+0000 7f0d6bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c15d590 con 0x7f0d6c0ab440 2023-10-16T12:34:04.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.315+0000 7f0d6bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c15d870 con 0x7f0d6c0ab8c0 2023-10-16T12:34:04.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d6c0ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d6c0ad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48352/0 (socket says 172.21.15.60:48352) 2023-10-16T12:34:04.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 -- 172.21.15.60:0/3797978555 learned_addr learned my addr 172.21.15.60:0/3797978555 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:04.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d627fc640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d6c0ab440 0x7f0d6c150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 -- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d6c0ab8c0 msgr2=0x7f0d6c151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6b7fe640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d6c0ab8c0 0x7f0d6c151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d6c0ab8c0 0x7f0d6c151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 -- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d6c0ab440 msgr2=0x7f0d6c150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d6c0ab440 0x7f0d6c150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d6affd640 1 -- 172.21.15.60:0/3797978555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c15db50 con 0x7f0d580039c0 2023-10-16T12:34:04.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.316+0000 7f0d627fc640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d6c0ab440 0x7f0d6c150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:04.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.317+0000 7f0d6affd640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d6c0ad3b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c02bec0 tx=0x7f0d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:04.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.317+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c03b9f0 con 0x7f0d580039c0 2023-10-16T12:34:04.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.317+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c03bb50 con 0x7f0d580039c0 2023-10-16T12:34:04.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.317+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c030410 con 0x7f0d580039c0 2023-10-16T12:34:04.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.317+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c15de30 con 0x7f0d580039c0 2023-10-16T12:34:04.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.317+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c0a9ed0 con 0x7f0d580039c0 2023-10-16T12:34:04.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.318+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c15dfc0 con 0x7f0d580039c0 2023-10-16T12:34:04.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.319+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0d5c030570 con 0x7f0d580039c0 2023-10-16T12:34:04.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.320+0000 7f0d63fff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d4c06ae70 0x7f0d4c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.320+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d5c0abdc0 con 0x7f0d580039c0 2023-10-16T12:34:04.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.321+0000 7f0d627fc640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d4c06ae70 0x7f0d4c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.322+0000 7f0d627fc640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d4c06ae70 0x7f0d4c06d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c15bd30 tx=0x7f0d54000ef0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:04.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.325+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d5c076de0 con 0x7f0d580039c0 2023-10-16T12:34:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.522+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d6c0b56d0 con 0x7f0d580039c0 2023-10-16T12:34:04.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.522+0000 7f0d63fff640 1 -- 172.21.15.60:0/3797978555 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0d5c07a400 con 0x7f0d580039c0 2023-10-16T12:34:04.525 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:04.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.525+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d4c06ae70 msgr2=0x7f0d4c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.525+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d4c06ae70 0x7f0d4c06d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c15bd30 tx=0x7f0d54000ef0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.525+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 msgr2=0x7f0d6c0ad3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.525+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d6c0ad3b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c02bec0 tx=0x7f0d5c004730 comp rx=0 tx=0).stop 2023-10-16T12:34:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 shutdown_connections 2023-10-16T12:34:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f0d4c06ae70 0x7f0d4c06d330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d6c0ab8c0 0x7f0d6c151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d6c0ab440 0x7f0d6c150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 --2- 172.21.15.60:0/3797978555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d580039c0 0x7f0d6c0ad3b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 >> 172.21.15.60:0/3797978555 conn(0x7f0d6c0cf570 msgr2=0x7f0d6c0b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:04.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 shutdown_connections 2023-10-16T12:34:04.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.526+0000 7f0d6bfff640 1 -- 172.21.15.60:0/3797978555 wait complete. 2023-10-16T12:34:04.559 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-16T12:34:04.560 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:34:04.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.781+0000 7f2e4135a640 1 Processor -- start 2023-10-16T12:34:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.782+0000 7f2e4135a640 1 -- start start 2023-10-16T12:34:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e4135a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e3c15aaa0 0x7f2e3c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e4135a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2e3c15ae80 0x7f2e3c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e4135a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c15ed60 con 0x7f2e3c0ccaa0 2023-10-16T12:34:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e4135a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c06b810 con 0x7f2e3c15aaa0 2023-10-16T12:34:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e4135a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c0707e0 con 0x7f2e3c15ae80 2023-10-16T12:34:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e3affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e3c15aaa0 0x7f2e3c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e3affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e3c15aaa0 0x7f2e3c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48366/0 (socket says 172.21.15.60:48366) 2023-10-16T12:34:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.783+0000 7f2e3affd640 1 -- 172.21.15.60:0/731163246 learned_addr learned my addr 172.21.15.60:0/731163246 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:04.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.784+0000 7f2e3a7fc640 1 --2- 172.21.15.60:0/731163246 >> v2:172.21.15.144:3300/0 conn(0x7f2e3c15ae80 0x7f2e3c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.784+0000 7f2e3affd640 1 -- 172.21.15.60:0/731163246 >> v2:172.21.15.144:3300/0 conn(0x7f2e3c15ae80 msgr2=0x7f2e3c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:04.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.784+0000 7f2e3affd640 1 --2- 172.21.15.60:0/731163246 >> v2:172.21.15.144:3300/0 conn(0x7f2e3c15ae80 0x7f2e3c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.784+0000 7f2e3affd640 1 -- 172.21.15.60:0/731163246 >> v1:172.21.15.60:6789/0 conn(0x7f2e3c0ccaa0 legacy=0x7f2e3c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.784+0000 7f2e3affd640 1 -- 172.21.15.60:0/731163246 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c15ef90 con 0x7f2e3c15aaa0 2023-10-16T12:34:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.784+0000 7f2e39ffb640 1 -- 172.21.15.60:0/731163246 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679837881 0 0) 0x7f2e3c15ed60 con 0x7f2e3c0ccaa0 2023-10-16T12:34:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.785+0000 7f2e3affd640 1 --2- 172.21.15.60:0/731163246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15e820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0c59c0 tx=0x7f2e2c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324c1ebcfdbc5086 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.785+0000 7f2e39ffb640 1 -- 172.21.15.60:0/731163246 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c02cb10 con 0x7f2e3c15aaa0 2023-10-16T12:34:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.785+0000 7f2e39ffb640 1 -- 172.21.15.60:0/731163246 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2c02cc70 con 0x7f2e3c15aaa0 2023-10-16T12:34:04.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e39ffb640 1 -- 172.21.15.60:0/731163246 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c0343f0 con 0x7f2e3c15aaa0 2023-10-16T12:34:04.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 -- 172.21.15.60:0/731163246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15aaa0 msgr2=0x7f2e3c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 --2- 172.21.15.60:0/731163246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15e820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0c59c0 tx=0x7f2e2c02bf70 comp rx=0 tx=0).stop 2023-10-16T12:34:04.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 -- 172.21.15.60:0/731163246 shutdown_connections 2023-10-16T12:34:04.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 --2- 172.21.15.60:0/731163246 >> v2:172.21.15.144:3300/0 conn(0x7f2e3c15ae80 0x7f2e3c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 --2- 172.21.15.60:0/731163246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15e820 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 -- 172.21.15.60:0/731163246 >> 172.21.15.60:0/731163246 conn(0x7f2e3c0cb410 msgr2=0x7f2e3c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:04.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 -- 172.21.15.60:0/731163246 shutdown_connections 2023-10-16T12:34:04.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.786+0000 7f2e4135a640 1 -- 172.21.15.60:0/731163246 wait complete. 2023-10-16T12:34:04.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.788+0000 7f2e4135a640 1 Processor -- start 2023-10-16T12:34:04.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.788+0000 7f2e4135a640 1 -- start start 2023-10-16T12:34:04.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.789+0000 7f2e4135a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e3c0ccaa0 0x7f2e3c154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.789+0000 7f2e4135a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.789+0000 7f2e4135a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 0x7f2e3c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.789+0000 7f2e4135a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c157730 con 0x7f2e3c15ae80 2023-10-16T12:34:04.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.789+0000 7f2e4135a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c1579d0 con 0x7f2e3c15aaa0 2023-10-16T12:34:04.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.789+0000 7f2e4135a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c157cb0 con 0x7f2e3c0ccaa0 2023-10-16T12:34:04.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 0x7f2e3c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 0x7f2e3c15d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48378/0 (socket says 172.21.15.60:48378) 2023-10-16T12:34:04.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 -- 172.21.15.60:0/1876533825 learned_addr learned my addr 172.21.15.60:0/1876533825 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:04.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3affd640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e3c0ccaa0 0x7f2e3c154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3a7fc640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 -- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e3c0ccaa0 msgr2=0x7f2e3c154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e3c0ccaa0 0x7f2e3c154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 -- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e3c15aaa0 msgr2=0x7f2e3c15cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:04.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:04.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 -- 172.21.15.60:0/1876533825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c009b30 con 0x7f2e3c15ae80 2023-10-16T12:34:04.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.790+0000 7f2e3b7fe640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 0x7f2e3c15d460 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c480 tx=0x7f2e3400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:04.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.791+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34005480 con 0x7f2e3c15ae80 2023-10-16T12:34:04.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.791+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e340035f0 con 0x7f2e3c15ae80 2023-10-16T12:34:04.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.791+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c157f90 con 0x7f2e3c15ae80 2023-10-16T12:34:04.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.791+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e340039a0 con 0x7f2e3c15ae80 2023-10-16T12:34:04.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.791+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c15d9a0 con 0x7f2e3c15ae80 2023-10-16T12:34:04.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.792+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c1584b0 con 0x7f2e3c15ae80 2023-10-16T12:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.793+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2e34003b00 con 0x7f2e3c15ae80 2023-10-16T12:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.794+0000 7f2e337fe640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2e1c06ae70 0x7f2e1c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.794+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e34089210 con 0x7f2e3c15ae80 2023-10-16T12:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.795+0000 7f2e3affd640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2e1c06ae70 0x7f2e1c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.796+0000 7f2e3affd640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2e1c06ae70 0x7f2e1c06d330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c035000 tx=0x7f2e2c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:04.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.799+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e3c1584b0 con 0x7f2e3c15ae80 2023-10-16T12:34:05.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:04.998+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e3c1584b0 con 0x7f2e3c15ae80 2023-10-16T12:34:05.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.000+0000 7f2e337fe640 1 -- 172.21.15.60:0/1876533825 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f2e3c1584b0 con 0x7f2e3c15ae80 2023-10-16T12:34:05.003 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:05.003 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:05.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2e1c06ae70 msgr2=0x7f2e1c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2e1c06ae70 0x7f2e1c06d330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c035000 tx=0x7f2e2c030000 comp rx=0 tx=0).stop 2023-10-16T12:34:05.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 msgr2=0x7f2e3c15d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 0x7f2e3c15d460 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c480 tx=0x7f2e3400c940 comp rx=0 tx=0).stop 2023-10-16T12:34:05.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 shutdown_connections 2023-10-16T12:34:05.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2e1c06ae70 0x7f2e1c06d330 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e3c15ae80 0x7f2e3c15d460 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e3c15aaa0 0x7f2e3c15cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 --2- 172.21.15.60:0/1876533825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e3c0ccaa0 0x7f2e3c154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 >> 172.21.15.60:0/1876533825 conn(0x7f2e3c0cb410 msgr2=0x7f2e3c0ce940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:05.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 shutdown_connections 2023-10-16T12:34:05.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.003+0000 7f2e4135a640 1 -- 172.21.15.60:0/1876533825 wait complete. 2023-10-16T12:34:05.025 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:05.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.255+0000 7f459359e640 1 Processor -- start 2023-10-16T12:34:05.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.255+0000 7f459359e640 1 -- start start 2023-10-16T12:34:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.257+0000 7f459359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4594070380 0x7f45940707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.257+0000 7f459359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45940ca680 0x7f45940cab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.257+0000 7f459359e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459415af90 con 0x7f459415abb0 2023-10-16T12:34:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.257+0000 7f459359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4594070d30 con 0x7f4594070380 2023-10-16T12:34:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.257+0000 7f459359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45940b3010 con 0x7f45940ca680 2023-10-16T12:34:05.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.258+0000 7f459259c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f459415abb0 0x7f45940b25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60370/0 (socket says 172.21.15.60:60370) 2023-10-16T12:34:05.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.258+0000 7f459259c640 1 -- 172.21.15.60:0/3285435625 learned_addr learned my addr 172.21.15.60:0/3285435625 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.258+0000 7f4592d9d640 1 --2- 172.21.15.60:0/3285435625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45940ca680 0x7f45940cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.258+0000 7f458bfff640 1 --2- 172.21.15.60:0/3285435625 >> v2:172.21.15.144:3300/0 conn(0x7f4594070380 0x7f45940707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.259+0000 7f4592d9d640 1 -- 172.21.15.60:0/3285435625 >> v2:172.21.15.144:3300/0 conn(0x7f4594070380 msgr2=0x7f45940707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.259+0000 7f4592d9d640 1 --2- 172.21.15.60:0/3285435625 >> v2:172.21.15.144:3300/0 conn(0x7f4594070380 0x7f45940707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.259+0000 7f4592d9d640 1 -- 172.21.15.60:0/3285435625 >> v1:172.21.15.144:6789/0 conn(0x7f459415abb0 legacy=0x7f45940b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.259+0000 7f4592d9d640 1 -- 172.21.15.60:0/3285435625 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45940b32f0 con 0x7f45940ca680 2023-10-16T12:34:05.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.260+0000 7f4592d9d640 1 --2- 172.21.15.60:0/3285435625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45940ca680 0x7f45940cab10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f458c009d90 tx=0x7f458c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c91fc22963a4115 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.261+0000 7f4591d9b640 1 -- 172.21.15.60:0/3285435625 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c02cc50 con 0x7f45940ca680 2023-10-16T12:34:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.261+0000 7f4591d9b640 1 -- 172.21.15.60:0/3285435625 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c02cdb0 con 0x7f45940ca680 2023-10-16T12:34:05.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.261+0000 7f4591d9b640 1 -- 172.21.15.60:0/3285435625 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c034660 con 0x7f45940ca680 2023-10-16T12:34:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 -- 172.21.15.60:0/3285435625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45940ca680 msgr2=0x7f45940cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 --2- 172.21.15.60:0/3285435625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45940ca680 0x7f45940cab10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f458c009d90 tx=0x7f458c02bea0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 -- 172.21.15.60:0/3285435625 shutdown_connections 2023-10-16T12:34:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 --2- 172.21.15.60:0/3285435625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45940ca680 0x7f45940cab10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 --2- 172.21.15.60:0/3285435625 >> v2:172.21.15.144:3300/0 conn(0x7f4594070380 0x7f45940707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 -- 172.21.15.60:0/3285435625 >> 172.21.15.60:0/3285435625 conn(0x7f45940cf790 msgr2=0x7f45940cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 -- 172.21.15.60:0/3285435625 shutdown_connections 2023-10-16T12:34:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.262+0000 7f459359e640 1 -- 172.21.15.60:0/3285435625 wait complete. 2023-10-16T12:34:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.265+0000 7f459359e640 1 Processor -- start 2023-10-16T12:34:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.265+0000 7f459359e640 1 -- start start 2023-10-16T12:34:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.266+0000 7f459359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4594070380 0x7f459406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.266+0000 7f459359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 0x7f459406c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.266+0000 7f459359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f459415abb0 0x7f45941f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.266+0000 7f459359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459406c880 con 0x7f45940ca680 2023-10-16T12:34:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.266+0000 7f459359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459406cbb0 con 0x7f4594070380 2023-10-16T12:34:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.266+0000 7f459359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459406b8a0 con 0x7f459415abb0 2023-10-16T12:34:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.267+0000 7f458bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 0x7f459406c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.267+0000 7f458bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 0x7f459406c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48388/0 (socket says 172.21.15.60:48388) 2023-10-16T12:34:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.267+0000 7f458bfff640 1 -- 172.21.15.60:0/1385310636 learned_addr learned my addr 172.21.15.60:0/1385310636 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.267+0000 7f459259c640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4594070380 0x7f459406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.267+0000 7f4592d9d640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f459415abb0 0x7f45941f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458bfff640 1 -- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f459415abb0 msgr2=0x7f45941f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458bfff640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f459415abb0 0x7f45941f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458bfff640 1 -- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4594070380 msgr2=0x7f459406d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458bfff640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4594070380 0x7f459406d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458bfff640 1 -- 172.21.15.60:0/1385310636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c009a70 con 0x7f45940ca680 2023-10-16T12:34:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458bfff640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 0x7f459406c340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f457c00e9b0 tx=0x7f457c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c009670 con 0x7f45940ca680 2023-10-16T12:34:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c00c9a0 con 0x7f45940ca680 2023-10-16T12:34:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.268+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459406cdc0 con 0x7f45940ca680 2023-10-16T12:34:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.269+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c00cd50 con 0x7f45940ca680 2023-10-16T12:34:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.269+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459406be20 con 0x7f45940ca680 2023-10-16T12:34:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.270+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459404e9b0 con 0x7f45940ca680 2023-10-16T12:34:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.271+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f457c005990 con 0x7f45940ca680 2023-10-16T12:34:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.272+0000 7f458affd640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f457406ae70 0x7f457406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.272+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f457c089830 con 0x7f45940ca680 2023-10-16T12:34:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.273+0000 7f459259c640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f457406ae70 0x7f457406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.274+0000 7f459259c640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f457406ae70 0x7f457406d330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4584007fd0 tx=0x7f4584007e80 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.276+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f457c054a00 con 0x7f45940ca680 2023-10-16T12:34:05.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.470+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45940c5b00 con 0x7f45940ca680 2023-10-16T12:34:05.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.472+0000 7f458affd640 1 -- 172.21.15.60:0/1385310636 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f457c0170a0 con 0x7f45940ca680 2023-10-16T12:34:05.475 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:05.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f457406ae70 msgr2=0x7f457406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f457406ae70 0x7f457406d330 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4584007fd0 tx=0x7f4584007e80 comp rx=0 tx=0).stop 2023-10-16T12:34:05.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 msgr2=0x7f459406c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 0x7f459406c340 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f457c00e9b0 tx=0x7f457c00ee70 comp rx=0 tx=0).stop 2023-10-16T12:34:05.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 shutdown_connections 2023-10-16T12:34:05.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f457406ae70 0x7f457406d330 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f459415abb0 0x7f45941f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45940ca680 0x7f459406c340 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 --2- 172.21.15.60:0/1385310636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4594070380 0x7f459406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 >> 172.21.15.60:0/1385310636 conn(0x7f45940cf790 msgr2=0x7f45940cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:05.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 shutdown_connections 2023-10-16T12:34:05.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.475+0000 7f459359e640 1 -- 172.21.15.60:0/1385310636 wait complete. 2023-10-16T12:34:05.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.718+0000 7fb1c8abb640 1 Processor -- start 2023-10-16T12:34:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.718+0000 7fb1c8abb640 1 -- start start 2023-10-16T12:34:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.720+0000 7fb1c8abb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1c406fb50 0x7fb1c40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.720+0000 7fb1c8abb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c406c9e0 0x7fb1c40ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.720+0000 7fb1c8abb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c40ca560 con 0x7fb1c40c9060 2023-10-16T12:34:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.720+0000 7fb1c8abb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c40c9950 con 0x7fb1c406fb50 2023-10-16T12:34:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.720+0000 7fb1c8abb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40c9c30 con 0x7fb1c406c9e0 2023-10-16T12:34:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1c406fb50 0x7fb1c40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1c406fb50 0x7fb1c40cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48400/0 (socket says 172.21.15.60:48400) 2023-10-16T12:34:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 -- 172.21.15.60:0/3075575294 learned_addr learned my addr 172.21.15.60:0/3075575294 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1c3fff640 1 --2- 172.21.15.60:0/3075575294 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c406c9e0 0x7fb1c40ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 -- 172.21.15.60:0/3075575294 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c406c9e0 msgr2=0x7fb1c40ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/3075575294 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c406c9e0 0x7fb1c40ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 -- 172.21.15.60:0/3075575294 >> v1:172.21.15.144:6789/0 conn(0x7fb1c40c9060 legacy=0x7fb1c406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.721+0000 7fb1bbfff640 1 -- 172.21.15.60:0/3075575294 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40c9f10 con 0x7fb1c406fb50 2023-10-16T12:34:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.722+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/3075575294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40cc8a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac009870 tx=0x7fb1ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82521a23af4aa8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c2ffd640 1 -- 172.21.15.60:0/3075575294 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac031640 con 0x7fb1c406fb50 2023-10-16T12:34:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c2ffd640 1 -- 172.21.15.60:0/3075575294 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ac0317a0 con 0x7fb1c406fb50 2023-10-16T12:34:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c2ffd640 1 -- 172.21.15.60:0/3075575294 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac031b30 con 0x7fb1c406fb50 2023-10-16T12:34:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c8abb640 1 -- 172.21.15.60:0/3075575294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 msgr2=0x7fb1c40cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/3075575294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40cc8a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac009870 tx=0x7fb1ac02b8f0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c8abb640 1 -- 172.21.15.60:0/3075575294 shutdown_connections 2023-10-16T12:34:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/3075575294 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c406c9e0 0x7fb1c40ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/3075575294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40cc8a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.723+0000 7fb1c8abb640 1 -- 172.21.15.60:0/3075575294 >> 172.21.15.60:0/3075575294 conn(0x7fb1c40ce6a0 msgr2=0x7fb1c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.724+0000 7fb1c8abb640 1 -- 172.21.15.60:0/3075575294 shutdown_connections 2023-10-16T12:34:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.724+0000 7fb1c8abb640 1 -- 172.21.15.60:0/3075575294 wait complete. 2023-10-16T12:34:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.725+0000 7fb1c8abb640 1 Processor -- start 2023-10-16T12:34:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.725+0000 7fb1c8abb640 1 -- start start 2023-10-16T12:34:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.726+0000 7fb1c8abb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1c406c9e0 0x7fb1c415ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.726+0000 7fb1c8abb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.726+0000 7fb1c8abb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c40c9060 0x7fb1c415d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1c8abb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c415b420 con 0x7fb1c406fb50 2023-10-16T12:34:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1c8abb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c415b700 con 0x7fb1c406c9e0 2023-10-16T12:34:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1c8abb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c415b9e0 con 0x7fb1c40c9060 2023-10-16T12:34:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48410/0 (socket says 172.21.15.60:48410) 2023-10-16T12:34:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 -- 172.21.15.60:0/911224797 learned_addr learned my addr 172.21.15.60:0/911224797 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1c3fff640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c40c9060 0x7fb1c415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1c37fe640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1c406c9e0 0x7fb1c415ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 -- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c40c9060 msgr2=0x7fb1c415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c40c9060 0x7fb1c415d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 -- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1c406c9e0 msgr2=0x7fb1c415ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1c406c9e0 0x7fb1c415ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1bbfff640 1 -- 172.21.15.60:0/911224797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac0094d0 con 0x7fb1c406fb50 2023-10-16T12:34:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.727+0000 7fb1c37fe640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1c406c9e0 0x7fb1c415ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.728+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40c8210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac009870 tx=0x7fb1ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.728+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac03ac60 con 0x7fb1c406fb50 2023-10-16T12:34:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.728+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ac03adc0 con 0x7fb1c406fb50 2023-10-16T12:34:05.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.728+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac044630 con 0x7fb1c406fb50 2023-10-16T12:34:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.728+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c415bcc0 con 0x7fb1c406fb50 2023-10-16T12:34:05.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.728+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c415c280 con 0x7fb1c406fb50 2023-10-16T12:34:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.729+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c404e9b0 con 0x7fb1c406fb50 2023-10-16T12:34:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.730+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb1ac02f050 con 0x7fb1c406fb50 2023-10-16T12:34:05.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.731+0000 7fb1c0ff9640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a406aec0 0x7fb1a406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.731+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1ac0ab520 con 0x7fb1c406fb50 2023-10-16T12:34:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.732+0000 7fb1c37fe640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a406aec0 0x7fb1a406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.733+0000 7fb1c37fe640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a406aec0 0x7fb1a406d380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4003eb0 tx=0x7fb1b4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:05.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.735+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1ac076640 con 0x7fb1c406fb50 2023-10-16T12:34:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.930+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1c40b4620 con 0x7fb1c406fb50 2023-10-16T12:34:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.931+0000 7fb1c0ff9640 1 -- 172.21.15.60:0/911224797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1ac079c60 con 0x7fb1c406fb50 2023-10-16T12:34:05.934 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a406aec0 msgr2=0x7fb1a406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a406aec0 0x7fb1a406d380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4003eb0 tx=0x7fb1b4009040 comp rx=0 tx=0).stop 2023-10-16T12:34:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 msgr2=0x7fb1c40c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40c8210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac009870 tx=0x7fb1ac0046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 shutdown_connections 2023-10-16T12:34:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a406aec0 0x7fb1a406d380 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1c40c9060 0x7fb1c415d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1c406fb50 0x7fb1c40c8210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 --2- 172.21.15.60:0/911224797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1c406c9e0 0x7fb1c415ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 >> 172.21.15.60:0/911224797 conn(0x7fb1c40ce6a0 msgr2=0x7fb1c40af930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 shutdown_connections 2023-10-16T12:34:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:05.934+0000 7fb1c8abb640 1 -- 172.21.15.60:0/911224797 wait complete. 2023-10-16T12:34:05.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:06.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.185+0000 7f2b292a2640 1 Processor -- start 2023-10-16T12:34:06.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.185+0000 7f2b292a2640 1 -- start start 2023-10-16T12:34:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.186+0000 7f2b292a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24160530 0x7f2b241609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.186+0000 7f2b292a2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b24069fa0 0x7f2b2406a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.186+0000 7f2b292a2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b240ac890 con 0x7f2b24065ab0 2023-10-16T12:34:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.186+0000 7f2b292a2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b240ae800 con 0x7f2b24069fa0 2023-10-16T12:34:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.186+0000 7f2b292a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240ae1f0 con 0x7f2b24160530 2023-10-16T12:34:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.187+0000 7f2b237fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b24069fa0 0x7f2b2406a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.187+0000 7f2b227fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2b24065ab0 0x7f2b24160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44154/0 (socket says 172.21.15.60:44154) 2023-10-16T12:34:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.187+0000 7f2b227fc640 1 -- 172.21.15.60:0/1532225793 learned_addr learned my addr 172.21.15.60:0/1532225793 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.187+0000 7f2b22ffd640 1 --2- 172.21.15.60:0/1532225793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24160530 0x7f2b241609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.188+0000 7f2b21ffb640 1 -- 172.21.15.60:0/1532225793 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575393801 0 0) 0x7f2b240ac890 con 0x7f2b24065ab0 2023-10-16T12:34:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.188+0000 7f2b237fe640 1 -- 172.21.15.60:0/1532225793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24160530 msgr2=0x7f2b241609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.188+0000 7f2b237fe640 1 --2- 172.21.15.60:0/1532225793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24160530 0x7f2b241609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.188+0000 7f2b237fe640 1 -- 172.21.15.60:0/1532225793 >> v1:172.21.15.60:6789/0 conn(0x7f2b24065ab0 legacy=0x7f2b24160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.188+0000 7f2b237fe640 1 -- 172.21.15.60:0/1532225793 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240acae0 con 0x7f2b24069fa0 2023-10-16T12:34:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.189+0000 7f2b237fe640 1 --2- 172.21.15.60:0/1532225793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24069fa0 0x7f2b2406a400 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0097a0 tx=0x7f2b1c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b023f88f529a09d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.189+0000 7f2b21ffb640 1 -- 172.21.15.60:0/1532225793 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c031780 con 0x7f2b24069fa0 2023-10-16T12:34:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.189+0000 7f2b21ffb640 1 -- 172.21.15.60:0/1532225793 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c0318e0 con 0x7f2b24069fa0 2023-10-16T12:34:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.189+0000 7f2b21ffb640 1 -- 172.21.15.60:0/1532225793 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c031c70 con 0x7f2b24069fa0 2023-10-16T12:34:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 -- 172.21.15.60:0/1532225793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24069fa0 msgr2=0x7f2b2406a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 --2- 172.21.15.60:0/1532225793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24069fa0 0x7f2b2406a400 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0097a0 tx=0x7f2b1c02b820 comp rx=0 tx=0).stop 2023-10-16T12:34:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 -- 172.21.15.60:0/1532225793 shutdown_connections 2023-10-16T12:34:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 --2- 172.21.15.60:0/1532225793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24069fa0 0x7f2b2406a400 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 --2- 172.21.15.60:0/1532225793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24160530 0x7f2b241609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 -- 172.21.15.60:0/1532225793 >> 172.21.15.60:0/1532225793 conn(0x7f2b2406bc10 msgr2=0x7f2b2406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 -- 172.21.15.60:0/1532225793 shutdown_connections 2023-10-16T12:34:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.190+0000 7f2b292a2640 1 -- 172.21.15.60:0/1532225793 wait complete. 2023-10-16T12:34:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.192+0000 7f2b292a2640 1 Processor -- start 2023-10-16T12:34:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.192+0000 7f2b292a2640 1 -- start start 2023-10-16T12:34:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b292a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24065ab0 0x7f2b241f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b292a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b24069fa0 0x7f2b2414dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b292a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 0x7f2b240c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b292a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2414e720 con 0x7f2b24160530 2023-10-16T12:34:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b292a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2414ea00 con 0x7f2b24069fa0 2023-10-16T12:34:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b292a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2414ece0 con 0x7f2b24065ab0 2023-10-16T12:34:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 0x7f2b240c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 0x7f2b240c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48428/0 (socket says 172.21.15.60:48428) 2023-10-16T12:34:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 -- 172.21.15.60:0/466602418 learned_addr learned my addr 172.21.15.60:0/466602418 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b22ffd640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24065ab0 0x7f2b241f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 -- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24065ab0 msgr2=0x7f2b241f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24065ab0 0x7f2b241f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 -- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b24069fa0 msgr2=0x7f2b2414dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b24069fa0 0x7f2b2414dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.193+0000 7f2b237fe640 1 -- 172.21.15.60:0/466602418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c009400 con 0x7f2b24160530 2023-10-16T12:34:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.194+0000 7f2b237fe640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 0x7f2b240c64f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0096f0 tx=0x7f2b1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.194+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c032820 con 0x7f2b24160530 2023-10-16T12:34:06.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.194+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c032980 con 0x7f2b24160530 2023-10-16T12:34:06.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.194+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c032d10 con 0x7f2b24160530 2023-10-16T12:34:06.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.194+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b240c3eb0 con 0x7f2b24160530 2023-10-16T12:34:06.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.194+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b240c6a30 con 0x7f2b24160530 2023-10-16T12:34:06.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.196+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b240c6c20 con 0x7f2b24160530 2023-10-16T12:34:06.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.197+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2b1c04e440 con 0x7f2b24160530 2023-10-16T12:34:06.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.198+0000 7f2b1b7fe640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2b0406ae70 0x7f2b0406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.198+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b1c0ab630 con 0x7f2b24160530 2023-10-16T12:34:06.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.198+0000 7f2b22ffd640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2b0406ae70 0x7f2b0406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.200+0000 7f2b22ffd640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2b0406ae70 0x7f2b0406d330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2b14007380 tx=0x7f2b1400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:06.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.202+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b1c076820 con 0x7f2b24160530 2023-10-16T12:34:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.396+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b2404e160 con 0x7f2b24160530 2023-10-16T12:34:06.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.397+0000 7f2b1b7fe640 1 -- 172.21.15.60:0/466602418 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b1c079e40 con 0x7f2b24160530 2023-10-16T12:34:06.400 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:06.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2b0406ae70 msgr2=0x7f2b0406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2b0406ae70 0x7f2b0406d330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2b14007380 tx=0x7f2b1400b040 comp rx=0 tx=0).stop 2023-10-16T12:34:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 msgr2=0x7f2b240c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 0x7f2b240c64f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0096f0 tx=0x7f2b1c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 shutdown_connections 2023-10-16T12:34:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f2b0406ae70 0x7f2b0406d330 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b24160530 0x7f2b240c64f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b24069fa0 0x7f2b2414dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 --2- 172.21.15.60:0/466602418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b24065ab0 0x7f2b241f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 >> 172.21.15.60:0/466602418 conn(0x7f2b2406bc10 msgr2=0x7f2b240ce120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 shutdown_connections 2023-10-16T12:34:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.400+0000 7f2b292a2640 1 -- 172.21.15.60:0/466602418 wait complete. 2023-10-16T12:34:06.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T12:34:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.639+0000 7ff210a0d640 1 Processor -- start 2023-10-16T12:34:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.639+0000 7ff210a0d640 1 -- start start 2023-10-16T12:34:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.640+0000 7ff210a0d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff20c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.641+0000 7ff210a0d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c0b53b0 con 0x7ff20c0cdae0 2023-10-16T12:34:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.641+0000 7ff210a0d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c160b60 con 0x7ff20c0d1f80 2023-10-16T12:34:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.641+0000 7ff210a0d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c1617e0 con 0x7ff20c0cf2b0 2023-10-16T12:34:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.641+0000 7ff20affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff20c0cdae0 0x7ff20c0b4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44162/0 (socket says 172.21.15.60:44162) 2023-10-16T12:34:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.641+0000 7ff20affd640 1 -- 172.21.15.60:0/2893482725 learned_addr learned my addr 172.21.15.60:0/2893482725 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.641+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff20c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983038669 0 0) 0x7ff20c0b53b0 con 0x7ff20c0cdae0 2023-10-16T12:34:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f8003520 con 0x7ff20c0cdae0 2023-10-16T12:34:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3422061447 0 0) 0x7ff1f8003520 con 0x7ff20c0cdae0 2023-10-16T12:34:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 msgr2=0x7ff20c0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff20c0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 >> v1:172.21.15.144:6789/0 conn(0x7ff20c0d1f80 legacy=0x7ff20c0b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c160e40 con 0x7ff20c0cdae0 2023-10-16T12:34:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff1f4001670 con 0x7ff20c0cdae0 2023-10-16T12:34:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 >> v1:172.21.15.60:6789/0 conn(0x7ff20c0cdae0 legacy=0x7ff20c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff1f80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8003f70 0x7ff1f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1f8004d00 0x7ff1f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c1617e0 con 0x7ff1f8004d00 2023-10-16T12:34:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f800bb70 con 0x7ff1f8003f70 2023-10-16T12:34:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f800bd30 con 0x7ff20c0cf2b0 2023-10-16T12:34:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.642+0000 7ff20bfff640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1f8004d00 0x7ff1f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff1f80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20affd640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8003f70 0x7ff1f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20bfff640 1 -- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 msgr2=0x7ff1f80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20bfff640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff1f80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20bfff640 1 -- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8003f70 msgr2=0x7ff1f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:06.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20bfff640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8003f70 0x7ff1f80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20bfff640 1 -- 172.21.15.60:0/2893482725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f8003520 con 0x7ff1f8004d00 2023-10-16T12:34:06.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cf2b0 0x7ff1f80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:06.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.643+0000 7ff20affd640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8003f70 0x7ff1f80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:06.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.644+0000 7ff20bfff640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1f8004d00 0x7ff1f8005180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff204001720 tx=0x7ff1f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cefdc801b675c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:06.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.644+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040052d0 con 0x7ff1f8004d00 2023-10-16T12:34:06.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.644+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20402e050 con 0x7ff1f8004d00 2023-10-16T12:34:06.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.644+0000 7ff20a7fc640 1 -- 172.21.15.60:0/2893482725 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204003830 con 0x7ff1f8004d00 2023-10-16T12:34:06.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff210a0d640 1 -- 172.21.15.60:0/2893482725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1f8004d00 msgr2=0x7ff1f8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff210a0d640 1 --2- 172.21.15.60:0/2893482725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1f8004d00 0x7ff1f8005180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff204001720 tx=0x7ff1f80057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff20bfff640 1 -- 172.21.15.60:0/2893482725 reap_dead start 2023-10-16T12:34:06.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff210a0d640 1 -- 172.21.15.60:0/2893482725 shutdown_connections 2023-10-16T12:34:06.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff210a0d640 1 -- 172.21.15.60:0/2893482725 >> 172.21.15.60:0/2893482725 conn(0x7ff20c06bc60 msgr2=0x7ff20c06c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff210a0d640 1 -- 172.21.15.60:0/2893482725 shutdown_connections 2023-10-16T12:34:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.645+0000 7ff210a0d640 1 -- 172.21.15.60:0/2893482725 wait complete. 2023-10-16T12:34:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.647+0000 7ff210a0d640 1 Processor -- start 2023-10-16T12:34:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.647+0000 7ff210a0d640 1 -- start start 2023-10-16T12:34:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff210a0d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8004d00 0x7ff20c14e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff210a0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 0x7ff20c0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff210a0d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cc5a0 0x7ff20c0caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff210a0d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0cca20 con 0x7ff20c152370 2023-10-16T12:34:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff210a0d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c1510a0 con 0x7ff1f8004d00 2023-10-16T12:34:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff210a0d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c151ee0 con 0x7ff20c0cc5a0 2023-10-16T12:34:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff20affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 0x7ff20c0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff20affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 0x7ff20c0cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48448/0 (socket says 172.21.15.60:48448) 2023-10-16T12:34:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.648+0000 7ff20affd640 1 -- 172.21.15.60:0/763017888 learned_addr learned my addr 172.21.15.60:0/763017888 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8004d00 0x7ff20c14e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20affd640 1 -- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cc5a0 msgr2=0x7ff20c0caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20affd640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cc5a0 0x7ff20c0caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20affd640 1 -- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8004d00 msgr2=0x7ff20c14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20affd640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8004d00 0x7ff20c14e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20affd640 1 -- 172.21.15.60:0/763017888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c1512d0 con 0x7ff20c152370 2023-10-16T12:34:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8004d00 0x7ff20c14e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff20affd640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 0x7ff20c0cc060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400d850 tx=0x7ff1f400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40074e0 con 0x7ff20c152370 2023-10-16T12:34:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4007640 con 0x7ff20c152370 2023-10-16T12:34:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4004a00 con 0x7ff20c152370 2023-10-16T12:34:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c1515b0 con 0x7ff20c152370 2023-10-16T12:34:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.649+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c151b70 con 0x7ff20c152370 2023-10-16T12:34:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.651+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c15e2f0 con 0x7ff20c152370 2023-10-16T12:34:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.651+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff1f4004b60 con 0x7ff20c152370 2023-10-16T12:34:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.652+0000 7ff203fff640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff1ec06ae70 0x7ff1ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.652+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1f4089280 con 0x7ff20c152370 2023-10-16T12:34:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.653+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff1ec06ae70 0x7ff1ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.654+0000 7ff20b7fe640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff1ec06ae70 0x7ff1ec06d330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0cd600 tx=0x7ff1fc00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.657+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1f4054340 con 0x7ff20c152370 2023-10-16T12:34:06.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:06.915+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7ff20c04e9b0 con 0x7ff20c152370 2023-10-16T12:34:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.479+0000 7ff203fff640 1 -- 172.21.15.60:0/763017888 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff1f4057960 con 0x7ff20c152370 2023-10-16T12:34:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff1ec06ae70 msgr2=0x7ff1ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff1ec06ae70 0x7ff1ec06d330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0cd600 tx=0x7ff1fc00b040 comp rx=0 tx=0).stop 2023-10-16T12:34:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 msgr2=0x7ff20c0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 0x7ff20c0cc060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400d850 tx=0x7ff1f400dd10 comp rx=0 tx=0).stop 2023-10-16T12:34:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 shutdown_connections 2023-10-16T12:34:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff1ec06ae70 0x7ff1ec06d330 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20c0cc5a0 0x7ff20c0caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20c152370 0x7ff20c0cc060 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 --2- 172.21.15.60:0/763017888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1f8004d00 0x7ff20c14e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 >> 172.21.15.60:0/763017888 conn(0x7ff20c06bc60 msgr2=0x7ff20c0cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 shutdown_connections 2023-10-16T12:34:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.483+0000 7ff210a0d640 1 -- 172.21.15.60:0/763017888 wait complete. 2023-10-16T12:34:07.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:34:07.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.736+0000 7fe1e1cd1640 1 Processor -- start 2023-10-16T12:34:07.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.737+0000 7fe1e1cd1640 1 -- start start 2023-10-16T12:34:07.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.738+0000 7fe1e1cd1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe1dc06c040 0x7fe1dc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:07.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.738+0000 7fe1e1cd1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc15f4a0 con 0x7fe1dc0cda80 2023-10-16T12:34:07.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.738+0000 7fe1e1cd1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc06c960 con 0x7fe1dc0ad970 2023-10-16T12:34:07.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.738+0000 7fe1e1cd1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc15b9d0 con 0x7fe1dc06c040 2023-10-16T12:34:07.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.738+0000 7fe1e0ccf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe1dc06c040 0x7fe1dc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:07.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.739+0000 7fe1e14d0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe1dc0cda80 0x7fe1dc15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44176/0 (socket says 172.21.15.60:44176) 2023-10-16T12:34:07.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.739+0000 7fe1e14d0640 1 -- 172.21.15.60:0/1921604520 learned_addr learned my addr 172.21.15.60:0/1921604520 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:07.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.739+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/1921604520 >> v1:172.21.15.144:6789/0 conn(0x7fe1dc0ad970 legacy=0x7fe1dc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:07.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.739+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/1921604520 >> v1:172.21.15.60:6789/0 conn(0x7fe1dc0cda80 legacy=0x7fe1dc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.739+0000 7fe1db7fe640 1 -- 172.21.15.60:0/1921604520 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409969439 0 0) 0x7fe1dc15f4a0 con 0x7fe1dc0cda80 2023-10-16T12:34:07.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.739+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/1921604520 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc15bcb0 con 0x7fe1dc06c040 2023-10-16T12:34:07.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.740+0000 7fe1e0ccf640 1 --2- 172.21.15.60:0/1921604520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc06c420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00097a0 tx=0x7fe1d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56a72cfc98aa5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:07.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1db7fe640 1 -- 172.21.15.60:0/1921604520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0031830 con 0x7fe1dc06c040 2023-10-16T12:34:07.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1db7fe640 1 -- 172.21.15.60:0/1921604520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d0031990 con 0x7fe1dc06c040 2023-10-16T12:34:07.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1db7fe640 1 -- 172.21.15.60:0/1921604520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0031d20 con 0x7fe1dc06c040 2023-10-16T12:34:07.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/1921604520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 msgr2=0x7fe1dc06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/1921604520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc06c420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00097a0 tx=0x7fe1d002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/1921604520 shutdown_connections 2023-10-16T12:34:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/1921604520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc06c420 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/1921604520 >> 172.21.15.60:0/1921604520 conn(0x7fe1dc0ce6a0 msgr2=0x7fe1dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.741+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/1921604520 shutdown_connections 2023-10-16T12:34:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.742+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/1921604520 wait complete. 2023-10-16T12:34:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.743+0000 7fe1e1cd1640 1 Processor -- start 2023-10-16T12:34:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.743+0000 7fe1e1cd1640 1 -- start start 2023-10-16T12:34:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e1cd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc1ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e1cd1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1dc0ad970 0x7fe1dc0c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e1cd1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1dc0cda80 0x7fe1dc0aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e1cd1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0c5810 con 0x7fe1dc06c040 2023-10-16T12:34:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e1cd1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0c5af0 con 0x7fe1dc0ad970 2023-10-16T12:34:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e1cd1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0c5dd0 con 0x7fe1dc0cda80 2023-10-16T12:34:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.744+0000 7fe1e0ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc1ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:07.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc1ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48470/0 (socket says 172.21.15.60:48470) 2023-10-16T12:34:07.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/4101648566 learned_addr learned my addr 172.21.15.60:0/4101648566 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:07.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1dbfff640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1dc0ad970 0x7fe1dc0c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:07.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1dc0cda80 msgr2=0x7fe1dc0aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1dc0cda80 0x7fe1dc0aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1dc0ad970 msgr2=0x7fe1dc0c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1dc0ad970 0x7fe1dc0c42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.745+0000 7fe1e0ccf640 1 -- 172.21.15.60:0/4101648566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d0009400 con 0x7fe1dc06c040 2023-10-16T12:34:07.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.746+0000 7fe1e0ccf640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc1ef9b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0000c00 tx=0x7fe1d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:07.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.746+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0004230 con 0x7fe1dc06c040 2023-10-16T12:34:07.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.746+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d0004390 con 0x7fe1dc06c040 2023-10-16T12:34:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.746+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d003a910 con 0x7fe1dc06c040 2023-10-16T12:34:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.746+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc0c60b0 con 0x7fe1dc06c040 2023-10-16T12:34:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.746+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc0c7cd0 con 0x7fe1dc06c040 2023-10-16T12:34:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.747+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc06b330 con 0x7fe1dc06c040 2023-10-16T12:34:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.748+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe1d003aa70 con 0x7fe1dc06c040 2023-10-16T12:34:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.749+0000 7fe1d97fa640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe1c006ae70 0x7fe1c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.749+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1d00ab9e0 con 0x7fe1dc06c040 2023-10-16T12:34:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.750+0000 7fe1dbfff640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe1c006ae70 0x7fe1c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.751+0000 7fe1dbfff640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe1c006ae70 0x7fe1c006d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0c8460 tx=0x7fe1c8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.753+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1d00769f0 con 0x7fe1dc06c040 2023-10-16T12:34:07.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.948+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe1dc06b330 con 0x7fe1dc06c040 2023-10-16T12:34:07.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.954+0000 7fe1d97fa640 1 -- 172.21.15.60:0/4101648566 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe1d007a010 con 0x7fe1dc06c040 2023-10-16T12:34:07.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe1c006ae70 msgr2=0x7fe1c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe1c006ae70 0x7fe1c006d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0c8460 tx=0x7fe1c8009040 comp rx=0 tx=0).stop 2023-10-16T12:34:07.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 msgr2=0x7fe1dc1ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:07.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc1ef9b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0000c00 tx=0x7fe1d00046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 shutdown_connections 2023-10-16T12:34:07.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe1c006ae70 0x7fe1c006d330 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1dc0cda80 0x7fe1dc0aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1dc0ad970 0x7fe1dc0c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 --2- 172.21.15.60:0/4101648566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1dc06c040 0x7fe1dc1ef9b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 >> 172.21.15.60:0/4101648566 conn(0x7fe1dc0ce6a0 msgr2=0x7fe1dc06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 shutdown_connections 2023-10-16T12:34:07.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:07.957+0000 7fe1e1cd1640 1 -- 172.21.15.60:0/4101648566 wait complete. 2023-10-16T12:34:07.978 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:34:07.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:34:08.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:34:08.213 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:34:08.214 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:08.214 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:08.277 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:08.278 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:08.335 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:08.336 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:08.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:08.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.601+0000 7fcd088ca640 1 Processor -- start 2023-10-16T12:34:08.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.602+0000 7fcd088ca640 1 -- start start 2023-10-16T12:34:08.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd088ca640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcd040ad970 0x7fcd040cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd088ca640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0415b9d0 con 0x7fcd0406c040 2023-10-16T12:34:08.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd088ca640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0415bcb0 con 0x7fcd040ccfb0 2023-10-16T12:34:08.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd088ca640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0415bf90 con 0x7fcd040ad970 2023-10-16T12:34:08.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd037fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd0406c040 0x7fcd0406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44184/0 (socket says 172.21.15.60:44184) 2023-10-16T12:34:08.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd037fe640 1 -- 172.21.15.60:0/83741692 learned_addr learned my addr 172.21.15.60:0/83741692 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:08.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd02ffd640 1 --2- 172.21.15.60:0/83741692 >> v2:172.21.15.144:3300/0 conn(0x7fcd040ad970 0x7fcd040cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.604+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49862870 0 0) 0x7fcd0415b9d0 con 0x7fcd0406c040 2023-10-16T12:34:08.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf0003520 con 0x7fcd0406c040 2023-10-16T12:34:08.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349030968 0 0) 0x7fccf0003520 con 0x7fcd0406c040 2023-10-16T12:34:08.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 >> v2:172.21.15.144:3300/0 conn(0x7fcd040ad970 msgr2=0x7fcd040cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:08.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 --2- 172.21.15.60:0/83741692 >> v2:172.21.15.144:3300/0 conn(0x7fcd040ad970 0x7fcd040cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 >> v1:172.21.15.144:6789/0 conn(0x7fcd040ccfb0 legacy=0x7fcd040cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd040cc270 con 0x7fcd0406c040 2023-10-16T12:34:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fccf40019f0 con 0x7fcd0406c040 2023-10-16T12:34:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 >> v1:172.21.15.60:6789/0 conn(0x7fcd0406c040 legacy=0x7fcd0406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccf00039c0 0x7fccf0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf00047c0 0x7fccf0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccf0005180 0x7fccf000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0415bf90 con 0x7fccf0005180 2023-10-16T12:34:08.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf000c080 con 0x7fccf00047c0 2023-10-16T12:34:08.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf000c240 con 0x7fccf00039c0 2023-10-16T12:34:08.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.605+0000 7fcd03fff640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccf0005180 0x7fccf000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd02ffd640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf00047c0 0x7fccf0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd037fe640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccf00039c0 0x7fccf0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd03fff640 1 -- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccf00039c0 msgr2=0x7fccf0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd03fff640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccf00039c0 0x7fccf0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd03fff640 1 -- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf00047c0 msgr2=0x7fccf0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd03fff640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf00047c0 0x7fccf0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.606+0000 7fcd03fff640 1 -- 172.21.15.60:0/83741692 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf0003520 con 0x7fccf0005180 2023-10-16T12:34:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.607+0000 7fcd03fff640 1 -- 172.21.15.60:0/83741692 reap_dead start 2023-10-16T12:34:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.607+0000 7fcd03fff640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccf0005180 0x7fccf000b980 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fccfc001750 tx=0x7fccf00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e8d7e7f1cce7d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.607+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc0052d0 con 0x7fccf0005180 2023-10-16T12:34:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.607+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc02e050 con 0x7fccf0005180 2023-10-16T12:34:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.607+0000 7fcd027fc640 1 -- 172.21.15.60:0/83741692 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc0035f0 con 0x7fccf0005180 2023-10-16T12:34:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 -- 172.21.15.60:0/83741692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccf0005180 msgr2=0x7fccf000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccf0005180 0x7fccf000b980 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fccfc001750 tx=0x7fccf00057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 -- 172.21.15.60:0/83741692 shutdown_connections 2023-10-16T12:34:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 --2- 172.21.15.60:0/83741692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccf0005180 0x7fccf000b980 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 -- 172.21.15.60:0/83741692 >> 172.21.15.60:0/83741692 conn(0x7fcd040ce6a0 msgr2=0x7fcd040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:08.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 -- 172.21.15.60:0/83741692 shutdown_connections 2023-10-16T12:34:08.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.608+0000 7fcd088ca640 1 -- 172.21.15.60:0/83741692 wait complete. 2023-10-16T12:34:08.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.610+0000 7fcd088ca640 1 Processor -- start 2023-10-16T12:34:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.610+0000 7fcd088ca640 1 -- start start 2023-10-16T12:34:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd088ca640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf0005180 0x7fcd04156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd088ca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 0x7fcd0415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd088ca640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd0415c630 0x7fcd0415eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd088ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd04156cb0 con 0x7fcd0415c1d0 2023-10-16T12:34:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd088ca640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0415cbe0 con 0x7fccf0005180 2023-10-16T12:34:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd088ca640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0415cec0 con 0x7fcd0415c630 2023-10-16T12:34:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd02ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 0x7fcd0415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd02ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 0x7fcd0415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48488/0 (socket says 172.21.15.60:48488) 2023-10-16T12:34:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.611+0000 7fcd02ffd640 1 -- 172.21.15.60:0/1259161429 learned_addr learned my addr 172.21.15.60:0/1259161429 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd037fe640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf0005180 0x7fcd04156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd03fff640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd0415c630 0x7fcd0415eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd02ffd640 1 -- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd0415c630 msgr2=0x7fcd0415eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd02ffd640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd0415c630 0x7fcd0415eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd02ffd640 1 -- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf0005180 msgr2=0x7fcd04156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd02ffd640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf0005180 0x7fcd04156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd02ffd640 1 -- 172.21.15.60:0/1259161429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0415d1a0 con 0x7fcd0415c1d0 2023-10-16T12:34:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd03fff640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd0415c630 0x7fcd0415eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.612+0000 7fcd02ffd640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 0x7fcd0415e5c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fccec011560 tx=0x7fccec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.613+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec015930 con 0x7fcd0415c1d0 2023-10-16T12:34:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.613+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec015a90 con 0x7fcd0415c1d0 2023-10-16T12:34:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.613+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec015e40 con 0x7fcd0415c1d0 2023-10-16T12:34:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.613+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0415d480 con 0x7fcd0415c1d0 2023-10-16T12:34:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.613+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd041f6280 con 0x7fcd0415c1d0 2023-10-16T12:34:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.614+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0406b330 con 0x7fcd0415c1d0 2023-10-16T12:34:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.615+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fccec01d400 con 0x7fcd0415c1d0 2023-10-16T12:34:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.616+0000 7fccfbfff640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcce406aea0 0x7fcce406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.616+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccec089dc0 con 0x7fcd0415c1d0 2023-10-16T12:34:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.616+0000 7fcd037fe640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcce406aea0 0x7fcce406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.618+0000 7fcd037fe640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcce406aea0 0x7fcce406d360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fccf40013d0 tx=0x7fccf4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.620+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccec054e60 con 0x7fcd0415c1d0 2023-10-16T12:34:08.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.817+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0404e9b0 con 0x7fcd0415c1d0 2023-10-16T12:34:08.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.821+0000 7fccfbfff640 1 -- 172.21.15.60:0/1259161429 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fccec058480 con 0x7fcd0415c1d0 2023-10-16T12:34:08.824 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:08.824 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:33:25.574606+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcce406aea0 msgr2=0x7fcce406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcce406aea0 0x7fcce406d360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fccf40013d0 tx=0x7fccf4009040 comp rx=0 tx=0).stop 2023-10-16T12:34:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 msgr2=0x7fcd0415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 0x7fcd0415e5c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fccec011560 tx=0x7fccec009f90 comp rx=0 tx=0).stop 2023-10-16T12:34:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 shutdown_connections 2023-10-16T12:34:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcce406aea0 0x7fcce406d360 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd0415c630 0x7fcd0415eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0415c1d0 0x7fcd0415e5c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 --2- 172.21.15.60:0/1259161429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf0005180 0x7fcd04156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.823+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 >> 172.21.15.60:0/1259161429 conn(0x7fcd040ce6a0 msgr2=0x7fcd040ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:08.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.824+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 shutdown_connections 2023-10-16T12:34:08.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:08.824+0000 7fcd088ca640 1 -- 172.21.15.60:0/1259161429 wait complete. 2023-10-16T12:34:08.845 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:08.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:09.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.073+0000 7fb16d561640 1 Processor -- start 2023-10-16T12:34:09.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.074+0000 7fb16d561640 1 -- start start 2023-10-16T12:34:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb16d561640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb16815ad70 0x7fb168069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb16d561640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb1680c9ca0 0x7fb16806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb16d561640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680ca3b0 0x7fb1680ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb16d561640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680cad80 con 0x7fb16815ad70 2023-10-16T12:34:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb16d561640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680cb030 con 0x7fb1680c9ca0 2023-10-16T12:34:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb16d561640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680c91c0 con 0x7fb1680ca3b0 2023-10-16T12:34:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb166ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb16815ad70 0x7fb168069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb166ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb16815ad70 0x7fb168069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54218/0 (socket says 172.21.15.60:54218) 2023-10-16T12:34:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.075+0000 7fb166ffd640 1 -- 172.21.15.60:0/2960366179 learned_addr learned my addr 172.21.15.60:0/2960366179 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.076+0000 7fb166ffd640 1 -- 172.21.15.60:0/2960366179 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680ca3b0 msgr2=0x7fb1680ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.076+0000 7fb166ffd640 1 --2- 172.21.15.60:0/2960366179 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680ca3b0 0x7fb1680ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.076+0000 7fb166ffd640 1 -- 172.21.15.60:0/2960366179 >> v2:172.21.15.144:3300/0 conn(0x7fb1680c9ca0 msgr2=0x7fb16806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.076+0000 7fb166ffd640 1 --2- 172.21.15.60:0/2960366179 >> v2:172.21.15.144:3300/0 conn(0x7fb1680c9ca0 0x7fb16806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.076+0000 7fb166ffd640 1 -- 172.21.15.60:0/2960366179 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1680c94a0 con 0x7fb16815ad70 2023-10-16T12:34:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.077+0000 7fb166ffd640 1 --2- 172.21.15.60:0/2960366179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16815ad70 0x7fb168069da0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb1680ce870 tx=0x7fb15802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3354f22d4e7c019b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.077+0000 7fb165ffb640 1 -- 172.21.15.60:0/2960366179 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15802ca70 con 0x7fb16815ad70 2023-10-16T12:34:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.077+0000 7fb165ffb640 1 -- 172.21.15.60:0/2960366179 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15802cbd0 con 0x7fb16815ad70 2023-10-16T12:34:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.077+0000 7fb165ffb640 1 -- 172.21.15.60:0/2960366179 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1580343f0 con 0x7fb16815ad70 2023-10-16T12:34:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 -- 172.21.15.60:0/2960366179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16815ad70 msgr2=0x7fb168069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 --2- 172.21.15.60:0/2960366179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16815ad70 0x7fb168069da0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb1680ce870 tx=0x7fb15802bed0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 -- 172.21.15.60:0/2960366179 shutdown_connections 2023-10-16T12:34:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 --2- 172.21.15.60:0/2960366179 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1680ca3b0 0x7fb1680ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 --2- 172.21.15.60:0/2960366179 >> v2:172.21.15.144:3300/0 conn(0x7fb1680c9ca0 0x7fb16806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 --2- 172.21.15.60:0/2960366179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16815ad70 0x7fb168069da0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 -- 172.21.15.60:0/2960366179 >> 172.21.15.60:0/2960366179 conn(0x7fb1680ce9b0 msgr2=0x7fb1680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 -- 172.21.15.60:0/2960366179 shutdown_connections 2023-10-16T12:34:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.078+0000 7fb16d561640 1 -- 172.21.15.60:0/2960366179 wait complete. 2023-10-16T12:34:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.080+0000 7fb16d561640 1 Processor -- start 2023-10-16T12:34:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.080+0000 7fb16d561640 1 -- start start 2023-10-16T12:34:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb16d561640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680c9ca0 0x7fb1680aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb16d561640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 0x7fb1680c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb16d561640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16815ad70 0x7fb1680c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb16d561640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680c5cb0 con 0x7fb1680ca3b0 2023-10-16T12:34:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb16d561640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680a9bc0 con 0x7fb1680c9ca0 2023-10-16T12:34:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb16d561640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680a9320 con 0x7fb16815ad70 2023-10-16T12:34:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb1667fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 0x7fb1680c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb1667fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 0x7fb1680c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54226/0 (socket says 172.21.15.60:54226) 2023-10-16T12:34:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.081+0000 7fb1667fc640 1 -- 172.21.15.60:0/115555312 learned_addr learned my addr 172.21.15.60:0/115555312 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb166ffd640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680c9ca0 0x7fb1680aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1677fe640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16815ad70 0x7fb1680c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1667fc640 1 -- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16815ad70 msgr2=0x7fb1680c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1667fc640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16815ad70 0x7fb1680c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1667fc640 1 -- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680c9ca0 msgr2=0x7fb1680aa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1667fc640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680c9ca0 0x7fb1680aa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1667fc640 1 -- 172.21.15.60:0/115555312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158009b30 con 0x7fb1680ca3b0 2023-10-16T12:34:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb1667fc640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 0x7fb1680c8030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb15000b4a0 tx=0x7fb15000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150010610 con 0x7fb1680ca3b0 2023-10-16T12:34:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb150010770 con 0x7fb1680ca3b0 2023-10-16T12:34:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150010b20 con 0x7fb1680ca3b0 2023-10-16T12:34:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.082+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1680a9600 con 0x7fb1680ca3b0 2023-10-16T12:34:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.083+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1680a8e90 con 0x7fb1680ca3b0 2023-10-16T12:34:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.084+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1680c5df0 con 0x7fb1680ca3b0 2023-10-16T12:34:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.085+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb150010c80 con 0x7fb1680ca3b0 2023-10-16T12:34:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.086+0000 7fb15f7fe640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb14806ae70 0x7fb14806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.086+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb150089220 con 0x7fb1680ca3b0 2023-10-16T12:34:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.086+0000 7fb166ffd640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb14806ae70 0x7fb14806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.088+0000 7fb166ffd640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb14806ae70 0x7fb14806d330 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb158035000 tx=0x7fb158030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.091+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1680c5df0 con 0x7fb1680ca3b0 2023-10-16T12:34:09.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.284+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1680c5df0 con 0x7fb1680ca3b0 2023-10-16T12:34:09.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.285+0000 7fb15f7fe640 1 -- 172.21.15.60:0/115555312 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1680c5df0 con 0x7fb1680ca3b0 2023-10-16T12:34:09.287 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:09.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb14806ae70 msgr2=0x7fb14806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb14806ae70 0x7fb14806d330 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb158035000 tx=0x7fb158030000 comp rx=0 tx=0).stop 2023-10-16T12:34:09.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 msgr2=0x7fb1680c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 0x7fb1680c8030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb15000b4a0 tx=0x7fb15000b960 comp rx=0 tx=0).stop 2023-10-16T12:34:09.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 shutdown_connections 2023-10-16T12:34:09.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb14806ae70 0x7fb14806d330 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb16815ad70 0x7fb1680c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1680ca3b0 0x7fb1680c8030 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 --2- 172.21.15.60:0/115555312 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1680c9ca0 0x7fb1680aa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 >> 172.21.15.60:0/115555312 conn(0x7fb1680ce9b0 msgr2=0x7fb16815dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:09.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 shutdown_connections 2023-10-16T12:34:09.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.288+0000 7fb16d561640 1 -- 172.21.15.60:0/115555312 wait complete. 2023-10-16T12:34:09.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:09.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.537+0000 7feabed33640 1 Processor -- start 2023-10-16T12:34:09.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.537+0000 7feabed33640 1 -- start start 2023-10-16T12:34:09.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.538+0000 7feabed33640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feab80cfaf0 0x7feab80b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabed33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80d3980 con 0x7feab80cf710 2023-10-16T12:34:09.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabed33640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80ce630 con 0x7feab80b3420 2023-10-16T12:34:09.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabed33640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab80d3c60 con 0x7feab80cfaf0 2023-10-16T12:34:09.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabd2d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feab80cfaf0 0x7feab80b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabcacf640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feab80cf710 0x7feab80b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50414/0 (socket says 172.21.15.60:50414) 2023-10-16T12:34:09.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabcacf640 1 -- 172.21.15.60:0/2220985058 learned_addr learned my addr 172.21.15.60:0/2220985058 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabd2d0640 1 -- 172.21.15.60:0/2220985058 >> v1:172.21.15.144:6789/0 conn(0x7feab80b3420 legacy=0x7feab80b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feabd2d0640 1 -- 172.21.15.60:0/2220985058 >> v1:172.21.15.60:6789/0 conn(0x7feab80cf710 legacy=0x7feab80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.540+0000 7feabd2d0640 1 -- 172.21.15.60:0/2220985058 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80ce910 con 0x7feab80cfaf0 2023-10-16T12:34:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.539+0000 7feab77fe640 1 -- 172.21.15.60:0/2220985058 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335676645 0 0) 0x7feab80d3980 con 0x7feab80cf710 2023-10-16T12:34:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.541+0000 7feabd2d0640 1 --2- 172.21.15.60:0/2220985058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80cfaf0 0x7feab80b63d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7feab0009d90 tx=0x7feab002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7ea3d9a074fd68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.541+0000 7feab77fe640 1 -- 172.21.15.60:0/2220985058 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab002cbd0 con 0x7feab80cfaf0 2023-10-16T12:34:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.541+0000 7feab77fe640 1 -- 172.21.15.60:0/2220985058 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab002cd30 con 0x7feab80cfaf0 2023-10-16T12:34:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.541+0000 7feab77fe640 1 -- 172.21.15.60:0/2220985058 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab002cea0 con 0x7feab80cfaf0 2023-10-16T12:34:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 -- 172.21.15.60:0/2220985058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80cfaf0 msgr2=0x7feab80b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 --2- 172.21.15.60:0/2220985058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80cfaf0 0x7feab80b63d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7feab0009d90 tx=0x7feab002be20 comp rx=0 tx=0).stop 2023-10-16T12:34:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 -- 172.21.15.60:0/2220985058 shutdown_connections 2023-10-16T12:34:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 --2- 172.21.15.60:0/2220985058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80cfaf0 0x7feab80b63d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 -- 172.21.15.60:0/2220985058 >> 172.21.15.60:0/2220985058 conn(0x7feab80d2cc0 msgr2=0x7feab80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 -- 172.21.15.60:0/2220985058 shutdown_connections 2023-10-16T12:34:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.542+0000 7feabed33640 1 -- 172.21.15.60:0/2220985058 wait complete. 2023-10-16T12:34:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.544+0000 7feabed33640 1 Processor -- start 2023-10-16T12:34:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.544+0000 7feabed33640 1 -- start start 2023-10-16T12:34:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabed33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 0x7feab80c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabed33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feab80cf710 0x7feab80cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabed33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feab80cfaf0 0x7feab815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabed33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80cd070 con 0x7feab80b3420 2023-10-16T12:34:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabed33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab80c7860 con 0x7feab80cf710 2023-10-16T12:34:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabed33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab80ccdb0 con 0x7feab80cfaf0 2023-10-16T12:34:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 0x7feab80c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 0x7feab80c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54238/0 (socket says 172.21.15.60:54238) 2023-10-16T12:34:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 -- 172.21.15.60:0/1732616931 learned_addr learned my addr 172.21.15.60:0/1732616931 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feab7fff640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feab80cf710 0x7feab80cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 -- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feab80cfaf0 msgr2=0x7feab815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feab80cfaf0 0x7feab815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 -- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feab80cf710 msgr2=0x7feab80cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feab80cf710 0x7feab80cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feabcacf640 1 -- 172.21.15.60:0/1732616931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0009a70 con 0x7feab80b3420 2023-10-16T12:34:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.545+0000 7feab7fff640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feab80cf710 0x7feab80cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.546+0000 7feabcacf640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 0x7feab80c7320 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7feaac0010e0 tx=0x7feaac00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.546+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac007480 con 0x7feab80b3420 2023-10-16T12:34:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.546+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac0075e0 con 0x7feab80b3420 2023-10-16T12:34:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.546+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac0059c0 con 0x7feab80b3420 2023-10-16T12:34:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.546+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab80c7d90 con 0x7feab80b3420 2023-10-16T12:34:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.546+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab80c8350 con 0x7feab80b3420 2023-10-16T12:34:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.548+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab804e160 con 0x7feab80b3420 2023-10-16T12:34:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.549+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feaac005b20 con 0x7feab80b3420 2023-10-16T12:34:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.549+0000 7feab57fa640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fea9c06aea0 0x7fea9c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.550+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaac088b80 con 0x7feab80b3420 2023-10-16T12:34:09.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.550+0000 7feab7fff640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fea9c06aea0 0x7fea9c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:09.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.551+0000 7feab7fff640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fea9c06aea0 0x7fea9c06d360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feab8071890 tx=0x7feaa4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:09.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.559+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feaac053c30 con 0x7feab80b3420 2023-10-16T12:34:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.753+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feab808c490 con 0x7feab80b3420 2023-10-16T12:34:09.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.754+0000 7feab57fa640 1 -- 172.21.15.60:0/1732616931 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7feaac017020 con 0x7feab80b3420 2023-10-16T12:34:09.757 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:09.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.756+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fea9c06aea0 msgr2=0x7fea9c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.756+0000 7feabed33640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fea9c06aea0 0x7fea9c06d360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feab8071890 tx=0x7feaa4009040 comp rx=0 tx=0).stop 2023-10-16T12:34:09.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.756+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 msgr2=0x7feab80c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:09.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.756+0000 7feabed33640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 0x7feab80c7320 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7feaac0010e0 tx=0x7feaac00c9b0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 shutdown_connections 2023-10-16T12:34:09.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fea9c06aea0 0x7fea9c06d360 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feab80cfaf0 0x7feab815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feab80cf710 0x7feab80cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 --2- 172.21.15.60:0/1732616931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feab80b3420 0x7feab80c7320 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:09.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 >> 172.21.15.60:0/1732616931 conn(0x7feab80d2cc0 msgr2=0x7feab80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:09.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 shutdown_connections 2023-10-16T12:34:09.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:09.757+0000 7feabed33640 1 -- 172.21.15.60:0/1732616931 wait complete. 2023-10-16T12:34:09.780 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.011+0000 7f16b7fff640 1 Processor -- start 2023-10-16T12:34:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.011+0000 7f16b7fff640 1 -- start start 2023-10-16T12:34:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16b80ad970 0x7f16b815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b7fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806c960 con 0x7f16b815ebc0 2023-10-16T12:34:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b7fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b815b9d0 con 0x7f16b806c040 2023-10-16T12:34:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b7fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b815bcb0 con 0x7f16b80ad970 2023-10-16T12:34:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54254/0 (socket says 172.21.15.60:54254) 2023-10-16T12:34:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.013+0000 7f16b6ffd640 1 -- 172.21.15.60:0/2196238753 learned_addr learned my addr 172.21.15.60:0/2196238753 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.014+0000 7f16b6ffd640 1 -- 172.21.15.60:0/2196238753 >> v2:172.21.15.144:3300/0 conn(0x7f16b80ad970 msgr2=0x7f16b815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.014+0000 7f16b6ffd640 1 --2- 172.21.15.60:0/2196238753 >> v2:172.21.15.144:3300/0 conn(0x7f16b80ad970 0x7f16b815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.014+0000 7f16b6ffd640 1 -- 172.21.15.60:0/2196238753 >> v1:172.21.15.144:6789/0 conn(0x7f16b815ebc0 legacy=0x7f16b80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.014+0000 7f16b6ffd640 1 -- 172.21.15.60:0/2196238753 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b815bf90 con 0x7f16b806c040 2023-10-16T12:34:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.015+0000 7f16b6ffd640 1 --2- 172.21.15.60:0/2196238753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b806c420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f16b80cb060 tx=0x7f16a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be00dcbb5d58ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.015+0000 7f16b5ffb640 1 -- 172.21.15.60:0/2196238753 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8031870 con 0x7f16b806c040 2023-10-16T12:34:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.015+0000 7f16b5ffb640 1 -- 172.21.15.60:0/2196238753 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a80319d0 con 0x7f16b806c040 2023-10-16T12:34:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.015+0000 7f16b5ffb640 1 -- 172.21.15.60:0/2196238753 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8031d60 con 0x7f16b806c040 2023-10-16T12:34:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 -- 172.21.15.60:0/2196238753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 msgr2=0x7f16b806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 --2- 172.21.15.60:0/2196238753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b806c420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f16b80cb060 tx=0x7f16a802b910 comp rx=0 tx=0).stop 2023-10-16T12:34:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 -- 172.21.15.60:0/2196238753 shutdown_connections 2023-10-16T12:34:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 --2- 172.21.15.60:0/2196238753 >> v2:172.21.15.144:3300/0 conn(0x7f16b80ad970 0x7f16b815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 --2- 172.21.15.60:0/2196238753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 -- 172.21.15.60:0/2196238753 >> 172.21.15.60:0/2196238753 conn(0x7f16b80ce6a0 msgr2=0x7f16b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 -- 172.21.15.60:0/2196238753 shutdown_connections 2023-10-16T12:34:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.016+0000 7f16b7fff640 1 -- 172.21.15.60:0/2196238753 wait complete. 2023-10-16T12:34:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.018+0000 7f16b7fff640 1 Processor -- start 2023-10-16T12:34:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.018+0000 7f16b7fff640 1 -- start start 2023-10-16T12:34:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b80ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80ad970 0x7f16b80c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b815ebc0 0x7f16b81f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b7fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c93e0 con 0x7f16b806c040 2023-10-16T12:34:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b7fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b3570 con 0x7f16b815ebc0 2023-10-16T12:34:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b3850 con 0x7f16b80ad970 2023-10-16T12:34:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b80ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b80ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54256/0 (socket says 172.21.15.60:54256) 2023-10-16T12:34:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.019+0000 7f16b6ffd640 1 -- 172.21.15.60:0/142963042 learned_addr learned my addr 172.21.15.60:0/142963042 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b6ffd640 1 -- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80ad970 msgr2=0x7f16b80c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b67fc640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80ad970 0x7f16b80c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b77fe640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b815ebc0 0x7f16b81f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b6ffd640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80ad970 0x7f16b80c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b6ffd640 1 -- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b815ebc0 msgr2=0x7f16b81f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b6ffd640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b815ebc0 0x7f16b81f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b6ffd640 1 -- 172.21.15.60:0/142963042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a80094d0 con 0x7f16b806c040 2023-10-16T12:34:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b67fc640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80ad970 0x7f16b80c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b77fe640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b815ebc0 0x7f16b81f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b6ffd640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b80ae8c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f16a8000c00 tx=0x7f16a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8031840 con 0x7f16b806c040 2023-10-16T12:34:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a80443f0 con 0x7f16b806c040 2023-10-16T12:34:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b80c8d90 con 0x7f16b806c040 2023-10-16T12:34:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.020+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a80447a0 con 0x7f16b806c040 2023-10-16T12:34:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.021+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80c9db0 con 0x7f16b806c040 2023-10-16T12:34:10.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.021+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b806b330 con 0x7f16b806c040 2023-10-16T12:34:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.023+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f16a804d450 con 0x7f16b806c040 2023-10-16T12:34:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.024+0000 7f169ffff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16a406ae70 0x7f16a406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.024+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16a802d040 con 0x7f16b806c040 2023-10-16T12:34:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.024+0000 7f16b67fc640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16a406ae70 0x7f16a406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.026+0000 7f16b67fc640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16a406ae70 0x7f16a406d330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f16b81f5bf0 tx=0x7f16ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.029+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16a8076e00 con 0x7f16b806c040 2023-10-16T12:34:10.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.223+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16b804e9b0 con 0x7f16b806c040 2023-10-16T12:34:10.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.224+0000 7f169ffff640 1 -- 172.21.15.60:0/142963042 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f16a8035020 con 0x7f16b806c040 2023-10-16T12:34:10.227 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:10.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.226+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16a406ae70 msgr2=0x7f16a406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.226+0000 7f16b7fff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16a406ae70 0x7f16a406d330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f16b81f5bf0 tx=0x7f16ac008040 comp rx=0 tx=0).stop 2023-10-16T12:34:10.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.226+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 msgr2=0x7f16b80ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.226+0000 7f16b7fff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b80ae8c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f16a8000c00 tx=0x7f16a80046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 shutdown_connections 2023-10-16T12:34:10.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f16a406ae70 0x7f16a406d330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80ad970 0x7f16b80c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b815ebc0 0x7f16b81f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 --2- 172.21.15.60:0/142963042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b80ae8c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 >> 172.21.15.60:0/142963042 conn(0x7f16b80ce6a0 msgr2=0x7f16b80c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:10.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 shutdown_connections 2023-10-16T12:34:10.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.227+0000 7f16b7fff640 1 -- 172.21.15.60:0/142963042 wait complete. 2023-10-16T12:34:10.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:34:10.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.475+0000 7f65bd678640 1 Processor -- start 2023-10-16T12:34:10.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.476+0000 7f65bd678640 1 -- start start 2023-10-16T12:34:10.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.477+0000 7f65bd678640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65b8065ab0 0x7f65b80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.477+0000 7f65bd678640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65b80ce280 0x7f65b80ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.477+0000 7f65bd678640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b80acd70 con 0x7f65b806d220 2023-10-16T12:34:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.477+0000 7f65bd678640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b80ac4a0 con 0x7f65b8065ab0 2023-10-16T12:34:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.477+0000 7f65bd678640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b8065f40 con 0x7f65b80ce280 2023-10-16T12:34:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65b8065ab0 0x7f65b80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65b8065ab0 0x7f65b80c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54270/0 (socket says 172.21.15.60:54270) 2023-10-16T12:34:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 -- 172.21.15.60:0/839679935 learned_addr learned my addr 172.21.15.60:0/839679935 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:10.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 -- 172.21.15.60:0/839679935 >> v2:172.21.15.144:3300/0 conn(0x7f65b80ce280 msgr2=0x7f65b80ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:10.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 --2- 172.21.15.60:0/839679935 >> v2:172.21.15.144:3300/0 conn(0x7f65b80ce280 0x7f65b80ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 -- 172.21.15.60:0/839679935 >> v1:172.21.15.144:6789/0 conn(0x7f65b806d220 legacy=0x7f65b80cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:10.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.478+0000 7f65b77fe640 1 -- 172.21.15.60:0/839679935 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b80ac720 con 0x7f65b8065ab0 2023-10-16T12:34:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.479+0000 7f65b77fe640 1 --2- 172.21.15.60:0/839679935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8065ab0 0x7f65b80c95e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f65b8071fd0 tx=0x7f65a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1b0a7dc02fe1ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.480+0000 7f65b6ffd640 1 -- 172.21.15.60:0/839679935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a00317e0 con 0x7f65b8065ab0 2023-10-16T12:34:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.480+0000 7f65b6ffd640 1 -- 172.21.15.60:0/839679935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a0031940 con 0x7f65b8065ab0 2023-10-16T12:34:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.480+0000 7f65b6ffd640 1 -- 172.21.15.60:0/839679935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0031cd0 con 0x7f65b8065ab0 2023-10-16T12:34:10.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.480+0000 7f65bd678640 1 -- 172.21.15.60:0/839679935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8065ab0 msgr2=0x7f65b80c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.480+0000 7f65bd678640 1 --2- 172.21.15.60:0/839679935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8065ab0 0x7f65b80c95e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f65b8071fd0 tx=0x7f65a002b880 comp rx=0 tx=0).stop 2023-10-16T12:34:10.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.481+0000 7f65bd678640 1 -- 172.21.15.60:0/839679935 shutdown_connections 2023-10-16T12:34:10.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.481+0000 7f65bd678640 1 --2- 172.21.15.60:0/839679935 >> v2:172.21.15.144:3300/0 conn(0x7f65b80ce280 0x7f65b80ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.481+0000 7f65bd678640 1 --2- 172.21.15.60:0/839679935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8065ab0 0x7f65b80c95e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.481+0000 7f65bd678640 1 -- 172.21.15.60:0/839679935 >> 172.21.15.60:0/839679935 conn(0x7f65b80b0a80 msgr2=0x7f65b80b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:10.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.481+0000 7f65bd678640 1 -- 172.21.15.60:0/839679935 shutdown_connections 2023-10-16T12:34:10.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.481+0000 7f65bd678640 1 -- 172.21.15.60:0/839679935 wait complete. 2023-10-16T12:34:10.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.482+0000 7f65bd678640 1 Processor -- start 2023-10-16T12:34:10.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.483+0000 7f65bd678640 1 -- start start 2023-10-16T12:34:10.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bd678640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b806d220 0x7f65b81f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bd678640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 0x7f65b81593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bd678640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b806a890 0x7f65b8150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bd678640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b806cc40 con 0x7f65b80ce280 2023-10-16T12:34:10.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bd678640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b806b380 con 0x7f65b806d220 2023-10-16T12:34:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bd678640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b81f6890 con 0x7f65b806a890 2023-10-16T12:34:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 0x7f65b81593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 0x7f65b81593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54272/0 (socket says 172.21.15.60:54272) 2023-10-16T12:34:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 -- 172.21.15.60:0/901814735 learned_addr learned my addr 172.21.15.60:0/901814735 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b7fff640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b806d220 0x7f65b81f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bce77640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b806a890 0x7f65b8150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 -- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b806a890 msgr2=0x7f65b8150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b806a890 0x7f65b8150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 -- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b806d220 msgr2=0x7f65b81f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b806d220 0x7f65b81f6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b77fe640 1 -- 172.21.15.60:0/901814735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a00094d0 con 0x7f65b80ce280 2023-10-16T12:34:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65b7fff640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b806d220 0x7f65b81f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.484+0000 7f65bce77640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b806a890 0x7f65b8150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.485+0000 7f65b77fe640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 0x7f65b81593a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f65a0000c00 tx=0x7f65a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.485+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0032880 con 0x7f65b80ce280 2023-10-16T12:34:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.485+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a00329e0 con 0x7f65b80ce280 2023-10-16T12:34:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.485+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0032d90 con 0x7f65b80ce280 2023-10-16T12:34:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.485+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b806adc0 con 0x7f65b80ce280 2023-10-16T12:34:10.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.485+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b80b2da0 con 0x7f65b80ce280 2023-10-16T12:34:10.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.487+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b804e9b0 con 0x7f65b80ce280 2023-10-16T12:34:10.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.488+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f65a004d3f0 con 0x7f65b80ce280 2023-10-16T12:34:10.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.489+0000 7f65b4ff9640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f659806ae70 0x7f659806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:10.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.489+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65a00ab580 con 0x7f65b80ce280 2023-10-16T12:34:10.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.489+0000 7f65b7fff640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f659806ae70 0x7f659806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:10.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.491+0000 7f65b7fff640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f659806ae70 0x7f659806d330 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007fd0 tx=0x7f65a8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:10.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.494+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65a0076620 con 0x7f65b80ce280 2023-10-16T12:34:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.711+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65b806d000 con 0x7f65b80ce280 2023-10-16T12:34:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.712+0000 7f65b4ff9640 1 -- 172.21.15.60:0/901814735 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f65a0080030 con 0x7f65b80ce280 2023-10-16T12:34:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2023-10-16T12:34:10.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f659806ae70 msgr2=0x7f659806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f659806ae70 0x7f659806d330 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007fd0 tx=0x7f65a8009040 comp rx=0 tx=0).stop 2023-10-16T12:34:10.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 msgr2=0x7f65b81593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:10.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 0x7f65b81593a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f65a0000c00 tx=0x7f65a00046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 shutdown_connections 2023-10-16T12:34:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f659806ae70 0x7f659806d330 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b806a890 0x7f65b8150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b80ce280 0x7f65b81593a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 --2- 172.21.15.60:0/901814735 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b806d220 0x7f65b81f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 >> 172.21.15.60:0/901814735 conn(0x7f65b80b0a80 msgr2=0x7f65b80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:10.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.714+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 shutdown_connections 2023-10-16T12:34:10.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:10.715+0000 7f65bd678640 1 -- 172.21.15.60:0/901814735 wait complete. 2023-10-16T12:34:10.735 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:34:10.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:10.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-16T12:34:10.847 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:34:10.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:10.848 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:34:10.917 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:34:10.918 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:10.978 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:34:10.979 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:11.035 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:34:11.036 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:11.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:11.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.321+0000 7fc0df59e640 1 Processor -- start 2023-10-16T12:34:11.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.321+0000 7fc0df59e640 1 -- start start 2023-10-16T12:34:11.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.322+0000 7fc0df59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e0159950 0x7fc0e015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0df59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc0e0070450 0x7fc0e00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0df59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0e006c100 0x7fc0e006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0df59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e015e490 con 0x7fc0e006c100 2023-10-16T12:34:11.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0df59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e0070df0 con 0x7fc0e0070450 2023-10-16T12:34:11.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0df59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e015aef0 con 0x7fc0e0159950 2023-10-16T12:34:11.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0ded9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0e006c100 0x7fc0e006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0ded9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0e006c100 0x7fc0e006c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54274/0 (socket says 172.21.15.60:54274) 2023-10-16T12:34:11.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0ded9d640 1 -- 172.21.15.60:0/3756743962 learned_addr learned my addr 172.21.15.60:0/3756743962 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0ded9d640 1 -- 172.21.15.60:0/3756743962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e0159950 msgr2=0x7fc0e015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0d5d9b640 1 --2- 172.21.15.60:0/3756743962 >> v2:172.21.15.144:3300/0 conn(0x7fc0e0070450 0x7fc0e00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.323+0000 7fc0de59c640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e0159950 0x7fc0e015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.324+0000 7fc0ded9d640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e0159950 0x7fc0e015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.324+0000 7fc0ded9d640 1 -- 172.21.15.60:0/3756743962 >> v2:172.21.15.144:3300/0 conn(0x7fc0e0070450 msgr2=0x7fc0e00708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.324+0000 7fc0ded9d640 1 --2- 172.21.15.60:0/3756743962 >> v2:172.21.15.144:3300/0 conn(0x7fc0e0070450 0x7fc0e00708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.324+0000 7fc0ded9d640 1 -- 172.21.15.60:0/3756743962 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e015d910 con 0x7fc0e006c100 2023-10-16T12:34:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.324+0000 7fc0d5d9b640 1 --2- 172.21.15.60:0/3756743962 >> v2:172.21.15.144:3300/0 conn(0x7fc0e0070450 0x7fc0e00708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.324+0000 7fc0de59c640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e0159950 0x7fc0e015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.325+0000 7fc0ded9d640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e006c100 0x7fc0e006c570 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00719e0 tx=0x7fc0d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357400da4ff8c8ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.325+0000 7fc0ddd9b640 1 -- 172.21.15.60:0/3756743962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80317c0 con 0x7fc0e006c100 2023-10-16T12:34:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.325+0000 7fc0ddd9b640 1 -- 172.21.15.60:0/3756743962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d8031920 con 0x7fc0e006c100 2023-10-16T12:34:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.325+0000 7fc0ddd9b640 1 -- 172.21.15.60:0/3756743962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8031cb0 con 0x7fc0e006c100 2023-10-16T12:34:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 -- 172.21.15.60:0/3756743962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e006c100 msgr2=0x7fc0e006c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e006c100 0x7fc0e006c570 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00719e0 tx=0x7fc0d802b860 comp rx=0 tx=0).stop 2023-10-16T12:34:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 -- 172.21.15.60:0/3756743962 shutdown_connections 2023-10-16T12:34:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e006c100 0x7fc0e006c570 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 --2- 172.21.15.60:0/3756743962 >> v2:172.21.15.144:3300/0 conn(0x7fc0e0070450 0x7fc0e00708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 --2- 172.21.15.60:0/3756743962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e0159950 0x7fc0e015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 -- 172.21.15.60:0/3756743962 >> 172.21.15.60:0/3756743962 conn(0x7fc0e00ce6a0 msgr2=0x7fc0e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:11.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 -- 172.21.15.60:0/3756743962 shutdown_connections 2023-10-16T12:34:11.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.326+0000 7fc0df59e640 1 -- 172.21.15.60:0/3756743962 wait complete. 2023-10-16T12:34:11.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.328+0000 7fc0df59e640 1 Processor -- start 2023-10-16T12:34:11.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.328+0000 7fc0df59e640 1 -- start start 2023-10-16T12:34:11.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0df59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e006c100 0x7fc0e00b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0df59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 0x7fc0e0151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0df59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e0159950 0x7fc0e0157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0df59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0167540 con 0x7fc0e0070450 2023-10-16T12:34:11.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0df59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00b3630 con 0x7fc0e0159950 2023-10-16T12:34:11.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0df59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0166930 con 0x7fc0e006c100 2023-10-16T12:34:11.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0d5d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 0x7fc0e0151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0d5d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 0x7fc0e0151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54288/0 (socket says 172.21.15.60:54288) 2023-10-16T12:34:11.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0d5d9b640 1 -- 172.21.15.60:0/2227905463 learned_addr learned my addr 172.21.15.60:0/2227905463 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:11.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0d5d9b640 1 -- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e006c100 msgr2=0x7fc0e00b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0de59c640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e006c100 0x7fc0e00b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0ded9d640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e0159950 0x7fc0e0157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.329+0000 7fc0d5d9b640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e006c100 0x7fc0e00b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d5d9b640 1 -- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e0159950 msgr2=0x7fc0e0157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d5d9b640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e0159950 0x7fc0e0157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d5d9b640 1 -- 172.21.15.60:0/2227905463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80094d0 con 0x7fc0e0070450 2023-10-16T12:34:11.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0de59c640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e006c100 0x7fc0e00b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:11.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0ded9d640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e0159950 0x7fc0e0157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:11.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d5d9b640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 0x7fc0e0151e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800cf60 tx=0x7fc0c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:11.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c800d840 con 0x7fc0e0070450 2023-10-16T12:34:11.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c800d9a0 con 0x7fc0e0070450 2023-10-16T12:34:11.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c800dd30 con 0x7fc0e0070450 2023-10-16T12:34:11.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e0166c10 con 0x7fc0e0070450 2023-10-16T12:34:11.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.330+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e01671d0 con 0x7fc0e0070450 2023-10-16T12:34:11.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.332+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e004e9b0 con 0x7fc0e0070450 2023-10-16T12:34:11.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.333+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc0c8015440 con 0x7fc0e0070450 2023-10-16T12:34:11.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.334+0000 7fc0d77fe640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc0c006ae70 0x7fc0c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.334+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fc0c8089ff0 con 0x7fc0e0070450 2023-10-16T12:34:11.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.335+0000 7fc0de59c640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc0c006ae70 0x7fc0c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.336+0000 7fc0de59c640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc0c006ae70 0x7fc0c006d330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00088b0 tx=0x7fc0d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:11.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.339+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0c80bcfe0 con 0x7fc0e0070450 2023-10-16T12:34:11.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.539+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0e00b4620 con 0x7fc0e0070450 2023-10-16T12:34:11.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.540+0000 7fc0d77fe640 1 -- 172.21.15.60:0/2227905463 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16714 (secure 0 0 0) 0x7fc0c801a090 con 0x7fc0e0070450 2023-10-16T12:34:11.543 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:11.544 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:11.506118+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.543+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc0c006ae70 msgr2=0x7fc0c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.543+0000 7fc0df59e640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc0c006ae70 0x7fc0c006d330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00088b0 tx=0x7fc0d0008040 comp rx=0 tx=0).stop 2023-10-16T12:34:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.543+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 msgr2=0x7fc0e0151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.543+0000 7fc0df59e640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 0x7fc0e0151e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800cf60 tx=0x7fc0c800cf90 comp rx=0 tx=0).stop 2023-10-16T12:34:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 shutdown_connections 2023-10-16T12:34:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc0c006ae70 0x7fc0c006d330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e0159950 0x7fc0e0157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0e0070450 0x7fc0e0151e20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 --2- 172.21.15.60:0/2227905463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc0e006c100 0x7fc0e00b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 >> 172.21.15.60:0/2227905463 conn(0x7fc0e00ce6a0 msgr2=0x7fc0e00cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 shutdown_connections 2023-10-16T12:34:11.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.544+0000 7fc0df59e640 1 -- 172.21.15.60:0/2227905463 wait complete. 2023-10-16T12:34:11.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:11.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:11.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.770+0000 7fc46b409640 1 Processor -- start 2023-10-16T12:34:11.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.770+0000 7fc46b409640 1 -- start start 2023-10-16T12:34:11.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc46b409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46415af80 0x7fc4640cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc46b409640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc464065ab0 0x7fc464065f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc46b409640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc464066450 con 0x7fc46406faf0 2023-10-16T12:34:11.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc46b409640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4640d0930 con 0x7fc464065ab0 2023-10-16T12:34:11.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc46b409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46415eff0 con 0x7fc46415af80 2023-10-16T12:34:11.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc4699a6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc464065ab0 0x7fc464065f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.772+0000 7fc4691a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46415af80 0x7fc4640cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.773+0000 7fc4699a6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc464065ab0 0x7fc464065f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:56696/0 (socket says 172.21.15.60:56696) 2023-10-16T12:34:11.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.773+0000 7fc4699a6640 1 -- 172.21.15.60:0/638460038 learned_addr learned my addr 172.21.15.60:0/638460038 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:11.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.773+0000 7fc4691a5640 1 -- 172.21.15.60:0/638460038 >> v2:172.21.15.144:3300/0 conn(0x7fc464065ab0 msgr2=0x7fc464065f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:11.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.773+0000 7fc4691a5640 1 --2- 172.21.15.60:0/638460038 >> v2:172.21.15.144:3300/0 conn(0x7fc464065ab0 0x7fc464065f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.773+0000 7fc4691a5640 1 -- 172.21.15.60:0/638460038 >> v1:172.21.15.144:6789/0 conn(0x7fc46406faf0 legacy=0x7fc46415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.773+0000 7fc4691a5640 1 -- 172.21.15.60:0/638460038 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46415ecc0 con 0x7fc46415af80 2023-10-16T12:34:11.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.775+0000 7fc4691a5640 1 --2- 172.21.15.60:0/638460038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46415af80 0x7fc4640cb1c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc460009d90 tx=0x7fc46002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90c5b7d7067097e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:11.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc45bfff640 1 -- 172.21.15.60:0/638460038 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46002cbc0 con 0x7fc46415af80 2023-10-16T12:34:11.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc45bfff640 1 -- 172.21.15.60:0/638460038 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46002cd20 con 0x7fc46415af80 2023-10-16T12:34:11.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc45bfff640 1 -- 172.21.15.60:0/638460038 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460034660 con 0x7fc46415af80 2023-10-16T12:34:11.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 -- 172.21.15.60:0/638460038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46415af80 msgr2=0x7fc4640cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 --2- 172.21.15.60:0/638460038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46415af80 0x7fc4640cb1c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc460009d90 tx=0x7fc46002be10 comp rx=0 tx=0).stop 2023-10-16T12:34:11.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 -- 172.21.15.60:0/638460038 shutdown_connections 2023-10-16T12:34:11.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 --2- 172.21.15.60:0/638460038 >> v2:172.21.15.144:3300/0 conn(0x7fc464065ab0 0x7fc464065f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 --2- 172.21.15.60:0/638460038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc46415af80 0x7fc4640cb1c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 -- 172.21.15.60:0/638460038 >> 172.21.15.60:0/638460038 conn(0x7fc4640d0ab0 msgr2=0x7fc4640d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:11.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.776+0000 7fc46b409640 1 -- 172.21.15.60:0/638460038 shutdown_connections 2023-10-16T12:34:11.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.777+0000 7fc46b409640 1 -- 172.21.15.60:0/638460038 wait complete. 2023-10-16T12:34:11.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.778+0000 7fc46b409640 1 Processor -- start 2023-10-16T12:34:11.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.778+0000 7fc46b409640 1 -- start start 2023-10-16T12:34:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.779+0000 7fc46b409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc464065ab0 0x7fc4640ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.779+0000 7fc46b409640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46406faf0 0x7fc4640aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.779+0000 7fc46b409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 0x7fc46414cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.779+0000 7fc46b409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46406c300 con 0x7fc46415af80 2023-10-16T12:34:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.779+0000 7fc46b409640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640af2d0 con 0x7fc46406faf0 2023-10-16T12:34:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.779+0000 7fc46b409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46406b960 con 0x7fc464065ab0 2023-10-16T12:34:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 0x7fc46414cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 0x7fc46414cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54304/0 (socket says 172.21.15.60:54304) 2023-10-16T12:34:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 -- 172.21.15.60:0/2520562836 learned_addr learned my addr 172.21.15.60:0/2520562836 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4691a5640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc464065ab0 0x7fc4640ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4689a4640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46406faf0 0x7fc4640aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 -- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc464065ab0 msgr2=0x7fc4640ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc464065ab0 0x7fc4640ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 -- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46406faf0 msgr2=0x7fc4640aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46406faf0 0x7fc4640aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.780+0000 7fc4699a6640 1 -- 172.21.15.60:0/2520562836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460009a70 con 0x7fc46415af80 2023-10-16T12:34:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.781+0000 7fc4699a6640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 0x7fc46414cea0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00e8a0 tx=0x7fc45c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.781+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c009e80 con 0x7fc46415af80 2023-10-16T12:34:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.781+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c00c940 con 0x7fc46415af80 2023-10-16T12:34:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.781+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c00ccf0 con 0x7fc46415af80 2023-10-16T12:34:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.781+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46406bc40 con 0x7fc46415af80 2023-10-16T12:34:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.781+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46414d3e0 con 0x7fc46415af80 2023-10-16T12:34:11.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.782+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46406c440 con 0x7fc46415af80 2023-10-16T12:34:11.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.783+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc45c013470 con 0x7fc46415af80 2023-10-16T12:34:11.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.784+0000 7fc459ffb640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44806ae70 0x7fc44806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:11.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.785+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fc45c089790 con 0x7fc46415af80 2023-10-16T12:34:11.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.785+0000 7fc4691a5640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44806ae70 0x7fc44806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:11.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.786+0000 7fc4691a5640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44806ae70 0x7fc44806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc460036000 tx=0x7fc460030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:11.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.789+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc45c0547d0 con 0x7fc46415af80 2023-10-16T12:34:11.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.984+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc46404e160 con 0x7fc46415af80 2023-10-16T12:34:11.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.984+0000 7fc459ffb640 1 -- 172.21.15.60:0/2520562836 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc45c057df0 con 0x7fc46415af80 2023-10-16T12:34:11.987 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:11.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44806ae70 msgr2=0x7fc44806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44806ae70 0x7fc44806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc460036000 tx=0x7fc460030000 comp rx=0 tx=0).stop 2023-10-16T12:34:11.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 msgr2=0x7fc46414cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:11.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 0x7fc46414cea0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00e8a0 tx=0x7fc45c00ed60 comp rx=0 tx=0).stop 2023-10-16T12:34:11.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 shutdown_connections 2023-10-16T12:34:11.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc44806ae70 0x7fc44806d330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc46415af80 0x7fc46414cea0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc46406faf0 0x7fc4640aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 --2- 172.21.15.60:0/2520562836 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc464065ab0 0x7fc4640ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:11.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 >> 172.21.15.60:0/2520562836 conn(0x7fc4640d0ab0 msgr2=0x7fc46415b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:11.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 shutdown_connections 2023-10-16T12:34:11.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:11.987+0000 7fc46b409640 1 -- 172.21.15.60:0/2520562836 wait complete. 2023-10-16T12:34:12.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.216+0000 7fa26149f640 1 Processor -- start 2023-10-16T12:34:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.216+0000 7fa26149f640 1 -- start start 2023-10-16T12:34:12.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.218+0000 7fa26149f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 0x7fa25c0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.218+0000 7fa26149f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c15b9d0 con 0x7fa25c0ad970 2023-10-16T12:34:12.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.218+0000 7fa26149f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c15bcb0 con 0x7fa25c06c040 2023-10-16T12:34:12.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.218+0000 7fa26149f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c15b7d0 con 0x7fa25c15b320 2023-10-16T12:34:12.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.218+0000 7fa253fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa25c0ad970 0x7fa25c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50430/0 (socket says 172.21.15.60:50430) 2023-10-16T12:34:12.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.218+0000 7fa253fff640 1 -- 172.21.15.60:0/1110624976 learned_addr learned my addr 172.21.15.60:0/1110624976 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:12.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa260c9e640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 0x7fa25c0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593318077 0 0) 0x7fa25c15b9d0 con 0x7fa25c0ad970 2023-10-16T12:34:12.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa248003520 con 0x7fa25c0ad970 2023-10-16T12:34:12.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689348806 0 0) 0x7fa248003520 con 0x7fa25c0ad970 2023-10-16T12:34:12.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 msgr2=0x7fa25c0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 0x7fa25c0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 >> v1:172.21.15.144:6789/0 conn(0x7fa25c06c040 legacy=0x7fa25c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c15bf90 con 0x7fa25c0ad970 2023-10-16T12:34:12.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa244001670 con 0x7fa25c0ad970 2023-10-16T12:34:12.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 >> v1:172.21.15.60:6789/0 conn(0x7fa25c0ad970 legacy=0x7fa25c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480039a0 0x7fa248003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2480047c0 0x7fa248004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 0x7fa248005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c15b7d0 con 0x7fa2480047c0 2023-10-16T12:34:12.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24800ba60 con 0x7fa2480039a0 2023-10-16T12:34:12.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.219+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24800bc20 con 0x7fa25c15b320 2023-10-16T12:34:12.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa253fff640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2480047c0 0x7fa248004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa25bfff640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480039a0 0x7fa248003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa260c9e640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 0x7fa248005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa253fff640 1 -- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 msgr2=0x7fa248005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa253fff640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25c15b320 0x7fa248005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa253fff640 1 -- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480039a0 msgr2=0x7fa248003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa253fff640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480039a0 0x7fa248003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.220+0000 7fa253fff640 1 -- 172.21.15.60:0/1110624976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248003520 con 0x7fa2480047c0 2023-10-16T12:34:12.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.221+0000 7fa253fff640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2480047c0 0x7fa248004c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0ade20 tx=0x7fa2480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b105141ac881ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:12.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244005ad0 con 0x7fa2480047c0 2023-10-16T12:34:12.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244005c50 con 0x7fa2480047c0 2023-10-16T12:34:12.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa25b7fe640 1 -- 172.21.15.60:0/1110624976 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24402c850 con 0x7fa2480047c0 2023-10-16T12:34:12.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa26149f640 1 -- 172.21.15.60:0/1110624976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2480047c0 msgr2=0x7fa248004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa26149f640 1 --2- 172.21.15.60:0/1110624976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2480047c0 0x7fa248004c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0ade20 tx=0x7fa2480057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa260c9e640 1 -- 172.21.15.60:0/1110624976 reap_dead start 2023-10-16T12:34:12.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa26149f640 1 -- 172.21.15.60:0/1110624976 shutdown_connections 2023-10-16T12:34:12.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.222+0000 7fa26149f640 1 -- 172.21.15.60:0/1110624976 >> 172.21.15.60:0/1110624976 conn(0x7fa25c0ce6a0 msgr2=0x7fa25c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:12.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.223+0000 7fa26149f640 1 -- 172.21.15.60:0/1110624976 shutdown_connections 2023-10-16T12:34:12.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.223+0000 7fa26149f640 1 -- 172.21.15.60:0/1110624976 wait complete. 2023-10-16T12:34:12.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.224+0000 7fa26149f640 1 Processor -- start 2023-10-16T12:34:12.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.224+0000 7fa26149f640 1 -- start start 2023-10-16T12:34:12.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.225+0000 7fa26149f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480047c0 0x7fa25c0c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.225+0000 7fa26149f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 0x7fa25c0b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.225+0000 7fa26149f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25c0b2070 0x7fa25c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.225+0000 7fa26149f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0c7280 con 0x7fa25c0c4e00 2023-10-16T12:34:12.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.225+0000 7fa26149f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0c7500 con 0x7fa25c0b2070 2023-10-16T12:34:12.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.225+0000 7fa26149f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0c77e0 con 0x7fa2480047c0 2023-10-16T12:34:12.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 0x7fa25c0b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 0x7fa25c0b1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54326/0 (socket says 172.21.15.60:54326) 2023-10-16T12:34:12.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 -- 172.21.15.60:0/2828494533 learned_addr learned my addr 172.21.15.60:0/2828494533 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:12.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa260c9e640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25c0b2070 0x7fa25c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 -- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480047c0 msgr2=0x7fa25c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480047c0 0x7fa25c0c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 -- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25c0b2070 msgr2=0x7fa25c155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25c0b2070 0x7fa25c155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 -- 172.21.15.60:0/2828494533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0b1330 con 0x7fa25c0c4e00 2023-10-16T12:34:12.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.226+0000 7fa253fff640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 0x7fa25c0b1b30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa24400c300 tx=0x7fa244001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:12.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.227+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24402cc60 con 0x7fa25c0c4e00 2023-10-16T12:34:12.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.227+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24402cdc0 con 0x7fa25c0c4e00 2023-10-16T12:34:12.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.227+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244007290 con 0x7fa25c0c4e00 2023-10-16T12:34:12.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.227+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c0b1610 con 0x7fa25c0c4e00 2023-10-16T12:34:12.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.227+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c0b3110 con 0x7fa25c0c4e00 2023-10-16T12:34:12.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.228+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c06b330 con 0x7fa25c0c4e00 2023-10-16T12:34:12.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.229+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa244032070 con 0x7fa25c0c4e00 2023-10-16T12:34:12.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.230+0000 7fa2597fa640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa23c06ae70 0x7fa23c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.230+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fa24403c040 con 0x7fa25c0c4e00 2023-10-16T12:34:12.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.231+0000 7fa25bfff640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa23c06ae70 0x7fa23c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.232+0000 7fa25bfff640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa23c06ae70 0x7fa23c06d330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa24c007fd0 tx=0x7fa24c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:12.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.234+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa244075bc0 con 0x7fa25c0c4e00 2023-10-16T12:34:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.420+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa25c0b17a0 con 0x7fa25c0c4e00 2023-10-16T12:34:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.421+0000 7fa2597fa640 1 -- 172.21.15.60:0/2828494533 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa2440450a0 con 0x7fa25c0c4e00 2023-10-16T12:34:12.424 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa23c06ae70 msgr2=0x7fa23c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa23c06ae70 0x7fa23c06d330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa24c007fd0 tx=0x7fa24c009040 comp rx=0 tx=0).stop 2023-10-16T12:34:12.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 msgr2=0x7fa25c0b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 0x7fa25c0b1b30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa24400c300 tx=0x7fa244001a00 comp rx=0 tx=0).stop 2023-10-16T12:34:12.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 shutdown_connections 2023-10-16T12:34:12.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa23c06ae70 0x7fa23c06d330 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25c0b2070 0x7fa25c155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa25c0c4e00 0x7fa25c0b1b30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 --2- 172.21.15.60:0/2828494533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480047c0 0x7fa25c0c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 >> 172.21.15.60:0/2828494533 conn(0x7fa25c0ce6a0 msgr2=0x7fa25c15d550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 shutdown_connections 2023-10-16T12:34:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.424+0000 7fa26149f640 1 -- 172.21.15.60:0/2828494533 wait complete. 2023-10-16T12:34:12.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:12.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.666+0000 7f9c154aa640 1 Processor -- start 2023-10-16T12:34:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.667+0000 7f9c154aa640 1 -- start start 2023-10-16T12:34:12.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c154aa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 0x7f9c100d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c154aa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c100cded0 0x7f9c100b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c154aa640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1015d150 con 0x7f9c100b8480 2023-10-16T12:34:12.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c154aa640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c100d1180 con 0x7f9c100cded0 2023-10-16T12:34:12.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c154aa640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100d1460 con 0x7f9c1015cd70 2023-10-16T12:34:12.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c0f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9c100b8480 0x7f9c100cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50438/0 (socket says 172.21.15.60:50438) 2023-10-16T12:34:12.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.668+0000 7f9c0f7fe640 1 -- 172.21.15.60:0/4157485892 learned_addr learned my addr 172.21.15.60:0/4157485892 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:12.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 0x7f9c100d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c14ca9640 1 --2- 172.21.15.60:0/4157485892 >> v2:172.21.15.144:3300/0 conn(0x7f9c100cded0 0x7f9c100b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86823592 0 0) 0x7f9c1015d150 con 0x7f9c100b8480 2023-10-16T12:34:12.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf0003520 con 0x7f9c100b8480 2023-10-16T12:34:12.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4056073129 0 0) 0x7f9bf0003520 con 0x7f9c100b8480 2023-10-16T12:34:12.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 msgr2=0x7f9c100d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 0x7f9c100d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 >> v2:172.21.15.144:3300/0 conn(0x7f9c100cded0 msgr2=0x7f9c100b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 --2- 172.21.15.60:0/4157485892 >> v2:172.21.15.144:3300/0 conn(0x7f9c100cded0 0x7f9c100b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100d1740 con 0x7f9c100b8480 2023-10-16T12:34:12.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9bf80019f0 con 0x7f9c100b8480 2023-10-16T12:34:12.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 >> v1:172.21.15.60:6789/0 conn(0x7f9c100b8480 legacy=0x7f9c100cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf0003970 0x7f9bf0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 0x7f9bf00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0004d00 0x7f9bf0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100d1180 con 0x7f9bf0003970 2023-10-16T12:34:12.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100d1460 con 0x7f9bf0004d00 2023-10-16T12:34:12.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf000ba00 con 0x7f9c1015cd70 2023-10-16T12:34:12.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.669+0000 7f9c14ca9640 1 --2- 172.21.15.60:0/4157485892 >> v2:172.21.15.144:3300/0 conn(0x7f9c100cded0 0x7f9c100b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:12.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.670+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf0003970 0x7f9bf0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.670+0000 7f9c0ffff640 1 -- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 msgr2=0x7f9bf00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.670+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c1015cd70 0x7f9bf00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.670+0000 7f9c0ffff640 1 -- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0004d00 msgr2=0x7f9bf0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:12.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.670+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0004d00 0x7f9bf0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.670+0000 7f9c0ffff640 1 -- 172.21.15.60:0/4157485892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf0003520 con 0x7f9bf0003970 2023-10-16T12:34:12.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.671+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf0003970 0x7f9bf0003d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009aa0 tx=0x7f9bf00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bb4a389f1eb5c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:12.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0002c710 con 0x7f9bf0003970 2023-10-16T12:34:12.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0002c870 con 0x7f9bf0003970 2023-10-16T12:34:12.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c0effd640 1 -- 172.21.15.60:0/4157485892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0002cc20 con 0x7f9bf0003970 2023-10-16T12:34:12.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c154aa640 1 -- 172.21.15.60:0/4157485892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf0003970 msgr2=0x7f9bf0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c154aa640 1 --2- 172.21.15.60:0/4157485892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf0003970 0x7f9bf0003d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009aa0 tx=0x7f9bf00057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c14ca9640 1 -- 172.21.15.60:0/4157485892 reap_dead start 2023-10-16T12:34:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c154aa640 1 -- 172.21.15.60:0/4157485892 shutdown_connections 2023-10-16T12:34:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c154aa640 1 -- 172.21.15.60:0/4157485892 >> 172.21.15.60:0/4157485892 conn(0x7f9c100cf570 msgr2=0x7f9c100cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c154aa640 1 -- 172.21.15.60:0/4157485892 shutdown_connections 2023-10-16T12:34:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.672+0000 7f9c154aa640 1 -- 172.21.15.60:0/4157485892 wait complete. 2023-10-16T12:34:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.674+0000 7f9c154aa640 1 Processor -- start 2023-10-16T12:34:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.674+0000 7f9c154aa640 1 -- start start 2023-10-16T12:34:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.675+0000 7f9c154aa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0003970 0x7f9c101f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.675+0000 7f9c154aa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c100c9bc0 0x7f9c100b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.675+0000 7f9c154aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 0x7f9c100b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.675+0000 7f9c154aa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100cad90 con 0x7f9c100a94f0 2023-10-16T12:34:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.675+0000 7f9c154aa640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100ca110 con 0x7f9bf0003970 2023-10-16T12:34:12.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.675+0000 7f9c154aa640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100ca3f0 con 0x7f9c100c9bc0 2023-10-16T12:34:12.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 0x7f9c100b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 0x7f9c100b3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54352/0 (socket says 172.21.15.60:54352) 2023-10-16T12:34:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 -- 172.21.15.60:0/3557795989 learned_addr learned my addr 172.21.15.60:0/3557795989 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0003970 0x7f9c101f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 -- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c100c9bc0 msgr2=0x7f9c100b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c100c9bc0 0x7f9c100b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 -- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0003970 msgr2=0x7f9c101f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0003970 0x7f9c101f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 -- 172.21.15.60:0/3557795989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100ca6d0 con 0x7f9c100a94f0 2023-10-16T12:34:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0003970 0x7f9c101f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c14ca9640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 0x7f9c100b3f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9c08006fd0 tx=0x7f9c0800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:12.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.676+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0800c070 con 0x7f9c100a94f0 2023-10-16T12:34:12.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.677+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08004230 con 0x7f9c100a94f0 2023-10-16T12:34:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.677+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08003610 con 0x7f9c100a94f0 2023-10-16T12:34:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.677+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c100ca9b0 con 0x7f9c100a94f0 2023-10-16T12:34:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.677+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c101f5980 con 0x7f9c100a94f0 2023-10-16T12:34:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.678+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c100b2360 con 0x7f9c100a94f0 2023-10-16T12:34:12.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.679+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9c08003770 con 0x7f9c100a94f0 2023-10-16T12:34:12.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.680+0000 7f9c0cff9640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9bf406ae70 0x7f9bf406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:12.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.680+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f9c08088b40 con 0x7f9c100a94f0 2023-10-16T12:34:12.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.680+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9bf406ae70 0x7f9bf406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:12.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.682+0000 7f9c0ffff640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9bf406ae70 0x7f9bf406d330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9c00036000 tx=0x7f9c00030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:12.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.684+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c08053b90 con 0x7f9c100a94f0 2023-10-16T12:34:12.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.880+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c1004e9b0 con 0x7f9c100a94f0 2023-10-16T12:34:12.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.881+0000 7f9c0cff9640 1 -- 172.21.15.60:0/3557795989 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c080571b0 con 0x7f9c100a94f0 2023-10-16T12:34:12.883 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9bf406ae70 msgr2=0x7f9bf406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9bf406ae70 0x7f9bf406d330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9c00036000 tx=0x7f9c00030000 comp rx=0 tx=0).stop 2023-10-16T12:34:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 msgr2=0x7f9c100b3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 0x7f9c100b3f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9c08006fd0 tx=0x7f9c0800e830 comp rx=0 tx=0).stop 2023-10-16T12:34:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 shutdown_connections 2023-10-16T12:34:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9bf406ae70 0x7f9bf406d330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c100c9bc0 0x7f9c100b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c100a94f0 0x7f9c100b3f40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.883+0000 7f9c154aa640 1 --2- 172.21.15.60:0/3557795989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf0003970 0x7f9c101f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:12.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.884+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 >> 172.21.15.60:0/3557795989 conn(0x7f9c100cf570 msgr2=0x7f9c100b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:12.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.884+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 shutdown_connections 2023-10-16T12:34:12.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:12.884+0000 7f9c154aa640 1 -- 172.21.15.60:0/3557795989 wait complete. 2023-10-16T12:34:12.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:34:13.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.122+0000 7f7c31dce640 1 Processor -- start 2023-10-16T12:34:13.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.122+0000 7f7c31dce640 1 -- start start 2023-10-16T12:34:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c31dce640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c2c06a4b0 0x7f7c2c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c31dce640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c0cb030 con 0x7f7c2c15ad70 2023-10-16T12:34:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c31dce640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c0c91c0 con 0x7f7c2c0c9ca0 2023-10-16T12:34:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c31dce640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c94a0 con 0x7f7c2c06a4b0 2023-10-16T12:34:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c2b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7c2c15ad70 0x7f7c2c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50450/0 (socket says 172.21.15.60:50450) 2023-10-16T12:34:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c2b7fe640 1 -- 172.21.15.60:0/3654397533 learned_addr learned my addr 172.21.15.60:0/3654397533 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.124+0000 7f7c23fff640 1 --2- 172.21.15.60:0/3654397533 >> v2:172.21.15.144:3300/0 conn(0x7f7c2c06a4b0 0x7f7c2c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456040399 0 0) 0x7f7c2c0cb030 con 0x7f7c2c15ad70 2023-10-16T12:34:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c18003520 con 0x7f7c2c15ad70 2023-10-16T12:34:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959704564 0 0) 0x7f7c18003520 con 0x7f7c2c15ad70 2023-10-16T12:34:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 >> v2:172.21.15.144:3300/0 conn(0x7f7c2c06a4b0 msgr2=0x7f7c2c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 --2- 172.21.15.60:0/3654397533 >> v2:172.21.15.144:3300/0 conn(0x7f7c2c06a4b0 0x7f7c2c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 >> v1:172.21.15.144:6789/0 conn(0x7f7c2c0c9ca0 legacy=0x7f7c2c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0c9780 con 0x7f7c2c15ad70 2023-10-16T12:34:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7c1c001670 con 0x7f7c2c15ad70 2023-10-16T12:34:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 >> v1:172.21.15.60:6789/0 conn(0x7f7c2c15ad70 legacy=0x7f7c2c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c180039c0 0x7f7c18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c180047c0 0x7f7c18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c94a0 con 0x7f7c18005180 2023-10-16T12:34:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1800c080 con 0x7f7c180047c0 2023-10-16T12:34:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.125+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1800c240 con 0x7f7c180039c0 2023-10-16T12:34:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2bfff640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c23fff640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c180047c0 0x7f7c18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2bfff640 1 -- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c180039c0 msgr2=0x7f7c18003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2b7fe640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c180039c0 0x7f7c18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2bfff640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c180039c0 0x7f7c18003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2bfff640 1 -- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c180047c0 msgr2=0x7f7c18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2bfff640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c180047c0 0x7f7c18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c2bfff640 1 -- 172.21.15.60:0/3654397533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18003520 con 0x7f7c18005180 2023-10-16T12:34:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.126+0000 7f7c23fff640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c180047c0 0x7f7c18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.127+0000 7f7c2bfff640 1 -- 172.21.15.60:0/3654397533 reap_dead start 2023-10-16T12:34:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.127+0000 7f7c2bfff640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c1800b980 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7c24001e50 tx=0x7f7c180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cdeeae24accef2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24003b70 con 0x7f7c18005180 2023-10-16T12:34:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c24003d10 con 0x7f7c18005180 2023-10-16T12:34:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c2affd640 1 -- 172.21.15.60:0/3654397533 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24005310 con 0x7f7c18005180 2023-10-16T12:34:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 -- 172.21.15.60:0/3654397533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 msgr2=0x7f7c1800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c1800b980 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7c24001e50 tx=0x7f7c180057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 -- 172.21.15.60:0/3654397533 shutdown_connections 2023-10-16T12:34:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 --2- 172.21.15.60:0/3654397533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c1800b980 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 -- 172.21.15.60:0/3654397533 >> 172.21.15.60:0/3654397533 conn(0x7f7c2c0ce9b0 msgr2=0x7f7c2c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 -- 172.21.15.60:0/3654397533 shutdown_connections 2023-10-16T12:34:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.128+0000 7f7c31dce640 1 -- 172.21.15.60:0/3654397533 wait complete. 2023-10-16T12:34:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.130+0000 7f7c31dce640 1 Processor -- start 2023-10-16T12:34:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.130+0000 7f7c31dce640 1 -- start start 2023-10-16T12:34:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.131+0000 7f7c31dce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c2c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.131+0000 7f7c31dce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c2c0ad680 0x7f7c2c0c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.131+0000 7f7c31dce640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2c0c69d0 0x7f7c2c0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.131+0000 7f7c31dce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c30b0 con 0x7f7c18005180 2023-10-16T12:34:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.131+0000 7f7c31dce640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c24a0 con 0x7f7c2c0c69d0 2023-10-16T12:34:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.131+0000 7f7c31dce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c2780 con 0x7f7c2c0ad680 2023-10-16T12:34:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c2c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c2c151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54366/0 (socket says 172.21.15.60:54366) 2023-10-16T12:34:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 -- 172.21.15.60:0/2805380290 learned_addr learned my addr 172.21.15.60:0/2805380290 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2bfff640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2c0c69d0 0x7f7c2c0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 -- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c2c0ad680 msgr2=0x7f7c2c0c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c2c0ad680 0x7f7c2c0c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 -- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2c0c69d0 msgr2=0x7f7c2c0c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2c0c69d0 0x7f7c2c0c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 -- 172.21.15.60:0/2805380290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0c2a60 con 0x7f7c18005180 2023-10-16T12:34:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.132+0000 7f7c2b7fe640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c2c151d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00d770 tx=0x7f7c1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.133+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0072d0 con 0x7f7c18005180 2023-10-16T12:34:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.133+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c007430 con 0x7f7c18005180 2023-10-16T12:34:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.133+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0047e0 con 0x7f7c18005180 2023-10-16T12:34:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.133+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c0c2d40 con 0x7f7c18005180 2023-10-16T12:34:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.133+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c152290 con 0x7f7c18005180 2023-10-16T12:34:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.134+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c04e160 con 0x7f7c18005180 2023-10-16T12:34:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.135+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7c1c004940 con 0x7f7c18005180 2023-10-16T12:34:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.136+0000 7f7c28ff9640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7c0c06ae70 0x7f7c0c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.136+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f7c1c088e00 con 0x7f7c18005180 2023-10-16T12:34:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.137+0000 7f7c23fff640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7c0c06ae70 0x7f7c0c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.138+0000 7f7c23fff640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7c0c06ae70 0x7f7c0c06d330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0cdea0 tx=0x7f7c14008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:13.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.140+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c1c053e60 con 0x7f7c18005180 2023-10-16T12:34:13.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.350+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7c2c0cef30 con 0x7f7c18005180 2023-10-16T12:34:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.351+0000 7f7c28ff9640 1 -- 172.21.15.60:0/2805380290 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7c1c019020 con 0x7f7c18005180 2023-10-16T12:34:13.354 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2023-10-16T12:34:13.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7c0c06ae70 msgr2=0x7f7c0c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7c0c06ae70 0x7f7c0c06d330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0cdea0 tx=0x7f7c14008040 comp rx=0 tx=0).stop 2023-10-16T12:34:13.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 msgr2=0x7f7c2c151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c2c151d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00d770 tx=0x7f7c1c00dc30 comp rx=0 tx=0).stop 2023-10-16T12:34:13.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 shutdown_connections 2023-10-16T12:34:13.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7c0c06ae70 0x7f7c0c06d330 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c2c0ad680 0x7f7c2c0c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2c0c69d0 0x7f7c2c0c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 --2- 172.21.15.60:0/2805380290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c18005180 0x7f7c2c151d50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 >> 172.21.15.60:0/2805380290 conn(0x7f7c2c0ce9b0 msgr2=0x7f7c2c0b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:13.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 shutdown_connections 2023-10-16T12:34:13.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.354+0000 7f7c31dce640 1 -- 172.21.15.60:0/2805380290 wait complete. 2023-10-16T12:34:13.374 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-16T12:34:13.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:13.375 DEBUG:teuthology.orchestra.run.smithi144:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-16T12:34:13.487 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:34:13.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:13.488 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-16T12:34:13.545 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:13.546 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:13.611 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:13.611 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:13.669 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:13.670 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:13.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:13.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.953+0000 7ff7db390640 1 Processor -- start 2023-10-16T12:34:13.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.954+0000 7ff7db390640 1 -- start start 2023-10-16T12:34:13.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.955+0000 7ff7db390640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d40cd0c0 0x7ff7d41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.955+0000 7ff7db390640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7d415db00 0x7ff7d40b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.955+0000 7ff7db390640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40b0db0 con 0x7ff7d40b0410 2023-10-16T12:34:13.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.955+0000 7ff7db390640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d415e3e0 con 0x7ff7d415db00 2023-10-16T12:34:13.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.955+0000 7ff7db390640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40cd4a0 con 0x7ff7d40cd0c0 2023-10-16T12:34:13.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7d415db00 0x7ff7d40b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7d415db00 0x7ff7d40b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54372/0 (socket says 172.21.15.60:54372) 2023-10-16T12:34:13.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 -- 172.21.15.60:0/669623283 learned_addr learned my addr 172.21.15.60:0/669623283 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:13.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d912c640 1 --2- 172.21.15.60:0/669623283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d40cd0c0 0x7ff7d41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 -- 172.21.15.60:0/669623283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d40cd0c0 msgr2=0x7ff7d41615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:13.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 --2- 172.21.15.60:0/669623283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d40cd0c0 0x7ff7d41615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 -- 172.21.15.60:0/669623283 >> v1:172.21.15.144:6789/0 conn(0x7ff7d40b0410 legacy=0x7ff7d415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:13.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.956+0000 7ff7d992d640 1 -- 172.21.15.60:0/669623283 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d415e100 con 0x7ff7d415db00 2023-10-16T12:34:13.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.957+0000 7ff7d992d640 1 --2- 172.21.15.60:0/669623283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d415db00 0x7ff7d40b0870 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009870 tx=0x7ff7cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2c68940a5dc988 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:13.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.958+0000 7ff7cbfff640 1 -- 172.21.15.60:0/669623283 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0317c0 con 0x7ff7d415db00 2023-10-16T12:34:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.958+0000 7ff7cbfff640 1 -- 172.21.15.60:0/669623283 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc031920 con 0x7ff7d415db00 2023-10-16T12:34:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.958+0000 7ff7cbfff640 1 -- 172.21.15.60:0/669623283 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc031cb0 con 0x7ff7d415db00 2023-10-16T12:34:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.958+0000 7ff7db390640 1 -- 172.21.15.60:0/669623283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d415db00 msgr2=0x7ff7d40b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.958+0000 7ff7db390640 1 --2- 172.21.15.60:0/669623283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d415db00 0x7ff7d40b0870 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009870 tx=0x7ff7cc02b860 comp rx=0 tx=0).stop 2023-10-16T12:34:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.959+0000 7ff7db390640 1 -- 172.21.15.60:0/669623283 shutdown_connections 2023-10-16T12:34:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.959+0000 7ff7db390640 1 --2- 172.21.15.60:0/669623283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d415db00 0x7ff7d40b0870 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.959+0000 7ff7db390640 1 --2- 172.21.15.60:0/669623283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d40cd0c0 0x7ff7d41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.959+0000 7ff7db390640 1 -- 172.21.15.60:0/669623283 >> 172.21.15.60:0/669623283 conn(0x7ff7d40cb410 msgr2=0x7ff7d40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.959+0000 7ff7db390640 1 -- 172.21.15.60:0/669623283 shutdown_connections 2023-10-16T12:34:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.959+0000 7ff7db390640 1 -- 172.21.15.60:0/669623283 wait complete. 2023-10-16T12:34:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.960+0000 7ff7db390640 1 Processor -- start 2023-10-16T12:34:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.961+0000 7ff7db390640 1 -- start start 2023-10-16T12:34:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7db390640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 0x7ff7d40b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7db390640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d40cd0c0 0x7ff7d40c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7db390640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d415db00 0x7ff7d415b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7db390640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41e72c0 con 0x7ff7d40b0410 2023-10-16T12:34:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7db390640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41e66b0 con 0x7ff7d40cd0c0 2023-10-16T12:34:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7db390640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41e6990 con 0x7ff7d415db00 2023-10-16T12:34:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 0x7ff7d40b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 0x7ff7d40b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54382/0 (socket says 172.21.15.60:54382) 2023-10-16T12:34:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 -- 172.21.15.60:0/1455224534 learned_addr learned my addr 172.21.15.60:0/1455224534 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d992d640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d415db00 0x7ff7d415b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d892b640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d40cd0c0 0x7ff7d40c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 -- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d415db00 msgr2=0x7ff7d415b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d415db00 0x7ff7d415b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 -- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d40cd0c0 msgr2=0x7ff7d40c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.962+0000 7ff7d912c640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d40cd0c0 0x7ff7d40c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7d912c640 1 -- 172.21.15.60:0/1455224534 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc0094d0 con 0x7ff7d40b0410 2023-10-16T12:34:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7d912c640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 0x7ff7d40b3cf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0007e40 tx=0x7ff7d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0014620 con 0x7ff7d40b0410 2023-10-16T12:34:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d000ee90 con 0x7ff7d40b0410 2023-10-16T12:34:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d41e6c70 con 0x7ff7d40b0410 2023-10-16T12:34:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d00149c0 con 0x7ff7d40b0410 2023-10-16T12:34:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.963+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d40b4230 con 0x7ff7d40b0410 2023-10-16T12:34:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.965+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d404e160 con 0x7ff7d40b0410 2023-10-16T12:34:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.965+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff7d0014b20 con 0x7ff7d40b0410 2023-10-16T12:34:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.966+0000 7ff7c9ffb640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff7b806ae70 0x7ff7b806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.966+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ff7d008a0d0 con 0x7ff7d40b0410 2023-10-16T12:34:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.967+0000 7ff7d892b640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff7b806ae70 0x7ff7b806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.968+0000 7ff7d892b640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff7b806ae70 0x7ff7b806d330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4006fd0 tx=0x7ff7c4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:13.971+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff7d0054ff0 con 0x7ff7d40b0410 2023-10-16T12:34:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.168+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d4160dc0 con 0x7ff7d40b0410 2023-10-16T12:34:14.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.170+0000 7ff7c9ffb640 1 -- 172.21.15.60:0/1455224534 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16714 (secure 0 0 0) 0x7ff7d0058610 con 0x7ff7d40b0410 2023-10-16T12:34:14.173 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:14.173 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:11.506118+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.172+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff7b806ae70 msgr2=0x7ff7b806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff7b806ae70 0x7ff7b806d330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4006fd0 tx=0x7ff7c4008040 comp rx=0 tx=0).stop 2023-10-16T12:34:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 msgr2=0x7ff7d40b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 0x7ff7d40b3cf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0007e40 tx=0x7ff7d000e4b0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 shutdown_connections 2023-10-16T12:34:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ff7b806ae70 0x7ff7b806d330 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7d415db00 0x7ff7d415b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7d40cd0c0 0x7ff7d40c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 --2- 172.21.15.60:0/1455224534 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7d40b0410 0x7ff7d40b3cf0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 >> 172.21.15.60:0/1455224534 conn(0x7ff7d40cb410 msgr2=0x7ff7d40c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 shutdown_connections 2023-10-16T12:34:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.173+0000 7ff7db390640 1 -- 172.21.15.60:0/1455224534 wait complete. 2023-10-16T12:34:14.196 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:14.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:14.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.430+0000 7f9f34a67640 1 Processor -- start 2023-10-16T12:34:14.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.431+0000 7f9f34a67640 1 -- start start 2023-10-16T12:34:14.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.432+0000 7f9f34a67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 0x7f9f300cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.432+0000 7f9f34a67640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3006be30 con 0x7f9f300b43c0 2023-10-16T12:34:14.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.432+0000 7f9f34a67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300ada10 con 0x7f9f300b0ea0 2023-10-16T12:34:14.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.433+0000 7f9f34a67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3006ca00 con 0x7f9f300b3ec0 2023-10-16T12:34:14.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.433+0000 7f9f2ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9f300b43c0 0x7f9f3015aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50458/0 (socket says 172.21.15.60:50458) 2023-10-16T12:34:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.433+0000 7f9f2ffff640 1 -- 172.21.15.60:0/3221862144 learned_addr learned my addr 172.21.15.60:0/3221862144 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.433+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 0x7f9f300cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.433+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027606005 0 0) 0x7f9f3006be30 con 0x7f9f300b43c0 2023-10-16T12:34:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1c003520 con 0x7f9f300b43c0 2023-10-16T12:34:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187861366 0 0) 0x7f9f300ada10 con 0x7f9f300b0ea0 2023-10-16T12:34:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3006be30 con 0x7f9f300b0ea0 2023-10-16T12:34:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3138827138 0 0) 0x7f9f1c003520 con 0x7f9f300b43c0 2023-10-16T12:34:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 msgr2=0x7f9f300cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 0x7f9f300cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 >> v1:172.21.15.144:6789/0 conn(0x7f9f300b0ea0 legacy=0x7f9f3015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3006c060 con 0x7f9f300b43c0 2023-10-16T12:34:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9f280019f0 con 0x7f9f300b43c0 2023-10-16T12:34:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 >> v1:172.21.15.60:6789/0 conn(0x7f9f300b43c0 legacy=0x7f9f3015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f1c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c0047c0 0x7f9f1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 0x7f9f1c005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3006ca00 con 0x7f9f1c003b30 2023-10-16T12:34:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c00bb80 con 0x7f9f1c0047c0 2023-10-16T12:34:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.434+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c00bd40 con 0x7f9f300b3ec0 2023-10-16T12:34:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f1c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2ffff640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 0x7f9f1c005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f27fff640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c0047c0 0x7f9f1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 msgr2=0x7f9f1c005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300b3ec0 0x7f9f1c005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c0047c0 msgr2=0x7f9f1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f1c0047c0 0x7f9f1c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.435+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/3221862144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c003520 con 0x7f9f1c003b30 2023-10-16T12:34:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.436+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f1c003f30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009ac0 tx=0x7f9f1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d53c7c6d978989 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.436+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2002c6b0 con 0x7f9f1c003b30 2023-10-16T12:34:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2002c810 con 0x7f9f1c003b30 2023-10-16T12:34:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f2effd640 1 -- 172.21.15.60:0/3221862144 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2002cba0 con 0x7f9f1c003b30 2023-10-16T12:34:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f34a67640 1 -- 172.21.15.60:0/3221862144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 msgr2=0x7f9f1c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f34a67640 1 --2- 172.21.15.60:0/3221862144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f1c003f30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009ac0 tx=0x7f9f1c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f2ffff640 1 -- 172.21.15.60:0/3221862144 reap_dead start 2023-10-16T12:34:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f34a67640 1 -- 172.21.15.60:0/3221862144 shutdown_connections 2023-10-16T12:34:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f34a67640 1 -- 172.21.15.60:0/3221862144 >> 172.21.15.60:0/3221862144 conn(0x7f9f300b0170 msgr2=0x7f9f300b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f34a67640 1 -- 172.21.15.60:0/3221862144 shutdown_connections 2023-10-16T12:34:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.437+0000 7f9f34a67640 1 -- 172.21.15.60:0/3221862144 wait complete. 2023-10-16T12:34:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.439+0000 7f9f34a67640 1 Processor -- start 2023-10-16T12:34:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.439+0000 7f9f34a67640 1 -- start start 2023-10-16T12:34:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f34a67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f3015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f34a67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f301596d0 0x7f9f30159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f34a67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300702c0 0x7f9f30070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f34a67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3015e090 con 0x7f9f1c003b30 2023-10-16T12:34:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f34a67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300c4b30 con 0x7f9f301596d0 2023-10-16T12:34:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f34a67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300c4280 con 0x7f9f300702c0 2023-10-16T12:34:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f2f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f3015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f2f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f3015d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54396/0 (socket says 172.21.15.60:54396) 2023-10-16T12:34:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.440+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/634399713 learned_addr learned my addr 172.21.15.60:0/634399713 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f27fff640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f301596d0 0x7f9f30159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300702c0 msgr2=0x7f9f30070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300702c0 0x7f9f30070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f301596d0 msgr2=0x7f9f30159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f301596d0 0x7f9f30159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f2f7fe640 1 -- 172.21.15.60:0/634399713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3015ddb0 con 0x7f9f1c003b30 2023-10-16T12:34:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.441+0000 7f9f2f7fe640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f3015d7d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9f20037040 tx=0x7f9f20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.442+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20003810 con 0x7f9f1c003b30 2023-10-16T12:34:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.442+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f20003970 con 0x7f9f1c003b30 2023-10-16T12:34:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.442+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20003d00 con 0x7f9f1c003b30 2023-10-16T12:34:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.442+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f300c4470 con 0x7f9f1c003b30 2023-10-16T12:34:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.442+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f300c4f50 con 0x7f9f1c003b30 2023-10-16T12:34:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.443+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f30070c80 con 0x7f9f1c003b30 2023-10-16T12:34:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.444+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9f20040450 con 0x7f9f1c003b30 2023-10-16T12:34:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.445+0000 7f9f2cff9640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9f1006ae70 0x7f9f1006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.445+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f9f200ab6d0 con 0x7f9f1c003b30 2023-10-16T12:34:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.445+0000 7f9f27fff640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9f1006ae70 0x7f9f1006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.447+0000 7f9f27fff640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9f1006ae70 0x7f9f1006d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9f180010e0 tx=0x7f9f18009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.449+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f20076680 con 0x7f9f1c003b30 2023-10-16T12:34:14.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.646+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f30074380 con 0x7f9f1c003b30 2023-10-16T12:34:14.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.647+0000 7f9f2cff9640 1 -- 172.21.15.60:0/634399713 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f200044e0 con 0x7f9f1c003b30 2023-10-16T12:34:14.649 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.649+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9f1006ae70 msgr2=0x7f9f1006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.649+0000 7f9f34a67640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9f1006ae70 0x7f9f1006d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9f180010e0 tx=0x7f9f18009040 comp rx=0 tx=0).stop 2023-10-16T12:34:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.649+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 msgr2=0x7f9f3015d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.649+0000 7f9f34a67640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f3015d7d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9f20037040 tx=0x7f9f20004730 comp rx=0 tx=0).stop 2023-10-16T12:34:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 shutdown_connections 2023-10-16T12:34:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f9f1006ae70 0x7f9f1006d330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f300702c0 0x7f9f30070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f301596d0 0x7f9f30159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 --2- 172.21.15.60:0/634399713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c003b30 0x7f9f3015d7d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 >> 172.21.15.60:0/634399713 conn(0x7f9f300b0170 msgr2=0x7f9f300ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 shutdown_connections 2023-10-16T12:34:14.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.650+0000 7f9f34a67640 1 -- 172.21.15.60:0/634399713 wait complete. 2023-10-16T12:34:14.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:14.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.896+0000 7feaf5fba640 1 Processor -- start 2023-10-16T12:34:14.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.896+0000 7feaf5fba640 1 -- start start 2023-10-16T12:34:14.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf5fba640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf5fba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feaf00cda80 0x7feaf015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf5fba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf006c960 con 0x7feaf00ad970 2023-10-16T12:34:14.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf5fba640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf015b9d0 con 0x7feaf00cda80 2023-10-16T12:34:14.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf5fba640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf015bcb0 con 0x7feaf006c040 2023-10-16T12:34:14.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf57b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feaf00cda80 0x7feaf015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf57b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feaf00cda80 0x7feaf015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54408/0 (socket says 172.21.15.60:54408) 2023-10-16T12:34:14.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaf57b9640 1 -- 172.21.15.60:0/3296749391 learned_addr learned my addr 172.21.15.60:0/3296749391 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:14.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.898+0000 7feaeffff640 1 --1- 172.21.15.60:0/3296749391 >> v1:172.21.15.60:6789/0 conn(0x7feaf00ad970 0x7feaf015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50460/0 (socket says 172.21.15.60:50460) 2023-10-16T12:34:14.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.899+0000 7feaf4fb8640 1 --2- 172.21.15.60:0/3296749391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.899+0000 7feaf57b9640 1 -- 172.21.15.60:0/3296749391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 msgr2=0x7feaf006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:14.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.899+0000 7feaf57b9640 1 --2- 172.21.15.60:0/3296749391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.899+0000 7feaf57b9640 1 -- 172.21.15.60:0/3296749391 >> v1:172.21.15.60:6789/0 conn(0x7feaf00ad970 legacy=0x7feaf015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.899+0000 7feaf57b9640 1 -- 172.21.15.60:0/3296749391 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf015bf90 con 0x7feaf00cda80 2023-10-16T12:34:14.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.899+0000 7feaef7fe640 1 -- 172.21.15.60:0/3296749391 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975044323 0 0) 0x7feaf006c960 con 0x7feaf00ad970 2023-10-16T12:34:14.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.900+0000 7feaf57b9640 1 --2- 172.21.15.60:0/3296749391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf015ed90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e50 tx=0x7feae802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef781bf3af0976a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:14.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaef7fe640 1 -- 172.21.15.60:0/3296749391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae802caf0 con 0x7feaf00cda80 2023-10-16T12:34:14.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaef7fe640 1 -- 172.21.15.60:0/3296749391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae802cc50 con 0x7feaf00cda80 2023-10-16T12:34:14.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaef7fe640 1 -- 172.21.15.60:0/3296749391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80343f0 con 0x7feaf00cda80 2023-10-16T12:34:14.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 -- 172.21.15.60:0/3296749391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 msgr2=0x7feaf015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 --2- 172.21.15.60:0/3296749391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf015ed90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e50 tx=0x7feae802bf50 comp rx=0 tx=0).stop 2023-10-16T12:34:14.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 -- 172.21.15.60:0/3296749391 shutdown_connections 2023-10-16T12:34:14.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 --2- 172.21.15.60:0/3296749391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf015ed90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 --2- 172.21.15.60:0/3296749391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 -- 172.21.15.60:0/3296749391 >> 172.21.15.60:0/3296749391 conn(0x7feaf00ce6a0 msgr2=0x7feaf00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:14.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 -- 172.21.15.60:0/3296749391 shutdown_connections 2023-10-16T12:34:14.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.901+0000 7feaf5fba640 1 -- 172.21.15.60:0/3296749391 wait complete. 2023-10-16T12:34:14.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.903+0000 7feaf5fba640 1 Processor -- start 2023-10-16T12:34:14.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.903+0000 7feaf5fba640 1 -- start start 2023-10-16T12:34:14.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.904+0000 7feaf5fba640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf0151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.904+0000 7feaf5fba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaf00ad970 0x7feaf01592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.904+0000 7feaf5fba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf00c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.904+0000 7feaf5fba640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01f6b20 con 0x7feaf00cda80 2023-10-16T12:34:14.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.904+0000 7feaf5fba640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01597f0 con 0x7feaf00ad970 2023-10-16T12:34:14.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.904+0000 7feaf5fba640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01f5fa0 con 0x7feaf006c040 2023-10-16T12:34:14.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf00c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf00c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54416/0 (socket says 172.21.15.60:54416) 2023-10-16T12:34:14.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 -- 172.21.15.60:0/1011842834 learned_addr learned my addr 172.21.15.60:0/1011842834 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:14.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf4fb8640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf0151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 -- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 msgr2=0x7feaf0151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf0151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 -- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaf00ad970 msgr2=0x7feaf01592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:14.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaf00ad970 0x7feaf01592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 -- 172.21.15.60:0/1011842834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae8009b30 con 0x7feaf00cda80 2023-10-16T12:34:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.905+0000 7feaf57b9640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf00c9610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e50 tx=0x7feae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.906+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8009510 con 0x7feaf00cda80 2023-10-16T12:34:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.906+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae8040470 con 0x7feaf00cda80 2023-10-16T12:34:14.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.906+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8040800 con 0x7feaf00cda80 2023-10-16T12:34:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.906+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf01f6280 con 0x7feaf00cda80 2023-10-16T12:34:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.906+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf01f6840 con 0x7feaf00cda80 2023-10-16T12:34:14.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.907+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf006b330 con 0x7feaf00cda80 2023-10-16T12:34:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.908+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feae8040960 con 0x7feaf00cda80 2023-10-16T12:34:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.909+0000 7feaed7fa640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fead406ae70 0x7fead406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:14.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.909+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7feae80ac3e0 con 0x7feaf00cda80 2023-10-16T12:34:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.910+0000 7feaf4fb8640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fead406ae70 0x7fead406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.911+0000 7feaf4fb8640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fead406ae70 0x7fead406d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feae4006fd0 tx=0x7feae4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:14.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:14.914+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feae80774b0 con 0x7feaf00cda80 2023-10-16T12:34:15.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.109+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feaf0156540 con 0x7feaf00cda80 2023-10-16T12:34:15.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.110+0000 7feaed7fa640 1 -- 172.21.15.60:0/1011842834 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7feaf0156540 con 0x7feaf00cda80 2023-10-16T12:34:15.112 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fead406ae70 msgr2=0x7fead406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fead406ae70 0x7fead406d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feae4006fd0 tx=0x7feae4008040 comp rx=0 tx=0).stop 2023-10-16T12:34:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 msgr2=0x7feaf00c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf00c9610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e50 tx=0x7feae8004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 shutdown_connections 2023-10-16T12:34:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fead406ae70 0x7fead406d330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feaf00cda80 0x7feaf00c9610 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaf00ad970 0x7feaf01592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 --2- 172.21.15.60:0/1011842834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feaf006c040 0x7feaf0151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.112+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 >> 172.21.15.60:0/1011842834 conn(0x7feaf00ce6a0 msgr2=0x7feaf015a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.113+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 shutdown_connections 2023-10-16T12:34:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.113+0000 7feaf5fba640 1 -- 172.21.15.60:0/1011842834 wait complete. 2023-10-16T12:34:15.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.358+0000 7f4f32fb7640 1 Processor -- start 2023-10-16T12:34:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.359+0000 7f4f32fb7640 1 -- start start 2023-10-16T12:34:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.360+0000 7f4f32fb7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.360+0000 7f4f32fb7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f2c06c140 0x7f4f2c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.360+0000 7f4f32fb7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c070360 con 0x7f4f2c159ab0 2023-10-16T12:34:15.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.360+0000 7f4f32fb7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c06cb10 con 0x7f4f2c06c140 2023-10-16T12:34:15.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.360+0000 7f4f32fb7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c15d890 con 0x7f4f2c159e90 2023-10-16T12:34:15.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.361+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.361+0000 7f4f31554640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f2c06c140 0x7f4f2c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.361+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:55170/0 (socket says 172.21.15.60:55170) 2023-10-16T12:34:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.361+0000 7f4f2bfff640 1 -- 172.21.15.60:0/2402727332 learned_addr learned my addr 172.21.15.60:0/2402727332 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.362+0000 7f4f31554640 1 -- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c159e90 msgr2=0x7f4f2c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.362+0000 7f4f31554640 1 --2- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.362+0000 7f4f31554640 1 -- 172.21.15.60:0/2402727332 >> v1:172.21.15.144:6789/0 conn(0x7f4f2c159ab0 legacy=0x7f4f2c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.362+0000 7f4f31554640 1 -- 172.21.15.60:0/2402727332 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0cd1f0 con 0x7f4f2c06c140 2023-10-16T12:34:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.364+0000 7f4f31554640 1 --2- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c06c140 0x7f4f2c06c5d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009dc0 tx=0x7f4f2402bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3effdcd723f8e0b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.364+0000 7f4f2b7fe640 1 -- 172.21.15.60:0/2402727332 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2402cb10 con 0x7f4f2c06c140 2023-10-16T12:34:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.364+0000 7f4f2b7fe640 1 -- 172.21.15.60:0/2402727332 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2402cc70 con 0x7f4f2c06c140 2023-10-16T12:34:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.364+0000 7f4f2b7fe640 1 -- 172.21.15.60:0/2402727332 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24034440 con 0x7f4f2c06c140 2023-10-16T12:34:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 -- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c06c140 msgr2=0x7f4f2c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c06c140 0x7f4f2c06c5d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009dc0 tx=0x7f4f2402bd60 comp rx=0 tx=0).stop 2023-10-16T12:34:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 -- 172.21.15.60:0/2402727332 shutdown_connections 2023-10-16T12:34:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c06c140 0x7f4f2c06c5d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/2402727332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c159e90 0x7f4f2c0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 -- 172.21.15.60:0/2402727332 >> 172.21.15.60:0/2402727332 conn(0x7f4f2c0ce9b0 msgr2=0x7f4f2c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 -- 172.21.15.60:0/2402727332 shutdown_connections 2023-10-16T12:34:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.365+0000 7f4f32fb7640 1 -- 172.21.15.60:0/2402727332 wait complete. 2023-10-16T12:34:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.367+0000 7f4f32fb7640 1 Processor -- start 2023-10-16T12:34:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.367+0000 7f4f32fb7640 1 -- start start 2023-10-16T12:34:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f32fb7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c06c140 0x7f4f2c0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f32fb7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 0x7f4f2c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f32fb7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c159e90 0x7f4f2c0c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f32fb7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1574a0 con 0x7f4f2c159ab0 2023-10-16T12:34:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f32fb7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1f6f40 con 0x7f4f2c159e90 2023-10-16T12:34:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f32fb7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c7470 con 0x7f4f2c06c140 2023-10-16T12:34:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 0x7f4f2c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 0x7f4f2c1f5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54428/0 (socket says 172.21.15.60:54428) 2023-10-16T12:34:15.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.368+0000 7f4f2bfff640 1 -- 172.21.15.60:0/1092194588 learned_addr learned my addr 172.21.15.60:0/1092194588 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:15.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f30d53640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c06c140 0x7f4f2c0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f31554640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c159e90 0x7f4f2c0c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f2bfff640 1 -- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c06c140 msgr2=0x7f4f2c0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f2bfff640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c06c140 0x7f4f2c0c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f2bfff640 1 -- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c159e90 msgr2=0x7f4f2c0c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f2bfff640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c159e90 0x7f4f2c0c6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f2bfff640 1 -- 172.21.15.60:0/1092194588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f24009a70 con 0x7f4f2c159ab0 2023-10-16T12:34:15.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f2bfff640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 0x7f4f2c1f5fa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009fd0 tx=0x7f4f1800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:15.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1800c940 con 0x7f4f2c159ab0 2023-10-16T12:34:15.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.369+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1800caa0 con 0x7f4f2c159ab0 2023-10-16T12:34:15.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.370+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1800ce30 con 0x7f4f2c159ab0 2023-10-16T12:34:15.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.370+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c1f66a0 con 0x7f4f2c159ab0 2023-10-16T12:34:15.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.370+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c1f6c60 con 0x7f4f2c159ab0 2023-10-16T12:34:15.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.371+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c04e160 con 0x7f4f2c159ab0 2023-10-16T12:34:15.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.372+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f18013470 con 0x7f4f2c159ab0 2023-10-16T12:34:15.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.373+0000 7f4f297fa640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f1006aec0 0x7f4f1006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.373+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f4f18089810 con 0x7f4f2c159ab0 2023-10-16T12:34:15.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.373+0000 7f4f30d53640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f1006aec0 0x7f4f1006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.375+0000 7f4f30d53640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f1006aec0 0x7f4f1006d380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4f20000f60 tx=0x7f4f20009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:15.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.377+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f18054840 con 0x7f4f2c159ab0 2023-10-16T12:34:15.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.574+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4f2c157200 con 0x7f4f2c159ab0 2023-10-16T12:34:15.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.575+0000 7f4f297fa640 1 -- 172.21.15.60:0/1092194588 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4f18057e60 con 0x7f4f2c159ab0 2023-10-16T12:34:15.577 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:15.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.577+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f1006aec0 msgr2=0x7f4f1006d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.577+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f1006aec0 0x7f4f1006d380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4f20000f60 tx=0x7f4f20009040 comp rx=0 tx=0).stop 2023-10-16T12:34:15.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.577+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 msgr2=0x7f4f2c1f5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.577+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 0x7f4f2c1f5fa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009fd0 tx=0x7f4f1800ed40 comp rx=0 tx=0).stop 2023-10-16T12:34:15.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 shutdown_connections 2023-10-16T12:34:15.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f1006aec0 0x7f4f1006d380 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f2c159e90 0x7f4f2c0c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f2c159ab0 0x7f4f2c1f5fa0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 --2- 172.21.15.60:0/1092194588 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2c06c140 0x7f4f2c0c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 >> 172.21.15.60:0/1092194588 conn(0x7f4f2c0ce9b0 msgr2=0x7f4f2c0afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:15.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 shutdown_connections 2023-10-16T12:34:15.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.578+0000 7f4f32fb7640 1 -- 172.21.15.60:0/1092194588 wait complete. 2023-10-16T12:34:15.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:34:15.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.822+0000 7f0857fff640 1 Processor -- start 2023-10-16T12:34:15.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.823+0000 7f0857fff640 1 -- start start 2023-10-16T12:34:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.824+0000 7f0857fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f085806c040 0x7f085806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.825+0000 7f0857fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580cda80 0x7f085815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.825+0000 7f0857fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f085806c960 con 0x7f08580ad970 2023-10-16T12:34:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.825+0000 7f0857fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085815b9d0 con 0x7f085806c040 2023-10-16T12:34:15.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.825+0000 7f0857fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085815bcb0 con 0x7f08580cda80 2023-10-16T12:34:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.825+0000 7f0856ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f085806c040 0x7f085806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.825+0000 7f08577fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580cda80 0x7f085815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.826+0000 7f0856ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f085806c040 0x7f085806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:56846/0 (socket says 172.21.15.60:56846) 2023-10-16T12:34:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.826+0000 7f0856ffd640 1 -- 172.21.15.60:0/1064749177 learned_addr learned my addr 172.21.15.60:0/1064749177 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:15.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.826+0000 7f0856ffd640 1 -- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580cda80 msgr2=0x7f085815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:15.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.826+0000 7f0856ffd640 1 --2- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580cda80 0x7f085815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.826+0000 7f0856ffd640 1 -- 172.21.15.60:0/1064749177 >> v1:172.21.15.144:6789/0 conn(0x7f08580ad970 legacy=0x7f085815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.827+0000 7f0856ffd640 1 -- 172.21.15.60:0/1064749177 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085815bf90 con 0x7f085806c040 2023-10-16T12:34:15.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.828+0000 7f0856ffd640 1 --2- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f085806c040 0x7f085806c420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0848009870 tx=0x7f084802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c9ca73c4f045c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:15.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.828+0000 7f08567fc640 1 -- 172.21.15.60:0/1064749177 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848030840 con 0x7f085806c040 2023-10-16T12:34:15.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f08567fc640 1 -- 172.21.15.60:0/1064749177 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08480309a0 con 0x7f085806c040 2023-10-16T12:34:15.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f08567fc640 1 -- 172.21.15.60:0/1064749177 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848030d30 con 0x7f085806c040 2023-10-16T12:34:15.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 -- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f085806c040 msgr2=0x7f085806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 --2- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f085806c040 0x7f085806c420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0848009870 tx=0x7f084802b980 comp rx=0 tx=0).stop 2023-10-16T12:34:15.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 -- 172.21.15.60:0/1064749177 shutdown_connections 2023-10-16T12:34:15.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 --2- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580cda80 0x7f085815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 --2- 172.21.15.60:0/1064749177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f085806c040 0x7f085806c420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 -- 172.21.15.60:0/1064749177 >> 172.21.15.60:0/1064749177 conn(0x7f08580ce6a0 msgr2=0x7f08580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:15.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 -- 172.21.15.60:0/1064749177 shutdown_connections 2023-10-16T12:34:15.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.829+0000 7f0857fff640 1 -- 172.21.15.60:0/1064749177 wait complete. 2023-10-16T12:34:15.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.831+0000 7f0857fff640 1 Processor -- start 2023-10-16T12:34:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.831+0000 7f0857fff640 1 -- start start 2023-10-16T12:34:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0857fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 0x7f08580c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0857fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580ad970 0x7f085815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0857fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08580cda80 0x7f085815d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0857fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08580af510 con 0x7f085806c040 2023-10-16T12:34:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0857fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08580c6e50 con 0x7f08580cda80 2023-10-16T12:34:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0857fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08580c70d0 con 0x7f08580ad970 2023-10-16T12:34:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.832+0000 7f0856ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 0x7f08580c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 0x7f08580c6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54442/0 (socket says 172.21.15.60:54442) 2023-10-16T12:34:15.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 -- 172.21.15.60:0/795787365 learned_addr learned my addr 172.21.15.60:0/795787365 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:15.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f084ffff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580ad970 0x7f085815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f08577fe640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08580cda80 0x7f085815d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 -- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580ad970 msgr2=0x7f085815d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580ad970 0x7f085815d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 -- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08580cda80 msgr2=0x7f085815d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08580cda80 0x7f085815d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 -- 172.21.15.60:0/795787365 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08480094d0 con 0x7f085806c040 2023-10-16T12:34:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f084ffff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580ad970 0x7f085815d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f08577fe640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08580cda80 0x7f085815d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.833+0000 7f0856ffd640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 0x7f08580c6910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0848000c00 tx=0x7f0848004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.834+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848039c90 con 0x7f085806c040 2023-10-16T12:34:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.834+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0848039df0 con 0x7f085806c040 2023-10-16T12:34:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.834+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848043640 con 0x7f085806c040 2023-10-16T12:34:15.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.834+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08580c73b0 con 0x7f085806c040 2023-10-16T12:34:15.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.834+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08580c7970 con 0x7f085806c040 2023-10-16T12:34:15.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.835+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085806b330 con 0x7f085806c040 2023-10-16T12:34:15.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.836+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f08480437a0 con 0x7f085806c040 2023-10-16T12:34:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.837+0000 7f0854ff9640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f083806ae70 0x7f083806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.837+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f084803e070 con 0x7f085806c040 2023-10-16T12:34:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.838+0000 7f084ffff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f083806ae70 0x7f083806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:15.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.839+0000 7f084ffff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f083806ae70 0x7f083806d330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0840000f60 tx=0x7f0840009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:15.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:15.842+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08480776e0 con 0x7f085806c040 2023-10-16T12:34:16.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.049+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f085804e9b0 con 0x7f085806c040 2023-10-16T12:34:16.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.050+0000 7f0854ff9640 1 -- 172.21.15.60:0/795787365 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f084807ad00 con 0x7f085806c040 2023-10-16T12:34:16.053 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2023-10-16T12:34:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f083806ae70 msgr2=0x7f083806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f083806ae70 0x7f083806d330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0840000f60 tx=0x7f0840009040 comp rx=0 tx=0).stop 2023-10-16T12:34:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 msgr2=0x7f08580c6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 0x7f08580c6910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0848000c00 tx=0x7f0848004730 comp rx=0 tx=0).stop 2023-10-16T12:34:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 shutdown_connections 2023-10-16T12:34:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f083806ae70 0x7f083806d330 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08580ad970 0x7f085815d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08580cda80 0x7f085815d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 --2- 172.21.15.60:0/795787365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f085806c040 0x7f08580c6910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 >> 172.21.15.60:0/795787365 conn(0x7f08580ce6a0 msgr2=0x7f08580c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:16.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 shutdown_connections 2023-10-16T12:34:16.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.053+0000 7f0857fff640 1 -- 172.21.15.60:0/795787365 wait complete. 2023-10-16T12:34:16.073 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-16T12:34:16.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:16.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-16T12:34:16.168 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:34:16.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:16.169 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-16T12:34:16.221 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:34:16.221 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:16.283 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:34:16.283 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:16.337 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:34:16.338 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:16.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:16.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.634+0000 7f5504464640 1 Processor -- start 2023-10-16T12:34:16.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.634+0000 7f5504464640 1 -- start start 2023-10-16T12:34:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.635+0000 7f5504464640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f54fc15aaa0 0x7f54fc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.635+0000 7f5504464640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54fc15ae80 0x7f54fc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.635+0000 7f5504464640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54fc15ed60 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.635+0000 7f5504464640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc06b810 con 0x7f54fc15ae80 2023-10-16T12:34:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.635+0000 7f5504464640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc0707e0 con 0x7f54fc15aaa0 2023-10-16T12:34:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54fc15ae80 0x7f54fc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54fc15ae80 0x7f54fc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54446/0 (socket says 172.21.15.60:54446) 2023-10-16T12:34:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 -- 172.21.15.60:0/2006973265 learned_addr learned my addr 172.21.15.60:0/2006973265 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f5502200640 1 --2- 172.21.15.60:0/2006973265 >> v2:172.21.15.144:3300/0 conn(0x7f54fc15aaa0 0x7f54fc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 -- 172.21.15.60:0/2006973265 >> v2:172.21.15.144:3300/0 conn(0x7f54fc15aaa0 msgr2=0x7f54fc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 --2- 172.21.15.60:0/2006973265 >> v2:172.21.15.144:3300/0 conn(0x7f54fc15aaa0 0x7f54fc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 -- 172.21.15.60:0/2006973265 >> v1:172.21.15.144:6789/0 conn(0x7f54fc0ccaa0 legacy=0x7f54fc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.636+0000 7f55019ff640 1 -- 172.21.15.60:0/2006973265 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc15ef90 con 0x7f54fc15ae80 2023-10-16T12:34:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.638+0000 7f55019ff640 1 --2- 172.21.15.60:0/2006973265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc15ae80 0x7f54fc0cc560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0cd990 tx=0x7f54ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0eef7d8adead4e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.638+0000 7f55011fe640 1 -- 172.21.15.60:0/2006973265 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec031870 con 0x7f54fc15ae80 2023-10-16T12:34:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.638+0000 7f55011fe640 1 -- 172.21.15.60:0/2006973265 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec0319d0 con 0x7f54fc15ae80 2023-10-16T12:34:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.638+0000 7f55011fe640 1 -- 172.21.15.60:0/2006973265 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec031d60 con 0x7f54fc15ae80 2023-10-16T12:34:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.638+0000 7f5504464640 1 -- 172.21.15.60:0/2006973265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc15ae80 msgr2=0x7f54fc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.638+0000 7f5504464640 1 --2- 172.21.15.60:0/2006973265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc15ae80 0x7f54fc0cc560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f54fc0cd990 tx=0x7f54ec02b910 comp rx=0 tx=0).stop 2023-10-16T12:34:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.639+0000 7f5504464640 1 -- 172.21.15.60:0/2006973265 shutdown_connections 2023-10-16T12:34:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.639+0000 7f5504464640 1 --2- 172.21.15.60:0/2006973265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc15ae80 0x7f54fc0cc560 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.639+0000 7f5504464640 1 --2- 172.21.15.60:0/2006973265 >> v2:172.21.15.144:3300/0 conn(0x7f54fc15aaa0 0x7f54fc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.639+0000 7f5504464640 1 -- 172.21.15.60:0/2006973265 >> 172.21.15.60:0/2006973265 conn(0x7f54fc0cb410 msgr2=0x7f54fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.639+0000 7f5504464640 1 -- 172.21.15.60:0/2006973265 shutdown_connections 2023-10-16T12:34:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.639+0000 7f5504464640 1 -- 172.21.15.60:0/2006973265 wait complete. 2023-10-16T12:34:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.640+0000 7f5504464640 1 Processor -- start 2023-10-16T12:34:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.641+0000 7f5504464640 1 -- start start 2023-10-16T12:34:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5504464640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 0x7f54fc1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5504464640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54fc15aaa0 0x7f54fc0c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5504464640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54fc15ae80 0x7f54fc0af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5504464640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0c3cf0 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5504464640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0c24c0 con 0x7f54fc15aaa0 2023-10-16T12:34:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5504464640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0c88b0 con 0x7f54fc15ae80 2023-10-16T12:34:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 0x7f54fc1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 0x7f54fc1560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54452/0 (socket says 172.21.15.60:54452) 2023-10-16T12:34:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 -- 172.21.15.60:0/436074416 learned_addr learned my addr 172.21.15.60:0/436074416 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f55019ff640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54fc15aaa0 0x7f54fc0c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 -- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54fc15ae80 msgr2=0x7f54fc0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54fc15ae80 0x7f54fc0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 -- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54fc15aaa0 msgr2=0x7f54fc0c37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54fc15aaa0 0x7f54fc0c37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.642+0000 7f5502200640 1 -- 172.21.15.60:0/436074416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ec0094d0 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.643+0000 7f5502200640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 0x7f54fc1560a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f54f800d830 tx=0x7f54f800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.643+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f800b6c0 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.643+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f800b820 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.643+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc0c2710 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.643+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f800bbb0 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.643+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc0c2cd0 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.644+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc04e160 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.645+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f54f801e460 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.647+0000 7f54f2ffd640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f54e006ae70 0x7f54e006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.647+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f54f8089e20 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.647+0000 7f55019ff640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f54e006ae70 0x7f54e006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.648+0000 7f55019ff640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f54e006ae70 0x7f54e006d330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f54ec036000 tx=0x7f54ec03c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:16.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.651+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54f8054e60 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.852+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54fc0cdc60 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.855+0000 7f54f2ffd640 1 -- 172.21.15.60:0/436074416 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16714 (secure 0 0 0) 0x7f54f8058480 con 0x7f54fc0ccaa0 2023-10-16T12:34:16.858 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:16.858 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:11.506118+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.857+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f54e006ae70 msgr2=0x7f54e006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.857+0000 7f5504464640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f54e006ae70 0x7f54e006d330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f54ec036000 tx=0x7f54ec03c040 comp rx=0 tx=0).stop 2023-10-16T12:34:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.857+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 msgr2=0x7f54fc1560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.857+0000 7f5504464640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 0x7f54fc1560a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f54f800d830 tx=0x7f54f800dcf0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 shutdown_connections 2023-10-16T12:34:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f54e006ae70 0x7f54e006d330 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54fc15ae80 0x7f54fc0af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54fc15aaa0 0x7f54fc0c37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 --2- 172.21.15.60:0/436074416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54fc0ccaa0 0x7f54fc1560a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 >> 172.21.15.60:0/436074416 conn(0x7f54fc0cb410 msgr2=0x7f54fc0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 shutdown_connections 2023-10-16T12:34:16.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:16.858+0000 7f5504464640 1 -- 172.21.15.60:0/436074416 wait complete. 2023-10-16T12:34:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:16.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:17.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.103+0000 7f0804802640 1 Processor -- start 2023-10-16T12:34:17.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.104+0000 7f0804802640 1 -- start start 2023-10-16T12:34:17.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.105+0000 7f0804802640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f080015ad70 0x7f0800069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.105+0000 7f0804802640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000cb030 con 0x7f08000c9ca0 2023-10-16T12:34:17.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.105+0000 7f0804802640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000c91c0 con 0x7f08000caac0 2023-10-16T12:34:17.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.105+0000 7f0804802640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08000c94a0 con 0x7f080015ad70 2023-10-16T12:34:17.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.106+0000 7f07f7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f08000c9ca0 0x7f08000ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50462/0 (socket says 172.21.15.60:50462) 2023-10-16T12:34:17.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.106+0000 7f07f7fff640 1 -- 172.21.15.60:0/546017114 learned_addr learned my addr 172.21.15.60:0/546017114 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:17.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.106+0000 7f07fe59c640 1 --2- 172.21.15.60:0/546017114 >> v2:172.21.15.144:3300/0 conn(0x7f080015ad70 0x7f0800069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.106+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174300294 0 0) 0x7f08000cb030 con 0x7f08000c9ca0 2023-10-16T12:34:17.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.106+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ec003520 con 0x7f08000c9ca0 2023-10-16T12:34:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735507038 0 0) 0x7f07ec003520 con 0x7f08000c9ca0 2023-10-16T12:34:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 >> v2:172.21.15.144:3300/0 conn(0x7f080015ad70 msgr2=0x7f0800069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 --2- 172.21.15.60:0/546017114 >> v2:172.21.15.144:3300/0 conn(0x7f080015ad70 0x7f0800069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 >> v1:172.21.15.144:6789/0 conn(0x7f08000caac0 legacy=0x7f08000ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000c9780 con 0x7f08000c9ca0 2023-10-16T12:34:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f07e8001670 con 0x7f08000c9ca0 2023-10-16T12:34:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 >> v1:172.21.15.60:6789/0 conn(0x7f08000c9ca0 legacy=0x7f08000ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ec003970 0x7f07ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f07ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07ec005180 0x7f07ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000c94a0 con 0x7f07ec0047c0 2023-10-16T12:34:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec00c020 con 0x7f07ec003970 2023-10-16T12:34:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec00c1e0 con 0x7f07ec005180 2023-10-16T12:34:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.107+0000 7f07f7fff640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f07ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07fe59c640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ec003970 0x7f07ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07f7fff640 1 -- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07ec005180 msgr2=0x7f07ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07fed9d640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07ec005180 0x7f07ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07f7fff640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07ec005180 0x7f07ec00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07f7fff640 1 -- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ec003970 msgr2=0x7f07ec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07f7fff640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ec003970 0x7f07ec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07f7fff640 1 -- 172.21.15.60:0/546017114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec003520 con 0x7f07ec0047c0 2023-10-16T12:34:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07fe59c640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ec003970 0x7f07ec003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.108+0000 7f07fed9d640 1 -- 172.21.15.60:0/546017114 reap_dead start 2023-10-16T12:34:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.109+0000 7f07f7fff640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f07ec004c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c190 tx=0x7f07ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112c90b9c4f15709 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.109+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8005ad0 con 0x7f07ec0047c0 2023-10-16T12:34:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.109+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e8005c30 con 0x7f07ec0047c0 2023-10-16T12:34:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.109+0000 7f07fdd9b640 1 -- 172.21.15.60:0/546017114 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e802cae0 con 0x7f07ec0047c0 2023-10-16T12:34:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 -- 172.21.15.60:0/546017114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 msgr2=0x7f07ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f07ec004c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c190 tx=0x7f07ec0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 -- 172.21.15.60:0/546017114 shutdown_connections 2023-10-16T12:34:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 --2- 172.21.15.60:0/546017114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f07ec004c40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 -- 172.21.15.60:0/546017114 >> 172.21.15.60:0/546017114 conn(0x7f08000ce9b0 msgr2=0x7f08000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 -- 172.21.15.60:0/546017114 shutdown_connections 2023-10-16T12:34:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.110+0000 7f0804802640 1 -- 172.21.15.60:0/546017114 wait complete. 2023-10-16T12:34:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.112+0000 7f0804802640 1 Processor -- start 2023-10-16T12:34:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.112+0000 7f0804802640 1 -- start start 2023-10-16T12:34:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f0804802640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f0800158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f0804802640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08000b2220 0x7f08000c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f0804802640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08000b26a0 0x7f08000c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f0804802640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0800159430 con 0x7f07ec0047c0 2023-10-16T12:34:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f0804802640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000b0590 con 0x7f08000b2220 2023-10-16T12:34:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f0804802640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000b0870 con 0x7f08000b26a0 2023-10-16T12:34:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f07fe59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f0800158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f07fe59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f0800158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54476/0 (socket says 172.21.15.60:54476) 2023-10-16T12:34:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.113+0000 7f07fe59c640 1 -- 172.21.15.60:0/3944415381 learned_addr learned my addr 172.21.15.60:0/3944415381 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07f7fff640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08000b2220 0x7f08000c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fed9d640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08000b26a0 0x7f08000c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fe59c640 1 -- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08000b26a0 msgr2=0x7f08000c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fe59c640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08000b26a0 0x7f08000c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fe59c640 1 -- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08000b2220 msgr2=0x7f08000c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fe59c640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08000b2220 0x7f08000c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fe59c640 1 -- 172.21.15.60:0/3944415381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000b0b50 con 0x7f07ec0047c0 2023-10-16T12:34:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07fe59c640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f0800158ef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f07f00107c0 tx=0x7f07f0010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f000eb60 con 0x7f07ec0047c0 2023-10-16T12:34:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.114+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f0010ed0 con 0x7f07ec0047c0 2023-10-16T12:34:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.115+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f001c400 con 0x7f07ec0047c0 2023-10-16T12:34:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.115+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08000b0e30 con 0x7f07ec0047c0 2023-10-16T12:34:17.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.115+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08000b30c0 con 0x7f07ec0047c0 2023-10-16T12:34:17.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.116+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08000b32b0 con 0x7f07ec0047c0 2023-10-16T12:34:17.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.117+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f07f001c560 con 0x7f07ec0047c0 2023-10-16T12:34:17.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.118+0000 7f07f6ffd640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f07e006aea0 0x7f07e006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.118+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f07f0089bc0 con 0x7f07ec0047c0 2023-10-16T12:34:17.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.119+0000 7f07f7fff640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f07e006aea0 0x7f07e006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.120+0000 7f07f7fff640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f07e006aea0 0x7f07e006d360 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f07e8034000 tx=0x7f07e8009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:17.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.122+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08000b32b0 con 0x7f07ec0047c0 2023-10-16T12:34:17.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.318+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08000b32b0 con 0x7f07ec0047c0 2023-10-16T12:34:17.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.319+0000 7f07f6ffd640 1 -- 172.21.15.60:0/3944415381 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08000b32b0 con 0x7f07ec0047c0 2023-10-16T12:34:17.321 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.321+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f07e006aea0 msgr2=0x7f07e006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.321+0000 7f0804802640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f07e006aea0 0x7f07e006d360 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f07e8034000 tx=0x7f07e8009030 comp rx=0 tx=0).stop 2023-10-16T12:34:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.321+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 msgr2=0x7f0800158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.321+0000 7f0804802640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f0800158ef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f07f00107c0 tx=0x7f07f0010c80 comp rx=0 tx=0).stop 2023-10-16T12:34:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 shutdown_connections 2023-10-16T12:34:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f07e006aea0 0x7f07e006d360 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08000b26a0 0x7f08000c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08000b2220 0x7f08000c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 --2- 172.21.15.60:0/3944415381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07ec0047c0 0x7f0800158ef0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 >> 172.21.15.60:0/3944415381 conn(0x7f08000ce9b0 msgr2=0x7f0800159fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 shutdown_connections 2023-10-16T12:34:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.322+0000 7f0804802640 1 -- 172.21.15.60:0/3944415381 wait complete. 2023-10-16T12:34:17.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:17.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.571+0000 7f7d15c1a640 1 Processor -- start 2023-10-16T12:34:17.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.571+0000 7f7d15c1a640 1 -- start start 2023-10-16T12:34:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.572+0000 7f7d15c1a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7d1006fb50 0x7f7d100cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.572+0000 7f7d15c1a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d1006c9e0 0x7f7d100ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.572+0000 7f7d15c1a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d100ca560 con 0x7f7d100c9060 2023-10-16T12:34:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.572+0000 7f7d15c1a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100c9950 con 0x7f7d1006c9e0 2023-10-16T12:34:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.572+0000 7f7d15c1a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100c9c30 con 0x7f7d1006fb50 2023-10-16T12:34:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d1006c9e0 0x7f7d100ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d1006c9e0 0x7f7d100ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54482/0 (socket says 172.21.15.60:54482) 2023-10-16T12:34:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 -- 172.21.15.60:0/1145155931 learned_addr learned my addr 172.21.15.60:0/1145155931 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:17.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d0ffff640 1 --2- 172.21.15.60:0/1145155931 >> v2:172.21.15.144:3300/0 conn(0x7f7d1006fb50 0x7f7d100cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 -- 172.21.15.60:0/1145155931 >> v2:172.21.15.144:3300/0 conn(0x7f7d1006fb50 msgr2=0x7f7d100cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 --2- 172.21.15.60:0/1145155931 >> v2:172.21.15.144:3300/0 conn(0x7f7d1006fb50 0x7f7d100cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 -- 172.21.15.60:0/1145155931 >> v1:172.21.15.144:6789/0 conn(0x7f7d100c9060 legacy=0x7f7d1006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.573+0000 7f7d15419640 1 -- 172.21.15.60:0/1145155931 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d100c9f10 con 0x7f7d1006c9e0 2023-10-16T12:34:17.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.575+0000 7f7d15419640 1 --2- 172.21.15.60:0/1145155931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d1006c9e0 0x7f7d100ccde0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d10164cb0 tx=0x7f7d0802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa9cdb60b5c4993 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:17.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.575+0000 7f7d0f7fe640 1 -- 172.21.15.60:0/1145155931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0802ca80 con 0x7f7d1006c9e0 2023-10-16T12:34:17.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.575+0000 7f7d0f7fe640 1 -- 172.21.15.60:0/1145155931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0802cbe0 con 0x7f7d1006c9e0 2023-10-16T12:34:17.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.575+0000 7f7d0f7fe640 1 -- 172.21.15.60:0/1145155931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d080343f0 con 0x7f7d1006c9e0 2023-10-16T12:34:17.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1145155931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d1006c9e0 msgr2=0x7f7d100ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1145155931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d1006c9e0 0x7f7d100ccde0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d10164cb0 tx=0x7f7d0802bee0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1145155931 shutdown_connections 2023-10-16T12:34:17.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1145155931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d1006c9e0 0x7f7d100ccde0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1145155931 >> v2:172.21.15.144:3300/0 conn(0x7f7d1006fb50 0x7f7d100cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1145155931 >> 172.21.15.60:0/1145155931 conn(0x7f7d100ce6a0 msgr2=0x7f7d100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:17.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1145155931 shutdown_connections 2023-10-16T12:34:17.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.576+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1145155931 wait complete. 2023-10-16T12:34:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.578+0000 7f7d15c1a640 1 Processor -- start 2023-10-16T12:34:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.579+0000 7f7d15c1a640 1 -- start start 2023-10-16T12:34:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15c1a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1006c9e0 0x7f7d100c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15c1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d1006fb50 0x7f7d101562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15c1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 0x7f7d101636e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15c1a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1015a830 con 0x7f7d100c9060 2023-10-16T12:34:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15c1a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d100c77d0 con 0x7f7d1006c9e0 2023-10-16T12:34:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15c1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10156e00 con 0x7f7d1006fb50 2023-10-16T12:34:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15419640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 0x7f7d101636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15419640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 0x7f7d101636e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54488/0 (socket says 172.21.15.60:54488) 2023-10-16T12:34:17.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d15419640 1 -- 172.21.15.60:0/1397485763 learned_addr learned my addr 172.21.15.60:0/1397485763 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:17.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d0ffff640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d1006fb50 0x7f7d101562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.580+0000 7f7d14c18640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1006c9e0 0x7f7d100c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d15419640 1 -- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d1006fb50 msgr2=0x7f7d101562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d15419640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d1006fb50 0x7f7d101562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d15419640 1 -- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1006c9e0 msgr2=0x7f7d100c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d15419640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1006c9e0 0x7f7d100c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d15419640 1 -- 172.21.15.60:0/1397485763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08009b30 con 0x7f7d100c9060 2023-10-16T12:34:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d0ffff640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d1006fb50 0x7f7d101562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d14c18640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1006c9e0 0x7f7d100c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:17.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d15419640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 0x7f7d101636e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009f40 tx=0x7f7d08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:17.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.581+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08009510 con 0x7f7d100c9060 2023-10-16T12:34:17.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.582+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0803e450 con 0x7f7d100c9060 2023-10-16T12:34:17.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.582+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0803e7e0 con 0x7f7d100c9060 2023-10-16T12:34:17.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.582+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d10159e90 con 0x7f7d100c9060 2023-10-16T12:34:17.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.582+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1015a450 con 0x7f7d100c9060 2023-10-16T12:34:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.583+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d1004e9b0 con 0x7f7d100c9060 2023-10-16T12:34:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.584+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7d0802cd50 con 0x7f7d100c9060 2023-10-16T12:34:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.585+0000 7f7d0d7fa640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cf406aea0 0x7f7cf406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:17.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.585+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f7d080ac440 con 0x7f7d100c9060 2023-10-16T12:34:17.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.586+0000 7f7d14c18640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cf406aea0 0x7f7cf406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:17.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.587+0000 7f7d14c18640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cf406aea0 0x7f7cf406d360 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400a830 tx=0x7f7d04008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:17.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.592+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d08077640 con 0x7f7d100c9060 2023-10-16T12:34:17.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.788+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d1015a5e0 con 0x7f7d100c9060 2023-10-16T12:34:17.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.789+0000 7f7d0d7fa640 1 -- 172.21.15.60:0/1397485763 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d0807ac60 con 0x7f7d100c9060 2023-10-16T12:34:17.792 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:17.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cf406aea0 msgr2=0x7f7cf406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cf406aea0 0x7f7cf406d360 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400a830 tx=0x7f7d04008040 comp rx=0 tx=0).stop 2023-10-16T12:34:17.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 msgr2=0x7f7d101636e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:17.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 0x7f7d101636e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009f40 tx=0x7f7d08004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:17.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 shutdown_connections 2023-10-16T12:34:17.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f7cf406aea0 0x7f7cf406d360 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d1006fb50 0x7f7d101562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d100c9060 0x7f7d101636e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 --2- 172.21.15.60:0/1397485763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d1006c9e0 0x7f7d100c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:17.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.792+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 >> 172.21.15.60:0/1397485763 conn(0x7f7d100ce6a0 msgr2=0x7f7d100cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:17.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.793+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 shutdown_connections 2023-10-16T12:34:17.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:17.793+0000 7f7d15c1a640 1 -- 172.21.15.60:0/1397485763 wait complete. 2023-10-16T12:34:17.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:18.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.036+0000 7f0438402640 1 Processor -- start 2023-10-16T12:34:18.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.037+0000 7f0438402640 1 -- start start 2023-10-16T12:34:18.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.038+0000 7f0438402640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f04300661c0 0x7f043006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.038+0000 7f0438402640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043015eff0 con 0x7f04300cb1c0 2023-10-16T12:34:18.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.038+0000 7f0438402640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043015ecc0 con 0x7f043015af80 2023-10-16T12:34:18.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.038+0000 7f0438402640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04300d0930 con 0x7f04300661c0 2023-10-16T12:34:18.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.039+0000 7f043599d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f04300cb1c0 0x7f0430065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50466/0 (socket says 172.21.15.60:50466) 2023-10-16T12:34:18.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.039+0000 7f043599d640 1 -- 172.21.15.60:0/2108411300 learned_addr learned my addr 172.21.15.60:0/2108411300 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:18.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.039+0000 7f043699f640 1 --2- 172.21.15.60:0/2108411300 >> v2:172.21.15.144:3300/0 conn(0x7f04300661c0 0x7f043006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.039+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150655872 0 0) 0x7f043015eff0 con 0x7f04300cb1c0 2023-10-16T12:34:18.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.039+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041c003520 con 0x7f04300cb1c0 2023-10-16T12:34:18.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911178905 0 0) 0x7f041c003520 con 0x7f04300cb1c0 2023-10-16T12:34:18.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 >> v2:172.21.15.144:3300/0 conn(0x7f04300661c0 msgr2=0x7f043006faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:18.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 --2- 172.21.15.60:0/2108411300 >> v2:172.21.15.144:3300/0 conn(0x7f04300661c0 0x7f043006faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 >> v1:172.21.15.144:6789/0 conn(0x7f043015af80 legacy=0x7f043015f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043015b360 con 0x7f04300cb1c0 2023-10-16T12:34:18.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0420001670 con 0x7f04300cb1c0 2023-10-16T12:34:18.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 >> v1:172.21.15.60:6789/0 conn(0x7f04300cb1c0 legacy=0x7f0430065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f041c0039c0 0x7f041c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f041c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f041c0051d0 0x7f041c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04300d0930 con 0x7f041c0047c0 2023-10-16T12:34:18.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c00c120 con 0x7f041c0039c0 2023-10-16T12:34:18.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c00c2e0 con 0x7f041c0051d0 2023-10-16T12:34:18.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043599d640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f041c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.040+0000 7f043619e640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f041c0039c0 0x7f041c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043699f640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f041c0051d0 0x7f041c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043599d640 1 -- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f041c0051d0 msgr2=0x7f041c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:18.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043599d640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f041c0051d0 0x7f041c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043599d640 1 -- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f041c0039c0 msgr2=0x7f041c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:18.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043599d640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f041c0039c0 0x7f041c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043599d640 1 -- 172.21.15.60:0/2108411300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c003520 con 0x7f041c0047c0 2023-10-16T12:34:18.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043699f640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f041c0051d0 0x7f041c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:18.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043619e640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f041c0039c0 0x7f041c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:18.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.041+0000 7f043699f640 1 -- 172.21.15.60:0/2108411300 reap_dead start 2023-10-16T12:34:18.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.042+0000 7f043599d640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f041c004c40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0420001eb0 tx=0x7f041c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c93c29d9f642c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:18.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.042+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420005ad0 con 0x7f041c0047c0 2023-10-16T12:34:18.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0420005c30 con 0x7f041c0047c0 2023-10-16T12:34:18.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f043519c640 1 -- 172.21.15.60:0/2108411300 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04200072b0 con 0x7f041c0047c0 2023-10-16T12:34:18.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f0438402640 1 -- 172.21.15.60:0/2108411300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 msgr2=0x7f041c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f0438402640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f041c004c40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0420001eb0 tx=0x7f041c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f0438402640 1 -- 172.21.15.60:0/2108411300 shutdown_connections 2023-10-16T12:34:18.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f0438402640 1 --2- 172.21.15.60:0/2108411300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f041c004c40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.043+0000 7f0438402640 1 -- 172.21.15.60:0/2108411300 >> 172.21.15.60:0/2108411300 conn(0x7f04300d0ab0 msgr2=0x7f04300d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:18.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.044+0000 7f0438402640 1 -- 172.21.15.60:0/2108411300 shutdown_connections 2023-10-16T12:34:18.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.044+0000 7f0438402640 1 -- 172.21.15.60:0/2108411300 wait complete. 2023-10-16T12:34:18.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.045+0000 7f0438402640 1 Processor -- start 2023-10-16T12:34:18.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.045+0000 7f0438402640 1 -- start start 2023-10-16T12:34:18.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.046+0000 7f0438402640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f04300ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.046+0000 7f0438402640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04300aee90 0x7f04300b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.046+0000 7f0438402640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04300b4580 0x7f043014f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.046+0000 7f0438402640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04301585c0 con 0x7f041c0047c0 2023-10-16T12:34:18.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.046+0000 7f0438402640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04300ca100 con 0x7f04300b4580 2023-10-16T12:34:18.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.046+0000 7f0438402640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04300caad0 con 0x7f04300aee90 2023-10-16T12:34:18.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f04300ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f04300ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54494/0 (socket says 172.21.15.60:54494) 2023-10-16T12:34:18.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 -- 172.21.15.60:0/691637979 learned_addr learned my addr 172.21.15.60:0/691637979 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:18.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043599d640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04300aee90 0x7f04300b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043699f640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04300b4580 0x7f043014f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 -- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04300aee90 msgr2=0x7f04300b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04300aee90 0x7f04300b63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 -- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04300b4580 msgr2=0x7f043014f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04300b4580 0x7f043014f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.047+0000 7f043619e640 1 -- 172.21.15.60:0/691637979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04300c9510 con 0x7f041c0047c0 2023-10-16T12:34:18.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.048+0000 7f043619e640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f04300ca590 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f042c00e390 tx=0x7f042c00e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:18.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.048+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0052e0 con 0x7f041c0047c0 2023-10-16T12:34:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.048+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c005440 con 0x7f041c0047c0 2023-10-16T12:34:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.048+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c003810 con 0x7f041c0047c0 2023-10-16T12:34:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.048+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04300b6920 con 0x7f041c0047c0 2023-10-16T12:34:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.048+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04300c9ab0 con 0x7f041c0047c0 2023-10-16T12:34:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.049+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043004e160 con 0x7f041c0047c0 2023-10-16T12:34:18.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.050+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f042c003970 con 0x7f041c0047c0 2023-10-16T12:34:18.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.051+0000 7f0426ffd640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f041406ae70 0x7f041406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.051+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f042c088e60 con 0x7f041c0047c0 2023-10-16T12:34:18.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.052+0000 7f043599d640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f041406ae70 0x7f041406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.053+0000 7f043599d640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f041406ae70 0x7f041406d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0420034000 tx=0x7f0420009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:18.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.056+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f042c053ec0 con 0x7f041c0047c0 2023-10-16T12:34:18.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.252+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f043015e890 con 0x7f041c0047c0 2023-10-16T12:34:18.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.253+0000 7f0426ffd640 1 -- 172.21.15.60:0/691637979 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f042c019020 con 0x7f041c0047c0 2023-10-16T12:34:18.255 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:18.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.255+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f041406ae70 msgr2=0x7f041406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.255+0000 7f0438402640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f041406ae70 0x7f041406d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0420034000 tx=0x7f0420009030 comp rx=0 tx=0).stop 2023-10-16T12:34:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 msgr2=0x7f04300ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f04300ca590 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f042c00e390 tx=0x7f042c00e850 comp rx=0 tx=0).stop 2023-10-16T12:34:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 shutdown_connections 2023-10-16T12:34:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f041406ae70 0x7f041406d330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f04300aee90 0x7f04300b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04300b4580 0x7f043014f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 --2- 172.21.15.60:0/691637979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f041c0047c0 0x7f04300ca590 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 >> 172.21.15.60:0/691637979 conn(0x7f04300d0ab0 msgr2=0x7f04300acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:18.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 shutdown_connections 2023-10-16T12:34:18.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.256+0000 7f0438402640 1 -- 172.21.15.60:0/691637979 wait complete. 2023-10-16T12:34:18.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:34:18.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.503+0000 7fd5f1935640 1 Processor -- start 2023-10-16T12:34:18.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.504+0000 7fd5f1935640 1 -- start start 2023-10-16T12:34:18.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f1935640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd5ec070950 0x7fd5ec06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f1935640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ec0ca3a0 con 0x7fd5ec1573e0 2023-10-16T12:34:18.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f1935640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ec0c9d70 con 0x7fd5ec0ca830 2023-10-16T12:34:18.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f1935640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec1577c0 con 0x7fd5ec070950 2023-10-16T12:34:18.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f0933640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd5ec1573e0 0x7fd5ec159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50468/0 (socket says 172.21.15.60:50468) 2023-10-16T12:34:18.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f0933640 1 -- 172.21.15.60:0/740499963 learned_addr learned my addr 172.21.15.60:0/740499963 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:18.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.505+0000 7fd5f1134640 1 --2- 172.21.15.60:0/740499963 >> v2:172.21.15.60:3300/0 conn(0x7fd5ec070950 0x7fd5ec06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.506+0000 7fd5eb7fe640 1 -- 172.21.15.60:0/740499963 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029599114 0 0) 0x7fd5ec0ca3a0 con 0x7fd5ec1573e0 2023-10-16T12:34:18.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.506+0000 7fd5eb7fe640 1 -- 172.21.15.60:0/740499963 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5dc005730 con 0x7fd5ec1573e0 2023-10-16T12:34:18.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.506+0000 7fd5f1134640 1 -- 172.21.15.60:0/740499963 >> v1:172.21.15.144:6789/0 conn(0x7fd5ec0ca830 legacy=0x7fd5ec070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:18.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.506+0000 7fd5f1134640 1 -- 172.21.15.60:0/740499963 >> v1:172.21.15.60:6789/0 conn(0x7fd5ec1573e0 legacy=0x7fd5ec159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.506+0000 7fd5f1134640 1 -- 172.21.15.60:0/740499963 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec15a9a0 con 0x7fd5ec070950 2023-10-16T12:34:18.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.507+0000 7fd5f1134640 1 --2- 172.21.15.60:0/740499963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec06bdc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009dc0 tx=0x7fd5e402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a453ffa67fcf54 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:18.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.507+0000 7fd5eb7fe640 1 -- 172.21.15.60:0/740499963 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e402cc40 con 0x7fd5ec070950 2023-10-16T12:34:18.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.507+0000 7fd5eb7fe640 1 -- 172.21.15.60:0/740499963 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e402cda0 con 0x7fd5ec070950 2023-10-16T12:34:18.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.507+0000 7fd5eb7fe640 1 -- 172.21.15.60:0/740499963 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4034660 con 0x7fd5ec070950 2023-10-16T12:34:18.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 -- 172.21.15.60:0/740499963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 msgr2=0x7fd5ec06bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 --2- 172.21.15.60:0/740499963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec06bdc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009dc0 tx=0x7fd5e402be90 comp rx=0 tx=0).stop 2023-10-16T12:34:18.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 -- 172.21.15.60:0/740499963 shutdown_connections 2023-10-16T12:34:18.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 --2- 172.21.15.60:0/740499963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec06bdc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 -- 172.21.15.60:0/740499963 >> 172.21.15.60:0/740499963 conn(0x7fd5ec0cc230 msgr2=0x7fd5ec0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:18.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 -- 172.21.15.60:0/740499963 shutdown_connections 2023-10-16T12:34:18.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.508+0000 7fd5f1935640 1 -- 172.21.15.60:0/740499963 wait complete. 2023-10-16T12:34:18.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.510+0000 7fd5f1935640 1 Processor -- start 2023-10-16T12:34:18.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.510+0000 7fd5f1935640 1 -- start start 2023-10-16T12:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.511+0000 7fd5f1935640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec1f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.511+0000 7fd5f1935640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5ec0ca830 0x7fd5ec15bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.511+0000 7fd5f1935640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5ec1573e0 0x7fd5ec0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.511+0000 7fd5f1935640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec1edd80 con 0x7fd5ec070950 2023-10-16T12:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.511+0000 7fd5f1935640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec15c230 con 0x7fd5ec1573e0 2023-10-16T12:34:18.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.511+0000 7fd5f1935640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c6970 con 0x7fd5ec0ca830 2023-10-16T12:34:18.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec1f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec1f1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54506/0 (socket says 172.21.15.60:54506) 2023-10-16T12:34:18.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 -- 172.21.15.60:0/1180151085 learned_addr learned my addr 172.21.15.60:0/1180151085 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:18.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5ebfff640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5ec0ca830 0x7fd5ec15bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 -- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5ec0ca830 msgr2=0x7fd5ec15bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f1134640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5ec1573e0 0x7fd5ec0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5ec0ca830 0x7fd5ec15bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 -- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5ec1573e0 msgr2=0x7fd5ec0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5ec1573e0 0x7fd5ec0b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5f0933640 1 -- 172.21.15.60:0/1180151085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e4009a70 con 0x7fd5ec070950 2023-10-16T12:34:18.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.512+0000 7fd5ebfff640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5ec0ca830 0x7fd5ec15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:18.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.513+0000 7fd5f0933640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec1f1ef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c500 tx=0x7fd5e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:18.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.513+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0005ce0 con 0x7fd5ec070950 2023-10-16T12:34:18.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.513+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e0005e40 con 0x7fd5ec070950 2023-10-16T12:34:18.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.513+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0007330 con 0x7fd5ec070950 2023-10-16T12:34:18.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.513+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec0c6c50 con 0x7fd5ec070950 2023-10-16T12:34:18.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.513+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec0c7210 con 0x7fd5ec070950 2023-10-16T12:34:18.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.514+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec0cacc0 con 0x7fd5ec070950 2023-10-16T12:34:18.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.515+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd5e0007490 con 0x7fd5ec070950 2023-10-16T12:34:18.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.516+0000 7fd5e97fa640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd5d006ae70 0x7fd5d006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:18.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.516+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fd5e0088d10 con 0x7fd5ec070950 2023-10-16T12:34:18.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.517+0000 7fd5ebfff640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd5d006ae70 0x7fd5d006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:18.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.518+0000 7fd5ebfff640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd5d006ae70 0x7fd5d006d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec15c800 tx=0x7fd5d8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:18.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.520+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5ec0cacc0 con 0x7fd5ec070950 2023-10-16T12:34:18.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.737+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd5ec0cacc0 con 0x7fd5ec070950 2023-10-16T12:34:18.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.738+0000 7fd5e97fa640 1 -- 172.21.15.60:0/1180151085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd5ec0cacc0 con 0x7fd5ec070950 2023-10-16T12:34:18.741 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2023-10-16T12:34:18.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.741+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd5d006ae70 msgr2=0x7fd5d006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd5d006ae70 0x7fd5d006d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec15c800 tx=0x7fd5d8008040 comp rx=0 tx=0).stop 2023-10-16T12:34:18.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 msgr2=0x7fd5ec1f1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:18.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec1f1ef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c500 tx=0x7fd5e000c9c0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 shutdown_connections 2023-10-16T12:34:18.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd5d006ae70 0x7fd5d006d330 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5ec0ca830 0x7fd5ec15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5ec1573e0 0x7fd5ec0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 --2- 172.21.15.60:0/1180151085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5ec070950 0x7fd5ec1f1ef0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:18.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 >> 172.21.15.60:0/1180151085 conn(0x7fd5ec0cc230 msgr2=0x7fd5ec156c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:18.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 shutdown_connections 2023-10-16T12:34:18.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:18.742+0000 7fd5f1935640 1 -- 172.21.15.60:0/1180151085 wait complete. 2023-10-16T12:34:18.763 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-16T12:34:18.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:18.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-16T12:34:18.874 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T12:34:18.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:18.875 DEBUG:teuthology.orchestra.run.smithi144:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-16T12:34:18.929 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:34:18.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:34:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.162+0000 7f1cd8e85640 1 Processor -- start 2023-10-16T12:34:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.162+0000 7f1cd8e85640 1 -- start start 2023-10-16T12:34:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.164+0000 7f1cd8e85640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cd415c9f0 0x7f1cd40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.164+0000 7f1cd8e85640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.164+0000 7f1cd8e85640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd4160a90 con 0x7f1cd4070f20 2023-10-16T12:34:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.164+0000 7f1cd8e85640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40cdec0 con 0x7f1cd415c9f0 2023-10-16T12:34:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.164+0000 7f1cd8e85640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40cd900 con 0x7f1cd40c9230 2023-10-16T12:34:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.165+0000 7f1cd37fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cd415c9f0 0x7f1cd40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.165+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.165+0000 7f1cd37fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cd415c9f0 0x7f1cd40c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:38194/0 (socket says 172.21.15.60:38194) 2023-10-16T12:34:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.165+0000 7f1cd37fe640 1 -- 172.21.15.60:0/3191675365 learned_addr learned my addr 172.21.15.60:0/3191675365 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.166+0000 7f1ccbfff640 1 -- 172.21.15.60:0/3191675365 >> v2:172.21.15.144:3300/0 conn(0x7f1cd415c9f0 msgr2=0x7f1cd40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.166+0000 7f1ccbfff640 1 --2- 172.21.15.60:0/3191675365 >> v2:172.21.15.144:3300/0 conn(0x7f1cd415c9f0 0x7f1cd40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.166+0000 7f1ccbfff640 1 -- 172.21.15.60:0/3191675365 >> v1:172.21.15.144:6789/0 conn(0x7f1cd4070f20 legacy=0x7f1cd40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.166+0000 7f1ccbfff640 1 -- 172.21.15.60:0/3191675365 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40cdbe0 con 0x7f1cd40c9230 2023-10-16T12:34:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.167+0000 7f1ccbfff640 1 --2- 172.21.15.60:0/3191675365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd40709e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0097a0 tx=0x7f1cbc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37bd04f8d62f9773 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.168+0000 7f1cd2ffd640 1 -- 172.21.15.60:0/3191675365 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc02d040 con 0x7f1cd40c9230 2023-10-16T12:34:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.168+0000 7f1cd2ffd640 1 -- 172.21.15.60:0/3191675365 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc031810 con 0x7f1cd40c9230 2023-10-16T12:34:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.168+0000 7f1cd2ffd640 1 -- 172.21.15.60:0/3191675365 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc031ba0 con 0x7f1cd40c9230 2023-10-16T12:34:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.168+0000 7f1cd8e85640 1 -- 172.21.15.60:0/3191675365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 msgr2=0x7f1cd40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.168+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/3191675365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd40709e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0097a0 tx=0x7f1cbc02b8b0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.169+0000 7f1cd8e85640 1 -- 172.21.15.60:0/3191675365 shutdown_connections 2023-10-16T12:34:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.169+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/3191675365 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd40709e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.169+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/3191675365 >> v2:172.21.15.144:3300/0 conn(0x7f1cd415c9f0 0x7f1cd40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.169+0000 7f1cd8e85640 1 -- 172.21.15.60:0/3191675365 >> 172.21.15.60:0/3191675365 conn(0x7f1cd40ce6a0 msgr2=0x7f1cd40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.169+0000 7f1cd8e85640 1 -- 172.21.15.60:0/3191675365 shutdown_connections 2023-10-16T12:34:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.169+0000 7f1cd8e85640 1 -- 172.21.15.60:0/3191675365 wait complete. 2023-10-16T12:34:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.171+0000 7f1cd8e85640 1 Processor -- start 2023-10-16T12:34:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.171+0000 7f1cd8e85640 1 -- start start 2023-10-16T12:34:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.172+0000 7f1cd8e85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 0x7f1cd40c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd8e85640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd4158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd8e85640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cd415c9f0 0x7f1cd4158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd8e85640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40c4f10 con 0x7f1cd4070f20 2023-10-16T12:34:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd8e85640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40c4300 con 0x7f1cd415c9f0 2023-10-16T12:34:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd8e85640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40c45e0 con 0x7f1cd40c9230 2023-10-16T12:34:19.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 0x7f1cd40c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 0x7f1cd40c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54706/0 (socket says 172.21.15.60:54706) 2023-10-16T12:34:19.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 -- 172.21.15.60:0/2054544973 learned_addr learned my addr 172.21.15.60:0/2054544973 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:19.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1ccbfff640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd4158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 -- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 msgr2=0x7f1cd4158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd4158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd3fff640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cd415c9f0 0x7f1cd4158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 -- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cd415c9f0 msgr2=0x7f1cd4158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cd415c9f0 0x7f1cd4158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd37fe640 1 -- 172.21.15.60:0/2054544973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc009400 con 0x7f1cd4070f20 2023-10-16T12:34:19.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.173+0000 7f1cd3fff640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cd415c9f0 0x7f1cd4158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:19.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.174+0000 7f1cd37fe640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 0x7f1cd40c9c90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009fd0 tx=0x7f1cc400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:19.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.174+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc400c940 con 0x7f1cd4070f20 2023-10-16T12:34:19.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.174+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc400caa0 con 0x7f1cd4070f20 2023-10-16T12:34:19.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.174+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc400ce30 con 0x7f1cd4070f20 2023-10-16T12:34:19.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.174+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd40c48c0 con 0x7f1cd4070f20 2023-10-16T12:34:19.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.174+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd40ca1d0 con 0x7f1cd4070f20 2023-10-16T12:34:19.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.175+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd40ca3c0 con 0x7f1cd4070f20 2023-10-16T12:34:19.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.176+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1cc4013470 con 0x7f1cd4070f20 2023-10-16T12:34:19.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.178+0000 7f1cd0ff9640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1cb406aea0 0x7f1cb406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.178+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f1cc4089810 con 0x7f1cd4070f20 2023-10-16T12:34:19.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.178+0000 7f1ccbfff640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1cb406aea0 0x7f1cb406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.179+0000 7f1ccbfff640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1cb406aea0 0x7f1cb406d360 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc036000 tx=0x7f1cbc03c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:19.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.183+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cc40547c0 con 0x7f1cd4070f20 2023-10-16T12:34:19.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.420+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd4070160 con 0x7f1cd4070f20 2023-10-16T12:34:19.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.421+0000 7f1cd0ff9640 1 -- 172.21.15.60:0/2054544973 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f1cc4057de0 con 0x7f1cd4070f20 2023-10-16T12:34:19.424 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:19.424 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:34:19.425 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:34:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1cb406aea0 msgr2=0x7f1cb406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1cb406aea0 0x7f1cb406d360 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc036000 tx=0x7f1cbc03c040 comp rx=0 tx=0).stop 2023-10-16T12:34:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 msgr2=0x7f1cd40c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 0x7f1cd40c9c90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009fd0 tx=0x7f1cc400ed40 comp rx=0 tx=0).stop 2023-10-16T12:34:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 shutdown_connections 2023-10-16T12:34:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1cb406aea0 0x7f1cb406d360 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cd40c9230 0x7f1cd4158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cd415c9f0 0x7f1cd4158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 --2- 172.21.15.60:0/2054544973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cd4070f20 0x7f1cd40c9c90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 >> 172.21.15.60:0/2054544973 conn(0x7f1cd40ce6a0 msgr2=0x7f1cd40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 shutdown_connections 2023-10-16T12:34:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.424+0000 7f1cd8e85640 1 -- 172.21.15.60:0/2054544973 wait complete. 2023-10-16T12:34:19.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:19.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.672+0000 7f250ea59640 1 Processor -- start 2023-10-16T12:34:19.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.672+0000 7f250ea59640 1 -- start start 2023-10-16T12:34:19.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.674+0000 7f250ea59640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f250806fb50 0x7f25080cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.674+0000 7f250ea59640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f250806c9e0 0x7f25080ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.674+0000 7f250ea59640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25080ca560 con 0x7f25080c9060 2023-10-16T12:34:19.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.674+0000 7f250ea59640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080c9950 con 0x7f250806c9e0 2023-10-16T12:34:19.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.674+0000 7f250ea59640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080c9c30 con 0x7f250806fb50 2023-10-16T12:34:19.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f250806c9e0 0x7f25080ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f250806c9e0 0x7f25080ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54720/0 (socket says 172.21.15.60:54720) 2023-10-16T12:34:19.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 -- 172.21.15.60:0/3359864001 learned_addr learned my addr 172.21.15.60:0/3359864001 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:19.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250d256640 1 --2- 172.21.15.60:0/3359864001 >> v2:172.21.15.144:3300/0 conn(0x7f250806fb50 0x7f25080cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 -- 172.21.15.60:0/3359864001 >> v2:172.21.15.144:3300/0 conn(0x7f250806fb50 msgr2=0x7f25080cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:19.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 --2- 172.21.15.60:0/3359864001 >> v2:172.21.15.144:3300/0 conn(0x7f250806fb50 0x7f25080cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 -- 172.21.15.60:0/3359864001 >> v1:172.21.15.144:6789/0 conn(0x7f25080c9060 legacy=0x7f250806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:19.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.675+0000 7f250e258640 1 -- 172.21.15.60:0/3359864001 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080c9f10 con 0x7f250806c9e0 2023-10-16T12:34:19.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.676+0000 7f250e258640 1 --2- 172.21.15.60:0/3359864001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f25080ccde0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2508164cb0 tx=0x7f250002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d30c743a618d2a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:19.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.677+0000 7f250ca55640 1 -- 172.21.15.60:0/3359864001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250002cb20 con 0x7f250806c9e0 2023-10-16T12:34:19.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.677+0000 7f250ca55640 1 -- 172.21.15.60:0/3359864001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250002cc80 con 0x7f250806c9e0 2023-10-16T12:34:19.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.677+0000 7f250ca55640 1 -- 172.21.15.60:0/3359864001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000343f0 con 0x7f250806c9e0 2023-10-16T12:34:19.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 -- 172.21.15.60:0/3359864001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 msgr2=0x7f25080ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 --2- 172.21.15.60:0/3359864001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f25080ccde0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2508164cb0 tx=0x7f250002bf80 comp rx=0 tx=0).stop 2023-10-16T12:34:19.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 -- 172.21.15.60:0/3359864001 shutdown_connections 2023-10-16T12:34:19.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 --2- 172.21.15.60:0/3359864001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f25080ccde0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 --2- 172.21.15.60:0/3359864001 >> v2:172.21.15.144:3300/0 conn(0x7f250806fb50 0x7f25080cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 -- 172.21.15.60:0/3359864001 >> 172.21.15.60:0/3359864001 conn(0x7f25080ce6a0 msgr2=0x7f25080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:19.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 -- 172.21.15.60:0/3359864001 shutdown_connections 2023-10-16T12:34:19.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.678+0000 7f250ea59640 1 -- 172.21.15.60:0/3359864001 wait complete. 2023-10-16T12:34:19.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.680+0000 7f250ea59640 1 Processor -- start 2023-10-16T12:34:19.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.680+0000 7f250ea59640 1 -- start start 2023-10-16T12:34:19.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250ea59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f2508163c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250ea59640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250806fb50 0x7f25080c3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250ea59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25080c9060 0x7f25080c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250ea59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508162910 con 0x7f250806c9e0 2023-10-16T12:34:19.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250ea59640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508162b60 con 0x7f250806fb50 2023-10-16T12:34:19.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250ea59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2508162e40 con 0x7f25080c9060 2023-10-16T12:34:19.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f2508163c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f2508163c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54722/0 (socket says 172.21.15.60:54722) 2023-10-16T12:34:19.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 -- 172.21.15.60:0/3164245233 learned_addr learned my addr 172.21.15.60:0/3164245233 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:19.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250d256640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250806fb50 0x7f25080c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 -- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25080c9060 msgr2=0x7f25080c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25080c9060 0x7f25080c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 -- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250806fb50 msgr2=0x7f25080c3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250806fb50 0x7f25080c3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.681+0000 7f250da57640 1 -- 172.21.15.60:0/3164245233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500009b30 con 0x7f250806c9e0 2023-10-16T12:34:19.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.682+0000 7f250da57640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f2508163c00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f250400c400 tx=0x7f250400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:19.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.682+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040052d0 con 0x7f250806c9e0 2023-10-16T12:34:19.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.682+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504005430 con 0x7f250806c9e0 2023-10-16T12:34:19.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.682+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2508163120 con 0x7f250806c9e0 2023-10-16T12:34:19.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.682+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504003830 con 0x7f250806c9e0 2023-10-16T12:34:19.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.682+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25081636e0 con 0x7f250806c9e0 2023-10-16T12:34:19.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.683+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250804e9b0 con 0x7f250806c9e0 2023-10-16T12:34:19.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.684+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2504003990 con 0x7f250806c9e0 2023-10-16T12:34:19.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.685+0000 7f24fe7fc640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f24ec06aea0 0x7f24ec06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:19.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.685+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f2504088b90 con 0x7f250806c9e0 2023-10-16T12:34:19.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.686+0000 7f250d256640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f24ec06aea0 0x7f24ec06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:19.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.687+0000 7f250d256640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f24ec06aea0 0x7f24ec06d360 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25080ad6a0 tx=0x7f24f8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:19.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.690+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2504053be0 con 0x7f250806c9e0 2023-10-16T12:34:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.892+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25080b4620 con 0x7f250806c9e0 2023-10-16T12:34:19.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.894+0000 7f24fe7fc640 1 -- 172.21.15.60:0/3164245233 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16714 (secure 0 0 0) 0x7f2504057200 con 0x7f250806c9e0 2023-10-16T12:34:19.897 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:19.898 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:11.506118+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:19.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f24ec06aea0 msgr2=0x7f24ec06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f24ec06aea0 0x7f24ec06d360 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25080ad6a0 tx=0x7f24f8008040 comp rx=0 tx=0).stop 2023-10-16T12:34:19.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 msgr2=0x7f2508163c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:19.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f2508163c00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f250400c400 tx=0x7f250400c8c0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 shutdown_connections 2023-10-16T12:34:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f24ec06aea0 0x7f24ec06d360 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25080c9060 0x7f25080c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f250806fb50 0x7f25080c3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 --2- 172.21.15.60:0/3164245233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250806c9e0 0x7f2508163c00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:19.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 >> 172.21.15.60:0/3164245233 conn(0x7f25080ce6a0 msgr2=0x7f250815fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:19.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.897+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 shutdown_connections 2023-10-16T12:34:19.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:19.898+0000 7f250ea59640 1 -- 172.21.15.60:0/3164245233 wait complete. 2023-10-16T12:34:19.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:33:22.201537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:33:23.456249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:19.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:20.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.155+0000 7ffad7eec640 1 Processor -- start 2023-10-16T12:34:20.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.156+0000 7ffad7eec640 1 -- start start 2023-10-16T12:34:20.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.157+0000 7ffad7eec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad015ad70 0x7ffad0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.157+0000 7ffad7eec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffad00c9ca0 0x7ffad006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.157+0000 7ffad7eec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffad00ca3b0 0x7ffad00ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.157+0000 7ffad7eec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00cad80 con 0x7ffad00ca3b0 2023-10-16T12:34:20.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.157+0000 7ffad7eec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00cb030 con 0x7ffad00c9ca0 2023-10-16T12:34:20.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.157+0000 7ffad7eec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00c91c0 con 0x7ffad015ad70 2023-10-16T12:34:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffad00ca3b0 0x7ffad00ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffad00ca3b0 0x7ffad00ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54724/0 (socket says 172.21.15.60:54724) 2023-10-16T12:34:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 -- 172.21.15.60:0/1720281954 learned_addr learned my addr 172.21.15.60:0/1720281954 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad5c88640 1 --2- 172.21.15.60:0/1720281954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad015ad70 0x7ffad0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad5487640 1 --2- 172.21.15.60:0/1720281954 >> v2:172.21.15.144:3300/0 conn(0x7ffad00c9ca0 0x7ffad006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 -- 172.21.15.60:0/1720281954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad015ad70 msgr2=0x7ffad0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 --2- 172.21.15.60:0/1720281954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad015ad70 0x7ffad0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 -- 172.21.15.60:0/1720281954 >> v2:172.21.15.144:3300/0 conn(0x7ffad00c9ca0 msgr2=0x7ffad006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 --2- 172.21.15.60:0/1720281954 >> v2:172.21.15.144:3300/0 conn(0x7ffad00c9ca0 0x7ffad006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad6489640 1 -- 172.21.15.60:0/1720281954 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00c94a0 con 0x7ffad00ca3b0 2023-10-16T12:34:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.158+0000 7ffad5487640 1 --2- 172.21.15.60:0/1720281954 >> v2:172.21.15.144:3300/0 conn(0x7ffad00c9ca0 0x7ffad006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.160+0000 7ffad6489640 1 --2- 172.21.15.60:0/1720281954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00ca3b0 0x7ffad00ca840 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffad006af00 tx=0x7ffac802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d83381deddd35e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:20.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.160+0000 7ffad4c86640 1 -- 172.21.15.60:0/1720281954 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac802ca70 con 0x7ffad00ca3b0 2023-10-16T12:34:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.160+0000 7ffad4c86640 1 -- 172.21.15.60:0/1720281954 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac802cbd0 con 0x7ffad00ca3b0 2023-10-16T12:34:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.160+0000 7ffad4c86640 1 -- 172.21.15.60:0/1720281954 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80343f0 con 0x7ffad00ca3b0 2023-10-16T12:34:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 -- 172.21.15.60:0/1720281954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00ca3b0 msgr2=0x7ffad00ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 --2- 172.21.15.60:0/1720281954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00ca3b0 0x7ffad00ca840 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffad006af00 tx=0x7ffac802bed0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 -- 172.21.15.60:0/1720281954 shutdown_connections 2023-10-16T12:34:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 --2- 172.21.15.60:0/1720281954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00ca3b0 0x7ffad00ca840 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 --2- 172.21.15.60:0/1720281954 >> v2:172.21.15.144:3300/0 conn(0x7ffad00c9ca0 0x7ffad006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 --2- 172.21.15.60:0/1720281954 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad015ad70 0x7ffad0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 -- 172.21.15.60:0/1720281954 >> 172.21.15.60:0/1720281954 conn(0x7ffad00ce9b0 msgr2=0x7ffad00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 -- 172.21.15.60:0/1720281954 shutdown_connections 2023-10-16T12:34:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.161+0000 7ffad7eec640 1 -- 172.21.15.60:0/1720281954 wait complete. 2023-10-16T12:34:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.163+0000 7ffad7eec640 1 Processor -- start 2023-10-16T12:34:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.163+0000 7ffad7eec640 1 -- start start 2023-10-16T12:34:20.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad7eec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 0x7ffad00b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad7eec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad00ca3b0 0x7ffad01f5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad7eec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffad015ad70 0x7ffad00c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad7eec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00b1f80 con 0x7ffad00c9ca0 2023-10-16T12:34:20.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad7eec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00b2260 con 0x7ffad015ad70 2023-10-16T12:34:20.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad7eec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00b2540 con 0x7ffad00ca3b0 2023-10-16T12:34:20.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad5c88640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 0x7ffad00b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad5c88640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 0x7ffad00b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54740/0 (socket says 172.21.15.60:54740) 2023-10-16T12:34:20.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.164+0000 7ffad5c88640 1 -- 172.21.15.60:0/3307118000 learned_addr learned my addr 172.21.15.60:0/3307118000 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:20.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad6489640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffad015ad70 0x7ffad00c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5487640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad00ca3b0 0x7ffad01f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5c88640 1 -- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad00ca3b0 msgr2=0x7ffad01f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5c88640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad00ca3b0 0x7ffad01f5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5c88640 1 -- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffad015ad70 msgr2=0x7ffad00c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5c88640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffad015ad70 0x7ffad00c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5c88640 1 -- 172.21.15.60:0/3307118000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8009b30 con 0x7ffad00c9ca0 2023-10-16T12:34:20.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5487640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad00ca3b0 0x7ffad01f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:20.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad6489640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffad015ad70 0x7ffad00c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:20.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad5c88640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 0x7ffad00b04f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00d5f0 tx=0x7ffacc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:20.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc0125f0 con 0x7ffad00c9ca0 2023-10-16T12:34:20.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc012750 con 0x7ffad00c9ca0 2023-10-16T12:34:20.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc012b00 con 0x7ffad00c9ca0 2023-10-16T12:34:20.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.165+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad00b2820 con 0x7ffad00c9ca0 2023-10-16T12:34:20.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.166+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad01f5d80 con 0x7ffad00c9ca0 2023-10-16T12:34:20.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.167+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad01f5f40 con 0x7ffad00c9ca0 2023-10-16T12:34:20.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.168+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffacc012c60 con 0x7ffad00c9ca0 2023-10-16T12:34:20.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.169+0000 7ffac67fc640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ffab406ae70 0x7ffab406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.169+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ffacc089b50 con 0x7ffad00c9ca0 2023-10-16T12:34:20.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.169+0000 7ffad5487640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ffab406ae70 0x7ffab406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.171+0000 7ffad5487640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ffab406ae70 0x7ffab406d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffad01f66d0 tx=0x7ffac0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:20.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.173+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffacc054b90 con 0x7ffad00c9ca0 2023-10-16T12:34:20.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.370+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffad004e160 con 0x7ffad00c9ca0 2023-10-16T12:34:20.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.371+0000 7ffac67fc640 1 -- 172.21.15.60:0/3307118000 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffacc0581b0 con 0x7ffad00c9ca0 2023-10-16T12:34:20.373 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:20.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ffab406ae70 msgr2=0x7ffab406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ffab406ae70 0x7ffab406d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffad01f66d0 tx=0x7ffac0008040 comp rx=0 tx=0).stop 2023-10-16T12:34:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 msgr2=0x7ffad00b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 0x7ffad00b04f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00d5f0 tx=0x7ffacc00dab0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 shutdown_connections 2023-10-16T12:34:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7ffab406ae70 0x7ffab406d330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffad00ca3b0 0x7ffad01f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffad015ad70 0x7ffad00c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 --2- 172.21.15.60:0/3307118000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffad00c9ca0 0x7ffad00b04f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 >> 172.21.15.60:0/3307118000 conn(0x7ffad00ce9b0 msgr2=0x7ffad015dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 shutdown_connections 2023-10-16T12:34:20.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.374+0000 7ffad7eec640 1 -- 172.21.15.60:0/3307118000 wait complete. 2023-10-16T12:34:20.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:34:20.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:20.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:20.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.612+0000 7f1cb0a4b640 1 Processor -- start 2023-10-16T12:34:20.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.613+0000 7f1cb0a4b640 1 -- start start 2023-10-16T12:34:20.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.614+0000 7f1cb0a4b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac15ad70 0x7f1cac069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.614+0000 7f1cb0a4b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac0cb030 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.614+0000 7f1cb0a4b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac0c91c0 con 0x7f1cac0caac0 2023-10-16T12:34:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.614+0000 7f1cb0a4b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0c94a0 con 0x7f1cac15ad70 2023-10-16T12:34:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.615+0000 7f1ca9d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1cac0c9ca0 0x7f1cac0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51078/0 (socket says 172.21.15.60:51078) 2023-10-16T12:34:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.615+0000 7f1ca9d9b640 1 -- 172.21.15.60:0/1114636008 learned_addr learned my addr 172.21.15.60:0/1114636008 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.615+0000 7f1caa59c640 1 --2- 172.21.15.60:0/1114636008 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac15ad70 0x7f1cac069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.615+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105109479 0 0) 0x7f1cac0cb030 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.615+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c94003520 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.616+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155716230 0 0) 0x7f1cac0c91c0 con 0x7f1cac0caac0 2023-10-16T12:34:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.616+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cac0cb030 con 0x7f1cac0caac0 2023-10-16T12:34:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.616+0000 7f1caa59c640 1 -- 172.21.15.60:0/1114636008 >> v1:172.21.15.144:6789/0 conn(0x7f1cac0caac0 legacy=0x7f1cac0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.616+0000 7f1caa59c640 1 -- 172.21.15.60:0/1114636008 >> v1:172.21.15.60:6789/0 conn(0x7f1cac0c9ca0 legacy=0x7f1cac0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.616+0000 7f1caa59c640 1 -- 172.21.15.60:0/1114636008 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0c9780 con 0x7f1cac15ad70 2023-10-16T12:34:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.617+0000 7f1caa59c640 1 --2- 172.21.15.60:0/1114636008 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac15ad70 0x7f1cac069da0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0097d0 tx=0x7f1c9c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d125c122c6c22424 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.618+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c02d040 con 0x7f1cac15ad70 2023-10-16T12:34:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.618+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c031810 con 0x7f1cac15ad70 2023-10-16T12:34:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.618+0000 7f1ca959a640 1 -- 172.21.15.60:0/1114636008 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c031ba0 con 0x7f1cac15ad70 2023-10-16T12:34:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/1114636008 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac15ad70 msgr2=0x7f1cac069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/1114636008 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac15ad70 0x7f1cac069da0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0097d0 tx=0x7f1c9c02b8b0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/1114636008 shutdown_connections 2023-10-16T12:34:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/1114636008 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac15ad70 0x7f1cac069da0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/1114636008 >> 172.21.15.60:0/1114636008 conn(0x7f1cac0ce9b0 msgr2=0x7f1cac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/1114636008 shutdown_connections 2023-10-16T12:34:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.619+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/1114636008 wait complete. 2023-10-16T12:34:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.621+0000 7f1cb0a4b640 1 Processor -- start 2023-10-16T12:34:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.621+0000 7f1cb0a4b640 1 -- start start 2023-10-16T12:34:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1cb0a4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 0x7f1cac14b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1cb0a4b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac0caac0 0x7f1cac0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1cb0a4b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cac15ad70 0x7f1cac1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1cb0a4b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0c5fd0 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1cb0a4b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0c62b0 con 0x7f1cac15ad70 2023-10-16T12:34:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1cb0a4b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0c6590 con 0x7f1cac0caac0 2023-10-16T12:34:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.622+0000 7f1caa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 0x7f1cac14b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 0x7f1cac14b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54750/0 (socket says 172.21.15.60:54750) 2023-10-16T12:34:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 -- 172.21.15.60:0/2479436814 learned_addr learned my addr 172.21.15.60:0/2479436814 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1ca9d9b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac0caac0 0x7f1cac0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caad9d640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cac15ad70 0x7f1cac1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 -- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac0caac0 msgr2=0x7f1cac0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac0caac0 0x7f1cac0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 -- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cac15ad70 msgr2=0x7f1cac1576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cac15ad70 0x7f1cac1576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 -- 172.21.15.60:0/2479436814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c009400 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.623+0000 7f1caa59c640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 0x7f1cac14b230 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0097d0 tx=0x7f1c9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.624+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c03ad60 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.624+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c0443f0 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.624+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0445a0 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.624+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac0c6870 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.624+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac0cdb20 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.626+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac0c6d90 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.627+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c9c04d450 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.627+0000 7f1c9affd640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c8c06af40 0x7f1c8c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.627+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f1c9c0ab8b0 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.628+0000 7f1ca9d9b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c8c06af40 0x7f1c8c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:20.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.629+0000 7f1ca9d9b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c8c06af40 0x7f1c8c06d400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1cac1f1410 tx=0x7f1ca000d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:20.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.636+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cac0c6d90 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.853+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1cac0c6d90 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.854+0000 7f1c9affd640 1 -- 172.21.15.60:0/2479436814 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1c9c035020 con 0x7f1cac0c9ca0 2023-10-16T12:34:20.857 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:34:20.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.858+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c8c06af40 msgr2=0x7f1c8c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.858+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c8c06af40 0x7f1c8c06d400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1cac1f1410 tx=0x7f1ca000d040 comp rx=0 tx=0).stop 2023-10-16T12:34:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.858+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 msgr2=0x7f1cac14b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.858+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 0x7f1cac14b230 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0097d0 tx=0x7f1c9c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 shutdown_connections 2023-10-16T12:34:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1c8c06af40 0x7f1c8c06d400 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cac0caac0 0x7f1cac0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cac15ad70 0x7f1cac1576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 --2- 172.21.15.60:0/2479436814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cac0c9ca0 0x7f1cac14b230 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:20.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 >> 172.21.15.60:0/2479436814 conn(0x7f1cac0ce9b0 msgr2=0x7f1cac0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:20.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 shutdown_connections 2023-10-16T12:34:20.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:20.859+0000 7f1cb0a4b640 1 -- 172.21.15.60:0/2479436814 wait complete. 2023-10-16T12:34:20.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:34:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.079+0000 7f8c9a34e640 1 Processor -- start 2023-10-16T12:34:21.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.079+0000 7f8c9a34e640 1 -- start start 2023-10-16T12:34:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.080+0000 7f8c9a34e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c94159950 0x7f8c9415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.080+0000 7f8c9a34e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8c9406c810 0x7f8c940708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.080+0000 7f8c9a34e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9415aef0 con 0x7f8c94070450 2023-10-16T12:34:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.080+0000 7f8c9a34e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c94070e20 con 0x7f8c94159950 2023-10-16T12:34:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.080+0000 7f8c9a34e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9415e490 con 0x7f8c9406c810 2023-10-16T12:34:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c94159950 0x7f8c9415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c94159950 0x7f8c9415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54754/0 (socket says 172.21.15.60:54754) 2023-10-16T12:34:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 -- 172.21.15.60:0/824770550 learned_addr learned my addr 172.21.15.60:0/824770550 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c99b4d640 1 --2- 172.21.15.60:0/824770550 >> v2:172.21.15.144:3300/0 conn(0x7f8c9406c810 0x7f8c940708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 -- 172.21.15.60:0/824770550 >> v2:172.21.15.144:3300/0 conn(0x7f8c9406c810 msgr2=0x7f8c940708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 --2- 172.21.15.60:0/824770550 >> v2:172.21.15.144:3300/0 conn(0x7f8c9406c810 0x7f8c940708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 -- 172.21.15.60:0/824770550 >> v1:172.21.15.144:6789/0 conn(0x7f8c94070450 legacy=0x7f8c9406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c9934c640 1 -- 172.21.15.60:0/824770550 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9415d910 con 0x7f8c94159950 2023-10-16T12:34:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.081+0000 7f8c99b4d640 1 --2- 172.21.15.60:0/824770550 >> v2:172.21.15.144:3300/0 conn(0x7f8c9406c810 0x7f8c940708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.082+0000 7f8c9934c640 1 --2- 172.21.15.60:0/824770550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c9415a9b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8c940716a0 tx=0x7f8c9002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1c8e1ab2fbb1ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c8bfff640 1 -- 172.21.15.60:0/824770550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c900317e0 con 0x7f8c94159950 2023-10-16T12:34:21.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c8bfff640 1 -- 172.21.15.60:0/824770550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c90031940 con 0x7f8c94159950 2023-10-16T12:34:21.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c8bfff640 1 -- 172.21.15.60:0/824770550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90031cd0 con 0x7f8c94159950 2023-10-16T12:34:21.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 -- 172.21.15.60:0/824770550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 msgr2=0x7f8c9415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/824770550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c9415a9b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8c940716a0 tx=0x7f8c9002b880 comp rx=0 tx=0).stop 2023-10-16T12:34:21.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 -- 172.21.15.60:0/824770550 shutdown_connections 2023-10-16T12:34:21.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/824770550 >> v2:172.21.15.144:3300/0 conn(0x7f8c9406c810 0x7f8c940708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/824770550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c9415a9b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 -- 172.21.15.60:0/824770550 >> 172.21.15.60:0/824770550 conn(0x7f8c940ce6a0 msgr2=0x7f8c940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:21.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 -- 172.21.15.60:0/824770550 shutdown_connections 2023-10-16T12:34:21.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.083+0000 7f8c9a34e640 1 -- 172.21.15.60:0/824770550 wait complete. 2023-10-16T12:34:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.085+0000 7f8c9a34e640 1 Processor -- start 2023-10-16T12:34:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.085+0000 7f8c9a34e640 1 -- start start 2023-10-16T12:34:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.086+0000 7f8c9a34e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c9406c810 0x7f8c940b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c9a34e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c94070450 0x7f8c9414eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c9a34e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c94152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c9a34e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94151e40 con 0x7f8c94159950 2023-10-16T12:34:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c9a34e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9415d4c0 con 0x7f8c94070450 2023-10-16T12:34:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c9a34e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94152050 con 0x7f8c9406c810 2023-10-16T12:34:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c99b4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c94152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c99b4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c94152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54756/0 (socket says 172.21.15.60:54756) 2023-10-16T12:34:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c99b4d640 1 -- 172.21.15.60:0/889314739 learned_addr learned my addr 172.21.15.60:0/889314739 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c9934c640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c9406c810 0x7f8c940b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.087+0000 7f8c98b4b640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c94070450 0x7f8c9414eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c99b4d640 1 -- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c9406c810 msgr2=0x7f8c940b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c99b4d640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c9406c810 0x7f8c940b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c99b4d640 1 -- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c94070450 msgr2=0x7f8c9414eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c99b4d640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c94070450 0x7f8c9414eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c99b4d640 1 -- 172.21.15.60:0/889314739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900094d0 con 0x7f8c94159950 2023-10-16T12:34:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c99b4d640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c94152b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00cef0 tx=0x7f8c8c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00d840 con 0x7f8c94159950 2023-10-16T12:34:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c00d9c0 con 0x7f8c94159950 2023-10-16T12:34:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c94152330 con 0x7f8c94159950 2023-10-16T12:34:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.088+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00dd70 con 0x7f8c94159950 2023-10-16T12:34:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.089+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9414f430 con 0x7f8c94159950 2023-10-16T12:34:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.090+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c94152850 con 0x7f8c94159950 2023-10-16T12:34:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.091+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c8c01e3b0 con 0x7f8c94159950 2023-10-16T12:34:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.092+0000 7f8c89ffb640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c7806ae70 0x7f8c7806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.092+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f8c8c08a0e0 con 0x7f8c94159950 2023-10-16T12:34:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.092+0000 7f8c9934c640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c7806ae70 0x7f8c7806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.094+0000 7f8c9934c640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c7806ae70 0x7f8c7806d330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8c90036000 tx=0x7f8c9003d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.096+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c94152850 con 0x7f8c94159950 2023-10-16T12:34:21.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.294+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8c94152850 con 0x7f8c94159950 2023-10-16T12:34:21.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.295+0000 7f8c89ffb640 1 -- 172.21.15.60:0/889314739 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8c94152850 con 0x7f8c94159950 2023-10-16T12:34:21.297 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:34:21.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c7806ae70 msgr2=0x7f8c7806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c7806ae70 0x7f8c7806d330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8c90036000 tx=0x7f8c9003d040 comp rx=0 tx=0).stop 2023-10-16T12:34:21.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 msgr2=0x7f8c94152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c94152b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00cef0 tx=0x7f8c8c00a710 comp rx=0 tx=0).stop 2023-10-16T12:34:21.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 shutdown_connections 2023-10-16T12:34:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8c7806ae70 0x7f8c7806d330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c94159950 0x7f8c94152b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c94070450 0x7f8c9414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 --2- 172.21.15.60:0/889314739 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c9406c810 0x7f8c940b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 >> 172.21.15.60:0/889314739 conn(0x7f8c940ce6a0 msgr2=0x7f8c940b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 shutdown_connections 2023-10-16T12:34:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.297+0000 7f8c9a34e640 1 -- 172.21.15.60:0/889314739 wait complete. 2023-10-16T12:34:21.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:34:21.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.509+0000 7fa4a3fff640 1 Processor -- start 2023-10-16T12:34:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.509+0000 7fa4a3fff640 1 -- start start 2023-10-16T12:34:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a40b8480 0x7fa4a40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4a40d0be0 0x7fa4a40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a3fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a415d150 con 0x7fa4a415cd70 2023-10-16T12:34:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a3fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40d1180 con 0x7fa4a40d0be0 2023-10-16T12:34:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40d1460 con 0x7fa4a40b8480 2023-10-16T12:34:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4a40d0be0 0x7fa4a40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4a40d0be0 0x7fa4a40cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54762/0 (socket says 172.21.15.60:54762) 2023-10-16T12:34:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 -- 172.21.15.60:0/1298796816 learned_addr learned my addr 172.21.15.60:0/1298796816 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:21.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a27fc640 1 --2- 172.21.15.60:0/1298796816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a40b8480 0x7fa4a40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 -- 172.21.15.60:0/1298796816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a40b8480 msgr2=0x7fa4a40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 --2- 172.21.15.60:0/1298796816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a40b8480 0x7fa4a40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 -- 172.21.15.60:0/1298796816 >> v1:172.21.15.144:6789/0 conn(0x7fa4a415cd70 legacy=0x7fa4a40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.511+0000 7fa4a37fe640 1 -- 172.21.15.60:0/1298796816 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40d1740 con 0x7fa4a40d0be0 2023-10-16T12:34:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a37fe640 1 --2- 172.21.15.60:0/1298796816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40d0be0 0x7fa4a40cdd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009e50 tx=0x7fa49c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec7a36fce5a5f42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a1ffb640 1 -- 172.21.15.60:0/1298796816 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c02ca70 con 0x7fa4a40d0be0 2023-10-16T12:34:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a1ffb640 1 -- 172.21.15.60:0/1298796816 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c02cbd0 con 0x7fa4a40d0be0 2023-10-16T12:34:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a1ffb640 1 -- 172.21.15.60:0/1298796816 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0343f0 con 0x7fa4a40d0be0 2023-10-16T12:34:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a3fff640 1 -- 172.21.15.60:0/1298796816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40d0be0 msgr2=0x7fa4a40cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/1298796816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40d0be0 0x7fa4a40cdd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009e50 tx=0x7fa49c02bed0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a3fff640 1 -- 172.21.15.60:0/1298796816 shutdown_connections 2023-10-16T12:34:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/1298796816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40d0be0 0x7fa4a40cdd00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/1298796816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a40b8480 0x7fa4a40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.513+0000 7fa4a3fff640 1 -- 172.21.15.60:0/1298796816 >> 172.21.15.60:0/1298796816 conn(0x7fa4a40cf570 msgr2=0x7fa4a40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.514+0000 7fa4a3fff640 1 -- 172.21.15.60:0/1298796816 shutdown_connections 2023-10-16T12:34:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.514+0000 7fa4a3fff640 1 -- 172.21.15.60:0/1298796816 wait complete. 2023-10-16T12:34:21.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.515+0000 7fa4a3fff640 1 Processor -- start 2023-10-16T12:34:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.515+0000 7fa4a3fff640 1 -- start start 2023-10-16T12:34:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.516+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 0x7fa4a40cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.516+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4a40d0be0 0x7fa4a4151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.516+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a415cd70 0x7fa4a40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.516+0000 7fa4a3fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4152290 con 0x7fa4a40b8480 2023-10-16T12:34:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.516+0000 7fa4a3fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40c7cd0 con 0x7fa4a40d0be0 2023-10-16T12:34:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.516+0000 7fa4a3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40c7fb0 con 0x7fa4a415cd70 2023-10-16T12:34:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 0x7fa4a40cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 0x7fa4a40cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54770/0 (socket says 172.21.15.60:54770) 2023-10-16T12:34:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2380169189 learned_addr learned my addr 172.21.15.60:0/2380169189 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a37fe640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a415cd70 0x7fa4a40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a27fc640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4a40d0be0 0x7fa4a4151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a415cd70 msgr2=0x7fa4a40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a415cd70 0x7fa4a40c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4a40d0be0 msgr2=0x7fa4a4151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4a40d0be0 0x7fa4a4151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2380169189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c009b30 con 0x7fa4a40b8480 2023-10-16T12:34:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a37fe640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a415cd70 0x7fa4a40c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a27fc640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4a40d0be0 0x7fa4a4151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.517+0000 7fa4a2ffd640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 0x7fa4a40cbc20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c430 tx=0x7fa49400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.518+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494011790 con 0x7fa4a40b8480 2023-10-16T12:34:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.518+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4940118f0 con 0x7fa4a40b8480 2023-10-16T12:34:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.518+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a40c8290 con 0x7fa4a40b8480 2023-10-16T12:34:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.518+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494011ce0 con 0x7fa4a40b8480 2023-10-16T12:34:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.518+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a40c8850 con 0x7fa4a40b8480 2023-10-16T12:34:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.519+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a404e9b0 con 0x7fa4a40b8480 2023-10-16T12:34:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.520+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa49401c460 con 0x7fa4a40b8480 2023-10-16T12:34:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.521+0000 7fa48b7fe640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa49006ae70 0x7fa49006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.521+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fa494089680 con 0x7fa4a40b8480 2023-10-16T12:34:21.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.521+0000 7fa4a27fc640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa49006ae70 0x7fa49006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.523+0000 7fa4a27fc640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa49006ae70 0x7fa49006d330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa498006fd0 tx=0x7fa498008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.525+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa494054630 con 0x7fa4a40b8480 2023-10-16T12:34:21.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.757+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a40b7280 con 0x7fa4a40b8480 2023-10-16T12:34:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.758+0000 7fa48b7fe640 1 -- 172.21.15.60:0/2380169189 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fa494057c50 con 0x7fa4a40b8480 2023-10-16T12:34:21.760 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:21.761 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:34:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:34:21.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.760+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa49006ae70 msgr2=0x7fa49006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.760+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa49006ae70 0x7fa49006d330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa498006fd0 tx=0x7fa498008040 comp rx=0 tx=0).stop 2023-10-16T12:34:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 msgr2=0x7fa4a40cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 0x7fa4a40cbc20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c430 tx=0x7fa49400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 shutdown_connections 2023-10-16T12:34:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa49006ae70 0x7fa49006d330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4a415cd70 0x7fa4a40c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4a40d0be0 0x7fa4a4151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2380169189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a40b8480 0x7fa4a40cbc20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 >> 172.21.15.60:0/2380169189 conn(0x7fa4a40cf570 msgr2=0x7fa4a40d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 shutdown_connections 2023-10-16T12:34:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.761+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2380169189 wait complete. 2023-10-16T12:34:21.780 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:21.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.986+0000 7f6159197640 1 Processor -- start 2023-10-16T12:34:21.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.986+0000 7f6159197640 1 -- start start 2023-10-16T12:34:21.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.987+0000 7f6159197640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61540ce390 0x7f615415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f6159197640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f615415ed60 con 0x7f615415aaa0 2023-10-16T12:34:21.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f6159197640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61540707e0 con 0x7f615415ae80 2023-10-16T12:34:21.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f6159197640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615406b810 con 0x7f61540ce390 2023-10-16T12:34:21.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f6152d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f615415aaa0 0x7f61540cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51092/0 (socket says 172.21.15.60:51092) 2023-10-16T12:34:21.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f6152d9d640 1 -- 172.21.15.60:0/2829426806 learned_addr learned my addr 172.21.15.60:0/2829426806 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:21.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f615359e640 1 --2- 172.21.15.60:0/2829426806 >> v2:172.21.15.60:3300/0 conn(0x7f61540ce390 0x7f615415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:21.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f615259c640 1 -- 172.21.15.60:0/2829426806 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084740434 0 0) 0x7f615415ed60 con 0x7f615415aaa0 2023-10-16T12:34:21.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f615259c640 1 -- 172.21.15.60:0/2829426806 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6140003520 con 0x7f615415aaa0 2023-10-16T12:34:21.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.988+0000 7f615359e640 1 -- 172.21.15.60:0/2829426806 >> v1:172.21.15.144:6789/0 conn(0x7f615415ae80 legacy=0x7f6154070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.989+0000 7f615359e640 1 -- 172.21.15.60:0/2829426806 >> v1:172.21.15.60:6789/0 conn(0x7f615415aaa0 legacy=0x7f61540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.989+0000 7f615359e640 1 -- 172.21.15.60:0/2829426806 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615415ef90 con 0x7f61540ce390 2023-10-16T12:34:21.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.990+0000 7f615359e640 1 --2- 172.21.15.60:0/2829426806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61540ce390 0x7f615415e820 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f614c009d90 tx=0x7f614c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d49961c7f3ecd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:21.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.990+0000 7f615259c640 1 -- 172.21.15.60:0/2829426806 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c02ccf0 con 0x7f61540ce390 2023-10-16T12:34:21.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.990+0000 7f615259c640 1 -- 172.21.15.60:0/2829426806 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c02ce50 con 0x7f61540ce390 2023-10-16T12:34:21.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.990+0000 7f615259c640 1 -- 172.21.15.60:0/2829426806 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c034660 con 0x7f61540ce390 2023-10-16T12:34:21.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 -- 172.21.15.60:0/2829426806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61540ce390 msgr2=0x7f615415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:21.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 --2- 172.21.15.60:0/2829426806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61540ce390 0x7f615415e820 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f614c009d90 tx=0x7f614c02bf40 comp rx=0 tx=0).stop 2023-10-16T12:34:21.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 -- 172.21.15.60:0/2829426806 shutdown_connections 2023-10-16T12:34:21.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 --2- 172.21.15.60:0/2829426806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61540ce390 0x7f615415e820 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:21.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 -- 172.21.15.60:0/2829426806 >> 172.21.15.60:0/2829426806 conn(0x7f61540cb410 msgr2=0x7f61540cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:21.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 -- 172.21.15.60:0/2829426806 shutdown_connections 2023-10-16T12:34:21.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.991+0000 7f6159197640 1 -- 172.21.15.60:0/2829426806 wait complete. 2023-10-16T12:34:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.993+0000 7f6159197640 1 Processor -- start 2023-10-16T12:34:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.993+0000 7f6159197640 1 -- start start 2023-10-16T12:34:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6159197640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61540ce390 0x7f615415dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6159197640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f615415aaa0 0x7f61540a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6159197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 0x7f61541575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6159197640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61541f1520 con 0x7f615415ae80 2023-10-16T12:34:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6159197640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540c84d0 con 0x7f61540ce390 2023-10-16T12:34:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6159197640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61540c74c0 con 0x7f615415aaa0 2023-10-16T12:34:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f615359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 0x7f61541575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f615359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 0x7f61541575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54784/0 (socket says 172.21.15.60:54784) 2023-10-16T12:34:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f615359e640 1 -- 172.21.15.60:0/1859585507 learned_addr learned my addr 172.21.15.60:0/1859585507 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f6152d9d640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61540ce390 0x7f615415dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.994+0000 7f614bfff640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f615415aaa0 0x7f61540a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f615359e640 1 -- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f615415aaa0 msgr2=0x7f61540a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f615359e640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f615415aaa0 0x7f61540a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f615359e640 1 -- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61540ce390 msgr2=0x7f615415dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f615359e640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61540ce390 0x7f615415dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f615359e640 1 -- 172.21.15.60:0/1859585507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c009a70 con 0x7f615415ae80 2023-10-16T12:34:22.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f615359e640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 0x7f61541575d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f614c037040 tx=0x7f614c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c03d600 con 0x7f615415ae80 2023-10-16T12:34:22.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c03d760 con 0x7f615415ae80 2023-10-16T12:34:22.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61540c8040 con 0x7f615415ae80 2023-10-16T12:34:22.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c03db10 con 0x7f615415ae80 2023-10-16T12:34:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.995+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61540c79f0 con 0x7f615415ae80 2023-10-16T12:34:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.997+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61540c81d0 con 0x7f615415ae80 2023-10-16T12:34:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.998+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f614c03b040 con 0x7f615415ae80 2023-10-16T12:34:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.998+0000 7f614b7fe640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f613406aea0 0x7f613406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.999+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f614c0abb10 con 0x7f615415ae80 2023-10-16T12:34:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:21.999+0000 7f6152d9d640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f613406aea0 0x7f613406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.000+0000 7f6152d9d640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f613406aea0 0x7f613406d360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61440011e0 tx=0x7f6144009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.003+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f614c076bf0 con 0x7f615415ae80 2023-10-16T12:34:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.200+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f615404e160 con 0x7f615415ae80 2023-10-16T12:34:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.203+0000 7f614b7fe640 1 -- 172.21.15.60:0/1859585507 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16714 (secure 0 0 0) 0x7f614c009450 con 0x7f615415ae80 2023-10-16T12:34:22.206 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:22.206 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:11.506118+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f613406aea0 msgr2=0x7f613406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f613406aea0 0x7f613406d360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61440011e0 tx=0x7f6144009040 comp rx=0 tx=0).stop 2023-10-16T12:34:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 msgr2=0x7f61541575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 0x7f61541575d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f614c037040 tx=0x7f614c004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 shutdown_connections 2023-10-16T12:34:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f613406aea0 0x7f613406d360 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f615415aaa0 0x7f61540a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615415ae80 0x7f61541575d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 --2- 172.21.15.60:0/1859585507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61540ce390 0x7f615415dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 >> 172.21.15.60:0/1859585507 conn(0x7f61540cb410 msgr2=0x7f61540b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:22.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 shutdown_connections 2023-10-16T12:34:22.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.206+0000 7f6159197640 1 -- 172.21.15.60:0/1859585507 wait complete. 2023-10-16T12:34:22.227 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:34:22.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:34:22.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.446+0000 7f132e7ff640 1 Processor -- start 2023-10-16T12:34:22.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.446+0000 7f132e7ff640 1 -- start start 2023-10-16T12:34:22.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132e7ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f132815c9f0 0x7f13280c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132e7ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1328160a90 con 0x7f132815dde0 2023-10-16T12:34:22.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132e7ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13280cd900 con 0x7f13280c9230 2023-10-16T12:34:22.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132e7ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13280cdec0 con 0x7f132815c9f0 2023-10-16T12:34:22.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132d7fd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f132815c9f0 0x7f13280c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132dffe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f132815dde0 0x7f13280ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51098/0 (socket says 172.21.15.60:51098) 2023-10-16T12:34:22.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.448+0000 7f132dffe640 1 -- 172.21.15.60:0/888866257 learned_addr learned my addr 172.21.15.60:0/888866257 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:22.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.449+0000 7f131ffff640 1 -- 172.21.15.60:0/888866257 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711569835 0 0) 0x7f1328160a90 con 0x7f132815dde0 2023-10-16T12:34:22.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.449+0000 7f131ffff640 1 -- 172.21.15.60:0/888866257 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1314003520 con 0x7f132815dde0 2023-10-16T12:34:22.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.449+0000 7f132d7fd640 1 -- 172.21.15.60:0/888866257 >> v1:172.21.15.144:6789/0 conn(0x7f13280c9230 legacy=0x7f13280709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.449+0000 7f132d7fd640 1 -- 172.21.15.60:0/888866257 >> v1:172.21.15.60:6789/0 conn(0x7f132815dde0 legacy=0x7f13280ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.449+0000 7f132d7fd640 1 -- 172.21.15.60:0/888866257 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13280cdbe0 con 0x7f132815c9f0 2023-10-16T12:34:22.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.451+0000 7f132d7fd640 1 --2- 172.21.15.60:0/888866257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f13280c8cf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1324009d90 tx=0x7f132402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bf83f3e1e3bd74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.451+0000 7f131ffff640 1 -- 172.21.15.60:0/888866257 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132402ccf0 con 0x7f132815c9f0 2023-10-16T12:34:22.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.451+0000 7f131ffff640 1 -- 172.21.15.60:0/888866257 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132402ce50 con 0x7f132815c9f0 2023-10-16T12:34:22.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.451+0000 7f131ffff640 1 -- 172.21.15.60:0/888866257 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324034660 con 0x7f132815c9f0 2023-10-16T12:34:22.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.451+0000 7f132e7ff640 1 -- 172.21.15.60:0/888866257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 msgr2=0x7f13280c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.451+0000 7f132e7ff640 1 --2- 172.21.15.60:0/888866257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f13280c8cf0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1324009d90 tx=0x7f132402bf40 comp rx=0 tx=0).stop 2023-10-16T12:34:22.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.452+0000 7f132e7ff640 1 -- 172.21.15.60:0/888866257 shutdown_connections 2023-10-16T12:34:22.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.452+0000 7f132e7ff640 1 --2- 172.21.15.60:0/888866257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f13280c8cf0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.452+0000 7f132e7ff640 1 -- 172.21.15.60:0/888866257 >> 172.21.15.60:0/888866257 conn(0x7f13280ce6a0 msgr2=0x7f13280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:22.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.452+0000 7f132e7ff640 1 -- 172.21.15.60:0/888866257 shutdown_connections 2023-10-16T12:34:22.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.452+0000 7f132e7ff640 1 -- 172.21.15.60:0/888866257 wait complete. 2023-10-16T12:34:22.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.453+0000 7f132e7ff640 1 Processor -- start 2023-10-16T12:34:22.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.454+0000 7f132e7ff640 1 -- start start 2023-10-16T12:34:22.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.454+0000 7f132e7ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13280c9230 0x7f1328156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.454+0000 7f132e7ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f1328158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132e7ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f132815dde0 0x7f13281590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132e7ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13280c21d0 con 0x7f132815c9f0 2023-10-16T12:34:22.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132e7ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13280c2470 con 0x7f13280c9230 2023-10-16T12:34:22.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132e7ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13280c2750 con 0x7f132815dde0 2023-10-16T12:34:22.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f1328158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f1328158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54798/0 (socket says 172.21.15.60:54798) 2023-10-16T12:34:22.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 -- 172.21.15.60:0/542507956 learned_addr learned my addr 172.21.15.60:0/542507956 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:22.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132dffe640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f132815dde0 0x7f13281590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132d7fd640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13280c9230 0x7f1328156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 -- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f132815dde0 msgr2=0x7f13281590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f132815dde0 0x7f13281590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 -- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13280c9230 msgr2=0x7f1328156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13280c9230 0x7f1328156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132cffc640 1 -- 172.21.15.60:0/542507956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1324009a70 con 0x7f132815c9f0 2023-10-16T12:34:22.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.455+0000 7f132dffe640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f132815dde0 0x7f13281590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:22.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.456+0000 7f132cffc640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f1328158b60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f131800c450 tx=0x7f131800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.456+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13180052d0 con 0x7f132815c9f0 2023-10-16T12:34:22.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.456+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1318005430 con 0x7f132815c9f0 2023-10-16T12:34:22.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.456+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318012870 con 0x7f132815c9f0 2023-10-16T12:34:22.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.456+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13280c2a30 con 0x7f132815c9f0 2023-10-16T12:34:22.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.456+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13280c3210 con 0x7f132815c9f0 2023-10-16T12:34:22.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.457+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13280668c0 con 0x7f132815c9f0 2023-10-16T12:34:22.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.458+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f13180129d0 con 0x7f132815c9f0 2023-10-16T12:34:22.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.459+0000 7f131dffb640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f130c06ae70 0x7f130c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.459+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f13180891b0 con 0x7f132815c9f0 2023-10-16T12:34:22.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.460+0000 7f132d7fd640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f130c06ae70 0x7f130c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.461+0000 7f132d7fd640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f130c06ae70 0x7f130c06d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1324036000 tx=0x7f1324030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.464+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1318054200 con 0x7f132815c9f0 2023-10-16T12:34:22.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.699+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f132804e9b0 con 0x7f132815c9f0 2023-10-16T12:34:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.700+0000 7f131dffb640 1 -- 172.21.15.60:0/542507956 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f1318057820 con 0x7f132815c9f0 2023-10-16T12:34:22.703 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:22.703 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:34:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:34:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f130c06ae70 msgr2=0x7f130c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f130c06ae70 0x7f130c06d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1324036000 tx=0x7f1324030000 comp rx=0 tx=0).stop 2023-10-16T12:34:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 msgr2=0x7f1328158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f1328158b60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f131800c450 tx=0x7f131800c910 comp rx=0 tx=0).stop 2023-10-16T12:34:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 shutdown_connections 2023-10-16T12:34:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f130c06ae70 0x7f130c06d330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f132815dde0 0x7f13281590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f132815c9f0 0x7f1328158b60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 --2- 172.21.15.60:0/542507956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13280c9230 0x7f1328156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 >> 172.21.15.60:0/542507956 conn(0x7f13280ce6a0 msgr2=0x7f132815a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 shutdown_connections 2023-10-16T12:34:22.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.703+0000 7f132e7ff640 1 -- 172.21.15.60:0/542507956 wait complete. 2023-10-16T12:34:22.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:34:22.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.946+0000 7f079f6a2640 1 Processor -- start 2023-10-16T12:34:22.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.946+0000 7f079f6a2640 1 -- start start 2023-10-16T12:34:22.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.947+0000 7f079f6a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0798159f00 0x7f07980b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.947+0000 7f079f6a2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f079815d880 0x7f07980b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.947+0000 7f079f6a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079806f7f0 con 0x7f0798159b20 2023-10-16T12:34:22.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.947+0000 7f079f6a2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079806c5e0 con 0x7f079815d880 2023-10-16T12:34:22.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.947+0000 7f079f6a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079806c8c0 con 0x7f0798159f00 2023-10-16T12:34:22.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.948+0000 7f079dc3f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f079815d880 0x7f07980b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.948+0000 7f079dc3f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f079815d880 0x7f07980b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54808/0 (socket says 172.21.15.60:54808) 2023-10-16T12:34:22.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.948+0000 7f079dc3f640 1 -- 172.21.15.60:0/1851154134 learned_addr learned my addr 172.21.15.60:0/1851154134 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:22.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.948+0000 7f079cc3d640 1 --2- 172.21.15.60:0/1851154134 >> v2:172.21.15.144:3300/0 conn(0x7f0798159f00 0x7f07980b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.949+0000 7f079dc3f640 1 -- 172.21.15.60:0/1851154134 >> v2:172.21.15.144:3300/0 conn(0x7f0798159f00 msgr2=0x7f07980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:22.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.949+0000 7f079dc3f640 1 --2- 172.21.15.60:0/1851154134 >> v2:172.21.15.144:3300/0 conn(0x7f0798159f00 0x7f07980b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.949+0000 7f079dc3f640 1 -- 172.21.15.60:0/1851154134 >> v1:172.21.15.144:6789/0 conn(0x7f0798159b20 legacy=0x7f079806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.949+0000 7f079dc3f640 1 -- 172.21.15.60:0/1851154134 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079806cba0 con 0x7f079815d880 2023-10-16T12:34:22.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.950+0000 7f079dc3f640 1 --2- 172.21.15.60:0/1851154134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f07980b0950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f079815d770 tx=0x7f079002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f117b6efd37b7ad0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f078ffff640 1 -- 172.21.15.60:0/1851154134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079002cb20 con 0x7f079815d880 2023-10-16T12:34:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f078ffff640 1 -- 172.21.15.60:0/1851154134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079002cc80 con 0x7f079815d880 2023-10-16T12:34:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f078ffff640 1 -- 172.21.15.60:0/1851154134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07900343f0 con 0x7f079815d880 2023-10-16T12:34:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 -- 172.21.15.60:0/1851154134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 msgr2=0x7f07980b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1851154134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f07980b0950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f079815d770 tx=0x7f079002bf80 comp rx=0 tx=0).stop 2023-10-16T12:34:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 -- 172.21.15.60:0/1851154134 shutdown_connections 2023-10-16T12:34:22.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1851154134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f07980b0950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1851154134 >> v2:172.21.15.144:3300/0 conn(0x7f0798159f00 0x7f07980b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 -- 172.21.15.60:0/1851154134 >> 172.21.15.60:0/1851154134 conn(0x7f07980cb410 msgr2=0x7f07980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:22.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 -- 172.21.15.60:0/1851154134 shutdown_connections 2023-10-16T12:34:22.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.951+0000 7f079f6a2640 1 -- 172.21.15.60:0/1851154134 wait complete. 2023-10-16T12:34:22.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.953+0000 7f079f6a2640 1 Processor -- start 2023-10-16T12:34:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.953+0000 7f079f6a2640 1 -- start start 2023-10-16T12:34:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.954+0000 7f079f6a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0798159b20 0x7f07980c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.954+0000 7f079f6a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0798159f00 0x7f079814be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.954+0000 7f079f6a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f079815a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.954+0000 7f079f6a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07980c3480 con 0x7f079815d880 2023-10-16T12:34:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.954+0000 7f079f6a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079814c3d0 con 0x7f0798159f00 2023-10-16T12:34:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.954+0000 7f079f6a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079815b710 con 0x7f0798159b20 2023-10-16T12:34:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f079815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f079815a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54810/0 (socket says 172.21.15.60:54810) 2023-10-16T12:34:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 -- 172.21.15.60:0/1260404863 learned_addr learned my addr 172.21.15.60:0/1260404863 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079d43e640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0798159b20 0x7f07980c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 -- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0798159b20 msgr2=0x7f07980c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0798159b20 0x7f07980c49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 -- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0798159f00 msgr2=0x7f079814be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0798159f00 0x7f079814be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 -- 172.21.15.60:0/1260404863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0790009b30 con 0x7f079815d880 2023-10-16T12:34:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.955+0000 7f079dc3f640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f079815a380 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0790036040 tx=0x7f0790004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.956+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790009510 con 0x7f079815d880 2023-10-16T12:34:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.956+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079003e450 con 0x7f079815d880 2023-10-16T12:34:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.956+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079003e800 con 0x7f079815d880 2023-10-16T12:34:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.956+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079815a8c0 con 0x7f079815d880 2023-10-16T12:34:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.956+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079815ae70 con 0x7f079815d880 2023-10-16T12:34:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.957+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079815bbc0 con 0x7f079815d880 2023-10-16T12:34:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.958+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f079003e960 con 0x7f079815d880 2023-10-16T12:34:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.959+0000 7f078dffb640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f077c06aea0 0x7f077c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.959+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f07900ac1e0 con 0x7f079815d880 2023-10-16T12:34:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.959+0000 7f079d43e640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f077c06aea0 0x7f077c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.961+0000 7f079d43e640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f077c06aea0 0x7f077c06d360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0794001500 tx=0x7f0794009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:22.963+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f079815bbc0 con 0x7f079815d880 2023-10-16T12:34:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.202+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f079815bbc0 con 0x7f079815d880 2023-10-16T12:34:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.204+0000 7f078dffb640 1 -- 172.21.15.60:0/1260404863 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f079815bbc0 con 0x7f079815d880 2023-10-16T12:34:23.206 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:23.207 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/835459902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":835459902},{"type":"v1","addr":"172.21.15.60:6833","nonce":835459902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4212,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1371013330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1371013330},{"type":"v1","addr":"172.21.15.60:6835","nonce":1371013330}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3732602645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3732602645},{"type":"v1","addr":"172.21.15.144:6835","nonce":3732602645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:33:24.434648+0000","modified":"2023-10-16T12:33:28.085691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.144:6837/872963903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":872963903},{"type":"v1","addr":"172.21.15.144:6837","nonce":872963903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:34:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-10-16T12:34:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f077c06aea0 msgr2=0x7f077c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f077c06aea0 0x7f077c06d360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0794001500 tx=0x7f0794009040 comp rx=0 tx=0).stop 2023-10-16T12:34:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 msgr2=0x7f079815a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f079815a380 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0790036040 tx=0x7f0790004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 shutdown_connections 2023-10-16T12:34:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f077c06aea0 0x7f077c06d360 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f079815d880 0x7f079815a380 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0798159f00 0x7f079814be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 --2- 172.21.15.60:0/1260404863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0798159b20 0x7f07980c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 >> 172.21.15.60:0/1260404863 conn(0x7f07980cb410 msgr2=0x7f07980ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 shutdown_connections 2023-10-16T12:34:23.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.207+0000 7f079f6a2640 1 -- 172.21.15.60:0/1260404863 wait complete. 2023-10-16T12:34:23.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.458+0000 7fa0ade51640 1 Processor -- start 2023-10-16T12:34:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.458+0000 7fa0ade51640 1 -- start start 2023-10-16T12:34:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0ade51640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa0a80ce390 0x7fa0a815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0ade51640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a815ed60 con 0x7fa0a815ae80 2023-10-16T12:34:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0ade51640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80707e0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0ade51640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a806b810 con 0x7fa0a80ce390 2023-10-16T12:34:23.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0a6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa0a815ae80 0x7fa0a8070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51100/0 (socket says 172.21.15.60:51100) 2023-10-16T12:34:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0a6ffd640 1 -- 172.21.15.60:0/2225911065 learned_addr learned my addr 172.21.15.60:0/2225911065 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.460+0000 7fa0a7fff640 1 --2- 172.21.15.60:0/2225911065 >> v2:172.21.15.144:3300/0 conn(0x7fa0a80ce390 0x7fa0a815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.461+0000 7fa0a67fc640 1 -- 172.21.15.60:0/2225911065 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569781749 0 0) 0x7fa0a815ed60 con 0x7fa0a815ae80 2023-10-16T12:34:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.461+0000 7fa0a67fc640 1 -- 172.21.15.60:0/2225911065 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa090003520 con 0x7fa0a815ae80 2023-10-16T12:34:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.461+0000 7fa0a7fff640 1 -- 172.21.15.60:0/2225911065 >> v1:172.21.15.144:6789/0 conn(0x7fa0a815aaa0 legacy=0x7fa0a80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.461+0000 7fa0a7fff640 1 -- 172.21.15.60:0/2225911065 >> v1:172.21.15.60:6789/0 conn(0x7fa0a815ae80 legacy=0x7fa0a8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.461+0000 7fa0a67fc640 1 -- 172.21.15.60:0/2225911065 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3772394853 0 0) 0x7fa090003520 con 0x7fa0a815ae80 2023-10-16T12:34:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.461+0000 7fa0a7fff640 1 -- 172.21.15.60:0/2225911065 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a815ef90 con 0x7fa0a80ce390 2023-10-16T12:34:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.463+0000 7fa0a7fff640 1 --2- 172.21.15.60:0/2225911065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a815e820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d90 tx=0x7fa0a002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb101a60f02aa3d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.463+0000 7fa0a67fc640 1 -- 172.21.15.60:0/2225911065 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c001670 con 0x7fa0a80ce390 2023-10-16T12:34:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.463+0000 7fa0a67fc640 1 -- 172.21.15.60:0/2225911065 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a002c940 con 0x7fa0a80ce390 2023-10-16T12:34:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.463+0000 7fa0a67fc640 1 -- 172.21.15.60:0/2225911065 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002ccd0 con 0x7fa0a80ce390 2023-10-16T12:34:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 -- 172.21.15.60:0/2225911065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 msgr2=0x7fa0a815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 --2- 172.21.15.60:0/2225911065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a815e820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d90 tx=0x7fa0a002be10 comp rx=0 tx=0).stop 2023-10-16T12:34:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 -- 172.21.15.60:0/2225911065 shutdown_connections 2023-10-16T12:34:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 --2- 172.21.15.60:0/2225911065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a815e820 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 -- 172.21.15.60:0/2225911065 >> 172.21.15.60:0/2225911065 conn(0x7fa0a80cb410 msgr2=0x7fa0a80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 -- 172.21.15.60:0/2225911065 shutdown_connections 2023-10-16T12:34:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.464+0000 7fa0ade51640 1 -- 172.21.15.60:0/2225911065 wait complete. 2023-10-16T12:34:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.466+0000 7fa0ade51640 1 Processor -- start 2023-10-16T12:34:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.466+0000 7fa0ade51640 1 -- start start 2023-10-16T12:34:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.467+0000 7fa0ade51640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a81f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.467+0000 7fa0ade51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 0x7fa0a8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.467+0000 7fa0ade51640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0a815ae80 0x7fa0a8159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.467+0000 7fa0ade51640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8157960 con 0x7fa0a815aaa0 2023-10-16T12:34:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.467+0000 7fa0ade51640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81f3440 con 0x7fa0a80ce390 2023-10-16T12:34:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.467+0000 7fa0ade51640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81f3720 con 0x7fa0a815ae80 2023-10-16T12:34:23.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 0x7fa0a8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 0x7fa0a8155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54812/0 (socket says 172.21.15.60:54812) 2023-10-16T12:34:23.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 -- 172.21.15.60:0/290245314 learned_addr learned my addr 172.21.15.60:0/290245314 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:23.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a77fe640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a81f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a7fff640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0a815ae80 0x7fa0a8159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 -- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0a815ae80 msgr2=0x7fa0a8159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0a815ae80 0x7fa0a8159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 -- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 msgr2=0x7fa0a81f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a81f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 -- 172.21.15.60:0/290245314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a0009a70 con 0x7fa0a815aaa0 2023-10-16T12:34:23.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a7fff640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0a815ae80 0x7fa0a8159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:23.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.468+0000 7fa0a6ffd640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 0x7fa0a8155800 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00c750 tx=0x7fa09c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:23.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.469+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c005ad0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.469+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09c005c30 con 0x7fa0a815aaa0 2023-10-16T12:34:23.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.469+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c007070 con 0x7fa0a815aaa0 2023-10-16T12:34:23.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.469+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a81f3a00 con 0x7fa0a815aaa0 2023-10-16T12:34:23.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.469+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a8155d40 con 0x7fa0a815aaa0 2023-10-16T12:34:23.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.470+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a8157aa0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.471+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa09c0071d0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.472+0000 7fa08ffff640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa09406ae70 0x7fa09406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.473+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fa09c0893c0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.473+0000 7fa0a77fe640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa09406ae70 0x7fa09406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.474+0000 7fa0a77fe640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa09406ae70 0x7fa09406d330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa098001640 tx=0x7fa098009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.477+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0a8157aa0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.675+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0a8157aa0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.678+0000 7fa08ffff640 1 -- 172.21.15.60:0/290245314 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16714 (secure 0 0 0) 0x7fa0a8157aa0 con 0x7fa0a815aaa0 2023-10-16T12:34:23.681 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:23.681 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:11.506118+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:33:22.201537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:33:23.456249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:23.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.680+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa09406ae70 msgr2=0x7fa09406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.680+0000 7fa0ade51640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa09406ae70 0x7fa09406d330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa098001640 tx=0x7fa098009040 comp rx=0 tx=0).stop 2023-10-16T12:34:23.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.680+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 msgr2=0x7fa0a8155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.680+0000 7fa0ade51640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 0x7fa0a8155800 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00c750 tx=0x7fa09c00cc10 comp rx=0 tx=0).stop 2023-10-16T12:34:23.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 shutdown_connections 2023-10-16T12:34:23.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa09406ae70 0x7fa09406d330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0a815ae80 0x7fa0a8159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0a815aaa0 0x7fa0a8155800 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 --2- 172.21.15.60:0/290245314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a80ce390 0x7fa0a81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 >> 172.21.15.60:0/290245314 conn(0x7fa0a80cb410 msgr2=0x7fa0a80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:23.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 shutdown_connections 2023-10-16T12:34:23.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.681+0000 7fa0ade51640 1 -- 172.21.15.60:0/290245314 wait complete. 2023-10-16T12:34:23.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:34:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.925+0000 7f1e07fff640 1 Processor -- start 2023-10-16T12:34:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.926+0000 7f1e07fff640 1 -- start start 2023-10-16T12:34:23.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.927+0000 7f1e07fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e0815c9f0 0x7f1e080c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.927+0000 7f1e07fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1e0815dde0 0x7f1e080ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.927+0000 7f1e07fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e08160a90 con 0x7f1e080c9230 2023-10-16T12:34:23.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.927+0000 7f1e07fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e080cdec0 con 0x7f1e0815c9f0 2023-10-16T12:34:23.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.927+0000 7f1e07fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e080cd900 con 0x7f1e0815dde0 2023-10-16T12:34:23.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e0815c9f0 0x7f1e080c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e0815c9f0 0x7f1e080c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54828/0 (socket says 172.21.15.60:54828) 2023-10-16T12:34:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 -- 172.21.15.60:0/2144499131 learned_addr learned my addr 172.21.15.60:0/2144499131 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e077fe640 1 --2- 172.21.15.60:0/2144499131 >> v2:172.21.15.144:3300/0 conn(0x7f1e0815dde0 0x7f1e080ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 -- 172.21.15.60:0/2144499131 >> v2:172.21.15.144:3300/0 conn(0x7f1e0815dde0 msgr2=0x7f1e080ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 --2- 172.21.15.60:0/2144499131 >> v2:172.21.15.144:3300/0 conn(0x7f1e0815dde0 0x7f1e080ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 -- 172.21.15.60:0/2144499131 >> v1:172.21.15.144:6789/0 conn(0x7f1e080c9230 legacy=0x7f1e080709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.928+0000 7f1e06ffd640 1 -- 172.21.15.60:0/2144499131 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e080cdbe0 con 0x7f1e0815c9f0 2023-10-16T12:34:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.930+0000 7f1e06ffd640 1 --2- 172.21.15.60:0/2144499131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e080c8cf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1e080b0120 tx=0x7f1df802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be94c13935faf12 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.930+0000 7f1e05ffb640 1 -- 172.21.15.60:0/2144499131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8031870 con 0x7f1e0815c9f0 2023-10-16T12:34:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.930+0000 7f1e05ffb640 1 -- 172.21.15.60:0/2144499131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df80319d0 con 0x7f1e0815c9f0 2023-10-16T12:34:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.930+0000 7f1e05ffb640 1 -- 172.21.15.60:0/2144499131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8031d60 con 0x7f1e0815c9f0 2023-10-16T12:34:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 -- 172.21.15.60:0/2144499131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 msgr2=0x7f1e080c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2144499131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e080c8cf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1e080b0120 tx=0x7f1df802b910 comp rx=0 tx=0).stop 2023-10-16T12:34:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 -- 172.21.15.60:0/2144499131 shutdown_connections 2023-10-16T12:34:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2144499131 >> v2:172.21.15.144:3300/0 conn(0x7f1e0815dde0 0x7f1e080ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2144499131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e080c8cf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 -- 172.21.15.60:0/2144499131 >> 172.21.15.60:0/2144499131 conn(0x7f1e080ce6a0 msgr2=0x7f1e080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 -- 172.21.15.60:0/2144499131 shutdown_connections 2023-10-16T12:34:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.931+0000 7f1e07fff640 1 -- 172.21.15.60:0/2144499131 wait complete. 2023-10-16T12:34:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.933+0000 7f1e07fff640 1 Processor -- start 2023-10-16T12:34:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.933+0000 7f1e07fff640 1 -- start start 2023-10-16T12:34:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e080c9230 0x7f1e08146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e0806c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0815dde0 0x7f1e081f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e07fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080c7930 con 0x7f1e0815c9f0 2023-10-16T12:34:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e07fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080c1780 con 0x7f1e0815dde0 2023-10-16T12:34:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e07fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0806cce0 con 0x7f1e080c9230 2023-10-16T12:34:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e067fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e0806c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e067fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e0806c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54836/0 (socket says 172.21.15.60:54836) 2023-10-16T12:34:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.934+0000 7f1e067fc640 1 -- 172.21.15.60:0/2679521129 learned_addr learned my addr 172.21.15.60:0/2679521129 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e06ffd640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e080c9230 0x7f1e08146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e077fe640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0815dde0 0x7f1e081f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e067fc640 1 -- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e080c9230 msgr2=0x7f1e08146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e067fc640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e080c9230 0x7f1e08146600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e067fc640 1 -- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0815dde0 msgr2=0x7f1e081f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e067fc640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0815dde0 0x7f1e081f2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e067fc640 1 -- 172.21.15.60:0/2679521129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80094d0 con 0x7f1e0815c9f0 2023-10-16T12:34:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e077fe640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0815dde0 0x7f1e081f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1e067fc640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e0806c7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c490 tx=0x7f1dfc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.935+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc012830 con 0x7f1e0815c9f0 2023-10-16T12:34:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.936+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc012990 con 0x7f1e0815c9f0 2023-10-16T12:34:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.936+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e081f2cd0 con 0x7f1e0815c9f0 2023-10-16T12:34:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.936+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc012d40 con 0x7f1e0815c9f0 2023-10-16T12:34:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.936+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e081f3290 con 0x7f1e0815c9f0 2023-10-16T12:34:23.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.937+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08161670 con 0x7f1e0815c9f0 2023-10-16T12:34:23.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.938+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1dfc01b400 con 0x7f1e0815c9f0 2023-10-16T12:34:23.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.939+0000 7f1def7fe640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1df406ae70 0x7f1df406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.939+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f1dfc089600 con 0x7f1e0815c9f0 2023-10-16T12:34:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.939+0000 7f1e06ffd640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1df406ae70 0x7f1df406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.940+0000 7f1e06ffd640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1df406ae70 0x7f1df406d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1df8036000 tx=0x7f1df803c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:23.943+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1dfc054640 con 0x7f1e0815c9f0 2023-10-16T12:34:24.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.172+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1e080c7a70 con 0x7f1e0815c9f0 2023-10-16T12:34:24.175 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:34:24.173+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:34:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.177+0000 7f1def7fe640 1 -- 172.21.15.60:0/2679521129 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1e080c7a70 con 0x7f1e0815c9f0 2023-10-16T12:34:24.179 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:34:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.179+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1df406ae70 msgr2=0x7f1df406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:24.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.179+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1df406ae70 0x7f1df406d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1df8036000 tx=0x7f1df803c040 comp rx=0 tx=0).stop 2023-10-16T12:34:24.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.179+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 msgr2=0x7f1e0806c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:24.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.179+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e0806c7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c490 tx=0x7f1dfc00c950 comp rx=0 tx=0).stop 2023-10-16T12:34:24.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.179+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 shutdown_connections 2023-10-16T12:34:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1df406ae70 0x7f1df406d330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0815dde0 0x7f1e081f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0815c9f0 0x7f1e0806c7a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 --2- 172.21.15.60:0/2679521129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e080c9230 0x7f1e08146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 >> 172.21.15.60:0/2679521129 conn(0x7f1e080ce6a0 msgr2=0x7f1e08160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 shutdown_connections 2023-10-16T12:34:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.180+0000 7f1e07fff640 1 -- 172.21.15.60:0/2679521129 wait complete. 2023-10-16T12:34:24.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:34:24.220 INFO:tasks.ceph.mds.d.smithi144.stderr: -7516> 2023-10-16T12:33:28.040+0000 7fcaacd3a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:34:24.312 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:34:24.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.411+0000 7f6792380640 1 Processor -- start 2023-10-16T12:34:24.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.412+0000 7f6792380640 1 -- start start 2023-10-16T12:34:24.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.413+0000 7f6792380640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f678c15cd70 0x7f678c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:24.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.413+0000 7f6792380640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0cded0 0x7f678c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:24.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.413+0000 7f6792380640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c15d150 con 0x7f678c0b8480 2023-10-16T12:34:24.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.413+0000 7f6792380640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c0d1180 con 0x7f678c15cd70 2023-10-16T12:34:24.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.413+0000 7f6792380640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0d1460 con 0x7f678c0cded0 2023-10-16T12:34:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.414+0000 7f6791b7f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0cded0 0x7f678c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.414+0000 7f679137e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f678c15cd70 0x7f678c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.414+0000 7f6791b7f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0cded0 0x7f678c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:59410/0 (socket says 172.21.15.60:59410) 2023-10-16T12:34:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.414+0000 7f6791b7f640 1 -- 172.21.15.60:0/2411991270 learned_addr learned my addr 172.21.15.60:0/2411991270 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.415+0000 7f679137e640 1 -- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0cded0 msgr2=0x7f678c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.415+0000 7f679137e640 1 --2- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0cded0 0x7f678c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.415+0000 7f679137e640 1 -- 172.21.15.60:0/2411991270 >> v1:172.21.15.144:6789/0 conn(0x7f678c0b8480 legacy=0x7f678c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.415+0000 7f679137e640 1 -- 172.21.15.60:0/2411991270 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0d1740 con 0x7f678c15cd70 2023-10-16T12:34:24.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.416+0000 7f679137e640 1 --2- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c15cd70 0x7f678c0d2220 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f67880097a0 tx=0x7f678802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f99d361f9de314 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:24.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.416+0000 7f6783fff640 1 -- 172.21.15.60:0/2411991270 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678802d040 con 0x7f678c15cd70 2023-10-16T12:34:24.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6783fff640 1 -- 172.21.15.60:0/2411991270 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6788031810 con 0x7f678c15cd70 2023-10-16T12:34:24.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6783fff640 1 -- 172.21.15.60:0/2411991270 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788031ba0 con 0x7f678c15cd70 2023-10-16T12:34:24.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6792380640 1 -- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c15cd70 msgr2=0x7f678c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6792380640 1 --2- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c15cd70 0x7f678c0d2220 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f67880097a0 tx=0x7f678802b8b0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6792380640 1 -- 172.21.15.60:0/2411991270 shutdown_connections 2023-10-16T12:34:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6792380640 1 --2- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0cded0 0x7f678c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6792380640 1 --2- 172.21.15.60:0/2411991270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c15cd70 0x7f678c0d2220 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.417+0000 7f6792380640 1 -- 172.21.15.60:0/2411991270 >> 172.21.15.60:0/2411991270 conn(0x7f678c0cf570 msgr2=0x7f678c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.418+0000 7f6792380640 1 -- 172.21.15.60:0/2411991270 shutdown_connections 2023-10-16T12:34:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.418+0000 7f6792380640 1 -- 172.21.15.60:0/2411991270 wait complete. 2023-10-16T12:34:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.420+0000 7f6792380640 1 Processor -- start 2023-10-16T12:34:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.420+0000 7f6792380640 1 -- start start 2023-10-16T12:34:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.421+0000 7f6792380640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0b8480 0x7f678c15fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.421+0000 7f6792380640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0cded0 0x7f678c14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.421+0000 7f6792380640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 0x7f678c156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.421+0000 7f6792380640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c160340 con 0x7f678c15cd70 2023-10-16T12:34:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.421+0000 7f6792380640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15ed60 con 0x7f678c0cded0 2023-10-16T12:34:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.421+0000 7f6792380640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15f040 con 0x7f678c0b8480 2023-10-16T12:34:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 0x7f678c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 0x7f678c156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54838/0 (socket says 172.21.15.60:54838) 2023-10-16T12:34:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 -- 172.21.15.60:0/2408366139 learned_addr learned my addr 172.21.15.60:0/2408366139 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f679137e640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0b8480 0x7f678c15fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6790b7d640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0cded0 0x7f678c14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 -- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0b8480 msgr2=0x7f678c15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0b8480 0x7f678c15fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 -- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0cded0 msgr2=0x7f678c14cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0cded0 0x7f678c14cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 -- 172.21.15.60:0/2408366139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788009400 con 0x7f678c15cd70 2023-10-16T12:34:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6790b7d640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0cded0 0x7f678c14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.422+0000 7f6791b7f640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 0x7f678c156770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f678400efc0 tx=0x7f678400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.423+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678400cd20 con 0x7f678c15cd70 2023-10-16T12:34:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.423+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678400f040 con 0x7f678c15cd70 2023-10-16T12:34:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.423+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678400cea0 con 0x7f678c15cd70 2023-10-16T12:34:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.423+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c15f320 con 0x7f678c15cd70 2023-10-16T12:34:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.423+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c15f8e0 con 0x7f678c15cd70 2023-10-16T12:34:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.424+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c04e9b0 con 0x7f678c15cd70 2023-10-16T12:34:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.425+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6784003bd0 con 0x7f678c15cd70 2023-10-16T12:34:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.426+0000 7f6781ffb640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f677006ae70 0x7f677006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.426+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f6784089d80 con 0x7f678c15cd70 2023-10-16T12:34:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.427+0000 7f679137e640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f677006ae70 0x7f677006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:24.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.428+0000 7f679137e640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f677006ae70 0x7f677006d330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6788036000 tx=0x7f678803c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:24.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.431+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6784054cd0 con 0x7f678c15cd70 2023-10-16T12:34:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:24.671+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f678c0b4220 con 0x7f678c15cd70 2023-10-16T12:34:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.180+0000 7f6781ffb640 1 -- 172.21.15.60:0/2408366139 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f67840582f0 con 0x7f678c15cd70 2023-10-16T12:34:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f677006ae70 msgr2=0x7f677006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f677006ae70 0x7f677006d330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6788036000 tx=0x7f678803c040 comp rx=0 tx=0).stop 2023-10-16T12:34:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 msgr2=0x7f678c156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 0x7f678c156770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f678400efc0 tx=0x7f678400c490 comp rx=0 tx=0).stop 2023-10-16T12:34:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 shutdown_connections 2023-10-16T12:34:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f677006ae70 0x7f677006d330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c15cd70 0x7f678c156770 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0cded0 0x7f678c14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 --2- 172.21.15.60:0/2408366139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0b8480 0x7f678c15fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.184+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 >> 172.21.15.60:0/2408366139 conn(0x7f678c0cf570 msgr2=0x7f678c0d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.185+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 shutdown_connections 2023-10-16T12:34:25.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.185+0000 7f6792380640 1 -- 172.21.15.60:0/2408366139 wait complete. 2023-10-16T12:34:25.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:34:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.437+0000 7fe1aed01640 1 Processor -- start 2023-10-16T12:34:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.437+0000 7fe1aed01640 1 -- start start 2023-10-16T12:34:25.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1aed01640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a815cf60 0x7fe1a806f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:25.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1aed01640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe1a806f9d0 0x7fe1a80c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:25.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1aed01640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80c9620 con 0x7fe1a80c9ff0 2023-10-16T12:34:25.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1aed01640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a815d340 con 0x7fe1a806f9d0 2023-10-16T12:34:25.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1aed01640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80cc860 con 0x7fe1a815cf60 2023-10-16T12:34:25.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1a7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe1a806f9d0 0x7fe1a80c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:25.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1a7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe1a806f9d0 0x7fe1a80c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54840/0 (socket says 172.21.15.60:54840) 2023-10-16T12:34:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.439+0000 7fe1a7fff640 1 -- 172.21.15.60:0/964578667 learned_addr learned my addr 172.21.15.60:0/964578667 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.440+0000 7fe1aca9d640 1 --2- 172.21.15.60:0/964578667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a815cf60 0x7fe1a806f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.440+0000 7fe1a7fff640 1 -- 172.21.15.60:0/964578667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a815cf60 msgr2=0x7fe1a806f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.440+0000 7fe1a7fff640 1 --2- 172.21.15.60:0/964578667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a815cf60 0x7fe1a806f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.440+0000 7fe1a7fff640 1 -- 172.21.15.60:0/964578667 >> v1:172.21.15.144:6789/0 conn(0x7fe1a80c9ff0 legacy=0x7fe1a80b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:25.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.440+0000 7fe1a7fff640 1 -- 172.21.15.60:0/964578667 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80ad4a0 con 0x7fe1a806f9d0 2023-10-16T12:34:25.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.441+0000 7fe1a7fff640 1 --2- 172.21.15.60:0/964578667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a806f9d0 0x7fe1a80c9ab0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe194009870 tx=0x7fe19402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23395cfe892bc47f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:25.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.441+0000 7fe1a77fe640 1 -- 172.21.15.60:0/964578667 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194031640 con 0x7fe1a806f9d0 2023-10-16T12:34:25.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.441+0000 7fe1a77fe640 1 -- 172.21.15.60:0/964578667 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1940317a0 con 0x7fe1a806f9d0 2023-10-16T12:34:25.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1a77fe640 1 -- 172.21.15.60:0/964578667 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194031b30 con 0x7fe1a806f9d0 2023-10-16T12:34:25.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 -- 172.21.15.60:0/964578667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a806f9d0 msgr2=0x7fe1a80c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:25.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 --2- 172.21.15.60:0/964578667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a806f9d0 0x7fe1a80c9ab0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe194009870 tx=0x7fe19402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 -- 172.21.15.60:0/964578667 shutdown_connections 2023-10-16T12:34:25.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 --2- 172.21.15.60:0/964578667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a806f9d0 0x7fe1a80c9ab0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 --2- 172.21.15.60:0/964578667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a815cf60 0x7fe1a806f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 -- 172.21.15.60:0/964578667 >> 172.21.15.60:0/964578667 conn(0x7fe1a80cb360 msgr2=0x7fe1a80cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:25.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.442+0000 7fe1aed01640 1 -- 172.21.15.60:0/964578667 shutdown_connections 2023-10-16T12:34:25.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.443+0000 7fe1aed01640 1 -- 172.21.15.60:0/964578667 wait complete. 2023-10-16T12:34:25.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.444+0000 7fe1aed01640 1 Processor -- start 2023-10-16T12:34:25.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.444+0000 7fe1aed01640 1 -- start start 2023-10-16T12:34:25.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.445+0000 7fe1aed01640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a806f9d0 0x7fe1a8158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:25.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.445+0000 7fe1aed01640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a80c9ff0 0x7fe1a80c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:25.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.445+0000 7fe1aed01640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 0x7fe1a815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:25.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.445+0000 7fe1aed01640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8159620 con 0x7fe1a815cf60 2023-10-16T12:34:25.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.445+0000 7fe1aed01640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8160000 con 0x7fe1a80c9ff0 2023-10-16T12:34:25.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.445+0000 7fe1aed01640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a81602e0 con 0x7fe1a806f9d0 2023-10-16T12:34:25.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 0x7fe1a815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:25.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 0x7fe1a815d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54856/0 (socket says 172.21.15.60:54856) 2023-10-16T12:34:25.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 -- 172.21.15.60:0/1895189018 learned_addr learned my addr 172.21.15.60:0/1895189018 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:25.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 -- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a806f9d0 msgr2=0x7fe1a8158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:25.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a806f9d0 0x7fe1a8158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 -- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a80c9ff0 msgr2=0x7fe1a80c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:25.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a80c9ff0 0x7fe1a80c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:25.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 -- 172.21.15.60:0/1895189018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940094d0 con 0x7fe1a815cf60 2023-10-16T12:34:25.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.446+0000 7fe1ad29e640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 0x7fe1a815d940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000b970 tx=0x7fe1a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:25.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.447+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a000cb40 con 0x7fe1a815cf60 2023-10-16T12:34:25.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.447+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a000cca0 con 0x7fe1a815cf60 2023-10-16T12:34:25.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.447+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0014680 con 0x7fe1a815cf60 2023-10-16T12:34:25.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.447+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a80b3220 con 0x7fe1a815cf60 2023-10-16T12:34:25.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.447+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a80b37e0 con 0x7fe1a815cf60 2023-10-16T12:34:25.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.448+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a80b39d0 con 0x7fe1a815cf60 2023-10-16T12:34:25.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.449+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe1a001d3b0 con 0x7fe1a815cf60 2023-10-16T12:34:25.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.450+0000 7fe1a57fa640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe18c06ae70 0x7fe18c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:25.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.450+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fe1a00895b0 con 0x7fe1a815cf60 2023-10-16T12:34:25.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.451+0000 7fe1aca9d640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe18c06ae70 0x7fe18c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:25.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.452+0000 7fe1aca9d640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe18c06ae70 0x7fe18c06d330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe19c006fd0 tx=0x7fe19c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:25.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.455+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1a0054500 con 0x7fe1a815cf60 2023-10-16T12:34:25.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:25.661+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe1a804e160 con 0x7fe1a815cf60 2023-10-16T12:34:26.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.626+0000 7fe1a57fa640 1 -- 172.21.15.60:0/1895189018 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe1a0057b20 con 0x7fe1a815cf60 2023-10-16T12:34:26.629 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:34:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe18c06ae70 msgr2=0x7fe18c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe18c06ae70 0x7fe18c06d330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe19c006fd0 tx=0x7fe19c008040 comp rx=0 tx=0).stop 2023-10-16T12:34:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 msgr2=0x7fe1a815d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 0x7fe1a815d940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000b970 tx=0x7fe1a000be30 comp rx=0 tx=0).stop 2023-10-16T12:34:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 shutdown_connections 2023-10-16T12:34:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe18c06ae70 0x7fe18c06d330 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a815cf60 0x7fe1a815d940 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1a80c9ff0 0x7fe1a80c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 --2- 172.21.15.60:0/1895189018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1a806f9d0 0x7fe1a8158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.629+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 >> 172.21.15.60:0/1895189018 conn(0x7fe1a80cb360 msgr2=0x7fe1a80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.630+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 shutdown_connections 2023-10-16T12:34:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.630+0000 7fe1aed01640 1 -- 172.21.15.60:0/1895189018 wait complete. 2023-10-16T12:34:26.658 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:34:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.891+0000 7f11cefbf640 1 Processor -- start 2023-10-16T12:34:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.891+0000 7f11cefbf640 1 -- start start 2023-10-16T12:34:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.893+0000 7f11cefbf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f11c815cd70 0x7f11c80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.893+0000 7f11cefbf640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11c80cded0 0x7f11c80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.893+0000 7f11cefbf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c815d150 con 0x7f11c80b8480 2023-10-16T12:34:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.893+0000 7f11cefbf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80d1180 con 0x7f11c815cd70 2023-10-16T12:34:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.893+0000 7f11cefbf640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80d1460 con 0x7f11c80cded0 2023-10-16T12:34:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.893+0000 7f11cdfbd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f11c815cd70 0x7f11c80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11cdfbd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f11c815cd70 0x7f11c80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54870/0 (socket says 172.21.15.60:54870) 2023-10-16T12:34:26.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11cdfbd640 1 -- 172.21.15.60:0/986952244 learned_addr learned my addr 172.21.15.60:0/986952244 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:26.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11ce7be640 1 --2- 172.21.15.60:0/986952244 >> v2:172.21.15.144:3300/0 conn(0x7f11c80cded0 0x7f11c80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:26.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11cdfbd640 1 -- 172.21.15.60:0/986952244 >> v2:172.21.15.144:3300/0 conn(0x7f11c80cded0 msgr2=0x7f11c80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:26.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11cdfbd640 1 --2- 172.21.15.60:0/986952244 >> v2:172.21.15.144:3300/0 conn(0x7f11c80cded0 0x7f11c80b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11cdfbd640 1 -- 172.21.15.60:0/986952244 >> v1:172.21.15.144:6789/0 conn(0x7f11c80b8480 legacy=0x7f11c80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:26.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11cdfbd640 1 -- 172.21.15.60:0/986952244 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80d1740 con 0x7f11c815cd70 2023-10-16T12:34:26.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.894+0000 7f11ce7be640 1 --2- 172.21.15.60:0/986952244 >> v2:172.21.15.144:3300/0 conn(0x7f11c80cded0 0x7f11c80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:26.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.895+0000 7f11cdfbd640 1 --2- 172.21.15.60:0/986952244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c815cd70 0x7f11c80d2220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f11c80b8910 tx=0x7f11c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea5b4db67756ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:26.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.896+0000 7f11ccfbb640 1 -- 172.21.15.60:0/986952244 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4031870 con 0x7f11c815cd70 2023-10-16T12:34:26.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.896+0000 7f11ccfbb640 1 -- 172.21.15.60:0/986952244 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c40319d0 con 0x7f11c815cd70 2023-10-16T12:34:26.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.896+0000 7f11ccfbb640 1 -- 172.21.15.60:0/986952244 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4031d60 con 0x7f11c815cd70 2023-10-16T12:34:26.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.896+0000 7f11cefbf640 1 -- 172.21.15.60:0/986952244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c815cd70 msgr2=0x7f11c80d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:26.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.896+0000 7f11cefbf640 1 --2- 172.21.15.60:0/986952244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c815cd70 0x7f11c80d2220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f11c80b8910 tx=0x7f11c402b910 comp rx=0 tx=0).stop 2023-10-16T12:34:26.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.897+0000 7f11cefbf640 1 -- 172.21.15.60:0/986952244 shutdown_connections 2023-10-16T12:34:26.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.897+0000 7f11cefbf640 1 --2- 172.21.15.60:0/986952244 >> v2:172.21.15.144:3300/0 conn(0x7f11c80cded0 0x7f11c80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.897+0000 7f11cefbf640 1 --2- 172.21.15.60:0/986952244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c815cd70 0x7f11c80d2220 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.897+0000 7f11cefbf640 1 -- 172.21.15.60:0/986952244 >> 172.21.15.60:0/986952244 conn(0x7f11c80cf570 msgr2=0x7f11c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.897+0000 7f11cefbf640 1 -- 172.21.15.60:0/986952244 shutdown_connections 2023-10-16T12:34:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.897+0000 7f11cefbf640 1 -- 172.21.15.60:0/986952244 wait complete. 2023-10-16T12:34:26.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.898+0000 7f11cefbf640 1 Processor -- start 2023-10-16T12:34:26.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.899+0000 7f11cefbf640 1 -- start start 2023-10-16T12:34:26.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cefbf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11c80b8480 0x7f11c80cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:26.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cefbf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 0x7f11c815ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:26.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cefbf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11c815cd70 0x7f11c815f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:26.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cefbf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c815df30 con 0x7f11c80cded0 2023-10-16T12:34:26.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cefbf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c815e350 con 0x7f11c80b8480 2023-10-16T12:34:26.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cefbf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c815d340 con 0x7f11c815cd70 2023-10-16T12:34:26.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 0x7f11c815ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:26.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 0x7f11c815ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54872/0 (socket says 172.21.15.60:54872) 2023-10-16T12:34:26.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 -- 172.21.15.60:0/1734136871 learned_addr learned my addr 172.21.15.60:0/1734136871 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:26.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cdfbd640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11c80b8480 0x7f11c80cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:26.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11ce7be640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11c815cd70 0x7f11c815f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:26.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 -- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11c815cd70 msgr2=0x7f11c815f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:26.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11c815cd70 0x7f11c815f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 -- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11c80b8480 msgr2=0x7f11c80cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:26.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.900+0000 7f11cd7bc640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11c80b8480 0x7f11c80cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:26.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11cd7bc640 1 -- 172.21.15.60:0/1734136871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40094d0 con 0x7f11c80cded0 2023-10-16T12:34:26.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11cdfbd640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11c80b8480 0x7f11c80cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:26.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11cd7bc640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 0x7f11c815ee70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c430 tx=0x7f11b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:26.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8011790 con 0x7f11c80cded0 2023-10-16T12:34:26.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b80118f0 con 0x7f11c80cded0 2023-10-16T12:34:26.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8011ce0 con 0x7f11c80cded0 2023-10-16T12:34:26.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c815f8f0 con 0x7f11c80cded0 2023-10-16T12:34:26.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.901+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c815d870 con 0x7f11c80cded0 2023-10-16T12:34:26.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.902+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c815e070 con 0x7f11c80cded0 2023-10-16T12:34:26.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.903+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f11b801c460 con 0x7f11c80cded0 2023-10-16T12:34:26.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.904+0000 7f11be7fc640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f11ac06ae70 0x7f11ac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:26.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.904+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5452+0+0 (secure 0 0 0) 0x7f11b8089470 con 0x7f11c80cded0 2023-10-16T12:34:26.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.905+0000 7f11cdfbd640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f11ac06ae70 0x7f11ac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:26.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.906+0000 7f11cdfbd640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f11ac06ae70 0x7f11ac06d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f11c4036000 tx=0x7f11c403c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:26.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:26.909+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11c815e070 con 0x7f11c80cded0 2023-10-16T12:34:27.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.108+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f11c815e070 con 0x7f11c80cded0 2023-10-16T12:34:27.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.629+0000 7f11be7fc640 1 -- 172.21.15.60:0/1734136871 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f11c815e070 con 0x7f11c80cded0 2023-10-16T12:34:27.632 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:34:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f11ac06ae70 msgr2=0x7f11ac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f11ac06ae70 0x7f11ac06d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f11c4036000 tx=0x7f11c403c040 comp rx=0 tx=0).stop 2023-10-16T12:34:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 msgr2=0x7f11c815ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 0x7f11c815ee70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c430 tx=0x7f11b800c8f0 comp rx=0 tx=0).stop 2023-10-16T12:34:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 shutdown_connections 2023-10-16T12:34:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f11ac06ae70 0x7f11ac06d330 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11c815cd70 0x7f11c815f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11c80cded0 0x7f11c815ee70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 --2- 172.21.15.60:0/1734136871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11c80b8480 0x7f11c80cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.633+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 >> 172.21.15.60:0/1734136871 conn(0x7f11c80cf570 msgr2=0x7f11c80d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.634+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 shutdown_connections 2023-10-16T12:34:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:27.634+0000 7f11cefbf640 1 -- 172.21.15.60:0/1734136871 wait complete. 2023-10-16T12:34:27.663 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:34:27.663 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:34:27.664 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:34:27.664 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:34:27.664 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:34:27.664 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:34:27.665 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:34:27.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:34:27.665 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:34:27.665 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:34:27.666 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:34:27.666 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:34:27.666 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:34:27.667 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:34:27.667 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:34:27.667 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:34:27.685 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:34:27.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:34:27.687 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:34:27.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:34:27.689 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:34:27.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:34:27.735 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:34:27.736 DEBUG:teuthology.parallel:result is None 2023-10-16T12:34:27.738 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:34:27.738 DEBUG:teuthology.parallel:result is None 2023-10-16T12:34:27.743 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:34:27.744 DEBUG:teuthology.parallel:result is None 2023-10-16T12:34:27.906 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:34:27.906 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:34:27.918 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:34:32.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-10-16T12:34:33.689 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:34:33.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:34:33.697 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:34:33.698 DEBUG:teuthology.parallel:result is None 2023-10-16T12:34:33.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:34:33.883 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:34:33.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.936+0000 7f3814747640 1 Processor -- start 2023-10-16T12:34:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.936+0000 7f3814747640 1 -- start start 2023-10-16T12:34:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3814747640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0addb0 0x7f380c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3814747640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f380c0c9000 0x7f380c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3814747640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c0cb140 con 0x7f380c06a2e0 2023-10-16T12:34:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3814747640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c161a50 con 0x7f380c0c9000 2023-10-16T12:34:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3814747640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c06a750 con 0x7f380c0addb0 2023-10-16T12:34:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3812ce4640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f380c06a2e0 0x7f380c0cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49792/0 (socket says 172.21.15.60:49792) 2023-10-16T12:34:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3812ce4640 1 -- 172.21.15.60:0/854144988 learned_addr learned my addr 172.21.15.60:0/854144988 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f3811ce2640 1 --2- 172.21.15.60:0/854144988 >> v2:172.21.15.60:3300/0 conn(0x7f380c0c9000 0x7f380c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.938+0000 7f38124e3640 1 --2- 172.21.15.60:0/854144988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0addb0 0x7f380c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.939+0000 7f38114e1640 1 -- 172.21.15.60:0/854144988 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592084769 0 0) 0x7f380c0cb140 con 0x7f380c06a2e0 2023-10-16T12:34:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.939+0000 7f38114e1640 1 -- 172.21.15.60:0/854144988 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f8003520 con 0x7f380c06a2e0 2023-10-16T12:34:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.939+0000 7f3811ce2640 1 -- 172.21.15.60:0/854144988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0addb0 msgr2=0x7f380c0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.939+0000 7f3811ce2640 1 --2- 172.21.15.60:0/854144988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0addb0 0x7f380c0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.939+0000 7f3811ce2640 1 -- 172.21.15.60:0/854144988 >> v1:172.21.15.60:6789/0 conn(0x7f380c06a2e0 legacy=0x7f380c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.939+0000 7f3811ce2640 1 -- 172.21.15.60:0/854144988 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c06b030 con 0x7f380c0c9000 2023-10-16T12:34:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.941+0000 7f3811ce2640 1 --2- 172.21.15.60:0/854144988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c0c9000 0x7f380c069da0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0097d0 tx=0x7f37fc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1bc06b9d7e4953 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.941+0000 7f38114e1640 1 -- 172.21.15.60:0/854144988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc031800 con 0x7f380c0c9000 2023-10-16T12:34:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.941+0000 7f38114e1640 1 -- 172.21.15.60:0/854144988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc031960 con 0x7f380c0c9000 2023-10-16T12:34:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.941+0000 7f38114e1640 1 -- 172.21.15.60:0/854144988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc031cf0 con 0x7f380c0c9000 2023-10-16T12:34:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 -- 172.21.15.60:0/854144988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c0c9000 msgr2=0x7f380c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 --2- 172.21.15.60:0/854144988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c0c9000 0x7f380c069da0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0097d0 tx=0x7f37fc02b8a0 comp rx=0 tx=0).stop 2023-10-16T12:34:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 -- 172.21.15.60:0/854144988 shutdown_connections 2023-10-16T12:34:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 --2- 172.21.15.60:0/854144988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c0c9000 0x7f380c069da0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 --2- 172.21.15.60:0/854144988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0addb0 0x7f380c0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 -- 172.21.15.60:0/854144988 >> 172.21.15.60:0/854144988 conn(0x7f380c0ce9b0 msgr2=0x7f380c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 -- 172.21.15.60:0/854144988 shutdown_connections 2023-10-16T12:34:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.942+0000 7f3814747640 1 -- 172.21.15.60:0/854144988 wait complete. 2023-10-16T12:34:33.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.944+0000 7f3814747640 1 Processor -- start 2023-10-16T12:34:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.944+0000 7f3814747640 1 -- start start 2023-10-16T12:34:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3814747640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 0x7f380c163f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3814747640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f380c0addb0 0x7f380c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3814747640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0c9000 0x7f380c14fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3814747640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0c5a80 con 0x7f380c06a2e0 2023-10-16T12:34:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3814747640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c163760 con 0x7f380c0addb0 2023-10-16T12:34:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3814747640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0c5c90 con 0x7f380c0c9000 2023-10-16T12:34:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 0x7f380c163f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 0x7f380c163f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44456/0 (socket says 172.21.15.60:44456) 2023-10-16T12:34:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 -- 172.21.15.60:0/3322917940 learned_addr learned my addr 172.21.15.60:0/3322917940 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3811ce2640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f380c0addb0 0x7f380c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f3812ce4640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0c9000 0x7f380c14fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 -- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0c9000 msgr2=0x7f380c14fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0c9000 0x7f380c14fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 -- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f380c0addb0 msgr2=0x7f380c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.945+0000 7f38124e3640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f380c0addb0 0x7f380c0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f38124e3640 1 -- 172.21.15.60:0/3322917940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc009400 con 0x7f380c06a2e0 2023-10-16T12:34:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f3811ce2640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f380c0addb0 0x7f380c0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f38124e3640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 0x7f380c163f20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f380800efc0 tx=0x7f380800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808014480 con 0x7f380c06a2e0 2023-10-16T12:34:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380800f040 con 0x7f380c06a2e0 2023-10-16T12:34:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808014800 con 0x7f380c06a2e0 2023-10-16T12:34:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c0c5f70 con 0x7f380c06a2e0 2023-10-16T12:34:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.946+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c0c6250 con 0x7f380c06a2e0 2023-10-16T12:34:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.947+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c04e160 con 0x7f380c06a2e0 2023-10-16T12:34:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.948+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3808014960 con 0x7f380c06a2e0 2023-10-16T12:34:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.949+0000 7f3802ffd640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f37f006aef0 0x7f37f006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.949+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3808089e20 con 0x7f380c06a2e0 2023-10-16T12:34:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.950+0000 7f3811ce2640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f37f006aef0 0x7f37f006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.951+0000 7f3811ce2640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f37f006aef0 0x7f37f006d3b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37fc036000 tx=0x7f37fc03c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:33.954+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3808055240 con 0x7f380c06a2e0 2023-10-16T12:34:34.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:34.166+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f380c06c5b0 con 0x7f380c06a2e0 2023-10-16T12:34:35.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.398+0000 7f3802ffd640 1 -- 172.21.15.60:0/3322917940 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3808058860 con 0x7f380c06a2e0 2023-10-16T12:34:35.401 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:34:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f37f006aef0 msgr2=0x7f37f006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f37f006aef0 0x7f37f006d3b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37fc036000 tx=0x7f37fc03c040 comp rx=0 tx=0).stop 2023-10-16T12:34:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 msgr2=0x7f380c163f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 0x7f380c163f20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f380800efc0 tx=0x7f380800c490 comp rx=0 tx=0).stop 2023-10-16T12:34:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 shutdown_connections 2023-10-16T12:34:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f37f006aef0 0x7f37f006d3b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f380c0c9000 0x7f380c14fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f380c0addb0 0x7f380c0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 --2- 172.21.15.60:0/3322917940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f380c06a2e0 0x7f380c163f20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 >> 172.21.15.60:0/3322917940 conn(0x7f380c0ce9b0 msgr2=0x7f380c0ae4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 shutdown_connections 2023-10-16T12:34:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.402+0000 7f3814747640 1 -- 172.21.15.60:0/3322917940 wait complete. 2023-10-16T12:34:35.425 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:34:35.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.649+0000 7f127759e640 1 Processor -- start 2023-10-16T12:34:35.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.650+0000 7f127759e640 1 -- start start 2023-10-16T12:34:35.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.651+0000 7f127759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f127806c040 0x7f127806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.651+0000 7f127759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12780ad970 0x7f127815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.651+0000 7f127759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127806c960 con 0x7f127815ebc0 2023-10-16T12:34:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.651+0000 7f127759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127815b9d0 con 0x7f12780ad970 2023-10-16T12:34:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.651+0000 7f127759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127815bcb0 con 0x7f127806c040 2023-10-16T12:34:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12780ad970 0x7f127815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12780ad970 0x7f127815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44458/0 (socket says 172.21.15.60:44458) 2023-10-16T12:34:35.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 -- 172.21.15.60:0/3577603725 learned_addr learned my addr 172.21.15.60:0/3577603725 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:35.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f127659c640 1 --2- 172.21.15.60:0/3577603725 >> v2:172.21.15.144:3300/0 conn(0x7f127806c040 0x7f127806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:35.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 -- 172.21.15.60:0/3577603725 >> v2:172.21.15.144:3300/0 conn(0x7f127806c040 msgr2=0x7f127806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:35.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 --2- 172.21.15.60:0/3577603725 >> v2:172.21.15.144:3300/0 conn(0x7f127806c040 0x7f127806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 -- 172.21.15.60:0/3577603725 >> v1:172.21.15.144:6789/0 conn(0x7f127815ebc0 legacy=0x7f12780cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.652+0000 7f126dd9b640 1 -- 172.21.15.60:0/3577603725 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127815bf90 con 0x7f12780ad970 2023-10-16T12:34:35.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.654+0000 7f126dd9b640 1 --2- 172.21.15.60:0/3577603725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12780ad970 0x7f127815e680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f12780cf350 tx=0x7f126002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4c3b386458c137 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:35.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.654+0000 7f1275d9b640 1 -- 172.21.15.60:0/3577603725 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260030860 con 0x7f12780ad970 2023-10-16T12:34:35.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.654+0000 7f1275d9b640 1 -- 172.21.15.60:0/3577603725 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12600309c0 con 0x7f12780ad970 2023-10-16T12:34:35.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.654+0000 7f1275d9b640 1 -- 172.21.15.60:0/3577603725 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260030d50 con 0x7f12780ad970 2023-10-16T12:34:35.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 -- 172.21.15.60:0/3577603725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12780ad970 msgr2=0x7f127815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 --2- 172.21.15.60:0/3577603725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12780ad970 0x7f127815e680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f12780cf350 tx=0x7f126002b9a0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 -- 172.21.15.60:0/3577603725 shutdown_connections 2023-10-16T12:34:35.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 --2- 172.21.15.60:0/3577603725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12780ad970 0x7f127815e680 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 --2- 172.21.15.60:0/3577603725 >> v2:172.21.15.144:3300/0 conn(0x7f127806c040 0x7f127806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 -- 172.21.15.60:0/3577603725 >> 172.21.15.60:0/3577603725 conn(0x7f12780ce6a0 msgr2=0x7f12780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:35.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 -- 172.21.15.60:0/3577603725 shutdown_connections 2023-10-16T12:34:35.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.655+0000 7f127759e640 1 -- 172.21.15.60:0/3577603725 wait complete. 2023-10-16T12:34:35.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.657+0000 7f127759e640 1 Processor -- start 2023-10-16T12:34:35.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.657+0000 7f127759e640 1 -- start start 2023-10-16T12:34:35.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 0x7f12780af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:35.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12780ad970 0x7f1278151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:35.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127815ebc0 0x7f12780b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:35.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278152120 con 0x7f127806c040 2023-10-16T12:34:35.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12780b26d0 con 0x7f127815ebc0 2023-10-16T12:34:35.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12780b1ac0 con 0x7f12780ad970 2023-10-16T12:34:35.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 0x7f12780af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:35.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 0x7f12780af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44462/0 (socket says 172.21.15.60:44462) 2023-10-16T12:34:35.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.658+0000 7f127659c640 1 -- 172.21.15.60:0/2630862666 learned_addr learned my addr 172.21.15.60:0/2630862666 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:35.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f127659c640 1 -- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12780ad970 msgr2=0x7f1278151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f126dd9b640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12780ad970 0x7f1278151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:35.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f127659c640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12780ad970 0x7f1278151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f127659c640 1 -- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127815ebc0 msgr2=0x7f12780b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f127659c640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127815ebc0 0x7f12780b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f127659c640 1 -- 172.21.15.60:0/2630862666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600094d0 con 0x7f127806c040 2023-10-16T12:34:35.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.659+0000 7f127659c640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 0x7f12780af360 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1268009d30 tx=0x7f126800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:35.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.660+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268009ba0 con 0x7f127806c040 2023-10-16T12:34:35.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.660+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126800f040 con 0x7f127806c040 2023-10-16T12:34:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.660+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680146c0 con 0x7f127806c040 2023-10-16T12:34:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.660+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12780b1da0 con 0x7f127806c040 2023-10-16T12:34:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.660+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12780b2360 con 0x7f127806c040 2023-10-16T12:34:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.661+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127806b330 con 0x7f127806c040 2023-10-16T12:34:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.662+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1268014820 con 0x7f127806c040 2023-10-16T12:34:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.663+0000 7f126f7fe640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f125806ae70 0x7f125806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.663+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1268089f80 con 0x7f127806c040 2023-10-16T12:34:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.664+0000 7f126dd9b640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f125806ae70 0x7f125806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.665+0000 7f126dd9b640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f125806ae70 0x7f125806d330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1260036000 tx=0x7f126003b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.668+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12680552e0 con 0x7f127806c040 2023-10-16T12:34:35.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.934+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f127804e9b0 con 0x7f127806c040 2023-10-16T12:34:35.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.936+0000 7f126f7fe640 1 -- 172.21.15.60:0/2630862666 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f1268058900 con 0x7f127806c040 2023-10-16T12:34:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:35.939 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:34:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:34:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:34:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:34:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:34:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:34:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:34:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:34:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:34:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:34:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:34:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:34:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:34:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:34:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:34:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:34:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:34:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:34:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:34:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:34:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:34:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:34:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:34:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:34:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:34:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:34:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:34:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:34:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:34:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:34:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:34:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:34:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:34:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:34:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:34:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:34:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:34:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:34:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:34:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:34:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:34:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:34:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:34:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:34:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:34:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:34:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:34:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:34:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:34:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:34:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:34:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:34:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:34:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:34:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:34:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:34:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:34:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:34:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:34:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:34:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:34:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:34:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-mds", 2023-10-16T12:34:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCALS1ltlmsIxAA8HjiIv99nZri/ezwQrU+Dw==", 2023-10-16T12:34:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-mds" 2023-10-16T12:34:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-mgr", 2023-10-16T12:34:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCALS1l332sIxAA3/crMT/S7erAx03EvMk/PQ==", 2023-10-16T12:34:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-16T12:34:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-osd", 2023-10-16T12:34:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCALS1lgpesIxAARfbkZOEu+vXEorh+cyejrw==", 2023-10-16T12:34:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-osd" 2023-10-16T12:34:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rbd", 2023-10-16T12:34:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCALS1l5a+sIxAAc/HDj4GELZ+ZYhDpqBKO5g==", 2023-10-16T12:34:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-16T12:34:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-16T12:34:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCALS1lDcmsIxAAez3j0UAgXWMWNLSoydKiag==", 2023-10-16T12:34:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-16T12:34:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rgw", 2023-10-16T12:34:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCALS1lp+KsIxAAVyz75s081rBB5VqRZsK2Wg==", 2023-10-16T12:34:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-16T12:34:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:34:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:34:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:34:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:34:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:34:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:34:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:34:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:34:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:34:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:34:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:34:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:34:35.983 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:34:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f125806ae70 msgr2=0x7f125806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f125806ae70 0x7f125806d330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1260036000 tx=0x7f126003b040 comp rx=0 tx=0).stop 2023-10-16T12:34:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 msgr2=0x7f12780af360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 0x7f12780af360 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1268009d30 tx=0x7f126800c490 comp rx=0 tx=0).stop 2023-10-16T12:34:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 shutdown_connections 2023-10-16T12:34:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f125806ae70 0x7f125806d330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12780ad970 0x7f1278151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127815ebc0 0x7f12780b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 --2- 172.21.15.60:0/2630862666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127806c040 0x7f12780af360 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:35.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 >> 172.21.15.60:0/2630862666 conn(0x7f12780ce6a0 msgr2=0x7f12780ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:35.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.938+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 shutdown_connections 2023-10-16T12:34:35.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:35.939+0000 7f127759e640 1 -- 172.21.15.60:0/2630862666 wait complete. 2023-10-16T12:34:35.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-16T12:34:36.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.215+0000 7fd278f5e640 1 Processor -- start 2023-10-16T12:34:36.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.215+0000 7fd278f5e640 1 -- start start 2023-10-16T12:34:36.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.216+0000 7fd278f5e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd2740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.216+0000 7fd278f5e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27415d150 con 0x7fd2740d0be0 2023-10-16T12:34:36.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.216+0000 7fd278f5e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2740d1180 con 0x7fd27415cd70 2023-10-16T12:34:36.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.216+0000 7fd278f5e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740d1460 con 0x7fd2740b8480 2023-10-16T12:34:36.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.217+0000 7fd273fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd2740d0be0 0x7fd2740b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49794/0 (socket says 172.21.15.60:49794) 2023-10-16T12:34:36.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.217+0000 7fd273fff640 1 -- 172.21.15.60:0/4153851263 learned_addr learned my addr 172.21.15.60:0/4153851263 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:36.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.217+0000 7fd272ffd640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd2740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.217+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055684953 0 0) 0x7fd27415d150 con 0x7fd2740d0be0 2023-10-16T12:34:36.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.217+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c003520 con 0x7fd2740d0be0 2023-10-16T12:34:36.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012070413 0 0) 0x7fd25c003520 con 0x7fd2740d0be0 2023-10-16T12:34:36.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 msgr2=0x7fd2740cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:36.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd2740cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 >> v1:172.21.15.144:6789/0 conn(0x7fd27415cd70 legacy=0x7fd2740d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740d1740 con 0x7fd2740d0be0 2023-10-16T12:34:36.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd26c0019f0 con 0x7fd2740d0be0 2023-10-16T12:34:36.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd272ffd640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd2740cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:36.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 >> v1:172.21.15.60:6789/0 conn(0x7fd2740d0be0 legacy=0x7fd2740b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd25c0039a0 0x7fd25c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0047c0 0x7fd25c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd25c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740d1460 con 0x7fd25c0039a0 2023-10-16T12:34:36.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c00ba80 con 0x7fd25c0047c0 2023-10-16T12:34:36.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c00bc40 con 0x7fd2740b8480 2023-10-16T12:34:36.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd2737fe640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd25c0039a0 0x7fd25c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd273fff640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd25c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.218+0000 7fd272ffd640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0047c0 0x7fd25c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.219+0000 7fd2737fe640 1 -- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 msgr2=0x7fd25c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:36.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.219+0000 7fd2737fe640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2740b8480 0x7fd25c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.219+0000 7fd2737fe640 1 -- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0047c0 msgr2=0x7fd25c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:36.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.219+0000 7fd2737fe640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0047c0 0x7fd25c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.219+0000 7fd2737fe640 1 -- 172.21.15.60:0/4153851263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c003520 con 0x7fd25c0039a0 2023-10-16T12:34:36.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.219+0000 7fd272ffd640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0047c0 0x7fd25c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:36.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.220+0000 7fd2737fe640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd25c0039a0 0x7fd25c003dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd264001750 tx=0x7fd25c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20569ff5ad17837 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:36.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.220+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640052d0 con 0x7fd25c0039a0 2023-10-16T12:34:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.220+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26402e050 con 0x7fd25c0039a0 2023-10-16T12:34:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.220+0000 7fd2727fc640 1 -- 172.21.15.60:0/4153851263 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2640035f0 con 0x7fd25c0039a0 2023-10-16T12:34:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.220+0000 7fd278f5e640 1 -- 172.21.15.60:0/4153851263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd25c0039a0 msgr2=0x7fd25c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.220+0000 7fd278f5e640 1 --2- 172.21.15.60:0/4153851263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd25c0039a0 0x7fd25c003dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd264001750 tx=0x7fd25c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.221+0000 7fd273fff640 1 -- 172.21.15.60:0/4153851263 reap_dead start 2023-10-16T12:34:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.221+0000 7fd278f5e640 1 -- 172.21.15.60:0/4153851263 shutdown_connections 2023-10-16T12:34:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.221+0000 7fd278f5e640 1 -- 172.21.15.60:0/4153851263 >> 172.21.15.60:0/4153851263 conn(0x7fd2740cf570 msgr2=0x7fd2740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.221+0000 7fd278f5e640 1 -- 172.21.15.60:0/4153851263 shutdown_connections 2023-10-16T12:34:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.221+0000 7fd278f5e640 1 -- 172.21.15.60:0/4153851263 wait complete. 2023-10-16T12:34:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.223+0000 7fd278f5e640 1 Processor -- start 2023-10-16T12:34:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.223+0000 7fd278f5e640 1 -- start start 2023-10-16T12:34:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd278f5e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0039a0 0x7fd2741541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd278f5e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 0x7fd274155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd278f5e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd274151de0 0x7fd2740cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd278f5e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c9c70 con 0x7fd27415e390 2023-10-16T12:34:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd278f5e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd274155980 con 0x7fd25c0039a0 2023-10-16T12:34:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd278f5e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740c9ec0 con 0x7fd274151de0 2023-10-16T12:34:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd272ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 0x7fd274155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd272ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 0x7fd274155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44480/0 (socket says 172.21.15.60:44480) 2023-10-16T12:34:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd272ffd640 1 -- 172.21.15.60:0/1947637567 learned_addr learned my addr 172.21.15.60:0/1947637567 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:36.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd2737fe640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0039a0 0x7fd2741541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd272ffd640 1 -- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd274151de0 msgr2=0x7fd2740cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.224+0000 7fd273fff640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd274151de0 0x7fd2740cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd272ffd640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd274151de0 0x7fd2740cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd272ffd640 1 -- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0039a0 msgr2=0x7fd2741541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd272ffd640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0039a0 0x7fd2741541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd272ffd640 1 -- 172.21.15.60:0/1947637567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740ca1a0 con 0x7fd27415e390 2023-10-16T12:34:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd2737fe640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0039a0 0x7fd2741541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd272ffd640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 0x7fd274155160 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd26800f550 tx=0x7fd26800fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268014890 con 0x7fd27415e390 2023-10-16T12:34:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26800c030 con 0x7fd27415e390 2023-10-16T12:34:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2740ca480 con 0x7fd27415e390 2023-10-16T12:34:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268014bf0 con 0x7fd27415e390 2023-10-16T12:34:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.225+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27415e7f0 con 0x7fd27415e390 2023-10-16T12:34:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.227+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27404e9b0 con 0x7fd27415e390 2023-10-16T12:34:36.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.228+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd268014d50 con 0x7fd27415e390 2023-10-16T12:34:36.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.229+0000 7fd25bfff640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd26006ae70 0x7fd26006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.229+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd268089580 con 0x7fd27415e390 2023-10-16T12:34:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.230+0000 7fd2737fe640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd26006ae70 0x7fd26006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.231+0000 7fd2737fe640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd26006ae70 0x7fd26006d330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd264033000 tx=0x7fd264009030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.233+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd268054980 con 0x7fd27415e390 2023-10-16T12:34:36.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.500+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fd27415c240 con 0x7fd27415e390 2023-10-16T12:34:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.506+0000 7fd25bfff640 1 -- 172.21.15.60:0/1947637567 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd268057fa0 con 0x7fd27415e390 2023-10-16T12:34:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.508+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd26006ae70 msgr2=0x7fd26006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.508+0000 7fd278f5e640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd26006ae70 0x7fd26006d330 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd264033000 tx=0x7fd264009030 comp rx=0 tx=0).stop 2023-10-16T12:34:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 msgr2=0x7fd274155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 0x7fd274155160 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd26800f550 tx=0x7fd26800fa10 comp rx=0 tx=0).stop 2023-10-16T12:34:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 shutdown_connections 2023-10-16T12:34:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd26006ae70 0x7fd26006d330 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd274151de0 0x7fd2740cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27415e390 0x7fd274155160 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 --2- 172.21.15.60:0/1947637567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd25c0039a0 0x7fd2741541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 >> 172.21.15.60:0/1947637567 conn(0x7fd2740cf570 msgr2=0x7fd2740b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 shutdown_connections 2023-10-16T12:34:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.509+0000 7fd278f5e640 1 -- 172.21.15.60:0/1947637567 wait complete. 2023-10-16T12:34:36.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-16T12:34:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.742+0000 7f5495c78640 1 Processor -- start 2023-10-16T12:34:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.742+0000 7f5495c78640 1 -- start start 2023-10-16T12:34:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.743+0000 7f5495c78640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5490159950 0x7f549015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.743+0000 7f5495c78640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5490070450 0x7f54900708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.743+0000 7f5495c78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f549006c100 0x7f549006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f5495c78640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5490070e20 con 0x7f5490070450 2023-10-16T12:34:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f5495c78640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549015e490 con 0x7f5490159950 2023-10-16T12:34:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f5495c78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549015aef0 con 0x7f549006c100 2023-10-16T12:34:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f548ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5490070450 0x7f54900708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f548ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5490070450 0x7f54900708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44488/0 (socket says 172.21.15.60:44488) 2023-10-16T12:34:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f548ffff640 1 -- 172.21.15.60:0/2372211625 learned_addr learned my addr 172.21.15.60:0/2372211625 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f5495477640 1 --2- 172.21.15.60:0/2372211625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f549006c100 0x7f549006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f5494c76640 1 --2- 172.21.15.60:0/2372211625 >> v2:172.21.15.144:3300/0 conn(0x7f5490159950 0x7f549015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f548ffff640 1 -- 172.21.15.60:0/2372211625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f549006c100 msgr2=0x7f549006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.744+0000 7f548ffff640 1 --2- 172.21.15.60:0/2372211625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f549006c100 0x7f549006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.745+0000 7f548ffff640 1 -- 172.21.15.60:0/2372211625 >> v2:172.21.15.144:3300/0 conn(0x7f5490159950 msgr2=0x7f549015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.745+0000 7f548ffff640 1 --2- 172.21.15.60:0/2372211625 >> v2:172.21.15.144:3300/0 conn(0x7f5490159950 0x7f549015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.745+0000 7f548ffff640 1 -- 172.21.15.60:0/2372211625 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549015d910 con 0x7f5490070450 2023-10-16T12:34:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.746+0000 7f548ffff640 1 --2- 172.21.15.60:0/2372211625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490070450 0x7f54900708e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54900ca950 tx=0x7f547c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335aa2ed39935921 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.746+0000 7f548f7fe640 1 -- 172.21.15.60:0/2372211625 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c0317e0 con 0x7f5490070450 2023-10-16T12:34:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.746+0000 7f548f7fe640 1 -- 172.21.15.60:0/2372211625 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c031940 con 0x7f5490070450 2023-10-16T12:34:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.746+0000 7f548f7fe640 1 -- 172.21.15.60:0/2372211625 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c031cd0 con 0x7f5490070450 2023-10-16T12:34:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 -- 172.21.15.60:0/2372211625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490070450 msgr2=0x7f54900708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 --2- 172.21.15.60:0/2372211625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490070450 0x7f54900708e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54900ca950 tx=0x7f547c02b880 comp rx=0 tx=0).stop 2023-10-16T12:34:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 -- 172.21.15.60:0/2372211625 shutdown_connections 2023-10-16T12:34:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 --2- 172.21.15.60:0/2372211625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f549006c100 0x7f549006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 --2- 172.21.15.60:0/2372211625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490070450 0x7f54900708e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 --2- 172.21.15.60:0/2372211625 >> v2:172.21.15.144:3300/0 conn(0x7f5490159950 0x7f549015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 -- 172.21.15.60:0/2372211625 >> 172.21.15.60:0/2372211625 conn(0x7f54900ce6a0 msgr2=0x7f54900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 -- 172.21.15.60:0/2372211625 shutdown_connections 2023-10-16T12:34:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.747+0000 7f5495c78640 1 -- 172.21.15.60:0/2372211625 wait complete. 2023-10-16T12:34:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.749+0000 7f5495c78640 1 Processor -- start 2023-10-16T12:34:36.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.749+0000 7f5495c78640 1 -- start start 2023-10-16T12:34:36.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5495c78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 0x7f549014e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5495c78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490070450 0x7f54900b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5495c78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490159950 0x7f54900b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5495c78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54900c5b80 con 0x7f549006c100 2023-10-16T12:34:36.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5495c78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54900b1160 con 0x7f5490070450 2023-10-16T12:34:36.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5495c78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54900b0ba0 con 0x7f5490159950 2023-10-16T12:34:36.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 0x7f549014e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 0x7f549014e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44492/0 (socket says 172.21.15.60:44492) 2023-10-16T12:34:36.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 -- 172.21.15.60:0/2053235687 learned_addr learned my addr 172.21.15.60:0/2053235687 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:36.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f548ffff640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490070450 0x7f54900b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 -- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490159950 msgr2=0x7f54900b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490159950 0x7f54900b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 -- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490070450 msgr2=0x7f54900b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:36.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.751+0000 7f5494c76640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490070450 0x7f54900b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:36.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f5494c76640 1 -- 172.21.15.60:0/2053235687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0094d0 con 0x7f549006c100 2023-10-16T12:34:36.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f548ffff640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490070450 0x7f54900b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:36.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f5494c76640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 0x7f549014e9f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f548400d800 tx=0x7f548400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:36.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548400b6c0 con 0x7f549006c100 2023-10-16T12:34:36.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548400b820 con 0x7f549006c100 2023-10-16T12:34:36.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54900b0e80 con 0x7f549006c100 2023-10-16T12:34:36.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548400bbb0 con 0x7f549006c100 2023-10-16T12:34:36.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.752+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549014ef30 con 0x7f549006c100 2023-10-16T12:34:36.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.753+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549004e9b0 con 0x7f549006c100 2023-10-16T12:34:36.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.755+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f548400bd10 con 0x7f549006c100 2023-10-16T12:34:36.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.756+0000 7f548d7fa640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f547406aea0 0x7f547406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:36.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.756+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5484089830 con 0x7f549006c100 2023-10-16T12:34:36.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.757+0000 7f548ffff640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f547406aea0 0x7f547406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:36.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.758+0000 7f548ffff640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f547406aea0 0x7f547406d360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f547c036000 tx=0x7f547c03d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:36.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:36.760+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5484054b90 con 0x7f549006c100 2023-10-16T12:34:37.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.025+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f54900b4620 con 0x7f549006c100 2023-10-16T12:34:37.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.030+0000 7f548d7fa640 1 -- 172.21.15.60:0/2053235687 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f54840581b0 con 0x7f549006c100 2023-10-16T12:34:37.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f547406aea0 msgr2=0x7f547406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f547406aea0 0x7f547406d360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f547c036000 tx=0x7f547c03d040 comp rx=0 tx=0).stop 2023-10-16T12:34:37.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 msgr2=0x7f549014e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 0x7f549014e9f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f548400d800 tx=0x7f548400dcc0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 shutdown_connections 2023-10-16T12:34:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f547406aea0 0x7f547406d360 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490159950 0x7f54900b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490070450 0x7f54900b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 --2- 172.21.15.60:0/2053235687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549006c100 0x7f549014e9f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 >> 172.21.15.60:0/2053235687 conn(0x7f54900ce6a0 msgr2=0x7f54900cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 shutdown_connections 2023-10-16T12:34:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.033+0000 7f5495c78640 1 -- 172.21.15.60:0/2053235687 wait complete. 2023-10-16T12:34:37.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-16T12:34:37.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.271+0000 7f4f23fff640 1 Processor -- start 2023-10-16T12:34:37.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.271+0000 7f4f23fff640 1 -- start start 2023-10-16T12:34:37.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.273+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 0x7f4f2406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.273+0000 7f4f23fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2415f4a0 con 0x7f4f240ad970 2023-10-16T12:34:37.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.273+0000 7f4f23fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2406c960 con 0x7f4f240cda80 2023-10-16T12:34:37.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.273+0000 7f4f23fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2415b9d0 con 0x7f4f2406c040 2023-10-16T12:34:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.273+0000 7f4f1bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4f240ad970 0x7f4f2415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49804/0 (socket says 172.21.15.60:49804) 2023-10-16T12:34:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.273+0000 7f4f1bfff640 1 -- 172.21.15.60:0/310573022 learned_addr learned my addr 172.21.15.60:0/310573022 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f22ffd640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 0x7f4f2406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783795902 0 0) 0x7f4f2415f4a0 con 0x7f4f240ad970 2023-10-16T12:34:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f10003520 con 0x7f4f240ad970 2023-10-16T12:34:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145541184 0 0) 0x7f4f10003520 con 0x7f4f240ad970 2023-10-16T12:34:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 msgr2=0x7f4f2406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 0x7f4f2406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 >> v1:172.21.15.144:6789/0 conn(0x7f4f240cda80 legacy=0x7f4f2415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2415bcb0 con 0x7f4f240ad970 2023-10-16T12:34:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.274+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4f0c001670 con 0x7f4f240ad970 2023-10-16T12:34:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 >> v1:172.21.15.60:6789/0 conn(0x7f4f240ad970 legacy=0x7f4f2415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 0x7f4f100039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f10003f70 0x7f4f100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2415b9d0 con 0x7f4f10004d00 2023-10-16T12:34:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1000bac0 con 0x7f4f10003f70 2023-10-16T12:34:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1000bc80 con 0x7f4f2406c040 2023-10-16T12:34:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f237fe640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f22ffd640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 0x7f4f100039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f237fe640 1 -- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 msgr2=0x7f4f100039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f237fe640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f2406c040 0x7f4f100039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f237fe640 1 -- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f10003f70 msgr2=0x7f4f100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f237fe640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f10003f70 0x7f4f100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.275+0000 7f4f237fe640 1 -- 172.21.15.60:0/310573022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f10003520 con 0x7f4f10004d00 2023-10-16T12:34:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f237fe640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f10005180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c001e50 tx=0x7f4f100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67167a2e4d37883 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c003b70 con 0x7f4f10004d00 2023-10-16T12:34:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c003cd0 con 0x7f4f10004d00 2023-10-16T12:34:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f227fc640 1 -- 172.21.15.60:0/310573022 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c005310 con 0x7f4f10004d00 2023-10-16T12:34:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f23fff640 1 -- 172.21.15.60:0/310573022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 msgr2=0x7f4f10005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f23fff640 1 --2- 172.21.15.60:0/310573022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f10005180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c001e50 tx=0x7f4f100057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.277+0000 7f4f237fe640 1 -- 172.21.15.60:0/310573022 reap_dead start 2023-10-16T12:34:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.278+0000 7f4f23fff640 1 -- 172.21.15.60:0/310573022 shutdown_connections 2023-10-16T12:34:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.278+0000 7f4f23fff640 1 -- 172.21.15.60:0/310573022 >> 172.21.15.60:0/310573022 conn(0x7f4f240ce6a0 msgr2=0x7f4f240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.278+0000 7f4f23fff640 1 -- 172.21.15.60:0/310573022 shutdown_connections 2023-10-16T12:34:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.278+0000 7f4f23fff640 1 -- 172.21.15.60:0/310573022 wait complete. 2023-10-16T12:34:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.279+0000 7f4f23fff640 1 Processor -- start 2023-10-16T12:34:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.280+0000 7f4f23fff640 1 -- start start 2023-10-16T12:34:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.280+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f240acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.280+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f240c7d90 0x7f4f240c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f24153780 0x7f4f24153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f23fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f240aec10 con 0x7f4f10004d00 2023-10-16T12:34:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f23fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f24154120 con 0x7f4f24153780 2023-10-16T12:34:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f23fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f240ca560 con 0x7f4f240c7d90 2023-10-16T12:34:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f240acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f240acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44510/0 (socket says 172.21.15.60:44510) 2023-10-16T12:34:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 -- 172.21.15.60:0/3193195516 learned_addr learned my addr 172.21.15.60:0/3193195516 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f237fe640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f24153780 0x7f4f24153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f1bfff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f240c7d90 0x7f4f240c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 -- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f240c7d90 msgr2=0x7f4f240c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f240c7d90 0x7f4f240c81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 -- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f24153780 msgr2=0x7f4f24153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f24153780 0x7f4f24153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f22ffd640 1 -- 172.21.15.60:0/3193195516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f240c9950 con 0x7f4f10004d00 2023-10-16T12:34:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.281+0000 7f4f237fe640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f24153780 0x7f4f24153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.282+0000 7f4f22ffd640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f240acb00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f14011600 tx=0x7f4f14011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.282+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1400f920 con 0x7f4f10004d00 2023-10-16T12:34:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.282+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1400fa80 con 0x7f4f10004d00 2023-10-16T12:34:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.282+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1400fe10 con 0x7f4f10004d00 2023-10-16T12:34:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.282+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f240c9c30 con 0x7f4f10004d00 2023-10-16T12:34:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.282+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f240ca1f0 con 0x7f4f10004d00 2023-10-16T12:34:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.283+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2406b330 con 0x7f4f10004d00 2023-10-16T12:34:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.284+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f14015460 con 0x7f4f10004d00 2023-10-16T12:34:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.285+0000 7f4f1b7fe640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f0406ae70 0x7f4f0406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.285+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4f14089ca0 con 0x7f4f10004d00 2023-10-16T12:34:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.286+0000 7f4f1bfff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f0406ae70 0x7f4f0406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.287+0000 7f4f1bfff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f0406ae70 0x7f4f0406d330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0017a0 tx=0x7f4f0c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.289+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f14054f70 con 0x7f4f10004d00 2023-10-16T12:34:37.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.558+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f4f2404e9b0 con 0x7f4f10004d00 2023-10-16T12:34:37.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.563+0000 7f4f1b7fe640 1 -- 172.21.15.60:0/3193195516 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4f14022020 con 0x7f4f10004d00 2023-10-16T12:34:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f0406ae70 msgr2=0x7f4f0406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f0406ae70 0x7f4f0406d330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0017a0 tx=0x7f4f0c009040 comp rx=0 tx=0).stop 2023-10-16T12:34:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 msgr2=0x7f4f240acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f240acb00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f14011600 tx=0x7f4f14011ac0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 shutdown_connections 2023-10-16T12:34:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4f0406ae70 0x7f4f0406d330 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4f240c7d90 0x7f4f240c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f24153780 0x7f4f24153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 --2- 172.21.15.60:0/3193195516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f10004d00 0x7f4f240acb00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 >> 172.21.15.60:0/3193195516 conn(0x7f4f240ce6a0 msgr2=0x7f4f2415c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 shutdown_connections 2023-10-16T12:34:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.566+0000 7f4f23fff640 1 -- 172.21.15.60:0/3193195516 wait complete. 2023-10-16T12:34:37.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-16T12:34:37.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.819+0000 7fe536128640 1 Processor -- start 2023-10-16T12:34:37.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.820+0000 7fe536128640 1 -- start start 2023-10-16T12:34:37.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe536128640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe530160c80 0x7fe530161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe536128640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5300ad7a0 0x7fe530070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe536128640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300ae1a0 con 0x7fe5300adc00 2023-10-16T12:34:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe536128640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5300ae480 con 0x7fe5300ad7a0 2023-10-16T12:34:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe536128640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300ae760 con 0x7fe530160c80 2023-10-16T12:34:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe534925640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5300ad7a0 0x7fe530070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe535927640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe5300adc00 0x7fe530142400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49816/0 (socket says 172.21.15.60:49816) 2023-10-16T12:34:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.821+0000 7fe535927640 1 -- 172.21.15.60:0/436120536 learned_addr learned my addr 172.21.15.60:0/436120536 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.822+0000 7fe535126640 1 --2- 172.21.15.60:0/436120536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe530160c80 0x7fe530161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.822+0000 7fe534925640 1 -- 172.21.15.60:0/436120536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe530160c80 msgr2=0x7fe530161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:37.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.822+0000 7fe534925640 1 --2- 172.21.15.60:0/436120536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe530160c80 0x7fe530161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.822+0000 7fe534925640 1 -- 172.21.15.60:0/436120536 >> v1:172.21.15.60:6789/0 conn(0x7fe5300adc00 legacy=0x7fe530142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.822+0000 7fe534925640 1 -- 172.21.15.60:0/436120536 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300aea40 con 0x7fe5300ad7a0 2023-10-16T12:34:37.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.823+0000 7fe534925640 1 --2- 172.21.15.60:0/436120536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5300ad7a0 0x7fe530070240 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097a0 tx=0x7fe52402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc550a09f4d26af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:37.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.823+0000 7fe52ffff640 1 -- 172.21.15.60:0/436120536 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52402d040 con 0x7fe5300ad7a0 2023-10-16T12:34:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.823+0000 7fe52ffff640 1 -- 172.21.15.60:0/436120536 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524031810 con 0x7fe5300ad7a0 2023-10-16T12:34:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.823+0000 7fe52ffff640 1 -- 172.21.15.60:0/436120536 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524031ba0 con 0x7fe5300ad7a0 2023-10-16T12:34:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 -- 172.21.15.60:0/436120536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5300ad7a0 msgr2=0x7fe530070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 --2- 172.21.15.60:0/436120536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5300ad7a0 0x7fe530070240 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097a0 tx=0x7fe52402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 -- 172.21.15.60:0/436120536 shutdown_connections 2023-10-16T12:34:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 --2- 172.21.15.60:0/436120536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5300ad7a0 0x7fe530070240 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 --2- 172.21.15.60:0/436120536 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe530160c80 0x7fe530161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 -- 172.21.15.60:0/436120536 >> 172.21.15.60:0/436120536 conn(0x7fe5300cc230 msgr2=0x7fe5300cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 -- 172.21.15.60:0/436120536 shutdown_connections 2023-10-16T12:34:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.824+0000 7fe536128640 1 -- 172.21.15.60:0/436120536 wait complete. 2023-10-16T12:34:37.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.826+0000 7fe536128640 1 Processor -- start 2023-10-16T12:34:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.826+0000 7fe536128640 1 -- start start 2023-10-16T12:34:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe536128640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5300ad7a0 0x7fe53015a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe536128640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5300adc00 0x7fe5301f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe536128640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 0x7fe5300c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe536128640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301f6b10 con 0x7fe530160c80 2023-10-16T12:34:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe536128640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301f6550 con 0x7fe5300adc00 2023-10-16T12:34:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe536128640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301f6830 con 0x7fe5300ad7a0 2023-10-16T12:34:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.827+0000 7fe535927640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 0x7fe5300c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 0x7fe5300c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44516/0 (socket says 172.21.15.60:44516) 2023-10-16T12:34:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 -- 172.21.15.60:0/782855009 learned_addr learned my addr 172.21.15.60:0/782855009 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535126640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5300ad7a0 0x7fe53015a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe534925640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5300adc00 0x7fe5301f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 -- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5300ad7a0 msgr2=0x7fe53015a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5300ad7a0 0x7fe53015a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 -- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5300adc00 msgr2=0x7fe5301f5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5300adc00 0x7fe5301f5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 -- 172.21.15.60:0/782855009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009400 con 0x7fe530160c80 2023-10-16T12:34:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.828+0000 7fe535927640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 0x7fe5300c9d70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe52800c420 tx=0x7fe52800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.829+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528005ce0 con 0x7fe530160c80 2023-10-16T12:34:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.829+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528005e40 con 0x7fe530160c80 2023-10-16T12:34:37.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.829+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280072d0 con 0x7fe530160c80 2023-10-16T12:34:37.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.829+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5300c6920 con 0x7fe530160c80 2023-10-16T12:34:37.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.829+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5301f6190 con 0x7fe530160c80 2023-10-16T12:34:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.830+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5300c6e40 con 0x7fe530160c80 2023-10-16T12:34:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.831+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe5280074f0 con 0x7fe530160c80 2023-10-16T12:34:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.832+0000 7fe52dffb640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe51c06ae70 0x7fe51c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.832+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe528088710 con 0x7fe530160c80 2023-10-16T12:34:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.833+0000 7fe535126640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe51c06ae70 0x7fe51c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.834+0000 7fe535126640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe51c06ae70 0x7fe51c06d330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5200099c0 tx=0x7fe520003990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:37.837+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5300c6e40 con 0x7fe530160c80 2023-10-16T12:34:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.101+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe5300c6e40 con 0x7fe530160c80 2023-10-16T12:34:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.105+0000 7fe52dffb640 1 -- 172.21.15.60:0/782855009 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe5300c6e40 con 0x7fe530160c80 2023-10-16T12:34:38.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe51c06ae70 msgr2=0x7fe51c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe51c06ae70 0x7fe51c06d330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe5200099c0 tx=0x7fe520003990 comp rx=0 tx=0).stop 2023-10-16T12:34:38.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 msgr2=0x7fe5300c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 0x7fe5300c9d70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe52800c420 tx=0x7fe52800c8e0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 shutdown_connections 2023-10-16T12:34:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe51c06ae70 0x7fe51c06d330 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe530160c80 0x7fe5300c9d70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5300adc00 0x7fe5301f5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 --2- 172.21.15.60:0/782855009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5300ad7a0 0x7fe53015a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 >> 172.21.15.60:0/782855009 conn(0x7fe5300cc230 msgr2=0x7fe5300cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 shutdown_connections 2023-10-16T12:34:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.108+0000 7fe536128640 1 -- 172.21.15.60:0/782855009 wait complete. 2023-10-16T12:34:38.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-16T12:34:38.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.348+0000 7fa6cf09f640 1 Processor -- start 2023-10-16T12:34:38.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.348+0000 7fa6cf09f640 1 -- start start 2023-10-16T12:34:38.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cf09f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa6c8153740 0x7fa6c815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cf09f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa6c80ad4d0 0x7fa6c80ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cf09f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c815cd50 0x7fa6c815d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cf09f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c815ac60 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cf09f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c8070030 con 0x7fa6c8153740 2023-10-16T12:34:38.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cf09f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c8070310 con 0x7fa6c815cd50 2023-10-16T12:34:38.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cd89c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa6c80ad4d0 0x7fa6c80ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cd89c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa6c80ad4d0 0x7fa6c80ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44524/0 (socket says 172.21.15.60:44524) 2023-10-16T12:34:38.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.350+0000 7fa6cd89c640 1 -- 172.21.15.60:0/744667680 learned_addr learned my addr 172.21.15.60:0/744667680 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6ce89e640 1 --2- 172.21.15.60:0/744667680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c815cd50 0x7fa6c815d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6cd89c640 1 -- 172.21.15.60:0/744667680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c815cd50 msgr2=0x7fa6c815d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6ce09d640 1 --2- 172.21.15.60:0/744667680 >> v2:172.21.15.144:3300/0 conn(0x7fa6c8153740 0x7fa6c815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6cd89c640 1 --2- 172.21.15.60:0/744667680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c815cd50 0x7fa6c815d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6cd89c640 1 -- 172.21.15.60:0/744667680 >> v2:172.21.15.144:3300/0 conn(0x7fa6c8153740 msgr2=0x7fa6c815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6cd89c640 1 --2- 172.21.15.60:0/744667680 >> v2:172.21.15.144:3300/0 conn(0x7fa6c8153740 0x7fa6c815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6cd89c640 1 -- 172.21.15.60:0/744667680 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80705f0 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.351+0000 7fa6ce09d640 1 --2- 172.21.15.60:0/744667680 >> v2:172.21.15.144:3300/0 conn(0x7fa6c8153740 0x7fa6c815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.352+0000 7fa6cd89c640 1 --2- 172.21.15.60:0/744667680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80ad930 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8073ca0 tx=0x7fa6b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93b91d7e6dd71d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.352+0000 7fa6cd09b640 1 -- 172.21.15.60:0/744667680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8031870 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.352+0000 7fa6cd09b640 1 -- 172.21.15.60:0/744667680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b80319d0 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.352+0000 7fa6cd09b640 1 -- 172.21.15.60:0/744667680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8031d60 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 -- 172.21.15.60:0/744667680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 msgr2=0x7fa6c80ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/744667680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80ad930 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8073ca0 tx=0x7fa6b802b910 comp rx=0 tx=0).stop 2023-10-16T12:34:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 -- 172.21.15.60:0/744667680 shutdown_connections 2023-10-16T12:34:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/744667680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c815cd50 0x7fa6c815d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/744667680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80ad930 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/744667680 >> v2:172.21.15.144:3300/0 conn(0x7fa6c8153740 0x7fa6c815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 -- 172.21.15.60:0/744667680 >> 172.21.15.60:0/744667680 conn(0x7fa6c80ce6a0 msgr2=0x7fa6c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 -- 172.21.15.60:0/744667680 shutdown_connections 2023-10-16T12:34:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.353+0000 7fa6cf09f640 1 -- 172.21.15.60:0/744667680 wait complete. 2023-10-16T12:34:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.355+0000 7fa6cf09f640 1 Processor -- start 2023-10-16T12:34:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.355+0000 7fa6cf09f640 1 -- start start 2023-10-16T12:34:38.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6cf09f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6cf09f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c8153740 0x7fa6c81504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6cf09f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c815cd50 0x7fa6c80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6cf09f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80c53e0 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6cf09f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80b3610 con 0x7fa6c815cd50 2023-10-16T12:34:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6cf09f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80c4e20 con 0x7fa6c8153740 2023-10-16T12:34:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6ce09d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6ce09d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44526/0 (socket says 172.21.15.60:44526) 2023-10-16T12:34:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.356+0000 7fa6ce09d640 1 -- 172.21.15.60:0/16776007 learned_addr learned my addr 172.21.15.60:0/16776007 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6cd89c640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c8153740 0x7fa6c81504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6ce09d640 1 -- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c8153740 msgr2=0x7fa6c81504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6ce09d640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c8153740 0x7fa6c81504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6ce09d640 1 -- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c815cd50 msgr2=0x7fa6c80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6ce09d640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c815cd50 0x7fa6c80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6ce09d640 1 -- 172.21.15.60:0/16776007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80094d0 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6ce09d640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80b40b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400c7c0 tx=0x7fa6c400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4007a30 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4007b90 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400ce70 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.357+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c80c5100 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.358+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c80b3b90 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.358+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c80b3d80 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.360+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa6c40113f0 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.360+0000 7fa6beffd640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa6ac06ae70 0x7fa6ac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.361+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa6c4089450 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.361+0000 7fa6cd89c640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa6ac06ae70 0x7fa6ac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.362+0000 7fa6cd89c640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa6ac06ae70 0x7fa6ac06d330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8036000 tx=0x7fa6b803c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:38.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.365+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6c4054840 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.633+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fa6c806fc30 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.638+0000 7fa6beffd640 1 -- 172.21.15.60:0/16776007 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa6c4057e60 con 0x7fa6c80ad4d0 2023-10-16T12:34:38.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa6ac06ae70 msgr2=0x7fa6ac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa6ac06ae70 0x7fa6ac06d330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8036000 tx=0x7fa6b803c040 comp rx=0 tx=0).stop 2023-10-16T12:34:38.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 msgr2=0x7fa6c80b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80b40b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400c7c0 tx=0x7fa6c400cc80 comp rx=0 tx=0).stop 2023-10-16T12:34:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 shutdown_connections 2023-10-16T12:34:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fa6ac06ae70 0x7fa6ac06d330 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c8153740 0x7fa6c81504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c815cd50 0x7fa6c80b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 --2- 172.21.15.60:0/16776007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c80ad4d0 0x7fa6c80b40b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 >> 172.21.15.60:0/16776007 conn(0x7fa6c80ce6a0 msgr2=0x7fa6c80adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 shutdown_connections 2023-10-16T12:34:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.641+0000 7fa6cf09f640 1 -- 172.21.15.60:0/16776007 wait complete. 2023-10-16T12:34:38.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-16T12:34:38.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.887+0000 7f972154e640 1 Processor -- start 2023-10-16T12:34:38.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.887+0000 7f972154e640 1 -- start start 2023-10-16T12:34:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.888+0000 7f972154e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 0x7f971c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f972154e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f971c06c810 0x7f971c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f972154e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971c070df0 con 0x7f971c070450 2023-10-16T12:34:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f972154e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971c15e490 con 0x7f971c06c810 2023-10-16T12:34:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f972154e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971c15aef0 con 0x7f971c159950 2023-10-16T12:34:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f971b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f971c070450 0x7f971c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51364/0 (socket says 172.21.15.60:51364) 2023-10-16T12:34:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f971b7fe640 1 -- 172.21.15.60:0/1933935394 learned_addr learned my addr 172.21.15.60:0/1933935394 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:38.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410521270 0 0) 0x7f971c070df0 con 0x7f971c070450 2023-10-16T12:34:38.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f971bfff640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 0x7f971c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f9720d4d640 1 --2- 172.21.15.60:0/1933935394 >> v2:172.21.15.144:3300/0 conn(0x7f971c06c810 0x7f971c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.889+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9704005730 con 0x7f971c070450 2023-10-16T12:34:38.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544805190 0 0) 0x7f9704005730 con 0x7f971c070450 2023-10-16T12:34:38.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 msgr2=0x7f971c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:38.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 0x7f971c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 >> v2:172.21.15.144:3300/0 conn(0x7f971c06c810 msgr2=0x7f971c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:38.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 --2- 172.21.15.60:0/1933935394 >> v2:172.21.15.144:3300/0 conn(0x7f971c06c810 0x7f971c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c06cc80 con 0x7f971c070450 2023-10-16T12:34:38.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9710001670 con 0x7f971c070450 2023-10-16T12:34:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 >> v1:172.21.15.60:6789/0 conn(0x7f971c070450 legacy=0x7f971c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 0x7f9704005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006150 0x7f97040069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9704006f10 0x7f9704007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c15e490 con 0x7f9704006f10 2023-10-16T12:34:38.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c15aef0 con 0x7f9704006150 2023-10-16T12:34:38.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.890+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970400dcf0 con 0x7f971c159950 2023-10-16T12:34:38.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f9720d4d640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9704006f10 0x7f9704007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f971b7fe640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006150 0x7f97040069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f971bfff640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 0x7f9704005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f9720d4d640 1 -- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 msgr2=0x7f9704005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:38.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f9720d4d640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c159950 0x7f9704005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f9720d4d640 1 -- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006150 msgr2=0x7f97040069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:38.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f9720d4d640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006150 0x7f97040069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.891+0000 7f9720d4d640 1 -- 172.21.15.60:0/1933935394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704005730 con 0x7f9704006f10 2023-10-16T12:34:38.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.892+0000 7f9720d4d640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9704006f10 0x7f9704007390 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f971400ba30 tx=0x7f9704007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca6867cfef7435c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:38.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.892+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714032680 con 0x7f9704006f10 2023-10-16T12:34:38.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.892+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97140327e0 con 0x7f9704006f10 2023-10-16T12:34:38.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.892+0000 7f971affd640 1 -- 172.21.15.60:0/1933935394 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714032b70 con 0x7f9704006f10 2023-10-16T12:34:38.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f972154e640 1 -- 172.21.15.60:0/1933935394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9704006f10 msgr2=0x7f9704007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f972154e640 1 --2- 172.21.15.60:0/1933935394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9704006f10 0x7f9704007390 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f971400ba30 tx=0x7f9704007a00 comp rx=0 tx=0).stop 2023-10-16T12:34:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f9720d4d640 1 -- 172.21.15.60:0/1933935394 reap_dead start 2023-10-16T12:34:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f972154e640 1 -- 172.21.15.60:0/1933935394 shutdown_connections 2023-10-16T12:34:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f972154e640 1 -- 172.21.15.60:0/1933935394 >> 172.21.15.60:0/1933935394 conn(0x7f971c0ce6a0 msgr2=0x7f971c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f972154e640 1 -- 172.21.15.60:0/1933935394 shutdown_connections 2023-10-16T12:34:38.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.893+0000 7f972154e640 1 -- 172.21.15.60:0/1933935394 wait complete. 2023-10-16T12:34:38.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.895+0000 7f972154e640 1 Processor -- start 2023-10-16T12:34:38.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.895+0000 7f972154e640 1 -- start start 2023-10-16T12:34:38.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f972154e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006f10 0x7f971c15c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f972154e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 0x7f971c0b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f972154e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c0c4da0 0x7f971c150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f972154e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c0b41c0 con 0x7f971c0c4920 2023-10-16T12:34:38.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f972154e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c15b090 con 0x7f9704006f10 2023-10-16T12:34:38.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f972154e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971c15b370 con 0x7f971c0c4da0 2023-10-16T12:34:38.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f971b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 0x7f971c0b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f971b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 0x7f971c0b1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41946/0 (socket says 172.21.15.60:41946) 2023-10-16T12:34:38.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.896+0000 7f971b7fe640 1 -- 172.21.15.60:0/2525634013 learned_addr learned my addr 172.21.15.60:0/2525634013 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:38.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971b7fe640 1 -- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c0c4da0 msgr2=0x7f971c150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f9720d4d640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c0c4da0 0x7f971c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971bfff640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006f10 0x7f971c15c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971b7fe640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c0c4da0 0x7f971c150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971b7fe640 1 -- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006f10 msgr2=0x7f971c15c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971b7fe640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006f10 0x7f971c15c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971b7fe640 1 -- 172.21.15.60:0/2525634013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c15b650 con 0x7f971c0c4920 2023-10-16T12:34:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f971b7fe640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 0x7f971c0b1530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f971000e7f0 tx=0x7f971000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.897+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971000eea0 con 0x7f971c0c4920 2023-10-16T12:34:38.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.898+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971000c870 con 0x7f971c0c4920 2023-10-16T12:34:38.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.898+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971000cc00 con 0x7f971c0c4920 2023-10-16T12:34:38.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.898+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c15b930 con 0x7f971c0c4920 2023-10-16T12:34:38.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.898+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c15bef0 con 0x7f971c0c4920 2023-10-16T12:34:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.899+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c04e9b0 con 0x7f971c0c4920 2023-10-16T12:34:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.900+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f971001d460 con 0x7f971c0c4920 2023-10-16T12:34:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.901+0000 7f9718ff9640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f970806ae70 0x7f970806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.901+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9710089380 con 0x7f971c0c4920 2023-10-16T12:34:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.901+0000 7f971bfff640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f970806ae70 0x7f970806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:38.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.903+0000 7f971bfff640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f970806ae70 0x7f970806d330 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f971c0b4450 tx=0x7f970c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:38.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:38.906+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97100546e0 con 0x7f971c0c4920 2023-10-16T12:34:39.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.180+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f971c0b4620 con 0x7f971c0c4920 2023-10-16T12:34:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.186+0000 7f9718ff9640 1 -- 172.21.15.60:0/2525634013 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9710057d00 con 0x7f971c0c4920 2023-10-16T12:34:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.188+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f970806ae70 msgr2=0x7f970806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.188+0000 7f972154e640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f970806ae70 0x7f970806d330 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f971c0b4450 tx=0x7f970c00b040 comp rx=0 tx=0).stop 2023-10-16T12:34:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.188+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 msgr2=0x7f971c0b1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.188+0000 7f972154e640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 0x7f971c0b1530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f971000e7f0 tx=0x7f971000ecb0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 shutdown_connections 2023-10-16T12:34:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f970806ae70 0x7f970806d330 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f971c0c4da0 0x7f971c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f971c0c4920 0x7f971c0b1530 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 --2- 172.21.15.60:0/2525634013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9704006f10 0x7f971c15c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 >> 172.21.15.60:0/2525634013 conn(0x7f971c0ce6a0 msgr2=0x7f971c070850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 shutdown_connections 2023-10-16T12:34:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.189+0000 7f972154e640 1 -- 172.21.15.60:0/2525634013 wait complete. 2023-10-16T12:34:39.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:39.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2023-10-16T12:34:39.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.419+0000 7f6f6373e640 1 Processor -- start 2023-10-16T12:34:39.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.419+0000 7f6f6373e640 1 -- start start 2023-10-16T12:34:39.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f6373e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15aaa0 0x7f6f5c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f6373e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f5c0ce390 0x7f6f5c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f6373e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5c15ed60 con 0x7f6f5c15ae80 2023-10-16T12:34:39.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f6373e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5c06b810 con 0x7f6f5c0ce390 2023-10-16T12:34:39.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f6373e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0707e0 con 0x7f6f5c15aaa0 2023-10-16T12:34:39.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f614da640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15aaa0 0x7f6f5c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f61cdb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f5c0ce390 0x7f6f5c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f614da640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15aaa0 0x7f6f5c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:58816/0 (socket says 172.21.15.60:58816) 2023-10-16T12:34:39.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.421+0000 7f6f614da640 1 -- 172.21.15.60:0/304260000 learned_addr learned my addr 172.21.15.60:0/304260000 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:39.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.422+0000 7f6f61cdb640 1 -- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15aaa0 msgr2=0x7f6f5c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:39.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.422+0000 7f6f61cdb640 1 --2- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15aaa0 0x7f6f5c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.422+0000 7f6f61cdb640 1 -- 172.21.15.60:0/304260000 >> v1:172.21.15.144:6789/0 conn(0x7f6f5c15ae80 legacy=0x7f6f5c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.422+0000 7f6f61cdb640 1 -- 172.21.15.60:0/304260000 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c15ef90 con 0x7f6f5c0ce390 2023-10-16T12:34:39.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.424+0000 7f6f61cdb640 1 --2- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c070f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009d90 tx=0x7f6f5402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=818f83a6e3dc9211 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:39.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.424+0000 7f6f53fff640 1 -- 172.21.15.60:0/304260000 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5402cc50 con 0x7f6f5c0ce390 2023-10-16T12:34:39.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.424+0000 7f6f53fff640 1 -- 172.21.15.60:0/304260000 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5402cdb0 con 0x7f6f5c0ce390 2023-10-16T12:34:39.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.424+0000 7f6f53fff640 1 -- 172.21.15.60:0/304260000 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54034660 con 0x7f6f5c0ce390 2023-10-16T12:34:39.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 -- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c0ce390 msgr2=0x7f6f5c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 --2- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c070f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009d90 tx=0x7f6f5402bea0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 -- 172.21.15.60:0/304260000 shutdown_connections 2023-10-16T12:34:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 --2- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c070f50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 --2- 172.21.15.60:0/304260000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15aaa0 0x7f6f5c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 -- 172.21.15.60:0/304260000 >> 172.21.15.60:0/304260000 conn(0x7f6f5c0cb410 msgr2=0x7f6f5c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 -- 172.21.15.60:0/304260000 shutdown_connections 2023-10-16T12:34:39.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.425+0000 7f6f6373e640 1 -- 172.21.15.60:0/304260000 wait complete. 2023-10-16T12:34:39.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.427+0000 7f6f6373e640 1 Processor -- start 2023-10-16T12:34:39.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.428+0000 7f6f6373e640 1 -- start start 2023-10-16T12:34:39.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f6373e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f6373e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c15aaa0 0x7f6f5c15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f6373e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15ae80 0x7f6f5c1f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f6373e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c157c00 con 0x7f6f5c0ce390 2023-10-16T12:34:39.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f6373e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c157ee0 con 0x7f6f5c15aaa0 2023-10-16T12:34:39.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f6373e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c1581c0 con 0x7f6f5c15ae80 2023-10-16T12:34:39.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f614da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f614da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c0ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41956/0 (socket says 172.21.15.60:41956) 2023-10-16T12:34:39.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f614da640 1 -- 172.21.15.60:0/3859949123 learned_addr learned my addr 172.21.15.60:0/3859949123 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:39.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f61cdb640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15ae80 0x7f6f5c1f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.429+0000 7f6f60cd9640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c15aaa0 0x7f6f5c15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f614da640 1 -- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15ae80 msgr2=0x7f6f5c1f56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f614da640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15ae80 0x7f6f5c1f56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f614da640 1 -- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c15aaa0 msgr2=0x7f6f5c15b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f614da640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c15aaa0 0x7f6f5c15b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f614da640 1 -- 172.21.15.60:0/3859949123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54009a70 con 0x7f6f5c0ce390 2023-10-16T12:34:39.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f60cd9640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c15aaa0 0x7f6f5c15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:39.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f614da640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c0ae950 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009fd0 tx=0x7f6f5800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:39.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5800c940 con 0x7f6f5c0ce390 2023-10-16T12:34:39.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5800caa0 con 0x7f6f5c0ce390 2023-10-16T12:34:39.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5800ce30 con 0x7f6f5c0ce390 2023-10-16T12:34:39.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.430+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c1584a0 con 0x7f6f5c0ce390 2023-10-16T12:34:39.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.431+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c15be90 con 0x7f6f5c0ce390 2023-10-16T12:34:39.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.432+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c04e160 con 0x7f6f5c0ce390 2023-10-16T12:34:39.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.433+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6f58013470 con 0x7f6f5c0ce390 2023-10-16T12:34:39.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.434+0000 7f6f51ffb640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6f4006ae70 0x7f6f4006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.434+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6f58089460 con 0x7f6f5c0ce390 2023-10-16T12:34:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.434+0000 7f6f60cd9640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6f4006ae70 0x7f6f4006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.435+0000 7f6f60cd9640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6f4006ae70 0x7f6f4006d330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0cdb50 tx=0x7f6f4c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.438+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f58054850 con 0x7f6f5c0ce390 2023-10-16T12:34:39.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.637+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f5c15e350 con 0x7f6f5c0ce390 2023-10-16T12:34:39.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.639+0000 7f6f51ffb640 1 -- 172.21.15.60:0/3859949123 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+12845 (secure 0 0 0) 0x7f6f58057e70 con 0x7f6f5c0ce390 2023-10-16T12:34:39.642 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:39.642 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:35.396150+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:39.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.642+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6f4006ae70 msgr2=0x7f6f4006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.642+0000 7f6f6373e640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6f4006ae70 0x7f6f4006d330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0cdb50 tx=0x7f6f4c009040 comp rx=0 tx=0).stop 2023-10-16T12:34:39.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.642+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 msgr2=0x7f6f5c0ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.642+0000 7f6f6373e640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c0ae950 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009fd0 tx=0x7f6f5800ed40 comp rx=0 tx=0).stop 2023-10-16T12:34:39.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 shutdown_connections 2023-10-16T12:34:39.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f6f4006ae70 0x7f6f4006d330 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f5c15ae80 0x7f6f5c1f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f5c15aaa0 0x7f6f5c15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 --2- 172.21.15.60:0/3859949123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5c0ce390 0x7f6f5c0ae950 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 >> 172.21.15.60:0/3859949123 conn(0x7f6f5c0cb410 msgr2=0x7f6f5c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:39.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 shutdown_connections 2023-10-16T12:34:39.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.643+0000 7f6f6373e640 1 -- 172.21.15.60:0/3859949123 wait complete. 2023-10-16T12:34:39.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:34:39.667 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:34:39.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.898+0000 7fb1c2d97640 1 Processor -- start 2023-10-16T12:34:39.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.898+0000 7fb1c2d97640 1 -- start start 2023-10-16T12:34:39.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1c2d97640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1bc1618b0 0x7fb1bc065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1c2d97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1c2d97640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1bc066420 con 0x7fb1bc15d330 2023-10-16T12:34:39.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1c2d97640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1bc06b1e0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1c2d97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc06cee0 con 0x7fb1bc065ff0 2023-10-16T12:34:39.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1bbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1bc1618b0 0x7fb1bc065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1bbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1bc1618b0 0x7fb1bc065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41968/0 (socket says 172.21.15.60:41968) 2023-10-16T12:34:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.899+0000 7fb1bbfff640 1 -- 172.21.15.60:0/2721443309 learned_addr learned my addr 172.21.15.60:0/2721443309 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.900+0000 7fb1bbfff640 1 -- 172.21.15.60:0/2721443309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 msgr2=0x7fb1bc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.900+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/2721443309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.900+0000 7fb1bbfff640 1 -- 172.21.15.60:0/2721443309 >> v1:172.21.15.144:6789/0 conn(0x7fb1bc15d330 legacy=0x7fb1bc06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.900+0000 7fb1bbfff640 1 -- 172.21.15.60:0/2721443309 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc0d0890 con 0x7fb1bc1618b0 2023-10-16T12:34:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.901+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/2721443309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc065ab0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0b6bc0 tx=0x7fb1b002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fb7f5b5ae165ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.901+0000 7fb1bb7fe640 1 -- 172.21.15.60:0/2721443309 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00317c0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.901+0000 7fb1bb7fe640 1 -- 172.21.15.60:0/2721443309 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b0031920 con 0x7fb1bc1618b0 2023-10-16T12:34:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.901+0000 7fb1bb7fe640 1 -- 172.21.15.60:0/2721443309 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0031cb0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2721443309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 msgr2=0x7fb1bc065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2721443309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc065ab0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0b6bc0 tx=0x7fb1b002b860 comp rx=0 tx=0).stop 2023-10-16T12:34:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2721443309 shutdown_connections 2023-10-16T12:34:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2721443309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2721443309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc065ab0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2721443309 >> 172.21.15.60:0/2721443309 conn(0x7fb1bc0d0ab0 msgr2=0x7fb1bc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2721443309 shutdown_connections 2023-10-16T12:34:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.902+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2721443309 wait complete. 2023-10-16T12:34:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.904+0000 7fb1c2d97640 1 Processor -- start 2023-10-16T12:34:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.904+0000 7fb1c2d97640 1 -- start start 2023-10-16T12:34:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c2d97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c2d97640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1bc15d330 0x7fb1bc0cec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c2d97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c2d97640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc1600e0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c2d97640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc1f2d80 con 0x7fb1bc15d330 2023-10-16T12:34:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c2d97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc15ec60 con 0x7fb1bc065ff0 2023-10-16T12:34:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c1334640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c1334640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41984/0 (socket says 172.21.15.60:41984) 2023-10-16T12:34:39.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.905+0000 7fb1c1334640 1 -- 172.21.15.60:0/2813291799 learned_addr learned my addr 172.21.15.60:0/2813291799 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:39.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c0b33640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1bbfff640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1bc15d330 0x7fb1bc0cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c1334640 1 -- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 msgr2=0x7fb1bc15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c1334640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c1334640 1 -- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1bc15d330 msgr2=0x7fb1bc0cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c1334640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1bc15d330 0x7fb1bc0cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c1334640 1 -- 172.21.15.60:0/2813291799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b00094d0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c1334640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc1f3e40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400b410 tx=0x7fb1b400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4011830 con 0x7fb1bc1618b0 2023-10-16T12:34:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b40119b0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4011d60 con 0x7fb1bc1618b0 2023-10-16T12:34:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.906+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1bc15ef40 con 0x7fb1bc1618b0 2023-10-16T12:34:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.907+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1bc15f500 con 0x7fb1bc1618b0 2023-10-16T12:34:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.908+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1bc1f2ef0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.909+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb1b401a400 con 0x7fb1bc1618b0 2023-10-16T12:34:39.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.910+0000 7fb1b97fa640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a806ae70 0x7fb1a806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:39.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.910+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb1b40893e0 con 0x7fb1bc1618b0 2023-10-16T12:34:39.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.911+0000 7fb1c0b33640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a806ae70 0x7fb1a806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:39.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.912+0000 7fb1c0b33640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a806ae70 0x7fb1a806d330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc157f70 tx=0x7fb1ac0094f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:39.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:39.914+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1b40547d0 con 0x7fb1bc1618b0 2023-10-16T12:34:40.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.111+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1bc04e160 con 0x7fb1bc1618b0 2023-10-16T12:34:40.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.112+0000 7fb1b97fa640 1 -- 172.21.15.60:0/2813291799 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1b4057df0 con 0x7fb1bc1618b0 2023-10-16T12:34:40.114 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:34:40.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a806ae70 msgr2=0x7fb1a806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a806ae70 0x7fb1a806d330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc157f70 tx=0x7fb1ac0094f0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 msgr2=0x7fb1bc1f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc1f3e40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400b410 tx=0x7fb1b400b8d0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 shutdown_connections 2023-10-16T12:34:40.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1a806ae70 0x7fb1a806d330 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1bc1618b0 0x7fb1bc1f3e40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1bc15d330 0x7fb1bc0cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 --2- 172.21.15.60:0/2813291799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb1bc065ff0 0x7fb1bc15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.114+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 >> 172.21.15.60:0/2813291799 conn(0x7fb1bc0d0ab0 msgr2=0x7fb1bc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:40.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.115+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 shutdown_connections 2023-10-16T12:34:40.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.115+0000 7fb1c2d97640 1 -- 172.21.15.60:0/2813291799 wait complete. 2023-10-16T12:34:40.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:34:40.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.351+0000 7f077359e640 1 Processor -- start 2023-10-16T12:34:40.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.351+0000 7f077359e640 1 -- start start 2023-10-16T12:34:40.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.352+0000 7f077359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f077415dde0 0x7f07740709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f077359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0774070f20 0x7f07740ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f077359e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0774160a90 con 0x7f077415c9f0 2023-10-16T12:34:40.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f077359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0774071350 con 0x7f0774070f20 2023-10-16T12:34:40.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f077359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740cdec0 con 0x7f077415dde0 2023-10-16T12:34:40.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f0772d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0774070f20 0x7f07740ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f0772d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0774070f20 0x7f07740ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41996/0 (socket says 172.21.15.60:41996) 2023-10-16T12:34:40.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f0772d9d640 1 -- 172.21.15.60:0/2751432978 learned_addr learned my addr 172.21.15.60:0/2751432978 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.353+0000 7f0771d9b640 1 --2- 172.21.15.60:0/2751432978 >> v2:172.21.15.144:3300/0 conn(0x7f077415dde0 0x7f07740709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.354+0000 7f0772d9d640 1 -- 172.21.15.60:0/2751432978 >> v2:172.21.15.144:3300/0 conn(0x7f077415dde0 msgr2=0x7f07740709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.354+0000 7f0772d9d640 1 --2- 172.21.15.60:0/2751432978 >> v2:172.21.15.144:3300/0 conn(0x7f077415dde0 0x7f07740709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.354+0000 7f0772d9d640 1 -- 172.21.15.60:0/2751432978 >> v1:172.21.15.144:6789/0 conn(0x7f077415c9f0 legacy=0x7f07740c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.354+0000 7f0772d9d640 1 -- 172.21.15.60:0/2751432978 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740cd900 con 0x7f0774070f20 2023-10-16T12:34:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.355+0000 7f0772d9d640 1 --2- 172.21.15.60:0/2751432978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0774070f20 0x7f07740ac890 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f07740b1260 tx=0x7f076c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbe22253a6f7ee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.355+0000 7f077159a640 1 -- 172.21.15.60:0/2751432978 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c031870 con 0x7f0774070f20 2023-10-16T12:34:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.355+0000 7f077159a640 1 -- 172.21.15.60:0/2751432978 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c0319d0 con 0x7f0774070f20 2023-10-16T12:34:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.355+0000 7f077159a640 1 -- 172.21.15.60:0/2751432978 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c031d60 con 0x7f0774070f20 2023-10-16T12:34:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 -- 172.21.15.60:0/2751432978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0774070f20 msgr2=0x7f07740ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 --2- 172.21.15.60:0/2751432978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0774070f20 0x7f07740ac890 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f07740b1260 tx=0x7f076c02b910 comp rx=0 tx=0).stop 2023-10-16T12:34:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 -- 172.21.15.60:0/2751432978 shutdown_connections 2023-10-16T12:34:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 --2- 172.21.15.60:0/2751432978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0774070f20 0x7f07740ac890 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 --2- 172.21.15.60:0/2751432978 >> v2:172.21.15.144:3300/0 conn(0x7f077415dde0 0x7f07740709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 -- 172.21.15.60:0/2751432978 >> 172.21.15.60:0/2751432978 conn(0x7f07740ce6a0 msgr2=0x7f07740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:40.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 -- 172.21.15.60:0/2751432978 shutdown_connections 2023-10-16T12:34:40.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.356+0000 7f077359e640 1 -- 172.21.15.60:0/2751432978 wait complete. 2023-10-16T12:34:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.358+0000 7f077359e640 1 Processor -- start 2023-10-16T12:34:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.358+0000 7f077359e640 1 -- start start 2023-10-16T12:34:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f077359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0774070f20 0x7f07741f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f077359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 0x7f0774142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f077359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f077415dde0 0x7f077406c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f077359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077406b480 con 0x7f077415c9f0 2023-10-16T12:34:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f077359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741f23b0 con 0x7f077415dde0 2023-10-16T12:34:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f077359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077406b690 con 0x7f0774070f20 2023-10-16T12:34:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f0771d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 0x7f0774142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f0771d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 0x7f0774142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42010/0 (socket says 172.21.15.60:42010) 2023-10-16T12:34:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.359+0000 7f0771d9b640 1 -- 172.21.15.60:0/2756674397 learned_addr learned my addr 172.21.15.60:0/2756674397 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0772d9d640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f077415dde0 0x7f077406c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f077259c640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0774070f20 0x7f07741f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0771d9b640 1 -- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0774070f20 msgr2=0x7f07741f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0771d9b640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0774070f20 0x7f07741f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0771d9b640 1 -- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f077415dde0 msgr2=0x7f077406c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0771d9b640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f077415dde0 0x7f077406c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0771d9b640 1 -- 172.21.15.60:0/2756674397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0094d0 con 0x7f077415c9f0 2023-10-16T12:34:40.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f0771d9b640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 0x7f0774142bf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0764007e40 tx=0x7f076400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:40.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764014620 con 0x7f077415c9f0 2023-10-16T12:34:40.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076400ee90 con 0x7f077415c9f0 2023-10-16T12:34:40.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640149c0 con 0x7f077415c9f0 2023-10-16T12:34:40.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.360+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077406b970 con 0x7f077415c9f0 2023-10-16T12:34:40.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.361+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077406bf30 con 0x7f077415c9f0 2023-10-16T12:34:40.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.362+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077406c0f0 con 0x7f077415c9f0 2023-10-16T12:34:40.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.363+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0764014b20 con 0x7f077415c9f0 2023-10-16T12:34:40.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.364+0000 7f076affd640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f075406ae70 0x7f075406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.364+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0764089c00 con 0x7f077415c9f0 2023-10-16T12:34:40.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.364+0000 7f077259c640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f075406ae70 0x7f075406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.365+0000 7f077259c640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f075406ae70 0x7f075406d330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f075c004ea0 tx=0x7f075c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:40.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.368+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0764054ff0 con 0x7f077415c9f0 2023-10-16T12:34:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.568+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f077404e9b0 con 0x7f077415c9f0 2023-10-16T12:34:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.569+0000 7f076affd640 1 -- 172.21.15.60:0/2756674397 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v32) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0764058610 con 0x7f077415c9f0 2023-10-16T12:34:40.572 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:40.572 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:34:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:34:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:34:40.573 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:40.573 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:34:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f075406ae70 msgr2=0x7f075406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f075406ae70 0x7f075406d330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f075c004ea0 tx=0x7f075c009040 comp rx=0 tx=0).stop 2023-10-16T12:34:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 msgr2=0x7f0774142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 0x7f0774142bf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0764007e40 tx=0x7f076400e4b0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 shutdown_connections 2023-10-16T12:34:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f075406ae70 0x7f075406d330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f077415dde0 0x7f077406c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f077415c9f0 0x7f0774142bf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 --2- 172.21.15.60:0/2756674397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0774070f20 0x7f07741f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 >> 172.21.15.60:0/2756674397 conn(0x7f07740ce6a0 msgr2=0x7f077415ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 shutdown_connections 2023-10-16T12:34:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.572+0000 7f077359e640 1 -- 172.21.15.60:0/2756674397 wait complete. 2023-10-16T12:34:40.592 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:34:40.593 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:34:40.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.824+0000 7fad3ce4d640 1 Processor -- start 2023-10-16T12:34:40.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.824+0000 7fad3ce4d640 1 -- start start 2023-10-16T12:34:40.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad3ce4d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad38159bd0 0x7fad38159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad3ce4d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad3806b3e0 0x7fad3806cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad3ce4d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3815a810 con 0x7fad3806d2c0 2023-10-16T12:34:40.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad3ce4d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad380c5740 con 0x7fad3806b3e0 2023-10-16T12:34:40.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad3ce4d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3806d750 con 0x7fad38159bd0 2023-10-16T12:34:40.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad377fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad38159bd0 0x7fad38159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad36ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad3806b3e0 0x7fad3806cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad377fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad38159bd0 0x7fad38159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:58880/0 (socket says 172.21.15.60:58880) 2023-10-16T12:34:40.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.826+0000 7fad377fe640 1 -- 172.21.15.60:0/1823691249 learned_addr learned my addr 172.21.15.60:0/1823691249 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:40.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.827+0000 7fad36ffd640 1 -- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad38159bd0 msgr2=0x7fad38159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:40.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.827+0000 7fad36ffd640 1 --2- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad38159bd0 0x7fad38159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.827+0000 7fad36ffd640 1 -- 172.21.15.60:0/1823691249 >> v1:172.21.15.144:6789/0 conn(0x7fad3806d2c0 legacy=0x7fad3815d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.827+0000 7fad36ffd640 1 -- 172.21.15.60:0/1823691249 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3815aaf0 con 0x7fad3806b3e0 2023-10-16T12:34:40.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.829+0000 7fad36ffd640 1 --2- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad3806cd80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0097a0 tx=0x7fad2c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a271535572694962 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:40.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad367fc640 1 -- 172.21.15.60:0/1823691249 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c02e070 con 0x7fad3806b3e0 2023-10-16T12:34:40.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad367fc640 1 -- 172.21.15.60:0/1823691249 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c0315a0 con 0x7fad3806b3e0 2023-10-16T12:34:40.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad367fc640 1 -- 172.21.15.60:0/1823691249 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c031930 con 0x7fad3806b3e0 2023-10-16T12:34:40.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad3ce4d640 1 -- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 msgr2=0x7fad3806cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad3806cd80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0097a0 tx=0x7fad2c02b840 comp rx=0 tx=0).stop 2023-10-16T12:34:40.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad3ce4d640 1 -- 172.21.15.60:0/1823691249 shutdown_connections 2023-10-16T12:34:40.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad3806cd80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/1823691249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad38159bd0 0x7fad38159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.830+0000 7fad3ce4d640 1 -- 172.21.15.60:0/1823691249 >> 172.21.15.60:0/1823691249 conn(0x7fad380ce820 msgr2=0x7fad380cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:40.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.831+0000 7fad3ce4d640 1 -- 172.21.15.60:0/1823691249 shutdown_connections 2023-10-16T12:34:40.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.831+0000 7fad3ce4d640 1 -- 172.21.15.60:0/1823691249 wait complete. 2023-10-16T12:34:40.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.832+0000 7fad3ce4d640 1 Processor -- start 2023-10-16T12:34:40.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 -- start start 2023-10-16T12:34:40.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad380b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad3806d2c0 0x7fad381f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 0x7fad380c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad38153840 con 0x7fad38159bd0 2023-10-16T12:34:40.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380b30d0 con 0x7fad3806b3e0 2023-10-16T12:34:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.833+0000 7fad3ce4d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380b33b0 con 0x7fad3806d2c0 2023-10-16T12:34:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 0x7fad380c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 0x7fad380c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42016/0 (socket says 172.21.15.60:42016) 2023-10-16T12:34:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 -- 172.21.15.60:0/3260076895 learned_addr learned my addr 172.21.15.60:0/3260076895 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:40.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad377fe640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad380b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad36ffd640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad3806d2c0 0x7fad381f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 -- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad3806d2c0 msgr2=0x7fad381f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad3806d2c0 0x7fad381f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 -- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 msgr2=0x7fad380b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:40.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad380b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:40.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad37fff640 1 -- 172.21.15.60:0/3260076895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c009400 con 0x7fad38159bd0 2023-10-16T12:34:40.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.834+0000 7fad36ffd640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad3806d2c0 0x7fad381f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:40.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.835+0000 7fad37fff640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 0x7fad380c4490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fad3000c6b0 tx=0x7fad3000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:40.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.835+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30003b70 con 0x7fad38159bd0 2023-10-16T12:34:40.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.835+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30003cf0 con 0x7fad38159bd0 2023-10-16T12:34:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.835+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30003ea0 con 0x7fad38159bd0 2023-10-16T12:34:40.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.835+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad380b3690 con 0x7fad38159bd0 2023-10-16T12:34:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.835+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad380b3c50 con 0x7fad38159bd0 2023-10-16T12:34:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.837+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad380c9d00 con 0x7fad38159bd0 2023-10-16T12:34:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.837+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fad30005310 con 0x7fad38159bd0 2023-10-16T12:34:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.838+0000 7fad1ffff640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2406ae70 0x7fad2406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.838+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad30089170 con 0x7fad38159bd0 2023-10-16T12:34:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.839+0000 7fad377fe640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2406ae70 0x7fad2406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:40.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.840+0000 7fad377fe640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2406ae70 0x7fad2406d330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fad38158ea0 tx=0x7fad28008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:40.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:40.843+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad300544d0 con 0x7fad38159bd0 2023-10-16T12:34:41.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.041+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fad3804e9b0 con 0x7fad38159bd0 2023-10-16T12:34:41.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.862+0000 7fad1ffff640 1 -- 172.21.15.60:0/3260076895 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v33) v1 ==== 138+0+0 (secure 0 0 0) 0x7fad30057af0 con 0x7fad38159bd0 2023-10-16T12:34:41.865 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:34:41.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2406ae70 msgr2=0x7fad2406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:41.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2406ae70 0x7fad2406d330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fad38158ea0 tx=0x7fad28008040 comp rx=0 tx=0).stop 2023-10-16T12:34:41.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 msgr2=0x7fad380c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:41.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 0x7fad380c4490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fad3000c6b0 tx=0x7fad3000cb70 comp rx=0 tx=0).stop 2023-10-16T12:34:41.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 shutdown_connections 2023-10-16T12:34:41.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fad2406ae70 0x7fad2406d330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:41.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fad3806d2c0 0x7fad381f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:41.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad38159bd0 0x7fad380c4490 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:41.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 --2- 172.21.15.60:0/3260076895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad3806b3e0 0x7fad380b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:41.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 >> 172.21.15.60:0/3260076895 conn(0x7fad380ce820 msgr2=0x7fad380af320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:41.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 shutdown_connections 2023-10-16T12:34:41.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:41.866+0000 7fad3ce4d640 1 -- 172.21.15.60:0/3260076895 wait complete. 2023-10-16T12:34:41.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:34:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.118+0000 7f8b9a64b640 1 Processor -- start 2023-10-16T12:34:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.118+0000 7f8b9a64b640 1 -- start start 2023-10-16T12:34:42.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b9a64b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b9415aaa0 0x7f8b9415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:42.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b9a64b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940707e0 con 0x7f8b940ce390 2023-10-16T12:34:42.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b9a64b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9406b7a0 con 0x7f8b9415ae80 2023-10-16T12:34:42.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b9a64b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9415ef20 con 0x7f8b9415aaa0 2023-10-16T12:34:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b93fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b9415aaa0 0x7f8b9415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b98be8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8b940ce390 0x7f8b94070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51378/0 (socket says 172.21.15.60:51378) 2023-10-16T12:34:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.119+0000 7f8b98be8640 1 -- 172.21.15.60:0/636797977 learned_addr learned my addr 172.21.15.60:0/636797977 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.120+0000 7f8b93fff640 1 -- 172.21.15.60:0/636797977 >> v1:172.21.15.144:6789/0 conn(0x7f8b9415ae80 legacy=0x7f8b940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.120+0000 7f8b93fff640 1 -- 172.21.15.60:0/636797977 >> v1:172.21.15.60:6789/0 conn(0x7f8b940ce390 legacy=0x7f8b94070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.120+0000 7f8b93fff640 1 -- 172.21.15.60:0/636797977 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9415f200 con 0x7f8b9415aaa0 2023-10-16T12:34:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.120+0000 7f8b937fe640 1 -- 172.21.15.60:0/636797977 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467385906 0 0) 0x7f8b940707e0 con 0x7f8b940ce390 2023-10-16T12:34:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.121+0000 7f8b93fff640 1 --2- 172.21.15.60:0/636797977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b9415aaa0 0x7f8b9415e820 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8b840097a0 tx=0x7f8b8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2d5de2322878ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b937fe640 1 -- 172.21.15.60:0/636797977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84031830 con 0x7f8b9415aaa0 2023-10-16T12:34:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b937fe640 1 -- 172.21.15.60:0/636797977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b84031990 con 0x7f8b9415aaa0 2023-10-16T12:34:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b937fe640 1 -- 172.21.15.60:0/636797977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84031d20 con 0x7f8b9415aaa0 2023-10-16T12:34:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 -- 172.21.15.60:0/636797977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b9415aaa0 msgr2=0x7f8b9415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/636797977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b9415aaa0 0x7f8b9415e820 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8b840097a0 tx=0x7f8b8402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 -- 172.21.15.60:0/636797977 shutdown_connections 2023-10-16T12:34:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/636797977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b9415aaa0 0x7f8b9415e820 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 -- 172.21.15.60:0/636797977 >> 172.21.15.60:0/636797977 conn(0x7f8b940cb410 msgr2=0x7f8b940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 -- 172.21.15.60:0/636797977 shutdown_connections 2023-10-16T12:34:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.122+0000 7f8b9a64b640 1 -- 172.21.15.60:0/636797977 wait complete. 2023-10-16T12:34:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.124+0000 7f8b9a64b640 1 Processor -- start 2023-10-16T12:34:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.124+0000 7f8b9a64b640 1 -- start start 2023-10-16T12:34:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b9a64b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 0x7f8b94156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b9a64b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9415aaa0 0x7f8b94159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b9a64b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9415ae80 0x7f8b9415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b9a64b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b940c4850 con 0x7f8b940ce390 2023-10-16T12:34:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b9a64b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94156f50 con 0x7f8b9415aaa0 2023-10-16T12:34:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b9a64b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b940c3f20 con 0x7f8b9415ae80 2023-10-16T12:34:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.125+0000 7f8b93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 0x7f8b94156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 0x7f8b94156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42044/0 (socket says 172.21.15.60:42044) 2023-10-16T12:34:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 -- 172.21.15.60:0/681642217 learned_addr learned my addr 172.21.15.60:0/681642217 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 -- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9415ae80 msgr2=0x7f8b9415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9415ae80 0x7f8b9415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 -- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9415aaa0 msgr2=0x7f8b94159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9415aaa0 0x7f8b94159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 -- 172.21.15.60:0/681642217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84009400 con 0x7f8b940ce390 2023-10-16T12:34:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.126+0000 7f8b93fff640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 0x7f8b94156850 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b840098c0 tx=0x7f8b840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.127+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8403ba70 con 0x7f8b940ce390 2023-10-16T12:34:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.127+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8403bbf0 con 0x7f8b940ce390 2023-10-16T12:34:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.127+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b940c4200 con 0x7f8b940ce390 2023-10-16T12:34:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.127+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b840453f0 con 0x7f8b940ce390 2023-10-16T12:34:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.127+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b940c4ce0 con 0x7f8b940ce390 2023-10-16T12:34:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.128+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9404e160 con 0x7f8b940ce390 2023-10-16T12:34:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.129+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b8404e450 con 0x7f8b940ce390 2023-10-16T12:34:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.130+0000 7f8b917fa640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8b7806ae70 0x7f8b7806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.130+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8b840abb60 con 0x7f8b940ce390 2023-10-16T12:34:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.131+0000 7f8b8b7fe640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8b7806ae70 0x7f8b7806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.132+0000 7f8b8b7fe640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8b7806ae70 0x7f8b7806d330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c000f60 tx=0x7f8b7c0079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.135+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b84076ef0 con 0x7f8b940ce390 2023-10-16T12:34:42.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.335+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8b940cdc60 con 0x7f8b940ce390 2023-10-16T12:34:42.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.869+0000 7f8b917fa640 1 -- 172.21.15.60:0/681642217 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v34) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8b8407a510 con 0x7f8b940ce390 2023-10-16T12:34:42.872 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:34:42.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8b7806ae70 msgr2=0x7f8b7806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8b7806ae70 0x7f8b7806d330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c000f60 tx=0x7f8b7c0079b0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 msgr2=0x7f8b94156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:42.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 0x7f8b94156850 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b840098c0 tx=0x7f8b840046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 shutdown_connections 2023-10-16T12:34:42.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f8b7806ae70 0x7f8b7806d330 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9415ae80 0x7f8b9415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9415aaa0 0x7f8b94159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 --2- 172.21.15.60:0/681642217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b940ce390 0x7f8b94156850 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:42.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 >> 172.21.15.60:0/681642217 conn(0x7f8b940cb410 msgr2=0x7f8b9406c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:42.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 shutdown_connections 2023-10-16T12:34:42.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:42.873+0000 7f8b9a64b640 1 -- 172.21.15.60:0/681642217 wait complete. 2023-10-16T12:34:42.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:34:43.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.100+0000 7fcf07be3640 1 Processor -- start 2023-10-16T12:34:43.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.101+0000 7fcf07be3640 1 -- start start 2023-10-16T12:34:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.102+0000 7fcf07be3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fcf00069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.102+0000 7fcf07be3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcf0006a2e0 0x7fcf000c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.102+0000 7fcf07be3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0006a6f0 con 0x7fcf0015d150 2023-10-16T12:34:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.102+0000 7fcf07be3640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0015d530 con 0x7fcf0006a2e0 2023-10-16T12:34:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.102+0000 7fcf07be3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000ac890 con 0x7fcf000ad350 2023-10-16T12:34:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.102+0000 7fcf0597f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcf0015d150 0x7fcf00160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51390/0 (socket says 172.21.15.60:51390) 2023-10-16T12:34:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0597f640 1 -- 172.21.15.60:0/776548275 learned_addr learned my addr 172.21.15.60:0/776548275 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:43.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf06180640 1 --2- 172.21.15.60:0/776548275 >> v2:172.21.15.144:3300/0 conn(0x7fcf0006a2e0 0x7fcf000c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927287605 0 0) 0x7fcf0006a6f0 con 0x7fcf0015d150 2023-10-16T12:34:43.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec003520 con 0x7fcf0015d150 2023-10-16T12:34:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0517e640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fcf00069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436293100 0 0) 0x7fceec003520 con 0x7fcf0015d150 2023-10-16T12:34:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 msgr2=0x7fcf00069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.103+0000 7fcf0497d640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fcf00069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 >> v2:172.21.15.144:3300/0 conn(0x7fcf0006a2e0 msgr2=0x7fcf000c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 --2- 172.21.15.60:0/776548275 >> v2:172.21.15.144:3300/0 conn(0x7fcf0006a2e0 0x7fcf000c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00161650 con 0x7fcf0015d150 2023-10-16T12:34:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcefc001670 con 0x7fcf0015d150 2023-10-16T12:34:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 >> v1:172.21.15.60:6789/0 conn(0x7fcf0015d150 legacy=0x7fcf00160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceec003970 0x7fceec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fceec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fceec004d00 0x7fceec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0015d530 con 0x7fceec003970 2023-10-16T12:34:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000ac890 con 0x7fceec004d00 2023-10-16T12:34:43.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec00b980 con 0x7fcf000ad350 2023-10-16T12:34:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0597f640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceec003970 0x7fceec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0517e640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fceec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf06180640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fceec004d00 0x7fceec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.104+0000 7fcf0597f640 1 -- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 msgr2=0x7fceec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.105+0000 7fcf0597f640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fceec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.105+0000 7fcf0597f640 1 -- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fceec004d00 msgr2=0x7fceec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:43.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.105+0000 7fcf0597f640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fceec004d00 0x7fceec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.105+0000 7fcf0597f640 1 -- 172.21.15.60:0/776548275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec003520 con 0x7fceec003970 2023-10-16T12:34:43.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.105+0000 7fcf0517e640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf000ad350 0x7fceec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:43.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.106+0000 7fcf0597f640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceec003970 0x7fceec003d90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00c270 tx=0x7fceec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a0580265636e6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.106+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc02cae0 con 0x7fceec003970 2023-10-16T12:34:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.106+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc02cc40 con 0x7fceec003970 2023-10-16T12:34:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.106+0000 7fcf0497d640 1 -- 172.21.15.60:0/776548275 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0058d0 con 0x7fceec003970 2023-10-16T12:34:43.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf07be3640 1 -- 172.21.15.60:0/776548275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceec003970 msgr2=0x7fceec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf07be3640 1 --2- 172.21.15.60:0/776548275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fceec003970 0x7fceec003d90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00c270 tx=0x7fceec0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf06180640 1 -- 172.21.15.60:0/776548275 reap_dead start 2023-10-16T12:34:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf07be3640 1 -- 172.21.15.60:0/776548275 shutdown_connections 2023-10-16T12:34:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf07be3640 1 -- 172.21.15.60:0/776548275 >> 172.21.15.60:0/776548275 conn(0x7fcf000ce9b0 msgr2=0x7fcf000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf07be3640 1 -- 172.21.15.60:0/776548275 shutdown_connections 2023-10-16T12:34:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.107+0000 7fcf07be3640 1 -- 172.21.15.60:0/776548275 wait complete. 2023-10-16T12:34:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.109+0000 7fcf07be3640 1 Processor -- start 2023-10-16T12:34:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.109+0000 7fcf07be3640 1 -- start start 2023-10-16T12:34:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf07be3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fceec003970 0x7fcf001f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf07be3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 0x7fcf000c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf07be3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf001602a0 0x7fcf000c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf07be3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0015dd80 con 0x7fcf000c6440 2023-10-16T12:34:43.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf07be3640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf001f26e0 con 0x7fcf001602a0 2023-10-16T12:34:43.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf07be3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000c68c0 con 0x7fceec003970 2023-10-16T12:34:43.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0517e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 0x7fcf000c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0517e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 0x7fcf000c7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42052/0 (socket says 172.21.15.60:42052) 2023-10-16T12:34:43.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0517e640 1 -- 172.21.15.60:0/1268376351 learned_addr learned my addr 172.21.15.60:0/1268376351 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:43.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0597f640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fceec003970 0x7fcf001f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf06180640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf001602a0 0x7fcf000c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0517e640 1 -- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fceec003970 msgr2=0x7fcf001f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0517e640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fceec003970 0x7fcf001f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.110+0000 7fcf0517e640 1 -- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf001602a0 msgr2=0x7fcf000c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcf0517e640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf001602a0 0x7fcf000c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcf0517e640 1 -- 172.21.15.60:0/1268376351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000c6ba0 con 0x7fcf000c6440 2023-10-16T12:34:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcf06180640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf001602a0 0x7fcf000c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcf0517e640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 0x7fcf000c7110 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e7e0 tx=0x7fcef000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef000c6d0 con 0x7fcf000c6440 2023-10-16T12:34:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef000c830 con 0x7fcf000c6440 2023-10-16T12:34:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef000cba0 con 0x7fcf000c6440 2023-10-16T12:34:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0015d730 con 0x7fcf000c6440 2023-10-16T12:34:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.111+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0015f4e0 con 0x7fcf000c6440 2023-10-16T12:34:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.112+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0015df30 con 0x7fcf000c6440 2023-10-16T12:34:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.114+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcef000cd00 con 0x7fcf000c6440 2023-10-16T12:34:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.115+0000 7fcef67fc640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcee406ae70 0x7fcee406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.115+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcef0089400 con 0x7fcf000c6440 2023-10-16T12:34:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.115+0000 7fcf0597f640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcee406ae70 0x7fcee406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.117+0000 7fcf0597f640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcee406ae70 0x7fcee406d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcefc034000 tx=0x7fcefc007000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.120+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf0015df30 con 0x7fcf000c6440 2023-10-16T12:34:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.358+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcf0004e160 con 0x7fcf000c6440 2023-10-16T12:34:43.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.913+0000 7fcef67fc640 1 -- 172.21.15.60:0/1268376351 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fcef0054470 con 0x7fcf000c6440 2023-10-16T12:34:43.918 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 4 and data pool 5 2023-10-16T12:34:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.930+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcee406ae70 msgr2=0x7fcee406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.930+0000 7fcf07be3640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcee406ae70 0x7fcee406d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcefc034000 tx=0x7fcefc007000 comp rx=0 tx=0).stop 2023-10-16T12:34:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.930+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 msgr2=0x7fcf000c7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.930+0000 7fcf07be3640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 0x7fcf000c7110 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e7e0 tx=0x7fcef000eca0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.932+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 shutdown_connections 2023-10-16T12:34:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.932+0000 7fcf07be3640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fcee406ae70 0x7fcee406d330 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.932+0000 7fcf07be3640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf001602a0 0x7fcf000c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.932+0000 7fcf07be3640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf000c6440 0x7fcf000c7110 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.932+0000 7fcf07be3640 1 --2- 172.21.15.60:0/1268376351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fceec003970 0x7fcf001f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.932+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 >> 172.21.15.60:0/1268376351 conn(0x7fcf000ce9b0 msgr2=0x7fcf0006c390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.933+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 shutdown_connections 2023-10-16T12:34:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:43.933+0000 7fcf07be3640 1 -- 172.21.15.60:0/1268376351 wait complete. 2023-10-16T12:34:43.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:44.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.150+0000 7f1fc8bc7640 1 Processor -- start 2023-10-16T12:34:44.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.150+0000 7f1fc8bc7640 1 -- start start 2023-10-16T12:34:44.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc8bc7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1fc415ae80 0x7f1fc415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc8bc7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 0x7f1fc4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc8bc7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc415ed60 con 0x7f1fc415aaa0 2023-10-16T12:34:44.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc8bc7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc406b810 con 0x7f1fc415ae80 2023-10-16T12:34:44.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc8bc7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc40707e0 con 0x7f1fc40ce390 2023-10-16T12:34:44.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1fc415aaa0 0x7f1fc40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51396/0 (socket says 172.21.15.60:51396) 2023-10-16T12:34:44.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc259c640 1 -- 172.21.15.60:0/2055780835 learned_addr learned my addr 172.21.15.60:0/2055780835 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:44.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc2d9d640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 0x7f1fc4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242719403 0 0) 0x7f1fc415ed60 con 0x7f1fc415aaa0 2023-10-16T12:34:44.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fbbfff640 1 --2- 172.21.15.60:0/2055780835 >> v2:172.21.15.144:3300/0 conn(0x7f1fc415ae80 0x7f1fc415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.151+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0003520 con 0x7f1fc415aaa0 2023-10-16T12:34:44.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514551871 0 0) 0x7f1fb0003520 con 0x7f1fc415aaa0 2023-10-16T12:34:44.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 msgr2=0x7f1fc4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:44.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 0x7f1fc4070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 >> v2:172.21.15.144:3300/0 conn(0x7f1fc415ae80 msgr2=0x7f1fc415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:44.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 --2- 172.21.15.60:0/2055780835 >> v2:172.21.15.144:3300/0 conn(0x7f1fc415ae80 0x7f1fc415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc415ef90 con 0x7f1fc415aaa0 2023-10-16T12:34:44.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1fb4001690 con 0x7f1fc415aaa0 2023-10-16T12:34:44.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 >> v1:172.21.15.60:6789/0 conn(0x7f1fc415aaa0 legacy=0x7f1fc40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fb0003970 0x7f1fb0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 0x7f1fb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0004d00 0x7f1fb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc406b810 con 0x7f1fb0003970 2023-10-16T12:34:44.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc40707e0 con 0x7f1fb0004d00 2023-10-16T12:34:44.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb000b980 con 0x7f1fc40ce390 2023-10-16T12:34:44.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fbbfff640 1 --2- 172.21.15.60:0/2055780835 >> v2:172.21.15.144:3300/0 conn(0x7f1fc415ae80 0x7f1fc415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:44.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.152+0000 7f1fc259c640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fb0003970 0x7f1fb0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.153+0000 7f1fc259c640 1 -- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 msgr2=0x7f1fb00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:44.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.153+0000 7f1fc259c640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40ce390 0x7f1fb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.153+0000 7f1fc259c640 1 -- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0004d00 msgr2=0x7f1fb0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:44.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.153+0000 7f1fc259c640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0004d00 0x7f1fb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.153+0000 7f1fc259c640 1 -- 172.21.15.60:0/2055780835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0003520 con 0x7f1fb0003970 2023-10-16T12:34:44.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.154+0000 7f1fc259c640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fb0003970 0x7f1fb0003d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400c240 tx=0x7f1fb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5852c1ed6129f859 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.154+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4005ad0 con 0x7f1fb0003970 2023-10-16T12:34:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.154+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4005c50 con 0x7f1fb0003970 2023-10-16T12:34:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.154+0000 7f1fc1d9b640 1 -- 172.21.15.60:0/2055780835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb402c8a0 con 0x7f1fb0003970 2023-10-16T12:34:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/2055780835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fb0003970 msgr2=0x7f1fb0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/2055780835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fb0003970 0x7f1fb0003d90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400c240 tx=0x7f1fb00057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc2d9d640 1 -- 172.21.15.60:0/2055780835 reap_dead start 2023-10-16T12:34:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/2055780835 shutdown_connections 2023-10-16T12:34:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/2055780835 >> 172.21.15.60:0/2055780835 conn(0x7f1fc40cb410 msgr2=0x7f1fc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/2055780835 shutdown_connections 2023-10-16T12:34:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.155+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/2055780835 wait complete. 2023-10-16T12:34:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.157+0000 7f1fc8bc7640 1 Processor -- start 2023-10-16T12:34:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.157+0000 7f1fc8bc7640 1 -- start start 2023-10-16T12:34:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0003970 0x7f1fc41f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40c4db0 0x7f1fc4155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 0x7f1fc415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc40c9460 con 0x7f1fc415bfe0 2023-10-16T12:34:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc41f3220 con 0x7f1fb0003970 2023-10-16T12:34:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc415c9a0 con 0x7f1fc40c4db0 2023-10-16T12:34:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 0x7f1fc415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 0x7f1fc415c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42070/0 (socket says 172.21.15.60:42070) 2023-10-16T12:34:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 -- 172.21.15.60:0/3562498227 learned_addr learned my addr 172.21.15.60:0/3562498227 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fbbfff640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40c4db0 0x7f1fc4155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 -- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40c4db0 msgr2=0x7f1fc4155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40c4db0 0x7f1fc4155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 -- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0003970 msgr2=0x7f1fc41f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0003970 0x7f1fc41f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 -- 172.21.15.60:0/3562498227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc415e3d0 con 0x7f1fc415bfe0 2023-10-16T12:34:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fbbfff640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40c4db0 0x7f1fc4155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc2d9d640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 0x7f1fc415c460 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40c5a10 tx=0x7f1fbc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc010040 con 0x7f1fc415bfe0 2023-10-16T12:34:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fbc00bb10 con 0x7f1fc415bfe0 2023-10-16T12:34:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc014400 con 0x7f1fc415bfe0 2023-10-16T12:34:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.159+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc4156370 con 0x7f1fc415bfe0 2023-10-16T12:34:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.160+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc415b6a0 con 0x7f1fc415bfe0 2023-10-16T12:34:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.161+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc404e160 con 0x7f1fc415bfe0 2023-10-16T12:34:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.163+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1fbc014560 con 0x7f1fc415bfe0 2023-10-16T12:34:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.164+0000 7f1fbaffd640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1fa406aef0 0x7f1fa406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.164+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fbc0897b0 con 0x7f1fc415bfe0 2023-10-16T12:34:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.164+0000 7f1fc259c640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1fa406aef0 0x7f1fa406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.165+0000 7f1fc259c640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1fa406aef0 0x7f1fa406d3b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4034000 tx=0x7f1fb402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.169+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1fbc0549d0 con 0x7f1fc415bfe0 2023-10-16T12:34:44.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.356+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fc415e3d0 con 0x7f1fc415bfe0 2023-10-16T12:34:44.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.358+0000 7f1fbaffd640 1 -- 172.21.15.60:0/3562498227 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f1fc415e3d0 con 0x7f1fc415bfe0 2023-10-16T12:34:44.361 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:44.362 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:43.360038+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:44.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1fa406aef0 msgr2=0x7f1fa406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1fa406aef0 0x7f1fa406d3b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4034000 tx=0x7f1fb402e000 comp rx=0 tx=0).stop 2023-10-16T12:34:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 msgr2=0x7f1fc415c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 0x7f1fc415c460 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40c5a10 tx=0x7f1fbc00b480 comp rx=0 tx=0).stop 2023-10-16T12:34:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 shutdown_connections 2023-10-16T12:34:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1fa406aef0 0x7f1fa406d3b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1fc40c4db0 0x7f1fc4155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc415bfe0 0x7f1fc415c460 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 --2- 172.21.15.60:0/3562498227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1fb0003970 0x7f1fc41f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 >> 172.21.15.60:0/3562498227 conn(0x7f1fc40cb410 msgr2=0x7f1fc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:44.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 shutdown_connections 2023-10-16T12:34:44.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.361+0000 7f1fc8bc7640 1 -- 172.21.15.60:0/3562498227 wait complete. 2023-10-16T12:34:44.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:44.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.581+0000 7fb21e5d2640 1 Processor -- start 2023-10-16T12:34:44.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.581+0000 7fb21e5d2640 1 -- start start 2023-10-16T12:34:44.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21e5d2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb218159950 0x7fb21815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21e5d2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb218070450 0x7fb2180708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21e5d2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21806c100 0x7fb21806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21e5d2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb218070e20 con 0x7fb218159950 2023-10-16T12:34:44.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21e5d2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21815e490 con 0x7fb218070450 2023-10-16T12:34:44.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21e5d2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21815aef0 con 0x7fb21806c100 2023-10-16T12:34:44.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb218159950 0x7fb21815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb218159950 0x7fb21815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42074/0 (socket says 172.21.15.60:42074) 2023-10-16T12:34:44.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 -- 172.21.15.60:0/646942757 learned_addr learned my addr 172.21.15.60:0/646942757 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:44.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21cdcf640 1 --2- 172.21.15.60:0/646942757 >> v2:172.21.15.144:3300/0 conn(0x7fb218070450 0x7fb2180708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 -- 172.21.15.60:0/646942757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21806c100 msgr2=0x7fb21806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:44.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21ddd1640 1 --2- 172.21.15.60:0/646942757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21806c100 0x7fb21806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/646942757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21806c100 0x7fb21806c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 -- 172.21.15.60:0/646942757 >> v2:172.21.15.144:3300/0 conn(0x7fb218070450 msgr2=0x7fb2180708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:44.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/646942757 >> v2:172.21.15.144:3300/0 conn(0x7fb218070450 0x7fb2180708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.583+0000 7fb21d5d0640 1 -- 172.21.15.60:0/646942757 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21815d910 con 0x7fb218159950 2023-10-16T12:34:44.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/646942757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb21815a9b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2180cab70 tx=0x7fb21402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9288e3c1e8207dee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:44.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb20ffff640 1 -- 172.21.15.60:0/646942757 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2140317c0 con 0x7fb218159950 2023-10-16T12:34:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb20ffff640 1 -- 172.21.15.60:0/646942757 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb214031920 con 0x7fb218159950 2023-10-16T12:34:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb20ffff640 1 -- 172.21.15.60:0/646942757 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214031cb0 con 0x7fb218159950 2023-10-16T12:34:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 -- 172.21.15.60:0/646942757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 msgr2=0x7fb21815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/646942757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb21815a9b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2180cab70 tx=0x7fb21402b860 comp rx=0 tx=0).stop 2023-10-16T12:34:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 -- 172.21.15.60:0/646942757 shutdown_connections 2023-10-16T12:34:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/646942757 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21806c100 0x7fb21806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/646942757 >> v2:172.21.15.144:3300/0 conn(0x7fb218070450 0x7fb2180708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/646942757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb21815a9b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 -- 172.21.15.60:0/646942757 >> 172.21.15.60:0/646942757 conn(0x7fb2180ce6a0 msgr2=0x7fb2180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 -- 172.21.15.60:0/646942757 shutdown_connections 2023-10-16T12:34:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.585+0000 7fb21e5d2640 1 -- 172.21.15.60:0/646942757 wait complete. 2023-10-16T12:34:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 Processor -- start 2023-10-16T12:34:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 -- start start 2023-10-16T12:34:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb21806c100 0x7fb2181519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb218070450 0x7fb2180c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb2180c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180c2a00 con 0x7fb218159950 2023-10-16T12:34:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180c2ce0 con 0x7fb21806c100 2023-10-16T12:34:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.588+0000 7fb21e5d2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180c2fc0 con 0x7fb218070450 2023-10-16T12:34:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb2180c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb2180c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42078/0 (socket says 172.21.15.60:42078) 2023-10-16T12:34:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 -- 172.21.15.60:0/3841150031 learned_addr learned my addr 172.21.15.60:0/3841150031 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:44.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21cdcf640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb218070450 0x7fb2180c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb21806c100 0x7fb2181519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 -- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb218070450 msgr2=0x7fb2180c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb218070450 0x7fb2180c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 -- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb21806c100 msgr2=0x7fb2181519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb21806c100 0x7fb2181519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21ddd1640 1 -- 172.21.15.60:0/3841150031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2140094d0 con 0x7fb218159950 2023-10-16T12:34:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.589+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb21806c100 0x7fb2181519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.590+0000 7fb21ddd1640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb2180c3aa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb21000cc30 tx=0x7fb21000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.590+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000ce40 con 0x7fb218159950 2023-10-16T12:34:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.590+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb210003990 con 0x7fb218159950 2023-10-16T12:34:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.590+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210003d20 con 0x7fb218159950 2023-10-16T12:34:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.590+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2180c32a0 con 0x7fb218159950 2023-10-16T12:34:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.590+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2180c26f0 con 0x7fb218159950 2023-10-16T12:34:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.591+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2180c37c0 con 0x7fb218159950 2023-10-16T12:34:44.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.592+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb210003e80 con 0x7fb218159950 2023-10-16T12:34:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.593+0000 7fb20dffb640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1fc06ae70 0x7fb1fc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.593+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb210089a80 con 0x7fb218159950 2023-10-16T12:34:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.595+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1fc06ae70 0x7fb1fc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.596+0000 7fb21d5d0640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1fc06ae70 0x7fb1fc06d330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb214036000 tx=0x7fb21403d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:44.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.598+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2180c37c0 con 0x7fb218159950 2023-10-16T12:34:44.685 INFO:tasks.ceph.mds.c.smithi060.stderr:2023-10-16T12:34:44.682+0000 7fa45717b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:34:44.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.791+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2180c37c0 con 0x7fb218159950 2023-10-16T12:34:44.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.793+0000 7fb20dffb640 1 -- 172.21.15.60:0/3841150031 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16570 (secure 0 0 0) 0x7fb2180c37c0 con 0x7fb218159950 2023-10-16T12:34:44.810 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:44.810 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:43.360038+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:44.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1fc06ae70 msgr2=0x7fb1fc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1fc06ae70 0x7fb1fc06d330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb214036000 tx=0x7fb21403d040 comp rx=0 tx=0).stop 2023-10-16T12:34:44.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 msgr2=0x7fb2180c3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:44.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb2180c3aa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb21000cc30 tx=0x7fb21000a5c0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 shutdown_connections 2023-10-16T12:34:44.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb1fc06ae70 0x7fb1fc06d330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb218070450 0x7fb2180c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb218159950 0x7fb2180c3aa0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 --2- 172.21.15.60:0/3841150031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb21806c100 0x7fb2181519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:44.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.795+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 >> 172.21.15.60:0/3841150031 conn(0x7fb2180ce6a0 msgr2=0x7fb2180af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:44.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.796+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 shutdown_connections 2023-10-16T12:34:44.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:44.796+0000 7fb21e5d2640 1 -- 172.21.15.60:0/3841150031 wait complete. 2023-10-16T12:34:44.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:34:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.039+0000 7f9855393640 1 Processor -- start 2023-10-16T12:34:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.039+0000 7f9855393640 1 -- start start 2023-10-16T12:34:45.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f9855393640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f985015cd70 0x7f98500d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f9855393640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985015d150 con 0x7f98500cded0 2023-10-16T12:34:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f9855393640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98500d1180 con 0x7f98500b8480 2023-10-16T12:34:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f9855393640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98500d1460 con 0x7f985015cd70 2023-10-16T12:34:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f9854b92640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f98500cded0 0x7f98500b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51412/0 (socket says 172.21.15.60:51412) 2023-10-16T12:34:45.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f9854b92640 1 -- 172.21.15.60:0/3635012339 learned_addr learned my addr 172.21.15.60:0/3635012339 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:45.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428477284 0 0) 0x7f985015d150 con 0x7f98500cded0 2023-10-16T12:34:45.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f984ffff640 1 --2- 172.21.15.60:0/3635012339 >> v2:172.21.15.144:3300/0 conn(0x7f985015cd70 0x7f98500d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9838003520 con 0x7f98500cded0 2023-10-16T12:34:45.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4153728980 0 0) 0x7f9838003520 con 0x7f98500cded0 2023-10-16T12:34:45.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.041+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 >> v2:172.21.15.144:3300/0 conn(0x7f985015cd70 msgr2=0x7f98500d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 --2- 172.21.15.60:0/3635012339 >> v2:172.21.15.144:3300/0 conn(0x7f985015cd70 0x7f98500d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 >> v1:172.21.15.144:6789/0 conn(0x7f98500b8480 legacy=0x7f98500cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500d1740 con 0x7f98500cded0 2023-10-16T12:34:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f98480019f0 con 0x7f98500cded0 2023-10-16T12:34:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 >> v1:172.21.15.60:6789/0 conn(0x7f98500cded0 legacy=0x7f98500b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9838003970 0x7f9838003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98380047c0 0x7f9838004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9838005180 0x7f983800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500d1460 con 0x7f9838003970 2023-10-16T12:34:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983800c000 con 0x7f9838005180 2023-10-16T12:34:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983800c1c0 con 0x7f98380047c0 2023-10-16T12:34:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984ffff640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9838003970 0x7f9838003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f9854b92640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9838005180 0x7f983800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984ffff640 1 -- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98380047c0 msgr2=0x7f9838004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984ffff640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98380047c0 0x7f9838004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.042+0000 7f984ffff640 1 -- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9838005180 msgr2=0x7f983800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.043+0000 7f984ffff640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9838005180 0x7f983800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.043+0000 7f984ffff640 1 -- 172.21.15.60:0/3635012339 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838003520 con 0x7f9838003970 2023-10-16T12:34:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.043+0000 7f9854b92640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9838005180 0x7f983800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.043+0000 7f9854b92640 1 -- 172.21.15.60:0/3635012339 reap_dead start 2023-10-16T12:34:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f984ffff640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9838003970 0x7f9838003d70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9840009ac0 tx=0x7f98380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7196afbc717dfebf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:45.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984002c6b0 con 0x7f9838003970 2023-10-16T12:34:45.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984002c810 con 0x7f9838003970 2023-10-16T12:34:45.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f984effd640 1 -- 172.21.15.60:0/3635012339 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984002cba0 con 0x7f9838003970 2023-10-16T12:34:45.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 -- 172.21.15.60:0/3635012339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9838003970 msgr2=0x7f9838003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9838003970 0x7f9838003d70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9840009ac0 tx=0x7f98380057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 -- 172.21.15.60:0/3635012339 shutdown_connections 2023-10-16T12:34:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 --2- 172.21.15.60:0/3635012339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9838003970 0x7f9838003d70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 -- 172.21.15.60:0/3635012339 >> 172.21.15.60:0/3635012339 conn(0x7f98500cf570 msgr2=0x7f98500cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 -- 172.21.15.60:0/3635012339 shutdown_connections 2023-10-16T12:34:45.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.045+0000 7f9855393640 1 -- 172.21.15.60:0/3635012339 wait complete. 2023-10-16T12:34:45.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.047+0000 7f9855393640 1 Processor -- start 2023-10-16T12:34:45.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.047+0000 7f9855393640 1 -- start start 2023-10-16T12:34:45.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.047+0000 7f9855393640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9838003970 0x7f98500c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9855393640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98500b3df0 0x7f98500b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9855393640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 0x7f985015f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9855393640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500b4790 con 0x7f985015ef30 2023-10-16T12:34:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9855393640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985015fca0 con 0x7f98500b3df0 2023-10-16T12:34:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9855393640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985015ff80 con 0x7f9838003970 2023-10-16T12:34:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 0x7f985015f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 0x7f985015f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42106/0 (socket says 172.21.15.60:42106) 2023-10-16T12:34:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 -- 172.21.15.60:0/1129044853 learned_addr learned my addr 172.21.15.60:0/1129044853 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f984ffff640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9838003970 0x7f98500c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f984f7fe640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98500b3df0 0x7f98500b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 -- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9838003970 msgr2=0x7f98500c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9838003970 0x7f98500c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 -- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98500b3df0 msgr2=0x7f98500b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98500b3df0 0x7f98500b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f9854b92640 1 -- 172.21.15.60:0/1129044853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9850160260 con 0x7f985015ef30 2023-10-16T12:34:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f984ffff640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9838003970 0x7f98500c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.048+0000 7f984f7fe640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98500b3df0 0x7f98500b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.049+0000 7f9854b92640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 0x7f985015f3b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f98500cd0f0 tx=0x7f98480021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.049+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984800b800 con 0x7f985015ef30 2023-10-16T12:34:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.049+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984800b960 con 0x7f985015ef30 2023-10-16T12:34:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.049+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984800bcd0 con 0x7f985015ef30 2023-10-16T12:34:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.049+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9850156800 con 0x7f985015ef30 2023-10-16T12:34:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.049+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98500c8c20 con 0x7f985015ef30 2023-10-16T12:34:45.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.050+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9850156d20 con 0x7f985015ef30 2023-10-16T12:34:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.051+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9848007070 con 0x7f985015ef30 2023-10-16T12:34:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.052+0000 7f984cff9640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f983c06aea0 0x7f983c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.052+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9848089990 con 0x7f985015ef30 2023-10-16T12:34:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.053+0000 7f984ffff640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f983c06aea0 0x7f983c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.054+0000 7f984ffff640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f983c06aea0 0x7f983c06d360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9840036000 tx=0x7f9840030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:45.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.057+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9850156d20 con 0x7f985015ef30 2023-10-16T12:34:45.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.294+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9850156d20 con 0x7f985015ef30 2023-10-16T12:34:45.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.926+0000 7f984cff9640 1 -- 172.21.15.60:0/1129044853 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9850156d20 con 0x7f985015ef30 2023-10-16T12:34:45.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f983c06aea0 msgr2=0x7f983c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:45.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f983c06aea0 0x7f983c06d360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9840036000 tx=0x7f9840030000 comp rx=0 tx=0).stop 2023-10-16T12:34:45.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 msgr2=0x7f985015f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:45.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 0x7f985015f3b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f98500cd0f0 tx=0x7f98480021d0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 shutdown_connections 2023-10-16T12:34:45.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f983c06aea0 0x7f983c06d360 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f985015ef30 0x7f985015f3b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98500b3df0 0x7f98500b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 --2- 172.21.15.60:0/1129044853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9838003970 0x7f98500c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:45.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 >> 172.21.15.60:0/1129044853 conn(0x7f98500cf570 msgr2=0x7f98500d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:45.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 shutdown_connections 2023-10-16T12:34:45.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:45.929+0000 7f9855393640 1 -- 172.21.15.60:0/1129044853 wait complete. 2023-10-16T12:34:45.955 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:34:45.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:34:46.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.184+0000 7fda7a32e640 1 Processor -- start 2023-10-16T12:34:46.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.185+0000 7fda7a32e640 1 -- start start 2023-10-16T12:34:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.186+0000 7fda7a32e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.186+0000 7fda7a32e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda740b8480 0x7fda740cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.186+0000 7fda7a32e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda740d0be0 0x7fda740cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.186+0000 7fda7a32e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7415d150 con 0x7fda740d0be0 2023-10-16T12:34:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.186+0000 7fda7a32e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda740d1180 con 0x7fda740b8480 2023-10-16T12:34:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.186+0000 7fda7a32e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda740d1460 con 0x7fda7415cd70 2023-10-16T12:34:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda740d0be0 0x7fda740cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:46.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda740d0be0 0x7fda740cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42118/0 (socket says 172.21.15.60:42118) 2023-10-16T12:34:46.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 -- 172.21.15.60:0/2491239764 learned_addr learned my addr 172.21.15.60:0/2491239764 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:46.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda7932c640 1 --2- 172.21.15.60:0/2491239764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:46.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda78b2b640 1 --2- 172.21.15.60:0/2491239764 >> v2:172.21.15.144:3300/0 conn(0x7fda740b8480 0x7fda740cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 -- 172.21.15.60:0/2491239764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 msgr2=0x7fda740d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 --2- 172.21.15.60:0/2491239764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 -- 172.21.15.60:0/2491239764 >> v2:172.21.15.144:3300/0 conn(0x7fda740b8480 msgr2=0x7fda740cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 --2- 172.21.15.60:0/2491239764 >> v2:172.21.15.144:3300/0 conn(0x7fda740b8480 0x7fda740cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda79b2d640 1 -- 172.21.15.60:0/2491239764 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda740d1740 con 0x7fda740d0be0 2023-10-16T12:34:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.187+0000 7fda78b2b640 1 --2- 172.21.15.60:0/2491239764 >> v2:172.21.15.144:3300/0 conn(0x7fda740b8480 0x7fda740cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:46.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.188+0000 7fda79b2d640 1 --2- 172.21.15.60:0/2491239764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740d0be0 0x7fda740cdd00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fda74066ab0 tx=0x7fda6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221d0e47bab55d75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:46.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.189+0000 7fda6bfff640 1 -- 172.21.15.60:0/2491239764 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c02ca80 con 0x7fda740d0be0 2023-10-16T12:34:46.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.189+0000 7fda6bfff640 1 -- 172.21.15.60:0/2491239764 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c02cbe0 con 0x7fda740d0be0 2023-10-16T12:34:46.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.189+0000 7fda6bfff640 1 -- 172.21.15.60:0/2491239764 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0343f0 con 0x7fda740d0be0 2023-10-16T12:34:46.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.189+0000 7fda7a32e640 1 -- 172.21.15.60:0/2491239764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740d0be0 msgr2=0x7fda740cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:46.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.189+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2491239764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740d0be0 0x7fda740cdd00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fda74066ab0 tx=0x7fda6c02bee0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 -- 172.21.15.60:0/2491239764 shutdown_connections 2023-10-16T12:34:46.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2491239764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740d0be0 0x7fda740cdd00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2491239764 >> v2:172.21.15.144:3300/0 conn(0x7fda740b8480 0x7fda740cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2491239764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 -- 172.21.15.60:0/2491239764 >> 172.21.15.60:0/2491239764 conn(0x7fda740cf570 msgr2=0x7fda740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 -- 172.21.15.60:0/2491239764 shutdown_connections 2023-10-16T12:34:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.190+0000 7fda7a32e640 1 -- 172.21.15.60:0/2491239764 wait complete. 2023-10-16T12:34:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.191+0000 7fda7a32e640 1 Processor -- start 2023-10-16T12:34:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.192+0000 7fda7a32e640 1 -- start start 2023-10-16T12:34:46.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7a32e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 0x7fda740c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7a32e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda740d0be0 0x7fda74150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7a32e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7a32e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda741f1f10 con 0x7fda740b8480 2023-10-16T12:34:46.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7a32e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740c8e00 con 0x7fda740d0be0 2023-10-16T12:34:46.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7a32e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda740c8b50 con 0x7fda7415cd70 2023-10-16T12:34:46.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 0x7fda740c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:46.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 0x7fda740c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42126/0 (socket says 172.21.15.60:42126) 2023-10-16T12:34:46.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 -- 172.21.15.60:0/2117945697 learned_addr learned my addr 172.21.15.60:0/2117945697 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:46.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 -- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 msgr2=0x7fda740c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:46.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 -- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda740d0be0 msgr2=0x7fda74150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:46.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda78b2b640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda740d0be0 0x7fda74150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:46.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda740d0be0 0x7fda74150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.193+0000 7fda7932c640 1 -- 172.21.15.60:0/2117945697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c009b30 con 0x7fda740b8480 2023-10-16T12:34:46.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.194+0000 7fda7932c640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 0x7fda740c6ef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fda7000d830 tx=0x7fda7000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:46.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.194+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000b6c0 con 0x7fda740b8480 2023-10-16T12:34:46.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.194+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000b820 con 0x7fda740b8480 2023-10-16T12:34:46.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.194+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000bbb0 con 0x7fda740b8480 2023-10-16T12:34:46.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.194+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda741f2100 con 0x7fda740b8480 2023-10-16T12:34:46.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.194+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda741f26c0 con 0x7fda740b8480 2023-10-16T12:34:46.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.195+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda740d1a00 con 0x7fda740b8480 2023-10-16T12:34:46.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.197+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fda7001e460 con 0x7fda740b8480 2023-10-16T12:34:46.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.198+0000 7fda69ffb640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fda5806ae70 0x7fda5806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:46.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.198+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda70089c90 con 0x7fda740b8480 2023-10-16T12:34:46.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.198+0000 7fda78b2b640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fda5806ae70 0x7fda5806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:46.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.200+0000 7fda78b2b640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fda5806ae70 0x7fda5806d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda740cd010 tx=0x7fda64008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:46.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.202+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda740d1a00 con 0x7fda740b8480 2023-10-16T12:34:46.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.439+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fda740d1a00 con 0x7fda740b8480 2023-10-16T12:34:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.929+0000 7fda69ffb640 1 -- 172.21.15.60:0/2117945697 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda740d1a00 con 0x7fda740b8480 2023-10-16T12:34:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.932+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fda5806ae70 msgr2=0x7fda5806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.932+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fda5806ae70 0x7fda5806d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda740cd010 tx=0x7fda64008040 comp rx=0 tx=0).stop 2023-10-16T12:34:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.932+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 msgr2=0x7fda740c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.932+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 0x7fda740c6ef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fda7000d830 tx=0x7fda7000dcf0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 shutdown_connections 2023-10-16T12:34:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fda5806ae70 0x7fda5806d330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda7415cd70 0x7fda740c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda740d0be0 0x7fda74150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 --2- 172.21.15.60:0/2117945697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda740b8480 0x7fda740c6ef0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 >> 172.21.15.60:0/2117945697 conn(0x7fda740cf570 msgr2=0x7fda740b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 shutdown_connections 2023-10-16T12:34:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:46.933+0000 7fda7a32e640 1 -- 172.21.15.60:0/2117945697 wait complete. 2023-10-16T12:34:46.959 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:34:46.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:34:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.191+0000 7f10bf3d5640 1 Processor -- start 2023-10-16T12:34:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.191+0000 7f10bf3d5640 1 -- start start 2023-10-16T12:34:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.192+0000 7f10bf3d5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f10b80afef0 0x7f10b80cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.192+0000 7f10bf3d5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80d0180 con 0x7f10b80b2bb0 2023-10-16T12:34:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.192+0000 7f10bf3d5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80cfea0 con 0x7f10b806aeb0 2023-10-16T12:34:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.192+0000 7f10bf3d5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b80b04d0 con 0x7f10b80afef0 2023-10-16T12:34:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.192+0000 7f10bd972640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f10b80b2bb0 0x7f10b80b0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51422/0 (socket says 172.21.15.60:51422) 2023-10-16T12:34:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.192+0000 7f10bd972640 1 -- 172.21.15.60:0/2575983947 learned_addr learned my addr 172.21.15.60:0/2575983947 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10bd171640 1 --2- 172.21.15.60:0/2575983947 >> v2:172.21.15.144:3300/0 conn(0x7f10b80afef0 0x7f10b80cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799086507 0 0) 0x7f10b80d0180 con 0x7f10b80b2bb0 2023-10-16T12:34:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a4003520 con 0x7f10b80b2bb0 2023-10-16T12:34:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4196355462 0 0) 0x7f10a4003520 con 0x7f10b80b2bb0 2023-10-16T12:34:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 >> v2:172.21.15.144:3300/0 conn(0x7f10b80afef0 msgr2=0x7f10b80cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 --2- 172.21.15.60:0/2575983947 >> v2:172.21.15.144:3300/0 conn(0x7f10b80afef0 0x7f10b80cf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 >> v1:172.21.15.144:6789/0 conn(0x7f10b806aeb0 legacy=0x7f10b80b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80b07b0 con 0x7f10b80b2bb0 2023-10-16T12:34:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.193+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f10b0001670 con 0x7f10b80b2bb0 2023-10-16T12:34:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 >> v1:172.21.15.60:6789/0 conn(0x7f10b80b2bb0 legacy=0x7f10b80b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a4003970 0x7f10a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a40047c0 0x7f10a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10a40052a0 0x7f10a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80b04d0 con 0x7f10a40052a0 2023-10-16T12:34:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a400c170 con 0x7f10a4003970 2023-10-16T12:34:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a400c330 con 0x7f10a40047c0 2023-10-16T12:34:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10bd972640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10a40052a0 0x7f10a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10bd171640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a4003970 0x7f10a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10bc970640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a40047c0 0x7f10a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.194+0000 7f10bd972640 1 -- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a40047c0 msgr2=0x7f10a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.195+0000 7f10bd972640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a40047c0 0x7f10a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.195+0000 7f10bd972640 1 -- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a4003970 msgr2=0x7f10a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:47.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.195+0000 7f10bd972640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a4003970 0x7f10a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.195+0000 7f10bd972640 1 -- 172.21.15.60:0/2575983947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a4003520 con 0x7f10a40052a0 2023-10-16T12:34:47.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.195+0000 7f10bc970640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10a40047c0 0x7f10a4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:47.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.195+0000 7f10bd972640 1 -- 172.21.15.60:0/2575983947 reap_dead start 2023-10-16T12:34:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.196+0000 7f10bd972640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10a40052a0 0x7f10a400b980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f10b80d0a80 tx=0x7f10a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd13833a43fa9e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.196+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0005ad0 con 0x7f10a40052a0 2023-10-16T12:34:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.196+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b0005c50 con 0x7f10a40052a0 2023-10-16T12:34:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.196+0000 7f10affff640 1 -- 172.21.15.60:0/2575983947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b002c830 con 0x7f10a40052a0 2023-10-16T12:34:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.196+0000 7f10bf3d5640 1 -- 172.21.15.60:0/2575983947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10a40052a0 msgr2=0x7f10a400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.196+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10a40052a0 0x7f10a400b980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f10b80d0a80 tx=0x7f10a40057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.197+0000 7f10bf3d5640 1 -- 172.21.15.60:0/2575983947 shutdown_connections 2023-10-16T12:34:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.197+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/2575983947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10a40052a0 0x7f10a400b980 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.197+0000 7f10bf3d5640 1 -- 172.21.15.60:0/2575983947 >> 172.21.15.60:0/2575983947 conn(0x7f10b80d0ad0 msgr2=0x7f10b80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.197+0000 7f10bf3d5640 1 -- 172.21.15.60:0/2575983947 shutdown_connections 2023-10-16T12:34:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.197+0000 7f10bf3d5640 1 -- 172.21.15.60:0/2575983947 wait complete. 2023-10-16T12:34:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.198+0000 7f10bf3d5640 1 Processor -- start 2023-10-16T12:34:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.199+0000 7f10bf3d5640 1 -- start start 2023-10-16T12:34:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bf3d5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a40052a0 0x7f10b80cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bf3d5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 0x7f10b80cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bf3d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10b80cd6a0 0x7f10b81f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bf3d5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80cce10 con 0x7f10b80b59e0 2023-10-16T12:34:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bf3d5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80cdb30 con 0x7f10a40052a0 2023-10-16T12:34:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bf3d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b80b4c40 con 0x7f10b80cd6a0 2023-10-16T12:34:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 0x7f10b80cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 0x7f10b80cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42136/0 (socket says 172.21.15.60:42136) 2023-10-16T12:34:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 -- 172.21.15.60:0/1998254701 learned_addr learned my addr 172.21.15.60:0/1998254701 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bd171640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a40052a0 0x7f10b80cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bd972640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10b80cd6a0 0x7f10b81f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 -- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10b80cd6a0 msgr2=0x7f10b81f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10b80cd6a0 0x7f10b81f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 -- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a40052a0 msgr2=0x7f10b80cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a40052a0 0x7f10b80cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.200+0000 7f10bc970640 1 -- 172.21.15.60:0/1998254701 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80b4f20 con 0x7f10b80b59e0 2023-10-16T12:34:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.201+0000 7f10bc970640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 0x7f10b80cd160 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f10a800dae0 tx=0x7f10a8002070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.201+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8003da0 con 0x7f10b80b59e0 2023-10-16T12:34:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.201+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a800b710 con 0x7f10b80b59e0 2023-10-16T12:34:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.201+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a800bac0 con 0x7f10b80b59e0 2023-10-16T12:34:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.201+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b80b5200 con 0x7f10b80b59e0 2023-10-16T12:34:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.201+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b81f33e0 con 0x7f10b80b59e0 2023-10-16T12:34:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.202+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b804e160 con 0x7f10b80b59e0 2023-10-16T12:34:47.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.203+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f10a800bc20 con 0x7f10b80b59e0 2023-10-16T12:34:47.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.204+0000 7f10adffb640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f109c06ae70 0x7f109c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.204+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a8089950 con 0x7f10b80b59e0 2023-10-16T12:34:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.205+0000 7f10bd171640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f109c06ae70 0x7f109c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.206+0000 7f10bd171640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f109c06ae70 0x7f109c06d330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f10b80afa50 tx=0x7f10b4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.211+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10a8054960 con 0x7f10b80b59e0 2023-10-16T12:34:47.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.445+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10b80c78b0 con 0x7f10b80b59e0 2023-10-16T12:34:47.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.446+0000 7f10adffb640 1 -- 172.21.15.60:0/1998254701 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f10a8057f80 con 0x7f10b80b59e0 2023-10-16T12:34:47.448 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:47.448 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:34:43.360823+0000","modified":"2023-10-16T12:34:46.440178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/3714850681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3714850681},{"type":"v1","addr":"172.21.15.60:6835","nonce":3714850681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:34:47.448 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2023-10-16T12:34:47.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f109c06ae70 msgr2=0x7f109c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f109c06ae70 0x7f109c06d330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f10b80afa50 tx=0x7f10b4008040 comp rx=0 tx=0).stop 2023-10-16T12:34:47.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 msgr2=0x7f10b80cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 0x7f10b80cd160 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f10a800dae0 tx=0x7f10a8002070 comp rx=0 tx=0).stop 2023-10-16T12:34:47.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 shutdown_connections 2023-10-16T12:34:47.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f109c06ae70 0x7f109c06d330 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10b80cd6a0 0x7f10b81f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10b80b59e0 0x7f10b80cd160 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 --2- 172.21.15.60:0/1998254701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10a40052a0 0x7f10b80cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 >> 172.21.15.60:0/1998254701 conn(0x7f10b80d0ad0 msgr2=0x7f10b80c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 shutdown_connections 2023-10-16T12:34:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.449+0000 7f10bf3d5640 1 -- 172.21.15.60:0/1998254701 wait complete. 2023-10-16T12:34:47.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:34:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.694+0000 7f99b677c640 1 Processor -- start 2023-10-16T12:34:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.694+0000 7f99b677c640 1 -- start start 2023-10-16T12:34:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.695+0000 7f99b677c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99b015ae80 0x7f99b015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.695+0000 7f99b677c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b015ed60 con 0x7f99b00ce390 2023-10-16T12:34:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.695+0000 7f99b677c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00707e0 con 0x7f99b015aaa0 2023-10-16T12:34:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.695+0000 7f99b677c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b006b810 con 0x7f99b015ae80 2023-10-16T12:34:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.696+0000 7f99b4d19640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f99b00ce390 0x7f99b0070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51438/0 (socket says 172.21.15.60:51438) 2023-10-16T12:34:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.696+0000 7f99b4d19640 1 -- 172.21.15.60:0/1663136244 learned_addr learned my addr 172.21.15.60:0/1663136244 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.696+0000 7f99a77fe640 1 --2- 172.21.15.60:0/1663136244 >> v2:172.21.15.144:3300/0 conn(0x7f99b015ae80 0x7f99b015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.696+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839777824 0 0) 0x7f99b015ed60 con 0x7f99b00ce390 2023-10-16T12:34:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.696+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f999c003520 con 0x7f99b00ce390 2023-10-16T12:34:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956352515 0 0) 0x7f999c003520 con 0x7f99b00ce390 2023-10-16T12:34:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 >> v2:172.21.15.144:3300/0 conn(0x7f99b015ae80 msgr2=0x7f99b015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1663136244 >> v2:172.21.15.144:3300/0 conn(0x7f99b015ae80 0x7f99b015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 >> v1:172.21.15.144:6789/0 conn(0x7f99b015aaa0 legacy=0x7f99b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b015ef90 con 0x7f99b00ce390 2023-10-16T12:34:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f99a8001670 con 0x7f99b00ce390 2023-10-16T12:34:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 >> v1:172.21.15.60:6789/0 conn(0x7f99b00ce390 legacy=0x7f99b0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c0039c0 0x7f999c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f999c0047c0 0x7f999c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f999c005180 0x7f999c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b006b810 con 0x7f999c005180 2023-10-16T12:34:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c00c080 con 0x7f999c0039c0 2023-10-16T12:34:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c00c240 con 0x7f999c0047c0 2023-10-16T12:34:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99b4d19640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f999c005180 0x7f999c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.697+0000 7f99affff640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c0039c0 0x7f999c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99a77fe640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f999c0047c0 0x7f999c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99b4d19640 1 -- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f999c0047c0 msgr2=0x7f999c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99b4d19640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f999c0047c0 0x7f999c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99b4d19640 1 -- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c0039c0 msgr2=0x7f999c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99b4d19640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c0039c0 0x7f999c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99b4d19640 1 -- 172.21.15.60:0/1663136244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c003520 con 0x7f999c005180 2023-10-16T12:34:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.698+0000 7f99affff640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c0039c0 0x7f999c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.699+0000 7f99b4d19640 1 -- 172.21.15.60:0/1663136244 reap_dead start 2023-10-16T12:34:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.699+0000 7f99b4d19640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f999c005180 0x7f999c00b980 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c0e0 tx=0x7f999c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564ab2792aed22bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.699+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8005ad0 con 0x7f999c005180 2023-10-16T12:34:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.699+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a8005c70 con 0x7f999c005180 2023-10-16T12:34:47.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.699+0000 7f99af7fe640 1 -- 172.21.15.60:0/1663136244 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a802c7d0 con 0x7f999c005180 2023-10-16T12:34:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 -- 172.21.15.60:0/1663136244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f999c005180 msgr2=0x7f999c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f999c005180 0x7f999c00b980 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c0e0 tx=0x7f999c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 -- 172.21.15.60:0/1663136244 shutdown_connections 2023-10-16T12:34:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 --2- 172.21.15.60:0/1663136244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f999c005180 0x7f999c00b980 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 -- 172.21.15.60:0/1663136244 >> 172.21.15.60:0/1663136244 conn(0x7f99b00cb410 msgr2=0x7f99b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 -- 172.21.15.60:0/1663136244 shutdown_connections 2023-10-16T12:34:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.700+0000 7f99b677c640 1 -- 172.21.15.60:0/1663136244 wait complete. 2023-10-16T12:34:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.703+0000 7f99b677c640 1 Processor -- start 2023-10-16T12:34:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.703+0000 7f99b677c640 1 -- start start 2023-10-16T12:34:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.704+0000 7f99b677c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c005180 0x7f99b015d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.704+0000 7f99b677c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99b0158d30 0x7f99b01498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.704+0000 7f99b677c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 0x7f99b0149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.704+0000 7f99b677c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015c410 con 0x7f99b01591b0 2023-10-16T12:34:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.704+0000 7f99b677c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015c680 con 0x7f999c005180 2023-10-16T12:34:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.704+0000 7f99b677c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015c960 con 0x7f99b0158d30 2023-10-16T12:34:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 0x7f99b0149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 0x7f99b0149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42156/0 (socket says 172.21.15.60:42156) 2023-10-16T12:34:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 -- 172.21.15.60:0/2031099009 learned_addr learned my addr 172.21.15.60:0/2031099009 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99affff640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c005180 0x7f99b015d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99a77fe640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99b0158d30 0x7f99b01498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 -- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99b0158d30 msgr2=0x7f99b01498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99b0158d30 0x7f99b01498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 -- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c005180 msgr2=0x7f99b015d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c005180 0x7f99b015d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99b4d19640 1 -- 172.21.15.60:0/2031099009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b015cc40 con 0x7f99b01591b0 2023-10-16T12:34:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.705+0000 7f99affff640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c005180 0x7f99b015d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.706+0000 7f99b4d19640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 0x7f99b0149df0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c0e0 tx=0x7f99b015f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.706+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8039050 con 0x7f99b01591b0 2023-10-16T12:34:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.706+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a803a040 con 0x7f99b01591b0 2023-10-16T12:34:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.706+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b014a330 con 0x7f99b01591b0 2023-10-16T12:34:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.706+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a80072b0 con 0x7f99b01591b0 2023-10-16T12:34:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.706+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b01ef890 con 0x7f99b01591b0 2023-10-16T12:34:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.707+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b004e160 con 0x7f99b01591b0 2023-10-16T12:34:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.708+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f99a8032070 con 0x7f99b01591b0 2023-10-16T12:34:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.709+0000 7f99ad7fa640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f999006ae70 0x7f999006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.709+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99a80aacf0 con 0x7f99b01591b0 2023-10-16T12:34:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.710+0000 7f99affff640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f999006ae70 0x7f999006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.711+0000 7f99affff640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f999006ae70 0x7f999006d330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99a0001d40 tx=0x7f99a0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.715+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99a8075d00 con 0x7f99b01591b0 2023-10-16T12:34:47.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.913+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99b00cdc60 con 0x7f99b01591b0 2023-10-16T12:34:47.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.916+0000 7f99ad7fa640 1 -- 172.21.15.60:0/2031099009 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f99a8046070 con 0x7f99b01591b0 2023-10-16T12:34:47.917 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:47.918 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:34:44.911174+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:34:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.918+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f999006ae70 msgr2=0x7f999006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.918+0000 7f99b677c640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f999006ae70 0x7f999006d330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99a0001d40 tx=0x7f99a0009040 comp rx=0 tx=0).stop 2023-10-16T12:34:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.918+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 msgr2=0x7f99b0149df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.918+0000 7f99b677c640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 0x7f99b0149df0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c0e0 tx=0x7f99b015f830 comp rx=0 tx=0).stop 2023-10-16T12:34:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 shutdown_connections 2023-10-16T12:34:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f999006ae70 0x7f999006d330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99b0158d30 0x7f99b01498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01591b0 0x7f99b0149df0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 --2- 172.21.15.60:0/2031099009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f999c005180 0x7f99b015d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 >> 172.21.15.60:0/2031099009 conn(0x7f99b00cb410 msgr2=0x7f99b006f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 shutdown_connections 2023-10-16T12:34:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:47.919+0000 7f99b677c640 1 -- 172.21.15.60:0/2031099009 wait complete. 2023-10-16T12:34:47.941 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:34:47.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:34:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.168+0000 7fb80e98e640 1 Processor -- start 2023-10-16T12:34:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.168+0000 7fb80e98e640 1 -- start start 2023-10-16T12:34:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80e98e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb80806fb50 0x7fb8080cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80e98e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080ca560 con 0x7fb80806c9e0 2023-10-16T12:34:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80e98e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080c9950 con 0x7fb8080c9060 2023-10-16T12:34:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80e98e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8080c9c30 con 0x7fb80806fb50 2023-10-16T12:34:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80e18d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb80806c9e0 0x7fb8080ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51442/0 (socket says 172.21.15.60:51442) 2023-10-16T12:34:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80e18d640 1 -- 172.21.15.60:0/3059447821 learned_addr learned my addr 172.21.15.60:0/3059447821 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.170+0000 7fb80d18b640 1 --2- 172.21.15.60:0/3059447821 >> v2:172.21.15.144:3300/0 conn(0x7fb80806fb50 0x7fb8080cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282833040 0 0) 0x7fb8080ca560 con 0x7fb80806c9e0 2023-10-16T12:34:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4003520 con 0x7fb80806c9e0 2023-10-16T12:34:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3769413061 0 0) 0x7fb7f4003520 con 0x7fb80806c9e0 2023-10-16T12:34:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 >> v2:172.21.15.144:3300/0 conn(0x7fb80806fb50 msgr2=0x7fb8080cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 --2- 172.21.15.60:0/3059447821 >> v2:172.21.15.144:3300/0 conn(0x7fb80806fb50 0x7fb8080cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 >> v1:172.21.15.144:6789/0 conn(0x7fb8080c9060 legacy=0x7fb80806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080c9f10 con 0x7fb80806c9e0 2023-10-16T12:34:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb800001670 con 0x7fb80806c9e0 2023-10-16T12:34:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 >> v1:172.21.15.60:6789/0 conn(0x7fb80806c9e0 legacy=0x7fb8080ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f4003970 0x7fb7f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f40047c0 0x7fb7f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7f4005180 0x7fb7f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080c9c30 con 0x7fb7f4005180 2023-10-16T12:34:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400c020 con 0x7fb7f4003970 2023-10-16T12:34:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.171+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400c1e0 con 0x7fb7f40047c0 2023-10-16T12:34:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7f4005180 0x7fb7f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80d98c640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f4003970 0x7fb7f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80e18d640 1 -- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f40047c0 msgr2=0x7fb7f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f40047c0 0x7fb7f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80e18d640 1 -- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f4003970 msgr2=0x7fb7f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f4003970 0x7fb7f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.172+0000 7fb80e18d640 1 -- 172.21.15.60:0/3059447821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4003520 con 0x7fb7f4005180 2023-10-16T12:34:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.173+0000 7fb80e18d640 1 -- 172.21.15.60:0/3059447821 reap_dead start 2023-10-16T12:34:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.173+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7f4005180 0x7fb7f400b980 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c210 tx=0x7fb7f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd33b298255878d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.173+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800005ad0 con 0x7fb7f4005180 2023-10-16T12:34:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.173+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800005c30 con 0x7fb7f4005180 2023-10-16T12:34:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.173+0000 7fb80c98a640 1 -- 172.21.15.60:0/3059447821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80002c890 con 0x7fb7f4005180 2023-10-16T12:34:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 -- 172.21.15.60:0/3059447821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7f4005180 msgr2=0x7fb7f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7f4005180 0x7fb7f400b980 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c210 tx=0x7fb7f40057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 -- 172.21.15.60:0/3059447821 shutdown_connections 2023-10-16T12:34:48.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3059447821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7f4005180 0x7fb7f400b980 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 -- 172.21.15.60:0/3059447821 >> 172.21.15.60:0/3059447821 conn(0x7fb8080ce6a0 msgr2=0x7fb8080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 -- 172.21.15.60:0/3059447821 shutdown_connections 2023-10-16T12:34:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.174+0000 7fb80e98e640 1 -- 172.21.15.60:0/3059447821 wait complete. 2023-10-16T12:34:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.176+0000 7fb80e98e640 1 Processor -- start 2023-10-16T12:34:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.176+0000 7fb80e98e640 1 -- start start 2023-10-16T12:34:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e98e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f4005180 0x7fb8081f9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e98e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb80815a5e0 0x7fb8080c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e98e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 0x7fb8081592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e98e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081fae80 con 0x7fb8080c5770 2023-10-16T12:34:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e98e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb808158e30 con 0x7fb80815a5e0 2023-10-16T12:34:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e98e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb808158820 con 0x7fb7f4005180 2023-10-16T12:34:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e18d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 0x7fb8081592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e18d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 0x7fb8081592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42172/0 (socket says 172.21.15.60:42172) 2023-10-16T12:34:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.177+0000 7fb80e18d640 1 -- 172.21.15.60:0/3083597955 learned_addr learned my addr 172.21.15.60:0/3083597955 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80d18b640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb80815a5e0 0x7fb8080c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80d98c640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f4005180 0x7fb8081f9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e18d640 1 -- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f4005180 msgr2=0x7fb8081f9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f4005180 0x7fb8081f9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e18d640 1 -- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb80815a5e0 msgr2=0x7fb8080c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb80815a5e0 0x7fb8080c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e18d640 1 -- 172.21.15.60:0/3083597955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8081fa8c0 con 0x7fb8080c5770 2023-10-16T12:34:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e18d640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 0x7fb8081592c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb800000c00 tx=0x7fb800001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000049e0 con 0x7fb8080c5770 2023-10-16T12:34:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800004b60 con 0x7fb8080c5770 2023-10-16T12:34:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8081faba0 con 0x7fb8080c5770 2023-10-16T12:34:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.178+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800007070 con 0x7fb8080c5770 2023-10-16T12:34:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.179+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8081fa490 con 0x7fb8080c5770 2023-10-16T12:34:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.180+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80804e9b0 con 0x7fb8080c5770 2023-10-16T12:34:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.181+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb800032070 con 0x7fb8080c5770 2023-10-16T12:34:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.182+0000 7fb7fe7fc640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb7ec06aea0 0x7fb7ec06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.182+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb80003c070 con 0x7fb8080c5770 2023-10-16T12:34:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.183+0000 7fb80d98c640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb7ec06aea0 0x7fb7ec06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.184+0000 7fb80d98c640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb7ec06aea0 0x7fb7ec06d360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb804007fd0 tx=0x7fb804009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.187+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8000ad030 con 0x7fb8080c5770 2023-10-16T12:34:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.382+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8080c1f10 con 0x7fb7ec06aea0 2023-10-16T12:34:48.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.411+0000 7fb7fe7fc640 1 -- 172.21.15.60:0/3083597955 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286631 (secure 0 0 0) 0x7fb804005740 con 0x7fb7ec06aea0 2023-10-16T12:34:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:34:48.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb7ec06aea0 msgr2=0x7fb7ec06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb7ec06aea0 0x7fb7ec06d360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb804007fd0 tx=0x7fb804009040 comp rx=0 tx=0).stop 2023-10-16T12:34:48.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 msgr2=0x7fb8081592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 0x7fb8081592c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb800000c00 tx=0x7fb800001a00 comp rx=0 tx=0).stop 2023-10-16T12:34:48.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 shutdown_connections 2023-10-16T12:34:48.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fb7ec06aea0 0x7fb7ec06d360 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8080c5770 0x7fb8081592c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb80815a5e0 0x7fb8080c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 --2- 172.21.15.60:0/3083597955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7f4005180 0x7fb8081f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 >> 172.21.15.60:0/3083597955 conn(0x7fb8080ce6a0 msgr2=0x7fb80806a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 shutdown_connections 2023-10-16T12:34:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.417+0000 7fb80e98e640 1 -- 172.21.15.60:0/3083597955 wait complete. 2023-10-16T12:34:48.438 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-10-16T12:34:47.759444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237204,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737516,"statfs":{"total":773094113280,"available":772851216384,"internally_reserved":0,"allocated":4497408,"data_stored":2465684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.893611"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949247+0000","last_change":"2023-10-16T12:34:43.930198+0000","last_active":"2023-10-16T12:34:44.949247+0000","last_peered":"2023-10-16T12:34:44.949247+0000","last_clean":"2023-10-16T12:34:44.949247+0000","last_became_active":"2023-10-16T12:34:43.929005+0000","last_became_peered":"2023-10-16T12:34:43.929005+0000","last_unstale":"2023-10-16T12:34:44.949247+0000","last_undegraded":"2023-10-16T12:34:44.949247+0000","last_fullsized":"2023-10-16T12:34:44.949247+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:49.302703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950274+0000","last_change":"2023-10-16T12:34:43.922962+0000","last_active":"2023-10-16T12:34:44.950274+0000","last_peered":"2023-10-16T12:34:44.950274+0000","last_clean":"2023-10-16T12:34:44.950274+0000","last_became_active":"2023-10-16T12:34:43.922670+0000","last_became_peered":"2023-10-16T12:34:43.922670+0000","last_unstale":"2023-10-16T12:34:44.950274+0000","last_undegraded":"2023-10-16T12:34:44.950274+0000","last_fullsized":"2023-10-16T12:34:44.950274+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921687+0000","last_change":"2023-10-16T12:34:43.935492+0000","last_active":"2023-10-16T12:34:44.921687+0000","last_peered":"2023-10-16T12:34:44.921687+0000","last_clean":"2023-10-16T12:34:44.921687+0000","last_became_active":"2023-10-16T12:34:43.935215+0000","last_became_peered":"2023-10-16T12:34:43.935215+0000","last_unstale":"2023-10-16T12:34:44.921687+0000","last_undegraded":"2023-10-16T12:34:44.921687+0000","last_fullsized":"2023-10-16T12:34:44.921687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.771260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917372+0000","last_change":"2023-10-16T12:34:43.938392+0000","last_active":"2023-10-16T12:34:44.917372+0000","last_peered":"2023-10-16T12:34:44.917372+0000","last_clean":"2023-10-16T12:34:44.917372+0000","last_became_active":"2023-10-16T12:34:43.937961+0000","last_became_peered":"2023-10-16T12:34:43.937961+0000","last_unstale":"2023-10-16T12:34:44.917372+0000","last_undegraded":"2023-10-16T12:34:44.917372+0000","last_fullsized":"2023-10-16T12:34:44.917372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:33.202722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948363+0000","last_change":"2023-10-16T12:34:43.950180+0000","last_active":"2023-10-16T12:34:44.948363+0000","last_peered":"2023-10-16T12:34:44.948363+0000","last_clean":"2023-10-16T12:34:44.948363+0000","last_became_active":"2023-10-16T12:34:43.949836+0000","last_became_peered":"2023-10-16T12:34:43.949836+0000","last_unstale":"2023-10-16T12:34:44.948363+0000","last_undegraded":"2023-10-16T12:34:44.948363+0000","last_fullsized":"2023-10-16T12:34:44.948363+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:38.679343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948115+0000","last_change":"2023-10-16T12:34:43.936331+0000","last_active":"2023-10-16T12:34:44.948115+0000","last_peered":"2023-10-16T12:34:44.948115+0000","last_clean":"2023-10-16T12:34:44.948115+0000","last_became_active":"2023-10-16T12:34:43.929952+0000","last_became_peered":"2023-10-16T12:34:43.929952+0000","last_unstale":"2023-10-16T12:34:44.948115+0000","last_undegraded":"2023-10-16T12:34:44.948115+0000","last_fullsized":"2023-10-16T12:34:44.948115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:28.037742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950228+0000","last_change":"2023-10-16T12:34:43.919371+0000","last_active":"2023-10-16T12:34:44.950228+0000","last_peered":"2023-10-16T12:34:44.950228+0000","last_clean":"2023-10-16T12:34:44.950228+0000","last_became_active":"2023-10-16T12:34:43.918990+0000","last_became_peered":"2023-10-16T12:34:43.918990+0000","last_unstale":"2023-10-16T12:34:44.950228+0000","last_undegraded":"2023-10-16T12:34:44.950228+0000","last_fullsized":"2023-10-16T12:34:44.950228+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.771260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949909+0000","last_change":"2023-10-16T12:34:43.928269+0000","last_active":"2023-10-16T12:34:44.949909+0000","last_peered":"2023-10-16T12:34:44.949909+0000","last_clean":"2023-10-16T12:34:44.949909+0000","last_became_active":"2023-10-16T12:34:43.927728+0000","last_became_peered":"2023-10-16T12:34:43.927728+0000","last_unstale":"2023-10-16T12:34:44.949909+0000","last_undegraded":"2023-10-16T12:34:44.949909+0000","last_fullsized":"2023-10-16T12:34:44.949909+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:38.692665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.924059+0000","last_change":"2023-10-16T12:34:43.937739+0000","last_active":"2023-10-16T12:34:44.924059+0000","last_peered":"2023-10-16T12:34:44.924059+0000","last_clean":"2023-10-16T12:34:44.924059+0000","last_became_active":"2023-10-16T12:34:43.937429+0000","last_became_peered":"2023-10-16T12:34:43.937429+0000","last_unstale":"2023-10-16T12:34:44.924059+0000","last_undegraded":"2023-10-16T12:34:44.924059+0000","last_fullsized":"2023-10-16T12:34:44.924059+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:22.197181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918033+0000","last_change":"2023-10-16T12:34:43.942762+0000","last_active":"2023-10-16T12:34:44.918033+0000","last_peered":"2023-10-16T12:34:44.918033+0000","last_clean":"2023-10-16T12:34:44.918033+0000","last_became_active":"2023-10-16T12:34:43.937861+0000","last_became_peered":"2023-10-16T12:34:43.937861+0000","last_unstale":"2023-10-16T12:34:44.918033+0000","last_undegraded":"2023-10-16T12:34:44.918033+0000","last_fullsized":"2023-10-16T12:34:44.918033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:37.757634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951128+0000","last_change":"2023-10-16T12:34:43.926827+0000","last_active":"2023-10-16T12:34:44.951128+0000","last_peered":"2023-10-16T12:34:44.951128+0000","last_clean":"2023-10-16T12:34:44.951128+0000","last_became_active":"2023-10-16T12:34:43.924813+0000","last_became_peered":"2023-10-16T12:34:43.924813+0000","last_unstale":"2023-10-16T12:34:44.951128+0000","last_undegraded":"2023-10-16T12:34:44.951128+0000","last_fullsized":"2023-10-16T12:34:44.951128+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:50.221503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921325+0000","last_change":"2023-10-16T12:34:43.941570+0000","last_active":"2023-10-16T12:34:44.921325+0000","last_peered":"2023-10-16T12:34:44.921325+0000","last_clean":"2023-10-16T12:34:44.921325+0000","last_became_active":"2023-10-16T12:34:43.941226+0000","last_became_peered":"2023-10-16T12:34:43.941226+0000","last_unstale":"2023-10-16T12:34:44.921325+0000","last_undegraded":"2023-10-16T12:34:44.921325+0000","last_fullsized":"2023-10-16T12:34:44.921325+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:32.221349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948744+0000","last_change":"2023-10-16T12:34:43.935981+0000","last_active":"2023-10-16T12:34:44.948744+0000","last_peered":"2023-10-16T12:34:44.948744+0000","last_clean":"2023-10-16T12:34:44.948744+0000","last_became_active":"2023-10-16T12:34:43.935634+0000","last_became_peered":"2023-10-16T12:34:43.935634+0000","last_unstale":"2023-10-16T12:34:44.948744+0000","last_undegraded":"2023-10-16T12:34:44.948744+0000","last_fullsized":"2023-10-16T12:34:44.948744+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:16.661136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949535+0000","last_change":"2023-10-16T12:34:43.936353+0000","last_active":"2023-10-16T12:34:44.949535+0000","last_peered":"2023-10-16T12:34:44.949535+0000","last_clean":"2023-10-16T12:34:44.949535+0000","last_became_active":"2023-10-16T12:34:43.927332+0000","last_became_peered":"2023-10-16T12:34:43.927332+0000","last_unstale":"2023-10-16T12:34:44.949535+0000","last_undegraded":"2023-10-16T12:34:44.949535+0000","last_fullsized":"2023-10-16T12:34:44.949535+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:48.267707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921283+0000","last_change":"2023-10-16T12:34:43.924656+0000","last_active":"2023-10-16T12:34:44.921283+0000","last_peered":"2023-10-16T12:34:44.921283+0000","last_clean":"2023-10-16T12:34:44.921283+0000","last_became_active":"2023-10-16T12:34:43.924330+0000","last_became_peered":"2023-10-16T12:34:43.924330+0000","last_unstale":"2023-10-16T12:34:44.921283+0000","last_undegraded":"2023-10-16T12:34:44.921283+0000","last_fullsized":"2023-10-16T12:34:44.921283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:38.692665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950515+0000","last_change":"2023-10-16T12:34:43.922370+0000","last_active":"2023-10-16T12:34:44.950515+0000","last_peered":"2023-10-16T12:34:44.950515+0000","last_clean":"2023-10-16T12:34:44.950515+0000","last_became_active":"2023-10-16T12:34:43.922085+0000","last_became_peered":"2023-10-16T12:34:43.922085+0000","last_unstale":"2023-10-16T12:34:44.950515+0000","last_undegraded":"2023-10-16T12:34:44.950515+0000","last_fullsized":"2023-10-16T12:34:44.950515+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:02.974058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951273+0000","last_change":"2023-10-16T12:34:43.911722+0000","last_active":"2023-10-16T12:34:44.951273+0000","last_peered":"2023-10-16T12:34:44.951273+0000","last_clean":"2023-10-16T12:34:44.951273+0000","last_became_active":"2023-10-16T12:34:43.911121+0000","last_became_peered":"2023-10-16T12:34:43.911121+0000","last_unstale":"2023-10-16T12:34:44.951273+0000","last_undegraded":"2023-10-16T12:34:44.951273+0000","last_fullsized":"2023-10-16T12:34:44.951273+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:56.298567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950414+0000","last_change":"2023-10-16T12:34:43.926887+0000","last_active":"2023-10-16T12:34:44.950414+0000","last_peered":"2023-10-16T12:34:44.950414+0000","last_clean":"2023-10-16T12:34:44.950414+0000","last_became_active":"2023-10-16T12:34:43.926593+0000","last_became_peered":"2023-10-16T12:34:43.926593+0000","last_unstale":"2023-10-16T12:34:44.950414+0000","last_undegraded":"2023-10-16T12:34:44.950414+0000","last_fullsized":"2023-10-16T12:34:44.950414+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:50.221503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950563+0000","last_change":"2023-10-16T12:34:43.932550+0000","last_active":"2023-10-16T12:34:44.950563+0000","last_peered":"2023-10-16T12:34:44.950563+0000","last_clean":"2023-10-16T12:34:44.950563+0000","last_became_active":"2023-10-16T12:34:43.927044+0000","last_became_peered":"2023-10-16T12:34:43.927044+0000","last_unstale":"2023-10-16T12:34:44.950563+0000","last_undegraded":"2023-10-16T12:34:44.950563+0000","last_fullsized":"2023-10-16T12:34:44.950563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:56.298567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948918+0000","last_change":"2023-10-16T12:34:43.929021+0000","last_active":"2023-10-16T12:34:44.948918+0000","last_peered":"2023-10-16T12:34:44.948918+0000","last_clean":"2023-10-16T12:34:44.948918+0000","last_became_active":"2023-10-16T12:34:43.928706+0000","last_became_peered":"2023-10-16T12:34:43.928706+0000","last_unstale":"2023-10-16T12:34:44.948918+0000","last_undegraded":"2023-10-16T12:34:44.948918+0000","last_fullsized":"2023-10-16T12:34:44.948918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:44.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922989+0000","last_change":"2023-10-16T12:34:43.932210+0000","last_active":"2023-10-16T12:34:44.922989+0000","last_peered":"2023-10-16T12:34:44.922989+0000","last_clean":"2023-10-16T12:34:44.922989+0000","last_became_active":"2023-10-16T12:34:43.931913+0000","last_became_peered":"2023-10-16T12:34:43.931913+0000","last_unstale":"2023-10-16T12:34:44.922989+0000","last_undegraded":"2023-10-16T12:34:44.922989+0000","last_fullsized":"2023-10-16T12:34:44.922989+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:50.157019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923036+0000","last_change":"2023-10-16T12:34:43.928342+0000","last_active":"2023-10-16T12:34:44.923036+0000","last_peered":"2023-10-16T12:34:44.923036+0000","last_clean":"2023-10-16T12:34:44.923036+0000","last_became_active":"2023-10-16T12:34:43.928067+0000","last_became_peered":"2023-10-16T12:34:43.928067+0000","last_unstale":"2023-10-16T12:34:44.923036+0000","last_undegraded":"2023-10-16T12:34:44.923036+0000","last_fullsized":"2023-10-16T12:34:44.923036+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:39.725825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950688+0000","last_change":"2023-10-16T12:34:43.928954+0000","last_active":"2023-10-16T12:34:44.950688+0000","last_peered":"2023-10-16T12:34:44.950688+0000","last_clean":"2023-10-16T12:34:44.950688+0000","last_became_active":"2023-10-16T12:34:43.928684+0000","last_became_peered":"2023-10-16T12:34:43.928684+0000","last_unstale":"2023-10-16T12:34:44.950688+0000","last_undegraded":"2023-10-16T12:34:44.950688+0000","last_fullsized":"2023-10-16T12:34:44.950688+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:07.757544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.920774+0000","last_change":"2023-10-16T12:34:43.926337+0000","last_active":"2023-10-16T12:34:44.920774+0000","last_peered":"2023-10-16T12:34:44.920774+0000","last_clean":"2023-10-16T12:34:44.920774+0000","last_became_active":"2023-10-16T12:34:43.926043+0000","last_became_peered":"2023-10-16T12:34:43.926043+0000","last_unstale":"2023-10-16T12:34:44.920774+0000","last_undegraded":"2023-10-16T12:34:44.920774+0000","last_fullsized":"2023-10-16T12:34:44.920774+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:03.305753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918020+0000","last_change":"2023-10-16T12:34:43.924982+0000","last_active":"2023-10-16T12:34:44.918020+0000","last_peered":"2023-10-16T12:34:44.918020+0000","last_clean":"2023-10-16T12:34:44.918020+0000","last_became_active":"2023-10-16T12:34:43.924701+0000","last_became_peered":"2023-10-16T12:34:43.924701+0000","last_unstale":"2023-10-16T12:34:44.918020+0000","last_undegraded":"2023-10-16T12:34:44.918020+0000","last_fullsized":"2023-10-16T12:34:44.918020+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:22.197181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950561+0000","last_change":"2023-10-16T12:34:43.915881+0000","last_active":"2023-10-16T12:34:44.950561+0000","last_peered":"2023-10-16T12:34:44.950561+0000","last_clean":"2023-10-16T12:34:44.950561+0000","last_became_active":"2023-10-16T12:34:43.914247+0000","last_became_peered":"2023-10-16T12:34:43.914247+0000","last_unstale":"2023-10-16T12:34:44.950561+0000","last_undegraded":"2023-10-16T12:34:44.950561+0000","last_fullsized":"2023-10-16T12:34:44.950561+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:55.889498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918052+0000","last_change":"2023-10-16T12:34:43.934418+0000","last_active":"2023-10-16T12:34:44.918052+0000","last_peered":"2023-10-16T12:34:44.918052+0000","last_clean":"2023-10-16T12:34:44.918052+0000","last_became_active":"2023-10-16T12:34:43.934119+0000","last_became_peered":"2023-10-16T12:34:43.934119+0000","last_unstale":"2023-10-16T12:34:44.918052+0000","last_undegraded":"2023-10-16T12:34:44.918052+0000","last_fullsized":"2023-10-16T12:34:44.918052+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:24:53.328597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922433+0000","last_change":"2023-10-16T12:34:43.929167+0000","last_active":"2023-10-16T12:34:44.922433+0000","last_peered":"2023-10-16T12:34:44.922433+0000","last_clean":"2023-10-16T12:34:44.922433+0000","last_became_active":"2023-10-16T12:34:43.928892+0000","last_became_peered":"2023-10-16T12:34:43.928892+0000","last_unstale":"2023-10-16T12:34:44.922433+0000","last_undegraded":"2023-10-16T12:34:44.922433+0000","last_fullsized":"2023-10-16T12:34:44.922433+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:48.267707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923038+0000","last_change":"2023-10-16T12:34:43.928710+0000","last_active":"2023-10-16T12:34:44.923038+0000","last_peered":"2023-10-16T12:34:44.923038+0000","last_clean":"2023-10-16T12:34:44.923038+0000","last_became_active":"2023-10-16T12:34:43.928375+0000","last_became_peered":"2023-10-16T12:34:43.928375+0000","last_unstale":"2023-10-16T12:34:44.923038+0000","last_undegraded":"2023-10-16T12:34:44.923038+0000","last_fullsized":"2023-10-16T12:34:44.923038+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:51.950513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949937+0000","last_change":"2023-10-16T12:34:43.912431+0000","last_active":"2023-10-16T12:34:44.949937+0000","last_peered":"2023-10-16T12:34:44.949937+0000","last_clean":"2023-10-16T12:34:44.949937+0000","last_became_active":"2023-10-16T12:34:43.911980+0000","last_became_peered":"2023-10-16T12:34:43.911980+0000","last_unstale":"2023-10-16T12:34:44.949937+0000","last_undegraded":"2023-10-16T12:34:44.949937+0000","last_fullsized":"2023-10-16T12:34:44.949937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:32.221349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948749+0000","last_change":"2023-10-16T12:34:43.926999+0000","last_active":"2023-10-16T12:34:44.948749+0000","last_peered":"2023-10-16T12:34:44.948749+0000","last_clean":"2023-10-16T12:34:44.948749+0000","last_became_active":"2023-10-16T12:34:43.926644+0000","last_became_peered":"2023-10-16T12:34:43.926644+0000","last_unstale":"2023-10-16T12:34:44.948749+0000","last_undegraded":"2023-10-16T12:34:44.948749+0000","last_fullsized":"2023-10-16T12:34:44.948749+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:24.393948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949938+0000","last_change":"2023-10-16T12:34:43.933027+0000","last_active":"2023-10-16T12:34:44.949938+0000","last_peered":"2023-10-16T12:34:44.949938+0000","last_clean":"2023-10-16T12:34:44.949938+0000","last_became_active":"2023-10-16T12:34:43.932717+0000","last_became_peered":"2023-10-16T12:34:43.932717+0000","last_unstale":"2023-10-16T12:34:44.949938+0000","last_undegraded":"2023-10-16T12:34:44.949938+0000","last_fullsized":"2023-10-16T12:34:44.949938+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:11.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952142+0000","last_change":"2023-10-16T12:34:43.932251+0000","last_active":"2023-10-16T12:34:44.952142+0000","last_peered":"2023-10-16T12:34:44.952142+0000","last_clean":"2023-10-16T12:34:44.952142+0000","last_became_active":"2023-10-16T12:34:43.925031+0000","last_became_peered":"2023-10-16T12:34:43.925031+0000","last_unstale":"2023-10-16T12:34:44.952142+0000","last_undegraded":"2023-10-16T12:34:44.952142+0000","last_fullsized":"2023-10-16T12:34:44.952142+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:51.950513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949324+0000","last_change":"2023-10-16T12:34:42.913313+0000","last_active":"2023-10-16T12:34:44.949324+0000","last_peered":"2023-10-16T12:34:44.949324+0000","last_clean":"2023-10-16T12:34:44.949324+0000","last_became_active":"2023-10-16T12:34:42.912917+0000","last_became_peered":"2023-10-16T12:34:42.912917+0000","last_unstale":"2023-10-16T12:34:44.949324+0000","last_undegraded":"2023-10-16T12:34:44.949324+0000","last_fullsized":"2023-10-16T12:34:44.949324+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:50.353010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949197+0000","last_change":"2023-10-16T12:34:43.932393+0000","last_active":"2023-10-16T12:34:44.949197+0000","last_peered":"2023-10-16T12:34:44.949197+0000","last_clean":"2023-10-16T12:34:44.949197+0000","last_became_active":"2023-10-16T12:34:43.932130+0000","last_became_peered":"2023-10-16T12:34:43.932130+0000","last_unstale":"2023-10-16T12:34:44.949197+0000","last_undegraded":"2023-10-16T12:34:44.949197+0000","last_fullsized":"2023-10-16T12:34:44.949197+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:37.700253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952008+0000","last_change":"2023-10-16T12:34:42.904347+0000","last_active":"2023-10-16T12:34:44.952008+0000","last_peered":"2023-10-16T12:34:44.952008+0000","last_clean":"2023-10-16T12:34:44.952008+0000","last_became_active":"2023-10-16T12:34:42.904071+0000","last_became_peered":"2023-10-16T12:34:42.904071+0000","last_unstale":"2023-10-16T12:34:44.952008+0000","last_undegraded":"2023-10-16T12:34:44.952008+0000","last_fullsized":"2023-10-16T12:34:44.952008+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:34.175108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923392+0000","last_change":"2023-10-16T12:34:43.947491+0000","last_active":"2023-10-16T12:34:44.923392+0000","last_peered":"2023-10-16T12:34:44.923392+0000","last_clean":"2023-10-16T12:34:44.923392+0000","last_became_active":"2023-10-16T12:34:43.947245+0000","last_became_peered":"2023-10-16T12:34:43.947245+0000","last_unstale":"2023-10-16T12:34:44.923392+0000","last_undegraded":"2023-10-16T12:34:44.923392+0000","last_fullsized":"2023-10-16T12:34:44.923392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:33.202722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950866+0000","last_change":"2023-10-16T12:34:42.901921+0000","last_active":"2023-10-16T12:34:44.950866+0000","last_peered":"2023-10-16T12:34:44.950866+0000","last_clean":"2023-10-16T12:34:44.950866+0000","last_became_active":"2023-10-16T12:34:42.901601+0000","last_became_peered":"2023-10-16T12:34:42.901601+0000","last_unstale":"2023-10-16T12:34:44.950866+0000","last_undegraded":"2023-10-16T12:34:44.950866+0000","last_fullsized":"2023-10-16T12:34:44.950866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:31.368211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918126+0000","last_change":"2023-10-16T12:34:43.923262+0000","last_active":"2023-10-16T12:34:44.918126+0000","last_peered":"2023-10-16T12:34:44.918126+0000","last_clean":"2023-10-16T12:34:44.918126+0000","last_became_active":"2023-10-16T12:34:43.922060+0000","last_became_peered":"2023-10-16T12:34:43.922060+0000","last_unstale":"2023-10-16T12:34:44.918126+0000","last_undegraded":"2023-10-16T12:34:44.918126+0000","last_fullsized":"2023-10-16T12:34:44.918126+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:23.676757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948904+0000","last_change":"2023-10-16T12:34:42.902554+0000","last_active":"2023-10-16T12:34:44.948904+0000","last_peered":"2023-10-16T12:34:44.948904+0000","last_clean":"2023-10-16T12:34:44.948904+0000","last_became_active":"2023-10-16T12:34:42.899186+0000","last_became_peered":"2023-10-16T12:34:42.899186+0000","last_unstale":"2023-10-16T12:34:44.948904+0000","last_undegraded":"2023-10-16T12:34:44.948904+0000","last_fullsized":"2023-10-16T12:34:44.948904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:32.668405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950357+0000","last_change":"2023-10-16T12:34:43.931639+0000","last_active":"2023-10-16T12:34:44.950357+0000","last_peered":"2023-10-16T12:34:44.950357+0000","last_clean":"2023-10-16T12:34:44.950357+0000","last_became_active":"2023-10-16T12:34:43.930333+0000","last_became_peered":"2023-10-16T12:34:43.930333+0000","last_unstale":"2023-10-16T12:34:44.950357+0000","last_undegraded":"2023-10-16T12:34:44.950357+0000","last_fullsized":"2023-10-16T12:34:44.950357+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:48.267707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923183+0000","last_change":"2023-10-16T12:34:42.904493+0000","last_active":"2023-10-16T12:34:44.923183+0000","last_peered":"2023-10-16T12:34:44.923183+0000","last_clean":"2023-10-16T12:34:44.923183+0000","last_became_active":"2023-10-16T12:34:42.904195+0000","last_became_peered":"2023-10-16T12:34:42.904195+0000","last_unstale":"2023-10-16T12:34:44.923183+0000","last_undegraded":"2023-10-16T12:34:44.923183+0000","last_fullsized":"2023-10-16T12:34:44.923183+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:56.384420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921398+0000","last_change":"2023-10-16T12:34:43.929647+0000","last_active":"2023-10-16T12:34:44.921398+0000","last_peered":"2023-10-16T12:34:44.921398+0000","last_clean":"2023-10-16T12:34:44.921398+0000","last_became_active":"2023-10-16T12:34:43.928770+0000","last_became_peered":"2023-10-16T12:34:43.928770+0000","last_unstale":"2023-10-16T12:34:44.921398+0000","last_undegraded":"2023-10-16T12:34:44.921398+0000","last_fullsized":"2023-10-16T12:34:44.921398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:07.978483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922949+0000","last_change":"2023-10-16T12:34:42.902944+0000","last_active":"2023-10-16T12:34:44.922949+0000","last_peered":"2023-10-16T12:34:44.922949+0000","last_clean":"2023-10-16T12:34:44.922949+0000","last_became_active":"2023-10-16T12:34:42.902580+0000","last_became_peered":"2023-10-16T12:34:42.902580+0000","last_unstale":"2023-10-16T12:34:44.922949+0000","last_undegraded":"2023-10-16T12:34:44.922949+0000","last_fullsized":"2023-10-16T12:34:44.922949+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:35.173303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921666+0000","last_change":"2023-10-16T12:34:43.934012+0000","last_active":"2023-10-16T12:34:44.921666+0000","last_peered":"2023-10-16T12:34:44.921666+0000","last_clean":"2023-10-16T12:34:44.921666+0000","last_became_active":"2023-10-16T12:34:43.933748+0000","last_became_peered":"2023-10-16T12:34:43.933748+0000","last_unstale":"2023-10-16T12:34:44.921666+0000","last_undegraded":"2023-10-16T12:34:44.921666+0000","last_fullsized":"2023-10-16T12:34:44.921666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:42.104379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949197+0000","last_change":"2023-10-16T12:34:42.899296+0000","last_active":"2023-10-16T12:34:44.949197+0000","last_peered":"2023-10-16T12:34:44.949197+0000","last_clean":"2023-10-16T12:34:44.949197+0000","last_became_active":"2023-10-16T12:34:42.898994+0000","last_became_peered":"2023-10-16T12:34:42.898994+0000","last_unstale":"2023-10-16T12:34:44.949197+0000","last_undegraded":"2023-10-16T12:34:44.949197+0000","last_fullsized":"2023-10-16T12:34:44.949197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:24:52.314462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.916959+0000","last_change":"2023-10-16T12:34:43.923159+0000","last_active":"2023-10-16T12:34:44.916959+0000","last_peered":"2023-10-16T12:34:44.916959+0000","last_clean":"2023-10-16T12:34:44.916959+0000","last_became_active":"2023-10-16T12:34:43.922129+0000","last_became_peered":"2023-10-16T12:34:43.922129+0000","last_unstale":"2023-10-16T12:34:44.916959+0000","last_undegraded":"2023-10-16T12:34:44.916959+0000","last_fullsized":"2023-10-16T12:34:44.916959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:39.725825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921879+0000","last_change":"2023-10-16T12:34:42.911662+0000","last_active":"2023-10-16T12:34:44.921879+0000","last_peered":"2023-10-16T12:34:44.921879+0000","last_clean":"2023-10-16T12:34:44.921879+0000","last_became_active":"2023-10-16T12:34:42.910293+0000","last_became_peered":"2023-10-16T12:34:42.910293+0000","last_unstale":"2023-10-16T12:34:44.921879+0000","last_undegraded":"2023-10-16T12:34:44.921879+0000","last_fullsized":"2023-10-16T12:34:44.921879+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:43.802570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922239+0000","last_change":"2023-10-16T12:34:43.941648+0000","last_active":"2023-10-16T12:34:44.922239+0000","last_peered":"2023-10-16T12:34:44.922239+0000","last_clean":"2023-10-16T12:34:44.922239+0000","last_became_active":"2023-10-16T12:34:43.941213+0000","last_became_peered":"2023-10-16T12:34:43.941213+0000","last_unstale":"2023-10-16T12:34:44.922239+0000","last_undegraded":"2023-10-16T12:34:44.922239+0000","last_fullsized":"2023-10-16T12:34:44.922239+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:28.037742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950952+0000","last_change":"2023-10-16T12:34:42.907472+0000","last_active":"2023-10-16T12:34:44.950952+0000","last_peered":"2023-10-16T12:34:44.950952+0000","last_clean":"2023-10-16T12:34:44.950952+0000","last_became_active":"2023-10-16T12:34:42.905716+0000","last_became_peered":"2023-10-16T12:34:42.905716+0000","last_unstale":"2023-10-16T12:34:44.950952+0000","last_undegraded":"2023-10-16T12:34:44.950952+0000","last_fullsized":"2023-10-16T12:34:44.950952+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:02.291618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950862+0000","last_change":"2023-10-16T12:34:43.926749+0000","last_active":"2023-10-16T12:34:44.950862+0000","last_peered":"2023-10-16T12:34:44.950862+0000","last_clean":"2023-10-16T12:34:44.950862+0000","last_became_active":"2023-10-16T12:34:43.926508+0000","last_became_peered":"2023-10-16T12:34:43.926508+0000","last_unstale":"2023-10-16T12:34:44.950862+0000","last_undegraded":"2023-10-16T12:34:44.950862+0000","last_fullsized":"2023-10-16T12:34:44.950862+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:59.555128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952098+0000","last_change":"2023-10-16T12:34:42.900177+0000","last_active":"2023-10-16T12:34:44.952098+0000","last_peered":"2023-10-16T12:34:44.952098+0000","last_clean":"2023-10-16T12:34:44.952098+0000","last_became_active":"2023-10-16T12:34:42.899888+0000","last_became_peered":"2023-10-16T12:34:42.899888+0000","last_unstale":"2023-10-16T12:34:44.952098+0000","last_undegraded":"2023-10-16T12:34:44.952098+0000","last_fullsized":"2023-10-16T12:34:44.952098+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.358016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951091+0000","last_change":"2023-10-16T12:34:43.912161+0000","last_active":"2023-10-16T12:34:44.951091+0000","last_peered":"2023-10-16T12:34:44.951091+0000","last_clean":"2023-10-16T12:34:44.951091+0000","last_became_active":"2023-10-16T12:34:43.911383+0000","last_became_peered":"2023-10-16T12:34:43.911383+0000","last_unstale":"2023-10-16T12:34:44.951091+0000","last_undegraded":"2023-10-16T12:34:44.951091+0000","last_fullsized":"2023-10-16T12:34:44.951091+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:10.834194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921759+0000","last_change":"2023-10-16T12:34:42.909164+0000","last_active":"2023-10-16T12:34:44.921759+0000","last_peered":"2023-10-16T12:34:44.921759+0000","last_clean":"2023-10-16T12:34:44.921759+0000","last_became_active":"2023-10-16T12:34:42.908234+0000","last_became_peered":"2023-10-16T12:34:42.908234+0000","last_unstale":"2023-10-16T12:34:44.921759+0000","last_undegraded":"2023-10-16T12:34:44.921759+0000","last_fullsized":"2023-10-16T12:34:44.921759+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:27.563110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921303+0000","last_change":"2023-10-16T12:34:43.935231+0000","last_active":"2023-10-16T12:34:44.921303+0000","last_peered":"2023-10-16T12:34:44.921303+0000","last_clean":"2023-10-16T12:34:44.921303+0000","last_became_active":"2023-10-16T12:34:43.934831+0000","last_became_peered":"2023-10-16T12:34:43.934831+0000","last_unstale":"2023-10-16T12:34:44.921303+0000","last_undegraded":"2023-10-16T12:34:44.921303+0000","last_fullsized":"2023-10-16T12:34:44.921303+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:11.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":18,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951152+0000","last_change":"2023-10-16T12:34:42.901772+0000","last_active":"2023-10-16T12:34:44.951152+0000","last_peered":"2023-10-16T12:34:44.951152+0000","last_clean":"2023-10-16T12:34:44.951152+0000","last_became_active":"2023-10-16T12:34:42.901477+0000","last_became_peered":"2023-10-16T12:34:42.901477+0000","last_unstale":"2023-10-16T12:34:44.951152+0000","last_undegraded":"2023-10-16T12:34:44.951152+0000","last_fullsized":"2023-10-16T12:34:44.951152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:36.743499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.916931+0000","last_change":"2023-10-16T12:34:43.933805+0000","last_active":"2023-10-16T12:34:44.916931+0000","last_peered":"2023-10-16T12:34:44.916931+0000","last_clean":"2023-10-16T12:34:44.916931+0000","last_became_active":"2023-10-16T12:34:43.926694+0000","last_became_peered":"2023-10-16T12:34:43.926694+0000","last_unstale":"2023-10-16T12:34:44.916931+0000","last_undegraded":"2023-10-16T12:34:44.916931+0000","last_fullsized":"2023-10-16T12:34:44.916931+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:51.950513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923258+0000","last_change":"2023-10-16T12:34:42.911025+0000","last_active":"2023-10-16T12:34:44.923258+0000","last_peered":"2023-10-16T12:34:44.923258+0000","last_clean":"2023-10-16T12:34:44.923258+0000","last_became_active":"2023-10-16T12:34:42.910753+0000","last_became_peered":"2023-10-16T12:34:42.910753+0000","last_unstale":"2023-10-16T12:34:44.923258+0000","last_undegraded":"2023-10-16T12:34:44.923258+0000","last_fullsized":"2023-10-16T12:34:44.923258+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:34.175108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950139+0000","last_change":"2023-10-16T12:34:43.920622+0000","last_active":"2023-10-16T12:34:44.950139+0000","last_peered":"2023-10-16T12:34:44.950139+0000","last_clean":"2023-10-16T12:34:44.950139+0000","last_became_active":"2023-10-16T12:34:43.920327+0000","last_became_peered":"2023-10-16T12:34:43.920327+0000","last_unstale":"2023-10-16T12:34:44.950139+0000","last_undegraded":"2023-10-16T12:34:44.950139+0000","last_fullsized":"2023-10-16T12:34:44.950139+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:02.974058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917040+0000","last_change":"2023-10-16T12:34:42.910360+0000","last_active":"2023-10-16T12:34:44.917040+0000","last_peered":"2023-10-16T12:34:44.917040+0000","last_clean":"2023-10-16T12:34:44.917040+0000","last_became_active":"2023-10-16T12:34:42.910010+0000","last_became_peered":"2023-10-16T12:34:42.910010+0000","last_unstale":"2023-10-16T12:34:44.917040+0000","last_undegraded":"2023-10-16T12:34:44.917040+0000","last_fullsized":"2023-10-16T12:34:44.917040+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:41.090244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950279+0000","last_change":"2023-10-16T12:34:43.932414+0000","last_active":"2023-10-16T12:34:44.950279+0000","last_peered":"2023-10-16T12:34:44.950279+0000","last_clean":"2023-10-16T12:34:44.950279+0000","last_became_active":"2023-10-16T12:34:43.932117+0000","last_became_peered":"2023-10-16T12:34:43.932117+0000","last_unstale":"2023-10-16T12:34:44.950279+0000","last_undegraded":"2023-10-16T12:34:44.950279+0000","last_fullsized":"2023-10-16T12:34:44.950279+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:28.037742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949294+0000","last_change":"2023-10-16T12:34:42.902621+0000","last_active":"2023-10-16T12:34:44.949294+0000","last_peered":"2023-10-16T12:34:44.949294+0000","last_clean":"2023-10-16T12:34:44.949294+0000","last_became_active":"2023-10-16T12:34:42.902351+0000","last_became_peered":"2023-10-16T12:34:42.902351+0000","last_unstale":"2023-10-16T12:34:44.949294+0000","last_undegraded":"2023-10-16T12:34:44.949294+0000","last_fullsized":"2023-10-16T12:34:44.949294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:06.743408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948888+0000","last_change":"2023-10-16T12:34:43.927554+0000","last_active":"2023-10-16T12:34:44.948888+0000","last_peered":"2023-10-16T12:34:44.948888+0000","last_clean":"2023-10-16T12:34:44.948888+0000","last_became_active":"2023-10-16T12:34:43.927274+0000","last_became_peered":"2023-10-16T12:34:43.927274+0000","last_unstale":"2023-10-16T12:34:44.948888+0000","last_undegraded":"2023-10-16T12:34:44.948888+0000","last_fullsized":"2023-10-16T12:34:44.948888+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:00.908601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.947794+0000","last_change":"2023-10-16T12:34:42.900613+0000","last_active":"2023-10-16T12:34:44.947794+0000","last_peered":"2023-10-16T12:34:44.947794+0000","last_clean":"2023-10-16T12:34:44.947794+0000","last_became_active":"2023-10-16T12:34:42.900350+0000","last_became_peered":"2023-10-16T12:34:42.900350+0000","last_unstale":"2023-10-16T12:34:44.947794+0000","last_undegraded":"2023-10-16T12:34:44.947794+0000","last_fullsized":"2023-10-16T12:34:44.947794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:21.183046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923905+0000","last_change":"2023-10-16T12:34:43.931499+0000","last_active":"2023-10-16T12:34:44.923905+0000","last_peered":"2023-10-16T12:34:44.923905+0000","last_clean":"2023-10-16T12:34:44.923905+0000","last_became_active":"2023-10-16T12:34:43.931226+0000","last_became_peered":"2023-10-16T12:34:43.931226+0000","last_unstale":"2023-10-16T12:34:44.923905+0000","last_undegraded":"2023-10-16T12:34:44.923905+0000","last_fullsized":"2023-10-16T12:34:44.923905+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:32.382347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918113+0000","last_change":"2023-10-16T12:34:42.905547+0000","last_active":"2023-10-16T12:34:44.918113+0000","last_peered":"2023-10-16T12:34:44.918113+0000","last_clean":"2023-10-16T12:34:44.918113+0000","last_became_active":"2023-10-16T12:34:42.905234+0000","last_became_peered":"2023-10-16T12:34:42.905234+0000","last_unstale":"2023-10-16T12:34:44.918113+0000","last_undegraded":"2023-10-16T12:34:44.918113+0000","last_fullsized":"2023-10-16T12:34:44.918113+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:38.379882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922484+0000","last_change":"2023-10-16T12:34:43.932591+0000","last_active":"2023-10-16T12:34:44.922484+0000","last_peered":"2023-10-16T12:34:44.922484+0000","last_clean":"2023-10-16T12:34:44.922484+0000","last_became_active":"2023-10-16T12:34:43.931829+0000","last_became_peered":"2023-10-16T12:34:43.931829+0000","last_unstale":"2023-10-16T12:34:44.922484+0000","last_undegraded":"2023-10-16T12:34:44.922484+0000","last_fullsized":"2023-10-16T12:34:44.922484+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:02.974058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923832+0000","last_change":"2023-10-16T12:34:42.910945+0000","last_active":"2023-10-16T12:34:44.923832+0000","last_peered":"2023-10-16T12:34:44.923832+0000","last_clean":"2023-10-16T12:34:44.923832+0000","last_became_active":"2023-10-16T12:34:42.910648+0000","last_became_peered":"2023-10-16T12:34:42.910648+0000","last_unstale":"2023-10-16T12:34:44.923832+0000","last_undegraded":"2023-10-16T12:34:44.923832+0000","last_fullsized":"2023-10-16T12:34:44.923832+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:46:35.902322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917513+0000","last_change":"2023-10-16T12:34:43.938237+0000","last_active":"2023-10-16T12:34:44.917513+0000","last_peered":"2023-10-16T12:34:44.917513+0000","last_clean":"2023-10-16T12:34:44.917513+0000","last_became_active":"2023-10-16T12:34:43.937101+0000","last_became_peered":"2023-10-16T12:34:43.937101+0000","last_unstale":"2023-10-16T12:34:44.917513+0000","last_undegraded":"2023-10-16T12:34:44.917513+0000","last_fullsized":"2023-10-16T12:34:44.917513+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:32.382347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923926+0000","last_change":"2023-10-16T12:34:42.900482+0000","last_active":"2023-10-16T12:34:44.923926+0000","last_peered":"2023-10-16T12:34:44.923926+0000","last_clean":"2023-10-16T12:34:44.923926+0000","last_became_active":"2023-10-16T12:34:42.900088+0000","last_became_peered":"2023-10-16T12:34:42.900088+0000","last_unstale":"2023-10-16T12:34:44.923926+0000","last_undegraded":"2023-10-16T12:34:44.923926+0000","last_fullsized":"2023-10-16T12:34:44.923926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.358016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948552+0000","last_change":"2023-10-16T12:34:43.929644+0000","last_active":"2023-10-16T12:34:44.948552+0000","last_peered":"2023-10-16T12:34:44.948552+0000","last_clean":"2023-10-16T12:34:44.948552+0000","last_became_active":"2023-10-16T12:34:43.929346+0000","last_became_peered":"2023-10-16T12:34:43.929346+0000","last_unstale":"2023-10-16T12:34:44.948552+0000","last_undegraded":"2023-10-16T12:34:44.948552+0000","last_fullsized":"2023-10-16T12:34:44.948552+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:01:23.152173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923405+0000","last_change":"2023-10-16T12:34:42.908083+0000","last_active":"2023-10-16T12:34:44.923405+0000","last_peered":"2023-10-16T12:34:44.923405+0000","last_clean":"2023-10-16T12:34:44.923405+0000","last_became_active":"2023-10-16T12:34:42.907812+0000","last_became_peered":"2023-10-16T12:34:42.907812+0000","last_unstale":"2023-10-16T12:34:44.923405+0000","last_undegraded":"2023-10-16T12:34:44.923405+0000","last_fullsized":"2023-10-16T12:34:44.923405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:51.182438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948283+0000","last_change":"2023-10-16T12:34:43.931488+0000","last_active":"2023-10-16T12:34:44.948283+0000","last_peered":"2023-10-16T12:34:44.948283+0000","last_clean":"2023-10-16T12:34:44.948283+0000","last_became_active":"2023-10-16T12:34:43.931129+0000","last_became_peered":"2023-10-16T12:34:43.931129+0000","last_unstale":"2023-10-16T12:34:44.948283+0000","last_undegraded":"2023-10-16T12:34:44.948283+0000","last_fullsized":"2023-10-16T12:34:44.948283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:25.750031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951513+0000","last_change":"2023-10-16T12:34:42.897386+0000","last_active":"2023-10-16T12:34:44.951513+0000","last_peered":"2023-10-16T12:34:44.951513+0000","last_clean":"2023-10-16T12:34:44.951513+0000","last_became_active":"2023-10-16T12:34:42.897078+0000","last_became_peered":"2023-10-16T12:34:42.897078+0000","last_unstale":"2023-10-16T12:34:44.951513+0000","last_undegraded":"2023-10-16T12:34:44.951513+0000","last_fullsized":"2023-10-16T12:34:44.951513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:09.153738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948974+0000","last_change":"2023-10-16T12:34:43.930074+0000","last_active":"2023-10-16T12:34:44.948974+0000","last_peered":"2023-10-16T12:34:44.948974+0000","last_clean":"2023-10-16T12:34:44.948974+0000","last_became_active":"2023-10-16T12:34:43.929115+0000","last_became_peered":"2023-10-16T12:34:43.929115+0000","last_unstale":"2023-10-16T12:34:44.948974+0000","last_undegraded":"2023-10-16T12:34:44.948974+0000","last_fullsized":"2023-10-16T12:34:44.948974+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:36.187438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948876+0000","last_change":"2023-10-16T12:34:42.913016+0000","last_active":"2023-10-16T12:34:44.948876+0000","last_peered":"2023-10-16T12:34:44.948876+0000","last_clean":"2023-10-16T12:34:44.948876+0000","last_became_active":"2023-10-16T12:34:42.911685+0000","last_became_peered":"2023-10-16T12:34:42.911685+0000","last_unstale":"2023-10-16T12:34:44.948876+0000","last_undegraded":"2023-10-16T12:34:44.948876+0000","last_fullsized":"2023-10-16T12:34:44.948876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:51:14.752734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948168+0000","last_change":"2023-10-16T12:34:43.928322+0000","last_active":"2023-10-16T12:34:44.948168+0000","last_peered":"2023-10-16T12:34:44.948168+0000","last_clean":"2023-10-16T12:34:44.948168+0000","last_became_active":"2023-10-16T12:34:43.928060+0000","last_became_peered":"2023-10-16T12:34:43.928060+0000","last_unstale":"2023-10-16T12:34:44.948168+0000","last_undegraded":"2023-10-16T12:34:44.948168+0000","last_fullsized":"2023-10-16T12:34:44.948168+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:12.148748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921728+0000","last_change":"2023-10-16T12:34:42.907306+0000","last_active":"2023-10-16T12:34:44.921728+0000","last_peered":"2023-10-16T12:34:44.921728+0000","last_clean":"2023-10-16T12:34:44.921728+0000","last_became_active":"2023-10-16T12:34:42.906994+0000","last_became_peered":"2023-10-16T12:34:42.906994+0000","last_unstale":"2023-10-16T12:34:44.921728+0000","last_undegraded":"2023-10-16T12:34:44.921728+0000","last_fullsized":"2023-10-16T12:34:44.921728+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:38.379882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950064+0000","last_change":"2023-10-16T12:34:43.915114+0000","last_active":"2023-10-16T12:34:44.950064+0000","last_peered":"2023-10-16T12:34:44.950064+0000","last_clean":"2023-10-16T12:34:44.950064+0000","last_became_active":"2023-10-16T12:34:43.911672+0000","last_became_peered":"2023-10-16T12:34:43.911672+0000","last_unstale":"2023-10-16T12:34:44.950064+0000","last_undegraded":"2023-10-16T12:34:44.950064+0000","last_fullsized":"2023-10-16T12:34:44.950064+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:07.978483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.920762+0000","last_change":"2023-10-16T12:34:42.902063+0000","last_active":"2023-10-16T12:34:44.920762+0000","last_peered":"2023-10-16T12:34:44.920762+0000","last_clean":"2023-10-16T12:34:44.920762+0000","last_became_active":"2023-10-16T12:34:42.901765+0000","last_became_peered":"2023-10-16T12:34:42.901765+0000","last_unstale":"2023-10-16T12:34:44.920762+0000","last_undegraded":"2023-10-16T12:34:44.920762+0000","last_fullsized":"2023-10-16T12:34:44.920762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:22.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948743+0000","last_change":"2023-10-16T12:34:43.929127+0000","last_active":"2023-10-16T12:34:44.948743+0000","last_peered":"2023-10-16T12:34:44.948743+0000","last_clean":"2023-10-16T12:34:44.948743+0000","last_became_active":"2023-10-16T12:34:43.928794+0000","last_became_peered":"2023-10-16T12:34:43.928794+0000","last_unstale":"2023-10-16T12:34:44.948743+0000","last_undegraded":"2023-10-16T12:34:44.948743+0000","last_fullsized":"2023-10-16T12:34:44.948743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:51.065728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918156+0000","last_change":"2023-10-16T12:34:42.903231+0000","last_active":"2023-10-16T12:34:44.918156+0000","last_peered":"2023-10-16T12:34:44.918156+0000","last_clean":"2023-10-16T12:34:44.918156+0000","last_became_active":"2023-10-16T12:34:42.902913+0000","last_became_peered":"2023-10-16T12:34:42.902913+0000","last_unstale":"2023-10-16T12:34:44.918156+0000","last_undegraded":"2023-10-16T12:34:44.918156+0000","last_fullsized":"2023-10-16T12:34:44.918156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:09.153738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922485+0000","last_change":"2023-10-16T12:34:43.935703+0000","last_active":"2023-10-16T12:34:44.922485+0000","last_peered":"2023-10-16T12:34:44.922485+0000","last_clean":"2023-10-16T12:34:44.922485+0000","last_became_active":"2023-10-16T12:34:43.935344+0000","last_became_peered":"2023-10-16T12:34:43.935344+0000","last_unstale":"2023-10-16T12:34:44.922485+0000","last_undegraded":"2023-10-16T12:34:44.922485+0000","last_fullsized":"2023-10-16T12:34:44.922485+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952184+0000","last_change":"2023-10-16T12:34:42.897264+0000","last_active":"2023-10-16T12:34:44.952184+0000","last_peered":"2023-10-16T12:34:44.952184+0000","last_clean":"2023-10-16T12:34:44.952184+0000","last_became_active":"2023-10-16T12:34:42.896857+0000","last_became_peered":"2023-10-16T12:34:42.896857+0000","last_unstale":"2023-10-16T12:34:44.952184+0000","last_undegraded":"2023-10-16T12:34:44.952184+0000","last_fullsized":"2023-10-16T12:34:44.952184+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:38.379882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951630+0000","last_change":"2023-10-16T12:34:43.926542+0000","last_active":"2023-10-16T12:34:44.951630+0000","last_peered":"2023-10-16T12:34:44.951630+0000","last_clean":"2023-10-16T12:34:44.951630+0000","last_became_active":"2023-10-16T12:34:43.926259+0000","last_became_peered":"2023-10-16T12:34:43.926259+0000","last_unstale":"2023-10-16T12:34:44.951630+0000","last_undegraded":"2023-10-16T12:34:44.951630+0000","last_fullsized":"2023-10-16T12:34:44.951630+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:50.157019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917604+0000","last_change":"2023-10-16T12:34:42.899972+0000","last_active":"2023-10-16T12:34:44.917604+0000","last_peered":"2023-10-16T12:34:44.917604+0000","last_clean":"2023-10-16T12:34:44.917604+0000","last_became_active":"2023-10-16T12:34:42.899536+0000","last_became_peered":"2023-10-16T12:34:42.899536+0000","last_unstale":"2023-10-16T12:34:44.917604+0000","last_undegraded":"2023-10-16T12:34:44.917604+0000","last_fullsized":"2023-10-16T12:34:44.917604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:06.743408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948651+0000","last_change":"2023-10-16T12:34:43.926988+0000","last_active":"2023-10-16T12:34:44.948651+0000","last_peered":"2023-10-16T12:34:44.948651+0000","last_clean":"2023-10-16T12:34:44.948651+0000","last_became_active":"2023-10-16T12:34:43.925046+0000","last_became_peered":"2023-10-16T12:34:43.925046+0000","last_unstale":"2023-10-16T12:34:44.948651+0000","last_undegraded":"2023-10-16T12:34:44.948651+0000","last_fullsized":"2023-10-16T12:34:44.948651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:19.266077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951701+0000","last_change":"2023-10-16T12:34:42.903567+0000","last_active":"2023-10-16T12:34:44.951701+0000","last_peered":"2023-10-16T12:34:44.951701+0000","last_clean":"2023-10-16T12:34:44.951701+0000","last_became_active":"2023-10-16T12:34:42.901990+0000","last_became_peered":"2023-10-16T12:34:42.901990+0000","last_unstale":"2023-10-16T12:34:44.951701+0000","last_undegraded":"2023-10-16T12:34:44.951701+0000","last_fullsized":"2023-10-16T12:34:44.951701+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:56.384420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951633+0000","last_change":"2023-10-16T12:34:43.928127+0000","last_active":"2023-10-16T12:34:44.951633+0000","last_peered":"2023-10-16T12:34:44.951633+0000","last_clean":"2023-10-16T12:34:44.951633+0000","last_became_active":"2023-10-16T12:34:43.927718+0000","last_became_peered":"2023-10-16T12:34:43.927718+0000","last_unstale":"2023-10-16T12:34:44.951633+0000","last_undegraded":"2023-10-16T12:34:44.951633+0000","last_fullsized":"2023-10-16T12:34:44.951633+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:42.104379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917101+0000","last_change":"2023-10-16T12:34:42.903353+0000","last_active":"2023-10-16T12:34:44.917101+0000","last_peered":"2023-10-16T12:34:44.917101+0000","last_clean":"2023-10-16T12:34:44.917101+0000","last_became_active":"2023-10-16T12:34:42.903040+0000","last_became_peered":"2023-10-16T12:34:42.903040+0000","last_unstale":"2023-10-16T12:34:44.917101+0000","last_undegraded":"2023-10-16T12:34:44.917101+0000","last_fullsized":"2023-10-16T12:34:44.917101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:49.142884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.947842+0000","last_change":"2023-10-16T12:34:43.926671+0000","last_active":"2023-10-16T12:34:44.947842+0000","last_peered":"2023-10-16T12:34:44.947842+0000","last_clean":"2023-10-16T12:34:44.947842+0000","last_became_active":"2023-10-16T12:34:43.926317+0000","last_became_peered":"2023-10-16T12:34:43.926317+0000","last_unstale":"2023-10-16T12:34:44.947842+0000","last_undegraded":"2023-10-16T12:34:44.947842+0000","last_fullsized":"2023-10-16T12:34:44.947842+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.771260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922233+0000","last_change":"2023-10-16T12:34:42.902999+0000","last_active":"2023-10-16T12:34:44.922233+0000","last_peered":"2023-10-16T12:34:44.922233+0000","last_clean":"2023-10-16T12:34:44.922233+0000","last_became_active":"2023-10-16T12:34:42.902709+0000","last_became_peered":"2023-10-16T12:34:42.902709+0000","last_unstale":"2023-10-16T12:34:44.922233+0000","last_undegraded":"2023-10-16T12:34:44.922233+0000","last_fullsized":"2023-10-16T12:34:44.922233+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:09.153738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921460+0000","last_change":"2023-10-16T12:34:43.927931+0000","last_active":"2023-10-16T12:34:44.921460+0000","last_peered":"2023-10-16T12:34:44.921460+0000","last_clean":"2023-10-16T12:34:44.921460+0000","last_became_active":"2023-10-16T12:34:43.927627+0000","last_became_peered":"2023-10-16T12:34:43.927627+0000","last_unstale":"2023-10-16T12:34:44.921460+0000","last_undegraded":"2023-10-16T12:34:44.921460+0000","last_fullsized":"2023-10-16T12:34:44.921460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:07.757544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948018+0000","last_change":"2023-10-16T12:34:42.902828+0000","last_active":"2023-10-16T12:34:44.948018+0000","last_peered":"2023-10-16T12:34:44.948018+0000","last_clean":"2023-10-16T12:34:44.948018+0000","last_became_active":"2023-10-16T12:34:42.902549+0000","last_became_peered":"2023-10-16T12:34:42.902549+0000","last_unstale":"2023-10-16T12:34:44.948018+0000","last_undegraded":"2023-10-16T12:34:44.948018+0000","last_fullsized":"2023-10-16T12:34:44.948018+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:27.563110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950806+0000","last_change":"2023-10-16T12:34:43.928337+0000","last_active":"2023-10-16T12:34:44.950806+0000","last_peered":"2023-10-16T12:34:44.950806+0000","last_clean":"2023-10-16T12:34:44.950806+0000","last_became_active":"2023-10-16T12:34:43.927812+0000","last_became_peered":"2023-10-16T12:34:43.927812+0000","last_unstale":"2023-10-16T12:34:44.950806+0000","last_undegraded":"2023-10-16T12:34:44.950806+0000","last_fullsized":"2023-10-16T12:34:44.950806+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948784+0000","last_change":"2023-10-16T12:34:42.913894+0000","last_active":"2023-10-16T12:34:44.948784+0000","last_peered":"2023-10-16T12:34:44.948784+0000","last_clean":"2023-10-16T12:34:44.948784+0000","last_became_active":"2023-10-16T12:34:42.913535+0000","last_became_peered":"2023-10-16T12:34:42.913535+0000","last_unstale":"2023-10-16T12:34:44.948784+0000","last_undegraded":"2023-10-16T12:34:44.948784+0000","last_fullsized":"2023-10-16T12:34:44.948784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:52.019635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-16T12:34:43.903418+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:34:43.903418+0000","last_peered":"2023-10-16T12:34:43.903418+0000","last_clean":"2023-10-16T12:34:43.903418+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:34:43.903418+0000","last_undegraded":"2023-10-16T12:34:43.903418+0000","last_fullsized":"2023-10-16T12:34:43.903418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":1003520,"data_stored":750474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.399},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.5600000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.458}]},{"osd":2,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.115},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.639}]},{"osd":3,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.587},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.714}]},{"osd":4,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.242},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.5}]},{"osd":5,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.675}]},{"osd":6,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":1.75},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":2.1379999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":160632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.306},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.028}]},{"osd":1,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.397}]},{"osd":2,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.554},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.2}]},{"osd":3,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.446}]},{"osd":4,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.498}]},{"osd":5,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.363,"15min":1.363},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.5230000000000001,"5min":2.5230000000000001,"15min":2.5230000000000001},"last":1.7290000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":1.363},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.5209999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":160688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.234},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":1.127}]},{"osd":1,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.21,"15min":1.21},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.5960000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.302},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.75700000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.95699999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.611,"15min":1.611},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":5.3250000000000002,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":1.855}]},{"osd":6,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.484},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.4119999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":5.1269999999999998,"5min":5.1269999999999998,"15min":5.1269999999999998},"last":1.806},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":1.6539999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":413696,"data_stored":160120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":1.008}]},{"osd":2,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.3240000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.236},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.95099999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":7.9000000000000004,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":1.272},{"interface":"front","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":7.3460000000000001,"5min":7.3460000000000001,"15min":7.3460000000000001},"last":1.3500000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.365}]},{"osd":7,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.302},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.159}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":161650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.0740000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.28},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.3959999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.4610000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":3.6930000000000001,"5min":3.6930000000000001,"15min":3.6930000000000001},"last":1.514}]},{"osd":5,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.5569999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":3.9809999999999999},"last":1.708}]},{"osd":7,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.772},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.756}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":160086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.74299999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.036},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.323}]},{"osd":3,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.099}]},{"osd":4,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.244},{"interface":"front","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.5549999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.649},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.2809999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.411},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.488,"5min":2.488,"15min":2.488},"last":1.381}]},{"osd":7,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.452},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":2.6429999999999998,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":1.4930000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":161650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.165}]},{"osd":2,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.8100000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.462}]},{"osd":4,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":3.0659999999999998,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9630000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.609},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.5149999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":2.9350000000000001,"5min":2.9350000000000001,"15min":2.9350000000000001},"last":2.1280000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.7470000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.516,"5min":1.516,"15min":1.516},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.903}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":750384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.3420000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.202},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.151},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.415,"15min":1.415},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":4.726,"5min":4.726,"15min":4.726},"last":1.262},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":4.843,"5min":4.843,"15min":4.843},"last":1.4359999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9059999999999997,"5min":4.9059999999999997,"15min":4.9059999999999997},"last":1.3979999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.296},{"interface":"front","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.2270000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.3969999999999998,"5min":2.3969999999999998,"15min":2.3969999999999998},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.4730000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:34:48.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:34:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.709+0000 7fc4b9c32640 1 Processor -- start 2023-10-16T12:34:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.709+0000 7fc4b9c32640 1 -- start start 2023-10-16T12:34:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.710+0000 7fc4b9c32640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4b415cd70 0x7fc4b40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.710+0000 7fc4b9c32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b9c32640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b415d150 con 0x7fc4b40b8480 2023-10-16T12:34:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b9c32640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b40d1180 con 0x7fc4b415cd70 2023-10-16T12:34:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b9c32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b40d1460 con 0x7fc4b40cded0 2023-10-16T12:34:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4b415cd70 0x7fc4b40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4b415cd70 0x7fc4b40d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42184/0 (socket says 172.21.15.60:42184) 2023-10-16T12:34:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 -- 172.21.15.60:0/3023370522 learned_addr learned my addr 172.21.15.60:0/3023370522 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b3fff640 1 --1- 172.21.15.60:0/3023370522 >> v1:172.21.15.60:6789/0 conn(0x7fc4b40b8480 0x7fc4b40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51456/0 (socket says 172.21.15.60:51456) 2023-10-16T12:34:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 -- 172.21.15.60:0/3023370522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 msgr2=0x7fc4b40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b9431640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b40b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b37fe640 1 -- 172.21.15.60:0/3023370522 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317299374 0 0) 0x7fc4b415d150 con 0x7fc4b40b8480 2023-10-16T12:34:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 -- 172.21.15.60:0/3023370522 >> v1:172.21.15.60:6789/0 conn(0x7fc4b40b8480 legacy=0x7fc4b40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b8c30640 1 -- 172.21.15.60:0/3023370522 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b40d1740 con 0x7fc4b415cd70 2023-10-16T12:34:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.711+0000 7fc4b9431640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.713+0000 7fc4b8c30640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b415cd70 0x7fc4b40d2220 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009870 tx=0x7fc4a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df80fd379ce04ff0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.713+0000 7fc4b37fe640 1 -- 172.21.15.60:0/3023370522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8030840 con 0x7fc4b415cd70 2023-10-16T12:34:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.713+0000 7fc4b37fe640 1 -- 172.21.15.60:0/3023370522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a80309a0 con 0x7fc4b415cd70 2023-10-16T12:34:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.713+0000 7fc4b37fe640 1 -- 172.21.15.60:0/3023370522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8030d30 con 0x7fc4b415cd70 2023-10-16T12:34:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.713+0000 7fc4b9c32640 1 -- 172.21.15.60:0/3023370522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b415cd70 msgr2=0x7fc4b40d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.713+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b415cd70 0x7fc4b40d2220 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009870 tx=0x7fc4a802b980 comp rx=0 tx=0).stop 2023-10-16T12:34:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.714+0000 7fc4b9c32640 1 -- 172.21.15.60:0/3023370522 shutdown_connections 2023-10-16T12:34:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.714+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.714+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/3023370522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b415cd70 0x7fc4b40d2220 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.714+0000 7fc4b9c32640 1 -- 172.21.15.60:0/3023370522 >> 172.21.15.60:0/3023370522 conn(0x7fc4b40cf570 msgr2=0x7fc4b40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.714+0000 7fc4b9c32640 1 -- 172.21.15.60:0/3023370522 shutdown_connections 2023-10-16T12:34:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.714+0000 7fc4b9c32640 1 -- 172.21.15.60:0/3023370522 wait complete. 2023-10-16T12:34:48.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.715+0000 7fc4b9c32640 1 Processor -- start 2023-10-16T12:34:48.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 -- start start 2023-10-16T12:34:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 0x7fc4b4159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b41f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b415cd70 0x7fc4b40b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4146020 con 0x7fc4b40b8480 2023-10-16T12:34:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4158960 con 0x7fc4b415cd70 2023-10-16T12:34:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.716+0000 7fc4b9c32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4158c40 con 0x7fc4b40cded0 2023-10-16T12:34:48.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 0x7fc4b4159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 0x7fc4b4159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42190/0 (socket says 172.21.15.60:42190) 2023-10-16T12:34:48.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 -- 172.21.15.60:0/2269237311 learned_addr learned my addr 172.21.15.60:0/2269237311 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:48.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 -- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 msgr2=0x7fc4b41f5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b41f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 -- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b415cd70 msgr2=0x7fc4b40b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b415cd70 0x7fc4b40b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.717+0000 7fc4b8c30640 1 -- 172.21.15.60:0/2269237311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80094d0 con 0x7fc4b40b8480 2023-10-16T12:34:48.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.718+0000 7fc4b8c30640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 0x7fc4b4159440 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009870 tx=0x7fc4a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:48.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.718+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a80307f0 con 0x7fc4b40b8480 2023-10-16T12:34:48.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.718+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a8039d90 con 0x7fc4b40b8480 2023-10-16T12:34:48.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.718+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4158f20 con 0x7fc4b40b8480 2023-10-16T12:34:48.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.718+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8043630 con 0x7fc4b40b8480 2023-10-16T12:34:48.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.718+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b41585a0 con 0x7fc4b40b8480 2023-10-16T12:34:48.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.719+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4160360 con 0x7fc4b40b8480 2023-10-16T12:34:48.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.720+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc4a804c3b0 con 0x7fc4b40b8480 2023-10-16T12:34:48.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.721+0000 7fc4b17fa640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49806ae70 0x7fc49806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:48.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.721+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4a80abd30 con 0x7fc4b40b8480 2023-10-16T12:34:48.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.722+0000 7fc4b3fff640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49806ae70 0x7fc49806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:48.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.723+0000 7fc4b3fff640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49806ae70 0x7fc49806d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0007fd0 tx=0x7fc4a0007940 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:48.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.725+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4a8076d40 con 0x7fc4b40b8480 2023-10-16T12:34:48.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.919+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4b4158ad0 con 0x7fc49806ae70 2023-10-16T12:34:48.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.943+0000 7fc4b17fa640 1 -- 172.21.15.60:0/2269237311 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286631 (secure 0 0 0) 0x7fc4b4158ad0 con 0x7fc49806ae70 2023-10-16T12:34:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:48.949 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:34:48.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.948+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49806ae70 msgr2=0x7fc49806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.948+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49806ae70 0x7fc49806d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0007fd0 tx=0x7fc4a0007940 comp rx=0 tx=0).stop 2023-10-16T12:34:48.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.948+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 msgr2=0x7fc4b4159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:48.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 0x7fc4b4159440 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009870 tx=0x7fc4a8004730 comp rx=0 tx=0).stop 2023-10-16T12:34:48.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 shutdown_connections 2023-10-16T12:34:48.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fc49806ae70 0x7fc49806d330 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40cded0 0x7fc4b41f5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b415cd70 0x7fc4b40b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 --2- 172.21.15.60:0/2269237311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40b8480 0x7fc4b4159440 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:48.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 >> 172.21.15.60:0/2269237311 conn(0x7fc4b40cf570 msgr2=0x7fc4b415bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:48.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 shutdown_connections 2023-10-16T12:34:48.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:48.949+0000 7fc4b9c32640 1 -- 172.21.15.60:0/2269237311 wait complete. 2023-10-16T12:34:48.971 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-10-16T12:34:47.759444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237204,"kb_used_data":4392,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737516,"statfs":{"total":773094113280,"available":772851216384,"internally_reserved":0,"allocated":4497408,"data_stored":2465684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.893611"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949247+0000","last_change":"2023-10-16T12:34:43.930198+0000","last_active":"2023-10-16T12:34:44.949247+0000","last_peered":"2023-10-16T12:34:44.949247+0000","last_clean":"2023-10-16T12:34:44.949247+0000","last_became_active":"2023-10-16T12:34:43.929005+0000","last_became_peered":"2023-10-16T12:34:43.929005+0000","last_unstale":"2023-10-16T12:34:44.949247+0000","last_undegraded":"2023-10-16T12:34:44.949247+0000","last_fullsized":"2023-10-16T12:34:44.949247+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:49.302703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950274+0000","last_change":"2023-10-16T12:34:43.922962+0000","last_active":"2023-10-16T12:34:44.950274+0000","last_peered":"2023-10-16T12:34:44.950274+0000","last_clean":"2023-10-16T12:34:44.950274+0000","last_became_active":"2023-10-16T12:34:43.922670+0000","last_became_peered":"2023-10-16T12:34:43.922670+0000","last_unstale":"2023-10-16T12:34:44.950274+0000","last_undegraded":"2023-10-16T12:34:44.950274+0000","last_fullsized":"2023-10-16T12:34:44.950274+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921687+0000","last_change":"2023-10-16T12:34:43.935492+0000","last_active":"2023-10-16T12:34:44.921687+0000","last_peered":"2023-10-16T12:34:44.921687+0000","last_clean":"2023-10-16T12:34:44.921687+0000","last_became_active":"2023-10-16T12:34:43.935215+0000","last_became_peered":"2023-10-16T12:34:43.935215+0000","last_unstale":"2023-10-16T12:34:44.921687+0000","last_undegraded":"2023-10-16T12:34:44.921687+0000","last_fullsized":"2023-10-16T12:34:44.921687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.771260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917372+0000","last_change":"2023-10-16T12:34:43.938392+0000","last_active":"2023-10-16T12:34:44.917372+0000","last_peered":"2023-10-16T12:34:44.917372+0000","last_clean":"2023-10-16T12:34:44.917372+0000","last_became_active":"2023-10-16T12:34:43.937961+0000","last_became_peered":"2023-10-16T12:34:43.937961+0000","last_unstale":"2023-10-16T12:34:44.917372+0000","last_undegraded":"2023-10-16T12:34:44.917372+0000","last_fullsized":"2023-10-16T12:34:44.917372+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:33.202722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948363+0000","last_change":"2023-10-16T12:34:43.950180+0000","last_active":"2023-10-16T12:34:44.948363+0000","last_peered":"2023-10-16T12:34:44.948363+0000","last_clean":"2023-10-16T12:34:44.948363+0000","last_became_active":"2023-10-16T12:34:43.949836+0000","last_became_peered":"2023-10-16T12:34:43.949836+0000","last_unstale":"2023-10-16T12:34:44.948363+0000","last_undegraded":"2023-10-16T12:34:44.948363+0000","last_fullsized":"2023-10-16T12:34:44.948363+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:38.679343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948115+0000","last_change":"2023-10-16T12:34:43.936331+0000","last_active":"2023-10-16T12:34:44.948115+0000","last_peered":"2023-10-16T12:34:44.948115+0000","last_clean":"2023-10-16T12:34:44.948115+0000","last_became_active":"2023-10-16T12:34:43.929952+0000","last_became_peered":"2023-10-16T12:34:43.929952+0000","last_unstale":"2023-10-16T12:34:44.948115+0000","last_undegraded":"2023-10-16T12:34:44.948115+0000","last_fullsized":"2023-10-16T12:34:44.948115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:28.037742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950228+0000","last_change":"2023-10-16T12:34:43.919371+0000","last_active":"2023-10-16T12:34:44.950228+0000","last_peered":"2023-10-16T12:34:44.950228+0000","last_clean":"2023-10-16T12:34:44.950228+0000","last_became_active":"2023-10-16T12:34:43.918990+0000","last_became_peered":"2023-10-16T12:34:43.918990+0000","last_unstale":"2023-10-16T12:34:44.950228+0000","last_undegraded":"2023-10-16T12:34:44.950228+0000","last_fullsized":"2023-10-16T12:34:44.950228+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.771260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949909+0000","last_change":"2023-10-16T12:34:43.928269+0000","last_active":"2023-10-16T12:34:44.949909+0000","last_peered":"2023-10-16T12:34:44.949909+0000","last_clean":"2023-10-16T12:34:44.949909+0000","last_became_active":"2023-10-16T12:34:43.927728+0000","last_became_peered":"2023-10-16T12:34:43.927728+0000","last_unstale":"2023-10-16T12:34:44.949909+0000","last_undegraded":"2023-10-16T12:34:44.949909+0000","last_fullsized":"2023-10-16T12:34:44.949909+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:38.692665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.924059+0000","last_change":"2023-10-16T12:34:43.937739+0000","last_active":"2023-10-16T12:34:44.924059+0000","last_peered":"2023-10-16T12:34:44.924059+0000","last_clean":"2023-10-16T12:34:44.924059+0000","last_became_active":"2023-10-16T12:34:43.937429+0000","last_became_peered":"2023-10-16T12:34:43.937429+0000","last_unstale":"2023-10-16T12:34:44.924059+0000","last_undegraded":"2023-10-16T12:34:44.924059+0000","last_fullsized":"2023-10-16T12:34:44.924059+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:22.197181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918033+0000","last_change":"2023-10-16T12:34:43.942762+0000","last_active":"2023-10-16T12:34:44.918033+0000","last_peered":"2023-10-16T12:34:44.918033+0000","last_clean":"2023-10-16T12:34:44.918033+0000","last_became_active":"2023-10-16T12:34:43.937861+0000","last_became_peered":"2023-10-16T12:34:43.937861+0000","last_unstale":"2023-10-16T12:34:44.918033+0000","last_undegraded":"2023-10-16T12:34:44.918033+0000","last_fullsized":"2023-10-16T12:34:44.918033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:37.757634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951128+0000","last_change":"2023-10-16T12:34:43.926827+0000","last_active":"2023-10-16T12:34:44.951128+0000","last_peered":"2023-10-16T12:34:44.951128+0000","last_clean":"2023-10-16T12:34:44.951128+0000","last_became_active":"2023-10-16T12:34:43.924813+0000","last_became_peered":"2023-10-16T12:34:43.924813+0000","last_unstale":"2023-10-16T12:34:44.951128+0000","last_undegraded":"2023-10-16T12:34:44.951128+0000","last_fullsized":"2023-10-16T12:34:44.951128+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:50.221503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921325+0000","last_change":"2023-10-16T12:34:43.941570+0000","last_active":"2023-10-16T12:34:44.921325+0000","last_peered":"2023-10-16T12:34:44.921325+0000","last_clean":"2023-10-16T12:34:44.921325+0000","last_became_active":"2023-10-16T12:34:43.941226+0000","last_became_peered":"2023-10-16T12:34:43.941226+0000","last_unstale":"2023-10-16T12:34:44.921325+0000","last_undegraded":"2023-10-16T12:34:44.921325+0000","last_fullsized":"2023-10-16T12:34:44.921325+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:32.221349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948744+0000","last_change":"2023-10-16T12:34:43.935981+0000","last_active":"2023-10-16T12:34:44.948744+0000","last_peered":"2023-10-16T12:34:44.948744+0000","last_clean":"2023-10-16T12:34:44.948744+0000","last_became_active":"2023-10-16T12:34:43.935634+0000","last_became_peered":"2023-10-16T12:34:43.935634+0000","last_unstale":"2023-10-16T12:34:44.948744+0000","last_undegraded":"2023-10-16T12:34:44.948744+0000","last_fullsized":"2023-10-16T12:34:44.948744+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:16.661136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949535+0000","last_change":"2023-10-16T12:34:43.936353+0000","last_active":"2023-10-16T12:34:44.949535+0000","last_peered":"2023-10-16T12:34:44.949535+0000","last_clean":"2023-10-16T12:34:44.949535+0000","last_became_active":"2023-10-16T12:34:43.927332+0000","last_became_peered":"2023-10-16T12:34:43.927332+0000","last_unstale":"2023-10-16T12:34:44.949535+0000","last_undegraded":"2023-10-16T12:34:44.949535+0000","last_fullsized":"2023-10-16T12:34:44.949535+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:48.267707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921283+0000","last_change":"2023-10-16T12:34:43.924656+0000","last_active":"2023-10-16T12:34:44.921283+0000","last_peered":"2023-10-16T12:34:44.921283+0000","last_clean":"2023-10-16T12:34:44.921283+0000","last_became_active":"2023-10-16T12:34:43.924330+0000","last_became_peered":"2023-10-16T12:34:43.924330+0000","last_unstale":"2023-10-16T12:34:44.921283+0000","last_undegraded":"2023-10-16T12:34:44.921283+0000","last_fullsized":"2023-10-16T12:34:44.921283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:38.692665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950515+0000","last_change":"2023-10-16T12:34:43.922370+0000","last_active":"2023-10-16T12:34:44.950515+0000","last_peered":"2023-10-16T12:34:44.950515+0000","last_clean":"2023-10-16T12:34:44.950515+0000","last_became_active":"2023-10-16T12:34:43.922085+0000","last_became_peered":"2023-10-16T12:34:43.922085+0000","last_unstale":"2023-10-16T12:34:44.950515+0000","last_undegraded":"2023-10-16T12:34:44.950515+0000","last_fullsized":"2023-10-16T12:34:44.950515+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:02.974058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951273+0000","last_change":"2023-10-16T12:34:43.911722+0000","last_active":"2023-10-16T12:34:44.951273+0000","last_peered":"2023-10-16T12:34:44.951273+0000","last_clean":"2023-10-16T12:34:44.951273+0000","last_became_active":"2023-10-16T12:34:43.911121+0000","last_became_peered":"2023-10-16T12:34:43.911121+0000","last_unstale":"2023-10-16T12:34:44.951273+0000","last_undegraded":"2023-10-16T12:34:44.951273+0000","last_fullsized":"2023-10-16T12:34:44.951273+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:56.298567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950414+0000","last_change":"2023-10-16T12:34:43.926887+0000","last_active":"2023-10-16T12:34:44.950414+0000","last_peered":"2023-10-16T12:34:44.950414+0000","last_clean":"2023-10-16T12:34:44.950414+0000","last_became_active":"2023-10-16T12:34:43.926593+0000","last_became_peered":"2023-10-16T12:34:43.926593+0000","last_unstale":"2023-10-16T12:34:44.950414+0000","last_undegraded":"2023-10-16T12:34:44.950414+0000","last_fullsized":"2023-10-16T12:34:44.950414+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:50.221503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950563+0000","last_change":"2023-10-16T12:34:43.932550+0000","last_active":"2023-10-16T12:34:44.950563+0000","last_peered":"2023-10-16T12:34:44.950563+0000","last_clean":"2023-10-16T12:34:44.950563+0000","last_became_active":"2023-10-16T12:34:43.927044+0000","last_became_peered":"2023-10-16T12:34:43.927044+0000","last_unstale":"2023-10-16T12:34:44.950563+0000","last_undegraded":"2023-10-16T12:34:44.950563+0000","last_fullsized":"2023-10-16T12:34:44.950563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:56.298567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948918+0000","last_change":"2023-10-16T12:34:43.929021+0000","last_active":"2023-10-16T12:34:44.948918+0000","last_peered":"2023-10-16T12:34:44.948918+0000","last_clean":"2023-10-16T12:34:44.948918+0000","last_became_active":"2023-10-16T12:34:43.928706+0000","last_became_peered":"2023-10-16T12:34:43.928706+0000","last_unstale":"2023-10-16T12:34:44.948918+0000","last_undegraded":"2023-10-16T12:34:44.948918+0000","last_fullsized":"2023-10-16T12:34:44.948918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:44.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922989+0000","last_change":"2023-10-16T12:34:43.932210+0000","last_active":"2023-10-16T12:34:44.922989+0000","last_peered":"2023-10-16T12:34:44.922989+0000","last_clean":"2023-10-16T12:34:44.922989+0000","last_became_active":"2023-10-16T12:34:43.931913+0000","last_became_peered":"2023-10-16T12:34:43.931913+0000","last_unstale":"2023-10-16T12:34:44.922989+0000","last_undegraded":"2023-10-16T12:34:44.922989+0000","last_fullsized":"2023-10-16T12:34:44.922989+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:50.157019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923036+0000","last_change":"2023-10-16T12:34:43.928342+0000","last_active":"2023-10-16T12:34:44.923036+0000","last_peered":"2023-10-16T12:34:44.923036+0000","last_clean":"2023-10-16T12:34:44.923036+0000","last_became_active":"2023-10-16T12:34:43.928067+0000","last_became_peered":"2023-10-16T12:34:43.928067+0000","last_unstale":"2023-10-16T12:34:44.923036+0000","last_undegraded":"2023-10-16T12:34:44.923036+0000","last_fullsized":"2023-10-16T12:34:44.923036+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:39.725825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950688+0000","last_change":"2023-10-16T12:34:43.928954+0000","last_active":"2023-10-16T12:34:44.950688+0000","last_peered":"2023-10-16T12:34:44.950688+0000","last_clean":"2023-10-16T12:34:44.950688+0000","last_became_active":"2023-10-16T12:34:43.928684+0000","last_became_peered":"2023-10-16T12:34:43.928684+0000","last_unstale":"2023-10-16T12:34:44.950688+0000","last_undegraded":"2023-10-16T12:34:44.950688+0000","last_fullsized":"2023-10-16T12:34:44.950688+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:07.757544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.920774+0000","last_change":"2023-10-16T12:34:43.926337+0000","last_active":"2023-10-16T12:34:44.920774+0000","last_peered":"2023-10-16T12:34:44.920774+0000","last_clean":"2023-10-16T12:34:44.920774+0000","last_became_active":"2023-10-16T12:34:43.926043+0000","last_became_peered":"2023-10-16T12:34:43.926043+0000","last_unstale":"2023-10-16T12:34:44.920774+0000","last_undegraded":"2023-10-16T12:34:44.920774+0000","last_fullsized":"2023-10-16T12:34:44.920774+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:03.305753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918020+0000","last_change":"2023-10-16T12:34:43.924982+0000","last_active":"2023-10-16T12:34:44.918020+0000","last_peered":"2023-10-16T12:34:44.918020+0000","last_clean":"2023-10-16T12:34:44.918020+0000","last_became_active":"2023-10-16T12:34:43.924701+0000","last_became_peered":"2023-10-16T12:34:43.924701+0000","last_unstale":"2023-10-16T12:34:44.918020+0000","last_undegraded":"2023-10-16T12:34:44.918020+0000","last_fullsized":"2023-10-16T12:34:44.918020+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:22.197181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950561+0000","last_change":"2023-10-16T12:34:43.915881+0000","last_active":"2023-10-16T12:34:44.950561+0000","last_peered":"2023-10-16T12:34:44.950561+0000","last_clean":"2023-10-16T12:34:44.950561+0000","last_became_active":"2023-10-16T12:34:43.914247+0000","last_became_peered":"2023-10-16T12:34:43.914247+0000","last_unstale":"2023-10-16T12:34:44.950561+0000","last_undegraded":"2023-10-16T12:34:44.950561+0000","last_fullsized":"2023-10-16T12:34:44.950561+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:55.889498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918052+0000","last_change":"2023-10-16T12:34:43.934418+0000","last_active":"2023-10-16T12:34:44.918052+0000","last_peered":"2023-10-16T12:34:44.918052+0000","last_clean":"2023-10-16T12:34:44.918052+0000","last_became_active":"2023-10-16T12:34:43.934119+0000","last_became_peered":"2023-10-16T12:34:43.934119+0000","last_unstale":"2023-10-16T12:34:44.918052+0000","last_undegraded":"2023-10-16T12:34:44.918052+0000","last_fullsized":"2023-10-16T12:34:44.918052+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:24:53.328597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922433+0000","last_change":"2023-10-16T12:34:43.929167+0000","last_active":"2023-10-16T12:34:44.922433+0000","last_peered":"2023-10-16T12:34:44.922433+0000","last_clean":"2023-10-16T12:34:44.922433+0000","last_became_active":"2023-10-16T12:34:43.928892+0000","last_became_peered":"2023-10-16T12:34:43.928892+0000","last_unstale":"2023-10-16T12:34:44.922433+0000","last_undegraded":"2023-10-16T12:34:44.922433+0000","last_fullsized":"2023-10-16T12:34:44.922433+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:48.267707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923038+0000","last_change":"2023-10-16T12:34:43.928710+0000","last_active":"2023-10-16T12:34:44.923038+0000","last_peered":"2023-10-16T12:34:44.923038+0000","last_clean":"2023-10-16T12:34:44.923038+0000","last_became_active":"2023-10-16T12:34:43.928375+0000","last_became_peered":"2023-10-16T12:34:43.928375+0000","last_unstale":"2023-10-16T12:34:44.923038+0000","last_undegraded":"2023-10-16T12:34:44.923038+0000","last_fullsized":"2023-10-16T12:34:44.923038+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:51.950513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949937+0000","last_change":"2023-10-16T12:34:43.912431+0000","last_active":"2023-10-16T12:34:44.949937+0000","last_peered":"2023-10-16T12:34:44.949937+0000","last_clean":"2023-10-16T12:34:44.949937+0000","last_became_active":"2023-10-16T12:34:43.911980+0000","last_became_peered":"2023-10-16T12:34:43.911980+0000","last_unstale":"2023-10-16T12:34:44.949937+0000","last_undegraded":"2023-10-16T12:34:44.949937+0000","last_fullsized":"2023-10-16T12:34:44.949937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:32.221349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948749+0000","last_change":"2023-10-16T12:34:43.926999+0000","last_active":"2023-10-16T12:34:44.948749+0000","last_peered":"2023-10-16T12:34:44.948749+0000","last_clean":"2023-10-16T12:34:44.948749+0000","last_became_active":"2023-10-16T12:34:43.926644+0000","last_became_peered":"2023-10-16T12:34:43.926644+0000","last_unstale":"2023-10-16T12:34:44.948749+0000","last_undegraded":"2023-10-16T12:34:44.948749+0000","last_fullsized":"2023-10-16T12:34:44.948749+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:24.393948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949938+0000","last_change":"2023-10-16T12:34:43.933027+0000","last_active":"2023-10-16T12:34:44.949938+0000","last_peered":"2023-10-16T12:34:44.949938+0000","last_clean":"2023-10-16T12:34:44.949938+0000","last_became_active":"2023-10-16T12:34:43.932717+0000","last_became_peered":"2023-10-16T12:34:43.932717+0000","last_unstale":"2023-10-16T12:34:44.949938+0000","last_undegraded":"2023-10-16T12:34:44.949938+0000","last_fullsized":"2023-10-16T12:34:44.949938+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:11.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952142+0000","last_change":"2023-10-16T12:34:43.932251+0000","last_active":"2023-10-16T12:34:44.952142+0000","last_peered":"2023-10-16T12:34:44.952142+0000","last_clean":"2023-10-16T12:34:44.952142+0000","last_became_active":"2023-10-16T12:34:43.925031+0000","last_became_peered":"2023-10-16T12:34:43.925031+0000","last_unstale":"2023-10-16T12:34:44.952142+0000","last_undegraded":"2023-10-16T12:34:44.952142+0000","last_fullsized":"2023-10-16T12:34:44.952142+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:51.950513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949324+0000","last_change":"2023-10-16T12:34:42.913313+0000","last_active":"2023-10-16T12:34:44.949324+0000","last_peered":"2023-10-16T12:34:44.949324+0000","last_clean":"2023-10-16T12:34:44.949324+0000","last_became_active":"2023-10-16T12:34:42.912917+0000","last_became_peered":"2023-10-16T12:34:42.912917+0000","last_unstale":"2023-10-16T12:34:44.949324+0000","last_undegraded":"2023-10-16T12:34:44.949324+0000","last_fullsized":"2023-10-16T12:34:44.949324+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:50.353010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949197+0000","last_change":"2023-10-16T12:34:43.932393+0000","last_active":"2023-10-16T12:34:44.949197+0000","last_peered":"2023-10-16T12:34:44.949197+0000","last_clean":"2023-10-16T12:34:44.949197+0000","last_became_active":"2023-10-16T12:34:43.932130+0000","last_became_peered":"2023-10-16T12:34:43.932130+0000","last_unstale":"2023-10-16T12:34:44.949197+0000","last_undegraded":"2023-10-16T12:34:44.949197+0000","last_fullsized":"2023-10-16T12:34:44.949197+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:37.700253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952008+0000","last_change":"2023-10-16T12:34:42.904347+0000","last_active":"2023-10-16T12:34:44.952008+0000","last_peered":"2023-10-16T12:34:44.952008+0000","last_clean":"2023-10-16T12:34:44.952008+0000","last_became_active":"2023-10-16T12:34:42.904071+0000","last_became_peered":"2023-10-16T12:34:42.904071+0000","last_unstale":"2023-10-16T12:34:44.952008+0000","last_undegraded":"2023-10-16T12:34:44.952008+0000","last_fullsized":"2023-10-16T12:34:44.952008+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:34.175108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923392+0000","last_change":"2023-10-16T12:34:43.947491+0000","last_active":"2023-10-16T12:34:44.923392+0000","last_peered":"2023-10-16T12:34:44.923392+0000","last_clean":"2023-10-16T12:34:44.923392+0000","last_became_active":"2023-10-16T12:34:43.947245+0000","last_became_peered":"2023-10-16T12:34:43.947245+0000","last_unstale":"2023-10-16T12:34:44.923392+0000","last_undegraded":"2023-10-16T12:34:44.923392+0000","last_fullsized":"2023-10-16T12:34:44.923392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:33.202722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950866+0000","last_change":"2023-10-16T12:34:42.901921+0000","last_active":"2023-10-16T12:34:44.950866+0000","last_peered":"2023-10-16T12:34:44.950866+0000","last_clean":"2023-10-16T12:34:44.950866+0000","last_became_active":"2023-10-16T12:34:42.901601+0000","last_became_peered":"2023-10-16T12:34:42.901601+0000","last_unstale":"2023-10-16T12:34:44.950866+0000","last_undegraded":"2023-10-16T12:34:44.950866+0000","last_fullsized":"2023-10-16T12:34:44.950866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:31.368211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918126+0000","last_change":"2023-10-16T12:34:43.923262+0000","last_active":"2023-10-16T12:34:44.918126+0000","last_peered":"2023-10-16T12:34:44.918126+0000","last_clean":"2023-10-16T12:34:44.918126+0000","last_became_active":"2023-10-16T12:34:43.922060+0000","last_became_peered":"2023-10-16T12:34:43.922060+0000","last_unstale":"2023-10-16T12:34:44.918126+0000","last_undegraded":"2023-10-16T12:34:44.918126+0000","last_fullsized":"2023-10-16T12:34:44.918126+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:23.676757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948904+0000","last_change":"2023-10-16T12:34:42.902554+0000","last_active":"2023-10-16T12:34:44.948904+0000","last_peered":"2023-10-16T12:34:44.948904+0000","last_clean":"2023-10-16T12:34:44.948904+0000","last_became_active":"2023-10-16T12:34:42.899186+0000","last_became_peered":"2023-10-16T12:34:42.899186+0000","last_unstale":"2023-10-16T12:34:44.948904+0000","last_undegraded":"2023-10-16T12:34:44.948904+0000","last_fullsized":"2023-10-16T12:34:44.948904+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:32.668405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950357+0000","last_change":"2023-10-16T12:34:43.931639+0000","last_active":"2023-10-16T12:34:44.950357+0000","last_peered":"2023-10-16T12:34:44.950357+0000","last_clean":"2023-10-16T12:34:44.950357+0000","last_became_active":"2023-10-16T12:34:43.930333+0000","last_became_peered":"2023-10-16T12:34:43.930333+0000","last_unstale":"2023-10-16T12:34:44.950357+0000","last_undegraded":"2023-10-16T12:34:44.950357+0000","last_fullsized":"2023-10-16T12:34:44.950357+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:56:48.267707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923183+0000","last_change":"2023-10-16T12:34:42.904493+0000","last_active":"2023-10-16T12:34:44.923183+0000","last_peered":"2023-10-16T12:34:44.923183+0000","last_clean":"2023-10-16T12:34:44.923183+0000","last_became_active":"2023-10-16T12:34:42.904195+0000","last_became_peered":"2023-10-16T12:34:42.904195+0000","last_unstale":"2023-10-16T12:34:44.923183+0000","last_undegraded":"2023-10-16T12:34:44.923183+0000","last_fullsized":"2023-10-16T12:34:44.923183+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:56.384420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921398+0000","last_change":"2023-10-16T12:34:43.929647+0000","last_active":"2023-10-16T12:34:44.921398+0000","last_peered":"2023-10-16T12:34:44.921398+0000","last_clean":"2023-10-16T12:34:44.921398+0000","last_became_active":"2023-10-16T12:34:43.928770+0000","last_became_peered":"2023-10-16T12:34:43.928770+0000","last_unstale":"2023-10-16T12:34:44.921398+0000","last_undegraded":"2023-10-16T12:34:44.921398+0000","last_fullsized":"2023-10-16T12:34:44.921398+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:07.978483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922949+0000","last_change":"2023-10-16T12:34:42.902944+0000","last_active":"2023-10-16T12:34:44.922949+0000","last_peered":"2023-10-16T12:34:44.922949+0000","last_clean":"2023-10-16T12:34:44.922949+0000","last_became_active":"2023-10-16T12:34:42.902580+0000","last_became_peered":"2023-10-16T12:34:42.902580+0000","last_unstale":"2023-10-16T12:34:44.922949+0000","last_undegraded":"2023-10-16T12:34:44.922949+0000","last_fullsized":"2023-10-16T12:34:44.922949+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:35.173303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921666+0000","last_change":"2023-10-16T12:34:43.934012+0000","last_active":"2023-10-16T12:34:44.921666+0000","last_peered":"2023-10-16T12:34:44.921666+0000","last_clean":"2023-10-16T12:34:44.921666+0000","last_became_active":"2023-10-16T12:34:43.933748+0000","last_became_peered":"2023-10-16T12:34:43.933748+0000","last_unstale":"2023-10-16T12:34:44.921666+0000","last_undegraded":"2023-10-16T12:34:44.921666+0000","last_fullsized":"2023-10-16T12:34:44.921666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:42.104379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949197+0000","last_change":"2023-10-16T12:34:42.899296+0000","last_active":"2023-10-16T12:34:44.949197+0000","last_peered":"2023-10-16T12:34:44.949197+0000","last_clean":"2023-10-16T12:34:44.949197+0000","last_became_active":"2023-10-16T12:34:42.898994+0000","last_became_peered":"2023-10-16T12:34:42.898994+0000","last_unstale":"2023-10-16T12:34:44.949197+0000","last_undegraded":"2023-10-16T12:34:44.949197+0000","last_fullsized":"2023-10-16T12:34:44.949197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:24:52.314462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.916959+0000","last_change":"2023-10-16T12:34:43.923159+0000","last_active":"2023-10-16T12:34:44.916959+0000","last_peered":"2023-10-16T12:34:44.916959+0000","last_clean":"2023-10-16T12:34:44.916959+0000","last_became_active":"2023-10-16T12:34:43.922129+0000","last_became_peered":"2023-10-16T12:34:43.922129+0000","last_unstale":"2023-10-16T12:34:44.916959+0000","last_undegraded":"2023-10-16T12:34:44.916959+0000","last_fullsized":"2023-10-16T12:34:44.916959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:39.725825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921879+0000","last_change":"2023-10-16T12:34:42.911662+0000","last_active":"2023-10-16T12:34:44.921879+0000","last_peered":"2023-10-16T12:34:44.921879+0000","last_clean":"2023-10-16T12:34:44.921879+0000","last_became_active":"2023-10-16T12:34:42.910293+0000","last_became_peered":"2023-10-16T12:34:42.910293+0000","last_unstale":"2023-10-16T12:34:44.921879+0000","last_undegraded":"2023-10-16T12:34:44.921879+0000","last_fullsized":"2023-10-16T12:34:44.921879+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:43:43.802570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922239+0000","last_change":"2023-10-16T12:34:43.941648+0000","last_active":"2023-10-16T12:34:44.922239+0000","last_peered":"2023-10-16T12:34:44.922239+0000","last_clean":"2023-10-16T12:34:44.922239+0000","last_became_active":"2023-10-16T12:34:43.941213+0000","last_became_peered":"2023-10-16T12:34:43.941213+0000","last_unstale":"2023-10-16T12:34:44.922239+0000","last_undegraded":"2023-10-16T12:34:44.922239+0000","last_fullsized":"2023-10-16T12:34:44.922239+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:28.037742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950952+0000","last_change":"2023-10-16T12:34:42.907472+0000","last_active":"2023-10-16T12:34:44.950952+0000","last_peered":"2023-10-16T12:34:44.950952+0000","last_clean":"2023-10-16T12:34:44.950952+0000","last_became_active":"2023-10-16T12:34:42.905716+0000","last_became_peered":"2023-10-16T12:34:42.905716+0000","last_unstale":"2023-10-16T12:34:44.950952+0000","last_undegraded":"2023-10-16T12:34:44.950952+0000","last_fullsized":"2023-10-16T12:34:44.950952+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:02.291618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950862+0000","last_change":"2023-10-16T12:34:43.926749+0000","last_active":"2023-10-16T12:34:44.950862+0000","last_peered":"2023-10-16T12:34:44.950862+0000","last_clean":"2023-10-16T12:34:44.950862+0000","last_became_active":"2023-10-16T12:34:43.926508+0000","last_became_peered":"2023-10-16T12:34:43.926508+0000","last_unstale":"2023-10-16T12:34:44.950862+0000","last_undegraded":"2023-10-16T12:34:44.950862+0000","last_fullsized":"2023-10-16T12:34:44.950862+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:59.555128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952098+0000","last_change":"2023-10-16T12:34:42.900177+0000","last_active":"2023-10-16T12:34:44.952098+0000","last_peered":"2023-10-16T12:34:44.952098+0000","last_clean":"2023-10-16T12:34:44.952098+0000","last_became_active":"2023-10-16T12:34:42.899888+0000","last_became_peered":"2023-10-16T12:34:42.899888+0000","last_unstale":"2023-10-16T12:34:44.952098+0000","last_undegraded":"2023-10-16T12:34:44.952098+0000","last_fullsized":"2023-10-16T12:34:44.952098+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.358016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951091+0000","last_change":"2023-10-16T12:34:43.912161+0000","last_active":"2023-10-16T12:34:44.951091+0000","last_peered":"2023-10-16T12:34:44.951091+0000","last_clean":"2023-10-16T12:34:44.951091+0000","last_became_active":"2023-10-16T12:34:43.911383+0000","last_became_peered":"2023-10-16T12:34:43.911383+0000","last_unstale":"2023-10-16T12:34:44.951091+0000","last_undegraded":"2023-10-16T12:34:44.951091+0000","last_fullsized":"2023-10-16T12:34:44.951091+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:10.834194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921759+0000","last_change":"2023-10-16T12:34:42.909164+0000","last_active":"2023-10-16T12:34:44.921759+0000","last_peered":"2023-10-16T12:34:44.921759+0000","last_clean":"2023-10-16T12:34:44.921759+0000","last_became_active":"2023-10-16T12:34:42.908234+0000","last_became_peered":"2023-10-16T12:34:42.908234+0000","last_unstale":"2023-10-16T12:34:44.921759+0000","last_undegraded":"2023-10-16T12:34:44.921759+0000","last_fullsized":"2023-10-16T12:34:44.921759+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:27.563110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921303+0000","last_change":"2023-10-16T12:34:43.935231+0000","last_active":"2023-10-16T12:34:44.921303+0000","last_peered":"2023-10-16T12:34:44.921303+0000","last_clean":"2023-10-16T12:34:44.921303+0000","last_became_active":"2023-10-16T12:34:43.934831+0000","last_became_peered":"2023-10-16T12:34:43.934831+0000","last_unstale":"2023-10-16T12:34:44.921303+0000","last_undegraded":"2023-10-16T12:34:44.921303+0000","last_fullsized":"2023-10-16T12:34:44.921303+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:11.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":18,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951152+0000","last_change":"2023-10-16T12:34:42.901772+0000","last_active":"2023-10-16T12:34:44.951152+0000","last_peered":"2023-10-16T12:34:44.951152+0000","last_clean":"2023-10-16T12:34:44.951152+0000","last_became_active":"2023-10-16T12:34:42.901477+0000","last_became_peered":"2023-10-16T12:34:42.901477+0000","last_unstale":"2023-10-16T12:34:44.951152+0000","last_undegraded":"2023-10-16T12:34:44.951152+0000","last_fullsized":"2023-10-16T12:34:44.951152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:36.743499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.916931+0000","last_change":"2023-10-16T12:34:43.933805+0000","last_active":"2023-10-16T12:34:44.916931+0000","last_peered":"2023-10-16T12:34:44.916931+0000","last_clean":"2023-10-16T12:34:44.916931+0000","last_became_active":"2023-10-16T12:34:43.926694+0000","last_became_peered":"2023-10-16T12:34:43.926694+0000","last_unstale":"2023-10-16T12:34:44.916931+0000","last_undegraded":"2023-10-16T12:34:44.916931+0000","last_fullsized":"2023-10-16T12:34:44.916931+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:51.950513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923258+0000","last_change":"2023-10-16T12:34:42.911025+0000","last_active":"2023-10-16T12:34:44.923258+0000","last_peered":"2023-10-16T12:34:44.923258+0000","last_clean":"2023-10-16T12:34:44.923258+0000","last_became_active":"2023-10-16T12:34:42.910753+0000","last_became_peered":"2023-10-16T12:34:42.910753+0000","last_unstale":"2023-10-16T12:34:44.923258+0000","last_undegraded":"2023-10-16T12:34:44.923258+0000","last_fullsized":"2023-10-16T12:34:44.923258+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:34.175108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950139+0000","last_change":"2023-10-16T12:34:43.920622+0000","last_active":"2023-10-16T12:34:44.950139+0000","last_peered":"2023-10-16T12:34:44.950139+0000","last_clean":"2023-10-16T12:34:44.950139+0000","last_became_active":"2023-10-16T12:34:43.920327+0000","last_became_peered":"2023-10-16T12:34:43.920327+0000","last_unstale":"2023-10-16T12:34:44.950139+0000","last_undegraded":"2023-10-16T12:34:44.950139+0000","last_fullsized":"2023-10-16T12:34:44.950139+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:02.974058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917040+0000","last_change":"2023-10-16T12:34:42.910360+0000","last_active":"2023-10-16T12:34:44.917040+0000","last_peered":"2023-10-16T12:34:44.917040+0000","last_clean":"2023-10-16T12:34:44.917040+0000","last_became_active":"2023-10-16T12:34:42.910010+0000","last_became_peered":"2023-10-16T12:34:42.910010+0000","last_unstale":"2023-10-16T12:34:44.917040+0000","last_undegraded":"2023-10-16T12:34:44.917040+0000","last_fullsized":"2023-10-16T12:34:44.917040+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:41.090244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950279+0000","last_change":"2023-10-16T12:34:43.932414+0000","last_active":"2023-10-16T12:34:44.950279+0000","last_peered":"2023-10-16T12:34:44.950279+0000","last_clean":"2023-10-16T12:34:44.950279+0000","last_became_active":"2023-10-16T12:34:43.932117+0000","last_became_peered":"2023-10-16T12:34:43.932117+0000","last_unstale":"2023-10-16T12:34:44.950279+0000","last_undegraded":"2023-10-16T12:34:44.950279+0000","last_fullsized":"2023-10-16T12:34:44.950279+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:28.037742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.949294+0000","last_change":"2023-10-16T12:34:42.902621+0000","last_active":"2023-10-16T12:34:44.949294+0000","last_peered":"2023-10-16T12:34:44.949294+0000","last_clean":"2023-10-16T12:34:44.949294+0000","last_became_active":"2023-10-16T12:34:42.902351+0000","last_became_peered":"2023-10-16T12:34:42.902351+0000","last_unstale":"2023-10-16T12:34:44.949294+0000","last_undegraded":"2023-10-16T12:34:44.949294+0000","last_fullsized":"2023-10-16T12:34:44.949294+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:06.743408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948888+0000","last_change":"2023-10-16T12:34:43.927554+0000","last_active":"2023-10-16T12:34:44.948888+0000","last_peered":"2023-10-16T12:34:44.948888+0000","last_clean":"2023-10-16T12:34:44.948888+0000","last_became_active":"2023-10-16T12:34:43.927274+0000","last_became_peered":"2023-10-16T12:34:43.927274+0000","last_unstale":"2023-10-16T12:34:44.948888+0000","last_undegraded":"2023-10-16T12:34:44.948888+0000","last_fullsized":"2023-10-16T12:34:44.948888+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:00.908601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.947794+0000","last_change":"2023-10-16T12:34:42.900613+0000","last_active":"2023-10-16T12:34:44.947794+0000","last_peered":"2023-10-16T12:34:44.947794+0000","last_clean":"2023-10-16T12:34:44.947794+0000","last_became_active":"2023-10-16T12:34:42.900350+0000","last_became_peered":"2023-10-16T12:34:42.900350+0000","last_unstale":"2023-10-16T12:34:44.947794+0000","last_undegraded":"2023-10-16T12:34:44.947794+0000","last_fullsized":"2023-10-16T12:34:44.947794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:21.183046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923905+0000","last_change":"2023-10-16T12:34:43.931499+0000","last_active":"2023-10-16T12:34:44.923905+0000","last_peered":"2023-10-16T12:34:44.923905+0000","last_clean":"2023-10-16T12:34:44.923905+0000","last_became_active":"2023-10-16T12:34:43.931226+0000","last_became_peered":"2023-10-16T12:34:43.931226+0000","last_unstale":"2023-10-16T12:34:44.923905+0000","last_undegraded":"2023-10-16T12:34:44.923905+0000","last_fullsized":"2023-10-16T12:34:44.923905+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:32.382347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918113+0000","last_change":"2023-10-16T12:34:42.905547+0000","last_active":"2023-10-16T12:34:44.918113+0000","last_peered":"2023-10-16T12:34:44.918113+0000","last_clean":"2023-10-16T12:34:44.918113+0000","last_became_active":"2023-10-16T12:34:42.905234+0000","last_became_peered":"2023-10-16T12:34:42.905234+0000","last_unstale":"2023-10-16T12:34:44.918113+0000","last_undegraded":"2023-10-16T12:34:44.918113+0000","last_fullsized":"2023-10-16T12:34:44.918113+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:38.379882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922484+0000","last_change":"2023-10-16T12:34:43.932591+0000","last_active":"2023-10-16T12:34:44.922484+0000","last_peered":"2023-10-16T12:34:44.922484+0000","last_clean":"2023-10-16T12:34:44.922484+0000","last_became_active":"2023-10-16T12:34:43.931829+0000","last_became_peered":"2023-10-16T12:34:43.931829+0000","last_unstale":"2023-10-16T12:34:44.922484+0000","last_undegraded":"2023-10-16T12:34:44.922484+0000","last_fullsized":"2023-10-16T12:34:44.922484+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:02.974058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923832+0000","last_change":"2023-10-16T12:34:42.910945+0000","last_active":"2023-10-16T12:34:44.923832+0000","last_peered":"2023-10-16T12:34:44.923832+0000","last_clean":"2023-10-16T12:34:44.923832+0000","last_became_active":"2023-10-16T12:34:42.910648+0000","last_became_peered":"2023-10-16T12:34:42.910648+0000","last_unstale":"2023-10-16T12:34:44.923832+0000","last_undegraded":"2023-10-16T12:34:44.923832+0000","last_fullsized":"2023-10-16T12:34:44.923832+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:46:35.902322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917513+0000","last_change":"2023-10-16T12:34:43.938237+0000","last_active":"2023-10-16T12:34:44.917513+0000","last_peered":"2023-10-16T12:34:44.917513+0000","last_clean":"2023-10-16T12:34:44.917513+0000","last_became_active":"2023-10-16T12:34:43.937101+0000","last_became_peered":"2023-10-16T12:34:43.937101+0000","last_unstale":"2023-10-16T12:34:44.917513+0000","last_undegraded":"2023-10-16T12:34:44.917513+0000","last_fullsized":"2023-10-16T12:34:44.917513+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:32.382347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923926+0000","last_change":"2023-10-16T12:34:42.900482+0000","last_active":"2023-10-16T12:34:44.923926+0000","last_peered":"2023-10-16T12:34:44.923926+0000","last_clean":"2023-10-16T12:34:44.923926+0000","last_became_active":"2023-10-16T12:34:42.900088+0000","last_became_peered":"2023-10-16T12:34:42.900088+0000","last_unstale":"2023-10-16T12:34:44.923926+0000","last_undegraded":"2023-10-16T12:34:44.923926+0000","last_fullsized":"2023-10-16T12:34:44.923926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:17.358016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948552+0000","last_change":"2023-10-16T12:34:43.929644+0000","last_active":"2023-10-16T12:34:44.948552+0000","last_peered":"2023-10-16T12:34:44.948552+0000","last_clean":"2023-10-16T12:34:44.948552+0000","last_became_active":"2023-10-16T12:34:43.929346+0000","last_became_peered":"2023-10-16T12:34:43.929346+0000","last_unstale":"2023-10-16T12:34:44.948552+0000","last_undegraded":"2023-10-16T12:34:44.948552+0000","last_fullsized":"2023-10-16T12:34:44.948552+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:01:23.152173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.923405+0000","last_change":"2023-10-16T12:34:42.908083+0000","last_active":"2023-10-16T12:34:44.923405+0000","last_peered":"2023-10-16T12:34:44.923405+0000","last_clean":"2023-10-16T12:34:44.923405+0000","last_became_active":"2023-10-16T12:34:42.907812+0000","last_became_peered":"2023-10-16T12:34:42.907812+0000","last_unstale":"2023-10-16T12:34:44.923405+0000","last_undegraded":"2023-10-16T12:34:44.923405+0000","last_fullsized":"2023-10-16T12:34:44.923405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:51.182438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948283+0000","last_change":"2023-10-16T12:34:43.931488+0000","last_active":"2023-10-16T12:34:44.948283+0000","last_peered":"2023-10-16T12:34:44.948283+0000","last_clean":"2023-10-16T12:34:44.948283+0000","last_became_active":"2023-10-16T12:34:43.931129+0000","last_became_peered":"2023-10-16T12:34:43.931129+0000","last_unstale":"2023-10-16T12:34:44.948283+0000","last_undegraded":"2023-10-16T12:34:44.948283+0000","last_fullsized":"2023-10-16T12:34:44.948283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:25.750031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951513+0000","last_change":"2023-10-16T12:34:42.897386+0000","last_active":"2023-10-16T12:34:44.951513+0000","last_peered":"2023-10-16T12:34:44.951513+0000","last_clean":"2023-10-16T12:34:44.951513+0000","last_became_active":"2023-10-16T12:34:42.897078+0000","last_became_peered":"2023-10-16T12:34:42.897078+0000","last_unstale":"2023-10-16T12:34:44.951513+0000","last_undegraded":"2023-10-16T12:34:44.951513+0000","last_fullsized":"2023-10-16T12:34:44.951513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:09.153738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948974+0000","last_change":"2023-10-16T12:34:43.930074+0000","last_active":"2023-10-16T12:34:44.948974+0000","last_peered":"2023-10-16T12:34:44.948974+0000","last_clean":"2023-10-16T12:34:44.948974+0000","last_became_active":"2023-10-16T12:34:43.929115+0000","last_became_peered":"2023-10-16T12:34:43.929115+0000","last_unstale":"2023-10-16T12:34:44.948974+0000","last_undegraded":"2023-10-16T12:34:44.948974+0000","last_fullsized":"2023-10-16T12:34:44.948974+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:36.187438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948876+0000","last_change":"2023-10-16T12:34:42.913016+0000","last_active":"2023-10-16T12:34:44.948876+0000","last_peered":"2023-10-16T12:34:44.948876+0000","last_clean":"2023-10-16T12:34:44.948876+0000","last_became_active":"2023-10-16T12:34:42.911685+0000","last_became_peered":"2023-10-16T12:34:42.911685+0000","last_unstale":"2023-10-16T12:34:44.948876+0000","last_undegraded":"2023-10-16T12:34:44.948876+0000","last_fullsized":"2023-10-16T12:34:44.948876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:51:14.752734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948168+0000","last_change":"2023-10-16T12:34:43.928322+0000","last_active":"2023-10-16T12:34:44.948168+0000","last_peered":"2023-10-16T12:34:44.948168+0000","last_clean":"2023-10-16T12:34:44.948168+0000","last_became_active":"2023-10-16T12:34:43.928060+0000","last_became_peered":"2023-10-16T12:34:43.928060+0000","last_unstale":"2023-10-16T12:34:44.948168+0000","last_undegraded":"2023-10-16T12:34:44.948168+0000","last_fullsized":"2023-10-16T12:34:44.948168+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:12.148748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921728+0000","last_change":"2023-10-16T12:34:42.907306+0000","last_active":"2023-10-16T12:34:44.921728+0000","last_peered":"2023-10-16T12:34:44.921728+0000","last_clean":"2023-10-16T12:34:44.921728+0000","last_became_active":"2023-10-16T12:34:42.906994+0000","last_became_peered":"2023-10-16T12:34:42.906994+0000","last_unstale":"2023-10-16T12:34:44.921728+0000","last_undegraded":"2023-10-16T12:34:44.921728+0000","last_fullsized":"2023-10-16T12:34:44.921728+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:38.379882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950064+0000","last_change":"2023-10-16T12:34:43.915114+0000","last_active":"2023-10-16T12:34:44.950064+0000","last_peered":"2023-10-16T12:34:44.950064+0000","last_clean":"2023-10-16T12:34:44.950064+0000","last_became_active":"2023-10-16T12:34:43.911672+0000","last_became_peered":"2023-10-16T12:34:43.911672+0000","last_unstale":"2023-10-16T12:34:44.950064+0000","last_undegraded":"2023-10-16T12:34:44.950064+0000","last_fullsized":"2023-10-16T12:34:44.950064+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:07.978483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.920762+0000","last_change":"2023-10-16T12:34:42.902063+0000","last_active":"2023-10-16T12:34:44.920762+0000","last_peered":"2023-10-16T12:34:44.920762+0000","last_clean":"2023-10-16T12:34:44.920762+0000","last_became_active":"2023-10-16T12:34:42.901765+0000","last_became_peered":"2023-10-16T12:34:42.901765+0000","last_unstale":"2023-10-16T12:34:44.920762+0000","last_undegraded":"2023-10-16T12:34:44.920762+0000","last_fullsized":"2023-10-16T12:34:44.920762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:22.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948743+0000","last_change":"2023-10-16T12:34:43.929127+0000","last_active":"2023-10-16T12:34:44.948743+0000","last_peered":"2023-10-16T12:34:44.948743+0000","last_clean":"2023-10-16T12:34:44.948743+0000","last_became_active":"2023-10-16T12:34:43.928794+0000","last_became_peered":"2023-10-16T12:34:43.928794+0000","last_unstale":"2023-10-16T12:34:44.948743+0000","last_undegraded":"2023-10-16T12:34:44.948743+0000","last_fullsized":"2023-10-16T12:34:44.948743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:51.065728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.918156+0000","last_change":"2023-10-16T12:34:42.903231+0000","last_active":"2023-10-16T12:34:44.918156+0000","last_peered":"2023-10-16T12:34:44.918156+0000","last_clean":"2023-10-16T12:34:44.918156+0000","last_became_active":"2023-10-16T12:34:42.902913+0000","last_became_peered":"2023-10-16T12:34:42.902913+0000","last_unstale":"2023-10-16T12:34:44.918156+0000","last_undegraded":"2023-10-16T12:34:44.918156+0000","last_fullsized":"2023-10-16T12:34:44.918156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:09.153738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922485+0000","last_change":"2023-10-16T12:34:43.935703+0000","last_active":"2023-10-16T12:34:44.922485+0000","last_peered":"2023-10-16T12:34:44.922485+0000","last_clean":"2023-10-16T12:34:44.922485+0000","last_became_active":"2023-10-16T12:34:43.935344+0000","last_became_peered":"2023-10-16T12:34:43.935344+0000","last_unstale":"2023-10-16T12:34:44.922485+0000","last_undegraded":"2023-10-16T12:34:44.922485+0000","last_fullsized":"2023-10-16T12:34:44.922485+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.952184+0000","last_change":"2023-10-16T12:34:42.897264+0000","last_active":"2023-10-16T12:34:44.952184+0000","last_peered":"2023-10-16T12:34:44.952184+0000","last_clean":"2023-10-16T12:34:44.952184+0000","last_became_active":"2023-10-16T12:34:42.896857+0000","last_became_peered":"2023-10-16T12:34:42.896857+0000","last_unstale":"2023-10-16T12:34:44.952184+0000","last_undegraded":"2023-10-16T12:34:44.952184+0000","last_fullsized":"2023-10-16T12:34:44.952184+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:38.379882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951630+0000","last_change":"2023-10-16T12:34:43.926542+0000","last_active":"2023-10-16T12:34:44.951630+0000","last_peered":"2023-10-16T12:34:44.951630+0000","last_clean":"2023-10-16T12:34:44.951630+0000","last_became_active":"2023-10-16T12:34:43.926259+0000","last_became_peered":"2023-10-16T12:34:43.926259+0000","last_unstale":"2023-10-16T12:34:44.951630+0000","last_undegraded":"2023-10-16T12:34:44.951630+0000","last_fullsized":"2023-10-16T12:34:44.951630+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:50.157019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917604+0000","last_change":"2023-10-16T12:34:42.899972+0000","last_active":"2023-10-16T12:34:44.917604+0000","last_peered":"2023-10-16T12:34:44.917604+0000","last_clean":"2023-10-16T12:34:44.917604+0000","last_became_active":"2023-10-16T12:34:42.899536+0000","last_became_peered":"2023-10-16T12:34:42.899536+0000","last_unstale":"2023-10-16T12:34:44.917604+0000","last_undegraded":"2023-10-16T12:34:44.917604+0000","last_fullsized":"2023-10-16T12:34:44.917604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:06.743408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948651+0000","last_change":"2023-10-16T12:34:43.926988+0000","last_active":"2023-10-16T12:34:44.948651+0000","last_peered":"2023-10-16T12:34:44.948651+0000","last_clean":"2023-10-16T12:34:44.948651+0000","last_became_active":"2023-10-16T12:34:43.925046+0000","last_became_peered":"2023-10-16T12:34:43.925046+0000","last_unstale":"2023-10-16T12:34:44.948651+0000","last_undegraded":"2023-10-16T12:34:44.948651+0000","last_fullsized":"2023-10-16T12:34:44.948651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:19.266077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951701+0000","last_change":"2023-10-16T12:34:42.903567+0000","last_active":"2023-10-16T12:34:44.951701+0000","last_peered":"2023-10-16T12:34:44.951701+0000","last_clean":"2023-10-16T12:34:44.951701+0000","last_became_active":"2023-10-16T12:34:42.901990+0000","last_became_peered":"2023-10-16T12:34:42.901990+0000","last_unstale":"2023-10-16T12:34:44.951701+0000","last_undegraded":"2023-10-16T12:34:44.951701+0000","last_fullsized":"2023-10-16T12:34:44.951701+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:26:56.384420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.951633+0000","last_change":"2023-10-16T12:34:43.928127+0000","last_active":"2023-10-16T12:34:44.951633+0000","last_peered":"2023-10-16T12:34:44.951633+0000","last_clean":"2023-10-16T12:34:44.951633+0000","last_became_active":"2023-10-16T12:34:43.927718+0000","last_became_peered":"2023-10-16T12:34:43.927718+0000","last_unstale":"2023-10-16T12:34:44.951633+0000","last_undegraded":"2023-10-16T12:34:44.951633+0000","last_fullsized":"2023-10-16T12:34:44.951633+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:42.104379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.917101+0000","last_change":"2023-10-16T12:34:42.903353+0000","last_active":"2023-10-16T12:34:44.917101+0000","last_peered":"2023-10-16T12:34:44.917101+0000","last_clean":"2023-10-16T12:34:44.917101+0000","last_became_active":"2023-10-16T12:34:42.903040+0000","last_became_peered":"2023-10-16T12:34:42.903040+0000","last_unstale":"2023-10-16T12:34:44.917101+0000","last_undegraded":"2023-10-16T12:34:44.917101+0000","last_fullsized":"2023-10-16T12:34:44.917101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:49.142884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.947842+0000","last_change":"2023-10-16T12:34:43.926671+0000","last_active":"2023-10-16T12:34:44.947842+0000","last_peered":"2023-10-16T12:34:44.947842+0000","last_clean":"2023-10-16T12:34:44.947842+0000","last_became_active":"2023-10-16T12:34:43.926317+0000","last_became_peered":"2023-10-16T12:34:43.926317+0000","last_unstale":"2023-10-16T12:34:44.947842+0000","last_undegraded":"2023-10-16T12:34:44.947842+0000","last_fullsized":"2023-10-16T12:34:44.947842+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.771260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.922233+0000","last_change":"2023-10-16T12:34:42.902999+0000","last_active":"2023-10-16T12:34:44.922233+0000","last_peered":"2023-10-16T12:34:44.922233+0000","last_clean":"2023-10-16T12:34:44.922233+0000","last_became_active":"2023-10-16T12:34:42.902709+0000","last_became_peered":"2023-10-16T12:34:42.902709+0000","last_unstale":"2023-10-16T12:34:44.922233+0000","last_undegraded":"2023-10-16T12:34:44.922233+0000","last_fullsized":"2023-10-16T12:34:44.922233+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:09.153738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.921460+0000","last_change":"2023-10-16T12:34:43.927931+0000","last_active":"2023-10-16T12:34:44.921460+0000","last_peered":"2023-10-16T12:34:44.921460+0000","last_clean":"2023-10-16T12:34:44.921460+0000","last_became_active":"2023-10-16T12:34:43.927627+0000","last_became_peered":"2023-10-16T12:34:43.927627+0000","last_unstale":"2023-10-16T12:34:44.921460+0000","last_undegraded":"2023-10-16T12:34:44.921460+0000","last_fullsized":"2023-10-16T12:34:44.921460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:07.757544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948018+0000","last_change":"2023-10-16T12:34:42.902828+0000","last_active":"2023-10-16T12:34:44.948018+0000","last_peered":"2023-10-16T12:34:44.948018+0000","last_clean":"2023-10-16T12:34:44.948018+0000","last_became_active":"2023-10-16T12:34:42.902549+0000","last_became_peered":"2023-10-16T12:34:42.902549+0000","last_unstale":"2023-10-16T12:34:44.948018+0000","last_undegraded":"2023-10-16T12:34:44.948018+0000","last_fullsized":"2023-10-16T12:34:44.948018+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:27.563110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.950806+0000","last_change":"2023-10-16T12:34:43.928337+0000","last_active":"2023-10-16T12:34:44.950806+0000","last_peered":"2023-10-16T12:34:44.950806+0000","last_clean":"2023-10-16T12:34:44.950806+0000","last_became_active":"2023-10-16T12:34:43.927812+0000","last_became_peered":"2023-10-16T12:34:43.927812+0000","last_unstale":"2023-10-16T12:34:44.950806+0000","last_undegraded":"2023-10-16T12:34:44.950806+0000","last_fullsized":"2023-10-16T12:34:44.950806+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:42.862929+0000","last_clean_scrub_stamp":"2023-10-16T12:34:42.862929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.835949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-16T12:34:44.948784+0000","last_change":"2023-10-16T12:34:42.913894+0000","last_active":"2023-10-16T12:34:44.948784+0000","last_peered":"2023-10-16T12:34:44.948784+0000","last_clean":"2023-10-16T12:34:44.948784+0000","last_became_active":"2023-10-16T12:34:42.913535+0000","last_became_peered":"2023-10-16T12:34:42.913535+0000","last_unstale":"2023-10-16T12:34:44.948784+0000","last_undegraded":"2023-10-16T12:34:44.948784+0000","last_fullsized":"2023-10-16T12:34:44.948784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:34:41.848794+0000","last_clean_scrub_stamp":"2023-10-16T12:34:41.848794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:52.019635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-10-16T12:34:43.903418+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:34:43.903418+0000","last_peered":"2023-10-16T12:34:43.903418+0000","last_clean":"2023-10-16T12:34:43.903418+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:34:43.903418+0000","last_undegraded":"2023-10-16T12:34:43.903418+0000","last_fullsized":"2023-10-16T12:34:43.903418+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":1003520,"data_stored":750474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.399},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.5600000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.458}]},{"osd":2,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.115},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.639}]},{"osd":3,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.587},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.714}]},{"osd":4,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.242},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.5}]},{"osd":5,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.675}]},{"osd":6,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":1.75},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":2.1379999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":160632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.306},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.028}]},{"osd":1,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.397}]},{"osd":2,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.554},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.2}]},{"osd":3,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.446}]},{"osd":4,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.498}]},{"osd":5,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.363,"15min":1.363},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.5230000000000001,"5min":2.5230000000000001,"15min":2.5230000000000001},"last":1.7290000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":1.363},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.5209999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":160688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.234},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":1.127}]},{"osd":1,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.21,"15min":1.21},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.5960000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.302},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.75700000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.95699999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.7909999999999999,"5min":2.7909999999999999,"15min":2.7909999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.611,"15min":1.611},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":5.3250000000000002,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":1.855}]},{"osd":6,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.484},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.4119999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":5.1269999999999998,"5min":5.1269999999999998,"15min":5.1269999999999998},"last":1.806},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":1.6539999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":413696,"data_stored":160120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":1.008}]},{"osd":2,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.3240000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.236},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.95099999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":7.9000000000000004,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":1.272},{"interface":"front","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":7.3460000000000001,"5min":7.3460000000000001,"15min":7.3460000000000001},"last":1.3500000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.411,"15min":1.411},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.365}]},{"osd":7,"last update":"Mon Oct 16 12:34:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.302},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.5289999999999999,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.159}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":161650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.0740000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.28},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.3959999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.4610000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":3.6930000000000001,"5min":3.6930000000000001,"15min":3.6930000000000001},"last":1.514}]},{"osd":5,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":3.8300000000000001,"5min":3.8300000000000001,"15min":3.8300000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.5569999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":3.9809999999999999},"last":1.708}]},{"osd":7,"last update":"Mon Oct 16 12:34:21 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.772},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.756}]}]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":160086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.74299999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.036},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.323}]},{"osd":3,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":1.099}]},{"osd":4,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.244},{"interface":"front","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.5549999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.649},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.2809999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.411},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.488,"5min":2.488,"15min":2.488},"last":1.381}]},{"osd":7,"last update":"Mon Oct 16 12:34:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.452},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":2.6429999999999998,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":1.4930000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":161650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.165}]},{"osd":2,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.8100000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.462}]},{"osd":4,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":3.0659999999999998,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.9630000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":1.609},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.5149999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":2.9350000000000001,"5min":2.9350000000000001,"15min":2.9350000000000001},"last":2.1280000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.7470000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:34:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.516,"5min":1.516,"15min":1.516},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.903}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":750384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.3420000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.202},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.151},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.415,"15min":1.415},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":4.726,"5min":4.726,"15min":4.726},"last":1.262},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":4.843,"5min":4.843,"15min":4.843},"last":1.4359999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":4.9059999999999997,"5min":4.9059999999999997,"15min":4.9059999999999997},"last":1.3979999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.296},{"interface":"front","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.2270000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:34:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.3969999999999998,"5min":2.3969999999999998,"15min":2.3969999999999998},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.4730000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:34:48.976 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:34:48.977 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:34:49.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.255+0000 7f4c0b251640 1 Processor -- start 2023-10-16T12:34:49.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.256+0000 7f4c0b251640 1 -- start start 2023-10-16T12:34:49.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.257+0000 7f4c0b251640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4c04159f00 0x7f4c040b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.257+0000 7f4c0b251640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4c0415d8c0 0x7f4c040b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.257+0000 7f4c0b251640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0406f890 con 0x7f4c04159b20 2023-10-16T12:34:49.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.257+0000 7f4c0b251640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c04070d30 con 0x7f4c0415d8c0 2023-10-16T12:34:49.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.257+0000 7f4c0b251640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0406c650 con 0x7f4c04159f00 2023-10-16T12:34:49.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c097ee640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4c0415d8c0 0x7f4c040b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c08fed640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4c04159b20 0x7f4c0406bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39266/0 (socket says 172.21.15.60:39266) 2023-10-16T12:34:49.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c08fed640 1 -- 172.21.15.60:0/1372435404 learned_addr learned my addr 172.21.15.60:0/1372435404 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:49.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c03fff640 1 --2- 172.21.15.60:0/1372435404 >> v2:172.21.15.144:3300/0 conn(0x7f4c04159f00 0x7f4c040b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c097ee640 1 -- 172.21.15.60:0/1372435404 >> v2:172.21.15.144:3300/0 conn(0x7f4c04159f00 msgr2=0x7f4c040b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:49.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c097ee640 1 --2- 172.21.15.60:0/1372435404 >> v2:172.21.15.144:3300/0 conn(0x7f4c04159f00 0x7f4c040b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c097ee640 1 -- 172.21.15.60:0/1372435404 >> v1:172.21.15.60:6789/0 conn(0x7f4c04159b20 legacy=0x7f4c0406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c097ee640 1 -- 172.21.15.60:0/1372435404 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0406c930 con 0x7f4c0415d8c0 2023-10-16T12:34:49.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.258+0000 7f4c037fe640 1 -- 172.21.15.60:0/1372435404 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485629486 0 0) 0x7f4c0406f890 con 0x7f4c04159b20 2023-10-16T12:34:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.260+0000 7f4c097ee640 1 --2- 172.21.15.60:0/1372435404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c0415d8c0 0x7f4c040b0950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c040cc510 tx=0x7f4bfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cf987689070dc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.260+0000 7f4c037fe640 1 -- 172.21.15.60:0/1372435404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc031830 con 0x7f4c0415d8c0 2023-10-16T12:34:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.260+0000 7f4c037fe640 1 -- 172.21.15.60:0/1372435404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc031990 con 0x7f4c0415d8c0 2023-10-16T12:34:49.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.260+0000 7f4c037fe640 1 -- 172.21.15.60:0/1372435404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc031d20 con 0x7f4c0415d8c0 2023-10-16T12:34:49.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.260+0000 7f4c0b251640 1 -- 172.21.15.60:0/1372435404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c0415d8c0 msgr2=0x7f4c040b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.260+0000 7f4c0b251640 1 --2- 172.21.15.60:0/1372435404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c0415d8c0 0x7f4c040b0950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c040cc510 tx=0x7f4bfc02b8d0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.261+0000 7f4c0b251640 1 -- 172.21.15.60:0/1372435404 shutdown_connections 2023-10-16T12:34:49.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.261+0000 7f4c0b251640 1 --2- 172.21.15.60:0/1372435404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c0415d8c0 0x7f4c040b0950 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.261+0000 7f4c0b251640 1 --2- 172.21.15.60:0/1372435404 >> v2:172.21.15.144:3300/0 conn(0x7f4c04159f00 0x7f4c040b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.261+0000 7f4c0b251640 1 -- 172.21.15.60:0/1372435404 >> 172.21.15.60:0/1372435404 conn(0x7f4c040cb410 msgr2=0x7f4c040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:49.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.261+0000 7f4c0b251640 1 -- 172.21.15.60:0/1372435404 shutdown_connections 2023-10-16T12:34:49.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.261+0000 7f4c0b251640 1 -- 172.21.15.60:0/1372435404 wait complete. 2023-10-16T12:34:49.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.263+0000 7f4c0b251640 1 Processor -- start 2023-10-16T12:34:49.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.263+0000 7f4c0b251640 1 -- start start 2023-10-16T12:34:49.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c0b251640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 0x7f4c041f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c0b251640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04159f00 0x7f4c04157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c0b251640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415d8c0 0x7f4c0415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c0b251640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c041f3210 con 0x7f4c04159b20 2023-10-16T12:34:49.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c0b251640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c041f34f0 con 0x7f4c04159f00 2023-10-16T12:34:49.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c0b251640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c041f37d0 con 0x7f4c0415d8c0 2023-10-16T12:34:49.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 0x7f4c041f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 0x7f4c041f2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51538/0 (socket says 172.21.15.60:51538) 2023-10-16T12:34:49.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 -- 172.21.15.60:0/2260905328 learned_addr learned my addr 172.21.15.60:0/2260905328 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:49.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c03fff640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04159f00 0x7f4c04157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c097ee640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415d8c0 0x7f4c0415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 -- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415d8c0 msgr2=0x7f4c0415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415d8c0 0x7f4c0415c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 -- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04159f00 msgr2=0x7f4c04157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.264+0000 7f4c08fed640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04159f00 0x7f4c04157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c08fed640 1 -- 172.21.15.60:0/2260905328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc009400 con 0x7f4c04159b20 2023-10-16T12:34:49.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c08fed640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 0x7f4c041f2bb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400c4d0 tx=0x7f4bf400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:49.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf40074e0 con 0x7f4c04159b20 2023-10-16T12:34:49.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf4007660 con 0x7f4c04159b20 2023-10-16T12:34:49.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4004a20 con 0x7f4c04159b20 2023-10-16T12:34:49.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c04158470 con 0x7f4c04159b20 2023-10-16T12:34:49.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.265+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0415cd20 con 0x7f4c04159b20 2023-10-16T12:34:49.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.267+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0404e160 con 0x7f4c04159b20 2023-10-16T12:34:49.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.268+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4bf4004b80 con 0x7f4c04159b20 2023-10-16T12:34:49.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.269+0000 7f4c017fa640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4be406ae70 0x7f4be406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.269+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bf4088720 con 0x7f4c04159b20 2023-10-16T12:34:49.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.269+0000 7f4c03fff640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4be406ae70 0x7f4be406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.271+0000 7f4c03fff640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4be406ae70 0x7f4be406d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c040c6f80 tx=0x7f4bec003990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:49.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.273+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bf4053860 con 0x7f4c04159b20 2023-10-16T12:34:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.540+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4c040b10b0 con 0x7f4c04159b20 2023-10-16T12:34:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.544+0000 7f4c017fa640 1 -- 172.21.15.60:0/2260905328 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4bf40180e0 con 0x7f4c04159b20 2023-10-16T12:34:49.545 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:34:49.546 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:34:49.546 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:34:49.546 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:34:49.546 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:34:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:34:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.546+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4be406ae70 msgr2=0x7f4be406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.546+0000 7f4c0b251640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4be406ae70 0x7f4be406d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c040c6f80 tx=0x7f4bec003990 comp rx=0 tx=0).stop 2023-10-16T12:34:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.546+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 msgr2=0x7f4c041f2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.546+0000 7f4c0b251640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 0x7f4c041f2bb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400c4d0 tx=0x7f4bf400c990 comp rx=0 tx=0).stop 2023-10-16T12:34:49.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 shutdown_connections 2023-10-16T12:34:49.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f4be406ae70 0x7f4be406d330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c0415d8c0 0x7f4c0415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c04159f00 0x7f4c04157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 --2- 172.21.15.60:0/2260905328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c04159b20 0x7f4c041f2bb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 >> 172.21.15.60:0/2260905328 conn(0x7f4c040cb410 msgr2=0x7f4c040cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 shutdown_connections 2023-10-16T12:34:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.547+0000 7f4c0b251640 1 -- 172.21.15.60:0/2260905328 wait complete. 2023-10-16T12:34:49.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:34:49.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.800+0000 7fe6c1864640 1 Processor -- start 2023-10-16T12:34:49.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.800+0000 7fe6c1864640 1 -- start start 2023-10-16T12:34:49.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c1864640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6bc15b320 0x7fe6bc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c1864640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc15b9d0 con 0x7fe6bc06c040 2023-10-16T12:34:49.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c1864640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6bc15bcb0 con 0x7fe6bc0ad970 2023-10-16T12:34:49.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c1864640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6bc15b7d0 con 0x7fe6bc15b320 2023-10-16T12:34:49.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c0862640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe6bc06c040 0x7fe6bc06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39280/0 (socket says 172.21.15.60:39280) 2023-10-16T12:34:49.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c0862640 1 -- 172.21.15.60:0/3617444852 learned_addr learned my addr 172.21.15.60:0/3617444852 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:49.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.802+0000 7fe6c1063640 1 --2- 172.21.15.60:0/3617444852 >> v2:172.21.15.60:3300/0 conn(0x7fe6bc15b320 0x7fe6bc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.803+0000 7fe6bb7fe640 1 -- 172.21.15.60:0/3617444852 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096756750 0 0) 0x7fe6bc15b9d0 con 0x7fe6bc06c040 2023-10-16T12:34:49.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.803+0000 7fe6bb7fe640 1 -- 172.21.15.60:0/3617444852 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac003520 con 0x7fe6bc06c040 2023-10-16T12:34:49.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.803+0000 7fe6c1063640 1 -- 172.21.15.60:0/3617444852 >> v1:172.21.15.144:6789/0 conn(0x7fe6bc0ad970 legacy=0x7fe6bc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:49.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.803+0000 7fe6c1063640 1 -- 172.21.15.60:0/3617444852 >> v1:172.21.15.60:6789/0 conn(0x7fe6bc06c040 legacy=0x7fe6bc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.803+0000 7fe6c1063640 1 -- 172.21.15.60:0/3617444852 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc15bf90 con 0x7fe6bc15b320 2023-10-16T12:34:49.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.804+0000 7fe6c1063640 1 --2- 172.21.15.60:0/3617444852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0cd180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009dc0 tx=0x7fe6b402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b68b2377f810d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:49.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.804+0000 7fe6bb7fe640 1 -- 172.21.15.60:0/3617444852 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b402cc40 con 0x7fe6bc15b320 2023-10-16T12:34:49.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.804+0000 7fe6bb7fe640 1 -- 172.21.15.60:0/3617444852 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b402cda0 con 0x7fe6bc15b320 2023-10-16T12:34:49.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.804+0000 7fe6bb7fe640 1 -- 172.21.15.60:0/3617444852 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4034660 con 0x7fe6bc15b320 2023-10-16T12:34:49.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 -- 172.21.15.60:0/3617444852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc15b320 msgr2=0x7fe6bc0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3617444852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0cd180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009dc0 tx=0x7fe6b402be90 comp rx=0 tx=0).stop 2023-10-16T12:34:49.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 -- 172.21.15.60:0/3617444852 shutdown_connections 2023-10-16T12:34:49.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3617444852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0cd180 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 -- 172.21.15.60:0/3617444852 >> 172.21.15.60:0/3617444852 conn(0x7fe6bc0ce6a0 msgr2=0x7fe6bc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:49.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 -- 172.21.15.60:0/3617444852 shutdown_connections 2023-10-16T12:34:49.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.805+0000 7fe6c1864640 1 -- 172.21.15.60:0/3617444852 wait complete. 2023-10-16T12:34:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.807+0000 7fe6c1864640 1 Processor -- start 2023-10-16T12:34:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.807+0000 7fe6c1864640 1 -- start start 2023-10-16T12:34:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c1864640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 0x7fe6bc0b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c1864640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6bc0ad970 0x7fe6bc155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c1864640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c1864640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc156510 con 0x7fe6bc06c040 2023-10-16T12:34:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c1864640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc0c7ad0 con 0x7fe6bc15b320 2023-10-16T12:34:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c1864640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc0b1430 con 0x7fe6bc0ad970 2023-10-16T12:34:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.808+0000 7fe6c0862640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 0x7fe6bc0b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 0x7fe6bc0b1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51558/0 (socket says 172.21.15.60:51558) 2023-10-16T12:34:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 -- 172.21.15.60:0/3926757896 learned_addr learned my addr 172.21.15.60:0/3926757896 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6bbfff640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6bc0ad970 0x7fe6bc155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c1063640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 -- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6bc0ad970 msgr2=0x7fe6bc155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6bc0ad970 0x7fe6bc155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 -- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6bc15b320 msgr2=0x7fe6bc0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 -- 172.21.15.60:0/3926757896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b4009a70 con 0x7fe6bc06c040 2023-10-16T12:34:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.809+0000 7fe6c0862640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 0x7fe6bc0b1f10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c410 tx=0x7fe6b000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:49.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.810+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0005ce0 con 0x7fe6bc06c040 2023-10-16T12:34:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.810+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b0005e40 con 0x7fe6bc06c040 2023-10-16T12:34:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.810+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc0b1710 con 0x7fe6bc06c040 2023-10-16T12:34:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.810+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00072f0 con 0x7fe6bc06c040 2023-10-16T12:34:49.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.810+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6bc156e60 con 0x7fe6bc06c040 2023-10-16T12:34:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.811+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6bc06b330 con 0x7fe6bc06c040 2023-10-16T12:34:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.812+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe6b0007510 con 0x7fe6bc06c040 2023-10-16T12:34:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.813+0000 7fe6b97fa640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe6a006ae70 0x7fe6a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.813+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6b0088ca0 con 0x7fe6bc06c040 2023-10-16T12:34:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.813+0000 7fe6bbfff640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe6a006ae70 0x7fe6a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.815+0000 7fe6bbfff640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe6a006ae70 0x7fe6a006d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0c8770 tx=0x7fe6a8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:49.817+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6b0053e00 con 0x7fe6bc06c040 2023-10-16T12:34:50.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.058+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6bc0b1290 con 0x7fe6bc06c040 2023-10-16T12:34:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.059+0000 7fe6b97fa640 1 -- 172.21.15.60:0/3926757896 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fe6bc0b1290 con 0x7fe6bc06c040 2023-10-16T12:34:50.060 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:34:50.061 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:34:43.360823+0000","modified":"2023-10-16T12:34:46.440178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/3714850681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3714850681},{"type":"v1","addr":"172.21.15.60:6835","nonce":3714850681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:34:50.061 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2023-10-16T12:34:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe6a006ae70 msgr2=0x7fe6a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe6a006ae70 0x7fe6a006d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0c8770 tx=0x7fe6a8009040 comp rx=0 tx=0).stop 2023-10-16T12:34:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 msgr2=0x7fe6bc0b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 0x7fe6bc0b1f10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c410 tx=0x7fe6b000c8d0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 shutdown_connections 2023-10-16T12:34:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fe6a006ae70 0x7fe6a006d330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6bc0ad970 0x7fe6bc155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6bc15b320 0x7fe6bc0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 --2- 172.21.15.60:0/3926757896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6bc06c040 0x7fe6bc0b1f10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 >> 172.21.15.60:0/3926757896 conn(0x7fe6bc0ce6a0 msgr2=0x7fe6bc15a030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 shutdown_connections 2023-10-16T12:34:50.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.062+0000 7fe6c1864640 1 -- 172.21.15.60:0/3926757896 wait complete. 2023-10-16T12:34:50.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:34:43.360823+0000', 'modified': '2023-10-16T12:34:46.440178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5004}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5004': {'gid': 5004, 'name': 'c', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.60:6835/3714850681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6833', 'nonce': 3714850681}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 3714850681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:34:50.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:34:50.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T12:34:50.711 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:50.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.726+0000 7f1db3ee0640 1 Processor -- start 2023-10-16T12:34:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.726+0000 7f1db3ee0640 1 -- start start 2023-10-16T12:34:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db3ee0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1dac0caac0 0x7f1dac0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db3ee0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0cb030 con 0x7f1dac06a4b0 2023-10-16T12:34:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db3ee0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0c91c0 con 0x7f1dac15ad70 2023-10-16T12:34:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db3ee0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0c94a0 con 0x7f1dac0caac0 2023-10-16T12:34:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db147b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1dac06a4b0 0x7f1dac0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39284/0 (socket says 172.21.15.60:39284) 2023-10-16T12:34:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db147b640 1 -- 172.21.15.60:0/3911377497 learned_addr learned my addr 172.21.15.60:0/3911377497 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.728+0000 7f1db247d640 1 --2- 172.21.15.60:0/3911377497 >> v2:172.21.15.60:3300/0 conn(0x7f1dac0caac0 0x7f1dac0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.729+0000 7f1db0c7a640 1 -- 172.21.15.60:0/3911377497 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977389784 0 0) 0x7f1dac0cb030 con 0x7f1dac06a4b0 2023-10-16T12:34:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.729+0000 7f1db0c7a640 1 -- 172.21.15.60:0/3911377497 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d98003520 con 0x7f1dac06a4b0 2023-10-16T12:34:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.729+0000 7f1db247d640 1 -- 172.21.15.60:0/3911377497 >> v1:172.21.15.144:6789/0 conn(0x7f1dac15ad70 legacy=0x7f1dac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.729+0000 7f1db247d640 1 -- 172.21.15.60:0/3911377497 >> v1:172.21.15.60:6789/0 conn(0x7f1dac06a4b0 legacy=0x7f1dac0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.729+0000 7f1db247d640 1 -- 172.21.15.60:0/3911377497 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0c9780 con 0x7f1dac0caac0 2023-10-16T12:34:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.730+0000 7f1db247d640 1 --2- 172.21.15.60:0/3911377497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac0caac0 0x7f1dac0ad4d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009d90 tx=0x7f1da402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706c21a7547c2c7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.730+0000 7f1db0c7a640 1 -- 172.21.15.60:0/3911377497 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da402ccf0 con 0x7f1dac0caac0 2023-10-16T12:34:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.730+0000 7f1db0c7a640 1 -- 172.21.15.60:0/3911377497 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da402ce50 con 0x7f1dac0caac0 2023-10-16T12:34:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.730+0000 7f1db0c7a640 1 -- 172.21.15.60:0/3911377497 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4034660 con 0x7f1dac0caac0 2023-10-16T12:34:50.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 -- 172.21.15.60:0/3911377497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac0caac0 msgr2=0x7f1dac0ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/3911377497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac0caac0 0x7f1dac0ad4d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009d90 tx=0x7f1da402bf40 comp rx=0 tx=0).stop 2023-10-16T12:34:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 -- 172.21.15.60:0/3911377497 shutdown_connections 2023-10-16T12:34:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/3911377497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac0caac0 0x7f1dac0ad4d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 -- 172.21.15.60:0/3911377497 >> 172.21.15.60:0/3911377497 conn(0x7f1dac0ce9b0 msgr2=0x7f1dac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 -- 172.21.15.60:0/3911377497 shutdown_connections 2023-10-16T12:34:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.731+0000 7f1db3ee0640 1 -- 172.21.15.60:0/3911377497 wait complete. 2023-10-16T12:34:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.733+0000 7f1db3ee0640 1 Processor -- start 2023-10-16T12:34:50.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.733+0000 7f1db3ee0640 1 -- start start 2023-10-16T12:34:50.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db3ee0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dac06a4b0 0x7f1dac1f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db3ee0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1dac0caac0 0x7f1dac0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db3ee0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 0x7f1dac158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db3ee0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0c79e0 con 0x7f1dac15ad70 2023-10-16T12:34:50.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db3ee0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0c2640 con 0x7f1dac06a4b0 2023-10-16T12:34:50.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db3ee0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0c2900 con 0x7f1dac0caac0 2023-10-16T12:34:50.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.734+0000 7f1db247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 0x7f1dac158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 0x7f1dac158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51572/0 (socket says 172.21.15.60:51572) 2023-10-16T12:34:50.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 -- 172.21.15.60:0/2490239984 learned_addr learned my addr 172.21.15.60:0/2490239984 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:50.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db147b640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1dac0caac0 0x7f1dac0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dac06a4b0 0x7f1dac1f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 -- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1dac0caac0 msgr2=0x7f1dac0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1dac0caac0 0x7f1dac0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 -- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dac06a4b0 msgr2=0x7f1dac1f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dac06a4b0 0x7f1dac1f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.735+0000 7f1db247d640 1 -- 172.21.15.60:0/2490239984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009a70 con 0x7f1dac15ad70 2023-10-16T12:34:50.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.736+0000 7f1db247d640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 0x7f1dac158d50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1da4037040 tx=0x7f1da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.736+0000 7f1da27fc640 1 -- 172.21.15.60:0/2490239984 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da403d600 con 0x7f1dac15ad70 2023-10-16T12:34:50.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.736+0000 7f1da27fc640 1 -- 172.21.15.60:0/2490239984 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da403d760 con 0x7f1dac15ad70 2023-10-16T12:34:50.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.736+0000 7f1da27fc640 1 -- 172.21.15.60:0/2490239984 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da403db10 con 0x7f1dac15ad70 2023-10-16T12:34:50.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.736+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac0c2be0 con 0x7f1dac15ad70 2023-10-16T12:34:50.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.736+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac0c31a0 con 0x7f1dac15ad70 2023-10-16T12:34:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.738+0000 7f1da27fc640 1 -- 172.21.15.60:0/2490239984 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1da403b040 con 0x7f1dac15ad70 2023-10-16T12:34:50.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.740+0000 7f1da27fc640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1d9006ae70 0x7f1d9006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.740+0000 7f1da27fc640 1 -- 172.21.15.60:0/2490239984 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1da40abb00 con 0x7f1dac15ad70 2023-10-16T12:34:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.741+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1d9006ae70 0x7f1d9006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.741+0000 7f1db6a79740 1 Processor -- start 2023-10-16T12:34:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.741+0000 7f1db6a79740 1 -- start start 2023-10-16T12:34:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b60f80 0x5639f7b33e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1d9006ae70 0x7f1d9006d330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1dac1f5450 tx=0x7f1da8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b61360 0x5639f7b63d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 0x5639f7b64250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db6a79740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7a547a0 con 0x5639f7b343c0 2023-10-16T12:34:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7aa6740 con 0x5639f7b60f80 2023-10-16T12:34:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639f7aa3e30 con 0x5639f7b61360 2023-10-16T12:34:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 0x5639f7b64250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 0x5639f7b64250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51580/0 (socket says 172.21.15.60:51580) 2023-10-16T12:34:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db247d640 1 -- 172.21.15.60:0/3906828007 learned_addr learned my addr 172.21.15.60:0/3906828007 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:50.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db147b640 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b61360 0x5639f7b63d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.742+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b60f80 0x5639f7b33e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1db247d640 1 -- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b61360 msgr2=0x5639f7b63d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1db247d640 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b61360 0x5639f7b63d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1db247d640 1 -- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b60f80 msgr2=0x5639f7b33e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1db247d640 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b60f80 0x5639f7b33e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1db247d640 1 -- 172.21.15.60:0/3906828007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639f7a63910 con 0x5639f7b343c0 2023-10-16T12:34:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1db247d640 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 0x5639f7b64250 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1da407a9c0 tx=0x7f1da4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ada7b37903a82c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1d8f7fe640 1 -- 172.21.15.60:0/3906828007 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da407ead0 con 0x5639f7b343c0 2023-10-16T12:34:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1d8f7fe640 1 -- 172.21.15.60:0/3906828007 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4082c80 con 0x5639f7b343c0 2023-10-16T12:34:50.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.743+0000 7f1d8f7fe640 1 -- 172.21.15.60:0/3906828007 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4092120 con 0x5639f7b343c0 2023-10-16T12:34:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 -- 172.21.15.60:0/3906828007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 msgr2=0x5639f7b64250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 0x5639f7b64250 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1da407a9c0 tx=0x7f1da4009750 comp rx=0 tx=0).stop 2023-10-16T12:34:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 -- 172.21.15.60:0/3906828007 shutdown_connections 2023-10-16T12:34:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b61360 0x5639f7b63d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b343c0 0x5639f7b64250 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3906828007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b60f80 0x5639f7b33e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 -- 172.21.15.60:0/3906828007 >> 172.21.15.60:0/3906828007 conn(0x5639f7b36190 msgr2=0x5639f7b7ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 -- 172.21.15.60:0/3906828007 shutdown_connections 2023-10-16T12:34:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.744+0000 7f1db6a79740 1 -- 172.21.15.60:0/3906828007 wait complete. 2023-10-16T12:34:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.745+0000 7f1db6a79740 1 -- start start 2023-10-16T12:34:50.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.745+0000 7f1db6a79740 1 Processor -- start 2023-10-16T12:34:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b66760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b66ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b5f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7a9f720 con 0x5639f7b5ece0 2023-10-16T12:34:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7a3d970 con 0x5639f7b61360 2023-10-16T12:34:50.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639f7a3d610 con 0x5639f7b60f80 2023-10-16T12:34:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db6a79740 10 client.0 _ll_register_callbacks cb 0x5639f7a5f9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:34:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.746+0000 7f1db247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b5f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b5f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51592/0 (socket says 172.21.15.60:51592) 2023-10-16T12:34:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 -- 172.21.15.60:0/4167565834 learned_addr learned my addr 172.21.15.60:0/4167565834 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b66760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db147b640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b66ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 -- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 msgr2=0x5639f7b66760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b66760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 -- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 msgr2=0x5639f7b66ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b66ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 -- 172.21.15.60:0/4167565834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac0c31a0 con 0x5639f7b5ece0 2023-10-16T12:34:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db247d640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b5f160 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1da407a9c0 tx=0x7f1da4034f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52913ba82f8391da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:34:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1d8cff9640 1 -- 172.21.15.60:0/4167565834 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da40a9070 con 0x5639f7b5ece0 2023-10-16T12:34:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639f7ac6160 con 0x5639f7b5ece0 2023-10-16T12:34:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.747+0000 7f1d8cff9640 1 -- 172.21.15.60:0/4167565834 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1da407ead0 con 0x5639f7b5ece0 2023-10-16T12:34:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1d8cff9640 1 -- 172.21.15.60:0/4167565834 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1da403dc70 con 0x5639f7b5ece0 2023-10-16T12:34:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 10 client.5187 fetch_fsmap learned FSMap version 17 2023-10-16T12:34:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5639f7a9dda0 con 0x5639f7b5ece0 2023-10-16T12:34:50.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1d8cff9640 1 -- 172.21.15.60:0/4167565834 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5639f7ac6160 con 0x5639f7b5ece0 2023-10-16T12:34:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1d8cff9640 1 -- 172.21.15.60:0/4167565834 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1da408a070 con 0x5639f7b5ece0 2023-10-16T12:34:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 10 client.5187 fetch_fsmap finished waiting for FSMap version 17 2023-10-16T12:34:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 10 client.5187 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T12:34:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 20 client.5187 populate_metadata read hostname 'smithi060' 2023-10-16T12:34:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b5fb70 0x5639f7b9c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 4 client.5187 mds_command: new command op to 5004 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:34:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.748+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 --> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639f7a71cc0 con 0x5639f7b5fb70 2023-10-16T12:34:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.749+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b5fb70 0x5639f7b9c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.749+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b5fb70 0x5639f7b9c420 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0207c26e92256c server_cookie=c21a753825a4e1c7 in_seq=0 out_seq=0 2023-10-16T12:34:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.749+0000 7f1d8cff9640 10 client.5187 ms_handle_connect on v2:172.21.15.60:6833/3714850681 2023-10-16T12:34:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.750+0000 7f1d8cff9640 1 -- 172.21.15.60:0/4167565834 <== mds.0 v2:172.21.15.60:6833/3714850681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5639f7a71cc0 con 0x5639f7b5fb70 2023-10-16T12:34:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.750+0000 7f1d8cff9640 10 client.5187 handle_command_reply: tid=0 2023-10-16T12:34:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.750+0000 7f1db6a79740 1 client.5187 shutdown 2023-10-16T12:34:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 msgr2=0x5639f7b5f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b5f160 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1da407a9c0 tx=0x7f1da4034f90 comp rx=0 tx=0).stop 2023-10-16T12:34:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 shutdown_connections 2023-10-16T12:34:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b5fb70 0x5639f7b9c420 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b5f160 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b66ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 --2- 172.21.15.60:0/4167565834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b66760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1d8cff9640 0 client.5187 ms_handle_reset on v2:172.21.15.60:6833/3714850681 2023-10-16T12:34:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 >> 172.21.15.60:0/4167565834 conn(0x5639f7b343c0 msgr2=0x5639f7b37e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 shutdown_connections 2023-10-16T12:34:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 1 -- 172.21.15.60:0/4167565834 wait complete. 2023-10-16T12:34:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.751+0000 7f1db6a79740 20 client.5187 trim_cache size 0 max 16384 2023-10-16T12:34:50.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.768+0000 7f1db6a79740 1 Processor -- start 2023-10-16T12:34:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.768+0000 7f1db6a79740 1 -- start start 2023-10-16T12:34:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b37e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b32e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b33380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db6a79740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7b59830 con 0x5639f7b5ece0 2023-10-16T12:34:50.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7803f20 con 0x5639f7b61360 2023-10-16T12:34:50.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639f7b66dd0 con 0x5639f7b60f80 2023-10-16T12:34:50.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.769+0000 7f1db1c7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b37e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b37e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51600/0 (socket says 172.21.15.60:51600) 2023-10-16T12:34:50.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 -- 172.21.15.60:0/1457920653 learned_addr learned my addr 172.21.15.60:0/1457920653 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:50.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db147b640 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b32e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db247d640 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b33380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 -- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 msgr2=0x5639f7b32e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b32e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 -- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 msgr2=0x5639f7b33380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b33380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 -- 172.21.15.60:0/1457920653 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639f7a3cb80 con 0x5639f7b5ece0 2023-10-16T12:34:50.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.770+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b37e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1da80051e0 tx=0x7f1da8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e2100ccd926287 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.771+0000 7f1d8ffff640 1 -- 172.21.15.60:0/1457920653 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80013a0 con 0x5639f7b5ece0 2023-10-16T12:34:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.771+0000 7f1d8ffff640 1 -- 172.21.15.60:0/1457920653 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da80253d0 con 0x5639f7b5ece0 2023-10-16T12:34:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.771+0000 7f1d8ffff640 1 -- 172.21.15.60:0/1457920653 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8025540 con 0x5639f7b5ece0 2023-10-16T12:34:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.771+0000 7f1db6a79740 1 -- 172.21.15.60:0/1457920653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 msgr2=0x5639f7b37e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.771+0000 7f1db6a79740 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b37e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1da80051e0 tx=0x7f1da8012000 comp rx=0 tx=0).stop 2023-10-16T12:34:50.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 -- 172.21.15.60:0/1457920653 shutdown_connections 2023-10-16T12:34:50.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b60f80 0x5639f7b32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b33380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 --2- 172.21.15.60:0/1457920653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b5ece0 0x5639f7b37e10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 -- 172.21.15.60:0/1457920653 >> 172.21.15.60:0/1457920653 conn(0x5639f7b343c0 msgr2=0x5639f7b67bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 -- 172.21.15.60:0/1457920653 shutdown_connections 2023-10-16T12:34:50.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 -- 172.21.15.60:0/1457920653 wait complete. 2023-10-16T12:34:50.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.772+0000 7f1db6a79740 1 -- start start 2023-10-16T12:34:50.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.773+0000 7f1db6a79740 1 Processor -- start 2023-10-16T12:34:50.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b5ece0 0x5639f7be04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 0x5639f7be6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b9d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7be6500 con 0x5639f7b60f80 2023-10-16T12:34:50.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639f7b9dd60 con 0x5639f7b61360 2023-10-16T12:34:50.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639f7b33af0 con 0x5639f7b5ece0 2023-10-16T12:34:50.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db6a79740 10 client.0 _ll_register_callbacks cb 0x5639f7b34310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:34:50.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db147b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 0x5639f7be6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db147b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 0x5639f7be6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51616/0 (socket says 172.21.15.60:51616) 2023-10-16T12:34:50.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db147b640 1 -- 172.21.15.60:0/3174071619 learned_addr learned my addr 172.21.15.60:0/3174071619 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:50.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b5ece0 0x5639f7be04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.774+0000 7f1db247d640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b9d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db147b640 1 -- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b5ece0 msgr2=0x5639f7be04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db147b640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b5ece0 0x5639f7be04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db147b640 1 -- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 msgr2=0x5639f7b9d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db147b640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b9d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db147b640 1 -- 172.21.15.60:0/3174071619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da801d040 con 0x5639f7b60f80 2023-10-16T12:34:50.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db147b640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 0x5639f7be6f10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0011e0 tx=0x7f1d9c0016a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffd3fb3b6ce9d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:50.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:34:50.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1d8cff9640 1 -- 172.21.15.60:0/3174071619 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c006bd0 con 0x5639f7b60f80 2023-10-16T12:34:50.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1d8cff9640 1 -- 172.21.15.60:0/3174071619 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d9c017640 con 0x5639f7b60f80 2023-10-16T12:34:50.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.775+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1da4082c80 con 0x5639f7b60f80 2023-10-16T12:34:50.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1d8cff9640 1 -- 172.21.15.60:0/3174071619 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d9c017800 con 0x5639f7b60f80 2023-10-16T12:34:50.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 10 client.5193 fetch_fsmap learned FSMap version 17 2023-10-16T12:34:50.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5639f7a62170 con 0x5639f7b60f80 2023-10-16T12:34:50.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1d8cff9640 1 -- 172.21.15.60:0/3174071619 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4082c80 con 0x5639f7b60f80 2023-10-16T12:34:50.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1d8cff9640 1 -- 172.21.15.60:0/3174071619 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1d9c012030 con 0x5639f7b60f80 2023-10-16T12:34:50.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 10 client.5193 fetch_fsmap finished waiting for FSMap version 17 2023-10-16T12:34:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 10 client.5193 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T12:34:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 20 client.5193 populate_metadata read hostname 'smithi060' 2023-10-16T12:34:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b47db0 0x5639f7b48190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 4 client.5193 mds_command: new command op to 5004 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:34:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 --> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5639f7a98700 con 0x5639f7b47db0 2023-10-16T12:34:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.776+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b47db0 0x5639f7b48190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1db1c7c640 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b47db0 0x5639f7b48190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d27e735e8d9a1d38 server_cookie=9077bc8f881c4fb6 in_seq=0 out_seq=0 2023-10-16T12:34:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1d8cff9640 10 client.5193 ms_handle_connect on v2:172.21.15.60:6833/3714850681 2023-10-16T12:34:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1d8cff9640 1 -- 172.21.15.60:0/3174071619 <== mds.0 v2:172.21.15.60:6833/3714850681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5639f7a98700 con 0x5639f7b47db0 2023-10-16T12:34:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1d8cff9640 10 client.5193 handle_command_reply: tid=0 2023-10-16T12:34:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1db6a79740 1 client.5193 shutdown 2023-10-16T12:34:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 msgr2=0x5639f7be6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.777+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 0x5639f7be6f10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0011e0 tx=0x7f1d9c0016a0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 shutdown_connections 2023-10-16T12:34:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:6833/3714850681,v1:172.21.15.60:6835/3714850681] conn(0x5639f7b47db0 0x5639f7b48190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5639f7b61360 0x5639f7b9d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5639f7b60f80 0x5639f7be6f10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1d8cff9640 0 client.5193 ms_handle_reset on v2:172.21.15.60:6833/3714850681 2023-10-16T12:34:50.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 --2- 172.21.15.60:0/3174071619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5639f7b5ece0 0x5639f7be04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 >> 172.21.15.60:0/3174071619 conn(0x5639f7b343c0 msgr2=0x5639f7b91450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 shutdown_connections 2023-10-16T12:34:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 1 -- 172.21.15.60:0/3174071619 wait complete. 2023-10-16T12:34:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.778+0000 7f1db6a79740 20 client.5193 trim_cache size 0 max 16384 2023-10-16T12:34:50.791 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:34:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:34:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:34:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 5004, 2023-10-16T12:34:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:34:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:34:50.792 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:34:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8616357560000001, 2023-10-16T12:34:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 17, 2023-10-16T12:34:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 36, 2023-10-16T12:34:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 36, 2023-10-16T12:34:50.793 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 22.868082579999999 2023-10-16T12:34:50.794 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:34:50.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1d9006ae70 msgr2=0x7f1d9006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1d9006ae70 0x7f1d9006d330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1dac1f5450 tx=0x7f1da8009040 comp rx=0 tx=0).stop 2023-10-16T12:34:50.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 msgr2=0x7f1dac158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:50.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 0x7f1dac158d50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1da4037040 tx=0x7f1da4004a40 comp rx=0 tx=0).stop 2023-10-16T12:34:50.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 shutdown_connections 2023-10-16T12:34:50.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f1d9006ae70 0x7f1d9006d330 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1dac0caac0 0x7f1dac0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dac15ad70 0x7f1dac158d50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 --2- 172.21.15.60:0/2490239984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dac06a4b0 0x7f1dac1f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:50.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 >> 172.21.15.60:0/2490239984 conn(0x7f1dac0ce9b0 msgr2=0x7f1dac0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:50.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 shutdown_connections 2023-10-16T12:34:50.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:50.779+0000 7f1db3ee0640 1 -- 172.21.15.60:0/2490239984 wait complete. 2023-10-16T12:34:50.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:34:51.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.066+0000 7f3678820640 1 Processor -- start 2023-10-16T12:34:51.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.067+0000 7f3678820640 1 -- start start 2023-10-16T12:34:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.068+0000 7f3678820640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f367415a9d0 0x7f3674071c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.068+0000 7f3678820640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36740b2360 0x7f36740b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.068+0000 7f3678820640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3674160b30 0x7f3674160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.068+0000 7f3678820640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36740d0770 con 0x7f36740b2360 2023-10-16T12:34:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.068+0000 7f3678820640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36740b0050 con 0x7f3674160b30 2023-10-16T12:34:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.068+0000 7f3678820640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36740b2d00 con 0x7f367415a9d0 2023-10-16T12:34:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36740b2360 0x7f36740b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36740b2360 0x7f36740b27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51620/0 (socket says 172.21.15.60:51620) 2023-10-16T12:34:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 -- 172.21.15.60:0/2500913492 learned_addr learned my addr 172.21.15.60:0/2500913492 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:51.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 -- 172.21.15.60:0/2500913492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f367415a9d0 msgr2=0x7f3674071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:51.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367359e640 1 --2- 172.21.15.60:0/2500913492 >> v2:172.21.15.144:3300/0 conn(0x7f3674160b30 0x7f3674160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f3672d9d640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f367415a9d0 0x7f3674071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f367415a9d0 0x7f3674071c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 -- 172.21.15.60:0/2500913492 >> v2:172.21.15.144:3300/0 conn(0x7f3674160b30 msgr2=0x7f3674160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 --2- 172.21.15.60:0/2500913492 >> v2:172.21.15.144:3300/0 conn(0x7f3674160b30 0x7f3674160fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367259c640 1 -- 172.21.15.60:0/2500913492 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674072170 con 0x7f36740b2360 2023-10-16T12:34:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f367359e640 1 --2- 172.21.15.60:0/2500913492 >> v2:172.21.15.144:3300/0 conn(0x7f3674160b30 0x7f3674160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.069+0000 7f3672d9d640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f367415a9d0 0x7f3674071c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.070+0000 7f367259c640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36740b2360 0x7f36740b27c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3674071720 tx=0x7f366802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5e4d7cfeeb25f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.071+0000 7f3671d9b640 1 -- 172.21.15.60:0/2500913492 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680317e0 con 0x7f36740b2360 2023-10-16T12:34:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.071+0000 7f3671d9b640 1 -- 172.21.15.60:0/2500913492 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668031940 con 0x7f36740b2360 2023-10-16T12:34:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.071+0000 7f3671d9b640 1 -- 172.21.15.60:0/2500913492 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668031cd0 con 0x7f36740b2360 2023-10-16T12:34:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.071+0000 7f3678820640 1 -- 172.21.15.60:0/2500913492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36740b2360 msgr2=0x7f36740b27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.071+0000 7f3678820640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36740b2360 0x7f36740b27c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3674071720 tx=0x7f366802b880 comp rx=0 tx=0).stop 2023-10-16T12:34:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 -- 172.21.15.60:0/2500913492 shutdown_connections 2023-10-16T12:34:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 --2- 172.21.15.60:0/2500913492 >> v2:172.21.15.144:3300/0 conn(0x7f3674160b30 0x7f3674160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36740b2360 0x7f36740b27c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 --2- 172.21.15.60:0/2500913492 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f367415a9d0 0x7f3674071c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 -- 172.21.15.60:0/2500913492 >> 172.21.15.60:0/2500913492 conn(0x7f367406d560 msgr2=0x7f367406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:51.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 -- 172.21.15.60:0/2500913492 shutdown_connections 2023-10-16T12:34:51.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.072+0000 7f3678820640 1 -- 172.21.15.60:0/2500913492 wait complete. 2023-10-16T12:34:51.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.074+0000 7f3678820640 1 Processor -- start 2023-10-16T12:34:51.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.074+0000 7f3678820640 1 -- start start 2023-10-16T12:34:51.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3678820640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36740b2360 0x7f36740b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3678820640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 0x7f3674156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3678820640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3674160b30 0x7f3674157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3678820640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36740b6320 con 0x7f367415a9d0 2023-10-16T12:34:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3678820640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36740c7b50 con 0x7f3674160b30 2023-10-16T12:34:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3678820640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36740b5d60 con 0x7f36740b2360 2023-10-16T12:34:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f367259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 0x7f3674156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f367259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 0x7f3674156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51636/0 (socket says 172.21.15.60:51636) 2023-10-16T12:34:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f367259c640 1 -- 172.21.15.60:0/1369557029 learned_addr learned my addr 172.21.15.60:0/1369557029 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f367359e640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3674160b30 0x7f3674157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.075+0000 7f3672d9d640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36740b2360 0x7f36740b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367259c640 1 -- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36740b2360 msgr2=0x7f36740b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367259c640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36740b2360 0x7f36740b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367259c640 1 -- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3674160b30 msgr2=0x7f3674157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367259c640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3674160b30 0x7f3674157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367259c640 1 -- 172.21.15.60:0/1369557029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36680094d0 con 0x7f367415a9d0 2023-10-16T12:34:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367359e640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3674160b30 0x7f3674157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f3672d9d640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36740b2360 0x7f36740b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f367259c640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 0x7f3674156be0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f366802b8c0 tx=0x7f36680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680329c0 con 0x7f367415a9d0 2023-10-16T12:34:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668032b40 con 0x7f367415a9d0 2023-10-16T12:34:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680453f0 con 0x7f367415a9d0 2023-10-16T12:34:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.076+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36740b6040 con 0x7f367415a9d0 2023-10-16T12:34:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.077+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36740b11f0 con 0x7f367415a9d0 2023-10-16T12:34:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.078+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36740b1440 con 0x7f367415a9d0 2023-10-16T12:34:51.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.079+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3668045550 con 0x7f367415a9d0 2023-10-16T12:34:51.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.080+0000 7f365b7fe640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f366006ae70 0x7f366006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:51.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.080+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36680aba70 con 0x7f367415a9d0 2023-10-16T12:34:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.080+0000 7f3672d9d640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f366006ae70 0x7f366006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.081+0000 7f3672d9d640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f366006ae70 0x7f366006d330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3664008fc0 tx=0x7f3664008510 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.084+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3668076c30 con 0x7f367415a9d0 2023-10-16T12:34:51.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.320+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f36740c8030 con 0x7f367415a9d0 2023-10-16T12:34:51.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.321+0000 7f365b7fe640 1 -- 172.21.15.60:0/1369557029 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f36740c8030 con 0x7f367415a9d0 2023-10-16T12:34:51.323 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:34:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f366006ae70 msgr2=0x7f366006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f366006ae70 0x7f366006d330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3664008fc0 tx=0x7f3664008510 comp rx=0 tx=0).stop 2023-10-16T12:34:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 msgr2=0x7f3674156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:51.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 0x7f3674156be0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f366802b8c0 tx=0x7f36680046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 shutdown_connections 2023-10-16T12:34:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f366006ae70 0x7f366006d330 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3674160b30 0x7f3674157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f367415a9d0 0x7f3674156be0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 --2- 172.21.15.60:0/1369557029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36740b2360 0x7f36740b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.324+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 >> 172.21.15.60:0/1369557029 conn(0x7f367406d560 msgr2=0x7f36740cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.325+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 shutdown_connections 2023-10-16T12:34:51.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:51.325+0000 7f3678820640 1 -- 172.21.15.60:0/1369557029 wait complete. 2023-10-16T12:34:51.346 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:34:51.347 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:34:51.347 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:34:51.348 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:34:51.348 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:34:51.348 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:34:51.349 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:34:51.349 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:34:51.349 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:34:51.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:51.350 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:34:51.413 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:34:51.414 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:34:51.414 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:34:51.414 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.415 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:34:51.415 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.415 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:34:51.415 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:34:51.415 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:34:51.416 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:34:51.416 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:34:51.416 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:34:51.416 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:34:51.417 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:34:51.417 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:34:51.417 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:34:51.417 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42296sec preferred_lft 42296sec 2023-10-16T12:34:51.417 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:34:51.418 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.418 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:34:51.418 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:34:51.418 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:34:51.418 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:34:51.419 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:34:51.419 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:34:51.419 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.419 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:34:51.420 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.420 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:34:51.420 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:34:51.420 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:34:51.420 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.421 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:34:51.421 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:34:51.421 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:34:51.421 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:34:51.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:51.423 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:34:51.524 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:34:51.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:51.529 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:34:51.593 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:34:51.593 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:34:51.593 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:34:51.594 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:34:51.594 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:34:51.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:51.595 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:34:51.595 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:34:51.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:34:51.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:34:51.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:34:51.597 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:34:51.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:51.786 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:34:51.840 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:34:51.840 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:34:51.841 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:34:51.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:51.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:34:51.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:34:51.940 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:34:51.994 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:34:51.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:34:51.995 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:34:52.048 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:34:52.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:52.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:34:52.201 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:34:52.202 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:34:52.236 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:34:52.236 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:34:52.236 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:34:52.236 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:34:52.237 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:34:52.239 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:34:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:34:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:34:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:34:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:34:52.243 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:34:52.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:52.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:34:52.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:52.334 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:34:52.404 INFO:teuthology.orchestra.run.smithi060.stdout:client4927 2023-10-16T12:34:52.404 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:34:52.405 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:34:52.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:34:52.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.692+0000 7f7251f6a640 1 Processor -- start 2023-10-16T12:34:52.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.692+0000 7f7251f6a640 1 -- start start 2023-10-16T12:34:52.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.694+0000 7f7251f6a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 0x7f724c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.694+0000 7f7251f6a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c15d150 con 0x7f724c15cd70 2023-10-16T12:34:52.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.694+0000 7f7251f6a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c0d1180 con 0x7f724c0d0be0 2023-10-16T12:34:52.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.694+0000 7f7251f6a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c0d1460 con 0x7f724c0b8480 2023-10-16T12:34:52.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.694+0000 7f7250f68640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f724c15cd70 0x7f724c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39288/0 (socket says 172.21.15.60:39288) 2023-10-16T12:34:52.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.694+0000 7f7250f68640 1 -- 172.21.15.60:0/3659434944 learned_addr learned my addr 172.21.15.60:0/3659434944 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:52.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724bfff640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 0x7f724c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813573388 0 0) 0x7f724c15d150 con 0x7f724c15cd70 2023-10-16T12:34:52.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f723c003520 con 0x7f724c15cd70 2023-10-16T12:34:52.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1653083988 0 0) 0x7f723c003520 con 0x7f724c15cd70 2023-10-16T12:34:52.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 msgr2=0x7f724c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:52.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 0x7f724c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 >> v1:172.21.15.144:6789/0 conn(0x7f724c0d0be0 legacy=0x7f724c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c0d1740 con 0x7f724c15cd70 2023-10-16T12:34:52.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7240001670 con 0x7f724c15cd70 2023-10-16T12:34:52.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 >> v1:172.21.15.60:6789/0 conn(0x7f724c15cd70 legacy=0x7f724c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.695+0000 7f724b7fe640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 0x7f723c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724b7fe640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f723c003f20 0x7f723c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724b7fe640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f723c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c0d1460 con 0x7f723c004d00 2023-10-16T12:34:52.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c00ba60 con 0x7f723c003f20 2023-10-16T12:34:52.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c00bc20 con 0x7f724c0b8480 2023-10-16T12:34:52.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7251769640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f723c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7250f68640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 0x7f723c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724bfff640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f723c003f20 0x7f723c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7251769640 1 -- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 msgr2=0x7f723c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:52.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7251769640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c0b8480 0x7f723c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7251769640 1 -- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f723c003f20 msgr2=0x7f723c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:52.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7251769640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f723c003f20 0x7f723c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f7251769640 1 -- 172.21.15.60:0/3659434944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c003520 con 0x7f723c004d00 2023-10-16T12:34:52.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.696+0000 7f724bfff640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f723c003f20 0x7f723c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:52.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.697+0000 7f7251769640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f723c005180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7244001e50 tx=0x7f723c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc904e38f9eb39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:52.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244003b70 con 0x7f723c004d00 2023-10-16T12:34:52.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7244003cd0 con 0x7f723c004d00 2023-10-16T12:34:52.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f724b7fe640 1 -- 172.21.15.60:0/3659434944 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244005310 con 0x7f723c004d00 2023-10-16T12:34:52.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f7251f6a640 1 -- 172.21.15.60:0/3659434944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 msgr2=0x7f723c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3659434944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f723c005180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7244001e50 tx=0x7f723c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f7251769640 1 -- 172.21.15.60:0/3659434944 reap_dead start 2023-10-16T12:34:52.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f7251f6a640 1 -- 172.21.15.60:0/3659434944 shutdown_connections 2023-10-16T12:34:52.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.698+0000 7f7251f6a640 1 -- 172.21.15.60:0/3659434944 >> 172.21.15.60:0/3659434944 conn(0x7f724c0cf570 msgr2=0x7f724c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:52.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.699+0000 7f7251f6a640 1 -- 172.21.15.60:0/3659434944 shutdown_connections 2023-10-16T12:34:52.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.699+0000 7f7251f6a640 1 -- 172.21.15.60:0/3659434944 wait complete. 2023-10-16T12:34:52.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.700+0000 7f7251f6a640 1 Processor -- start 2023-10-16T12:34:52.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.701+0000 7f7251f6a640 1 -- start start 2023-10-16T12:34:52.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.701+0000 7f7251f6a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f724c152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251f6a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c157cc0 0x7f724c06bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251f6a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f724c158140 0x7f724c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251f6a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c1f64f0 con 0x7f723c004d00 2023-10-16T12:34:52.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251f6a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c1f5970 con 0x7f724c158140 2023-10-16T12:34:52.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251f6a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c1f5c50 con 0x7f724c157cc0 2023-10-16T12:34:52.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f724c152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f724c152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51654/0 (socket says 172.21.15.60:51654) 2023-10-16T12:34:52.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 -- 172.21.15.60:0/3438604141 learned_addr learned my addr 172.21.15.60:0/3438604141 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:52.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251769640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f724c158140 0x7f724c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 -- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c157cc0 msgr2=0x7f724c06bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c157cc0 0x7f724c06bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 -- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f724c158140 msgr2=0x7f724c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f724c158140 0x7f724c06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7250f68640 1 -- 172.21.15.60:0/3438604141 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c1f5f30 con 0x7f723c004d00 2023-10-16T12:34:52.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.702+0000 7f7251769640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f724c158140 0x7f724c06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:52.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.703+0000 7f7250f68640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f724c152bd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f724000e7f0 tx=0x7f724000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:52.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.703+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000c8d0 con 0x7f723c004d00 2023-10-16T12:34:52.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.703+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000ca30 con 0x7f723c004d00 2023-10-16T12:34:52.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.703+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724c1f6210 con 0x7f723c004d00 2023-10-16T12:34:52.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.703+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000cdc0 con 0x7f723c004d00 2023-10-16T12:34:52.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.703+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724c153110 con 0x7f723c004d00 2023-10-16T12:34:52.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.704+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c04e9b0 con 0x7f723c004d00 2023-10-16T12:34:52.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.705+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f724001d460 con 0x7f723c004d00 2023-10-16T12:34:52.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.706+0000 7f72497fa640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f723006ae70 0x7f723006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:52.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.706+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7240089760 con 0x7f723c004d00 2023-10-16T12:34:52.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.707+0000 7f724bfff640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f723006ae70 0x7f723006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:52.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.708+0000 7f724bfff640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f723006ae70 0x7f723006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f724c06c9f0 tx=0x7f7238008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:52.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.710+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7240054770 con 0x7f723c004d00 2023-10-16T12:34:52.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.932+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f724c0d1d00 con 0x7f723006ae70 2023-10-16T12:34:52.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.937+0000 7f72497fa640 1 -- 172.21.15.60:0/3438604141 <== mgr.4111 v2:172.21.15.144:6832/1973903962 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f72380107f0 con 0x7f723006ae70 2023-10-16T12:34:52.938 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:34:52.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:34:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:34:52.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:34:52.939 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:34:52.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f723006ae70 msgr2=0x7f723006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f723006ae70 0x7f723006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f724c06c9f0 tx=0x7f7238008040 comp rx=0 tx=0).stop 2023-10-16T12:34:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 msgr2=0x7f724c152bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f724c152bd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f724000e7f0 tx=0x7f724000ecb0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 shutdown_connections 2023-10-16T12:34:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f723006ae70 0x7f723006d330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f724c157cc0 0x7f724c06bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f724c158140 0x7f724c06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 --2- 172.21.15.60:0/3438604141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723c004d00 0x7f724c152bd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 >> 172.21.15.60:0/3438604141 conn(0x7f724c0cf570 msgr2=0x7f724c15ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 shutdown_connections 2023-10-16T12:34:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:52.939+0000 7f7251f6a640 1 -- 172.21.15.60:0/3438604141 wait complete. 2023-10-16T12:34:52.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:34:53.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.185+0000 7f5208f73640 1 Processor -- start 2023-10-16T12:34:53.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.185+0000 7f5208f73640 1 -- start start 2023-10-16T12:34:53.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.186+0000 7f5208f73640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f520415ad70 0x7f5204069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.186+0000 7f5208f73640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52040c9ca0 0x7f520406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.186+0000 7f5208f73640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52040cb030 con 0x7f52040ca3b0 2023-10-16T12:34:53.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.186+0000 7f5208f73640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040c91c0 con 0x7f52040c9ca0 2023-10-16T12:34:53.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.186+0000 7f5208f73640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040c94a0 con 0x7f520415ad70 2023-10-16T12:34:53.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5201d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52040c9ca0 0x7f520406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5202d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f52040ca3b0 0x7f52040ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39294/0 (socket says 172.21.15.60:39294) 2023-10-16T12:34:53.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5202d9d640 1 -- 172.21.15.60:0/4284888036 learned_addr learned my addr 172.21.15.60:0/4284888036 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:53.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f520259c640 1 --2- 172.21.15.60:0/4284888036 >> v2:172.21.15.144:3300/0 conn(0x7f520415ad70 0x7f5204069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5201d9b640 1 -- 172.21.15.60:0/4284888036 >> v2:172.21.15.144:3300/0 conn(0x7f520415ad70 msgr2=0x7f5204069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:53.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5201d9b640 1 --2- 172.21.15.60:0/4284888036 >> v2:172.21.15.144:3300/0 conn(0x7f520415ad70 0x7f5204069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5201d9b640 1 -- 172.21.15.60:0/4284888036 >> v1:172.21.15.60:6789/0 conn(0x7f52040ca3b0 legacy=0x7f52040ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f5201d9b640 1 -- 172.21.15.60:0/4284888036 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040c9780 con 0x7f52040c9ca0 2023-10-16T12:34:53.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.187+0000 7f520159a640 1 -- 172.21.15.60:0/4284888036 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772794764 0 0) 0x7f52040cb030 con 0x7f52040ca3b0 2023-10-16T12:34:53.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.189+0000 7f5201d9b640 1 --2- 172.21.15.60:0/4284888036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f520406a2e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f52040cf780 tx=0x7f51f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fadb51d6c22d3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:53.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.189+0000 7f520159a640 1 -- 172.21.15.60:0/4284888036 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8031830 con 0x7f52040c9ca0 2023-10-16T12:34:53.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.189+0000 7f520159a640 1 -- 172.21.15.60:0/4284888036 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8031990 con 0x7f52040c9ca0 2023-10-16T12:34:53.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.189+0000 7f520159a640 1 -- 172.21.15.60:0/4284888036 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8031d20 con 0x7f52040c9ca0 2023-10-16T12:34:53.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.189+0000 7f5208f73640 1 -- 172.21.15.60:0/4284888036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 msgr2=0x7f520406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.189+0000 7f5208f73640 1 --2- 172.21.15.60:0/4284888036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f520406a2e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f52040cf780 tx=0x7f51f802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.190+0000 7f5208f73640 1 -- 172.21.15.60:0/4284888036 shutdown_connections 2023-10-16T12:34:53.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.190+0000 7f5208f73640 1 --2- 172.21.15.60:0/4284888036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f520406a2e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.190+0000 7f5208f73640 1 --2- 172.21.15.60:0/4284888036 >> v2:172.21.15.144:3300/0 conn(0x7f520415ad70 0x7f5204069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.190+0000 7f5208f73640 1 -- 172.21.15.60:0/4284888036 >> 172.21.15.60:0/4284888036 conn(0x7f52040ce9b0 msgr2=0x7f52040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:53.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.190+0000 7f5208f73640 1 -- 172.21.15.60:0/4284888036 shutdown_connections 2023-10-16T12:34:53.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.190+0000 7f5208f73640 1 -- 172.21.15.60:0/4284888036 wait complete. 2023-10-16T12:34:53.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.191+0000 7f5208f73640 1 Processor -- start 2023-10-16T12:34:53.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.192+0000 7f5208f73640 1 -- start start 2023-10-16T12:34:53.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.192+0000 7f5208f73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f5204158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.192+0000 7f5208f73640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52040ca3b0 0x7f52040ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f5208f73640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f520415ad70 0x7f52040cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f5208f73640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c7ab0 con 0x7f52040c9ca0 2023-10-16T12:34:53.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f5208f73640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c6a10 con 0x7f520415ad70 2023-10-16T12:34:53.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f5208f73640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c6cf0 con 0x7f52040ca3b0 2023-10-16T12:34:53.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f520259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f5204158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f520259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f5204158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51670/0 (socket says 172.21.15.60:51670) 2023-10-16T12:34:53.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f520259c640 1 -- 172.21.15.60:0/4271348951 learned_addr learned my addr 172.21.15.60:0/4271348951 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f5202d9d640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f520415ad70 0x7f52040cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.193+0000 7f5201d9b640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52040ca3b0 0x7f52040ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f520259c640 1 -- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52040ca3b0 msgr2=0x7f52040ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f520259c640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52040ca3b0 0x7f52040ae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f520259c640 1 -- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f520415ad70 msgr2=0x7f52040cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f520259c640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f520415ad70 0x7f52040cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f520259c640 1 -- 172.21.15.60:0/4271348951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f8009400 con 0x7f52040c9ca0 2023-10-16T12:34:53.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f520259c640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f5204158200 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d5f0 tx=0x7f51f400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:53.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f40125f0 con 0x7f52040c9ca0 2023-10-16T12:34:53.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f4012750 con 0x7f52040c9ca0 2023-10-16T12:34:53.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52040c6fd0 con 0x7f52040c9ca0 2023-10-16T12:34:53.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f4012b00 con 0x7f52040c9ca0 2023-10-16T12:34:53.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.194+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52040c7590 con 0x7f52040c9ca0 2023-10-16T12:34:53.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.196+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52040c7720 con 0x7f52040c9ca0 2023-10-16T12:34:53.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.197+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f51f4012c60 con 0x7f52040c9ca0 2023-10-16T12:34:53.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.198+0000 7f51f2ffd640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f51e406aea0 0x7f51e406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.198+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51f4089aa0 con 0x7f52040c9ca0 2023-10-16T12:34:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.198+0000 7f5201d9b640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f51e406aea0 0x7f51e406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.200+0000 7f5201d9b640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f51e406aea0 0x7f51e406d360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51f8036000 tx=0x7f51f8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:53.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.202+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51f4054ab0 con 0x7f52040c9ca0 2023-10-16T12:34:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.449+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f520404e160 con 0x7f52040c9ca0 2023-10-16T12:34:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.560+0000 7f51f2ffd640 1 -- 172.21.15.60:0/4271348951 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f51f40580d0 con 0x7f52040c9ca0 2023-10-16T12:34:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f51e406aea0 msgr2=0x7f51e406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f51e406aea0 0x7f51e406d360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51f8036000 tx=0x7f51f8008030 comp rx=0 tx=0).stop 2023-10-16T12:34:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 msgr2=0x7f5204158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f5204158200 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d5f0 tx=0x7f51f400dab0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 shutdown_connections 2023-10-16T12:34:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f51e406aea0 0x7f51e406d360 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52040ca3b0 0x7f52040ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f520415ad70 0x7f52040cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 --2- 172.21.15.60:0/4271348951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52040c9ca0 0x7f5204158200 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.563+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 >> 172.21.15.60:0/4271348951 conn(0x7f52040ce9b0 msgr2=0x7f5204070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.564+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 shutdown_connections 2023-10-16T12:34:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.564+0000 7f5208f73640 1 -- 172.21.15.60:0/4271348951 wait complete. 2023-10-16T12:34:53.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:34:53.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.794+0000 7f3eeb1e3640 1 Processor -- start 2023-10-16T12:34:53.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.794+0000 7f3eeb1e3640 1 -- start start 2023-10-16T12:34:53.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3eeb1e3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee415ad70 0x7f3ee4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3eeb1e3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ee40c9ca0 0x7f3ee406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3eeb1e3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40cb030 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3eeb1e3640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee40c91c0 con 0x7f3ee40c9ca0 2023-10-16T12:34:53.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3eeb1e3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40c94a0 con 0x7f3ee415ad70 2023-10-16T12:34:53.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3ee3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ee40c9ca0 0x7f3ee406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3ee9780640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3ee40ca3b0 0x7f3ee40ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39310/0 (socket says 172.21.15.60:39310) 2023-10-16T12:34:53.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.796+0000 7f3ee9780640 1 -- 172.21.15.60:0/1176448206 learned_addr learned my addr 172.21.15.60:0/1176448206 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:53.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.797+0000 7f3ee8f7f640 1 --2- 172.21.15.60:0/1176448206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee415ad70 0x7f3ee4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.797+0000 7f3ee3fff640 1 -- 172.21.15.60:0/1176448206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee415ad70 msgr2=0x7f3ee4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.797+0000 7f3ee3fff640 1 --2- 172.21.15.60:0/1176448206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee415ad70 0x7f3ee4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.797+0000 7f3ee3fff640 1 -- 172.21.15.60:0/1176448206 >> v1:172.21.15.60:6789/0 conn(0x7f3ee40ca3b0 legacy=0x7f3ee40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.797+0000 7f3ee3fff640 1 -- 172.21.15.60:0/1176448206 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40c9780 con 0x7f3ee40c9ca0 2023-10-16T12:34:53.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.799+0000 7f3ee3fff640 1 --2- 172.21.15.60:0/1176448206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40c9ca0 0x7f3ee406a2e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00097a0 tx=0x7f3ed002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c0ad401a5c2523 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:53.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.799+0000 7f3ee37fe640 1 -- 172.21.15.60:0/1176448206 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed002d040 con 0x7f3ee40c9ca0 2023-10-16T12:34:53.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.799+0000 7f3ee37fe640 1 -- 172.21.15.60:0/1176448206 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed0031810 con 0x7f3ee40c9ca0 2023-10-16T12:34:53.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3ee37fe640 1 -- 172.21.15.60:0/1176448206 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0031ba0 con 0x7f3ee40c9ca0 2023-10-16T12:34:53.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/1176448206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40c9ca0 msgr2=0x7f3ee406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/1176448206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40c9ca0 0x7f3ee406a2e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00097a0 tx=0x7f3ed002b8b0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/1176448206 shutdown_connections 2023-10-16T12:34:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/1176448206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40c9ca0 0x7f3ee406a2e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/1176448206 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee415ad70 0x7f3ee4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/1176448206 >> 172.21.15.60:0/1176448206 conn(0x7f3ee40ce9b0 msgr2=0x7f3ee40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/1176448206 shutdown_connections 2023-10-16T12:34:53.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.800+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/1176448206 wait complete. 2023-10-16T12:34:53.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.802+0000 7f3eeb1e3640 1 Processor -- start 2023-10-16T12:34:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.802+0000 7f3eeb1e3640 1 -- start start 2023-10-16T12:34:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.803+0000 7f3eeb1e3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee40c9ca0 0x7f3ee4157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.803+0000 7f3eeb1e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 0x7f3ee40acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.803+0000 7f3eeb1e3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee415ad70 0x7f3ee40b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.803+0000 7f3eeb1e3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b05b0 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.803+0000 7f3eeb1e3640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40c7c50 con 0x7f3ee415ad70 2023-10-16T12:34:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.803+0000 7f3eeb1e3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee4158160 con 0x7f3ee40c9ca0 2023-10-16T12:34:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 0x7f3ee40acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 0x7f3ee40acd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51706/0 (socket says 172.21.15.60:51706) 2023-10-16T12:34:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 -- 172.21.15.60:0/2591512162 learned_addr learned my addr 172.21.15.60:0/2591512162 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee8f7f640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee40c9ca0 0x7f3ee4157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee9780640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee415ad70 0x7f3ee40b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 -- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee40c9ca0 msgr2=0x7f3ee4157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee40c9ca0 0x7f3ee4157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 -- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee415ad70 msgr2=0x7f3ee40b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:53.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee415ad70 0x7f3ee40b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:53.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.804+0000 7f3ee3fff640 1 -- 172.21.15.60:0/2591512162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0009400 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.805+0000 7f3ee3fff640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 0x7f3ee40acd20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00098c0 tx=0x7f3ed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:53.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.805+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed003ad60 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.805+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed0044400 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.805+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee40c8790 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.805+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00447b0 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.805+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee40c81d0 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.807+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee4159bd0 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.809+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f3ed004d440 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.810+0000 7f3ee17fa640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 0x7f3ec806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.810+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ed00abad0 con 0x7f3ee40ca3b0 2023-10-16T12:34:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.810+0000 7f3ee8f7f640 1 -- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 msgr2=0x7f3ec806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.810+0000 7f3ee8f7f640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 0x7f3ec806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:34:53.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:53.815+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ed0076ae0 con 0x7f3ee40ca3b0 2023-10-16T12:34:54.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.009+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3ee404e160 con 0x7f3ee40ca3b0 2023-10-16T12:34:54.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.011+0000 7f3ee8f7f640 1 -- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 msgr2=0x7f3ec806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:54.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.011+0000 7f3ee8f7f640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 0x7f3ec806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:34:54.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.016+0000 7f3ee17fa640 1 -- 172.21.15.60:0/2591512162 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3ed007a100 con 0x7f3ee40ca3b0 2023-10-16T12:34:54.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 msgr2=0x7f3ec806d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:34:54.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 0x7f3ec806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 msgr2=0x7f3ee40acd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 0x7f3ee40acd20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00098c0 tx=0x7f3ed00046c0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 shutdown_connections 2023-10-16T12:34:54.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f3ec806aec0 0x7f3ec806d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee415ad70 0x7f3ee40b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee40ca3b0 0x7f3ee40acd20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 --2- 172.21.15.60:0/2591512162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee40c9ca0 0x7f3ee4157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 >> 172.21.15.60:0/2591512162 conn(0x7f3ee40ce9b0 msgr2=0x7f3ee415b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:54.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.018+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 shutdown_connections 2023-10-16T12:34:54.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.019+0000 7f3eeb1e3640 1 -- 172.21.15.60:0/2591512162 wait complete. 2023-10-16T12:34:54.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:34:54.088 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.086+0000 7f8e59838040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:34:54.108 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.107+0000 7faf4062f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:34:54.142 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.139+0000 7f8e59838040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:34:54.166 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.165+0000 7faf4062f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:34:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.245+0000 7f5e472af640 1 Processor -- start 2023-10-16T12:34:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.245+0000 7f5e472af640 1 -- start start 2023-10-16T12:34:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e472af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159ab0 0x7f5e400ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e472af640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e40159e90 0x7f5e4006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e472af640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e400cd1f0 con 0x7f5e4006c140 2023-10-16T12:34:54.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e472af640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4015d890 con 0x7f5e40159e90 2023-10-16T12:34:54.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e472af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400cd4d0 con 0x7f5e40159ab0 2023-10-16T12:34:54.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e4584c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e4006c140 0x7f5e4015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39318/0 (socket says 172.21.15.60:39318) 2023-10-16T12:34:54.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e4584c640 1 -- 172.21.15.60:0/2435840088 learned_addr learned my addr 172.21.15.60:0/2435840088 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:54.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.247+0000 7f5e4484a640 1 --2- 172.21.15.60:0/2435840088 >> v2:172.21.15.60:3300/0 conn(0x7f5e40159e90 0x7f5e4006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.248+0000 7f5e4504b640 1 --2- 172.21.15.60:0/2435840088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159ab0 0x7f5e400ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.248+0000 7f5e4484a640 1 -- 172.21.15.60:0/2435840088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159ab0 msgr2=0x7f5e400ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:54.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.248+0000 7f5e4484a640 1 --2- 172.21.15.60:0/2435840088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159ab0 0x7f5e400ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.248+0000 7f5e4484a640 1 -- 172.21.15.60:0/2435840088 >> v1:172.21.15.60:6789/0 conn(0x7f5e4006c140 legacy=0x7f5e4015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.248+0000 7f5e4484a640 1 -- 172.21.15.60:0/2435840088 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e400cd7b0 con 0x7f5e40159e90 2023-10-16T12:34:54.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.248+0000 7f5e37fff640 1 -- 172.21.15.60:0/2435840088 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864303482 0 0) 0x7f5e400cd1f0 con 0x7f5e4006c140 2023-10-16T12:34:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.250+0000 7f5e4484a640 1 --2- 172.21.15.60:0/2435840088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e40159e90 0x7f5e4006fc50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097d0 tx=0x7f5e3002d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34de09dee7730e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.250+0000 7f5e37fff640 1 -- 172.21.15.60:0/2435840088 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30032860 con 0x7f5e40159e90 2023-10-16T12:34:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.250+0000 7f5e37fff640 1 -- 172.21.15.60:0/2435840088 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e300329c0 con 0x7f5e40159e90 2023-10-16T12:34:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.250+0000 7f5e37fff640 1 -- 172.21.15.60:0/2435840088 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30032d50 con 0x7f5e40159e90 2023-10-16T12:34:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 -- 172.21.15.60:0/2435840088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e40159e90 msgr2=0x7f5e4006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 --2- 172.21.15.60:0/2435840088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e40159e90 0x7f5e4006fc50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097d0 tx=0x7f5e3002d9a0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 -- 172.21.15.60:0/2435840088 shutdown_connections 2023-10-16T12:34:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 --2- 172.21.15.60:0/2435840088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e40159e90 0x7f5e4006fc50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 --2- 172.21.15.60:0/2435840088 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159ab0 0x7f5e400ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 -- 172.21.15.60:0/2435840088 >> 172.21.15.60:0/2435840088 conn(0x7f5e400ce9b0 msgr2=0x7f5e400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 -- 172.21.15.60:0/2435840088 shutdown_connections 2023-10-16T12:34:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.251+0000 7f5e472af640 1 -- 172.21.15.60:0/2435840088 wait complete. 2023-10-16T12:34:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.253+0000 7f5e472af640 1 Processor -- start 2023-10-16T12:34:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.253+0000 7f5e472af640 1 -- start start 2023-10-16T12:34:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e472af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 0x7f5e40156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e472af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e40159ab0 0x7f5e400c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e472af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159e90 0x7f5e400acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e472af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400c2900 con 0x7f5e4006c140 2023-10-16T12:34:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e472af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400c2be0 con 0x7f5e40159ab0 2023-10-16T12:34:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e472af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400c2ec0 con 0x7f5e40159e90 2023-10-16T12:34:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e4504b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 0x7f5e40156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e4504b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 0x7f5e40156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51722/0 (socket says 172.21.15.60:51722) 2023-10-16T12:34:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.254+0000 7f5e4504b640 1 -- 172.21.15.60:0/375247553 learned_addr learned my addr 172.21.15.60:0/375247553 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4484a640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e40159ab0 0x7f5e400c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4584c640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159e90 0x7f5e400acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4504b640 1 -- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159e90 msgr2=0x7f5e400acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4504b640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159e90 0x7f5e400acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4504b640 1 -- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e40159ab0 msgr2=0x7f5e400c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4504b640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e40159ab0 0x7f5e400c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4504b640 1 -- 172.21.15.60:0/375247553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30009400 con 0x7f5e4006c140 2023-10-16T12:34:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4484a640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e40159ab0 0x7f5e400c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:34:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e4504b640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 0x7f5e40156180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00e870 tx=0x7f5e3c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0094f0 con 0x7f5e4006c140 2023-10-16T12:34:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c00c940 con 0x7f5e4006c140 2023-10-16T12:34:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e400c31a0 con 0x7f5e4006c140 2023-10-16T12:34:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.255+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c00ccf0 con 0x7f5e4006c140 2023-10-16T12:34:54.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.256+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e400c24e0 con 0x7f5e4006c140 2023-10-16T12:34:54.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.257+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e400c26d0 con 0x7f5e4006c140 2023-10-16T12:34:54.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.258+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f5e3c013470 con 0x7f5e4006c140 2023-10-16T12:34:54.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.259+0000 7f5e35ffb640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 0x7f5e2406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.259+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e3c089780 con 0x7f5e4006c140 2023-10-16T12:34:54.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.259+0000 7f5e4484a640 1 -- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 msgr2=0x7f5e2406d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:54.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.259+0000 7f5e4484a640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 0x7f5e2406d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:34:54.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.263+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e3c054820 con 0x7f5e4006c140 2023-10-16T12:34:54.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.460+0000 7f5e4484a640 1 -- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 msgr2=0x7f5e2406d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:54.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.460+0000 7f5e4484a640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 0x7f5e2406d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:34:54.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.461+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5e4004e160 con 0x7f5e4006c140 2023-10-16T12:34:54.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.466+0000 7f5e35ffb640 1 -- 172.21.15.60:0/375247553 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5e3c057e40 con 0x7f5e4006c140 2023-10-16T12:34:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 msgr2=0x7f5e2406d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:34:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 0x7f5e2406d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 msgr2=0x7f5e40156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 0x7f5e40156180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00e870 tx=0x7f5e3c00ed30 comp rx=0 tx=0).stop 2023-10-16T12:34:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 shutdown_connections 2023-10-16T12:34:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7f5e2406aec0 0x7f5e2406d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e40159e90 0x7f5e400acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e40159ab0 0x7f5e400c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 --2- 172.21.15.60:0/375247553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e4006c140 0x7f5e40156180 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.468+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 >> 172.21.15.60:0/375247553 conn(0x7f5e400ce9b0 msgr2=0x7f5e400ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.469+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 shutdown_connections 2023-10-16T12:34:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.469+0000 7f5e472af640 1 -- 172.21.15.60:0/375247553 wait complete. 2023-10-16T12:34:54.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:54.489 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:34:54.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:54.544 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-16T12:34:54.590 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.587+0000 7f8e59838040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:34:54.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-10-16T12:34:54.643 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.640+0000 7f8e59838040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:34:54.650 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.649+0000 7faf4062f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:34:54.692 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.690+0000 7f8e59838040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:34:54.707 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.705+0000 7faf4062f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:34:54.753 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.750+0000 7f8e59838040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:34:54.761 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.760+0000 7faf4062f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:34:54.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.795+0000 7fd55b4b4640 1 Processor -- start 2023-10-16T12:34:54.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.796+0000 7fd55b4b4640 1 -- start start 2023-10-16T12:34:54.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd55b4b4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd55406a4b0 0x7fd5540ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd55b4b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 0x7fd5540ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd55b4b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5540cb030 con 0x7fd55415ad70 2023-10-16T12:34:54.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd55b4b4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5540cae30 con 0x7fd55406a4b0 2023-10-16T12:34:54.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd55b4b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540c91c0 con 0x7fd5540c9ca0 2023-10-16T12:34:54.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd559250640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd55415ad70 0x7fd554069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39330/0 (socket says 172.21.15.60:39330) 2023-10-16T12:34:54.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd559250640 1 -- 172.21.15.60:0/1392009817 learned_addr learned my addr 172.21.15.60:0/1392009817 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:54.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.797+0000 7fd558a4f640 1 --2- 172.21.15.60:0/1392009817 >> v2:172.21.15.144:3300/0 conn(0x7fd55406a4b0 0x7fd5540ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd559a51640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 0x7fd5540ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117826136 0 0) 0x7fd5540cb030 con 0x7fd55415ad70 2023-10-16T12:34:54.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd540003520 con 0x7fd55415ad70 2023-10-16T12:34:54.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387037038 0 0) 0x7fd540003520 con 0x7fd55415ad70 2023-10-16T12:34:54.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 msgr2=0x7fd5540ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:54.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 0x7fd5540ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 >> v2:172.21.15.144:3300/0 conn(0x7fd55406a4b0 msgr2=0x7fd5540ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:54.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 --2- 172.21.15.60:0/1392009817 >> v2:172.21.15.144:3300/0 conn(0x7fd55406a4b0 0x7fd5540ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5540c94a0 con 0x7fd55415ad70 2023-10-16T12:34:54.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd550001670 con 0x7fd55415ad70 2023-10-16T12:34:54.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 >> v1:172.21.15.60:6789/0 conn(0x7fd55415ad70 legacy=0x7fd554069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 0x7fd540003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd540003ef0 0x7fd5400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd540004d00 0x7fd540005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540cae30 con 0x7fd540004d00 2023-10-16T12:34:54.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540c91c0 con 0x7fd540003ef0 2023-10-16T12:34:54.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54000b980 con 0x7fd5540c9ca0 2023-10-16T12:34:54.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.798+0000 7fd559a51640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd540004d00 0x7fd540005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd559250640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 0x7fd540003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd558a4f640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd540003ef0 0x7fd5400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd559a51640 1 -- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 msgr2=0x7fd540003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:54.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd559a51640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5540c9ca0 0x7fd540003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd559a51640 1 -- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd540003ef0 msgr2=0x7fd5400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:54.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd559a51640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd540003ef0 0x7fd5400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.799+0000 7fd559a51640 1 -- 172.21.15.60:0/1392009817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540003520 con 0x7fd540004d00 2023-10-16T12:34:54.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.800+0000 7fd559a51640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd540004d00 0x7fd540005180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009fc0 tx=0x7fd5400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6887d5fdba25ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:54.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.800+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c02cc00 con 0x7fd540004d00 2023-10-16T12:34:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.800+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c00f040 con 0x7fd540004d00 2023-10-16T12:34:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.800+0000 7fd54bfff640 1 -- 172.21.15.60:0/1392009817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c034440 con 0x7fd540004d00 2023-10-16T12:34:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1392009817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd540004d00 msgr2=0x7fd540005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd55b4b4640 1 --2- 172.21.15.60:0/1392009817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd540004d00 0x7fd540005180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009fc0 tx=0x7fd5400057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd559a51640 1 -- 172.21.15.60:0/1392009817 reap_dead start 2023-10-16T12:34:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1392009817 shutdown_connections 2023-10-16T12:34:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1392009817 >> 172.21.15.60:0/1392009817 conn(0x7fd5540ce9b0 msgr2=0x7fd5540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1392009817 shutdown_connections 2023-10-16T12:34:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.801+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1392009817 wait complete. 2023-10-16T12:34:54.811 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.799+0000 7f8e59838040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:34:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.803+0000 7fd55b4b4640 1 Processor -- start 2023-10-16T12:34:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.803+0000 7fd55b4b4640 1 -- start start 2023-10-16T12:34:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd55b4b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd554158cc0 0x7fd5541f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd55b4b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5541f60a0 0x7fd5540ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd55b4b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5540ade20 0x7fd5540c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd55b4b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5540ae2a0 con 0x7fd5540ade20 2023-10-16T12:34:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd55b4b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5541f30a0 con 0x7fd5541f60a0 2023-10-16T12:34:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd55b4b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5541f26d0 con 0x7fd554158cc0 2023-10-16T12:34:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5540ade20 0x7fd5540c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5540ade20 0x7fd5540c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51738/0 (socket says 172.21.15.60:51738) 2023-10-16T12:34:54.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 -- 172.21.15.60:0/1635788977 learned_addr learned my addr 172.21.15.60:0/1635788977 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:54.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd558a4f640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5541f60a0 0x7fd5540ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:54.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd554158cc0 msgr2=0x7fd5541f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd554158cc0 0x7fd5541f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5541f60a0 msgr2=0x7fd5540ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.804+0000 7fd559a51640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5541f60a0 0x7fd5540ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd559a51640 1 -- 172.21.15.60:0/1635788977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55415d780 con 0x7fd5540ade20 2023-10-16T12:34:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd558a4f640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5541f60a0 0x7fd5540ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:54.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd559a51640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5540ade20 0x7fd5540c5a80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd54c002230 tx=0x7fd54c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:54.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c003990 con 0x7fd5540ade20 2023-10-16T12:34:54.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c003af0 con 0x7fd5540ade20 2023-10-16T12:34:54.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c003ea0 con 0x7fd5540ade20 2023-10-16T12:34:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5541f20a0 con 0x7fd5540ade20 2023-10-16T12:34:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.805+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55415dd00 con 0x7fd5540ade20 2023-10-16T12:34:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.806+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55404e160 con 0x7fd5540ade20 2023-10-16T12:34:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.808+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fd54c034900 con 0x7fd5540ade20 2023-10-16T12:34:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.808+0000 7fd549ffb640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 0x7fd53806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:54.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.809+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd54c0ab330 con 0x7fd5540ade20 2023-10-16T12:34:54.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.809+0000 7fd559250640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 msgr2=0x7fd53806d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:54.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.809+0000 7fd559250640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 0x7fd53806d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:34:54.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:54.813+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd54c076590 con 0x7fd5540ade20 2023-10-16T12:34:54.828 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.827+0000 7faf4062f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:34:54.881 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.880+0000 7faf4062f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:34:54.913 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.910+0000 7f8e59838040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:34:54.975 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:54.972+0000 7f8e59838040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:34:55.000 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:54.998+0000 7faf4062f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:34:55.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:55.010+0000 7fd559250640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 msgr2=0x7fd53806d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:55.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:55.010+0000 7fd559250640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 0x7fd53806d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:34:55.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.021+0000 7f8e59838040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:34:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:55.035+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 --> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5540c9780 con 0x7fd53806aef0 2023-10-16T12:34:55.066 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.065+0000 7faf4062f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:34:55.081 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.078+0000 7f8e59838040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:34:55.118 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.117+0000 7faf4062f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:34:55.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.127+0000 7f8e59838040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:34:55.178 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.177+0000 7faf4062f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:34:55.232 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.230+0000 7faf4062f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:34:55.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:55.411+0000 7fd559250640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 msgr2=0x7fd53806d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:55.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:55.411+0000 7fd559250640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 0x7fd53806d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:34:55.508 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.505+0000 7f8e59838040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:34:55.638 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.637+0000 7faf4062f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:34:55.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.663+0000 7f8e59838040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:34:55.774 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.771+0000 7f8e59838040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:34:55.808 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.807+0000 7faf4062f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:34:55.824 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.821+0000 7f8e59838040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:34:55.924 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.922+0000 7faf4062f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:34:55.933 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:55.931+0000 7f8e59838040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:34:55.977 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:55.975+0000 7faf4062f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:34:56.040 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:56.037+0000 7f8e59838040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:34:56.091 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:56.089+0000 7faf4062f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:34:56.195 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:56.192+0000 7f8e59838040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:34:56.203 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:56.202+0000 7faf4062f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:34:56.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:56.212+0000 7fd559250640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 msgr2=0x7fd53806d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1973903962 2023-10-16T12:34:56.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:56.212+0000 7fd559250640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 0x7fd53806d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:34:56.266 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:56.263+0000 7f8e59838040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:34:56.368 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:56.367+0000 7faf4062f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:34:56.408 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:56.405+0000 7f8e59838040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:34:56.420 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:56.419+0000 7faf4062f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:34:56.547 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:56.544+0000 7faf4062f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:34:57.234 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:57.231+0000 7f8e59838040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:34:57.403 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:57.402+0000 7faf4062f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:34:57.529 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:57.526+0000 7f8e59838040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:34:57.585 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:57.582+0000 7f8e59838040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:34:57.698 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:57.695+0000 7f8e59838040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:34:57.721 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:57.719+0000 7faf4062f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:34:57.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:57.721+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 7) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fd54c076960 con 0x7fd5540ade20 2023-10-16T12:34:57.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:57.722+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 msgr2=0x7fd53806d3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:34:57.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:57.722+0000 7fd549ffb640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/1973903962,v1:172.21.15.144:6833/1973903962] conn(0x7fd53806aef0 0x7fd53806d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:57.790 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:57.788+0000 7faf4062f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:34:57.928 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:34:57.927+0000 7faf4062f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:34:58.060 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.057+0000 7f8dd933e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.061 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.057+0000 7f8dd933e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.061 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.057+0000 7f8dd933e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.061 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.057+0000 7f8dd933e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.061 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.057+0000 7f8dd933e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.061 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.058+0000 7f8dd62f8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.062 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.058+0000 7f8dd62f8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.062 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.058+0000 7f8dd62f8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.062 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.058+0000 7f8dd62f8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.062 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:34:58.058+0000 7f8dd62f8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:34:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.733+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd54c0046c0 con 0x7fd5540ade20 2023-10-16T12:34:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.734+0000 7fd549ffb640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fd538035ad0 0x7fd538074db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.734+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd538008050 con 0x7fd538035ad0 2023-10-16T12:34:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.742+0000 7fd559250640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fd538035ad0 0x7fd538074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.746+0000 7fd559250640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fd538035ad0 0x7fd538074db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd550001eb0 tx=0x7fd550009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:58.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.967+0000 7fd549ffb640 1 -- 172.21.15.60:0/1635788977 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd538008050 con 0x7fd538035ad0 2023-10-16T12:34:58.969 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-16T12:34:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fd538035ad0 msgr2=0x7fd538074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fd538035ad0 0x7fd538074db0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd550001eb0 tx=0x7fd550009040 comp rx=0 tx=0).stop 2023-10-16T12:34:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5540ade20 msgr2=0x7fd5540c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 --2- 172.21.15.60:0/1635788977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5540ade20 0x7fd5540c5a80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd54c002230 tx=0x7fd54c009750 comp rx=0 tx=0).stop 2023-10-16T12:34:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd559a51640 1 -- 172.21.15.60:0/1635788977 reap_dead start 2023-10-16T12:34:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 shutdown_connections 2023-10-16T12:34:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 >> 172.21.15.60:0/1635788977 conn(0x7fd5540ce9b0 msgr2=0x7fd554069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:58.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 shutdown_connections 2023-10-16T12:34:58.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:58.971+0000 7fd55b4b4640 1 -- 172.21.15.60:0/1635788977 wait complete. 2023-10-16T12:34:58.994 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-16T12:34:58.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:58.995 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-16T12:34:59.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-10-16T12:34:59.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.280+0000 7fa4b0cda640 1 Processor -- start 2023-10-16T12:34:59.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.281+0000 7fa4b0cda640 1 -- start start 2023-10-16T12:34:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.282+0000 7fa4b0cda640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 0x7fa4ac069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.282+0000 7fa4b0cda640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac0cb030 con 0x7fa4ac0caac0 2023-10-16T12:34:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.282+0000 7fa4b0cda640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac0c91c0 con 0x7fa4ac0c9ca0 2023-10-16T12:34:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.282+0000 7fa4b0cda640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0c94a0 con 0x7fa4ac15ad70 2023-10-16T12:34:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.282+0000 7fa4aad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa4ac0caac0 0x7fa4ac0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41628/0 (socket says 172.21.15.60:41628) 2023-10-16T12:34:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4aad9d640 1 -- 172.21.15.60:0/677669624 learned_addr learned my addr 172.21.15.60:0/677669624 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:34:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468901773 0 0) 0x7fa4ac0cb030 con 0x7fa4ac0caac0 2023-10-16T12:34:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4aa59c640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 0x7fa4ac069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498003520 con 0x7fa4ac0caac0 2023-10-16T12:34:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1800962526 0 0) 0x7fa498003520 con 0x7fa4ac0caac0 2023-10-16T12:34:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 msgr2=0x7fa4ac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 0x7fa4ac069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 >> v1:172.21.15.144:6789/0 conn(0x7fa4ac0c9ca0 legacy=0x7fa4ac0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0c9780 con 0x7fa4ac0caac0 2023-10-16T12:34:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa4a40019f0 con 0x7fa4ac0caac0 2023-10-16T12:34:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 >> v1:172.21.15.60:6789/0 conn(0x7fa4ac0caac0 legacy=0x7fa4ac0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4980039a0 0x7fa498003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.283+0000 7fa4a9d9b640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4980047c0 0x7fa498004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a9d9b640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 0x7fa4980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0c94a0 con 0x7fa4980047c0 2023-10-16T12:34:59.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800bab0 con 0x7fa4980039a0 2023-10-16T12:34:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49800bc70 con 0x7fa4ac15ad70 2023-10-16T12:34:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4980047c0 0x7fa498004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4aa59c640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4980039a0 0x7fa498003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4aad9d640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 0x7fa4980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a3fff640 1 -- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 msgr2=0x7fa4980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4ac15ad70 0x7fa4980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a3fff640 1 -- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4980039a0 msgr2=0x7fa498003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:34:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4980039a0 0x7fa498003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4a3fff640 1 -- 172.21.15.60:0/677669624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498003520 con 0x7fa4980047c0 2023-10-16T12:34:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.284+0000 7fa4aa59c640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4980039a0 0x7fa498003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:34:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.286+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4980047c0 0x7fa498004c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa494001750 tx=0x7fa4980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1236d5d5537386b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.286+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940052d0 con 0x7fa4980047c0 2023-10-16T12:34:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.286+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49402e050 con 0x7fa4980047c0 2023-10-16T12:34:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.286+0000 7fa4a9d9b640 1 -- 172.21.15.60:0/677669624 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940035f0 con 0x7fa4980047c0 2023-10-16T12:34:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4b0cda640 1 -- 172.21.15.60:0/677669624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4980047c0 msgr2=0x7fa498004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/677669624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4980047c0 0x7fa498004c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa494001750 tx=0x7fa4980057f0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4aad9d640 1 -- 172.21.15.60:0/677669624 reap_dead start 2023-10-16T12:34:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4b0cda640 1 -- 172.21.15.60:0/677669624 shutdown_connections 2023-10-16T12:34:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4b0cda640 1 -- 172.21.15.60:0/677669624 >> 172.21.15.60:0/677669624 conn(0x7fa4ac0ce9b0 msgr2=0x7fa4ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4b0cda640 1 -- 172.21.15.60:0/677669624 shutdown_connections 2023-10-16T12:34:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.287+0000 7fa4b0cda640 1 -- 172.21.15.60:0/677669624 wait complete. 2023-10-16T12:34:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.289+0000 7fa4b0cda640 1 Processor -- start 2023-10-16T12:34:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.289+0000 7fa4b0cda640 1 -- start start 2023-10-16T12:34:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4b0cda640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4980047c0 0x7fa4ac0c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4b0cda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 0x7fa4ac15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4b0cda640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ac15b8d0 0x7fa4ac15d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4b0cda640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac15d9c0 con 0x7fa4ac15b450 2023-10-16T12:34:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4b0cda640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac15cab0 con 0x7fa4ac15b8d0 2023-10-16T12:34:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4b0cda640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac15bea0 con 0x7fa4980047c0 2023-10-16T12:34:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 0x7fa4ac15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 0x7fa4ac15cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45968/0 (socket says 172.21.15.60:45968) 2023-10-16T12:34:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2732991801 learned_addr learned my addr 172.21.15.60:0/2732991801 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:34:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4980047c0 msgr2=0x7fa4ac0c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4980047c0 0x7fa4ac0c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ac15b8d0 msgr2=0x7fa4ac15d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ac15b8d0 0x7fa4ac15d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.290+0000 7fa4a3fff640 1 -- 172.21.15.60:0/2732991801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac15c180 con 0x7fa4ac15b450 2023-10-16T12:34:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.291+0000 7fa4a3fff640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 0x7fa4ac15cf40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa494008fd0 tx=0x7fa494001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.291+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940076c0 con 0x7fa4ac15b450 2023-10-16T12:34:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.291+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa494007a30 con 0x7fa4ac15b450 2023-10-16T12:34:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.291+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac15c460 con 0x7fa4ac15b450 2023-10-16T12:34:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.291+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494006c00 con 0x7fa4ac15b450 2023-10-16T12:34:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.291+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac0c46e0 con 0x7fa4ac15b450 2023-10-16T12:34:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.292+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac0c4870 con 0x7fa4ac15b450 2023-10-16T12:34:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.293+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa4940035f0 con 0x7fa4ac15b450 2023-10-16T12:34:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.294+0000 7fa4a2ffd640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa48c06ad40 0x7fa48c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:34:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.294+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa4940ab450 con 0x7fa4ac15b450 2023-10-16T12:34:59.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.295+0000 7fa4aa59c640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa48c06ad40 0x7fa48c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:34:59.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.296+0000 7fa4aa59c640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa48c06ad40 0x7fa48c06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa49c006fd0 tx=0x7fa49c009350 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:34:59.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.299+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa494076670 con 0x7fa4ac15b450 2023-10-16T12:34:59.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.518+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4ac0adbe0 con 0x7fa48c06ad40 2023-10-16T12:34:59.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.713+0000 7fa4a2ffd640 1 -- 172.21.15.60:0/2732991801 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa49c0094e0 con 0x7fa48c06ad40 2023-10-16T12:34:59.715 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-16T12:34:59.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.716+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa48c06ad40 msgr2=0x7fa48c06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.716+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa48c06ad40 0x7fa48c06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa49c006fd0 tx=0x7fa49c009350 comp rx=0 tx=0).stop 2023-10-16T12:34:59.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.716+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 msgr2=0x7fa4ac15cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:34:59.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.716+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 0x7fa4ac15cf40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa494008fd0 tx=0x7fa494001120 comp rx=0 tx=0).stop 2023-10-16T12:34:59.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 shutdown_connections 2023-10-16T12:34:59.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa48c06ad40 0x7fa48c06d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4ac15b8d0 0x7fa4ac15d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4ac15b450 0x7fa4ac15cf40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 --2- 172.21.15.60:0/2732991801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4980047c0 0x7fa4ac0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:34:59.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 >> 172.21.15.60:0/2732991801 conn(0x7fa4ac0ce9b0 msgr2=0x7fa4ac0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:34:59.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 shutdown_connections 2023-10-16T12:34:59.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:34:59.717+0000 7fa4b0cda640 1 -- 172.21.15.60:0/2732991801 wait complete. 2023-10-16T12:34:59.740 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-16T12:34:59.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:34:59.741 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-16T12:34:59.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-10-16T12:35:00.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.026+0000 7f3deda47640 1 Processor -- start 2023-10-16T12:35:00.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.026+0000 7f3deda47640 1 -- start start 2023-10-16T12:35:00.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.027+0000 7f3deda47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de815cd70 0x7f3de80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.027+0000 7f3deda47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3de80b8480 0x7f3de80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.027+0000 7f3deda47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3de80d0be0 0x7f3de80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.027+0000 7f3deda47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de815d150 con 0x7f3de80d0be0 2023-10-16T12:35:00.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.027+0000 7f3deda47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de80d1180 con 0x7f3de80b8480 2023-10-16T12:35:00.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.027+0000 7f3deda47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de80d1460 con 0x7f3de815cd70 2023-10-16T12:35:00.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.028+0000 7f3ded246640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3de80d0be0 0x7f3de80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.028+0000 7f3ded246640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3de80d0be0 0x7f3de80cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45976/0 (socket says 172.21.15.60:45976) 2023-10-16T12:35:00.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.028+0000 7f3ded246640 1 -- 172.21.15.60:0/2136763237 learned_addr learned my addr 172.21.15.60:0/2136763237 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:00.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.028+0000 7f3deca45640 1 --2- 172.21.15.60:0/2136763237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de815cd70 0x7f3de80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.028+0000 7f3de7fff640 1 --2- 172.21.15.60:0/2136763237 >> v2:172.21.15.144:3300/0 conn(0x7f3de80b8480 0x7f3de80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.029+0000 7f3ded246640 1 -- 172.21.15.60:0/2136763237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de815cd70 msgr2=0x7f3de80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.029+0000 7f3ded246640 1 --2- 172.21.15.60:0/2136763237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de815cd70 0x7f3de80d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.029+0000 7f3ded246640 1 -- 172.21.15.60:0/2136763237 >> v2:172.21.15.144:3300/0 conn(0x7f3de80b8480 msgr2=0x7f3de80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:00.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.029+0000 7f3ded246640 1 --2- 172.21.15.60:0/2136763237 >> v2:172.21.15.144:3300/0 conn(0x7f3de80b8480 0x7f3de80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.029+0000 7f3ded246640 1 -- 172.21.15.60:0/2136763237 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de80d1740 con 0x7f3de80d0be0 2023-10-16T12:35:00.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.030+0000 7f3ded246640 1 --2- 172.21.15.60:0/2136763237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80d0be0 0x7f3de80cdd00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3de8066ab0 tx=0x7f3de002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81783a81f2ec6c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:00.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.030+0000 7f3de77fe640 1 -- 172.21.15.60:0/2136763237 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0031860 con 0x7f3de80d0be0 2023-10-16T12:35:00.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.030+0000 7f3de77fe640 1 -- 172.21.15.60:0/2136763237 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de00319c0 con 0x7f3de80d0be0 2023-10-16T12:35:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.030+0000 7f3de77fe640 1 -- 172.21.15.60:0/2136763237 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0031d50 con 0x7f3de80d0be0 2023-10-16T12:35:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 -- 172.21.15.60:0/2136763237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80d0be0 msgr2=0x7f3de80cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 --2- 172.21.15.60:0/2136763237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80d0be0 0x7f3de80cdd00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3de8066ab0 tx=0x7f3de002b900 comp rx=0 tx=0).stop 2023-10-16T12:35:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 -- 172.21.15.60:0/2136763237 shutdown_connections 2023-10-16T12:35:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 --2- 172.21.15.60:0/2136763237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80d0be0 0x7f3de80cdd00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 --2- 172.21.15.60:0/2136763237 >> v2:172.21.15.144:3300/0 conn(0x7f3de80b8480 0x7f3de80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 --2- 172.21.15.60:0/2136763237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de815cd70 0x7f3de80d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 -- 172.21.15.60:0/2136763237 >> 172.21.15.60:0/2136763237 conn(0x7f3de80cf570 msgr2=0x7f3de80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:00.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 -- 172.21.15.60:0/2136763237 shutdown_connections 2023-10-16T12:35:00.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.031+0000 7f3deda47640 1 -- 172.21.15.60:0/2136763237 wait complete. 2023-10-16T12:35:00.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.033+0000 7f3deda47640 1 Processor -- start 2023-10-16T12:35:00.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.033+0000 7f3deda47640 1 -- start start 2023-10-16T12:35:00.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deda47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 0x7f3de8154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deda47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de80d0be0 0x7f3de8155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deda47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de815cd70 0x7f3de815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deda47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8155650 con 0x7f3de80b8480 2023-10-16T12:35:00.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deda47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de81558b0 con 0x7f3de815cd70 2023-10-16T12:35:00.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deda47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de80c64f0 con 0x7f3de80d0be0 2023-10-16T12:35:00.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 0x7f3de8154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 0x7f3de8154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45978/0 (socket says 172.21.15.60:45978) 2023-10-16T12:35:00.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 -- 172.21.15.60:0/1747637894 learned_addr learned my addr 172.21.15.60:0/1747637894 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:00.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 -- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de80d0be0 msgr2=0x7f3de8155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de80d0be0 0x7f3de8155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 -- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de815cd70 msgr2=0x7f3de815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de815cd70 0x7f3de815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.034+0000 7f3deca45640 1 -- 172.21.15.60:0/1747637894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00094d0 con 0x7f3de80b8480 2023-10-16T12:35:00.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.035+0000 7f3deca45640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 0x7f3de8154120 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800d5f0 tx=0x7f3dd800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:00.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.035+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80125f0 con 0x7f3de80b8480 2023-10-16T12:35:00.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.035+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd8012750 con 0x7f3de80b8480 2023-10-16T12:35:00.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.035+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de80c67d0 con 0x7f3de80b8480 2023-10-16T12:35:00.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.035+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8012b00 con 0x7f3de80b8480 2023-10-16T12:35:00.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.035+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de80c6d90 con 0x7f3de80b8480 2023-10-16T12:35:00.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.036+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de80c6f50 con 0x7f3de80b8480 2023-10-16T12:35:00.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.037+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3dd8012c60 con 0x7f3de80b8480 2023-10-16T12:35:00.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.038+0000 7f3de57fa640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3dd406ad10 0x7f3dd406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.038+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3dd80898c0 con 0x7f3de80b8480 2023-10-16T12:35:00.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.039+0000 7f3de7fff640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3dd406ad10 0x7f3dd406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.040+0000 7f3de7fff640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3dd406ad10 0x7f3dd406d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0070b0 tx=0x7f3ddc007040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:00.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.043+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3dd8054ae0 con 0x7f3de80b8480 2023-10-16T12:35:00.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.259+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3de80c6660 con 0x7f3dd406ad10 2023-10-16T12:35:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.424+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3dd8054eb0 con 0x7f3de80b8480 2023-10-16T12:35:00.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.471+0000 7f3de57fa640 1 -- 172.21.15.60:0/1747637894 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3de80c6660 con 0x7f3dd406ad10 2023-10-16T12:35:00.472 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-16T12:35:00.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3dd406ad10 msgr2=0x7f3dd406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3dd406ad10 0x7f3dd406d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0070b0 tx=0x7f3ddc007040 comp rx=0 tx=0).stop 2023-10-16T12:35:00.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 msgr2=0x7f3de8154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 0x7f3de8154120 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800d5f0 tx=0x7f3dd800dab0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 shutdown_connections 2023-10-16T12:35:00.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3dd406ad10 0x7f3dd406d1d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3de80d0be0 0x7f3de8155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3de815cd70 0x7f3de815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 --2- 172.21.15.60:0/1747637894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3de80b8480 0x7f3de8154120 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 >> 172.21.15.60:0/1747637894 conn(0x7f3de80cf570 msgr2=0x7f3de80b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:00.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 shutdown_connections 2023-10-16T12:35:00.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.474+0000 7f3deda47640 1 -- 172.21.15.60:0/1747637894 wait complete. 2023-10-16T12:35:00.495 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-16T12:35:00.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:00.496 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-16T12:35:00.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-10-16T12:35:00.739 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:35:00.736+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:35:00.739 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:35:00.736+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:35:00.739 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:35:00.736+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:35:00.739 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:35:00.736+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:35:00.740 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:35:00.736+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:35:00.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.777+0000 7efe605ce640 1 Processor -- start 2023-10-16T12:35:00.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.777+0000 7efe605ce640 1 -- start start 2023-10-16T12:35:00.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.779+0000 7efe605ce640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe58161400 0x7efe5806d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.779+0000 7efe605ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe581617e0 0x7efe5806c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.779+0000 7efe605ce640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580cf060 con 0x7efe5806b630 2023-10-16T12:35:00.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.779+0000 7efe605ce640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe580cf2e0 con 0x7efe58161400 2023-10-16T12:35:00.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.779+0000 7efe605ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe580cf7e0 con 0x7efe581617e0 2023-10-16T12:35:00.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.780+0000 7efe5eb6b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efe5806b630 0x7efe580705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41632/0 (socket says 172.21.15.60:41632) 2023-10-16T12:35:00.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.780+0000 7efe5eb6b640 1 -- 172.21.15.60:0/289916042 learned_addr learned my addr 172.21.15.60:0/289916042 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:00.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.780+0000 7efe5e36a640 1 --2- 172.21.15.60:0/289916042 >> v2:172.21.15.144:3300/0 conn(0x7efe58161400 0x7efe5806d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.780+0000 7efe5db69640 1 --2- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe581617e0 0x7efe5806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.781+0000 7efe5d368640 1 -- 172.21.15.60:0/289916042 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379426863 0 0) 0x7efe580cf060 con 0x7efe5806b630 2023-10-16T12:35:00.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.781+0000 7efe5d368640 1 -- 172.21.15.60:0/289916042 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44003520 con 0x7efe5806b630 2023-10-16T12:35:00.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.781+0000 7efe5e36a640 1 -- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe581617e0 msgr2=0x7efe5806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:00.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.781+0000 7efe5e36a640 1 --2- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe581617e0 0x7efe5806c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.781+0000 7efe5e36a640 1 -- 172.21.15.60:0/289916042 >> v1:172.21.15.60:6789/0 conn(0x7efe5806b630 legacy=0x7efe580705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.781+0000 7efe5e36a640 1 -- 172.21.15.60:0/289916042 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5806a850 con 0x7efe58161400 2023-10-16T12:35:00.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.782+0000 7efe5e36a640 1 --2- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe58161400 0x7efe5806d2e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe540097a0 tx=0x7efe5402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8943efaef14c2dee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:00.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.783+0000 7efe5d368640 1 -- 172.21.15.60:0/289916042 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5402d040 con 0x7efe58161400 2023-10-16T12:35:00.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.783+0000 7efe5d368640 1 -- 172.21.15.60:0/289916042 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe54031810 con 0x7efe58161400 2023-10-16T12:35:00.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.783+0000 7efe5d368640 1 -- 172.21.15.60:0/289916042 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54031b80 con 0x7efe58161400 2023-10-16T12:35:00.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 -- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe58161400 msgr2=0x7efe5806d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 --2- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe58161400 0x7efe5806d2e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe540097a0 tx=0x7efe5402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 -- 172.21.15.60:0/289916042 shutdown_connections 2023-10-16T12:35:00.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 --2- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe581617e0 0x7efe5806c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 --2- 172.21.15.60:0/289916042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe58161400 0x7efe5806d2e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 -- 172.21.15.60:0/289916042 >> 172.21.15.60:0/289916042 conn(0x7efe580cfac0 msgr2=0x7efe580cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:00.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 -- 172.21.15.60:0/289916042 shutdown_connections 2023-10-16T12:35:00.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.784+0000 7efe605ce640 1 -- 172.21.15.60:0/289916042 wait complete. 2023-10-16T12:35:00.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.786+0000 7efe605ce640 1 Processor -- start 2023-10-16T12:35:00.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.786+0000 7efe605ce640 1 -- start start 2023-10-16T12:35:00.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe605ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5806b630 0x7efe58149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe605ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 0x7efe5815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe605ce640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe581617e0 0x7efe5815d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe605ce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580c4870 con 0x7efe58161400 2023-10-16T12:35:00.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe605ce640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe581f6b80 con 0x7efe581617e0 2023-10-16T12:35:00.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe605ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe581603c0 con 0x7efe5806b630 2023-10-16T12:35:00.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe5db69640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 0x7efe5815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe5db69640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 0x7efe5815d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45988/0 (socket says 172.21.15.60:45988) 2023-10-16T12:35:00.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe5db69640 1 -- 172.21.15.60:0/3273776301 learned_addr learned my addr 172.21.15.60:0/3273776301 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:00.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe5eb6b640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe581617e0 0x7efe5815d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.787+0000 7efe5e36a640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5806b630 0x7efe58149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5db69640 1 -- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5806b630 msgr2=0x7efe58149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5db69640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5806b630 0x7efe58149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5db69640 1 -- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe581617e0 msgr2=0x7efe5815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:00.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5db69640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe581617e0 0x7efe5815d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:00.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5db69640 1 -- 172.21.15.60:0/3273776301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54009400 con 0x7efe58161400 2023-10-16T12:35:00.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5eb6b640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe581617e0 0x7efe5815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:35:00.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.788+0000 7efe5db69640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 0x7efe5815d270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efe4800e840 tx=0x7efe4800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:00.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.789+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe480094f0 con 0x7efe58161400 2023-10-16T12:35:00.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.789+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800c940 con 0x7efe58161400 2023-10-16T12:35:00.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.789+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800ccd0 con 0x7efe58161400 2023-10-16T12:35:00.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.789+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe581606a0 con 0x7efe58161400 2023-10-16T12:35:00.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.789+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5815f690 con 0x7efe58161400 2023-10-16T12:35:00.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.791+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe580c4640 con 0x7efe58161400 2023-10-16T12:35:00.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.792+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efe48013470 con 0x7efe58161400 2023-10-16T12:35:00.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.793+0000 7efe4effd640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7efe3c06afe0 0x7efe3c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:00.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.793+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7efe480899a0 con 0x7efe58161400 2023-10-16T12:35:00.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.793+0000 7efe5e36a640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7efe3c06afe0 0x7efe3c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:00.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.795+0000 7efe5e36a640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7efe3c06afe0 0x7efe3c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe54036000 tx=0x7efe5403c040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:00.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:00.799+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe580c4640 con 0x7efe58161400 2023-10-16T12:35:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.041+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efe5806ab30 con 0x7efe3c06afe0 2023-10-16T12:35:01.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.343+0000 7efe4effd640 1 -- 172.21.15.60:0/3273776301 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7efe54031b00 con 0x7efe3c06afe0 2023-10-16T12:35:01.345 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-16T12:35:01.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7efe3c06afe0 msgr2=0x7efe3c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:01.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7efe3c06afe0 0x7efe3c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe54036000 tx=0x7efe5403c040 comp rx=0 tx=0).stop 2023-10-16T12:35:01.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 msgr2=0x7efe5815d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:01.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 0x7efe5815d270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efe4800e840 tx=0x7efe4800ed00 comp rx=0 tx=0).stop 2023-10-16T12:35:01.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 shutdown_connections 2023-10-16T12:35:01.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7efe3c06afe0 0x7efe3c06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe581617e0 0x7efe5815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe58161400 0x7efe5815d270 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 --2- 172.21.15.60:0/3273776301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5806b630 0x7efe58149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 >> 172.21.15.60:0/3273776301 conn(0x7efe580cfac0 msgr2=0x7efe581629d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 shutdown_connections 2023-10-16T12:35:01.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.347+0000 7efe605ce640 1 -- 172.21.15.60:0/3273776301 wait complete. 2023-10-16T12:35:01.370 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-16T12:35:01.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:01.371 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-16T12:35:01.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-10-16T12:35:01.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.654+0000 7fbbe2213640 1 Processor -- start 2023-10-16T12:35:01.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.654+0000 7fbbe2213640 1 -- start start 2023-10-16T12:35:01.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.655+0000 7fbbe2213640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbbdc15cd70 0x7fbbdc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:01.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.655+0000 7fbbe2213640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbbdc0cded0 0x7fbbdc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:01.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.655+0000 7fbbe2213640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbdc15d150 con 0x7fbbdc0b8480 2023-10-16T12:35:01.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.655+0000 7fbbe2213640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbdc0d1180 con 0x7fbbdc15cd70 2023-10-16T12:35:01.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe2213640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbdc0d1460 con 0x7fbbdc0cded0 2023-10-16T12:35:01.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbbdc15cd70 0x7fbbdc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:01.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbbdc15cd70 0x7fbbdc0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46000/0 (socket says 172.21.15.60:46000) 2023-10-16T12:35:01.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 -- 172.21.15.60:0/2579376385 learned_addr learned my addr 172.21.15.60:0/2579376385 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:01.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 -- 172.21.15.60:0/2579376385 >> v2:172.21.15.144:3300/0 conn(0x7fbbdc0cded0 msgr2=0x7fbbdc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:01.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 --2- 172.21.15.60:0/2579376385 >> v2:172.21.15.144:3300/0 conn(0x7fbbdc0cded0 0x7fbbdc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 -- 172.21.15.60:0/2579376385 >> v1:172.21.15.144:6789/0 conn(0x7fbbdc0b8480 legacy=0x7fbbdc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:01.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.656+0000 7fbbe1211640 1 -- 172.21.15.60:0/2579376385 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc0d1740 con 0x7fbbdc15cd70 2023-10-16T12:35:01.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.657+0000 7fbbe1211640 1 --2- 172.21.15.60:0/2579376385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc15cd70 0x7fbbdc0d2220 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0b8910 tx=0x7fbbd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3414151f54d40e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:01.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.658+0000 7fbbd3fff640 1 -- 172.21.15.60:0/2579376385 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8031870 con 0x7fbbdc15cd70 2023-10-16T12:35:01.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.658+0000 7fbbd3fff640 1 -- 172.21.15.60:0/2579376385 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd80319d0 con 0x7fbbdc15cd70 2023-10-16T12:35:01.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.658+0000 7fbbd3fff640 1 -- 172.21.15.60:0/2579376385 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8031d60 con 0x7fbbdc15cd70 2023-10-16T12:35:01.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.658+0000 7fbbe2213640 1 -- 172.21.15.60:0/2579376385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc15cd70 msgr2=0x7fbbdc0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:01.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.658+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2579376385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc15cd70 0x7fbbdc0d2220 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0b8910 tx=0x7fbbd802b910 comp rx=0 tx=0).stop 2023-10-16T12:35:01.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.659+0000 7fbbe2213640 1 -- 172.21.15.60:0/2579376385 shutdown_connections 2023-10-16T12:35:01.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.659+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2579376385 >> v2:172.21.15.144:3300/0 conn(0x7fbbdc0cded0 0x7fbbdc0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.659+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2579376385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc15cd70 0x7fbbdc0d2220 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.659+0000 7fbbe2213640 1 -- 172.21.15.60:0/2579376385 >> 172.21.15.60:0/2579376385 conn(0x7fbbdc0cf570 msgr2=0x7fbbdc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:01.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.659+0000 7fbbe2213640 1 -- 172.21.15.60:0/2579376385 shutdown_connections 2023-10-16T12:35:01.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.659+0000 7fbbe2213640 1 -- 172.21.15.60:0/2579376385 wait complete. 2023-10-16T12:35:01.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.660+0000 7fbbe2213640 1 Processor -- start 2023-10-16T12:35:01.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.660+0000 7fbbe2213640 1 -- start start 2023-10-16T12:35:01.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.661+0000 7fbbe2213640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 0x7fbbdc158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:01.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.661+0000 7fbbe2213640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbdc0cded0 0x7fbbdc159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:01.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.661+0000 7fbbe2213640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbdc15cd70 0x7fbbdc0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:01.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.661+0000 7fbbe2213640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc1599b0 con 0x7fbbdc0b8480 2023-10-16T12:35:01.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.661+0000 7fbbe2213640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc1f2850 con 0x7fbbdc15cd70 2023-10-16T12:35:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.661+0000 7fbbe2213640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc1f2b30 con 0x7fbbdc0cded0 2023-10-16T12:35:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 0x7fbbdc158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 0x7fbbdc158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46006/0 (socket says 172.21.15.60:46006) 2023-10-16T12:35:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 -- 172.21.15.60:0/2016432235 learned_addr learned my addr 172.21.15.60:0/2016432235 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 -- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbdc0cded0 msgr2=0x7fbbdc159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbdc0cded0 0x7fbbdc159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 -- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbdc15cd70 msgr2=0x7fbbdc0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbdc15cd70 0x7fbbdc0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 -- 172.21.15.60:0/2016432235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd80094d0 con 0x7fbbdc0b8480 2023-10-16T12:35:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.662+0000 7fbbe1211640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 0x7fbbdc158df0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8000c00 tx=0x7fbbd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.663+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8031840 con 0x7fbbdc0b8480 2023-10-16T12:35:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.663+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd80443f0 con 0x7fbbdc0b8480 2023-10-16T12:35:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.663+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80447a0 con 0x7fbbdc0b8480 2023-10-16T12:35:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.663+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbdc1f2e10 con 0x7fbbdc0b8480 2023-10-16T12:35:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.663+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbdc1f24f0 con 0x7fbbdc0b8480 2023-10-16T12:35:01.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.664+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbdc1f3050 con 0x7fbbdc0b8480 2023-10-16T12:35:01.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.665+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fbbd804d450 con 0x7fbbdc0b8480 2023-10-16T12:35:01.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.666+0000 7fbbd1ffb640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbbc006afe0 0x7fbbc006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:01.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.666+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbbd802d040 con 0x7fbbdc0b8480 2023-10-16T12:35:01.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.666+0000 7fbbe0a10640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbbc006afe0 0x7fbbc006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:01.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.668+0000 7fbbe0a10640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbbc006afe0 0x7fbbc006d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0b0c20 tx=0x7fbbcc0079b0 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:01.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.670+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbdc1f3050 con 0x7fbbdc0b8480 2023-10-16T12:35:01.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:01.890+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbdc1f29c0 con 0x7fbbc006afe0 2023-10-16T12:35:02.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.088+0000 7fbbd1ffb640 1 -- 172.21.15.60:0/2016432235 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbbdc1f29c0 con 0x7fbbc006afe0 2023-10-16T12:35:02.090 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-16T12:35:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbbc006afe0 msgr2=0x7fbbc006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbbc006afe0 0x7fbbc006d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0b0c20 tx=0x7fbbcc0079b0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 msgr2=0x7fbbdc158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 0x7fbbdc158df0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8000c00 tx=0x7fbbd80046c0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 shutdown_connections 2023-10-16T12:35:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbbc006afe0 0x7fbbc006d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbbdc0cded0 0x7fbbdc159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbbdc15cd70 0x7fbbdc0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 --2- 172.21.15.60:0/2016432235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbdc0b8480 0x7fbbdc158df0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.091+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 >> 172.21.15.60:0/2016432235 conn(0x7fbbdc0cf570 msgr2=0x7fbbdc0b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.092+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 shutdown_connections 2023-10-16T12:35:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.092+0000 7fbbe2213640 1 -- 172.21.15.60:0/2016432235 wait complete. 2023-10-16T12:35:02.113 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-16T12:35:02.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:02.114 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-16T12:35:02.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-10-16T12:35:02.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.399+0000 7fdd424ec640 1 Processor -- start 2023-10-16T12:35:02.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.399+0000 7fdd424ec640 1 -- start start 2023-10-16T12:35:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.400+0000 7fdd424ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd3c0cf710 0x7fdd3c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:02.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd424ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd3c0b6cd0 0x7fdd3c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd424ec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3c0d3980 con 0x7fdd3c0cfaf0 2023-10-16T12:35:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd424ec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3c0d3c60 con 0x7fdd3c0b6cd0 2023-10-16T12:35:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd424ec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3c0ce630 con 0x7fdd3c0cf710 2023-10-16T12:35:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd40a89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd3c0b6cd0 0x7fdd3c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:02.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd40a89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd3c0b6cd0 0x7fdd3c0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46008/0 (socket says 172.21.15.60:46008) 2023-10-16T12:35:02.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd40a89640 1 -- 172.21.15.60:0/3633742269 learned_addr learned my addr 172.21.15.60:0/3633742269 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:02.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd3bfff640 1 --2- 172.21.15.60:0/3633742269 >> v2:172.21.15.144:3300/0 conn(0x7fdd3c0cf710 0x7fdd3c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:02.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.401+0000 7fdd40a89640 1 -- 172.21.15.60:0/3633742269 >> v2:172.21.15.144:3300/0 conn(0x7fdd3c0cf710 msgr2=0x7fdd3c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:02.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.402+0000 7fdd40a89640 1 --2- 172.21.15.60:0/3633742269 >> v2:172.21.15.144:3300/0 conn(0x7fdd3c0cf710 0x7fdd3c0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.402+0000 7fdd40a89640 1 -- 172.21.15.60:0/3633742269 >> v1:172.21.15.144:6789/0 conn(0x7fdd3c0cfaf0 legacy=0x7fdd3c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:02.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.402+0000 7fdd40a89640 1 -- 172.21.15.60:0/3633742269 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0ce910 con 0x7fdd3c0b6cd0 2023-10-16T12:35:02.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.402+0000 7fdd3bfff640 1 --2- 172.21.15.60:0/3633742269 >> v2:172.21.15.144:3300/0 conn(0x7fdd3c0cf710 0x7fdd3c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:02.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.403+0000 7fdd40a89640 1 --2- 172.21.15.60:0/3633742269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0b6cd0 0x7fdd3c0b3250 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0d2c20 tx=0x7fdd3402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbc4db8a8359bb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:02.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.403+0000 7fdd3affd640 1 -- 172.21.15.60:0/3633742269 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd340317e0 con 0x7fdd3c0b6cd0 2023-10-16T12:35:02.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.403+0000 7fdd3affd640 1 -- 172.21.15.60:0/3633742269 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34031940 con 0x7fdd3c0b6cd0 2023-10-16T12:35:02.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.403+0000 7fdd3affd640 1 -- 172.21.15.60:0/3633742269 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34031cd0 con 0x7fdd3c0b6cd0 2023-10-16T12:35:02.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 -- 172.21.15.60:0/3633742269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0b6cd0 msgr2=0x7fdd3c0b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 --2- 172.21.15.60:0/3633742269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0b6cd0 0x7fdd3c0b3250 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0d2c20 tx=0x7fdd3402b880 comp rx=0 tx=0).stop 2023-10-16T12:35:02.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 -- 172.21.15.60:0/3633742269 shutdown_connections 2023-10-16T12:35:02.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 --2- 172.21.15.60:0/3633742269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0b6cd0 0x7fdd3c0b3250 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 --2- 172.21.15.60:0/3633742269 >> v2:172.21.15.144:3300/0 conn(0x7fdd3c0cf710 0x7fdd3c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 -- 172.21.15.60:0/3633742269 >> 172.21.15.60:0/3633742269 conn(0x7fdd3c0d2cc0 msgr2=0x7fdd3c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:02.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 -- 172.21.15.60:0/3633742269 shutdown_connections 2023-10-16T12:35:02.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.404+0000 7fdd424ec640 1 -- 172.21.15.60:0/3633742269 wait complete. 2023-10-16T12:35:02.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.406+0000 7fdd424ec640 1 Processor -- start 2023-10-16T12:35:02.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.406+0000 7fdd424ec640 1 -- start start 2023-10-16T12:35:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd424ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd3c0b6cd0 0x7fdd3c15eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd424ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 0x7fdd3c0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd424ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0cfaf0 0x7fdd3c09e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd424ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0cab80 con 0x7fdd3c0cf710 2023-10-16T12:35:02.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd424ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0cae60 con 0x7fdd3c0cfaf0 2023-10-16T12:35:02.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd424ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0cb140 con 0x7fdd3c0b6cd0 2023-10-16T12:35:02.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.407+0000 7fdd3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 0x7fdd3c0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:02.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 0x7fdd3c0cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46022/0 (socket says 172.21.15.60:46022) 2023-10-16T12:35:02.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 -- 172.21.15.60:0/779497036 learned_addr learned my addr 172.21.15.60:0/779497036 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:02.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd40a89640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0cfaf0 0x7fdd3c09e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:02.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 -- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd3c0b6cd0 msgr2=0x7fdd3c15eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd3c0b6cd0 0x7fdd3c15eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 -- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0cfaf0 msgr2=0x7fdd3c09e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0cfaf0 0x7fdd3c09e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 -- 172.21.15.60:0/779497036 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340094d0 con 0x7fdd3c0cf710 2023-10-16T12:35:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.408+0000 7fdd3b7fe640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 0x7fdd3c0cc620 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400c430 tx=0x7fdd2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.409+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24011790 con 0x7fdd3c0cf710 2023-10-16T12:35:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.409+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd240118f0 con 0x7fdd3c0cf710 2023-10-16T12:35:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.409+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24011ce0 con 0x7fdd3c0cf710 2023-10-16T12:35:02.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.409+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c0cb420 con 0x7fdd3c0cf710 2023-10-16T12:35:02.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.409+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c15f0e0 con 0x7fdd3c0cf710 2023-10-16T12:35:02.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.410+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c1f2400 con 0x7fdd3c0cf710 2023-10-16T12:35:02.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.411+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fdd2401c460 con 0x7fdd3c0cf710 2023-10-16T12:35:02.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.412+0000 7fdd38ff9640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdd1c06afe0 0x7fdd1c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:02.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.412+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdd24089860 con 0x7fdd3c0cf710 2023-10-16T12:35:02.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.413+0000 7fdd3bfff640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdd1c06afe0 0x7fdd1c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:02.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.414+0000 7fdd3bfff640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdd1c06afe0 0x7fdd1c06d4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0cc2d0 tx=0x7fdd2c008380 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:02.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.417+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd240546f0 con 0x7fdd3c0cf710 2023-10-16T12:35:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.451+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fdd24054ac0 con 0x7fdd3c0cf710 2023-10-16T12:35:02.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.636+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd3c0d1d50 con 0x7fdd1c06afe0 2023-10-16T12:35:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.831+0000 7fdd38ff9640 1 -- 172.21.15.60:0/779497036 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fdd2c012070 con 0x7fdd1c06afe0 2023-10-16T12:35:02.833 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-16T12:35:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdd1c06afe0 msgr2=0x7fdd1c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdd1c06afe0 0x7fdd1c06d4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0cc2d0 tx=0x7fdd2c008380 comp rx=0 tx=0).stop 2023-10-16T12:35:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 msgr2=0x7fdd3c0cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 0x7fdd3c0cc620 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400c430 tx=0x7fdd2400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 shutdown_connections 2023-10-16T12:35:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdd1c06afe0 0x7fdd1c06d4a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0cfaf0 0x7fdd3c09e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd3c0cf710 0x7fdd3c0cc620 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 --2- 172.21.15.60:0/779497036 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd3c0b6cd0 0x7fdd3c15eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 >> 172.21.15.60:0/779497036 conn(0x7fdd3c0d2cc0 msgr2=0x7fdd3c0d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:02.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 shutdown_connections 2023-10-16T12:35:02.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:02.835+0000 7fdd424ec640 1 -- 172.21.15.60:0/779497036 wait complete. 2023-10-16T12:35:02.860 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-16T12:35:02.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:02.861 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-16T12:35:02.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-10-16T12:35:03.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.156+0000 7fcd9084d640 1 Processor -- start 2023-10-16T12:35:03.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.156+0000 7fcd9084d640 1 -- start start 2023-10-16T12:35:03.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.157+0000 7fcd9084d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcd8c06c040 0x7fcd8c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.157+0000 7fcd9084d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd8c0cda80 0x7fcd8c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.157+0000 7fcd9084d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c06c960 con 0x7fcd8c0ad970 2023-10-16T12:35:03.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.157+0000 7fcd9084d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8c15b9d0 con 0x7fcd8c0cda80 2023-10-16T12:35:03.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.157+0000 7fcd9084d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8c15bcb0 con 0x7fcd8c06c040 2023-10-16T12:35:03.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd8c0cda80 0x7fcd8c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd8c0ad970 0x7fcd8c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41646/0 (socket says 172.21.15.60:41646) 2023-10-16T12:35:03.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8affd640 1 -- 172.21.15.60:0/529219930 learned_addr learned my addr 172.21.15.60:0/529219930 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:03.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8b7fe640 1 --2- 172.21.15.60:0/529219930 >> v2:172.21.15.144:3300/0 conn(0x7fcd8c06c040 0x7fcd8c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8bfff640 1 -- 172.21.15.60:0/529219930 >> v2:172.21.15.144:3300/0 conn(0x7fcd8c06c040 msgr2=0x7fcd8c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:03.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8bfff640 1 --2- 172.21.15.60:0/529219930 >> v2:172.21.15.144:3300/0 conn(0x7fcd8c06c040 0x7fcd8c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8bfff640 1 -- 172.21.15.60:0/529219930 >> v1:172.21.15.60:6789/0 conn(0x7fcd8c0ad970 legacy=0x7fcd8c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8bfff640 1 -- 172.21.15.60:0/529219930 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c15bf90 con 0x7fcd8c0cda80 2023-10-16T12:35:03.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.158+0000 7fcd8b7fe640 1 --2- 172.21.15.60:0/529219930 >> v2:172.21.15.144:3300/0 conn(0x7fcd8c06c040 0x7fcd8c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:03.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.159+0000 7fcd8bfff640 1 --2- 172.21.15.60:0/529219930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0cda80 0x7fcd8c15ed90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0cf350 tx=0x7fcd8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e298c59dad4f032 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:03.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.160+0000 7fcd8a7fc640 1 -- 172.21.15.60:0/529219930 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84031830 con 0x7fcd8c0cda80 2023-10-16T12:35:03.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.160+0000 7fcd8a7fc640 1 -- 172.21.15.60:0/529219930 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd84031990 con 0x7fcd8c0cda80 2023-10-16T12:35:03.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.160+0000 7fcd8a7fc640 1 -- 172.21.15.60:0/529219930 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84031d20 con 0x7fcd8c0cda80 2023-10-16T12:35:03.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.160+0000 7fcd9084d640 1 -- 172.21.15.60:0/529219930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0cda80 msgr2=0x7fcd8c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.160+0000 7fcd9084d640 1 --2- 172.21.15.60:0/529219930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0cda80 0x7fcd8c15ed90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0cf350 tx=0x7fcd8402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.161+0000 7fcd9084d640 1 -- 172.21.15.60:0/529219930 shutdown_connections 2023-10-16T12:35:03.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.161+0000 7fcd9084d640 1 --2- 172.21.15.60:0/529219930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0cda80 0x7fcd8c15ed90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.161+0000 7fcd9084d640 1 --2- 172.21.15.60:0/529219930 >> v2:172.21.15.144:3300/0 conn(0x7fcd8c06c040 0x7fcd8c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.161+0000 7fcd9084d640 1 -- 172.21.15.60:0/529219930 >> 172.21.15.60:0/529219930 conn(0x7fcd8c0ce6a0 msgr2=0x7fcd8c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:03.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.161+0000 7fcd9084d640 1 -- 172.21.15.60:0/529219930 shutdown_connections 2023-10-16T12:35:03.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.161+0000 7fcd9084d640 1 -- 172.21.15.60:0/529219930 wait complete. 2023-10-16T12:35:03.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.162+0000 7fcd9084d640 1 Processor -- start 2023-10-16T12:35:03.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.163+0000 7fcd9084d640 1 -- start start 2023-10-16T12:35:03.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd9084d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd8c06c040 0x7fcd8c0c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd9084d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 0x7fcd8c14db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd9084d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd8c0cda80 0x7fcd8c0b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd9084d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0c8100 con 0x7fcd8c0ad970 2023-10-16T12:35:03.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd9084d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0c9e20 con 0x7fcd8c06c040 2023-10-16T12:35:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd9084d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c14e070 con 0x7fcd8c0cda80 2023-10-16T12:35:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 0x7fcd8c14db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 0x7fcd8c14db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46040/0 (socket says 172.21.15.60:46040) 2023-10-16T12:35:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 -- 172.21.15.60:0/632769890 learned_addr learned my addr 172.21.15.60:0/632769890 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:03.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 -- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd8c0cda80 msgr2=0x7fcd8c0b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd8c0cda80 0x7fcd8c0b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 -- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd8c06c040 msgr2=0x7fcd8c0c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8b7fe640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd8c06c040 0x7fcd8c0c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd8c06c040 0x7fcd8c0c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.164+0000 7fcd8affd640 1 -- 172.21.15.60:0/632769890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84009400 con 0x7fcd8c0ad970 2023-10-16T12:35:03.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.165+0000 7fcd8affd640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 0x7fcd8c14db30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c500 tx=0x7fcd7400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:03.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.165+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74005ce0 con 0x7fcd8c0ad970 2023-10-16T12:35:03.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.165+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd74005e40 con 0x7fcd8c0ad970 2023-10-16T12:35:03.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.165+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd740072f0 con 0x7fcd8c0ad970 2023-10-16T12:35:03.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.165+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c0b37d0 con 0x7fcd8c0ad970 2023-10-16T12:35:03.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.165+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c0c90e0 con 0x7fcd8c0ad970 2023-10-16T12:35:03.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.166+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c06b330 con 0x7fcd8c0ad970 2023-10-16T12:35:03.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.167+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcd74007450 con 0x7fcd8c0ad970 2023-10-16T12:35:03.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.168+0000 7fcd83fff640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fcd6c06b150 0x7fcd6c06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.168+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fcd74088f40 con 0x7fcd8c0ad970 2023-10-16T12:35:03.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.169+0000 7fcd8b7fe640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fcd6c06b150 0x7fcd6c06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.170+0000 7fcd8b7fe640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fcd6c06b150 0x7fcd6c06d610 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c008f80 tx=0x7fcd7c0084d0 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:03.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.172+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd740bbfe0 con 0x7fcd8c0ad970 2023-10-16T12:35:03.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.391+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd8c0cc550 con 0x7fcd6c06b150 2023-10-16T12:35:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.588+0000 7fcd83fff640 1 -- 172.21.15.60:0/632769890 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fcd7c008b90 con 0x7fcd6c06b150 2023-10-16T12:35:03.590 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-16T12:35:03.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fcd6c06b150 msgr2=0x7fcd6c06d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fcd6c06b150 0x7fcd6c06d610 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c008f80 tx=0x7fcd7c0084d0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 msgr2=0x7fcd8c14db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 0x7fcd8c14db30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c500 tx=0x7fcd7400c9c0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 shutdown_connections 2023-10-16T12:35:03.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fcd6c06b150 0x7fcd6c06d610 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd8c0cda80 0x7fcd8c0b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd8c0ad970 0x7fcd8c14db30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 --2- 172.21.15.60:0/632769890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd8c06c040 0x7fcd8c0c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.591+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 >> 172.21.15.60:0/632769890 conn(0x7fcd8c0ce6a0 msgr2=0x7fcd8c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.592+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 shutdown_connections 2023-10-16T12:35:03.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.592+0000 7fcd9084d640 1 -- 172.21.15.60:0/632769890 wait complete. 2023-10-16T12:35:03.615 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-16T12:35:03.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:03.616 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-16T12:35:03.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-10-16T12:35:03.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.908+0000 7f4cd23e9640 1 Processor -- start 2023-10-16T12:35:03.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.909+0000 7f4cd23e9640 1 -- start start 2023-10-16T12:35:03.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.910+0000 7f4cd23e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4ccc15b320 0x7f4ccc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.910+0000 7f4cd23e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc15b9d0 con 0x7f4ccc06c040 2023-10-16T12:35:03.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.910+0000 7f4cd23e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc15bcb0 con 0x7f4ccc0ad970 2023-10-16T12:35:03.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.910+0000 7f4cd23e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc15b7d0 con 0x7f4ccc15b320 2023-10-16T12:35:03.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.911+0000 7f4cd13e7640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ccc06c040 0x7f4ccc06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41660/0 (socket says 172.21.15.60:41660) 2023-10-16T12:35:03.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.911+0000 7f4cd13e7640 1 -- 172.21.15.60:0/1164834281 learned_addr learned my addr 172.21.15.60:0/1164834281 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:03.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.911+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/1164834281 >> v2:172.21.15.144:3300/0 conn(0x7f4ccc15b320 0x7f4ccc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.911+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103642881 0 0) 0x7f4ccc15b9d0 con 0x7f4ccc06c040 2023-10-16T12:35:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.911+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cbc003520 con 0x7f4ccc06c040 2023-10-16T12:35:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1222976502 0 0) 0x7f4cbc003520 con 0x7f4ccc06c040 2023-10-16T12:35:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 >> v2:172.21.15.144:3300/0 conn(0x7f4ccc15b320 msgr2=0x7f4ccc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 --2- 172.21.15.60:0/1164834281 >> v2:172.21.15.144:3300/0 conn(0x7f4ccc15b320 0x7f4ccc0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 >> v1:172.21.15.144:6789/0 conn(0x7f4ccc0ad970 legacy=0x7f4ccc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc15bf90 con 0x7f4ccc06c040 2023-10-16T12:35:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4cc00019f0 con 0x7f4ccc06c040 2023-10-16T12:35:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 >> v1:172.21.15.60:6789/0 conn(0x7f4ccc06c040 legacy=0x7f4ccc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cbc0039a0 0x7f4cbc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cbc0047c0 0x7f4cbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4cbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc15b7d0 con 0x7f4cbc005180 2023-10-16T12:35:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc00c050 con 0x7f4cbc0039a0 2023-10-16T12:35:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc00c210 con 0x7f4cbc0047c0 2023-10-16T12:35:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.912+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4cbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.913+0000 7f4cd1be8640 1 -- 172.21.15.60:0/1164834281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cbc0047c0 msgr2=0x7f4cbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.913+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cbc0047c0 0x7f4cbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.913+0000 7f4cd1be8640 1 -- 172.21.15.60:0/1164834281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cbc0039a0 msgr2=0x7f4cbc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.913+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cbc0039a0 0x7f4cbc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.913+0000 7f4cd1be8640 1 -- 172.21.15.60:0/1164834281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc003520 con 0x7f4cbc005180 2023-10-16T12:35:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4cd1be8640 1 -- 172.21.15.60:0/1164834281 reap_dead start 2023-10-16T12:35:03.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4cbc00b980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009ac0 tx=0x7f4cbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94098d43ae40834 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:03.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4004020 con 0x7f4cbc005180 2023-10-16T12:35:03.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4004180 con 0x7f4cbc005180 2023-10-16T12:35:03.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4ccbfff640 1 -- 172.21.15.60:0/1164834281 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40044f0 con 0x7f4cbc005180 2023-10-16T12:35:03.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4cd23e9640 1 -- 172.21.15.60:0/1164834281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 msgr2=0x7f4cbc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.914+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4cbc00b980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009ac0 tx=0x7f4cbc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.915+0000 7f4cd23e9640 1 -- 172.21.15.60:0/1164834281 shutdown_connections 2023-10-16T12:35:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.915+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/1164834281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4cbc00b980 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.915+0000 7f4cd23e9640 1 -- 172.21.15.60:0/1164834281 >> 172.21.15.60:0/1164834281 conn(0x7f4ccc0ce6a0 msgr2=0x7f4ccc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.915+0000 7f4cd23e9640 1 -- 172.21.15.60:0/1164834281 shutdown_connections 2023-10-16T12:35:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.915+0000 7f4cd23e9640 1 -- 172.21.15.60:0/1164834281 wait complete. 2023-10-16T12:35:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.917+0000 7f4cd23e9640 1 Processor -- start 2023-10-16T12:35:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.917+0000 7f4cd23e9640 1 -- start start 2023-10-16T12:35:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd23e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4ccc0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd23e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ccc0b3ad0 0x7f4ccc14e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd23e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ccc0b2880 0x7f4ccc0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd23e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0b3250 con 0x7f4cbc005180 2023-10-16T12:35:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd23e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0b4120 con 0x7f4ccc0b2880 2023-10-16T12:35:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd23e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc152750 con 0x7f4ccc0b3ad0 2023-10-16T12:35:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd13e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4ccc0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd13e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4ccc0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46060/0 (socket says 172.21.15.60:46060) 2023-10-16T12:35:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.918+0000 7f4cd13e7640 1 -- 172.21.15.60:0/3443893896 learned_addr learned my addr 172.21.15.60:0/3443893896 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd0be6640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ccc0b3ad0 0x7f4ccc14e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ccc0b2880 0x7f4ccc0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd13e7640 1 -- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ccc0b3ad0 msgr2=0x7f4ccc14e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd13e7640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ccc0b3ad0 0x7f4ccc14e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd13e7640 1 -- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ccc0b2880 msgr2=0x7f4ccc0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd13e7640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ccc0b2880 0x7f4ccc0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd13e7640 1 -- 172.21.15.60:0/3443893896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc0b36b0 con 0x7f4cbc005180 2023-10-16T12:35:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd1be8640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ccc0b2880 0x7f4ccc0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd0be6640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ccc0b3ad0 0x7f4ccc14e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd13e7640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4ccc0cdaf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000caa0 tx=0x7f4cc000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:03.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000a6d0 con 0x7f4cbc005180 2023-10-16T12:35:03.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc000a850 con 0x7f4cbc005180 2023-10-16T12:35:03.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000ac00 con 0x7f4cbc005180 2023-10-16T12:35:03.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.919+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc0aff20 con 0x7f4cbc005180 2023-10-16T12:35:03.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.920+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc0b04e0 con 0x7f4cbc005180 2023-10-16T12:35:03.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.921+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc06b330 con 0x7f4cbc005180 2023-10-16T12:35:03.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.922+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4cc00070d0 con 0x7f4cbc005180 2023-10-16T12:35:03.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.923+0000 7f4cc9ffb640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f4cb006b100 0x7f4cb006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:03.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.923+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4cc00893c0 con 0x7f4cbc005180 2023-10-16T12:35:03.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.924+0000 7f4cd0be6640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f4cb006b100 0x7f4cb006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:03.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.925+0000 7f4cd0be6640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f4cb006b100 0x7f4cb006d5c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0ade20 tx=0x7f4cb8009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:03.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:03.928+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4cc0053f70 con 0x7f4cbc005180 2023-10-16T12:35:04.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.147+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ccc0cc550 con 0x7f4cb006b100 2023-10-16T12:35:04.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.344+0000 7f4cc9ffb640 1 -- 172.21.15.60:0/3443893896 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4cb8001330 con 0x7f4cb006b100 2023-10-16T12:35:04.346 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-16T12:35:04.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.348+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f4cb006b100 msgr2=0x7f4cb006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:04.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f4cb006b100 0x7f4cb006d5c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0ade20 tx=0x7f4cb8009040 comp rx=0 tx=0).stop 2023-10-16T12:35:04.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 msgr2=0x7f4ccc0cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:04.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4ccc0cdaf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000caa0 tx=0x7f4cc000cea0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 shutdown_connections 2023-10-16T12:35:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f4cb006b100 0x7f4cb006d5c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ccc0b3ad0 0x7f4ccc14e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ccc0b2880 0x7f4ccc0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 --2- 172.21.15.60:0/3443893896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cbc005180 0x7f4ccc0cdaf0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 >> 172.21.15.60:0/3443893896 conn(0x7f4ccc0ce6a0 msgr2=0x7f4ccc0ca2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:04.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 shutdown_connections 2023-10-16T12:35:04.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.349+0000 7f4cd23e9640 1 -- 172.21.15.60:0/3443893896 wait complete. 2023-10-16T12:35:04.373 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-16T12:35:04.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:04.374 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-16T12:35:04.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-10-16T12:35:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.658+0000 7f396a8bb640 1 Processor -- start 2023-10-16T12:35:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.658+0000 7f396a8bb640 1 -- start start 2023-10-16T12:35:04.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f396a8bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 0x7f39640d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f396a8bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f39640cded0 0x7f39640b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f396a8bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396415d150 con 0x7f39640b8480 2023-10-16T12:35:04.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f396a8bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39640d1180 con 0x7f39640cded0 2023-10-16T12:35:04.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f396a8bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39640d1460 con 0x7f396415cd70 2023-10-16T12:35:04.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f39690b8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f39640b8480 0x7f39640cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41674/0 (socket says 172.21.15.60:41674) 2023-10-16T12:35:04.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f39690b8640 1 -- 172.21.15.60:0/3528698455 learned_addr learned my addr 172.21.15.60:0/3528698455 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:04.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f396a0ba640 1 --2- 172.21.15.60:0/3528698455 >> v2:172.21.15.144:3300/0 conn(0x7f39640cded0 0x7f39640b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.660+0000 7f39698b9640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 0x7f39640d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982912766 0 0) 0x7f396415d150 con 0x7f39640b8480 2023-10-16T12:35:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3950003520 con 0x7f39640b8480 2023-10-16T12:35:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567278368 0 0) 0x7f3950003520 con 0x7f39640b8480 2023-10-16T12:35:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 msgr2=0x7f39640d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 0x7f39640d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 >> v2:172.21.15.144:3300/0 conn(0x7f39640cded0 msgr2=0x7f39640b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 --2- 172.21.15.60:0/3528698455 >> v2:172.21.15.144:3300/0 conn(0x7f39640cded0 0x7f39640b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39640d1740 con 0x7f39640b8480 2023-10-16T12:35:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f3954001670 con 0x7f39640b8480 2023-10-16T12:35:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 >> v1:172.21.15.60:6789/0 conn(0x7f39640b8480 legacy=0x7f39640cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3950003970 0x7f3950003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39500047c0 0x7f3950004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 0x7f3950005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640d1180 con 0x7f39500047c0 2023-10-16T12:35:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640d1460 con 0x7f3950003970 2023-10-16T12:35:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.661+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395000b980 con 0x7f396415cd70 2023-10-16T12:35:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39690b8640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39500047c0 0x7f3950004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f396a0ba640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 0x7f3950005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39698b9640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3950003970 0x7f3950003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39690b8640 1 -- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 msgr2=0x7f3950005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39690b8640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f396415cd70 0x7f3950005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39690b8640 1 -- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3950003970 msgr2=0x7f3950003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39690b8640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3950003970 0x7f3950003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.662+0000 7f39690b8640 1 -- 172.21.15.60:0/3528698455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3950003520 con 0x7f39500047c0 2023-10-16T12:35:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.663+0000 7f39690b8640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39500047c0 0x7f3950004c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f395400c110 tx=0x7f39500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c32700715e9ae40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:04.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.663+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954005ad0 con 0x7f39500047c0 2023-10-16T12:35:04.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.663+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3954005c50 con 0x7f39500047c0 2023-10-16T12:35:04.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.663+0000 7f39688b7640 1 -- 172.21.15.60:0/3528698455 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395402c810 con 0x7f39500047c0 2023-10-16T12:35:04.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a8bb640 1 -- 172.21.15.60:0/3528698455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39500047c0 msgr2=0x7f3950004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:04.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a8bb640 1 --2- 172.21.15.60:0/3528698455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39500047c0 0x7f3950004c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f395400c110 tx=0x7f39500057f0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a0ba640 1 -- 172.21.15.60:0/3528698455 reap_dead start 2023-10-16T12:35:04.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a8bb640 1 -- 172.21.15.60:0/3528698455 shutdown_connections 2023-10-16T12:35:04.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a8bb640 1 -- 172.21.15.60:0/3528698455 >> 172.21.15.60:0/3528698455 conn(0x7f39640cf570 msgr2=0x7f39640cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:04.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a8bb640 1 -- 172.21.15.60:0/3528698455 shutdown_connections 2023-10-16T12:35:04.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.664+0000 7f396a8bb640 1 -- 172.21.15.60:0/3528698455 wait complete. 2023-10-16T12:35:04.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.666+0000 7f396a8bb640 1 Processor -- start 2023-10-16T12:35:04.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.666+0000 7f396a8bb640 1 -- start start 2023-10-16T12:35:04.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a8bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39500047c0 0x7f396415ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a8bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39640cc000 0x7f39640aa640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a8bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 0x7f396415a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a8bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39641f2520 con 0x7f39640cc480 2023-10-16T12:35:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a8bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39641f2800 con 0x7f39500047c0 2023-10-16T12:35:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a8bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39641f2ae0 con 0x7f39640cc000 2023-10-16T12:35:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a0ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 0x7f396415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a0ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 0x7f396415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46076/0 (socket says 172.21.15.60:46076) 2023-10-16T12:35:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.667+0000 7f396a0ba640 1 -- 172.21.15.60:0/946860870 learned_addr learned my addr 172.21.15.60:0/946860870 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:04.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f39698b9640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39500047c0 0x7f396415ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f39690b8640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39640cc000 0x7f39640aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f396a0ba640 1 -- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39640cc000 msgr2=0x7f39640aa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:04.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f396a0ba640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39640cc000 0x7f39640aa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f396a0ba640 1 -- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39500047c0 msgr2=0x7f396415ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:04.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f396a0ba640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39500047c0 0x7f396415ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:04.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f396a0ba640 1 -- 172.21.15.60:0/946860870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39641f2dc0 con 0x7f39640cc480 2023-10-16T12:35:04.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f396a0ba640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 0x7f396415a020 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f39640b56a0 tx=0x7f395c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:04.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.668+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c003990 con 0x7f39640cc480 2023-10-16T12:35:04.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.669+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c011040 con 0x7f39640cc480 2023-10-16T12:35:04.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.669+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c003d50 con 0x7f39640cc480 2023-10-16T12:35:04.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.669+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39641f30a0 con 0x7f39640cc480 2023-10-16T12:35:04.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.669+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39640aab80 con 0x7f39640cc480 2023-10-16T12:35:04.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.670+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396415d340 con 0x7f39640cc480 2023-10-16T12:35:04.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.671+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f395c003eb0 con 0x7f39640cc480 2023-10-16T12:35:04.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.672+0000 7f395a7fc640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f394806b100 0x7f394806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:04.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.673+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f395c08a240 con 0x7f39640cc480 2023-10-16T12:35:04.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.673+0000 7f39698b9640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f394806b100 0x7f394806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:04.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.674+0000 7f39698b9640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f394806b100 0x7f394806d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f39640cc930 tx=0x7f3960008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:04.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.677+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f396415d340 con 0x7f39640cc480 2023-10-16T12:35:04.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:04.894+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f39640d1d00 con 0x7f394806b100 2023-10-16T12:35:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.096+0000 7f395a7fc640 1 -- 172.21.15.60:0/946860870 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3960009700 con 0x7f394806b100 2023-10-16T12:35:05.097 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-16T12:35:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.099+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f394806b100 msgr2=0x7f394806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.099+0000 7f396a8bb640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f394806b100 0x7f394806d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f39640cc930 tx=0x7f3960008040 comp rx=0 tx=0).stop 2023-10-16T12:35:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 msgr2=0x7f396415a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 0x7f396415a020 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f39640b56a0 tx=0x7f395c00f4a0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 shutdown_connections 2023-10-16T12:35:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f394806b100 0x7f394806d5c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f39640cc000 0x7f39640aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39640cc480 0x7f396415a020 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 --2- 172.21.15.60:0/946860870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39500047c0 0x7f396415ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 >> 172.21.15.60:0/946860870 conn(0x7f39640cf570 msgr2=0x7f39640b2830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 shutdown_connections 2023-10-16T12:35:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.100+0000 7f396a8bb640 1 -- 172.21.15.60:0/946860870 wait complete. 2023-10-16T12:35:05.123 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-16T12:35:05.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:35:05.124 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-16T12:35:05.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-10-16T12:35:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.412+0000 7f9635a51640 1 Processor -- start 2023-10-16T12:35:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.412+0000 7f9635a51640 1 -- start start 2023-10-16T12:35:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.414+0000 7f9635a51640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96300addb0 0x7f96300caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.414+0000 7f9635a51640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96300c9000 0x7f9630069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f9635a51640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963006a6f0 con 0x7f963006a2e0 2023-10-16T12:35:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f9635a51640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9630161a50 con 0x7f96300addb0 2023-10-16T12:35:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f9635a51640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96300cb140 con 0x7f96300c9000 2023-10-16T12:35:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f962f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f963006a2e0 0x7f96300cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41690/0 (socket says 172.21.15.60:41690) 2023-10-16T12:35:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f962effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96300addb0 0x7f96300caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:05.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f962f7fe640 1 -- 172.21.15.60:0/3184299965 learned_addr learned my addr 172.21.15.60:0/3184299965 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f96267fc640 1 --2- 172.21.15.60:0/3184299965 >> v2:172.21.15.144:3300/0 conn(0x7f96300c9000 0x7f9630069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.415+0000 7f962e7fc640 1 -- 172.21.15.60:0/3184299965 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000022294 0 0) 0x7f963006a6f0 con 0x7f963006a2e0 2023-10-16T12:35:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.416+0000 7f962e7fc640 1 -- 172.21.15.60:0/3184299965 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c003520 con 0x7f963006a2e0 2023-10-16T12:35:05.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.416+0000 7f962effd640 1 -- 172.21.15.60:0/3184299965 >> v2:172.21.15.144:3300/0 conn(0x7f96300c9000 msgr2=0x7f9630069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.416+0000 7f962effd640 1 --2- 172.21.15.60:0/3184299965 >> v2:172.21.15.144:3300/0 conn(0x7f96300c9000 0x7f9630069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.416+0000 7f962effd640 1 -- 172.21.15.60:0/3184299965 >> v1:172.21.15.60:6789/0 conn(0x7f963006a2e0 legacy=0x7f96300cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.416+0000 7f962effd640 1 -- 172.21.15.60:0/3184299965 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963006b030 con 0x7f96300addb0 2023-10-16T12:35:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.417+0000 7f962effd640 1 --2- 172.21.15.60:0/3184299965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300caa80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f96300cdbe0 tx=0x7f962002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef0bc2ce1e22fde server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.417+0000 7f962e7fc640 1 -- 172.21.15.60:0/3184299965 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620031830 con 0x7f96300addb0 2023-10-16T12:35:05.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.417+0000 7f962e7fc640 1 -- 172.21.15.60:0/3184299965 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620031990 con 0x7f96300addb0 2023-10-16T12:35:05.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.417+0000 7f962e7fc640 1 -- 172.21.15.60:0/3184299965 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620031d00 con 0x7f96300addb0 2023-10-16T12:35:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 -- 172.21.15.60:0/3184299965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 msgr2=0x7f96300caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 --2- 172.21.15.60:0/3184299965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300caa80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f96300cdbe0 tx=0x7f962002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 -- 172.21.15.60:0/3184299965 shutdown_connections 2023-10-16T12:35:05.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 --2- 172.21.15.60:0/3184299965 >> v2:172.21.15.144:3300/0 conn(0x7f96300c9000 0x7f9630069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 --2- 172.21.15.60:0/3184299965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300caa80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 -- 172.21.15.60:0/3184299965 >> 172.21.15.60:0/3184299965 conn(0x7f96300ce9b0 msgr2=0x7f96300cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 -- 172.21.15.60:0/3184299965 shutdown_connections 2023-10-16T12:35:05.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.418+0000 7f9635a51640 1 -- 172.21.15.60:0/3184299965 wait complete. 2023-10-16T12:35:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.420+0000 7f9635a51640 1 Processor -- start 2023-10-16T12:35:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.420+0000 7f9635a51640 1 -- start start 2023-10-16T12:35:05.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f9635a51640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f963006a2e0 0x7f96300b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f9635a51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f9635a51640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96300c9000 0x7f96301f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f9635a51640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b3780 con 0x7f96300addb0 2023-10-16T12:35:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f9635a51640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b3a60 con 0x7f963006a2e0 2023-10-16T12:35:05.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f9635a51640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b3d40 con 0x7f96300c9000 2023-10-16T12:35:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f96267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f96267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46098/0 (socket says 172.21.15.60:46098) 2023-10-16T12:35:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.421+0000 7f96267fc640 1 -- 172.21.15.60:0/3142637893 learned_addr learned my addr 172.21.15.60:0/3142637893 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f962effd640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f963006a2e0 0x7f96300b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f96267fc640 1 -- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96300c9000 msgr2=0x7f96301f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f96267fc640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96300c9000 0x7f96301f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f96267fc640 1 -- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f963006a2e0 msgr2=0x7f96300b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f96267fc640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f963006a2e0 0x7f96300b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f96267fc640 1 -- 172.21.15.60:0/3142637893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620009400 con 0x7f96300addb0 2023-10-16T12:35:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f96267fc640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300b1ae0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f961800d5f0 tx=0x7f961800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:05.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96180125f0 con 0x7f96300addb0 2023-10-16T12:35:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618012750 con 0x7f96300addb0 2023-10-16T12:35:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618012b00 con 0x7f96300addb0 2023-10-16T12:35:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.422+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96300b4020 con 0x7f96300addb0 2023-10-16T12:35:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.423+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96300b4820 con 0x7f96300addb0 2023-10-16T12:35:05.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.424+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963004e160 con 0x7f96300addb0 2023-10-16T12:35:05.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.425+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9618012c60 con 0x7f96300addb0 2023-10-16T12:35:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.426+0000 7f962cff9640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f961006b100 0x7f961006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.426+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f961808a090 con 0x7f96300addb0 2023-10-16T12:35:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.426+0000 7f962effd640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f961006b100 0x7f961006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:05.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.427+0000 7f962effd640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f961006b100 0x7f961006d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9620036000 tx=0x7f9620008030 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:05.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.430+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9618054cd0 con 0x7f96300addb0 2023-10-16T12:35:05.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.649+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f96300b49e0 con 0x7f961006b100 2023-10-16T12:35:05.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.849+0000 7f962cff9640 1 -- 172.21.15.60:0/3142637893 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f96300b49e0 con 0x7f961006b100 2023-10-16T12:35:05.851 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-16T12:35:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f961006b100 msgr2=0x7f961006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f961006b100 0x7f961006d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9620036000 tx=0x7f9620008030 comp rx=0 tx=0).stop 2023-10-16T12:35:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 msgr2=0x7f96300b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300b1ae0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f961800d5f0 tx=0x7f961800dab0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 shutdown_connections 2023-10-16T12:35:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f961006b100 0x7f961006d5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96300c9000 0x7f96301f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96300addb0 0x7f96300b1ae0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 --2- 172.21.15.60:0/3142637893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f963006a2e0 0x7f96300b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.853+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 >> 172.21.15.60:0/3142637893 conn(0x7f96300ce9b0 msgr2=0x7f96300af450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.854+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 shutdown_connections 2023-10-16T12:35:05.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:05.854+0000 7f9635a51640 1 -- 172.21.15.60:0/3142637893 wait complete. 2023-10-16T12:35:05.878 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-16T12:35:05.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-10-16T12:35:06.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.104+0000 7f90bffff640 1 Processor -- start 2023-10-16T12:35:06.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.104+0000 7f90bffff640 1 -- start start 2023-10-16T12:35:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.105+0000 7f90bffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f90c015c9f0 0x7f90c00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90bffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90c00c9230 0x7f90c00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90bffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c0160a90 con 0x7f90c0070f20 2023-10-16T12:35:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90bffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c00cdec0 con 0x7f90c00c9230 2023-10-16T12:35:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90bffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c00cd900 con 0x7f90c015c9f0 2023-10-16T12:35:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90b67fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90c00c9230 0x7f90c00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90bf7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f90c0070f20 0x7f90c00ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41696/0 (socket says 172.21.15.60:41696) 2023-10-16T12:35:06.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90bf7fe640 1 -- 172.21.15.60:0/3163650986 learned_addr learned my addr 172.21.15.60:0/3163650986 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:06.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.106+0000 7f90beffd640 1 --2- 172.21.15.60:0/3163650986 >> v2:172.21.15.144:3300/0 conn(0x7f90c015c9f0 0x7f90c00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.107+0000 7f90b67fc640 1 -- 172.21.15.60:0/3163650986 >> v2:172.21.15.144:3300/0 conn(0x7f90c015c9f0 msgr2=0x7f90c00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:06.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.107+0000 7f90b67fc640 1 --2- 172.21.15.60:0/3163650986 >> v2:172.21.15.144:3300/0 conn(0x7f90c015c9f0 0x7f90c00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.107+0000 7f90b67fc640 1 -- 172.21.15.60:0/3163650986 >> v1:172.21.15.60:6789/0 conn(0x7f90c0070f20 legacy=0x7f90c00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.107+0000 7f90b67fc640 1 -- 172.21.15.60:0/3163650986 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c00cdbe0 con 0x7f90c00c9230 2023-10-16T12:35:06.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.107+0000 7f90beffd640 1 --2- 172.21.15.60:0/3163650986 >> v2:172.21.15.144:3300/0 conn(0x7f90c015c9f0 0x7f90c00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:06.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.108+0000 7f90b67fc640 1 --2- 172.21.15.60:0/3163650986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c00709e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f90c00ad4e0 tx=0x7f90a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92706d2fdc6181a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:06.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.108+0000 7f90be7fc640 1 -- 172.21.15.60:0/3163650986 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8031830 con 0x7f90c00c9230 2023-10-16T12:35:06.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.108+0000 7f90be7fc640 1 -- 172.21.15.60:0/3163650986 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a8031990 con 0x7f90c00c9230 2023-10-16T12:35:06.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.108+0000 7f90be7fc640 1 -- 172.21.15.60:0/3163650986 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8031d20 con 0x7f90c00c9230 2023-10-16T12:35:06.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 -- 172.21.15.60:0/3163650986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 msgr2=0x7f90c00709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 --2- 172.21.15.60:0/3163650986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c00709e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f90c00ad4e0 tx=0x7f90a802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 -- 172.21.15.60:0/3163650986 shutdown_connections 2023-10-16T12:35:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 --2- 172.21.15.60:0/3163650986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c00709e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 --2- 172.21.15.60:0/3163650986 >> v2:172.21.15.144:3300/0 conn(0x7f90c015c9f0 0x7f90c00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 -- 172.21.15.60:0/3163650986 >> 172.21.15.60:0/3163650986 conn(0x7f90c00ce6a0 msgr2=0x7f90c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 -- 172.21.15.60:0/3163650986 shutdown_connections 2023-10-16T12:35:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.109+0000 7f90bffff640 1 -- 172.21.15.60:0/3163650986 wait complete. 2023-10-16T12:35:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.111+0000 7f90bffff640 1 Processor -- start 2023-10-16T12:35:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.111+0000 7f90bffff640 1 -- start start 2023-10-16T12:35:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90bffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c0070f20 0x7f90c0150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90bffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c015c9f0 0x7f90c0158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90bffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00c8540 con 0x7f90c00c9230 2023-10-16T12:35:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90bffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c0151130 con 0x7f90c015c9f0 2023-10-16T12:35:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90bffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00c2270 con 0x7f90c0070f20 2023-10-16T12:35:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90b67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90b67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c006a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46120/0 (socket says 172.21.15.60:46120) 2023-10-16T12:35:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.112+0000 7f90b67fc640 1 -- 172.21.15.60:0/2170440346 learned_addr learned my addr 172.21.15.60:0/2170440346 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90beffd640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c0070f20 0x7f90c0150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b67fc640 1 -- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c0070f20 msgr2=0x7f90c0150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b67fc640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c0070f20 0x7f90c0150bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b67fc640 1 -- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c015c9f0 msgr2=0x7f90c0158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b67fc640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c015c9f0 0x7f90c0158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b67fc640 1 -- 172.21.15.60:0/2170440346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8009400 con 0x7f90c00c9230 2023-10-16T12:35:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b67fc640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c006a7b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009890 tx=0x7f90a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a803ba70 con 0x7f90c00c9230 2023-10-16T12:35:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.113+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a803bbf0 con 0x7f90c00c9230 2023-10-16T12:35:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.114+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8045400 con 0x7f90c00c9230 2023-10-16T12:35:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.114+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c006acf0 con 0x7f90c00c9230 2023-10-16T12:35:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.114+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c00c2510 con 0x7f90c00c9230 2023-10-16T12:35:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.115+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c004e9b0 con 0x7f90c00c9230 2023-10-16T12:35:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.116+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f90a804e440 con 0x7f90c00c9230 2023-10-16T12:35:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.117+0000 7f90b7fff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f90a406b100 0x7f90a406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.117+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f90a80abfe0 con 0x7f90c00c9230 2023-10-16T12:35:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.117+0000 7f90beffd640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f90a406b100 0x7f90a406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.119+0000 7f90beffd640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f90a406b100 0x7f90a406d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90b000b840 tx=0x7f90b0008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.121+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f90a8076b30 con 0x7f90c00c9230 2023-10-16T12:35:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.341+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f90c00704c0 con 0x7f90a406b100 2023-10-16T12:35:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.494+0000 7f90b7fff640 1 -- 172.21.15.60:0/2170440346 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f90b000c040 con 0x7f90a406b100 2023-10-16T12:35:06.496 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-16T12:35:06.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.497+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f90a406b100 msgr2=0x7f90a406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.497+0000 7f90bffff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f90a406b100 0x7f90a406d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90b000b840 tx=0x7f90b0008040 comp rx=0 tx=0).stop 2023-10-16T12:35:06.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.497+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 msgr2=0x7f90c006a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.497+0000 7f90bffff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c006a7b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009890 tx=0x7f90a80046c0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 shutdown_connections 2023-10-16T12:35:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f90a406b100 0x7f90a406d5c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c015c9f0 0x7f90c0158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00c9230 0x7f90c006a7b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 --2- 172.21.15.60:0/2170440346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c0070f20 0x7f90c0150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 >> 172.21.15.60:0/2170440346 conn(0x7f90c00ce6a0 msgr2=0x7f90c00c1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:06.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 shutdown_connections 2023-10-16T12:35:06.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.498+0000 7f90bffff640 1 -- 172.21.15.60:0/2170440346 wait complete. 2023-10-16T12:35:06.522 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-16T12:35:06.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-10-16T12:35:06.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.750+0000 7feb2b59e640 1 Processor -- start 2023-10-16T12:35:06.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.750+0000 7feb2b59e640 1 -- start start 2023-10-16T12:35:06.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb2c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feb2c06c9e0 0x7feb2c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0ca560 con 0x7feb2c0c9440 2023-10-16T12:35:06.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2b59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0c9950 con 0x7feb2c06c9e0 2023-10-16T12:35:06.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0c9c30 con 0x7feb2c0c9060 2023-10-16T12:35:06.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feb2c0c9440 0x7feb2c164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41704/0 (socket says 172.21.15.60:41704) 2023-10-16T12:35:06.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2ad9d640 1 -- 172.21.15.60:0/2970511974 learned_addr learned my addr 172.21.15.60:0/2970511974 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:06.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb2a59c640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb2c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.752+0000 7feb23fff640 1 --2- 172.21.15.60:0/2970511974 >> v2:172.21.15.144:3300/0 conn(0x7feb2c06c9e0 0x7feb2c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619671500 0 0) 0x7feb2c0ca560 con 0x7feb2c0c9440 2023-10-16T12:35:06.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb18003520 con 0x7feb2c0c9440 2023-10-16T12:35:06.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3071342996 0 0) 0x7feb18003520 con 0x7feb2c0c9440 2023-10-16T12:35:06.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 msgr2=0x7feb2c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:06.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb2c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 >> v2:172.21.15.144:3300/0 conn(0x7feb2c06c9e0 msgr2=0x7feb2c06f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:06.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 --2- 172.21.15.60:0/2970511974 >> v2:172.21.15.144:3300/0 conn(0x7feb2c06c9e0 0x7feb2c06f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0c9f10 con 0x7feb2c0c9440 2023-10-16T12:35:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7feb240019f0 con 0x7feb2c0c9440 2023-10-16T12:35:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 >> v1:172.21.15.60:6789/0 conn(0x7feb2c0c9440 legacy=0x7feb2c164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb18003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb18004d00 0x7feb18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0c9950 con 0x7feb18003f40 2023-10-16T12:35:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0c9c30 con 0x7feb18004d00 2023-10-16T12:35:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.753+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1800b980 con 0x7feb2c0c9060 2023-10-16T12:35:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb23fff640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb2a59c640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb18003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb23fff640 1 -- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 msgr2=0x7feb18003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb23fff640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb18003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb23fff640 1 -- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb18004d00 msgr2=0x7feb18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb2ad9d640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb18004d00 0x7feb18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb23fff640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb18004d00 0x7feb18005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb23fff640 1 -- 172.21.15.60:0/2970511974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18003520 con 0x7feb18003f40 2023-10-16T12:35:06.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.754+0000 7feb2a59c640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c9060 0x7feb18003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:06.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.755+0000 7feb23fff640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb180047c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7feb14009ac0 tx=0x7feb180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c602caacaaa8451 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:06.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1402c6b0 con 0x7feb18003f40 2023-10-16T12:35:06.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1402c810 con 0x7feb18003f40 2023-10-16T12:35:06.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb29d9b640 1 -- 172.21.15.60:0/2970511974 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1402cba0 con 0x7feb18003f40 2023-10-16T12:35:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2b59e640 1 -- 172.21.15.60:0/2970511974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 msgr2=0x7feb180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2b59e640 1 --2- 172.21.15.60:0/2970511974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb180047c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7feb14009ac0 tx=0x7feb180057f0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2ad9d640 1 -- 172.21.15.60:0/2970511974 reap_dead start 2023-10-16T12:35:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2b59e640 1 -- 172.21.15.60:0/2970511974 shutdown_connections 2023-10-16T12:35:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2b59e640 1 -- 172.21.15.60:0/2970511974 >> 172.21.15.60:0/2970511974 conn(0x7feb2c0ce6a0 msgr2=0x7feb2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2b59e640 1 -- 172.21.15.60:0/2970511974 shutdown_connections 2023-10-16T12:35:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.756+0000 7feb2b59e640 1 -- 172.21.15.60:0/2970511974 wait complete. 2023-10-16T12:35:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.758+0000 7feb2b59e640 1 Processor -- start 2023-10-16T12:35:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.758+0000 7feb2b59e640 1 -- start start 2023-10-16T12:35:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.759+0000 7feb2b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb2c1f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.759+0000 7feb2b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c5770 0x7feb2c161560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.759+0000 7feb2b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb2c0b3e60 0x7feb2c163050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.759+0000 7feb2b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0b3440 con 0x7feb18003f40 2023-10-16T12:35:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.759+0000 7feb2b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c1f65d0 con 0x7feb2c0b3e60 2023-10-16T12:35:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.759+0000 7feb2b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c160f00 con 0x7feb2c0c5770 2023-10-16T12:35:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb2c1f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb2c1f6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46136/0 (socket says 172.21.15.60:46136) 2023-10-16T12:35:06.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 -- 172.21.15.60:0/4100055044 learned_addr learned my addr 172.21.15.60:0/4100055044 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:06.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 -- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c5770 msgr2=0x7feb2c161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c5770 0x7feb2c161560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 -- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb2c0b3e60 msgr2=0x7feb2c163050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:06.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb2c0b3e60 0x7feb2c163050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:06.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 -- 172.21.15.60:0/4100055044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c1611e0 con 0x7feb18003f40 2023-10-16T12:35:06.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.760+0000 7feb2a59c640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb2c1f6090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0108b0 tx=0x7feb1c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:06.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.761+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00eb60 con 0x7feb18003f40 2023-10-16T12:35:06.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.761+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c00ecc0 con 0x7feb18003f40 2023-10-16T12:35:06.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.761+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c01c400 con 0x7feb18003f40 2023-10-16T12:35:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.761+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c0b3060 con 0x7feb18003f40 2023-10-16T12:35:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.761+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c163590 con 0x7feb18003f40 2023-10-16T12:35:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.762+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c04e9b0 con 0x7feb18003f40 2023-10-16T12:35:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.763+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7feb1c01c560 con 0x7feb18003f40 2023-10-16T12:35:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.764+0000 7feb22ffd640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb0c06b100 0x7feb0c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.765+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7feb1c08a160 con 0x7feb18003f40 2023-10-16T12:35:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.765+0000 7feb23fff640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb0c06b100 0x7feb0c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.766+0000 7feb23fff640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb0c06b100 0x7feb0c06d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feb14036000 tx=0x7feb1402e000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.769+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb1c054e50 con 0x7feb18003f40 2023-10-16T12:35:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:06.991+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7feb2c0c1f10 con 0x7feb0c06b100 2023-10-16T12:35:07.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.140+0000 7feb22ffd640 1 -- 172.21.15.60:0/4100055044 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7feb140040c0 con 0x7feb0c06b100 2023-10-16T12:35:07.142 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-16T12:35:07.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb0c06b100 msgr2=0x7feb0c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb0c06b100 0x7feb0c06d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feb14036000 tx=0x7feb1402e000 comp rx=0 tx=0).stop 2023-10-16T12:35:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 msgr2=0x7feb2c1f6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb2c1f6090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0108b0 tx=0x7feb1c010d70 comp rx=0 tx=0).stop 2023-10-16T12:35:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 shutdown_connections 2023-10-16T12:35:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb0c06b100 0x7feb0c06d5c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb2c0c5770 0x7feb2c161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb2c0b3e60 0x7feb2c163050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 --2- 172.21.15.60:0/4100055044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb18003f40 0x7feb2c1f6090 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 >> 172.21.15.60:0/4100055044 conn(0x7feb2c0ce6a0 msgr2=0x7feb2c06b020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:07.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 shutdown_connections 2023-10-16T12:35:07.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.144+0000 7feb2b59e640 1 -- 172.21.15.60:0/4100055044 wait complete. 2023-10-16T12:35:07.166 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-16T12:35:07.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-10-16T12:35:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.392+0000 7f275847c640 1 Processor -- start 2023-10-16T12:35:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.393+0000 7f275847c640 1 -- start start 2023-10-16T12:35:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.394+0000 7f275847c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 0x7f27500ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.394+0000 7f275847c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2750070360 con 0x7f2750159ab0 2023-10-16T12:35:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f275847c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500cd1f0 con 0x7f2750159e90 2023-10-16T12:35:07.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f275847c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275015d890 con 0x7f275006c850 2023-10-16T12:35:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f2756218640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2750159ab0 0x7f275006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41706/0 (socket says 172.21.15.60:41706) 2023-10-16T12:35:07.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f2756218640 1 -- 172.21.15.60:0/454990946 learned_addr learned my addr 172.21.15.60:0/454990946 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f2756a19640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 0x7f27500ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659428977 0 0) 0x7f2750070360 con 0x7f2750159ab0 2023-10-16T12:35:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.395+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f273c005730 con 0x7f2750159ab0 2023-10-16T12:35:07.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2104644029 0 0) 0x7f273c005730 con 0x7f2750159ab0 2023-10-16T12:35:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 msgr2=0x7f27500ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 0x7f27500ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 >> v1:172.21.15.144:6789/0 conn(0x7f2750159e90 legacy=0x7f275006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500cd4d0 con 0x7f2750159ab0 2023-10-16T12:35:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f274c001670 con 0x7f2750159ab0 2023-10-16T12:35:07.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 >> v1:172.21.15.60:6789/0 conn(0x7f2750159ab0 legacy=0x7f275006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c005bb0 0x7f273c005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f273c0069d0 0x7f273c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 0x7f273c007460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275015d890 con 0x7f273c0069d0 2023-10-16T12:35:07.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c00dd80 con 0x7f273c005bb0 2023-10-16T12:35:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c00df40 con 0x7f275006c850 2023-10-16T12:35:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.396+0000 7f2755a17640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f273c0069d0 0x7f273c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2756218640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c005bb0 0x7f273c005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2756a19640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 0x7f273c007460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2755a17640 1 -- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 msgr2=0x7f273c007460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2755a17640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f275006c850 0x7f273c007460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2755a17640 1 -- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c005bb0 msgr2=0x7f273c005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2755a17640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c005bb0 0x7f273c005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.397+0000 7f2755a17640 1 -- 172.21.15.60:0/454990946 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c005730 con 0x7f273c0069d0 2023-10-16T12:35:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.398+0000 7f2755a17640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f273c0069d0 0x7f273c006e50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2740001750 tx=0x7f273c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf77bff90015982 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.398+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27400052d0 con 0x7f273c0069d0 2023-10-16T12:35:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.398+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274002e050 con 0x7f273c0069d0 2023-10-16T12:35:07.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.398+0000 7f2755216640 1 -- 172.21.15.60:0/454990946 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27400035f0 con 0x7f273c0069d0 2023-10-16T12:35:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f275847c640 1 -- 172.21.15.60:0/454990946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f273c0069d0 msgr2=0x7f273c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f275847c640 1 --2- 172.21.15.60:0/454990946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f273c0069d0 0x7f273c006e50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2740001750 tx=0x7f273c007a00 comp rx=0 tx=0).stop 2023-10-16T12:35:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f2756a19640 1 -- 172.21.15.60:0/454990946 reap_dead start 2023-10-16T12:35:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f275847c640 1 -- 172.21.15.60:0/454990946 shutdown_connections 2023-10-16T12:35:07.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f275847c640 1 -- 172.21.15.60:0/454990946 >> 172.21.15.60:0/454990946 conn(0x7f27500ce9b0 msgr2=0x7f27500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f275847c640 1 -- 172.21.15.60:0/454990946 shutdown_connections 2023-10-16T12:35:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.399+0000 7f275847c640 1 -- 172.21.15.60:0/454990946 wait complete. 2023-10-16T12:35:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.401+0000 7f275847c640 1 Processor -- start 2023-10-16T12:35:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.401+0000 7f275847c640 1 -- start start 2023-10-16T12:35:07.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.402+0000 7f275847c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c0069d0 0x7f2750156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.402+0000 7f275847c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 0x7f27500c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.402+0000 7f275847c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27500c7cf0 0x7f2750158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.402+0000 7f275847c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275015a3a0 con 0x7f275015b440 2023-10-16T12:35:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.402+0000 7f275847c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275015a680 con 0x7f273c0069d0 2023-10-16T12:35:07.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.402+0000 7f275847c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275015a960 con 0x7f27500c7cf0 2023-10-16T12:35:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 0x7f27500c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 0x7f27500c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46142/0 (socket says 172.21.15.60:46142) 2023-10-16T12:35:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 -- 172.21.15.60:0/4057086544 learned_addr learned my addr 172.21.15.60:0/4057086544 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2756218640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c0069d0 0x7f2750156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2756a19640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27500c7cf0 0x7f2750158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 -- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27500c7cf0 msgr2=0x7f2750158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27500c7cf0 0x7f2750158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 -- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c0069d0 msgr2=0x7f2750156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c0069d0 0x7f2750156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 -- 172.21.15.60:0/4057086544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275015ac40 con 0x7f275015b440 2023-10-16T12:35:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2756218640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c0069d0 0x7f2750156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2756a19640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27500c7cf0 0x7f2750158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.403+0000 7f2755a17640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 0x7f27500c77b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2740008fd0 tx=0x7f2740001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.404+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27400076c0 con 0x7f275015b440 2023-10-16T12:35:07.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.404+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2740007a30 con 0x7f275015b440 2023-10-16T12:35:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.404+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740006c00 con 0x7f275015b440 2023-10-16T12:35:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.404+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275015af20 con 0x7f275015b440 2023-10-16T12:35:07.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.404+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2750156950 con 0x7f275015b440 2023-10-16T12:35:07.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.405+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275004e160 con 0x7f275015b440 2023-10-16T12:35:07.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.406+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f27400035f0 con 0x7f275015b440 2023-10-16T12:35:07.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.407+0000 7f2746ffd640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f273406b100 0x7f273406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:07.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.407+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f27400abcd0 con 0x7f275015b440 2023-10-16T12:35:07.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.408+0000 7f2756218640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f273406b100 0x7f273406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:07.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.409+0000 7f2756218640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f273406b100 0x7f273406d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f274c001750 tx=0x7f274c001d70 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:07.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.411+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f274007a480 con 0x7f275015b440 2023-10-16T12:35:07.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.634+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f27500cda90 con 0x7f273406b100 2023-10-16T12:35:07.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.790+0000 7f2746ffd640 1 -- 172.21.15.60:0/4057086544 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f274c001430 con 0x7f273406b100 2023-10-16T12:35:07.791 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-16T12:35:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.793+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f273406b100 msgr2=0x7f273406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.793+0000 7f275847c640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f273406b100 0x7f273406d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f274c001750 tx=0x7f274c001d70 comp rx=0 tx=0).stop 2023-10-16T12:35:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.793+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 msgr2=0x7f27500c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.793+0000 7f275847c640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 0x7f27500c77b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2740008fd0 tx=0x7f2740001120 comp rx=0 tx=0).stop 2023-10-16T12:35:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 shutdown_connections 2023-10-16T12:35:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f273406b100 0x7f273406d5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27500c7cf0 0x7f2750158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f275015b440 0x7f27500c77b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 --2- 172.21.15.60:0/4057086544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f273c0069d0 0x7f2750156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 >> 172.21.15.60:0/4057086544 conn(0x7f27500ce9b0 msgr2=0x7f27500b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 shutdown_connections 2023-10-16T12:35:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:07.794+0000 7f275847c640 1 -- 172.21.15.60:0/4057086544 wait complete. 2023-10-16T12:35:07.817 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-16T12:35:07.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-10-16T12:35:08.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.048+0000 7f570bf41640 1 Processor -- start 2023-10-16T12:35:08.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.048+0000 7f570bf41640 1 -- start start 2023-10-16T12:35:08.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f570bf41640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f570bf41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040caac0 0x7f570406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f570bf41640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040cb030 con 0x7f57040c9ca0 2023-10-16T12:35:08.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f570bf41640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57040c91c0 con 0x7f570415ad70 2023-10-16T12:35:08.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f570bf41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040c94a0 con 0x7f57040caac0 2023-10-16T12:35:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f5709cdd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f5709cdd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46154/0 (socket says 172.21.15.60:46154) 2023-10-16T12:35:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.049+0000 7f5709cdd640 1 -- 172.21.15.60:0/2224012077 learned_addr learned my addr 172.21.15.60:0/2224012077 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.050+0000 7f5709cdd640 1 -- 172.21.15.60:0/2224012077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040caac0 msgr2=0x7f570406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.050+0000 7f570a4de640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040caac0 0x7f570406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.050+0000 7f5709cdd640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040caac0 0x7f570406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.050+0000 7f5709cdd640 1 -- 172.21.15.60:0/2224012077 >> v1:172.21.15.144:6789/0 conn(0x7f57040c9ca0 legacy=0x7f57040ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.050+0000 7f5709cdd640 1 -- 172.21.15.60:0/2224012077 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57040c9780 con 0x7f570415ad70 2023-10-16T12:35:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.050+0000 7f570a4de640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040caac0 0x7f570406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.051+0000 7f5709cdd640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f5704069da0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5700009870 tx=0x7f570002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb97d340cb3e619 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.051+0000 7f5708cdb640 1 -- 172.21.15.60:0/2224012077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700031640 con 0x7f570415ad70 2023-10-16T12:35:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.051+0000 7f5708cdb640 1 -- 172.21.15.60:0/2224012077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57000317a0 con 0x7f570415ad70 2023-10-16T12:35:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.051+0000 7f5708cdb640 1 -- 172.21.15.60:0/2224012077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5700031b30 con 0x7f570415ad70 2023-10-16T12:35:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 -- 172.21.15.60:0/2224012077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 msgr2=0x7f5704069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f5704069da0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5700009870 tx=0x7f570002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 -- 172.21.15.60:0/2224012077 shutdown_connections 2023-10-16T12:35:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040caac0 0x7f570406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 --2- 172.21.15.60:0/2224012077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 -- 172.21.15.60:0/2224012077 >> 172.21.15.60:0/2224012077 conn(0x7f57040ce9b0 msgr2=0x7f57040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 -- 172.21.15.60:0/2224012077 shutdown_connections 2023-10-16T12:35:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.052+0000 7f570bf41640 1 -- 172.21.15.60:0/2224012077 wait complete. 2023-10-16T12:35:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.054+0000 7f570bf41640 1 Processor -- start 2023-10-16T12:35:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.054+0000 7f570bf41640 1 -- start start 2023-10-16T12:35:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570bf41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040c9ca0 0x7f57040c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570bf41640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57040caac0 0x7f5704159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570bf41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f57040c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570bf41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040c2aa0 con 0x7f570415ad70 2023-10-16T12:35:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570bf41640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040c2d80 con 0x7f57040caac0 2023-10-16T12:35:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570bf41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040c3060 con 0x7f57040c9ca0 2023-10-16T12:35:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570a4de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f57040c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570a4de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f57040c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46156/0 (socket says 172.21.15.60:46156) 2023-10-16T12:35:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.055+0000 7f570a4de640 1 -- 172.21.15.60:0/4108518320 learned_addr learned my addr 172.21.15.60:0/4108518320 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f5709cdd640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040c9ca0 0x7f57040c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f570a4de640 1 -- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040c9ca0 msgr2=0x7f57040c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f57094dc640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57040caac0 0x7f5704159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f570a4de640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040c9ca0 0x7f57040c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f570a4de640 1 -- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57040caac0 msgr2=0x7f5704159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f570a4de640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57040caac0 0x7f5704159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f570a4de640 1 -- 172.21.15.60:0/4108518320 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57000094d0 con 0x7f570415ad70 2023-10-16T12:35:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f5709cdd640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040c9ca0 0x7f57040c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f570a4de640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f57040c3b40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00cef0 tx=0x7f56fc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc00d840 con 0x7f570415ad70 2023-10-16T12:35:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.056+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc00d9c0 con 0x7f570415ad70 2023-10-16T12:35:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.057+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57040c3340 con 0x7f570415ad70 2023-10-16T12:35:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.057+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc00dd70 con 0x7f570415ad70 2023-10-16T12:35:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.057+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57040c2050 con 0x7f570415ad70 2023-10-16T12:35:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.058+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57040c3860 con 0x7f570415ad70 2023-10-16T12:35:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.059+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f56fc01e3b0 con 0x7f570415ad70 2023-10-16T12:35:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.060+0000 7f56fa7fc640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f56e806b100 0x7f56e806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.060+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f56fc08a6e0 con 0x7f570415ad70 2023-10-16T12:35:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.060+0000 7f5709cdd640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f56e806b100 0x7f56e806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.061+0000 7f5709cdd640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f56e806b100 0x7f56e806d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5700036000 tx=0x7f570003c040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.064+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57040c3860 con 0x7f570415ad70 2023-10-16T12:35:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.286+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f57040c2ef0 con 0x7f56e806b100 2023-10-16T12:35:08.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.439+0000 7f56fa7fc640 1 -- 172.21.15.60:0/4108518320 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f57040c2ef0 con 0x7f56e806b100 2023-10-16T12:35:08.441 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-16T12:35:08.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f56e806b100 msgr2=0x7f56e806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f56e806b100 0x7f56e806d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5700036000 tx=0x7f570003c040 comp rx=0 tx=0).stop 2023-10-16T12:35:08.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 msgr2=0x7f57040c3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f57040c3b40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00cef0 tx=0x7f56fc00a710 comp rx=0 tx=0).stop 2023-10-16T12:35:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 shutdown_connections 2023-10-16T12:35:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f56e806b100 0x7f56e806d5c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f570415ad70 0x7f57040c3b40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57040caac0 0x7f5704159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 --2- 172.21.15.60:0/4108518320 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57040c9ca0 0x7f57040c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.443+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 >> 172.21.15.60:0/4108518320 conn(0x7f57040ce9b0 msgr2=0x7f57040cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:08.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.444+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 shutdown_connections 2023-10-16T12:35:08.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.444+0000 7f570bf41640 1 -- 172.21.15.60:0/4108518320 wait complete. 2023-10-16T12:35:08.466 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-16T12:35:08.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-10-16T12:35:08.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.697+0000 7faabddd5640 1 Processor -- start 2023-10-16T12:35:08.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.697+0000 7faabddd5640 1 -- start start 2023-10-16T12:35:08.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faabddd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 0x7faab80ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faabddd5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab80cb030 con 0x7faab806a4b0 2023-10-16T12:35:08.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faabddd5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab80c91c0 con 0x7faab815ad70 2023-10-16T12:35:08.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faabddd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab80c94a0 con 0x7faab80caac0 2023-10-16T12:35:08.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faab6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faab806a4b0 0x7faab80ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41710/0 (socket says 172.21.15.60:41710) 2023-10-16T12:35:08.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faab6ffd640 1 -- 172.21.15.60:0/4156869033 learned_addr learned my addr 172.21.15.60:0/4156869033 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:08.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faab7fff640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 0x7faab80ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.699+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777674932 0 0) 0x7faab80cb030 con 0x7faab806a4b0 2023-10-16T12:35:08.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa0003520 con 0x7faab806a4b0 2023-10-16T12:35:08.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458247296 0 0) 0x7faaa0003520 con 0x7faab806a4b0 2023-10-16T12:35:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 msgr2=0x7faab80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 0x7faab80ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 >> v1:172.21.15.144:6789/0 conn(0x7faab815ad70 legacy=0x7faab8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab80c9780 con 0x7faab806a4b0 2023-10-16T12:35:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7faaac001670 con 0x7faab806a4b0 2023-10-16T12:35:08.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 >> v1:172.21.15.60:6789/0 conn(0x7faab806a4b0 legacy=0x7faab80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 0x7faaa00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa0003f70 0x7faaa00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa0004d00 0x7faaa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab80c94a0 con 0x7faaa0003f70 2023-10-16T12:35:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa000bac0 con 0x7faaa0004d00 2023-10-16T12:35:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa000bc80 con 0x7faab80caac0 2023-10-16T12:35:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.700+0000 7faab6ffd640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa0003f70 0x7faaa00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab77fe640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 0x7faaa00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab7fff640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa0004d00 0x7faaa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab6ffd640 1 -- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 msgr2=0x7faaa00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:08.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab6ffd640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faab80caac0 0x7faaa00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab6ffd640 1 -- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa0004d00 msgr2=0x7faaa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:08.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab6ffd640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa0004d00 0x7faaa0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab6ffd640 1 -- 172.21.15.60:0/4156869033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa0003520 con 0x7faaa0003f70 2023-10-16T12:35:08.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.701+0000 7faab7fff640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa0004d00 0x7faaa0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:08.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.702+0000 7faab6ffd640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa0003f70 0x7faaa00047c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c0e0 tx=0x7faaa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ce0e108cc64b69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:08.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.702+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac005ad0 con 0x7faaa0003f70 2023-10-16T12:35:08.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.702+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac005c30 con 0x7faaa0003f70 2023-10-16T12:35:08.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.702+0000 7faab67fc640 1 -- 172.21.15.60:0/4156869033 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0072b0 con 0x7faaa0003f70 2023-10-16T12:35:08.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faabddd5640 1 -- 172.21.15.60:0/4156869033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa0003f70 msgr2=0x7faaa00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faabddd5640 1 --2- 172.21.15.60:0/4156869033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa0003f70 0x7faaa00047c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c0e0 tx=0x7faaa00057f0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faab7fff640 1 -- 172.21.15.60:0/4156869033 reap_dead start 2023-10-16T12:35:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faabddd5640 1 -- 172.21.15.60:0/4156869033 shutdown_connections 2023-10-16T12:35:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faabddd5640 1 -- 172.21.15.60:0/4156869033 >> 172.21.15.60:0/4156869033 conn(0x7faab80ce9b0 msgr2=0x7faab80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faabddd5640 1 -- 172.21.15.60:0/4156869033 shutdown_connections 2023-10-16T12:35:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.703+0000 7faabddd5640 1 -- 172.21.15.60:0/4156869033 wait complete. 2023-10-16T12:35:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.705+0000 7faabddd5640 1 Processor -- start 2023-10-16T12:35:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.705+0000 7faabddd5640 1 -- start start 2023-10-16T12:35:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.706+0000 7faabddd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa0003f70 0x7faab80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.706+0000 7faabddd5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faab80c24a0 0x7faab8145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.706+0000 7faabddd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 0x7faab80b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.706+0000 7faabddd5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab81467f0 con 0x7faab80b2310 2023-10-16T12:35:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.706+0000 7faabddd5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab80b2cb0 con 0x7faab80c24a0 2023-10-16T12:35:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.706+0000 7faabddd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab80c2c80 con 0x7faaa0003f70 2023-10-16T12:35:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 0x7faab80b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 0x7faab80b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46184/0 (socket says 172.21.15.60:46184) 2023-10-16T12:35:08.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 -- 172.21.15.60:0/4204082854 learned_addr learned my addr 172.21.15.60:0/4204082854 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab77fe640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa0003f70 0x7faab80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab6ffd640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faab80c24a0 0x7faab8145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 -- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa0003f70 msgr2=0x7faab80c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa0003f70 0x7faab80c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 -- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faab80c24a0 msgr2=0x7faab8145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:08.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faab80c24a0 0x7faab8145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:08.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.707+0000 7faab7fff640 1 -- 172.21.15.60:0/4204082854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8146130 con 0x7faab80b2310 2023-10-16T12:35:08.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.708+0000 7faab7fff640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 0x7faab80b2770 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faab806d860 tx=0x7faab000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:08.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.708+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0017400 con 0x7faab80b2310 2023-10-16T12:35:08.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.708+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab0011040 con 0x7faab80b2310 2023-10-16T12:35:08.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.708+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0017780 con 0x7faab80b2310 2023-10-16T12:35:08.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.708+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab8146410 con 0x7faab80b2310 2023-10-16T12:35:08.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.708+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab80c2920 con 0x7faab80b2310 2023-10-16T12:35:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.710+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab8146930 con 0x7faab80b2310 2023-10-16T12:35:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.710+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faab00203b0 con 0x7faab80b2310 2023-10-16T12:35:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.711+0000 7faa9ffff640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7faaa406b100 0x7faaa406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.711+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7faab0089f30 con 0x7faab80b2310 2023-10-16T12:35:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.712+0000 7faab77fe640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7faaa406b100 0x7faaa406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.713+0000 7faab77fe640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7faaa406b100 0x7faaa406d5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa8009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.716+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faab8146930 con 0x7faab80b2310 2023-10-16T12:35:08.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:08.938+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faab80c9ca0 con 0x7faaa406b100 2023-10-16T12:35:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.094+0000 7faa9ffff640 1 -- 172.21.15.60:0/4204082854 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7faaa8005170 con 0x7faaa406b100 2023-10-16T12:35:09.097 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-16T12:35:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.097+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7faaa406b100 msgr2=0x7faaa406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.097+0000 7faabddd5640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7faaa406b100 0x7faaa406d5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faaa8007fd0 tx=0x7faaa8009040 comp rx=0 tx=0).stop 2023-10-16T12:35:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.097+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 msgr2=0x7faab80b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.097+0000 7faabddd5640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 0x7faab80b2770 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faab806d860 tx=0x7faab000f4a0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 shutdown_connections 2023-10-16T12:35:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7faaa406b100 0x7faaa406d5c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faab80b2310 0x7faab80b2770 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faab80c24a0 0x7faab8145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 --2- 172.21.15.60:0/4204082854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa0003f70 0x7faab80c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 >> 172.21.15.60:0/4204082854 conn(0x7faab80ce9b0 msgr2=0x7faab80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 shutdown_connections 2023-10-16T12:35:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.098+0000 7faabddd5640 1 -- 172.21.15.60:0/4204082854 wait complete. 2023-10-16T12:35:09.122 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-16T12:35:09.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-10-16T12:35:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.352+0000 7fa28f59e640 1 Processor -- start 2023-10-16T12:35:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.352+0000 7fa28f59e640 1 -- start start 2023-10-16T12:35:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.353+0000 7fa28f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015cd70 0x7fa2900d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.353+0000 7fa28f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29015d150 con 0x7fa2900b8480 2023-10-16T12:35:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.353+0000 7fa28f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2900d1180 con 0x7fa2900cded0 2023-10-16T12:35:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.353+0000 7fa28f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900d1460 con 0x7fa29015cd70 2023-10-16T12:35:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.354+0000 7fa28dd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa2900b8480 0x7fa2900cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37650/0 (socket says 172.21.15.60:37650) 2023-10-16T12:35:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.354+0000 7fa28dd9b640 1 -- 172.21.15.60:0/3699016824 learned_addr learned my addr 172.21.15.60:0/3699016824 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.354+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015cd70 0x7fa2900d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801307525 0 0) 0x7fa29015d150 con 0x7fa2900b8480 2023-10-16T12:35:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c003520 con 0x7fa2900b8480 2023-10-16T12:35:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987347698 0 0) 0x7fa2900d1180 con 0x7fa2900cded0 2023-10-16T12:35:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29015d150 con 0x7fa2900cded0 2023-10-16T12:35:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074312594 0 0) 0x7fa27c003520 con 0x7fa2900b8480 2023-10-16T12:35:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015cd70 msgr2=0x7fa2900d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015cd70 0x7fa2900d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 >> v1:172.21.15.144:6789/0 conn(0x7fa2900cded0 legacy=0x7fa2900b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900d1740 con 0x7fa2900b8480 2023-10-16T12:35:09.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa278001670 con 0x7fa2900b8480 2023-10-16T12:35:09.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 >> v1:172.21.15.60:6789/0 conn(0x7fa2900b8480 legacy=0x7fa2900cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa27c003b30 0x7fa27c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c0047c0 0x7fa27c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa27c005180 0x7fa27c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900d1460 con 0x7fa27c003b30 2023-10-16T12:35:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c00bfa0 con 0x7fa27c0047c0 2023-10-16T12:35:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c00c160 con 0x7fa27c005180 2023-10-16T12:35:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.355+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015cd70 0x7fa2900d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:35:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa27c003b30 0x7fa27c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28dd9b640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c0047c0 0x7fa27c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28ed9d640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa27c005180 0x7fa27c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28e59c640 1 -- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa27c005180 msgr2=0x7fa27c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:09.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa27c005180 0x7fa27c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28e59c640 1 -- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c0047c0 msgr2=0x7fa27c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:09.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c0047c0 0x7fa27c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28e59c640 1 -- 172.21.15.60:0/3699016824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c003520 con 0x7fa27c003b30 2023-10-16T12:35:09.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.356+0000 7fa28ed9d640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa27c005180 0x7fa27c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:09.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.357+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa27c003b30 0x7fa27c003f30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa290066a80 tx=0x7fa27c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09a360f3640b3cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:09.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280004030 con 0x7fa27c003b30 2023-10-16T12:35:09.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa280004190 con 0x7fa27c003b30 2023-10-16T12:35:09.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28d59a640 1 -- 172.21.15.60:0/3699016824 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280004520 con 0x7fa27c003b30 2023-10-16T12:35:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28f59e640 1 -- 172.21.15.60:0/3699016824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa27c003b30 msgr2=0x7fa27c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3699016824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa27c003b30 0x7fa27c003f30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa290066a80 tx=0x7fa27c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28ed9d640 1 -- 172.21.15.60:0/3699016824 reap_dead start 2023-10-16T12:35:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28f59e640 1 -- 172.21.15.60:0/3699016824 shutdown_connections 2023-10-16T12:35:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.358+0000 7fa28f59e640 1 -- 172.21.15.60:0/3699016824 >> 172.21.15.60:0/3699016824 conn(0x7fa2900cf570 msgr2=0x7fa2900cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.359+0000 7fa28f59e640 1 -- 172.21.15.60:0/3699016824 shutdown_connections 2023-10-16T12:35:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.359+0000 7fa28f59e640 1 -- 172.21.15.60:0/3699016824 wait complete. 2023-10-16T12:35:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.360+0000 7fa28f59e640 1 Processor -- start 2023-10-16T12:35:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.361+0000 7fa28f59e640 1 -- start start 2023-10-16T12:35:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.361+0000 7fa28f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c003b30 0x7fa290150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 0x7fa2900cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015f750 0x7fa29015fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900cca10 con 0x7fa2900cc050 2023-10-16T12:35:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900a94f0 con 0x7fa27c003b30 2023-10-16T12:35:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900aa070 con 0x7fa29015f750 2023-10-16T12:35:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 0x7fa2900cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 0x7fa2900cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39138/0 (socket says 172.21.15.60:39138) 2023-10-16T12:35:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 -- 172.21.15.60:0/3895817233 learned_addr learned my addr 172.21.15.60:0/3895817233 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c003b30 0x7fa290150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28ed9d640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015f750 0x7fa29015fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 -- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015f750 msgr2=0x7fa29015fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015f750 0x7fa29015fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 -- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c003b30 msgr2=0x7fa290150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c003b30 0x7fa290150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28dd9b640 1 -- 172.21.15.60:0/3895817233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900aa500 con 0x7fa2900cc050 2023-10-16T12:35:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.362+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c003b30 0x7fa290150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.363+0000 7fa28dd9b640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 0x7fa2900cc4d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa27800e7f0 tx=0x7fa27800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.363+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27800c8d0 con 0x7fa2900cc050 2023-10-16T12:35:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.363+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27800ca30 con 0x7fa2900cc050 2023-10-16T12:35:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.363+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27800cde0 con 0x7fa2900cc050 2023-10-16T12:35:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.363+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2900a9740 con 0x7fa2900cc050 2023-10-16T12:35:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.363+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2900a9d00 con 0x7fa2900cc050 2023-10-16T12:35:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.364+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29004e9b0 con 0x7fa2900cc050 2023-10-16T12:35:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.366+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa27801d460 con 0x7fa2900cc050 2023-10-16T12:35:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.366+0000 7fa286ffd640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa27006b100 0x7fa27006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.367+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa278089af0 con 0x7fa2900cc050 2023-10-16T12:35:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.367+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa27006b100 0x7fa27006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.368+0000 7fa28e59c640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa27006b100 0x7fa27006d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa280035000 tx=0x7fa280030000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.371+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa278054800 con 0x7fa2900cc050 2023-10-16T12:35:09.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.590+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2900d1d00 con 0x7fa27006b100 2023-10-16T12:35:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.746+0000 7fa286ffd640 1 -- 172.21.15.60:0/3895817233 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa280005990 con 0x7fa27006b100 2023-10-16T12:35:09.750 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-16T12:35:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa27006b100 msgr2=0x7fa27006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa27006b100 0x7fa27006d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa280035000 tx=0x7fa280030000 comp rx=0 tx=0).stop 2023-10-16T12:35:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 msgr2=0x7fa2900cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 0x7fa2900cc4d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa27800e7f0 tx=0x7fa27800ecb0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 shutdown_connections 2023-10-16T12:35:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa27006b100 0x7fa27006d5c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa29015f750 0x7fa29015fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2900cc050 0x7fa2900cc4d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 --2- 172.21.15.60:0/3895817233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa27c003b30 0x7fa290150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.750+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 >> 172.21.15.60:0/3895817233 conn(0x7fa2900cf570 msgr2=0x7fa29015b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.751+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 shutdown_connections 2023-10-16T12:35:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:09.751+0000 7fa28f59e640 1 -- 172.21.15.60:0/3895817233 wait complete. 2023-10-16T12:35:09.772 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-16T12:35:09.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-10-16T12:35:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.004+0000 7fbe723e7640 1 Processor -- start 2023-10-16T12:35:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.004+0000 7fbe723e7640 1 -- start start 2023-10-16T12:35:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe723e7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe6c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe723e7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c070360 con 0x7fbe6c159ab0 2023-10-16T12:35:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe723e7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6c0cd1f0 con 0x7fbe6c06c140 2023-10-16T12:35:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe723e7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c15d890 con 0x7fbe6c159e90 2023-10-16T12:35:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe6bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbe6c159ab0 0x7fbe6c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37666/0 (socket says 172.21.15.60:37666) 2023-10-16T12:35:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe6bfff640 1 -- 172.21.15.60:0/4213111241 learned_addr learned my addr 172.21.15.60:0/4213111241 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe63fff640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe6c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.006+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209327925 0 0) 0x7fbe6c070360 con 0x7fbe6c159ab0 2023-10-16T12:35:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe58003520 con 0x7fbe6c159ab0 2023-10-16T12:35:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4068344711 0 0) 0x7fbe58003520 con 0x7fbe6c159ab0 2023-10-16T12:35:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 msgr2=0x7fbe6c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe6c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 >> v1:172.21.15.144:6789/0 conn(0x7fbe6c06c140 legacy=0x7fbe6c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c0cd4d0 con 0x7fbe6c159ab0 2023-10-16T12:35:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fbe5c001670 con 0x7fbe6c159ab0 2023-10-16T12:35:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 >> v1:172.21.15.60:6789/0 conn(0x7fbe6c159ab0 legacy=0x7fbe6c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe580039a0 0x7fbe58003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe58004d00 0x7fbe58005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c15d890 con 0x7fbe58004d00 2023-10-16T12:35:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5800bb90 con 0x7fbe580039a0 2023-10-16T12:35:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.007+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5800df60 con 0x7fbe6c159e90 2023-10-16T12:35:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe70984640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe58004d00 0x7fbe58005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe63fff640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe6bfff640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe580039a0 0x7fbe58003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe70984640 1 -- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 msgr2=0x7fbe580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe70984640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe70984640 1 -- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe580039a0 msgr2=0x7fbe58003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe70984640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe580039a0 0x7fbe58003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe70984640 1 -- 172.21.15.60:0/4213111241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58003520 con 0x7fbe58004d00 2023-10-16T12:35:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.008+0000 7fbe63fff640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6c159e90 0x7fbe580047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:35:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.009+0000 7fbe70984640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe58004d00 0x7fbe58005160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbe64001e50 tx=0x7fbe580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b08dd5cb9eeba3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.009+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64003b70 con 0x7fbe58004d00 2023-10-16T12:35:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.009+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe64003cd0 con 0x7fbe58004d00 2023-10-16T12:35:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.009+0000 7fbe6b7fe640 1 -- 172.21.15.60:0/4213111241 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64005310 con 0x7fbe58004d00 2023-10-16T12:35:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe723e7640 1 -- 172.21.15.60:0/4213111241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe58004d00 msgr2=0x7fbe58005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe723e7640 1 --2- 172.21.15.60:0/4213111241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe58004d00 0x7fbe58005160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbe64001e50 tx=0x7fbe580057f0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe70984640 1 -- 172.21.15.60:0/4213111241 reap_dead start 2023-10-16T12:35:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe723e7640 1 -- 172.21.15.60:0/4213111241 shutdown_connections 2023-10-16T12:35:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe723e7640 1 -- 172.21.15.60:0/4213111241 >> 172.21.15.60:0/4213111241 conn(0x7fbe6c0ce9b0 msgr2=0x7fbe6c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe723e7640 1 -- 172.21.15.60:0/4213111241 shutdown_connections 2023-10-16T12:35:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.010+0000 7fbe723e7640 1 -- 172.21.15.60:0/4213111241 wait complete. 2023-10-16T12:35:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.012+0000 7fbe723e7640 1 Processor -- start 2023-10-16T12:35:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.012+0000 7fbe723e7640 1 -- start start 2023-10-16T12:35:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.013+0000 7fbe723e7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe58004d00 0x7fbe6c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.013+0000 7fbe723e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 0x7fbe6c141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.013+0000 7fbe723e7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6c06c520 0x7fbe6c06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.013+0000 7fbe723e7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c06cee0 con 0x7fbe6c15cc40 2023-10-16T12:35:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.013+0000 7fbe723e7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c1544b0 con 0x7fbe6c06c520 2023-10-16T12:35:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.013+0000 7fbe723e7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6c141e70 con 0x7fbe58004d00 2023-10-16T12:35:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 0x7fbe6c141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 0x7fbe6c141930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39166/0 (socket says 172.21.15.60:39166) 2023-10-16T12:35:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 -- 172.21.15.60:0/1453631672 learned_addr learned my addr 172.21.15.60:0/1453631672 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe6bfff640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe58004d00 0x7fbe6c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe70984640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6c06c520 0x7fbe6c06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 -- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe58004d00 msgr2=0x7fbe6c1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe58004d00 0x7fbe6c1531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 -- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6c06c520 msgr2=0x7fbe6c06c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6c06c520 0x7fbe6c06c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 -- 172.21.15.60:0/1453631672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c1538a0 con 0x7fbe6c15cc40 2023-10-16T12:35:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe70984640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6c06c520 0x7fbe6c06c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.014+0000 7fbe63fff640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 0x7fbe6c141930 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbe54011730 tx=0x7fbe54011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.015+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5400f960 con 0x7fbe6c15cc40 2023-10-16T12:35:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.015+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5400fac0 con 0x7fbe6c15cc40 2023-10-16T12:35:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.015+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5400fe70 con 0x7fbe6c15cc40 2023-10-16T12:35:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.015+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6c153b80 con 0x7fbe6c15cc40 2023-10-16T12:35:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.015+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6c154140 con 0x7fbe6c15cc40 2023-10-16T12:35:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.016+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6c04e160 con 0x7fbe6c15cc40 2023-10-16T12:35:10.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.017+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbe54015460 con 0x7fbe6c15cc40 2023-10-16T12:35:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.018+0000 7fbe697fa640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbe4c06b100 0x7fbe4c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.018+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbe5408a5b0 con 0x7fbe6c15cc40 2023-10-16T12:35:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.019+0000 7fbe6bfff640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbe4c06b100 0x7fbe4c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.020+0000 7fbe6bfff640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbe4c06b100 0x7fbe4c06d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c001dc0 tx=0x7fbe5c000fc0 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.023+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe54055160 con 0x7fbe6c15cc40 2023-10-16T12:35:10.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.245+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe6c0cda90 con 0x7fbe4c06b100 2023-10-16T12:35:10.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.402+0000 7fbe697fa640 1 -- 172.21.15.60:0/1453631672 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fbe5c001430 con 0x7fbe4c06b100 2023-10-16T12:35:10.405 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-16T12:35:10.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbe4c06b100 msgr2=0x7fbe4c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbe4c06b100 0x7fbe4c06d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c001dc0 tx=0x7fbe5c000fc0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 msgr2=0x7fbe6c141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 0x7fbe6c141930 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbe54011730 tx=0x7fbe54011bf0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 shutdown_connections 2023-10-16T12:35:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fbe4c06b100 0x7fbe4c06d5c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6c06c520 0x7fbe6c06c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe6c15cc40 0x7fbe6c141930 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 --2- 172.21.15.60:0/1453631672 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe58004d00 0x7fbe6c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 >> 172.21.15.60:0/1453631672 conn(0x7fbe6c0ce9b0 msgr2=0x7fbe6c06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:10.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 shutdown_connections 2023-10-16T12:35:10.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.406+0000 7fbe723e7640 1 -- 172.21.15.60:0/1453631672 wait complete. 2023-10-16T12:35:10.430 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-16T12:35:10.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-10-16T12:35:10.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.656+0000 7fa3f5a75640 1 Processor -- start 2023-10-16T12:35:10.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.656+0000 7fa3f5a75640 1 -- start start 2023-10-16T12:35:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.657+0000 7fa3f5a75640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3f00b3420 0x7fa3f00b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.657+0000 7fa3f5a75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.657+0000 7fa3f5a75640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00d3980 con 0x7fa3f00cf710 2023-10-16T12:35:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.657+0000 7fa3f5a75640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f00d3c60 con 0x7fa3f00b3420 2023-10-16T12:35:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.657+0000 7fa3f5a75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00ce630 con 0x7fa3f00cfaf0 2023-10-16T12:35:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.658+0000 7fa3ef7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa3f00cf710 0x7fa3f00b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37682/0 (socket says 172.21.15.60:37682) 2023-10-16T12:35:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.658+0000 7fa3ef7fe640 1 -- 172.21.15.60:0/331766403 learned_addr learned my addr 172.21.15.60:0/331766403 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.658+0000 7fa3effff640 1 --2- 172.21.15.60:0/331766403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.658+0000 7fa3eeffd640 1 --2- 172.21.15.60:0/331766403 >> v2:172.21.15.144:3300/0 conn(0x7fa3f00b3420 0x7fa3f00b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.659+0000 7fa3ee7fc640 1 -- 172.21.15.60:0/331766403 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096205759 0 0) 0x7fa3f00d3980 con 0x7fa3f00cf710 2023-10-16T12:35:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.659+0000 7fa3ee7fc640 1 -- 172.21.15.60:0/331766403 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc003520 con 0x7fa3f00cf710 2023-10-16T12:35:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.659+0000 7fa3effff640 1 -- 172.21.15.60:0/331766403 >> v2:172.21.15.144:3300/0 conn(0x7fa3f00b3420 msgr2=0x7fa3f00b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.659+0000 7fa3effff640 1 --2- 172.21.15.60:0/331766403 >> v2:172.21.15.144:3300/0 conn(0x7fa3f00b3420 0x7fa3f00b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.659+0000 7fa3effff640 1 -- 172.21.15.60:0/331766403 >> v1:172.21.15.60:6789/0 conn(0x7fa3f00cf710 legacy=0x7fa3f00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.659+0000 7fa3effff640 1 -- 172.21.15.60:0/331766403 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00ce910 con 0x7fa3f00cfaf0 2023-10-16T12:35:10.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.660+0000 7fa3effff640 1 --2- 172.21.15.60:0/331766403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00b6200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80097a0 tx=0x7fa3e802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f9fe44b36438df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.661+0000 7fa3ee7fc640 1 -- 172.21.15.60:0/331766403 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8031780 con 0x7fa3f00cfaf0 2023-10-16T12:35:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.661+0000 7fa3ee7fc640 1 -- 172.21.15.60:0/331766403 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e80318e0 con 0x7fa3f00cfaf0 2023-10-16T12:35:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.661+0000 7fa3ee7fc640 1 -- 172.21.15.60:0/331766403 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8031c70 con 0x7fa3f00cfaf0 2023-10-16T12:35:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 -- 172.21.15.60:0/331766403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 msgr2=0x7fa3f00b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/331766403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00b6200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80097a0 tx=0x7fa3e802b820 comp rx=0 tx=0).stop 2023-10-16T12:35:10.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 -- 172.21.15.60:0/331766403 shutdown_connections 2023-10-16T12:35:10.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/331766403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00b6200 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/331766403 >> v2:172.21.15.144:3300/0 conn(0x7fa3f00b3420 0x7fa3f00b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 -- 172.21.15.60:0/331766403 >> 172.21.15.60:0/331766403 conn(0x7fa3f00d2cc0 msgr2=0x7fa3f00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:10.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 -- 172.21.15.60:0/331766403 shutdown_connections 2023-10-16T12:35:10.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.662+0000 7fa3f5a75640 1 -- 172.21.15.60:0/331766403 wait complete. 2023-10-16T12:35:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.664+0000 7fa3f5a75640 1 Processor -- start 2023-10-16T12:35:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.664+0000 7fa3f5a75640 1 -- start start 2023-10-16T12:35:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3f5a75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 0x7fa3f0151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3f5a75640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3f00cf710 0x7fa3f01f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3f5a75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3f5a75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00ca300 con 0x7fa3f00b3420 2023-10-16T12:35:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3f5a75640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00ca5e0 con 0x7fa3f00cf710 2023-10-16T12:35:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3f5a75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00ca8c0 con 0x7fa3f00cfaf0 2023-10-16T12:35:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 0x7fa3f0151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 0x7fa3f0151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39168/0 (socket says 172.21.15.60:39168) 2023-10-16T12:35:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3ef7fe640 1 -- 172.21.15.60:0/2740723906 learned_addr learned my addr 172.21.15.60:0/2740723906 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3effff640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.665+0000 7fa3eeffd640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3f00cf710 0x7fa3f01f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3ef7fe640 1 -- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 msgr2=0x7fa3f00cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3ef7fe640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3ef7fe640 1 -- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3f00cf710 msgr2=0x7fa3f01f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3ef7fe640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3f00cf710 0x7fa3f01f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3ef7fe640 1 -- 172.21.15.60:0/2740723906 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8009400 con 0x7fa3f00b3420 2023-10-16T12:35:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3ef7fe640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 0x7fa3f0151f90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00015e0 tx=0x7fa3e000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e000a640 con 0x7fa3f00b3420 2023-10-16T12:35:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e000a7a0 con 0x7fa3f00b3420 2023-10-16T12:35:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e000ab90 con 0x7fa3f00b3420 2023-10-16T12:35:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f00caba0 con 0x7fa3f00b3420 2023-10-16T12:35:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.666+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f01524d0 con 0x7fa3f00b3420 2023-10-16T12:35:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.668+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f004e160 con 0x7fa3f00b3420 2023-10-16T12:35:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.669+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fa3e000acf0 con 0x7fa3f00b3420 2023-10-16T12:35:10.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.669+0000 7fa3e7fff640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa3d006b150 0x7fa3d006d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:10.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.669+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa3e0089470 con 0x7fa3f00b3420 2023-10-16T12:35:10.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.670+0000 7fa3eeffd640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa3d006b150 0x7fa3d006d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:10.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.671+0000 7fa3eeffd640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa3d006b150 0x7fa3d006d610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa3f01f56c0 tx=0x7fa3d8008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:10.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.674+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3e00541a0 con 0x7fa3f00b3420 2023-10-16T12:35:10.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:10.894+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f00d1d50 con 0x7fa3d006b150 2023-10-16T12:35:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.051+0000 7fa3e7fff640 1 -- 172.21.15.60:0/2740723906 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa3d8009680 con 0x7fa3d006b150 2023-10-16T12:35:11.054 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-16T12:35:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.054+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa3d006b150 msgr2=0x7fa3d006d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.054+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa3d006b150 0x7fa3d006d610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa3f01f56c0 tx=0x7fa3d8008040 comp rx=0 tx=0).stop 2023-10-16T12:35:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.054+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 msgr2=0x7fa3f0151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.054+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 0x7fa3f0151f90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00015e0 tx=0x7fa3e000ccd0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 shutdown_connections 2023-10-16T12:35:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fa3d006b150 0x7fa3d006d610 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3f00cfaf0 0x7fa3f00cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3f00cf710 0x7fa3f01f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 --2- 172.21.15.60:0/2740723906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3f00b3420 0x7fa3f0151f90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 >> 172.21.15.60:0/2740723906 conn(0x7fa3f00d2cc0 msgr2=0x7fa3f00d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 shutdown_connections 2023-10-16T12:35:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.055+0000 7fa3f5a75640 1 -- 172.21.15.60:0/2740723906 wait complete. 2023-10-16T12:35:11.079 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-16T12:35:11.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-10-16T12:35:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.299+0000 7f5cb20f7640 1 Processor -- start 2023-10-16T12:35:11.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.299+0000 7f5cb20f7640 1 -- start start 2023-10-16T12:35:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.301+0000 7f5cb20f7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5cac15d150 0x7f5cac0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.301+0000 7f5cb20f7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac160b90 0x7f5cac160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.301+0000 7f5cb20f7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5cac069da0 0x7f5cac06a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.301+0000 7f5cb20f7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac15d530 con 0x7f5cac069da0 2023-10-16T12:35:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.301+0000 7f5cb20f7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac06a6f0 con 0x7f5cac15d150 2023-10-16T12:35:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.301+0000 7f5cb20f7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac0ac890 con 0x7f5cac160b90 2023-10-16T12:35:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5cac069da0 0x7f5cac06a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5cac069da0 0x7f5cac06a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39182/0 (socket says 172.21.15.60:39182) 2023-10-16T12:35:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 -- 172.21.15.60:0/3427755207 learned_addr learned my addr 172.21.15.60:0/3427755207 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5ca2ffd640 1 --2- 172.21.15.60:0/3427755207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac160b90 0x7f5cac160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cab7fe640 1 --2- 172.21.15.60:0/3427755207 >> v2:172.21.15.144:3300/0 conn(0x7f5cac15d150 0x7f5cac0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 -- 172.21.15.60:0/3427755207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac160b90 msgr2=0x7f5cac160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 --2- 172.21.15.60:0/3427755207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac160b90 0x7f5cac160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 -- 172.21.15.60:0/3427755207 >> v2:172.21.15.144:3300/0 conn(0x7f5cac15d150 msgr2=0x7f5cac0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 --2- 172.21.15.60:0/3427755207 >> v2:172.21.15.144:3300/0 conn(0x7f5cac15d150 0x7f5cac0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.302+0000 7f5cabfff640 1 -- 172.21.15.60:0/3427755207 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac161650 con 0x7f5cac069da0 2023-10-16T12:35:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.303+0000 7f5cabfff640 1 --2- 172.21.15.60:0/3427755207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac069da0 0x7f5cac06a1b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0b15a0 tx=0x7f5ca402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fbb0cb043a5d8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5caaffd640 1 -- 172.21.15.60:0/3427755207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca402ca80 con 0x7f5cac069da0 2023-10-16T12:35:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5caaffd640 1 -- 172.21.15.60:0/3427755207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca402cbe0 con 0x7f5cac069da0 2023-10-16T12:35:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5caaffd640 1 -- 172.21.15.60:0/3427755207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca40343f0 con 0x7f5cac069da0 2023-10-16T12:35:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 -- 172.21.15.60:0/3427755207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac069da0 msgr2=0x7f5cac06a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/3427755207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac069da0 0x7f5cac06a1b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0b15a0 tx=0x7f5ca402bee0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 -- 172.21.15.60:0/3427755207 shutdown_connections 2023-10-16T12:35:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/3427755207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac069da0 0x7f5cac06a1b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/3427755207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac160b90 0x7f5cac160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/3427755207 >> v2:172.21.15.144:3300/0 conn(0x7f5cac15d150 0x7f5cac0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.304+0000 7f5cb20f7640 1 -- 172.21.15.60:0/3427755207 >> 172.21.15.60:0/3427755207 conn(0x7f5cac0ce9b0 msgr2=0x7f5cac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.305+0000 7f5cb20f7640 1 -- 172.21.15.60:0/3427755207 shutdown_connections 2023-10-16T12:35:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.305+0000 7f5cb20f7640 1 -- 172.21.15.60:0/3427755207 wait complete. 2023-10-16T12:35:11.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.306+0000 7f5cb20f7640 1 Processor -- start 2023-10-16T12:35:11.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.306+0000 7f5cb20f7640 1 -- start start 2023-10-16T12:35:11.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.307+0000 7f5cb20f7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac069da0 0x7f5cac0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.307+0000 7f5cb20f7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac15d150 0x7f5cac0c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.307+0000 7f5cb20f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 0x7f5cac0c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.307+0000 7f5cb20f7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac14e900 con 0x7f5cac160b90 2023-10-16T12:35:11.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.307+0000 7f5cb20f7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac0c4c90 con 0x7f5cac15d150 2023-10-16T12:35:11.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.307+0000 7f5cb20f7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac0ae240 con 0x7f5cac069da0 2023-10-16T12:35:11.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 0x7f5cac0c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 0x7f5cac0c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39188/0 (socket says 172.21.15.60:39188) 2023-10-16T12:35:11.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 -- 172.21.15.60:0/2174248390 learned_addr learned my addr 172.21.15.60:0/2174248390 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 -- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac069da0 msgr2=0x7f5cac0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac069da0 0x7f5cac0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 -- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac15d150 msgr2=0x7f5cac0c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5ca2ffd640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac15d150 0x7f5cac0c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac15d150 0x7f5cac0c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5cabfff640 1 -- 172.21.15.60:0/2174248390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4009b30 con 0x7f5cac160b90 2023-10-16T12:35:11.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.308+0000 7f5ca2ffd640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac15d150 0x7f5cac0c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:35:11.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.309+0000 7f5cabfff640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 0x7f5cac0c8190 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009f40 tx=0x7f5ca4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:11.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.309+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4009510 con 0x7f5cac160b90 2023-10-16T12:35:11.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.309+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca403e450 con 0x7f5cac160b90 2023-10-16T12:35:11.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.309+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca403e800 con 0x7f5cac160b90 2023-10-16T12:35:11.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.309+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac0c3130 con 0x7f5cac160b90 2023-10-16T12:35:11.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.309+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac0ae490 con 0x7f5cac160b90 2023-10-16T12:35:11.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.310+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac04e160 con 0x7f5cac160b90 2023-10-16T12:35:11.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.311+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5ca402cd50 con 0x7f5cac160b90 2023-10-16T12:35:11.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.313+0000 7f5ca8ff9640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5c8c06b130 0x7f5c8c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.313+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5ca40ac340 con 0x7f5cac160b90 2023-10-16T12:35:11.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.313+0000 7f5cab7fe640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5c8c06b130 0x7f5c8c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.314+0000 7f5cab7fe640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5c8c06b130 0x7f5c8c06d5f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009870 tx=0x7f5c9c008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:11.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.317+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ca4076f90 con 0x7f5cac160b90 2023-10-16T12:35:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.543+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cac0acb50 con 0x7f5c8c06b130 2023-10-16T12:35:11.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.698+0000 7f5ca8ff9640 1 -- 172.21.15.60:0/2174248390 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5c9c0090c0 con 0x7f5c8c06b130 2023-10-16T12:35:11.701 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-16T12:35:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5c8c06b130 msgr2=0x7f5c8c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5c8c06b130 0x7f5c8c06d5f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009870 tx=0x7f5c9c008040 comp rx=0 tx=0).stop 2023-10-16T12:35:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 msgr2=0x7f5cac0c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 0x7f5cac0c8190 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009f40 tx=0x7f5ca4004a40 comp rx=0 tx=0).stop 2023-10-16T12:35:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 shutdown_connections 2023-10-16T12:35:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5c8c06b130 0x7f5c8c06d5f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5cac160b90 0x7f5cac0c8190 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac15d150 0x7f5cac0c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 --2- 172.21.15.60:0/2174248390 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5cac069da0 0x7f5cac0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 >> 172.21.15.60:0/2174248390 conn(0x7f5cac0ce9b0 msgr2=0x7f5cac0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.701+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 shutdown_connections 2023-10-16T12:35:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.702+0000 7f5cb20f7640 1 -- 172.21.15.60:0/2174248390 wait complete. 2023-10-16T12:35:11.725 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-16T12:35:11.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-10-16T12:35:11.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.950+0000 7f32056b4640 1 Processor -- start 2023-10-16T12:35:11.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.950+0000 7f32056b4640 1 -- start start 2023-10-16T12:35:11.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.951+0000 7f32056b4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32000cb1c0 0x7f320006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.951+0000 7f32056b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3200065ab0 0x7f3200065f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.951+0000 7f32056b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320015eff0 con 0x7f320015af80 2023-10-16T12:35:11.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.951+0000 7f32056b4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3200066480 con 0x7f32000cb1c0 2023-10-16T12:35:11.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.951+0000 7f32056b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32000d0930 con 0x7f3200065ab0 2023-10-16T12:35:11.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31feffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f320015af80 0x7f320015f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37690/0 (socket says 172.21.15.60:37690) 2023-10-16T12:35:11.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31feffd640 1 -- 172.21.15.60:0/537385355 learned_addr learned my addr 172.21.15.60:0/537385355 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:35:11.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fe7fc640 1 --2- 172.21.15.60:0/537385355 >> v2:172.21.15.60:3300/0 conn(0x7f32000cb1c0 0x7f320006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31ff7fe640 1 --2- 172.21.15.60:0/537385355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3200065ab0 0x7f3200065f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fdffb640 1 -- 172.21.15.60:0/537385355 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339621390 0 0) 0x7f320015eff0 con 0x7f320015af80 2023-10-16T12:35:11.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fdffb640 1 -- 172.21.15.60:0/537385355 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec003520 con 0x7f320015af80 2023-10-16T12:35:11.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fe7fc640 1 -- 172.21.15.60:0/537385355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3200065ab0 msgr2=0x7f3200065f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:35:11.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fe7fc640 1 --2- 172.21.15.60:0/537385355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3200065ab0 0x7f3200065f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fe7fc640 1 -- 172.21.15.60:0/537385355 >> v1:172.21.15.60:6789/0 conn(0x7f320015af80 legacy=0x7f320015f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.952+0000 7f31fe7fc640 1 -- 172.21.15.60:0/537385355 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320015ecc0 con 0x7f32000cb1c0 2023-10-16T12:35:11.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.954+0000 7f31fe7fc640 1 --2- 172.21.15.60:0/537385355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000cb1c0 0x7f320006faf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f31e80097d0 tx=0x7f31e802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629d5ba20762103a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:11.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.954+0000 7f31fdffb640 1 -- 172.21.15.60:0/537385355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8031770 con 0x7f32000cb1c0 2023-10-16T12:35:11.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.954+0000 7f31fdffb640 1 -- 172.21.15.60:0/537385355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e80318d0 con 0x7f32000cb1c0 2023-10-16T12:35:11.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.954+0000 7f31fdffb640 1 -- 172.21.15.60:0/537385355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8031c60 con 0x7f32000cb1c0 2023-10-16T12:35:11.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 -- 172.21.15.60:0/537385355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000cb1c0 msgr2=0x7f320006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 --2- 172.21.15.60:0/537385355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000cb1c0 0x7f320006faf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f31e80097d0 tx=0x7f31e802b810 comp rx=0 tx=0).stop 2023-10-16T12:35:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 -- 172.21.15.60:0/537385355 shutdown_connections 2023-10-16T12:35:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 --2- 172.21.15.60:0/537385355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3200065ab0 0x7f3200065f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 --2- 172.21.15.60:0/537385355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32000cb1c0 0x7f320006faf0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 -- 172.21.15.60:0/537385355 >> 172.21.15.60:0/537385355 conn(0x7f32000d0ab0 msgr2=0x7f32000d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:11.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 -- 172.21.15.60:0/537385355 shutdown_connections 2023-10-16T12:35:11.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.955+0000 7f32056b4640 1 -- 172.21.15.60:0/537385355 wait complete. 2023-10-16T12:35:11.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.957+0000 7f32056b4640 1 Processor -- start 2023-10-16T12:35:11.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.957+0000 7f32056b4640 1 -- start start 2023-10-16T12:35:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f32056b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 0x7f320015e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f32056b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32000cb1c0 0x7f320015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f32056b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f320015af80 0x7f32001effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f32056b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32000c4b10 con 0x7f3200065ab0 2023-10-16T12:35:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f32056b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32000c7210 con 0x7f32000cb1c0 2023-10-16T12:35:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f32056b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32000c74f0 con 0x7f320015af80 2023-10-16T12:35:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f31feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 0x7f320015e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f31feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 0x7f320015e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39210/0 (socket says 172.21.15.60:39210) 2023-10-16T12:35:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.958+0000 7f31feffd640 1 -- 172.21.15.60:0/87727594 learned_addr learned my addr 172.21.15.60:0/87727594 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:35:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31fe7fc640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32000cb1c0 0x7f320015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31ff7fe640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f320015af80 0x7f32001effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31feffd640 1 -- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f320015af80 msgr2=0x7f32001effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31feffd640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f320015af80 0x7f32001effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31feffd640 1 -- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32000cb1c0 msgr2=0x7f320015b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31feffd640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32000cb1c0 0x7f320015b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31feffd640 1 -- 172.21.15.60:0/87727594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8009400 con 0x7f3200065ab0 2023-10-16T12:35:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31feffd640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 0x7f320015e040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f31f000cbc0 tx=0x7f31f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f000ad40 con 0x7f3200065ab0 2023-10-16T12:35:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f0012400 con 0x7f3200065ab0 2023-10-16T12:35:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.959+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f000aea0 con 0x7f3200065ab0 2023-10-16T12:35:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.960+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32000c4550 con 0x7f3200065ab0 2023-10-16T12:35:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.960+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320015bc40 con 0x7f3200065ab0 2023-10-16T12:35:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.961+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320004e160 con 0x7f3200065ab0 2023-10-16T12:35:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.962+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 10) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f31f0007070 con 0x7f3200065ab0 2023-10-16T12:35:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.963+0000 7f31f77fe640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f31e006b100 0x7f31e006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:35:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.963+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f31f0089c10 con 0x7f3200065ab0 2023-10-16T12:35:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.963+0000 7f31fe7fc640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f31e006b100 0x7f31e006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:35:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.965+0000 7f31fe7fc640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f31e006b100 0x7f31e006d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f31e8036000 tx=0x7f31e803d040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:35:11.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:11.967+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31f0054870 con 0x7f3200065ab0 2023-10-16T12:35:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.187+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f320015b360 con 0x7f31e006b100 2023-10-16T12:35:12.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.340+0000 7f31f77fe640 1 -- 172.21.15.60:0/87727594 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f31e8031a40 con 0x7f31e006b100 2023-10-16T12:35:12.343 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-16T12:35:12.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.343+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f31e006b100 msgr2=0x7f31e006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:12.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.343+0000 7f32056b4640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f31e006b100 0x7f31e006d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f31e8036000 tx=0x7f31e803d040 comp rx=0 tx=0).stop 2023-10-16T12:35:12.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.343+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 msgr2=0x7f320015e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:35:12.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.343+0000 7f32056b4640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 0x7f320015e040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f31f000cbc0 tx=0x7f31f000a430 comp rx=0 tx=0).stop 2023-10-16T12:35:12.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 shutdown_connections 2023-10-16T12:35:12.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f31e006b100 0x7f31e006d5c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:12.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f320015af80 0x7f32001effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:12.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32000cb1c0 0x7f320015b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:12.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 --2- 172.21.15.60:0/87727594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3200065ab0 0x7f320015e040 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:35:12.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 >> 172.21.15.60:0/87727594 conn(0x7f32000d0ab0 msgr2=0x7f32000b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:35:12.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 shutdown_connections 2023-10-16T12:35:12.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:35:12.344+0000 7f32056b4640 1 -- 172.21.15.60:0/87727594 wait complete. 2023-10-16T12:35:12.366 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-16T12:35:20.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:20.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:51.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:51.327 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:21.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:21.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:51.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:51.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:21.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:21.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:52.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:52.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:22.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:22.230 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:52.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:52.385 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:22.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:22.634 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:53.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:53.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:00.464 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:40:00.461+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:40:00.464 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:40:00.461+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:40:00.464 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:40:00.461+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:40:00.465 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:40:00.461+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:40:00.465 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:40:00.461+0000 7f8dc82dc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:40:23.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:23.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:53.335 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:53.345 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:23.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:23.500 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:53.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:53.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:23.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:23.801 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:54.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:54.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:24.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:24.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:54.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:54.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:24.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:24.713 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:54.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:54.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:25.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:25.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:55.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:55.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:22.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-10-16T12:46:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.612+0000 7f356421c640 1 Processor -- start 2023-10-16T12:46:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.612+0000 7f356421c640 1 -- start start 2023-10-16T12:46:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.613+0000 7f356421c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4d50 0x7f355c0cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.613+0000 7f356421c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f355c071bc0 0x7f355c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f356421c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0cabe0 con 0x7f355c0b4580 2023-10-16T12:46:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f356421c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c0b4960 con 0x7f355c071bc0 2023-10-16T12:46:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f356421c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c06ff20 con 0x7f355c0b4d50 2023-10-16T12:46:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35627b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f355c071bc0 0x7f355c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35627b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f355c071bc0 0x7f355c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58620/0 (socket says 172.21.15.60:58620) 2023-10-16T12:46:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35627b9640 1 -- 172.21.15.60:0/3990669573 learned_addr learned my addr 172.21.15.60:0/3990669573 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:22.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35617b7640 1 --2- 172.21.15.60:0/3990669573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4d50 0x7f355c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:22.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35627b9640 1 -- 172.21.15.60:0/3990669573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4d50 msgr2=0x7f355c0cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:22.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35627b9640 1 --2- 172.21.15.60:0/3990669573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4d50 0x7f355c0cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:22.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.614+0000 7f35627b9640 1 -- 172.21.15.60:0/3990669573 >> v1:172.21.15.60:6789/0 conn(0x7f355c0b4580 legacy=0x7f355c0714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:22.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.615+0000 7f35627b9640 1 -- 172.21.15.60:0/3990669573 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0ca2b0 con 0x7f355c071bc0 2023-10-16T12:46:22.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.616+0000 7f35627b9640 1 --2- 172.21.15.60:0/3990669573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c069da0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3554009870 tx=0x7f355402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ba096bd899583c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:22.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.616+0000 7f3560fb6640 1 -- 172.21.15.60:0/3990669573 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554030840 con 0x7f355c071bc0 2023-10-16T12:46:22.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.616+0000 7f3560fb6640 1 -- 172.21.15.60:0/3990669573 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35540309a0 con 0x7f355c071bc0 2023-10-16T12:46:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.616+0000 7f3560fb6640 1 -- 172.21.15.60:0/3990669573 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554030d30 con 0x7f355c071bc0 2023-10-16T12:46:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 -- 172.21.15.60:0/3990669573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 msgr2=0x7f355c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 --2- 172.21.15.60:0/3990669573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c069da0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3554009870 tx=0x7f355402b980 comp rx=0 tx=0).stop 2023-10-16T12:46:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 -- 172.21.15.60:0/3990669573 shutdown_connections 2023-10-16T12:46:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 --2- 172.21.15.60:0/3990669573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c069da0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 --2- 172.21.15.60:0/3990669573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4d50 0x7f355c0cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 -- 172.21.15.60:0/3990669573 >> 172.21.15.60:0/3990669573 conn(0x7f355c06bc10 msgr2=0x7f355c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 -- 172.21.15.60:0/3990669573 shutdown_connections 2023-10-16T12:46:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.617+0000 7f356421c640 1 -- 172.21.15.60:0/3990669573 wait complete. 2023-10-16T12:46:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.619+0000 7f356421c640 1 Processor -- start 2023-10-16T12:46:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.619+0000 7f356421c640 1 -- start start 2023-10-16T12:46:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f356421c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f356421c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4580 0x7f355c1577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f356421c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f355c0b4d50 0x7f355c0c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f356421c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0c9490 con 0x7f355c071bc0 2023-10-16T12:46:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f356421c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c157cf0 con 0x7f355c0b4d50 2023-10-16T12:46:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f356421c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0c8bf0 con 0x7f355c0b4580 2023-10-16T12:46:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f3561fb8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f3561fb8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58632/0 (socket says 172.21.15.60:58632) 2023-10-16T12:46:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f3561fb8640 1 -- 172.21.15.60:0/1804965185 learned_addr learned my addr 172.21.15.60:0/1804965185 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f35617b7640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4580 0x7f355c1577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f3561fb8640 1 -- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4580 msgr2=0x7f355c1577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f35627b9640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f355c0b4d50 0x7f355c0c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.620+0000 7f3561fb8640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4580 0x7f355c1577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f3561fb8640 1 -- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f355c0b4d50 msgr2=0x7f355c0c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f3561fb8640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f355c0b4d50 0x7f355c0c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f3561fb8640 1 -- 172.21.15.60:0/1804965185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540094d0 con 0x7f355c071bc0 2023-10-16T12:46:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f35627b9640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f355c0b4d50 0x7f355c0c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f3561fb8640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c15dca0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c870 tx=0x7f354c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c003b70 con 0x7f355c071bc0 2023-10-16T12:46:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c003cf0 con 0x7f355c071bc0 2023-10-16T12:46:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c0c8ed0 con 0x7f355c071bc0 2023-10-16T12:46:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c00aa90 con 0x7f355c071bc0 2023-10-16T12:46:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.621+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c0c2e80 con 0x7f355c071bc0 2023-10-16T12:46:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.622+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c04e160 con 0x7f355c071bc0 2023-10-16T12:46:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.624+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f354c00abf0 con 0x7f355c071bc0 2023-10-16T12:46:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.625+0000 7f355a7fc640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f354806b070 0x7f354806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.625+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f354c08a350 con 0x7f355c071bc0 2023-10-16T12:46:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.626+0000 7f35617b7640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f354806b070 0x7f354806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:22.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.627+0000 7f35617b7640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f354806b070 0x7f354806d530 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f355c071120 tx=0x7f3550008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:22.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.628+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f354c055160 con 0x7f355c071bc0 2023-10-16T12:46:22.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:22.850+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f355c0ca870 con 0x7f354806b070 2023-10-16T12:46:23.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.069+0000 7f355a7fc640 1 -- 172.21.15.60:0/1804965185 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3550009540 con 0x7f354806b070 2023-10-16T12:46:23.071 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T12:46:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f354806b070 msgr2=0x7f354806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f354806b070 0x7f354806d530 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f355c071120 tx=0x7f3550008040 comp rx=0 tx=0).stop 2023-10-16T12:46:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 msgr2=0x7f355c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c15dca0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c870 tx=0x7f354c00cd30 comp rx=0 tx=0).stop 2023-10-16T12:46:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 shutdown_connections 2023-10-16T12:46:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f354806b070 0x7f354806d530 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f355c0b4580 0x7f355c1577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f355c0b4d50 0x7f355c0c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 --2- 172.21.15.60:0/1804965185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f355c071bc0 0x7f355c15dca0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.072+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 >> 172.21.15.60:0/1804965185 conn(0x7f355c06bc10 msgr2=0x7f355c0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.073+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 shutdown_connections 2023-10-16T12:46:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.073+0000 7f356421c640 1 -- 172.21.15.60:0/1804965185 wait complete. 2023-10-16T12:46:23.097 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T12:46:23.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-10-16T12:46:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.331+0000 7fca09800640 1 Processor -- start 2023-10-16T12:46:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.331+0000 7fca09800640 1 -- start start 2023-10-16T12:46:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.332+0000 7fca09800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca0415d150 0x7fca040ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca09800640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca04160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca09800640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca04069da0 0x7fca0406a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca09800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0415d530 con 0x7fca0415d150 2023-10-16T12:46:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca09800640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0406a6f0 con 0x7fca04069da0 2023-10-16T12:46:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca09800640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca040ac890 con 0x7fca04160b90 2023-10-16T12:46:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca02ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca0415d150 0x7fca040ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca02ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca0415d150 0x7fca040ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58638/0 (socket says 172.21.15.60:58638) 2023-10-16T12:46:23.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca02ffd640 1 -- 172.21.15.60:0/4204546503 learned_addr learned my addr 172.21.15.60:0/4204546503 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:23.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca027fc640 1 --2- 172.21.15.60:0/4204546503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca04160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca02ffd640 1 -- 172.21.15.60:0/4204546503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 msgr2=0x7fca04160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:23.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca037fe640 1 --2- 172.21.15.60:0/4204546503 >> v2:172.21.15.144:3300/0 conn(0x7fca04069da0 0x7fca0406a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca02ffd640 1 --2- 172.21.15.60:0/4204546503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca04160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.333+0000 7fca02ffd640 1 -- 172.21.15.60:0/4204546503 >> v2:172.21.15.144:3300/0 conn(0x7fca04069da0 msgr2=0x7fca0406a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:23.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.334+0000 7fca02ffd640 1 --2- 172.21.15.60:0/4204546503 >> v2:172.21.15.144:3300/0 conn(0x7fca04069da0 0x7fca0406a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.334+0000 7fca02ffd640 1 -- 172.21.15.60:0/4204546503 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca04161650 con 0x7fca0415d150 2023-10-16T12:46:23.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.334+0000 7fca037fe640 1 --2- 172.21.15.60:0/4204546503 >> v2:172.21.15.144:3300/0 conn(0x7fca04069da0 0x7fca0406a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:23.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.335+0000 7fca02ffd640 1 --2- 172.21.15.60:0/4204546503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca040ad350 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fca040cf990 tx=0x7fc9f402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a76bb1d8813e47a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:23.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.335+0000 7fca01ffb640 1 -- 172.21.15.60:0/4204546503 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f402ca80 con 0x7fca0415d150 2023-10-16T12:46:23.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.335+0000 7fca01ffb640 1 -- 172.21.15.60:0/4204546503 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f402cbe0 con 0x7fca0415d150 2023-10-16T12:46:23.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.335+0000 7fca01ffb640 1 -- 172.21.15.60:0/4204546503 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40343f0 con 0x7fca0415d150 2023-10-16T12:46:23.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 -- 172.21.15.60:0/4204546503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 msgr2=0x7fca040ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 --2- 172.21.15.60:0/4204546503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca040ad350 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fca040cf990 tx=0x7fc9f402bee0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 -- 172.21.15.60:0/4204546503 shutdown_connections 2023-10-16T12:46:23.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 --2- 172.21.15.60:0/4204546503 >> v2:172.21.15.144:3300/0 conn(0x7fca04069da0 0x7fca0406a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 --2- 172.21.15.60:0/4204546503 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca04160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 --2- 172.21.15.60:0/4204546503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca040ad350 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 -- 172.21.15.60:0/4204546503 >> 172.21.15.60:0/4204546503 conn(0x7fca040ce9b0 msgr2=0x7fca040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 -- 172.21.15.60:0/4204546503 shutdown_connections 2023-10-16T12:46:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.336+0000 7fca09800640 1 -- 172.21.15.60:0/4204546503 wait complete. 2023-10-16T12:46:23.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.338+0000 7fca09800640 1 Processor -- start 2023-10-16T12:46:23.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.338+0000 7fca09800640 1 -- start start 2023-10-16T12:46:23.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca09800640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca04069da0 0x7fca040b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca09800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca04155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca09800640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca040af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca09800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca04160750 con 0x7fca0415d150 2023-10-16T12:46:23.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca09800640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0415e240 con 0x7fca04069da0 2023-10-16T12:46:23.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca09800640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0415e520 con 0x7fca04160b90 2023-10-16T12:46:23.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca027fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca04155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca027fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca04155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58644/0 (socket says 172.21.15.60:58644) 2023-10-16T12:46:23.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca027fc640 1 -- 172.21.15.60:0/3804603793 learned_addr learned my addr 172.21.15.60:0/3804603793 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:23.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.339+0000 7fca02ffd640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca04069da0 0x7fca040b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca037fe640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca040af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca027fc640 1 -- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 msgr2=0x7fca040af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca027fc640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca040af870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca027fc640 1 -- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca04069da0 msgr2=0x7fca040b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca027fc640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca04069da0 0x7fca040b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca027fc640 1 -- 172.21.15.60:0/3804603793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f4009b30 con 0x7fca0415d150 2023-10-16T12:46:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca037fe640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca040af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca027fc640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca04155c90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00c7f0 tx=0x7fc9ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec004220 con 0x7fca0415d150 2023-10-16T12:46:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec0043a0 con 0x7fca0415d150 2023-10-16T12:46:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0415e800 con 0x7fca0415d150 2023-10-16T12:46:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.340+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec007860 con 0x7fca0415d150 2023-10-16T12:46:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.341+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca040afdb0 con 0x7fca0415d150 2023-10-16T12:46:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.342+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0415ed20 con 0x7fca0415d150 2023-10-16T12:46:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.343+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc9ec007bd0 con 0x7fca0415d150 2023-10-16T12:46:23.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.344+0000 7fc9fb7fe640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc9e406b070 0x7fc9e406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:23.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.344+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc9ec089af0 con 0x7fca0415d150 2023-10-16T12:46:23.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.345+0000 7fca02ffd640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc9e406b070 0x7fc9e406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:23.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.346+0000 7fca02ffd640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc9e406b070 0x7fc9e406d530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4035000 tx=0x7fc9f4030000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:23.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.349+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca0415ed20 con 0x7fca0415d150 2023-10-16T12:46:23.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.575+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fca0415e690 con 0x7fc9e406b070 2023-10-16T12:46:23.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.787+0000 7fc9fb7fe640 1 -- 172.21.15.60:0/3804603793 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fca0415e690 con 0x7fc9e406b070 2023-10-16T12:46:23.790 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T12:46:23.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc9e406b070 msgr2=0x7fc9e406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc9e406b070 0x7fc9e406d530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4035000 tx=0x7fc9f4030000 comp rx=0 tx=0).stop 2023-10-16T12:46:23.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 msgr2=0x7fca04155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:23.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca04155c90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00c7f0 tx=0x7fc9ec00ccb0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 shutdown_connections 2023-10-16T12:46:23.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc9e406b070 0x7fc9e406d530 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca04160b90 0x7fca040af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca0415d150 0x7fca04155c90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 --2- 172.21.15.60:0/3804603793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca04069da0 0x7fca040b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:23.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 >> 172.21.15.60:0/3804603793 conn(0x7fca040ce9b0 msgr2=0x7fca040c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:23.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 shutdown_connections 2023-10-16T12:46:23.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:23.791+0000 7fca09800640 1 -- 172.21.15.60:0/3804603793 wait complete. 2023-10-16T12:46:23.814 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T12:46:23.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-10-16T12:46:24.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.043+0000 7f3108fc9640 1 Processor -- start 2023-10-16T12:46:24.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.044+0000 7f3108fc9640 1 -- start start 2023-10-16T12:46:24.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.045+0000 7f3108fc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f310415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.045+0000 7f3108fc9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f310406c810 0x7f31040708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.045+0000 7f3108fc9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3104070df0 con 0x7f3104070450 2023-10-16T12:46:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.045+0000 7f3108fc9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310415e490 con 0x7f310406c810 2023-10-16T12:46:24.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.045+0000 7f3108fc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310415aef0 con 0x7f3104159950 2023-10-16T12:46:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.046+0000 7f31037fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f310415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.046+0000 7f3103fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f310406c810 0x7f31040708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.046+0000 7f31037fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f310415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:42800/0 (socket says 172.21.15.60:42800) 2023-10-16T12:46:24.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.046+0000 7f31037fe640 1 -- 172.21.15.60:0/3319725 learned_addr learned my addr 172.21.15.60:0/3319725 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:24.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.047+0000 7f31037fe640 1 -- 172.21.15.60:0/3319725 >> v2:172.21.15.144:3300/0 conn(0x7f310406c810 msgr2=0x7f31040708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:24.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.047+0000 7f31037fe640 1 --2- 172.21.15.60:0/3319725 >> v2:172.21.15.144:3300/0 conn(0x7f310406c810 0x7f31040708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.047+0000 7f31037fe640 1 -- 172.21.15.60:0/3319725 >> v1:172.21.15.144:6789/0 conn(0x7f3104070450 legacy=0x7f310406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.047+0000 7f31037fe640 1 -- 172.21.15.60:0/3319725 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310406cc80 con 0x7f3104159950 2023-10-16T12:46:24.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.047+0000 7f3103fff640 1 --2- 172.21.15.60:0/3319725 >> v2:172.21.15.144:3300/0 conn(0x7f310406c810 0x7f31040708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:46:24.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.048+0000 7f31037fe640 1 --2- 172.21.15.60:0/3319725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f310415a9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009870 tx=0x7f30f402db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e421464ef89e479 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:24.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.049+0000 7f31027fc640 1 -- 172.21.15.60:0/3319725 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f402f8c0 con 0x7f3104159950 2023-10-16T12:46:24.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.049+0000 7f31027fc640 1 -- 172.21.15.60:0/3319725 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f402fa20 con 0x7f3104159950 2023-10-16T12:46:24.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.049+0000 7f31027fc640 1 -- 172.21.15.60:0/3319725 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f402fdb0 con 0x7f3104159950 2023-10-16T12:46:24.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 -- 172.21.15.60:0/3319725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 msgr2=0x7f310415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 --2- 172.21.15.60:0/3319725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f310415a9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009870 tx=0x7f30f402db00 comp rx=0 tx=0).stop 2023-10-16T12:46:24.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 -- 172.21.15.60:0/3319725 shutdown_connections 2023-10-16T12:46:24.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 --2- 172.21.15.60:0/3319725 >> v2:172.21.15.144:3300/0 conn(0x7f310406c810 0x7f31040708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 --2- 172.21.15.60:0/3319725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f310415a9b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 -- 172.21.15.60:0/3319725 >> 172.21.15.60:0/3319725 conn(0x7f31040ce6a0 msgr2=0x7f31040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:24.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 -- 172.21.15.60:0/3319725 shutdown_connections 2023-10-16T12:46:24.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.050+0000 7f3108fc9640 1 -- 172.21.15.60:0/3319725 wait complete. 2023-10-16T12:46:24.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.052+0000 7f3108fc9640 1 Processor -- start 2023-10-16T12:46:24.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.052+0000 7f3108fc9640 1 -- start start 2023-10-16T12:46:24.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f3108fc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 0x7f31041581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f3108fc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3104070450 0x7f31040c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f3108fc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f31040c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f3108fc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3104158c40 con 0x7f310406c810 2023-10-16T12:46:24.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f3108fc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040c8590 con 0x7f3104070450 2023-10-16T12:46:24.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f3108fc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31041f5fa0 con 0x7f3104159950 2023-10-16T12:46:24.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.053+0000 7f31037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 0x7f31041581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 0x7f31041581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58656/0 (socket says 172.21.15.60:58656) 2023-10-16T12:46:24.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 -- 172.21.15.60:0/2362655716 learned_addr learned my addr 172.21.15.60:0/2362655716 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:24.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f3102ffd640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3104070450 0x7f31040c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f3103fff640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f31040c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 -- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 msgr2=0x7f31040c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f31040c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 -- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3104070450 msgr2=0x7f31040c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3104070450 0x7f31040c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f31037fe640 1 -- 172.21.15.60:0/2362655716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc009a70 con 0x7f310406c810 2023-10-16T12:46:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f3102ffd640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3104070450 0x7f31040c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.054+0000 7f3103fff640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f31040c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.055+0000 7f31037fe640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 0x7f31041581b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f30f402db40 tx=0x7f30f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.055+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4042880 con 0x7f310406c810 2023-10-16T12:46:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.055+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f4042a20 con 0x7f310406c810 2023-10-16T12:46:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.055+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f40094d0 con 0x7f310406c810 2023-10-16T12:46:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.055+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4042dd0 con 0x7f310406c810 2023-10-16T12:46:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.055+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31041f6560 con 0x7f310406c810 2023-10-16T12:46:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.056+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310415d4e0 con 0x7f310406c810 2023-10-16T12:46:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.057+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f30f4040440 con 0x7f310406c810 2023-10-16T12:46:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.058+0000 7f30ebfff640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f30f006b070 0x7f30f006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.058+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f30f4033070 con 0x7f310406c810 2023-10-16T12:46:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.059+0000 7f3102ffd640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f30f006b070 0x7f30f006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.060+0000 7f3102ffd640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f30f006b070 0x7f30f006d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30f8007fd0 tx=0x7f30f8009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.062+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30f4077450 con 0x7f310406c810 2023-10-16T12:46:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.284+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f310415dbf0 con 0x7f30f006b070 2023-10-16T12:46:24.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.493+0000 7f30ebfff640 1 -- 172.21.15.60:0/2362655716 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f30f8001d30 con 0x7f30f006b070 2023-10-16T12:46:24.496 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T12:46:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.496+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f30f006b070 msgr2=0x7f30f006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.496+0000 7f3108fc9640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f30f006b070 0x7f30f006d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30f8007fd0 tx=0x7f30f8009040 comp rx=0 tx=0).stop 2023-10-16T12:46:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 msgr2=0x7f31041581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 0x7f31041581b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f30f402db40 tx=0x7f30f40046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 shutdown_connections 2023-10-16T12:46:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f30f006b070 0x7f30f006d530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3104159950 0x7f31040c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3104070450 0x7f31040c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 --2- 172.21.15.60:0/2362655716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310406c810 0x7f31041581b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 >> 172.21.15.60:0/2362655716 conn(0x7f31040ce6a0 msgr2=0x7f3104159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 shutdown_connections 2023-10-16T12:46:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.497+0000 7f3108fc9640 1 -- 172.21.15.60:0/2362655716 wait complete. 2023-10-16T12:46:24.520 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T12:46:24.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-10-16T12:46:24.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.749+0000 7fee86f7c640 1 Processor -- start 2023-10-16T12:46:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.749+0000 7fee86f7c640 1 -- start start 2023-10-16T12:46:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee86f7c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee8006a4b0 0x7fee800c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee86f7c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee800ac890 con 0x7fee8015d150 2023-10-16T12:46:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee86f7c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee800ad7a0 con 0x7fee800ad350 2023-10-16T12:46:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee86f7c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8015d530 con 0x7fee8006a4b0 2023-10-16T12:46:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee84d18640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fee8015d150 0x7fee80160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49494/0 (socket says 172.21.15.60:49494) 2023-10-16T12:46:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee85519640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee8006a4b0 0x7fee800c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.751+0000 7fee84d18640 1 -- 172.21.15.60:0/1873695160 learned_addr learned my addr 172.21.15.60:0/1873695160 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.752+0000 7fee7f7fe640 1 -- 172.21.15.60:0/1873695160 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194637678 0 0) 0x7fee800ac890 con 0x7fee8015d150 2023-10-16T12:46:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.752+0000 7fee7f7fe640 1 -- 172.21.15.60:0/1873695160 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee70003520 con 0x7fee8015d150 2023-10-16T12:46:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.752+0000 7fee85519640 1 -- 172.21.15.60:0/1873695160 >> v1:172.21.15.144:6789/0 conn(0x7fee800ad350 legacy=0x7fee80069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.752+0000 7fee85519640 1 -- 172.21.15.60:0/1873695160 >> v1:172.21.15.60:6789/0 conn(0x7fee8015d150 legacy=0x7fee80160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.752+0000 7fee85519640 1 -- 172.21.15.60:0/1873695160 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee80161650 con 0x7fee8006a4b0 2023-10-16T12:46:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.753+0000 7fee85519640 1 --2- 172.21.15.60:0/1873695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8006a4b0 0x7fee800c9000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fee78009d90 tx=0x7fee7802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2873ab92e265861 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.754+0000 7fee7f7fe640 1 -- 172.21.15.60:0/1873695160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7802cd00 con 0x7fee8006a4b0 2023-10-16T12:46:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.754+0000 7fee7f7fe640 1 -- 172.21.15.60:0/1873695160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7802ce60 con 0x7fee8006a4b0 2023-10-16T12:46:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.754+0000 7fee7f7fe640 1 -- 172.21.15.60:0/1873695160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78034660 con 0x7fee8006a4b0 2023-10-16T12:46:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.754+0000 7fee86f7c640 1 -- 172.21.15.60:0/1873695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8006a4b0 msgr2=0x7fee800c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.754+0000 7fee86f7c640 1 --2- 172.21.15.60:0/1873695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8006a4b0 0x7fee800c9000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fee78009d90 tx=0x7fee7802bf50 comp rx=0 tx=0).stop 2023-10-16T12:46:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.755+0000 7fee86f7c640 1 -- 172.21.15.60:0/1873695160 shutdown_connections 2023-10-16T12:46:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.755+0000 7fee86f7c640 1 --2- 172.21.15.60:0/1873695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8006a4b0 0x7fee800c9000 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.755+0000 7fee86f7c640 1 -- 172.21.15.60:0/1873695160 >> 172.21.15.60:0/1873695160 conn(0x7fee800ce9b0 msgr2=0x7fee800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.755+0000 7fee86f7c640 1 -- 172.21.15.60:0/1873695160 shutdown_connections 2023-10-16T12:46:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.755+0000 7fee86f7c640 1 -- 172.21.15.60:0/1873695160 wait complete. 2023-10-16T12:46:24.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.756+0000 7fee86f7c640 1 Processor -- start 2023-10-16T12:46:24.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.756+0000 7fee86f7c640 1 -- start start 2023-10-16T12:46:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.757+0000 7fee86f7c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee8006a4b0 0x7fee8015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.757+0000 7fee86f7c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee800ad350 0x7fee801553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.757+0000 7fee86f7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 0x7fee80155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.757+0000 7fee86f7c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee801e7090 con 0x7fee8015d150 2023-10-16T12:46:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.757+0000 7fee86f7c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8015b260 con 0x7fee8006a4b0 2023-10-16T12:46:24.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.757+0000 7fee86f7c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8015b540 con 0x7fee800ad350 2023-10-16T12:46:24.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 0x7fee80155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 0x7fee80155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58678/0 (socket says 172.21.15.60:58678) 2023-10-16T12:46:24.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 -- 172.21.15.60:0/3293042490 learned_addr learned my addr 172.21.15.60:0/3293042490 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:24.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee84d18640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee8006a4b0 0x7fee8015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee7ffff640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee800ad350 0x7fee801553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 -- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee800ad350 msgr2=0x7fee801553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee800ad350 0x7fee801553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 -- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee8006a4b0 msgr2=0x7fee8015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee8006a4b0 0x7fee8015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee85519640 1 -- 172.21.15.60:0/3293042490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee78009a70 con 0x7fee8015d150 2023-10-16T12:46:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.758+0000 7fee84d18640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee8006a4b0 0x7fee8015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.759+0000 7fee85519640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 0x7fee80155930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fee78009eb0 tx=0x7fee78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.759+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7803d600 con 0x7fee8015d150 2023-10-16T12:46:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.759+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7803d780 con 0x7fee8015d150 2023-10-16T12:46:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.759+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7803db30 con 0x7fee8015d150 2023-10-16T12:46:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.759+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8015b820 con 0x7fee8015d150 2023-10-16T12:46:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.759+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8014f7e0 con 0x7fee8015d150 2023-10-16T12:46:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.760+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8014fa00 con 0x7fee8015d150 2023-10-16T12:46:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.761+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fee7803b040 con 0x7fee8015d150 2023-10-16T12:46:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.762+0000 7fee7d7fa640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fee6406b070 0x7fee6406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.763+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fee780abfa0 con 0x7fee8015d150 2023-10-16T12:46:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.763+0000 7fee84d18640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fee6406b070 0x7fee6406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.764+0000 7fee84d18640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fee6406b070 0x7fee6406d530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fee74007fd0 tx=0x7fee74009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.767+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee78076cc0 con 0x7fee8015d150 2023-10-16T12:46:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:24.996+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fee800acb50 con 0x7fee6406b070 2023-10-16T12:46:25.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.214+0000 7fee7d7fa640 1 -- 172.21.15.60:0/3293042490 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fee74001d20 con 0x7fee6406b070 2023-10-16T12:46:25.217 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T12:46:25.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fee6406b070 msgr2=0x7fee6406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fee6406b070 0x7fee6406d530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fee74007fd0 tx=0x7fee74009040 comp rx=0 tx=0).stop 2023-10-16T12:46:25.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 msgr2=0x7fee80155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 0x7fee80155930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fee78009eb0 tx=0x7fee78004a40 comp rx=0 tx=0).stop 2023-10-16T12:46:25.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 shutdown_connections 2023-10-16T12:46:25.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fee6406b070 0x7fee6406d530 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee800ad350 0x7fee801553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee8015d150 0x7fee80155930 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 --2- 172.21.15.60:0/3293042490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee8006a4b0 0x7fee8015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.217+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 >> 172.21.15.60:0/3293042490 conn(0x7fee800ce9b0 msgr2=0x7fee8006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.218+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 shutdown_connections 2023-10-16T12:46:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.218+0000 7fee86f7c640 1 -- 172.21.15.60:0/3293042490 wait complete. 2023-10-16T12:46:25.242 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T12:46:25.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-10-16T12:46:25.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.465+0000 7f8e056ec640 1 Processor -- start 2023-10-16T12:46:25.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.465+0000 7f8e056ec640 1 -- start start 2023-10-16T12:46:25.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8e056ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e00160240 0x7f8e000caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:25.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8e056ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8e00160620 0x7f8e000ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:25.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8e056ec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0006afe0 con 0x7f8e000b1ca0 2023-10-16T12:46:25.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8e056ec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e000cf540 con 0x7f8e00160240 2023-10-16T12:46:25.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8e056ec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e000cf820 con 0x7f8e00160620 2023-10-16T12:46:25.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8dfffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e00160240 0x7f8e000caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:25.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8dfffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e00160240 0x7f8e000caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58680/0 (socket says 172.21.15.60:58680) 2023-10-16T12:46:25.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.467+0000 7f8dfffff640 1 -- 172.21.15.60:0/336314837 learned_addr learned my addr 172.21.15.60:0/336314837 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:25.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.468+0000 7f8dfffff640 1 -- 172.21.15.60:0/336314837 >> v2:172.21.15.144:3300/0 conn(0x7f8e00160620 msgr2=0x7f8e000ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:25.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.468+0000 7f8dff7fe640 1 --2- 172.21.15.60:0/336314837 >> v2:172.21.15.144:3300/0 conn(0x7f8e00160620 0x7f8e000ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.468+0000 7f8dfffff640 1 --2- 172.21.15.60:0/336314837 >> v2:172.21.15.144:3300/0 conn(0x7f8e00160620 0x7f8e000ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.468+0000 7f8dfffff640 1 -- 172.21.15.60:0/336314837 >> v1:172.21.15.144:6789/0 conn(0x7f8e000b1ca0 legacy=0x7f8e000cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.468+0000 7f8dfffff640 1 -- 172.21.15.60:0/336314837 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000cfb00 con 0x7f8e00160240 2023-10-16T12:46:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.469+0000 7f8dfffff640 1 --2- 172.21.15.60:0/336314837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e00160240 0x7f8e000caf60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8e000739f0 tx=0x7f8df002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15461f7f08636f2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:25.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.469+0000 7f8dfeffd640 1 -- 172.21.15.60:0/336314837 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df002c9f0 con 0x7f8e00160240 2023-10-16T12:46:25.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.469+0000 7f8dfeffd640 1 -- 172.21.15.60:0/336314837 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df002cb50 con 0x7f8e00160240 2023-10-16T12:46:25.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.469+0000 7f8dfeffd640 1 -- 172.21.15.60:0/336314837 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df00343f0 con 0x7f8e00160240 2023-10-16T12:46:25.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 -- 172.21.15.60:0/336314837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e00160240 msgr2=0x7f8e000caf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 --2- 172.21.15.60:0/336314837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e00160240 0x7f8e000caf60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8e000739f0 tx=0x7f8df002be50 comp rx=0 tx=0).stop 2023-10-16T12:46:25.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 -- 172.21.15.60:0/336314837 shutdown_connections 2023-10-16T12:46:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 --2- 172.21.15.60:0/336314837 >> v2:172.21.15.144:3300/0 conn(0x7f8e00160620 0x7f8e000ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 --2- 172.21.15.60:0/336314837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e00160240 0x7f8e000caf60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 -- 172.21.15.60:0/336314837 >> 172.21.15.60:0/336314837 conn(0x7f8e00099a10 msgr2=0x7f8e000d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 -- 172.21.15.60:0/336314837 shutdown_connections 2023-10-16T12:46:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.470+0000 7f8e056ec640 1 -- 172.21.15.60:0/336314837 wait complete. 2023-10-16T12:46:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.472+0000 7f8e056ec640 1 Processor -- start 2023-10-16T12:46:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.472+0000 7f8e056ec640 1 -- start start 2023-10-16T12:46:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8e056ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 0x7f8e000b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8e056ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e00160240 0x7f8e001e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8e056ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e00160620 0x7f8e001506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8e056ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e00151980 con 0x7f8e000b1ca0 2023-10-16T12:46:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8e056ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000b45c0 con 0x7f8e00160620 2023-10-16T12:46:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8e056ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000b42e0 con 0x7f8e00160240 2023-10-16T12:46:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8dfffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 0x7f8e000b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8dfffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 0x7f8e000b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58684/0 (socket says 172.21.15.60:58684) 2023-10-16T12:46:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.473+0000 7f8dfffff640 1 -- 172.21.15.60:0/508359097 learned_addr learned my addr 172.21.15.60:0/508359097 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfffff640 1 -- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e00160240 msgr2=0x7f8e001e5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfffff640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e00160240 0x7f8e001e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfffff640 1 -- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e00160620 msgr2=0x7f8e001506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8e04eeb640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e00160620 0x7f8e001506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfffff640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e00160620 0x7f8e001506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfffff640 1 -- 172.21.15.60:0/508359097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df0009b30 con 0x7f8e000b1ca0 2023-10-16T12:46:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8e04eeb640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e00160620 0x7f8e001506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfffff640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 0x7f8e000b3d00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8df0000f80 tx=0x7f8df0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df003e450 con 0x7f8e000b1ca0 2023-10-16T12:46:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df003e5f0 con 0x7f8e000b1ca0 2023-10-16T12:46:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df003e9a0 con 0x7f8e000b1ca0 2023-10-16T12:46:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.474+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e00151c20 con 0x7f8e000b1ca0 2023-10-16T12:46:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.475+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e00150c10 con 0x7f8e000b1ca0 2023-10-16T12:46:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.476+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e001517e0 con 0x7f8e000b1ca0 2023-10-16T12:46:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.477+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8df003eb00 con 0x7f8e000b1ca0 2023-10-16T12:46:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.478+0000 7f8dfcff9640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f8dec06b070 0x7f8dec06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.478+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8df00aca30 con 0x7f8e000b1ca0 2023-10-16T12:46:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.479+0000 7f8dff7fe640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f8dec06b070 0x7f8dec06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.480+0000 7f8dff7fe640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f8dec06b070 0x7f8dec06d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8e00159110 tx=0x7f8df4008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.483+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e001517e0 con 0x7f8e000b1ca0 2023-10-16T12:46:25.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:25.599 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:25.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.706+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e000cfde0 con 0x7f8dec06b070 2023-10-16T12:46:25.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.904+0000 7f8dfcff9640 1 -- 172.21.15.60:0/508359097 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8df400a420 con 0x7f8dec06b070 2023-10-16T12:46:25.907 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T12:46:25.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f8dec06b070 msgr2=0x7f8dec06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f8dec06b070 0x7f8dec06d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8e00159110 tx=0x7f8df4008040 comp rx=0 tx=0).stop 2023-10-16T12:46:25.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 msgr2=0x7f8e000b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:25.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 0x7f8e000b3d00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8df0000f80 tx=0x7f8df0004a40 comp rx=0 tx=0).stop 2023-10-16T12:46:25.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 shutdown_connections 2023-10-16T12:46:25.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f8dec06b070 0x7f8dec06d530 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e00160240 0x7f8e001e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.907+0000 7f8e056ec640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e00160620 0x7f8e001506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.908+0000 7f8e056ec640 1 --2- 172.21.15.60:0/508359097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e000b1ca0 0x7f8e000b3d00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:25.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.908+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 >> 172.21.15.60:0/508359097 conn(0x7f8e00099a10 msgr2=0x7f8e001552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:25.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.908+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 shutdown_connections 2023-10-16T12:46:25.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:25.908+0000 7f8e056ec640 1 -- 172.21.15.60:0/508359097 wait complete. 2023-10-16T12:46:25.930 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T12:46:25.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-10-16T12:46:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.152+0000 7f29e1547640 1 Processor -- start 2023-10-16T12:46:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.152+0000 7f29e1547640 1 -- start start 2023-10-16T12:46:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e1547640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f29dc15c9f0 0x7f29dc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e1547640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc0c9230 0x7f29dc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e1547640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29dc070f20 0x7f29dc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e1547640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29dc0cdec0 con 0x7f29dc070f20 2023-10-16T12:46:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e1547640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29dc160a90 con 0x7f29dc15c9f0 2023-10-16T12:46:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e1547640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc0cd900 con 0x7f29dc0c9230 2023-10-16T12:46:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e0d46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29dc070f20 0x7f29dc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e0d46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29dc070f20 0x7f29dc0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58690/0 (socket says 172.21.15.60:58690) 2023-10-16T12:46:26.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29e0d46640 1 -- 172.21.15.60:0/1709930905 learned_addr learned my addr 172.21.15.60:0/1709930905 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:26.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29db7fe640 1 --2- 172.21.15.60:0/1709930905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc0c9230 0x7f29dc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.154+0000 7f29dbfff640 1 --2- 172.21.15.60:0/1709930905 >> v2:172.21.15.144:3300/0 conn(0x7f29dc15c9f0 0x7f29dc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.155+0000 7f29e0d46640 1 -- 172.21.15.60:0/1709930905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc0c9230 msgr2=0x7f29dc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:26.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.155+0000 7f29e0d46640 1 --2- 172.21.15.60:0/1709930905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc0c9230 0x7f29dc0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.155+0000 7f29e0d46640 1 -- 172.21.15.60:0/1709930905 >> v2:172.21.15.144:3300/0 conn(0x7f29dc15c9f0 msgr2=0x7f29dc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.155+0000 7f29e0d46640 1 --2- 172.21.15.60:0/1709930905 >> v2:172.21.15.144:3300/0 conn(0x7f29dc15c9f0 0x7f29dc0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.155+0000 7f29e0d46640 1 -- 172.21.15.60:0/1709930905 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc0cdbe0 con 0x7f29dc070f20 2023-10-16T12:46:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.155+0000 7f29dbfff640 1 --2- 172.21.15.60:0/1709930905 >> v2:172.21.15.144:3300/0 conn(0x7f29dc15c9f0 0x7f29dc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.156+0000 7f29e0d46640 1 --2- 172.21.15.60:0/1709930905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc070f20 0x7f29dc0ac890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0ad4e0 tx=0x7f29d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5936770ca8e35a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29daffd640 1 -- 172.21.15.60:0/1709930905 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d402ca80 con 0x7f29dc070f20 2023-10-16T12:46:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29daffd640 1 -- 172.21.15.60:0/1709930905 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d402cbe0 con 0x7f29dc070f20 2023-10-16T12:46:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29daffd640 1 -- 172.21.15.60:0/1709930905 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d40343f0 con 0x7f29dc070f20 2023-10-16T12:46:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 -- 172.21.15.60:0/1709930905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc070f20 msgr2=0x7f29dc0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 --2- 172.21.15.60:0/1709930905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc070f20 0x7f29dc0ac890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0ad4e0 tx=0x7f29d402bee0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 -- 172.21.15.60:0/1709930905 shutdown_connections 2023-10-16T12:46:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 --2- 172.21.15.60:0/1709930905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc070f20 0x7f29dc0ac890 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 --2- 172.21.15.60:0/1709930905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc0c9230 0x7f29dc0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 --2- 172.21.15.60:0/1709930905 >> v2:172.21.15.144:3300/0 conn(0x7f29dc15c9f0 0x7f29dc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.157+0000 7f29e1547640 1 -- 172.21.15.60:0/1709930905 >> 172.21.15.60:0/1709930905 conn(0x7f29dc0ce6a0 msgr2=0x7f29dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.158+0000 7f29e1547640 1 -- 172.21.15.60:0/1709930905 shutdown_connections 2023-10-16T12:46:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.158+0000 7f29e1547640 1 -- 172.21.15.60:0/1709930905 wait complete. 2023-10-16T12:46:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.159+0000 7f29e1547640 1 Processor -- start 2023-10-16T12:46:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.160+0000 7f29e1547640 1 -- start start 2023-10-16T12:46:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.160+0000 7f29e1547640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc070f20 0x7f29dc149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.160+0000 7f29e1547640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29dc0c9230 0x7f29dc145de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e1547640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 0x7f29dc0c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e1547640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc0c3ac0 con 0x7f29dc15c9f0 2023-10-16T12:46:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e1547640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1496d0 con 0x7f29dc0c9230 2023-10-16T12:46:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e1547640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc0c3d40 con 0x7f29dc070f20 2023-10-16T12:46:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 0x7f29dc0c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 0x7f29dc0c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58704/0 (socket says 172.21.15.60:58704) 2023-10-16T12:46:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 -- 172.21.15.60:0/3919791691 learned_addr learned my addr 172.21.15.60:0/3919791691 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29db7fe640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29dc0c9230 0x7f29dc145de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29dbfff640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc070f20 0x7f29dc149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 -- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc070f20 msgr2=0x7f29dc149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc070f20 0x7f29dc149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 -- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29dc0c9230 msgr2=0x7f29dc145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29dc0c9230 0x7f29dc145de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29e0d46640 1 -- 172.21.15.60:0/3919791691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d4009b30 con 0x7f29dc15c9f0 2023-10-16T12:46:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.161+0000 7f29dbfff640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc070f20 0x7f29dc149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.162+0000 7f29e0d46640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 0x7f29dc0c4dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036040 tx=0x7f29d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.162+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4009510 con 0x7f29dc15c9f0 2023-10-16T12:46:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.162+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d403e450 con 0x7f29dc15c9f0 2023-10-16T12:46:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.162+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d403e7e0 con 0x7f29dc15c9f0 2023-10-16T12:46:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.162+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc0c4020 con 0x7f29dc15c9f0 2023-10-16T12:46:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.162+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc0c45e0 con 0x7f29dc15c9f0 2023-10-16T12:46:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.164+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29dc04e9b0 con 0x7f29dc15c9f0 2023-10-16T12:46:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.165+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f29d402cd50 con 0x7f29dc15c9f0 2023-10-16T12:46:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.165+0000 7f29d8ff9640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f29c806b070 0x7f29c806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.166+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f29d40ac600 con 0x7f29dc15c9f0 2023-10-16T12:46:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.166+0000 7f29dbfff640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f29c806b070 0x7f29c806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.167+0000 7f29dbfff640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f29c806b070 0x7f29c806d530 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29cc006fd0 tx=0x7f29cc008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.170+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29d4077330 con 0x7f29dc15c9f0 2023-10-16T12:46:26.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.387+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f29dc0704c0 con 0x7f29c806b070 2023-10-16T12:46:26.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.603+0000 7f29d8ff9640 1 -- 172.21.15.60:0/3919791691 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f29cc00d550 con 0x7f29c806b070 2023-10-16T12:46:26.606 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T12:46:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.606+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f29c806b070 msgr2=0x7f29c806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.606+0000 7f29e1547640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f29c806b070 0x7f29c806d530 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29cc006fd0 tx=0x7f29cc008040 comp rx=0 tx=0).stop 2023-10-16T12:46:26.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.606+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 msgr2=0x7f29dc0c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 0x7f29dc0c4dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036040 tx=0x7f29d4004a40 comp rx=0 tx=0).stop 2023-10-16T12:46:26.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 shutdown_connections 2023-10-16T12:46:26.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f29c806b070 0x7f29c806d530 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29dc15c9f0 0x7f29dc0c4dc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29dc0c9230 0x7f29dc145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 --2- 172.21.15.60:0/3919791691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29dc070f20 0x7f29dc149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 >> 172.21.15.60:0/3919791691 conn(0x7f29dc0ce6a0 msgr2=0x7f29dc0cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 shutdown_connections 2023-10-16T12:46:26.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.607+0000 7f29e1547640 1 -- 172.21.15.60:0/3919791691 wait complete. 2023-10-16T12:46:26.630 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T12:46:26.630 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-10-16T12:46:26.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.856+0000 7f4444215640 1 Processor -- start 2023-10-16T12:46:26.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.856+0000 7f4444215640 1 -- start start 2023-10-16T12:46:26.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.858+0000 7f4444215640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f443c15aee0 0x7f443c0b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.858+0000 7f4444215640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c0ae0d0 0x7f443c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.858+0000 7f4444215640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c15b2c0 con 0x7f443c0ce820 2023-10-16T12:46:26.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.858+0000 7f4444215640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c0ae5d0 con 0x7f443c15aee0 2023-10-16T12:46:26.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.858+0000 7f4444215640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0ae8b0 con 0x7f443c0ae0d0 2023-10-16T12:46:26.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.859+0000 7f4441fb1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f443c15aee0 0x7f443c0b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.859+0000 7f44417b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c0ae0d0 0x7f443c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.859+0000 7f44417b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c0ae0d0 0x7f443c0cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:42858/0 (socket says 172.21.15.60:42858) 2023-10-16T12:46:26.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.859+0000 7f44417b0640 1 -- 172.21.15.60:0/2513510251 learned_addr learned my addr 172.21.15.60:0/2513510251 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:26.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.859+0000 7f4441fb1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f443c15aee0 0x7f443c0b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:60474/0 (socket says 172.21.15.60:60474) 2023-10-16T12:46:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.860+0000 7f4441fb1640 1 -- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c0ae0d0 msgr2=0x7f443c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.860+0000 7f4441fb1640 1 --2- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c0ae0d0 0x7f443c0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.860+0000 7f4441fb1640 1 -- 172.21.15.60:0/2513510251 >> v1:172.21.15.144:6789/0 conn(0x7f443c0ce820 legacy=0x7f443c0af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.860+0000 7f4441fb1640 1 -- 172.21.15.60:0/2513510251 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0aeb90 con 0x7f443c15aee0 2023-10-16T12:46:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.861+0000 7f4441fb1640 1 --2- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c15aee0 0x7f443c0b3920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4438009790 tx=0x7f443802b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51da49235c87a952 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:26.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.862+0000 7f4440faf640 1 -- 172.21.15.60:0/2513510251 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4438031890 con 0x7f443c15aee0 2023-10-16T12:46:26.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.862+0000 7f4440faf640 1 -- 172.21.15.60:0/2513510251 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44380319f0 con 0x7f443c15aee0 2023-10-16T12:46:26.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.862+0000 7f4440faf640 1 -- 172.21.15.60:0/2513510251 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4438031d80 con 0x7f443c15aee0 2023-10-16T12:46:26.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.862+0000 7f4444215640 1 -- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c15aee0 msgr2=0x7f443c0b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.862+0000 7f4444215640 1 --2- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c15aee0 0x7f443c0b3920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4438009790 tx=0x7f443802b930 comp rx=0 tx=0).stop 2023-10-16T12:46:26.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.863+0000 7f4444215640 1 -- 172.21.15.60:0/2513510251 shutdown_connections 2023-10-16T12:46:26.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.863+0000 7f4444215640 1 --2- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c0ae0d0 0x7f443c0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.863+0000 7f4444215640 1 --2- 172.21.15.60:0/2513510251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c15aee0 0x7f443c0b3920 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.863+0000 7f4444215640 1 -- 172.21.15.60:0/2513510251 >> 172.21.15.60:0/2513510251 conn(0x7f443c0d0ad0 msgr2=0x7f443c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.863+0000 7f4444215640 1 -- 172.21.15.60:0/2513510251 shutdown_connections 2023-10-16T12:46:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.863+0000 7f4444215640 1 -- 172.21.15.60:0/2513510251 wait complete. 2023-10-16T12:46:26.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.864+0000 7f4444215640 1 Processor -- start 2023-10-16T12:46:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.865+0000 7f4444215640 1 -- start start 2023-10-16T12:46:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.865+0000 7f4444215640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0ae0d0 0x7f443c1f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.865+0000 7f4444215640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 0x7f443c144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f4444215640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c15aee0 0x7f443c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f4444215640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1f6a70 con 0x7f443c0ce820 2023-10-16T12:46:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f4444215640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0b27c0 con 0x7f443c0ae0d0 2023-10-16T12:46:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f4444215640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0b2aa0 con 0x7f443c15aee0 2023-10-16T12:46:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 0x7f443c144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 0x7f443c144fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58718/0 (socket says 172.21.15.60:58718) 2023-10-16T12:46:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 -- 172.21.15.60:0/1151401696 learned_addr learned my addr 172.21.15.60:0/1151401696 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 -- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c15aee0 msgr2=0x7f443c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44427b2640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c15aee0 0x7f443c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c15aee0 0x7f443c0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 -- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0ae0d0 msgr2=0x7f443c1f6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:26.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0ae0d0 0x7f443c1f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:26.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.866+0000 7f44417b0640 1 -- 172.21.15.60:0/1151401696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44380093a0 con 0x7f443c0ce820 2023-10-16T12:46:26.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.867+0000 7f44417b0640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 0x7f443c144fa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f442c009fd0 tx=0x7f442c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:26.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.867+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c00c940 con 0x7f443c0ce820 2023-10-16T12:46:26.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.867+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c00caa0 con 0x7f443c0ce820 2023-10-16T12:46:26.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.867+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c00ce30 con 0x7f443c0ce820 2023-10-16T12:46:26.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.867+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c0b2d80 con 0x7f443c0ce820 2023-10-16T12:46:26.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.867+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c0b3340 con 0x7f443c0ce820 2023-10-16T12:46:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.869+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c04e160 con 0x7f443c0ce820 2023-10-16T12:46:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.869+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f442c013470 con 0x7f443c0ce820 2023-10-16T12:46:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.870+0000 7f44327fc640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f442006b0a0 0x7f442006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.871+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f442c089bd0 con 0x7f443c0ce820 2023-10-16T12:46:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.871+0000 7f4441fb1640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f442006b0a0 0x7f442006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.872+0000 7f4441fb1640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f442006b0a0 0x7f442006d560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4438036000 tx=0x7f4438030000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:26.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:26.875+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f442c0548f0 con 0x7f443c0ce820 2023-10-16T12:46:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.099+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f443c0af150 con 0x7f442006b0a0 2023-10-16T12:46:27.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.304+0000 7f44327fc640 1 -- 172.21.15.60:0/1151401696 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4438031b60 con 0x7f442006b0a0 2023-10-16T12:46:27.307 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T12:46:27.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.307+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f442006b0a0 msgr2=0x7f442006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.307+0000 7f4444215640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f442006b0a0 0x7f442006d560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4438036000 tx=0x7f4438030000 comp rx=0 tx=0).stop 2023-10-16T12:46:27.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.307+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 msgr2=0x7f443c144fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.307+0000 7f4444215640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 0x7f443c144fa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f442c009fd0 tx=0x7f442c00ed40 comp rx=0 tx=0).stop 2023-10-16T12:46:27.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 shutdown_connections 2023-10-16T12:46:27.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f442006b0a0 0x7f442006d560 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f443c15aee0 0x7f443c0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f443c0ce820 0x7f443c144fa0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 --2- 172.21.15.60:0/1151401696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0ae0d0 0x7f443c1f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 >> 172.21.15.60:0/1151401696 conn(0x7f443c0d0ad0 msgr2=0x7f443c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:27.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 shutdown_connections 2023-10-16T12:46:27.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.308+0000 7f4444215640 1 -- 172.21.15.60:0/1151401696 wait complete. 2023-10-16T12:46:27.330 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T12:46:27.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-10-16T12:46:27.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.554+0000 7ff2cf0c2640 1 Processor -- start 2023-10-16T12:46:27.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.554+0000 7ff2cf0c2640 1 -- start start 2023-10-16T12:46:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2cf0c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2c80ccb00 0x7ff2c80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2cf0c2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c80cd4a0 con 0x7ff2c8153740 2023-10-16T12:46:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2cf0c2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c815ac60 con 0x7ff2c80ad4d0 2023-10-16T12:46:27.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2cf0c2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c8070030 con 0x7ff2c80ccb00 2023-10-16T12:46:27.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2ce0c0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff2c8153740 0x7ff2c815a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49510/0 (socket says 172.21.15.60:49510) 2023-10-16T12:46:27.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2ce0c0640 1 -- 172.21.15.60:0/897583658 learned_addr learned my addr 172.21.15.60:0/897583658 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2cd8bf640 1 --2- 172.21.15.60:0/897583658 >> v2:172.21.15.60:3300/0 conn(0x7ff2c80ccb00 0x7ff2c80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.556+0000 7ff2cd0be640 1 -- 172.21.15.60:0/897583658 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971990240 0 0) 0x7ff2c80cd4a0 con 0x7ff2c8153740 2023-10-16T12:46:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.557+0000 7ff2cd0be640 1 -- 172.21.15.60:0/897583658 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b4005730 con 0x7ff2c8153740 2023-10-16T12:46:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.557+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/897583658 >> v1:172.21.15.144:6789/0 conn(0x7ff2c80ad4d0 legacy=0x7ff2c80ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.557+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/897583658 >> v1:172.21.15.60:6789/0 conn(0x7ff2c8153740 legacy=0x7ff2c815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.557+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/897583658 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8070310 con 0x7ff2c80ccb00 2023-10-16T12:46:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.558+0000 7ff2cd8bf640 1 --2- 172.21.15.60:0/897583658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80ccf60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80097a0 tx=0x7ff2b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad02f61a86cd71d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.558+0000 7ff2cd0be640 1 -- 172.21.15.60:0/897583658 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8031830 con 0x7ff2c80ccb00 2023-10-16T12:46:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.558+0000 7ff2cd0be640 1 -- 172.21.15.60:0/897583658 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b8031990 con 0x7ff2c80ccb00 2023-10-16T12:46:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.558+0000 7ff2cd0be640 1 -- 172.21.15.60:0/897583658 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8031d20 con 0x7ff2c80ccb00 2023-10-16T12:46:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/897583658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 msgr2=0x7ff2c80ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/897583658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80ccf60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80097a0 tx=0x7ff2b802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/897583658 shutdown_connections 2023-10-16T12:46:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/897583658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80ccf60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/897583658 >> 172.21.15.60:0/897583658 conn(0x7ff2c80ce6a0 msgr2=0x7ff2c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/897583658 shutdown_connections 2023-10-16T12:46:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.559+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/897583658 wait complete. 2023-10-16T12:46:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.561+0000 7ff2cf0c2640 1 Processor -- start 2023-10-16T12:46:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.561+0000 7ff2cf0c2640 1 -- start start 2023-10-16T12:46:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.562+0000 7ff2cf0c2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2c80ad4d0 0x7ff2c8149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.562+0000 7ff2cf0c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.562+0000 7ff2cf0c2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2c8153740 0x7ff2c80aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.562+0000 7ff2cf0c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81562f0 con 0x7ff2c80ccb00 2023-10-16T12:46:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.562+0000 7ff2cf0c2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c814a3c0 con 0x7ff2c8153740 2023-10-16T12:46:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.562+0000 7ff2cf0c2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8155250 con 0x7ff2c80ad4d0 2023-10-16T12:46:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80b2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58740/0 (socket says 172.21.15.60:58740) 2023-10-16T12:46:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/1693136176 learned_addr learned my addr 172.21.15.60:0/1693136176 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2ce0c0640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2c80ad4d0 0x7ff2c8149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2ce8c1640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2c8153740 0x7ff2c80aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2c80ad4d0 msgr2=0x7ff2c8149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2c80ad4d0 0x7ff2c8149e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2c8153740 msgr2=0x7ff2c80aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2c8153740 0x7ff2c80aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2cd8bf640 1 -- 172.21.15.60:0/1693136176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b8009400 con 0x7ff2c80ccb00 2023-10-16T12:46:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.563+0000 7ff2ce8c1640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2c8153740 0x7ff2c80aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.564+0000 7ff2cd8bf640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80b2120 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff2b802bee0 tx=0x7ff2b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.564+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b803a6f0 con 0x7ff2c80ccb00 2023-10-16T12:46:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.564+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b803a870 con 0x7ff2c80ccb00 2023-10-16T12:46:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.564+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b803ac20 con 0x7ff2c80ccb00 2023-10-16T12:46:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.564+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c8155530 con 0x7ff2c80ccb00 2023-10-16T12:46:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.564+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c8155af0 con 0x7ff2c80ccb00 2023-10-16T12:46:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.565+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c806fc30 con 0x7ff2c80ccb00 2023-10-16T12:46:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.567+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff2b804c3b0 con 0x7ff2c80ccb00 2023-10-16T12:46:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.567+0000 7ff2beffd640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff2ac06b070 0x7ff2ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.568+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff2b80ab960 con 0x7ff2c80ccb00 2023-10-16T12:46:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.568+0000 7ff2ce0c0640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff2ac06b070 0x7ff2ac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.569+0000 7ff2ce0c0640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff2ac06b070 0x7ff2ac06d530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40011e0 tx=0x7ff2c4009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:27.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.572+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2b80764e0 con 0x7ff2c80ccb00 2023-10-16T12:46:27.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:27.794+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2c80708d0 con 0x7ff2ac06b070 2023-10-16T12:46:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.012+0000 7ff2beffd640 1 -- 172.21.15.60:0/1693136176 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff2c4001330 con 0x7ff2ac06b070 2023-10-16T12:46:28.015 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T12:46:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff2ac06b070 msgr2=0x7ff2ac06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff2ac06b070 0x7ff2ac06d530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40011e0 tx=0x7ff2c4009040 comp rx=0 tx=0).stop 2023-10-16T12:46:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 msgr2=0x7ff2c80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80b2120 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff2b802bee0 tx=0x7ff2b80046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 shutdown_connections 2023-10-16T12:46:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff2ac06b070 0x7ff2ac06d530 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2c8153740 0x7ff2c80aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2c80ccb00 0x7ff2c80b2120 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 --2- 172.21.15.60:0/1693136176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2c80ad4d0 0x7ff2c8149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 >> 172.21.15.60:0/1693136176 conn(0x7ff2c80ce6a0 msgr2=0x7ff2c815c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 shutdown_connections 2023-10-16T12:46:28.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.016+0000 7ff2cf0c2640 1 -- 172.21.15.60:0/1693136176 wait complete. 2023-10-16T12:46:28.044 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T12:46:28.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-10-16T12:46:28.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.272+0000 7f0132f4e640 1 Processor -- start 2023-10-16T12:46:28.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.273+0000 7f0132f4e640 1 -- start start 2023-10-16T12:46:28.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.274+0000 7f0132f4e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f012c06c040 0x7f012c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:28.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.274+0000 7f0132f4e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f012c0cda80 0x7f012c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:28.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.274+0000 7f0132f4e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c06c960 con 0x7f012c0ad970 2023-10-16T12:46:28.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.274+0000 7f0132f4e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c15b9d0 con 0x7f012c06c040 2023-10-16T12:46:28.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.274+0000 7f0132f4e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c15bcb0 con 0x7f012c0cda80 2023-10-16T12:46:28.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f012c06c040 0x7f012c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:28.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f012c06c040 0x7f012c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58742/0 (socket says 172.21.15.60:58742) 2023-10-16T12:46:28.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 -- 172.21.15.60:0/3465481749 learned_addr learned my addr 172.21.15.60:0/3465481749 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:28.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f013274d640 1 --2- 172.21.15.60:0/3465481749 >> v2:172.21.15.144:3300/0 conn(0x7f012c0cda80 0x7f012c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:28.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 -- 172.21.15.60:0/3465481749 >> v2:172.21.15.144:3300/0 conn(0x7f012c0cda80 msgr2=0x7f012c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:28.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 --2- 172.21.15.60:0/3465481749 >> v2:172.21.15.144:3300/0 conn(0x7f012c0cda80 0x7f012c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 -- 172.21.15.60:0/3465481749 >> v1:172.21.15.144:6789/0 conn(0x7f012c0ad970 legacy=0x7f012c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:28.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f0131f4c640 1 -- 172.21.15.60:0/3465481749 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c15bf90 con 0x7f012c06c040 2023-10-16T12:46:28.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.275+0000 7f013274d640 1 --2- 172.21.15.60:0/3465481749 >> v2:172.21.15.144:3300/0 conn(0x7f012c0cda80 0x7f012c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:28.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.276+0000 7f0131f4c640 1 --2- 172.21.15.60:0/3465481749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c06c420 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f012c0cf300 tx=0x7f012802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bded77a36ae6790 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:28.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.277+0000 7f0130f4a640 1 -- 172.21.15.60:0/3465481749 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128031870 con 0x7f012c06c040 2023-10-16T12:46:28.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.277+0000 7f0130f4a640 1 -- 172.21.15.60:0/3465481749 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01280319d0 con 0x7f012c06c040 2023-10-16T12:46:28.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.277+0000 7f0130f4a640 1 -- 172.21.15.60:0/3465481749 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128031d60 con 0x7f012c06c040 2023-10-16T12:46:28.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 -- 172.21.15.60:0/3465481749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 msgr2=0x7f012c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3465481749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c06c420 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f012c0cf300 tx=0x7f012802b910 comp rx=0 tx=0).stop 2023-10-16T12:46:28.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 -- 172.21.15.60:0/3465481749 shutdown_connections 2023-10-16T12:46:28.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3465481749 >> v2:172.21.15.144:3300/0 conn(0x7f012c0cda80 0x7f012c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3465481749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c06c420 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 -- 172.21.15.60:0/3465481749 >> 172.21.15.60:0/3465481749 conn(0x7f012c0ce6a0 msgr2=0x7f012c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:28.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 -- 172.21.15.60:0/3465481749 shutdown_connections 2023-10-16T12:46:28.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.278+0000 7f0132f4e640 1 -- 172.21.15.60:0/3465481749 wait complete. 2023-10-16T12:46:28.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.280+0000 7f0132f4e640 1 Processor -- start 2023-10-16T12:46:28.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.280+0000 7f0132f4e640 1 -- start start 2023-10-16T12:46:28.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0132f4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c0ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:28.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0132f4e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f012c0ad970 0x7f012c14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:28.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0132f4e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f012c0cda80 0x7f012c1e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:28.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0132f4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0c9d30 con 0x7f012c06c040 2023-10-16T12:46:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0132f4e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c1e5290 con 0x7f012c0cda80 2023-10-16T12:46:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0132f4e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0c9700 con 0x7f012c0ad970 2023-10-16T12:46:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0131f4c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c0ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0131f4c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c0ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58752/0 (socket says 172.21.15.60:58752) 2023-10-16T12:46:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0131f4c640 1 -- 172.21.15.60:0/3027017743 learned_addr learned my addr 172.21.15.60:0/3027017743 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0131f4c640 1 -- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f012c0ad970 msgr2=0x7f012c14dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f013174b640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f012c0ad970 0x7f012c14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f013274d640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f012c0cda80 0x7f012c1e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0131f4c640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f012c0ad970 0x7f012c14dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.282+0000 7f0131f4c640 1 -- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f012c0cda80 msgr2=0x7f012c1e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f0131f4c640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f012c0cda80 0x7f012c1e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f0131f4c640 1 -- 172.21.15.60:0/3027017743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01280094d0 con 0x7f012c06c040 2023-10-16T12:46:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f013174b640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f012c0ad970 0x7f012c14dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f0131f4c640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c0ae100 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f012802b950 tx=0x7f01280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012803ad10 con 0x7f012c06c040 2023-10-16T12:46:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012803ae70 con 0x7f012c06c040 2023-10-16T12:46:28.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c1e4cd0 con 0x7f012c06c040 2023-10-16T12:46:28.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.283+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128044610 con 0x7f012c06c040 2023-10-16T12:46:28.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.284+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c0c9950 con 0x7f012c06c040 2023-10-16T12:46:28.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.285+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c06b330 con 0x7f012c06c040 2023-10-16T12:46:28.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.286+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f0128044770 con 0x7f012c06c040 2023-10-16T12:46:28.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.287+0000 7f01227fc640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f011006b070 0x7f011006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:28.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.287+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f012802d040 con 0x7f012c06c040 2023-10-16T12:46:28.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.288+0000 7f013174b640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f011006b070 0x7f011006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:28.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.289+0000 7f013174b640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f011006b070 0x7f011006d530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f011c007fd0 tx=0x7f011c009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:28.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.292+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0128077080 con 0x7f012c06c040 2023-10-16T12:46:28.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.514+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f012c0cc550 con 0x7f011006b070 2023-10-16T12:46:28.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.729+0000 7f01227fc640 1 -- 172.21.15.60:0/3027017743 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f011c0012e0 con 0x7f011006b070 2023-10-16T12:46:28.732 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T12:46:28.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f011006b070 msgr2=0x7f011006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f011006b070 0x7f011006d530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f011c007fd0 tx=0x7f011c009040 comp rx=0 tx=0).stop 2023-10-16T12:46:28.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 msgr2=0x7f012c0ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:28.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c0ae100 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f012802b950 tx=0x7f01280046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 shutdown_connections 2023-10-16T12:46:28.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f011006b070 0x7f011006d530 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f012c0ad970 0x7f012c14dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f012c0cda80 0x7f012c1e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 --2- 172.21.15.60:0/3027017743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f012c06c040 0x7f012c0ae100 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:28.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 >> 172.21.15.60:0/3027017743 conn(0x7f012c0ce6a0 msgr2=0x7f012c0b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:28.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 shutdown_connections 2023-10-16T12:46:28.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:28.733+0000 7f0132f4e640 1 -- 172.21.15.60:0/3027017743 wait complete. 2023-10-16T12:46:28.760 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T12:46:28.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-10-16T12:46:29.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.000+0000 7f567359e640 1 Processor -- start 2023-10-16T12:46:29.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.000+0000 7f567359e640 1 -- start start 2023-10-16T12:46:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.001+0000 7f567359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f56740cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.001+0000 7f567359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f56740cd800 0x7f56740cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f567359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56740cafd0 con 0x7f5674161b60 2023-10-16T12:46:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f567359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56740cbc50 con 0x7f56740cd800 2023-10-16T12:46:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f567359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740ce1d0 con 0x7f5674071b90 2023-10-16T12:46:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f567259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5674161b60 0x7f5674161f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54774/0 (socket says 172.21.15.60:54774) 2023-10-16T12:46:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f567259c640 1 -- 172.21.15.60:0/1996057625 learned_addr learned my addr 172.21.15.60:0/1996057625 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f5672d9d640 1 --2- 172.21.15.60:0/1996057625 >> v2:172.21.15.144:3300/0 conn(0x7f56740cd800 0x7f56740cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.002+0000 7f5671d9b640 1 --2- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f56740cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f567159a640 1 -- 172.21.15.60:0/1996057625 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36431899 0 0) 0x7f56740cafd0 con 0x7f5674161b60 2023-10-16T12:46:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f567159a640 1 -- 172.21.15.60:0/1996057625 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5660003520 con 0x7f5674161b60 2023-10-16T12:46:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f567159a640 1 -- 172.21.15.60:0/1996057625 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 330763461 0 0) 0x7f5660003520 con 0x7f5674161b60 2023-10-16T12:46:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f5672d9d640 1 -- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 msgr2=0x7f56740cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f5672d9d640 1 --2- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f56740cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f5672d9d640 1 -- 172.21.15.60:0/1996057625 >> v1:172.21.15.60:6789/0 conn(0x7f5674161b60 legacy=0x7f5674161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.003+0000 7f5672d9d640 1 -- 172.21.15.60:0/1996057625 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740cb2b0 con 0x7f56740cd800 2023-10-16T12:46:29.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.004+0000 7f5672d9d640 1 --2- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56740cd800 0x7f56740cdc90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f566c0097a0 tx=0x7f566c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=decb91c91275b5c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:29.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567159a640 1 -- 172.21.15.60:0/1996057625 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640019f0 con 0x7f56740cd800 2023-10-16T12:46:29.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567159a640 1 -- 172.21.15.60:0/1996057625 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c02e050 con 0x7f56740cd800 2023-10-16T12:46:29.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567159a640 1 -- 172.21.15.60:0/1996057625 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c030730 con 0x7f56740cd800 2023-10-16T12:46:29.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567359e640 1 -- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56740cd800 msgr2=0x7f56740cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567359e640 1 --2- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56740cd800 0x7f56740cdc90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f566c0097a0 tx=0x7f566c02b790 comp rx=0 tx=0).stop 2023-10-16T12:46:29.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567359e640 1 -- 172.21.15.60:0/1996057625 shutdown_connections 2023-10-16T12:46:29.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567359e640 1 --2- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56740cd800 0x7f56740cdc90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567359e640 1 --2- 172.21.15.60:0/1996057625 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f56740cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.005+0000 7f567359e640 1 -- 172.21.15.60:0/1996057625 >> 172.21.15.60:0/1996057625 conn(0x7f5674065ed0 msgr2=0x7f56740662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:29.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.006+0000 7f567359e640 1 -- 172.21.15.60:0/1996057625 shutdown_connections 2023-10-16T12:46:29.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.006+0000 7f567359e640 1 -- 172.21.15.60:0/1996057625 wait complete. 2023-10-16T12:46:29.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.007+0000 7f567359e640 1 Processor -- start 2023-10-16T12:46:29.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 -- start start 2023-10-16T12:46:29.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f5674160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 0x7f56740c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5674161b60 0x7f56741ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5674160d40 con 0x7f56740cd800 2023-10-16T12:46:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56740c4270 con 0x7f5674161b60 2023-10-16T12:46:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.008+0000 7f567359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740c4550 con 0x7f5674071b90 2023-10-16T12:46:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 0x7f56740c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 0x7f56740c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42240/0 (socket says 172.21.15.60:42240) 2023-10-16T12:46:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 -- 172.21.15.60:0/3266776117 learned_addr learned my addr 172.21.15.60:0/3266776117 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5672d9d640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5674161b60 0x7f56741ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f567259c640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f5674160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 -- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 msgr2=0x7f5674160800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f5674160800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 -- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5674161b60 msgr2=0x7f56741ea9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5674161b60 0x7f56741ea9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.009+0000 7f5671d9b640 1 -- 172.21.15.60:0/3266776117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c009400 con 0x7f56740cd800 2023-10-16T12:46:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.010+0000 7f5671d9b640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 0x7f56740c9850 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f565c00e9b0 tx=0x7f565c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.010+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c009670 con 0x7f56740cd800 2023-10-16T12:46:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.010+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c00c9a0 con 0x7f56740cd800 2023-10-16T12:46:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.010+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c00cd50 con 0x7f56740cd800 2023-10-16T12:46:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.010+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56740c4830 con 0x7f56740cd800 2023-10-16T12:46:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.010+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56741e75c0 con 0x7f56740cd800 2023-10-16T12:46:29.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.011+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567404e9b0 con 0x7f56740cd800 2023-10-16T12:46:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.012+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f565c013470 con 0x7f56740cd800 2023-10-16T12:46:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.013+0000 7f566affd640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f565406b070 0x7f565406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.013+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f565c089d50 con 0x7f56740cd800 2023-10-16T12:46:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.014+0000 7f567259c640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f565406b070 0x7f565406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.015+0000 7f567259c640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f565406b070 0x7f565406d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5664001e70 tx=0x7f5664009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:29.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.017+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f565c0549e0 con 0x7f56740cd800 2023-10-16T12:46:29.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.243+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f56740cb870 con 0x7f565406b070 2023-10-16T12:46:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.453+0000 7f566affd640 1 -- 172.21.15.60:0/3266776117 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f566400ee10 con 0x7f565406b070 2023-10-16T12:46:29.456 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T12:46:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f565406b070 msgr2=0x7f565406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f565406b070 0x7f565406d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5664001e70 tx=0x7f5664009040 comp rx=0 tx=0).stop 2023-10-16T12:46:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 msgr2=0x7f56740c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 0x7f56740c9850 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f565c00e9b0 tx=0x7f565c00ee70 comp rx=0 tx=0).stop 2023-10-16T12:46:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 shutdown_connections 2023-10-16T12:46:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f565406b070 0x7f565406d530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5674161b60 0x7f56741ea9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56740cd800 0x7f56740c9850 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 --2- 172.21.15.60:0/3266776117 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5674071b90 0x7f5674160800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 >> 172.21.15.60:0/3266776117 conn(0x7f5674065ed0 msgr2=0x7f567406aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 shutdown_connections 2023-10-16T12:46:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.457+0000 7f567359e640 1 -- 172.21.15.60:0/3266776117 wait complete. 2023-10-16T12:46:29.481 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T12:46:29.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:29.482 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T12:46:29.544 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_35_00_UTC 2023-10-16T12:46:29.544 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:29.544 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:29.544 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:29.545 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:29.545 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:29.545 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:29.545 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:29.545 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:29.546 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:29.546 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:29.546 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:29.547 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_35_00_UTC', 'scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_01_UTC', 'scheduled-2023-10-16-12_38_01_UTC', 'scheduled-2023-10-16-12_39_01_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_01_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:29.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-10-16T12:46:29.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.777+0000 7f2914d33640 1 Processor -- start 2023-10-16T12:46:29.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.778+0000 7f2914d33640 1 -- start start 2023-10-16T12:46:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.779+0000 7f2914d33640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f291015ad70 0x7f2910069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.779+0000 7f2914d33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 0x7f291006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.779+0000 7f2914d33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29100cb030 con 0x7f29100c9ca0 2023-10-16T12:46:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.779+0000 7f2914d33640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29100c91c0 con 0x7f291015ad70 2023-10-16T12:46:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.779+0000 7f2914d33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c94a0 con 0x7f29100caac0 2023-10-16T12:46:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.780+0000 7f290dd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f29100c9ca0 0x7f29100ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54782/0 (socket says 172.21.15.60:54782) 2023-10-16T12:46:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.780+0000 7f290dd9b640 1 -- 172.21.15.60:0/1054436650 learned_addr learned my addr 172.21.15.60:0/1054436650 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.780+0000 7f290ed9d640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 0x7f291006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.780+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358379791 0 0) 0x7f29100cb030 con 0x7f29100c9ca0 2023-10-16T12:46:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.780+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f8003520 con 0x7f29100c9ca0 2023-10-16T12:46:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.780+0000 7f290e59c640 1 --2- 172.21.15.60:0/1054436650 >> v2:172.21.15.144:3300/0 conn(0x7f291015ad70 0x7f2910069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994162797 0 0) 0x7f28f8003520 con 0x7f29100c9ca0 2023-10-16T12:46:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 msgr2=0x7f291006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 0x7f291006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 >> v2:172.21.15.144:3300/0 conn(0x7f291015ad70 msgr2=0x7f2910069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 --2- 172.21.15.60:0/1054436650 >> v2:172.21.15.144:3300/0 conn(0x7f291015ad70 0x7f2910069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29100c9780 con 0x7f29100c9ca0 2023-10-16T12:46:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2904001670 con 0x7f29100c9ca0 2023-10-16T12:46:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 >> v1:172.21.15.60:6789/0 conn(0x7f29100c9ca0 legacy=0x7f29100ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 0x7f28f8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f28f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28f8004d00 0x7f28f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c91c0 con 0x7f28f8003ef0 2023-10-16T12:46:29.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c94a0 con 0x7f28f8004d00 2023-10-16T12:46:29.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f800b980 con 0x7f29100caac0 2023-10-16T12:46:29.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.781+0000 7f290dd9b640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f28f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290e59c640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 0x7f28f8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290ed9d640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28f8004d00 0x7f28f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290dd9b640 1 -- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 msgr2=0x7f28f8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290dd9b640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29100caac0 0x7f28f8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290dd9b640 1 -- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28f8004d00 msgr2=0x7f28f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290dd9b640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28f8004d00 0x7f28f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.782+0000 7f290dd9b640 1 -- 172.21.15.60:0/1054436650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8003520 con 0x7f28f8003ef0 2023-10-16T12:46:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.783+0000 7f290dd9b640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f28f80047c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f290400c210 tx=0x7f28f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7302a2ab9834c97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.783+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904005ad0 con 0x7f28f8003ef0 2023-10-16T12:46:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.783+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2904005c30 con 0x7f28f8003ef0 2023-10-16T12:46:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.783+0000 7f290d59a640 1 -- 172.21.15.60:0/1054436650 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290402c8c0 con 0x7f28f8003ef0 2023-10-16T12:46:29.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f2914d33640 1 -- 172.21.15.60:0/1054436650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 msgr2=0x7f28f80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f2914d33640 1 --2- 172.21.15.60:0/1054436650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f28f80047c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f290400c210 tx=0x7f28f80057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f290ed9d640 1 -- 172.21.15.60:0/1054436650 reap_dead start 2023-10-16T12:46:29.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f2914d33640 1 -- 172.21.15.60:0/1054436650 shutdown_connections 2023-10-16T12:46:29.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f2914d33640 1 -- 172.21.15.60:0/1054436650 >> 172.21.15.60:0/1054436650 conn(0x7f29100ce9b0 msgr2=0x7f29100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:29.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f2914d33640 1 -- 172.21.15.60:0/1054436650 shutdown_connections 2023-10-16T12:46:29.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.784+0000 7f2914d33640 1 -- 172.21.15.60:0/1054436650 wait complete. 2023-10-16T12:46:29.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.786+0000 7f2914d33640 1 Processor -- start 2023-10-16T12:46:29.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.786+0000 7f2914d33640 1 -- start start 2023-10-16T12:46:29.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f2914d33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f291015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f2914d33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291015b450 0x7f2910155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f2914d33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f291006b030 0x7f291006b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f2914d33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c5af0 con 0x7f28f8003ef0 2023-10-16T12:46:29.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f2914d33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291006b9f0 con 0x7f291006b030 2023-10-16T12:46:29.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f2914d33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2910155fb0 con 0x7f291015b450 2023-10-16T12:46:29.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f290e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f291015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f290e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f291015d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42244/0 (socket says 172.21.15.60:42244) 2023-10-16T12:46:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.787+0000 7f290e59c640 1 -- 172.21.15.60:0/3251022129 learned_addr learned my addr 172.21.15.60:0/3251022129 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f290e59c640 1 -- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291015b450 msgr2=0x7f2910155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f290e59c640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291015b450 0x7f2910155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f290e59c640 1 -- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f291006b030 msgr2=0x7f291006b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f290e59c640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f291006b030 0x7f291006b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f290e59c640 1 -- 172.21.15.60:0/3251022129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291015b8d0 con 0x7f28f8003ef0 2023-10-16T12:46:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f290e59c640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f291015d1b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2900011600 tx=0x7f2900011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900015870 con 0x7f28f8003ef0 2023-10-16T12:46:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.788+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29000159d0 con 0x7f28f8003ef0 2023-10-16T12:46:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.789+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291015bba0 con 0x7f28f8003ef0 2023-10-16T12:46:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.789+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900015d40 con 0x7f28f8003ef0 2023-10-16T12:46:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.789+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291015c160 con 0x7f28f8003ef0 2023-10-16T12:46:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.790+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291015d6f0 con 0x7f28f8003ef0 2023-10-16T12:46:29.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.791+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2900013430 con 0x7f28f8003ef0 2023-10-16T12:46:29.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.792+0000 7f28feffd640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f28f006b070 0x7f28f006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:29.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.792+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f290008a250 con 0x7f28f8003ef0 2023-10-16T12:46:29.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.793+0000 7f290dd9b640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f28f006b070 0x7f28f006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:29.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.794+0000 7f290dd9b640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f28f006b070 0x7f28f006d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2904034000 tx=0x7f290402e000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:29.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:29.797+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2900054f70 con 0x7f28f8003ef0 2023-10-16T12:46:30.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.016+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f291006a2e0 con 0x7f28f006b070 2023-10-16T12:46:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.036+0000 7f28feffd640 1 -- 172.21.15.60:0/3251022129 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2904005970 con 0x7f28f006b070 2023-10-16T12:46:30.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:30.039 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:34:58", "first": "2023-10-16T12:35:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T12:46:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f28f006b070 msgr2=0x7f28f006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f28f006b070 0x7f28f006d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2904034000 tx=0x7f290402e000 comp rx=0 tx=0).stop 2023-10-16T12:46:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 msgr2=0x7f291015d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f291015d1b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2900011600 tx=0x7f2900011ac0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 shutdown_connections 2023-10-16T12:46:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f28f006b070 0x7f28f006d530 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f291015b450 0x7f2910155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f291006b030 0x7f291006b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 --2- 172.21.15.60:0/3251022129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28f8003ef0 0x7f291015d1b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 >> 172.21.15.60:0/3251022129 conn(0x7f29100ce9b0 msgr2=0x7f291015a040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 shutdown_connections 2023-10-16T12:46:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.039+0000 7f2914d33640 1 -- 172.21.15.60:0/3251022129 wait complete. 2023-10-16T12:46:30.060 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:34:58", "first": "2023-10-16T12:35:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T12:46:30.060 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:34:58', 'first': '2023-10-16T12:35:00', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:30.061 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T12:46:30.061 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T12:46:30.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:30.062 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T12:46:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_35_00_UTC 2023-10-16T12:46:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:30.124 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_35_00_UTC', 'scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_01_UTC', 'scheduled-2023-10-16-12_38_01_UTC', 'scheduled-2023-10-16-12_39_00_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_01_UTC', 'scheduled-2023-10-16-12_44_01_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:30.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-10-16T12:46:30.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.347+0000 7f851a8ad640 1 Processor -- start 2023-10-16T12:46:30.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.348+0000 7f851a8ad640 1 -- start start 2023-10-16T12:46:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.349+0000 7f851a8ad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851406c040 0x7f851406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.349+0000 7f851a8ad640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f85140ad970 0x7f851415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.349+0000 7f851a8ad640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851406c960 con 0x7f851415ebc0 2023-10-16T12:46:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.349+0000 7f851a8ad640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851415b9d0 con 0x7f85140ad970 2023-10-16T12:46:30.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.349+0000 7f851a8ad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851415bcb0 con 0x7f851406c040 2023-10-16T12:46:30.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85190aa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f85140ad970 0x7f851415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85190aa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f85140ad970 0x7f851415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42254/0 (socket says 172.21.15.60:42254) 2023-10-16T12:46:30.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85190aa640 1 -- 172.21.15.60:0/2351362915 learned_addr learned my addr 172.21.15.60:0/2351362915 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85198ab640 1 --2- 172.21.15.60:0/2351362915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851406c040 0x7f851406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85190aa640 1 -- 172.21.15.60:0/2351362915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851406c040 msgr2=0x7f851406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85190aa640 1 --2- 172.21.15.60:0/2351362915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851406c040 0x7f851406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.350+0000 7f85190aa640 1 -- 172.21.15.60:0/2351362915 >> v1:172.21.15.144:6789/0 conn(0x7f851415ebc0 legacy=0x7f85140cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.351+0000 7f85190aa640 1 -- 172.21.15.60:0/2351362915 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851415bf90 con 0x7f85140ad970 2023-10-16T12:46:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.352+0000 7f85190aa640 1 --2- 172.21.15.60:0/2351362915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85140ad970 0x7f851415e680 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f85040098a0 tx=0x7f850402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2796f79532557bfa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.352+0000 7f85188a9640 1 -- 172.21.15.60:0/2351362915 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504030840 con 0x7f85140ad970 2023-10-16T12:46:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.352+0000 7f85188a9640 1 -- 172.21.15.60:0/2351362915 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85040309a0 con 0x7f85140ad970 2023-10-16T12:46:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.352+0000 7f85188a9640 1 -- 172.21.15.60:0/2351362915 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504030d30 con 0x7f85140ad970 2023-10-16T12:46:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 -- 172.21.15.60:0/2351362915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85140ad970 msgr2=0x7f851415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 --2- 172.21.15.60:0/2351362915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85140ad970 0x7f851415e680 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f85040098a0 tx=0x7f850402b980 comp rx=0 tx=0).stop 2023-10-16T12:46:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 -- 172.21.15.60:0/2351362915 shutdown_connections 2023-10-16T12:46:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 --2- 172.21.15.60:0/2351362915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f85140ad970 0x7f851415e680 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 --2- 172.21.15.60:0/2351362915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851406c040 0x7f851406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 -- 172.21.15.60:0/2351362915 >> 172.21.15.60:0/2351362915 conn(0x7f85140ce6a0 msgr2=0x7f85140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 -- 172.21.15.60:0/2351362915 shutdown_connections 2023-10-16T12:46:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.353+0000 7f851a8ad640 1 -- 172.21.15.60:0/2351362915 wait complete. 2023-10-16T12:46:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.355+0000 7f851a8ad640 1 Processor -- start 2023-10-16T12:46:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.355+0000 7f851a8ad640 1 -- start start 2023-10-16T12:46:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f851a8ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 0x7f8514157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f851a8ad640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85140ad970 0x7f85140c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f851a8ad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851415ebc0 0x7f85140c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f851a8ad640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140c6b70 con 0x7f851406c040 2023-10-16T12:46:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f851a8ad640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85141f25e0 con 0x7f85140ad970 2023-10-16T12:46:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f851a8ad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140aed10 con 0x7f851415ebc0 2023-10-16T12:46:30.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f85198ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 0x7f8514157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f85198ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 0x7f8514157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42262/0 (socket says 172.21.15.60:42262) 2023-10-16T12:46:30.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.356+0000 7f85198ab640 1 -- 172.21.15.60:0/3821819885 learned_addr learned my addr 172.21.15.60:0/3821819885 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:30.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85190aa640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85140ad970 0x7f85140c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f851a0ac640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851415ebc0 0x7f85140c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85198ab640 1 -- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851415ebc0 msgr2=0x7f85140c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85198ab640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851415ebc0 0x7f85140c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85198ab640 1 -- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85140ad970 msgr2=0x7f85140c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85198ab640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85140ad970 0x7f85140c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85198ab640 1 -- 172.21.15.60:0/3821819885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85040094d0 con 0x7f851406c040 2023-10-16T12:46:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85190aa640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85140ad970 0x7f85140c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f85198ab640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 0x7f8514157910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8510009d10 tx=0x7f851000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8510009b80 con 0x7f851406c040 2023-10-16T12:46:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851000f040 con 0x7f851406c040 2023-10-16T12:46:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8510014680 con 0x7f851406c040 2023-10-16T12:46:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.357+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85140aeff0 con 0x7f851406c040 2023-10-16T12:46:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.358+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85140af5b0 con 0x7f851406c040 2023-10-16T12:46:30.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.359+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851406b330 con 0x7f851406c040 2023-10-16T12:46:30.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.360+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f85100147e0 con 0x7f851406c040 2023-10-16T12:46:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.361+0000 7f850a7fc640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84f806b070 0x7f84f806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.361+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f851008a6d0 con 0x7f851406c040 2023-10-16T12:46:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.361+0000 7f85190aa640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84f806b070 0x7f84f806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.363+0000 7f85190aa640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84f806b070 0x7f84f806d530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8504036000 tx=0x7f850403b040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.365+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85100552e0 con 0x7f851406c040 2023-10-16T12:46:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.582+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85141f2470 con 0x7f84f806b070 2023-10-16T12:46:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.600+0000 7f850a7fc640 1 -- 172.21.15.60:0/3821819885 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f85141f2470 con 0x7f84f806b070 2023-10-16T12:46:30.602 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:30.602 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:34:59", "first": "2023-10-16T12:35:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T12:46:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84f806b070 msgr2=0x7f84f806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84f806b070 0x7f84f806d530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8504036000 tx=0x7f850403b040 comp rx=0 tx=0).stop 2023-10-16T12:46:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 msgr2=0x7f8514157910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 0x7f8514157910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8510009d10 tx=0x7f851000c490 comp rx=0 tx=0).stop 2023-10-16T12:46:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 shutdown_connections 2023-10-16T12:46:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84f806b070 0x7f84f806d530 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f851415ebc0 0x7f85140c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85140ad970 0x7f85140c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 --2- 172.21.15.60:0/3821819885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f851406c040 0x7f8514157910 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 >> 172.21.15.60:0/3821819885 conn(0x7f85140ce6a0 msgr2=0x7f85140ca7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 shutdown_connections 2023-10-16T12:46:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.602+0000 7f851a8ad640 1 -- 172.21.15.60:0/3821819885 wait complete. 2023-10-16T12:46:30.624 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:34:59", "first": "2023-10-16T12:35:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-16T12:46:30.624 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:34:59', 'first': '2023-10-16T12:35:00', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:30.624 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-16T12:46:30.625 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-16T12:46:30.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:30.625 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T12:46:30.683 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:30.684 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:30.684 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:30.684 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:30.684 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:30.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:30.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:30.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:30.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:30.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_01_UTC 2023-10-16T12:46:30.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:30.686 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_01_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_01_UTC', 'scheduled-2023-10-16-12_39_01_UTC', 'scheduled-2023-10-16-12_40_01_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_01_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_01_UTC', 'scheduled-2023-10-16-12_45_01_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:30.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-10-16T12:46:30.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.914+0000 7f1090b47640 1 Processor -- start 2023-10-16T12:46:30.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.915+0000 7f1090b47640 1 -- start start 2023-10-16T12:46:30.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f1090b47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f108c15cd70 0x7f108c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f1090b47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108c15d150 con 0x7f108c0cded0 2023-10-16T12:46:30.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f1090b47640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108c0d1180 con 0x7f108c0b8480 2023-10-16T12:46:30.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f1090b47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108c0d1460 con 0x7f108c15cd70 2023-10-16T12:46:30.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f108b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f108c15cd70 0x7f108c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f108bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f108c0cded0 0x7f108c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54798/0 (socket says 172.21.15.60:54798) 2023-10-16T12:46:30.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.916+0000 7f108bfff640 1 -- 172.21.15.60:0/1815742016 learned_addr learned my addr 172.21.15.60:0/1815742016 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:30.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.917+0000 7f108a7fc640 1 -- 172.21.15.60:0/1815742016 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030322056 0 0) 0x7f108c15d150 con 0x7f108c0cded0 2023-10-16T12:46:30.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.917+0000 7f108a7fc640 1 -- 172.21.15.60:0/1815742016 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1074003520 con 0x7f108c0cded0 2023-10-16T12:46:30.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.917+0000 7f108b7fe640 1 -- 172.21.15.60:0/1815742016 >> v1:172.21.15.144:6789/0 conn(0x7f108c0b8480 legacy=0x7f108c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.917+0000 7f108b7fe640 1 -- 172.21.15.60:0/1815742016 >> v1:172.21.15.60:6789/0 conn(0x7f108c0cded0 legacy=0x7f108c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.917+0000 7f108b7fe640 1 -- 172.21.15.60:0/1815742016 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0d1740 con 0x7f108c15cd70 2023-10-16T12:46:30.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.918+0000 7f108b7fe640 1 --2- 172.21.15.60:0/1815742016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c15cd70 0x7f108c0d2220 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f107c0097a0 tx=0x7f107c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b49a9b5b524a55 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:30.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f108a7fc640 1 -- 172.21.15.60:0/1815742016 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c030610 con 0x7f108c15cd70 2023-10-16T12:46:30.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f108a7fc640 1 -- 172.21.15.60:0/1815742016 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107c030770 con 0x7f108c15cd70 2023-10-16T12:46:30.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f108a7fc640 1 -- 172.21.15.60:0/1815742016 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c030ae0 con 0x7f108c15cd70 2023-10-16T12:46:30.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 -- 172.21.15.60:0/1815742016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c15cd70 msgr2=0x7f108c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 --2- 172.21.15.60:0/1815742016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c15cd70 0x7f108c0d2220 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f107c0097a0 tx=0x7f107c02b960 comp rx=0 tx=0).stop 2023-10-16T12:46:30.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 -- 172.21.15.60:0/1815742016 shutdown_connections 2023-10-16T12:46:30.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 --2- 172.21.15.60:0/1815742016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c15cd70 0x7f108c0d2220 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 -- 172.21.15.60:0/1815742016 >> 172.21.15.60:0/1815742016 conn(0x7f108c0cf570 msgr2=0x7f108c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:30.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 -- 172.21.15.60:0/1815742016 shutdown_connections 2023-10-16T12:46:30.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.919+0000 7f1090b47640 1 -- 172.21.15.60:0/1815742016 wait complete. 2023-10-16T12:46:30.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.921+0000 7f1090b47640 1 Processor -- start 2023-10-16T12:46:30.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.921+0000 7f1090b47640 1 -- start start 2023-10-16T12:46:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f1090b47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 0x7f108c0cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f1090b47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108c0cded0 0x7f108c1f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f1090b47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108c15cd70 0x7f108c1f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f1090b47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1f2d00 con 0x7f108c0b8480 2023-10-16T12:46:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f1090b47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c0cac10 con 0x7f108c15cd70 2023-10-16T12:46:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f1090b47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c0caeb0 con 0x7f108c0cded0 2023-10-16T12:46:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.922+0000 7f108b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 0x7f108c0cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 0x7f108c0cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42276/0 (socket says 172.21.15.60:42276) 2023-10-16T12:46:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 -- 172.21.15.60:0/1002200464 learned_addr learned my addr 172.21.15.60:0/1002200464 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108bfff640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108c15cd70 0x7f108c1f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108affd640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108c0cded0 0x7f108c1f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 -- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108c0cded0 msgr2=0x7f108c1f2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108c0cded0 0x7f108c1f2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 -- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108c15cd70 msgr2=0x7f108c1f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108c15cd70 0x7f108c1f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108b7fe640 1 -- 172.21.15.60:0/1002200464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c009400 con 0x7f108c0b8480 2023-10-16T12:46:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108affd640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108c0cded0 0x7f108c1f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.923+0000 7f108bfff640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108c15cd70 0x7f108c1f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.924+0000 7f108b7fe640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 0x7f108c0cbf50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f107c000c00 tx=0x7f107c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.924+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c03f070 con 0x7f108c0b8480 2023-10-16T12:46:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.924+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107c030e30 con 0x7f108c0b8480 2023-10-16T12:46:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.924+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108c0cb190 con 0x7f108c0b8480 2023-10-16T12:46:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.924+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c004230 con 0x7f108c0b8480 2023-10-16T12:46:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.924+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c0cb750 con 0x7f108c0b8480 2023-10-16T12:46:30.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.925+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108c04e9b0 con 0x7f108c0b8480 2023-10-16T12:46:30.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.926+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f107c043070 con 0x7f108c0b8480 2023-10-16T12:46:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.927+0000 7f1073fff640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f107806b070 0x7f107806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.927+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f107c0ab960 con 0x7f108c0b8480 2023-10-16T12:46:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.928+0000 7f108affd640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f107806b070 0x7f107806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.929+0000 7f108affd640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f107806b070 0x7f107806d530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f108c15b270 tx=0x7f1080009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:30.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:30.931+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f107c076600 con 0x7f108c0b8480 2023-10-16T12:46:31.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.150+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f108c0d1d00 con 0x7f107806b070 2023-10-16T12:46:31.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.169+0000 7f1073fff640 1 -- 172.21.15.60:0/1002200464 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f1080001340 con 0x7f107806b070 2023-10-16T12:46:31.171 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:31.172 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:00", "first": "2023-10-16T12:36:01", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:31.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f107806b070 msgr2=0x7f107806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f107806b070 0x7f107806d530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f108c15b270 tx=0x7f1080009040 comp rx=0 tx=0).stop 2023-10-16T12:46:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 msgr2=0x7f108c0cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 0x7f108c0cbf50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f107c000c00 tx=0x7f107c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 shutdown_connections 2023-10-16T12:46:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f107806b070 0x7f107806d530 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108c0cded0 0x7f108c1f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108c15cd70 0x7f108c1f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 --2- 172.21.15.60:0/1002200464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f108c0b8480 0x7f108c0cbf50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 >> 172.21.15.60:0/1002200464 conn(0x7f108c0cf570 msgr2=0x7f108c0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:31.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 shutdown_connections 2023-10-16T12:46:31.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.172+0000 7f1090b47640 1 -- 172.21.15.60:0/1002200464 wait complete. 2023-10-16T12:46:31.193 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:00", "first": "2023-10-16T12:36:01", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:31.193 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:00', 'first': '2023-10-16T12:36:01', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:31.194 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:31.194 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:31.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:31.195 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T12:46:31.248 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:31.248 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:31.248 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:31.249 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:31.249 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:31.249 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:31.249 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:31.249 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:31.250 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:31.250 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:31.250 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:31.251 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_01_UTC', 'scheduled-2023-10-16-12_38_01_UTC', 'scheduled-2023-10-16-12_39_01_UTC', 'scheduled-2023-10-16-12_40_01_UTC', 'scheduled-2023-10-16-12_41_01_UTC', 'scheduled-2023-10-16-12_42_01_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_01_UTC'] 2023-10-16T12:46:31.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-10-16T12:46:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.473+0000 7f9f8714e640 1 Processor -- start 2023-10-16T12:46:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.473+0000 7f9f8714e640 1 -- start start 2023-10-16T12:46:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8714e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9f800ca0a0 0x7f9f8006c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8714e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8015d910 con 0x7f9f80159950 2023-10-16T12:46:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8714e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8015dbf0 con 0x7f9f80070b60 2023-10-16T12:46:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8714e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8015e490 con 0x7f9f800ca0a0 2023-10-16T12:46:31.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8694d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9f800ca0a0 0x7f9f8006c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:31.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8614c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9f80159950 0x7f9f80070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54802/0 (socket says 172.21.15.60:54802) 2023-10-16T12:46:31.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8614c640 1 -- 172.21.15.60:0/570132830 learned_addr learned my addr 172.21.15.60:0/570132830 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:31.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.475+0000 7f9f8514a640 1 -- 172.21.15.60:0/570132830 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942136506 0 0) 0x7f9f8015d910 con 0x7f9f80159950 2023-10-16T12:46:31.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.476+0000 7f9f8514a640 1 -- 172.21.15.60:0/570132830 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6c003520 con 0x7f9f80159950 2023-10-16T12:46:31.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.476+0000 7f9f8694d640 1 -- 172.21.15.60:0/570132830 >> v1:172.21.15.144:6789/0 conn(0x7f9f80070b60 legacy=0x7f9f8006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:31.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.476+0000 7f9f8694d640 1 -- 172.21.15.60:0/570132830 >> v1:172.21.15.60:6789/0 conn(0x7f9f80159950 legacy=0x7f9f80070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.476+0000 7f9f8694d640 1 -- 172.21.15.60:0/570132830 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8015ded0 con 0x7f9f800ca0a0 2023-10-16T12:46:31.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.477+0000 7f9f8694d640 1 --2- 172.21.15.60:0/570132830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8006c810 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009d90 tx=0x7f9f7802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7beb970560947c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:31.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.477+0000 7f9f8514a640 1 -- 172.21.15.60:0/570132830 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7802cc60 con 0x7f9f800ca0a0 2023-10-16T12:46:31.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.477+0000 7f9f8514a640 1 -- 172.21.15.60:0/570132830 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7802cdc0 con 0x7f9f800ca0a0 2023-10-16T12:46:31.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.477+0000 7f9f8514a640 1 -- 172.21.15.60:0/570132830 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78034660 con 0x7f9f800ca0a0 2023-10-16T12:46:31.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 -- 172.21.15.60:0/570132830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 msgr2=0x7f9f8006c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 --2- 172.21.15.60:0/570132830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8006c810 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009d90 tx=0x7f9f7802beb0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 -- 172.21.15.60:0/570132830 shutdown_connections 2023-10-16T12:46:31.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 --2- 172.21.15.60:0/570132830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8006c810 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 -- 172.21.15.60:0/570132830 >> 172.21.15.60:0/570132830 conn(0x7f9f800ce6a0 msgr2=0x7f9f800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:31.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 -- 172.21.15.60:0/570132830 shutdown_connections 2023-10-16T12:46:31.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.478+0000 7f9f8714e640 1 -- 172.21.15.60:0/570132830 wait complete. 2023-10-16T12:46:31.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.480+0000 7f9f8714e640 1 Processor -- start 2023-10-16T12:46:31.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.480+0000 7f9f8714e640 1 -- start start 2023-10-16T12:46:31.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8714e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f80070b60 0x7f9f800c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:31.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8714e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:31.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8714e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f80159950 0x7f9f80144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:31.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8714e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c8680 con 0x7f9f800ca0a0 2023-10-16T12:46:31.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8714e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80151560 con 0x7f9f80070b60 2023-10-16T12:46:31.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8714e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80151840 con 0x7f9f80159950 2023-10-16T12:46:31.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8594b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:31.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8594b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8014fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42292/0 (socket says 172.21.15.60:42292) 2023-10-16T12:46:31.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8594b640 1 -- 172.21.15.60:0/1993057016 learned_addr learned my addr 172.21.15.60:0/1993057016 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:31.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.481+0000 7f9f8614c640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f80070b60 0x7f9f800c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:31.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8694d640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f80159950 0x7f9f80144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:31.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8594b640 1 -- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f80159950 msgr2=0x7f9f80144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8594b640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f80159950 0x7f9f80144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8594b640 1 -- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f80070b60 msgr2=0x7f9f800c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8594b640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f80070b60 0x7f9f800c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8594b640 1 -- 172.21.15.60:0/1993057016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f78009a70 con 0x7f9f800ca0a0 2023-10-16T12:46:31.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f8594b640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8014fde0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c450 tx=0x7f9f7000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:31.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700052d0 con 0x7f9f800ca0a0 2023-10-16T12:46:31.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70005430 con 0x7f9f800ca0a0 2023-10-16T12:46:31.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.482+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70012870 con 0x7f9f800ca0a0 2023-10-16T12:46:31.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.483+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f80151b20 con 0x7f9f800ca0a0 2023-10-16T12:46:31.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.483+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f800c8160 con 0x7f9f800ca0a0 2023-10-16T12:46:31.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.484+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8004e9b0 con 0x7f9f800ca0a0 2023-10-16T12:46:31.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.485+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f9f700129d0 con 0x7f9f800ca0a0 2023-10-16T12:46:31.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.486+0000 7f9f76ffd640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:31.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.486+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9f70089510 con 0x7f9f800ca0a0 2023-10-16T12:46:31.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.487+0000 7f9f8614c640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:31.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.488+0000 7f9f8614c640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f9f6406b0a0 0x7f9f6406d560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0011e0 tx=0x7f9f7c009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:31.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.490+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f700542d0 con 0x7f9f800ca0a0 2023-10-16T12:46:31.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.708+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f80159d30 con 0x7f9f6406b0a0 2023-10-16T12:46:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.727+0000 7f9f76ffd640 1 -- 172.21.15.60:0/1993057016 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f9f7c001330 con 0x7f9f6406b0a0 2023-10-16T12:46:31.734 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:31.734 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:01", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f9f6406b0a0 msgr2=0x7f9f6406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f9f6406b0a0 0x7f9f6406d560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0011e0 tx=0x7f9f7c009040 comp rx=0 tx=0).stop 2023-10-16T12:46:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 msgr2=0x7f9f8014fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8014fde0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c450 tx=0x7f9f7000c910 comp rx=0 tx=0).stop 2023-10-16T12:46:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 shutdown_connections 2023-10-16T12:46:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f9f6406b0a0 0x7f9f6406d560 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9f80159950 0x7f9f80144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f800ca0a0 0x7f9f8014fde0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 --2- 172.21.15.60:0/1993057016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f80070b60 0x7f9f800c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.730+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 >> 172.21.15.60:0/1993057016 conn(0x7f9f800ce6a0 msgr2=0x7f9f800c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.731+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 shutdown_connections 2023-10-16T12:46:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:31.731+0000 7f9f8714e640 1 -- 172.21.15.60:0/1993057016 wait complete. 2023-10-16T12:46:31.757 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:01", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:31.758 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:01', 'first': '2023-10-16T12:36:00', 'last': '2023-10-16T12:46:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:31.758 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:31.758 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:31.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:31.759 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T12:46:31.826 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:31.827 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:31.827 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:31.827 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:31.827 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:31.827 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:31.828 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:31.828 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:31.828 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:31.828 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:31.828 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:31.829 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_00_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_01_UTC', 'scheduled-2023-10-16-12_44_01_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_01_UTC'] 2023-10-16T12:46:31.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-10-16T12:46:32.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.066+0000 7ff4ebf33640 1 Processor -- start 2023-10-16T12:46:32.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.067+0000 7ff4ebf33640 1 -- start start 2023-10-16T12:46:32.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.068+0000 7ff4ebf33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4e40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.068+0000 7ff4ebf33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40cb030 con 0x7ff4e415ad70 2023-10-16T12:46:32.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4ebf33640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40c91c0 con 0x7ff4e40c9ca0 2023-10-16T12:46:32.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4ebf33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c94a0 con 0x7ff4e406a4b0 2023-10-16T12:46:32.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4e9ccf640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff4e415ad70 0x7ff4e4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54818/0 (socket says 172.21.15.60:54818) 2023-10-16T12:46:32.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4e9ccf640 1 -- 172.21.15.60:0/3455240435 learned_addr learned my addr 172.21.15.60:0/3455240435 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:32.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4e40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151296843 0 0) 0x7ff4e40cb030 con 0x7ff4e415ad70 2023-10-16T12:46:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.069+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d0003520 con 0x7ff4e415ad70 2023-10-16T12:46:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066460248 0 0) 0x7ff4d0003520 con 0x7ff4e415ad70 2023-10-16T12:46:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 msgr2=0x7ff4e40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4e40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 >> v1:172.21.15.144:6789/0 conn(0x7ff4e40c9ca0 legacy=0x7ff4e40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e40c9780 con 0x7ff4e415ad70 2023-10-16T12:46:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff4e00019f0 con 0x7ff4e415ad70 2023-10-16T12:46:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 >> v1:172.21.15.60:6789/0 conn(0x7ff4e415ad70 legacy=0x7ff4e4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4d00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4d0004d00 0x7ff4d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c94a0 con 0x7ff4d0003f70 2023-10-16T12:46:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d000bac0 con 0x7ff4d0004d00 2023-10-16T12:46:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d000bc80 con 0x7ff4e406a4b0 2023-10-16T12:46:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.070+0000 7ff4e9ccf640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4d00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4e94ce640 1 -- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 msgr2=0x7ff4d00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4d00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4e94ce640 1 -- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4d0004d00 msgr2=0x7ff4d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4ea4d0640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4d0004d00 0x7ff4d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4d0004d00 0x7ff4d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4e94ce640 1 -- 172.21.15.60:0/3455240435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d0003520 con 0x7ff4d0003f70 2023-10-16T12:46:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4ea4d0640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4d0004d00 0x7ff4d0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.071+0000 7ff4e9ccf640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e406a4b0 0x7ff4d00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.072+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4d00047c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009ac0 tx=0x7ff4d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2459c2627095909 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.072+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d402c600 con 0x7ff4d0003f70 2023-10-16T12:46:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.072+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d402c760 con 0x7ff4d0003f70 2023-10-16T12:46:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.072+0000 7ff4e8ccd640 1 -- 172.21.15.60:0/3455240435 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d402caf0 con 0x7ff4d0003f70 2023-10-16T12:46:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ebf33640 1 -- 172.21.15.60:0/3455240435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 msgr2=0x7ff4d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/3455240435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4d00047c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009ac0 tx=0x7ff4d00057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ea4d0640 1 -- 172.21.15.60:0/3455240435 reap_dead start 2023-10-16T12:46:32.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ebf33640 1 -- 172.21.15.60:0/3455240435 shutdown_connections 2023-10-16T12:46:32.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ebf33640 1 -- 172.21.15.60:0/3455240435 >> 172.21.15.60:0/3455240435 conn(0x7ff4e40ce9b0 msgr2=0x7ff4e40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:32.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ebf33640 1 -- 172.21.15.60:0/3455240435 shutdown_connections 2023-10-16T12:46:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.073+0000 7ff4ebf33640 1 -- 172.21.15.60:0/3455240435 wait complete. 2023-10-16T12:46:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.075+0000 7ff4ebf33640 1 Processor -- start 2023-10-16T12:46:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.075+0000 7ff4ebf33640 1 -- start start 2023-10-16T12:46:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4ebf33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4e40af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4ebf33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4e40c5a80 0x7ff4e414dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4ebf33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e40c5f00 0x7ff4e414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4ebf33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c70c0 con 0x7ff4d0003f70 2023-10-16T12:46:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4ebf33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c6440 con 0x7ff4e40c5a80 2023-10-16T12:46:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4ebf33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c6720 con 0x7ff4e40c5f00 2023-10-16T12:46:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4e9ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4e40af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4e9ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4e40af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42312/0 (socket says 172.21.15.60:42312) 2023-10-16T12:46:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4e9ccf640 1 -- 172.21.15.60:0/224710945 learned_addr learned my addr 172.21.15.60:0/224710945 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.076+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4e40c5a80 0x7ff4e414dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4e9ccf640 1 -- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e40c5f00 msgr2=0x7ff4e414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4e9ccf640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e40c5f00 0x7ff4e414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4e9ccf640 1 -- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4e40c5a80 msgr2=0x7ff4e414dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4e9ccf640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4e40c5a80 0x7ff4e414dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4e9ccf640 1 -- 172.21.15.60:0/224710945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e40c6a00 con 0x7ff4d0003f70 2023-10-16T12:46:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4e9ccf640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4e40af3b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0007c40 tx=0x7ff4e000b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0004d60 con 0x7ff4d0003f70 2023-10-16T12:46:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e00070d0 con 0x7ff4d0003f70 2023-10-16T12:46:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e40c6ce0 con 0x7ff4d0003f70 2023-10-16T12:46:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0007460 con 0x7ff4d0003f70 2023-10-16T12:46:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.077+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e40afad0 con 0x7ff4d0003f70 2023-10-16T12:46:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.079+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e404e160 con 0x7ff4d0003f70 2023-10-16T12:46:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.080+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff4e00075c0 con 0x7ff4d0003f70 2023-10-16T12:46:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.081+0000 7ff4da7fc640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff4c806b070 0x7ff4c806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.081+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff4e0089b90 con 0x7ff4d0003f70 2023-10-16T12:46:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.081+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff4c806b070 0x7ff4c806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.083+0000 7ff4e94ce640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff4c806b070 0x7ff4c806d530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4036000 tx=0x7ff4d4030000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:32.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.085+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4e0054710 con 0x7ff4d0003f70 2023-10-16T12:46:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.303+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4e40adbe0 con 0x7ff4c806b070 2023-10-16T12:46:32.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.320+0000 7ff4da7fc640 1 -- 172.21.15.60:0/224710945 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ff4d4004140 con 0x7ff4c806b070 2023-10-16T12:46:32.322 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:32.323 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:01", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:32.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.322+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff4c806b070 msgr2=0x7ff4c806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.322+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff4c806b070 0x7ff4c806d530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4036000 tx=0x7ff4d4030000 comp rx=0 tx=0).stop 2023-10-16T12:46:32.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.322+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 msgr2=0x7ff4e40af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.322+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4e40af3b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0007c40 tx=0x7ff4e000b560 comp rx=0 tx=0).stop 2023-10-16T12:46:32.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 shutdown_connections 2023-10-16T12:46:32.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7ff4c806b070 0x7ff4c806d530 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff4e40c5f00 0x7ff4e414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4e40c5a80 0x7ff4e414dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 --2- 172.21.15.60:0/224710945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4d0003f70 0x7ff4e40af3b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 >> 172.21.15.60:0/224710945 conn(0x7ff4e40ce9b0 msgr2=0x7ff4e415b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:32.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 shutdown_connections 2023-10-16T12:46:32.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.323+0000 7ff4ebf33640 1 -- 172.21.15.60:0/224710945 wait complete. 2023-10-16T12:46:32.344 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:01", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:32.345 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:01', 'first': '2023-10-16T12:36:00', 'last': '2023-10-16T12:46:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:32.345 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:32.345 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:32.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:32.346 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T12:46:32.406 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:32.406 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:32.406 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:32.407 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:32.407 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:32.407 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:32.407 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:32.407 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:32.408 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:32.408 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:32.408 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:32.409 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_01_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_01_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_01_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_01_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:32.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-10-16T12:46:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.640+0000 7fdb48083640 1 Processor -- start 2023-10-16T12:46:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.641+0000 7fdb48083640 1 -- start start 2023-10-16T12:46:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.642+0000 7fdb48083640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb4015ad70 0x7fdb40069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.642+0000 7fdb48083640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb400c9ca0 0x7fdb4006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.642+0000 7fdb48083640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400ca3b0 0x7fdb400ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.642+0000 7fdb48083640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb400cad80 con 0x7fdb400c9ca0 2023-10-16T12:46:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.642+0000 7fdb48083640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb400cb030 con 0x7fdb4015ad70 2023-10-16T12:46:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.642+0000 7fdb48083640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400c91c0 con 0x7fdb400ca3b0 2023-10-16T12:46:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb400c9ca0 0x7fdb4006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb400c9ca0 0x7fdb4006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42322/0 (socket says 172.21.15.60:42322) 2023-10-16T12:46:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 -- 172.21.15.60:0/1533858810 learned_addr learned my addr 172.21.15.60:0/1533858810 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb45e1f640 1 --2- 172.21.15.60:0/1533858810 >> v2:172.21.15.144:3300/0 conn(0x7fdb4015ad70 0x7fdb40069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb46620640 1 --2- 172.21.15.60:0/1533858810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400ca3b0 0x7fdb400ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 -- 172.21.15.60:0/1533858810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400ca3b0 msgr2=0x7fdb400ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 --2- 172.21.15.60:0/1533858810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400ca3b0 0x7fdb400ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 -- 172.21.15.60:0/1533858810 >> v2:172.21.15.144:3300/0 conn(0x7fdb4015ad70 msgr2=0x7fdb40069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 --2- 172.21.15.60:0/1533858810 >> v2:172.21.15.144:3300/0 conn(0x7fdb4015ad70 0x7fdb40069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.643+0000 7fdb4561e640 1 -- 172.21.15.60:0/1533858810 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400c94a0 con 0x7fdb400c9ca0 2023-10-16T12:46:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.645+0000 7fdb4561e640 1 --2- 172.21.15.60:0/1533858810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb400c9ca0 0x7fdb4006a2e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdb400cf780 tx=0x7fdb3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831443984274a3ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.645+0000 7fdb44e1d640 1 -- 172.21.15.60:0/1533858810 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30031870 con 0x7fdb400c9ca0 2023-10-16T12:46:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb44e1d640 1 -- 172.21.15.60:0/1533858810 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb300319d0 con 0x7fdb400c9ca0 2023-10-16T12:46:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb44e1d640 1 -- 172.21.15.60:0/1533858810 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30031d60 con 0x7fdb400c9ca0 2023-10-16T12:46:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 -- 172.21.15.60:0/1533858810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb400c9ca0 msgr2=0x7fdb4006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 --2- 172.21.15.60:0/1533858810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb400c9ca0 0x7fdb4006a2e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdb400cf780 tx=0x7fdb3002b910 comp rx=0 tx=0).stop 2023-10-16T12:46:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 -- 172.21.15.60:0/1533858810 shutdown_connections 2023-10-16T12:46:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 --2- 172.21.15.60:0/1533858810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400ca3b0 0x7fdb400ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 --2- 172.21.15.60:0/1533858810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb400c9ca0 0x7fdb4006a2e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 --2- 172.21.15.60:0/1533858810 >> v2:172.21.15.144:3300/0 conn(0x7fdb4015ad70 0x7fdb40069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 -- 172.21.15.60:0/1533858810 >> 172.21.15.60:0/1533858810 conn(0x7fdb400ce9b0 msgr2=0x7fdb400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 -- 172.21.15.60:0/1533858810 shutdown_connections 2023-10-16T12:46:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.646+0000 7fdb48083640 1 -- 172.21.15.60:0/1533858810 wait complete. 2023-10-16T12:46:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.648+0000 7fdb48083640 1 Processor -- start 2023-10-16T12:46:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.648+0000 7fdb48083640 1 -- start start 2023-10-16T12:46:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.649+0000 7fdb48083640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400c9ca0 0x7fdb400c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.649+0000 7fdb48083640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb400ca3b0 0x7fdb400c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb48083640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 0x7fdb401f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb48083640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400c47f0 con 0x7fdb4015ad70 2023-10-16T12:46:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb48083640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4015b850 con 0x7fdb400ca3b0 2023-10-16T12:46:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb48083640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400c4a40 con 0x7fdb400c9ca0 2023-10-16T12:46:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 0x7fdb401f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 0x7fdb401f5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42332/0 (socket says 172.21.15.60:42332) 2023-10-16T12:46:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 -- 172.21.15.60:0/3384723999 learned_addr learned my addr 172.21.15.60:0/3384723999 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb45e1f640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400c9ca0 0x7fdb400c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb4561e640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb400ca3b0 0x7fdb400c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 -- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400c9ca0 msgr2=0x7fdb400c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400c9ca0 0x7fdb400c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 -- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb400ca3b0 msgr2=0x7fdb400c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb400ca3b0 0x7fdb400c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.650+0000 7fdb46620640 1 -- 172.21.15.60:0/3384723999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb300094d0 con 0x7fdb4015ad70 2023-10-16T12:46:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.651+0000 7fdb46620640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 0x7fdb401f5630 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdb38007e60 tx=0x7fdb3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.651+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38014620 con 0x7fdb4015ad70 2023-10-16T12:46:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.651+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3800ee90 con 0x7fdb4015ad70 2023-10-16T12:46:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.651+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb400b3340 con 0x7fdb4015ad70 2023-10-16T12:46:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.651+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380149e0 con 0x7fdb4015ad70 2023-10-16T12:46:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.651+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb40158d00 con 0x7fdb4015ad70 2023-10-16T12:46:32.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.653+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb40158ef0 con 0x7fdb4015ad70 2023-10-16T12:46:32.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.654+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fdb38014b40 con 0x7fdb4015ad70 2023-10-16T12:46:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.654+0000 7fdb367fc640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdb2406b0a0 0x7fdb2406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.655+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdb3808a3b0 con 0x7fdb4015ad70 2023-10-16T12:46:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.655+0000 7fdb45e1f640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdb2406b0a0 0x7fdb2406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.656+0000 7fdb45e1f640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdb2406b0a0 0x7fdb2406d560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c006fd0 tx=0x7fdb3c008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.659+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb380550e0 con 0x7fdb4015ad70 2023-10-16T12:46:32.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.878+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb400c9780 con 0x7fdb2406b0a0 2023-10-16T12:46:32.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.898+0000 7fdb367fc640 1 -- 172.21.15.60:0/3384723999 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fdb3c00f860 con 0x7fdb2406b0a0 2023-10-16T12:46:32.901 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:32.901 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:02", "first": "2023-10-16T12:36:01", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:32.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdb2406b0a0 msgr2=0x7fdb2406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdb2406b0a0 0x7fdb2406d560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c006fd0 tx=0x7fdb3c008040 comp rx=0 tx=0).stop 2023-10-16T12:46:32.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 msgr2=0x7fdb401f5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:32.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 0x7fdb401f5630 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdb38007e60 tx=0x7fdb3800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 shutdown_connections 2023-10-16T12:46:32.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fdb2406b0a0 0x7fdb2406d560 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4015ad70 0x7fdb401f5630 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb400ca3b0 0x7fdb400c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 --2- 172.21.15.60:0/3384723999 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb400c9ca0 0x7fdb400c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:32.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 >> 172.21.15.60:0/3384723999 conn(0x7fdb400ce9b0 msgr2=0x7fdb4006b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:32.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 shutdown_connections 2023-10-16T12:46:32.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:32.901+0000 7fdb48083640 1 -- 172.21.15.60:0/3384723999 wait complete. 2023-10-16T12:46:32.924 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:02", "first": "2023-10-16T12:36:01", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:32.925 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:02', 'first': '2023-10-16T12:36:01', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:32.925 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:32.925 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:32.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:32.926 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T12:46:32.990 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:32.990 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:32.991 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:32.991 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:32.991 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:32.991 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:32.991 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:32.992 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:32.992 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:32.992 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_01_UTC 2023-10-16T12:46:32.992 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:32.993 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_01_UTC', 'scheduled-2023-10-16-12_40_01_UTC', 'scheduled-2023-10-16-12_41_01_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_01_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:32.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-10-16T12:46:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.225+0000 7f10f106e640 1 Processor -- start 2023-10-16T12:46:33.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.226+0000 7f10f106e640 1 -- start start 2023-10-16T12:46:33.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.227+0000 7f10f106e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10ec06c040 0x7f10ec06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.227+0000 7f10f106e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f10ec0cda80 0x7f10ec15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.227+0000 7f10f106e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec06c960 con 0x7f10ec0ad970 2023-10-16T12:46:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.227+0000 7f10f106e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec15b9d0 con 0x7f10ec06c040 2023-10-16T12:46:33.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.227+0000 7f10f106e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec15bcb0 con 0x7f10ec0cda80 2023-10-16T12:46:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10ec06c040 0x7f10ec06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10ec06c040 0x7f10ec06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42334/0 (socket says 172.21.15.60:42334) 2023-10-16T12:46:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 -- 172.21.15.60:0/1413931442 learned_addr learned my addr 172.21.15.60:0/1413931442 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10f086d640 1 --2- 172.21.15.60:0/1413931442 >> v2:172.21.15.144:3300/0 conn(0x7f10ec0cda80 0x7f10ec15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 -- 172.21.15.60:0/1413931442 >> v2:172.21.15.144:3300/0 conn(0x7f10ec0cda80 msgr2=0x7f10ec15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:33.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 --2- 172.21.15.60:0/1413931442 >> v2:172.21.15.144:3300/0 conn(0x7f10ec0cda80 0x7f10ec15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 -- 172.21.15.60:0/1413931442 >> v1:172.21.15.144:6789/0 conn(0x7f10ec0ad970 legacy=0x7f10ec15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:33.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.228+0000 7f10ebfff640 1 -- 172.21.15.60:0/1413931442 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec15bf90 con 0x7f10ec06c040 2023-10-16T12:46:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.230+0000 7f10ebfff640 1 --2- 172.21.15.60:0/1413931442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec06c040 0x7f10ec06c420 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0cf300 tx=0x7f10dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3bb15689039879 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10eaffd640 1 -- 172.21.15.60:0/1413931442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc031870 con 0x7f10ec06c040 2023-10-16T12:46:33.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10eaffd640 1 -- 172.21.15.60:0/1413931442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc0319d0 con 0x7f10ec06c040 2023-10-16T12:46:33.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10eaffd640 1 -- 172.21.15.60:0/1413931442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc031d60 con 0x7f10ec06c040 2023-10-16T12:46:33.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10f106e640 1 -- 172.21.15.60:0/1413931442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec06c040 msgr2=0x7f10ec06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10f106e640 1 --2- 172.21.15.60:0/1413931442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec06c040 0x7f10ec06c420 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0cf300 tx=0x7f10dc02b910 comp rx=0 tx=0).stop 2023-10-16T12:46:33.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10f106e640 1 -- 172.21.15.60:0/1413931442 shutdown_connections 2023-10-16T12:46:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10f106e640 1 --2- 172.21.15.60:0/1413931442 >> v2:172.21.15.144:3300/0 conn(0x7f10ec0cda80 0x7f10ec15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10f106e640 1 --2- 172.21.15.60:0/1413931442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec06c040 0x7f10ec06c420 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.231+0000 7f10f106e640 1 -- 172.21.15.60:0/1413931442 >> 172.21.15.60:0/1413931442 conn(0x7f10ec0ce6a0 msgr2=0x7f10ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.232+0000 7f10f106e640 1 -- 172.21.15.60:0/1413931442 shutdown_connections 2023-10-16T12:46:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.232+0000 7f10f106e640 1 -- 172.21.15.60:0/1413931442 wait complete. 2023-10-16T12:46:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.233+0000 7f10f106e640 1 Processor -- start 2023-10-16T12:46:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.233+0000 7f10f106e640 1 -- start start 2023-10-16T12:46:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.234+0000 7f10f106e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ec06c040 0x7f10ec15ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.234+0000 7f10f106e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10ec0ad970 0x7f10ec15d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.234+0000 7f10f106e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 0x7f10ec15d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.234+0000 7f10f106e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec1f3350 con 0x7f10ec0cda80 2023-10-16T12:46:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.234+0000 7f10f106e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0b0f10 con 0x7f10ec06c040 2023-10-16T12:46:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.234+0000 7f10f106e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0c67c0 con 0x7f10ec0ad970 2023-10-16T12:46:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 0x7f10ec15d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 0x7f10ec15d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42344/0 (socket says 172.21.15.60:42344) 2023-10-16T12:46:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 -- 172.21.15.60:0/2754979495 learned_addr learned my addr 172.21.15.60:0/2754979495 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10ebfff640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ec06c040 0x7f10ec15ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10eb7fe640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10ec0ad970 0x7f10ec15d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 -- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10ec0ad970 msgr2=0x7f10ec15d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10ec0ad970 0x7f10ec15d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 -- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ec06c040 msgr2=0x7f10ec15ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ec06c040 0x7f10ec15ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.235+0000 7f10f086d640 1 -- 172.21.15.60:0/2754979495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0094d0 con 0x7f10ec0cda80 2023-10-16T12:46:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.236+0000 7f10f086d640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 0x7f10ec15d890 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f10e400efc0 tx=0x7f10e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.236+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e4009ba0 con 0x7f10ec0cda80 2023-10-16T12:46:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.236+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e400f040 con 0x7f10ec0cda80 2023-10-16T12:46:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.236+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e40146c0 con 0x7f10ec0cda80 2023-10-16T12:46:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.236+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec0c6aa0 con 0x7f10ec0cda80 2023-10-16T12:46:33.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.236+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec0c7060 con 0x7f10ec0cda80 2023-10-16T12:46:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.237+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec06b330 con 0x7f10ec0cda80 2023-10-16T12:46:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.238+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f10e4014820 con 0x7f10ec0cda80 2023-10-16T12:46:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.239+0000 7f10e8ff9640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f10d806b070 0x7f10d806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.239+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f10e408a6a0 con 0x7f10ec0cda80 2023-10-16T12:46:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.240+0000 7f10ebfff640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f10d806b070 0x7f10d806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.241+0000 7f10ebfff640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f10d806b070 0x7f10d806d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f10dc036000 tx=0x7f10dc03c040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.243+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10e40552b0 con 0x7f10ec0cda80 2023-10-16T12:46:33.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.461+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10ec0cc550 con 0x7f10d806b070 2023-10-16T12:46:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.479+0000 7f10e8ff9640 1 -- 172.21.15.60:0/2754979495 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f10dc031c40 con 0x7f10d806b070 2023-10-16T12:46:33.482 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:33.482 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:03", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f10d806b070 msgr2=0x7f10d806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f10d806b070 0x7f10d806d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f10dc036000 tx=0x7f10dc03c040 comp rx=0 tx=0).stop 2023-10-16T12:46:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 msgr2=0x7f10ec15d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 0x7f10ec15d890 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f10e400efc0 tx=0x7f10e400c490 comp rx=0 tx=0).stop 2023-10-16T12:46:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 shutdown_connections 2023-10-16T12:46:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f10d806b070 0x7f10d806d530 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10ec0ad970 0x7f10ec15d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10ec0cda80 0x7f10ec15d890 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 --2- 172.21.15.60:0/2754979495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10ec06c040 0x7f10ec15ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 >> 172.21.15.60:0/2754979495 conn(0x7f10ec0ce6a0 msgr2=0x7f10ec0c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 shutdown_connections 2023-10-16T12:46:33.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.482+0000 7f10f106e640 1 -- 172.21.15.60:0/2754979495 wait complete. 2023-10-16T12:46:33.504 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:03", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:33.505 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:03', 'first': '2023-10-16T12:36:00', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:33.505 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:33.505 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:33.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:33.506 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T12:46:33.567 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:33.567 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:33.567 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:33.567 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:33.568 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:33.568 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:33.568 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:33.568 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:33.568 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:33.569 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:33.569 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:33.570 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_00_UTC', 'scheduled-2023-10-16-12_40_01_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_01_UTC'] 2023-10-16T12:46:33.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-10-16T12:46:33.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.800+0000 7f388392c640 1 Processor -- start 2023-10-16T12:46:33.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.800+0000 7f388392c640 1 -- start start 2023-10-16T12:46:33.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.801+0000 7f388392c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f387c15d330 0x7f387c1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.801+0000 7f388392c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c06c7d0 0x7f387c065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.801+0000 7f388392c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f387c06cc80 0x7f387c065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.801+0000 7f388392c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387c06b1e0 con 0x7f387c06cc80 2023-10-16T12:46:33.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.801+0000 7f388392c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387c0d0890 con 0x7f387c15d330 2023-10-16T12:46:33.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.801+0000 7f388392c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0d05b0 con 0x7f387c06c7d0 2023-10-16T12:46:33.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f387c06cc80 0x7f387c065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f387c06cc80 0x7f387c065ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42350/0 (socket says 172.21.15.60:42350) 2023-10-16T12:46:33.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 -- 172.21.15.60:0/3389538768 learned_addr learned my addr 172.21.15.60:0/3389538768 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:33.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3880ec7640 1 --2- 172.21.15.60:0/3389538768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c06c7d0 0x7f387c065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f38816c8640 1 --2- 172.21.15.60:0/3389538768 >> v2:172.21.15.144:3300/0 conn(0x7f387c15d330 0x7f387c1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 -- 172.21.15.60:0/3389538768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c06c7d0 msgr2=0x7f387c065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:33.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 --2- 172.21.15.60:0/3389538768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c06c7d0 0x7f387c065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 -- 172.21.15.60:0/3389538768 >> v2:172.21.15.144:3300/0 conn(0x7f387c15d330 msgr2=0x7f387c1618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:33.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 --2- 172.21.15.60:0/3389538768 >> v2:172.21.15.144:3300/0 conn(0x7f387c15d330 0x7f387c1618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.802+0000 7f3881ec9640 1 -- 172.21.15.60:0/3389538768 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c160dd0 con 0x7f387c06cc80 2023-10-16T12:46:33.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.803+0000 7f38816c8640 1 --2- 172.21.15.60:0/3389538768 >> v2:172.21.15.144:3300/0 conn(0x7f387c15d330 0x7f387c1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:33.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.804+0000 7f3881ec9640 1 --2- 172.21.15.60:0/3389538768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c065ff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f387c0744f0 tx=0x7f387402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56713eccefe0f0e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:33.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.804+0000 7f3873fff640 1 -- 172.21.15.60:0/3389538768 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387402c9e0 con 0x7f387c06cc80 2023-10-16T12:46:33.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.804+0000 7f3873fff640 1 -- 172.21.15.60:0/3389538768 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387402cb40 con 0x7f387c06cc80 2023-10-16T12:46:33.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.804+0000 7f3873fff640 1 -- 172.21.15.60:0/3389538768 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740343f0 con 0x7f387c06cc80 2023-10-16T12:46:33.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 -- 172.21.15.60:0/3389538768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 msgr2=0x7f387c065ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 --2- 172.21.15.60:0/3389538768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c065ff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f387c0744f0 tx=0x7f387402be40 comp rx=0 tx=0).stop 2023-10-16T12:46:33.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 -- 172.21.15.60:0/3389538768 shutdown_connections 2023-10-16T12:46:33.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 --2- 172.21.15.60:0/3389538768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c065ff0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 --2- 172.21.15.60:0/3389538768 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c06c7d0 0x7f387c065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 --2- 172.21.15.60:0/3389538768 >> v2:172.21.15.144:3300/0 conn(0x7f387c15d330 0x7f387c1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 -- 172.21.15.60:0/3389538768 >> 172.21.15.60:0/3389538768 conn(0x7f387c0d0ab0 msgr2=0x7f387c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:33.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 -- 172.21.15.60:0/3389538768 shutdown_connections 2023-10-16T12:46:33.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.805+0000 7f388392c640 1 -- 172.21.15.60:0/3389538768 wait complete. 2023-10-16T12:46:33.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.807+0000 7f388392c640 1 Processor -- start 2023-10-16T12:46:33.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.807+0000 7f388392c640 1 -- start start 2023-10-16T12:46:33.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f388392c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f387c06c7d0 0x7f387c157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f388392c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c0cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f388392c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c15d330 0x7f387c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f388392c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0c4860 con 0x7f387c06cc80 2023-10-16T12:46:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f388392c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c157880 con 0x7f387c06c7d0 2023-10-16T12:46:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f388392c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0c6980 con 0x7f387c15d330 2023-10-16T12:46:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f3880ec7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c0cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f3880ec7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c0cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42362/0 (socket says 172.21.15.60:42362) 2023-10-16T12:46:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.808+0000 7f3880ec7640 1 -- 172.21.15.60:0/3956423821 learned_addr learned my addr 172.21.15.60:0/3956423821 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3881ec9640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c15d330 0x7f387c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f38816c8640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f387c06c7d0 0x7f387c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3880ec7640 1 -- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c15d330 msgr2=0x7f387c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3880ec7640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c15d330 0x7f387c0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3880ec7640 1 -- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f387c06c7d0 msgr2=0x7f387c157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3880ec7640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f387c06c7d0 0x7f387c157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3880ec7640 1 -- 172.21.15.60:0/3956423821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874009b30 con 0x7f387c06cc80 2023-10-16T12:46:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3881ec9640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c15d330 0x7f387c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:33.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.809+0000 7f3880ec7640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c0cbb70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f386c00d5f0 tx=0x7f386c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:33.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.810+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c0125f0 con 0x7f387c06cc80 2023-10-16T12:46:33.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.810+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386c012750 con 0x7f387c06cc80 2023-10-16T12:46:33.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.810+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c0c6c60 con 0x7f387c06cc80 2023-10-16T12:46:33.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.810+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c012b00 con 0x7f387c06cc80 2023-10-16T12:46:33.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.810+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c0c7220 con 0x7f387c06cc80 2023-10-16T12:46:33.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.811+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c0c73e0 con 0x7f387c06cc80 2023-10-16T12:46:33.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.812+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f386c012c60 con 0x7f387c06cc80 2023-10-16T12:46:33.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.813+0000 7f3871ffb640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f386006b0a0 0x7f386006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:33.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.813+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f386c089fe0 con 0x7f387c06cc80 2023-10-16T12:46:33.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.814+0000 7f38816c8640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f386006b0a0 0x7f386006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:33.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.815+0000 7f38816c8640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f386006b0a0 0x7f386006d560 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3878006fd0 tx=0x7f3878008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:33.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:33.817+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f386c054c70 con 0x7f387c06cc80 2023-10-16T12:46:34.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.037+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f387c161390 con 0x7f386006b0a0 2023-10-16T12:46:34.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.056+0000 7f3871ffb640 1 -- 172.21.15.60:0/3956423821 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f38780094c0 con 0x7f386006b0a0 2023-10-16T12:46:34.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:34.059 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:04", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:34.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.058+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f386006b0a0 msgr2=0x7f386006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.058+0000 7f388392c640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f386006b0a0 0x7f386006d560 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3878006fd0 tx=0x7f3878008040 comp rx=0 tx=0).stop 2023-10-16T12:46:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.058+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 msgr2=0x7f387c0cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c0cbb70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f386c00d5f0 tx=0x7f386c00dab0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 shutdown_connections 2023-10-16T12:46:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f386006b0a0 0x7f386006d560 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f387c15d330 0x7f387c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f387c06cc80 0x7f387c0cbb70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 --2- 172.21.15.60:0/3956423821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f387c06c7d0 0x7f387c157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 >> 172.21.15.60:0/3956423821 conn(0x7f387c0d0ab0 msgr2=0x7f387c0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 shutdown_connections 2023-10-16T12:46:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.059+0000 7f388392c640 1 -- 172.21.15.60:0/3956423821 wait complete. 2023-10-16T12:46:34.081 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:04", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:34.082 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:04', 'first': '2023-10-16T12:36:00', 'last': '2023-10-16T12:46:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:34.082 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:34.083 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:34.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:34.083 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T12:46:34.141 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:34.142 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:34.142 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:34.142 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:34.142 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:34.142 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:34.143 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:34.143 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:34.143 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:34.143 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:34.144 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:34.145 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_01_UTC', 'scheduled-2023-10-16-12_37_01_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_00_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_01_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:34.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-10-16T12:46:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.374+0000 7fc7bc67c640 1 Processor -- start 2023-10-16T12:46:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.375+0000 7fc7bc67c640 1 -- start start 2023-10-16T12:46:34.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.376+0000 7fc7bc67c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc7b40b4580 0x7fc7b40cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.376+0000 7fc7bc67c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 0x7fc7b40714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.376+0000 7fc7bc67c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40cabe0 con 0x7fc7b40719f0 2023-10-16T12:46:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.376+0000 7fc7bc67c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b40b4960 con 0x7fc7b40b4580 2023-10-16T12:46:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.376+0000 7fc7bc67c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b406ff20 con 0x7fc7b40b4d50 2023-10-16T12:46:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.376+0000 7fc7bac19640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc7b40719f0 0x7fc7b415e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54822/0 (socket says 172.21.15.60:54822) 2023-10-16T12:46:34.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7bac19640 1 -- 172.21.15.60:0/3316625230 learned_addr learned my addr 172.21.15.60:0/3316625230 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9c17640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 0x7fc7b40714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7ba418640 1 --2- 172.21.15.60:0/3316625230 >> v2:172.21.15.144:3300/0 conn(0x7fc7b40b4580 0x7fc7b40cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024381789 0 0) 0x7fc7b40cabe0 con 0x7fc7b40719f0 2023-10-16T12:46:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0003520 con 0x7fc7b40719f0 2023-10-16T12:46:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1945493694 0 0) 0x7fc7a0003520 con 0x7fc7b40719f0 2023-10-16T12:46:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 msgr2=0x7fc7b40714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 0x7fc7b40714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 >> v2:172.21.15.144:3300/0 conn(0x7fc7b40b4580 msgr2=0x7fc7b40cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 --2- 172.21.15.60:0/3316625230 >> v2:172.21.15.144:3300/0 conn(0x7fc7b40b4580 0x7fc7b40cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.377+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b40ca2b0 con 0x7fc7b40719f0 2023-10-16T12:46:34.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc7ac0019f0 con 0x7fc7b40719f0 2023-10-16T12:46:34.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 >> v1:172.21.15.60:6789/0 conn(0x7fc7b40719f0 legacy=0x7fc7b415e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 0x7fc7a0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a0003f40 0x7fc7a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a0004d00 0x7fc7a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40b4960 con 0x7fc7a0004d00 2023-10-16T12:46:34.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b406ff20 con 0x7fc7a0003f40 2023-10-16T12:46:34.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a000b980 con 0x7fc7b40b4d50 2023-10-16T12:46:34.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7bac19640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a0004d00 0x7fc7a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7ba418640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 0x7fc7a0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9c17640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a0003f40 0x7fc7a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7bac19640 1 -- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 msgr2=0x7fc7a0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7bac19640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7b40b4d50 0x7fc7a0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7bac19640 1 -- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a0003f40 msgr2=0x7fc7a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7bac19640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a0003f40 0x7fc7a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7bac19640 1 -- 172.21.15.60:0/3316625230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0003520 con 0x7fc7a0004d00 2023-10-16T12:46:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.378+0000 7fc7b9c17640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a0003f40 0x7fc7a00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.380+0000 7fc7bac19640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a0004d00 0x7fc7a0005180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00a840 tx=0x7fc7a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf3e2e2cb9d009 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:34.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.380+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac034630 con 0x7fc7a0004d00 2023-10-16T12:46:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.380+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac02f070 con 0x7fc7a0004d00 2023-10-16T12:46:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.380+0000 7fc7b9416640 1 -- 172.21.15.60:0/3316625230 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac034a30 con 0x7fc7a0004d00 2023-10-16T12:46:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bc67c640 1 -- 172.21.15.60:0/3316625230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a0004d00 msgr2=0x7fc7a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/3316625230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a0004d00 0x7fc7a0005180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00a840 tx=0x7fc7a00057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bac19640 1 -- 172.21.15.60:0/3316625230 reap_dead start 2023-10-16T12:46:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bc67c640 1 -- 172.21.15.60:0/3316625230 shutdown_connections 2023-10-16T12:46:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bc67c640 1 -- 172.21.15.60:0/3316625230 >> 172.21.15.60:0/3316625230 conn(0x7fc7b406bc10 msgr2=0x7fc7b406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bc67c640 1 -- 172.21.15.60:0/3316625230 shutdown_connections 2023-10-16T12:46:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.381+0000 7fc7bc67c640 1 -- 172.21.15.60:0/3316625230 wait complete. 2023-10-16T12:46:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.383+0000 7fc7bc67c640 1 Processor -- start 2023-10-16T12:46:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.383+0000 7fc7bc67c640 1 -- start start 2023-10-16T12:46:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bc67c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a0004d00 0x7fc7b40ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bc67c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b415c280 0x7fc7b415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bc67c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 0x7fc7b40c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bc67c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40c7660 con 0x7fc7b415cc40 2023-10-16T12:46:34.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bc67c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4148ef0 con 0x7fc7b415c280 2023-10-16T12:46:34.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bc67c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b415d0c0 con 0x7fc7a0004d00 2023-10-16T12:46:34.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bac19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 0x7fc7b40c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bac19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 0x7fc7b40c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42374/0 (socket says 172.21.15.60:42374) 2023-10-16T12:46:34.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7bac19640 1 -- 172.21.15.60:0/2106024864 learned_addr learned my addr 172.21.15.60:0/2106024864 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:34.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7b9c17640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b415c280 0x7fc7b415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.384+0000 7fc7ba418640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a0004d00 0x7fc7b40ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bac19640 1 -- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a0004d00 msgr2=0x7fc7b40ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bac19640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a0004d00 0x7fc7b40ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bac19640 1 -- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b415c280 msgr2=0x7fc7b415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bac19640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b415c280 0x7fc7b415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bac19640 1 -- 172.21.15.60:0/2106024864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b40c7870 con 0x7fc7b415cc40 2023-10-16T12:46:34.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7ba418640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a0004d00 0x7fc7b40ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:34.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bac19640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 0x7fc7b40c8630 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00a7e0 tx=0x7fc7b415b1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:34.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac034430 con 0x7fc7b415cc40 2023-10-16T12:46:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac034590 con 0x7fc7b415cc40 2023-10-16T12:46:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac034900 con 0x7fc7b415cc40 2023-10-16T12:46:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.385+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b40c7b50 con 0x7fc7b415cc40 2023-10-16T12:46:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.386+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b40c8110 con 0x7fc7b415cc40 2023-10-16T12:46:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.387+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b404e160 con 0x7fc7b415cc40 2023-10-16T12:46:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.388+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc7ac031030 con 0x7fc7b415cc40 2023-10-16T12:46:34.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.388+0000 7fc7aaffd640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc79806b0a0 0x7fc79806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.389+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc7ac0abe40 con 0x7fc7b415cc40 2023-10-16T12:46:34.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.389+0000 7fc7ba418640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc79806b0a0 0x7fc79806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.390+0000 7fc7ba418640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc79806b0a0 0x7fc79806d560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009a80 tx=0x7fc7b0008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:34.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.393+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7ac076ae0 con 0x7fc7b415cc40 2023-10-16T12:46:34.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.610+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7b40ca870 con 0x7fc79806b0a0 2023-10-16T12:46:34.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.628+0000 7fc7aaffd640 1 -- 172.21.15.60:0/2106024864 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fc7b0012a10 con 0x7fc79806b0a0 2023-10-16T12:46:34.630 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:34.630 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:04", "first": "2023-10-16T12:36:01", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.630+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc79806b0a0 msgr2=0x7fc79806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.630+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc79806b0a0 0x7fc79806d560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009a80 tx=0x7fc7b0008040 comp rx=0 tx=0).stop 2023-10-16T12:46:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.630+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 msgr2=0x7fc7b40c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.630+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 0x7fc7b40c8630 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00a7e0 tx=0x7fc7b415b1c0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 shutdown_connections 2023-10-16T12:46:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc79806b0a0 0x7fc79806d560 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b415cc40 0x7fc7b40c8630 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7b415c280 0x7fc7b415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 --2- 172.21.15.60:0/2106024864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a0004d00 0x7fc7b40ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 >> 172.21.15.60:0/2106024864 conn(0x7fc7b406bc10 msgr2=0x7fc7b4069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 shutdown_connections 2023-10-16T12:46:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.631+0000 7fc7bc67c640 1 -- 172.21.15.60:0/2106024864 wait complete. 2023-10-16T12:46:34.652 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:04", "first": "2023-10-16T12:36:01", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:34.653 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:04', 'first': '2023-10-16T12:36:01', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:34.653 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:34.654 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:34.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:34.655 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T12:46:34.711 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:34.712 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:34.712 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:34.712 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:34.712 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:34.712 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:34.713 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:34.713 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:34.713 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:34.713 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:34.713 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:34.714 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_00_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_01_UTC', 'scheduled-2023-10-16-12_43_01_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_00_UTC'] 2023-10-16T12:46:34.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-10-16T12:46:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.942+0000 7f26d8df4640 1 Processor -- start 2023-10-16T12:46:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.943+0000 7f26d8df4640 1 -- start start 2023-10-16T12:46:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.944+0000 7f26d8df4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26d415cd70 0x7f26d40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.944+0000 7f26d8df4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d415d150 con 0x7f26d40b8480 2023-10-16T12:46:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.944+0000 7f26d8df4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40d1180 con 0x7f26d40cded0 2023-10-16T12:46:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.944+0000 7f26d8df4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d40d1460 con 0x7f26d415cd70 2023-10-16T12:46:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.945+0000 7f26cbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f26d40b8480 0x7f26d40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54832/0 (socket says 172.21.15.60:54832) 2023-10-16T12:46:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.945+0000 7f26cbfff640 1 -- 172.21.15.60:0/2660228957 learned_addr learned my addr 172.21.15.60:0/2660228957 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:34.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.945+0000 7f26d37fe640 1 --2- 172.21.15.60:0/2660228957 >> v2:172.21.15.60:3300/0 conn(0x7f26d415cd70 0x7f26d40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.945+0000 7f26d2ffd640 1 -- 172.21.15.60:0/2660228957 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761265585 0 0) 0x7f26d415d150 con 0x7f26d40b8480 2023-10-16T12:46:34.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.945+0000 7f26d2ffd640 1 -- 172.21.15.60:0/2660228957 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0003520 con 0x7f26d40b8480 2023-10-16T12:46:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.946+0000 7f26d37fe640 1 -- 172.21.15.60:0/2660228957 >> v1:172.21.15.144:6789/0 conn(0x7f26d40cded0 legacy=0x7f26d40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.946+0000 7f26d37fe640 1 -- 172.21.15.60:0/2660228957 >> v1:172.21.15.60:6789/0 conn(0x7f26d40b8480 legacy=0x7f26d40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.946+0000 7f26d2ffd640 1 -- 172.21.15.60:0/2660228957 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2659365916 0 0) 0x7f26c0003520 con 0x7f26d40b8480 2023-10-16T12:46:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.946+0000 7f26d37fe640 1 -- 172.21.15.60:0/2660228957 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40d1740 con 0x7f26d415cd70 2023-10-16T12:46:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.947+0000 7f26d37fe640 1 --2- 172.21.15.60:0/2660228957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d415cd70 0x7f26d40d2220 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f26c40097a0 tx=0x7f26c402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9408debb54be77f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.947+0000 7f26d2ffd640 1 -- 172.21.15.60:0/2660228957 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c402f050 con 0x7f26d415cd70 2023-10-16T12:46:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d2ffd640 1 -- 172.21.15.60:0/2660228957 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c4031500 con 0x7f26d415cd70 2023-10-16T12:46:34.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d2ffd640 1 -- 172.21.15.60:0/2660228957 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc001670 con 0x7f26d415cd70 2023-10-16T12:46:34.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 -- 172.21.15.60:0/2660228957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d415cd70 msgr2=0x7f26d40d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2660228957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d415cd70 0x7f26d40d2220 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f26c40097a0 tx=0x7f26c402b820 comp rx=0 tx=0).stop 2023-10-16T12:46:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 -- 172.21.15.60:0/2660228957 shutdown_connections 2023-10-16T12:46:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2660228957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d415cd70 0x7f26d40d2220 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 -- 172.21.15.60:0/2660228957 >> 172.21.15.60:0/2660228957 conn(0x7f26d40cf570 msgr2=0x7f26d40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 -- 172.21.15.60:0/2660228957 shutdown_connections 2023-10-16T12:46:34.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.948+0000 7f26d8df4640 1 -- 172.21.15.60:0/2660228957 wait complete. 2023-10-16T12:46:34.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.950+0000 7f26d8df4640 1 Processor -- start 2023-10-16T12:46:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.950+0000 7f26d8df4640 1 -- start start 2023-10-16T12:46:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.951+0000 7f26d8df4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26d40b8480 0x7f26d41f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.951+0000 7f26d8df4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 0x7f26d40b6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.951+0000 7f26d8df4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26d415cd70 0x7f26d40a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.951+0000 7f26d8df4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d4149b80 con 0x7f26d40cded0 2023-10-16T12:46:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.951+0000 7f26d8df4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d40b71e0 con 0x7f26d40b8480 2023-10-16T12:46:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.951+0000 7f26d8df4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d40b74a0 con 0x7f26d415cd70 2023-10-16T12:46:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 0x7f26d40b6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 0x7f26d40b6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42392/0 (socket says 172.21.15.60:42392) 2023-10-16T12:46:34.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 -- 172.21.15.60:0/2379718721 learned_addr learned my addr 172.21.15.60:0/2379718721 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26d37fe640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26d40b8480 0x7f26d41f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26d3fff640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26d415cd70 0x7f26d40a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 -- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26d415cd70 msgr2=0x7f26d40a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26d415cd70 0x7f26d40a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 -- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26d40b8480 msgr2=0x7f26d41f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26d40b8480 0x7f26d41f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26cbfff640 1 -- 172.21.15.60:0/2379718721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c4009400 con 0x7f26d40cded0 2023-10-16T12:46:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26d3fff640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26d415cd70 0x7f26d40a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.952+0000 7f26d37fe640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26d40b8480 0x7f26d41f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:34.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.953+0000 7f26cbfff640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 0x7f26d40b6740 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00c760 tx=0x7f26bc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.953+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc005ad0 con 0x7f26d40cded0 2023-10-16T12:46:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.953+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc005c30 con 0x7f26d40cded0 2023-10-16T12:46:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.953+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26bc007070 con 0x7f26d40cded0 2023-10-16T12:46:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.953+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d4149dd0 con 0x7f26d40cded0 2023-10-16T12:46:34.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.953+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d414fe50 con 0x7f26d40cded0 2023-10-16T12:46:34.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.954+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d414a2f0 con 0x7f26d40cded0 2023-10-16T12:46:34.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.955+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f26bc0071d0 con 0x7f26d40cded0 2023-10-16T12:46:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.956+0000 7f26d0ff9640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26b406b070 0x7f26b406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.956+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f26bc089900 con 0x7f26d40cded0 2023-10-16T12:46:34.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.957+0000 7f26d37fe640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26b406b070 0x7f26b406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:34.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.958+0000 7f26d37fe640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26b406b070 0x7f26b406d530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26c4035000 tx=0x7f26c4008030 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:34.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:34.961+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26d414a2f0 con 0x7f26d40cded0 2023-10-16T12:46:35.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.180+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26d40d1d00 con 0x7f26b406b070 2023-10-16T12:46:35.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.199+0000 7f26d0ff9640 1 -- 172.21.15.60:0/2379718721 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f26c4031970 con 0x7f26b406b070 2023-10-16T12:46:35.202 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:35.202 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:05", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:35.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26b406b070 msgr2=0x7f26b406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26b406b070 0x7f26b406d530 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26c4035000 tx=0x7f26c4008030 comp rx=0 tx=0).stop 2023-10-16T12:46:35.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 msgr2=0x7f26d40b6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 0x7f26d40b6740 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00c760 tx=0x7f26bc00cc20 comp rx=0 tx=0).stop 2023-10-16T12:46:35.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 shutdown_connections 2023-10-16T12:46:35.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26b406b070 0x7f26b406d530 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.201+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26d415cd70 0x7f26d40a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.202+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26d40cded0 0x7f26d40b6740 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.202+0000 7f26d8df4640 1 --2- 172.21.15.60:0/2379718721 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26d40b8480 0x7f26d41f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.202+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 >> 172.21.15.60:0/2379718721 conn(0x7f26d40cf570 msgr2=0x7f26d415bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:35.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.202+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 shutdown_connections 2023-10-16T12:46:35.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.202+0000 7f26d8df4640 1 -- 172.21.15.60:0/2379718721 wait complete. 2023-10-16T12:46:35.224 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:35:05", "first": "2023-10-16T12:36:00", "last": "2023-10-16T12:46:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:46:35.225 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:35:05', 'first': '2023-10-16T12:36:00', 'last': '2023-10-16T12:46:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:46:35.225 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:46:35.225 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:46:35.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-10-16T12:46:35.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.452+0000 7f1d0a821640 1 Processor -- start 2023-10-16T12:46:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.452+0000 7f1d0a821640 1 -- start start 2023-10-16T12:46:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.453+0000 7f1d0a821640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 0x7f1d040ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d0a821640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040cb030 con 0x7f1d040c9ca0 2023-10-16T12:46:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d0a821640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040c91c0 con 0x7f1d0415ad70 2023-10-16T12:46:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d0a821640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040c94a0 con 0x7f1d0406a4b0 2023-10-16T12:46:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d08dbe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1d040c9ca0 0x7f1d040ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54846/0 (socket says 172.21.15.60:54846) 2023-10-16T12:46:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d08dbe640 1 -- 172.21.15.60:0/3655621123 learned_addr learned my addr 172.21.15.60:0/3655621123 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d037fe640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 0x7f1d040ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.454+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283590841 0 0) 0x7f1d040cb030 con 0x7f1d040c9ca0 2023-10-16T12:46:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf0003520 con 0x7f1d040c9ca0 2023-10-16T12:46:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481211592 0 0) 0x7f1cf0003520 con 0x7f1d040c9ca0 2023-10-16T12:46:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 msgr2=0x7f1d040ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:35.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 0x7f1d040ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 >> v1:172.21.15.144:6789/0 conn(0x7f1d0415ad70 legacy=0x7f1d04069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040c9780 con 0x7f1d040c9ca0 2023-10-16T12:46:35.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1cfc001670 con 0x7f1d040c9ca0 2023-10-16T12:46:35.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 >> v1:172.21.15.60:6789/0 conn(0x7f1d040c9ca0 legacy=0x7f1d040ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1cf0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf00047c0 0x7f1cf0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 0x7f1cf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040c94a0 con 0x7f1cf00039a0 2023-10-16T12:46:35.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf000bab0 con 0x7f1cf00047c0 2023-10-16T12:46:35.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf000bc70 con 0x7f1d0406a4b0 2023-10-16T12:46:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.455+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1cf0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d037fe640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf00047c0 0x7f1cf0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d08dbe640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 0x7f1cf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d03fff640 1 -- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 msgr2=0x7f1cf00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d0406a4b0 0x7f1cf00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d03fff640 1 -- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf00047c0 msgr2=0x7f1cf0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf00047c0 0x7f1cf0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.456+0000 7f1d03fff640 1 -- 172.21.15.60:0/3655621123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf0003520 con 0x7f1cf00039a0 2023-10-16T12:46:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.457+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1cf0003dc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1cec001750 tx=0x7f1cf00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807b5f43c56b6358 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.457+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0052d0 con 0x7f1cf00039a0 2023-10-16T12:46:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.457+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec02e050 con 0x7f1cf00039a0 2023-10-16T12:46:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.457+0000 7f1d02ffd640 1 -- 172.21.15.60:0/3655621123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0035f0 con 0x7f1cf00039a0 2023-10-16T12:46:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d0a821640 1 -- 172.21.15.60:0/3655621123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 msgr2=0x7f1cf0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3655621123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1cf0003dc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1cec001750 tx=0x7f1cf00057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d08dbe640 1 -- 172.21.15.60:0/3655621123 reap_dead start 2023-10-16T12:46:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d0a821640 1 -- 172.21.15.60:0/3655621123 shutdown_connections 2023-10-16T12:46:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d0a821640 1 -- 172.21.15.60:0/3655621123 >> 172.21.15.60:0/3655621123 conn(0x7f1d040ce9b0 msgr2=0x7f1d040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d0a821640 1 -- 172.21.15.60:0/3655621123 shutdown_connections 2023-10-16T12:46:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.458+0000 7f1d0a821640 1 -- 172.21.15.60:0/3655621123 wait complete. 2023-10-16T12:46:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.460+0000 7f1d0a821640 1 Processor -- start 2023-10-16T12:46:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.460+0000 7f1d0a821640 1 -- start start 2023-10-16T12:46:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d0a821640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1d041f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d0a821640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d04158190 0x7f1d040c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d0a821640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d040b0bd0 0x7f1d040acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d0a821640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d041f29c0 con 0x7f1cf00039a0 2023-10-16T12:46:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d0a821640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040c5a80 con 0x7f1d04158190 2023-10-16T12:46:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d0a821640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040c6cf0 con 0x7f1d040b0bd0 2023-10-16T12:46:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d03fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1d041f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d03fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1d041f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42410/0 (socket says 172.21.15.60:42410) 2023-10-16T12:46:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.461+0000 7f1d03fff640 1 -- 172.21.15.60:0/3144224790 learned_addr learned my addr 172.21.15.60:0/3144224790 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d03fff640 1 -- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d040b0bd0 msgr2=0x7f1d040acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d040b0bd0 0x7f1d040acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d03fff640 1 -- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d04158190 msgr2=0x7f1d040c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d04158190 0x7f1d040c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d03fff640 1 -- 172.21.15.60:0/3144224790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d041588f0 con 0x7f1cf00039a0 2023-10-16T12:46:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d03fff640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1d041f2480 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1cec001750 tx=0x7f1cec001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec03c630 con 0x7f1cf00039a0 2023-10-16T12:46:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec03c7b0 con 0x7f1cf00039a0 2023-10-16T12:46:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec03cb60 con 0x7f1cf00039a0 2023-10-16T12:46:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.462+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d040c5cd0 con 0x7f1cf00039a0 2023-10-16T12:46:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.463+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d040c6290 con 0x7f1cf00039a0 2023-10-16T12:46:35.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.464+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d040c6420 con 0x7f1cf00039a0 2023-10-16T12:46:35.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.465+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f1cec03ccc0 con 0x7f1cf00039a0 2023-10-16T12:46:35.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.466+0000 7f1d00ff9640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f1ce406b070 0x7f1ce406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:35.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.466+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1cec0ab930 con 0x7f1cf00039a0 2023-10-16T12:46:35.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.466+0000 7f1d037fe640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f1ce406b070 0x7f1ce406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:35.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.468+0000 7f1d037fe640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f1ce406b070 0x7f1ce406d530 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d040a8990 tx=0x7f1cf4003990 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:35.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.471+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cec076670 con 0x7f1cf00039a0 2023-10-16T12:46:35.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.696+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d0406c6c0 con 0x7f1ce406b070 2023-10-16T12:46:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.937+0000 7f1d00ff9640 1 -- 172.21.15.60:0/3144224790 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f1d0406c6c0 con 0x7f1ce406b070 2023-10-16T12:46:35.939 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-16T12:46:35.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f1ce406b070 msgr2=0x7f1ce406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f1ce406b070 0x7f1ce406d530 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d040a8990 tx=0x7f1cf4003990 comp rx=0 tx=0).stop 2023-10-16T12:46:35.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 msgr2=0x7f1d041f2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:35.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1d041f2480 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1cec001750 tx=0x7f1cec001120 comp rx=0 tx=0).stop 2023-10-16T12:46:35.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 shutdown_connections 2023-10-16T12:46:35.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f1ce406b070 0x7f1ce406d530 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d040b0bd0 0x7f1d040acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d04158190 0x7f1d040c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 --2- 172.21.15.60:0/3144224790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf00039a0 0x7f1d041f2480 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:35.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.940+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 >> 172.21.15.60:0/3144224790 conn(0x7f1d040ce9b0 msgr2=0x7f1d040cdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:35.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.941+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 shutdown_connections 2023-10-16T12:46:35.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:35.941+0000 7f1d0a821640 1 -- 172.21.15.60:0/3144224790 wait complete. 2023-10-16T12:46:35.965 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-16T12:46:35.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:35.966 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T12:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_35_00_UTC 2023-10-16T12:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:36.028 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_35_00_UTC 2023-10-16T12:46:36.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.029 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_35_00_UTC) 2023-10-16T12:46:36.096 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:36.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.097 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:36.168 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:36.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.170 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_37_01_UTC) 2023-10-16T12:46:36.245 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:36.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.246 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_38_01_UTC) 2023-10-16T12:46:36.324 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:36.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.325 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_39_01_UTC) 2023-10-16T12:46:36.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:36.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.404 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T12:46:36.481 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:36.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.482 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:36.559 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:36.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.560 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_42_01_UTC) 2023-10-16T12:46:36.641 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:36.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.642 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T12:46:36.724 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:36.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.725 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T12:46:36.796 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:36.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.798 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:36.877 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:36.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.878 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:36.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:36.959 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-16T12:46:37.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-10-16T12:46:37.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.271+0000 7f447020e640 1 Processor -- start 2023-10-16T12:46:37.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.271+0000 7f447020e640 1 -- start start 2023-10-16T12:46:37.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.273+0000 7f447020e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44680addb0 0x7f44680caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:37.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.273+0000 7f447020e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f446806a4b0 0x7f44681650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:37.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.273+0000 7f447020e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680cb140 con 0x7f44680c9000 2023-10-16T12:46:37.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.273+0000 7f447020e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4468161a50 con 0x7f446806a4b0 2023-10-16T12:46:37.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.273+0000 7f447020e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446806b030 con 0x7f44680addb0 2023-10-16T12:46:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446d7a9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f44680c9000 0x7f4468069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54856/0 (socket says 172.21.15.60:54856) 2023-10-16T12:46:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446d7a9640 1 -- 172.21.15.60:0/858429947 learned_addr learned my addr 172.21.15.60:0/858429947 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446e7ab640 1 --2- 172.21.15.60:0/858429947 >> v2:172.21.15.60:3300/0 conn(0x7f446806a4b0 0x7f44681650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446dfaa640 1 --2- 172.21.15.60:0/858429947 >> v2:172.21.15.144:3300/0 conn(0x7f44680addb0 0x7f44680caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446cfa8640 1 -- 172.21.15.60:0/858429947 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663439548 0 0) 0x7f44680cb140 con 0x7f44680c9000 2023-10-16T12:46:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446e7ab640 1 -- 172.21.15.60:0/858429947 >> v2:172.21.15.144:3300/0 conn(0x7f44680addb0 msgr2=0x7f44680caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446e7ab640 1 --2- 172.21.15.60:0/858429947 >> v2:172.21.15.144:3300/0 conn(0x7f44680addb0 0x7f44680caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446e7ab640 1 -- 172.21.15.60:0/858429947 >> v1:172.21.15.60:6789/0 conn(0x7f44680c9000 legacy=0x7f4468069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.274+0000 7f446e7ab640 1 -- 172.21.15.60:0/858429947 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446806b310 con 0x7f446806a4b0 2023-10-16T12:46:37.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.276+0000 7f446e7ab640 1 --2- 172.21.15.60:0/858429947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f446806a4b0 0x7f44681650d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f44680b15a0 tx=0x7f446002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdd24785799f868 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:37.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.276+0000 7f446cfa8640 1 -- 172.21.15.60:0/858429947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460031830 con 0x7f446806a4b0 2023-10-16T12:46:37.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.276+0000 7f446cfa8640 1 -- 172.21.15.60:0/858429947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460031990 con 0x7f446806a4b0 2023-10-16T12:46:37.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.276+0000 7f446cfa8640 1 -- 172.21.15.60:0/858429947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460031d20 con 0x7f446806a4b0 2023-10-16T12:46:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 -- 172.21.15.60:0/858429947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f446806a4b0 msgr2=0x7f44681650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 --2- 172.21.15.60:0/858429947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f446806a4b0 0x7f44681650d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f44680b15a0 tx=0x7f446002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 -- 172.21.15.60:0/858429947 shutdown_connections 2023-10-16T12:46:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 --2- 172.21.15.60:0/858429947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f446806a4b0 0x7f44681650d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 --2- 172.21.15.60:0/858429947 >> v2:172.21.15.144:3300/0 conn(0x7f44680addb0 0x7f44680caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 -- 172.21.15.60:0/858429947 >> 172.21.15.60:0/858429947 conn(0x7f44680ce9b0 msgr2=0x7f44680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 -- 172.21.15.60:0/858429947 shutdown_connections 2023-10-16T12:46:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.277+0000 7f447020e640 1 -- 172.21.15.60:0/858429947 wait complete. 2023-10-16T12:46:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.279+0000 7f447020e640 1 Processor -- start 2023-10-16T12:46:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.279+0000 7f447020e640 1 -- start start 2023-10-16T12:46:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f447020e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f446806a4b0 0x7f44681f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f447020e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44680addb0 0x7f44681f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f447020e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 0x7f44680ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f447020e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44681f7810 con 0x7f44680c9000 2023-10-16T12:46:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f447020e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680b18d0 con 0x7f446806a4b0 2023-10-16T12:46:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f447020e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44680b1bb0 con 0x7f44680addb0 2023-10-16T12:46:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f446e7ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 0x7f44680ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f446e7ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 0x7f44680ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42424/0 (socket says 172.21.15.60:42424) 2023-10-16T12:46:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f446e7ab640 1 -- 172.21.15.60:0/2472115250 learned_addr learned my addr 172.21.15.60:0/2472115250 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f446dfaa640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f446806a4b0 0x7f44681f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.281+0000 7f446d7a9640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44680addb0 0x7f44681f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f446e7ab640 1 -- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44680addb0 msgr2=0x7f44681f7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f446e7ab640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44680addb0 0x7f44681f7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f446e7ab640 1 -- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f446806a4b0 msgr2=0x7f44681f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f446e7ab640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f446806a4b0 0x7f44681f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f446e7ab640 1 -- 172.21.15.60:0/2472115250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460009400 con 0x7f44680c9000 2023-10-16T12:46:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f446e7ab640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 0x7f44680ae640 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4460009890 tx=0x7f44600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460004230 con 0x7f44680c9000 2023-10-16T12:46:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460004390 con 0x7f44680c9000 2023-10-16T12:46:37.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.282+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446003a8a0 con 0x7f44680c9000 2023-10-16T12:46:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.283+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44680b1e90 con 0x7f44680c9000 2023-10-16T12:46:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.283+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44680b2450 con 0x7f44680c9000 2023-10-16T12:46:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.284+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446804e160 con 0x7f44680c9000 2023-10-16T12:46:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.285+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f446003aa00 con 0x7f44680c9000 2023-10-16T12:46:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.286+0000 7f445e7fc640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f444c06b070 0x7f444c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.286+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f44600ac360 con 0x7f44680c9000 2023-10-16T12:46:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.286+0000 7f446dfaa640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f444c06b070 0x7f444c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.288+0000 7f446dfaa640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f444c06b070 0x7f444c06d530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f44680c55a0 tx=0x7f44640039b0 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.290+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4460076ee0 con 0x7f44680c9000 2023-10-16T12:46:37.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.508+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f446806b8d0 con 0x7f444c06b070 2023-10-16T12:46:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.751+0000 7f445e7fc640 1 -- 172.21.15.60:0/2472115250 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f44640096f0 con 0x7f444c06b070 2023-10-16T12:46:37.753 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-16T12:46:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f444c06b070 msgr2=0x7f444c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f444c06b070 0x7f444c06d530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f44680c55a0 tx=0x7f44640039b0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 msgr2=0x7f44680ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 0x7f44680ae640 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4460009890 tx=0x7f44600046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 shutdown_connections 2023-10-16T12:46:37.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f444c06b070 0x7f444c06d530 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44680addb0 0x7f44681f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44680c9000 0x7f44680ae640 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.754+0000 7f447020e640 1 --2- 172.21.15.60:0/2472115250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f446806a4b0 0x7f44681f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:37.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.755+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 >> 172.21.15.60:0/2472115250 conn(0x7f44680ce9b0 msgr2=0x7f44680cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:37.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.755+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 shutdown_connections 2023-10-16T12:46:37.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:37.755+0000 7f447020e640 1 -- 172.21.15.60:0/2472115250 wait complete. 2023-10-16T12:46:37.780 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-16T12:46:37.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:37.781 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T12:46:37.841 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_35_00_UTC 2023-10-16T12:46:37.841 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:37.842 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:37.842 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:37.842 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:37.842 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:37.842 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:37.843 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:37.843 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:37.843 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:37.843 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:37.843 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:37.844 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_35_00_UTC 2023-10-16T12:46:37.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:37.845 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_35_00_UTC) 2023-10-16T12:46:37.919 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:37.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:37.920 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:37.998 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:37.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:37.999 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_37_01_UTC) 2023-10-16T12:46:38.068 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:38.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.069 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_38_01_UTC) 2023-10-16T12:46:38.143 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:38.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.143 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_39_00_UTC) 2023-10-16T12:46:38.219 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:38.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.220 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T12:46:38.289 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:38.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.290 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:38.359 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:38.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.360 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T12:46:38.432 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:38.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.433 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_43_01_UTC) 2023-10-16T12:46:38.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:38.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.510 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_44_01_UTC) 2023-10-16T12:46:38.579 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:38.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.580 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:38.651 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:38.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.652 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:38.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:38.719 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-16T12:46:38.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-10-16T12:46:38.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.990+0000 7f3e61116640 1 Processor -- start 2023-10-16T12:46:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.990+0000 7f3e61116640 1 -- start start 2023-10-16T12:46:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.991+0000 7f3e61116640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.991+0000 7f3e61116640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e5c0c9ca0 0x7f3e5c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.991+0000 7f3e61116640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c0cb030 con 0x7f3e5c0ca3b0 2023-10-16T12:46:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e61116640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c91c0 con 0x7f3e5c0c9ca0 2023-10-16T12:46:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e61116640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c94a0 con 0x7f3e5c15ad70 2023-10-16T12:46:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e53fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e5c0c9ca0 0x7f3e5c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e53fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e5c0c9ca0 0x7f3e5c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46168/0 (socket says 172.21.15.60:46168) 2023-10-16T12:46:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e53fff640 1 -- 172.21.15.60:0/4060269094 learned_addr learned my addr 172.21.15.60:0/4060269094 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:38.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e5ad9d640 1 --2- 172.21.15.60:0/4060269094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:38.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e53fff640 1 -- 172.21.15.60:0/4060269094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 msgr2=0x7f3e5c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.992+0000 7f3e53fff640 1 --2- 172.21.15.60:0/4060269094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.993+0000 7f3e53fff640 1 -- 172.21.15.60:0/4060269094 >> v1:172.21.15.144:6789/0 conn(0x7f3e5c0ca3b0 legacy=0x7f3e5c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.993+0000 7f3e53fff640 1 -- 172.21.15.60:0/4060269094 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0c9780 con 0x7f3e5c0c9ca0 2023-10-16T12:46:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.994+0000 7f3e53fff640 1 --2- 172.21.15.60:0/4060269094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c06a2e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009870 tx=0x7f3e4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f993d20749385ecf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.994+0000 7f3e5a59c640 1 -- 172.21.15.60:0/4060269094 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44031640 con 0x7f3e5c0c9ca0 2023-10-16T12:46:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.994+0000 7f3e5a59c640 1 -- 172.21.15.60:0/4060269094 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e440317a0 con 0x7f3e5c0c9ca0 2023-10-16T12:46:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.994+0000 7f3e5a59c640 1 -- 172.21.15.60:0/4060269094 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44031b30 con 0x7f3e5c0c9ca0 2023-10-16T12:46:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 -- 172.21.15.60:0/4060269094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0c9ca0 msgr2=0x7f3e5c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 --2- 172.21.15.60:0/4060269094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c06a2e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009870 tx=0x7f3e4402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:46:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 -- 172.21.15.60:0/4060269094 shutdown_connections 2023-10-16T12:46:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 --2- 172.21.15.60:0/4060269094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c06a2e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 --2- 172.21.15.60:0/4060269094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 -- 172.21.15.60:0/4060269094 >> 172.21.15.60:0/4060269094 conn(0x7f3e5c0ce9b0 msgr2=0x7f3e5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 -- 172.21.15.60:0/4060269094 shutdown_connections 2023-10-16T12:46:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.995+0000 7f3e61116640 1 -- 172.21.15.60:0/4060269094 wait complete. 2023-10-16T12:46:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.997+0000 7f3e61116640 1 Processor -- start 2023-10-16T12:46:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.997+0000 7f3e61116640 1 -- start start 2023-10-16T12:46:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e61116640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e61116640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 0x7f3e5c0b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e61116640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c0b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e61116640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c74a0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e61116640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c152090 con 0x7f3e5c0c9ca0 2023-10-16T12:46:39.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e61116640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0c6b70 con 0x7f3e5c15ad70 2023-10-16T12:46:39.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 0x7f3e5c0b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:39.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 0x7f3e5c0b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46178/0 (socket says 172.21.15.60:46178) 2023-10-16T12:46:39.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 -- 172.21.15.60:0/1510947347 learned_addr learned my addr 172.21.15.60:0/1510947347 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:39.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e5b59e640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c0b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 -- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 msgr2=0x7f3e5c0b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e5ad9d640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c0b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 -- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e5c0c9ca0 msgr2=0x7f3e5c151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.998+0000 7f3e53fff640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e53fff640 1 -- 172.21.15.60:0/1510947347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e440094d0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e5b59e640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c0b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e53fff640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 0x7f3e5c0b0e90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3e4402b930 tx=0x7f3e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4403ac60 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4403adc0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44042630 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5c0c6de0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:38.999+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c0b18e0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.001+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c04e160 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.002+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3e4402f050 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.002+0000 7f3e537fe640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3e3c06b070 0x7f3e3c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.003+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6407+0+0 (secure 0 0 0) 0x7f3e440ab5b0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.003+0000 7f3e5ad9d640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3e3c06b070 0x7f3e3c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.004+0000 7f3e5ad9d640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3e3c06b070 0x7f3e3c06d530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.007+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e440760a0 con 0x7f3e5c0ca3b0 2023-10-16T12:46:39.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.286+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e5c0ad4d0 con 0x7f3e3c06b070 2023-10-16T12:46:39.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.513+0000 7f3e537fe640 1 -- 172.21.15.60:0/1510947347 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f3e4c009480 con 0x7f3e3c06b070 2023-10-16T12:46:39.516 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-16T12:46:39.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3e3c06b070 msgr2=0x7f3e3c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:39.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3e3c06b070 0x7f3e3c06d530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c008040 comp rx=0 tx=0).stop 2023-10-16T12:46:39.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 msgr2=0x7f3e5c0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:39.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 0x7f3e5c0b0e90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3e4402b930 tx=0x7f3e440046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 shutdown_connections 2023-10-16T12:46:39.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f3e3c06b070 0x7f3e3c06d530 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e5c15ad70 0x7f3e5c0b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e5c0ca3b0 0x7f3e5c0b0e90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 --2- 172.21.15.60:0/1510947347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e5c0c9ca0 0x7f3e5c151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 >> 172.21.15.60:0/1510947347 conn(0x7f3e5c0ce9b0 msgr2=0x7f3e5c06a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 shutdown_connections 2023-10-16T12:46:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:39.517+0000 7f3e61116640 1 -- 172.21.15.60:0/1510947347 wait complete. 2023-10-16T12:46:39.540 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-16T12:46:39.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.541 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T12:46:39.596 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:39.596 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:39.597 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:39.597 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:39.597 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:39.597 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:39.597 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:39.598 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:39.598 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:39.598 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_01_UTC 2023-10-16T12:46:39.598 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:39.599 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:39.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.600 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_36_01_UTC) 2023-10-16T12:46:39.672 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:39.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.673 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T12:46:39.744 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:39.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.745 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_38_01_UTC) 2023-10-16T12:46:39.855 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:39.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.856 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_39_01_UTC) 2023-10-16T12:46:39.927 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:39.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.928 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_40_01_UTC) 2023-10-16T12:46:39.991 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:39.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:39.992 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:40.042 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:40.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:40.043 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_42_01_UTC) 2023-10-16T12:46:40.098 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:40.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:40.098 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T12:46:40.160 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:40.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:40.161 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_44_01_UTC) 2023-10-16T12:46:40.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_45_01_UTC 2023-10-16T12:46:40.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:40.233 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_45_01_UTC) 2023-10-16T12:46:40.301 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:40.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:40.302 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:40.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:40.372 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-16T12:46:40.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-10-16T12:46:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.671+0000 7f45a5013640 1 Processor -- start 2023-10-16T12:46:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.671+0000 7f45a5013640 1 -- start start 2023-10-16T12:46:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.672+0000 7f45a5013640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45a006c040 0x7f45a006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.672+0000 7f45a5013640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f45a00ad970 0x7f45a015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f45a5013640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a006c960 con 0x7f45a015ebc0 2023-10-16T12:46:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f45a5013640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a015b9d0 con 0x7f45a006c040 2023-10-16T12:46:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f45a5013640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a015bcb0 con 0x7f45a00ad970 2023-10-16T12:46:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f45a4812640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f45a015ebc0 0x7f45a00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46998/0 (socket says 172.21.15.60:46998) 2023-10-16T12:46:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f459ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45a006c040 0x7f45a006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f45a4812640 1 -- 172.21.15.60:0/3735842510 learned_addr learned my addr 172.21.15.60:0/3735842510 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f4597fff640 1 --2- 172.21.15.60:0/3735842510 >> v2:172.21.15.144:3300/0 conn(0x7f45a00ad970 0x7f45a015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.673+0000 7f459f7fe640 1 -- 172.21.15.60:0/3735842510 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597918568 0 0) 0x7f45a006c960 con 0x7f45a015ebc0 2023-10-16T12:46:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.674+0000 7f459f7fe640 1 -- 172.21.15.60:0/3735842510 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f458c003520 con 0x7f45a015ebc0 2023-10-16T12:46:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.674+0000 7f459ffff640 1 -- 172.21.15.60:0/3735842510 >> v2:172.21.15.144:3300/0 conn(0x7f45a00ad970 msgr2=0x7f45a015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.674+0000 7f459ffff640 1 --2- 172.21.15.60:0/3735842510 >> v2:172.21.15.144:3300/0 conn(0x7f45a00ad970 0x7f45a015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.674+0000 7f459ffff640 1 -- 172.21.15.60:0/3735842510 >> v1:172.21.15.60:6789/0 conn(0x7f45a015ebc0 legacy=0x7f45a00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.674+0000 7f459ffff640 1 -- 172.21.15.60:0/3735842510 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a015bf90 con 0x7f45a006c040 2023-10-16T12:46:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.674+0000 7f4597fff640 1 --2- 172.21.15.60:0/3735842510 >> v2:172.21.15.144:3300/0 conn(0x7f45a00ad970 0x7f45a015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.675+0000 7f459ffff640 1 --2- 172.21.15.60:0/3735842510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a006c040 0x7f45a006c420 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f45a00cb060 tx=0x7f459002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b61a4c7e190b3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.675+0000 7f459f7fe640 1 -- 172.21.15.60:0/3735842510 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590031830 con 0x7f45a006c040 2023-10-16T12:46:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.675+0000 7f459f7fe640 1 -- 172.21.15.60:0/3735842510 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4590031990 con 0x7f45a006c040 2023-10-16T12:46:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.675+0000 7f459f7fe640 1 -- 172.21.15.60:0/3735842510 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590031d20 con 0x7f45a006c040 2023-10-16T12:46:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 -- 172.21.15.60:0/3735842510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a006c040 msgr2=0x7f45a006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 --2- 172.21.15.60:0/3735842510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a006c040 0x7f45a006c420 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f45a00cb060 tx=0x7f459002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:46:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 -- 172.21.15.60:0/3735842510 shutdown_connections 2023-10-16T12:46:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 --2- 172.21.15.60:0/3735842510 >> v2:172.21.15.144:3300/0 conn(0x7f45a00ad970 0x7f45a015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:40.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 --2- 172.21.15.60:0/3735842510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a006c040 0x7f45a006c420 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 -- 172.21.15.60:0/3735842510 >> 172.21.15.60:0/3735842510 conn(0x7f45a00ce6a0 msgr2=0x7f45a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 -- 172.21.15.60:0/3735842510 shutdown_connections 2023-10-16T12:46:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.676+0000 7f45a5013640 1 -- 172.21.15.60:0/3735842510 wait complete. 2023-10-16T12:46:40.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.678+0000 7f45a5013640 1 Processor -- start 2023-10-16T12:46:40.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.678+0000 7f45a5013640 1 -- start start 2023-10-16T12:46:40.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.679+0000 7f45a5013640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45a006c040 0x7f45a00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.679+0000 7f45a5013640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45a00ad970 0x7f45a00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.679+0000 7f45a5013640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 0x7f45a00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.679+0000 7f45a5013640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00b1610 con 0x7f45a015ebc0 2023-10-16T12:46:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.679+0000 7f45a5013640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00b12c0 con 0x7f45a006c040 2023-10-16T12:46:40.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.679+0000 7f45a5013640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00c9540 con 0x7f45a00ad970 2023-10-16T12:46:40.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 0x7f45a00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:40.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 0x7f45a00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46194/0 (socket says 172.21.15.60:46194) 2023-10-16T12:46:40.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 -- 172.21.15.60:0/272286108 learned_addr learned my addr 172.21.15.60:0/272286108 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:40.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f4597fff640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45a00ad970 0x7f45a00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:40.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f459ffff640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45a006c040 0x7f45a00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:40.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 -- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45a00ad970 msgr2=0x7f45a00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:40.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45a00ad970 0x7f45a00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:40.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 -- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45a006c040 msgr2=0x7f45a00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:40.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45a006c040 0x7f45a00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:40.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 -- 172.21.15.60:0/272286108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4590009400 con 0x7f45a015ebc0 2023-10-16T12:46:40.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.680+0000 7f45a4812640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 0x7f45a00acdd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f459800cb40 tx=0x7f459800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:40.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.681+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800aca0 con 0x7f45a015ebc0 2023-10-16T12:46:40.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.681+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459800ae00 con 0x7f45a015ebc0 2023-10-16T12:46:40.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.681+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598012620 con 0x7f45a015ebc0 2023-10-16T12:46:40.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.681+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a00c9820 con 0x7f45a015ebc0 2023-10-16T12:46:40.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.681+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a00c9230 con 0x7f45a015ebc0 2023-10-16T12:46:40.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.682+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a006b330 con 0x7f45a015ebc0 2023-10-16T12:46:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.683+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f4598007070 con 0x7f45a015ebc0 2023-10-16T12:46:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.684+0000 7f459d7fa640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f458006b070 0x7f458006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.684+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6471+0+0 (secure 0 0 0) 0x7f4598089d90 con 0x7f45a015ebc0 2023-10-16T12:46:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.685+0000 7f459ffff640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f458006b070 0x7f458006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.686+0000 7f459ffff640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f458006b070 0x7f458006d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4590036000 tx=0x7f459003c040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.689+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4598054820 con 0x7f45a015ebc0 2023-10-16T12:46:40.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:40.924+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f45a00c50b0 con 0x7f458006b070 2023-10-16T12:46:41.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.183+0000 7f459d7fa640 1 -- 172.21.15.60:0/272286108 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f45a00c50b0 con 0x7f458006b070 2023-10-16T12:46:41.186 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-16T12:46:41.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f458006b070 msgr2=0x7f458006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:41.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f458006b070 0x7f458006d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4590036000 tx=0x7f459003c040 comp rx=0 tx=0).stop 2023-10-16T12:46:41.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 msgr2=0x7f45a00acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:41.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 0x7f45a00acdd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f459800cb40 tx=0x7f459800a060 comp rx=0 tx=0).stop 2023-10-16T12:46:41.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 shutdown_connections 2023-10-16T12:46:41.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f458006b070 0x7f458006d530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:41.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45a00ad970 0x7f45a00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:41.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45a015ebc0 0x7f45a00acdd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:41.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 --2- 172.21.15.60:0/272286108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45a006c040 0x7f45a00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:41.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 >> 172.21.15.60:0/272286108 conn(0x7f45a00ce6a0 msgr2=0x7f45a00cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.187+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 shutdown_connections 2023-10-16T12:46:41.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:41.188+0000 7f45a5013640 1 -- 172.21.15.60:0/272286108 wait complete. 2023-10-16T12:46:41.210 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-16T12:46:41.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.211 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T12:46:41.266 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:41.267 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:41.267 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:41.267 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:41.267 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:41.267 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:41.269 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:41.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.270 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:41.342 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:41.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.343 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_37_01_UTC) 2023-10-16T12:46:41.416 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_38_01_UTC 2023-10-16T12:46:41.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.417 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_38_01_UTC) 2023-10-16T12:46:41.491 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:41.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.492 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_39_01_UTC) 2023-10-16T12:46:41.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:41.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.571 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_40_01_UTC) 2023-10-16T12:46:41.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:41.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.644 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_41_01_UTC) 2023-10-16T12:46:41.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:41.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.720 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_42_01_UTC) 2023-10-16T12:46:41.810 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:41.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.811 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T12:46:41.879 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:41.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.880 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T12:46:41.930 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:41.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:41.930 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:42.006 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:42.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:42.007 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_46_01_UTC) 2023-10-16T12:46:42.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:42.056 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-16T12:46:42.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-10-16T12:46:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.375+0000 7feb7e407640 1 Processor -- start 2023-10-16T12:46:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.375+0000 7feb7e407640 1 -- start start 2023-10-16T12:46:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb7e407640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb780caac0 0x7feb780ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:42.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb7e407640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb780cb030 con 0x7feb7815ad70 2023-10-16T12:46:42.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb7e407640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb780c91c0 con 0x7feb7806a4b0 2023-10-16T12:46:42.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb7e407640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb780c94a0 con 0x7feb780caac0 2023-10-16T12:46:42.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb77fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feb7815ad70 0x7feb78069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47006/0 (socket says 172.21.15.60:47006) 2023-10-16T12:46:42.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb77fff640 1 -- 172.21.15.60:0/4096279961 learned_addr learned my addr 172.21.15.60:0/4096279961 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:42.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.377+0000 7feb7c9a4640 1 --2- 172.21.15.60:0/4096279961 >> v2:172.21.15.60:3300/0 conn(0x7feb780caac0 0x7feb780ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.378+0000 7feb7c9a4640 1 -- 172.21.15.60:0/4096279961 >> v1:172.21.15.144:6789/0 conn(0x7feb7806a4b0 legacy=0x7feb780ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.378+0000 7feb777fe640 1 -- 172.21.15.60:0/4096279961 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933756932 0 0) 0x7feb780cb030 con 0x7feb7815ad70 2023-10-16T12:46:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.378+0000 7feb7c9a4640 1 -- 172.21.15.60:0/4096279961 >> v1:172.21.15.60:6789/0 conn(0x7feb7815ad70 legacy=0x7feb78069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.378+0000 7feb7c9a4640 1 -- 172.21.15.60:0/4096279961 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb780c9780 con 0x7feb780caac0 2023-10-16T12:46:42.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.379+0000 7feb7c9a4640 1 --2- 172.21.15.60:0/4096279961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb780caac0 0x7feb780ad4d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7feb700097a0 tx=0x7feb7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1257e65e4bd16f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:42.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.379+0000 7feb777fe640 1 -- 172.21.15.60:0/4096279961 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70031830 con 0x7feb780caac0 2023-10-16T12:46:42.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.379+0000 7feb777fe640 1 -- 172.21.15.60:0/4096279961 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb70031990 con 0x7feb780caac0 2023-10-16T12:46:42.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.379+0000 7feb777fe640 1 -- 172.21.15.60:0/4096279961 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70031d20 con 0x7feb780caac0 2023-10-16T12:46:42.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 -- 172.21.15.60:0/4096279961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb780caac0 msgr2=0x7feb780ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:42.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 --2- 172.21.15.60:0/4096279961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb780caac0 0x7feb780ad4d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7feb700097a0 tx=0x7feb7002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 -- 172.21.15.60:0/4096279961 shutdown_connections 2023-10-16T12:46:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 --2- 172.21.15.60:0/4096279961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb780caac0 0x7feb780ad4d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 -- 172.21.15.60:0/4096279961 >> 172.21.15.60:0/4096279961 conn(0x7feb780ce9b0 msgr2=0x7feb780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 -- 172.21.15.60:0/4096279961 shutdown_connections 2023-10-16T12:46:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.380+0000 7feb7e407640 1 -- 172.21.15.60:0/4096279961 wait complete. 2023-10-16T12:46:42.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.382+0000 7feb7e407640 1 Processor -- start 2023-10-16T12:46:42.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.382+0000 7feb7e407640 1 -- start start 2023-10-16T12:46:42.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7e407640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb7806a4b0 0x7feb780c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7e407640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb780caac0 0x7feb78158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7e407640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 0x7feb780b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7e407640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780af650 con 0x7feb7815ad70 2023-10-16T12:46:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7e407640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78158840 con 0x7feb780caac0 2023-10-16T12:46:42.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7e407640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb780af370 con 0x7feb7806a4b0 2023-10-16T12:46:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7c9a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 0x7feb780b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7c9a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 0x7feb780b1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46208/0 (socket says 172.21.15.60:46208) 2023-10-16T12:46:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.383+0000 7feb7c9a4640 1 -- 172.21.15.60:0/951820358 learned_addr learned my addr 172.21.15.60:0/951820358 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb6f7fe640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb780caac0 0x7feb78158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:42.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb77fff640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb7806a4b0 0x7feb780c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb7c9a4640 1 -- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb7806a4b0 msgr2=0x7feb780c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb7c9a4640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb7806a4b0 0x7feb780c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb7c9a4640 1 -- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb780caac0 msgr2=0x7feb78158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb7c9a4640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb780caac0 0x7feb78158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb7c9a4640 1 -- 172.21.15.60:0/951820358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb70009400 con 0x7feb7815ad70 2023-10-16T12:46:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb77fff640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb7806a4b0 0x7feb780c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.384+0000 7feb7c9a4640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 0x7feb780b1e60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7feb700098c0 tx=0x7feb700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.385+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7003b9f0 con 0x7feb7815ad70 2023-10-16T12:46:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.385+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7003bb50 con 0x7feb7815ad70 2023-10-16T12:46:42.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.385+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb700453f0 con 0x7feb7815ad70 2023-10-16T12:46:42.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.385+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7815de10 con 0x7feb7815ad70 2023-10-16T12:46:42.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.385+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb780aefb0 con 0x7feb7815ad70 2023-10-16T12:46:42.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.386+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7815dc70 con 0x7feb7815ad70 2023-10-16T12:46:42.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.387+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7feb70045550 con 0x7feb7815ad70 2023-10-16T12:46:42.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.388+0000 7feb757fa640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb5806b070 0x7feb5806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:42.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.388+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6535+0+0 (secure 0 0 0) 0x7feb700ac2b0 con 0x7feb7815ad70 2023-10-16T12:46:42.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.389+0000 7feb77fff640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb5806b070 0x7feb5806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:42.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.390+0000 7feb77fff640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb5806b070 0x7feb5806d530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb68007fd0 tx=0x7feb68009040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:42.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.392+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb7815dc70 con 0x7feb7815ad70 2023-10-16T12:46:42.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.611+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7feb780c9ca0 con 0x7feb5806b070 2023-10-16T12:46:42.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.848+0000 7feb757fa640 1 -- 172.21.15.60:0/951820358 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7feb68004b60 con 0x7feb5806b070 2023-10-16T12:46:42.853 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-16T12:46:42.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.855+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb5806b070 msgr2=0x7feb5806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:42.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.855+0000 7feb7e407640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb5806b070 0x7feb5806d530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb68007fd0 tx=0x7feb68009040 comp rx=0 tx=0).stop 2023-10-16T12:46:42.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.855+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 msgr2=0x7feb780b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:42.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.855+0000 7feb7e407640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 0x7feb780b1e60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7feb700098c0 tx=0x7feb700046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 shutdown_connections 2023-10-16T12:46:42.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7feb5806b070 0x7feb5806d530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb7815ad70 0x7feb780b1e60 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb780caac0 0x7feb78158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 --2- 172.21.15.60:0/951820358 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb7806a4b0 0x7feb780c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:42.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 >> 172.21.15.60:0/951820358 conn(0x7feb780ce9b0 msgr2=0x7feb780ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:42.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.856+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 shutdown_connections 2023-10-16T12:46:42.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:42.857+0000 7feb7e407640 1 -- 172.21.15.60:0/951820358 wait complete. 2023-10-16T12:46:42.884 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-16T12:46:42.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:42.885 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T12:46:42.928 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:42.928 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:42.929 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:42.929 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:42.929 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:42.929 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:42.930 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:42.930 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:42.930 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:42.930 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:42.930 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:42.931 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:42.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:42.932 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:43.023 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:43.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.024 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T12:46:43.077 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:43.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.078 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T12:46:43.137 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:43.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.138 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_39_00_UTC) 2023-10-16T12:46:43.195 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:43.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.196 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T12:46:43.261 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:43.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.262 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:43.326 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:43.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.327 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T12:46:43.392 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:43.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.393 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_43_01_UTC) 2023-10-16T12:46:43.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:43.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.464 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_44_01_UTC) 2023-10-16T12:46:43.535 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:43.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.535 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:43.611 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:43.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.612 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_46_01_UTC) 2023-10-16T12:46:43.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:43.688 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-16T12:46:43.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-10-16T12:46:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.987+0000 7fc3fbd6c640 1 Processor -- start 2023-10-16T12:46:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.987+0000 7fc3fbd6c640 1 -- start start 2023-10-16T12:46:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.988+0000 7fc3fbd6c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3f4159e90 0x7fc3f40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.988+0000 7fc3fbd6c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f4070360 con 0x7fc3f406c140 2023-10-16T12:46:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.988+0000 7fc3fbd6c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f40cd1f0 con 0x7fc3f4159ab0 2023-10-16T12:46:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.988+0000 7fc3fbd6c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f415d890 con 0x7fc3f4159e90 2023-10-16T12:46:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.989+0000 7fc3fa309640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc3f406c140 0x7fc3f415dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47022/0 (socket says 172.21.15.60:47022) 2023-10-16T12:46:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.989+0000 7fc3fa309640 1 -- 172.21.15.60:0/1734781194 learned_addr learned my addr 172.21.15.60:0/1734781194 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.989+0000 7fc3f9307640 1 --2- 172.21.15.60:0/1734781194 >> v2:172.21.15.144:3300/0 conn(0x7fc3f4159e90 0x7fc3f40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.989+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160822115 0 0) 0x7fc3f4070360 con 0x7fc3f406c140 2023-10-16T12:46:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.989+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e0003520 con 0x7fc3f406c140 2023-10-16T12:46:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173140752 0 0) 0x7fc3e0003520 con 0x7fc3f406c140 2023-10-16T12:46:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 >> v2:172.21.15.144:3300/0 conn(0x7fc3f4159e90 msgr2=0x7fc3f40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 --2- 172.21.15.60:0/1734781194 >> v2:172.21.15.144:3300/0 conn(0x7fc3f4159e90 0x7fc3f40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 >> v1:172.21.15.144:6789/0 conn(0x7fc3f4159ab0 legacy=0x7fc3f406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40cd4d0 con 0x7fc3f406c140 2023-10-16T12:46:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc3ec0019f0 con 0x7fc3f406c140 2023-10-16T12:46:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 >> v1:172.21.15.60:6789/0 conn(0x7fc3f406c140 legacy=0x7fc3f415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3e00039c0 0x7fc3e0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:43.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3e00047c0 0x7fc3e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:43.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00052a0 0x7fc3e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:43.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f415d890 con 0x7fc3e00039c0 2023-10-16T12:46:43.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e000c1a0 con 0x7fc3e00047c0 2023-10-16T12:46:43.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e000c360 con 0x7fc3e00052a0 2023-10-16T12:46:43.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f9307640 1 --2- 172.21.15.60:0/1734781194 >> v2:172.21.15.144:3300/0 conn(0x7fc3f4159e90 0x7fc3f40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:46:43.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.990+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3e00039c0 0x7fc3e0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:43.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3fa309640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00052a0 0x7fc3e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:43.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3f9b08640 1 -- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00052a0 msgr2=0x7fc3e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:43.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00052a0 0x7fc3e000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:43.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3f9b08640 1 -- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3e00047c0 msgr2=0x7fc3e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:43.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3e00047c0 0x7fc3e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:43.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3f9b08640 1 -- 172.21.15.60:0/1734781194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e0003520 con 0x7fc3e00039c0 2023-10-16T12:46:43.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.991+0000 7fc3fa309640 1 -- 172.21.15.60:0/1734781194 reap_dead start 2023-10-16T12:46:44.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.992+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3e00039c0 0x7fc3e0003de0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0001750 tx=0x7fc3e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fcafb47cf444624 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:44.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.992+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00052d0 con 0x7fc3e00039c0 2023-10-16T12:46:44.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.992+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f002e050 con 0x7fc3e00039c0 2023-10-16T12:46:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.992+0000 7fc3f8b06640 1 -- 172.21.15.60:0/1734781194 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0003830 con 0x7fc3e00039c0 2023-10-16T12:46:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1734781194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3e00039c0 msgr2=0x7fc3e0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3e00039c0 0x7fc3e0003de0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0001750 tx=0x7fc3e00057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1734781194 shutdown_connections 2023-10-16T12:46:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1734781194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3e00039c0 0x7fc3e0003de0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1734781194 >> 172.21.15.60:0/1734781194 conn(0x7fc3f40ce9b0 msgr2=0x7fc3f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1734781194 shutdown_connections 2023-10-16T12:46:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.993+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1734781194 wait complete. 2023-10-16T12:46:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.995+0000 7fc3fbd6c640 1 Processor -- start 2023-10-16T12:46:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.995+0000 7fc3fbd6c640 1 -- start start 2023-10-16T12:46:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fbd6c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00039c0 0x7fc3f41e30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fbd6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f41e2960 0x7fc3f41e3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fbd6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 0x7fc3f40c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fbd6c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41e4420 con 0x7fc3f40c8060 2023-10-16T12:46:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fbd6c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40c7bd0 con 0x7fc3f41e2960 2023-10-16T12:46:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fbd6c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40c6fc0 con 0x7fc3e00039c0 2023-10-16T12:46:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 0x7fc3f40c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 0x7fc3f40c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46214/0 (socket says 172.21.15.60:46214) 2023-10-16T12:46:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 -- 172.21.15.60:0/1522989296 learned_addr learned my addr 172.21.15.60:0/1522989296 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00039c0 0x7fc3f41e30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3f9307640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f41e2960 0x7fc3f41e3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 -- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00039c0 msgr2=0x7fc3f41e30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00039c0 0x7fc3f41e30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 -- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f41e2960 msgr2=0x7fc3f41e3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f41e2960 0x7fc3f41e3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3fa309640 1 -- 172.21.15.60:0/1522989296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40c72a0 con 0x7fc3f40c8060 2023-10-16T12:46:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.996+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00039c0 0x7fc3f41e30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3f9307640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f41e2960 0x7fc3f41e3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3fa309640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 0x7fc3f40c96f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40c24b0 tx=0x7fc3ec00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00b870 con 0x7fc3f40c8060 2023-10-16T12:46:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec00b9f0 con 0x7fc3f40c8060 2023-10-16T12:46:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00bda0 con 0x7fc3f40c8060 2023-10-16T12:46:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f40c7580 con 0x7fc3f40c8060 2023-10-16T12:46:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.997+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f41e2de0 con 0x7fc3f40c8060 2023-10-16T12:46:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.998+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f404e160 con 0x7fc3f40c8060 2023-10-16T12:46:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:43.999+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc3ec007070 con 0x7fc3f40c8060 2023-10-16T12:46:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.000+0000 7fc3ea7fc640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc3d806b0a0 0x7fc3d806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.000+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6535+0+0 (secure 0 0 0) 0x7fc3ec08a0b0 con 0x7fc3f40c8060 2023-10-16T12:46:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.001+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc3d806b0a0 0x7fc3d806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.002+0000 7fc3f9b08640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc3d806b0a0 0x7fc3d806d560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0033000 tx=0x7fc3f0009030 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.005+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3ec0549f0 con 0x7fc3f40c8060 2023-10-16T12:46:44.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.227+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3f40cda90 con 0x7fc3d806b0a0 2023-10-16T12:46:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.464+0000 7fc3ea7fc640 1 -- 172.21.15.60:0/1522989296 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc3f0007040 con 0x7fc3d806b0a0 2023-10-16T12:46:44.467 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-16T12:46:44.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc3d806b0a0 msgr2=0x7fc3d806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:44.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc3d806b0a0 0x7fc3d806d560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0033000 tx=0x7fc3f0009030 comp rx=0 tx=0).stop 2023-10-16T12:46:44.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 msgr2=0x7fc3f40c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:44.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 0x7fc3f40c96f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40c24b0 tx=0x7fc3ec00b0d0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 shutdown_connections 2023-10-16T12:46:44.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fc3d806b0a0 0x7fc3d806d560 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f40c8060 0x7fc3f40c96f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f41e2960 0x7fc3f41e3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 --2- 172.21.15.60:0/1522989296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3e00039c0 0x7fc3f41e30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:44.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.468+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 >> 172.21.15.60:0/1522989296 conn(0x7fc3f40ce9b0 msgr2=0x7fc3f4158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:44.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.469+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 shutdown_connections 2023-10-16T12:46:44.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:44.469+0000 7fc3fbd6c640 1 -- 172.21.15.60:0/1522989296 wait complete. 2023-10-16T12:46:44.490 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-16T12:46:44.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.491 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T12:46:44.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:44.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:44.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:44.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:44.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:44.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:44.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:44.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:44.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:44.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:44.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:44.550 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:44.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.551 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_36_01_UTC) 2023-10-16T12:46:44.618 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:44.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.619 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T12:46:44.687 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:44.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.688 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T12:46:44.755 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:44.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.756 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_39_01_UTC) 2023-10-16T12:46:44.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:44.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.854 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T12:46:44.913 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:44.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.914 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_41_01_UTC) 2023-10-16T12:46:44.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:44.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:44.963 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T12:46:45.013 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:45.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:45.013 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_43_01_UTC) 2023-10-16T12:46:45.067 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:45.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:45.068 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T12:46:45.143 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:45.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:45.144 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:45.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:45.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:45.206 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:45.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:45.254 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-16T12:46:45.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-10-16T12:46:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.576+0000 7f2e7776c640 1 Processor -- start 2023-10-16T12:46:45.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.577+0000 7f2e7776c640 1 -- start start 2023-10-16T12:46:45.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.578+0000 7f2e7776c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e7015ad70 0x7f2e70069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.578+0000 7f2e7776c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e700caac0 0x7f2e7006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.578+0000 7f2e7776c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e700cb030 con 0x7f2e700c9ca0 2023-10-16T12:46:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.578+0000 7f2e7776c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e700c91c0 con 0x7f2e700caac0 2023-10-16T12:46:45.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.578+0000 7f2e7776c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700c94a0 con 0x7f2e7015ad70 2023-10-16T12:46:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e74d07640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2e700c9ca0 0x7f2e700ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47028/0 (socket says 172.21.15.60:47028) 2023-10-16T12:46:45.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e74d07640 1 -- 172.21.15.60:0/1384230592 learned_addr learned my addr 172.21.15.60:0/1384230592 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e75d09640 1 --2- 172.21.15.60:0/1384230592 >> v2:172.21.15.60:3300/0 conn(0x7f2e700caac0 0x7f2e7006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e75508640 1 --2- 172.21.15.60:0/1384230592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e7015ad70 0x7f2e70069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e67fff640 1 -- 172.21.15.60:0/1384230592 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108820250 0 0) 0x7f2e700cb030 con 0x7f2e700c9ca0 2023-10-16T12:46:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e67fff640 1 -- 172.21.15.60:0/1384230592 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c003520 con 0x7f2e700c9ca0 2023-10-16T12:46:45.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e75d09640 1 -- 172.21.15.60:0/1384230592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e7015ad70 msgr2=0x7f2e70069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:45.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e75d09640 1 --2- 172.21.15.60:0/1384230592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e7015ad70 0x7f2e70069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:45.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e75d09640 1 -- 172.21.15.60:0/1384230592 >> v1:172.21.15.60:6789/0 conn(0x7f2e700c9ca0 legacy=0x7f2e700ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:45.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.579+0000 7f2e75d09640 1 -- 172.21.15.60:0/1384230592 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e700c9780 con 0x7f2e700caac0 2023-10-16T12:46:45.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.581+0000 7f2e75d09640 1 --2- 172.21.15.60:0/1384230592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e7006a2e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e6802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bec6015877f2502 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:45.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.581+0000 7f2e67fff640 1 -- 172.21.15.60:0/1384230592 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6802cc50 con 0x7f2e700caac0 2023-10-16T12:46:45.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.581+0000 7f2e67fff640 1 -- 172.21.15.60:0/1384230592 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6802cdb0 con 0x7f2e700caac0 2023-10-16T12:46:45.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.581+0000 7f2e67fff640 1 -- 172.21.15.60:0/1384230592 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68034640 con 0x7f2e700caac0 2023-10-16T12:46:45.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 -- 172.21.15.60:0/1384230592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 msgr2=0x7f2e7006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:45.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 --2- 172.21.15.60:0/1384230592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e7006a2e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d90 tx=0x7f2e6802bea0 comp rx=0 tx=0).stop 2023-10-16T12:46:45.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 -- 172.21.15.60:0/1384230592 shutdown_connections 2023-10-16T12:46:45.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 --2- 172.21.15.60:0/1384230592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e7006a2e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:45.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 --2- 172.21.15.60:0/1384230592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e7015ad70 0x7f2e70069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:45.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 -- 172.21.15.60:0/1384230592 >> 172.21.15.60:0/1384230592 conn(0x7f2e700ce9b0 msgr2=0x7f2e700cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:45.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 -- 172.21.15.60:0/1384230592 shutdown_connections 2023-10-16T12:46:45.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.582+0000 7f2e7776c640 1 -- 172.21.15.60:0/1384230592 wait complete. 2023-10-16T12:46:45.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.583+0000 7f2e7776c640 1 Processor -- start 2023-10-16T12:46:45.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.584+0000 7f2e7776c640 1 -- start start 2023-10-16T12:46:45.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e7776c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e700c9ca0 0x7f2e700b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:45.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e7776c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e701589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:45.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e7776c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7015ad70 0x7f2e700c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:45.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e7776c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700b38f0 con 0x7f2e700caac0 2023-10-16T12:46:45.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e7776c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700ac030 con 0x7f2e7015ad70 2023-10-16T12:46:45.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e7776c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700ac310 con 0x7f2e700c9ca0 2023-10-16T12:46:45.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e701589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e701589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46234/0 (socket says 172.21.15.60:46234) 2023-10-16T12:46:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 -- 172.21.15.60:0/4041364668 learned_addr learned my addr 172.21.15.60:0/4041364668 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e75508640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e700c9ca0 0x7f2e700b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e75d09640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7015ad70 0x7f2e700c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 -- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e700c9ca0 msgr2=0x7f2e700b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e700c9ca0 0x7f2e700b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 -- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7015ad70 msgr2=0x7f2e700c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7015ad70 0x7f2e700c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.585+0000 7f2e74d07640 1 -- 172.21.15.60:0/4041364668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009a70 con 0x7f2e700caac0 2023-10-16T12:46:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.586+0000 7f2e74d07640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e701589f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000c430 tx=0x7f2e6000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.586+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60005ce0 con 0x7f2e700caac0 2023-10-16T12:46:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.586+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60005e40 con 0x7f2e700caac0 2023-10-16T12:46:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.586+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e700ac5f0 con 0x7f2e700caac0 2023-10-16T12:46:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.586+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e600072f0 con 0x7f2e700caac0 2023-10-16T12:46:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.586+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e700acbb0 con 0x7f2e700caac0 2023-10-16T12:46:45.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.587+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7004e160 con 0x7f2e700caac0 2023-10-16T12:46:45.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.589+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2e60007450 con 0x7f2e700caac0 2023-10-16T12:46:45.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.590+0000 7f2e65ffb640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f2e5406b070 0x7f2e5406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:45.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.590+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2e600893f0 con 0x7f2e700caac0 2023-10-16T12:46:45.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.590+0000 7f2e75508640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f2e5406b070 0x7f2e5406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:45.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.592+0000 7f2e75508640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f2e5406b070 0x7f2e5406d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c006fd0 tx=0x7f2e6c008040 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:45.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.595+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e60053e60 con 0x7f2e700caac0 2023-10-16T12:46:45.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:45.836+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e700ad4d0 con 0x7f2e5406b070 2023-10-16T12:46:46.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.037+0000 7f2e65ffb640 1 -- 172.21.15.60:0/4041364668 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f2e6c00f860 con 0x7f2e5406b070 2023-10-16T12:46:46.040 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-16T12:46:46.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f2e5406b070 msgr2=0x7f2e5406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f2e5406b070 0x7f2e5406d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c006fd0 tx=0x7f2e6c008040 comp rx=0 tx=0).stop 2023-10-16T12:46:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 msgr2=0x7f2e701589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e701589f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000c430 tx=0x7f2e6000c8f0 comp rx=0 tx=0).stop 2023-10-16T12:46:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 shutdown_connections 2023-10-16T12:46:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f2e5406b070 0x7f2e5406d530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:46.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7015ad70 0x7f2e700c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:46.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e700caac0 0x7f2e701589f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:46.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 --2- 172.21.15.60:0/4041364668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e700c9ca0 0x7f2e700b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:46.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 >> 172.21.15.60:0/4041364668 conn(0x7f2e700ce9b0 msgr2=0x7f2e700af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:46.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 shutdown_connections 2023-10-16T12:46:46.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:46.041+0000 7f2e7776c640 1 -- 172.21.15.60:0/4041364668 wait complete. 2023-10-16T12:46:46.063 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-16T12:46:46.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.064 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T12:46:46.114 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:46.115 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:46.115 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:46.115 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:46.115 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:46.115 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:46.116 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:46.116 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:46.116 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:46.116 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_01_UTC 2023-10-16T12:46:46.116 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:46.117 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:46.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.118 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:46.184 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:46.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.185 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T12:46:46.255 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:46.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.256 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T12:46:46.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_39_01_UTC 2023-10-16T12:46:46.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.330 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_39_01_UTC) 2023-10-16T12:46:46.398 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:46.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.399 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_40_01_UTC) 2023-10-16T12:46:46.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_41_01_UTC 2023-10-16T12:46:46.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.464 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_41_01_UTC) 2023-10-16T12:46:46.529 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:46.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.530 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T12:46:46.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:46.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.596 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T12:46:46.666 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:46.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.667 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T12:46:46.737 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_45_01_UTC 2023-10-16T12:46:46.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.738 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_45_01_UTC) 2023-10-16T12:46:46.809 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:46.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.810 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:46.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:46.896 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-16T12:46:46.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-10-16T12:46:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.224+0000 7fda26d33640 1 Processor -- start 2023-10-16T12:46:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.224+0000 7fda26d33640 1 -- start start 2023-10-16T12:46:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.226+0000 7fda26d33640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda2006ade0 0x7fda2006b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.226+0000 7fda26d33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.226+0000 7fda26d33640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200cdc30 con 0x7fda20071b30 2023-10-16T12:46:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.226+0000 7fda26d33640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2006b780 con 0x7fda2006ade0 2023-10-16T12:46:47.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.226+0000 7fda26d33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200ca1d0 con 0x7fda200c9800 2023-10-16T12:46:47.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.227+0000 7fda1ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda2006ade0 0x7fda2006b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.227+0000 7fda1ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda2006ade0 0x7fda2006b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:48422/0 (socket says 172.21.15.60:48422) 2023-10-16T12:46:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.227+0000 7fda1ffff640 1 -- 172.21.15.60:0/3369571616 learned_addr learned my addr 172.21.15.60:0/3369571616 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.228+0000 7fda1ffff640 1 -- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 msgr2=0x7fda200c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.228+0000 7fda1ffff640 1 --2- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.228+0000 7fda1ffff640 1 -- 172.21.15.60:0/3369571616 >> v1:172.21.15.144:6789/0 conn(0x7fda20071b30 legacy=0x7fda2006fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.228+0000 7fda1ffff640 1 -- 172.21.15.60:0/3369571616 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200ccd80 con 0x7fda2006ade0 2023-10-16T12:46:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.229+0000 7fda1ffff640 1 --2- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda2006ade0 0x7fda2006b240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00bb00 tx=0x7fda0c02dea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb6f24728dcaf4a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.230+0000 7fda1f7fe640 1 -- 172.21.15.60:0/3369571616 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c02ec50 con 0x7fda2006ade0 2023-10-16T12:46:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.230+0000 7fda1f7fe640 1 -- 172.21.15.60:0/3369571616 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c032070 con 0x7fda2006ade0 2023-10-16T12:46:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.230+0000 7fda1f7fe640 1 -- 172.21.15.60:0/3369571616 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c038400 con 0x7fda2006ade0 2023-10-16T12:46:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 -- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda2006ade0 msgr2=0x7fda2006b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 --2- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda2006ade0 0x7fda2006b240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00bb00 tx=0x7fda0c02dea0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 -- 172.21.15.60:0/3369571616 shutdown_connections 2023-10-16T12:46:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 --2- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 --2- 172.21.15.60:0/3369571616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda2006ade0 0x7fda2006b240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 -- 172.21.15.60:0/3369571616 >> 172.21.15.60:0/3369571616 conn(0x7fda2006bc10 msgr2=0x7fda2006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 -- 172.21.15.60:0/3369571616 shutdown_connections 2023-10-16T12:46:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.231+0000 7fda26d33640 1 -- 172.21.15.60:0/3369571616 wait complete. 2023-10-16T12:46:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.233+0000 7fda26d33640 1 Processor -- start 2023-10-16T12:46:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.233+0000 7fda26d33640 1 -- start start 2023-10-16T12:46:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda26d33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 0x7fda20157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:47.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda26d33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda20071b30 0x7fda20156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda26d33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda26d33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b31d0 con 0x7fda2006ade0 2023-10-16T12:46:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda26d33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201571b0 con 0x7fda20071b30 2023-10-16T12:46:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda26d33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2015a530 con 0x7fda200c9800 2023-10-16T12:46:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.234+0000 7fda24acf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 0x7fda20157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 0x7fda20157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46248/0 (socket says 172.21.15.60:46248) 2023-10-16T12:46:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 -- 172.21.15.60:0/2122667904 learned_addr learned my addr 172.21.15.60:0/2122667904 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda1ffff640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda20071b30 0x7fda20156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda252d0640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 -- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 msgr2=0x7fda200b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 -- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda20071b30 msgr2=0x7fda20156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda20071b30 0x7fda20156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 -- 172.21.15.60:0/2122667904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c00b760 con 0x7fda2006ade0 2023-10-16T12:46:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda252d0640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.235+0000 7fda24acf640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 0x7fda20157c50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fda1400c430 tx=0x7fda1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.236+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14011790 con 0x7fda2006ade0 2023-10-16T12:46:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.236+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda140118f0 con 0x7fda2006ade0 2023-10-16T12:46:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.236+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14011ca0 con 0x7fda2006ade0 2023-10-16T12:46:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.236+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200b2930 con 0x7fda2006ade0 2023-10-16T12:46:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.236+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200b2ef0 con 0x7fda2006ade0 2023-10-16T12:46:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.238+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2004e160 con 0x7fda2006ade0 2023-10-16T12:46:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.239+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fda1401c460 con 0x7fda2006ade0 2023-10-16T12:46:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.240+0000 7fda1d7fa640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fda0406b070 0x7fda0406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.241+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6631+0+0 (secure 0 0 0) 0x7fda14089c00 con 0x7fda2006ade0 2023-10-16T12:46:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.241+0000 7fda1ffff640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fda0406b070 0x7fda0406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.243+0000 7fda1ffff640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fda0406b070 0x7fda0406d530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda0c039000 tx=0x7fda0c034000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.246+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda14090080 con 0x7fda2006ade0 2023-10-16T12:46:47.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.473+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fda200cd300 con 0x7fda0406b070 2023-10-16T12:46:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.718+0000 7fda1d7fa640 1 -- 172.21.15.60:0/2122667904 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fda0c02ec70 con 0x7fda0406b070 2023-10-16T12:46:47.722 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-16T12:46:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.721+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fda0406b070 msgr2=0x7fda0406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.721+0000 7fda26d33640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fda0406b070 0x7fda0406d530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda0c039000 tx=0x7fda0c034000 comp rx=0 tx=0).stop 2023-10-16T12:46:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.721+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 msgr2=0x7fda20157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.721+0000 7fda26d33640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 0x7fda20157c50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fda1400c430 tx=0x7fda1400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 shutdown_connections 2023-10-16T12:46:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fda0406b070 0x7fda0406d530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda200c9800 0x7fda200b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda20071b30 0x7fda20156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 --2- 172.21.15.60:0/2122667904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda2006ade0 0x7fda20157c50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 >> 172.21.15.60:0/2122667904 conn(0x7fda2006bc10 msgr2=0x7fda20158870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 shutdown_connections 2023-10-16T12:46:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:47.722+0000 7fda26d33640 1 -- 172.21.15.60:0/2122667904 wait complete. 2023-10-16T12:46:47.747 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-16T12:46:47.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:47.748 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T12:46:47.804 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:47.804 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:47.805 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:47.805 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:47.805 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:47.805 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:47.806 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:47.806 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:47.806 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:47.806 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:47.807 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:47.808 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:47.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:47.808 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:47.911 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:47.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:47.913 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T12:46:47.961 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:47.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:47.962 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T12:46:48.013 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:48.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.014 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_39_00_UTC) 2023-10-16T12:46:48.144 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_40_01_UTC 2023-10-16T12:46:48.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.145 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_40_01_UTC) 2023-10-16T12:46:48.198 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:48.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.199 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:48.248 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:48.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.249 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T12:46:48.306 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:48.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.307 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T12:46:48.379 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:48.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.380 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T12:46:48.451 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:48.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.451 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:48.530 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_46_01_UTC 2023-10-16T12:46:48.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.531 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_46_01_UTC) 2023-10-16T12:46:48.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:48.614 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-16T12:46:48.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-10-16T12:46:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.925+0000 7f270357c640 1 Processor -- start 2023-10-16T12:46:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.926+0000 7f270357c640 1 -- start start 2023-10-16T12:46:48.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f270357c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26fc159e90 0x7f26fc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:48.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f270357c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc070360 con 0x7f26fc159ab0 2023-10-16T12:46:48.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f270357c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0cd1f0 con 0x7f26fc06c140 2023-10-16T12:46:48.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f270357c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc15d890 con 0x7f26fc159e90 2023-10-16T12:46:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f2700b17640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26fc159e90 0x7f26fc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f2701318640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f26fc159ab0 0x7f26fc06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34908/0 (socket says 172.21.15.60:34908) 2023-10-16T12:46:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.927+0000 7f2701318640 1 -- 172.21.15.60:0/2191625327 learned_addr learned my addr 172.21.15.60:0/2191625327 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.928+0000 7f26f3fff640 1 -- 172.21.15.60:0/2191625327 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365564296 0 0) 0x7f26fc070360 con 0x7f26fc159ab0 2023-10-16T12:46:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.928+0000 7f2700b17640 1 -- 172.21.15.60:0/2191625327 >> v1:172.21.15.144:6789/0 conn(0x7f26fc06c140 legacy=0x7f26fc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.928+0000 7f2700b17640 1 -- 172.21.15.60:0/2191625327 >> v1:172.21.15.60:6789/0 conn(0x7f26fc159ab0 legacy=0x7f26fc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.928+0000 7f2700b17640 1 -- 172.21.15.60:0/2191625327 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc0cd4d0 con 0x7f26fc159e90 2023-10-16T12:46:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.929+0000 7f2700b17640 1 --2- 172.21.15.60:0/2191625327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159e90 0x7f26fc0ce370 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0097a0 tx=0x7f26ec02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b324176941e01b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.929+0000 7f26f3fff640 1 -- 172.21.15.60:0/2191625327 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec02f050 con 0x7f26fc159e90 2023-10-16T12:46:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.929+0000 7f26f3fff640 1 -- 172.21.15.60:0/2191625327 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec031700 con 0x7f26fc159e90 2023-10-16T12:46:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.929+0000 7f26f3fff640 1 -- 172.21.15.60:0/2191625327 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec031a90 con 0x7f26fc159e90 2023-10-16T12:46:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 -- 172.21.15.60:0/2191625327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159e90 msgr2=0x7f26fc0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 --2- 172.21.15.60:0/2191625327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159e90 0x7f26fc0ce370 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0097a0 tx=0x7f26ec02b7a0 comp rx=0 tx=0).stop 2023-10-16T12:46:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 -- 172.21.15.60:0/2191625327 shutdown_connections 2023-10-16T12:46:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 --2- 172.21.15.60:0/2191625327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159e90 0x7f26fc0ce370 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 -- 172.21.15.60:0/2191625327 >> 172.21.15.60:0/2191625327 conn(0x7f26fc0ce9b0 msgr2=0x7f26fc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 -- 172.21.15.60:0/2191625327 shutdown_connections 2023-10-16T12:46:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.930+0000 7f270357c640 1 -- 172.21.15.60:0/2191625327 wait complete. 2023-10-16T12:46:48.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.932+0000 7f270357c640 1 Processor -- start 2023-10-16T12:46:48.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.932+0000 7f270357c640 1 -- start start 2023-10-16T12:46:48.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f270357c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26fc06c140 0x7f26fc1e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:48.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f270357c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 0x7f26fc14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:48.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f270357c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26fc159e90 0x7f26fc15cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:48.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f270357c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc15c710 con 0x7f26fc159ab0 2023-10-16T12:46:48.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f270357c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc15ba90 con 0x7f26fc06c140 2023-10-16T12:46:48.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f270357c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc15bd70 con 0x7f26fc159e90 2023-10-16T12:46:48.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.933+0000 7f2700b17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 0x7f26fc14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 0x7f26fc14fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37488/0 (socket says 172.21.15.60:37488) 2023-10-16T12:46:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 -- 172.21.15.60:0/2301703229 learned_addr learned my addr 172.21.15.60:0/2301703229 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2701318640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26fc06c140 0x7f26fc1e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2701b19640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26fc159e90 0x7f26fc15cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 -- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26fc159e90 msgr2=0x7f26fc15cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:48.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26fc159e90 0x7f26fc15cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:48.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 -- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26fc06c140 msgr2=0x7f26fc1e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:48.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26fc06c140 0x7f26fc1e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:48.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.934+0000 7f2700b17640 1 -- 172.21.15.60:0/2301703229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec009400 con 0x7f26fc159ab0 2023-10-16T12:46:48.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.936+0000 7f2700b17640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 0x7f26fc14fa10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036040 tx=0x7f26ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:48.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.940+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec033cf0 con 0x7f26fc159ab0 2023-10-16T12:46:48.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.940+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec033e70 con 0x7f26fc159ab0 2023-10-16T12:46:48.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.940+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec03e6a0 con 0x7f26fc159ab0 2023-10-16T12:46:48.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.940+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc15c050 con 0x7f26fc159ab0 2023-10-16T12:46:48.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.941+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc1e3db0 con 0x7f26fc159ab0 2023-10-16T12:46:48.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.943+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc15c850 con 0x7f26fc159ab0 2023-10-16T12:46:48.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.946+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f26ec03a040 con 0x7f26fc159ab0 2023-10-16T12:46:48.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.946+0000 7f26f1ffb640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26e006b070 0x7f26e006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:48.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.947+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6471+0+0 (secure 0 0 0) 0x7f26ec0ac1f0 con 0x7f26fc159ab0 2023-10-16T12:46:48.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.947+0000 7f2701318640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26e006b070 0x7f26e006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:48.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.949+0000 7f2701318640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26e006b070 0x7f26e006d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26f80012e0 tx=0x7f26f8007ef0 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:48.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:48.950+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26ec076da0 con 0x7f26fc159ab0 2023-10-16T12:46:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.170+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f26fc0cda90 con 0x7f26e006b070 2023-10-16T12:46:49.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.409+0000 7f26f1ffb640 1 -- 172.21.15.60:0/2301703229 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f26f8005c70 con 0x7f26e006b070 2023-10-16T12:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-16T12:46:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26e006b070 msgr2=0x7f26e006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:49.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26e006b070 0x7f26e006d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26f80012e0 tx=0x7f26f8007ef0 comp rx=0 tx=0).stop 2023-10-16T12:46:49.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 msgr2=0x7f26fc14fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:49.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 0x7f26fc14fa10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036040 tx=0x7f26ec0046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:49.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 shutdown_connections 2023-10-16T12:46:49.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f26e006b070 0x7f26e006d530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:49.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26fc159e90 0x7f26fc15cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:49.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26fc159ab0 0x7f26fc14fa10 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036040 tx=0x7f26ec0046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:49.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 --2- 172.21.15.60:0/2301703229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26fc06c140 0x7f26fc1e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:49.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.412+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 >> 172.21.15.60:0/2301703229 conn(0x7f26fc0ce9b0 msgr2=0x7f26fc06c520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:49.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.413+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 shutdown_connections 2023-10-16T12:46:49.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:49.413+0000 7f270357c640 1 -- 172.21.15.60:0/2301703229 wait complete. 2023-10-16T12:46:49.435 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-16T12:46:49.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.435 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T12:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:49.496 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_36_01_UTC 2023-10-16T12:46:49.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.497 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_36_01_UTC) 2023-10-16T12:46:49.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_37_01_UTC 2023-10-16T12:46:49.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.571 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_37_01_UTC) 2023-10-16T12:46:49.646 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:49.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.647 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T12:46:49.752 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:49.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.754 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_39_00_UTC) 2023-10-16T12:46:49.809 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:49.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.810 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T12:46:49.865 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:49.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.866 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:49.936 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T12:46:49.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.938 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T12:46:49.992 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T12:46:49.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:49.993 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T12:46:50.052 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_44_01_UTC 2023-10-16T12:46:50.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:50.053 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_44_01_UTC) 2023-10-16T12:46:50.118 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:50.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:50.119 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:50.208 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:50.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:50.209 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:50.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:50.262 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-16T12:46:50.309 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-10-16T12:46:50.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.579+0000 7fe75535a640 1 Processor -- start 2023-10-16T12:46:50.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.579+0000 7fe75535a640 1 -- start start 2023-10-16T12:46:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.581+0000 7fe75535a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7500addb0 0x7fe7500caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.581+0000 7fe75535a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7500cb140 con 0x7fe7500c9000 2023-10-16T12:46:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.581+0000 7fe75535a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75006b030 con 0x7fe75006a4b0 2023-10-16T12:46:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.581+0000 7fe75535a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe750161a50 con 0x7fe7500addb0 2023-10-16T12:46:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74e7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe7500c9000 0x7fe750069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34918/0 (socket says 172.21.15.60:34918) 2023-10-16T12:46:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74e7fc640 1 -- 172.21.15.60:0/2922068915 learned_addr learned my addr 172.21.15.60:0/2922068915 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74effd640 1 --2- 172.21.15.60:0/2922068915 >> v2:172.21.15.60:3300/0 conn(0x7fe7500addb0 0x7fe7500caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:50.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74dffb640 1 -- 172.21.15.60:0/2922068915 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111221550 0 0) 0x7fe7500cb140 con 0x7fe7500c9000 2023-10-16T12:46:50.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74dffb640 1 -- 172.21.15.60:0/2922068915 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73c003520 con 0x7fe7500c9000 2023-10-16T12:46:50.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74effd640 1 -- 172.21.15.60:0/2922068915 >> v1:172.21.15.144:6789/0 conn(0x7fe75006a4b0 legacy=0x7fe7501650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:50.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74effd640 1 -- 172.21.15.60:0/2922068915 >> v1:172.21.15.60:6789/0 conn(0x7fe7500c9000 legacy=0x7fe750069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:50.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.582+0000 7fe74effd640 1 -- 172.21.15.60:0/2922068915 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75006b310 con 0x7fe7500addb0 2023-10-16T12:46:50.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.583+0000 7fe74effd640 1 --2- 172.21.15.60:0/2922068915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500addb0 0x7fe7500caa80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe740009d90 tx=0x7fe74002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fbbd704cb8d06a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:50.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe74dffb640 1 -- 172.21.15.60:0/2922068915 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74002cbd0 con 0x7fe7500addb0 2023-10-16T12:46:50.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe74dffb640 1 -- 172.21.15.60:0/2922068915 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74002cd30 con 0x7fe7500addb0 2023-10-16T12:46:50.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe74dffb640 1 -- 172.21.15.60:0/2922068915 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74002cea0 con 0x7fe7500addb0 2023-10-16T12:46:50.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 -- 172.21.15.60:0/2922068915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500addb0 msgr2=0x7fe7500caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:50.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 --2- 172.21.15.60:0/2922068915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500addb0 0x7fe7500caa80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe740009d90 tx=0x7fe74002be20 comp rx=0 tx=0).stop 2023-10-16T12:46:50.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 -- 172.21.15.60:0/2922068915 shutdown_connections 2023-10-16T12:46:50.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 --2- 172.21.15.60:0/2922068915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500addb0 0x7fe7500caa80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:50.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 -- 172.21.15.60:0/2922068915 >> 172.21.15.60:0/2922068915 conn(0x7fe7500ce9b0 msgr2=0x7fe7500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:50.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 -- 172.21.15.60:0/2922068915 shutdown_connections 2023-10-16T12:46:50.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.584+0000 7fe75535a640 1 -- 172.21.15.60:0/2922068915 wait complete. 2023-10-16T12:46:50.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.586+0000 7fe75535a640 1 Processor -- start 2023-10-16T12:46:50.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.586+0000 7fe75535a640 1 -- start start 2023-10-16T12:46:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.587+0000 7fe75535a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe75006a4b0 0x7fe75006fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.587+0000 7fe75535a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500addb0 0x7fe7500b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.587+0000 7fe75535a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 0x7fe7500b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.587+0000 7fe75535a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500703a0 con 0x7fe7500c9000 2023-10-16T12:46:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.587+0000 7fe75535a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe750070680 con 0x7fe75006a4b0 2023-10-16T12:46:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.587+0000 7fe75535a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe750070960 con 0x7fe7500addb0 2023-10-16T12:46:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 0x7fe7500b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 0x7fe7500b3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37504/0 (socket says 172.21.15.60:37504) 2023-10-16T12:46:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 -- 172.21.15.60:0/659256538 learned_addr learned my addr 172.21.15.60:0/659256538 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74effd640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe75006a4b0 0x7fe75006fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 -- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500addb0 msgr2=0x7fe7500b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500addb0 0x7fe7500b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 -- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe75006a4b0 msgr2=0x7fe75006fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe75006a4b0 0x7fe75006fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 -- 172.21.15.60:0/659256538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740009a70 con 0x7fe7500c9000 2023-10-16T12:46:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74effd640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe75006a4b0 0x7fe75006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.588+0000 7fe74f7fe640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 0x7fe7500b3c50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c390 tx=0x7fe74800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.589+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7480035f0 con 0x7fe7500c9000 2023-10-16T12:46:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.589+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748003750 con 0x7fe7500c9000 2023-10-16T12:46:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.589+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe750070c40 con 0x7fe7500c9000 2023-10-16T12:46:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.589+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748003b00 con 0x7fe7500c9000 2023-10-16T12:46:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.589+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7500b2310 con 0x7fe7500c9000 2023-10-16T12:46:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.590+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7500c1ee0 con 0x7fe7500c9000 2023-10-16T12:46:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.591+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe748003c60 con 0x7fe7500c9000 2023-10-16T12:46:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.592+0000 7fe7477fe640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe73006b070 0x7fe73006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.592+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7107+0+0 (secure 0 0 0) 0x7fe748089580 con 0x7fe7500c9000 2023-10-16T12:46:50.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.593+0000 7fe74effd640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe73006b070 0x7fe73006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:50.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.594+0000 7fe74effd640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe73006b070 0x7fe73006d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe740036000 tx=0x7fe740030000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:50.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.597+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe748053d90 con 0x7fe7500c9000 2023-10-16T12:46:50.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:50.823+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 --> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe75006b8d0 con 0x7fe73006b070 2023-10-16T12:46:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.064+0000 7fe7477fe640 1 -- 172.21.15.60:0/659256538 <== mgr.4948 v2:172.21.15.144:6832/696341254 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe7400450c0 con 0x7fe73006b070 2023-10-16T12:46:51.067 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-16T12:46:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe73006b070 msgr2=0x7fe73006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe73006b070 0x7fe73006d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe740036000 tx=0x7fe740030000 comp rx=0 tx=0).stop 2023-10-16T12:46:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 msgr2=0x7fe7500b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 0x7fe7500b3c50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c390 tx=0x7fe74800c850 comp rx=0 tx=0).stop 2023-10-16T12:46:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 shutdown_connections 2023-10-16T12:46:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe73006b070 0x7fe73006d530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7500addb0 0x7fe7500b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7500c9000 0x7fe7500b3c50 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.067+0000 7fe75535a640 1 --2- 172.21.15.60:0/659256538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe75006a4b0 0x7fe75006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.068+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 >> 172.21.15.60:0/659256538 conn(0x7fe7500ce9b0 msgr2=0x7fe7500af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.068+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 shutdown_connections 2023-10-16T12:46:51.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:51.068+0000 7fe75535a640 1 -- 172.21.15.60:0/659256538 wait complete. 2023-10-16T12:46:51.092 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-16T12:46:51.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.093 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T12:46:51.151 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:51.151 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:51.152 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:51.152 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:51.152 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:51.152 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:51.152 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:51.153 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:51.153 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:51.153 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:51.153 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:51.154 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T12:46:51.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.155 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T12:46:51.233 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T12:46:51.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.234 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T12:46:51.299 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T12:46:51.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.300 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T12:46:51.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_39_00_UTC 2023-10-16T12:46:51.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.372 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_39_00_UTC) 2023-10-16T12:46:51.453 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T12:46:51.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.454 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T12:46:51.533 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T12:46:51.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.534 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T12:46:51.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_42_01_UTC 2023-10-16T12:46:51.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.611 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_42_01_UTC) 2023-10-16T12:46:51.690 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_43_01_UTC 2023-10-16T12:46:51.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.691 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_43_01_UTC) 2023-10-16T12:46:51.780 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T12:46:51.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.781 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T12:46:51.835 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T12:46:51.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.836 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T12:46:51.902 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T12:46:51.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.903 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T12:46:51.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:51.979 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-16T12:46:52.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:46:52.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.264+0000 7f21cbfff640 1 Processor -- start 2023-10-16T12:46:52.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.265+0000 7f21cbfff640 1 -- start start 2023-10-16T12:46:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21cbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f21cc15cba0 0x7f21cc071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21cbfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0d0500 con 0x7f21cc0cdf10 2023-10-16T12:46:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21cbfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0d07c0 con 0x7f21cc0720d0 2023-10-16T12:46:52.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21cbfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc0d0aa0 con 0x7f21cc15cba0 2023-10-16T12:46:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21caffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f21cc15cba0 0x7f21cc071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:52.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21cb7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f21cc0cdf10 0x7f21cc06aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34928/0 (socket says 172.21.15.60:34928) 2023-10-16T12:46:52.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.267+0000 7f21cb7fe640 1 -- 172.21.15.60:0/3644113401 learned_addr learned my addr 172.21.15.60:0/3644113401 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:52.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.268+0000 7f21caffd640 1 -- 172.21.15.60:0/3644113401 >> v1:172.21.15.144:6789/0 conn(0x7f21cc0720d0 legacy=0x7f21cc0cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:52.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.268+0000 7f21caffd640 1 -- 172.21.15.60:0/3644113401 >> v1:172.21.15.60:6789/0 conn(0x7f21cc0cdf10 legacy=0x7f21cc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:52.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.268+0000 7f21caffd640 1 -- 172.21.15.60:0/3644113401 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0d0d80 con 0x7f21cc15cba0 2023-10-16T12:46:52.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.269+0000 7f21caffd640 1 --2- 172.21.15.60:0/3644113401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc071b90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097a0 tx=0x7f21bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ae4d80980c840d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:52.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.269+0000 7f21ca7fc640 1 -- 172.21.15.60:0/3644113401 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031830 con 0x7f21cc15cba0 2023-10-16T12:46:52.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.269+0000 7f21ca7fc640 1 -- 172.21.15.60:0/3644113401 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc031990 con 0x7f21cc15cba0 2023-10-16T12:46:52.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.269+0000 7f21ca7fc640 1 -- 172.21.15.60:0/3644113401 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031d20 con 0x7f21cc15cba0 2023-10-16T12:46:52.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 -- 172.21.15.60:0/3644113401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 msgr2=0x7f21cc071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:52.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 --2- 172.21.15.60:0/3644113401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc071b90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097a0 tx=0x7f21bc02b8d0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 -- 172.21.15.60:0/3644113401 shutdown_connections 2023-10-16T12:46:52.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 --2- 172.21.15.60:0/3644113401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc071b90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 -- 172.21.15.60:0/3644113401 >> 172.21.15.60:0/3644113401 conn(0x7f21cc065ed0 msgr2=0x7f21cc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:52.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 -- 172.21.15.60:0/3644113401 shutdown_connections 2023-10-16T12:46:52.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.270+0000 7f21cbfff640 1 -- 172.21.15.60:0/3644113401 wait complete. 2023-10-16T12:46:52.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.272+0000 7f21cbfff640 1 Processor -- start 2023-10-16T12:46:52.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.272+0000 7f21cbfff640 1 -- start start 2023-10-16T12:46:52.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21cc0720d0 0x7f21cc1f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:52.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21cc0cdf10 0x7f21cc1e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:52.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:52.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cbfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1e8a40 con 0x7f21cc15cba0 2023-10-16T12:46:52.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cbfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1e7b90 con 0x7f21cc0720d0 2023-10-16T12:46:52.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cbfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1e7e30 con 0x7f21cc0cdf10 2023-10-16T12:46:52.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:52.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc0acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37530/0 (socket says 172.21.15.60:37530) 2023-10-16T12:46:52.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.273+0000 7f21cb7fe640 1 -- 172.21.15.60:0/2306886147 learned_addr learned my addr 172.21.15.60:0/2306886147 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cb7fe640 1 -- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21cc0cdf10 msgr2=0x7f21cc1e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cb7fe640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21cc0cdf10 0x7f21cc1e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cb7fe640 1 -- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21cc0720d0 msgr2=0x7f21cc1f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cb7fe640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21cc0720d0 0x7f21cc1f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cb7fe640 1 -- 172.21.15.60:0/2306886147 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc009400 con 0x7f21cc15cba0 2023-10-16T12:46:52.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cb7fe640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc0acb00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f21c400cbc0 tx=0x7f21c400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:52.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c400ad40 con 0x7f21cc15cba0 2023-10-16T12:46:52.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c4012400 con 0x7f21cc15cba0 2023-10-16T12:46:52.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c400aea0 con 0x7f21cc15cba0 2023-10-16T12:46:52.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc1e8110 con 0x7f21cc15cba0 2023-10-16T12:46:52.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.274+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc1e86d0 con 0x7f21cc15cba0 2023-10-16T12:46:52.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.276+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc04e9b0 con 0x7f21cc15cba0 2023-10-16T12:46:52.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.277+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f21c4007070 con 0x7f21cc15cba0 2023-10-16T12:46:52.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.277+0000 7f21c37fe640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f21ac06b070 0x7f21ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:52.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.278+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f21c4089b40 con 0x7f21cc15cba0 2023-10-16T12:46:52.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.278+0000 7f21caffd640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f21ac06b070 0x7f21ac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:52.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.279+0000 7f21caffd640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f21ac06b070 0x7f21ac06d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f21bc036000 tx=0x7f21bc030000 comp rx=0 tx=0).ready entity=mgr.4948 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:52.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.282+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21c4054690 con 0x7f21cc15cba0 2023-10-16T12:46:52.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.530+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f21cc1e8b80 con 0x7f21cc15cba0 2023-10-16T12:46:52.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.860+0000 7f21c37fe640 1 -- 172.21.15.60:0/2306886147 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v13) v1 ==== 93+0+0 (secure 0 0 0) 0x7f21c4017020 con 0x7f21cc15cba0 2023-10-16T12:46:52.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.863+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f21ac06b070 msgr2=0x7f21ac06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:52.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.863+0000 7f21cbfff640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f21ac06b070 0x7f21ac06d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f21bc036000 tx=0x7f21bc030000 comp rx=0 tx=0).stop 2023-10-16T12:46:52.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.863+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 msgr2=0x7f21cc0acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:52.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.863+0000 7f21cbfff640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc0acb00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f21c400cbc0 tx=0x7f21c400a430 comp rx=0 tx=0).stop 2023-10-16T12:46:52.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.865+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 shutdown_connections 2023-10-16T12:46:52.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.865+0000 7f21cbfff640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f21ac06b070 0x7f21ac06d530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.865+0000 7f21cbfff640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21cc0cdf10 0x7f21cc1e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.865+0000 7f21cbfff640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21cc15cba0 0x7f21cc0acb00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.865+0000 7f21cbfff640 1 --2- 172.21.15.60:0/2306886147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21cc0720d0 0x7f21cc1f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:52.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.865+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 >> 172.21.15.60:0/2306886147 conn(0x7f21cc065ed0 msgr2=0x7f21cc160770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:52.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.866+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 shutdown_connections 2023-10-16T12:46:52.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:52.866+0000 7f21cbfff640 1 -- 172.21.15.60:0/2306886147 wait complete. 2023-10-16T12:46:52.890 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:46:52.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:46:53.020 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:46:53.108 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:46:53.109 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:46:53.109 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:46:53.157 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:46:53.158 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:46:53.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:46:53.430 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:53.429+0000 7f68e2812040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:46:53.443 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:53.441+0000 7f72e9245040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:46:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.459+0000 7f44ee894640 1 Processor -- start 2023-10-16T12:46:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.460+0000 7f44ee894640 1 -- start start 2023-10-16T12:46:53.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.461+0000 7f44ee894640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 0x7f44e8164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.461+0000 7f44ee894640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e8071450 con 0x7f44e8070fd0 2023-10-16T12:46:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.461+0000 7f44ee894640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e80cc600 con 0x7f44e80adb50 2023-10-16T12:46:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.461+0000 7f44ee894640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cc8e0 con 0x7f44e80b0b20 2023-10-16T12:46:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44dffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f44e8070fd0 0x7f44e80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34940/0 (socket says 172.21.15.60:34940) 2023-10-16T12:46:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44dffff640 1 -- 172.21.15.60:0/3685575337 learned_addr learned my addr 172.21.15.60:0/3685575337 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959376871 0 0) 0x7f44e8071450 con 0x7f44e8070fd0 2023-10-16T12:46:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003520 con 0x7f44e8070fd0 2023-10-16T12:46:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44ece31640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 0x7f44e8164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813986732 0 0) 0x7f44d4003520 con 0x7f44e8070fd0 2023-10-16T12:46:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 msgr2=0x7f44e8164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 0x7f44e8164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 >> v1:172.21.15.144:6789/0 conn(0x7f44e80adb50 legacy=0x7f44e806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80ccbc0 con 0x7f44e8070fd0 2023-10-16T12:46:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.462+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f44d0001670 con 0x7f44e8070fd0 2023-10-16T12:46:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 >> v1:172.21.15.60:6789/0 conn(0x7f44e8070fd0 legacy=0x7f44e80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d40039a0 0x7f44d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d40047c0 0x7f44d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 0x7f44d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cc8e0 con 0x7f44d40039a0 2023-10-16T12:46:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400bae0 con 0x7f44d40047c0 2023-10-16T12:46:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400bca0 con 0x7f44e80b0b20 2023-10-16T12:46:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d40039a0 0x7f44d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44dffff640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d40047c0 0x7f44d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.463+0000 7f44ece31640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 0x7f44d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.464+0000 7f44e7fff640 1 -- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 msgr2=0x7f44d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.464+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44e80b0b20 0x7f44d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.464+0000 7f44e7fff640 1 -- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d40047c0 msgr2=0x7f44d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:53.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.464+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d40047c0 0x7f44d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.464+0000 7f44e7fff640 1 -- 172.21.15.60:0/3685575337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d4003520 con 0x7f44d40039a0 2023-10-16T12:46:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.465+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d40039a0 0x7f44d4003dc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f44d8001750 tx=0x7f44d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdddf2fdf59cede server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.465+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d80052d0 con 0x7f44d40039a0 2023-10-16T12:46:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.465+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d802e050 con 0x7f44d40039a0 2023-10-16T12:46:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.465+0000 7f44e77fe640 1 -- 172.21.15.60:0/3685575337 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d80035f0 con 0x7f44d40039a0 2023-10-16T12:46:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ee894640 1 -- 172.21.15.60:0/3685575337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d40039a0 msgr2=0x7f44d4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ee894640 1 --2- 172.21.15.60:0/3685575337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d40039a0 0x7f44d4003dc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f44d8001750 tx=0x7f44d40057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ece31640 1 -- 172.21.15.60:0/3685575337 reap_dead start 2023-10-16T12:46:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ee894640 1 -- 172.21.15.60:0/3685575337 shutdown_connections 2023-10-16T12:46:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ee894640 1 -- 172.21.15.60:0/3685575337 >> 172.21.15.60:0/3685575337 conn(0x7f44e80cb410 msgr2=0x7f44e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ee894640 1 -- 172.21.15.60:0/3685575337 shutdown_connections 2023-10-16T12:46:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.466+0000 7f44ee894640 1 -- 172.21.15.60:0/3685575337 wait complete. 2023-10-16T12:46:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.468+0000 7f44ee894640 1 Processor -- start 2023-10-16T12:46:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.468+0000 7f44ee894640 1 -- start start 2023-10-16T12:46:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44ee894640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d40039a0 0x7f44e8155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44ee894640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 0x7f44e80af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44ee894640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44e81fabe0 0x7f44e81fb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44ee894640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81fbbb0 con 0x7f44e80aed40 2023-10-16T12:46:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44ee894640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80af700 con 0x7f44e81fabe0 2023-10-16T12:46:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44ee894640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80bf330 con 0x7f44d40039a0 2023-10-16T12:46:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 0x7f44e80af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 0x7f44e80af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37538/0 (socket says 172.21.15.60:37538) 2023-10-16T12:46:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.469+0000 7f44dffff640 1 -- 172.21.15.60:0/3085602512 learned_addr learned my addr 172.21.15.60:0/3085602512 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d40039a0 0x7f44e8155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44ece31640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44e81fabe0 0x7f44e81fb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44dffff640 1 -- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d40039a0 msgr2=0x7f44e8155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44dffff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d40039a0 0x7f44e8155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44dffff640 1 -- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44e81fabe0 msgr2=0x7f44e81fb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44dffff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44e81fabe0 0x7f44e81fb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44dffff640 1 -- 172.21.15.60:0/3085602512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e81fb8d0 con 0x7f44e80aed40 2023-10-16T12:46:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d40039a0 0x7f44e8155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44ece31640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44e81fabe0 0x7f44e81fb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.470+0000 7f44dffff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 0x7f44e80af1c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f44d000e730 tx=0x7f44d000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.471+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000ee40 con 0x7f44e80aed40 2023-10-16T12:46:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.471+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d000c8d0 con 0x7f44e80aed40 2023-10-16T12:46:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.471+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000cc80 con 0x7f44e80aed40 2023-10-16T12:46:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.471+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80bf5d0 con 0x7f44e80aed40 2023-10-16T12:46:53.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.471+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e80bfb90 con 0x7f44e80aed40 2023-10-16T12:46:53.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.472+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e804e160 con 0x7f44e80aed40 2023-10-16T12:46:53.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.473+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f44d0013420 con 0x7f44e80aed40 2023-10-16T12:46:53.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.474+0000 7f44e57fa640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 0x7f44c806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.474+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f44d0089930 con 0x7f44e80aed40 2023-10-16T12:46:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.474+0000 7f44e7fff640 1 -- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 msgr2=0x7f44c806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.474+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 0x7f44c806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.479+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44d0054400 con 0x7f44e80aed40 2023-10-16T12:46:53.490 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:53.488+0000 7f68e2812040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:46:53.506 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:53.504+0000 7f72e9245040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:46:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.675+0000 7f44e7fff640 1 -- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 msgr2=0x7f44c806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.675+0000 7f44e7fff640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 0x7f44c806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:46:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.675+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44e80ad790 con 0x7f44e80aed40 2023-10-16T12:46:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.677+0000 7f44e57fa640 1 -- 172.21.15.60:0/3085602512 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17873 (secure 0 0 0) 0x7f44d0057a20 con 0x7f44e80aed40 2023-10-16T12:46:53.680 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:53.680 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":52,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:46:52.692005+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":52,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2619332853":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/1199236332":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/1714573694":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6832/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/3946760462":"2023-10-17T12:34:57.701960+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2},{"begin":11,"length":2},{"begin":14,"length":2},{"begin":17,"length":3},{"begin":23,"length":7},{"begin":35,"length":4},{"begin":40,"length":1},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":58,"length":2},{"begin":62,"length":3},{"begin":66,"length":4},{"begin":76,"length":1},{"begin":78,"length":2},{"begin":81,"length":1},{"begin":83,"length":1},{"begin":87,"length":3},{"begin":92,"length":2},{"begin":96,"length":4},{"begin":102,"length":1},{"begin":105,"length":2},{"begin":109,"length":1},{"begin":111,"length":3},{"begin":159,"length":62}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":7,"length":1},{"begin":15,"length":1},{"begin":29,"length":1},{"begin":38,"length":1},{"begin":47,"length":1},{"begin":55,"length":1},{"begin":105,"length":1},{"begin":214,"length":7}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:46:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.679+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 msgr2=0x7f44c806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.679+0000 7f44ee894640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 0x7f44c806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 msgr2=0x7f44e80af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 0x7f44e80af1c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f44d000e730 tx=0x7f44d000ebf0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 shutdown_connections 2023-10-16T12:46:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f44c806b050 0x7f44c806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44e81fabe0 0x7f44e81fb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44e80aed40 0x7f44e80af1c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 --2- 172.21.15.60:0/3085602512 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d40039a0 0x7f44e8155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 >> 172.21.15.60:0/3085602512 conn(0x7f44e80cb410 msgr2=0x7f44e8160bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 shutdown_connections 2023-10-16T12:46:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.680+0000 7f44ee894640 1 -- 172.21.15.60:0/3085602512 wait complete. 2023-10-16T12:46:53.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:34:41.099103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:34:42.392121+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 52, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:46:53.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:46:53.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.928+0000 7f0f09c87640 1 Processor -- start 2023-10-16T12:46:53.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.928+0000 7f0f09c87640 1 -- start start 2023-10-16T12:46:53.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.929+0000 7f0f09c87640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0f04070830 0x7f0f0406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.929+0000 7f0f09c87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f0406c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.929+0000 7f0f09c87640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f04161740 con 0x7f0f04070c10 2023-10-16T12:46:53.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.929+0000 7f0f09c87640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0406c680 con 0x7f0f04070830 2023-10-16T12:46:53.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.929+0000 7f0f09c87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0406b0f0 con 0x7f0f040cf5f0 2023-10-16T12:46:53.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0f04070830 0x7f0f0406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f02ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f0406c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0f04070830 0x7f0f0406d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:38452/0 (socket says 172.21.15.60:38452) 2023-10-16T12:46:53.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 -- 172.21.15.60:0/475061884 learned_addr learned my addr 172.21.15.60:0/475061884 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:53.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 -- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 msgr2=0x7f0f0406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:53.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 --2- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f0406c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 -- 172.21.15.60:0/475061884 >> v1:172.21.15.144:6789/0 conn(0x7f0f04070c10 legacy=0x7f0f040cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.930+0000 7f0f037fe640 1 -- 172.21.15.60:0/475061884 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f041619b0 con 0x7f0f04070830 2023-10-16T12:46:53.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.931+0000 7f0f037fe640 1 --2- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070830 0x7f0f0406d3c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009870 tx=0x7f0ef402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691623b734aeabef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:53.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f027fc640 1 -- 172.21.15.60:0/475061884 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40317c0 con 0x7f0f04070830 2023-10-16T12:46:53.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f027fc640 1 -- 172.21.15.60:0/475061884 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef4031920 con 0x7f0f04070830 2023-10-16T12:46:53.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f027fc640 1 -- 172.21.15.60:0/475061884 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4031cb0 con 0x7f0f04070830 2023-10-16T12:46:53.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f09c87640 1 -- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070830 msgr2=0x7f0f0406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f09c87640 1 --2- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070830 0x7f0f0406d3c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009870 tx=0x7f0ef402b860 comp rx=0 tx=0).stop 2023-10-16T12:46:53.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f09c87640 1 -- 172.21.15.60:0/475061884 shutdown_connections 2023-10-16T12:46:53.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f09c87640 1 --2- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f0406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f09c87640 1 --2- 172.21.15.60:0/475061884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070830 0x7f0f0406d3c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.932+0000 7f0f09c87640 1 -- 172.21.15.60:0/475061884 >> 172.21.15.60:0/475061884 conn(0x7f0f040cfac0 msgr2=0x7f0f040cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:53.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.933+0000 7f0f09c87640 1 -- 172.21.15.60:0/475061884 shutdown_connections 2023-10-16T12:46:53.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.933+0000 7f0f09c87640 1 -- 172.21.15.60:0/475061884 wait complete. 2023-10-16T12:46:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.934+0000 7f0f09c87640 1 Processor -- start 2023-10-16T12:46:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.934+0000 7f0f09c87640 1 -- start start 2023-10-16T12:46:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.935+0000 7f0f09c87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 0x7f0f0414be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.935+0000 7f0f09c87640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070c10 0x7f0f0414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.935+0000 7f0f09c87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f040b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.935+0000 7f0f09c87640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041504c0 con 0x7f0f04070830 2023-10-16T12:46:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.935+0000 7f0f09c87640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041507a0 con 0x7f0f04070c10 2023-10-16T12:46:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f09c87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f04150a80 con 0x7f0f040cf5f0 2023-10-16T12:46:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 0x7f0f0414be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f02ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070c10 0x7f0f0414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 0x7f0f0414be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37544/0 (socket says 172.21.15.60:37544) 2023-10-16T12:46:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 -- 172.21.15.60:0/2982355651 learned_addr learned my addr 172.21.15.60:0/2982355651 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f03fff640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f040b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 -- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 msgr2=0x7f0f040b13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f040b13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 -- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070c10 msgr2=0x7f0f0414fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070c10 0x7f0f0414fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f037fe640 1 -- 172.21.15.60:0/2982355651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef40094d0 con 0x7f0f04070830 2023-10-16T12:46:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.936+0000 7f0f02ffd640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070c10 0x7f0f0414fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:46:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.937+0000 7f0f037fe640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 0x7f0f0414be00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0ef402b8a0 tx=0x7f0ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:53.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.937+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4032860 con 0x7f0f04070830 2023-10-16T12:46:53.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.937+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef40329c0 con 0x7f0f04070830 2023-10-16T12:46:53.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.937+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4032d70 con 0x7f0f04070830 2023-10-16T12:46:53.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.937+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f04150d60 con 0x7f0f04070830 2023-10-16T12:46:53.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.937+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0414c340 con 0x7f0f04070830 2023-10-16T12:46:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.939+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0404e160 con 0x7f0f04070830 2023-10-16T12:46:53.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.940+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f0ef404e450 con 0x7f0f04070830 2023-10-16T12:46:53.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.941+0000 7f0eebfff640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 0x7f0ef006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:53.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.941+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f0ef40abd00 con 0x7f0f04070830 2023-10-16T12:46:53.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.941+0000 7f0f02ffd640 1 -- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 msgr2=0x7f0ef006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:53.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.941+0000 7f0f02ffd640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 0x7f0ef006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:53.945+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ef4076810 con 0x7f0f04070830 2023-10-16T12:46:53.976 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:53.974+0000 7f68e2812040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:46:54.028 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.026+0000 7f68e2812040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:46:54.031 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.029+0000 7f72e9245040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:46:54.078 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.076+0000 7f68e2812040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:46:54.087 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.084+0000 7f72e9245040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:46:54.138 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.137+0000 7f68e2812040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:46:54.140 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.138+0000 7f72e9245040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:46:54.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.138+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f040b5a40 con 0x7f0f04070830 2023-10-16T12:46:54.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.139+0000 7f0eebfff640 1 -- 172.21.15.60:0/2982355651 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ef4079e30 con 0x7f0f04070830 2023-10-16T12:46:54.142 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:46:54.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f02ffd640 1 -- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 msgr2=0x7f0ef006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:54.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f02ffd640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 0x7f0ef006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:46:54.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 msgr2=0x7f0ef006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:54.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 0x7f0ef006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 msgr2=0x7f0f0414be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 0x7f0f0414be00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0ef402b8a0 tx=0x7f0ef40046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 shutdown_connections 2023-10-16T12:46:54.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f0ef006b020 0x7f0ef006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0f040cf5f0 0x7f0f040b13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f04070c10 0x7f0f0414fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 --2- 172.21.15.60:0/2982355651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f04070830 0x7f0f0414be00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 >> 172.21.15.60:0/2982355651 conn(0x7f0f040cfac0 msgr2=0x7f0f040cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:54.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 shutdown_connections 2023-10-16T12:46:54.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.142+0000 7f0f09c87640 1 -- 172.21.15.60:0/2982355651 wait complete. 2023-10-16T12:46:54.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:46:54.187 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.186+0000 7f68e2812040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:46:54.208 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.206+0000 7f72e9245040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:46:54.261 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.259+0000 7f72e9245040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:46:54.297 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.295+0000 7f68e2812040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:46:54.360 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.358+0000 7f68e2812040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:46:54.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.366+0000 7f71d20ec640 1 Processor -- start 2023-10-16T12:46:54.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.366+0000 7f71d20ec640 1 -- start start 2023-10-16T12:46:54.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.367+0000 7f71d20ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f71cc0cded0 0x7f71cc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.367+0000 7f71d20ec640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71cc15d150 con 0x7f71cc15cd70 2023-10-16T12:46:54.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.367+0000 7f71d20ec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71cc0d1180 con 0x7f71cc0b8480 2023-10-16T12:46:54.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.367+0000 7f71d20ec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71cc0d1460 con 0x7f71cc0cded0 2023-10-16T12:46:54.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.368+0000 7f71d10ea640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f71cc15cd70 0x7f71cc0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34942/0 (socket says 172.21.15.60:34942) 2023-10-16T12:46:54.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.368+0000 7f71d10ea640 1 -- 172.21.15.60:0/2153275769 learned_addr learned my addr 172.21.15.60:0/2153275769 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:54.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.368+0000 7f71d18eb640 1 --2- 172.21.15.60:0/2153275769 >> v2:172.21.15.144:3300/0 conn(0x7f71cc0cded0 0x7f71cc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.368+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127329226 0 0) 0x7f71cc15d150 con 0x7f71cc15cd70 2023-10-16T12:46:54.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.368+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b8003520 con 0x7f71cc15cd70 2023-10-16T12:46:54.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1732488589 0 0) 0x7f71b8003520 con 0x7f71cc15cd70 2023-10-16T12:46:54.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 >> v2:172.21.15.144:3300/0 conn(0x7f71cc0cded0 msgr2=0x7f71cc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:54.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 --2- 172.21.15.60:0/2153275769 >> v2:172.21.15.144:3300/0 conn(0x7f71cc0cded0 0x7f71cc0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 >> v1:172.21.15.144:6789/0 conn(0x7f71cc0b8480 legacy=0x7f71cc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc0d1740 con 0x7f71cc15cd70 2023-10-16T12:46:54.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f71c8001670 con 0x7f71cc15cd70 2023-10-16T12:46:54.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 >> v1:172.21.15.60:6789/0 conn(0x7f71cc15cd70 legacy=0x7f71cc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b80039c0 0x7f71b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b80047c0 0x7f71b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b8005180 0x7f71b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0d1460 con 0x7f71b80047c0 2023-10-16T12:46:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b800c080 con 0x7f71b8005180 2023-10-16T12:46:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b800c240 con 0x7f71b80039c0 2023-10-16T12:46:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71d08e9640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b80047c0 0x7f71b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71d10ea640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b80039c0 0x7f71b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.369+0000 7f71d18eb640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b8005180 0x7f71b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d08e9640 1 -- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b80039c0 msgr2=0x7f71b8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d08e9640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b80039c0 0x7f71b8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d08e9640 1 -- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b8005180 msgr2=0x7f71b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:54.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d08e9640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b8005180 0x7f71b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d08e9640 1 -- 172.21.15.60:0/2153275769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8003520 con 0x7f71b80047c0 2023-10-16T12:46:54.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d18eb640 1 -- 172.21.15.60:0/2153275769 reap_dead start 2023-10-16T12:46:54.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.370+0000 7f71d10ea640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b80039c0 0x7f71b8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:54.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.371+0000 7f71d08e9640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b80047c0 0x7f71b8004c40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f71bc001750 tx=0x7f71b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67fa51bd2667a9b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:54.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.371+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0052d0 con 0x7f71b80047c0 2023-10-16T12:46:54.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.371+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc02e050 con 0x7f71b80047c0 2023-10-16T12:46:54.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.371+0000 7f71c3fff640 1 -- 172.21.15.60:0/2153275769 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc003830 con 0x7f71b80047c0 2023-10-16T12:46:54.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 -- 172.21.15.60:0/2153275769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b80047c0 msgr2=0x7f71b8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b80047c0 0x7f71b8004c40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f71bc001750 tx=0x7f71b80057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 -- 172.21.15.60:0/2153275769 shutdown_connections 2023-10-16T12:46:54.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 --2- 172.21.15.60:0/2153275769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b80047c0 0x7f71b8004c40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 -- 172.21.15.60:0/2153275769 >> 172.21.15.60:0/2153275769 conn(0x7f71cc0cf570 msgr2=0x7f71cc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:54.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 -- 172.21.15.60:0/2153275769 shutdown_connections 2023-10-16T12:46:54.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.372+0000 7f71d20ec640 1 -- 172.21.15.60:0/2153275769 wait complete. 2023-10-16T12:46:54.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.373+0000 7f71d20ec640 1 Processor -- start 2023-10-16T12:46:54.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.374+0000 7f71d20ec640 1 -- start start 2023-10-16T12:46:54.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.374+0000 7f71d20ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b80047c0 0x7f71cc0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.374+0000 7f71d20ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 0x7f71cc0d2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d20ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71cc0d31a0 0x7f71cc1f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d20ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc148dc0 con 0x7f71cc159260 2023-10-16T12:46:54.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d20ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc1581c0 con 0x7f71b80047c0 2023-10-16T12:46:54.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d20ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc1584a0 con 0x7f71cc0d31a0 2023-10-16T12:46:54.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 0x7f71cc0d2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 0x7f71cc0d2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37566/0 (socket says 172.21.15.60:37566) 2023-10-16T12:46:54.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 -- 172.21.15.60:0/1356196985 learned_addr learned my addr 172.21.15.60:0/1356196985 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:54.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d10ea640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b80047c0 0x7f71cc0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d18eb640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71cc0d31a0 0x7f71cc1f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 -- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71cc0d31a0 msgr2=0x7f71cc1f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71cc0d31a0 0x7f71cc1f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 -- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b80047c0 msgr2=0x7f71cc0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b80047c0 0x7f71cc0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d08e9640 1 -- 172.21.15.60:0/1356196985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc158780 con 0x7f71cc159260 2023-10-16T12:46:54.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.375+0000 7f71d18eb640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71cc0d31a0 0x7f71cc1f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.376+0000 7f71d08e9640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 0x7f71cc0d2c60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f71bc008fd0 tx=0x7f71bc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.376+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0076c0 con 0x7f71cc159260 2023-10-16T12:46:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.376+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc007a30 con 0x7f71cc159260 2023-10-16T12:46:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.376+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc006be0 con 0x7f71cc159260 2023-10-16T12:46:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.376+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71cc158a60 con 0x7f71cc159260 2023-10-16T12:46:54.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.376+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71cc157de0 con 0x7f71cc159260 2023-10-16T12:46:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.377+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71cc157f70 con 0x7f71cc159260 2023-10-16T12:46:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.378+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f71bc006d40 con 0x7f71cc159260 2023-10-16T12:46:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.380+0000 7f71c1ffb640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 0x7f71b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.380+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f71bc0abff0 con 0x7f71cc159260 2023-10-16T12:46:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.380+0000 7f71d10ea640 1 -- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 msgr2=0x7f71b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.380+0000 7f71d10ea640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 0x7f71b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:54.390 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.382+0000 7f72e9245040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:46:54.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.384+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f71bc0defe0 con 0x7f71cc159260 2023-10-16T12:46:54.410 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.408+0000 7f68e2812040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:46:54.451 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.448+0000 7f72e9245040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:46:54.466 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.464+0000 7f68e2812040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:46:54.502 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.500+0000 7f72e9245040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:46:54.516 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.514+0000 7f68e2812040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:46:54.562 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.560+0000 7f72e9245040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:46:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.578+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71cc04e9b0 con 0x7f71cc159260 2023-10-16T12:46:54.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.579+0000 7f71c1ffb640 1 -- 172.21.15.60:0/1356196985 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f71bc04b020 con 0x7f71cc159260 2023-10-16T12:46:54.581 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:46:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d10ea640 1 -- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 msgr2=0x7f71b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d10ea640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 0x7f71b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:46:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 msgr2=0x7f71b006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 0x7f71b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 msgr2=0x7f71cc0d2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 0x7f71cc0d2c60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f71bc008fd0 tx=0x7f71bc001120 comp rx=0 tx=0).stop 2023-10-16T12:46:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 shutdown_connections 2023-10-16T12:46:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f71b006b020 0x7f71b006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71cc0d31a0 0x7f71cc1f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71cc159260 0x7f71cc0d2c60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 --2- 172.21.15.60:0/1356196985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b80047c0 0x7f71cc0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 >> 172.21.15.60:0/1356196985 conn(0x7f71cc0cf570 msgr2=0x7f71cc0b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:54.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.581+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 shutdown_connections 2023-10-16T12:46:54.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.582+0000 7f71d20ec640 1 -- 172.21.15.60:0/1356196985 wait complete. 2023-10-16T12:46:54.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:46:54.616 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:54.614+0000 7f72e9245040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:46:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.814+0000 7f50dfd47640 1 Processor -- start 2023-10-16T12:46:54.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.814+0000 7f50dfd47640 1 -- start start 2023-10-16T12:46:54.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50dfd47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50d8159ab0 0x7f50d80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50dfd47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d806c140 0x7f50d806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50dfd47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d8070360 con 0x7f50d8159e90 2023-10-16T12:46:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50dfd47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d806cb10 con 0x7f50d8159ab0 2023-10-16T12:46:54.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50dfd47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d815d890 con 0x7f50d806c140 2023-10-16T12:46:54.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50ddae3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50d8159ab0 0x7f50d80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50ddae3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50d8159ab0 0x7f50d80ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37580/0 (socket says 172.21.15.60:37580) 2023-10-16T12:46:54.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.816+0000 7f50ddae3640 1 -- 172.21.15.60:0/3221312771 learned_addr learned my addr 172.21.15.60:0/3221312771 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:54.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.817+0000 7f50de2e4640 1 --2- 172.21.15.60:0/3221312771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d806c140 0x7f50d806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.817+0000 7f50ddae3640 1 -- 172.21.15.60:0/3221312771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d806c140 msgr2=0x7f50d806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:54.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.817+0000 7f50ddae3640 1 --2- 172.21.15.60:0/3221312771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d806c140 0x7f50d806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.817+0000 7f50ddae3640 1 -- 172.21.15.60:0/3221312771 >> v1:172.21.15.60:6789/0 conn(0x7f50d8159e90 legacy=0x7f50d806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.817+0000 7f50ddae3640 1 -- 172.21.15.60:0/3221312771 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80cd1f0 con 0x7f50d8159ab0 2023-10-16T12:46:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.817+0000 7f50dcae1640 1 -- 172.21.15.60:0/3221312771 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459253843 0 0) 0x7f50d8070360 con 0x7f50d8159e90 2023-10-16T12:46:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.818+0000 7f50ddae3640 1 --2- 172.21.15.60:0/3221312771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d8159ab0 0x7f50d80ce370 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009e50 tx=0x7f50d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6267a813fcfa7ef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.818+0000 7f50dcae1640 1 -- 172.21.15.60:0/3221312771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d402c9e0 con 0x7f50d8159ab0 2023-10-16T12:46:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dcae1640 1 -- 172.21.15.60:0/3221312771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d402cb40 con 0x7f50d8159ab0 2023-10-16T12:46:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dcae1640 1 -- 172.21.15.60:0/3221312771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d40343f0 con 0x7f50d8159ab0 2023-10-16T12:46:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 -- 172.21.15.60:0/3221312771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d8159ab0 msgr2=0x7f50d80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3221312771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d8159ab0 0x7f50d80ce370 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009e50 tx=0x7f50d402be40 comp rx=0 tx=0).stop 2023-10-16T12:46:54.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 -- 172.21.15.60:0/3221312771 shutdown_connections 2023-10-16T12:46:54.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3221312771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d806c140 0x7f50d806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3221312771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d8159ab0 0x7f50d80ce370 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 -- 172.21.15.60:0/3221312771 >> 172.21.15.60:0/3221312771 conn(0x7f50d80ce9b0 msgr2=0x7f50d80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:54.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 -- 172.21.15.60:0/3221312771 shutdown_connections 2023-10-16T12:46:54.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.819+0000 7f50dfd47640 1 -- 172.21.15.60:0/3221312771 wait complete. 2023-10-16T12:46:54.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.822+0000 7f50dfd47640 1 Processor -- start 2023-10-16T12:46:54.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.822+0000 7f50dfd47640 1 -- start start 2023-10-16T12:46:54.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dfd47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 0x7f50d80c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dfd47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d8159ab0 0x7f50d815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dfd47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d8159e90 0x7f50d80c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dfd47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80c2de0 con 0x7f50d806c140 2023-10-16T12:46:54.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dfd47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80c30c0 con 0x7f50d8159ab0 2023-10-16T12:46:54.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dfd47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80c33a0 con 0x7f50d8159e90 2023-10-16T12:46:54.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50ddae3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 0x7f50d80c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50ddae3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 0x7f50d80c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37582/0 (socket says 172.21.15.60:37582) 2023-10-16T12:46:54.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50ddae3640 1 -- 172.21.15.60:0/3401768306 learned_addr learned my addr 172.21.15.60:0/3401768306 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:54.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.824+0000 7f50dd2e2640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d8159ab0 0x7f50d815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50de2e4640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d8159e90 0x7f50d80c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:54.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ddae3640 1 -- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d8159e90 msgr2=0x7f50d80c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ddae3640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d8159e90 0x7f50d80c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ddae3640 1 -- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d8159ab0 msgr2=0x7f50d815c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:54.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ddae3640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d8159ab0 0x7f50d815c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:54.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ddae3640 1 -- 172.21.15.60:0/3401768306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d4009b30 con 0x7f50d806c140 2023-10-16T12:46:54.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50dd2e2640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d8159ab0 0x7f50d815c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:54.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ddae3640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 0x7f50d80c24e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f50d4000f80 tx=0x7f50d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:54.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.825+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4009510 con 0x7f50d806c140 2023-10-16T12:46:54.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.826+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d403e450 con 0x7f50d806c140 2023-10-16T12:46:54.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.826+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d403e800 con 0x7f50d806c140 2023-10-16T12:46:54.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.826+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d80c3680 con 0x7f50d806c140 2023-10-16T12:46:54.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.826+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d80c2a20 con 0x7f50d806c140 2023-10-16T12:46:54.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.827+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d804e160 con 0x7f50d806c140 2023-10-16T12:46:54.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.828+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f50d403e960 con 0x7f50d806c140 2023-10-16T12:46:54.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.829+0000 7f50ce7fc640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 0x7f50bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:54.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.829+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6471+0+0 (secure 0 0 0) 0x7f50d40aca00 con 0x7f50d806c140 2023-10-16T12:46:54.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.830+0000 7f50dd2e2640 1 -- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 msgr2=0x7f50bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:54.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.830+0000 7f50dd2e2640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 0x7f50bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:54.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:54.833+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50d4077520 con 0x7f50d806c140 2023-10-16T12:46:54.914 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:54.912+0000 7f68e2812040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:46:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.031+0000 7f50dd2e2640 1 -- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 msgr2=0x7f50bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.031+0000 7f50dd2e2640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 0x7f50bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:46:55.077 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.075+0000 7f68e2812040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:46:55.084 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.082+0000 7f72e9245040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:46:55.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.095+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f50d80adf10 con 0x7f50d806c140 2023-10-16T12:46:55.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.096+0000 7f50ce7fc640 1 -- 172.21.15.60:0/3401768306 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f50d407ab40 con 0x7f50d806c140 2023-10-16T12:46:55.098 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:46:55.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 msgr2=0x7f50bc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:55.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 0x7f50bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 msgr2=0x7f50d80c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 0x7f50d80c24e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f50d4000f80 tx=0x7f50d4004a40 comp rx=0 tx=0).stop 2023-10-16T12:46:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 shutdown_connections 2023-10-16T12:46:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f50bc06b020 0x7f50bc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d8159e90 0x7f50d80c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d8159ab0 0x7f50d815c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 --2- 172.21.15.60:0/3401768306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d806c140 0x7f50d80c24e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.098+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 >> 172.21.15.60:0/3401768306 conn(0x7f50d80ce9b0 msgr2=0x7f50d8158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.099+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 shutdown_connections 2023-10-16T12:46:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.099+0000 7f50dfd47640 1 -- 172.21.15.60:0/3401768306 wait complete. 2023-10-16T12:46:55.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:46:55.185 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.181+0000 7f68e2812040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:46:55.233 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.231+0000 7f68e2812040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:46:55.253 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.251+0000 7f72e9245040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:46:55.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.339+0000 7f68e2812040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:46:55.369 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.366+0000 7f72e9245040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:46:55.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.368+0000 7f84d8ee0640 1 Processor -- start 2023-10-16T12:46:55.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.368+0000 7f84d8ee0640 1 -- start start 2023-10-16T12:46:55.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.370+0000 7f84d8ee0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 0x7f84d40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.370+0000 7f84d8ee0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d406cee0 con 0x7f84d41618b0 2023-10-16T12:46:55.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.370+0000 7f84d8ee0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d40d0890 con 0x7f84d415d330 2023-10-16T12:46:55.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.370+0000 7f84d8ee0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d406b1e0 con 0x7f84d40661c0 2023-10-16T12:46:55.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.370+0000 7f84cbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f84d41618b0 0x7f84d4065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34948/0 (socket says 172.21.15.60:34948) 2023-10-16T12:46:55.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.370+0000 7f84cbfff640 1 -- 172.21.15.60:0/1308791857 learned_addr learned my addr 172.21.15.60:0/1308791857 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 0x7f84d40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251444318 0 0) 0x7f84d406cee0 con 0x7f84d41618b0 2023-10-16T12:46:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c0003520 con 0x7f84d41618b0 2023-10-16T12:46:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982120375 0 0) 0x7f84d40d0890 con 0x7f84d415d330 2023-10-16T12:46:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d406cee0 con 0x7f84d415d330 2023-10-16T12:46:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822626824 0 0) 0x7f84c0003520 con 0x7f84d41618b0 2023-10-16T12:46:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 msgr2=0x7f84d40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 0x7f84d40ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 >> v1:172.21.15.144:6789/0 conn(0x7f84d415d330 legacy=0x7f84d406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.371+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d40d05b0 con 0x7f84d41618b0 2023-10-16T12:46:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f84bc001670 con 0x7f84d41618b0 2023-10-16T12:46:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 >> v1:172.21.15.60:6789/0 conn(0x7f84d41618b0 legacy=0x7f84d4065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 0x7f84c0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c00040e0 0x7f84c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0004d00 0x7f84c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d406b1e0 con 0x7f84c00040e0 2023-10-16T12:46:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c000bb40 con 0x7f84c0004d00 2023-10-16T12:46:55.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c000bd00 con 0x7f84d40661c0 2023-10-16T12:46:55.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c00040e0 0x7f84c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d259c640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 0x7f84c0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0004d00 0x7f84c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84cbfff640 1 -- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 msgr2=0x7f84c0003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:55.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40661c0 0x7f84c0003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84cbfff640 1 -- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0004d00 msgr2=0x7f84c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:55.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0004d00 0x7f84c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84cbfff640 1 -- 172.21.15.60:0/1308791857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0003520 con 0x7f84c00040e0 2023-10-16T12:46:55.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.372+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c0004d00 0x7f84c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:55.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c00040e0 0x7f84c00047c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f84bc007f90 tx=0x7f84c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab531ed50ed0224 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:55.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc005ad0 con 0x7f84c00040e0 2023-10-16T12:46:55.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc005c30 con 0x7f84c00040e0 2023-10-16T12:46:55.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84d1d9b640 1 -- 172.21.15.60:0/1308791857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc007290 con 0x7f84c00040e0 2023-10-16T12:46:55.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1308791857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c00040e0 msgr2=0x7f84c00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1308791857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c00040e0 0x7f84c00047c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f84bc007f90 tx=0x7f84c00057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.374+0000 7f84d2d9d640 1 -- 172.21.15.60:0/1308791857 reap_dead start 2023-10-16T12:46:55.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.375+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1308791857 shutdown_connections 2023-10-16T12:46:55.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.375+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1308791857 >> 172.21.15.60:0/1308791857 conn(0x7f84d40d0ab0 msgr2=0x7f84d40d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:55.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.375+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1308791857 shutdown_connections 2023-10-16T12:46:55.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.375+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1308791857 wait complete. 2023-10-16T12:46:55.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.376+0000 7f84d8ee0640 1 Processor -- start 2023-10-16T12:46:55.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.377+0000 7f84d8ee0640 1 -- start start 2023-10-16T12:46:55.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.377+0000 7f84d8ee0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c00040e0 0x7f84d40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.377+0000 7f84d8ee0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 0x7f84d415b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84d8ee0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d415bd60 0x7f84d40ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84d8ee0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40ca4c0 con 0x7f84d40acdd0 2023-10-16T12:46:55.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84d8ee0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40c98b0 con 0x7f84c00040e0 2023-10-16T12:46:55.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84d8ee0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40c9b90 con 0x7f84d415bd60 2023-10-16T12:46:55.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 0x7f84d415b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 0x7f84d415b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37590/0 (socket says 172.21.15.60:37590) 2023-10-16T12:46:55.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 -- 172.21.15.60:0/1085517772 learned_addr learned my addr 172.21.15.60:0/1085517772 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:55.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84d259c640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c00040e0 0x7f84d40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d415bd60 0x7f84d40ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:55.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 -- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d415bd60 msgr2=0x7f84d40ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d415bd60 0x7f84d40ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 -- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c00040e0 msgr2=0x7f84d40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c00040e0 0x7f84d40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.378+0000 7f84cbfff640 1 -- 172.21.15.60:0/1085517772 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d40c9e70 con 0x7f84d40acdd0 2023-10-16T12:46:55.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.379+0000 7f84cbfff640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 0x7f84d415b820 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02c310 tx=0x7f84bc001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:55.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.379+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc039030 con 0x7f84d40acdd0 2023-10-16T12:46:55.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.379+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc007480 con 0x7f84d40acdd0 2023-10-16T12:46:55.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.379+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc03c040 con 0x7f84d40acdd0 2023-10-16T12:46:55.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.379+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d40ca150 con 0x7f84d40acdd0 2023-10-16T12:46:55.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.379+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d40ad760 con 0x7f84d40acdd0 2023-10-16T12:46:55.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.380+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d404e160 con 0x7f84d40acdd0 2023-10-16T12:46:55.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.382+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f84bc032070 con 0x7f84d40acdd0 2023-10-16T12:46:55.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.384+0000 7f84caffd640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 0x7f84b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:55.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.384+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6471+0+0 (secure 0 0 0) 0x7f84bc0aae60 con 0x7f84d40acdd0 2023-10-16T12:46:55.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.384+0000 7f84d259c640 1 -- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 msgr2=0x7f84b406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:55.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.384+0000 7f84d259c640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 0x7f84b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:55.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.388+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84bc075a20 con 0x7f84d40acdd0 2023-10-16T12:46:55.424 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.421+0000 7f72e9245040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:46:55.449 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.447+0000 7f68e2812040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:46:55.537 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.535+0000 7f72e9245040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:46:55.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.585+0000 7f84d259c640 1 -- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 msgr2=0x7f84b406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:55.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.585+0000 7f84d259c640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 0x7f84b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:46:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.596+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f84d406f3a0 con 0x7f84d40acdd0 2023-10-16T12:46:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.597+0000 7f84caffd640 1 -- 172.21.15.60:0/1085517772 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v54) v1 ==== 80+0+360 (secure 0 0 0) 0x7f84bc079040 con 0x7f84d40acdd0 2023-10-16T12:46:55.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2619332853 2023-10-17T12:34:57.701960+0000 2023-10-16T12:46:55.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1199236332 2023-10-17T12:34:57.701960+0000 2023-10-16T12:46:55.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1714573694 2023-10-17T12:34:57.701960+0000 2023-10-16T12:46:55.601 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/1973903962 2023-10-17T12:34:57.701960+0000 2023-10-16T12:46:55.601 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/1973903962 2023-10-17T12:34:57.701960+0000 2023-10-16T12:46:55.601 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3946760462 2023-10-17T12:34:57.701960+0000 2023-10-16T12:46:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:listed 6 entries 2023-10-16T12:46:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 msgr2=0x7f84b406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 0x7f84b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 msgr2=0x7f84d415b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 0x7f84d415b820 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02c310 tx=0x7f84bc001930 comp rx=0 tx=0).stop 2023-10-16T12:46:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 shutdown_connections 2023-10-16T12:46:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f84b406b020 0x7f84b406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d415bd60 0x7f84d40ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d40acdd0 0x7f84d415b820 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 --2- 172.21.15.60:0/1085517772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c00040e0 0x7f84d40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 >> 172.21.15.60:0/1085517772 conn(0x7f84d40d0ab0 msgr2=0x7f84d40c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 shutdown_connections 2023-10-16T12:46:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:55.600+0000 7f84d8ee0640 1 -- 172.21.15.60:0/1085517772 wait complete. 2023-10-16T12:46:55.605 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.604+0000 7f68e2812040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:46:55.621 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:46:55.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:55.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:46:55.650 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.648+0000 7f72e9245040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:46:55.654 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.653+0000 7f68e2812040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:46:55.727 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:46:55.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:46:55.727 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:46:55.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:55.775 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:55.773+0000 7f68e2812040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:46:55.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:55.816 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:46:55.817 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:46:55.819 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.817+0000 7f72e9245040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:46:55.872 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.870+0000 7f72e9245040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:46:55.878 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:46:55.878 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:46:55.879 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:46:55.932 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:46:55.932 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:46:55.933 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:46:55.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:46:55.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:46:56.004 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:55.996+0000 7f72e9245040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:46:56.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.206+0000 7f5f4bfff640 1 Processor -- start 2023-10-16T12:46:56.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.206+0000 7f5f4bfff640 1 -- start start 2023-10-16T12:46:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.207+0000 7f5f4bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f4c0ad970 0x7f5f4c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.207+0000 7f5f4bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c15b9d0 con 0x7f5f4c06c040 2023-10-16T12:46:56.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4bfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c15bcb0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c15bf90 con 0x7f5f4c0ccfb0 2023-10-16T12:46:56.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f4c0ad970 0x7f5f4c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f4c0ad970 0x7f5f4c0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37596/0 (socket says 172.21.15.60:37596) 2023-10-16T12:46:56.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/3231390231 learned_addr learned my addr 172.21.15.60:0/3231390231 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:56.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4affd640 1 --1- 172.21.15.60:0/3231390231 >> v1:172.21.15.144:6789/0 conn(0x7f5f4c06c040 0x7f5f4c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:53622/0 (socket says 172.21.15.60:53622) 2023-10-16T12:46:56.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4b7fe640 1 --2- 172.21.15.60:0/3231390231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/3231390231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 msgr2=0x7f5f4c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:56.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 --2- 172.21.15.60:0/3231390231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/3231390231 >> v1:172.21.15.144:6789/0 conn(0x7f5f4c06c040 legacy=0x7f5f4c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.208+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/3231390231 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0cc270 con 0x7f5f4c0ad970 2023-10-16T12:46:56.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4a7fc640 1 --2- 172.21.15.60:0/3231390231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c0cca70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009870 tx=0x7f5f3402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac53ae3ea277815 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:56.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f49ffb640 1 -- 172.21.15.60:0/3231390231 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34030840 con 0x7f5f4c0ad970 2023-10-16T12:46:56.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f49ffb640 1 -- 172.21.15.60:0/3231390231 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f340309a0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f49ffb640 1 -- 172.21.15.60:0/3231390231 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34030d30 con 0x7f5f4c0ad970 2023-10-16T12:46:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4bfff640 1 -- 172.21.15.60:0/3231390231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 msgr2=0x7f5f4c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/3231390231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c0cca70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009870 tx=0x7f5f3402b980 comp rx=0 tx=0).stop 2023-10-16T12:46:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4bfff640 1 -- 172.21.15.60:0/3231390231 shutdown_connections 2023-10-16T12:46:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/3231390231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/3231390231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c0cca70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.210+0000 7f5f4bfff640 1 -- 172.21.15.60:0/3231390231 >> 172.21.15.60:0/3231390231 conn(0x7f5f4c0ce6a0 msgr2=0x7f5f4c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.211+0000 7f5f4bfff640 1 -- 172.21.15.60:0/3231390231 shutdown_connections 2023-10-16T12:46:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.211+0000 7f5f4bfff640 1 -- 172.21.15.60:0/3231390231 wait complete. 2023-10-16T12:46:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.212+0000 7f5f4bfff640 1 Processor -- start 2023-10-16T12:46:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.212+0000 7f5f4bfff640 1 -- start start 2023-10-16T12:46:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f4c06c040 0x7f5f4c0c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c1f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c1f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c1f25f0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0c5070 con 0x7f5f4c06c040 2023-10-16T12:46:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0ae230 con 0x7f5f4c0ccfb0 2023-10-16T12:46:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.213+0000 7f5f4a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c1f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f4c06c040 0x7f5f4c0c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c1f1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37600/0 (socket says 172.21.15.60:37600) 2023-10-16T12:46:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/2311947887 learned_addr learned my addr 172.21.15.60:0/2311947887 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4b7fe640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c1f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 msgr2=0x7f5f4c1f20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c1f20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f4c06c040 msgr2=0x7f5f4c0c2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f4c06c040 0x7f5f4c0c2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 -- 172.21.15.60:0/2311947887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340094d0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.214+0000 7f5f4a7fc640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c1f1b70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5f340097c0 tx=0x7f5f34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:56.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.215+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34039c90 con 0x7f5f4c0ad970 2023-10-16T12:46:56.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.215+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f34039df0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.215+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f340446a0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.215+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c0ae510 con 0x7f5f4c0ad970 2023-10-16T12:46:56.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.215+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c0aead0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.216+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c06b330 con 0x7f5f4c0ad970 2023-10-16T12:46:56.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.217+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f5f3402f390 con 0x7f5f4c0ad970 2023-10-16T12:46:56.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.218+0000 7f5f43fff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 0x7f5f2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.218+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6471+0+0 (secure 0 0 0) 0x7f5f3403e070 con 0x7f5f4c0ad970 2023-10-16T12:46:56.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.218+0000 7f5f4affd640 1 -- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 msgr2=0x7f5f2c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:56.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.218+0000 7f5f4affd640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 0x7f5f2c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:56.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.223+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f34076f60 con 0x7f5f4c0ad970 2023-10-16T12:46:56.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.419+0000 7f5f4affd640 1 -- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 msgr2=0x7f5f2c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.419+0000 7f5f4affd640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 0x7f5f2c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:46:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.455+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4c04e9b0 con 0x7f5f4c0ad970 2023-10-16T12:46:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.456+0000 7f5f43fff640 1 -- 172.21.15.60:0/2311947887 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f5f3407a580 con 0x7f5f4c0ad970 2023-10-16T12:46:56.459 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:56.459 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:34:43.360823+0000","modified":"2023-10-16T12:34:46.440178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/3714850681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3714850681},{"type":"v1","addr":"172.21.15.60:6835","nonce":3714850681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:46:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2023-10-16T12:46:56.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.458+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 msgr2=0x7f5f2c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:56.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.458+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 0x7f5f2c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.458+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 msgr2=0x7f5f4c1f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.458+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c1f1b70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5f340097c0 tx=0x7f5f34004730 comp rx=0 tx=0).stop 2023-10-16T12:46:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 shutdown_connections 2023-10-16T12:46:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7f5f2c06b020 0x7f5f2c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f4c0ccfb0 0x7f5f4c1f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f4c0ad970 0x7f5f4c1f1b70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 --2- 172.21.15.60:0/2311947887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f4c06c040 0x7f5f4c0c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 >> 172.21.15.60:0/2311947887 conn(0x7f5f4c0ce6a0 msgr2=0x7f5f4c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 shutdown_connections 2023-10-16T12:46:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.459+0000 7f5f4bfff640 1 -- 172.21.15.60:0/2311947887 wait complete. 2023-10-16T12:46:56.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:46:56.606 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:56.604+0000 7f68e2812040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:46:56.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.674+0000 7fe86b13e640 1 Processor -- start 2023-10-16T12:46:56.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.675+0000 7fe86b13e640 1 -- start start 2023-10-16T12:46:56.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.676+0000 7fe86b13e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 0x7fe86406c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.676+0000 7fe86b13e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe86415e540 0x7fe8640b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.676+0000 7fe86b13e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8640cd630 con 0x7fe86415e0c0 2023-10-16T12:46:56.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.677+0000 7fe86b13e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8640ce1b0 con 0x7fe86415e540 2023-10-16T12:46:56.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.677+0000 7fe86b13e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86406cd10 con 0x7fe86406c3f0 2023-10-16T12:46:56.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.677+0000 7fe868eda640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 0x7fe86406c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.677+0000 7fe868eda640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 0x7fe86406c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:49346/0 (socket says 172.21.15.60:49346) 2023-10-16T12:46:56.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.677+0000 7fe868eda640 1 -- 172.21.15.60:0/1905680033 learned_addr learned my addr 172.21.15.60:0/1905680033 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:56.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.677+0000 7fe8696db640 1 --2- 172.21.15.60:0/1905680033 >> v2:172.21.15.144:3300/0 conn(0x7fe86415e540 0x7fe8640b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.678+0000 7fe868eda640 1 -- 172.21.15.60:0/1905680033 >> v2:172.21.15.144:3300/0 conn(0x7fe86415e540 msgr2=0x7fe8640b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:56.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.678+0000 7fe868eda640 1 --2- 172.21.15.60:0/1905680033 >> v2:172.21.15.144:3300/0 conn(0x7fe86415e540 0x7fe8640b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.678+0000 7fe868eda640 1 -- 172.21.15.60:0/1905680033 >> v1:172.21.15.144:6789/0 conn(0x7fe86415e0c0 legacy=0x7fe8640b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.678+0000 7fe868eda640 1 -- 172.21.15.60:0/1905680033 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8640d2170 con 0x7fe86406c3f0 2023-10-16T12:46:56.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.679+0000 7fe868eda640 1 --2- 172.21.15.60:0/1905680033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 0x7fe86406c7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe854009e50 tx=0x7fe85402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78506e9f6fdd5c43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.679+0000 7fe8637fe640 1 -- 172.21.15.60:0/1905680033 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85402cb00 con 0x7fe86406c3f0 2023-10-16T12:46:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.679+0000 7fe8637fe640 1 -- 172.21.15.60:0/1905680033 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85402cc60 con 0x7fe86406c3f0 2023-10-16T12:46:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.679+0000 7fe8637fe640 1 -- 172.21.15.60:0/1905680033 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8540343f0 con 0x7fe86406c3f0 2023-10-16T12:46:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 -- 172.21.15.60:0/1905680033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 msgr2=0x7fe86406c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1905680033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 0x7fe86406c7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe854009e50 tx=0x7fe85402bf60 comp rx=0 tx=0).stop 2023-10-16T12:46:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 -- 172.21.15.60:0/1905680033 shutdown_connections 2023-10-16T12:46:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1905680033 >> v2:172.21.15.144:3300/0 conn(0x7fe86415e540 0x7fe8640b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1905680033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86406c3f0 0x7fe86406c7d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 -- 172.21.15.60:0/1905680033 >> 172.21.15.60:0/1905680033 conn(0x7fe8640d2cb0 msgr2=0x7fe8640d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.680+0000 7fe86b13e640 1 -- 172.21.15.60:0/1905680033 shutdown_connections 2023-10-16T12:46:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.681+0000 7fe86b13e640 1 -- 172.21.15.60:0/1905680033 wait complete. 2023-10-16T12:46:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.682+0000 7fe86b13e640 1 Processor -- start 2023-10-16T12:46:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.683+0000 7fe86b13e640 1 -- start start 2023-10-16T12:46:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe86b13e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe86406c3f0 0x7fe8640ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe86b13e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 0x7fe8640cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe86b13e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86415e540 0x7fe8640ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe86b13e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640cbba0 con 0x7fe86415e0c0 2023-10-16T12:46:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe86b13e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640cb300 con 0x7fe86406c3f0 2023-10-16T12:46:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe86b13e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640cb5e0 con 0x7fe86415e540 2023-10-16T12:46:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe863fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 0x7fe8640cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe863fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 0x7fe8640cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37606/0 (socket says 172.21.15.60:37606) 2023-10-16T12:46:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe863fff640 1 -- 172.21.15.60:0/1069074399 learned_addr learned my addr 172.21.15.60:0/1069074399 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe868eda640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe86406c3f0 0x7fe8640ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.684+0000 7fe8696db640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86415e540 0x7fe8640ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe863fff640 1 -- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86415e540 msgr2=0x7fe8640ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe863fff640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86415e540 0x7fe8640ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe863fff640 1 -- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe86406c3f0 msgr2=0x7fe8640ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe863fff640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe86406c3f0 0x7fe8640ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe863fff640 1 -- 172.21.15.60:0/1069074399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe854009b30 con 0x7fe86415e0c0 2023-10-16T12:46:56.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe8696db640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86415e540 0x7fe8640ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:46:56.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.685+0000 7fe863fff640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 0x7fe8640cc670 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c400 tx=0x7fe84c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:56.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.686+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0052d0 con 0x7fe86415e0c0 2023-10-16T12:46:56.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.686+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c005450 con 0x7fe86415e0c0 2023-10-16T12:46:56.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.686+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c007260 con 0x7fe86415e0c0 2023-10-16T12:46:56.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.686+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8640cb8c0 con 0x7fe86415e0c0 2023-10-16T12:46:56.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.686+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8640cad30 con 0x7fe86415e0c0 2023-10-16T12:46:56.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.687+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8641592a0 con 0x7fe86415e0c0 2023-10-16T12:46:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.689+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fe84c0073c0 con 0x7fe86415e0c0 2023-10-16T12:46:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.691+0000 7fe8617fa640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe84406b070 0x7fe84406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.691+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6471+0+0 (secure 0 0 0) 0x7fe84c088f50 con 0x7fe86415e0c0 2023-10-16T12:46:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.691+0000 7fe868eda640 1 -- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe84406b070 msgr2=0x7fe84406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/696341254 2023-10-16T12:46:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.691+0000 7fe868eda640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe84406b070 0x7fe84406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:46:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.695+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8641592a0 con 0x7fe86415e0c0 2023-10-16T12:46:56.841 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:56.839+0000 7f72e9245040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:46:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.885+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8641592a0 con 0x7fe86415e0c0 2023-10-16T12:46:56.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.887+0000 7fe8617fa640 1 -- 172.21.15.60:0/1069074399 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17730 (secure 0 0 0) 0x7fe8641592a0 con 0x7fe86415e0c0 2023-10-16T12:46:56.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:56.891 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":54,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:46:54.698281+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":225,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2619332853":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/1199236332":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/1714573694":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6832/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/3946760462":"2023-10-17T12:34:57.701960+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2},{"begin":11,"length":2},{"begin":14,"length":2},{"begin":17,"length":3},{"begin":23,"length":7},{"begin":35,"length":4},{"begin":40,"length":1},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":58,"length":2},{"begin":62,"length":3},{"begin":66,"length":5},{"begin":76,"length":1},{"begin":78,"length":2},{"begin":81,"length":3},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":92,"length":2},{"begin":96,"length":4},{"begin":101,"length":2},{"begin":104,"length":3},{"begin":109,"length":1},{"begin":111,"length":3},{"begin":159,"length":67}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":104,"length":1},{"begin":225,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:46:56.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe84406b070 msgr2=0x7fe84406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:46:56.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe84406b070 0x7fe84406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 msgr2=0x7fe8640cc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 0x7fe8640cc670 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c400 tx=0x7fe84c00c8c0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 shutdown_connections 2023-10-16T12:46:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:6832/696341254,v1:172.21.15.144:6833/696341254] conn(0x7fe84406b070 0x7fe84406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe86415e540 0x7fe8640ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe86415e0c0 0x7fe8640cc670 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 --2- 172.21.15.60:0/1069074399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe86406c3f0 0x7fe8640ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:56.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 >> 172.21.15.60:0/1069074399 conn(0x7fe8640d2cb0 msgr2=0x7fe8640b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:56.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.890+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 shutdown_connections 2023-10-16T12:46:56.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:56.891+0000 7fe86b13e640 1 -- 172.21.15.60:0/1069074399 wait complete. 2023-10-16T12:46:56.905 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:56.904+0000 7f68e2812040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:46:56.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:34:41.099103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:34:42.392121+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 225, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:46:56.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:46:56.963 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:56.961+0000 7f68e2812040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:46:57.076 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.074+0000 7f68e2812040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:46:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.116+0000 7f8346de2640 1 Processor -- start 2023-10-16T12:46:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.116+0000 7f8346de2640 1 -- start start 2023-10-16T12:46:57.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f8346de2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f83401430c0 0x7f834006b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f8346de2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400cc790 con 0x7f83400cc220 2023-10-16T12:46:57.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f8346de2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400cca70 con 0x7f83400acf90 2023-10-16T12:46:57.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f8346de2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83400ccd50 con 0x7f83401430c0 2023-10-16T12:46:57.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f8345de0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f83401430c0 0x7f834006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f83455df640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f83400cc220 0x7f83400ac880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34958/0 (socket says 172.21.15.60:34958) 2023-10-16T12:46:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.118+0000 7f83455df640 1 -- 172.21.15.60:0/2124818670 learned_addr learned my addr 172.21.15.60:0/2124818670 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.119+0000 7f8344dde640 1 -- 172.21.15.60:0/2124818670 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577653264 0 0) 0x7f83400cc790 con 0x7f83400cc220 2023-10-16T12:46:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.119+0000 7f8344dde640 1 -- 172.21.15.60:0/2124818670 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832c003520 con 0x7f83400cc220 2023-10-16T12:46:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.119+0000 7f8344dde640 1 -- 172.21.15.60:0/2124818670 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232521421 0 0) 0x7f83400cca70 con 0x7f83400acf90 2023-10-16T12:46:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.119+0000 7f8345de0640 1 -- 172.21.15.60:0/2124818670 >> v1:172.21.15.144:6789/0 conn(0x7f83400acf90 legacy=0x7f83400ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.119+0000 7f8345de0640 1 -- 172.21.15.60:0/2124818670 >> v1:172.21.15.60:6789/0 conn(0x7f83400cc220 legacy=0x7f83400ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.119+0000 7f8345de0640 1 -- 172.21.15.60:0/2124818670 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400cd030 con 0x7f83401430c0 2023-10-16T12:46:57.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.120+0000 7f8345de0640 1 --2- 172.21.15.60:0/2124818670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83401430c0 0x7f834006b260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f833c0097a0 tx=0x7f833c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c2f411db4285b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:57.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.120+0000 7f8344dde640 1 -- 172.21.15.60:0/2124818670 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c030610 con 0x7f83401430c0 2023-10-16T12:46:57.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.120+0000 7f8344dde640 1 -- 172.21.15.60:0/2124818670 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c030770 con 0x7f83401430c0 2023-10-16T12:46:57.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.120+0000 7f8344dde640 1 -- 172.21.15.60:0/2124818670 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c030b00 con 0x7f83401430c0 2023-10-16T12:46:57.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 -- 172.21.15.60:0/2124818670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83401430c0 msgr2=0x7f834006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 --2- 172.21.15.60:0/2124818670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83401430c0 0x7f834006b260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f833c0097a0 tx=0x7f833c02b960 comp rx=0 tx=0).stop 2023-10-16T12:46:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 -- 172.21.15.60:0/2124818670 shutdown_connections 2023-10-16T12:46:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 --2- 172.21.15.60:0/2124818670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83401430c0 0x7f834006b260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 -- 172.21.15.60:0/2124818670 >> 172.21.15.60:0/2124818670 conn(0x7f83400ce650 msgr2=0x7f83400cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 -- 172.21.15.60:0/2124818670 shutdown_connections 2023-10-16T12:46:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.121+0000 7f8346de2640 1 -- 172.21.15.60:0/2124818670 wait complete. 2023-10-16T12:46:57.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.123+0000 7f8346de2640 1 Processor -- start 2023-10-16T12:46:57.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.123+0000 7f8346de2640 1 -- start start 2023-10-16T12:46:57.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8346de2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 0x7f834013d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8346de2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83400cc220 0x7f8340145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8346de2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83401430c0 0x7f834015f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8346de2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834015f3a0 con 0x7f83400cc220 2023-10-16T12:46:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8346de2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834015e790 con 0x7f83400acf90 2023-10-16T12:46:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8346de2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834015ea70 con 0x7f83401430c0 2023-10-16T12:46:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f83455df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83400cc220 0x7f8340145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8345de0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 0x7f834013d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8345de0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 0x7f834013d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:38552/0 (socket says 172.21.15.60:38552) 2023-10-16T12:46:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f8345de0640 1 -- 172.21.15.60:0/944277424 learned_addr learned my addr 172.21.15.60:0/944277424 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.124+0000 7f83465e1640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83401430c0 0x7f834015f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8345de0640 1 -- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83401430c0 msgr2=0x7f834015f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8345de0640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83401430c0 0x7f834015f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8345de0640 1 -- 172.21.15.60:0/944277424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83400cc220 msgr2=0x7f8340145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8345de0640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83400cc220 0x7f8340145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8345de0640 1 -- 172.21.15.60:0/944277424 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c009400 con 0x7f83400acf90 2023-10-16T12:46:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f83455df640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83400cc220 0x7f8340145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:46:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8345de0640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 0x7f834013d160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f833c0098c0 tx=0x7f833c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c03e070 con 0x7f83400acf90 2023-10-16T12:46:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c030e30 con 0x7f83400acf90 2023-10-16T12:46:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c0399a0 con 0x7f83400acf90 2023-10-16T12:46:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.125+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834015ed50 con 0x7f83400acf90 2023-10-16T12:46:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.126+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83401464c0 con 0x7f83400acf90 2023-10-16T12:46:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.127+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8340146650 con 0x7f83400acf90 2023-10-16T12:46:57.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.128+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 14) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f833c042070 con 0x7f83400acf90 2023-10-16T12:46:57.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.128+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6814+0+0 (secure 0 0 0) 0x7f833c0aaef0 con 0x7f83400acf90 2023-10-16T12:46:57.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.134+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8340146650 con 0x7f83400acf90 2023-10-16T12:46:57.167 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f68646cb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.167 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f68646cb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.167 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f68646cb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f68646cb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f68646cb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f6866ed0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f6866ed0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.168 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f6866ed0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.169 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f6866ed0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.169 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:46:57.165+0000 7f6866ed0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:46:57.171 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:57.169+0000 7f72e9245040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:46:57.232 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:57.230+0000 7f72e9245040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:46:57.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.330+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8340146650 con 0x7f83400acf90 2023-10-16T12:46:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.332+0000 7f83367fc640 1 -- 172.21.15.60:0/944277424 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f833c049020 con 0x7f83400acf90 2023-10-16T12:46:57.334 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:46:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 msgr2=0x7f834013d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 0x7f834013d160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f833c0098c0 tx=0x7f833c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 shutdown_connections 2023-10-16T12:46:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83401430c0 0x7f834015f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83400cc220 0x7f8340145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 --2- 172.21.15.60:0/944277424 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83400acf90 0x7f834013d160 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 >> 172.21.15.60:0/944277424 conn(0x7f83400ce650 msgr2=0x7f834015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 shutdown_connections 2023-10-16T12:46:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.335+0000 7f8346de2640 1 -- 172.21.15.60:0/944277424 wait complete. 2023-10-16T12:46:57.352 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:46:57.350+0000 7f72e9245040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:46:57.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:46:57.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.551+0000 7fda9de8a640 1 Processor -- start 2023-10-16T12:46:57.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.551+0000 7fda9de8a640 1 -- start start 2023-10-16T12:46:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.552+0000 7fda9de8a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 0x7fda98161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.552+0000 7fda9de8a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda980ad7a0 0x7fda98070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.552+0000 7fda9de8a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda980ae1a0 con 0x7fda980adc00 2023-10-16T12:46:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.552+0000 7fda9de8a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda980ae480 con 0x7fda980ad7a0 2023-10-16T12:46:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.552+0000 7fda9de8a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980ae760 con 0x7fda98160c80 2023-10-16T12:46:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.553+0000 7fda9d689640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fda980adc00 0x7fda98142400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34970/0 (socket says 172.21.15.60:34970) 2023-10-16T12:46:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.553+0000 7fda9d689640 1 -- 172.21.15.60:0/1675587368 learned_addr learned my addr 172.21.15.60:0/1675587368 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.553+0000 7fda97fff640 1 --2- 172.21.15.60:0/1675587368 >> v2:172.21.15.144:3300/0 conn(0x7fda980ad7a0 0x7fda98070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.553+0000 7fda9ce88640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 0x7fda98161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639201364 0 0) 0x7fda980ae1a0 con 0x7fda980adc00 2023-10-16T12:46:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda78003520 con 0x7fda980adc00 2023-10-16T12:46:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1981269583 0 0) 0x7fda78003520 con 0x7fda980adc00 2023-10-16T12:46:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 msgr2=0x7fda98161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 0x7fda98161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 >> v2:172.21.15.144:3300/0 conn(0x7fda980ad7a0 msgr2=0x7fda98070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 --2- 172.21.15.60:0/1675587368 >> v2:172.21.15.144:3300/0 conn(0x7fda980ad7a0 0x7fda98070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980aea40 con 0x7fda980adc00 2023-10-16T12:46:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fda90001670 con 0x7fda980adc00 2023-10-16T12:46:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 >> v1:172.21.15.60:6789/0 conn(0x7fda980adc00 legacy=0x7fda98142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda78003970 0x7fda78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 0x7fda780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda78004d00 0x7fda78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980ae480 con 0x7fda78004d00 2023-10-16T12:46:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980ae760 con 0x7fda78003970 2023-10-16T12:46:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.554+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7800b980 con 0x7fda98160c80 2023-10-16T12:46:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9d689640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda78004d00 0x7fda78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9d689640 1 -- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 msgr2=0x7fda780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda97fff640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 0x7fda780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9ce88640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda78003970 0x7fda78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9d689640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda98160c80 0x7fda780047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9d689640 1 -- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda78003970 msgr2=0x7fda78003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9d689640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda78003970 0x7fda78003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.555+0000 7fda9d689640 1 -- 172.21.15.60:0/1675587368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda78003520 con 0x7fda78004d00 2023-10-16T12:46:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.556+0000 7fda9d689640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda78004d00 0x7fda78005180 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c110 tx=0x7fda780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8129c0f5b79985 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.556+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90005ad0 con 0x7fda78004d00 2023-10-16T12:46:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.556+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda90005c50 con 0x7fda78004d00 2023-10-16T12:46:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.556+0000 7fda977fe640 1 -- 172.21.15.60:0/1675587368 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9002c830 con 0x7fda78004d00 2023-10-16T12:46:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9de8a640 1 -- 172.21.15.60:0/1675587368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda78004d00 msgr2=0x7fda78005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9de8a640 1 --2- 172.21.15.60:0/1675587368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda78004d00 0x7fda78005180 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c110 tx=0x7fda780057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9d689640 1 -- 172.21.15.60:0/1675587368 reap_dead start 2023-10-16T12:46:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9de8a640 1 -- 172.21.15.60:0/1675587368 shutdown_connections 2023-10-16T12:46:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9de8a640 1 -- 172.21.15.60:0/1675587368 >> 172.21.15.60:0/1675587368 conn(0x7fda980cc230 msgr2=0x7fda980cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9de8a640 1 -- 172.21.15.60:0/1675587368 shutdown_connections 2023-10-16T12:46:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.557+0000 7fda9de8a640 1 -- 172.21.15.60:0/1675587368 wait complete. 2023-10-16T12:46:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.559+0000 7fda9de8a640 1 Processor -- start 2023-10-16T12:46:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.559+0000 7fda9de8a640 1 -- start start 2023-10-16T12:46:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda9de8a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda78004d00 0x7fda98155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda9de8a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 0x7fda98154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda9de8a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda98155a00 0x7fda98155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda9de8a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981e6a00 con 0x7fda9815bde0 2023-10-16T12:46:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda9de8a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980c59c0 con 0x7fda98155a00 2023-10-16T12:46:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda9de8a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda981563c0 con 0x7fda78004d00 2023-10-16T12:46:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 0x7fda98154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 0x7fda98154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37642/0 (socket says 172.21.15.60:37642) 2023-10-16T12:46:57.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.560+0000 7fda97fff640 1 -- 172.21.15.60:0/1041729710 learned_addr learned my addr 172.21.15.60:0/1041729710 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:57.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda9ce88640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda78004d00 0x7fda98155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda9d689640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda98155a00 0x7fda98155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda97fff640 1 -- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda78004d00 msgr2=0x7fda98155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda97fff640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda78004d00 0x7fda98155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda97fff640 1 -- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda98155a00 msgr2=0x7fda98155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda97fff640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda98155a00 0x7fda98155e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda97fff640 1 -- 172.21.15.60:0/1041729710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98158cf0 con 0x7fda9815bde0 2023-10-16T12:46:57.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda9ce88640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda78004d00 0x7fda98155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:57.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda97fff640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 0x7fda98154720 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fda8000f3d0 tx=0x7fda8000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:57.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80016620 con 0x7fda9815bde0 2023-10-16T12:46:57.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda800167a0 con 0x7fda9815bde0 2023-10-16T12:46:57.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80016b50 con 0x7fda9815bde0 2023-10-16T12:46:57.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda98154e90 con 0x7fda9815bde0 2023-10-16T12:46:57.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.561+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda980c5eb0 con 0x7fda9815bde0 2023-10-16T12:46:57.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.563+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda981e6b40 con 0x7fda9815bde0 2023-10-16T12:46:57.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.563+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 14) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fda80016cb0 con 0x7fda9815bde0 2023-10-16T12:46:57.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.565+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6814+0+0 (secure 0 0 0) 0x7fda8008a6e0 con 0x7fda9815bde0 2023-10-16T12:46:57.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.571+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda981e6b40 con 0x7fda9815bde0 2023-10-16T12:46:57.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.764+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda981e6b40 con 0x7fda9815bde0 2023-10-16T12:46:57.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.765+0000 7fda957fa640 1 -- 172.21.15.60:0/1041729710 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda981e6b40 con 0x7fda9815bde0 2023-10-16T12:46:57.768 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:46:57.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 msgr2=0x7fda98154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:57.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 0x7fda98154720 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fda8000f3d0 tx=0x7fda8000f890 comp rx=0 tx=0).stop 2023-10-16T12:46:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 shutdown_connections 2023-10-16T12:46:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda98155a00 0x7fda98155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda9815bde0 0x7fda98154720 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 --2- 172.21.15.60:0/1041729710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda78004d00 0x7fda98155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:57.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 >> 172.21.15.60:0/1041729710 conn(0x7fda980cc230 msgr2=0x7fda9815e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:57.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 shutdown_connections 2023-10-16T12:46:57.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:57.768+0000 7fda9de8a640 1 -- 172.21.15.60:0/1041729710 wait complete. 2023-10-16T12:46:57.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:46:58.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.026+0000 7f25a2a27640 1 Processor -- start 2023-10-16T12:46:58.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.026+0000 7f25a2a27640 1 -- start start 2023-10-16T12:46:58.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a2a27640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f259c06fb50 0x7f259c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a2a27640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f259c06c9e0 0x7f259c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a2a27640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c0ca560 con 0x7f259c0c9060 2023-10-16T12:46:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a2a27640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0c9950 con 0x7f259c06fb50 2023-10-16T12:46:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a2a27640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0c9c30 con 0x7f259c06c9e0 2023-10-16T12:46:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a1224640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f259c06fb50 0x7f259c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a1224640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f259c06fb50 0x7f259c0cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37656/0 (socket says 172.21.15.60:37656) 2023-10-16T12:46:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.028+0000 7f25a1224640 1 -- 172.21.15.60:0/1057159528 learned_addr learned my addr 172.21.15.60:0/1057159528 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.029+0000 7f25a2226640 1 --2- 172.21.15.60:0/1057159528 >> v2:172.21.15.144:3300/0 conn(0x7f259c06c9e0 0x7f259c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.029+0000 7f25a1224640 1 -- 172.21.15.60:0/1057159528 >> v2:172.21.15.144:3300/0 conn(0x7f259c06c9e0 msgr2=0x7f259c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.029+0000 7f25a1224640 1 --2- 172.21.15.60:0/1057159528 >> v2:172.21.15.144:3300/0 conn(0x7f259c06c9e0 0x7f259c0ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.029+0000 7f25a1224640 1 -- 172.21.15.60:0/1057159528 >> v1:172.21.15.60:6789/0 conn(0x7f259c0c9060 legacy=0x7f259c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.029+0000 7f25a1224640 1 -- 172.21.15.60:0/1057159528 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0c9f10 con 0x7f259c06fb50 2023-10-16T12:46:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.030+0000 7f25a1224640 1 --2- 172.21.15.60:0/1057159528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06fb50 0x7f259c0cc8a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f259c0ce550 tx=0x7f258c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a0921d7f2faeea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.030+0000 7f25a0a23640 1 -- 172.21.15.60:0/1057159528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c030860 con 0x7f259c06fb50 2023-10-16T12:46:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.030+0000 7f25a0a23640 1 -- 172.21.15.60:0/1057159528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c0309c0 con 0x7f259c06fb50 2023-10-16T12:46:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.030+0000 7f25a0a23640 1 -- 172.21.15.60:0/1057159528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c030d50 con 0x7f259c06fb50 2023-10-16T12:46:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 -- 172.21.15.60:0/1057159528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06fb50 msgr2=0x7f259c0cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 --2- 172.21.15.60:0/1057159528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06fb50 0x7f259c0cc8a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f259c0ce550 tx=0x7f258c02b9a0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 -- 172.21.15.60:0/1057159528 shutdown_connections 2023-10-16T12:46:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 --2- 172.21.15.60:0/1057159528 >> v2:172.21.15.144:3300/0 conn(0x7f259c06c9e0 0x7f259c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 --2- 172.21.15.60:0/1057159528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06fb50 0x7f259c0cc8a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 -- 172.21.15.60:0/1057159528 >> 172.21.15.60:0/1057159528 conn(0x7f259c0ce6a0 msgr2=0x7f259c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 -- 172.21.15.60:0/1057159528 shutdown_connections 2023-10-16T12:46:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.031+0000 7f25a2a27640 1 -- 172.21.15.60:0/1057159528 wait complete. 2023-10-16T12:46:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.033+0000 7f25a2a27640 1 Processor -- start 2023-10-16T12:46:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.033+0000 7f25a2a27640 1 -- start start 2023-10-16T12:46:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a2a27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 0x7f259c0c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a2a27640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f259c06fb50 0x7f259c155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a2a27640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f259c0c9060 0x7f259c155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a2a27640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c14f920 con 0x7f259c06c9e0 2023-10-16T12:46:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a2a27640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0c7280 con 0x7f259c0c9060 2023-10-16T12:46:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a2a27640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0c7560 con 0x7f259c06fb50 2023-10-16T12:46:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a1a25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 0x7f259c0c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a1a25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 0x7f259c0c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37668/0 (socket says 172.21.15.60:37668) 2023-10-16T12:46:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.034+0000 7f25a1a25640 1 -- 172.21.15.60:0/402684839 learned_addr learned my addr 172.21.15.60:0/402684839 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a1a25640 1 -- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f259c06fb50 msgr2=0x7f259c155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a1a25640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f259c06fb50 0x7f259c155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a1a25640 1 -- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f259c0c9060 msgr2=0x7f259c155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a1a25640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f259c0c9060 0x7f259c155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a1a25640 1 -- 172.21.15.60:0/402684839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c0094d0 con 0x7f259c06c9e0 2023-10-16T12:46:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a1a25640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 0x7f259c0c67c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f259800c730 tx=0x7f259800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598003b70 con 0x7f259c06c9e0 2023-10-16T12:46:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2598003cf0 con 0x7f259c06c9e0 2023-10-16T12:46:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598003ea0 con 0x7f259c06c9e0 2023-10-16T12:46:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c0c7840 con 0x7f259c06c9e0 2023-10-16T12:46:58.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.035+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c0c7e00 con 0x7f259c06c9e0 2023-10-16T12:46:58.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.036+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c04e9b0 con 0x7f259c06c9e0 2023-10-16T12:46:58.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.037+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 14) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f259800aa90 con 0x7f259c06c9e0 2023-10-16T12:46:58.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.038+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6814+0+0 (secure 0 0 0) 0x7f2598089440 con 0x7f259c06c9e0 2023-10-16T12:46:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.042+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2598054400 con 0x7f259c06c9e0 2023-10-16T12:46:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.108+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 15) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f259800ad40 con 0x7f259c06c9e0 2023-10-16T12:46:58.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.108+0000 7f25927fc640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2580070ae0 0x7f2580072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.114+0000 7f25a1224640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2580070ae0 0x7f2580072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.126+0000 7f25a1224640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2580070ae0 0x7f2580072ed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f258c036000 tx=0x7f258c03b040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:58.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.247+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f259c0b4620 con 0x7f259c06c9e0 2023-10-16T12:46:58.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.248+0000 7f25927fc640 1 -- 172.21.15.60:0/402684839 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f25980719b0 con 0x7f259c06c9e0 2023-10-16T12:46:58.250 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:46:58.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2580070ae0 msgr2=0x7f2580072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2580070ae0 0x7f2580072ed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f258c036000 tx=0x7f258c03b040 comp rx=0 tx=0).stop 2023-10-16T12:46:58.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 msgr2=0x7f259c0c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 0x7f259c0c67c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f259800c730 tx=0x7f259800cbf0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 shutdown_connections 2023-10-16T12:46:58.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2580070ae0 0x7f2580072ed0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f259c06fb50 0x7f259c155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f259c0c9060 0x7f259c155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 --2- 172.21.15.60:0/402684839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f259c06c9e0 0x7f259c0c67c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.250+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 >> 172.21.15.60:0/402684839 conn(0x7f259c0ce6a0 msgr2=0x7f259c0af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:58.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.251+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 shutdown_connections 2023-10-16T12:46:58.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.251+0000 7f25a2a27640 1 -- 172.21.15.60:0/402684839 wait complete. 2023-10-16T12:46:58.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:46:58.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.505+0000 7fb92def2640 1 Processor -- start 2023-10-16T12:46:58.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.505+0000 7fb92def2640 1 -- start start 2023-10-16T12:46:58.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.506+0000 7fb92def2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 0x7fb92815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.507+0000 7fb92def2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92815ed60 con 0x7fb92815ae80 2023-10-16T12:46:58.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.507+0000 7fb92def2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9280707e0 con 0x7fb9280ce390 2023-10-16T12:46:58.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.507+0000 7fb92def2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92806b810 con 0x7fb92815aaa0 2023-10-16T12:46:58.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.507+0000 7fb91ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb92815ae80 0x7fb9280cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34988/0 (socket says 172.21.15.60:34988) 2023-10-16T12:46:58.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.507+0000 7fb91ffff640 1 -- 172.21.15.60:0/2280911025 learned_addr learned my addr 172.21.15.60:0/2280911025 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:58.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.507+0000 7fb9277fe640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 0x7fb92815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592568326 0 0) 0x7fb92815ed60 con 0x7fb92815ae80 2023-10-16T12:46:58.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb914003520 con 0x7fb92815ae80 2023-10-16T12:46:58.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834471641 0 0) 0x7fb914003520 con 0x7fb92815ae80 2023-10-16T12:46:58.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 msgr2=0x7fb92815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:58.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 0x7fb92815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 >> v1:172.21.15.144:6789/0 conn(0x7fb9280ce390 legacy=0x7fb928070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92815ef90 con 0x7fb92815ae80 2023-10-16T12:46:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb910001670 con 0x7fb92815ae80 2023-10-16T12:46:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 >> v1:172.21.15.60:6789/0 conn(0x7fb92815ae80 legacy=0x7fb9280cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140039a0 0x7fb914003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9140047c0 0x7fb914004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 0x7fb914005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92806b810 con 0x7fb9140047c0 2023-10-16T12:46:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91400ba30 con 0x7fb9140039a0 2023-10-16T12:46:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91400bbf0 con 0x7fb92815aaa0 2023-10-16T12:46:58.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.508+0000 7fb91ffff640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9140047c0 0x7fb914004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.509+0000 7fb91ffff640 1 -- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 msgr2=0x7fb914005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:58.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.509+0000 7fb91ffff640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815aaa0 0x7fb914005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.509+0000 7fb9277fe640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140039a0 0x7fb914003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.509+0000 7fb91ffff640 1 -- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140039a0 msgr2=0x7fb914003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:58.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.509+0000 7fb91ffff640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140039a0 0x7fb914003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.509+0000 7fb91ffff640 1 -- 172.21.15.60:0/2280911025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914003520 con 0x7fb9140047c0 2023-10-16T12:46:58.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.510+0000 7fb91ffff640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9140047c0 0x7fb914004c40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c210 tx=0x7fb9140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8959dfa4316de9d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:58.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910005ad0 con 0x7fb9140047c0 2023-10-16T12:46:58.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910005c30 con 0x7fb9140047c0 2023-10-16T12:46:58.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb926ffd640 1 -- 172.21.15.60:0/2280911025 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91002c870 con 0x7fb9140047c0 2023-10-16T12:46:58.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb92def2640 1 -- 172.21.15.60:0/2280911025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9140047c0 msgr2=0x7fb914004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb92def2640 1 --2- 172.21.15.60:0/2280911025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9140047c0 0x7fb914004c40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c210 tx=0x7fb9140057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb927fff640 1 -- 172.21.15.60:0/2280911025 reap_dead start 2023-10-16T12:46:58.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb92def2640 1 -- 172.21.15.60:0/2280911025 shutdown_connections 2023-10-16T12:46:58.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.511+0000 7fb92def2640 1 -- 172.21.15.60:0/2280911025 >> 172.21.15.60:0/2280911025 conn(0x7fb9280cb410 msgr2=0x7fb9280cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:58.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.512+0000 7fb92def2640 1 -- 172.21.15.60:0/2280911025 shutdown_connections 2023-10-16T12:46:58.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.512+0000 7fb92def2640 1 -- 172.21.15.60:0/2280911025 wait complete. 2023-10-16T12:46:58.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.513+0000 7fb92def2640 1 Processor -- start 2023-10-16T12:46:58.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 -- start start 2023-10-16T12:46:58.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140047c0 0x7fb9281f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815d9e0 0x7fb9281f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 0x7fb9280b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281f2ec0 con 0x7fb9281f6390 2023-10-16T12:46:58.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281f3820 con 0x7fb9140047c0 2023-10-16T12:46:58.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.514+0000 7fb92def2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281f2860 con 0x7fb92815d9e0 2023-10-16T12:46:58.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 0x7fb9280b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 0x7fb9280b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37678/0 (socket says 172.21.15.60:37678) 2023-10-16T12:46:58.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 -- 172.21.15.60:0/806607359 learned_addr learned my addr 172.21.15.60:0/806607359 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:58.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb9277fe640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140047c0 0x7fb9281f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 -- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815d9e0 msgr2=0x7fb9281f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb91ffff640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815d9e0 0x7fb9281f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815d9e0 0x7fb9281f5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 -- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140047c0 msgr2=0x7fb9281f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140047c0 0x7fb9281f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb927fff640 1 -- 172.21.15.60:0/806607359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9281f6880 con 0x7fb9281f6390 2023-10-16T12:46:58.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.515+0000 7fb9277fe640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140047c0 0x7fb9281f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:58.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.516+0000 7fb927fff640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 0x7fb9280b1950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb9280ce2e0 tx=0x7fb92000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:58.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.516+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920005d40 con 0x7fb9281f6390 2023-10-16T12:46:58.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.516+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92000ced0 con 0x7fb9281f6390 2023-10-16T12:46:58.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.516+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9281f6b60 con 0x7fb9281f6390 2023-10-16T12:46:58.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.516+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000a7f0 con 0x7fb9281f6390 2023-10-16T12:46:58.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.516+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92815de60 con 0x7fb9281f6390 2023-10-16T12:46:58.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.517+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9281f6d50 con 0x7fb9281f6390 2023-10-16T12:46:58.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.518+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 15) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb92000a950 con 0x7fb9281f6390 2023-10-16T12:46:58.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.519+0000 7fb924ff9640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb90806ad10 0x7fb90806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:58.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.520+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6814+0+0 (secure 0 0 0) 0x7fb920088ad0 con 0x7fb9281f6390 2023-10-16T12:46:58.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.520+0000 7fb9277fe640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb90806ad10 0x7fb90806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:58.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.521+0000 7fb9277fe640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb90806ad10 0x7fb90806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb9280ca8a0 tx=0x7fb918008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:58.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.524+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9281f6d50 con 0x7fb9281f6390 2023-10-16T12:46:58.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.759+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb92815f530 con 0x7fb9281f6390 2023-10-16T12:46:58.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.760+0000 7fb924ff9640 1 -- 172.21.15.60:0/806607359 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fb92815f530 con 0x7fb9281f6390 2023-10-16T12:46:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:58.762 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:34:43.360823+0000","modified":"2023-10-16T12:34:46.440178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/3714850681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3714850681},{"type":"v1","addr":"172.21.15.60:6835","nonce":3714850681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:46:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2023-10-16T12:46:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb90806ad10 msgr2=0x7fb90806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb90806ad10 0x7fb90806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb9280ca8a0 tx=0x7fb918008040 comp rx=0 tx=0).stop 2023-10-16T12:46:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 msgr2=0x7fb9280b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 0x7fb9280b1950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb9280ce2e0 tx=0x7fb92000cb90 comp rx=0 tx=0).stop 2023-10-16T12:46:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 shutdown_connections 2023-10-16T12:46:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb90806ad10 0x7fb90806d1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb92815d9e0 0x7fb9281f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9281f6390 0x7fb9280b1950 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 --2- 172.21.15.60:0/806607359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9140047c0 0x7fb9281f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.762+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 >> 172.21.15.60:0/806607359 conn(0x7fb9280cb410 msgr2=0x7fb9280cc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.763+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 shutdown_connections 2023-10-16T12:46:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:58.763+0000 7fb92def2640 1 -- 172.21.15.60:0/806607359 wait complete. 2023-10-16T12:46:58.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:46:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.011+0000 7fbfafc6f640 1 Processor -- start 2023-10-16T12:46:59.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.011+0000 7fbfafc6f640 1 -- start start 2023-10-16T12:46:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.012+0000 7fbfafc6f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfa815ae80 0x7fbfa815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.012+0000 7fbfafc6f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.012+0000 7fbfafc6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa815ed60 con 0x7fbfa815aaa0 2023-10-16T12:46:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.012+0000 7fbfafc6f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa806b810 con 0x7fbfa815ae80 2023-10-16T12:46:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.012+0000 7fbfafc6f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80707e0 con 0x7fbfa80ce390 2023-10-16T12:46:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfa815ae80 0x7fbfa815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfa815ae80 0x7fbfa815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47106/0 (socket says 172.21.15.60:47106) 2023-10-16T12:46:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 -- 172.21.15.60:0/3366224495 learned_addr learned my addr 172.21.15.60:0/3366224495 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 -- 172.21.15.60:0/3366224495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 msgr2=0x7fbfa8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfae20c640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa8070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 -- 172.21.15.60:0/3366224495 >> v1:172.21.15.144:6789/0 conn(0x7fbfa815aaa0 legacy=0x7fbfa80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfad20a640 1 -- 172.21.15.60:0/3366224495 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa815ef90 con 0x7fbfa815ae80 2023-10-16T12:46:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.013+0000 7fbfae20c640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa8070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:46:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.015+0000 7fbfad20a640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa815e820 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80c59c0 tx=0x7fbf9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438a8ff9edbb967f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.015+0000 7fbfaca09640 1 -- 172.21.15.60:0/3366224495 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94031830 con 0x7fbfa815ae80 2023-10-16T12:46:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.015+0000 7fbfaca09640 1 -- 172.21.15.60:0/3366224495 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94031990 con 0x7fbfa815ae80 2023-10-16T12:46:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.015+0000 7fbfaca09640 1 -- 172.21.15.60:0/3366224495 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94031d20 con 0x7fbfa815ae80 2023-10-16T12:46:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 -- 172.21.15.60:0/3366224495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 msgr2=0x7fbfa815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa815e820 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80c59c0 tx=0x7fbf9402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 -- 172.21.15.60:0/3366224495 shutdown_connections 2023-10-16T12:46:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa8070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/3366224495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa815e820 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 -- 172.21.15.60:0/3366224495 >> 172.21.15.60:0/3366224495 conn(0x7fbfa80cb410 msgr2=0x7fbfa80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 -- 172.21.15.60:0/3366224495 shutdown_connections 2023-10-16T12:46:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.016+0000 7fbfafc6f640 1 -- 172.21.15.60:0/3366224495 wait complete. 2023-10-16T12:46:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.018+0000 7fbfafc6f640 1 Processor -- start 2023-10-16T12:46:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.018+0000 7fbfafc6f640 1 -- start start 2023-10-16T12:46:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfafc6f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfafc6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfa815aaa0 0x7fbfa80b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfafc6f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa81f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfafc6f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80c2e20 con 0x7fbfa815ae80 2023-10-16T12:46:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfafc6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa81f6710 con 0x7fbfa815aaa0 2023-10-16T12:46:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfafc6f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa81f7070 con 0x7fbfa80ce390 2023-10-16T12:46:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfae20c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa81f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfae20c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa81f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47116/0 (socket says 172.21.15.60:47116) 2023-10-16T12:46:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.019+0000 7fbfae20c640 1 -- 172.21.15.60:0/681004858 learned_addr learned my addr 172.21.15.60:0/681004858 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfada0b640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfad20a640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfa815aaa0 0x7fbfa80b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfae20c640 1 -- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 msgr2=0x7fbfa81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfae20c640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfae20c640 1 -- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfa815aaa0 msgr2=0x7fbfa80b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfae20c640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfa815aaa0 0x7fbfa80b0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfae20c640 1 -- 172.21.15.60:0/681004858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf940094d0 con 0x7fbfa815ae80 2023-10-16T12:46:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfae20c640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa81f6b30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000c820 tx=0x7fbfa000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.020+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0007a30 con 0x7fbfa815ae80 2023-10-16T12:46:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.021+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa0007b90 con 0x7fbfa815ae80 2023-10-16T12:46:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.021+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa000ce10 con 0x7fbfa815ae80 2023-10-16T12:46:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.021+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa80c35c0 con 0x7fbfa815ae80 2023-10-16T12:46:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.021+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa80c3000 con 0x7fbfa815ae80 2023-10-16T12:46:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.022+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa81f6850 con 0x7fbfa815ae80 2023-10-16T12:46:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.023+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 15) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbfa00113f0 con 0x7fbfa815ae80 2023-10-16T12:46:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.024+0000 7fbfa67fc640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbf8c06acc0 0x7fbf8c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.024+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6814+0+0 (secure 0 0 0) 0x7fbfa00898c0 con 0x7fbfa815ae80 2023-10-16T12:46:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.025+0000 7fbfada0b640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbf8c06acc0 0x7fbf8c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.026+0000 7fbfada0b640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbf8c06acc0 0x7fbf8c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c005520 tx=0x7fbf9c009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.029+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbfa81f6850 con 0x7fbfa815ae80 2023-10-16T12:46:59.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.119+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 16) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbfa0054b60 con 0x7fbfa815ae80 2023-10-16T12:46:59.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.234+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfa81f6850 con 0x7fbfa815ae80 2023-10-16T12:46:59.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.236+0000 7fbfa67fc640 1 -- 172.21.15.60:0/681004858 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18227 (secure 0 0 0) 0x7fbfa81f6850 con 0x7fbfa815ae80 2023-10-16T12:46:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:59.240 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":55,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:46:57.079658+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":225,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/674545145":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2596584944":"2023-10-17T12:46:57.079584+0000","172.21.15.144:6832/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/696341254":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/1714573694":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6832/696341254":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/3946760462":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/680953068":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2301614647":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2412206403":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/25541997":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/1199236332":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/2619332853":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/2315449004":"2023-10-17T12:46:57.079584+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2},{"begin":11,"length":2},{"begin":14,"length":2},{"begin":17,"length":3},{"begin":23,"length":7},{"begin":35,"length":4},{"begin":40,"length":1},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":58,"length":2},{"begin":62,"length":3},{"begin":66,"length":5},{"begin":76,"length":1},{"begin":78,"length":2},{"begin":81,"length":3},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":92,"length":2},{"begin":96,"length":4},{"begin":101,"length":2},{"begin":104,"length":3},{"begin":109,"length":1},{"begin":111,"length":3},{"begin":159,"length":67}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:46:59.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbf8c06acc0 msgr2=0x7fbf8c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbf8c06acc0 0x7fbf8c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c005520 tx=0x7fbf9c009040 comp rx=0 tx=0).stop 2023-10-16T12:46:59.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 msgr2=0x7fbfa81f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa81f6b30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000c820 tx=0x7fbfa000cce0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 shutdown_connections 2023-10-16T12:46:59.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbf8c06acc0 0x7fbf8c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfa815ae80 0x7fbfa81f6b30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfa815aaa0 0x7fbfa80b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 --2- 172.21.15.60:0/681004858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfa80ce390 0x7fbfa81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 >> 172.21.15.60:0/681004858 conn(0x7fbfa80cb410 msgr2=0x7fbfa80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:59.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 shutdown_connections 2023-10-16T12:46:59.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.240+0000 7fbfafc6f640 1 -- 172.21.15.60:0/681004858 wait complete. 2023-10-16T12:46:59.260 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:46:59.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:46:59.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.476+0000 7ff6b62ad640 1 Processor -- start 2023-10-16T12:46:59.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.476+0000 7ff6b62ad640 1 -- start start 2023-10-16T12:46:59.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.478+0000 7ff6b62ad640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff6b00ac890 0x7ff6b00accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.478+0000 7ff6b62ad640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b0160a90 con 0x7ff6b015dde0 2023-10-16T12:46:59.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.478+0000 7ff6b62ad640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b00ad200 con 0x7ff6b015c9f0 2023-10-16T12:46:59.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.478+0000 7ff6b62ad640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00cdec0 con 0x7ff6b00ac890 2023-10-16T12:46:59.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.478+0000 7ff6b4aaa640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff6b015dde0 0x7ff6b00709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52160/0 (socket says 172.21.15.60:52160) 2023-10-16T12:46:59.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.478+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/1398272884 learned_addr learned my addr 172.21.15.60:0/1398272884 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:46:59.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6b5aac640 1 --2- 172.21.15.60:0/1398272884 >> v2:172.21.15.144:3300/0 conn(0x7ff6b00ac890 0x7ff6b00accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494647931 0 0) 0x7ff6b0160a90 con 0x7ff6b015dde0 2023-10-16T12:46:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff69c003520 con 0x7ff6b015dde0 2023-10-16T12:46:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953931468 0 0) 0x7ff69c003520 con 0x7ff6b015dde0 2023-10-16T12:46:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 >> v2:172.21.15.144:3300/0 conn(0x7ff6b00ac890 msgr2=0x7ff6b00accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 --2- 172.21.15.60:0/1398272884 >> v2:172.21.15.144:3300/0 conn(0x7ff6b00ac890 0x7ff6b00accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 >> v1:172.21.15.144:6789/0 conn(0x7ff6b015c9f0 legacy=0x7ff6b00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00cd900 con 0x7ff6b015dde0 2023-10-16T12:46:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff6a0001670 con 0x7ff6b015dde0 2023-10-16T12:46:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 >> v1:172.21.15.60:6789/0 conn(0x7ff6b015dde0 legacy=0x7ff6b00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c003970 0x7ff69c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff69c0047c0 0x7ff69c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.479+0000 7ff6a7fff640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff69c005180 0x7ff69c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c00c050 con 0x7ff69c0047c0 2023-10-16T12:46:59.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c00c190 con 0x7ff69c005180 2023-10-16T12:46:59.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c00c350 con 0x7ff69c003970 2023-10-16T12:46:59.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff69c0047c0 0x7ff69c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b52ab640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c003970 0x7ff69c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c003970 msgr2=0x7ff69c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c003970 0x7ff69c003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b5aac640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff69c005180 0x7ff69c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff69c005180 msgr2=0x7ff69c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:59.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff69c005180 0x7ff69c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/1398272884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c003520 con 0x7ff69c0047c0 2023-10-16T12:46:59.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.480+0000 7ff6b5aac640 1 -- 172.21.15.60:0/1398272884 reap_dead start 2023-10-16T12:46:59.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.481+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff69c0047c0 0x7ff69c004c40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000c240 tx=0x7ff69c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27be4e11def903d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0005ad0 con 0x7ff69c0047c0 2023-10-16T12:46:59.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a0005c30 con 0x7ff69c0047c0 2023-10-16T12:46:59.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6a7fff640 1 -- 172.21.15.60:0/1398272884 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a002c850 con 0x7ff69c0047c0 2023-10-16T12:46:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 -- 172.21.15.60:0/1398272884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff69c0047c0 msgr2=0x7ff69c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff69c0047c0 0x7ff69c004c40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff6a000c240 tx=0x7ff69c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 -- 172.21.15.60:0/1398272884 shutdown_connections 2023-10-16T12:46:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/1398272884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff69c0047c0 0x7ff69c004c40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 -- 172.21.15.60:0/1398272884 >> 172.21.15.60:0/1398272884 conn(0x7ff6b00ce6a0 msgr2=0x7ff6b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 -- 172.21.15.60:0/1398272884 shutdown_connections 2023-10-16T12:46:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.482+0000 7ff6b62ad640 1 -- 172.21.15.60:0/1398272884 wait complete. 2023-10-16T12:46:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.484+0000 7ff6b62ad640 1 Processor -- start 2023-10-16T12:46:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.484+0000 7ff6b62ad640 1 -- start start 2023-10-16T12:46:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.485+0000 7ff6b62ad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c0047c0 0x7ff6b01f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.485+0000 7ff6b62ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 0x7ff6b0069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.485+0000 7ff6b62ad640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6b006a490 0x7ff6b00af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.485+0000 7ff6b62ad640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01f30e0 con 0x7ff6b00c2750 2023-10-16T12:46:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.485+0000 7ff6b62ad640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01f2460 con 0x7ff6b006a490 2023-10-16T12:46:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.485+0000 7ff6b62ad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01f2740 con 0x7ff69c0047c0 2023-10-16T12:46:59.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 0x7ff6b0069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 0x7ff6b0069f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47138/0 (socket says 172.21.15.60:47138) 2023-10-16T12:46:59.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/2595296555 learned_addr learned my addr 172.21.15.60:0/2595296555 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:59.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b52ab640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c0047c0 0x7ff6b01f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b5aac640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6b006a490 0x7ff6b00af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c0047c0 msgr2=0x7ff6b01f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c0047c0 0x7ff6b01f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6b006a490 msgr2=0x7ff6b00af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6b006a490 0x7ff6b00af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.486+0000 7ff6b4aaa640 1 -- 172.21.15.60:0/2595296555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b01f2a20 con 0x7ff6b00c2750 2023-10-16T12:46:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.487+0000 7ff6b4aaa640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 0x7ff6b0069f50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff6a002c3f0 tx=0x7ff6a002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.487+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a003c040 con 0x7ff6b00c2750 2023-10-16T12:46:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.487+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a002cc60 con 0x7ff6b00c2750 2023-10-16T12:46:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.487+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0007070 con 0x7ff6b00c2750 2023-10-16T12:46:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.487+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b01f2d00 con 0x7ff6b00c2750 2023-10-16T12:46:59.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.487+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b00afaa0 con 0x7ff6b00c2750 2023-10-16T12:46:59.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.489+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b004e9b0 con 0x7ff6b00c2750 2023-10-16T12:46:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.490+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff6a0032070 con 0x7ff6b00c2750 2023-10-16T12:46:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.491+0000 7ff6a5ffb640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff69406af50 0x7ff69406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.491+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6814+0+0 (secure 0 0 0) 0x7ff6a00aaf80 con 0x7ff6b00c2750 2023-10-16T12:46:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.492+0000 7ff6b52ab640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff69406af50 0x7ff69406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.493+0000 7ff6b52ab640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff69406af50 0x7ff69406d410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00c2dc0 tx=0x7ff6ac009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.496+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6a0075d00 con 0x7ff6b00c2750 2023-10-16T12:46:59.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.732+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b00afc90 con 0x7ff6b00c2750 2023-10-16T12:46:59.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.733+0000 7ff6a5ffb640 1 -- 172.21.15.60:0/2595296555 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7ff6a0079320 con 0x7ff6b00c2750 2023-10-16T12:46:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:46:59.736 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:34:43.360823+0000","modified":"2023-10-16T12:34:46.440178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/3714850681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3714850681},{"type":"v1","addr":"172.21.15.60:6835","nonce":3714850681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:46:59.736 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2023-10-16T12:46:59.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff69406af50 msgr2=0x7ff69406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff69406af50 0x7ff69406d410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00c2dc0 tx=0x7ff6ac009040 comp rx=0 tx=0).stop 2023-10-16T12:46:59.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 msgr2=0x7ff6b0069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 0x7ff6b0069f50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff6a002c3f0 tx=0x7ff6a002e070 comp rx=0 tx=0).stop 2023-10-16T12:46:59.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 shutdown_connections 2023-10-16T12:46:59.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff69406af50 0x7ff69406d410 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6b006a490 0x7ff6b00af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6b00c2750 0x7ff6b0069f50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 --2- 172.21.15.60:0/2595296555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff69c0047c0 0x7ff6b01f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 >> 172.21.15.60:0/2595296555 conn(0x7ff6b00ce6a0 msgr2=0x7ff6b0070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:59.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 shutdown_connections 2023-10-16T12:46:59.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.736+0000 7ff6b62ad640 1 -- 172.21.15.60:0/2595296555 wait complete. 2023-10-16T12:46:59.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:46:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.983+0000 7fce47bc0640 1 Processor -- start 2023-10-16T12:46:59.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.983+0000 7fce47bc0640 1 -- start start 2023-10-16T12:46:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce47bc0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fce40159f00 0x7fce400b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce47bc0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fce4015d880 0x7fce400b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce47bc0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4006f7f0 con 0x7fce40159b20 2023-10-16T12:46:59.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce47bc0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4006c5e0 con 0x7fce4015d880 2023-10-16T12:46:59.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce47bc0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4006c8c0 con 0x7fce40159f00 2023-10-16T12:46:59.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce4615d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fce4015d880 0x7fce400b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce4615d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fce4015d880 0x7fce400b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47144/0 (socket says 172.21.15.60:47144) 2023-10-16T12:46:59.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce4615d640 1 -- 172.21.15.60:0/2887977495 learned_addr learned my addr 172.21.15.60:0/2887977495 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:59.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.985+0000 7fce4595c640 1 --1- 172.21.15.60:0/2887977495 >> v1:172.21.15.60:6789/0 conn(0x7fce40159b20 0x7fce4006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52170/0 (socket says 172.21.15.60:52170) 2023-10-16T12:46:59.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.986+0000 7fce4515b640 1 --2- 172.21.15.60:0/2887977495 >> v2:172.21.15.144:3300/0 conn(0x7fce40159f00 0x7fce400b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.986+0000 7fce4615d640 1 -- 172.21.15.60:0/2887977495 >> v2:172.21.15.144:3300/0 conn(0x7fce40159f00 msgr2=0x7fce400b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:46:59.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.986+0000 7fce4615d640 1 --2- 172.21.15.60:0/2887977495 >> v2:172.21.15.144:3300/0 conn(0x7fce40159f00 0x7fce400b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.986+0000 7fce4615d640 1 -- 172.21.15.60:0/2887977495 >> v1:172.21.15.60:6789/0 conn(0x7fce40159b20 legacy=0x7fce4006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.986+0000 7fce4615d640 1 -- 172.21.15.60:0/2887977495 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4006cba0 con 0x7fce4015d880 2023-10-16T12:46:59.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.986+0000 7fce4495a640 1 -- 172.21.15.60:0/2887977495 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259509731 0 0) 0x7fce4006f7f0 con 0x7fce40159b20 2023-10-16T12:46:59.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.987+0000 7fce4615d640 1 --2- 172.21.15.60:0/2887977495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce4015d880 0x7fce400b0950 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fce4015d770 tx=0x7fce3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235b0d03cffb3513 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:46:59.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.987+0000 7fce4495a640 1 -- 172.21.15.60:0/2887977495 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38030860 con 0x7fce4015d880 2023-10-16T12:46:59.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.987+0000 7fce4495a640 1 -- 172.21.15.60:0/2887977495 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce380309c0 con 0x7fce4015d880 2023-10-16T12:46:59.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.987+0000 7fce4495a640 1 -- 172.21.15.60:0/2887977495 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38030d50 con 0x7fce4015d880 2023-10-16T12:46:59.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 -- 172.21.15.60:0/2887977495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce4015d880 msgr2=0x7fce400b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2887977495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce4015d880 0x7fce400b0950 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fce4015d770 tx=0x7fce3802b9a0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 -- 172.21.15.60:0/2887977495 shutdown_connections 2023-10-16T12:46:59.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2887977495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce4015d880 0x7fce400b0950 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2887977495 >> v2:172.21.15.144:3300/0 conn(0x7fce40159f00 0x7fce400b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:46:59.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 -- 172.21.15.60:0/2887977495 >> 172.21.15.60:0/2887977495 conn(0x7fce400cb410 msgr2=0x7fce400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:46:59.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 -- 172.21.15.60:0/2887977495 shutdown_connections 2023-10-16T12:46:59.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.988+0000 7fce47bc0640 1 -- 172.21.15.60:0/2887977495 wait complete. 2023-10-16T12:46:59.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.990+0000 7fce47bc0640 1 Processor -- start 2023-10-16T12:46:59.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.990+0000 7fce47bc0640 1 -- start start 2023-10-16T12:46:59.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce47bc0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 0x7fce40157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce47bc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce40159f00 0x7fce4014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce47bc0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce4015d880 0x7fce400b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:46:59.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce47bc0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400c8530 con 0x7fce40159b20 2023-10-16T12:46:59.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce47bc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce40070b20 con 0x7fce40159f00 2023-10-16T12:46:59.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce47bc0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce400ceac0 con 0x7fce4015d880 2023-10-16T12:46:59.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce4595c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 0x7fce40157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce4595c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 0x7fce40157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47146/0 (socket says 172.21.15.60:47146) 2023-10-16T12:46:59.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.991+0000 7fce4595c640 1 -- 172.21.15.60:0/2105334843 learned_addr learned my addr 172.21.15.60:0/2105334843 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:46:59.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4515b640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce40159f00 0x7fce4014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4615d640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce4015d880 0x7fce400b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:46:59.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4595c640 1 -- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce4015d880 msgr2=0x7fce400b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:46:59.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4595c640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce4015d880 0x7fce400b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4595c640 1 -- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce40159f00 msgr2=0x7fce4014cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4595c640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce40159f00 0x7fce4014cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4595c640 1 -- 172.21.15.60:0/2105334843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380094d0 con 0x7fce40159b20 2023-10-16T12:47:00.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4615d640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce4015d880 0x7fce400b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:00.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce4595c640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 0x7fce40157d30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c7c0 tx=0x7fce3c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.992+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c00a640 con 0x7fce40159b20 2023-10-16T12:47:00.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.993+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3c00a7c0 con 0x7fce40159b20 2023-10-16T12:47:00.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.993+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c00ab70 con 0x7fce40159b20 2023-10-16T12:47:00.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.993+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce400ceda0 con 0x7fce40159b20 2023-10-16T12:47:00.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.993+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4015a310 con 0x7fce40159b20 2023-10-16T12:47:00.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.994+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4006c270 con 0x7fce40159b20 2023-10-16T12:47:00.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.995+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fce3c00acd0 con 0x7fce40159b20 2023-10-16T12:47:00.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.996+0000 7fce367fc640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fce2406af00 0x7fce2406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.996+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6814+0+0 (secure 0 0 0) 0x7fce3c0890f0 con 0x7fce40159b20 2023-10-16T12:47:00.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.996+0000 7fce4515b640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fce2406af00 0x7fce2406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:46:59.998+0000 7fce4515b640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fce2406af00 0x7fce2406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fce4015d430 tx=0x7fce30008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.000+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce3c053c70 con 0x7fce40159b20 2023-10-16T12:47:00.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.249+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce4004e160 con 0x7fce40159b20 2023-10-16T12:47:00.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.250+0000 7fce367fc640 1 -- 172.21.15.60:0/2105334843 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fce3c057290 con 0x7fce40159b20 2023-10-16T12:47:00.253 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:00.253 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:34:43.360823+0000","modified":"2023-10-16T12:34:46.440178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5004":{"gid":5004,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/3714850681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3714850681},{"type":"v1","addr":"172.21.15.60:6835","nonce":3714850681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:47:00.253 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2023-10-16T12:47:00.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fce2406af00 msgr2=0x7fce2406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fce2406af00 0x7fce2406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fce4015d430 tx=0x7fce30008040 comp rx=0 tx=0).stop 2023-10-16T12:47:00.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 msgr2=0x7fce40157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 0x7fce40157d30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c7c0 tx=0x7fce3c00cc80 comp rx=0 tx=0).stop 2023-10-16T12:47:00.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 shutdown_connections 2023-10-16T12:47:00.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fce2406af00 0x7fce2406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce4015d880 0x7fce400b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce40159f00 0x7fce4014cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 --2- 172.21.15.60:0/2105334843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce40159b20 0x7fce40157d30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 >> 172.21.15.60:0/2105334843 conn(0x7fce400cb410 msgr2=0x7fce4006d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:00.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 shutdown_connections 2023-10-16T12:47:00.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.253+0000 7fce47bc0640 1 -- 172.21.15.60:0/2105334843 wait complete. 2023-10-16T12:47:00.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:47:00.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.497+0000 7f9ac257c640 1 Processor -- start 2023-10-16T12:47:00.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.497+0000 7f9ac257c640 1 -- start start 2023-10-16T12:47:00.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.498+0000 7f9ac257c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc06c3f0 0x7f9abc06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.498+0000 7f9ac257c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9abc15e0c0 0x7f9abc0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.498+0000 7f9ac257c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9abc15e540 0x7f9abc0b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.498+0000 7f9ac257c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0cd630 con 0x7f9abc15e0c0 2023-10-16T12:47:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.498+0000 7f9ac257c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0ce1b0 con 0x7f9abc15e540 2023-10-16T12:47:00.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.498+0000 7f9ac257c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc06cd10 con 0x7f9abc06c3f0 2023-10-16T12:47:00.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9abc15e0c0 0x7f9abc0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9abc15e0c0 0x7f9abc0b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47158/0 (socket says 172.21.15.60:47158) 2023-10-16T12:47:00.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 -- 172.21.15.60:0/40637 learned_addr learned my addr 172.21.15.60:0/40637 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:00.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9ac0b19640 1 --2- 172.21.15.60:0/40637 >> v2:172.21.15.144:3300/0 conn(0x7f9abc15e540 0x7f9abc0b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abbfff640 1 --2- 172.21.15.60:0/40637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc06c3f0 0x7f9abc06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 -- 172.21.15.60:0/40637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc06c3f0 msgr2=0x7f9abc06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:00.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 --2- 172.21.15.60:0/40637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc06c3f0 0x7f9abc06c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 -- 172.21.15.60:0/40637 >> v2:172.21.15.144:3300/0 conn(0x7f9abc15e540 msgr2=0x7f9abc0b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:00.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 --2- 172.21.15.60:0/40637 >> v2:172.21.15.144:3300/0 conn(0x7f9abc15e540 0x7f9abc0b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.499+0000 7f9abb7fe640 1 -- 172.21.15.60:0/40637 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc0cd880 con 0x7f9abc15e0c0 2023-10-16T12:47:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.501+0000 7f9abb7fe640 1 --2- 172.21.15.60:0/40637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc0b4730 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9abc160db0 tx=0x7f9ab002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad04260b22be4bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.501+0000 7f9abaffd640 1 -- 172.21.15.60:0/40637 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0031870 con 0x7f9abc15e0c0 2023-10-16T12:47:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.501+0000 7f9abaffd640 1 -- 172.21.15.60:0/40637 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab00319d0 con 0x7f9abc15e0c0 2023-10-16T12:47:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.501+0000 7f9abaffd640 1 -- 172.21.15.60:0/40637 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0031d60 con 0x7f9abc15e0c0 2023-10-16T12:47:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 -- 172.21.15.60:0/40637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 msgr2=0x7f9abc0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 --2- 172.21.15.60:0/40637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc0b4730 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9abc160db0 tx=0x7f9ab002b910 comp rx=0 tx=0).stop 2023-10-16T12:47:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 -- 172.21.15.60:0/40637 shutdown_connections 2023-10-16T12:47:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 --2- 172.21.15.60:0/40637 >> v2:172.21.15.144:3300/0 conn(0x7f9abc15e540 0x7f9abc0b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 --2- 172.21.15.60:0/40637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc0b4730 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 --2- 172.21.15.60:0/40637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc06c3f0 0x7f9abc06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 -- 172.21.15.60:0/40637 >> 172.21.15.60:0/40637 conn(0x7f9abc0d2cb0 msgr2=0x7f9abc0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 -- 172.21.15.60:0/40637 shutdown_connections 2023-10-16T12:47:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.502+0000 7f9ac257c640 1 -- 172.21.15.60:0/40637 wait complete. 2023-10-16T12:47:00.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.504+0000 7f9ac257c640 1 Processor -- start 2023-10-16T12:47:00.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.504+0000 7f9ac257c640 1 -- start start 2023-10-16T12:47:00.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9ac257c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc06c3f0 0x7f9abc15e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9ac257c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc14d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9ac257c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15e540 0x7f9abc0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9ac257c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc158da0 con 0x7f9abc15e0c0 2023-10-16T12:47:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9ac257c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc152e70 con 0x7f9abc06c3f0 2023-10-16T12:47:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9ac257c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc15ef00 con 0x7f9abc15e540 2023-10-16T12:47:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9abb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc14d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9abb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc14d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47174/0 (socket says 172.21.15.60:47174) 2023-10-16T12:47:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.505+0000 7f9abb7fe640 1 -- 172.21.15.60:0/1622319604 learned_addr learned my addr 172.21.15.60:0/1622319604 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abbfff640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc06c3f0 0x7f9abc15e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abb7fe640 1 -- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15e540 msgr2=0x7f9abc0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abb7fe640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15e540 0x7f9abc0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abb7fe640 1 -- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc06c3f0 msgr2=0x7f9abc15e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abb7fe640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc06c3f0 0x7f9abc15e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abb7fe640 1 -- 172.21.15.60:0/1622319604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00094d0 con 0x7f9abc15e0c0 2023-10-16T12:47:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9abb7fe640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc14d810 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9ab002b950 tx=0x7f9ab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.506+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab003ad30 con 0x7f9abc15e0c0 2023-10-16T12:47:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.507+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab00443f0 con 0x7f9abc15e0c0 2023-10-16T12:47:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.507+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab00447a0 con 0x7f9abc15e0c0 2023-10-16T12:47:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.507+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc0c64e0 con 0x7f9abc15e0c0 2023-10-16T12:47:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.507+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc152820 con 0x7f9abc15e0c0 2023-10-16T12:47:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.508+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc153000 con 0x7f9abc15e0c0 2023-10-16T12:47:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.509+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9ab004d450 con 0x7f9abc15e0c0 2023-10-16T12:47:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.510+0000 7f9ab8ff9640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9aa806af00 0x7f9aa806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.510+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6814+0+0 (secure 0 0 0) 0x7f9ab002d040 con 0x7f9abc15e0c0 2023-10-16T12:47:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.510+0000 7f9abbfff640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9aa806af00 0x7f9aa806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.512+0000 7f9abbfff640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9aa806af00 0x7f9aa806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9aac004450 tx=0x7f9aac003a30 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.514+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ab0076c00 con 0x7f9abc15e0c0 2023-10-16T12:47:00.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.711+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9abc04e160 con 0x7f9abc15e0c0 2023-10-16T12:47:00.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.713+0000 7f9ab8ff9640 1 -- 172.21.15.60:0/1622319604 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18227 (secure 0 0 0) 0x7f9ab0035070 con 0x7f9abc15e0c0 2023-10-16T12:47:00.716 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:00.716 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":56,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:46:59.686420+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:34:41.099103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-16T12:34:42.392121+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":225,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/674545145":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2596584944":"2023-10-17T12:46:57.079584+0000","172.21.15.144:6832/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/696341254":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/1714573694":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6832/696341254":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/3946760462":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/680953068":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2301614647":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2412206403":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/25541997":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/1199236332":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/2619332853":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/2315449004":"2023-10-17T12:46:57.079584+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2},{"begin":11,"length":2},{"begin":14,"length":2},{"begin":17,"length":3},{"begin":23,"length":7},{"begin":35,"length":4},{"begin":40,"length":1},{"begin":43,"length":2},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":58,"length":2},{"begin":62,"length":3},{"begin":66,"length":5},{"begin":76,"length":1},{"begin":78,"length":2},{"begin":81,"length":3},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":92,"length":2},{"begin":96,"length":4},{"begin":101,"length":2},{"begin":104,"length":3},{"begin":109,"length":1},{"begin":111,"length":3},{"begin":159,"length":67}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:47:00.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9aa806af00 msgr2=0x7f9aa806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9aa806af00 0x7f9aa806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9aac004450 tx=0x7f9aac003a30 comp rx=0 tx=0).stop 2023-10-16T12:47:00.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 msgr2=0x7f9abc14d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc14d810 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9ab002b950 tx=0x7f9ab00046c0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 shutdown_connections 2023-10-16T12:47:00.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9aa806af00 0x7f9aa806d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15e540 0x7f9abc0c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc15e0c0 0x7f9abc14d810 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 --2- 172.21.15.60:0/1622319604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc06c3f0 0x7f9abc15e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 >> 172.21.15.60:0/1622319604 conn(0x7f9abc0d2cb0 msgr2=0x7f9abc0b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:00.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 shutdown_connections 2023-10-16T12:47:00.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.716+0000 7f9ac257c640 1 -- 172.21.15.60:0/1622319604 wait complete. 2023-10-16T12:47:00.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:47:00.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.968+0000 7fdb3491c640 1 Processor -- start 2023-10-16T12:47:00.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.968+0000 7fdb3491c640 1 -- start start 2023-10-16T12:47:00.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb3491c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb3006c040 0x7fdb3006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb3491c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb300ad970 0x7fdb3015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb3491c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3015ebc0 0x7fdb3015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb3491c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3006c960 con 0x7fdb3006c040 2023-10-16T12:47:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb3491c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3015b9d0 con 0x7fdb300ad970 2023-10-16T12:47:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb3491c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3015bcb0 con 0x7fdb3015ebc0 2023-10-16T12:47:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb2ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb3006c040 0x7fdb3006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb2ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb3006c040 0x7fdb3006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47180/0 (socket says 172.21.15.60:47180) 2023-10-16T12:47:00.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb2ed9d640 1 -- 172.21.15.60:0/1027983366 learned_addr learned my addr 172.21.15.60:0/1027983366 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:00.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.970+0000 7fdb2f59e640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3015ebc0 0x7fdb3015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2ed9d640 1 -- 172.21.15.60:0/1027983366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3015ebc0 msgr2=0x7fdb3015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2ed9d640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3015ebc0 0x7fdb3015f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2ed9d640 1 -- 172.21.15.60:0/1027983366 >> v2:172.21.15.144:3300/0 conn(0x7fdb300ad970 msgr2=0x7fdb3015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2e59c640 1 --2- 172.21.15.60:0/1027983366 >> v2:172.21.15.144:3300/0 conn(0x7fdb300ad970 0x7fdb3015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2ed9d640 1 --2- 172.21.15.60:0/1027983366 >> v2:172.21.15.144:3300/0 conn(0x7fdb300ad970 0x7fdb3015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2ed9d640 1 -- 172.21.15.60:0/1027983366 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3015bf90 con 0x7fdb3006c040 2023-10-16T12:47:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2f59e640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3015ebc0 0x7fdb3015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.971+0000 7fdb2e59c640 1 --2- 172.21.15.60:0/1027983366 >> v2:172.21.15.144:3300/0 conn(0x7fdb300ad970 0x7fdb3015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.972+0000 7fdb2ed9d640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3006c040 0x7fdb3006c420 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdb300cb060 tx=0x7fdb2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53f07a1c52333ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.972+0000 7fdb2dd9b640 1 -- 172.21.15.60:0/1027983366 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2002ca80 con 0x7fdb3006c040 2023-10-16T12:47:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.972+0000 7fdb2dd9b640 1 -- 172.21.15.60:0/1027983366 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2002cbe0 con 0x7fdb3006c040 2023-10-16T12:47:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.972+0000 7fdb2dd9b640 1 -- 172.21.15.60:0/1027983366 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200343f0 con 0x7fdb3006c040 2023-10-16T12:47:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 -- 172.21.15.60:0/1027983366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3006c040 msgr2=0x7fdb3006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3006c040 0x7fdb3006c420 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdb300cb060 tx=0x7fdb2002bee0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 -- 172.21.15.60:0/1027983366 shutdown_connections 2023-10-16T12:47:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3015ebc0 0x7fdb3015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 --2- 172.21.15.60:0/1027983366 >> v2:172.21.15.144:3300/0 conn(0x7fdb300ad970 0x7fdb3015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 --2- 172.21.15.60:0/1027983366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3006c040 0x7fdb3006c420 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 -- 172.21.15.60:0/1027983366 >> 172.21.15.60:0/1027983366 conn(0x7fdb300ce6a0 msgr2=0x7fdb300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 -- 172.21.15.60:0/1027983366 shutdown_connections 2023-10-16T12:47:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.973+0000 7fdb3491c640 1 -- 172.21.15.60:0/1027983366 wait complete. 2023-10-16T12:47:00.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.975+0000 7fdb3491c640 1 Processor -- start 2023-10-16T12:47:00.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.976+0000 7fdb3491c640 1 -- start start 2023-10-16T12:47:00.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb3491c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3006c040 0x7fdb300c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb3491c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb300ad970 0x7fdb300c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb3491c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 0x7fdb30156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb3491c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300c8cf0 con 0x7fdb3015ebc0 2023-10-16T12:47:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb3491c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300c9b50 con 0x7fdb300ad970 2023-10-16T12:47:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb3491c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300c8f40 con 0x7fdb3006c040 2023-10-16T12:47:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb2f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 0x7fdb30156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb2f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 0x7fdb30156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47182/0 (socket says 172.21.15.60:47182) 2023-10-16T12:47:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.977+0000 7fdb2f59e640 1 -- 172.21.15.60:0/4222713334 learned_addr learned my addr 172.21.15.60:0/4222713334 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2ed9d640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3006c040 0x7fdb300c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2e59c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb300ad970 0x7fdb300c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2f59e640 1 -- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3006c040 msgr2=0x7fdb300c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2f59e640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3006c040 0x7fdb300c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2f59e640 1 -- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb300ad970 msgr2=0x7fdb300c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:00.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2f59e640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb300ad970 0x7fdb300c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:00.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2f59e640 1 -- 172.21.15.60:0/4222713334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb20009b30 con 0x7fdb3015ebc0 2023-10-16T12:47:00.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.978+0000 7fdb2f59e640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 0x7fdb30156980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800c7f0 tx=0x7fdb2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.979+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28007820 con 0x7fdb3015ebc0 2023-10-16T12:47:00.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.979+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb28007980 con 0x7fdb3015ebc0 2023-10-16T12:47:00.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.979+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28007d70 con 0x7fdb3015ebc0 2023-10-16T12:47:00.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.979+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb300c9220 con 0x7fdb3015ebc0 2023-10-16T12:47:00.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.979+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb300c97e0 con 0x7fdb3015ebc0 2023-10-16T12:47:00.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.980+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3006b330 con 0x7fdb3015ebc0 2023-10-16T12:47:00.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.981+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdb280113f0 con 0x7fdb3015ebc0 2023-10-16T12:47:00.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.982+0000 7fdb177fe640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fdb1c06af00 0x7fdb1c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:00.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.982+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6814+0+0 (secure 0 0 0) 0x7fdb28089e30 con 0x7fdb3015ebc0 2023-10-16T12:47:00.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.983+0000 7fdb2ed9d640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fdb1c06af00 0x7fdb1c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:00.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.984+0000 7fdb2ed9d640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fdb1c06af00 0x7fdb1c06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdb20035000 tx=0x7fdb20030000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:00.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:00.987+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb28054a20 con 0x7fdb3015ebc0 2023-10-16T12:47:01.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.146+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb28054df0 con 0x7fdb3015ebc0 2023-10-16T12:47:01.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.234+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdb300c9cc0 con 0x7fdb3015ebc0 2023-10-16T12:47:01.237 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:47:01.235+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:47:01.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.238+0000 7fdb177fe640 1 -- 172.21.15.60:0/4222713334 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdb28071350 con 0x7fdb3015ebc0 2023-10-16T12:47:01.241 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:47:01.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fdb1c06af00 msgr2=0x7fdb1c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:01.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fdb1c06af00 0x7fdb1c06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdb20035000 tx=0x7fdb20030000 comp rx=0 tx=0).stop 2023-10-16T12:47:01.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 msgr2=0x7fdb30156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:01.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 0x7fdb30156980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800c7f0 tx=0x7fdb2800ccb0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 shutdown_connections 2023-10-16T12:47:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fdb1c06af00 0x7fdb1c06d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb3015ebc0 0x7fdb30156980 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb300ad970 0x7fdb300c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 --2- 172.21.15.60:0/4222713334 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb3006c040 0x7fdb300c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.241+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 >> 172.21.15.60:0/4222713334 conn(0x7fdb300ce6a0 msgr2=0x7fdb300cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.242+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 shutdown_connections 2023-10-16T12:47:01.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.242+0000 7fdb3491c640 1 -- 172.21.15.60:0/4222713334 wait complete. 2023-10-16T12:47:01.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:47:01.352 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:47:01.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.473+0000 7f8e85701640 1 Processor -- start 2023-10-16T12:47:01.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.473+0000 7f8e85701640 1 -- start start 2023-10-16T12:47:01.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.474+0000 7f8e85701640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8e800caa80 0x7f8e80069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:01.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.474+0000 7f8e85701640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e8006a2e0 0x7f8e801650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:01.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.474+0000 7f8e85701640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e800caef0 con 0x7f8e800addb0 2023-10-16T12:47:01.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.474+0000 7f8e85701640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e80161a50 con 0x7f8e8006a2e0 2023-10-16T12:47:01.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.474+0000 7f8e85701640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8006a6f0 con 0x7f8e800caa80 2023-10-16T12:47:01.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e8006a2e0 0x7f8e801650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:01.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7effd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8e800addb0 0x7f8e800c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52186/0 (socket says 172.21.15.60:52186) 2023-10-16T12:47:01.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7effd640 1 -- 172.21.15.60:0/3753953658 learned_addr learned my addr 172.21.15.60:0/3753953658 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e77fff640 1 --2- 172.21.15.60:0/3753953658 >> v2:172.21.15.144:3300/0 conn(0x7f8e800caa80 0x7f8e80069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/3753953658 >> v2:172.21.15.144:3300/0 conn(0x7f8e800caa80 msgr2=0x7f8e80069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7f7fe640 1 --2- 172.21.15.60:0/3753953658 >> v2:172.21.15.144:3300/0 conn(0x7f8e800caa80 0x7f8e80069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/3753953658 >> v1:172.21.15.60:6789/0 conn(0x7f8e800addb0 legacy=0x7f8e800c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/3753953658 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e800cb140 con 0x7f8e8006a2e0 2023-10-16T12:47:01.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.475+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/3753953658 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466275111 0 0) 0x7f8e800caef0 con 0x7f8e800addb0 2023-10-16T12:47:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.477+0000 7f8e7f7fe640 1 --2- 172.21.15.60:0/3753953658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e801650d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8e800cdbe0 tx=0x7f8e7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6e2e1b8571c336 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.477+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/3753953658 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7802cc80 con 0x7f8e8006a2e0 2023-10-16T12:47:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.477+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/3753953658 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7802cde0 con 0x7f8e8006a2e0 2023-10-16T12:47:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.477+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/3753953658 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e78034660 con 0x7f8e8006a2e0 2023-10-16T12:47:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 -- 172.21.15.60:0/3753953658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 msgr2=0x7f8e801650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 --2- 172.21.15.60:0/3753953658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e801650d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8e800cdbe0 tx=0x7f8e7802bed0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 -- 172.21.15.60:0/3753953658 shutdown_connections 2023-10-16T12:47:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 --2- 172.21.15.60:0/3753953658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e801650d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 --2- 172.21.15.60:0/3753953658 >> v2:172.21.15.144:3300/0 conn(0x7f8e800caa80 0x7f8e80069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 -- 172.21.15.60:0/3753953658 >> 172.21.15.60:0/3753953658 conn(0x7f8e800ce9b0 msgr2=0x7f8e800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 -- 172.21.15.60:0/3753953658 shutdown_connections 2023-10-16T12:47:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.478+0000 7f8e85701640 1 -- 172.21.15.60:0/3753953658 wait complete. 2023-10-16T12:47:01.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.480+0000 7f8e85701640 1 Processor -- start 2023-10-16T12:47:01.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.480+0000 7f8e85701640 1 -- start start 2023-10-16T12:47:01.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e85701640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e80151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e85701640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e800addb0 0x7f8e800c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e85701640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e800caa80 0x7f8e800c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e85701640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e80151760 con 0x7f8e8006a2e0 2023-10-16T12:47:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e85701640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e80151b10 con 0x7f8e800caa80 2023-10-16T12:47:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e85701640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e80070580 con 0x7f8e800addb0 2023-10-16T12:47:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e7effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e80151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e7effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e80151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47212/0 (socket says 172.21.15.60:47212) 2023-10-16T12:47:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e7effd640 1 -- 172.21.15.60:0/458425484 learned_addr learned my addr 172.21.15.60:0/458425484 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.481+0000 7f8e77fff640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e800addb0 0x7f8e800c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e7effd640 1 -- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e800addb0 msgr2=0x7f8e800c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e7effd640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e800addb0 0x7f8e800c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e7effd640 1 -- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e800caa80 msgr2=0x7f8e800c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e7effd640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e800caa80 0x7f8e800c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e7effd640 1 -- 172.21.15.60:0/458425484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78009a70 con 0x7f8e8006a2e0 2023-10-16T12:47:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e7effd640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e80151220 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c500 tx=0x7f8e7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70007560 con 0x7f8e8006a2e0 2023-10-16T12:47:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e700047e0 con 0x7f8e8006a2e0 2023-10-16T12:47:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e80070860 con 0x7f8e8006a2e0 2023-10-16T12:47:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70004b90 con 0x7f8e8006a2e0 2023-10-16T12:47:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.482+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e80070e20 con 0x7f8e8006a2e0 2023-10-16T12:47:01.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.483+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8004e160 con 0x7f8e8006a2e0 2023-10-16T12:47:01.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.485+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8e70004cf0 con 0x7f8e8006a2e0 2023-10-16T12:47:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.485+0000 7f8e777fe640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8e6006af90 0x7f8e6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.486+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f8e70088f70 con 0x7f8e8006a2e0 2023-10-16T12:47:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.486+0000 7f8e77fff640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8e6006af90 0x7f8e6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.487+0000 7f8e77fff640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8e6006af90 0x7f8e6006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006fd0 tx=0x7f8e68008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:01.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.490+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e70053c20 con 0x7f8e8006a2e0 2023-10-16T12:47:01.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:01.723+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8e8006c5b0 con 0x7f8e8006a2e0 2023-10-16T12:47:02.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.242+0000 7f8e777fe640 1 -- 172.21.15.60:0/458425484 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8e700180a0 con 0x7f8e8006a2e0 2023-10-16T12:47:02.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8e6006af90 msgr2=0x7f8e6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:02.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8e6006af90 0x7f8e6006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006fd0 tx=0x7f8e68008040 comp rx=0 tx=0).stop 2023-10-16T12:47:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 msgr2=0x7f8e80151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e80151220 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c500 tx=0x7f8e7000c9c0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 shutdown_connections 2023-10-16T12:47:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8e6006af90 0x7f8e6006d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e800addb0 0x7f8e800c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e800caa80 0x7f8e800c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 --2- 172.21.15.60:0/458425484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e8006a2e0 0x7f8e80151220 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.245+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 >> 172.21.15.60:0/458425484 conn(0x7f8e800ce9b0 msgr2=0x7f8e800ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.246+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 shutdown_connections 2023-10-16T12:47:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.246+0000 7f8e85701640 1 -- 172.21.15.60:0/458425484 wait complete. 2023-10-16T12:47:02.269 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:47:02.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.492+0000 7fc049155640 1 Processor -- start 2023-10-16T12:47:02.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.492+0000 7fc049155640 1 -- start start 2023-10-16T12:47:02.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.493+0000 7fc049155640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc04406c040 0x7fc04406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:02.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.493+0000 7fc049155640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc0440ad970 0x7fc04415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:02.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.493+0000 7fc049155640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04406c960 con 0x7fc04415ebc0 2023-10-16T12:47:02.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.493+0000 7fc049155640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04415b9d0 con 0x7fc04406c040 2023-10-16T12:47:02.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.493+0000 7fc049155640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04415bcb0 con 0x7fc0440ad970 2023-10-16T12:47:02.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc04406c040 0x7fc04406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:02.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc04406c040 0x7fc04406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47218/0 (socket says 172.21.15.60:47218) 2023-10-16T12:47:02.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 -- 172.21.15.60:0/1082074599 learned_addr learned my addr 172.21.15.60:0/1082074599 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:02.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 -- 172.21.15.60:0/1082074599 >> v2:172.21.15.144:3300/0 conn(0x7fc0440ad970 msgr2=0x7fc04415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:02.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 --2- 172.21.15.60:0/1082074599 >> v2:172.21.15.144:3300/0 conn(0x7fc0440ad970 0x7fc04415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 -- 172.21.15.60:0/1082074599 >> v1:172.21.15.144:6789/0 conn(0x7fc04415ebc0 legacy=0x7fc0440cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:02.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.494+0000 7fc043fff640 1 -- 172.21.15.60:0/1082074599 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04415bf90 con 0x7fc04406c040 2023-10-16T12:47:02.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.495+0000 7fc043fff640 1 --2- 172.21.15.60:0/1082074599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04406c040 0x7fc04406c420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc0440cb060 tx=0x7fc03402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1c6dd2186cdff9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:02.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc042ffd640 1 -- 172.21.15.60:0/1082074599 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034031870 con 0x7fc04406c040 2023-10-16T12:47:02.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc042ffd640 1 -- 172.21.15.60:0/1082074599 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0340319d0 con 0x7fc04406c040 2023-10-16T12:47:02.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc042ffd640 1 -- 172.21.15.60:0/1082074599 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034031d60 con 0x7fc04406c040 2023-10-16T12:47:02.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc049155640 1 -- 172.21.15.60:0/1082074599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04406c040 msgr2=0x7fc04406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:02.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc049155640 1 --2- 172.21.15.60:0/1082074599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04406c040 0x7fc04406c420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc0440cb060 tx=0x7fc03402b910 comp rx=0 tx=0).stop 2023-10-16T12:47:02.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc049155640 1 -- 172.21.15.60:0/1082074599 shutdown_connections 2023-10-16T12:47:02.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc049155640 1 --2- 172.21.15.60:0/1082074599 >> v2:172.21.15.144:3300/0 conn(0x7fc0440ad970 0x7fc04415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.496+0000 7fc049155640 1 --2- 172.21.15.60:0/1082074599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04406c040 0x7fc04406c420 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.497+0000 7fc049155640 1 -- 172.21.15.60:0/1082074599 >> 172.21.15.60:0/1082074599 conn(0x7fc0440ce6a0 msgr2=0x7fc0440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:02.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.497+0000 7fc049155640 1 -- 172.21.15.60:0/1082074599 shutdown_connections 2023-10-16T12:47:02.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.497+0000 7fc049155640 1 -- 172.21.15.60:0/1082074599 wait complete. 2023-10-16T12:47:02.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.498+0000 7fc049155640 1 Processor -- start 2023-10-16T12:47:02.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 -- start start 2023-10-16T12:47:02.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc04406c040 0x7fc0440a8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:02.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0440ad970 0x7fc0440afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:02.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 0x7fc04414acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:02.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc044148720 con 0x7fc04415ebc0 2023-10-16T12:47:02.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0441477c0 con 0x7fc0440ad970 2023-10-16T12:47:02.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.499+0000 7fc049155640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc044147aa0 con 0x7fc04406c040 2023-10-16T12:47:02.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 0x7fc04414acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:02.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 0x7fc04414acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47222/0 (socket says 172.21.15.60:47222) 2023-10-16T12:47:02.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 -- 172.21.15.60:0/3136191908 learned_addr learned my addr 172.21.15.60:0/3136191908 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:02.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc043fff640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc04406c040 0x7fc0440a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:02.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 -- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc04406c040 msgr2=0x7fc0440a8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:02.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc04406c040 0x7fc0440a8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 -- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0440ad970 msgr2=0x7fc0440afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:02.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc0437fe640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0440ad970 0x7fc0440afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:02.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0440ad970 0x7fc0440afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:02.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.500+0000 7fc048954640 1 -- 172.21.15.60:0/3136191908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340094d0 con 0x7fc04415ebc0 2023-10-16T12:47:02.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.501+0000 7fc048954640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 0x7fc04414acb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00c780 tx=0x7fc03c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:02.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.501+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c003d80 con 0x7fc04415ebc0 2023-10-16T12:47:02.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.501+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c00ce90 con 0x7fc04415ebc0 2023-10-16T12:47:02.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.501+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0052b0 con 0x7fc04415ebc0 2023-10-16T12:47:02.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.501+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc044147d80 con 0x7fc04415ebc0 2023-10-16T12:47:02.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.501+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc044148340 con 0x7fc04415ebc0 2023-10-16T12:47:02.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.502+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04406b330 con 0x7fc04415ebc0 2023-10-16T12:47:02.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.503+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc03c005410 con 0x7fc04415ebc0 2023-10-16T12:47:02.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.505+0000 7fc040ff9640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc03006af90 0x7fc03006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:02.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.505+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc03c088f30 con 0x7fc04415ebc0 2023-10-16T12:47:02.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.505+0000 7fc043fff640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc03006af90 0x7fc03006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:02.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.507+0000 7fc043fff640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc03006af90 0x7fc03006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc034036000 tx=0x7fc03403c040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:02.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.509+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc03c053be0 con 0x7fc04415ebc0 2023-10-16T12:47:02.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:02.707+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc044148860 con 0x7fc04415ebc0 2023-10-16T12:47:03.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.252+0000 7fc040ff9640 1 -- 172.21.15.60:0/3136191908 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc044148860 con 0x7fc04415ebc0 2023-10-16T12:47:03.255 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:47:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc03006af90 msgr2=0x7fc03006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc03006af90 0x7fc03006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc034036000 tx=0x7fc03403c040 comp rx=0 tx=0).stop 2023-10-16T12:47:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 msgr2=0x7fc04414acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 0x7fc04414acb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00c780 tx=0x7fc03c00cc40 comp rx=0 tx=0).stop 2023-10-16T12:47:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 shutdown_connections 2023-10-16T12:47:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc03006af90 0x7fc03006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc04415ebc0 0x7fc04414acb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0440ad970 0x7fc0440afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 --2- 172.21.15.60:0/3136191908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc04406c040 0x7fc0440a8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.259+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 >> 172.21.15.60:0/3136191908 conn(0x7fc0440ce6a0 msgr2=0x7fc0440cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:03.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.260+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 shutdown_connections 2023-10-16T12:47:03.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.260+0000 7fc049155640 1 -- 172.21.15.60:0/3136191908 wait complete. 2023-10-16T12:47:03.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:47:03.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.516+0000 7efcf62e0640 1 Processor -- start 2023-10-16T12:47:03.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.516+0000 7efcf62e0640 1 -- start start 2023-10-16T12:47:03.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.517+0000 7efcf62e0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efcf00b0360 0x7efcf00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:03.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.517+0000 7efcf62e0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf015b310 con 0x7efcf015ac10 2023-10-16T12:47:03.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.517+0000 7efcf62e0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf006f640 con 0x7efcf00ca510 2023-10-16T12:47:03.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.517+0000 7efcf62e0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf015f800 con 0x7efcf00b0360 2023-10-16T12:47:03.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efceffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efcf015ac10 0x7efcf006c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52198/0 (socket says 172.21.15.60:52198) 2023-10-16T12:47:03.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efceffff640 1 -- 172.21.15.60:0/3142241942 learned_addr learned my addr 172.21.15.60:0/3142241942 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:03.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efce7fff640 1 --2- 172.21.15.60:0/3142241942 >> v2:172.21.15.60:3300/0 conn(0x7efcf00b0360 0x7efcf00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:03.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efcef7fe640 1 -- 172.21.15.60:0/3142241942 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238762785 0 0) 0x7efcf015b310 con 0x7efcf015ac10 2023-10-16T12:47:03.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efcef7fe640 1 -- 172.21.15.60:0/3142241942 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcdc003520 con 0x7efcf015ac10 2023-10-16T12:47:03.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efce7fff640 1 -- 172.21.15.60:0/3142241942 >> v1:172.21.15.144:6789/0 conn(0x7efcf00ca510 legacy=0x7efcf00ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:03.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efce7fff640 1 -- 172.21.15.60:0/3142241942 >> v1:172.21.15.60:6789/0 conn(0x7efcf015ac10 legacy=0x7efcf006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:03.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.518+0000 7efce7fff640 1 -- 172.21.15.60:0/3142241942 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf015b5b0 con 0x7efcf00b0360 2023-10-16T12:47:03.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.519+0000 7efce7fff640 1 --2- 172.21.15.60:0/3142241942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf00b07f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efcd80097a0 tx=0x7efcd802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2303657adc2465a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:03.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcef7fe640 1 -- 172.21.15.60:0/3142241942 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd802e070 con 0x7efcf00b0360 2023-10-16T12:47:03.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcef7fe640 1 -- 172.21.15.60:0/3142241942 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd80315a0 con 0x7efcf00b0360 2023-10-16T12:47:03.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcef7fe640 1 -- 172.21.15.60:0/3142241942 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8031930 con 0x7efcf00b0360 2023-10-16T12:47:03.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcf62e0640 1 -- 172.21.15.60:0/3142241942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 msgr2=0x7efcf00b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:03.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3142241942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf00b07f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efcd80097a0 tx=0x7efcd802b840 comp rx=0 tx=0).stop 2023-10-16T12:47:03.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcf62e0640 1 -- 172.21.15.60:0/3142241942 shutdown_connections 2023-10-16T12:47:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3142241942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf00b07f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.520+0000 7efcf62e0640 1 -- 172.21.15.60:0/3142241942 >> 172.21.15.60:0/3142241942 conn(0x7efcf00cb360 msgr2=0x7efcf00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:03.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.521+0000 7efcf62e0640 1 -- 172.21.15.60:0/3142241942 shutdown_connections 2023-10-16T12:47:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.521+0000 7efcf62e0640 1 -- 172.21.15.60:0/3142241942 wait complete. 2023-10-16T12:47:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.522+0000 7efcf62e0640 1 Processor -- start 2023-10-16T12:47:03.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 -- start start 2023-10-16T12:47:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf0155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:03.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcf00ca510 0x7efcf01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcf015ac10 0x7efcf00c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00c7380 con 0x7efcf00b0360 2023-10-16T12:47:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00b2ef0 con 0x7efcf00ca510 2023-10-16T12:47:03.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.523+0000 7efcf62e0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00ccf10 con 0x7efcf015ac10 2023-10-16T12:47:03.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf0155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:03.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf0155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47244/0 (socket says 172.21.15.60:47244) 2023-10-16T12:47:03.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 -- 172.21.15.60:0/3721247602 learned_addr learned my addr 172.21.15.60:0/3721247602 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:03.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 -- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcf015ac10 msgr2=0x7efcf00c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:03.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcf015ac10 0x7efcf00c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 -- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcf00ca510 msgr2=0x7efcf01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:03.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efce7fff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcf00ca510 0x7efcf01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:03.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcf00ca510 0x7efcf01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:03.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efceffff640 1 -- 172.21.15.60:0/3721247602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd8009400 con 0x7efcf00b0360 2023-10-16T12:47:03.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.524+0000 7efce7fff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcf00ca510 0x7efcf01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:03.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.525+0000 7efceffff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf0155ba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efce000c390 tx=0x7efce000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:03.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.525+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0005ce0 con 0x7efcf00b0360 2023-10-16T12:47:03.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.525+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0005e40 con 0x7efcf00b0360 2023-10-16T12:47:03.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.525+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce00072f0 con 0x7efcf00b0360 2023-10-16T12:47:03.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.525+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf01560e0 con 0x7efcf00b0360 2023-10-16T12:47:03.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.525+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf00ccba0 con 0x7efcf00b0360 2023-10-16T12:47:03.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.526+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf00cc6f0 con 0x7efcf00b0360 2023-10-16T12:47:03.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.527+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efce0007450 con 0x7efcf00b0360 2023-10-16T12:47:03.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.529+0000 7efced7fa640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7efcd006af90 0x7efcd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.529+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6002+0+0 (secure 0 0 0) 0x7efce0088ef0 con 0x7efcf00b0360 2023-10-16T12:47:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.529+0000 7efce7fff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7efcd006af90 0x7efcd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.531+0000 7efce7fff640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7efcd006af90 0x7efcd006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efcd8036000 tx=0x7efcd803d040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.533+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efce0053d50 con 0x7efcf00b0360 2023-10-16T12:47:03.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:03.729+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efcf004e160 con 0x7efcf00b0360 2023-10-16T12:47:04.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.255+0000 7efced7fa640 1 -- 172.21.15.60:0/3721247602 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7efce0017020 con 0x7efcf00b0360 2023-10-16T12:47:04.258 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:47:04.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.259+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7efcd006af90 msgr2=0x7efcd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.259+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7efcd006af90 0x7efcd006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efcd8036000 tx=0x7efcd803d040 comp rx=0 tx=0).stop 2023-10-16T12:47:04.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.259+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 msgr2=0x7efcf0155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.259+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf0155ba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efce000c390 tx=0x7efce000c850 comp rx=0 tx=0).stop 2023-10-16T12:47:04.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.260+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 shutdown_connections 2023-10-16T12:47:04.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.260+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7efcd006af90 0x7efcd006d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.260+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcf015ac10 0x7efcf00c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.260+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcf00ca510 0x7efcf01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.260+0000 7efcf62e0640 1 --2- 172.21.15.60:0/3721247602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00b0360 0x7efcf0155ba0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.260+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 >> 172.21.15.60:0/3721247602 conn(0x7efcf00cb360 msgr2=0x7efcf00cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:04.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.261+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 shutdown_connections 2023-10-16T12:47:04.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.261+0000 7efcf62e0640 1 -- 172.21.15.60:0/3721247602 wait complete. 2023-10-16T12:47:04.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:47:04.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.508+0000 7f93b9867640 1 Processor -- start 2023-10-16T12:47:04.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.508+0000 7f93b9867640 1 -- start start 2023-10-16T12:47:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.509+0000 7f93b9867640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b40cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b9867640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93b40b0ea0 0x7f93b40cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b9867640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b406be30 con 0x7f93b40b43c0 2023-10-16T12:47:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b9867640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b406ca00 con 0x7f93b40b0ea0 2023-10-16T12:47:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b9867640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40ada10 con 0x7f93b40b3ec0 2023-10-16T12:47:04.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b8865640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b40cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b8865640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b40cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:56566/0 (socket says 172.21.15.60:56566) 2023-10-16T12:47:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.510+0000 7f93b8865640 1 -- 172.21.15.60:0/2768975776 learned_addr learned my addr 172.21.15.60:0/2768975776 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.511+0000 7f93abfff640 1 --2- 172.21.15.60:0/2768975776 >> v2:172.21.15.144:3300/0 conn(0x7f93b40b0ea0 0x7f93b40cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.511+0000 7f93b8865640 1 -- 172.21.15.60:0/2768975776 >> v2:172.21.15.144:3300/0 conn(0x7f93b40b0ea0 msgr2=0x7f93b40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.511+0000 7f93b8865640 1 --2- 172.21.15.60:0/2768975776 >> v2:172.21.15.144:3300/0 conn(0x7f93b40b0ea0 0x7f93b40cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.511+0000 7f93b8865640 1 -- 172.21.15.60:0/2768975776 >> v1:172.21.15.144:6789/0 conn(0x7f93b40b43c0 legacy=0x7f93b415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.511+0000 7f93b8865640 1 -- 172.21.15.60:0/2768975776 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b406c060 con 0x7f93b40b3ec0 2023-10-16T12:47:04.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.512+0000 7f93b8865640 1 --2- 172.21.15.60:0/2768975776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b40cd260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009870 tx=0x7f93b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bda21b0d44b68a1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:04.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.513+0000 7f93ab7fe640 1 -- 172.21.15.60:0/2768975776 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0031640 con 0x7f93b40b3ec0 2023-10-16T12:47:04.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.513+0000 7f93ab7fe640 1 -- 172.21.15.60:0/2768975776 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b00317a0 con 0x7f93b40b3ec0 2023-10-16T12:47:04.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.513+0000 7f93ab7fe640 1 -- 172.21.15.60:0/2768975776 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0031b30 con 0x7f93b40b3ec0 2023-10-16T12:47:04.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.513+0000 7f93b9867640 1 -- 172.21.15.60:0/2768975776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 msgr2=0x7f93b40cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.513+0000 7f93b9867640 1 --2- 172.21.15.60:0/2768975776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b40cd260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009870 tx=0x7f93b002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.514+0000 7f93b9867640 1 -- 172.21.15.60:0/2768975776 shutdown_connections 2023-10-16T12:47:04.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.514+0000 7f93b9867640 1 --2- 172.21.15.60:0/2768975776 >> v2:172.21.15.144:3300/0 conn(0x7f93b40b0ea0 0x7f93b40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.514+0000 7f93b9867640 1 --2- 172.21.15.60:0/2768975776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b40cd260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.514+0000 7f93b9867640 1 -- 172.21.15.60:0/2768975776 >> 172.21.15.60:0/2768975776 conn(0x7f93b40b0170 msgr2=0x7f93b40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:04.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.514+0000 7f93b9867640 1 -- 172.21.15.60:0/2768975776 shutdown_connections 2023-10-16T12:47:04.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.514+0000 7f93b9867640 1 -- 172.21.15.60:0/2768975776 wait complete. 2023-10-16T12:47:04.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.515+0000 7f93b9867640 1 Processor -- start 2023-10-16T12:47:04.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.516+0000 7f93b9867640 1 -- start start 2023-10-16T12:47:04.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9867640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93b40b0ea0 0x7f93b40c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9867640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9867640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 0x7f93b4145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9867640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b41f2830 con 0x7f93b40b43c0 2023-10-16T12:47:04.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9867640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b415aaf0 con 0x7f93b40b0ea0 2023-10-16T12:47:04.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9867640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b415add0 con 0x7f93b40b3ec0 2023-10-16T12:47:04.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 0x7f93b4145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 0x7f93b4145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47260/0 (socket says 172.21.15.60:47260) 2023-10-16T12:47:04.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b9066640 1 -- 172.21.15.60:0/87278728 learned_addr learned my addr 172.21.15.60:0/87278728 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:04.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93abfff640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.517+0000 7f93b8865640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93b40b0ea0 0x7f93b40c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9066640 1 -- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 msgr2=0x7f93b415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9066640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b415be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9066640 1 -- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93b40b0ea0 msgr2=0x7f93b40c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9066640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93b40b0ea0 0x7f93b40c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9066640 1 -- 172.21.15.60:0/87278728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00094d0 con 0x7f93b40b43c0 2023-10-16T12:47:04.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b8865640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93b40b0ea0 0x7f93b40c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:04.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9066640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 0x7f93b4145980 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00c730 tx=0x7f93ac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:04.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac003b70 con 0x7f93b40b43c0 2023-10-16T12:47:04.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac003cd0 con 0x7f93b40b43c0 2023-10-16T12:47:04.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00aa90 con 0x7f93b40b43c0 2023-10-16T12:47:04.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b415b0b0 con 0x7f93b40b43c0 2023-10-16T12:47:04.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.518+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b415b670 con 0x7f93b40b43c0 2023-10-16T12:47:04.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.520+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b404e9b0 con 0x7f93b40b43c0 2023-10-16T12:47:04.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.520+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93ac00abf0 con 0x7f93b40b43c0 2023-10-16T12:47:04.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.521+0000 7f93a97fa640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f93a006af90 0x7f93a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.522+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f93ac0895a0 con 0x7f93b40b43c0 2023-10-16T12:47:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.522+0000 7f93b8865640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f93a006af90 0x7f93a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.523+0000 7f93b8865640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f93a006af90 0x7f93a006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93b0036000 tx=0x7f93b003c040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:04.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.526+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93ac054510 con 0x7f93b40b43c0 2023-10-16T12:47:04.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.724+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f93b406abd0 con 0x7f93b40b43c0 2023-10-16T12:47:04.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.731+0000 7f93a97fa640 1 -- 172.21.15.60:0/87278728 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7f93ac057b30 con 0x7f93b40b43c0 2023-10-16T12:47:04.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.733+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f93a006af90 msgr2=0x7f93a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.733+0000 7f93b9867640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f93a006af90 0x7f93a006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f93b0036000 tx=0x7f93b003c040 comp rx=0 tx=0).stop 2023-10-16T12:47:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.733+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 msgr2=0x7f93b4145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.733+0000 7f93b9867640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 0x7f93b4145980 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00c730 tx=0x7f93ac00cbf0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 shutdown_connections 2023-10-16T12:47:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f93a006af90 0x7f93a006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93b40b3ec0 0x7f93b415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93b40b43c0 0x7f93b4145980 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 --2- 172.21.15.60:0/87278728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93b40b0ea0 0x7f93b40c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 >> 172.21.15.60:0/87278728 conn(0x7f93b40b0170 msgr2=0x7f93b40acaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 shutdown_connections 2023-10-16T12:47:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.734+0000 7f93b9867640 1 -- 172.21.15.60:0/87278728 wait complete. 2023-10-16T12:47:04.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:47:04.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.966+0000 7f30b8000640 1 Processor -- start 2023-10-16T12:47:04.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.967+0000 7f30b8000640 1 -- start start 2023-10-16T12:47:04.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.968+0000 7f30b8000640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.968+0000 7f30b8000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00cb030 con 0x7f30b00c9ca0 2023-10-16T12:47:04.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.968+0000 7f30b8000640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00c91c0 con 0x7f30b00caac0 2023-10-16T12:47:04.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.968+0000 7f30b8000640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b00c94a0 con 0x7f30b015ad70 2023-10-16T12:47:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.969+0000 7f30b559b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f30b00c9ca0 0x7f30b00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52208/0 (socket says 172.21.15.60:52208) 2023-10-16T12:47:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.969+0000 7f30b559b640 1 -- 172.21.15.60:0/2096869518 learned_addr learned my addr 172.21.15.60:0/2096869518 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.969+0000 7f30b5d9c640 1 --2- 172.21.15.60:0/2096869518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.969+0000 7f30b4d9a640 1 -- 172.21.15.60:0/2096869518 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491740089 0 0) 0x7f30b00cb030 con 0x7f30b00c9ca0 2023-10-16T12:47:04.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.969+0000 7f30b4d9a640 1 -- 172.21.15.60:0/2096869518 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f309c003520 con 0x7f30b00c9ca0 2023-10-16T12:47:04.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.970+0000 7f30b5d9c640 1 -- 172.21.15.60:0/2096869518 >> v1:172.21.15.144:6789/0 conn(0x7f30b00caac0 legacy=0x7f30b00ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.970+0000 7f30b5d9c640 1 -- 172.21.15.60:0/2096869518 >> v1:172.21.15.60:6789/0 conn(0x7f30b00c9ca0 legacy=0x7f30b00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.970+0000 7f30b4d9a640 1 -- 172.21.15.60:0/2096869518 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848441263 0 0) 0x7f309c003520 con 0x7f30b00c9ca0 2023-10-16T12:47:04.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.970+0000 7f30b5d9c640 1 -- 172.21.15.60:0/2096869518 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b00c9780 con 0x7f30b015ad70 2023-10-16T12:47:04.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.971+0000 7f30b5d9c640 1 --2- 172.21.15.60:0/2096869518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b0069da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0097d0 tx=0x7f30ac02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d0bf653e49016b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:04.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.971+0000 7f30b4d9a640 1 -- 172.21.15.60:0/2096869518 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0001670 con 0x7f30b015ad70 2023-10-16T12:47:04.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b4d9a640 1 -- 172.21.15.60:0/2096869518 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac02e050 con 0x7f30b015ad70 2023-10-16T12:47:04.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b4d9a640 1 -- 172.21.15.60:0/2096869518 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac030730 con 0x7f30b015ad70 2023-10-16T12:47:04.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 -- 172.21.15.60:0/2096869518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 msgr2=0x7f30b0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 --2- 172.21.15.60:0/2096869518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b0069da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0097d0 tx=0x7f30ac02b800 comp rx=0 tx=0).stop 2023-10-16T12:47:04.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 -- 172.21.15.60:0/2096869518 shutdown_connections 2023-10-16T12:47:04.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 --2- 172.21.15.60:0/2096869518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b0069da0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 -- 172.21.15.60:0/2096869518 >> 172.21.15.60:0/2096869518 conn(0x7f30b00ce9b0 msgr2=0x7f30b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:04.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 -- 172.21.15.60:0/2096869518 shutdown_connections 2023-10-16T12:47:04.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.972+0000 7f30b8000640 1 -- 172.21.15.60:0/2096869518 wait complete. 2023-10-16T12:47:04.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.974+0000 7f30b8000640 1 Processor -- start 2023-10-16T12:47:04.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.975+0000 7f30b8000640 1 -- start start 2023-10-16T12:47:04.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b8000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 0x7f30b0158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b8000640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30b00caac0 0x7f30b00ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b8000640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b00adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b8000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b00c4c40 con 0x7f30b00c9ca0 2023-10-16T12:47:04.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b8000640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b00c3fc0 con 0x7f30b00caac0 2023-10-16T12:47:04.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b8000640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b00c42a0 con 0x7f30b015ad70 2023-10-16T12:47:04.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b5d9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 0x7f30b0158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b5d9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 0x7f30b0158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47270/0 (socket says 172.21.15.60:47270) 2023-10-16T12:47:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b5d9c640 1 -- 172.21.15.60:0/939900944 learned_addr learned my addr 172.21.15.60:0/939900944 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.976+0000 7f30b659d640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b00adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b559b640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30b00caac0 0x7f30b00ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b5d9c640 1 -- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 msgr2=0x7f30b00adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b5d9c640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b00adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b5d9c640 1 -- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30b00caac0 msgr2=0x7f30b00ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b5d9c640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30b00caac0 0x7f30b00ad5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b5d9c640 1 -- 172.21.15.60:0/939900944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac009400 con 0x7f30b00c9ca0 2023-10-16T12:47:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30b5d9c640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 0x7f30b0158d30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0097d0 tx=0x7f30ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:04.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac004220 con 0x7f30b00c9ca0 2023-10-16T12:47:04.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.977+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac004380 con 0x7f30b00c9ca0 2023-10-16T12:47:04.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.978+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac03c860 con 0x7f30b00c9ca0 2023-10-16T12:47:04.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.978+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b00c4580 con 0x7f30b00c9ca0 2023-10-16T12:47:04.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.978+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b0159270 con 0x7f30b00c9ca0 2023-10-16T12:47:04.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.979+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b00c4d80 con 0x7f30b00c9ca0 2023-10-16T12:47:04.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.980+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30ac039050 con 0x7f30b00c9ca0 2023-10-16T12:47:04.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.981+0000 7f30a67fc640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f309406af90 0x7f309406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:04.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.981+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f30ac0ab2c0 con 0x7f30b00c9ca0 2023-10-16T12:47:04.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.982+0000 7f30b559b640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f309406af90 0x7f309406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:04.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.983+0000 7f30b559b640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f309406af90 0x7f309406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30b00b4800 tx=0x7f30a0009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:04.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:04.986+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30b00c4d80 con 0x7f30b00c9ca0 2023-10-16T12:47:05.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.191+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f30b00c4d80 con 0x7f30b00c9ca0 2023-10-16T12:47:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.197+0000 7f30a67fc640 1 -- 172.21.15.60:0/939900944 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f30b00c4d80 con 0x7f30b00c9ca0 2023-10-16T12:47:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f309406af90 msgr2=0x7f309406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f309406af90 0x7f309406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30b00b4800 tx=0x7f30a0009040 comp rx=0 tx=0).stop 2023-10-16T12:47:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 msgr2=0x7f30b0158d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 0x7f30b0158d30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0097d0 tx=0x7f30ac0046c0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 shutdown_connections 2023-10-16T12:47:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f309406af90 0x7f309406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30b015ad70 0x7f30b00adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30b00caac0 0x7f30b00ad5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 --2- 172.21.15.60:0/939900944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b00c9ca0 0x7f30b0158d30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 >> 172.21.15.60:0/939900944 conn(0x7f30b00ce9b0 msgr2=0x7f30b00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.200+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 shutdown_connections 2023-10-16T12:47:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.201+0000 7f30b8000640 1 -- 172.21.15.60:0/939900944 wait complete. 2023-10-16T12:47:05.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:47:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.447+0000 7f59cd0bb640 1 Processor -- start 2023-10-16T12:47:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.447+0000 7f59cd0bb640 1 -- start start 2023-10-16T12:47:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59cd0bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f59c806c040 0x7f59c806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59cd0bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 0x7f59c815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59cd0bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c806c960 con 0x7f59c80ad970 2023-10-16T12:47:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59cd0bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c815b9d0 con 0x7f59c806c040 2023-10-16T12:47:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59cd0bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c815bcb0 con 0x7f59c80cda80 2023-10-16T12:47:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59bffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f59c80ad970 0x7f59c815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52220/0 (socket says 172.21.15.60:52220) 2023-10-16T12:47:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.449+0000 7f59bffff640 1 -- 172.21.15.60:0/330090502 learned_addr learned my addr 172.21.15.60:0/330090502 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c7fff640 1 --2- 172.21.15.60:0/330090502 >> v2:172.21.15.144:3300/0 conn(0x7f59c806c040 0x7f59c806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59cc8ba640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 0x7f59c815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171251715 0 0) 0x7f59c806c960 con 0x7f59c80ad970 2023-10-16T12:47:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b4003520 con 0x7f59c80ad970 2023-10-16T12:47:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751863111 0 0) 0x7f59b4003520 con 0x7f59c80ad970 2023-10-16T12:47:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 msgr2=0x7f59c815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 0x7f59c815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 >> v2:172.21.15.144:3300/0 conn(0x7f59c806c040 msgr2=0x7f59c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 --2- 172.21.15.60:0/330090502 >> v2:172.21.15.144:3300/0 conn(0x7f59c806c040 0x7f59c806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c815bf90 con 0x7f59c80ad970 2023-10-16T12:47:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f59b0001670 con 0x7f59c80ad970 2023-10-16T12:47:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 >> v1:172.21.15.60:6789/0 conn(0x7f59c80ad970 legacy=0x7f59c815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59b4003970 0x7f59b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b40047c0 0x7f59b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 0x7f59b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c815b9d0 con 0x7f59b4003970 2023-10-16T12:47:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c815bcb0 con 0x7f59b40047c0 2023-10-16T12:47:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.450+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b400b980 con 0x7f59c80cda80 2023-10-16T12:47:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.451+0000 7f59c7fff640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59b4003970 0x7f59b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.451+0000 7f59c7fff640 1 -- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 msgr2=0x7f59b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.451+0000 7f59c7fff640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80cda80 0x7f59b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.451+0000 7f59c7fff640 1 -- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b40047c0 msgr2=0x7f59b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.451+0000 7f59c7fff640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b40047c0 0x7f59b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.451+0000 7f59c7fff640 1 -- 172.21.15.60:0/330090502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4003520 con 0x7f59b4003970 2023-10-16T12:47:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.452+0000 7f59c7fff640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59b4003970 0x7f59b4003d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009af0 tx=0x7f59b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4704b37d6f366a39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b802c710 con 0x7f59b4003970 2023-10-16T12:47:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b802c870 con 0x7f59b4003970 2023-10-16T12:47:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59c77fe640 1 -- 172.21.15.60:0/330090502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b802cc00 con 0x7f59b4003970 2023-10-16T12:47:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cd0bb640 1 -- 172.21.15.60:0/330090502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59b4003970 msgr2=0x7f59b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/330090502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59b4003970 0x7f59b4003d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009af0 tx=0x7f59b40057f0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cc8ba640 1 -- 172.21.15.60:0/330090502 reap_dead start 2023-10-16T12:47:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cd0bb640 1 -- 172.21.15.60:0/330090502 shutdown_connections 2023-10-16T12:47:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cd0bb640 1 -- 172.21.15.60:0/330090502 >> 172.21.15.60:0/330090502 conn(0x7f59c80ce6a0 msgr2=0x7f59c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cd0bb640 1 -- 172.21.15.60:0/330090502 shutdown_connections 2023-10-16T12:47:05.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.453+0000 7f59cd0bb640 1 -- 172.21.15.60:0/330090502 wait complete. 2023-10-16T12:47:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.455+0000 7f59cd0bb640 1 Processor -- start 2023-10-16T12:47:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.455+0000 7f59cd0bb640 1 -- start start 2023-10-16T12:47:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.456+0000 7f59cd0bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b4003970 0x7f59c80b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.456+0000 7f59cd0bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 0x7f59c80c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.456+0000 7f59cd0bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80ae3a0 0x7f59c814d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.456+0000 7f59cd0bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80c5910 con 0x7f59c80b1980 2023-10-16T12:47:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.456+0000 7f59cd0bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80b1e00 con 0x7f59b4003970 2023-10-16T12:47:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.456+0000 7f59cd0bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80b2080 con 0x7f59c80ae3a0 2023-10-16T12:47:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 0x7f59c80c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 0x7f59c80c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47294/0 (socket says 172.21.15.60:47294) 2023-10-16T12:47:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 -- 172.21.15.60:0/2924067293 learned_addr learned my addr 172.21.15.60:0/2924067293 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59cc8ba640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80ae3a0 0x7f59c814d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 -- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80ae3a0 msgr2=0x7f59c814d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80ae3a0 0x7f59c814d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 -- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b4003970 msgr2=0x7f59c80b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b4003970 0x7f59c80b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 -- 172.21.15.60:0/2924067293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c80b2360 con 0x7f59c80b1980 2023-10-16T12:47:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59cc8ba640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80ae3a0 0x7f59c814d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:05.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.457+0000 7f59bffff640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 0x7f59c80c6ec0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c700 tx=0x7f59b000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.458+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b00072d0 con 0x7f59c80b1980 2023-10-16T12:47:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.458+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b0007450 con 0x7f59c80b1980 2023-10-16T12:47:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.458+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b00047e0 con 0x7f59c80b1980 2023-10-16T12:47:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.458+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c80b2640 con 0x7f59c80b1980 2023-10-16T12:47:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.458+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c80c7400 con 0x7f59c80b1980 2023-10-16T12:47:05.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.459+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c806b330 con 0x7f59c80b1980 2023-10-16T12:47:05.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.460+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f59b0004960 con 0x7f59c80b1980 2023-10-16T12:47:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.461+0000 7f59c57fa640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f59a806af90 0x7f59a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.461+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f59b0088a80 con 0x7f59c80b1980 2023-10-16T12:47:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.462+0000 7f59c7fff640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f59a806af90 0x7f59a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.463+0000 7f59c7fff640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f59a806af90 0x7f59a806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59b8036000 tx=0x7f59b802e000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:05.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.466+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59b0053a80 con 0x7f59c80b1980 2023-10-16T12:47:05.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.664+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f59c80c5a50 con 0x7f59c80b1980 2023-10-16T12:47:05.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.672+0000 7f59c57fa640 1 -- 172.21.15.60:0/2924067293 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f59c80c5a50 con 0x7f59c80b1980 2023-10-16T12:47:05.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f59a806af90 msgr2=0x7f59a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f59a806af90 0x7f59a806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59b8036000 tx=0x7f59b802e000 comp rx=0 tx=0).stop 2023-10-16T12:47:05.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 msgr2=0x7f59c80c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 0x7f59c80c6ec0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c700 tx=0x7f59b000cbc0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 shutdown_connections 2023-10-16T12:47:05.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f59a806af90 0x7f59a806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59c80ae3a0 0x7f59c814d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59c80b1980 0x7f59c80c6ec0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 --2- 172.21.15.60:0/2924067293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59b4003970 0x7f59c80b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 >> 172.21.15.60:0/2924067293 conn(0x7f59c80ce6a0 msgr2=0x7f59c815e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:05.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 shutdown_connections 2023-10-16T12:47:05.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.675+0000 7f59cd0bb640 1 -- 172.21.15.60:0/2924067293 wait complete. 2023-10-16T12:47:05.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T12:47:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.926+0000 7fae09c40640 1 Processor -- start 2023-10-16T12:47:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.926+0000 7fae09c40640 1 -- start start 2023-10-16T12:47:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.927+0000 7fae09c40640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fae0406c4d0 0x7fae040cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.927+0000 7fae09c40640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae040cc980 0x7fae0406f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.927+0000 7fae09c40640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae040ce460 con 0x7fae0406f480 2023-10-16T12:47:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.927+0000 7fae09c40640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae040adf50 con 0x7fae0406c4d0 2023-10-16T12:47:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.928+0000 7fae09c40640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0406fe50 con 0x7fae040cc980 2023-10-16T12:47:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.928+0000 7fae0943f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae040cc980 0x7fae0406f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.928+0000 7fae0943f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae040cc980 0x7fae0406f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:56628/0 (socket says 172.21.15.60:56628) 2023-10-16T12:47:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.928+0000 7fae03fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fae0406f480 0x7fae040cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:50674/0 (socket says 172.21.15.60:50674) 2023-10-16T12:47:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.928+0000 7fae0943f640 1 -- 172.21.15.60:0/2054477468 learned_addr learned my addr 172.21.15.60:0/2054477468 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.928+0000 7fae08c3e640 1 --2- 172.21.15.60:0/2054477468 >> v2:172.21.15.144:3300/0 conn(0x7fae0406c4d0 0x7fae040cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.929+0000 7fae08c3e640 1 -- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae040cc980 msgr2=0x7fae0406f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.929+0000 7fae08c3e640 1 --2- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae040cc980 0x7fae0406f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.929+0000 7fae08c3e640 1 -- 172.21.15.60:0/2054477468 >> v1:172.21.15.144:6789/0 conn(0x7fae0406f480 legacy=0x7fae040cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.929+0000 7fae08c3e640 1 -- 172.21.15.60:0/2054477468 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae040cce80 con 0x7fae0406c4d0 2023-10-16T12:47:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.930+0000 7fae08c3e640 1 --2- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406c4d0 0x7fae040cdf20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009d90 tx=0x7fadf402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26a568f45085dd78 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.931+0000 7fae037fe640 1 -- 172.21.15.60:0/2054477468 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf402cc60 con 0x7fae0406c4d0 2023-10-16T12:47:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.931+0000 7fae037fe640 1 -- 172.21.15.60:0/2054477468 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf402cdc0 con 0x7fae0406c4d0 2023-10-16T12:47:05.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.931+0000 7fae037fe640 1 -- 172.21.15.60:0/2054477468 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4034660 con 0x7fae0406c4d0 2023-10-16T12:47:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.931+0000 7fae09c40640 1 -- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406c4d0 msgr2=0x7fae040cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.931+0000 7fae09c40640 1 --2- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406c4d0 0x7fae040cdf20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009d90 tx=0x7fadf402beb0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.932+0000 7fae09c40640 1 -- 172.21.15.60:0/2054477468 shutdown_connections 2023-10-16T12:47:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.932+0000 7fae09c40640 1 --2- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae040cc980 0x7fae0406f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.932+0000 7fae09c40640 1 --2- 172.21.15.60:0/2054477468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406c4d0 0x7fae040cdf20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.932+0000 7fae09c40640 1 -- 172.21.15.60:0/2054477468 >> 172.21.15.60:0/2054477468 conn(0x7fae040b0a90 msgr2=0x7fae040ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.932+0000 7fae09c40640 1 -- 172.21.15.60:0/2054477468 shutdown_connections 2023-10-16T12:47:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.932+0000 7fae09c40640 1 -- 172.21.15.60:0/2054477468 wait complete. 2023-10-16T12:47:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.933+0000 7fae09c40640 1 Processor -- start 2023-10-16T12:47:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.934+0000 7fae09c40640 1 -- start start 2023-10-16T12:47:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.934+0000 7fae09c40640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae0406c4d0 0x7fae040bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae09c40640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406f480 0x7fae0415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae09c40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 0x7fae04162e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae09c40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0415efa0 con 0x7fae040cc980 2023-10-16T12:47:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae09c40640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae040c1480 con 0x7fae0406f480 2023-10-16T12:47:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae09c40640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040c0900 con 0x7fae0406c4d0 2023-10-16T12:47:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 0x7fae04162e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 0x7fae04162e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47310/0 (socket says 172.21.15.60:47310) 2023-10-16T12:47:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 -- 172.21.15.60:0/1228578742 learned_addr learned my addr 172.21.15.60:0/1228578742 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae03fff640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406f480 0x7fae0415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae08c3e640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae0406c4d0 0x7fae040bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 -- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae0406c4d0 msgr2=0x7fae040bdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae0406c4d0 0x7fae040bdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 -- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406f480 msgr2=0x7fae0415ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406f480 0x7fae0415ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae0943f640 1 -- 172.21.15.60:0/1228578742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4009a70 con 0x7fae040cc980 2023-10-16T12:47:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.935+0000 7fae08c3e640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae0406c4d0 0x7fae040bdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.936+0000 7fae0943f640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 0x7fae04162e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00e870 tx=0x7fadfc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.936+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc009660 con 0x7fae040cc980 2023-10-16T12:47:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.936+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc00c940 con 0x7fae040cc980 2023-10-16T12:47:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.936+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc00ccf0 con 0x7fae040cc980 2023-10-16T12:47:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.936+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae040c0be0 con 0x7fae040cc980 2023-10-16T12:47:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.937+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae040c11a0 con 0x7fae040cc980 2023-10-16T12:47:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.938+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040b1300 con 0x7fae040cc980 2023-10-16T12:47:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.939+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fadfc013470 con 0x7fae040cc980 2023-10-16T12:47:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.940+0000 7fae017fa640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fadf006af90 0x7fadf006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.940+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fadfc089ae0 con 0x7fae040cc980 2023-10-16T12:47:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.941+0000 7fae08c3e640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fadf006af90 0x7fadf006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.942+0000 7fae08c3e640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fadf006af90 0x7fadf006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fadf4036000 tx=0x7fadf4030000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:05.945+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae040b1300 con 0x7fae040cc980 2023-10-16T12:47:06.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.216+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fae040b1300 con 0x7fae040cc980 2023-10-16T12:47:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.275+0000 7fae017fa640 1 -- 172.21.15.60:0/1228578742 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v486) v1 ==== 158+0+0 (secure 0 0 0) 0x7fae040b1300 con 0x7fae040cc980 2023-10-16T12:47:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fadf006af90 msgr2=0x7fadf006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fadf006af90 0x7fadf006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fadf4036000 tx=0x7fadf4030000 comp rx=0 tx=0).stop 2023-10-16T12:47:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 msgr2=0x7fae04162e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 0x7fae04162e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00e870 tx=0x7fadfc00ed30 comp rx=0 tx=0).stop 2023-10-16T12:47:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 shutdown_connections 2023-10-16T12:47:06.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fadf006af90 0x7fadf006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae040cc980 0x7fae04162e40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae0406f480 0x7fae0415ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 --2- 172.21.15.60:0/1228578742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae0406c4d0 0x7fae040bdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 >> 172.21.15.60:0/1228578742 conn(0x7fae040b0a90 msgr2=0x7fae040c1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:06.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 shutdown_connections 2023-10-16T12:47:06.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.278+0000 7fae09c40640 1 -- 172.21.15.60:0/1228578742 wait complete. 2023-10-16T12:47:06.301 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-16T12:47:06.302 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-16T12:47:06.302 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:47:06.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:47:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.521+0000 7f27d359e640 1 Processor -- start 2023-10-16T12:47:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.522+0000 7f27d359e640 1 -- start start 2023-10-16T12:47:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27d40b8480 0x7f27d40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40d0be0 0x7f27d40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d415d150 con 0x7f27d415cd70 2023-10-16T12:47:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d40d1180 con 0x7f27d40b8480 2023-10-16T12:47:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40d1460 con 0x7f27d40d0be0 2023-10-16T12:47:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f27d415cd70 0x7f27d40d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52236/0 (socket says 172.21.15.60:52236) 2023-10-16T12:47:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27d259c640 1 -- 172.21.15.60:0/3638891362 learned_addr learned my addr 172.21.15.60:0/3638891362 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.523+0000 7f27cbfff640 1 --2- 172.21.15.60:0/3638891362 >> v2:172.21.15.60:3300/0 conn(0x7f27d40b8480 0x7f27d40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27d2d9d640 1 --2- 172.21.15.60:0/3638891362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40d0be0 0x7f27d40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27d1d9b640 1 -- 172.21.15.60:0/3638891362 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630882603 0 0) 0x7f27d415d150 con 0x7f27d415cd70 2023-10-16T12:47:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27d1d9b640 1 -- 172.21.15.60:0/3638891362 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c0003520 con 0x7f27d415cd70 2023-10-16T12:47:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27cbfff640 1 -- 172.21.15.60:0/3638891362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40d0be0 msgr2=0x7f27d40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27cbfff640 1 --2- 172.21.15.60:0/3638891362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40d0be0 0x7f27d40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27cbfff640 1 -- 172.21.15.60:0/3638891362 >> v1:172.21.15.60:6789/0 conn(0x7f27d415cd70 legacy=0x7f27d40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.524+0000 7f27cbfff640 1 -- 172.21.15.60:0/3638891362 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d40d1740 con 0x7f27d40b8480 2023-10-16T12:47:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.525+0000 7f27cbfff640 1 --2- 172.21.15.60:0/3638891362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d40b8480 0x7f27d40cd7c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097a0 tx=0x7f27bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc8991c3658fbb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d1d9b640 1 -- 172.21.15.60:0/3638891362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc031830 con 0x7f27d40b8480 2023-10-16T12:47:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d1d9b640 1 -- 172.21.15.60:0/3638891362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc031990 con 0x7f27d40b8480 2023-10-16T12:47:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d1d9b640 1 -- 172.21.15.60:0/3638891362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc031d20 con 0x7f27d40b8480 2023-10-16T12:47:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 -- 172.21.15.60:0/3638891362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d40b8480 msgr2=0x7f27d40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 --2- 172.21.15.60:0/3638891362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d40b8480 0x7f27d40cd7c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097a0 tx=0x7f27bc02b8d0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 -- 172.21.15.60:0/3638891362 shutdown_connections 2023-10-16T12:47:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 --2- 172.21.15.60:0/3638891362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40d0be0 0x7f27d40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 --2- 172.21.15.60:0/3638891362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d40b8480 0x7f27d40cd7c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 -- 172.21.15.60:0/3638891362 >> 172.21.15.60:0/3638891362 conn(0x7f27d40cf570 msgr2=0x7f27d40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 -- 172.21.15.60:0/3638891362 shutdown_connections 2023-10-16T12:47:06.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.526+0000 7f27d359e640 1 -- 172.21.15.60:0/3638891362 wait complete. 2023-10-16T12:47:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.528+0000 7f27d359e640 1 Processor -- start 2023-10-16T12:47:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.528+0000 7f27d359e640 1 -- start start 2023-10-16T12:47:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.529+0000 7f27d359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40b8480 0x7f27d4153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.529+0000 7f27d359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27d40d0be0 0x7f27d41540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.529+0000 7f27d359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 0x7f27d415d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.529+0000 7f27d359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4148690 con 0x7f27d415cd70 2023-10-16T12:47:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.529+0000 7f27d359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41547f0 con 0x7f27d40d0be0 2023-10-16T12:47:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.529+0000 7f27d359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4154ad0 con 0x7f27d40b8480 2023-10-16T12:47:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 0x7f27d415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 0x7f27d415d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47330/0 (socket says 172.21.15.60:47330) 2023-10-16T12:47:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 -- 172.21.15.60:0/1821223847 learned_addr learned my addr 172.21.15.60:0/1821223847 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 -- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40b8480 msgr2=0x7f27d4153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40b8480 0x7f27d4153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 -- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27d40d0be0 msgr2=0x7f27d41540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27cbfff640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27d40d0be0 0x7f27d41540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27d40d0be0 0x7f27d41540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27d2d9d640 1 -- 172.21.15.60:0/1821223847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009400 con 0x7f27d415cd70 2023-10-16T12:47:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.530+0000 7f27cbfff640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27d40d0be0 0x7f27d41540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.531+0000 7f27d2d9d640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 0x7f27d415d340 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00efc0 tx=0x7f27cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.531+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc009b10 con 0x7f27d415cd70 2023-10-16T12:47:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.531+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc00f040 con 0x7f27d415cd70 2023-10-16T12:47:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.531+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d4154db0 con 0x7f27d415cd70 2023-10-16T12:47:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.531+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0146c0 con 0x7f27d415cd70 2023-10-16T12:47:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.531+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d4155370 con 0x7f27d415cd70 2023-10-16T12:47:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.532+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d4155560 con 0x7f27d415cd70 2023-10-16T12:47:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.533+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f27cc014820 con 0x7f27d415cd70 2023-10-16T12:47:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.534+0000 7f27caffd640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f27b406af90 0x7f27b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.534+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f27cc08a3b0 con 0x7f27d415cd70 2023-10-16T12:47:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.535+0000 7f27d259c640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f27b406af90 0x7f27b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.536+0000 7f27d259c640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f27b406af90 0x7f27b406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f27c4001610 tx=0x7f27c4007930 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.539+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27cc055210 con 0x7f27d415cd70 2023-10-16T12:47:06.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.737+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27d404e9b0 con 0x7f27d415cd70 2023-10-16T12:47:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.739+0000 7f27caffd640 1 -- 172.21.15.60:0/1821223847 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13925 (secure 0 0 0) 0x7f27cc058830 con 0x7f27d415cd70 2023-10-16T12:47:06.741 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:06.742 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":59,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:47:04.252787+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/674545145":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2596584944":"2023-10-17T12:46:57.079584+0000","172.21.15.144:6832/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/1973903962":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6833/696341254":"2023-10-17T12:46:57.079584+0000","172.21.15.60:6835/3714850681":"2023-10-17T12:47:01.236398+0000","172.21.15.144:0/1714573694":"2023-10-17T12:34:57.701960+0000","172.21.15.144:6832/696341254":"2023-10-17T12:46:57.079584+0000","172.21.15.60:6833/3714850681":"2023-10-17T12:47:01.236398+0000","172.21.15.144:0/3946760462":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/680953068":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2301614647":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/2412206403":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/25541997":"2023-10-17T12:46:57.079584+0000","172.21.15.144:0/1199236332":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/2619332853":"2023-10-17T12:34:57.701960+0000","172.21.15.144:0/2315449004":"2023-10-17T12:46:57.079584+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:47:06.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.741+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f27b406af90 msgr2=0x7f27b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.741+0000 7f27d359e640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f27b406af90 0x7f27b406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f27c4001610 tx=0x7f27c4007930 comp rx=0 tx=0).stop 2023-10-16T12:47:06.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.741+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 msgr2=0x7f27d415d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:06.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.741+0000 7f27d359e640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 0x7f27d415d340 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00efc0 tx=0x7f27cc00c490 comp rx=0 tx=0).stop 2023-10-16T12:47:06.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 shutdown_connections 2023-10-16T12:47:06.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f27b406af90 0x7f27b406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27d415cd70 0x7f27d415d340 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27d40d0be0 0x7f27d41540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 --2- 172.21.15.60:0/1821223847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27d40b8480 0x7f27d4153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 >> 172.21.15.60:0/1821223847 conn(0x7f27d40cf570 msgr2=0x7f27d40b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 shutdown_connections 2023-10-16T12:47:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.742+0000 7f27d359e640 1 -- 172.21.15.60:0/1821223847 wait complete. 2023-10-16T12:47:06.764 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:47:06.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:47:07.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.998+0000 7fda5a3e9640 1 Processor -- start 2023-10-16T12:47:07.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:06.998+0000 7fda5a3e9640 1 -- start start 2023-10-16T12:47:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda5a3e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cbf0 0x7fda541613b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda5a3e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda5406c550 0x7fda5406c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda5a3e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda54160f20 con 0x7fda5415cfd0 2023-10-16T12:47:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda5a3e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5406cf00 con 0x7fda5406c550 2023-10-16T12:47:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda5a3e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda54160910 con 0x7fda5415cbf0 2023-10-16T12:47:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda59be8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fda5406c550 0x7fda5406c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda58be6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fda5415cfd0 0x7fda540cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52242/0 (socket says 172.21.15.60:52242) 2023-10-16T12:47:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.000+0000 7fda58be6640 1 -- 172.21.15.60:0/1647211991 learned_addr learned my addr 172.21.15.60:0/1647211991 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.001+0000 7fda593e7640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cbf0 0x7fda541613b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.001+0000 7fda59be8640 1 -- 172.21.15.60:0/1647211991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cbf0 msgr2=0x7fda541613b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.001+0000 7fda59be8640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cbf0 0x7fda541613b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.001+0000 7fda59be8640 1 -- 172.21.15.60:0/1647211991 >> v1:172.21.15.60:6789/0 conn(0x7fda5415cfd0 legacy=0x7fda540cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.001+0000 7fda59be8640 1 -- 172.21.15.60:0/1647211991 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda540cd5c0 con 0x7fda5406c550 2023-10-16T12:47:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.001+0000 7fda593e7640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cbf0 0x7fda541613b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.002+0000 7fda59be8640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda5406c9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009d90 tx=0x7fda4c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0b5cf197dc8693 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:07.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.003+0000 7fda4bfff640 1 -- 172.21.15.60:0/1647211991 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c02cc60 con 0x7fda5406c550 2023-10-16T12:47:07.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.003+0000 7fda4bfff640 1 -- 172.21.15.60:0/1647211991 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c02cdc0 con 0x7fda5406c550 2023-10-16T12:47:07.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.003+0000 7fda4bfff640 1 -- 172.21.15.60:0/1647211991 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c034660 con 0x7fda5406c550 2023-10-16T12:47:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.003+0000 7fda5a3e9640 1 -- 172.21.15.60:0/1647211991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 msgr2=0x7fda5406c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.003+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda5406c9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009d90 tx=0x7fda4c02beb0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.004+0000 7fda5a3e9640 1 -- 172.21.15.60:0/1647211991 shutdown_connections 2023-10-16T12:47:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.004+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda5406c9c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.004+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/1647211991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cbf0 0x7fda541613b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.004+0000 7fda5a3e9640 1 -- 172.21.15.60:0/1647211991 >> 172.21.15.60:0/1647211991 conn(0x7fda540b23b0 msgr2=0x7fda540b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.004+0000 7fda5a3e9640 1 -- 172.21.15.60:0/1647211991 shutdown_connections 2023-10-16T12:47:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.004+0000 7fda5a3e9640 1 -- 172.21.15.60:0/1647211991 wait complete. 2023-10-16T12:47:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.005+0000 7fda5a3e9640 1 Processor -- start 2023-10-16T12:47:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.006+0000 7fda5a3e9640 1 -- start start 2023-10-16T12:47:07.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda5a3e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda54158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda5a3e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda5415cbf0 0x7fda540c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda5a3e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cfd0 0x7fda540c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda5a3e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda540af740 con 0x7fda5406c550 2023-10-16T12:47:07.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda5a3e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda540ade80 con 0x7fda5415cbf0 2023-10-16T12:47:07.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda5a3e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda540ae160 con 0x7fda5415cfd0 2023-10-16T12:47:07.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda54158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda54158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47348/0 (socket says 172.21.15.60:47348) 2023-10-16T12:47:07.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 -- 172.21.15.60:0/3590306033 learned_addr learned my addr 172.21.15.60:0/3590306033 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:07.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 -- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cfd0 msgr2=0x7fda540c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cfd0 0x7fda540c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 -- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda5415cbf0 msgr2=0x7fda540c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda5415cbf0 0x7fda540c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.007+0000 7fda593e7640 1 -- 172.21.15.60:0/3590306033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c009a70 con 0x7fda5406c550 2023-10-16T12:47:07.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.008+0000 7fda593e7640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda54158180 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c7f0 tx=0x7fda5000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:07.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.008+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50007a30 con 0x7fda5406c550 2023-10-16T12:47:07.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.008+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda50007b90 con 0x7fda5406c550 2023-10-16T12:47:07.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.008+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5000ce40 con 0x7fda5406c550 2023-10-16T12:47:07.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.008+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda540c4540 con 0x7fda5406c550 2023-10-16T12:47:07.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.008+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda540c4b00 con 0x7fda5406c550 2023-10-16T12:47:07.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.009+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda540c4d20 con 0x7fda5406c550 2023-10-16T12:47:07.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.011+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fda500113f0 con 0x7fda5406c550 2023-10-16T12:47:07.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.012+0000 7fda49ffb640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fda3806afe0 0x7fda3806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.012+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fda500899a0 con 0x7fda5406c550 2023-10-16T12:47:07.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.012+0000 7fda58be6640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fda3806afe0 0x7fda3806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.013+0000 7fda58be6640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fda3806afe0 0x7fda3806d4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda541e60e0 tx=0x7fda44007a40 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:07.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.017+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda50054910 con 0x7fda5406c550 2023-10-16T12:47:07.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.217+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda5404e9b0 con 0x7fda5406c550 2023-10-16T12:47:07.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.218+0000 7fda49ffb640 1 -- 172.21.15.60:0/3590306033 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda50057f30 con 0x7fda5406c550 2023-10-16T12:47:07.221 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:47:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fda3806afe0 msgr2=0x7fda3806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fda3806afe0 0x7fda3806d4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda541e60e0 tx=0x7fda44007a40 comp rx=0 tx=0).stop 2023-10-16T12:47:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 msgr2=0x7fda54158180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda54158180 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c7f0 tx=0x7fda5000cbf0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 shutdown_connections 2023-10-16T12:47:07.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fda3806afe0 0x7fda3806d4a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda5415cfd0 0x7fda540c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda5415cbf0 0x7fda540c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 --2- 172.21.15.60:0/3590306033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda5406c550 0x7fda54158180 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 >> 172.21.15.60:0/3590306033 conn(0x7fda540b23b0 msgr2=0x7fda540b18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:07.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 shutdown_connections 2023-10-16T12:47:07.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.221+0000 7fda5a3e9640 1 -- 172.21.15.60:0/3590306033 wait complete. 2023-10-16T12:47:07.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T12:47:07.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.475+0000 7f9ce6df5640 1 Processor -- start 2023-10-16T12:47:07.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.475+0000 7f9ce6df5640 1 -- start start 2023-10-16T12:47:07.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.476+0000 7f9ce6df5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9ce006c3f0 0x7f9ce006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.476+0000 7f9ce6df5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9ce015e540 0x7f9ce00b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.476+0000 7f9ce6df5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce00cd630 con 0x7f9ce015e0c0 2023-10-16T12:47:07.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.476+0000 7f9ce6df5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce00ce1b0 con 0x7f9ce015e540 2023-10-16T12:47:07.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.476+0000 7f9ce6df5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce006cd10 con 0x7f9ce006c3f0 2023-10-16T12:47:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.477+0000 7f9ce5392640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9ce015e540 0x7f9ce00b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.477+0000 7f9ce5392640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9ce015e540 0x7f9ce00b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47364/0 (socket says 172.21.15.60:47364) 2023-10-16T12:47:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.477+0000 7f9ce5392640 1 -- 172.21.15.60:0/441343186 learned_addr learned my addr 172.21.15.60:0/441343186 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.477+0000 7f9ce4b91640 1 --2- 172.21.15.60:0/441343186 >> v2:172.21.15.144:3300/0 conn(0x7f9ce006c3f0 0x7f9ce006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.478+0000 7f9ce5392640 1 -- 172.21.15.60:0/441343186 >> v2:172.21.15.144:3300/0 conn(0x7f9ce006c3f0 msgr2=0x7f9ce006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.478+0000 7f9ce5392640 1 --2- 172.21.15.60:0/441343186 >> v2:172.21.15.144:3300/0 conn(0x7f9ce006c3f0 0x7f9ce006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.478+0000 7f9ce5392640 1 -- 172.21.15.60:0/441343186 >> v1:172.21.15.144:6789/0 conn(0x7f9ce015e0c0 legacy=0x7f9ce00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.478+0000 7f9ce5392640 1 -- 172.21.15.60:0/441343186 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce00d2170 con 0x7f9ce015e540 2023-10-16T12:47:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.479+0000 7f9ce5392640 1 --2- 172.21.15.60:0/441343186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e540 0x7f9ce00b1710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00d3d00 tx=0x7f9cd802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95ec62994cf4fd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.479+0000 7f9cdf7fe640 1 -- 172.21.15.60:0/441343186 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8030860 con 0x7f9ce015e540 2023-10-16T12:47:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.479+0000 7f9cdf7fe640 1 -- 172.21.15.60:0/441343186 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd80309c0 con 0x7f9ce015e540 2023-10-16T12:47:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.479+0000 7f9cdf7fe640 1 -- 172.21.15.60:0/441343186 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8030d50 con 0x7f9ce015e540 2023-10-16T12:47:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 -- 172.21.15.60:0/441343186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e540 msgr2=0x7f9ce00b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/441343186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e540 0x7f9ce00b1710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00d3d00 tx=0x7f9cd802b9a0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 -- 172.21.15.60:0/441343186 shutdown_connections 2023-10-16T12:47:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/441343186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e540 0x7f9ce00b1710 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/441343186 >> v2:172.21.15.144:3300/0 conn(0x7f9ce006c3f0 0x7f9ce006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 -- 172.21.15.60:0/441343186 >> 172.21.15.60:0/441343186 conn(0x7f9ce00d2cb0 msgr2=0x7f9ce00d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 -- 172.21.15.60:0/441343186 shutdown_connections 2023-10-16T12:47:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.480+0000 7f9ce6df5640 1 -- 172.21.15.60:0/441343186 wait complete. 2023-10-16T12:47:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.482+0000 7f9ce6df5640 1 Processor -- start 2023-10-16T12:47:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.482+0000 7f9ce6df5640 1 -- start start 2023-10-16T12:47:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9ce6df5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce006c3f0 0x7f9ce0155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9ce6df5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 0x7f9ce00cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9ce6df5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce015e540 0x7f9ce01f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9ce6df5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce01580f0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9ce6df5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce01583d0 con 0x7f9ce015e540 2023-10-16T12:47:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9ce6df5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce01586b0 con 0x7f9ce006c3f0 2023-10-16T12:47:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9cdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 0x7f9ce00cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9cdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 0x7f9ce00cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47372/0 (socket says 172.21.15.60:47372) 2023-10-16T12:47:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.483+0000 7f9cdffff640 1 -- 172.21.15.60:0/3077866532 learned_addr learned my addr 172.21.15.60:0/3077866532 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9ce4b91640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce006c3f0 0x7f9ce0155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdffff640 1 -- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce006c3f0 msgr2=0x7f9ce0155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdffff640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce006c3f0 0x7f9ce0155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdffff640 1 -- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce015e540 msgr2=0x7f9ce01f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdffff640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce015e540 0x7f9ce01f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdffff640 1 -- 172.21.15.60:0/3077866532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd80094d0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdffff640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 0x7f9ce00cc020 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400c870 tx=0x7f9cd400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4003b70 con 0x7f9ce015e0c0 2023-10-16T12:47:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd4003cf0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400aa90 con 0x7f9ce015e0c0 2023-10-16T12:47:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.484+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce0158990 con 0x7f9ce015e0c0 2023-10-16T12:47:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.485+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce01556b0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.486+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce0158eb0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.487+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9cd400abf0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.488+0000 7f9cdd7fa640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9ccc06af90 0x7f9ccc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.488+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f9cd40895d0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.488+0000 7f9ce4b91640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9ccc06af90 0x7f9ccc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.490+0000 7f9ce4b91640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9ccc06af90 0x7f9ccc06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0006fd0 tx=0x7f9cd0008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:07.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.492+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ce0158eb0 con 0x7f9ce015e0c0 2023-10-16T12:47:07.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:07.764+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f9ce0158eb0 con 0x7f9ce015e0c0 2023-10-16T12:47:08.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.287+0000 7f9cdd7fa640 1 -- 172.21.15.60:0/3077866532 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v488) v1 ==== 157+0+0 (secure 0 0 0) 0x7f9ce0158eb0 con 0x7f9ce015e0c0 2023-10-16T12:47:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9ccc06af90 msgr2=0x7f9ccc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9ccc06af90 0x7f9ccc06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0006fd0 tx=0x7f9cd0008040 comp rx=0 tx=0).stop 2023-10-16T12:47:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 msgr2=0x7f9ce00cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 0x7f9ce00cc020 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400c870 tx=0x7f9cd400cd30 comp rx=0 tx=0).stop 2023-10-16T12:47:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 shutdown_connections 2023-10-16T12:47:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f9ccc06af90 0x7f9ccc06d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce015e540 0x7f9ce01f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce015e0c0 0x7f9ce00cc020 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 --2- 172.21.15.60:0/3077866532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce006c3f0 0x7f9ce0155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 >> 172.21.15.60:0/3077866532 conn(0x7f9ce00d2cb0 msgr2=0x7f9ce006ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:08.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 shutdown_connections 2023-10-16T12:47:08.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.291+0000 7f9ce6df5640 1 -- 172.21.15.60:0/3077866532 wait complete. 2023-10-16T12:47:08.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:47:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.542+0000 7fe71ffff640 1 Processor -- start 2023-10-16T12:47:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.542+0000 7fe71ffff640 1 -- start start 2023-10-16T12:47:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe72006c040 0x7fe72006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe7200ad970 0x7fe72015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72006c960 con 0x7fe72015ebc0 2023-10-16T12:47:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72015b9d0 con 0x7fe72006c040 2023-10-16T12:47:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71ffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72015bcb0 con 0x7fe7200ad970 2023-10-16T12:47:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe72006c040 0x7fe72006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe72006c040 0x7fe72006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47378/0 (socket says 172.21.15.60:47378) 2023-10-16T12:47:08.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.544+0000 7fe71effd640 1 -- 172.21.15.60:0/3173451246 learned_addr learned my addr 172.21.15.60:0/3173451246 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:08.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.545+0000 7fe71effd640 1 -- 172.21.15.60:0/3173451246 >> v2:172.21.15.144:3300/0 conn(0x7fe7200ad970 msgr2=0x7fe72015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:08.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.545+0000 7fe71effd640 1 --2- 172.21.15.60:0/3173451246 >> v2:172.21.15.144:3300/0 conn(0x7fe7200ad970 0x7fe72015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.545+0000 7fe71effd640 1 -- 172.21.15.60:0/3173451246 >> v1:172.21.15.144:6789/0 conn(0x7fe72015ebc0 legacy=0x7fe7200cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:08.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.545+0000 7fe71effd640 1 -- 172.21.15.60:0/3173451246 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72015bf90 con 0x7fe72006c040 2023-10-16T12:47:08.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.546+0000 7fe71effd640 1 --2- 172.21.15.60:0/3173451246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72006c040 0x7fe72006c420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe7200cb060 tx=0x7fe71002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7440fe705cfedd83 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:08.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.546+0000 7fe71e7fc640 1 -- 172.21.15.60:0/3173451246 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710031870 con 0x7fe72006c040 2023-10-16T12:47:08.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.546+0000 7fe71e7fc640 1 -- 172.21.15.60:0/3173451246 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7100319d0 con 0x7fe72006c040 2023-10-16T12:47:08.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.546+0000 7fe71e7fc640 1 -- 172.21.15.60:0/3173451246 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710031d60 con 0x7fe72006c040 2023-10-16T12:47:08.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 -- 172.21.15.60:0/3173451246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72006c040 msgr2=0x7fe72006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 --2- 172.21.15.60:0/3173451246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72006c040 0x7fe72006c420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe7200cb060 tx=0x7fe71002b910 comp rx=0 tx=0).stop 2023-10-16T12:47:08.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 -- 172.21.15.60:0/3173451246 shutdown_connections 2023-10-16T12:47:08.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 --2- 172.21.15.60:0/3173451246 >> v2:172.21.15.144:3300/0 conn(0x7fe7200ad970 0x7fe72015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 --2- 172.21.15.60:0/3173451246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72006c040 0x7fe72006c420 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 -- 172.21.15.60:0/3173451246 >> 172.21.15.60:0/3173451246 conn(0x7fe7200ce6a0 msgr2=0x7fe7200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 -- 172.21.15.60:0/3173451246 shutdown_connections 2023-10-16T12:47:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.547+0000 7fe71ffff640 1 -- 172.21.15.60:0/3173451246 wait complete. 2023-10-16T12:47:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.549+0000 7fe71ffff640 1 Processor -- start 2023-10-16T12:47:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.549+0000 7fe71ffff640 1 -- start start 2023-10-16T12:47:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe72006c040 0x7fe7200c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7200ad970 0x7fe720151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 0x7fe7200b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200c6dc0 con 0x7fe72015ebc0 2023-10-16T12:47:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200b1d10 con 0x7fe72006c040 2023-10-16T12:47:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200b1ff0 con 0x7fe7200ad970 2023-10-16T12:47:08.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 0x7fe7200b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:08.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 0x7fe7200b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47392/0 (socket says 172.21.15.60:47392) 2023-10-16T12:47:08.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 -- 172.21.15.60:0/2012715498 learned_addr learned my addr 172.21.15.60:0/2012715498 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:08.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 -- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7200ad970 msgr2=0x7fe720151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7200ad970 0x7fe720151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 -- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe72006c040 msgr2=0x7fe7200c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe72006c040 0x7fe7200c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.550+0000 7fe71f7fe640 1 -- 172.21.15.60:0/2012715498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7100094d0 con 0x7fe72015ebc0 2023-10-16T12:47:08.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.551+0000 7fe71f7fe640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 0x7fe7200b12c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c780 tx=0x7fe71800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:08.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.551+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718003d80 con 0x7fe72015ebc0 2023-10-16T12:47:08.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.551+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71800ce90 con 0x7fe72015ebc0 2023-10-16T12:47:08.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.551+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180052b0 con 0x7fe72015ebc0 2023-10-16T12:47:08.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.551+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7200c7080 con 0x7fe72015ebc0 2023-10-16T12:47:08.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.551+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7201522d0 con 0x7fe72015ebc0 2023-10-16T12:47:08.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.552+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72006b330 con 0x7fe72015ebc0 2023-10-16T12:47:08.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.553+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe718005410 con 0x7fe72015ebc0 2023-10-16T12:47:08.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.555+0000 7fe7177fe640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fe70006af90 0x7fe70006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:08.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.555+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fe718088b60 con 0x7fe72015ebc0 2023-10-16T12:47:08.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.556+0000 7fe71effd640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fe70006af90 0x7fe70006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:08.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.557+0000 7fe71effd640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fe70006af90 0x7fe70006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe710036000 tx=0x7fe710030000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:08.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.559+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe718053c00 con 0x7fe72015ebc0 2023-10-16T12:47:08.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.753+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe7200c75a0 con 0x7fe72015ebc0 2023-10-16T12:47:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.760+0000 7fe7177fe640 1 -- 172.21.15.60:0/2012715498 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe7200c75a0 con 0x7fe72015ebc0 2023-10-16T12:47:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fe70006af90 msgr2=0x7fe70006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fe70006af90 0x7fe70006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe710036000 tx=0x7fe710030000 comp rx=0 tx=0).stop 2023-10-16T12:47:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 msgr2=0x7fe7200b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 0x7fe7200b12c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c780 tx=0x7fe71800cc40 comp rx=0 tx=0).stop 2023-10-16T12:47:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 shutdown_connections 2023-10-16T12:47:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fe70006af90 0x7fe70006d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe7200ad970 0x7fe720151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe72015ebc0 0x7fe7200b12c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 --2- 172.21.15.60:0/2012715498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe72006c040 0x7fe7200c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 >> 172.21.15.60:0/2012715498 conn(0x7fe7200ce6a0 msgr2=0x7fe72015a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 shutdown_connections 2023-10-16T12:47:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:08.763+0000 7fe71ffff640 1 -- 172.21.15.60:0/2012715498 wait complete. 2023-10-16T12:47:08.786 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:47:08.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:47:08.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:47:09.026 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:47:09.027 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:47:09.027 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:47:09.090 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:47:09.091 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:47:09.091 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:47:09.154 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:47:09.155 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:47:09.155 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:47:09.219 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:47:09.220 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:47:09.220 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:47:09.280 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:47:09.280 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:47:09.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:47:09.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.518+0000 7f578c9e9640 1 Processor -- start 2023-10-16T12:47:09.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.519+0000 7f578c9e9640 1 -- start start 2023-10-16T12:47:09.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f578c9e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57880c9060 0x7f578806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:09.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f578c9e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57880c9440 0x7f578806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:09.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f578c9e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880ca560 con 0x7f578806c9e0 2023-10-16T12:47:09.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f578c9e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57880c9950 con 0x7f57880c9060 2023-10-16T12:47:09.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f578c9e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57880c9c30 con 0x7f57880c9440 2023-10-16T12:47:09.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f57877fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57880c9060 0x7f578806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:09.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f57877fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57880c9060 0x7f578806f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45218/0 (socket says 172.21.15.60:45218) 2023-10-16T12:47:09.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f57877fe640 1 -- 172.21.15.60:0/783901296 learned_addr learned my addr 172.21.15.60:0/783901296 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:09.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f57877fe640 1 -- 172.21.15.60:0/783901296 >> v2:172.21.15.144:3300/0 conn(0x7f57880c9440 msgr2=0x7f578806f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:09.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f57877fe640 1 --2- 172.21.15.60:0/783901296 >> v2:172.21.15.144:3300/0 conn(0x7f57880c9440 0x7f578806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.521+0000 7f57877fe640 1 -- 172.21.15.60:0/783901296 >> v1:172.21.15.144:6789/0 conn(0x7f578806c9e0 legacy=0x7f5788164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.522+0000 7f57877fe640 1 -- 172.21.15.60:0/783901296 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57880c9f10 con 0x7f57880c9060 2023-10-16T12:47:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.523+0000 7f57877fe640 1 --2- 172.21.15.60:0/783901296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f578806f440 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f57880ce550 tx=0x7f577802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edb598baa3ae7c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.523+0000 7f57867fc640 1 -- 172.21.15.60:0/783901296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778031870 con 0x7f57880c9060 2023-10-16T12:47:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.523+0000 7f57867fc640 1 -- 172.21.15.60:0/783901296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57780319d0 con 0x7f57880c9060 2023-10-16T12:47:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.523+0000 7f57867fc640 1 -- 172.21.15.60:0/783901296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778031d60 con 0x7f57880c9060 2023-10-16T12:47:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 -- 172.21.15.60:0/783901296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 msgr2=0x7f578806f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 --2- 172.21.15.60:0/783901296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f578806f440 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f57880ce550 tx=0x7f577802b910 comp rx=0 tx=0).stop 2023-10-16T12:47:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 -- 172.21.15.60:0/783901296 shutdown_connections 2023-10-16T12:47:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 --2- 172.21.15.60:0/783901296 >> v2:172.21.15.144:3300/0 conn(0x7f57880c9440 0x7f578806f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 --2- 172.21.15.60:0/783901296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f578806f440 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 -- 172.21.15.60:0/783901296 >> 172.21.15.60:0/783901296 conn(0x7f57880ce6a0 msgr2=0x7f57880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 -- 172.21.15.60:0/783901296 shutdown_connections 2023-10-16T12:47:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.524+0000 7f578c9e9640 1 -- 172.21.15.60:0/783901296 wait complete. 2023-10-16T12:47:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.526+0000 7f578c9e9640 1 Processor -- start 2023-10-16T12:47:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.526+0000 7f578c9e9640 1 -- start start 2023-10-16T12:47:09.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f578c9e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f578806c9e0 0x7f57881636c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f578c9e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f57881f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f578c9e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57880c9440 0x7f5788161f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f578c9e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5788162620 con 0x7f57880c9060 2023-10-16T12:47:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f578c9e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5788162900 con 0x7f578806c9e0 2023-10-16T12:47:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f578c9e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5788162be0 con 0x7f57880c9440 2023-10-16T12:47:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f5786ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f57881f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f5786ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f57881f62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45220/0 (socket says 172.21.15.60:45220) 2023-10-16T12:47:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.527+0000 7f5786ffd640 1 -- 172.21.15.60:0/591497382 learned_addr learned my addr 172.21.15.60:0/591497382 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f57877fe640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f578806c9e0 0x7f57881636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5787fff640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57880c9440 0x7f5788161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5786ffd640 1 -- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57880c9440 msgr2=0x7f5788161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5786ffd640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57880c9440 0x7f5788161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5786ffd640 1 -- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f578806c9e0 msgr2=0x7f57881636c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5786ffd640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f578806c9e0 0x7f57881636c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5786ffd640 1 -- 172.21.15.60:0/591497382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57780094d0 con 0x7f57880c9060 2023-10-16T12:47:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5787fff640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57880c9440 0x7f5788161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f5786ffd640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f57881f62e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f577c00c400 tx=0x7f577c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c0035f0 con 0x7f57880c9060 2023-10-16T12:47:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c003750 con 0x7f57880c9060 2023-10-16T12:47:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c003b40 con 0x7f57880c9060 2023-10-16T12:47:09.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.528+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5788162ec0 con 0x7f57880c9060 2023-10-16T12:47:09.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.529+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57880c41c0 con 0x7f57880c9060 2023-10-16T12:47:09.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.530+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578804e9b0 con 0x7f57880c9060 2023-10-16T12:47:09.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.531+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f577c003ca0 con 0x7f57880c9060 2023-10-16T12:47:09.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.532+0000 7f576ffff640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f577406af90 0x7f577406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:09.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.532+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7f577c0889a0 con 0x7f57880c9060 2023-10-16T12:47:09.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.532+0000 7f57877fe640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f577406af90 0x7f577406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:09.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.534+0000 7f57877fe640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f577406af90 0x7f577406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5778036000 tx=0x7f577803c040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:09.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.536+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f577c0539c0 con 0x7f57880c9060 2023-10-16T12:47:09.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.780+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57880b4620 con 0x7f57880c9060 2023-10-16T12:47:09.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.781+0000 7f576ffff640 1 -- 172.21.15.60:0/591497382 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f577c017090 con 0x7f57880c9060 2023-10-16T12:47:09.784 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:09.784 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4634,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3658240614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3658240614},{"type":"v1","addr":"172.21.15.144:6835","nonce":3658240614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4825,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1265863417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1265863417},{"type":"v1","addr":"172.21.15.144:6837","nonce":1265863417}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5001,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1790637224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1790637224},{"type":"v1","addr":"172.21.15.60:6834","nonce":1790637224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1561859139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":1561859139},{"type":"v1","addr":"172.21.15.60:6835","nonce":1561859139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2023-10-16T12:47:09.784 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 19 2023-10-16T12:47:09.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f577406af90 msgr2=0x7f577406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:09.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f577406af90 0x7f577406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5778036000 tx=0x7f577803c040 comp rx=0 tx=0).stop 2023-10-16T12:47:09.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 msgr2=0x7f57881f62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:09.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f57881f62e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f577c00c400 tx=0x7f577c00c8c0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 shutdown_connections 2023-10-16T12:47:09.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f577406af90 0x7f577406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57880c9440 0x7f5788161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57880c9060 0x7f57881f62e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 --2- 172.21.15.60:0/591497382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f578806c9e0 0x7f57881636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:09.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 >> 172.21.15.60:0/591497382 conn(0x7f57880ce6a0 msgr2=0x7f578806b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:09.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 shutdown_connections 2023-10-16T12:47:09.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:09.784+0000 7f578c9e9640 1 -- 172.21.15.60:0/591497382 wait complete. 2023-10-16T12:47:09.805 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:47:09.805 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:47:09.805 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:47:09.806 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:47:09.806 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:47:09.806 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:47:09.807 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:47:09.807 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:47:09.807 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:47:09.807 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:47:09.808 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:47:09.808 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:47:09.808 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:47:09.808 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:47:09.809 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:47:09.809 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:47:09.829 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:47:09.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:47:09.831 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:47:09.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:47:09.834 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:47:09.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:47:09.839 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:47:09.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:47:09.879 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:47:09.880 DEBUG:teuthology.parallel:result is None 2023-10-16T12:47:09.886 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:47:09.887 DEBUG:teuthology.parallel:result is None 2023-10-16T12:47:09.888 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:47:09.888 DEBUG:teuthology.parallel:result is None 2023-10-16T12:47:09.895 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:47:09.895 DEBUG:teuthology.parallel:result is None 2023-10-16T12:47:09.896 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:47:10.044 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:47:10.045 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:47:10.046 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:47:10.049 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:47:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.088+0000 7fba1e187640 1 Processor -- start 2023-10-16T12:47:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.088+0000 7fba1e187640 1 -- start start 2023-10-16T12:47:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1e187640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba1815cd70 0x7fba180d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1e187640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba180b8480 0x7fba180cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1e187640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1815d150 con 0x7fba180d0be0 2023-10-16T12:47:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1e187640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba180d1180 con 0x7fba1815cd70 2023-10-16T12:47:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1e187640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba180d1460 con 0x7fba180b8480 2023-10-16T12:47:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1d185640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba1815cd70 0x7fba180d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1d986640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fba180d0be0 0x7fba180b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40632/0 (socket says 172.21.15.60:40632) 2023-10-16T12:47:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.089+0000 7fba1d986640 1 -- 172.21.15.60:0/1647796168 learned_addr learned my addr 172.21.15.60:0/1647796168 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.090+0000 7fba1c984640 1 --2- 172.21.15.60:0/1647796168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba180b8480 0x7fba180cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.090+0000 7fba1d185640 1 -- 172.21.15.60:0/1647796168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba180b8480 msgr2=0x7fba180cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.090+0000 7fba1d185640 1 --2- 172.21.15.60:0/1647796168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba180b8480 0x7fba180cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.090+0000 7fba1d185640 1 -- 172.21.15.60:0/1647796168 >> v1:172.21.15.60:6789/0 conn(0x7fba180d0be0 legacy=0x7fba180b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.090+0000 7fba1d185640 1 -- 172.21.15.60:0/1647796168 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba180d1740 con 0x7fba1815cd70 2023-10-16T12:47:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.090+0000 7fba0ffff640 1 -- 172.21.15.60:0/1647796168 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186801930 0 0) 0x7fba1815d150 con 0x7fba180d0be0 2023-10-16T12:47:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.091+0000 7fba1d185640 1 --2- 172.21.15.60:0/1647796168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1815cd70 0x7fba180d2220 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fba140097a0 tx=0x7fba1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1bf0cd8c50883a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.091+0000 7fba0ffff640 1 -- 172.21.15.60:0/1647796168 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14031830 con 0x7fba1815cd70 2023-10-16T12:47:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.091+0000 7fba0ffff640 1 -- 172.21.15.60:0/1647796168 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba14031990 con 0x7fba1815cd70 2023-10-16T12:47:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.091+0000 7fba0ffff640 1 -- 172.21.15.60:0/1647796168 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14031d20 con 0x7fba1815cd70 2023-10-16T12:47:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 -- 172.21.15.60:0/1647796168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1815cd70 msgr2=0x7fba180d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 --2- 172.21.15.60:0/1647796168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1815cd70 0x7fba180d2220 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fba140097a0 tx=0x7fba1402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 -- 172.21.15.60:0/1647796168 shutdown_connections 2023-10-16T12:47:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 --2- 172.21.15.60:0/1647796168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba180b8480 0x7fba180cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 --2- 172.21.15.60:0/1647796168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1815cd70 0x7fba180d2220 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 -- 172.21.15.60:0/1647796168 >> 172.21.15.60:0/1647796168 conn(0x7fba180cf570 msgr2=0x7fba180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 -- 172.21.15.60:0/1647796168 shutdown_connections 2023-10-16T12:47:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.092+0000 7fba1e187640 1 -- 172.21.15.60:0/1647796168 wait complete. 2023-10-16T12:47:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.094+0000 7fba1e187640 1 Processor -- start 2023-10-16T12:47:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.094+0000 7fba1e187640 1 -- start start 2023-10-16T12:47:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1e187640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 0x7fba18158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1e187640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba180d0be0 0x7fba180b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:10.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1e187640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba1815cd70 0x7fba181f5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1e187640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba18152a60 con 0x7fba180b8480 2023-10-16T12:47:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1e187640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b4e10 con 0x7fba180d0be0 2023-10-16T12:47:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1e187640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b50f0 con 0x7fba1815cd70 2023-10-16T12:47:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1d185640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 0x7fba18158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:10.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1d185640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 0x7fba18158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45258/0 (socket says 172.21.15.60:45258) 2023-10-16T12:47:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1d185640 1 -- 172.21.15.60:0/101179238 learned_addr learned my addr 172.21.15.60:0/101179238 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1c984640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba180d0be0 0x7fba180b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.095+0000 7fba1d986640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba1815cd70 0x7fba181f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d185640 1 -- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba1815cd70 msgr2=0x7fba181f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d185640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba1815cd70 0x7fba181f5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d185640 1 -- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba180d0be0 msgr2=0x7fba180b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d185640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba180d0be0 0x7fba180b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d185640 1 -- 172.21.15.60:0/101179238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba14009400 con 0x7fba180b8480 2023-10-16T12:47:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d986640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba1815cd70 0x7fba181f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:47:10.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1d185640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 0x7fba18158470 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fba1402bee0 tx=0x7fba140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1403a6f0 con 0x7fba180b8480 2023-10-16T12:47:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1403a870 con 0x7fba180b8480 2023-10-16T12:47:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba180b53d0 con 0x7fba180b8480 2023-10-16T12:47:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1403ac20 con 0x7fba180b8480 2023-10-16T12:47:10.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.096+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba181f6590 con 0x7fba180b8480 2023-10-16T12:47:10.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.098+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba180b58f0 con 0x7fba180b8480 2023-10-16T12:47:10.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.098+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fba1404c3b0 con 0x7fba180b8480 2023-10-16T12:47:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.099+0000 7fba0dffb640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb9fc06afc0 0x7fb9fc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.100+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5588+0+0 (secure 0 0 0) 0x7fba140ab5c0 con 0x7fba180b8480 2023-10-16T12:47:10.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.100+0000 7fba1c984640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb9fc06afc0 0x7fb9fc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.101+0000 7fba1c984640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb9fc06afc0 0x7fb9fc06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba18156a40 tx=0x7fba08008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:10.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.104+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba14076690 con 0x7fba180b8480 2023-10-16T12:47:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.311+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fba1804e9b0 con 0x7fba180b8480 2023-10-16T12:47:10.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.786+0000 7fba0dffb640 1 -- 172.21.15.60:0/101179238 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7fba14079cb0 con 0x7fba180b8480 2023-10-16T12:47:10.789 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:47:10.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb9fc06afc0 msgr2=0x7fb9fc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:10.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb9fc06afc0 0x7fb9fc06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba18156a40 tx=0x7fba08008040 comp rx=0 tx=0).stop 2023-10-16T12:47:10.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 msgr2=0x7fba18158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:10.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 0x7fba18158470 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fba1402bee0 tx=0x7fba140046c0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 shutdown_connections 2023-10-16T12:47:10.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fb9fc06afc0 0x7fb9fc06d480 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba1815cd70 0x7fba181f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba180d0be0 0x7fba180b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 --2- 172.21.15.60:0/101179238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba180b8480 0x7fba18158470 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:10.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 >> 172.21.15.60:0/101179238 conn(0x7fba180cf570 msgr2=0x7fba180d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:10.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 shutdown_connections 2023-10-16T12:47:10.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:10.790+0000 7fba1e187640 1 -- 172.21.15.60:0/101179238 wait complete. 2023-10-16T12:47:10.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:47:11.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.035+0000 7f7a0ee6f640 1 Processor -- start 2023-10-16T12:47:11.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.035+0000 7f7a0ee6f640 1 -- start start 2023-10-16T12:47:11.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.037+0000 7f7a0ee6f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f7a081615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.037+0000 7f7a0ee6f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0815e3e0 con 0x7f7a0815d3f0 2023-10-16T12:47:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.037+0000 7f7a0ee6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a080cd4a0 con 0x7f7a080cd0c0 2023-10-16T12:47:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.037+0000 7f7a0ee6f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0815e100 con 0x7f7a080b0b20 2023-10-16T12:47:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.037+0000 7f7a0d40c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7a0815d3f0 0x7f7a0815d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40646/0 (socket says 172.21.15.60:40646) 2023-10-16T12:47:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.037+0000 7f7a0d40c640 1 -- 172.21.15.60:0/316974980 learned_addr learned my addr 172.21.15.60:0/316974980 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:11.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a07fff640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f7a081615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256077992 0 0) 0x7f7a0815e3e0 con 0x7f7a0815d3f0 2023-10-16T12:47:11.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f8003520 con 0x7f7a0815d3f0 2023-10-16T12:47:11.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126458350 0 0) 0x7f79f8003520 con 0x7f7a0815d3f0 2023-10-16T12:47:11.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 msgr2=0x7f7a081615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:11.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f7a081615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 >> v1:172.21.15.144:6789/0 conn(0x7f7a080cd0c0 legacy=0x7f7a080b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08160b10 con 0x7f7a0815d3f0 2023-10-16T12:47:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7a000019f0 con 0x7f7a0815d3f0 2023-10-16T12:47:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 >> v1:172.21.15.60:6789/0 conn(0x7f7a0815d3f0 legacy=0x7f7a0815d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80039a0 0x7f79f8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.038+0000 7f7a077fe640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79f80047c0 0x7f79f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a077fe640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f79f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0815e100 con 0x7f79f80047c0 2023-10-16T12:47:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f800bae0 con 0x7f79f80039a0 2023-10-16T12:47:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f800bca0 con 0x7f7a080b0b20 2023-10-16T12:47:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a07fff640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79f80047c0 0x7f79f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a0cc0b640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80039a0 0x7f79f8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a0d40c640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f79f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a07fff640 1 -- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 msgr2=0x7f79f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a07fff640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f79f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a07fff640 1 -- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80039a0 msgr2=0x7f79f8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a07fff640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80039a0 0x7f79f8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a07fff640 1 -- 172.21.15.60:0/316974980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f8003520 con 0x7f79f80047c0 2023-10-16T12:47:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.039+0000 7f7a0d40c640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a080b0b20 0x7f79f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.041+0000 7f7a07fff640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79f80047c0 0x7f79f8004c40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009ac0 tx=0x7f79f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167a92233836c179 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.041+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f402c6b0 con 0x7f79f80047c0 2023-10-16T12:47:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.041+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f402c810 con 0x7f79f80047c0 2023-10-16T12:47:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.041+0000 7f7a077fe640 1 -- 172.21.15.60:0/316974980 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f402cba0 con 0x7f79f80047c0 2023-10-16T12:47:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/316974980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79f80047c0 msgr2=0x7f79f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/316974980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79f80047c0 0x7f79f8004c40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009ac0 tx=0x7f79f80057f0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0d40c640 1 -- 172.21.15.60:0/316974980 reap_dead start 2023-10-16T12:47:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/316974980 shutdown_connections 2023-10-16T12:47:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/316974980 >> 172.21.15.60:0/316974980 conn(0x7f7a080cb410 msgr2=0x7f7a080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/316974980 shutdown_connections 2023-10-16T12:47:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.042+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/316974980 wait complete. 2023-10-16T12:47:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.044+0000 7f7a0ee6f640 1 Processor -- start 2023-10-16T12:47:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.044+0000 7f7a0ee6f640 1 -- start start 2023-10-16T12:47:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0ee6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80047c0 0x7f7a081f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0ee6f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a0815b010 0x7f7a0815b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0ee6f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 0x7f7a08148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0ee6f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a08159320 con 0x7f7a08148980 2023-10-16T12:47:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0ee6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0815b9d0 con 0x7f79f80047c0 2023-10-16T12:47:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0ee6f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081594b0 con 0x7f7a0815b010 2023-10-16T12:47:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0d40c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 0x7f7a08148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0d40c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 0x7f7a08148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45272/0 (socket says 172.21.15.60:45272) 2023-10-16T12:47:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0d40c640 1 -- 172.21.15.60:0/145588708 learned_addr learned my addr 172.21.15.60:0/145588708 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:11.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.045+0000 7f7a0cc0b640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80047c0 0x7f7a081f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a07fff640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a0815b010 0x7f7a0815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0d40c640 1 -- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a0815b010 msgr2=0x7f7a0815b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0d40c640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a0815b010 0x7f7a0815b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0d40c640 1 -- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80047c0 msgr2=0x7f7a081f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0d40c640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80047c0 0x7f7a081f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0d40c640 1 -- 172.21.15.60:0/145588708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a081f2c10 con 0x7f7a08148980 2023-10-16T12:47:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0d40c640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 0x7f7a08148e00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7a080cc190 tx=0x7f7a0000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00004d60 con 0x7f7a08148980 2023-10-16T12:47:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0000bb20 con 0x7f7a08148980 2023-10-16T12:47:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000bed0 con 0x7f7a08148980 2023-10-16T12:47:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a081f2ef0 con 0x7f7a08148980 2023-10-16T12:47:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.046+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a081f34b0 con 0x7f7a08148980 2023-10-16T12:47:11.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.047+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0804e160 con 0x7f7a08148980 2023-10-16T12:47:11.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.049+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a00007070 con 0x7f7a08148980 2023-10-16T12:47:11.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.050+0000 7f7a057fa640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f79ec06af90 0x7f79ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.050+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7a000898f0 con 0x7f7a08148980 2023-10-16T12:47:11.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.050+0000 7f7a0cc0b640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f79ec06af90 0x7f79ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.051+0000 7f7a0cc0b640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f79ec06af90 0x7f79ec06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7a080afc20 tx=0x7f79fc009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:11.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.054+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a00054a30 con 0x7f7a08148980 2023-10-16T12:47:11.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.324+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7a080cba40 con 0x7f7a08148980 2023-10-16T12:47:11.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.325+0000 7f7a057fa640 1 -- 172.21.15.60:0/145588708 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7a00058050 con 0x7f7a08148980 2023-10-16T12:47:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:11.328 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:47:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:47:11.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:47:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:47:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.329 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:47:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:47:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.330 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:47:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:47:11.331 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:47:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:47:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:47:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:47:11.333 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:47:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:47:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:47:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:47:11.335 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:47:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:47:11.336 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:47:11.337 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:47:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:47:11.339 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:47:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:47:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:47:11.341 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:47:11.343 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:47:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:47:11.345 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:47:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.346 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.347 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:47:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:47:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.348 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:47:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:47:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.350 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:47:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:47:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:47:11.353 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:47:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.354 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:47:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:47:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:47:11.355 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:47:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:47:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:47:11.356 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:47:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:47:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:47:11.357 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:47:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:47:11.358 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:47:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:47:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:47:11.359 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:47:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:47:11.360 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:47:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:47:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:47:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:47:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:47:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:47:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.362 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:47:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:47:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:47:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:47:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:47:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.364 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:47:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:47:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:47:11.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:47:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:47:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:47:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:47:11.366 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:47:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:47:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:11.367 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:47:11.367 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:47:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f79ec06af90 msgr2=0x7f79ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f79ec06af90 0x7f79ec06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7a080afc20 tx=0x7f79fc009040 comp rx=0 tx=0).stop 2023-10-16T12:47:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 msgr2=0x7f7a08148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 0x7f7a08148e00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7a080cc190 tx=0x7f7a0000b4a0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 shutdown_connections 2023-10-16T12:47:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f79ec06af90 0x7f79ec06d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a0815b010 0x7f7a0815b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a08148980 0x7f7a08148e00 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 --2- 172.21.15.60:0/145588708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f80047c0 0x7f7a081f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 >> 172.21.15.60:0/145588708 conn(0x7f7a080cb410 msgr2=0x7f7a080c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 shutdown_connections 2023-10-16T12:47:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.328+0000 7f7a0ee6f640 1 -- 172.21.15.60:0/145588708 wait complete. 2023-10-16T12:47:11.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:47:11.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.605+0000 7f6cf8915640 1 Processor -- start 2023-10-16T12:47:11.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.605+0000 7f6cf8915640 1 -- start start 2023-10-16T12:47:11.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf8915640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6cf406c040 0x7f6cf406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf8915640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf415f4a0 con 0x7f6cf40cda80 2023-10-16T12:47:11.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf8915640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf406c960 con 0x7f6cf40ad970 2023-10-16T12:47:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf8915640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf415b9d0 con 0x7f6cf406c040 2023-10-16T12:47:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6cf406c040 0x7f6cf406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf359e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6cf40cda80 0x7f6cf415ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40648/0 (socket says 172.21.15.60:40648) 2023-10-16T12:47:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.607+0000 7f6cf359e640 1 -- 172.21.15.60:0/3527904821 learned_addr learned my addr 172.21.15.60:0/3527904821 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.608+0000 7f6cf2d9d640 1 -- 172.21.15.60:0/3527904821 >> v1:172.21.15.144:6789/0 conn(0x7f6cf40ad970 legacy=0x7f6cf415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.608+0000 7f6cf2d9d640 1 -- 172.21.15.60:0/3527904821 >> v1:172.21.15.60:6789/0 conn(0x7f6cf40cda80 legacy=0x7f6cf415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.608+0000 7f6cf2d9d640 1 -- 172.21.15.60:0/3527904821 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf415bcb0 con 0x7f6cf406c040 2023-10-16T12:47:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.608+0000 7f6cf259c640 1 -- 172.21.15.60:0/3527904821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965321439 0 0) 0x7f6cf415f4a0 con 0x7f6cf40cda80 2023-10-16T12:47:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.609+0000 7f6cf2d9d640 1 --2- 172.21.15.60:0/3527904821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf406c040 0x7f6cf406c420 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097a0 tx=0x7f6ce402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76fdf45059a466b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.609+0000 7f6cf259c640 1 -- 172.21.15.60:0/3527904821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4031830 con 0x7f6cf406c040 2023-10-16T12:47:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.609+0000 7f6cf259c640 1 -- 172.21.15.60:0/3527904821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce4031990 con 0x7f6cf406c040 2023-10-16T12:47:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.609+0000 7f6cf259c640 1 -- 172.21.15.60:0/3527904821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4031d20 con 0x7f6cf406c040 2023-10-16T12:47:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 -- 172.21.15.60:0/3527904821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf406c040 msgr2=0x7f6cf406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 --2- 172.21.15.60:0/3527904821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf406c040 0x7f6cf406c420 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40097a0 tx=0x7f6ce402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 -- 172.21.15.60:0/3527904821 shutdown_connections 2023-10-16T12:47:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 --2- 172.21.15.60:0/3527904821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf406c040 0x7f6cf406c420 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 -- 172.21.15.60:0/3527904821 >> 172.21.15.60:0/3527904821 conn(0x7f6cf40ce6a0 msgr2=0x7f6cf40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 -- 172.21.15.60:0/3527904821 shutdown_connections 2023-10-16T12:47:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.610+0000 7f6cf8915640 1 -- 172.21.15.60:0/3527904821 wait complete. 2023-10-16T12:47:11.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.612+0000 7f6cf8915640 1 Processor -- start 2023-10-16T12:47:11.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.612+0000 7f6cf8915640 1 -- start start 2023-10-16T12:47:11.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf8915640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6cf406c040 0x7f6cf40c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf8915640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cf40ad970 0x7f6cf40b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf8915640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 0x7f6cf4151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf8915640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40c25d0 con 0x7f6cf40cda80 2023-10-16T12:47:11.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf8915640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40c6980 con 0x7f6cf40ad970 2023-10-16T12:47:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf8915640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40b36c0 con 0x7f6cf406c040 2023-10-16T12:47:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 0x7f6cf4151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 0x7f6cf4151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45280/0 (socket says 172.21.15.60:45280) 2023-10-16T12:47:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf359e640 1 -- 172.21.15.60:0/1837023809 learned_addr learned my addr 172.21.15.60:0/1837023809 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cebfff640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cf40ad970 0x7f6cf40b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf359e640 1 -- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6cf406c040 msgr2=0x7f6cf40c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf359e640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6cf406c040 0x7f6cf40c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.613+0000 7f6cf359e640 1 -- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cf40ad970 msgr2=0x7f6cf40b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6cf359e640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cf40ad970 0x7f6cf40b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6cf359e640 1 -- 172.21.15.60:0/1837023809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4009400 con 0x7f6cf40cda80 2023-10-16T12:47:11.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6cebfff640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cf40ad970 0x7f6cf40b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:11.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6cf359e640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 0x7f6cf4151020 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c810 tx=0x7f6cec00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:11.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec00a640 con 0x7f6cf40cda80 2023-10-16T12:47:11.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec00a7c0 con 0x7f6cf40cda80 2023-10-16T12:47:11.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec00ab70 con 0x7f6cf40cda80 2023-10-16T12:47:11.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf4151560 con 0x7f6cf40cda80 2023-10-16T12:47:11.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.614+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf40c60e0 con 0x7f6cf40cda80 2023-10-16T12:47:11.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.615+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf406b330 con 0x7f6cf40cda80 2023-10-16T12:47:11.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.616+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6cec00acd0 con 0x7f6cf40cda80 2023-10-16T12:47:11.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.617+0000 7f6ceb7fe640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6cd406af90 0x7f6cd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:11.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.617+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6cec088940 con 0x7f6cf40cda80 2023-10-16T12:47:11.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.618+0000 7f6cf2d9d640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6cd406af90 0x7f6cd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:11.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.619+0000 7f6cf2d9d640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6cd406af90 0x7f6cd406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4036000 tx=0x7f6ce4030000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:11.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.622+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6cec053c20 con 0x7f6cf40cda80 2023-10-16T12:47:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.822+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cf40c2710 con 0x7f6cf40cda80 2023-10-16T12:47:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.824+0000 7f6ceb7fe640 1 -- 172.21.15.60:0/1837023809 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+12845 (secure 0 0 0) 0x7f6cec018020 con 0x7f6cf40cda80 2023-10-16T12:47:11.826 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":63,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:47:10.802529+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:47:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.826+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6cd406af90 msgr2=0x7f6cd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.826+0000 7f6cf8915640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6cd406af90 0x7f6cd406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4036000 tx=0x7f6ce4030000 comp rx=0 tx=0).stop 2023-10-16T12:47:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.826+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 msgr2=0x7f6cf4151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:11.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.826+0000 7f6cf8915640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 0x7f6cf4151020 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c810 tx=0x7f6cec00ccd0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 shutdown_connections 2023-10-16T12:47:11.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6cd406af90 0x7f6cd406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40cda80 0x7f6cf4151020 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cf40ad970 0x7f6cf40b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 --2- 172.21.15.60:0/1837023809 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6cf406c040 0x7f6cf40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 >> 172.21.15.60:0/1837023809 conn(0x7f6cf40ce6a0 msgr2=0x7f6cf415e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 shutdown_connections 2023-10-16T12:47:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:11.827+0000 7f6cf8915640 1 -- 172.21.15.60:0/1837023809 wait complete. 2023-10-16T12:47:11.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:47:11.849 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:47:12.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.074+0000 7f916f421640 1 Processor -- start 2023-10-16T12:47:12.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.074+0000 7f916f421640 1 -- start start 2023-10-16T12:47:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.075+0000 7f916f421640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f916815f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.075+0000 7f916f421640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916806c140 con 0x7f91680cbf20 2023-10-16T12:47:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.075+0000 7f916f421640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91680af380 con 0x7f916815aee0 2023-10-16T12:47:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.075+0000 7f916f421640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916815b2c0 con 0x7f91680aeef0 2023-10-16T12:47:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f916c9bc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f91680cbf20 0x7f916815e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40650/0 (socket says 172.21.15.60:40650) 2023-10-16T12:47:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f916c9bc640 1 -- 172.21.15.60:0/1550985539 learned_addr learned my addr 172.21.15.60:0/1550985539 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f916d9be640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f916815f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739717592 0 0) 0x7f916806c140 con 0x7f91680cbf20 2023-10-16T12:47:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9154003520 con 0x7f91680cbf20 2023-10-16T12:47:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210409140 0 0) 0x7f9154003520 con 0x7f91680cbf20 2023-10-16T12:47:12.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 msgr2=0x7f916815f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f915bfff640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f916815f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.076+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 >> v1:172.21.15.144:6789/0 conn(0x7f916815aee0 legacy=0x7f91680cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916806b270 con 0x7f91680cbf20 2023-10-16T12:47:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f915c001670 con 0x7f91680cbf20 2023-10-16T12:47:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 >> v1:172.21.15.60:6789/0 conn(0x7f91680cbf20 legacy=0x7f916815e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f91540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9154003ee0 0x7f91540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154004d00 0x7f9154005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916815b2c0 con 0x7f9154004d00 2023-10-16T12:47:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915400ba90 con 0x7f9154003ee0 2023-10-16T12:47:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915400bc50 con 0x7f91680aeef0 2023-10-16T12:47:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f916d9be640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154004d00 0x7f9154005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f916c9bc640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9154003ee0 0x7f91540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f916d9be640 1 -- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 msgr2=0x7f91540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:12.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f916d1bd640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f91540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.077+0000 7f916d9be640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f91540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.078+0000 7f916d9be640 1 -- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9154003ee0 msgr2=0x7f91540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.078+0000 7f916d9be640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9154003ee0 0x7f91540047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.078+0000 7f916d9be640 1 -- 172.21.15.60:0/1550985539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154003520 con 0x7f9154004d00 2023-10-16T12:47:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.078+0000 7f916d1bd640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91680aeef0 0x7f91540039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f916d9be640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154004d00 0x7f9154005180 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9160009fc0 tx=0x7f91540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6818f7ea5e161aa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916002cbf0 con 0x7f9154004d00 2023-10-16T12:47:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000f040 con 0x7f9154004d00 2023-10-16T12:47:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f915bfff640 1 -- 172.21.15.60:0/1550985539 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160034440 con 0x7f9154004d00 2023-10-16T12:47:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f916f421640 1 -- 172.21.15.60:0/1550985539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154004d00 msgr2=0x7f9154005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f916f421640 1 --2- 172.21.15.60:0/1550985539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154004d00 0x7f9154005180 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9160009fc0 tx=0x7f91540057f0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.079+0000 7f916d9be640 1 -- 172.21.15.60:0/1550985539 reap_dead start 2023-10-16T12:47:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.080+0000 7f916f421640 1 -- 172.21.15.60:0/1550985539 shutdown_connections 2023-10-16T12:47:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.080+0000 7f916f421640 1 -- 172.21.15.60:0/1550985539 >> 172.21.15.60:0/1550985539 conn(0x7f91680cfac0 msgr2=0x7f91680cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.080+0000 7f916f421640 1 -- 172.21.15.60:0/1550985539 shutdown_connections 2023-10-16T12:47:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.080+0000 7f916f421640 1 -- 172.21.15.60:0/1550985539 wait complete. 2023-10-16T12:47:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.082+0000 7f916f421640 1 Processor -- start 2023-10-16T12:47:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.082+0000 7f916f421640 1 -- start start 2023-10-16T12:47:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916f421640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9154004d00 0x7f91680c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916f421640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91681508f0 0x7f91681f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916f421640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 0x7f91681f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916f421640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91681513b0 con 0x7f91681f2450 2023-10-16T12:47:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916f421640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91681f2e10 con 0x7f91681508f0 2023-10-16T12:47:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916f421640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91680c79d0 con 0x7f9154004d00 2023-10-16T12:47:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 0x7f91681f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 0x7f91681f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45300/0 (socket says 172.21.15.60:45300) 2023-10-16T12:47:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 -- 172.21.15.60:0/3445151610 learned_addr learned my addr 172.21.15.60:0/3445151610 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d1bd640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9154004d00 0x7f91680c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 -- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9154004d00 msgr2=0x7f91680c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9154004d00 0x7f91680c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916c9bc640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91681508f0 0x7f91681f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 -- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91681508f0 msgr2=0x7f91681f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.083+0000 7f916d9be640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91681508f0 0x7f91681f1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f916d9be640 1 -- 172.21.15.60:0/3445151610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680b1020 con 0x7f91681f2450 2023-10-16T12:47:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f916d1bd640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9154004d00 0x7f91680c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f916d9be640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 0x7f91681f28d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9160002230 tx=0x7f9160009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916002de80 con 0x7f91681f2450 2023-10-16T12:47:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9160034ad0 con 0x7f91681f2450 2023-10-16T12:47:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160034e60 con 0x7f91681f2450 2023-10-16T12:47:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91680b1300 con 0x7f91681f2450 2023-10-16T12:47:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.084+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91680b18c0 con 0x7f91681f2450 2023-10-16T12:47:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.086+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916804e160 con 0x7f91681f2450 2023-10-16T12:47:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.086+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f916003e440 con 0x7f91681f2450 2023-10-16T12:47:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.087+0000 7f9159ffb640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f914c06af90 0x7f914c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.087+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91600aaa20 con 0x7f91681f2450 2023-10-16T12:47:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.088+0000 7f916d1bd640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f914c06af90 0x7f914c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.089+0000 7f916d1bd640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f914c06af90 0x7f914c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9164001270 tx=0x7f9164009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.092+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9160075d00 con 0x7f91681f2450 2023-10-16T12:47:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.290+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91680b1a80 con 0x7f91681f2450 2023-10-16T12:47:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.291+0000 7f9159ffb640 1 -- 172.21.15.60:0/3445151610 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9160079320 con 0x7f91681f2450 2023-10-16T12:47:12.294 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:47:12.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f914c06af90 msgr2=0x7f914c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f914c06af90 0x7f914c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9164001270 tx=0x7f9164009040 comp rx=0 tx=0).stop 2023-10-16T12:47:12.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 msgr2=0x7f91681f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 0x7f91681f28d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9160002230 tx=0x7f9160009750 comp rx=0 tx=0).stop 2023-10-16T12:47:12.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 shutdown_connections 2023-10-16T12:47:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f914c06af90 0x7f914c06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91681f2450 0x7f91681f28d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91681508f0 0x7f91681f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 --2- 172.21.15.60:0/3445151610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9154004d00 0x7f91680c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.294+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 >> 172.21.15.60:0/3445151610 conn(0x7f91680cfac0 msgr2=0x7f916806c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.295+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 shutdown_connections 2023-10-16T12:47:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.295+0000 7f916f421640 1 -- 172.21.15.60:0/3445151610 wait complete. 2023-10-16T12:47:12.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:47:12.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.549+0000 7f4ec0b2e640 1 Processor -- start 2023-10-16T12:47:12.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.549+0000 7f4ec0b2e640 1 -- start start 2023-10-16T12:47:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ec0b2e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ebc159290 0x7f4ebc0cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ec0b2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc0c9cf0 0x7f4ebc0ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ec0b2e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0ca580 con 0x7f4ebc06fa20 2023-10-16T12:47:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ec0b2e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc0ad5e0 con 0x7f4ebc159290 2023-10-16T12:47:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ec0b2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0ca2a0 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ebb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ebc159290 0x7f4ebc0cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ebaffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ebc06fa20 0x7f4ebc0c95e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40652/0 (socket says 172.21.15.60:40652) 2023-10-16T12:47:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.551+0000 7f4ebaffd640 1 -- 172.21.15.60:0/2917616381 learned_addr learned my addr 172.21.15.60:0/2917616381 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.552+0000 7f4ebbfff640 1 --2- 172.21.15.60:0/2917616381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc0c9cf0 0x7f4ebc0ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.552+0000 7f4ebb7fe640 1 -- 172.21.15.60:0/2917616381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc0c9cf0 msgr2=0x7f4ebc0ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.552+0000 7f4ebb7fe640 1 --2- 172.21.15.60:0/2917616381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc0c9cf0 0x7f4ebc0ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.552+0000 7f4ebb7fe640 1 -- 172.21.15.60:0/2917616381 >> v1:172.21.15.60:6789/0 conn(0x7f4ebc06fa20 legacy=0x7f4ebc0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.552+0000 7f4ebb7fe640 1 -- 172.21.15.60:0/2917616381 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0acf20 con 0x7f4ebc159290 2023-10-16T12:47:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.552+0000 7f4eba7fc640 1 -- 172.21.15.60:0/2917616381 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781800880 0 0) 0x7f4ebc0ca580 con 0x7f4ebc06fa20 2023-10-16T12:47:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.553+0000 7f4ebb7fe640 1 --2- 172.21.15.60:0/2917616381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc159290 0x7f4ebc0cdb70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009dc0 tx=0x7f4eac02be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1208125357efcc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4eba7fc640 1 -- 172.21.15.60:0/2917616381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac02cc30 con 0x7f4ebc159290 2023-10-16T12:47:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4eba7fc640 1 -- 172.21.15.60:0/2917616381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac02cd90 con 0x7f4ebc159290 2023-10-16T12:47:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4eba7fc640 1 -- 172.21.15.60:0/2917616381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac034660 con 0x7f4ebc159290 2023-10-16T12:47:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2917616381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc159290 msgr2=0x7f4ebc0cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2917616381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc159290 0x7f4ebc0cdb70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009dc0 tx=0x7f4eac02be80 comp rx=0 tx=0).stop 2023-10-16T12:47:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2917616381 shutdown_connections 2023-10-16T12:47:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2917616381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc0c9cf0 0x7f4ebc0ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2917616381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc159290 0x7f4ebc0cdb70 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2917616381 >> 172.21.15.60:0/2917616381 conn(0x7f4ebc0b0a80 msgr2=0x7f4ebc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2917616381 shutdown_connections 2023-10-16T12:47:12.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.554+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2917616381 wait complete. 2023-10-16T12:47:12.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.556+0000 7f4ec0b2e640 1 Processor -- start 2023-10-16T12:47:12.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.557+0000 7f4ec0b2e640 1 -- start start 2023-10-16T12:47:12.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ec0b2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc06fa20 0x7f4ebc156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ec0b2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 0x7f4ebc09c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ec0b2e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ebc159290 0x7f4ebc0ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ec0b2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc09c850 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ec0b2e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc156c40 con 0x7f4ebc159290 2023-10-16T12:47:12.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ec0b2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc09b270 con 0x7f4ebc06fa20 2023-10-16T12:47:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 0x7f4ebc09c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 0x7f4ebc09c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45328/0 (socket says 172.21.15.60:45328) 2023-10-16T12:47:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 -- 172.21.15.60:0/2733600643 learned_addr learned my addr 172.21.15.60:0/2733600643 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebb7fe640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc06fa20 0x7f4ebc156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebbfff640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ebc159290 0x7f4ebc0ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 -- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc06fa20 msgr2=0x7f4ebc156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc06fa20 0x7f4ebc156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 -- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ebc159290 msgr2=0x7f4ebc0ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ebc159290 0x7f4ebc0ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebaffd640 1 -- 172.21.15.60:0/2733600643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac009a70 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.558+0000 7f4ebbfff640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ebc159290 0x7f4ebc0ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:12.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.559+0000 7f4ebaffd640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 0x7f4ebc09c310 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c4d0 tx=0x7f4eb000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:12.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.559+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00074e0 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.559+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0007660 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.559+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0004a20 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.559+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc09b550 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.559+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc09bb10 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.560+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc04e9b0 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.561+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4eb0004b80 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.562+0000 7f4ea3fff640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4ea806af90 0x7f4ea806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:12.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.562+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4eb0088640 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.563+0000 7f4ebb7fe640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4ea806af90 0x7f4ea806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:12.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.564+0000 7f4ebb7fe640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4ea806af90 0x7f4ea806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4eac036000 tx=0x7f4eac030000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:12.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.567+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4eb0053920 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.767+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4ebc0b4f10 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.768+0000 7f4ea3fff640 1 -- 172.21.15.60:0/2733600643 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4eb0018090 con 0x7f4ebc0c9cf0 2023-10-16T12:47:12.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:12.771 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:47:12.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:12.772 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:47:12.772 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:47:12.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:12.772 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:47:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.771+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4ea806af90 msgr2=0x7f4ea806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.771+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4ea806af90 0x7f4ea806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4eac036000 tx=0x7f4eac030000 comp rx=0 tx=0).stop 2023-10-16T12:47:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 msgr2=0x7f4ebc09c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 0x7f4ebc09c310 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c4d0 tx=0x7f4eb000c990 comp rx=0 tx=0).stop 2023-10-16T12:47:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 shutdown_connections 2023-10-16T12:47:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4ea806af90 0x7f4ea806d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ebc159290 0x7f4ebc0ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ebc0c9cf0 0x7f4ebc09c310 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 --2- 172.21.15.60:0/2733600643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ebc06fa20 0x7f4ebc156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 >> 172.21.15.60:0/2733600643 conn(0x7f4ebc0b0a80 msgr2=0x7f4ebc15b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 shutdown_connections 2023-10-16T12:47:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:12.772+0000 7f4ec0b2e640 1 -- 172.21.15.60:0/2733600643 wait complete. 2023-10-16T12:47:12.795 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:47:12.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:47:13.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.026+0000 7f8a9a36c640 1 Processor -- start 2023-10-16T12:47:13.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.026+0000 7f8a9a36c640 1 -- start start 2023-10-16T12:47:13.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.027+0000 7f8a9a36c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a9415ad80 0x7f8a9406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.027+0000 7f8a9a36c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a940cae40 0x7f8a9406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.027+0000 7f8a9a36c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9406b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.027+0000 7f8a9a36c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a940c8e70 con 0x7f8a940cae40 2023-10-16T12:47:13.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.027+0000 7f8a9a36c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a940b4170 con 0x7f8a9415ad80 2023-10-16T12:47:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.027+0000 7f8a9a36c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a940adde0 con 0x7f8a940ad7f0 2023-10-16T12:47:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a940cae40 0x7f8a9406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a940cae40 0x7f8a9406d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45336/0 (socket says 172.21.15.60:45336) 2023-10-16T12:47:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 -- 172.21.15.60:0/86589707 learned_addr learned my addr 172.21.15.60:0/86589707 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a99b6b640 1 --2- 172.21.15.60:0/86589707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9406b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a9936a640 1 --2- 172.21.15.60:0/86589707 >> v2:172.21.15.144:3300/0 conn(0x7f8a9415ad80 0x7f8a9406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 -- 172.21.15.60:0/86589707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 msgr2=0x7f8a9406b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 --2- 172.21.15.60:0/86589707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9406b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 -- 172.21.15.60:0/86589707 >> v2:172.21.15.144:3300/0 conn(0x7f8a9415ad80 msgr2=0x7f8a9406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 --2- 172.21.15.60:0/86589707 >> v2:172.21.15.144:3300/0 conn(0x7f8a9415ad80 0x7f8a9406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a98b69640 1 -- 172.21.15.60:0/86589707 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a940b4410 con 0x7f8a940cae40 2023-10-16T12:47:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.028+0000 7f8a9936a640 1 --2- 172.21.15.60:0/86589707 >> v2:172.21.15.144:3300/0 conn(0x7f8a9415ad80 0x7f8a9406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.029+0000 7f8a98b69640 1 --2- 172.21.15.60:0/86589707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a940cae40 0x7f8a9406d0b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a94070710 tx=0x7f8a8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31834fc901c9a130 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.030+0000 7f8a8bfff640 1 -- 172.21.15.60:0/86589707 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031870 con 0x7f8a940cae40 2023-10-16T12:47:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.030+0000 7f8a8bfff640 1 -- 172.21.15.60:0/86589707 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a840319d0 con 0x7f8a940cae40 2023-10-16T12:47:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.030+0000 7f8a8bfff640 1 -- 172.21.15.60:0/86589707 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031d60 con 0x7f8a940cae40 2023-10-16T12:47:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.030+0000 7f8a9a36c640 1 -- 172.21.15.60:0/86589707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a940cae40 msgr2=0x7f8a9406d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.030+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/86589707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a940cae40 0x7f8a9406d0b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a94070710 tx=0x7f8a8402b910 comp rx=0 tx=0).stop 2023-10-16T12:47:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 -- 172.21.15.60:0/86589707 shutdown_connections 2023-10-16T12:47:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/86589707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9406b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/86589707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a940cae40 0x7f8a9406d0b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/86589707 >> v2:172.21.15.144:3300/0 conn(0x7f8a9415ad80 0x7f8a9406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 -- 172.21.15.60:0/86589707 >> 172.21.15.60:0/86589707 conn(0x7f8a940ce820 msgr2=0x7f8a940cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 -- 172.21.15.60:0/86589707 shutdown_connections 2023-10-16T12:47:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.031+0000 7f8a9a36c640 1 -- 172.21.15.60:0/86589707 wait complete. 2023-10-16T12:47:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.032+0000 7f8a9a36c640 1 Processor -- start 2023-10-16T12:47:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.033+0000 7f8a9a36c640 1 -- start start 2023-10-16T12:47:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.033+0000 7f8a9a36c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9414fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.033+0000 7f8a9a36c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a940cae40 0x7f8a9415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a9a36c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 0x7f8a94158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a9a36c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a94154920 con 0x7f8a9415ad80 2023-10-16T12:47:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a9a36c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a94153d30 con 0x7f8a940cae40 2023-10-16T12:47:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a9a36c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a94153f80 con 0x7f8a940ad7f0 2023-10-16T12:47:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 0x7f8a94158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 0x7f8a94158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45350/0 (socket says 172.21.15.60:45350) 2023-10-16T12:47:13.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 -- 172.21.15.60:0/1360730125 learned_addr learned my addr 172.21.15.60:0/1360730125 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:13.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a9936a640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9414fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a98b69640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a940cae40 0x7f8a9415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 -- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 msgr2=0x7f8a9414fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:13.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9414fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 -- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a940cae40 msgr2=0x7f8a9415dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:13.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a940cae40 0x7f8a9415dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a99b6b640 1 -- 172.21.15.60:0/1360730125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840094d0 con 0x7f8a9415ad80 2023-10-16T12:47:13.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.034+0000 7f8a9936a640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9414fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:13.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.035+0000 7f8a99b6b640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 0x7f8a94158300 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00ccf0 tx=0x7f8a8c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:13.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.035+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c003b90 con 0x7f8a9415ad80 2023-10-16T12:47:13.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.035+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c003cf0 con 0x7f8a9415ad80 2023-10-16T12:47:13.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.035+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a94154260 con 0x7f8a9415ad80 2023-10-16T12:47:13.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.035+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c004240 con 0x7f8a9415ad80 2023-10-16T12:47:13.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.035+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a94150430 con 0x7f8a9415ad80 2023-10-16T12:47:13.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.036+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9404e9b0 con 0x7f8a9415ad80 2023-10-16T12:47:13.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.037+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a8c0043a0 con 0x7f8a9415ad80 2023-10-16T12:47:13.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.038+0000 7f8a89ffb640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8a7806af90 0x7f8a7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:13.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.038+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8a8c012070 con 0x7f8a9415ad80 2023-10-16T12:47:13.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.039+0000 7f8a9936a640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8a7806af90 0x7f8a7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:13.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.040+0000 7f8a9936a640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8a7806af90 0x7f8a7806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8a90006fd0 tx=0x7f8a90008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:13.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.042+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a8c055060 con 0x7f8a9415ad80 2023-10-16T12:47:13.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.240+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8a940c9d80 con 0x7f8a9415ad80 2023-10-16T12:47:13.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.808+0000 7f8a89ffb640 1 -- 172.21.15.60:0/1360730125 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8a8c058680 con 0x7f8a9415ad80 2023-10-16T12:47:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:47:13.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8a7806af90 msgr2=0x7f8a7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:13.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8a7806af90 0x7f8a7806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8a90006fd0 tx=0x7f8a90008040 comp rx=0 tx=0).stop 2023-10-16T12:47:13.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 msgr2=0x7f8a94158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:13.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 0x7f8a94158300 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00ccf0 tx=0x7f8a8c00a660 comp rx=0 tx=0).stop 2023-10-16T12:47:13.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 shutdown_connections 2023-10-16T12:47:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f8a7806af90 0x7f8a7806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a9415ad80 0x7f8a94158300 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a940cae40 0x7f8a9415dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 --2- 172.21.15.60:0/1360730125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a940ad7f0 0x7f8a9414fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.811+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 >> 172.21.15.60:0/1360730125 conn(0x7f8a940ce820 msgr2=0x7f8a940aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.812+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 shutdown_connections 2023-10-16T12:47:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:13.812+0000 7f8a9a36c640 1 -- 172.21.15.60:0/1360730125 wait complete. 2023-10-16T12:47:13.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:47:14.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.064+0000 7f036a90a640 1 Processor -- start 2023-10-16T12:47:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.064+0000 7f036a90a640 1 -- start start 2023-10-16T12:47:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a90a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f036415ac70 0x7f03640cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a90a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a90a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03640cc850 con 0x7f03640b4160 2023-10-16T12:47:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a90a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03640ccb30 con 0x7f036415ac70 2023-10-16T12:47:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a90a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03640cce10 con 0x7f036406a4b0 2023-10-16T12:47:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f036415ac70 0x7f03640cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f036415ac70 0x7f03640cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45364/0 (socket says 172.21.15.60:45364) 2023-10-16T12:47:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 -- 172.21.15.60:0/2993351318 learned_addr learned my addr 172.21.15.60:0/2993351318 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369107640 1 --1- 172.21.15.60:0/2993351318 >> v1:172.21.15.60:6789/0 conn(0x7f03640b4160 0x7f0364069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40660/0 (socket says 172.21.15.60:40660) 2023-10-16T12:47:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 -- 172.21.15.60:0/2993351318 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 msgr2=0x7f03640cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a109640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0368906640 1 -- 172.21.15.60:0/2993351318 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754811322 0 0) 0x7f03640cc850 con 0x7f03640b4160 2023-10-16T12:47:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 -- 172.21.15.60:0/2993351318 >> v1:172.21.15.60:6789/0 conn(0x7f03640b4160 legacy=0x7f0364069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f0369908640 1 -- 172.21.15.60:0/2993351318 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03640cd0f0 con 0x7f036415ac70 2023-10-16T12:47:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.066+0000 7f036a109640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.068+0000 7f0369908640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03640cc270 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0360009870 tx=0x7f036002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d7981c0d29fdd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.068+0000 7f0368906640 1 -- 172.21.15.60:0/2993351318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360030840 con 0x7f036415ac70 2023-10-16T12:47:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.068+0000 7f0368906640 1 -- 172.21.15.60:0/2993351318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03600309a0 con 0x7f036415ac70 2023-10-16T12:47:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.068+0000 7f0368906640 1 -- 172.21.15.60:0/2993351318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360030d30 con 0x7f036415ac70 2023-10-16T12:47:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.068+0000 7f036a90a640 1 -- 172.21.15.60:0/2993351318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 msgr2=0x7f03640cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.068+0000 7f036a90a640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03640cc270 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0360009870 tx=0x7f036002b980 comp rx=0 tx=0).stop 2023-10-16T12:47:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.069+0000 7f036a90a640 1 -- 172.21.15.60:0/2993351318 shutdown_connections 2023-10-16T12:47:14.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.069+0000 7f036a90a640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.069+0000 7f036a90a640 1 --2- 172.21.15.60:0/2993351318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03640cc270 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.069+0000 7f036a90a640 1 -- 172.21.15.60:0/2993351318 >> 172.21.15.60:0/2993351318 conn(0x7f03640ce6a0 msgr2=0x7f03640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:14.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.069+0000 7f036a90a640 1 -- 172.21.15.60:0/2993351318 shutdown_connections 2023-10-16T12:47:14.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.069+0000 7f036a90a640 1 -- 172.21.15.60:0/2993351318 wait complete. 2023-10-16T12:47:14.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.070+0000 7f036a90a640 1 Processor -- start 2023-10-16T12:47:14.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.071+0000 7f036a90a640 1 -- start start 2023-10-16T12:47:14.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640c2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:14.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03640b4160 0x7f03641f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:14.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03641ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:14.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03641f0ff0 con 0x7f036415ac70 2023-10-16T12:47:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03641f3600 con 0x7f03640b4160 2023-10-16T12:47:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03641f38e0 con 0x7f036406a4b0 2023-10-16T12:47:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03641ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03641ef790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45376/0 (socket says 172.21.15.60:45376) 2023-10-16T12:47:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 -- 172.21.15.60:0/3369631023 learned_addr learned my addr 172.21.15.60:0/3369631023 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:14.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f0369908640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:14.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f0369107640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03640b4160 0x7f03641f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:14.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 -- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 msgr2=0x7f03640c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:14.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 -- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03640b4160 msgr2=0x7f03641f0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:14.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03640b4160 0x7f03641f0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 -- 172.21.15.60:0/3369631023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03600094d0 con 0x7f036415ac70 2023-10-16T12:47:14.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f0369107640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03640b4160 0x7f03641f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:14.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f0369908640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:14.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a109640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03641ef790 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f035c00cf20 tx=0x7f035c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:14.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c00d840 con 0x7f036415ac70 2023-10-16T12:47:14.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c00d9a0 con 0x7f036415ac70 2023-10-16T12:47:14.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c00dd50 con 0x7f036415ac70 2023-10-16T12:47:14.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03640c77f0 con 0x7f036415ac70 2023-10-16T12:47:14.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.073+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03640c7db0 con 0x7f036415ac70 2023-10-16T12:47:14.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.074+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03641f3a70 con 0x7f036415ac70 2023-10-16T12:47:14.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.076+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f035c01e3b0 con 0x7f036415ac70 2023-10-16T12:47:14.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.077+0000 7f035a7fc640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f034806afc0 0x7f034806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:14.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.077+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f035c089ff0 con 0x7f036415ac70 2023-10-16T12:47:14.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.077+0000 7f0369908640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f034806afc0 0x7f034806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:14.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.079+0000 7f0369908640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f034806afc0 0x7f034806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0360036000 tx=0x7f036003b040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:14.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.082+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f035c055160 con 0x7f036415ac70 2023-10-16T12:47:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.280+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f036404e9b0 con 0x7f036415ac70 2023-10-16T12:47:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.815+0000 7f035a7fc640 1 -- 172.21.15.60:0/3369631023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7f035c01a0d0 con 0x7f036415ac70 2023-10-16T12:47:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:47:14.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f034806afc0 msgr2=0x7f034806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f034806afc0 0x7f034806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0360036000 tx=0x7f036003b040 comp rx=0 tx=0).stop 2023-10-16T12:47:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 msgr2=0x7f03641ef790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03641ef790 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f035c00cf20 tx=0x7f035c00a7d0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 shutdown_connections 2023-10-16T12:47:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f034806afc0 0x7f034806d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f036415ac70 0x7f03641ef790 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03640b4160 0x7f03641f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 --2- 172.21.15.60:0/3369631023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f036406a4b0 0x7f03640c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 >> 172.21.15.60:0/3369631023 conn(0x7f03640ce6a0 msgr2=0x7f0364158600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 shutdown_connections 2023-10-16T12:47:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:14.819+0000 7f036a90a640 1 -- 172.21.15.60:0/3369631023 wait complete. 2023-10-16T12:47:14.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:47:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.045+0000 7fd15c8d1640 1 Processor -- start 2023-10-16T12:47:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.046+0000 7fd15c8d1640 1 -- start start 2023-10-16T12:47:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd15c8d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd15815cd70 0x7fd1580d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd15c8d1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1580b8480 0x7fd1580cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd15c8d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580d0be0 0x7fd1580cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd15c8d1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15815d150 con 0x7fd1580b8480 2023-10-16T12:47:15.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd15c8d1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1580d1180 con 0x7fd15815cd70 2023-10-16T12:47:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd15c8d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580d1460 con 0x7fd1580d0be0 2023-10-16T12:47:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.047+0000 7fd14ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1580b8480 0x7fd1580cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1580b8480 0x7fd1580cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45384/0 (socket says 172.21.15.60:45384) 2023-10-16T12:47:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 -- 172.21.15.60:0/1487019302 learned_addr learned my addr 172.21.15.60:0/1487019302 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd1577fe640 1 --2- 172.21.15.60:0/1487019302 >> v2:172.21.15.144:3300/0 conn(0x7fd15815cd70 0x7fd1580d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd157fff640 1 --2- 172.21.15.60:0/1487019302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580d0be0 0x7fd1580cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 -- 172.21.15.60:0/1487019302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580d0be0 msgr2=0x7fd1580cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 --2- 172.21.15.60:0/1487019302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580d0be0 0x7fd1580cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 -- 172.21.15.60:0/1487019302 >> v2:172.21.15.144:3300/0 conn(0x7fd15815cd70 msgr2=0x7fd1580d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 --2- 172.21.15.60:0/1487019302 >> v2:172.21.15.144:3300/0 conn(0x7fd15815cd70 0x7fd1580d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.048+0000 7fd14ffff640 1 -- 172.21.15.60:0/1487019302 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580d1740 con 0x7fd1580b8480 2023-10-16T12:47:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.049+0000 7fd14ffff640 1 --2- 172.21.15.60:0/1487019302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1580b8480 0x7fd1580cd7c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd1580b56a0 tx=0x7fd14002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca28e8e8df05ee35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.050+0000 7fd156ffd640 1 -- 172.21.15.60:0/1487019302 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140031860 con 0x7fd1580b8480 2023-10-16T12:47:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.050+0000 7fd156ffd640 1 -- 172.21.15.60:0/1487019302 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1400319c0 con 0x7fd1580b8480 2023-10-16T12:47:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.050+0000 7fd156ffd640 1 -- 172.21.15.60:0/1487019302 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140031d50 con 0x7fd1580b8480 2023-10-16T12:47:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.050+0000 7fd15c8d1640 1 -- 172.21.15.60:0/1487019302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1580b8480 msgr2=0x7fd1580cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.050+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/1487019302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1580b8480 0x7fd1580cd7c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd1580b56a0 tx=0x7fd14002b900 comp rx=0 tx=0).stop 2023-10-16T12:47:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 -- 172.21.15.60:0/1487019302 shutdown_connections 2023-10-16T12:47:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/1487019302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580d0be0 0x7fd1580cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/1487019302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1580b8480 0x7fd1580cd7c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/1487019302 >> v2:172.21.15.144:3300/0 conn(0x7fd15815cd70 0x7fd1580d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 -- 172.21.15.60:0/1487019302 >> 172.21.15.60:0/1487019302 conn(0x7fd1580cf570 msgr2=0x7fd1580cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 -- 172.21.15.60:0/1487019302 shutdown_connections 2023-10-16T12:47:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.051+0000 7fd15c8d1640 1 -- 172.21.15.60:0/1487019302 wait complete. 2023-10-16T12:47:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.053+0000 7fd15c8d1640 1 Processor -- start 2023-10-16T12:47:15.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.053+0000 7fd15c8d1640 1 -- start start 2023-10-16T12:47:15.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd15c8d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580b8480 0x7fd1581f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd15c8d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1580d0be0 0x7fd1581f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd15c8d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 0x7fd158155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd15c8d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580cb520 con 0x7fd15815cd70 2023-10-16T12:47:15.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd15c8d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580cac10 con 0x7fd1580d0be0 2023-10-16T12:47:15.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd15c8d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1581eff90 con 0x7fd1580b8480 2023-10-16T12:47:15.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd157fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 0x7fd158155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd157fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 0x7fd158155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45386/0 (socket says 172.21.15.60:45386) 2023-10-16T12:47:15.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd157fff640 1 -- 172.21.15.60:0/567604604 learned_addr learned my addr 172.21.15.60:0/567604604 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:15.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd14ffff640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1580d0be0 0x7fd1581f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.054+0000 7fd1577fe640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580b8480 0x7fd1581f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd157fff640 1 -- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580b8480 msgr2=0x7fd1581f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:15.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd157fff640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580b8480 0x7fd1581f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd157fff640 1 -- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1580d0be0 msgr2=0x7fd1581f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:15.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd157fff640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1580d0be0 0x7fd1581f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd157fff640 1 -- 172.21.15.60:0/567604604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400094d0 con 0x7fd15815cd70 2023-10-16T12:47:15.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd157fff640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 0x7fd158155210 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd150007d90 tx=0x7fd15000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:15.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150014420 con 0x7fd15815cd70 2023-10-16T12:47:15.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1500145a0 con 0x7fd15815cd70 2023-10-16T12:47:15.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd158155a60 con 0x7fd15815cd70 2023-10-16T12:47:15.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150014950 con 0x7fd15815cd70 2023-10-16T12:47:15.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.055+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1580cb1b0 con 0x7fd15815cd70 2023-10-16T12:47:15.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.057+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1580cb690 con 0x7fd15815cd70 2023-10-16T12:47:15.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.058+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd150014ab0 con 0x7fd15815cd70 2023-10-16T12:47:15.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.059+0000 7fd154ff9640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fd13806af90 0x7fd13806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:15.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.059+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd15008a130 con 0x7fd15815cd70 2023-10-16T12:47:15.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.060+0000 7fd1577fe640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fd13806af90 0x7fd13806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:15.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.061+0000 7fd1577fe640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fd13806af90 0x7fd13806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd148006fd0 tx=0x7fd148008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:15.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.065+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd150055090 con 0x7fd15815cd70 2023-10-16T12:47:15.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.296+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd15804e9b0 con 0x7fd15815cd70 2023-10-16T12:47:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.840+0000 7fd154ff9640 1 -- 172.21.15.60:0/567604604 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd1500586b0 con 0x7fd15815cd70 2023-10-16T12:47:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 6 and data pool 7 2023-10-16T12:47:15.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fd13806af90 msgr2=0x7fd13806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:15.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fd13806af90 0x7fd13806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd148006fd0 tx=0x7fd148008040 comp rx=0 tx=0).stop 2023-10-16T12:47:15.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 msgr2=0x7fd158155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:15.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 0x7fd158155210 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd150007d90 tx=0x7fd15000e4b0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 shutdown_connections 2023-10-16T12:47:15.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fd13806af90 0x7fd13806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd15815cd70 0x7fd158155210 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1580d0be0 0x7fd1581f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.844+0000 7fd15c8d1640 1 --2- 172.21.15.60:0/567604604 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1580b8480 0x7fd1581f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:15.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.846+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 >> 172.21.15.60:0/567604604 conn(0x7fd1580cf570 msgr2=0x7fd1580d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:15.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.849+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 shutdown_connections 2023-10-16T12:47:15.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:15.849+0000 7fd15c8d1640 1 -- 172.21.15.60:0/567604604 wait complete. 2023-10-16T12:47:15.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:47:16.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.078+0000 7f123a7c0640 1 Processor -- start 2023-10-16T12:47:16.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.078+0000 7f123a7c0640 1 -- start start 2023-10-16T12:47:16.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.079+0000 7f123a7c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f123415aaa0 0x7f123415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f123a7c0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 0x7f12340cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f123a7c0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123415ed60 con 0x7f12340ccaa0 2023-10-16T12:47:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f123a7c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123406b810 con 0x7f123415aaa0 2023-10-16T12:47:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f123a7c0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340707e0 con 0x7f123415ae80 2023-10-16T12:47:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f1238d5d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f12340ccaa0 0x7f1234070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40666/0 (socket says 172.21.15.60:40666) 2023-10-16T12:47:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f1238d5d640 1 -- 172.21.15.60:0/914786926 learned_addr learned my addr 172.21.15.60:0/914786926 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f1233fff640 1 --2- 172.21.15.60:0/914786926 >> v2:172.21.15.144:3300/0 conn(0x7f123415aaa0 0x7f123415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f122bfff640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 0x7f12340cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.080+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201495363 0 0) 0x7f123415ed60 con 0x7f12340ccaa0 2023-10-16T12:47:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1220003520 con 0x7f12340ccaa0 2023-10-16T12:47:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2308286667 0 0) 0x7f1220003520 con 0x7f12340ccaa0 2023-10-16T12:47:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 msgr2=0x7f12340cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 0x7f12340cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 >> v2:172.21.15.144:3300/0 conn(0x7f123415aaa0 msgr2=0x7f123415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 --2- 172.21.15.60:0/914786926 >> v2:172.21.15.144:3300/0 conn(0x7f123415aaa0 0x7f123415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123415ef90 con 0x7f12340ccaa0 2023-10-16T12:47:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f122c0019f0 con 0x7f12340ccaa0 2023-10-16T12:47:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 >> v1:172.21.15.60:6789/0 conn(0x7f12340ccaa0 legacy=0x7f1234070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1220003970 0x7f1220003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 0x7f12200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1220004d00 0x7f1220005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123406b810 con 0x7f1220004d00 2023-10-16T12:47:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12340707e0 con 0x7f1220003970 2023-10-16T12:47:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122000b980 con 0x7f123415ae80 2023-10-16T12:47:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.081+0000 7f1238d5d640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1220004d00 0x7f1220005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.082+0000 7f1233fff640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1220003970 0x7f1220003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.082+0000 7f1238d5d640 1 -- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 msgr2=0x7f12200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.082+0000 7f1238d5d640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f123415ae80 0x7f12200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.082+0000 7f1238d5d640 1 -- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1220003970 msgr2=0x7f1220003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.082+0000 7f1238d5d640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1220003970 0x7f1220003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.082+0000 7f1238d5d640 1 -- 172.21.15.60:0/914786926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220003520 con 0x7f1220004d00 2023-10-16T12:47:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.083+0000 7f1238d5d640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1220004d00 0x7f1220005180 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f122c00a890 tx=0x7f12200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce16990aa2dc006b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.083+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c034630 con 0x7f1220004d00 2023-10-16T12:47:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.083+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c02f070 con 0x7f1220004d00 2023-10-16T12:47:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.083+0000 7f12337fe640 1 -- 172.21.15.60:0/914786926 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c0349f0 con 0x7f1220004d00 2023-10-16T12:47:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f123a7c0640 1 -- 172.21.15.60:0/914786926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1220004d00 msgr2=0x7f1220005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f123a7c0640 1 --2- 172.21.15.60:0/914786926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1220004d00 0x7f1220005180 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f122c00a890 tx=0x7f12200057f0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f1238d5d640 1 -- 172.21.15.60:0/914786926 reap_dead start 2023-10-16T12:47:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f123a7c0640 1 -- 172.21.15.60:0/914786926 shutdown_connections 2023-10-16T12:47:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f123a7c0640 1 -- 172.21.15.60:0/914786926 >> 172.21.15.60:0/914786926 conn(0x7f12340cb410 msgr2=0x7f12340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f123a7c0640 1 -- 172.21.15.60:0/914786926 shutdown_connections 2023-10-16T12:47:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.084+0000 7f123a7c0640 1 -- 172.21.15.60:0/914786926 wait complete. 2023-10-16T12:47:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.086+0000 7f123a7c0640 1 Processor -- start 2023-10-16T12:47:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.086+0000 7f123a7c0640 1 -- start start 2023-10-16T12:47:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f123a7c0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1220004d00 0x7f12340c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f123a7c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12340c24a0 0x7f12341590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f123a7c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 0x7f12340af730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f123a7c0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12340afc70 con 0x7f12340af280 2023-10-16T12:47:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f123a7c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12340c3500 con 0x7f12340c24a0 2023-10-16T12:47:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f123a7c0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340c2980 con 0x7f1220004d00 2023-10-16T12:47:16.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f1238d5d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 0x7f12340af730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f1238d5d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 0x7f12340af730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45396/0 (socket says 172.21.15.60:45396) 2023-10-16T12:47:16.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.087+0000 7f1238d5d640 1 -- 172.21.15.60:0/3275666041 learned_addr learned my addr 172.21.15.60:0/3275666041 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:16.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1233fff640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1220004d00 0x7f12340c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f122bfff640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12340c24a0 0x7f12341590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1238d5d640 1 -- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1220004d00 msgr2=0x7f12340c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1238d5d640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1220004d00 0x7f12340c3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1238d5d640 1 -- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12340c24a0 msgr2=0x7f12341590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1238d5d640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12340c24a0 0x7f12341590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1238d5d640 1 -- 172.21.15.60:0/3275666041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340c2c60 con 0x7f12340af280 2023-10-16T12:47:16.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f1238d5d640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 0x7f12340af730 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f122c00a890 tx=0x7f122c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:16.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c034840 con 0x7f12340af280 2023-10-16T12:47:16.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c0349a0 con 0x7f12340af280 2023-10-16T12:47:16.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.088+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12340c2f40 con 0x7f12340af280 2023-10-16T12:47:16.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.089+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c034d10 con 0x7f12340af280 2023-10-16T12:47:16.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.089+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12340c41a0 con 0x7f12340af280 2023-10-16T12:47:16.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.090+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123415b2f0 con 0x7f12340af280 2023-10-16T12:47:16.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.091+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f122c031030 con 0x7f12340af280 2023-10-16T12:47:16.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.092+0000 7f12317fa640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f121406af90 0x7f121406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.092+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f122c0aba70 con 0x7f12340af280 2023-10-16T12:47:16.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.092+0000 7f1233fff640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f121406af90 0x7f121406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.093+0000 7f1233fff640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f121406af90 0x7f121406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1224009a90 tx=0x7f1224008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:16.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.096+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f123415b2f0 con 0x7f12340af280 2023-10-16T12:47:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.290+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f123415b2f0 con 0x7f12340af280 2023-10-16T12:47:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.292+0000 7f12317fa640 1 -- 172.21.15.60:0/3275666041 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16514 (secure 0 0 0) 0x7f123415b2f0 con 0x7f12340af280 2023-10-16T12:47:16.294 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:16.294 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":66,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:47:15.297588+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:47:16.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.294+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f121406af90 msgr2=0x7f121406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.294+0000 7f123a7c0640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f121406af90 0x7f121406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1224009a90 tx=0x7f1224008040 comp rx=0 tx=0).stop 2023-10-16T12:47:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.294+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 msgr2=0x7f12340af730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.294+0000 7f123a7c0640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 0x7f12340af730 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f122c00a890 tx=0x7f122c001d80 comp rx=0 tx=0).stop 2023-10-16T12:47:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 shutdown_connections 2023-10-16T12:47:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f121406af90 0x7f121406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12340af280 0x7f12340af730 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12340c24a0 0x7f12341590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 --2- 172.21.15.60:0/3275666041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1220004d00 0x7f12340c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 >> 172.21.15.60:0/3275666041 conn(0x7f12340cb410 msgr2=0x7f12340b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:16.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 shutdown_connections 2023-10-16T12:47:16.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.295+0000 7f123a7c0640 1 -- 172.21.15.60:0/3275666041 wait complete. 2023-10-16T12:47:16.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:47:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.533+0000 7f008b59e640 1 Processor -- start 2023-10-16T12:47:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.534+0000 7f008b59e640 1 -- start start 2023-10-16T12:47:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.535+0000 7f008b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f008c0b8480 0x7f008c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.535+0000 7f008b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0d0be0 0x7f008c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.535+0000 7f008b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008c15d150 con 0x7f008c15cd70 2023-10-16T12:47:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.535+0000 7f008b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c0d1180 con 0x7f008c0b8480 2023-10-16T12:47:16.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.535+0000 7f008b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0d1460 con 0x7f008c0d0be0 2023-10-16T12:47:16.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f008a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f008c15cd70 0x7f008c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40674/0 (socket says 172.21.15.60:40674) 2023-10-16T12:47:16.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f008a59c640 1 -- 172.21.15.60:0/4028982613 learned_addr learned my addr 172.21.15.60:0/4028982613 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:16.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f0089d9b640 1 --2- 172.21.15.60:0/4028982613 >> v2:172.21.15.60:3300/0 conn(0x7f008c0b8480 0x7f008c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f008959a640 1 -- 172.21.15.60:0/4028982613 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514767220 0 0) 0x7f008c15d150 con 0x7f008c15cd70 2023-10-16T12:47:16.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f008959a640 1 -- 172.21.15.60:0/4028982613 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078003520 con 0x7f008c15cd70 2023-10-16T12:47:16.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f008ad9d640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0d0be0 0x7f008c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f0089d9b640 1 -- 172.21.15.60:0/4028982613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0d0be0 msgr2=0x7f008c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:16.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f0089d9b640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0d0be0 0x7f008c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f0089d9b640 1 -- 172.21.15.60:0/4028982613 >> v1:172.21.15.60:6789/0 conn(0x7f008c15cd70 legacy=0x7f008c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f0089d9b640 1 -- 172.21.15.60:0/4028982613 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0d1740 con 0x7f008c0b8480 2023-10-16T12:47:16.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.536+0000 7f008ad9d640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0d0be0 0x7f008c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:16.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.537+0000 7f0089d9b640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c0b8480 0x7f008c0cd7c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f00740097a0 tx=0x7f007402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f76429e7c7bc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:16.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008959a640 1 -- 172.21.15.60:0/4028982613 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007402d040 con 0x7f008c0b8480 2023-10-16T12:47:16.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008959a640 1 -- 172.21.15.60:0/4028982613 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0074031810 con 0x7f008c0b8480 2023-10-16T12:47:16.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008959a640 1 -- 172.21.15.60:0/4028982613 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0074031b80 con 0x7f008c0b8480 2023-10-16T12:47:16.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 -- 172.21.15.60:0/4028982613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c0b8480 msgr2=0x7f008c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c0b8480 0x7f008c0cd7c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f00740097a0 tx=0x7f007402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 -- 172.21.15.60:0/4028982613 shutdown_connections 2023-10-16T12:47:16.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0d0be0 0x7f008c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 --2- 172.21.15.60:0/4028982613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c0b8480 0x7f008c0cd7c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 -- 172.21.15.60:0/4028982613 >> 172.21.15.60:0/4028982613 conn(0x7f008c0cf570 msgr2=0x7f008c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:16.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 -- 172.21.15.60:0/4028982613 shutdown_connections 2023-10-16T12:47:16.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.538+0000 7f008b59e640 1 -- 172.21.15.60:0/4028982613 wait complete. 2023-10-16T12:47:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.540+0000 7f008b59e640 1 Processor -- start 2023-10-16T12:47:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.540+0000 7f008b59e640 1 -- start start 2023-10-16T12:47:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0b8480 0x7f008c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008c0d0be0 0x7f008c14d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 0x7f008c0b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1f5fc0 con 0x7f008c15cd70 2023-10-16T12:47:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0ccd10 con 0x7f008c0d0be0 2023-10-16T12:47:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0ca100 con 0x7f008c0b8480 2023-10-16T12:47:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 0x7f008c0b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 0x7f008c0b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45416/0 (socket says 172.21.15.60:45416) 2023-10-16T12:47:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.541+0000 7f008ad9d640 1 -- 172.21.15.60:0/219887993 learned_addr learned my addr 172.21.15.60:0/219887993 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008ad9d640 1 -- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0b8480 msgr2=0x7f008c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008ad9d640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0b8480 0x7f008c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f0089d9b640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008c0d0be0 0x7f008c14d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008ad9d640 1 -- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008c0d0be0 msgr2=0x7f008c14d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008ad9d640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008c0d0be0 0x7f008c14d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008ad9d640 1 -- 172.21.15.60:0/219887993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0074009400 con 0x7f008c15cd70 2023-10-16T12:47:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008ad9d640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 0x7f008c0b4120 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f008400c640 tx=0x7f008400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084011800 con 0x7f008c15cd70 2023-10-16T12:47:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084011960 con 0x7f008c15cd70 2023-10-16T12:47:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084011d10 con 0x7f008c15cd70 2023-10-16T12:47:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c15f360 con 0x7f008c15cd70 2023-10-16T12:47:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.542+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c1f5900 con 0x7f008c15cd70 2023-10-16T12:47:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.543+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c1f5760 con 0x7f008c15cd70 2023-10-16T12:47:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.545+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0084007370 con 0x7f008c15cd70 2023-10-16T12:47:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.545+0000 7f0082ffd640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f006c06af90 0x7f006c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.546+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00840899c0 con 0x7f008c15cd70 2023-10-16T12:47:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.546+0000 7f008a59c640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f006c06af90 0x7f006c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:16.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.547+0000 7f008a59c640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f006c06af90 0x7f006c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f007c0012e0 tx=0x7f007c001650 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:16.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.550+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f008c1f5760 con 0x7f008c15cd70 2023-10-16T12:47:16.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.739+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f008c1f5760 con 0x7f008c15cd70 2023-10-16T12:47:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.742+0000 7f0082ffd640 1 -- 172.21.15.60:0/219887993 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16514 (secure 0 0 0) 0x7f008c1f5760 con 0x7f008c15cd70 2023-10-16T12:47:16.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:16.745 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":66,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:47:15.297588+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:47:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f006c06af90 msgr2=0x7f006c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f006c06af90 0x7f006c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f007c0012e0 tx=0x7f007c001650 comp rx=0 tx=0).stop 2023-10-16T12:47:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 msgr2=0x7f008c0b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 0x7f008c0b4120 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f008400c640 tx=0x7f008400cb00 comp rx=0 tx=0).stop 2023-10-16T12:47:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 shutdown_connections 2023-10-16T12:47:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f006c06af90 0x7f006c06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f008c15cd70 0x7f008c0b4120 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008c0d0be0 0x7f008c14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 --2- 172.21.15.60:0/219887993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f008c0b8480 0x7f008c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:16.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 >> 172.21.15.60:0/219887993 conn(0x7f008c0cf570 msgr2=0x7f008c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:16.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 shutdown_connections 2023-10-16T12:47:16.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.745+0000 7f008b59e640 1 -- 172.21.15.60:0/219887993 wait complete. 2023-10-16T12:47:16.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:47:16.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.997+0000 7f99dbdc9640 1 Processor -- start 2023-10-16T12:47:16.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.998+0000 7f99dbdc9640 1 -- start start 2023-10-16T12:47:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.999+0000 7f99dbdc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.999+0000 7f99dbdc9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f99d415d410 0x7f99d4161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.999+0000 7f99dbdc9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99d4161990 0x7f99d40cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.999+0000 7f99dbdc9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d40d2b50 con 0x7f99d415d410 2023-10-16T12:47:17.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.999+0000 7f99dbdc9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d40b5d90 con 0x7f99d4161990 2023-10-16T12:47:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:16.999+0000 7f99dbdc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4160fc0 con 0x7f99d415d030 2023-10-16T12:47:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f99d415d410 0x7f99d4161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f99d415d410 0x7f99d4161450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45430/0 (socket says 172.21.15.60:45430) 2023-10-16T12:47:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 -- 172.21.15.60:0/571552868 learned_addr learned my addr 172.21.15.60:0/571552868 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 -- 172.21.15.60:0/571552868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 msgr2=0x7f99d40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 --2- 172.21.15.60:0/571552868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 -- 172.21.15.60:0/571552868 >> v2:172.21.15.144:3300/0 conn(0x7f99d4161990 msgr2=0x7f99d40cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 --2- 172.21.15.60:0/571552868 >> v2:172.21.15.144:3300/0 conn(0x7f99d4161990 0x7f99d40cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.000+0000 7f99d9364640 1 -- 172.21.15.60:0/571552868 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40d25d0 con 0x7f99d415d410 2023-10-16T12:47:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.001+0000 7f99d9364640 1 --2- 172.21.15.60:0/571552868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4161450 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f99d4073850 tx=0x7f99c402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9d974dfffe07e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.001+0000 7f99d8b63640 1 -- 172.21.15.60:0/571552868 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4030850 con 0x7f99d415d410 2023-10-16T12:47:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99d8b63640 1 -- 172.21.15.60:0/571552868 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c40309b0 con 0x7f99d415d410 2023-10-16T12:47:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99d8b63640 1 -- 172.21.15.60:0/571552868 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4030d40 con 0x7f99d415d410 2023-10-16T12:47:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 -- 172.21.15.60:0/571552868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 msgr2=0x7f99d4161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/571552868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4161450 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f99d4073850 tx=0x7f99c402b990 comp rx=0 tx=0).stop 2023-10-16T12:47:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 -- 172.21.15.60:0/571552868 shutdown_connections 2023-10-16T12:47:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/571552868 >> v2:172.21.15.144:3300/0 conn(0x7f99d4161990 0x7f99d40cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/571552868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4161450 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/571552868 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.002+0000 7f99dbdc9640 1 -- 172.21.15.60:0/571552868 >> 172.21.15.60:0/571552868 conn(0x7f99d40cf720 msgr2=0x7f99d40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:17.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.003+0000 7f99dbdc9640 1 -- 172.21.15.60:0/571552868 shutdown_connections 2023-10-16T12:47:17.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.003+0000 7f99dbdc9640 1 -- 172.21.15.60:0/571552868 wait complete. 2023-10-16T12:47:17.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.005+0000 7f99dbdc9640 1 Processor -- start 2023-10-16T12:47:17.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.005+0000 7f99dbdc9640 1 -- start start 2023-10-16T12:47:17.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99dbdc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99dbdc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99dbdc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d4161990 0x7f99d415b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99dbdc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d415e7f0 con 0x7f99d415d410 2023-10-16T12:47:17.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99dbdc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40ccb30 con 0x7f99d4161990 2023-10-16T12:47:17.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99dbdc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d415dc70 con 0x7f99d415d030 2023-10-16T12:47:17.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99d9364640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99d9364640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45438/0 (socket says 172.21.15.60:45438) 2023-10-16T12:47:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99d9364640 1 -- 172.21.15.60:0/330210203 learned_addr learned my addr 172.21.15.60:0/330210203 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99da366640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d4161990 0x7f99d415b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99d9b65640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99d9364640 1 -- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 msgr2=0x7f99d40cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.007+0000 7f99d9364640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99d9364640 1 -- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d4161990 msgr2=0x7f99d415b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99d9364640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d4161990 0x7f99d415b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99d9364640 1 -- 172.21.15.60:0/330210203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c40094d0 con 0x7f99d415d410 2023-10-16T12:47:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99da366640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d4161990 0x7f99d415b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99d9364640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4158250 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f99c4000c00 tx=0x7f99c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4039d40 con 0x7f99d415d410 2023-10-16T12:47:17.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c4043400 con 0x7f99d415d410 2023-10-16T12:47:17.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4039ea0 con 0x7f99d415d410 2023-10-16T12:47:17.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d415df50 con 0x7f99d415d410 2023-10-16T12:47:17.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.008+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d415e510 con 0x7f99d415d410 2023-10-16T12:47:17.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.009+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d40c67d0 con 0x7f99d415d410 2023-10-16T12:47:17.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.011+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99c402e070 con 0x7f99d415d410 2023-10-16T12:47:17.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.011+0000 7f99ca7fc640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f99b806af90 0x7f99b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:17.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.012+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99c403e070 con 0x7f99d415d410 2023-10-16T12:47:17.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.012+0000 7f99d9b65640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f99b806af90 0x7f99b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:17.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.013+0000 7f99d9b65640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f99b806af90 0x7f99b806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99d40cbe00 tx=0x7f99d0008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:17.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.016+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99d40c67d0 con 0x7f99d415d410 2023-10-16T12:47:17.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.241+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f99d40c67d0 con 0x7f99d415d410 2023-10-16T12:47:17.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.874+0000 7f99ca7fc640 1 -- 172.21.15.60:0/330210203 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f99d40c67d0 con 0x7f99d415d410 2023-10-16T12:47:17.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.878+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f99b806af90 msgr2=0x7f99b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:17.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.878+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f99b806af90 0x7f99b806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99d40cbe00 tx=0x7f99d0008040 comp rx=0 tx=0).stop 2023-10-16T12:47:17.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.878+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 msgr2=0x7f99d4158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:17.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.878+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4158250 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f99c4000c00 tx=0x7f99c4004730 comp rx=0 tx=0).stop 2023-10-16T12:47:17.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 shutdown_connections 2023-10-16T12:47:17.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f99b806af90 0x7f99b806d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d4161990 0x7f99d415b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99d415d410 0x7f99d4158250 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 --2- 172.21.15.60:0/330210203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99d415d030 0x7f99d40cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:17.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 >> 172.21.15.60:0/330210203 conn(0x7f99d40cf720 msgr2=0x7f99d40b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:17.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 shutdown_connections 2023-10-16T12:47:17.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:17.879+0000 7f99dbdc9640 1 -- 172.21.15.60:0/330210203 wait complete. 2023-10-16T12:47:17.899 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:47:17.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:47:18.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.125+0000 7fc2e7fff640 1 Processor -- start 2023-10-16T12:47:18.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.125+0000 7fc2e7fff640 1 -- start start 2023-10-16T12:47:18.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2e806c040 0x7fc2e806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:18.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2e80cda80 0x7fc2e815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:18.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e7fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e806c960 con 0x7fc2e80ad970 2023-10-16T12:47:18.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e7fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e815b9d0 con 0x7fc2e806c040 2023-10-16T12:47:18.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e7fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e815bcb0 con 0x7fc2e80cda80 2023-10-16T12:47:18.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2e806c040 0x7fc2e806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:18.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2e806c040 0x7fc2e806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45452/0 (socket says 172.21.15.60:45452) 2023-10-16T12:47:18.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/867998996 learned_addr learned my addr 172.21.15.60:0/867998996 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:18.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e67fc640 1 --1- 172.21.15.60:0/867998996 >> v1:172.21.15.60:6789/0 conn(0x7fc2e80ad970 0x7fc2e815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40676/0 (socket says 172.21.15.60:40676) 2023-10-16T12:47:18.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/867998996 >> v2:172.21.15.144:3300/0 conn(0x7fc2e80cda80 msgr2=0x7fc2e815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:18.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e77fe640 1 --2- 172.21.15.60:0/867998996 >> v2:172.21.15.144:3300/0 conn(0x7fc2e80cda80 0x7fc2e815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:18.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/867998996 >> v2:172.21.15.144:3300/0 conn(0x7fc2e80cda80 0x7fc2e815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.127+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/867998996 >> v1:172.21.15.60:6789/0 conn(0x7fc2e80ad970 legacy=0x7fc2e815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:18.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.128+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/867998996 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e815bf90 con 0x7fc2e806c040 2023-10-16T12:47:18.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.128+0000 7fc2e77fe640 1 --2- 172.21.15.60:0/867998996 >> v2:172.21.15.144:3300/0 conn(0x7fc2e80cda80 0x7fc2e815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:18.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.129+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/867998996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e806c040 0x7fc2e806c420 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80cf300 tx=0x7fc2d802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f21cec8acc248e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:18.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.129+0000 7fc2e5ffb640 1 -- 172.21.15.60:0/867998996 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8030870 con 0x7fc2e806c040 2023-10-16T12:47:18.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.129+0000 7fc2e5ffb640 1 -- 172.21.15.60:0/867998996 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d80309d0 con 0x7fc2e806c040 2023-10-16T12:47:18.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.129+0000 7fc2e5ffb640 1 -- 172.21.15.60:0/867998996 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8030d60 con 0x7fc2e806c040 2023-10-16T12:47:18.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.129+0000 7fc2e7fff640 1 -- 172.21.15.60:0/867998996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e806c040 msgr2=0x7fc2e806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:18.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.129+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/867998996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e806c040 0x7fc2e806c420 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80cf300 tx=0x7fc2d802b9b0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.130+0000 7fc2e7fff640 1 -- 172.21.15.60:0/867998996 shutdown_connections 2023-10-16T12:47:18.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.130+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/867998996 >> v2:172.21.15.144:3300/0 conn(0x7fc2e80cda80 0x7fc2e815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.130+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/867998996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e806c040 0x7fc2e806c420 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.130+0000 7fc2e7fff640 1 -- 172.21.15.60:0/867998996 >> 172.21.15.60:0/867998996 conn(0x7fc2e80ce6a0 msgr2=0x7fc2e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:18.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.130+0000 7fc2e7fff640 1 -- 172.21.15.60:0/867998996 shutdown_connections 2023-10-16T12:47:18.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.130+0000 7fc2e7fff640 1 -- 172.21.15.60:0/867998996 wait complete. 2023-10-16T12:47:18.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.131+0000 7fc2e7fff640 1 Processor -- start 2023-10-16T12:47:18.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.131+0000 7fc2e7fff640 1 -- start start 2023-10-16T12:47:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.132+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e806c040 0x7fc2e81f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.132+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e80ad970 0x7fc2e80af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.132+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 0x7fc2e81f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.132+0000 7fc2e7fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80b0300 con 0x7fc2e80cda80 2023-10-16T12:47:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.132+0000 7fc2e7fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80ae1a0 con 0x7fc2e80ad970 2023-10-16T12:47:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.132+0000 7fc2e7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80ae480 con 0x7fc2e806c040 2023-10-16T12:47:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 0x7fc2e81f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 0x7fc2e81f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45466/0 (socket says 172.21.15.60:45466) 2023-10-16T12:47:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 -- 172.21.15.60:0/3754463071 learned_addr learned my addr 172.21.15.60:0/3754463071 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e806c040 0x7fc2e81f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e67fc640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e80ad970 0x7fc2e80af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 -- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e806c040 msgr2=0x7fc2e81f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e806c040 0x7fc2e81f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 -- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e80ad970 msgr2=0x7fc2e80af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e80ad970 0x7fc2e80af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 -- 172.21.15.60:0/3754463071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d80094d0 con 0x7fc2e80cda80 2023-10-16T12:47:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e806c040 0x7fc2e81f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.133+0000 7fc2e77fe640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 0x7fc2e81f3540 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000cd50 tx=0x7fc2e000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.134+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000f680 con 0x7fc2e80cda80 2023-10-16T12:47:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.134+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000f800 con 0x7fc2e80cda80 2023-10-16T12:47:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.134+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000fbb0 con 0x7fc2e80cda80 2023-10-16T12:47:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.134+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e80ae760 con 0x7fc2e80cda80 2023-10-16T12:47:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.134+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e80b05e0 con 0x7fc2e80cda80 2023-10-16T12:47:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.135+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e806b330 con 0x7fc2e80cda80 2023-10-16T12:47:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.136+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc2e000b040 con 0x7fc2e80cda80 2023-10-16T12:47:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.137+0000 7fc2cf7fe640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc2d406af90 0x7fc2d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:18.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.137+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2e008a250 con 0x7fc2e80cda80 2023-10-16T12:47:18.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.138+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc2d406af90 0x7fc2d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:18.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.139+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc2d406af90 0x7fc2d406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8036000 tx=0x7fc2d803b040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:18.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.141+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc2e00550d0 con 0x7fc2e80cda80 2023-10-16T12:47:18.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.374+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc2e804e9b0 con 0x7fc2e80cda80 2023-10-16T12:47:18.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.877+0000 7fc2cf7fe640 1 -- 172.21.15.60:0/3754463071 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc2e00586f0 con 0x7fc2e80cda80 2023-10-16T12:47:18.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc2d406af90 msgr2=0x7fc2d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:18.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc2d406af90 0x7fc2d406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8036000 tx=0x7fc2d803b040 comp rx=0 tx=0).stop 2023-10-16T12:47:18.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 msgr2=0x7fc2e81f3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:18.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 0x7fc2e81f3540 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000cd50 tx=0x7fc2e000a650 comp rx=0 tx=0).stop 2023-10-16T12:47:18.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 shutdown_connections 2023-10-16T12:47:18.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fc2d406af90 0x7fc2d406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2e80cda80 0x7fc2e81f3540 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e80ad970 0x7fc2e80af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 --2- 172.21.15.60:0/3754463071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e806c040 0x7fc2e81f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 >> 172.21.15.60:0/3754463071 conn(0x7fc2e80ce6a0 msgr2=0x7fc2e806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 shutdown_connections 2023-10-16T12:47:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:18.881+0000 7fc2e7fff640 1 -- 172.21.15.60:0/3754463071 wait complete. 2023-10-16T12:47:18.904 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:47:18.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:47:19.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.124+0000 7f807010f640 1 Processor -- start 2023-10-16T12:47:19.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.124+0000 7f807010f640 1 -- start start 2023-10-16T12:47:19.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f807010f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8068070fd0 0x7f80680b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f807010f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8068071450 con 0x7f80680adf30 2023-10-16T12:47:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f807010f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80680cc600 con 0x7f80680adb50 2023-10-16T12:47:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f807010f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80680cc8e0 con 0x7f8068070fd0 2023-10-16T12:47:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f806e6ac640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f80680adf30 0x7f8068164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51168/0 (socket says 172.21.15.60:51168) 2023-10-16T12:47:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f806e6ac640 1 -- 172.21.15.60:0/2446383676 learned_addr learned my addr 172.21.15.60:0/2446383676 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.126+0000 7f806d6aa640 1 --2- 172.21.15.60:0/2446383676 >> v2:172.21.15.144:3300/0 conn(0x7f8068070fd0 0x7f80680b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.127+0000 7f806cea9640 1 -- 172.21.15.60:0/2446383676 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326937097 0 0) 0x7f8068071450 con 0x7f80680adf30 2023-10-16T12:47:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.127+0000 7f806cea9640 1 -- 172.21.15.60:0/2446383676 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8054003520 con 0x7f80680adf30 2023-10-16T12:47:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.127+0000 7f806cea9640 1 -- 172.21.15.60:0/2446383676 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178998201 0 0) 0x7f8054003520 con 0x7f80680adf30 2023-10-16T12:47:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.127+0000 7f806d6aa640 1 -- 172.21.15.60:0/2446383676 >> v1:172.21.15.144:6789/0 conn(0x7f80680adb50 legacy=0x7f806806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.127+0000 7f806d6aa640 1 -- 172.21.15.60:0/2446383676 >> v1:172.21.15.60:6789/0 conn(0x7f80680adf30 legacy=0x7f8068164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.127+0000 7f806d6aa640 1 -- 172.21.15.60:0/2446383676 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80680ccbc0 con 0x7f8068070fd0 2023-10-16T12:47:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.129+0000 7f806d6aa640 1 --2- 172.21.15.60:0/2446383676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8068070fd0 0x7f80680b0410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f80580097a0 tx=0x7f805802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=544ba41f92daecab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.129+0000 7f806cea9640 1 -- 172.21.15.60:0/2446383676 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805802f050 con 0x7f8068070fd0 2023-10-16T12:47:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.129+0000 7f806cea9640 1 -- 172.21.15.60:0/2446383676 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058031500 con 0x7f8068070fd0 2023-10-16T12:47:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.129+0000 7f806cea9640 1 -- 172.21.15.60:0/2446383676 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80600019f0 con 0x7f8068070fd0 2023-10-16T12:47:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 -- 172.21.15.60:0/2446383676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8068070fd0 msgr2=0x7f80680b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 --2- 172.21.15.60:0/2446383676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8068070fd0 0x7f80680b0410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f80580097a0 tx=0x7f805802b820 comp rx=0 tx=0).stop 2023-10-16T12:47:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 -- 172.21.15.60:0/2446383676 shutdown_connections 2023-10-16T12:47:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 --2- 172.21.15.60:0/2446383676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8068070fd0 0x7f80680b0410 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 -- 172.21.15.60:0/2446383676 >> 172.21.15.60:0/2446383676 conn(0x7f80680cb410 msgr2=0x7f80680cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 -- 172.21.15.60:0/2446383676 shutdown_connections 2023-10-16T12:47:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.130+0000 7f807010f640 1 -- 172.21.15.60:0/2446383676 wait complete. 2023-10-16T12:47:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.132+0000 7f807010f640 1 Processor -- start 2023-10-16T12:47:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.132+0000 7f807010f640 1 -- start start 2023-10-16T12:47:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f807010f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8068070fd0 0x7f806815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f807010f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80680adb50 0x7f8068163c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f807010f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 0x7f80680b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f807010f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80680c1500 con 0x7f80680adf30 2023-10-16T12:47:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f807010f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8068161a60 con 0x7f80680adb50 2023-10-16T12:47:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f807010f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80681fa3d0 con 0x7f8068070fd0 2023-10-16T12:47:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f806e6ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 0x7f80680b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f806e6ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 0x7f80680b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34094/0 (socket says 172.21.15.60:34094) 2023-10-16T12:47:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.133+0000 7f806e6ac640 1 -- 172.21.15.60:0/393027824 learned_addr learned my addr 172.21.15.60:0/393027824 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806deab640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8068070fd0 0x7f806815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806d6aa640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80680adb50 0x7f8068163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806e6ac640 1 -- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8068070fd0 msgr2=0x7f806815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806e6ac640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8068070fd0 0x7f806815f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806e6ac640 1 -- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80680adb50 msgr2=0x7f8068163c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806e6ac640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80680adb50 0x7f8068163c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806e6ac640 1 -- 172.21.15.60:0/393027824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8058009400 con 0x7f80680adf30 2023-10-16T12:47:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f806e6ac640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 0x7f80680b38d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8060001f70 tx=0x7f806000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060012400 con 0x7f80680adf30 2023-10-16T12:47:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8060012560 con 0x7f80680adf30 2023-10-16T12:47:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060012910 con 0x7f80680adf30 2023-10-16T12:47:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.134+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80681fa6b0 con 0x7f80680adf30 2023-10-16T12:47:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.135+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80681fac70 con 0x7f80680adf30 2023-10-16T12:47:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.136+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8068161c20 con 0x7f80680adf30 2023-10-16T12:47:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.137+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8060012a70 con 0x7f80680adf30 2023-10-16T12:47:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.138+0000 7f805e7fc640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f804c06af90 0x7f804c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.138+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f806008a070 con 0x7f80680adf30 2023-10-16T12:47:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.138+0000 7f806deab640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f804c06af90 0x7f804c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.139+0000 7f806deab640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f804c06af90 0x7f804c06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8064000f60 tx=0x7f8064009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.142+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f80600bcfe0 con 0x7f80680adf30 2023-10-16T12:47:19.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.366+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f806804e160 con 0x7f80680adf30 2023-10-16T12:47:19.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.368+0000 7f805e7fc640 1 -- 172.21.15.60:0/393027824 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f8060054f70 con 0x7f80680adf30 2023-10-16T12:47:19.369 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:19.370 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:47:15.298066+0000","modified":"2023-10-16T12:47:18.376045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/528818860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":528818860},{"type":"v1","addr":"172.21.15.144:6837","nonce":528818860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:47:19.370 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2023-10-16T12:47:19.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f804c06af90 msgr2=0x7f804c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f804c06af90 0x7f804c06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8064000f60 tx=0x7f8064009040 comp rx=0 tx=0).stop 2023-10-16T12:47:19.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 msgr2=0x7f80680b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 0x7f80680b38d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8060001f70 tx=0x7f806000a430 comp rx=0 tx=0).stop 2023-10-16T12:47:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 shutdown_connections 2023-10-16T12:47:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f804c06af90 0x7f804c06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80680adf30 0x7f80680b38d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80680adb50 0x7f8068163c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 --2- 172.21.15.60:0/393027824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8068070fd0 0x7f806815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.370+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 >> 172.21.15.60:0/393027824 conn(0x7f80680cb410 msgr2=0x7f80680ae4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.371+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 shutdown_connections 2023-10-16T12:47:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.371+0000 7f807010f640 1 -- 172.21.15.60:0/393027824 wait complete. 2023-10-16T12:47:19.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:47:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.607+0000 7ff236209640 1 Processor -- start 2023-10-16T12:47:19.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.607+0000 7ff236209640 1 -- start start 2023-10-16T12:47:19.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.608+0000 7ff236209640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015aaa0 0x7ff23015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff236209640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff23015ae80 0x7ff2300cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff236209640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2300ccaa0 0x7ff230070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff236209640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23006b810 con 0x7ff2300ccaa0 2023-10-16T12:47:19.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff236209640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23015ed60 con 0x7ff23015ae80 2023-10-16T12:47:19.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff236209640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300707e0 con 0x7ff23015aaa0 2023-10-16T12:47:19.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff22ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2300ccaa0 0x7ff230070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff22f7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015aaa0 0x7ff23015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff22ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2300ccaa0 0x7ff230070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34102/0 (socket says 172.21.15.60:34102) 2023-10-16T12:47:19.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff22ffff640 1 -- 172.21.15.60:0/4113452004 learned_addr learned my addr 172.21.15.60:0/4113452004 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.609+0000 7ff227fff640 1 --2- 172.21.15.60:0/4113452004 >> v2:172.21.15.144:3300/0 conn(0x7ff23015ae80 0x7ff2300cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.610+0000 7ff22ffff640 1 -- 172.21.15.60:0/4113452004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015aaa0 msgr2=0x7ff23015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.610+0000 7ff22ffff640 1 --2- 172.21.15.60:0/4113452004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015aaa0 0x7ff23015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.610+0000 7ff22ffff640 1 -- 172.21.15.60:0/4113452004 >> v2:172.21.15.144:3300/0 conn(0x7ff23015ae80 msgr2=0x7ff2300cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.610+0000 7ff22ffff640 1 --2- 172.21.15.60:0/4113452004 >> v2:172.21.15.144:3300/0 conn(0x7ff23015ae80 0x7ff2300cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.610+0000 7ff22ffff640 1 -- 172.21.15.60:0/4113452004 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23015ef90 con 0x7ff2300ccaa0 2023-10-16T12:47:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.611+0000 7ff22ffff640 1 --2- 172.21.15.60:0/4113452004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff230070f50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff2300cb270 tx=0x7ff22802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d82e615a3abcc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff22effd640 1 -- 172.21.15.60:0/4113452004 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22802ca70 con 0x7ff2300ccaa0 2023-10-16T12:47:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff22effd640 1 -- 172.21.15.60:0/4113452004 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22802cbd0 con 0x7ff2300ccaa0 2023-10-16T12:47:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff22effd640 1 -- 172.21.15.60:0/4113452004 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280343f0 con 0x7ff2300ccaa0 2023-10-16T12:47:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 -- 172.21.15.60:0/4113452004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 msgr2=0x7ff230070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 --2- 172.21.15.60:0/4113452004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff230070f50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff2300cb270 tx=0x7ff22802bed0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 -- 172.21.15.60:0/4113452004 shutdown_connections 2023-10-16T12:47:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 --2- 172.21.15.60:0/4113452004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff230070f50 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 --2- 172.21.15.60:0/4113452004 >> v2:172.21.15.144:3300/0 conn(0x7ff23015ae80 0x7ff2300cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 --2- 172.21.15.60:0/4113452004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015aaa0 0x7ff23015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 -- 172.21.15.60:0/4113452004 >> 172.21.15.60:0/4113452004 conn(0x7ff2300cb410 msgr2=0x7ff2300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.612+0000 7ff236209640 1 -- 172.21.15.60:0/4113452004 shutdown_connections 2023-10-16T12:47:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.613+0000 7ff236209640 1 -- 172.21.15.60:0/4113452004 wait complete. 2023-10-16T12:47:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.615+0000 7ff236209640 1 Processor -- start 2023-10-16T12:47:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.615+0000 7ff236209640 1 -- start start 2023-10-16T12:47:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff236209640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff2301562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff236209640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff23015aaa0 0x7ff23015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff236209640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015ae80 0x7ff23015d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff236209640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23015cc10 con 0x7ff2300ccaa0 2023-10-16T12:47:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff236209640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23015bc50 con 0x7ff23015aaa0 2023-10-16T12:47:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff236209640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23015bf30 con 0x7ff23015ae80 2023-10-16T12:47:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff22f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff2301562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff22f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff2301562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34106/0 (socket says 172.21.15.60:34106) 2023-10-16T12:47:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.616+0000 7ff22f7fe640 1 -- 172.21.15.60:0/1656531378 learned_addr learned my addr 172.21.15.60:0/1656531378 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22f7fe640 1 -- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015ae80 msgr2=0x7ff23015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22f7fe640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015ae80 0x7ff23015d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22f7fe640 1 -- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff23015aaa0 msgr2=0x7ff23015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22f7fe640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff23015aaa0 0x7ff23015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22f7fe640 1 -- 172.21.15.60:0/1656531378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228009b30 con 0x7ff2300ccaa0 2023-10-16T12:47:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22f7fe640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff2301562c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff22000e870 tx=0x7ff22000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:19.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000ee60 con 0x7ff2300ccaa0 2023-10-16T12:47:19.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22000c940 con 0x7ff2300ccaa0 2023-10-16T12:47:19.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23015c8a0 con 0x7ff2300ccaa0 2023-10-16T12:47:19.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.617+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000ccf0 con 0x7ff2300ccaa0 2023-10-16T12:47:19.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.618+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23015b860 con 0x7ff2300ccaa0 2023-10-16T12:47:19.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.619+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23015ca30 con 0x7ff2300ccaa0 2023-10-16T12:47:19.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.620+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff220013470 con 0x7ff2300ccaa0 2023-10-16T12:47:19.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.620+0000 7ff22cff9640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff21006af90 0x7ff21006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:19.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.621+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2200898b0 con 0x7ff2300ccaa0 2023-10-16T12:47:19.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.621+0000 7ff227fff640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff21006af90 0x7ff21006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:19.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.622+0000 7ff227fff640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff21006af90 0x7ff21006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff218006a90 tx=0x7ff218006a20 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:19.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.625+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2200547b0 con 0x7ff2300ccaa0 2023-10-16T12:47:19.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.815+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff23004e160 con 0x7ff2300ccaa0 2023-10-16T12:47:19.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.818+0000 7ff22cff9640 1 -- 172.21.15.60:0/1656531378 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16501 (secure 0 0 0) 0x7ff220021090 con 0x7ff2300ccaa0 2023-10-16T12:47:19.820 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:19.821 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":67,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:47:16.854824+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:47:19.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff21006af90 msgr2=0x7ff21006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff21006af90 0x7ff21006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff218006a90 tx=0x7ff218006a20 comp rx=0 tx=0).stop 2023-10-16T12:47:19.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 msgr2=0x7ff2301562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:19.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff2301562c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff22000e870 tx=0x7ff22000ed30 comp rx=0 tx=0).stop 2023-10-16T12:47:19.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 shutdown_connections 2023-10-16T12:47:19.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7ff21006af90 0x7ff21006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff23015ae80 0x7ff23015d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff23015aaa0 0x7ff23015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 --2- 172.21.15.60:0/1656531378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2300ccaa0 0x7ff2301562c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:19.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 >> 172.21.15.60:0/1656531378 conn(0x7ff2300cb410 msgr2=0x7ff2300ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:19.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 shutdown_connections 2023-10-16T12:47:19.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:19.821+0000 7ff236209640 1 -- 172.21.15.60:0/1656531378 wait complete. 2023-10-16T12:47:19.842 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:47:19.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:47:20.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.067+0000 7f11d5e32640 1 Processor -- start 2023-10-16T12:47:20.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.067+0000 7f11d5e32640 1 -- start start 2023-10-16T12:47:20.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.069+0000 7f11d5e32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 0x7f11d015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.069+0000 7f11d5e32640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11d015ae80 0x7f11d00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.069+0000 7f11d5e32640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d015ed60 con 0x7f11d00ccaa0 2023-10-16T12:47:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.069+0000 7f11d5e32640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d006b810 con 0x7f11d015ae80 2023-10-16T12:47:20.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.069+0000 7f11d5e32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00707e0 con 0x7f11d015aaa0 2023-10-16T12:47:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.070+0000 7f11cf7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 0x7f11d015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.070+0000 7f11c7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11d015ae80 0x7f11d00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.070+0000 7f11cf7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 0x7f11d015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:48044/0 (socket says 172.21.15.60:48044) 2023-10-16T12:47:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.070+0000 7f11cf7fe640 1 -- 172.21.15.60:0/3982122162 learned_addr learned my addr 172.21.15.60:0/3982122162 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.071+0000 7f11cf7fe640 1 -- 172.21.15.60:0/3982122162 >> v2:172.21.15.144:3300/0 conn(0x7f11d015ae80 msgr2=0x7f11d00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.071+0000 7f11cf7fe640 1 --2- 172.21.15.60:0/3982122162 >> v2:172.21.15.144:3300/0 conn(0x7f11d015ae80 0x7f11d00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.071+0000 7f11cf7fe640 1 -- 172.21.15.60:0/3982122162 >> v1:172.21.15.144:6789/0 conn(0x7f11d00ccaa0 legacy=0x7f11d0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.071+0000 7f11cf7fe640 1 -- 172.21.15.60:0/3982122162 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d015ef90 con 0x7f11d015aaa0 2023-10-16T12:47:20.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.071+0000 7f11c7fff640 1 --2- 172.21.15.60:0/3982122162 >> v2:172.21.15.144:3300/0 conn(0x7f11d015ae80 0x7f11d00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:47:20.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.072+0000 7f11cf7fe640 1 --2- 172.21.15.60:0/3982122162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 0x7f11d015e820 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009870 tx=0x7f11c002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e7731846cfe559 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:20.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11ceffd640 1 -- 172.21.15.60:0/3982122162 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0030840 con 0x7f11d015aaa0 2023-10-16T12:47:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11ceffd640 1 -- 172.21.15.60:0/3982122162 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c00309a0 con 0x7f11d015aaa0 2023-10-16T12:47:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11ceffd640 1 -- 172.21.15.60:0/3982122162 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0030d30 con 0x7f11d015aaa0 2023-10-16T12:47:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11d5e32640 1 -- 172.21.15.60:0/3982122162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 msgr2=0x7f11d015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11d5e32640 1 --2- 172.21.15.60:0/3982122162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 0x7f11d015e820 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009870 tx=0x7f11c002b980 comp rx=0 tx=0).stop 2023-10-16T12:47:20.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11d5e32640 1 -- 172.21.15.60:0/3982122162 shutdown_connections 2023-10-16T12:47:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11d5e32640 1 --2- 172.21.15.60:0/3982122162 >> v2:172.21.15.144:3300/0 conn(0x7f11d015ae80 0x7f11d00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11d5e32640 1 --2- 172.21.15.60:0/3982122162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015aaa0 0x7f11d015e820 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.073+0000 7f11d5e32640 1 -- 172.21.15.60:0/3982122162 >> 172.21.15.60:0/3982122162 conn(0x7f11d00cb410 msgr2=0x7f11d00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.074+0000 7f11d5e32640 1 -- 172.21.15.60:0/3982122162 shutdown_connections 2023-10-16T12:47:20.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.074+0000 7f11d5e32640 1 -- 172.21.15.60:0/3982122162 wait complete. 2023-10-16T12:47:20.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.075+0000 7f11d5e32640 1 Processor -- start 2023-10-16T12:47:20.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.075+0000 7f11d5e32640 1 -- start start 2023-10-16T12:47:20.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.076+0000 7f11d5e32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 0x7f11d00c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.076+0000 7f11d5e32640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11d015aaa0 0x7f11d0153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.076+0000 7f11d5e32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015ae80 0x7f11d0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.076+0000 7f11d5e32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00c47e0 con 0x7f11d00ccaa0 2023-10-16T12:47:20.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.076+0000 7f11d5e32640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00c4ac0 con 0x7f11d015aaa0 2023-10-16T12:47:20.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.076+0000 7f11d5e32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00c4da0 con 0x7f11d015ae80 2023-10-16T12:47:20.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 0x7f11d00c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 0x7f11d00c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34118/0 (socket says 172.21.15.60:34118) 2023-10-16T12:47:20.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 -- 172.21.15.60:0/2465970823 learned_addr learned my addr 172.21.15.60:0/2465970823 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:20.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11c7fff640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11d015aaa0 0x7f11d0153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cffff640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015ae80 0x7f11d0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 -- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015ae80 msgr2=0x7f11d0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015ae80 0x7f11d0156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 -- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11d015aaa0 msgr2=0x7f11d0153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11d015aaa0 0x7f11d0153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cf7fe640 1 -- 172.21.15.60:0/2465970823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b8009400 con 0x7f11d00ccaa0 2023-10-16T12:47:20.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.077+0000 7f11cffff640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015ae80 0x7f11d0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:47:20.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11c7fff640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11d015aaa0 0x7f11d0153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:47:20.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11cf7fe640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 0x7f11d00c3f00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f11c0000c00 tx=0x7f11c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:20.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0039c90 con 0x7f11d00ccaa0 2023-10-16T12:47:20.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c0039df0 con 0x7f11d00ccaa0 2023-10-16T12:47:20.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0043640 con 0x7f11d00ccaa0 2023-10-16T12:47:20.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c00094d0 con 0x7f11d00ccaa0 2023-10-16T12:47:20.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.078+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d015c140 con 0x7f11d00ccaa0 2023-10-16T12:47:20.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.080+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d004e160 con 0x7f11d00ccaa0 2023-10-16T12:47:20.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.081+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f11c00437a0 con 0x7f11d00ccaa0 2023-10-16T12:47:20.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.082+0000 7f11ccff9640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f11b406afe0 0x7f11b406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.082+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11c003e070 con 0x7f11d00ccaa0 2023-10-16T12:47:20.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.082+0000 7f11c7fff640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f11b406afe0 0x7f11b406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.084+0000 7f11c7fff640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f11b406afe0 0x7f11b406d4a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11d015a1d0 tx=0x7f11b8008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:20.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.086+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11c00776e0 con 0x7f11d00ccaa0 2023-10-16T12:47:20.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.286+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 --> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11d00c4c30 con 0x7f11b406afe0 2023-10-16T12:47:20.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.316+0000 7f11ccff9640 1 -- 172.21.15.60:0/2465970823 <== mgr.4910 v2:172.21.15.144:6832/3875614959 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286806 (secure 0 0 0) 0x7f11d00c4c30 con 0x7f11b406afe0 2023-10-16T12:47:20.322 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:20.324 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:47:20.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.321+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f11b406afe0 msgr2=0x7f11b406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.321+0000 7f11d5e32640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f11b406afe0 0x7f11b406d4a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11d015a1d0 tx=0x7f11b8008040 comp rx=0 tx=0).stop 2023-10-16T12:47:20.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.321+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 msgr2=0x7f11d00c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.321+0000 7f11d5e32640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 0x7f11d00c3f00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f11c0000c00 tx=0x7f11c0004730 comp rx=0 tx=0).stop 2023-10-16T12:47:20.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 shutdown_connections 2023-10-16T12:47:20.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f11b406afe0 0x7f11b406d4a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f11d015ae80 0x7f11d0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11d015aaa0 0x7f11d0153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 --2- 172.21.15.60:0/2465970823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11d00ccaa0 0x7f11d00c3f00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 >> 172.21.15.60:0/2465970823 conn(0x7f11d00cb410 msgr2=0x7f11d00b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:20.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 shutdown_connections 2023-10-16T12:47:20.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.322+0000 7f11d5e32640 1 -- 172.21.15.60:0/2465970823 wait complete. 2023-10-16T12:47:20.343 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T12:47:19.119971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326220,"kb_used_data":7384,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754648500,"statfs":{"total":773094113280,"available":772760064000,"internally_reserved":0,"allocated":7561216,"data_stored":4135388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":326356536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.307763"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715094+0000","last_change":"2023-10-16T12:47:15.881133+0000","last_active":"2023-10-16T12:47:17.715094+0000","last_peered":"2023-10-16T12:47:17.715094+0000","last_clean":"2023-10-16T12:47:17.715094+0000","last_became_active":"2023-10-16T12:47:15.880870+0000","last_became_peered":"2023-10-16T12:47:15.880870+0000","last_unstale":"2023-10-16T12:47:17.715094+0000","last_undegraded":"2023-10-16T12:47:17.715094+0000","last_fullsized":"2023-10-16T12:47:17.715094+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:15.398759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871776+0000","last_change":"2023-10-16T12:47:15.861659+0000","last_active":"2023-10-16T12:47:16.871776+0000","last_peered":"2023-10-16T12:47:16.871776+0000","last_clean":"2023-10-16T12:47:16.871776+0000","last_became_active":"2023-10-16T12:47:15.861348+0000","last_became_peered":"2023-10-16T12:47:15.861348+0000","last_unstale":"2023-10-16T12:47:16.871776+0000","last_undegraded":"2023-10-16T12:47:16.871776+0000","last_fullsized":"2023-10-16T12:47:16.871776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:32:27.290787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870806+0000","last_change":"2023-10-16T12:47:15.865388+0000","last_active":"2023-10-16T12:47:16.870806+0000","last_peered":"2023-10-16T12:47:16.870806+0000","last_clean":"2023-10-16T12:47:16.870806+0000","last_became_active":"2023-10-16T12:47:15.863227+0000","last_became_peered":"2023-10-16T12:47:15.863227+0000","last_unstale":"2023-10-16T12:47:16.870806+0000","last_undegraded":"2023-10-16T12:47:16.870806+0000","last_fullsized":"2023-10-16T12:47:16.870806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:12.908101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867919+0000","last_change":"2023-10-16T12:47:15.880187+0000","last_active":"2023-10-16T12:47:16.867919+0000","last_peered":"2023-10-16T12:47:16.867919+0000","last_clean":"2023-10-16T12:47:16.867919+0000","last_became_active":"2023-10-16T12:47:15.879944+0000","last_became_peered":"2023-10-16T12:47:15.879944+0000","last_unstale":"2023-10-16T12:47:16.867919+0000","last_undegraded":"2023-10-16T12:47:16.867919+0000","last_fullsized":"2023-10-16T12:47:16.867919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:38.432677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872665+0000","last_change":"2023-10-16T12:47:15.861615+0000","last_active":"2023-10-16T12:47:16.872665+0000","last_peered":"2023-10-16T12:47:16.872665+0000","last_clean":"2023-10-16T12:47:16.872665+0000","last_became_active":"2023-10-16T12:47:15.861304+0000","last_became_peered":"2023-10-16T12:47:15.861304+0000","last_unstale":"2023-10-16T12:47:16.872665+0000","last_undegraded":"2023-10-16T12:47:16.872665+0000","last_fullsized":"2023-10-16T12:47:16.872665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:36.363962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593177+0000","last_change":"2023-10-16T12:47:15.868036+0000","last_active":"2023-10-16T12:47:17.593177+0000","last_peered":"2023-10-16T12:47:17.593177+0000","last_clean":"2023-10-16T12:47:17.593177+0000","last_became_active":"2023-10-16T12:47:15.867649+0000","last_became_peered":"2023-10-16T12:47:15.867649+0000","last_unstale":"2023-10-16T12:47:17.593177+0000","last_undegraded":"2023-10-16T12:47:17.593177+0000","last_fullsized":"2023-10-16T12:47:17.593177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.301876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713742+0000","last_change":"2023-10-16T12:47:15.870827+0000","last_active":"2023-10-16T12:47:17.713742+0000","last_peered":"2023-10-16T12:47:17.713742+0000","last_clean":"2023-10-16T12:47:17.713742+0000","last_became_active":"2023-10-16T12:47:15.870354+0000","last_became_peered":"2023-10-16T12:47:15.870354+0000","last_unstale":"2023-10-16T12:47:17.713742+0000","last_undegraded":"2023-10-16T12:47:17.713742+0000","last_fullsized":"2023-10-16T12:47:17.713742+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:19.740976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866820+0000","last_change":"2023-10-16T12:47:15.883869+0000","last_active":"2023-10-16T12:47:16.866820+0000","last_peered":"2023-10-16T12:47:16.866820+0000","last_clean":"2023-10-16T12:47:16.866820+0000","last_became_active":"2023-10-16T12:47:15.882940+0000","last_became_peered":"2023-10-16T12:47:15.882940+0000","last_unstale":"2023-10-16T12:47:16.866820+0000","last_undegraded":"2023-10-16T12:47:16.866820+0000","last_fullsized":"2023-10-16T12:47:16.866820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:41.278440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868566+0000","last_change":"2023-10-16T12:47:15.865294+0000","last_active":"2023-10-16T12:47:16.868566+0000","last_peered":"2023-10-16T12:47:16.868566+0000","last_clean":"2023-10-16T12:47:16.868566+0000","last_became_active":"2023-10-16T12:47:15.864967+0000","last_became_peered":"2023-10-16T12:47:15.864967+0000","last_unstale":"2023-10-16T12:47:16.868566+0000","last_undegraded":"2023-10-16T12:47:16.868566+0000","last_fullsized":"2023-10-16T12:47:16.868566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:14.366218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868348+0000","last_change":"2023-10-16T12:47:15.881539+0000","last_active":"2023-10-16T12:47:16.868348+0000","last_peered":"2023-10-16T12:47:16.868348+0000","last_clean":"2023-10-16T12:47:16.868348+0000","last_became_active":"2023-10-16T12:47:15.881275+0000","last_became_peered":"2023-10-16T12:47:15.881275+0000","last_unstale":"2023-10-16T12:47:16.868348+0000","last_undegraded":"2023-10-16T12:47:16.868348+0000","last_fullsized":"2023-10-16T12:47:16.868348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:33.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868336+0000","last_change":"2023-10-16T12:47:15.864078+0000","last_active":"2023-10-16T12:47:16.868336+0000","last_peered":"2023-10-16T12:47:16.868336+0000","last_clean":"2023-10-16T12:47:16.868336+0000","last_became_active":"2023-10-16T12:47:15.863447+0000","last_became_peered":"2023-10-16T12:47:15.863447+0000","last_unstale":"2023-10-16T12:47:16.868336+0000","last_undegraded":"2023-10-16T12:47:16.868336+0000","last_fullsized":"2023-10-16T12:47:16.868336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:05.594442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872394+0000","last_change":"2023-10-16T12:47:15.866787+0000","last_active":"2023-10-16T12:47:16.872394+0000","last_peered":"2023-10-16T12:47:16.872394+0000","last_clean":"2023-10-16T12:47:16.872394+0000","last_became_active":"2023-10-16T12:47:15.866185+0000","last_became_peered":"2023-10-16T12:47:15.866185+0000","last_unstale":"2023-10-16T12:47:16.872394+0000","last_undegraded":"2023-10-16T12:47:16.872394+0000","last_fullsized":"2023-10-16T12:47:16.872394+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:35.132589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713395+0000","last_change":"2023-10-16T12:47:15.881464+0000","last_active":"2023-10-16T12:47:17.713395+0000","last_peered":"2023-10-16T12:47:17.713395+0000","last_clean":"2023-10-16T12:47:17.713395+0000","last_became_active":"2023-10-16T12:47:15.880203+0000","last_became_peered":"2023-10-16T12:47:15.880203+0000","last_unstale":"2023-10-16T12:47:17.713395+0000","last_undegraded":"2023-10-16T12:47:17.713395+0000","last_fullsized":"2023-10-16T12:47:17.713395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:02.183359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867947+0000","last_change":"2023-10-16T12:47:15.864127+0000","last_active":"2023-10-16T12:47:16.867947+0000","last_peered":"2023-10-16T12:47:16.867947+0000","last_clean":"2023-10-16T12:47:16.867947+0000","last_became_active":"2023-10-16T12:47:15.863430+0000","last_became_peered":"2023-10-16T12:47:15.863430+0000","last_unstale":"2023-10-16T12:47:16.867947+0000","last_undegraded":"2023-10-16T12:47:16.867947+0000","last_fullsized":"2023-10-16T12:47:16.867947+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:12.440413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593109+0000","last_change":"2023-10-16T12:47:15.862090+0000","last_active":"2023-10-16T12:47:17.593109+0000","last_peered":"2023-10-16T12:47:17.593109+0000","last_clean":"2023-10-16T12:47:17.593109+0000","last_became_active":"2023-10-16T12:47:15.861684+0000","last_became_peered":"2023-10-16T12:47:15.861684+0000","last_unstale":"2023-10-16T12:47:17.593109+0000","last_undegraded":"2023-10-16T12:47:17.593109+0000","last_fullsized":"2023-10-16T12:47:17.593109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:45.289233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713803+0000","last_change":"2023-10-16T12:47:15.869579+0000","last_active":"2023-10-16T12:47:17.713803+0000","last_peered":"2023-10-16T12:47:17.713803+0000","last_clean":"2023-10-16T12:47:17.713803+0000","last_became_active":"2023-10-16T12:47:15.869303+0000","last_became_peered":"2023-10-16T12:47:15.869303+0000","last_unstale":"2023-10-16T12:47:17.713803+0000","last_undegraded":"2023-10-16T12:47:17.713803+0000","last_fullsized":"2023-10-16T12:47:17.713803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:33:23.634134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871706+0000","last_change":"2023-10-16T12:47:15.868752+0000","last_active":"2023-10-16T12:47:16.871706+0000","last_peered":"2023-10-16T12:47:16.871706+0000","last_clean":"2023-10-16T12:47:16.871706+0000","last_became_active":"2023-10-16T12:47:15.868439+0000","last_became_peered":"2023-10-16T12:47:15.868439+0000","last_unstale":"2023-10-16T12:47:16.871706+0000","last_undegraded":"2023-10-16T12:47:16.871706+0000","last_fullsized":"2023-10-16T12:47:16.871706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:29.974892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871274+0000","last_change":"2023-10-16T12:47:15.864974+0000","last_active":"2023-10-16T12:47:16.871274+0000","last_peered":"2023-10-16T12:47:16.871274+0000","last_clean":"2023-10-16T12:47:16.871274+0000","last_became_active":"2023-10-16T12:47:15.864681+0000","last_became_peered":"2023-10-16T12:47:15.864681+0000","last_unstale":"2023-10-16T12:47:16.871274+0000","last_undegraded":"2023-10-16T12:47:16.871274+0000","last_fullsized":"2023-10-16T12:47:16.871274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:51.718249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716488+0000","last_change":"2023-10-16T12:47:15.884654+0000","last_active":"2023-10-16T12:47:17.716488+0000","last_peered":"2023-10-16T12:47:17.716488+0000","last_clean":"2023-10-16T12:47:17.716488+0000","last_became_active":"2023-10-16T12:47:15.882247+0000","last_became_peered":"2023-10-16T12:47:15.882247+0000","last_unstale":"2023-10-16T12:47:17.716488+0000","last_undegraded":"2023-10-16T12:47:17.716488+0000","last_fullsized":"2023-10-16T12:47:17.716488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:33.605094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713707+0000","last_change":"2023-10-16T12:47:15.870082+0000","last_active":"2023-10-16T12:47:17.713707+0000","last_peered":"2023-10-16T12:47:17.713707+0000","last_clean":"2023-10-16T12:47:17.713707+0000","last_became_active":"2023-10-16T12:47:15.869773+0000","last_became_peered":"2023-10-16T12:47:15.869773+0000","last_unstale":"2023-10-16T12:47:17.713707+0000","last_undegraded":"2023-10-16T12:47:17.713707+0000","last_fullsized":"2023-10-16T12:47:17.713707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:01.337793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593346+0000","last_change":"2023-10-16T12:47:15.879746+0000","last_active":"2023-10-16T12:47:17.593346+0000","last_peered":"2023-10-16T12:47:17.593346+0000","last_clean":"2023-10-16T12:47:17.593346+0000","last_became_active":"2023-10-16T12:47:15.879426+0000","last_became_peered":"2023-10-16T12:47:15.879426+0000","last_unstale":"2023-10-16T12:47:17.593346+0000","last_undegraded":"2023-10-16T12:47:17.593346+0000","last_fullsized":"2023-10-16T12:47:17.593346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:11:48.086170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868157+0000","last_change":"2023-10-16T12:47:15.886162+0000","last_active":"2023-10-16T12:47:16.868157+0000","last_peered":"2023-10-16T12:47:16.868157+0000","last_clean":"2023-10-16T12:47:16.868157+0000","last_became_active":"2023-10-16T12:47:15.885430+0000","last_became_peered":"2023-10-16T12:47:15.885430+0000","last_unstale":"2023-10-16T12:47:16.868157+0000","last_undegraded":"2023-10-16T12:47:16.868157+0000","last_fullsized":"2023-10-16T12:47:16.868157+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:57.147140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714575+0000","last_change":"2023-10-16T12:47:15.882268+0000","last_active":"2023-10-16T12:47:17.714575+0000","last_peered":"2023-10-16T12:47:17.714575+0000","last_clean":"2023-10-16T12:47:17.714575+0000","last_became_active":"2023-10-16T12:47:15.882003+0000","last_became_peered":"2023-10-16T12:47:15.882003+0000","last_unstale":"2023-10-16T12:47:17.714575+0000","last_undegraded":"2023-10-16T12:47:17.714575+0000","last_fullsized":"2023-10-16T12:47:17.714575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:31.877461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714145+0000","last_change":"2023-10-16T12:47:15.873951+0000","last_active":"2023-10-16T12:47:17.714145+0000","last_peered":"2023-10-16T12:47:17.714145+0000","last_clean":"2023-10-16T12:47:17.714145+0000","last_became_active":"2023-10-16T12:47:15.873598+0000","last_became_peered":"2023-10-16T12:47:15.873598+0000","last_unstale":"2023-10-16T12:47:17.714145+0000","last_undegraded":"2023-10-16T12:47:17.714145+0000","last_fullsized":"2023-10-16T12:47:17.714145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:37.236166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716460+0000","last_change":"2023-10-16T12:47:15.882085+0000","last_active":"2023-10-16T12:47:17.716460+0000","last_peered":"2023-10-16T12:47:17.716460+0000","last_clean":"2023-10-16T12:47:17.716460+0000","last_became_active":"2023-10-16T12:47:15.879599+0000","last_became_peered":"2023-10-16T12:47:15.879599+0000","last_unstale":"2023-10-16T12:47:17.716460+0000","last_undegraded":"2023-10-16T12:47:17.716460+0000","last_fullsized":"2023-10-16T12:47:17.716460+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:24.788118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713082+0000","last_change":"2023-10-16T12:47:15.879603+0000","last_active":"2023-10-16T12:47:17.713082+0000","last_peered":"2023-10-16T12:47:17.713082+0000","last_clean":"2023-10-16T12:47:17.713082+0000","last_became_active":"2023-10-16T12:47:15.879310+0000","last_became_peered":"2023-10-16T12:47:15.879310+0000","last_unstale":"2023-10-16T12:47:17.713082+0000","last_undegraded":"2023-10-16T12:47:17.713082+0000","last_fullsized":"2023-10-16T12:47:17.713082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:06.479762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870884+0000","last_change":"2023-10-16T12:47:15.859873+0000","last_active":"2023-10-16T12:47:16.870884+0000","last_peered":"2023-10-16T12:47:16.870884+0000","last_clean":"2023-10-16T12:47:16.870884+0000","last_became_active":"2023-10-16T12:47:15.859474+0000","last_became_peered":"2023-10-16T12:47:15.859474+0000","last_unstale":"2023-10-16T12:47:16.870884+0000","last_undegraded":"2023-10-16T12:47:16.870884+0000","last_fullsized":"2023-10-16T12:47:16.870884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:55.824293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872727+0000","last_change":"2023-10-16T12:47:15.863582+0000","last_active":"2023-10-16T12:47:16.872727+0000","last_peered":"2023-10-16T12:47:16.872727+0000","last_clean":"2023-10-16T12:47:16.872727+0000","last_became_active":"2023-10-16T12:47:15.863163+0000","last_became_peered":"2023-10-16T12:47:15.863163+0000","last_unstale":"2023-10-16T12:47:16.872727+0000","last_undegraded":"2023-10-16T12:47:16.872727+0000","last_fullsized":"2023-10-16T12:47:16.872727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:22.063939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715391+0000","last_change":"2023-10-16T12:47:15.878832+0000","last_active":"2023-10-16T12:47:17.715391+0000","last_peered":"2023-10-16T12:47:17.715391+0000","last_clean":"2023-10-16T12:47:17.715391+0000","last_became_active":"2023-10-16T12:47:15.878163+0000","last_became_peered":"2023-10-16T12:47:15.878163+0000","last_unstale":"2023-10-16T12:47:17.715391+0000","last_undegraded":"2023-10-16T12:47:17.715391+0000","last_fullsized":"2023-10-16T12:47:17.715391+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:56.063314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870488+0000","last_change":"2023-10-16T12:47:15.870913+0000","last_active":"2023-10-16T12:47:16.870488+0000","last_peered":"2023-10-16T12:47:16.870488+0000","last_clean":"2023-10-16T12:47:16.870488+0000","last_became_active":"2023-10-16T12:47:15.870642+0000","last_became_peered":"2023-10-16T12:47:15.870642+0000","last_unstale":"2023-10-16T12:47:16.870488+0000","last_undegraded":"2023-10-16T12:47:16.870488+0000","last_fullsized":"2023-10-16T12:47:16.870488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:10.311108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593364+0000","last_change":"2023-10-16T12:47:15.861433+0000","last_active":"2023-10-16T12:47:17.593364+0000","last_peered":"2023-10-16T12:47:17.593364+0000","last_clean":"2023-10-16T12:47:17.593364+0000","last_became_active":"2023-10-16T12:47:15.861138+0000","last_became_peered":"2023-10-16T12:47:15.861138+0000","last_unstale":"2023-10-16T12:47:17.593364+0000","last_undegraded":"2023-10-16T12:47:17.593364+0000","last_fullsized":"2023-10-16T12:47:17.593364+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:49.013743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.712684+0000","last_change":"2023-10-16T12:47:15.873792+0000","last_active":"2023-10-16T12:47:17.712684+0000","last_peered":"2023-10-16T12:47:17.712684+0000","last_clean":"2023-10-16T12:47:17.712684+0000","last_became_active":"2023-10-16T12:47:15.873525+0000","last_became_peered":"2023-10-16T12:47:15.873525+0000","last_unstale":"2023-10-16T12:47:17.712684+0000","last_undegraded":"2023-10-16T12:47:17.712684+0000","last_fullsized":"2023-10-16T12:47:17.712684+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:25:23.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711852+0000","last_change":"2023-10-16T12:47:15.871937+0000","last_active":"2023-10-16T12:47:17.711852+0000","last_peered":"2023-10-16T12:47:17.711852+0000","last_clean":"2023-10-16T12:47:17.711852+0000","last_became_active":"2023-10-16T12:47:15.871676+0000","last_became_peered":"2023-10-16T12:47:15.871676+0000","last_unstale":"2023-10-16T12:47:17.711852+0000","last_undegraded":"2023-10-16T12:47:17.711852+0000","last_fullsized":"2023-10-16T12:47:17.711852+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:13.892096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"66'4","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594352+0000","last_change":"2023-10-16T12:47:14.853041+0000","last_active":"2023-10-16T12:47:17.594352+0000","last_peered":"2023-10-16T12:47:17.594352+0000","last_clean":"2023-10-16T12:47:17.594352+0000","last_became_active":"2023-10-16T12:47:14.852693+0000","last_became_peered":"2023-10-16T12:47:14.852693+0000","last_unstale":"2023-10-16T12:47:17.594352+0000","last_undegraded":"2023-10-16T12:47:17.594352+0000","last_fullsized":"2023-10-16T12:47:17.594352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:06.845364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716321+0000","last_change":"2023-10-16T12:47:15.879305+0000","last_active":"2023-10-16T12:47:17.716321+0000","last_peered":"2023-10-16T12:47:17.716321+0000","last_clean":"2023-10-16T12:47:17.716321+0000","last_became_active":"2023-10-16T12:47:15.878006+0000","last_became_peered":"2023-10-16T12:47:15.878006+0000","last_unstale":"2023-10-16T12:47:17.716321+0000","last_undegraded":"2023-10-16T12:47:17.716321+0000","last_fullsized":"2023-10-16T12:47:17.716321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:59.361855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871488+0000","last_change":"2023-10-16T12:47:14.851685+0000","last_active":"2023-10-16T12:47:16.871488+0000","last_peered":"2023-10-16T12:47:16.871488+0000","last_clean":"2023-10-16T12:47:16.871488+0000","last_became_active":"2023-10-16T12:47:14.851200+0000","last_became_peered":"2023-10-16T12:47:14.851200+0000","last_unstale":"2023-10-16T12:47:16.871488+0000","last_undegraded":"2023-10-16T12:47:16.871488+0000","last_fullsized":"2023-10-16T12:47:16.871488+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:26.245432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.873040+0000","last_change":"2023-10-16T12:47:15.864451+0000","last_active":"2023-10-16T12:47:16.873040+0000","last_peered":"2023-10-16T12:47:16.873040+0000","last_clean":"2023-10-16T12:47:16.873040+0000","last_became_active":"2023-10-16T12:47:15.863994+0000","last_became_peered":"2023-10-16T12:47:15.863994+0000","last_unstale":"2023-10-16T12:47:16.873040+0000","last_undegraded":"2023-10-16T12:47:16.873040+0000","last_fullsized":"2023-10-16T12:47:16.873040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:48.556188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711673+0000","last_change":"2023-10-16T12:47:14.845807+0000","last_active":"2023-10-16T12:47:17.711673+0000","last_peered":"2023-10-16T12:47:17.711673+0000","last_clean":"2023-10-16T12:47:17.711673+0000","last_became_active":"2023-10-16T12:47:14.845540+0000","last_became_peered":"2023-10-16T12:47:14.845540+0000","last_unstale":"2023-10-16T12:47:17.711673+0000","last_undegraded":"2023-10-16T12:47:17.711673+0000","last_fullsized":"2023-10-16T12:47:17.711673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:42.162801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711386+0000","last_change":"2023-10-16T12:47:15.879550+0000","last_active":"2023-10-16T12:47:17.711386+0000","last_peered":"2023-10-16T12:47:17.711386+0000","last_clean":"2023-10-16T12:47:17.711386+0000","last_became_active":"2023-10-16T12:47:15.879302+0000","last_became_peered":"2023-10-16T12:47:15.879302+0000","last_unstale":"2023-10-16T12:47:17.711386+0000","last_undegraded":"2023-10-16T12:47:17.711386+0000","last_fullsized":"2023-10-16T12:47:17.711386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:34.490300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714479+0000","last_change":"2023-10-16T12:47:14.853830+0000","last_active":"2023-10-16T12:47:17.714479+0000","last_peered":"2023-10-16T12:47:17.714479+0000","last_clean":"2023-10-16T12:47:17.714479+0000","last_became_active":"2023-10-16T12:47:14.851699+0000","last_became_peered":"2023-10-16T12:47:14.851699+0000","last_unstale":"2023-10-16T12:47:17.714479+0000","last_undegraded":"2023-10-16T12:47:17.714479+0000","last_fullsized":"2023-10-16T12:47:17.714479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:29.957872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870942+0000","last_change":"2023-10-16T12:47:15.882158+0000","last_active":"2023-10-16T12:47:16.870942+0000","last_peered":"2023-10-16T12:47:16.870942+0000","last_clean":"2023-10-16T12:47:16.870942+0000","last_became_active":"2023-10-16T12:47:15.881845+0000","last_became_peered":"2023-10-16T12:47:15.881845+0000","last_unstale":"2023-10-16T12:47:16.870942+0000","last_undegraded":"2023-10-16T12:47:16.870942+0000","last_fullsized":"2023-10-16T12:47:16.870942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:49.068292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593645+0000","last_change":"2023-10-16T12:47:14.851242+0000","last_active":"2023-10-16T12:47:17.593645+0000","last_peered":"2023-10-16T12:47:17.593645+0000","last_clean":"2023-10-16T12:47:17.593645+0000","last_became_active":"2023-10-16T12:47:14.850300+0000","last_became_peered":"2023-10-16T12:47:14.850300+0000","last_unstale":"2023-10-16T12:47:17.593645+0000","last_undegraded":"2023-10-16T12:47:17.593645+0000","last_fullsized":"2023-10-16T12:47:17.593645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:16.119849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867400+0000","last_change":"2023-10-16T12:47:15.859602+0000","last_active":"2023-10-16T12:47:16.867400+0000","last_peered":"2023-10-16T12:47:16.867400+0000","last_clean":"2023-10-16T12:47:16.867400+0000","last_became_active":"2023-10-16T12:47:15.858830+0000","last_became_peered":"2023-10-16T12:47:15.858830+0000","last_unstale":"2023-10-16T12:47:16.867400+0000","last_undegraded":"2023-10-16T12:47:16.867400+0000","last_fullsized":"2023-10-16T12:47:16.867400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:30.780039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871186+0000","last_change":"2023-10-16T12:47:14.843389+0000","last_active":"2023-10-16T12:47:16.871186+0000","last_peered":"2023-10-16T12:47:16.871186+0000","last_clean":"2023-10-16T12:47:16.871186+0000","last_became_active":"2023-10-16T12:47:14.843105+0000","last_became_peered":"2023-10-16T12:47:14.843105+0000","last_unstale":"2023-10-16T12:47:16.871186+0000","last_undegraded":"2023-10-16T12:47:16.871186+0000","last_fullsized":"2023-10-16T12:47:16.871186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:59.053243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866739+0000","last_change":"2023-10-16T12:47:15.883910+0000","last_active":"2023-10-16T12:47:16.866739+0000","last_peered":"2023-10-16T12:47:16.866739+0000","last_clean":"2023-10-16T12:47:16.866739+0000","last_became_active":"2023-10-16T12:47:15.883534+0000","last_became_peered":"2023-10-16T12:47:15.883534+0000","last_unstale":"2023-10-16T12:47:16.866739+0000","last_undegraded":"2023-10-16T12:47:16.866739+0000","last_fullsized":"2023-10-16T12:47:16.866739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:19.126746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871069+0000","last_change":"2023-10-16T12:47:14.840171+0000","last_active":"2023-10-16T12:47:16.871069+0000","last_peered":"2023-10-16T12:47:16.871069+0000","last_clean":"2023-10-16T12:47:16.871069+0000","last_became_active":"2023-10-16T12:47:14.838988+0000","last_became_peered":"2023-10-16T12:47:14.838988+0000","last_unstale":"2023-10-16T12:47:16.871069+0000","last_undegraded":"2023-10-16T12:47:16.871069+0000","last_fullsized":"2023-10-16T12:47:16.871069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:47.673081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714327+0000","last_change":"2023-10-16T12:47:15.882954+0000","last_active":"2023-10-16T12:47:17.714327+0000","last_peered":"2023-10-16T12:47:17.714327+0000","last_clean":"2023-10-16T12:47:17.714327+0000","last_became_active":"2023-10-16T12:47:15.882442+0000","last_became_peered":"2023-10-16T12:47:15.882442+0000","last_unstale":"2023-10-16T12:47:17.714327+0000","last_undegraded":"2023-10-16T12:47:17.714327+0000","last_fullsized":"2023-10-16T12:47:17.714327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:47.633166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866866+0000","last_change":"2023-10-16T12:47:14.838460+0000","last_active":"2023-10-16T12:47:16.866866+0000","last_peered":"2023-10-16T12:47:16.866866+0000","last_clean":"2023-10-16T12:47:16.866866+0000","last_became_active":"2023-10-16T12:47:14.837900+0000","last_became_peered":"2023-10-16T12:47:14.837900+0000","last_unstale":"2023-10-16T12:47:16.866866+0000","last_undegraded":"2023-10-16T12:47:16.866866+0000","last_fullsized":"2023-10-16T12:47:16.866866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:15.271899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871488+0000","last_change":"2023-10-16T12:47:15.864917+0000","last_active":"2023-10-16T12:47:16.871488+0000","last_peered":"2023-10-16T12:47:16.871488+0000","last_clean":"2023-10-16T12:47:16.871488+0000","last_became_active":"2023-10-16T12:47:15.864619+0000","last_became_peered":"2023-10-16T12:47:15.864619+0000","last_unstale":"2023-10-16T12:47:16.871488+0000","last_undegraded":"2023-10-16T12:47:16.871488+0000","last_fullsized":"2023-10-16T12:47:16.871488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:07.313911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872135+0000","last_change":"2023-10-16T12:47:14.840571+0000","last_active":"2023-10-16T12:47:16.872135+0000","last_peered":"2023-10-16T12:47:16.872135+0000","last_clean":"2023-10-16T12:47:16.872135+0000","last_became_active":"2023-10-16T12:47:14.840253+0000","last_became_peered":"2023-10-16T12:47:14.840253+0000","last_unstale":"2023-10-16T12:47:16.872135+0000","last_undegraded":"2023-10-16T12:47:16.872135+0000","last_fullsized":"2023-10-16T12:47:16.872135+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:23.868424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715202+0000","last_change":"2023-10-16T12:47:15.871092+0000","last_active":"2023-10-16T12:47:17.715202+0000","last_peered":"2023-10-16T12:47:17.715202+0000","last_clean":"2023-10-16T12:47:17.715202+0000","last_became_active":"2023-10-16T12:47:15.870802+0000","last_became_peered":"2023-10-16T12:47:15.870802+0000","last_unstale":"2023-10-16T12:47:17.715202+0000","last_undegraded":"2023-10-16T12:47:17.715202+0000","last_fullsized":"2023-10-16T12:47:17.715202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:22.260815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868432+0000","last_change":"2023-10-16T12:47:14.852327+0000","last_active":"2023-10-16T12:47:16.868432+0000","last_peered":"2023-10-16T12:47:16.868432+0000","last_clean":"2023-10-16T12:47:16.868432+0000","last_became_active":"2023-10-16T12:47:14.852013+0000","last_became_peered":"2023-10-16T12:47:14.852013+0000","last_unstale":"2023-10-16T12:47:16.868432+0000","last_undegraded":"2023-10-16T12:47:16.868432+0000","last_fullsized":"2023-10-16T12:47:16.868432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:44:28.187045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594409+0000","last_change":"2023-10-16T12:47:15.872942+0000","last_active":"2023-10-16T12:47:17.594409+0000","last_peered":"2023-10-16T12:47:17.594409+0000","last_clean":"2023-10-16T12:47:17.594409+0000","last_became_active":"2023-10-16T12:47:15.871624+0000","last_became_peered":"2023-10-16T12:47:15.871624+0000","last_unstale":"2023-10-16T12:47:17.594409+0000","last_undegraded":"2023-10-16T12:47:17.594409+0000","last_fullsized":"2023-10-16T12:47:17.594409+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:19.740976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"66'3","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714519+0000","last_change":"2023-10-16T12:47:14.850845+0000","last_active":"2023-10-16T12:47:17.714519+0000","last_peered":"2023-10-16T12:47:17.714519+0000","last_clean":"2023-10-16T12:47:17.714519+0000","last_became_active":"2023-10-16T12:47:14.850568+0000","last_became_peered":"2023-10-16T12:47:14.850568+0000","last_unstale":"2023-10-16T12:47:17.714519+0000","last_undegraded":"2023-10-16T12:47:17.714519+0000","last_fullsized":"2023-10-16T12:47:17.714519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:19.735554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713967+0000","last_change":"2023-10-16T12:47:15.875474+0000","last_active":"2023-10-16T12:47:17.713967+0000","last_peered":"2023-10-16T12:47:17.713967+0000","last_clean":"2023-10-16T12:47:17.713967+0000","last_became_active":"2023-10-16T12:47:15.875135+0000","last_became_peered":"2023-10-16T12:47:15.875135+0000","last_unstale":"2023-10-16T12:47:17.713967+0000","last_undegraded":"2023-10-16T12:47:17.713967+0000","last_fullsized":"2023-10-16T12:47:17.713967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:10.491685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"66'6","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713645+0000","last_change":"2023-10-16T12:47:14.838723+0000","last_active":"2023-10-16T12:47:17.713645+0000","last_peered":"2023-10-16T12:47:17.713645+0000","last_clean":"2023-10-16T12:47:17.713645+0000","last_became_active":"2023-10-16T12:47:14.838443+0000","last_became_peered":"2023-10-16T12:47:14.838443+0000","last_unstale":"2023-10-16T12:47:17.713645+0000","last_undegraded":"2023-10-16T12:47:17.713645+0000","last_fullsized":"2023-10-16T12:47:17.713645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:54.044963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714710+0000","last_change":"2023-10-16T12:47:15.877442+0000","last_active":"2023-10-16T12:47:17.714710+0000","last_peered":"2023-10-16T12:47:17.714710+0000","last_clean":"2023-10-16T12:47:17.714710+0000","last_became_active":"2023-10-16T12:47:15.877148+0000","last_became_peered":"2023-10-16T12:47:15.877148+0000","last_unstale":"2023-10-16T12:47:17.714710+0000","last_undegraded":"2023-10-16T12:47:17.714710+0000","last_fullsized":"2023-10-16T12:47:17.714710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:38.259521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593582+0000","last_change":"2023-10-16T12:47:14.847585+0000","last_active":"2023-10-16T12:47:17.593582+0000","last_peered":"2023-10-16T12:47:17.593582+0000","last_clean":"2023-10-16T12:47:17.593582+0000","last_became_active":"2023-10-16T12:47:14.847271+0000","last_became_peered":"2023-10-16T12:47:14.847271+0000","last_unstale":"2023-10-16T12:47:17.593582+0000","last_undegraded":"2023-10-16T12:47:17.593582+0000","last_fullsized":"2023-10-16T12:47:17.593582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:23.377778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867564+0000","last_change":"2023-10-16T12:47:15.872695+0000","last_active":"2023-10-16T12:47:16.867564+0000","last_peered":"2023-10-16T12:47:16.867564+0000","last_clean":"2023-10-16T12:47:16.867564+0000","last_became_active":"2023-10-16T12:47:15.872433+0000","last_became_peered":"2023-10-16T12:47:15.872433+0000","last_unstale":"2023-10-16T12:47:16.867564+0000","last_undegraded":"2023-10-16T12:47:16.867564+0000","last_fullsized":"2023-10-16T12:47:16.867564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:06.040135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713852+0000","last_change":"2023-10-16T12:47:14.847528+0000","last_active":"2023-10-16T12:47:17.713852+0000","last_peered":"2023-10-16T12:47:17.713852+0000","last_clean":"2023-10-16T12:47:17.713852+0000","last_became_active":"2023-10-16T12:47:14.847258+0000","last_became_peered":"2023-10-16T12:47:14.847258+0000","last_unstale":"2023-10-16T12:47:17.713852+0000","last_undegraded":"2023-10-16T12:47:17.713852+0000","last_fullsized":"2023-10-16T12:47:17.713852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:47.998720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593109+0000","last_change":"2023-10-16T12:47:15.868946+0000","last_active":"2023-10-16T12:47:17.593109+0000","last_peered":"2023-10-16T12:47:17.593109+0000","last_clean":"2023-10-16T12:47:17.593109+0000","last_became_active":"2023-10-16T12:47:15.868683+0000","last_became_peered":"2023-10-16T12:47:15.868683+0000","last_unstale":"2023-10-16T12:47:17.593109+0000","last_undegraded":"2023-10-16T12:47:17.593109+0000","last_fullsized":"2023-10-16T12:47:17.593109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:04.492293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593367+0000","last_change":"2023-10-16T12:47:14.847569+0000","last_active":"2023-10-16T12:47:17.593367+0000","last_peered":"2023-10-16T12:47:17.593367+0000","last_clean":"2023-10-16T12:47:17.593367+0000","last_became_active":"2023-10-16T12:47:14.846019+0000","last_became_peered":"2023-10-16T12:47:14.846019+0000","last_unstale":"2023-10-16T12:47:17.593367+0000","last_undegraded":"2023-10-16T12:47:17.593367+0000","last_fullsized":"2023-10-16T12:47:17.593367+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.541165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714101+0000","last_change":"2023-10-16T12:47:15.880147+0000","last_active":"2023-10-16T12:47:17.714101+0000","last_peered":"2023-10-16T12:47:17.714101+0000","last_clean":"2023-10-16T12:47:17.714101+0000","last_became_active":"2023-10-16T12:47:15.879359+0000","last_became_peered":"2023-10-16T12:47:15.879359+0000","last_unstale":"2023-10-16T12:47:17.714101+0000","last_undegraded":"2023-10-16T12:47:17.714101+0000","last_fullsized":"2023-10-16T12:47:17.714101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:48.381657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"66'8","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867058+0000","last_change":"2023-10-16T12:47:14.845074+0000","last_active":"2023-10-16T12:47:16.867058+0000","last_peered":"2023-10-16T12:47:16.867058+0000","last_clean":"2023-10-16T12:47:16.867058+0000","last_became_active":"2023-10-16T12:47:14.844782+0000","last_became_peered":"2023-10-16T12:47:14.844782+0000","last_unstale":"2023-10-16T12:47:16.867058+0000","last_undegraded":"2023-10-16T12:47:16.867058+0000","last_fullsized":"2023-10-16T12:47:16.867058+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:22.162022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713205+0000","last_change":"2023-10-16T12:47:15.872442+0000","last_active":"2023-10-16T12:47:17.713205+0000","last_peered":"2023-10-16T12:47:17.713205+0000","last_clean":"2023-10-16T12:47:17.713205+0000","last_became_active":"2023-10-16T12:47:15.872131+0000","last_became_peered":"2023-10-16T12:47:15.872131+0000","last_unstale":"2023-10-16T12:47:17.713205+0000","last_undegraded":"2023-10-16T12:47:17.713205+0000","last_fullsized":"2023-10-16T12:47:17.713205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:03.173587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713710+0000","last_change":"2023-10-16T12:47:14.843998+0000","last_active":"2023-10-16T12:47:17.713710+0000","last_peered":"2023-10-16T12:47:17.713710+0000","last_clean":"2023-10-16T12:47:17.713710+0000","last_became_active":"2023-10-16T12:47:14.843708+0000","last_became_peered":"2023-10-16T12:47:14.843708+0000","last_unstale":"2023-10-16T12:47:17.713710+0000","last_undegraded":"2023-10-16T12:47:17.713710+0000","last_fullsized":"2023-10-16T12:47:17.713710+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:29.721083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868644+0000","last_change":"2023-10-16T12:47:15.865878+0000","last_active":"2023-10-16T12:47:16.868644+0000","last_peered":"2023-10-16T12:47:16.868644+0000","last_clean":"2023-10-16T12:47:16.868644+0000","last_became_active":"2023-10-16T12:47:15.865579+0000","last_became_peered":"2023-10-16T12:47:15.865579+0000","last_unstale":"2023-10-16T12:47:16.868644+0000","last_undegraded":"2023-10-16T12:47:16.868644+0000","last_fullsized":"2023-10-16T12:47:16.868644+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:05.787181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871613+0000","last_change":"2023-10-16T12:47:14.842060+0000","last_active":"2023-10-16T12:47:16.871613+0000","last_peered":"2023-10-16T12:47:16.871613+0000","last_clean":"2023-10-16T12:47:16.871613+0000","last_became_active":"2023-10-16T12:47:14.841745+0000","last_became_peered":"2023-10-16T12:47:14.841745+0000","last_unstale":"2023-10-16T12:47:16.871613+0000","last_undegraded":"2023-10-16T12:47:16.871613+0000","last_fullsized":"2023-10-16T12:47:16.871613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:12.877073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715530+0000","last_change":"2023-10-16T12:47:15.870462+0000","last_active":"2023-10-16T12:47:17.715530+0000","last_peered":"2023-10-16T12:47:17.715530+0000","last_clean":"2023-10-16T12:47:17.715530+0000","last_became_active":"2023-10-16T12:47:15.870163+0000","last_became_peered":"2023-10-16T12:47:15.870163+0000","last_unstale":"2023-10-16T12:47:17.715530+0000","last_undegraded":"2023-10-16T12:47:17.715530+0000","last_fullsized":"2023-10-16T12:47:17.715530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:33.095535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.873382+0000","last_change":"2023-10-16T12:47:14.835953+0000","last_active":"2023-10-16T12:47:16.873382+0000","last_peered":"2023-10-16T12:47:16.873382+0000","last_clean":"2023-10-16T12:47:16.873382+0000","last_became_active":"2023-10-16T12:47:14.835446+0000","last_became_peered":"2023-10-16T12:47:14.835446+0000","last_unstale":"2023-10-16T12:47:16.873382+0000","last_undegraded":"2023-10-16T12:47:16.873382+0000","last_fullsized":"2023-10-16T12:47:16.873382+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:18.877023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716183+0000","last_change":"2023-10-16T12:47:15.883125+0000","last_active":"2023-10-16T12:47:17.716183+0000","last_peered":"2023-10-16T12:47:17.716183+0000","last_clean":"2023-10-16T12:47:17.716183+0000","last_became_active":"2023-10-16T12:47:15.882713+0000","last_became_peered":"2023-10-16T12:47:15.882713+0000","last_unstale":"2023-10-16T12:47:17.716183+0000","last_undegraded":"2023-10-16T12:47:17.716183+0000","last_fullsized":"2023-10-16T12:47:17.716183+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:13.198864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594269+0000","last_change":"2023-10-16T12:47:14.851357+0000","last_active":"2023-10-16T12:47:17.594269+0000","last_peered":"2023-10-16T12:47:17.594269+0000","last_clean":"2023-10-16T12:47:17.594269+0000","last_became_active":"2023-10-16T12:47:14.850881+0000","last_became_peered":"2023-10-16T12:47:14.850881+0000","last_unstale":"2023-10-16T12:47:17.594269+0000","last_undegraded":"2023-10-16T12:47:17.594269+0000","last_fullsized":"2023-10-16T12:47:17.594269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:45.719271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593310+0000","last_change":"2023-10-16T12:47:15.865546+0000","last_active":"2023-10-16T12:47:17.593310+0000","last_peered":"2023-10-16T12:47:17.593310+0000","last_clean":"2023-10-16T12:47:17.593310+0000","last_became_active":"2023-10-16T12:47:15.864473+0000","last_became_peered":"2023-10-16T12:47:15.864473+0000","last_unstale":"2023-10-16T12:47:17.593310+0000","last_undegraded":"2023-10-16T12:47:17.593310+0000","last_fullsized":"2023-10-16T12:47:17.593310+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:05.268814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594044+0000","last_change":"2023-10-16T12:47:14.851477+0000","last_active":"2023-10-16T12:47:17.594044+0000","last_peered":"2023-10-16T12:47:17.594044+0000","last_clean":"2023-10-16T12:47:17.594044+0000","last_became_active":"2023-10-16T12:47:14.849170+0000","last_became_peered":"2023-10-16T12:47:14.849170+0000","last_unstale":"2023-10-16T12:47:17.594044+0000","last_undegraded":"2023-10-16T12:47:17.594044+0000","last_fullsized":"2023-10-16T12:47:17.594044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:29.957872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711609+0000","last_change":"2023-10-16T12:47:15.860752+0000","last_active":"2023-10-16T12:47:17.711609+0000","last_peered":"2023-10-16T12:47:17.711609+0000","last_clean":"2023-10-16T12:47:17.711609+0000","last_became_active":"2023-10-16T12:47:15.860447+0000","last_became_peered":"2023-10-16T12:47:15.860447+0000","last_unstale":"2023-10-16T12:47:17.711609+0000","last_undegraded":"2023-10-16T12:47:17.711609+0000","last_fullsized":"2023-10-16T12:47:17.711609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:27.260455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870869+0000","last_change":"2023-10-16T12:47:14.850253+0000","last_active":"2023-10-16T12:47:16.870869+0000","last_peered":"2023-10-16T12:47:16.870869+0000","last_clean":"2023-10-16T12:47:16.870869+0000","last_became_active":"2023-10-16T12:47:14.849991+0000","last_became_peered":"2023-10-16T12:47:14.849991+0000","last_unstale":"2023-10-16T12:47:16.870869+0000","last_undegraded":"2023-10-16T12:47:16.870869+0000","last_fullsized":"2023-10-16T12:47:16.870869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:22.570270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.712716+0000","last_change":"2023-10-16T12:47:15.868019+0000","last_active":"2023-10-16T12:47:17.712716+0000","last_peered":"2023-10-16T12:47:17.712716+0000","last_clean":"2023-10-16T12:47:17.712716+0000","last_became_active":"2023-10-16T12:47:15.867168+0000","last_became_peered":"2023-10-16T12:47:15.867168+0000","last_unstale":"2023-10-16T12:47:17.712716+0000","last_undegraded":"2023-10-16T12:47:17.712716+0000","last_fullsized":"2023-10-16T12:47:17.712716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:51.003724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867233+0000","last_change":"2023-10-16T12:47:14.852006+0000","last_active":"2023-10-16T12:47:16.867233+0000","last_peered":"2023-10-16T12:47:16.867233+0000","last_clean":"2023-10-16T12:47:16.867233+0000","last_became_active":"2023-10-16T12:47:14.851187+0000","last_became_peered":"2023-10-16T12:47:14.851187+0000","last_unstale":"2023-10-16T12:47:16.867233+0000","last_undegraded":"2023-10-16T12:47:16.867233+0000","last_fullsized":"2023-10-16T12:47:16.867233+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:57.232158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866864+0000","last_change":"2023-10-16T12:47:15.870962+0000","last_active":"2023-10-16T12:47:16.866864+0000","last_peered":"2023-10-16T12:47:16.866864+0000","last_clean":"2023-10-16T12:47:16.866864+0000","last_became_active":"2023-10-16T12:47:15.870701+0000","last_became_peered":"2023-10-16T12:47:15.870701+0000","last_unstale":"2023-10-16T12:47:16.866864+0000","last_undegraded":"2023-10-16T12:47:16.866864+0000","last_fullsized":"2023-10-16T12:47:16.866864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:08.204380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713301+0000","last_change":"2023-10-16T12:47:14.846747+0000","last_active":"2023-10-16T12:47:17.713301+0000","last_peered":"2023-10-16T12:47:17.713301+0000","last_clean":"2023-10-16T12:47:17.713301+0000","last_became_active":"2023-10-16T12:47:14.846430+0000","last_became_peered":"2023-10-16T12:47:14.846430+0000","last_unstale":"2023-10-16T12:47:17.713301+0000","last_undegraded":"2023-10-16T12:47:17.713301+0000","last_fullsized":"2023-10-16T12:47:17.713301+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:06.845364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716510+0000","last_change":"2023-10-16T12:47:15.876390+0000","last_active":"2023-10-16T12:47:17.716510+0000","last_peered":"2023-10-16T12:47:17.716510+0000","last_clean":"2023-10-16T12:47:17.716510+0000","last_became_active":"2023-10-16T12:47:15.876040+0000","last_became_peered":"2023-10-16T12:47:15.876040+0000","last_unstale":"2023-10-16T12:47:17.716510+0000","last_undegraded":"2023-10-16T12:47:17.716510+0000","last_fullsized":"2023-10-16T12:47:17.716510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:43.059000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594182+0000","last_change":"2023-10-16T12:47:14.839347+0000","last_active":"2023-10-16T12:47:17.594182+0000","last_peered":"2023-10-16T12:47:17.594182+0000","last_clean":"2023-10-16T12:47:17.594182+0000","last_became_active":"2023-10-16T12:47:14.839073+0000","last_became_peered":"2023-10-16T12:47:14.839073+0000","last_unstale":"2023-10-16T12:47:17.594182+0000","last_undegraded":"2023-10-16T12:47:17.594182+0000","last_fullsized":"2023-10-16T12:47:17.594182+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:28.769276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868014+0000","last_change":"2023-10-16T12:47:15.858197+0000","last_active":"2023-10-16T12:47:16.868014+0000","last_peered":"2023-10-16T12:47:16.868014+0000","last_clean":"2023-10-16T12:47:16.868014+0000","last_became_active":"2023-10-16T12:47:15.857884+0000","last_became_peered":"2023-10-16T12:47:15.857884+0000","last_unstale":"2023-10-16T12:47:16.868014+0000","last_undegraded":"2023-10-16T12:47:16.868014+0000","last_fullsized":"2023-10-16T12:47:16.868014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:32.745085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715114+0000","last_change":"2023-10-16T12:47:14.855238+0000","last_active":"2023-10-16T12:47:17.715114+0000","last_peered":"2023-10-16T12:47:17.715114+0000","last_clean":"2023-10-16T12:47:17.715114+0000","last_became_active":"2023-10-16T12:47:14.854906+0000","last_became_peered":"2023-10-16T12:47:14.854906+0000","last_unstale":"2023-10-16T12:47:17.715114+0000","last_undegraded":"2023-10-16T12:47:17.715114+0000","last_fullsized":"2023-10-16T12:47:17.715114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:28.764239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714309+0000","last_change":"2023-10-16T12:47:15.873271+0000","last_active":"2023-10-16T12:47:17.714309+0000","last_peered":"2023-10-16T12:47:17.714309+0000","last_clean":"2023-10-16T12:47:17.714309+0000","last_became_active":"2023-10-16T12:47:15.873006+0000","last_became_peered":"2023-10-16T12:47:15.873006+0000","last_unstale":"2023-10-16T12:47:17.714309+0000","last_undegraded":"2023-10-16T12:47:17.714309+0000","last_fullsized":"2023-10-16T12:47:17.714309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:51:42.973193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711158+0000","last_change":"2023-10-16T12:47:14.845070+0000","last_active":"2023-10-16T12:47:17.711158+0000","last_peered":"2023-10-16T12:47:17.711158+0000","last_clean":"2023-10-16T12:47:17.711158+0000","last_became_active":"2023-10-16T12:47:14.844808+0000","last_became_peered":"2023-10-16T12:47:14.844808+0000","last_unstale":"2023-10-16T12:47:17.711158+0000","last_undegraded":"2023-10-16T12:47:17.711158+0000","last_fullsized":"2023-10-16T12:47:17.711158+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:40.081833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593588+0000","last_change":"2023-10-16T12:47:15.868310+0000","last_active":"2023-10-16T12:47:17.593588+0000","last_peered":"2023-10-16T12:47:17.593588+0000","last_clean":"2023-10-16T12:47:17.593588+0000","last_became_active":"2023-10-16T12:47:15.865995+0000","last_became_peered":"2023-10-16T12:47:15.865995+0000","last_unstale":"2023-10-16T12:47:17.593588+0000","last_undegraded":"2023-10-16T12:47:17.593588+0000","last_fullsized":"2023-10-16T12:47:17.593588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:29.779262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714217+0000","last_change":"2023-10-16T12:47:14.853707+0000","last_active":"2023-10-16T12:47:17.714217+0000","last_peered":"2023-10-16T12:47:17.714217+0000","last_clean":"2023-10-16T12:47:17.714217+0000","last_became_active":"2023-10-16T12:47:14.853328+0000","last_became_peered":"2023-10-16T12:47:14.853328+0000","last_unstale":"2023-10-16T12:47:17.714217+0000","last_undegraded":"2023-10-16T12:47:17.714217+0000","last_fullsized":"2023-10-16T12:47:17.714217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:03.477270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593411+0000","last_change":"2023-10-16T12:47:15.868092+0000","last_active":"2023-10-16T12:47:17.593411+0000","last_peered":"2023-10-16T12:47:17.593411+0000","last_clean":"2023-10-16T12:47:17.593411+0000","last_became_active":"2023-10-16T12:47:15.867676+0000","last_became_peered":"2023-10-16T12:47:15.867676+0000","last_unstale":"2023-10-16T12:47:17.593411+0000","last_undegraded":"2023-10-16T12:47:17.593411+0000","last_fullsized":"2023-10-16T12:47:17.593411+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:20.750577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871182+0000","last_change":"2023-10-16T12:47:14.842781+0000","last_active":"2023-10-16T12:47:16.871182+0000","last_peered":"2023-10-16T12:47:16.871182+0000","last_clean":"2023-10-16T12:47:16.871182+0000","last_became_active":"2023-10-16T12:47:14.840875+0000","last_became_peered":"2023-10-16T12:47:14.840875+0000","last_unstale":"2023-10-16T12:47:16.871182+0000","last_undegraded":"2023-10-16T12:47:16.871182+0000","last_fullsized":"2023-10-16T12:47:16.871182+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:42.162801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870576+0000","last_change":"2023-10-16T12:47:15.865170+0000","last_active":"2023-10-16T12:47:16.870576+0000","last_peered":"2023-10-16T12:47:16.870576+0000","last_clean":"2023-10-16T12:47:16.870576+0000","last_became_active":"2023-10-16T12:47:15.864892+0000","last_became_peered":"2023-10-16T12:47:15.864892+0000","last_unstale":"2023-10-16T12:47:16.870576+0000","last_undegraded":"2023-10-16T12:47:16.870576+0000","last_fullsized":"2023-10-16T12:47:16.870576+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:35.792307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711266+0000","last_change":"2023-10-16T12:47:14.856068+0000","last_active":"2023-10-16T12:47:17.711266+0000","last_peered":"2023-10-16T12:47:17.711266+0000","last_clean":"2023-10-16T12:47:17.711266+0000","last_became_active":"2023-10-16T12:47:14.855678+0000","last_became_peered":"2023-10-16T12:47:14.855678+0000","last_unstale":"2023-10-16T12:47:17.711266+0000","last_undegraded":"2023-10-16T12:47:17.711266+0000","last_fullsized":"2023-10-16T12:47:17.711266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:59.053243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870265+0000","last_change":"2023-10-16T12:47:15.871600+0000","last_active":"2023-10-16T12:47:16.870265+0000","last_peered":"2023-10-16T12:47:16.870265+0000","last_clean":"2023-10-16T12:47:16.870265+0000","last_became_active":"2023-10-16T12:47:15.867470+0000","last_became_peered":"2023-10-16T12:47:15.867470+0000","last_unstale":"2023-10-16T12:47:16.870265+0000","last_undegraded":"2023-10-16T12:47:16.870265+0000","last_fullsized":"2023-10-16T12:47:16.870265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:02.574295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866982+0000","last_change":"2023-10-16T12:47:14.843191+0000","last_active":"2023-10-16T12:47:16.866982+0000","last_peered":"2023-10-16T12:47:16.866982+0000","last_clean":"2023-10-16T12:47:16.866982+0000","last_became_active":"2023-10-16T12:47:14.842208+0000","last_became_peered":"2023-10-16T12:47:14.842208+0000","last_unstale":"2023-10-16T12:47:16.866982+0000","last_undegraded":"2023-10-16T12:47:16.866982+0000","last_fullsized":"2023-10-16T12:47:16.866982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:30.449917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868444+0000","last_change":"2023-10-16T12:47:15.870312+0000","last_active":"2023-10-16T12:47:16.868444+0000","last_peered":"2023-10-16T12:47:16.868444+0000","last_clean":"2023-10-16T12:47:16.868444+0000","last_became_active":"2023-10-16T12:47:15.868896+0000","last_became_peered":"2023-10-16T12:47:15.868896+0000","last_unstale":"2023-10-16T12:47:16.868444+0000","last_undegraded":"2023-10-16T12:47:16.868444+0000","last_fullsized":"2023-10-16T12:47:16.868444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:16.958321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.873775+0000","last_change":"2023-10-16T12:47:14.845137+0000","last_active":"2023-10-16T12:47:16.873775+0000","last_peered":"2023-10-16T12:47:16.873775+0000","last_clean":"2023-10-16T12:47:16.873775+0000","last_became_active":"2023-10-16T12:47:14.844845+0000","last_became_peered":"2023-10-16T12:47:14.844845+0000","last_unstale":"2023-10-16T12:47:16.873775+0000","last_undegraded":"2023-10-16T12:47:16.873775+0000","last_fullsized":"2023-10-16T12:47:16.873775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:07.222854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"55'139","reported_seq":290,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T12:47:15.860723+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:47:15.860723+0000","last_peered":"2023-10-16T12:47:15.860723+0000","last_clean":"2023-10-16T12:47:15.860723+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:47:15.860723+0000","last_undegraded":"2023-10-16T12:47:15.860723+0000","last_fullsized":"2023-10-16T12:47:15.860723+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":956037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3260000000000001,"15min":1.282},"min":{"1min":0.91400000000000003,"5min":0.77800000000000002,"15min":0.65100000000000002},"max":{"1min":1.6599999999999999,"5min":1.7050000000000001,"15min":2.1890000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.325},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.746,"5min":1.8420000000000001,"15min":1.8899999999999999},"last":1.2989999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2250000000000001,"15min":1.214},"min":{"1min":0.85799999999999998,"5min":0.81200000000000006,"15min":0.67300000000000004},"max":{"1min":1.5309999999999999,"5min":1.637,"15min":1.752},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2250000000000001,"15min":1.1950000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.77100000000000002},"max":{"1min":1.577,"5min":1.708,"15min":1.843},"last":1.3779999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.222,"15min":1.2070000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.59799999999999998},"max":{"1min":1.591,"5min":1.657,"15min":2.2130000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.337,"15min":1.3400000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.63500000000000001},"max":{"1min":1.6659999999999999,"5min":1.837,"15min":2.3450000000000002},"last":1.133}]},{"osd":3,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.28,"15min":1.3009999999999999},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.69099999999999995},"max":{"1min":1.508,"5min":1.673,"15min":2.0150000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.391,"15min":1.391},"min":{"1min":1.2210000000000001,"5min":0.97099999999999997,"15min":0.85999999999999999},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":2.0019999999999998},"last":1.468}]},{"osd":4,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.1819999999999999,"15min":1.2030000000000001},"min":{"1min":0.82899999999999996,"5min":0.72099999999999997,"15min":0.67300000000000004},"max":{"1min":1.7030000000000001,"5min":1.744,"15min":1.9710000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.323,"15min":1.351},"min":{"1min":0.96699999999999997,"5min":0.73099999999999998,"15min":0.56499999999999995},"max":{"1min":1.736,"5min":1.748,"15min":2.282},"last":1.169}]},{"osd":5,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3180000000000001,"15min":1.3400000000000001},"min":{"1min":1.0720000000000001,"5min":0.89300000000000002,"15min":0.78900000000000003},"max":{"1min":1.627,"5min":1.671,"15min":2.0529999999999999},"last":1.115},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.4330000000000001,"15min":1.482},"min":{"1min":1.242,"5min":0.98899999999999999,"15min":0.61299999999999999},"max":{"1min":1.768,"5min":1.8169999999999999,"15min":2.2000000000000002},"last":1.5349999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.548,"15min":1.5700000000000001},"min":{"1min":1.1419999999999999,"5min":1.026,"15min":0.72699999999999998},"max":{"1min":1.833,"5min":1.865,"15min":2.2639999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.419,"15min":1.4470000000000001},"min":{"1min":1.298,"5min":0.89100000000000001,"15min":0.83399999999999996},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1379999999999999},"last":1.2150000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":366251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.325,"15min":1.282},"min":{"1min":0.749,"5min":0.749,"15min":0.71499999999999997},"max":{"1min":1.5840000000000001,"5min":1.657,"15min":3.7450000000000001},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.343,"15min":1.3109999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.57999999999999996},"max":{"1min":1.528,"5min":1.6819999999999999,"15min":3.819},"last":1.536}]},{"osd":1,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2050000000000001,"15min":1.1579999999999999},"min":{"1min":0.86899999999999999,"5min":0.82899999999999996,"15min":0.621},"max":{"1min":1.5329999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.216,"15min":1.218},"min":{"1min":0.86199999999999999,"5min":0.72199999999999998,"15min":0.66800000000000004},"max":{"1min":1.7110000000000001,"5min":2.0459999999999998,"15min":3.8519999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.236,"15min":1.21},"min":{"1min":0.93000000000000005,"5min":0.83999999999999997,"15min":0.68300000000000005},"max":{"1min":1.663,"5min":1.663,"15min":1.762},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.272,"15min":1.2749999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.755},"max":{"1min":1.516,"5min":1.9970000000000001,"15min":3.9009999999999998},"last":1.2270000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3260000000000001,"15min":1.3440000000000001},"min":{"1min":0.93899999999999995,"5min":0.92400000000000004,"15min":0.751},"max":{"1min":1.867,"5min":2.3900000000000001,"15min":4.2380000000000004},"last":1.492},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.3280000000000001,"15min":1.3560000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.012,"5min":2.012,"15min":4.4269999999999996},"last":1.087}]},{"osd":4,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3400000000000001,"15min":1.359},"min":{"1min":0.89000000000000001,"5min":0.78000000000000003,"15min":0.57699999999999996},"max":{"1min":2.0830000000000002,"5min":2.3029999999999999,"15min":2.4100000000000001},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4419999999999999,"15min":1.49},"min":{"1min":1.127,"5min":0.82099999999999995,"15min":0.67300000000000004},"max":{"1min":1.764,"5min":1.7709999999999999,"15min":2.3050000000000002},"last":1.3460000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.3009999999999999,"15min":1.3169999999999999},"min":{"1min":0.872,"5min":0.79600000000000004,"15min":0.63900000000000001},"max":{"1min":1.716,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.425,"5min":1.4079999999999999,"15min":1.476},"min":{"1min":1.1000000000000001,"5min":0.92000000000000004,"15min":0.63900000000000001},"max":{"1min":1.9570000000000001,"5min":2.1699999999999999,"15min":2.5230000000000001},"last":1.27}]},{"osd":7,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3660000000000001,"15min":1.4490000000000001},"min":{"1min":0.97899999999999998,"5min":0.96299999999999997,"15min":0.495},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4470000000000001,"15min":1.474},"min":{"1min":0.98699999999999999,"5min":0.96399999999999997,"15min":0.59099999999999997},"max":{"1min":1.825,"5min":2.2610000000000001,"15min":2.2719999999999998},"last":1.3979999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":371949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1819999999999999,"15min":1.1519999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1639999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.67400000000000004},"max":{"1min":1.641,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":1.0569999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.222,"15min":1.2470000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.639,"5min":1.639,"15min":2.415},"last":1.169},{"interface":"front","average":{"1min":1.123,"5min":1.226,"15min":1.226},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.452,"5min":1.6899999999999999,"15min":1.9419999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2709999999999999,"15min":1.294},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.74199999999999999},"max":{"1min":1.5840000000000001,"5min":1.6519999999999999,"15min":2.0259999999999998},"last":1.669},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2829999999999999,"15min":1.258},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.123}]},{"osd":3,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2969999999999999,"15min":1.3109999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":2.2690000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.341,"5min":1.351,"15min":1.3720000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.361}]},{"osd":4,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.21,"15min":1.2150000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.996,"5min":1.996,"15min":2.7909999999999999},"last":1.006},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3169999999999999,"15min":1.361},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.806,"5min":1.806,"15min":5.3250000000000002},"last":1.296}]},{"osd":6,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3500000000000001,"15min":1.345},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.927,"5min":1.927,"15min":2.343},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.395,"15min":1.383},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.6520000000000001},"last":1.323}]},{"osd":7,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4610000000000001,"15min":1.4870000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.889,"5min":1.889,"15min":5.1269999999999998},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.5189999999999999,"15min":1.5289999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.4929999999999999},"last":1.615}]}]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58336,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313504,"statfs":{"total":96636764160,"available":96577028096,"internally_reserved":0,"allocated":806912,"data_stored":373491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2410000000000001,"15min":1.1870000000000001},"min":{"1min":0.82299999999999995,"5min":0.80600000000000005,"15min":0.64500000000000002},"max":{"1min":1.665,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.159,"15min":1.147},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.623},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.9159999999999999},"last":1.5069999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.194,"15min":1.2050000000000001},"min":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.68799999999999994},"max":{"1min":1.542,"5min":1.8400000000000001,"15min":1.865},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.266,"15min":1.28},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.68799999999999994},"max":{"1min":1.6359999999999999,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.681}]},{"osd":2,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.232},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.753},"max":{"1min":1.7370000000000001,"5min":1.865,"15min":1.865},"last":1.115},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.363,"15min":1.3580000000000001},"min":{"1min":0.92200000000000004,"5min":0.91800000000000004,"15min":0.73399999999999999},"max":{"1min":1.502,"5min":1.758,"15min":1.9910000000000001},"last":1.6240000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2709999999999999,"15min":1.2589999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.752},"max":{"1min":1.7130000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.383,"15min":1.3740000000000001},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.76500000000000001},"max":{"1min":1.819,"5min":1.956,"15min":1.956},"last":1.4039999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2689999999999999,"15min":1.391},"min":{"1min":1.0129999999999999,"5min":0.87,"15min":0.50900000000000001},"max":{"1min":1.8260000000000001,"5min":2.0339999999999998,"15min":7.9000000000000004},"last":1.319},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.288,"15min":1.369},"min":{"1min":0.81999999999999995,"5min":0.81399999999999995,"15min":0.53600000000000003},"max":{"1min":1.913,"5min":1.913,"15min":7.3460000000000001},"last":1.1739999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.91700000000000004,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":1.718,"5min":1.788,"15min":2.0139999999999998},"last":1.248},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.472,"15min":1.504},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.65300000000000002},"max":{"1min":2.0139999999999998,"5min":2.1339999999999999,"15min":2.6269999999999998},"last":1.5509999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.4530000000000001,"15min":1.476},"min":{"1min":1.1220000000000001,"5min":0.88400000000000001,"15min":0.54200000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.254},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.5609999999999999,"15min":1.5980000000000001},"min":{"1min":1.1419999999999999,"5min":0.94799999999999995,"15min":0.69199999999999995},"max":{"1min":1.966,"5min":2.1699999999999999,"15min":2.5289999999999999},"last":1.478}]}]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40276,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331564,"statfs":{"total":96636764160,"available":96595521536,"internally_reserved":0,"allocated":794624,"data_stored":366217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.95999999999999996,"15min":0.95099999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.3260000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.206,"15min":1.1599999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5960000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.4079999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0329999999999999,"15min":1.026},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.39500000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.8149999999999999},"last":1.137},{"interface":"front","average":{"1min":1.014,"5min":1.0920000000000001,"15min":1.0820000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"max":{"1min":1.4690000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.79900000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0980000000000001,"15min":1.0660000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.712,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.3759999999999999,"15min":1.349},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.58299999999999996},"max":{"1min":1.8600000000000001,"5min":1.921,"15min":2.0569999999999999},"last":1.3069999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.5,"15min":1.476},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.77600000000000002},"max":{"1min":1.788,"5min":1.956,"15min":2.2730000000000001},"last":1.569},{"interface":"front","average":{"1min":1.383,"5min":1.4370000000000001,"15min":1.4279999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.63500000000000001},"max":{"1min":1.821,"5min":1.859,"15min":3.6930000000000001},"last":1.671}]},{"osd":5,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.60699999999999998},"max":{"1min":1.8600000000000001,"5min":1.8779999999999999,"15min":3.8300000000000001},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4630000000000001,"15min":1.4570000000000001},"min":{"1min":0.98899999999999999,"5min":0.95399999999999996,"15min":0.64900000000000002},"max":{"1min":1.9990000000000001,"5min":2.1030000000000002,"15min":2.3540000000000001},"last":1.8340000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.468,"15min":1.4550000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.65300000000000002},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.415},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.4359999999999999,"15min":1.4370000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.238,"5min":2.3740000000000001,"15min":3.9809999999999999},"last":1.458}]},{"osd":7,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5549999999999999,"15min":1.5389999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.80300000000000005},"max":{"1min":2.1099999999999999,"5min":2.198,"15min":2.431},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.657,"15min":1.6060000000000001},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.95099999999999996},"max":{"1min":2.0649999999999999,"5min":2.552,"15min":2.552},"last":1.5129999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40280,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331560,"statfs":{"total":96636764160,"available":96595517440,"internally_reserved":0,"allocated":798720,"data_stored":367213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.94699999999999995,"15min":0.93799999999999994},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":1.742,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1120000000000001,"15min":1.081},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.5},"max":{"1min":1.5529999999999999,"5min":1.946,"15min":2.085},"last":0.80400000000000005}]},{"osd":1,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.145,"15min":1.107},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.50900000000000001},"max":{"1min":1.8260000000000001,"5min":1.851,"15min":1.9330000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.22,"5min":1.268,"15min":1.2050000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45500000000000002},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.4289999999999998},"last":1.5680000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1060000000000001,"15min":1.0620000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6879999999999999,"5min":1.859,"15min":1.859},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.337,"15min":1.2929999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.59199999999999997},"max":{"1min":1.831,"5min":1.831,"15min":2.1970000000000001},"last":0.96099999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4890000000000001,"15min":1.4470000000000001},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.89000000000000001},"max":{"1min":1.958,"5min":2.3159999999999998,"15min":2.383},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.458,"15min":1.446},"min":{"1min":1.014,"5min":1.014,"15min":0.85899999999999999},"max":{"1min":1.9390000000000001,"5min":2.2530000000000001,"15min":2.294},"last":1.2410000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5329999999999999,"15min":1.482},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.83499999999999996},"max":{"1min":2.23,"5min":2.23,"15min":2.3340000000000001},"last":1.347},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4510000000000001,"15min":1.427},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.502}]},{"osd":6,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.456,"15min":1.4610000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.85999999999999999},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.5820000000000001,"15min":1.5649999999999999},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":0.95799999999999996},"max":{"1min":2.294,"5min":2.3279999999999998,"15min":2.488},"last":1.2170000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6339999999999999,"15min":1.599},"min":{"1min":1.113,"5min":1.113,"15min":0.96699999999999997},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5469999999999999,"15min":1.5589999999999999},"min":{"1min":1.014,"5min":1.014,"15min":0.90400000000000003},"max":{"1min":2.573,"5min":2.573,"15min":2.6429999999999998},"last":1.3839999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54172,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317668,"statfs":{"total":96636764160,"available":96581292032,"internally_reserved":0,"allocated":798720,"data_stored":371949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.91300000000000003,"15min":0.89700000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.02,"5min":1.103,"15min":1.085},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":1.456,"5min":1.6040000000000001,"15min":1.7669999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.01,"15min":0.98899999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8460000000000001,"5min":2.262,"15min":2.262},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2250000000000001,"15min":1.1799999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.90700000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.325,"15min":1.2949999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.677,"5min":1.907,"15min":2.2349999999999999},"last":1.018},{"interface":"front","average":{"1min":1.24,"5min":1.2310000000000001,"15min":1.204},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.46800000000000003},"max":{"1min":1.617,"5min":1.883,"15min":2.1789999999999998},"last":0.79800000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.304,"15min":1.3},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.788,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":1.482},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.448,"15min":1.45},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9750000000000001,"5min":2.0649999999999999,"15min":3.0659999999999998},"last":1.2729999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.4239999999999999,"15min":1.4379999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.754,"5min":2.0070000000000001,"15min":2.0499999999999998},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.4430000000000001,"15min":1.4059999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":1.9930000000000001,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.6839999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.542,"15min":1.54},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.839,"5min":2.1899999999999999,"15min":2.9350000000000001},"last":1.206},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.526,"15min":1.5269999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.8859999999999999,"5min":2.246,"15min":2.246},"last":1.2490000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.637,"15min":1.623},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.948,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.554,"15min":1.5549999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.921,"5min":2.2370000000000001,"15min":2.323},"last":1.613}]}]},{"osd":0,"up_from":14,"seq":60129542314,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1392640,"data_stored":962281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.879,"15min":0.90200000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.274,"5min":1.573,"15min":2.5720000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.121,"15min":1.0629999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.32700000000000001},"max":{"1min":1.6870000000000001,"5min":1.7190000000000001,"15min":2.1800000000000002},"last":1.1850000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.097,"15min":1.111},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.482,"5min":1.7210000000000001,"15min":2.7650000000000001},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1950000000000001,"15min":1.143},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.438},"max":{"1min":1.952,"5min":1.952,"15min":1.952},"last":1.52}]},{"osd":3,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1359999999999999,"15min":1.0580000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.315},"max":{"1min":1.575,"5min":1.9319999999999999,"15min":3.0339999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.276,"15min":1.286},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.45700000000000002},"max":{"1min":1.6160000000000001,"5min":1.794,"15min":3.6579999999999999},"last":1.3720000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.353,"15min":1.359},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8069999999999999,"5min":1.9950000000000001,"15min":4.726},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.411,"15min":1.4590000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.756,"5min":1.9490000000000001,"15min":4.843},"last":1.966}]},{"osd":5,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.4710000000000001,"15min":1.4039999999999999},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.2229999999999999,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":2.024},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.4279999999999999,"15min":1.4470000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.02,"5min":2.0270000000000001,"15min":4.9059999999999997},"last":1.427}]},{"osd":6,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5209999999999999,"15min":1.522},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.73299999999999998},"max":{"1min":2.0009999999999999,"5min":2.3519999999999999,"15min":3.7890000000000001},"last":2.0750000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.573,"15min":1.506},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.76200000000000001},"max":{"1min":2.145,"5min":2.363,"15min":2.6589999999999998},"last":1.8919999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.6739999999999999,"15min":1.5980000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.82599999999999996},"max":{"1min":2.3490000000000002,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":2.1080000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.5720000000000001,"15min":1.5660000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":4.5890000000000004},"last":1.6639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:47:20.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:47:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.619+0000 7f98b5561640 1 Processor -- start 2023-10-16T12:47:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.619+0000 7f98b5561640 1 -- start start 2023-10-16T12:47:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.621+0000 7f98b5561640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 0x7f98b015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.621+0000 7f98b5561640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b015ed60 con 0x7f98b015aaa0 2023-10-16T12:47:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.621+0000 7f98b5561640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b00707e0 con 0x7f98b015ae80 2023-10-16T12:47:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.621+0000 7f98b5561640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b006b810 con 0x7f98b00ce390 2023-10-16T12:47:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.622+0000 7f98aeffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f98b015aaa0 0x7f98b00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51172/0 (socket says 172.21.15.60:51172) 2023-10-16T12:47:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.622+0000 7f98aeffd640 1 -- 172.21.15.60:0/1261880437 learned_addr learned my addr 172.21.15.60:0/1261880437 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.622+0000 7f98af7fe640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 0x7f98b015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.622+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840436422 0 0) 0x7f98b015ed60 con 0x7f98b015aaa0 2023-10-16T12:47:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f989c003520 con 0x7f98b015aaa0 2023-10-16T12:47:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855505549 0 0) 0x7f98b00707e0 con 0x7f98b015ae80 2023-10-16T12:47:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b015ed60 con 0x7f98b015ae80 2023-10-16T12:47:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754815419 0 0) 0x7f989c003520 con 0x7f98b015aaa0 2023-10-16T12:47:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 msgr2=0x7f98b015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 0x7f98b015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 >> v1:172.21.15.144:6789/0 conn(0x7f98b015ae80 legacy=0x7f98b0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b015ef90 con 0x7f98b015aaa0 2023-10-16T12:47:20.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f98a0001690 con 0x7f98b015aaa0 2023-10-16T12:47:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 >> v1:172.21.15.60:6789/0 conn(0x7f98b015aaa0 legacy=0x7f98b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f989c003b30 0x7f989c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 0x7f989c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c004d00 0x7f989c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b006b810 con 0x7f989c003b30 2023-10-16T12:47:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c00ba30 con 0x7f989c004d00 2023-10-16T12:47:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989c00bba0 con 0x7f98b00ce390 2023-10-16T12:47:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.623+0000 7f98aeffd640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f989c003b30 0x7f989c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98ae7fc640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 0x7f989c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98aeffd640 1 -- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 msgr2=0x7f989c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:20.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98aeffd640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98b00ce390 0x7f989c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98af7fe640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c004d00 0x7f989c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98aeffd640 1 -- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c004d00 msgr2=0x7f989c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:20.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98aeffd640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f989c004d00 0x7f989c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.624+0000 7f98aeffd640 1 -- 172.21.15.60:0/1261880437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c003520 con 0x7f989c003b30 2023-10-16T12:47:20.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.625+0000 7f98aeffd640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f989c003b30 0x7f989c003f30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f98b00c2490 tx=0x7f989c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aedc04e7223260f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:20.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0005ad0 con 0x7f989c003b30 2023-10-16T12:47:20.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a0005c50 con 0x7f989c003b30 2023-10-16T12:47:20.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98adffb640 1 -- 172.21.15.60:0/1261880437 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a002c810 con 0x7f989c003b30 2023-10-16T12:47:20.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98b5561640 1 -- 172.21.15.60:0/1261880437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f989c003b30 msgr2=0x7f989c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98b5561640 1 --2- 172.21.15.60:0/1261880437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f989c003b30 0x7f989c003f30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f98b00c2490 tx=0x7f989c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98af7fe640 1 -- 172.21.15.60:0/1261880437 reap_dead start 2023-10-16T12:47:20.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98b5561640 1 -- 172.21.15.60:0/1261880437 shutdown_connections 2023-10-16T12:47:20.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.626+0000 7f98b5561640 1 -- 172.21.15.60:0/1261880437 >> 172.21.15.60:0/1261880437 conn(0x7f98b00cb410 msgr2=0x7f98b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:20.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.627+0000 7f98b5561640 1 -- 172.21.15.60:0/1261880437 shutdown_connections 2023-10-16T12:47:20.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.627+0000 7f98b5561640 1 -- 172.21.15.60:0/1261880437 wait complete. 2023-10-16T12:47:20.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.629+0000 7f98b5561640 1 Processor -- start 2023-10-16T12:47:20.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.629+0000 7f98b5561640 1 -- start start 2023-10-16T12:47:20.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98b5561640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c003b30 0x7f98b00c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98b5561640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 0x7f98b00af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98b5561640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98b00acb00 0x7f98b014eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98b5561640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b015d1e0 con 0x7f98b00caca0 2023-10-16T12:47:20.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98b5561640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b015cf20 con 0x7f98b00acb00 2023-10-16T12:47:20.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98b5561640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b00ca0f0 con 0x7f989c003b30 2023-10-16T12:47:20.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98ae7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 0x7f98b00af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98ae7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 0x7f98b00af9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34132/0 (socket says 172.21.15.60:34132) 2023-10-16T12:47:20.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98ae7fc640 1 -- 172.21.15.60:0/3992893198 learned_addr learned my addr 172.21.15.60:0/3992893198 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:20.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98aeffd640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c003b30 0x7f98b00c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98af7fe640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98b00acb00 0x7f98b014eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.630+0000 7f98ae7fc640 1 -- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c003b30 msgr2=0x7f98b00c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98ae7fc640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c003b30 0x7f98b00c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98ae7fc640 1 -- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98b00acb00 msgr2=0x7f98b014eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98ae7fc640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98b00acb00 0x7f98b014eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98ae7fc640 1 -- 172.21.15.60:0/3992893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b006c5b0 con 0x7f98b00caca0 2023-10-16T12:47:20.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98aeffd640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c003b30 0x7f98b00c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:20.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98ae7fc640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 0x7f98b00af9e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f989800eb50 tx=0x7f989800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:20.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800ee50 con 0x7f98b00caca0 2023-10-16T12:47:20.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989800cbc0 con 0x7f98b00caca0 2023-10-16T12:47:20.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898003970 con 0x7f98b00caca0 2023-10-16T12:47:20.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b015b450 con 0x7f98b00caca0 2023-10-16T12:47:20.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.631+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b00aff20 con 0x7f98b00caca0 2023-10-16T12:47:20.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.633+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b015d350 con 0x7f98b00caca0 2023-10-16T12:47:20.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.633+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9898003ad0 con 0x7f98b00caca0 2023-10-16T12:47:20.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.634+0000 7f98a77fe640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f989006afe0 0x7f989006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:20.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.634+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98980899f0 con 0x7f98b00caca0 2023-10-16T12:47:20.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.635+0000 7f98aeffd640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f989006afe0 0x7f989006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:20.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.636+0000 7f98aeffd640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f989006afe0 0x7f989006d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98a0034000 tx=0x7f98a0007000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:20.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.639+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98b015d350 con 0x7f98b00caca0 2023-10-16T12:47:20.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.835+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 --> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98b015f550 con 0x7f989006afe0 2023-10-16T12:47:20.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.858+0000 7f98a77fe640 1 -- 172.21.15.60:0/3992893198 <== mgr.4910 v2:172.21.15.144:6832/3875614959 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286806 (secure 0 0 0) 0x7f98a0001590 con 0x7f989006afe0 2023-10-16T12:47:20.863 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:20.865 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:47:20.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f989006afe0 msgr2=0x7f989006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f989006afe0 0x7f989006d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98a0034000 tx=0x7f98a0007000 comp rx=0 tx=0).stop 2023-10-16T12:47:20.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 msgr2=0x7f98b00af9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:20.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 0x7f98b00af9e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f989800eb50 tx=0x7f989800c4b0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 shutdown_connections 2023-10-16T12:47:20.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f989006afe0 0x7f989006d4a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98b00acb00 0x7f98b014eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00caca0 0x7f98b00af9e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 --2- 172.21.15.60:0/3992893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f989c003b30 0x7f98b00c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:20.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 >> 172.21.15.60:0/3992893198 conn(0x7f98b00cb410 msgr2=0x7f98b006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:20.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.863+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 shutdown_connections 2023-10-16T12:47:20.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:20.864+0000 7f98b5561640 1 -- 172.21.15.60:0/3992893198 wait complete. 2023-10-16T12:47:20.884 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T12:47:19.119971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326220,"kb_used_data":7384,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754648500,"statfs":{"total":773094113280,"available":772760064000,"internally_reserved":0,"allocated":7561216,"data_stored":4135388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":326356536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.307763"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715094+0000","last_change":"2023-10-16T12:47:15.881133+0000","last_active":"2023-10-16T12:47:17.715094+0000","last_peered":"2023-10-16T12:47:17.715094+0000","last_clean":"2023-10-16T12:47:17.715094+0000","last_became_active":"2023-10-16T12:47:15.880870+0000","last_became_peered":"2023-10-16T12:47:15.880870+0000","last_unstale":"2023-10-16T12:47:17.715094+0000","last_undegraded":"2023-10-16T12:47:17.715094+0000","last_fullsized":"2023-10-16T12:47:17.715094+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:15.398759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871776+0000","last_change":"2023-10-16T12:47:15.861659+0000","last_active":"2023-10-16T12:47:16.871776+0000","last_peered":"2023-10-16T12:47:16.871776+0000","last_clean":"2023-10-16T12:47:16.871776+0000","last_became_active":"2023-10-16T12:47:15.861348+0000","last_became_peered":"2023-10-16T12:47:15.861348+0000","last_unstale":"2023-10-16T12:47:16.871776+0000","last_undegraded":"2023-10-16T12:47:16.871776+0000","last_fullsized":"2023-10-16T12:47:16.871776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:32:27.290787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870806+0000","last_change":"2023-10-16T12:47:15.865388+0000","last_active":"2023-10-16T12:47:16.870806+0000","last_peered":"2023-10-16T12:47:16.870806+0000","last_clean":"2023-10-16T12:47:16.870806+0000","last_became_active":"2023-10-16T12:47:15.863227+0000","last_became_peered":"2023-10-16T12:47:15.863227+0000","last_unstale":"2023-10-16T12:47:16.870806+0000","last_undegraded":"2023-10-16T12:47:16.870806+0000","last_fullsized":"2023-10-16T12:47:16.870806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:12.908101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867919+0000","last_change":"2023-10-16T12:47:15.880187+0000","last_active":"2023-10-16T12:47:16.867919+0000","last_peered":"2023-10-16T12:47:16.867919+0000","last_clean":"2023-10-16T12:47:16.867919+0000","last_became_active":"2023-10-16T12:47:15.879944+0000","last_became_peered":"2023-10-16T12:47:15.879944+0000","last_unstale":"2023-10-16T12:47:16.867919+0000","last_undegraded":"2023-10-16T12:47:16.867919+0000","last_fullsized":"2023-10-16T12:47:16.867919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:38.432677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872665+0000","last_change":"2023-10-16T12:47:15.861615+0000","last_active":"2023-10-16T12:47:16.872665+0000","last_peered":"2023-10-16T12:47:16.872665+0000","last_clean":"2023-10-16T12:47:16.872665+0000","last_became_active":"2023-10-16T12:47:15.861304+0000","last_became_peered":"2023-10-16T12:47:15.861304+0000","last_unstale":"2023-10-16T12:47:16.872665+0000","last_undegraded":"2023-10-16T12:47:16.872665+0000","last_fullsized":"2023-10-16T12:47:16.872665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:36.363962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593177+0000","last_change":"2023-10-16T12:47:15.868036+0000","last_active":"2023-10-16T12:47:17.593177+0000","last_peered":"2023-10-16T12:47:17.593177+0000","last_clean":"2023-10-16T12:47:17.593177+0000","last_became_active":"2023-10-16T12:47:15.867649+0000","last_became_peered":"2023-10-16T12:47:15.867649+0000","last_unstale":"2023-10-16T12:47:17.593177+0000","last_undegraded":"2023-10-16T12:47:17.593177+0000","last_fullsized":"2023-10-16T12:47:17.593177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.301876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713742+0000","last_change":"2023-10-16T12:47:15.870827+0000","last_active":"2023-10-16T12:47:17.713742+0000","last_peered":"2023-10-16T12:47:17.713742+0000","last_clean":"2023-10-16T12:47:17.713742+0000","last_became_active":"2023-10-16T12:47:15.870354+0000","last_became_peered":"2023-10-16T12:47:15.870354+0000","last_unstale":"2023-10-16T12:47:17.713742+0000","last_undegraded":"2023-10-16T12:47:17.713742+0000","last_fullsized":"2023-10-16T12:47:17.713742+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:19.740976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866820+0000","last_change":"2023-10-16T12:47:15.883869+0000","last_active":"2023-10-16T12:47:16.866820+0000","last_peered":"2023-10-16T12:47:16.866820+0000","last_clean":"2023-10-16T12:47:16.866820+0000","last_became_active":"2023-10-16T12:47:15.882940+0000","last_became_peered":"2023-10-16T12:47:15.882940+0000","last_unstale":"2023-10-16T12:47:16.866820+0000","last_undegraded":"2023-10-16T12:47:16.866820+0000","last_fullsized":"2023-10-16T12:47:16.866820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:41.278440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868566+0000","last_change":"2023-10-16T12:47:15.865294+0000","last_active":"2023-10-16T12:47:16.868566+0000","last_peered":"2023-10-16T12:47:16.868566+0000","last_clean":"2023-10-16T12:47:16.868566+0000","last_became_active":"2023-10-16T12:47:15.864967+0000","last_became_peered":"2023-10-16T12:47:15.864967+0000","last_unstale":"2023-10-16T12:47:16.868566+0000","last_undegraded":"2023-10-16T12:47:16.868566+0000","last_fullsized":"2023-10-16T12:47:16.868566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:14.366218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868348+0000","last_change":"2023-10-16T12:47:15.881539+0000","last_active":"2023-10-16T12:47:16.868348+0000","last_peered":"2023-10-16T12:47:16.868348+0000","last_clean":"2023-10-16T12:47:16.868348+0000","last_became_active":"2023-10-16T12:47:15.881275+0000","last_became_peered":"2023-10-16T12:47:15.881275+0000","last_unstale":"2023-10-16T12:47:16.868348+0000","last_undegraded":"2023-10-16T12:47:16.868348+0000","last_fullsized":"2023-10-16T12:47:16.868348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:33.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868336+0000","last_change":"2023-10-16T12:47:15.864078+0000","last_active":"2023-10-16T12:47:16.868336+0000","last_peered":"2023-10-16T12:47:16.868336+0000","last_clean":"2023-10-16T12:47:16.868336+0000","last_became_active":"2023-10-16T12:47:15.863447+0000","last_became_peered":"2023-10-16T12:47:15.863447+0000","last_unstale":"2023-10-16T12:47:16.868336+0000","last_undegraded":"2023-10-16T12:47:16.868336+0000","last_fullsized":"2023-10-16T12:47:16.868336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:05.594442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872394+0000","last_change":"2023-10-16T12:47:15.866787+0000","last_active":"2023-10-16T12:47:16.872394+0000","last_peered":"2023-10-16T12:47:16.872394+0000","last_clean":"2023-10-16T12:47:16.872394+0000","last_became_active":"2023-10-16T12:47:15.866185+0000","last_became_peered":"2023-10-16T12:47:15.866185+0000","last_unstale":"2023-10-16T12:47:16.872394+0000","last_undegraded":"2023-10-16T12:47:16.872394+0000","last_fullsized":"2023-10-16T12:47:16.872394+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:35.132589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713395+0000","last_change":"2023-10-16T12:47:15.881464+0000","last_active":"2023-10-16T12:47:17.713395+0000","last_peered":"2023-10-16T12:47:17.713395+0000","last_clean":"2023-10-16T12:47:17.713395+0000","last_became_active":"2023-10-16T12:47:15.880203+0000","last_became_peered":"2023-10-16T12:47:15.880203+0000","last_unstale":"2023-10-16T12:47:17.713395+0000","last_undegraded":"2023-10-16T12:47:17.713395+0000","last_fullsized":"2023-10-16T12:47:17.713395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:02.183359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867947+0000","last_change":"2023-10-16T12:47:15.864127+0000","last_active":"2023-10-16T12:47:16.867947+0000","last_peered":"2023-10-16T12:47:16.867947+0000","last_clean":"2023-10-16T12:47:16.867947+0000","last_became_active":"2023-10-16T12:47:15.863430+0000","last_became_peered":"2023-10-16T12:47:15.863430+0000","last_unstale":"2023-10-16T12:47:16.867947+0000","last_undegraded":"2023-10-16T12:47:16.867947+0000","last_fullsized":"2023-10-16T12:47:16.867947+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:12.440413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593109+0000","last_change":"2023-10-16T12:47:15.862090+0000","last_active":"2023-10-16T12:47:17.593109+0000","last_peered":"2023-10-16T12:47:17.593109+0000","last_clean":"2023-10-16T12:47:17.593109+0000","last_became_active":"2023-10-16T12:47:15.861684+0000","last_became_peered":"2023-10-16T12:47:15.861684+0000","last_unstale":"2023-10-16T12:47:17.593109+0000","last_undegraded":"2023-10-16T12:47:17.593109+0000","last_fullsized":"2023-10-16T12:47:17.593109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:45.289233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713803+0000","last_change":"2023-10-16T12:47:15.869579+0000","last_active":"2023-10-16T12:47:17.713803+0000","last_peered":"2023-10-16T12:47:17.713803+0000","last_clean":"2023-10-16T12:47:17.713803+0000","last_became_active":"2023-10-16T12:47:15.869303+0000","last_became_peered":"2023-10-16T12:47:15.869303+0000","last_unstale":"2023-10-16T12:47:17.713803+0000","last_undegraded":"2023-10-16T12:47:17.713803+0000","last_fullsized":"2023-10-16T12:47:17.713803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:33:23.634134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871706+0000","last_change":"2023-10-16T12:47:15.868752+0000","last_active":"2023-10-16T12:47:16.871706+0000","last_peered":"2023-10-16T12:47:16.871706+0000","last_clean":"2023-10-16T12:47:16.871706+0000","last_became_active":"2023-10-16T12:47:15.868439+0000","last_became_peered":"2023-10-16T12:47:15.868439+0000","last_unstale":"2023-10-16T12:47:16.871706+0000","last_undegraded":"2023-10-16T12:47:16.871706+0000","last_fullsized":"2023-10-16T12:47:16.871706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:29.974892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871274+0000","last_change":"2023-10-16T12:47:15.864974+0000","last_active":"2023-10-16T12:47:16.871274+0000","last_peered":"2023-10-16T12:47:16.871274+0000","last_clean":"2023-10-16T12:47:16.871274+0000","last_became_active":"2023-10-16T12:47:15.864681+0000","last_became_peered":"2023-10-16T12:47:15.864681+0000","last_unstale":"2023-10-16T12:47:16.871274+0000","last_undegraded":"2023-10-16T12:47:16.871274+0000","last_fullsized":"2023-10-16T12:47:16.871274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:42:51.718249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716488+0000","last_change":"2023-10-16T12:47:15.884654+0000","last_active":"2023-10-16T12:47:17.716488+0000","last_peered":"2023-10-16T12:47:17.716488+0000","last_clean":"2023-10-16T12:47:17.716488+0000","last_became_active":"2023-10-16T12:47:15.882247+0000","last_became_peered":"2023-10-16T12:47:15.882247+0000","last_unstale":"2023-10-16T12:47:17.716488+0000","last_undegraded":"2023-10-16T12:47:17.716488+0000","last_fullsized":"2023-10-16T12:47:17.716488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:33.605094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713707+0000","last_change":"2023-10-16T12:47:15.870082+0000","last_active":"2023-10-16T12:47:17.713707+0000","last_peered":"2023-10-16T12:47:17.713707+0000","last_clean":"2023-10-16T12:47:17.713707+0000","last_became_active":"2023-10-16T12:47:15.869773+0000","last_became_peered":"2023-10-16T12:47:15.869773+0000","last_unstale":"2023-10-16T12:47:17.713707+0000","last_undegraded":"2023-10-16T12:47:17.713707+0000","last_fullsized":"2023-10-16T12:47:17.713707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:01.337793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593346+0000","last_change":"2023-10-16T12:47:15.879746+0000","last_active":"2023-10-16T12:47:17.593346+0000","last_peered":"2023-10-16T12:47:17.593346+0000","last_clean":"2023-10-16T12:47:17.593346+0000","last_became_active":"2023-10-16T12:47:15.879426+0000","last_became_peered":"2023-10-16T12:47:15.879426+0000","last_unstale":"2023-10-16T12:47:17.593346+0000","last_undegraded":"2023-10-16T12:47:17.593346+0000","last_fullsized":"2023-10-16T12:47:17.593346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:11:48.086170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868157+0000","last_change":"2023-10-16T12:47:15.886162+0000","last_active":"2023-10-16T12:47:16.868157+0000","last_peered":"2023-10-16T12:47:16.868157+0000","last_clean":"2023-10-16T12:47:16.868157+0000","last_became_active":"2023-10-16T12:47:15.885430+0000","last_became_peered":"2023-10-16T12:47:15.885430+0000","last_unstale":"2023-10-16T12:47:16.868157+0000","last_undegraded":"2023-10-16T12:47:16.868157+0000","last_fullsized":"2023-10-16T12:47:16.868157+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:57.147140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714575+0000","last_change":"2023-10-16T12:47:15.882268+0000","last_active":"2023-10-16T12:47:17.714575+0000","last_peered":"2023-10-16T12:47:17.714575+0000","last_clean":"2023-10-16T12:47:17.714575+0000","last_became_active":"2023-10-16T12:47:15.882003+0000","last_became_peered":"2023-10-16T12:47:15.882003+0000","last_unstale":"2023-10-16T12:47:17.714575+0000","last_undegraded":"2023-10-16T12:47:17.714575+0000","last_fullsized":"2023-10-16T12:47:17.714575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:31.877461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714145+0000","last_change":"2023-10-16T12:47:15.873951+0000","last_active":"2023-10-16T12:47:17.714145+0000","last_peered":"2023-10-16T12:47:17.714145+0000","last_clean":"2023-10-16T12:47:17.714145+0000","last_became_active":"2023-10-16T12:47:15.873598+0000","last_became_peered":"2023-10-16T12:47:15.873598+0000","last_unstale":"2023-10-16T12:47:17.714145+0000","last_undegraded":"2023-10-16T12:47:17.714145+0000","last_fullsized":"2023-10-16T12:47:17.714145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:43:37.236166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716460+0000","last_change":"2023-10-16T12:47:15.882085+0000","last_active":"2023-10-16T12:47:17.716460+0000","last_peered":"2023-10-16T12:47:17.716460+0000","last_clean":"2023-10-16T12:47:17.716460+0000","last_became_active":"2023-10-16T12:47:15.879599+0000","last_became_peered":"2023-10-16T12:47:15.879599+0000","last_unstale":"2023-10-16T12:47:17.716460+0000","last_undegraded":"2023-10-16T12:47:17.716460+0000","last_fullsized":"2023-10-16T12:47:17.716460+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:24.788118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713082+0000","last_change":"2023-10-16T12:47:15.879603+0000","last_active":"2023-10-16T12:47:17.713082+0000","last_peered":"2023-10-16T12:47:17.713082+0000","last_clean":"2023-10-16T12:47:17.713082+0000","last_became_active":"2023-10-16T12:47:15.879310+0000","last_became_peered":"2023-10-16T12:47:15.879310+0000","last_unstale":"2023-10-16T12:47:17.713082+0000","last_undegraded":"2023-10-16T12:47:17.713082+0000","last_fullsized":"2023-10-16T12:47:17.713082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:06.479762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870884+0000","last_change":"2023-10-16T12:47:15.859873+0000","last_active":"2023-10-16T12:47:16.870884+0000","last_peered":"2023-10-16T12:47:16.870884+0000","last_clean":"2023-10-16T12:47:16.870884+0000","last_became_active":"2023-10-16T12:47:15.859474+0000","last_became_peered":"2023-10-16T12:47:15.859474+0000","last_unstale":"2023-10-16T12:47:16.870884+0000","last_undegraded":"2023-10-16T12:47:16.870884+0000","last_fullsized":"2023-10-16T12:47:16.870884+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:55.824293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872727+0000","last_change":"2023-10-16T12:47:15.863582+0000","last_active":"2023-10-16T12:47:16.872727+0000","last_peered":"2023-10-16T12:47:16.872727+0000","last_clean":"2023-10-16T12:47:16.872727+0000","last_became_active":"2023-10-16T12:47:15.863163+0000","last_became_peered":"2023-10-16T12:47:15.863163+0000","last_unstale":"2023-10-16T12:47:16.872727+0000","last_undegraded":"2023-10-16T12:47:16.872727+0000","last_fullsized":"2023-10-16T12:47:16.872727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:22.063939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715391+0000","last_change":"2023-10-16T12:47:15.878832+0000","last_active":"2023-10-16T12:47:17.715391+0000","last_peered":"2023-10-16T12:47:17.715391+0000","last_clean":"2023-10-16T12:47:17.715391+0000","last_became_active":"2023-10-16T12:47:15.878163+0000","last_became_peered":"2023-10-16T12:47:15.878163+0000","last_unstale":"2023-10-16T12:47:17.715391+0000","last_undegraded":"2023-10-16T12:47:17.715391+0000","last_fullsized":"2023-10-16T12:47:17.715391+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:56.063314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870488+0000","last_change":"2023-10-16T12:47:15.870913+0000","last_active":"2023-10-16T12:47:16.870488+0000","last_peered":"2023-10-16T12:47:16.870488+0000","last_clean":"2023-10-16T12:47:16.870488+0000","last_became_active":"2023-10-16T12:47:15.870642+0000","last_became_peered":"2023-10-16T12:47:15.870642+0000","last_unstale":"2023-10-16T12:47:16.870488+0000","last_undegraded":"2023-10-16T12:47:16.870488+0000","last_fullsized":"2023-10-16T12:47:16.870488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:10.311108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593364+0000","last_change":"2023-10-16T12:47:15.861433+0000","last_active":"2023-10-16T12:47:17.593364+0000","last_peered":"2023-10-16T12:47:17.593364+0000","last_clean":"2023-10-16T12:47:17.593364+0000","last_became_active":"2023-10-16T12:47:15.861138+0000","last_became_peered":"2023-10-16T12:47:15.861138+0000","last_unstale":"2023-10-16T12:47:17.593364+0000","last_undegraded":"2023-10-16T12:47:17.593364+0000","last_fullsized":"2023-10-16T12:47:17.593364+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:49.013743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.712684+0000","last_change":"2023-10-16T12:47:15.873792+0000","last_active":"2023-10-16T12:47:17.712684+0000","last_peered":"2023-10-16T12:47:17.712684+0000","last_clean":"2023-10-16T12:47:17.712684+0000","last_became_active":"2023-10-16T12:47:15.873525+0000","last_became_peered":"2023-10-16T12:47:15.873525+0000","last_unstale":"2023-10-16T12:47:17.712684+0000","last_undegraded":"2023-10-16T12:47:17.712684+0000","last_fullsized":"2023-10-16T12:47:17.712684+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:25:23.146648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711852+0000","last_change":"2023-10-16T12:47:15.871937+0000","last_active":"2023-10-16T12:47:17.711852+0000","last_peered":"2023-10-16T12:47:17.711852+0000","last_clean":"2023-10-16T12:47:17.711852+0000","last_became_active":"2023-10-16T12:47:15.871676+0000","last_became_peered":"2023-10-16T12:47:15.871676+0000","last_unstale":"2023-10-16T12:47:17.711852+0000","last_undegraded":"2023-10-16T12:47:17.711852+0000","last_fullsized":"2023-10-16T12:47:17.711852+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:13.892096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"66'4","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594352+0000","last_change":"2023-10-16T12:47:14.853041+0000","last_active":"2023-10-16T12:47:17.594352+0000","last_peered":"2023-10-16T12:47:17.594352+0000","last_clean":"2023-10-16T12:47:17.594352+0000","last_became_active":"2023-10-16T12:47:14.852693+0000","last_became_peered":"2023-10-16T12:47:14.852693+0000","last_unstale":"2023-10-16T12:47:17.594352+0000","last_undegraded":"2023-10-16T12:47:17.594352+0000","last_fullsized":"2023-10-16T12:47:17.594352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:06.845364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716321+0000","last_change":"2023-10-16T12:47:15.879305+0000","last_active":"2023-10-16T12:47:17.716321+0000","last_peered":"2023-10-16T12:47:17.716321+0000","last_clean":"2023-10-16T12:47:17.716321+0000","last_became_active":"2023-10-16T12:47:15.878006+0000","last_became_peered":"2023-10-16T12:47:15.878006+0000","last_unstale":"2023-10-16T12:47:17.716321+0000","last_undegraded":"2023-10-16T12:47:17.716321+0000","last_fullsized":"2023-10-16T12:47:17.716321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:59.361855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871488+0000","last_change":"2023-10-16T12:47:14.851685+0000","last_active":"2023-10-16T12:47:16.871488+0000","last_peered":"2023-10-16T12:47:16.871488+0000","last_clean":"2023-10-16T12:47:16.871488+0000","last_became_active":"2023-10-16T12:47:14.851200+0000","last_became_peered":"2023-10-16T12:47:14.851200+0000","last_unstale":"2023-10-16T12:47:16.871488+0000","last_undegraded":"2023-10-16T12:47:16.871488+0000","last_fullsized":"2023-10-16T12:47:16.871488+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:26.245432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.873040+0000","last_change":"2023-10-16T12:47:15.864451+0000","last_active":"2023-10-16T12:47:16.873040+0000","last_peered":"2023-10-16T12:47:16.873040+0000","last_clean":"2023-10-16T12:47:16.873040+0000","last_became_active":"2023-10-16T12:47:15.863994+0000","last_became_peered":"2023-10-16T12:47:15.863994+0000","last_unstale":"2023-10-16T12:47:16.873040+0000","last_undegraded":"2023-10-16T12:47:16.873040+0000","last_fullsized":"2023-10-16T12:47:16.873040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:48.556188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711673+0000","last_change":"2023-10-16T12:47:14.845807+0000","last_active":"2023-10-16T12:47:17.711673+0000","last_peered":"2023-10-16T12:47:17.711673+0000","last_clean":"2023-10-16T12:47:17.711673+0000","last_became_active":"2023-10-16T12:47:14.845540+0000","last_became_peered":"2023-10-16T12:47:14.845540+0000","last_unstale":"2023-10-16T12:47:17.711673+0000","last_undegraded":"2023-10-16T12:47:17.711673+0000","last_fullsized":"2023-10-16T12:47:17.711673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:42.162801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711386+0000","last_change":"2023-10-16T12:47:15.879550+0000","last_active":"2023-10-16T12:47:17.711386+0000","last_peered":"2023-10-16T12:47:17.711386+0000","last_clean":"2023-10-16T12:47:17.711386+0000","last_became_active":"2023-10-16T12:47:15.879302+0000","last_became_peered":"2023-10-16T12:47:15.879302+0000","last_unstale":"2023-10-16T12:47:17.711386+0000","last_undegraded":"2023-10-16T12:47:17.711386+0000","last_fullsized":"2023-10-16T12:47:17.711386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:34.490300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714479+0000","last_change":"2023-10-16T12:47:14.853830+0000","last_active":"2023-10-16T12:47:17.714479+0000","last_peered":"2023-10-16T12:47:17.714479+0000","last_clean":"2023-10-16T12:47:17.714479+0000","last_became_active":"2023-10-16T12:47:14.851699+0000","last_became_peered":"2023-10-16T12:47:14.851699+0000","last_unstale":"2023-10-16T12:47:17.714479+0000","last_undegraded":"2023-10-16T12:47:17.714479+0000","last_fullsized":"2023-10-16T12:47:17.714479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:29.957872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870942+0000","last_change":"2023-10-16T12:47:15.882158+0000","last_active":"2023-10-16T12:47:16.870942+0000","last_peered":"2023-10-16T12:47:16.870942+0000","last_clean":"2023-10-16T12:47:16.870942+0000","last_became_active":"2023-10-16T12:47:15.881845+0000","last_became_peered":"2023-10-16T12:47:15.881845+0000","last_unstale":"2023-10-16T12:47:16.870942+0000","last_undegraded":"2023-10-16T12:47:16.870942+0000","last_fullsized":"2023-10-16T12:47:16.870942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:49.068292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593645+0000","last_change":"2023-10-16T12:47:14.851242+0000","last_active":"2023-10-16T12:47:17.593645+0000","last_peered":"2023-10-16T12:47:17.593645+0000","last_clean":"2023-10-16T12:47:17.593645+0000","last_became_active":"2023-10-16T12:47:14.850300+0000","last_became_peered":"2023-10-16T12:47:14.850300+0000","last_unstale":"2023-10-16T12:47:17.593645+0000","last_undegraded":"2023-10-16T12:47:17.593645+0000","last_fullsized":"2023-10-16T12:47:17.593645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:16.119849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867400+0000","last_change":"2023-10-16T12:47:15.859602+0000","last_active":"2023-10-16T12:47:16.867400+0000","last_peered":"2023-10-16T12:47:16.867400+0000","last_clean":"2023-10-16T12:47:16.867400+0000","last_became_active":"2023-10-16T12:47:15.858830+0000","last_became_peered":"2023-10-16T12:47:15.858830+0000","last_unstale":"2023-10-16T12:47:16.867400+0000","last_undegraded":"2023-10-16T12:47:16.867400+0000","last_fullsized":"2023-10-16T12:47:16.867400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:30.780039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871186+0000","last_change":"2023-10-16T12:47:14.843389+0000","last_active":"2023-10-16T12:47:16.871186+0000","last_peered":"2023-10-16T12:47:16.871186+0000","last_clean":"2023-10-16T12:47:16.871186+0000","last_became_active":"2023-10-16T12:47:14.843105+0000","last_became_peered":"2023-10-16T12:47:14.843105+0000","last_unstale":"2023-10-16T12:47:16.871186+0000","last_undegraded":"2023-10-16T12:47:16.871186+0000","last_fullsized":"2023-10-16T12:47:16.871186+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:59.053243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866739+0000","last_change":"2023-10-16T12:47:15.883910+0000","last_active":"2023-10-16T12:47:16.866739+0000","last_peered":"2023-10-16T12:47:16.866739+0000","last_clean":"2023-10-16T12:47:16.866739+0000","last_became_active":"2023-10-16T12:47:15.883534+0000","last_became_peered":"2023-10-16T12:47:15.883534+0000","last_unstale":"2023-10-16T12:47:16.866739+0000","last_undegraded":"2023-10-16T12:47:16.866739+0000","last_fullsized":"2023-10-16T12:47:16.866739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:19.126746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871069+0000","last_change":"2023-10-16T12:47:14.840171+0000","last_active":"2023-10-16T12:47:16.871069+0000","last_peered":"2023-10-16T12:47:16.871069+0000","last_clean":"2023-10-16T12:47:16.871069+0000","last_became_active":"2023-10-16T12:47:14.838988+0000","last_became_peered":"2023-10-16T12:47:14.838988+0000","last_unstale":"2023-10-16T12:47:16.871069+0000","last_undegraded":"2023-10-16T12:47:16.871069+0000","last_fullsized":"2023-10-16T12:47:16.871069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:47.673081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714327+0000","last_change":"2023-10-16T12:47:15.882954+0000","last_active":"2023-10-16T12:47:17.714327+0000","last_peered":"2023-10-16T12:47:17.714327+0000","last_clean":"2023-10-16T12:47:17.714327+0000","last_became_active":"2023-10-16T12:47:15.882442+0000","last_became_peered":"2023-10-16T12:47:15.882442+0000","last_unstale":"2023-10-16T12:47:17.714327+0000","last_undegraded":"2023-10-16T12:47:17.714327+0000","last_fullsized":"2023-10-16T12:47:17.714327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:47.633166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866866+0000","last_change":"2023-10-16T12:47:14.838460+0000","last_active":"2023-10-16T12:47:16.866866+0000","last_peered":"2023-10-16T12:47:16.866866+0000","last_clean":"2023-10-16T12:47:16.866866+0000","last_became_active":"2023-10-16T12:47:14.837900+0000","last_became_peered":"2023-10-16T12:47:14.837900+0000","last_unstale":"2023-10-16T12:47:16.866866+0000","last_undegraded":"2023-10-16T12:47:16.866866+0000","last_fullsized":"2023-10-16T12:47:16.866866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:15.271899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871488+0000","last_change":"2023-10-16T12:47:15.864917+0000","last_active":"2023-10-16T12:47:16.871488+0000","last_peered":"2023-10-16T12:47:16.871488+0000","last_clean":"2023-10-16T12:47:16.871488+0000","last_became_active":"2023-10-16T12:47:15.864619+0000","last_became_peered":"2023-10-16T12:47:15.864619+0000","last_unstale":"2023-10-16T12:47:16.871488+0000","last_undegraded":"2023-10-16T12:47:16.871488+0000","last_fullsized":"2023-10-16T12:47:16.871488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:07.313911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.872135+0000","last_change":"2023-10-16T12:47:14.840571+0000","last_active":"2023-10-16T12:47:16.872135+0000","last_peered":"2023-10-16T12:47:16.872135+0000","last_clean":"2023-10-16T12:47:16.872135+0000","last_became_active":"2023-10-16T12:47:14.840253+0000","last_became_peered":"2023-10-16T12:47:14.840253+0000","last_unstale":"2023-10-16T12:47:16.872135+0000","last_undegraded":"2023-10-16T12:47:16.872135+0000","last_fullsized":"2023-10-16T12:47:16.872135+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:23.868424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715202+0000","last_change":"2023-10-16T12:47:15.871092+0000","last_active":"2023-10-16T12:47:17.715202+0000","last_peered":"2023-10-16T12:47:17.715202+0000","last_clean":"2023-10-16T12:47:17.715202+0000","last_became_active":"2023-10-16T12:47:15.870802+0000","last_became_peered":"2023-10-16T12:47:15.870802+0000","last_unstale":"2023-10-16T12:47:17.715202+0000","last_undegraded":"2023-10-16T12:47:17.715202+0000","last_fullsized":"2023-10-16T12:47:17.715202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:22.260815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868432+0000","last_change":"2023-10-16T12:47:14.852327+0000","last_active":"2023-10-16T12:47:16.868432+0000","last_peered":"2023-10-16T12:47:16.868432+0000","last_clean":"2023-10-16T12:47:16.868432+0000","last_became_active":"2023-10-16T12:47:14.852013+0000","last_became_peered":"2023-10-16T12:47:14.852013+0000","last_unstale":"2023-10-16T12:47:16.868432+0000","last_undegraded":"2023-10-16T12:47:16.868432+0000","last_fullsized":"2023-10-16T12:47:16.868432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:44:28.187045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594409+0000","last_change":"2023-10-16T12:47:15.872942+0000","last_active":"2023-10-16T12:47:17.594409+0000","last_peered":"2023-10-16T12:47:17.594409+0000","last_clean":"2023-10-16T12:47:17.594409+0000","last_became_active":"2023-10-16T12:47:15.871624+0000","last_became_peered":"2023-10-16T12:47:15.871624+0000","last_unstale":"2023-10-16T12:47:17.594409+0000","last_undegraded":"2023-10-16T12:47:17.594409+0000","last_fullsized":"2023-10-16T12:47:17.594409+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:19.740976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"66'3","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714519+0000","last_change":"2023-10-16T12:47:14.850845+0000","last_active":"2023-10-16T12:47:17.714519+0000","last_peered":"2023-10-16T12:47:17.714519+0000","last_clean":"2023-10-16T12:47:17.714519+0000","last_became_active":"2023-10-16T12:47:14.850568+0000","last_became_peered":"2023-10-16T12:47:14.850568+0000","last_unstale":"2023-10-16T12:47:17.714519+0000","last_undegraded":"2023-10-16T12:47:17.714519+0000","last_fullsized":"2023-10-16T12:47:17.714519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:19.735554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713967+0000","last_change":"2023-10-16T12:47:15.875474+0000","last_active":"2023-10-16T12:47:17.713967+0000","last_peered":"2023-10-16T12:47:17.713967+0000","last_clean":"2023-10-16T12:47:17.713967+0000","last_became_active":"2023-10-16T12:47:15.875135+0000","last_became_peered":"2023-10-16T12:47:15.875135+0000","last_unstale":"2023-10-16T12:47:17.713967+0000","last_undegraded":"2023-10-16T12:47:17.713967+0000","last_fullsized":"2023-10-16T12:47:17.713967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:10.491685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"66'6","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713645+0000","last_change":"2023-10-16T12:47:14.838723+0000","last_active":"2023-10-16T12:47:17.713645+0000","last_peered":"2023-10-16T12:47:17.713645+0000","last_clean":"2023-10-16T12:47:17.713645+0000","last_became_active":"2023-10-16T12:47:14.838443+0000","last_became_peered":"2023-10-16T12:47:14.838443+0000","last_unstale":"2023-10-16T12:47:17.713645+0000","last_undegraded":"2023-10-16T12:47:17.713645+0000","last_fullsized":"2023-10-16T12:47:17.713645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:54.044963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714710+0000","last_change":"2023-10-16T12:47:15.877442+0000","last_active":"2023-10-16T12:47:17.714710+0000","last_peered":"2023-10-16T12:47:17.714710+0000","last_clean":"2023-10-16T12:47:17.714710+0000","last_became_active":"2023-10-16T12:47:15.877148+0000","last_became_peered":"2023-10-16T12:47:15.877148+0000","last_unstale":"2023-10-16T12:47:17.714710+0000","last_undegraded":"2023-10-16T12:47:17.714710+0000","last_fullsized":"2023-10-16T12:47:17.714710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:38.259521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593582+0000","last_change":"2023-10-16T12:47:14.847585+0000","last_active":"2023-10-16T12:47:17.593582+0000","last_peered":"2023-10-16T12:47:17.593582+0000","last_clean":"2023-10-16T12:47:17.593582+0000","last_became_active":"2023-10-16T12:47:14.847271+0000","last_became_peered":"2023-10-16T12:47:14.847271+0000","last_unstale":"2023-10-16T12:47:17.593582+0000","last_undegraded":"2023-10-16T12:47:17.593582+0000","last_fullsized":"2023-10-16T12:47:17.593582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:23.377778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867564+0000","last_change":"2023-10-16T12:47:15.872695+0000","last_active":"2023-10-16T12:47:16.867564+0000","last_peered":"2023-10-16T12:47:16.867564+0000","last_clean":"2023-10-16T12:47:16.867564+0000","last_became_active":"2023-10-16T12:47:15.872433+0000","last_became_peered":"2023-10-16T12:47:15.872433+0000","last_unstale":"2023-10-16T12:47:16.867564+0000","last_undegraded":"2023-10-16T12:47:16.867564+0000","last_fullsized":"2023-10-16T12:47:16.867564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:06.040135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713852+0000","last_change":"2023-10-16T12:47:14.847528+0000","last_active":"2023-10-16T12:47:17.713852+0000","last_peered":"2023-10-16T12:47:17.713852+0000","last_clean":"2023-10-16T12:47:17.713852+0000","last_became_active":"2023-10-16T12:47:14.847258+0000","last_became_peered":"2023-10-16T12:47:14.847258+0000","last_unstale":"2023-10-16T12:47:17.713852+0000","last_undegraded":"2023-10-16T12:47:17.713852+0000","last_fullsized":"2023-10-16T12:47:17.713852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:47.998720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593109+0000","last_change":"2023-10-16T12:47:15.868946+0000","last_active":"2023-10-16T12:47:17.593109+0000","last_peered":"2023-10-16T12:47:17.593109+0000","last_clean":"2023-10-16T12:47:17.593109+0000","last_became_active":"2023-10-16T12:47:15.868683+0000","last_became_peered":"2023-10-16T12:47:15.868683+0000","last_unstale":"2023-10-16T12:47:17.593109+0000","last_undegraded":"2023-10-16T12:47:17.593109+0000","last_fullsized":"2023-10-16T12:47:17.593109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:04.492293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593367+0000","last_change":"2023-10-16T12:47:14.847569+0000","last_active":"2023-10-16T12:47:17.593367+0000","last_peered":"2023-10-16T12:47:17.593367+0000","last_clean":"2023-10-16T12:47:17.593367+0000","last_became_active":"2023-10-16T12:47:14.846019+0000","last_became_peered":"2023-10-16T12:47:14.846019+0000","last_unstale":"2023-10-16T12:47:17.593367+0000","last_undegraded":"2023-10-16T12:47:17.593367+0000","last_fullsized":"2023-10-16T12:47:17.593367+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.541165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714101+0000","last_change":"2023-10-16T12:47:15.880147+0000","last_active":"2023-10-16T12:47:17.714101+0000","last_peered":"2023-10-16T12:47:17.714101+0000","last_clean":"2023-10-16T12:47:17.714101+0000","last_became_active":"2023-10-16T12:47:15.879359+0000","last_became_peered":"2023-10-16T12:47:15.879359+0000","last_unstale":"2023-10-16T12:47:17.714101+0000","last_undegraded":"2023-10-16T12:47:17.714101+0000","last_fullsized":"2023-10-16T12:47:17.714101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:48.381657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"66'8","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867058+0000","last_change":"2023-10-16T12:47:14.845074+0000","last_active":"2023-10-16T12:47:16.867058+0000","last_peered":"2023-10-16T12:47:16.867058+0000","last_clean":"2023-10-16T12:47:16.867058+0000","last_became_active":"2023-10-16T12:47:14.844782+0000","last_became_peered":"2023-10-16T12:47:14.844782+0000","last_unstale":"2023-10-16T12:47:16.867058+0000","last_undegraded":"2023-10-16T12:47:16.867058+0000","last_fullsized":"2023-10-16T12:47:16.867058+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:22.162022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713205+0000","last_change":"2023-10-16T12:47:15.872442+0000","last_active":"2023-10-16T12:47:17.713205+0000","last_peered":"2023-10-16T12:47:17.713205+0000","last_clean":"2023-10-16T12:47:17.713205+0000","last_became_active":"2023-10-16T12:47:15.872131+0000","last_became_peered":"2023-10-16T12:47:15.872131+0000","last_unstale":"2023-10-16T12:47:17.713205+0000","last_undegraded":"2023-10-16T12:47:17.713205+0000","last_fullsized":"2023-10-16T12:47:17.713205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:03.173587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713710+0000","last_change":"2023-10-16T12:47:14.843998+0000","last_active":"2023-10-16T12:47:17.713710+0000","last_peered":"2023-10-16T12:47:17.713710+0000","last_clean":"2023-10-16T12:47:17.713710+0000","last_became_active":"2023-10-16T12:47:14.843708+0000","last_became_peered":"2023-10-16T12:47:14.843708+0000","last_unstale":"2023-10-16T12:47:17.713710+0000","last_undegraded":"2023-10-16T12:47:17.713710+0000","last_fullsized":"2023-10-16T12:47:17.713710+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:29.721083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868644+0000","last_change":"2023-10-16T12:47:15.865878+0000","last_active":"2023-10-16T12:47:16.868644+0000","last_peered":"2023-10-16T12:47:16.868644+0000","last_clean":"2023-10-16T12:47:16.868644+0000","last_became_active":"2023-10-16T12:47:15.865579+0000","last_became_peered":"2023-10-16T12:47:15.865579+0000","last_unstale":"2023-10-16T12:47:16.868644+0000","last_undegraded":"2023-10-16T12:47:16.868644+0000","last_fullsized":"2023-10-16T12:47:16.868644+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:05.787181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871613+0000","last_change":"2023-10-16T12:47:14.842060+0000","last_active":"2023-10-16T12:47:16.871613+0000","last_peered":"2023-10-16T12:47:16.871613+0000","last_clean":"2023-10-16T12:47:16.871613+0000","last_became_active":"2023-10-16T12:47:14.841745+0000","last_became_peered":"2023-10-16T12:47:14.841745+0000","last_unstale":"2023-10-16T12:47:16.871613+0000","last_undegraded":"2023-10-16T12:47:16.871613+0000","last_fullsized":"2023-10-16T12:47:16.871613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:12.877073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715530+0000","last_change":"2023-10-16T12:47:15.870462+0000","last_active":"2023-10-16T12:47:17.715530+0000","last_peered":"2023-10-16T12:47:17.715530+0000","last_clean":"2023-10-16T12:47:17.715530+0000","last_became_active":"2023-10-16T12:47:15.870163+0000","last_became_peered":"2023-10-16T12:47:15.870163+0000","last_unstale":"2023-10-16T12:47:17.715530+0000","last_undegraded":"2023-10-16T12:47:17.715530+0000","last_fullsized":"2023-10-16T12:47:17.715530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:33.095535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.873382+0000","last_change":"2023-10-16T12:47:14.835953+0000","last_active":"2023-10-16T12:47:16.873382+0000","last_peered":"2023-10-16T12:47:16.873382+0000","last_clean":"2023-10-16T12:47:16.873382+0000","last_became_active":"2023-10-16T12:47:14.835446+0000","last_became_peered":"2023-10-16T12:47:14.835446+0000","last_unstale":"2023-10-16T12:47:16.873382+0000","last_undegraded":"2023-10-16T12:47:16.873382+0000","last_fullsized":"2023-10-16T12:47:16.873382+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:18.877023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716183+0000","last_change":"2023-10-16T12:47:15.883125+0000","last_active":"2023-10-16T12:47:17.716183+0000","last_peered":"2023-10-16T12:47:17.716183+0000","last_clean":"2023-10-16T12:47:17.716183+0000","last_became_active":"2023-10-16T12:47:15.882713+0000","last_became_peered":"2023-10-16T12:47:15.882713+0000","last_unstale":"2023-10-16T12:47:17.716183+0000","last_undegraded":"2023-10-16T12:47:17.716183+0000","last_fullsized":"2023-10-16T12:47:17.716183+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:13.198864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594269+0000","last_change":"2023-10-16T12:47:14.851357+0000","last_active":"2023-10-16T12:47:17.594269+0000","last_peered":"2023-10-16T12:47:17.594269+0000","last_clean":"2023-10-16T12:47:17.594269+0000","last_became_active":"2023-10-16T12:47:14.850881+0000","last_became_peered":"2023-10-16T12:47:14.850881+0000","last_unstale":"2023-10-16T12:47:17.594269+0000","last_undegraded":"2023-10-16T12:47:17.594269+0000","last_fullsized":"2023-10-16T12:47:17.594269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:45.719271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593310+0000","last_change":"2023-10-16T12:47:15.865546+0000","last_active":"2023-10-16T12:47:17.593310+0000","last_peered":"2023-10-16T12:47:17.593310+0000","last_clean":"2023-10-16T12:47:17.593310+0000","last_became_active":"2023-10-16T12:47:15.864473+0000","last_became_peered":"2023-10-16T12:47:15.864473+0000","last_unstale":"2023-10-16T12:47:17.593310+0000","last_undegraded":"2023-10-16T12:47:17.593310+0000","last_fullsized":"2023-10-16T12:47:17.593310+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:05.268814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594044+0000","last_change":"2023-10-16T12:47:14.851477+0000","last_active":"2023-10-16T12:47:17.594044+0000","last_peered":"2023-10-16T12:47:17.594044+0000","last_clean":"2023-10-16T12:47:17.594044+0000","last_became_active":"2023-10-16T12:47:14.849170+0000","last_became_peered":"2023-10-16T12:47:14.849170+0000","last_unstale":"2023-10-16T12:47:17.594044+0000","last_undegraded":"2023-10-16T12:47:17.594044+0000","last_fullsized":"2023-10-16T12:47:17.594044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:29.957872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711609+0000","last_change":"2023-10-16T12:47:15.860752+0000","last_active":"2023-10-16T12:47:17.711609+0000","last_peered":"2023-10-16T12:47:17.711609+0000","last_clean":"2023-10-16T12:47:17.711609+0000","last_became_active":"2023-10-16T12:47:15.860447+0000","last_became_peered":"2023-10-16T12:47:15.860447+0000","last_unstale":"2023-10-16T12:47:17.711609+0000","last_undegraded":"2023-10-16T12:47:17.711609+0000","last_fullsized":"2023-10-16T12:47:17.711609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:27.260455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870869+0000","last_change":"2023-10-16T12:47:14.850253+0000","last_active":"2023-10-16T12:47:16.870869+0000","last_peered":"2023-10-16T12:47:16.870869+0000","last_clean":"2023-10-16T12:47:16.870869+0000","last_became_active":"2023-10-16T12:47:14.849991+0000","last_became_peered":"2023-10-16T12:47:14.849991+0000","last_unstale":"2023-10-16T12:47:16.870869+0000","last_undegraded":"2023-10-16T12:47:16.870869+0000","last_fullsized":"2023-10-16T12:47:16.870869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:22.570270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.712716+0000","last_change":"2023-10-16T12:47:15.868019+0000","last_active":"2023-10-16T12:47:17.712716+0000","last_peered":"2023-10-16T12:47:17.712716+0000","last_clean":"2023-10-16T12:47:17.712716+0000","last_became_active":"2023-10-16T12:47:15.867168+0000","last_became_peered":"2023-10-16T12:47:15.867168+0000","last_unstale":"2023-10-16T12:47:17.712716+0000","last_undegraded":"2023-10-16T12:47:17.712716+0000","last_fullsized":"2023-10-16T12:47:17.712716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:51.003724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.867233+0000","last_change":"2023-10-16T12:47:14.852006+0000","last_active":"2023-10-16T12:47:16.867233+0000","last_peered":"2023-10-16T12:47:16.867233+0000","last_clean":"2023-10-16T12:47:16.867233+0000","last_became_active":"2023-10-16T12:47:14.851187+0000","last_became_peered":"2023-10-16T12:47:14.851187+0000","last_unstale":"2023-10-16T12:47:16.867233+0000","last_undegraded":"2023-10-16T12:47:16.867233+0000","last_fullsized":"2023-10-16T12:47:16.867233+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:43:57.232158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866864+0000","last_change":"2023-10-16T12:47:15.870962+0000","last_active":"2023-10-16T12:47:16.866864+0000","last_peered":"2023-10-16T12:47:16.866864+0000","last_clean":"2023-10-16T12:47:16.866864+0000","last_became_active":"2023-10-16T12:47:15.870701+0000","last_became_peered":"2023-10-16T12:47:15.870701+0000","last_unstale":"2023-10-16T12:47:16.866864+0000","last_undegraded":"2023-10-16T12:47:16.866864+0000","last_fullsized":"2023-10-16T12:47:16.866864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:08.204380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.713301+0000","last_change":"2023-10-16T12:47:14.846747+0000","last_active":"2023-10-16T12:47:17.713301+0000","last_peered":"2023-10-16T12:47:17.713301+0000","last_clean":"2023-10-16T12:47:17.713301+0000","last_became_active":"2023-10-16T12:47:14.846430+0000","last_became_peered":"2023-10-16T12:47:14.846430+0000","last_unstale":"2023-10-16T12:47:17.713301+0000","last_undegraded":"2023-10-16T12:47:17.713301+0000","last_fullsized":"2023-10-16T12:47:17.713301+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:06.845364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.716510+0000","last_change":"2023-10-16T12:47:15.876390+0000","last_active":"2023-10-16T12:47:17.716510+0000","last_peered":"2023-10-16T12:47:17.716510+0000","last_clean":"2023-10-16T12:47:17.716510+0000","last_became_active":"2023-10-16T12:47:15.876040+0000","last_became_peered":"2023-10-16T12:47:15.876040+0000","last_unstale":"2023-10-16T12:47:17.716510+0000","last_undegraded":"2023-10-16T12:47:17.716510+0000","last_fullsized":"2023-10-16T12:47:17.716510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:43.059000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.594182+0000","last_change":"2023-10-16T12:47:14.839347+0000","last_active":"2023-10-16T12:47:17.594182+0000","last_peered":"2023-10-16T12:47:17.594182+0000","last_clean":"2023-10-16T12:47:17.594182+0000","last_became_active":"2023-10-16T12:47:14.839073+0000","last_became_peered":"2023-10-16T12:47:14.839073+0000","last_unstale":"2023-10-16T12:47:17.594182+0000","last_undegraded":"2023-10-16T12:47:17.594182+0000","last_fullsized":"2023-10-16T12:47:17.594182+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:28.769276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868014+0000","last_change":"2023-10-16T12:47:15.858197+0000","last_active":"2023-10-16T12:47:16.868014+0000","last_peered":"2023-10-16T12:47:16.868014+0000","last_clean":"2023-10-16T12:47:16.868014+0000","last_became_active":"2023-10-16T12:47:15.857884+0000","last_became_peered":"2023-10-16T12:47:15.857884+0000","last_unstale":"2023-10-16T12:47:16.868014+0000","last_undegraded":"2023-10-16T12:47:16.868014+0000","last_fullsized":"2023-10-16T12:47:16.868014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:32.745085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.715114+0000","last_change":"2023-10-16T12:47:14.855238+0000","last_active":"2023-10-16T12:47:17.715114+0000","last_peered":"2023-10-16T12:47:17.715114+0000","last_clean":"2023-10-16T12:47:17.715114+0000","last_became_active":"2023-10-16T12:47:14.854906+0000","last_became_peered":"2023-10-16T12:47:14.854906+0000","last_unstale":"2023-10-16T12:47:17.715114+0000","last_undegraded":"2023-10-16T12:47:17.715114+0000","last_fullsized":"2023-10-16T12:47:17.715114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:28.764239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714309+0000","last_change":"2023-10-16T12:47:15.873271+0000","last_active":"2023-10-16T12:47:17.714309+0000","last_peered":"2023-10-16T12:47:17.714309+0000","last_clean":"2023-10-16T12:47:17.714309+0000","last_became_active":"2023-10-16T12:47:15.873006+0000","last_became_peered":"2023-10-16T12:47:15.873006+0000","last_unstale":"2023-10-16T12:47:17.714309+0000","last_undegraded":"2023-10-16T12:47:17.714309+0000","last_fullsized":"2023-10-16T12:47:17.714309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:51:42.973193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711158+0000","last_change":"2023-10-16T12:47:14.845070+0000","last_active":"2023-10-16T12:47:17.711158+0000","last_peered":"2023-10-16T12:47:17.711158+0000","last_clean":"2023-10-16T12:47:17.711158+0000","last_became_active":"2023-10-16T12:47:14.844808+0000","last_became_peered":"2023-10-16T12:47:14.844808+0000","last_unstale":"2023-10-16T12:47:17.711158+0000","last_undegraded":"2023-10-16T12:47:17.711158+0000","last_fullsized":"2023-10-16T12:47:17.711158+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:40.081833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593588+0000","last_change":"2023-10-16T12:47:15.868310+0000","last_active":"2023-10-16T12:47:17.593588+0000","last_peered":"2023-10-16T12:47:17.593588+0000","last_clean":"2023-10-16T12:47:17.593588+0000","last_became_active":"2023-10-16T12:47:15.865995+0000","last_became_peered":"2023-10-16T12:47:15.865995+0000","last_unstale":"2023-10-16T12:47:17.593588+0000","last_undegraded":"2023-10-16T12:47:17.593588+0000","last_fullsized":"2023-10-16T12:47:17.593588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:29.779262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.714217+0000","last_change":"2023-10-16T12:47:14.853707+0000","last_active":"2023-10-16T12:47:17.714217+0000","last_peered":"2023-10-16T12:47:17.714217+0000","last_clean":"2023-10-16T12:47:17.714217+0000","last_became_active":"2023-10-16T12:47:14.853328+0000","last_became_peered":"2023-10-16T12:47:14.853328+0000","last_unstale":"2023-10-16T12:47:17.714217+0000","last_undegraded":"2023-10-16T12:47:17.714217+0000","last_fullsized":"2023-10-16T12:47:17.714217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:03.477270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.593411+0000","last_change":"2023-10-16T12:47:15.868092+0000","last_active":"2023-10-16T12:47:17.593411+0000","last_peered":"2023-10-16T12:47:17.593411+0000","last_clean":"2023-10-16T12:47:17.593411+0000","last_became_active":"2023-10-16T12:47:15.867676+0000","last_became_peered":"2023-10-16T12:47:15.867676+0000","last_unstale":"2023-10-16T12:47:17.593411+0000","last_undegraded":"2023-10-16T12:47:17.593411+0000","last_fullsized":"2023-10-16T12:47:17.593411+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:20.750577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.871182+0000","last_change":"2023-10-16T12:47:14.842781+0000","last_active":"2023-10-16T12:47:16.871182+0000","last_peered":"2023-10-16T12:47:16.871182+0000","last_clean":"2023-10-16T12:47:16.871182+0000","last_became_active":"2023-10-16T12:47:14.840875+0000","last_became_peered":"2023-10-16T12:47:14.840875+0000","last_unstale":"2023-10-16T12:47:16.871182+0000","last_undegraded":"2023-10-16T12:47:16.871182+0000","last_fullsized":"2023-10-16T12:47:16.871182+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:42.162801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870576+0000","last_change":"2023-10-16T12:47:15.865170+0000","last_active":"2023-10-16T12:47:16.870576+0000","last_peered":"2023-10-16T12:47:16.870576+0000","last_clean":"2023-10-16T12:47:16.870576+0000","last_became_active":"2023-10-16T12:47:15.864892+0000","last_became_peered":"2023-10-16T12:47:15.864892+0000","last_unstale":"2023-10-16T12:47:16.870576+0000","last_undegraded":"2023-10-16T12:47:16.870576+0000","last_fullsized":"2023-10-16T12:47:16.870576+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:35.792307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:17.711266+0000","last_change":"2023-10-16T12:47:14.856068+0000","last_active":"2023-10-16T12:47:17.711266+0000","last_peered":"2023-10-16T12:47:17.711266+0000","last_clean":"2023-10-16T12:47:17.711266+0000","last_became_active":"2023-10-16T12:47:14.855678+0000","last_became_peered":"2023-10-16T12:47:14.855678+0000","last_unstale":"2023-10-16T12:47:17.711266+0000","last_undegraded":"2023-10-16T12:47:17.711266+0000","last_fullsized":"2023-10-16T12:47:17.711266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:59.053243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.870265+0000","last_change":"2023-10-16T12:47:15.871600+0000","last_active":"2023-10-16T12:47:16.870265+0000","last_peered":"2023-10-16T12:47:16.870265+0000","last_clean":"2023-10-16T12:47:16.870265+0000","last_became_active":"2023-10-16T12:47:15.867470+0000","last_became_peered":"2023-10-16T12:47:15.867470+0000","last_unstale":"2023-10-16T12:47:16.870265+0000","last_undegraded":"2023-10-16T12:47:16.870265+0000","last_fullsized":"2023-10-16T12:47:16.870265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:02.574295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.866982+0000","last_change":"2023-10-16T12:47:14.843191+0000","last_active":"2023-10-16T12:47:16.866982+0000","last_peered":"2023-10-16T12:47:16.866982+0000","last_clean":"2023-10-16T12:47:16.866982+0000","last_became_active":"2023-10-16T12:47:14.842208+0000","last_became_peered":"2023-10-16T12:47:14.842208+0000","last_unstale":"2023-10-16T12:47:16.866982+0000","last_undegraded":"2023-10-16T12:47:16.866982+0000","last_fullsized":"2023-10-16T12:47:16.866982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:30.449917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.868444+0000","last_change":"2023-10-16T12:47:15.870312+0000","last_active":"2023-10-16T12:47:16.868444+0000","last_peered":"2023-10-16T12:47:16.868444+0000","last_clean":"2023-10-16T12:47:16.868444+0000","last_became_active":"2023-10-16T12:47:15.868896+0000","last_became_peered":"2023-10-16T12:47:15.868896+0000","last_unstale":"2023-10-16T12:47:16.868444+0000","last_undegraded":"2023-10-16T12:47:16.868444+0000","last_fullsized":"2023-10-16T12:47:16.868444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:14.808926+0000","last_clean_scrub_stamp":"2023-10-16T12:47:14.808926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:16.958321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"66'2","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:47:16.873775+0000","last_change":"2023-10-16T12:47:14.845137+0000","last_active":"2023-10-16T12:47:16.873775+0000","last_peered":"2023-10-16T12:47:16.873775+0000","last_clean":"2023-10-16T12:47:16.873775+0000","last_became_active":"2023-10-16T12:47:14.844845+0000","last_became_peered":"2023-10-16T12:47:14.844845+0000","last_unstale":"2023-10-16T12:47:16.873775+0000","last_undegraded":"2023-10-16T12:47:16.873775+0000","last_fullsized":"2023-10-16T12:47:16.873775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:47:13.793903+0000","last_clean_scrub_stamp":"2023-10-16T12:47:13.793903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:07.222854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"55'139","reported_seq":290,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T12:47:15.860723+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:47:15.860723+0000","last_peered":"2023-10-16T12:47:15.860723+0000","last_clean":"2023-10-16T12:47:15.860723+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:47:15.860723+0000","last_undegraded":"2023-10-16T12:47:15.860723+0000","last_fullsized":"2023-10-16T12:47:15.860723+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":956037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3260000000000001,"15min":1.282},"min":{"1min":0.91400000000000003,"5min":0.77800000000000002,"15min":0.65100000000000002},"max":{"1min":1.6599999999999999,"5min":1.7050000000000001,"15min":2.1890000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.325},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.746,"5min":1.8420000000000001,"15min":1.8899999999999999},"last":1.2989999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2250000000000001,"15min":1.214},"min":{"1min":0.85799999999999998,"5min":0.81200000000000006,"15min":0.67300000000000004},"max":{"1min":1.5309999999999999,"5min":1.637,"15min":1.752},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2250000000000001,"15min":1.1950000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.77100000000000002},"max":{"1min":1.577,"5min":1.708,"15min":1.843},"last":1.3779999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.222,"15min":1.2070000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.59799999999999998},"max":{"1min":1.591,"5min":1.657,"15min":2.2130000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.337,"15min":1.3400000000000001},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.63500000000000001},"max":{"1min":1.6659999999999999,"5min":1.837,"15min":2.3450000000000002},"last":1.133}]},{"osd":3,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.28,"15min":1.3009999999999999},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.69099999999999995},"max":{"1min":1.508,"5min":1.673,"15min":2.0150000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.391,"15min":1.391},"min":{"1min":1.2210000000000001,"5min":0.97099999999999997,"15min":0.85999999999999999},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":2.0019999999999998},"last":1.468}]},{"osd":4,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.1819999999999999,"15min":1.2030000000000001},"min":{"1min":0.82899999999999996,"5min":0.72099999999999997,"15min":0.67300000000000004},"max":{"1min":1.7030000000000001,"5min":1.744,"15min":1.9710000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.323,"15min":1.351},"min":{"1min":0.96699999999999997,"5min":0.73099999999999998,"15min":0.56499999999999995},"max":{"1min":1.736,"5min":1.748,"15min":2.282},"last":1.169}]},{"osd":5,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3180000000000001,"15min":1.3400000000000001},"min":{"1min":1.0720000000000001,"5min":0.89300000000000002,"15min":0.78900000000000003},"max":{"1min":1.627,"5min":1.671,"15min":2.0529999999999999},"last":1.115},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.4330000000000001,"15min":1.482},"min":{"1min":1.242,"5min":0.98899999999999999,"15min":0.61299999999999999},"max":{"1min":1.768,"5min":1.8169999999999999,"15min":2.2000000000000002},"last":1.5349999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.548,"15min":1.5700000000000001},"min":{"1min":1.1419999999999999,"5min":1.026,"15min":0.72699999999999998},"max":{"1min":1.833,"5min":1.865,"15min":2.2639999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.419,"15min":1.4470000000000001},"min":{"1min":1.298,"5min":0.89100000000000001,"15min":0.83399999999999996},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.1379999999999999},"last":1.2150000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":790528,"data_stored":366251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.325,"15min":1.282},"min":{"1min":0.749,"5min":0.749,"15min":0.71499999999999997},"max":{"1min":1.5840000000000001,"5min":1.657,"15min":3.7450000000000001},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.343,"15min":1.3109999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.57999999999999996},"max":{"1min":1.528,"5min":1.6819999999999999,"15min":3.819},"last":1.536}]},{"osd":1,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2050000000000001,"15min":1.1579999999999999},"min":{"1min":0.86899999999999999,"5min":0.82899999999999996,"15min":0.621},"max":{"1min":1.5329999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.216,"15min":1.218},"min":{"1min":0.86199999999999999,"5min":0.72199999999999998,"15min":0.66800000000000004},"max":{"1min":1.7110000000000001,"5min":2.0459999999999998,"15min":3.8519999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.236,"15min":1.21},"min":{"1min":0.93000000000000005,"5min":0.83999999999999997,"15min":0.68300000000000005},"max":{"1min":1.663,"5min":1.663,"15min":1.762},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.272,"15min":1.2749999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.755},"max":{"1min":1.516,"5min":1.9970000000000001,"15min":3.9009999999999998},"last":1.2270000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3260000000000001,"15min":1.3440000000000001},"min":{"1min":0.93899999999999995,"5min":0.92400000000000004,"15min":0.751},"max":{"1min":1.867,"5min":2.3900000000000001,"15min":4.2380000000000004},"last":1.492},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.3280000000000001,"15min":1.3560000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.012,"5min":2.012,"15min":4.4269999999999996},"last":1.087}]},{"osd":4,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.3400000000000001,"15min":1.359},"min":{"1min":0.89000000000000001,"5min":0.78000000000000003,"15min":0.57699999999999996},"max":{"1min":2.0830000000000002,"5min":2.3029999999999999,"15min":2.4100000000000001},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4419999999999999,"15min":1.49},"min":{"1min":1.127,"5min":0.82099999999999995,"15min":0.67300000000000004},"max":{"1min":1.764,"5min":1.7709999999999999,"15min":2.3050000000000002},"last":1.3460000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.3009999999999999,"15min":1.3169999999999999},"min":{"1min":0.872,"5min":0.79600000000000004,"15min":0.63900000000000001},"max":{"1min":1.716,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.425,"5min":1.4079999999999999,"15min":1.476},"min":{"1min":1.1000000000000001,"5min":0.92000000000000004,"15min":0.63900000000000001},"max":{"1min":1.9570000000000001,"5min":2.1699999999999999,"15min":2.5230000000000001},"last":1.27}]},{"osd":7,"last update":"Mon Oct 16 12:46:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3660000000000001,"15min":1.4490000000000001},"min":{"1min":0.97899999999999998,"5min":0.96299999999999997,"15min":0.495},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4470000000000001,"15min":1.474},"min":{"1min":0.98699999999999999,"5min":0.96399999999999997,"15min":0.59099999999999997},"max":{"1min":1.825,"5min":2.2610000000000001,"15min":2.2719999999999998},"last":1.3979999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":371949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1819999999999999,"15min":1.1519999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1639999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.67400000000000004},"max":{"1min":1.641,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":1.0569999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.222,"15min":1.2470000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.639,"5min":1.639,"15min":2.415},"last":1.169},{"interface":"front","average":{"1min":1.123,"5min":1.226,"15min":1.226},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.452,"5min":1.6899999999999999,"15min":1.9419999999999999},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2709999999999999,"15min":1.294},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.74199999999999999},"max":{"1min":1.5840000000000001,"5min":1.6519999999999999,"15min":2.0259999999999998},"last":1.669},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2829999999999999,"15min":1.258},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.123}]},{"osd":3,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2969999999999999,"15min":1.3109999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":2.2690000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.341,"5min":1.351,"15min":1.3720000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.361}]},{"osd":4,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.21,"15min":1.2150000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.996,"5min":1.996,"15min":2.7909999999999999},"last":1.006},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3169999999999999,"15min":1.361},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.806,"5min":1.806,"15min":5.3250000000000002},"last":1.296}]},{"osd":6,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3500000000000001,"15min":1.345},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.927,"5min":1.927,"15min":2.343},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.395,"15min":1.383},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.6520000000000001},"last":1.323}]},{"osd":7,"last update":"Mon Oct 16 12:46:43 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4610000000000001,"15min":1.4870000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.889,"5min":1.889,"15min":5.1269999999999998},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.5189999999999999,"15min":1.5289999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.4929999999999999},"last":1.615}]}]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58336,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313504,"statfs":{"total":96636764160,"available":96577028096,"internally_reserved":0,"allocated":806912,"data_stored":373491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2410000000000001,"15min":1.1870000000000001},"min":{"1min":0.82299999999999995,"5min":0.80600000000000005,"15min":0.64500000000000002},"max":{"1min":1.665,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.159,"15min":1.147},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.623},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.9159999999999999},"last":1.5069999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.194,"15min":1.2050000000000001},"min":{"1min":0.84099999999999997,"5min":0.80000000000000004,"15min":0.68799999999999994},"max":{"1min":1.542,"5min":1.8400000000000001,"15min":1.865},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.266,"15min":1.28},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.68799999999999994},"max":{"1min":1.6359999999999999,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.681}]},{"osd":2,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.232},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.753},"max":{"1min":1.7370000000000001,"5min":1.865,"15min":1.865},"last":1.115},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.363,"15min":1.3580000000000001},"min":{"1min":0.92200000000000004,"5min":0.91800000000000004,"15min":0.73399999999999999},"max":{"1min":1.502,"5min":1.758,"15min":1.9910000000000001},"last":1.6240000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2709999999999999,"15min":1.2589999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.752},"max":{"1min":1.7130000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.383,"15min":1.3740000000000001},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.76500000000000001},"max":{"1min":1.819,"5min":1.956,"15min":1.956},"last":1.4039999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2689999999999999,"15min":1.391},"min":{"1min":1.0129999999999999,"5min":0.87,"15min":0.50900000000000001},"max":{"1min":1.8260000000000001,"5min":2.0339999999999998,"15min":7.9000000000000004},"last":1.319},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.288,"15min":1.369},"min":{"1min":0.81999999999999995,"5min":0.81399999999999995,"15min":0.53600000000000003},"max":{"1min":1.913,"5min":1.913,"15min":7.3460000000000001},"last":1.1739999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.91700000000000004,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":1.718,"5min":1.788,"15min":2.0139999999999998},"last":1.248},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.472,"15min":1.504},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.65300000000000002},"max":{"1min":2.0139999999999998,"5min":2.1339999999999999,"15min":2.6269999999999998},"last":1.5509999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.4530000000000001,"15min":1.476},"min":{"1min":1.1220000000000001,"5min":0.88400000000000001,"15min":0.54200000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.254},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.5609999999999999,"15min":1.5980000000000001},"min":{"1min":1.1419999999999999,"5min":0.94799999999999995,"15min":0.69199999999999995},"max":{"1min":1.966,"5min":2.1699999999999999,"15min":2.5289999999999999},"last":1.478}]}]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40276,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331564,"statfs":{"total":96636764160,"available":96595521536,"internally_reserved":0,"allocated":794624,"data_stored":366217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.95999999999999996,"15min":0.95099999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.3260000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.206,"15min":1.1599999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5960000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.4079999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0329999999999999,"15min":1.026},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.39500000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.8149999999999999},"last":1.137},{"interface":"front","average":{"1min":1.014,"5min":1.0920000000000001,"15min":1.0820000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.45600000000000002},"max":{"1min":1.4690000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.79900000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0980000000000001,"15min":1.0660000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.712,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.3759999999999999,"15min":1.349},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.58299999999999996},"max":{"1min":1.8600000000000001,"5min":1.921,"15min":2.0569999999999999},"last":1.3069999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.5,"15min":1.476},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.77600000000000002},"max":{"1min":1.788,"5min":1.956,"15min":2.2730000000000001},"last":1.569},{"interface":"front","average":{"1min":1.383,"5min":1.4370000000000001,"15min":1.4279999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.63500000000000001},"max":{"1min":1.821,"5min":1.859,"15min":3.6930000000000001},"last":1.671}]},{"osd":5,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.60699999999999998},"max":{"1min":1.8600000000000001,"5min":1.8779999999999999,"15min":3.8300000000000001},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4630000000000001,"15min":1.4570000000000001},"min":{"1min":0.98899999999999999,"5min":0.95399999999999996,"15min":0.64900000000000002},"max":{"1min":1.9990000000000001,"5min":2.1030000000000002,"15min":2.3540000000000001},"last":1.8340000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.468,"15min":1.4550000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.65300000000000002},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.415},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.4359999999999999,"15min":1.4370000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.238,"5min":2.3740000000000001,"15min":3.9809999999999999},"last":1.458}]},{"osd":7,"last update":"Mon Oct 16 12:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5549999999999999,"15min":1.5389999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.80300000000000005},"max":{"1min":2.1099999999999999,"5min":2.198,"15min":2.431},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.657,"15min":1.6060000000000001},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.95099999999999996},"max":{"1min":2.0649999999999999,"5min":2.552,"15min":2.552},"last":1.5129999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40280,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331560,"statfs":{"total":96636764160,"available":96595517440,"internally_reserved":0,"allocated":798720,"data_stored":367213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.94699999999999995,"15min":0.93799999999999994},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":1.742,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1120000000000001,"15min":1.081},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.5},"max":{"1min":1.5529999999999999,"5min":1.946,"15min":2.085},"last":0.80400000000000005}]},{"osd":1,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.145,"15min":1.107},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.50900000000000001},"max":{"1min":1.8260000000000001,"5min":1.851,"15min":1.9330000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.22,"5min":1.268,"15min":1.2050000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45500000000000002},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.4289999999999998},"last":1.5680000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1060000000000001,"15min":1.0620000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6879999999999999,"5min":1.859,"15min":1.859},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.337,"15min":1.2929999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.59199999999999997},"max":{"1min":1.831,"5min":1.831,"15min":2.1970000000000001},"last":0.96099999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4890000000000001,"15min":1.4470000000000001},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.89000000000000001},"max":{"1min":1.958,"5min":2.3159999999999998,"15min":2.383},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.458,"15min":1.446},"min":{"1min":1.014,"5min":1.014,"15min":0.85899999999999999},"max":{"1min":1.9390000000000001,"5min":2.2530000000000001,"15min":2.294},"last":1.2410000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5329999999999999,"15min":1.482},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.83499999999999996},"max":{"1min":2.23,"5min":2.23,"15min":2.3340000000000001},"last":1.347},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4510000000000001,"15min":1.427},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.502}]},{"osd":6,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.456,"15min":1.4610000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.85999999999999999},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.5820000000000001,"15min":1.5649999999999999},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":0.95799999999999996},"max":{"1min":2.294,"5min":2.3279999999999998,"15min":2.488},"last":1.2170000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:46:39 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.6339999999999999,"15min":1.599},"min":{"1min":1.113,"5min":1.113,"15min":0.96699999999999997},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5469999999999999,"15min":1.5589999999999999},"min":{"1min":1.014,"5min":1.014,"15min":0.90400000000000003},"max":{"1min":2.573,"5min":2.573,"15min":2.6429999999999998},"last":1.3839999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54172,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317668,"statfs":{"total":96636764160,"available":96581292032,"internally_reserved":0,"allocated":798720,"data_stored":371949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.91300000000000003,"15min":0.89700000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.435},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.02,"5min":1.103,"15min":1.085},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":1.456,"5min":1.6040000000000001,"15min":1.7669999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.01,"15min":0.98899999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8460000000000001,"5min":2.262,"15min":2.262},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2250000000000001,"15min":1.1799999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.90700000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.325,"15min":1.2949999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.677,"5min":1.907,"15min":2.2349999999999999},"last":1.018},{"interface":"front","average":{"1min":1.24,"5min":1.2310000000000001,"15min":1.204},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.46800000000000003},"max":{"1min":1.617,"5min":1.883,"15min":2.1789999999999998},"last":0.79800000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.304,"15min":1.3},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.788,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":1.482},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.448,"15min":1.45},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9750000000000001,"5min":2.0649999999999999,"15min":3.0659999999999998},"last":1.2729999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.4239999999999999,"15min":1.4379999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.754,"5min":2.0070000000000001,"15min":2.0499999999999998},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.4430000000000001,"15min":1.4059999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":1.9930000000000001,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.6839999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.542,"15min":1.54},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.839,"5min":2.1899999999999999,"15min":2.9350000000000001},"last":1.206},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.526,"15min":1.5269999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.8859999999999999,"5min":2.246,"15min":2.246},"last":1.2490000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:46:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.637,"15min":1.623},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.948,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.554,"15min":1.5549999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.921,"5min":2.2370000000000001,"15min":2.323},"last":1.613}]}]},{"osd":0,"up_from":14,"seq":60129542314,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1392640,"data_stored":962281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.879,"15min":0.90200000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.274,"5min":1.573,"15min":2.5720000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.121,"15min":1.0629999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.32700000000000001},"max":{"1min":1.6870000000000001,"5min":1.7190000000000001,"15min":2.1800000000000002},"last":1.1850000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.097,"15min":1.111},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.482,"5min":1.7210000000000001,"15min":2.7650000000000001},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1950000000000001,"15min":1.143},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.438},"max":{"1min":1.952,"5min":1.952,"15min":1.952},"last":1.52}]},{"osd":3,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1359999999999999,"15min":1.0580000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.315},"max":{"1min":1.575,"5min":1.9319999999999999,"15min":3.0339999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.276,"15min":1.286},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.45700000000000002},"max":{"1min":1.6160000000000001,"5min":1.794,"15min":3.6579999999999999},"last":1.3720000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.353,"15min":1.359},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8069999999999999,"5min":1.9950000000000001,"15min":4.726},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.411,"15min":1.4590000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.756,"5min":1.9490000000000001,"15min":4.843},"last":1.966}]},{"osd":5,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.4710000000000001,"15min":1.4039999999999999},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.2229999999999999,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":2.024},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.4279999999999999,"15min":1.4470000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.02,"5min":2.0270000000000001,"15min":4.9059999999999997},"last":1.427}]},{"osd":6,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5209999999999999,"15min":1.522},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.73299999999999998},"max":{"1min":2.0009999999999999,"5min":2.3519999999999999,"15min":3.7890000000000001},"last":2.0750000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.573,"15min":1.506},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.76200000000000001},"max":{"1min":2.145,"5min":2.363,"15min":2.6589999999999998},"last":1.8919999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:46:48 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.6739999999999999,"15min":1.5980000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.82599999999999996},"max":{"1min":2.3490000000000002,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":2.1080000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.5720000000000001,"15min":1.5660000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":4.5890000000000004},"last":1.6639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:47:20.890 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:47:20.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:47:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.171+0000 7f4be33fb640 1 Processor -- start 2023-10-16T12:47:21.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.171+0000 7f4be33fb640 1 -- start start 2023-10-16T12:47:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.172+0000 7f4be33fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bdc0b1690 0x7f4bdc0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.172+0000 7f4be33fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0b4f00 con 0x7f4bdc0b4ab0 2023-10-16T12:47:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be33fb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0cdbf0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be33fb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cde90 con 0x7f4bdc0b1690 2023-10-16T12:47:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be1bf8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bdc0b1690 0x7f4bdc0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be1bf8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bdc0b1690 0x7f4bdc0b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34140/0 (socket says 172.21.15.60:34140) 2023-10-16T12:47:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be1bf8640 1 -- 172.21.15.60:0/2526685350 learned_addr learned my addr 172.21.15.60:0/2526685350 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:21.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be1bf8640 1 -- 172.21.15.60:0/2526685350 >> v1:172.21.15.144:6789/0 conn(0x7f4bdc15cb50 legacy=0x7f4bdc161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:21.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be1bf8640 1 -- 172.21.15.60:0/2526685350 >> v1:172.21.15.60:6789/0 conn(0x7f4bdc0b4ab0 legacy=0x7f4bdc0d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.173+0000 7f4be1bf8640 1 -- 172.21.15.60:0/2526685350 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0ce170 con 0x7f4bdc0b1690 2023-10-16T12:47:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.174+0000 7f4be1bf8640 1 --2- 172.21.15.60:0/2526685350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc0b1690 0x7f4bdc0b4570 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009870 tx=0x7f4bcc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcddeab2a77a68b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.175+0000 7f4be13f7640 1 -- 172.21.15.60:0/2526685350 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0317e0 con 0x7f4bdc0b1690 2023-10-16T12:47:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.175+0000 7f4be13f7640 1 -- 172.21.15.60:0/2526685350 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc031940 con 0x7f4bdc0b1690 2023-10-16T12:47:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.175+0000 7f4be13f7640 1 -- 172.21.15.60:0/2526685350 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc031cd0 con 0x7f4bdc0b1690 2023-10-16T12:47:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.175+0000 7f4be33fb640 1 -- 172.21.15.60:0/2526685350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc0b1690 msgr2=0x7f4bdc0b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.175+0000 7f4be33fb640 1 --2- 172.21.15.60:0/2526685350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc0b1690 0x7f4bdc0b4570 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009870 tx=0x7f4bcc02b880 comp rx=0 tx=0).stop 2023-10-16T12:47:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.176+0000 7f4be33fb640 1 -- 172.21.15.60:0/2526685350 shutdown_connections 2023-10-16T12:47:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.176+0000 7f4be33fb640 1 --2- 172.21.15.60:0/2526685350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc0b1690 0x7f4bdc0b4570 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.176+0000 7f4be33fb640 1 -- 172.21.15.60:0/2526685350 >> 172.21.15.60:0/2526685350 conn(0x7f4bdc0cf570 msgr2=0x7f4bdc0cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:21.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.176+0000 7f4be33fb640 1 -- 172.21.15.60:0/2526685350 shutdown_connections 2023-10-16T12:47:21.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.176+0000 7f4be33fb640 1 -- 172.21.15.60:0/2526685350 wait complete. 2023-10-16T12:47:21.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.177+0000 7f4be33fb640 1 Processor -- start 2023-10-16T12:47:21.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.178+0000 7f4be33fb640 1 -- start start 2023-10-16T12:47:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be33fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bdc0b1690 0x7f4bdc152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be33fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bdc0b4ab0 0x7f4bdc0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be33fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 0x7f4bdc0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be33fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc15dcb0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be33fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc15df90 con 0x7f4bdc0b4ab0 2023-10-16T12:47:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be33fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc1508d0 con 0x7f4bdc0b1690 2023-10-16T12:47:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 0x7f4bdc0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 0x7f4bdc0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34156/0 (socket says 172.21.15.60:34156) 2023-10-16T12:47:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 -- 172.21.15.60:0/1466694917 learned_addr learned my addr 172.21.15.60:0/1466694917 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be1bf8640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bdc0b4ab0 0x7f4bdc0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be23f9640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bdc0b1690 0x7f4bdc152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 -- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bdc0b1690 msgr2=0x7f4bdc152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bdc0b1690 0x7f4bdc152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 -- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bdc0b4ab0 msgr2=0x7f4bdc0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.179+0000 7f4be2bfa640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bdc0b4ab0 0x7f4bdc0cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4be2bfa640 1 -- 172.21.15.60:0/1466694917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc0094d0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4be23f9640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bdc0b1690 0x7f4bdc152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4be2bfa640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 0x7f4bdc0c74a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400c3c0 tx=0x7f4bd400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40052d0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd4005450 con 0x7f4bdc15cb50 2023-10-16T12:47:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4003880 con 0x7f4bdc15cb50 2023-10-16T12:47:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc150bb0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.180+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc0c79e0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.182+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc04e9b0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.183+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4bd40039e0 con 0x7f4bdc15cb50 2023-10-16T12:47:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.183+0000 7f4bd2ffd640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4bc006af90 0x7f4bc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.184+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bd4088c10 con 0x7f4bdc15cb50 2023-10-16T12:47:21.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.184+0000 7f4be23f9640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4bc006af90 0x7f4bc006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.185+0000 7f4be23f9640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4bc006af90 0x7f4bc006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8007e90 tx=0x7f4bd80073a0 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:21.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.188+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bd4053b10 con 0x7f4bdc15cb50 2023-10-16T12:47:21.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.455+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4bdc0b4f00 con 0x7f4bdc15cb50 2023-10-16T12:47:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.460+0000 7f4bd2ffd640 1 -- 172.21.15.60:0/1466694917 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4bdc0b4f00 con 0x7f4bdc15cb50 2023-10-16T12:47:21.462 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:47:21.462 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:47:21.463 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:47:21.463 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:47:21.463 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:47:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:47:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4bc006af90 msgr2=0x7f4bc006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4bc006af90 0x7f4bc006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8007e90 tx=0x7f4bd80073a0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 msgr2=0x7f4bdc0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 0x7f4bdc0c74a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400c3c0 tx=0x7f4bd400c880 comp rx=0 tx=0).stop 2023-10-16T12:47:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 shutdown_connections 2023-10-16T12:47:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f4bc006af90 0x7f4bc006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bdc15cb50 0x7f4bdc0c74a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bdc0b4ab0 0x7f4bdc0cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 --2- 172.21.15.60:0/1466694917 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bdc0b1690 0x7f4bdc152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 >> 172.21.15.60:0/1466694917 conn(0x7f4bdc0cf570 msgr2=0x7f4bdc0d3310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.463+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 shutdown_connections 2023-10-16T12:47:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.464+0000 7f4be33fb640 1 -- 172.21.15.60:0/1466694917 wait complete. 2023-10-16T12:47:21.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:47:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.694+0000 7fbea087b640 1 Processor -- start 2023-10-16T12:47:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.695+0000 7fbea087b640 1 -- start start 2023-10-16T12:47:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.696+0000 7fbea087b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbe9c15c9f0 0x7fbe9c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.696+0000 7fbea087b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbe9c0c9230 0x7fbe9c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.696+0000 7fbea087b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c160a90 con 0x7fbe9c070f20 2023-10-16T12:47:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.696+0000 7fbea087b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0cdec0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.696+0000 7fbea087b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0cd900 con 0x7fbe9c0c9230 2023-10-16T12:47:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.697+0000 7fbe9ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbe9c15c9f0 0x7fbe9c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.697+0000 7fbe9ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbe9c15c9f0 0x7fbe9c0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34170/0 (socket says 172.21.15.60:34170) 2023-10-16T12:47:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.697+0000 7fbe9ad9d640 1 -- 172.21.15.60:0/1958889833 learned_addr learned my addr 172.21.15.60:0/1958889833 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.697+0000 7fbe9a59c640 1 --2- 172.21.15.60:0/1958889833 >> v2:172.21.15.144:3300/0 conn(0x7fbe9c0c9230 0x7fbe9c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.697+0000 7fbe9ad9d640 1 -- 172.21.15.60:0/1958889833 >> v2:172.21.15.144:3300/0 conn(0x7fbe9c0c9230 msgr2=0x7fbe9c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.697+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/1958889833 >> v2:172.21.15.144:3300/0 conn(0x7fbe9c0c9230 0x7fbe9c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.698+0000 7fbe9ad9d640 1 -- 172.21.15.60:0/1958889833 >> v1:172.21.15.60:6789/0 conn(0x7fbe9c070f20 legacy=0x7fbe9c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.698+0000 7fbe9ad9d640 1 -- 172.21.15.60:0/1958889833 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0cdbe0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.698+0000 7fbe99d9b640 1 -- 172.21.15.60:0/1958889833 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514179496 0 0) 0x7fbe9c160a90 con 0x7fbe9c070f20 2023-10-16T12:47:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.699+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/1958889833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c0c8cf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0b1260 tx=0x7fbe8c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddcc7997bf79bee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.699+0000 7fbe99d9b640 1 -- 172.21.15.60:0/1958889833 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c02cb20 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.699+0000 7fbe99d9b640 1 -- 172.21.15.60:0/1958889833 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c02cc80 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.699+0000 7fbe99d9b640 1 -- 172.21.15.60:0/1958889833 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0343f0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 -- 172.21.15.60:0/1958889833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 msgr2=0x7fbe9c0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 --2- 172.21.15.60:0/1958889833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c0c8cf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0b1260 tx=0x7fbe8c02bf80 comp rx=0 tx=0).stop 2023-10-16T12:47:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 -- 172.21.15.60:0/1958889833 shutdown_connections 2023-10-16T12:47:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 --2- 172.21.15.60:0/1958889833 >> v2:172.21.15.144:3300/0 conn(0x7fbe9c0c9230 0x7fbe9c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 --2- 172.21.15.60:0/1958889833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c0c8cf0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 -- 172.21.15.60:0/1958889833 >> 172.21.15.60:0/1958889833 conn(0x7fbe9c0ce6a0 msgr2=0x7fbe9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 -- 172.21.15.60:0/1958889833 shutdown_connections 2023-10-16T12:47:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.700+0000 7fbea087b640 1 -- 172.21.15.60:0/1958889833 wait complete. 2023-10-16T12:47:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.702+0000 7fbea087b640 1 Processor -- start 2023-10-16T12:47:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.702+0000 7fbea087b640 1 -- start start 2023-10-16T12:47:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbea087b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe9c070f20 0x7fbe9c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbea087b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe9c0c9230 0x7fbe9c1f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbea087b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c1f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbea087b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c6da0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbea087b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c21d0 con 0x7fbe9c070f20 2023-10-16T12:47:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbea087b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c15f750 con 0x7fbe9c0c9230 2023-10-16T12:47:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbe9b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c1f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbe9b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c1f62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34180/0 (socket says 172.21.15.60:34180) 2023-10-16T12:47:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.703+0000 7fbe9b59e640 1 -- 172.21.15.60:0/1830701057 learned_addr learned my addr 172.21.15.60:0/1830701057 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe9c070f20 0x7fbe9c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9a59c640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe9c0c9230 0x7fbe9c1f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9b59e640 1 -- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe9c0c9230 msgr2=0x7fbe9c1f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9b59e640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe9c0c9230 0x7fbe9c1f5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9b59e640 1 -- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe9c070f20 msgr2=0x7fbe9c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9b59e640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe9c070f20 0x7fbe9c15ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9b59e640 1 -- 172.21.15.60:0/1830701057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c009b30 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe9b59e640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c1f62f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c430 tx=0x7fbe9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe940035f0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.704+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe94003770 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.705+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c15fa30 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.705+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94003b20 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.705+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c15fdc0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.706+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c04e9b0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.707+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe94003c80 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.708+0000 7fbe837fe640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbe8806af90 0x7fbe8806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.708+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe94088e60 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.708+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbe8806af90 0x7fbe8806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.710+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbe8806af90 0x7fbe8806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c035000 tx=0x7fbe8c030000 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.712+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe94053d60 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.803 INFO:tasks.ceph.mds.d.smithi144.stderr:2023-10-16T12:47:21.802+0000 7f24d4d55640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:47:21.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.950+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe9c0adac0 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.952+0000 7fbe837fe640 1 -- 172.21.15.60:0/1830701057 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fbe94017020 con 0x7fbe9c15c9f0 2023-10-16T12:47:21.953 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:21.954 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:47:15.298066+0000","modified":"2023-10-16T12:47:18.376045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/528818860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":528818860},{"type":"v1","addr":"172.21.15.144:6837","nonce":528818860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:47:21.954 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2023-10-16T12:47:21.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbe8806af90 msgr2=0x7fbe8806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbe8806af90 0x7fbe8806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c035000 tx=0x7fbe8c030000 comp rx=0 tx=0).stop 2023-10-16T12:47:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 msgr2=0x7fbe9c1f62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c1f62f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c430 tx=0x7fbe9400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 shutdown_connections 2023-10-16T12:47:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7fbe8806af90 0x7fbe8806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe9c0c9230 0x7fbe9c1f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c15c9f0 0x7fbe9c1f62f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 --2- 172.21.15.60:0/1830701057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe9c070f20 0x7fbe9c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 >> 172.21.15.60:0/1830701057 conn(0x7fbe9c0ce6a0 msgr2=0x7fbe9c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:21.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 shutdown_connections 2023-10-16T12:47:21.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:21.955+0000 7fbea087b640 1 -- 172.21.15.60:0/1830701057 wait complete. 2023-10-16T12:47:22.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:47:15.298066+0000', 'modified': '2023-10-16T12:47:18.376045+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5874': {'gid': 5874, 'name': 'd', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.144:6837/528818860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6836', 'nonce': 528818860}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 528818860}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:47:22.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:47:22.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T12:47:22.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.618+0000 7f44ab53c640 1 Processor -- start 2023-10-16T12:47:22.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.618+0000 7f44ab53c640 1 -- start start 2023-10-16T12:47:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44ab53c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44a415b320 0x7f44a40cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44ab53c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a415b9d0 con 0x7f44a406c040 2023-10-16T12:47:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44ab53c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a415bcb0 con 0x7f44a40ad970 2023-10-16T12:47:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44ab53c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a415b7d0 con 0x7f44a415b320 2023-10-16T12:47:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44aad3b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44a415b320 0x7f44a40cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44aad3b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44a415b320 0x7f44a40cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34190/0 (socket says 172.21.15.60:34190) 2023-10-16T12:47:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.620+0000 7f44aad3b640 1 -- 172.21.15.60:0/1977598360 learned_addr learned my addr 172.21.15.60:0/1977598360 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.621+0000 7f44aad3b640 1 -- 172.21.15.60:0/1977598360 >> v1:172.21.15.144:6789/0 conn(0x7f44a40ad970 legacy=0x7f44a40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.621+0000 7f44aad3b640 1 -- 172.21.15.60:0/1977598360 >> v1:172.21.15.60:6789/0 conn(0x7f44a406c040 legacy=0x7f44a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.621+0000 7f44aad3b640 1 -- 172.21.15.60:0/1977598360 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a415bf90 con 0x7f44a415b320 2023-10-16T12:47:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.622+0000 7f44aad3b640 1 --2- 172.21.15.60:0/1977598360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a415b320 0x7f44a40cd180 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f449c009e80 tx=0x7f449c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43af79c7dc0b9631 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.622+0000 7f44a9538640 1 -- 172.21.15.60:0/1977598360 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c02cb90 con 0x7f44a415b320 2023-10-16T12:47:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.622+0000 7f44a9538640 1 -- 172.21.15.60:0/1977598360 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c02ccf0 con 0x7f44a415b320 2023-10-16T12:47:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.622+0000 7f44a9538640 1 -- 172.21.15.60:0/1977598360 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c034610 con 0x7f44a415b320 2023-10-16T12:47:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 -- 172.21.15.60:0/1977598360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a415b320 msgr2=0x7f44a40cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 --2- 172.21.15.60:0/1977598360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a415b320 0x7f44a40cd180 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f449c009e80 tx=0x7f449c02bff0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 -- 172.21.15.60:0/1977598360 shutdown_connections 2023-10-16T12:47:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 --2- 172.21.15.60:0/1977598360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a415b320 0x7f44a40cd180 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 -- 172.21.15.60:0/1977598360 >> 172.21.15.60:0/1977598360 conn(0x7f44a40ce6a0 msgr2=0x7f44a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 -- 172.21.15.60:0/1977598360 shutdown_connections 2023-10-16T12:47:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.623+0000 7f44ab53c640 1 -- 172.21.15.60:0/1977598360 wait complete. 2023-10-16T12:47:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.625+0000 7f44ab53c640 1 Processor -- start 2023-10-16T12:47:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.625+0000 7f44ab53c640 1 -- start start 2023-10-16T12:47:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.626+0000 7f44ab53c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 0x7f44a40ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.626+0000 7f44ab53c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44a40ad970 0x7f44a414c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.626+0000 7f44ab53c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44a415b320 0x7f44a414ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.626+0000 7f44ab53c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40ca6e0 con 0x7f44a406c040 2023-10-16T12:47:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.626+0000 7f44ab53c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40b1100 con 0x7f44a40ad970 2023-10-16T12:47:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.626+0000 7f44ab53c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40b1290 con 0x7f44a415b320 2023-10-16T12:47:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 0x7f44a40ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 0x7f44a40ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34196/0 (socket says 172.21.15.60:34196) 2023-10-16T12:47:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 -- 172.21.15.60:0/3058266015 learned_addr learned my addr 172.21.15.60:0/3058266015 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44a40ad970 0x7f44a414c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 -- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44a415b320 msgr2=0x7f44a414ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aad3b640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44a415b320 0x7f44a414ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44a415b320 0x7f44a414ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 -- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44a40ad970 msgr2=0x7f44a414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44a40ad970 0x7f44a414c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44aa53a640 1 -- 172.21.15.60:0/3058266015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c009b30 con 0x7f44a406c040 2023-10-16T12:47:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.627+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44a40ad970 0x7f44a414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.628+0000 7f44aa53a640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 0x7f44a40ca1a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c400 tx=0x7f44a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.628+0000 7f449affd640 1 -- 172.21.15.60:0/3058266015 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a00052d0 con 0x7f44a406c040 2023-10-16T12:47:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.628+0000 7f449affd640 1 -- 172.21.15.60:0/3058266015 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0005430 con 0x7f44a406c040 2023-10-16T12:47:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.628+0000 7f449affd640 1 -- 172.21.15.60:0/3058266015 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0003830 con 0x7f44a406c040 2023-10-16T12:47:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.628+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a40b15d0 con 0x7f44a406c040 2023-10-16T12:47:22.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.628+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a40c9c80 con 0x7f44a406c040 2023-10-16T12:47:22.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.631+0000 7f449affd640 1 -- 172.21.15.60:0/3058266015 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f44a0003990 con 0x7f44a406c040 2023-10-16T12:47:22.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.632+0000 7f449affd640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f448806af90 0x7f448806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.633+0000 7f449affd640 1 -- 172.21.15.60:0/3058266015 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44a0088db0 con 0x7f44a406c040 2023-10-16T12:47:22.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.633+0000 7f44af40c740 1 Processor -- start 2023-10-16T12:47:22.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.633+0000 7f44af40c740 1 -- start start 2023-10-16T12:47:22.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.633+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f448806af90 0x7f448806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44af40c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 0x55a1bee6f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c550 0x55a1bee9ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee6f5b0 0x55a1bee9f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44af40c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bed8f7a0 con 0x55a1bee9c170 2023-10-16T12:47:22.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bede1740 con 0x55a1bee9c550 2023-10-16T12:47:22.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a1beddee30 con 0x55a1bee6f5b0 2023-10-16T12:47:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44aa53a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 0x55a1bee6f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44aa53a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 0x55a1bee6f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34208/0 (socket says 172.21.15.60:34208) 2023-10-16T12:47:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.634+0000 7f44aa53a640 1 -- 172.21.15.60:0/2555951784 learned_addr learned my addr 172.21.15.60:0/2555951784 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aad3b640 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee6f5b0 0x55a1bee9f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f448806af90 0x7f448806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f44a41576e0 tx=0x7f4494009040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aa53a640 1 -- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee6f5b0 msgr2=0x55a1bee9f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44a9d39640 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c550 0x55a1bee9ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aa53a640 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee6f5b0 0x55a1bee9f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aa53a640 1 -- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c550 msgr2=0x55a1bee9ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aa53a640 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c550 0x55a1bee9ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aa53a640 1 -- 172.21.15.60:0/2555951784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1bed9e910 con 0x55a1bee9c170 2023-10-16T12:47:22.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aad3b640 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee6f5b0 0x55a1bee9f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:22.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f44aa53a640 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 0x55a1bee6f070 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000f30 tx=0x7f44a0057970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4a631cc83c5355 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f4487fff640 1 -- 172.21.15.60:0/2555951784 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a006f170 con 0x55a1bee9c170 2023-10-16T12:47:22.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f4487fff640 1 -- 172.21.15.60:0/2555951784 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0087230 con 0x55a1bee9c170 2023-10-16T12:47:22.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.635+0000 7f4487fff640 1 -- 172.21.15.60:0/2555951784 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a006f2d0 con 0x55a1bee9c170 2023-10-16T12:47:22.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 -- 172.21.15.60:0/2555951784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 msgr2=0x55a1bee6f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 0x55a1bee6f070 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000f30 tx=0x7f44a0057970 comp rx=0 tx=0).stop 2023-10-16T12:47:22.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 -- 172.21.15.60:0/2555951784 shutdown_connections 2023-10-16T12:47:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee6f5b0 0x55a1bee9f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c550 0x55a1bee9ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 --2- 172.21.15.60:0/2555951784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c170 0x55a1bee6f070 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 -- 172.21.15.60:0/2555951784 >> 172.21.15.60:0/2555951784 conn(0x55a1bee71380 msgr2=0x55a1beeb9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 -- 172.21.15.60:0/2555951784 shutdown_connections 2023-10-16T12:47:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.636+0000 7f44af40c740 1 -- 172.21.15.60:0/2555951784 wait complete. 2023-10-16T12:47:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.637+0000 7f44af40c740 1 -- start start 2023-10-16T12:47:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.637+0000 7f44af40c740 1 Processor -- start 2023-10-16T12:47:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1bee90430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1beec9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bedda720 con 0x55a1beeb1b60 2023-10-16T12:47:22.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bed78970 con 0x55a1bee9c170 2023-10-16T12:47:22.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a1bed78610 con 0x55a1bee9c550 2023-10-16T12:47:22.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.638+0000 7f44af40c740 10 client.0 _ll_register_callbacks cb 0x55a1bed9a9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:47:22.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1beec9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1beec9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34224/0 (socket says 172.21.15.60:34224) 2023-10-16T12:47:22.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 -- 172.21.15.60:0/985144055 learned_addr learned my addr 172.21.15.60:0/985144055 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 -- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 msgr2=0x55a1bee90430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1bee90430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 -- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 msgr2=0x55a1beeb3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.639+0000 7f44aad3b640 1 -- 172.21.15.60:0/985144055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a40c9c80 con 0x55a1beeb1b60 2023-10-16T12:47:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44aad3b640 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1beec9e70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f449c036040 tx=0x7f449c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257befa14176652 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44857fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:47:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44857fa640 1 -- 172.21.15.60:0/985144055 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c034bf0 con 0x55a1beeb1b60 2023-10-16T12:47:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44857fa640 1 -- 172.21.15.60:0/985144055 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f449c034d50 con 0x55a1beeb1b60 2023-10-16T12:47:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a1bee01160 con 0x55a1beeb1b60 2023-10-16T12:47:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44857fa640 1 -- 172.21.15.60:0/985144055 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f449c048410 con 0x55a1beeb1b60 2023-10-16T12:47:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44af40c740 10 client.6018 fetch_fsmap learned FSMap version 26 2023-10-16T12:47:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55a1bedd8da0 con 0x55a1beeb1b60 2023-10-16T12:47:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.640+0000 7f44857fa640 1 -- 172.21.15.60:0/985144055 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a1bee01160 con 0x55a1beeb1b60 2023-10-16T12:47:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44857fa640 1 -- 172.21.15.60:0/985144055 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f449c048570 con 0x55a1beeb1b60 2023-10-16T12:47:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44af40c740 10 client.6018 fetch_fsmap finished waiting for FSMap version 26 2023-10-16T12:47:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44af40c740 10 client.6018 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:47:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44af40c740 20 client.6018 populate_metadata read hostname 'smithi060' 2023-10-16T12:47:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44af40c740 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1beea2560 0x55a1beed7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44af40c740 4 client.6018 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:47:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 --> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a1bedaccc0 con 0x55a1beea2560 2023-10-16T12:47:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.641+0000 7f44aa53a640 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1beea2560 0x55a1beed7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.642+0000 7f44aa53a640 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1beea2560 0x55a1beed7610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f730674438d9b6fe server_cookie=cc6cbf75bd6bdaa4 in_seq=0 out_seq=0 2023-10-16T12:47:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.642+0000 7f44857fa640 10 client.6018 ms_handle_connect on v2:172.21.15.144:6836/528818860 2023-10-16T12:47:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.644+0000 7f44857fa640 1 -- 172.21.15.60:0/985144055 <== mds.0 v2:172.21.15.144:6836/528818860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a1bedaccc0 con 0x55a1beea2560 2023-10-16T12:47:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.644+0000 7f44857fa640 10 client.6018 handle_command_reply: tid=0 2023-10-16T12:47:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.644+0000 7f44af40c740 1 client.6018 shutdown 2023-10-16T12:47:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 msgr2=0x55a1beec9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1beec9e70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f449c036040 tx=0x7f449c004ab0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 shutdown_connections 2023-10-16T12:47:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1beea2560 0x55a1beed7610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1bee90430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1beec9e70 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 --2- 172.21.15.60:0/985144055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44857fa640 0 client.6018 ms_handle_reset on v2:172.21.15.144:6836/528818860 2023-10-16T12:47:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 >> 172.21.15.60:0/985144055 conn(0x55a1bee6f5b0 msgr2=0x55a1bee9eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 shutdown_connections 2023-10-16T12:47:22.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 1 -- 172.21.15.60:0/985144055 wait complete. 2023-10-16T12:47:22.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.645+0000 7f44af40c740 20 client.6018 trim_cache size 0 max 16384 2023-10-16T12:47:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.662+0000 7f44af40c740 1 Processor -- start 2023-10-16T12:47:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.662+0000 7f44af40c740 1 -- start start 2023-10-16T12:47:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1beeb3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44af40c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1bee6e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44af40c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1beb3ef20 con 0x55a1beeb1b60 2023-10-16T12:47:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1beebe1a0 con 0x55a1bee9c170 2023-10-16T12:47:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a1bee6a9c0 con 0x55a1bee9c550 2023-10-16T12:47:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44aad3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1bee6e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44aad3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1bee6e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34240/0 (socket says 172.21.15.60:34240) 2023-10-16T12:47:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.663+0000 7f44aad3b640 1 -- 172.21.15.60:0/3670904267 learned_addr learned my addr 172.21.15.60:0/3670904267 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aa53a640 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1beeb3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aad3b640 1 -- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 msgr2=0x55a1beeb3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aad3b640 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1beeb3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aad3b640 1 -- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 msgr2=0x55a1beeb1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aad3b640 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aad3b640 1 -- 172.21.15.60:0/3670904267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1bed77b80 con 0x55a1beeb1b60 2023-10-16T12:47:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.664+0000 7f44aad3b640 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1bee6e0f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f449c0072a0 tx=0x7f449c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d90d697c9ae4054 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f4498ff9640 1 -- 172.21.15.60:0/3670904267 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c009510 con 0x55a1beeb1b60 2023-10-16T12:47:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f4498ff9640 1 -- 172.21.15.60:0/3670904267 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c048410 con 0x55a1beeb1b60 2023-10-16T12:47:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f4498ff9640 1 -- 172.21.15.60:0/3670904267 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c0349f0 con 0x55a1beeb1b60 2023-10-16T12:47:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 -- 172.21.15.60:0/3670904267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 msgr2=0x55a1bee6e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1bee6e0f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f449c0072a0 tx=0x7f449c03f000 comp rx=0 tx=0).stop 2023-10-16T12:47:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 -- 172.21.15.60:0/3670904267 shutdown_connections 2023-10-16T12:47:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1bee9c550 0x55a1beeb3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1beeb1b60 0x55a1bee6e0f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 --2- 172.21.15.60:0/3670904267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beeb1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 -- 172.21.15.60:0/3670904267 >> 172.21.15.60:0/3670904267 conn(0x55a1bee6f5b0 msgr2=0x55a1bee761a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 -- 172.21.15.60:0/3670904267 shutdown_connections 2023-10-16T12:47:22.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.665+0000 7f44af40c740 1 -- 172.21.15.60:0/3670904267 wait complete. 2023-10-16T12:47:22.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.666+0000 7f44af40c740 1 -- start start 2023-10-16T12:47:22.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.666+0000 7f44af40c740 1 Processor -- start 2023-10-16T12:47:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beebc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 0x55a1beebcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1beeb1b60 0x55a1bee8edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bee73e10 con 0x55a1bee9c550 2023-10-16T12:47:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1bee74450 con 0x55a1bee9c170 2023-10-16T12:47:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a1beebbdf0 con 0x55a1beeb1b60 2023-10-16T12:47:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44af40c740 10 client.0 _ll_register_callbacks cb 0x7f449c009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:47:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.667+0000 7f44a9d39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 0x55a1beebcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 0x55a1beebcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34248/0 (socket says 172.21.15.60:34248) 2023-10-16T12:47:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 -- 172.21.15.60:0/3662520750 learned_addr learned my addr 172.21.15.60:0/3662520750 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 -- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1beeb1b60 msgr2=0x55a1bee8edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1beeb1b60 0x55a1bee8edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 -- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 msgr2=0x55a1beebc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beebc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 -- 172.21.15.60:0/3662520750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1bedd8da0 con 0x55a1bee9c550 2023-10-16T12:47:22.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44a9d39640 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 0x55a1beebcc10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f44a41f9fb0 tx=0x7f4494001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f3b1c44e6c6996 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.668+0000 7f44857fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:47:22.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44857fa640 1 -- 172.21.15.60:0/3662520750 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494014d00 con 0x55a1bee9c550 2023-10-16T12:47:22.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44857fa640 1 -- 172.21.15.60:0/3662520750 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44940203f0 con 0x55a1bee9c550 2023-10-16T12:47:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44a0087230 con 0x55a1bee9c550 2023-10-16T12:47:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44857fa640 1 -- 172.21.15.60:0/3662520750 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4494014e60 con 0x55a1bee9c550 2023-10-16T12:47:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 10 client.6024 fetch_fsmap learned FSMap version 26 2023-10-16T12:47:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55a1bed9d170 con 0x55a1bee9c550 2023-10-16T12:47:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44857fa640 1 -- 172.21.15.60:0/3662520750 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0087230 con 0x55a1bee9c550 2023-10-16T12:47:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44857fa640 1 -- 172.21.15.60:0/3662520750 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f449401f070 con 0x55a1bee9c550 2023-10-16T12:47:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 10 client.6024 fetch_fsmap finished waiting for FSMap version 26 2023-10-16T12:47:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 10 client.6024 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:47:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 20 client.6024 populate_metadata read hostname 'smithi060' 2023-10-16T12:47:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1bee8f330 0x55a1bef25c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 4 client.6024 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:47:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.669+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 --> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a1bedd3700 con 0x55a1bee8f330 2023-10-16T12:47:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.670+0000 7f44aa53a640 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1bee8f330 0x55a1bef25c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.671+0000 7f44aa53a640 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1bee8f330 0x55a1bef25c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e1a4831a68ad4fc server_cookie=c3dd03f3b6ec1848 in_seq=0 out_seq=0 2023-10-16T12:47:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.671+0000 7f44857fa640 10 client.6024 ms_handle_connect on v2:172.21.15.144:6836/528818860 2023-10-16T12:47:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.671+0000 7f44857fa640 1 -- 172.21.15.60:0/3662520750 <== mds.0 v2:172.21.15.144:6836/528818860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a1bedd3700 con 0x55a1bee8f330 2023-10-16T12:47:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.671+0000 7f44857fa640 10 client.6024 handle_command_reply: tid=0 2023-10-16T12:47:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.671+0000 7f44af40c740 1 client.6024 shutdown 2023-10-16T12:47:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 msgr2=0x55a1beebcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 0x55a1beebcc10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f44a41f9fb0 tx=0x7f4494001080 comp rx=0 tx=0).stop 2023-10-16T12:47:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 shutdown_connections 2023-10-16T12:47:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:6836/528818860,v1:172.21.15.144:6837/528818860] conn(0x55a1bee8f330 0x55a1bef25c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a1beeb1b60 0x55a1bee8edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a1bee9c550 0x55a1beebcc10 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44857fa640 0 client.6024 ms_handle_reset on v2:172.21.15.144:6836/528818860 2023-10-16T12:47:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 --2- 172.21.15.60:0/3662520750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a1bee9c170 0x55a1beebc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 >> 172.21.15.60:0/3662520750 conn(0x55a1bee6f5b0 msgr2=0x55a1bee6e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 shutdown_connections 2023-10-16T12:47:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 1 -- 172.21.15.60:0/3662520750 wait complete. 2023-10-16T12:47:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.672+0000 7f44af40c740 20 client.6024 trim_cache size 0 max 16384 2023-10-16T12:47:22.687 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:47:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:47:22.687 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:47:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 5874, 2023-10-16T12:47:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:47:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:47:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:47:22.688 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8287360880000003, 2023-10-16T12:47:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 26, 2023-10-16T12:47:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 67, 2023-10-16T12:47:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 67, 2023-10-16T12:47:22.689 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.618275836 2023-10-16T12:47:22.689 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:47:22.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f448806af90 msgr2=0x7f448806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f448806af90 0x7f448806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f44a41576e0 tx=0x7f4494009040 comp rx=0 tx=0).stop 2023-10-16T12:47:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 msgr2=0x7f44a40ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 0x7f44a40ca1a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c400 tx=0x7f44a000c8c0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 shutdown_connections 2023-10-16T12:47:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f448806af90 0x7f448806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44a415b320 0x7f44a414ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44a40ad970 0x7f44a414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 --2- 172.21.15.60:0/3058266015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44a406c040 0x7f44a40ca1a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 >> 172.21.15.60:0/3058266015 conn(0x7f44a40ce6a0 msgr2=0x7f44a415de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 shutdown_connections 2023-10-16T12:47:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.673+0000 7f44ab53c640 1 -- 172.21.15.60:0/3058266015 wait complete. 2023-10-16T12:47:22.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:47:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.920+0000 7f88ea111640 1 Processor -- start 2023-10-16T12:47:22.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.920+0000 7f88ea111640 1 -- start start 2023-10-16T12:47:22.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.921+0000 7f88ea111640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 0x7f88e40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.921+0000 7f88ea111640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e40c8cf0 con 0x7f88e415cdb0 2023-10-16T12:47:22.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.921+0000 7f88ea111640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e40cc6d0 con 0x7f88e40acdd0 2023-10-16T12:47:22.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.921+0000 7f88ea111640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e415d190 con 0x7f88e40cc270 2023-10-16T12:47:22.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.922+0000 7f88e910f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f88e415cdb0 0x7f88e40c9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51208/0 (socket says 172.21.15.60:51208) 2023-10-16T12:47:22.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.922+0000 7f88e910f640 1 -- 172.21.15.60:0/147157286 learned_addr learned my addr 172.21.15.60:0/147157286 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:22.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.922+0000 7f88e890e640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 0x7f88e40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.922+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076080449 0 0) 0x7f88e40c8cf0 con 0x7f88e415cdb0 2023-10-16T12:47:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.922+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d0003520 con 0x7f88e415cdb0 2023-10-16T12:47:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1619912481 0 0) 0x7f88d0003520 con 0x7f88e415cdb0 2023-10-16T12:47:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 msgr2=0x7f88e40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 0x7f88e40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 >> v1:172.21.15.144:6789/0 conn(0x7f88e40acdd0 legacy=0x7f88e406fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40ccca0 con 0x7f88e415cdb0 2023-10-16T12:47:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f88e0001670 con 0x7f88e415cdb0 2023-10-16T12:47:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 >> v1:172.21.15.60:6789/0 conn(0x7f88e415cdb0 legacy=0x7f88e40c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 0x7f88d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d0004d00 0x7f88d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e415d190 con 0x7f88d00039a0 2023-10-16T12:47:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d000bb40 con 0x7f88d0004d00 2023-10-16T12:47:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d000bd00 con 0x7f88e40cc270 2023-10-16T12:47:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88e910f640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88e910f640 1 -- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 msgr2=0x7f88d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88e910f640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40cc270 0x7f88d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.923+0000 7f88e910f640 1 -- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d0004d00 msgr2=0x7f88d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.924+0000 7f88e910f640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88d0004d00 0x7f88d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.924+0000 7f88e910f640 1 -- 172.21.15.60:0/147157286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d0003520 con 0x7f88d00039a0 2023-10-16T12:47:22.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88e910f640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88d0003dc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f88e000c0e0 tx=0x7f88d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320c9f006d03ee48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0005ad0 con 0x7f88d00039a0 2023-10-16T12:47:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0005c30 con 0x7f88d00039a0 2023-10-16T12:47:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88dbfff640 1 -- 172.21.15.60:0/147157286 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e002c810 con 0x7f88d00039a0 2023-10-16T12:47:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88ea111640 1 -- 172.21.15.60:0/147157286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 msgr2=0x7f88d0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88ea111640 1 --2- 172.21.15.60:0/147157286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88d0003dc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f88e000c0e0 tx=0x7f88d00057f0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.925+0000 7f88e9910640 1 -- 172.21.15.60:0/147157286 reap_dead start 2023-10-16T12:47:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.926+0000 7f88ea111640 1 -- 172.21.15.60:0/147157286 shutdown_connections 2023-10-16T12:47:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.926+0000 7f88ea111640 1 -- 172.21.15.60:0/147157286 >> 172.21.15.60:0/147157286 conn(0x7f88e40b0a90 msgr2=0x7f88e40ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.926+0000 7f88ea111640 1 -- 172.21.15.60:0/147157286 shutdown_connections 2023-10-16T12:47:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.926+0000 7f88ea111640 1 -- 172.21.15.60:0/147157286 wait complete. 2023-10-16T12:47:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.927+0000 7f88ea111640 1 Processor -- start 2023-10-16T12:47:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.928+0000 7f88ea111640 1 -- start start 2023-10-16T12:47:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88ea111640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88e414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88ea111640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40c69d0 0x7f88e40cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88ea111640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88e40cb660 0x7f88e406adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88ea111640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40cbb10 con 0x7f88d00039a0 2023-10-16T12:47:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88ea111640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e406cc50 con 0x7f88e40cb660 2023-10-16T12:47:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88ea111640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e4150100 con 0x7f88e40c69d0 2023-10-16T12:47:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88e414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88e414fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34276/0 (socket says 172.21.15.60:34276) 2023-10-16T12:47:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 -- 172.21.15.60:0/4273848972 learned_addr learned my addr 172.21.15.60:0/4273848972 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 -- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40c69d0 msgr2=0x7f88e40cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40c69d0 0x7f88e40cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 -- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88e40cb660 msgr2=0x7f88e406adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88e40cb660 0x7f88e406adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.929+0000 7f88e910f640 1 -- 172.21.15.60:0/4273848972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e406bd50 con 0x7f88d00039a0 2023-10-16T12:47:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.930+0000 7f88e910f640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88e414fbc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f88e002c3b0 tx=0x7f88e0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.930+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e003c040 con 0x7f88d00039a0 2023-10-16T12:47:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.930+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e002cc20 con 0x7f88d00039a0 2023-10-16T12:47:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.930+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0007070 con 0x7f88d00039a0 2023-10-16T12:47:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.930+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e406ba70 con 0x7f88d00039a0 2023-10-16T12:47:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.930+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e40c25c0 con 0x7f88d00039a0 2023-10-16T12:47:22.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.931+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e40c2020 con 0x7f88d00039a0 2023-10-16T12:47:22.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.932+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88e0032070 con 0x7f88d00039a0 2023-10-16T12:47:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.934+0000 7f88d9ffb640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f88c806af90 0x7f88c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.934+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88e00aa920 con 0x7f88d00039a0 2023-10-16T12:47:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.934+0000 7f88e890e640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f88c806af90 0x7f88c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.936+0000 7f88e890e640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f88c806af90 0x7f88c806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88d40070b0 tx=0x7f88d400d040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:22.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:22.938+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88e00758b0 con 0x7f88d00039a0 2023-10-16T12:47:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.178+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f88e40c9200 con 0x7f88d00039a0 2023-10-16T12:47:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.179+0000 7f88d9ffb640 1 -- 172.21.15.60:0/4273848972 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f88e40c9200 con 0x7f88d00039a0 2023-10-16T12:47:23.181 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:47:23.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f88c806af90 msgr2=0x7f88c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:23.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f88c806af90 0x7f88c806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88d40070b0 tx=0x7f88d400d040 comp rx=0 tx=0).stop 2023-10-16T12:47:23.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 msgr2=0x7f88e414fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:23.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88e414fbc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f88e002c3b0 tx=0x7f88e0001930 comp rx=0 tx=0).stop 2023-10-16T12:47:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 shutdown_connections 2023-10-16T12:47:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f88c806af90 0x7f88c806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88e40c69d0 0x7f88e40cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88e40cb660 0x7f88e406adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 --2- 172.21.15.60:0/4273848972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00039a0 0x7f88e414fbc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:23.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.182+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 >> 172.21.15.60:0/4273848972 conn(0x7f88e40b0a90 msgr2=0x7f88e40cd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:23.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.183+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 shutdown_connections 2023-10-16T12:47:23.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:23.183+0000 7f88ea111640 1 -- 172.21.15.60:0/4273848972 wait complete. 2023-10-16T12:47:23.202 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:47:23.203 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:47:23.203 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:47:23.203 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:47:23.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:47:23.204 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:47:23.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:47:23.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:47:23.205 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:47:23.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:23.206 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:47:23.263 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:47:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:47:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:47:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:47:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.264 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:47:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:47:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:47:23.265 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:47:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:47:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:47:23.265 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:47:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:47:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:47:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:47:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41544sec preferred_lft 41544sec 2023-10-16T12:47:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:47:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.267 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:47:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:47:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:47:23.268 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:47:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:47:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:47:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:47:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.269 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:47:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:47:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:47:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.270 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:47:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:47:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:47:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:47:23.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:23.272 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:47:23.375 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:47:23.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:23.381 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:47:23.441 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:47:23.442 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:47:23.442 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:47:23.442 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:47:23.442 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:47:23.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:23.443 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:47:23.443 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:47:23.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:47:23.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:47:23.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:47:23.445 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:47:23.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:23.633 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:47:23.691 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:47:23.692 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:47:23.693 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:47:23.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:23.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:47:23.798 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:47:23.933 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:47:24.004 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:47:24.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:47:24.005 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:47:24.062 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:47:24.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:24.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:47:24.219 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:47:24.219 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:47:24.255 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:47:24.255 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:47:24.256 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:47:24.256 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:47:24.256 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:47:24.257 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:47:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:47:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:47:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:47:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:47:24.262 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:47:24.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:24.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:47:24.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:24.350 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:47:24.436 INFO:teuthology.orchestra.run.smithi060.stdout:client5102 2023-10-16T12:47:24.437 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:47:24.437 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:47:24.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:47:24.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.736+0000 7f2a38c87640 1 Processor -- start 2023-10-16T12:47:24.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.736+0000 7f2a38c87640 1 -- start start 2023-10-16T12:47:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.737+0000 7f2a38c87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3415d150 0x7f2a340ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.737+0000 7f2a38c87640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a3406a4b0 0x7f2a34160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a38c87640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340ac890 con 0x7f2a34160b90 2023-10-16T12:47:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a38c87640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3415d530 con 0x7f2a3406a4b0 2023-10-16T12:47:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a38c87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a34161650 con 0x7f2a3415d150 2023-10-16T12:47:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a3406a4b0 0x7f2a34160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a3406a4b0 0x7f2a34160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34282/0 (socket says 172.21.15.60:34282) 2023-10-16T12:47:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 -- 172.21.15.60:0/476239354 learned_addr learned my addr 172.21.15.60:0/476239354 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a3259c640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3415d150 0x7f2a340ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 -- 172.21.15.60:0/476239354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3415d150 msgr2=0x7f2a340ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3415d150 0x7f2a340ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 -- 172.21.15.60:0/476239354 >> v1:172.21.15.60:6789/0 conn(0x7f2a34160b90 legacy=0x7f2a34069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a32d9d640 1 -- 172.21.15.60:0/476239354 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34161930 con 0x7f2a3406a4b0 2023-10-16T12:47:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.738+0000 7f2a3259c640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3415d150 0x7f2a340ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.739+0000 7f2a32d9d640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3406a4b0 0x7f2a34160fb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce9c2e81a4fc06e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a3159a640 1 -- 172.21.15.60:0/476239354 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c030840 con 0x7f2a3406a4b0 2023-10-16T12:47:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a3159a640 1 -- 172.21.15.60:0/476239354 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2c0309a0 con 0x7f2a3406a4b0 2023-10-16T12:47:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a3159a640 1 -- 172.21.15.60:0/476239354 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c030d30 con 0x7f2a3406a4b0 2023-10-16T12:47:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a38c87640 1 -- 172.21.15.60:0/476239354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3406a4b0 msgr2=0x7f2a34160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a38c87640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3406a4b0 0x7f2a34160fb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c02b980 comp rx=0 tx=0).stop 2023-10-16T12:47:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a38c87640 1 -- 172.21.15.60:0/476239354 shutdown_connections 2023-10-16T12:47:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a38c87640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a3406a4b0 0x7f2a34160fb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.740+0000 7f2a38c87640 1 --2- 172.21.15.60:0/476239354 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3415d150 0x7f2a340ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.741+0000 7f2a38c87640 1 -- 172.21.15.60:0/476239354 >> 172.21.15.60:0/476239354 conn(0x7f2a340ce9b0 msgr2=0x7f2a340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.741+0000 7f2a38c87640 1 -- 172.21.15.60:0/476239354 shutdown_connections 2023-10-16T12:47:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.741+0000 7f2a38c87640 1 -- 172.21.15.60:0/476239354 wait complete. 2023-10-16T12:47:24.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.742+0000 7f2a38c87640 1 Processor -- start 2023-10-16T12:47:24.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 -- start start 2023-10-16T12:47:24.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3406a4b0 0x7f2a340b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:24.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a3415d150 0x7f2a34146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:24.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 0x7f2a34147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:24.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340b36f0 con 0x7f2a34160b90 2023-10-16T12:47:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340ae620 con 0x7f2a3415d150 2023-10-16T12:47:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.743+0000 7f2a38c87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340b30a0 con 0x7f2a3406a4b0 2023-10-16T12:47:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 0x7f2a34147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 0x7f2a34147450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34284/0 (socket says 172.21.15.60:34284) 2023-10-16T12:47:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 -- 172.21.15.60:0/1395571733 learned_addr learned my addr 172.21.15.60:0/1395571733 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a3259c640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3406a4b0 0x7f2a340b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 -- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3406a4b0 msgr2=0x7f2a340b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a31d9b640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a3415d150 0x7f2a34146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3406a4b0 0x7f2a340b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 -- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a3415d150 msgr2=0x7f2a34146f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a3415d150 0x7f2a34146f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a32d9d640 1 -- 172.21.15.60:0/1395571733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2c0094d0 con 0x7f2a34160b90 2023-10-16T12:47:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.744+0000 7f2a3259c640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3406a4b0 0x7f2a340b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.745+0000 7f2a32d9d640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 0x7f2a34147450 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.745+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c0307d0 con 0x7f2a34160b90 2023-10-16T12:47:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.745+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2c043400 con 0x7f2a34160b90 2023-10-16T12:47:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.745+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a340b3380 con 0x7f2a34160b90 2023-10-16T12:47:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.745+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c0437b0 con 0x7f2a34160b90 2023-10-16T12:47:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.745+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a340ae240 con 0x7f2a34160b90 2023-10-16T12:47:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.746+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a340b38a0 con 0x7f2a34160b90 2023-10-16T12:47:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.747+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a2c043910 con 0x7f2a34160b90 2023-10-16T12:47:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.748+0000 7f2a2affd640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2a1406af90 0x7f2a1406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.748+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a2c03e070 con 0x7f2a34160b90 2023-10-16T12:47:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.749+0000 7f2a3259c640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2a1406af90 0x7f2a1406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.750+0000 7f2a3259c640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2a1406af90 0x7f2a1406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400d430 tx=0x7f2a24007530 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.753+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a340b38a0 con 0x7f2a34160b90 2023-10-16T12:47:24.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.974+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 --> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a340ace30 con 0x7f2a1406af90 2023-10-16T12:47:24.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.978+0000 7f2a2affd640 1 -- 172.21.15.60:0/1395571733 <== mgr.4910 v2:172.21.15.144:6832/3875614959 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2a24018470 con 0x7f2a1406af90 2023-10-16T12:47:24.979 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:47:24.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:47:24.980 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:47:24.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:47:24.980 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:47:24.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2a1406af90 msgr2=0x7f2a1406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:24.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2a1406af90 0x7f2a1406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400d430 tx=0x7f2a24007530 comp rx=0 tx=0).stop 2023-10-16T12:47:24.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 msgr2=0x7f2a34147450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:24.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 0x7f2a34147450 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c004730 comp rx=0 tx=0).stop 2023-10-16T12:47:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 shutdown_connections 2023-10-16T12:47:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f2a1406af90 0x7f2a1406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a34160b90 0x7f2a34147450 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a3415d150 0x7f2a34146f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 --2- 172.21.15.60:0/1395571733 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2a3406a4b0 0x7f2a340b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.980+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 >> 172.21.15.60:0/1395571733 conn(0x7f2a340ce9b0 msgr2=0x7f2a340ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.981+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 shutdown_connections 2023-10-16T12:47:24.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:24.981+0000 7f2a38c87640 1 -- 172.21.15.60:0/1395571733 wait complete. 2023-10-16T12:47:25.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:47:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.230+0000 7f7a7e311640 1 Processor -- start 2023-10-16T12:47:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.230+0000 7f7a7e311640 1 -- start start 2023-10-16T12:47:25.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7e311640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a7815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7e311640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7a78070450 0x7f7a780708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7e311640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a7806c100 0x7f7a7806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7e311640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7815e490 con 0x7f7a7806c100 2023-10-16T12:47:25.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7e311640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a78070df0 con 0x7f7a78070450 2023-10-16T12:47:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7e311640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815aef0 con 0x7f7a78159950 2023-10-16T12:47:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7db10640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a7806c100 0x7f7a7806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7db10640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a7806c100 0x7f7a7806c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34298/0 (socket says 172.21.15.60:34298) 2023-10-16T12:47:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.232+0000 7f7a7db10640 1 -- 172.21.15.60:0/2422267134 learned_addr learned my addr 172.21.15.60:0/2422267134 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7d30f640 1 --2- 172.21.15.60:0/2422267134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a7815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7cb0e640 1 --2- 172.21.15.60:0/2422267134 >> v2:172.21.15.144:3300/0 conn(0x7f7a78070450 0x7f7a780708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7db10640 1 -- 172.21.15.60:0/2422267134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 msgr2=0x7f7a7815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7db10640 1 --2- 172.21.15.60:0/2422267134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a7815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7db10640 1 -- 172.21.15.60:0/2422267134 >> v2:172.21.15.144:3300/0 conn(0x7f7a78070450 msgr2=0x7f7a780708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7db10640 1 --2- 172.21.15.60:0/2422267134 >> v2:172.21.15.144:3300/0 conn(0x7f7a78070450 0x7f7a780708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.233+0000 7f7a7db10640 1 -- 172.21.15.60:0/2422267134 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7815d910 con 0x7f7a7806c100 2023-10-16T12:47:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.234+0000 7f7a7db10640 1 --2- 172.21.15.60:0/2422267134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a7806c100 0x7f7a7806c570 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7a780719e0 tx=0x7f7a7002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c14f6e5d8697c21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.234+0000 7f7a6bfff640 1 -- 172.21.15.60:0/2422267134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7002c9e0 con 0x7f7a7806c100 2023-10-16T12:47:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.234+0000 7f7a6bfff640 1 -- 172.21.15.60:0/2422267134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7002cb40 con 0x7f7a7806c100 2023-10-16T12:47:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.234+0000 7f7a6bfff640 1 -- 172.21.15.60:0/2422267134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a700343f0 con 0x7f7a7806c100 2023-10-16T12:47:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 -- 172.21.15.60:0/2422267134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a7806c100 msgr2=0x7f7a7806c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:25.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 --2- 172.21.15.60:0/2422267134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a7806c100 0x7f7a7806c570 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7a780719e0 tx=0x7f7a7002be40 comp rx=0 tx=0).stop 2023-10-16T12:47:25.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 -- 172.21.15.60:0/2422267134 shutdown_connections 2023-10-16T12:47:25.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 --2- 172.21.15.60:0/2422267134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a7806c100 0x7f7a7806c570 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 --2- 172.21.15.60:0/2422267134 >> v2:172.21.15.144:3300/0 conn(0x7f7a78070450 0x7f7a780708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 --2- 172.21.15.60:0/2422267134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a7815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 -- 172.21.15.60:0/2422267134 >> 172.21.15.60:0/2422267134 conn(0x7f7a780ce6a0 msgr2=0x7f7a780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:25.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 -- 172.21.15.60:0/2422267134 shutdown_connections 2023-10-16T12:47:25.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.235+0000 7f7a7e311640 1 -- 172.21.15.60:0/2422267134 wait complete. 2023-10-16T12:47:25.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.237+0000 7f7a7e311640 1 Processor -- start 2023-10-16T12:47:25.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.237+0000 7f7a7e311640 1 -- start start 2023-10-16T12:47:25.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7e311640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a7806c100 0x7f7a78157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7e311640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 0x7f7a78157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7e311640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a781f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7e311640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815ce00 con 0x7f7a78070450 2023-10-16T12:47:25.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7e311640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a78157ad0 con 0x7f7a7806c100 2023-10-16T12:47:25.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7e311640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7815c180 con 0x7f7a78159950 2023-10-16T12:47:25.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7cb0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 0x7f7a78157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7cb0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 0x7f7a78157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34308/0 (socket says 172.21.15.60:34308) 2023-10-16T12:47:25.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.238+0000 7f7a7cb0e640 1 -- 172.21.15.60:0/1492238695 learned_addr learned my addr 172.21.15.60:0/1492238695 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:25.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7db10640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a781f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7d30f640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a7806c100 0x7f7a78157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7cb0e640 1 -- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 msgr2=0x7f7a781f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:25.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7cb0e640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a781f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7cb0e640 1 -- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a7806c100 msgr2=0x7f7a78157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:25.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7cb0e640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a7806c100 0x7f7a78157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7cb0e640 1 -- 172.21.15.60:0/1492238695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a70009b30 con 0x7f7a78070450 2023-10-16T12:47:25.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7db10640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a781f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:25.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7d30f640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a7806c100 0x7f7a78157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:25.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7cb0e640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 0x7f7a78157590 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00d800 tx=0x7f7a6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:25.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c00b6c0 con 0x7f7a78070450 2023-10-16T12:47:25.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c00b820 con 0x7f7a78070450 2023-10-16T12:47:25.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c00bbb0 con 0x7f7a78070450 2023-10-16T12:47:25.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.239+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7815c460 con 0x7f7a78070450 2023-10-16T12:47:25.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.240+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7815ca20 con 0x7f7a78070450 2023-10-16T12:47:25.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.241+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7815a490 con 0x7f7a78070450 2023-10-16T12:47:25.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.242+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a6c00bd10 con 0x7f7a78070450 2023-10-16T12:47:25.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.242+0000 7f7a69ffb640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f7a5c06af90 0x7f7a5c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:25.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.243+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a6c089dd0 con 0x7f7a78070450 2023-10-16T12:47:25.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.243+0000 7f7a7d30f640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f7a5c06af90 0x7f7a5c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:25.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.244+0000 7f7a7d30f640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f7a5c06af90 0x7f7a5c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a74006fd0 tx=0x7f7a74008040 comp rx=0 tx=0).ready entity=mgr.4910 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:25.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.247+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a6c054cd0 con 0x7f7a78070450 2023-10-16T12:47:25.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.494+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7a7815dea0 con 0x7f7a78070450 2023-10-16T12:47:25.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:25.913 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:25.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.934+0000 7f7a69ffb640 1 -- 172.21.15.60:0/1492238695 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v18) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7a6c0582f0 con 0x7f7a78070450 2023-10-16T12:47:25.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f7a5c06af90 msgr2=0x7f7a5c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:25.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f7a5c06af90 0x7f7a5c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a74006fd0 tx=0x7f7a74008040 comp rx=0 tx=0).stop 2023-10-16T12:47:25.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 msgr2=0x7f7a78157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:25.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 0x7f7a78157590 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00d800 tx=0x7f7a6c00dcc0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 shutdown_connections 2023-10-16T12:47:25.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f7a5c06af90 0x7f7a5c06d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a78159950 0x7f7a781f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a78070450 0x7f7a78157590 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 --2- 172.21.15.60:0/1492238695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a7806c100 0x7f7a78157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:25.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.938+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 >> 172.21.15.60:0/1492238695 conn(0x7f7a780ce6a0 msgr2=0x7f7a781564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:25.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.940+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 shutdown_connections 2023-10-16T12:47:25.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:25.942+0000 7f7a7e311640 1 -- 172.21.15.60:0/1492238695 wait complete. 2023-10-16T12:47:25.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:47:26.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.207+0000 7f6ed86c9640 1 Processor -- start 2023-10-16T12:47:26.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.207+0000 7f6ed86c9640 1 -- start start 2023-10-16T12:47:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.208+0000 7f6ed86c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ed015ad70 0x7f6ed0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.208+0000 7f6ed86c9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ed00c9ca0 0x7f6ed006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.208+0000 7f6ed86c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed00cb030 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.208+0000 7f6ed86c9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed00c91c0 con 0x7f6ed00c9ca0 2023-10-16T12:47:26.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.208+0000 7f6ed86c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed00c94a0 con 0x7f6ed015ad70 2023-10-16T12:47:26.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed5c64640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ed00c9ca0 0x7f6ed006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed6c66640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6ed00ca3b0 0x7f6ed00ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51236/0 (socket says 172.21.15.60:51236) 2023-10-16T12:47:26.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed6c66640 1 -- 172.21.15.60:0/171490652 learned_addr learned my addr 172.21.15.60:0/171490652 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:26.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed6465640 1 --2- 172.21.15.60:0/171490652 >> v2:172.21.15.144:3300/0 conn(0x7f6ed015ad70 0x7f6ed0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed5c64640 1 -- 172.21.15.60:0/171490652 >> v2:172.21.15.144:3300/0 conn(0x7f6ed015ad70 msgr2=0x7f6ed0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:26.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed5c64640 1 --2- 172.21.15.60:0/171490652 >> v2:172.21.15.144:3300/0 conn(0x7f6ed015ad70 0x7f6ed0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed5c64640 1 -- 172.21.15.60:0/171490652 >> v1:172.21.15.60:6789/0 conn(0x7f6ed00ca3b0 legacy=0x7f6ed00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed5463640 1 -- 172.21.15.60:0/171490652 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328467335 0 0) 0x7f6ed00cb030 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.209+0000 7f6ed5c64640 1 -- 172.21.15.60:0/171490652 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed00c9780 con 0x7f6ed00c9ca0 2023-10-16T12:47:26.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.210+0000 7f6ed5c64640 1 --2- 172.21.15.60:0/171490652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00c9ca0 0x7f6ed006a2e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00cf780 tx=0x7f6ec402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f7d0872b59ade8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:26.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.211+0000 7f6ed5463640 1 -- 172.21.15.60:0/171490652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4031830 con 0x7f6ed00c9ca0 2023-10-16T12:47:26.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.211+0000 7f6ed5463640 1 -- 172.21.15.60:0/171490652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4031990 con 0x7f6ed00c9ca0 2023-10-16T12:47:26.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.211+0000 7f6ed5463640 1 -- 172.21.15.60:0/171490652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4031d20 con 0x7f6ed00c9ca0 2023-10-16T12:47:26.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.211+0000 7f6ed86c9640 1 -- 172.21.15.60:0/171490652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00c9ca0 msgr2=0x7f6ed006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.211+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/171490652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00c9ca0 0x7f6ed006a2e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00cf780 tx=0x7f6ec402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.212+0000 7f6ed86c9640 1 -- 172.21.15.60:0/171490652 shutdown_connections 2023-10-16T12:47:26.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.212+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/171490652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00c9ca0 0x7f6ed006a2e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.212+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/171490652 >> v2:172.21.15.144:3300/0 conn(0x7f6ed015ad70 0x7f6ed0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.212+0000 7f6ed86c9640 1 -- 172.21.15.60:0/171490652 >> 172.21.15.60:0/171490652 conn(0x7f6ed00ce9b0 msgr2=0x7f6ed00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:26.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.212+0000 7f6ed86c9640 1 -- 172.21.15.60:0/171490652 shutdown_connections 2023-10-16T12:47:26.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.212+0000 7f6ed86c9640 1 -- 172.21.15.60:0/171490652 wait complete. 2023-10-16T12:47:26.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.213+0000 7f6ed86c9640 1 Processor -- start 2023-10-16T12:47:26.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.214+0000 7f6ed86c9640 1 -- start start 2023-10-16T12:47:26.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.214+0000 7f6ed86c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ed00c9ca0 0x7f6ed0158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.214+0000 7f6ed86c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 0x7f6ed015be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed86c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ed015ad70 0x7f6ed01e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed86c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed014e0c0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed86c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed015c670 con 0x7f6ed015ad70 2023-10-16T12:47:26.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed86c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed015c950 con 0x7f6ed00c9ca0 2023-10-16T12:47:26.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 0x7f6ed015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 0x7f6ed015be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34330/0 (socket says 172.21.15.60:34330) 2023-10-16T12:47:26.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 -- 172.21.15.60:0/3917577948 learned_addr learned my addr 172.21.15.60:0/3917577948 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:26.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed6465640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ed00c9ca0 0x7f6ed0158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed6c66640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ed015ad70 0x7f6ed01e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 -- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ed00c9ca0 msgr2=0x7f6ed0158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ed00c9ca0 0x7f6ed0158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 -- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ed015ad70 msgr2=0x7f6ed01e4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.215+0000 7f6ed5c64640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ed015ad70 0x7f6ed01e4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.216+0000 7f6ed5c64640 1 -- 172.21.15.60:0/3917577948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec4009400 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.216+0000 7f6ed5c64640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 0x7f6ed015be10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009890 tx=0x7f6ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.216+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec403a630 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.216+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec403a7d0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.216+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec403ab80 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.216+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed015cc30 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.217+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed015d1f0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.218+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed015d3d0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.219+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6ec403ace0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.220+0000 7f6eceffd640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 0x7f6ebc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.221+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ec40ab540 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.221+0000 7f6ed6465640 1 -- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 msgr2=0x7f6ebc06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.221+0000 7f6ed6465640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 0x7f6ebc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:47:26.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.225+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ec40765a0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.421+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6ed004e160 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.422+0000 7f6ed6465640 1 -- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 msgr2=0x7f6ebc06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.422+0000 7f6ed6465640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 0x7f6ebc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:47:26.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.426+0000 7f6eceffd640 1 -- 172.21.15.60:0/3917577948 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6ec4079bc0 con 0x7f6ed00ca3b0 2023-10-16T12:47:26.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 msgr2=0x7f6ebc06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:47:26.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 0x7f6ebc06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 msgr2=0x7f6ed015be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 0x7f6ed015be10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009890 tx=0x7f6ec40046c0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 shutdown_connections 2023-10-16T12:47:26.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f6ebc06afe0 0x7f6ebc06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ed015ad70 0x7f6ed01e4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ed00ca3b0 0x7f6ed015be10 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 --2- 172.21.15.60:0/3917577948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ed00c9ca0 0x7f6ed0158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 >> 172.21.15.60:0/3917577948 conn(0x7f6ed00ce9b0 msgr2=0x7f6ed00b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 shutdown_connections 2023-10-16T12:47:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.429+0000 7f6ed86c9640 1 -- 172.21.15.60:0/3917577948 wait complete. 2023-10-16T12:47:26.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:26.444+0000 7f5a5ac38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:47:26.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:47:26.499 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:26.497+0000 7f5a5ac38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:47:26.504 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:26.503+0000 7f143c22c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:47:26.562 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:26.561+0000 7f143c22c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:47:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.656+0000 7f191b97c640 1 Processor -- start 2023-10-16T12:47:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.656+0000 7f191b97c640 1 -- start start 2023-10-16T12:47:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f191b97c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1914160530 0x7f19141609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f191b97c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914069fa0 0x7f191406a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f191b97c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19140ac890 con 0x7f1914065ab0 2023-10-16T12:47:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f191b97c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19140ae800 con 0x7f1914160530 2023-10-16T12:47:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f191b97c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19140ae1f0 con 0x7f1914069fa0 2023-10-16T12:47:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f1919718640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1914160530 0x7f19141609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f1919718640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1914160530 0x7f19141609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34334/0 (socket says 172.21.15.60:34334) 2023-10-16T12:47:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.658+0000 7f1919718640 1 -- 172.21.15.60:0/3292774891 learned_addr learned my addr 172.21.15.60:0/3292774891 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.659+0000 7f1919718640 1 -- 172.21.15.60:0/3292774891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914069fa0 msgr2=0x7f191406a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.659+0000 7f1919718640 1 --2- 172.21.15.60:0/3292774891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914069fa0 0x7f191406a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.659+0000 7f1919718640 1 -- 172.21.15.60:0/3292774891 >> v1:172.21.15.144:6789/0 conn(0x7f1914065ab0 legacy=0x7f1914160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.659+0000 7f1919718640 1 -- 172.21.15.60:0/3292774891 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140acae0 con 0x7f1914160530 2023-10-16T12:47:26.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.660+0000 7f1919718640 1 --2- 172.21.15.60:0/3292774891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914160530 0x7f19141609f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1910009870 tx=0x7f191002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3579ef22dac8d1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:26.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.660+0000 7f190bfff640 1 -- 172.21.15.60:0/3292774891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19100317c0 con 0x7f1914160530 2023-10-16T12:47:26.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.660+0000 7f190bfff640 1 -- 172.21.15.60:0/3292774891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910031920 con 0x7f1914160530 2023-10-16T12:47:26.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.660+0000 7f190bfff640 1 -- 172.21.15.60:0/3292774891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910031cb0 con 0x7f1914160530 2023-10-16T12:47:26.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 -- 172.21.15.60:0/3292774891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914160530 msgr2=0x7f19141609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 --2- 172.21.15.60:0/3292774891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914160530 0x7f19141609f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1910009870 tx=0x7f191002b860 comp rx=0 tx=0).stop 2023-10-16T12:47:26.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 -- 172.21.15.60:0/3292774891 shutdown_connections 2023-10-16T12:47:26.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 --2- 172.21.15.60:0/3292774891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914069fa0 0x7f191406a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 --2- 172.21.15.60:0/3292774891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914160530 0x7f19141609f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 -- 172.21.15.60:0/3292774891 >> 172.21.15.60:0/3292774891 conn(0x7f191406bc10 msgr2=0x7f191406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:26.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 -- 172.21.15.60:0/3292774891 shutdown_connections 2023-10-16T12:47:26.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.661+0000 7f191b97c640 1 -- 172.21.15.60:0/3292774891 wait complete. 2023-10-16T12:47:26.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.663+0000 7f191b97c640 1 Processor -- start 2023-10-16T12:47:26.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.663+0000 7f191b97c640 1 -- start start 2023-10-16T12:47:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f191b97c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1914065ab0 0x7f19141f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f191b97c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 0x7f19141f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f191b97c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914160530 0x7f19140c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f191b97c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19140c4d50 con 0x7f1914069fa0 2023-10-16T12:47:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f191b97c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19141f6ff0 con 0x7f1914065ab0 2023-10-16T12:47:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f191b97c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19140c40d0 con 0x7f1914160530 2023-10-16T12:47:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f1918f17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 0x7f19141f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f1918f17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 0x7f19141f6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34350/0 (socket says 172.21.15.60:34350) 2023-10-16T12:47:26.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f1918f17640 1 -- 172.21.15.60:0/2352977610 learned_addr learned my addr 172.21.15.60:0/2352977610 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:26.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.664+0000 7f1919718640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1914065ab0 0x7f19141f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1919f19640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914160530 0x7f19140c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:26.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1918f17640 1 -- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914160530 msgr2=0x7f19140c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1918f17640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914160530 0x7f19140c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1918f17640 1 -- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1914065ab0 msgr2=0x7f19141f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1918f17640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1914065ab0 0x7f19141f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1918f17640 1 -- 172.21.15.60:0/2352977610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100094d0 con 0x7f1914069fa0 2023-10-16T12:47:26.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1919f19640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914160530 0x7f19140c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:26.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1918f17640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 0x7f19141f6ab0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f190400c430 tx=0x7f190400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:26.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1904011790 con 0x7f1914069fa0 2023-10-16T12:47:26.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19040118f0 con 0x7f1914069fa0 2023-10-16T12:47:26.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19140c43b0 con 0x7f1914069fa0 2023-10-16T12:47:26.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.665+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1904011ce0 con 0x7f1914069fa0 2023-10-16T12:47:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.666+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19140c4970 con 0x7f1914069fa0 2023-10-16T12:47:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.667+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19140c6260 con 0x7f1914069fa0 2023-10-16T12:47:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.668+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f190401c460 con 0x7f1914069fa0 2023-10-16T12:47:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.668+0000 7f1909ffb640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 0x7f18f806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.669+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1904089760 con 0x7f1914069fa0 2023-10-16T12:47:26.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.669+0000 7f1919718640 1 -- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 msgr2=0x7f18f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.669+0000 7f1919718640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 0x7f18f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:47:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.673+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19040546f0 con 0x7f1914069fa0 2023-10-16T12:47:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.868+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f191404e160 con 0x7f1914069fa0 2023-10-16T12:47:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.870+0000 7f1919718640 1 -- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 msgr2=0x7f18f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.870+0000 7f1919718640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 0x7f18f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:47:26.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.873+0000 7f1909ffb640 1 -- 172.21.15.60:0/2352977610 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1904057d10 con 0x7f1914069fa0 2023-10-16T12:47:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.876+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 msgr2=0x7f18f806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:47:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 0x7f18f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 msgr2=0x7f19141f6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 0x7f19141f6ab0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f190400c430 tx=0x7f190400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 shutdown_connections 2023-10-16T12:47:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f18f806afe0 0x7f18f806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1914160530 0x7f19140c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1914069fa0 0x7f19141f6ab0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 --2- 172.21.15.60:0/2352977610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1914065ab0 0x7f19141f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 >> 172.21.15.60:0/2352977610 conn(0x7f191406bc10 msgr2=0x7f19140b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 shutdown_connections 2023-10-16T12:47:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:26.877+0000 7f191b97c640 1 -- 172.21.15.60:0/2352977610 wait complete. 2023-10-16T12:47:26.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:26.897 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:47:26.949 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:26.946+0000 7f5a5ac38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:47:26.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T12:47:27.001 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:26.998+0000 7f5a5ac38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:47:27.046 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.045+0000 7f143c22c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:47:27.050 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.048+0000 7f5a5ac38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:47:27.102 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.100+0000 7f143c22c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:47:27.110 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.108+0000 7f5a5ac38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:47:27.155 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.154+0000 7f143c22c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:47:27.158 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.156+0000 7f5a5ac38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:47:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.179+0000 7f328e6da640 1 Processor -- start 2023-10-16T12:47:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.179+0000 7f328e6da640 1 -- start start 2023-10-16T12:47:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.180+0000 7f328e6da640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32880caac0 0x7f32880ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.180+0000 7f328e6da640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32880cb030 con 0x7f328815ad70 2023-10-16T12:47:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f328e6da640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32880c91c0 con 0x7f328806a4b0 2023-10-16T12:47:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f328e6da640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32880c94a0 con 0x7f32880caac0 2023-10-16T12:47:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f3287fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f328815ad70 0x7f3288069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51240/0 (socket says 172.21.15.60:51240) 2023-10-16T12:47:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f3287fff640 1 -- 172.21.15.60:0/5102249 learned_addr learned my addr 172.21.15.60:0/5102249 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:47:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f328cc77640 1 --2- 172.21.15.60:0/5102249 >> v2:172.21.15.60:3300/0 conn(0x7f32880caac0 0x7f32880ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f3286ffd640 1 -- 172.21.15.60:0/5102249 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589081957 0 0) 0x7f32880cb030 con 0x7f328815ad70 2023-10-16T12:47:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f3286ffd640 1 -- 172.21.15.60:0/5102249 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3270003520 con 0x7f328815ad70 2023-10-16T12:47:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f328cc77640 1 -- 172.21.15.60:0/5102249 >> v1:172.21.15.144:6789/0 conn(0x7f328806a4b0 legacy=0x7f32880ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f328cc77640 1 -- 172.21.15.60:0/5102249 >> v1:172.21.15.60:6789/0 conn(0x7f328815ad70 legacy=0x7f3288069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.181+0000 7f328cc77640 1 -- 172.21.15.60:0/5102249 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880c9780 con 0x7f32880caac0 2023-10-16T12:47:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.183+0000 7f328cc77640 1 --2- 172.21.15.60:0/5102249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880ad4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f32800097a0 tx=0x7f328002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb93b39f4db358d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.183+0000 7f3286ffd640 1 -- 172.21.15.60:0/5102249 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280031830 con 0x7f32880caac0 2023-10-16T12:47:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.183+0000 7f3286ffd640 1 -- 172.21.15.60:0/5102249 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3280031990 con 0x7f32880caac0 2023-10-16T12:47:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.183+0000 7f3286ffd640 1 -- 172.21.15.60:0/5102249 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280031d00 con 0x7f32880caac0 2023-10-16T12:47:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.183+0000 7f328e6da640 1 -- 172.21.15.60:0/5102249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 msgr2=0x7f32880ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.183+0000 7f328e6da640 1 --2- 172.21.15.60:0/5102249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880ad4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f32800097a0 tx=0x7f328002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:47:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.184+0000 7f328e6da640 1 -- 172.21.15.60:0/5102249 shutdown_connections 2023-10-16T12:47:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.184+0000 7f328e6da640 1 --2- 172.21.15.60:0/5102249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880ad4d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.184+0000 7f328e6da640 1 -- 172.21.15.60:0/5102249 >> 172.21.15.60:0/5102249 conn(0x7f32880ce9b0 msgr2=0x7f32880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:27.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.184+0000 7f328e6da640 1 -- 172.21.15.60:0/5102249 shutdown_connections 2023-10-16T12:47:27.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.184+0000 7f328e6da640 1 -- 172.21.15.60:0/5102249 wait complete. 2023-10-16T12:47:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.185+0000 7f328e6da640 1 Processor -- start 2023-10-16T12:47:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.186+0000 7f328e6da640 1 -- start start 2023-10-16T12:47:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.186+0000 7f328e6da640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f328806a4b0 0x7f32880c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.186+0000 7f328e6da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f328e6da640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328815ad70 0x7f32880aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f328e6da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32880c7f00 con 0x7f32880caac0 2023-10-16T12:47:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f328e6da640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32880c72f0 con 0x7f328815ad70 2023-10-16T12:47:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f328e6da640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880c75d0 con 0x7f328806a4b0 2023-10-16T12:47:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34378/0 (socket says 172.21.15.60:34378) 2023-10-16T12:47:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 -- 172.21.15.60:0/3696974020 learned_addr learned my addr 172.21.15.60:0/3696974020 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f328806a4b0 0x7f32880c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f328cc77640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328815ad70 0x7f32880aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f328806a4b0 msgr2=0x7f32880c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f328806a4b0 0x7f32880c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328815ad70 msgr2=0x7f32880aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328815ad70 0x7f32880aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.187+0000 7f32877fe640 1 -- 172.21.15.60:0/3696974020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3280009400 con 0x7f32880caac0 2023-10-16T12:47:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.188+0000 7f32877fe640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880c4fa0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c430 tx=0x7f327c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.188+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c011790 con 0x7f32880caac0 2023-10-16T12:47:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.188+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c0118f0 con 0x7f32880caac0 2023-10-16T12:47:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.188+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c011ce0 con 0x7f32880caac0 2023-10-16T12:47:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.188+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32880c78b0 con 0x7f32880caac0 2023-10-16T12:47:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.188+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32880c29e0 con 0x7f32880caac0 2023-10-16T12:47:27.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.189+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32880c80b0 con 0x7f32880caac0 2023-10-16T12:47:27.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.190+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f327c01c460 con 0x7f32880caac0 2023-10-16T12:47:27.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.191+0000 7f3284ff9640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 0x7f327406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:27.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.191+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f327c089700 con 0x7f32880caac0 2023-10-16T12:47:27.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.191+0000 7f3287fff640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 msgr2=0x7f327406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:27.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.191+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 0x7f327406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:47:27.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.200+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32880c80b0 con 0x7f32880caac0 2023-10-16T12:47:27.225 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.223+0000 7f143c22c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:47:27.269 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.266+0000 7f5a5ac38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:47:27.277 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.275+0000 7f143c22c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:47:27.331 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.328+0000 7f5a5ac38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:47:27.379 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.376+0000 7f5a5ac38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:47:27.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.392+0000 7f3287fff640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 msgr2=0x7f327406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:27.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.392+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 0x7f327406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:47:27.394 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.392+0000 7f143c22c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:47:27.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.422+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 --> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f32880c9ca0 con 0x7f327406b010 2023-10-16T12:47:27.436 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.433+0000 7f5a5ac38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:47:27.460 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.458+0000 7f143c22c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:47:27.485 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.482+0000 7f5a5ac38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:47:27.511 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.510+0000 7f143c22c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:47:27.570 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.568+0000 7f143c22c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:47:27.622 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:27.621+0000 7f143c22c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:47:27.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.793+0000 7f3287fff640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 msgr2=0x7f327406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:27.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:27.793+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 0x7f327406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:47:27.860 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:27.857+0000 7f5a5ac38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:47:28.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.015+0000 7f5a5ac38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:47:28.020 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.019+0000 7f143c22c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:47:28.124 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.122+0000 7f5a5ac38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:47:28.175 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.172+0000 7f5a5ac38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:47:28.191 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.190+0000 7f143c22c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:47:28.281 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.278+0000 7f5a5ac38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:47:28.305 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.303+0000 7f143c22c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:47:28.357 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.356+0000 7f143c22c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:47:28.386 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.383+0000 7f5a5ac38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:47:28.471 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.469+0000 7f143c22c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:47:28.541 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.539+0000 7f5a5ac38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:47:28.582 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.581+0000 7f143c22c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:47:28.590 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.587+0000 7f5a5ac38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:47:28.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:28.594+0000 7f3287fff640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 msgr2=0x7f327406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3875614959 2023-10-16T12:47:28.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:28.594+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 0x7f327406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:47:28.707 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:28.704+0000 7f5a5ac38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:47:28.748 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.746+0000 7f143c22c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:47:28.799 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.798+0000 7f143c22c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:47:28.925 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:28.923+0000 7f143c22c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:47:29.528 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:29.525+0000 7f5a5ac38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:47:29.795 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:29.793+0000 7f143c22c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:47:29.819 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:29.816+0000 7f5a5ac38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:47:29.876 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:29.873+0000 7f5a5ac38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:47:29.989 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:29.986+0000 7f5a5ac38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:47:30.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:30.010+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 19) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f327c054a60 con 0x7f32880caac0 2023-10-16T12:47:30.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:30.010+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 msgr2=0x7f327406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:47:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:30.010+0000 7f3284ff9640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/3875614959,v1:172.21.15.144:6833/3875614959] conn(0x7f327406b010 0x7f327406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:30.120 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:30.119+0000 7f143c22c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:47:30.185 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:30.183+0000 7f143c22c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:47:30.305 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59d5ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.306 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59d5ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.306 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59d5ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.306 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59d5ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.306 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59d5ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.307 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59da72c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.307 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59da72c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.307 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59da72c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.307 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59da72c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.307 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:47:30.303+0000 7f59da72c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:47:30.309 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:47:30.308+0000 7f143c22c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:47:31.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.024+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 20) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f327c087b80 con 0x7f32880caac0 2023-10-16T12:47:31.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.025+0000 7f3284ff9640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f32740355d0 0x7f32740359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.025+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 --> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3274035ef0 con 0x7f32740355d0 2023-10-16T12:47:31.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.033+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f32740355d0 0x7f32740359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:31.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.040+0000 7f3287fff640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f32740355d0 0x7f32740359b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3278005c00 tx=0x7f3278009040 comp rx=0 tx=0).ready entity=mgr.5111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:31.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.240+0000 7f3284ff9640 1 -- 172.21.15.60:0/3696974020 <== mgr.5111 v2:172.21.15.144:6832/745273919 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3274035ef0 con 0x7f32740355d0 2023-10-16T12:47:31.242 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1 2023-10-16T12:47:31.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f32740355d0 msgr2=0x7f32740359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f32740355d0 0x7f32740359b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3278005c00 tx=0x7f3278009040 comp rx=0 tx=0).stop 2023-10-16T12:47:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 msgr2=0x7f32880c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 --2- 172.21.15.60:0/3696974020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32880caac0 0x7f32880c4fa0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c430 tx=0x7f327c00c8f0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328cc77640 1 -- 172.21.15.60:0/3696974020 reap_dead start 2023-10-16T12:47:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 shutdown_connections 2023-10-16T12:47:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 >> 172.21.15.60:0/3696974020 conn(0x7f32880ce9b0 msgr2=0x7f32880ca310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 shutdown_connections 2023-10-16T12:47:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.244+0000 7f328e6da640 1 -- 172.21.15.60:0/3696974020 wait complete. 2023-10-16T12:47:31.265 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T12:47:31.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-10-16T12:47:31.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.490+0000 7f3ad8edf640 1 Processor -- start 2023-10-16T12:47:31.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.490+0000 7f3ad8edf640 1 -- start start 2023-10-16T12:47:31.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad8edf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad8edf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ad40b4740 0x7f3ad40b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad8edf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad40b3730 con 0x7f3ad415aa60 2023-10-16T12:47:31.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad8edf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad40b3980 con 0x7f3ad40b4740 2023-10-16T12:47:31.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad8edf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40b3c60 con 0x7f3ad40cc2e0 2023-10-16T12:47:31.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ad40b4740 0x7f3ad40b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:31.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ad40b4740 0x7f3ad40b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43002/0 (socket says 172.21.15.60:43002) 2023-10-16T12:47:31.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 -- 172.21.15.60:0/1304607111 learned_addr learned my addr 172.21.15.60:0/1304607111 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:31.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad2ffd640 1 --2- 172.21.15.60:0/1304607111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:31.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 -- 172.21.15.60:0/1304607111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 msgr2=0x7f3ad40cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:31.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 --2- 172.21.15.60:0/1304607111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40cc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 -- 172.21.15.60:0/1304607111 >> v1:172.21.15.144:6789/0 conn(0x7f3ad415aa60 legacy=0x7f3ad40af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:31.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.492+0000 7f3ad3fff640 1 -- 172.21.15.60:0/1304607111 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad40b3f40 con 0x7f3ad40b4740 2023-10-16T12:47:31.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.494+0000 7f3ad3fff640 1 --2- 172.21.15.60:0/1304607111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad40b4bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009870 tx=0x7f3acc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e690d448c50c8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:31.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.494+0000 7f3ad27fc640 1 -- 172.21.15.60:0/1304607111 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc031640 con 0x7f3ad40b4740 2023-10-16T12:47:31.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.494+0000 7f3ad27fc640 1 -- 172.21.15.60:0/1304607111 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc0317a0 con 0x7f3ad40b4740 2023-10-16T12:47:31.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.494+0000 7f3ad27fc640 1 -- 172.21.15.60:0/1304607111 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3acc031b30 con 0x7f3ad40b4740 2023-10-16T12:47:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 -- 172.21.15.60:0/1304607111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 msgr2=0x7f3ad40b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/1304607111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad40b4bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009870 tx=0x7f3acc02b8f0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 -- 172.21.15.60:0/1304607111 shutdown_connections 2023-10-16T12:47:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/1304607111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad40b4bd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/1304607111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 -- 172.21.15.60:0/1304607111 >> 172.21.15.60:0/1304607111 conn(0x7f3ad4065ed0 msgr2=0x7f3ad40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 -- 172.21.15.60:0/1304607111 shutdown_connections 2023-10-16T12:47:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.495+0000 7f3ad8edf640 1 -- 172.21.15.60:0/1304607111 wait complete. 2023-10-16T12:47:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.497+0000 7f3ad8edf640 1 Processor -- start 2023-10-16T12:47:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.498+0000 7f3ad8edf640 1 -- start start 2023-10-16T12:47:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad8edf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad8edf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad8edf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad415aa60 0x7f3ad40c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad8edf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad406b490 con 0x7f3ad40b4740 2023-10-16T12:47:31.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad8edf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad406b770 con 0x7f3ad415aa60 2023-10-16T12:47:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad8edf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad406ba50 con 0x7f3ad40cc2e0 2023-10-16T12:47:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad406aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43012/0 (socket says 172.21.15.60:43012) 2023-10-16T12:47:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.499+0000 7f3ad37fe640 1 -- 172.21.15.60:0/4104817386 learned_addr learned my addr 172.21.15.60:0/4104817386 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad2ffd640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad37fe640 1 -- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 msgr2=0x7f3ad40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad37fe640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad37fe640 1 -- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad415aa60 msgr2=0x7f3ad40c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad37fe640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad415aa60 0x7f3ad40c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad37fe640 1 -- 172.21.15.60:0/4104817386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0094d0 con 0x7f3ad40b4740 2023-10-16T12:47:31.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad37fe640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad406aeb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400ba50 tx=0x7f3ac400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400cbe0 con 0x7f3ad40b4740 2023-10-16T12:47:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac400cd80 con 0x7f3ad40b4740 2023-10-16T12:47:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4014680 con 0x7f3ad40b4740 2023-10-16T12:47:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.500+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad406bd30 con 0x7f3ad40b4740 2023-10-16T12:47:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.501+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad40c73e0 con 0x7f3ad40b4740 2023-10-16T12:47:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.502+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad404e9b0 con 0x7f3ad40b4740 2023-10-16T12:47:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.503+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3ac40147e0 con 0x7f3ad40b4740 2023-10-16T12:47:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.504+0000 7f3acbfff640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f3ab406ad40 0x7f3ab406d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.504+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3ac40891e0 con 0x7f3ad40b4740 2023-10-16T12:47:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.504+0000 7f3ad2ffd640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f3ab406ad40 0x7f3ab406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:31.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.506+0000 7f3ad2ffd640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f3ab406ad40 0x7f3ab406d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006fd0 tx=0x7f3abc008040 comp rx=0 tx=0).ready entity=mgr.5111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:31.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.509+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ac4054520 con 0x7f3ad40b4740 2023-10-16T12:47:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.733+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 --> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ad406b8e0 con 0x7f3ab406ad40 2023-10-16T12:47:31.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.900+0000 7f3acbfff640 1 -- 172.21.15.60:0/4104817386 <== mgr.5111 v2:172.21.15.144:6832/745273919 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3ad406b8e0 con 0x7f3ab406ad40 2023-10-16T12:47:31.902 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1 2023-10-16T12:47:31.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f3ab406ad40 msgr2=0x7f3ab406d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f3ab406ad40 0x7f3ab406d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006fd0 tx=0x7f3abc008040 comp rx=0 tx=0).stop 2023-10-16T12:47:31.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 msgr2=0x7f3ad406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:31.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad406aeb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400ba50 tx=0x7f3ac400bf10 comp rx=0 tx=0).stop 2023-10-16T12:47:31.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 shutdown_connections 2023-10-16T12:47:31.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f3ab406ad40 0x7f3ab406d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad40cc2e0 0x7f3ad40c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad415aa60 0x7f3ad40c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 --2- 172.21.15.60:0/4104817386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad40b4740 0x7f3ad406aeb0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:31.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 >> 172.21.15.60:0/4104817386 conn(0x7f3ad4065ed0 msgr2=0x7f3ad40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:31.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.903+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 shutdown_connections 2023-10-16T12:47:31.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:31.904+0000 7f3ad8edf640 1 -- 172.21.15.60:0/4104817386 wait complete. 2023-10-16T12:47:31.926 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T12:47:31.926 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-12_48_00 in ~29s... 2023-10-16T12:47:31.927 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-12_49_00 in ~89s... 2023-10-16T12:47:31.927 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-10-16T12:47:31.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T12:47:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.154+0000 7efdf6652640 1 Processor -- start 2023-10-16T12:47:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.154+0000 7efdf6652640 1 -- start start 2023-10-16T12:47:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.155+0000 7efdf6652640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efdf0153740 0x7efdf015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.155+0000 7efdf6652640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00cc270 0x7efdf00cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.155+0000 7efdf6652640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf015ac60 con 0x7efdf00ad4d0 2023-10-16T12:47:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf6652640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf0070030 con 0x7efdf0153740 2023-10-16T12:47:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf6652640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0070310 con 0x7efdf00cc270 2023-10-16T12:47:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efdf0153740 0x7efdf015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efdf0153740 0x7efdf015a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43026/0 (socket says 172.21.15.60:43026) 2023-10-16T12:47:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 -- 172.21.15.60:0/3996740633 learned_addr learned my addr 172.21.15.60:0/3996740633 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5e51640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00cc270 0x7efdf00cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:32.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 -- 172.21.15.60:0/3996740633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00cc270 msgr2=0x7efdf00cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:47:32.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00cc270 0x7efdf00cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 -- 172.21.15.60:0/3996740633 >> v1:172.21.15.60:6789/0 conn(0x7efdf00ad4d0 legacy=0x7efdf00ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:32.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5650640 1 -- 172.21.15.60:0/3996740633 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00705f0 con 0x7efdf0153740 2023-10-16T12:47:32.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.156+0000 7efdf5e51640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00cc270 0x7efdf00cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:47:32.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.158+0000 7efdf5650640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0153740 0x7efdf015a0e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7efdec00bb10 tx=0x7efdec02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766fbd94ea51cdcd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:32.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.158+0000 7efde7fff640 1 -- 172.21.15.60:0/3996740633 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec036440 con 0x7efdf0153740 2023-10-16T12:47:32.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.158+0000 7efde7fff640 1 -- 172.21.15.60:0/3996740633 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec0365a0 con 0x7efdf0153740 2023-10-16T12:47:32.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.158+0000 7efde7fff640 1 -- 172.21.15.60:0/3996740633 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec036930 con 0x7efdf0153740 2023-10-16T12:47:32.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.158+0000 7efdf6652640 1 -- 172.21.15.60:0/3996740633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0153740 msgr2=0x7efdf015a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:32.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.158+0000 7efdf6652640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0153740 0x7efdf015a0e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7efdec00bb10 tx=0x7efdec02e150 comp rx=0 tx=0).stop 2023-10-16T12:47:32.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.159+0000 7efdf6652640 1 -- 172.21.15.60:0/3996740633 shutdown_connections 2023-10-16T12:47:32.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.159+0000 7efdf6652640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00cc270 0x7efdf00cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.159+0000 7efdf6652640 1 --2- 172.21.15.60:0/3996740633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0153740 0x7efdf015a0e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.159+0000 7efdf6652640 1 -- 172.21.15.60:0/3996740633 >> 172.21.15.60:0/3996740633 conn(0x7efdf00ce6a0 msgr2=0x7efdf00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:32.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.159+0000 7efdf6652640 1 -- 172.21.15.60:0/3996740633 shutdown_connections 2023-10-16T12:47:32.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.159+0000 7efdf6652640 1 -- 172.21.15.60:0/3996740633 wait complete. 2023-10-16T12:47:32.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.161+0000 7efdf6652640 1 Processor -- start 2023-10-16T12:47:32.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.161+0000 7efdf6652640 1 -- start start 2023-10-16T12:47:32.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf6652640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00ad4d0 0x7efdf0151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:32.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf6652640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 0x7efdf0155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:32.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf6652640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdf0153740 0x7efdf01558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:32.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf6652640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0156d60 con 0x7efdf00cc270 2023-10-16T12:47:32.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf6652640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00b42f0 con 0x7efdf0153740 2023-10-16T12:47:32.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf6652640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0156280 con 0x7efdf00ad4d0 2023-10-16T12:47:32.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf4e4f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 0x7efdf0155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:32.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf4e4f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 0x7efdf0155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43042/0 (socket says 172.21.15.60:43042) 2023-10-16T12:47:32.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf4e4f640 1 -- 172.21.15.60:0/1418804194 learned_addr learned my addr 172.21.15.60:0/1418804194 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:47:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf5e51640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdf0153740 0x7efdf01558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.162+0000 7efdf5650640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00ad4d0 0x7efdf0151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf4e4f640 1 -- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00ad4d0 msgr2=0x7efdf0151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf4e4f640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00ad4d0 0x7efdf0151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf4e4f640 1 -- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdf0153740 msgr2=0x7efdf01558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:32.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf4e4f640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdf0153740 0x7efdf01558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf4e4f640 1 -- 172.21.15.60:0/1418804194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec00b770 con 0x7efdf00cc270 2023-10-16T12:47:32.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf5e51640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdf0153740 0x7efdf01558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:47:32.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf4e4f640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 0x7efdf0155370 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7efde000c450 tx=0x7efde000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:32.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00052d0 con 0x7efdf00cc270 2023-10-16T12:47:32.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0005430 con 0x7efdf00cc270 2023-10-16T12:47:32.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0012870 con 0x7efdf00cc270 2023-10-16T12:47:32.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf0156560 con 0x7efdf00cc270 2023-10-16T12:47:32.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.163+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf0152460 con 0x7efdf00cc270 2023-10-16T12:47:32.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.165+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf0156a80 con 0x7efdf00cc270 2023-10-16T12:47:32.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.166+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 21) v1 ==== 89349+0+0 (secure 0 0 0) 0x7efde00129d0 con 0x7efdf00cc270 2023-10-16T12:47:32.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.166+0000 7efde5ffb640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efdd406afe0 0x7efdd406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:47:32.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.167+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6053+0+0 (secure 0 0 0) 0x7efde0089530 con 0x7efdf00cc270 2023-10-16T12:47:32.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.167+0000 7efdf5650640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efdd406afe0 0x7efdd406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:47:32.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.168+0000 7efdf5650640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efdd406afe0 0x7efdd406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efdec038000 tx=0x7efdec032000 comp rx=0 tx=0).ready entity=mgr.5111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:47:32.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.173+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efdf0156a80 con 0x7efdf00cc270 2023-10-16T12:47:32.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.395+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 --> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efdf00708d0 con 0x7efdd406afe0 2023-10-16T12:47:32.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.411+0000 7efde5ffb640 1 -- 172.21.15.60:0/1418804194 <== mgr.5111 v2:172.21.15.144:6832/745273919 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7efdec036710 con 0x7efdd406afe0 2023-10-16T12:47:32.413 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:47:32.413 INFO:teuthology.orchestra.run.smithi060.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T12:47:32.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efdd406afe0 msgr2=0x7efdd406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:32.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efdd406afe0 0x7efdd406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efdec038000 tx=0x7efdec032000 comp rx=0 tx=0).stop 2023-10-16T12:47:32.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 msgr2=0x7efdf0155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:47:32.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 0x7efdf0155370 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7efde000c450 tx=0x7efde000c910 comp rx=0 tx=0).stop 2023-10-16T12:47:32.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 shutdown_connections 2023-10-16T12:47:32.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efdd406afe0 0x7efdd406d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdf0153740 0x7efdf01558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00cc270 0x7efdf0155370 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 --2- 172.21.15.60:0/1418804194 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efdf00ad4d0 0x7efdf0151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:47:32.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 >> 172.21.15.60:0/1418804194 conn(0x7efdf00ce6a0 msgr2=0x7efdf00ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:47:32.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 shutdown_connections 2023-10-16T12:47:32.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:47:32.414+0000 7efdf6652640 1 -- 172.21.15.60:0/1418804194 wait complete. 2023-10-16T12:47:32.434 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T12:47:32.434 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-10-16T12:47:32.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:32.435 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:32.493 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:32.493 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:32.494 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:33.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:33.495 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:33.563 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:33.563 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:33.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:34.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:34.565 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:34.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:34.632 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:34.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:35.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:35.635 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:35.702 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:35.703 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:35.703 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:36.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:36.705 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:36.775 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:36.775 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:36.775 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:37.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:37.777 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:37.846 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:37.846 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:37.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:38.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:38.848 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:38.916 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:38.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:38.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:39.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:39.919 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:39.985 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:39.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:39.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:40.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:40.988 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:41.058 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:41.058 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:41.059 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:42.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:42.061 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:42.128 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:42.128 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:42.129 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:43.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:43.133 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:43.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:43.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:43.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:44.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:44.202 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:44.271 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:44.271 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:44.271 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:45.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:45.273 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:45.341 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:45.341 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:45.342 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:46.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:46.344 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:46.412 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:46.413 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:46.413 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:47.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:47.416 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:47.484 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:47.484 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:47.485 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:48.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:48.486 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:48.552 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:48.553 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:48.553 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:49.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:49.554 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:49.621 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:49.621 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:49.622 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:50.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:50.624 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:50.693 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:50.693 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:50.694 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:51.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:51.695 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:51.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:51.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:51.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:52.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:52.765 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:52.833 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:52.834 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:52.834 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:53.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:53.836 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:53.899 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:53.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:53.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:54.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:54.902 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:54.970 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:54.970 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:54.970 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:55.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:55.972 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:56.038 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:56.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:56.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:56.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:56.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:57.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:57.041 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:57.109 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:57.109 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:57.109 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:58.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:58.111 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:58.178 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:58.179 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:58.179 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:47:59.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:47:59.181 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:47:59.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:47:59.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:47:59.249 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:00.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:00.251 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:00.318 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:48:00.318 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:00.319 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:00.923 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:48:00.922+0000 7f59c96ca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:48:00.924 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:48:00.922+0000 7f59c96ca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:48:00.924 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:48:00.922+0000 7f59c96ca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:48:00.924 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:48:00.922+0000 7f59c96ca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:48:00.924 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:48:00.922+0000 7f59c96ca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:48:01.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:01.320 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:01.388 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:01.389 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:01.389 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:01.390 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:01.390 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-12_48_00_UTC'] 2023-10-16T12:48:01.391 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697460451.9266763, now = 1697460481.3909986, timo = 29 2023-10-16T12:48:02.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:02.393 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:02.459 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:02.461 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:02.461 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:02.461 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:03.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:03.463 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:03.529 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:03.530 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:03.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:03.531 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:04.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:04.533 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:04.599 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:04.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:04.601 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:04.601 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:05.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:05.603 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:05.669 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:05.670 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:05.670 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:05.671 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:06.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:06.672 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:06.740 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:06.741 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:06.741 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:06.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:07.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:07.744 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:07.810 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:07.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:07.812 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:07.812 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:08.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:08.813 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:08.879 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:08.880 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:08.881 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:08.881 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:09.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:09.883 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:09.948 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:09.949 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:09.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:09.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:10.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:10.951 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:11.016 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:11.018 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:11.018 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:11.018 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:12.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:12.020 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:12.086 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:12.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:12.088 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:12.088 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:13.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:13.091 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:13.157 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:13.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:13.159 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:13.159 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:14.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:14.160 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:14.225 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:14.226 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:14.227 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:14.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:15.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:15.229 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:15.294 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:15.296 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:15.296 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:15.297 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:16.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:16.298 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:16.365 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:16.366 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:16.367 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:16.367 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:17.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:17.369 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:17.435 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:17.436 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:17.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:17.437 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:18.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:18.439 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:18.505 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:18.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:18.507 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:18.507 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:19.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:19.509 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:19.576 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:19.577 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:19.577 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:19.578 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:20.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:20.580 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:20.644 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:20.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:20.646 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:20.646 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:21.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:21.648 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:21.714 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:21.716 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:21.716 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:21.716 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:22.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:22.718 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:22.786 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:22.787 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:22.788 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:22.788 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:23.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:23.790 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:23.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:23.858 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:23.859 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:23.859 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:24.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:24.860 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:24.926 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:24.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:24.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:24.928 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:25.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:25.929 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:25.994 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:25.996 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:25.996 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:25.997 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:26.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:26.190 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:26.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:26.999 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:27.063 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:27.064 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:27.065 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:27.065 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:28.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:28.067 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:28.130 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:28.131 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:28.131 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:28.132 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:29.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:29.133 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:29.201 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:29.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:29.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:29.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:30.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:30.205 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:30.272 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:30.273 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:30.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:30.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:31.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:31.276 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:31.342 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:31.344 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:31.344 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:31.344 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:32.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:32.346 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:32.410 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:32.412 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:32.412 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:32.412 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:33.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:33.414 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:33.479 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:33.480 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:33.481 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:33.481 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:34.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:34.483 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:34.550 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:34.551 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:34.552 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:34.552 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:35.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:35.554 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:35.619 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:35.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:35.621 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:35.621 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:36.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:36.623 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:36.691 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:36.692 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:36.693 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:36.693 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:37.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:37.695 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:37.760 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:37.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:37.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:37.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:38.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:38.764 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:38.830 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:38.832 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:38.832 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:38.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:39.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:39.834 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:39.896 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:39.897 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:39.898 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:39.898 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:40.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:40.901 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:40.968 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:40.970 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:40.970 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:40.970 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:41.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:41.972 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:42.042 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:42.043 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:42.043 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:42.044 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:43.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:43.046 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:43.111 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:43.112 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:43.113 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:43.113 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:44.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:44.115 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:44.182 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:44.184 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:44.184 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:44.185 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:45.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:45.186 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:45.253 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:45.255 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:45.255 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:45.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:46.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:46.257 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:46.324 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:46.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:46.325 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:46.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:47.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:47.327 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:47.395 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:47.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:47.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:47.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:48.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:48.398 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:48.464 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:48.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:48.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:48.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:49.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:49.468 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:49.532 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:49.533 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:49.533 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:49.534 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:50.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:50.535 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:50.601 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:50.603 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:50.603 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:50.604 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:51.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:51.605 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:51.668 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:51.670 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:51.670 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:51.670 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:52.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:52.675 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:52.741 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:52.743 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:52.743 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:52.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:53.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:53.745 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:53.811 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:53.812 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:53.813 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:53.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:54.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:54.815 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:54.881 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:54.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:54.883 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:54.883 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:55.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:55.885 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:55.951 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:55.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:55.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:55.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:56.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:56.343 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:56.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:56.955 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:57.021 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:57.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:57.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:57.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:58.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:58.025 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:58.090 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:58.092 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:58.092 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:58.092 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:48:59.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:48:59.094 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:48:59.160 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:48:59.161 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:48:59.162 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:48:59.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:49:00.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:00.164 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:49:00.229 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:49:00.229 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_49_00_UTC 2023-10-16T12:49:00.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_49_00_UTC', 'scheduled-2023-10-16-12_48_00_UTC'} 2023-10-16T12:49:00.231 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-12_49_00_UTC'} 2023-10-16T12:49:00.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:49:00.231 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-12_49_00_UTC'] 2023-10-16T12:49:00.232 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697460451.9266763, now = 1697460540.232237, timo = 89 2023-10-16T12:49:01.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T12:49:01.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.475+0000 7f7d691ac640 1 Processor -- start 2023-10-16T12:49:01.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.476+0000 7f7d691ac640 1 -- start start 2023-10-16T12:49:01.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d691ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d691ac640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d6406c100 0x7f7d6406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d691ac640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6415aef0 con 0x7f7d64159950 2023-10-16T12:49:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d691ac640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6415e490 con 0x7f7d6406c100 2023-10-16T12:49:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d691ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6415d910 con 0x7f7d64070b60 2023-10-16T12:49:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d63fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7d64159950 0x7f7d64070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45184/0 (socket says 172.21.15.60:45184) 2023-10-16T12:49:01.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d63fff640 1 -- 172.21.15.60:0/4085315540 learned_addr learned my addr 172.21.15.60:0/4085315540 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:01.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.477+0000 7f7d689ab640 1 --2- 172.21.15.60:0/4085315540 >> v2:172.21.15.60:3300/0 conn(0x7f7d6406c100 0x7f7d6406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d5bfff640 1 --2- 172.21.15.60:0/4085315540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d637fe640 1 -- 172.21.15.60:0/4085315540 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443012042 0 0) 0x7f7d54001110 con 0x7f7d64159950 2023-10-16T12:49:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d637fe640 1 -- 172.21.15.60:0/4085315540 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50003520 con 0x7f7d64159950 2023-10-16T12:49:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d689ab640 1 -- 172.21.15.60:0/4085315540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 msgr2=0x7f7d6415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d689ab640 1 --2- 172.21.15.60:0/4085315540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d689ab640 1 -- 172.21.15.60:0/4085315540 >> v1:172.21.15.60:6789/0 conn(0x7f7d64159950 legacy=0x7f7d64070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.478+0000 7f7d689ab640 1 -- 172.21.15.60:0/4085315540 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6415dbf0 con 0x7f7d6406c100 2023-10-16T12:49:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.479+0000 7f7d689ab640 1 --2- 172.21.15.60:0/4085315540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d6406c550 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009dc0 tx=0x7f7d5c02bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1128d730df32e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d637fe640 1 -- 172.21.15.60:0/4085315540 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c02cba0 con 0x7f7d6406c100 2023-10-16T12:49:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d637fe640 1 -- 172.21.15.60:0/4085315540 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c02cd00 con 0x7f7d6406c100 2023-10-16T12:49:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d637fe640 1 -- 172.21.15.60:0/4085315540 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c034640 con 0x7f7d6406c100 2023-10-16T12:49:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 -- 172.21.15.60:0/4085315540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 msgr2=0x7f7d6406c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 --2- 172.21.15.60:0/4085315540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d6406c550 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009dc0 tx=0x7f7d5c02bdf0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 -- 172.21.15.60:0/4085315540 shutdown_connections 2023-10-16T12:49:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 --2- 172.21.15.60:0/4085315540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d6406c550 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 --2- 172.21.15.60:0/4085315540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 -- 172.21.15.60:0/4085315540 >> 172.21.15.60:0/4085315540 conn(0x7f7d640ce6a0 msgr2=0x7f7d640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 -- 172.21.15.60:0/4085315540 shutdown_connections 2023-10-16T12:49:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.480+0000 7f7d691ac640 1 -- 172.21.15.60:0/4085315540 wait complete. 2023-10-16T12:49:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.482+0000 7f7d691ac640 1 Processor -- start 2023-10-16T12:49:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.482+0000 7f7d691ac640 1 -- start start 2023-10-16T12:49:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d691ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d641505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d691ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6414f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:01.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d691ac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d64159950 0x7f7d6415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d691ac640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d64150430 con 0x7f7d6406c100 2023-10-16T12:49:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d691ac640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6414f850 con 0x7f7d64159950 2023-10-16T12:49:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d691ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6414fb30 con 0x7f7d64070b60 2023-10-16T12:49:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.483+0000 7f7d63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d641505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:01.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d641505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52610/0 (socket says 172.21.15.60:52610) 2023-10-16T12:49:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 -- 172.21.15.60:0/3298526988 learned_addr learned my addr 172.21.15.60:0/3298526988 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d5bfff640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6414f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 -- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 msgr2=0x7f7d6414f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6414f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d689ab640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d64159950 0x7f7d6415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:01.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 -- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d64159950 msgr2=0x7f7d6415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:01.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d64159950 0x7f7d6415b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 -- 172.21.15.60:0/3298526988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c009a70 con 0x7f7d6406c100 2023-10-16T12:49:01.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.484+0000 7f7d63fff640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d641505c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400c390 tx=0x7f7d5400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.485+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54005ce0 con 0x7f7d6406c100 2023-10-16T12:49:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.485+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54005e60 con 0x7f7d6406c100 2023-10-16T12:49:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.485+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54007310 con 0x7f7d6406c100 2023-10-16T12:49:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.485+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6414fe10 con 0x7f7d6406c100 2023-10-16T12:49:01.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.485+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6415b410 con 0x7f7d6406c100 2023-10-16T12:49:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.486+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6404e9b0 con 0x7f7d6406c100 2023-10-16T12:49:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.487+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 23) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7d54007470 con 0x7f7d6406c100 2023-10-16T12:49:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.488+0000 7f7d617fa640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f7d4406b150 0x7f7d4406d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.488+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f7d54089160 con 0x7f7d6406c100 2023-10-16T12:49:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.489+0000 7f7d5bfff640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f7d4406b150 0x7f7d4406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.490+0000 7f7d5bfff640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f7d4406b150 0x7f7d4406d610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d640c5610 tx=0x7f7d4c008040 comp rx=0 tx=0).ready entity=mgr.5111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.493+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d54053c70 con 0x7f7d6406c100 2023-10-16T12:49:01.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.711+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 --> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d6415e1b0 con 0x7f7d4406b150 2023-10-16T12:49:01.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.911+0000 7f7d617fa640 1 -- 172.21.15.60:0/3298526988 <== mgr.5111 v2:172.21.15.144:6832/745273919 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7d4c009d60 con 0x7f7d4406b150 2023-10-16T12:49:01.913 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path snap_test_dir1 2023-10-16T12:49:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f7d4406b150 msgr2=0x7f7d4406d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f7d4406b150 0x7f7d4406d610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d640c5610 tx=0x7f7d4c008040 comp rx=0 tx=0).stop 2023-10-16T12:49:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 msgr2=0x7f7d641505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d641505c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400c390 tx=0x7f7d5400c850 comp rx=0 tx=0).stop 2023-10-16T12:49:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 shutdown_connections 2023-10-16T12:49:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f7d4406b150 0x7f7d4406d610 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7d64070b60 0x7f7d6414f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d64159950 0x7f7d6415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 --2- 172.21.15.60:0/3298526988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d6406c100 0x7f7d641505c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.914+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 >> 172.21.15.60:0/3298526988 conn(0x7f7d640ce6a0 msgr2=0x7f7d6406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.915+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 shutdown_connections 2023-10-16T12:49:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:01.915+0000 7f7d691ac640 1 -- 172.21.15.60:0/3298526988 wait complete. 2023-10-16T12:49:01.938 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T12:49:01.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:01.939 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:49:01.996 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:49:01.996 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_49_00_UTC 2023-10-16T12:49:01.997 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_48_00_UTC 2023-10-16T12:49:01.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:01.998 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_48_00_UTC) 2023-10-16T12:49:02.069 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_49_00_UTC 2023-10-16T12:49:02.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:02.070 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_49_00_UTC) 2023-10-16T12:49:02.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:02.147 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T12:49:02.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:49:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.454+0000 7fd1ae9a9640 1 Processor -- start 2023-10-16T12:49:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.454+0000 7fd1ae9a9640 1 -- start start 2023-10-16T12:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1ae9a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1ae9a9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1ae9a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1ae9a9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a806b1e0 con 0x7fd1a806c7d0 2023-10-16T12:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1ae9a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a80d0890 con 0x7fd1a815d330 2023-10-16T12:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1ae9a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80d05b0 con 0x7fd1a806cc80 2023-10-16T12:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1a77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1a77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52614/0 (socket says 172.21.15.60:52614) 2023-10-16T12:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.456+0000 7fd1a77fe640 1 -- 172.21.15.60:0/301494132 learned_addr learned my addr 172.21.15.60:0/301494132 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1a7fff640 1 --2- 172.21.15.60:0/301494132 >> v2:172.21.15.144:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1acf46640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1a77fe640 1 -- 172.21.15.60:0/301494132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a806cc80 msgr2=0x7fd1a8065ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1a77fe640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1a77fe640 1 -- 172.21.15.60:0/301494132 >> v2:172.21.15.144:3300/0 conn(0x7fd1a815d330 msgr2=0x7fd1a81618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1a77fe640 1 --2- 172.21.15.60:0/301494132 >> v2:172.21.15.144:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1a77fe640 1 -- 172.21.15.60:0/301494132 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8160dd0 con 0x7fd1a806c7d0 2023-10-16T12:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.457+0000 7fd1acf46640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.458+0000 7fd1a77fe640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80744f0 tx=0x7fd19002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d4a52c4dfeee49 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:02.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.459+0000 7fd1a6ffd640 1 -- 172.21.15.60:0/301494132 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900317e0 con 0x7fd1a806c7d0 2023-10-16T12:49:02.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.459+0000 7fd1a6ffd640 1 -- 172.21.15.60:0/301494132 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd190031940 con 0x7fd1a806c7d0 2023-10-16T12:49:02.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.459+0000 7fd1a6ffd640 1 -- 172.21.15.60:0/301494132 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190031cd0 con 0x7fd1a806c7d0 2023-10-16T12:49:02.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.459+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/301494132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806c7d0 msgr2=0x7fd1a8065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:02.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.459+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80744f0 tx=0x7fd19002b880 comp rx=0 tx=0).stop 2023-10-16T12:49:02.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/301494132 shutdown_connections 2023-10-16T12:49:02.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a806cc80 0x7fd1a8065ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/301494132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806c7d0 0x7fd1a8065ab0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/301494132 >> v2:172.21.15.144:3300/0 conn(0x7fd1a815d330 0x7fd1a81618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/301494132 >> 172.21.15.60:0/301494132 conn(0x7fd1a80d0ab0 msgr2=0x7fd1a80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:02.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/301494132 shutdown_connections 2023-10-16T12:49:02.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.460+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/301494132 wait complete. 2023-10-16T12:49:02.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.462+0000 7fd1ae9a9640 1 Processor -- start 2023-10-16T12:49:02.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.462+0000 7fd1ae9a9640 1 -- start start 2023-10-16T12:49:02.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1ae9a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a806c7d0 0x7fd1a815f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1ae9a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 0x7fd1a80b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1ae9a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a815d330 0x7fd1a80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1ae9a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a815f9a0 con 0x7fd1a806cc80 2023-10-16T12:49:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1ae9a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a815fc80 con 0x7fd1a806c7d0 2023-10-16T12:49:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1ae9a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a815ff60 con 0x7fd1a815d330 2023-10-16T12:49:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1a77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 0x7fd1a80b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1a77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 0x7fd1a80b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52630/0 (socket says 172.21.15.60:52630) 2023-10-16T12:49:02.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1a77fe640 1 -- 172.21.15.60:0/2086807075 learned_addr learned my addr 172.21.15.60:0/2086807075 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:02.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1a7fff640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a806c7d0 0x7fd1a815f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:02.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1a77fe640 1 -- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a815d330 msgr2=0x7fd1a80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:02.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.463+0000 7fd1a77fe640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a815d330 0x7fd1a80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a77fe640 1 -- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a806c7d0 msgr2=0x7fd1a815f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:02.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a77fe640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a806c7d0 0x7fd1a815f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a77fe640 1 -- 172.21.15.60:0/2086807075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1900094d0 con 0x7fd1a806cc80 2023-10-16T12:49:02.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a77fe640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 0x7fd1a80b23d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd190000c00 tx=0x7fd1900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:02.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190032880 con 0x7fd1a806cc80 2023-10-16T12:49:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd190032a20 con 0x7fd1a806cc80 2023-10-16T12:49:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190032dd0 con 0x7fd1a806cc80 2023-10-16T12:49:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a80c4f50 con 0x7fd1a806cc80 2023-10-16T12:49:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.464+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a815f6b0 con 0x7fd1a806cc80 2023-10-16T12:49:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.465+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a80c5470 con 0x7fd1a806cc80 2023-10-16T12:49:02.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.466+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 23) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd19004e450 con 0x7fd1a806cc80 2023-10-16T12:49:02.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.467+0000 7fd1a4ff9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fd18806b130 0x7fd18806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:02.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.467+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fd1900abef0 con 0x7fd1a806cc80 2023-10-16T12:49:02.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.468+0000 7fd1a7fff640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fd18806b130 0x7fd18806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:02.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.469+0000 7fd1a7fff640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fd18806b130 0x7fd18806d5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd198009e10 tx=0x7fd19800b040 comp rx=0 tx=0).ready entity=mgr.5111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:02.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.472+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1a80c5470 con 0x7fd1a806cc80 2023-10-16T12:49:02.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.719+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd1a80c5470 con 0x7fd1a806cc80 2023-10-16T12:49:02.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.813+0000 7fd1a4ff9640 1 -- 172.21.15.60:0/2086807075 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v24) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd1a80c5470 con 0x7fd1a806cc80 2023-10-16T12:49:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fd18806b130 msgr2=0x7fd18806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fd18806b130 0x7fd18806d5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd198009e10 tx=0x7fd19800b040 comp rx=0 tx=0).stop 2023-10-16T12:49:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 msgr2=0x7fd1a80b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 0x7fd1a80b23d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd190000c00 tx=0x7fd1900046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 shutdown_connections 2023-10-16T12:49:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fd18806b130 0x7fd18806d5f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a815d330 0x7fd1a80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a806cc80 0x7fd1a80b23d0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 --2- 172.21.15.60:0/2086807075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a806c7d0 0x7fd1a815f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.819+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 >> 172.21.15.60:0/2086807075 conn(0x7fd1a80d0ab0 msgr2=0x7fd1a8065ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.820+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 shutdown_connections 2023-10-16T12:49:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:02.820+0000 7fd1ae9a9640 1 -- 172.21.15.60:0/2086807075 wait complete. 2023-10-16T12:49:02.844 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:49:02.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:02.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:03.044 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:49:03.045 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:03.045 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:03.100 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:03.100 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:03.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:03.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:03.344+0000 7ff67183e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:49:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.362+0000 7fe85a415640 1 Processor -- start 2023-10-16T12:49:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.362+0000 7fe85a415640 1 -- start start 2023-10-16T12:49:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.363+0000 7fe85a415640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 0x7fe8540b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.363+0000 7fe85a415640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85415c1b0 con 0x7fe85406fa90 2023-10-16T12:49:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.363+0000 7fe85a415640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540ce2d0 con 0x7fe85415ace0 2023-10-16T12:49:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.363+0000 7fe85a415640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540b63d0 con 0x7fe85415b0c0 2023-10-16T12:49:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.363+0000 7fe8589b2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe85406fa90 0x7fe8540cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45190/0 (socket says 172.21.15.60:45190) 2023-10-16T12:49:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.363+0000 7fe8589b2640 1 -- 172.21.15.60:0/210241089 learned_addr learned my addr 172.21.15.60:0/210241089 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe8537fe640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 0x7fe8540b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919109454 0 0) 0x7fe85415c1b0 con 0x7fe85406fa90 2023-10-16T12:49:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe83c003520 con 0x7fe85406fa90 2023-10-16T12:49:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714675277 0 0) 0x7fe83c003520 con 0x7fe85406fa90 2023-10-16T12:49:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 msgr2=0x7fe8540b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 0x7fe8540b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 >> v1:172.21.15.144:6789/0 conn(0x7fe85415ace0 legacy=0x7fe85415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540b66b0 con 0x7fe85406fa90 2023-10-16T12:49:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe84c001670 con 0x7fe85406fa90 2023-10-16T12:49:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.364+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 >> v1:172.21.15.60:6789/0 conn(0x7fe85406fa90 legacy=0x7fe8540cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe852ffd640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 0x7fe83c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe852ffd640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83c003ee0 0x7fe83c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe852ffd640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe83c004d00 0x7fe83c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540b63d0 con 0x7fe83c004d00 2023-10-16T12:49:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c00bac0 con 0x7fe83c003ee0 2023-10-16T12:49:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c00bc80 con 0x7fe85415b0c0 2023-10-16T12:49:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe8589b2640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe83c004d00 0x7fe83c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe8537fe640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83c003ee0 0x7fe83c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe853fff640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 0x7fe83c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe8589b2640 1 -- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 msgr2=0x7fe83c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe8589b2640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe85415b0c0 0x7fe83c0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.365+0000 7fe8589b2640 1 -- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83c003ee0 msgr2=0x7fe83c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.366+0000 7fe8589b2640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83c003ee0 0x7fe83c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.366+0000 7fe8589b2640 1 -- 172.21.15.60:0/210241089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c003520 con 0x7fe83c004d00 2023-10-16T12:49:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.366+0000 7fe8537fe640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83c003ee0 0x7fe83c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.367+0000 7fe8589b2640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe83c004d00 0x7fe83c005180 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c190 tx=0x7fe83c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b132cdfe38258a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.367+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c005ad0 con 0x7fe83c004d00 2023-10-16T12:49:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.367+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c005c30 con 0x7fe83c004d00 2023-10-16T12:49:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.367+0000 7fe852ffd640 1 -- 172.21.15.60:0/210241089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c02c810 con 0x7fe83c004d00 2023-10-16T12:49:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe85a415640 1 -- 172.21.15.60:0/210241089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe83c004d00 msgr2=0x7fe83c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe85a415640 1 --2- 172.21.15.60:0/210241089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe83c004d00 0x7fe83c005180 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c190 tx=0x7fe83c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe8589b2640 1 -- 172.21.15.60:0/210241089 reap_dead start 2023-10-16T12:49:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe85a415640 1 -- 172.21.15.60:0/210241089 shutdown_connections 2023-10-16T12:49:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe85a415640 1 -- 172.21.15.60:0/210241089 >> 172.21.15.60:0/210241089 conn(0x7fe85406d700 msgr2=0x7fe85406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe85a415640 1 -- 172.21.15.60:0/210241089 shutdown_connections 2023-10-16T12:49:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.368+0000 7fe85a415640 1 -- 172.21.15.60:0/210241089 wait complete. 2023-10-16T12:49:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.370+0000 7fe85a415640 1 Processor -- start 2023-10-16T12:49:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.370+0000 7fe85a415640 1 -- start start 2023-10-16T12:49:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe85a415640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe83c004d00 0x7fe854151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe85a415640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 0x7fe8540ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe85a415640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe85415d400 0x7fe8541f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe85a415640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85415d000 con 0x7fe85415bea0 2023-10-16T12:49:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe85a415640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85415c380 con 0x7fe85415d400 2023-10-16T12:49:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe85a415640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85415c660 con 0x7fe83c004d00 2023-10-16T12:49:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe8537fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 0x7fe8540ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe8537fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 0x7fe8540ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52670/0 (socket says 172.21.15.60:52670) 2023-10-16T12:49:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.371+0000 7fe8537fe640 1 -- 172.21.15.60:0/2226670682 learned_addr learned my addr 172.21.15.60:0/2226670682 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8589b2640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe85415d400 0x7fe8541f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe853fff640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe83c004d00 0x7fe854151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8537fe640 1 -- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe83c004d00 msgr2=0x7fe854151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8537fe640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe83c004d00 0x7fe854151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8537fe640 1 -- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe85415d400 msgr2=0x7fe8541f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8537fe640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe85415d400 0x7fe8541f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8537fe640 1 -- 172.21.15.60:0/2226670682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85415c940 con 0x7fe85415bea0 2023-10-16T12:49:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe8537fe640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 0x7fe8540ca320 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe8480116d0 tx=0x7fe848011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800f9c0 con 0x7fe85415bea0 2023-10-16T12:49:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84800fb20 con 0x7fe85415bea0 2023-10-16T12:49:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800fed0 con 0x7fe85415bea0 2023-10-16T12:49:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.372+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85415cc20 con 0x7fe85415bea0 2023-10-16T12:49:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.373+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85415d870 con 0x7fe85415bea0 2023-10-16T12:49:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.374+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85404e160 con 0x7fe85415bea0 2023-10-16T12:49:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.375+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe848015460 con 0x7fe85415bea0 2023-10-16T12:49:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.376+0000 7fe850ff9640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 0x7fe84006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.376+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fe84808a210 con 0x7fe85415bea0 2023-10-16T12:49:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.376+0000 7fe853fff640 1 -- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 msgr2=0x7fe84006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.376+0000 7fe853fff640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 0x7fe84006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:03.387 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:03.379+0000 7fc60d83a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:49:03.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.384+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8480550b0 con 0x7fe85415bea0 2023-10-16T12:49:03.400 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:03.399+0000 7ff67183e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:49:03.439 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:03.438+0000 7fc60d83a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:49:03.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.577+0000 7fe853fff640 1 -- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 msgr2=0x7fe84006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:03.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.577+0000 7fe853fff640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 0x7fe84006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:03.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.581+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe85406e3a0 con 0x7fe85415bea0 2023-10-16T12:49:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.583+0000 7fe850ff9640 1 -- 172.21.15.60:0/2226670682 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17098 (secure 0 0 0) 0x7fe8480586d0 con 0x7fe85415bea0 2023-10-16T12:49:03.585 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:03.585 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:48:31.745474+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6832/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/788226979":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3543205521":"2023-10-17T12:47:29.992623+0000","172.21.15.144:6833/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/411702881":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3534362968":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/2319504696":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3965421321":"2023-10-17T12:47:29.992623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:03.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.585+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 msgr2=0x7fe84006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:03.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.585+0000 7fe85a415640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 0x7fe84006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 msgr2=0x7fe8540ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 0x7fe8540ca320 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe8480116d0 tx=0x7fe848011b90 comp rx=0 tx=0).stop 2023-10-16T12:49:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 shutdown_connections 2023-10-16T12:49:03.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fe84006af90 0x7fe84006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe85415d400 0x7fe8541f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe85415bea0 0x7fe8540ca320 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 --2- 172.21.15.60:0/2226670682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe83c004d00 0x7fe854151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 >> 172.21.15.60:0/2226670682 conn(0x7fe85406d700 msgr2=0x7fe8540af710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 shutdown_connections 2023-10-16T12:49:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.586+0000 7fe85a415640 1 -- 172.21.15.60:0/2226670682 wait complete. 2023-10-16T12:49:03.605 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:47:13.297149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:47:14.334033+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:03.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:03.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.814+0000 7efd98a34640 1 Processor -- start 2023-10-16T12:49:03.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.814+0000 7efd98a34640 1 -- start start 2023-10-16T12:49:03.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd98a34640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd9415ad70 0x7efd94069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd98a34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efd940caac0 0x7efd9406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd98a34640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd940cb030 con 0x7efd940c9ca0 2023-10-16T12:49:03.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd98a34640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd940c91c0 con 0x7efd9415ad70 2023-10-16T12:49:03.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd98a34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd940c94a0 con 0x7efd940caac0 2023-10-16T12:49:03.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd9259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd9415ad70 0x7efd94069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd91d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd940c9ca0 0x7efd940ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45202/0 (socket says 172.21.15.60:45202) 2023-10-16T12:49:03.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.816+0000 7efd91d9b640 1 -- 172.21.15.60:0/1253900107 learned_addr learned my addr 172.21.15.60:0/1253900107 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:03.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.817+0000 7efd9259c640 1 -- 172.21.15.60:0/1253900107 >> v2:172.21.15.144:3300/0 conn(0x7efd940caac0 msgr2=0x7efd9406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:03.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.817+0000 7efd92d9d640 1 --2- 172.21.15.60:0/1253900107 >> v2:172.21.15.144:3300/0 conn(0x7efd940caac0 0x7efd9406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.817+0000 7efd9259c640 1 --2- 172.21.15.60:0/1253900107 >> v2:172.21.15.144:3300/0 conn(0x7efd940caac0 0x7efd9406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.817+0000 7efd9259c640 1 -- 172.21.15.60:0/1253900107 >> v1:172.21.15.60:6789/0 conn(0x7efd940c9ca0 legacy=0x7efd940ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.817+0000 7efd9159a640 1 -- 172.21.15.60:0/1253900107 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825908165 0 0) 0x7efd940cb030 con 0x7efd940c9ca0 2023-10-16T12:49:03.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.817+0000 7efd9259c640 1 -- 172.21.15.60:0/1253900107 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940c9780 con 0x7efd9415ad70 2023-10-16T12:49:03.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.818+0000 7efd9259c640 1 --2- 172.21.15.60:0/1253900107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd9415ad70 0x7efd94069da0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efd9406d090 tx=0x7efd8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705adcd5f947135a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:03.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd9159a640 1 -- 172.21.15.60:0/1253900107 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84031830 con 0x7efd9415ad70 2023-10-16T12:49:03.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd9159a640 1 -- 172.21.15.60:0/1253900107 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd84031990 con 0x7efd9415ad70 2023-10-16T12:49:03.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd9159a640 1 -- 172.21.15.60:0/1253900107 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84031d20 con 0x7efd9415ad70 2023-10-16T12:49:03.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd98a34640 1 -- 172.21.15.60:0/1253900107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd9415ad70 msgr2=0x7efd94069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd98a34640 1 --2- 172.21.15.60:0/1253900107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd9415ad70 0x7efd94069da0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efd9406d090 tx=0x7efd8402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd98a34640 1 -- 172.21.15.60:0/1253900107 shutdown_connections 2023-10-16T12:49:03.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.819+0000 7efd98a34640 1 --2- 172.21.15.60:0/1253900107 >> v2:172.21.15.144:3300/0 conn(0x7efd940caac0 0x7efd9406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.820+0000 7efd98a34640 1 --2- 172.21.15.60:0/1253900107 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd9415ad70 0x7efd94069da0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.820+0000 7efd98a34640 1 -- 172.21.15.60:0/1253900107 >> 172.21.15.60:0/1253900107 conn(0x7efd940ce9b0 msgr2=0x7efd940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.820+0000 7efd98a34640 1 -- 172.21.15.60:0/1253900107 shutdown_connections 2023-10-16T12:49:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.820+0000 7efd98a34640 1 -- 172.21.15.60:0/1253900107 wait complete. 2023-10-16T12:49:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.821+0000 7efd98a34640 1 Processor -- start 2023-10-16T12:49:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.822+0000 7efd98a34640 1 -- start start 2023-10-16T12:49:03.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd98a34640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd940c9ca0 0x7efd94158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd98a34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 0x7efd940c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd98a34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9415ad70 0x7efd940c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd98a34640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd941f26a0 con 0x7efd940caac0 2023-10-16T12:49:03.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd98a34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd941f2980 con 0x7efd9415ad70 2023-10-16T12:49:03.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd98a34640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd941f2c60 con 0x7efd940c9ca0 2023-10-16T12:49:03.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 0x7efd940c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 0x7efd940c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52698/0 (socket says 172.21.15.60:52698) 2023-10-16T12:49:03.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 -- 172.21.15.60:0/3804583250 learned_addr learned my addr 172.21.15.60:0/3804583250 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:03.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd9259c640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd940c9ca0 0x7efd94158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 -- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd940c9ca0 msgr2=0x7efd94158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd940c9ca0 0x7efd94158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 -- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9415ad70 msgr2=0x7efd940c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:03.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd92d9d640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9415ad70 0x7efd940c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:03.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9415ad70 0x7efd940c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:03.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.823+0000 7efd91d9b640 1 -- 172.21.15.60:0/3804583250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84009400 con 0x7efd940caac0 2023-10-16T12:49:03.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.824+0000 7efd91d9b640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 0x7efd940c6530 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00c4d0 tx=0x7efd7c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:03.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.824+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c0074e0 con 0x7efd940caac0 2023-10-16T12:49:03.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.824+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c007660 con 0x7efd940caac0 2023-10-16T12:49:03.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.824+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd941f2f40 con 0x7efd940caac0 2023-10-16T12:49:03.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.824+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c004a20 con 0x7efd940caac0 2023-10-16T12:49:03.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.824+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd94158840 con 0x7efd940caac0 2023-10-16T12:49:03.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.825+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd941f3130 con 0x7efd940caac0 2023-10-16T12:49:03.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.826+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7efd7c004b80 con 0x7efd940caac0 2023-10-16T12:49:03.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.827+0000 7efd8affd640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 0x7efd7406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:03.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.827+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7efd7c088dd0 con 0x7efd940caac0 2023-10-16T12:49:03.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.827+0000 7efd9259c640 1 -- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 msgr2=0x7efd7406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:03.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.828+0000 7efd9259c640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 0x7efd7406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:03.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:03.832+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd7c053c70 con 0x7efd940caac0 2023-10-16T12:49:03.843 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:03.842+0000 7ff67183e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:49:03.896 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:03.894+0000 7ff67183e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:49:03.919 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:03.918+0000 7fc60d83a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:49:03.945 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:03.944+0000 7ff67183e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:49:03.974 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:03.973+0000 7fc60d83a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:49:04.006 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.004+0000 7ff67183e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:49:04.027 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.026+0000 7fc60d83a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:49:04.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.029+0000 7efd9259c640 1 -- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 msgr2=0x7efd7406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:04.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.029+0000 7efd9259c640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 0x7efd7406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:04.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.029+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd941f27e0 con 0x7efd940caac0 2023-10-16T12:49:04.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.030+0000 7efd8affd640 1 -- 172.21.15.60:0/3804583250 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd941f27e0 con 0x7efd940caac0 2023-10-16T12:49:04.032 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:04.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 msgr2=0x7efd7406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:04.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 0x7efd7406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 msgr2=0x7efd940c6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 0x7efd940c6530 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00c4d0 tx=0x7efd7c00c990 comp rx=0 tx=0).stop 2023-10-16T12:49:04.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 shutdown_connections 2023-10-16T12:49:04.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7efd7406af90 0x7efd7406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9415ad70 0x7efd940c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd940caac0 0x7efd940c6530 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 --2- 172.21.15.60:0/3804583250 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd940c9ca0 0x7efd94158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.032+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 >> 172.21.15.60:0/3804583250 conn(0x7efd940ce9b0 msgr2=0x7efd940acce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:04.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.033+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 shutdown_connections 2023-10-16T12:49:04.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.033+0000 7efd98a34640 1 -- 172.21.15.60:0/3804583250 wait complete. 2023-10-16T12:49:04.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:49:04.054 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.053+0000 7ff67183e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:49:04.095 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.094+0000 7fc60d83a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:49:04.148 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.147+0000 7fc60d83a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:49:04.164 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.163+0000 7ff67183e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:49:04.226 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.224+0000 7ff67183e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:04.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.264+0000 7f5e3240d640 1 Processor -- start 2023-10-16T12:49:04.266 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.264+0000 7fc60d83a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:49:04.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.265+0000 7f5e3240d640 1 -- start start 2023-10-16T12:49:04.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.266+0000 7f5e3240d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e2c0cded0 0x7f5e2c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.266+0000 7f5e3240d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c15d150 con 0x7f5e2c0b8480 2023-10-16T12:49:04.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.266+0000 7f5e3240d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0d1180 con 0x7f5e2c15cd70 2023-10-16T12:49:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.266+0000 7f5e3240d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0d1460 con 0x7f5e2c0cded0 2023-10-16T12:49:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e30c0a640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e2c0b8480 0x7f5e2c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45214/0 (socket says 172.21.15.60:45214) 2023-10-16T12:49:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e30c0a640 1 -- 172.21.15.60:0/170849313 learned_addr learned my addr 172.21.15.60:0/170849313 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:04.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e31c0c640 1 --2- 172.21.15.60:0/170849313 >> v2:172.21.15.60:3300/0 conn(0x7f5e2c0cded0 0x7f5e2c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e23fff640 1 -- 172.21.15.60:0/170849313 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740574819 0 0) 0x7f5e2c15d150 con 0x7f5e2c0b8480 2023-10-16T12:49:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e23fff640 1 -- 172.21.15.60:0/170849313 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e18003520 con 0x7f5e2c0b8480 2023-10-16T12:49:04.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e31c0c640 1 -- 172.21.15.60:0/170849313 >> v1:172.21.15.144:6789/0 conn(0x7f5e2c15cd70 legacy=0x7f5e2c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e31c0c640 1 -- 172.21.15.60:0/170849313 >> v1:172.21.15.60:6789/0 conn(0x7f5e2c0b8480 legacy=0x7f5e2c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.267+0000 7f5e31c0c640 1 -- 172.21.15.60:0/170849313 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c0d1740 con 0x7f5e2c0cded0 2023-10-16T12:49:04.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.269+0000 7f5e31c0c640 1 --2- 172.21.15.60:0/170849313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c0b5c90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009d90 tx=0x7f5e2402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94589aba38f9af4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:04.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.269+0000 7f5e23fff640 1 -- 172.21.15.60:0/170849313 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2402ccf0 con 0x7f5e2c0cded0 2023-10-16T12:49:04.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.270+0000 7f5e23fff640 1 -- 172.21.15.60:0/170849313 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2402ce50 con 0x7f5e2c0cded0 2023-10-16T12:49:04.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.270+0000 7f5e23fff640 1 -- 172.21.15.60:0/170849313 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24034660 con 0x7f5e2c0cded0 2023-10-16T12:49:04.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.270+0000 7f5e3240d640 1 -- 172.21.15.60:0/170849313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0cded0 msgr2=0x7f5e2c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.270+0000 7f5e3240d640 1 --2- 172.21.15.60:0/170849313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c0b5c90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009d90 tx=0x7f5e2402bf40 comp rx=0 tx=0).stop 2023-10-16T12:49:04.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.271+0000 7f5e3240d640 1 -- 172.21.15.60:0/170849313 shutdown_connections 2023-10-16T12:49:04.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.271+0000 7f5e3240d640 1 --2- 172.21.15.60:0/170849313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c0b5c90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.271+0000 7f5e3240d640 1 -- 172.21.15.60:0/170849313 >> 172.21.15.60:0/170849313 conn(0x7f5e2c0cf570 msgr2=0x7f5e2c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:04.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.271+0000 7f5e3240d640 1 -- 172.21.15.60:0/170849313 shutdown_connections 2023-10-16T12:49:04.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.271+0000 7f5e3240d640 1 -- 172.21.15.60:0/170849313 wait complete. 2023-10-16T12:49:04.277 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.273+0000 7ff67183e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:49:04.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.273+0000 7f5e3240d640 1 Processor -- start 2023-10-16T12:49:04.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.273+0000 7f5e3240d640 1 -- start start 2023-10-16T12:49:04.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3240d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 0x7f5e2c15add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3240d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c1f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3240d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e2c15cd70 0x7f5e2c1f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3240d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c1f28c0 con 0x7f5e2c0b8480 2023-10-16T12:49:04.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3240d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0a94f0 con 0x7f5e2c0cded0 2023-10-16T12:49:04.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3240d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c15b310 con 0x7f5e2c15cd70 2023-10-16T12:49:04.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 0x7f5e2c15add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 0x7f5e2c15add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52714/0 (socket says 172.21.15.60:52714) 2023-10-16T12:49:04.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 -- 172.21.15.60:0/897945528 learned_addr learned my addr 172.21.15.60:0/897945528 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:04.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e30c0a640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c1f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 -- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e2c15cd70 msgr2=0x7f5e2c1f2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e2c15cd70 0x7f5e2c1f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 -- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e2c0cded0 msgr2=0x7f5e2c1f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c1f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e3140b640 1 -- 172.21.15.60:0/897945528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24009a70 con 0x7f5e2c0b8480 2023-10-16T12:49:04.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.275+0000 7f5e30c0a640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c1f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:04.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.276+0000 7f5e3140b640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 0x7f5e2c15add0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c400 tx=0x7f5e2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:04.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.276+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280052d0 con 0x7f5e2c0b8480 2023-10-16T12:49:04.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.276+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28005430 con 0x7f5e2c0b8480 2023-10-16T12:49:04.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.276+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c0b3600 con 0x7f5e2c0b8480 2023-10-16T12:49:04.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.276+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28003830 con 0x7f5e2c0b8480 2023-10-16T12:49:04.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.276+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2c0b3bc0 con 0x7f5e2c0b8480 2023-10-16T12:49:04.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.278+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2c0a9660 con 0x7f5e2c0b8480 2023-10-16T12:49:04.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.278+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5e28003990 con 0x7f5e2c0b8480 2023-10-16T12:49:04.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.279+0000 7f5e21ffb640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 0x7f5e1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.279+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f5e28088f40 con 0x7f5e2c0b8480 2023-10-16T12:49:04.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.279+0000 7f5e30c0a640 1 -- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 msgr2=0x7f5e1006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:04.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.279+0000 7f5e30c0a640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 0x7f5e1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:04.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.283+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e2c0a9660 con 0x7f5e2c0b8480 2023-10-16T12:49:04.333 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.331+0000 7ff67183e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:49:04.352 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.351+0000 7fc60d83a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:04.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.380+0000 7ff67183e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:49:04.405 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.403+0000 7fc60d83a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:49:04.465 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.463+0000 7fc60d83a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:49:04.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.480+0000 7f5e30c0a640 1 -- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 msgr2=0x7f5e1006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:04.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.480+0000 7f5e30c0a640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 0x7f5e1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:04.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.498+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e2c0a9660 con 0x7f5e2c0b8480 2023-10-16T12:49:04.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.499+0000 7f5e21ffb640 1 -- 172.21.15.60:0/897945528 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e28017020 con 0x7f5e2c0b8480 2023-10-16T12:49:04.500 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:49:04.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.501+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 msgr2=0x7f5e1006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:04.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.501+0000 7f5e3240d640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 0x7f5e1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.501+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 msgr2=0x7f5e2c15add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.501+0000 7f5e3240d640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 0x7f5e2c15add0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c400 tx=0x7f5e2800c8c0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 shutdown_connections 2023-10-16T12:49:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f5e1006af90 0x7f5e1006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e2c15cd70 0x7f5e2c1f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e2c0cded0 0x7f5e2c1f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 --2- 172.21.15.60:0/897945528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e2c0b8480 0x7f5e2c15add0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 >> 172.21.15.60:0/897945528 conn(0x7f5e2c0cf570 msgr2=0x7f5e2c15d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 shutdown_connections 2023-10-16T12:49:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.502+0000 7f5e3240d640 1 -- 172.21.15.60:0/897945528 wait complete. 2023-10-16T12:49:04.519 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.518+0000 7fc60d83a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:49:04.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:49:04.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.759+0000 7ff67183e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:49:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.771+0000 7f28859ce640 1 Processor -- start 2023-10-16T12:49:04.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.772+0000 7f28859ce640 1 -- start start 2023-10-16T12:49:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.773+0000 7f28859ce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28800cf710 0x7f28800b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.773+0000 7f28859ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800cfaf0 0x7f28800b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.773+0000 7f28859ce640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28800d3980 con 0x7f28800b38e0 2023-10-16T12:49:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.773+0000 7f28859ce640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28800d3c60 con 0x7f28800cf710 2023-10-16T12:49:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.773+0000 7f28859ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800ce630 con 0x7f28800cfaf0 2023-10-16T12:49:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28800cf710 0x7f28800b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28800cf710 0x7f28800b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52724/0 (socket says 172.21.15.60:52724) 2023-10-16T12:49:04.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 -- 172.21.15.60:0/329187930 learned_addr learned my addr 172.21.15.60:0/329187930 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:04.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 -- 172.21.15.60:0/329187930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800cfaf0 msgr2=0x7f28800b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:04.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287e7fc640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800cfaf0 0x7f28800b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800cfaf0 0x7f28800b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 -- 172.21.15.60:0/329187930 >> v1:172.21.15.144:6789/0 conn(0x7f28800b38e0 legacy=0x7f28800d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:04.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287effd640 1 -- 172.21.15.60:0/329187930 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800ce910 con 0x7f28800cf710 2023-10-16T12:49:04.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.774+0000 7f287e7fc640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800cfaf0 0x7f28800b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:04.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.775+0000 7f287effd640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28800b2d10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2870009850 tx=0x7f287002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff39031c731653ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:04.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.775+0000 7f287dffb640 1 -- 172.21.15.60:0/329187930 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287002e070 con 0x7f28800cf710 2023-10-16T12:49:04.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f287dffb640 1 -- 172.21.15.60:0/329187930 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2870004030 con 0x7f28800cf710 2023-10-16T12:49:04.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f287dffb640 1 -- 172.21.15.60:0/329187930 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870004420 con 0x7f28800cf710 2023-10-16T12:49:04.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f28859ce640 1 -- 172.21.15.60:0/329187930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 msgr2=0x7f28800b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f28859ce640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28800b2d10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2870009850 tx=0x7f287002b840 comp rx=0 tx=0).stop 2023-10-16T12:49:04.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f28859ce640 1 -- 172.21.15.60:0/329187930 shutdown_connections 2023-10-16T12:49:04.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f28859ce640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800cfaf0 0x7f28800b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f28859ce640 1 --2- 172.21.15.60:0/329187930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28800b2d10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.776+0000 7f28859ce640 1 -- 172.21.15.60:0/329187930 >> 172.21.15.60:0/329187930 conn(0x7f28800d2cc0 msgr2=0x7f28800d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:04.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.777+0000 7f28859ce640 1 -- 172.21.15.60:0/329187930 shutdown_connections 2023-10-16T12:49:04.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.777+0000 7f28859ce640 1 -- 172.21.15.60:0/329187930 wait complete. 2023-10-16T12:49:04.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.778+0000 7f28859ce640 1 Processor -- start 2023-10-16T12:49:04.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.778+0000 7f28859ce640 1 -- start start 2023-10-16T12:49:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.779+0000 7f28859ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800b38e0 0x7f28801f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.779+0000 7f28859ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28801f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.779+0000 7f28859ce640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28800cfaf0 0x7f28800caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.779+0000 7f28859ce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28801f6a20 con 0x7f28800cf710 2023-10-16T12:49:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.779+0000 7f28859ce640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800cc040 con 0x7f28800cfaf0 2023-10-16T12:49:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.779+0000 7f28859ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800c7e20 con 0x7f28800b38e0 2023-10-16T12:49:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28801f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28801f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52740/0 (socket says 172.21.15.60:52740) 2023-10-16T12:49:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 -- 172.21.15.60:0/332346134 learned_addr learned my addr 172.21.15.60:0/332346134 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287effd640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800b38e0 0x7f28801f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 -- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800b38e0 msgr2=0x7f28801f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800b38e0 0x7f28801f64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 -- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28800cfaf0 msgr2=0x7f28800caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28800cfaf0 0x7f28800caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 -- 172.21.15.60:0/332346134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870009460 con 0x7f28800cf710 2023-10-16T12:49:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.780+0000 7f287e7fc640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28801f3630 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f287400c820 tx=0x7f287400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.781+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874007a30 con 0x7f28800cf710 2023-10-16T12:49:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.781+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874007b90 con 0x7f28800cf710 2023-10-16T12:49:04.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.781+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28800c80c0 con 0x7f28800cf710 2023-10-16T12:49:04.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.781+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287400ce10 con 0x7f28800cf710 2023-10-16T12:49:04.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.781+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28800cba80 con 0x7f28800cf710 2023-10-16T12:49:04.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.782+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288004e160 con 0x7f28800cf710 2023-10-16T12:49:04.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.783+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f28740113f0 con 0x7f28800cf710 2023-10-16T12:49:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.784+0000 7f2867fff640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 0x7f286c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.785+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f2874089a70 con 0x7f28800cf710 2023-10-16T12:49:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.785+0000 7f287effd640 1 -- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 msgr2=0x7f286c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:04.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.785+0000 7f287effd640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 0x7f286c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:04.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.789+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2874054800 con 0x7f28800cf710 2023-10-16T12:49:04.923 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:04.921+0000 7ff67183e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:49:04.929 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:04.928+0000 7fc60d83a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:49:04.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.984+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28800cbe20 con 0x7f28800cf710 2023-10-16T12:49:04.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.985+0000 7f2867fff640 1 -- 172.21.15.60:0/332346134 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f287401f0a0 con 0x7f28800cf710 2023-10-16T12:49:04.987 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:49:04.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.986+0000 7f287effd640 1 -- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 msgr2=0x7f286c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:04.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.986+0000 7f287effd640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 0x7f286c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:04.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 msgr2=0x7f286c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:04.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 0x7f286c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 msgr2=0x7f28801f3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:04.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28801f3630 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f287400c820 tx=0x7f287400cce0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 shutdown_connections 2023-10-16T12:49:04.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7f286c06af90 0x7f286c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28800cfaf0 0x7f28800caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28800cf710 0x7f28801f3630 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 --2- 172.21.15.60:0/332346134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28800b38e0 0x7f28801f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:04.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 >> 172.21.15.60:0/332346134 conn(0x7f28800d2cc0 msgr2=0x7f28800d1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:04.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 shutdown_connections 2023-10-16T12:49:04.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:04.988+0000 7f28859ce640 1 -- 172.21.15.60:0/332346134 wait complete. 2023-10-16T12:49:05.006 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:49:05.030 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.028+0000 7ff67183e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:49:05.081 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.079+0000 7ff67183e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:49:05.102 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.101+0000 7fc60d83a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:49:05.189 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.187+0000 7ff67183e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:05.218 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.216+0000 7fc60d83a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:49:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.218+0000 7fb09a590640 1 Processor -- start 2023-10-16T12:49:05.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.218+0000 7fb09a590640 1 -- start start 2023-10-16T12:49:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.219+0000 7fb09a590640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb0940caac0 0x7fb0940ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.219+0000 7fb09a590640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940cb030 con 0x7fb09415ad70 2023-10-16T12:49:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.220+0000 7fb09a590640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940c91c0 con 0x7fb09406a4b0 2023-10-16T12:49:05.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.220+0000 7fb09a590640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0940c94a0 con 0x7fb0940caac0 2023-10-16T12:49:05.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.220+0000 7fb093fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb09415ad70 0x7fb094069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45220/0 (socket says 172.21.15.60:45220) 2023-10-16T12:49:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.220+0000 7fb093fff640 1 -- 172.21.15.60:0/817055217 learned_addr learned my addr 172.21.15.60:0/817055217 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.220+0000 7fb098b2d640 1 --2- 172.21.15.60:0/817055217 >> v2:172.21.15.144:3300/0 conn(0x7fb0940caac0 0x7fb0940ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.220+0000 7fb092ffd640 1 -- 172.21.15.60:0/817055217 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559258942 0 0) 0x7fb0940cb030 con 0x7fb09415ad70 2023-10-16T12:49:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.221+0000 7fb092ffd640 1 -- 172.21.15.60:0/817055217 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c003520 con 0x7fb09415ad70 2023-10-16T12:49:05.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.221+0000 7fb098b2d640 1 -- 172.21.15.60:0/817055217 >> v1:172.21.15.144:6789/0 conn(0x7fb09406a4b0 legacy=0x7fb0940ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.221+0000 7fb098b2d640 1 -- 172.21.15.60:0/817055217 >> v1:172.21.15.60:6789/0 conn(0x7fb09415ad70 legacy=0x7fb094069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.221+0000 7fb092ffd640 1 -- 172.21.15.60:0/817055217 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978440674 0 0) 0x7fb07c003520 con 0x7fb09415ad70 2023-10-16T12:49:05.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.221+0000 7fb098b2d640 1 -- 172.21.15.60:0/817055217 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0940c9780 con 0x7fb0940caac0 2023-10-16T12:49:05.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.222+0000 7fb098b2d640 1 --2- 172.21.15.60:0/817055217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0940caac0 0x7fb0940ad4d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009d90 tx=0x7fb08c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6d4dbd9a60c1ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb092ffd640 1 -- 172.21.15.60:0/817055217 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084001670 con 0x7fb0940caac0 2023-10-16T12:49:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb092ffd640 1 -- 172.21.15.60:0/817055217 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08c02c940 con 0x7fb0940caac0 2023-10-16T12:49:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb092ffd640 1 -- 172.21.15.60:0/817055217 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c02ccd0 con 0x7fb0940caac0 2023-10-16T12:49:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb09a590640 1 -- 172.21.15.60:0/817055217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0940caac0 msgr2=0x7fb0940ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:05.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb09a590640 1 --2- 172.21.15.60:0/817055217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0940caac0 0x7fb0940ad4d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009d90 tx=0x7fb08c02be10 comp rx=0 tx=0).stop 2023-10-16T12:49:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb09a590640 1 -- 172.21.15.60:0/817055217 shutdown_connections 2023-10-16T12:49:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb09a590640 1 --2- 172.21.15.60:0/817055217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0940caac0 0x7fb0940ad4d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.223+0000 7fb09a590640 1 -- 172.21.15.60:0/817055217 >> 172.21.15.60:0/817055217 conn(0x7fb0940ce9b0 msgr2=0x7fb0940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.224+0000 7fb09a590640 1 -- 172.21.15.60:0/817055217 shutdown_connections 2023-10-16T12:49:05.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.224+0000 7fb09a590640 1 -- 172.21.15.60:0/817055217 wait complete. 2023-10-16T12:49:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.226+0000 7fb09a590640 1 Processor -- start 2023-10-16T12:49:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.226+0000 7fb09a590640 1 -- start start 2023-10-16T12:49:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb09a590640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 0x7fb0941f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb09a590640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0940caac0 0x7fb09415d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb09a590640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb09415ad70 0x7fb09415de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb09a590640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941f0fd0 con 0x7fb09406a4b0 2023-10-16T12:49:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb09a590640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941f35e0 con 0x7fb09415ad70 2023-10-16T12:49:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb09a590640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941f13f0 con 0x7fb0940caac0 2023-10-16T12:49:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb093fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 0x7fb0941f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb093fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 0x7fb0941f30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52752/0 (socket says 172.21.15.60:52752) 2023-10-16T12:49:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.227+0000 7fb093fff640 1 -- 172.21.15.60:0/1582292731 learned_addr learned my addr 172.21.15.60:0/1582292731 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb0937fe640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0940caac0 0x7fb09415d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb098b2d640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb09415ad70 0x7fb09415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb093fff640 1 -- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0940caac0 msgr2=0x7fb09415d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb093fff640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0940caac0 0x7fb09415d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb093fff640 1 -- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb09415ad70 msgr2=0x7fb09415de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:05.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb093fff640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb09415ad70 0x7fb09415de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb093fff640 1 -- 172.21.15.60:0/1582292731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c009a70 con 0x7fb09406a4b0 2023-10-16T12:49:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb098b2d640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb09415ad70 0x7fb09415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb0937fe640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0940caac0 0x7fb09415d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb093fff640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 0x7fb0941f30a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c750 tx=0x7fb08400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.228+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084005ad0 con 0x7fb09406a4b0 2023-10-16T12:49:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.229+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084005c30 con 0x7fb09406a4b0 2023-10-16T12:49:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.229+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084007070 con 0x7fb09406a4b0 2023-10-16T12:49:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.229+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0941f0630 con 0x7fb09406a4b0 2023-10-16T12:49:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.229+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0941f0bf0 con 0x7fb09406a4b0 2023-10-16T12:49:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.230+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0941f1110 con 0x7fb09406a4b0 2023-10-16T12:49:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.231+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb0840071d0 con 0x7fb09406a4b0 2023-10-16T12:49:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.232+0000 7fb090ff9640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 0x7fb08006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.232+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fb084088fe0 con 0x7fb09406a4b0 2023-10-16T12:49:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.232+0000 7fb0937fe640 1 -- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 msgr2=0x7fb08006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.232+0000 7fb0937fe640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 0x7fb08006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.236+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb084053e00 con 0x7fb09406a4b0 2023-10-16T12:49:05.273 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.272+0000 7fc60d83a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:49:05.297 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.295+0000 7ff67183e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:49:05.388 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.387+0000 7fc60d83a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:05.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.433+0000 7fb0937fe640 1 -- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 msgr2=0x7fb08006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:05.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.433+0000 7fb0937fe640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 0x7fb08006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.441+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb09404e160 con 0x7fb09406a4b0 2023-10-16T12:49:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.442+0000 7fb090ff9640 1 -- 172.21.15.60:0/1582292731 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v70) v1 ==== 80+0+476 (secure 0 0 0) 0x7fb084017020 con 0x7fb09406a4b0 2023-10-16T12:49:05.444 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/3875614959 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.444 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/788226979 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.444 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3543205521 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.445 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/3875614959 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.445 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/411702881 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.445 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3534362968 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.445 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2319504696 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.445 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3965421321 2023-10-17T12:47:29.992623+0000 2023-10-16T12:49:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T12:49:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 msgr2=0x7fb08006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 0x7fb08006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 msgr2=0x7fb0941f30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 0x7fb0941f30a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c750 tx=0x7fb08400cc10 comp rx=0 tx=0).stop 2023-10-16T12:49:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 shutdown_connections 2023-10-16T12:49:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fb08006af90 0x7fb08006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb0940caac0 0x7fb09415d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb09415ad70 0x7fb09415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 --2- 172.21.15.60:0/1582292731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb09406a4b0 0x7fb0941f30a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 >> 172.21.15.60:0/1582292731 conn(0x7fb0940ce9b0 msgr2=0x7fb0940ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 shutdown_connections 2023-10-16T12:49:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:05.445+0000 7fb09a590640 1 -- 172.21.15.60:0/1582292731 wait complete. 2023-10-16T12:49:05.453 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.451+0000 7ff67183e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:05.465 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:49:05.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:05.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:49:05.501 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.500+0000 7ff67183e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:05.503 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.502+0000 7fc60d83a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:49:05.556 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:49:05.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:05.557 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:49:05.622 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:05.620+0000 7ff67183e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:49:05.652 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:05.652 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:05.674 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.673+0000 7fc60d83a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:05.706 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:05.706 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:05.707 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:05.726 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.725+0000 7fc60d83a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:05.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:49:05.766 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:05.766 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:05.821 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:05.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:05.860 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:05.858+0000 7fc60d83a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:49:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.025+0000 7ff58e50f640 1 Processor -- start 2023-10-16T12:49:06.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.025+0000 7ff58e50f640 1 -- start start 2023-10-16T12:49:06.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.026+0000 7ff58e50f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff58815aaa0 0x7ff58815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.026+0000 7ff58e50f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815ae80 0x7ff5880cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.026+0000 7ff58e50f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5880ccaa0 0x7ff588070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.026+0000 7ff58e50f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58806b810 con 0x7ff5880ccaa0 2023-10-16T12:49:06.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.026+0000 7ff58e50f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58815ed60 con 0x7ff58815aaa0 2023-10-16T12:49:06.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.026+0000 7ff58e50f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880707e0 con 0x7ff58815ae80 2023-10-16T12:49:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5880ccaa0 0x7ff588070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5880ccaa0 0x7ff588070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52762/0 (socket says 172.21.15.60:52762) 2023-10-16T12:49:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 -- 172.21.15.60:0/666825873 learned_addr learned my addr 172.21.15.60:0/666825873 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff57ffff640 1 --2- 172.21.15.60:0/666825873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815ae80 0x7ff5880cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff587fff640 1 --2- 172.21.15.60:0/666825873 >> v2:172.21.15.144:3300/0 conn(0x7ff58815aaa0 0x7ff58815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 -- 172.21.15.60:0/666825873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815ae80 msgr2=0x7ff5880cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:06.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 --2- 172.21.15.60:0/666825873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815ae80 0x7ff5880cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 -- 172.21.15.60:0/666825873 >> v2:172.21.15.144:3300/0 conn(0x7ff58815aaa0 msgr2=0x7ff58815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:06.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 --2- 172.21.15.60:0/666825873 >> v2:172.21.15.144:3300/0 conn(0x7ff58815aaa0 0x7ff58815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.027+0000 7ff58caac640 1 -- 172.21.15.60:0/666825873 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58815ef90 con 0x7ff5880ccaa0 2023-10-16T12:49:06.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.029+0000 7ff58caac640 1 --2- 172.21.15.60:0/666825873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880ccaa0 0x7ff588070f50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff5880cd990 tx=0x7ff58002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2501fc9936d5f87c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:06.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.029+0000 7ff5877fe640 1 -- 172.21.15.60:0/666825873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58002ca70 con 0x7ff5880ccaa0 2023-10-16T12:49:06.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.029+0000 7ff5877fe640 1 -- 172.21.15.60:0/666825873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58002cbd0 con 0x7ff5880ccaa0 2023-10-16T12:49:06.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.029+0000 7ff5877fe640 1 -- 172.21.15.60:0/666825873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5800343f0 con 0x7ff5880ccaa0 2023-10-16T12:49:06.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 -- 172.21.15.60:0/666825873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880ccaa0 msgr2=0x7ff588070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 --2- 172.21.15.60:0/666825873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880ccaa0 0x7ff588070f50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff5880cd990 tx=0x7ff58002bed0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 -- 172.21.15.60:0/666825873 shutdown_connections 2023-10-16T12:49:06.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 --2- 172.21.15.60:0/666825873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880ccaa0 0x7ff588070f50 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 --2- 172.21.15.60:0/666825873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815ae80 0x7ff5880cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 --2- 172.21.15.60:0/666825873 >> v2:172.21.15.144:3300/0 conn(0x7ff58815aaa0 0x7ff58815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 -- 172.21.15.60:0/666825873 >> 172.21.15.60:0/666825873 conn(0x7ff5880cb410 msgr2=0x7ff5880cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:06.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 -- 172.21.15.60:0/666825873 shutdown_connections 2023-10-16T12:49:06.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.030+0000 7ff58e50f640 1 -- 172.21.15.60:0/666825873 wait complete. 2023-10-16T12:49:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.032+0000 7ff58e50f640 1 Processor -- start 2023-10-16T12:49:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.032+0000 7ff58e50f640 1 -- start start 2023-10-16T12:49:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58e50f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5880ccaa0 0x7ff588155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58e50f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815aaa0 0x7ff588155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58e50f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 0x7ff5881f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58e50f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5881f2630 con 0x7ff58815ae80 2023-10-16T12:49:06.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58e50f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5881f2910 con 0x7ff5880ccaa0 2023-10-16T12:49:06.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58e50f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5881f2bf0 con 0x7ff58815aaa0 2023-10-16T12:49:06.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58caac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 0x7ff5881f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58caac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 0x7ff5881f08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52770/0 (socket says 172.21.15.60:52770) 2023-10-16T12:49:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff58caac640 1 -- 172.21.15.60:0/3899189085 learned_addr learned my addr 172.21.15.60:0/3899189085 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff57ffff640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815aaa0 0x7ff588155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.033+0000 7ff587fff640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5880ccaa0 0x7ff588155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58caac640 1 -- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815aaa0 msgr2=0x7ff588155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58caac640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815aaa0 0x7ff588155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58caac640 1 -- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5880ccaa0 msgr2=0x7ff588155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58caac640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5880ccaa0 0x7ff588155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58caac640 1 -- 172.21.15.60:0/3899189085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580009b30 con 0x7ff58815ae80 2023-10-16T12:49:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58caac640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 0x7ff5881f08f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff580036040 tx=0x7ff580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580009510 con 0x7ff58815ae80 2023-10-16T12:49:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58003e450 con 0x7ff58815ae80 2023-10-16T12:49:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5881f2ed0 con 0x7ff58815ae80 2023-10-16T12:49:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.034+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58003e800 con 0x7ff58815ae80 2023-10-16T12:49:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.035+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5880ceef0 con 0x7ff58815ae80 2023-10-16T12:49:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.036+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5881f33f0 con 0x7ff58815ae80 2023-10-16T12:49:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.037+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff58003e960 con 0x7ff58815ae80 2023-10-16T12:49:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.038+0000 7ff5857fa640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 0x7ff56806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.038+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6199+0+0 (secure 0 0 0) 0x7ff5800ac6d0 con 0x7ff58815ae80 2023-10-16T12:49:06.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.039+0000 7ff587fff640 1 -- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 msgr2=0x7ff56806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:06.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.039+0000 7ff587fff640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 0x7ff56806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:06.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.043+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5881f33f0 con 0x7ff58815ae80 2023-10-16T12:49:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.240+0000 7ff587fff640 1 -- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 msgr2=0x7ff56806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.240+0000 7ff587fff640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 0x7ff56806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:06.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.270+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5881f33f0 con 0x7ff58815ae80 2023-10-16T12:49:06.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.271+0000 7ff5857fa640 1 -- 172.21.15.60:0/3899189085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4916 (secure 0 0 0) 0x7ff5881f33f0 con 0x7ff58815ae80 2023-10-16T12:49:06.273 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:06.273 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:47:15.298066+0000","modified":"2023-10-16T12:47:18.376045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/528818860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":528818860},{"type":"v1","addr":"172.21.15.144:6837","nonce":528818860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:49:06.273 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2023-10-16T12:49:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.273+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 msgr2=0x7ff56806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.273+0000 7ff58e50f640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 0x7ff56806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.273+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 msgr2=0x7ff5881f08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.273+0000 7ff58e50f640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 0x7ff5881f08f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff580036040 tx=0x7ff580004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.273+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 shutdown_connections 2023-10-16T12:49:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.273+0000 7ff58e50f640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7ff56806af90 0x7ff56806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.274+0000 7ff58e50f640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58815aaa0 0x7ff588155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.274+0000 7ff58e50f640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58815ae80 0x7ff5881f08f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.274+0000 7ff58e50f640 1 --2- 172.21.15.60:0/3899189085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5880ccaa0 0x7ff588155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.274+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 >> 172.21.15.60:0/3899189085 conn(0x7ff5880cb410 msgr2=0x7ff5880b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.274+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 shutdown_connections 2023-10-16T12:49:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.274+0000 7ff58e50f640 1 -- 172.21.15.60:0/3899189085 wait complete. 2023-10-16T12:49:06.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:06.462 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:06.460+0000 7ff67183e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:49:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.525+0000 7fcf951b0640 1 Processor -- start 2023-10-16T12:49:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.525+0000 7fcf951b0640 1 -- start start 2023-10-16T12:49:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.526+0000 7fcf951b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcf9006b8c0 0x7fcf900cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.526+0000 7fcf951b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf900b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf951b0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcf900b09c0 0x7fcf90160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf951b0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900ce990 con 0x7fcf900b09c0 2023-10-16T12:49:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf951b0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900d0310 con 0x7fcf9006b8c0 2023-10-16T12:49:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf951b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cf940 con 0x7fcf9006bca0 2023-10-16T12:49:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcf900b09c0 0x7fcf90160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcf900b09c0 0x7fcf90160a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52776/0 (socket says 172.21.15.60:52776) 2023-10-16T12:49:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 -- 172.21.15.60:0/3707552623 learned_addr learned my addr 172.21.15.60:0/3707552623 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf87fff640 1 --2- 172.21.15.60:0/3707552623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf900b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8ed9d640 1 --2- 172.21.15.60:0/3707552623 >> v2:172.21.15.144:3300/0 conn(0x7fcf9006b8c0 0x7fcf900cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 -- 172.21.15.60:0/3707552623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 msgr2=0x7fcf900b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 --2- 172.21.15.60:0/3707552623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf900b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 -- 172.21.15.60:0/3707552623 >> v2:172.21.15.144:3300/0 conn(0x7fcf9006b8c0 msgr2=0x7fcf900cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 --2- 172.21.15.60:0/3707552623 >> v2:172.21.15.144:3300/0 conn(0x7fcf9006b8c0 0x7fcf900cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.527+0000 7fcf8f59e640 1 -- 172.21.15.60:0/3707552623 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9015e290 con 0x7fcf900b09c0 2023-10-16T12:49:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf8f59e640 1 --2- 172.21.15.60:0/3707552623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf900b09c0 0x7fcf90160a30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcf900711f0 tx=0x7fcf8802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997a9857996738c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf8e59c640 1 -- 172.21.15.60:0/3707552623 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8802ca80 con 0x7fcf900b09c0 2023-10-16T12:49:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf8e59c640 1 -- 172.21.15.60:0/3707552623 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8802cbe0 con 0x7fcf900b09c0 2023-10-16T12:49:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf8e59c640 1 -- 172.21.15.60:0/3707552623 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880343f0 con 0x7fcf900b09c0 2023-10-16T12:49:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 -- 172.21.15.60:0/3707552623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf900b09c0 msgr2=0x7fcf90160a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 --2- 172.21.15.60:0/3707552623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf900b09c0 0x7fcf90160a30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcf900711f0 tx=0x7fcf8802bee0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 -- 172.21.15.60:0/3707552623 shutdown_connections 2023-10-16T12:49:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 --2- 172.21.15.60:0/3707552623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf900b09c0 0x7fcf90160a30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 --2- 172.21.15.60:0/3707552623 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf900b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 --2- 172.21.15.60:0/3707552623 >> v2:172.21.15.144:3300/0 conn(0x7fcf9006b8c0 0x7fcf900cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.529+0000 7fcf951b0640 1 -- 172.21.15.60:0/3707552623 >> 172.21.15.60:0/3707552623 conn(0x7fcf90099ea0 msgr2=0x7fcf900d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.530+0000 7fcf951b0640 1 -- 172.21.15.60:0/3707552623 shutdown_connections 2023-10-16T12:49:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.530+0000 7fcf951b0640 1 -- 172.21.15.60:0/3707552623 wait complete. 2023-10-16T12:49:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.531+0000 7fcf951b0640 1 Processor -- start 2023-10-16T12:49:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.531+0000 7fcf951b0640 1 -- start start 2023-10-16T12:49:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.532+0000 7fcf951b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 0x7fcf90151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.532+0000 7fcf951b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf901508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.532+0000 7fcf951b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf900b09c0 0x7fcf900c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.532+0000 7fcf951b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cc480 con 0x7fcf9006b8c0 2023-10-16T12:49:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.532+0000 7fcf951b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cc760 con 0x7fcf900b09c0 2023-10-16T12:49:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.532+0000 7fcf951b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cca40 con 0x7fcf9006bca0 2023-10-16T12:49:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 0x7fcf90151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 0x7fcf90151de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52784/0 (socket says 172.21.15.60:52784) 2023-10-16T12:49:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 -- 172.21.15.60:0/4273792233 learned_addr learned my addr 172.21.15.60:0/4273792233 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf87fff640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf901508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8f59e640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf900b09c0 0x7fcf900c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 -- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 msgr2=0x7fcf901508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf901508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 -- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf900b09c0 msgr2=0x7fcf900c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf900b09c0 0x7fcf900c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.533+0000 7fcf8ed9d640 1 -- 172.21.15.60:0/4273792233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88009b30 con 0x7fcf9006b8c0 2023-10-16T12:49:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.534+0000 7fcf8ed9d640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 0x7fcf90151de0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000d800 tx=0x7fcf8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.534+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800096a0 con 0x7fcf9006b8c0 2023-10-16T12:49:06.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.534+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf80009800 con 0x7fcf9006b8c0 2023-10-16T12:49:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.534+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80009b90 con 0x7fcf9006b8c0 2023-10-16T12:49:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.534+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf900ccd20 con 0x7fcf9006b8c0 2023-10-16T12:49:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.534+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf90152320 con 0x7fcf9006b8c0 2023-10-16T12:49:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.535+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf900cf2d0 con 0x7fcf9006b8c0 2023-10-16T12:49:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.536+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 24) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fcf80009cf0 con 0x7fcf9006b8c0 2023-10-16T12:49:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.538+0000 7fcf877fe640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 0x7fcf7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.538+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fcf80089f20 con 0x7fcf9006b8c0 2023-10-16T12:49:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.538+0000 7fcf87fff640 1 -- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 msgr2=0x7fcf7006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.538+0000 7fcf87fff640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 0x7fcf7006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.542+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf900cf2d0 con 0x7fcf9006b8c0 2023-10-16T12:49:06.703 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:06.702+0000 7fc60d83a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:49:06.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.739+0000 7fcf87fff640 1 -- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 msgr2=0x7fcf7006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/745273919 2023-10-16T12:49:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.739+0000 7fcf87fff640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 0x7fcf7006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.740+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf900cf2d0 con 0x7fcf9006b8c0 2023-10-16T12:49:06.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.742+0000 7fcf877fe640 1 -- 172.21.15.60:0/4273792233 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+17252 (secure 0 0 0) 0x7fcf900cf2d0 con 0x7fcf9006b8c0 2023-10-16T12:49:06.743 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:06.744 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":71,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:05.836367+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6832/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/788226979":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3543205521":"2023-10-17T12:47:29.992623+0000","172.21.15.144:6833/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/411702881":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3534362968":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/2319504696":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3965421321":"2023-10-17T12:47:29.992623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 msgr2=0x7fcf7006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 0x7fcf7006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 msgr2=0x7fcf90151de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 0x7fcf90151de0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000d800 tx=0x7fcf8000dcc0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 shutdown_connections 2023-10-16T12:49:06.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:6832/745273919,v1:172.21.15.144:6833/745273919] conn(0x7fcf7006af90 0x7fcf7006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf9006bca0 0x7fcf901508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf900b09c0 0x7fcf900c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 --2- 172.21.15.60:0/4273792233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9006b8c0 0x7fcf90151de0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.744+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 >> 172.21.15.60:0/4273792233 conn(0x7fcf90099ea0 msgr2=0x7fcf9015ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:06.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.745+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 shutdown_connections 2023-10-16T12:49:06.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.745+0000 7fcf951b0640 1 -- 172.21.15.60:0/4273792233 wait complete. 2023-10-16T12:49:06.754 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:06.753+0000 7ff67183e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:49:06.763 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:47:13.297149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:47:14.334033+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:06.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:06.810 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:06.809+0000 7ff67183e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:49:06.940 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:06.938+0000 7ff67183e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:49:06.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.992+0000 7f879d4d7640 1 Processor -- start 2023-10-16T12:49:06.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.993+0000 7f879d4d7640 1 -- start start 2023-10-16T12:49:06.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f879d4d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f879d4d7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87980c9ca0 0x7f87980ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:06.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f879d4d7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980cb030 con 0x7f879815ad70 2023-10-16T12:49:06.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f879d4d7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980cae30 con 0x7f87980c9ca0 2023-10-16T12:49:06.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f879d4d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87980c91c0 con 0x7f879806a4b0 2023-10-16T12:49:06.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f8796ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f879815ad70 0x7f8798069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:42878/0 (socket says 172.21.15.60:42878) 2023-10-16T12:49:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f8796ffd640 1 -- 172.21.15.60:0/3896716902 learned_addr learned my addr 172.21.15.60:0/3896716902 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f87977fe640 1 --2- 172.21.15.60:0/3896716902 >> v2:172.21.15.60:3300/0 conn(0x7f87980c9ca0 0x7f87980ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.994+0000 7f878e7fc640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f87967fc640 1 -- 172.21.15.60:0/3896716902 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145750217 0 0) 0x7f87980cb030 con 0x7f879815ad70 2023-10-16T12:49:06.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f87967fc640 1 -- 172.21.15.60:0/3896716902 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784003520 con 0x7f879815ad70 2023-10-16T12:49:06.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f87977fe640 1 -- 172.21.15.60:0/3896716902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 msgr2=0x7f87980ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:06.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f87977fe640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:06.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f87977fe640 1 -- 172.21.15.60:0/3896716902 >> v1:172.21.15.144:6789/0 conn(0x7f879815ad70 legacy=0x7f8798069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:06.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f87977fe640 1 -- 172.21.15.60:0/3896716902 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980c94a0 con 0x7f87980c9ca0 2023-10-16T12:49:07.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.995+0000 7f878e7fc640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.996+0000 7f87977fe640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87980c9ca0 0x7f87980ca8f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8790009d90 tx=0x7f879002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af78e88d981a8b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f87967fc640 1 -- 172.21.15.60:0/3896716902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879002cc50 con 0x7f87980c9ca0 2023-10-16T12:49:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f87967fc640 1 -- 172.21.15.60:0/3896716902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879002cdb0 con 0x7f87980c9ca0 2023-10-16T12:49:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f87967fc640 1 -- 172.21.15.60:0/3896716902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8790034660 con 0x7f87980c9ca0 2023-10-16T12:49:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f879d4d7640 1 -- 172.21.15.60:0/3896716902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87980c9ca0 msgr2=0x7f87980ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f879d4d7640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87980c9ca0 0x7f87980ca8f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8790009d90 tx=0x7f879002bea0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f879d4d7640 1 -- 172.21.15.60:0/3896716902 shutdown_connections 2023-10-16T12:49:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f879d4d7640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87980c9ca0 0x7f87980ca8f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f879d4d7640 1 --2- 172.21.15.60:0/3896716902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.997+0000 7f879d4d7640 1 -- 172.21.15.60:0/3896716902 >> 172.21.15.60:0/3896716902 conn(0x7f87980ce9b0 msgr2=0x7f87980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.998+0000 7f879d4d7640 1 -- 172.21.15.60:0/3896716902 shutdown_connections 2023-10-16T12:49:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.998+0000 7f879d4d7640 1 -- 172.21.15.60:0/3896716902 wait complete. 2023-10-16T12:49:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.999+0000 7f879d4d7640 1 Processor -- start 2023-10-16T12:49:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:06.999+0000 7f879d4d7640 1 -- start start 2023-10-16T12:49:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.000+0000 7f879d4d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.000+0000 7f879d4d7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87980c9ca0 0x7f87981efc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.000+0000 7f879d4d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 0x7f87980ab640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.000+0000 7f879d4d7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87980c6f30 con 0x7f879815ad70 2023-10-16T12:49:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.000+0000 7f879d4d7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87980a8e00 con 0x7f87980c9ca0 2023-10-16T12:49:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.000+0000 7f879d4d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87980a90a0 con 0x7f879806a4b0 2023-10-16T12:49:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 0x7f87980ab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f8796ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f878e7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87980c9ca0 0x7f87981efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 0x7f87980ab640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52806/0 (socket says 172.21.15.60:52806) 2023-10-16T12:49:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 -- 172.21.15.60:0/367816384 learned_addr learned my addr 172.21.15.60:0/367816384 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 -- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 msgr2=0x7f87980ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 -- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87980c9ca0 msgr2=0x7f87981efc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87980c9ca0 0x7f87981efc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f87977fe640 1 -- 172.21.15.60:0/367816384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780009400 con 0x7f879815ad70 2023-10-16T12:49:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f8796ffd640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.001+0000 7f878e7fc640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87980c9ca0 0x7f87981efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.002+0000 7f87977fe640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 0x7f87980ab640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8790009d90 tx=0x7f8790004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.002+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879003e610 con 0x7f879815ad70 2023-10-16T12:49:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.002+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879003e770 con 0x7f879815ad70 2023-10-16T12:49:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.002+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879003eb20 con 0x7f879815ad70 2023-10-16T12:49:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.002+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8790009a70 con 0x7f879815ad70 2023-10-16T12:49:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.002+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87980a9940 con 0x7f879815ad70 2023-10-16T12:49:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.003+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87980a94a0 con 0x7f879815ad70 2023-10-16T12:49:07.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.004+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 25) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f879003b050 con 0x7f879815ad70 2023-10-16T12:49:07.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.006+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f87900abaf0 con 0x7f879815ad70 2023-10-16T12:49:07.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.011+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8790076e50 con 0x7f879815ad70 2023-10-16T12:49:07.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.032+0000 7ff5f6b6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.032+0000 7ff5f6b6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.032+0000 7ff5f6b6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.032+0000 7ff5f6b6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.035 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.032+0000 7ff5f6b6a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.035 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.033+0000 7ff5f2361640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.035 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.033+0000 7ff5f2361640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.036 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.033+0000 7ff5f2361640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.036 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.033+0000 7ff5f2361640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.036 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:07.033+0000 7ff5f2361640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:07.044 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:07.043+0000 7fc60d83a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:49:07.105 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:07.103+0000 7fc60d83a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:49:07.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.207+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87980a9260 con 0x7f879815ad70 2023-10-16T12:49:07.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.208+0000 7f878ffff640 1 -- 172.21.15.60:0/367816384 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8790039020 con 0x7f879815ad70 2023-10-16T12:49:07.209 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:07.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.210+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 msgr2=0x7f87980ab640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.210+0000 7f879d4d7640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 0x7f87980ab640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8790009d90 tx=0x7f8790004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:07.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 shutdown_connections 2023-10-16T12:49:07.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879815ad70 0x7f87980ab640 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87980c9ca0 0x7f87981efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 --2- 172.21.15.60:0/367816384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f879806a4b0 0x7f87980ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 >> 172.21.15.60:0/367816384 conn(0x7f87980ce9b0 msgr2=0x7f87980ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:07.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 shutdown_connections 2023-10-16T12:49:07.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.211+0000 7f879d4d7640 1 -- 172.21.15.60:0/367816384 wait complete. 2023-10-16T12:49:07.224 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:07.223+0000 7fc60d83a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:49:07.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:49:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.438+0000 7fc49ed4e640 1 Processor -- start 2023-10-16T12:49:07.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.439+0000 7fc49ed4e640 1 -- start start 2023-10-16T12:49:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.440+0000 7fc49ed4e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc498070830 0x7fc49806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.440+0000 7fc49ed4e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4980cf5f0 0x7fc49806c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.440+0000 7fc49ed4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc498161740 con 0x7fc498070c10 2023-10-16T12:49:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.440+0000 7fc49ed4e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc49806c680 con 0x7fc4980cf5f0 2023-10-16T12:49:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.440+0000 7fc49ed4e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc49806b0f0 con 0x7fc498070830 2023-10-16T12:49:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4980cf5f0 0x7fc49806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4980cf5f0 0x7fc49806c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52810/0 (socket says 172.21.15.60:52810) 2023-10-16T12:49:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 -- 172.21.15.60:0/264699633 learned_addr learned my addr 172.21.15.60:0/264699633 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:07.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc49caea640 1 --2- 172.21.15.60:0/264699633 >> v2:172.21.15.144:3300/0 conn(0x7fc498070830 0x7fc49806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 -- 172.21.15.60:0/264699633 >> v2:172.21.15.144:3300/0 conn(0x7fc498070830 msgr2=0x7fc49806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 --2- 172.21.15.60:0/264699633 >> v2:172.21.15.144:3300/0 conn(0x7fc498070830 0x7fc49806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 -- 172.21.15.60:0/264699633 >> v1:172.21.15.60:6789/0 conn(0x7fc498070c10 legacy=0x7fc4980cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc497fff640 1 -- 172.21.15.60:0/264699633 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4981619b0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.441+0000 7fc49caea640 1 --2- 172.21.15.60:0/264699633 >> v2:172.21.15.144:3300/0 conn(0x7fc498070830 0x7fc49806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.443+0000 7fc497fff640 1 --2- 172.21.15.60:0/264699633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc49806c140 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc49806c010 tx=0x7fc48402b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635875767baf45e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.443+0000 7fc4977fe640 1 -- 172.21.15.60:0/264699633 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4840317f0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.443+0000 7fc4977fe640 1 -- 172.21.15.60:0/264699633 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484031950 con 0x7fc4980cf5f0 2023-10-16T12:49:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.443+0000 7fc4977fe640 1 -- 172.21.15.60:0/264699633 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484031ce0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.443+0000 7fc49ed4e640 1 -- 172.21.15.60:0/264699633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 msgr2=0x7fc49806c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.443+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/264699633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc49806c140 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc49806c010 tx=0x7fc48402b890 comp rx=0 tx=0).stop 2023-10-16T12:49:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.444+0000 7fc49ed4e640 1 -- 172.21.15.60:0/264699633 shutdown_connections 2023-10-16T12:49:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.444+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/264699633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc49806c140 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.444+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/264699633 >> v2:172.21.15.144:3300/0 conn(0x7fc498070830 0x7fc49806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.444+0000 7fc49ed4e640 1 -- 172.21.15.60:0/264699633 >> 172.21.15.60:0/264699633 conn(0x7fc4980cfac0 msgr2=0x7fc4980cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.444+0000 7fc49ed4e640 1 -- 172.21.15.60:0/264699633 shutdown_connections 2023-10-16T12:49:07.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.444+0000 7fc49ed4e640 1 -- 172.21.15.60:0/264699633 wait complete. 2023-10-16T12:49:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.446+0000 7fc49ed4e640 1 Processor -- start 2023-10-16T12:49:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.446+0000 7fc49ed4e640 1 -- start start 2023-10-16T12:49:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49ed4e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc498070830 0x7fc49815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49ed4e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc498070c10 0x7fc4980c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49ed4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc4981f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49ed4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980c7f50 con 0x7fc4980cf5f0 2023-10-16T12:49:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49ed4e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980c8230 con 0x7fc498070c10 2023-10-16T12:49:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49ed4e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980c8510 con 0x7fc498070830 2023-10-16T12:49:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49d2eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc4981f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49d2eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc4981f27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52826/0 (socket says 172.21.15.60:52826) 2023-10-16T12:49:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49d2eb640 1 -- 172.21.15.60:0/1629260929 learned_addr learned my addr 172.21.15.60:0/1629260929 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.447+0000 7fc49caea640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc498070830 0x7fc49815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc497fff640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc498070c10 0x7fc4980c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49d2eb640 1 -- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc498070830 msgr2=0x7fc49815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49d2eb640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc498070830 0x7fc49815b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49d2eb640 1 -- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc498070c10 msgr2=0x7fc4980c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49d2eb640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc498070c10 0x7fc4980c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49d2eb640 1 -- 172.21.15.60:0/1629260929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4840094d0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49d2eb640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc4981f27d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc49000c7a0 tx=0x7fc49000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490003b70 con 0x7fc4980cf5f0 2023-10-16T12:49:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc490003cf0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490003ea0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.448+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4980c87f0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.449+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4980c7be0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.450+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4980c8d10 con 0x7fc4980cf5f0 2023-10-16T12:49:07.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.451+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 25) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc49000aa90 con 0x7fc4980cf5f0 2023-10-16T12:49:07.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.452+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fc4900891d0 con 0x7fc4980cf5f0 2023-10-16T12:49:07.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.457+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4980c8d10 con 0x7fc4980cf5f0 2023-10-16T12:49:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.661+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4980c8d10 con 0x7fc4980cf5f0 2023-10-16T12:49:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.662+0000 7fc4957fa640 1 -- 172.21.15.60:0/1629260929 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4980c8d10 con 0x7fc4980cf5f0 2023-10-16T12:49:07.666 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:49:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 msgr2=0x7fc4981f27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc4981f27d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc49000c7a0 tx=0x7fc49000cc60 comp rx=0 tx=0).stop 2023-10-16T12:49:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 shutdown_connections 2023-10-16T12:49:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4980cf5f0 0x7fc4981f27d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc498070c10 0x7fc4980c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 --2- 172.21.15.60:0/1629260929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc498070830 0x7fc49815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.664+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 >> 172.21.15.60:0/1629260929 conn(0x7fc4980cfac0 msgr2=0x7fc498160080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.665+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 shutdown_connections 2023-10-16T12:49:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.665+0000 7fc49ed4e640 1 -- 172.21.15.60:0/1629260929 wait complete. 2023-10-16T12:49:07.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:49:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.915+0000 7facc0e2d640 1 Processor -- start 2023-10-16T12:49:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.915+0000 7facc0e2d640 1 -- start start 2023-10-16T12:49:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facc0e2d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7facbc0caac0 0x7facbc0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facc0e2d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0cb030 con 0x7facbc15ad70 2023-10-16T12:49:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facc0e2d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0c91c0 con 0x7facbc06a4b0 2023-10-16T12:49:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facc0e2d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc0c94a0 con 0x7facbc0caac0 2023-10-16T12:49:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facba59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7facbc15ad70 0x7facbc069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45246/0 (socket says 172.21.15.60:45246) 2023-10-16T12:49:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facba59c640 1 -- 172.21.15.60:0/624892418 learned_addr learned my addr 172.21.15.60:0/624892418 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facbad9d640 1 --2- 172.21.15.60:0/624892418 >> v2:172.21.15.144:3300/0 conn(0x7facbc0caac0 0x7facbc0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.917+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191967338 0 0) 0x7facbc0cb030 con 0x7facbc15ad70 2023-10-16T12:49:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca8003520 con 0x7facbc15ad70 2023-10-16T12:49:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649464274 0 0) 0x7faca8003520 con 0x7facbc15ad70 2023-10-16T12:49:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 >> v2:172.21.15.144:3300/0 conn(0x7facbc0caac0 msgr2=0x7facbc0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 --2- 172.21.15.60:0/624892418 >> v2:172.21.15.144:3300/0 conn(0x7facbc0caac0 0x7facbc0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 >> v1:172.21.15.144:6789/0 conn(0x7facbc06a4b0 legacy=0x7facbc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0c9780 con 0x7facbc15ad70 2023-10-16T12:49:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7facac0019f0 con 0x7facbc15ad70 2023-10-16T12:49:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 >> v1:172.21.15.60:6789/0 conn(0x7facbc15ad70 legacy=0x7facbc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80039c0 0x7faca8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca80047c0 0x7faca8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca8005180 0x7faca800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0c94a0 con 0x7faca8005180 2023-10-16T12:49:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca800c080 con 0x7faca80039c0 2023-10-16T12:49:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca800c240 con 0x7faca80047c0 2023-10-16T12:49:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.918+0000 7facbad9d640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca8005180 0x7faca800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facb9d9b640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca80047c0 0x7faca8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facba59c640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80039c0 0x7faca8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facbad9d640 1 -- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca80047c0 msgr2=0x7faca8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facbad9d640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca80047c0 0x7faca8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facbad9d640 1 -- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80039c0 msgr2=0x7faca8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facbad9d640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80039c0 0x7faca8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facbad9d640 1 -- 172.21.15.60:0/624892418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8003520 con 0x7faca8005180 2023-10-16T12:49:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facb9d9b640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca80047c0 0x7faca8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.919+0000 7facba59c640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80039c0 0x7faca8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.920+0000 7facbad9d640 1 -- 172.21.15.60:0/624892418 reap_dead start 2023-10-16T12:49:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.920+0000 7facbad9d640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca8005180 0x7faca800b980 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7facb4009ac0 tx=0x7faca80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eb71441f074c52 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.920+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4004020 con 0x7faca8005180 2023-10-16T12:49:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.920+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb4004180 con 0x7faca8005180 2023-10-16T12:49:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.920+0000 7facb959a640 1 -- 172.21.15.60:0/624892418 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb402c710 con 0x7faca8005180 2023-10-16T12:49:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 -- 172.21.15.60:0/624892418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca8005180 msgr2=0x7faca800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca8005180 0x7faca800b980 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7facb4009ac0 tx=0x7faca80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 -- 172.21.15.60:0/624892418 shutdown_connections 2023-10-16T12:49:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 --2- 172.21.15.60:0/624892418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca8005180 0x7faca800b980 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 -- 172.21.15.60:0/624892418 >> 172.21.15.60:0/624892418 conn(0x7facbc0ce9b0 msgr2=0x7facbc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 -- 172.21.15.60:0/624892418 shutdown_connections 2023-10-16T12:49:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.921+0000 7facc0e2d640 1 -- 172.21.15.60:0/624892418 wait complete. 2023-10-16T12:49:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.923+0000 7facc0e2d640 1 Processor -- start 2023-10-16T12:49:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.923+0000 7facc0e2d640 1 -- start start 2023-10-16T12:49:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facc0e2d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca8005180 0x7facbc0b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facc0e2d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facbc0ad700 0x7facbc1f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facc0e2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 0x7facbc15c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facc0e2d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0adb80 con 0x7facbc1594b0 2023-10-16T12:49:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facc0e2d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1f64d0 con 0x7faca8005180 2023-10-16T12:49:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facc0e2d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0addc0 con 0x7facbc0ad700 2023-10-16T12:49:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facbad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 0x7facbc15c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facbad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 0x7facbc15c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52846/0 (socket says 172.21.15.60:52846) 2023-10-16T12:49:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facbad9d640 1 -- 172.21.15.60:0/706334729 learned_addr learned my addr 172.21.15.60:0/706334729 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facb9d9b640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facbc0ad700 0x7facbc1f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.924+0000 7facba59c640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca8005180 0x7facbc0b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facbad9d640 1 -- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facbc0ad700 msgr2=0x7facbc1f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facbad9d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facbc0ad700 0x7facbc1f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facbad9d640 1 -- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca8005180 msgr2=0x7facbc0b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facbad9d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca8005180 0x7facbc0b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facbad9d640 1 -- 172.21.15.60:0/706334729 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0ae0a0 con 0x7facbc1594b0 2023-10-16T12:49:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facbad9d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 0x7facbc15c460 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7facb402c380 tx=0x7facb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb403a070 con 0x7facbc1594b0 2023-10-16T12:49:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb40042f0 con 0x7facbc1594b0 2023-10-16T12:49:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4003610 con 0x7facbc1594b0 2023-10-16T12:49:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.925+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc0ccbe0 con 0x7facbc1594b0 2023-10-16T12:49:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.926+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc0cd1a0 con 0x7facbc1594b0 2023-10-16T12:49:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.927+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc0ae290 con 0x7facbc1594b0 2023-10-16T12:49:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.927+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 25) v1 ==== 88940+0+0 (secure 0 0 0) 0x7facb4003790 con 0x7facbc1594b0 2023-10-16T12:49:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.928+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6483+0+0 (secure 0 0 0) 0x7facb40a9810 con 0x7facbc1594b0 2023-10-16T12:49:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.933+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facb40749a0 con 0x7facbc1594b0 2023-10-16T12:49:07.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.962+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 26) v1 ==== 89067+0+0 (secure 0 0 0) 0x7facb4074450 con 0x7facbc1594b0 2023-10-16T12:49:07.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.963+0000 7facb2ffd640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fac9c0706e0 0x7fac9c072ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:07.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.964+0000 7facba59c640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fac9c0706e0 0x7fac9c072ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:07.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:07.966+0000 7facba59c640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fac9c0706e0 0x7fac9c072ad0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7facac001580 tx=0x7facac009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.133+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7facbc04e160 con 0x7facbc1594b0 2023-10-16T12:49:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.135+0000 7facb2ffd640 1 -- 172.21.15.60:0/706334729 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) v1 ==== 100+0+11 (secure 0 0 0) 0x7facb4060cc0 con 0x7facbc1594b0 2023-10-16T12:49:08.136 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:49:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fac9c0706e0 msgr2=0x7fac9c072ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fac9c0706e0 0x7fac9c072ad0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7facac001580 tx=0x7facac009040 comp rx=0 tx=0).stop 2023-10-16T12:49:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 msgr2=0x7facbc15c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 0x7facbc15c460 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7facb402c380 tx=0x7facb4004730 comp rx=0 tx=0).stop 2023-10-16T12:49:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 shutdown_connections 2023-10-16T12:49:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fac9c0706e0 0x7fac9c072ad0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facbc0ad700 0x7facbc1f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facbc1594b0 0x7facbc15c460 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 --2- 172.21.15.60:0/706334729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca8005180 0x7facbc0b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.137+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 >> 172.21.15.60:0/706334729 conn(0x7facbc0ce9b0 msgr2=0x7facbc0c27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.138+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 shutdown_connections 2023-10-16T12:49:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.138+0000 7facc0e2d640 1 -- 172.21.15.60:0/706334729 wait complete. 2023-10-16T12:49:08.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:08.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.383+0000 7ff8f4bff640 1 Processor -- start 2023-10-16T12:49:08.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.383+0000 7ff8f4bff640 1 -- start start 2023-10-16T12:49:08.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8f4bff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8f006c040 0x7ff8f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8f4bff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 0x7ff8f015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8f4bff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f006c960 con 0x7ff8f00ad970 2023-10-16T12:49:08.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8f4bff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f015b9d0 con 0x7ff8f006c040 2023-10-16T12:49:08.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8f4bff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015bcb0 con 0x7ff8f00cda80 2023-10-16T12:49:08.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8e6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff8f00ad970 0x7ff8f015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45254/0 (socket says 172.21.15.60:45254) 2023-10-16T12:49:08.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8e6ffd640 1 -- 172.21.15.60:0/451085522 learned_addr learned my addr 172.21.15.60:0/451085522 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:08.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8ef7fe640 1 --2- 172.21.15.60:0/451085522 >> v2:172.21.15.144:3300/0 conn(0x7ff8f006c040 0x7ff8f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.385+0000 7ff8effff640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 0x7ff8f015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31926369 0 0) 0x7ff8f006c960 con 0x7ff8f00ad970 2023-10-16T12:49:08.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc003520 con 0x7ff8f00ad970 2023-10-16T12:49:08.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527148991 0 0) 0x7ff8dc003520 con 0x7ff8f00ad970 2023-10-16T12:49:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 msgr2=0x7ff8f015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 0x7ff8f015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 >> v2:172.21.15.144:3300/0 conn(0x7ff8f006c040 msgr2=0x7ff8f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 --2- 172.21.15.60:0/451085522 >> v2:172.21.15.144:3300/0 conn(0x7ff8f006c040 0x7ff8f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f015bf90 con 0x7ff8f00ad970 2023-10-16T12:49:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff8d8001670 con 0x7ff8f00ad970 2023-10-16T12:49:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 >> v1:172.21.15.60:6789/0 conn(0x7ff8f00ad970 legacy=0x7ff8f015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 0x7ff8dc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc004d00 0x7ff8dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015b9d0 con 0x7ff8dc003ef0 2023-10-16T12:49:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015bcb0 con 0x7ff8dc004d00 2023-10-16T12:49:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.386+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00b980 con 0x7ff8f00cda80 2023-10-16T12:49:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8e6ffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8ef7fe640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 0x7ff8dc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8e6ffd640 1 -- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 msgr2=0x7ff8dc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8e6ffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00cda80 0x7ff8dc003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8e6ffd640 1 -- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc004d00 msgr2=0x7ff8dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8effff640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc004d00 0x7ff8dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8e6ffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc004d00 0x7ff8dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8e6ffd640 1 -- 172.21.15.60:0/451085522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc003520 con 0x7ff8dc003ef0 2023-10-16T12:49:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.387+0000 7ff8effff640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc004d00 0x7ff8dc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.388+0000 7ff8e6ffd640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8dc0047c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800c210 tx=0x7ff8dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5717bae7d1821eff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8005ad0 con 0x7ff8dc003ef0 2023-10-16T12:49:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8005c30 con 0x7ff8dc003ef0 2023-10-16T12:49:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8eeffd640 1 -- 172.21.15.60:0/451085522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d802cae0 con 0x7ff8dc003ef0 2023-10-16T12:49:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8f4bff640 1 -- 172.21.15.60:0/451085522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 msgr2=0x7ff8dc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/451085522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8dc0047c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800c210 tx=0x7ff8dc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8effff640 1 -- 172.21.15.60:0/451085522 reap_dead start 2023-10-16T12:49:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8f4bff640 1 -- 172.21.15.60:0/451085522 shutdown_connections 2023-10-16T12:49:08.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8f4bff640 1 -- 172.21.15.60:0/451085522 >> 172.21.15.60:0/451085522 conn(0x7ff8f00ce6a0 msgr2=0x7ff8f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:08.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.389+0000 7ff8f4bff640 1 -- 172.21.15.60:0/451085522 shutdown_connections 2023-10-16T12:49:08.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.390+0000 7ff8f4bff640 1 -- 172.21.15.60:0/451085522 wait complete. 2023-10-16T12:49:08.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.391+0000 7ff8f4bff640 1 Processor -- start 2023-10-16T12:49:08.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.391+0000 7ff8f4bff640 1 -- start start 2023-10-16T12:49:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.392+0000 7ff8f4bff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8f00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.392+0000 7ff8f4bff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00c5cb0 0x7ff8f00c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.392+0000 7ff8f4bff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f00c7bf0 0x7ff8f00b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.392+0000 7ff8f4bff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00c6b80 con 0x7ff8dc003ef0 2023-10-16T12:49:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.392+0000 7ff8f4bff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00c6110 con 0x7ff8f00c7bf0 2023-10-16T12:49:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.392+0000 7ff8f4bff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0148250 con 0x7ff8f00c5cb0 2023-10-16T12:49:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8f00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8f00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52862/0 (socket says 172.21.15.60:52862) 2023-10-16T12:49:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 -- 172.21.15.60:0/1357792770 learned_addr learned my addr 172.21.15.60:0/1357792770 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 -- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00c5cb0 msgr2=0x7ff8f00c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00c5cb0 0x7ff8f00c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 -- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f00c7bf0 msgr2=0x7ff8f00b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f00c7bf0 0x7ff8f00b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 -- 172.21.15.60:0/1357792770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00c8390 con 0x7ff8dc003ef0 2023-10-16T12:49:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.393+0000 7ff8ef7fe640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8f00c5770 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000f3d0 tx=0x7ff8e000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.394+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0016620 con 0x7ff8dc003ef0 2023-10-16T12:49:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.394+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e00167a0 con 0x7ff8dc003ef0 2023-10-16T12:49:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.394+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0016b50 con 0x7ff8dc003ef0 2023-10-16T12:49:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.394+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f00b1440 con 0x7ff8dc003ef0 2023-10-16T12:49:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.394+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f00b1a00 con 0x7ff8dc003ef0 2023-10-16T12:49:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.395+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f00b1b90 con 0x7ff8dc003ef0 2023-10-16T12:49:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.396+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff8e0016cb0 con 0x7ff8dc003ef0 2023-10-16T12:49:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.397+0000 7ff8ecff9640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff8d006ace0 0x7ff8d006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.397+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6483+0+0 (secure 0 0 0) 0x7ff8e008a560 con 0x7ff8dc003ef0 2023-10-16T12:49:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.398+0000 7ff8e6ffd640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff8d006ace0 0x7ff8d006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.399+0000 7ff8e6ffd640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff8d006ace0 0x7ff8d006d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8034000 tx=0x7ff8d802e000 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.402+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8e0055630 con 0x7ff8dc003ef0 2023-10-16T12:49:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.635+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8f004e9b0 con 0x7ff8dc003ef0 2023-10-16T12:49:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.636+0000 7ff8ecff9640 1 -- 172.21.15.60:0/1357792770 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4916 (secure 0 0 0) 0x7ff8e0058c50 con 0x7ff8dc003ef0 2023-10-16T12:49:08.637 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:08.638 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:47:15.298066+0000","modified":"2023-10-16T12:47:18.376045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/528818860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":528818860},{"type":"v1","addr":"172.21.15.144:6837","nonce":528818860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:49:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2023-10-16T12:49:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff8d006ace0 msgr2=0x7ff8d006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff8d006ace0 0x7ff8d006d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8034000 tx=0x7ff8d802e000 comp rx=0 tx=0).stop 2023-10-16T12:49:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 msgr2=0x7ff8f00c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8f00c5770 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000f3d0 tx=0x7ff8e000f890 comp rx=0 tx=0).stop 2023-10-16T12:49:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 shutdown_connections 2023-10-16T12:49:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff8d006ace0 0x7ff8d006d1a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f00c5cb0 0x7ff8f00c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f00c7bf0 0x7ff8f00b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 --2- 172.21.15.60:0/1357792770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc003ef0 0x7ff8f00c5770 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 >> 172.21.15.60:0/1357792770 conn(0x7ff8f00ce6a0 msgr2=0x7ff8f015de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 shutdown_connections 2023-10-16T12:49:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.639+0000 7ff8f4bff640 1 -- 172.21.15.60:0/1357792770 wait complete. 2023-10-16T12:49:08.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:08.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.887+0000 7f88480c2640 1 Processor -- start 2023-10-16T12:49:08.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.887+0000 7f88480c2640 1 -- start start 2023-10-16T12:49:08.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f88480c2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015aaa0 0x7f884015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f88480c2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 0x7f8840070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f88480c2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884015ed60 con 0x7f884015ae80 2023-10-16T12:49:08.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f88480c2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f884006b810 con 0x7f88400ce390 2023-10-16T12:49:08.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f88480c2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88400707e0 con 0x7f884015aaa0 2023-10-16T12:49:08.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f884665f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 0x7f8840070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f8845e5e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015aaa0 0x7f884015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f884665f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 0x7f8840070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:60752/0 (socket says 172.21.15.60:60752) 2023-10-16T12:49:08.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.889+0000 7f884665f640 1 -- 172.21.15.60:0/1871204658 learned_addr learned my addr 172.21.15.60:0/1871204658 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:08.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.890+0000 7f8845e5e640 1 -- 172.21.15.60:0/1871204658 >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 msgr2=0x7f8840070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:08.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.890+0000 7f8845e5e640 1 --2- 172.21.15.60:0/1871204658 >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 0x7f8840070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.890+0000 7f8845e5e640 1 -- 172.21.15.60:0/1871204658 >> v1:172.21.15.144:6789/0 conn(0x7f884015ae80 legacy=0x7f88400cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.890+0000 7f8845e5e640 1 -- 172.21.15.60:0/1871204658 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884015ef90 con 0x7f884015aaa0 2023-10-16T12:49:08.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.890+0000 7f884665f640 1 --2- 172.21.15.60:0/1871204658 >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 0x7f8840070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:08.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.892+0000 7f8845e5e640 1 --2- 172.21.15.60:0/1871204658 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015aaa0 0x7f884015e820 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f883c0097a0 tx=0x7f883c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efcfa18811b1116f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.893+0000 7f8844e5c640 1 -- 172.21.15.60:0/1871204658 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c02d040 con 0x7f884015aaa0 2023-10-16T12:49:08.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.893+0000 7f8844e5c640 1 -- 172.21.15.60:0/1871204658 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c031810 con 0x7f884015aaa0 2023-10-16T12:49:08.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.893+0000 7f8844e5c640 1 -- 172.21.15.60:0/1871204658 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c031ba0 con 0x7f884015aaa0 2023-10-16T12:49:08.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 -- 172.21.15.60:0/1871204658 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015aaa0 msgr2=0x7f884015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 --2- 172.21.15.60:0/1871204658 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015aaa0 0x7f884015e820 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f883c0097a0 tx=0x7f883c02b8b0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 -- 172.21.15.60:0/1871204658 shutdown_connections 2023-10-16T12:49:08.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 --2- 172.21.15.60:0/1871204658 >> v2:172.21.15.144:3300/0 conn(0x7f88400ce390 0x7f8840070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 --2- 172.21.15.60:0/1871204658 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015aaa0 0x7f884015e820 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 -- 172.21.15.60:0/1871204658 >> 172.21.15.60:0/1871204658 conn(0x7f88400cb410 msgr2=0x7f88400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:08.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 -- 172.21.15.60:0/1871204658 shutdown_connections 2023-10-16T12:49:08.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.894+0000 7f88480c2640 1 -- 172.21.15.60:0/1871204658 wait complete. 2023-10-16T12:49:08.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.896+0000 7f88480c2640 1 Processor -- start 2023-10-16T12:49:08.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.897+0000 7f88480c2640 1 -- start start 2023-10-16T12:49:08.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f88480c2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88400ce390 0x7f88400c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f88480c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 0x7f884014ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f88480c2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015ae80 0x7f88400c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f88480c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88400c29e0 con 0x7f884015aaa0 2023-10-16T12:49:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f88480c2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88400c2bd0 con 0x7f88400ce390 2023-10-16T12:49:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f88480c2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88400c2eb0 con 0x7f884015ae80 2023-10-16T12:49:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f884565d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 0x7f884014ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f884565d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 0x7f884014ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47524/0 (socket says 172.21.15.60:47524) 2023-10-16T12:49:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.898+0000 7f884565d640 1 -- 172.21.15.60:0/3707067608 learned_addr learned my addr 172.21.15.60:0/3707067608 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f8845e5e640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88400ce390 0x7f88400c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884665f640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015ae80 0x7f88400c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884565d640 1 -- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015ae80 msgr2=0x7f88400c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884565d640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015ae80 0x7f88400c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884565d640 1 -- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88400ce390 msgr2=0x7f88400c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:08.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884565d640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88400ce390 0x7f88400c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:08.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884565d640 1 -- 172.21.15.60:0/3707067608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838009b30 con 0x7f884015aaa0 2023-10-16T12:49:08.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f884565d640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 0x7f884014ac30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f883000c400 tx=0x7f883000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88300052d0 con 0x7f884015aaa0 2023-10-16T12:49:08.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8830005450 con 0x7f884015aaa0 2023-10-16T12:49:08.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830007260 con 0x7f884015aaa0 2023-10-16T12:49:08.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.899+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c009400 con 0x7f884015aaa0 2023-10-16T12:49:08.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.900+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88400c3470 con 0x7f884015aaa0 2023-10-16T12:49:08.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.901+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884004e160 con 0x7f884015aaa0 2023-10-16T12:49:08.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.902+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f88300073c0 con 0x7f884015aaa0 2023-10-16T12:49:08.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.903+0000 7f88367fc640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f882406acc0 0x7f882406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:08.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.903+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f8830088970 con 0x7f884015aaa0 2023-10-16T12:49:08.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.903+0000 7f8845e5e640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f882406acc0 0x7f882406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:08.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.905+0000 7f8845e5e640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f882406acc0 0x7f882406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f883c036000 tx=0x7f883c03c040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:08.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.909+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8830053a50 con 0x7f884015aaa0 2023-10-16T12:49:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:08.980+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 27) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8830053e20 con 0x7f884015aaa0 2023-10-16T12:49:09.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.116+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88400ce0c0 con 0x7f884015aaa0 2023-10-16T12:49:09.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.118+0000 7f88367fc640 1 -- 172.21.15.60:0/3707067608 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+17669 (secure 0 0 0) 0x7f8830074660 con 0x7f884015aaa0 2023-10-16T12:49:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:09.120 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":73,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:06.943557+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2397437661":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/2355510359":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6833/745273919":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/571939833":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6832/745273919":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/2772412730":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/3600960564":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6832/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/529537373":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/788226979":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3543205521":"2023-10-17T12:47:29.992623+0000","172.21.15.144:6833/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/411702881":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3534362968":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/2319504696":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3965421321":"2023-10-17T12:47:29.992623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:09.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.120+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f882406acc0 msgr2=0x7f882406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.120+0000 7f88480c2640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f882406acc0 0x7f882406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f883c036000 tx=0x7f883c03c040 comp rx=0 tx=0).stop 2023-10-16T12:49:09.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.120+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 msgr2=0x7f884014ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.120+0000 7f88480c2640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 0x7f884014ac30 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f883000c400 tx=0x7f883000c8c0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 shutdown_connections 2023-10-16T12:49:09.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f882406acc0 0x7f882406d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f884015ae80 0x7f88400c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f884015aaa0 0x7f884014ac30 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 --2- 172.21.15.60:0/3707067608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88400ce390 0x7f88400c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 >> 172.21.15.60:0/3707067608 conn(0x7f88400cb410 msgr2=0x7f884006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:09.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 shutdown_connections 2023-10-16T12:49:09.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.121+0000 7f88480c2640 1 -- 172.21.15.60:0/3707067608 wait complete. 2023-10-16T12:49:09.141 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:49:09.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:09.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.364+0000 7fc4d8dc7640 1 Processor -- start 2023-10-16T12:49:09.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.365+0000 7fc4d8dc7640 1 -- start start 2023-10-16T12:49:09.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.366+0000 7fc4d8dc7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4d4159b20 0x7fc4d406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.366+0000 7fc4d8dc7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.366+0000 7fc4d8dc7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d406f7f0 con 0x7fc4d406c120 2023-10-16T12:49:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.366+0000 7fc4d8dc7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d406c5e0 con 0x7fc4d4159b20 2023-10-16T12:49:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.366+0000 7fc4d8dc7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d406c8c0 con 0x7fc4d4159f00 2023-10-16T12:49:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.367+0000 7fc4d259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4d4159b20 0x7fc4d406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.367+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.367+0000 7fc4d2d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc4d406c120 0x7fc4d40cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:47772/0 (socket says 172.21.15.60:47772) 2023-10-16T12:49:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.367+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:51962/0 (socket says 172.21.15.60:51962) 2023-10-16T12:49:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.367+0000 7fc4d2d9d640 1 -- 172.21.15.60:0/1728081974 learned_addr learned my addr 172.21.15.60:0/1728081974 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.368+0000 7fc4d259c640 1 -- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 msgr2=0x7fc4d40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:09.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.368+0000 7fc4d259c640 1 --2- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.368+0000 7fc4d259c640 1 -- 172.21.15.60:0/1728081974 >> v1:172.21.15.144:6789/0 conn(0x7fc4d406c120 legacy=0x7fc4d40cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.368+0000 7fc4d259c640 1 -- 172.21.15.60:0/1728081974 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d406cba0 con 0x7fc4d4159b20 2023-10-16T12:49:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.368+0000 7fc4cbfff640 1 --2- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.369+0000 7fc4d259c640 1 --2- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d406bbe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40097a0 tx=0x7fc4c402dac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a28c3c32bf65ff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.370+0000 7fc4d1d9b640 1 -- 172.21.15.60:0/1728081974 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c402f880 con 0x7fc4d4159b20 2023-10-16T12:49:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.370+0000 7fc4d1d9b640 1 -- 172.21.15.60:0/1728081974 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c402f9e0 con 0x7fc4d4159b20 2023-10-16T12:49:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.370+0000 7fc4d1d9b640 1 -- 172.21.15.60:0/1728081974 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c402fd70 con 0x7fc4d4159b20 2023-10-16T12:49:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 msgr2=0x7fc4d406bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d406bbe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40097a0 tx=0x7fc4c402dac0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/1728081974 shutdown_connections 2023-10-16T12:49:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/1728081974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d406bbe0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/1728081974 >> 172.21.15.60:0/1728081974 conn(0x7fc4d40cb410 msgr2=0x7fc4d40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/1728081974 shutdown_connections 2023-10-16T12:49:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.371+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/1728081974 wait complete. 2023-10-16T12:49:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.373+0000 7fc4d8dc7640 1 Processor -- start 2023-10-16T12:49:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.373+0000 7fc4d8dc7640 1 -- start start 2023-10-16T12:49:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d8dc7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 0x7fc4d40c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d8dc7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d4151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d8dc7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d8dc7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d40c8120 con 0x7fc4d406c120 2023-10-16T12:49:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d8dc7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d40c64c0 con 0x7fc4d4159b20 2023-10-16T12:49:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d8dc7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d40c5b90 con 0x7fc4d4159f00 2023-10-16T12:49:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 0x7fc4d40c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 0x7fc4d40c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47532/0 (socket says 172.21.15.60:47532) 2023-10-16T12:49:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d259c640 1 -- 172.21.15.60:0/879650535 learned_addr learned my addr 172.21.15.60:0/879650535 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d2d9d640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4d259c640 1 -- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 msgr2=0x7fc4d40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.374+0000 7fc4cbfff640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d4151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d259c640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d259c640 1 -- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 msgr2=0x7fc4d4151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d259c640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d4151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d259c640 1 -- 172.21.15.60:0/879650535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc009400 con 0x7fc4d406c120 2023-10-16T12:49:09.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d2d9d640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:09.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d259c640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 0x7fc4d40c6950 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009890 tx=0x7fc4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:09.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4030ac0 con 0x7fc4d406c120 2023-10-16T12:49:09.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4030c40 con 0x7fc4d406c120 2023-10-16T12:49:09.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c4009400 con 0x7fc4d406c120 2023-10-16T12:49:09.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4042410 con 0x7fc4d406c120 2023-10-16T12:49:09.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.375+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d40c6150 con 0x7fc4d406c120 2023-10-16T12:49:09.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.377+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d404e160 con 0x7fc4d406c120 2023-10-16T12:49:09.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.378+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc4c4039400 con 0x7fc4d406c120 2023-10-16T12:49:09.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.378+0000 7fc4caffd640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc4b406af00 0x7fc4b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.379+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fc4c40ab810 con 0x7fc4d406c120 2023-10-16T12:49:09.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.379+0000 7fc4cbfff640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc4b406af00 0x7fc4b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.380+0000 7fc4cbfff640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc4b406af00 0x7fc4b406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4d41f62b0 tx=0x7fc4bc008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:09.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.383+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4c4076660 con 0x7fc4d406c120 2023-10-16T12:49:09.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.619+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4d40c2200 con 0x7fc4d406c120 2023-10-16T12:49:09.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.620+0000 7fc4caffd640 1 -- 172.21.15.60:0/879650535 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fc4c4079c80 con 0x7fc4d406c120 2023-10-16T12:49:09.622 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:09.622 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:47:15.298066+0000","modified":"2023-10-16T12:47:18.376045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/528818860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":528818860},{"type":"v1","addr":"172.21.15.144:6837","nonce":528818860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:49:09.623 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2023-10-16T12:49:09.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc4b406af00 msgr2=0x7fc4b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc4b406af00 0x7fc4b406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc4d41f62b0 tx=0x7fc4bc008040 comp rx=0 tx=0).stop 2023-10-16T12:49:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 msgr2=0x7fc4d40c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 0x7fc4d40c6950 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009890 tx=0x7fc4c40046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 shutdown_connections 2023-10-16T12:49:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc4b406af00 0x7fc4b406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4d4159f00 0x7fc4d40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d4159b20 0x7fc4d4151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 --2- 172.21.15.60:0/879650535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4d406c120 0x7fc4d40c6950 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 >> 172.21.15.60:0/879650535 conn(0x7fc4d40cb410 msgr2=0x7fc4d40c44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 shutdown_connections 2023-10-16T12:49:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.623+0000 7fc4d8dc7640 1 -- 172.21.15.60:0/879650535 wait complete. 2023-10-16T12:49:09.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:09.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.872+0000 7fe5a6cd7640 1 Processor -- start 2023-10-16T12:49:09.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.872+0000 7fe5a6cd7640 1 -- start start 2023-10-16T12:49:09.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.874+0000 7fe5a6cd7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5a006d640 0x7fe5a00ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.875+0000 7fe5a6cd7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a015ce30 con 0x7fe5a015d2c0 2023-10-16T12:49:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.875+0000 7fe5a6cd7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a00c9080 con 0x7fe5a0158ed0 2023-10-16T12:49:09.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.875+0000 7fe5a6cd7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a00c9360 con 0x7fe5a006d640 2023-10-16T12:49:09.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.875+0000 7fe5a5274640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe5a015d2c0 0x7fe5a00b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38854/0 (socket says 172.21.15.60:38854) 2023-10-16T12:49:09.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.875+0000 7fe5a5274640 1 -- 172.21.15.60:0/1512187626 learned_addr learned my addr 172.21.15.60:0/1512187626 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.876+0000 7fe5a4a73640 1 --2- 172.21.15.60:0/1512187626 >> v2:172.21.15.60:3300/0 conn(0x7fe5a006d640 0x7fe5a00ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.877+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397500270 0 0) 0x7fe5a015ce30 con 0x7fe5a015d2c0 2023-10-16T12:49:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.877+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c003520 con 0x7fe5a015d2c0 2023-10-16T12:49:09.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472578014 0 0) 0x7fe58c003520 con 0x7fe5a015d2c0 2023-10-16T12:49:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 >> v2:172.21.15.60:3300/0 conn(0x7fe5a006d640 msgr2=0x7fe5a00ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 --2- 172.21.15.60:0/1512187626 >> v2:172.21.15.60:3300/0 conn(0x7fe5a006d640 0x7fe5a00ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 >> v1:172.21.15.144:6789/0 conn(0x7fe5a0158ed0 legacy=0x7fe5a00ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a00c9640 con 0x7fe5a015d2c0 2023-10-16T12:49:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe5980019f0 con 0x7fe5a015d2c0 2023-10-16T12:49:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 >> v1:172.21.15.60:6789/0 conn(0x7fe5a015d2c0 legacy=0x7fe5a00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe58c0039c0 0x7fe58c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0047c0 0x7fe58c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58c0052a0 0x7fe58c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00c9360 con 0x7fe58c0052a0 2023-10-16T12:49:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0105d0 con 0x7fe58c0047c0 2023-10-16T12:49:09.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.878+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c010790 con 0x7fe58c0039c0 2023-10-16T12:49:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a5274640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58c0052a0 0x7fe58c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe59ffff640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0047c0 0x7fe58c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a4a73640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe58c0039c0 0x7fe58c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a5274640 1 -- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe58c0039c0 msgr2=0x7fe58c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a5274640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe58c0039c0 0x7fe58c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a5274640 1 -- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0047c0 msgr2=0x7fe58c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a5274640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0047c0 0x7fe58c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.879+0000 7fe5a5274640 1 -- 172.21.15.60:0/1512187626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c003520 con 0x7fe58c0052a0 2023-10-16T12:49:09.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.880+0000 7fe5a5274640 1 -- 172.21.15.60:0/1512187626 reap_dead start 2023-10-16T12:49:09.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.881+0000 7fe5a5274640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58c0052a0 0x7fe58c00b980 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe59800a830 tx=0x7fe58c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def195d61187d79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.881+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5980072d0 con 0x7fe58c0052a0 2023-10-16T12:49:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.881+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59802f070 con 0x7fe58c0052a0 2023-10-16T12:49:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.881+0000 7fe59f7fe640 1 -- 172.21.15.60:0/1512187626 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5980076b0 con 0x7fe58c0052a0 2023-10-16T12:49:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/1512187626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58c0052a0 msgr2=0x7fe58c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58c0052a0 0x7fe58c00b980 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe59800a830 tx=0x7fe58c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/1512187626 shutdown_connections 2023-10-16T12:49:09.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/1512187626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58c0052a0 0x7fe58c00b980 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/1512187626 >> 172.21.15.60:0/1512187626 conn(0x7fe5a00cb410 msgr2=0x7fe5a00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/1512187626 shutdown_connections 2023-10-16T12:49:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.882+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/1512187626 wait complete. 2023-10-16T12:49:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.884+0000 7fe5a6cd7640 1 Processor -- start 2023-10-16T12:49:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.884+0000 7fe5a6cd7640 1 -- start start 2023-10-16T12:49:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a6cd7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0052a0 0x7fe5a00c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a6cd7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 0x7fe5a006f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a6cd7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a00b44e0 0x7fe5a006f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a6cd7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00c8ab0 con 0x7fe5a00b4060 2023-10-16T12:49:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a6cd7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a01f1ca0 con 0x7fe58c0052a0 2023-10-16T12:49:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a6cd7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a01eef90 con 0x7fe5a00b44e0 2023-10-16T12:49:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe59ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 0x7fe5a006f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe59ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 0x7fe5a006f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47568/0 (socket says 172.21.15.60:47568) 2023-10-16T12:49:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe59ffff640 1 -- 172.21.15.60:0/158559779 learned_addr learned my addr 172.21.15.60:0/158559779 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.885+0000 7fe5a4a73640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0052a0 0x7fe5a00c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe5a5274640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a00b44e0 0x7fe5a006f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59ffff640 1 -- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a00b44e0 msgr2=0x7fe5a006f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59ffff640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a00b44e0 0x7fe5a006f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59ffff640 1 -- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0052a0 msgr2=0x7fe5a00c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59ffff640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0052a0 0x7fe5a00c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59ffff640 1 -- 172.21.15.60:0/158559779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a01ef270 con 0x7fe5a00b4060 2023-10-16T12:49:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59ffff640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 0x7fe5a006f450 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe58800d470 tx=0x7fe58800d930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588005480 con 0x7fe5a00b4060 2023-10-16T12:49:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe588003610 con 0x7fe5a00b4060 2023-10-16T12:49:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a00c7a90 con 0x7fe5a00b4060 2023-10-16T12:49:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.886+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588003980 con 0x7fe5a00b4060 2023-10-16T12:49:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.887+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a00c8050 con 0x7fe5a00b4060 2023-10-16T12:49:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.888+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a00c81e0 con 0x7fe5a00b4060 2023-10-16T12:49:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.889+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe588003ae0 con 0x7fe5a00b4060 2023-10-16T12:49:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.889+0000 7fe59d7fa640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe58006af30 0x7fe58006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.890+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fe5880899a0 con 0x7fe5a00b4060 2023-10-16T12:49:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.890+0000 7fe5a4a73640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe58006af30 0x7fe58006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:09.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.891+0000 7fe5a4a73640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe58006af30 0x7fe58006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe590006fd0 tx=0x7fe590008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:09.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:09.894+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5880545d0 con 0x7fe5a00b4060 2023-10-16T12:49:10.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.134+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a004e160 con 0x7fe5a00b4060 2023-10-16T12:49:10.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.135+0000 7fe59d7fa640 1 -- 172.21.15.60:0/158559779 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fe588001160 con 0x7fe5a00b4060 2023-10-16T12:49:10.137 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:10.137 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:47:15.298066+0000","modified":"2023-10-16T12:47:18.376045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"d","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/528818860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":528818860},{"type":"v1","addr":"172.21.15.144:6837","nonce":528818860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:49:10.137 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2023-10-16T12:49:10.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe58006af30 msgr2=0x7fe58006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe58006af30 0x7fe58006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe590006fd0 tx=0x7fe590008040 comp rx=0 tx=0).stop 2023-10-16T12:49:10.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 msgr2=0x7fe5a006f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 0x7fe5a006f450 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe58800d470 tx=0x7fe58800d930 comp rx=0 tx=0).stop 2023-10-16T12:49:10.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 shutdown_connections 2023-10-16T12:49:10.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe58006af30 0x7fe58006d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5a00b44e0 0x7fe5a006f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5a00b4060 0x7fe5a006f450 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 --2- 172.21.15.60:0/158559779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58c0052a0 0x7fe5a00c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 >> 172.21.15.60:0/158559779 conn(0x7fe5a00cb410 msgr2=0x7fe5a00b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:10.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 shutdown_connections 2023-10-16T12:49:10.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.138+0000 7fe5a6cd7640 1 -- 172.21.15.60:0/158559779 wait complete. 2023-10-16T12:49:10.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:10.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.381+0000 7f02b9a6f640 1 Processor -- start 2023-10-16T12:49:10.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.382+0000 7f02b9a6f640 1 -- start start 2023-10-16T12:49:10.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.383+0000 7f02b9a6f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 0x7f02b415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.383+0000 7f02b9a6f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02b40ce390 0x7f02b4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.383+0000 7f02b9a6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b415ed60 con 0x7f02b415aaa0 2023-10-16T12:49:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.383+0000 7f02b9a6f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b406b810 con 0x7f02b40ce390 2023-10-16T12:49:10.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.383+0000 7f02b9a6f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40707e0 con 0x7f02b415ae80 2023-10-16T12:49:10.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.384+0000 7f02abfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 0x7f02b415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.384+0000 7f02b3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02b40ce390 0x7f02b4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.384+0000 7f02b37fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f02b415aaa0 0x7f02b40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:47778/0 (socket says 172.21.15.60:47778) 2023-10-16T12:49:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.384+0000 7f02abfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 0x7f02b415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:51998/0 (socket says 172.21.15.60:51998) 2023-10-16T12:49:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.384+0000 7f02b37fe640 1 -- 172.21.15.60:0/2758269563 learned_addr learned my addr 172.21.15.60:0/2758269563 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.384+0000 7f02b3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02b40ce390 0x7f02b4070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:60806/0 (socket says 172.21.15.60:60806) 2023-10-16T12:49:10.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.385+0000 7f02abfff640 1 -- 172.21.15.60:0/2758269563 >> v2:172.21.15.144:3300/0 conn(0x7f02b40ce390 msgr2=0x7f02b4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.385+0000 7f02abfff640 1 --2- 172.21.15.60:0/2758269563 >> v2:172.21.15.144:3300/0 conn(0x7f02b40ce390 0x7f02b4070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.385+0000 7f02abfff640 1 -- 172.21.15.60:0/2758269563 >> v1:172.21.15.144:6789/0 conn(0x7f02b415aaa0 legacy=0x7f02b40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.385+0000 7f02abfff640 1 -- 172.21.15.60:0/2758269563 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b415ef90 con 0x7f02b415ae80 2023-10-16T12:49:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.386+0000 7f02abfff640 1 --2- 172.21.15.60:0/2758269563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 0x7f02b415e820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f029c0097a0 tx=0x7f029c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=532e1806a19fa14c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.387+0000 7f02b2ffd640 1 -- 172.21.15.60:0/2758269563 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c02d040 con 0x7f02b415ae80 2023-10-16T12:49:10.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.387+0000 7f02b2ffd640 1 -- 172.21.15.60:0/2758269563 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c031810 con 0x7f02b415ae80 2023-10-16T12:49:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.387+0000 7f02b2ffd640 1 -- 172.21.15.60:0/2758269563 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c031ba0 con 0x7f02b415ae80 2023-10-16T12:49:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 -- 172.21.15.60:0/2758269563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 msgr2=0x7f02b415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/2758269563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 0x7f02b415e820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f029c0097a0 tx=0x7f029c02b8b0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 -- 172.21.15.60:0/2758269563 shutdown_connections 2023-10-16T12:49:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/2758269563 >> v2:172.21.15.144:3300/0 conn(0x7f02b40ce390 0x7f02b4070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/2758269563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415ae80 0x7f02b415e820 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 -- 172.21.15.60:0/2758269563 >> 172.21.15.60:0/2758269563 conn(0x7f02b40cb410 msgr2=0x7f02b40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:10.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 -- 172.21.15.60:0/2758269563 shutdown_connections 2023-10-16T12:49:10.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.388+0000 7f02b9a6f640 1 -- 172.21.15.60:0/2758269563 wait complete. 2023-10-16T12:49:10.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.390+0000 7f02b9a6f640 1 Processor -- start 2023-10-16T12:49:10.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.391+0000 7f02b9a6f640 1 -- start start 2023-10-16T12:49:10.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b9a6f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 0x7f02b41f1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b9a6f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415aaa0 0x7f02b40c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b9a6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b415ae80 0x7f02b40c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b9a6f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41580c0 con 0x7f02b40ce390 2023-10-16T12:49:10.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b9a6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4157440 con 0x7f02b415ae80 2023-10-16T12:49:10.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b9a6f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b4157720 con 0x7f02b415aaa0 2023-10-16T12:49:10.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 0x7f02b41f1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 0x7f02b41f1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47582/0 (socket says 172.21.15.60:47582) 2023-10-16T12:49:10.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.392+0000 7f02b37fe640 1 -- 172.21.15.60:0/3783186183 learned_addr learned my addr 172.21.15.60:0/3783186183 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:10.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02abfff640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415aaa0 0x7f02b40c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b3fff640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b415ae80 0x7f02b40c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b37fe640 1 -- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415aaa0 msgr2=0x7f02b40c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b37fe640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415aaa0 0x7f02b40c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b37fe640 1 -- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b415ae80 msgr2=0x7f02b40c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b37fe640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b415ae80 0x7f02b40c29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b37fe640 1 -- 172.21.15.60:0/3783186183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c009400 con 0x7f02b40ce390 2023-10-16T12:49:10.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02abfff640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415aaa0 0x7f02b40c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:10.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b37fe640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 0x7f02b41f1210 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c4a0 tx=0x7f02a400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:10.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.393+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4005ce0 con 0x7f02b40ce390 2023-10-16T12:49:10.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.394+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4005e40 con 0x7f02b40ce390 2023-10-16T12:49:10.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.394+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4007270 con 0x7f02b40ce390 2023-10-16T12:49:10.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.394+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b4157a00 con 0x7f02b40ce390 2023-10-16T12:49:10.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.394+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b41f1750 con 0x7f02b40ce390 2023-10-16T12:49:10.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.395+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b404e160 con 0x7f02b40ce390 2023-10-16T12:49:10.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.396+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f02a40073d0 con 0x7f02b40ce390 2023-10-16T12:49:10.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.397+0000 7f02b0ff9640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f029406af30 0x7f029406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.398+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f02a4088da0 con 0x7f02b40ce390 2023-10-16T12:49:10.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.398+0000 7f02abfff640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f029406af30 0x7f029406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.399+0000 7f02abfff640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f029406af30 0x7f029406d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f029c036000 tx=0x7f029c03c040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:10.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.402+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02a4053b90 con 0x7f02b40ce390 2023-10-16T12:49:10.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.600+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02b415e8b0 con 0x7f02b40ce390 2023-10-16T12:49:10.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.603+0000 7f02b0ff9640 1 -- 172.21.15.60:0/3783186183 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+17669 (secure 0 0 0) 0x7f02a4017020 con 0x7f02b40ce390 2023-10-16T12:49:10.605 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:10.606 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":74,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:09.747276+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:47:13.297149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-16T12:47:14.334033+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2397437661":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/2355510359":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6833/745273919":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/571939833":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6832/745273919":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/2772412730":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/3600960564":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6832/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/529537373":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/788226979":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3543205521":"2023-10-17T12:47:29.992623+0000","172.21.15.144:6833/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/411702881":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3534362968":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/2319504696":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3965421321":"2023-10-17T12:47:29.992623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f029406af30 msgr2=0x7f029406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f029406af30 0x7f029406d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f029c036000 tx=0x7f029c03c040 comp rx=0 tx=0).stop 2023-10-16T12:49:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 msgr2=0x7f02b41f1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 0x7f02b41f1210 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c4a0 tx=0x7f02a400c960 comp rx=0 tx=0).stop 2023-10-16T12:49:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 shutdown_connections 2023-10-16T12:49:10.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f029406af30 0x7f029406d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02b415aaa0 0x7f02b40c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b415ae80 0x7f02b40c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 --2- 172.21.15.60:0/3783186183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b40ce390 0x7f02b41f1210 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 >> 172.21.15.60:0/3783186183 conn(0x7f02b40cb410 msgr2=0x7f02b40cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:10.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.606+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 shutdown_connections 2023-10-16T12:49:10.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.607+0000 7f02b9a6f640 1 -- 172.21.15.60:0/3783186183 wait complete. 2023-10-16T12:49:10.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:49:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.854+0000 7f859e7e9640 1 Processor -- start 2023-10-16T12:49:10.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.854+0000 7f859e7e9640 1 -- start start 2023-10-16T12:49:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.855+0000 7f859e7e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f859815c9f0 0x7f85980c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.855+0000 7f859e7e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815dde0 0x7f85980ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.855+0000 7f859e7e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8598160a90 con 0x7f85980c9230 2023-10-16T12:49:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.855+0000 7f859e7e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980cdec0 con 0x7f859815c9f0 2023-10-16T12:49:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.855+0000 7f859e7e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980cd900 con 0x7f859815dde0 2023-10-16T12:49:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f859815c9f0 0x7f85980c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f859815c9f0 0x7f85980c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47586/0 (socket says 172.21.15.60:47586) 2023-10-16T12:49:10.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 -- 172.21.15.60:0/1566496744 learned_addr learned my addr 172.21.15.60:0/1566496744 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 -- 172.21.15.60:0/1566496744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815dde0 msgr2=0x7f85980ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 --2- 172.21.15.60:0/1566496744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815dde0 0x7f85980ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 -- 172.21.15.60:0/1566496744 >> v1:172.21.15.60:6789/0 conn(0x7f85980c9230 legacy=0x7f85980709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.856+0000 7f859d7e7640 1 -- 172.21.15.60:0/1566496744 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980cdbe0 con 0x7f859815c9f0 2023-10-16T12:49:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.857+0000 7f859d7e7640 1 --2- 172.21.15.60:0/1566496744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815c9f0 0x7f85980c8cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8594009870 tx=0x7f859402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4664c8666bafafef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.858+0000 7f858ffff640 1 -- 172.21.15.60:0/1566496744 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594030840 con 0x7f859815c9f0 2023-10-16T12:49:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.858+0000 7f858ffff640 1 -- 172.21.15.60:0/1566496744 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85940309a0 con 0x7f859815c9f0 2023-10-16T12:49:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.858+0000 7f858ffff640 1 -- 172.21.15.60:0/1566496744 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594030d30 con 0x7f859815c9f0 2023-10-16T12:49:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.858+0000 7f859e7e9640 1 -- 172.21.15.60:0/1566496744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815c9f0 msgr2=0x7f85980c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.858+0000 7f859e7e9640 1 --2- 172.21.15.60:0/1566496744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815c9f0 0x7f85980c8cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8594009870 tx=0x7f859402b980 comp rx=0 tx=0).stop 2023-10-16T12:49:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.859+0000 7f859e7e9640 1 -- 172.21.15.60:0/1566496744 shutdown_connections 2023-10-16T12:49:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.859+0000 7f859e7e9640 1 --2- 172.21.15.60:0/1566496744 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815dde0 0x7f85980ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.859+0000 7f859e7e9640 1 --2- 172.21.15.60:0/1566496744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815c9f0 0x7f85980c8cf0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.859+0000 7f859e7e9640 1 -- 172.21.15.60:0/1566496744 >> 172.21.15.60:0/1566496744 conn(0x7f85980ce6a0 msgr2=0x7f85980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.859+0000 7f859e7e9640 1 -- 172.21.15.60:0/1566496744 shutdown_connections 2023-10-16T12:49:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.859+0000 7f859e7e9640 1 -- 172.21.15.60:0/1566496744 wait complete. 2023-10-16T12:49:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.860+0000 7f859e7e9640 1 Processor -- start 2023-10-16T12:49:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.861+0000 7f859e7e9640 1 -- start start 2023-10-16T12:49:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.861+0000 7f859e7e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85980c9230 0x7f85980c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.861+0000 7f859e7e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815c9f0 0x7f859814c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859e7e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 0x7f859815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859e7e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85981f3630 con 0x7f859815dde0 2023-10-16T12:49:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859e7e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85981f2760 con 0x7f85980c9230 2023-10-16T12:49:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859e7e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85981f2a20 con 0x7f859815c9f0 2023-10-16T12:49:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 0x7f859815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 0x7f859815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47590/0 (socket says 172.21.15.60:47590) 2023-10-16T12:49:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 -- 172.21.15.60:0/3127551899 learned_addr learned my addr 172.21.15.60:0/3127551899 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 -- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815c9f0 msgr2=0x7f859814c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815c9f0 0x7f859814c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 -- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85980c9230 msgr2=0x7f85980c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85980c9230 0x7f85980c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.862+0000 7f859dfe8640 1 -- 172.21.15.60:0/3127551899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85940094d0 con 0x7f859815dde0 2023-10-16T12:49:10.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.863+0000 7f859dfe8640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 0x7f859815a920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f859000b9e0 tx=0x7f859000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:10.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.863+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000cb20 con 0x7f859815dde0 2023-10-16T12:49:10.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.863+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859000cca0 con 0x7f859815dde0 2023-10-16T12:49:10.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.863+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590014680 con 0x7f859815dde0 2023-10-16T12:49:10.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.863+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85981f2d00 con 0x7f859815dde0 2023-10-16T12:49:10.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.863+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85981f32c0 con 0x7f859815dde0 2023-10-16T12:49:10.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.864+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598161670 con 0x7f859815dde0 2023-10-16T12:49:10.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.865+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f859001d3b0 con 0x7f859815dde0 2023-10-16T12:49:10.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.866+0000 7f858dffb640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f857c06af00 0x7f857c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:10.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.866+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f8590089860 con 0x7f859815dde0 2023-10-16T12:49:10.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.867+0000 7f859d7e7640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f857c06af00 0x7f857c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:10.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.868+0000 7f859d7e7640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f857c06af00 0x7f857c06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8594036000 tx=0x7f859403b040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:10.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.871+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8590054610 con 0x7f859815dde0 2023-10-16T12:49:10.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:10.996+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85900549e0 con 0x7f859815dde0 2023-10-16T12:49:11.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.109+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f859804e9b0 con 0x7f859815dde0 2023-10-16T12:49:11.111 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:49:11.110+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:49:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.114+0000 7f858dffb640 1 -- 172.21.15.60:0/3127551899 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8590072840 con 0x7f859815dde0 2023-10-16T12:49:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:49:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f857c06af00 msgr2=0x7f857c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f857c06af00 0x7f857c06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8594036000 tx=0x7f859403b040 comp rx=0 tx=0).stop 2023-10-16T12:49:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 msgr2=0x7f859815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 0x7f859815a920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f859000b9e0 tx=0x7f859000bea0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 shutdown_connections 2023-10-16T12:49:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f857c06af00 0x7f857c06d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f859815c9f0 0x7f859814c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859815dde0 0x7f859815a920 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 --2- 172.21.15.60:0/3127551899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85980c9230 0x7f85980c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 >> 172.21.15.60:0/3127551899 conn(0x7f85980ce6a0 msgr2=0x7f8598160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 shutdown_connections 2023-10-16T12:49:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.117+0000 7f859e7e9640 1 -- 172.21.15.60:0/3127551899 wait complete. 2023-10-16T12:49:11.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:49:11.246 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:49:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.357+0000 7fb4cf59e640 1 Processor -- start 2023-10-16T12:49:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.358+0000 7fb4cf59e640 1 -- start start 2023-10-16T12:49:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.359+0000 7fb4cf59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d015c9f0 0x7fb4d00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.359+0000 7fb4cf59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d0160a90 con 0x7fb4d015dde0 2023-10-16T12:49:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.360+0000 7fb4cf59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d00cd900 con 0x7fb4d00c9230 2023-10-16T12:49:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.360+0000 7fb4cf59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00cdec0 con 0x7fb4d015c9f0 2023-10-16T12:49:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.360+0000 7fb4ce59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d015c9f0 0x7fb4d00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.360+0000 7fb4ced9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb4d015dde0 0x7fb4d00ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38860/0 (socket says 172.21.15.60:38860) 2023-10-16T12:49:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.360+0000 7fb4ced9d640 1 -- 172.21.15.60:0/3406569469 learned_addr learned my addr 172.21.15.60:0/3406569469 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.360+0000 7fb4c7fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb4d00c9230 0x7fb4d00709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:47794/0 (socket says 172.21.15.60:47794) 2023-10-16T12:49:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.361+0000 7fb4cdd9b640 1 -- 172.21.15.60:0/3406569469 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776822579 0 0) 0x7fb4d0160a90 con 0x7fb4d015dde0 2023-10-16T12:49:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.361+0000 7fb4cdd9b640 1 -- 172.21.15.60:0/3406569469 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4bc003520 con 0x7fb4d015dde0 2023-10-16T12:49:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.361+0000 7fb4ce59c640 1 -- 172.21.15.60:0/3406569469 >> v1:172.21.15.144:6789/0 conn(0x7fb4d00c9230 legacy=0x7fb4d00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.361+0000 7fb4ce59c640 1 -- 172.21.15.60:0/3406569469 >> v1:172.21.15.60:6789/0 conn(0x7fb4d015dde0 legacy=0x7fb4d00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.361+0000 7fb4ce59c640 1 -- 172.21.15.60:0/3406569469 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d00cdbe0 con 0x7fb4d015c9f0 2023-10-16T12:49:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.362+0000 7fb4ce59c640 1 --2- 172.21.15.60:0/3406569469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d015c9f0 0x7fb4d00c8cf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00097d0 tx=0x7fb4c002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3049f9fbd2077c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.363+0000 7fb4cdd9b640 1 -- 172.21.15.60:0/3406569469 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0030800 con 0x7fb4d015c9f0 2023-10-16T12:49:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.363+0000 7fb4cdd9b640 1 -- 172.21.15.60:0/3406569469 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c0030960 con 0x7fb4d015c9f0 2023-10-16T12:49:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.363+0000 7fb4cdd9b640 1 -- 172.21.15.60:0/3406569469 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0030cf0 con 0x7fb4d015c9f0 2023-10-16T12:49:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 -- 172.21.15.60:0/3406569469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d015c9f0 msgr2=0x7fb4d00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/3406569469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d015c9f0 0x7fb4d00c8cf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00097d0 tx=0x7fb4c002b940 comp rx=0 tx=0).stop 2023-10-16T12:49:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 -- 172.21.15.60:0/3406569469 shutdown_connections 2023-10-16T12:49:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/3406569469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d015c9f0 0x7fb4d00c8cf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 -- 172.21.15.60:0/3406569469 >> 172.21.15.60:0/3406569469 conn(0x7fb4d00ce6a0 msgr2=0x7fb4d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 -- 172.21.15.60:0/3406569469 shutdown_connections 2023-10-16T12:49:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.364+0000 7fb4cf59e640 1 -- 172.21.15.60:0/3406569469 wait complete. 2023-10-16T12:49:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.366+0000 7fb4cf59e640 1 Processor -- start 2023-10-16T12:49:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.366+0000 7fb4cf59e640 1 -- start start 2023-10-16T12:49:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4cf59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d00c9230 0x7fb4d015e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4cf59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 0x7fb4d0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4cf59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4d015dde0 0x7fb4d01f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4cf59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d015f9b0 con 0x7fb4d015c9f0 2023-10-16T12:49:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4cf59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d01483a0 con 0x7fb4d015dde0 2023-10-16T12:49:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4cf59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d015ed30 con 0x7fb4d00c9230 2023-10-16T12:49:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4c7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 0x7fb4d0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4c7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 0x7fb4d0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47606/0 (socket says 172.21.15.60:47606) 2023-10-16T12:49:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4c7fff640 1 -- 172.21.15.60:0/2440066255 learned_addr learned my addr 172.21.15.60:0/2440066255 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4ce59c640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d00c9230 0x7fb4d015e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.367+0000 7fb4ced9d640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4d015dde0 0x7fb4d01f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c7fff640 1 -- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d00c9230 msgr2=0x7fb4d015e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c7fff640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d00c9230 0x7fb4d015e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c7fff640 1 -- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4d015dde0 msgr2=0x7fb4d01f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c7fff640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4d015dde0 0x7fb4d01f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c7fff640 1 -- 172.21.15.60:0/2440066255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0009400 con 0x7fb4d015c9f0 2023-10-16T12:49:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c7fff640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 0x7fb4d0069da0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800c500 tx=0x7fb4b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8007500 con 0x7fb4d015c9f0 2023-10-16T12:49:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b8007660 con 0x7fb4d015c9f0 2023-10-16T12:49:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80049e0 con 0x7fb4d015c9f0 2023-10-16T12:49:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d015f010 con 0x7fb4d015c9f0 2023-10-16T12:49:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.368+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d015f5d0 con 0x7fb4d015c9f0 2023-10-16T12:49:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.369+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d00aff40 con 0x7fb4d015c9f0 2023-10-16T12:49:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.371+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb4b8004b40 con 0x7fb4d015c9f0 2023-10-16T12:49:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.372+0000 7fb4c6ffd640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fb4b006af90 0x7fb4b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.372+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fb4b8088c60 con 0x7fb4d015c9f0 2023-10-16T12:49:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.372+0000 7fb4ce59c640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fb4b006af90 0x7fb4b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.373+0000 7fb4ce59c640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fb4b006af90 0x7fb4b006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0036000 tx=0x7fb4c0008030 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.376+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4b8053960 con 0x7fb4d015c9f0 2023-10-16T12:49:11.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:11.604+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb4d004e9b0 con 0x7fb4d015c9f0 2023-10-16T12:49:12.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.118+0000 7fb4c6ffd640 1 -- 172.21.15.60:0/2440066255 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4b80180a0 con 0x7fb4d015c9f0 2023-10-16T12:49:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fb4b006af90 msgr2=0x7fb4b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fb4b006af90 0x7fb4b006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0036000 tx=0x7fb4c0008030 comp rx=0 tx=0).stop 2023-10-16T12:49:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 msgr2=0x7fb4d0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 0x7fb4d0069da0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800c500 tx=0x7fb4b800c9c0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 shutdown_connections 2023-10-16T12:49:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fb4b006af90 0x7fb4b006d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4d015dde0 0x7fb4d01f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d015c9f0 0x7fb4d0069da0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 --2- 172.21.15.60:0/2440066255 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4d00c9230 0x7fb4d015e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 >> 172.21.15.60:0/2440066255 conn(0x7fb4d00ce6a0 msgr2=0x7fb4d00ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 shutdown_connections 2023-10-16T12:49:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.123+0000 7fb4cf59e640 1 -- 172.21.15.60:0/2440066255 wait complete. 2023-10-16T12:49:12.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:49:12.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.386+0000 7f8a44a14640 1 Processor -- start 2023-10-16T12:49:12.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.386+0000 7f8a44a14640 1 -- start start 2023-10-16T12:49:12.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a44a14640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a4006c040 0x7f8a4006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:12.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a44a14640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a400cda80 0x7f8a4015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:12.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a44a14640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4006c960 con 0x7f8a400ad970 2023-10-16T12:49:12.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a44a14640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4015b9d0 con 0x7f8a400cda80 2023-10-16T12:49:12.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a44a14640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4015bcb0 con 0x7f8a4006c040 2023-10-16T12:49:12.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a3f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a400cda80 0x7f8a4015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:12.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a3f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a400cda80 0x7f8a4015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47612/0 (socket says 172.21.15.60:47612) 2023-10-16T12:49:12.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.388+0000 7f8a3f59e640 1 -- 172.21.15.60:0/715360200 learned_addr learned my addr 172.21.15.60:0/715360200 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:12.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.389+0000 7f8a3ed9d640 1 --2- 172.21.15.60:0/715360200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a4006c040 0x7f8a4006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:12.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.389+0000 7f8a3f59e640 1 -- 172.21.15.60:0/715360200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a4006c040 msgr2=0x7f8a4006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:12.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.389+0000 7f8a3f59e640 1 --2- 172.21.15.60:0/715360200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a4006c040 0x7f8a4006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.389+0000 7f8a3f59e640 1 -- 172.21.15.60:0/715360200 >> v1:172.21.15.144:6789/0 conn(0x7f8a400ad970 legacy=0x7f8a4015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:12.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.389+0000 7f8a3f59e640 1 -- 172.21.15.60:0/715360200 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4015bf90 con 0x7f8a400cda80 2023-10-16T12:49:12.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.391+0000 7f8a3f59e640 1 --2- 172.21.15.60:0/715360200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a400cda80 0x7f8a4015ed90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009e50 tx=0x7f8a3802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a603fbf88719b6d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.391+0000 7f8a3e59c640 1 -- 172.21.15.60:0/715360200 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3802cb00 con 0x7f8a400cda80 2023-10-16T12:49:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.391+0000 7f8a3e59c640 1 -- 172.21.15.60:0/715360200 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3802cc60 con 0x7f8a400cda80 2023-10-16T12:49:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.391+0000 7f8a3e59c640 1 -- 172.21.15.60:0/715360200 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380343f0 con 0x7f8a400cda80 2023-10-16T12:49:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.391+0000 7f8a44a14640 1 -- 172.21.15.60:0/715360200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a400cda80 msgr2=0x7f8a4015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.391+0000 7f8a44a14640 1 --2- 172.21.15.60:0/715360200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a400cda80 0x7f8a4015ed90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009e50 tx=0x7f8a3802bf60 comp rx=0 tx=0).stop 2023-10-16T12:49:12.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.392+0000 7f8a44a14640 1 -- 172.21.15.60:0/715360200 shutdown_connections 2023-10-16T12:49:12.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.392+0000 7f8a44a14640 1 --2- 172.21.15.60:0/715360200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a400cda80 0x7f8a4015ed90 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.392+0000 7f8a44a14640 1 --2- 172.21.15.60:0/715360200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a4006c040 0x7f8a4006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.392+0000 7f8a44a14640 1 -- 172.21.15.60:0/715360200 >> 172.21.15.60:0/715360200 conn(0x7f8a400ce6a0 msgr2=0x7f8a400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:12.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.392+0000 7f8a44a14640 1 -- 172.21.15.60:0/715360200 shutdown_connections 2023-10-16T12:49:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.392+0000 7f8a44a14640 1 -- 172.21.15.60:0/715360200 wait complete. 2023-10-16T12:49:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.394+0000 7f8a44a14640 1 Processor -- start 2023-10-16T12:49:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.395+0000 7f8a44a14640 1 -- start start 2023-10-16T12:49:12.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a44a14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 0x7f8a4015de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:12.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a44a14640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a400ad970 0x7f8a40150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:12.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a44a14640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a400cda80 0x7f8a4015a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a44a14640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400b0fc0 con 0x7f8a4006c040 2023-10-16T12:49:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a44a14640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400b1280 con 0x7f8a400cda80 2023-10-16T12:49:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a44a14640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400b1560 con 0x7f8a400ad970 2023-10-16T12:49:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a3ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 0x7f8a4015de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a3ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 0x7f8a4015de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47620/0 (socket says 172.21.15.60:47620) 2023-10-16T12:49:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.396+0000 7f8a3ed9d640 1 -- 172.21.15.60:0/2303012098 learned_addr learned my addr 172.21.15.60:0/2303012098 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a37fff640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a400ad970 0x7f8a40150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3f59e640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a400cda80 0x7f8a4015a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3ed9d640 1 -- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a400ad970 msgr2=0x7f8a40150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3ed9d640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a400ad970 0x7f8a40150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3ed9d640 1 -- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a400cda80 msgr2=0x7f8a4015a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3ed9d640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a400cda80 0x7f8a4015a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3ed9d640 1 -- 172.21.15.60:0/2303012098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38009b30 con 0x7f8a4006c040 2023-10-16T12:49:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3f59e640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a400cda80 0x7f8a4015a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a3ed9d640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 0x7f8a4015de30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d5f0 tx=0x7f8a3000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a300125f0 con 0x7f8a4006c040 2023-10-16T12:49:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a30012750 con 0x7f8a4006c040 2023-10-16T12:49:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.397+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a400b1840 con 0x7f8a4006c040 2023-10-16T12:49:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.398+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a30012b00 con 0x7f8a4006c040 2023-10-16T12:49:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.398+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4015e370 con 0x7f8a4006c040 2023-10-16T12:49:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.399+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4006b330 con 0x7f8a4006c040 2023-10-16T12:49:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.400+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a30012c60 con 0x7f8a4006c040 2023-10-16T12:49:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.400+0000 7f8a377fe640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8a2006afc0 0x7f8a2006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.401+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6541+0+0 (secure 0 0 0) 0x7f8a30089fe0 con 0x7f8a4006c040 2023-10-16T12:49:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.401+0000 7f8a37fff640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8a2006afc0 0x7f8a2006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.402+0000 7f8a37fff640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8a2006afc0 0x7f8a2006d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a28007fd0 tx=0x7f8a28009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.405+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a30054ba0 con 0x7f8a4006c040 2023-10-16T12:49:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:12.600+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a4004e9b0 con 0x7f8a4006c040 2023-10-16T12:49:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.130+0000 7f8a377fe640 1 -- 172.21.15.60:0/2303012098 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v76) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8a300581c0 con 0x7f8a4006c040 2023-10-16T12:49:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:49:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8a2006afc0 msgr2=0x7f8a2006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8a2006afc0 0x7f8a2006d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a28007fd0 tx=0x7f8a28009040 comp rx=0 tx=0).stop 2023-10-16T12:49:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 msgr2=0x7f8a4015de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 0x7f8a4015de30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d5f0 tx=0x7f8a3000dab0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 shutdown_connections 2023-10-16T12:49:13.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8a2006afc0 0x7f8a2006d480 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a400ad970 0x7f8a40150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a400cda80 0x7f8a4015a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 --2- 172.21.15.60:0/2303012098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a4006c040 0x7f8a4015de30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 >> 172.21.15.60:0/2303012098 conn(0x7f8a400ce6a0 msgr2=0x7f8a400ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:13.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 shutdown_connections 2023-10-16T12:49:13.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.134+0000 7f8a44a14640 1 -- 172.21.15.60:0/2303012098 wait complete. 2023-10-16T12:49:13.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:49:13.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.391+0000 7f880f59e640 1 Processor -- start 2023-10-16T12:49:13.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.391+0000 7f880f59e640 1 -- start start 2023-10-16T12:49:13.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f880f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f88100ad970 0x7f88100cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:13.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f880f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881015b9d0 con 0x7f88100ccfb0 2023-10-16T12:49:13.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f880f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881015bcb0 con 0x7f881006c040 2023-10-16T12:49:13.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f880f59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881015bf90 con 0x7f88100ad970 2023-10-16T12:49:13.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f8807fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f88100ad970 0x7f88100cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:13.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f880ed9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f88100ccfb0 0x7f88100cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38862/0 (socket says 172.21.15.60:38862) 2023-10-16T12:49:13.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.393+0000 7f880ed9d640 1 -- 172.21.15.60:0/2590536710 learned_addr learned my addr 172.21.15.60:0/2590536710 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:13.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.394+0000 7f8807fff640 1 -- 172.21.15.60:0/2590536710 >> v1:172.21.15.144:6789/0 conn(0x7f881006c040 legacy=0x7f881006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:13.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.394+0000 7f8807fff640 1 -- 172.21.15.60:0/2590536710 >> v1:172.21.15.60:6789/0 conn(0x7f88100ccfb0 legacy=0x7f88100cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:13.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.394+0000 7f8807fff640 1 -- 172.21.15.60:0/2590536710 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100cc270 con 0x7f88100ad970 2023-10-16T12:49:13.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.394+0000 7f880dd9b640 1 -- 172.21.15.60:0/2590536710 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763101010 0 0) 0x7f881015b9d0 con 0x7f88100ccfb0 2023-10-16T12:49:13.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.396+0000 7f8807fff640 1 --2- 172.21.15.60:0/2590536710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ad970 0x7f88100cca70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f87f80097d0 tx=0x7f87f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8629ea843b4e088 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.396+0000 7f880dd9b640 1 -- 172.21.15.60:0/2590536710 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8031830 con 0x7f88100ad970 2023-10-16T12:49:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.396+0000 7f880dd9b640 1 -- 172.21.15.60:0/2590536710 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f8031990 con 0x7f88100ad970 2023-10-16T12:49:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.396+0000 7f880dd9b640 1 -- 172.21.15.60:0/2590536710 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8031d20 con 0x7f88100ad970 2023-10-16T12:49:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.396+0000 7f880f59e640 1 -- 172.21.15.60:0/2590536710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ad970 msgr2=0x7f88100cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:13.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.396+0000 7f880f59e640 1 --2- 172.21.15.60:0/2590536710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ad970 0x7f88100cca70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f87f80097d0 tx=0x7f87f802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.397+0000 7f880f59e640 1 -- 172.21.15.60:0/2590536710 shutdown_connections 2023-10-16T12:49:13.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.397+0000 7f880f59e640 1 --2- 172.21.15.60:0/2590536710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ad970 0x7f88100cca70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.397+0000 7f880f59e640 1 -- 172.21.15.60:0/2590536710 >> 172.21.15.60:0/2590536710 conn(0x7f88100ce6a0 msgr2=0x7f88100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:13.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.397+0000 7f880f59e640 1 -- 172.21.15.60:0/2590536710 shutdown_connections 2023-10-16T12:49:13.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.397+0000 7f880f59e640 1 -- 172.21.15.60:0/2590536710 wait complete. 2023-10-16T12:49:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.399+0000 7f880f59e640 1 Processor -- start 2023-10-16T12:49:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.399+0000 7f880f59e640 1 -- start start 2023-10-16T12:49:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f881006c040 0x7f8810153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88100ad970 0x7f88101531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:13.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 0x7f88100b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100ae7f0 con 0x7f88100ccfb0 2023-10-16T12:49:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100af370 con 0x7f88100ad970 2023-10-16T12:49:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88101540e0 con 0x7f881006c040 2023-10-16T12:49:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 0x7f88100b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:13.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 0x7f88100b3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47644/0 (socket says 172.21.15.60:47644) 2023-10-16T12:49:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880ed9d640 1 -- 172.21.15.60:0/880470895 learned_addr learned my addr 172.21.15.60:0/880470895 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f880e59c640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f881006c040 0x7f8810153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.400+0000 7f8807fff640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88100ad970 0x7f88101531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880ed9d640 1 -- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f881006c040 msgr2=0x7f8810153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880ed9d640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f881006c040 0x7f8810153840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880ed9d640 1 -- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88100ad970 msgr2=0x7f88101531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880ed9d640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88100ad970 0x7f88101531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880ed9d640 1 -- 172.21.15.60:0/880470895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8009400 con 0x7f88100ccfb0 2023-10-16T12:49:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f8807fff640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88100ad970 0x7f88101531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880ed9d640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 0x7f88100b3930 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f880800c7e0 tx=0x7f880800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800a640 con 0x7f88100ccfb0 2023-10-16T12:49:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880800a7c0 con 0x7f88100ccfb0 2023-10-16T12:49:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800ab70 con 0x7f88100ccfb0 2023-10-16T12:49:13.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88100aea40 con 0x7f88100ccfb0 2023-10-16T12:49:13.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.401+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88100af000 con 0x7f88100ccfb0 2023-10-16T12:49:13.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.402+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100af7e0 con 0x7f88100ccfb0 2023-10-16T12:49:13.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.403+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f880800acd0 con 0x7f88100ccfb0 2023-10-16T12:49:13.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.404+0000 7f8806ffd640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f87f006af90 0x7f87f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:13.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.404+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6103+0+0 (secure 0 0 0) 0x7f8808088f50 con 0x7f88100ccfb0 2023-10-16T12:49:13.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.405+0000 7f880e59c640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f87f006af90 0x7f87f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:13.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.406+0000 7f880e59c640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f87f006af90 0x7f87f006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8800006fd0 tx=0x7f8800008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:13.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.409+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88100af7e0 con 0x7f88100ccfb0 2023-10-16T12:49:13.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:13.604+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f88100af7e0 con 0x7f88100ccfb0 2023-10-16T12:49:14.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.145+0000 7f8806ffd640 1 -- 172.21.15.60:0/880470895 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v77) v1 ==== 169+0+0 (secure 0 0 0) 0x7f88100af7e0 con 0x7f88100ccfb0 2023-10-16T12:49:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:49:14.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.150+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f87f006af90 msgr2=0x7f87f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.150+0000 7f880f59e640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f87f006af90 0x7f87f006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8800006fd0 tx=0x7f8800008040 comp rx=0 tx=0).stop 2023-10-16T12:49:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.150+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 msgr2=0x7f88100b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.150+0000 7f880f59e640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 0x7f88100b3930 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f880800c7e0 tx=0x7f880800cca0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.151+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 shutdown_connections 2023-10-16T12:49:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.151+0000 7f880f59e640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f87f006af90 0x7f87f006d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.151+0000 7f880f59e640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88100ccfb0 0x7f88100b3930 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.151+0000 7f880f59e640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88100ad970 0x7f88101531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.151+0000 7f880f59e640 1 --2- 172.21.15.60:0/880470895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f881006c040 0x7f8810153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.151+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 >> 172.21.15.60:0/880470895 conn(0x7f88100ce6a0 msgr2=0x7f881006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.155+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 shutdown_connections 2023-10-16T12:49:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.155+0000 7f880f59e640 1 -- 172.21.15.60:0/880470895 wait complete. 2023-10-16T12:49:14.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:49:14.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.390+0000 7fcbeffff640 1 Processor -- start 2023-10-16T12:49:14.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.390+0000 7fcbeffff640 1 -- start start 2023-10-16T12:49:14.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.392+0000 7fcbeffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcbf00cd970 0x7fcbf015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.392+0000 7fcbeffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf006be30 con 0x7fcbf00b3ec0 2023-10-16T12:49:14.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.392+0000 7fcbeffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00ada10 con 0x7fcbf00b0ea0 2023-10-16T12:49:14.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.392+0000 7fcbeffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf006ca00 con 0x7fcbf00cd970 2023-10-16T12:49:14.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.392+0000 7fcbeeffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcbf00b3ec0 0x7fcbf00cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38874/0 (socket says 172.21.15.60:38874) 2023-10-16T12:49:14.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.392+0000 7fcbeeffd640 1 -- 172.21.15.60:0/262420730 learned_addr learned my addr 172.21.15.60:0/262420730 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:14.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610228539 0 0) 0x7fcbf006be30 con 0x7fcbf00b3ec0 2023-10-16T12:49:14.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc003520 con 0x7fcbf00b3ec0 2023-10-16T12:49:14.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbef7fe640 1 --2- 172.21.15.60:0/262420730 >> v2:172.21.15.144:3300/0 conn(0x7fcbf00cd970 0x7fcbf015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431793981 0 0) 0x7fcbdc003520 con 0x7fcbf00b3ec0 2023-10-16T12:49:14.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 >> v2:172.21.15.144:3300/0 conn(0x7fcbf00cd970 msgr2=0x7fcbf015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:14.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 --2- 172.21.15.60:0/262420730 >> v2:172.21.15.144:3300/0 conn(0x7fcbf00cd970 0x7fcbf015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 >> v1:172.21.15.144:6789/0 conn(0x7fcbf00b0ea0 legacy=0x7fcbf015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf006c060 con 0x7fcbf00b3ec0 2023-10-16T12:49:14.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcbe0001670 con 0x7fcbf00b3ec0 2023-10-16T12:49:14.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 >> v1:172.21.15.60:6789/0 conn(0x7fcbf00b3ec0 legacy=0x7fcbf00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbdc0039c0 0x7fcbdc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbdc0047c0 0x7fcbdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0052a0 0x7fcbdc00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf006ca00 con 0x7fcbdc0047c0 2023-10-16T12:49:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc0105d0 con 0x7fcbdc0052a0 2023-10-16T12:49:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.393+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc010790 con 0x7fcbdc0039c0 2023-10-16T12:49:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbdc0047c0 0x7fcbdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbeeffd640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbdc0039c0 0x7fcbdc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbef7fe640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0052a0 0x7fcbdc00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbe7fff640 1 -- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbdc0039c0 msgr2=0x7fcbdc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbdc0039c0 0x7fcbdc003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbe7fff640 1 -- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0052a0 msgr2=0x7fcbdc00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0052a0 0x7fcbdc00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbe7fff640 1 -- 172.21.15.60:0/262420730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc003520 con 0x7fcbdc0047c0 2023-10-16T12:49:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbef7fe640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0052a0 0x7fcbdc00fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.394+0000 7fcbef7fe640 1 -- 172.21.15.60:0/262420730 reap_dead start 2023-10-16T12:49:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.395+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbdc0047c0 0x7fcbdc004c40 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8001750 tx=0x7fcbdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d7b2d43cdea0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:14.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80052d0 con 0x7fcbdc0047c0 2023-10-16T12:49:14.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd802e050 con 0x7fcbdc0047c0 2023-10-16T12:49:14.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbee7fc640 1 -- 172.21.15.60:0/262420730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8003830 con 0x7fcbdc0047c0 2023-10-16T12:49:14.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 -- 172.21.15.60:0/262420730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbdc0047c0 msgr2=0x7fcbdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbdc0047c0 0x7fcbdc004c40 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8001750 tx=0x7fcbdc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 -- 172.21.15.60:0/262420730 shutdown_connections 2023-10-16T12:49:14.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 --2- 172.21.15.60:0/262420730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbdc0047c0 0x7fcbdc004c40 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 -- 172.21.15.60:0/262420730 >> 172.21.15.60:0/262420730 conn(0x7fcbf00b0170 msgr2=0x7fcbf00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:14.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 -- 172.21.15.60:0/262420730 shutdown_connections 2023-10-16T12:49:14.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.396+0000 7fcbeffff640 1 -- 172.21.15.60:0/262420730 wait complete. 2023-10-16T12:49:14.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.398+0000 7fcbeffff640 1 Processor -- start 2023-10-16T12:49:14.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.398+0000 7fcbeffff640 1 -- start start 2023-10-16T12:49:14.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.399+0000 7fcbeffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0047c0 0x7fcbf00c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.399+0000 7fcbeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 0x7fcbf01efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbeffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbf0151790 0x7fcbf01effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbeffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00c3c10 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbeffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00c41d0 con 0x7fcbdc0047c0 2023-10-16T12:49:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbeffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0155190 con 0x7fcbf0151790 2023-10-16T12:49:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 0x7fcbf01efaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 0x7fcbf01efaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47662/0 (socket says 172.21.15.60:47662) 2023-10-16T12:49:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 -- 172.21.15.60:0/3144515664 learned_addr learned my addr 172.21.15.60:0/3144515664 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbeeffd640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0047c0 0x7fcbf00c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbef7fe640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbf0151790 0x7fcbf01effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 -- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbf0151790 msgr2=0x7fcbf01effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbf0151790 0x7fcbf01effe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 -- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0047c0 msgr2=0x7fcbf00c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0047c0 0x7fcbf00c4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbe7fff640 1 -- 172.21.15.60:0/3144515664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf0155470 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.400+0000 7fcbeeffd640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0047c0 0x7fcbf00c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.401+0000 7fcbe7fff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 0x7fcbf01efaa0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8005040 tx=0x7fcbd8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.401+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80076c0 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.401+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8007a30 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.401+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf00c3e60 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.401+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8006be0 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.401+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf0152030 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.402+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf004e9b0 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.403+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcbd8006d40 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.405+0000 7fcbecff9640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fcbd006af90 0x7fcbd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.405+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fcbd80ab410 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.405+0000 7fcbeeffd640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fcbd006af90 0x7fcbd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.407+0000 7fcbeeffd640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fcbd006af90 0x7fcbd006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0000c00 tx=0x7fcbe0009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.409+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbd80762e0 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.608+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fcbf00c4310 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.614+0000 7fcbecff9640 1 -- 172.21.15.60:0/3144515664 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcbd8079900 con 0x7fcbf00c4ba0 2023-10-16T12:49:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fcbd006af90 msgr2=0x7fcbd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fcbd006af90 0x7fcbd006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0000c00 tx=0x7fcbe0009040 comp rx=0 tx=0).stop 2023-10-16T12:49:14.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 msgr2=0x7fcbf01efaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 0x7fcbf01efaa0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8005040 tx=0x7fcbd8001120 comp rx=0 tx=0).stop 2023-10-16T12:49:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 shutdown_connections 2023-10-16T12:49:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fcbd006af90 0x7fcbd006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbf0151790 0x7fcbf01effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00c4ba0 0x7fcbf01efaa0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 --2- 172.21.15.60:0/3144515664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbdc0047c0 0x7fcbf00c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 >> 172.21.15.60:0/3144515664 conn(0x7fcbf00b0170 msgr2=0x7fcbf00b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 shutdown_connections 2023-10-16T12:49:14.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.617+0000 7fcbeffff640 1 -- 172.21.15.60:0/3144515664 wait complete. 2023-10-16T12:49:14.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:49:14.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.869+0000 7f6b49fa0640 1 Processor -- start 2023-10-16T12:49:14.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.869+0000 7f6b49fa0640 1 -- start start 2023-10-16T12:49:14.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b49fa0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b4406c040 0x7f6b4406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b49fa0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440cda80 0x7f6b4415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b49fa0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4406c960 con 0x7f6b440ad970 2023-10-16T12:49:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b49fa0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4415b9d0 con 0x7f6b4406c040 2023-10-16T12:49:14.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b49fa0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4415bcb0 con 0x7f6b440cda80 2023-10-16T12:49:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b48f9e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b4406c040 0x7f6b4406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b48f9e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b4406c040 0x7f6b4406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47678/0 (socket says 172.21.15.60:47678) 2023-10-16T12:49:14.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b3bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b440ad970 0x7f6b4415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38876/0 (socket says 172.21.15.60:38876) 2023-10-16T12:49:14.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.870+0000 7f6b48f9e640 1 -- 172.21.15.60:0/3432615123 learned_addr learned my addr 172.21.15.60:0/3432615123 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:14.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.871+0000 7f6b4979f640 1 --2- 172.21.15.60:0/3432615123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440cda80 0x7f6b4415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.871+0000 7f6b48f9e640 1 -- 172.21.15.60:0/3432615123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440cda80 msgr2=0x7f6b4415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:14.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.871+0000 7f6b48f9e640 1 --2- 172.21.15.60:0/3432615123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440cda80 0x7f6b4415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.871+0000 7f6b48f9e640 1 -- 172.21.15.60:0/3432615123 >> v1:172.21.15.60:6789/0 conn(0x7f6b440ad970 legacy=0x7f6b4415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.871+0000 7f6b48f9e640 1 -- 172.21.15.60:0/3432615123 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4415bf90 con 0x7f6b4406c040 2023-10-16T12:49:14.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.871+0000 7f6b3b7fe640 1 -- 172.21.15.60:0/3432615123 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721984862 0 0) 0x7f6b4406c960 con 0x7f6b440ad970 2023-10-16T12:49:14.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.872+0000 7f6b48f9e640 1 --2- 172.21.15.60:0/3432615123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b4406c420 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009e50 tx=0x7f6b4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94102da914411ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:14.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b3b7fe640 1 -- 172.21.15.60:0/3432615123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4002cb00 con 0x7f6b4406c040 2023-10-16T12:49:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b3b7fe640 1 -- 172.21.15.60:0/3432615123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4002cc60 con 0x7f6b4406c040 2023-10-16T12:49:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b3b7fe640 1 -- 172.21.15.60:0/3432615123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400343f0 con 0x7f6b4406c040 2023-10-16T12:49:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 -- 172.21.15.60:0/3432615123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 msgr2=0x7f6b4406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/3432615123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b4406c420 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009e50 tx=0x7f6b4002bf60 comp rx=0 tx=0).stop 2023-10-16T12:49:14.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 -- 172.21.15.60:0/3432615123 shutdown_connections 2023-10-16T12:49:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/3432615123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440cda80 0x7f6b4415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/3432615123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b4406c420 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 -- 172.21.15.60:0/3432615123 >> 172.21.15.60:0/3432615123 conn(0x7f6b440ce6a0 msgr2=0x7f6b440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:14.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 -- 172.21.15.60:0/3432615123 shutdown_connections 2023-10-16T12:49:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.873+0000 7f6b49fa0640 1 -- 172.21.15.60:0/3432615123 wait complete. 2023-10-16T12:49:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.875+0000 7f6b49fa0640 1 Processor -- start 2023-10-16T12:49:14.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.875+0000 7f6b49fa0640 1 -- start start 2023-10-16T12:49:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b49fa0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b441f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b49fa0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440ad970 0x7f6b440c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b49fa0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b440cda80 0x7f6b440ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b49fa0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440af6f0 con 0x7f6b4406c040 2023-10-16T12:49:14.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b49fa0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441f31a0 con 0x7f6b440cda80 2023-10-16T12:49:14.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b49fa0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440aeae0 con 0x7f6b440ad970 2023-10-16T12:49:14.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b48f9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b441f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b48f9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b441f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47692/0 (socket says 172.21.15.60:47692) 2023-10-16T12:49:14.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.876+0000 7f6b48f9e640 1 -- 172.21.15.60:0/1448048377 learned_addr learned my addr 172.21.15.60:0/1448048377 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:14.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b3bfff640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440ad970 0x7f6b440c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b48f9e640 1 -- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440ad970 msgr2=0x7f6b440c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b4979f640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b440cda80 0x7f6b440ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b48f9e640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440ad970 0x7f6b440c4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b48f9e640 1 -- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b440cda80 msgr2=0x7f6b440ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b48f9e640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b440cda80 0x7f6b440ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b48f9e640 1 -- 172.21.15.60:0/1448048377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40009b30 con 0x7f6b4406c040 2023-10-16T12:49:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b4979f640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b440cda80 0x7f6b440ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.877+0000 7f6b48f9e640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b441f2c60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6b40036040 tx=0x7f6b40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.878+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40009510 con 0x7f6b4406c040 2023-10-16T12:49:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.878+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4003e450 con 0x7f6b4406c040 2023-10-16T12:49:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.878+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4003e800 con 0x7f6b4406c040 2023-10-16T12:49:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.878+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b440aedc0 con 0x7f6b4406c040 2023-10-16T12:49:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.878+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b440af380 con 0x7f6b4406c040 2023-10-16T12:49:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.879+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4406b330 con 0x7f6b4406c040 2023-10-16T12:49:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.880+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b4003e960 con 0x7f6b4406c040 2023-10-16T12:49:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.881+0000 7f6b397fa640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6b3006af90 0x7f6b3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.881+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f6b400ac570 con 0x7f6b4406c040 2023-10-16T12:49:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.882+0000 7f6b3bfff640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6b3006af90 0x7f6b3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.883+0000 7f6b3bfff640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6b3006af90 0x7f6b3006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b34007fd0 tx=0x7f6b34009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:14.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:14.886+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b40077550 con 0x7f6b4406c040 2023-10-16T12:49:15.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.096+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6b440af510 con 0x7f6b4406c040 2023-10-16T12:49:15.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.101+0000 7f6b397fa640 1 -- 172.21.15.60:0/1448048377 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6b4007ab70 con 0x7f6b4406c040 2023-10-16T12:49:15.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6b3006af90 msgr2=0x7f6b3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6b3006af90 0x7f6b3006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b34007fd0 tx=0x7f6b34009040 comp rx=0 tx=0).stop 2023-10-16T12:49:15.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 msgr2=0x7f6b441f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b441f2c60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6b40036040 tx=0x7f6b40004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:15.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 shutdown_connections 2023-10-16T12:49:15.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6b3006af90 0x7f6b3006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b440ad970 0x7f6b440c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b440cda80 0x7f6b440ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 --2- 172.21.15.60:0/1448048377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b4406c040 0x7f6b441f2c60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 >> 172.21.15.60:0/1448048377 conn(0x7f6b440ce6a0 msgr2=0x7f6b440ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:15.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 shutdown_connections 2023-10-16T12:49:15.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.104+0000 7f6b49fa0640 1 -- 172.21.15.60:0/1448048377 wait complete. 2023-10-16T12:49:15.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:49:15.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.336+0000 7fa878ba9640 1 Processor -- start 2023-10-16T12:49:15.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.336+0000 7fa878ba9640 1 -- start start 2023-10-16T12:49:15.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa878ba9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa8740caac0 0x7fa8740ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa878ba9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740cb030 con 0x7fa87415ad70 2023-10-16T12:49:15.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa878ba9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740c91c0 con 0x7fa87406a4b0 2023-10-16T12:49:15.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa878ba9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8740c94a0 con 0x7fa8740caac0 2023-10-16T12:49:15.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa87259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa87415ad70 0x7fa874069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38878/0 (socket says 172.21.15.60:38878) 2023-10-16T12:49:15.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa87259c640 1 -- 172.21.15.60:0/2013131253 learned_addr learned my addr 172.21.15.60:0/2013131253 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:15.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.338+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188971830 0 0) 0x7fa8740cb030 con 0x7fa87415ad70 2023-10-16T12:49:15.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa860003520 con 0x7fa87415ad70 2023-10-16T12:49:15.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa872d9d640 1 --2- 172.21.15.60:0/2013131253 >> v2:172.21.15.144:3300/0 conn(0x7fa8740caac0 0x7fa8740ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 536480166 0 0) 0x7fa860003520 con 0x7fa87415ad70 2023-10-16T12:49:15.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 >> v2:172.21.15.144:3300/0 conn(0x7fa8740caac0 msgr2=0x7fa8740ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 --2- 172.21.15.60:0/2013131253 >> v2:172.21.15.144:3300/0 conn(0x7fa8740caac0 0x7fa8740ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 >> v1:172.21.15.144:6789/0 conn(0x7fa87406a4b0 legacy=0x7fa8740ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8740c9780 con 0x7fa87415ad70 2023-10-16T12:49:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa864001670 con 0x7fa87415ad70 2023-10-16T12:49:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 >> v1:172.21.15.60:6789/0 conn(0x7fa87415ad70 legacy=0x7fa874069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa860003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8600051d0 0x7fa86000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740c94a0 con 0x7fa8600039c0 2023-10-16T12:49:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86000c120 con 0x7fa8600051d0 2023-10-16T12:49:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86000c2e0 con 0x7fa8600047c0 2023-10-16T12:49:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.339+0000 7fa87259c640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa860003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa86bfff640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa87259c640 1 -- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8600047c0 msgr2=0x7fa860004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa872d9d640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8600051d0 0x7fa86000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa87259c640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa87259c640 1 -- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8600051d0 msgr2=0x7fa86000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa87259c640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8600051d0 0x7fa86000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa87259c640 1 -- 172.21.15.60:0/2013131253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860003520 con 0x7fa8600039c0 2023-10-16T12:49:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa86bfff640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8600047c0 0x7fa860004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa872d9d640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8600051d0 0x7fa86000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.340+0000 7fa872d9d640 1 -- 172.21.15.60:0/2013131253 reap_dead start 2023-10-16T12:49:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.341+0000 7fa87259c640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa860003de0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa86400c0e0 tx=0x7fa8600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9c8a89de276b37 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864005ad0 con 0x7fa8600039c0 2023-10-16T12:49:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864005c50 con 0x7fa8600039c0 2023-10-16T12:49:15.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa871d9b640 1 -- 172.21.15.60:0/2013131253 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86402c7f0 con 0x7fa8600039c0 2023-10-16T12:49:15.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 -- 172.21.15.60:0/2013131253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 msgr2=0x7fa860003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa860003de0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa86400c0e0 tx=0x7fa8600057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 -- 172.21.15.60:0/2013131253 shutdown_connections 2023-10-16T12:49:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 --2- 172.21.15.60:0/2013131253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa860003de0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 -- 172.21.15.60:0/2013131253 >> 172.21.15.60:0/2013131253 conn(0x7fa8740ce9b0 msgr2=0x7fa8740cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 -- 172.21.15.60:0/2013131253 shutdown_connections 2023-10-16T12:49:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.342+0000 7fa878ba9640 1 -- 172.21.15.60:0/2013131253 wait complete. 2023-10-16T12:49:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.344+0000 7fa878ba9640 1 Processor -- start 2023-10-16T12:49:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.344+0000 7fa878ba9640 1 -- start start 2023-10-16T12:49:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.345+0000 7fa878ba9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa87414e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.345+0000 7fa878ba9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa87414a590 0x7fa87414aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.345+0000 7fa878ba9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8741486c0 0x7fa874148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.345+0000 7fa878ba9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87414af50 con 0x7fa8600039c0 2023-10-16T12:49:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.345+0000 7fa878ba9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8741493f0 con 0x7fa8741486c0 2023-10-16T12:49:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.345+0000 7fa878ba9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8741496d0 con 0x7fa87414a590 2023-10-16T12:49:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa87414e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa87414e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47722/0 (socket says 172.21.15.60:47722) 2023-10-16T12:49:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 -- 172.21.15.60:0/1047763134 learned_addr learned my addr 172.21.15.60:0/1047763134 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa87414a590 0x7fa87414aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa872d9d640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8741486c0 0x7fa874148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 -- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa87414a590 msgr2=0x7fa87414aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa87414a590 0x7fa87414aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 -- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8741486c0 msgr2=0x7fa874148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8741486c0 0x7fa874148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.346+0000 7fa87259c640 1 -- 172.21.15.60:0/1047763134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8741499b0 con 0x7fa8600039c0 2023-10-16T12:49:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.347+0000 7fa87259c640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa87414e470 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa864000c00 tx=0x7fa86402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.347+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86403c040 con 0x7fa8600039c0 2023-10-16T12:49:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.347+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86402cce0 con 0x7fa8600039c0 2023-10-16T12:49:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.347+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640072b0 con 0x7fa8600039c0 2023-10-16T12:49:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.347+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa874149c90 con 0x7fa8600039c0 2023-10-16T12:49:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.347+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87414e9b0 con 0x7fa8600039c0 2023-10-16T12:49:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.348+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87404e160 con 0x7fa8600039c0 2023-10-16T12:49:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.349+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa864032070 con 0x7fa8600039c0 2023-10-16T12:49:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.350+0000 7fa86affd640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa85406b010 0x7fa85406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.351+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fa8640aab00 con 0x7fa8600039c0 2023-10-16T12:49:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.351+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa85406b010 0x7fa85406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.352+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa85406b010 0x7fa85406d4d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa8740b3dc0 tx=0x7fa85c009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.355+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8640ace60 con 0x7fa8600039c0 2023-10-16T12:49:15.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.552+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa874069f40 con 0x7fa8600039c0 2023-10-16T12:49:15.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.558+0000 7fa86affd640 1 -- 172.21.15.60:0/1047763134 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa864075b60 con 0x7fa8600039c0 2023-10-16T12:49:15.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa85406b010 msgr2=0x7fa85406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa85406b010 0x7fa85406d4d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa8740b3dc0 tx=0x7fa85c009040 comp rx=0 tx=0).stop 2023-10-16T12:49:15.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 msgr2=0x7fa87414e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa87414e470 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa864000c00 tx=0x7fa86402e070 comp rx=0 tx=0).stop 2023-10-16T12:49:15.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 shutdown_connections 2023-10-16T12:49:15.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa85406b010 0x7fa85406d4d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa87414a590 0x7fa87414aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8741486c0 0x7fa874148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 --2- 172.21.15.60:0/1047763134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8600039c0 0x7fa87414e470 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.560+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 >> 172.21.15.60:0/1047763134 conn(0x7fa8740ce9b0 msgr2=0x7fa8740c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:15.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.561+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 shutdown_connections 2023-10-16T12:49:15.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.561+0000 7fa878ba9640 1 -- 172.21.15.60:0/1047763134 wait complete. 2023-10-16T12:49:15.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T12:49:15.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.806+0000 7f28ecae7640 1 Processor -- start 2023-10-16T12:49:15.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.807+0000 7f28ecae7640 1 -- start start 2023-10-16T12:49:15.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.808+0000 7f28ecae7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 0x7f28e815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.808+0000 7f28ecae7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e815ed60 con 0x7f28e80ce390 2023-10-16T12:49:15.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.808+0000 7f28ecae7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80707e0 con 0x7f28e815ae80 2023-10-16T12:49:15.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.808+0000 7f28ecae7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e806b810 con 0x7f28e815aaa0 2023-10-16T12:49:15.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.809+0000 7f28e6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f28e80ce390 0x7f28e8070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38894/0 (socket says 172.21.15.60:38894) 2023-10-16T12:49:15.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.809+0000 7f28e6d9d640 1 -- 172.21.15.60:0/4043311404 learned_addr learned my addr 172.21.15.60:0/4043311404 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:15.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.809+0000 7f28e659c640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 0x7f28e815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.809+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251475654 0 0) 0x7f28e815ed60 con 0x7f28e80ce390 2023-10-16T12:49:15.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.809+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d4003520 con 0x7f28e80ce390 2023-10-16T12:49:15.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 500624786 0 0) 0x7f28d4003520 con 0x7f28e80ce390 2023-10-16T12:49:15.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 msgr2=0x7f28e815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:15.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 0x7f28e815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 >> v1:172.21.15.144:6789/0 conn(0x7f28e815ae80 legacy=0x7f28e80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e815ef90 con 0x7f28e80ce390 2023-10-16T12:49:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f28e00019f0 con 0x7f28e80ce390 2023-10-16T12:49:15.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 >> v1:172.21.15.60:6789/0 conn(0x7f28e80ce390 legacy=0x7f28e8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d40039a0 0x7f28d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 0x7f28d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d4004d00 0x7f28d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e806b810 con 0x7f28d4004d00 2023-10-16T12:49:15.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d400ba60 con 0x7f28d40039a0 2023-10-16T12:49:15.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d400bc20 con 0x7f28e815aaa0 2023-10-16T12:49:15.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e6d9d640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d4004d00 0x7f28d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28dffff640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 0x7f28d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e659c640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d40039a0 0x7f28d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e6d9d640 1 -- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 msgr2=0x7f28d40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:15.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.810+0000 7f28e6d9d640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815aaa0 0x7f28d40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.811+0000 7f28e6d9d640 1 -- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d40039a0 msgr2=0x7f28d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:15.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.811+0000 7f28e6d9d640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d40039a0 0x7f28d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.811+0000 7f28e6d9d640 1 -- 172.21.15.60:0/4043311404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4003520 con 0x7f28d4004d00 2023-10-16T12:49:15.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.811+0000 7f28e659c640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d40039a0 0x7f28d4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.812+0000 7f28e6d9d640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d4004d00 0x7f28d4005180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f28e0002230 tx=0x7f28d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc583a739783a10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.812+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0034630 con 0x7f28d4004d00 2023-10-16T12:49:15.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.812+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e002f070 con 0x7f28d4004d00 2023-10-16T12:49:15.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.812+0000 7f28e5d9b640 1 -- 172.21.15.60:0/4043311404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0034a10 con 0x7f28d4004d00 2023-10-16T12:49:15.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28ecae7640 1 -- 172.21.15.60:0/4043311404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d4004d00 msgr2=0x7f28d4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28ecae7640 1 --2- 172.21.15.60:0/4043311404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d4004d00 0x7f28d4005180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f28e0002230 tx=0x7f28d40057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28e6d9d640 1 -- 172.21.15.60:0/4043311404 reap_dead start 2023-10-16T12:49:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28ecae7640 1 -- 172.21.15.60:0/4043311404 shutdown_connections 2023-10-16T12:49:15.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28ecae7640 1 -- 172.21.15.60:0/4043311404 >> 172.21.15.60:0/4043311404 conn(0x7f28e80cb410 msgr2=0x7f28e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:15.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28ecae7640 1 -- 172.21.15.60:0/4043311404 shutdown_connections 2023-10-16T12:49:15.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.813+0000 7f28ecae7640 1 -- 172.21.15.60:0/4043311404 wait complete. 2023-10-16T12:49:15.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.815+0000 7f28ecae7640 1 Processor -- start 2023-10-16T12:49:15.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.815+0000 7f28ecae7640 1 -- start start 2023-10-16T12:49:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28ecae7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d4004d00 0x7f28e81594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28ecae7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 0x7f28e80c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28ecae7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815d6c0 0x7f28e815db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28ecae7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e81f3c60 con 0x7f28e814a750 2023-10-16T12:49:15.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28ecae7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e81f3f40 con 0x7f28d4004d00 2023-10-16T12:49:15.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28ecae7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e81f4220 con 0x7f28e815d6c0 2023-10-16T12:49:15.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 0x7f28e80c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 0x7f28e80c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47732/0 (socket says 172.21.15.60:47732) 2023-10-16T12:49:15.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 -- 172.21.15.60:0/186848901 learned_addr learned my addr 172.21.15.60:0/186848901 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:15.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 -- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815d6c0 msgr2=0x7f28e815db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815d6c0 0x7f28e815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 -- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d4004d00 msgr2=0x7f28e81594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:15.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d4004d00 0x7f28e81594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.816+0000 7f28dffff640 1 -- 172.21.15.60:0/186848901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e81f4500 con 0x7f28e814a750 2023-10-16T12:49:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.817+0000 7f28dffff640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 0x7f28e80c5a60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f28d0001eb0 tx=0x7f28d000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.817+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00052e0 con 0x7f28e814a750 2023-10-16T12:49:15.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.817+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d0005440 con 0x7f28e814a750 2023-10-16T12:49:15.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.817+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0003830 con 0x7f28e814a750 2023-10-16T12:49:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.817+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e815e060 con 0x7f28e814a750 2023-10-16T12:49:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.817+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e8159bf0 con 0x7f28e814a750 2023-10-16T12:49:15.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.819+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e81f4080 con 0x7f28e814a750 2023-10-16T12:49:15.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.819+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f28d0003990 con 0x7f28e814a750 2023-10-16T12:49:15.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.820+0000 7f28deffd640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f28c806af90 0x7f28c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:15.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.820+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f28d0089260 con 0x7f28e814a750 2023-10-16T12:49:15.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.821+0000 7f28e659c640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f28c806af90 0x7f28c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:15.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.822+0000 7f28e659c640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f28c806af90 0x7f28c806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f28d80099e0 tx=0x7f28d8008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:15.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:15.824+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28e81f4080 con 0x7f28e814a750 2023-10-16T12:49:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.091+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f28e81f4360 con 0x7f28e814a750 2023-10-16T12:49:16.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.150+0000 7f28deffd640 1 -- 172.21.15.60:0/186848901 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v571) v1 ==== 154+0+0 (secure 0 0 0) 0x7f28e81f4360 con 0x7f28e814a750 2023-10-16T12:49:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f28c806af90 msgr2=0x7f28c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f28c806af90 0x7f28c806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f28d80099e0 tx=0x7f28d8008040 comp rx=0 tx=0).stop 2023-10-16T12:49:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 msgr2=0x7f28e80c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 0x7f28e80c5a60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f28d0001eb0 tx=0x7f28d000e9e0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 shutdown_connections 2023-10-16T12:49:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f28c806af90 0x7f28c806d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28e815d6c0 0x7f28e815db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e814a750 0x7f28e80c5a60 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 --2- 172.21.15.60:0/186848901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28d4004d00 0x7f28e81594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 >> 172.21.15.60:0/186848901 conn(0x7f28e80cb410 msgr2=0x7f28e806f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.153+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 shutdown_connections 2023-10-16T12:49:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.154+0000 7f28ecae7640 1 -- 172.21.15.60:0/186848901 wait complete. 2023-10-16T12:49:16.175 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-16T12:49:16.175 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-16T12:49:16.176 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:49:16.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.394+0000 7f7686b1d640 1 Processor -- start 2023-10-16T12:49:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.394+0000 7f7686b1d640 1 -- start start 2023-10-16T12:49:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.396+0000 7f7686b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76800ce390 0x7f768015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.396+0000 7f7686b1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768015ed60 con 0x7f768015aaa0 2023-10-16T12:49:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.396+0000 7f7686b1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800707e0 con 0x7f768015ae80 2023-10-16T12:49:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.396+0000 7f7686b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768006b810 con 0x7f76800ce390 2023-10-16T12:49:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.396+0000 7f76848b9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f768015aaa0 0x7f76800cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38910/0 (socket says 172.21.15.60:38910) 2023-10-16T12:49:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.396+0000 7f76848b9640 1 -- 172.21.15.60:0/3757224599 learned_addr learned my addr 172.21.15.60:0/3757224599 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247641535 0 0) 0x7f768015ed60 con 0x7f768015aaa0 2023-10-16T12:49:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f76850ba640 1 --2- 172.21.15.60:0/3757224599 >> v2:172.21.15.144:3300/0 conn(0x7f76800ce390 0x7f768015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7670003520 con 0x7f768015aaa0 2023-10-16T12:49:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491017882 0 0) 0x7f7670003520 con 0x7f768015aaa0 2023-10-16T12:49:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 >> v2:172.21.15.144:3300/0 conn(0x7f76800ce390 msgr2=0x7f768015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 --2- 172.21.15.60:0/3757224599 >> v2:172.21.15.144:3300/0 conn(0x7f76800ce390 0x7f768015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 >> v1:172.21.15.144:6789/0 conn(0x7f768015ae80 legacy=0x7f7680070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768015ef90 con 0x7f768015aaa0 2023-10-16T12:49:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7674001670 con 0x7f768015aaa0 2023-10-16T12:49:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 >> v1:172.21.15.60:6789/0 conn(0x7f768015aaa0 legacy=0x7f76800cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7670003970 0x7f7670003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76700047c0 0x7f7670004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f767000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768006b810 con 0x7f7670005180 2023-10-16T12:49:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c0d0 con 0x7f76700047c0 2023-10-16T12:49:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.397+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c290 con 0x7f7670003970 2023-10-16T12:49:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76850ba640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f767000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f767ffff640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76700047c0 0x7f7670004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76848b9640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7670003970 0x7f7670003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76850ba640 1 -- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7670003970 msgr2=0x7f7670003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76850ba640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7670003970 0x7f7670003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76850ba640 1 -- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76700047c0 msgr2=0x7f7670004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76850ba640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76700047c0 0x7f7670004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.398+0000 7f76850ba640 1 -- 172.21.15.60:0/3757224599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670003520 con 0x7f7670005180 2023-10-16T12:49:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.399+0000 7f76850ba640 1 -- 172.21.15.60:0/3757224599 reap_dead start 2023-10-16T12:49:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.400+0000 7f76850ba640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f767000b980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7678007360 tx=0x7f76700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ebebb7b1d15e9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.400+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767802ccf0 con 0x7f7670005180 2023-10-16T12:49:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.400+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767802ce90 con 0x7f7670005180 2023-10-16T12:49:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.400+0000 7f767f7fe640 1 -- 172.21.15.60:0/3757224599 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678003bd0 con 0x7f7670005180 2023-10-16T12:49:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.400+0000 7f7686b1d640 1 -- 172.21.15.60:0/3757224599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 msgr2=0x7f767000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.400+0000 7f7686b1d640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f767000b980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7678007360 tx=0x7f76700057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.401+0000 7f7686b1d640 1 -- 172.21.15.60:0/3757224599 shutdown_connections 2023-10-16T12:49:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.401+0000 7f7686b1d640 1 --2- 172.21.15.60:0/3757224599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f767000b980 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.401+0000 7f7686b1d640 1 -- 172.21.15.60:0/3757224599 >> 172.21.15.60:0/3757224599 conn(0x7f76800cb410 msgr2=0x7f76800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.401+0000 7f7686b1d640 1 -- 172.21.15.60:0/3757224599 shutdown_connections 2023-10-16T12:49:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.401+0000 7f7686b1d640 1 -- 172.21.15.60:0/3757224599 wait complete. 2023-10-16T12:49:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.402+0000 7f7686b1d640 1 Processor -- start 2023-10-16T12:49:16.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.403+0000 7f7686b1d640 1 -- start start 2023-10-16T12:49:16.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.403+0000 7f7686b1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f768014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f7686b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76800c2930 0x7f76800ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f7686b1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7680150cd0 0x7f76800acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f7686b1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76800b0d10 con 0x7f7670005180 2023-10-16T12:49:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f7686b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76800c2e80 con 0x7f76800c2930 2023-10-16T12:49:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f7686b1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76800b0f60 con 0x7f7680150cd0 2023-10-16T12:49:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f768014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f768014fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47746/0 (socket says 172.21.15.60:47746) 2023-10-16T12:49:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 -- 172.21.15.60:0/853114212 learned_addr learned my addr 172.21.15.60:0/853114212 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:16.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 -- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7680150cd0 msgr2=0x7f76800acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76850ba640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7680150cd0 0x7f76800acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f767ffff640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76800c2930 0x7f76800ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7680150cd0 0x7f76800acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 -- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76800c2930 msgr2=0x7f76800ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76800c2930 0x7f76800ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.404+0000 7f76848b9640 1 -- 172.21.15.60:0/853114212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800c3120 con 0x7f7670005180 2023-10-16T12:49:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.405+0000 7f76848b9640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f768014fed0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f767400e780 tx=0x7f767400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.405+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767400c6c0 con 0x7f7670005180 2023-10-16T12:49:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.405+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767400c860 con 0x7f7670005180 2023-10-16T12:49:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.405+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767400cc10 con 0x7f7670005180 2023-10-16T12:49:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.405+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76800c3490 con 0x7f7670005180 2023-10-16T12:49:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.405+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76801506f0 con 0x7f7670005180 2023-10-16T12:49:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.406+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768004e160 con 0x7f7670005180 2023-10-16T12:49:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.407+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f767401d460 con 0x7f7670005180 2023-10-16T12:49:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.408+0000 7f767d7fa640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f766406af90 0x7f766406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.408+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7674089680 con 0x7f7670005180 2023-10-16T12:49:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.409+0000 7f767ffff640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f766406af90 0x7f766406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.410+0000 7f767ffff640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f766406af90 0x7f766406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.412+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76740545d0 con 0x7f7670005180 2023-10-16T12:49:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.603+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76800b15d0 con 0x7f7670005180 2023-10-16T12:49:16.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.605+0000 7f767d7fa640 1 -- 172.21.15.60:0/853114212 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13988 (secure 0 0 0) 0x7f7674057bf0 con 0x7f7670005180 2023-10-16T12:49:16.607 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:16.607 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":77,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:14.131078+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/528818860":"2023-10-17T12:49:11.110704+0000","172.21.15.144:6836/528818860":"2023-10-17T12:49:11.110704+0000","172.21.15.144:0/2397437661":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/2355510359":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6833/745273919":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/571939833":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6832/745273919":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/2772412730":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/3600960564":"2023-10-17T12:49:06.943439+0000","172.21.15.144:6832/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/529537373":"2023-10-17T12:49:06.943439+0000","172.21.15.144:0/788226979":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3543205521":"2023-10-17T12:47:29.992623+0000","172.21.15.144:6833/3875614959":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/411702881":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3534362968":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/2319504696":"2023-10-17T12:47:29.992623+0000","172.21.15.144:0/3965421321":"2023-10-17T12:47:29.992623+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.607+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f766406af90 msgr2=0x7f766406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.607+0000 7f7686b1d640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f766406af90 0x7f766406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c009040 comp rx=0 tx=0).stop 2023-10-16T12:49:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 msgr2=0x7f768014fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f768014fed0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f767400e780 tx=0x7f767400ec40 comp rx=0 tx=0).stop 2023-10-16T12:49:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 shutdown_connections 2023-10-16T12:49:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f766406af90 0x7f766406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7680150cd0 0x7f76800acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76800c2930 0x7f76800ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 --2- 172.21.15.60:0/853114212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7670005180 0x7f768014fed0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 >> 172.21.15.60:0/853114212 conn(0x7f76800cb410 msgr2=0x7f76801587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 shutdown_connections 2023-10-16T12:49:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.608+0000 7f7686b1d640 1 -- 172.21.15.60:0/853114212 wait complete. 2023-10-16T12:49:16.628 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:16.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.852+0000 7f59f97d5640 1 Processor -- start 2023-10-16T12:49:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.853+0000 7f59f97d5640 1 -- start start 2023-10-16T12:49:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.854+0000 7f59f97d5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59f40661c0 0x7f59f40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.854+0000 7f59f97d5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f406cee0 con 0x7f59f41618b0 2023-10-16T12:49:16.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.854+0000 7f59f97d5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40d0890 con 0x7f59f415d330 2023-10-16T12:49:16.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.854+0000 7f59f97d5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f406b1e0 con 0x7f59f40661c0 2023-10-16T12:49:16.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59f40661c0 0x7f59f40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59f40661c0 0x7f59f40ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47760/0 (socket says 172.21.15.60:47760) 2023-10-16T12:49:16.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f37fe640 1 -- 172.21.15.60:0/3450693182 learned_addr learned my addr 172.21.15.60:0/3450693182 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:16.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f27fc640 1 --1- 172.21.15.60:0/3450693182 >> v1:172.21.15.60:6789/0 conn(0x7f59f41618b0 0x7f59f4065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38914/0 (socket says 172.21.15.60:38914) 2023-10-16T12:49:16.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f37fe640 1 -- 172.21.15.60:0/3450693182 >> v1:172.21.15.144:6789/0 conn(0x7f59f415d330 legacy=0x7f59f406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f37fe640 1 -- 172.21.15.60:0/3450693182 >> v1:172.21.15.60:6789/0 conn(0x7f59f41618b0 legacy=0x7f59f4065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.855+0000 7f59f37fe640 1 -- 172.21.15.60:0/3450693182 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f40d05b0 con 0x7f59f40661c0 2023-10-16T12:49:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.857+0000 7f59f37fe640 1 --2- 172.21.15.60:0/3450693182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f40661c0 0x7f59f40ce9b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009870 tx=0x7f59ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f3d34b319c1724 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.857+0000 7f59f1ffb640 1 -- 172.21.15.60:0/3450693182 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec030860 con 0x7f59f40661c0 2023-10-16T12:49:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.857+0000 7f59f1ffb640 1 -- 172.21.15.60:0/3450693182 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec0309c0 con 0x7f59f40661c0 2023-10-16T12:49:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.857+0000 7f59f1ffb640 1 -- 172.21.15.60:0/3450693182 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec030d50 con 0x7f59f40661c0 2023-10-16T12:49:16.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.857+0000 7f59f97d5640 1 -- 172.21.15.60:0/3450693182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f40661c0 msgr2=0x7f59f40ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.857+0000 7f59f97d5640 1 --2- 172.21.15.60:0/3450693182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f40661c0 0x7f59f40ce9b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009870 tx=0x7f59ec02b9a0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.858+0000 7f59f97d5640 1 -- 172.21.15.60:0/3450693182 shutdown_connections 2023-10-16T12:49:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.858+0000 7f59f97d5640 1 --2- 172.21.15.60:0/3450693182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f40661c0 0x7f59f40ce9b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.858+0000 7f59f97d5640 1 -- 172.21.15.60:0/3450693182 >> 172.21.15.60:0/3450693182 conn(0x7f59f40d0ab0 msgr2=0x7f59f40d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.858+0000 7f59f97d5640 1 -- 172.21.15.60:0/3450693182 shutdown_connections 2023-10-16T12:49:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.858+0000 7f59f97d5640 1 -- 172.21.15.60:0/3450693182 wait complete. 2023-10-16T12:49:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.859+0000 7f59f97d5640 1 Processor -- start 2023-10-16T12:49:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.860+0000 7f59f97d5640 1 -- start start 2023-10-16T12:49:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f97d5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59f40661c0 0x7f59f40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f97d5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 0x7f59f40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f97d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59f41618b0 0x7f59f4159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f97d5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40ae990 con 0x7f59f415d330 2023-10-16T12:49:16.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f97d5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4159580 con 0x7f59f40661c0 2023-10-16T12:49:16.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f97d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40c5b20 con 0x7f59f41618b0 2023-10-16T12:49:16.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 0x7f59f40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 0x7f59f40c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47764/0 (socket says 172.21.15.60:47764) 2023-10-16T12:49:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 -- 172.21.15.60:0/2153582337 learned_addr learned my addr 172.21.15.60:0/2153582337 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f2ffd640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59f40661c0 0x7f59f40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f37fe640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59f41618b0 0x7f59f4159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 -- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59f41618b0 msgr2=0x7f59f4159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59f41618b0 0x7f59f4159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 -- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59f40661c0 msgr2=0x7f59f40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59f40661c0 0x7f59f40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f27fc640 1 -- 172.21.15.60:0/2153582337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0094d0 con 0x7f59f415d330 2023-10-16T12:49:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.861+0000 7f59f2ffd640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59f40661c0 0x7f59f40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.862+0000 7f59f27fc640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 0x7f59f40c7080 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c4d0 tx=0x7f59dc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.862+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc007480 con 0x7f59f415d330 2023-10-16T12:49:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.862+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc0075e0 con 0x7f59f415d330 2023-10-16T12:49:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.862+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc0049e0 con 0x7f59f415d330 2023-10-16T12:49:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.862+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f40c5e00 con 0x7f59f415d330 2023-10-16T12:49:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.862+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f40c63c0 con 0x7f59f415d330 2023-10-16T12:49:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.863+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f404e160 con 0x7f59f415d330 2023-10-16T12:49:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.865+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f59dc004b40 con 0x7f59f415d330 2023-10-16T12:49:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.865+0000 7f59eb7fe640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f59d406af90 0x7f59d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.866+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f59dc088860 con 0x7f59f415d330 2023-10-16T12:49:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.866+0000 7f59f2ffd640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f59d406af90 0x7f59d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.867+0000 7f59f2ffd640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f59d406af90 0x7f59d406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f59e40012e0 tx=0x7f59e4009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:16.870+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59dc053860 con 0x7f59f415d330 2023-10-16T12:49:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.070+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59f406f3a0 con 0x7f59f415d330 2023-10-16T12:49:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.071+0000 7f59eb7fe640 1 -- 172.21.15.60:0/2153582337 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59dc0180a0 con 0x7f59f415d330 2023-10-16T12:49:17.073 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:17.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f59d406af90 msgr2=0x7f59d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f59d406af90 0x7f59d406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f59e40012e0 tx=0x7f59e4009040 comp rx=0 tx=0).stop 2023-10-16T12:49:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 msgr2=0x7f59f40c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 0x7f59f40c7080 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c4d0 tx=0x7f59dc00c990 comp rx=0 tx=0).stop 2023-10-16T12:49:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 shutdown_connections 2023-10-16T12:49:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f59d406af90 0x7f59d406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59f41618b0 0x7f59f4159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59f415d330 0x7f59f40c7080 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 --2- 172.21.15.60:0/2153582337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59f40661c0 0x7f59f40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 >> 172.21.15.60:0/2153582337 conn(0x7f59f40d0ab0 msgr2=0x7f59f415a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 shutdown_connections 2023-10-16T12:49:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.074+0000 7f59f97d5640 1 -- 172.21.15.60:0/2153582337 wait complete. 2023-10-16T12:49:17.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T12:49:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.328+0000 7fe61759e640 1 Processor -- start 2023-10-16T12:49:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.328+0000 7fe61759e640 1 -- start start 2023-10-16T12:49:17.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.329+0000 7fe61759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe61815aad0 0x7fe6180cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.329+0000 7fe61759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe618070d70 0x7fe61806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.329+0000 7fe61759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6180712c0 0x7fe61806c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.329+0000 7fe61759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61806c9c0 con 0x7fe618070d70 2023-10-16T12:49:17.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.329+0000 7fe61759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6180b4620 con 0x7fe61815aad0 2023-10-16T12:49:17.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.329+0000 7fe61759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180b4340 con 0x7fe6180712c0 2023-10-16T12:49:17.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe618070d70 0x7fe61806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:17.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe618070d70 0x7fe61806bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47780/0 (socket says 172.21.15.60:47780) 2023-10-16T12:49:17.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 -- 172.21.15.60:0/2786905218 learned_addr learned my addr 172.21.15.60:0/2786905218 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:17.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe61659c640 1 --2- 172.21.15.60:0/2786905218 >> v2:172.21.15.144:3300/0 conn(0x7fe61815aad0 0x7fe6180cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:17.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe616d9d640 1 --2- 172.21.15.60:0/2786905218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6180712c0 0x7fe61806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:17.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 -- 172.21.15.60:0/2786905218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6180712c0 msgr2=0x7fe61806c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:17.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 --2- 172.21.15.60:0/2786905218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6180712c0 0x7fe61806c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 -- 172.21.15.60:0/2786905218 >> v2:172.21.15.144:3300/0 conn(0x7fe61815aad0 msgr2=0x7fe6180cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:17.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 --2- 172.21.15.60:0/2786905218 >> v2:172.21.15.144:3300/0 conn(0x7fe61815aad0 0x7fe6180cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.330+0000 7fe60ffff640 1 -- 172.21.15.60:0/2786905218 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61806f520 con 0x7fe618070d70 2023-10-16T12:49:17.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.332+0000 7fe60ffff640 1 --2- 172.21.15.60:0/2786905218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe618070d70 0x7fe61806bf40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe6180cf4f0 tx=0x7fe60002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bec38255305c669 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:17.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.332+0000 7fe615d9b640 1 -- 172.21.15.60:0/2786905218 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600030850 con 0x7fe618070d70 2023-10-16T12:49:17.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.332+0000 7fe615d9b640 1 -- 172.21.15.60:0/2786905218 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6000309b0 con 0x7fe618070d70 2023-10-16T12:49:17.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.332+0000 7fe615d9b640 1 -- 172.21.15.60:0/2786905218 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600030d40 con 0x7fe618070d70 2023-10-16T12:49:17.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 -- 172.21.15.60:0/2786905218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe618070d70 msgr2=0x7fe61806bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:17.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 --2- 172.21.15.60:0/2786905218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe618070d70 0x7fe61806bf40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe6180cf4f0 tx=0x7fe60002b990 comp rx=0 tx=0).stop 2023-10-16T12:49:17.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 -- 172.21.15.60:0/2786905218 shutdown_connections 2023-10-16T12:49:17.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 --2- 172.21.15.60:0/2786905218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6180712c0 0x7fe61806c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 --2- 172.21.15.60:0/2786905218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe618070d70 0x7fe61806bf40 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 --2- 172.21.15.60:0/2786905218 >> v2:172.21.15.144:3300/0 conn(0x7fe61815aad0 0x7fe6180cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 -- 172.21.15.60:0/2786905218 >> 172.21.15.60:0/2786905218 conn(0x7fe6180b0a90 msgr2=0x7fe6180ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:17.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 -- 172.21.15.60:0/2786905218 shutdown_connections 2023-10-16T12:49:17.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.333+0000 7fe61759e640 1 -- 172.21.15.60:0/2786905218 wait complete. 2023-10-16T12:49:17.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.335+0000 7fe61759e640 1 Processor -- start 2023-10-16T12:49:17.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.336+0000 7fe61759e640 1 -- start start 2023-10-16T12:49:17.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe61759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe618070d70 0x7fe6180c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe61759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 0x7fe6180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe61759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61815aad0 0x7fe6180acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe61759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180c8cf0 con 0x7fe6180712c0 2023-10-16T12:49:17.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe61759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180c9590 con 0x7fe618070d70 2023-10-16T12:49:17.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe61759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180c2fa0 con 0x7fe61815aad0 2023-10-16T12:49:17.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe60ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 0x7fe6180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:17.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe60ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 0x7fe6180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47794/0 (socket says 172.21.15.60:47794) 2023-10-16T12:49:17.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.337+0000 7fe60ffff640 1 -- 172.21.15.60:0/2564199758 learned_addr learned my addr 172.21.15.60:0/2564199758 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:17.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe61659c640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe618070d70 0x7fe6180c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:17.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60ffff640 1 -- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61815aad0 msgr2=0x7fe6180acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:17.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60ffff640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61815aad0 0x7fe6180acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60ffff640 1 -- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe618070d70 msgr2=0x7fe6180c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:17.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60ffff640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe618070d70 0x7fe6180c23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:17.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60ffff640 1 -- 172.21.15.60:0/2564199758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000094d0 con 0x7fe6180712c0 2023-10-16T12:49:17.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe61659c640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe618070d70 0x7fe6180c23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:17.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60ffff640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 0x7fe6180ac890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe60002b9d0 tx=0x7fe600004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:17.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600039d60 con 0x7fe6180712c0 2023-10-16T12:49:17.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600043400 con 0x7fe6180712c0 2023-10-16T12:49:17.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6000437b0 con 0x7fe6180712c0 2023-10-16T12:49:17.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6180c8f40 con 0x7fe6180712c0 2023-10-16T12:49:17.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.338+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6180c2bc0 con 0x7fe6180712c0 2023-10-16T12:49:17.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.339+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6180c9700 con 0x7fe6180712c0 2023-10-16T12:49:17.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.340+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe60002e070 con 0x7fe6180712c0 2023-10-16T12:49:17.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.341+0000 7fe60effd640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe5f806afe0 0x7fe5f806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.342+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe60003e070 con 0x7fe6180712c0 2023-10-16T12:49:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.342+0000 7fe61659c640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe5f806afe0 0x7fe5f806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.343+0000 7fe61659c640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe5f806afe0 0x7fe5f806d4a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe608006fd0 tx=0x7fe608008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.346+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6000776b0 con 0x7fe6180712c0 2023-10-16T12:49:17.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:17.609+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fe61804e9b0 con 0x7fe6180712c0 2023-10-16T12:49:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.165+0000 7fe60effd640 1 -- 172.21.15.60:0/2564199758 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v573) v1 ==== 159+0+0 (secure 0 0 0) 0x7fe60007acd0 con 0x7fe6180712c0 2023-10-16T12:49:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.168+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe5f806afe0 msgr2=0x7fe5f806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.168+0000 7fe61759e640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe5f806afe0 0x7fe5f806d4a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe608006fd0 tx=0x7fe608008040 comp rx=0 tx=0).stop 2023-10-16T12:49:18.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.168+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 msgr2=0x7fe6180ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.168+0000 7fe61759e640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 0x7fe6180ac890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe60002b9d0 tx=0x7fe600004730 comp rx=0 tx=0).stop 2023-10-16T12:49:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.168+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 shutdown_connections 2023-10-16T12:49:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.168+0000 7fe61759e640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe5f806afe0 0x7fe5f806d4a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.169+0000 7fe61759e640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61815aad0 0x7fe6180acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.169+0000 7fe61759e640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6180712c0 0x7fe6180ac890 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.169+0000 7fe61759e640 1 --2- 172.21.15.60:0/2564199758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe618070d70 0x7fe6180c23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.169+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 >> 172.21.15.60:0/2564199758 conn(0x7fe6180b0a90 msgr2=0x7fe6180ae1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.169+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 shutdown_connections 2023-10-16T12:49:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.169+0000 7fe61759e640 1 -- 172.21.15.60:0/2564199758 wait complete. 2023-10-16T12:49:18.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:49:18.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.420+0000 7facb90c2640 1 Processor -- start 2023-10-16T12:49:18.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.420+0000 7facb90c2640 1 -- start start 2023-10-16T12:49:18.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.421+0000 7facb90c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facb40ad970 0x7facb40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:18.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.421+0000 7facb90c2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb415b9d0 con 0x7facb40ccfb0 2023-10-16T12:49:18.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.421+0000 7facb90c2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb415bcb0 con 0x7facb406c040 2023-10-16T12:49:18.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.421+0000 7facb90c2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb415bf90 con 0x7facb40ad970 2023-10-16T12:49:18.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.422+0000 7facb88c1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7facb40ccfb0 0x7facb40cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38916/0 (socket says 172.21.15.60:38916) 2023-10-16T12:49:18.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.422+0000 7facb88c1640 1 -- 172.21.15.60:0/2608499346 learned_addr learned my addr 172.21.15.60:0/2608499346 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:18.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.422+0000 7facb37fe640 1 --2- 172.21.15.60:0/2608499346 >> v2:172.21.15.60:3300/0 conn(0x7facb40ad970 0x7facb40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:18.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.422+0000 7facb2ffd640 1 -- 172.21.15.60:0/2608499346 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522778104 0 0) 0x7facb415b9d0 con 0x7facb40ccfb0 2023-10-16T12:49:18.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.422+0000 7facb2ffd640 1 -- 172.21.15.60:0/2608499346 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac9c003520 con 0x7facb40ccfb0 2023-10-16T12:49:18.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.423+0000 7facb37fe640 1 -- 172.21.15.60:0/2608499346 >> v1:172.21.15.144:6789/0 conn(0x7facb406c040 legacy=0x7facb406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.423+0000 7facb37fe640 1 -- 172.21.15.60:0/2608499346 >> v1:172.21.15.60:6789/0 conn(0x7facb40ccfb0 legacy=0x7facb40cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.423+0000 7facb37fe640 1 -- 172.21.15.60:0/2608499346 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb40cc270 con 0x7facb40ad970 2023-10-16T12:49:18.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.424+0000 7facb37fe640 1 --2- 172.21.15.60:0/2608499346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40cca70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7faca8009dc0 tx=0x7faca802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc770818839042cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:18.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.424+0000 7facb2ffd640 1 -- 172.21.15.60:0/2608499346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca802cce0 con 0x7facb40ad970 2023-10-16T12:49:18.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.424+0000 7facb2ffd640 1 -- 172.21.15.60:0/2608499346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca802ce40 con 0x7facb40ad970 2023-10-16T12:49:18.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.424+0000 7facb2ffd640 1 -- 172.21.15.60:0/2608499346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca8034660 con 0x7facb40ad970 2023-10-16T12:49:18.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 -- 172.21.15.60:0/2608499346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 msgr2=0x7facb40cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 --2- 172.21.15.60:0/2608499346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40cca70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7faca8009dc0 tx=0x7faca802bf30 comp rx=0 tx=0).stop 2023-10-16T12:49:18.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 -- 172.21.15.60:0/2608499346 shutdown_connections 2023-10-16T12:49:18.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 --2- 172.21.15.60:0/2608499346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40cca70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 -- 172.21.15.60:0/2608499346 >> 172.21.15.60:0/2608499346 conn(0x7facb40ce6a0 msgr2=0x7facb40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:18.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 -- 172.21.15.60:0/2608499346 shutdown_connections 2023-10-16T12:49:18.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.425+0000 7facb90c2640 1 -- 172.21.15.60:0/2608499346 wait complete. 2023-10-16T12:49:18.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.427+0000 7facb90c2640 1 Processor -- start 2023-10-16T12:49:18.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.427+0000 7facb90c2640 1 -- start start 2023-10-16T12:49:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb90c2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facb406c040 0x7facb40b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb90c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb90c2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facb40ccfb0 0x7facb41f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb90c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40aee30 con 0x7facb40ad970 2023-10-16T12:49:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb90c2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40b3e50 con 0x7facb40ccfb0 2023-10-16T12:49:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb90c2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb41f2b30 con 0x7facb406c040 2023-10-16T12:49:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47804/0 (socket says 172.21.15.60:47804) 2023-10-16T12:49:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb37fe640 1 -- 172.21.15.60:0/237978303 learned_addr learned my addr 172.21.15.60:0/237978303 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:18.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb88c1640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facb40ccfb0 0x7facb41f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:18.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.428+0000 7facb3fff640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facb406c040 0x7facb40b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:18.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb37fe640 1 -- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facb406c040 msgr2=0x7facb40b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb37fe640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facb406c040 0x7facb40b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb37fe640 1 -- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facb40ccfb0 msgr2=0x7facb41f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb37fe640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facb40ccfb0 0x7facb41f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb37fe640 1 -- 172.21.15.60:0/237978303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8009a70 con 0x7facb40ad970 2023-10-16T12:49:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb3fff640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facb406c040 0x7facb40b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb88c1640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facb40ccfb0 0x7facb41f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:18.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb37fe640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40b3910 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7faca8037040 tx=0x7faca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:18.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca803d600 con 0x7facb40ad970 2023-10-16T12:49:18.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca803d760 con 0x7facb40ad970 2023-10-16T12:49:18.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca803db10 con 0x7facb40ad970 2023-10-16T12:49:18.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.429+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb41f2dc0 con 0x7facb40ad970 2023-10-16T12:49:18.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.430+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb41f3380 con 0x7facb40ad970 2023-10-16T12:49:18.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.431+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb40cdb10 con 0x7facb40ad970 2023-10-16T12:49:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.431+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faca803b040 con 0x7facb40ad970 2023-10-16T12:49:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.432+0000 7facb0ff9640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faca006afe0 0x7faca006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.433+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7faca80ab6a0 con 0x7facb40ad970 2023-10-16T12:49:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.433+0000 7facb3fff640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faca006afe0 0x7faca006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.434+0000 7facb3fff640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faca006afe0 0x7faca006d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7facb415e220 tx=0x7faca4009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:18.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.437+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faca8076670 con 0x7facb40ad970 2023-10-16T12:49:18.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.633+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7facb404e9b0 con 0x7facb40ad970 2023-10-16T12:49:18.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.638+0000 7facb0ff9640 1 -- 172.21.15.60:0/237978303 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7faca8079c90 con 0x7facb40ad970 2023-10-16T12:49:18.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faca006afe0 msgr2=0x7faca006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faca006afe0 0x7faca006d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7facb415e220 tx=0x7faca4009040 comp rx=0 tx=0).stop 2023-10-16T12:49:18.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 msgr2=0x7facb40b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:18.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40b3910 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7faca8037040 tx=0x7faca8004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:18.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 shutdown_connections 2023-10-16T12:49:18.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faca006afe0 0x7faca006d4a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facb40ccfb0 0x7facb41f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facb40ad970 0x7facb40b3910 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 --2- 172.21.15.60:0/237978303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facb406c040 0x7facb40b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 >> 172.21.15.60:0/237978303 conn(0x7facb40ce6a0 msgr2=0x7facb406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 shutdown_connections 2023-10-16T12:49:18.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:18.641+0000 7facb90c2640 1 -- 172.21.15.60:0/237978303 wait complete. 2023-10-16T12:49:18.663 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:49:18.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:18.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:18.921 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:49:18.921 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:18.921 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:18.985 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:49:18.986 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:18.986 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:19.045 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:19.045 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:19.045 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:19.110 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:49:19.111 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:19.111 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:19.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:19.173 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:49:19.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:19.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.410+0000 7f2c070df640 1 Processor -- start 2023-10-16T12:49:19.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.411+0000 7f2c070df640 1 -- start start 2023-10-16T12:49:19.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.412+0000 7f2c070df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c0015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.412+0000 7f2c070df640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2c0006c100 0x7f2c0006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.412+0000 7f2c070df640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0015aef0 con 0x7f2c00159950 2023-10-16T12:49:19.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.412+0000 7f2c070df640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0015e490 con 0x7f2c0006c100 2023-10-16T12:49:19.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.412+0000 7f2c070df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0015d910 con 0x7f2c00070b60 2023-10-16T12:49:19.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.413+0000 7f2c058dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c0015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.413+0000 7f2c068de640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2c0006c100 0x7f2c0006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.413+0000 7f2c060dd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f2c00159950 0x7f2c00070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:51610/0 (socket says 172.21.15.60:51610) 2023-10-16T12:49:19.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.413+0000 7f2c060dd640 1 -- 172.21.15.60:0/1624961606 learned_addr learned my addr 172.21.15.60:0/1624961606 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:19.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.414+0000 7f2c068de640 1 -- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 msgr2=0x7f2c0015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:19.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.414+0000 7f2c068de640 1 --2- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c0015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.414+0000 7f2c068de640 1 -- 172.21.15.60:0/1624961606 >> v1:172.21.15.144:6789/0 conn(0x7f2c00159950 legacy=0x7f2c00070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.414+0000 7f2c050db640 1 -- 172.21.15.60:0/1624961606 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994311654 0 0) 0x7f2bfc001110 con 0x7f2c00159950 2023-10-16T12:49:19.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.414+0000 7f2c068de640 1 -- 172.21.15.60:0/1624961606 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0015dbf0 con 0x7f2c0006c100 2023-10-16T12:49:19.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.416+0000 7f2c068de640 1 --2- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c0006c100 0x7f2c0006c550 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009d90 tx=0x7f2bf802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=195b02eee6cf81bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:19.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.416+0000 7f2c050db640 1 -- 172.21.15.60:0/1624961606 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf802cbc0 con 0x7f2c0006c100 2023-10-16T12:49:19.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.416+0000 7f2c050db640 1 -- 172.21.15.60:0/1624961606 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf802cd20 con 0x7f2c0006c100 2023-10-16T12:49:19.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.416+0000 7f2c050db640 1 -- 172.21.15.60:0/1624961606 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8034660 con 0x7f2c0006c100 2023-10-16T12:49:19.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 -- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c0006c100 msgr2=0x7f2c0006c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 --2- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c0006c100 0x7f2c0006c550 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009d90 tx=0x7f2bf802be10 comp rx=0 tx=0).stop 2023-10-16T12:49:19.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 -- 172.21.15.60:0/1624961606 shutdown_connections 2023-10-16T12:49:19.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 --2- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c0006c100 0x7f2c0006c550 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 --2- 172.21.15.60:0/1624961606 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c0015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 -- 172.21.15.60:0/1624961606 >> 172.21.15.60:0/1624961606 conn(0x7f2c000ce6a0 msgr2=0x7f2c000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:19.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 -- 172.21.15.60:0/1624961606 shutdown_connections 2023-10-16T12:49:19.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.417+0000 7f2c070df640 1 -- 172.21.15.60:0/1624961606 wait complete. 2023-10-16T12:49:19.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.419+0000 7f2c070df640 1 Processor -- start 2023-10-16T12:49:19.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.419+0000 7f2c070df640 1 -- start start 2023-10-16T12:49:19.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c070df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 0x7f2c0015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c070df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c000c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c070df640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c00159950 0x7f2c000c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c070df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c001ee070 con 0x7f2c0006c100 2023-10-16T12:49:19.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c070df640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0015d2c0 con 0x7f2c00159950 2023-10-16T12:49:19.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c070df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c000c36d0 con 0x7f2c00070b60 2023-10-16T12:49:19.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c060dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 0x7f2c0015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c060dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 0x7f2c0015cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42712/0 (socket says 172.21.15.60:42712) 2023-10-16T12:49:19.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.420+0000 7f2c060dd640 1 -- 172.21.15.60:0/1121220123 learned_addr learned my addr 172.21.15.60:0/1121220123 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:19.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c058dc640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c000c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c068de640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c00159950 0x7f2c000c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c060dd640 1 -- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 msgr2=0x7f2c000c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c060dd640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c000c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c060dd640 1 -- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c00159950 msgr2=0x7f2c000c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c060dd640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c00159950 0x7f2c000c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c060dd640 1 -- 172.21.15.60:0/1121220123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8009a70 con 0x7f2c0006c100 2023-10-16T12:49:19.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.421+0000 7f2c060dd640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 0x7f2c0015cd80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00c430 tx=0x7f2bfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.422+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc0074e0 con 0x7f2c0006c100 2023-10-16T12:49:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.422+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc007660 con 0x7f2c0006c100 2023-10-16T12:49:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.422+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc004a20 con 0x7f2c0006c100 2023-10-16T12:49:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.422+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c000c39b0 con 0x7f2c0006c100 2023-10-16T12:49:19.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.422+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c000c3f70 con 0x7f2c0006c100 2023-10-16T12:49:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.423+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0004e9b0 con 0x7f2c0006c100 2023-10-16T12:49:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.424+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2bfc004b80 con 0x7f2c0006c100 2023-10-16T12:49:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.425+0000 7f2bf6ffd640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f2be406af90 0x7f2be406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.425+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f2bfc089430 con 0x7f2c0006c100 2023-10-16T12:49:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.426+0000 7f2c058dc640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f2be406af90 0x7f2be406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.427+0000 7f2c058dc640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f2be406af90 0x7f2be406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2c0015c8a0 tx=0x7f2bf000b3a0 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:19.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.429+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2bfc054380 con 0x7f2c0006c100 2023-10-16T12:49:19.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.666+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c000b4620 con 0x7f2c0006c100 2023-10-16T12:49:19.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.667+0000 7f2bf6ffd640 1 -- 172.21.15.60:0/1121220123 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f2bfc0579a0 con 0x7f2c0006c100 2023-10-16T12:49:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:19.669 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3763459726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3763459726},{"type":"v1","addr":"172.21.15.60:6834","nonce":3763459726}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3043296902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3043296902},{"type":"v1","addr":"172.21.15.144:6835","nonce":3043296902}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1964725217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1964725217},{"type":"v1","addr":"172.21.15.144:6837","nonce":1964725217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5868,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4158055533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4158055533},{"type":"v1","addr":"172.21.15.60:6835","nonce":4158055533}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2023-10-16T12:49:19.669 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2023-10-16T12:49:19.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f2be406af90 msgr2=0x7f2be406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f2be406af90 0x7f2be406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2c0015c8a0 tx=0x7f2bf000b3a0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 msgr2=0x7f2c0015cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 0x7f2c0015cd80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00c430 tx=0x7f2bfc00c8f0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 shutdown_connections 2023-10-16T12:49:19.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f2be406af90 0x7f2be406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c00070b60 0x7f2c000c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c00159950 0x7f2c000c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 --2- 172.21.15.60:0/1121220123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c0006c100 0x7f2c0015cd80 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 >> 172.21.15.60:0/1121220123 conn(0x7f2c000ce6a0 msgr2=0x7f2c00070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:19.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 shutdown_connections 2023-10-16T12:49:19.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.670+0000 7f2c070df640 1 -- 172.21.15.60:0/1121220123 wait complete. 2023-10-16T12:49:19.692 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:49:19.692 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:49:19.693 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:49:19.693 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:19.693 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:49:19.693 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:49:19.694 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:49:19.694 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:19.694 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:49:19.695 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:49:19.695 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:49:19.695 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:19.695 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:49:19.696 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:49:19.696 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:49:19.696 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:19.711 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:49:19.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:49:19.715 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:49:19.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:49:19.716 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:49:19.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:49:19.718 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:49:19.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:49:19.765 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:49:19.766 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:19.767 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:49:19.767 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:19.773 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:49:19.773 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:19.774 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:49:19.775 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:19.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:49:19.928 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:49:19.931 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:49:19.934 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:49:19.935 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:49:19.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.973+0000 7ff524541640 1 Processor -- start 2023-10-16T12:49:19.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.973+0000 7ff524541640 1 -- start start 2023-10-16T12:49:19.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff524541640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff51c159ab0 0x7ff51c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff524541640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c06c140 0x7ff51c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff524541640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c070360 con 0x7ff51c159e90 2023-10-16T12:49:19.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff524541640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51c06cb10 con 0x7ff51c159ab0 2023-10-16T12:49:19.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff524541640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c15d890 con 0x7ff51c06c140 2023-10-16T12:49:19.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff5222dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff51c159ab0 0x7ff51c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff521adc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff51c159e90 0x7ff51c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45484/0 (socket says 172.21.15.60:45484) 2023-10-16T12:49:19.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.974+0000 7ff521adc640 1 -- 172.21.15.60:0/1395811915 learned_addr learned my addr 172.21.15.60:0/1395811915 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:19.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.975+0000 7ff522ade640 1 --2- 172.21.15.60:0/1395811915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c06c140 0x7ff51c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.975+0000 7ff5222dd640 1 -- 172.21.15.60:0/1395811915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c06c140 msgr2=0x7ff51c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:19.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.975+0000 7ff5222dd640 1 --2- 172.21.15.60:0/1395811915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c06c140 0x7ff51c06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.975+0000 7ff5212db640 1 -- 172.21.15.60:0/1395811915 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880676093 0 0) 0x7ff51c070360 con 0x7ff51c159e90 2023-10-16T12:49:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.975+0000 7ff5222dd640 1 -- 172.21.15.60:0/1395811915 >> v1:172.21.15.60:6789/0 conn(0x7ff51c159e90 legacy=0x7ff51c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.975+0000 7ff5222dd640 1 -- 172.21.15.60:0/1395811915 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c0cd1f0 con 0x7ff51c159ab0 2023-10-16T12:49:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.976+0000 7ff5222dd640 1 --2- 172.21.15.60:0/1395811915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c159ab0 0x7ff51c0ce370 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7ff5180097a0 tx=0x7ff51802d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a545726a4fa30e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.976+0000 7ff5212db640 1 -- 172.21.15.60:0/1395811915 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518032860 con 0x7ff51c159ab0 2023-10-16T12:49:19.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.976+0000 7ff5212db640 1 -- 172.21.15.60:0/1395811915 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5180329c0 con 0x7ff51c159ab0 2023-10-16T12:49:19.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.976+0000 7ff5212db640 1 -- 172.21.15.60:0/1395811915 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518032d50 con 0x7ff51c159ab0 2023-10-16T12:49:19.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 -- 172.21.15.60:0/1395811915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c159ab0 msgr2=0x7ff51c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 --2- 172.21.15.60:0/1395811915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c159ab0 0x7ff51c0ce370 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7ff5180097a0 tx=0x7ff51802d9a0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 -- 172.21.15.60:0/1395811915 shutdown_connections 2023-10-16T12:49:19.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 --2- 172.21.15.60:0/1395811915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c06c140 0x7ff51c06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 --2- 172.21.15.60:0/1395811915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c159ab0 0x7ff51c0ce370 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 -- 172.21.15.60:0/1395811915 >> 172.21.15.60:0/1395811915 conn(0x7ff51c0ce9b0 msgr2=0x7ff51c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:19.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 -- 172.21.15.60:0/1395811915 shutdown_connections 2023-10-16T12:49:19.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.977+0000 7ff524541640 1 -- 172.21.15.60:0/1395811915 wait complete. 2023-10-16T12:49:19.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.979+0000 7ff524541640 1 Processor -- start 2023-10-16T12:49:19.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.979+0000 7ff524541640 1 -- start start 2023-10-16T12:49:19.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff524541640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 0x7ff51c0c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff524541640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c159ab0 0x7ff51c15c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff524541640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff51c159e90 0x7ff51c0c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff524541640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0c2de0 con 0x7ff51c06c140 2023-10-16T12:49:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff524541640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0c30c0 con 0x7ff51c159e90 2023-10-16T12:49:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff524541640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0c33a0 con 0x7ff51c159ab0 2023-10-16T12:49:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff5222dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 0x7ff51c0c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff5222dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 0x7ff51c0c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42736/0 (socket says 172.21.15.60:42736) 2023-10-16T12:49:19.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.980+0000 7ff5222dd640 1 -- 172.21.15.60:0/2675830540 learned_addr learned my addr 172.21.15.60:0/2675830540 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:19.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff521adc640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c159ab0 0x7ff51c15c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff522ade640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff51c159e90 0x7ff51c0c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff5222dd640 1 -- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c159ab0 msgr2=0x7ff51c15c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff5222dd640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c159ab0 0x7ff51c15c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff5222dd640 1 -- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff51c159e90 msgr2=0x7ff51c0c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:19.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff5222dd640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff51c159e90 0x7ff51c0c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:19.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff5222dd640 1 -- 172.21.15.60:0/2675830540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518009400 con 0x7ff51c06c140 2023-10-16T12:49:19.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff522ade640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff51c159e90 0x7ff51c0c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:19.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff5222dd640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 0x7ff51c0c24e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff51802dfb0 tx=0x7ff5180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:19.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.981+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51803ba90 con 0x7ff51c06c140 2023-10-16T12:49:19.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.982+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51803bc10 con 0x7ff51c06c140 2023-10-16T12:49:19.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.982+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518046450 con 0x7ff51c06c140 2023-10-16T12:49:19.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.982+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c0c3680 con 0x7ff51c06c140 2023-10-16T12:49:19.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.982+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c0c2a20 con 0x7ff51c06c140 2023-10-16T12:49:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.983+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c04e160 con 0x7ff51c06c140 2023-10-16T12:49:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.984+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff5180313b0 con 0x7ff51c06c140 2023-10-16T12:49:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.985+0000 7ff512ffd640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff50006af90 0x7ff50006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.985+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ff5180ac080 con 0x7ff51c06c140 2023-10-16T12:49:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.986+0000 7ff521adc640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff50006af90 0x7ff50006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.986+0000 7ff521adc640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff50006af90 0x7ff50006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff51c157670 tx=0x7ff50c007b50 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:19.990+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5180defe0 con 0x7ff51c06c140 2023-10-16T12:49:20.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:20.199+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff51c0adf10 con 0x7ff51c06c140 2023-10-16T12:49:21.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.677+0000 7ff512ffd640 1 -- 172.21.15.60:0/2675830540 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v82) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff518076f50 con 0x7ff51c06c140 2023-10-16T12:49:21.680 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:49:21.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.680+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff50006af90 msgr2=0x7ff50006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:21.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.680+0000 7ff524541640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff50006af90 0x7ff50006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff51c157670 tx=0x7ff50c007b50 comp rx=0 tx=0).stop 2023-10-16T12:49:21.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 msgr2=0x7ff51c0c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:21.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 0x7ff51c0c24e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff51802dfb0 tx=0x7ff5180046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 shutdown_connections 2023-10-16T12:49:21.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ff50006af90 0x7ff50006d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff51c159ab0 0x7ff51c15c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff51c159e90 0x7ff51c0c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 --2- 172.21.15.60:0/2675830540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff51c06c140 0x7ff51c0c24e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 >> 172.21.15.60:0/2675830540 conn(0x7ff51c0ce9b0 msgr2=0x7ff51c158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 shutdown_connections 2023-10-16T12:49:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.681+0000 7ff524541640 1 -- 172.21.15.60:0/2675830540 wait complete. 2023-10-16T12:49:21.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:49:21.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.932+0000 7f8423a75640 1 Processor -- start 2023-10-16T12:49:21.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.932+0000 7f8423a75640 1 -- start start 2023-10-16T12:49:21.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.933+0000 7f8423a75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15aaa0 0x7f841c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.933+0000 7f8423a75640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f841c0ce390 0x7f841c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.933+0000 7f8423a75640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841c15ed60 con 0x7f841c15ae80 2023-10-16T12:49:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.933+0000 7f8423a75640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841c06b810 con 0x7f841c0ce390 2023-10-16T12:49:21.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.933+0000 7f8423a75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c0707e0 con 0x7f841c15aaa0 2023-10-16T12:49:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.933+0000 7f8422012640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f841c0ce390 0x7f841c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:21.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8422012640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f841c0ce390 0x7f841c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42750/0 (socket says 172.21.15.60:42750) 2023-10-16T12:49:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8421010640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f841c15ae80 0x7f841c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45490/0 (socket says 172.21.15.60:45490) 2023-10-16T12:49:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8422012640 1 -- 172.21.15.60:0/1790878948 learned_addr learned my addr 172.21.15.60:0/1790878948 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8421811640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15aaa0 0x7f841c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8422012640 1 -- 172.21.15.60:0/1790878948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15aaa0 msgr2=0x7f841c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:21.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8422012640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15aaa0 0x7f841c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8422012640 1 -- 172.21.15.60:0/1790878948 >> v1:172.21.15.60:6789/0 conn(0x7f841c15ae80 legacy=0x7f841c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:21.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8422012640 1 -- 172.21.15.60:0/1790878948 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c15ef90 con 0x7f841c0ce390 2023-10-16T12:49:21.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.934+0000 7f8421811640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15aaa0 0x7f841c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.935+0000 7f8422012640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c070f50 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8414009e50 tx=0x7f841402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab29afdbcf134df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.936+0000 7f842080f640 1 -- 172.21.15.60:0/1790878948 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841402cb00 con 0x7f841c0ce390 2023-10-16T12:49:21.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.936+0000 7f842080f640 1 -- 172.21.15.60:0/1790878948 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841402cc60 con 0x7f841c0ce390 2023-10-16T12:49:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.936+0000 7f842080f640 1 -- 172.21.15.60:0/1790878948 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84140343f0 con 0x7f841c0ce390 2023-10-16T12:49:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.936+0000 7f8423a75640 1 -- 172.21.15.60:0/1790878948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 msgr2=0x7f841c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.936+0000 7f8423a75640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c070f50 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8414009e50 tx=0x7f841402bf60 comp rx=0 tx=0).stop 2023-10-16T12:49:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.937+0000 7f8423a75640 1 -- 172.21.15.60:0/1790878948 shutdown_connections 2023-10-16T12:49:21.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.937+0000 7f8423a75640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c070f50 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.937+0000 7f8423a75640 1 --2- 172.21.15.60:0/1790878948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15aaa0 0x7f841c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.937+0000 7f8423a75640 1 -- 172.21.15.60:0/1790878948 >> 172.21.15.60:0/1790878948 conn(0x7f841c0cb410 msgr2=0x7f841c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.937+0000 7f8423a75640 1 -- 172.21.15.60:0/1790878948 shutdown_connections 2023-10-16T12:49:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.937+0000 7f8423a75640 1 -- 172.21.15.60:0/1790878948 wait complete. 2023-10-16T12:49:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.938+0000 7f8423a75640 1 Processor -- start 2023-10-16T12:49:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.939+0000 7f8423a75640 1 -- start start 2023-10-16T12:49:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8423a75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8423a75640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f841c15aaa0 0x7f841c0c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8423a75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15ae80 0x7f841c0c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8423a75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c15d350 con 0x7f841c0ce390 2023-10-16T12:49:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8423a75640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c1f3260 con 0x7f841c15aaa0 2023-10-16T12:49:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8423a75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c15c9b0 con 0x7f841c15ae80 2023-10-16T12:49:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42766/0 (socket says 172.21.15.60:42766) 2023-10-16T12:49:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 -- 172.21.15.60:0/4208614700 learned_addr learned my addr 172.21.15.60:0/4208614700 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 -- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15ae80 msgr2=0x7f841c0c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15ae80 0x7f841c0c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 -- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f841c15aaa0 msgr2=0x7f841c0c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f841c15aaa0 0x7f841c0c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.940+0000 7f8421811640 1 -- 172.21.15.60:0/4208614700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8414009b30 con 0x7f841c0ce390 2023-10-16T12:49:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.941+0000 7f8421811640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c159a70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f841800e7e0 tx=0x7f841800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.941+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418004030 con 0x7f841c0ce390 2023-10-16T12:49:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.941+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8418004190 con 0x7f841c0ce390 2023-10-16T12:49:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.941+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c15cc90 con 0x7f841c0ce390 2023-10-16T12:49:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.941+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418004520 con 0x7f841c0ce390 2023-10-16T12:49:21.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.941+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c0c3a30 con 0x7f841c0ce390 2023-10-16T12:49:21.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.942+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c159fb0 con 0x7f841c0ce390 2023-10-16T12:49:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.943+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f841800c6d0 con 0x7f841c0ce390 2023-10-16T12:49:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.944+0000 7f84127fc640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f840006af90 0x7f840006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.944+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8418089200 con 0x7f841c0ce390 2023-10-16T12:49:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.945+0000 7f8421010640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f840006af90 0x7f840006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:21.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.946+0000 7f8421010640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f840006af90 0x7f840006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f841c0c5350 tx=0x7f840c0079c0 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:21.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:21.949+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84180543d0 con 0x7f841c0ce390 2023-10-16T12:49:22.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.214+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f841c04e160 con 0x7f841c0ce390 2023-10-16T12:49:22.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.215+0000 7f84127fc640 1 -- 172.21.15.60:0/4208614700 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f84180579f0 con 0x7f841c0ce390 2023-10-16T12:49:22.218 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:22.218 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:49:22.218 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:49:22.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.219 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:49:22.219 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:49:22.219 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:49:22.220 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:49:22.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.221 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:49:22.221 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:49:22.221 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.221 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.221 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:49:22.222 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:49:22.222 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.222 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.223 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:49:22.223 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:49:22.223 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.223 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.223 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:49:22.224 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:49:22.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.225 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:49:22.225 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:49:22.225 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.225 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.225 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:49:22.225 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:49:22.226 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.226 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.226 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:49:22.227 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:49:22.227 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.227 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.228 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:49:22.228 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:49:22.229 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.229 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.229 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.229 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.229 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.230 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:49:22.230 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:49:22.230 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.230 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.231 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.231 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.231 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:49:22.234 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.234 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.234 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.234 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.235 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:49:22.235 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:49:22.236 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.236 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.236 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.236 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.236 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:49:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:49:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.238 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.238 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.238 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.239 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:49:22.239 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:49:22.239 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.239 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:49:22.239 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:49:22.240 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.240 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.240 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:49:22.241 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:49:22.241 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.241 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.241 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:49:22.241 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:49:22.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.242 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:49:22.242 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:49:22.242 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.243 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.243 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:49:22.243 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:49:22.243 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:49:22.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.244 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:49:22.244 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:49:22.244 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.245 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.245 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:49:22.245 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:49:22.245 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:49:22.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.246 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:49:22.246 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:49:22.246 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:49:22.247 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:49:22.247 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:49:22.247 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:49:22.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:49:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:49:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:49:22.249 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:49:22.249 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:49:22.249 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.250 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:49:22.250 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:49:22.250 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:49:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:49:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:49:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:49:22.252 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:49:22.252 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:49:22.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:49:22.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:49:22.252 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:49:22.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:22.253 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:49:22.253 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:49:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.217+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f840006af90 msgr2=0x7f840006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.217+0000 7f8423a75640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f840006af90 0x7f840006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f841c0c5350 tx=0x7f840c0079c0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.217+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 msgr2=0x7f841c159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.217+0000 7f8423a75640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c159a70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f841800e7e0 tx=0x7f841800eca0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 shutdown_connections 2023-10-16T12:49:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f840006af90 0x7f840006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f841c15ae80 0x7f841c0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f841c15aaa0 0x7f841c0c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 --2- 172.21.15.60:0/4208614700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f841c0ce390 0x7f841c159a70 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 >> 172.21.15.60:0/4208614700 conn(0x7f841c0cb410 msgr2=0x7f841c0b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 shutdown_connections 2023-10-16T12:49:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.218+0000 7f8423a75640 1 -- 172.21.15.60:0/4208614700 wait complete. 2023-10-16T12:49:22.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.477+0000 7f543ffff640 1 Processor -- start 2023-10-16T12:49:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.478+0000 7f543ffff640 1 -- start start 2023-10-16T12:49:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.479+0000 7f543ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f544015cbe0 0x7f54400d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.479+0000 7f543ffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54400d15a0 con 0x7f544015cfc0 2023-10-16T12:49:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.479+0000 7f543ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54400d1880 con 0x7f54400b4c80 2023-10-16T12:49:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.479+0000 7f543ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54400d1b60 con 0x7f544015cbe0 2023-10-16T12:49:22.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f544015cbe0 0x7f54400d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f544015cbe0 0x7f54400d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42774/0 (socket says 172.21.15.60:42774) 2023-10-16T12:49:22.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543effd640 1 -- 172.21.15.60:0/3156080699 learned_addr learned my addr 172.21.15.60:0/3156080699 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:22.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543e7fc640 1 --1- 172.21.15.60:0/3156080699 >> v1:172.21.15.60:6789/0 conn(0x7f544015cfc0 0x7f54400b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45492/0 (socket says 172.21.15.60:45492) 2023-10-16T12:49:22.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543effd640 1 -- 172.21.15.60:0/3156080699 >> v1:172.21.15.144:6789/0 conn(0x7f54400b4c80 legacy=0x7f54400b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:22.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543effd640 1 -- 172.21.15.60:0/3156080699 >> v1:172.21.15.60:6789/0 conn(0x7f544015cfc0 legacy=0x7f54400b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.480+0000 7f543effd640 1 -- 172.21.15.60:0/3156080699 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400d1e40 con 0x7f544015cbe0 2023-10-16T12:49:22.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.481+0000 7f543effd640 1 --2- 172.21.15.60:0/3156080699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400d0b50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5430009870 tx=0x7f543002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c8f37074a9ac8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:22.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.482+0000 7f543dffb640 1 -- 172.21.15.60:0/3156080699 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5430030850 con 0x7f544015cbe0 2023-10-16T12:49:22.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.482+0000 7f543dffb640 1 -- 172.21.15.60:0/3156080699 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54300309b0 con 0x7f544015cbe0 2023-10-16T12:49:22.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.482+0000 7f543dffb640 1 -- 172.21.15.60:0/3156080699 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5430030d40 con 0x7f544015cbe0 2023-10-16T12:49:22.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.482+0000 7f543ffff640 1 -- 172.21.15.60:0/3156080699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 msgr2=0x7f54400d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.482+0000 7f543ffff640 1 --2- 172.21.15.60:0/3156080699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400d0b50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5430009870 tx=0x7f543002b990 comp rx=0 tx=0).stop 2023-10-16T12:49:22.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.483+0000 7f543ffff640 1 -- 172.21.15.60:0/3156080699 shutdown_connections 2023-10-16T12:49:22.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.483+0000 7f543ffff640 1 --2- 172.21.15.60:0/3156080699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400d0b50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.483+0000 7f543ffff640 1 -- 172.21.15.60:0/3156080699 >> 172.21.15.60:0/3156080699 conn(0x7f54400cf570 msgr2=0x7f54400cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:22.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.483+0000 7f543ffff640 1 -- 172.21.15.60:0/3156080699 shutdown_connections 2023-10-16T12:49:22.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.483+0000 7f543ffff640 1 -- 172.21.15.60:0/3156080699 wait complete. 2023-10-16T12:49:22.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.484+0000 7f543ffff640 1 Processor -- start 2023-10-16T12:49:22.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.485+0000 7f543ffff640 1 -- start start 2023-10-16T12:49:22.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54400b4c80 0x7f5440153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f544015cfc0 0x7f54400c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400cba90 con 0x7f544015cbe0 2023-10-16T12:49:22.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400cae80 con 0x7f54400b4c80 2023-10-16T12:49:22.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400cb160 con 0x7f544015cfc0 2023-10-16T12:49:22.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42778/0 (socket says 172.21.15.60:42778) 2023-10-16T12:49:22.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 -- 172.21.15.60:0/2999128652 learned_addr learned my addr 172.21.15.60:0/2999128652 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:22.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543effd640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54400b4c80 0x7f5440153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543f7fe640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f544015cfc0 0x7f54400c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 -- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f544015cfc0 msgr2=0x7f54400c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f544015cfc0 0x7f54400c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 -- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54400b4c80 msgr2=0x7f5440153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54400b4c80 0x7f5440153920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.486+0000 7f543e7fc640 1 -- 172.21.15.60:0/2999128652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54300094d0 con 0x7f544015cbe0 2023-10-16T12:49:22.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.487+0000 7f543e7fc640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400c6620 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f543400c3c0 tx=0x7f543400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:22.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.487+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340074e0 con 0x7f544015cbe0 2023-10-16T12:49:22.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.487+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434007640 con 0x7f544015cbe0 2023-10-16T12:49:22.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.487+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434004a00 con 0x7f544015cbe0 2023-10-16T12:49:22.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.487+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54400cb440 con 0x7f544015cbe0 2023-10-16T12:49:22.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.487+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54400c8a80 con 0x7f544015cbe0 2023-10-16T12:49:22.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.488+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54400cbc40 con 0x7f544015cbe0 2023-10-16T12:49:22.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.490+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5434004b60 con 0x7f544015cbe0 2023-10-16T12:49:22.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.491+0000 7f54277fe640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f542c06af90 0x7f542c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.491+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5434088dd0 con 0x7f544015cbe0 2023-10-16T12:49:22.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.491+0000 7f543effd640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f542c06af90 0x7f542c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.492+0000 7f543effd640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f542c06af90 0x7f542c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5430036000 tx=0x7f543003b040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:22.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.495+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54400cbc40 con 0x7f544015cbe0 2023-10-16T12:49:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.695+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f544004e9b0 con 0x7f544015cbe0 2023-10-16T12:49:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.697+0000 7f54277fe640 1 -- 172.21.15.60:0/2999128652 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+12845 (secure 0 0 0) 0x7f543408a090 con 0x7f544015cbe0 2023-10-16T12:49:22.700 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:22.700 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":82,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:21.674636+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f542c06af90 msgr2=0x7f542c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f542c06af90 0x7f542c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5430036000 tx=0x7f543003b040 comp rx=0 tx=0).stop 2023-10-16T12:49:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 msgr2=0x7f54400c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400c6620 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f543400c3c0 tx=0x7f543400c880 comp rx=0 tx=0).stop 2023-10-16T12:49:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 shutdown_connections 2023-10-16T12:49:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f542c06af90 0x7f542c06d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f544015cfc0 0x7f54400c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544015cbe0 0x7f54400c6620 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 --2- 172.21.15.60:0/2999128652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54400b4c80 0x7f5440153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 >> 172.21.15.60:0/2999128652 conn(0x7f54400cf570 msgr2=0x7f544015c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 shutdown_connections 2023-10-16T12:49:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.700+0000 7f543ffff640 1 -- 172.21.15.60:0/2999128652 wait complete. 2023-10-16T12:49:22.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:22.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:22.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.955+0000 7f6ab3ff5640 1 Processor -- start 2023-10-16T12:49:22.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.955+0000 7f6ab3ff5640 1 -- start start 2023-10-16T12:49:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab3ff5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6aac06a4b0 0x7f6aac0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab3ff5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac0c9ca0 0x7f6aac0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab3ff5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac0cb030 con 0x7f6aac15ad70 2023-10-16T12:49:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab3ff5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aac0cae30 con 0x7f6aac06a4b0 2023-10-16T12:49:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab3ff5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0c91c0 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab1590640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6aac06a4b0 0x7f6aac0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab1590640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6aac06a4b0 0x7f6aac0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42780/0 (socket says 172.21.15.60:42780) 2023-10-16T12:49:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.957+0000 7f6ab1590640 1 -- 172.21.15.60:0/2619812267 learned_addr learned my addr 172.21.15.60:0/2619812267 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.958+0000 7f6ab2592640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac0c9ca0 0x7f6aac0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.958+0000 7f6ab1590640 1 -- 172.21.15.60:0/2619812267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac0c9ca0 msgr2=0x7f6aac0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.958+0000 7f6ab1590640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac0c9ca0 0x7f6aac0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.958+0000 7f6ab1590640 1 -- 172.21.15.60:0/2619812267 >> v1:172.21.15.60:6789/0 conn(0x7f6aac15ad70 legacy=0x7f6aac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.958+0000 7f6ab1590640 1 -- 172.21.15.60:0/2619812267 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac0c94a0 con 0x7f6aac06a4b0 2023-10-16T12:49:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.958+0000 7f6ab2592640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac0c9ca0 0x7f6aac0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.959+0000 7f6ab1590640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac06a4b0 0x7f6aac0ca3b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009870 tx=0x7f6a9c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26a1a7a3e3ac90d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.959+0000 7f6ab0d8f640 1 -- 172.21.15.60:0/2619812267 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c030840 con 0x7f6aac06a4b0 2023-10-16T12:49:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.959+0000 7f6ab0d8f640 1 -- 172.21.15.60:0/2619812267 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c0309a0 con 0x7f6aac06a4b0 2023-10-16T12:49:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.959+0000 7f6ab0d8f640 1 -- 172.21.15.60:0/2619812267 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c030d30 con 0x7f6aac06a4b0 2023-10-16T12:49:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/2619812267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac06a4b0 msgr2=0x7f6aac0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac06a4b0 0x7f6aac0ca3b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009870 tx=0x7f6a9c02b980 comp rx=0 tx=0).stop 2023-10-16T12:49:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/2619812267 shutdown_connections 2023-10-16T12:49:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac0c9ca0 0x7f6aac0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/2619812267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac06a4b0 0x7f6aac0ca3b0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/2619812267 >> 172.21.15.60:0/2619812267 conn(0x7f6aac0ce9b0 msgr2=0x7f6aac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/2619812267 shutdown_connections 2023-10-16T12:49:22.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.960+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/2619812267 wait complete. 2023-10-16T12:49:22.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.962+0000 7f6ab3ff5640 1 Processor -- start 2023-10-16T12:49:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.962+0000 7f6ab3ff5640 1 -- start start 2023-10-16T12:49:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab3ff5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac06a4b0 0x7f6aac1564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab3ff5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 0x7f6aac0c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab3ff5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6aac15ad70 0x7f6aac156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab3ff5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0adf90 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab3ff5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0c79f0 con 0x7f6aac15ad70 2023-10-16T12:49:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab3ff5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0ad9d0 con 0x7f6aac06a4b0 2023-10-16T12:49:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.963+0000 7f6ab1590640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 0x7f6aac0c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 0x7f6aac0c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42788/0 (socket says 172.21.15.60:42788) 2023-10-16T12:49:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 -- 172.21.15.60:0/1644895939 learned_addr learned my addr 172.21.15.60:0/1644895939 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1d91640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac06a4b0 0x7f6aac1564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab2592640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6aac15ad70 0x7f6aac156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 -- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac06a4b0 msgr2=0x7f6aac1564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac06a4b0 0x7f6aac1564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 -- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6aac15ad70 msgr2=0x7f6aac156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6aac15ad70 0x7f6aac156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 -- 172.21.15.60:0/1644895939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c0094d0 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.964+0000 7f6ab1590640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 0x7f6aac0c7e80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c02b9c0 tx=0x7f6a9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.965+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c030810 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.965+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c044480 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.965+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c044850 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.965+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aac0adcb0 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.965+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aac1569e0 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.966+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aac0c74e0 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.967+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a9c02f390 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.968+0000 7f6aa27fc640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6a9006af90 0x7f6a9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:22.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.968+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a9c03e070 con 0x7f6aac0c9ca0 2023-10-16T12:49:22.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.969+0000 7f6ab1d91640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6a9006af90 0x7f6a9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:22.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.970+0000 7f6ab1d91640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6a9006af90 0x7f6a9006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0c8820 tx=0x7f6aa8009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:22.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:22.972+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6aac0c74e0 con 0x7f6aac0c9ca0 2023-10-16T12:49:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.170+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6aac0c74e0 con 0x7f6aac0c9ca0 2023-10-16T12:49:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.170+0000 7f6aa27fc640 1 -- 172.21.15.60:0/1644895939 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6aac0c74e0 con 0x7f6aac0c9ca0 2023-10-16T12:49:23.173 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6a9006af90 msgr2=0x7f6a9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6a9006af90 0x7f6a9006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0c8820 tx=0x7f6aa8009040 comp rx=0 tx=0).stop 2023-10-16T12:49:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 msgr2=0x7f6aac0c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 0x7f6aac0c7e80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c02b9c0 tx=0x7f6a9c004730 comp rx=0 tx=0).stop 2023-10-16T12:49:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 shutdown_connections 2023-10-16T12:49:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6a9006af90 0x7f6a9006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6aac15ad70 0x7f6aac156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6aac0c9ca0 0x7f6aac0c7e80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 --2- 172.21.15.60:0/1644895939 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6aac06a4b0 0x7f6aac1564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.173+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 >> 172.21.15.60:0/1644895939 conn(0x7f6aac0ce9b0 msgr2=0x7f6aac0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.174+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 shutdown_connections 2023-10-16T12:49:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.174+0000 7f6ab3ff5640 1 -- 172.21.15.60:0/1644895939 wait complete. 2023-10-16T12:49:23.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:49:23.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.424+0000 7fde97fa9640 1 Processor -- start 2023-10-16T12:49:23.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.425+0000 7fde97fa9640 1 -- start start 2023-10-16T12:49:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.426+0000 7fde97fa9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fde90159f00 0x7fde900b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.426+0000 7fde97fa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde900b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.426+0000 7fde97fa9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9006f7f0 con 0x7fde90159b20 2023-10-16T12:49:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.426+0000 7fde97fa9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9006c5e0 con 0x7fde90159f00 2023-10-16T12:49:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.426+0000 7fde97fa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9006c8c0 con 0x7fde9015d880 2023-10-16T12:49:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95d45640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fde90159b20 0x7fde9006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45508/0 (socket says 172.21.15.60:45508) 2023-10-16T12:49:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95d45640 1 -- 172.21.15.60:0/3476569084 learned_addr learned my addr 172.21.15.60:0/3476569084 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:23.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95544640 1 --2- 172.21.15.60:0/3476569084 >> v2:172.21.15.60:3300/0 conn(0x7fde90159f00 0x7fde900b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde96546640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde900b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95544640 1 -- 172.21.15.60:0/3476569084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 msgr2=0x7fde900b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:23.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95544640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde900b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95544640 1 -- 172.21.15.60:0/3476569084 >> v1:172.21.15.60:6789/0 conn(0x7fde90159b20 legacy=0x7fde9006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde95544640 1 -- 172.21.15.60:0/3476569084 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9006cba0 con 0x7fde90159f00 2023-10-16T12:49:23.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.427+0000 7fde96546640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde900b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:23.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.428+0000 7fde95544640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900b0410 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fde80009d90 tx=0x7fde8002e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726723dad92b2be6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:23.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.429+0000 7fde94d43640 1 -- 172.21.15.60:0/3476569084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8002ee60 con 0x7fde90159f00 2023-10-16T12:49:23.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.429+0000 7fde94d43640 1 -- 172.21.15.60:0/3476569084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80036440 con 0x7fde90159f00 2023-10-16T12:49:23.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.429+0000 7fde94d43640 1 -- 172.21.15.60:0/3476569084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde800367d0 con 0x7fde90159f00 2023-10-16T12:49:23.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.429+0000 7fde97fa9640 1 -- 172.21.15.60:0/3476569084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 msgr2=0x7fde900b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.429+0000 7fde97fa9640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900b0410 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fde80009d90 tx=0x7fde8002e0b0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.430+0000 7fde97fa9640 1 -- 172.21.15.60:0/3476569084 shutdown_connections 2023-10-16T12:49:23.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.430+0000 7fde97fa9640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde900b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.430+0000 7fde97fa9640 1 --2- 172.21.15.60:0/3476569084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900b0410 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.430+0000 7fde97fa9640 1 -- 172.21.15.60:0/3476569084 >> 172.21.15.60:0/3476569084 conn(0x7fde900cb410 msgr2=0x7fde900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:23.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.430+0000 7fde97fa9640 1 -- 172.21.15.60:0/3476569084 shutdown_connections 2023-10-16T12:49:23.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.430+0000 7fde97fa9640 1 -- 172.21.15.60:0/3476569084 wait complete. 2023-10-16T12:49:23.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.432+0000 7fde97fa9640 1 Processor -- start 2023-10-16T12:49:23.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.432+0000 7fde97fa9640 1 -- start start 2023-10-16T12:49:23.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde97fa9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde90159b20 0x7fde901f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde97fa9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde97fa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde90151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde97fa9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde901f7600 con 0x7fde90159f00 2023-10-16T12:49:23.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde97fa9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde901f6d60 con 0x7fde90159b20 2023-10-16T12:49:23.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde97fa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde901f7040 con 0x7fde9015d880 2023-10-16T12:49:23.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde95544640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde95544640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42810/0 (socket says 172.21.15.60:42810) 2023-10-16T12:49:23.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.433+0000 7fde95544640 1 -- 172.21.15.60:0/1202320872 learned_addr learned my addr 172.21.15.60:0/1202320872 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:23.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde96546640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde90151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95d45640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde90159b20 0x7fde901f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95544640 1 -- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 msgr2=0x7fde90151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95544640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde90151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95544640 1 -- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde90159b20 msgr2=0x7fde901f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95544640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde90159b20 0x7fde901f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95544640 1 -- 172.21.15.60:0/1202320872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde80009a70 con 0x7fde90159f00 2023-10-16T12:49:23.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde95544640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900c77b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fde80009d90 tx=0x7fde80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:23.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8003f850 con 0x7fde90159f00 2023-10-16T12:49:23.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8003f9f0 con 0x7fde90159f00 2023-10-16T12:49:23.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8003fda0 con 0x7fde90159f00 2023-10-16T12:49:23.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.434+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde901f7320 con 0x7fde90159f00 2023-10-16T12:49:23.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.435+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde90151fe0 con 0x7fde90159f00 2023-10-16T12:49:23.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.436+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde90152170 con 0x7fde90159f00 2023-10-16T12:49:23.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.437+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fde8003d070 con 0x7fde90159f00 2023-10-16T12:49:23.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.438+0000 7fde867fc640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fde7406af90 0x7fde7406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.438+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fde800abe10 con 0x7fde90159f00 2023-10-16T12:49:23.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.438+0000 7fde95d45640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fde7406af90 0x7fde7406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.439+0000 7fde95d45640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fde7406af90 0x7fde7406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde8c005880 tx=0x7fde8c009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:23.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.442+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fde90152170 con 0x7fde90159f00 2023-10-16T12:49:23.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.642+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fde90157f50 con 0x7fde90159f00 2023-10-16T12:49:23.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.643+0000 7fde867fc640 1 -- 172.21.15.60:0/1202320872 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7fde8003b060 con 0x7fde90159f00 2023-10-16T12:49:23.646 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:23.646 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:49:23.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:23.647 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:49:23.647 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:49:23.647 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:23.647 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:49:23.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fde7406af90 msgr2=0x7fde7406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fde7406af90 0x7fde7406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde8c005880 tx=0x7fde8c009040 comp rx=0 tx=0).stop 2023-10-16T12:49:23.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 msgr2=0x7fde900c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900c77b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fde80009d90 tx=0x7fde80004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:23.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 shutdown_connections 2023-10-16T12:49:23.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fde7406af90 0x7fde7406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fde9015d880 0x7fde90151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde90159f00 0x7fde900c77b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 --2- 172.21.15.60:0/1202320872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde90159b20 0x7fde901f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.646+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 >> 172.21.15.60:0/1202320872 conn(0x7fde900cb410 msgr2=0x7fde900cde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:23.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.647+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 shutdown_connections 2023-10-16T12:49:23.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.647+0000 7fde97fa9640 1 -- 172.21.15.60:0/1202320872 wait complete. 2023-10-16T12:49:23.669 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:49:23.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:49:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.883+0000 7ffa6e490640 1 Processor -- start 2023-10-16T12:49:23.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.883+0000 7ffa6e490640 1 -- start start 2023-10-16T12:49:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6e490640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa6815cd20 0x7ffa680cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6e490640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa6815d100 0x7ffa680b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6e490640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa680b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6e490640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680b8430 con 0x7ffa6815d100 2023-10-16T12:49:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6e490640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680d2160 con 0x7ffa6815cd20 2023-10-16T12:49:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6e490640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680ce0b0 con 0x7ffa680b1810 2023-10-16T12:49:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6cc8d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa6815d100 0x7ffa680b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6cc8d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa6815d100 0x7ffa680b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42824/0 (socket says 172.21.15.60:42824) 2023-10-16T12:49:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6cc8d640 1 -- 172.21.15.60:0/4285867105 learned_addr learned my addr 172.21.15.60:0/4285867105 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.885+0000 7ffa6dc8f640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa680b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6d48e640 1 --2- 172.21.15.60:0/4285867105 >> v2:172.21.15.144:3300/0 conn(0x7ffa6815cd20 0x7ffa680cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6cc8d640 1 -- 172.21.15.60:0/4285867105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 msgr2=0x7ffa680b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6cc8d640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa680b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6cc8d640 1 -- 172.21.15.60:0/4285867105 >> v2:172.21.15.144:3300/0 conn(0x7ffa6815cd20 msgr2=0x7ffa680cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6cc8d640 1 --2- 172.21.15.60:0/4285867105 >> v2:172.21.15.144:3300/0 conn(0x7ffa6815cd20 0x7ffa680cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6cc8d640 1 -- 172.21.15.60:0/4285867105 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680b8680 con 0x7ffa6815d100 2023-10-16T12:49:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.886+0000 7ffa6dc8f640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa680b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.887+0000 7ffa6cc8d640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680b4590 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffa680cf4f0 tx=0x7ffa5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2078b2152604ddad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.888+0000 7ffa5ffff640 1 -- 172.21.15.60:0/4285867105 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58030860 con 0x7ffa6815d100 2023-10-16T12:49:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.888+0000 7ffa5ffff640 1 -- 172.21.15.60:0/4285867105 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa580309c0 con 0x7ffa6815d100 2023-10-16T12:49:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.888+0000 7ffa5ffff640 1 -- 172.21.15.60:0/4285867105 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58030d50 con 0x7ffa6815d100 2023-10-16T12:49:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.888+0000 7ffa6e490640 1 -- 172.21.15.60:0/4285867105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 msgr2=0x7ffa680b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.888+0000 7ffa6e490640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680b4590 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffa680cf4f0 tx=0x7ffa5802b9a0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 -- 172.21.15.60:0/4285867105 shutdown_connections 2023-10-16T12:49:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa680b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 --2- 172.21.15.60:0/4285867105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680b4590 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 --2- 172.21.15.60:0/4285867105 >> v2:172.21.15.144:3300/0 conn(0x7ffa6815cd20 0x7ffa680cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 -- 172.21.15.60:0/4285867105 >> 172.21.15.60:0/4285867105 conn(0x7ffa680cf590 msgr2=0x7ffa680cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 -- 172.21.15.60:0/4285867105 shutdown_connections 2023-10-16T12:49:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.889+0000 7ffa6e490640 1 -- 172.21.15.60:0/4285867105 wait complete. 2023-10-16T12:49:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.891+0000 7ffa6e490640 1 Processor -- start 2023-10-16T12:49:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.891+0000 7ffa6e490640 1 -- start start 2023-10-16T12:49:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6e490640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa6814f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6e490640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa6815cd20 0x7ffa6815d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6e490640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6e490640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681f3df0 con 0x7ffa6815d100 2023-10-16T12:49:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6e490640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681f3170 con 0x7ffa6815cd20 2023-10-16T12:49:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6e490640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681f3450 con 0x7ffa680b1810 2023-10-16T12:49:23.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6dc8f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6dc8f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42828/0 (socket says 172.21.15.60:42828) 2023-10-16T12:49:23.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6dc8f640 1 -- 172.21.15.60:0/1484674495 learned_addr learned my addr 172.21.15.60:0/1484674495 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:23.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6d48e640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa6814f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.893+0000 7ffa6cc8d640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa6815cd20 0x7ffa6815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6dc8f640 1 -- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 msgr2=0x7ffa6814f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6dc8f640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa6814f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6dc8f640 1 -- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa6815cd20 msgr2=0x7ffa6815d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6dc8f640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa6815cd20 0x7ffa6815d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6dc8f640 1 -- 172.21.15.60:0/1484674495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa580094d0 con 0x7ffa6815d100 2023-10-16T12:49:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6dc8f640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680cc3c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffa60007e10 tx=0x7ffa6000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60003b90 con 0x7ffa6815d100 2023-10-16T12:49:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6000f040 con 0x7ffa6815d100 2023-10-16T12:49:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6000de80 con 0x7ffa6815d100 2023-10-16T12:49:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa681f3730 con 0x7ffa6815d100 2023-10-16T12:49:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.894+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa681f2e70 con 0x7ffa6815d100 2023-10-16T12:49:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.896+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6804e9b0 con 0x7ffa6815d100 2023-10-16T12:49:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.897+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa60004030 con 0x7ffa6815d100 2023-10-16T12:49:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.898+0000 7ffa5dffb640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ffa4c06af90 0x7ffa4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.898+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffa600898b0 con 0x7ffa6815d100 2023-10-16T12:49:23.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.899+0000 7ffa6d48e640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ffa4c06af90 0x7ffa4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:23.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.900+0000 7ffa6d48e640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ffa4c06af90 0x7ffa4c06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffa64006fd0 tx=0x7ffa64008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:23.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:23.903+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa60054a90 con 0x7ffa6815d100 2023-10-16T12:49:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.093+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffa680d23b0 con 0x7ffa6815d100 2023-10-16T12:49:24.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.695+0000 7ffa5dffb640 1 -- 172.21.15.60:0/1484674495 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffa6001a0b0 con 0x7ffa6815d100 2023-10-16T12:49:24.698 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:49:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ffa4c06af90 msgr2=0x7ffa4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ffa4c06af90 0x7ffa4c06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffa64006fd0 tx=0x7ffa64008040 comp rx=0 tx=0).stop 2023-10-16T12:49:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 msgr2=0x7ffa680cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680cc3c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffa60007e10 tx=0x7ffa6000d4a0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 shutdown_connections 2023-10-16T12:49:24.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7ffa4c06af90 0x7ffa4c06d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa6815d100 0x7ffa680cc3c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa6815cd20 0x7ffa6815d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 --2- 172.21.15.60:0/1484674495 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa680b1810 0x7ffa6814f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 >> 172.21.15.60:0/1484674495 conn(0x7ffa680cf590 msgr2=0x7ffa6806c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:24.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.699+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 shutdown_connections 2023-10-16T12:49:24.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.700+0000 7ffa6e490640 1 -- 172.21.15.60:0/1484674495 wait complete. 2023-10-16T12:49:24.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:49:24.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.942+0000 7f48c2690640 1 Processor -- start 2023-10-16T12:49:24.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.943+0000 7f48c2690640 1 -- start start 2023-10-16T12:49:24.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.944+0000 7f48c2690640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f48bc0ce390 0x7f48bc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.944+0000 7f48c2690640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0707e0 con 0x7f48bc15aaa0 2023-10-16T12:49:24.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.944+0000 7f48c2690640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc06b7a0 con 0x7f48bc15ae80 2023-10-16T12:49:24.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.944+0000 7f48c2690640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc15ef20 con 0x7f48bc0ce390 2023-10-16T12:49:24.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.944+0000 7f48bbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f48bc15aaa0 0x7f48bc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45518/0 (socket says 172.21.15.60:45518) 2023-10-16T12:49:24.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.944+0000 7f48bbfff640 1 -- 172.21.15.60:0/3451856075 learned_addr learned my addr 172.21.15.60:0/3451856075 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:24.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516378291 0 0) 0x7f48bc0707e0 con 0x7f48bc15aaa0 2023-10-16T12:49:24.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a4003520 con 0x7f48bc15aaa0 2023-10-16T12:49:24.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48c0c2d640 1 --2- 172.21.15.60:0/3451856075 >> v2:172.21.15.144:3300/0 conn(0x7f48bc0ce390 0x7f48bc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728758570 0 0) 0x7f48a4003520 con 0x7f48bc15aaa0 2023-10-16T12:49:24.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 >> v2:172.21.15.144:3300/0 conn(0x7f48bc0ce390 msgr2=0x7f48bc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:24.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 --2- 172.21.15.60:0/3451856075 >> v2:172.21.15.144:3300/0 conn(0x7f48bc0ce390 0x7f48bc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 >> v1:172.21.15.144:6789/0 conn(0x7f48bc15ae80 legacy=0x7f48bc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc15f200 con 0x7f48bc15aaa0 2023-10-16T12:49:24.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.945+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f48ac001670 con 0x7f48bc15aaa0 2023-10-16T12:49:24.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 >> v1:172.21.15.60:6789/0 conn(0x7f48bc15aaa0 legacy=0x7f48bc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48a40047c0 0x7f48a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48a40052a0 0x7f48a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc15ef20 con 0x7f48a40039c0 2023-10-16T12:49:24.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a400c1c0 con 0x7f48a40047c0 2023-10-16T12:49:24.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a400c380 con 0x7f48a40052a0 2023-10-16T12:49:24.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bb7fe640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48a40047c0 0x7f48a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48c0c2d640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48a40052a0 0x7f48a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bbfff640 1 -- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48a40052a0 msgr2=0x7f48a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:24.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48a40052a0 0x7f48a400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bbfff640 1 -- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48a40047c0 msgr2=0x7f48a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:24.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48a40047c0 0x7f48a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48bbfff640 1 -- 172.21.15.60:0/3451856075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a4003520 con 0x7f48a40039c0 2023-10-16T12:49:24.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.946+0000 7f48c0c2d640 1 -- 172.21.15.60:0/3451856075 reap_dead start 2023-10-16T12:49:24.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.948+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48a4003de0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c0e0 tx=0x7f48a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfc975a6277d184 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:24.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.948+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac005ad0 con 0x7f48a40039c0 2023-10-16T12:49:24.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.948+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac005c30 con 0x7f48a40039c0 2023-10-16T12:49:24.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.948+0000 7f48baffd640 1 -- 172.21.15.60:0/3451856075 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac02ca00 con 0x7f48a40039c0 2023-10-16T12:49:24.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.948+0000 7f48c2690640 1 -- 172.21.15.60:0/3451856075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 msgr2=0x7f48a4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.948+0000 7f48c2690640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48a4003de0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00c0e0 tx=0x7f48a40057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.949+0000 7f48c2690640 1 -- 172.21.15.60:0/3451856075 shutdown_connections 2023-10-16T12:49:24.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.949+0000 7f48c2690640 1 --2- 172.21.15.60:0/3451856075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48a4003de0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.949+0000 7f48c2690640 1 -- 172.21.15.60:0/3451856075 >> 172.21.15.60:0/3451856075 conn(0x7f48bc0cb410 msgr2=0x7f48bc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:24.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.949+0000 7f48c2690640 1 -- 172.21.15.60:0/3451856075 shutdown_connections 2023-10-16T12:49:24.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.949+0000 7f48c2690640 1 -- 172.21.15.60:0/3451856075 wait complete. 2023-10-16T12:49:24.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.950+0000 7f48c2690640 1 Processor -- start 2023-10-16T12:49:24.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.951+0000 7f48c2690640 1 -- start start 2023-10-16T12:49:24.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48c2690640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48bc1f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48c2690640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48bc14bd90 0x7f48bc0c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48c2690640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48bc0c2ff0 0x7f48bc1e4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48c2690640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0c3570 con 0x7f48a40039c0 2023-10-16T12:49:24.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48c2690640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0c3850 con 0x7f48bc14bd90 2023-10-16T12:49:24.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48c2690640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0c3b30 con 0x7f48bc0c2ff0 2023-10-16T12:49:24.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48bbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48bc1f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48bbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48bc1f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42840/0 (socket says 172.21.15.60:42840) 2023-10-16T12:49:24.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.952+0000 7f48bbfff640 1 -- 172.21.15.60:0/3075240674 learned_addr learned my addr 172.21.15.60:0/3075240674 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:24.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bb7fe640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48bc14bd90 0x7f48bc0c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48c0c2d640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48bc0c2ff0 0x7f48bc1e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bbfff640 1 -- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48bc0c2ff0 msgr2=0x7f48bc1e4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48bc0c2ff0 0x7f48bc1e4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bbfff640 1 -- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48bc14bd90 msgr2=0x7f48bc0c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:24.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48bc14bd90 0x7f48bc0c2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:24.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bbfff640 1 -- 172.21.15.60:0/3075240674 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0c3e10 con 0x7f48a40039c0 2023-10-16T12:49:24.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48bbfff640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48bc1f2b80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f48ac000c00 tx=0x7f48ac02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:24.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac03c040 con 0x7f48a40039c0 2023-10-16T12:49:24.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac02ccb0 con 0x7f48a40039c0 2023-10-16T12:49:24.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.953+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0072b0 con 0x7f48a40039c0 2023-10-16T12:49:24.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.954+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc0c40f0 con 0x7f48a40039c0 2023-10-16T12:49:24.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.954+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc14c210 con 0x7f48a40039c0 2023-10-16T12:49:24.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.955+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc0c4280 con 0x7f48a40039c0 2023-10-16T12:49:24.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.956+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f48ac032070 con 0x7f48a40039c0 2023-10-16T12:49:24.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.957+0000 7f48b8ff9640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48a806afc0 0x7f48a806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:24.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.957+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f48ac0ab040 con 0x7f48a40039c0 2023-10-16T12:49:24.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.958+0000 7f48bb7fe640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48a806afc0 0x7f48a806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:24.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.959+0000 7f48bb7fe640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48a806afc0 0x7f48a806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f48bc15e170 tx=0x7f48b0009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:24.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:24.961+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48ac076240 con 0x7f48a40039c0 2023-10-16T12:49:25.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.157+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f48bc04e160 con 0x7f48a40039c0 2023-10-16T12:49:25.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.702+0000 7f48b8ff9640 1 -- 172.21.15.60:0/3075240674 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7f48ac079860 con 0x7f48a40039c0 2023-10-16T12:49:25.705 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:49:25.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48a806afc0 msgr2=0x7f48a806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48a806afc0 0x7f48a806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f48bc15e170 tx=0x7f48b0009040 comp rx=0 tx=0).stop 2023-10-16T12:49:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 msgr2=0x7f48bc1f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48bc1f2b80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f48ac000c00 tx=0x7f48ac02e070 comp rx=0 tx=0).stop 2023-10-16T12:49:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 shutdown_connections 2023-10-16T12:49:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48a806afc0 0x7f48a806d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48bc0c2ff0 0x7f48bc1e4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48bc14bd90 0x7f48bc0c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 --2- 172.21.15.60:0/3075240674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48a40039c0 0x7f48bc1f2b80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.707+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 >> 172.21.15.60:0/3075240674 conn(0x7f48bc0cb410 msgr2=0x7f48bc0cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:25.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.708+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 shutdown_connections 2023-10-16T12:49:25.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.708+0000 7f48c2690640 1 -- 172.21.15.60:0/3075240674 wait complete. 2023-10-16T12:49:25.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:49:25.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.941+0000 7f34260de640 1 Processor -- start 2023-10-16T12:49:25.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.941+0000 7f34260de640 1 -- start start 2023-10-16T12:49:25.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.943+0000 7f34260de640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f34200ad350 0x7f3420069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:25.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.943+0000 7f34260de640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342006a730 con 0x7f342015d150 2023-10-16T12:49:25.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.943+0000 7f34260de640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34200ac890 con 0x7f342006a2e0 2023-10-16T12:49:25.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.943+0000 7f34260de640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342015d530 con 0x7f34200ad350 2023-10-16T12:49:25.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.943+0000 7f341f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f342015d150 0x7f3420160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45522/0 (socket says 172.21.15.60:45522) 2023-10-16T12:49:25.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.943+0000 7f341f7fe640 1 -- 172.21.15.60:0/2889213648 learned_addr learned my addr 172.21.15.60:0/2889213648 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:25.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.944+0000 7f341effd640 1 --2- 172.21.15.60:0/2889213648 >> v2:172.21.15.144:3300/0 conn(0x7f34200ad350 0x7f3420069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:25.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.944+0000 7f341e7fc640 1 -- 172.21.15.60:0/2889213648 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434651324 0 0) 0x7f3410001110 con 0x7f342015d150 2023-10-16T12:49:25.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.944+0000 7f341e7fc640 1 -- 172.21.15.60:0/2889213648 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3408003520 con 0x7f342015d150 2023-10-16T12:49:25.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.945+0000 7f341effd640 1 -- 172.21.15.60:0/2889213648 >> v1:172.21.15.144:6789/0 conn(0x7f342006a2e0 legacy=0x7f34200c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.945+0000 7f341effd640 1 -- 172.21.15.60:0/2889213648 >> v1:172.21.15.60:6789/0 conn(0x7f342015d150 legacy=0x7f3420160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.945+0000 7f341e7fc640 1 -- 172.21.15.60:0/2889213648 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1923134634 0 0) 0x7f3408003520 con 0x7f342015d150 2023-10-16T12:49:25.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.945+0000 7f341effd640 1 -- 172.21.15.60:0/2889213648 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200ad7a0 con 0x7f34200ad350 2023-10-16T12:49:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.946+0000 7f341effd640 1 --2- 172.21.15.60:0/2889213648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34200ad350 0x7f3420069da0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f34140097a0 tx=0x7f341402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d9b9591ec84bb4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.946+0000 7f341e7fc640 1 -- 172.21.15.60:0/2889213648 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341402f050 con 0x7f34200ad350 2023-10-16T12:49:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.946+0000 7f341e7fc640 1 -- 172.21.15.60:0/2889213648 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3414031500 con 0x7f34200ad350 2023-10-16T12:49:25.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f341e7fc640 1 -- 172.21.15.60:0/2889213648 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410001800 con 0x7f34200ad350 2023-10-16T12:49:25.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 -- 172.21.15.60:0/2889213648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34200ad350 msgr2=0x7f3420069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 --2- 172.21.15.60:0/2889213648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34200ad350 0x7f3420069da0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f34140097a0 tx=0x7f341402b820 comp rx=0 tx=0).stop 2023-10-16T12:49:25.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 -- 172.21.15.60:0/2889213648 shutdown_connections 2023-10-16T12:49:25.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 --2- 172.21.15.60:0/2889213648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34200ad350 0x7f3420069da0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 -- 172.21.15.60:0/2889213648 >> 172.21.15.60:0/2889213648 conn(0x7f34200ce9b0 msgr2=0x7f34200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:25.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 -- 172.21.15.60:0/2889213648 shutdown_connections 2023-10-16T12:49:25.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.947+0000 7f34260de640 1 -- 172.21.15.60:0/2889213648 wait complete. 2023-10-16T12:49:25.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.949+0000 7f34260de640 1 Processor -- start 2023-10-16T12:49:25.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.949+0000 7f34260de640 1 -- start start 2023-10-16T12:49:25.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f34260de640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f342006a2e0 0x7f34201558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:25.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f34260de640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34200ad350 0x7f34201e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:25.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f34260de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 0x7f34200b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:25.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f34260de640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201e7e00 con 0x7f342015d150 2023-10-16T12:49:25.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f34260de640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201e8050 con 0x7f342006a2e0 2023-10-16T12:49:25.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f34260de640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34201e8330 con 0x7f34200ad350 2023-10-16T12:49:25.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.950+0000 7f341ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 0x7f34200b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:25.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 0x7f34200b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42856/0 (socket says 172.21.15.60:42856) 2023-10-16T12:49:25.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 -- 172.21.15.60:0/734207676 learned_addr learned my addr 172.21.15.60:0/734207676 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:25.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341f7fe640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f342006a2e0 0x7f34201558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:25.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341effd640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34200ad350 0x7f34201e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:25.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 -- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34200ad350 msgr2=0x7f34201e78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34200ad350 0x7f34201e78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 -- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f342006a2e0 msgr2=0x7f34201558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:25.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f342006a2e0 0x7f34201558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:25.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 -- 172.21.15.60:0/734207676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414009400 con 0x7f342015d150 2023-10-16T12:49:25.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.951+0000 7f341ffff640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 0x7f34200b3ef0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f341800c7a0 tx=0x7f341800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:25.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.952+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418003b70 con 0x7f342015d150 2023-10-16T12:49:25.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.952+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3418003cf0 con 0x7f342015d150 2023-10-16T12:49:25.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.952+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34201e8610 con 0x7f342015d150 2023-10-16T12:49:25.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.952+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418003ea0 con 0x7f342015d150 2023-10-16T12:49:25.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.952+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34201e8bd0 con 0x7f342015d150 2023-10-16T12:49:25.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.953+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342004e160 con 0x7f342015d150 2023-10-16T12:49:25.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.955+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f341800aa90 con 0x7f342015d150 2023-10-16T12:49:25.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.955+0000 7f3407fff640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f340c06afc0 0x7f340c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:25.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.956+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3418089f70 con 0x7f342015d150 2023-10-16T12:49:25.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.956+0000 7f341f7fe640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f340c06afc0 0x7f340c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:25.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.957+0000 7f341f7fe640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f340c06afc0 0x7f340c06d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3410005880 tx=0x7f3410009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:25.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:25.959+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34180550d0 con 0x7f342015d150 2023-10-16T12:49:26.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.199+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f34200ace20 con 0x7f342015d150 2023-10-16T12:49:26.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:26.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:26.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.770+0000 7f3407fff640 1 -- 172.21.15.60:0/734207676 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f34180586f0 con 0x7f342015d150 2023-10-16T12:49:26.773 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 8 and data pool 9 2023-10-16T12:49:26.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.779+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f340c06afc0 msgr2=0x7f340c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:26.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.779+0000 7f34260de640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f340c06afc0 0x7f340c06d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3410005880 tx=0x7f3410009040 comp rx=0 tx=0).stop 2023-10-16T12:49:26.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.779+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 msgr2=0x7f34200b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:26.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.779+0000 7f34260de640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 0x7f34200b3ef0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f341800c7a0 tx=0x7f341800cc60 comp rx=0 tx=0).stop 2023-10-16T12:49:26.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 shutdown_connections 2023-10-16T12:49:26.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f340c06afc0 0x7f340c06d480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:26.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34200ad350 0x7f34201e78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:26.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f342015d150 0x7f34200b3ef0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:26.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 --2- 172.21.15.60:0/734207676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f342006a2e0 0x7f34201558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:26.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 >> 172.21.15.60:0/734207676 conn(0x7f34200ce9b0 msgr2=0x7f342006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:26.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 shutdown_connections 2023-10-16T12:49:26.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.780+0000 7f34260de640 1 -- 172.21.15.60:0/734207676 wait complete. 2023-10-16T12:49:26.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:26.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.994+0000 7f4bf2844640 1 Processor -- start 2023-10-16T12:49:26.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.994+0000 7f4bf2844640 1 -- start start 2023-10-16T12:49:26.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.995+0000 7f4bf2844640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bec15ae80 0x7f4bec15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.995+0000 7f4bf2844640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4bec0ce390 0x7f4bec070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.995+0000 7f4bf2844640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec15ed60 con 0x7f4bec15aaa0 2023-10-16T12:49:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.995+0000 7f4bf2844640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec06b810 con 0x7f4bec15ae80 2023-10-16T12:49:26.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.995+0000 7f4bf2844640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec0707e0 con 0x7f4bec0ce390 2023-10-16T12:49:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bec15ae80 0x7f4bec15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bec15ae80 0x7f4bec15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42862/0 (socket says 172.21.15.60:42862) 2023-10-16T12:49:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 -- 172.21.15.60:0/3133859123 learned_addr learned my addr 172.21.15.60:0/3133859123 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4bf0de1640 1 --2- 172.21.15.60:0/3133859123 >> v2:172.21.15.144:3300/0 conn(0x7f4bec0ce390 0x7f4bec070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 -- 172.21.15.60:0/3133859123 >> v2:172.21.15.144:3300/0 conn(0x7f4bec0ce390 msgr2=0x7f4bec070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:27.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 --2- 172.21.15.60:0/3133859123 >> v2:172.21.15.144:3300/0 conn(0x7f4bec0ce390 0x7f4bec070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 -- 172.21.15.60:0/3133859123 >> v1:172.21.15.144:6789/0 conn(0x7f4bec15aaa0 legacy=0x7f4bec0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:27.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4beb7fe640 1 -- 172.21.15.60:0/3133859123 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec15ef90 con 0x7f4bec15ae80 2023-10-16T12:49:27.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.996+0000 7f4bf0de1640 1 --2- 172.21.15.60:0/3133859123 >> v2:172.21.15.144:3300/0 conn(0x7f4bec0ce390 0x7f4bec070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:27.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.997+0000 7f4beb7fe640 1 --2- 172.21.15.60:0/3133859123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec15e820 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0c2490 tx=0x7f4bdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2bc3085d73d0b34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4beaffd640 1 -- 172.21.15.60:0/3133859123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc031870 con 0x7f4bec15ae80 2023-10-16T12:49:27.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4beaffd640 1 -- 172.21.15.60:0/3133859123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc0319d0 con 0x7f4bec15ae80 2023-10-16T12:49:27.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4beaffd640 1 -- 172.21.15.60:0/3133859123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc031d60 con 0x7f4bec15ae80 2023-10-16T12:49:27.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 -- 172.21.15.60:0/3133859123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 msgr2=0x7f4bec15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 --2- 172.21.15.60:0/3133859123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec15e820 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0c2490 tx=0x7f4bdc02b910 comp rx=0 tx=0).stop 2023-10-16T12:49:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 -- 172.21.15.60:0/3133859123 shutdown_connections 2023-10-16T12:49:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 --2- 172.21.15.60:0/3133859123 >> v2:172.21.15.144:3300/0 conn(0x7f4bec0ce390 0x7f4bec070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 --2- 172.21.15.60:0/3133859123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec15e820 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 -- 172.21.15.60:0/3133859123 >> 172.21.15.60:0/3133859123 conn(0x7f4bec0cb410 msgr2=0x7f4bec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 -- 172.21.15.60:0/3133859123 shutdown_connections 2023-10-16T12:49:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:26.998+0000 7f4bf2844640 1 -- 172.21.15.60:0/3133859123 wait complete. 2023-10-16T12:49:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.000+0000 7f4bf2844640 1 Processor -- start 2023-10-16T12:49:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.000+0000 7f4bf2844640 1 -- start start 2023-10-16T12:49:27.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.001+0000 7f4bf2844640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bec0ce390 0x7f4bec147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.001+0000 7f4bf2844640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bec15aaa0 0x7f4bec14d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.001+0000 7f4bf2844640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.001+0000 7f4bf2844640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec1479b0 con 0x7f4bec15ae80 2023-10-16T12:49:27.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.001+0000 7f4bf2844640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec147c00 con 0x7f4bec15aaa0 2023-10-16T12:49:27.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.001+0000 7f4bf2844640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec147ee0 con 0x7f4bec0ce390 2023-10-16T12:49:27.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42864/0 (socket says 172.21.15.60:42864) 2023-10-16T12:49:27.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 -- 172.21.15.60:0/2799066023 learned_addr learned my addr 172.21.15.60:0/2799066023 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:27.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bebfff640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bec0ce390 0x7f4bec147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4beb7fe640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bec15aaa0 0x7f4bec14d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 -- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bec0ce390 msgr2=0x7f4bec147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bec0ce390 0x7f4bec147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 -- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bec15aaa0 msgr2=0x7f4bec14d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bec15aaa0 0x7f4bec14d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 -- 172.21.15.60:0/2799066023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0094d0 con 0x7f4bec15ae80 2023-10-16T12:49:27.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bebfff640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bec0ce390 0x7f4bec147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:27.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4beb7fe640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bec15aaa0 0x7f4bec14d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:27.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.002+0000 7f4bf0de1640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec149a40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4be400cef0 tx=0x7f4be400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.003+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be400d840 con 0x7f4bec15ae80 2023-10-16T12:49:27.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.003+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be400d9c0 con 0x7f4bec15ae80 2023-10-16T12:49:27.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.003+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be400dd70 con 0x7f4bec15ae80 2023-10-16T12:49:27.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.003+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec1481c0 con 0x7f4bec15ae80 2023-10-16T12:49:27.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.003+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec14da40 con 0x7f4bec15ae80 2023-10-16T12:49:27.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.004+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec14dc30 con 0x7f4bec15ae80 2023-10-16T12:49:27.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.005+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4be401e3b0 con 0x7f4bec15ae80 2023-10-16T12:49:27.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.006+0000 7f4be8ff9640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f4bcc06af90 0x7f4bcc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.006+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4be408a1e0 con 0x7f4bec15ae80 2023-10-16T12:49:27.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.007+0000 7f4bebfff640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f4bcc06af90 0x7f4bcc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.008+0000 7f4bebfff640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f4bcc06af90 0x7f4bcc06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4007fd0 tx=0x7f4bd4009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.011+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bec14dc30 con 0x7f4bec15ae80 2023-10-16T12:49:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.208+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bec1486e0 con 0x7f4bec15ae80 2023-10-16T12:49:27.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.210+0000 7f4be8ff9640 1 -- 172.21.15.60:0/2799066023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f4bec1486e0 con 0x7f4bec15ae80 2023-10-16T12:49:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:27.213 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":85,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:26.201261+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f4bcc06af90 msgr2=0x7f4bcc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f4bcc06af90 0x7f4bcc06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4007fd0 tx=0x7f4bd4009040 comp rx=0 tx=0).stop 2023-10-16T12:49:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 msgr2=0x7f4bec149a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec149a40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4be400cef0 tx=0x7f4be400a710 comp rx=0 tx=0).stop 2023-10-16T12:49:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 shutdown_connections 2023-10-16T12:49:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f4bcc06af90 0x7f4bcc06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bec15ae80 0x7f4bec149a40 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bec15aaa0 0x7f4bec14d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 --2- 172.21.15.60:0/2799066023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4bec0ce390 0x7f4bec147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.212+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 >> 172.21.15.60:0/2799066023 conn(0x7f4bec0cb410 msgr2=0x7f4bec0acaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:27.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.213+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 shutdown_connections 2023-10-16T12:49:27.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.213+0000 7f4bf2844640 1 -- 172.21.15.60:0/2799066023 wait complete. 2023-10-16T12:49:27.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:27.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.454+0000 7f13e1773640 1 Processor -- start 2023-10-16T12:49:27.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.455+0000 7f13e1773640 1 -- start start 2023-10-16T12:49:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.456+0000 7f13e1773640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f13dc0cce60 0x7f13dc0acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.456+0000 7f13e1773640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc15d120 con 0x7f13dc15cd40 2023-10-16T12:49:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.456+0000 7f13e1773640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc15e180 con 0x7f13dc15dcc0 2023-10-16T12:49:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.456+0000 7f13e1773640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13dc06b300 con 0x7f13dc0cce60 2023-10-16T12:49:27.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13dbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f13dc15cd40 0x7f13dc0cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45524/0 (socket says 172.21.15.60:45524) 2023-10-16T12:49:27.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13dbfff640 1 -- 172.21.15.60:0/1742811560 learned_addr learned my addr 172.21.15.60:0/1742811560 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:27.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921361867 0 0) 0x7f13dc15d120 con 0x7f13dc15cd40 2023-10-16T12:49:27.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003520 con 0x7f13dc15cd40 2023-10-16T12:49:27.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13e0f72640 1 --2- 172.21.15.60:0/1742811560 >> v2:172.21.15.144:3300/0 conn(0x7f13dc0cce60 0x7f13dc0acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2657288192 0 0) 0x7f13c8003520 con 0x7f13dc15cd40 2023-10-16T12:49:27.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 >> v2:172.21.15.144:3300/0 conn(0x7f13dc0cce60 msgr2=0x7f13dc0acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:27.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 --2- 172.21.15.60:0/1742811560 >> v2:172.21.15.144:3300/0 conn(0x7f13dc0cce60 0x7f13dc0acfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 >> v1:172.21.15.144:6789/0 conn(0x7f13dc15dcc0 legacy=0x7f13dc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc06b5e0 con 0x7f13dc15cd40 2023-10-16T12:49:27.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.457+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f13cc001670 con 0x7f13dc15cd40 2023-10-16T12:49:27.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 >> v1:172.21.15.60:6789/0 conn(0x7f13dc15cd40 legacy=0x7f13dc0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13c8005180 0x7f13c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc06b300 con 0x7f13c8003970 2023-10-16T12:49:27.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c800c0d0 con 0x7f13c80047c0 2023-10-16T12:49:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c800c290 con 0x7f13c8005180 2023-10-16T12:49:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13dbfff640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13d3fff640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13dbfff640 1 -- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13c8005180 msgr2=0x7f13c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13dbfff640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13c8005180 0x7f13c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13dbfff640 1 -- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c80047c0 msgr2=0x7f13c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:27.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13dbfff640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c80047c0 0x7f13c8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13dbfff640 1 -- 172.21.15.60:0/1742811560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c8003520 con 0x7f13c8003970 2023-10-16T12:49:27.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.458+0000 7f13e0f72640 1 -- 172.21.15.60:0/1742811560 reap_dead start 2023-10-16T12:49:27.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.459+0000 7f13dbfff640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00c240 tx=0x7f13c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5155a341d54d511b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.460+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc005ad0 con 0x7f13c8003970 2023-10-16T12:49:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.460+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc005c30 con 0x7f13c8003970 2023-10-16T12:49:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.460+0000 7f13db7fe640 1 -- 172.21.15.60:0/1742811560 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc02caf0 con 0x7f13c8003970 2023-10-16T12:49:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.460+0000 7f13e1773640 1 -- 172.21.15.60:0/1742811560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13c8003970 msgr2=0x7f13c8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.460+0000 7f13e1773640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00c240 tx=0x7f13c80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.461+0000 7f13e1773640 1 -- 172.21.15.60:0/1742811560 shutdown_connections 2023-10-16T12:49:27.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.461+0000 7f13e1773640 1 --2- 172.21.15.60:0/1742811560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13c8003970 0x7f13c8003d90 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.461+0000 7f13e1773640 1 -- 172.21.15.60:0/1742811560 >> 172.21.15.60:0/1742811560 conn(0x7f13dc0ceb80 msgr2=0x7f13dc0cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:27.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.461+0000 7f13e1773640 1 -- 172.21.15.60:0/1742811560 shutdown_connections 2023-10-16T12:49:27.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.461+0000 7f13e1773640 1 -- 172.21.15.60:0/1742811560 wait complete. 2023-10-16T12:49:27.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.463+0000 7f13e1773640 1 Processor -- start 2023-10-16T12:49:27.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.463+0000 7f13e1773640 1 -- start start 2023-10-16T12:49:27.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e1773640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c8003970 0x7f13dc0c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e1773640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13dc160260 0x7f13dc1606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e1773640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 0x7f13dc0aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e1773640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc160c20 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e1773640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0af640 con 0x7f13c8003970 2023-10-16T12:49:27.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e1773640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0af920 con 0x7f13dc160260 2023-10-16T12:49:27.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 0x7f13dc0aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 0x7f13dc0aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42878/0 (socket says 172.21.15.60:42878) 2023-10-16T12:49:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 -- 172.21.15.60:0/2163310477 learned_addr learned my addr 172.21.15.60:0/2163310477 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 -- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13dc160260 msgr2=0x7f13dc1606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13dc160260 0x7f13dc1606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 -- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c8003970 msgr2=0x7f13dc0c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c8003970 0x7f13dc0c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.464+0000 7f13e0f72640 1 -- 172.21.15.60:0/2163310477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0afc00 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.465+0000 7f13e0f72640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 0x7f13dc0aed50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0c9160 tx=0x7f13d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.465+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d4007e40 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.465+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4014420 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.465+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc0afee0 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.465+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d40147d0 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.465+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc0c8e70 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.466+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc0b03a0 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.467+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f13d4014930 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.468+0000 7f13d97fa640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f13bc06af90 0x7f13bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.468+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13d4089cb0 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.469+0000 7f13dbfff640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f13bc06af90 0x7f13bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.470+0000 7f13dbfff640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f13bc06af90 0x7f13bc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f13cc034000 tx=0x7f13cc039040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.472+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13d4054bb0 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.672+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13dc0afa60 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.675+0000 7f13d97fa640 1 -- 172.21.15.60:0/2163310477 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f13dc0afa60 con 0x7f13dc0ae8d0 2023-10-16T12:49:27.678 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:27.679 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":85,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:26.201261+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:27.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.677+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f13bc06af90 msgr2=0x7f13bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.677+0000 7f13e1773640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f13bc06af90 0x7f13bc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f13cc034000 tx=0x7f13cc039040 comp rx=0 tx=0).stop 2023-10-16T12:49:27.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.677+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 msgr2=0x7f13dc0aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.677+0000 7f13e1773640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 0x7f13dc0aed50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0c9160 tx=0x7f13d400e4b0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 shutdown_connections 2023-10-16T12:49:27.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f13bc06af90 0x7f13bc06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f13dc160260 0x7f13dc1606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13dc0ae8d0 0x7f13dc0aed50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 --2- 172.21.15.60:0/2163310477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13c8003970 0x7f13dc0c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 >> 172.21.15.60:0/2163310477 conn(0x7f13dc0ceb80 msgr2=0x7f13dc072f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 shutdown_connections 2023-10-16T12:49:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.678+0000 7f13e1773640 1 -- 172.21.15.60:0/2163310477 wait complete. 2023-10-16T12:49:27.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:49:27.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.924+0000 7f6ec0987640 1 Processor -- start 2023-10-16T12:49:27.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.924+0000 7f6ec0987640 1 -- start start 2023-10-16T12:49:27.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.925+0000 7f6ec0987640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ebc06c040 0x7f6ebc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ec0987640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ebc0cda80 0x7f6ebc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ec0987640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc06c960 con 0x7f6ebc0ad970 2023-10-16T12:49:27.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ec0987640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc15b9d0 con 0x7f6ebc0cda80 2023-10-16T12:49:27.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ec0987640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc15bcb0 con 0x7f6ebc06c040 2023-10-16T12:49:27.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ebbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ebc0cda80 0x7f6ebc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ebaffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6ebc0ad970 0x7f6ebc15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45538/0 (socket says 172.21.15.60:45538) 2023-10-16T12:49:27.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ebaffd640 1 -- 172.21.15.60:0/3874327225 learned_addr learned my addr 172.21.15.60:0/3874327225 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:27.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ebbfff640 1 --2- 172.21.15.60:0/3874327225 >> v2:172.21.15.60:3300/0 conn(0x7f6ebc0cda80 0x7f6ebc15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42890/0 (socket says 172.21.15.60:42890) 2023-10-16T12:49:27.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.926+0000 7f6ebb7fe640 1 --2- 172.21.15.60:0/3874327225 >> v2:172.21.15.144:3300/0 conn(0x7f6ebc06c040 0x7f6ebc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.927+0000 7f6eba7fc640 1 -- 172.21.15.60:0/3874327225 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351825291 0 0) 0x7f6ebc06c960 con 0x7f6ebc0ad970 2023-10-16T12:49:27.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.927+0000 7f6eba7fc640 1 -- 172.21.15.60:0/3874327225 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea8003520 con 0x7f6ebc0ad970 2023-10-16T12:49:27.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.927+0000 7f6ebbfff640 1 -- 172.21.15.60:0/3874327225 >> v2:172.21.15.144:3300/0 conn(0x7f6ebc06c040 msgr2=0x7f6ebc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:27.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.927+0000 7f6ebbfff640 1 --2- 172.21.15.60:0/3874327225 >> v2:172.21.15.144:3300/0 conn(0x7f6ebc06c040 0x7f6ebc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.927+0000 7f6ebbfff640 1 -- 172.21.15.60:0/3874327225 >> v1:172.21.15.60:6789/0 conn(0x7f6ebc0ad970 legacy=0x7f6ebc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.927+0000 7f6ebbfff640 1 -- 172.21.15.60:0/3874327225 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc15bf90 con 0x7f6ebc0cda80 2023-10-16T12:49:27.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.928+0000 7f6ebbfff640 1 --2- 172.21.15.60:0/3874327225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc15ed90 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc0cf350 tx=0x7f6eb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5196acbd2ff3ffeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.928+0000 7f6eba7fc640 1 -- 172.21.15.60:0/3874327225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4031830 con 0x7f6ebc0cda80 2023-10-16T12:49:27.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6eba7fc640 1 -- 172.21.15.60:0/3874327225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4031990 con 0x7f6ebc0cda80 2023-10-16T12:49:27.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6eba7fc640 1 -- 172.21.15.60:0/3874327225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4031d00 con 0x7f6ebc0cda80 2023-10-16T12:49:27.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 -- 172.21.15.60:0/3874327225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 msgr2=0x7f6ebc15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 --2- 172.21.15.60:0/3874327225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc15ed90 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc0cf350 tx=0x7f6eb402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 -- 172.21.15.60:0/3874327225 shutdown_connections 2023-10-16T12:49:27.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 --2- 172.21.15.60:0/3874327225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc15ed90 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 --2- 172.21.15.60:0/3874327225 >> v2:172.21.15.144:3300/0 conn(0x7f6ebc06c040 0x7f6ebc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 -- 172.21.15.60:0/3874327225 >> 172.21.15.60:0/3874327225 conn(0x7f6ebc0ce6a0 msgr2=0x7f6ebc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 -- 172.21.15.60:0/3874327225 shutdown_connections 2023-10-16T12:49:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.929+0000 7f6ec0987640 1 -- 172.21.15.60:0/3874327225 wait complete. 2023-10-16T12:49:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.931+0000 7f6ec0987640 1 Processor -- start 2023-10-16T12:49:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.931+0000 7f6ec0987640 1 -- start start 2023-10-16T12:49:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.932+0000 7f6ec0987640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ebc06c040 0x7f6ebc1f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.932+0000 7f6ec0987640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ebc0ad970 0x7f6ebc0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.932+0000 7f6ec0987640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.932+0000 7f6ec0987640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc1f2f50 con 0x7f6ebc0cda80 2023-10-16T12:49:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.932+0000 7f6ec0987640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0ca230 con 0x7f6ebc06c040 2023-10-16T12:49:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.932+0000 7f6ec0987640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0b0e20 con 0x7f6ebc0ad970 2023-10-16T12:49:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc0aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42904/0 (socket says 172.21.15.60:42904) 2023-10-16T12:49:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 -- 172.21.15.60:0/1864655813 learned_addr learned my addr 172.21.15.60:0/1864655813 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebaffd640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ebc0ad970 0x7f6ebc0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebb7fe640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ebc06c040 0x7f6ebc1f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 -- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ebc0ad970 msgr2=0x7f6ebc0b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ebc0ad970 0x7f6ebc0b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 -- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ebc06c040 msgr2=0x7f6ebc1f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ebc06c040 0x7f6ebc1f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebbfff640 1 -- 172.21.15.60:0/1864655813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb4009400 con 0x7f6ebc0cda80 2023-10-16T12:49:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.933+0000 7f6ebb7fe640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ebc06c040 0x7f6ebc1f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.934+0000 7f6ebbfff640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc0aeb10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4034040 tx=0x7f6eb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.934+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb403a830 con 0x7f6ebc0cda80 2023-10-16T12:49:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.934+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb403a990 con 0x7f6ebc0cda80 2023-10-16T12:49:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.934+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb403ad00 con 0x7f6ebc0cda80 2023-10-16T12:49:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.934+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc151680 con 0x7f6ebc0cda80 2023-10-16T12:49:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.934+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc0c9f50 con 0x7f6ebc0cda80 2023-10-16T12:49:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.935+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc06b330 con 0x7f6ebc0cda80 2023-10-16T12:49:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.936+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6eb404d450 con 0x7f6ebc0cda80 2023-10-16T12:49:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.937+0000 7f6eb3fff640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6e9c06afc0 0x7f6e9c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.937+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6eb40ab8f0 con 0x7f6ebc0cda80 2023-10-16T12:49:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.938+0000 7f6ebb7fe640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6e9c06afc0 0x7f6e9c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.939+0000 7f6ebb7fe640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6e9c06afc0 0x7f6e9c06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ac0 tx=0x7f6eac008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:27.942+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6eb40766e0 con 0x7f6ebc0cda80 2023-10-16T12:49:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.174+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6ebc04e9b0 con 0x7f6ebc0cda80 2023-10-16T12:49:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.780+0000 7f6eb3fff640 1 -- 172.21.15.60:0/1864655813 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6eb403ae60 con 0x7f6ebc0cda80 2023-10-16T12:49:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6e9c06afc0 msgr2=0x7f6e9c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6e9c06afc0 0x7f6e9c06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ac0 tx=0x7f6eac008040 comp rx=0 tx=0).stop 2023-10-16T12:49:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 msgr2=0x7f6ebc0aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc0aeb10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4034040 tx=0x7f6eb40046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:28.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 shutdown_connections 2023-10-16T12:49:28.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f6e9c06afc0 0x7f6e9c06d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:28.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ebc0ad970 0x7f6ebc0b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ebc0cda80 0x7f6ebc0aeb10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 --2- 172.21.15.60:0/1864655813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ebc06c040 0x7f6ebc1f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.784+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 >> 172.21.15.60:0/1864655813 conn(0x7f6ebc0ce6a0 msgr2=0x7f6ebc0ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.785+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 shutdown_connections 2023-10-16T12:49:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:28.785+0000 7f6ec0987640 1 -- 172.21.15.60:0/1864655813 wait complete. 2023-10-16T12:49:28.809 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:49:28.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:49:29.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.038+0000 7f233cc09640 1 Processor -- start 2023-10-16T12:49:29.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.038+0000 7f233cc09640 1 -- start start 2023-10-16T12:49:29.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233cc09640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f233806a4b0 0x7f23380ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:29.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233cc09640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23380c9ca0 0x7f23380ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233cc09640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380cb030 con 0x7f233815ad70 2023-10-16T12:49:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233cc09640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23380cae30 con 0x7f233806a4b0 2023-10-16T12:49:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233cc09640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380c91c0 con 0x7f23380c9ca0 2023-10-16T12:49:29.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f233815ad70 0x7f2338069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50260/0 (socket says 172.21.15.60:50260) 2023-10-16T12:49:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233659c640 1 -- 172.21.15.60:0/2602605423 learned_addr learned my addr 172.21.15.60:0/2602605423 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f2335d9b640 1 --2- 172.21.15.60:0/2602605423 >> v2:172.21.15.60:3300/0 conn(0x7f233806a4b0 0x7f23380ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:29.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f2336d9d640 1 --2- 172.21.15.60:0/2602605423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23380c9ca0 0x7f23380ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:29.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233559a640 1 -- 172.21.15.60:0/2602605423 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289508906 0 0) 0x7f23380cb030 con 0x7f233815ad70 2023-10-16T12:49:29.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.040+0000 7f233559a640 1 -- 172.21.15.60:0/2602605423 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2324003520 con 0x7f233815ad70 2023-10-16T12:49:29.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.041+0000 7f2335d9b640 1 -- 172.21.15.60:0/2602605423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23380c9ca0 msgr2=0x7f23380ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:29.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.041+0000 7f2335d9b640 1 --2- 172.21.15.60:0/2602605423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23380c9ca0 0x7f23380ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.041+0000 7f2335d9b640 1 -- 172.21.15.60:0/2602605423 >> v1:172.21.15.60:6789/0 conn(0x7f233815ad70 legacy=0x7f2338069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:29.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.041+0000 7f2335d9b640 1 -- 172.21.15.60:0/2602605423 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23380c94a0 con 0x7f233806a4b0 2023-10-16T12:49:29.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.042+0000 7f2335d9b640 1 --2- 172.21.15.60:0/2602605423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233806a4b0 0x7f23380ca3b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f23200097a0 tx=0x7f232002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eec832017790146 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:29.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.042+0000 7f233559a640 1 -- 172.21.15.60:0/2602605423 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232002d040 con 0x7f233806a4b0 2023-10-16T12:49:29.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.042+0000 7f233559a640 1 -- 172.21.15.60:0/2602605423 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320031810 con 0x7f233806a4b0 2023-10-16T12:49:29.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.042+0000 7f233559a640 1 -- 172.21.15.60:0/2602605423 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320031ba0 con 0x7f233806a4b0 2023-10-16T12:49:29.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 -- 172.21.15.60:0/2602605423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233806a4b0 msgr2=0x7f23380ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:29.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 --2- 172.21.15.60:0/2602605423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233806a4b0 0x7f23380ca3b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f23200097a0 tx=0x7f232002b8b0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 -- 172.21.15.60:0/2602605423 shutdown_connections 2023-10-16T12:49:29.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 --2- 172.21.15.60:0/2602605423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23380c9ca0 0x7f23380ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 --2- 172.21.15.60:0/2602605423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233806a4b0 0x7f23380ca3b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 -- 172.21.15.60:0/2602605423 >> 172.21.15.60:0/2602605423 conn(0x7f23380ce9b0 msgr2=0x7f23380cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:29.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 -- 172.21.15.60:0/2602605423 shutdown_connections 2023-10-16T12:49:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.043+0000 7f233cc09640 1 -- 172.21.15.60:0/2602605423 wait complete. 2023-10-16T12:49:29.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.045+0000 7f233cc09640 1 Processor -- start 2023-10-16T12:49:29.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.045+0000 7f233cc09640 1 -- start start 2023-10-16T12:49:29.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233cc09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f233806a4b0 0x7f2338156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:29.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233cc09640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 0x7f233815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:29.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233cc09640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f233815ad70 0x7f233815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:29.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233cc09640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23381f6380 con 0x7f23380c9ca0 2023-10-16T12:49:29.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233cc09640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233815d490 con 0x7f233806a4b0 2023-10-16T12:49:29.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233cc09640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23381f67a0 con 0x7f233815ad70 2023-10-16T12:49:29.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 0x7f233815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:29.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 0x7f233815cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42568/0 (socket says 172.21.15.60:42568) 2023-10-16T12:49:29.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 -- 172.21.15.60:0/1462962714 learned_addr learned my addr 172.21.15.60:0/1462962714 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:29.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f233659c640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f233806a4b0 0x7f2338156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:29.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 -- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f233815ad70 msgr2=0x7f233815d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:29.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f233815ad70 0x7f233815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 -- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f233806a4b0 msgr2=0x7f2338156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:29.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f233806a4b0 0x7f2338156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.046+0000 7f2335d9b640 1 -- 172.21.15.60:0/1462962714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320009400 con 0x7f23380c9ca0 2023-10-16T12:49:29.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.047+0000 7f2335d9b640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 0x7f233815cad0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f23200098c0 tx=0x7f23200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:29.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.047+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320006e10 con 0x7f23380c9ca0 2023-10-16T12:49:29.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.047+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232003ad60 con 0x7f23380c9ca0 2023-10-16T12:49:29.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.047+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320044620 con 0x7f23380c9ca0 2023-10-16T12:49:29.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.047+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23381f59e0 con 0x7f23380c9ca0 2023-10-16T12:49:29.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.047+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23381f5fa0 con 0x7f23380c9ca0 2023-10-16T12:49:29.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.048+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23381f64c0 con 0x7f23380c9ca0 2023-10-16T12:49:29.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.049+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f232004d440 con 0x7f23380c9ca0 2023-10-16T12:49:29.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.050+0000 7f232effd640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f231806af90 0x7f231806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:29.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.050+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23200abb60 con 0x7f23380c9ca0 2023-10-16T12:49:29.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.051+0000 7f233659c640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f231806af90 0x7f231806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:29.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.052+0000 7f233659c640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f231806af90 0x7f231806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2328001560 tx=0x7f2328009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:29.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.055+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23381f64c0 con 0x7f23380c9ca0 2023-10-16T12:49:29.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.282+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f23381f64c0 con 0x7f23380c9ca0 2023-10-16T12:49:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.790+0000 7f232effd640 1 -- 172.21.15.60:0/1462962714 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f23381f64c0 con 0x7f23380c9ca0 2023-10-16T12:49:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.793+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f231806af90 msgr2=0x7f231806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.793+0000 7f233cc09640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f231806af90 0x7f231806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2328001560 tx=0x7f2328009040 comp rx=0 tx=0).stop 2023-10-16T12:49:29.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.793+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 msgr2=0x7f233815cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.793+0000 7f233cc09640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 0x7f233815cad0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f23200098c0 tx=0x7f23200046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 shutdown_connections 2023-10-16T12:49:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f231806af90 0x7f231806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f233815ad70 0x7f233815d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23380c9ca0 0x7f233815cad0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 --2- 172.21.15.60:0/1462962714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f233806a4b0 0x7f2338156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 >> 172.21.15.60:0/1462962714 conn(0x7f23380ce9b0 msgr2=0x7f23380cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 shutdown_connections 2023-10-16T12:49:29.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:29.794+0000 7f233cc09640 1 -- 172.21.15.60:0/1462962714 wait complete. 2023-10-16T12:49:29.817 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:49:29.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:30.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.029+0000 7f250c341640 1 Processor -- start 2023-10-16T12:49:30.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.029+0000 7f250c341640 1 -- start start 2023-10-16T12:49:30.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250c341640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f25040ce390 0x7f250415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250c341640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250415ed60 con 0x7f250415aaa0 2023-10-16T12:49:30.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250c341640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25040707e0 con 0x7f250415ae80 2023-10-16T12:49:30.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250c341640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250406b810 con 0x7f25040ce390 2023-10-16T12:49:30.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250a8de640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f25040ce390 0x7f250415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250a8de640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f25040ce390 0x7f250415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42580/0 (socket says 172.21.15.60:42580) 2023-10-16T12:49:30.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250a8de640 1 -- 172.21.15.60:0/3422433342 learned_addr learned my addr 172.21.15.60:0/3422433342 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:30.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.031+0000 7f250a8de640 1 -- 172.21.15.60:0/3422433342 >> v1:172.21.15.144:6789/0 conn(0x7f250415ae80 legacy=0x7f2504070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:30.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.032+0000 7f250a8de640 1 -- 172.21.15.60:0/3422433342 >> v1:172.21.15.60:6789/0 conn(0x7f250415aaa0 legacy=0x7f25040cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.032+0000 7f250a8de640 1 -- 172.21.15.60:0/3422433342 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250415ef90 con 0x7f25040ce390 2023-10-16T12:49:30.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.032+0000 7f25090db640 1 -- 172.21.15.60:0/3422433342 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798239263 0 0) 0x7f250415ed60 con 0x7f250415aaa0 2023-10-16T12:49:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.033+0000 7f250a8de640 1 --2- 172.21.15.60:0/3422433342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25040ce390 0x7f250415e820 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009e50 tx=0x7f24fc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910bf9f38e603b36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.033+0000 7f25090db640 1 -- 172.21.15.60:0/3422433342 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc02cb20 con 0x7f25040ce390 2023-10-16T12:49:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.033+0000 7f25090db640 1 -- 172.21.15.60:0/3422433342 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc02cc80 con 0x7f25040ce390 2023-10-16T12:49:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.033+0000 7f25090db640 1 -- 172.21.15.60:0/3422433342 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0343f0 con 0x7f25040ce390 2023-10-16T12:49:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.033+0000 7f250c341640 1 -- 172.21.15.60:0/3422433342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25040ce390 msgr2=0x7f250415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.033+0000 7f250c341640 1 --2- 172.21.15.60:0/3422433342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25040ce390 0x7f250415e820 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009e50 tx=0x7f24fc02bf80 comp rx=0 tx=0).stop 2023-10-16T12:49:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.034+0000 7f250c341640 1 -- 172.21.15.60:0/3422433342 shutdown_connections 2023-10-16T12:49:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.034+0000 7f250c341640 1 --2- 172.21.15.60:0/3422433342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25040ce390 0x7f250415e820 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.034+0000 7f250c341640 1 -- 172.21.15.60:0/3422433342 >> 172.21.15.60:0/3422433342 conn(0x7f25040cb410 msgr2=0x7f25040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.034+0000 7f250c341640 1 -- 172.21.15.60:0/3422433342 shutdown_connections 2023-10-16T12:49:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.034+0000 7f250c341640 1 -- 172.21.15.60:0/3422433342 wait complete. 2023-10-16T12:49:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.035+0000 7f250c341640 1 Processor -- start 2023-10-16T12:49:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.036+0000 7f250c341640 1 -- start start 2023-10-16T12:49:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.036+0000 7f250c341640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25040ce390 0x7f2504157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.036+0000 7f250c341640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 0x7f250415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250c341640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f250415ae80 0x7f25040ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250c341640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25041584c0 con 0x7f250415aaa0 2023-10-16T12:49:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250c341640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040c91d0 con 0x7f25040ce390 2023-10-16T12:49:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250c341640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25040c9440 con 0x7f250415ae80 2023-10-16T12:49:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 0x7f250415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 0x7f250415dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42594/0 (socket says 172.21.15.60:42594) 2023-10-16T12:49:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 -- 172.21.15.60:0/3830460340 learned_addr learned my addr 172.21.15.60:0/3830460340 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250a0dd640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25040ce390 0x7f2504157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250a8de640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f250415ae80 0x7f25040ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 -- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f250415ae80 msgr2=0x7f25040ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f250415ae80 0x7f25040ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 -- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25040ce390 msgr2=0x7f2504157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25040ce390 0x7f2504157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f25098dc640 1 -- 172.21.15.60:0/3830460340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc009b30 con 0x7f250415aaa0 2023-10-16T12:49:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.037+0000 7f250a8de640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f250415ae80 0x7f25040ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.038+0000 7f25098dc640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 0x7f250415dc00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f24f400b3c0 tx=0x7f24f400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.038+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4011830 con 0x7f250415aaa0 2023-10-16T12:49:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.038+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f4011990 con 0x7f250415aaa0 2023-10-16T12:49:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.038+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4011d40 con 0x7f250415aaa0 2023-10-16T12:49:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.038+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25040c9720 con 0x7f250415aaa0 2023-10-16T12:49:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.038+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25040c9ce0 con 0x7f250415aaa0 2023-10-16T12:49:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.039+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250404e160 con 0x7f250415aaa0 2023-10-16T12:49:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.040+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f24f401b460 con 0x7f250415aaa0 2023-10-16T12:49:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.041+0000 7f24faffd640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f24e806af90 0x7f24e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.041+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24f40892e0 con 0x7f250415aaa0 2023-10-16T12:49:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.042+0000 7f250a0dd640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f24e806af90 0x7f24e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.043+0000 7f250a0dd640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f24e806af90 0x7f24e806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2500000f60 tx=0x7f2500009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.046+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f24f4054270 con 0x7f250415aaa0 2023-10-16T12:49:30.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.283+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25040cdc60 con 0x7f250415aaa0 2023-10-16T12:49:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.284+0000 7f24faffd640 1 -- 172.21.15.60:0/3830460340 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f24f4057890 con 0x7f250415aaa0 2023-10-16T12:49:30.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:30.288 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:49:26.202053+0000","modified":"2023-10-16T12:49:29.284142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2404598903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2404598903},{"type":"v1","addr":"172.21.15.60:6835","nonce":2404598903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:49:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2023-10-16T12:49:30.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.287+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f24e806af90 msgr2=0x7f24e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.287+0000 7f250c341640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f24e806af90 0x7f24e806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2500000f60 tx=0x7f2500009040 comp rx=0 tx=0).stop 2023-10-16T12:49:30.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.287+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 msgr2=0x7f250415dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.287+0000 7f250c341640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 0x7f250415dc00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f24f400b3c0 tx=0x7f24f400b880 comp rx=0 tx=0).stop 2023-10-16T12:49:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 shutdown_connections 2023-10-16T12:49:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f24e806af90 0x7f24e806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f250415ae80 0x7f25040ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f250415aaa0 0x7f250415dc00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 --2- 172.21.15.60:0/3830460340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25040ce390 0x7f2504157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 >> 172.21.15.60:0/3830460340 conn(0x7f25040cb410 msgr2=0x7f25040b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 shutdown_connections 2023-10-16T12:49:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.288+0000 7f250c341640 1 -- 172.21.15.60:0/3830460340 wait complete. 2023-10-16T12:49:30.309 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.537+0000 7fe9d0a2b640 1 Processor -- start 2023-10-16T12:49:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.537+0000 7fe9d0a2b640 1 -- start start 2023-10-16T12:49:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.538+0000 7fe9d0a2b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe9cc0cfc10 0x7fe9cc0d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.538+0000 7fe9d0a2b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9cc0b3e70 0x7fe9cc0d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9d0a2b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc0b3940 con 0x7fe9cc0b3070 2023-10-16T12:49:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9d0a2b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc0b45e0 con 0x7fe9cc0b3e70 2023-10-16T12:49:30.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9d0a2b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc0c7880 con 0x7fe9cc0cfc10 2023-10-16T12:49:30.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9cad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9cc0b3e70 0x7fe9cc0d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9cad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9cc0b3e70 0x7fe9cc0d1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42610/0 (socket says 172.21.15.60:42610) 2023-10-16T12:49:30.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9cad9d640 1 -- 172.21.15.60:0/3847067499 learned_addr learned my addr 172.21.15.60:0/3847067499 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:30.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.539+0000 7fe9c9d9b640 1 --2- 172.21.15.60:0/3847067499 >> v2:172.21.15.144:3300/0 conn(0x7fe9cc0cfc10 0x7fe9cc0d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.540+0000 7fe9cad9d640 1 -- 172.21.15.60:0/3847067499 >> v2:172.21.15.144:3300/0 conn(0x7fe9cc0cfc10 msgr2=0x7fe9cc0d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:30.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.540+0000 7fe9cad9d640 1 --2- 172.21.15.60:0/3847067499 >> v2:172.21.15.144:3300/0 conn(0x7fe9cc0cfc10 0x7fe9cc0d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.540+0000 7fe9cad9d640 1 -- 172.21.15.60:0/3847067499 >> v1:172.21.15.144:6789/0 conn(0x7fe9cc0b3070 legacy=0x7fe9cc0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:30.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.540+0000 7fe9cad9d640 1 -- 172.21.15.60:0/3847067499 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc0b3b90 con 0x7fe9cc0b3e70 2023-10-16T12:49:30.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.541+0000 7fe9cad9d640 1 --2- 172.21.15.60:0/3847067499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc0d1040 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0b58e0 tx=0x7fe9c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abcdfd3012361b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:30.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.541+0000 7fe9c959a640 1 -- 172.21.15.60:0/3847067499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4030860 con 0x7fe9cc0b3e70 2023-10-16T12:49:30.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.541+0000 7fe9c959a640 1 -- 172.21.15.60:0/3847067499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c40309c0 con 0x7fe9cc0b3e70 2023-10-16T12:49:30.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.541+0000 7fe9c959a640 1 -- 172.21.15.60:0/3847067499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4030d50 con 0x7fe9cc0b3e70 2023-10-16T12:49:30.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3847067499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3e70 msgr2=0x7fe9cc0d1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3847067499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc0d1040 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0b58e0 tx=0x7fe9c402b9a0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3847067499 shutdown_connections 2023-10-16T12:49:30.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3847067499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc0d1040 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3847067499 >> v2:172.21.15.144:3300/0 conn(0x7fe9cc0cfc10 0x7fe9cc0d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3847067499 >> 172.21.15.60:0/3847067499 conn(0x7fe9cc0d2d20 msgr2=0x7fe9cc0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:30.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3847067499 shutdown_connections 2023-10-16T12:49:30.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.542+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3847067499 wait complete. 2023-10-16T12:49:30.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.544+0000 7fe9d0a2b640 1 Processor -- start 2023-10-16T12:49:30.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.544+0000 7fe9d0a2b640 1 -- start start 2023-10-16T12:49:30.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9d0a2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 0x7fe9cc0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9d0a2b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc15be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9d0a2b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9cc0cfc10 0x7fe9cc15f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9d0a2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0b0a00 con 0x7fe9cc0b3070 2023-10-16T12:49:30.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9d0a2b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0afdf0 con 0x7fe9cc0b3e70 2023-10-16T12:49:30.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9d0a2b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc0b00d0 con 0x7fe9cc0cfc10 2023-10-16T12:49:30.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9ca59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 0x7fe9cc0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9ca59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 0x7fe9cc0cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42618/0 (socket says 172.21.15.60:42618) 2023-10-16T12:49:30.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9ca59c640 1 -- 172.21.15.60:0/3766376730 learned_addr learned my addr 172.21.15.60:0/3766376730 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:30.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.545+0000 7fe9c9d9b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9cad9d640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9cc0cfc10 0x7fe9cc15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9ca59c640 1 -- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9cc0cfc10 msgr2=0x7fe9cc15f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9ca59c640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9cc0cfc10 0x7fe9cc15f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9ca59c640 1 -- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9cc0b3e70 msgr2=0x7fe9cc15be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9ca59c640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc15be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9ca59c640 1 -- 172.21.15.60:0/3766376730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c40094d0 con 0x7fe9cc0b3070 2023-10-16T12:49:30.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9c9d9b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc15be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:30.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9ca59c640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 0x7fe9cc0cbea0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c7c0 tx=0x7fe9bc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:30.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.546+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc003bc0 con 0x7fe9cc0b3070 2023-10-16T12:49:30.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.547+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc003d40 con 0x7fe9cc0b3070 2023-10-16T12:49:30.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.547+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc0b03b0 con 0x7fe9cc0b3070 2023-10-16T12:49:30.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.547+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00aa90 con 0x7fe9cc0b3070 2023-10-16T12:49:30.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.547+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc0cc3e0 con 0x7fe9cc0b3070 2023-10-16T12:49:30.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.548+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc04e160 con 0x7fe9cc0b3070 2023-10-16T12:49:30.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.549+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe9bc00abf0 con 0x7fe9cc0b3070 2023-10-16T12:49:30.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.550+0000 7fe9c2ffd640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe9ac06afe0 0x7fe9ac06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:30.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.550+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9bc089610 con 0x7fe9cc0b3070 2023-10-16T12:49:30.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.551+0000 7fe9c9d9b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe9ac06afe0 0x7fe9ac06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:30.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.552+0000 7fe9c9d9b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe9ac06afe0 0x7fe9ac06d4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4006fd0 tx=0x7fe9b4008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:30.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.554+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9bc054620 con 0x7fe9cc0b3070 2023-10-16T12:49:30.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.756+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9cc1e5d30 con 0x7fe9cc0b3070 2023-10-16T12:49:30.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.759+0000 7fe9c2ffd640 1 -- 172.21.15.60:0/3766376730 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16553 (secure 0 0 0) 0x7fe9bc05e020 con 0x7fe9cc0b3070 2023-10-16T12:49:30.762 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:30.762 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":86,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:27.728805+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:30.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe9ac06afe0 msgr2=0x7fe9ac06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe9ac06afe0 0x7fe9ac06d4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4006fd0 tx=0x7fe9b4008040 comp rx=0 tx=0).stop 2023-10-16T12:49:30.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 msgr2=0x7fe9cc0cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:30.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 0x7fe9cc0cbea0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c7c0 tx=0x7fe9bc00cc80 comp rx=0 tx=0).stop 2023-10-16T12:49:30.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 shutdown_connections 2023-10-16T12:49:30.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fe9ac06afe0 0x7fe9ac06d4a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9cc0cfc10 0x7fe9cc15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9cc0b3e70 0x7fe9cc15be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 --2- 172.21.15.60:0/3766376730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9cc0b3070 0x7fe9cc0cbea0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:30.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 >> 172.21.15.60:0/3766376730 conn(0x7fe9cc0d2d20 msgr2=0x7fe9cc0d3100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:30.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 shutdown_connections 2023-10-16T12:49:30.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:30.762+0000 7fe9d0a2b640 1 -- 172.21.15.60:0/3766376730 wait complete. 2023-10-16T12:49:30.775 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T12:49:30.773+0000 7f63cd168640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:49:30.784 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:49:30.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:49:31.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.009+0000 7f720da59640 1 Processor -- start 2023-10-16T12:49:31.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.009+0000 7f720da59640 1 -- start start 2023-10-16T12:49:31.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.010+0000 7f720da59640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f720815cbe0 0x7f72080d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.010+0000 7f720da59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72080b4c80 0x7f72080b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.010+0000 7f720da59640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72080d15a0 con 0x7f720815cfc0 2023-10-16T12:49:31.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.010+0000 7f720da59640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72080d1880 con 0x7f720815cbe0 2023-10-16T12:49:31.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.010+0000 7f720da59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080d1b60 con 0x7f72080b4c80 2023-10-16T12:49:31.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720ca57640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f720815cbe0 0x7f72080d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f7207fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f720815cfc0 0x7f72080b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50274/0 (socket says 172.21.15.60:50274) 2023-10-16T12:49:31.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f7207fff640 1 -- 172.21.15.60:0/3415599632 learned_addr learned my addr 172.21.15.60:0/3415599632 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:31.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720d258640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72080b4c80 0x7f72080b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720ca57640 1 -- 172.21.15.60:0/3415599632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72080b4c80 msgr2=0x7f72080b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:31.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720ca57640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72080b4c80 0x7f72080b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720ca57640 1 -- 172.21.15.60:0/3415599632 >> v1:172.21.15.60:6789/0 conn(0x7f720815cfc0 legacy=0x7f72080b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720ca57640 1 -- 172.21.15.60:0/3415599632 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72080d1e40 con 0x7f720815cbe0 2023-10-16T12:49:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.011+0000 7f720d258640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72080b4c80 0x7f72080b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.012+0000 7f720ca57640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720815cbe0 0x7f72080d0b50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009d90 tx=0x7f71f802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23d33f6d74b2e6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:31.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.013+0000 7f72077fe640 1 -- 172.21.15.60:0/3415599632 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f802ccf0 con 0x7f720815cbe0 2023-10-16T12:49:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.013+0000 7f72077fe640 1 -- 172.21.15.60:0/3415599632 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f802ce50 con 0x7f720815cbe0 2023-10-16T12:49:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.013+0000 7f72077fe640 1 -- 172.21.15.60:0/3415599632 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8034660 con 0x7f720815cbe0 2023-10-16T12:49:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.013+0000 7f720da59640 1 -- 172.21.15.60:0/3415599632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720815cbe0 msgr2=0x7f72080d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.013+0000 7f720da59640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720815cbe0 0x7f72080d0b50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009d90 tx=0x7f71f802bf40 comp rx=0 tx=0).stop 2023-10-16T12:49:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.014+0000 7f720da59640 1 -- 172.21.15.60:0/3415599632 shutdown_connections 2023-10-16T12:49:31.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.014+0000 7f720da59640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f72080b4c80 0x7f72080b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.014+0000 7f720da59640 1 --2- 172.21.15.60:0/3415599632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720815cbe0 0x7f72080d0b50 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.014+0000 7f720da59640 1 -- 172.21.15.60:0/3415599632 >> 172.21.15.60:0/3415599632 conn(0x7f72080cf570 msgr2=0x7f72080cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:31.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.014+0000 7f720da59640 1 -- 172.21.15.60:0/3415599632 shutdown_connections 2023-10-16T12:49:31.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.014+0000 7f720da59640 1 -- 172.21.15.60:0/3415599632 wait complete. 2023-10-16T12:49:31.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.015+0000 7f720da59640 1 Processor -- start 2023-10-16T12:49:31.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.016+0000 7f720da59640 1 -- start start 2023-10-16T12:49:31.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720da59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 0x7f720815ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720da59640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f720815cbe0 0x7f7208145380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720da59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f720815cfc0 0x7f7208151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720da59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72081433f0 con 0x7f72080b4c80 2023-10-16T12:49:31.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720da59640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720815f660 con 0x7f720815cbe0 2023-10-16T12:49:31.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720da59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720815f940 con 0x7f720815cfc0 2023-10-16T12:49:31.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 0x7f720815ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 0x7f720815ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42642/0 (socket says 172.21.15.60:42642) 2023-10-16T12:49:31.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 -- 172.21.15.60:0/723512527 learned_addr learned my addr 172.21.15.60:0/723512527 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:31.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 -- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f720815cfc0 msgr2=0x7f7208151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f720815cfc0 0x7f7208151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 -- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f720815cbe0 msgr2=0x7f7208145380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f720815cbe0 0x7f7208145380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.017+0000 7f720ca57640 1 -- 172.21.15.60:0/723512527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f8009a70 con 0x7f72080b4c80 2023-10-16T12:49:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.018+0000 7f720ca57640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 0x7f720815ee30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f71f802dfd0 tx=0x7f71f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.018+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f803d600 con 0x7f72080b4c80 2023-10-16T12:49:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.018+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f803d7a0 con 0x7f72080b4c80 2023-10-16T12:49:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.018+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f803db50 con 0x7f72080b4c80 2023-10-16T12:49:31.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.018+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720815fc20 con 0x7f72080b4c80 2023-10-16T12:49:31.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.018+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720815f370 con 0x7f72080b4c80 2023-10-16T12:49:31.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.019+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720804e9b0 con 0x7f72080b4c80 2023-10-16T12:49:31.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.020+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f71f803b040 con 0x7f72080b4c80 2023-10-16T12:49:31.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.021+0000 7f72057fa640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f71e806af90 0x7f71e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.021+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71f80abc20 con 0x7f72080b4c80 2023-10-16T12:49:31.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.022+0000 7f7207fff640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f71e806af90 0x7f71e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.023+0000 7f7207fff640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f71e806af90 0x7f71e806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72081f60a0 tx=0x7f71f0007a40 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:31.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.026+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f71f8076b20 con 0x7f72080b4c80 2023-10-16T12:49:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.218+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 --> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72080d2120 con 0x7f71e806af90 2023-10-16T12:49:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.248+0000 7f72057fa640 1 -- 172.21.15.60:0/723512527 <== mgr.5186 v2:172.21.15.144:6832/2872949001 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286648 (secure 0 0 0) 0x7f71f00070d0 con 0x7f71e806af90 2023-10-16T12:49:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:31.257 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:49:31.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.254+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f71e806af90 msgr2=0x7f71e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.254+0000 7f720da59640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f71e806af90 0x7f71e806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72081f60a0 tx=0x7f71f0007a40 comp rx=0 tx=0).stop 2023-10-16T12:49:31.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.254+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 msgr2=0x7f720815ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.254+0000 7f720da59640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 0x7f720815ee30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f71f802dfd0 tx=0x7f71f8004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:31.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 shutdown_connections 2023-10-16T12:49:31.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f71e806af90 0x7f71e806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f720815cfc0 0x7f7208151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f720815cbe0 0x7f7208145380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 --2- 172.21.15.60:0/723512527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72080b4c80 0x7f720815ee30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 >> 172.21.15.60:0/723512527 conn(0x7f72080cf570 msgr2=0x7f72080b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 shutdown_connections 2023-10-16T12:49:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.255+0000 7f720da59640 1 -- 172.21.15.60:0/723512527 wait complete. 2023-10-16T12:49:31.277 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T12:49:30.990083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":340576,"kb_used_data":9260,"kb_used_omap":12,"kb_used_meta":331187,"kb_avail":754634144,"statfs":{"total":773094113280,"available":772745363456,"internally_reserved":0,"allocated":9482240,"data_stored":5098064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":339136056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001973"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.782184+0000","last_change":"2023-10-16T12:49:26.779800+0000","last_active":"2023-10-16T12:49:26.782184+0000","last_peered":"2023-10-16T12:49:26.782184+0000","last_clean":"2023-10-16T12:49:26.782184+0000","last_became_active":"2023-10-16T12:49:26.779504+0000","last_became_peered":"2023-10-16T12:49:26.779504+0000","last_unstale":"2023-10-16T12:49:26.782184+0000","last_undegraded":"2023-10-16T12:49:26.782184+0000","last_fullsized":"2023-10-16T12:49:26.782184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:46.019798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.782207+0000","last_change":"2023-10-16T12:49:26.774328+0000","last_active":"2023-10-16T12:49:26.782207+0000","last_peered":"2023-10-16T12:49:26.782207+0000","last_clean":"2023-10-16T12:49:26.782207+0000","last_became_active":"2023-10-16T12:49:26.771401+0000","last_became_peered":"2023-10-16T12:49:26.771401+0000","last_unstale":"2023-10-16T12:49:26.782207+0000","last_undegraded":"2023-10-16T12:49:26.782207+0000","last_fullsized":"2023-10-16T12:49:26.782207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.315108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763391+0000","last_change":"2023-10-16T12:49:26.758417+0000","last_active":"2023-10-16T12:49:26.763391+0000","last_peered":"2023-10-16T12:49:26.763391+0000","last_clean":"2023-10-16T12:49:26.763391+0000","last_became_active":"2023-10-16T12:49:26.757948+0000","last_became_peered":"2023-10-16T12:49:26.757948+0000","last_unstale":"2023-10-16T12:49:26.763391+0000","last_undegraded":"2023-10-16T12:49:26.763391+0000","last_fullsized":"2023-10-16T12:49:26.763391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:48.142634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740156+0000","last_change":"2023-10-16T12:49:26.769606+0000","last_active":"2023-10-16T12:49:27.740156+0000","last_peered":"2023-10-16T12:49:27.740156+0000","last_clean":"2023-10-16T12:49:27.740156+0000","last_became_active":"2023-10-16T12:49:26.768287+0000","last_became_peered":"2023-10-16T12:49:26.768287+0000","last_unstale":"2023-10-16T12:49:27.740156+0000","last_undegraded":"2023-10-16T12:49:27.740156+0000","last_fullsized":"2023-10-16T12:49:27.740156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:51.358206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.770471+0000","last_change":"2023-10-16T12:49:26.769830+0000","last_active":"2023-10-16T12:49:26.770471+0000","last_peered":"2023-10-16T12:49:26.770471+0000","last_clean":"2023-10-16T12:49:26.770471+0000","last_became_active":"2023-10-16T12:49:26.768941+0000","last_became_peered":"2023-10-16T12:49:26.768941+0000","last_unstale":"2023-10-16T12:49:26.770471+0000","last_undegraded":"2023-10-16T12:49:26.770471+0000","last_fullsized":"2023-10-16T12:49:26.770471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:46.679516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.773472+0000","last_change":"2023-10-16T12:49:26.765676+0000","last_active":"2023-10-16T12:49:26.773472+0000","last_peered":"2023-10-16T12:49:26.773472+0000","last_clean":"2023-10-16T12:49:26.773472+0000","last_became_active":"2023-10-16T12:49:26.765356+0000","last_became_peered":"2023-10-16T12:49:26.765356+0000","last_unstale":"2023-10-16T12:49:26.773472+0000","last_undegraded":"2023-10-16T12:49:26.773472+0000","last_fullsized":"2023-10-16T12:49:26.773472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:15.408002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.737941+0000","last_change":"2023-10-16T12:49:26.768164+0000","last_active":"2023-10-16T12:49:27.737941+0000","last_peered":"2023-10-16T12:49:27.737941+0000","last_clean":"2023-10-16T12:49:27.737941+0000","last_became_active":"2023-10-16T12:49:26.767840+0000","last_became_peered":"2023-10-16T12:49:26.767840+0000","last_unstale":"2023-10-16T12:49:27.737941+0000","last_undegraded":"2023-10-16T12:49:27.737941+0000","last_fullsized":"2023-10-16T12:49:27.737941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:25.123970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.761569+0000","last_change":"2023-10-16T12:49:26.757776+0000","last_active":"2023-10-16T12:49:26.761569+0000","last_peered":"2023-10-16T12:49:26.761569+0000","last_clean":"2023-10-16T12:49:26.761569+0000","last_became_active":"2023-10-16T12:49:26.757439+0000","last_became_peered":"2023-10-16T12:49:26.757439+0000","last_unstale":"2023-10-16T12:49:26.761569+0000","last_undegraded":"2023-10-16T12:49:26.761569+0000","last_fullsized":"2023-10-16T12:49:26.761569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:41.176774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739855+0000","last_change":"2023-10-16T12:49:26.761605+0000","last_active":"2023-10-16T12:49:27.739855+0000","last_peered":"2023-10-16T12:49:27.739855+0000","last_clean":"2023-10-16T12:49:27.739855+0000","last_became_active":"2023-10-16T12:49:26.761185+0000","last_became_peered":"2023-10-16T12:49:26.761185+0000","last_unstale":"2023-10-16T12:49:27.739855+0000","last_undegraded":"2023-10-16T12:49:27.739855+0000","last_fullsized":"2023-10-16T12:49:27.739855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:09.616899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734477+0000","last_change":"2023-10-16T12:49:26.768282+0000","last_active":"2023-10-16T12:49:27.734477+0000","last_peered":"2023-10-16T12:49:27.734477+0000","last_clean":"2023-10-16T12:49:27.734477+0000","last_became_active":"2023-10-16T12:49:26.767961+0000","last_became_peered":"2023-10-16T12:49:26.767961+0000","last_unstale":"2023-10-16T12:49:27.734477+0000","last_undegraded":"2023-10-16T12:49:27.734477+0000","last_fullsized":"2023-10-16T12:49:27.734477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:53.860402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734324+0000","last_change":"2023-10-16T12:49:26.784401+0000","last_active":"2023-10-16T12:49:27.734324+0000","last_peered":"2023-10-16T12:49:27.734324+0000","last_clean":"2023-10-16T12:49:27.734324+0000","last_became_active":"2023-10-16T12:49:26.784078+0000","last_became_peered":"2023-10-16T12:49:26.784078+0000","last_unstale":"2023-10-16T12:49:27.734324+0000","last_undegraded":"2023-10-16T12:49:27.734324+0000","last_fullsized":"2023-10-16T12:49:27.734324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:58:59.268866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740529+0000","last_change":"2023-10-16T12:49:26.774983+0000","last_active":"2023-10-16T12:49:27.740529+0000","last_peered":"2023-10-16T12:49:27.740529+0000","last_clean":"2023-10-16T12:49:27.740529+0000","last_became_active":"2023-10-16T12:49:26.774563+0000","last_became_peered":"2023-10-16T12:49:26.774563+0000","last_unstale":"2023-10-16T12:49:27.740529+0000","last_undegraded":"2023-10-16T12:49:27.740529+0000","last_fullsized":"2023-10-16T12:49:27.740529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:48.441349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.768552+0000","last_change":"2023-10-16T12:49:26.766950+0000","last_active":"2023-10-16T12:49:26.768552+0000","last_peered":"2023-10-16T12:49:26.768552+0000","last_clean":"2023-10-16T12:49:26.768552+0000","last_became_active":"2023-10-16T12:49:26.766058+0000","last_became_peered":"2023-10-16T12:49:26.766058+0000","last_unstale":"2023-10-16T12:49:26.768552+0000","last_undegraded":"2023-10-16T12:49:26.768552+0000","last_fullsized":"2023-10-16T12:49:26.768552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:29.168043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740233+0000","last_change":"2023-10-16T12:49:26.747953+0000","last_active":"2023-10-16T12:49:27.740233+0000","last_peered":"2023-10-16T12:49:27.740233+0000","last_clean":"2023-10-16T12:49:27.740233+0000","last_became_active":"2023-10-16T12:49:26.747200+0000","last_became_peered":"2023-10-16T12:49:26.747200+0000","last_unstale":"2023-10-16T12:49:27.740233+0000","last_undegraded":"2023-10-16T12:49:27.740233+0000","last_fullsized":"2023-10-16T12:49:27.740233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:43.467438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740304+0000","last_change":"2023-10-16T12:49:26.757837+0000","last_active":"2023-10-16T12:49:27.740304+0000","last_peered":"2023-10-16T12:49:27.740304+0000","last_clean":"2023-10-16T12:49:27.740304+0000","last_became_active":"2023-10-16T12:49:26.757344+0000","last_became_peered":"2023-10-16T12:49:26.757344+0000","last_unstale":"2023-10-16T12:49:27.740304+0000","last_undegraded":"2023-10-16T12:49:27.740304+0000","last_fullsized":"2023-10-16T12:49:27.740304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:41.015857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779510+0000","last_change":"2023-10-16T12:49:26.775293+0000","last_active":"2023-10-16T12:49:26.779510+0000","last_peered":"2023-10-16T12:49:26.779510+0000","last_clean":"2023-10-16T12:49:26.779510+0000","last_became_active":"2023-10-16T12:49:26.772954+0000","last_became_peered":"2023-10-16T12:49:26.772954+0000","last_unstale":"2023-10-16T12:49:26.779510+0000","last_undegraded":"2023-10-16T12:49:26.779510+0000","last_fullsized":"2023-10-16T12:49:26.779510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.822248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776202+0000","last_change":"2023-10-16T12:49:26.770701+0000","last_active":"2023-10-16T12:49:26.776202+0000","last_peered":"2023-10-16T12:49:26.776202+0000","last_clean":"2023-10-16T12:49:26.776202+0000","last_became_active":"2023-10-16T12:49:26.769619+0000","last_became_peered":"2023-10-16T12:49:26.769619+0000","last_unstale":"2023-10-16T12:49:26.776202+0000","last_undegraded":"2023-10-16T12:49:26.776202+0000","last_fullsized":"2023-10-16T12:49:26.776202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:17.639724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.771894+0000","last_change":"2023-10-16T12:49:26.764518+0000","last_active":"2023-10-16T12:49:26.771894+0000","last_peered":"2023-10-16T12:49:26.771894+0000","last_clean":"2023-10-16T12:49:26.771894+0000","last_became_active":"2023-10-16T12:49:26.764217+0000","last_became_peered":"2023-10-16T12:49:26.764217+0000","last_unstale":"2023-10-16T12:49:26.771894+0000","last_undegraded":"2023-10-16T12:49:26.771894+0000","last_fullsized":"2023-10-16T12:49:26.771894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:51.358206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.788018+0000","last_change":"2023-10-16T12:49:26.783981+0000","last_active":"2023-10-16T12:49:26.788018+0000","last_peered":"2023-10-16T12:49:26.788018+0000","last_clean":"2023-10-16T12:49:26.788018+0000","last_became_active":"2023-10-16T12:49:26.783696+0000","last_became_peered":"2023-10-16T12:49:26.783696+0000","last_unstale":"2023-10-16T12:49:26.788018+0000","last_undegraded":"2023-10-16T12:49:26.788018+0000","last_fullsized":"2023-10-16T12:49:26.788018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:47.251171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.785467+0000","last_change":"2023-10-16T12:49:26.780743+0000","last_active":"2023-10-16T12:49:26.785467+0000","last_peered":"2023-10-16T12:49:26.785467+0000","last_clean":"2023-10-16T12:49:26.785467+0000","last_became_active":"2023-10-16T12:49:26.780418+0000","last_became_peered":"2023-10-16T12:49:26.780418+0000","last_unstale":"2023-10-16T12:49:26.785467+0000","last_undegraded":"2023-10-16T12:49:26.785467+0000","last_fullsized":"2023-10-16T12:49:26.785467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:07.127743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.773205+0000","last_change":"2023-10-16T12:49:26.771376+0000","last_active":"2023-10-16T12:49:26.773205+0000","last_peered":"2023-10-16T12:49:26.773205+0000","last_clean":"2023-10-16T12:49:26.773205+0000","last_became_active":"2023-10-16T12:49:26.771061+0000","last_became_peered":"2023-10-16T12:49:26.771061+0000","last_unstale":"2023-10-16T12:49:26.773205+0000","last_undegraded":"2023-10-16T12:49:26.773205+0000","last_fullsized":"2023-10-16T12:49:26.773205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:48.441349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776109+0000","last_change":"2023-10-16T12:49:26.770161+0000","last_active":"2023-10-16T12:49:26.776109+0000","last_peered":"2023-10-16T12:49:26.776109+0000","last_clean":"2023-10-16T12:49:26.776109+0000","last_became_active":"2023-10-16T12:49:26.768059+0000","last_became_peered":"2023-10-16T12:49:26.768059+0000","last_unstale":"2023-10-16T12:49:26.776109+0000","last_undegraded":"2023-10-16T12:49:26.776109+0000","last_fullsized":"2023-10-16T12:49:26.776109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:26.259122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763672+0000","last_change":"2023-10-16T12:49:26.759428+0000","last_active":"2023-10-16T12:49:26.763672+0000","last_peered":"2023-10-16T12:49:26.763672+0000","last_clean":"2023-10-16T12:49:26.763672+0000","last_became_active":"2023-10-16T12:49:26.758703+0000","last_became_peered":"2023-10-16T12:49:26.758703+0000","last_unstale":"2023-10-16T12:49:26.763672+0000","last_undegraded":"2023-10-16T12:49:26.763672+0000","last_fullsized":"2023-10-16T12:49:26.763672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:13.767009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.770400+0000","last_change":"2023-10-16T12:49:26.768576+0000","last_active":"2023-10-16T12:49:26.770400+0000","last_peered":"2023-10-16T12:49:26.770400+0000","last_clean":"2023-10-16T12:49:26.770400+0000","last_became_active":"2023-10-16T12:49:26.766846+0000","last_became_peered":"2023-10-16T12:49:26.766846+0000","last_unstale":"2023-10-16T12:49:26.770400+0000","last_undegraded":"2023-10-16T12:49:26.770400+0000","last_fullsized":"2023-10-16T12:49:26.770400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:38.979498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.764786+0000","last_change":"2023-10-16T12:49:26.761191+0000","last_active":"2023-10-16T12:49:26.764786+0000","last_peered":"2023-10-16T12:49:26.764786+0000","last_clean":"2023-10-16T12:49:26.764786+0000","last_became_active":"2023-10-16T12:49:26.758480+0000","last_became_peered":"2023-10-16T12:49:26.758480+0000","last_unstale":"2023-10-16T12:49:26.764786+0000","last_undegraded":"2023-10-16T12:49:26.764786+0000","last_fullsized":"2023-10-16T12:49:26.764786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:59.443397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741456+0000","last_change":"2023-10-16T12:49:26.772360+0000","last_active":"2023-10-16T12:49:27.741456+0000","last_peered":"2023-10-16T12:49:27.741456+0000","last_clean":"2023-10-16T12:49:27.741456+0000","last_became_active":"2023-10-16T12:49:26.772036+0000","last_became_peered":"2023-10-16T12:49:26.772036+0000","last_unstale":"2023-10-16T12:49:27.741456+0000","last_undegraded":"2023-10-16T12:49:27.741456+0000","last_fullsized":"2023-10-16T12:49:27.741456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:48.142634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779040+0000","last_change":"2023-10-16T12:49:26.768467+0000","last_active":"2023-10-16T12:49:26.779040+0000","last_peered":"2023-10-16T12:49:26.779040+0000","last_clean":"2023-10-16T12:49:26.779040+0000","last_became_active":"2023-10-16T12:49:26.767054+0000","last_became_peered":"2023-10-16T12:49:26.767054+0000","last_unstale":"2023-10-16T12:49:26.779040+0000","last_undegraded":"2023-10-16T12:49:26.779040+0000","last_fullsized":"2023-10-16T12:49:26.779040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:45.675716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.781001+0000","last_change":"2023-10-16T12:49:26.772981+0000","last_active":"2023-10-16T12:49:26.781001+0000","last_peered":"2023-10-16T12:49:26.781001+0000","last_clean":"2023-10-16T12:49:26.781001+0000","last_became_active":"2023-10-16T12:49:26.772716+0000","last_became_peered":"2023-10-16T12:49:26.772716+0000","last_unstale":"2023-10-16T12:49:26.781001+0000","last_undegraded":"2023-10-16T12:49:26.781001+0000","last_fullsized":"2023-10-16T12:49:26.781001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:00:32.951148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.782028+0000","last_change":"2023-10-16T12:49:26.774504+0000","last_active":"2023-10-16T12:49:26.782028+0000","last_peered":"2023-10-16T12:49:26.782028+0000","last_clean":"2023-10-16T12:49:26.782028+0000","last_became_active":"2023-10-16T12:49:26.774147+0000","last_became_peered":"2023-10-16T12:49:26.774147+0000","last_unstale":"2023-10-16T12:49:26.782028+0000","last_undegraded":"2023-10-16T12:49:26.782028+0000","last_fullsized":"2023-10-16T12:49:26.782028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:39.748496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.759178+0000","last_change":"2023-10-16T12:49:26.753094+0000","last_active":"2023-10-16T12:49:26.759178+0000","last_peered":"2023-10-16T12:49:26.759178+0000","last_clean":"2023-10-16T12:49:26.759178+0000","last_became_active":"2023-10-16T12:49:26.750269+0000","last_became_peered":"2023-10-16T12:49:26.750269+0000","last_unstale":"2023-10-16T12:49:26.759178+0000","last_undegraded":"2023-10-16T12:49:26.759178+0000","last_fullsized":"2023-10-16T12:49:26.759178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:28.460635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.765145+0000","last_change":"2023-10-16T12:49:26.759075+0000","last_active":"2023-10-16T12:49:26.765145+0000","last_peered":"2023-10-16T12:49:26.765145+0000","last_clean":"2023-10-16T12:49:26.765145+0000","last_became_active":"2023-10-16T12:49:26.758711+0000","last_became_peered":"2023-10-16T12:49:26.758711+0000","last_unstale":"2023-10-16T12:49:26.765145+0000","last_undegraded":"2023-10-16T12:49:26.765145+0000","last_fullsized":"2023-10-16T12:49:26.765145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:38.177995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.766121+0000","last_change":"2023-10-16T12:49:26.760582+0000","last_active":"2023-10-16T12:49:26.766121+0000","last_peered":"2023-10-16T12:49:26.766121+0000","last_clean":"2023-10-16T12:49:26.766121+0000","last_became_active":"2023-10-16T12:49:26.760130+0000","last_became_peered":"2023-10-16T12:49:26.760130+0000","last_unstale":"2023-10-16T12:49:26.766121+0000","last_undegraded":"2023-10-16T12:49:26.766121+0000","last_fullsized":"2023-10-16T12:49:26.766121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:15.191125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776427+0000","last_change":"2023-10-16T12:49:26.765739+0000","last_active":"2023-10-16T12:49:26.776427+0000","last_peered":"2023-10-16T12:49:26.776427+0000","last_clean":"2023-10-16T12:49:26.776427+0000","last_became_active":"2023-10-16T12:49:26.765420+0000","last_became_peered":"2023-10-16T12:49:26.765420+0000","last_unstale":"2023-10-16T12:49:26.776427+0000","last_undegraded":"2023-10-16T12:49:26.776427+0000","last_fullsized":"2023-10-16T12:49:26.776427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:09.290033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"85'4","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791868+0000","last_change":"2023-10-16T12:49:25.738940+0000","last_active":"2023-10-16T12:49:26.791868+0000","last_peered":"2023-10-16T12:49:26.791868+0000","last_clean":"2023-10-16T12:49:26.791868+0000","last_became_active":"2023-10-16T12:49:25.738695+0000","last_became_peered":"2023-10-16T12:49:25.738695+0000","last_unstale":"2023-10-16T12:49:26.791868+0000","last_undegraded":"2023-10-16T12:49:26.791868+0000","last_fullsized":"2023-10-16T12:49:26.791868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:47.922516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734608+0000","last_change":"2023-10-16T12:49:26.763182+0000","last_active":"2023-10-16T12:49:27.734608+0000","last_peered":"2023-10-16T12:49:27.734608+0000","last_clean":"2023-10-16T12:49:27.734608+0000","last_became_active":"2023-10-16T12:49:26.762890+0000","last_became_peered":"2023-10-16T12:49:26.762890+0000","last_unstale":"2023-10-16T12:49:27.734608+0000","last_undegraded":"2023-10-16T12:49:27.734608+0000","last_fullsized":"2023-10-16T12:49:27.734608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:26.285968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.790766+0000","last_change":"2023-10-16T12:49:25.729100+0000","last_active":"2023-10-16T12:49:26.790766+0000","last_peered":"2023-10-16T12:49:26.790766+0000","last_clean":"2023-10-16T12:49:26.790766+0000","last_became_active":"2023-10-16T12:49:25.728822+0000","last_became_peered":"2023-10-16T12:49:25.728822+0000","last_unstale":"2023-10-16T12:49:26.790766+0000","last_undegraded":"2023-10-16T12:49:26.790766+0000","last_fullsized":"2023-10-16T12:49:26.790766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:44.209674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.766531+0000","last_change":"2023-10-16T12:49:26.763254+0000","last_active":"2023-10-16T12:49:26.766531+0000","last_peered":"2023-10-16T12:49:26.766531+0000","last_clean":"2023-10-16T12:49:26.766531+0000","last_became_active":"2023-10-16T12:49:26.762909+0000","last_became_peered":"2023-10-16T12:49:26.762909+0000","last_unstale":"2023-10-16T12:49:26.766531+0000","last_undegraded":"2023-10-16T12:49:26.766531+0000","last_fullsized":"2023-10-16T12:49:26.766531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:46.592546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.789854+0000","last_change":"2023-10-16T12:49:25.727459+0000","last_active":"2023-10-16T12:49:26.789854+0000","last_peered":"2023-10-16T12:49:26.789854+0000","last_clean":"2023-10-16T12:49:26.789854+0000","last_became_active":"2023-10-16T12:49:25.725586+0000","last_became_peered":"2023-10-16T12:49:25.725586+0000","last_unstale":"2023-10-16T12:49:26.789854+0000","last_undegraded":"2023-10-16T12:49:26.789854+0000","last_fullsized":"2023-10-16T12:49:26.789854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:55.417552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.778881+0000","last_change":"2023-10-16T12:49:26.774882+0000","last_active":"2023-10-16T12:49:26.778881+0000","last_peered":"2023-10-16T12:49:26.778881+0000","last_clean":"2023-10-16T12:49:26.778881+0000","last_became_active":"2023-10-16T12:49:26.769371+0000","last_became_peered":"2023-10-16T12:49:26.769371+0000","last_unstale":"2023-10-16T12:49:26.778881+0000","last_undegraded":"2023-10-16T12:49:26.778881+0000","last_fullsized":"2023-10-16T12:49:26.778881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:07.389207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"85'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740842+0000","last_change":"2023-10-16T12:49:25.742509+0000","last_active":"2023-10-16T12:49:27.740842+0000","last_peered":"2023-10-16T12:49:27.740842+0000","last_clean":"2023-10-16T12:49:27.740842+0000","last_became_active":"2023-10-16T12:49:25.742140+0000","last_became_peered":"2023-10-16T12:49:25.742140+0000","last_unstale":"2023-10-16T12:49:27.740842+0000","last_undegraded":"2023-10-16T12:49:27.740842+0000","last_fullsized":"2023-10-16T12:49:27.740842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:58.098283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.780328+0000","last_change":"2023-10-16T12:49:26.772637+0000","last_active":"2023-10-16T12:49:26.780328+0000","last_peered":"2023-10-16T12:49:26.780328+0000","last_clean":"2023-10-16T12:49:26.780328+0000","last_became_active":"2023-10-16T12:49:26.770770+0000","last_became_peered":"2023-10-16T12:49:26.770770+0000","last_unstale":"2023-10-16T12:49:26.780328+0000","last_undegraded":"2023-10-16T12:49:26.780328+0000","last_fullsized":"2023-10-16T12:49:26.780328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:48.142634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739047+0000","last_change":"2023-10-16T12:49:25.734734+0000","last_active":"2023-10-16T12:49:27.739047+0000","last_peered":"2023-10-16T12:49:27.739047+0000","last_clean":"2023-10-16T12:49:27.739047+0000","last_became_active":"2023-10-16T12:49:25.734419+0000","last_became_peered":"2023-10-16T12:49:25.734419+0000","last_unstale":"2023-10-16T12:49:27.739047+0000","last_undegraded":"2023-10-16T12:49:27.739047+0000","last_fullsized":"2023-10-16T12:49:27.739047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:21.521408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.790365+0000","last_change":"2023-10-16T12:49:26.782630+0000","last_active":"2023-10-16T12:49:26.790365+0000","last_peered":"2023-10-16T12:49:26.790365+0000","last_clean":"2023-10-16T12:49:26.790365+0000","last_became_active":"2023-10-16T12:49:26.782342+0000","last_became_peered":"2023-10-16T12:49:26.782342+0000","last_unstale":"2023-10-16T12:49:26.790365+0000","last_undegraded":"2023-10-16T12:49:26.790365+0000","last_fullsized":"2023-10-16T12:49:26.790365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:55.184341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.789428+0000","last_change":"2023-10-16T12:49:25.739996+0000","last_active":"2023-10-16T12:49:26.789428+0000","last_peered":"2023-10-16T12:49:26.789428+0000","last_clean":"2023-10-16T12:49:26.789428+0000","last_became_active":"2023-10-16T12:49:25.737324+0000","last_became_peered":"2023-10-16T12:49:25.737324+0000","last_unstale":"2023-10-16T12:49:26.789428+0000","last_undegraded":"2023-10-16T12:49:26.789428+0000","last_fullsized":"2023-10-16T12:49:26.789428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:40.172041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.768769+0000","last_change":"2023-10-16T12:49:26.766836+0000","last_active":"2023-10-16T12:49:26.768769+0000","last_peered":"2023-10-16T12:49:26.768769+0000","last_clean":"2023-10-16T12:49:26.768769+0000","last_became_active":"2023-10-16T12:49:26.766396+0000","last_became_peered":"2023-10-16T12:49:26.766396+0000","last_unstale":"2023-10-16T12:49:26.768769+0000","last_undegraded":"2023-10-16T12:49:26.768769+0000","last_fullsized":"2023-10-16T12:49:26.768769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:41.754039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.738932+0000","last_change":"2023-10-16T12:49:25.738490+0000","last_active":"2023-10-16T12:49:27.738932+0000","last_peered":"2023-10-16T12:49:27.738932+0000","last_clean":"2023-10-16T12:49:27.738932+0000","last_became_active":"2023-10-16T12:49:25.738207+0000","last_became_peered":"2023-10-16T12:49:25.738207+0000","last_unstale":"2023-10-16T12:49:27.738932+0000","last_undegraded":"2023-10-16T12:49:27.738932+0000","last_fullsized":"2023-10-16T12:49:27.738932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:12.277394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776089+0000","last_change":"2023-10-16T12:49:26.770727+0000","last_active":"2023-10-16T12:49:26.776089+0000","last_peered":"2023-10-16T12:49:26.776089+0000","last_clean":"2023-10-16T12:49:26.776089+0000","last_became_active":"2023-10-16T12:49:26.770237+0000","last_became_peered":"2023-10-16T12:49:26.770237+0000","last_unstale":"2023-10-16T12:49:26.776089+0000","last_undegraded":"2023-10-16T12:49:26.776089+0000","last_fullsized":"2023-10-16T12:49:26.776089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:48.441349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739954+0000","last_change":"2023-10-16T12:49:25.725211+0000","last_active":"2023-10-16T12:49:27.739954+0000","last_peered":"2023-10-16T12:49:27.739954+0000","last_clean":"2023-10-16T12:49:27.739954+0000","last_became_active":"2023-10-16T12:49:25.724905+0000","last_became_peered":"2023-10-16T12:49:25.724905+0000","last_unstale":"2023-10-16T12:49:27.739954+0000","last_undegraded":"2023-10-16T12:49:27.739954+0000","last_fullsized":"2023-10-16T12:49:27.739954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:48.520263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739853+0000","last_change":"2023-10-16T12:49:26.764605+0000","last_active":"2023-10-16T12:49:27.739853+0000","last_peered":"2023-10-16T12:49:27.739853+0000","last_clean":"2023-10-16T12:49:27.739853+0000","last_became_active":"2023-10-16T12:49:26.763311+0000","last_became_peered":"2023-10-16T12:49:26.763311+0000","last_unstale":"2023-10-16T12:49:27.739853+0000","last_undegraded":"2023-10-16T12:49:27.739853+0000","last_fullsized":"2023-10-16T12:49:27.739853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:07.738593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.749356+0000","last_change":"2023-10-16T12:49:25.736697+0000","last_active":"2023-10-16T12:49:26.749356+0000","last_peered":"2023-10-16T12:49:26.749356+0000","last_clean":"2023-10-16T12:49:26.749356+0000","last_became_active":"2023-10-16T12:49:25.736409+0000","last_became_peered":"2023-10-16T12:49:25.736409+0000","last_unstale":"2023-10-16T12:49:26.749356+0000","last_undegraded":"2023-10-16T12:49:26.749356+0000","last_fullsized":"2023-10-16T12:49:26.749356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:54.178505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.772356+0000","last_change":"2023-10-16T12:49:26.765659+0000","last_active":"2023-10-16T12:49:26.772356+0000","last_peered":"2023-10-16T12:49:26.772356+0000","last_clean":"2023-10-16T12:49:26.772356+0000","last_became_active":"2023-10-16T12:49:26.765150+0000","last_became_peered":"2023-10-16T12:49:26.765150+0000","last_unstale":"2023-10-16T12:49:26.772356+0000","last_undegraded":"2023-10-16T12:49:26.772356+0000","last_fullsized":"2023-10-16T12:49:26.772356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:17.639724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.761151+0000","last_change":"2023-10-16T12:49:25.736947+0000","last_active":"2023-10-16T12:49:26.761151+0000","last_peered":"2023-10-16T12:49:26.761151+0000","last_clean":"2023-10-16T12:49:26.761151+0000","last_became_active":"2023-10-16T12:49:25.736678+0000","last_became_peered":"2023-10-16T12:49:25.736678+0000","last_unstale":"2023-10-16T12:49:26.761151+0000","last_undegraded":"2023-10-16T12:49:26.761151+0000","last_fullsized":"2023-10-16T12:49:26.761151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:14.911519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733943+0000","last_change":"2023-10-16T12:49:26.762298+0000","last_active":"2023-10-16T12:49:27.733943+0000","last_peered":"2023-10-16T12:49:27.733943+0000","last_clean":"2023-10-16T12:49:27.733943+0000","last_became_active":"2023-10-16T12:49:26.761954+0000","last_became_peered":"2023-10-16T12:49:26.761954+0000","last_unstale":"2023-10-16T12:49:27.733943+0000","last_undegraded":"2023-10-16T12:49:27.733943+0000","last_fullsized":"2023-10-16T12:49:27.733943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:30.628184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"85'3","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.788610+0000","last_change":"2023-10-16T12:49:25.737273+0000","last_active":"2023-10-16T12:49:26.788610+0000","last_peered":"2023-10-16T12:49:26.788610+0000","last_clean":"2023-10-16T12:49:26.788610+0000","last_became_active":"2023-10-16T12:49:25.737008+0000","last_became_peered":"2023-10-16T12:49:25.737008+0000","last_unstale":"2023-10-16T12:49:26.788610+0000","last_undegraded":"2023-10-16T12:49:26.788610+0000","last_fullsized":"2023-10-16T12:49:26.788610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:38.742660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763428+0000","last_change":"2023-10-16T12:49:26.761793+0000","last_active":"2023-10-16T12:49:26.763428+0000","last_peered":"2023-10-16T12:49:26.763428+0000","last_clean":"2023-10-16T12:49:26.763428+0000","last_became_active":"2023-10-16T12:49:26.761093+0000","last_became_peered":"2023-10-16T12:49:26.761093+0000","last_unstale":"2023-10-16T12:49:26.763428+0000","last_undegraded":"2023-10-16T12:49:26.763428+0000","last_fullsized":"2023-10-16T12:49:26.763428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.315108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"85'6","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739449+0000","last_change":"2023-10-16T12:49:25.735817+0000","last_active":"2023-10-16T12:49:27.739449+0000","last_peered":"2023-10-16T12:49:27.739449+0000","last_clean":"2023-10-16T12:49:27.739449+0000","last_became_active":"2023-10-16T12:49:25.735465+0000","last_became_peered":"2023-10-16T12:49:25.735465+0000","last_unstale":"2023-10-16T12:49:27.739449+0000","last_undegraded":"2023-10-16T12:49:27.739449+0000","last_fullsized":"2023-10-16T12:49:27.739449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:58.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739400+0000","last_change":"2023-10-16T12:49:26.769826+0000","last_active":"2023-10-16T12:49:27.739400+0000","last_peered":"2023-10-16T12:49:27.739400+0000","last_clean":"2023-10-16T12:49:27.739400+0000","last_became_active":"2023-10-16T12:49:26.769529+0000","last_became_peered":"2023-10-16T12:49:26.769529+0000","last_unstale":"2023-10-16T12:49:27.739400+0000","last_undegraded":"2023-10-16T12:49:27.739400+0000","last_fullsized":"2023-10-16T12:49:27.739400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:19.089644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791402+0000","last_change":"2023-10-16T12:49:25.733869+0000","last_active":"2023-10-16T12:49:26.791402+0000","last_peered":"2023-10-16T12:49:26.791402+0000","last_clean":"2023-10-16T12:49:26.791402+0000","last_became_active":"2023-10-16T12:49:25.733469+0000","last_became_peered":"2023-10-16T12:49:25.733469+0000","last_unstale":"2023-10-16T12:49:26.791402+0000","last_undegraded":"2023-10-16T12:49:26.791402+0000","last_fullsized":"2023-10-16T12:49:26.791402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:26.495695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.771949+0000","last_change":"2023-10-16T12:49:26.766534+0000","last_active":"2023-10-16T12:49:26.771949+0000","last_peered":"2023-10-16T12:49:26.771949+0000","last_clean":"2023-10-16T12:49:26.771949+0000","last_became_active":"2023-10-16T12:49:26.766248+0000","last_became_peered":"2023-10-16T12:49:26.766248+0000","last_unstale":"2023-10-16T12:49:26.771949+0000","last_undegraded":"2023-10-16T12:49:26.771949+0000","last_fullsized":"2023-10-16T12:49:26.771949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:08.747676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.746976+0000","last_change":"2023-10-16T12:49:25.739833+0000","last_active":"2023-10-16T12:49:26.746976+0000","last_peered":"2023-10-16T12:49:26.746976+0000","last_clean":"2023-10-16T12:49:26.746976+0000","last_became_active":"2023-10-16T12:49:25.737220+0000","last_became_peered":"2023-10-16T12:49:25.737220+0000","last_unstale":"2023-10-16T12:49:26.746976+0000","last_undegraded":"2023-10-16T12:49:26.746976+0000","last_fullsized":"2023-10-16T12:49:26.746976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:42.104908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.737992+0000","last_change":"2023-10-16T12:49:26.751234+0000","last_active":"2023-10-16T12:49:27.737992+0000","last_peered":"2023-10-16T12:49:27.737992+0000","last_clean":"2023-10-16T12:49:27.737992+0000","last_became_active":"2023-10-16T12:49:26.750473+0000","last_became_peered":"2023-10-16T12:49:26.750473+0000","last_unstale":"2023-10-16T12:49:27.737992+0000","last_undegraded":"2023-10-16T12:49:27.737992+0000","last_fullsized":"2023-10-16T12:49:27.737992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:53.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739138+0000","last_change":"2023-10-16T12:49:25.740744+0000","last_active":"2023-10-16T12:49:27.739138+0000","last_peered":"2023-10-16T12:49:27.739138+0000","last_clean":"2023-10-16T12:49:27.739138+0000","last_became_active":"2023-10-16T12:49:25.740308+0000","last_became_peered":"2023-10-16T12:49:25.740308+0000","last_unstale":"2023-10-16T12:49:27.739138+0000","last_undegraded":"2023-10-16T12:49:27.739138+0000","last_fullsized":"2023-10-16T12:49:27.739138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:21.539277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763650+0000","last_change":"2023-10-16T12:49:26.759085+0000","last_active":"2023-10-16T12:49:26.763650+0000","last_peered":"2023-10-16T12:49:26.763650+0000","last_clean":"2023-10-16T12:49:26.763650+0000","last_became_active":"2023-10-16T12:49:26.758138+0000","last_became_peered":"2023-10-16T12:49:26.758138+0000","last_unstale":"2023-10-16T12:49:26.763650+0000","last_undegraded":"2023-10-16T12:49:26.763650+0000","last_fullsized":"2023-10-16T12:49:26.763650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:19.089644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791026+0000","last_change":"2023-10-16T12:49:25.740357+0000","last_active":"2023-10-16T12:49:26.791026+0000","last_peered":"2023-10-16T12:49:26.791026+0000","last_clean":"2023-10-16T12:49:26.791026+0000","last_became_active":"2023-10-16T12:49:25.739455+0000","last_became_peered":"2023-10-16T12:49:25.739455+0000","last_unstale":"2023-10-16T12:49:26.791026+0000","last_undegraded":"2023-10-16T12:49:26.791026+0000","last_fullsized":"2023-10-16T12:49:26.791026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:16.140142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763722+0000","last_change":"2023-10-16T12:49:26.759318+0000","last_active":"2023-10-16T12:49:26.763722+0000","last_peered":"2023-10-16T12:49:26.763722+0000","last_clean":"2023-10-16T12:49:26.763722+0000","last_became_active":"2023-10-16T12:49:26.757244+0000","last_became_peered":"2023-10-16T12:49:26.757244+0000","last_unstale":"2023-10-16T12:49:26.763722+0000","last_undegraded":"2023-10-16T12:49:26.763722+0000","last_fullsized":"2023-10-16T12:49:26.763722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:52.310280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.790360+0000","last_change":"2023-10-16T12:49:25.733778+0000","last_active":"2023-10-16T12:49:26.790360+0000","last_peered":"2023-10-16T12:49:26.790360+0000","last_clean":"2023-10-16T12:49:26.790360+0000","last_became_active":"2023-10-16T12:49:25.733497+0000","last_became_peered":"2023-10-16T12:49:25.733497+0000","last_unstale":"2023-10-16T12:49:26.790360+0000","last_undegraded":"2023-10-16T12:49:26.790360+0000","last_fullsized":"2023-10-16T12:49:26.790360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:04.144571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741065+0000","last_change":"2023-10-16T12:49:26.774187+0000","last_active":"2023-10-16T12:49:27.741065+0000","last_peered":"2023-10-16T12:49:27.741065+0000","last_clean":"2023-10-16T12:49:27.741065+0000","last_became_active":"2023-10-16T12:49:26.773874+0000","last_became_peered":"2023-10-16T12:49:26.773874+0000","last_unstale":"2023-10-16T12:49:27.741065+0000","last_undegraded":"2023-10-16T12:49:27.741065+0000","last_fullsized":"2023-10-16T12:49:27.741065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:52.310280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791297+0000","last_change":"2023-10-16T12:49:25.738021+0000","last_active":"2023-10-16T12:49:26.791297+0000","last_peered":"2023-10-16T12:49:26.791297+0000","last_clean":"2023-10-16T12:49:26.791297+0000","last_became_active":"2023-10-16T12:49:25.737704+0000","last_became_peered":"2023-10-16T12:49:25.737704+0000","last_unstale":"2023-10-16T12:49:26.791297+0000","last_undegraded":"2023-10-16T12:49:26.791297+0000","last_fullsized":"2023-10-16T12:49:26.791297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:14.431428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.781489+0000","last_change":"2023-10-16T12:49:26.774938+0000","last_active":"2023-10-16T12:49:26.781489+0000","last_peered":"2023-10-16T12:49:26.781489+0000","last_clean":"2023-10-16T12:49:26.781489+0000","last_became_active":"2023-10-16T12:49:26.774692+0000","last_became_peered":"2023-10-16T12:49:26.774692+0000","last_unstale":"2023-10-16T12:49:26.781489+0000","last_undegraded":"2023-10-16T12:49:26.781489+0000","last_fullsized":"2023-10-16T12:49:26.781489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:48.445429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741148+0000","last_change":"2023-10-16T12:49:25.742216+0000","last_active":"2023-10-16T12:49:27.741148+0000","last_peered":"2023-10-16T12:49:27.741148+0000","last_clean":"2023-10-16T12:49:27.741148+0000","last_became_active":"2023-10-16T12:49:25.741888+0000","last_became_peered":"2023-10-16T12:49:25.741888+0000","last_unstale":"2023-10-16T12:49:27.741148+0000","last_undegraded":"2023-10-16T12:49:27.741148+0000","last_fullsized":"2023-10-16T12:49:27.741148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:12.761173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763412+0000","last_change":"2023-10-16T12:49:26.761491+0000","last_active":"2023-10-16T12:49:26.763412+0000","last_peered":"2023-10-16T12:49:26.763412+0000","last_clean":"2023-10-16T12:49:26.763412+0000","last_became_active":"2023-10-16T12:49:26.760444+0000","last_became_peered":"2023-10-16T12:49:26.760444+0000","last_unstale":"2023-10-16T12:49:26.763412+0000","last_undegraded":"2023-10-16T12:49:26.763412+0000","last_fullsized":"2023-10-16T12:49:26.763412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:26.259122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.748006+0000","last_change":"2023-10-16T12:49:25.725852+0000","last_active":"2023-10-16T12:49:26.748006+0000","last_peered":"2023-10-16T12:49:26.748006+0000","last_clean":"2023-10-16T12:49:26.748006+0000","last_became_active":"2023-10-16T12:49:25.725483+0000","last_became_peered":"2023-10-16T12:49:25.725483+0000","last_unstale":"2023-10-16T12:49:26.748006+0000","last_undegraded":"2023-10-16T12:49:26.748006+0000","last_fullsized":"2023-10-16T12:49:26.748006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:58.209696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739800+0000","last_change":"2023-10-16T12:49:26.772707+0000","last_active":"2023-10-16T12:49:27.739800+0000","last_peered":"2023-10-16T12:49:27.739800+0000","last_clean":"2023-10-16T12:49:27.739800+0000","last_became_active":"2023-10-16T12:49:26.771407+0000","last_became_peered":"2023-10-16T12:49:26.771407+0000","last_unstale":"2023-10-16T12:49:27.739800+0000","last_undegraded":"2023-10-16T12:49:27.739800+0000","last_fullsized":"2023-10-16T12:49:27.739800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:28.460635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791513+0000","last_change":"2023-10-16T12:49:25.733089+0000","last_active":"2023-10-16T12:49:26.791513+0000","last_peered":"2023-10-16T12:49:26.791513+0000","last_clean":"2023-10-16T12:49:26.791513+0000","last_became_active":"2023-10-16T12:49:25.732748+0000","last_became_peered":"2023-10-16T12:49:25.732748+0000","last_unstale":"2023-10-16T12:49:26.791513+0000","last_undegraded":"2023-10-16T12:49:26.791513+0000","last_fullsized":"2023-10-16T12:49:26.791513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:58.240759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779115+0000","last_change":"2023-10-16T12:49:26.775681+0000","last_active":"2023-10-16T12:49:26.779115+0000","last_peered":"2023-10-16T12:49:26.779115+0000","last_clean":"2023-10-16T12:49:26.779115+0000","last_became_active":"2023-10-16T12:49:26.772475+0000","last_became_peered":"2023-10-16T12:49:26.772475+0000","last_unstale":"2023-10-16T12:49:26.779115+0000","last_undegraded":"2023-10-16T12:49:26.779115+0000","last_fullsized":"2023-10-16T12:49:26.779115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:39.534158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733821+0000","last_change":"2023-10-16T12:49:25.735427+0000","last_active":"2023-10-16T12:49:27.733821+0000","last_peered":"2023-10-16T12:49:27.733821+0000","last_clean":"2023-10-16T12:49:27.733821+0000","last_became_active":"2023-10-16T12:49:25.735153+0000","last_became_peered":"2023-10-16T12:49:25.735153+0000","last_unstale":"2023-10-16T12:49:27.733821+0000","last_undegraded":"2023-10-16T12:49:27.733821+0000","last_fullsized":"2023-10-16T12:49:27.733821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:58.895115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.766149+0000","last_change":"2023-10-16T12:49:26.761804+0000","last_active":"2023-10-16T12:49:26.766149+0000","last_peered":"2023-10-16T12:49:26.766149+0000","last_clean":"2023-10-16T12:49:26.766149+0000","last_became_active":"2023-10-16T12:49:26.761501+0000","last_became_peered":"2023-10-16T12:49:26.761501+0000","last_unstale":"2023-10-16T12:49:26.766149+0000","last_undegraded":"2023-10-16T12:49:26.766149+0000","last_fullsized":"2023-10-16T12:49:26.766149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:20.304637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.738636+0000","last_change":"2023-10-16T12:49:25.738870+0000","last_active":"2023-10-16T12:49:27.738636+0000","last_peered":"2023-10-16T12:49:27.738636+0000","last_clean":"2023-10-16T12:49:27.738636+0000","last_became_active":"2023-10-16T12:49:25.738558+0000","last_became_peered":"2023-10-16T12:49:25.738558+0000","last_unstale":"2023-10-16T12:49:27.738636+0000","last_undegraded":"2023-10-16T12:49:27.738636+0000","last_fullsized":"2023-10-16T12:49:27.738636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:04:38.702610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741517+0000","last_change":"2023-10-16T12:49:26.771786+0000","last_active":"2023-10-16T12:49:27.741517+0000","last_peered":"2023-10-16T12:49:27.741517+0000","last_clean":"2023-10-16T12:49:27.741517+0000","last_became_active":"2023-10-16T12:49:26.769157+0000","last_became_peered":"2023-10-16T12:49:26.769157+0000","last_unstale":"2023-10-16T12:49:27.741517+0000","last_undegraded":"2023-10-16T12:49:27.741517+0000","last_fullsized":"2023-10-16T12:49:27.741517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:41.176774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"85'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734710+0000","last_change":"2023-10-16T12:49:25.725015+0000","last_active":"2023-10-16T12:49:27.734710+0000","last_peered":"2023-10-16T12:49:27.734710+0000","last_clean":"2023-10-16T12:49:27.734710+0000","last_became_active":"2023-10-16T12:49:25.724672+0000","last_became_peered":"2023-10-16T12:49:25.724672+0000","last_unstale":"2023-10-16T12:49:27.734710+0000","last_undegraded":"2023-10-16T12:49:27.734710+0000","last_fullsized":"2023-10-16T12:49:27.734710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:30.631949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741194+0000","last_change":"2023-10-16T12:49:26.774785+0000","last_active":"2023-10-16T12:49:27.741194+0000","last_peered":"2023-10-16T12:49:27.741194+0000","last_clean":"2023-10-16T12:49:27.741194+0000","last_became_active":"2023-10-16T12:49:26.774391+0000","last_became_peered":"2023-10-16T12:49:26.774391+0000","last_unstale":"2023-10-16T12:49:27.741194+0000","last_undegraded":"2023-10-16T12:49:27.741194+0000","last_fullsized":"2023-10-16T12:49:27.741194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:08.747676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.756760+0000","last_change":"2023-10-16T12:49:25.738491+0000","last_active":"2023-10-16T12:49:26.756760+0000","last_peered":"2023-10-16T12:49:26.756760+0000","last_clean":"2023-10-16T12:49:26.756760+0000","last_became_active":"2023-10-16T12:49:25.738080+0000","last_became_peered":"2023-10-16T12:49:25.738080+0000","last_unstale":"2023-10-16T12:49:26.756760+0000","last_undegraded":"2023-10-16T12:49:26.756760+0000","last_fullsized":"2023-10-16T12:49:26.756760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:43.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734259+0000","last_change":"2023-10-16T12:49:26.770163+0000","last_active":"2023-10-16T12:49:27.734259+0000","last_peered":"2023-10-16T12:49:27.734259+0000","last_clean":"2023-10-16T12:49:27.734259+0000","last_became_active":"2023-10-16T12:49:26.768935+0000","last_became_peered":"2023-10-16T12:49:26.768935+0000","last_unstale":"2023-10-16T12:49:27.734259+0000","last_undegraded":"2023-10-16T12:49:27.734259+0000","last_fullsized":"2023-10-16T12:49:27.734259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:21.378894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.747496+0000","last_change":"2023-10-16T12:49:25.739372+0000","last_active":"2023-10-16T12:49:26.747496+0000","last_peered":"2023-10-16T12:49:26.747496+0000","last_clean":"2023-10-16T12:49:26.747496+0000","last_became_active":"2023-10-16T12:49:25.738093+0000","last_became_peered":"2023-10-16T12:49:25.738093+0000","last_unstale":"2023-10-16T12:49:26.747496+0000","last_undegraded":"2023-10-16T12:49:26.747496+0000","last_fullsized":"2023-10-16T12:49:26.747496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:06.625845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779060+0000","last_change":"2023-10-16T12:49:26.775988+0000","last_active":"2023-10-16T12:49:26.779060+0000","last_peered":"2023-10-16T12:49:26.779060+0000","last_clean":"2023-10-16T12:49:26.779060+0000","last_became_active":"2023-10-16T12:49:26.769786+0000","last_became_peered":"2023-10-16T12:49:26.769786+0000","last_unstale":"2023-10-16T12:49:26.779060+0000","last_undegraded":"2023-10-16T12:49:26.779060+0000","last_fullsized":"2023-10-16T12:49:26.779060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:27.642654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740771+0000","last_change":"2023-10-16T12:49:25.737922+0000","last_active":"2023-10-16T12:49:27.740771+0000","last_peered":"2023-10-16T12:49:27.740771+0000","last_clean":"2023-10-16T12:49:27.740771+0000","last_became_active":"2023-10-16T12:49:25.737584+0000","last_became_peered":"2023-10-16T12:49:25.737584+0000","last_unstale":"2023-10-16T12:49:27.740771+0000","last_undegraded":"2023-10-16T12:49:27.740771+0000","last_fullsized":"2023-10-16T12:49:27.740771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:16.140142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.771056+0000","last_change":"2023-10-16T12:49:26.760148+0000","last_active":"2023-10-16T12:49:26.771056+0000","last_peered":"2023-10-16T12:49:26.771056+0000","last_clean":"2023-10-16T12:49:26.771056+0000","last_became_active":"2023-10-16T12:49:26.759818+0000","last_became_peered":"2023-10-16T12:49:26.759818+0000","last_unstale":"2023-10-16T12:49:26.771056+0000","last_undegraded":"2023-10-16T12:49:26.771056+0000","last_fullsized":"2023-10-16T12:49:26.771056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:52.310280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733696+0000","last_change":"2023-10-16T12:49:25.738179+0000","last_active":"2023-10-16T12:49:27.733696+0000","last_peered":"2023-10-16T12:49:27.733696+0000","last_clean":"2023-10-16T12:49:27.733696+0000","last_became_active":"2023-10-16T12:49:25.737864+0000","last_became_peered":"2023-10-16T12:49:25.737864+0000","last_unstale":"2023-10-16T12:49:27.733696+0000","last_undegraded":"2023-10-16T12:49:27.733696+0000","last_fullsized":"2023-10-16T12:49:27.733696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:15.150187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.773224+0000","last_change":"2023-10-16T12:49:26.771861+0000","last_active":"2023-10-16T12:49:26.773224+0000","last_peered":"2023-10-16T12:49:26.773224+0000","last_clean":"2023-10-16T12:49:26.773224+0000","last_became_active":"2023-10-16T12:49:26.771579+0000","last_became_peered":"2023-10-16T12:49:26.771579+0000","last_unstale":"2023-10-16T12:49:26.773224+0000","last_undegraded":"2023-10-16T12:49:26.773224+0000","last_fullsized":"2023-10-16T12:49:26.773224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:41.176774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739384+0000","last_change":"2023-10-16T12:49:25.732369+0000","last_active":"2023-10-16T12:49:27.739384+0000","last_peered":"2023-10-16T12:49:27.739384+0000","last_clean":"2023-10-16T12:49:27.739384+0000","last_became_active":"2023-10-16T12:49:25.732052+0000","last_became_peered":"2023-10-16T12:49:25.732052+0000","last_unstale":"2023-10-16T12:49:27.739384+0000","last_undegraded":"2023-10-16T12:49:27.739384+0000","last_fullsized":"2023-10-16T12:49:27.739384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:41.571703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733717+0000","last_change":"2023-10-16T12:49:26.769061+0000","last_active":"2023-10-16T12:49:27.733717+0000","last_peered":"2023-10-16T12:49:27.733717+0000","last_clean":"2023-10-16T12:49:27.733717+0000","last_became_active":"2023-10-16T12:49:26.768744+0000","last_became_peered":"2023-10-16T12:49:26.768744+0000","last_unstale":"2023-10-16T12:49:27.733717+0000","last_undegraded":"2023-10-16T12:49:27.733717+0000","last_fullsized":"2023-10-16T12:49:27.733717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:40.666471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733996+0000","last_change":"2023-10-16T12:49:25.726132+0000","last_active":"2023-10-16T12:49:27.733996+0000","last_peered":"2023-10-16T12:49:27.733996+0000","last_clean":"2023-10-16T12:49:27.733996+0000","last_became_active":"2023-10-16T12:49:25.725830+0000","last_became_peered":"2023-10-16T12:49:25.725830+0000","last_unstale":"2023-10-16T12:49:27.733996+0000","last_undegraded":"2023-10-16T12:49:27.733996+0000","last_fullsized":"2023-10-16T12:49:27.733996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:57.967543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776068+0000","last_change":"2023-10-16T12:49:26.770503+0000","last_active":"2023-10-16T12:49:26.776068+0000","last_peered":"2023-10-16T12:49:26.776068+0000","last_clean":"2023-10-16T12:49:26.776068+0000","last_became_active":"2023-10-16T12:49:26.769871+0000","last_became_peered":"2023-10-16T12:49:26.769871+0000","last_unstale":"2023-10-16T12:49:26.776068+0000","last_undegraded":"2023-10-16T12:49:26.776068+0000","last_fullsized":"2023-10-16T12:49:26.776068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:51.358206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.760843+0000","last_change":"2023-10-16T12:49:25.739754+0000","last_active":"2023-10-16T12:49:26.760843+0000","last_peered":"2023-10-16T12:49:26.760843+0000","last_clean":"2023-10-16T12:49:26.760843+0000","last_became_active":"2023-10-16T12:49:25.739259+0000","last_became_peered":"2023-10-16T12:49:25.739259+0000","last_unstale":"2023-10-16T12:49:26.760843+0000","last_undegraded":"2023-10-16T12:49:26.760843+0000","last_fullsized":"2023-10-16T12:49:26.760843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:09.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.780347+0000","last_change":"2023-10-16T12:49:26.771700+0000","last_active":"2023-10-16T12:49:26.780347+0000","last_peered":"2023-10-16T12:49:26.780347+0000","last_clean":"2023-10-16T12:49:26.780347+0000","last_became_active":"2023-10-16T12:49:26.771262+0000","last_became_peered":"2023-10-16T12:49:26.771262+0000","last_unstale":"2023-10-16T12:49:26.780347+0000","last_undegraded":"2023-10-16T12:49:26.780347+0000","last_fullsized":"2023-10-16T12:49:26.780347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:08.747676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.746910+0000","last_change":"2023-10-16T12:49:25.739319+0000","last_active":"2023-10-16T12:49:26.746910+0000","last_peered":"2023-10-16T12:49:26.746910+0000","last_clean":"2023-10-16T12:49:26.746910+0000","last_became_active":"2023-10-16T12:49:25.739043+0000","last_became_peered":"2023-10-16T12:49:25.739043+0000","last_unstale":"2023-10-16T12:49:26.746910+0000","last_undegraded":"2023-10-16T12:49:26.746910+0000","last_fullsized":"2023-10-16T12:49:26.746910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:54.566500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":384,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:49:25.728711+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:49:25.728711+0000","last_peered":"2023-10-16T12:49:25.728711+0000","last_clean":"2023-10-16T12:49:25.728711+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:49:25.728711+0000","last_undegraded":"2023-10-16T12:49:25.728711+0000","last_fullsized":"2023-10-16T12:49:25.728711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542338,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1626112,"data_stored":1077229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.3,"15min":1.294},"min":{"1min":0.73499999999999999,"5min":0.66100000000000003,"15min":0.65100000000000002},"max":{"1min":1.72,"5min":1.72,"15min":2.1890000000000001},"last":1.728},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.325,"15min":1.3440000000000001},"min":{"1min":1.0089999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6479999999999999,"5min":1.8420000000000001,"15min":1.8899999999999999},"last":1.5940000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.218,"15min":1.2170000000000001},"min":{"1min":0.873,"5min":0.72199999999999998,"15min":0.67300000000000004},"max":{"1min":1.6539999999999999,"5min":1.78,"15min":1.78},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1850000000000001,"15min":1.208},"min":{"1min":0.875,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.667,"5min":1.6779999999999999,"15min":1.843},"last":1.2430000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.204,"15min":1.2090000000000001},"min":{"1min":0.94899999999999995,"5min":0.71299999999999997,"15min":0.59799999999999998},"max":{"1min":1.5880000000000001,"5min":1.591,"15min":2.2130000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2909999999999999,"15min":1.3260000000000001},"min":{"1min":1.0449999999999999,"5min":0.81899999999999995,"15min":0.63500000000000001},"max":{"1min":1.5549999999999999,"5min":1.7230000000000001,"15min":2.3450000000000002},"last":1.5129999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.292,"15min":1.3},"min":{"1min":0.879,"5min":0.80000000000000004,"15min":0.69099999999999995},"max":{"1min":1.847,"5min":1.847,"15min":2.0150000000000001},"last":1.429},{"interface":"front","average":{"1min":1.417,"5min":1.395,"15min":1.3999999999999999},"min":{"1min":1.006,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.661,"5min":1.8009999999999999,"15min":2.0019999999999998},"last":1.385}]},{"osd":4,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.153,"15min":1.194},"min":{"1min":0.86699999999999999,"5min":0.72099999999999997,"15min":0.67300000000000004},"max":{"1min":1.597,"5min":1.7030000000000001,"15min":1.9710000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.341,"15min":1.3440000000000001},"min":{"1min":0.98299999999999998,"5min":0.83199999999999996,"15min":0.56499999999999995},"max":{"1min":1.77,"5min":1.9339999999999999,"15min":2.282},"last":1.631}]},{"osd":5,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.044,"5min":0.99399999999999999,"15min":0.78900000000000003},"max":{"1min":1.8320000000000001,"5min":1.9139999999999999,"15min":2.0529999999999999},"last":1.284},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4890000000000001,"15min":1.4790000000000001},"min":{"1min":1.2070000000000001,"5min":0.98899999999999999,"15min":0.61299999999999999},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.2000000000000002},"last":1.1919999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.569,"15min":1.5700000000000001},"min":{"1min":1.3819999999999999,"5min":1.026,"15min":0.72699999999999998},"max":{"1min":1.9370000000000001,"5min":2.1110000000000002,"15min":2.2639999999999998},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.4450000000000001,"15min":1.4530000000000001},"min":{"1min":0.997,"5min":0.89100000000000001,"15min":0.83399999999999996},"max":{"1min":1.7210000000000001,"5min":2.1280000000000001,"15min":2.1379999999999999},"last":1.673}]}]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":486701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2949999999999999,"15min":1.2829999999999999},"min":{"1min":0.81899999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.6619999999999999,"5min":1.784,"15min":3.7450000000000001},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.3240000000000001,"15min":1.3320000000000001},"min":{"1min":0.95199999999999996,"5min":0.748,"15min":0.57999999999999996},"max":{"1min":1.6259999999999999,"5min":2.0270000000000001,"15min":3.819},"last":1.1990000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2250000000000001,"15min":1.1870000000000001},"min":{"1min":0.97799999999999998,"5min":0.67400000000000004,"15min":0.621},"max":{"1min":1.5529999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2330000000000001,"15min":1.2130000000000001},"min":{"1min":0.90700000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.599,"5min":2.0459999999999998,"15min":3.8519999999999999},"last":1.518}]},{"osd":2,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2130000000000001,"15min":1.218},"min":{"1min":0.85699999999999998,"5min":0.627,"15min":0.627},"max":{"1min":1.6339999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.752},{"interface":"front","average":{"1min":1.292,"5min":1.264,"15min":1.284},"min":{"1min":0.91300000000000003,"5min":0.88500000000000001,"15min":0.755},"max":{"1min":1.653,"5min":1.9970000000000001,"15min":3.9009999999999998},"last":0.90200000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3320000000000001,"15min":1.333},"min":{"1min":1.016,"5min":0.80100000000000005,"15min":0.751},"max":{"1min":1.6459999999999999,"5min":2.3900000000000001,"15min":4.2380000000000004},"last":1.331},{"interface":"front","average":{"1min":1.373,"5min":1.296,"15min":1.351},"min":{"1min":1.0860000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6859999999999999,"5min":2.012,"15min":4.4269999999999996},"last":0.871}]},{"osd":4,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.361,"15min":1.345},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.57699999999999996},"max":{"1min":1.7070000000000001,"5min":2.3029999999999999,"15min":2.4100000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.4239999999999999,"15min":1.476},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.67300000000000004},"max":{"1min":1.839,"5min":2.2189999999999999,"15min":2.3050000000000002},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.306,"15min":1.3129999999999999},"min":{"1min":0.86499999999999999,"5min":0.79600000000000004,"15min":0.63900000000000001},"max":{"1min":1.772,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.145},{"interface":"front","average":{"1min":1.456,"5min":1.4259999999999999,"15min":1.454},"min":{"1min":1.075,"5min":1.075,"15min":0.63900000000000001},"max":{"1min":2.214,"5min":2.214,"15min":2.5230000000000001},"last":1.466}]},{"osd":7,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.3919999999999999,"15min":1.4450000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.495},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.454,"15min":1.4610000000000001},"min":{"1min":1.1220000000000001,"5min":0.96399999999999997,"15min":0.59099999999999997},"max":{"1min":2.008,"5min":2.2610000000000001,"15min":2.2719999999999998},"last":1.272}]}]},{"osd":5,"up_from":14,"seq":60129542340,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335480,"statfs":{"total":96636764160,"available":96599531520,"internally_reserved":0,"allocated":1044480,"data_stored":493653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1459999999999999,"15min":1.159},"min":{"1min":0.80100000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7150000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.1879999999999999,"15min":1.177},"min":{"1min":0.85899999999999999,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.516,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":0.97699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.222,"15min":1.25},"min":{"1min":0.94199999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.679,"5min":1.728,"15min":2.415},"last":1.52},{"interface":"front","average":{"1min":1.236,"5min":1.1950000000000001,"15min":1.2250000000000001},"min":{"1min":0.96899999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.708,"5min":1.708,"15min":1.9419999999999999},"last":1.1379999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2649999999999999,"15min":1.298},"min":{"1min":0.94799999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.625,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.591},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2649999999999999,"15min":1.268},"min":{"1min":0.97599999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.548,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.29}]},{"osd":3,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.329,"15min":1.3140000000000001},"min":{"1min":0.97399999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6479999999999999,"5min":1.6870000000000001,"15min":2.2690000000000001},"last":1.103},{"interface":"front","average":{"1min":1.403,"5min":1.3560000000000001,"15min":1.377},"min":{"1min":1.0609999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.6220000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.6819999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89000000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7170000000000001,"5min":1.996,"15min":2.7909999999999999},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.329,"15min":1.327},"min":{"1min":0.90700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.8120000000000001,"5min":1.877,"15min":5.3250000000000002},"last":1.3979999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3200000000000001,"15min":1.3280000000000001},"min":{"1min":0.96799999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.696,"5min":1.927,"15min":2.343},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.377,"15min":1.377},"min":{"1min":1.095,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9119999999999999,"5min":2.3079999999999998,"15min":2.6520000000000001},"last":1.6399999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4370000000000001,"15min":1.4590000000000001},"min":{"1min":1.054,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7170000000000001,"5min":1.889,"15min":5.1269999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.506,"15min":1.51},"min":{"1min":1.131,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.7629999999999999,"5min":2.0659999999999998,"15min":2.4929999999999999},"last":1.4830000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542340,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58576,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313264,"statfs":{"total":96636764160,"available":96576782336,"internally_reserved":0,"allocated":1048576,"data_stored":494613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2090000000000001,"15min":1.2030000000000001},"min":{"1min":0.78700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.766,"5min":1.766,"15min":2.2200000000000002},"last":0.873},{"interface":"front","average":{"1min":1.155,"5min":1.125,"15min":1.1539999999999999},"min":{"1min":0.80700000000000005,"5min":0.65800000000000003,"15min":0.623},"max":{"1min":1.645,"5min":1.7070000000000001,"15min":1.9159999999999999},"last":1.139}]},{"osd":1,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1599999999999999,"15min":1.1970000000000001},"min":{"1min":0.79900000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.865},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2529999999999999,"15min":1.272},"min":{"1min":0.91200000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.724,"5min":1.724,"15min":1.9390000000000001},"last":1.091}]},{"osd":2,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.97799999999999998,"5min":0.81899999999999995,"15min":0.753},"max":{"1min":1.667,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3120000000000001,"15min":1.3400000000000001},"min":{"1min":0.93100000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.9910000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2490000000000001,"15min":1.27},"min":{"1min":0.91900000000000004,"5min":0.80900000000000005,"15min":0.752},"max":{"1min":1.5289999999999999,"5min":1.716,"15min":1.8100000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.3640000000000001,"15min":1.3819999999999999},"min":{"1min":1.236,"5min":0.86799999999999999,"15min":0.76500000000000001},"max":{"1min":1.7869999999999999,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.044}]},{"osd":5,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.304,"15min":1.327},"min":{"1min":0.94799999999999995,"5min":0.86799999999999999,"15min":0.50900000000000001},"max":{"1min":1.6579999999999999,"5min":2.0800000000000001,"15min":7.9000000000000004},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.2929999999999999,"15min":1.319},"min":{"1min":0.96999999999999997,"5min":0.76400000000000001,"15min":0.53600000000000003},"max":{"1min":1.625,"5min":2.3700000000000001,"15min":7.3460000000000001},"last":1.405}]},{"osd":6,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3819999999999999,"15min":1.387},"min":{"1min":0.98199999999999998,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":1.8380000000000001,"5min":2.153,"15min":2.153},"last":1.329},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4590000000000001,"15min":1.486},"min":{"1min":1.0369999999999999,"5min":0.90400000000000003,"15min":0.65300000000000002},"max":{"1min":1.96,"5min":2.1339999999999999,"15min":2.6269999999999998},"last":1.282}]},{"osd":7,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4350000000000001,"15min":1.4690000000000001},"min":{"1min":0.95199999999999996,"5min":0.88600000000000001,"15min":0.54200000000000004},"max":{"1min":1.76,"5min":2.056,"15min":2.254},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.5620000000000001,"15min":1.595},"min":{"1min":1.1659999999999999,"5min":0.94799999999999995,"15min":0.69199999999999995},"max":{"1min":1.893,"5min":2.262,"15min":2.5289999999999999},"last":1.492}]}]},{"osd":3,"up_from":14,"seq":60129542340,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40512,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331328,"statfs":{"total":96636764160,"available":96595279872,"internally_reserved":0,"allocated":1032192,"data_stored":486757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.94399999999999995,"15min":0.95899999999999996},"min":{"1min":0.61099999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.482,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.194,"5min":1.1839999999999999,"15min":1.181},"min":{"1min":0.66400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5800000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0229999999999999,"15min":1.0369999999999999},"min":{"1min":0.69299999999999995,"5min":0.53500000000000003,"15min":0.39500000000000002},"max":{"1min":1.4099999999999999,"5min":1.758,"15min":1.8149999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.0840000000000001,"15min":1.087},"min":{"1min":0.57799999999999996,"5min":0.50600000000000001,"15min":0.45600000000000002},"max":{"1min":1.476,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.80700000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.0960000000000001,"15min":1.0840000000000001},"min":{"1min":0.71999999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.6539999999999999,"5min":1.7949999999999999,"15min":1.9510000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3460000000000001,"15min":1.367},"min":{"1min":0.79600000000000004,"5min":0.61499999999999999,"15min":0.58299999999999996},"max":{"1min":1.7010000000000001,"5min":1.8600000000000001,"15min":2.0569999999999999},"last":0.996}]},{"osd":4,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.4930000000000001,"15min":1.4910000000000001},"min":{"1min":1.1579999999999999,"5min":0.80400000000000005,"15min":0.77600000000000002},"max":{"1min":1.9239999999999999,"5min":1.956,"15min":2.2730000000000001},"last":1.371},{"interface":"front","average":{"1min":1.431,"5min":1.4299999999999999,"15min":1.427},"min":{"1min":0.97499999999999998,"5min":0.64800000000000002,"15min":0.63500000000000001},"max":{"1min":1.869,"5min":1.8939999999999999,"15min":3.6930000000000001},"last":1.1579999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4119999999999999,"15min":1.405},"min":{"1min":1.0069999999999999,"5min":0.65900000000000003,"15min":0.60699999999999998},"max":{"1min":1.9910000000000001,"5min":2.028,"15min":3.8300000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.47,"15min":1.4730000000000001},"min":{"1min":1.0269999999999999,"5min":0.95399999999999996,"15min":0.64900000000000002},"max":{"1min":1.827,"5min":2.1030000000000002,"15min":2.3540000000000001},"last":1.232}]},{"osd":6,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4530000000000001,"15min":1.4550000000000001},"min":{"1min":0.88900000000000001,"5min":0.83999999999999997,"15min":0.65300000000000002},"max":{"1min":1.905,"5min":2.2989999999999999,"15min":2.415},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4490000000000001,"15min":1.4379999999999999},"min":{"1min":1.196,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8700000000000001,"5min":2.3740000000000001,"15min":3.9809999999999999},"last":1.3340000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.554,"15min":1.5469999999999999},"min":{"1min":1.0660000000000001,"5min":0.81000000000000005,"15min":0.80300000000000005},"max":{"1min":1.917,"5min":2.198,"15min":2.431},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.621,"15min":1.613},"min":{"1min":1.0389999999999999,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":1.986,"5min":2.552,"15min":2.552},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542338,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44676,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327164,"statfs":{"total":96636764160,"available":96591015936,"internally_reserved":0,"allocated":1032192,"data_stored":486757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.46999999999999997,"5min":0.315,"15min":0.315},"max":{"1min":1.0980000000000001,"5min":1.742,"15min":1.7889999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.091,"5min":1.087,"15min":1.0880000000000001},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.655,"5min":1.7070000000000001,"15min":2.085},"last":0.69099999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.133,"15min":1.1180000000000001},"min":{"1min":0.59699999999999998,"5min":0.51500000000000001,"15min":0.50900000000000001},"max":{"1min":1.458,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.107},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.206,"15min":1.22},"min":{"1min":0.76400000000000001,"5min":0.46500000000000002,"15min":0.45500000000000002},"max":{"1min":2.347,"5min":2.347,"15min":2.4289999999999998},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0680000000000001,"15min":1.0740000000000001},"min":{"1min":0.54700000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.4610000000000001,"5min":1.788,"15min":1.859},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3069999999999999,"15min":1.3049999999999999},"min":{"1min":0.64500000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.8049999999999999,"5min":2.1000000000000001,"15min":2.1970000000000001},"last":1.177}]},{"osd":4,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.456,"15min":1.4510000000000001},"min":{"1min":0.94699999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.0619999999999998,"5min":2.431,"15min":2.431},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4239999999999999,"15min":1.4430000000000001},"min":{"1min":1.038,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8360000000000001,"5min":2.0670000000000002,"15min":2.294},"last":1.2}]},{"osd":5,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4910000000000001,"15min":1.4930000000000001},"min":{"1min":1.095,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.117,"5min":2.23,"15min":2.3340000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3939999999999999,"15min":1.421},"min":{"1min":0.85899999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.8759999999999999,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.696}]},{"osd":6,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.417,"15min":1.448},"min":{"1min":0.97399999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.181,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.583},{"interface":"front","average":{"1min":1.502,"5min":1.5629999999999999,"15min":1.5609999999999999},"min":{"1min":1.0740000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.2789999999999999,"5min":2.294,"15min":2.488},"last":1.3640000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5920000000000001,"15min":1.6040000000000001},"min":{"1min":1.157,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.202,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.169},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.55},"min":{"1min":0.99399999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.0390000000000001,"5min":2.573,"15min":2.6429999999999998},"last":1.401}]}]},{"osd":1,"up_from":14,"seq":60129542338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58568,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313272,"statfs":{"total":96636764160,"available":96576790528,"internally_reserved":0,"allocated":1040384,"data_stored":493091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.90700000000000003,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.423,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.0820000000000001,"15min":1.0960000000000001},"min":{"1min":0.80500000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.931,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.98699999999999999,"15min":0.995},"min":{"1min":0.76800000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.603,"5min":1.8460000000000001,"15min":2.262},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.1859999999999999},"min":{"1min":0.85699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.5980000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.3109999999999999,"15min":1.3180000000000001},"min":{"1min":1.2170000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.988,"5min":1.988,"15min":2.2349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.272,"5min":1.208,"15min":1.22},"min":{"1min":0.94899999999999995,"5min":0.56799999999999995,"15min":0.46800000000000003},"max":{"1min":1.625,"5min":1.9199999999999999,"15min":2.1789999999999998},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.2929999999999999,"15min":1.3140000000000001},"min":{"1min":0.99299999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.032,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4510000000000001,"15min":1.4470000000000001},"min":{"1min":1.1559999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.8839999999999999,"5min":2.0649999999999999,"15min":3.0659999999999998},"last":0.95099999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.3939999999999999,"15min":1.4390000000000001},"min":{"1min":1.107,"5min":0.755,"15min":0.755},"max":{"1min":1.95,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.4470000000000001,"15min":1.4299999999999999},"min":{"1min":1.0640000000000001,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":1.996,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.2450000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.532,"15min":1.54},"min":{"1min":1.2549999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.0129999999999999,"5min":2.1899999999999999,"15min":2.9350000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4970000000000001,"15min":1.5289999999999999},"min":{"1min":1.0780000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.0739999999999998,"5min":2.387,"15min":2.387},"last":0.82899999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.5940000000000001,"15min":1.6259999999999999},"min":{"1min":1.29,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.1760000000000002,"5min":2.2429999999999999,"15min":2.2879999999999998},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.655,"5min":1.5549999999999999,"15min":1.5660000000000001},"min":{"1min":1.2110000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.153,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.173}]}]},{"osd":0,"up_from":14,"seq":60129542339,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36940,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334900,"statfs":{"total":96636764160,"available":96598937600,"internally_reserved":0,"allocated":1634304,"data_stored":1079263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.876,"15min":0.90300000000000002},"min":{"1min":0.45700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.335,"5min":1.5089999999999999,"15min":2.5720000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.0980000000000001,"15min":1.0860000000000001},"min":{"1min":0.59799999999999998,"5min":0.47799999999999998,"15min":0.32700000000000001},"max":{"1min":1.706,"5min":1.8109999999999999,"15min":2.1800000000000002},"last":0.92500000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.095,"15min":1.1299999999999999},"min":{"1min":0.63700000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.6679999999999999,"5min":1.8120000000000001,"15min":2.7650000000000001},"last":1.579},{"interface":"front","average":{"1min":1.194,"5min":1.1539999999999999,"15min":1.165},"min":{"1min":0.71299999999999997,"5min":0.496,"15min":0.438},"max":{"1min":1.637,"5min":1.952,"15min":1.952},"last":1.1120000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0680000000000001,"15min":1.0800000000000001},"min":{"1min":0.68000000000000005,"5min":0.41299999999999998,"15min":0.315},"max":{"1min":1.696,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2629999999999999,"15min":1.3069999999999999},"min":{"1min":0.746,"5min":0.56100000000000005,"15min":0.45700000000000002},"max":{"1min":1.6379999999999999,"5min":1.8740000000000001,"15min":3.6579999999999999},"last":1.1799999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.339,"15min":1.347},"min":{"1min":0.98799999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":4.726},"last":1.389},{"interface":"front","average":{"1min":1.448,"5min":1.409,"15min":1.454},"min":{"1min":0.88300000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.7809999999999999,"5min":1.966,"15min":4.843},"last":1.722}]},{"osd":5,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.423,"15min":1.4219999999999999},"min":{"1min":0.91000000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.857,"5min":2.2229999999999999,"15min":2.2440000000000002},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":1.135,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.7569999999999999,"5min":2.02,"15min":4.9059999999999997},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.51,"15min":1.532},"min":{"1min":0.98099999999999998,"5min":0.78300000000000003,"15min":0.73299999999999998},"max":{"1min":1.7869999999999999,"5min":2.3519999999999999,"15min":3.7890000000000001},"last":1.829},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.496,"15min":1.5189999999999999},"min":{"1min":0.96699999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.9379999999999999,"5min":2.145,"15min":2.6589999999999998},"last":1.641}]},{"osd":7,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6060000000000001,"15min":1.6180000000000001},"min":{"1min":1.0249999999999999,"5min":0.92000000000000004,"15min":0.82599999999999996},"max":{"1min":2.0219999999999998,"5min":2.3490000000000002,"15min":2.4729999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.5369999999999999,"15min":1.5629999999999999},"min":{"1min":1.2789999999999999,"5min":0.755,"15min":0.755},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":4.5890000000000004},"last":1.5409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:49:31.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:49:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.554+0000 7f8d3759e640 1 Processor -- start 2023-10-16T12:49:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.555+0000 7f8d3759e640 1 -- start start 2023-10-16T12:49:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.556+0000 7f8d3759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d3815dde0 0x7f8d380709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.556+0000 7f8d3759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d38070f20 0x7f8d380ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.556+0000 7f8d3759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d38160a90 con 0x7f8d3815c9f0 2023-10-16T12:49:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.556+0000 7f8d3759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380cdec0 con 0x7f8d3815dde0 2023-10-16T12:49:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.556+0000 7f8d3759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380cd900 con 0x7f8d38070f20 2023-10-16T12:49:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d3659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8d3815c9f0 0x7f8d380c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50276/0 (socket says 172.21.15.60:50276) 2023-10-16T12:49:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d3659c640 1 -- 172.21.15.60:0/2716742488 learned_addr learned my addr 172.21.15.60:0/2716742488 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d35d9b640 1 --2- 172.21.15.60:0/2716742488 >> v2:172.21.15.60:3300/0 conn(0x7f8d3815dde0 0x7f8d380709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d36d9d640 1 --2- 172.21.15.60:0/2716742488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d38070f20 0x7f8d380ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d35d9b640 1 -- 172.21.15.60:0/2716742488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d38070f20 msgr2=0x7f8d380ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d35d9b640 1 --2- 172.21.15.60:0/2716742488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d38070f20 0x7f8d380ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d3559a640 1 -- 172.21.15.60:0/2716742488 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646603901 0 0) 0x7f8d38160a90 con 0x7f8d3815c9f0 2023-10-16T12:49:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d35d9b640 1 -- 172.21.15.60:0/2716742488 >> v1:172.21.15.60:6789/0 conn(0x7f8d3815c9f0 legacy=0x7f8d380c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.557+0000 7f8d35d9b640 1 -- 172.21.15.60:0/2716742488 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d380cdbe0 con 0x7f8d3815dde0 2023-10-16T12:49:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.558+0000 7f8d35d9b640 1 --2- 172.21.15.60:0/2716742488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d380709e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f8d200097a0 tx=0x7f8d2002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e8db935dae0664 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3559a640 1 -- 172.21.15.60:0/2716742488 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2002d040 con 0x7f8d3815dde0 2023-10-16T12:49:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3559a640 1 -- 172.21.15.60:0/2716742488 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d20031810 con 0x7f8d3815dde0 2023-10-16T12:49:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3559a640 1 -- 172.21.15.60:0/2716742488 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20031ba0 con 0x7f8d3815dde0 2023-10-16T12:49:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3759e640 1 -- 172.21.15.60:0/2716742488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 msgr2=0x7f8d380709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3759e640 1 --2- 172.21.15.60:0/2716742488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d380709e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f8d200097a0 tx=0x7f8d2002b8b0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3759e640 1 -- 172.21.15.60:0/2716742488 shutdown_connections 2023-10-16T12:49:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3759e640 1 --2- 172.21.15.60:0/2716742488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d38070f20 0x7f8d380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3759e640 1 --2- 172.21.15.60:0/2716742488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d380709e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.559+0000 7f8d3759e640 1 -- 172.21.15.60:0/2716742488 >> 172.21.15.60:0/2716742488 conn(0x7f8d380ce6a0 msgr2=0x7f8d380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.560+0000 7f8d3759e640 1 -- 172.21.15.60:0/2716742488 shutdown_connections 2023-10-16T12:49:31.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.560+0000 7f8d3759e640 1 -- 172.21.15.60:0/2716742488 wait complete. 2023-10-16T12:49:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.561+0000 7f8d3759e640 1 Processor -- start 2023-10-16T12:49:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 -- start start 2023-10-16T12:49:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d38070f20 0x7f8d380c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d3815c9f0 0x7f8d3815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d3806adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380c2f80 con 0x7f8d3815dde0 2023-10-16T12:49:31.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380c60c0 con 0x7f8d38070f20 2023-10-16T12:49:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.562+0000 7f8d3759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380c63a0 con 0x7f8d3815c9f0 2023-10-16T12:49:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d3806adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d3806adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42654/0 (socket says 172.21.15.60:42654) 2023-10-16T12:49:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 -- 172.21.15.60:0/1405565950 learned_addr learned my addr 172.21.15.60:0/1405565950 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:31.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d3659c640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d38070f20 0x7f8d380c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d35d9b640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d3815c9f0 0x7f8d3815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 -- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d3815c9f0 msgr2=0x7f8d3815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d3815c9f0 0x7f8d3815a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 -- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d38070f20 msgr2=0x7f8d380c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d38070f20 0x7f8d380c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d36d9d640 1 -- 172.21.15.60:0/1405565950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d20009400 con 0x7f8d3815dde0 2023-10-16T12:49:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d3659c640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d38070f20 0x7f8d380c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.563+0000 7f8d35d9b640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d3815c9f0 0x7f8d3815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.564+0000 7f8d36d9d640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d3806adf0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8d30007e40 tx=0x7f8d3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:31.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.564+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30014620 con 0x7f8d3815dde0 2023-10-16T12:49:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.564+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3000ee90 con 0x7f8d3815dde0 2023-10-16T12:49:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.564+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300149e0 con 0x7f8d3815dde0 2023-10-16T12:49:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.564+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d380c6680 con 0x7f8d3815dde0 2023-10-16T12:49:31.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.564+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d380c5e90 con 0x7f8d3815dde0 2023-10-16T12:49:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.565+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d381611a0 con 0x7f8d3815dde0 2023-10-16T12:49:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.566+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8d30014b40 con 0x7f8d3815dde0 2023-10-16T12:49:31.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.567+0000 7f8d2effd640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8d1806afc0 0x7f8d1806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.567+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3008a110 con 0x7f8d3815dde0 2023-10-16T12:49:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.568+0000 7f8d3659c640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8d1806afc0 0x7f8d1806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:31.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.569+0000 7f8d3659c640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8d1806afc0 0x7f8d1806d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d28000f60 tx=0x7f8d28009040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.571+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8d300550a0 con 0x7f8d3815dde0 2023-10-16T12:49:31.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.764+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 --> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d380704c0 con 0x7f8d1806afc0 2023-10-16T12:49:31.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.787+0000 7f8d2effd640 1 -- 172.21.15.60:0/1405565950 <== mgr.5186 v2:172.21.15.144:6832/2872949001 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286648 (secure 0 0 0) 0x7f8d280013d0 con 0x7f8d1806afc0 2023-10-16T12:49:31.794 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:31.796 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:49:31.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8d1806afc0 msgr2=0x7f8d1806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8d1806afc0 0x7f8d1806d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d28000f60 tx=0x7f8d28009040 comp rx=0 tx=0).stop 2023-10-16T12:49:31.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 msgr2=0x7f8d3806adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:31.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d3806adf0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8d30007e40 tx=0x7f8d3000e4b0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 shutdown_connections 2023-10-16T12:49:31.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f8d1806afc0 0x7f8d1806d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8d3815c9f0 0x7f8d3815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d3815dde0 0x7f8d3806adf0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 --2- 172.21.15.60:0/1405565950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d38070f20 0x7f8d380c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:31.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 >> 172.21.15.60:0/1405565950 conn(0x7f8d380ce6a0 msgr2=0x7f8d381606b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:31.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 shutdown_connections 2023-10-16T12:49:31.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:31.794+0000 7f8d3759e640 1 -- 172.21.15.60:0/1405565950 wait complete. 2023-10-16T12:49:31.817 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T12:49:30.990083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":340576,"kb_used_data":9260,"kb_used_omap":12,"kb_used_meta":331187,"kb_avail":754634144,"statfs":{"total":773094113280,"available":772745363456,"internally_reserved":0,"allocated":9482240,"data_stored":5098064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":339136056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001973"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.782184+0000","last_change":"2023-10-16T12:49:26.779800+0000","last_active":"2023-10-16T12:49:26.782184+0000","last_peered":"2023-10-16T12:49:26.782184+0000","last_clean":"2023-10-16T12:49:26.782184+0000","last_became_active":"2023-10-16T12:49:26.779504+0000","last_became_peered":"2023-10-16T12:49:26.779504+0000","last_unstale":"2023-10-16T12:49:26.782184+0000","last_undegraded":"2023-10-16T12:49:26.782184+0000","last_fullsized":"2023-10-16T12:49:26.782184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:46.019798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.782207+0000","last_change":"2023-10-16T12:49:26.774328+0000","last_active":"2023-10-16T12:49:26.782207+0000","last_peered":"2023-10-16T12:49:26.782207+0000","last_clean":"2023-10-16T12:49:26.782207+0000","last_became_active":"2023-10-16T12:49:26.771401+0000","last_became_peered":"2023-10-16T12:49:26.771401+0000","last_unstale":"2023-10-16T12:49:26.782207+0000","last_undegraded":"2023-10-16T12:49:26.782207+0000","last_fullsized":"2023-10-16T12:49:26.782207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.315108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763391+0000","last_change":"2023-10-16T12:49:26.758417+0000","last_active":"2023-10-16T12:49:26.763391+0000","last_peered":"2023-10-16T12:49:26.763391+0000","last_clean":"2023-10-16T12:49:26.763391+0000","last_became_active":"2023-10-16T12:49:26.757948+0000","last_became_peered":"2023-10-16T12:49:26.757948+0000","last_unstale":"2023-10-16T12:49:26.763391+0000","last_undegraded":"2023-10-16T12:49:26.763391+0000","last_fullsized":"2023-10-16T12:49:26.763391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:48.142634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740156+0000","last_change":"2023-10-16T12:49:26.769606+0000","last_active":"2023-10-16T12:49:27.740156+0000","last_peered":"2023-10-16T12:49:27.740156+0000","last_clean":"2023-10-16T12:49:27.740156+0000","last_became_active":"2023-10-16T12:49:26.768287+0000","last_became_peered":"2023-10-16T12:49:26.768287+0000","last_unstale":"2023-10-16T12:49:27.740156+0000","last_undegraded":"2023-10-16T12:49:27.740156+0000","last_fullsized":"2023-10-16T12:49:27.740156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:51.358206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.770471+0000","last_change":"2023-10-16T12:49:26.769830+0000","last_active":"2023-10-16T12:49:26.770471+0000","last_peered":"2023-10-16T12:49:26.770471+0000","last_clean":"2023-10-16T12:49:26.770471+0000","last_became_active":"2023-10-16T12:49:26.768941+0000","last_became_peered":"2023-10-16T12:49:26.768941+0000","last_unstale":"2023-10-16T12:49:26.770471+0000","last_undegraded":"2023-10-16T12:49:26.770471+0000","last_fullsized":"2023-10-16T12:49:26.770471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:46.679516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.773472+0000","last_change":"2023-10-16T12:49:26.765676+0000","last_active":"2023-10-16T12:49:26.773472+0000","last_peered":"2023-10-16T12:49:26.773472+0000","last_clean":"2023-10-16T12:49:26.773472+0000","last_became_active":"2023-10-16T12:49:26.765356+0000","last_became_peered":"2023-10-16T12:49:26.765356+0000","last_unstale":"2023-10-16T12:49:26.773472+0000","last_undegraded":"2023-10-16T12:49:26.773472+0000","last_fullsized":"2023-10-16T12:49:26.773472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:15.408002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.737941+0000","last_change":"2023-10-16T12:49:26.768164+0000","last_active":"2023-10-16T12:49:27.737941+0000","last_peered":"2023-10-16T12:49:27.737941+0000","last_clean":"2023-10-16T12:49:27.737941+0000","last_became_active":"2023-10-16T12:49:26.767840+0000","last_became_peered":"2023-10-16T12:49:26.767840+0000","last_unstale":"2023-10-16T12:49:27.737941+0000","last_undegraded":"2023-10-16T12:49:27.737941+0000","last_fullsized":"2023-10-16T12:49:27.737941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:25.123970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.761569+0000","last_change":"2023-10-16T12:49:26.757776+0000","last_active":"2023-10-16T12:49:26.761569+0000","last_peered":"2023-10-16T12:49:26.761569+0000","last_clean":"2023-10-16T12:49:26.761569+0000","last_became_active":"2023-10-16T12:49:26.757439+0000","last_became_peered":"2023-10-16T12:49:26.757439+0000","last_unstale":"2023-10-16T12:49:26.761569+0000","last_undegraded":"2023-10-16T12:49:26.761569+0000","last_fullsized":"2023-10-16T12:49:26.761569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:41.176774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739855+0000","last_change":"2023-10-16T12:49:26.761605+0000","last_active":"2023-10-16T12:49:27.739855+0000","last_peered":"2023-10-16T12:49:27.739855+0000","last_clean":"2023-10-16T12:49:27.739855+0000","last_became_active":"2023-10-16T12:49:26.761185+0000","last_became_peered":"2023-10-16T12:49:26.761185+0000","last_unstale":"2023-10-16T12:49:27.739855+0000","last_undegraded":"2023-10-16T12:49:27.739855+0000","last_fullsized":"2023-10-16T12:49:27.739855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:09.616899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734477+0000","last_change":"2023-10-16T12:49:26.768282+0000","last_active":"2023-10-16T12:49:27.734477+0000","last_peered":"2023-10-16T12:49:27.734477+0000","last_clean":"2023-10-16T12:49:27.734477+0000","last_became_active":"2023-10-16T12:49:26.767961+0000","last_became_peered":"2023-10-16T12:49:26.767961+0000","last_unstale":"2023-10-16T12:49:27.734477+0000","last_undegraded":"2023-10-16T12:49:27.734477+0000","last_fullsized":"2023-10-16T12:49:27.734477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:53:53.860402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734324+0000","last_change":"2023-10-16T12:49:26.784401+0000","last_active":"2023-10-16T12:49:27.734324+0000","last_peered":"2023-10-16T12:49:27.734324+0000","last_clean":"2023-10-16T12:49:27.734324+0000","last_became_active":"2023-10-16T12:49:26.784078+0000","last_became_peered":"2023-10-16T12:49:26.784078+0000","last_unstale":"2023-10-16T12:49:27.734324+0000","last_undegraded":"2023-10-16T12:49:27.734324+0000","last_fullsized":"2023-10-16T12:49:27.734324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:58:59.268866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740529+0000","last_change":"2023-10-16T12:49:26.774983+0000","last_active":"2023-10-16T12:49:27.740529+0000","last_peered":"2023-10-16T12:49:27.740529+0000","last_clean":"2023-10-16T12:49:27.740529+0000","last_became_active":"2023-10-16T12:49:26.774563+0000","last_became_peered":"2023-10-16T12:49:26.774563+0000","last_unstale":"2023-10-16T12:49:27.740529+0000","last_undegraded":"2023-10-16T12:49:27.740529+0000","last_fullsized":"2023-10-16T12:49:27.740529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:48.441349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.768552+0000","last_change":"2023-10-16T12:49:26.766950+0000","last_active":"2023-10-16T12:49:26.768552+0000","last_peered":"2023-10-16T12:49:26.768552+0000","last_clean":"2023-10-16T12:49:26.768552+0000","last_became_active":"2023-10-16T12:49:26.766058+0000","last_became_peered":"2023-10-16T12:49:26.766058+0000","last_unstale":"2023-10-16T12:49:26.768552+0000","last_undegraded":"2023-10-16T12:49:26.768552+0000","last_fullsized":"2023-10-16T12:49:26.768552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:29.168043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740233+0000","last_change":"2023-10-16T12:49:26.747953+0000","last_active":"2023-10-16T12:49:27.740233+0000","last_peered":"2023-10-16T12:49:27.740233+0000","last_clean":"2023-10-16T12:49:27.740233+0000","last_became_active":"2023-10-16T12:49:26.747200+0000","last_became_peered":"2023-10-16T12:49:26.747200+0000","last_unstale":"2023-10-16T12:49:27.740233+0000","last_undegraded":"2023-10-16T12:49:27.740233+0000","last_fullsized":"2023-10-16T12:49:27.740233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:43.467438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740304+0000","last_change":"2023-10-16T12:49:26.757837+0000","last_active":"2023-10-16T12:49:27.740304+0000","last_peered":"2023-10-16T12:49:27.740304+0000","last_clean":"2023-10-16T12:49:27.740304+0000","last_became_active":"2023-10-16T12:49:26.757344+0000","last_became_peered":"2023-10-16T12:49:26.757344+0000","last_unstale":"2023-10-16T12:49:27.740304+0000","last_undegraded":"2023-10-16T12:49:27.740304+0000","last_fullsized":"2023-10-16T12:49:27.740304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:41.015857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779510+0000","last_change":"2023-10-16T12:49:26.775293+0000","last_active":"2023-10-16T12:49:26.779510+0000","last_peered":"2023-10-16T12:49:26.779510+0000","last_clean":"2023-10-16T12:49:26.779510+0000","last_became_active":"2023-10-16T12:49:26.772954+0000","last_became_peered":"2023-10-16T12:49:26.772954+0000","last_unstale":"2023-10-16T12:49:26.779510+0000","last_undegraded":"2023-10-16T12:49:26.779510+0000","last_fullsized":"2023-10-16T12:49:26.779510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:46.822248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776202+0000","last_change":"2023-10-16T12:49:26.770701+0000","last_active":"2023-10-16T12:49:26.776202+0000","last_peered":"2023-10-16T12:49:26.776202+0000","last_clean":"2023-10-16T12:49:26.776202+0000","last_became_active":"2023-10-16T12:49:26.769619+0000","last_became_peered":"2023-10-16T12:49:26.769619+0000","last_unstale":"2023-10-16T12:49:26.776202+0000","last_undegraded":"2023-10-16T12:49:26.776202+0000","last_fullsized":"2023-10-16T12:49:26.776202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:17.639724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.771894+0000","last_change":"2023-10-16T12:49:26.764518+0000","last_active":"2023-10-16T12:49:26.771894+0000","last_peered":"2023-10-16T12:49:26.771894+0000","last_clean":"2023-10-16T12:49:26.771894+0000","last_became_active":"2023-10-16T12:49:26.764217+0000","last_became_peered":"2023-10-16T12:49:26.764217+0000","last_unstale":"2023-10-16T12:49:26.771894+0000","last_undegraded":"2023-10-16T12:49:26.771894+0000","last_fullsized":"2023-10-16T12:49:26.771894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:51.358206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.788018+0000","last_change":"2023-10-16T12:49:26.783981+0000","last_active":"2023-10-16T12:49:26.788018+0000","last_peered":"2023-10-16T12:49:26.788018+0000","last_clean":"2023-10-16T12:49:26.788018+0000","last_became_active":"2023-10-16T12:49:26.783696+0000","last_became_peered":"2023-10-16T12:49:26.783696+0000","last_unstale":"2023-10-16T12:49:26.788018+0000","last_undegraded":"2023-10-16T12:49:26.788018+0000","last_fullsized":"2023-10-16T12:49:26.788018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:47.251171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.785467+0000","last_change":"2023-10-16T12:49:26.780743+0000","last_active":"2023-10-16T12:49:26.785467+0000","last_peered":"2023-10-16T12:49:26.785467+0000","last_clean":"2023-10-16T12:49:26.785467+0000","last_became_active":"2023-10-16T12:49:26.780418+0000","last_became_peered":"2023-10-16T12:49:26.780418+0000","last_unstale":"2023-10-16T12:49:26.785467+0000","last_undegraded":"2023-10-16T12:49:26.785467+0000","last_fullsized":"2023-10-16T12:49:26.785467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:07.127743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.773205+0000","last_change":"2023-10-16T12:49:26.771376+0000","last_active":"2023-10-16T12:49:26.773205+0000","last_peered":"2023-10-16T12:49:26.773205+0000","last_clean":"2023-10-16T12:49:26.773205+0000","last_became_active":"2023-10-16T12:49:26.771061+0000","last_became_peered":"2023-10-16T12:49:26.771061+0000","last_unstale":"2023-10-16T12:49:26.773205+0000","last_undegraded":"2023-10-16T12:49:26.773205+0000","last_fullsized":"2023-10-16T12:49:26.773205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:48.441349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776109+0000","last_change":"2023-10-16T12:49:26.770161+0000","last_active":"2023-10-16T12:49:26.776109+0000","last_peered":"2023-10-16T12:49:26.776109+0000","last_clean":"2023-10-16T12:49:26.776109+0000","last_became_active":"2023-10-16T12:49:26.768059+0000","last_became_peered":"2023-10-16T12:49:26.768059+0000","last_unstale":"2023-10-16T12:49:26.776109+0000","last_undegraded":"2023-10-16T12:49:26.776109+0000","last_fullsized":"2023-10-16T12:49:26.776109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:26.259122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763672+0000","last_change":"2023-10-16T12:49:26.759428+0000","last_active":"2023-10-16T12:49:26.763672+0000","last_peered":"2023-10-16T12:49:26.763672+0000","last_clean":"2023-10-16T12:49:26.763672+0000","last_became_active":"2023-10-16T12:49:26.758703+0000","last_became_peered":"2023-10-16T12:49:26.758703+0000","last_unstale":"2023-10-16T12:49:26.763672+0000","last_undegraded":"2023-10-16T12:49:26.763672+0000","last_fullsized":"2023-10-16T12:49:26.763672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:13.767009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.770400+0000","last_change":"2023-10-16T12:49:26.768576+0000","last_active":"2023-10-16T12:49:26.770400+0000","last_peered":"2023-10-16T12:49:26.770400+0000","last_clean":"2023-10-16T12:49:26.770400+0000","last_became_active":"2023-10-16T12:49:26.766846+0000","last_became_peered":"2023-10-16T12:49:26.766846+0000","last_unstale":"2023-10-16T12:49:26.770400+0000","last_undegraded":"2023-10-16T12:49:26.770400+0000","last_fullsized":"2023-10-16T12:49:26.770400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:38.979498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.764786+0000","last_change":"2023-10-16T12:49:26.761191+0000","last_active":"2023-10-16T12:49:26.764786+0000","last_peered":"2023-10-16T12:49:26.764786+0000","last_clean":"2023-10-16T12:49:26.764786+0000","last_became_active":"2023-10-16T12:49:26.758480+0000","last_became_peered":"2023-10-16T12:49:26.758480+0000","last_unstale":"2023-10-16T12:49:26.764786+0000","last_undegraded":"2023-10-16T12:49:26.764786+0000","last_fullsized":"2023-10-16T12:49:26.764786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:59.443397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741456+0000","last_change":"2023-10-16T12:49:26.772360+0000","last_active":"2023-10-16T12:49:27.741456+0000","last_peered":"2023-10-16T12:49:27.741456+0000","last_clean":"2023-10-16T12:49:27.741456+0000","last_became_active":"2023-10-16T12:49:26.772036+0000","last_became_peered":"2023-10-16T12:49:26.772036+0000","last_unstale":"2023-10-16T12:49:27.741456+0000","last_undegraded":"2023-10-16T12:49:27.741456+0000","last_fullsized":"2023-10-16T12:49:27.741456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:48.142634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779040+0000","last_change":"2023-10-16T12:49:26.768467+0000","last_active":"2023-10-16T12:49:26.779040+0000","last_peered":"2023-10-16T12:49:26.779040+0000","last_clean":"2023-10-16T12:49:26.779040+0000","last_became_active":"2023-10-16T12:49:26.767054+0000","last_became_peered":"2023-10-16T12:49:26.767054+0000","last_unstale":"2023-10-16T12:49:26.779040+0000","last_undegraded":"2023-10-16T12:49:26.779040+0000","last_fullsized":"2023-10-16T12:49:26.779040+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:45.675716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.781001+0000","last_change":"2023-10-16T12:49:26.772981+0000","last_active":"2023-10-16T12:49:26.781001+0000","last_peered":"2023-10-16T12:49:26.781001+0000","last_clean":"2023-10-16T12:49:26.781001+0000","last_became_active":"2023-10-16T12:49:26.772716+0000","last_became_peered":"2023-10-16T12:49:26.772716+0000","last_unstale":"2023-10-16T12:49:26.781001+0000","last_undegraded":"2023-10-16T12:49:26.781001+0000","last_fullsized":"2023-10-16T12:49:26.781001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:00:32.951148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.782028+0000","last_change":"2023-10-16T12:49:26.774504+0000","last_active":"2023-10-16T12:49:26.782028+0000","last_peered":"2023-10-16T12:49:26.782028+0000","last_clean":"2023-10-16T12:49:26.782028+0000","last_became_active":"2023-10-16T12:49:26.774147+0000","last_became_peered":"2023-10-16T12:49:26.774147+0000","last_unstale":"2023-10-16T12:49:26.782028+0000","last_undegraded":"2023-10-16T12:49:26.782028+0000","last_fullsized":"2023-10-16T12:49:26.782028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:39.748496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.759178+0000","last_change":"2023-10-16T12:49:26.753094+0000","last_active":"2023-10-16T12:49:26.759178+0000","last_peered":"2023-10-16T12:49:26.759178+0000","last_clean":"2023-10-16T12:49:26.759178+0000","last_became_active":"2023-10-16T12:49:26.750269+0000","last_became_peered":"2023-10-16T12:49:26.750269+0000","last_unstale":"2023-10-16T12:49:26.759178+0000","last_undegraded":"2023-10-16T12:49:26.759178+0000","last_fullsized":"2023-10-16T12:49:26.759178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:28.460635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.765145+0000","last_change":"2023-10-16T12:49:26.759075+0000","last_active":"2023-10-16T12:49:26.765145+0000","last_peered":"2023-10-16T12:49:26.765145+0000","last_clean":"2023-10-16T12:49:26.765145+0000","last_became_active":"2023-10-16T12:49:26.758711+0000","last_became_peered":"2023-10-16T12:49:26.758711+0000","last_unstale":"2023-10-16T12:49:26.765145+0000","last_undegraded":"2023-10-16T12:49:26.765145+0000","last_fullsized":"2023-10-16T12:49:26.765145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:38.177995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.766121+0000","last_change":"2023-10-16T12:49:26.760582+0000","last_active":"2023-10-16T12:49:26.766121+0000","last_peered":"2023-10-16T12:49:26.766121+0000","last_clean":"2023-10-16T12:49:26.766121+0000","last_became_active":"2023-10-16T12:49:26.760130+0000","last_became_peered":"2023-10-16T12:49:26.760130+0000","last_unstale":"2023-10-16T12:49:26.766121+0000","last_undegraded":"2023-10-16T12:49:26.766121+0000","last_fullsized":"2023-10-16T12:49:26.766121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:15.191125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776427+0000","last_change":"2023-10-16T12:49:26.765739+0000","last_active":"2023-10-16T12:49:26.776427+0000","last_peered":"2023-10-16T12:49:26.776427+0000","last_clean":"2023-10-16T12:49:26.776427+0000","last_became_active":"2023-10-16T12:49:26.765420+0000","last_became_peered":"2023-10-16T12:49:26.765420+0000","last_unstale":"2023-10-16T12:49:26.776427+0000","last_undegraded":"2023-10-16T12:49:26.776427+0000","last_fullsized":"2023-10-16T12:49:26.776427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:09.290033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"85'4","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791868+0000","last_change":"2023-10-16T12:49:25.738940+0000","last_active":"2023-10-16T12:49:26.791868+0000","last_peered":"2023-10-16T12:49:26.791868+0000","last_clean":"2023-10-16T12:49:26.791868+0000","last_became_active":"2023-10-16T12:49:25.738695+0000","last_became_peered":"2023-10-16T12:49:25.738695+0000","last_unstale":"2023-10-16T12:49:26.791868+0000","last_undegraded":"2023-10-16T12:49:26.791868+0000","last_fullsized":"2023-10-16T12:49:26.791868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:47.922516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734608+0000","last_change":"2023-10-16T12:49:26.763182+0000","last_active":"2023-10-16T12:49:27.734608+0000","last_peered":"2023-10-16T12:49:27.734608+0000","last_clean":"2023-10-16T12:49:27.734608+0000","last_became_active":"2023-10-16T12:49:26.762890+0000","last_became_peered":"2023-10-16T12:49:26.762890+0000","last_unstale":"2023-10-16T12:49:27.734608+0000","last_undegraded":"2023-10-16T12:49:27.734608+0000","last_fullsized":"2023-10-16T12:49:27.734608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:26.285968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.790766+0000","last_change":"2023-10-16T12:49:25.729100+0000","last_active":"2023-10-16T12:49:26.790766+0000","last_peered":"2023-10-16T12:49:26.790766+0000","last_clean":"2023-10-16T12:49:26.790766+0000","last_became_active":"2023-10-16T12:49:25.728822+0000","last_became_peered":"2023-10-16T12:49:25.728822+0000","last_unstale":"2023-10-16T12:49:26.790766+0000","last_undegraded":"2023-10-16T12:49:26.790766+0000","last_fullsized":"2023-10-16T12:49:26.790766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:44.209674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.766531+0000","last_change":"2023-10-16T12:49:26.763254+0000","last_active":"2023-10-16T12:49:26.766531+0000","last_peered":"2023-10-16T12:49:26.766531+0000","last_clean":"2023-10-16T12:49:26.766531+0000","last_became_active":"2023-10-16T12:49:26.762909+0000","last_became_peered":"2023-10-16T12:49:26.762909+0000","last_unstale":"2023-10-16T12:49:26.766531+0000","last_undegraded":"2023-10-16T12:49:26.766531+0000","last_fullsized":"2023-10-16T12:49:26.766531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:46.592546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.789854+0000","last_change":"2023-10-16T12:49:25.727459+0000","last_active":"2023-10-16T12:49:26.789854+0000","last_peered":"2023-10-16T12:49:26.789854+0000","last_clean":"2023-10-16T12:49:26.789854+0000","last_became_active":"2023-10-16T12:49:25.725586+0000","last_became_peered":"2023-10-16T12:49:25.725586+0000","last_unstale":"2023-10-16T12:49:26.789854+0000","last_undegraded":"2023-10-16T12:49:26.789854+0000","last_fullsized":"2023-10-16T12:49:26.789854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:55.417552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.778881+0000","last_change":"2023-10-16T12:49:26.774882+0000","last_active":"2023-10-16T12:49:26.778881+0000","last_peered":"2023-10-16T12:49:26.778881+0000","last_clean":"2023-10-16T12:49:26.778881+0000","last_became_active":"2023-10-16T12:49:26.769371+0000","last_became_peered":"2023-10-16T12:49:26.769371+0000","last_unstale":"2023-10-16T12:49:26.778881+0000","last_undegraded":"2023-10-16T12:49:26.778881+0000","last_fullsized":"2023-10-16T12:49:26.778881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:07.389207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"85'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740842+0000","last_change":"2023-10-16T12:49:25.742509+0000","last_active":"2023-10-16T12:49:27.740842+0000","last_peered":"2023-10-16T12:49:27.740842+0000","last_clean":"2023-10-16T12:49:27.740842+0000","last_became_active":"2023-10-16T12:49:25.742140+0000","last_became_peered":"2023-10-16T12:49:25.742140+0000","last_unstale":"2023-10-16T12:49:27.740842+0000","last_undegraded":"2023-10-16T12:49:27.740842+0000","last_fullsized":"2023-10-16T12:49:27.740842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:58.098283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.780328+0000","last_change":"2023-10-16T12:49:26.772637+0000","last_active":"2023-10-16T12:49:26.780328+0000","last_peered":"2023-10-16T12:49:26.780328+0000","last_clean":"2023-10-16T12:49:26.780328+0000","last_became_active":"2023-10-16T12:49:26.770770+0000","last_became_peered":"2023-10-16T12:49:26.770770+0000","last_unstale":"2023-10-16T12:49:26.780328+0000","last_undegraded":"2023-10-16T12:49:26.780328+0000","last_fullsized":"2023-10-16T12:49:26.780328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:48.142634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739047+0000","last_change":"2023-10-16T12:49:25.734734+0000","last_active":"2023-10-16T12:49:27.739047+0000","last_peered":"2023-10-16T12:49:27.739047+0000","last_clean":"2023-10-16T12:49:27.739047+0000","last_became_active":"2023-10-16T12:49:25.734419+0000","last_became_peered":"2023-10-16T12:49:25.734419+0000","last_unstale":"2023-10-16T12:49:27.739047+0000","last_undegraded":"2023-10-16T12:49:27.739047+0000","last_fullsized":"2023-10-16T12:49:27.739047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:21.521408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.790365+0000","last_change":"2023-10-16T12:49:26.782630+0000","last_active":"2023-10-16T12:49:26.790365+0000","last_peered":"2023-10-16T12:49:26.790365+0000","last_clean":"2023-10-16T12:49:26.790365+0000","last_became_active":"2023-10-16T12:49:26.782342+0000","last_became_peered":"2023-10-16T12:49:26.782342+0000","last_unstale":"2023-10-16T12:49:26.790365+0000","last_undegraded":"2023-10-16T12:49:26.790365+0000","last_fullsized":"2023-10-16T12:49:26.790365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:55.184341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.789428+0000","last_change":"2023-10-16T12:49:25.739996+0000","last_active":"2023-10-16T12:49:26.789428+0000","last_peered":"2023-10-16T12:49:26.789428+0000","last_clean":"2023-10-16T12:49:26.789428+0000","last_became_active":"2023-10-16T12:49:25.737324+0000","last_became_peered":"2023-10-16T12:49:25.737324+0000","last_unstale":"2023-10-16T12:49:26.789428+0000","last_undegraded":"2023-10-16T12:49:26.789428+0000","last_fullsized":"2023-10-16T12:49:26.789428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:40.172041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.768769+0000","last_change":"2023-10-16T12:49:26.766836+0000","last_active":"2023-10-16T12:49:26.768769+0000","last_peered":"2023-10-16T12:49:26.768769+0000","last_clean":"2023-10-16T12:49:26.768769+0000","last_became_active":"2023-10-16T12:49:26.766396+0000","last_became_peered":"2023-10-16T12:49:26.766396+0000","last_unstale":"2023-10-16T12:49:26.768769+0000","last_undegraded":"2023-10-16T12:49:26.768769+0000","last_fullsized":"2023-10-16T12:49:26.768769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:41.754039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.738932+0000","last_change":"2023-10-16T12:49:25.738490+0000","last_active":"2023-10-16T12:49:27.738932+0000","last_peered":"2023-10-16T12:49:27.738932+0000","last_clean":"2023-10-16T12:49:27.738932+0000","last_became_active":"2023-10-16T12:49:25.738207+0000","last_became_peered":"2023-10-16T12:49:25.738207+0000","last_unstale":"2023-10-16T12:49:27.738932+0000","last_undegraded":"2023-10-16T12:49:27.738932+0000","last_fullsized":"2023-10-16T12:49:27.738932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:12.277394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776089+0000","last_change":"2023-10-16T12:49:26.770727+0000","last_active":"2023-10-16T12:49:26.776089+0000","last_peered":"2023-10-16T12:49:26.776089+0000","last_clean":"2023-10-16T12:49:26.776089+0000","last_became_active":"2023-10-16T12:49:26.770237+0000","last_became_peered":"2023-10-16T12:49:26.770237+0000","last_unstale":"2023-10-16T12:49:26.776089+0000","last_undegraded":"2023-10-16T12:49:26.776089+0000","last_fullsized":"2023-10-16T12:49:26.776089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:48.441349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739954+0000","last_change":"2023-10-16T12:49:25.725211+0000","last_active":"2023-10-16T12:49:27.739954+0000","last_peered":"2023-10-16T12:49:27.739954+0000","last_clean":"2023-10-16T12:49:27.739954+0000","last_became_active":"2023-10-16T12:49:25.724905+0000","last_became_peered":"2023-10-16T12:49:25.724905+0000","last_unstale":"2023-10-16T12:49:27.739954+0000","last_undegraded":"2023-10-16T12:49:27.739954+0000","last_fullsized":"2023-10-16T12:49:27.739954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:48.520263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739853+0000","last_change":"2023-10-16T12:49:26.764605+0000","last_active":"2023-10-16T12:49:27.739853+0000","last_peered":"2023-10-16T12:49:27.739853+0000","last_clean":"2023-10-16T12:49:27.739853+0000","last_became_active":"2023-10-16T12:49:26.763311+0000","last_became_peered":"2023-10-16T12:49:26.763311+0000","last_unstale":"2023-10-16T12:49:27.739853+0000","last_undegraded":"2023-10-16T12:49:27.739853+0000","last_fullsized":"2023-10-16T12:49:27.739853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:07.738593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.749356+0000","last_change":"2023-10-16T12:49:25.736697+0000","last_active":"2023-10-16T12:49:26.749356+0000","last_peered":"2023-10-16T12:49:26.749356+0000","last_clean":"2023-10-16T12:49:26.749356+0000","last_became_active":"2023-10-16T12:49:25.736409+0000","last_became_peered":"2023-10-16T12:49:25.736409+0000","last_unstale":"2023-10-16T12:49:26.749356+0000","last_undegraded":"2023-10-16T12:49:26.749356+0000","last_fullsized":"2023-10-16T12:49:26.749356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:54.178505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.772356+0000","last_change":"2023-10-16T12:49:26.765659+0000","last_active":"2023-10-16T12:49:26.772356+0000","last_peered":"2023-10-16T12:49:26.772356+0000","last_clean":"2023-10-16T12:49:26.772356+0000","last_became_active":"2023-10-16T12:49:26.765150+0000","last_became_peered":"2023-10-16T12:49:26.765150+0000","last_unstale":"2023-10-16T12:49:26.772356+0000","last_undegraded":"2023-10-16T12:49:26.772356+0000","last_fullsized":"2023-10-16T12:49:26.772356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:17.639724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.761151+0000","last_change":"2023-10-16T12:49:25.736947+0000","last_active":"2023-10-16T12:49:26.761151+0000","last_peered":"2023-10-16T12:49:26.761151+0000","last_clean":"2023-10-16T12:49:26.761151+0000","last_became_active":"2023-10-16T12:49:25.736678+0000","last_became_peered":"2023-10-16T12:49:25.736678+0000","last_unstale":"2023-10-16T12:49:26.761151+0000","last_undegraded":"2023-10-16T12:49:26.761151+0000","last_fullsized":"2023-10-16T12:49:26.761151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:14.911519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733943+0000","last_change":"2023-10-16T12:49:26.762298+0000","last_active":"2023-10-16T12:49:27.733943+0000","last_peered":"2023-10-16T12:49:27.733943+0000","last_clean":"2023-10-16T12:49:27.733943+0000","last_became_active":"2023-10-16T12:49:26.761954+0000","last_became_peered":"2023-10-16T12:49:26.761954+0000","last_unstale":"2023-10-16T12:49:27.733943+0000","last_undegraded":"2023-10-16T12:49:27.733943+0000","last_fullsized":"2023-10-16T12:49:27.733943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:30.628184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"85'3","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.788610+0000","last_change":"2023-10-16T12:49:25.737273+0000","last_active":"2023-10-16T12:49:26.788610+0000","last_peered":"2023-10-16T12:49:26.788610+0000","last_clean":"2023-10-16T12:49:26.788610+0000","last_became_active":"2023-10-16T12:49:25.737008+0000","last_became_peered":"2023-10-16T12:49:25.737008+0000","last_unstale":"2023-10-16T12:49:26.788610+0000","last_undegraded":"2023-10-16T12:49:26.788610+0000","last_fullsized":"2023-10-16T12:49:26.788610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:38.742660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763428+0000","last_change":"2023-10-16T12:49:26.761793+0000","last_active":"2023-10-16T12:49:26.763428+0000","last_peered":"2023-10-16T12:49:26.763428+0000","last_clean":"2023-10-16T12:49:26.763428+0000","last_became_active":"2023-10-16T12:49:26.761093+0000","last_became_peered":"2023-10-16T12:49:26.761093+0000","last_unstale":"2023-10-16T12:49:26.763428+0000","last_undegraded":"2023-10-16T12:49:26.763428+0000","last_fullsized":"2023-10-16T12:49:26.763428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.315108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"85'6","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739449+0000","last_change":"2023-10-16T12:49:25.735817+0000","last_active":"2023-10-16T12:49:27.739449+0000","last_peered":"2023-10-16T12:49:27.739449+0000","last_clean":"2023-10-16T12:49:27.739449+0000","last_became_active":"2023-10-16T12:49:25.735465+0000","last_became_peered":"2023-10-16T12:49:25.735465+0000","last_unstale":"2023-10-16T12:49:27.739449+0000","last_undegraded":"2023-10-16T12:49:27.739449+0000","last_fullsized":"2023-10-16T12:49:27.739449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:58.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739400+0000","last_change":"2023-10-16T12:49:26.769826+0000","last_active":"2023-10-16T12:49:27.739400+0000","last_peered":"2023-10-16T12:49:27.739400+0000","last_clean":"2023-10-16T12:49:27.739400+0000","last_became_active":"2023-10-16T12:49:26.769529+0000","last_became_peered":"2023-10-16T12:49:26.769529+0000","last_unstale":"2023-10-16T12:49:27.739400+0000","last_undegraded":"2023-10-16T12:49:27.739400+0000","last_fullsized":"2023-10-16T12:49:27.739400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:19.089644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791402+0000","last_change":"2023-10-16T12:49:25.733869+0000","last_active":"2023-10-16T12:49:26.791402+0000","last_peered":"2023-10-16T12:49:26.791402+0000","last_clean":"2023-10-16T12:49:26.791402+0000","last_became_active":"2023-10-16T12:49:25.733469+0000","last_became_peered":"2023-10-16T12:49:25.733469+0000","last_unstale":"2023-10-16T12:49:26.791402+0000","last_undegraded":"2023-10-16T12:49:26.791402+0000","last_fullsized":"2023-10-16T12:49:26.791402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:26.495695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.771949+0000","last_change":"2023-10-16T12:49:26.766534+0000","last_active":"2023-10-16T12:49:26.771949+0000","last_peered":"2023-10-16T12:49:26.771949+0000","last_clean":"2023-10-16T12:49:26.771949+0000","last_became_active":"2023-10-16T12:49:26.766248+0000","last_became_peered":"2023-10-16T12:49:26.766248+0000","last_unstale":"2023-10-16T12:49:26.771949+0000","last_undegraded":"2023-10-16T12:49:26.771949+0000","last_fullsized":"2023-10-16T12:49:26.771949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:08.747676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.746976+0000","last_change":"2023-10-16T12:49:25.739833+0000","last_active":"2023-10-16T12:49:26.746976+0000","last_peered":"2023-10-16T12:49:26.746976+0000","last_clean":"2023-10-16T12:49:26.746976+0000","last_became_active":"2023-10-16T12:49:25.737220+0000","last_became_peered":"2023-10-16T12:49:25.737220+0000","last_unstale":"2023-10-16T12:49:26.746976+0000","last_undegraded":"2023-10-16T12:49:26.746976+0000","last_fullsized":"2023-10-16T12:49:26.746976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:42.104908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.737992+0000","last_change":"2023-10-16T12:49:26.751234+0000","last_active":"2023-10-16T12:49:27.737992+0000","last_peered":"2023-10-16T12:49:27.737992+0000","last_clean":"2023-10-16T12:49:27.737992+0000","last_became_active":"2023-10-16T12:49:26.750473+0000","last_became_peered":"2023-10-16T12:49:26.750473+0000","last_unstale":"2023-10-16T12:49:27.737992+0000","last_undegraded":"2023-10-16T12:49:27.737992+0000","last_fullsized":"2023-10-16T12:49:27.737992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:53.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739138+0000","last_change":"2023-10-16T12:49:25.740744+0000","last_active":"2023-10-16T12:49:27.739138+0000","last_peered":"2023-10-16T12:49:27.739138+0000","last_clean":"2023-10-16T12:49:27.739138+0000","last_became_active":"2023-10-16T12:49:25.740308+0000","last_became_peered":"2023-10-16T12:49:25.740308+0000","last_unstale":"2023-10-16T12:49:27.739138+0000","last_undegraded":"2023-10-16T12:49:27.739138+0000","last_fullsized":"2023-10-16T12:49:27.739138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:21.539277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763650+0000","last_change":"2023-10-16T12:49:26.759085+0000","last_active":"2023-10-16T12:49:26.763650+0000","last_peered":"2023-10-16T12:49:26.763650+0000","last_clean":"2023-10-16T12:49:26.763650+0000","last_became_active":"2023-10-16T12:49:26.758138+0000","last_became_peered":"2023-10-16T12:49:26.758138+0000","last_unstale":"2023-10-16T12:49:26.763650+0000","last_undegraded":"2023-10-16T12:49:26.763650+0000","last_fullsized":"2023-10-16T12:49:26.763650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:19.089644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791026+0000","last_change":"2023-10-16T12:49:25.740357+0000","last_active":"2023-10-16T12:49:26.791026+0000","last_peered":"2023-10-16T12:49:26.791026+0000","last_clean":"2023-10-16T12:49:26.791026+0000","last_became_active":"2023-10-16T12:49:25.739455+0000","last_became_peered":"2023-10-16T12:49:25.739455+0000","last_unstale":"2023-10-16T12:49:26.791026+0000","last_undegraded":"2023-10-16T12:49:26.791026+0000","last_fullsized":"2023-10-16T12:49:26.791026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:16.140142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763722+0000","last_change":"2023-10-16T12:49:26.759318+0000","last_active":"2023-10-16T12:49:26.763722+0000","last_peered":"2023-10-16T12:49:26.763722+0000","last_clean":"2023-10-16T12:49:26.763722+0000","last_became_active":"2023-10-16T12:49:26.757244+0000","last_became_peered":"2023-10-16T12:49:26.757244+0000","last_unstale":"2023-10-16T12:49:26.763722+0000","last_undegraded":"2023-10-16T12:49:26.763722+0000","last_fullsized":"2023-10-16T12:49:26.763722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:52.310280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"85'1","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.790360+0000","last_change":"2023-10-16T12:49:25.733778+0000","last_active":"2023-10-16T12:49:26.790360+0000","last_peered":"2023-10-16T12:49:26.790360+0000","last_clean":"2023-10-16T12:49:26.790360+0000","last_became_active":"2023-10-16T12:49:25.733497+0000","last_became_peered":"2023-10-16T12:49:25.733497+0000","last_unstale":"2023-10-16T12:49:26.790360+0000","last_undegraded":"2023-10-16T12:49:26.790360+0000","last_fullsized":"2023-10-16T12:49:26.790360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:04.144571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741065+0000","last_change":"2023-10-16T12:49:26.774187+0000","last_active":"2023-10-16T12:49:27.741065+0000","last_peered":"2023-10-16T12:49:27.741065+0000","last_clean":"2023-10-16T12:49:27.741065+0000","last_became_active":"2023-10-16T12:49:26.773874+0000","last_became_peered":"2023-10-16T12:49:26.773874+0000","last_unstale":"2023-10-16T12:49:27.741065+0000","last_undegraded":"2023-10-16T12:49:27.741065+0000","last_fullsized":"2023-10-16T12:49:27.741065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:52.310280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791297+0000","last_change":"2023-10-16T12:49:25.738021+0000","last_active":"2023-10-16T12:49:26.791297+0000","last_peered":"2023-10-16T12:49:26.791297+0000","last_clean":"2023-10-16T12:49:26.791297+0000","last_became_active":"2023-10-16T12:49:25.737704+0000","last_became_peered":"2023-10-16T12:49:25.737704+0000","last_unstale":"2023-10-16T12:49:26.791297+0000","last_undegraded":"2023-10-16T12:49:26.791297+0000","last_fullsized":"2023-10-16T12:49:26.791297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:14.431428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.781489+0000","last_change":"2023-10-16T12:49:26.774938+0000","last_active":"2023-10-16T12:49:26.781489+0000","last_peered":"2023-10-16T12:49:26.781489+0000","last_clean":"2023-10-16T12:49:26.781489+0000","last_became_active":"2023-10-16T12:49:26.774692+0000","last_became_peered":"2023-10-16T12:49:26.774692+0000","last_unstale":"2023-10-16T12:49:26.781489+0000","last_undegraded":"2023-10-16T12:49:26.781489+0000","last_fullsized":"2023-10-16T12:49:26.781489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:48.445429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741148+0000","last_change":"2023-10-16T12:49:25.742216+0000","last_active":"2023-10-16T12:49:27.741148+0000","last_peered":"2023-10-16T12:49:27.741148+0000","last_clean":"2023-10-16T12:49:27.741148+0000","last_became_active":"2023-10-16T12:49:25.741888+0000","last_became_peered":"2023-10-16T12:49:25.741888+0000","last_unstale":"2023-10-16T12:49:27.741148+0000","last_undegraded":"2023-10-16T12:49:27.741148+0000","last_fullsized":"2023-10-16T12:49:27.741148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:12.761173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.763412+0000","last_change":"2023-10-16T12:49:26.761491+0000","last_active":"2023-10-16T12:49:26.763412+0000","last_peered":"2023-10-16T12:49:26.763412+0000","last_clean":"2023-10-16T12:49:26.763412+0000","last_became_active":"2023-10-16T12:49:26.760444+0000","last_became_peered":"2023-10-16T12:49:26.760444+0000","last_unstale":"2023-10-16T12:49:26.763412+0000","last_undegraded":"2023-10-16T12:49:26.763412+0000","last_fullsized":"2023-10-16T12:49:26.763412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:26.259122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.748006+0000","last_change":"2023-10-16T12:49:25.725852+0000","last_active":"2023-10-16T12:49:26.748006+0000","last_peered":"2023-10-16T12:49:26.748006+0000","last_clean":"2023-10-16T12:49:26.748006+0000","last_became_active":"2023-10-16T12:49:25.725483+0000","last_became_peered":"2023-10-16T12:49:25.725483+0000","last_unstale":"2023-10-16T12:49:26.748006+0000","last_undegraded":"2023-10-16T12:49:26.748006+0000","last_fullsized":"2023-10-16T12:49:26.748006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:58.209696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739800+0000","last_change":"2023-10-16T12:49:26.772707+0000","last_active":"2023-10-16T12:49:27.739800+0000","last_peered":"2023-10-16T12:49:27.739800+0000","last_clean":"2023-10-16T12:49:27.739800+0000","last_became_active":"2023-10-16T12:49:26.771407+0000","last_became_peered":"2023-10-16T12:49:26.771407+0000","last_unstale":"2023-10-16T12:49:27.739800+0000","last_undegraded":"2023-10-16T12:49:27.739800+0000","last_fullsized":"2023-10-16T12:49:27.739800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:28.460635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.791513+0000","last_change":"2023-10-16T12:49:25.733089+0000","last_active":"2023-10-16T12:49:26.791513+0000","last_peered":"2023-10-16T12:49:26.791513+0000","last_clean":"2023-10-16T12:49:26.791513+0000","last_became_active":"2023-10-16T12:49:25.732748+0000","last_became_peered":"2023-10-16T12:49:25.732748+0000","last_unstale":"2023-10-16T12:49:26.791513+0000","last_undegraded":"2023-10-16T12:49:26.791513+0000","last_fullsized":"2023-10-16T12:49:26.791513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:41:58.240759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779115+0000","last_change":"2023-10-16T12:49:26.775681+0000","last_active":"2023-10-16T12:49:26.779115+0000","last_peered":"2023-10-16T12:49:26.779115+0000","last_clean":"2023-10-16T12:49:26.779115+0000","last_became_active":"2023-10-16T12:49:26.772475+0000","last_became_peered":"2023-10-16T12:49:26.772475+0000","last_unstale":"2023-10-16T12:49:26.779115+0000","last_undegraded":"2023-10-16T12:49:26.779115+0000","last_fullsized":"2023-10-16T12:49:26.779115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:39.534158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733821+0000","last_change":"2023-10-16T12:49:25.735427+0000","last_active":"2023-10-16T12:49:27.733821+0000","last_peered":"2023-10-16T12:49:27.733821+0000","last_clean":"2023-10-16T12:49:27.733821+0000","last_became_active":"2023-10-16T12:49:25.735153+0000","last_became_peered":"2023-10-16T12:49:25.735153+0000","last_unstale":"2023-10-16T12:49:27.733821+0000","last_undegraded":"2023-10-16T12:49:27.733821+0000","last_fullsized":"2023-10-16T12:49:27.733821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:58.895115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.766149+0000","last_change":"2023-10-16T12:49:26.761804+0000","last_active":"2023-10-16T12:49:26.766149+0000","last_peered":"2023-10-16T12:49:26.766149+0000","last_clean":"2023-10-16T12:49:26.766149+0000","last_became_active":"2023-10-16T12:49:26.761501+0000","last_became_peered":"2023-10-16T12:49:26.761501+0000","last_unstale":"2023-10-16T12:49:26.766149+0000","last_undegraded":"2023-10-16T12:49:26.766149+0000","last_fullsized":"2023-10-16T12:49:26.766149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:20.304637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.738636+0000","last_change":"2023-10-16T12:49:25.738870+0000","last_active":"2023-10-16T12:49:27.738636+0000","last_peered":"2023-10-16T12:49:27.738636+0000","last_clean":"2023-10-16T12:49:27.738636+0000","last_became_active":"2023-10-16T12:49:25.738558+0000","last_became_peered":"2023-10-16T12:49:25.738558+0000","last_unstale":"2023-10-16T12:49:27.738636+0000","last_undegraded":"2023-10-16T12:49:27.738636+0000","last_fullsized":"2023-10-16T12:49:27.738636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:04:38.702610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741517+0000","last_change":"2023-10-16T12:49:26.771786+0000","last_active":"2023-10-16T12:49:27.741517+0000","last_peered":"2023-10-16T12:49:27.741517+0000","last_clean":"2023-10-16T12:49:27.741517+0000","last_became_active":"2023-10-16T12:49:26.769157+0000","last_became_peered":"2023-10-16T12:49:26.769157+0000","last_unstale":"2023-10-16T12:49:27.741517+0000","last_undegraded":"2023-10-16T12:49:27.741517+0000","last_fullsized":"2023-10-16T12:49:27.741517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:41.176774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"85'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734710+0000","last_change":"2023-10-16T12:49:25.725015+0000","last_active":"2023-10-16T12:49:27.734710+0000","last_peered":"2023-10-16T12:49:27.734710+0000","last_clean":"2023-10-16T12:49:27.734710+0000","last_became_active":"2023-10-16T12:49:25.724672+0000","last_became_peered":"2023-10-16T12:49:25.724672+0000","last_unstale":"2023-10-16T12:49:27.734710+0000","last_undegraded":"2023-10-16T12:49:27.734710+0000","last_fullsized":"2023-10-16T12:49:27.734710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:30.631949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.741194+0000","last_change":"2023-10-16T12:49:26.774785+0000","last_active":"2023-10-16T12:49:27.741194+0000","last_peered":"2023-10-16T12:49:27.741194+0000","last_clean":"2023-10-16T12:49:27.741194+0000","last_became_active":"2023-10-16T12:49:26.774391+0000","last_became_peered":"2023-10-16T12:49:26.774391+0000","last_unstale":"2023-10-16T12:49:27.741194+0000","last_undegraded":"2023-10-16T12:49:27.741194+0000","last_fullsized":"2023-10-16T12:49:27.741194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:08.747676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.756760+0000","last_change":"2023-10-16T12:49:25.738491+0000","last_active":"2023-10-16T12:49:26.756760+0000","last_peered":"2023-10-16T12:49:26.756760+0000","last_clean":"2023-10-16T12:49:26.756760+0000","last_became_active":"2023-10-16T12:49:25.738080+0000","last_became_peered":"2023-10-16T12:49:25.738080+0000","last_unstale":"2023-10-16T12:49:26.756760+0000","last_undegraded":"2023-10-16T12:49:26.756760+0000","last_fullsized":"2023-10-16T12:49:26.756760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:43.292714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.734259+0000","last_change":"2023-10-16T12:49:26.770163+0000","last_active":"2023-10-16T12:49:27.734259+0000","last_peered":"2023-10-16T12:49:27.734259+0000","last_clean":"2023-10-16T12:49:27.734259+0000","last_became_active":"2023-10-16T12:49:26.768935+0000","last_became_peered":"2023-10-16T12:49:26.768935+0000","last_unstale":"2023-10-16T12:49:27.734259+0000","last_undegraded":"2023-10-16T12:49:27.734259+0000","last_fullsized":"2023-10-16T12:49:27.734259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:21.378894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.747496+0000","last_change":"2023-10-16T12:49:25.739372+0000","last_active":"2023-10-16T12:49:26.747496+0000","last_peered":"2023-10-16T12:49:26.747496+0000","last_clean":"2023-10-16T12:49:26.747496+0000","last_became_active":"2023-10-16T12:49:25.738093+0000","last_became_peered":"2023-10-16T12:49:25.738093+0000","last_unstale":"2023-10-16T12:49:26.747496+0000","last_undegraded":"2023-10-16T12:49:26.747496+0000","last_fullsized":"2023-10-16T12:49:26.747496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:06.625845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.779060+0000","last_change":"2023-10-16T12:49:26.775988+0000","last_active":"2023-10-16T12:49:26.779060+0000","last_peered":"2023-10-16T12:49:26.779060+0000","last_clean":"2023-10-16T12:49:26.779060+0000","last_became_active":"2023-10-16T12:49:26.769786+0000","last_became_peered":"2023-10-16T12:49:26.769786+0000","last_unstale":"2023-10-16T12:49:26.779060+0000","last_undegraded":"2023-10-16T12:49:26.779060+0000","last_fullsized":"2023-10-16T12:49:26.779060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:27.642654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.740771+0000","last_change":"2023-10-16T12:49:25.737922+0000","last_active":"2023-10-16T12:49:27.740771+0000","last_peered":"2023-10-16T12:49:27.740771+0000","last_clean":"2023-10-16T12:49:27.740771+0000","last_became_active":"2023-10-16T12:49:25.737584+0000","last_became_peered":"2023-10-16T12:49:25.737584+0000","last_unstale":"2023-10-16T12:49:27.740771+0000","last_undegraded":"2023-10-16T12:49:27.740771+0000","last_fullsized":"2023-10-16T12:49:27.740771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:16.140142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.771056+0000","last_change":"2023-10-16T12:49:26.760148+0000","last_active":"2023-10-16T12:49:26.771056+0000","last_peered":"2023-10-16T12:49:26.771056+0000","last_clean":"2023-10-16T12:49:26.771056+0000","last_became_active":"2023-10-16T12:49:26.759818+0000","last_became_peered":"2023-10-16T12:49:26.759818+0000","last_unstale":"2023-10-16T12:49:26.771056+0000","last_undegraded":"2023-10-16T12:49:26.771056+0000","last_fullsized":"2023-10-16T12:49:26.771056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:52.310280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733696+0000","last_change":"2023-10-16T12:49:25.738179+0000","last_active":"2023-10-16T12:49:27.733696+0000","last_peered":"2023-10-16T12:49:27.733696+0000","last_clean":"2023-10-16T12:49:27.733696+0000","last_became_active":"2023-10-16T12:49:25.737864+0000","last_became_peered":"2023-10-16T12:49:25.737864+0000","last_unstale":"2023-10-16T12:49:27.733696+0000","last_undegraded":"2023-10-16T12:49:27.733696+0000","last_fullsized":"2023-10-16T12:49:27.733696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:15.150187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.773224+0000","last_change":"2023-10-16T12:49:26.771861+0000","last_active":"2023-10-16T12:49:26.773224+0000","last_peered":"2023-10-16T12:49:26.773224+0000","last_clean":"2023-10-16T12:49:26.773224+0000","last_became_active":"2023-10-16T12:49:26.771579+0000","last_became_peered":"2023-10-16T12:49:26.771579+0000","last_unstale":"2023-10-16T12:49:26.773224+0000","last_undegraded":"2023-10-16T12:49:26.773224+0000","last_fullsized":"2023-10-16T12:49:26.773224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:41.176774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.739384+0000","last_change":"2023-10-16T12:49:25.732369+0000","last_active":"2023-10-16T12:49:27.739384+0000","last_peered":"2023-10-16T12:49:27.739384+0000","last_clean":"2023-10-16T12:49:27.739384+0000","last_became_active":"2023-10-16T12:49:25.732052+0000","last_became_peered":"2023-10-16T12:49:25.732052+0000","last_unstale":"2023-10-16T12:49:27.739384+0000","last_undegraded":"2023-10-16T12:49:27.739384+0000","last_fullsized":"2023-10-16T12:49:27.739384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:41.571703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733717+0000","last_change":"2023-10-16T12:49:26.769061+0000","last_active":"2023-10-16T12:49:27.733717+0000","last_peered":"2023-10-16T12:49:27.733717+0000","last_clean":"2023-10-16T12:49:27.733717+0000","last_became_active":"2023-10-16T12:49:26.768744+0000","last_became_peered":"2023-10-16T12:49:26.768744+0000","last_unstale":"2023-10-16T12:49:27.733717+0000","last_undegraded":"2023-10-16T12:49:27.733717+0000","last_fullsized":"2023-10-16T12:49:27.733717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:40.666471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-16T12:49:27.733996+0000","last_change":"2023-10-16T12:49:25.726132+0000","last_active":"2023-10-16T12:49:27.733996+0000","last_peered":"2023-10-16T12:49:27.733996+0000","last_clean":"2023-10-16T12:49:27.733996+0000","last_became_active":"2023-10-16T12:49:25.725830+0000","last_became_peered":"2023-10-16T12:49:25.725830+0000","last_unstale":"2023-10-16T12:49:27.733996+0000","last_undegraded":"2023-10-16T12:49:27.733996+0000","last_fullsized":"2023-10-16T12:49:27.733996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:57.967543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.776068+0000","last_change":"2023-10-16T12:49:26.770503+0000","last_active":"2023-10-16T12:49:26.776068+0000","last_peered":"2023-10-16T12:49:26.776068+0000","last_clean":"2023-10-16T12:49:26.776068+0000","last_became_active":"2023-10-16T12:49:26.769871+0000","last_became_peered":"2023-10-16T12:49:26.769871+0000","last_unstale":"2023-10-16T12:49:26.776068+0000","last_undegraded":"2023-10-16T12:49:26.776068+0000","last_fullsized":"2023-10-16T12:49:26.776068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:51.358206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.760843+0000","last_change":"2023-10-16T12:49:25.739754+0000","last_active":"2023-10-16T12:49:26.760843+0000","last_peered":"2023-10-16T12:49:26.760843+0000","last_clean":"2023-10-16T12:49:26.760843+0000","last_became_active":"2023-10-16T12:49:25.739259+0000","last_became_peered":"2023-10-16T12:49:25.739259+0000","last_unstale":"2023-10-16T12:49:26.760843+0000","last_undegraded":"2023-10-16T12:49:26.760843+0000","last_fullsized":"2023-10-16T12:49:26.760843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:09.694841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.780347+0000","last_change":"2023-10-16T12:49:26.771700+0000","last_active":"2023-10-16T12:49:26.780347+0000","last_peered":"2023-10-16T12:49:26.780347+0000","last_clean":"2023-10-16T12:49:26.780347+0000","last_became_active":"2023-10-16T12:49:26.771262+0000","last_became_peered":"2023-10-16T12:49:26.771262+0000","last_unstale":"2023-10-16T12:49:26.780347+0000","last_undegraded":"2023-10-16T12:49:26.780347+0000","last_fullsized":"2023-10-16T12:49:26.780347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:25.696135+0000","last_clean_scrub_stamp":"2023-10-16T12:49:25.696135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:08.747676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:49:26.746910+0000","last_change":"2023-10-16T12:49:25.739319+0000","last_active":"2023-10-16T12:49:26.746910+0000","last_peered":"2023-10-16T12:49:26.746910+0000","last_clean":"2023-10-16T12:49:26.746910+0000","last_became_active":"2023-10-16T12:49:25.739043+0000","last_became_peered":"2023-10-16T12:49:25.739043+0000","last_unstale":"2023-10-16T12:49:26.746910+0000","last_undegraded":"2023-10-16T12:49:26.746910+0000","last_fullsized":"2023-10-16T12:49:26.746910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:49:24.690299+0000","last_clean_scrub_stamp":"2023-10-16T12:49:24.690299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:54.566500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":384,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:49:25.728711+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:49:25.728711+0000","last_peered":"2023-10-16T12:49:25.728711+0000","last_clean":"2023-10-16T12:49:25.728711+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:49:25.728711+0000","last_undegraded":"2023-10-16T12:49:25.728711+0000","last_fullsized":"2023-10-16T12:49:25.728711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542338,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1626112,"data_stored":1077229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.3,"15min":1.294},"min":{"1min":0.73499999999999999,"5min":0.66100000000000003,"15min":0.65100000000000002},"max":{"1min":1.72,"5min":1.72,"15min":2.1890000000000001},"last":1.728},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.325,"15min":1.3440000000000001},"min":{"1min":1.0089999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6479999999999999,"5min":1.8420000000000001,"15min":1.8899999999999999},"last":1.5940000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.218,"15min":1.2170000000000001},"min":{"1min":0.873,"5min":0.72199999999999998,"15min":0.67300000000000004},"max":{"1min":1.6539999999999999,"5min":1.78,"15min":1.78},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1850000000000001,"15min":1.208},"min":{"1min":0.875,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.667,"5min":1.6779999999999999,"15min":1.843},"last":1.2430000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.204,"15min":1.2090000000000001},"min":{"1min":0.94899999999999995,"5min":0.71299999999999997,"15min":0.59799999999999998},"max":{"1min":1.5880000000000001,"5min":1.591,"15min":2.2130000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2909999999999999,"15min":1.3260000000000001},"min":{"1min":1.0449999999999999,"5min":0.81899999999999995,"15min":0.63500000000000001},"max":{"1min":1.5549999999999999,"5min":1.7230000000000001,"15min":2.3450000000000002},"last":1.5129999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.292,"15min":1.3},"min":{"1min":0.879,"5min":0.80000000000000004,"15min":0.69099999999999995},"max":{"1min":1.847,"5min":1.847,"15min":2.0150000000000001},"last":1.429},{"interface":"front","average":{"1min":1.417,"5min":1.395,"15min":1.3999999999999999},"min":{"1min":1.006,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.661,"5min":1.8009999999999999,"15min":2.0019999999999998},"last":1.385}]},{"osd":4,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.153,"15min":1.194},"min":{"1min":0.86699999999999999,"5min":0.72099999999999997,"15min":0.67300000000000004},"max":{"1min":1.597,"5min":1.7030000000000001,"15min":1.9710000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.341,"15min":1.3440000000000001},"min":{"1min":0.98299999999999998,"5min":0.83199999999999996,"15min":0.56499999999999995},"max":{"1min":1.77,"5min":1.9339999999999999,"15min":2.282},"last":1.631}]},{"osd":5,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3360000000000001,"15min":1.339},"min":{"1min":1.044,"5min":0.99399999999999999,"15min":0.78900000000000003},"max":{"1min":1.8320000000000001,"5min":1.9139999999999999,"15min":2.0529999999999999},"last":1.284},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4890000000000001,"15min":1.4790000000000001},"min":{"1min":1.2070000000000001,"5min":0.98899999999999999,"15min":0.61299999999999999},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.2000000000000002},"last":1.1919999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.569,"15min":1.5700000000000001},"min":{"1min":1.3819999999999999,"5min":1.026,"15min":0.72699999999999998},"max":{"1min":1.9370000000000001,"5min":2.1110000000000002,"15min":2.2639999999999998},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.4450000000000001,"15min":1.4530000000000001},"min":{"1min":0.997,"5min":0.89100000000000001,"15min":0.83399999999999996},"max":{"1min":1.7210000000000001,"5min":2.1280000000000001,"15min":2.1379999999999999},"last":1.673}]}]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":486701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2949999999999999,"15min":1.2829999999999999},"min":{"1min":0.81899999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.6619999999999999,"5min":1.784,"15min":3.7450000000000001},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.3240000000000001,"15min":1.3320000000000001},"min":{"1min":0.95199999999999996,"5min":0.748,"15min":0.57999999999999996},"max":{"1min":1.6259999999999999,"5min":2.0270000000000001,"15min":3.819},"last":1.1990000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2250000000000001,"15min":1.1870000000000001},"min":{"1min":0.97799999999999998,"5min":0.67400000000000004,"15min":0.621},"max":{"1min":1.5529999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2330000000000001,"15min":1.2130000000000001},"min":{"1min":0.90700000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.599,"5min":2.0459999999999998,"15min":3.8519999999999999},"last":1.518}]},{"osd":2,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2130000000000001,"15min":1.218},"min":{"1min":0.85699999999999998,"5min":0.627,"15min":0.627},"max":{"1min":1.6339999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.752},{"interface":"front","average":{"1min":1.292,"5min":1.264,"15min":1.284},"min":{"1min":0.91300000000000003,"5min":0.88500000000000001,"15min":0.755},"max":{"1min":1.653,"5min":1.9970000000000001,"15min":3.9009999999999998},"last":0.90200000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3320000000000001,"15min":1.333},"min":{"1min":1.016,"5min":0.80100000000000005,"15min":0.751},"max":{"1min":1.6459999999999999,"5min":2.3900000000000001,"15min":4.2380000000000004},"last":1.331},{"interface":"front","average":{"1min":1.373,"5min":1.296,"15min":1.351},"min":{"1min":1.0860000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6859999999999999,"5min":2.012,"15min":4.4269999999999996},"last":0.871}]},{"osd":4,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.361,"15min":1.345},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.57699999999999996},"max":{"1min":1.7070000000000001,"5min":2.3029999999999999,"15min":2.4100000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.4239999999999999,"15min":1.476},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.67300000000000004},"max":{"1min":1.839,"5min":2.2189999999999999,"15min":2.3050000000000002},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.306,"15min":1.3129999999999999},"min":{"1min":0.86499999999999999,"5min":0.79600000000000004,"15min":0.63900000000000001},"max":{"1min":1.772,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.145},{"interface":"front","average":{"1min":1.456,"5min":1.4259999999999999,"15min":1.454},"min":{"1min":1.075,"5min":1.075,"15min":0.63900000000000001},"max":{"1min":2.214,"5min":2.214,"15min":2.5230000000000001},"last":1.466}]},{"osd":7,"last update":"Mon Oct 16 12:48:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.3919999999999999,"15min":1.4450000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.495},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.454,"15min":1.4610000000000001},"min":{"1min":1.1220000000000001,"5min":0.96399999999999997,"15min":0.59099999999999997},"max":{"1min":2.008,"5min":2.2610000000000001,"15min":2.2719999999999998},"last":1.272}]}]},{"osd":5,"up_from":14,"seq":60129542340,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335480,"statfs":{"total":96636764160,"available":96599531520,"internally_reserved":0,"allocated":1044480,"data_stored":493653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1459999999999999,"15min":1.159},"min":{"1min":0.80100000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7150000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.1879999999999999,"15min":1.177},"min":{"1min":0.85899999999999999,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.516,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":0.97699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.222,"15min":1.25},"min":{"1min":0.94199999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.679,"5min":1.728,"15min":2.415},"last":1.52},{"interface":"front","average":{"1min":1.236,"5min":1.1950000000000001,"15min":1.2250000000000001},"min":{"1min":0.96899999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.708,"5min":1.708,"15min":1.9419999999999999},"last":1.1379999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2649999999999999,"15min":1.298},"min":{"1min":0.94799999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.625,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.591},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2649999999999999,"15min":1.268},"min":{"1min":0.97599999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.548,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.29}]},{"osd":3,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.329,"15min":1.3140000000000001},"min":{"1min":0.97399999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6479999999999999,"5min":1.6870000000000001,"15min":2.2690000000000001},"last":1.103},{"interface":"front","average":{"1min":1.403,"5min":1.3560000000000001,"15min":1.377},"min":{"1min":1.0609999999999999,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.6220000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.6819999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1899999999999999,"15min":1.204},"min":{"1min":0.89000000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7170000000000001,"5min":1.996,"15min":2.7909999999999999},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.329,"15min":1.327},"min":{"1min":0.90700000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.8120000000000001,"5min":1.877,"15min":5.3250000000000002},"last":1.3979999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3200000000000001,"15min":1.3280000000000001},"min":{"1min":0.96799999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.696,"5min":1.927,"15min":2.343},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.377,"15min":1.377},"min":{"1min":1.095,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9119999999999999,"5min":2.3079999999999998,"15min":2.6520000000000001},"last":1.6399999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:48:44 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4370000000000001,"15min":1.4590000000000001},"min":{"1min":1.054,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.7170000000000001,"5min":1.889,"15min":5.1269999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.506,"15min":1.51},"min":{"1min":1.131,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.7629999999999999,"5min":2.0659999999999998,"15min":2.4929999999999999},"last":1.4830000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542340,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58576,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313264,"statfs":{"total":96636764160,"available":96576782336,"internally_reserved":0,"allocated":1048576,"data_stored":494613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2090000000000001,"15min":1.2030000000000001},"min":{"1min":0.78700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.766,"5min":1.766,"15min":2.2200000000000002},"last":0.873},{"interface":"front","average":{"1min":1.155,"5min":1.125,"15min":1.1539999999999999},"min":{"1min":0.80700000000000005,"5min":0.65800000000000003,"15min":0.623},"max":{"1min":1.645,"5min":1.7070000000000001,"15min":1.9159999999999999},"last":1.139}]},{"osd":1,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1599999999999999,"15min":1.1970000000000001},"min":{"1min":0.79900000000000004,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.681,"5min":1.681,"15min":1.865},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2529999999999999,"15min":1.272},"min":{"1min":0.91200000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.724,"5min":1.724,"15min":1.9390000000000001},"last":1.091}]},{"osd":2,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.97799999999999998,"5min":0.81899999999999995,"15min":0.753},"max":{"1min":1.667,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3120000000000001,"15min":1.3400000000000001},"min":{"1min":0.93100000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.9910000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2490000000000001,"15min":1.27},"min":{"1min":0.91900000000000004,"5min":0.80900000000000005,"15min":0.752},"max":{"1min":1.5289999999999999,"5min":1.716,"15min":1.8100000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.3640000000000001,"15min":1.3819999999999999},"min":{"1min":1.236,"5min":0.86799999999999999,"15min":0.76500000000000001},"max":{"1min":1.7869999999999999,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.044}]},{"osd":5,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.304,"15min":1.327},"min":{"1min":0.94799999999999995,"5min":0.86799999999999999,"15min":0.50900000000000001},"max":{"1min":1.6579999999999999,"5min":2.0800000000000001,"15min":7.9000000000000004},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.2929999999999999,"15min":1.319},"min":{"1min":0.96999999999999997,"5min":0.76400000000000001,"15min":0.53600000000000003},"max":{"1min":1.625,"5min":2.3700000000000001,"15min":7.3460000000000001},"last":1.405}]},{"osd":6,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3819999999999999,"15min":1.387},"min":{"1min":0.98199999999999998,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":1.8380000000000001,"5min":2.153,"15min":2.153},"last":1.329},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4590000000000001,"15min":1.486},"min":{"1min":1.0369999999999999,"5min":0.90400000000000003,"15min":0.65300000000000002},"max":{"1min":1.96,"5min":2.1339999999999999,"15min":2.6269999999999998},"last":1.282}]},{"osd":7,"last update":"Mon Oct 16 12:48:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4350000000000001,"15min":1.4690000000000001},"min":{"1min":0.95199999999999996,"5min":0.88600000000000001,"15min":0.54200000000000004},"max":{"1min":1.76,"5min":2.056,"15min":2.254},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.5620000000000001,"15min":1.595},"min":{"1min":1.1659999999999999,"5min":0.94799999999999995,"15min":0.69199999999999995},"max":{"1min":1.893,"5min":2.262,"15min":2.5289999999999999},"last":1.492}]}]},{"osd":3,"up_from":14,"seq":60129542340,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40512,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331328,"statfs":{"total":96636764160,"available":96595279872,"internally_reserved":0,"allocated":1032192,"data_stored":486757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.94399999999999995,"15min":0.95899999999999996},"min":{"1min":0.61099999999999999,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.482,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.194,"5min":1.1839999999999999,"15min":1.181},"min":{"1min":0.66400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5800000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0229999999999999,"15min":1.0369999999999999},"min":{"1min":0.69299999999999995,"5min":0.53500000000000003,"15min":0.39500000000000002},"max":{"1min":1.4099999999999999,"5min":1.758,"15min":1.8149999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.0840000000000001,"15min":1.087},"min":{"1min":0.57799999999999996,"5min":0.50600000000000001,"15min":0.45600000000000002},"max":{"1min":1.476,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.80700000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.0960000000000001,"15min":1.0840000000000001},"min":{"1min":0.71999999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.6539999999999999,"5min":1.7949999999999999,"15min":1.9510000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3460000000000001,"15min":1.367},"min":{"1min":0.79600000000000004,"5min":0.61499999999999999,"15min":0.58299999999999996},"max":{"1min":1.7010000000000001,"5min":1.8600000000000001,"15min":2.0569999999999999},"last":0.996}]},{"osd":4,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.4930000000000001,"15min":1.4910000000000001},"min":{"1min":1.1579999999999999,"5min":0.80400000000000005,"15min":0.77600000000000002},"max":{"1min":1.9239999999999999,"5min":1.956,"15min":2.2730000000000001},"last":1.371},{"interface":"front","average":{"1min":1.431,"5min":1.4299999999999999,"15min":1.427},"min":{"1min":0.97499999999999998,"5min":0.64800000000000002,"15min":0.63500000000000001},"max":{"1min":1.869,"5min":1.8939999999999999,"15min":3.6930000000000001},"last":1.1579999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4119999999999999,"15min":1.405},"min":{"1min":1.0069999999999999,"5min":0.65900000000000003,"15min":0.60699999999999998},"max":{"1min":1.9910000000000001,"5min":2.028,"15min":3.8300000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.47,"15min":1.4730000000000001},"min":{"1min":1.0269999999999999,"5min":0.95399999999999996,"15min":0.64900000000000002},"max":{"1min":1.827,"5min":2.1030000000000002,"15min":2.3540000000000001},"last":1.232}]},{"osd":6,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4530000000000001,"15min":1.4550000000000001},"min":{"1min":0.88900000000000001,"5min":0.83999999999999997,"15min":0.65300000000000002},"max":{"1min":1.905,"5min":2.2989999999999999,"15min":2.415},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4490000000000001,"15min":1.4379999999999999},"min":{"1min":1.196,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8700000000000001,"5min":2.3740000000000001,"15min":3.9809999999999999},"last":1.3340000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:00 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.554,"15min":1.5469999999999999},"min":{"1min":1.0660000000000001,"5min":0.81000000000000005,"15min":0.80300000000000005},"max":{"1min":1.917,"5min":2.198,"15min":2.431},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.621,"15min":1.613},"min":{"1min":1.0389999999999999,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":1.986,"5min":2.552,"15min":2.552},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542338,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44676,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327164,"statfs":{"total":96636764160,"available":96591015936,"internally_reserved":0,"allocated":1032192,"data_stored":486757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.46999999999999997,"5min":0.315,"15min":0.315},"max":{"1min":1.0980000000000001,"5min":1.742,"15min":1.7889999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.091,"5min":1.087,"15min":1.0880000000000001},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.655,"5min":1.7070000000000001,"15min":2.085},"last":0.69099999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.133,"15min":1.1180000000000001},"min":{"1min":0.59699999999999998,"5min":0.51500000000000001,"15min":0.50900000000000001},"max":{"1min":1.458,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.107},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.206,"15min":1.22},"min":{"1min":0.76400000000000001,"5min":0.46500000000000002,"15min":0.45500000000000002},"max":{"1min":2.347,"5min":2.347,"15min":2.4289999999999998},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0680000000000001,"15min":1.0740000000000001},"min":{"1min":0.54700000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.4610000000000001,"5min":1.788,"15min":1.859},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3069999999999999,"15min":1.3049999999999999},"min":{"1min":0.64500000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.8049999999999999,"5min":2.1000000000000001,"15min":2.1970000000000001},"last":1.177}]},{"osd":4,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.456,"15min":1.4510000000000001},"min":{"1min":0.94699999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.0619999999999998,"5min":2.431,"15min":2.431},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4239999999999999,"15min":1.4430000000000001},"min":{"1min":1.038,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8360000000000001,"5min":2.0670000000000002,"15min":2.294},"last":1.2}]},{"osd":5,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4910000000000001,"15min":1.4930000000000001},"min":{"1min":1.095,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.117,"5min":2.23,"15min":2.3340000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3939999999999999,"15min":1.421},"min":{"1min":0.85899999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.8759999999999999,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.696}]},{"osd":6,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.417,"15min":1.448},"min":{"1min":0.97399999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.181,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.583},{"interface":"front","average":{"1min":1.502,"5min":1.5629999999999999,"15min":1.5609999999999999},"min":{"1min":1.0740000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.2789999999999999,"5min":2.294,"15min":2.488},"last":1.3640000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5920000000000001,"15min":1.6040000000000001},"min":{"1min":1.157,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.202,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.169},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5129999999999999,"15min":1.55},"min":{"1min":0.99399999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.0390000000000001,"5min":2.573,"15min":2.6429999999999998},"last":1.401}]}]},{"osd":1,"up_from":14,"seq":60129542338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58568,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313272,"statfs":{"total":96636764160,"available":96576790528,"internally_reserved":0,"allocated":1040384,"data_stored":493091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.90700000000000003,"15min":0.90400000000000003},"min":{"1min":0.69499999999999995,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.423,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.0820000000000001,"15min":1.0960000000000001},"min":{"1min":0.80500000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.931,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.98699999999999999,"15min":0.995},"min":{"1min":0.76800000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.603,"5min":1.8460000000000001,"15min":2.262},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.232,"5min":1.2010000000000001,"15min":1.1859999999999999},"min":{"1min":0.85699999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.5980000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.3109999999999999,"15min":1.3180000000000001},"min":{"1min":1.2170000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.988,"5min":1.988,"15min":2.2349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.272,"5min":1.208,"15min":1.22},"min":{"1min":0.94899999999999995,"5min":0.56799999999999995,"15min":0.46800000000000003},"max":{"1min":1.625,"5min":1.9199999999999999,"15min":2.1789999999999998},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.2929999999999999,"15min":1.3140000000000001},"min":{"1min":0.99299999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.032,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4510000000000001,"15min":1.4470000000000001},"min":{"1min":1.1559999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.8839999999999999,"5min":2.0649999999999999,"15min":3.0659999999999998},"last":0.95099999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.3939999999999999,"15min":1.4390000000000001},"min":{"1min":1.107,"5min":0.755,"15min":0.755},"max":{"1min":1.95,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.4470000000000001,"15min":1.4299999999999999},"min":{"1min":1.0640000000000001,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":1.996,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.2450000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.532,"15min":1.54},"min":{"1min":1.2549999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.0129999999999999,"5min":2.1899999999999999,"15min":2.9350000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4970000000000001,"15min":1.5289999999999999},"min":{"1min":1.0780000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.0739999999999998,"5min":2.387,"15min":2.387},"last":0.82899999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:48:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.5940000000000001,"15min":1.6259999999999999},"min":{"1min":1.29,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.1760000000000002,"5min":2.2429999999999999,"15min":2.2879999999999998},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.655,"5min":1.5549999999999999,"15min":1.5660000000000001},"min":{"1min":1.2110000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.153,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.173}]}]},{"osd":0,"up_from":14,"seq":60129542339,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36940,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334900,"statfs":{"total":96636764160,"available":96598937600,"internally_reserved":0,"allocated":1634304,"data_stored":1079263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.876,"15min":0.90300000000000002},"min":{"1min":0.45700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.335,"5min":1.5089999999999999,"15min":2.5720000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.0980000000000001,"15min":1.0860000000000001},"min":{"1min":0.59799999999999998,"5min":0.47799999999999998,"15min":0.32700000000000001},"max":{"1min":1.706,"5min":1.8109999999999999,"15min":2.1800000000000002},"last":0.92500000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.095,"15min":1.1299999999999999},"min":{"1min":0.63700000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.6679999999999999,"5min":1.8120000000000001,"15min":2.7650000000000001},"last":1.579},{"interface":"front","average":{"1min":1.194,"5min":1.1539999999999999,"15min":1.165},"min":{"1min":0.71299999999999997,"5min":0.496,"15min":0.438},"max":{"1min":1.637,"5min":1.952,"15min":1.952},"last":1.1120000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0680000000000001,"15min":1.0800000000000001},"min":{"1min":0.68000000000000005,"5min":0.41299999999999998,"15min":0.315},"max":{"1min":1.696,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2629999999999999,"15min":1.3069999999999999},"min":{"1min":0.746,"5min":0.56100000000000005,"15min":0.45700000000000002},"max":{"1min":1.6379999999999999,"5min":1.8740000000000001,"15min":3.6579999999999999},"last":1.1799999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.339,"15min":1.347},"min":{"1min":0.98799999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":4.726},"last":1.389},{"interface":"front","average":{"1min":1.448,"5min":1.409,"15min":1.454},"min":{"1min":0.88300000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.7809999999999999,"5min":1.966,"15min":4.843},"last":1.722}]},{"osd":5,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.423,"15min":1.4219999999999999},"min":{"1min":0.91000000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.857,"5min":2.2229999999999999,"15min":2.2440000000000002},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":1.135,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.7569999999999999,"5min":2.02,"15min":4.9059999999999997},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.51,"15min":1.532},"min":{"1min":0.98099999999999998,"5min":0.78300000000000003,"15min":0.73299999999999998},"max":{"1min":1.7869999999999999,"5min":2.3519999999999999,"15min":3.7890000000000001},"last":1.829},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.496,"15min":1.5189999999999999},"min":{"1min":0.96699999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.9379999999999999,"5min":2.145,"15min":2.6589999999999998},"last":1.641}]},{"osd":7,"last update":"Mon Oct 16 12:48:56 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6060000000000001,"15min":1.6180000000000001},"min":{"1min":1.0249999999999999,"5min":0.92000000000000004,"15min":0.82599999999999996},"max":{"1min":2.0219999999999998,"5min":2.3490000000000002,"15min":2.4729999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.5369999999999999,"15min":1.5629999999999999},"min":{"1min":1.2789999999999999,"5min":0.755,"15min":0.755},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":4.5890000000000004},"last":1.5409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:49:31.823 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:49:31.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:49:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.102+0000 7fa15c94e640 1 Processor -- start 2023-10-16T12:49:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.102+0000 7fa15c94e640 1 -- start start 2023-10-16T12:49:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa15c94e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa15815cb50 0x7fa158161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa15c94e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa1580b4c80 0x7fa1580d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa15c94e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1581616c0 con 0x7fa1580b1690 2023-10-16T12:49:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa15c94e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580cdbf0 con 0x7fa15815cb50 2023-10-16T12:49:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa15c94e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580cde90 con 0x7fa1580b4c80 2023-10-16T12:49:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa14ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa1580b1690 0x7fa1580b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50286/0 (socket says 172.21.15.60:50286) 2023-10-16T12:49:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.104+0000 7fa14ffff640 1 -- 172.21.15.60:0/1333276192 learned_addr learned my addr 172.21.15.60:0/1333276192 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa1577fe640 1 --2- 172.21.15.60:0/1333276192 >> v2:172.21.15.60:3300/0 conn(0x7fa15815cb50 0x7fa158161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa157fff640 1 --2- 172.21.15.60:0/1333276192 >> v2:172.21.15.144:3300/0 conn(0x7fa1580b4c80 0x7fa1580d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa156ffd640 1 -- 172.21.15.60:0/1333276192 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268391727 0 0) 0x7fa1581616c0 con 0x7fa1580b1690 2023-10-16T12:49:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa156ffd640 1 -- 172.21.15.60:0/1333276192 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa144003520 con 0x7fa1580b1690 2023-10-16T12:49:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa1577fe640 1 -- 172.21.15.60:0/1333276192 >> v2:172.21.15.144:3300/0 conn(0x7fa1580b4c80 msgr2=0x7fa1580d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa1577fe640 1 --2- 172.21.15.60:0/1333276192 >> v2:172.21.15.144:3300/0 conn(0x7fa1580b4c80 0x7fa1580d1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa1577fe640 1 -- 172.21.15.60:0/1333276192 >> v1:172.21.15.60:6789/0 conn(0x7fa1580b1690 legacy=0x7fa1580b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.105+0000 7fa1577fe640 1 -- 172.21.15.60:0/1333276192 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580ce170 con 0x7fa15815cb50 2023-10-16T12:49:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa1577fe640 1 --2- 172.21.15.60:0/1333276192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815cb50 0x7fa158161180 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa1580b1ae0 tx=0x7fa14802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fe8189f5de7ef6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa156ffd640 1 -- 172.21.15.60:0/1333276192 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148031710 con 0x7fa15815cb50 2023-10-16T12:49:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa156ffd640 1 -- 172.21.15.60:0/1333276192 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14802f050 con 0x7fa15815cb50 2023-10-16T12:49:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa156ffd640 1 -- 172.21.15.60:0/1333276192 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148031a90 con 0x7fa15815cb50 2023-10-16T12:49:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa15c94e640 1 -- 172.21.15.60:0/1333276192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815cb50 msgr2=0x7fa158161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa15c94e640 1 --2- 172.21.15.60:0/1333276192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815cb50 0x7fa158161180 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa1580b1ae0 tx=0x7fa14802b7b0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa15c94e640 1 -- 172.21.15.60:0/1333276192 shutdown_connections 2023-10-16T12:49:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa15c94e640 1 --2- 172.21.15.60:0/1333276192 >> v2:172.21.15.144:3300/0 conn(0x7fa1580b4c80 0x7fa1580d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa15c94e640 1 --2- 172.21.15.60:0/1333276192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815cb50 0x7fa158161180 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.107+0000 7fa15c94e640 1 -- 172.21.15.60:0/1333276192 >> 172.21.15.60:0/1333276192 conn(0x7fa1580cf570 msgr2=0x7fa1580cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.108+0000 7fa15c94e640 1 -- 172.21.15.60:0/1333276192 shutdown_connections 2023-10-16T12:49:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.108+0000 7fa15c94e640 1 -- 172.21.15.60:0/1333276192 wait complete. 2023-10-16T12:49:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.109+0000 7fa15c94e640 1 Processor -- start 2023-10-16T12:49:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.110+0000 7fa15c94e640 1 -- start start 2023-10-16T12:49:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.110+0000 7fa15c94e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580b1690 0x7fa1580c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa15c94e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 0x7fa1580cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa15c94e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815cb50 0x7fa1580cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa15c94e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580c7930 con 0x7fa1580b4c80 2023-10-16T12:49:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa15c94e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580c7c10 con 0x7fa1580b1690 2023-10-16T12:49:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa15c94e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580c7ef0 con 0x7fa15815cb50 2023-10-16T12:49:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 0x7fa1580cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 0x7fa1580cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42678/0 (socket says 172.21.15.60:42678) 2023-10-16T12:49:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 -- 172.21.15.60:0/3931353403 learned_addr learned my addr 172.21.15.60:0/3931353403 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa1577fe640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580b1690 0x7fa1580c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 -- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815cb50 msgr2=0x7fa1580cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa157fff640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815cb50 0x7fa1580cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815cb50 0x7fa1580cc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 -- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580b1690 msgr2=0x7fa1580c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580b1690 0x7fa1580c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa14ffff640 1 -- 172.21.15.60:0/3931353403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148009400 con 0x7fa1580b4c80 2023-10-16T12:49:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.111+0000 7fa1577fe640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580b1690 0x7fa1580c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.112+0000 7fa14ffff640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 0x7fa1580cbbd0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c430 tx=0x7fa14000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.112+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140005ce0 con 0x7fa1580b4c80 2023-10-16T12:49:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.112+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140005e40 con 0x7fa1580b4c80 2023-10-16T12:49:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.112+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1400072f0 con 0x7fa1580b4c80 2023-10-16T12:49:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.112+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1580c81d0 con 0x7fa1580b4c80 2023-10-16T12:49:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.112+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1580c89d0 con 0x7fa1580b4c80 2023-10-16T12:49:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.113+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1580c86f0 con 0x7fa1580b4c80 2023-10-16T12:49:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.114+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa140007450 con 0x7fa1580b4c80 2023-10-16T12:49:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.115+0000 7fa154ff9640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa13806af90 0x7fa13806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.115+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa140088ec0 con 0x7fa1580b4c80 2023-10-16T12:49:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.116+0000 7fa1577fe640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa13806af90 0x7fa13806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.117+0000 7fa1577fe640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa13806af90 0x7fa13806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa148035000 tx=0x7fa148008030 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.120+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1580c86f0 con 0x7fa1580b4c80 2023-10-16T12:49:32.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.382+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa1580c86f0 con 0x7fa1580b4c80 2023-10-16T12:49:32.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.388+0000 7fa154ff9640 1 -- 172.21.15.60:0/3931353403 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa1580c86f0 con 0x7fa1580b4c80 2023-10-16T12:49:32.390 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:49:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:49:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:49:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:49:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:49:32.391 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:49:32.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa13806af90 msgr2=0x7fa13806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa13806af90 0x7fa13806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa148035000 tx=0x7fa148008030 comp rx=0 tx=0).stop 2023-10-16T12:49:32.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 msgr2=0x7fa1580cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 0x7fa1580cbbd0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c430 tx=0x7fa14000c8f0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 shutdown_connections 2023-10-16T12:49:32.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fa13806af90 0x7fa13806d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815cb50 0x7fa1580cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580b4c80 0x7fa1580cbbd0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.390+0000 7fa15c94e640 1 --2- 172.21.15.60:0/3931353403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580b1690 0x7fa1580c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.391+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 >> 172.21.15.60:0/3931353403 conn(0x7fa1580cf570 msgr2=0x7fa1580cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:32.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.391+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 shutdown_connections 2023-10-16T12:49:32.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.391+0000 7fa15c94e640 1 -- 172.21.15.60:0/3931353403 wait complete. 2023-10-16T12:49:32.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:32.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.631+0000 7fc1effff640 1 Processor -- start 2023-10-16T12:49:32.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.632+0000 7fc1effff640 1 -- start start 2023-10-16T12:49:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.633+0000 7fc1effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 0x7fc1f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.633+0000 7fc1effff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 0x7fc1f015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.633+0000 7fc1effff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f006c960 con 0x7fc1f00ad970 2023-10-16T12:49:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.633+0000 7fc1effff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f015b9d0 con 0x7fc1f006c040 2023-10-16T12:49:32.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.633+0000 7fc1effff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f015bcb0 con 0x7fc1f00cda80 2023-10-16T12:49:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.634+0000 7fc1e67fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc1f00ad970 0x7fc1f015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50302/0 (socket says 172.21.15.60:50302) 2023-10-16T12:49:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.634+0000 7fc1e67fc640 1 -- 172.21.15.60:0/4021612399 learned_addr learned my addr 172.21.15.60:0/4021612399 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.634+0000 7fc1eeffd640 1 --2- 172.21.15.60:0/4021612399 >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 0x7fc1f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.634+0000 7fc1ef7fe640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 0x7fc1f015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.634+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153027842 0 0) 0x7fc1f006c960 con 0x7fc1f00ad970 2023-10-16T12:49:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.634+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1dc003520 con 0x7fc1f00ad970 2023-10-16T12:49:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905182147 0 0) 0x7fc1dc003520 con 0x7fc1f00ad970 2023-10-16T12:49:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 msgr2=0x7fc1f015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 0x7fc1f015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 msgr2=0x7fc1f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 --2- 172.21.15.60:0/4021612399 >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 0x7fc1f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f015bf90 con 0x7fc1f00ad970 2023-10-16T12:49:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc1d8001670 con 0x7fc1f00ad970 2023-10-16T12:49:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 >> v1:172.21.15.60:6789/0 conn(0x7fc1f00ad970 legacy=0x7fc1f015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 0x7fc1dc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc003f40 0x7fc1dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc004d00 0x7fc1dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f015b9d0 con 0x7fc1dc003f40 2023-10-16T12:49:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f015bcb0 con 0x7fc1dc004d00 2023-10-16T12:49:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc00b980 con 0x7fc1f00cda80 2023-10-16T12:49:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc003f40 0x7fc1dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.635+0000 7fc1eeffd640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 0x7fc1dc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.636+0000 7fc1ef7fe640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc004d00 0x7fc1dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.636+0000 7fc1e67fc640 1 -- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 msgr2=0x7fc1dc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.636+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00cda80 0x7fc1dc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.636+0000 7fc1e67fc640 1 -- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc004d00 msgr2=0x7fc1dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.636+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc004d00 0x7fc1dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.636+0000 7fc1e67fc640 1 -- 172.21.15.60:0/4021612399 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc003520 con 0x7fc1dc003f40 2023-10-16T12:49:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.637+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc003f40 0x7fc1dc0047c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c110 tx=0x7fc1dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a6f9e5c8275596 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.637+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8005ad0 con 0x7fc1dc003f40 2023-10-16T12:49:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.637+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d8005c50 con 0x7fc1dc003f40 2023-10-16T12:49:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.637+0000 7fc1ee7fc640 1 -- 172.21.15.60:0/4021612399 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d802c7e0 con 0x7fc1dc003f40 2023-10-16T12:49:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1effff640 1 -- 172.21.15.60:0/4021612399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc003f40 msgr2=0x7fc1dc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1effff640 1 --2- 172.21.15.60:0/4021612399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc003f40 0x7fc1dc0047c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c110 tx=0x7fc1dc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1ef7fe640 1 -- 172.21.15.60:0/4021612399 reap_dead start 2023-10-16T12:49:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1effff640 1 -- 172.21.15.60:0/4021612399 shutdown_connections 2023-10-16T12:49:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1effff640 1 -- 172.21.15.60:0/4021612399 >> 172.21.15.60:0/4021612399 conn(0x7fc1f00ce6a0 msgr2=0x7fc1f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1effff640 1 -- 172.21.15.60:0/4021612399 shutdown_connections 2023-10-16T12:49:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.638+0000 7fc1effff640 1 -- 172.21.15.60:0/4021612399 wait complete. 2023-10-16T12:49:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.640+0000 7fc1effff640 1 Processor -- start 2023-10-16T12:49:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.640+0000 7fc1effff640 1 -- start start 2023-10-16T12:49:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1effff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003f40 0x7fc1f00c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 0x7fc1f01f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1effff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f0070150 0x7fc1f0070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1effff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0070b40 con 0x7fc1f00c7820 2023-10-16T12:49:32.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1effff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00c66c0 con 0x7fc1f0070150 2023-10-16T12:49:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1effff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00c2370 con 0x7fc1dc003f40 2023-10-16T12:49:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1e67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 0x7fc1f01f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1e67fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 0x7fc1f01f3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42706/0 (socket says 172.21.15.60:42706) 2023-10-16T12:49:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1e67fc640 1 -- 172.21.15.60:0/1003979181 learned_addr learned my addr 172.21.15.60:0/1003979181 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:32.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.641+0000 7fc1eeffd640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003f40 0x7fc1f00c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1ef7fe640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f0070150 0x7fc1f0070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e67fc640 1 -- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003f40 msgr2=0x7fc1f00c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003f40 0x7fc1f00c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e67fc640 1 -- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f0070150 msgr2=0x7fc1f0070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f0070150 0x7fc1f0070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e67fc640 1 -- 172.21.15.60:0/1003979181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00c2650 con 0x7fc1f00c7820 2023-10-16T12:49:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e67fc640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 0x7fc1f01f3590 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc1d802c380 tx=0x7fc1d8001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8039070 con 0x7fc1f00c7820 2023-10-16T12:49:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d802cbf0 con 0x7fc1f00c7820 2023-10-16T12:49:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d803c040 con 0x7fc1f00c7820 2023-10-16T12:49:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.642+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f00c2930 con 0x7fc1f00c7820 2023-10-16T12:49:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.643+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f00c2ef0 con 0x7fc1f00c7820 2023-10-16T12:49:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.644+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f006b330 con 0x7fc1f00c7820 2023-10-16T12:49:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.645+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc1d8032070 con 0x7fc1f00c7820 2023-10-16T12:49:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.646+0000 7fc1e7fff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc1d006afc0 0x7fc1d006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.646+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1d80aaac0 con 0x7fc1f00c7820 2023-10-16T12:49:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.646+0000 7fc1eeffd640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc1d006afc0 0x7fc1d006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.647+0000 7fc1eeffd640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc1d006afc0 0x7fc1d006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0007520 tx=0x7fc1e000b040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.650+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc1d8075ae0 con 0x7fc1f00c7820 2023-10-16T12:49:32.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.885+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f004e9b0 con 0x7fc1f00c7820 2023-10-16T12:49:32.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.886+0000 7fc1e7fff640 1 -- 172.21.15.60:0/1003979181 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fc1d8079100 con 0x7fc1f00c7820 2023-10-16T12:49:32.889 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:32.889 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:49:26.202053+0000","modified":"2023-10-16T12:49:29.284142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2404598903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2404598903},{"type":"v1","addr":"172.21.15.60:6835","nonce":2404598903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:49:32.890 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2023-10-16T12:49:32.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc1d006afc0 msgr2=0x7fc1d006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc1d006afc0 0x7fc1d006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0007520 tx=0x7fc1e000b040 comp rx=0 tx=0).stop 2023-10-16T12:49:32.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 msgr2=0x7fc1f01f3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:32.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 0x7fc1f01f3590 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc1d802c380 tx=0x7fc1d8001930 comp rx=0 tx=0).stop 2023-10-16T12:49:32.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 shutdown_connections 2023-10-16T12:49:32.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7fc1d006afc0 0x7fc1d006d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f0070150 0x7fc1f0070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1f00c7820 0x7fc1f01f3590 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 --2- 172.21.15.60:0/1003979181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003f40 0x7fc1f00c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:32.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 >> 172.21.15.60:0/1003979181 conn(0x7fc1f00ce6a0 msgr2=0x7fc1f00b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:32.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 shutdown_connections 2023-10-16T12:49:32.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:32.889+0000 7fc1effff640 1 -- 172.21.15.60:0/1003979181 wait complete. 2023-10-16T12:49:33.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:49:26.202053+0000', 'modified': '2023-10-16T12:49:29.284142+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6399}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6399': {'gid': 6399, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/2404598903', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6833', 'nonce': 2404598903}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2404598903}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:49:33.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:49:33.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:49:33.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.550+0000 7febf3561640 1 Processor -- start 2023-10-16T12:49:33.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.550+0000 7febf3561640 1 -- start start 2023-10-16T12:49:33.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.552+0000 7febf3561640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febec15ad70 0x7febec069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.552+0000 7febf3561640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0cb030 con 0x7febec0c9ca0 2023-10-16T12:49:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.552+0000 7febf3561640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0c91c0 con 0x7febec0caac0 2023-10-16T12:49:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.552+0000 7febf3561640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec0c94a0 con 0x7febec15ad70 2023-10-16T12:49:33.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.552+0000 7febf0afc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7febec0c9ca0 0x7febec0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50310/0 (socket says 172.21.15.60:50310) 2023-10-16T12:49:33.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.552+0000 7febf0afc640 1 -- 172.21.15.60:0/829280415 learned_addr learned my addr 172.21.15.60:0/829280415 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febf12fd640 1 --2- 172.21.15.60:0/829280415 >> v2:172.21.15.144:3300/0 conn(0x7febec15ad70 0x7febec069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113291588 0 0) 0x7febec0cb030 con 0x7febec0c9ca0 2023-10-16T12:49:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd8003520 con 0x7febec0c9ca0 2023-10-16T12:49:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 658174153 0 0) 0x7febd8003520 con 0x7febec0c9ca0 2023-10-16T12:49:33.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 >> v2:172.21.15.144:3300/0 conn(0x7febec15ad70 msgr2=0x7febec069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 --2- 172.21.15.60:0/829280415 >> v2:172.21.15.144:3300/0 conn(0x7febec15ad70 0x7febec069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 >> v1:172.21.15.144:6789/0 conn(0x7febec0caac0 legacy=0x7febec0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec0c9780 con 0x7febec0c9ca0 2023-10-16T12:49:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7febdc001670 con 0x7febec0c9ca0 2023-10-16T12:49:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 >> v1:172.21.15.60:6789/0 conn(0x7febec0c9ca0 legacy=0x7febec0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.553+0000 7febe3fff640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8003970 0x7febd8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febe3fff640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febd80047c0 0x7febd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febe3fff640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd8005180 0x7febd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec0c94a0 con 0x7febd8005180 2023-10-16T12:49:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd800bfd0 con 0x7febd80047c0 2023-10-16T12:49:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd800c190 con 0x7febd8003970 2023-10-16T12:49:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf1afe640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd8005180 0x7febd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf12fd640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8003970 0x7febd8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf0afc640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febd80047c0 0x7febd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf1afe640 1 -- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8003970 msgr2=0x7febd8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf1afe640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8003970 0x7febd8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf1afe640 1 -- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febd80047c0 msgr2=0x7febd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf1afe640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febd80047c0 0x7febd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf1afe640 1 -- 172.21.15.60:0/829280415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd8003520 con 0x7febd8005180 2023-10-16T12:49:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.554+0000 7febf0afc640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febd80047c0 0x7febd8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.555+0000 7febf1afe640 1 -- 172.21.15.60:0/829280415 reap_dead start 2023-10-16T12:49:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.555+0000 7febf1afe640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd8005180 0x7febd800b980 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7febe4001f60 tx=0x7febd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85553020a242df34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe4003b70 con 0x7febd8005180 2023-10-16T12:49:33.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe4003cf0 con 0x7febd8005180 2023-10-16T12:49:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febe3fff640 1 -- 172.21.15.60:0/829280415 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe40052b0 con 0x7febd8005180 2023-10-16T12:49:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febf3561640 1 -- 172.21.15.60:0/829280415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd8005180 msgr2=0x7febd800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febf3561640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd8005180 0x7febd800b980 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7febe4001f60 tx=0x7febd80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febf3561640 1 -- 172.21.15.60:0/829280415 shutdown_connections 2023-10-16T12:49:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.556+0000 7febf3561640 1 --2- 172.21.15.60:0/829280415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd8005180 0x7febd800b980 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.557+0000 7febf3561640 1 -- 172.21.15.60:0/829280415 >> 172.21.15.60:0/829280415 conn(0x7febec0ce9b0 msgr2=0x7febec0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.557+0000 7febf3561640 1 -- 172.21.15.60:0/829280415 shutdown_connections 2023-10-16T12:49:33.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.557+0000 7febf3561640 1 -- 172.21.15.60:0/829280415 wait complete. 2023-10-16T12:49:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.559+0000 7febf3561640 1 Processor -- start 2023-10-16T12:49:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.559+0000 7febf3561640 1 -- start start 2023-10-16T12:49:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.560+0000 7febf3561640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8005180 0x7febec159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.560+0000 7febf3561640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 0x7febec0b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.560+0000 7febf3561640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febec0b2b90 0x7febec0a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.560+0000 7febf3561640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec0b3010 con 0x7febec159310 2023-10-16T12:49:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.560+0000 7febf3561640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec15de60 con 0x7febec0b2b90 2023-10-16T12:49:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.560+0000 7febf3561640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febec159790 con 0x7febd8005180 2023-10-16T12:49:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 0x7febec0b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 0x7febec0b2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42714/0 (socket says 172.21.15.60:42714) 2023-10-16T12:49:33.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 -- 172.21.15.60:0/3029026867 learned_addr learned my addr 172.21.15.60:0/3029026867 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:33.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf12fd640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8005180 0x7febec159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 -- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8005180 msgr2=0x7febec159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8005180 0x7febec159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 -- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febec0b2b90 msgr2=0x7febec0a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febec0b2b90 0x7febec0a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.561+0000 7febf0afc640 1 -- 172.21.15.60:0/3029026867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec15d8e0 con 0x7febec159310 2023-10-16T12:49:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.562+0000 7febf0afc640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 0x7febec0b2650 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7febdc00d760 tx=0x7febdc00dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.562+0000 7febe1ffb640 1 -- 172.21.15.60:0/3029026867 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc0074e0 con 0x7febec159310 2023-10-16T12:49:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.562+0000 7febe1ffb640 1 -- 172.21.15.60:0/3029026867 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc007640 con 0x7febec159310 2023-10-16T12:49:33.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.562+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec15db80 con 0x7febec159310 2023-10-16T12:49:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.562+0000 7febe1ffb640 1 -- 172.21.15.60:0/3029026867 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc004a00 con 0x7febec159310 2023-10-16T12:49:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.562+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febec0a8870 con 0x7febec159310 2023-10-16T12:49:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.564+0000 7febe1ffb640 1 -- 172.21.15.60:0/3029026867 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7febdc004b60 con 0x7febec159310 2023-10-16T12:49:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.565+0000 7febe1ffb640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7febd006af90 0x7febd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.565+0000 7febe1ffb640 1 -- 172.21.15.60:0/3029026867 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febdc089530 con 0x7febec159310 2023-10-16T12:49:33.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.566+0000 7febf61d3740 1 Processor -- start 2023-10-16T12:49:33.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.566+0000 7febf12fd640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7febd006af90 0x7febd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.566+0000 7febf61d3740 1 -- start start 2023-10-16T12:49:33.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf61d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe829a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82ca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe829a7d0 0x564fe82ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf61d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe81b67a0 con 0x564fe82c7390 2023-10-16T12:49:33.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe8208740 con 0x564fe82c7770 2023-10-16T12:49:33.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x564fe8205e30 con 0x564fe829a7d0 2023-10-16T12:49:33.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf12fd640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7febd006af90 0x7febd006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febec0b38a0 tx=0x7febe8008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe829a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe829a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42726/0 (socket says 172.21.15.60:42726) 2023-10-16T12:49:33.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.567+0000 7febf12fd640 1 -- 172.21.15.60:0/3255253626 learned_addr learned my addr 172.21.15.60:0/3255253626 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:33.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf0afc640 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf1afe640 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe829a7d0 0x564fe82ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf12fd640 1 -- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe829a7d0 msgr2=0x564fe82ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf12fd640 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe829a7d0 0x564fe82ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf12fd640 1 -- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 msgr2=0x564fe82ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf12fd640 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf12fd640 1 -- 172.21.15.60:0/3255253626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fe81c5910 con 0x564fe82c7390 2023-10-16T12:49:33.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf0afc640 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:33.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febf12fd640 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe829a290 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7febe8012cd0 tx=0x7febe8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96fe7ad3de8b6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febd6ffd640 1 -- 172.21.15.60:0/3255253626 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8022440 con 0x564fe82c7390 2023-10-16T12:49:33.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febd6ffd640 1 -- 172.21.15.60:0/3255253626 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe80225a0 con 0x564fe82c7390 2023-10-16T12:49:33.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.568+0000 7febd6ffd640 1 -- 172.21.15.60:0/3255253626 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8022910 con 0x564fe82c7390 2023-10-16T12:49:33.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 -- 172.21.15.60:0/3255253626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 msgr2=0x564fe829a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe829a290 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7febe8012cd0 tx=0x7febe8015000 comp rx=0 tx=0).stop 2023-10-16T12:49:33.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 -- 172.21.15.60:0/3255253626 shutdown_connections 2023-10-16T12:49:33.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe829a7d0 0x564fe82ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 --2- 172.21.15.60:0/3255253626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe829a290 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 -- 172.21.15.60:0/3255253626 >> 172.21.15.60:0/3255253626 conn(0x564fe829c5a0 msgr2=0x564fe82e5120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 -- 172.21.15.60:0/3255253626 shutdown_connections 2023-10-16T12:49:33.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.569+0000 7febf61d3740 1 -- 172.21.15.60:0/3255253626 wait complete. 2023-10-16T12:49:33.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.570+0000 7febf61d3740 1 -- start start 2023-10-16T12:49:33.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.570+0000 7febf61d3740 1 Processor -- start 2023-10-16T12:49:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82c7770 0x564fe82cc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82cca60 0x564fe82df750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe8201720 con 0x564fe82c7390 2023-10-16T12:49:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe819f970 con 0x564fe82cca60 2023-10-16T12:49:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x564fe819f610 con 0x564fe82c7770 2023-10-16T12:49:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf61d3740 10 client.0 _ll_register_callbacks cb 0x564fe81c19d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:49:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82b7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42734/0 (socket says 172.21.15.60:42734) 2023-10-16T12:49:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.571+0000 7febf12fd640 1 -- 172.21.15.60:0/2952468987 learned_addr learned my addr 172.21.15.60:0/2952468987 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf12fd640 1 -- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82c7770 msgr2=0x564fe82cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf12fd640 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82c7770 0x564fe82cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf12fd640 1 -- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82cca60 msgr2=0x564fe82df750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf1afe640 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82cca60 0x564fe82df750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf12fd640 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82cca60 0x564fe82df750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf12fd640 1 -- 172.21.15.60:0/2952468987 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe8012950 con 0x564fe82c7390 2023-10-16T12:49:33.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febf12fd640 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82b7d90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7febe80096d0 tx=0x7febe8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa36fb1476a58b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.572+0000 7febcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:49:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febcbfff640 1 -- 172.21.15.60:0/2952468987 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8022d90 con 0x564fe82c7390 2023-10-16T12:49:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febcbfff640 1 -- 172.21.15.60:0/2952468987 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febe8034410 con 0x564fe82c7390 2023-10-16T12:49:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564fe8228160 con 0x564fe82c7390 2023-10-16T12:49:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febcbfff640 1 -- 172.21.15.60:0/2952468987 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7febe80345d0 con 0x564fe82c7390 2023-10-16T12:49:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 10 client.6552 fetch_fsmap learned FSMap version 34 2023-10-16T12:49:33.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x564fe81ffda0 con 0x564fe82c7390 2023-10-16T12:49:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febcbfff640 1 -- 172.21.15.60:0/2952468987 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564fe8228160 con 0x564fe82c7390 2023-10-16T12:49:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febcbfff640 1 -- 172.21.15.60:0/2952468987 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 34) v1 ==== 3294+0+0 (secure 0 0 0) 0x7febe8034730 con 0x564fe82c7390 2023-10-16T12:49:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 10 client.6552 fetch_fsmap finished waiting for FSMap version 34 2023-10-16T12:49:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 10 client.6552 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:49:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 20 client.6552 populate_metadata read hostname 'smithi060' 2023-10-16T12:49:33.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe82f9370 0x564fe82dcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 4 client.6552 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:49:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.573+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 --> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fe81d3cc0 con 0x564fe82f9370 2023-10-16T12:49:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.574+0000 7febf0afc640 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe82f9370 0x564fe82dcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.574+0000 7febf0afc640 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe82f9370 0x564fe82dcfe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=181df4f64d03d08 server_cookie=80691f52e96d9b4 in_seq=0 out_seq=0 2023-10-16T12:49:33.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.574+0000 7febcbfff640 10 client.6552 ms_handle_connect on v2:172.21.15.60:6833/2404598903 2023-10-16T12:49:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febcbfff640 1 -- 172.21.15.60:0/2952468987 <== mds.0 v2:172.21.15.60:6833/2404598903 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564fe81d3cc0 con 0x564fe82f9370 2023-10-16T12:49:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febcbfff640 10 client.6552 handle_command_reply: tid=0 2023-10-16T12:49:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 client.6552 shutdown 2023-10-16T12:49:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 msgr2=0x564fe82b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82b7d90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7febe80096d0 tx=0x7febe8004730 comp rx=0 tx=0).stop 2023-10-16T12:49:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 shutdown_connections 2023-10-16T12:49:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe82f9370 0x564fe82dcfe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82c7770 0x564fe82cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82cca60 0x564fe82df750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 --2- 172.21.15.60:0/2952468987 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82b7d90 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febcbfff640 0 client.6552 ms_handle_reset on v2:172.21.15.60:6833/2404598903 2023-10-16T12:49:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 >> 172.21.15.60:0/2952468987 conn(0x564fe829a7d0 msgr2=0x564fe829e140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 shutdown_connections 2023-10-16T12:49:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.575+0000 7febf61d3740 1 -- 172.21.15.60:0/2952468987 wait complete. 2023-10-16T12:49:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.576+0000 7febf61d3740 20 client.6552 trim_cache size 0 max 16384 2023-10-16T12:49:33.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.592+0000 7febf61d3740 1 Processor -- start 2023-10-16T12:49:33.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 -- start start 2023-10-16T12:49:33.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82a0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82b8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe7f65f20 con 0x564fe82c7390 2023-10-16T12:49:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe82b7f20 con 0x564fe82c7770 2023-10-16T12:49:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.593+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x564fe82e0e20 con 0x564fe82cca60 2023-10-16T12:49:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42746/0 (socket says 172.21.15.60:42746) 2023-10-16T12:49:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 -- 172.21.15.60:0/1940769269 learned_addr learned my addr 172.21.15.60:0/1940769269 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf0afc640 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf1afe640 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 -- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 msgr2=0x564fe82b89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 -- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 msgr2=0x564fe82b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.594+0000 7febf12fd640 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.595+0000 7febf12fd640 1 -- 172.21.15.60:0/1940769269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fe819eb80 con 0x564fe82c7390 2023-10-16T12:49:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.595+0000 7febf12fd640 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82a0680 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7febe8013010 tx=0x7febe802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d419b1ee75909ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.595+0000 7febd77fe640 1 -- 172.21.15.60:0/1940769269 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8034ea0 con 0x564fe82c7390 2023-10-16T12:49:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.595+0000 7febd77fe640 1 -- 172.21.15.60:0/1940769269 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe8009200 con 0x564fe82c7390 2023-10-16T12:49:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.595+0000 7febd77fe640 1 -- 172.21.15.60:0/1940769269 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8034610 con 0x564fe82c7390 2023-10-16T12:49:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 -- 172.21.15.60:0/1940769269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 msgr2=0x564fe82a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82a0680 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7febe8013010 tx=0x7febe802e040 comp rx=0 tx=0).stop 2023-10-16T12:49:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 -- 172.21.15.60:0/1940769269 shutdown_connections 2023-10-16T12:49:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe82c7770 0x564fe82b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 --2- 172.21.15.60:0/1940769269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7390 0x564fe82a0680 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 -- 172.21.15.60:0/1940769269 >> 172.21.15.60:0/1940769269 conn(0x564fe829a7d0 msgr2=0x564fe82e8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 -- 172.21.15.60:0/1940769269 shutdown_connections 2023-10-16T12:49:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.596+0000 7febf61d3740 1 -- 172.21.15.60:0/1940769269 wait complete. 2023-10-16T12:49:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.597+0000 7febf61d3740 1 -- start start 2023-10-16T12:49:33.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.597+0000 7febf61d3740 1 Processor -- start 2023-10-16T12:49:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 0x564fe82b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe8347e90 0x564fe834ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe82df3d0 con 0x564fe82c7770 2023-10-16T12:49:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe82cb880 con 0x564fe8347e90 2023-10-16T12:49:33.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x564fe82de7e0 con 0x564fe82cca60 2023-10-16T12:49:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf61d3740 10 client.0 _ll_register_callbacks cb 0x564fe829a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:49:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 0x564fe82b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf12fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 0x564fe82b83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42760/0 (socket says 172.21.15.60:42760) 2023-10-16T12:49:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.598+0000 7febf12fd640 1 -- 172.21.15.60:0/2512973443 learned_addr learned my addr 172.21.15.60:0/2512973443 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:33.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf12fd640 1 -- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 msgr2=0x564fe82b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf12fd640 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf1afe640 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe8347e90 0x564fe834ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf12fd640 1 -- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe8347e90 msgr2=0x564fe834ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf12fd640 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe8347e90 0x564fe834ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf12fd640 1 -- 172.21.15.60:0/2512973443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fe81ffda0 con 0x564fe82c7770 2023-10-16T12:49:33.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf12fd640 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 0x564fe82b83d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7febe8009d80 tx=0x7febe8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8483655b8adb726b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:49:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febcbfff640 1 -- 172.21.15.60:0/2512973443 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8034ea0 con 0x564fe82c7770 2023-10-16T12:49:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febcbfff640 1 -- 172.21.15.60:0/2512973443 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febe8034820 con 0x564fe82c7770 2023-10-16T12:49:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.599+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564fe81fa700 con 0x564fe82c7770 2023-10-16T12:49:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febcbfff640 1 -- 172.21.15.60:0/2512973443 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7febe8009200 con 0x564fe82c7770 2023-10-16T12:49:33.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 10 client.6558 fetch_fsmap learned FSMap version 34 2023-10-16T12:49:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x564fe81c4170 con 0x564fe82c7770 2023-10-16T12:49:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febcbfff640 1 -- 172.21.15.60:0/2512973443 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564fe81fa700 con 0x564fe82c7770 2023-10-16T12:49:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febcbfff640 1 -- 172.21.15.60:0/2512973443 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 34) v1 ==== 3294+0+0 (secure 0 0 0) 0x7febe803c070 con 0x564fe82c7770 2023-10-16T12:49:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 10 client.6558 fetch_fsmap finished waiting for FSMap version 34 2023-10-16T12:49:33.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 10 client.6558 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:49:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 20 client.6558 populate_metadata read hostname 'smithi060' 2023-10-16T12:49:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe834e330 0x564fe834e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 4 client.6558 mds_command: new command op to 6399 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:49:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 --> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564fe81fa420 con 0x564fe834e330 2023-10-16T12:49:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.600+0000 7febf0afc640 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe834e330 0x564fe834e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf0afc640 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe834e330 0x564fe834e710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b79a972d2e18423 server_cookie=e4319cc5d9941dcb in_seq=0 out_seq=0 2023-10-16T12:49:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febcbfff640 10 client.6558 ms_handle_connect on v2:172.21.15.60:6833/2404598903 2023-10-16T12:49:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febcbfff640 1 -- 172.21.15.60:0/2512973443 <== mds.0 v2:172.21.15.60:6833/2404598903 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x564fe81fa420 con 0x564fe834e330 2023-10-16T12:49:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febcbfff640 10 client.6558 handle_command_reply: tid=0 2023-10-16T12:49:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 client.6558 shutdown 2023-10-16T12:49:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 msgr2=0x564fe82b83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 0x564fe82b83d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7febe8009d80 tx=0x7febe8018000 comp rx=0 tx=0).stop 2023-10-16T12:49:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 shutdown_connections 2023-10-16T12:49:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:6833/2404598903,v1:172.21.15.60:6835/2404598903] conn(0x564fe834e330 0x564fe834e710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564fe82cca60 0x564fe82b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564fe8347e90 0x564fe834ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 --2- 172.21.15.60:0/2512973443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564fe82c7770 0x564fe82b83d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febcbfff640 0 client.6558 ms_handle_reset on v2:172.21.15.60:6833/2404598903 2023-10-16T12:49:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 >> 172.21.15.60:0/2512973443 conn(0x564fe829a7d0 msgr2=0x564fe82a04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.601+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 shutdown_connections 2023-10-16T12:49:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf61d3740 1 -- 172.21.15.60:0/2512973443 wait complete. 2023-10-16T12:49:33.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf61d3740 20 client.6558 trim_cache size 0 max 16384 2023-10-16T12:49:33.614 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:49:33.615 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:49:33.615 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:49:33.615 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 6399, 2023-10-16T12:49:33.615 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:49:33.615 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:49:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:49:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8302626220000002, 2023-10-16T12:49:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 34, 2023-10-16T12:49:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 86, 2023-10-16T12:49:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 86, 2023-10-16T12:49:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.661248495000001 2023-10-16T12:49:33.617 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:49:33.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7febd006af90 msgr2=0x7febd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7febd006af90 0x7febd006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febec0b38a0 tx=0x7febe8008040 comp rx=0 tx=0).stop 2023-10-16T12:49:33.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 msgr2=0x7febec0b2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 0x7febec0b2650 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7febdc00d760 tx=0x7febdc00dc20 comp rx=0 tx=0).stop 2023-10-16T12:49:33.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 shutdown_connections 2023-10-16T12:49:33.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7febd006af90 0x7febd006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febec0b2b90 0x7febec0a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febec159310 0x7febec0b2650 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 --2- 172.21.15.60:0/3029026867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febd8005180 0x7febec159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 >> 172.21.15.60:0/3029026867 conn(0x7febec0ce9b0 msgr2=0x7febec0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.602+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 shutdown_connections 2023-10-16T12:49:33.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.603+0000 7febf3561640 1 -- 172.21.15.60:0/3029026867 wait complete. 2023-10-16T12:49:33.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:49:33.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.840+0000 7f8655dfb640 1 Processor -- start 2023-10-16T12:49:33.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.840+0000 7f8655dfb640 1 -- start start 2023-10-16T12:49:33.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8655dfb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86500cdae0 0x7f86500cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8655dfb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500d1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8655dfb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8650160b60 con 0x7f86500cf2b0 2023-10-16T12:49:33.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8655dfb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86501617e0 con 0x7f86500cdae0 2023-10-16T12:49:33.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8655dfb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8650160e40 con 0x7f86500d0f70 2023-10-16T12:49:33.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f864ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86500cdae0 0x7f86500cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8654df9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86500cf2b0 0x7f86500cf690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50314/0 (socket says 172.21.15.60:50314) 2023-10-16T12:49:33.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f8654df9640 1 -- 172.21.15.60:0/81029557 learned_addr learned my addr 172.21.15.60:0/81029557 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:33.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.842+0000 7f86555fa640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.843+0000 7f864ffff640 1 -- 172.21.15.60:0/81029557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 msgr2=0x7f86500d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:33.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.843+0000 7f864ffff640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500d1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.843+0000 7f864ffff640 1 -- 172.21.15.60:0/81029557 >> v1:172.21.15.60:6789/0 conn(0x7f86500cf2b0 legacy=0x7f86500cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.843+0000 7f864ffff640 1 -- 172.21.15.60:0/81029557 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650161120 con 0x7f86500cdae0 2023-10-16T12:49:33.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.843+0000 7f86555fa640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:33.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.844+0000 7f864ffff640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500cdf40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f863c0097a0 tx=0x7f863c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b503ec8f5583e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f864f7fe640 1 -- 172.21.15.60:0/81029557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c031780 con 0x7f86500cdae0 2023-10-16T12:49:33.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f864f7fe640 1 -- 172.21.15.60:0/81029557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c0318e0 con 0x7f86500cdae0 2023-10-16T12:49:33.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f864f7fe640 1 -- 172.21.15.60:0/81029557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c031c70 con 0x7f86500cdae0 2023-10-16T12:49:33.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 -- 172.21.15.60:0/81029557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 msgr2=0x7f86500cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500cdf40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f863c0097a0 tx=0x7f863c02b820 comp rx=0 tx=0).stop 2023-10-16T12:49:33.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 -- 172.21.15.60:0/81029557 shutdown_connections 2023-10-16T12:49:33.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 --2- 172.21.15.60:0/81029557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500cdf40 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 -- 172.21.15.60:0/81029557 >> 172.21.15.60:0/81029557 conn(0x7f865006bc60 msgr2=0x7f865006c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:33.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 -- 172.21.15.60:0/81029557 shutdown_connections 2023-10-16T12:49:33.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.845+0000 7f8655dfb640 1 -- 172.21.15.60:0/81029557 wait complete. 2023-10-16T12:49:33.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.847+0000 7f8655dfb640 1 Processor -- start 2023-10-16T12:49:33.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.847+0000 7f8655dfb640 1 -- start start 2023-10-16T12:49:33.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8655dfb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8655dfb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86500cf2b0 0x7f86501f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8655dfb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8655dfb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86500ca580 con 0x7f86500cdae0 2023-10-16T12:49:33.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8655dfb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86500b2c00 con 0x7f86500cf2b0 2023-10-16T12:49:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8655dfb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86500c9be0 con 0x7f86500d0f70 2023-10-16T12:49:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8654df9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8654df9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500b2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42776/0 (socket says 172.21.15.60:42776) 2023-10-16T12:49:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.848+0000 7f8654df9640 1 -- 172.21.15.60:0/1554473031 learned_addr learned my addr 172.21.15.60:0/1554473031 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f86555fa640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f8654df9640 1 -- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 msgr2=0x7f86500b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f864ffff640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86500cf2b0 0x7f86501f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f8654df9640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500b4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f8654df9640 1 -- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86500cf2b0 msgr2=0x7f86501f20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:33.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f8654df9640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86500cf2b0 0x7f86501f20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:33.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f8654df9640 1 -- 172.21.15.60:0/1554473031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c009400 con 0x7f86500cdae0 2023-10-16T12:49:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f8654df9640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500b2570 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f864400c400 tx=0x7f864400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.849+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644005ce0 con 0x7f86500cdae0 2023-10-16T12:49:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.850+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644005e40 con 0x7f86500cdae0 2023-10-16T12:49:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.850+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86440072d0 con 0x7f86500cdae0 2023-10-16T12:49:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.850+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86500c9ec0 con 0x7f86500cdae0 2023-10-16T12:49:33.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.850+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86500cca40 con 0x7f86500cdae0 2023-10-16T12:49:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.851+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865004e9b0 con 0x7f86500cdae0 2023-10-16T12:49:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.852+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f86440074f0 con 0x7f86500cdae0 2023-10-16T12:49:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.853+0000 7f864d7fa640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f863406af90 0x7f863406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.853+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8644088e90 con 0x7f86500cdae0 2023-10-16T12:49:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.853+0000 7f864ffff640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f863406af90 0x7f863406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.854+0000 7f864ffff640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f863406af90 0x7f863406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f863c036000 tx=0x7f863c03d040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:33.857+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8644053d90 con 0x7f86500cdae0 2023-10-16T12:49:34.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.089+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f86500d07b0 con 0x7f86500cdae0 2023-10-16T12:49:34.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.090+0000 7f864d7fa640 1 -- 172.21.15.60:0/1554473031 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f864408a070 con 0x7f86500cdae0 2023-10-16T12:49:34.092 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:49:34.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f863406af90 msgr2=0x7f863406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:34.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f863406af90 0x7f863406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f863c036000 tx=0x7f863c03d040 comp rx=0 tx=0).stop 2023-10-16T12:49:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 msgr2=0x7f86500b2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500b2570 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f864400c400 tx=0x7f864400c8c0 comp rx=0 tx=0).stop 2023-10-16T12:49:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 shutdown_connections 2023-10-16T12:49:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f863406af90 0x7f863406d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86500d0f70 0x7f86500b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86500cf2b0 0x7f86501f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 --2- 172.21.15.60:0/1554473031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86500cdae0 0x7f86500b2570 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 >> 172.21.15.60:0/1554473031 conn(0x7f865006bc60 msgr2=0x7f86500cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 shutdown_connections 2023-10-16T12:49:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:34.093+0000 7f8655dfb640 1 -- 172.21.15.60:0/1554473031 wait complete. 2023-10-16T12:49:34.115 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:49:34.115 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:49:34.116 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:49:34.116 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:49:34.116 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:49:34.117 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:49:34.117 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:49:34.117 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:49:34.117 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:49:34.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.118 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:49:34.174 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:49:34.174 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:49:34.174 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:49:34.174 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.175 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:49:34.175 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.175 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:49:34.175 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:49:34.175 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:49:34.175 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:49:34.176 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:49:34.176 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:49:34.176 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:49:34.176 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:49:34.176 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:49:34.177 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:49:34.177 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41413sec preferred_lft 41413sec 2023-10-16T12:49:34.177 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:49:34.177 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.177 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:49:34.177 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:49:34.178 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:49:34.178 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:49:34.178 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:49:34.178 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:49:34.178 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.179 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:49:34.179 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.179 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:49:34.179 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:49:34.179 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:49:34.180 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.180 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:49:34.180 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:49:34.180 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:49:34.180 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:49:34.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.181 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:49:34.280 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:49:34.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.286 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:49:34.343 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:49:34.344 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:49:34.344 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:49:34.344 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:49:34.344 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:49:34.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.345 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:49:34.346 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:49:34.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:49:34.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:49:34.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:49:34.347 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:49:34.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.556 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:49:34.610 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:49:34.611 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:49:34.612 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:49:34.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:49:34.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:49:34.720 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:49:34.780 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:49:34.780 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:49:34.781 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:49:34.835 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:49:34.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:34.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:49:34.975 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:49:34.975 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:49:35.009 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:49:35.010 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:49:35.010 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:49:35.010 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:49:35.011 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:49:35.011 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:49:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:49:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:49:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:49:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:49:35.016 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:49:35.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:35.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:49:35.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:35.112 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:49:35.181 INFO:teuthology.orchestra.run.smithi060.stdout:client5788 2023-10-16T12:49:35.181 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:35.182 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:35.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:49:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.460+0000 7f48df59e640 1 Processor -- start 2023-10-16T12:49:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.460+0000 7f48df59e640 1 -- start start 2023-10-16T12:49:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.461+0000 7f48df59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f48e015aaa0 0x7f48e00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48df59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 0x7f48e00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48df59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e00cad00 con 0x7f48e0070240 2023-10-16T12:49:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48df59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e00b2d00 con 0x7f48e015aaa0 2023-10-16T12:49:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48df59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e00b2fe0 con 0x7f48e00ca8a0 2023-10-16T12:49:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48ded9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f48e0070240 0x7f48e00c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50330/0 (socket says 172.21.15.60:50330) 2023-10-16T12:49:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48ded9d640 1 -- 172.21.15.60:0/1994409021 learned_addr learned my addr 172.21.15.60:0/1994409021 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48de59c640 1 --2- 172.21.15.60:0/1994409021 >> v2:172.21.15.144:3300/0 conn(0x7f48e015aaa0 0x7f48e00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.462+0000 7f48ddd9b640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 0x7f48e00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263221879 0 0) 0x7f48e00cad00 con 0x7f48e0070240 2023-10-16T12:49:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48c8003520 con 0x7f48e0070240 2023-10-16T12:49:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2529506681 0 0) 0x7f48c8003520 con 0x7f48e0070240 2023-10-16T12:49:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 msgr2=0x7f48e00b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 0x7f48e00b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 >> v2:172.21.15.144:3300/0 conn(0x7f48e015aaa0 msgr2=0x7f48e00b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 --2- 172.21.15.60:0/1994409021 >> v2:172.21.15.144:3300/0 conn(0x7f48e015aaa0 0x7f48e00b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e00b32c0 con 0x7f48e0070240 2023-10-16T12:49:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f48d80019f0 con 0x7f48e0070240 2023-10-16T12:49:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48de59c640 1 --2- 172.21.15.60:0/1994409021 >> v2:172.21.15.144:3300/0 conn(0x7f48e015aaa0 0x7f48e00b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 >> v1:172.21.15.60:6789/0 conn(0x7f48e0070240 legacy=0x7f48e00c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 0x7f48c8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8003f40 0x7f48c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48c8004d00 0x7f48c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.463+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e00b2d00 con 0x7f48c8004d00 2023-10-16T12:49:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e00b2fe0 con 0x7f48c8003f40 2023-10-16T12:49:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c800b980 con 0x7f48e00ca8a0 2023-10-16T12:49:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ded9d640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48c8004d00 0x7f48c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48de59c640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 0x7f48c8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ddd9b640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8003f40 0x7f48c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ded9d640 1 -- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 msgr2=0x7f48c8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ded9d640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e00ca8a0 0x7f48c8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ded9d640 1 -- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8003f40 msgr2=0x7f48c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ded9d640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8003f40 0x7f48c80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ded9d640 1 -- 172.21.15.60:0/1994409021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8003520 con 0x7f48c8004d00 2023-10-16T12:49:35.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.464+0000 7f48ddd9b640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8003f40 0x7f48c80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:35.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.466+0000 7f48ded9d640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48c8004d00 0x7f48c8005180 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f48d800a840 tx=0x7f48c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0f0d9324503de0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:35.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.466+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8034630 con 0x7f48c8004d00 2023-10-16T12:49:35.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.466+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d802f070 con 0x7f48c8004d00 2023-10-16T12:49:35.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.466+0000 7f48dd59a640 1 -- 172.21.15.60:0/1994409021 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8034a30 con 0x7f48c8004d00 2023-10-16T12:49:35.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48df59e640 1 -- 172.21.15.60:0/1994409021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48c8004d00 msgr2=0x7f48c8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48df59e640 1 --2- 172.21.15.60:0/1994409021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48c8004d00 0x7f48c8005180 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f48d800a840 tx=0x7f48c80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48ded9d640 1 -- 172.21.15.60:0/1994409021 reap_dead start 2023-10-16T12:49:35.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48df59e640 1 -- 172.21.15.60:0/1994409021 shutdown_connections 2023-10-16T12:49:35.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48df59e640 1 -- 172.21.15.60:0/1994409021 >> 172.21.15.60:0/1994409021 conn(0x7f48e00cc230 msgr2=0x7f48e00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:35.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48df59e640 1 -- 172.21.15.60:0/1994409021 shutdown_connections 2023-10-16T12:49:35.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.467+0000 7f48df59e640 1 -- 172.21.15.60:0/1994409021 wait complete. 2023-10-16T12:49:35.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.469+0000 7f48df59e640 1 Processor -- start 2023-10-16T12:49:35.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.469+0000 7f48df59e640 1 -- start start 2023-10-16T12:49:35.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48df59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8004d00 0x7f48e0159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48df59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 0x7f48e01f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48df59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e01f5f70 0x7f48e00ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48df59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e00c80b0 con 0x7f48e0159d70 2023-10-16T12:49:35.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48df59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0159bb0 con 0x7f48c8004d00 2023-10-16T12:49:35.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48df59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e01f57b0 con 0x7f48e01f5f70 2023-10-16T12:49:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48ddd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 0x7f48e01f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48ddd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 0x7f48e01f21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42800/0 (socket says 172.21.15.60:42800) 2023-10-16T12:49:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48ddd9b640 1 -- 172.21.15.60:0/3843517449 learned_addr learned my addr 172.21.15.60:0/3843517449 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.470+0000 7f48de59c640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8004d00 0x7f48e0159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ded9d640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e01f5f70 0x7f48e00ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ddd9b640 1 -- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e01f5f70 msgr2=0x7f48e00ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ddd9b640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e01f5f70 0x7f48e00ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ddd9b640 1 -- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8004d00 msgr2=0x7f48e0159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ddd9b640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8004d00 0x7f48e0159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ddd9b640 1 -- 172.21.15.60:0/3843517449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e01f2a80 con 0x7f48e0159d70 2023-10-16T12:49:35.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ddd9b640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 0x7f48e01f21e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f48d40115c0 tx=0x7f48d4011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:35.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4015870 con 0x7f48e0159d70 2023-10-16T12:49:35.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d40159d0 con 0x7f48e0159d70 2023-10-16T12:49:35.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4015d80 con 0x7f48e0159d70 2023-10-16T12:49:35.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.471+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e01f2d60 con 0x7f48e0159d70 2023-10-16T12:49:35.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.472+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e01f5a50 con 0x7f48e0159d70 2023-10-16T12:49:35.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.473+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e004e9b0 con 0x7f48e0159d70 2023-10-16T12:49:35.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.474+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f48d400f480 con 0x7f48e0159d70 2023-10-16T12:49:35.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.475+0000 7f48ceffd640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48c006af90 0x7f48c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.475+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48d4089fc0 con 0x7f48e0159d70 2023-10-16T12:49:35.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.475+0000 7f48de59c640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48c006af90 0x7f48c006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.476+0000 7f48de59c640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48c006af90 0x7f48c006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009c20 tx=0x7f48d0008040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:35.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.479+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48d4054f50 con 0x7f48e0159d70 2023-10-16T12:49:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.700+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 --> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f48e00b3880 con 0x7f48c006af90 2023-10-16T12:49:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.704+0000 7f48ceffd640 1 -- 172.21.15.60:0/3843517449 <== mgr.5186 v2:172.21.15.144:6832/2872949001 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f48d0009870 con 0x7f48c006af90 2023-10-16T12:49:35.706 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:49:35.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:49:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:49:35.707 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:49:35.707 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:49:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48c006af90 msgr2=0x7f48c006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48c006af90 0x7f48c006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009c20 tx=0x7f48d0008040 comp rx=0 tx=0).stop 2023-10-16T12:49:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 msgr2=0x7f48e01f21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 0x7f48e01f21e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f48d40115c0 tx=0x7f48d4011a80 comp rx=0 tx=0).stop 2023-10-16T12:49:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 shutdown_connections 2023-10-16T12:49:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f48c006af90 0x7f48c006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48e01f5f70 0x7f48e00ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48e0159d70 0x7f48e01f21e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 --2- 172.21.15.60:0/3843517449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c8004d00 0x7f48e0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.706+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 >> 172.21.15.60:0/3843517449 conn(0x7f48e00cc230 msgr2=0x7f48e0070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:35.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.707+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 shutdown_connections 2023-10-16T12:49:35.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.707+0000 7f48df59e640 1 -- 172.21.15.60:0/3843517449 wait complete. 2023-10-16T12:49:35.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:49:35.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.952+0000 7f359ef05640 1 Processor -- start 2023-10-16T12:49:35.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.953+0000 7f359ef05640 1 -- start start 2023-10-16T12:49:35.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.954+0000 7f359ef05640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f359815ad70 0x7f3598069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.954+0000 7f359ef05640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f35980caac0 0x7f359806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.954+0000 7f359ef05640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35980cb030 con 0x7f35980c9ca0 2023-10-16T12:49:35.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.954+0000 7f359ef05640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35980c91c0 con 0x7f359815ad70 2023-10-16T12:49:35.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.954+0000 7f359ef05640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35980c94a0 con 0x7f35980caac0 2023-10-16T12:49:35.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.954+0000 7f359cca1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f359815ad70 0x7f3598069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.955+0000 7f359cca1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f359815ad70 0x7f3598069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42806/0 (socket says 172.21.15.60:42806) 2023-10-16T12:49:35.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.955+0000 7f359cca1640 1 -- 172.21.15.60:0/2168998535 learned_addr learned my addr 172.21.15.60:0/2168998535 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:35.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.955+0000 7f359cca1640 1 -- 172.21.15.60:0/2168998535 >> v2:172.21.15.144:3300/0 conn(0x7f35980caac0 msgr2=0x7f359806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:35.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.955+0000 7f359cca1640 1 --2- 172.21.15.60:0/2168998535 >> v2:172.21.15.144:3300/0 conn(0x7f35980caac0 0x7f359806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.955+0000 7f359cca1640 1 -- 172.21.15.60:0/2168998535 >> v1:172.21.15.144:6789/0 conn(0x7f35980c9ca0 legacy=0x7f35980ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:35.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.955+0000 7f359cca1640 1 -- 172.21.15.60:0/2168998535 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980c9780 con 0x7f359815ad70 2023-10-16T12:49:35.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.956+0000 7f359cca1640 1 --2- 172.21.15.60:0/2168998535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f3598069da0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f359806d090 tx=0x7f358c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8246462f9e3ce967 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:35.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f35977fe640 1 -- 172.21.15.60:0/2168998535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c031870 con 0x7f359815ad70 2023-10-16T12:49:35.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f35977fe640 1 -- 172.21.15.60:0/2168998535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c0319d0 con 0x7f359815ad70 2023-10-16T12:49:35.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f35977fe640 1 -- 172.21.15.60:0/2168998535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c031d60 con 0x7f359815ad70 2023-10-16T12:49:35.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f359ef05640 1 -- 172.21.15.60:0/2168998535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 msgr2=0x7f3598069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f359ef05640 1 --2- 172.21.15.60:0/2168998535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f3598069da0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f359806d090 tx=0x7f358c02b910 comp rx=0 tx=0).stop 2023-10-16T12:49:35.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f359ef05640 1 -- 172.21.15.60:0/2168998535 shutdown_connections 2023-10-16T12:49:35.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f359ef05640 1 --2- 172.21.15.60:0/2168998535 >> v2:172.21.15.144:3300/0 conn(0x7f35980caac0 0x7f359806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f359ef05640 1 --2- 172.21.15.60:0/2168998535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f3598069da0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.957+0000 7f359ef05640 1 -- 172.21.15.60:0/2168998535 >> 172.21.15.60:0/2168998535 conn(0x7f35980ce9b0 msgr2=0x7f35980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:35.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.958+0000 7f359ef05640 1 -- 172.21.15.60:0/2168998535 shutdown_connections 2023-10-16T12:49:35.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.958+0000 7f359ef05640 1 -- 172.21.15.60:0/2168998535 wait complete. 2023-10-16T12:49:35.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.959+0000 7f359ef05640 1 Processor -- start 2023-10-16T12:49:35.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.959+0000 7f359ef05640 1 -- start start 2023-10-16T12:49:35.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.960+0000 7f359ef05640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35980c9ca0 0x7f35981f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.960+0000 7f359ef05640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35980caac0 0x7f35980c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.960+0000 7f359ef05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f35980c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.960+0000 7f359ef05640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981f32c0 con 0x7f359815ad70 2023-10-16T12:49:35.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.960+0000 7f359ef05640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981f6840 con 0x7f35980c9ca0 2023-10-16T12:49:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.960+0000 7f359ef05640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35981f2740 con 0x7f35980caac0 2023-10-16T12:49:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f35980c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f35980c5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42816/0 (socket says 172.21.15.60:42816) 2023-10-16T12:49:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 -- 172.21.15.60:0/3462632888 learned_addr learned my addr 172.21.15.60:0/3462632888 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359cca1640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35980c9ca0 0x7f35981f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 -- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35980caac0 msgr2=0x7f35980c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f3597fff640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35980caac0 0x7f35980c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35980caac0 0x7f35980c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 -- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35980c9ca0 msgr2=0x7f35981f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35980c9ca0 0x7f35981f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359d4a2640 1 -- 172.21.15.60:0/3462632888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0094d0 con 0x7f359815ad70 2023-10-16T12:49:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.961+0000 7f359cca1640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35980c9ca0 0x7f35981f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.962+0000 7f359d4a2640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f35980c5df0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f359000b970 tx=0x7f359000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.962+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000cb40 con 0x7f359815ad70 2023-10-16T12:49:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.962+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359000ccc0 con 0x7f359815ad70 2023-10-16T12:49:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.962+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590014680 con 0x7f359815ad70 2023-10-16T12:49:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.962+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35981f2a20 con 0x7f359815ad70 2023-10-16T12:49:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.962+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35981f2fe0 con 0x7f359815ad70 2023-10-16T12:49:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.963+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35980c5b10 con 0x7f359815ad70 2023-10-16T12:49:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.964+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 28) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f359001d3b0 con 0x7f359815ad70 2023-10-16T12:49:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.965+0000 7f35957fa640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f357c06af90 0x7f357c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.965+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3590089720 con 0x7f359815ad70 2023-10-16T12:49:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.966+0000 7f359cca1640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f357c06af90 0x7f357c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.967+0000 7f359cca1640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f357c06af90 0x7f357c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f358c036000 tx=0x7f358c03c040 comp rx=0 tx=0).ready entity=mgr.5186 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:35.969+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35900546b0 con 0x7f359815ad70 2023-10-16T12:49:36.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.214+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f359804e160 con 0x7f359815ad70 2023-10-16T12:49:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.822+0000 7f35957fa640 1 -- 172.21.15.60:0/3462632888 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v29) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3590057cd0 con 0x7f359815ad70 2023-10-16T12:49:36.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.827+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f357c06af90 msgr2=0x7f357c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.827+0000 7f359ef05640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f357c06af90 0x7f357c06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f358c036000 tx=0x7f358c03c040 comp rx=0 tx=0).stop 2023-10-16T12:49:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.827+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 msgr2=0x7f35980c5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.827+0000 7f359ef05640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f35980c5df0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f359000b970 tx=0x7f359000be30 comp rx=0 tx=0).stop 2023-10-16T12:49:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 shutdown_connections 2023-10-16T12:49:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f357c06af90 0x7f357c06d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35980caac0 0x7f35980c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f359815ad70 0x7f35980c5df0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 --2- 172.21.15.60:0/3462632888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35980c9ca0 0x7f35981f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 >> 172.21.15.60:0/3462632888 conn(0x7f35980ce9b0 msgr2=0x7f35980ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 shutdown_connections 2023-10-16T12:49:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:36.828+0000 7f359ef05640 1 -- 172.21.15.60:0/3462632888 wait complete. 2023-10-16T12:49:36.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:49:37.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.054+0000 7f1505978640 1 Processor -- start 2023-10-16T12:49:37.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.054+0000 7f1505978640 1 -- start start 2023-10-16T12:49:37.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.056+0000 7f1505978640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1500159e40 0x7f15000cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.056+0000 7f1505978640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f150006b5c0 0x7f150006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.056+0000 7f1505978640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15000c3610 con 0x7f15000ca780 2023-10-16T12:49:37.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.056+0000 7f1505978640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150015a220 con 0x7f1500159e40 2023-10-16T12:49:37.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.056+0000 7f1505978640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15000cafd0 con 0x7f150006b5c0 2023-10-16T12:49:37.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.056+0000 7f14feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1500159e40 0x7f15000cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1500159e40 0x7f15000cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42846/0 (socket says 172.21.15.60:42846) 2023-10-16T12:49:37.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14feffd640 1 -- 172.21.15.60:0/2528025168 learned_addr learned my addr 172.21.15.60:0/2528025168 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:37.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14ff7fe640 1 --2- 172.21.15.60:0/2528025168 >> v2:172.21.15.144:3300/0 conn(0x7f150006b5c0 0x7f150006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14feffd640 1 -- 172.21.15.60:0/2528025168 >> v2:172.21.15.144:3300/0 conn(0x7f150006b5c0 msgr2=0x7f150006ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:37.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14feffd640 1 --2- 172.21.15.60:0/2528025168 >> v2:172.21.15.144:3300/0 conn(0x7f150006b5c0 0x7f150006ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14feffd640 1 -- 172.21.15.60:0/2528025168 >> v1:172.21.15.144:6789/0 conn(0x7f15000ca780 legacy=0x7f150006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:37.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14feffd640 1 -- 172.21.15.60:0/2528025168 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000cb2b0 con 0x7f1500159e40 2023-10-16T12:49:37.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.057+0000 7f14ff7fe640 1 --2- 172.21.15.60:0/2528025168 >> v2:172.21.15.144:3300/0 conn(0x7f150006b5c0 0x7f150006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:37.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.058+0000 7f14feffd640 1 --2- 172.21.15.60:0/2528025168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1500159e40 0x7f15000cda70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f150006fee0 tx=0x7f14f002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19aaa02fbb4b39db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:37.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.059+0000 7f14fe7fc640 1 -- 172.21.15.60:0/2528025168 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0031890 con 0x7f1500159e40 2023-10-16T12:49:37.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.059+0000 7f14fe7fc640 1 -- 172.21.15.60:0/2528025168 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f00319f0 con 0x7f1500159e40 2023-10-16T12:49:37.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.059+0000 7f14fe7fc640 1 -- 172.21.15.60:0/2528025168 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0031d80 con 0x7f1500159e40 2023-10-16T12:49:37.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.059+0000 7f1505978640 1 -- 172.21.15.60:0/2528025168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1500159e40 msgr2=0x7f15000cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.059+0000 7f1505978640 1 --2- 172.21.15.60:0/2528025168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1500159e40 0x7f15000cda70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f150006fee0 tx=0x7f14f002b930 comp rx=0 tx=0).stop 2023-10-16T12:49:37.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.060+0000 7f1505978640 1 -- 172.21.15.60:0/2528025168 shutdown_connections 2023-10-16T12:49:37.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.060+0000 7f1505978640 1 --2- 172.21.15.60:0/2528025168 >> v2:172.21.15.144:3300/0 conn(0x7f150006b5c0 0x7f150006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.060+0000 7f1505978640 1 --2- 172.21.15.60:0/2528025168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1500159e40 0x7f15000cda70 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.060+0000 7f1505978640 1 -- 172.21.15.60:0/2528025168 >> 172.21.15.60:0/2528025168 conn(0x7f15000cfac0 msgr2=0x7f15000cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:37.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.060+0000 7f1505978640 1 -- 172.21.15.60:0/2528025168 shutdown_connections 2023-10-16T12:49:37.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.060+0000 7f1505978640 1 -- 172.21.15.60:0/2528025168 wait complete. 2023-10-16T12:49:37.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.062+0000 7f1505978640 1 Processor -- start 2023-10-16T12:49:37.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.062+0000 7f1505978640 1 -- start start 2023-10-16T12:49:37.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.063+0000 7f1505978640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150006b5c0 0x7f15001e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f1505978640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 0x7f15000c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f1505978640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1500159e40 0x7f15000c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f1505978640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15001e7850 con 0x7f15000ca780 2023-10-16T12:49:37.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f1505978640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15000c6cc0 con 0x7f150006b5c0 2023-10-16T12:49:37.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f1505978640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15001e5630 con 0x7f1500159e40 2023-10-16T12:49:37.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 0x7f15000c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 0x7f15000c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42854/0 (socket says 172.21.15.60:42854) 2023-10-16T12:49:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f14f7fff640 1 -- 172.21.15.60:0/2828186655 learned_addr learned my addr 172.21.15.60:0/2828186655 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f14feffd640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150006b5c0 0x7f15001e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.064+0000 7f14ff7fe640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1500159e40 0x7f15000c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14f7fff640 1 -- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1500159e40 msgr2=0x7f15000c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14f7fff640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1500159e40 0x7f15000c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14f7fff640 1 -- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150006b5c0 msgr2=0x7f15001e50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14f7fff640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150006b5c0 0x7f15001e50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14f7fff640 1 -- 172.21.15.60:0/2828186655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00094d0 con 0x7f15000ca780 2023-10-16T12:49:37.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14f7fff640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 0x7f15000c4520 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f14e800c430 tx=0x7f14e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:37.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8011790 con 0x7f15000ca780 2023-10-16T12:49:37.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e80118f0 con 0x7f15000ca780 2023-10-16T12:49:37.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15000c6f60 con 0x7f15000ca780 2023-10-16T12:49:37.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8011ce0 con 0x7f15000ca780 2023-10-16T12:49:37.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.065+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15000c7520 con 0x7f15000ca780 2023-10-16T12:49:37.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.067+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15000cf250 con 0x7f15000ca780 2023-10-16T12:49:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.068+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 29) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f14e801c460 con 0x7f15000ca780 2023-10-16T12:49:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.068+0000 7f14fcff9640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 0x7f14e006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.069+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14e80896b0 con 0x7f15000ca780 2023-10-16T12:49:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.069+0000 7f14feffd640 1 -- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 msgr2=0x7f14e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:37.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.069+0000 7f14feffd640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 0x7f14e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:37.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.077+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14e8054640 con 0x7f15000ca780 2023-10-16T12:49:37.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.270+0000 7f14feffd640 1 -- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 msgr2=0x7f14e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:37.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.270+0000 7f14feffd640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 0x7f14e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:37.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.277+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f150004e160 con 0x7f15000ca780 2023-10-16T12:49:37.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.283+0000 7f14fcff9640 1 -- 172.21.15.60:0/2828186655 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f14e8057c60 con 0x7f15000ca780 2023-10-16T12:49:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 msgr2=0x7f14e006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 0x7f14e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 msgr2=0x7f15000c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 0x7f15000c4520 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f14e800c430 tx=0x7f14e800c8f0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 shutdown_connections 2023-10-16T12:49:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f14e006afe0 0x7f14e006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1500159e40 0x7f15000c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15000ca780 0x7f15000c4520 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 --2- 172.21.15.60:0/2828186655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f150006b5c0 0x7f15001e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 >> 172.21.15.60:0/2828186655 conn(0x7f15000cfac0 msgr2=0x7f15000c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 shutdown_connections 2023-10-16T12:49:37.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.286+0000 7f1505978640 1 -- 172.21.15.60:0/2828186655 wait complete. 2023-10-16T12:49:37.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:49:37.351 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:37.348+0000 7f344dc12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:49:37.375 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:37.373+0000 7fe9c7c3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:49:37.404 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:37.401+0000 7f344dc12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:49:37.434 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:37.432+0000 7fe9c7c3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:49:37.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.517+0000 7f19b359e640 1 Processor -- start 2023-10-16T12:49:37.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.517+0000 7f19b359e640 1 -- start start 2023-10-16T12:49:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.518+0000 7f19b359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19b40cded0 0x7f19b40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.518+0000 7f19b359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b415d150 con 0x7f19b40b8480 2023-10-16T12:49:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.518+0000 7f19b359e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b40d1180 con 0x7f19b415cd70 2023-10-16T12:49:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.518+0000 7f19b359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b40d1460 con 0x7f19b40cded0 2023-10-16T12:49:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.519+0000 7f19abfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f19b40b8480 0x7f19b40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50340/0 (socket says 172.21.15.60:50340) 2023-10-16T12:49:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.519+0000 7f19abfff640 1 -- 172.21.15.60:0/1343363852 learned_addr learned my addr 172.21.15.60:0/1343363852 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.519+0000 7f19b2d9d640 1 --2- 172.21.15.60:0/1343363852 >> v2:172.21.15.144:3300/0 conn(0x7f19b40cded0 0x7f19b40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.519+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939477289 0 0) 0x7f19b415d150 con 0x7f19b40b8480 2023-10-16T12:49:37.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.519+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a0003520 con 0x7f19b40b8480 2023-10-16T12:49:37.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2586698452 0 0) 0x7f19a0003520 con 0x7f19b40b8480 2023-10-16T12:49:37.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 >> v2:172.21.15.144:3300/0 conn(0x7f19b40cded0 msgr2=0x7f19b40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:37.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 --2- 172.21.15.60:0/1343363852 >> v2:172.21.15.144:3300/0 conn(0x7f19b40cded0 0x7f19b40b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 >> v1:172.21.15.144:6789/0 conn(0x7f19b415cd70 legacy=0x7f19b40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b40d1740 con 0x7f19b40b8480 2023-10-16T12:49:37.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f199c001670 con 0x7f19b40b8480 2023-10-16T12:49:37.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 >> v1:172.21.15.60:6789/0 conn(0x7f19b40b8480 legacy=0x7f19b40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19a00039c0 0x7f19a0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00047c0 0x7f19a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19a0005180 0x7f19a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40d1460 con 0x7f19a00039c0 2023-10-16T12:49:37.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a000c080 con 0x7f19a00047c0 2023-10-16T12:49:37.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a000c240 con 0x7f19a0005180 2023-10-16T12:49:37.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b259c640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19a00039c0 0x7f19a0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19abfff640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00047c0 0x7f19a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b2d9d640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19a0005180 0x7f19a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b259c640 1 -- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19a0005180 msgr2=0x7f19a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:37.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.520+0000 7f19b259c640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19a0005180 0x7f19a000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.521+0000 7f19b259c640 1 -- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00047c0 msgr2=0x7f19a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:37.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.521+0000 7f19b259c640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00047c0 0x7f19a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.521+0000 7f19b259c640 1 -- 172.21.15.60:0/1343363852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a0003520 con 0x7f19a00039c0 2023-10-16T12:49:37.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.521+0000 7f19b2d9d640 1 -- 172.21.15.60:0/1343363852 reap_dead start 2023-10-16T12:49:37.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.522+0000 7f19b259c640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19a00039c0 0x7f19a0003de0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f19a4001750 tx=0x7f19a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfc823321a0cd75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:37.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.522+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a40052d0 con 0x7f19a00039c0 2023-10-16T12:49:37.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.522+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a402e050 con 0x7f19a00039c0 2023-10-16T12:49:37.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.522+0000 7f19b1d9b640 1 -- 172.21.15.60:0/1343363852 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a4003830 con 0x7f19a00039c0 2023-10-16T12:49:37.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.522+0000 7f19b359e640 1 -- 172.21.15.60:0/1343363852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19a00039c0 msgr2=0x7f19a0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.522+0000 7f19b359e640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19a00039c0 0x7f19a0003de0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f19a4001750 tx=0x7f19a00057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.523+0000 7f19b359e640 1 -- 172.21.15.60:0/1343363852 shutdown_connections 2023-10-16T12:49:37.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.523+0000 7f19b359e640 1 --2- 172.21.15.60:0/1343363852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19a00039c0 0x7f19a0003de0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.523+0000 7f19b359e640 1 -- 172.21.15.60:0/1343363852 >> 172.21.15.60:0/1343363852 conn(0x7f19b40cf570 msgr2=0x7f19b40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:37.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.523+0000 7f19b359e640 1 -- 172.21.15.60:0/1343363852 shutdown_connections 2023-10-16T12:49:37.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.523+0000 7f19b359e640 1 -- 172.21.15.60:0/1343363852 wait complete. 2023-10-16T12:49:37.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.524+0000 7f19b359e640 1 Processor -- start 2023-10-16T12:49:37.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 -- start start 2023-10-16T12:49:37.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00039c0 0x7f19b40a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 0x7f19b40ab760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19b40abca0 0x7f19b41f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40ac100 con 0x7f19b40aa440 2023-10-16T12:49:37.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40aaa70 con 0x7f19a00039c0 2023-10-16T12:49:37.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.525+0000 7f19b359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40c9d40 con 0x7f19b40abca0 2023-10-16T12:49:37.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 0x7f19b40ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 0x7f19b40ab760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42882/0 (socket says 172.21.15.60:42882) 2023-10-16T12:49:37.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 -- 172.21.15.60:0/1368261528 learned_addr learned my addr 172.21.15.60:0/1368261528 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:37.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19b2d9d640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19b40abca0 0x7f19b41f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 -- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19b40abca0 msgr2=0x7f19b41f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19b40abca0 0x7f19b41f1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 -- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00039c0 msgr2=0x7f19b40a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19b259c640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00039c0 0x7f19b40a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:37.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00039c0 0x7f19b40a9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 -- 172.21.15.60:0/1368261528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b40ca020 con 0x7f19b40aa440 2023-10-16T12:49:37.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.526+0000 7f19abfff640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 0x7f19b40ab760 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f199c00e700 tx=0x7f199c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:37.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.527+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c00c6c0 con 0x7f19b40aa440 2023-10-16T12:49:37.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.527+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c00c840 con 0x7f19b40aa440 2023-10-16T12:49:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.527+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c00cbf0 con 0x7f19b40aa440 2023-10-16T12:49:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.527+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b40ca300 con 0x7f19b40aa440 2023-10-16T12:49:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.527+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b41f2250 con 0x7f19b40aa440 2023-10-16T12:49:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.528+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b40c9ba0 con 0x7f19b40aa440 2023-10-16T12:49:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.529+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 29) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f199c00cd50 con 0x7f19b40aa440 2023-10-16T12:49:37.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.530+0000 7f19aaffd640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 0x7f199406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:37.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.530+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f199c089430 con 0x7f19b40aa440 2023-10-16T12:49:37.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.530+0000 7f19b259c640 1 -- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 msgr2=0x7f199406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:37.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.531+0000 7f19b259c640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 0x7f199406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:37.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.539+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19b40c9ba0 con 0x7f19b40aa440 2023-10-16T12:49:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.732+0000 7f19b259c640 1 -- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 msgr2=0x7f199406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:37.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.732+0000 7f19b259c640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 0x7f199406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.737+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f19b40c9ba0 con 0x7f19b40aa440 2023-10-16T12:49:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.745+0000 7f19aaffd640 1 -- 172.21.15.60:0/1368261528 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f19b40c9ba0 con 0x7f19b40aa440 2023-10-16T12:49:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 msgr2=0x7f199406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 0x7f199406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 msgr2=0x7f19b40ab760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 0x7f19b40ab760 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f199c00e700 tx=0x7f199c00ebc0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 shutdown_connections 2023-10-16T12:49:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7f199406b010 0x7f199406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19b40abca0 0x7f19b41f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19b40aa440 0x7f19b40ab760 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 --2- 172.21.15.60:0/1368261528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19a00039c0 0x7f19b40a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 >> 172.21.15.60:0/1368261528 conn(0x7f19b40cf570 msgr2=0x7f19b40b75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 shutdown_connections 2023-10-16T12:49:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:37.747+0000 7f19b359e640 1 -- 172.21.15.60:0/1368261528 wait complete. 2023-10-16T12:49:37.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:37.769 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:49:37.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T12:49:37.855 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:37.852+0000 7f344dc12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:49:37.908 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:37.905+0000 7f344dc12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:49:37.920 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:37.918+0000 7fe9c7c3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:49:37.957 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:37.955+0000 7f344dc12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:49:37.977 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:37.974+0000 7fe9c7c3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:49:38.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.015+0000 7f344dc12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:49:38.030 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.028+0000 7fe9c7c3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:49:38.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.033+0000 7faf57075640 1 Processor -- start 2023-10-16T12:49:38.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.034+0000 7faf57075640 1 -- start start 2023-10-16T12:49:38.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.035+0000 7faf57075640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 0x7faf501659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.035+0000 7faf57075640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5015f590 con 0x7faf5006a4b0 2023-10-16T12:49:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.035+0000 7faf57075640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf500d05b0 con 0x7faf5015aa10 2023-10-16T12:49:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.035+0000 7faf57075640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf500cf7f0 con 0x7faf500b0e10 2023-10-16T12:49:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.035+0000 7faf55872640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faf5006a4b0 0x7faf500b0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50356/0 (socket says 172.21.15.60:50356) 2023-10-16T12:49:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.035+0000 7faf55872640 1 -- 172.21.15.60:0/1856606263 learned_addr learned my addr 172.21.15.60:0/1856606263 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:38.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf56874640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 0x7faf501659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478919822 0 0) 0x7faf5015f590 con 0x7faf5006a4b0 2023-10-16T12:49:38.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf3c003520 con 0x7faf5006a4b0 2023-10-16T12:49:38.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414594517 0 0) 0x7faf3c003520 con 0x7faf5006a4b0 2023-10-16T12:49:38.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 msgr2=0x7faf501659a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:38.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 0x7faf501659a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:38.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 >> v1:172.21.15.144:6789/0 conn(0x7faf5015aa10 legacy=0x7faf50069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:38.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf500cfad0 con 0x7faf5006a4b0 2023-10-16T12:49:38.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7faf40001670 con 0x7faf5006a4b0 2023-10-16T12:49:38.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 >> v1:172.21.15.60:6789/0 conn(0x7faf5006a4b0 legacy=0x7faf500b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:38.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf3c0039a0 0x7faf3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.036+0000 7faf55071640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 0x7faf3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf55071640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf3c004d00 0x7faf3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf500cf7f0 con 0x7faf3c004d00 2023-10-16T12:49:38.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c00ba30 con 0x7faf3c0039a0 2023-10-16T12:49:38.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c00bbf0 con 0x7faf500b0e10 2023-10-16T12:49:38.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56874640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf3c004d00 0x7faf3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56073640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf3c0039a0 0x7faf3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf55872640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 0x7faf3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56874640 1 -- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 msgr2=0x7faf3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:38.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56874640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf500b0e10 0x7faf3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:38.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56874640 1 -- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf3c0039a0 msgr2=0x7faf3c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:38.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56874640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf3c0039a0 0x7faf3c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:38.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.037+0000 7faf56874640 1 -- 172.21.15.60:0/1856606263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c003520 con 0x7faf3c004d00 2023-10-16T12:49:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.039+0000 7faf56874640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf3c004d00 0x7faf3c005180 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7faf5006e380 tx=0x7faf3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ba452a793e0be8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.039+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4802cb90 con 0x7faf3c004d00 2023-10-16T12:49:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.039+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4800f040 con 0x7faf3c004d00 2023-10-16T12:49:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.039+0000 7faf55071640 1 -- 172.21.15.60:0/1856606263 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf48034440 con 0x7faf3c004d00 2023-10-16T12:49:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf57075640 1 -- 172.21.15.60:0/1856606263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf3c004d00 msgr2=0x7faf3c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf57075640 1 --2- 172.21.15.60:0/1856606263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf3c004d00 0x7faf3c005180 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7faf5006e380 tx=0x7faf3c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf56874640 1 -- 172.21.15.60:0/1856606263 reap_dead start 2023-10-16T12:49:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf57075640 1 -- 172.21.15.60:0/1856606263 shutdown_connections 2023-10-16T12:49:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf57075640 1 -- 172.21.15.60:0/1856606263 >> 172.21.15.60:0/1856606263 conn(0x7faf5006aeb0 msgr2=0x7faf5006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf57075640 1 -- 172.21.15.60:0/1856606263 shutdown_connections 2023-10-16T12:49:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.040+0000 7faf57075640 1 -- 172.21.15.60:0/1856606263 wait complete. 2023-10-16T12:49:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.042+0000 7faf57075640 1 Processor -- start 2023-10-16T12:49:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.042+0000 7faf57075640 1 -- start start 2023-10-16T12:49:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf57075640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf3c004d00 0x7faf5015c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf57075640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf5014cc00 0x7faf500cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf57075640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf500cbc30 0x7faf5006c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf57075640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5014d080 con 0x7faf5014cc00 2023-10-16T12:49:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf57075640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5015b240 con 0x7faf500cbc30 2023-10-16T12:49:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf57075640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5015b520 con 0x7faf3c004d00 2023-10-16T12:49:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.043+0000 7faf55872640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf5014cc00 0x7faf500cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf5014cc00 0x7faf500cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42906/0 (socket says 172.21.15.60:42906) 2023-10-16T12:49:38.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 -- 172.21.15.60:0/2693361850 learned_addr learned my addr 172.21.15.60:0/2693361850 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:38.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf3c004d00 0x7faf5015c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf3c004d00 msgr2=0x7faf5015c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:38.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf56874640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf500cbc30 0x7faf5006c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:38.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf3c004d00 0x7faf5015c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:38.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf500cbc30 msgr2=0x7faf5006c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:38.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf500cbc30 0x7faf5006c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:38.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 -- 172.21.15.60:0/2693361850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5015b800 con 0x7faf5014cc00 2023-10-16T12:49:38.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf56874640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf500cbc30 0x7faf5006c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:38.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.044+0000 7faf55872640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf5014cc00 0x7faf500cb6f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7faf4000e8c0 tx=0x7faf4000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:38.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.045+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4000c8d0 con 0x7faf5014cc00 2023-10-16T12:49:38.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.045+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4000ca30 con 0x7faf5014cc00 2023-10-16T12:49:38.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.045+0000 7faf57075640 1 -- 172.21.15.60:0/2693361850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5015bae0 con 0x7faf5014cc00 2023-10-16T12:49:38.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.045+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4000cde0 con 0x7faf5014cc00 2023-10-16T12:49:38.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.045+0000 7faf57075640 1 -- 172.21.15.60:0/2693361850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5015adf0 con 0x7faf5014cc00 2023-10-16T12:49:38.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.046+0000 7faf57075640 1 -- 172.21.15.60:0/2693361850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf500674f0 con 0x7faf5014cc00 2023-10-16T12:49:38.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.047+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 29) v1 ==== 89348+0+0 (secure 0 0 0) 0x7faf4001d460 con 0x7faf5014cc00 2023-10-16T12:49:38.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.048+0000 7faf46ffd640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 0x7faf3406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:38.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.049+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf40089980 con 0x7faf5014cc00 2023-10-16T12:49:38.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.049+0000 7faf56073640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 msgr2=0x7faf3406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:38.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.049+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 0x7faf3406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:38.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.052+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf40054880 con 0x7faf5014cc00 2023-10-16T12:49:38.067 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.065+0000 7f344dc12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:49:38.098 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.096+0000 7fe9c7c3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:49:38.151 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.149+0000 7fe9c7c3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:49:38.178 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.175+0000 7f344dc12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:49:38.239 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.237+0000 7f344dc12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:38.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.250+0000 7faf56073640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 msgr2=0x7faf3406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:38.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.250+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 0x7faf3406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:38.268 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.265+0000 7fe9c7c3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:49:38.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.268+0000 7faf57075640 1 -- 172.21.15.60:0/2693361850 --> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7faf5015b690 con 0x7faf3406afe0 2023-10-16T12:49:38.288 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.285+0000 7f344dc12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:49:38.334 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.331+0000 7fe9c7c3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:38.345 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.343+0000 7f344dc12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:49:38.386 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.383+0000 7fe9c7c3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:49:38.396 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.393+0000 7f344dc12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:49:38.444 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.442+0000 7fe9c7c3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:49:38.498 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.495+0000 7fe9c7c3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:49:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.651+0000 7faf56073640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 msgr2=0x7faf3406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:38.651+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 0x7faf3406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:49:38.775 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.772+0000 7f344dc12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:49:38.899 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:38.897+0000 7fe9c7c3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:49:38.933 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:38.930+0000 7f344dc12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:49:39.041 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.038+0000 7f344dc12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:49:39.067 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.065+0000 7fe9c7c3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:49:39.090 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.088+0000 7f344dc12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:49:39.180 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.178+0000 7fe9c7c3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:49:39.197 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.194+0000 7f344dc12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:39.232 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.230+0000 7fe9c7c3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:49:39.304 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.302+0000 7f344dc12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:49:39.346 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.344+0000 7fe9c7c3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:39.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:39.452+0000 7faf56073640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 msgr2=0x7faf3406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2872949001 2023-10-16T12:49:39.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:39.452+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 0x7faf3406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:49:39.458 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.456+0000 7fe9c7c3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:49:39.460 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.457+0000 7f344dc12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:39.509 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.506+0000 7f344dc12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:39.623 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.621+0000 7fe9c7c3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:39.628 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:39.625+0000 7f344dc12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:49:39.675 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.673+0000 7fe9c7c3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:39.801 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:39.798+0000 7fe9c7c3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:49:40.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:40.442+0000 7f344dc12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:49:40.668 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:40.665+0000 7fe9c7c3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:49:40.737 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:40.734+0000 7f344dc12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:49:40.793 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:40.790+0000 7f344dc12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:49:40.907 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:40.904+0000 7f344dc12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:49:40.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:40.927+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 30) v1 ==== 88957+0+0 (secure 0 0 0) 0x7faf40054c50 con 0x7faf5014cc00 2023-10-16T12:49:40.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:40.928+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 msgr2=0x7faf3406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:40.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:40.928+0000 7faf46ffd640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2872949001,v1:172.21.15.144:6833/2872949001] conn(0x7faf3406afe0 0x7faf3406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:40.991 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:40.988+0000 7fe9c7c3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:49:41.056 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:41.053+0000 7fe9c7c3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:49:41.179 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:41.176+0000 7fe9c7c3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:49:41.238 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ca287640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.239 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ca287640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.239 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ca287640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.239 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ca287640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.239 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ca287640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.239 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ccacc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.240 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ccacc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.240 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ccacc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.240 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ccacc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.240 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.235+0000 7f33ccacc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:41.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:41.937+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 31) v1 ==== 89084+0+0 (secure 0 0 0) 0x7faf4001ad40 con 0x7faf5014cc00 2023-10-16T12:49:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:41.938+0000 7faf46ffd640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7faf340355a0 0x7faf34035980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:41.938+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 --> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7faf5015b690 con 0x7faf340355a0 2023-10-16T12:49:41.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:41.948+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7faf340355a0 0x7faf34035980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:41.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:41.957+0000 7faf56073640 1 --2- 172.21.15.60:0/2693361850 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7faf340355a0 0x7faf34035980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faf4c007c80 tx=0x7faf4c009040 comp rx=0 tx=0).ready entity=mgr.5806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:41.978 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:41.976+0000 7f33d42db640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T12:49:41.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:41.976+0000 7faf46ffd640 1 -- 172.21.15.60:0/2693361850 <== mgr.5806 v2:172.21.15.144:6832/2100131827 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7faf5015b690 con 0x7faf340355a0 2023-10-16T12:49:41.979 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T12:49:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:49:41.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:41.997 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T12:49:42.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:49:42.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.286+0000 7f519640e640 1 Processor -- start 2023-10-16T12:49:42.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.286+0000 7f519640e640 1 -- start start 2023-10-16T12:49:42.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.287+0000 7f519640e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 0x7f51900ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.287+0000 7f519640e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5190070360 con 0x7f5190159e90 2023-10-16T12:49:42.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.287+0000 7f519640e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51900cd1f0 con 0x7f519006c140 2023-10-16T12:49:42.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.287+0000 7f519640e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519015d890 con 0x7f5190159ab0 2023-10-16T12:49:42.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.288+0000 7f5187fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5190159e90 0x7f519006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38682/0 (socket says 172.21.15.60:38682) 2023-10-16T12:49:42.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.288+0000 7f5187fff640 1 -- 172.21.15.60:0/548147904 learned_addr learned my addr 172.21.15.60:0/548147904 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:42.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.288+0000 7f518ffff640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 0x7f51900ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.288+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046833109 0 0) 0x7f5190070360 con 0x7f5190159e90 2023-10-16T12:49:42.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.288+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f517c003520 con 0x7f5190159e90 2023-10-16T12:49:42.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439778900 0 0) 0x7f517c003520 con 0x7f5190159e90 2023-10-16T12:49:42.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 msgr2=0x7f51900ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:42.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 0x7f51900ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 >> v1:172.21.15.144:6789/0 conn(0x7f519006c140 legacy=0x7f519015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51900cd4d0 con 0x7f5190159e90 2023-10-16T12:49:42.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5178001670 con 0x7f5190159e90 2023-10-16T12:49:42.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 >> v1:172.21.15.60:6789/0 conn(0x7f5190159e90 legacy=0x7f519006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 0x7f517c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517c004d00 0x7f517c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519015d890 con 0x7f517c0039a0 2023-10-16T12:49:42.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00bb40 con 0x7f517c004d00 2023-10-16T12:49:42.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c00bd00 con 0x7f5190159ab0 2023-10-16T12:49:42.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.289+0000 7f518ffff640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f51949ab640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517c004d00 0x7f517c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f5187fff640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 0x7f517c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f518ffff640 1 -- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 msgr2=0x7f517c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:42.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f518ffff640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5190159ab0 0x7f517c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f518ffff640 1 -- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517c004d00 msgr2=0x7f517c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:42.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f518ffff640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517c004d00 0x7f517c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f518ffff640 1 -- 172.21.15.60:0/548147904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c003520 con 0x7f517c0039a0 2023-10-16T12:49:42.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.290+0000 7f51949ab640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517c004d00 0x7f517c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:42.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.291+0000 7f518ffff640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5180009ac0 tx=0x7f517c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4478dc95a51a11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:42.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.291+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518002c6b0 con 0x7f517c0039a0 2023-10-16T12:49:42.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.291+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518002c810 con 0x7f517c0039a0 2023-10-16T12:49:42.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.291+0000 7f518f7fe640 1 -- 172.21.15.60:0/548147904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518002cba0 con 0x7f517c0039a0 2023-10-16T12:49:42.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f519640e640 1 -- 172.21.15.60:0/548147904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f517c0039a0 msgr2=0x7f517c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f519640e640 1 --2- 172.21.15.60:0/548147904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f517c0039a0 0x7f517c003dc0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5180009ac0 tx=0x7f517c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f51949ab640 1 -- 172.21.15.60:0/548147904 reap_dead start 2023-10-16T12:49:42.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f519640e640 1 -- 172.21.15.60:0/548147904 shutdown_connections 2023-10-16T12:49:42.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f519640e640 1 -- 172.21.15.60:0/548147904 >> 172.21.15.60:0/548147904 conn(0x7f51900ce9b0 msgr2=0x7f51900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:42.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f519640e640 1 -- 172.21.15.60:0/548147904 shutdown_connections 2023-10-16T12:49:42.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.292+0000 7f519640e640 1 -- 172.21.15.60:0/548147904 wait complete. 2023-10-16T12:49:42.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.294+0000 7f519640e640 1 Processor -- start 2023-10-16T12:49:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.294+0000 7f519640e640 1 -- start start 2023-10-16T12:49:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f519640e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 0x7f519014ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f519640e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51900c6b00 0x7f51900c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f519640e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51900c3be0 0x7f51900c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f519640e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51900c74c0 con 0x7f51900c5050 2023-10-16T12:49:42.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f519640e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51900c5430 con 0x7f51900c6b00 2023-10-16T12:49:42.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f519640e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51900c3210 con 0x7f51900c3be0 2023-10-16T12:49:42.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f518ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 0x7f519014ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f518ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 0x7f519014ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60608/0 (socket says 172.21.15.60:60608) 2023-10-16T12:49:42.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.295+0000 7f518ffff640 1 -- 172.21.15.60:0/121115773 learned_addr learned my addr 172.21.15.60:0/121115773 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:42.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f51949ab640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51900c3be0 0x7f51900c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518ffff640 1 -- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51900c3be0 msgr2=0x7f51900c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f5187fff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51900c6b00 0x7f51900c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518ffff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51900c3be0 0x7f51900c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518ffff640 1 -- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51900c6b00 msgr2=0x7f51900c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518ffff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51900c6b00 0x7f51900c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518ffff640 1 -- 172.21.15.60:0/121115773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519014df60 con 0x7f51900c5050 2023-10-16T12:49:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f5187fff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51900c6b00 0x7f51900c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518ffff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 0x7f519014ea40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f518002c2a0 tx=0x7f5180004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:42.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180003810 con 0x7f51900c5050 2023-10-16T12:49:42.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5180003970 con 0x7f51900c5050 2023-10-16T12:49:42.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180003d20 con 0x7f51900c5050 2023-10-16T12:49:42.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.296+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519014e240 con 0x7f51900c5050 2023-10-16T12:49:42.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.297+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51900c45a0 con 0x7f51900c5050 2023-10-16T12:49:42.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.298+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519014dda0 con 0x7f51900c5050 2023-10-16T12:49:42.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.298+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 31) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f518003b070 con 0x7f51900c5050 2023-10-16T12:49:42.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.299+0000 7f518d7fa640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f517006ad40 0x7f517006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:42.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.300+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f51800aae80 con 0x7f51900c5050 2023-10-16T12:49:42.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.300+0000 7f5187fff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f517006ad40 0x7f517006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:42.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.301+0000 7f5187fff640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f517006ad40 0x7f517006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f51900b4070 tx=0x7f5178009040 comp rx=0 tx=0).ready entity=mgr.5806 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:42.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.304+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51800761c0 con 0x7f51900c5050 2023-10-16T12:49:42.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.549+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f519004e160 con 0x7f51900c5050 2023-10-16T12:49:42.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.982+0000 7f518d7fa640 1 -- 172.21.15.60:0/121115773 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v32) v1 ==== 93+0+0 (secure 0 0 0) 0x7f51800797e0 con 0x7f51900c5050 2023-10-16T12:49:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f517006ad40 msgr2=0x7f517006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f517006ad40 0x7f517006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f51900b4070 tx=0x7f5178009040 comp rx=0 tx=0).stop 2023-10-16T12:49:42.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 msgr2=0x7f519014ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:42.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 0x7f519014ea40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f518002c2a0 tx=0x7f5180004730 comp rx=0 tx=0).stop 2023-10-16T12:49:42.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 shutdown_connections 2023-10-16T12:49:42.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f517006ad40 0x7f517006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51900c3be0 0x7f51900c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51900c6b00 0x7f51900c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 --2- 172.21.15.60:0/121115773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51900c5050 0x7f519014ea40 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.986+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 >> 172.21.15.60:0/121115773 conn(0x7f51900ce9b0 msgr2=0x7f519006c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.987+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 shutdown_connections 2023-10-16T12:49:42.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:42.987+0000 7f519640e640 1 -- 172.21.15.60:0/121115773 wait complete. 2023-10-16T12:49:43.014 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:49:43.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:43.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:43.213 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:49:43.213 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:43.214 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:43.264 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:43.265 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:43.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:43.518 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:43.516+0000 7fa55b64f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:49:43.562 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:43.560+0000 7f2ac1cb6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:49:43.571 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:43.570+0000 7fa55b64f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:49:43.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.574+0000 7f8163d1d640 1 Processor -- start 2023-10-16T12:49:43.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.574+0000 7f8163d1d640 1 -- start start 2023-10-16T12:49:43.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.576+0000 7f8163d1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f815c0b0b20 0x7f815c164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:43.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.576+0000 7f8163d1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815c071450 con 0x7f815c0adb50 2023-10-16T12:49:43.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.576+0000 7f8163d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815c0cc600 con 0x7f815c070fd0 2023-10-16T12:49:43.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.576+0000 7f8163d1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f815c0cc8e0 con 0x7f815c0b0b20 2023-10-16T12:49:43.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.576+0000 7f8161ab9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f815c0adb50 0x7f815c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38696/0 (socket says 172.21.15.60:38696) 2023-10-16T12:49:43.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.576+0000 7f8161ab9640 1 -- 172.21.15.60:0/1869797307 learned_addr learned my addr 172.21.15.60:0/1869797307 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:43.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f81622ba640 1 --2- 172.21.15.60:0/1869797307 >> v2:172.21.15.144:3300/0 conn(0x7f815c0b0b20 0x7f815c164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:43.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f8160ab7640 1 -- 172.21.15.60:0/1869797307 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151999958 0 0) 0x7f815c071450 con 0x7f815c0adb50 2023-10-16T12:49:43.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f8160ab7640 1 -- 172.21.15.60:0/1869797307 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8148003520 con 0x7f815c0adb50 2023-10-16T12:49:43.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f81622ba640 1 -- 172.21.15.60:0/1869797307 >> v1:172.21.15.144:6789/0 conn(0x7f815c070fd0 legacy=0x7f815c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:43.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f81622ba640 1 -- 172.21.15.60:0/1869797307 >> v1:172.21.15.60:6789/0 conn(0x7f815c0adb50 legacy=0x7f815c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:43.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f8160ab7640 1 -- 172.21.15.60:0/1869797307 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916034817 0 0) 0x7f8148003520 con 0x7f815c0adb50 2023-10-16T12:49:43.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.577+0000 7f81622ba640 1 -- 172.21.15.60:0/1869797307 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c0ccbc0 con 0x7f815c0b0b20 2023-10-16T12:49:43.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.579+0000 7f81622ba640 1 --2- 172.21.15.60:0/1869797307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0b0b20 0x7f815c164f60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8154009d90 tx=0x7f815402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02a90b6345db880 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:43.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.579+0000 7f8160ab7640 1 -- 172.21.15.60:0/1869797307 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158001670 con 0x7f815c0b0b20 2023-10-16T12:49:43.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.579+0000 7f8160ab7640 1 -- 172.21.15.60:0/1869797307 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815402c940 con 0x7f815c0b0b20 2023-10-16T12:49:43.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.579+0000 7f8160ab7640 1 -- 172.21.15.60:0/1869797307 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815402ccb0 con 0x7f815c0b0b20 2023-10-16T12:49:43.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 -- 172.21.15.60:0/1869797307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0b0b20 msgr2=0x7f815c164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:43.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 --2- 172.21.15.60:0/1869797307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0b0b20 0x7f815c164f60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8154009d90 tx=0x7f815402be10 comp rx=0 tx=0).stop 2023-10-16T12:49:43.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 -- 172.21.15.60:0/1869797307 shutdown_connections 2023-10-16T12:49:43.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 --2- 172.21.15.60:0/1869797307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0b0b20 0x7f815c164f60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 -- 172.21.15.60:0/1869797307 >> 172.21.15.60:0/1869797307 conn(0x7f815c0cb410 msgr2=0x7f815c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:43.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 -- 172.21.15.60:0/1869797307 shutdown_connections 2023-10-16T12:49:43.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.580+0000 7f8163d1d640 1 -- 172.21.15.60:0/1869797307 wait complete. 2023-10-16T12:49:43.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.582+0000 7f8163d1d640 1 Processor -- start 2023-10-16T12:49:43.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.582+0000 7f8163d1d640 1 -- start start 2023-10-16T12:49:43.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8163d1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815c070fd0 0x7f815c1f78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:43.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8163d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0adb50 0x7f815c14ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:43.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8163d1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 0x7f815c0c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:43.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8163d1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c0b4540 con 0x7f815c0b0b20 2023-10-16T12:49:43.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8163d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c0c5890 con 0x7f815c0adb50 2023-10-16T12:49:43.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8163d1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c0c55b0 con 0x7f815c070fd0 2023-10-16T12:49:43.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f81622ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 0x7f815c0c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:43.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f81622ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 0x7f815c0c4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60628/0 (socket says 172.21.15.60:60628) 2023-10-16T12:49:43.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f81622ba640 1 -- 172.21.15.60:0/2708107070 learned_addr learned my addr 172.21.15.60:0/2708107070 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:43.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f8161ab9640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815c070fd0 0x7f815c1f78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:43.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.583+0000 7f81612b8640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0adb50 0x7f815c14ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:43.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81622ba640 1 -- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815c070fd0 msgr2=0x7f815c1f78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:43.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81622ba640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815c070fd0 0x7f815c1f78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81622ba640 1 -- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0adb50 msgr2=0x7f815c14ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:43.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81622ba640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0adb50 0x7f815c14ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81622ba640 1 -- 172.21.15.60:0/2708107070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8154009a70 con 0x7f815c0b0b20 2023-10-16T12:49:43.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81622ba640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 0x7f815c0c4d60 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8154009eb0 tx=0x7f8154004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:43.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154007000 con 0x7f815c0b0b20 2023-10-16T12:49:43.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8154004250 con 0x7f815c0b0b20 2023-10-16T12:49:43.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154004620 con 0x7f815c0b0b20 2023-10-16T12:49:43.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c162b70 con 0x7f815c0b0b20 2023-10-16T12:49:43.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.584+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c163130 con 0x7f815c0b0b20 2023-10-16T12:49:43.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.585+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815c04e160 con 0x7f815c0b0b20 2023-10-16T12:49:43.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.587+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8154034440 con 0x7f815c0b0b20 2023-10-16T12:49:43.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.587+0000 7f81527fc640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:43.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.587+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f81540ac1e0 con 0x7f815c0b0b20 2023-10-16T12:49:43.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.588+0000 7f8161ab9640 1 -- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 msgr2=0x7f814006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:43.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.588+0000 7f8161ab9640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:43.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.592+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81540770b0 con 0x7f815c0b0b20 2023-10-16T12:49:43.622 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:43.620+0000 7f2ac1cb6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:49:43.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.786+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f815c0b4910 con 0x7f815c0b0b20 2023-10-16T12:49:43.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.788+0000 7f81527fc640 1 -- 172.21.15.60:0/2708107070 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+17060 (secure 0 0 0) 0x7f815407a6d0 con 0x7f815c0b0b20 2023-10-16T12:49:43.791 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:43.791 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:40.910632+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/954403568":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2870481770":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/3154142782":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2412969789":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2725919540":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/342775588":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2872949001":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6832/2872949001":"2023-10-17T12:49:40.910583+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.789+0000 7f8161ab9640 1 -- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 msgr2=0x7f814006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:43.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.789+0000 7f8161ab9640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.790+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 msgr2=0x7f814006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.790+0000 7f8163d1d640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.790+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 msgr2=0x7f815c0c4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.790+0000 7f8163d1d640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 0x7f815c0c4d60 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8154009eb0 tx=0x7f8154004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.790+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 shutdown_connections 2023-10-16T12:49:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.790+0000 7f8163d1d640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f815c0b0b20 0x7f815c0c4d60 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.791+0000 7f8163d1d640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f815c0adb50 0x7f815c14ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.791+0000 7f8163d1d640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.791+0000 7f8163d1d640 1 --2- 172.21.15.60:0/2708107070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815c070fd0 0x7f815c1f78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.791+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 >> 172.21.15.60:0/2708107070 conn(0x7f815c0cb410 msgr2=0x7f815c160cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.791+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 shutdown_connections 2023-10-16T12:49:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:43.791+0000 7f8163d1d640 1 -- 172.21.15.60:0/2708107070 wait complete. 2023-10-16T12:49:43.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:49:24.152341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:49:25.216482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:43.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:44.014 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.013+0000 7fa55b64f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:49:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.014+0000 7fc11f59e640 1 Processor -- start 2023-10-16T12:49:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.014+0000 7fc11f59e640 1 -- start start 2023-10-16T12:49:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.016+0000 7fc11f59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc120070f30 0x7fc12006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.016+0000 7fc11f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc12006b3e0 0x7fc1200c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.016+0000 7fc11f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200cd260 con 0x7fc120159ad0 2023-10-16T12:49:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.016+0000 7fc11f59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12015de30 con 0x7fc120070f30 2023-10-16T12:49:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.016+0000 7fc11f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12015e110 con 0x7fc12006b3e0 2023-10-16T12:49:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc117fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc120070f30 0x7fc12006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc11ed9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc12006b3e0 0x7fc1200c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc117fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc120070f30 0x7fc12006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:36538/0 (socket says 172.21.15.60:36538) 2023-10-16T12:49:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc117fff640 1 -- 172.21.15.60:0/292746352 learned_addr learned my addr 172.21.15.60:0/292746352 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc11ed9d640 1 -- 172.21.15.60:0/292746352 >> v2:172.21.15.144:3300/0 conn(0x7fc120070f30 msgr2=0x7fc12006aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc11ed9d640 1 --2- 172.21.15.60:0/292746352 >> v2:172.21.15.144:3300/0 conn(0x7fc120070f30 0x7fc12006aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc11ed9d640 1 -- 172.21.15.60:0/292746352 >> v1:172.21.15.144:6789/0 conn(0x7fc120159ad0 legacy=0x7fc12015d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.017+0000 7fc11ed9d640 1 -- 172.21.15.60:0/292746352 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12015e3f0 con 0x7fc12006b3e0 2023-10-16T12:49:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.019+0000 7fc11ed9d640 1 --2- 172.21.15.60:0/292746352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc12006b3e0 0x7fc1200c9de0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc118009d90 tx=0x7fc11802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf199a81f7813f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.019+0000 7fc11dd9b640 1 -- 172.21.15.60:0/292746352 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11802cc50 con 0x7fc12006b3e0 2023-10-16T12:49:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.019+0000 7fc11dd9b640 1 -- 172.21.15.60:0/292746352 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11802cdb0 con 0x7fc12006b3e0 2023-10-16T12:49:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.019+0000 7fc11dd9b640 1 -- 172.21.15.60:0/292746352 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118034660 con 0x7fc12006b3e0 2023-10-16T12:49:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 -- 172.21.15.60:0/292746352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc12006b3e0 msgr2=0x7fc1200c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 --2- 172.21.15.60:0/292746352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc12006b3e0 0x7fc1200c9de0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc118009d90 tx=0x7fc11802bea0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 -- 172.21.15.60:0/292746352 shutdown_connections 2023-10-16T12:49:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 --2- 172.21.15.60:0/292746352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc12006b3e0 0x7fc1200c9de0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 --2- 172.21.15.60:0/292746352 >> v2:172.21.15.144:3300/0 conn(0x7fc120070f30 0x7fc12006aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 -- 172.21.15.60:0/292746352 >> 172.21.15.60:0/292746352 conn(0x7fc1200b0170 msgr2=0x7fc1200b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 -- 172.21.15.60:0/292746352 shutdown_connections 2023-10-16T12:49:44.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.020+0000 7fc11f59e640 1 -- 172.21.15.60:0/292746352 wait complete. 2023-10-16T12:49:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.022+0000 7fc11f59e640 1 Processor -- start 2023-10-16T12:49:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.022+0000 7fc11f59e640 1 -- start start 2023-10-16T12:49:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 0x7fc12015a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc120070f30 0x7fc12014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc120159ad0 0x7fc12014dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1200c58e0 con 0x7fc12006b3e0 2023-10-16T12:49:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12015c8f0 con 0x7fc120159ad0 2023-10-16T12:49:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12015cbd0 con 0x7fc120070f30 2023-10-16T12:49:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 0x7fc12015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 0x7fc12015a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60630/0 (socket says 172.21.15.60:60630) 2023-10-16T12:49:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 -- 172.21.15.60:0/2192424128 learned_addr learned my addr 172.21.15.60:0/2192424128 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc117fff640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc120070f30 0x7fc12014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11ed9d640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc120159ad0 0x7fc12014dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 -- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc120070f30 msgr2=0x7fc12014d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc120070f30 0x7fc12014d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 -- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc120159ad0 msgr2=0x7fc12014dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc120159ad0 0x7fc12014dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.023+0000 7fc11e59c640 1 -- 172.21.15.60:0/2192424128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc118009a70 con 0x7fc12006b3e0 2023-10-16T12:49:44.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.024+0000 7fc11e59c640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 0x7fc12015a150 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c400 tx=0x7fc11000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:44.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.024+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1100052d0 con 0x7fc12006b3e0 2023-10-16T12:49:44.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.024+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc110005430 con 0x7fc12006b3e0 2023-10-16T12:49:44.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.024+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc110003830 con 0x7fc12006b3e0 2023-10-16T12:49:44.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.024+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12015ceb0 con 0x7fc12006b3e0 2023-10-16T12:49:44.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.024+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12015a7e0 con 0x7fc12006b3e0 2023-10-16T12:49:44.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.026+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12015d0a0 con 0x7fc12006b3e0 2023-10-16T12:49:44.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.026+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc110003990 con 0x7fc12006b3e0 2023-10-16T12:49:44.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.027+0000 7fc116ffd640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 0x7fc10006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.027+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc110088f10 con 0x7fc12006b3e0 2023-10-16T12:49:44.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.027+0000 7fc117fff640 1 -- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 msgr2=0x7fc10006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:44.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.027+0000 7fc117fff640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 0x7fc10006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:44.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.031+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc110053cb0 con 0x7fc12006b3e0 2023-10-16T12:49:44.067 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.065+0000 7fa55b64f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:49:44.103 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.101+0000 7f2ac1cb6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:49:44.116 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.115+0000 7fa55b64f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:49:44.158 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.156+0000 7f2ac1cb6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:49:44.176 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.174+0000 7fa55b64f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:49:44.211 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.209+0000 7f2ac1cb6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:49:44.224 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.222+0000 7fa55b64f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:49:44.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.224+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc12004e9b0 con 0x7fc12006b3e0 2023-10-16T12:49:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.225+0000 7fc116ffd640 1 -- 172.21.15.60:0/2192424128 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1100170a0 con 0x7fc12006b3e0 2023-10-16T12:49:44.228 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:44.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.228+0000 7fc117fff640 1 -- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 msgr2=0x7fc10006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:44.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.228+0000 7fc117fff640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 0x7fc10006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:44.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 msgr2=0x7fc10006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:44.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 0x7fc10006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 msgr2=0x7fc12015a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 0x7fc12015a150 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c400 tx=0x7fc11000c8c0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 shutdown_connections 2023-10-16T12:49:44.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7fc10006af90 0x7fc10006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc120070f30 0x7fc12014d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc120159ad0 0x7fc12014dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 --2- 172.21.15.60:0/2192424128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12006b3e0 0x7fc12015a150 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 >> 172.21.15.60:0/2192424128 conn(0x7fc1200b0170 msgr2=0x7fc12006c520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 shutdown_connections 2023-10-16T12:49:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.229+0000 7fc11f59e640 1 -- 172.21.15.60:0/2192424128 wait complete. 2023-10-16T12:49:44.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:49:44.286 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.283+0000 7f2ac1cb6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:49:44.332 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.330+0000 7fa55b64f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:49:44.338 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.336+0000 7f2ac1cb6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:49:44.394 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.392+0000 7fa55b64f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:44.442 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.440+0000 7fa55b64f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:49:44.457 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.454+0000 7f2ac1cb6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:49:44.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.475+0000 7f34a0f6c640 1 Processor -- start 2023-10-16T12:49:44.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.475+0000 7f34a0f6c640 1 -- start start 2023-10-16T12:49:44.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.477+0000 7f34a0f6c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.477+0000 7f34a0f6c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f349c0cfaf0 0x7f349c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.477+0000 7f34a0f6c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349c0d3980 con 0x7f349c0b38e0 2023-10-16T12:49:44.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.477+0000 7f34a0f6c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c0d3c60 con 0x7f349c0cfaf0 2023-10-16T12:49:44.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.477+0000 7f34a0f6c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0ce630 con 0x7f349c0cf710 2023-10-16T12:49:44.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f3493fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f349c0cfaf0 0x7f349c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f349a59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f349a59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:50062/0 (socket says 172.21.15.60:50062) 2023-10-16T12:49:44.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f3493fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f349c0cfaf0 0x7f349c0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:36550/0 (socket says 172.21.15.60:36550) 2023-10-16T12:49:44.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f349a59c640 1 -- 172.21.15.60:0/1381263360 learned_addr learned my addr 172.21.15.60:0/1381263360 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:44.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f349ad9d640 1 --1- 172.21.15.60:0/1381263360 >> v1:172.21.15.144:6789/0 conn(0x7f349c0b38e0 0x7f349c0d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:44280/0 (socket says 172.21.15.60:44280) 2023-10-16T12:49:44.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f3493fff640 1 -- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 msgr2=0x7f349c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:44.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f3493fff640 1 --2- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f3493fff640 1 -- 172.21.15.60:0/1381263360 >> v1:172.21.15.144:6789/0 conn(0x7f349c0b38e0 legacy=0x7f349c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f3493fff640 1 -- 172.21.15.60:0/1381263360 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0ce910 con 0x7f349c0cfaf0 2023-10-16T12:49:44.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.478+0000 7f349a59c640 1 --2- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:49:44.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.480+0000 7f3493fff640 1 --2- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0b3250 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f34840097a0 tx=0x7f348402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86af47f3207e7af0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:44.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.480+0000 7f3499d9b640 1 -- 172.21.15.60:0/1381263360 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484031780 con 0x7f349c0cfaf0 2023-10-16T12:49:44.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.480+0000 7f3499d9b640 1 -- 172.21.15.60:0/1381263360 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34840318e0 con 0x7f349c0cfaf0 2023-10-16T12:49:44.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.480+0000 7f3499d9b640 1 -- 172.21.15.60:0/1381263360 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484031c70 con 0x7f349c0cfaf0 2023-10-16T12:49:44.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 -- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 msgr2=0x7f349c0b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0b3250 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f34840097a0 tx=0x7f348402b820 comp rx=0 tx=0).stop 2023-10-16T12:49:44.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 -- 172.21.15.60:0/1381263360 shutdown_connections 2023-10-16T12:49:44.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0b3250 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/1381263360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 -- 172.21.15.60:0/1381263360 >> 172.21.15.60:0/1381263360 conn(0x7f349c0d2cc0 msgr2=0x7f349c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:44.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 -- 172.21.15.60:0/1381263360 shutdown_connections 2023-10-16T12:49:44.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.481+0000 7f34a0f6c640 1 -- 172.21.15.60:0/1381263360 wait complete. 2023-10-16T12:49:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.483+0000 7f34a0f6c640 1 Processor -- start 2023-10-16T12:49:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.483+0000 7f34a0f6c640 1 -- start start 2023-10-16T12:49:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f34a0f6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 0x7f349c156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f34a0f6c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f34a0f6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f34a0f6c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1583b0 con 0x7f349c0b38e0 2023-10-16T12:49:44.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f34a0f6c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c156740 con 0x7f349c0cfaf0 2023-10-16T12:49:44.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f34a0f6c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0b1830 con 0x7f349c0cf710 2023-10-16T12:49:44.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f349a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 0x7f349c156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f349a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 0x7f349c156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60632/0 (socket says 172.21.15.60:60632) 2023-10-16T12:49:44.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f349a59c640 1 -- 172.21.15.60:0/3463764323 learned_addr learned my addr 172.21.15.60:0/3463764323 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:44.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f3493fff640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.485+0000 7f349ad9d640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f349a59c640 1 -- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 msgr2=0x7f349c0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f349a59c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f349a59c640 1 -- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 msgr2=0x7f349c0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f349a59c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f349a59c640 1 -- 172.21.15.60:0/3463764323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c009460 con 0x7f349c0b38e0 2023-10-16T12:49:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f349a59c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 0x7f349c156200 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f348c00e9b0 tx=0x7f348c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c00c9a0 con 0x7f349c0b38e0 2023-10-16T12:49:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348c00cb00 con 0x7f349c0b38e0 2023-10-16T12:49:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c00ce70 con 0x7f349c0b38e0 2023-10-16T12:49:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3484009400 con 0x7f349c0b38e0 2023-10-16T12:49:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.486+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c0b1db0 con 0x7f349c0b38e0 2023-10-16T12:49:44.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.487+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c04e160 con 0x7f349c0b38e0 2023-10-16T12:49:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.488+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f348c005990 con 0x7f349c0b38e0 2023-10-16T12:49:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.489+0000 7f3492ffd640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 0x7f347c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.489+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f348c089c60 con 0x7f349c0b38e0 2023-10-16T12:49:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.490+0000 7f3493fff640 1 -- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 msgr2=0x7f347c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:44.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.490+0000 7f3493fff640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 0x7f347c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:44.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.494+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f348c054b30 con 0x7f349c0b38e0 2023-10-16T12:49:44.501 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.499+0000 7fa55b64f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:49:44.525 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.523+0000 7f2ac1cb6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:44.550 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.549+0000 7fa55b64f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:49:44.578 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.576+0000 7f2ac1cb6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:49:44.638 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.636+0000 7f2ac1cb6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:49:44.691 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:44.688+0000 7f2ac1cb6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:49:44.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.691+0000 7f3493fff640 1 -- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 msgr2=0x7f347c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:44.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.691+0000 7f3493fff640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 0x7f347c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:44.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.697+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f349c0d2580 con 0x7f349c0b38e0 2023-10-16T12:49:44.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.698+0000 7f3492ffd640 1 -- 172.21.15.60:0/3463764323 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f348c0200a0 con 0x7f349c0b38e0 2023-10-16T12:49:44.700 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:49:44.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.700+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 msgr2=0x7f347c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:44.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.700+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 0x7f347c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.700+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 msgr2=0x7f349c156200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.700+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 0x7f349c156200 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f348c00e9b0 tx=0x7f348c00ee70 comp rx=0 tx=0).stop 2023-10-16T12:49:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 shutdown_connections 2023-10-16T12:49:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f347c06af90 0x7f347c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f349c0cf710 0x7f349c0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f349c0cfaf0 0x7f349c0cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 --2- 172.21.15.60:0/3463764323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f349c0b38e0 0x7f349c156200 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 >> 172.21.15.60:0/3463764323 conn(0x7f349c0d2cc0 msgr2=0x7f349c0b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:44.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 shutdown_connections 2023-10-16T12:49:44.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.701+0000 7f34a0f6c640 1 -- 172.21.15.60:0/3463764323 wait complete. 2023-10-16T12:49:44.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:49:44.928 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:44.926+0000 7fa55b64f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:49:44.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.982+0000 7f6ad0f41640 1 Processor -- start 2023-10-16T12:49:44.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.982+0000 7f6ad0f41640 1 -- start start 2023-10-16T12:49:44.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.983+0000 7f6ad0f41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.983+0000 7f6ad0f41640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6ad0f41640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc0cb030 con 0x7f6acc15ad70 2023-10-16T12:49:44.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6ad0f41640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc0cae30 con 0x7f6acc0c9ca0 2023-10-16T12:49:44.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6ad0f41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c91c0 con 0x7f6acc06a4b0 2023-10-16T12:49:44.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6aca59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6acc15ad70 0x7f6acc069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38702/0 (socket says 172.21.15.60:38702) 2023-10-16T12:49:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6aca59c640 1 -- 172.21.15.60:0/3138323464 learned_addr learned my addr 172.21.15.60:0/3138323464 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6acad9d640 1 --2- 172.21.15.60:0/3138323464 >> v2:172.21.15.144:3300/0 conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6ac3fff640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.984+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188347511 0 0) 0x7f6acc0cb030 con 0x7f6acc15ad70 2023-10-16T12:49:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab8003520 con 0x7f6acc15ad70 2023-10-16T12:49:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833405075 0 0) 0x7f6ab8003520 con 0x7f6acc15ad70 2023-10-16T12:49:44.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 msgr2=0x7f6acc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:44.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 0x7f6acc0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 >> v2:172.21.15.144:3300/0 conn(0x7f6acc0c9ca0 msgr2=0x7f6acc0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:44.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 --2- 172.21.15.60:0/3138323464 >> v2:172.21.15.144:3300/0 conn(0x7f6acc0c9ca0 0x7f6acc0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0c94a0 con 0x7f6acc15ad70 2023-10-16T12:49:44.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f6abc001670 con 0x7f6acc15ad70 2023-10-16T12:49:44.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 >> v1:172.21.15.60:6789/0 conn(0x7f6acc15ad70 legacy=0x7f6acc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8003970 0x7f6ab8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 0x7f6ab80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8004d00 0x7f6ab8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0cae30 con 0x7f6ab8003970 2023-10-16T12:49:44.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c91c0 con 0x7f6ab8004d00 2023-10-16T12:49:44.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab800b980 con 0x7f6acc06a4b0 2023-10-16T12:49:44.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.985+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8003970 0x7f6ab8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6ac3fff640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 0x7f6ab80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6acad9d640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8004d00 0x7f6ab8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6aca59c640 1 -- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 msgr2=0x7f6ab80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc06a4b0 0x7f6ab80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6aca59c640 1 -- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8004d00 msgr2=0x7f6ab8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8004d00 0x7f6ab8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.986+0000 7f6aca59c640 1 -- 172.21.15.60:0/3138323464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8003520 con 0x7f6ab8003970 2023-10-16T12:49:44.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.987+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8003970 0x7f6ab8003d90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c210 tx=0x7f6ab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59b8a5e96c7224a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:44.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.988+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc005ad0 con 0x7f6ab8003970 2023-10-16T12:49:44.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.988+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc005c30 con 0x7f6ab8003970 2023-10-16T12:49:44.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.988+0000 7f6ac9d9b640 1 -- 172.21.15.60:0/3138323464 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc02c8d0 con 0x7f6ab8003970 2023-10-16T12:49:44.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3138323464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8003970 msgr2=0x7f6ab8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:44.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3138323464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8003970 0x7f6ab8003d90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c210 tx=0x7f6ab80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:44.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6acad9d640 1 -- 172.21.15.60:0/3138323464 reap_dead start 2023-10-16T12:49:44.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3138323464 shutdown_connections 2023-10-16T12:49:44.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3138323464 >> 172.21.15.60:0/3138323464 conn(0x7f6acc0ce9b0 msgr2=0x7f6acc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:44.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3138323464 shutdown_connections 2023-10-16T12:49:44.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.989+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3138323464 wait complete. 2023-10-16T12:49:44.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.991+0000 7f6ad0f41640 1 Processor -- start 2023-10-16T12:49:44.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.991+0000 7f6ad0f41640 1 -- start start 2023-10-16T12:49:44.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6ad0f41640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8003970 0x7f6acc14f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6ad0f41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc0aecf0 0x7f6acc0b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6ad0f41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 0x7f6acc0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:44.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6ad0f41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b2cf0 con 0x7f6acc0b2870 2023-10-16T12:49:44.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6ad0f41640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c8310 con 0x7f6ab8003970 2023-10-16T12:49:44.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6ad0f41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c7850 con 0x7f6acc0aecf0 2023-10-16T12:49:44.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6acad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 0x7f6acc0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6acad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 0x7f6acc0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60656/0 (socket says 172.21.15.60:60656) 2023-10-16T12:49:45.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.992+0000 7f6acad9d640 1 -- 172.21.15.60:0/3938212139 learned_addr learned my addr 172.21.15.60:0/3938212139 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:45.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8003970 0x7f6acc14f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6ac3fff640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc0aecf0 0x7f6acc0b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6acad9d640 1 -- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc0aecf0 msgr2=0x7f6acc0b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6acad9d640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc0aecf0 0x7f6acc0b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6acad9d640 1 -- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8003970 msgr2=0x7f6acc14f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6acad9d640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8003970 0x7f6acc14f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6acad9d640 1 -- 172.21.15.60:0/3938212139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0ac930 con 0x7f6acc0b2870 2023-10-16T12:49:45.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8003970 0x7f6acc14f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:45.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6acad9d640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 0x7f6acc0c6c00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6acc06d860 tx=0x7f6ac400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:45.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400e040 con 0x7f6acc0b2870 2023-10-16T12:49:45.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac400f040 con 0x7f6acc0b2870 2023-10-16T12:49:45.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4015430 con 0x7f6acc0b2870 2023-10-16T12:49:45.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.993+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc0acc10 con 0x7f6acc0b2870 2023-10-16T12:49:45.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.994+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc0af170 con 0x7f6acc0b2870 2023-10-16T12:49:45.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.995+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc04e160 con 0x7f6acc0b2870 2023-10-16T12:49:45.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.996+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6ac4015590 con 0x7f6acc0b2870 2023-10-16T12:49:45.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.997+0000 7f6ac2ffd640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.997+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6ac408a190 con 0x7f6acc0b2870 2023-10-16T12:49:45.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.997+0000 7f6aca59c640 1 -- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 msgr2=0x7f6aac06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:45.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:44.997+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:45.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.002+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ac4054e20 con 0x7f6acc0b2870 2023-10-16T12:49:45.089 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.087+0000 7fa55b64f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:49:45.103 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.100+0000 7f2ac1cb6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:49:45.195 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.194+0000 7fa55b64f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:49:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.196+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6acc0b2ff0 con 0x7f6acc0b2870 2023-10-16T12:49:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.197+0000 7f6ac2ffd640 1 -- 172.21.15.60:0/3938212139 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ac4058440 con 0x7f6acc0b2870 2023-10-16T12:49:45.200 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:49:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.198+0000 7f6aca59c640 1 -- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 msgr2=0x7f6aac06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.198+0000 7f6aca59c640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:45.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 msgr2=0x7f6aac06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 msgr2=0x7f6acc0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 0x7f6acc0c6c00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6acc06d860 tx=0x7f6ac400f580 comp rx=0 tx=0).stop 2023-10-16T12:49:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 shutdown_connections 2023-10-16T12:49:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6acc0aecf0 0x7f6acc0b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6acc0b2870 0x7f6acc0c6c00 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 --2- 172.21.15.60:0/3938212139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab8003970 0x7f6acc14f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 >> 172.21.15.60:0/3938212139 conn(0x7f6acc0ce9b0 msgr2=0x7f6acc15dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 shutdown_connections 2023-10-16T12:49:45.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.200+0000 7f6ad0f41640 1 -- 172.21.15.60:0/3938212139 wait complete. 2023-10-16T12:49:45.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:49:45.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.243+0000 7fa55b64f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:49:45.274 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.272+0000 7f2ac1cb6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:49:45.350 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.348+0000 7fa55b64f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:45.389 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.387+0000 7f2ac1cb6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:49:45.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.427+0000 7f0cba6ca640 1 Processor -- start 2023-10-16T12:49:45.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.427+0000 7f0cba6ca640 1 -- start start 2023-10-16T12:49:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cba6ca640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb415cd70 0x7f0cb40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cba6ca640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0cb40b8480 0x7f0cb40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cba6ca640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb415d150 con 0x7f0cb40d0be0 2023-10-16T12:49:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cba6ca640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb40d1180 con 0x7f0cb40b8480 2023-10-16T12:49:45.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cba6ca640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb40d1460 con 0x7f0cb415cd70 2023-10-16T12:49:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cb9ec9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0cb40d0be0 0x7f0cb40b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38716/0 (socket says 172.21.15.60:38716) 2023-10-16T12:49:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cb9ec9640 1 -- 172.21.15.60:0/1348718748 learned_addr learned my addr 172.21.15.60:0/1348718748 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.429+0000 7f0cb8ec7640 1 --2- 172.21.15.60:0/1348718748 >> v2:172.21.15.144:3300/0 conn(0x7f0cb40b8480 0x7f0cb40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cb96c8640 1 --2- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb415cd70 0x7f0cb40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cabfff640 1 -- 172.21.15.60:0/1348718748 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330195216 0 0) 0x7f0cb415d150 con 0x7f0cb40d0be0 2023-10-16T12:49:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cabfff640 1 -- 172.21.15.60:0/1348718748 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca0003520 con 0x7f0cb40d0be0 2023-10-16T12:49:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cb8ec7640 1 -- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb415cd70 msgr2=0x7f0cb40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cb8ec7640 1 --2- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb415cd70 0x7f0cb40d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cb8ec7640 1 -- 172.21.15.60:0/1348718748 >> v1:172.21.15.60:6789/0 conn(0x7f0cb40d0be0 legacy=0x7f0cb40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cabfff640 1 -- 172.21.15.60:0/1348718748 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 798050876 0 0) 0x7f0ca0003520 con 0x7f0cb40d0be0 2023-10-16T12:49:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.430+0000 7f0cb8ec7640 1 -- 172.21.15.60:0/1348718748 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb40d1740 con 0x7f0cb40b8480 2023-10-16T12:49:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.431+0000 7f0cb8ec7640 1 --2- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40b8480 0x7f0cb40cd7c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40097a0 tx=0x7f0ca402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=329e1cd30a750536 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.432+0000 7f0cabfff640 1 -- 172.21.15.60:0/1348718748 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0019f0 con 0x7f0cb40b8480 2023-10-16T12:49:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.432+0000 7f0cabfff640 1 -- 172.21.15.60:0/1348718748 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca402e050 con 0x7f0cb40b8480 2023-10-16T12:49:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.432+0000 7f0cabfff640 1 -- 172.21.15.60:0/1348718748 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4030710 con 0x7f0cb40b8480 2023-10-16T12:49:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40b8480 msgr2=0x7f0cb40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40b8480 0x7f0cb40cd7c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40097a0 tx=0x7f0ca402b790 comp rx=0 tx=0).stop 2023-10-16T12:49:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1348718748 shutdown_connections 2023-10-16T12:49:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40b8480 0x7f0cb40cd7c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1348718748 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb415cd70 0x7f0cb40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1348718748 >> 172.21.15.60:0/1348718748 conn(0x7f0cb40cf570 msgr2=0x7f0cb40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1348718748 shutdown_connections 2023-10-16T12:49:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.433+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1348718748 wait complete. 2023-10-16T12:49:45.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.435+0000 7f0cba6ca640 1 Processor -- start 2023-10-16T12:49:45.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.436+0000 7f0cba6ca640 1 -- start start 2023-10-16T12:49:45.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.437+0000 7f0cba6ca640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb40b8480 0x7f0cb41f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.437+0000 7f0cba6ca640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40d0be0 0x7f0cb41551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.437+0000 7f0cba6ca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 0x7f0cb41527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.437+0000 7f0cba6ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb40c9350 con 0x7f0cb415cd70 2023-10-16T12:49:45.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.437+0000 7f0cba6ca640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4155b30 con 0x7f0cb40d0be0 2023-10-16T12:49:45.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.437+0000 7f0cba6ca640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb415e8e0 con 0x7f0cb40b8480 2023-10-16T12:49:45.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 0x7f0cb41527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 0x7f0cb41527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60660/0 (socket says 172.21.15.60:60660) 2023-10-16T12:49:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 -- 172.21.15.60:0/1190881292 learned_addr learned my addr 172.21.15.60:0/1190881292 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb96c8640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb40b8480 0x7f0cb41f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb8ec7640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40d0be0 0x7f0cb41551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 -- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb40b8480 msgr2=0x7f0cb41f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb40b8480 0x7f0cb41f50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 -- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40d0be0 msgr2=0x7f0cb41551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40d0be0 0x7f0cb41551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 -- 172.21.15.60:0/1190881292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4009400 con 0x7f0cb415cd70 2023-10-16T12:49:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.438+0000 7f0cb9ec9640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 0x7f0cb41527c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00cb90 tx=0x7f0cac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.439+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00ad40 con 0x7f0cb415cd70 2023-10-16T12:49:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.439+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac012400 con 0x7f0cb415cd70 2023-10-16T12:49:45.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.439+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00aea0 con 0x7f0cb415cd70 2023-10-16T12:49:45.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.439+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb415dfb0 con 0x7f0cb415cd70 2023-10-16T12:49:45.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.439+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb415e570 con 0x7f0cb415cd70 2023-10-16T12:49:45.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.440+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb404e9b0 con 0x7f0cb415cd70 2023-10-16T12:49:45.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.441+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0cac0127b0 con 0x7f0cb415cd70 2023-10-16T12:49:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.442+0000 7f0ca9ffb640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 0x7f0c9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:45.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.443+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0cac089fe0 con 0x7f0cb415cd70 2023-10-16T12:49:45.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.443+0000 7f0cb96c8640 1 -- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 msgr2=0x7f0c9806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:45.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.443+0000 7f0cb96c8640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 0x7f0c9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:45.447 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.444+0000 7f2ac1cb6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:49:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.447+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0cac054dc0 con 0x7f0cb415cd70 2023-10-16T12:49:45.457 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.455+0000 7fa55b64f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:49:45.561 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.559+0000 7f2ac1cb6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:45.611 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.609+0000 7fa55b64f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.644+0000 7f0cb96c8640 1 -- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 msgr2=0x7f0c9806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.644+0000 7f0cb96c8640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 0x7f0c9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:45.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.656+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0cb415ea20 con 0x7f0cb415cd70 2023-10-16T12:49:45.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.657+0000 7f0ca9ffb640 1 -- 172.21.15.60:0/1190881292 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v87) v1 ==== 80+0+476 (secure 0 0 0) 0x7f0cac0583e0 con 0x7f0cb415cd70 2023-10-16T12:49:45.659 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/954403568 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.660 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2870481770 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.660 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3154142782 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.660 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2412969789 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.660 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2725919540 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.660 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/342775588 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.661 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/2872949001 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.661 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/2872949001 2023-10-17T12:49:40.910583+0000 2023-10-16T12:49:45.661 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T12:49:45.661 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.658+0000 7fa55b64f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:45.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 msgr2=0x7f0c9806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 0x7f0c9806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 msgr2=0x7f0cb41527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 0x7f0cb41527c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00cb90 tx=0x7f0cac00a430 comp rx=0 tx=0).stop 2023-10-16T12:49:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 shutdown_connections 2023-10-16T12:49:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cb415cd70 0x7f0cb41527c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb40d0be0 0x7f0cb41551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f0c9806af90 0x7f0c9806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 --2- 172.21.15.60:0/1190881292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0cb40b8480 0x7f0cb41f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 >> 172.21.15.60:0/1190881292 conn(0x7f0cb40cf570 msgr2=0x7f0cb40b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 shutdown_connections 2023-10-16T12:49:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:45.660+0000 7f0cba6ca640 1 -- 172.21.15.60:0/1190881292 wait complete. 2023-10-16T12:49:45.676 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.674+0000 7f2ac1cb6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:49:45.681 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:49:45.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:45.681 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:49:45.770 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:49:45.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:49:45.771 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:49:45.779 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:45.777+0000 7fa55b64f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:49:45.821 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:45.822 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:45.850 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.848+0000 7f2ac1cb6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:49:45.872 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:45.873 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:45.873 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:45.903 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:45.901+0000 7f2ac1cb6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:49:45.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:49:45.928 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:45.928 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:45.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:45.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:46.033 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:46.031+0000 7f2ac1cb6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:49:46.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.198+0000 7f5933fff640 1 Processor -- start 2023-10-16T12:49:46.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.199+0000 7f5933fff640 1 -- start start 2023-10-16T12:49:46.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.200+0000 7f5933fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f593406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.200+0000 7f5933fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f59340ad970 0x7f593415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.200+0000 7f5933fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593406c960 con 0x7f593415ebc0 2023-10-16T12:49:46.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.200+0000 7f5933fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593415b9d0 con 0x7f59340ad970 2023-10-16T12:49:46.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.200+0000 7f5933fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593415bcb0 con 0x7f593406c040 2023-10-16T12:49:46.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59337fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f593415ebc0 0x7f59340cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38732/0 (socket says 172.21.15.60:38732) 2023-10-16T12:49:46.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59337fe640 1 -- 172.21.15.60:0/2994142992 learned_addr learned my addr 172.21.15.60:0/2994142992 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:46.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f592bfff640 1 --2- 172.21.15.60:0/2994142992 >> v2:172.21.15.144:3300/0 conn(0x7f59340ad970 0x7f593415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f5932ffd640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f593406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106479258 0 0) 0x7f593406c960 con 0x7f593415ebc0 2023-10-16T12:49:46.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5920003520 con 0x7f593415ebc0 2023-10-16T12:49:46.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1938509795 0 0) 0x7f5920003520 con 0x7f593415ebc0 2023-10-16T12:49:46.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 msgr2=0x7f593406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:46.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f593406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 >> v2:172.21.15.144:3300/0 conn(0x7f59340ad970 msgr2=0x7f593415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:46.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 --2- 172.21.15.60:0/2994142992 >> v2:172.21.15.144:3300/0 conn(0x7f59340ad970 0x7f593415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593415bf90 con 0x7f593415ebc0 2023-10-16T12:49:46.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f592c0019f0 con 0x7f593415ebc0 2023-10-16T12:49:46.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.201+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 >> v1:172.21.15.60:6789/0 conn(0x7f593415ebc0 legacy=0x7f59340cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59327fc640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5920003970 0x7f5920003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59327fc640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f59200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59327fc640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5920004d00 0x7f5920005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593415b9d0 con 0x7f5920003970 2023-10-16T12:49:46.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593415bcb0 con 0x7f5920004d00 2023-10-16T12:49:46.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592000b980 con 0x7f593406c040 2023-10-16T12:49:46.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f5932ffd640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5920003970 0x7f5920003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59337fe640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5920004d00 0x7f5920005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f592bfff640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f59200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f5932ffd640 1 -- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 msgr2=0x7f59200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f5932ffd640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f59200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f5932ffd640 1 -- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5920004d00 msgr2=0x7f5920005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f5932ffd640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5920004d00 0x7f5920005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f5932ffd640 1 -- 172.21.15.60:0/2994142992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5920003520 con 0x7f5920003970 2023-10-16T12:49:46.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f59337fe640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5920004d00 0x7f5920005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:46.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.202+0000 7f592bfff640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f593406c040 0x7f59200047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:46.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.203+0000 7f5932ffd640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5920003970 0x7f5920003d90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f592400ba60 tx=0x7f59200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31ceab4f279f0c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:46.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924032630 con 0x7f5920003970 2023-10-16T12:49:46.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5924032790 con 0x7f5920003970 2023-10-16T12:49:46.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f59327fc640 1 -- 172.21.15.60:0/2994142992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924032b20 con 0x7f5920003970 2023-10-16T12:49:46.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f5933fff640 1 -- 172.21.15.60:0/2994142992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5920003970 msgr2=0x7f5920003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f5933fff640 1 --2- 172.21.15.60:0/2994142992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5920003970 0x7f5920003d90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f592400ba60 tx=0x7f59200057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f59337fe640 1 -- 172.21.15.60:0/2994142992 reap_dead start 2023-10-16T12:49:46.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f5933fff640 1 -- 172.21.15.60:0/2994142992 shutdown_connections 2023-10-16T12:49:46.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f5933fff640 1 -- 172.21.15.60:0/2994142992 >> 172.21.15.60:0/2994142992 conn(0x7f59340ce6a0 msgr2=0x7f59340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:46.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.204+0000 7f5933fff640 1 -- 172.21.15.60:0/2994142992 shutdown_connections 2023-10-16T12:49:46.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.205+0000 7f5933fff640 1 -- 172.21.15.60:0/2994142992 wait complete. 2023-10-16T12:49:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.206+0000 7f5933fff640 1 Processor -- start 2023-10-16T12:49:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.206+0000 7f5933fff640 1 -- start start 2023-10-16T12:49:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.207+0000 7f5933fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5920003970 0x7f593415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.207+0000 7f5933fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 0x7f59340ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.207+0000 7f5933fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59340ac890 0x7f59340acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.207+0000 7f5933fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59340af790 con 0x7f59340ae0c0 2023-10-16T12:49:46.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.207+0000 7f5933fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59340ad250 con 0x7f59340ac890 2023-10-16T12:49:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.207+0000 7f5933fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59340aeb80 con 0x7f5920003970 2023-10-16T12:49:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 0x7f59340ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 0x7f59340ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60674/0 (socket says 172.21.15.60:60674) 2023-10-16T12:49:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 -- 172.21.15.60:0/168525157 learned_addr learned my addr 172.21.15.60:0/168525157 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f59337fe640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59340ac890 0x7f59340acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 -- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5920003970 msgr2=0x7f593415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f5932ffd640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5920003970 0x7f593415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5920003970 0x7f593415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 -- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59340ac890 msgr2=0x7f59340acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59340ac890 0x7f59340acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 -- 172.21.15.60:0/168525157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59340aee60 con 0x7f59340ae0c0 2023-10-16T12:49:46.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f59337fe640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59340ac890 0x7f59340acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:46.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.208+0000 7f592bfff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 0x7f59340ae540 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f591c0109d0 tx=0x7f591c010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:46.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.209+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c00ec20 con 0x7f59340ae0c0 2023-10-16T12:49:46.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.209+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c00ed80 con 0x7f59340ae0c0 2023-10-16T12:49:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.209+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59340af140 con 0x7f59340ae0c0 2023-10-16T12:49:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.209+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c01c620 con 0x7f59340ae0c0 2023-10-16T12:49:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.209+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593415b120 con 0x7f59340ae0c0 2023-10-16T12:49:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.210+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593406b330 con 0x7f59340ae0c0 2023-10-16T12:49:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.211+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f591c01c780 con 0x7f59340ae0c0 2023-10-16T12:49:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.212+0000 7f592b7fe640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 0x7f591406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.212+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f591c089e90 con 0x7f59340ae0c0 2023-10-16T12:49:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.212+0000 7f5932ffd640 1 -- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 msgr2=0x7f591406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.212+0000 7f5932ffd640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 0x7f591406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.216+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f591c054d60 con 0x7f59340ae0c0 2023-10-16T12:49:46.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.413+0000 7f5932ffd640 1 -- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 msgr2=0x7f591406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:46.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.413+0000 7f5932ffd640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 0x7f591406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:46.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.449+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f593404e9b0 con 0x7f59340ae0c0 2023-10-16T12:49:46.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.450+0000 7f592b7fe640 1 -- 172.21.15.60:0/168525157 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f591c058380 con 0x7f59340ae0c0 2023-10-16T12:49:46.452 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:46.453 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:49:26.202053+0000","modified":"2023-10-16T12:49:29.284142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2404598903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2404598903},{"type":"v1","addr":"172.21.15.60:6835","nonce":2404598903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:49:46.453 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2023-10-16T12:49:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 msgr2=0x7f591406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 0x7f591406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 msgr2=0x7f59340ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 0x7f59340ae540 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f591c0109d0 tx=0x7f591c010e90 comp rx=0 tx=0).stop 2023-10-16T12:49:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 shutdown_connections 2023-10-16T12:49:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59340ac890 0x7f59340acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59340ae0c0 0x7f59340ae540 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f591406afe0 0x7f591406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 --2- 172.21.15.60:0/168525157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5920003970 0x7f593415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.452+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 >> 172.21.15.60:0/168525157 conn(0x7f59340ce6a0 msgr2=0x7f59340c98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:46.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.453+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 shutdown_connections 2023-10-16T12:49:46.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.453+0000 7f5933fff640 1 -- 172.21.15.60:0/168525157 wait complete. 2023-10-16T12:49:46.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:46.614 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:46.612+0000 7fa55b64f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:49:46.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.705+0000 7f8c17fff640 1 Processor -- start 2023-10-16T12:49:46.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.706+0000 7f8c17fff640 1 -- start start 2023-10-16T12:49:46.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c17fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8c1815cbe0 0x7f8c180d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c17fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c1815cfc0 0x7f8c180b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c17fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c180d15a0 con 0x7f8c180d1090 2023-10-16T12:49:46.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c17fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180d1880 con 0x7f8c1815cfc0 2023-10-16T12:49:46.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c17fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180d1b60 con 0x7f8c1815cbe0 2023-10-16T12:49:46.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c0ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c1815cfc0 0x7f8c180b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c177fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8c180d1090 0x7f8c1806bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38744/0 (socket says 172.21.15.60:38744) 2023-10-16T12:49:46.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.707+0000 7f8c177fe640 1 -- 172.21.15.60:0/2417214429 learned_addr learned my addr 172.21.15.60:0/2417214429 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:46.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.708+0000 7f8c16ffd640 1 --2- 172.21.15.60:0/2417214429 >> v2:172.21.15.144:3300/0 conn(0x7f8c1815cbe0 0x7f8c180d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.708+0000 7f8c0ffff640 1 -- 172.21.15.60:0/2417214429 >> v2:172.21.15.144:3300/0 conn(0x7f8c1815cbe0 msgr2=0x7f8c180d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:46.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.708+0000 7f8c0ffff640 1 --2- 172.21.15.60:0/2417214429 >> v2:172.21.15.144:3300/0 conn(0x7f8c1815cbe0 0x7f8c180d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.708+0000 7f8c0ffff640 1 -- 172.21.15.60:0/2417214429 >> v1:172.21.15.60:6789/0 conn(0x7f8c180d1090 legacy=0x7f8c1806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.708+0000 7f8c0ffff640 1 -- 172.21.15.60:0/2417214429 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180d1e40 con 0x7f8c1815cfc0 2023-10-16T12:49:46.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.708+0000 7f8c16ffd640 1 --2- 172.21.15.60:0/2417214429 >> v2:172.21.15.144:3300/0 conn(0x7f8c1815cbe0 0x7f8c180d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:46.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.709+0000 7f8c0ffff640 1 --2- 172.21.15.60:0/2417214429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cfc0 0x7f8c180b4570 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8c180b53f0 tx=0x7f8c0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad9943799bfea92 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:46.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c167fc640 1 -- 172.21.15.60:0/2417214429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00030610 con 0x7f8c1815cfc0 2023-10-16T12:49:46.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c167fc640 1 -- 172.21.15.60:0/2417214429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c00030770 con 0x7f8c1815cfc0 2023-10-16T12:49:46.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c167fc640 1 -- 172.21.15.60:0/2417214429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00030b00 con 0x7f8c1815cfc0 2023-10-16T12:49:46.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 -- 172.21.15.60:0/2417214429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cfc0 msgr2=0x7f8c180b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 --2- 172.21.15.60:0/2417214429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cfc0 0x7f8c180b4570 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8c180b53f0 tx=0x7f8c0002b960 comp rx=0 tx=0).stop 2023-10-16T12:49:46.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 -- 172.21.15.60:0/2417214429 shutdown_connections 2023-10-16T12:49:46.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 --2- 172.21.15.60:0/2417214429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cfc0 0x7f8c180b4570 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 --2- 172.21.15.60:0/2417214429 >> v2:172.21.15.144:3300/0 conn(0x7f8c1815cbe0 0x7f8c180d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 -- 172.21.15.60:0/2417214429 >> 172.21.15.60:0/2417214429 conn(0x7f8c180cf570 msgr2=0x7f8c180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:46.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 -- 172.21.15.60:0/2417214429 shutdown_connections 2023-10-16T12:49:46.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.710+0000 7f8c17fff640 1 -- 172.21.15.60:0/2417214429 wait complete. 2023-10-16T12:49:46.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.712+0000 7f8c17fff640 1 Processor -- start 2023-10-16T12:49:46.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.712+0000 7f8c17fff640 1 -- start start 2023-10-16T12:49:46.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c180d1090 0x7f8c18145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 0x7f8c1815a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c1815cfc0 0x7f8c1815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c17fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18146270 con 0x7f8c1815cbe0 2023-10-16T12:49:46.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c17fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180cba10 con 0x7f8c180d1090 2023-10-16T12:49:46.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c17fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c181492c0 con 0x7f8c1815cfc0 2023-10-16T12:49:46.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c0ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 0x7f8c1815a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c0ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 0x7f8c1815a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60694/0 (socket says 172.21.15.60:60694) 2023-10-16T12:49:46.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.713+0000 7f8c0ffff640 1 -- 172.21.15.60:0/1244800718 learned_addr learned my addr 172.21.15.60:0/1244800718 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:46.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c16ffd640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c180d1090 0x7f8c18145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c177fe640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c1815cfc0 0x7f8c1815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:46.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c0ffff640 1 -- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c1815cfc0 msgr2=0x7f8c1815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c0ffff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c1815cfc0 0x7f8c1815a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c0ffff640 1 -- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c180d1090 msgr2=0x7f8c18145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c0ffff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c180d1090 0x7f8c18145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c0ffff640 1 -- 172.21.15.60:0/1244800718 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00009400 con 0x7f8c1815cbe0 2023-10-16T12:49:46.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c16ffd640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c180d1090 0x7f8c18145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:46.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.714+0000 7f8c0ffff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 0x7f8c1815a300 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8c00000c00 tx=0x7f8c000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:46.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.715+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0003e070 con 0x7f8c1815cbe0 2023-10-16T12:49:46.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.715+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c00006e10 con 0x7f8c1815cbe0 2023-10-16T12:49:46.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.715+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c180cbc90 con 0x7f8c1815cbe0 2023-10-16T12:49:46.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.715+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00039c50 con 0x7f8c1815cbe0 2023-10-16T12:49:46.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.715+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c180cc250 con 0x7f8c1815cbe0 2023-10-16T12:49:46.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.716+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1804e9b0 con 0x7f8c1815cbe0 2023-10-16T12:49:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.717+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 32) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8c0004b400 con 0x7f8c1815cbe0 2023-10-16T12:49:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.718+0000 7f8c14ff9640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 0x7f8bf806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.718+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8c000abe60 con 0x7f8c1815cbe0 2023-10-16T12:49:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.718+0000 7f8c16ffd640 1 -- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 msgr2=0x7f8bf806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:46.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.718+0000 7f8c16ffd640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 0x7f8bf806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:49:46.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.722+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c00076c30 con 0x7f8c1815cbe0 2023-10-16T12:49:46.878 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:46.875+0000 7f2ac1cb6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:49:46.907 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:46.905+0000 7fa55b64f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:49:46.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.914+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c180cd7c0 con 0x7f8c1815cbe0 2023-10-16T12:49:46.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.917+0000 7f8c14ff9640 1 -- 172.21.15.60:0/1244800718 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+17060 (secure 0 0 0) 0x7f8c00034090 con 0x7f8c1815cbe0 2023-10-16T12:49:46.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:46.920 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:40.910632+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/954403568":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2870481770":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/3154142782":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2412969789":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2725919540":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/342775588":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2872949001":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6832/2872949001":"2023-10-17T12:49:40.910583+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:46.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.919+0000 7f8c16ffd640 1 -- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 msgr2=0x7f8bf806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2100131827 2023-10-16T12:49:46.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.919+0000 7f8c16ffd640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 0x7f8bf806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:49:46.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 msgr2=0x7f8bf806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:49:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 0x7f8bf806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 msgr2=0x7f8c1815a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 0x7f8c1815a300 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8c00000c00 tx=0x7f8c000046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 shutdown_connections 2023-10-16T12:49:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:6832/2100131827,v1:172.21.15.144:6833/2100131827] conn(0x7f8bf806af90 0x7f8bf806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8c1815cfc0 0x7f8c1815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c1815cbe0 0x7f8c1815a300 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 --2- 172.21.15.60:0/1244800718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c180d1090 0x7f8c18145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 >> 172.21.15.60:0/1244800718 conn(0x7f8c180cf570 msgr2=0x7f8c1815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 shutdown_connections 2023-10-16T12:49:46.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:46.920+0000 7f8c17fff640 1 -- 172.21.15.60:0/1244800718 wait complete. 2023-10-16T12:49:46.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:49:24.152341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:49:25.216482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:46.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:46.964 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:46.962+0000 7fa55b64f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:49:47.077 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.075+0000 7fa55b64f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:49:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.157+0000 7f3b180a2640 1 Processor -- start 2023-10-16T12:49:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.157+0000 7f3b180a2640 1 -- start start 2023-10-16T12:49:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.158+0000 7f3b180a2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b1015ad70 0x7f3b10069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.158+0000 7f3b180a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b100c9ca0 0x7f3b1006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.158+0000 7f3b180a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100cb030 con 0x7f3b100ca3b0 2023-10-16T12:49:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.158+0000 7f3b180a2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b100c91c0 con 0x7f3b1015ad70 2023-10-16T12:49:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.158+0000 7f3b180a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b100c94a0 con 0x7f3b100c9ca0 2023-10-16T12:49:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b15e3e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b1015ad70 0x7f3b10069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b1563d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b100c9ca0 0x7f3b1006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b1663f640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3b100ca3b0 0x7f3b100ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:44286/0 (socket says 172.21.15.60:44286) 2023-10-16T12:49:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b1663f640 1 -- 172.21.15.60:0/1004332587 learned_addr learned my addr 172.21.15.60:0/1004332587 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b15e3e640 1 -- 172.21.15.60:0/1004332587 >> v2:172.21.15.144:3300/0 conn(0x7f3b100c9ca0 msgr2=0x7f3b1006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b15e3e640 1 --2- 172.21.15.60:0/1004332587 >> v2:172.21.15.144:3300/0 conn(0x7f3b100c9ca0 0x7f3b1006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b15e3e640 1 -- 172.21.15.60:0/1004332587 >> v1:172.21.15.144:6789/0 conn(0x7f3b100ca3b0 legacy=0x7f3b100ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.159+0000 7f3b15e3e640 1 -- 172.21.15.60:0/1004332587 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100c9780 con 0x7f3b1015ad70 2023-10-16T12:49:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.160+0000 7f3b15e3e640 1 --2- 172.21.15.60:0/1004332587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b10069da0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3b100ce870 tx=0x7f3b0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d450e1a4c301006 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b14e3c640 1 -- 172.21.15.60:0/1004332587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c02e070 con 0x7f3b1015ad70 2023-10-16T12:49:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b14e3c640 1 -- 172.21.15.60:0/1004332587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0c0315a0 con 0x7f3b1015ad70 2023-10-16T12:49:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b14e3c640 1 -- 172.21.15.60:0/1004332587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c031930 con 0x7f3b1015ad70 2023-10-16T12:49:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 -- 172.21.15.60:0/1004332587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 msgr2=0x7f3b10069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 --2- 172.21.15.60:0/1004332587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b10069da0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3b100ce870 tx=0x7f3b0c02b840 comp rx=0 tx=0).stop 2023-10-16T12:49:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 -- 172.21.15.60:0/1004332587 shutdown_connections 2023-10-16T12:49:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 --2- 172.21.15.60:0/1004332587 >> v2:172.21.15.144:3300/0 conn(0x7f3b100c9ca0 0x7f3b1006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 --2- 172.21.15.60:0/1004332587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b10069da0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 -- 172.21.15.60:0/1004332587 >> 172.21.15.60:0/1004332587 conn(0x7f3b100ce9b0 msgr2=0x7f3b100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:47.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 -- 172.21.15.60:0/1004332587 shutdown_connections 2023-10-16T12:49:47.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.161+0000 7f3b180a2640 1 -- 172.21.15.60:0/1004332587 wait complete. 2023-10-16T12:49:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.163+0000 7f3b180a2640 1 Processor -- start 2023-10-16T12:49:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.163+0000 7f3b180a2640 1 -- start start 2023-10-16T12:49:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b180a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b100c9ca0 0x7f3b100c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b180a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b100ca3b0 0x7f3b1015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b180a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b101f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b180a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100c5e60 con 0x7f3b1015ad70 2023-10-16T12:49:47.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b180a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1015bd50 con 0x7f3b100ca3b0 2023-10-16T12:49:47.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b180a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1015c030 con 0x7f3b100c9ca0 2023-10-16T12:49:47.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b1663f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b101f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b1663f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b101f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60718/0 (socket says 172.21.15.60:60718) 2023-10-16T12:49:47.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b1663f640 1 -- 172.21.15.60:0/3081752895 learned_addr learned my addr 172.21.15.60:0/3081752895 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b1563d640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b100ca3b0 0x7f3b1015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.164+0000 7f3b15e3e640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b100c9ca0 0x7f3b100c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b1663f640 1 -- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b100c9ca0 msgr2=0x7f3b100c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b1663f640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b100c9ca0 0x7f3b100c48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b1663f640 1 -- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b100ca3b0 msgr2=0x7f3b1015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b1663f640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b100ca3b0 0x7f3b1015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b1663f640 1 -- 172.21.15.60:0/3081752895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c009400 con 0x7f3b1015ad70 2023-10-16T12:49:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b1663f640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b101f3520 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800caf0 tx=0x7f3b0800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0800aca0 con 0x7f3b1015ad70 2023-10-16T12:49:47.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0800ae20 con 0x7f3b1015ad70 2023-10-16T12:49:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1015c310 con 0x7f3b1015ad70 2023-10-16T12:49:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.165+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08012640 con 0x7f3b1015ad70 2023-10-16T12:49:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.166+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1015b990 con 0x7f3b1015ad70 2023-10-16T12:49:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.167+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1004e160 con 0x7f3b1015ad70 2023-10-16T12:49:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.168+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 33) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3b080127a0 con 0x7f3b1015ad70 2023-10-16T12:49:47.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.169+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3b080897c0 con 0x7f3b1015ad70 2023-10-16T12:49:47.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.173+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b08054ac0 con 0x7f3b1015ad70 2023-10-16T12:49:47.176 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.169+0000 7fa4e0a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.177 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.169+0000 7fa4e0a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.177 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.169+0000 7fa4e0a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.177 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.169+0000 7fa4e0a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.177 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.169+0000 7fa4e0a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.177 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.170+0000 7fa4dda1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.177 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.170+0000 7fa4dda1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.178 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.170+0000 7fa4dda1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.178 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.170+0000 7fa4dda1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.178 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:49:47.170+0000 7fa4dda1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:49:47.214 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:47.212+0000 7f2ac1cb6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:49:47.274 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:47.272+0000 7f2ac1cb6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:49:47.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.361+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b100c4e70 con 0x7f3b1015ad70 2023-10-16T12:49:47.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.362+0000 7f3b067fc640 1 -- 172.21.15.60:0/3081752895 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b08087e60 con 0x7f3b1015ad70 2023-10-16T12:49:47.365 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 msgr2=0x7f3b101f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b101f3520 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800caf0 tx=0x7f3b0800a060 comp rx=0 tx=0).stop 2023-10-16T12:49:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 shutdown_connections 2023-10-16T12:49:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b1015ad70 0x7f3b101f3520 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b100ca3b0 0x7f3b1015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 --2- 172.21.15.60:0/3081752895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b100c9ca0 0x7f3b100c48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 >> 172.21.15.60:0/3081752895 conn(0x7f3b100ce9b0 msgr2=0x7f3b100ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:47.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.365+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 shutdown_connections 2023-10-16T12:49:47.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.366+0000 7f3b180a2640 1 -- 172.21.15.60:0/3081752895 wait complete. 2023-10-16T12:49:47.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:49:47.395 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:49:47.392+0000 7f2ac1cb6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:49:47.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.595+0000 7f7b4db1d640 1 Processor -- start 2023-10-16T12:49:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.595+0000 7f7b4db1d640 1 -- start start 2023-10-16T12:49:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.596+0000 7f7b4db1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b480ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.596+0000 7f7b4db1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b4806c140 0x7f7b4806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b4db1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b48070360 con 0x7f7b48159ab0 2023-10-16T12:49:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b4db1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4806cb10 con 0x7f7b4806c140 2023-10-16T12:49:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b4db1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4815d890 con 0x7f7b48159e90 2023-10-16T12:49:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b477fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7b48159ab0 0x7f7b4806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38746/0 (socket says 172.21.15.60:38746) 2023-10-16T12:49:47.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b477fe640 1 -- 172.21.15.60:0/2850506964 learned_addr learned my addr 172.21.15.60:0/2850506964 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969662873 0 0) 0x7f7b48070360 con 0x7f7b48159ab0 2023-10-16T12:49:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b47fff640 1 --2- 172.21.15.60:0/2850506964 >> v2:172.21.15.144:3300/0 conn(0x7f7b4806c140 0x7f7b4806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b30003520 con 0x7f7b48159ab0 2023-10-16T12:49:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.597+0000 7f7b46ffd640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b480ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600999565 0 0) 0x7f7b30003520 con 0x7f7b48159ab0 2023-10-16T12:49:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 msgr2=0x7f7b480ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b480ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 >> v2:172.21.15.144:3300/0 conn(0x7f7b4806c140 msgr2=0x7f7b4806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 --2- 172.21.15.60:0/2850506964 >> v2:172.21.15.144:3300/0 conn(0x7f7b4806c140 0x7f7b4806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b480cd1f0 con 0x7f7b48159ab0 2023-10-16T12:49:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7b38001670 con 0x7f7b48159ab0 2023-10-16T12:49:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 >> v1:172.21.15.60:6789/0 conn(0x7f7b48159ab0 legacy=0x7f7b4806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b30003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b30003ef0 0x7f7b300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4806cb10 con 0x7f7b30004d00 2023-10-16T12:49:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4815d890 con 0x7f7b30003ef0 2023-10-16T12:49:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3000bb10 con 0x7f7b48159e90 2023-10-16T12:49:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b47fff640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b46ffd640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b30003ef0 0x7f7b300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.598+0000 7f7b477fe640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b30003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b47fff640 1 -- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 msgr2=0x7f7b30003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b47fff640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b30003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b47fff640 1 -- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b30003ef0 msgr2=0x7f7b300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b47fff640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b30003ef0 0x7f7b300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b47fff640 1 -- 172.21.15.60:0/2850506964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b30003520 con 0x7f7b30004d00 2023-10-16T12:49:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.599+0000 7f7b477fe640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48159e90 0x7f7b30003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.600+0000 7f7b47fff640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b30005180 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7b40010350 tx=0x7f7b300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b80e76769b29af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.600+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4002ee50 con 0x7f7b30004d00 2023-10-16T12:49:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.600+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b40011040 con 0x7f7b30004d00 2023-10-16T12:49:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.600+0000 7f7b467fc640 1 -- 172.21.15.60:0/2850506964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000b310 con 0x7f7b30004d00 2023-10-16T12:49:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b4db1d640 1 -- 172.21.15.60:0/2850506964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 msgr2=0x7f7b30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b4db1d640 1 --2- 172.21.15.60:0/2850506964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b30005180 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7b40010350 tx=0x7f7b300057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b47fff640 1 -- 172.21.15.60:0/2850506964 reap_dead start 2023-10-16T12:49:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b4db1d640 1 -- 172.21.15.60:0/2850506964 shutdown_connections 2023-10-16T12:49:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b4db1d640 1 -- 172.21.15.60:0/2850506964 >> 172.21.15.60:0/2850506964 conn(0x7f7b480ce9b0 msgr2=0x7f7b480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b4db1d640 1 -- 172.21.15.60:0/2850506964 shutdown_connections 2023-10-16T12:49:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.601+0000 7f7b4db1d640 1 -- 172.21.15.60:0/2850506964 wait complete. 2023-10-16T12:49:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.603+0000 7f7b4db1d640 1 Processor -- start 2023-10-16T12:49:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.603+0000 7f7b4db1d640 1 -- start start 2023-10-16T12:49:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b4db1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b481e63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b4db1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48158ff0 0x7f7b481f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b4db1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b480c7a90 0x7f7b480c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b4db1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b481e3950 con 0x7f7b30004d00 2023-10-16T12:49:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b4db1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b480c8450 con 0x7f7b480c7a90 2023-10-16T12:49:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b4db1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b481f6470 con 0x7f7b48158ff0 2023-10-16T12:49:47.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b481e63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b481e63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60730/0 (socket says 172.21.15.60:60730) 2023-10-16T12:49:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b477fe640 1 -- 172.21.15.60:0/4144610575 learned_addr learned my addr 172.21.15.60:0/4144610575 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.604+0000 7f7b46ffd640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48158ff0 0x7f7b481f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b47fff640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b480c7a90 0x7f7b480c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:47.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b477fe640 1 -- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48158ff0 msgr2=0x7f7b481f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b477fe640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48158ff0 0x7f7b481f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b477fe640 1 -- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b480c7a90 msgr2=0x7f7b480c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b477fe640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b480c7a90 0x7f7b480c7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b477fe640 1 -- 172.21.15.60:0/4144610575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b481e2cd0 con 0x7f7b30004d00 2023-10-16T12:49:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b47fff640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b480c7a90 0x7f7b480c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:47.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.605+0000 7f7b477fe640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b481e63f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7b38001eb0 tx=0x7f7b3800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.606+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380074e0 con 0x7f7b30004d00 2023-10-16T12:49:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.606+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b38007640 con 0x7f7b30004d00 2023-10-16T12:49:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.606+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380049e0 con 0x7f7b30004d00 2023-10-16T12:49:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.606+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b481e2fb0 con 0x7f7b30004d00 2023-10-16T12:49:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.606+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b481e3570 con 0x7f7b30004d00 2023-10-16T12:49:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.607+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4804e160 con 0x7f7b30004d00 2023-10-16T12:49:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.608+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 33) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7b38004b40 con 0x7f7b30004d00 2023-10-16T12:49:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.609+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7b38088e10 con 0x7f7b30004d00 2023-10-16T12:49:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.613+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b3808c030 con 0x7f7b30004d00 2023-10-16T12:49:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.808+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7b480ccb60 con 0x7f7b30004d00 2023-10-16T12:49:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.809+0000 7f7b2ffff640 1 -- 172.21.15.60:0/4144610575 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7b380190a0 con 0x7f7b30004d00 2023-10-16T12:49:47.811 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:49:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 msgr2=0x7f7b481e63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b481e63f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7b38001eb0 tx=0x7f7b3800dd40 comp rx=0 tx=0).stop 2023-10-16T12:49:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 shutdown_connections 2023-10-16T12:49:47.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b48158ff0 0x7f7b481f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b480c7a90 0x7f7b480c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 --2- 172.21.15.60:0/4144610575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b30004d00 0x7f7b481e63f0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 >> 172.21.15.60:0/4144610575 conn(0x7f7b480ce9b0 msgr2=0x7f7b4806b960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.811+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 shutdown_connections 2023-10-16T12:49:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:47.812+0000 7f7b4db1d640 1 -- 172.21.15.60:0/4144610575 wait complete. 2023-10-16T12:49:47.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:49:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.106+0000 7f211e7dc640 1 Processor -- start 2023-10-16T12:49:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.106+0000 7f211e7dc640 1 -- start start 2023-10-16T12:49:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.107+0000 7f211e7dc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f211815d150 0x7f21180ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f211e7dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 0x7f2118160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f211e7dc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2118069da0 0x7f211806a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f211e7dc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211815d530 con 0x7f211815d150 2023-10-16T12:49:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f211e7dc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211806a6f0 con 0x7f2118069da0 2023-10-16T12:49:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f211e7dc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21180ac890 con 0x7f2118160b90 2023-10-16T12:49:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f211815d150 0x7f21180ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f211815d150 0x7f21180ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60746/0 (socket says 172.21.15.60:60746) 2023-10-16T12:49:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 -- 172.21.15.60:0/452005099 learned_addr learned my addr 172.21.15.60:0/452005099 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:48.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 -- 172.21.15.60:0/452005099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 msgr2=0x7f2118160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:48.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 --2- 172.21.15.60:0/452005099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 0x7f2118160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 -- 172.21.15.60:0/452005099 >> v2:172.21.15.144:3300/0 conn(0x7f2118069da0 msgr2=0x7f211806a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:48.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 --2- 172.21.15.60:0/452005099 >> v2:172.21.15.144:3300/0 conn(0x7f2118069da0 0x7f211806a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.108+0000 7f2117fff640 1 -- 172.21.15.60:0/452005099 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2118161650 con 0x7f211815d150 2023-10-16T12:49:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.110+0000 7f2117fff640 1 --2- 172.21.15.60:0/452005099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f211815d150 0x7f21180ad350 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f21180cf990 tx=0x7f210802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164e636c8f899b65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.110+0000 7f2116ffd640 1 -- 172.21.15.60:0/452005099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210802ca80 con 0x7f211815d150 2023-10-16T12:49:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.110+0000 7f2116ffd640 1 -- 172.21.15.60:0/452005099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210802cbe0 con 0x7f211815d150 2023-10-16T12:49:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.110+0000 7f2116ffd640 1 -- 172.21.15.60:0/452005099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080343f0 con 0x7f211815d150 2023-10-16T12:49:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 -- 172.21.15.60:0/452005099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f211815d150 msgr2=0x7f21180ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 --2- 172.21.15.60:0/452005099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f211815d150 0x7f21180ad350 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f21180cf990 tx=0x7f210802bee0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 -- 172.21.15.60:0/452005099 shutdown_connections 2023-10-16T12:49:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 --2- 172.21.15.60:0/452005099 >> v2:172.21.15.144:3300/0 conn(0x7f2118069da0 0x7f211806a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 --2- 172.21.15.60:0/452005099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 0x7f2118160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 --2- 172.21.15.60:0/452005099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f211815d150 0x7f21180ad350 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 -- 172.21.15.60:0/452005099 >> 172.21.15.60:0/452005099 conn(0x7f21180ce9b0 msgr2=0x7f21180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 -- 172.21.15.60:0/452005099 shutdown_connections 2023-10-16T12:49:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.111+0000 7f211e7dc640 1 -- 172.21.15.60:0/452005099 wait complete. 2023-10-16T12:49:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.113+0000 7f211e7dc640 1 Processor -- start 2023-10-16T12:49:48.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.113+0000 7f211e7dc640 1 -- start start 2023-10-16T12:49:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.114+0000 7f211e7dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 0x7f2118155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.114+0000 7f211e7dc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f211815d150 0x7f21180c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.114+0000 7f211e7dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 0x7f21180c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.114+0000 7f211e7dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2118156390 con 0x7f2118069da0 2023-10-16T12:49:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.114+0000 7f211e7dc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21180c2e50 con 0x7f211815d150 2023-10-16T12:49:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.114+0000 7f211e7dc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21180c3130 con 0x7f2118160b90 2023-10-16T12:49:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 0x7f2118155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 0x7f2118155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60760/0 (socket says 172.21.15.60:60760) 2023-10-16T12:49:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 -- 172.21.15.60:0/4144774361 learned_addr learned my addr 172.21.15.60:0/4144774361 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 -- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 msgr2=0x7f21180c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 0x7f21180c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 -- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f211815d150 msgr2=0x7f21180c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f211815d150 0x7f21180c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 -- 172.21.15.60:0/4144774361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108009b30 con 0x7f2118069da0 2023-10-16T12:49:48.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.115+0000 7f2117fff640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 0x7f2118155e50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2108000f80 tx=0x7f2108004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.116+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210803e450 con 0x7f2118069da0 2023-10-16T12:49:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.116+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210803e5f0 con 0x7f2118069da0 2023-10-16T12:49:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.116+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210803e9a0 con 0x7f2118069da0 2023-10-16T12:49:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.116+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21180c3410 con 0x7f2118069da0 2023-10-16T12:49:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.116+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211815e170 con 0x7f2118069da0 2023-10-16T12:49:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.117+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211804e160 con 0x7f2118069da0 2023-10-16T12:49:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.118+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 34) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f210802cd50 con 0x7f2118069da0 2023-10-16T12:49:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.120+0000 7f2114ff9640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f20f806acf0 0x7f20f806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.120+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f21080ac080 con 0x7f2118069da0 2023-10-16T12:49:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.122+0000 7f21177fe640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f20f806acf0 0x7f20f806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.125+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21080defe0 con 0x7f2118069da0 2023-10-16T12:49:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.128+0000 7f21177fe640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f20f806acf0 0x7f20f806d1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f211815c130 tx=0x7f2100006c90 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:48.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.316+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f21180ade10 con 0x7f2118069da0 2023-10-16T12:49:48.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.316+0000 7f2114ff9640 1 -- 172.21.15.60:0/4144774361 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2108077210 con 0x7f2118069da0 2023-10-16T12:49:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:49:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f20f806acf0 msgr2=0x7f20f806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f20f806acf0 0x7f20f806d1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f211815c130 tx=0x7f2100006c90 comp rx=0 tx=0).stop 2023-10-16T12:49:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 msgr2=0x7f2118155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 0x7f2118155e50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2108000f80 tx=0x7f2108004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 shutdown_connections 2023-10-16T12:49:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f20f806acf0 0x7f20f806d1b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2118160b90 0x7f21180c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f211815d150 0x7f21180c24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 --2- 172.21.15.60:0/4144774361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2118069da0 0x7f2118155e50 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 >> 172.21.15.60:0/4144774361 conn(0x7f21180ce9b0 msgr2=0x7f211806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 shutdown_connections 2023-10-16T12:49:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.319+0000 7f211e7dc640 1 -- 172.21.15.60:0/4144774361 wait complete. 2023-10-16T12:49:48.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:48.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.549+0000 7f2e486bb640 1 Processor -- start 2023-10-16T12:49:48.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.549+0000 7f2e486bb640 1 -- start start 2023-10-16T12:49:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e486bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2e4015aaa0 0x7f2e4015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e486bb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e400ce390 0x7f2e40070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e486bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4015ed60 con 0x7f2e4015ae80 2023-10-16T12:49:48.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e486bb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4006b810 con 0x7f2e400ce390 2023-10-16T12:49:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e486bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e400707e0 con 0x7f2e4015aaa0 2023-10-16T12:49:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e46c58640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e400ce390 0x7f2e40070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e46c58640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e400ce390 0x7f2e40070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60768/0 (socket says 172.21.15.60:60768) 2023-10-16T12:49:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e46c58640 1 -- 172.21.15.60:0/1411491903 learned_addr learned my addr 172.21.15.60:0/1411491903 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:48.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.551+0000 7f2e45c56640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2e4015ae80 0x7f2e400cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38754/0 (socket says 172.21.15.60:38754) 2023-10-16T12:49:48.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.552+0000 7f2e46457640 1 --2- 172.21.15.60:0/1411491903 >> v2:172.21.15.144:3300/0 conn(0x7f2e4015aaa0 0x7f2e4015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.552+0000 7f2e46c58640 1 -- 172.21.15.60:0/1411491903 >> v2:172.21.15.144:3300/0 conn(0x7f2e4015aaa0 msgr2=0x7f2e4015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.552+0000 7f2e46c58640 1 --2- 172.21.15.60:0/1411491903 >> v2:172.21.15.144:3300/0 conn(0x7f2e4015aaa0 0x7f2e4015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.552+0000 7f2e46c58640 1 -- 172.21.15.60:0/1411491903 >> v1:172.21.15.60:6789/0 conn(0x7f2e4015ae80 legacy=0x7f2e400cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.552+0000 7f2e46c58640 1 -- 172.21.15.60:0/1411491903 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4015ef90 con 0x7f2e400ce390 2023-10-16T12:49:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.553+0000 7f2e46c58640 1 --2- 172.21.15.60:0/1411491903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e400ce390 0x7f2e40070f50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2e400cd990 tx=0x7f2e3802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba602271916cd5ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.553+0000 7f2e45455640 1 -- 172.21.15.60:0/1411491903 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3802cb10 con 0x7f2e400ce390 2023-10-16T12:49:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.553+0000 7f2e45455640 1 -- 172.21.15.60:0/1411491903 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3802cc70 con 0x7f2e400ce390 2023-10-16T12:49:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.553+0000 7f2e45455640 1 -- 172.21.15.60:0/1411491903 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e380343f0 con 0x7f2e400ce390 2023-10-16T12:49:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 -- 172.21.15.60:0/1411491903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e400ce390 msgr2=0x7f2e40070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1411491903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e400ce390 0x7f2e40070f50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2e400cd990 tx=0x7f2e3802bf70 comp rx=0 tx=0).stop 2023-10-16T12:49:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 -- 172.21.15.60:0/1411491903 shutdown_connections 2023-10-16T12:49:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1411491903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e400ce390 0x7f2e40070f50 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1411491903 >> v2:172.21.15.144:3300/0 conn(0x7f2e4015aaa0 0x7f2e4015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 -- 172.21.15.60:0/1411491903 >> 172.21.15.60:0/1411491903 conn(0x7f2e400cb410 msgr2=0x7f2e400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 -- 172.21.15.60:0/1411491903 shutdown_connections 2023-10-16T12:49:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.554+0000 7f2e486bb640 1 -- 172.21.15.60:0/1411491903 wait complete. 2023-10-16T12:49:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.556+0000 7f2e486bb640 1 Processor -- start 2023-10-16T12:49:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.556+0000 7f2e486bb640 1 -- start start 2023-10-16T12:49:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e486bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e400ce390 0x7f2e400c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e486bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e4015aaa0 0x7f2e401f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e486bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 0x7f2e400af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e486bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e400c3460 con 0x7f2e4015ae80 2023-10-16T12:49:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e486bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e400c2490 con 0x7f2e400ce390 2023-10-16T12:49:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e486bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e401f1810 con 0x7f2e4015aaa0 2023-10-16T12:49:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.557+0000 7f2e46c58640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 0x7f2e400af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 0x7f2e400af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60772/0 (socket says 172.21.15.60:60772) 2023-10-16T12:49:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 -- 172.21.15.60:0/1210152305 learned_addr learned my addr 172.21.15.60:0/1210152305 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46457640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e400ce390 0x7f2e400c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 -- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e4015aaa0 msgr2=0x7f2e401f12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e4015aaa0 0x7f2e401f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 -- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e400ce390 msgr2=0x7f2e400c2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e400ce390 0x7f2e400c2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 -- 172.21.15.60:0/1210152305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38009b30 con 0x7f2e4015ae80 2023-10-16T12:49:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46457640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e400ce390 0x7f2e400c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:48.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.558+0000 7f2e46c58640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 0x7f2e400af3b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2e38036040 tx=0x7f2e38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.559+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38009510 con 0x7f2e4015ae80 2023-10-16T12:49:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.559+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3803e450 con 0x7f2e4015ae80 2023-10-16T12:49:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.559+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3803e800 con 0x7f2e4015ae80 2023-10-16T12:49:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.559+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4015be40 con 0x7f2e4015ae80 2023-10-16T12:49:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.559+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4015c0b0 con 0x7f2e4015ae80 2023-10-16T12:49:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.560+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4015c2a0 con 0x7f2e4015ae80 2023-10-16T12:49:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.561+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 34) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2e3803e960 con 0x7f2e4015ae80 2023-10-16T12:49:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.562+0000 7f2e36ffd640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2e2406acc0 0x7f2e2406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.562+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2e380ac380 con 0x7f2e4015ae80 2023-10-16T12:49:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.563+0000 7f2e46457640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2e2406acc0 0x7f2e2406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:48.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.564+0000 7f2e46457640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2e2406acc0 0x7f2e2406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c006fd0 tx=0x7f2e3c008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.567+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e380773f0 con 0x7f2e4015ae80 2023-10-16T12:49:48.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.809+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e4004e160 con 0x7f2e4015ae80 2023-10-16T12:49:48.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.810+0000 7f2e36ffd640 1 -- 172.21.15.60:0/1210152305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f2e3807aa10 con 0x7f2e4015ae80 2023-10-16T12:49:48.813 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:48.813 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:49:26.202053+0000","modified":"2023-10-16T12:49:29.284142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2404598903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2404598903},{"type":"v1","addr":"172.21.15.60:6835","nonce":2404598903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:49:48.814 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2023-10-16T12:49:48.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2e2406acc0 msgr2=0x7f2e2406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2e2406acc0 0x7f2e2406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c006fd0 tx=0x7f2e3c008040 comp rx=0 tx=0).stop 2023-10-16T12:49:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 msgr2=0x7f2e400af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 0x7f2e400af3b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2e38036040 tx=0x7f2e38004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 shutdown_connections 2023-10-16T12:49:48.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2e2406acc0 0x7f2e2406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e4015aaa0 0x7f2e401f12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4015ae80 0x7f2e400af3b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 --2- 172.21.15.60:0/1210152305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e400ce390 0x7f2e400c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:48.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 >> 172.21.15.60:0/1210152305 conn(0x7f2e400cb410 msgr2=0x7f2e40070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:48.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 shutdown_connections 2023-10-16T12:49:48.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:48.813+0000 7f2e486bb640 1 -- 172.21.15.60:0/1210152305 wait complete. 2023-10-16T12:49:48.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:49.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.058+0000 7f5b26143640 1 Processor -- start 2023-10-16T12:49:49.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.059+0000 7f5b26143640 1 -- start start 2023-10-16T12:49:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.060+0000 7f5b26143640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b20160240 0x7f5b200caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.060+0000 7f5b26143640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b200b1ca0 0x7f5b200cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.060+0000 7f5b26143640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2006afe0 con 0x7f5b20160620 2023-10-16T12:49:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.060+0000 7f5b26143640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b200cf540 con 0x7f5b20160240 2023-10-16T12:49:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.060+0000 7f5b26143640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200cf820 con 0x7f5b200b1ca0 2023-10-16T12:49:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b20160240 0x7f5b200caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b20160240 0x7f5b200caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38110/0 (socket says 172.21.15.60:38110) 2023-10-16T12:49:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 -- 172.21.15.60:0/3433286585 learned_addr learned my addr 172.21.15.60:0/3433286585 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b24940640 1 --1- 172.21.15.60:0/3433286585 >> v1:172.21.15.60:6789/0 conn(0x7f5b20160620 0x7f5b200ceaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49958/0 (socket says 172.21.15.60:49958) 2023-10-16T12:49:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25942640 1 --2- 172.21.15.60:0/3433286585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b200b1ca0 0x7f5b200cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 -- 172.21.15.60:0/3433286585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b200b1ca0 msgr2=0x7f5b200cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 --2- 172.21.15.60:0/3433286585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b200b1ca0 0x7f5b200cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 -- 172.21.15.60:0/3433286585 >> v1:172.21.15.60:6789/0 conn(0x7f5b20160620 legacy=0x7f5b200ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b25141640 1 -- 172.21.15.60:0/3433286585 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200cfb00 con 0x7f5b20160240 2023-10-16T12:49:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.061+0000 7f5b1ffff640 1 -- 172.21.15.60:0/3433286585 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479945312 0 0) 0x7f5b2006afe0 con 0x7f5b20160620 2023-10-16T12:49:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.063+0000 7f5b25141640 1 --2- 172.21.15.60:0/3433286585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b20160240 0x7f5b200caf60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009e50 tx=0x7f5b1002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc22097c4cb54840 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:49.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.063+0000 7f5b1ffff640 1 -- 172.21.15.60:0/3433286585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1002ca70 con 0x7f5b20160240 2023-10-16T12:49:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.063+0000 7f5b1ffff640 1 -- 172.21.15.60:0/3433286585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1002cbd0 con 0x7f5b20160240 2023-10-16T12:49:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.063+0000 7f5b1ffff640 1 -- 172.21.15.60:0/3433286585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100343f0 con 0x7f5b20160240 2023-10-16T12:49:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.063+0000 7f5b26143640 1 -- 172.21.15.60:0/3433286585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b20160240 msgr2=0x7f5b200caf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.063+0000 7f5b26143640 1 --2- 172.21.15.60:0/3433286585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b20160240 0x7f5b200caf60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009e50 tx=0x7f5b1002bed0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.064+0000 7f5b26143640 1 -- 172.21.15.60:0/3433286585 shutdown_connections 2023-10-16T12:49:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.064+0000 7f5b26143640 1 --2- 172.21.15.60:0/3433286585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b200b1ca0 0x7f5b200cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.064+0000 7f5b26143640 1 --2- 172.21.15.60:0/3433286585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b20160240 0x7f5b200caf60 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.064+0000 7f5b26143640 1 -- 172.21.15.60:0/3433286585 >> 172.21.15.60:0/3433286585 conn(0x7f5b20099a10 msgr2=0x7f5b200d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.064+0000 7f5b26143640 1 -- 172.21.15.60:0/3433286585 shutdown_connections 2023-10-16T12:49:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.064+0000 7f5b26143640 1 -- 172.21.15.60:0/3433286585 wait complete. 2023-10-16T12:49:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.066+0000 7f5b26143640 1 Processor -- start 2023-10-16T12:49:49.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.066+0000 7f5b26143640 1 -- start start 2023-10-16T12:49:49.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b26143640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 0x7f5b2009de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b26143640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b20160240 0x7f5b2009ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b26143640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b20160620 0x7f5b2015e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b26143640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2009b280 con 0x7f5b200b1ca0 2023-10-16T12:49:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b26143640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200c4ef0 con 0x7f5b20160620 2023-10-16T12:49:49.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b26143640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200c5190 con 0x7f5b20160240 2023-10-16T12:49:49.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b25141640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 0x7f5b2009de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b25141640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 0x7f5b2009de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38118/0 (socket says 172.21.15.60:38118) 2023-10-16T12:49:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b25141640 1 -- 172.21.15.60:0/307524259 learned_addr learned my addr 172.21.15.60:0/307524259 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b25942640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b20160620 0x7f5b2015e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b25141640 1 -- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b20160240 msgr2=0x7f5b2009ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.067+0000 7f5b24940640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b20160240 0x7f5b2009ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b25141640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b20160240 0x7f5b2009ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b25141640 1 -- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b20160620 msgr2=0x7f5b2015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b25141640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b20160620 0x7f5b2015e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b25141640 1 -- 172.21.15.60:0/307524259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10009b30 con 0x7f5b200b1ca0 2023-10-16T12:49:49.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b25942640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b20160620 0x7f5b2015e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:49.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b24940640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b20160240 0x7f5b2009ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:49.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b25141640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 0x7f5b2009de50 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009e50 tx=0x7f5b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:49.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10009510 con 0x7f5b200b1ca0 2023-10-16T12:49:49.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1003e450 con 0x7f5b200b1ca0 2023-10-16T12:49:49.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1003e800 con 0x7f5b200b1ca0 2023-10-16T12:49:49.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.068+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b200c5470 con 0x7f5b200b1ca0 2023-10-16T12:49:49.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.069+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2009b4b0 con 0x7f5b200b1ca0 2023-10-16T12:49:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.070+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b200c5990 con 0x7f5b200b1ca0 2023-10-16T12:49:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.071+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 34) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5b1003e960 con 0x7f5b200b1ca0 2023-10-16T12:49:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.072+0000 7f5b1dffb640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b0c06acc0 0x7f5b0c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.072+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5b100ac400 con 0x7f5b200b1ca0 2023-10-16T12:49:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.072+0000 7f5b24940640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b0c06acc0 0x7f5b0c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.073+0000 7f5b24940640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b0c06acc0 0x7f5b0c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b200cbcd0 tx=0x7f5b14009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.076+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b200c5990 con 0x7f5b200b1ca0 2023-10-16T12:49:49.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.115+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 35) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5b10077840 con 0x7f5b200b1ca0 2023-10-16T12:49:49.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.281+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b200c5990 con 0x7f5b200b1ca0 2023-10-16T12:49:49.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.284+0000 7f5b1dffb640 1 -- 172.21.15.60:0/307524259 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17570 (secure 0 0 0) 0x7f5b200c5990 con 0x7f5b200b1ca0 2023-10-16T12:49:49.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:49.287 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":88,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:47.079899+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1428848083":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2802966120":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2857824565":"2023-10-17T12:49:47.079819+0000","172.21.15.144:6832/2100131827":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/954403568":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2870481770":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/1738450860":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/3154142782":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2938654985":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2412969789":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/3554060781":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2725919540":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2100131827":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/342775588":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2872949001":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6832/2872949001":"2023-10-17T12:49:40.910583+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:49.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b0c06acc0 msgr2=0x7f5b0c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b0c06acc0 0x7f5b0c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b200cbcd0 tx=0x7f5b14009040 comp rx=0 tx=0).stop 2023-10-16T12:49:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 msgr2=0x7f5b2009de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 0x7f5b2009de50 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009e50 tx=0x7f5b10004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 shutdown_connections 2023-10-16T12:49:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b0c06acc0 0x7f5b0c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b20160240 0x7f5b2009ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b20160620 0x7f5b2015e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 --2- 172.21.15.60:0/307524259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b200b1ca0 0x7f5b2009de50 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 >> 172.21.15.60:0/307524259 conn(0x7f5b20099a10 msgr2=0x7f5b2006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:49.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 shutdown_connections 2023-10-16T12:49:49.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.287+0000 7f5b26143640 1 -- 172.21.15.60:0/307524259 wait complete. 2023-10-16T12:49:49.310 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:49:49.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:49.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.536+0000 7fcb7052c640 1 Processor -- start 2023-10-16T12:49:49.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.537+0000 7fcb7052c640 1 -- start start 2023-10-16T12:49:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.538+0000 7fcb7052c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb6815aaa0 0x7fcb6815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.538+0000 7fcb7052c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6815ed60 con 0x7fcb6815ae80 2023-10-16T12:49:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.538+0000 7fcb7052c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb680707e0 con 0x7fcb680ce390 2023-10-16T12:49:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.538+0000 7fcb7052c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6806b810 con 0x7fcb6815aaa0 2023-10-16T12:49:49.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.539+0000 7fcb6dac7640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcb6815ae80 0x7fcb680cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49962/0 (socket says 172.21.15.60:49962) 2023-10-16T12:49:49.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.539+0000 7fcb6dac7640 1 -- 172.21.15.60:0/1638109091 learned_addr learned my addr 172.21.15.60:0/1638109091 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.539+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/1638109091 >> v2:172.21.15.144:3300/0 conn(0x7fcb6815aaa0 0x7fcb6815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.539+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546192569 0 0) 0x7fcb6815ed60 con 0x7fcb6815ae80 2023-10-16T12:49:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.539+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb54003520 con 0x7fcb6815ae80 2023-10-16T12:49:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2601740261 0 0) 0x7fcb54003520 con 0x7fcb6815ae80 2023-10-16T12:49:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 >> v2:172.21.15.144:3300/0 conn(0x7fcb6815aaa0 msgr2=0x7fcb6815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 --2- 172.21.15.60:0/1638109091 >> v2:172.21.15.144:3300/0 conn(0x7fcb6815aaa0 0x7fcb6815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 >> v1:172.21.15.144:6789/0 conn(0x7fcb680ce390 legacy=0x7fcb68070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6815ef90 con 0x7fcb6815ae80 2023-10-16T12:49:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcb58001670 con 0x7fcb6815ae80 2023-10-16T12:49:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 >> v1:172.21.15.60:6789/0 conn(0x7fcb6815ae80 legacy=0x7fcb680cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb54003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb540047c0 0x7fcb54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb54005180 0x7fcb5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6806b810 con 0x7fcb540039c0 2023-10-16T12:49:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5400c080 con 0x7fcb540047c0 2023-10-16T12:49:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5400c240 con 0x7fcb54005180 2023-10-16T12:49:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb54003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.540+0000 7fcb6dac7640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb540047c0 0x7fcb54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb54005180 msgr2=0x7fcb5400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6eac9640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb54005180 0x7fcb5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb54005180 0x7fcb5400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb540047c0 msgr2=0x7fcb54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb540047c0 0x7fcb54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/1638109091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54003520 con 0x7fcb540039c0 2023-10-16T12:49:49.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.541+0000 7fcb6eac9640 1 -- 172.21.15.60:0/1638109091 reap_dead start 2023-10-16T12:49:49.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.542+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb54003de0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009ac0 tx=0x7fcb540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdc5906cf40af41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:49.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.542+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6402c6b0 con 0x7fcb540039c0 2023-10-16T12:49:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.542+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6402c810 con 0x7fcb540039c0 2023-10-16T12:49:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.542+0000 7fcb6d2c6640 1 -- 172.21.15.60:0/1638109091 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6402cba0 con 0x7fcb540039c0 2023-10-16T12:49:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 -- 172.21.15.60:0/1638109091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 msgr2=0x7fcb54003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb54003de0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009ac0 tx=0x7fcb540057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 -- 172.21.15.60:0/1638109091 shutdown_connections 2023-10-16T12:49:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 --2- 172.21.15.60:0/1638109091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb54003de0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 -- 172.21.15.60:0/1638109091 >> 172.21.15.60:0/1638109091 conn(0x7fcb680cb410 msgr2=0x7fcb680cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 -- 172.21.15.60:0/1638109091 shutdown_connections 2023-10-16T12:49:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.543+0000 7fcb7052c640 1 -- 172.21.15.60:0/1638109091 wait complete. 2023-10-16T12:49:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.545+0000 7fcb7052c640 1 Processor -- start 2023-10-16T12:49:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.545+0000 7fcb7052c640 1 -- start start 2023-10-16T12:49:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb7052c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb681f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb7052c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb680c43e0 0x7fcb6815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb7052c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb680c4860 0x7fcb680b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb7052c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6815e130 con 0x7fcb540039c0 2023-10-16T12:49:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb7052c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6815de50 con 0x7fcb680c4860 2023-10-16T12:49:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb7052c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb681f6560 con 0x7fcb680c43e0 2023-10-16T12:49:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb6e2c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb681f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb6e2c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb681f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38130/0 (socket says 172.21.15.60:38130) 2023-10-16T12:49:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/4165291035 learned_addr learned my addr 172.21.15.60:0/4165291035 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb6eac9640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb680c4860 0x7fcb680b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.546+0000 7fcb6dac7640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb680c43e0 0x7fcb6815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb680c43e0 msgr2=0x7fcb6815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb680c43e0 0x7fcb6815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb680c4860 msgr2=0x7fcb680b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb680c4860 0x7fcb680b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb6e2c8640 1 -- 172.21.15.60:0/4165291035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb681f6840 con 0x7fcb540039c0 2023-10-16T12:49:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb6e2c8640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb681f7040 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009ac0 tx=0x7fcb64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64003810 con 0x7fcb540039c0 2023-10-16T12:49:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb64003970 con 0x7fcb540039c0 2023-10-16T12:49:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64003d00 con 0x7fcb540039c0 2023-10-16T12:49:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb681f6b20 con 0x7fcb540039c0 2023-10-16T12:49:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.547+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6815d8f0 con 0x7fcb540039c0 2023-10-16T12:49:49.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.548+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb681f6ce0 con 0x7fcb540039c0 2023-10-16T12:49:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.549+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 35) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fcb64040450 con 0x7fcb540039c0 2023-10-16T12:49:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.551+0000 7fcb5effd640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fcb4c06af00 0x7fcb4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.551+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcb640ab890 con 0x7fcb540039c0 2023-10-16T12:49:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.551+0000 7fcb6dac7640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fcb4c06af00 0x7fcb4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.553+0000 7fcb6dac7640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fcb4c06af00 0x7fcb4c06d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb58001050 tx=0x7fcb58009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:49.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.555+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb64076630 con 0x7fcb540039c0 2023-10-16T12:49:49.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.793+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6815dba0 con 0x7fcb540039c0 2023-10-16T12:49:49.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.795+0000 7fcb5effd640 1 -- 172.21.15.60:0/4165291035 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fcb64003e60 con 0x7fcb540039c0 2023-10-16T12:49:49.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:49.798 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:49:26.202053+0000","modified":"2023-10-16T12:49:29.284142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2404598903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2404598903},{"type":"v1","addr":"172.21.15.60:6835","nonce":2404598903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:49:49.798 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2023-10-16T12:49:49.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.797+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fcb4c06af00 msgr2=0x7fcb4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.797+0000 7fcb7052c640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fcb4c06af00 0x7fcb4c06d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb58001050 tx=0x7fcb58009040 comp rx=0 tx=0).stop 2023-10-16T12:49:49.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.797+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 msgr2=0x7fcb681f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:49.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.797+0000 7fcb7052c640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb681f7040 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009ac0 tx=0x7fcb64004730 comp rx=0 tx=0).stop 2023-10-16T12:49:49.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 shutdown_connections 2023-10-16T12:49:49.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fcb4c06af00 0x7fcb4c06d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb680c43e0 0x7fcb6815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb680c4860 0x7fcb680b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 --2- 172.21.15.60:0/4165291035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb540039c0 0x7fcb681f7040 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:49.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 >> 172.21.15.60:0/4165291035 conn(0x7fcb680cb410 msgr2=0x7fcb680b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:49.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 shutdown_connections 2023-10-16T12:49:49.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:49.798+0000 7fcb7052c640 1 -- 172.21.15.60:0/4165291035 wait complete. 2023-10-16T12:49:49.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:50.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.043+0000 7f75bcd0a640 1 Processor -- start 2023-10-16T12:49:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.043+0000 7f75bcd0a640 1 -- start start 2023-10-16T12:49:50.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.044+0000 7f75bcd0a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f75b815dde0 0x7f75b80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.044+0000 7f75bcd0a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8160a90 con 0x7f75b815c9f0 2023-10-16T12:49:50.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.044+0000 7f75bcd0a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b80cd900 con 0x7f75b8070f20 2023-10-16T12:49:50.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.044+0000 7f75bcd0a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b8071350 con 0x7f75b815dde0 2023-10-16T12:49:50.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.045+0000 7f75b77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75b815c9f0 0x7f75b80c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49976/0 (socket says 172.21.15.60:49976) 2023-10-16T12:49:50.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.045+0000 7f75b77fe640 1 -- 172.21.15.60:0/2905867438 learned_addr learned my addr 172.21.15.60:0/2905867438 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:50.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.045+0000 7f75b6ffd640 1 --2- 172.21.15.60:0/2905867438 >> v2:172.21.15.144:3300/0 conn(0x7f75b815dde0 0x7f75b80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.045+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914165436 0 0) 0x7f75b8160a90 con 0x7f75b815c9f0 2023-10-16T12:49:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.045+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a4003520 con 0x7f75b815c9f0 2023-10-16T12:49:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811574099 0 0) 0x7f75a4003520 con 0x7f75b815c9f0 2023-10-16T12:49:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 >> v2:172.21.15.144:3300/0 conn(0x7f75b815dde0 msgr2=0x7f75b80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 --2- 172.21.15.60:0/2905867438 >> v2:172.21.15.144:3300/0 conn(0x7f75b815dde0 0x7f75b80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 >> v1:172.21.15.144:6789/0 conn(0x7f75b8070f20 legacy=0x7f75b80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b80cdbe0 con 0x7f75b815c9f0 2023-10-16T12:49:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f75a8001670 con 0x7f75b815c9f0 2023-10-16T12:49:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 >> v1:172.21.15.60:6789/0 conn(0x7f75b815c9f0 legacy=0x7f75b80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75a40039c0 0x7f75a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40047c0 0x7f75a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75a40051d0 0x7f75a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b8071350 con 0x7f75a40039c0 2023-10-16T12:49:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a400c100 con 0x7f75a40051d0 2023-10-16T12:49:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a400c2c0 con 0x7f75a40047c0 2023-10-16T12:49:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b77fe640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75a40039c0 0x7f75a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b7fff640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75a40051d0 0x7f75a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b6ffd640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40047c0 0x7f75a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b77fe640 1 -- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40047c0 msgr2=0x7f75a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b77fe640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40047c0 0x7f75a4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.046+0000 7f75b77fe640 1 -- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75a40051d0 msgr2=0x7f75a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.047+0000 7f75b77fe640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75a40051d0 0x7f75a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.047+0000 7f75b77fe640 1 -- 172.21.15.60:0/2905867438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a4003520 con 0x7f75a40039c0 2023-10-16T12:49:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.047+0000 7f75b7fff640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75a40051d0 0x7f75a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.047+0000 7f75b7fff640 1 -- 172.21.15.60:0/2905867438 reap_dead start 2023-10-16T12:49:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.048+0000 7f75b77fe640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75a40039c0 0x7f75a4003dc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f75a800c160 tx=0x7f75a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b992655b0e31a8d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.048+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a8005ad0 con 0x7f75a40039c0 2023-10-16T12:49:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.048+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a8005c30 con 0x7f75a40039c0 2023-10-16T12:49:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.048+0000 7f75b67fc640 1 -- 172.21.15.60:0/2905867438 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a802c7e0 con 0x7f75a40039c0 2023-10-16T12:49:50.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.048+0000 7f75bcd0a640 1 -- 172.21.15.60:0/2905867438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75a40039c0 msgr2=0x7f75a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.048+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75a40039c0 0x7f75a4003dc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f75a800c160 tx=0x7f75a40057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.049+0000 7f75bcd0a640 1 -- 172.21.15.60:0/2905867438 shutdown_connections 2023-10-16T12:49:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.049+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/2905867438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75a40039c0 0x7f75a4003dc0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.049+0000 7f75bcd0a640 1 -- 172.21.15.60:0/2905867438 >> 172.21.15.60:0/2905867438 conn(0x7f75b80ce6a0 msgr2=0x7f75b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.049+0000 7f75bcd0a640 1 -- 172.21.15.60:0/2905867438 shutdown_connections 2023-10-16T12:49:50.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.049+0000 7f75bcd0a640 1 -- 172.21.15.60:0/2905867438 wait complete. 2023-10-16T12:49:50.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.051+0000 7f75bcd0a640 1 Processor -- start 2023-10-16T12:49:50.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.051+0000 7f75bcd0a640 1 -- start start 2023-10-16T12:49:50.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75bcd0a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40039c0 0x7f75b814eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75bcd0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 0x7f75b806bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75bcd0a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81f5cb0 0x7f75b81f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75bcd0a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b806b860 con 0x7f75b80c2190 2023-10-16T12:49:50.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75bcd0a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b814f080 con 0x7f75b81f5cb0 2023-10-16T12:49:50.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75bcd0a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b806c130 con 0x7f75a40039c0 2023-10-16T12:49:50.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 0x7f75b806bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 0x7f75b806bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38146/0 (socket says 172.21.15.60:38146) 2023-10-16T12:49:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b6ffd640 1 -- 172.21.15.60:0/3032173707 learned_addr learned my addr 172.21.15.60:0/3032173707 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b77fe640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40039c0 0x7f75b814eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b6ffd640 1 -- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40039c0 msgr2=0x7f75b814eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b7fff640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81f5cb0 0x7f75b81f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b6ffd640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40039c0 0x7f75b814eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.052+0000 7f75b6ffd640 1 -- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81f5cb0 msgr2=0x7f75b81f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75b6ffd640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81f5cb0 0x7f75b81f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75b6ffd640 1 -- 172.21.15.60:0/3032173707 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b80c26c0 con 0x7f75b80c2190 2023-10-16T12:49:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75b7fff640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81f5cb0 0x7f75b81f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75b6ffd640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 0x7f75b806bbf0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007da0 tx=0x7f75a000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:50.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00143f0 con 0x7f75b80c2190 2023-10-16T12:49:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a0014550 con 0x7f75b80c2190 2023-10-16T12:49:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00148e0 con 0x7f75b80c2190 2023-10-16T12:49:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b80c29a0 con 0x7f75b80c2190 2023-10-16T12:49:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.053+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b80c2f60 con 0x7f75b80c2190 2023-10-16T12:49:50.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.054+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b804e9b0 con 0x7f75b80c2190 2023-10-16T12:49:50.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.055+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 35) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f75a0014a40 con 0x7f75b80c2190 2023-10-16T12:49:50.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.057+0000 7f75affff640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f759806af30 0x7f759806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.057+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f75a008a680 con 0x7f75b80c2190 2023-10-16T12:49:50.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.057+0000 7f75b77fe640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f759806af30 0x7f759806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.059+0000 7f75b77fe640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f759806af30 0x7f759806d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75a8034000 tx=0x7f75a8009030 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:50.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.062+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75a0055280 con 0x7f75b80c2190 2023-10-16T12:49:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.306+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75b80adac0 con 0x7f75b80c2190 2023-10-16T12:49:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.307+0000 7f75affff640 1 -- 172.21.15.60:0/3032173707 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f75a00588a0 con 0x7f75b80c2190 2023-10-16T12:49:50.310 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:50.310 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:49:26.202053+0000","modified":"2023-10-16T12:49:29.284142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6399},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6399":{"gid":6399,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2404598903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2404598903},{"type":"v1","addr":"172.21.15.60:6835","nonce":2404598903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:49:50.310 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2023-10-16T12:49:50.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f759806af30 msgr2=0x7f759806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f759806af30 0x7f759806d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75a8034000 tx=0x7f75a8009030 comp rx=0 tx=0).stop 2023-10-16T12:49:50.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 msgr2=0x7f75b806bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 0x7f75b806bbf0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f75a0007da0 tx=0x7f75a000f520 comp rx=0 tx=0).stop 2023-10-16T12:49:50.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 shutdown_connections 2023-10-16T12:49:50.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f759806af30 0x7f759806d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81f5cb0 0x7f75b81f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b80c2190 0x7f75b806bbf0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 --2- 172.21.15.60:0/3032173707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75a40039c0 0x7f75b814eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 >> 172.21.15.60:0/3032173707 conn(0x7f75b80ce6a0 msgr2=0x7f75b80c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:50.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 shutdown_connections 2023-10-16T12:49:50.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.310+0000 7f75bcd0a640 1 -- 172.21.15.60:0/3032173707 wait complete. 2023-10-16T12:49:50.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.560+0000 7fc155189640 1 Processor -- start 2023-10-16T12:49:50.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.560+0000 7fc155189640 1 -- start start 2023-10-16T12:49:50.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.561+0000 7fc155189640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc15015aaa0 0x7fc15015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.561+0000 7fc155189640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15015ed60 con 0x7fc1500ce390 2023-10-16T12:49:50.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.561+0000 7fc155189640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1500707e0 con 0x7fc15015ae80 2023-10-16T12:49:50.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.561+0000 7fc155189640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15006b810 con 0x7fc15015aaa0 2023-10-16T12:49:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc15015aaa0 0x7fc15015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14f59e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc1500ce390 0x7fc150070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49990/0 (socket says 172.21.15.60:49990) 2023-10-16T12:49:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14f59e640 1 -- 172.21.15.60:0/3507149638 learned_addr learned my addr 172.21.15.60:0/3507149638 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14ed9d640 1 --2- 172.21.15.60:0/3507149638 >> v2:172.21.15.60:3300/0 conn(0x7fc15015aaa0 0x7fc15015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38162/0 (socket says 172.21.15.60:38162) 2023-10-16T12:49:50.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14ed9d640 1 -- 172.21.15.60:0/3507149638 >> v1:172.21.15.144:6789/0 conn(0x7fc15015ae80 legacy=0x7fc1500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:50.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14ed9d640 1 -- 172.21.15.60:0/3507149638 >> v1:172.21.15.60:6789/0 conn(0x7fc1500ce390 legacy=0x7fc150070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.562+0000 7fc14ed9d640 1 -- 172.21.15.60:0/3507149638 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15015ef90 con 0x7fc15015aaa0 2023-10-16T12:49:50.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.563+0000 7fc14ed9d640 1 --2- 172.21.15.60:0/3507149638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc15015aaa0 0x7fc15015e820 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc1400097a0 tx=0x7fc14002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a585e0cadabd5102 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:50.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc14e59c640 1 -- 172.21.15.60:0/3507149638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140031830 con 0x7fc15015aaa0 2023-10-16T12:49:50.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc14e59c640 1 -- 172.21.15.60:0/3507149638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc140031990 con 0x7fc15015aaa0 2023-10-16T12:49:50.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc14e59c640 1 -- 172.21.15.60:0/3507149638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140031d20 con 0x7fc15015aaa0 2023-10-16T12:49:50.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 -- 172.21.15.60:0/3507149638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc15015aaa0 msgr2=0x7fc15015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 --2- 172.21.15.60:0/3507149638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc15015aaa0 0x7fc15015e820 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc1400097a0 tx=0x7fc14002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 -- 172.21.15.60:0/3507149638 shutdown_connections 2023-10-16T12:49:50.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 --2- 172.21.15.60:0/3507149638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc15015aaa0 0x7fc15015e820 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 -- 172.21.15.60:0/3507149638 >> 172.21.15.60:0/3507149638 conn(0x7fc1500cb410 msgr2=0x7fc1500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:50.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 -- 172.21.15.60:0/3507149638 shutdown_connections 2023-10-16T12:49:50.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.564+0000 7fc155189640 1 -- 172.21.15.60:0/3507149638 wait complete. 2023-10-16T12:49:50.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.566+0000 7fc155189640 1 Processor -- start 2023-10-16T12:49:50.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.566+0000 7fc155189640 1 -- start start 2023-10-16T12:49:50.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc155189640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 0x7fc150147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc155189640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15015aaa0 0x7fc1500c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc155189640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15015ae80 0x7fc1500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc155189640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15015c380 con 0x7fc1500ce390 2023-10-16T12:49:50.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc155189640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1500c2770 con 0x7fc15015aaa0 2023-10-16T12:49:50.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc155189640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15015b700 con 0x7fc15015ae80 2023-10-16T12:49:50.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.567+0000 7fc14ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 0x7fc150147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 0x7fc150147d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38166/0 (socket says 172.21.15.60:38166) 2023-10-16T12:49:50.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 -- 172.21.15.60:0/4118759596 learned_addr learned my addr 172.21.15.60:0/4118759596 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:50.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 -- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15015ae80 msgr2=0x7fc1500ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15015ae80 0x7fc1500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 -- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15015aaa0 msgr2=0x7fc1500c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15015aaa0 0x7fc1500c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 -- 172.21.15.60:0/4118759596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc140009400 con 0x7fc1500ce390 2023-10-16T12:49:50.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.568+0000 7fc14ed9d640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 0x7fc150147d50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc1400098c0 tx=0x7fc1400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:50.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.569+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14003ba70 con 0x7fc1500ce390 2023-10-16T12:49:50.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.569+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14003bbf0 con 0x7fc1500ce390 2023-10-16T12:49:50.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.569+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1400453f0 con 0x7fc1500ce390 2023-10-16T12:49:50.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.569+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15015b9e0 con 0x7fc1500ce390 2023-10-16T12:49:50.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.569+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15015bfa0 con 0x7fc1500ce390 2023-10-16T12:49:50.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.570+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15004e160 con 0x7fc1500ce390 2023-10-16T12:49:50.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.571+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 35) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc14004e450 con 0x7fc1500ce390 2023-10-16T12:49:50.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.572+0000 7fc1477fe640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc13006af00 0x7fc13006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:50.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.572+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc1400abc90 con 0x7fc1500ce390 2023-10-16T12:49:50.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.573+0000 7fc147fff640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc13006af00 0x7fc13006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:50.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.574+0000 7fc147fff640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc13006af00 0x7fc13006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1501f6500 tx=0x7fc138006ba0 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:50.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.576+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc140076990 con 0x7fc1500ce390 2023-10-16T12:49:50.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.779+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc15004e160 con 0x7fc1500ce390 2023-10-16T12:49:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.782+0000 7fc1477fe640 1 -- 172.21.15.60:0/4118759596 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17570 (secure 0 0 0) 0x7fc140035020 con 0x7fc1500ce390 2023-10-16T12:49:50.785 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:50.785 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":88,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:47.079899+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:49:24.152341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-16T12:49:25.216482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1428848083":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2802966120":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2857824565":"2023-10-17T12:49:47.079819+0000","172.21.15.144:6832/2100131827":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/954403568":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2870481770":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/1738450860":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/3154142782":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2938654985":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2412969789":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/3554060781":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2725919540":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2100131827":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/342775588":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2872949001":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6832/2872949001":"2023-10-17T12:49:40.910583+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc13006af00 msgr2=0x7fc13006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc13006af00 0x7fc13006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1501f6500 tx=0x7fc138006ba0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 msgr2=0x7fc150147d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 0x7fc150147d50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc1400098c0 tx=0x7fc1400046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 shutdown_connections 2023-10-16T12:49:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc13006af00 0x7fc13006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc15015ae80 0x7fc1500ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.784+0000 7fc155189640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc15015aaa0 0x7fc1500c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.785+0000 7fc155189640 1 --2- 172.21.15.60:0/4118759596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500ce390 0x7fc150147d50 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.785+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 >> 172.21.15.60:0/4118759596 conn(0x7fc1500cb410 msgr2=0x7fc1500b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.785+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 shutdown_connections 2023-10-16T12:49:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:50.785+0000 7fc155189640 1 -- 172.21.15.60:0/4118759596 wait complete. 2023-10-16T12:49:50.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:49:51.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.015+0000 7f4105d47640 1 Processor -- start 2023-10-16T12:49:51.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.016+0000 7f4105d47640 1 -- start start 2023-10-16T12:49:51.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105d47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105d47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41000c9230 0x7f41000709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105d47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4100160a90 con 0x7f4100070f20 2023-10-16T12:49:51.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105d47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41000cdec0 con 0x7f41000c9230 2023-10-16T12:49:51.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105d47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41000cd900 con 0x7f410015c9f0 2023-10-16T12:49:51.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105546640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4100070f20 0x7f41000ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50000/0 (socket says 172.21.15.60:50000) 2023-10-16T12:49:51.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f4105546640 1 -- 172.21.15.60:0/1837903402 learned_addr learned my addr 172.21.15.60:0/1837903402 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:51.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.017+0000 7f40fffff640 1 --2- 172.21.15.60:0/1837903402 >> v2:172.21.15.60:3300/0 conn(0x7f41000c9230 0x7f41000709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f4104d45640 1 --2- 172.21.15.60:0/1837903402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f40ff7fe640 1 -- 172.21.15.60:0/1837903402 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957760891 0 0) 0x7f4100160a90 con 0x7f4100070f20 2023-10-16T12:49:51.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f40ff7fe640 1 -- 172.21.15.60:0/1837903402 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e8003520 con 0x7f4100070f20 2023-10-16T12:49:51.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f40fffff640 1 -- 172.21.15.60:0/1837903402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 msgr2=0x7f41000c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:51.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f40fffff640 1 --2- 172.21.15.60:0/1837903402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f40fffff640 1 -- 172.21.15.60:0/1837903402 >> v1:172.21.15.60:6789/0 conn(0x7f4100070f20 legacy=0x7f41000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.018+0000 7f40fffff640 1 -- 172.21.15.60:0/1837903402 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41000cdbe0 con 0x7f41000c9230 2023-10-16T12:49:51.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.019+0000 7f40fffff640 1 --2- 172.21.15.60:0/1837903402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000709e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f40f40097a0 tx=0x7f40f402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0af5eff5053875 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:51.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.019+0000 7f40ff7fe640 1 -- 172.21.15.60:0/1837903402 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f402d040 con 0x7f41000c9230 2023-10-16T12:49:51.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.019+0000 7f40ff7fe640 1 -- 172.21.15.60:0/1837903402 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f4031810 con 0x7f41000c9230 2023-10-16T12:49:51.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.019+0000 7f40ff7fe640 1 -- 172.21.15.60:0/1837903402 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4031b80 con 0x7f41000c9230 2023-10-16T12:49:51.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 -- 172.21.15.60:0/1837903402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 msgr2=0x7f41000709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 --2- 172.21.15.60:0/1837903402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000709e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f40f40097a0 tx=0x7f40f402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 -- 172.21.15.60:0/1837903402 shutdown_connections 2023-10-16T12:49:51.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 --2- 172.21.15.60:0/1837903402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000709e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 --2- 172.21.15.60:0/1837903402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 -- 172.21.15.60:0/1837903402 >> 172.21.15.60:0/1837903402 conn(0x7f41000ce6a0 msgr2=0x7f41000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:51.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 -- 172.21.15.60:0/1837903402 shutdown_connections 2023-10-16T12:49:51.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.020+0000 7f4105d47640 1 -- 172.21.15.60:0/1837903402 wait complete. 2023-10-16T12:49:51.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.022+0000 7f4105d47640 1 Processor -- start 2023-10-16T12:49:51.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.022+0000 7f4105d47640 1 -- start start 2023-10-16T12:49:51.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105d47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4100070f20 0x7f41000ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105d47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105d47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105d47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410006c6e0 con 0x7f41000c9230 2023-10-16T12:49:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105d47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41000c3a60 con 0x7f4100070f20 2023-10-16T12:49:51.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105d47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41000c3d40 con 0x7f410015c9f0 2023-10-16T12:49:51.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38180/0 (socket says 172.21.15.60:38180) 2023-10-16T12:49:51.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 -- 172.21.15.60:0/966649608 learned_addr learned my addr 172.21.15.60:0/966649608 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:51.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4104d45640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4100070f20 0x7f41000ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f4105546640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 -- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 msgr2=0x7f41000c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 -- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4100070f20 msgr2=0x7f41000ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4100070f20 0x7f41000ae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.023+0000 7f40fffff640 1 -- 172.21.15.60:0/966649608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4009400 con 0x7f41000c9230 2023-10-16T12:49:51.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.024+0000 7f40fffff640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000aeec0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f40f402b8f0 tx=0x7f40f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:51.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.024+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4006e10 con 0x7f41000c9230 2023-10-16T12:49:51.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.024+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f403ad60 con 0x7f41000c9230 2023-10-16T12:49:51.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.024+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41000c4020 con 0x7f41000c9230 2023-10-16T12:49:51.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.024+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4044610 con 0x7f41000c9230 2023-10-16T12:49:51.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.024+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41000c45e0 con 0x7f41000c9230 2023-10-16T12:49:51.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.025+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41000c9050 con 0x7f41000c9230 2023-10-16T12:49:51.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.026+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 35) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f40f404d450 con 0x7f41000c9230 2023-10-16T12:49:51.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.028+0000 7f40fd7fa640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f40ec06af00 0x7f40ec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.028+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f40f40abd10 con 0x7f41000c9230 2023-10-16T12:49:51.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.028+0000 7f4104d45640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f40ec06af00 0x7f40ec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.030+0000 7f4104d45640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f40ec06af00 0x7f40ec06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009f30 tx=0x7f40f000b040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:51.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.032+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40f40b1020 con 0x7f41000c9230 2023-10-16T12:49:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.130+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f40f4076d70 con 0x7f41000c9230 2023-10-16T12:49:51.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.277+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f41000c47d0 con 0x7f41000c9230 2023-10-16T12:49:51.280 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:49:51.278+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:49:51.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.281+0000 7f40fd7fa640 1 -- 172.21.15.60:0/966649608 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f40f408fe90 con 0x7f41000c9230 2023-10-16T12:49:51.284 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:49:51.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.284+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f40ec06af00 msgr2=0x7f40ec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.284+0000 7f4105d47640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f40ec06af00 0x7f40ec06d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009f30 tx=0x7f40f000b040 comp rx=0 tx=0).stop 2023-10-16T12:49:51.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.284+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 msgr2=0x7f41000aeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.284+0000 7f4105d47640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000aeec0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f40f402b8f0 tx=0x7f40f40046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 shutdown_connections 2023-10-16T12:49:51.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f40ec06af00 0x7f40ec06d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f410015c9f0 0x7f41000c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41000c9230 0x7f41000aeec0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 --2- 172.21.15.60:0/966649608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4100070f20 0x7f41000ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 >> 172.21.15.60:0/966649608 conn(0x7f41000ce6a0 msgr2=0x7f410015d800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:51.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 shutdown_connections 2023-10-16T12:49:51.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.285+0000 7f4105d47640 1 -- 172.21.15.60:0/966649608 wait complete. 2023-10-16T12:49:51.299 INFO:tasks.ceph.mds.a.smithi060.stderr: -3996> 2023-10-16T12:49:30.773+0000 7f63cd168640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:49:51.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:49:51.369 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:49:51.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.514+0000 7f52e5d16640 1 Processor -- start 2023-10-16T12:49:51.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.515+0000 7f52e5d16640 1 -- start start 2023-10-16T12:49:51.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.516+0000 7f52e5d16640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f52e006c040 0x7f52e006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.516+0000 7f52e5d16640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 0x7f52e015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.516+0000 7f52e5d16640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e006c960 con 0x7f52e015ebc0 2023-10-16T12:49:51.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.516+0000 7f52e5d16640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e015b9d0 con 0x7f52e006c040 2023-10-16T12:49:51.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.516+0000 7f52e5d16640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015bcb0 con 0x7f52e00ad970 2023-10-16T12:49:51.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.517+0000 7f52e5515640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f52e015ebc0 0x7f52e00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50016/0 (socket says 172.21.15.60:50016) 2023-10-16T12:49:51.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.517+0000 7f52e5515640 1 -- 172.21.15.60:0/756919829 learned_addr learned my addr 172.21.15.60:0/756919829 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:51.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.517+0000 7f52d7fff640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 0x7f52e015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.517+0000 7f52e4d14640 1 --2- 172.21.15.60:0/756919829 >> v2:172.21.15.144:3300/0 conn(0x7f52e006c040 0x7f52e006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.517+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662908561 0 0) 0x7f52e006c960 con 0x7f52e015ebc0 2023-10-16T12:49:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.517+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c8003520 con 0x7f52e015ebc0 2023-10-16T12:49:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4293157949 0 0) 0x7f52c8003520 con 0x7f52e015ebc0 2023-10-16T12:49:51.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 msgr2=0x7f52e015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:51.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 0x7f52e015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 >> v2:172.21.15.144:3300/0 conn(0x7f52e006c040 msgr2=0x7f52e006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:51.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 --2- 172.21.15.60:0/756919829 >> v2:172.21.15.144:3300/0 conn(0x7f52e006c040 0x7f52e006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e015bf90 con 0x7f52e015ebc0 2023-10-16T12:49:51.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f52d80019f0 con 0x7f52e015ebc0 2023-10-16T12:49:51.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 >> v1:172.21.15.60:6789/0 conn(0x7f52e015ebc0 legacy=0x7f52e00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 0x7f52c8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c8003ef0 0x7f52c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c8004d00 0x7f52c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015b9d0 con 0x7f52c8003ef0 2023-10-16T12:49:51.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015bcb0 con 0x7f52c8004d00 2023-10-16T12:49:51.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c800b980 con 0x7f52e00ad970 2023-10-16T12:49:51.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52d7fff640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c8003ef0 0x7f52c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.518+0000 7f52e4d14640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 0x7f52c8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52e5515640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c8004d00 0x7f52c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52d7fff640 1 -- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 msgr2=0x7f52c8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:51.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52d7fff640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52e00ad970 0x7f52c8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52d7fff640 1 -- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c8004d00 msgr2=0x7f52c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:51.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52d7fff640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c8004d00 0x7f52c8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52d7fff640 1 -- 172.21.15.60:0/756919829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8003520 con 0x7f52c8003ef0 2023-10-16T12:49:51.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.519+0000 7f52e5515640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c8004d00 0x7f52c8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:51.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.520+0000 7f52d7fff640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c8003ef0 0x7f52c80047c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009ac0 tx=0x7f52c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4214795214015215 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:51.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.520+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d002c6b0 con 0x7f52c8003ef0 2023-10-16T12:49:51.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.520+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d002c810 con 0x7f52c8003ef0 2023-10-16T12:49:51.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.520+0000 7f52d77fe640 1 -- 172.21.15.60:0/756919829 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d002cba0 con 0x7f52c8003ef0 2023-10-16T12:49:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5d16640 1 -- 172.21.15.60:0/756919829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c8003ef0 msgr2=0x7f52c80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5d16640 1 --2- 172.21.15.60:0/756919829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c8003ef0 0x7f52c80047c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009ac0 tx=0x7f52c80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5515640 1 -- 172.21.15.60:0/756919829 reap_dead start 2023-10-16T12:49:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5d16640 1 -- 172.21.15.60:0/756919829 shutdown_connections 2023-10-16T12:49:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5d16640 1 -- 172.21.15.60:0/756919829 >> 172.21.15.60:0/756919829 conn(0x7f52e00ce6a0 msgr2=0x7f52e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5d16640 1 -- 172.21.15.60:0/756919829 shutdown_connections 2023-10-16T12:49:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.521+0000 7f52e5d16640 1 -- 172.21.15.60:0/756919829 wait complete. 2023-10-16T12:49:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.523+0000 7f52e5d16640 1 Processor -- start 2023-10-16T12:49:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.523+0000 7f52e5d16640 1 -- start start 2023-10-16T12:49:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5d16640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c8003ef0 0x7f52e015d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5d16640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52e00c7650 0x7f52e00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5d16640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 0x7f52e0152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5d16640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00c6c40 con 0x7f52e00c9230 2023-10-16T12:49:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5d16640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00c71c0 con 0x7f52e00c7650 2023-10-16T12:49:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5d16640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00c9740 con 0x7f52c8003ef0 2023-10-16T12:49:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5515640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 0x7f52e0152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5515640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 0x7f52e0152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38212/0 (socket says 172.21.15.60:38212) 2023-10-16T12:49:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.524+0000 7f52e5515640 1 -- 172.21.15.60:0/1575657122 learned_addr learned my addr 172.21.15.60:0/1575657122 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e4d14640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c8003ef0 0x7f52e015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52d7fff640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52e00c7650 0x7f52e00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e5515640 1 -- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c8003ef0 msgr2=0x7f52e015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e5515640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c8003ef0 0x7f52e015d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e5515640 1 -- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52e00c7650 msgr2=0x7f52e00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e5515640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52e00c7650 0x7f52e00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e5515640 1 -- 172.21.15.60:0/1575657122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00c9a20 con 0x7f52e00c9230 2023-10-16T12:49:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.525+0000 7f52e5515640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 0x7f52e0152dd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f52d80012e0 tx=0x7f52d800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.526+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d800bb80 con 0x7f52e00c9230 2023-10-16T12:49:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.526+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d800bce0 con 0x7f52e00c9230 2023-10-16T12:49:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.526+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e00c6e50 con 0x7f52e00c9230 2023-10-16T12:49:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.526+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d80072b0 con 0x7f52e00c9230 2023-10-16T12:49:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.526+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e015dd60 con 0x7f52e00c9230 2023-10-16T12:49:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.527+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e015df50 con 0x7f52e00c9230 2023-10-16T12:49:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.528+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52d8007410 con 0x7f52e00c9230 2023-10-16T12:49:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.529+0000 7f52d57fa640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f52cc06af90 0x7f52cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.529+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f52d8089e90 con 0x7f52e00c9230 2023-10-16T12:49:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.529+0000 7f52e4d14640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f52cc06af90 0x7f52cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.531+0000 7f52e4d14640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f52cc06af90 0x7f52cc06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52e00c4f00 tx=0x7f52dc008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.533+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52d8054970 con 0x7f52e00c9230 2023-10-16T12:49:51.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:51.776+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f52e00c9bb0 con 0x7f52e00c9230 2023-10-16T12:49:52.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.285+0000 7f52d57fa640 1 -- 172.21.15.60:0/1575657122 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f52d808b020 con 0x7f52e00c9230 2023-10-16T12:49:52.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f52cc06af90 msgr2=0x7f52cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:52.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f52cc06af90 0x7f52cc06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52e00c4f00 tx=0x7f52dc008040 comp rx=0 tx=0).stop 2023-10-16T12:49:52.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 msgr2=0x7f52e0152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 0x7f52e0152dd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f52d80012e0 tx=0x7f52d800b4a0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 shutdown_connections 2023-10-16T12:49:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f52cc06af90 0x7f52cc06d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52e00c9230 0x7f52e0152dd0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52e00c7650 0x7f52e00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 --2- 172.21.15.60:0/1575657122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c8003ef0 0x7f52e015d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.288+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 >> 172.21.15.60:0/1575657122 conn(0x7f52e00ce6a0 msgr2=0x7f52e00b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.289+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 shutdown_connections 2023-10-16T12:49:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.289+0000 7f52e5d16640 1 -- 172.21.15.60:0/1575657122 wait complete. 2023-10-16T12:49:52.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:49:52.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.543+0000 7f2ff2f1d640 1 Processor -- start 2023-10-16T12:49:52.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.543+0000 7f2ff2f1d640 1 -- start start 2023-10-16T12:49:52.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff2f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2fec071e10 0x7f2fec15e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:52.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff2f1d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fec06b4c0 0x7f2fec06c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:52.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff2f1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec06bdb0 con 0x7f2fec159c50 2023-10-16T12:49:52.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff2f1d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec06c090 con 0x7f2fec06b4c0 2023-10-16T12:49:52.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff2f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec15e780 con 0x7f2fec071e10 2023-10-16T12:49:52.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff14ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fec06b4c0 0x7f2fec06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:52.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff14ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fec06b4c0 0x7f2fec06c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38218/0 (socket says 172.21.15.60:38218) 2023-10-16T12:49:52.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.545+0000 7f2ff14ba640 1 -- 172.21.15.60:0/2576269273 learned_addr learned my addr 172.21.15.60:0/2576269273 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:52.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.546+0000 7f2febfff640 1 --2- 172.21.15.60:0/2576269273 >> v2:172.21.15.144:3300/0 conn(0x7f2fec071e10 0x7f2fec15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:52.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.546+0000 7f2ff14ba640 1 -- 172.21.15.60:0/2576269273 >> v2:172.21.15.144:3300/0 conn(0x7f2fec071e10 msgr2=0x7f2fec15e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:52.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.546+0000 7f2ff14ba640 1 --2- 172.21.15.60:0/2576269273 >> v2:172.21.15.144:3300/0 conn(0x7f2fec071e10 0x7f2fec15e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.546+0000 7f2ff14ba640 1 -- 172.21.15.60:0/2576269273 >> v1:172.21.15.144:6789/0 conn(0x7f2fec159c50 legacy=0x7f2fec06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:52.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.546+0000 7f2ff14ba640 1 -- 172.21.15.60:0/2576269273 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec06fd20 con 0x7f2fec06b4c0 2023-10-16T12:49:52.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.547+0000 7f2ff14ba640 1 --2- 172.21.15.60:0/2576269273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec06b4c0 0x7f2fec06c520 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2fec072490 tx=0x7f2fe402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8847b105145111c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:52.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.547+0000 7f2feb7fe640 1 -- 172.21.15.60:0/2576269273 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe402c9f0 con 0x7f2fec06b4c0 2023-10-16T12:49:52.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.547+0000 7f2feb7fe640 1 -- 172.21.15.60:0/2576269273 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe402cb50 con 0x7f2fec06b4c0 2023-10-16T12:49:52.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.547+0000 7f2feb7fe640 1 -- 172.21.15.60:0/2576269273 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe40343f0 con 0x7f2fec06b4c0 2023-10-16T12:49:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/2576269273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec06b4c0 msgr2=0x7f2fec06c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/2576269273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec06b4c0 0x7f2fec06c520 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2fec072490 tx=0x7f2fe402be50 comp rx=0 tx=0).stop 2023-10-16T12:49:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/2576269273 shutdown_connections 2023-10-16T12:49:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/2576269273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec06b4c0 0x7f2fec06c520 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/2576269273 >> v2:172.21.15.144:3300/0 conn(0x7f2fec071e10 0x7f2fec15e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/2576269273 >> 172.21.15.60:0/2576269273 conn(0x7f2fec0b0cc0 msgr2=0x7f2fec0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:52.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/2576269273 shutdown_connections 2023-10-16T12:49:52.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.548+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/2576269273 wait complete. 2023-10-16T12:49:52.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.551+0000 7f2ff2f1d640 1 Processor -- start 2023-10-16T12:49:52.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.551+0000 7f2ff2f1d640 1 -- start start 2023-10-16T12:49:52.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.552+0000 7f2ff2f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fec06b4c0 0x7f2fec09e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:52.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.552+0000 7f2ff2f1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fec159c50 0x7f2fec155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:52.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.552+0000 7f2ff2f1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 0x7f2fec15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:52.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.552+0000 7f2ff2f1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec09e990 con 0x7f2fec156270 2023-10-16T12:49:52.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.552+0000 7f2ff2f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec155fa0 con 0x7f2fec06b4c0 2023-10-16T12:49:52.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.552+0000 7f2ff2f1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec09d3b0 con 0x7f2fec159c50 2023-10-16T12:49:52.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 0x7f2fec15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:52.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 0x7f2fec15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38220/0 (socket says 172.21.15.60:38220) 2023-10-16T12:49:52.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 -- 172.21.15.60:0/1235057304 learned_addr learned my addr 172.21.15.60:0/1235057304 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:52.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff0cb9640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fec06b4c0 0x7f2fec09e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:52.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2febfff640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fec159c50 0x7f2fec155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:52.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 -- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fec159c50 msgr2=0x7f2fec155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:52.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fec159c50 0x7f2fec155a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 -- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fec06b4c0 msgr2=0x7f2fec09e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:52.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fec06b4c0 0x7f2fec09e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.553+0000 7f2ff14ba640 1 -- 172.21.15.60:0/1235057304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe4009b30 con 0x7f2fec156270 2023-10-16T12:49:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.554+0000 7f2ff14ba640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 0x7f2fec15bc60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4036040 tx=0x7f2fe4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.554+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4009510 con 0x7f2fec156270 2023-10-16T12:49:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.554+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe403e450 con 0x7f2fec156270 2023-10-16T12:49:52.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.554+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec09d690 con 0x7f2fec156270 2023-10-16T12:49:52.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.554+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe403e7e0 con 0x7f2fec156270 2023-10-16T12:49:52.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.554+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec09dc50 con 0x7f2fec156270 2023-10-16T12:49:52.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.555+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec04e160 con 0x7f2fec156270 2023-10-16T12:49:52.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.556+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2fe403e940 con 0x7f2fec156270 2023-10-16T12:49:52.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.557+0000 7f2fe97fa640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2fd806afe0 0x7f2fd806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:52.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.557+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f2fe40ac860 con 0x7f2fec156270 2023-10-16T12:49:52.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.558+0000 7f2ff0cb9640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2fd806afe0 0x7f2fd806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:52.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.559+0000 7f2ff0cb9640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2fd806afe0 0x7f2fd806d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0006fd0 tx=0x7f2fe0008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:52.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.561+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fe40774d0 con 0x7f2fec156270 2023-10-16T12:49:52.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:52.757+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2fec15ce40 con 0x7f2fec156270 2023-10-16T12:49:53.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.296+0000 7f2fe97fa640 1 -- 172.21.15.60:0/1235057304 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2fe40430d0 con 0x7f2fec156270 2023-10-16T12:49:53.299 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:49:53.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.299+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2fd806afe0 msgr2=0x7f2fd806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:53.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.299+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2fd806afe0 0x7f2fd806d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0006fd0 tx=0x7f2fe0008040 comp rx=0 tx=0).stop 2023-10-16T12:49:53.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.299+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 msgr2=0x7f2fec15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:53.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.299+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 0x7f2fec15bc60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4036040 tx=0x7f2fe4004a40 comp rx=0 tx=0).stop 2023-10-16T12:49:53.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 shutdown_connections 2023-10-16T12:49:53.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2fd806afe0 0x7f2fd806d4a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fec159c50 0x7f2fec155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec156270 0x7f2fec15bc60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 --2- 172.21.15.60:0/1235057304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fec06b4c0 0x7f2fec09e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 >> 172.21.15.60:0/1235057304 conn(0x7f2fec0b0cc0 msgr2=0x7f2fec0c2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:53.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 shutdown_connections 2023-10-16T12:49:53.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.300+0000 7f2ff2f1d640 1 -- 172.21.15.60:0/1235057304 wait complete. 2023-10-16T12:49:53.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:49:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.542+0000 7f529759e640 1 Processor -- start 2023-10-16T12:49:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.543+0000 7f529759e640 1 -- start start 2023-10-16T12:49:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.544+0000 7f529759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f52980cd800 0x7f52980cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.544+0000 7f529759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52980ce180 con 0x7f529815cba0 2023-10-16T12:49:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.544+0000 7f529759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52980d0500 con 0x7f529806aeb0 2023-10-16T12:49:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.544+0000 7f529759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52980d07c0 con 0x7f52980cd800 2023-10-16T12:49:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.544+0000 7f529659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f529815cba0 0x7f5298071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50022/0 (socket says 172.21.15.60:50022) 2023-10-16T12:49:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.544+0000 7f529659c640 1 -- 172.21.15.60:0/1329014442 learned_addr learned my addr 172.21.15.60:0/1329014442 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f528ffff640 1 --2- 172.21.15.60:0/1329014442 >> v2:172.21.15.144:3300/0 conn(0x7f52980cd800 0x7f52980cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883230032 0 0) 0x7f52980ce180 con 0x7f529815cba0 2023-10-16T12:49:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5284003520 con 0x7f529815cba0 2023-10-16T12:49:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1549879053 0 0) 0x7f5284003520 con 0x7f529815cba0 2023-10-16T12:49:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 >> v2:172.21.15.144:3300/0 conn(0x7f52980cd800 msgr2=0x7f52980cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 --2- 172.21.15.60:0/1329014442 >> v2:172.21.15.144:3300/0 conn(0x7f52980cd800 0x7f52980cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 >> v1:172.21.15.144:6789/0 conn(0x7f529806aeb0 legacy=0x7f529806b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980d0aa0 con 0x7f529815cba0 2023-10-16T12:49:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5288001670 con 0x7f529815cba0 2023-10-16T12:49:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f528ffff640 1 --2- 172.21.15.60:0/1329014442 >> v2:172.21.15.144:3300/0 conn(0x7f52980cd800 0x7f52980cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.545+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 >> v1:172.21.15.60:6789/0 conn(0x7f529815cba0 legacy=0x7f5298071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5295d9b640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840039c0 0x7f5284003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5295d9b640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52840047c0 0x7f5284004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5295d9b640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52840051d0 0x7f528400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52980d07c0 con 0x7f52840047c0 2023-10-16T12:49:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528400c100 con 0x7f52840039c0 2023-10-16T12:49:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528400c2c0 con 0x7f52840051d0 2023-10-16T12:49:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f528ffff640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52840047c0 0x7f5284004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f529659c640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840039c0 0x7f5284003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f528ffff640 1 -- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52840051d0 msgr2=0x7f528400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f528ffff640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52840051d0 0x7f528400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f528ffff640 1 -- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840039c0 msgr2=0x7f5284003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f528ffff640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840039c0 0x7f5284003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f528ffff640 1 -- 172.21.15.60:0/1329014442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5284003520 con 0x7f52840047c0 2023-10-16T12:49:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f5296d9d640 1 -- 172.21.15.60:0/1329014442 reap_dead start 2023-10-16T12:49:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.546+0000 7f529659c640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840039c0 0x7f5284003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.548+0000 7f528ffff640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52840047c0 0x7f5284004c40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5280006fd0 tx=0x7f52840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e220cf21277d7ca0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.548+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280004230 con 0x7f52840047c0 2023-10-16T12:49:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.548+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280004390 con 0x7f52840047c0 2023-10-16T12:49:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.548+0000 7f5295d9b640 1 -- 172.21.15.60:0/1329014442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280003830 con 0x7f52840047c0 2023-10-16T12:49:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 -- 172.21.15.60:0/1329014442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52840047c0 msgr2=0x7f5284004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52840047c0 0x7f5284004c40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5280006fd0 tx=0x7f52840057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 -- 172.21.15.60:0/1329014442 shutdown_connections 2023-10-16T12:49:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 --2- 172.21.15.60:0/1329014442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52840047c0 0x7f5284004c40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 -- 172.21.15.60:0/1329014442 >> 172.21.15.60:0/1329014442 conn(0x7f5298065ed0 msgr2=0x7f52980662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 -- 172.21.15.60:0/1329014442 shutdown_connections 2023-10-16T12:49:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.549+0000 7f529759e640 1 -- 172.21.15.60:0/1329014442 wait complete. 2023-10-16T12:49:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.551+0000 7f529759e640 1 Processor -- start 2023-10-16T12:49:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.551+0000 7f529759e640 1 -- start start 2023-10-16T12:49:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840047c0 0x7f52980c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 0x7f5298159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52980b3500 0x7f52980b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52980c6ce0 con 0x7f529815dba0 2023-10-16T12:49:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52980c5e70 con 0x7f52840047c0 2023-10-16T12:49:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52980b3ec0 con 0x7f52980b3500 2023-10-16T12:49:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f528ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 0x7f5298159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f528ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 0x7f5298159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38248/0 (socket says 172.21.15.60:38248) 2023-10-16T12:49:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f528ffff640 1 -- 172.21.15.60:0/2920631550 learned_addr learned my addr 172.21.15.60:0/2920631550 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f529659c640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840047c0 0x7f52980c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.552+0000 7f5296d9d640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52980b3500 0x7f52980b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528ffff640 1 -- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52980b3500 msgr2=0x7f52980b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528ffff640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52980b3500 0x7f52980b3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528ffff640 1 -- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840047c0 msgr2=0x7f52980c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528ffff640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840047c0 0x7f52980c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528ffff640 1 -- 172.21.15.60:0/2920631550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980c6060 con 0x7f529815dba0 2023-10-16T12:49:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528ffff640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 0x7f5298159ce0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5280006fd0 tx=0x7f5280004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280043600 con 0x7f529815dba0 2023-10-16T12:49:53.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280043760 con 0x7f529815dba0 2023-10-16T12:49:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52980c6340 con 0x7f529815dba0 2023-10-16T12:49:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.553+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280043b10 con 0x7f529815dba0 2023-10-16T12:49:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.554+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52980c6900 con 0x7f529815dba0 2023-10-16T12:49:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.555+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52980c6e20 con 0x7f529815dba0 2023-10-16T12:49:53.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.556+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52800383f0 con 0x7f529815dba0 2023-10-16T12:49:53.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.557+0000 7f528effd640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f527806afe0 0x7f527806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.557+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6045+0+0 (secure 0 0 0) 0x7f52800aba20 con 0x7f529815dba0 2023-10-16T12:49:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.557+0000 7f529659c640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f527806afe0 0x7f527806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.558+0000 7f529659c640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f527806afe0 0x7f527806d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5288000c00 tx=0x7f5288009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:53.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.561+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52980c6e20 con 0x7f529815dba0 2023-10-16T12:49:53.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:53.755+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f529804e9b0 con 0x7f529815dba0 2023-10-16T12:49:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.299+0000 7f528effd640 1 -- 172.21.15.60:0/2920631550 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) v1 ==== 169+0+0 (secure 0 0 0) 0x7f52800480d0 con 0x7f529815dba0 2023-10-16T12:49:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:49:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f527806afe0 msgr2=0x7f527806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f527806afe0 0x7f527806d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5288000c00 tx=0x7f5288009040 comp rx=0 tx=0).stop 2023-10-16T12:49:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 msgr2=0x7f5298159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 0x7f5298159ce0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5280006fd0 tx=0x7f5280004730 comp rx=0 tx=0).stop 2023-10-16T12:49:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 shutdown_connections 2023-10-16T12:49:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f527806afe0 0x7f527806d4a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52980b3500 0x7f52980b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f529815dba0 0x7f5298159ce0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 --2- 172.21.15.60:0/2920631550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52840047c0 0x7f52980c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.303+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 >> 172.21.15.60:0/2920631550 conn(0x7f5298065ed0 msgr2=0x7f52980b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.304+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 shutdown_connections 2023-10-16T12:49:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.304+0000 7f529759e640 1 -- 172.21.15.60:0/2920631550 wait complete. 2023-10-16T12:49:54.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:49:54.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.536+0000 7f5ba227e640 1 Processor -- start 2023-10-16T12:49:54.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.537+0000 7f5ba227e640 1 -- start start 2023-10-16T12:49:54.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.538+0000 7f5ba227e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b9c0b2080 0x7f5b9c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:54.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.538+0000 7f5ba227e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b9c0cfc40 0x7f5b9c0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:54.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.538+0000 7f5ba227e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c0d0970 con 0x7f5b9c0b1ca0 2023-10-16T12:49:54.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.538+0000 7f5ba227e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9c0d0ea0 con 0x7f5b9c0b2080 2023-10-16T12:49:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.538+0000 7f5ba227e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9c0d0bc0 con 0x7f5b9c0cfc40 2023-10-16T12:49:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba0a7b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5b9c0b2080 0x7f5b9c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba127c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5b9c0b1ca0 0x7f5b9c0ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50030/0 (socket says 172.21.15.60:50030) 2023-10-16T12:49:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba127c640 1 -- 172.21.15.60:0/3208740274 learned_addr learned my addr 172.21.15.60:0/3208740274 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba1a7d640 1 --2- 172.21.15.60:0/3208740274 >> v2:172.21.15.144:3300/0 conn(0x7f5b9c0cfc40 0x7f5b9c0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/3208740274 >> v2:172.21.15.144:3300/0 conn(0x7f5b9c0cfc40 msgr2=0x7f5b9c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba0a7b640 1 --2- 172.21.15.60:0/3208740274 >> v2:172.21.15.144:3300/0 conn(0x7f5b9c0cfc40 0x7f5b9c0b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/3208740274 >> v1:172.21.15.60:6789/0 conn(0x7f5b9c0b1ca0 legacy=0x7f5b9c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.539+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/3208740274 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c0cd940 con 0x7f5b9c0b2080 2023-10-16T12:49:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.540+0000 7f5ba0a7b640 1 --2- 172.21.15.60:0/3208740274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c0d36b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0d1d30 tx=0x7f5b8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa534063f6079bd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.541+0000 7f5b93fff640 1 -- 172.21.15.60:0/3208740274 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c030610 con 0x7f5b9c0b2080 2023-10-16T12:49:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.541+0000 7f5b93fff640 1 -- 172.21.15.60:0/3208740274 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c030770 con 0x7f5b9c0b2080 2023-10-16T12:49:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.541+0000 7f5b93fff640 1 -- 172.21.15.60:0/3208740274 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c030b00 con 0x7f5b9c0b2080 2023-10-16T12:49:54.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.541+0000 7f5ba227e640 1 -- 172.21.15.60:0/3208740274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 msgr2=0x7f5b9c0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.541+0000 7f5ba227e640 1 --2- 172.21.15.60:0/3208740274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c0d36b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0d1d30 tx=0x7f5b8c02b960 comp rx=0 tx=0).stop 2023-10-16T12:49:54.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.542+0000 7f5ba227e640 1 -- 172.21.15.60:0/3208740274 shutdown_connections 2023-10-16T12:49:54.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.542+0000 7f5ba227e640 1 --2- 172.21.15.60:0/3208740274 >> v2:172.21.15.144:3300/0 conn(0x7f5b9c0cfc40 0x7f5b9c0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.542+0000 7f5ba227e640 1 --2- 172.21.15.60:0/3208740274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c0d36b0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.542+0000 7f5ba227e640 1 -- 172.21.15.60:0/3208740274 >> 172.21.15.60:0/3208740274 conn(0x7f5b9c0d2b90 msgr2=0x7f5b9c0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:54.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.542+0000 7f5ba227e640 1 -- 172.21.15.60:0/3208740274 shutdown_connections 2023-10-16T12:49:54.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.542+0000 7f5ba227e640 1 -- 172.21.15.60:0/3208740274 wait complete. 2023-10-16T12:49:54.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.544+0000 7f5ba227e640 1 Processor -- start 2023-10-16T12:49:54.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.544+0000 7f5ba227e640 1 -- start start 2023-10-16T12:49:54.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba227e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b9c0b1ca0 0x7f5b9c15c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:54.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba227e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:54.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba227e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b9c0cfc40 0x7f5b9c15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:54.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba227e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c0c6e20 con 0x7f5b9c0b2080 2023-10-16T12:49:54.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba227e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c1f6320 con 0x7f5b9c0cfc40 2023-10-16T12:49:54.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba227e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c15b550 con 0x7f5b9c0b1ca0 2023-10-16T12:49:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba0a7b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba0a7b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c15a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38278/0 (socket says 172.21.15.60:38278) 2023-10-16T12:49:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/615709212 learned_addr learned my addr 172.21.15.60:0/615709212 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba1a7d640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b9c0cfc40 0x7f5b9c15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.545+0000 7f5ba127c640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b9c0b1ca0 0x7f5b9c15c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b9c0b1ca0 msgr2=0x7f5b9c15c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba0a7b640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b9c0b1ca0 0x7f5b9c15c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b9c0cfc40 msgr2=0x7f5b9c15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba0a7b640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b9c0cfc40 0x7f5b9c15aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba0a7b640 1 -- 172.21.15.60:0/615709212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c009400 con 0x7f5b9c0b2080 2023-10-16T12:49:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba127c640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b9c0b1ca0 0x7f5b9c15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba0a7b640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c15a970 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009890 tx=0x7f5b8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c03e070 con 0x7f5b9c0b2080 2023-10-16T12:49:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c006e10 con 0x7f5b9c0b2080 2023-10-16T12:49:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c039c30 con 0x7f5b9c0b2080 2023-10-16T12:49:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9c15b830 con 0x7f5b9c0b2080 2023-10-16T12:49:54.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.546+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c0c6a40 con 0x7f5b9c0b2080 2023-10-16T12:49:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.548+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9c0c6f60 con 0x7f5b9c0b2080 2023-10-16T12:49:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.549+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5b8c0423f0 con 0x7f5b9c0b2080 2023-10-16T12:49:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.549+0000 7f5b91ffb640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b8006af90 0x7f5b8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.550+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f5b8c0ab5a0 con 0x7f5b9c0b2080 2023-10-16T12:49:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.550+0000 7f5ba127c640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b8006af90 0x7f5b8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.551+0000 7f5ba127c640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b8006af90 0x7f5b8006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5b98007fd0 tx=0x7f5b98009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.554+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b9c0c6f60 con 0x7f5b9c0b2080 2023-10-16T12:49:54.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.758+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5b9c0c6f60 con 0x7f5b9c0b2080 2023-10-16T12:49:54.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.765+0000 7f5b91ffb640 1 -- 172.21.15.60:0/615709212 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5b9c0c6f60 con 0x7f5b9c0b2080 2023-10-16T12:49:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.768+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b8006af90 msgr2=0x7f5b8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.768+0000 7f5ba227e640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b8006af90 0x7f5b8006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5b98007fd0 tx=0x7f5b98009040 comp rx=0 tx=0).stop 2023-10-16T12:49:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.768+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 msgr2=0x7f5b9c15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.768+0000 7f5ba227e640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c15a970 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009890 tx=0x7f5b8c0046c0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.768+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 shutdown_connections 2023-10-16T12:49:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5b8006af90 0x7f5b8006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b9c0cfc40 0x7f5b9c15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5b9c0b2080 0x7f5b9c15a970 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 --2- 172.21.15.60:0/615709212 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5b9c0b1ca0 0x7f5b9c15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 >> 172.21.15.60:0/615709212 conn(0x7f5b9c0d2b90 msgr2=0x7f5b9c0d1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 shutdown_connections 2023-10-16T12:49:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:54.769+0000 7f5ba227e640 1 -- 172.21.15.60:0/615709212 wait complete. 2023-10-16T12:49:54.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:49:55.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.016+0000 7f45e102d640 1 Processor -- start 2023-10-16T12:49:55.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.016+0000 7f45e102d640 1 -- start start 2023-10-16T12:49:55.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.018+0000 7f45e102d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f45dc0addb0 0x7f45dc0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.018+0000 7f45e102d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 0x7f45dc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.018+0000 7f45e102d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc0cb140 con 0x7f45dc06a2e0 2023-10-16T12:49:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.018+0000 7f45e102d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc161a50 con 0x7f45dc0addb0 2023-10-16T12:49:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.018+0000 7f45e102d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc06a750 con 0x7f45dc0c9000 2023-10-16T12:49:55.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.019+0000 7f45db59e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f45dc06a2e0 0x7f45dc0cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50042/0 (socket says 172.21.15.60:50042) 2023-10-16T12:49:55.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.019+0000 7f45db59e640 1 -- 172.21.15.60:0/3874638468 learned_addr learned my addr 172.21.15.60:0/3874638468 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:55.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.019+0000 7f45d3fff640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 0x7f45dc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.019+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765226784 0 0) 0x7f45dc0cb140 con 0x7f45dc06a2e0 2023-10-16T12:49:55.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.019+0000 7f45dad9d640 1 --2- 172.21.15.60:0/3874638468 >> v2:172.21.15.144:3300/0 conn(0x7f45dc0addb0 0x7f45dc0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.019+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c8003520 con 0x7f45dc06a2e0 2023-10-16T12:49:55.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420754975 0 0) 0x7f45c8003520 con 0x7f45dc06a2e0 2023-10-16T12:49:55.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 msgr2=0x7f45dc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 0x7f45dc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 >> v2:172.21.15.144:3300/0 conn(0x7f45dc0addb0 msgr2=0x7f45dc0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 --2- 172.21.15.60:0/3874638468 >> v2:172.21.15.144:3300/0 conn(0x7f45dc0addb0 0x7f45dc0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc06b030 con 0x7f45dc06a2e0 2023-10-16T12:49:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f45d40019f0 con 0x7f45dc06a2e0 2023-10-16T12:49:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 >> v1:172.21.15.60:6789/0 conn(0x7f45dc06a2e0 legacy=0x7f45dc0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 0x7f45c80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8003f70 0x7f45c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45c8004d00 0x7f45c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc161a50 con 0x7f45c8004d00 2023-10-16T12:49:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc06a750 con 0x7f45c8003f70 2023-10-16T12:49:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c800b9b0 con 0x7f45dc0c9000 2023-10-16T12:49:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45db59e640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45c8004d00 0x7f45c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.020+0000 7f45dad9d640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 0x7f45c80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.021+0000 7f45d3fff640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8003f70 0x7f45c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.021+0000 7f45db59e640 1 -- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 msgr2=0x7f45c80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.021+0000 7f45db59e640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc0c9000 0x7f45c80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.021+0000 7f45db59e640 1 -- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8003f70 msgr2=0x7f45c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.021+0000 7f45db59e640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8003f70 0x7f45c80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.021+0000 7f45db59e640 1 -- 172.21.15.60:0/3874638468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8003520 con 0x7f45c8004d00 2023-10-16T12:49:55.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.022+0000 7f45db59e640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45c8004d00 0x7f45c8005180 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f45d400a760 tx=0x7f45c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef87ae6d2b7dfa57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.022+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4034630 con 0x7f45c8004d00 2023-10-16T12:49:55.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.022+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d402f070 con 0x7f45c8004d00 2023-10-16T12:49:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.022+0000 7f45da59c640 1 -- 172.21.15.60:0/3874638468 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4034a50 con 0x7f45c8004d00 2023-10-16T12:49:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45e102d640 1 -- 172.21.15.60:0/3874638468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45c8004d00 msgr2=0x7f45c8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45e102d640 1 --2- 172.21.15.60:0/3874638468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45c8004d00 0x7f45c8005180 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f45d400a760 tx=0x7f45c80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45db59e640 1 -- 172.21.15.60:0/3874638468 reap_dead start 2023-10-16T12:49:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45e102d640 1 -- 172.21.15.60:0/3874638468 shutdown_connections 2023-10-16T12:49:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45e102d640 1 -- 172.21.15.60:0/3874638468 >> 172.21.15.60:0/3874638468 conn(0x7f45dc0ce9b0 msgr2=0x7f45dc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45e102d640 1 -- 172.21.15.60:0/3874638468 shutdown_connections 2023-10-16T12:49:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.023+0000 7f45e102d640 1 -- 172.21.15.60:0/3874638468 wait complete. 2023-10-16T12:49:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.025+0000 7f45e102d640 1 Processor -- start 2023-10-16T12:49:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.025+0000 7f45e102d640 1 -- start start 2023-10-16T12:49:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45e102d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8004d00 0x7f45dc160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45e102d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc164210 0x7f45dc1fabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45e102d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 0x7f45dc0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45e102d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1605a0 con 0x7f45dc0c29f0 2023-10-16T12:49:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45e102d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0b2e40 con 0x7f45c8004d00 2023-10-16T12:49:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45e102d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1fb130 con 0x7f45dc164210 2023-10-16T12:49:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45db59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 0x7f45dc0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45db59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 0x7f45dc0b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38286/0 (socket says 172.21.15.60:38286) 2023-10-16T12:49:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45db59e640 1 -- 172.21.15.60:0/1579676032 learned_addr learned my addr 172.21.15.60:0/1579676032 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45dad9d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8004d00 0x7f45dc160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.026+0000 7f45d3fff640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc164210 0x7f45dc1fabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45db59e640 1 -- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc164210 msgr2=0x7f45dc1fabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45db59e640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc164210 0x7f45dc1fabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45db59e640 1 -- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8004d00 msgr2=0x7f45dc160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45db59e640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8004d00 0x7f45dc160060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45db59e640 1 -- 172.21.15.60:0/1579676032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc162880 con 0x7f45dc0c29f0 2023-10-16T12:49:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45dad9d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8004d00 0x7f45dc160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45d3fff640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc164210 0x7f45dc1fabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45db59e640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 0x7f45dc0b2fd0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f45d4000f80 tx=0x7f45d4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4034840 con 0x7f45dc0c29f0 2023-10-16T12:49:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d40349a0 con 0x7f45dc0c29f0 2023-10-16T12:49:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4034d10 con 0x7f45dc0c29f0 2023-10-16T12:49:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc161c70 con 0x7f45dc0c29f0 2023-10-16T12:49:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.027+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc162230 con 0x7f45dc0c29f0 2023-10-16T12:49:55.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.029+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc162a10 con 0x7f45dc0c29f0 2023-10-16T12:49:55.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.029+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f45d4031030 con 0x7f45dc0c29f0 2023-10-16T12:49:55.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.030+0000 7f45d37fe640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f45bc06afc0 0x7f45bc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.031+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f45d40abb70 con 0x7f45dc0c29f0 2023-10-16T12:49:55.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.031+0000 7f45dad9d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f45bc06afc0 0x7f45bc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.032+0000 7f45dad9d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f45bc06afc0 0x7f45bc06d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009a80 tx=0x7f45cc008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.035+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45d4076b60 con 0x7f45dc0c29f0 2023-10-16T12:49:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.231+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f45dc162a10 con 0x7f45dc0c29f0 2023-10-16T12:49:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.238+0000 7f45d37fe640 1 -- 172.21.15.60:0/1579676032 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f45d407a180 con 0x7f45dc0c29f0 2023-10-16T12:49:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.240+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f45bc06afc0 msgr2=0x7f45bc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.240+0000 7f45e102d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f45bc06afc0 0x7f45bc06d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009a80 tx=0x7f45cc008040 comp rx=0 tx=0).stop 2023-10-16T12:49:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.240+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 msgr2=0x7f45dc0b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.240+0000 7f45e102d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 0x7f45dc0b2fd0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f45d4000f80 tx=0x7f45d4001cb0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 shutdown_connections 2023-10-16T12:49:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f45bc06afc0 0x7f45bc06d480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45dc164210 0x7f45dc1fabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45dc0c29f0 0x7f45dc0b2fd0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 --2- 172.21.15.60:0/1579676032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45c8004d00 0x7f45dc160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 >> 172.21.15.60:0/1579676032 conn(0x7f45dc0ce9b0 msgr2=0x7f45dc070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 shutdown_connections 2023-10-16T12:49:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.241+0000 7f45e102d640 1 -- 172.21.15.60:0/1579676032 wait complete. 2023-10-16T12:49:55.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:49:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.465+0000 7fe2f1135640 1 Processor -- start 2023-10-16T12:49:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.465+0000 7fe2f1135640 1 -- start start 2023-10-16T12:49:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2f1135640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe2ec070b60 0x7fe2ec15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2f1135640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec15aef0 con 0x7fe2ec159950 2023-10-16T12:49:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2f1135640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec15d910 con 0x7fe2ec06c100 2023-10-16T12:49:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2f1135640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2ec15e490 con 0x7fe2ec070b60 2023-10-16T12:49:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2ebfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2ec159950 0x7fe2ec070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50044/0 (socket says 172.21.15.60:50044) 2023-10-16T12:49:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2ebfff640 1 -- 172.21.15.60:0/2631191469 learned_addr learned my addr 172.21.15.60:0/2631191469 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.467+0000 7fe2e3fff640 1 --2- 172.21.15.60:0/2631191469 >> v2:172.21.15.144:3300/0 conn(0x7fe2ec070b60 0x7fe2ec15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515766375 0 0) 0x7fe2dc001110 con 0x7fe2ec159950 2023-10-16T12:49:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d8003520 con 0x7fe2ec159950 2023-10-16T12:49:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2141266963 0 0) 0x7fe2d8003520 con 0x7fe2ec159950 2023-10-16T12:49:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 >> v2:172.21.15.144:3300/0 conn(0x7fe2ec070b60 msgr2=0x7fe2ec15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 --2- 172.21.15.60:0/2631191469 >> v2:172.21.15.144:3300/0 conn(0x7fe2ec070b60 0x7fe2ec15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 >> v1:172.21.15.144:6789/0 conn(0x7fe2ec06c100 legacy=0x7fe2ec06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec15dbf0 con 0x7fe2ec159950 2023-10-16T12:49:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe2dc001800 con 0x7fe2ec159950 2023-10-16T12:49:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 >> v1:172.21.15.60:6789/0 conn(0x7fe2ec159950 legacy=0x7fe2ec070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d8003970 0x7fe2d8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d80047c0 0x7fe2d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d80052a0 0x7fe2d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec15e490 con 0x7fe2d8003970 2023-10-16T12:49:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d800c150 con 0x7fe2d80047c0 2023-10-16T12:49:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d800c310 con 0x7fe2d80052a0 2023-10-16T12:49:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.468+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d8003970 0x7fe2d8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.469+0000 7fe2ebfff640 1 -- 172.21.15.60:0/2631191469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d80052a0 msgr2=0x7fe2d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.469+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d80052a0 0x7fe2d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.469+0000 7fe2ebfff640 1 -- 172.21.15.60:0/2631191469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d80047c0 msgr2=0x7fe2d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.469+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d80047c0 0x7fe2d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.469+0000 7fe2ebfff640 1 -- 172.21.15.60:0/2631191469 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d8003520 con 0x7fe2d8003970 2023-10-16T12:49:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.469+0000 7fe2f0934640 1 -- 172.21.15.60:0/2631191469 reap_dead start 2023-10-16T12:49:55.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.470+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d8003970 0x7fe2d8003d70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc007e50 tx=0x7fe2d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbe6e4f777ff62f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.470+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc005ad0 con 0x7fe2d8003970 2023-10-16T12:49:55.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.470+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc005c50 con 0x7fe2d8003970 2023-10-16T12:49:55.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.470+0000 7fe2eb7fe640 1 -- 172.21.15.60:0/2631191469 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc02c890 con 0x7fe2d8003970 2023-10-16T12:49:55.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 -- 172.21.15.60:0/2631191469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d8003970 msgr2=0x7fe2d8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d8003970 0x7fe2d8003d70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc007e50 tx=0x7fe2d80057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 -- 172.21.15.60:0/2631191469 shutdown_connections 2023-10-16T12:49:55.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 --2- 172.21.15.60:0/2631191469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d8003970 0x7fe2d8003d70 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 -- 172.21.15.60:0/2631191469 >> 172.21.15.60:0/2631191469 conn(0x7fe2ec0ce6a0 msgr2=0x7fe2ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 -- 172.21.15.60:0/2631191469 shutdown_connections 2023-10-16T12:49:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.471+0000 7fe2f1135640 1 -- 172.21.15.60:0/2631191469 wait complete. 2023-10-16T12:49:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.473+0000 7fe2f1135640 1 Processor -- start 2023-10-16T12:49:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.473+0000 7fe2f1135640 1 -- start start 2023-10-16T12:49:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f1135640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d8003970 0x7fe2ec0c2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f1135640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0c3d50 0x7fe2ec0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f1135640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 0x7fe2ec1477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f1135640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0c4e40 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f1135640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0c4230 con 0x7fe2ec0c3d50 2023-10-16T12:49:55.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f1135640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0c4510 con 0x7fe2d8003970 2023-10-16T12:49:55.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f0934640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 0x7fe2ec1477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f0934640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 0x7fe2ec1477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38306/0 (socket says 172.21.15.60:38306) 2023-10-16T12:49:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.474+0000 7fe2f0934640 1 -- 172.21.15.60:0/107564740 learned_addr learned my addr 172.21.15.60:0/107564740 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d8003970 0x7fe2ec0c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2e3fff640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0c3d50 0x7fe2ec0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2f0934640 1 -- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d8003970 msgr2=0x7fe2ec0c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2f0934640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d8003970 0x7fe2ec0c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2f0934640 1 -- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0c3d50 msgr2=0x7fe2ec0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2f0934640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0c3d50 0x7fe2ec0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2f0934640 1 -- 172.21.15.60:0/107564740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0c47f0 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.475+0000 7fe2f0934640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 0x7fe2ec1477a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06f1c0 tx=0x7fe2e4010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.476+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400ea20 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.476+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e400eb80 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.476+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4016400 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.476+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec0c4ad0 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.476+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec0c2870 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.477+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec06db60 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.478+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2e4020440 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.479+0000 7fe2e97fa640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2cc06af90 0x7fe2cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.479+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe2e408cdd0 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.479+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2cc06af90 0x7fe2cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.481+0000 7fe2ebfff640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2cc06af90 0x7fe2cc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc034000 tx=0x7fe2dc009030 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.483+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2e4057e60 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.683+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe2ec0c2870 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.687+0000 7fe2e97fa640 1 -- 172.21.15.60:0/107564740 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe2ec0c2870 con 0x7fe2ec0b3d90 2023-10-16T12:49:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2cc06af90 msgr2=0x7fe2cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2cc06af90 0x7fe2cc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc034000 tx=0x7fe2dc009030 comp rx=0 tx=0).stop 2023-10-16T12:49:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 msgr2=0x7fe2ec1477a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 0x7fe2ec1477a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06f1c0 tx=0x7fe2e4010f00 comp rx=0 tx=0).stop 2023-10-16T12:49:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 shutdown_connections 2023-10-16T12:49:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2cc06af90 0x7fe2cc06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0b3d90 0x7fe2ec1477a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0c3d50 0x7fe2ec0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 --2- 172.21.15.60:0/107564740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2d8003970 0x7fe2ec0c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 >> 172.21.15.60:0/107564740 conn(0x7fe2ec0ce6a0 msgr2=0x7fe2ec0ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 shutdown_connections 2023-10-16T12:49:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.691+0000 7fe2f1135640 1 -- 172.21.15.60:0/107564740 wait complete. 2023-10-16T12:49:55.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T12:49:55.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.937+0000 7fb18ef83640 1 Processor -- start 2023-10-16T12:49:55.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.937+0000 7fb18ef83640 1 -- start start 2023-10-16T12:49:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18ef83640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb18815ace0 0x7fb1880b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18ef83640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb18815b0c0 0x7fb18815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18ef83640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880ce2d0 con 0x7fb18806fa90 2023-10-16T12:49:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18ef83640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1880b63d0 con 0x7fb18815b0c0 2023-10-16T12:49:55.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18ef83640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1880b66b0 con 0x7fb18815ace0 2023-10-16T12:49:55.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb17ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb18815b0c0 0x7fb18815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18d520640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb18806fa90 0x7fb1880cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50048/0 (socket says 172.21.15.60:50048) 2023-10-16T12:49:55.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18d520640 1 -- 172.21.15.60:0/2286597134 learned_addr learned my addr 172.21.15.60:0/2286597134 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:55.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb17ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb18815b0c0 0x7fb18815baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38314/0 (socket says 172.21.15.60:38314) 2023-10-16T12:49:55.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.939+0000 7fb18cd1f640 1 --2- 172.21.15.60:0/2286597134 >> v2:172.21.15.144:3300/0 conn(0x7fb18815ace0 0x7fb1880b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.940+0000 7fb17ffff640 1 -- 172.21.15.60:0/2286597134 >> v2:172.21.15.144:3300/0 conn(0x7fb18815ace0 msgr2=0x7fb1880b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:55.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.940+0000 7fb17ffff640 1 --2- 172.21.15.60:0/2286597134 >> v2:172.21.15.144:3300/0 conn(0x7fb18815ace0 0x7fb1880b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.940+0000 7fb17ffff640 1 -- 172.21.15.60:0/2286597134 >> v1:172.21.15.60:6789/0 conn(0x7fb18806fa90 legacy=0x7fb1880cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.940+0000 7fb17ffff640 1 -- 172.21.15.60:0/2286597134 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880b6990 con 0x7fb18815b0c0 2023-10-16T12:49:55.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.941+0000 7fb17ffff640 1 --2- 172.21.15.60:0/2286597134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb18815baa0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb1880b3630 tx=0x7fb17802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46ccae4534859d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.941+0000 7fb17f7fe640 1 -- 172.21.15.60:0/2286597134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17802e070 con 0x7fb18815b0c0 2023-10-16T12:49:55.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.941+0000 7fb17f7fe640 1 -- 172.21.15.60:0/2286597134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1780315a0 con 0x7fb18815b0c0 2023-10-16T12:49:55.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb17f7fe640 1 -- 172.21.15.60:0/2286597134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178031930 con 0x7fb18815b0c0 2023-10-16T12:49:55.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb18ef83640 1 -- 172.21.15.60:0/2286597134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 msgr2=0x7fb18815baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb18ef83640 1 --2- 172.21.15.60:0/2286597134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb18815baa0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fb1880b3630 tx=0x7fb17802b840 comp rx=0 tx=0).stop 2023-10-16T12:49:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb18ef83640 1 -- 172.21.15.60:0/2286597134 shutdown_connections 2023-10-16T12:49:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb18ef83640 1 --2- 172.21.15.60:0/2286597134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb18815baa0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb18ef83640 1 --2- 172.21.15.60:0/2286597134 >> v2:172.21.15.144:3300/0 conn(0x7fb18815ace0 0x7fb1880b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.942+0000 7fb18ef83640 1 -- 172.21.15.60:0/2286597134 >> 172.21.15.60:0/2286597134 conn(0x7fb18806d700 msgr2=0x7fb18806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.943+0000 7fb18ef83640 1 -- 172.21.15.60:0/2286597134 shutdown_connections 2023-10-16T12:49:55.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.943+0000 7fb18ef83640 1 -- 172.21.15.60:0/2286597134 wait complete. 2023-10-16T12:49:55.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.945+0000 7fb18ef83640 1 Processor -- start 2023-10-16T12:49:55.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.945+0000 7fb18ef83640 1 -- start start 2023-10-16T12:49:55.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18ef83640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb18806fa90 0x7fb1880c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18ef83640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18815ace0 0x7fb1880c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18ef83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb1880c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18ef83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880a8480 con 0x7fb18815b0c0 2023-10-16T12:49:55.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18ef83640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880c87f0 con 0x7fb18815ace0 2023-10-16T12:49:55.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18ef83640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1880c8ad0 con 0x7fb18806fa90 2023-10-16T12:49:55.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18d520640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb1880c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18d520640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb1880c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38320/0 (socket says 172.21.15.60:38320) 2023-10-16T12:49:55.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.946+0000 7fb18d520640 1 -- 172.21.15.60:0/3409849822 learned_addr learned my addr 172.21.15.60:0/3409849822 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:55.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb17ffff640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18815ace0 0x7fb1880c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18cd1f640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb18806fa90 0x7fb1880c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18d520640 1 -- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb18806fa90 msgr2=0x7fb1880c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18d520640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb18806fa90 0x7fb1880c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18d520640 1 -- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18815ace0 msgr2=0x7fb1880c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:55.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18d520640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18815ace0 0x7fb1880c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:55.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18d520640 1 -- 172.21.15.60:0/3409849822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178009400 con 0x7fb18815b0c0 2023-10-16T12:49:55.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb17ffff640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18815ace0 0x7fb1880c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:55.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18d520640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb1880c80e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb18000c420 tx=0x7fb18000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180005ce0 con 0x7fb18815b0c0 2023-10-16T12:49:55.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb180005e40 con 0x7fb18815b0c0 2023-10-16T12:49:55.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1800072d0 con 0x7fb18815b0c0 2023-10-16T12:49:55.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.947+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1880c8db0 con 0x7fb18815b0c0 2023-10-16T12:49:55.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.948+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1880c4680 con 0x7fb18815b0c0 2023-10-16T12:49:55.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.949+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1880c4870 con 0x7fb18815b0c0 2023-10-16T12:49:55.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.950+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb1800074f0 con 0x7fb18815b0c0 2023-10-16T12:49:55.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.950+0000 7fb17d7fa640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fb17406af90 0x7fb17406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:55.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.951+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fb180088cb0 con 0x7fb18815b0c0 2023-10-16T12:49:55.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.951+0000 7fb18cd1f640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fb17406af90 0x7fb17406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:55.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.952+0000 7fb18cd1f640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fb17406af90 0x7fb17406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb184009f30 tx=0x7fb18400b040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:55.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:55.955+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb180053c20 con 0x7fb18815b0c0 2023-10-16T12:49:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.222+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fb18804e160 con 0x7fb18815b0c0 2023-10-16T12:49:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.320+0000 7fb17d7fa640 1 -- 172.21.15.60:0/3409849822 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v606) v1 ==== 156+0+0 (secure 0 0 0) 0x7fb180017070 con 0x7fb18815b0c0 2023-10-16T12:49:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fb17406af90 msgr2=0x7fb17406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fb17406af90 0x7fb17406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb184009f30 tx=0x7fb18400b040 comp rx=0 tx=0).stop 2023-10-16T12:49:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 msgr2=0x7fb1880c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb1880c80e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb18000c420 tx=0x7fb18000c8e0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 shutdown_connections 2023-10-16T12:49:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fb17406af90 0x7fb17406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18815b0c0 0x7fb1880c80e0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18815ace0 0x7fb1880c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 --2- 172.21.15.60:0/3409849822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb18806fa90 0x7fb1880c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 >> 172.21.15.60:0/3409849822 conn(0x7fb18806d700 msgr2=0x7fb1880cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 shutdown_connections 2023-10-16T12:49:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.323+0000 7fb18ef83640 1 -- 172.21.15.60:0/3409849822 wait complete. 2023-10-16T12:49:56.347 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-16T12:49:56.348 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-16T12:49:56.348 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:49:56.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:49:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.570+0000 7f9e75640640 1 Processor -- start 2023-10-16T12:49:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.570+0000 7f9e75640640 1 -- start start 2023-10-16T12:49:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.571+0000 7f9e75640640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e700cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.571+0000 7f9e75640640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9e700ccfb0 0x7f9e700cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.571+0000 7f9e75640640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7015b9d0 con 0x7f9e7006c040 2023-10-16T12:49:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.571+0000 7f9e75640640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7015bcb0 con 0x7f9e700ccfb0 2023-10-16T12:49:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.571+0000 7f9e75640640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015bf90 con 0x7f9e700ad970 2023-10-16T12:49:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.572+0000 7f9e6ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9e7006c040 0x7f9e7006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50062/0 (socket says 172.21.15.60:50062) 2023-10-16T12:49:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.572+0000 7f9e6ffff640 1 -- 172.21.15.60:0/1973271215 learned_addr learned my addr 172.21.15.60:0/1973271215 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.572+0000 7f9e67fff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e700cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.572+0000 7f9e74e3f640 1 --2- 172.21.15.60:0/1973271215 >> v2:172.21.15.144:3300/0 conn(0x7f9e700ccfb0 0x7f9e700cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.572+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482638500 0 0) 0x7f9e7015b9d0 con 0x7f9e7006c040 2023-10-16T12:49:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.572+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c003520 con 0x7f9e7006c040 2023-10-16T12:49:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005165170 0 0) 0x7f9e5c003520 con 0x7f9e7006c040 2023-10-16T12:49:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 msgr2=0x7f9e700cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e700cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 >> v2:172.21.15.144:3300/0 conn(0x7f9e700ccfb0 msgr2=0x7f9e700cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 --2- 172.21.15.60:0/1973271215 >> v2:172.21.15.144:3300/0 conn(0x7f9e700ccfb0 0x7f9e700cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700cc270 con 0x7f9e7006c040 2023-10-16T12:49:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9e60001670 con 0x7f9e7006c040 2023-10-16T12:49:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e67fff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e700cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:49:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 >> v1:172.21.15.60:6789/0 conn(0x7f9e7006c040 legacy=0x7f9e7006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e5c003970 0x7f9e5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c004d00 0x7f9e5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015bcb0 con 0x7f9e5c003970 2023-10-16T12:49:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7015bf90 con 0x7f9e5c004d00 2023-10-16T12:49:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c00b9b0 con 0x7f9e700ad970 2023-10-16T12:49:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e5c003970 0x7f9e5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.573+0000 7f9e67fff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e74e3f640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c004d00 0x7f9e5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e6ffff640 1 -- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 msgr2=0x7f9e5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700ad970 0x7f9e5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e6ffff640 1 -- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c004d00 msgr2=0x7f9e5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c004d00 0x7f9e5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e6ffff640 1 -- 172.21.15.60:0/1973271215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c003520 con 0x7f9e5c003970 2023-10-16T12:49:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.574+0000 7f9e74e3f640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c004d00 0x7f9e5c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.575+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e5c003970 0x7f9e5c003d90 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c240 tx=0x7f9e5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f968b8aeb16852 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.575+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60005ad0 con 0x7f9e5c003970 2023-10-16T12:49:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.575+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60005c30 con 0x7f9e5c003970 2023-10-16T12:49:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.575+0000 7f9e6f7fe640 1 -- 172.21.15.60:0/1973271215 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6002c8b0 con 0x7f9e5c003970 2023-10-16T12:49:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e75640640 1 -- 172.21.15.60:0/1973271215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e5c003970 msgr2=0x7f9e5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e75640640 1 --2- 172.21.15.60:0/1973271215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e5c003970 0x7f9e5c003d90 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c240 tx=0x7f9e5c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e74e3f640 1 -- 172.21.15.60:0/1973271215 reap_dead start 2023-10-16T12:49:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e75640640 1 -- 172.21.15.60:0/1973271215 shutdown_connections 2023-10-16T12:49:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e75640640 1 -- 172.21.15.60:0/1973271215 >> 172.21.15.60:0/1973271215 conn(0x7f9e700ce6a0 msgr2=0x7f9e700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e75640640 1 -- 172.21.15.60:0/1973271215 shutdown_connections 2023-10-16T12:49:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.576+0000 7f9e75640640 1 -- 172.21.15.60:0/1973271215 wait complete. 2023-10-16T12:49:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.578+0000 7f9e75640640 1 Processor -- start 2023-10-16T12:49:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.578+0000 7f9e75640640 1 -- start start 2023-10-16T12:49:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e75640640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c003970 0x7f9e70153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e75640640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 0x7f9e700c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e75640640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700c7900 0x7f9e700c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e75640640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700c63e0 con 0x7f9e700b37c0 2023-10-16T12:49:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e75640640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700c6870 con 0x7f9e5c003970 2023-10-16T12:49:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e75640640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e700c5810 con 0x7f9e700c7900 2023-10-16T12:49:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e67fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 0x7f9e700c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e67fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 0x7f9e700c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38350/0 (socket says 172.21.15.60:38350) 2023-10-16T12:49:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.579+0000 7f9e67fff640 1 -- 172.21.15.60:0/1938487340 learned_addr learned my addr 172.21.15.60:0/1938487340 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c003970 0x7f9e70153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e74e3f640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700c7900 0x7f9e700c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e67fff640 1 -- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700c7900 msgr2=0x7f9e700c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e67fff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700c7900 0x7f9e700c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e67fff640 1 -- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c003970 msgr2=0x7f9e70153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e67fff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c003970 0x7f9e70153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e67fff640 1 -- 172.21.15.60:0/1938487340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e700c5ab0 con 0x7f9e700b37c0 2023-10-16T12:49:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e74e3f640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700c7900 0x7f9e700c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c003970 0x7f9e70153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e67fff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 0x7f9e700c73c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9e58010890 tx=0x7f9e58010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5800cb40 con 0x7f9e700b37c0 2023-10-16T12:49:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5800cca0 con 0x7f9e700b37c0 2023-10-16T12:49:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e700c5d90 con 0x7f9e700b37c0 2023-10-16T12:49:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e580156b0 con 0x7f9e700b37c0 2023-10-16T12:49:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.580+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e700ca280 con 0x7f9e700b37c0 2023-10-16T12:49:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.582+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e700c6550 con 0x7f9e700b37c0 2023-10-16T12:49:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.583+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9e58015810 con 0x7f9e700b37c0 2023-10-16T12:49:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.584+0000 7f9e6d7fa640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9e5006afc0 0x7f9e5006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.584+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f9e58089f20 con 0x7f9e700b37c0 2023-10-16T12:49:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.584+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9e5006afc0 0x7f9e5006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.586+0000 7f9e6ffff640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9e5006afc0 0x7f9e5006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e60034000 tx=0x7f9e60009030 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.588+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e58054e00 con 0x7f9e700b37c0 2023-10-16T12:49:56.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:56.662 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:56.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.787+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e700cc520 con 0x7f9e700b37c0 2023-10-16T12:49:56.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.789+0000 7f9e6d7fa640 1 -- 172.21.15.60:0/1938487340 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13992 (secure 0 0 0) 0x7f9e58058420 con 0x7f9e700b37c0 2023-10-16T12:49:56.792 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:56.792 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":91,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:49:54.296663+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6833/2404598903":"2023-10-17T12:49:51.279471+0000","172.21.15.144:0/1428848083":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2802966120":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2857824565":"2023-10-17T12:49:47.079819+0000","172.21.15.144:6832/2100131827":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/954403568":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2870481770":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/1738450860":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/3154142782":"2023-10-17T12:49:40.910583+0000","172.21.15.144:0/2938654985":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2412969789":"2023-10-17T12:49:40.910583+0000","172.21.15.60:6835/2404598903":"2023-10-17T12:49:51.279471+0000","172.21.15.144:0/3554060781":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/2725919540":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2100131827":"2023-10-17T12:49:47.079819+0000","172.21.15.144:0/342775588":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6833/2872949001":"2023-10-17T12:49:40.910583+0000","172.21.15.144:6832/2872949001":"2023-10-17T12:49:40.910583+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:49:56.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9e5006afc0 msgr2=0x7f9e5006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9e5006afc0 0x7f9e5006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e60034000 tx=0x7f9e60009030 comp rx=0 tx=0).stop 2023-10-16T12:49:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 msgr2=0x7f9e700c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 0x7f9e700c73c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9e58010890 tx=0x7f9e58010d50 comp rx=0 tx=0).stop 2023-10-16T12:49:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 shutdown_connections 2023-10-16T12:49:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9e5006afc0 0x7f9e5006d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e700c7900 0x7f9e700c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e700b37c0 0x7f9e700c73c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 --2- 172.21.15.60:0/1938487340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e5c003970 0x7f9e70153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 >> 172.21.15.60:0/1938487340 conn(0x7f9e700ce6a0 msgr2=0x7f9e70159da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 shutdown_connections 2023-10-16T12:49:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:56.792+0000 7f9e75640640 1 -- 172.21.15.60:0/1938487340 wait complete. 2023-10-16T12:49:56.812 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:49:56.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:49:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.084+0000 7f5dc5d61640 1 Processor -- start 2023-10-16T12:49:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.084+0000 7f5dc5d61640 1 -- start start 2023-10-16T12:49:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dc5d61640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5dc015ad70 0x7f5dc0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dc5d61640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00cb030 con 0x7f5dc00c9ca0 2023-10-16T12:49:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dc5d61640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00c91c0 con 0x7f5dc00caac0 2023-10-16T12:49:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dc5d61640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc00c94a0 con 0x7f5dc015ad70 2023-10-16T12:49:57.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dbeffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5dc00c9ca0 0x7f5dc00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50074/0 (socket says 172.21.15.60:50074) 2023-10-16T12:49:57.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dbeffd640 1 -- 172.21.15.60:0/1031282762 learned_addr learned my addr 172.21.15.60:0/1031282762 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.086+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/1031282762 >> v2:172.21.15.144:3300/0 conn(0x7f5dc015ad70 0x7f5dc0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430120360 0 0) 0x7f5dc00cb030 con 0x7f5dc00c9ca0 2023-10-16T12:49:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac003520 con 0x7f5dc00c9ca0 2023-10-16T12:49:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162066092 0 0) 0x7f5dac003520 con 0x7f5dc00c9ca0 2023-10-16T12:49:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 >> v2:172.21.15.144:3300/0 conn(0x7f5dc015ad70 msgr2=0x7f5dc0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 --2- 172.21.15.60:0/1031282762 >> v2:172.21.15.144:3300/0 conn(0x7f5dc015ad70 0x7f5dc0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 >> v1:172.21.15.144:6789/0 conn(0x7f5dc00caac0 legacy=0x7f5dc00ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00c9780 con 0x7f5dc00c9ca0 2023-10-16T12:49:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5da8001670 con 0x7f5dc00c9ca0 2023-10-16T12:49:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 >> v1:172.21.15.60:6789/0 conn(0x7f5dc00c9ca0 legacy=0x7f5dc00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dac003970 0x7f5dac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac0047c0 0x7f5dac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac005180 0x7f5dac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00c94a0 con 0x7f5dac003970 2023-10-16T12:49:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac00c020 con 0x7f5dac0047c0 2023-10-16T12:49:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac00c1e0 con 0x7f5dac005180 2023-10-16T12:49:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.087+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dac003970 0x7f5dac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbeffd640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac0047c0 0x7f5dac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbf7fe640 1 -- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac005180 msgr2=0x7f5dac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac005180 0x7f5dac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbf7fe640 1 -- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac0047c0 msgr2=0x7f5dac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dac0047c0 0x7f5dac004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbf7fe640 1 -- 172.21.15.60:0/1031282762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac003520 con 0x7f5dac003970 2023-10-16T12:49:57.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.088+0000 7f5dbffff640 1 -- 172.21.15.60:0/1031282762 reap_dead start 2023-10-16T12:49:57.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.089+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dac003970 0x7f5dac003d90 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009ac0 tx=0x7f5dac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65582a39e9195151 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:57.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.089+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db002c710 con 0x7f5dac003970 2023-10-16T12:49:57.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.089+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db002c870 con 0x7f5dac003970 2023-10-16T12:49:57.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.089+0000 7f5dbe7fc640 1 -- 172.21.15.60:0/1031282762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db002cc00 con 0x7f5dac003970 2023-10-16T12:49:57.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 -- 172.21.15.60:0/1031282762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dac003970 msgr2=0x7f5dac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dac003970 0x7f5dac003d90 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009ac0 tx=0x7f5dac0057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 -- 172.21.15.60:0/1031282762 shutdown_connections 2023-10-16T12:49:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/1031282762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dac003970 0x7f5dac003d90 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 -- 172.21.15.60:0/1031282762 >> 172.21.15.60:0/1031282762 conn(0x7f5dc00ce9b0 msgr2=0x7f5dc00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 -- 172.21.15.60:0/1031282762 shutdown_connections 2023-10-16T12:49:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.090+0000 7f5dc5d61640 1 -- 172.21.15.60:0/1031282762 wait complete. 2023-10-16T12:49:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.092+0000 7f5dc5d61640 1 Processor -- start 2023-10-16T12:49:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.092+0000 7f5dc5d61640 1 -- start start 2023-10-16T12:49:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dc5d61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac003970 0x7f5dc014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dc5d61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 0x7f5dc00c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dc5d61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dc00c24c0 0x7f5dc00c2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dc5d61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00c2e80 con 0x7f5dc00c8570 2023-10-16T12:49:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dc5d61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00c7aa0 con 0x7f5dc00c24c0 2023-10-16T12:49:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dc5d61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0150250 con 0x7f5dac003970 2023-10-16T12:49:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dbeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 0x7f5dc00c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dbeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 0x7f5dc00c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38376/0 (socket says 172.21.15.60:38376) 2023-10-16T12:49:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.093+0000 7f5dbeffd640 1 -- 172.21.15.60:0/2360386353 learned_addr learned my addr 172.21.15.60:0/2360386353 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dbeffd640 1 -- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac003970 msgr2=0x7f5dc014fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dbeffd640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac003970 0x7f5dc014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dbeffd640 1 -- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dc00c24c0 msgr2=0x7f5dc00c2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dbeffd640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dc00c24c0 0x7f5dc00c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dbeffd640 1 -- 172.21.15.60:0/2360386353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00c6e90 con 0x7f5dc00c8570 2023-10-16T12:49:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dbeffd640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 0x7f5dc00c89f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5da800d770 tx=0x7f5da800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da80074e0 con 0x7f5dc00c8570 2023-10-16T12:49:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da8007640 con 0x7f5dc00c8570 2023-10-16T12:49:57.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da8004a20 con 0x7f5dc00c8570 2023-10-16T12:49:57.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.094+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc00c7170 con 0x7f5dc00c8570 2023-10-16T12:49:57.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.095+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc00c7730 con 0x7f5dc00c8570 2023-10-16T12:49:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.096+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc004e160 con 0x7f5dc00c8570 2023-10-16T12:49:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.097+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5da8004b80 con 0x7f5dc00c8570 2023-10-16T12:49:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.098+0000 7f5db7fff640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5da006af90 0x7f5da006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.098+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f5da80890c0 con 0x7f5dc00c8570 2023-10-16T12:49:57.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.098+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5da006af90 0x7f5da006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.100+0000 7f5dbf7fe640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5da006af90 0x7f5da006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5db0036000 tx=0x7f5db002e000 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:57.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.102+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5da8053fd0 con 0x7f5dc00c8570 2023-10-16T12:49:57.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.299+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5dc00c4c00 con 0x7f5dc00c8570 2023-10-16T12:49:57.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.301+0000 7f5db7fff640 1 -- 172.21.15.60:0/2360386353 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5da80575f0 con 0x7f5dc00c8570 2023-10-16T12:49:57.303 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:49:57.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.303+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5da006af90 msgr2=0x7f5da006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.303+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5da006af90 0x7f5da006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5db0036000 tx=0x7f5db002e000 comp rx=0 tx=0).stop 2023-10-16T12:49:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.303+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 msgr2=0x7f5dc00c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.303+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 0x7f5dc00c89f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5da800d770 tx=0x7f5da800dc30 comp rx=0 tx=0).stop 2023-10-16T12:49:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 shutdown_connections 2023-10-16T12:49:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f5da006af90 0x7f5da006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dc00c24c0 0x7f5dc00c2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dc00c8570 0x7f5dc00c89f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 --2- 172.21.15.60:0/2360386353 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5dac003970 0x7f5dc014fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 >> 172.21.15.60:0/2360386353 conn(0x7f5dc00ce9b0 msgr2=0x7f5dc015d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 shutdown_connections 2023-10-16T12:49:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.304+0000 7f5dc5d61640 1 -- 172.21.15.60:0/2360386353 wait complete. 2023-10-16T12:49:57.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T12:49:57.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.550+0000 7fef05d47640 1 Processor -- start 2023-10-16T12:49:57.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.550+0000 7fef05d47640 1 -- start start 2023-10-16T12:49:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.551+0000 7fef05d47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef0015cbe0 0x7fef000d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.551+0000 7fef05d47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 0x7fef000b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.551+0000 7fef05d47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef000d15a0 con 0x7fef0015cfc0 2023-10-16T12:49:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.551+0000 7fef05d47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef000d1880 con 0x7fef0015cbe0 2023-10-16T12:49:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.551+0000 7fef05d47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef000d1b60 con 0x7fef000b4c80 2023-10-16T12:49:57.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.552+0000 7feefffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fef0015cfc0 0x7fef000b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50088/0 (socket says 172.21.15.60:50088) 2023-10-16T12:49:57.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.552+0000 7feefffff640 1 -- 172.21.15.60:0/2662689821 learned_addr learned my addr 172.21.15.60:0/2662689821 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:49:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.552+0000 7fef04d45640 1 --2- 172.21.15.60:0/2662689821 >> v2:172.21.15.144:3300/0 conn(0x7fef0015cbe0 0x7fef000d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.552+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173459100 0 0) 0x7fef000d15a0 con 0x7fef0015cfc0 2023-10-16T12:49:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.552+0000 7fef05546640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 0x7fef000b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.552+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef0003520 con 0x7fef0015cfc0 2023-10-16T12:49:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034450318 0 0) 0x7feef0003520 con 0x7fef0015cfc0 2023-10-16T12:49:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 msgr2=0x7fef000b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 0x7fef000b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 >> v2:172.21.15.144:3300/0 conn(0x7fef0015cbe0 msgr2=0x7fef000d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 --2- 172.21.15.60:0/2662689821 >> v2:172.21.15.144:3300/0 conn(0x7fef0015cbe0 0x7fef000d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef000d1e40 con 0x7fef0015cfc0 2023-10-16T12:49:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7feeec001670 con 0x7fef0015cfc0 2023-10-16T12:49:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 >> v1:172.21.15.60:6789/0 conn(0x7fef0015cfc0 legacy=0x7fef000b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef0003970 0x7feef0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 0x7feef00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef0004d00 0x7feef0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000d1880 con 0x7feef0004d00 2023-10-16T12:49:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000d1b60 con 0x7feef0003970 2023-10-16T12:49:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef000b9b0 con 0x7fef000b4c80 2023-10-16T12:49:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.553+0000 7fef05546640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef0004d00 0x7feef0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7fef04d45640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef0003970 0x7feef0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7feefffff640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 0x7feef00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7fef05546640 1 -- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 msgr2=0x7feef00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7fef05546640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef000b4c80 0x7feef00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7fef05546640 1 -- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef0003970 msgr2=0x7feef0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7fef05546640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef0003970 0x7feef0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.554+0000 7fef05546640 1 -- 172.21.15.60:0/2662689821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0003520 con 0x7feef0004d00 2023-10-16T12:49:57.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.555+0000 7fef05546640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef0004d00 0x7feef0005180 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7feef8009fc0 tx=0x7feef00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f22dd5d9782943 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:57.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.555+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef802cc20 con 0x7feef0004d00 2023-10-16T12:49:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.555+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef800f040 con 0x7feef0004d00 2023-10-16T12:49:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.555+0000 7feeff7fe640 1 -- 172.21.15.60:0/2662689821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8034440 con 0x7feef0004d00 2023-10-16T12:49:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05d47640 1 -- 172.21.15.60:0/2662689821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef0004d00 msgr2=0x7feef0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05d47640 1 --2- 172.21.15.60:0/2662689821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef0004d00 0x7feef0005180 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7feef8009fc0 tx=0x7feef00057f0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05546640 1 -- 172.21.15.60:0/2662689821 reap_dead start 2023-10-16T12:49:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05d47640 1 -- 172.21.15.60:0/2662689821 shutdown_connections 2023-10-16T12:49:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05d47640 1 -- 172.21.15.60:0/2662689821 >> 172.21.15.60:0/2662689821 conn(0x7fef000cf570 msgr2=0x7fef000cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05d47640 1 -- 172.21.15.60:0/2662689821 shutdown_connections 2023-10-16T12:49:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.556+0000 7fef05d47640 1 -- 172.21.15.60:0/2662689821 wait complete. 2023-10-16T12:49:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.558+0000 7fef05d47640 1 Processor -- start 2023-10-16T12:49:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.558+0000 7fef05d47640 1 -- start start 2023-10-16T12:49:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05d47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef000c8dc0 0x7fef00152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05d47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef00151910 0x7fef00152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05d47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 0x7fef00154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05d47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0015fd60 con 0x7fef00151e60 2023-10-16T12:49:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05d47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0015f150 con 0x7fef000c8dc0 2023-10-16T12:49:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05d47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0015f430 con 0x7fef00151910 2023-10-16T12:49:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05546640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 0x7fef00154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05546640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 0x7fef00154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38398/0 (socket says 172.21.15.60:38398) 2023-10-16T12:49:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.559+0000 7fef05546640 1 -- 172.21.15.60:0/3893179758 learned_addr learned my addr 172.21.15.60:0/3893179758 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef04d45640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef000c8dc0 0x7fef00152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7feefffff640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef00151910 0x7fef00152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef05546640 1 -- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef00151910 msgr2=0x7fef00152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef05546640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef00151910 0x7fef00152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef05546640 1 -- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef000c8dc0 msgr2=0x7fef00152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef05546640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef000c8dc0 0x7fef00152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef05546640 1 -- 172.21.15.60:0/3893179758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0015f710 con 0x7fef00151e60 2023-10-16T12:49:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7fef05546640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 0x7fef00154c30 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7feef8037040 tx=0x7feef8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8046070 con 0x7fef00151e60 2023-10-16T12:49:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.560+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef8034ad0 con 0x7fef00151e60 2023-10-16T12:49:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.561+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0015f9f0 con 0x7fef00151e60 2023-10-16T12:49:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.561+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8034e40 con 0x7fef00151e60 2023-10-16T12:49:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.561+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef000c91a0 con 0x7fef00151e60 2023-10-16T12:49:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.562+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0015fbb0 con 0x7fef00151e60 2023-10-16T12:49:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.563+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feef803b050 con 0x7fef00151e60 2023-10-16T12:49:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.564+0000 7feefd7fa640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7feee406af90 0x7feee406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.564+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7feef80aaa60 con 0x7fef00151e60 2023-10-16T12:49:57.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.565+0000 7fef04d45640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7feee406af90 0x7feee406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:57.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.566+0000 7fef04d45640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7feee406af90 0x7feee406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feef4009a90 tx=0x7feef4008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:57.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.569+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feef8075c20 con 0x7fef00151e60 2023-10-16T12:49:57.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:57.837+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fef0004e9b0 con 0x7fef00151e60 2023-10-16T12:49:58.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.335+0000 7feefd7fa640 1 -- 172.21.15.60:0/3893179758 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v608) v1 ==== 169+0+0 (secure 0 0 0) 0x7feef8079240 con 0x7fef00151e60 2023-10-16T12:49:58.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.338+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7feee406af90 msgr2=0x7feee406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.338+0000 7fef05d47640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7feee406af90 0x7feee406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feef4009a90 tx=0x7feef4008040 comp rx=0 tx=0).stop 2023-10-16T12:49:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 msgr2=0x7fef00154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 0x7fef00154c30 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7feef8037040 tx=0x7feef8009750 comp rx=0 tx=0).stop 2023-10-16T12:49:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 shutdown_connections 2023-10-16T12:49:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7feee406af90 0x7feee406d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef00151910 0x7fef00152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef00151e60 0x7fef00154c30 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 --2- 172.21.15.60:0/3893179758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef000c8dc0 0x7fef00152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 >> 172.21.15.60:0/3893179758 conn(0x7fef000cf570 msgr2=0x7fef000b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 shutdown_connections 2023-10-16T12:49:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.339+0000 7fef05d47640 1 -- 172.21.15.60:0/3893179758 wait complete. 2023-10-16T12:49:58.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:49:58.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.585+0000 7f808251d640 1 Processor -- start 2023-10-16T12:49:58.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.585+0000 7f808251d640 1 -- start start 2023-10-16T12:49:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.586+0000 7f808251d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c160ad0 0x7f807c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.586+0000 7f808251d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f807c0ae260 0x7f807c160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f808251d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c161900 con 0x7f807c0ccd80 2023-10-16T12:49:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f808251d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c06b790 con 0x7f807c0ae260 2023-10-16T12:49:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f808251d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0acdd0 con 0x7f807c160ad0 2023-10-16T12:49:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f807c0ae260 0x7f807c160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f807c0ae260 0x7f807c160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38410/0 (socket says 172.21.15.60:38410) 2023-10-16T12:49:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 -- 172.21.15.60:0/1796612321 learned_addr learned my addr 172.21.15.60:0/1796612321 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807bfff640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c160ad0 0x7f807c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 -- 172.21.15.60:0/1796612321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c160ad0 msgr2=0x7f807c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c160ad0 0x7f807c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 -- 172.21.15.60:0/1796612321 >> v1:172.21.15.144:6789/0 conn(0x7f807c0ccd80 legacy=0x7f807c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807b7fe640 1 -- 172.21.15.60:0/1796612321 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c1613f0 con 0x7f807c0ae260 2023-10-16T12:49:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.587+0000 7f807bfff640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c160ad0 0x7f807c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:58.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.589+0000 7f807b7fe640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c160eb0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8064009870 tx=0x7f806402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd9e05fd2ccb022 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:58.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.589+0000 7f807affd640 1 -- 172.21.15.60:0/1796612321 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064031640 con 0x7f807c0ae260 2023-10-16T12:49:58.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.589+0000 7f807affd640 1 -- 172.21.15.60:0/1796612321 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80640317a0 con 0x7f807c0ae260 2023-10-16T12:49:58.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.589+0000 7f807affd640 1 -- 172.21.15.60:0/1796612321 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064031b30 con 0x7f807c0ae260 2023-10-16T12:49:58.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 -- 172.21.15.60:0/1796612321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 msgr2=0x7f807c160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c160eb0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8064009870 tx=0x7f806402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 -- 172.21.15.60:0/1796612321 shutdown_connections 2023-10-16T12:49:58.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c160eb0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 --2- 172.21.15.60:0/1796612321 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c160ad0 0x7f807c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 -- 172.21.15.60:0/1796612321 >> 172.21.15.60:0/1796612321 conn(0x7f807c06bc10 msgr2=0x7f807c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:58.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 -- 172.21.15.60:0/1796612321 shutdown_connections 2023-10-16T12:49:58.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.590+0000 7f808251d640 1 -- 172.21.15.60:0/1796612321 wait complete. 2023-10-16T12:49:58.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.591+0000 7f808251d640 1 Processor -- start 2023-10-16T12:49:58.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.592+0000 7f808251d640 1 -- start start 2023-10-16T12:49:58.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.592+0000 7f808251d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f808251d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c0ccd80 0x7f807c0b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f808251d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c160ad0 0x7f807c0c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f808251d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c158bb0 con 0x7f807c0ae260 2023-10-16T12:49:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f808251d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0b3240 con 0x7f807c160ad0 2023-10-16T12:49:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f808251d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0b3520 con 0x7f807c0ccd80 2023-10-16T12:49:58.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:58.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38418/0 (socket says 172.21.15.60:38418) 2023-10-16T12:49:58.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 -- 172.21.15.60:0/290209940 learned_addr learned my addr 172.21.15.60:0/290209940 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:58.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f8080aba640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c160ad0 0x7f807c0c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807b7fe640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c0ccd80 0x7f807c0b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 -- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c0ccd80 msgr2=0x7f807c0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c0ccd80 0x7f807c0b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 -- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c160ad0 msgr2=0x7f807c0c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c160ad0 0x7f807c0c4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807bfff640 1 -- 172.21.15.60:0/290209940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640094d0 con 0x7f807c0ae260 2023-10-16T12:49:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f807b7fe640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c0ccd80 0x7f807c0b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.593+0000 7f8080aba640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c160ad0 0x7f807c0c4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:49:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.594+0000 7f807bfff640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c158670 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f806c00cf60 tx=0x7f806c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.594+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c00d840 con 0x7f807c0ae260 2023-10-16T12:49:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.594+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c00d9a0 con 0x7f807c0ae260 2023-10-16T12:49:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.594+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c00dd30 con 0x7f807c0ae260 2023-10-16T12:49:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.594+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c0b3800 con 0x7f807c0ae260 2023-10-16T12:49:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.594+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c15df90 con 0x7f807c0ae260 2023-10-16T12:49:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.595+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c04e160 con 0x7f807c0ae260 2023-10-16T12:49:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.596+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f806c015440 con 0x7f807c0ae260 2023-10-16T12:49:58.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.597+0000 7f8078ff9640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f805c06af90 0x7f805c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:58.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.597+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f806c08a0e0 con 0x7f807c0ae260 2023-10-16T12:49:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.598+0000 7f807b7fe640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f805c06af90 0x7f805c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.599+0000 7f807b7fe640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f805c06af90 0x7f805c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8064036000 tx=0x7f806403c040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.602+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f806c055170 con 0x7f807c0ae260 2023-10-16T12:49:58.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.807+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f807c158d50 con 0x7f807c0ae260 2023-10-16T12:49:58.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.812+0000 7f8078ff9640 1 -- 172.21.15.60:0/290209940 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f806c058790 con 0x7f807c0ae260 2023-10-16T12:49:58.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f805c06af90 msgr2=0x7f805c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f805c06af90 0x7f805c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8064036000 tx=0x7f806403c040 comp rx=0 tx=0).stop 2023-10-16T12:49:58.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 msgr2=0x7f807c158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:58.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c158670 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f806c00cf60 tx=0x7f806c00cf90 comp rx=0 tx=0).stop 2023-10-16T12:49:58.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 shutdown_connections 2023-10-16T12:49:58.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f805c06af90 0x7f805c06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f807c0ccd80 0x7f807c0b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c160ad0 0x7f807c0c4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 --2- 172.21.15.60:0/290209940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f807c0ae260 0x7f807c158670 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:58.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 >> 172.21.15.60:0/290209940 conn(0x7f807c06bc10 msgr2=0x7f807c0cd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:58.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 shutdown_connections 2023-10-16T12:49:58.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:58.815+0000 7f808251d640 1 -- 172.21.15.60:0/290209940 wait complete. 2023-10-16T12:49:58.836 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:49:58.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:58.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:49:59.080 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:49:59.081 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:59.081 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:59.145 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:49:59.146 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:59.146 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:59.206 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:59.206 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:49:59.206 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:59.270 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:49:59.271 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:49:59.271 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:49:59.329 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:49:59.330 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:49:59.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:49:59.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.569+0000 7f7c13fff640 1 Processor -- start 2023-10-16T12:49:59.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.569+0000 7f7c13fff640 1 -- start start 2023-10-16T12:49:59.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c1415cba0 0x7f7c14071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:59.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c13fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c140720d0 0x7f7c140cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:59.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c13fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c140d0500 con 0x7f7c140cdd40 2023-10-16T12:49:59.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c13fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c140ce1d0 con 0x7f7c140720d0 2023-10-16T12:49:59.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c13fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c140d07c0 con 0x7f7c1415cba0 2023-10-16T12:49:59.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c0a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c140720d0 0x7f7c140cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:59.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c0a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c140720d0 0x7f7c140cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53634/0 (socket says 172.21.15.60:53634) 2023-10-16T12:49:59.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.571+0000 7f7c0a7fc640 1 -- 172.21.15.60:0/2761759152 learned_addr learned my addr 172.21.15.60:0/2761759152 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.572+0000 7f7c12ffd640 1 --2- 172.21.15.60:0/2761759152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c1415cba0 0x7f7c14071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.572+0000 7f7c0a7fc640 1 -- 172.21.15.60:0/2761759152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c1415cba0 msgr2=0x7f7c14071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:49:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.572+0000 7f7c0a7fc640 1 --2- 172.21.15.60:0/2761759152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c1415cba0 0x7f7c14071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.572+0000 7f7c0a7fc640 1 -- 172.21.15.60:0/2761759152 >> v1:172.21.15.144:6789/0 conn(0x7f7c140cdd40 legacy=0x7f7c141607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.572+0000 7f7c0a7fc640 1 -- 172.21.15.60:0/2761759152 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c140d0aa0 con 0x7f7c140720d0 2023-10-16T12:49:59.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.573+0000 7f7c0a7fc640 1 --2- 172.21.15.60:0/2761759152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c140cd800 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009870 tx=0x7f7bfc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5def65f0ab31d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:59.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.573+0000 7f7c127fc640 1 -- 172.21.15.60:0/2761759152 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc030840 con 0x7f7c140720d0 2023-10-16T12:49:59.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c127fc640 1 -- 172.21.15.60:0/2761759152 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc0309a0 con 0x7f7c140720d0 2023-10-16T12:49:59.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c127fc640 1 -- 172.21.15.60:0/2761759152 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc030d30 con 0x7f7c140720d0 2023-10-16T12:49:59.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 -- 172.21.15.60:0/2761759152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 msgr2=0x7f7c140cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:59.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 --2- 172.21.15.60:0/2761759152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c140cd800 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009870 tx=0x7f7bfc02b980 comp rx=0 tx=0).stop 2023-10-16T12:49:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 -- 172.21.15.60:0/2761759152 shutdown_connections 2023-10-16T12:49:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 --2- 172.21.15.60:0/2761759152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c140cd800 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 --2- 172.21.15.60:0/2761759152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c1415cba0 0x7f7c14071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 -- 172.21.15.60:0/2761759152 >> 172.21.15.60:0/2761759152 conn(0x7f7c14065ed0 msgr2=0x7f7c140662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 -- 172.21.15.60:0/2761759152 shutdown_connections 2023-10-16T12:49:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.574+0000 7f7c13fff640 1 -- 172.21.15.60:0/2761759152 wait complete. 2023-10-16T12:49:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.576+0000 7f7c13fff640 1 Processor -- start 2023-10-16T12:49:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.576+0000 7f7c13fff640 1 -- start start 2023-10-16T12:49:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.577+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c141f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.577+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c140cdd40 0x7f7c14158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.577+0000 7f7c13fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c1415cba0 0x7f7c1415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.577+0000 7f7c13fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c141f46c0 con 0x7f7c140720d0 2023-10-16T12:49:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.577+0000 7f7c13fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c141f49a0 con 0x7f7c1415cba0 2023-10-16T12:49:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.577+0000 7f7c13fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c141f4c80 con 0x7f7c140cdd40 2023-10-16T12:49:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c141f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c141f5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53650/0 (socket says 172.21.15.60:53650) 2023-10-16T12:49:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 -- 172.21.15.60:0/1251563111 learned_addr learned my addr 172.21.15.60:0/1251563111 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:49:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 -- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c140cdd40 msgr2=0x7f7c14158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c140cdd40 0x7f7c14158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 -- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c1415cba0 msgr2=0x7f7c1415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:59.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c1415cba0 0x7f7c1415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 -- 172.21.15.60:0/1251563111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0094d0 con 0x7f7c140720d0 2023-10-16T12:49:59.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.578+0000 7f7c12ffd640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c141f5760 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400efc0 tx=0x7f7c0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:59.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.579+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04009b80 con 0x7f7c140720d0 2023-10-16T12:49:59.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.579+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400f040 con 0x7f7c140720d0 2023-10-16T12:49:59.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.579+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040146a0 con 0x7f7c140720d0 2023-10-16T12:49:59.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.579+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c141f4f60 con 0x7f7c140720d0 2023-10-16T12:49:59.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.579+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c141f4350 con 0x7f7c140720d0 2023-10-16T12:49:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.580+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1404e9b0 con 0x7f7c140720d0 2023-10-16T12:49:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.581+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7c04014800 con 0x7f7c140720d0 2023-10-16T12:49:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.582+0000 7f7c10ff9640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f7bf406af90 0x7f7bf406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:49:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.582+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7c0408a400 con 0x7f7c140720d0 2023-10-16T12:49:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.583+0000 7f7c0a7fc640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f7bf406af90 0x7f7bf406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:49:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.584+0000 7f7c0a7fc640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f7bf406af90 0x7f7bf406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc036000 tx=0x7f7bfc03b040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:49:59.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.586+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c040552e0 con 0x7f7c140720d0 2023-10-16T12:49:59.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.828+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c140ce4f0 con 0x7f7c140720d0 2023-10-16T12:49:59.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.829+0000 7f7c10ff9640 1 -- 172.21.15.60:0/1251563111 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f7c04058900 con 0x7f7c140720d0 2023-10-16T12:49:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:49:59.832 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5318,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4110513738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":4110513738},{"type":"v1","addr":"172.21.15.144:6837","nonce":4110513738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/359691286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":359691286},{"type":"v1","addr":"172.21.15.144:6835","nonce":359691286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/4014496239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4014496239},{"type":"v1","addr":"172.21.15.60:6834","nonce":4014496239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6786,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3547485090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3547485090},{"type":"v1","addr":"172.21.15.60:6835","nonce":3547485090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[]} 2023-10-16T12:49:59.832 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2023-10-16T12:49:59.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f7bf406af90 msgr2=0x7f7bf406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:59.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f7bf406af90 0x7f7bf406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc036000 tx=0x7f7bfc03b040 comp rx=0 tx=0).stop 2023-10-16T12:49:59.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 msgr2=0x7f7c141f5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:49:59.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c141f5760 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400efc0 tx=0x7f7c0400c490 comp rx=0 tx=0).stop 2023-10-16T12:49:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 shutdown_connections 2023-10-16T12:49:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f7bf406af90 0x7f7bf406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c140cdd40 0x7f7c14158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c1415cba0 0x7f7c1415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 --2- 172.21.15.60:0/1251563111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c140720d0 0x7f7c141f5760 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:49:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 >> 172.21.15.60:0/1251563111 conn(0x7f7c14065ed0 msgr2=0x7f7c140b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:49:59.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.832+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 shutdown_connections 2023-10-16T12:49:59.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:49:59.833+0000 7f7c13fff640 1 -- 172.21.15.60:0/1251563111 wait complete. 2023-10-16T12:49:59.855 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:49:59.855 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:49:59.856 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:49:59.856 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:59.856 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:49:59.857 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:49:59.857 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:49:59.857 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:59.857 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:49:59.858 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:49:59.858 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:49:59.858 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:59.858 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:49:59.859 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:49:59.859 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:49:59.859 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:49:59.875 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:49:59.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:49:59.882 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:49:59.882 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:49:59.884 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:49:59.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:49:59.885 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:49:59.885 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:49:59.932 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:49:59.933 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:59.933 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:49:59.934 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:59.940 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:49:59.940 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:59.941 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:49:59.941 DEBUG:teuthology.parallel:result is None 2023-10-16T12:49:59.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:50:00.097 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:50:00.099 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:50:00.101 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:50:00.113 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:50:00.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.144+0000 7fa15f11d640 1 Processor -- start 2023-10-16T12:50:00.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.144+0000 7fa15f11d640 1 -- start start 2023-10-16T12:50:00.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15f11d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1580caac0 0x7fa1580ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:00.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15f11d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1580cb030 con 0x7fa15815ad70 2023-10-16T12:50:00.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15f11d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1580c91c0 con 0x7fa15806a4b0 2023-10-16T12:50:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15f11d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580c94a0 con 0x7fa1580caac0 2023-10-16T12:50:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15ceb9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa15815ad70 0x7fa158069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41030/0 (socket says 172.21.15.60:41030) 2023-10-16T12:50:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15ceb9640 1 -- 172.21.15.60:0/1878066735 learned_addr learned my addr 172.21.15.60:0/1878066735 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15d6ba640 1 --2- 172.21.15.60:0/1878066735 >> v2:172.21.15.60:3300/0 conn(0x7fa1580caac0 0x7fa1580ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa1577fe640 1 -- 172.21.15.60:0/1878066735 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961559136 0 0) 0x7fa1580cb030 con 0x7fa15815ad70 2023-10-16T12:50:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa1577fe640 1 -- 172.21.15.60:0/1878066735 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa148003520 con 0x7fa15815ad70 2023-10-16T12:50:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15d6ba640 1 -- 172.21.15.60:0/1878066735 >> v1:172.21.15.144:6789/0 conn(0x7fa15806a4b0 legacy=0x7fa1580ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15d6ba640 1 -- 172.21.15.60:0/1878066735 >> v1:172.21.15.60:6789/0 conn(0x7fa15815ad70 legacy=0x7fa158069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.146+0000 7fa15d6ba640 1 -- 172.21.15.60:0/1878066735 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580c9780 con 0x7fa1580caac0 2023-10-16T12:50:00.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa15d6ba640 1 --2- 172.21.15.60:0/1878066735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1580ad4d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa150009d90 tx=0x7fa15002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fe2d2020942b5a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:00.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa1577fe640 1 -- 172.21.15.60:0/1878066735 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15002cd00 con 0x7fa1580caac0 2023-10-16T12:50:00.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa1577fe640 1 -- 172.21.15.60:0/1878066735 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15002ce60 con 0x7fa1580caac0 2023-10-16T12:50:00.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa1577fe640 1 -- 172.21.15.60:0/1878066735 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150034660 con 0x7fa1580caac0 2023-10-16T12:50:00.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa15f11d640 1 -- 172.21.15.60:0/1878066735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 msgr2=0x7fa1580ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa15f11d640 1 --2- 172.21.15.60:0/1878066735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1580ad4d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa150009d90 tx=0x7fa15002bf50 comp rx=0 tx=0).stop 2023-10-16T12:50:00.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa15f11d640 1 -- 172.21.15.60:0/1878066735 shutdown_connections 2023-10-16T12:50:00.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa15f11d640 1 --2- 172.21.15.60:0/1878066735 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1580ad4d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.148+0000 7fa15f11d640 1 -- 172.21.15.60:0/1878066735 >> 172.21.15.60:0/1878066735 conn(0x7fa1580ce9b0 msgr2=0x7fa1580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:00.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.149+0000 7fa15f11d640 1 -- 172.21.15.60:0/1878066735 shutdown_connections 2023-10-16T12:50:00.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.149+0000 7fa15f11d640 1 -- 172.21.15.60:0/1878066735 wait complete. 2023-10-16T12:50:00.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.150+0000 7fa15f11d640 1 Processor -- start 2023-10-16T12:50:00.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.150+0000 7fa15f11d640 1 -- start start 2023-10-16T12:50:00.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa15f11d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa15806a4b0 0x7fa1580c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:00.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa15f11d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1581f0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:00.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa15f11d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815ad70 0x7fa1580c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:00.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa15f11d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580c7c50 con 0x7fa1580caac0 2023-10-16T12:50:00.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa15f11d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580c6b50 con 0x7fa15806a4b0 2023-10-16T12:50:00.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa15f11d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580c7040 con 0x7fa15815ad70 2023-10-16T12:50:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.151+0000 7fa157fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1581f0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1581f0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53692/0 (socket says 172.21.15.60:53692) 2023-10-16T12:50:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 -- 172.21.15.60:0/4015401770 learned_addr learned my addr 172.21.15.60:0/4015401770 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa15d6ba640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815ad70 0x7fa1580c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa15ceb9640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa15806a4b0 0x7fa1580c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 -- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815ad70 msgr2=0x7fa1580c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815ad70 0x7fa1580c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 -- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa15806a4b0 msgr2=0x7fa1580c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa15806a4b0 0x7fa1580c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 -- 172.21.15.60:0/4015401770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150009a70 con 0x7fa1580caac0 2023-10-16T12:50:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa15d6ba640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815ad70 0x7fa1580c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa157fff640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1581f0090 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa14400c450 tx=0x7fa14400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.152+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1440052d0 con 0x7fa1580caac0 2023-10-16T12:50:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.153+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa144005430 con 0x7fa1580caac0 2023-10-16T12:50:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.153+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1580c7320 con 0x7fa1580caac0 2023-10-16T12:50:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.153+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa144012870 con 0x7fa1580caac0 2023-10-16T12:50:00.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.153+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1580c78e0 con 0x7fa1580caac0 2023-10-16T12:50:00.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.154+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1580c7e00 con 0x7fa1580caac0 2023-10-16T12:50:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.155+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa1440129d0 con 0x7fa1580caac0 2023-10-16T12:50:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.156+0000 7fa1557fa640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa13c06afc0 0x7fa13c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.156+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fa144089180 con 0x7fa1580caac0 2023-10-16T12:50:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.156+0000 7fa15ceb9640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa13c06afc0 0x7fa13c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.157+0000 7fa15ceb9640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa13c06afc0 0x7fa13c06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0011e0 tx=0x7fa14c009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:00.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.160+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1580c7e00 con 0x7fa1580caac0 2023-10-16T12:50:00.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.368+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa15804e160 con 0x7fa1580caac0 2023-10-16T12:50:00.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.843+0000 7fa1557fa640 1 -- 172.21.15.60:0/4015401770 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v93) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa144054200 con 0x7fa1580caac0 2023-10-16T12:50:00.846 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:50:00.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.847+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa13c06afc0 msgr2=0x7fa13c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.847+0000 7fa15f11d640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa13c06afc0 0x7fa13c06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0011e0 tx=0x7fa14c009040 comp rx=0 tx=0).stop 2023-10-16T12:50:00.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.847+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 msgr2=0x7fa1581f0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:00.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.847+0000 7fa15f11d640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1581f0090 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa14400c450 tx=0x7fa14400c910 comp rx=0 tx=0).stop 2023-10-16T12:50:00.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 shutdown_connections 2023-10-16T12:50:00.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa13c06afc0 0x7fa13c06d480 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa15815ad70 0x7fa1580c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1580caac0 0x7fa1581f0090 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 --2- 172.21.15.60:0/4015401770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa15806a4b0 0x7fa1580c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:00.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 >> 172.21.15.60:0/4015401770 conn(0x7fa1580ce9b0 msgr2=0x7fa1580ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:00.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 shutdown_connections 2023-10-16T12:50:00.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:00.848+0000 7fa15f11d640 1 -- 172.21.15.60:0/4015401770 wait complete. 2023-10-16T12:50:00.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:50:01.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.084+0000 7f86ced47640 1 Processor -- start 2023-10-16T12:50:01.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.084+0000 7f86ced47640 1 -- start start 2023-10-16T12:50:01.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ced47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c8070fd0 0x7f86c80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ced47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86c80adf30 0x7f86c80b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ced47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c8071450 con 0x7f86c80adb50 2023-10-16T12:50:01.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ced47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c80cc600 con 0x7f86c80adf30 2023-10-16T12:50:01.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ced47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80cc8e0 con 0x7f86c8070fd0 2023-10-16T12:50:01.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ccae3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86c80adb50 0x7f86c806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41032/0 (socket says 172.21.15.60:41032) 2023-10-16T12:50:01.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86ccae3640 1 -- 172.21.15.60:0/2337626109 learned_addr learned my addr 172.21.15.60:0/2337626109 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:01.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.086+0000 7f86cd2e4640 1 --2- 172.21.15.60:0/2337626109 >> v2:172.21.15.60:3300/0 conn(0x7f86c80adf30 0x7f86c80b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.087+0000 7f86c7fff640 1 --2- 172.21.15.60:0/2337626109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c8070fd0 0x7f86c80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.087+0000 7f86cd2e4640 1 -- 172.21.15.60:0/2337626109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c8070fd0 msgr2=0x7f86c80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:01.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.087+0000 7f86cd2e4640 1 --2- 172.21.15.60:0/2337626109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c8070fd0 0x7f86c80b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.087+0000 7f86cd2e4640 1 -- 172.21.15.60:0/2337626109 >> v1:172.21.15.60:6789/0 conn(0x7f86c80adb50 legacy=0x7f86c806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.087+0000 7f86c77fe640 1 -- 172.21.15.60:0/2337626109 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866930735 0 0) 0x7f86c8071450 con 0x7f86c80adb50 2023-10-16T12:50:01.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.087+0000 7f86cd2e4640 1 -- 172.21.15.60:0/2337626109 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c80ccbc0 con 0x7f86c80adf30 2023-10-16T12:50:01.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.088+0000 7f86cd2e4640 1 --2- 172.21.15.60:0/2337626109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adf30 0x7f86c80b0950 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009d90 tx=0x7f86c002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32afe9ecabbbbef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:01.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.088+0000 7f86c77fe640 1 -- 172.21.15.60:0/2337626109 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c002cc50 con 0x7f86c80adf30 2023-10-16T12:50:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.088+0000 7f86c77fe640 1 -- 172.21.15.60:0/2337626109 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c002cdb0 con 0x7f86c80adf30 2023-10-16T12:50:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.088+0000 7f86c77fe640 1 -- 172.21.15.60:0/2337626109 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0034660 con 0x7f86c80adf30 2023-10-16T12:50:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 -- 172.21.15.60:0/2337626109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adf30 msgr2=0x7f86c80b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 --2- 172.21.15.60:0/2337626109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adf30 0x7f86c80b0950 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009d90 tx=0x7f86c002bea0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 -- 172.21.15.60:0/2337626109 shutdown_connections 2023-10-16T12:50:01.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 --2- 172.21.15.60:0/2337626109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adf30 0x7f86c80b0950 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 --2- 172.21.15.60:0/2337626109 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c8070fd0 0x7f86c80b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 -- 172.21.15.60:0/2337626109 >> 172.21.15.60:0/2337626109 conn(0x7f86c80cb410 msgr2=0x7f86c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:01.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 -- 172.21.15.60:0/2337626109 shutdown_connections 2023-10-16T12:50:01.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.089+0000 7f86ced47640 1 -- 172.21.15.60:0/2337626109 wait complete. 2023-10-16T12:50:01.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.091+0000 7f86ced47640 1 Processor -- start 2023-10-16T12:50:01.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.091+0000 7f86ced47640 1 -- start start 2023-10-16T12:50:01.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86ced47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c8070fd0 0x7f86c815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86ced47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 0x7f86c8163200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86ced47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c80adf30 0x7f86c80c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86ced47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c8162ab0 con 0x7f86c80adb50 2023-10-16T12:50:01.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86ced47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c8164a00 con 0x7f86c8070fd0 2023-10-16T12:50:01.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86ced47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c9430 con 0x7f86c80adf30 2023-10-16T12:50:01.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86c7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 0x7f86c8163200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86c7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 0x7f86c8163200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53702/0 (socket says 172.21.15.60:53702) 2023-10-16T12:50:01.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.092+0000 7f86c7fff640 1 -- 172.21.15.60:0/1517011743 learned_addr learned my addr 172.21.15.60:0/1517011743 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:01.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c7fff640 1 -- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c80adf30 msgr2=0x7f86c80c27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c7fff640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c80adf30 0x7f86c80c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c7fff640 1 -- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c8070fd0 msgr2=0x7f86c815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c7fff640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c8070fd0 0x7f86c815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c7fff640 1 -- 172.21.15.60:0/1517011743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0009a70 con 0x7f86c80adb50 2023-10-16T12:50:01.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c7fff640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 0x7f86c8163200 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f86b400d5f0 tx=0x7f86b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:01.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40125f0 con 0x7f86c80adb50 2023-10-16T12:50:01.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b4012750 con 0x7f86c80adb50 2023-10-16T12:50:01.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4012b00 con 0x7f86c80adb50 2023-10-16T12:50:01.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.093+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c80c9710 con 0x7f86c80adb50 2023-10-16T12:50:01.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.094+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c8164570 con 0x7f86c80adb50 2023-10-16T12:50:01.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.095+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c80c9900 con 0x7f86c80adb50 2023-10-16T12:50:01.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.096+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f86b4012c60 con 0x7f86c80adb50 2023-10-16T12:50:01.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.096+0000 7f86c57fa640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f86ac06af90 0x7f86ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.097+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f86b40898e0 con 0x7f86c80adb50 2023-10-16T12:50:01.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.097+0000 7f86ccae3640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f86ac06af90 0x7f86ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.098+0000 7f86ccae3640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f86ac06af90 0x7f86ac06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86bc007fd0 tx=0x7f86bc007920 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:01.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.101+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f86c80c9900 con 0x7f86c80adb50 2023-10-16T12:50:01.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.367+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f86c80c9900 con 0x7f86c80adb50 2023-10-16T12:50:01.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.367+0000 7f86c57fa640 1 -- 172.21.15.60:0/1517011743 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f86c80c9900 con 0x7f86c80adb50 2023-10-16T12:50:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:01.370 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:50:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:50:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:50:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:50:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:50:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:50:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:50:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:50:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:50:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:50:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.378 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.378 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:01.378 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:01.378 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.378 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:50:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:50:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:50:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:50:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:50:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:50:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:50:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:50:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:50:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:50:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:50:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:50:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:50:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:50:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:50:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:50:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:50:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:50:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:50:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:50:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:50:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:50:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:50:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:50:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:50:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:50:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:50:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:50:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:50:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:50:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:50:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:50:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:50:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:50:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:50:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:50:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:50:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:50:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:50:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:50:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:50:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:50:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:50:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:50:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:50:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:50:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:50:01.406 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:50:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f86ac06af90 msgr2=0x7f86ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f86ac06af90 0x7f86ac06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86bc007fd0 tx=0x7f86bc007920 comp rx=0 tx=0).stop 2023-10-16T12:50:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 msgr2=0x7f86c8163200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 0x7f86c8163200 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f86b400d5f0 tx=0x7f86b400dab0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 shutdown_connections 2023-10-16T12:50:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f86ac06af90 0x7f86ac06d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86c80adf30 0x7f86c80c27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c80adb50 0x7f86c8163200 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 --2- 172.21.15.60:0/1517011743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c8070fd0 0x7f86c815eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 >> 172.21.15.60:0/1517011743 conn(0x7f86c80cb410 msgr2=0x7f86c8160bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.370+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 shutdown_connections 2023-10-16T12:50:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.371+0000 7f86ced47640 1 -- 172.21.15.60:0/1517011743 wait complete. 2023-10-16T12:50:01.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:01.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.637+0000 7f452a62e640 1 Processor -- start 2023-10-16T12:50:01.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.637+0000 7f452a62e640 1 -- start start 2023-10-16T12:50:01.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452a62e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f452406c040 0x7f452406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452a62e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45240ad970 0x7f452415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452a62e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452406c960 con 0x7f452415ebc0 2023-10-16T12:50:01.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452a62e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452415b9d0 con 0x7f452406c040 2023-10-16T12:50:01.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452a62e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452415bcb0 con 0x7f45240ad970 2023-10-16T12:50:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452962c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f452406c040 0x7f452406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452962c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f452406c040 0x7f452406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53716/0 (socket says 172.21.15.60:53716) 2023-10-16T12:50:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.639+0000 7f452962c640 1 -- 172.21.15.60:0/1532911837 learned_addr learned my addr 172.21.15.60:0/1532911837 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:01.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.640+0000 7f452962c640 1 -- 172.21.15.60:0/1532911837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45240ad970 msgr2=0x7f452415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.640+0000 7f452962c640 1 --2- 172.21.15.60:0/1532911837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45240ad970 0x7f452415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.640+0000 7f452962c640 1 -- 172.21.15.60:0/1532911837 >> v1:172.21.15.144:6789/0 conn(0x7f452415ebc0 legacy=0x7f45240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.640+0000 7f452962c640 1 -- 172.21.15.60:0/1532911837 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452415bf90 con 0x7f452406c040 2023-10-16T12:50:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.641+0000 7f452962c640 1 --2- 172.21.15.60:0/1532911837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452406c040 0x7f452406c420 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f45200098a0 tx=0x7f452002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526b2a0c297fa69b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.641+0000 7f451bfff640 1 -- 172.21.15.60:0/1532911837 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520031640 con 0x7f452406c040 2023-10-16T12:50:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.641+0000 7f451bfff640 1 -- 172.21.15.60:0/1532911837 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45200317a0 con 0x7f452406c040 2023-10-16T12:50:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.641+0000 7f451bfff640 1 -- 172.21.15.60:0/1532911837 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520031b30 con 0x7f452406c040 2023-10-16T12:50:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 -- 172.21.15.60:0/1532911837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452406c040 msgr2=0x7f452406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 --2- 172.21.15.60:0/1532911837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452406c040 0x7f452406c420 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f45200098a0 tx=0x7f452002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 -- 172.21.15.60:0/1532911837 shutdown_connections 2023-10-16T12:50:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 --2- 172.21.15.60:0/1532911837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45240ad970 0x7f452415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 --2- 172.21.15.60:0/1532911837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452406c040 0x7f452406c420 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 -- 172.21.15.60:0/1532911837 >> 172.21.15.60:0/1532911837 conn(0x7f45240ce6a0 msgr2=0x7f45240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 -- 172.21.15.60:0/1532911837 shutdown_connections 2023-10-16T12:50:01.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.642+0000 7f452a62e640 1 -- 172.21.15.60:0/1532911837 wait complete. 2023-10-16T12:50:01.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.644+0000 7f452a62e640 1 Processor -- start 2023-10-16T12:50:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.644+0000 7f452a62e640 1 -- start start 2023-10-16T12:50:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.645+0000 7f452a62e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f452406c040 0x7f45240c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.645+0000 7f452a62e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45240ad970 0x7f4524158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.645+0000 7f452a62e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 0x7f45241591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.645+0000 7f452a62e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524154220 con 0x7f452415ebc0 2023-10-16T12:50:01.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.645+0000 7f452a62e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45240ae4a0 con 0x7f45240ad970 2023-10-16T12:50:01.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.645+0000 7f452a62e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45241e6c30 con 0x7f452406c040 2023-10-16T12:50:01.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 0x7f45241591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 0x7f45241591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53718/0 (socket says 172.21.15.60:53718) 2023-10-16T12:50:01.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 -- 172.21.15.60:0/2536836857 learned_addr learned my addr 172.21.15.60:0/2536836857 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:01.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f452962c640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f452406c040 0x7f45240c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4528e2b640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45240ad970 0x7f4524158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 -- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f452406c040 msgr2=0x7f45240c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f452406c040 0x7f45240c8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 -- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45240ad970 msgr2=0x7f4524158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45240ad970 0x7f4524158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4529e2d640 1 -- 172.21.15.60:0/2536836857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45200094d0 con 0x7f452415ebc0 2023-10-16T12:50:01.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.646+0000 7f4528e2b640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45240ad970 0x7f4524158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:01.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.647+0000 7f4529e2d640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 0x7f45241591a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c780 tx=0x7f451c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:01.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.647+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c003d80 con 0x7f452415ebc0 2023-10-16T12:50:01.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.647+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c00ce90 con 0x7f452415ebc0 2023-10-16T12:50:01.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.647+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c0052b0 con 0x7f452415ebc0 2023-10-16T12:50:01.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.647+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45241e6670 con 0x7f452415ebc0 2023-10-16T12:50:01.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.647+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45240ae130 con 0x7f452415ebc0 2023-10-16T12:50:01.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.648+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452406b330 con 0x7f452415ebc0 2023-10-16T12:50:01.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.649+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f451c005410 con 0x7f452415ebc0 2023-10-16T12:50:01.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.650+0000 7f4519ffb640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f450806af90 0x7f450806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:01.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.650+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f451c0888c0 con 0x7f452415ebc0 2023-10-16T12:50:01.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.651+0000 7f452962c640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f450806af90 0x7f450806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:01.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.652+0000 7f452962c640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f450806af90 0x7f450806d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4520036000 tx=0x7f452003c040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:01.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.655+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f451c053b10 con 0x7f452415ebc0 2023-10-16T12:50:01.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.853+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45240ae610 con 0x7f452415ebc0 2023-10-16T12:50:01.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.854+0000 7f4519ffb640 1 -- 172.21.15.60:0/2536836857 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+12845 (secure 0 0 0) 0x7f451c017090 con 0x7f452415ebc0 2023-10-16T12:50:01.857 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:01.857 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:00.861196+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f450806af90 msgr2=0x7f450806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f450806af90 0x7f450806d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4520036000 tx=0x7f452003c040 comp rx=0 tx=0).stop 2023-10-16T12:50:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 msgr2=0x7f45241591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 0x7f45241591a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c780 tx=0x7f451c00cc40 comp rx=0 tx=0).stop 2023-10-16T12:50:01.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 shutdown_connections 2023-10-16T12:50:01.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f450806af90 0x7f450806d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452415ebc0 0x7f45241591a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45240ad970 0x7f4524158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 --2- 172.21.15.60:0/2536836857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f452406c040 0x7f45240c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:01.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 >> 172.21.15.60:0/2536836857 conn(0x7f45240ce6a0 msgr2=0x7f45240b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:01.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 shutdown_connections 2023-10-16T12:50:01.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:01.857+0000 7f452a62e640 1 -- 172.21.15.60:0/2536836857 wait complete. 2023-10-16T12:50:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:50:01.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:50:02.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.090+0000 7f9bc4661640 1 Processor -- start 2023-10-16T12:50:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.090+0000 7f9bc4661640 1 -- start start 2023-10-16T12:50:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc4661640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bbc15ad70 0x7f9bbc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc4661640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc0caac0 0x7f9bbc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc4661640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bbc0cb030 con 0x7f9bbc0c9ca0 2023-10-16T12:50:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc4661640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bbc0c91c0 con 0x7f9bbc15ad70 2023-10-16T12:50:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc4661640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0c94a0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc23fd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bbc15ad70 0x7f9bbc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc23fd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bbc15ad70 0x7f9bbc069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53730/0 (socket says 172.21.15.60:53730) 2023-10-16T12:50:02.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.091+0000 7f9bc23fd640 1 -- 172.21.15.60:0/2060014152 learned_addr learned my addr 172.21.15.60:0/2060014152 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:02.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.092+0000 7f9bc2bfe640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc0caac0 0x7f9bbc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.092+0000 7f9bc23fd640 1 -- 172.21.15.60:0/2060014152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc0caac0 msgr2=0x7f9bbc06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:02.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.092+0000 7f9bc23fd640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc0caac0 0x7f9bbc06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.092+0000 7f9bc23fd640 1 -- 172.21.15.60:0/2060014152 >> v1:172.21.15.144:6789/0 conn(0x7f9bbc0c9ca0 legacy=0x7f9bbc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:02.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.092+0000 7f9bc23fd640 1 -- 172.21.15.60:0/2060014152 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0c9780 con 0x7f9bbc15ad70 2023-10-16T12:50:02.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.092+0000 7f9bc2bfe640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc0caac0 0x7f9bbc06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:02.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.093+0000 7f9bc23fd640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc15ad70 0x7f9bbc069da0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009870 tx=0x7f9bb802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab42c112aa1b03b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:02.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.093+0000 7f9bc13fb640 1 -- 172.21.15.60:0/2060014152 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8031640 con 0x7f9bbc15ad70 2023-10-16T12:50:02.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.093+0000 7f9bc13fb640 1 -- 172.21.15.60:0/2060014152 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb80317a0 con 0x7f9bbc15ad70 2023-10-16T12:50:02.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.093+0000 7f9bc13fb640 1 -- 172.21.15.60:0/2060014152 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8031b30 con 0x7f9bbc15ad70 2023-10-16T12:50:02.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 -- 172.21.15.60:0/2060014152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc15ad70 msgr2=0x7f9bbc069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc15ad70 0x7f9bbc069da0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009870 tx=0x7f9bb802b8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 -- 172.21.15.60:0/2060014152 shutdown_connections 2023-10-16T12:50:02.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc0caac0 0x7f9bbc06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 --2- 172.21.15.60:0/2060014152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc15ad70 0x7f9bbc069da0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 -- 172.21.15.60:0/2060014152 >> 172.21.15.60:0/2060014152 conn(0x7f9bbc0ce9b0 msgr2=0x7f9bbc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:02.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 -- 172.21.15.60:0/2060014152 shutdown_connections 2023-10-16T12:50:02.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.094+0000 7f9bc4661640 1 -- 172.21.15.60:0/2060014152 wait complete. 2023-10-16T12:50:02.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.096+0000 7f9bc4661640 1 Processor -- start 2023-10-16T12:50:02.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.096+0000 7f9bc4661640 1 -- start start 2023-10-16T12:50:02.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc4661640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bbc0c9ca0 0x7f9bbc14f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc4661640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 0x7f9bbc0c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc4661640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc15ad70 0x7f9bbc15dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc4661640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc14f640 con 0x7f9bbc0caac0 2023-10-16T12:50:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc4661640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc15ca40 con 0x7f9bbc0c9ca0 2023-10-16T12:50:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc4661640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc15cd20 con 0x7f9bbc15ad70 2023-10-16T12:50:02.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc1bfc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 0x7f9bbc0c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc1bfc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 0x7f9bbc0c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53744/0 (socket says 172.21.15.60:53744) 2023-10-16T12:50:02.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc1bfc640 1 -- 172.21.15.60:0/3156400762 learned_addr learned my addr 172.21.15.60:0/3156400762 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:02.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc2bfe640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc15ad70 0x7f9bbc15dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.097+0000 7f9bc23fd640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bbc0c9ca0 0x7f9bbc14f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc1bfc640 1 -- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc15ad70 msgr2=0x7f9bbc15dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc1bfc640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc15ad70 0x7f9bbc15dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc1bfc640 1 -- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bbc0c9ca0 msgr2=0x7f9bbc14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc1bfc640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bbc0c9ca0 0x7f9bbc14f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc1bfc640 1 -- 172.21.15.60:0/3156400762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb80094d0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc23fd640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bbc0c9ca0 0x7f9bbc14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc1bfc640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 0x7f9bbc0c3060 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00c430 tx=0x7f9bac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac011790 con 0x7f9bbc0caac0 2023-10-16T12:50:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac0118f0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac011ce0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bbc15d000 con 0x7f9bbc0caac0 2023-10-16T12:50:02.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.098+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bbc15d5c0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.099+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bbc04e160 con 0x7f9bbc0caac0 2023-10-16T12:50:02.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.100+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9bac01c460 con 0x7f9bbc0caac0 2023-10-16T12:50:02.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.101+0000 7f9bb2ffd640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ba006af90 0x7f9ba006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.101+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9bac089490 con 0x7f9bbc0caac0 2023-10-16T12:50:02.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.102+0000 7f9bc23fd640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ba006af90 0x7f9ba006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.103+0000 7f9bc23fd640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ba006af90 0x7f9ba006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8036000 tx=0x7f9bb803c040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:02.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.105+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9bac0546e0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.300+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bbc15d7e0 con 0x7f9bbc0caac0 2023-10-16T12:50:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.301+0000 7f9bb2ffd640 1 -- 172.21.15.60:0/3156400762 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bac057d00 con 0x7f9bbc0caac0 2023-10-16T12:50:02.304 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:50:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ba006af90 msgr2=0x7f9ba006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ba006af90 0x7f9ba006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8036000 tx=0x7f9bb803c040 comp rx=0 tx=0).stop 2023-10-16T12:50:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 msgr2=0x7f9bbc0c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 0x7f9bbc0c3060 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00c430 tx=0x7f9bac00c8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 shutdown_connections 2023-10-16T12:50:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ba006af90 0x7f9ba006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bbc15ad70 0x7f9bbc15dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bbc0caac0 0x7f9bbc0c3060 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.303+0000 7f9bc4661640 1 --2- 172.21.15.60:0/3156400762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bbc0c9ca0 0x7f9bbc14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.304+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 >> 172.21.15.60:0/3156400762 conn(0x7f9bbc0ce9b0 msgr2=0x7f9bbc0afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.304+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 shutdown_connections 2023-10-16T12:50:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.304+0000 7f9bc4661640 1 -- 172.21.15.60:0/3156400762 wait complete. 2023-10-16T12:50:02.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:50:02.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.551+0000 7efcb8d21640 1 Processor -- start 2023-10-16T12:50:02.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.551+0000 7efcb8d21640 1 -- start start 2023-10-16T12:50:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.553+0000 7efcb8d21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 0x7efcb40cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.553+0000 7efcb8d21640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb415b9d0 con 0x7efcb40ad970 2023-10-16T12:50:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.553+0000 7efcb8d21640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb415bcb0 con 0x7efcb406c040 2023-10-16T12:50:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.553+0000 7efcb8d21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb415b7d0 con 0x7efcb415b320 2023-10-16T12:50:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.553+0000 7efcb2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efcb40ad970 0x7efcb40cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41046/0 (socket says 172.21.15.60:41046) 2023-10-16T12:50:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.553+0000 7efcb2ffd640 1 -- 172.21.15.60:0/416016618 learned_addr learned my addr 172.21.15.60:0/416016618 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb3fff640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 0x7efcb40cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998973045 0 0) 0x7efcb415b9d0 con 0x7efcb40ad970 2023-10-16T12:50:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c003520 con 0x7efcb40ad970 2023-10-16T12:50:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752636921 0 0) 0x7efc9c003520 con 0x7efcb40ad970 2023-10-16T12:50:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 msgr2=0x7efcb40cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 0x7efcb40cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 >> v1:172.21.15.144:6789/0 conn(0x7efcb406c040 legacy=0x7efcb406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb415bf90 con 0x7efcb40ad970 2023-10-16T12:50:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efca8001670 con 0x7efcb40ad970 2023-10-16T12:50:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.554+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 >> v1:172.21.15.60:6789/0 conn(0x7efcb40ad970 legacy=0x7efcb40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb27fc640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9c0039a0 0x7efc9c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb27fc640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 0x7efc9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb27fc640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc9c004d00 0x7efc9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb415b7d0 con 0x7efc9c004d00 2023-10-16T12:50:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00ba30 con 0x7efc9c0039a0 2023-10-16T12:50:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00bbc0 con 0x7efcb415b320 2023-10-16T12:50:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb3fff640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc9c004d00 0x7efc9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb37fe640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9c0039a0 0x7efc9c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb2ffd640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 0x7efc9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb3fff640 1 -- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 msgr2=0x7efc9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb3fff640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcb415b320 0x7efc9c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb3fff640 1 -- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9c0039a0 msgr2=0x7efc9c003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb3fff640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9c0039a0 0x7efc9c003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.555+0000 7efcb3fff640 1 -- 172.21.15.60:0/416016618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c003520 con 0x7efc9c004d00 2023-10-16T12:50:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.556+0000 7efcb3fff640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc9c004d00 0x7efc9c005180 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7efcac00e170 tx=0x7efc9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42afaa772b3e719 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac02cbf0 con 0x7efc9c004d00 2023-10-16T12:50:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac00f040 con 0x7efc9c004d00 2023-10-16T12:50:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb27fc640 1 -- 172.21.15.60:0/416016618 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac034440 con 0x7efc9c004d00 2023-10-16T12:50:02.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb8d21640 1 -- 172.21.15.60:0/416016618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc9c004d00 msgr2=0x7efc9c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb8d21640 1 --2- 172.21.15.60:0/416016618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc9c004d00 0x7efc9c005180 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7efcac00e170 tx=0x7efc9c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb3fff640 1 -- 172.21.15.60:0/416016618 reap_dead start 2023-10-16T12:50:02.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb8d21640 1 -- 172.21.15.60:0/416016618 shutdown_connections 2023-10-16T12:50:02.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.557+0000 7efcb8d21640 1 -- 172.21.15.60:0/416016618 >> 172.21.15.60:0/416016618 conn(0x7efcb40ce6a0 msgr2=0x7efcb40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.558+0000 7efcb8d21640 1 -- 172.21.15.60:0/416016618 shutdown_connections 2023-10-16T12:50:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.558+0000 7efcb8d21640 1 -- 172.21.15.60:0/416016618 wait complete. 2023-10-16T12:50:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.559+0000 7efcb8d21640 1 Processor -- start 2023-10-16T12:50:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 -- start start 2023-10-16T12:50:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc9c004d00 0x7efcb40b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcb41f0750 0x7efcb414b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 0x7efcb40ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40ae100 con 0x7efcb414bb20 2023-10-16T12:50:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb414bfa0 con 0x7efcb41f0750 2023-10-16T12:50:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.560+0000 7efcb8d21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40ae310 con 0x7efc9c004d00 2023-10-16T12:50:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 0x7efcb40ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 0x7efcb40ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53758/0 (socket says 172.21.15.60:53758) 2023-10-16T12:50:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 -- 172.21.15.60:0/367579232 learned_addr learned my addr 172.21.15.60:0/367579232 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb2ffd640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcb41f0750 0x7efcb414b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb37fe640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc9c004d00 0x7efcb40b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 -- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc9c004d00 msgr2=0x7efcb40b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc9c004d00 0x7efcb40b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 -- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcb41f0750 msgr2=0x7efcb414b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcb41f0750 0x7efcb414b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb3fff640 1 -- 172.21.15.60:0/367579232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb40ae5f0 con 0x7efcb414bb20 2023-10-16T12:50:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb37fe640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc9c004d00 0x7efcb40b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.561+0000 7efcb2ffd640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcb41f0750 0x7efcb414b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.562+0000 7efcb3fff640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 0x7efcb40ac440 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7efcac009f90 tx=0x7efcac009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.562+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac043070 con 0x7efcb414bb20 2023-10-16T12:50:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.562+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac0349c0 con 0x7efcb414bb20 2023-10-16T12:50:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.562+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac034d70 con 0x7efcb414bb20 2023-10-16T12:50:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.562+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb40ae8d0 con 0x7efcb414bb20 2023-10-16T12:50:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.562+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb40b1340 con 0x7efcb414bb20 2023-10-16T12:50:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.563+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb406b330 con 0x7efcb414bb20 2023-10-16T12:50:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.564+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efcac03e440 con 0x7efcb414bb20 2023-10-16T12:50:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.565+0000 7efc9bfff640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7efca006af90 0x7efca006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.565+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efcac0aaf60 con 0x7efcb414bb20 2023-10-16T12:50:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.566+0000 7efcb37fe640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7efca006af90 0x7efca006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.567+0000 7efcb37fe640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7efca006af90 0x7efca006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efca40017e0 tx=0x7efca4007b70 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.570+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efcac0761c0 con 0x7efcb414bb20 2023-10-16T12:50:02.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.769+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efcb40b14d0 con 0x7efcb414bb20 2023-10-16T12:50:02.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.770+0000 7efc9bfff640 1 -- 172.21.15.60:0/367579232 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) v1 ==== 84+0+65 (secure 0 0 0) 0x7efcac039070 con 0x7efcb414bb20 2023-10-16T12:50:02.773 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:02.773 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:50:02.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:02.773 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:50:02.773 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:50:02.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:02.774 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:50:02.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.772+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7efca006af90 msgr2=0x7efca006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.772+0000 7efcb8d21640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7efca006af90 0x7efca006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efca40017e0 tx=0x7efca4007b70 comp rx=0 tx=0).stop 2023-10-16T12:50:02.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.772+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 msgr2=0x7efcb40ac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:02.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.772+0000 7efcb8d21640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 0x7efcb40ac440 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7efcac009f90 tx=0x7efcac009750 comp rx=0 tx=0).stop 2023-10-16T12:50:02.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 shutdown_connections 2023-10-16T12:50:02.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7efca006af90 0x7efca006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcb414bb20 0x7efcb40ac440 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcb41f0750 0x7efcb414b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 --2- 172.21.15.60:0/367579232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc9c004d00 0x7efcb40b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:02.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 >> 172.21.15.60:0/367579232 conn(0x7efcb40ce6a0 msgr2=0x7efcb40c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:02.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 shutdown_connections 2023-10-16T12:50:02.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:02.773+0000 7efcb8d21640 1 -- 172.21.15.60:0/367579232 wait complete. 2023-10-16T12:50:02.793 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:50:02.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:50:03.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.027+0000 7f0032941640 1 Processor -- start 2023-10-16T12:50:03.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.027+0000 7f0032941640 1 -- start start 2023-10-16T12:50:03.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.029+0000 7f0032941640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 0x7f002c0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.029+0000 7f0032941640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c15dfb0 con 0x7f002c159da0 2023-10-16T12:50:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.029+0000 7f0032941640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c15e290 con 0x7f002c0cce20 2023-10-16T12:50:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.029+0000 7f0032941640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c15e570 con 0x7f002c1599c0 2023-10-16T12:50:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.029+0000 7f00237fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f002c159da0 0x7f002c15d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41050/0 (socket says 172.21.15.60:41050) 2023-10-16T12:50:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.029+0000 7f00237fe640 1 -- 172.21.15.60:0/36655611 learned_addr learned my addr 172.21.15.60:0/36655611 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:03.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997488348 0 0) 0x7f002c15dfb0 con 0x7f002c159da0 2023-10-16T12:50:03.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0018003520 con 0x7f002c159da0 2023-10-16T12:50:03.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002bfff640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 0x7f002c0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4040689306 0 0) 0x7f0018003520 con 0x7f002c159da0 2023-10-16T12:50:03.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 msgr2=0x7f002c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:03.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 0x7f002c0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 >> v1:172.21.15.144:6789/0 conn(0x7f002c0cce20 legacy=0x7f002c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c06fda0 con 0x7f002c159da0 2023-10-16T12:50:03.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0014001690 con 0x7f002c159da0 2023-10-16T12:50:03.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.030+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 >> v1:172.21.15.60:6789/0 conn(0x7f002c159da0 legacy=0x7f002c15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002b7fe640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0018003a10 0x7f0018003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002b7fe640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00180047c0 0x7f0018004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002b7fe640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 0x7f001800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c15e570 con 0x7f0018003a10 2023-10-16T12:50:03.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180054e0 con 0x7f00180047c0 2023-10-16T12:50:03.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0018005650 con 0x7f002c1599c0 2023-10-16T12:50:03.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002bfff640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0018003a10 0x7f0018003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f00237fe640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00180047c0 0x7f0018004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002bfff640 1 -- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 msgr2=0x7f001800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:03.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002bfff640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c1599c0 0x7f001800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002bfff640 1 -- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00180047c0 msgr2=0x7f0018004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:03.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002bfff640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00180047c0 0x7f0018004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f002bfff640 1 -- 172.21.15.60:0/36655611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018003520 con 0x7f0018003a10 2023-10-16T12:50:03.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.031+0000 7f00237fe640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00180047c0 0x7f0018004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:03.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.032+0000 7f002bfff640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0018003a10 0x7f0018003e30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f001c0073e0 tx=0x7f00180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1048e64d02ec4b9d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:03.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c02cce0 con 0x7f0018003a10 2023-10-16T12:50:03.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c02ce40 con 0x7f0018003a10 2023-10-16T12:50:03.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f002b7fe640 1 -- 172.21.15.60:0/36655611 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c034660 con 0x7f0018003a10 2023-10-16T12:50:03.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0032941640 1 -- 172.21.15.60:0/36655611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0018003a10 msgr2=0x7f0018003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0032941640 1 --2- 172.21.15.60:0/36655611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0018003a10 0x7f0018003e30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f001c0073e0 tx=0x7f00180057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0030ede640 1 -- 172.21.15.60:0/36655611 reap_dead start 2023-10-16T12:50:03.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0032941640 1 -- 172.21.15.60:0/36655611 shutdown_connections 2023-10-16T12:50:03.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0032941640 1 -- 172.21.15.60:0/36655611 >> 172.21.15.60:0/36655611 conn(0x7f002c0cb360 msgr2=0x7f002c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:03.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0032941640 1 -- 172.21.15.60:0/36655611 shutdown_connections 2023-10-16T12:50:03.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.033+0000 7f0032941640 1 -- 172.21.15.60:0/36655611 wait complete. 2023-10-16T12:50:03.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.035+0000 7f0032941640 1 Processor -- start 2023-10-16T12:50:03.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.035+0000 7f0032941640 1 -- start start 2023-10-16T12:50:03.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.036+0000 7f0032941640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0018003a10 0x7f002c0b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.036+0000 7f0032941640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 0x7f002c1f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.036+0000 7f0032941640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c0c7b80 0x7f002c0c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.036+0000 7f0032941640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0c8540 con 0x7f002c157a20 2023-10-16T12:50:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.036+0000 7f0032941640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c15a7a0 con 0x7f0018003a10 2023-10-16T12:50:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.036+0000 7f0032941640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c1f5960 con 0x7f002c0c7b80 2023-10-16T12:50:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 0x7f002c1f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 0x7f002c1f5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53774/0 (socket says 172.21.15.60:53774) 2023-10-16T12:50:03.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 -- 172.21.15.60:0/3031219774 learned_addr learned my addr 172.21.15.60:0/3031219774 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:03.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f002bfff640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0018003a10 0x7f002c0b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f0030ede640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c0c7b80 0x7f002c0c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 -- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c0c7b80 msgr2=0x7f002c0c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c0c7b80 0x7f002c0c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 -- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0018003a10 msgr2=0x7f002c0b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0018003a10 0x7f002c0b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 -- 172.21.15.60:0/3031219774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c15a200 con 0x7f002c157a20 2023-10-16T12:50:03.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f002bfff640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0018003a10 0x7f002c0b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:03.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.037+0000 7f00237fe640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 0x7f002c1f5420 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f001400d770 tx=0x7f001400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:03.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.038+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00140074e0 con 0x7f002c157a20 2023-10-16T12:50:03.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.038+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0014007640 con 0x7f002c157a20 2023-10-16T12:50:03.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.038+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014004a00 con 0x7f002c157a20 2023-10-16T12:50:03.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.038+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c15a4c0 con 0x7f002c157a20 2023-10-16T12:50:03.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.038+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c0ae7c0 con 0x7f002c157a20 2023-10-16T12:50:03.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.039+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c04e160 con 0x7f002c157a20 2023-10-16T12:50:03.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.040+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0014004b60 con 0x7f002c157a20 2023-10-16T12:50:03.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.041+0000 7f00297fa640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f000c06afc0 0x7f000c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:03.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.041+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0014088f50 con 0x7f002c157a20 2023-10-16T12:50:03.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.042+0000 7f002bfff640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f000c06afc0 0x7f000c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:03.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.043+0000 7f002bfff640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f000c06afc0 0x7f000c06d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f001c036000 tx=0x7f001c030000 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:03.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.046+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0014054090 con 0x7f002c157a20 2023-10-16T12:50:03.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.245+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f002c0b4860 con 0x7f002c157a20 2023-10-16T12:50:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.858+0000 7f00297fa640 1 -- 172.21.15.60:0/3031219774 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) v1 ==== 138+0+0 (secure 0 0 0) 0x7f00140190b0 con 0x7f002c157a20 2023-10-16T12:50:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:50:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f000c06afc0 msgr2=0x7f000c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f000c06afc0 0x7f000c06d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f001c036000 tx=0x7f001c030000 comp rx=0 tx=0).stop 2023-10-16T12:50:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 msgr2=0x7f002c1f5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 0x7f002c1f5420 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f001400d770 tx=0x7f001400dc30 comp rx=0 tx=0).stop 2023-10-16T12:50:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 shutdown_connections 2023-10-16T12:50:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f000c06afc0 0x7f000c06d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f002c0c7b80 0x7f002c0c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002c157a20 0x7f002c1f5420 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 --2- 172.21.15.60:0/3031219774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0018003a10 0x7f002c0b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 >> 172.21.15.60:0/3031219774 conn(0x7f002c0cb360 msgr2=0x7f002c06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.862+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 shutdown_connections 2023-10-16T12:50:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:03.863+0000 7f0032941640 1 -- 172.21.15.60:0/3031219774 wait complete. 2023-10-16T12:50:03.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:50:04.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.106+0000 7ff223347640 1 Processor -- start 2023-10-16T12:50:04.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.107+0000 7ff223347640 1 -- start start 2023-10-16T12:50:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.108+0000 7ff223347640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c160440 0x7ff21c0d1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff223347640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff21c0b1e10 0x7ff21c06bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff223347640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21c06c260 con 0x7ff21c06b8c0 2023-10-16T12:50:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff223347640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21c06b440 con 0x7ff21c0b1e10 2023-10-16T12:50:04.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff223347640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0d0260 con 0x7ff21c160440 2023-10-16T12:50:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff2218e4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff21c0b1e10 0x7ff21c06bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff2218e4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff21c0b1e10 0x7ff21c06bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53776/0 (socket says 172.21.15.60:53776) 2023-10-16T12:50:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff2218e4640 1 -- 172.21.15.60:0/2083463124 learned_addr learned my addr 172.21.15.60:0/2083463124 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.109+0000 7ff2210e3640 1 --2- 172.21.15.60:0/2083463124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c160440 0x7ff21c0d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.110+0000 7ff2218e4640 1 -- 172.21.15.60:0/2083463124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c160440 msgr2=0x7ff21c0d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.110+0000 7ff2218e4640 1 --2- 172.21.15.60:0/2083463124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c160440 0x7ff21c0d1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.110+0000 7ff2218e4640 1 -- 172.21.15.60:0/2083463124 >> v1:172.21.15.144:6789/0 conn(0x7ff21c06b8c0 legacy=0x7ff21c160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.110+0000 7ff2218e4640 1 -- 172.21.15.60:0/2083463124 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0b0970 con 0x7ff21c0b1e10 2023-10-16T12:50:04.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.111+0000 7ff2218e4640 1 --2- 172.21.15.60:0/2083463124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c0b1e10 0x7ff21c06bd20 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff2140098a0 tx=0x7ff21402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5a9bd38df106b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:04.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.111+0000 7ff213fff640 1 -- 172.21.15.60:0/2083463124 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214030840 con 0x7ff21c0b1e10 2023-10-16T12:50:04.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.111+0000 7ff213fff640 1 -- 172.21.15.60:0/2083463124 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2140309a0 con 0x7ff21c0b1e10 2023-10-16T12:50:04.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.111+0000 7ff213fff640 1 -- 172.21.15.60:0/2083463124 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214030d30 con 0x7ff21c0b1e10 2023-10-16T12:50:04.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 -- 172.21.15.60:0/2083463124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c0b1e10 msgr2=0x7ff21c06bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:04.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 --2- 172.21.15.60:0/2083463124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c0b1e10 0x7ff21c06bd20 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff2140098a0 tx=0x7ff21402b980 comp rx=0 tx=0).stop 2023-10-16T12:50:04.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 -- 172.21.15.60:0/2083463124 shutdown_connections 2023-10-16T12:50:04.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 --2- 172.21.15.60:0/2083463124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c0b1e10 0x7ff21c06bd20 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 --2- 172.21.15.60:0/2083463124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c160440 0x7ff21c0d1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 -- 172.21.15.60:0/2083463124 >> 172.21.15.60:0/2083463124 conn(0x7ff21c099ea0 msgr2=0x7ff21c0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:04.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 -- 172.21.15.60:0/2083463124 shutdown_connections 2023-10-16T12:50:04.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.112+0000 7ff223347640 1 -- 172.21.15.60:0/2083463124 wait complete. 2023-10-16T12:50:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.114+0000 7ff223347640 1 Processor -- start 2023-10-16T12:50:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.114+0000 7ff223347640 1 -- start start 2023-10-16T12:50:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff223347640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 0x7ff21c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff223347640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c0b1e10 0x7ff21c15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff223347640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff21c160440 0x7ff21c155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff223347640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c156da0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff223347640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0b56e0 con 0x7ff21c160440 2023-10-16T12:50:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff223347640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0b4390 con 0x7ff21c0b1e10 2023-10-16T12:50:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff2210e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 0x7ff21c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff2210e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 0x7ff21c15c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53786/0 (socket says 172.21.15.60:53786) 2023-10-16T12:50:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.115+0000 7ff2210e3640 1 -- 172.21.15.60:0/3393625076 learned_addr learned my addr 172.21.15.60:0/3393625076 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2218e4640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff21c160440 0x7ff21c155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2208e2640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c0b1e10 0x7ff21c15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2210e3640 1 -- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c0b1e10 msgr2=0x7ff21c15a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2210e3640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c0b1e10 0x7ff21c15a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2210e3640 1 -- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff21c160440 msgr2=0x7ff21c155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2210e3640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff21c160440 0x7ff21c155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2210e3640 1 -- 172.21.15.60:0/3393625076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140094d0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.116+0000 7ff2210e3640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 0x7ff21c15c200 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff218007ef0 tx=0x7ff21800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.117+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218014620 con 0x7ff21c06b8c0 2023-10-16T12:50:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.117+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21800ee90 con 0x7ff21c06b8c0 2023-10-16T12:50:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.117+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2180149c0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.117+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c0b4670 con 0x7ff21c06b8c0 2023-10-16T12:50:04.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.117+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c0b5370 con 0x7ff21c06b8c0 2023-10-16T12:50:04.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.118+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c04e160 con 0x7ff21c06b8c0 2023-10-16T12:50:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.119+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff218014b20 con 0x7ff21c06b8c0 2023-10-16T12:50:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.121+0000 7ff211ffb640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7ff20006af90 0x7ff20006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.121+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff21808ade0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.122+0000 7ff2208e2640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7ff20006af90 0x7ff20006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.123+0000 7ff2208e2640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7ff20006af90 0x7ff20006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff21c159ee0 tx=0x7ff20c009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:04.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.126+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff218055fd0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.328+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff21c0ce1a0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.872+0000 7ff211ffb640 1 -- 172.21.15.60:0/3393625076 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff2180595f0 con 0x7ff21c06b8c0 2023-10-16T12:50:04.875 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:50:04.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.876+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7ff20006af90 msgr2=0x7ff20006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:04.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.876+0000 7ff223347640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7ff20006af90 0x7ff20006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff21c159ee0 tx=0x7ff20c009040 comp rx=0 tx=0).stop 2023-10-16T12:50:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.876+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 msgr2=0x7ff21c15c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.876+0000 7ff223347640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 0x7ff21c15c200 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff218007ef0 tx=0x7ff21800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 shutdown_connections 2023-10-16T12:50:04.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7ff20006af90 0x7ff20006d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff21c0b1e10 0x7ff21c15a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff21c160440 0x7ff21c155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 --2- 172.21.15.60:0/3393625076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff21c06b8c0 0x7ff21c15c200 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 >> 172.21.15.60:0/3393625076 conn(0x7ff21c099ea0 msgr2=0x7ff21c06d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.877+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 shutdown_connections 2023-10-16T12:50:04.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:04.878+0000 7ff223347640 1 -- 172.21.15.60:0/3393625076 wait complete. 2023-10-16T12:50:04.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:50:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.103+0000 7f534dccf640 1 Processor -- start 2023-10-16T12:50:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.103+0000 7f534dccf640 1 -- start start 2023-10-16T12:50:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f534dccf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f534815d150 0x7f53480ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f534dccf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53480ac890 con 0x7f534806a4b0 2023-10-16T12:50:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f534dccf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348161650 con 0x7f5348160b90 2023-10-16T12:50:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f534dccf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534815d530 con 0x7f534815d150 2023-10-16T12:50:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f53477fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f534815d150 0x7f53480ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:05.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f5347fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f534806a4b0 0x7f53480c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41062/0 (socket says 172.21.15.60:41062) 2023-10-16T12:50:05.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.105+0000 7f5347fff640 1 -- 172.21.15.60:0/1462634503 learned_addr learned my addr 172.21.15.60:0/1462634503 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.106+0000 7f53477fe640 1 -- 172.21.15.60:0/1462634503 >> v1:172.21.15.144:6789/0 conn(0x7f5348160b90 legacy=0x7f5348069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.106+0000 7f53477fe640 1 -- 172.21.15.60:0/1462634503 >> v1:172.21.15.60:6789/0 conn(0x7f534806a4b0 legacy=0x7f53480c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.106+0000 7f53467fc640 1 -- 172.21.15.60:0/1462634503 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748721603 0 0) 0x7f53480ac890 con 0x7f534806a4b0 2023-10-16T12:50:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.106+0000 7f53477fe640 1 -- 172.21.15.60:0/1462634503 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5348161930 con 0x7f534815d150 2023-10-16T12:50:05.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.107+0000 7f53477fe640 1 --2- 172.21.15.60:0/1462634503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534815d150 0x7f53480ad350 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5338009d90 tx=0x7f533802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33f8829730df143 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f53467fc640 1 -- 172.21.15.60:0/1462634503 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533802cd00 con 0x7f534815d150 2023-10-16T12:50:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f53467fc640 1 -- 172.21.15.60:0/1462634503 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533802ce60 con 0x7f534815d150 2023-10-16T12:50:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f53467fc640 1 -- 172.21.15.60:0/1462634503 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338034660 con 0x7f534815d150 2023-10-16T12:50:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 -- 172.21.15.60:0/1462634503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534815d150 msgr2=0x7f53480ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 --2- 172.21.15.60:0/1462634503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534815d150 0x7f53480ad350 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5338009d90 tx=0x7f533802bf50 comp rx=0 tx=0).stop 2023-10-16T12:50:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 -- 172.21.15.60:0/1462634503 shutdown_connections 2023-10-16T12:50:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 --2- 172.21.15.60:0/1462634503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534815d150 0x7f53480ad350 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 -- 172.21.15.60:0/1462634503 >> 172.21.15.60:0/1462634503 conn(0x7f53480ce9b0 msgr2=0x7f53480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 -- 172.21.15.60:0/1462634503 shutdown_connections 2023-10-16T12:50:05.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.108+0000 7f534dccf640 1 -- 172.21.15.60:0/1462634503 wait complete. 2023-10-16T12:50:05.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.110+0000 7f534dccf640 1 Processor -- start 2023-10-16T12:50:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.110+0000 7f534dccf640 1 -- start start 2023-10-16T12:50:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f534dccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 0x7f53481f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f534dccf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f534815d150 0x7f53480c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f534dccf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5348160b90 0x7f53480c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:05.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f534dccf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5348156770 con 0x7f534806a4b0 2023-10-16T12:50:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f534dccf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53481f6ab0 con 0x7f5348160b90 2023-10-16T12:50:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f534dccf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53481f5ea0 con 0x7f534815d150 2023-10-16T12:50:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f53477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 0x7f53481f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f53477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 0x7f53481f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53804/0 (socket says 172.21.15.60:53804) 2023-10-16T12:50:05.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.111+0000 7f53477fe640 1 -- 172.21.15.60:0/3491961586 learned_addr learned my addr 172.21.15.60:0/3491961586 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f5346ffd640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f534815d150 0x7f53480c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f5347fff640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5348160b90 0x7f53480c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f53477fe640 1 -- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f534815d150 msgr2=0x7f53480c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f53477fe640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f534815d150 0x7f53480c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f53477fe640 1 -- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5348160b90 msgr2=0x7f53480c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f53477fe640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5348160b90 0x7f53480c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f53477fe640 1 -- 172.21.15.60:0/3491961586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5338009a70 con 0x7f534806a4b0 2023-10-16T12:50:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f5346ffd640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f534815d150 0x7f53480c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.112+0000 7f53477fe640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 0x7f53481f3450 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5338000f80 tx=0x7f5338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:05.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.113+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338003b90 con 0x7f534806a4b0 2023-10-16T12:50:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.113+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338003cf0 con 0x7f534806a4b0 2023-10-16T12:50:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.113+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338004250 con 0x7f534806a4b0 2023-10-16T12:50:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.113+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53481f6180 con 0x7f534806a4b0 2023-10-16T12:50:05.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.113+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53481f6740 con 0x7f534806a4b0 2023-10-16T12:50:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.114+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534804e160 con 0x7f534806a4b0 2023-10-16T12:50:05.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.115+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f533803b050 con 0x7f534806a4b0 2023-10-16T12:50:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.116+0000 7f533ffff640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f532806af90 0x7f532806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.116+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f53380ac4f0 con 0x7f534806a4b0 2023-10-16T12:50:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.117+0000 7f5346ffd640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f532806af90 0x7f532806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.118+0000 7f5346ffd640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f532806af90 0x7f532806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f53480cd2d0 tx=0x7f5330009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:05.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.120+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53380775d0 con 0x7f534806a4b0 2023-10-16T12:50:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.351+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f53480b1300 con 0x7f534806a4b0 2023-10-16T12:50:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.901+0000 7f533ffff640 1 -- 172.21.15.60:0/3491961586 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) v1 ==== 174+0+0 (secure 0 0 0) 0x7f533807abf0 con 0x7f534806a4b0 2023-10-16T12:50:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 10 and data pool 11 2023-10-16T12:50:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f532806af90 msgr2=0x7f532806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f532806af90 0x7f532806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f53480cd2d0 tx=0x7f5330009040 comp rx=0 tx=0).stop 2023-10-16T12:50:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 msgr2=0x7f53481f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 0x7f53481f3450 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5338000f80 tx=0x7f5338004a40 comp rx=0 tx=0).stop 2023-10-16T12:50:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 shutdown_connections 2023-10-16T12:50:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f532806af90 0x7f532806d450 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f53480cd2d0 tx=0x7f5330009040 comp rx=0 tx=0).stop 2023-10-16T12:50:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f534815d150 0x7f53480c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5348160b90 0x7f53480c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 --2- 172.21.15.60:0/3491961586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f534806a4b0 0x7f53481f3450 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.909+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 >> 172.21.15.60:0/3491961586 conn(0x7f53480ce9b0 msgr2=0x7f53480c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.910+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 shutdown_connections 2023-10-16T12:50:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:05.911+0000 7f534dccf640 1 -- 172.21.15.60:0/3491961586 wait complete. 2023-10-16T12:50:05.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:06.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.128+0000 7f5903fc9640 1 Processor -- start 2023-10-16T12:50:06.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.129+0000 7f5903fc9640 1 -- start start 2023-10-16T12:50:06.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.130+0000 7f5903fc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc070fd0 0x7f58fc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.130+0000 7f5903fc9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f58fc0adf30 0x7f58fc0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.130+0000 7f5903fc9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc071450 con 0x7f58fc0adb50 2023-10-16T12:50:06.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.130+0000 7f5903fc9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc0cc600 con 0x7f58fc0adf30 2023-10-16T12:50:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.130+0000 7f5903fc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0cc8e0 con 0x7f58fc070fd0 2023-10-16T12:50:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5901d65640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f58fc0adb50 0x7f58fc06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41070/0 (socket says 172.21.15.60:41070) 2023-10-16T12:50:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5901d65640 1 -- 172.21.15.60:0/1956900721 learned_addr learned my addr 172.21.15.60:0/1956900721 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5902566640 1 --2- 172.21.15.60:0/1956900721 >> v2:172.21.15.60:3300/0 conn(0x7f58fc0adf30 0x7f58fc0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5901564640 1 --2- 172.21.15.60:0/1956900721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc070fd0 0x7f58fc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5902566640 1 -- 172.21.15.60:0/1956900721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc070fd0 msgr2=0x7f58fc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:06.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5902566640 1 --2- 172.21.15.60:0/1956900721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc070fd0 0x7f58fc0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5902566640 1 -- 172.21.15.60:0/1956900721 >> v1:172.21.15.60:6789/0 conn(0x7f58fc0adb50 legacy=0x7f58fc06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.131+0000 7f5902566640 1 -- 172.21.15.60:0/1956900721 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc0ccbc0 con 0x7f58fc0adf30 2023-10-16T12:50:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.133+0000 7f5902566640 1 --2- 172.21.15.60:0/1956900721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc0adf30 0x7f58fc0b0950 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f58f40097a0 tx=0x7f58f402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c6e56edee7298f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.133+0000 7f5900d63640 1 -- 172.21.15.60:0/1956900721 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f402d040 con 0x7f58fc0adf30 2023-10-16T12:50:06.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.133+0000 7f5900d63640 1 -- 172.21.15.60:0/1956900721 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f4031810 con 0x7f58fc0adf30 2023-10-16T12:50:06.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.133+0000 7f5900d63640 1 -- 172.21.15.60:0/1956900721 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4031ba0 con 0x7f58fc0adf30 2023-10-16T12:50:06.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.133+0000 7f5903fc9640 1 -- 172.21.15.60:0/1956900721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc0adf30 msgr2=0x7f58fc0b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.133+0000 7f5903fc9640 1 --2- 172.21.15.60:0/1956900721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc0adf30 0x7f58fc0b0950 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f58f40097a0 tx=0x7f58f402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.134+0000 7f5903fc9640 1 -- 172.21.15.60:0/1956900721 shutdown_connections 2023-10-16T12:50:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.134+0000 7f5903fc9640 1 --2- 172.21.15.60:0/1956900721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc0adf30 0x7f58fc0b0950 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.134+0000 7f5903fc9640 1 --2- 172.21.15.60:0/1956900721 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc070fd0 0x7f58fc0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.134+0000 7f5903fc9640 1 -- 172.21.15.60:0/1956900721 >> 172.21.15.60:0/1956900721 conn(0x7f58fc0cb410 msgr2=0x7f58fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:06.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.134+0000 7f5903fc9640 1 -- 172.21.15.60:0/1956900721 shutdown_connections 2023-10-16T12:50:06.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.134+0000 7f5903fc9640 1 -- 172.21.15.60:0/1956900721 wait complete. 2023-10-16T12:50:06.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.135+0000 7f5903fc9640 1 Processor -- start 2023-10-16T12:50:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.136+0000 7f5903fc9640 1 -- start start 2023-10-16T12:50:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5903fc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 0x7f58fc0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5903fc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc0adb50 0x7f58fc154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5903fc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58fc0adf30 0x7f58fc0c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5903fc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc155230 con 0x7f58fc070fd0 2023-10-16T12:50:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5903fc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0b2310 con 0x7f58fc0adf30 2023-10-16T12:50:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5903fc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0b25f0 con 0x7f58fc0adb50 2023-10-16T12:50:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 0x7f58fc0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 0x7f58fc0b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53818/0 (socket says 172.21.15.60:53818) 2023-10-16T12:50:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 -- 172.21.15.60:0/2660398541 learned_addr learned my addr 172.21.15.60:0/2660398541 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 -- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc0adb50 msgr2=0x7f58fc154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc0adb50 0x7f58fc154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 -- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58fc0adf30 msgr2=0x7f58fc0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58fc0adf30 0x7f58fc0c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.137+0000 7f5901d65640 1 -- 172.21.15.60:0/2660398541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4009400 con 0x7f58fc070fd0 2023-10-16T12:50:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.138+0000 7f5901d65640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 0x7f58fc0b33b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f58f800cba0 tx=0x7f58f800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.138+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800ad40 con 0x7f58fc070fd0 2023-10-16T12:50:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.138+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f8012400 con 0x7f58fc070fd0 2023-10-16T12:50:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.138+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800aea0 con 0x7f58fc070fd0 2023-10-16T12:50:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.138+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc0b28d0 con 0x7f58fc070fd0 2023-10-16T12:50:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.138+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc0b2e90 con 0x7f58fc070fd0 2023-10-16T12:50:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.139+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc04e160 con 0x7f58fc070fd0 2023-10-16T12:50:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.140+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f58f8007070 con 0x7f58fc070fd0 2023-10-16T12:50:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.141+0000 7f58f27fc640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f58e006af90 0x7f58e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.141+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58f8089710 con 0x7f58fc070fd0 2023-10-16T12:50:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.142+0000 7f5901564640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f58e006af90 0x7f58e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.143+0000 7f5901564640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f58e006af90 0x7f58e006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58ec006fd0 tx=0x7f58ec008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.146+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58f80546c0 con 0x7f58fc070fd0 2023-10-16T12:50:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.343+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58fc0b4910 con 0x7f58fc070fd0 2023-10-16T12:50:06.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.347+0000 7f58f27fc640 1 -- 172.21.15.60:0/2660398541 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f58f8057ce0 con 0x7f58fc070fd0 2023-10-16T12:50:06.350 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:06.350 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":98,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:05.353944+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f58e006af90 msgr2=0x7f58e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f58e006af90 0x7f58e006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58ec006fd0 tx=0x7f58ec008040 comp rx=0 tx=0).stop 2023-10-16T12:50:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 msgr2=0x7f58fc0b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 0x7f58fc0b33b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f58f800cba0 tx=0x7f58f800a430 comp rx=0 tx=0).stop 2023-10-16T12:50:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 shutdown_connections 2023-10-16T12:50:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f58e006af90 0x7f58e006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58fc0adb50 0x7f58fc154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58fc0adf30 0x7f58fc0c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 --2- 172.21.15.60:0/2660398541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58fc070fd0 0x7f58fc0b33b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.349+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 >> 172.21.15.60:0/2660398541 conn(0x7f58fc0cb410 msgr2=0x7f58fc1656b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.350+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 shutdown_connections 2023-10-16T12:50:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.350+0000 7f5903fc9640 1 -- 172.21.15.60:0/2660398541 wait complete. 2023-10-16T12:50:06.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.590+0000 7fa871a6d640 1 Processor -- start 2023-10-16T12:50:06.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.590+0000 7fa871a6d640 1 -- start start 2023-10-16T12:50:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.592+0000 7fa871a6d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 0x7fa86c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.592+0000 7fa871a6d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c15b9d0 con 0x7fa86c06c040 2023-10-16T12:50:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.592+0000 7fa871a6d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c15bcb0 con 0x7fa86c0ccfb0 2023-10-16T12:50:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.592+0000 7fa871a6d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c15bf90 con 0x7fa86c0ad970 2023-10-16T12:50:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.592+0000 7fa870a6b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa86c06c040 0x7fa86c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41072/0 (socket says 172.21.15.60:41072) 2023-10-16T12:50:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.592+0000 7fa870a6b640 1 -- 172.21.15.60:0/446540388 learned_addr learned my addr 172.21.15.60:0/446540388 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86bfff640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 0x7fa86c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631468986 0 0) 0x7fa86c15b9d0 con 0x7fa86c06c040 2023-10-16T12:50:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa85c003520 con 0x7fa86c06c040 2023-10-16T12:50:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251611529 0 0) 0x7fa86c15bcb0 con 0x7fa86c0ccfb0 2023-10-16T12:50:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa86c15b9d0 con 0x7fa86c0ccfb0 2023-10-16T12:50:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 264134560 0 0) 0x7fa85c003520 con 0x7fa86c06c040 2023-10-16T12:50:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 msgr2=0x7fa86c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:06.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 0x7fa86c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 >> v1:172.21.15.144:6789/0 conn(0x7fa86c0ccfb0 legacy=0x7fa86c0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0cc270 con 0x7fa86c06c040 2023-10-16T12:50:06.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa8600015f0 con 0x7fa86c06c040 2023-10-16T12:50:06.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.593+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 >> v1:172.21.15.60:6789/0 conn(0x7fa86c06c040 legacy=0x7fa86c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86b7fe640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 0x7fa85c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86b7fe640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c0040e0 0x7fa85c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86b7fe640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa85c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c15bf90 con 0x7fa85c004d00 2023-10-16T12:50:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c00bac0 con 0x7fa85c0040e0 2023-10-16T12:50:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c00bc80 con 0x7fa86c0ad970 2023-10-16T12:50:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa87126c640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa85c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa86bfff640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c0040e0 0x7fa85c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa87126c640 1 -- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 msgr2=0x7fa85c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa87126c640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c0ad970 0x7fa85c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa87126c640 1 -- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c0040e0 msgr2=0x7fa85c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa87126c640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c0040e0 0x7fa85c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.594+0000 7fa87126c640 1 -- 172.21.15.60:0/446540388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c003520 con 0x7fa85c004d00 2023-10-16T12:50:06.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.595+0000 7fa87126c640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa85c005180 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa864001fd0 tx=0x7fa85c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3214f677b42f1bb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864003b70 con 0x7fa85c004d00 2023-10-16T12:50:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864003cd0 con 0x7fa85c004d00 2023-10-16T12:50:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa86b7fe640 1 -- 172.21.15.60:0/446540388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640052f0 con 0x7fa85c004d00 2023-10-16T12:50:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa871a6d640 1 -- 172.21.15.60:0/446540388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 msgr2=0x7fa85c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa871a6d640 1 --2- 172.21.15.60:0/446540388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa85c005180 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa864001fd0 tx=0x7fa85c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa87126c640 1 -- 172.21.15.60:0/446540388 reap_dead start 2023-10-16T12:50:06.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa871a6d640 1 -- 172.21.15.60:0/446540388 shutdown_connections 2023-10-16T12:50:06.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa871a6d640 1 -- 172.21.15.60:0/446540388 >> 172.21.15.60:0/446540388 conn(0x7fa86c0ce6a0 msgr2=0x7fa86c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:06.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa871a6d640 1 -- 172.21.15.60:0/446540388 shutdown_connections 2023-10-16T12:50:06.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.596+0000 7fa871a6d640 1 -- 172.21.15.60:0/446540388 wait complete. 2023-10-16T12:50:06.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.598+0000 7fa871a6d640 1 Processor -- start 2023-10-16T12:50:06.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.598+0000 7fa871a6d640 1 -- start start 2023-10-16T12:50:06.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.599+0000 7fa871a6d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa86c1f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.599+0000 7fa871a6d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa86c0c3b40 0x7fa86c15da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.599+0000 7fa871a6d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c156690 0x7fa86c156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.599+0000 7fa871a6d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c0c4370 con 0x7fa85c004d00 2023-10-16T12:50:06.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.599+0000 7fa871a6d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1f43c0 con 0x7fa86c0c3b40 2023-10-16T12:50:06.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.599+0000 7fa871a6d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c157080 con 0x7fa86c156690 2023-10-16T12:50:06.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa86c1f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa86c1f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53828/0 (socket says 172.21.15.60:53828) 2023-10-16T12:50:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 -- 172.21.15.60:0/1032980835 learned_addr learned my addr 172.21.15.60:0/1032980835 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 -- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c156690 msgr2=0x7fa86c156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c156690 0x7fa86c156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 -- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa86c0c3b40 msgr2=0x7fa86c15da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa86c0c3b40 0x7fa86c15da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa86c0c3b40 0x7fa86c15da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 -- 172.21.15.60:0/1032980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0cdcd0 con 0x7fa85c004d00 2023-10-16T12:50:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa86c0c3b40 0x7fa86c15da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.600+0000 7fa870a6b640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa86c1f0e10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa86000c6b0 tx=0x7fa86000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.601+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600072d0 con 0x7fa85c004d00 2023-10-16T12:50:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.601+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa860007430 con 0x7fa85c004d00 2023-10-16T12:50:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.601+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c1f3a20 con 0x7fa85c004d00 2023-10-16T12:50:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.601+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600047e0 con 0x7fa85c004d00 2023-10-16T12:50:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.601+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c1f3fe0 con 0x7fa85c004d00 2023-10-16T12:50:06.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.602+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c06b330 con 0x7fa85c004d00 2023-10-16T12:50:06.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.603+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa860004940 con 0x7fa85c004d00 2023-10-16T12:50:06.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.604+0000 7fa8697fa640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa85006af90 0x7fa85006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:06.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.605+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa860088ec0 con 0x7fa85c004d00 2023-10-16T12:50:06.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.605+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa85006af90 0x7fa85006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:06.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.606+0000 7fa86bfff640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa85006af90 0x7fa85006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0ad520 tx=0x7fa858008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:06.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.609+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa860053e50 con 0x7fa85c004d00 2023-10-16T12:50:06.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.808+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa86c0ad7e0 con 0x7fa85c004d00 2023-10-16T12:50:06.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.810+0000 7fa8697fa640 1 -- 172.21.15.60:0/1032980835 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fa860018020 con 0x7fa85c004d00 2023-10-16T12:50:06.812 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:06.813 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":98,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:05.353944+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:06.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.812+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa85006af90 msgr2=0x7fa85006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.812+0000 7fa871a6d640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa85006af90 0x7fa85006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0ad520 tx=0x7fa858008040 comp rx=0 tx=0).stop 2023-10-16T12:50:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.812+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 msgr2=0x7fa86c1f0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.812+0000 7fa871a6d640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa86c1f0e10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa86000c6b0 tx=0x7fa86000cb70 comp rx=0 tx=0).stop 2023-10-16T12:50:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 shutdown_connections 2023-10-16T12:50:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fa85006af90 0x7fa85006d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa86c156690 0x7fa86c156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa86c0c3b40 0x7fa86c15da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 --2- 172.21.15.60:0/1032980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c004d00 0x7fa86c1f0e10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 >> 172.21.15.60:0/1032980835 conn(0x7fa86c0ce6a0 msgr2=0x7fa86c159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 shutdown_connections 2023-10-16T12:50:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:06.813+0000 7fa871a6d640 1 -- 172.21.15.60:0/1032980835 wait complete. 2023-10-16T12:50:06.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:50:07.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.061+0000 7f4aa0dcb640 1 Processor -- start 2023-10-16T12:50:07.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.061+0000 7f4aa0dcb640 1 -- start start 2023-10-16T12:50:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.062+0000 7f4aa0dcb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a9c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.062+0000 7f4aa0dcb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c15f4a0 con 0x7f4a9c0cda80 2023-10-16T12:50:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.062+0000 7f4aa0dcb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c06c960 con 0x7f4a9c0ad970 2023-10-16T12:50:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.062+0000 7f4aa0dcb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15b9d0 con 0x7f4a9c06c040 2023-10-16T12:50:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.063+0000 7f4a9bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4a9c0cda80 0x7f4a9c15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41076/0 (socket says 172.21.15.60:41076) 2023-10-16T12:50:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.063+0000 7f4a9bfff640 1 -- 172.21.15.60:0/594102699 learned_addr learned my addr 172.21.15.60:0/594102699 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.063+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a9c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.063+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061305997 0 0) 0x7f4a9c15f4a0 con 0x7f4a9c0cda80 2023-10-16T12:50:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.063+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a84003520 con 0x7f4a9c0cda80 2023-10-16T12:50:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312018180 0 0) 0x7f4a84003520 con 0x7f4a9c0cda80 2023-10-16T12:50:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 msgr2=0x7f4a9c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a9c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 >> v1:172.21.15.144:6789/0 conn(0x7f4a9c0ad970 legacy=0x7f4a9c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c15bcb0 con 0x7f4a9c0cda80 2023-10-16T12:50:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4a940019f0 con 0x7f4a9c0cda80 2023-10-16T12:50:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 >> v1:172.21.15.60:6789/0 conn(0x7f4a9c0cda80 legacy=0x7f4a9c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a840039a0 0x7f4a84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840047c0 0x7f4a84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15b9d0 con 0x7f4a840039a0 2023-10-16T12:50:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8400ba60 con 0x7f4a840047c0 2023-10-16T12:50:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8400bc20 con 0x7f4a9c06c040 2023-10-16T12:50:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a840039a0 0x7f4a84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9bfff640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.064+0000 7f4a9b7fe640 1 -- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 msgr2=0x7f4a84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.065+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a84005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.065+0000 7f4a9b7fe640 1 -- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840047c0 msgr2=0x7f4a84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:07.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.065+0000 7f4a9affd640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840047c0 0x7f4a84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.065+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840047c0 0x7f4a84004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.065+0000 7f4a9b7fe640 1 -- 172.21.15.60:0/594102699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84003520 con 0x7f4a840039a0 2023-10-16T12:50:07.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.065+0000 7f4a9bfff640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9c06c040 0x7f4a84005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:07.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.066+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a840039a0 0x7f4a84003dc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009a90 tx=0x7f4a840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babb858def8f11c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:07.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.066+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c02c660 con 0x7f4a840039a0 2023-10-16T12:50:07.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.066+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c02c7c0 con 0x7f4a840039a0 2023-10-16T12:50:07.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.066+0000 7f4a9a7fc640 1 -- 172.21.15.60:0/594102699 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c02cb50 con 0x7f4a840039a0 2023-10-16T12:50:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/594102699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a840039a0 msgr2=0x7f4a84003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/594102699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a840039a0 0x7f4a84003dc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009a90 tx=0x7f4a840057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4a9bfff640 1 -- 172.21.15.60:0/594102699 reap_dead start 2023-10-16T12:50:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/594102699 shutdown_connections 2023-10-16T12:50:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/594102699 >> 172.21.15.60:0/594102699 conn(0x7f4a9c0ce6a0 msgr2=0x7f4a9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/594102699 shutdown_connections 2023-10-16T12:50:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.067+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/594102699 wait complete. 2023-10-16T12:50:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.069+0000 7f4aa0dcb640 1 Processor -- start 2023-10-16T12:50:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.069+0000 7f4aa0dcb640 1 -- start start 2023-10-16T12:50:07.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4aa0dcb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a840039a0 0x7f4a9c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4aa0dcb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 0x7f4a9c153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4aa0dcb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a9c0c4b30 0x7f4a9c0c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4aa0dcb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0b1870 con 0x7f4a9c0b2910 2023-10-16T12:50:07.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4aa0dcb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0b1b50 con 0x7f4a9c0c4b30 2023-10-16T12:50:07.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4aa0dcb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0b1e30 con 0x7f4a840039a0 2023-10-16T12:50:07.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 0x7f4a9c153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 0x7f4a9c153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53844/0 (socket says 172.21.15.60:53844) 2023-10-16T12:50:07.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.070+0000 7f4a9affd640 1 -- 172.21.15.60:0/1634668671 learned_addr learned my addr 172.21.15.60:0/1634668671 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:07.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a840039a0 0x7f4a9c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9affd640 1 -- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a840039a0 msgr2=0x7f4a9c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9affd640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a840039a0 0x7f4a9c0b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9affd640 1 -- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a9c0c4b30 msgr2=0x7f4a9c0c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9affd640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a9c0c4b30 0x7f4a9c0c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9affd640 1 -- 172.21.15.60:0/1634668671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c0b2110 con 0x7f4a9c0b2910 2023-10-16T12:50:07.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a9affd640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 0x7f4a9c153b30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000d4c0 tx=0x7f4a9000d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:07.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90005480 con 0x7f4a9c0b2910 2023-10-16T12:50:07.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a90003610 con 0x7f4a9c0b2910 2023-10-16T12:50:07.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c0b23f0 con 0x7f4a9c0b2910 2023-10-16T12:50:07.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900039e0 con 0x7f4a9c0b2910 2023-10-16T12:50:07.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.071+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c154280 con 0x7f4a9c0b2910 2023-10-16T12:50:07.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.073+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c06b330 con 0x7f4a9c0b2910 2023-10-16T12:50:07.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.074+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a90003b40 con 0x7f4a9c0b2910 2023-10-16T12:50:07.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.074+0000 7f4a83fff640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f4a8806afc0 0x7f4a8806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:07.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.075+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a900897d0 con 0x7f4a9c0b2910 2023-10-16T12:50:07.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.075+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f4a8806afc0 0x7f4a8806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:07.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.077+0000 7f4a9b7fe640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f4a8806afc0 0x7f4a8806d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c036000 tx=0x7f4a8c030000 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:07.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.081+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a900545d0 con 0x7f4a9c0b2910 2023-10-16T12:50:07.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.322+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4a9c04e9b0 con 0x7f4a9c0b2910 2023-10-16T12:50:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.913+0000 7f4a83fff640 1 -- 172.21.15.60:0/1634668671 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4a90057bf0 con 0x7f4a9c0b2910 2023-10-16T12:50:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f4a8806afc0 msgr2=0x7f4a8806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f4a8806afc0 0x7f4a8806d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c036000 tx=0x7f4a8c030000 comp rx=0 tx=0).stop 2023-10-16T12:50:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 msgr2=0x7f4a9c153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 0x7f4a9c153b30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000d4c0 tx=0x7f4a9000d980 comp rx=0 tx=0).stop 2023-10-16T12:50:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 shutdown_connections 2023-10-16T12:50:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f4a8806afc0 0x7f4a8806d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a9c0c4b30 0x7f4a9c0c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a9c0b2910 0x7f4a9c153b30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 --2- 172.21.15.60:0/1634668671 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a840039a0 0x7f4a9c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 >> 172.21.15.60:0/1634668671 conn(0x7f4a9c0ce6a0 msgr2=0x7f4a9c15db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 shutdown_connections 2023-10-16T12:50:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:07.917+0000 7f4aa0dcb640 1 -- 172.21.15.60:0/1634668671 wait complete. 2023-10-16T12:50:07.940 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:50:07.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:50:08.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.163+0000 7f658f521640 1 Processor -- start 2023-10-16T12:50:08.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.164+0000 7f658f521640 1 -- start start 2023-10-16T12:50:08.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.165+0000 7f658f521640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65880b8480 0x7f65880cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:08.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.165+0000 7f658f521640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65880d0be0 0x7f65880cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:08.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.165+0000 7f658f521640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658815d150 con 0x7f658815cd70 2023-10-16T12:50:08.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.165+0000 7f658f521640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65880d1180 con 0x7f65880d0be0 2023-10-16T12:50:08.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.165+0000 7f658f521640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65880d1460 con 0x7f65880b8480 2023-10-16T12:50:08.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65880d0be0 0x7f65880cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:08.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65880d0be0 0x7f65880cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53846/0 (socket says 172.21.15.60:53846) 2023-10-16T12:50:08.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 -- 172.21.15.60:0/177778878 learned_addr learned my addr 172.21.15.60:0/177778878 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:08.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658e51f640 1 --1- 172.21.15.60:0/177778878 >> v1:172.21.15.60:6789/0 conn(0x7f658815cd70 0x7f65880d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41084/0 (socket says 172.21.15.60:41084) 2023-10-16T12:50:08.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658dd1e640 1 --2- 172.21.15.60:0/177778878 >> v2:172.21.15.144:3300/0 conn(0x7f65880b8480 0x7f65880cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:08.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 -- 172.21.15.60:0/177778878 >> v2:172.21.15.144:3300/0 conn(0x7f65880b8480 msgr2=0x7f65880cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:08.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 --2- 172.21.15.60:0/177778878 >> v2:172.21.15.144:3300/0 conn(0x7f65880b8480 0x7f65880cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 -- 172.21.15.60:0/177778878 >> v1:172.21.15.60:6789/0 conn(0x7f658815cd70 legacy=0x7f65880d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.166+0000 7f658ed20640 1 -- 172.21.15.60:0/177778878 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65880d1740 con 0x7f65880d0be0 2023-10-16T12:50:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.167+0000 7f658ed20640 1 --2- 172.21.15.60:0/177778878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65880d0be0 0x7f65880cdd00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6588066a00 tx=0x7f658002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab938a3d98662b76 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.168+0000 7f658d51d640 1 -- 172.21.15.60:0/177778878 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658002ca80 con 0x7f65880d0be0 2023-10-16T12:50:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.168+0000 7f658d51d640 1 -- 172.21.15.60:0/177778878 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658002cbe0 con 0x7f65880d0be0 2023-10-16T12:50:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.168+0000 7f658d51d640 1 -- 172.21.15.60:0/177778878 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800343f0 con 0x7f65880d0be0 2023-10-16T12:50:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.168+0000 7f658f521640 1 -- 172.21.15.60:0/177778878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65880d0be0 msgr2=0x7f65880cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.168+0000 7f658f521640 1 --2- 172.21.15.60:0/177778878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65880d0be0 0x7f65880cdd00 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6588066a00 tx=0x7f658002bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.169+0000 7f658f521640 1 -- 172.21.15.60:0/177778878 shutdown_connections 2023-10-16T12:50:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.169+0000 7f658f521640 1 --2- 172.21.15.60:0/177778878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65880d0be0 0x7f65880cdd00 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.169+0000 7f658f521640 1 --2- 172.21.15.60:0/177778878 >> v2:172.21.15.144:3300/0 conn(0x7f65880b8480 0x7f65880cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.169+0000 7f658f521640 1 -- 172.21.15.60:0/177778878 >> 172.21.15.60:0/177778878 conn(0x7f65880cf570 msgr2=0x7f65880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.169+0000 7f658f521640 1 -- 172.21.15.60:0/177778878 shutdown_connections 2023-10-16T12:50:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.169+0000 7f658f521640 1 -- 172.21.15.60:0/177778878 wait complete. 2023-10-16T12:50:08.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.170+0000 7f658f521640 1 Processor -- start 2023-10-16T12:50:08.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.171+0000 7f658f521640 1 -- start start 2023-10-16T12:50:08.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658f521640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65880b8480 0x7f6588158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658f521640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65880d0be0 0x7f6588156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658f521640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 0x7f65880cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658f521640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6588159610 con 0x7f658815cd70 2023-10-16T12:50:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658f521640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658815e390 con 0x7f65880b8480 2023-10-16T12:50:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658f521640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658815e670 con 0x7f65880d0be0 2023-10-16T12:50:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 0x7f65880cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 0x7f65880cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53862/0 (socket says 172.21.15.60:53862) 2023-10-16T12:50:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 -- 172.21.15.60:0/3366808372 learned_addr learned my addr 172.21.15.60:0/3366808372 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658e51f640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65880b8480 0x7f6588158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 -- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65880d0be0 msgr2=0x7f6588156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65880d0be0 0x7f6588156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 -- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65880b8480 msgr2=0x7f6588158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65880b8480 0x7f6588158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658ed20640 1 -- 172.21.15.60:0/3366808372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6580009b30 con 0x7f658815cd70 2023-10-16T12:50:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.172+0000 7f658e51f640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65880b8480 0x7f6588158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.173+0000 7f658ed20640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 0x7f65880cc250 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6580036040 tx=0x7f6580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.173+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580009510 con 0x7f658815cd70 2023-10-16T12:50:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.173+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658003e450 con 0x7f658815cd70 2023-10-16T12:50:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.173+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658003e800 con 0x7f658815cd70 2023-10-16T12:50:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.173+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658815e950 con 0x7f658815cd70 2023-10-16T12:50:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.173+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6588159200 con 0x7f658815cd70 2023-10-16T12:50:08.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.175+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6588159390 con 0x7f658815cd70 2023-10-16T12:50:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.175+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f658002cd50 con 0x7f658815cd70 2023-10-16T12:50:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.176+0000 7f657effd640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f656c06af90 0x7f656c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.176+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65800ac200 con 0x7f658815cd70 2023-10-16T12:50:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.177+0000 7f658e51f640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f656c06af90 0x7f656c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.178+0000 7f658e51f640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f656c06af90 0x7f656c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6584000f60 tx=0x7f6584009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.181+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65800771a0 con 0x7f658815cd70 2023-10-16T12:50:08.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.429+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f658815ee10 con 0x7f658815cd70 2023-10-16T12:50:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.924+0000 7f657effd640 1 -- 172.21.15.60:0/3366808372 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f658007a7c0 con 0x7f658815cd70 2023-10-16T12:50:08.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f656c06af90 msgr2=0x7f656c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:08.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f656c06af90 0x7f656c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6584000f60 tx=0x7f6584009040 comp rx=0 tx=0).stop 2023-10-16T12:50:08.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 msgr2=0x7f65880cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:08.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 0x7f65880cc250 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6580036040 tx=0x7f6580004a40 comp rx=0 tx=0).stop 2023-10-16T12:50:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 shutdown_connections 2023-10-16T12:50:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f656c06af90 0x7f656c06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65880d0be0 0x7f6588156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f658815cd70 0x7f65880cc250 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 --2- 172.21.15.60:0/3366808372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65880b8480 0x7f6588158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 >> 172.21.15.60:0/3366808372 conn(0x7f65880cf570 msgr2=0x7f65880b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 shutdown_connections 2023-10-16T12:50:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:08.928+0000 7f658f521640 1 -- 172.21.15.60:0/3366808372 wait complete. 2023-10-16T12:50:08.953 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:50:08.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.176+0000 7f4074cca640 1 Processor -- start 2023-10-16T12:50:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.177+0000 7f4074cca640 1 -- start start 2023-10-16T12:50:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.178+0000 7f4074cca640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f407015aa60 0x7f40700af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.178+0000 7f4074cca640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40700cc9f0 0x7f40700b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.178+0000 7f4074cca640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40700b3730 con 0x7f40700afd60 2023-10-16T12:50:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.178+0000 7f4074cca640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40700b3980 con 0x7f40700cc9f0 2023-10-16T12:50:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.178+0000 7f4074cca640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40700b3c60 con 0x7f407015aa60 2023-10-16T12:50:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40700cc9f0 0x7f40700b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40700cc9f0 0x7f40700b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42636/0 (socket says 172.21.15.60:42636) 2023-10-16T12:50:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 -- 172.21.15.60:0/2078380296 learned_addr learned my addr 172.21.15.60:0/2078380296 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406f7fe640 1 --2- 172.21.15.60:0/2078380296 >> v2:172.21.15.144:3300/0 conn(0x7f407015aa60 0x7f40700af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 -- 172.21.15.60:0/2078380296 >> v2:172.21.15.144:3300/0 conn(0x7f407015aa60 msgr2=0x7f40700af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 --2- 172.21.15.60:0/2078380296 >> v2:172.21.15.144:3300/0 conn(0x7f407015aa60 0x7f40700af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 -- 172.21.15.60:0/2078380296 >> v1:172.21.15.144:6789/0 conn(0x7f40700afd60 legacy=0x7f40700cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406ffff640 1 -- 172.21.15.60:0/2078380296 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40700b3f40 con 0x7f40700cc9f0 2023-10-16T12:50:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.179+0000 7f406f7fe640 1 --2- 172.21.15.60:0/2078380296 >> v2:172.21.15.144:3300/0 conn(0x7f407015aa60 0x7f40700af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.181+0000 7f406ffff640 1 --2- 172.21.15.60:0/2078380296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700b4740 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f40700ce210 tx=0x7f406802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea10c453bd533cdd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.181+0000 7f406effd640 1 -- 172.21.15.60:0/2078380296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406802ca80 con 0x7f40700cc9f0 2023-10-16T12:50:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.181+0000 7f406effd640 1 -- 172.21.15.60:0/2078380296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406802cbe0 con 0x7f40700cc9f0 2023-10-16T12:50:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.181+0000 7f406effd640 1 -- 172.21.15.60:0/2078380296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40680343f0 con 0x7f40700cc9f0 2023-10-16T12:50:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 -- 172.21.15.60:0/2078380296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 msgr2=0x7f40700b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 --2- 172.21.15.60:0/2078380296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700b4740 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f40700ce210 tx=0x7f406802bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 -- 172.21.15.60:0/2078380296 shutdown_connections 2023-10-16T12:50:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 --2- 172.21.15.60:0/2078380296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700b4740 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 --2- 172.21.15.60:0/2078380296 >> v2:172.21.15.144:3300/0 conn(0x7f407015aa60 0x7f40700af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 -- 172.21.15.60:0/2078380296 >> 172.21.15.60:0/2078380296 conn(0x7f4070065ed0 msgr2=0x7f40700662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 -- 172.21.15.60:0/2078380296 shutdown_connections 2023-10-16T12:50:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.182+0000 7f4074cca640 1 -- 172.21.15.60:0/2078380296 wait complete. 2023-10-16T12:50:09.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.184+0000 7f4074cca640 1 Processor -- start 2023-10-16T12:50:09.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.184+0000 7f4074cca640 1 -- start start 2023-10-16T12:50:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4074cca640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40700afd60 0x7f40701efb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4074cca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4074cca640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f407015aa60 0x7f40700ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4074cca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407014b130 con 0x7f40700cc9f0 2023-10-16T12:50:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4074cca640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407015dff0 con 0x7f407015aa60 2023-10-16T12:50:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4074cca640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407015da30 con 0x7f40700afd60 2023-10-16T12:50:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700ce490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42644/0 (socket says 172.21.15.60:42644) 2023-10-16T12:50:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 -- 172.21.15.60:0/1852066945 learned_addr learned my addr 172.21.15.60:0/1852066945 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f406ffff640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f407015aa60 0x7f40700ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f406f7fe640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40700afd60 0x7f40701efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 -- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40700afd60 msgr2=0x7f40701efb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40700afd60 0x7f40701efb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 -- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f407015aa60 msgr2=0x7f40700ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f407015aa60 0x7f40700ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.185+0000 7f4067fff640 1 -- 172.21.15.60:0/1852066945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4068009b30 con 0x7f40700cc9f0 2023-10-16T12:50:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.186+0000 7f4067fff640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700ce490 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f405800c430 tx=0x7f405800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.186+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058011790 con 0x7f40700cc9f0 2023-10-16T12:50:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.186+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40580118f0 con 0x7f40700cc9f0 2023-10-16T12:50:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.186+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407015dd10 con 0x7f40700cc9f0 2023-10-16T12:50:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.186+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058011ce0 con 0x7f40700cc9f0 2023-10-16T12:50:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.186+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407015cd00 con 0x7f40700cc9f0 2023-10-16T12:50:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.187+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407014abb0 con 0x7f40700cc9f0 2023-10-16T12:50:09.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.188+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f405801c460 con 0x7f40700cc9f0 2023-10-16T12:50:09.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.190+0000 7f406cff9640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f405006af90 0x7f405006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.190+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40580896c0 con 0x7f40700cc9f0 2023-10-16T12:50:09.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.191+0000 7f406f7fe640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f405006af90 0x7f405006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.192+0000 7f406f7fe640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f405006af90 0x7f405006d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f406000b840 tx=0x7f4060008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:09.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.194+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f407014abb0 con 0x7f40700cc9f0 2023-10-16T12:50:09.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.428+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f407014abb0 con 0x7f40700cc9f0 2023-10-16T12:50:09.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.429+0000 7f406cff9640 1 -- 172.21.15.60:0/1852066945 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f407014abb0 con 0x7f40700cc9f0 2023-10-16T12:50:09.432 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:09.432 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:05.354835+0000","modified":"2023-10-16T12:50:08.431520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/4244730882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4244730882},{"type":"v1","addr":"172.21.15.60:6835","nonce":4244730882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:50:09.433 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2023-10-16T12:50:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f405006af90 msgr2=0x7f405006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f405006af90 0x7f405006d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f406000b840 tx=0x7f4060008040 comp rx=0 tx=0).stop 2023-10-16T12:50:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 msgr2=0x7f40700ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700ce490 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f405800c430 tx=0x7f405800c8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 shutdown_connections 2023-10-16T12:50:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f405006af90 0x7f405006d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f407015aa60 0x7f40700ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40700cc9f0 0x7f40700ce490 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 --2- 172.21.15.60:0/1852066945 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40700afd60 0x7f40701efb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.432+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 >> 172.21.15.60:0/1852066945 conn(0x7f4070065ed0 msgr2=0x7f40700b5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.433+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 shutdown_connections 2023-10-16T12:50:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.433+0000 7f4074cca640 1 -- 172.21.15.60:0/1852066945 wait complete. 2023-10-16T12:50:09.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.684+0000 7f578bfff640 1 Processor -- start 2023-10-16T12:50:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.684+0000 7f578bfff640 1 -- start start 2023-10-16T12:50:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.685+0000 7f578bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f578c0b8480 0x7f578c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f578bfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c15d150 con 0x7f578c15cd70 2023-10-16T12:50:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f578bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c0d1180 con 0x7f578c0d0be0 2023-10-16T12:50:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f578bfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c0d1460 con 0x7f578c0b8480 2023-10-16T12:50:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f578affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f578c15cd70 0x7f578c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49514/0 (socket says 172.21.15.60:49514) 2023-10-16T12:50:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f578affd640 1 -- 172.21.15.60:0/1035629201 learned_addr learned my addr 172.21.15.60:0/1035629201 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:09.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028705766 0 0) 0x7f578c15d150 con 0x7f578c15cd70 2023-10-16T12:50:09.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774003520 con 0x7f578c15cd70 2023-10-16T12:50:09.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.686+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1035629201 >> v2:172.21.15.144:3300/0 conn(0x7f578c0b8480 0x7f578c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1939587766 0 0) 0x7f5774003520 con 0x7f578c15cd70 2023-10-16T12:50:09.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 >> v2:172.21.15.144:3300/0 conn(0x7f578c0b8480 msgr2=0x7f578c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:09.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 --2- 172.21.15.60:0/1035629201 >> v2:172.21.15.144:3300/0 conn(0x7f578c0b8480 0x7f578c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 >> v1:172.21.15.144:6789/0 conn(0x7f578c0d0be0 legacy=0x7f578c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c0d1740 con 0x7f578c15cd70 2023-10-16T12:50:09.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f577c001670 con 0x7f578c15cd70 2023-10-16T12:50:09.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 >> v1:172.21.15.60:6789/0 conn(0x7f578c15cd70 legacy=0x7f578c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57740039c0 0x7f5774003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57740047c0 0x7f5774004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740052a0 0x7f577400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0d1460 con 0x7f57740047c0 2023-10-16T12:50:09.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577400c1c0 con 0x7f57740052a0 2023-10-16T12:50:09.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577400c380 con 0x7f57740039c0 2023-10-16T12:50:09.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.687+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57740047c0 0x7f5774004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578affd640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57740039c0 0x7f5774003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578b7fe640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740052a0 0x7f577400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578a7fc640 1 -- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57740039c0 msgr2=0x7f5774003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:09.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57740039c0 0x7f5774003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578a7fc640 1 -- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740052a0 msgr2=0x7f577400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:09.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740052a0 0x7f577400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578a7fc640 1 -- 172.21.15.60:0/1035629201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774003520 con 0x7f57740047c0 2023-10-16T12:50:09.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.688+0000 7f578b7fe640 1 -- 172.21.15.60:0/1035629201 reap_dead start 2023-10-16T12:50:09.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.689+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57740047c0 0x7f5774004c40 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f578000ba30 tx=0x7f57740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed23bc11d5ccfe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:09.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.689+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780032640 con 0x7f57740047c0 2023-10-16T12:50:09.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.689+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57800327a0 con 0x7f57740047c0 2023-10-16T12:50:09.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.689+0000 7f5789ffb640 1 -- 172.21.15.60:0/1035629201 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780032b30 con 0x7f57740047c0 2023-10-16T12:50:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 -- 172.21.15.60:0/1035629201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57740047c0 msgr2=0x7f5774004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57740047c0 0x7f5774004c40 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f578000ba30 tx=0x7f57740057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 -- 172.21.15.60:0/1035629201 shutdown_connections 2023-10-16T12:50:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 --2- 172.21.15.60:0/1035629201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57740047c0 0x7f5774004c40 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 -- 172.21.15.60:0/1035629201 >> 172.21.15.60:0/1035629201 conn(0x7f578c0cf570 msgr2=0x7f578c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:09.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 -- 172.21.15.60:0/1035629201 shutdown_connections 2023-10-16T12:50:09.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.690+0000 7f578bfff640 1 -- 172.21.15.60:0/1035629201 wait complete. 2023-10-16T12:50:09.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.692+0000 7f578bfff640 1 Processor -- start 2023-10-16T12:50:09.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.692+0000 7f578bfff640 1 -- start start 2023-10-16T12:50:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740047c0 0x7f578c0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 0x7f578c15d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f578c15f390 0x7f578c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c15dd20 con 0x7f578c15d360 2023-10-16T12:50:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c148360 con 0x7f57740047c0 2023-10-16T12:50:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c15e7e0 con 0x7f578c15f390 2023-10-16T12:50:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 0x7f578c15d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 0x7f578c15d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42656/0 (socket says 172.21.15.60:42656) 2023-10-16T12:50:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578a7fc640 1 -- 172.21.15.60:0/1440470747 learned_addr learned my addr 172.21.15.60:0/1440470747 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.693+0000 7f578affd640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740047c0 0x7f578c0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578b7fe640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f578c15f390 0x7f578c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578a7fc640 1 -- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f578c15f390 msgr2=0x7f578c15f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f578c15f390 0x7f578c15f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578a7fc640 1 -- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740047c0 msgr2=0x7f578c0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740047c0 0x7f578c0cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578a7fc640 1 -- 172.21.15.60:0/1440470747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c15fd50 con 0x7f578c15d360 2023-10-16T12:50:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578a7fc640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 0x7f578c15d7e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f578002bec0 tx=0x7f5780004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780004220 con 0x7f578c15d360 2023-10-16T12:50:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57800043a0 con 0x7f578c15d360 2023-10-16T12:50:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578003ca80 con 0x7f578c15d360 2023-10-16T12:50:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c15df40 con 0x7f578c15d360 2023-10-16T12:50:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.694+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c15e500 con 0x7f578c15d360 2023-10-16T12:50:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.696+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578c04e9b0 con 0x7f578c15d360 2023-10-16T12:50:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.697+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f578003cbe0 con 0x7f578c15d360 2023-10-16T12:50:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.697+0000 7f5773fff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f577806af90 0x7f577806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.698+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57800ac2e0 con 0x7f578c15d360 2023-10-16T12:50:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.698+0000 7f578affd640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f577806af90 0x7f577806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.699+0000 7f578affd640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f577806af90 0x7f577806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f577c001050 tx=0x7f577c009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:09.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.702+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5780077160 con 0x7f578c15d360 2023-10-16T12:50:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.907+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f578c04e9b0 con 0x7f578c15d360 2023-10-16T12:50:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.909+0000 7f5773fff640 1 -- 172.21.15.60:0/1440470747 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16556 (secure 0 0 0) 0x7f578007a780 con 0x7f578c15d360 2023-10-16T12:50:09.912 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:09.912 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":99,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:06.900850+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f577806af90 msgr2=0x7f577806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f577806af90 0x7f577806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f577c001050 tx=0x7f577c009040 comp rx=0 tx=0).stop 2023-10-16T12:50:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 msgr2=0x7f578c15d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 0x7f578c15d7e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f578002bec0 tx=0x7f5780004730 comp rx=0 tx=0).stop 2023-10-16T12:50:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 shutdown_connections 2023-10-16T12:50:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f577806af90 0x7f577806d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f578c15f390 0x7f578c15f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f578c15d360 0x7f578c15d7e0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 --2- 172.21.15.60:0/1440470747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57740047c0 0x7f578c0cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 >> 172.21.15.60:0/1440470747 conn(0x7f578c0cf570 msgr2=0x7f578c0d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 shutdown_connections 2023-10-16T12:50:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:09.912+0000 7f578bfff640 1 -- 172.21.15.60:0/1440470747 wait complete. 2023-10-16T12:50:09.935 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:50:09.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:50:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.159+0000 7fa017a1f640 1 Processor -- start 2023-10-16T12:50:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.159+0000 7fa017a1f640 1 -- start start 2023-10-16T12:50:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa017a1f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa010160420 0x7fa0100ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa017a1f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa010160800 0x7fa0100cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa017a1f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01006e170 con 0x7fa0100adc40 2023-10-16T12:50:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa017a1f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100cde30 con 0x7fa010160800 2023-10-16T12:50:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa017a1f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100cd460 con 0x7fa010160420 2023-10-16T12:50:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa014fba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa010160800 0x7fa0100cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa014fba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa010160800 0x7fa0100cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42664/0 (socket says 172.21.15.60:42664) 2023-10-16T12:50:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.160+0000 7fa014fba640 1 -- 172.21.15.60:0/2440661961 learned_addr learned my addr 172.21.15.60:0/2440661961 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:10.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.161+0000 7fa014fba640 1 -- 172.21.15.60:0/2440661961 >> v2:172.21.15.144:3300/0 conn(0x7fa010160420 msgr2=0x7fa0100ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:10.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.161+0000 7fa014fba640 1 --2- 172.21.15.60:0/2440661961 >> v2:172.21.15.144:3300/0 conn(0x7fa010160420 0x7fa0100ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.161+0000 7fa014fba640 1 -- 172.21.15.60:0/2440661961 >> v1:172.21.15.144:6789/0 conn(0x7fa0100adc40 legacy=0x7fa01006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.161+0000 7fa014fba640 1 -- 172.21.15.60:0/2440661961 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01006e3c0 con 0x7fa010160800 2023-10-16T12:50:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.162+0000 7fa014fba640 1 --2- 172.21.15.60:0/2440661961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160800 0x7fa0100cd8f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa01006f470 tx=0x7fa00002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e002d421b3622e19 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.162+0000 7fa007fff640 1 -- 172.21.15.60:0/2440661961 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000031870 con 0x7fa010160800 2023-10-16T12:50:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.162+0000 7fa007fff640 1 -- 172.21.15.60:0/2440661961 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0000319d0 con 0x7fa010160800 2023-10-16T12:50:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.162+0000 7fa007fff640 1 -- 172.21.15.60:0/2440661961 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000031d60 con 0x7fa010160800 2023-10-16T12:50:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 -- 172.21.15.60:0/2440661961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160800 msgr2=0x7fa0100cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2440661961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160800 0x7fa0100cd8f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa01006f470 tx=0x7fa00002b910 comp rx=0 tx=0).stop 2023-10-16T12:50:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 -- 172.21.15.60:0/2440661961 shutdown_connections 2023-10-16T12:50:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2440661961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160800 0x7fa0100cd8f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2440661961 >> v2:172.21.15.144:3300/0 conn(0x7fa010160420 0x7fa0100ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 -- 172.21.15.60:0/2440661961 >> 172.21.15.60:0/2440661961 conn(0x7fa0100b0d30 msgr2=0x7fa0100b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:10.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 -- 172.21.15.60:0/2440661961 shutdown_connections 2023-10-16T12:50:10.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.163+0000 7fa017a1f640 1 -- 172.21.15.60:0/2440661961 wait complete. 2023-10-16T12:50:10.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.165+0000 7fa017a1f640 1 Processor -- start 2023-10-16T12:50:10.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.165+0000 7fa017a1f640 1 -- start start 2023-10-16T12:50:10.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa017a1f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0100adc40 0x7fa01015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa017a1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 0x7fa01015db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa017a1f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa010160800 0x7fa0100afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa017a1f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100c2c10 con 0x7fa010160420 2023-10-16T12:50:10.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa017a1f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100c2eb0 con 0x7fa010160800 2023-10-16T12:50:10.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa017a1f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100c3190 con 0x7fa0100adc40 2023-10-16T12:50:10.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.166+0000 7fa014fba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 0x7fa01015db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 0x7fa01015db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42668/0 (socket says 172.21.15.60:42668) 2023-10-16T12:50:10.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 -- 172.21.15.60:0/2906410870 learned_addr learned my addr 172.21.15.60:0/2906410870 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:10.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 -- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0100adc40 msgr2=0x7fa01015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0100adc40 0x7fa01015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa015fbc640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa010160800 0x7fa0100afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 -- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa010160800 msgr2=0x7fa0100afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa010160800 0x7fa0100afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa014fba640 1 -- 172.21.15.60:0/2906410870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0000094d0 con 0x7fa010160420 2023-10-16T12:50:10.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.167+0000 7fa015fbc640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa010160800 0x7fa0100afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:10.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.168+0000 7fa014fba640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 0x7fa01015db60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fa00002b950 tx=0x7fa0000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:10.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.168+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00003ad30 con 0x7fa010160420 2023-10-16T12:50:10.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.168+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0000443f0 con 0x7fa010160420 2023-10-16T12:50:10.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.168+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000447a0 con 0x7fa010160420 2023-10-16T12:50:10.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.168+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0100c3470 con 0x7fa010160420 2023-10-16T12:50:10.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.168+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01015a5e0 con 0x7fa010160420 2023-10-16T12:50:10.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.169+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0100c6120 con 0x7fa010160420 2023-10-16T12:50:10.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.170+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa00004d450 con 0x7fa010160420 2023-10-16T12:50:10.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.171+0000 7fa005ffb640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ff406afc0 0x7f9ff406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.171+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00002d040 con 0x7fa010160420 2023-10-16T12:50:10.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.172+0000 7fa0157bb640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ff406afc0 0x7f9ff406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.173+0000 7fa0157bb640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ff406afc0 0x7f9ff406d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00ab90 tx=0x7fa00c009250 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:10.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.176+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa000076e10 con 0x7fa010160420 2023-10-16T12:50:10.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.368+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 --> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0100c3020 con 0x7f9ff406afc0 2023-10-16T12:50:10.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.397+0000 7fa005ffb640 1 -- 172.21.15.60:0/2906410870 <== mgr.5863 v2:172.21.15.144:6832/1915751278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286863 (secure 0 0 0) 0x7fa0100c3020 con 0x7f9ff406afc0 2023-10-16T12:50:10.405 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:10.406 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:50:10.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ff406afc0 msgr2=0x7f9ff406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ff406afc0 0x7f9ff406d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00ab90 tx=0x7fa00c009250 comp rx=0 tx=0).stop 2023-10-16T12:50:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 msgr2=0x7fa01015db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 0x7fa01015db60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fa00002b950 tx=0x7fa0000046c0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 shutdown_connections 2023-10-16T12:50:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f9ff406afc0 0x7f9ff406d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa010160800 0x7fa0100afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa010160420 0x7fa01015db60 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.404+0000 7fa017a1f640 1 --2- 172.21.15.60:0/2906410870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0100adc40 0x7fa01015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.405+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 >> 172.21.15.60:0/2906410870 conn(0x7fa0100b0d30 msgr2=0x7fa0100cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:10.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.405+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 shutdown_connections 2023-10-16T12:50:10.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.405+0000 7fa017a1f640 1 -- 172.21.15.60:0/2906410870 wait complete. 2023-10-16T12:50:10.428 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:50:09.126016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":307,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":341920,"kb_used_data":10596,"kb_used_omap":12,"kb_used_meta":331187,"kb_avail":754632800,"statfs":{"total":773094113280,"available":772743987200,"internally_reserved":0,"allocated":10850304,"data_stored":5751064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":339136056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.263902"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987487+0000","last_change":"2023-10-16T12:50:05.934061+0000","last_active":"2023-10-16T12:50:07.987487+0000","last_peered":"2023-10-16T12:50:07.987487+0000","last_clean":"2023-10-16T12:50:07.987487+0000","last_became_active":"2023-10-16T12:50:05.933670+0000","last_became_peered":"2023-10-16T12:50:05.933670+0000","last_unstale":"2023-10-16T12:50:07.987487+0000","last_undegraded":"2023-10-16T12:50:07.987487+0000","last_fullsized":"2023-10-16T12:50:07.987487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:36.519976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239949+0000","last_change":"2023-10-16T12:50:05.910733+0000","last_active":"2023-10-16T12:50:07.239949+0000","last_peered":"2023-10-16T12:50:07.239949+0000","last_clean":"2023-10-16T12:50:07.239949+0000","last_became_active":"2023-10-16T12:50:05.910266+0000","last_became_peered":"2023-10-16T12:50:05.910266+0000","last_unstale":"2023-10-16T12:50:07.239949+0000","last_undegraded":"2023-10-16T12:50:07.239949+0000","last_fullsized":"2023-10-16T12:50:07.239949+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:40.545491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914807+0000","last_change":"2023-10-16T12:50:05.904970+0000","last_active":"2023-10-16T12:50:06.914807+0000","last_peered":"2023-10-16T12:50:06.914807+0000","last_clean":"2023-10-16T12:50:06.914807+0000","last_became_active":"2023-10-16T12:50:05.904691+0000","last_became_peered":"2023-10-16T12:50:05.904691+0000","last_unstale":"2023-10-16T12:50:06.914807+0000","last_undegraded":"2023-10-16T12:50:06.914807+0000","last_fullsized":"2023-10-16T12:50:06.914807+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:42.657480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251175+0000","last_change":"2023-10-16T12:50:05.911077+0000","last_active":"2023-10-16T12:50:07.251175+0000","last_peered":"2023-10-16T12:50:07.251175+0000","last_clean":"2023-10-16T12:50:07.251175+0000","last_became_active":"2023-10-16T12:50:05.910038+0000","last_became_peered":"2023-10-16T12:50:05.910038+0000","last_unstale":"2023-10-16T12:50:07.251175+0000","last_undegraded":"2023-10-16T12:50:07.251175+0000","last_fullsized":"2023-10-16T12:50:07.251175+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:34.211707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888559+0000","last_change":"2023-10-16T12:50:05.930533+0000","last_active":"2023-10-16T12:50:07.888559+0000","last_peered":"2023-10-16T12:50:07.888559+0000","last_clean":"2023-10-16T12:50:07.888559+0000","last_became_active":"2023-10-16T12:50:05.930270+0000","last_became_peered":"2023-10-16T12:50:05.930270+0000","last_unstale":"2023-10-16T12:50:07.888559+0000","last_undegraded":"2023-10-16T12:50:07.888559+0000","last_fullsized":"2023-10-16T12:50:07.888559+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:48.780746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238677+0000","last_change":"2023-10-16T12:50:05.917259+0000","last_active":"2023-10-16T12:50:07.238677+0000","last_peered":"2023-10-16T12:50:07.238677+0000","last_clean":"2023-10-16T12:50:07.238677+0000","last_became_active":"2023-10-16T12:50:05.916885+0000","last_became_peered":"2023-10-16T12:50:05.916885+0000","last_unstale":"2023-10-16T12:50:07.238677+0000","last_undegraded":"2023-10-16T12:50:07.238677+0000","last_fullsized":"2023-10-16T12:50:07.238677+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:30.710940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920519+0000","last_change":"2023-10-16T12:50:05.898981+0000","last_active":"2023-10-16T12:50:06.920519+0000","last_peered":"2023-10-16T12:50:06.920519+0000","last_clean":"2023-10-16T12:50:06.920519+0000","last_became_active":"2023-10-16T12:50:05.898563+0000","last_became_peered":"2023-10-16T12:50:05.898563+0000","last_unstale":"2023-10-16T12:50:06.920519+0000","last_undegraded":"2023-10-16T12:50:06.920519+0000","last_fullsized":"2023-10-16T12:50:06.920519+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:22.063989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238923+0000","last_change":"2023-10-16T12:50:05.919024+0000","last_active":"2023-10-16T12:50:07.238923+0000","last_peered":"2023-10-16T12:50:07.238923+0000","last_clean":"2023-10-16T12:50:07.238923+0000","last_became_active":"2023-10-16T12:50:05.917739+0000","last_became_peered":"2023-10-16T12:50:05.917739+0000","last_unstale":"2023-10-16T12:50:07.238923+0000","last_undegraded":"2023-10-16T12:50:07.238923+0000","last_fullsized":"2023-10-16T12:50:07.238923+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:14.106247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888835+0000","last_change":"2023-10-16T12:50:05.929125+0000","last_active":"2023-10-16T12:50:07.888835+0000","last_peered":"2023-10-16T12:50:07.888835+0000","last_clean":"2023-10-16T12:50:07.888835+0000","last_became_active":"2023-10-16T12:50:05.928498+0000","last_became_peered":"2023-10-16T12:50:05.928498+0000","last_unstale":"2023-10-16T12:50:07.888835+0000","last_undegraded":"2023-10-16T12:50:07.888835+0000","last_fullsized":"2023-10-16T12:50:07.888835+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:22.985542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919516+0000","last_change":"2023-10-16T12:50:05.912433+0000","last_active":"2023-10-16T12:50:06.919516+0000","last_peered":"2023-10-16T12:50:06.919516+0000","last_clean":"2023-10-16T12:50:06.919516+0000","last_became_active":"2023-10-16T12:50:05.912174+0000","last_became_peered":"2023-10-16T12:50:05.912174+0000","last_unstale":"2023-10-16T12:50:06.919516+0000","last_undegraded":"2023-10-16T12:50:06.919516+0000","last_fullsized":"2023-10-16T12:50:06.919516+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:01.708959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987077+0000","last_change":"2023-10-16T12:50:05.914220+0000","last_active":"2023-10-16T12:50:07.987077+0000","last_peered":"2023-10-16T12:50:07.987077+0000","last_clean":"2023-10-16T12:50:07.987077+0000","last_became_active":"2023-10-16T12:50:05.913921+0000","last_became_peered":"2023-10-16T12:50:05.913921+0000","last_unstale":"2023-10-16T12:50:07.987077+0000","last_undegraded":"2023-10-16T12:50:07.987077+0000","last_fullsized":"2023-10-16T12:50:07.987077+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:39.457178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251036+0000","last_change":"2023-10-16T12:50:05.933150+0000","last_active":"2023-10-16T12:50:07.251036+0000","last_peered":"2023-10-16T12:50:07.251036+0000","last_clean":"2023-10-16T12:50:07.251036+0000","last_became_active":"2023-10-16T12:50:05.932861+0000","last_became_peered":"2023-10-16T12:50:05.932861+0000","last_unstale":"2023-10-16T12:50:07.251036+0000","last_undegraded":"2023-10-16T12:50:07.251036+0000","last_fullsized":"2023-10-16T12:50:07.251036+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:42.451199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985243+0000","last_change":"2023-10-16T12:50:05.913608+0000","last_active":"2023-10-16T12:50:07.985243+0000","last_peered":"2023-10-16T12:50:07.985243+0000","last_clean":"2023-10-16T12:50:07.985243+0000","last_became_active":"2023-10-16T12:50:05.913346+0000","last_became_peered":"2023-10-16T12:50:05.913346+0000","last_unstale":"2023-10-16T12:50:07.985243+0000","last_undegraded":"2023-10-16T12:50:07.985243+0000","last_fullsized":"2023-10-16T12:50:07.985243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:29.876830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888624+0000","last_change":"2023-10-16T12:50:05.931381+0000","last_active":"2023-10-16T12:50:07.888624+0000","last_peered":"2023-10-16T12:50:07.888624+0000","last_clean":"2023-10-16T12:50:07.888624+0000","last_became_active":"2023-10-16T12:50:05.929897+0000","last_became_peered":"2023-10-16T12:50:05.929897+0000","last_unstale":"2023-10-16T12:50:07.888624+0000","last_undegraded":"2023-10-16T12:50:07.888624+0000","last_fullsized":"2023-10-16T12:50:07.888624+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:22.631285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.984944+0000","last_change":"2023-10-16T12:50:05.913956+0000","last_active":"2023-10-16T12:50:07.984944+0000","last_peered":"2023-10-16T12:50:07.984944+0000","last_clean":"2023-10-16T12:50:07.984944+0000","last_became_active":"2023-10-16T12:50:05.913654+0000","last_became_peered":"2023-10-16T12:50:05.913654+0000","last_unstale":"2023-10-16T12:50:07.984944+0000","last_undegraded":"2023-10-16T12:50:07.984944+0000","last_fullsized":"2023-10-16T12:50:07.984944+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:44.658254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.917997+0000","last_change":"2023-10-16T12:50:05.917868+0000","last_active":"2023-10-16T12:50:06.917997+0000","last_peered":"2023-10-16T12:50:06.917997+0000","last_clean":"2023-10-16T12:50:06.917997+0000","last_became_active":"2023-10-16T12:50:05.917579+0000","last_became_peered":"2023-10-16T12:50:05.917579+0000","last_unstale":"2023-10-16T12:50:06.917997+0000","last_undegraded":"2023-10-16T12:50:06.917997+0000","last_fullsized":"2023-10-16T12:50:06.917997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:55.159359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888814+0000","last_change":"2023-10-16T12:50:05.928943+0000","last_active":"2023-10-16T12:50:07.888814+0000","last_peered":"2023-10-16T12:50:07.888814+0000","last_clean":"2023-10-16T12:50:07.888814+0000","last_became_active":"2023-10-16T12:50:05.928083+0000","last_became_peered":"2023-10-16T12:50:05.928083+0000","last_unstale":"2023-10-16T12:50:07.888814+0000","last_undegraded":"2023-10-16T12:50:07.888814+0000","last_fullsized":"2023-10-16T12:50:07.888814+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:20.179704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251593+0000","last_change":"2023-10-16T12:50:05.915602+0000","last_active":"2023-10-16T12:50:07.251593+0000","last_peered":"2023-10-16T12:50:07.251593+0000","last_clean":"2023-10-16T12:50:07.251593+0000","last_became_active":"2023-10-16T12:50:05.915322+0000","last_became_peered":"2023-10-16T12:50:05.915322+0000","last_unstale":"2023-10-16T12:50:07.251593+0000","last_undegraded":"2023-10-16T12:50:07.251593+0000","last_fullsized":"2023-10-16T12:50:07.251593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:17:22.249451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239484+0000","last_change":"2023-10-16T12:50:05.918803+0000","last_active":"2023-10-16T12:50:07.239484+0000","last_peered":"2023-10-16T12:50:07.239484+0000","last_clean":"2023-10-16T12:50:07.239484+0000","last_became_active":"2023-10-16T12:50:05.917735+0000","last_became_peered":"2023-10-16T12:50:05.917735+0000","last_unstale":"2023-10-16T12:50:07.239484+0000","last_undegraded":"2023-10-16T12:50:07.239484+0000","last_fullsized":"2023-10-16T12:50:07.239484+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:27.820235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985269+0000","last_change":"2023-10-16T12:50:05.904469+0000","last_active":"2023-10-16T12:50:07.985269+0000","last_peered":"2023-10-16T12:50:07.985269+0000","last_clean":"2023-10-16T12:50:07.985269+0000","last_became_active":"2023-10-16T12:50:05.903618+0000","last_became_peered":"2023-10-16T12:50:05.903618+0000","last_unstale":"2023-10-16T12:50:07.985269+0000","last_undegraded":"2023-10-16T12:50:07.985269+0000","last_fullsized":"2023-10-16T12:50:07.985269+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:04.981937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919102+0000","last_change":"2023-10-16T12:50:05.910322+0000","last_active":"2023-10-16T12:50:06.919102+0000","last_peered":"2023-10-16T12:50:06.919102+0000","last_clean":"2023-10-16T12:50:06.919102+0000","last_became_active":"2023-10-16T12:50:05.909199+0000","last_became_peered":"2023-10-16T12:50:05.909199+0000","last_unstale":"2023-10-16T12:50:06.919102+0000","last_undegraded":"2023-10-16T12:50:06.919102+0000","last_fullsized":"2023-10-16T12:50:06.919102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:38.651350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.918654+0000","last_change":"2023-10-16T12:50:05.921203+0000","last_active":"2023-10-16T12:50:06.918654+0000","last_peered":"2023-10-16T12:50:06.918654+0000","last_clean":"2023-10-16T12:50:06.918654+0000","last_became_active":"2023-10-16T12:50:05.920445+0000","last_became_peered":"2023-10-16T12:50:05.920445+0000","last_unstale":"2023-10-16T12:50:06.918654+0000","last_undegraded":"2023-10-16T12:50:06.918654+0000","last_fullsized":"2023-10-16T12:50:06.918654+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:22.157039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986487+0000","last_change":"2023-10-16T12:50:05.910485+0000","last_active":"2023-10-16T12:50:07.986487+0000","last_peered":"2023-10-16T12:50:07.986487+0000","last_clean":"2023-10-16T12:50:07.986487+0000","last_became_active":"2023-10-16T12:50:05.910157+0000","last_became_peered":"2023-10-16T12:50:05.910157+0000","last_unstale":"2023-10-16T12:50:07.986487+0000","last_undegraded":"2023-10-16T12:50:07.986487+0000","last_fullsized":"2023-10-16T12:50:07.986487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:17:53.141438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986443+0000","last_change":"2023-10-16T12:50:05.922961+0000","last_active":"2023-10-16T12:50:07.986443+0000","last_peered":"2023-10-16T12:50:07.986443+0000","last_clean":"2023-10-16T12:50:07.986443+0000","last_became_active":"2023-10-16T12:50:05.922700+0000","last_became_peered":"2023-10-16T12:50:05.922700+0000","last_unstale":"2023-10-16T12:50:07.986443+0000","last_undegraded":"2023-10-16T12:50:07.986443+0000","last_fullsized":"2023-10-16T12:50:07.986443+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:15.127530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919589+0000","last_change":"2023-10-16T12:50:05.910720+0000","last_active":"2023-10-16T12:50:06.919589+0000","last_peered":"2023-10-16T12:50:06.919589+0000","last_clean":"2023-10-16T12:50:06.919589+0000","last_became_active":"2023-10-16T12:50:05.908009+0000","last_became_peered":"2023-10-16T12:50:05.908009+0000","last_unstale":"2023-10-16T12:50:06.919589+0000","last_undegraded":"2023-10-16T12:50:06.919589+0000","last_fullsized":"2023-10-16T12:50:06.919589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:38.267965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920715+0000","last_change":"2023-10-16T12:50:05.916478+0000","last_active":"2023-10-16T12:50:06.920715+0000","last_peered":"2023-10-16T12:50:06.920715+0000","last_clean":"2023-10-16T12:50:06.920715+0000","last_became_active":"2023-10-16T12:50:05.916175+0000","last_became_peered":"2023-10-16T12:50:05.916175+0000","last_unstale":"2023-10-16T12:50:06.920715+0000","last_undegraded":"2023-10-16T12:50:06.920715+0000","last_fullsized":"2023-10-16T12:50:06.920715+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:57.966553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.915034+0000","last_change":"2023-10-16T12:50:05.908550+0000","last_active":"2023-10-16T12:50:06.915034+0000","last_peered":"2023-10-16T12:50:06.915034+0000","last_clean":"2023-10-16T12:50:06.915034+0000","last_became_active":"2023-10-16T12:50:05.908242+0000","last_became_peered":"2023-10-16T12:50:05.908242+0000","last_unstale":"2023-10-16T12:50:06.915034+0000","last_undegraded":"2023-10-16T12:50:06.915034+0000","last_fullsized":"2023-10-16T12:50:06.915034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:17.222741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919374+0000","last_change":"2023-10-16T12:50:05.931667+0000","last_active":"2023-10-16T12:50:06.919374+0000","last_peered":"2023-10-16T12:50:06.919374+0000","last_clean":"2023-10-16T12:50:06.919374+0000","last_became_active":"2023-10-16T12:50:05.930777+0000","last_became_peered":"2023-10-16T12:50:05.930777+0000","last_unstale":"2023-10-16T12:50:06.919374+0000","last_undegraded":"2023-10-16T12:50:06.919374+0000","last_fullsized":"2023-10-16T12:50:06.919374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:18.872293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914496+0000","last_change":"2023-10-16T12:50:05.902521+0000","last_active":"2023-10-16T12:50:06.914496+0000","last_peered":"2023-10-16T12:50:06.914496+0000","last_clean":"2023-10-16T12:50:06.914496+0000","last_became_active":"2023-10-16T12:50:05.902085+0000","last_became_peered":"2023-10-16T12:50:05.902085+0000","last_unstale":"2023-10-16T12:50:06.914496+0000","last_undegraded":"2023-10-16T12:50:06.914496+0000","last_fullsized":"2023-10-16T12:50:06.914496+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.034398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919803+0000","last_change":"2023-10-16T12:50:05.910068+0000","last_active":"2023-10-16T12:50:06.919803+0000","last_peered":"2023-10-16T12:50:06.919803+0000","last_clean":"2023-10-16T12:50:06.919803+0000","last_became_active":"2023-10-16T12:50:05.909772+0000","last_became_peered":"2023-10-16T12:50:05.909772+0000","last_unstale":"2023-10-16T12:50:06.919803+0000","last_undegraded":"2023-10-16T12:50:06.919803+0000","last_fullsized":"2023-10-16T12:50:06.919803+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:21.501022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921067+0000","last_change":"2023-10-16T12:50:05.912113+0000","last_active":"2023-10-16T12:50:06.921067+0000","last_peered":"2023-10-16T12:50:06.921067+0000","last_clean":"2023-10-16T12:50:06.921067+0000","last_became_active":"2023-10-16T12:50:05.911842+0000","last_became_peered":"2023-10-16T12:50:05.911842+0000","last_unstale":"2023-10-16T12:50:06.921067+0000","last_undegraded":"2023-10-16T12:50:06.921067+0000","last_fullsized":"2023-10-16T12:50:06.921067+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:04.103802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239183+0000","last_change":"2023-10-16T12:50:05.914968+0000","last_active":"2023-10-16T12:50:07.239183+0000","last_peered":"2023-10-16T12:50:07.239183+0000","last_clean":"2023-10-16T12:50:07.239183+0000","last_became_active":"2023-10-16T12:50:05.914662+0000","last_became_peered":"2023-10-16T12:50:05.914662+0000","last_unstale":"2023-10-16T12:50:07.239183+0000","last_undegraded":"2023-10-16T12:50:07.239183+0000","last_fullsized":"2023-10-16T12:50:07.239183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:26.145362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987166+0000","last_change":"2023-10-16T12:50:05.923905+0000","last_active":"2023-10-16T12:50:07.987166+0000","last_peered":"2023-10-16T12:50:07.987166+0000","last_clean":"2023-10-16T12:50:07.987166+0000","last_became_active":"2023-10-16T12:50:05.923285+0000","last_became_peered":"2023-10-16T12:50:05.923285+0000","last_unstale":"2023-10-16T12:50:07.987166+0000","last_undegraded":"2023-10-16T12:50:07.987166+0000","last_fullsized":"2023-10-16T12:50:07.987166+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.890992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987747+0000","last_change":"2023-10-16T12:50:04.912074+0000","last_active":"2023-10-16T12:50:07.987747+0000","last_peered":"2023-10-16T12:50:07.987747+0000","last_clean":"2023-10-16T12:50:07.987747+0000","last_became_active":"2023-10-16T12:50:04.911730+0000","last_became_peered":"2023-10-16T12:50:04.911730+0000","last_unstale":"2023-10-16T12:50:07.987747+0000","last_undegraded":"2023-10-16T12:50:07.987747+0000","last_fullsized":"2023-10-16T12:50:07.987747+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:48.094137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.889007+0000","last_change":"2023-10-16T12:50:05.928769+0000","last_active":"2023-10-16T12:50:07.889007+0000","last_peered":"2023-10-16T12:50:07.889007+0000","last_clean":"2023-10-16T12:50:07.889007+0000","last_became_active":"2023-10-16T12:50:05.927670+0000","last_became_peered":"2023-10-16T12:50:05.927670+0000","last_unstale":"2023-10-16T12:50:07.889007+0000","last_undegraded":"2023-10-16T12:50:07.889007+0000","last_fullsized":"2023-10-16T12:50:07.889007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:28.572915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985589+0000","last_change":"2023-10-16T12:50:04.897359+0000","last_active":"2023-10-16T12:50:07.985589+0000","last_peered":"2023-10-16T12:50:07.985589+0000","last_clean":"2023-10-16T12:50:07.985589+0000","last_became_active":"2023-10-16T12:50:04.897013+0000","last_became_peered":"2023-10-16T12:50:04.897013+0000","last_unstale":"2023-10-16T12:50:07.985589+0000","last_undegraded":"2023-10-16T12:50:07.985589+0000","last_fullsized":"2023-10-16T12:50:07.985589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:50.215116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239103+0000","last_change":"2023-10-16T12:50:05.922648+0000","last_active":"2023-10-16T12:50:07.239103+0000","last_peered":"2023-10-16T12:50:07.239103+0000","last_clean":"2023-10-16T12:50:07.239103+0000","last_became_active":"2023-10-16T12:50:05.921988+0000","last_became_peered":"2023-10-16T12:50:05.921988+0000","last_unstale":"2023-10-16T12:50:07.239103+0000","last_undegraded":"2023-10-16T12:50:07.239103+0000","last_fullsized":"2023-10-16T12:50:07.239103+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:00:56.969231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920889+0000","last_change":"2023-10-16T12:50:04.910287+0000","last_active":"2023-10-16T12:50:06.920889+0000","last_peered":"2023-10-16T12:50:06.920889+0000","last_clean":"2023-10-16T12:50:06.920889+0000","last_became_active":"2023-10-16T12:50:04.909949+0000","last_became_peered":"2023-10-16T12:50:04.909949+0000","last_unstale":"2023-10-16T12:50:06.920889+0000","last_undegraded":"2023-10-16T12:50:06.920889+0000","last_fullsized":"2023-10-16T12:50:06.920889+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:36.373869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987214+0000","last_change":"2023-10-16T12:50:05.916676+0000","last_active":"2023-10-16T12:50:07.987214+0000","last_peered":"2023-10-16T12:50:07.987214+0000","last_clean":"2023-10-16T12:50:07.987214+0000","last_became_active":"2023-10-16T12:50:05.916359+0000","last_became_peered":"2023-10-16T12:50:05.916359+0000","last_unstale":"2023-10-16T12:50:07.987214+0000","last_undegraded":"2023-10-16T12:50:07.987214+0000","last_fullsized":"2023-10-16T12:50:07.987214+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:42.657480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239436+0000","last_change":"2023-10-16T12:50:04.909778+0000","last_active":"2023-10-16T12:50:07.239436+0000","last_peered":"2023-10-16T12:50:07.239436+0000","last_clean":"2023-10-16T12:50:07.239436+0000","last_became_active":"2023-10-16T12:50:04.909153+0000","last_became_peered":"2023-10-16T12:50:04.909153+0000","last_unstale":"2023-10-16T12:50:07.239436+0000","last_undegraded":"2023-10-16T12:50:07.239436+0000","last_fullsized":"2023-10-16T12:50:07.239436+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:25.644099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920154+0000","last_change":"2023-10-16T12:50:05.905025+0000","last_active":"2023-10-16T12:50:06.920154+0000","last_peered":"2023-10-16T12:50:06.920154+0000","last_clean":"2023-10-16T12:50:06.920154+0000","last_became_active":"2023-10-16T12:50:05.904728+0000","last_became_peered":"2023-10-16T12:50:05.904728+0000","last_unstale":"2023-10-16T12:50:06.920154+0000","last_undegraded":"2023-10-16T12:50:06.920154+0000","last_fullsized":"2023-10-16T12:50:06.920154+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:43.914172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914841+0000","last_change":"2023-10-16T12:50:04.905258+0000","last_active":"2023-10-16T12:50:06.914841+0000","last_peered":"2023-10-16T12:50:06.914841+0000","last_clean":"2023-10-16T12:50:06.914841+0000","last_became_active":"2023-10-16T12:50:04.904945+0000","last_became_peered":"2023-10-16T12:50:04.904945+0000","last_unstale":"2023-10-16T12:50:06.914841+0000","last_undegraded":"2023-10-16T12:50:06.914841+0000","last_fullsized":"2023-10-16T12:50:06.914841+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:14.113323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919471+0000","last_change":"2023-10-16T12:50:05.918386+0000","last_active":"2023-10-16T12:50:06.919471+0000","last_peered":"2023-10-16T12:50:06.919471+0000","last_clean":"2023-10-16T12:50:06.919471+0000","last_became_active":"2023-10-16T12:50:05.918129+0000","last_became_peered":"2023-10-16T12:50:05.918129+0000","last_unstale":"2023-10-16T12:50:06.919471+0000","last_undegraded":"2023-10-16T12:50:06.919471+0000","last_fullsized":"2023-10-16T12:50:06.919471+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:52.447077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238599+0000","last_change":"2023-10-16T12:50:04.908491+0000","last_active":"2023-10-16T12:50:07.238599+0000","last_peered":"2023-10-16T12:50:07.238599+0000","last_clean":"2023-10-16T12:50:07.238599+0000","last_became_active":"2023-10-16T12:50:04.907899+0000","last_became_peered":"2023-10-16T12:50:04.907899+0000","last_unstale":"2023-10-16T12:50:07.238599+0000","last_undegraded":"2023-10-16T12:50:07.238599+0000","last_fullsized":"2023-10-16T12:50:07.238599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:41.094211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.887643+0000","last_change":"2023-10-16T12:50:05.917547+0000","last_active":"2023-10-16T12:50:07.887643+0000","last_peered":"2023-10-16T12:50:07.887643+0000","last_clean":"2023-10-16T12:50:07.887643+0000","last_became_active":"2023-10-16T12:50:05.916215+0000","last_became_peered":"2023-10-16T12:50:05.916215+0000","last_unstale":"2023-10-16T12:50:07.887643+0000","last_undegraded":"2023-10-16T12:50:07.887643+0000","last_fullsized":"2023-10-16T12:50:07.887643+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:01.691091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250583+0000","last_change":"2023-10-16T12:50:04.905235+0000","last_active":"2023-10-16T12:50:07.250583+0000","last_peered":"2023-10-16T12:50:07.250583+0000","last_clean":"2023-10-16T12:50:07.250583+0000","last_became_active":"2023-10-16T12:50:04.904889+0000","last_became_peered":"2023-10-16T12:50:04.904889+0000","last_unstale":"2023-10-16T12:50:07.250583+0000","last_undegraded":"2023-10-16T12:50:07.250583+0000","last_fullsized":"2023-10-16T12:50:07.250583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:27.558708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985060+0000","last_change":"2023-10-16T12:50:05.907559+0000","last_active":"2023-10-16T12:50:07.985060+0000","last_peered":"2023-10-16T12:50:07.985060+0000","last_clean":"2023-10-16T12:50:07.985060+0000","last_became_active":"2023-10-16T12:50:05.907274+0000","last_became_peered":"2023-10-16T12:50:05.907274+0000","last_unstale":"2023-10-16T12:50:07.985060+0000","last_undegraded":"2023-10-16T12:50:07.985060+0000","last_fullsized":"2023-10-16T12:50:07.985060+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:35.209542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.887678+0000","last_change":"2023-10-16T12:50:04.899018+0000","last_active":"2023-10-16T12:50:07.887678+0000","last_peered":"2023-10-16T12:50:07.887678+0000","last_clean":"2023-10-16T12:50:07.887678+0000","last_became_active":"2023-10-16T12:50:04.896458+0000","last_became_peered":"2023-10-16T12:50:04.896458+0000","last_unstale":"2023-10-16T12:50:07.887678+0000","last_undegraded":"2023-10-16T12:50:07.887678+0000","last_fullsized":"2023-10-16T12:50:07.887678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:37.714992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921968+0000","last_change":"2023-10-16T12:50:05.911458+0000","last_active":"2023-10-16T12:50:06.921968+0000","last_peered":"2023-10-16T12:50:06.921968+0000","last_clean":"2023-10-16T12:50:06.921968+0000","last_became_active":"2023-10-16T12:50:05.911165+0000","last_became_peered":"2023-10-16T12:50:05.911165+0000","last_unstale":"2023-10-16T12:50:06.921968+0000","last_undegraded":"2023-10-16T12:50:06.921968+0000","last_fullsized":"2023-10-16T12:50:06.921968+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:27.454739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919819+0000","last_change":"2023-10-16T12:50:04.899940+0000","last_active":"2023-10-16T12:50:06.919819+0000","last_peered":"2023-10-16T12:50:06.919819+0000","last_clean":"2023-10-16T12:50:06.919819+0000","last_became_active":"2023-10-16T12:50:04.899651+0000","last_became_peered":"2023-10-16T12:50:04.899651+0000","last_unstale":"2023-10-16T12:50:06.919819+0000","last_undegraded":"2023-10-16T12:50:06.919819+0000","last_fullsized":"2023-10-16T12:50:06.919819+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:19.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987398+0000","last_change":"2023-10-16T12:50:05.911584+0000","last_active":"2023-10-16T12:50:07.987398+0000","last_peered":"2023-10-16T12:50:07.987398+0000","last_clean":"2023-10-16T12:50:07.987398+0000","last_became_active":"2023-10-16T12:50:05.911243+0000","last_became_peered":"2023-10-16T12:50:05.911243+0000","last_unstale":"2023-10-16T12:50:07.987398+0000","last_undegraded":"2023-10-16T12:50:07.987398+0000","last_fullsized":"2023-10-16T12:50:07.987398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:42.818400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239961+0000","last_change":"2023-10-16T12:50:04.904736+0000","last_active":"2023-10-16T12:50:07.239961+0000","last_peered":"2023-10-16T12:50:07.239961+0000","last_clean":"2023-10-16T12:50:07.239961+0000","last_became_active":"2023-10-16T12:50:04.903775+0000","last_became_peered":"2023-10-16T12:50:04.903775+0000","last_unstale":"2023-10-16T12:50:07.239961+0000","last_undegraded":"2023-10-16T12:50:07.239961+0000","last_fullsized":"2023-10-16T12:50:07.239961+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:28.502627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239388+0000","last_change":"2023-10-16T12:50:05.908674+0000","last_active":"2023-10-16T12:50:07.239388+0000","last_peered":"2023-10-16T12:50:07.239388+0000","last_clean":"2023-10-16T12:50:07.239388+0000","last_became_active":"2023-10-16T12:50:05.908283+0000","last_became_peered":"2023-10-16T12:50:05.908283+0000","last_unstale":"2023-10-16T12:50:07.239388+0000","last_undegraded":"2023-10-16T12:50:07.239388+0000","last_fullsized":"2023-10-16T12:50:07.239388+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:51.067027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"98'3","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914916+0000","last_change":"2023-10-16T12:50:04.903704+0000","last_active":"2023-10-16T12:50:06.914916+0000","last_peered":"2023-10-16T12:50:06.914916+0000","last_clean":"2023-10-16T12:50:06.914916+0000","last_became_active":"2023-10-16T12:50:04.902242+0000","last_became_peered":"2023-10-16T12:50:04.902242+0000","last_unstale":"2023-10-16T12:50:06.914916+0000","last_undegraded":"2023-10-16T12:50:06.914916+0000","last_fullsized":"2023-10-16T12:50:06.914916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:11.966999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888077+0000","last_change":"2023-10-16T12:50:05.921318+0000","last_active":"2023-10-16T12:50:07.888077+0000","last_peered":"2023-10-16T12:50:07.888077+0000","last_clean":"2023-10-16T12:50:07.888077+0000","last_became_active":"2023-10-16T12:50:05.920593+0000","last_became_peered":"2023-10-16T12:50:05.920593+0000","last_unstale":"2023-10-16T12:50:07.888077+0000","last_undegraded":"2023-10-16T12:50:07.888077+0000","last_fullsized":"2023-10-16T12:50:07.888077+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:04.287816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"98'6","reported_seq":18,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251347+0000","last_change":"2023-10-16T12:50:04.899374+0000","last_active":"2023-10-16T12:50:07.251347+0000","last_peered":"2023-10-16T12:50:07.251347+0000","last_clean":"2023-10-16T12:50:07.251347+0000","last_became_active":"2023-10-16T12:50:04.899056+0000","last_became_peered":"2023-10-16T12:50:04.899056+0000","last_unstale":"2023-10-16T12:50:07.251347+0000","last_undegraded":"2023-10-16T12:50:07.251347+0000","last_fullsized":"2023-10-16T12:50:07.251347+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:23.241467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986921+0000","last_change":"2023-10-16T12:50:05.930835+0000","last_active":"2023-10-16T12:50:07.986921+0000","last_peered":"2023-10-16T12:50:07.986921+0000","last_clean":"2023-10-16T12:50:07.986921+0000","last_became_active":"2023-10-16T12:50:05.930564+0000","last_became_peered":"2023-10-16T12:50:05.930564+0000","last_unstale":"2023-10-16T12:50:07.986921+0000","last_undegraded":"2023-10-16T12:50:07.986921+0000","last_fullsized":"2023-10-16T12:50:07.986921+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:42.188513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250865+0000","last_change":"2023-10-16T12:50:04.901049+0000","last_active":"2023-10-16T12:50:07.250865+0000","last_peered":"2023-10-16T12:50:07.250865+0000","last_clean":"2023-10-16T12:50:07.250865+0000","last_became_active":"2023-10-16T12:50:04.900768+0000","last_became_peered":"2023-10-16T12:50:04.900768+0000","last_unstale":"2023-10-16T12:50:07.250865+0000","last_undegraded":"2023-10-16T12:50:07.250865+0000","last_fullsized":"2023-10-16T12:50:07.250865+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:07.708543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920355+0000","last_change":"2023-10-16T12:50:05.903183+0000","last_active":"2023-10-16T12:50:06.920355+0000","last_peered":"2023-10-16T12:50:06.920355+0000","last_clean":"2023-10-16T12:50:06.920355+0000","last_became_active":"2023-10-16T12:50:05.902766+0000","last_became_peered":"2023-10-16T12:50:05.902766+0000","last_unstale":"2023-10-16T12:50:06.920355+0000","last_undegraded":"2023-10-16T12:50:06.920355+0000","last_fullsized":"2023-10-16T12:50:06.920355+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:27.484829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238854+0000","last_change":"2023-10-16T12:50:04.909403+0000","last_active":"2023-10-16T12:50:07.238854+0000","last_peered":"2023-10-16T12:50:07.238854+0000","last_clean":"2023-10-16T12:50:07.238854+0000","last_became_active":"2023-10-16T12:50:04.906556+0000","last_became_peered":"2023-10-16T12:50:04.906556+0000","last_unstale":"2023-10-16T12:50:07.238854+0000","last_undegraded":"2023-10-16T12:50:07.238854+0000","last_fullsized":"2023-10-16T12:50:07.238854+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:11.978352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.984903+0000","last_change":"2023-10-16T12:50:05.917948+0000","last_active":"2023-10-16T12:50:07.984903+0000","last_peered":"2023-10-16T12:50:07.984903+0000","last_clean":"2023-10-16T12:50:07.984903+0000","last_became_active":"2023-10-16T12:50:05.917650+0000","last_became_peered":"2023-10-16T12:50:05.917650+0000","last_unstale":"2023-10-16T12:50:07.984903+0000","last_undegraded":"2023-10-16T12:50:07.984903+0000","last_fullsized":"2023-10-16T12:50:07.984903+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:17.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920749+0000","last_change":"2023-10-16T12:50:04.897941+0000","last_active":"2023-10-16T12:50:06.920749+0000","last_peered":"2023-10-16T12:50:06.920749+0000","last_clean":"2023-10-16T12:50:06.920749+0000","last_became_active":"2023-10-16T12:50:04.897614+0000","last_became_peered":"2023-10-16T12:50:04.897614+0000","last_unstale":"2023-10-16T12:50:06.920749+0000","last_undegraded":"2023-10-16T12:50:06.920749+0000","last_fullsized":"2023-10-16T12:50:06.920749+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:04.314762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.887654+0000","last_change":"2023-10-16T12:50:05.925498+0000","last_active":"2023-10-16T12:50:07.887654+0000","last_peered":"2023-10-16T12:50:07.887654+0000","last_clean":"2023-10-16T12:50:07.887654+0000","last_became_active":"2023-10-16T12:50:05.919426+0000","last_became_peered":"2023-10-16T12:50:05.919426+0000","last_unstale":"2023-10-16T12:50:07.887654+0000","last_undegraded":"2023-10-16T12:50:07.887654+0000","last_fullsized":"2023-10-16T12:50:07.887654+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:57.219093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.918024+0000","last_change":"2023-10-16T12:50:04.907902+0000","last_active":"2023-10-16T12:50:06.918024+0000","last_peered":"2023-10-16T12:50:06.918024+0000","last_clean":"2023-10-16T12:50:06.918024+0000","last_became_active":"2023-10-16T12:50:04.906698+0000","last_became_peered":"2023-10-16T12:50:04.906698+0000","last_unstale":"2023-10-16T12:50:06.918024+0000","last_undegraded":"2023-10-16T12:50:06.918024+0000","last_fullsized":"2023-10-16T12:50:06.918024+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:35.590819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919882+0000","last_change":"2023-10-16T12:50:05.915739+0000","last_active":"2023-10-16T12:50:06.919882+0000","last_peered":"2023-10-16T12:50:06.919882+0000","last_clean":"2023-10-16T12:50:06.919882+0000","last_became_active":"2023-10-16T12:50:05.915372+0000","last_became_peered":"2023-10-16T12:50:05.915372+0000","last_unstale":"2023-10-16T12:50:06.919882+0000","last_undegraded":"2023-10-16T12:50:06.919882+0000","last_fullsized":"2023-10-16T12:50:06.919882+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:56.309825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239857+0000","last_change":"2023-10-16T12:50:04.909488+0000","last_active":"2023-10-16T12:50:07.239857+0000","last_peered":"2023-10-16T12:50:07.239857+0000","last_clean":"2023-10-16T12:50:07.239857+0000","last_became_active":"2023-10-16T12:50:04.909134+0000","last_became_peered":"2023-10-16T12:50:04.909134+0000","last_unstale":"2023-10-16T12:50:07.239857+0000","last_undegraded":"2023-10-16T12:50:07.239857+0000","last_fullsized":"2023-10-16T12:50:07.239857+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:21.517636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987608+0000","last_change":"2023-10-16T12:50:05.915341+0000","last_active":"2023-10-16T12:50:07.987608+0000","last_peered":"2023-10-16T12:50:07.987608+0000","last_clean":"2023-10-16T12:50:07.987608+0000","last_became_active":"2023-10-16T12:50:05.915017+0000","last_became_peered":"2023-10-16T12:50:05.915017+0000","last_unstale":"2023-10-16T12:50:07.987608+0000","last_undegraded":"2023-10-16T12:50:07.987608+0000","last_fullsized":"2023-10-16T12:50:07.987608+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:57.752805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920113+0000","last_change":"2023-10-16T12:50:04.904388+0000","last_active":"2023-10-16T12:50:06.920113+0000","last_peered":"2023-10-16T12:50:06.920113+0000","last_clean":"2023-10-16T12:50:06.920113+0000","last_became_active":"2023-10-16T12:50:04.904122+0000","last_became_peered":"2023-10-16T12:50:04.904122+0000","last_unstale":"2023-10-16T12:50:06.920113+0000","last_undegraded":"2023-10-16T12:50:06.920113+0000","last_fullsized":"2023-10-16T12:50:06.920113+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:05.823130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251189+0000","last_change":"2023-10-16T12:50:05.933323+0000","last_active":"2023-10-16T12:50:07.251189+0000","last_peered":"2023-10-16T12:50:07.251189+0000","last_clean":"2023-10-16T12:50:07.251189+0000","last_became_active":"2023-10-16T12:50:05.933038+0000","last_became_peered":"2023-10-16T12:50:05.933038+0000","last_unstale":"2023-10-16T12:50:07.251189+0000","last_undegraded":"2023-10-16T12:50:07.251189+0000","last_fullsized":"2023-10-16T12:50:07.251189+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:28.709647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239670+0000","last_change":"2023-10-16T12:50:04.906298+0000","last_active":"2023-10-16T12:50:07.239670+0000","last_peered":"2023-10-16T12:50:07.239670+0000","last_clean":"2023-10-16T12:50:07.239670+0000","last_became_active":"2023-10-16T12:50:04.906002+0000","last_became_peered":"2023-10-16T12:50:04.906002+0000","last_unstale":"2023-10-16T12:50:07.239670+0000","last_undegraded":"2023-10-16T12:50:07.239670+0000","last_fullsized":"2023-10-16T12:50:07.239670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:05.586314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888472+0000","last_change":"2023-10-16T12:50:05.920923+0000","last_active":"2023-10-16T12:50:07.888472+0000","last_peered":"2023-10-16T12:50:07.888472+0000","last_clean":"2023-10-16T12:50:07.888472+0000","last_became_active":"2023-10-16T12:50:05.914388+0000","last_became_peered":"2023-10-16T12:50:05.914388+0000","last_unstale":"2023-10-16T12:50:07.888472+0000","last_undegraded":"2023-10-16T12:50:07.888472+0000","last_fullsized":"2023-10-16T12:50:07.888472+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:32.171397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986659+0000","last_change":"2023-10-16T12:50:04.912716+0000","last_active":"2023-10-16T12:50:07.986659+0000","last_peered":"2023-10-16T12:50:07.986659+0000","last_clean":"2023-10-16T12:50:07.986659+0000","last_became_active":"2023-10-16T12:50:04.912359+0000","last_became_peered":"2023-10-16T12:50:04.912359+0000","last_unstale":"2023-10-16T12:50:07.986659+0000","last_undegraded":"2023-10-16T12:50:07.986659+0000","last_fullsized":"2023-10-16T12:50:07.986659+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:11.966999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914641+0000","last_change":"2023-10-16T12:50:05.917272+0000","last_active":"2023-10-16T12:50:06.914641+0000","last_peered":"2023-10-16T12:50:06.914641+0000","last_clean":"2023-10-16T12:50:06.914641+0000","last_became_active":"2023-10-16T12:50:05.916097+0000","last_became_peered":"2023-10-16T12:50:05.916097+0000","last_unstale":"2023-10-16T12:50:06.914641+0000","last_undegraded":"2023-10-16T12:50:06.914641+0000","last_fullsized":"2023-10-16T12:50:06.914641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.890992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238915+0000","last_change":"2023-10-16T12:50:04.903020+0000","last_active":"2023-10-16T12:50:07.238915+0000","last_peered":"2023-10-16T12:50:07.238915+0000","last_clean":"2023-10-16T12:50:07.238915+0000","last_became_active":"2023-10-16T12:50:04.901234+0000","last_became_peered":"2023-10-16T12:50:04.901234+0000","last_unstale":"2023-10-16T12:50:07.238915+0000","last_undegraded":"2023-10-16T12:50:07.238915+0000","last_fullsized":"2023-10-16T12:50:07.238915+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:17.779879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914288+0000","last_change":"2023-10-16T12:50:05.918667+0000","last_active":"2023-10-16T12:50:06.914288+0000","last_peered":"2023-10-16T12:50:06.914288+0000","last_clean":"2023-10-16T12:50:06.914288+0000","last_became_active":"2023-10-16T12:50:05.918268+0000","last_became_peered":"2023-10-16T12:50:05.918268+0000","last_unstale":"2023-10-16T12:50:06.914288+0000","last_undegraded":"2023-10-16T12:50:06.914288+0000","last_fullsized":"2023-10-16T12:50:06.914288+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:17:53.141438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921300+0000","last_change":"2023-10-16T12:50:04.897323+0000","last_active":"2023-10-16T12:50:06.921300+0000","last_peered":"2023-10-16T12:50:06.921300+0000","last_clean":"2023-10-16T12:50:06.921300+0000","last_became_active":"2023-10-16T12:50:04.896993+0000","last_became_peered":"2023-10-16T12:50:04.896993+0000","last_unstale":"2023-10-16T12:50:06.921300+0000","last_undegraded":"2023-10-16T12:50:06.921300+0000","last_fullsized":"2023-10-16T12:50:06.921300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:11.773202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914579+0000","last_change":"2023-10-16T12:50:05.918396+0000","last_active":"2023-10-16T12:50:06.914579+0000","last_peered":"2023-10-16T12:50:06.914579+0000","last_clean":"2023-10-16T12:50:06.914579+0000","last_became_active":"2023-10-16T12:50:05.916755+0000","last_became_peered":"2023-10-16T12:50:05.916755+0000","last_unstale":"2023-10-16T12:50:06.914579+0000","last_undegraded":"2023-10-16T12:50:06.914579+0000","last_fullsized":"2023-10-16T12:50:06.914579+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:42.188513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.984828+0000","last_change":"2023-10-16T12:50:04.897379+0000","last_active":"2023-10-16T12:50:07.984828+0000","last_peered":"2023-10-16T12:50:07.984828+0000","last_clean":"2023-10-16T12:50:07.984828+0000","last_became_active":"2023-10-16T12:50:04.897041+0000","last_became_peered":"2023-10-16T12:50:04.897041+0000","last_unstale":"2023-10-16T12:50:07.984828+0000","last_undegraded":"2023-10-16T12:50:07.984828+0000","last_fullsized":"2023-10-16T12:50:07.984828+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:18.816111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985477+0000","last_change":"2023-10-16T12:50:05.926250+0000","last_active":"2023-10-16T12:50:07.985477+0000","last_peered":"2023-10-16T12:50:07.985477+0000","last_clean":"2023-10-16T12:50:07.985477+0000","last_became_active":"2023-10-16T12:50:05.925919+0000","last_became_peered":"2023-10-16T12:50:05.925919+0000","last_unstale":"2023-10-16T12:50:07.985477+0000","last_undegraded":"2023-10-16T12:50:07.985477+0000","last_fullsized":"2023-10-16T12:50:07.985477+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:56.764598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239915+0000","last_change":"2023-10-16T12:50:04.909166+0000","last_active":"2023-10-16T12:50:07.239915+0000","last_peered":"2023-10-16T12:50:07.239915+0000","last_clean":"2023-10-16T12:50:07.239915+0000","last_became_active":"2023-10-16T12:50:04.908875+0000","last_became_peered":"2023-10-16T12:50:04.908875+0000","last_unstale":"2023-10-16T12:50:07.239915+0000","last_undegraded":"2023-10-16T12:50:07.239915+0000","last_fullsized":"2023-10-16T12:50:07.239915+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:32.169152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921492+0000","last_change":"2023-10-16T12:50:05.900261+0000","last_active":"2023-10-16T12:50:06.921492+0000","last_peered":"2023-10-16T12:50:06.921492+0000","last_clean":"2023-10-16T12:50:06.921492+0000","last_became_active":"2023-10-16T12:50:05.899923+0000","last_became_peered":"2023-10-16T12:50:05.899923+0000","last_unstale":"2023-10-16T12:50:06.921492+0000","last_undegraded":"2023-10-16T12:50:06.921492+0000","last_fullsized":"2023-10-16T12:50:06.921492+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:14.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921718+0000","last_change":"2023-10-16T12:50:04.898402+0000","last_active":"2023-10-16T12:50:06.921718+0000","last_peered":"2023-10-16T12:50:06.921718+0000","last_clean":"2023-10-16T12:50:06.921718+0000","last_became_active":"2023-10-16T12:50:04.898114+0000","last_became_peered":"2023-10-16T12:50:04.898114+0000","last_unstale":"2023-10-16T12:50:06.921718+0000","last_undegraded":"2023-10-16T12:50:06.921718+0000","last_fullsized":"2023-10-16T12:50:06.921718+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:41.611358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986704+0000","last_change":"2023-10-16T12:50:05.921002+0000","last_active":"2023-10-16T12:50:07.986704+0000","last_peered":"2023-10-16T12:50:07.986704+0000","last_clean":"2023-10-16T12:50:07.986704+0000","last_became_active":"2023-10-16T12:50:05.920613+0000","last_became_peered":"2023-10-16T12:50:05.920613+0000","last_unstale":"2023-10-16T12:50:07.986704+0000","last_undegraded":"2023-10-16T12:50:07.986704+0000","last_fullsized":"2023-10-16T12:50:07.986704+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.034398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888395+0000","last_change":"2023-10-16T12:50:04.896080+0000","last_active":"2023-10-16T12:50:07.888395+0000","last_peered":"2023-10-16T12:50:07.888395+0000","last_clean":"2023-10-16T12:50:07.888395+0000","last_became_active":"2023-10-16T12:50:04.895802+0000","last_became_peered":"2023-10-16T12:50:04.895802+0000","last_unstale":"2023-10-16T12:50:07.888395+0000","last_undegraded":"2023-10-16T12:50:07.888395+0000","last_fullsized":"2023-10-16T12:50:07.888395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:40.040917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987368+0000","last_change":"2023-10-16T12:50:05.910412+0000","last_active":"2023-10-16T12:50:07.987368+0000","last_peered":"2023-10-16T12:50:07.987368+0000","last_clean":"2023-10-16T12:50:07.987368+0000","last_became_active":"2023-10-16T12:50:05.910121+0000","last_became_peered":"2023-10-16T12:50:05.910121+0000","last_unstale":"2023-10-16T12:50:07.987368+0000","last_undegraded":"2023-10-16T12:50:07.987368+0000","last_fullsized":"2023-10-16T12:50:07.987368+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:17.222741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985279+0000","last_change":"2023-10-16T12:50:04.901142+0000","last_active":"2023-10-16T12:50:07.985279+0000","last_peered":"2023-10-16T12:50:07.985279+0000","last_clean":"2023-10-16T12:50:07.985279+0000","last_became_active":"2023-10-16T12:50:04.900860+0000","last_became_peered":"2023-10-16T12:50:04.900860+0000","last_unstale":"2023-10-16T12:50:07.985279+0000","last_undegraded":"2023-10-16T12:50:07.985279+0000","last_fullsized":"2023-10-16T12:50:07.985279+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:11.297663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888526+0000","last_change":"2023-10-16T12:50:05.918582+0000","last_active":"2023-10-16T12:50:07.888526+0000","last_peered":"2023-10-16T12:50:07.888526+0000","last_clean":"2023-10-16T12:50:07.888526+0000","last_became_active":"2023-10-16T12:50:05.918316+0000","last_became_peered":"2023-10-16T12:50:05.918316+0000","last_unstale":"2023-10-16T12:50:07.888526+0000","last_undegraded":"2023-10-16T12:50:07.888526+0000","last_fullsized":"2023-10-16T12:50:07.888526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:46.902439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888444+0000","last_change":"2023-10-16T12:50:04.902038+0000","last_active":"2023-10-16T12:50:07.888444+0000","last_peered":"2023-10-16T12:50:07.888444+0000","last_clean":"2023-10-16T12:50:07.888444+0000","last_became_active":"2023-10-16T12:50:04.901771+0000","last_became_peered":"2023-10-16T12:50:04.901771+0000","last_unstale":"2023-10-16T12:50:07.888444+0000","last_undegraded":"2023-10-16T12:50:07.888444+0000","last_fullsized":"2023-10-16T12:50:07.888444+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:51.093009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238680+0000","last_change":"2023-10-16T12:50:05.921784+0000","last_active":"2023-10-16T12:50:07.238680+0000","last_peered":"2023-10-16T12:50:07.238680+0000","last_clean":"2023-10-16T12:50:07.238680+0000","last_became_active":"2023-10-16T12:50:05.921410+0000","last_became_peered":"2023-10-16T12:50:05.921410+0000","last_unstale":"2023-10-16T12:50:07.238680+0000","last_undegraded":"2023-10-16T12:50:07.238680+0000","last_fullsized":"2023-10-16T12:50:07.238680+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:29.623720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250454+0000","last_change":"2023-10-16T12:50:04.905982+0000","last_active":"2023-10-16T12:50:07.250454+0000","last_peered":"2023-10-16T12:50:07.250454+0000","last_clean":"2023-10-16T12:50:07.250454+0000","last_became_active":"2023-10-16T12:50:04.905738+0000","last_became_peered":"2023-10-16T12:50:04.905738+0000","last_unstale":"2023-10-16T12:50:07.250454+0000","last_undegraded":"2023-10-16T12:50:07.250454+0000","last_fullsized":"2023-10-16T12:50:07.250454+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:34.135812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250686+0000","last_change":"2023-10-16T12:50:05.907215+0000","last_active":"2023-10-16T12:50:07.250686+0000","last_peered":"2023-10-16T12:50:07.250686+0000","last_clean":"2023-10-16T12:50:07.250686+0000","last_became_active":"2023-10-16T12:50:05.906933+0000","last_became_peered":"2023-10-16T12:50:05.906933+0000","last_unstale":"2023-10-16T12:50:07.250686+0000","last_undegraded":"2023-10-16T12:50:07.250686+0000","last_fullsized":"2023-10-16T12:50:07.250686+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:16.174638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.918013+0000","last_change":"2023-10-16T12:50:04.907939+0000","last_active":"2023-10-16T12:50:06.918013+0000","last_peered":"2023-10-16T12:50:06.918013+0000","last_clean":"2023-10-16T12:50:06.918013+0000","last_became_active":"2023-10-16T12:50:04.907070+0000","last_became_peered":"2023-10-16T12:50:04.907070+0000","last_unstale":"2023-10-16T12:50:06.918013+0000","last_undegraded":"2023-10-16T12:50:06.918013+0000","last_fullsized":"2023-10-16T12:50:06.918013+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:23.780652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238587+0000","last_change":"2023-10-16T12:50:05.915065+0000","last_active":"2023-10-16T12:50:07.238587+0000","last_peered":"2023-10-16T12:50:07.238587+0000","last_clean":"2023-10-16T12:50:07.238587+0000","last_became_active":"2023-10-16T12:50:05.914753+0000","last_became_peered":"2023-10-16T12:50:05.914753+0000","last_unstale":"2023-10-16T12:50:07.238587+0000","last_undegraded":"2023-10-16T12:50:07.238587+0000","last_fullsized":"2023-10-16T12:50:07.238587+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:52.112031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239596+0000","last_change":"2023-10-16T12:50:04.905517+0000","last_active":"2023-10-16T12:50:07.239596+0000","last_peered":"2023-10-16T12:50:07.239596+0000","last_clean":"2023-10-16T12:50:07.239596+0000","last_became_active":"2023-10-16T12:50:04.905241+0000","last_became_peered":"2023-10-16T12:50:04.905241+0000","last_unstale":"2023-10-16T12:50:07.239596+0000","last_undegraded":"2023-10-16T12:50:07.239596+0000","last_fullsized":"2023-10-16T12:50:07.239596+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:50.136631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914583+0000","last_change":"2023-10-16T12:50:05.921170+0000","last_active":"2023-10-16T12:50:06.914583+0000","last_peered":"2023-10-16T12:50:06.914583+0000","last_clean":"2023-10-16T12:50:06.914583+0000","last_became_active":"2023-10-16T12:50:05.920882+0000","last_became_peered":"2023-10-16T12:50:05.920882+0000","last_unstale":"2023-10-16T12:50:06.914583+0000","last_undegraded":"2023-10-16T12:50:06.914583+0000","last_fullsized":"2023-10-16T12:50:06.914583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:39.457178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238926+0000","last_change":"2023-10-16T12:50:04.906490+0000","last_active":"2023-10-16T12:50:07.238926+0000","last_peered":"2023-10-16T12:50:07.238926+0000","last_clean":"2023-10-16T12:50:07.238926+0000","last_became_active":"2023-10-16T12:50:04.905491+0000","last_became_peered":"2023-10-16T12:50:04.905491+0000","last_unstale":"2023-10-16T12:50:07.238926+0000","last_undegraded":"2023-10-16T12:50:07.238926+0000","last_fullsized":"2023-10-16T12:50:07.238926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:19.435381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"88'175","reported_seq":447,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:50:05.900237+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:50:05.900237+0000","last_peered":"2023-10-16T12:50:05.900237+0000","last_clean":"2023-10-16T12:50:05.900237+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:50:05.900237+0000","last_undegraded":"2023-10-16T12:50:05.900237+0000","last_fullsized":"2023-10-16T12:50:05.900237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542345,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1794048,"data_stored":1157735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2609999999999999,"15min":1.2929999999999999},"min":{"1min":0.68200000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.728,"5min":1.728,"15min":1.784},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3049999999999999,"15min":1.3520000000000001},"min":{"1min":0.76100000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.772,"5min":1.8420000000000001,"15min":1.8899999999999999},"last":1.1839999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.196,"15min":1.214},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.627,"5min":1.78,"15min":1.78},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1879999999999999,"15min":1.2170000000000001},"min":{"1min":0.67600000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.87}]},{"osd":2,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1799999999999999,"15min":1.206},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.302,"15min":1.323},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.63500000000000001},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.087}]},{"osd":3,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3049999999999999,"15min":1.296},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.69099999999999995},"max":{"1min":1.5680000000000001,"5min":1.847,"15min":1.847},"last":1.135},{"interface":"front","average":{"1min":1.411,"5min":1.411,"15min":1.4059999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.0129999999999999,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.171,"15min":1.194},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.349,"15min":1.3360000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8440000000000001,"5min":1.9339999999999999,"15min":1.9359999999999999},"last":1.4930000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3240000000000001,"15min":1.3340000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.6279999999999999,"5min":1.9139999999999999,"15min":1.944},"last":1.452},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4890000000000001,"15min":1.4690000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.2769999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.5780000000000001,"15min":1.5649999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4750000000000001,"15min":1.452},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":3.1360000000000001,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.3109999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542346,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":573143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.2430000000000001,"15min":1.276},"min":{"1min":0.72299999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.829,"5min":1.829,"15min":3.7450000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2929999999999999,"15min":1.337},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.67800000000000005},"max":{"1min":1.8680000000000001,"5min":2.0270000000000001,"15min":3.819},"last":1.145}]},{"osd":1,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1919999999999999,"15min":1.1930000000000001},"min":{"1min":0.67600000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5569999999999999,"5min":1.6579999999999999,"15min":1.8979999999999999},"last":1.677},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.216,"15min":1.208},"min":{"1min":0.68100000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.637,"5min":1.821,"15min":3.8519999999999999},"last":1.181}]},{"osd":2,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2130000000000001,"15min":1.224},"min":{"1min":0.73999999999999999,"5min":0.627,"15min":0.627},"max":{"1min":1.671,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.2170000000000001,"15min":1.278},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7849999999999999,"5min":1.829,"15min":3.9009999999999998},"last":0.86399999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.333,"15min":1.3280000000000001},"min":{"1min":0.88800000000000001,"5min":0.80100000000000005,"15min":0.751},"max":{"1min":1.819,"5min":1.9399999999999999,"15min":4.2380000000000004},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2869999999999999,"15min":1.3500000000000001},"min":{"1min":0.82199999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.806,"5min":2.012,"15min":4.4269999999999996},"last":1.244}]},{"osd":4,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3560000000000001,"15min":1.3400000000000001},"min":{"1min":0.82899999999999996,"5min":0.71099999999999997,"15min":0.57699999999999996},"max":{"1min":1.9490000000000001,"5min":2.0830000000000002,"15min":2.3029999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.45,"15min":1.474},"min":{"1min":1.077,"5min":0.77300000000000002,"15min":0.76900000000000002},"max":{"1min":1.9430000000000001,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.8799999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2889999999999999,"15min":1.3089999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.849,"5min":1.9810000000000001,"15min":2.0910000000000002},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.4450000000000001,"15min":1.446},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.63900000000000001},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.7769999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.427,"15min":1.4430000000000001},"min":{"1min":0.99399999999999999,"5min":0.878,"15min":0.67900000000000005},"max":{"1min":1.8,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.446,"15min":1.452},"min":{"1min":1.0640000000000001,"5min":0.96399999999999997,"15min":0.71599999999999997},"max":{"1min":1.9690000000000001,"5min":2.1230000000000002,"15min":2.2610000000000001},"last":1.984}]}]},{"osd":5,"up_from":14,"seq":60129542347,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36520,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335320,"statfs":{"total":96636764160,"available":96599367680,"internally_reserved":0,"allocated":1208320,"data_stored":573141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.38200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.8480000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1779999999999999,"15min":1.1819999999999999},"min":{"1min":0.77800000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.645,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":0.83499999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.22,"15min":1.2549999999999999},"min":{"1min":0.69399999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.827},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1819999999999999,"15min":1.226},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.647,"5min":1.708,"15min":1.7889999999999999},"last":1.391}]},{"osd":2,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.26,"15min":1.2989999999999999},"min":{"1min":0.71599999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.591,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.044},{"interface":"front","average":{"1min":1.155,"5min":1.24,"15min":1.2669999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.554,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.26}]},{"osd":3,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.73099999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.363,"15min":1.3799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.929,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.306}]},{"osd":4,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1930000000000001,"15min":1.2050000000000001},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7430000000000001,"5min":1.996,"15min":1.996},"last":1.482},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.361,"15min":1.3140000000000001},"min":{"1min":0.55000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.0910000000000002},"last":1.4350000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.321,"15min":1.3220000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8,"5min":1.927,"15min":1.927},"last":1.536},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.3939999999999999,"15min":1.3799999999999999},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9470000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.6479999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4470000000000001,"15min":1.448},"min":{"1min":0.64300000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.3900000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5029999999999999},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.0209999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.6859999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58732,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313108,"statfs":{"total":96636764160,"available":96576622592,"internally_reserved":0,"allocated":1208320,"data_stored":573121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1799999999999999,"15min":1.2010000000000001},"min":{"1min":0.65700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4359999999999999,"5min":1.766,"15min":2.2200000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.032,"5min":1.1120000000000001,"15min":1.151},"min":{"1min":0.73499999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.569,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.375}]},{"osd":1,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.141,"15min":1.1899999999999999},"min":{"1min":0.69299999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5669999999999999,"5min":1.681,"15min":1.865},"last":1.157},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.228,"15min":1.26},"min":{"1min":0.76500000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.532,"5min":1.724,"15min":1.9390000000000001},"last":0.89000000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2549999999999999,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.81899999999999995,"15min":0.753},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.24,"5min":1.292,"15min":1.329},"min":{"1min":0.78800000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6379999999999999,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.2130000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2569999999999999,"15min":1.278},"min":{"1min":0.91600000000000004,"5min":0.80900000000000005,"15min":0.752},"max":{"1min":1.804,"5min":1.804,"15min":1.8100000000000001},"last":1.117},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.359,"15min":1.381},"min":{"1min":0.89800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.343}]},{"osd":5,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3420000000000001,"15min":1.3049999999999999},"min":{"1min":1.1699999999999999,"5min":0.86799999999999999,"15min":0.50900000000000001},"max":{"1min":1.917,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3220000000000001,"15min":1.3009999999999999},"min":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.53600000000000003},"max":{"1min":1.994,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.028}]},{"osd":6,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.407,"15min":1.391},"min":{"1min":0.92300000000000004,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.252},{"interface":"front","average":{"1min":1.569,"5min":1.488,"15min":1.4830000000000001},"min":{"1min":1.282,"5min":0.90400000000000003,"15min":0.66600000000000004},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.4870000000000001,"15min":1.4790000000000001},"min":{"1min":1.0600000000000001,"5min":0.88600000000000001,"15min":0.60399999999999998},"max":{"1min":2.109,"5min":2.109,"15min":2.254},"last":1.29},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.591,"15min":1.5960000000000001},"min":{"1min":1.026,"5min":0.94799999999999995,"15min":0.71299999999999997},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.4390000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40692,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331148,"statfs":{"total":96636764160,"available":96595095552,"internally_reserved":0,"allocated":1212416,"data_stored":573163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.91700000000000004,"15min":0.95399999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.5649999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.129},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.1180000000000001,"15min":1.169},"min":{"1min":0.44900000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.244,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.333}]},{"osd":1,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.004,"15min":1.0369999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.337,"5min":1.758,"15min":1.758},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.034,"15min":1.0740000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.45600000000000002},"max":{"1min":1.133,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.1659999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.0229999999999999,"15min":1.0740000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.1639999999999999,"5min":1.7949999999999999,"15min":1.9510000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.2689999999999999,"15min":1.3540000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.359,"5min":1.8600000000000001,"15min":1.921},"last":1.2949999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4530000000000001,"15min":1.484},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.77600000000000002},"max":{"1min":1.806,"5min":1.956,"15min":1.974},"last":1.514},{"interface":"front","average":{"1min":1.236,"5min":1.391,"15min":1.4119999999999999},"min":{"1min":0.82699999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.607,"5min":1.8939999999999999,"15min":2.0539999999999998},"last":1.254}]},{"osd":5,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.363,"15min":1.389},"min":{"1min":0.83899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5329999999999999,"5min":2.028,"15min":2.0539999999999998},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.4219999999999999,"15min":1.4670000000000001},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.80300000000000005},"max":{"1min":1.575,"5min":1.9990000000000001,"15min":2.3540000000000001},"last":1.698}]},{"osd":6,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.4139999999999999,"15min":1.4410000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.881,"5min":2.2989999999999999,"15min":2.3220000000000001},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.696,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.589}]},{"osd":7,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.512,"15min":1.5369999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6850000000000001,"5min":2.1339999999999999,"15min":2.431},"last":1.871},{"interface":"front","average":{"1min":1.381,"5min":1.5740000000000001,"15min":1.603},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.956,"5min":2.552,"15min":2.552},"last":2.0030000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542345,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44844,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326996,"statfs":{"total":96636764160,"available":96590843904,"internally_reserved":0,"allocated":1204224,"data_stored":568363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.90100000000000002,"15min":0.91600000000000004},"min":{"1min":0.44600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.494,"5min":1.742,"15min":1.7889999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.0549999999999999,"15min":1.093},"min":{"1min":0.67500000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.6579999999999999,"5min":1.7070000000000001,"15min":2.085},"last":1.3819999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1399999999999999,"15min":1.127},"min":{"1min":0.68000000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.5800000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.044,"5min":1.153,"15min":1.2210000000000001},"min":{"1min":0.56999999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.4370000000000001,"5min":2.347,"15min":2.4289999999999998},"last":1.0660000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.056,"15min":1.0820000000000001},"min":{"1min":0.502,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3839999999999999,"5min":1.6879999999999999,"15min":1.859},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.282,"15min":1.3049999999999999},"min":{"1min":0.56100000000000005,"5min":0.434,"15min":0.434},"max":{"1min":1.6000000000000001,"5min":2.1000000000000001,"15min":2.1970000000000001},"last":1.972}]},{"osd":4,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4399999999999999,"15min":1.452},"min":{"1min":1.0940000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.764,"5min":2.431,"15min":2.431},"last":1.526},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.395,"15min":1.4419999999999999},"min":{"1min":0.85099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7290000000000001,"5min":2.0670000000000002,"15min":2.2530000000000001},"last":1.181}]},{"osd":5,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.4650000000000001,"15min":1.4950000000000001},"min":{"1min":0.96299999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.7909999999999999,"5min":2.23,"15min":2.3250000000000002},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.365,"15min":1.421},"min":{"1min":0.84099999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.788,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.657}]},{"osd":6,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4059999999999999,"15min":1.4450000000000001},"min":{"1min":0.93400000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.881,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.413},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.542,"15min":1.5580000000000001},"min":{"1min":1.0640000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8400000000000001,"5min":2.294,"15min":2.3279999999999998},"last":1.8520000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.5660000000000001,"15min":1.607},"min":{"1min":0.97699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.0539999999999998,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.296},{"interface":"front","average":{"1min":1.464,"5min":1.4950000000000001,"15min":1.5449999999999999},"min":{"1min":0.88,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.944,"5min":2.573,"15min":2.573},"last":1.7789999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542345,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58748,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313092,"statfs":{"total":96636764160,"available":96576606208,"internally_reserved":0,"allocated":1224704,"data_stored":575279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90400000000000003,"15min":0.91100000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6359999999999999,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.0329999999999999,"15min":1.0860000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.7050000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.002}]},{"osd":2,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.93400000000000005,"15min":0.98899999999999999},"min":{"1min":0.46000000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.151,"5min":1.8460000000000001,"15min":2.262},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1579999999999999,"15min":1.1830000000000001},"min":{"1min":0.58399999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.9910000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.3919999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.254,"15min":1.319},"min":{"1min":0.51600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.77,"5min":1.988,"15min":2.2349999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.175,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.587,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.4490000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2749999999999999,"15min":1.3129999999999999},"min":{"1min":0.88400000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.647,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4370000000000001,"15min":1.4430000000000001},"min":{"1min":0.67900000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.6839999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.371,"15min":1.4319999999999999},"min":{"1min":0.77800000000000002,"5min":0.755,"15min":0.755},"max":{"1min":1.9390000000000001,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.77},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.411,"15min":1.431},"min":{"1min":0.93899999999999995,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":2.1179999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.5189999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.504,"15min":1.532},"min":{"1min":0.72099999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.202},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.472,"15min":1.5189999999999999},"min":{"1min":0.73899999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":2.1640000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.5620000000000001,"15min":1.617},"min":{"1min":0.82099999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.5269999999999999,"15min":1.5620000000000001},"min":{"1min":0.82599999999999996,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.9430000000000001,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.7230000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542346,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":1157119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.85299999999999998,"15min":0.89300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.655,"5min":1.655,"15min":2.5720000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.0509999999999999,"15min":1.089},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.439},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":2.1800000000000002},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.0660000000000001,"15min":1.127},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.579,"5min":1.8120000000000001,"15min":2.7650000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.1140000000000001,"15min":1.159},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.5880000000000001,"5min":1.952,"15min":1.952},"last":0.501}]},{"osd":3,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.016,"15min":1.0820000000000001},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.609,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.226,"15min":1.3009999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.696,"5min":1.8740000000000001,"15min":3.6579999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2929999999999999,"15min":1.3300000000000001},"min":{"1min":0.71899999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.542,"5min":1.8660000000000001,"15min":2.6920000000000002},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.3799999999999999,"15min":1.4390000000000001},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.845,"5min":1.966,"15min":3.544},"last":0.77800000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.377,"15min":1.4219999999999999},"min":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8320000000000001,"5min":2.2229999999999999,"15min":2.2440000000000002},"last":0.755},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3560000000000001,"15min":1.417},"min":{"1min":0.872,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.7490000000000001,"5min":2.02,"15min":2.8610000000000002},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.488,"15min":1.5289999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.9790000000000001,"5min":2.0750000000000002,"15min":3.7890000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.4470000000000001,"15min":1.516},"min":{"1min":0.92000000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.95,"5min":2.145,"15min":2.363},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5620000000000001,"15min":1.6200000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.056,"5min":2.3490000000000002,"15min":2.4729999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.389,"5min":1.4990000000000001,"15min":1.552},"min":{"1min":0.81599999999999995,"5min":0.755,"15min":0.755},"max":{"1min":1.8560000000000001,"5min":2.2050000000000001,"15min":3.1989999999999998},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:50:10.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:50:10.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.695+0000 7faaa7fff640 1 Processor -- start 2023-10-16T12:50:10.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.695+0000 7faaa7fff640 1 -- start start 2023-10-16T12:50:10.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.697+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 0x7faaa80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.697+0000 7faaa7fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa815b9d0 con 0x7faaa806c040 2023-10-16T12:50:10.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.697+0000 7faaa7fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa815bcb0 con 0x7faaa80ad970 2023-10-16T12:50:10.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.697+0000 7faaa7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa815b7d0 con 0x7faaa815b320 2023-10-16T12:50:10.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.698+0000 7faaa6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faaa806c040 0x7faaa806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49530/0 (socket says 172.21.15.60:49530) 2023-10-16T12:50:10.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.698+0000 7faaa6ffd640 1 -- 172.21.15.60:0/676224682 learned_addr learned my addr 172.21.15.60:0/676224682 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:10.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.698+0000 7faaa77fe640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 0x7faaa80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.698+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157325753 0 0) 0x7faaa815b9d0 con 0x7faaa806c040 2023-10-16T12:50:10.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.698+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa94003520 con 0x7faaa806c040 2023-10-16T12:50:10.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786747308 0 0) 0x7faa94003520 con 0x7faaa806c040 2023-10-16T12:50:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 msgr2=0x7faaa80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 0x7faaa80cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 >> v1:172.21.15.144:6789/0 conn(0x7faaa80ad970 legacy=0x7faaa80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa815bf90 con 0x7faaa806c040 2023-10-16T12:50:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7faa98001670 con 0x7faaa806c040 2023-10-16T12:50:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 >> v1:172.21.15.60:6789/0 conn(0x7faaa806c040 legacy=0x7faaa806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 0x7faa940039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faa940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa94004d00 0x7faa94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa815b7d0 con 0x7faa94003f20 2023-10-16T12:50:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9400ba60 con 0x7faa94004d00 2023-10-16T12:50:10.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9400bc20 con 0x7faaa815b320 2023-10-16T12:50:10.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faa9ffff640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faa940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.699+0000 7faaa6ffd640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 0x7faa940039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faaa77fe640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa94004d00 0x7faa94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faa9ffff640 1 -- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 msgr2=0x7faa940039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faa9ffff640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa815b320 0x7faa940039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faa9ffff640 1 -- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa94004d00 msgr2=0x7faa94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faa9ffff640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa94004d00 0x7faa94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faa9ffff640 1 -- 172.21.15.60:0/676224682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa94003520 con 0x7faa94003f20 2023-10-16T12:50:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.700+0000 7faaa77fe640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa94004d00 0x7faa94005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.701+0000 7faa9ffff640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faa940047c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7faa90001720 tx=0x7faa940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669b085efe1e0f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.702+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa900052d0 con 0x7faa94003f20 2023-10-16T12:50:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.702+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9002e050 con 0x7faa94003f20 2023-10-16T12:50:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.702+0000 7faaa67fc640 1 -- 172.21.15.60:0/676224682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90003830 con 0x7faa94003f20 2023-10-16T12:50:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.702+0000 7faaa7fff640 1 -- 172.21.15.60:0/676224682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 msgr2=0x7faa940047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.702+0000 7faaa7fff640 1 --2- 172.21.15.60:0/676224682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faa940047c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7faa90001720 tx=0x7faa940057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.703+0000 7faaa77fe640 1 -- 172.21.15.60:0/676224682 reap_dead start 2023-10-16T12:50:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.703+0000 7faaa7fff640 1 -- 172.21.15.60:0/676224682 shutdown_connections 2023-10-16T12:50:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.703+0000 7faaa7fff640 1 -- 172.21.15.60:0/676224682 >> 172.21.15.60:0/676224682 conn(0x7faaa80ce6a0 msgr2=0x7faaa80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.703+0000 7faaa7fff640 1 -- 172.21.15.60:0/676224682 shutdown_connections 2023-10-16T12:50:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.703+0000 7faaa7fff640 1 -- 172.21.15.60:0/676224682 wait complete. 2023-10-16T12:50:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.705+0000 7faaa7fff640 1 Processor -- start 2023-10-16T12:50:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.705+0000 7faaa7fff640 1 -- start start 2023-10-16T12:50:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faaa80b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa8157b20 0x7faaa8157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa80c2850 0x7faaa80c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa7fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa81584e0 con 0x7faa94003f20 2023-10-16T12:50:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa7fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80c6b00 con 0x7faaa80c2850 2023-10-16T12:50:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80c3240 con 0x7faaa8157b20 2023-10-16T12:50:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faaa80b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faaa80b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42684/0 (socket says 172.21.15.60:42684) 2023-10-16T12:50:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 -- 172.21.15.60:0/4216022726 learned_addr learned my addr 172.21.15.60:0/4216022726 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 -- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa8157b20 msgr2=0x7faaa8157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa8157b20 0x7faaa8157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 -- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa80c2850 msgr2=0x7faaa80c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa80c2850 0x7faaa80c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.706+0000 7faaa6ffd640 1 -- 172.21.15.60:0/4216022726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80c3510 con 0x7faa94003f20 2023-10-16T12:50:10.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.707+0000 7faaa6ffd640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faaa80b0860 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7faa9800e740 tx=0x7faa9800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:10.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.707+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800c660 con 0x7faa94003f20 2023-10-16T12:50:10.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.707+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9800c7c0 con 0x7faa94003f20 2023-10-16T12:50:10.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.707+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800cb70 con 0x7faa94003f20 2023-10-16T12:50:10.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.707+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa80c37f0 con 0x7faa94003f20 2023-10-16T12:50:10.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.707+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa80c7030 con 0x7faa94003f20 2023-10-16T12:50:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.708+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa806b330 con 0x7faa94003f20 2023-10-16T12:50:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.709+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa9800ccd0 con 0x7faa94003f20 2023-10-16T12:50:10.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.710+0000 7faa9f7fe640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7faa8806afe0 0x7faa8806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:10.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.710+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa980895f0 con 0x7faa94003f20 2023-10-16T12:50:10.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.711+0000 7faa9ffff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7faa8806afe0 0x7faa8806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:10.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.712+0000 7faa9ffff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7faa8806afe0 0x7faa8806d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa90033000 tx=0x7faa9002f000 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:10.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.714+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa98054600 con 0x7faa94003f20 2023-10-16T12:50:10.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.907+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 --> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faaa80cc550 con 0x7faa8806afe0 2023-10-16T12:50:10.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.930+0000 7faa9f7fe640 1 -- 172.21.15.60:0/4216022726 <== mgr.5863 v2:172.21.15.144:6832/1915751278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286863 (secure 0 0 0) 0x7faa90005340 con 0x7faa8806afe0 2023-10-16T12:50:10.934 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:10.938 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:50:10.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7faa8806afe0 msgr2=0x7faa8806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7faa8806afe0 0x7faa8806d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa90033000 tx=0x7faa9002f000 comp rx=0 tx=0).stop 2023-10-16T12:50:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 msgr2=0x7faaa80b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faaa80b0860 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7faa9800e740 tx=0x7faa9800ec00 comp rx=0 tx=0).stop 2023-10-16T12:50:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 shutdown_connections 2023-10-16T12:50:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7faa8806afe0 0x7faa8806d4a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa8157b20 0x7faaa8157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaa80c2850 0x7faaa80c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 --2- 172.21.15.60:0/4216022726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa94003f20 0x7faaa80b0860 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 >> 172.21.15.60:0/4216022726 conn(0x7faaa80ce6a0 msgr2=0x7faaa815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 shutdown_connections 2023-10-16T12:50:10.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:10.937+0000 7faaa7fff640 1 -- 172.21.15.60:0/4216022726 wait complete. 2023-10-16T12:50:10.962 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:50:09.126016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":307,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":341920,"kb_used_data":10596,"kb_used_omap":12,"kb_used_meta":331187,"kb_avail":754632800,"statfs":{"total":773094113280,"available":772743987200,"internally_reserved":0,"allocated":10850304,"data_stored":5751064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":339136056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.263902"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987487+0000","last_change":"2023-10-16T12:50:05.934061+0000","last_active":"2023-10-16T12:50:07.987487+0000","last_peered":"2023-10-16T12:50:07.987487+0000","last_clean":"2023-10-16T12:50:07.987487+0000","last_became_active":"2023-10-16T12:50:05.933670+0000","last_became_peered":"2023-10-16T12:50:05.933670+0000","last_unstale":"2023-10-16T12:50:07.987487+0000","last_undegraded":"2023-10-16T12:50:07.987487+0000","last_fullsized":"2023-10-16T12:50:07.987487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:36.519976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239949+0000","last_change":"2023-10-16T12:50:05.910733+0000","last_active":"2023-10-16T12:50:07.239949+0000","last_peered":"2023-10-16T12:50:07.239949+0000","last_clean":"2023-10-16T12:50:07.239949+0000","last_became_active":"2023-10-16T12:50:05.910266+0000","last_became_peered":"2023-10-16T12:50:05.910266+0000","last_unstale":"2023-10-16T12:50:07.239949+0000","last_undegraded":"2023-10-16T12:50:07.239949+0000","last_fullsized":"2023-10-16T12:50:07.239949+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:40.545491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914807+0000","last_change":"2023-10-16T12:50:05.904970+0000","last_active":"2023-10-16T12:50:06.914807+0000","last_peered":"2023-10-16T12:50:06.914807+0000","last_clean":"2023-10-16T12:50:06.914807+0000","last_became_active":"2023-10-16T12:50:05.904691+0000","last_became_peered":"2023-10-16T12:50:05.904691+0000","last_unstale":"2023-10-16T12:50:06.914807+0000","last_undegraded":"2023-10-16T12:50:06.914807+0000","last_fullsized":"2023-10-16T12:50:06.914807+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:42.657480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251175+0000","last_change":"2023-10-16T12:50:05.911077+0000","last_active":"2023-10-16T12:50:07.251175+0000","last_peered":"2023-10-16T12:50:07.251175+0000","last_clean":"2023-10-16T12:50:07.251175+0000","last_became_active":"2023-10-16T12:50:05.910038+0000","last_became_peered":"2023-10-16T12:50:05.910038+0000","last_unstale":"2023-10-16T12:50:07.251175+0000","last_undegraded":"2023-10-16T12:50:07.251175+0000","last_fullsized":"2023-10-16T12:50:07.251175+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:42:34.211707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888559+0000","last_change":"2023-10-16T12:50:05.930533+0000","last_active":"2023-10-16T12:50:07.888559+0000","last_peered":"2023-10-16T12:50:07.888559+0000","last_clean":"2023-10-16T12:50:07.888559+0000","last_became_active":"2023-10-16T12:50:05.930270+0000","last_became_peered":"2023-10-16T12:50:05.930270+0000","last_unstale":"2023-10-16T12:50:07.888559+0000","last_undegraded":"2023-10-16T12:50:07.888559+0000","last_fullsized":"2023-10-16T12:50:07.888559+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:48.780746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238677+0000","last_change":"2023-10-16T12:50:05.917259+0000","last_active":"2023-10-16T12:50:07.238677+0000","last_peered":"2023-10-16T12:50:07.238677+0000","last_clean":"2023-10-16T12:50:07.238677+0000","last_became_active":"2023-10-16T12:50:05.916885+0000","last_became_peered":"2023-10-16T12:50:05.916885+0000","last_unstale":"2023-10-16T12:50:07.238677+0000","last_undegraded":"2023-10-16T12:50:07.238677+0000","last_fullsized":"2023-10-16T12:50:07.238677+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:30.710940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920519+0000","last_change":"2023-10-16T12:50:05.898981+0000","last_active":"2023-10-16T12:50:06.920519+0000","last_peered":"2023-10-16T12:50:06.920519+0000","last_clean":"2023-10-16T12:50:06.920519+0000","last_became_active":"2023-10-16T12:50:05.898563+0000","last_became_peered":"2023-10-16T12:50:05.898563+0000","last_unstale":"2023-10-16T12:50:06.920519+0000","last_undegraded":"2023-10-16T12:50:06.920519+0000","last_fullsized":"2023-10-16T12:50:06.920519+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:22.063989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238923+0000","last_change":"2023-10-16T12:50:05.919024+0000","last_active":"2023-10-16T12:50:07.238923+0000","last_peered":"2023-10-16T12:50:07.238923+0000","last_clean":"2023-10-16T12:50:07.238923+0000","last_became_active":"2023-10-16T12:50:05.917739+0000","last_became_peered":"2023-10-16T12:50:05.917739+0000","last_unstale":"2023-10-16T12:50:07.238923+0000","last_undegraded":"2023-10-16T12:50:07.238923+0000","last_fullsized":"2023-10-16T12:50:07.238923+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:14.106247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888835+0000","last_change":"2023-10-16T12:50:05.929125+0000","last_active":"2023-10-16T12:50:07.888835+0000","last_peered":"2023-10-16T12:50:07.888835+0000","last_clean":"2023-10-16T12:50:07.888835+0000","last_became_active":"2023-10-16T12:50:05.928498+0000","last_became_peered":"2023-10-16T12:50:05.928498+0000","last_unstale":"2023-10-16T12:50:07.888835+0000","last_undegraded":"2023-10-16T12:50:07.888835+0000","last_fullsized":"2023-10-16T12:50:07.888835+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:22.985542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919516+0000","last_change":"2023-10-16T12:50:05.912433+0000","last_active":"2023-10-16T12:50:06.919516+0000","last_peered":"2023-10-16T12:50:06.919516+0000","last_clean":"2023-10-16T12:50:06.919516+0000","last_became_active":"2023-10-16T12:50:05.912174+0000","last_became_peered":"2023-10-16T12:50:05.912174+0000","last_unstale":"2023-10-16T12:50:06.919516+0000","last_undegraded":"2023-10-16T12:50:06.919516+0000","last_fullsized":"2023-10-16T12:50:06.919516+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:01.708959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987077+0000","last_change":"2023-10-16T12:50:05.914220+0000","last_active":"2023-10-16T12:50:07.987077+0000","last_peered":"2023-10-16T12:50:07.987077+0000","last_clean":"2023-10-16T12:50:07.987077+0000","last_became_active":"2023-10-16T12:50:05.913921+0000","last_became_peered":"2023-10-16T12:50:05.913921+0000","last_unstale":"2023-10-16T12:50:07.987077+0000","last_undegraded":"2023-10-16T12:50:07.987077+0000","last_fullsized":"2023-10-16T12:50:07.987077+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:39.457178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251036+0000","last_change":"2023-10-16T12:50:05.933150+0000","last_active":"2023-10-16T12:50:07.251036+0000","last_peered":"2023-10-16T12:50:07.251036+0000","last_clean":"2023-10-16T12:50:07.251036+0000","last_became_active":"2023-10-16T12:50:05.932861+0000","last_became_peered":"2023-10-16T12:50:05.932861+0000","last_unstale":"2023-10-16T12:50:07.251036+0000","last_undegraded":"2023-10-16T12:50:07.251036+0000","last_fullsized":"2023-10-16T12:50:07.251036+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:42.451199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985243+0000","last_change":"2023-10-16T12:50:05.913608+0000","last_active":"2023-10-16T12:50:07.985243+0000","last_peered":"2023-10-16T12:50:07.985243+0000","last_clean":"2023-10-16T12:50:07.985243+0000","last_became_active":"2023-10-16T12:50:05.913346+0000","last_became_peered":"2023-10-16T12:50:05.913346+0000","last_unstale":"2023-10-16T12:50:07.985243+0000","last_undegraded":"2023-10-16T12:50:07.985243+0000","last_fullsized":"2023-10-16T12:50:07.985243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:29.876830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888624+0000","last_change":"2023-10-16T12:50:05.931381+0000","last_active":"2023-10-16T12:50:07.888624+0000","last_peered":"2023-10-16T12:50:07.888624+0000","last_clean":"2023-10-16T12:50:07.888624+0000","last_became_active":"2023-10-16T12:50:05.929897+0000","last_became_peered":"2023-10-16T12:50:05.929897+0000","last_unstale":"2023-10-16T12:50:07.888624+0000","last_undegraded":"2023-10-16T12:50:07.888624+0000","last_fullsized":"2023-10-16T12:50:07.888624+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:22.631285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.984944+0000","last_change":"2023-10-16T12:50:05.913956+0000","last_active":"2023-10-16T12:50:07.984944+0000","last_peered":"2023-10-16T12:50:07.984944+0000","last_clean":"2023-10-16T12:50:07.984944+0000","last_became_active":"2023-10-16T12:50:05.913654+0000","last_became_peered":"2023-10-16T12:50:05.913654+0000","last_unstale":"2023-10-16T12:50:07.984944+0000","last_undegraded":"2023-10-16T12:50:07.984944+0000","last_fullsized":"2023-10-16T12:50:07.984944+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:44.658254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.917997+0000","last_change":"2023-10-16T12:50:05.917868+0000","last_active":"2023-10-16T12:50:06.917997+0000","last_peered":"2023-10-16T12:50:06.917997+0000","last_clean":"2023-10-16T12:50:06.917997+0000","last_became_active":"2023-10-16T12:50:05.917579+0000","last_became_peered":"2023-10-16T12:50:05.917579+0000","last_unstale":"2023-10-16T12:50:06.917997+0000","last_undegraded":"2023-10-16T12:50:06.917997+0000","last_fullsized":"2023-10-16T12:50:06.917997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:55.159359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888814+0000","last_change":"2023-10-16T12:50:05.928943+0000","last_active":"2023-10-16T12:50:07.888814+0000","last_peered":"2023-10-16T12:50:07.888814+0000","last_clean":"2023-10-16T12:50:07.888814+0000","last_became_active":"2023-10-16T12:50:05.928083+0000","last_became_peered":"2023-10-16T12:50:05.928083+0000","last_unstale":"2023-10-16T12:50:07.888814+0000","last_undegraded":"2023-10-16T12:50:07.888814+0000","last_fullsized":"2023-10-16T12:50:07.888814+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:20.179704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251593+0000","last_change":"2023-10-16T12:50:05.915602+0000","last_active":"2023-10-16T12:50:07.251593+0000","last_peered":"2023-10-16T12:50:07.251593+0000","last_clean":"2023-10-16T12:50:07.251593+0000","last_became_active":"2023-10-16T12:50:05.915322+0000","last_became_peered":"2023-10-16T12:50:05.915322+0000","last_unstale":"2023-10-16T12:50:07.251593+0000","last_undegraded":"2023-10-16T12:50:07.251593+0000","last_fullsized":"2023-10-16T12:50:07.251593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:17:22.249451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239484+0000","last_change":"2023-10-16T12:50:05.918803+0000","last_active":"2023-10-16T12:50:07.239484+0000","last_peered":"2023-10-16T12:50:07.239484+0000","last_clean":"2023-10-16T12:50:07.239484+0000","last_became_active":"2023-10-16T12:50:05.917735+0000","last_became_peered":"2023-10-16T12:50:05.917735+0000","last_unstale":"2023-10-16T12:50:07.239484+0000","last_undegraded":"2023-10-16T12:50:07.239484+0000","last_fullsized":"2023-10-16T12:50:07.239484+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:27.820235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985269+0000","last_change":"2023-10-16T12:50:05.904469+0000","last_active":"2023-10-16T12:50:07.985269+0000","last_peered":"2023-10-16T12:50:07.985269+0000","last_clean":"2023-10-16T12:50:07.985269+0000","last_became_active":"2023-10-16T12:50:05.903618+0000","last_became_peered":"2023-10-16T12:50:05.903618+0000","last_unstale":"2023-10-16T12:50:07.985269+0000","last_undegraded":"2023-10-16T12:50:07.985269+0000","last_fullsized":"2023-10-16T12:50:07.985269+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:04.981937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919102+0000","last_change":"2023-10-16T12:50:05.910322+0000","last_active":"2023-10-16T12:50:06.919102+0000","last_peered":"2023-10-16T12:50:06.919102+0000","last_clean":"2023-10-16T12:50:06.919102+0000","last_became_active":"2023-10-16T12:50:05.909199+0000","last_became_peered":"2023-10-16T12:50:05.909199+0000","last_unstale":"2023-10-16T12:50:06.919102+0000","last_undegraded":"2023-10-16T12:50:06.919102+0000","last_fullsized":"2023-10-16T12:50:06.919102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:38.651350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.918654+0000","last_change":"2023-10-16T12:50:05.921203+0000","last_active":"2023-10-16T12:50:06.918654+0000","last_peered":"2023-10-16T12:50:06.918654+0000","last_clean":"2023-10-16T12:50:06.918654+0000","last_became_active":"2023-10-16T12:50:05.920445+0000","last_became_peered":"2023-10-16T12:50:05.920445+0000","last_unstale":"2023-10-16T12:50:06.918654+0000","last_undegraded":"2023-10-16T12:50:06.918654+0000","last_fullsized":"2023-10-16T12:50:06.918654+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:22.157039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986487+0000","last_change":"2023-10-16T12:50:05.910485+0000","last_active":"2023-10-16T12:50:07.986487+0000","last_peered":"2023-10-16T12:50:07.986487+0000","last_clean":"2023-10-16T12:50:07.986487+0000","last_became_active":"2023-10-16T12:50:05.910157+0000","last_became_peered":"2023-10-16T12:50:05.910157+0000","last_unstale":"2023-10-16T12:50:07.986487+0000","last_undegraded":"2023-10-16T12:50:07.986487+0000","last_fullsized":"2023-10-16T12:50:07.986487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:17:53.141438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986443+0000","last_change":"2023-10-16T12:50:05.922961+0000","last_active":"2023-10-16T12:50:07.986443+0000","last_peered":"2023-10-16T12:50:07.986443+0000","last_clean":"2023-10-16T12:50:07.986443+0000","last_became_active":"2023-10-16T12:50:05.922700+0000","last_became_peered":"2023-10-16T12:50:05.922700+0000","last_unstale":"2023-10-16T12:50:07.986443+0000","last_undegraded":"2023-10-16T12:50:07.986443+0000","last_fullsized":"2023-10-16T12:50:07.986443+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:15.127530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919589+0000","last_change":"2023-10-16T12:50:05.910720+0000","last_active":"2023-10-16T12:50:06.919589+0000","last_peered":"2023-10-16T12:50:06.919589+0000","last_clean":"2023-10-16T12:50:06.919589+0000","last_became_active":"2023-10-16T12:50:05.908009+0000","last_became_peered":"2023-10-16T12:50:05.908009+0000","last_unstale":"2023-10-16T12:50:06.919589+0000","last_undegraded":"2023-10-16T12:50:06.919589+0000","last_fullsized":"2023-10-16T12:50:06.919589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:38.267965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920715+0000","last_change":"2023-10-16T12:50:05.916478+0000","last_active":"2023-10-16T12:50:06.920715+0000","last_peered":"2023-10-16T12:50:06.920715+0000","last_clean":"2023-10-16T12:50:06.920715+0000","last_became_active":"2023-10-16T12:50:05.916175+0000","last_became_peered":"2023-10-16T12:50:05.916175+0000","last_unstale":"2023-10-16T12:50:06.920715+0000","last_undegraded":"2023-10-16T12:50:06.920715+0000","last_fullsized":"2023-10-16T12:50:06.920715+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:57.966553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.915034+0000","last_change":"2023-10-16T12:50:05.908550+0000","last_active":"2023-10-16T12:50:06.915034+0000","last_peered":"2023-10-16T12:50:06.915034+0000","last_clean":"2023-10-16T12:50:06.915034+0000","last_became_active":"2023-10-16T12:50:05.908242+0000","last_became_peered":"2023-10-16T12:50:05.908242+0000","last_unstale":"2023-10-16T12:50:06.915034+0000","last_undegraded":"2023-10-16T12:50:06.915034+0000","last_fullsized":"2023-10-16T12:50:06.915034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:17.222741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919374+0000","last_change":"2023-10-16T12:50:05.931667+0000","last_active":"2023-10-16T12:50:06.919374+0000","last_peered":"2023-10-16T12:50:06.919374+0000","last_clean":"2023-10-16T12:50:06.919374+0000","last_became_active":"2023-10-16T12:50:05.930777+0000","last_became_peered":"2023-10-16T12:50:05.930777+0000","last_unstale":"2023-10-16T12:50:06.919374+0000","last_undegraded":"2023-10-16T12:50:06.919374+0000","last_fullsized":"2023-10-16T12:50:06.919374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:18.872293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914496+0000","last_change":"2023-10-16T12:50:05.902521+0000","last_active":"2023-10-16T12:50:06.914496+0000","last_peered":"2023-10-16T12:50:06.914496+0000","last_clean":"2023-10-16T12:50:06.914496+0000","last_became_active":"2023-10-16T12:50:05.902085+0000","last_became_peered":"2023-10-16T12:50:05.902085+0000","last_unstale":"2023-10-16T12:50:06.914496+0000","last_undegraded":"2023-10-16T12:50:06.914496+0000","last_fullsized":"2023-10-16T12:50:06.914496+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.034398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919803+0000","last_change":"2023-10-16T12:50:05.910068+0000","last_active":"2023-10-16T12:50:06.919803+0000","last_peered":"2023-10-16T12:50:06.919803+0000","last_clean":"2023-10-16T12:50:06.919803+0000","last_became_active":"2023-10-16T12:50:05.909772+0000","last_became_peered":"2023-10-16T12:50:05.909772+0000","last_unstale":"2023-10-16T12:50:06.919803+0000","last_undegraded":"2023-10-16T12:50:06.919803+0000","last_fullsized":"2023-10-16T12:50:06.919803+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:21.501022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921067+0000","last_change":"2023-10-16T12:50:05.912113+0000","last_active":"2023-10-16T12:50:06.921067+0000","last_peered":"2023-10-16T12:50:06.921067+0000","last_clean":"2023-10-16T12:50:06.921067+0000","last_became_active":"2023-10-16T12:50:05.911842+0000","last_became_peered":"2023-10-16T12:50:05.911842+0000","last_unstale":"2023-10-16T12:50:06.921067+0000","last_undegraded":"2023-10-16T12:50:06.921067+0000","last_fullsized":"2023-10-16T12:50:06.921067+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:04.103802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239183+0000","last_change":"2023-10-16T12:50:05.914968+0000","last_active":"2023-10-16T12:50:07.239183+0000","last_peered":"2023-10-16T12:50:07.239183+0000","last_clean":"2023-10-16T12:50:07.239183+0000","last_became_active":"2023-10-16T12:50:05.914662+0000","last_became_peered":"2023-10-16T12:50:05.914662+0000","last_unstale":"2023-10-16T12:50:07.239183+0000","last_undegraded":"2023-10-16T12:50:07.239183+0000","last_fullsized":"2023-10-16T12:50:07.239183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:26.145362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987166+0000","last_change":"2023-10-16T12:50:05.923905+0000","last_active":"2023-10-16T12:50:07.987166+0000","last_peered":"2023-10-16T12:50:07.987166+0000","last_clean":"2023-10-16T12:50:07.987166+0000","last_became_active":"2023-10-16T12:50:05.923285+0000","last_became_peered":"2023-10-16T12:50:05.923285+0000","last_unstale":"2023-10-16T12:50:07.987166+0000","last_undegraded":"2023-10-16T12:50:07.987166+0000","last_fullsized":"2023-10-16T12:50:07.987166+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.890992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987747+0000","last_change":"2023-10-16T12:50:04.912074+0000","last_active":"2023-10-16T12:50:07.987747+0000","last_peered":"2023-10-16T12:50:07.987747+0000","last_clean":"2023-10-16T12:50:07.987747+0000","last_became_active":"2023-10-16T12:50:04.911730+0000","last_became_peered":"2023-10-16T12:50:04.911730+0000","last_unstale":"2023-10-16T12:50:07.987747+0000","last_undegraded":"2023-10-16T12:50:07.987747+0000","last_fullsized":"2023-10-16T12:50:07.987747+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:48.094137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.889007+0000","last_change":"2023-10-16T12:50:05.928769+0000","last_active":"2023-10-16T12:50:07.889007+0000","last_peered":"2023-10-16T12:50:07.889007+0000","last_clean":"2023-10-16T12:50:07.889007+0000","last_became_active":"2023-10-16T12:50:05.927670+0000","last_became_peered":"2023-10-16T12:50:05.927670+0000","last_unstale":"2023-10-16T12:50:07.889007+0000","last_undegraded":"2023-10-16T12:50:07.889007+0000","last_fullsized":"2023-10-16T12:50:07.889007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:28.572915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985589+0000","last_change":"2023-10-16T12:50:04.897359+0000","last_active":"2023-10-16T12:50:07.985589+0000","last_peered":"2023-10-16T12:50:07.985589+0000","last_clean":"2023-10-16T12:50:07.985589+0000","last_became_active":"2023-10-16T12:50:04.897013+0000","last_became_peered":"2023-10-16T12:50:04.897013+0000","last_unstale":"2023-10-16T12:50:07.985589+0000","last_undegraded":"2023-10-16T12:50:07.985589+0000","last_fullsized":"2023-10-16T12:50:07.985589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:50.215116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239103+0000","last_change":"2023-10-16T12:50:05.922648+0000","last_active":"2023-10-16T12:50:07.239103+0000","last_peered":"2023-10-16T12:50:07.239103+0000","last_clean":"2023-10-16T12:50:07.239103+0000","last_became_active":"2023-10-16T12:50:05.921988+0000","last_became_peered":"2023-10-16T12:50:05.921988+0000","last_unstale":"2023-10-16T12:50:07.239103+0000","last_undegraded":"2023-10-16T12:50:07.239103+0000","last_fullsized":"2023-10-16T12:50:07.239103+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:00:56.969231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920889+0000","last_change":"2023-10-16T12:50:04.910287+0000","last_active":"2023-10-16T12:50:06.920889+0000","last_peered":"2023-10-16T12:50:06.920889+0000","last_clean":"2023-10-16T12:50:06.920889+0000","last_became_active":"2023-10-16T12:50:04.909949+0000","last_became_peered":"2023-10-16T12:50:04.909949+0000","last_unstale":"2023-10-16T12:50:06.920889+0000","last_undegraded":"2023-10-16T12:50:06.920889+0000","last_fullsized":"2023-10-16T12:50:06.920889+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:36.373869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987214+0000","last_change":"2023-10-16T12:50:05.916676+0000","last_active":"2023-10-16T12:50:07.987214+0000","last_peered":"2023-10-16T12:50:07.987214+0000","last_clean":"2023-10-16T12:50:07.987214+0000","last_became_active":"2023-10-16T12:50:05.916359+0000","last_became_peered":"2023-10-16T12:50:05.916359+0000","last_unstale":"2023-10-16T12:50:07.987214+0000","last_undegraded":"2023-10-16T12:50:07.987214+0000","last_fullsized":"2023-10-16T12:50:07.987214+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:32:42.657480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239436+0000","last_change":"2023-10-16T12:50:04.909778+0000","last_active":"2023-10-16T12:50:07.239436+0000","last_peered":"2023-10-16T12:50:07.239436+0000","last_clean":"2023-10-16T12:50:07.239436+0000","last_became_active":"2023-10-16T12:50:04.909153+0000","last_became_peered":"2023-10-16T12:50:04.909153+0000","last_unstale":"2023-10-16T12:50:07.239436+0000","last_undegraded":"2023-10-16T12:50:07.239436+0000","last_fullsized":"2023-10-16T12:50:07.239436+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:25.644099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920154+0000","last_change":"2023-10-16T12:50:05.905025+0000","last_active":"2023-10-16T12:50:06.920154+0000","last_peered":"2023-10-16T12:50:06.920154+0000","last_clean":"2023-10-16T12:50:06.920154+0000","last_became_active":"2023-10-16T12:50:05.904728+0000","last_became_peered":"2023-10-16T12:50:05.904728+0000","last_unstale":"2023-10-16T12:50:06.920154+0000","last_undegraded":"2023-10-16T12:50:06.920154+0000","last_fullsized":"2023-10-16T12:50:06.920154+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:43.914172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914841+0000","last_change":"2023-10-16T12:50:04.905258+0000","last_active":"2023-10-16T12:50:06.914841+0000","last_peered":"2023-10-16T12:50:06.914841+0000","last_clean":"2023-10-16T12:50:06.914841+0000","last_became_active":"2023-10-16T12:50:04.904945+0000","last_became_peered":"2023-10-16T12:50:04.904945+0000","last_unstale":"2023-10-16T12:50:06.914841+0000","last_undegraded":"2023-10-16T12:50:06.914841+0000","last_fullsized":"2023-10-16T12:50:06.914841+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:47:14.113323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919471+0000","last_change":"2023-10-16T12:50:05.918386+0000","last_active":"2023-10-16T12:50:06.919471+0000","last_peered":"2023-10-16T12:50:06.919471+0000","last_clean":"2023-10-16T12:50:06.919471+0000","last_became_active":"2023-10-16T12:50:05.918129+0000","last_became_peered":"2023-10-16T12:50:05.918129+0000","last_unstale":"2023-10-16T12:50:06.919471+0000","last_undegraded":"2023-10-16T12:50:06.919471+0000","last_fullsized":"2023-10-16T12:50:06.919471+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:52.447077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238599+0000","last_change":"2023-10-16T12:50:04.908491+0000","last_active":"2023-10-16T12:50:07.238599+0000","last_peered":"2023-10-16T12:50:07.238599+0000","last_clean":"2023-10-16T12:50:07.238599+0000","last_became_active":"2023-10-16T12:50:04.907899+0000","last_became_peered":"2023-10-16T12:50:04.907899+0000","last_unstale":"2023-10-16T12:50:07.238599+0000","last_undegraded":"2023-10-16T12:50:07.238599+0000","last_fullsized":"2023-10-16T12:50:07.238599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:41.094211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.887643+0000","last_change":"2023-10-16T12:50:05.917547+0000","last_active":"2023-10-16T12:50:07.887643+0000","last_peered":"2023-10-16T12:50:07.887643+0000","last_clean":"2023-10-16T12:50:07.887643+0000","last_became_active":"2023-10-16T12:50:05.916215+0000","last_became_peered":"2023-10-16T12:50:05.916215+0000","last_unstale":"2023-10-16T12:50:07.887643+0000","last_undegraded":"2023-10-16T12:50:07.887643+0000","last_fullsized":"2023-10-16T12:50:07.887643+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:01.691091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250583+0000","last_change":"2023-10-16T12:50:04.905235+0000","last_active":"2023-10-16T12:50:07.250583+0000","last_peered":"2023-10-16T12:50:07.250583+0000","last_clean":"2023-10-16T12:50:07.250583+0000","last_became_active":"2023-10-16T12:50:04.904889+0000","last_became_peered":"2023-10-16T12:50:04.904889+0000","last_unstale":"2023-10-16T12:50:07.250583+0000","last_undegraded":"2023-10-16T12:50:07.250583+0000","last_fullsized":"2023-10-16T12:50:07.250583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:27.558708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985060+0000","last_change":"2023-10-16T12:50:05.907559+0000","last_active":"2023-10-16T12:50:07.985060+0000","last_peered":"2023-10-16T12:50:07.985060+0000","last_clean":"2023-10-16T12:50:07.985060+0000","last_became_active":"2023-10-16T12:50:05.907274+0000","last_became_peered":"2023-10-16T12:50:05.907274+0000","last_unstale":"2023-10-16T12:50:07.985060+0000","last_undegraded":"2023-10-16T12:50:07.985060+0000","last_fullsized":"2023-10-16T12:50:07.985060+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:35.209542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.887678+0000","last_change":"2023-10-16T12:50:04.899018+0000","last_active":"2023-10-16T12:50:07.887678+0000","last_peered":"2023-10-16T12:50:07.887678+0000","last_clean":"2023-10-16T12:50:07.887678+0000","last_became_active":"2023-10-16T12:50:04.896458+0000","last_became_peered":"2023-10-16T12:50:04.896458+0000","last_unstale":"2023-10-16T12:50:07.887678+0000","last_undegraded":"2023-10-16T12:50:07.887678+0000","last_fullsized":"2023-10-16T12:50:07.887678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:32:37.714992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921968+0000","last_change":"2023-10-16T12:50:05.911458+0000","last_active":"2023-10-16T12:50:06.921968+0000","last_peered":"2023-10-16T12:50:06.921968+0000","last_clean":"2023-10-16T12:50:06.921968+0000","last_became_active":"2023-10-16T12:50:05.911165+0000","last_became_peered":"2023-10-16T12:50:05.911165+0000","last_unstale":"2023-10-16T12:50:06.921968+0000","last_undegraded":"2023-10-16T12:50:06.921968+0000","last_fullsized":"2023-10-16T12:50:06.921968+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:27.454739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919819+0000","last_change":"2023-10-16T12:50:04.899940+0000","last_active":"2023-10-16T12:50:06.919819+0000","last_peered":"2023-10-16T12:50:06.919819+0000","last_clean":"2023-10-16T12:50:06.919819+0000","last_became_active":"2023-10-16T12:50:04.899651+0000","last_became_peered":"2023-10-16T12:50:04.899651+0000","last_unstale":"2023-10-16T12:50:06.919819+0000","last_undegraded":"2023-10-16T12:50:06.919819+0000","last_fullsized":"2023-10-16T12:50:06.919819+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:19.511563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987398+0000","last_change":"2023-10-16T12:50:05.911584+0000","last_active":"2023-10-16T12:50:07.987398+0000","last_peered":"2023-10-16T12:50:07.987398+0000","last_clean":"2023-10-16T12:50:07.987398+0000","last_became_active":"2023-10-16T12:50:05.911243+0000","last_became_peered":"2023-10-16T12:50:05.911243+0000","last_unstale":"2023-10-16T12:50:07.987398+0000","last_undegraded":"2023-10-16T12:50:07.987398+0000","last_fullsized":"2023-10-16T12:50:07.987398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:42.818400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239961+0000","last_change":"2023-10-16T12:50:04.904736+0000","last_active":"2023-10-16T12:50:07.239961+0000","last_peered":"2023-10-16T12:50:07.239961+0000","last_clean":"2023-10-16T12:50:07.239961+0000","last_became_active":"2023-10-16T12:50:04.903775+0000","last_became_peered":"2023-10-16T12:50:04.903775+0000","last_unstale":"2023-10-16T12:50:07.239961+0000","last_undegraded":"2023-10-16T12:50:07.239961+0000","last_fullsized":"2023-10-16T12:50:07.239961+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:28.502627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239388+0000","last_change":"2023-10-16T12:50:05.908674+0000","last_active":"2023-10-16T12:50:07.239388+0000","last_peered":"2023-10-16T12:50:07.239388+0000","last_clean":"2023-10-16T12:50:07.239388+0000","last_became_active":"2023-10-16T12:50:05.908283+0000","last_became_peered":"2023-10-16T12:50:05.908283+0000","last_unstale":"2023-10-16T12:50:07.239388+0000","last_undegraded":"2023-10-16T12:50:07.239388+0000","last_fullsized":"2023-10-16T12:50:07.239388+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:51.067027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"98'3","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914916+0000","last_change":"2023-10-16T12:50:04.903704+0000","last_active":"2023-10-16T12:50:06.914916+0000","last_peered":"2023-10-16T12:50:06.914916+0000","last_clean":"2023-10-16T12:50:06.914916+0000","last_became_active":"2023-10-16T12:50:04.902242+0000","last_became_peered":"2023-10-16T12:50:04.902242+0000","last_unstale":"2023-10-16T12:50:06.914916+0000","last_undegraded":"2023-10-16T12:50:06.914916+0000","last_fullsized":"2023-10-16T12:50:06.914916+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:11.966999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888077+0000","last_change":"2023-10-16T12:50:05.921318+0000","last_active":"2023-10-16T12:50:07.888077+0000","last_peered":"2023-10-16T12:50:07.888077+0000","last_clean":"2023-10-16T12:50:07.888077+0000","last_became_active":"2023-10-16T12:50:05.920593+0000","last_became_peered":"2023-10-16T12:50:05.920593+0000","last_unstale":"2023-10-16T12:50:07.888077+0000","last_undegraded":"2023-10-16T12:50:07.888077+0000","last_fullsized":"2023-10-16T12:50:07.888077+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:04.287816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"98'6","reported_seq":18,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251347+0000","last_change":"2023-10-16T12:50:04.899374+0000","last_active":"2023-10-16T12:50:07.251347+0000","last_peered":"2023-10-16T12:50:07.251347+0000","last_clean":"2023-10-16T12:50:07.251347+0000","last_became_active":"2023-10-16T12:50:04.899056+0000","last_became_peered":"2023-10-16T12:50:04.899056+0000","last_unstale":"2023-10-16T12:50:07.251347+0000","last_undegraded":"2023-10-16T12:50:07.251347+0000","last_fullsized":"2023-10-16T12:50:07.251347+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:23.241467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986921+0000","last_change":"2023-10-16T12:50:05.930835+0000","last_active":"2023-10-16T12:50:07.986921+0000","last_peered":"2023-10-16T12:50:07.986921+0000","last_clean":"2023-10-16T12:50:07.986921+0000","last_became_active":"2023-10-16T12:50:05.930564+0000","last_became_peered":"2023-10-16T12:50:05.930564+0000","last_unstale":"2023-10-16T12:50:07.986921+0000","last_undegraded":"2023-10-16T12:50:07.986921+0000","last_fullsized":"2023-10-16T12:50:07.986921+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:42.188513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250865+0000","last_change":"2023-10-16T12:50:04.901049+0000","last_active":"2023-10-16T12:50:07.250865+0000","last_peered":"2023-10-16T12:50:07.250865+0000","last_clean":"2023-10-16T12:50:07.250865+0000","last_became_active":"2023-10-16T12:50:04.900768+0000","last_became_peered":"2023-10-16T12:50:04.900768+0000","last_unstale":"2023-10-16T12:50:07.250865+0000","last_undegraded":"2023-10-16T12:50:07.250865+0000","last_fullsized":"2023-10-16T12:50:07.250865+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:07.708543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920355+0000","last_change":"2023-10-16T12:50:05.903183+0000","last_active":"2023-10-16T12:50:06.920355+0000","last_peered":"2023-10-16T12:50:06.920355+0000","last_clean":"2023-10-16T12:50:06.920355+0000","last_became_active":"2023-10-16T12:50:05.902766+0000","last_became_peered":"2023-10-16T12:50:05.902766+0000","last_unstale":"2023-10-16T12:50:06.920355+0000","last_undegraded":"2023-10-16T12:50:06.920355+0000","last_fullsized":"2023-10-16T12:50:06.920355+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:27.484829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238854+0000","last_change":"2023-10-16T12:50:04.909403+0000","last_active":"2023-10-16T12:50:07.238854+0000","last_peered":"2023-10-16T12:50:07.238854+0000","last_clean":"2023-10-16T12:50:07.238854+0000","last_became_active":"2023-10-16T12:50:04.906556+0000","last_became_peered":"2023-10-16T12:50:04.906556+0000","last_unstale":"2023-10-16T12:50:07.238854+0000","last_undegraded":"2023-10-16T12:50:07.238854+0000","last_fullsized":"2023-10-16T12:50:07.238854+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:11.978352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.984903+0000","last_change":"2023-10-16T12:50:05.917948+0000","last_active":"2023-10-16T12:50:07.984903+0000","last_peered":"2023-10-16T12:50:07.984903+0000","last_clean":"2023-10-16T12:50:07.984903+0000","last_became_active":"2023-10-16T12:50:05.917650+0000","last_became_peered":"2023-10-16T12:50:05.917650+0000","last_unstale":"2023-10-16T12:50:07.984903+0000","last_undegraded":"2023-10-16T12:50:07.984903+0000","last_fullsized":"2023-10-16T12:50:07.984903+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:17.394990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920749+0000","last_change":"2023-10-16T12:50:04.897941+0000","last_active":"2023-10-16T12:50:06.920749+0000","last_peered":"2023-10-16T12:50:06.920749+0000","last_clean":"2023-10-16T12:50:06.920749+0000","last_became_active":"2023-10-16T12:50:04.897614+0000","last_became_peered":"2023-10-16T12:50:04.897614+0000","last_unstale":"2023-10-16T12:50:06.920749+0000","last_undegraded":"2023-10-16T12:50:06.920749+0000","last_fullsized":"2023-10-16T12:50:06.920749+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:04.314762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.887654+0000","last_change":"2023-10-16T12:50:05.925498+0000","last_active":"2023-10-16T12:50:07.887654+0000","last_peered":"2023-10-16T12:50:07.887654+0000","last_clean":"2023-10-16T12:50:07.887654+0000","last_became_active":"2023-10-16T12:50:05.919426+0000","last_became_peered":"2023-10-16T12:50:05.919426+0000","last_unstale":"2023-10-16T12:50:07.887654+0000","last_undegraded":"2023-10-16T12:50:07.887654+0000","last_fullsized":"2023-10-16T12:50:07.887654+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:57.219093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.918024+0000","last_change":"2023-10-16T12:50:04.907902+0000","last_active":"2023-10-16T12:50:06.918024+0000","last_peered":"2023-10-16T12:50:06.918024+0000","last_clean":"2023-10-16T12:50:06.918024+0000","last_became_active":"2023-10-16T12:50:04.906698+0000","last_became_peered":"2023-10-16T12:50:04.906698+0000","last_unstale":"2023-10-16T12:50:06.918024+0000","last_undegraded":"2023-10-16T12:50:06.918024+0000","last_fullsized":"2023-10-16T12:50:06.918024+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:35.590819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.919882+0000","last_change":"2023-10-16T12:50:05.915739+0000","last_active":"2023-10-16T12:50:06.919882+0000","last_peered":"2023-10-16T12:50:06.919882+0000","last_clean":"2023-10-16T12:50:06.919882+0000","last_became_active":"2023-10-16T12:50:05.915372+0000","last_became_peered":"2023-10-16T12:50:05.915372+0000","last_unstale":"2023-10-16T12:50:06.919882+0000","last_undegraded":"2023-10-16T12:50:06.919882+0000","last_fullsized":"2023-10-16T12:50:06.919882+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:32:56.309825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239857+0000","last_change":"2023-10-16T12:50:04.909488+0000","last_active":"2023-10-16T12:50:07.239857+0000","last_peered":"2023-10-16T12:50:07.239857+0000","last_clean":"2023-10-16T12:50:07.239857+0000","last_became_active":"2023-10-16T12:50:04.909134+0000","last_became_peered":"2023-10-16T12:50:04.909134+0000","last_unstale":"2023-10-16T12:50:07.239857+0000","last_undegraded":"2023-10-16T12:50:07.239857+0000","last_fullsized":"2023-10-16T12:50:07.239857+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:21.517636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987608+0000","last_change":"2023-10-16T12:50:05.915341+0000","last_active":"2023-10-16T12:50:07.987608+0000","last_peered":"2023-10-16T12:50:07.987608+0000","last_clean":"2023-10-16T12:50:07.987608+0000","last_became_active":"2023-10-16T12:50:05.915017+0000","last_became_peered":"2023-10-16T12:50:05.915017+0000","last_unstale":"2023-10-16T12:50:07.987608+0000","last_undegraded":"2023-10-16T12:50:07.987608+0000","last_fullsized":"2023-10-16T12:50:07.987608+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:57.752805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.920113+0000","last_change":"2023-10-16T12:50:04.904388+0000","last_active":"2023-10-16T12:50:06.920113+0000","last_peered":"2023-10-16T12:50:06.920113+0000","last_clean":"2023-10-16T12:50:06.920113+0000","last_became_active":"2023-10-16T12:50:04.904122+0000","last_became_peered":"2023-10-16T12:50:04.904122+0000","last_unstale":"2023-10-16T12:50:06.920113+0000","last_undegraded":"2023-10-16T12:50:06.920113+0000","last_fullsized":"2023-10-16T12:50:06.920113+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:05.823130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.251189+0000","last_change":"2023-10-16T12:50:05.933323+0000","last_active":"2023-10-16T12:50:07.251189+0000","last_peered":"2023-10-16T12:50:07.251189+0000","last_clean":"2023-10-16T12:50:07.251189+0000","last_became_active":"2023-10-16T12:50:05.933038+0000","last_became_peered":"2023-10-16T12:50:05.933038+0000","last_unstale":"2023-10-16T12:50:07.251189+0000","last_undegraded":"2023-10-16T12:50:07.251189+0000","last_fullsized":"2023-10-16T12:50:07.251189+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:28.709647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239670+0000","last_change":"2023-10-16T12:50:04.906298+0000","last_active":"2023-10-16T12:50:07.239670+0000","last_peered":"2023-10-16T12:50:07.239670+0000","last_clean":"2023-10-16T12:50:07.239670+0000","last_became_active":"2023-10-16T12:50:04.906002+0000","last_became_peered":"2023-10-16T12:50:04.906002+0000","last_unstale":"2023-10-16T12:50:07.239670+0000","last_undegraded":"2023-10-16T12:50:07.239670+0000","last_fullsized":"2023-10-16T12:50:07.239670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:05.586314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888472+0000","last_change":"2023-10-16T12:50:05.920923+0000","last_active":"2023-10-16T12:50:07.888472+0000","last_peered":"2023-10-16T12:50:07.888472+0000","last_clean":"2023-10-16T12:50:07.888472+0000","last_became_active":"2023-10-16T12:50:05.914388+0000","last_became_peered":"2023-10-16T12:50:05.914388+0000","last_unstale":"2023-10-16T12:50:07.888472+0000","last_undegraded":"2023-10-16T12:50:07.888472+0000","last_fullsized":"2023-10-16T12:50:07.888472+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:32.171397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986659+0000","last_change":"2023-10-16T12:50:04.912716+0000","last_active":"2023-10-16T12:50:07.986659+0000","last_peered":"2023-10-16T12:50:07.986659+0000","last_clean":"2023-10-16T12:50:07.986659+0000","last_became_active":"2023-10-16T12:50:04.912359+0000","last_became_peered":"2023-10-16T12:50:04.912359+0000","last_unstale":"2023-10-16T12:50:07.986659+0000","last_undegraded":"2023-10-16T12:50:07.986659+0000","last_fullsized":"2023-10-16T12:50:07.986659+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:11.966999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914641+0000","last_change":"2023-10-16T12:50:05.917272+0000","last_active":"2023-10-16T12:50:06.914641+0000","last_peered":"2023-10-16T12:50:06.914641+0000","last_clean":"2023-10-16T12:50:06.914641+0000","last_became_active":"2023-10-16T12:50:05.916097+0000","last_became_peered":"2023-10-16T12:50:05.916097+0000","last_unstale":"2023-10-16T12:50:06.914641+0000","last_undegraded":"2023-10-16T12:50:06.914641+0000","last_fullsized":"2023-10-16T12:50:06.914641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.890992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238915+0000","last_change":"2023-10-16T12:50:04.903020+0000","last_active":"2023-10-16T12:50:07.238915+0000","last_peered":"2023-10-16T12:50:07.238915+0000","last_clean":"2023-10-16T12:50:07.238915+0000","last_became_active":"2023-10-16T12:50:04.901234+0000","last_became_peered":"2023-10-16T12:50:04.901234+0000","last_unstale":"2023-10-16T12:50:07.238915+0000","last_undegraded":"2023-10-16T12:50:07.238915+0000","last_fullsized":"2023-10-16T12:50:07.238915+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:17.779879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914288+0000","last_change":"2023-10-16T12:50:05.918667+0000","last_active":"2023-10-16T12:50:06.914288+0000","last_peered":"2023-10-16T12:50:06.914288+0000","last_clean":"2023-10-16T12:50:06.914288+0000","last_became_active":"2023-10-16T12:50:05.918268+0000","last_became_peered":"2023-10-16T12:50:05.918268+0000","last_unstale":"2023-10-16T12:50:06.914288+0000","last_undegraded":"2023-10-16T12:50:06.914288+0000","last_fullsized":"2023-10-16T12:50:06.914288+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:17:53.141438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921300+0000","last_change":"2023-10-16T12:50:04.897323+0000","last_active":"2023-10-16T12:50:06.921300+0000","last_peered":"2023-10-16T12:50:06.921300+0000","last_clean":"2023-10-16T12:50:06.921300+0000","last_became_active":"2023-10-16T12:50:04.896993+0000","last_became_peered":"2023-10-16T12:50:04.896993+0000","last_unstale":"2023-10-16T12:50:06.921300+0000","last_undegraded":"2023-10-16T12:50:06.921300+0000","last_fullsized":"2023-10-16T12:50:06.921300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:11.773202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914579+0000","last_change":"2023-10-16T12:50:05.918396+0000","last_active":"2023-10-16T12:50:06.914579+0000","last_peered":"2023-10-16T12:50:06.914579+0000","last_clean":"2023-10-16T12:50:06.914579+0000","last_became_active":"2023-10-16T12:50:05.916755+0000","last_became_peered":"2023-10-16T12:50:05.916755+0000","last_unstale":"2023-10-16T12:50:06.914579+0000","last_undegraded":"2023-10-16T12:50:06.914579+0000","last_fullsized":"2023-10-16T12:50:06.914579+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:42.188513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.984828+0000","last_change":"2023-10-16T12:50:04.897379+0000","last_active":"2023-10-16T12:50:07.984828+0000","last_peered":"2023-10-16T12:50:07.984828+0000","last_clean":"2023-10-16T12:50:07.984828+0000","last_became_active":"2023-10-16T12:50:04.897041+0000","last_became_peered":"2023-10-16T12:50:04.897041+0000","last_unstale":"2023-10-16T12:50:07.984828+0000","last_undegraded":"2023-10-16T12:50:07.984828+0000","last_fullsized":"2023-10-16T12:50:07.984828+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:18.816111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985477+0000","last_change":"2023-10-16T12:50:05.926250+0000","last_active":"2023-10-16T12:50:07.985477+0000","last_peered":"2023-10-16T12:50:07.985477+0000","last_clean":"2023-10-16T12:50:07.985477+0000","last_became_active":"2023-10-16T12:50:05.925919+0000","last_became_peered":"2023-10-16T12:50:05.925919+0000","last_unstale":"2023-10-16T12:50:07.985477+0000","last_undegraded":"2023-10-16T12:50:07.985477+0000","last_fullsized":"2023-10-16T12:50:07.985477+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:56.764598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239915+0000","last_change":"2023-10-16T12:50:04.909166+0000","last_active":"2023-10-16T12:50:07.239915+0000","last_peered":"2023-10-16T12:50:07.239915+0000","last_clean":"2023-10-16T12:50:07.239915+0000","last_became_active":"2023-10-16T12:50:04.908875+0000","last_became_peered":"2023-10-16T12:50:04.908875+0000","last_unstale":"2023-10-16T12:50:07.239915+0000","last_undegraded":"2023-10-16T12:50:07.239915+0000","last_fullsized":"2023-10-16T12:50:07.239915+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:32.169152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921492+0000","last_change":"2023-10-16T12:50:05.900261+0000","last_active":"2023-10-16T12:50:06.921492+0000","last_peered":"2023-10-16T12:50:06.921492+0000","last_clean":"2023-10-16T12:50:06.921492+0000","last_became_active":"2023-10-16T12:50:05.899923+0000","last_became_peered":"2023-10-16T12:50:05.899923+0000","last_unstale":"2023-10-16T12:50:06.921492+0000","last_undegraded":"2023-10-16T12:50:06.921492+0000","last_fullsized":"2023-10-16T12:50:06.921492+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:14.007579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.921718+0000","last_change":"2023-10-16T12:50:04.898402+0000","last_active":"2023-10-16T12:50:06.921718+0000","last_peered":"2023-10-16T12:50:06.921718+0000","last_clean":"2023-10-16T12:50:06.921718+0000","last_became_active":"2023-10-16T12:50:04.898114+0000","last_became_peered":"2023-10-16T12:50:04.898114+0000","last_unstale":"2023-10-16T12:50:06.921718+0000","last_undegraded":"2023-10-16T12:50:06.921718+0000","last_fullsized":"2023-10-16T12:50:06.921718+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:41.611358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.986704+0000","last_change":"2023-10-16T12:50:05.921002+0000","last_active":"2023-10-16T12:50:07.986704+0000","last_peered":"2023-10-16T12:50:07.986704+0000","last_clean":"2023-10-16T12:50:07.986704+0000","last_became_active":"2023-10-16T12:50:05.920613+0000","last_became_peered":"2023-10-16T12:50:05.920613+0000","last_unstale":"2023-10-16T12:50:07.986704+0000","last_undegraded":"2023-10-16T12:50:07.986704+0000","last_fullsized":"2023-10-16T12:50:07.986704+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.034398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888395+0000","last_change":"2023-10-16T12:50:04.896080+0000","last_active":"2023-10-16T12:50:07.888395+0000","last_peered":"2023-10-16T12:50:07.888395+0000","last_clean":"2023-10-16T12:50:07.888395+0000","last_became_active":"2023-10-16T12:50:04.895802+0000","last_became_peered":"2023-10-16T12:50:04.895802+0000","last_unstale":"2023-10-16T12:50:07.888395+0000","last_undegraded":"2023-10-16T12:50:07.888395+0000","last_fullsized":"2023-10-16T12:50:07.888395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:40.040917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.987368+0000","last_change":"2023-10-16T12:50:05.910412+0000","last_active":"2023-10-16T12:50:07.987368+0000","last_peered":"2023-10-16T12:50:07.987368+0000","last_clean":"2023-10-16T12:50:07.987368+0000","last_became_active":"2023-10-16T12:50:05.910121+0000","last_became_peered":"2023-10-16T12:50:05.910121+0000","last_unstale":"2023-10-16T12:50:07.987368+0000","last_undegraded":"2023-10-16T12:50:07.987368+0000","last_fullsized":"2023-10-16T12:50:07.987368+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:17.222741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.985279+0000","last_change":"2023-10-16T12:50:04.901142+0000","last_active":"2023-10-16T12:50:07.985279+0000","last_peered":"2023-10-16T12:50:07.985279+0000","last_clean":"2023-10-16T12:50:07.985279+0000","last_became_active":"2023-10-16T12:50:04.900860+0000","last_became_peered":"2023-10-16T12:50:04.900860+0000","last_unstale":"2023-10-16T12:50:07.985279+0000","last_undegraded":"2023-10-16T12:50:07.985279+0000","last_fullsized":"2023-10-16T12:50:07.985279+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:11.297663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888526+0000","last_change":"2023-10-16T12:50:05.918582+0000","last_active":"2023-10-16T12:50:07.888526+0000","last_peered":"2023-10-16T12:50:07.888526+0000","last_clean":"2023-10-16T12:50:07.888526+0000","last_became_active":"2023-10-16T12:50:05.918316+0000","last_became_peered":"2023-10-16T12:50:05.918316+0000","last_unstale":"2023-10-16T12:50:07.888526+0000","last_undegraded":"2023-10-16T12:50:07.888526+0000","last_fullsized":"2023-10-16T12:50:07.888526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:46.902439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.888444+0000","last_change":"2023-10-16T12:50:04.902038+0000","last_active":"2023-10-16T12:50:07.888444+0000","last_peered":"2023-10-16T12:50:07.888444+0000","last_clean":"2023-10-16T12:50:07.888444+0000","last_became_active":"2023-10-16T12:50:04.901771+0000","last_became_peered":"2023-10-16T12:50:04.901771+0000","last_unstale":"2023-10-16T12:50:07.888444+0000","last_undegraded":"2023-10-16T12:50:07.888444+0000","last_fullsized":"2023-10-16T12:50:07.888444+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:51.093009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238680+0000","last_change":"2023-10-16T12:50:05.921784+0000","last_active":"2023-10-16T12:50:07.238680+0000","last_peered":"2023-10-16T12:50:07.238680+0000","last_clean":"2023-10-16T12:50:07.238680+0000","last_became_active":"2023-10-16T12:50:05.921410+0000","last_became_peered":"2023-10-16T12:50:05.921410+0000","last_unstale":"2023-10-16T12:50:07.238680+0000","last_undegraded":"2023-10-16T12:50:07.238680+0000","last_fullsized":"2023-10-16T12:50:07.238680+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:29.623720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250454+0000","last_change":"2023-10-16T12:50:04.905982+0000","last_active":"2023-10-16T12:50:07.250454+0000","last_peered":"2023-10-16T12:50:07.250454+0000","last_clean":"2023-10-16T12:50:07.250454+0000","last_became_active":"2023-10-16T12:50:04.905738+0000","last_became_peered":"2023-10-16T12:50:04.905738+0000","last_unstale":"2023-10-16T12:50:07.250454+0000","last_undegraded":"2023-10-16T12:50:07.250454+0000","last_fullsized":"2023-10-16T12:50:07.250454+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:34.135812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.250686+0000","last_change":"2023-10-16T12:50:05.907215+0000","last_active":"2023-10-16T12:50:07.250686+0000","last_peered":"2023-10-16T12:50:07.250686+0000","last_clean":"2023-10-16T12:50:07.250686+0000","last_became_active":"2023-10-16T12:50:05.906933+0000","last_became_peered":"2023-10-16T12:50:05.906933+0000","last_unstale":"2023-10-16T12:50:07.250686+0000","last_undegraded":"2023-10-16T12:50:07.250686+0000","last_fullsized":"2023-10-16T12:50:07.250686+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:16.174638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.918013+0000","last_change":"2023-10-16T12:50:04.907939+0000","last_active":"2023-10-16T12:50:06.918013+0000","last_peered":"2023-10-16T12:50:06.918013+0000","last_clean":"2023-10-16T12:50:06.918013+0000","last_became_active":"2023-10-16T12:50:04.907070+0000","last_became_peered":"2023-10-16T12:50:04.907070+0000","last_unstale":"2023-10-16T12:50:06.918013+0000","last_undegraded":"2023-10-16T12:50:06.918013+0000","last_fullsized":"2023-10-16T12:50:06.918013+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:23.780652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238587+0000","last_change":"2023-10-16T12:50:05.915065+0000","last_active":"2023-10-16T12:50:07.238587+0000","last_peered":"2023-10-16T12:50:07.238587+0000","last_clean":"2023-10-16T12:50:07.238587+0000","last_became_active":"2023-10-16T12:50:05.914753+0000","last_became_peered":"2023-10-16T12:50:05.914753+0000","last_unstale":"2023-10-16T12:50:07.238587+0000","last_undegraded":"2023-10-16T12:50:07.238587+0000","last_fullsized":"2023-10-16T12:50:07.238587+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:52.112031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.239596+0000","last_change":"2023-10-16T12:50:04.905517+0000","last_active":"2023-10-16T12:50:07.239596+0000","last_peered":"2023-10-16T12:50:07.239596+0000","last_clean":"2023-10-16T12:50:07.239596+0000","last_became_active":"2023-10-16T12:50:04.905241+0000","last_became_peered":"2023-10-16T12:50:04.905241+0000","last_unstale":"2023-10-16T12:50:07.239596+0000","last_undegraded":"2023-10-16T12:50:07.239596+0000","last_fullsized":"2023-10-16T12:50:07.239596+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:50.136631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:06.914583+0000","last_change":"2023-10-16T12:50:05.921170+0000","last_active":"2023-10-16T12:50:06.914583+0000","last_peered":"2023-10-16T12:50:06.914583+0000","last_clean":"2023-10-16T12:50:06.914583+0000","last_became_active":"2023-10-16T12:50:05.920882+0000","last_became_peered":"2023-10-16T12:50:05.920882+0000","last_unstale":"2023-10-16T12:50:06.914583+0000","last_undegraded":"2023-10-16T12:50:06.914583+0000","last_fullsized":"2023-10-16T12:50:06.914583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:04.859982+0000","last_clean_scrub_stamp":"2023-10-16T12:50:04.859982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:39.457178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:50:07.238926+0000","last_change":"2023-10-16T12:50:04.906490+0000","last_active":"2023-10-16T12:50:07.238926+0000","last_peered":"2023-10-16T12:50:07.238926+0000","last_clean":"2023-10-16T12:50:07.238926+0000","last_became_active":"2023-10-16T12:50:04.905491+0000","last_became_peered":"2023-10-16T12:50:04.905491+0000","last_unstale":"2023-10-16T12:50:07.238926+0000","last_undegraded":"2023-10-16T12:50:07.238926+0000","last_fullsized":"2023-10-16T12:50:07.238926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:03.845775+0000","last_clean_scrub_stamp":"2023-10-16T12:50:03.845775+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:19.435381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"88'175","reported_seq":447,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:50:05.900237+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:50:05.900237+0000","last_peered":"2023-10-16T12:50:05.900237+0000","last_clean":"2023-10-16T12:50:05.900237+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:50:05.900237+0000","last_undegraded":"2023-10-16T12:50:05.900237+0000","last_fullsized":"2023-10-16T12:50:05.900237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542345,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1794048,"data_stored":1157735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2609999999999999,"15min":1.2929999999999999},"min":{"1min":0.68200000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.728,"5min":1.728,"15min":1.784},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3049999999999999,"15min":1.3520000000000001},"min":{"1min":0.76100000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.772,"5min":1.8420000000000001,"15min":1.8899999999999999},"last":1.1839999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.196,"15min":1.214},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.627,"5min":1.78,"15min":1.78},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1879999999999999,"15min":1.2170000000000001},"min":{"1min":0.67600000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.87}]},{"osd":2,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1799999999999999,"15min":1.206},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.302,"15min":1.323},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.63500000000000001},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.087}]},{"osd":3,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3049999999999999,"15min":1.296},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.69099999999999995},"max":{"1min":1.5680000000000001,"5min":1.847,"15min":1.847},"last":1.135},{"interface":"front","average":{"1min":1.411,"5min":1.411,"15min":1.4059999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.0129999999999999,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.171,"15min":1.194},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.349,"15min":1.3360000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8440000000000001,"5min":1.9339999999999999,"15min":1.9359999999999999},"last":1.4930000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3240000000000001,"15min":1.3340000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.6279999999999999,"5min":1.9139999999999999,"15min":1.944},"last":1.452},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4890000000000001,"15min":1.4690000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.2769999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:49:41 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.5780000000000001,"15min":1.5649999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4750000000000001,"15min":1.452},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":3.1360000000000001,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.3109999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542346,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":573143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.2430000000000001,"15min":1.276},"min":{"1min":0.72299999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.829,"5min":1.829,"15min":3.7450000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2929999999999999,"15min":1.337},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.67800000000000005},"max":{"1min":1.8680000000000001,"5min":2.0270000000000001,"15min":3.819},"last":1.145}]},{"osd":1,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1919999999999999,"15min":1.1930000000000001},"min":{"1min":0.67600000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5569999999999999,"5min":1.6579999999999999,"15min":1.8979999999999999},"last":1.677},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.216,"15min":1.208},"min":{"1min":0.68100000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.637,"5min":1.821,"15min":3.8519999999999999},"last":1.181}]},{"osd":2,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2130000000000001,"15min":1.224},"min":{"1min":0.73999999999999999,"5min":0.627,"15min":0.627},"max":{"1min":1.671,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.2170000000000001,"15min":1.278},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.7849999999999999,"5min":1.829,"15min":3.9009999999999998},"last":0.86399999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.333,"15min":1.3280000000000001},"min":{"1min":0.88800000000000001,"5min":0.80100000000000005,"15min":0.751},"max":{"1min":1.819,"5min":1.9399999999999999,"15min":4.2380000000000004},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2869999999999999,"15min":1.3500000000000001},"min":{"1min":0.82199999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.806,"5min":2.012,"15min":4.4269999999999996},"last":1.244}]},{"osd":4,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3560000000000001,"15min":1.3400000000000001},"min":{"1min":0.82899999999999996,"5min":0.71099999999999997,"15min":0.57699999999999996},"max":{"1min":1.9490000000000001,"5min":2.0830000000000002,"15min":2.3029999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.45,"15min":1.474},"min":{"1min":1.077,"5min":0.77300000000000002,"15min":0.76900000000000002},"max":{"1min":1.9430000000000001,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.8799999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2889999999999999,"15min":1.3089999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.849,"5min":1.9810000000000001,"15min":2.0910000000000002},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.4450000000000001,"15min":1.446},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.63900000000000001},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.7769999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.427,"15min":1.4430000000000001},"min":{"1min":0.99399999999999999,"5min":0.878,"15min":0.67900000000000005},"max":{"1min":1.8,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.446,"15min":1.452},"min":{"1min":1.0640000000000001,"5min":0.96399999999999997,"15min":0.71599999999999997},"max":{"1min":1.9690000000000001,"5min":2.1230000000000002,"15min":2.2610000000000001},"last":1.984}]}]},{"osd":5,"up_from":14,"seq":60129542347,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36520,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335320,"statfs":{"total":96636764160,"available":96599367680,"internally_reserved":0,"allocated":1208320,"data_stored":573141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.38200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.8480000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1779999999999999,"15min":1.1819999999999999},"min":{"1min":0.77800000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.645,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":0.83499999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.22,"15min":1.2549999999999999},"min":{"1min":0.69399999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.827},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1819999999999999,"15min":1.226},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.647,"5min":1.708,"15min":1.7889999999999999},"last":1.391}]},{"osd":2,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.26,"15min":1.2989999999999999},"min":{"1min":0.71599999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.591,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.044},{"interface":"front","average":{"1min":1.155,"5min":1.24,"15min":1.2669999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.554,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.26}]},{"osd":3,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.73099999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.363,"15min":1.3799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.929,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.306}]},{"osd":4,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1930000000000001,"15min":1.2050000000000001},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7430000000000001,"5min":1.996,"15min":1.996},"last":1.482},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.361,"15min":1.3140000000000001},"min":{"1min":0.55000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.0910000000000002},"last":1.4350000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.321,"15min":1.3220000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8,"5min":1.927,"15min":1.927},"last":1.536},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.3939999999999999,"15min":1.3799999999999999},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9470000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.6479999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4470000000000001,"15min":1.448},"min":{"1min":0.64300000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.3900000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5029999999999999},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.0209999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.6859999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58732,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313108,"statfs":{"total":96636764160,"available":96576622592,"internally_reserved":0,"allocated":1208320,"data_stored":573121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1799999999999999,"15min":1.2010000000000001},"min":{"1min":0.65700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4359999999999999,"5min":1.766,"15min":2.2200000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.032,"5min":1.1120000000000001,"15min":1.151},"min":{"1min":0.73499999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.569,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.375}]},{"osd":1,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.141,"15min":1.1899999999999999},"min":{"1min":0.69299999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5669999999999999,"5min":1.681,"15min":1.865},"last":1.157},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.228,"15min":1.26},"min":{"1min":0.76500000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.532,"5min":1.724,"15min":1.9390000000000001},"last":0.89000000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2549999999999999,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.81899999999999995,"15min":0.753},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.24,"5min":1.292,"15min":1.329},"min":{"1min":0.78800000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6379999999999999,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.2130000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2569999999999999,"15min":1.278},"min":{"1min":0.91600000000000004,"5min":0.80900000000000005,"15min":0.752},"max":{"1min":1.804,"5min":1.804,"15min":1.8100000000000001},"last":1.117},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.359,"15min":1.381},"min":{"1min":0.89800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.343}]},{"osd":5,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3420000000000001,"15min":1.3049999999999999},"min":{"1min":1.1699999999999999,"5min":0.86799999999999999,"15min":0.50900000000000001},"max":{"1min":1.917,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3220000000000001,"15min":1.3009999999999999},"min":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.53600000000000003},"max":{"1min":1.994,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.028}]},{"osd":6,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.407,"15min":1.391},"min":{"1min":0.92300000000000004,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.252},{"interface":"front","average":{"1min":1.569,"5min":1.488,"15min":1.4830000000000001},"min":{"1min":1.282,"5min":0.90400000000000003,"15min":0.66600000000000004},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.4870000000000001,"15min":1.4790000000000001},"min":{"1min":1.0600000000000001,"5min":0.88600000000000001,"15min":0.60399999999999998},"max":{"1min":2.109,"5min":2.109,"15min":2.254},"last":1.29},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.591,"15min":1.5960000000000001},"min":{"1min":1.026,"5min":0.94799999999999995,"15min":0.71299999999999997},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.4390000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542347,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40692,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331148,"statfs":{"total":96636764160,"available":96595095552,"internally_reserved":0,"allocated":1212416,"data_stored":573163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.91700000000000004,"15min":0.95399999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.5649999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.129},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.1180000000000001,"15min":1.169},"min":{"1min":0.44900000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.244,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.333}]},{"osd":1,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.004,"15min":1.0369999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.337,"5min":1.758,"15min":1.758},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.034,"15min":1.0740000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.45600000000000002},"max":{"1min":1.133,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.1659999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.0229999999999999,"15min":1.0740000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.1639999999999999,"5min":1.7949999999999999,"15min":1.9510000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.2689999999999999,"15min":1.3540000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.359,"5min":1.8600000000000001,"15min":1.921},"last":1.2949999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4530000000000001,"15min":1.484},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.77600000000000002},"max":{"1min":1.806,"5min":1.956,"15min":1.974},"last":1.514},{"interface":"front","average":{"1min":1.236,"5min":1.391,"15min":1.4119999999999999},"min":{"1min":0.82699999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.607,"5min":1.8939999999999999,"15min":2.0539999999999998},"last":1.254}]},{"osd":5,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.363,"15min":1.389},"min":{"1min":0.83899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5329999999999999,"5min":2.028,"15min":2.0539999999999998},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.4219999999999999,"15min":1.4670000000000001},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.80300000000000005},"max":{"1min":1.575,"5min":1.9990000000000001,"15min":2.3540000000000001},"last":1.698}]},{"osd":6,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.4139999999999999,"15min":1.4410000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.881,"5min":2.2989999999999999,"15min":2.3220000000000001},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.696,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.589}]},{"osd":7,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.512,"15min":1.5369999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6850000000000001,"5min":2.1339999999999999,"15min":2.431},"last":1.871},{"interface":"front","average":{"1min":1.381,"5min":1.5740000000000001,"15min":1.603},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.956,"5min":2.552,"15min":2.552},"last":2.0030000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542345,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44844,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326996,"statfs":{"total":96636764160,"available":96590843904,"internally_reserved":0,"allocated":1204224,"data_stored":568363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.90100000000000002,"15min":0.91600000000000004},"min":{"1min":0.44600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.494,"5min":1.742,"15min":1.7889999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.0549999999999999,"15min":1.093},"min":{"1min":0.67500000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.6579999999999999,"5min":1.7070000000000001,"15min":2.085},"last":1.3819999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1399999999999999,"15min":1.127},"min":{"1min":0.68000000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.5800000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.044,"5min":1.153,"15min":1.2210000000000001},"min":{"1min":0.56999999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.4370000000000001,"5min":2.347,"15min":2.4289999999999998},"last":1.0660000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.056,"15min":1.0820000000000001},"min":{"1min":0.502,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3839999999999999,"5min":1.6879999999999999,"15min":1.859},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.282,"15min":1.3049999999999999},"min":{"1min":0.56100000000000005,"5min":0.434,"15min":0.434},"max":{"1min":1.6000000000000001,"5min":2.1000000000000001,"15min":2.1970000000000001},"last":1.972}]},{"osd":4,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4399999999999999,"15min":1.452},"min":{"1min":1.0940000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.764,"5min":2.431,"15min":2.431},"last":1.526},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.395,"15min":1.4419999999999999},"min":{"1min":0.85099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7290000000000001,"5min":2.0670000000000002,"15min":2.2530000000000001},"last":1.181}]},{"osd":5,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.4650000000000001,"15min":1.4950000000000001},"min":{"1min":0.96299999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.7909999999999999,"5min":2.23,"15min":2.3250000000000002},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.365,"15min":1.421},"min":{"1min":0.84099999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.788,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.657}]},{"osd":6,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4059999999999999,"15min":1.4450000000000001},"min":{"1min":0.93400000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.881,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.413},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.542,"15min":1.5580000000000001},"min":{"1min":1.0640000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8400000000000001,"5min":2.294,"15min":2.3279999999999998},"last":1.8520000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:45 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.5660000000000001,"15min":1.607},"min":{"1min":0.97699999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.0539999999999998,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.296},{"interface":"front","average":{"1min":1.464,"5min":1.4950000000000001,"15min":1.5449999999999999},"min":{"1min":0.88,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.944,"5min":2.573,"15min":2.573},"last":1.7789999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542345,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58748,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313092,"statfs":{"total":96636764160,"available":96576606208,"internally_reserved":0,"allocated":1224704,"data_stored":575279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90400000000000003,"15min":0.91100000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6359999999999999,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.0329999999999999,"15min":1.0860000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.7050000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.002}]},{"osd":2,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.93400000000000005,"15min":0.98899999999999999},"min":{"1min":0.46000000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.151,"5min":1.8460000000000001,"15min":2.262},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1579999999999999,"15min":1.1830000000000001},"min":{"1min":0.58399999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.9910000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.3919999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.254,"15min":1.319},"min":{"1min":0.51600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.77,"5min":1.988,"15min":2.2349999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.175,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.587,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.4490000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2749999999999999,"15min":1.3129999999999999},"min":{"1min":0.88400000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.647,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4370000000000001,"15min":1.4430000000000001},"min":{"1min":0.67900000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.6839999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.371,"15min":1.4319999999999999},"min":{"1min":0.77800000000000002,"5min":0.755,"15min":0.755},"max":{"1min":1.9390000000000001,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.77},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.411,"15min":1.431},"min":{"1min":0.93899999999999995,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":2.1179999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.5189999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.504,"15min":1.532},"min":{"1min":0.72099999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.202},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.472,"15min":1.5189999999999999},"min":{"1min":0.73899999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":2.1640000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.5620000000000001,"15min":1.617},"min":{"1min":0.82099999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.5269999999999999,"15min":1.5620000000000001},"min":{"1min":0.82599999999999996,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.9430000000000001,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.7230000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542346,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":1157119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.85299999999999998,"15min":0.89300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.655,"5min":1.655,"15min":2.5720000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.0509999999999999,"15min":1.089},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.439},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":2.1800000000000002},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.0660000000000001,"15min":1.127},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.579,"5min":1.8120000000000001,"15min":2.7650000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.1140000000000001,"15min":1.159},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.5880000000000001,"5min":1.952,"15min":1.952},"last":0.501}]},{"osd":3,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.016,"15min":1.0820000000000001},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.609,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.226,"15min":1.3009999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.696,"5min":1.8740000000000001,"15min":3.6579999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2929999999999999,"15min":1.3300000000000001},"min":{"1min":0.71899999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.542,"5min":1.8660000000000001,"15min":2.6920000000000002},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.3799999999999999,"15min":1.4390000000000001},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.845,"5min":1.966,"15min":3.544},"last":0.77800000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.377,"15min":1.4219999999999999},"min":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8320000000000001,"5min":2.2229999999999999,"15min":2.2440000000000002},"last":0.755},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3560000000000001,"15min":1.417},"min":{"1min":0.872,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.7490000000000001,"5min":2.02,"15min":2.8610000000000002},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.488,"15min":1.5289999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.9790000000000001,"5min":2.0750000000000002,"15min":3.7890000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.4470000000000001,"15min":1.516},"min":{"1min":0.92000000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.95,"5min":2.145,"15min":2.363},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5620000000000001,"15min":1.6200000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.056,"5min":2.3490000000000002,"15min":2.4729999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.389,"5min":1.4990000000000001,"15min":1.552},"min":{"1min":0.81599999999999995,"5min":0.755,"15min":0.755},"max":{"1min":1.8560000000000001,"5min":2.2050000000000001,"15min":3.1989999999999998},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:50:10.968 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:50:10.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:50:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.247+0000 7fc71ae44640 1 Processor -- start 2023-10-16T12:50:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.247+0000 7fc71ae44640 1 -- start start 2023-10-16T12:50:11.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc71ae44640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc714160420 0x7fc7140ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc71ae44640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc714160800 0x7fc7140cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc71ae44640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71406e170 con 0x7fc7140adc40 2023-10-16T12:50:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc71ae44640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7140cde30 con 0x7fc714160800 2023-10-16T12:50:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc71ae44640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7140cd460 con 0x7fc714160420 2023-10-16T12:50:11.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc713fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc714160800 0x7fc7140cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.249+0000 7fc713fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc714160800 0x7fc7140cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42688/0 (socket says 172.21.15.60:42688) 2023-10-16T12:50:11.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.250+0000 7fc713fff640 1 -- 172.21.15.60:0/1818908957 learned_addr learned my addr 172.21.15.60:0/1818908957 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:11.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.250+0000 7fc713fff640 1 -- 172.21.15.60:0/1818908957 >> v2:172.21.15.144:3300/0 conn(0x7fc714160420 msgr2=0x7fc7140ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:11.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.250+0000 7fc713fff640 1 --2- 172.21.15.60:0/1818908957 >> v2:172.21.15.144:3300/0 conn(0x7fc714160420 0x7fc7140ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.250+0000 7fc713fff640 1 -- 172.21.15.60:0/1818908957 >> v1:172.21.15.144:6789/0 conn(0x7fc7140adc40 legacy=0x7fc71406adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.250+0000 7fc713fff640 1 -- 172.21.15.60:0/1818908957 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71406e3c0 con 0x7fc714160800 2023-10-16T12:50:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.251+0000 7fc713fff640 1 --2- 172.21.15.60:0/1818908957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160800 0x7fc7140cd8f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc71406f470 tx=0x7fc70002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2301532bce98c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:11.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.251+0000 7fc7137fe640 1 -- 172.21.15.60:0/1818908957 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700031870 con 0x7fc714160800 2023-10-16T12:50:11.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc7137fe640 1 -- 172.21.15.60:0/1818908957 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7000319d0 con 0x7fc714160800 2023-10-16T12:50:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc7137fe640 1 -- 172.21.15.60:0/1818908957 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700031d60 con 0x7fc714160800 2023-10-16T12:50:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 -- 172.21.15.60:0/1818908957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160800 msgr2=0x7fc7140cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1818908957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160800 0x7fc7140cd8f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc71406f470 tx=0x7fc70002b910 comp rx=0 tx=0).stop 2023-10-16T12:50:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 -- 172.21.15.60:0/1818908957 shutdown_connections 2023-10-16T12:50:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1818908957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160800 0x7fc7140cd8f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1818908957 >> v2:172.21.15.144:3300/0 conn(0x7fc714160420 0x7fc7140ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 -- 172.21.15.60:0/1818908957 >> 172.21.15.60:0/1818908957 conn(0x7fc7140b0d30 msgr2=0x7fc7140b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:11.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 -- 172.21.15.60:0/1818908957 shutdown_connections 2023-10-16T12:50:11.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.252+0000 7fc71ae44640 1 -- 172.21.15.60:0/1818908957 wait complete. 2023-10-16T12:50:11.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.254+0000 7fc71ae44640 1 Processor -- start 2023-10-16T12:50:11.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.254+0000 7fc71ae44640 1 -- start start 2023-10-16T12:50:11.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.255+0000 7fc71ae44640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140adc40 0x7fc71415f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.255+0000 7fc71ae44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 0x7fc71415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.255+0000 7fc71ae44640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc714160800 0x7fc7141f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.255+0000 7fc71ae44640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140b29f0 con 0x7fc714160420 2023-10-16T12:50:11.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.255+0000 7fc71ae44640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71415a750 con 0x7fc714160800 2023-10-16T12:50:11.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.255+0000 7fc71ae44640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140b4530 con 0x7fc7140adc40 2023-10-16T12:50:11.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 0x7fc71415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 0x7fc71415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42696/0 (socket says 172.21.15.60:42696) 2023-10-16T12:50:11.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 -- 172.21.15.60:0/1973413635 learned_addr learned my addr 172.21.15.60:0/1973413635 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:11.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc718be0640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140adc40 0x7fc71415f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc7193e1640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc714160800 0x7fc7141f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 -- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140adc40 msgr2=0x7fc71415f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140adc40 0x7fc71415f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 -- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc714160800 msgr2=0x7fc7141f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc714160800 0x7fc7141f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc713fff640 1 -- 172.21.15.60:0/1973413635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000094d0 con 0x7fc714160420 2023-10-16T12:50:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc718be0640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140adc40 0x7fc71415f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:11.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.256+0000 7fc7193e1640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc714160800 0x7fc7141f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:11.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.257+0000 7fc713fff640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 0x7fc71415a210 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc7000090b0 tx=0x7fc7000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:11.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.257+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70003ad30 con 0x7fc714160420 2023-10-16T12:50:11.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.257+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7000443f0 con 0x7fc714160420 2023-10-16T12:50:11.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.257+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7000447a0 con 0x7fc714160420 2023-10-16T12:50:11.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.257+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71409d8c0 con 0x7fc714160420 2023-10-16T12:50:11.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.257+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7141f6e10 con 0x7fc714160420 2023-10-16T12:50:11.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.258+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71404e160 con 0x7fc714160420 2023-10-16T12:50:11.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.259+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc700044900 con 0x7fc714160420 2023-10-16T12:50:11.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.261+0000 7fc7117fa640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc6f806af90 0x7fc6f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.261+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc70002d040 con 0x7fc714160420 2023-10-16T12:50:11.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.262+0000 7fc718be0640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc6f806af90 0x7fc6f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.263+0000 7fc718be0640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc6f806af90 0x7fc6f806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc708006fd0 tx=0x7fc708008040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:11.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.265+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc700076f90 con 0x7fc714160420 2023-10-16T12:50:11.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.530+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc7140cd0c0 con 0x7fc714160420 2023-10-16T12:50:11.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.535+0000 7fc7117fa640 1 -- 172.21.15.60:0/1973413635 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc70007a5b0 con 0x7fc714160420 2023-10-16T12:50:11.538 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:50:11.538 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:50:11.538 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:50:11.539 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:50:11.539 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:50:11.541 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:50:11.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc6f806af90 msgr2=0x7fc6f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc6f806af90 0x7fc6f806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc708006fd0 tx=0x7fc708008040 comp rx=0 tx=0).stop 2023-10-16T12:50:11.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 msgr2=0x7fc71415a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 0x7fc71415a210 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc7000090b0 tx=0x7fc7000046c0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 shutdown_connections 2023-10-16T12:50:11.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fc6f806af90 0x7fc6f806d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc714160800 0x7fc7141f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc714160420 0x7fc71415a210 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 --2- 172.21.15.60:0/1973413635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140adc40 0x7fc71415f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.537+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 >> 172.21.15.60:0/1973413635 conn(0x7fc7140b0d30 msgr2=0x7fc7140c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:11.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.538+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 shutdown_connections 2023-10-16T12:50:11.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.538+0000 7fc71ae44640 1 -- 172.21.15.60:0/1973413635 wait complete. 2023-10-16T12:50:11.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:11.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.778+0000 7fe13091a640 1 Processor -- start 2023-10-16T12:50:11.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.778+0000 7fe13091a640 1 -- start start 2023-10-16T12:50:11.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.780+0000 7fe13091a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe12c0ad970 0x7fe12c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.780+0000 7fe13091a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c0ccfb0 0x7fe12c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.780+0000 7fe13091a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12c15b9d0 con 0x7fe12c06c040 2023-10-16T12:50:11.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.780+0000 7fe13091a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12c15bcb0 con 0x7fe12c0ad970 2023-10-16T12:50:11.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.780+0000 7fe13091a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c15bf90 con 0x7fe12c0ccfb0 2023-10-16T12:50:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12b7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe12c06c040 0x7fe12c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:52756/0 (socket says 172.21.15.60:52756) 2023-10-16T12:50:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12b7fe640 1 -- 172.21.15.60:0/961099955 learned_addr learned my addr 172.21.15.60:0/961099955 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12bfff640 1 --2- 172.21.15.60:0/961099955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c0ccfb0 0x7fe12c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12affd640 1 --2- 172.21.15.60:0/961099955 >> v2:172.21.15.144:3300/0 conn(0x7fe12c0ad970 0x7fe12c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12bfff640 1 -- 172.21.15.60:0/961099955 >> v2:172.21.15.144:3300/0 conn(0x7fe12c0ad970 msgr2=0x7fe12c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12bfff640 1 --2- 172.21.15.60:0/961099955 >> v2:172.21.15.144:3300/0 conn(0x7fe12c0ad970 0x7fe12c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.781+0000 7fe12bfff640 1 -- 172.21.15.60:0/961099955 >> v1:172.21.15.144:6789/0 conn(0x7fe12c06c040 legacy=0x7fe12c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.782+0000 7fe12bfff640 1 -- 172.21.15.60:0/961099955 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c0cc270 con 0x7fe12c0ccfb0 2023-10-16T12:50:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.783+0000 7fe12bfff640 1 --2- 172.21.15.60:0/961099955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c0ccfb0 0x7fe12c0cd460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe124009d90 tx=0x7fe12402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1dd5f0f11ad92a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe12a7fc640 1 -- 172.21.15.60:0/961099955 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12402cc50 con 0x7fe12c0ccfb0 2023-10-16T12:50:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe12a7fc640 1 -- 172.21.15.60:0/961099955 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12402cdb0 con 0x7fe12c0ccfb0 2023-10-16T12:50:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe12a7fc640 1 -- 172.21.15.60:0/961099955 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124034660 con 0x7fe12c0ccfb0 2023-10-16T12:50:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe13091a640 1 -- 172.21.15.60:0/961099955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c0ccfb0 msgr2=0x7fe12c0cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe13091a640 1 --2- 172.21.15.60:0/961099955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c0ccfb0 0x7fe12c0cd460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe124009d90 tx=0x7fe12402bea0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe13091a640 1 -- 172.21.15.60:0/961099955 shutdown_connections 2023-10-16T12:50:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe13091a640 1 --2- 172.21.15.60:0/961099955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c0ccfb0 0x7fe12c0cd460 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.784+0000 7fe13091a640 1 --2- 172.21.15.60:0/961099955 >> v2:172.21.15.144:3300/0 conn(0x7fe12c0ad970 0x7fe12c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.785+0000 7fe13091a640 1 -- 172.21.15.60:0/961099955 >> 172.21.15.60:0/961099955 conn(0x7fe12c0ce6a0 msgr2=0x7fe12c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:11.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.785+0000 7fe13091a640 1 -- 172.21.15.60:0/961099955 shutdown_connections 2023-10-16T12:50:11.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.785+0000 7fe13091a640 1 -- 172.21.15.60:0/961099955 wait complete. 2023-10-16T12:50:11.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.786+0000 7fe13091a640 1 Processor -- start 2023-10-16T12:50:11.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.787+0000 7fe13091a640 1 -- start start 2023-10-16T12:50:11.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.787+0000 7fe13091a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c06c040 0x7fe12c0c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.787+0000 7fe13091a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 0x7fe12c159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.787+0000 7fe13091a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe12c0ccfb0 0x7fe12c0b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe13091a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0c64b0 con 0x7fe12c0ad970 2023-10-16T12:50:11.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe13091a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0b08e0 con 0x7fe12c0ccfb0 2023-10-16T12:50:11.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe13091a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0c58a0 con 0x7fe12c06c040 2023-10-16T12:50:11.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 0x7fe12c159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 0x7fe12c159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42702/0 (socket says 172.21.15.60:42702) 2023-10-16T12:50:11.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 -- 172.21.15.60:0/3491325243 learned_addr learned my addr 172.21.15.60:0/3491325243 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:11.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12b7fe640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c06c040 0x7fe12c0c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12bfff640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe12c0ccfb0 0x7fe12c0b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 -- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c06c040 msgr2=0x7fe12c0c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c06c040 0x7fe12c0c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 -- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe12c0ccfb0 msgr2=0x7fe12c0b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe12c0ccfb0 0x7fe12c0b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.788+0000 7fe12affd640 1 -- 172.21.15.60:0/3491325243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124009a70 con 0x7fe12c0ad970 2023-10-16T12:50:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.789+0000 7fe12affd640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 0x7fe12c159030 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e980 tx=0x7fe12000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:11.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.789+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120009670 con 0x7fe12c0ad970 2023-10-16T12:50:11.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.789+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12000ca00 con 0x7fe12c0ad970 2023-10-16T12:50:11.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.789+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12000cd90 con 0x7fe12c0ad970 2023-10-16T12:50:11.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.789+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c0c5b80 con 0x7fe12c0ad970 2023-10-16T12:50:11.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.789+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c0c6140 con 0x7fe12c0ad970 2023-10-16T12:50:11.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.790+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c0c6300 con 0x7fe12c0ad970 2023-10-16T12:50:11.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.791+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe120013470 con 0x7fe12c0ad970 2023-10-16T12:50:11.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.792+0000 7fe113fff640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe11806af90 0x7fe11806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:11.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.792+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1200899c0 con 0x7fe12c0ad970 2023-10-16T12:50:11.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.793+0000 7fe12b7fe640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe11806af90 0x7fe11806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:11.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.794+0000 7fe12b7fe640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe11806af90 0x7fe11806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe11c005880 tx=0x7fe11c009040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:11.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:11.797+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe120054950 con 0x7fe12c0ad970 2023-10-16T12:50:11.859 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T12:50:11.856+0000 7f5ad2dba640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:50:12.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.035+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe12c06b330 con 0x7fe12c0ad970 2023-10-16T12:50:12.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.036+0000 7fe113fff640 1 -- 172.21.15.60:0/3491325243 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe120017090 con 0x7fe12c0ad970 2023-10-16T12:50:12.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:12.039 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:05.354835+0000","modified":"2023-10-16T12:50:08.431520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/4244730882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4244730882},{"type":"v1","addr":"172.21.15.60:6835","nonce":4244730882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:50:12.040 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2023-10-16T12:50:12.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe11806af90 msgr2=0x7fe11806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe11806af90 0x7fe11806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe11c005880 tx=0x7fe11c009040 comp rx=0 tx=0).stop 2023-10-16T12:50:12.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 msgr2=0x7fe12c159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 0x7fe12c159030 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e980 tx=0x7fe12000ee40 comp rx=0 tx=0).stop 2023-10-16T12:50:12.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 shutdown_connections 2023-10-16T12:50:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe11806af90 0x7fe11806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe12c0ccfb0 0x7fe12c0b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe12c0ad970 0x7fe12c159030 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 --2- 172.21.15.60:0/3491325243 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe12c06c040 0x7fe12c0c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 >> 172.21.15.60:0/3491325243 conn(0x7fe12c0ce6a0 msgr2=0x7fe12c15d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 shutdown_connections 2023-10-16T12:50:12.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.039+0000 7fe13091a640 1 -- 172.21.15.60:0/3491325243 wait complete. 2023-10-16T12:50:12.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:50:05.354835+0000', 'modified': '2023-10-16T12:50:08.431520+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6894}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6894': {'gid': 6894, 'name': 'a', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/4244730882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 4244730882}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 4244730882}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:50:12.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:50:12.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:50:12.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.702+0000 7fe2e9541640 1 Processor -- start 2023-10-16T12:50:12.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.702+0000 7fe2e9541640 1 -- start start 2023-10-16T12:50:12.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e9541640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2e406a4b0 0x7fe2e40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e9541640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e9541640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e40cb030 con 0x7fe2e415ad70 2023-10-16T12:50:12.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e9541640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e40cae30 con 0x7fe2e406a4b0 2023-10-16T12:50:12.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e9541640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e40c91c0 con 0x7fe2e40c9ca0 2023-10-16T12:50:12.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2dbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2e406a4b0 0x7fe2e40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2e415ad70 0x7fe2e4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49542/0 (socket says 172.21.15.60:49542) 2023-10-16T12:50:12.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/1135927049 learned_addr learned my addr 172.21.15.60:0/1135927049 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:12.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1135927049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 msgr2=0x7fe2e40ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.704+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1135927049 >> v1:172.21.15.60:6789/0 conn(0x7fe2e415ad70 legacy=0x7fe2e4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.705+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1135927049 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e40c94a0 con 0x7fe2e406a4b0 2023-10-16T12:50:12.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.705+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.706+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ca3b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0097a0 tx=0x7fe2cc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e266585ad0cf9406 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.706+0000 7fe2e27fc640 1 -- 172.21.15.60:0/1135927049 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc02d040 con 0x7fe2e406a4b0 2023-10-16T12:50:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.706+0000 7fe2e27fc640 1 -- 172.21.15.60:0/1135927049 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc031810 con 0x7fe2e406a4b0 2023-10-16T12:50:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.706+0000 7fe2e27fc640 1 -- 172.21.15.60:0/1135927049 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc031ba0 con 0x7fe2e406a4b0 2023-10-16T12:50:12.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 -- 172.21.15.60:0/1135927049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 msgr2=0x7fe2e40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ca3b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0097a0 tx=0x7fe2cc02b8b0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 -- 172.21.15.60:0/1135927049 shutdown_connections 2023-10-16T12:50:12.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 --2- 172.21.15.60:0/1135927049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ca3b0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 -- 172.21.15.60:0/1135927049 >> 172.21.15.60:0/1135927049 conn(0x7fe2e40ce9b0 msgr2=0x7fe2e40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 -- 172.21.15.60:0/1135927049 shutdown_connections 2023-10-16T12:50:12.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.707+0000 7fe2e9541640 1 -- 172.21.15.60:0/1135927049 wait complete. 2023-10-16T12:50:12.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.709+0000 7fe2e9541640 1 Processor -- start 2023-10-16T12:50:12.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.709+0000 7fe2e9541640 1 -- start start 2023-10-16T12:50:12.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e9541640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e9541640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e9541640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2e415ad70 0x7fe2e4150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e9541640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e40add40 con 0x7fe2e406a4b0 2023-10-16T12:50:12.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e9541640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e40cd630 con 0x7fe2e415ad70 2023-10-16T12:50:12.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e9541640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e40cd910 con 0x7fe2e40c9ca0 2023-10-16T12:50:12.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42738/0 (socket says 172.21.15.60:42738) 2023-10-16T12:50:12.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/3719341401 learned_addr learned my addr 172.21.15.60:0/3719341401 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:12.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2e415ad70 0x7fe2e4150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 msgr2=0x7fe2e40ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2e415ad70 msgr2=0x7fe2e4150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.710+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2e415ad70 0x7fe2e4150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/3719341401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc009400 con 0x7fe2e406a4b0 2023-10-16T12:50:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:12.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ce110 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400c480 tx=0x7fe2d400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2db7fe640 1 -- 172.21.15.60:0/3719341401 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4005ce0 con 0x7fe2e406a4b0 2023-10-16T12:50:12.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2db7fe640 1 -- 172.21.15.60:0/3719341401 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d4005e40 con 0x7fe2e406a4b0 2023-10-16T12:50:12.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2db7fe640 1 -- 172.21.15.60:0/3719341401 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40072f0 con 0x7fe2e406a4b0 2023-10-16T12:50:12.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e40cdbf0 con 0x7fe2e406a4b0 2023-10-16T12:50:12.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.711+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e40ae020 con 0x7fe2e406a4b0 2023-10-16T12:50:12.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.714+0000 7fe2db7fe640 1 -- 172.21.15.60:0/3719341401 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2d4007450 con 0x7fe2e406a4b0 2023-10-16T12:50:12.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.715+0000 7fe2db7fe640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2c406af90 0x7fe2c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.715+0000 7fe2db7fe640 1 -- 172.21.15.60:0/3719341401 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2d4088c50 con 0x7fe2e406a4b0 2023-10-16T12:50:12.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.716+0000 7fe2ec1b1740 1 Processor -- start 2023-10-16T12:50:12.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.716+0000 7fe2ec1b1740 1 -- start start 2023-10-16T12:50:12.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.716+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2c406af90 0x7fe2c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341064e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341094d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410653c0 0x564341095250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564340f857a0 con 0x564341092360 2023-10-16T12:50:12.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564340fd7740 con 0x564341091f80 2023-10-16T12:50:12.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x564340fd4e30 con 0x5643410653c0 2023-10-16T12:50:12.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2c406af90 0x7fe2c406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc036000 tx=0x7fe2cc03c040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341094d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341094d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42748/0 (socket says 172.21.15.60:42748) 2023-10-16T12:50:12.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.717+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1517802525 learned_addr learned my addr 172.21.15.60:0/1517802525 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:12.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341064e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410653c0 0x564341095250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410653c0 msgr2=0x564341095250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410653c0 0x564341095250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 msgr2=0x564341064e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341064e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2dbfff640 1 -- 172.21.15.60:0/1517802525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564340f94910 con 0x564341092360 2023-10-16T12:50:12.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.718+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341094d10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009890 tx=0x7fe2cc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b094e18f3a7ab33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2d8ff9640 1 -- 172.21.15.60:0/1517802525 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc03adb0 con 0x564341092360 2023-10-16T12:50:12.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2d8ff9640 1 -- 172.21.15.60:0/1517802525 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2cc04f440 con 0x564341092360 2023-10-16T12:50:12.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2d8ff9640 1 -- 172.21.15.60:0/1517802525 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc04f7b0 con 0x564341092360 2023-10-16T12:50:12.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1517802525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 msgr2=0x564341094d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341094d10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009890 tx=0x7fe2cc030000 comp rx=0 tx=0).stop 2023-10-16T12:50:12.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1517802525 shutdown_connections 2023-10-16T12:50:12.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410653c0 0x564341095250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341094d10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1517802525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341064e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.719+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1517802525 >> 172.21.15.60:0/1517802525 conn(0x564341067190 msgr2=0x5643410afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.720+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1517802525 shutdown_connections 2023-10-16T12:50:12.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.720+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1517802525 wait complete. 2023-10-16T12:50:12.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.720+0000 7fe2ec1b1740 1 -- start start 2023-10-16T12:50:12.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.721+0000 7fe2ec1b1740 1 Processor -- start 2023-10-16T12:50:12.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x56434107ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341082050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410aa720 0x564341080660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564340fd0720 con 0x564341092360 2023-10-16T12:50:12.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564340f6e970 con 0x564341091f80 2023-10-16T12:50:12.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x564340f6e610 con 0x5643410aa720 2023-10-16T12:50:12.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2ec1b1740 10 client.0 _ll_register_callbacks cb 0x564340f909d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:50:12.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341082050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341082050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42754/0 (socket says 172.21.15.60:42754) 2023-10-16T12:50:12.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2dbfff640 1 -- 172.21.15.60:0/3000953870 learned_addr learned my addr 172.21.15.60:0/3000953870 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:12.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410aa720 0x564341080660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x56434107ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2dbfff640 1 -- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410aa720 msgr2=0x564341080660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.722+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410aa720 0x564341080660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2dbfff640 1 -- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 msgr2=0x56434107ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x56434107ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2dbfff640 1 -- 172.21.15.60:0/3000953870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e40c94a0 con 0x564341092360 2023-10-16T12:50:12.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341082050 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc047040 tx=0x7fe2cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cca5b4cff377c80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:50:12.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/3000953870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0454c0 con 0x564341092360 2023-10-16T12:50:12.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/3000953870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2cc045620 con 0x564341092360 2023-10-16T12:50:12.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.723+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564340ff7160 con 0x564341092360 2023-10-16T12:50:12.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/3000953870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2cc0457e0 con 0x564341092360 2023-10-16T12:50:12.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 10 client.7029 fetch_fsmap learned FSMap version 43 2023-10-16T12:50:12.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x564340fceda0 con 0x564341092360 2023-10-16T12:50:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/3000953870 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564340ff7160 con 0x564341092360 2023-10-16T12:50:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/3000953870 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 43) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe2cc045940 con 0x564341092360 2023-10-16T12:50:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 10 client.7029 fetch_fsmap finished waiting for FSMap version 43 2023-10-16T12:50:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 10 client.7029 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:50:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 20 client.7029 populate_metadata read hostname 'smithi060' 2023-10-16T12:50:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x564341082b20 0x564341096d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 4 client.7029 mds_command: new command op to 6894 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:50:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 --> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564340fa2cc0 con 0x564341082b20 2023-10-16T12:50:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.724+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x564341082b20 0x564341096d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.725+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x564341082b20 0x564341096d50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb1cd50dd792dd8 server_cookie=c724dd8fee0c1863 in_seq=0 out_seq=0 2023-10-16T12:50:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.725+0000 7fe2c1ffb640 10 client.7029 ms_handle_connect on v2:172.21.15.60:6834/4244730882 2023-10-16T12:50:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/3000953870 <== mds.0 v2:172.21.15.60:6834/4244730882 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564340fa2cc0 con 0x564341082b20 2023-10-16T12:50:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2c1ffb640 10 client.7029 handle_command_reply: tid=0 2023-10-16T12:50:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 client.7029 shutdown 2023-10-16T12:50:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 msgr2=0x564341082050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341082050 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc047040 tx=0x7fe2cc004730 comp rx=0 tx=0).stop 2023-10-16T12:50:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 shutdown_connections 2023-10-16T12:50:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x564341082b20 0x564341096d50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5643410aa720 0x564341080660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341092360 0x564341082050 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3000953870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x56434107ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2c1ffb640 0 client.7029 ms_handle_reset on v2:172.21.15.60:6834/4244730882 2023-10-16T12:50:12.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 >> 172.21.15.60:0/3000953870 conn(0x5643410653c0 msgr2=0x5643410b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 shutdown_connections 2023-10-16T12:50:12.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3000953870 wait complete. 2023-10-16T12:50:12.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.726+0000 7fe2ec1b1740 20 client.7029 trim_cache size 0 max 16384 2023-10-16T12:50:12.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.743+0000 7fe2ec1b1740 1 Processor -- start 2023-10-16T12:50:12.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 -- start start 2023-10-16T12:50:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x56434110f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 0x5643410b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564340d34f20 con 0x5643410aa720 2023-10-16T12:50:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643410aa580 con 0x564341091f80 2023-10-16T12:50:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.744+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5643410aa330 con 0x564341092360 2023-10-16T12:50:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 0x5643410b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 0x5643410b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42768/0 (socket says 172.21.15.60:42768) 2023-10-16T12:50:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 -- 172.21.15.60:0/3365033634 learned_addr learned my addr 172.21.15.60:0/3365033634 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x56434110f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 -- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 msgr2=0x56434110f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x56434110f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 -- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 msgr2=0x564341069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341069190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.745+0000 7fe2e37fe640 1 -- 172.21.15.60:0/3365033634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564340f6db80 con 0x5643410aa720 2023-10-16T12:50:12.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.746+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 0x5643410b23c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc008fa0 tx=0x7fe2dc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec0bf6045a98ad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.746+0000 7fe2d97fa640 1 -- 172.21.15.60:0/3365033634 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00b8e0 con 0x5643410aa720 2023-10-16T12:50:12.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.746+0000 7fe2d97fa640 1 -- 172.21.15.60:0/3365033634 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc013040 con 0x5643410aa720 2023-10-16T12:50:12.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.746+0000 7fe2d97fa640 1 -- 172.21.15.60:0/3365033634 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00bc80 con 0x5643410aa720 2023-10-16T12:50:12.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3365033634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 msgr2=0x5643410b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 0x5643410b23c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc008fa0 tx=0x7fe2dc00c8b0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3365033634 shutdown_connections 2023-10-16T12:50:12.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x56434110f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5643410aa720 0x5643410b23c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/3365033634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564341091f80 0x564341069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3365033634 >> 172.21.15.60:0/3365033634 conn(0x5643410653c0 msgr2=0x564341068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3365033634 shutdown_connections 2023-10-16T12:50:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.747+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/3365033634 wait complete. 2023-10-16T12:50:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.748+0000 7fe2ec1b1740 1 -- start start 2023-10-16T12:50:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.748+0000 7fe2ec1b1740 1 Processor -- start 2023-10-16T12:50:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 0x564341068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x564341100130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5643410aa720 0x564341085440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564341100670 con 0x564341091f80 2023-10-16T12:50:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56434110ea80 con 0x5643410aa720 2023-10-16T12:50:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5643410a6070 con 0x564341092360 2023-10-16T12:50:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2ec1b1740 10 client.0 _ll_register_callbacks cb 0x7fe2d40577a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:50:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2e2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 0x564341068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2e2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 0x564341068ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42782/0 (socket says 172.21.15.60:42782) 2023-10-16T12:50:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.749+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/1989549679 learned_addr learned my addr 172.21.15.60:0/1989549679 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5643410aa720 0x564341085440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x564341100130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 msgr2=0x564341100130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x564341100130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5643410aa720 msgr2=0x564341085440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5643410aa720 0x564341085440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e2ffd640 1 -- 172.21.15.60:0/1989549679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc008bd0 con 0x564341091f80 2023-10-16T12:50:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e37fe640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5643410aa720 0x564341085440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x564341100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2e2ffd640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 0x564341068ef0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40078c0 tx=0x7fe2d4055cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06174945ad6728a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:50:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/1989549679 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40872d0 con 0x564341091f80 2023-10-16T12:50:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/1989549679 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2d4056e40 con 0x564341091f80 2023-10-16T12:50:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2cc04f440 con 0x564341091f80 2023-10-16T12:50:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.750+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/1989549679 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2d4057000 con 0x564341091f80 2023-10-16T12:50:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 10 client.7035 fetch_fsmap learned FSMap version 43 2023-10-16T12:50:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x564340f93170 con 0x564341091f80 2023-10-16T12:50:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/1989549679 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d4075020 con 0x564341091f80 2023-10-16T12:50:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/1989549679 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 43) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe2d4058bd0 con 0x564341091f80 2023-10-16T12:50:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 10 client.7035 fetch_fsmap finished waiting for FSMap version 43 2023-10-16T12:50:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 10 client.7035 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:50:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 20 client.7035 populate_metadata read hostname 'smithi060' 2023-10-16T12:50:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x56434110dcd0 0x56434110e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 4 client.7035 mds_command: new command op to 6894 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:50:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 --> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564340fc9700 con 0x56434110dcd0 2023-10-16T12:50:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.751+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x56434110dcd0 0x56434110e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2c1ffb640 10 client.7035 ms_handle_connect on v2:172.21.15.60:6834/4244730882 2023-10-16T12:50:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2dbfff640 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x56434110dcd0 0x56434110e0b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afd205c5cbacbec6 server_cookie=e2bbf7f8ba3d920a in_seq=0 out_seq=0 2023-10-16T12:50:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2c1ffb640 1 -- 172.21.15.60:0/1989549679 <== mds.0 v2:172.21.15.60:6834/4244730882 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x564340fc9700 con 0x56434110dcd0 2023-10-16T12:50:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2c1ffb640 10 client.7035 handle_command_reply: tid=0 2023-10-16T12:50:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 client.7035 shutdown 2023-10-16T12:50:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 msgr2=0x564341068ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 0x564341068ef0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40078c0 tx=0x7fe2d4055cf0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 shutdown_connections 2023-10-16T12:50:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:6834/4244730882,v1:172.21.15.60:6835/4244730882] conn(0x56434110dcd0 0x56434110e0b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x564341092360 0x564341100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5643410aa720 0x564341085440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 --2- 172.21.15.60:0/1989549679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564341091f80 0x564341068ef0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 >> 172.21.15.60:0/1989549679 conn(0x5643410653c0 msgr2=0x5643410b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.752+0000 7fe2c1ffb640 0 client.7035 ms_handle_reset on v2:172.21.15.60:6834/4244730882 2023-10-16T12:50:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 shutdown_connections 2023-10-16T12:50:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2ec1b1740 1 -- 172.21.15.60:0/1989549679 wait complete. 2023-10-16T12:50:12.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2ec1b1740 20 client.7035 trim_cache size 0 max 16384 2023-10-16T12:50:12.773 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:50:12.773 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:50:12.774 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:50:12.774 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 6894, 2023-10-16T12:50:12.774 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:50:12.774 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:50:12.774 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:50:12.775 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8497791399999999, 2023-10-16T12:50:12.775 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 43, 2023-10-16T12:50:12.775 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 99, 2023-10-16T12:50:12.775 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 99, 2023-10-16T12:50:12.775 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.635166492 2023-10-16T12:50:12.776 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:50:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2c406af90 msgr2=0x7fe2c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2c406af90 0x7fe2c406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc036000 tx=0x7fe2cc03c040 comp rx=0 tx=0).stop 2023-10-16T12:50:12.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 msgr2=0x7fe2e40ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ce110 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400c480 tx=0x7fe2d400c940 comp rx=0 tx=0).stop 2023-10-16T12:50:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 shutdown_connections 2023-10-16T12:50:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe2c406af90 0x7fe2c406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2e40c9ca0 0x7fe2e40ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2e415ad70 0x7fe2e4150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 --2- 172.21.15.60:0/3719341401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2e406a4b0 0x7fe2e40ce110 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.753+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 >> 172.21.15.60:0/3719341401 conn(0x7fe2e40ce9b0 msgr2=0x7fe2e40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.754+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 shutdown_connections 2023-10-16T12:50:12.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:12.754+0000 7fe2e9541640 1 -- 172.21.15.60:0/3719341401 wait complete. 2023-10-16T12:50:12.780 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:50:13.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.004+0000 7f2ae2a0e640 1 Processor -- start 2023-10-16T12:50:13.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.004+0000 7f2ae2a0e640 1 -- start start 2023-10-16T12:50:13.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2ae2a0e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2adc0cd0c0 0x7f2adc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:13.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2ae2a0e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2adc0b0410 0x7f2adc0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:13.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2ae2a0e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc15e3e0 con 0x7f2adc15d3f0 2023-10-16T12:50:13.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2ae2a0e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc0b0de0 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2ae2a0e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc0cd4a0 con 0x7f2adc0b0410 2023-10-16T12:50:13.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2adbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2adc0cd0c0 0x7f2adc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:13.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2adbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2adc0cd0c0 0x7f2adc1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42794/0 (socket says 172.21.15.60:42794) 2023-10-16T12:50:13.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.006+0000 7f2adbfff640 1 -- 172.21.15.60:0/2340261804 learned_addr learned my addr 172.21.15.60:0/2340261804 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:13.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.007+0000 7f2adbfff640 1 -- 172.21.15.60:0/2340261804 >> v2:172.21.15.144:3300/0 conn(0x7f2adc0b0410 msgr2=0x7f2adc0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:13.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.007+0000 7f2adbfff640 1 --2- 172.21.15.60:0/2340261804 >> v2:172.21.15.144:3300/0 conn(0x7f2adc0b0410 0x7f2adc0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.007+0000 7f2adbfff640 1 -- 172.21.15.60:0/2340261804 >> v1:172.21.15.60:6789/0 conn(0x7f2adc15d3f0 legacy=0x7f2adc15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:13.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.007+0000 7f2adbfff640 1 -- 172.21.15.60:0/2340261804 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc15e100 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.008+0000 7f2adbfff640 1 --2- 172.21.15.60:0/2340261804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc1615f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0cdad0 tx=0x7f2acc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ddb6265e4f1e44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:13.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.008+0000 7f2adaffd640 1 -- 172.21.15.60:0/2340261804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc031880 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.008+0000 7f2adaffd640 1 -- 172.21.15.60:0/2340261804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc0319e0 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.008+0000 7f2adaffd640 1 -- 172.21.15.60:0/2340261804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc031d70 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/2340261804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 msgr2=0x7f2adc1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:13.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/2340261804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc1615f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0cdad0 tx=0x7f2acc02b920 comp rx=0 tx=0).stop 2023-10-16T12:50:13.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/2340261804 shutdown_connections 2023-10-16T12:50:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/2340261804 >> v2:172.21.15.144:3300/0 conn(0x7f2adc0b0410 0x7f2adc0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/2340261804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc1615f0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/2340261804 >> 172.21.15.60:0/2340261804 conn(0x7f2adc0cb410 msgr2=0x7f2adc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/2340261804 shutdown_connections 2023-10-16T12:50:13.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.009+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/2340261804 wait complete. 2023-10-16T12:50:13.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.011+0000 7f2ae2a0e640 1 Processor -- start 2023-10-16T12:50:13.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.011+0000 7f2ae2a0e640 1 -- start start 2023-10-16T12:50:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2ae2a0e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2adc0b0410 0x7f2adc1e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2ae2a0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc0b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2ae2a0e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2adc15d3f0 0x7f2adc0af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2ae2a0e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0b26d0 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2ae2a0e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0c3900 con 0x7f2adc15d3f0 2023-10-16T12:50:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2ae2a0e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc1e8e00 con 0x7f2adc0b0410 2023-10-16T12:50:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2adb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc0b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2adb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc0b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42808/0 (socket says 172.21.15.60:42808) 2023-10-16T12:50:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.012+0000 7f2adb7fe640 1 -- 172.21.15.60:0/3888292004 learned_addr learned my addr 172.21.15.60:0/3888292004 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adbfff640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2adc0b0410 0x7f2adc1e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2ae0fab640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2adc15d3f0 0x7f2adc0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adb7fe640 1 -- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2adc0b0410 msgr2=0x7f2adc1e88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adb7fe640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2adc0b0410 0x7f2adc1e88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adb7fe640 1 -- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2adc15d3f0 msgr2=0x7f2adc0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adb7fe640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2adc15d3f0 0x7f2adc0af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adb7fe640 1 -- 172.21.15.60:0/3888292004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0094d0 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2adb7fe640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc0b3b60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000b410 tx=0x7f2ad000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0011830 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad00119b0 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0011d60 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.013+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc0b23f0 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.014+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc0c3590 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.015+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2adc0c3750 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.016+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ad001a400 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.017+0000 7f2ad8ff9640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2ac806afc0 0x7f2ac806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.017+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ad0089960 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.017+0000 7f2adbfff640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2ac806afc0 0x7f2ac806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.019+0000 7f2adbfff640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2ac806afc0 0x7f2ac806d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2acc036000 tx=0x7f2acc03c040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.021+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ad0054860 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.255+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2adc0c3110 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.256+0000 7f2ad8ff9640 1 -- 172.21.15.60:0/3888292004 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2adc0c3110 con 0x7f2adc0cd0c0 2023-10-16T12:50:13.258 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:50:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.258+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2ac806afc0 msgr2=0x7f2ac806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:13.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.258+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2ac806afc0 0x7f2ac806d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2acc036000 tx=0x7f2acc03c040 comp rx=0 tx=0).stop 2023-10-16T12:50:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.258+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 msgr2=0x7f2adc0b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.258+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc0b3b60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000b410 tx=0x7f2ad000b8d0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 shutdown_connections 2023-10-16T12:50:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f2ac806afc0 0x7f2ac806d480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2adc15d3f0 0x7f2adc0af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2adc0cd0c0 0x7f2adc0b3b60 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 --2- 172.21.15.60:0/3888292004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2adc0b0410 0x7f2adc1e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 >> 172.21.15.60:0/3888292004 conn(0x7f2adc0cb410 msgr2=0x7f2adc0c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 shutdown_connections 2023-10-16T12:50:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:13.259+0000 7f2ae2a0e640 1 -- 172.21.15.60:0/3888292004 wait complete. 2023-10-16T12:50:13.281 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:50:13.281 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:50:13.282 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:50:13.282 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:50:13.283 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:13.283 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:50:13.283 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:50:13.283 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:50:13.284 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:50:13.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:13.285 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:50:13.342 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:50:13.342 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:50:13.342 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:50:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:50:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.343 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:50:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:50:13.344 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:50:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:50:13.344 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:50:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:50:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:50:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41374sec preferred_lft 41374sec 2023-10-16T12:50:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:50:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.346 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:50:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:50:13.347 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:50:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:50:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:50:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.348 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:50:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:50:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:50:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.348 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:50:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:50:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:50:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:13.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:13.351 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:50:13.450 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:50:13.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:13.455 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:50:13.515 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:50:13.515 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:50:13.515 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:50:13.515 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:50:13.516 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:50:13.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:13.517 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:50:13.518 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:50:13.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:50:13.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:50:13.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:50:13.519 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:50:13.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:13.721 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:50:13.777 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:50:13.797 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:50:13.798 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:50:13.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:13.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:50:13.910 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:50:13.910 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:13.987 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:50:13.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:50:13.988 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:14.049 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:50:14.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:14.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:50:14.196 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:50:14.196 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:50:14.229 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:50:14.229 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:50:14.229 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:50:14.230 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:50:14.230 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:50:14.232 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:50:14.233 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:50:14.233 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:50:14.233 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:50:14.233 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:50:14.237 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:50:14.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:14.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:14.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:14.329 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:50:14.395 INFO:teuthology.orchestra.run.smithi060.stdout:client5627 2023-10-16T12:50:14.396 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:50:14.396 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:14.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:50:14.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.676+0000 7fdd9ec90640 1 Processor -- start 2023-10-16T12:50:14.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.676+0000 7fdd9ec90640 1 -- start start 2023-10-16T12:50:14.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.677+0000 7fdd9ec90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd9815ad70 0x7fdd98069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:14.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.677+0000 7fdd9ec90640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd980caac0 0x7fdd9806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:14.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.677+0000 7fdd9ec90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd980cb030 con 0x7fdd980c9ca0 2023-10-16T12:50:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.677+0000 7fdd9ec90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980c91c0 con 0x7fdd9815ad70 2023-10-16T12:50:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.677+0000 7fdd9ec90640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980c94a0 con 0x7fdd980caac0 2023-10-16T12:50:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd9ca2c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd9815ad70 0x7fdd98069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd97fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdd980c9ca0 0x7fdd980ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49552/0 (socket says 172.21.15.60:49552) 2023-10-16T12:50:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd97fff640 1 -- 172.21.15.60:0/3613867575 learned_addr learned my addr 172.21.15.60:0/3613867575 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:14.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd9d22d640 1 --2- 172.21.15.60:0/3613867575 >> v2:172.21.15.144:3300/0 conn(0x7fdd980caac0 0x7fdd9806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:14.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd9ca2c640 1 -- 172.21.15.60:0/3613867575 >> v2:172.21.15.144:3300/0 conn(0x7fdd980caac0 msgr2=0x7fdd9806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:14.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd9ca2c640 1 --2- 172.21.15.60:0/3613867575 >> v2:172.21.15.144:3300/0 conn(0x7fdd980caac0 0x7fdd9806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd9ca2c640 1 -- 172.21.15.60:0/3613867575 >> v1:172.21.15.60:6789/0 conn(0x7fdd980c9ca0 legacy=0x7fdd980ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:14.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd9ca2c640 1 -- 172.21.15.60:0/3613867575 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd980c9780 con 0x7fdd9815ad70 2023-10-16T12:50:14.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.678+0000 7fdd977fe640 1 -- 172.21.15.60:0/3613867575 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863410146 0 0) 0x7fdd980cb030 con 0x7fdd980c9ca0 2023-10-16T12:50:14.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.680+0000 7fdd9ca2c640 1 --2- 172.21.15.60:0/3613867575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9815ad70 0x7fdd98069da0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdd9806d090 tx=0x7fdd8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6315c2e6c7c746 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:14.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.680+0000 7fdd977fe640 1 -- 172.21.15.60:0/3613867575 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88031830 con 0x7fdd9815ad70 2023-10-16T12:50:14.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.680+0000 7fdd977fe640 1 -- 172.21.15.60:0/3613867575 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd88031990 con 0x7fdd9815ad70 2023-10-16T12:50:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.680+0000 7fdd977fe640 1 -- 172.21.15.60:0/3613867575 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88031d20 con 0x7fdd9815ad70 2023-10-16T12:50:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.680+0000 7fdd9ec90640 1 -- 172.21.15.60:0/3613867575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9815ad70 msgr2=0x7fdd98069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.680+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/3613867575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9815ad70 0x7fdd98069da0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdd9806d090 tx=0x7fdd8802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.681+0000 7fdd9ec90640 1 -- 172.21.15.60:0/3613867575 shutdown_connections 2023-10-16T12:50:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.681+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/3613867575 >> v2:172.21.15.144:3300/0 conn(0x7fdd980caac0 0x7fdd9806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.681+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/3613867575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9815ad70 0x7fdd98069da0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.681+0000 7fdd9ec90640 1 -- 172.21.15.60:0/3613867575 >> 172.21.15.60:0/3613867575 conn(0x7fdd980ce9b0 msgr2=0x7fdd980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.681+0000 7fdd9ec90640 1 -- 172.21.15.60:0/3613867575 shutdown_connections 2023-10-16T12:50:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.681+0000 7fdd9ec90640 1 -- 172.21.15.60:0/3613867575 wait complete. 2023-10-16T12:50:14.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.682+0000 7fdd9ec90640 1 Processor -- start 2023-10-16T12:50:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.683+0000 7fdd9ec90640 1 -- start start 2023-10-16T12:50:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ec90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd980c9ca0 0x7fdd9814a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ec90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 0x7fdd98150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ec90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9815ad70 0x7fdd9814aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ec90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd981f24b0 con 0x7fdd980caac0 2023-10-16T12:50:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ec90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9814cf50 con 0x7fdd9815ad70 2023-10-16T12:50:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ec90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9815b450 con 0x7fdd980c9ca0 2023-10-16T12:50:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 0x7fdd98150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 0x7fdd98150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42830/0 (socket says 172.21.15.60:42830) 2023-10-16T12:50:14.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd97fff640 1 -- 172.21.15.60:0/2241926919 learned_addr learned my addr 172.21.15.60:0/2241926919 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:14.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9d22d640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9815ad70 0x7fdd9814aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:14.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.684+0000 7fdd9ca2c640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd980c9ca0 0x7fdd9814a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:14.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd97fff640 1 -- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd980c9ca0 msgr2=0x7fdd9814a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:14.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd97fff640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd980c9ca0 0x7fdd9814a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd97fff640 1 -- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9815ad70 msgr2=0x7fdd9814aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd97fff640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9815ad70 0x7fdd9814aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd97fff640 1 -- 172.21.15.60:0/2241926919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009400 con 0x7fdd980caac0 2023-10-16T12:50:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd97fff640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 0x7fdd98150a10 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00c890 tx=0x7fdd8c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00a640 con 0x7fdd980caac0 2023-10-16T12:50:14.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c00a7c0 con 0x7fdd980caac0 2023-10-16T12:50:14.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9815b730 con 0x7fdd980caac0 2023-10-16T12:50:14.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00ab70 con 0x7fdd980caac0 2023-10-16T12:50:14.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.685+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9815bcf0 con 0x7fdd980caac0 2023-10-16T12:50:14.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.687+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9804e160 con 0x7fdd980caac0 2023-10-16T12:50:14.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.688+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd8c00acd0 con 0x7fdd980caac0 2023-10-16T12:50:14.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.689+0000 7fdd957fa640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fdd8406af90 0x7fdd8406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:14.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.689+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd8c088f30 con 0x7fdd980caac0 2023-10-16T12:50:14.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.689+0000 7fdd9ca2c640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fdd8406af90 0x7fdd8406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:14.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.691+0000 7fdd9ca2c640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fdd8406af90 0x7fdd8406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdd88036000 tx=0x7fdd8803c040 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:14.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.694+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd8c053f60 con 0x7fdd980caac0 2023-10-16T12:50:14.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.919+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 --> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd980ad4d0 con 0x7fdd8406af90 2023-10-16T12:50:14.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.923+0000 7fdd957fa640 1 -- 172.21.15.60:0/2241926919 <== mgr.5863 v2:172.21.15.144:6832/1915751278 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdd88031b00 con 0x7fdd8406af90 2023-10-16T12:50:14.925 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:50:14.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:14.926 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:50:14.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:14.926 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:50:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fdd8406af90 msgr2=0x7fdd8406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fdd8406af90 0x7fdd8406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdd88036000 tx=0x7fdd8803c040 comp rx=0 tx=0).stop 2023-10-16T12:50:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 msgr2=0x7fdd98150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:14.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 0x7fdd98150a10 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00c890 tx=0x7fdd8c00cd50 comp rx=0 tx=0).stop 2023-10-16T12:50:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 shutdown_connections 2023-10-16T12:50:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fdd8406af90 0x7fdd8406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9815ad70 0x7fdd9814aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd980caac0 0x7fdd98150a10 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 --2- 172.21.15.60:0/2241926919 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd980c9ca0 0x7fdd9814a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.925+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 >> 172.21.15.60:0/2241926919 conn(0x7fdd980ce9b0 msgr2=0x7fdd980af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.926+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 shutdown_connections 2023-10-16T12:50:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:14.926+0000 7fdd9ec90640 1 -- 172.21.15.60:0/2241926919 wait complete. 2023-10-16T12:50:14.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:50:15.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.176+0000 7fe2b359e640 1 Processor -- start 2023-10-16T12:50:15.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.177+0000 7fe2b359e640 1 -- start start 2023-10-16T12:50:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.178+0000 7fe2b359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe2b40ad970 0x7fe2b40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.178+0000 7fe2b359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b40ccfb0 0x7fe2b40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.178+0000 7fe2b359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b415b9d0 con 0x7fe2b406c040 2023-10-16T12:50:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.178+0000 7fe2b359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b415bcb0 con 0x7fe2b40ad970 2023-10-16T12:50:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.178+0000 7fe2b359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415bf90 con 0x7fe2b40ccfb0 2023-10-16T12:50:15.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2b259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2b406c040 0x7fe2b406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49558/0 (socket says 172.21.15.60:49558) 2023-10-16T12:50:15.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2b259c640 1 -- 172.21.15.60:0/3766068639 learned_addr learned my addr 172.21.15.60:0/3766068639 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2abfff640 1 --2- 172.21.15.60:0/3766068639 >> v2:172.21.15.144:3300/0 conn(0x7fe2b40ad970 0x7fe2b40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693863847 0 0) 0x7fe2b415b9d0 con 0x7fe2b406c040 2023-10-16T12:50:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b40ccfb0 0x7fe2b40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a0003520 con 0x7fe2b406c040 2023-10-16T12:50:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.179+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465217548 0 0) 0x7fe2a0003520 con 0x7fe2b406c040 2023-10-16T12:50:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b40ccfb0 msgr2=0x7fe2b40cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b40ccfb0 0x7fe2b40cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 >> v2:172.21.15.144:3300/0 conn(0x7fe2b40ad970 msgr2=0x7fe2b40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 --2- 172.21.15.60:0/3766068639 >> v2:172.21.15.144:3300/0 conn(0x7fe2b40ad970 0x7fe2b40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40cc270 con 0x7fe2b406c040 2023-10-16T12:50:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe2a4001670 con 0x7fe2b406c040 2023-10-16T12:50:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 >> v1:172.21.15.60:6789/0 conn(0x7fe2b406c040 legacy=0x7fe2b406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0003970 0x7fe2a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a00047c0 0x7fe2a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2a0005180 0x7fe2a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415bcb0 con 0x7fe2a0005180 2023-10-16T12:50:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415bf90 con 0x7fe2a00047c0 2023-10-16T12:50:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a000bf40 con 0x7fe2a0003970 2023-10-16T12:50:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2b40ccfb0 0x7fe2b40cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2a0005180 0x7fe2a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2b259c640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0003970 0x7fe2a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.180+0000 7fe2abfff640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a00047c0 0x7fe2a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.181+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0003970 msgr2=0x7fe2a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.181+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0003970 0x7fe2a0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.181+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a00047c0 msgr2=0x7fe2a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.181+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a00047c0 0x7fe2a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.181+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/3766068639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a0003520 con 0x7fe2a0005180 2023-10-16T12:50:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2a0005180 0x7fe2a000b980 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009f00 tx=0x7fe2a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f9b17a78015e30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac02cc70 con 0x7fe2a0005180 2023-10-16T12:50:15.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac02cdd0 con 0x7fe2a0005180 2023-10-16T12:50:15.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b1d9b640 1 -- 172.21.15.60:0/3766068639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac003bb0 con 0x7fe2a0005180 2023-10-16T12:50:15.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b359e640 1 -- 172.21.15.60:0/3766068639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2a0005180 msgr2=0x7fe2a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:15.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b359e640 1 --2- 172.21.15.60:0/3766068639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2a0005180 0x7fe2a000b980 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009f00 tx=0x7fe2a00057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.182+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/3766068639 reap_dead start 2023-10-16T12:50:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.183+0000 7fe2b359e640 1 -- 172.21.15.60:0/3766068639 shutdown_connections 2023-10-16T12:50:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.183+0000 7fe2b359e640 1 -- 172.21.15.60:0/3766068639 >> 172.21.15.60:0/3766068639 conn(0x7fe2b40ce6a0 msgr2=0x7fe2b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.183+0000 7fe2b359e640 1 -- 172.21.15.60:0/3766068639 shutdown_connections 2023-10-16T12:50:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.183+0000 7fe2b359e640 1 -- 172.21.15.60:0/3766068639 wait complete. 2023-10-16T12:50:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.185+0000 7fe2b359e640 1 Processor -- start 2023-10-16T12:50:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.185+0000 7fe2b359e640 1 -- start start 2023-10-16T12:50:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0005180 0x7fe2b415db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b4144ff0 0x7fe2b40aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 0x7fe2b41e6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b415e050 con 0x7fe2b40aaa60 2023-10-16T12:50:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4145830 con 0x7fe2b4144ff0 2023-10-16T12:50:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4145b10 con 0x7fe2a0005180 2023-10-16T12:50:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 0x7fe2b41e6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 0x7fe2b41e6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42844/0 (socket says 172.21.15.60:42844) 2023-10-16T12:50:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.186+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/195352352 learned_addr learned my addr 172.21.15.60:0/195352352 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b259c640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0005180 0x7fe2b415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2abfff640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b4144ff0 0x7fe2b40aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0005180 msgr2=0x7fe2b415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0005180 0x7fe2b415db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b4144ff0 msgr2=0x7fe2b40aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:15.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b4144ff0 0x7fe2b40aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b2d9d640 1 -- 172.21.15.60:0/195352352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b4145df0 con 0x7fe2b40aaa60 2023-10-16T12:50:15.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b259c640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0005180 0x7fe2b415db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:15.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.187+0000 7fe2b2d9d640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 0x7fe2b41e6630 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac007360 tx=0x7fe2ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.188+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac03c070 con 0x7fe2b40aaa60 2023-10-16T12:50:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.188+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac003e30 con 0x7fe2b40aaa60 2023-10-16T12:50:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.188+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac004730 con 0x7fe2b40aaa60 2023-10-16T12:50:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.188+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b41460d0 con 0x7fe2b40aaa60 2023-10-16T12:50:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.188+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b4146690 con 0x7fe2b40aaa60 2023-10-16T12:50:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.189+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b406b330 con 0x7fe2b40aaa60 2023-10-16T12:50:15.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.190+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 36) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2ac037460 con 0x7fe2b40aaa60 2023-10-16T12:50:15.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.191+0000 7fe2aaffd640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe29406af90 0x7fe29406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:15.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.191+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2ac0aace0 con 0x7fe2b40aaa60 2023-10-16T12:50:15.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.192+0000 7fe2b259c640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe29406af90 0x7fe29406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:15.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.193+0000 7fe2b259c640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe29406af90 0x7fe29406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4005990 tx=0x7fe2a4000f90 comp rx=0 tx=0).ready entity=mgr.5863 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:15.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.196+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2ac075c80 con 0x7fe2b40aaa60 2023-10-16T12:50:15.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.442+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe2b4146880 con 0x7fe2b40aaa60 2023-10-16T12:50:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.991+0000 7fe2aaffd640 1 -- 172.21.15.60:0/195352352 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v37) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe2ac0460a0 con 0x7fe2b40aaa60 2023-10-16T12:50:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe29406af90 msgr2=0x7fe29406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe29406af90 0x7fe29406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4005990 tx=0x7fe2a4000f90 comp rx=0 tx=0).stop 2023-10-16T12:50:15.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 msgr2=0x7fe2b41e6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:15.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 0x7fe2b41e6630 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac007360 tx=0x7fe2ac004ab0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 shutdown_connections 2023-10-16T12:50:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7fe29406af90 0x7fe29406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2b40aaa60 0x7fe2b41e6630 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2b4144ff0 0x7fe2b40aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 --2- 172.21.15.60:0/195352352 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2a0005180 0x7fe2b415db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.995+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 >> 172.21.15.60:0/195352352 conn(0x7fe2b40ce6a0 msgr2=0x7fe2b406a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.996+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 shutdown_connections 2023-10-16T12:50:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:15.998+0000 7fe2b359e640 1 -- 172.21.15.60:0/195352352 wait complete. 2023-10-16T12:50:16.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:50:16.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.247+0000 7f1c9941f640 1 Processor -- start 2023-10-16T12:50:16.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.247+0000 7f1c9941f640 1 -- start start 2023-10-16T12:50:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c9941f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c94065ab0 0x7f1c940ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c9941f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1c940b4e50 0x7f1c94069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c9941f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940c9930 con 0x7f1c940caf80 2023-10-16T12:50:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c9941f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9415b1e0 con 0x7f1c94065ab0 2023-10-16T12:50:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c9941f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c940c9c10 con 0x7f1c940b4e50 2023-10-16T12:50:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c8a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c94065ab0 0x7f1c940ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c8a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c94065ab0 0x7f1c940ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42868/0 (socket says 172.21.15.60:42868) 2023-10-16T12:50:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c8a7fc640 1 -- 172.21.15.60:0/1719176332 learned_addr learned my addr 172.21.15.60:0/1719176332 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.249+0000 7f1c937fe640 1 --2- 172.21.15.60:0/1719176332 >> v2:172.21.15.144:3300/0 conn(0x7f1c940b4e50 0x7f1c94069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.250+0000 7f1c8a7fc640 1 -- 172.21.15.60:0/1719176332 >> v2:172.21.15.144:3300/0 conn(0x7f1c940b4e50 msgr2=0x7f1c94069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.250+0000 7f1c8a7fc640 1 --2- 172.21.15.60:0/1719176332 >> v2:172.21.15.144:3300/0 conn(0x7f1c940b4e50 0x7f1c94069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.250+0000 7f1c8a7fc640 1 -- 172.21.15.60:0/1719176332 >> v1:172.21.15.144:6789/0 conn(0x7f1c940caf80 legacy=0x7f1c940cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.250+0000 7f1c8a7fc640 1 -- 172.21.15.60:0/1719176332 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c940c9ef0 con 0x7f1c94065ab0 2023-10-16T12:50:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.251+0000 7f1c8a7fc640 1 --2- 172.21.15.60:0/1719176332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c940ca9d0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1c9406cdc0 tx=0x7f1c7c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5bee291d742e5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.251+0000 7f1c927fc640 1 -- 172.21.15.60:0/1719176332 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c02ca80 con 0x7f1c94065ab0 2023-10-16T12:50:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.251+0000 7f1c927fc640 1 -- 172.21.15.60:0/1719176332 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c02cbe0 con 0x7f1c94065ab0 2023-10-16T12:50:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.251+0000 7f1c927fc640 1 -- 172.21.15.60:0/1719176332 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0343f0 con 0x7f1c94065ab0 2023-10-16T12:50:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 -- 172.21.15.60:0/1719176332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 msgr2=0x7f1c940ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 --2- 172.21.15.60:0/1719176332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c940ca9d0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1c9406cdc0 tx=0x7f1c7c02bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 -- 172.21.15.60:0/1719176332 shutdown_connections 2023-10-16T12:50:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 --2- 172.21.15.60:0/1719176332 >> v2:172.21.15.144:3300/0 conn(0x7f1c940b4e50 0x7f1c94069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 --2- 172.21.15.60:0/1719176332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c940ca9d0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 -- 172.21.15.60:0/1719176332 >> 172.21.15.60:0/1719176332 conn(0x7f1c9406bc10 msgr2=0x7f1c9406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 -- 172.21.15.60:0/1719176332 shutdown_connections 2023-10-16T12:50:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.252+0000 7f1c9941f640 1 -- 172.21.15.60:0/1719176332 wait complete. 2023-10-16T12:50:16.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.254+0000 7f1c9941f640 1 Processor -- start 2023-10-16T12:50:16.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.254+0000 7f1c9941f640 1 -- start start 2023-10-16T12:50:16.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c9941f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c9414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c9941f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c940b4e50 0x7f1c940c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c9941f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c940caf80 0x7f1c940c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c9941f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940cec30 con 0x7f1c94065ab0 2023-10-16T12:50:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c9941f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940cda70 con 0x7f1c940b4e50 2023-10-16T12:50:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c9941f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940cdd50 con 0x7f1c940caf80 2023-10-16T12:50:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c92ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c9414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c92ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c9414ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42884/0 (socket says 172.21.15.60:42884) 2023-10-16T12:50:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c92ffd640 1 -- 172.21.15.60:0/2017081720 learned_addr learned my addr 172.21.15.60:0/2017081720 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.255+0000 7f1c8a7fc640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c940b4e50 0x7f1c940c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c937fe640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c940caf80 0x7f1c940c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c92ffd640 1 -- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c940caf80 msgr2=0x7f1c940c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c92ffd640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c940caf80 0x7f1c940c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c92ffd640 1 -- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c940b4e50 msgr2=0x7f1c940c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c92ffd640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c940b4e50 0x7f1c940c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c92ffd640 1 -- 172.21.15.60:0/2017081720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c009b30 con 0x7f1c94065ab0 2023-10-16T12:50:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c92ffd640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c9414ea60 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400c450 tx=0x7f1c8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.256+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840052d0 con 0x7f1c94065ab0 2023-10-16T12:50:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.257+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c84005430 con 0x7f1c94065ab0 2023-10-16T12:50:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.257+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c940cd740 con 0x7f1c94065ab0 2023-10-16T12:50:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.257+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84012870 con 0x7f1c94065ab0 2023-10-16T12:50:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.257+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c940c2e30 con 0x7f1c94065ab0 2023-10-16T12:50:16.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.258+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c940cdf10 con 0x7f1c94065ab0 2023-10-16T12:50:16.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.259+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 37) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1c840129d0 con 0x7f1c94065ab0 2023-10-16T12:50:16.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.260+0000 7f1c8bfff640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 0x7f1c7406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.260+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c84089200 con 0x7f1c94065ab0 2023-10-16T12:50:16.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.260+0000 7f1c8a7fc640 1 -- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 msgr2=0x7f1c7406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:16.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.260+0000 7f1c8a7fc640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 0x7f1c7406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.264+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c84054230 con 0x7f1c94065ab0 2023-10-16T12:50:16.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.460+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1c940c3020 con 0x7f1c94065ab0 2023-10-16T12:50:16.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.461+0000 7f1c8a7fc640 1 -- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 msgr2=0x7f1c7406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:16.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.461+0000 7f1c8a7fc640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 0x7f1c7406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:16.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.465+0000 7f1c8bfff640 1 -- 172.21.15.60:0/2017081720 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1c940c3020 con 0x7f1c94065ab0 2023-10-16T12:50:16.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.467+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 msgr2=0x7f1c7406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:16.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.467+0000 7f1c9941f640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 0x7f1c7406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.467+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 msgr2=0x7f1c9414ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.467+0000 7f1c9941f640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c9414ea60 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400c450 tx=0x7f1c8400c910 comp rx=0 tx=0).stop 2023-10-16T12:50:16.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 shutdown_connections 2023-10-16T12:50:16.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f1c7406afe0 0x7f1c7406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c940caf80 0x7f1c940c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c940b4e50 0x7f1c940c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 --2- 172.21.15.60:0/2017081720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c94065ab0 0x7f1c9414ea60 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 >> 172.21.15.60:0/2017081720 conn(0x7f1c9406bc10 msgr2=0x7f1c940ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 shutdown_connections 2023-10-16T12:50:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.468+0000 7f1c9941f640 1 -- 172.21.15.60:0/2017081720 wait complete. 2023-10-16T12:50:16.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:50:16.512 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:16.510+0000 7f402bc2e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:50:16.554 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:16.551+0000 7fd8fa4b2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:50:16.565 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:16.563+0000 7f402bc2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:50:16.611 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:16.609+0000 7fd8fa4b2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:50:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.697+0000 7f44528e0640 1 Processor -- start 2023-10-16T12:50:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.698+0000 7f44528e0640 1 -- start start 2023-10-16T12:50:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.699+0000 7f44528e0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 0x7f444c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.699+0000 7f44528e0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444c161740 con 0x7f444c070830 2023-10-16T12:50:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.699+0000 7f44528e0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444c06b0f0 con 0x7f444c070c10 2023-10-16T12:50:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.699+0000 7f44528e0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c1619b0 con 0x7f444c0cf5f0 2023-10-16T12:50:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.699+0000 7f444bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f444c070830 0x7f444c06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49586/0 (socket says 172.21.15.60:49586) 2023-10-16T12:50:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.699+0000 7f444bfff640 1 -- 172.21.15.60:0/493120758 learned_addr learned my addr 172.21.15.60:0/493120758 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f4443fff640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 0x7f444c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869143307 0 0) 0x7f444c161740 con 0x7f444c070830 2023-10-16T12:50:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4438003520 con 0x7f444c070830 2023-10-16T12:50:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2200108140 0 0) 0x7f4438003520 con 0x7f444c070830 2023-10-16T12:50:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 msgr2=0x7f444c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 0x7f444c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 >> v1:172.21.15.144:6789/0 conn(0x7f444c070c10 legacy=0x7f444c0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c06b3d0 con 0x7f444c070830 2023-10-16T12:50:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f443c001670 con 0x7f444c070830 2023-10-16T12:50:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.700+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 >> v1:172.21.15.60:6789/0 conn(0x7f444c070830 legacy=0x7f444c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444b7fe640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44380039a0 0x7f4438003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444b7fe640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 0x7f44380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444b7fe640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4438004d00 0x7f4438005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c1619b0 con 0x7f44380039a0 2023-10-16T12:50:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443800bb10 con 0x7f4438004d00 2023-10-16T12:50:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443800bcd0 con 0x7f444c0cf5f0 2023-10-16T12:50:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444bfff640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44380039a0 0x7f4438003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f4450e7d640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4438004d00 0x7f4438005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f4443fff640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 0x7f44380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444bfff640 1 -- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 msgr2=0x7f44380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444bfff640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f444c0cf5f0 0x7f44380047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444bfff640 1 -- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4438004d00 msgr2=0x7f4438005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444bfff640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4438004d00 0x7f4438005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.701+0000 7f444bfff640 1 -- 172.21.15.60:0/493120758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4438003520 con 0x7f44380039a0 2023-10-16T12:50:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.703+0000 7f444bfff640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44380039a0 0x7f4438003d80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f443c00c190 tx=0x7f44380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d156020d868cab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.703+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c005ad0 con 0x7f44380039a0 2023-10-16T12:50:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.703+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c005c30 con 0x7f44380039a0 2023-10-16T12:50:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.703+0000 7f444b7fe640 1 -- 172.21.15.60:0/493120758 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c02c810 con 0x7f44380039a0 2023-10-16T12:50:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f44528e0640 1 -- 172.21.15.60:0/493120758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44380039a0 msgr2=0x7f4438003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f44528e0640 1 --2- 172.21.15.60:0/493120758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44380039a0 0x7f4438003d80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f443c00c190 tx=0x7f44380057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f4450e7d640 1 -- 172.21.15.60:0/493120758 reap_dead start 2023-10-16T12:50:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f44528e0640 1 -- 172.21.15.60:0/493120758 shutdown_connections 2023-10-16T12:50:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f44528e0640 1 -- 172.21.15.60:0/493120758 >> 172.21.15.60:0/493120758 conn(0x7f444c0cfac0 msgr2=0x7f444c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f44528e0640 1 -- 172.21.15.60:0/493120758 shutdown_connections 2023-10-16T12:50:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.704+0000 7f44528e0640 1 -- 172.21.15.60:0/493120758 wait complete. 2023-10-16T12:50:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.707+0000 7f44528e0640 1 Processor -- start 2023-10-16T12:50:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.707+0000 7f44528e0640 1 -- start start 2023-10-16T12:50:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.708+0000 7f44528e0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44380039a0 0x7f444c150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.708+0000 7f44528e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 0x7f444c0acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.708+0000 7f44528e0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f444c154570 0x7f444c0ad530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.708+0000 7f44528e0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0b1790 con 0x7f444c150bc0 2023-10-16T12:50:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.708+0000 7f44528e0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0b4bc0 con 0x7f444c154570 2023-10-16T12:50:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.708+0000 7f44528e0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0aff80 con 0x7f44380039a0 2023-10-16T12:50:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 0x7f444c0acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 0x7f444c0acff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42898/0 (socket says 172.21.15.60:42898) 2023-10-16T12:50:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 -- 172.21.15.60:0/3594019595 learned_addr learned my addr 172.21.15.60:0/3594019595 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f444bfff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44380039a0 0x7f444c150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 -- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44380039a0 msgr2=0x7f444c150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44380039a0 0x7f444c150680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4450e7d640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f444c154570 0x7f444c0ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 -- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f444c154570 msgr2=0x7f444c0ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f444c154570 0x7f444c0ad530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 -- 172.21.15.60:0/3594019595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c0af650 con 0x7f444c150bc0 2023-10-16T12:50:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f444bfff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44380039a0 0x7f444c150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:16.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.709+0000 7f4443fff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 0x7f444c0acff0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f44340108c0 tx=0x7f4434010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.710+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443400eb60 con 0x7f444c150bc0 2023-10-16T12:50:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.710+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443400ecc0 con 0x7f444c150bc0 2023-10-16T12:50:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.710+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443401c400 con 0x7f444c150bc0 2023-10-16T12:50:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.710+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c0af930 con 0x7f444c150bc0 2023-10-16T12:50:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.710+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444c0b4820 con 0x7f444c150bc0 2023-10-16T12:50:16.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.711+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444c0b0110 con 0x7f444c150bc0 2023-10-16T12:50:16.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.712+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 37) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f443401c560 con 0x7f444c150bc0 2023-10-16T12:50:16.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.713+0000 7f44497fa640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 0x7f442c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:16.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.714+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4434089d90 con 0x7f444c150bc0 2023-10-16T12:50:16.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.714+0000 7f444bfff640 1 -- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 msgr2=0x7f442c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:16.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.714+0000 7f444bfff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 0x7f442c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:16.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.718+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4434054dd0 con 0x7f444c150bc0 2023-10-16T12:50:16.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.912+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f444c04e160 con 0x7f444c150bc0 2023-10-16T12:50:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.915+0000 7f444bfff640 1 -- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 msgr2=0x7f442c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.915+0000 7f444bfff640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 0x7f442c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.919+0000 7f44497fa640 1 -- 172.21.15.60:0/3594019595 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f44340583f0 con 0x7f444c150bc0 2023-10-16T12:50:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 msgr2=0x7f442c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 0x7f442c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 msgr2=0x7f444c0acff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 0x7f444c0acff0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f44340108c0 tx=0x7f4434010d80 comp rx=0 tx=0).stop 2023-10-16T12:50:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 shutdown_connections 2023-10-16T12:50:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f442c06afe0 0x7f442c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f444c154570 0x7f444c0ad530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f444c150bc0 0x7f444c0acff0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 --2- 172.21.15.60:0/3594019595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44380039a0 0x7f444c150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 >> 172.21.15.60:0/3594019595 conn(0x7f444c0cfac0 msgr2=0x7f444c06c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 shutdown_connections 2023-10-16T12:50:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:16.922+0000 7f44528e0640 1 -- 172.21.15.60:0/3594019595 wait complete. 2023-10-16T12:50:16.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T12:50:17.013 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.010+0000 7f402bc2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:50:17.065 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.062+0000 7f402bc2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:50:17.094 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.092+0000 7fd8fa4b2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:50:17.114 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.112+0000 7f402bc2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:50:17.150 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.148+0000 7fd8fa4b2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:50:17.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.152+0000 7f9131997640 1 Processor -- start 2023-10-16T12:50:17.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.152+0000 7f9131997640 1 -- start start 2023-10-16T12:50:17.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.153+0000 7f9131997640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f912c0cb1c0 0x7f912c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:17.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.153+0000 7f9131997640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c065f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:17.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f9131997640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c15eff0 con 0x7f912c15af80 2023-10-16T12:50:17.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f9131997640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912c066480 con 0x7f912c0cb1c0 2023-10-16T12:50:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f9131997640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0d0930 con 0x7f912c065ab0 2023-10-16T12:50:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f912b7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c065f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f912a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f912c0cb1c0 0x7f912c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f912b7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c065f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:59510/0 (socket says 172.21.15.60:59510) 2023-10-16T12:50:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f912b7fe640 1 -- 172.21.15.60:0/2041544267 learned_addr learned my addr 172.21.15.60:0/2041544267 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.154+0000 7f912affd640 1 --1- 172.21.15.60:0/2041544267 >> v1:172.21.15.144:6789/0 conn(0x7f912c15af80 0x7f912c15f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:52784/0 (socket says 172.21.15.60:52784) 2023-10-16T12:50:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.155+0000 7f912a7fc640 1 -- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 msgr2=0x7f912c065f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.155+0000 7f912a7fc640 1 --2- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c065f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.155+0000 7f912a7fc640 1 -- 172.21.15.60:0/2041544267 >> v1:172.21.15.144:6789/0 conn(0x7f912c15af80 legacy=0x7f912c15f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.155+0000 7f912a7fc640 1 -- 172.21.15.60:0/2041544267 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c15ecc0 con 0x7f912c0cb1c0 2023-10-16T12:50:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.155+0000 7f912b7fe640 1 --2- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c065f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:50:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.156+0000 7f912a7fc640 1 --2- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c0cb1c0 0x7f912c06faf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f91200097d0 tx=0x7f912002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0046d50053b410c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.157+0000 7f9129ffb640 1 -- 172.21.15.60:0/2041544267 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120031780 con 0x7f912c0cb1c0 2023-10-16T12:50:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.157+0000 7f9129ffb640 1 -- 172.21.15.60:0/2041544267 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91200318e0 con 0x7f912c0cb1c0 2023-10-16T12:50:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.157+0000 7f9129ffb640 1 -- 172.21.15.60:0/2041544267 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120031c70 con 0x7f912c0cb1c0 2023-10-16T12:50:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 -- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c0cb1c0 msgr2=0x7f912c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 --2- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c0cb1c0 0x7f912c06faf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f91200097d0 tx=0x7f912002b820 comp rx=0 tx=0).stop 2023-10-16T12:50:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 -- 172.21.15.60:0/2041544267 shutdown_connections 2023-10-16T12:50:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 --2- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c065f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 --2- 172.21.15.60:0/2041544267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c0cb1c0 0x7f912c06faf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 -- 172.21.15.60:0/2041544267 >> 172.21.15.60:0/2041544267 conn(0x7f912c0d0ab0 msgr2=0x7f912c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 -- 172.21.15.60:0/2041544267 shutdown_connections 2023-10-16T12:50:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.158+0000 7f9131997640 1 -- 172.21.15.60:0/2041544267 wait complete. 2023-10-16T12:50:17.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.160+0000 7f9131997640 1 Processor -- start 2023-10-16T12:50:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.160+0000 7f9131997640 1 -- start start 2023-10-16T12:50:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.161+0000 7f9131997640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.161+0000 7f9131997640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f912c0cb1c0 0x7f912c0c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.161+0000 7f9131997640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c15af80 0x7f912c0c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f9131997640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0ca8d0 con 0x7f912c0cb1c0 2023-10-16T12:50:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f9131997640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0c3c40 con 0x7f912c15af80 2023-10-16T12:50:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f9131997640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0c9cc0 con 0x7f912c065ab0 2023-10-16T12:50:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f912c0cb1c0 0x7f912c0c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f912c0cb1c0 0x7f912c0c3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42900/0 (socket says 172.21.15.60:42900) 2023-10-16T12:50:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 -- 172.21.15.60:0/2856796202 learned_addr learned my addr 172.21.15.60:0/2856796202 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912b7fe640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c15af80 0x7f912c0c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 msgr2=0x7f912c147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f912c065ab0 0x7f912c147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c15af80 msgr2=0x7f912c0c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f912c15af80 0x7f912c0c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.162+0000 7f912a7fc640 1 -- 172.21.15.60:0/2856796202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124009b30 con 0x7f912c0cb1c0 2023-10-16T12:50:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.163+0000 7f912a7fc640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f912c0cb1c0 0x7f912c0c3700 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f91200097d0 tx=0x7f91200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.163+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120032820 con 0x7f912c0cb1c0 2023-10-16T12:50:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.163+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91200329a0 con 0x7f912c0cb1c0 2023-10-16T12:50:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.163+0000 7f9131997640 1 -- 172.21.15.60:0/2856796202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9120009400 con 0x7f912c0cb1c0 2023-10-16T12:50:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.163+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120032d50 con 0x7f912c0cb1c0 2023-10-16T12:50:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.163+0000 7f9131997640 1 -- 172.21.15.60:0/2856796202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c0ca280 con 0x7f912c0cb1c0 2023-10-16T12:50:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.165+0000 7f9131997640 1 -- 172.21.15.60:0/2856796202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912c04e160 con 0x7f912c0cb1c0 2023-10-16T12:50:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.166+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 37) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f912004e440 con 0x7f912c0cb1c0 2023-10-16T12:50:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.167+0000 7f9113fff640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 0x7f911806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.167+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91200ab9a0 con 0x7f912c0cb1c0 2023-10-16T12:50:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.167+0000 7f912affd640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 msgr2=0x7f911806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.167+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 0x7f911806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.171+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91200768e0 con 0x7f912c0cb1c0 2023-10-16T12:50:17.176 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.173+0000 7f402bc2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:50:17.206 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.204+0000 7fd8fa4b2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:50:17.224 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.222+0000 7f402bc2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:50:17.271 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.268+0000 7fd8fa4b2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:50:17.323 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.321+0000 7fd8fa4b2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:50:17.334 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.331+0000 7f402bc2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:50:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.368+0000 7f912affd640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 msgr2=0x7f911806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.368+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 0x7f911806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:17.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.388+0000 7f9131997640 1 -- 172.21.15.60:0/2856796202 --> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f912c15b360 con 0x7f911806afe0 2023-10-16T12:50:17.395 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.392+0000 7f402bc2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:17.440 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.438+0000 7fd8fa4b2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:50:17.444 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.441+0000 7f402bc2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:50:17.501 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.498+0000 7f402bc2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:50:17.506 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.504+0000 7fd8fa4b2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:17.550 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.548+0000 7f402bc2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:50:17.558 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.555+0000 7fd8fa4b2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:50:17.617 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.615+0000 7fd8fa4b2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:50:17.670 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:17.667+0000 7fd8fa4b2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:50:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.769+0000 7f912affd640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 msgr2=0x7f911806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:17.769+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 0x7f911806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:50:17.970 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:17.967+0000 7f402bc2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:50:18.071 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.068+0000 7fd8fa4b2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:50:18.126 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.123+0000 7f402bc2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:50:18.232 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.230+0000 7f402bc2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:50:18.238 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.236+0000 7fd8fa4b2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:50:18.281 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.279+0000 7f402bc2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:50:18.352 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.349+0000 7fd8fa4b2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:50:18.388 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.385+0000 7f402bc2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:18.404 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.402+0000 7fd8fa4b2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:50:18.495 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.492+0000 7f402bc2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:50:18.517 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.515+0000 7fd8fa4b2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:18.570+0000 7f912affd640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 msgr2=0x7f911806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1915751278 2023-10-16T12:50:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:18.570+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 0x7f911806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:50:18.630 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.627+0000 7fd8fa4b2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:50:18.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.646+0000 7f402bc2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:18.697 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.695+0000 7f402bc2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:18.794 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.791+0000 7fd8fa4b2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:18.815 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:18.812+0000 7f402bc2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:50:18.846 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.843+0000 7fd8fa4b2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:18.972 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:18.969+0000 7fd8fa4b2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:50:19.605 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:19.602+0000 7f402bc2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:50:19.859 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:19.856+0000 7fd8fa4b2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:50:19.899 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:19.897+0000 7f402bc2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:50:19.956 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:19.954+0000 7f402bc2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:50:20.068 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.065+0000 7f402bc2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:50:20.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:20.087+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 38) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f91200a9ec0 con 0x7f912c0cb1c0 2023-10-16T12:50:20.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:20.088+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 msgr2=0x7f911806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:20.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:20.088+0000 7f9113fff640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/1915751278,v1:172.21.15.144:6833/1915751278] conn(0x7f911806afe0 0x7f911806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:20.179 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:20.176+0000 7fd8fa4b2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:50:20.245 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:20.242+0000 7fd8fa4b2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:50:20.370 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3faa746640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.370 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3faa746640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.370 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3faa746640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3faa746640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3faa746640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3fa7f01640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3fa7f01640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3fa7f01640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3fa7f01640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:20.367+0000 7f3fa7f01640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:20.372 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:20.367+0000 7fd8fa4b2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:50:21.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.100+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 39) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9120039d90 con 0x7f912c0cb1c0 2023-10-16T12:50:21.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.101+0000 7f9113fff640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f9118035880 0x7f9118074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.101+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 --> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9118035c60 con 0x7f9118035880 2023-10-16T12:50:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.107+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f9118035880 0x7f9118074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.112+0000 7f912affd640 1 --2- 172.21.15.60:0/2856796202 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f9118035880 0x7f9118074ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f911c000f60 tx=0x7f911c001210 comp rx=0 tx=0).ready entity=mgr.5639 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:21.135 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:21.133+0000 7f3fb1f55640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T12:50:21.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.133+0000 7f9113fff640 1 -- 172.21.15.60:0/2856796202 <== mgr.5639 v2:172.21.15.144:6832/176900405 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f9118035c60 con 0x7f9118035880 2023-10-16T12:50:21.136 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T12:50:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:50:21.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:50:21.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.386+0000 7f2ce487b640 1 Processor -- start 2023-10-16T12:50:21.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.386+0000 7f2ce487b640 1 -- start start 2023-10-16T12:50:21.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.388+0000 7f2ce487b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ce00c9060 0x7f2ce006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.388+0000 7f2ce487b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00ca560 con 0x7f2ce006c9e0 2023-10-16T12:50:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.388+0000 7f2ce487b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce0165b70 con 0x7f2ce00c9440 2023-10-16T12:50:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.388+0000 7f2ce487b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce00c9950 con 0x7f2ce00c9060 2023-10-16T12:50:21.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.388+0000 7f2cd7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2ce006c9e0 0x7f2ce0164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49898/0 (socket says 172.21.15.60:49898) 2023-10-16T12:50:21.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.388+0000 7f2cd7fff640 1 -- 172.21.15.60:0/473684153 learned_addr learned my addr 172.21.15.60:0/473684153 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cded9d640 1 --2- 172.21.15.60:0/473684153 >> v2:172.21.15.144:3300/0 conn(0x7f2ce00c9060 0x7f2ce006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894184168 0 0) 0x7f2ce00ca560 con 0x7f2ce006c9e0 2023-10-16T12:50:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ccc003520 con 0x7f2ce006c9e0 2023-10-16T12:50:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107827287 0 0) 0x7f2ccc003520 con 0x7f2ce006c9e0 2023-10-16T12:50:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 >> v2:172.21.15.144:3300/0 conn(0x7f2ce00c9060 msgr2=0x7f2ce006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 --2- 172.21.15.60:0/473684153 >> v2:172.21.15.144:3300/0 conn(0x7f2ce00c9060 0x7f2ce006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 >> v1:172.21.15.144:6789/0 conn(0x7f2ce00c9440 legacy=0x7f2ce0165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce00c9c30 con 0x7f2ce006c9e0 2023-10-16T12:50:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2cc8001670 con 0x7f2ce006c9e0 2023-10-16T12:50:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 >> v1:172.21.15.60:6789/0 conn(0x7f2ce006c9e0 legacy=0x7f2ce0164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0039c0 0x7f2ccc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ccc0047c0 0x7f2ccc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc0051d0 0x7f2ccc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.389+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce00c9950 con 0x7f2ccc0051d0 2023-10-16T12:50:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc00c120 con 0x7f2ccc0047c0 2023-10-16T12:50:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc00c2e0 con 0x7f2ccc0039c0 2023-10-16T12:50:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc0051d0 0x7f2ccc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cd7fff640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ccc0047c0 0x7f2ccc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cded9d640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0039c0 0x7f2ccc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cdf59e640 1 -- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0039c0 msgr2=0x7f2ccc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0039c0 0x7f2ccc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cdf59e640 1 -- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ccc0047c0 msgr2=0x7f2ccc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ccc0047c0 0x7f2ccc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.390+0000 7f2cdf59e640 1 -- 172.21.15.60:0/473684153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc003520 con 0x7f2ccc0051d0 2023-10-16T12:50:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.391+0000 7f2cdf59e640 1 -- 172.21.15.60:0/473684153 reap_dead start 2023-10-16T12:50:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.391+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc0051d0 0x7f2ccc00b980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8001e20 tx=0x7f2ccc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c85c312b17d7b67 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8003b70 con 0x7f2ccc0051d0 2023-10-16T12:50:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8003cd0 con 0x7f2ccc0051d0 2023-10-16T12:50:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2cde59c640 1 -- 172.21.15.60:0/473684153 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80052f0 con 0x7f2ccc0051d0 2023-10-16T12:50:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2ce487b640 1 -- 172.21.15.60:0/473684153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc0051d0 msgr2=0x7f2ccc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2ce487b640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc0051d0 0x7f2ccc00b980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8001e20 tx=0x7f2ccc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2ce487b640 1 -- 172.21.15.60:0/473684153 shutdown_connections 2023-10-16T12:50:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2ce487b640 1 --2- 172.21.15.60:0/473684153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ccc0051d0 0x7f2ccc00b980 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.392+0000 7f2ce487b640 1 -- 172.21.15.60:0/473684153 >> 172.21.15.60:0/473684153 conn(0x7f2ce00ce6a0 msgr2=0x7f2ce00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.393+0000 7f2ce487b640 1 -- 172.21.15.60:0/473684153 shutdown_connections 2023-10-16T12:50:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.393+0000 7f2ce487b640 1 -- 172.21.15.60:0/473684153 wait complete. 2023-10-16T12:50:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.395+0000 7f2ce487b640 1 Processor -- start 2023-10-16T12:50:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.395+0000 7f2ce487b640 1 -- start start 2023-10-16T12:50:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2ce487b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0051d0 0x7f2ce0155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2ce487b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce0155640 0x7f2ce015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2ce487b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 0x7f2ce015a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2ce487b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce015adb0 con 0x7f2ce015a3f0 2023-10-16T12:50:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2ce487b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01615b0 con 0x7f2ce0155640 2023-10-16T12:50:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2ce487b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce0161890 con 0x7f2ccc0051d0 2023-10-16T12:50:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2cdf59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 0x7f2ce015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2cdf59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 0x7f2ce015a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53264/0 (socket says 172.21.15.60:53264) 2023-10-16T12:50:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2cdf59e640 1 -- 172.21.15.60:0/302777951 learned_addr learned my addr 172.21.15.60:0/302777951 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.396+0000 7f2cd7fff640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce0155640 0x7f2ce015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cded9d640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0051d0 0x7f2ce0155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cdf59e640 1 -- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0051d0 msgr2=0x7f2ce0155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0051d0 0x7f2ce0155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cdf59e640 1 -- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce0155640 msgr2=0x7f2ce015c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce0155640 0x7f2ce015c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cdf59e640 1 -- 172.21.15.60:0/302777951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce0161b70 con 0x7f2ce015a3f0 2023-10-16T12:50:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cd7fff640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce0155640 0x7f2ce015c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cdf59e640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 0x7f2ce015a870 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8001f40 tx=0x7f2cd80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.397+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8006d50 con 0x7f2ce015a3f0 2023-10-16T12:50:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.398+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8007020 con 0x7f2ce015a3f0 2023-10-16T12:50:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.398+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8007390 con 0x7f2ce015a3f0 2023-10-16T12:50:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.398+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce0161e50 con 0x7f2ce015a3f0 2023-10-16T12:50:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.398+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce015c800 con 0x7f2ce015a3f0 2023-10-16T12:50:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.399+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce0162370 con 0x7f2ce015a3f0 2023-10-16T12:50:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.400+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 39) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2cd803f030 con 0x7f2ce015a3f0 2023-10-16T12:50:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.401+0000 7f2cd77fe640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f2cc006ad10 0x7f2cc006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.401+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2cd8003e40 con 0x7f2ce015a3f0 2023-10-16T12:50:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.401+0000 7f2cded9d640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f2cc006ad10 0x7f2cc006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.403+0000 7f2cded9d640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f2cc006ad10 0x7f2cc006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0006fd0 tx=0x7f2cd0008040 comp rx=0 tx=0).ready entity=mgr.5639 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.406+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ce0162370 con 0x7f2ce015a3f0 2023-10-16T12:50:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:21.650+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f2ce0162370 con 0x7f2ce015a3f0 2023-10-16T12:50:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.106+0000 7f2cd77fe640 1 -- 172.21.15.60:0/302777951 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v40) v1 ==== 93+0+0 (secure 0 0 0) 0x7f2ce0162370 con 0x7f2ce015a3f0 2023-10-16T12:50:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.111+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f2cc006ad10 msgr2=0x7f2cc006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.111+0000 7f2ce487b640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f2cc006ad10 0x7f2cc006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0006fd0 tx=0x7f2cd0008040 comp rx=0 tx=0).stop 2023-10-16T12:50:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.111+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 msgr2=0x7f2ce015a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.111+0000 7f2ce487b640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 0x7f2ce015a870 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8001f40 tx=0x7f2cd80014a0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.112+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 shutdown_connections 2023-10-16T12:50:22.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.112+0000 7f2ce487b640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f2cc006ad10 0x7f2cc006d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.112+0000 7f2ce487b640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce015a3f0 0x7f2ce015a870 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.112+0000 7f2ce487b640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce0155640 0x7f2ce015c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.112+0000 7f2ce487b640 1 --2- 172.21.15.60:0/302777951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ccc0051d0 0x7f2ce0155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.112+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 >> 172.21.15.60:0/302777951 conn(0x7f2ce00ce6a0 msgr2=0x7f2ce015f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:22.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.113+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 shutdown_connections 2023-10-16T12:50:22.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.113+0000 7f2ce487b640 1 -- 172.21.15.60:0/302777951 wait complete. 2023-10-16T12:50:22.140 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:50:22.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:50:22.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:50:22.362 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:50:22.363 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:50:22.363 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:22.422 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:50:22.423 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:22.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:22.654 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:22.651+0000 7feb16812040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:50:22.657 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:22.655+0000 7f8c60e2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:50:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.677+0000 7ffad1d1d640 1 Processor -- start 2023-10-16T12:50:22.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.677+0000 7ffad1d1d640 1 -- start start 2023-10-16T12:50:22.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.678+0000 7ffad1d1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 0x7ffacc0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.678+0000 7ffad1d1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffacc0cee00 con 0x7ffacc0cf730 2023-10-16T12:50:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.678+0000 7ffad1d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffacc0aeba0 con 0x7ffacc15b010 2023-10-16T12:50:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.678+0000 7ffad1d1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0ae270 con 0x7ffacc0afee0 2023-10-16T12:50:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.678+0000 7ffacaffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffacc0cf730 0x7ffacc0af030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49914/0 (socket says 172.21.15.60:49914) 2023-10-16T12:50:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.678+0000 7ffacaffd640 1 -- 172.21.15.60:0/693810659 learned_addr learned my addr 172.21.15.60:0/693810659 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffacbfff640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 0x7ffacc0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46772260 0 0) 0x7ffac0001110 con 0x7ffacc0cf730 2023-10-16T12:50:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab4003520 con 0x7ffacc0cf730 2023-10-16T12:50:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3029055394 0 0) 0x7ffab4003520 con 0x7ffacc0cf730 2023-10-16T12:50:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 msgr2=0x7ffacc0ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 0x7ffacc0ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 >> v1:172.21.15.144:6789/0 conn(0x7ffacc15b010 legacy=0x7ffacc0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc0ae550 con 0x7ffacc0cf730 2023-10-16T12:50:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ffac0001800 con 0x7ffacc0cf730 2023-10-16T12:50:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 >> v1:172.21.15.60:6789/0 conn(0x7ffacc0cf730 legacy=0x7ffacc0af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffab40039a0 0x7ffab4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffab40047c0 0x7ffab4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 0x7ffab4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0ae270 con 0x7ffab40039a0 2023-10-16T12:50:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab400bb20 con 0x7ffab40047c0 2023-10-16T12:50:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.679+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab400bce0 con 0x7ffacc0afee0 2023-10-16T12:50:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffab40039a0 0x7ffab4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacb7fe640 1 -- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 msgr2=0x7ffab4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffacc0afee0 0x7ffab4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacb7fe640 1 -- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffab40047c0 msgr2=0x7ffab4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacaffd640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffab40047c0 0x7ffab4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffab40047c0 0x7ffab4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacb7fe640 1 -- 172.21.15.60:0/693810659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab4003520 con 0x7ffab40039a0 2023-10-16T12:50:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.680+0000 7ffacaffd640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffab40047c0 0x7ffab4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.681+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffab40039a0 0x7ffab4003dc0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ffabc007fd0 tx=0x7ffab40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ebd62c2ec5bc10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.681+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc0052d0 con 0x7ffab40039a0 2023-10-16T12:50:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffabc02e050 con 0x7ffab40039a0 2023-10-16T12:50:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffaca7fc640 1 -- 172.21.15.60:0/693810659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc003640 con 0x7ffab40039a0 2023-10-16T12:50:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffad1d1d640 1 -- 172.21.15.60:0/693810659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffab40039a0 msgr2=0x7ffab4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/693810659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffab40039a0 0x7ffab4003dc0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ffabc007fd0 tx=0x7ffab40057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffacbfff640 1 -- 172.21.15.60:0/693810659 reap_dead start 2023-10-16T12:50:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffad1d1d640 1 -- 172.21.15.60:0/693810659 shutdown_connections 2023-10-16T12:50:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffad1d1d640 1 -- 172.21.15.60:0/693810659 >> 172.21.15.60:0/693810659 conn(0x7ffacc0d0ad0 msgr2=0x7ffacc0d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffad1d1d640 1 -- 172.21.15.60:0/693810659 shutdown_connections 2023-10-16T12:50:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.682+0000 7ffad1d1d640 1 -- 172.21.15.60:0/693810659 wait complete. 2023-10-16T12:50:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.684+0000 7ffad1d1d640 1 Processor -- start 2023-10-16T12:50:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.684+0000 7ffad1d1d640 1 -- start start 2023-10-16T12:50:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffad1d1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffab40039a0 0x7ffacc0b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffad1d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffacc1598f0 0x7ffacc0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffad1d1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 0x7ffacc15c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffad1d1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0b4680 con 0x7ffacc158fd0 2023-10-16T12:50:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffad1d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0b5f40 con 0x7ffacc1598f0 2023-10-16T12:50:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffad1d1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0b40b0 con 0x7ffab40039a0 2023-10-16T12:50:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffacbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 0x7ffacc15c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffacbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 0x7ffacc15c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53288/0 (socket says 172.21.15.60:53288) 2023-10-16T12:50:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.685+0000 7ffacbfff640 1 -- 172.21.15.60:0/2184016871 learned_addr learned my addr 172.21.15.60:0/2184016871 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffab40039a0 0x7ffacc0b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacbfff640 1 -- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffab40039a0 msgr2=0x7ffacc0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacaffd640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffacc1598f0 0x7ffacc0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacbfff640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffab40039a0 0x7ffacc0b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacbfff640 1 -- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffacc1598f0 msgr2=0x7ffacc0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacbfff640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffacc1598f0 0x7ffacc0c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacbfff640 1 -- 172.21.15.60:0/2184016871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc0b21e0 con 0x7ffacc158fd0 2023-10-16T12:50:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacaffd640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffacc1598f0 0x7ffacc0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffacbfff640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 0x7ffacc15c930 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ffacc0b0390 tx=0x7ffac400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4010040 con 0x7ffacc158fd0 2023-10-16T12:50:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac400bab0 con 0x7ffacc158fd0 2023-10-16T12:50:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac400be40 con 0x7ffacc158fd0 2023-10-16T12:50:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.686+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc0b24a0 con 0x7ffacc158fd0 2023-10-16T12:50:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.687+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc0b42a0 con 0x7ffacc158fd0 2023-10-16T12:50:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.688+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc0b47c0 con 0x7ffacc158fd0 2023-10-16T12:50:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.689+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ffac401d400 con 0x7ffacc158fd0 2023-10-16T12:50:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.690+0000 7ffab3fff640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 0x7ffab806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.690+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ffac4089fe0 con 0x7ffacc158fd0 2023-10-16T12:50:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.690+0000 7ffacb7fe640 1 -- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 msgr2=0x7ffab806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.690+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 0x7ffab806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:22.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.694+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffacc0b47c0 con 0x7ffacc158fd0 2023-10-16T12:50:22.713 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:22.711+0000 7feb16812040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:50:22.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:22.714+0000 7f8c60e2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:50:22.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.891+0000 7ffacb7fe640 1 -- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 msgr2=0x7ffab806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:22.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.891+0000 7ffacb7fe640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 0x7ffab806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:22.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.893+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffacc0b47c0 con 0x7ffacc158fd0 2023-10-16T12:50:22.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.895+0000 7ffab3fff640 1 -- 172.21.15.60:0/2184016871 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+17066 (secure 0 0 0) 0x7ffacc0b47c0 con 0x7ffacc158fd0 2023-10-16T12:50:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:22.898 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":100,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:20.071860+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1468249085":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2269656792":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6832/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3435889746":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2501905996":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/1930765872":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6833/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3359683879":"2023-10-17T12:50:20.071781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:22.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 msgr2=0x7ffab806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:22.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 0x7ffab806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 msgr2=0x7ffacc15c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:22.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 0x7ffacc15c930 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ffacc0b0390 tx=0x7ffac400b480 comp rx=0 tx=0).stop 2023-10-16T12:50:22.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 shutdown_connections 2023-10-16T12:50:22.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7ffab806af90 0x7ffab806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffacc158fd0 0x7ffacc15c930 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffacc1598f0 0x7ffacc0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 --2- 172.21.15.60:0/2184016871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffab40039a0 0x7ffacc0b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:22.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.897+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 >> 172.21.15.60:0/2184016871 conn(0x7ffacc0d0ad0 msgr2=0x7ffacc06e590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:22.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.898+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 shutdown_connections 2023-10-16T12:50:22.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:22.898+0000 7ffad1d1d640 1 -- 172.21.15.60:0/2184016871 wait complete. 2023-10-16T12:50:22.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:50:03.305764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:50:04.385189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:50:22.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:50:23.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.126+0000 7fbdb9383640 1 Processor -- start 2023-10-16T12:50:23.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.126+0000 7fbdb9383640 1 -- start start 2023-10-16T12:50:23.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.127+0000 7fbdb9383640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbdb40661c0 0x7fbdb406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.127+0000 7fbdb9383640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb415eff0 con 0x7fbdb415af80 2023-10-16T12:50:23.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.127+0000 7fbdb9383640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb415ecc0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.127+0000 7fbdb9383640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb40d0930 con 0x7fbdb40661c0 2023-10-16T12:50:23.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbdb415af80 0x7fbdb415f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49926/0 (socket says 172.21.15.60:49926) 2023-10-16T12:50:23.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb2ffd640 1 -- 172.21.15.60:0/256265744 learned_addr learned my addr 172.21.15.60:0/256265744 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:23.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb37fe640 1 --2- 172.21.15.60:0/256265744 >> v2:172.21.15.60:3300/0 conn(0x7fbdb40661c0 0x7fbdb406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb1ffb640 1 -- 172.21.15.60:0/256265744 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988253135 0 0) 0x7fbdb415eff0 con 0x7fbdb415af80 2023-10-16T12:50:23.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb1ffb640 1 -- 172.21.15.60:0/256265744 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda0003520 con 0x7fbdb415af80 2023-10-16T12:50:23.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb37fe640 1 -- 172.21.15.60:0/256265744 >> v1:172.21.15.144:6789/0 conn(0x7fbdb40cb1c0 legacy=0x7fbdb4065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb37fe640 1 -- 172.21.15.60:0/256265744 >> v1:172.21.15.60:6789/0 conn(0x7fbdb415af80 legacy=0x7fbdb415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.128+0000 7fbdb37fe640 1 -- 172.21.15.60:0/256265744 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb415b360 con 0x7fbdb40661c0 2023-10-16T12:50:23.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.129+0000 7fbdb37fe640 1 --2- 172.21.15.60:0/256265744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40661c0 0x7fbdb406faf0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0097a0 tx=0x7fbdac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92e17223d2c74ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:23.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb1ffb640 1 -- 172.21.15.60:0/256265744 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac031830 con 0x7fbdb40661c0 2023-10-16T12:50:23.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb1ffb640 1 -- 172.21.15.60:0/256265744 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac031990 con 0x7fbdb40661c0 2023-10-16T12:50:23.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb1ffb640 1 -- 172.21.15.60:0/256265744 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac031d20 con 0x7fbdb40661c0 2023-10-16T12:50:23.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb9383640 1 -- 172.21.15.60:0/256265744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40661c0 msgr2=0x7fbdb406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb9383640 1 --2- 172.21.15.60:0/256265744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40661c0 0x7fbdb406faf0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0097a0 tx=0x7fbdac02b8d0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb9383640 1 -- 172.21.15.60:0/256265744 shutdown_connections 2023-10-16T12:50:23.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb9383640 1 --2- 172.21.15.60:0/256265744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40661c0 0x7fbdb406faf0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.130+0000 7fbdb9383640 1 -- 172.21.15.60:0/256265744 >> 172.21.15.60:0/256265744 conn(0x7fbdb40d0ab0 msgr2=0x7fbdb40d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:23.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.131+0000 7fbdb9383640 1 -- 172.21.15.60:0/256265744 shutdown_connections 2023-10-16T12:50:23.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.131+0000 7fbdb9383640 1 -- 172.21.15.60:0/256265744 wait complete. 2023-10-16T12:50:23.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.132+0000 7fbdb9383640 1 Processor -- start 2023-10-16T12:50:23.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.133+0000 7fbdb9383640 1 -- start start 2023-10-16T12:50:23.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb9383640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb40661c0 0x7fbdb4155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb9383640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 0x7fbdb40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb9383640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb415af80 0x7fbdb40ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb9383640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb415c920 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb9383640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb40c9c30 con 0x7fbdb40661c0 2023-10-16T12:50:23.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb9383640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb415c360 con 0x7fbdb415af80 2023-10-16T12:50:23.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 0x7fbdb40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 0x7fbdb40ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53308/0 (socket says 172.21.15.60:53308) 2023-10-16T12:50:23.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 -- 172.21.15.60:0/514968128 learned_addr learned my addr 172.21.15.60:0/514968128 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:23.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb2ffd640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb40661c0 0x7fbdb4155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb37fe640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb415af80 0x7fbdb40ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 -- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb415af80 msgr2=0x7fbdb40ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb415af80 0x7fbdb40ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 -- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb40661c0 msgr2=0x7fbdb4155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb40661c0 0x7fbdb4155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.134+0000 7fbdb27fc640 1 -- 172.21.15.60:0/514968128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac009400 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.135+0000 7fbdb27fc640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 0x7fbdb40ac890 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c400 tx=0x7fbd9c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:23.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.135+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0052d0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.135+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c005430 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.135+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c003830 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.135+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb415c640 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.135+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb40acdd0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.136+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb40c9de0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.137+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fbd9c003990 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.138+0000 7fbdab7fe640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 0x7fbd9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.138+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbd9c088ef0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.139+0000 7fbdb2ffd640 1 -- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 msgr2=0x7fbd9406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:23.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.139+0000 7fbdb2ffd640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 0x7fbd9406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:23.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.146+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbdb40c9de0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.163 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.161+0000 7f8c60e2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:50:23.202 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.200+0000 7feb16812040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:50:23.214 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.212+0000 7f8c60e2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:50:23.258 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.256+0000 7feb16812040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:50:23.264 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.261+0000 7f8c60e2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:50:23.312 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.310+0000 7feb16812040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:50:23.324 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.322+0000 7f8c60e2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:50:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.340+0000 7fbdb2ffd640 1 -- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 msgr2=0x7fbd9406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:23.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.340+0000 7fbdb2ffd640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 0x7fbd9406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:23.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.342+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbdb40cbcd0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.343+0000 7fbdab7fe640 1 -- 172.21.15.60:0/514968128 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd9c0170a0 con 0x7fbdb40cb1c0 2023-10-16T12:50:23.346 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:50:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 msgr2=0x7fbd9406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:23.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 0x7fbd9406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 msgr2=0x7fbdb40ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 0x7fbdb40ac890 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c400 tx=0x7fbd9c00c8c0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 shutdown_connections 2023-10-16T12:50:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fbd9406af90 0x7fbd9406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbdb415af80 0x7fbdb40ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbdb40cb1c0 0x7fbdb40ac890 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 --2- 172.21.15.60:0/514968128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdb40661c0 0x7fbdb4155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 >> 172.21.15.60:0/514968128 conn(0x7fbdb40d0ab0 msgr2=0x7fbdb40b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 shutdown_connections 2023-10-16T12:50:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.346+0000 7fbdb9383640 1 -- 172.21.15.60:0/514968128 wait complete. 2023-10-16T12:50:23.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:50:23.373 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.370+0000 7f8c60e2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:50:23.380 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.377+0000 7feb16812040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:50:23.433 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.431+0000 7feb16812040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:50:23.483 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.480+0000 7f8c60e2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:50:23.545 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.542+0000 7f8c60e2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:23.554 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.552+0000 7feb16812040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:50:23.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.581+0000 7fd568b90640 1 Processor -- start 2023-10-16T12:50:23.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.581+0000 7fd568b90640 1 -- start start 2023-10-16T12:50:23.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd568b90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd568b90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd564069fa0 0x7fd56406a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd568b90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5640ac890 con 0x7fd564160530 2023-10-16T12:50:23.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd568b90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5640ae800 con 0x7fd564069fa0 2023-10-16T12:50:23.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd568b90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640ae1f0 con 0x7fd564065ab0 2023-10-16T12:50:23.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd56259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd564160530 0x7fd5641609f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49934/0 (socket says 172.21.15.60:49934) 2023-10-16T12:50:23.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.583+0000 7fd56259c640 1 -- 172.21.15.60:0/3003969800 learned_addr learned my addr 172.21.15.60:0/3003969800 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:23.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd562d9d640 1 --2- 172.21.15.60:0/3003969800 >> v2:172.21.15.60:3300/0 conn(0x7fd564069fa0 0x7fd56406a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd559d9b640 1 --2- 172.21.15.60:0/3003969800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd561d9b640 1 -- 172.21.15.60:0/3003969800 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436352584 0 0) 0x7fd5640ac890 con 0x7fd564160530 2023-10-16T12:50:23.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd561d9b640 1 -- 172.21.15.60:0/3003969800 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd550003520 con 0x7fd564160530 2023-10-16T12:50:23.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd562d9d640 1 -- 172.21.15.60:0/3003969800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 msgr2=0x7fd564161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:23.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd562d9d640 1 --2- 172.21.15.60:0/3003969800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564161100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd562d9d640 1 -- 172.21.15.60:0/3003969800 >> v1:172.21.15.60:6789/0 conn(0x7fd564160530 legacy=0x7fd5641609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.584+0000 7fd562d9d640 1 -- 172.21.15.60:0/3003969800 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640acae0 con 0x7fd564069fa0 2023-10-16T12:50:23.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.585+0000 7fd562d9d640 1 --2- 172.21.15.60:0/3003969800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564069fa0 0x7fd56406a400 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009d90 tx=0x7fd55c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb383d74e5b7700d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:23.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.585+0000 7fd561d9b640 1 -- 172.21.15.60:0/3003969800 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c02cbd0 con 0x7fd564069fa0 2023-10-16T12:50:23.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.585+0000 7fd561d9b640 1 -- 172.21.15.60:0/3003969800 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c02cd30 con 0x7fd564069fa0 2023-10-16T12:50:23.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.585+0000 7fd561d9b640 1 -- 172.21.15.60:0/3003969800 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c02cea0 con 0x7fd564069fa0 2023-10-16T12:50:23.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 -- 172.21.15.60:0/3003969800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564069fa0 msgr2=0x7fd56406a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 --2- 172.21.15.60:0/3003969800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564069fa0 0x7fd56406a400 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009d90 tx=0x7fd55c02be20 comp rx=0 tx=0).stop 2023-10-16T12:50:23.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 -- 172.21.15.60:0/3003969800 shutdown_connections 2023-10-16T12:50:23.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 --2- 172.21.15.60:0/3003969800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564069fa0 0x7fd56406a400 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 --2- 172.21.15.60:0/3003969800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564161100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 -- 172.21.15.60:0/3003969800 >> 172.21.15.60:0/3003969800 conn(0x7fd56406bc10 msgr2=0x7fd56406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:23.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 -- 172.21.15.60:0/3003969800 shutdown_connections 2023-10-16T12:50:23.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.586+0000 7fd568b90640 1 -- 172.21.15.60:0/3003969800 wait complete. 2023-10-16T12:50:23.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.588+0000 7fd568b90640 1 Processor -- start 2023-10-16T12:50:23.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.588+0000 7fd568b90640 1 -- start start 2023-10-16T12:50:23.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.589+0000 7fd568b90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.589+0000 7fd568b90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd564069fa0 0x7fd5641570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.589+0000 7fd568b90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 0x7fd564157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.589+0000 7fd568b90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640b2ab0 con 0x7fd564160530 2023-10-16T12:50:23.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.589+0000 7fd568b90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640b2d90 con 0x7fd564069fa0 2023-10-16T12:50:23.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.589+0000 7fd568b90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640b3070 con 0x7fd564065ab0 2023-10-16T12:50:23.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 0x7fd564157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 0x7fd564157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53340/0 (socket says 172.21.15.60:53340) 2023-10-16T12:50:23.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 -- 172.21.15.60:0/2800130285 learned_addr learned my addr 172.21.15.60:0/2800130285 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:23.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd56259c640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 -- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 msgr2=0x7fd564153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd559d9b640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd564069fa0 0x7fd5641570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:23.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 -- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd564069fa0 msgr2=0x7fd5641570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd564069fa0 0x7fd5641570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 -- 172.21.15.60:0/2800130285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c009a70 con 0x7fd564160530 2023-10-16T12:50:23.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.590+0000 7fd562d9d640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 0x7fd564157620 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd55c02c430 tx=0x7fd55c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:23.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.591+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c02cea0 con 0x7fd564160530 2023-10-16T12:50:23.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.591+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c03e630 con 0x7fd564160530 2023-10-16T12:50:23.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.591+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c03e9e0 con 0x7fd564160530 2023-10-16T12:50:23.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.591+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5640b3350 con 0x7fd564160530 2023-10-16T12:50:23.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.591+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5640b4e90 con 0x7fd564160530 2023-10-16T12:50:23.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.592+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5640b3570 con 0x7fd564160530 2023-10-16T12:50:23.600 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.592+0000 7f8c60e2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:50:23.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.594+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd55c03b050 con 0x7fd564160530 2023-10-16T12:50:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.595+0000 7fd55b7fe640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fd54406af90 0x7fd54406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.595+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd55c0abfc0 con 0x7fd564160530 2023-10-16T12:50:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.595+0000 7fd56259c640 1 -- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fd54406af90 msgr2=0x7fd54406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.595+0000 7fd56259c640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fd54406af90 0x7fd54406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:23.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.599+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd55c0defe0 con 0x7fd564160530 2023-10-16T12:50:23.625 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.622+0000 7feb16812040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:23.652 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.649+0000 7f8c60e2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:50:23.677 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.674+0000 7feb16812040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:50:23.701 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:23.699+0000 7f8c60e2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:50:23.737 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.734+0000 7feb16812040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:50:23.789 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:23.787+0000 7feb16812040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:50:23.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.791+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5640b2bf0 con 0x7fd564160530 2023-10-16T12:50:23.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.791+0000 7fd55b7fe640 1 -- 172.21.15.60:0/2800130285 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd55c04e020 con 0x7fd564160530 2023-10-16T12:50:23.794 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:50:23.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fd54406af90 msgr2=0x7fd54406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:23.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fd54406af90 0x7fd54406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 msgr2=0x7fd564157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:23.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 0x7fd564157620 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd55c02c430 tx=0x7fd55c004a40 comp rx=0 tx=0).stop 2023-10-16T12:50:23.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 shutdown_connections 2023-10-16T12:50:23.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fd54406af90 0x7fd54406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd564160530 0x7fd564157620 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd564069fa0 0x7fd5641570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 --2- 172.21.15.60:0/2800130285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd564065ab0 0x7fd564153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:23.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 >> 172.21.15.60:0/2800130285 conn(0x7fd56406bc10 msgr2=0x7fd564071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:23.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 shutdown_connections 2023-10-16T12:50:23.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:23.794+0000 7fd568b90640 1 -- 172.21.15.60:0/2800130285 wait complete. 2023-10-16T12:50:23.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:50:24.077 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.074+0000 7f8c60e2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:50:24.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.081+0000 7fdedc087640 1 Processor -- start 2023-10-16T12:50:24.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.081+0000 7fdedc087640 1 -- start start 2023-10-16T12:50:24.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.083+0000 7fdedc087640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fded40b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.083+0000 7fdedc087640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40d18c0 con 0x7fded40cfaf0 2023-10-16T12:50:24.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.083+0000 7fdedc087640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40d3980 con 0x7fded40b6cd0 2023-10-16T12:50:24.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.083+0000 7fdedc087640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded40d3c60 con 0x7fded40cf710 2023-10-16T12:50:24.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.083+0000 7fded9622640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fded40cfaf0 0x7fded40d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49944/0 (socket says 172.21.15.60:49944) 2023-10-16T12:50:24.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.083+0000 7fded9622640 1 -- 172.21.15.60:0/2806047417 learned_addr learned my addr 172.21.15.60:0/2806047417 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded9e23640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fded40b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797848995 0 0) 0x7fded40d18c0 con 0x7fded40cfaf0 2023-10-16T12:50:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec0003520 con 0x7fded40cfaf0 2023-10-16T12:50:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972659476 0 0) 0x7fdec0003520 con 0x7fded40cfaf0 2023-10-16T12:50:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 msgr2=0x7fded40b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fded40b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 >> v1:172.21.15.144:6789/0 conn(0x7fded40b6cd0 legacy=0x7fded40d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40ce630 con 0x7fded40cfaf0 2023-10-16T12:50:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fdec4001670 con 0x7fded40cfaf0 2023-10-16T12:50:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 >> v1:172.21.15.60:6789/0 conn(0x7fded40cfaf0 legacy=0x7fded40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec00039a0 0x7fdec0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fdec00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.084+0000 7fded8e21640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec0004d00 0x7fdec0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40d3c60 con 0x7fdec00039a0 2023-10-16T12:50:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec000bb10 con 0x7fdec0004d00 2023-10-16T12:50:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec000bcd0 con 0x7fded40cf710 2023-10-16T12:50:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9e23640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec00039a0 0x7fdec0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fdeda624640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec0004d00 0x7fdec0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9622640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fdec00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9e23640 1 -- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 msgr2=0x7fdec00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:24.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9e23640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fdec00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9e23640 1 -- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec0004d00 msgr2=0x7fdec0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9e23640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec0004d00 0x7fdec0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9e23640 1 -- 172.21.15.60:0/2806047417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0003520 con 0x7fdec00039a0 2023-10-16T12:50:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fded9622640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40cf710 0x7fdec00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.085+0000 7fdeda624640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec0004d00 0x7fdec0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.086+0000 7fded9e23640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec00039a0 0x7fdec0003dc0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fded0009ab0 tx=0x7fdec00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4cabac16067699 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.086+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0005990 con 0x7fdec00039a0 2023-10-16T12:50:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded0005af0 con 0x7fdec00039a0 2023-10-16T12:50:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fded8e21640 1 -- 172.21.15.60:0/2806047417 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0005ea0 con 0x7fdec00039a0 2023-10-16T12:50:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdedc087640 1 -- 172.21.15.60:0/2806047417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec00039a0 msgr2=0x7fdec0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdedc087640 1 --2- 172.21.15.60:0/2806047417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec00039a0 0x7fdec0003dc0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fded0009ab0 tx=0x7fdec00057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdeda624640 1 -- 172.21.15.60:0/2806047417 reap_dead start 2023-10-16T12:50:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdedc087640 1 -- 172.21.15.60:0/2806047417 shutdown_connections 2023-10-16T12:50:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdedc087640 1 -- 172.21.15.60:0/2806047417 >> 172.21.15.60:0/2806047417 conn(0x7fded40d2cc0 msgr2=0x7fded40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdedc087640 1 -- 172.21.15.60:0/2806047417 shutdown_connections 2023-10-16T12:50:24.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.087+0000 7fdedc087640 1 -- 172.21.15.60:0/2806047417 wait complete. 2023-10-16T12:50:24.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.089+0000 7fdedc087640 1 Processor -- start 2023-10-16T12:50:24.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.089+0000 7fdedc087640 1 -- start start 2023-10-16T12:50:24.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdedc087640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec00039a0 0x7fded40b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdedc087640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40b1810 0x7fded40c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdedc087640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 0x7fded40cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdedc087640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded41469c0 con 0x7fded40c6d30 2023-10-16T12:50:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdedc087640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40c71b0 con 0x7fdec00039a0 2023-10-16T12:50:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdedc087640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded40b1db0 con 0x7fded40b1810 2023-10-16T12:50:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdeda624640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 0x7fded40cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdeda624640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 0x7fded40cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53356/0 (socket says 172.21.15.60:53356) 2023-10-16T12:50:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.090+0000 7fdeda624640 1 -- 172.21.15.60:0/2027013862 learned_addr learned my addr 172.21.15.60:0/2027013862 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fded9e23640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec00039a0 0x7fded40b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fded9622640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40b1810 0x7fded40c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeda624640 1 -- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40b1810 msgr2=0x7fded40c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeda624640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40b1810 0x7fded40c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeda624640 1 -- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec00039a0 msgr2=0x7fded40b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeda624640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec00039a0 0x7fded40b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeda624640 1 -- 172.21.15.60:0/2027013862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40b2090 con 0x7fded40c6d30 2023-10-16T12:50:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fded9e23640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec00039a0 0x7fded40b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeda624640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 0x7fded40cc5a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fded408ba60 tx=0x7fdecc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc006d50 con 0x7fded40c6d30 2023-10-16T12:50:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc003970 con 0x7fded40c6d30 2023-10-16T12:50:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc003d60 con 0x7fded40c6d30 2023-10-16T12:50:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.091+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded4146bd0 con 0x7fded40c6d30 2023-10-16T12:50:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.092+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded41496b0 con 0x7fded40c6d30 2023-10-16T12:50:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.093+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded404e160 con 0x7fded40c6d30 2023-10-16T12:50:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.094+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fdecc005070 con 0x7fded40c6d30 2023-10-16T12:50:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.095+0000 7fdeca7fc640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 0x7fdeb806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.095+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fdecc089870 con 0x7fded40c6d30 2023-10-16T12:50:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.095+0000 7fded9e23640 1 -- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 msgr2=0x7fdeb806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.095+0000 7fded9e23640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 0x7fdeb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.099+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdecc054470 con 0x7fded40c6d30 2023-10-16T12:50:24.199 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.197+0000 7feb16812040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:50:24.238 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.236+0000 7f8c60e2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:50:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.295+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fded40d0f90 con 0x7fded40c6d30 2023-10-16T12:50:24.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.296+0000 7fded9e23640 1 -- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 msgr2=0x7fdeb806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:24.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.296+0000 7fded9e23640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 0x7fdeb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:24.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.296+0000 7fdeca7fc640 1 -- 172.21.15.60:0/2027013862 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdecc01ce80 con 0x7fded40c6d30 2023-10-16T12:50:24.299 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:50:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.298+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 msgr2=0x7fdeb806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.298+0000 7fdedc087640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 0x7fdeb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.298+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 msgr2=0x7fded40cc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.298+0000 7fdedc087640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 0x7fded40cc5a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fded408ba60 tx=0x7fdecc00dc00 comp rx=0 tx=0).stop 2023-10-16T12:50:24.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 shutdown_connections 2023-10-16T12:50:24.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fdeb806af90 0x7fdeb806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fded40b1810 0x7fded40c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded40c6d30 0x7fded40cc5a0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 --2- 172.21.15.60:0/2027013862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec00039a0 0x7fded40b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 >> 172.21.15.60:0/2027013862 conn(0x7fded40d2cc0 msgr2=0x7fded40d2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 shutdown_connections 2023-10-16T12:50:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.299+0000 7fdedc087640 1 -- 172.21.15.60:0/2027013862 wait complete. 2023-10-16T12:50:24.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:50:24.345 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.342+0000 7f8c60e2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:50:24.369 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.367+0000 7feb16812040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:50:24.394 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.391+0000 7f8c60e2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:50:24.484 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.482+0000 7feb16812040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:50:24.500 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.497+0000 7f8c60e2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:24.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.524+0000 7fc7aca15640 1 Processor -- start 2023-10-16T12:50:24.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.525+0000 7fc7aca15640 1 -- start start 2023-10-16T12:50:24.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.526+0000 7fc7aca15640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7a80661c0 0x7fc7a806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.526+0000 7fc7aca15640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a815eff0 con 0x7fc7a815af80 2023-10-16T12:50:24.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.526+0000 7fc7aca15640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a815ecc0 con 0x7fc7a80cb1c0 2023-10-16T12:50:24.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.526+0000 7fc7aca15640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a80d0930 con 0x7fc7a80661c0 2023-10-16T12:50:24.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc7a815af80 0x7fc7a815f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49954/0 (socket says 172.21.15.60:49954) 2023-10-16T12:50:24.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a659c640 1 -- 172.21.15.60:0/4110328771 learned_addr learned my addr 172.21.15.60:0/4110328771 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:24.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a6d9d640 1 --2- 172.21.15.60:0/4110328771 >> v2:172.21.15.60:3300/0 conn(0x7fc7a80661c0 0x7fc7a806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a6d9d640 1 -- 172.21.15.60:0/4110328771 >> v1:172.21.15.144:6789/0 conn(0x7fc7a80cb1c0 legacy=0x7fc7a8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:24.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a6d9d640 1 -- 172.21.15.60:0/4110328771 >> v1:172.21.15.60:6789/0 conn(0x7fc7a815af80 legacy=0x7fc7a815f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a559a640 1 -- 172.21.15.60:0/4110328771 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076838246 0 0) 0x7fc7a815eff0 con 0x7fc7a815af80 2023-10-16T12:50:24.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.527+0000 7fc7a6d9d640 1 -- 172.21.15.60:0/4110328771 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a815b360 con 0x7fc7a80661c0 2023-10-16T12:50:24.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.528+0000 7fc7a6d9d640 1 --2- 172.21.15.60:0/4110328771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a806faf0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c18fd4b88fd3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:24.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7a559a640 1 -- 172.21.15.60:0/4110328771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a002e070 con 0x7fc7a80661c0 2023-10-16T12:50:24.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7a559a640 1 -- 172.21.15.60:0/4110328771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a00315a0 con 0x7fc7a80661c0 2023-10-16T12:50:24.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7a559a640 1 -- 172.21.15.60:0/4110328771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0031930 con 0x7fc7a80661c0 2023-10-16T12:50:24.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 -- 172.21.15.60:0/4110328771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 msgr2=0x7fc7a806faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 --2- 172.21.15.60:0/4110328771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a806faf0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a002b840 comp rx=0 tx=0).stop 2023-10-16T12:50:24.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 -- 172.21.15.60:0/4110328771 shutdown_connections 2023-10-16T12:50:24.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 --2- 172.21.15.60:0/4110328771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a806faf0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 -- 172.21.15.60:0/4110328771 >> 172.21.15.60:0/4110328771 conn(0x7fc7a80d0ab0 msgr2=0x7fc7a80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:24.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 -- 172.21.15.60:0/4110328771 shutdown_connections 2023-10-16T12:50:24.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.529+0000 7fc7aca15640 1 -- 172.21.15.60:0/4110328771 wait complete. 2023-10-16T12:50:24.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.531+0000 7fc7aca15640 1 Processor -- start 2023-10-16T12:50:24.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.531+0000 7fc7aca15640 1 -- start start 2023-10-16T12:50:24.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.532+0000 7fc7aca15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a80b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.532+0000 7fc7aca15640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a80cb1c0 0x7fc7a815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.532+0000 7fc7aca15640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a815af80 0x7fc7a8154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.532+0000 7fc7aca15640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80c8a30 con 0x7fc7a80661c0 2023-10-16T12:50:24.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.532+0000 7fc7aca15640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a815e300 con 0x7fc7a815af80 2023-10-16T12:50:24.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.532+0000 7fc7aca15640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80c7af0 con 0x7fc7a80cb1c0 2023-10-16T12:50:24.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a80b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a80b5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53384/0 (socket says 172.21.15.60:53384) 2023-10-16T12:50:24.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 -- 172.21.15.60:0/2157985094 learned_addr learned my addr 172.21.15.60:0/2157985094 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:24.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a5d9b640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a80cb1c0 0x7fc7a815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a6d9d640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a815af80 0x7fc7a8154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:24.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 -- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a80cb1c0 msgr2=0x7fc7a815ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a80cb1c0 0x7fc7a815ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 -- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a815af80 msgr2=0x7fc7a8154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a815af80 0x7fc7a8154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 -- 172.21.15.60:0/2157985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0009400 con 0x7fc7a80661c0 2023-10-16T12:50:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.533+0000 7fc7a659c640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a80b5fb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc79800c890 tx=0x7fc79800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.534+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79800a640 con 0x7fc7a80661c0 2023-10-16T12:50:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.534+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79800a7c0 con 0x7fc7a80661c0 2023-10-16T12:50:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.534+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79800ab70 con 0x7fc7a80661c0 2023-10-16T12:50:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.534+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a80c7db0 con 0x7fc7a80661c0 2023-10-16T12:50:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.534+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a80c8370 con 0x7fc7a80661c0 2023-10-16T12:50:24.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.535+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a804e160 con 0x7fc7a80661c0 2023-10-16T12:50:24.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.536+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc79800acd0 con 0x7fc7a80661c0 2023-10-16T12:50:24.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.537+0000 7fc79effd640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 0x7fc78806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.537+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc798088f20 con 0x7fc7a80661c0 2023-10-16T12:50:24.544 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.537+0000 7feb16812040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:50:24.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.537+0000 7fc7a5d9b640 1 -- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 msgr2=0x7fc78806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:24.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.537+0000 7fc7a5d9b640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 0x7fc78806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:24.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.541+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc798053cc0 con 0x7fc7a80661c0 2023-10-16T12:50:24.608 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.605+0000 7f8c60e2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:50:24.654 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.652+0000 7feb16812040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.738+0000 7fc7a5d9b640 1 -- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 msgr2=0x7fc78806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.738+0000 7fc7a5d9b640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 0x7fc78806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:24.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.749+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7a8156510 con 0x7fc7a80661c0 2023-10-16T12:50:24.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.750+0000 7fc79effd640 1 -- 172.21.15.60:0/2157985094 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v100) v1 ==== 80+0+478 (secure 0 0 0) 0x7fc798018020 con 0x7fc7a80661c0 2023-10-16T12:50:24.752 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1468249085 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.753 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2269656792 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.753 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/1915751278 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.753 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3435889746 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.753 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2501905996 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.754 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1930765872 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.754 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/1915751278 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.754 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3359683879 2023-10-17T12:50:20.071781+0000 2023-10-16T12:50:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T12:50:24.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.752+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 msgr2=0x7fc78806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.752+0000 7fc7aca15640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 0x7fc78806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.752+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 msgr2=0x7fc7a80b5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.752+0000 7fc7aca15640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a80b5fb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc79800c890 tx=0x7fc79800cd50 comp rx=0 tx=0).stop 2023-10-16T12:50:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 shutdown_connections 2023-10-16T12:50:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fc78806af90 0x7fc78806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a80cb1c0 0x7fc7a815ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a815af80 0x7fc7a8154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 --2- 172.21.15.60:0/2157985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a80661c0 0x7fc7a80b5fb0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 >> 172.21.15.60:0/2157985094 conn(0x7fc7a80d0ab0 msgr2=0x7fc7a815a220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 shutdown_connections 2023-10-16T12:50:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:24.753+0000 7fc7aca15640 1 -- 172.21.15.60:0/2157985094 wait complete. 2023-10-16T12:50:24.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.760+0000 7f8c60e2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:24.768 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.766+0000 7feb16812040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:50:24.772 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:50:24.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:24.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:24.812 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.810+0000 7f8c60e2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:24.869 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:50:24.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:24.870 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:50:24.915 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:50:24.915 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:24.932 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:24.929+0000 7f8c60e2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:50:24.940 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.937+0000 7feb16812040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:50:24.969 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:50:24.970 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:50:24.970 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:24.992 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:24.990+0000 7feb16812040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:50:25.026 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:50:25.027 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:50:25.028 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:25.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:50:25.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:25.119 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:25.117+0000 7feb16812040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:50:25.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.292+0000 7f81f8d1d640 1 Processor -- start 2023-10-16T12:50:25.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.292+0000 7f81f8d1d640 1 -- start start 2023-10-16T12:50:25.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f8d1d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f81f415aaa0 0x7f81f415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f8d1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f40ce390 0x7f81f4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f8d1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f415ed60 con 0x7f81f415ae80 2023-10-16T12:50:25.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f8d1d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f406b810 con 0x7f81f415aaa0 2023-10-16T12:50:25.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f8d1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40707e0 con 0x7f81f40ce390 2023-10-16T12:50:25.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f81f415aaa0 0x7f81f415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f1d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f81f415ae80 0x7f81f40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49966/0 (socket says 172.21.15.60:49966) 2023-10-16T12:50:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f81f415aaa0 0x7f81f415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53394/0 (socket says 172.21.15.60:53394) 2023-10-16T12:50:25.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.294+0000 7f81f1d9b640 1 -- 172.21.15.60:0/2680425516 learned_addr learned my addr 172.21.15.60:0/2680425516 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:25.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.295+0000 7f81f2d9d640 1 --2- 172.21.15.60:0/2680425516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f40ce390 0x7f81f4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.295+0000 7f81f259c640 1 -- 172.21.15.60:0/2680425516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f40ce390 msgr2=0x7f81f4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:25.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.295+0000 7f81f259c640 1 --2- 172.21.15.60:0/2680425516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f40ce390 0x7f81f4070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.295+0000 7f81f259c640 1 -- 172.21.15.60:0/2680425516 >> v1:172.21.15.60:6789/0 conn(0x7f81f415ae80 legacy=0x7f81f40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.295+0000 7f81f259c640 1 -- 172.21.15.60:0/2680425516 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f415ef90 con 0x7f81f415aaa0 2023-10-16T12:50:25.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.296+0000 7f81f259c640 1 --2- 172.21.15.60:0/2680425516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f415aaa0 0x7f81f415e820 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f81e40097a0 tx=0x7f81e402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6d1797aa9d0fc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:25.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.296+0000 7f81f159a640 1 -- 172.21.15.60:0/2680425516 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e402d040 con 0x7f81f415aaa0 2023-10-16T12:50:25.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.296+0000 7f81f159a640 1 -- 172.21.15.60:0/2680425516 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e4031810 con 0x7f81f415aaa0 2023-10-16T12:50:25.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.296+0000 7f81f159a640 1 -- 172.21.15.60:0/2680425516 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4031ba0 con 0x7f81f415aaa0 2023-10-16T12:50:25.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 -- 172.21.15.60:0/2680425516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f415aaa0 msgr2=0x7f81f415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/2680425516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f415aaa0 0x7f81f415e820 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f81e40097a0 tx=0x7f81e402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 -- 172.21.15.60:0/2680425516 shutdown_connections 2023-10-16T12:50:25.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/2680425516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f40ce390 0x7f81f4070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/2680425516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f415aaa0 0x7f81f415e820 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 -- 172.21.15.60:0/2680425516 >> 172.21.15.60:0/2680425516 conn(0x7f81f40cb410 msgr2=0x7f81f40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:25.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 -- 172.21.15.60:0/2680425516 shutdown_connections 2023-10-16T12:50:25.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.297+0000 7f81f8d1d640 1 -- 172.21.15.60:0/2680425516 wait complete. 2023-10-16T12:50:25.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.299+0000 7f81f8d1d640 1 Processor -- start 2023-10-16T12:50:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.299+0000 7f81f8d1d640 1 -- start start 2023-10-16T12:50:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f8d1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 0x7f81f41579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f8d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f415aaa0 0x7f81f415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f8d1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f415ae80 0x7f81f40c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f8d1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f415b320 con 0x7f81f40ce390 2023-10-16T12:50:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f8d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f415b570 con 0x7f81f415aaa0 2023-10-16T12:50:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f8d1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f415b850 con 0x7f81f415ae80 2023-10-16T12:50:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 0x7f81f41579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 0x7f81f41579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53402/0 (socket says 172.21.15.60:53402) 2023-10-16T12:50:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.300+0000 7f81f259c640 1 -- 172.21.15.60:0/1695796686 learned_addr learned my addr 172.21.15.60:0/1695796686 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f1d9b640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f415aaa0 0x7f81f415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f2d9d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f415ae80 0x7f81f40c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f259c640 1 -- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f415ae80 msgr2=0x7f81f40c4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f259c640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f415ae80 0x7f81f40c4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f259c640 1 -- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f415aaa0 msgr2=0x7f81f415c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f259c640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f415aaa0 0x7f81f415c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f259c640 1 -- 172.21.15.60:0/1695796686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4009400 con 0x7f81f40ce390 2023-10-16T12:50:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f259c640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 0x7f81f41579e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f81e40098c0 tx=0x7f81e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e403ad60 con 0x7f81f40ce390 2023-10-16T12:50:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e4044400 con 0x7f81f40ce390 2023-10-16T12:50:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e40447b0 con 0x7f81f40ce390 2023-10-16T12:50:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.301+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f415bb30 con 0x7f81f40ce390 2023-10-16T12:50:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.302+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f415c0f0 con 0x7f81f40ce390 2023-10-16T12:50:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.303+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f404e160 con 0x7f81f40ce390 2023-10-16T12:50:25.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.304+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f81e404d440 con 0x7f81f40ce390 2023-10-16T12:50:25.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.304+0000 7f81e2ffd640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 0x7f81d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.305+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f81e40abd30 con 0x7f81f40ce390 2023-10-16T12:50:25.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.305+0000 7f81f1d9b640 1 -- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 msgr2=0x7f81d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:25.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.305+0000 7f81f1d9b640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 0x7f81d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:25.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.309+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81e4076af0 con 0x7f81f40ce390 2023-10-16T12:50:25.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.506+0000 7f81f1d9b640 1 -- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 msgr2=0x7f81d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:25.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.506+0000 7f81f1d9b640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 0x7f81d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.542+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81f40cdc60 con 0x7f81f40ce390 2023-10-16T12:50:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.543+0000 7f81e2ffd640 1 -- 172.21.15.60:0/1695796686 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f81e4035090 con 0x7f81f40ce390 2023-10-16T12:50:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:25.546 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:05.354835+0000","modified":"2023-10-16T12:50:08.431520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/4244730882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4244730882},{"type":"v1","addr":"172.21.15.60:6835","nonce":4244730882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:50:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2023-10-16T12:50:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 msgr2=0x7f81d406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 0x7f81d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 msgr2=0x7f81f41579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 0x7f81f41579e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f81e40098c0 tx=0x7f81e40046c0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 shutdown_connections 2023-10-16T12:50:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7f81d406af90 0x7f81d406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81f415ae80 0x7f81f40c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f415aaa0 0x7f81f415c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 --2- 172.21.15.60:0/1695796686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81f40ce390 0x7f81f41579e0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 >> 172.21.15.60:0/1695796686 conn(0x7f81f40cb410 msgr2=0x7f81f406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 shutdown_connections 2023-10-16T12:50:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.546+0000 7f81f8d1d640 1 -- 172.21.15.60:0/1695796686 wait complete. 2023-10-16T12:50:25.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:25.774 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:25.771+0000 7f8c60e2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:50:25.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.802+0000 7fe8fa367640 1 Processor -- start 2023-10-16T12:50:25.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.802+0000 7fe8fa367640 1 -- start start 2023-10-16T12:50:25.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.804+0000 7fe8fa367640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8f4161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.804+0000 7fe8fa367640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe8f40ad7a0 0x7fe8f4070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.804+0000 7fe8fa367640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f40ae1a0 con 0x7fe8f40adc00 2023-10-16T12:50:25.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.804+0000 7fe8fa367640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f40ae480 con 0x7fe8f40ad7a0 2023-10-16T12:50:25.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.804+0000 7fe8fa367640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40ae760 con 0x7fe8f4160c80 2023-10-16T12:50:25.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.805+0000 7fe8f9b66640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe8f40adc00 0x7fe8f4142400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49972/0 (socket says 172.21.15.60:49972) 2023-10-16T12:50:25.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.805+0000 7fe8f9b66640 1 -- 172.21.15.60:0/3112615328 learned_addr learned my addr 172.21.15.60:0/3112615328 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:25.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.805+0000 7fe8f9365640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8f4161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.805+0000 7fe8f8b64640 1 --2- 172.21.15.60:0/3112615328 >> v2:172.21.15.144:3300/0 conn(0x7fe8f40ad7a0 0x7fe8f4070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.805+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957616850 0 0) 0x7fe8f40ae1a0 con 0x7fe8f40adc00 2023-10-16T12:50:25.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.805+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e0003520 con 0x7fe8f40adc00 2023-10-16T12:50:25.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 379394756 0 0) 0x7fe8e0003520 con 0x7fe8f40adc00 2023-10-16T12:50:25.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 msgr2=0x7fe8f4161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:25.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8f4161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 >> v2:172.21.15.144:3300/0 conn(0x7fe8f40ad7a0 msgr2=0x7fe8f4070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:25.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 --2- 172.21.15.60:0/3112615328 >> v2:172.21.15.144:3300/0 conn(0x7fe8f40ad7a0 0x7fe8f4070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f40aea40 con 0x7fe8f40adc00 2023-10-16T12:50:25.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe8ec0019f0 con 0x7fe8f40adc00 2023-10-16T12:50:25.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 >> v1:172.21.15.60:6789/0 conn(0x7fe8f40adc00 legacy=0x7fe8f4142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8e0003970 0x7fe8e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8e0004d00 0x7fe8e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40ae480 con 0x7fe8e0004d00 2023-10-16T12:50:25.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40ae760 con 0x7fe8e0003970 2023-10-16T12:50:25.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e000b980 con 0x7fe8f4160c80 2023-10-16T12:50:25.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8e0004d00 0x7fe8e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.806+0000 7fe8f9365640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8e0003970 0x7fe8e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f8b64640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f9b66640 1 -- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 msgr2=0x7fe8e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:25.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f9b66640 1 -- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8e0003970 msgr2=0x7fe8e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:25.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8e0003970 0x7fe8e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f9b66640 1 -- 172.21.15.60:0/3112615328 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0003520 con 0x7fe8e0004d00 2023-10-16T12:50:25.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.807+0000 7fe8f8b64640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8f4160c80 0x7fe8e00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:25.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.808+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8e0004d00 0x7fe8e0005180 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00a7b0 tx=0x7fe8e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624c26a361ca4130 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:25.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.808+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec034630 con 0x7fe8e0004d00 2023-10-16T12:50:25.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.808+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec02f070 con 0x7fe8e0004d00 2023-10-16T12:50:25.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.808+0000 7fe8ebfff640 1 -- 172.21.15.60:0/3112615328 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0349f0 con 0x7fe8e0004d00 2023-10-16T12:50:25.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8fa367640 1 -- 172.21.15.60:0/3112615328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8e0004d00 msgr2=0x7fe8e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8fa367640 1 --2- 172.21.15.60:0/3112615328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8e0004d00 0x7fe8e0005180 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00a7b0 tx=0x7fe8e00057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8f9b66640 1 -- 172.21.15.60:0/3112615328 reap_dead start 2023-10-16T12:50:25.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8fa367640 1 -- 172.21.15.60:0/3112615328 shutdown_connections 2023-10-16T12:50:25.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8fa367640 1 -- 172.21.15.60:0/3112615328 >> 172.21.15.60:0/3112615328 conn(0x7fe8f40cc230 msgr2=0x7fe8f40cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:25.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8fa367640 1 -- 172.21.15.60:0/3112615328 shutdown_connections 2023-10-16T12:50:25.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.809+0000 7fe8fa367640 1 -- 172.21.15.60:0/3112615328 wait complete. 2023-10-16T12:50:25.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.812+0000 7fe8fa367640 1 Processor -- start 2023-10-16T12:50:25.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.812+0000 7fe8fa367640 1 -- start start 2023-10-16T12:50:25.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8fa367640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8e0004d00 0x7fe8f4157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8fa367640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8f4157e00 0x7fe8f41e6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8fa367640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 0x7fe8f415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8fa367640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f415bce0 con 0x7fe8f415b2f0 2023-10-16T12:50:25.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8fa367640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4143ce0 con 0x7fe8f4157e00 2023-10-16T12:50:25.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8fa367640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4157b40 con 0x7fe8e0004d00 2023-10-16T12:50:25.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8f9b66640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 0x7fe8f415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8f9b66640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 0x7fe8f415b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53430/0 (socket says 172.21.15.60:53430) 2023-10-16T12:50:25.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8f9b66640 1 -- 172.21.15.60:0/1632555165 learned_addr learned my addr 172.21.15.60:0/1632555165 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:25.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8f9365640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8e0004d00 0x7fe8f4157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.814+0000 7fe8f8b64640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8f4157e00 0x7fe8f41e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:25.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f9b66640 1 -- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8e0004d00 msgr2=0x7fe8f4157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8e0004d00 0x7fe8f4157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f9b66640 1 -- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8f4157e00 msgr2=0x7fe8f41e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:25.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8f4157e00 0x7fe8f41e6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:25.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f9b66640 1 -- 172.21.15.60:0/1632555165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f41e7270 con 0x7fe8f415b2f0 2023-10-16T12:50:25.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f8b64640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8f4157e00 0x7fe8f41e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:25.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8f9b66640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 0x7fe8f415b7a0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00a8d0 tx=0x7fe8ec001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:25.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.815+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec001f20 con 0x7fe8f415b2f0 2023-10-16T12:50:25.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.816+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec034640 con 0x7fe8f415b2f0 2023-10-16T12:50:25.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.816+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f41e7550 con 0x7fe8f415b2f0 2023-10-16T12:50:25.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.816+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0349f0 con 0x7fe8f415b2f0 2023-10-16T12:50:25.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.816+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f41e6ea0 con 0x7fe8f415b2f0 2023-10-16T12:50:25.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.817+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f404e9b0 con 0x7fe8f415b2f0 2023-10-16T12:50:25.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.818+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 40) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe8ec031030 con 0x7fe8f415b2f0 2023-10-16T12:50:25.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.819+0000 7fe8e9ffb640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 0x7fe8d806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:25.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.820+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe8ec02ce40 con 0x7fe8f415b2f0 2023-10-16T12:50:25.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.820+0000 7fe8f9365640 1 -- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 msgr2=0x7fe8d806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:25.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.820+0000 7fe8f9365640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 0x7fe8d806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:50:25.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:25.824+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8ec076b00 con 0x7fe8f415b2f0 2023-10-16T12:50:25.987 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:25.985+0000 7feb16812040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:50:26.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.019+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8f404e9b0 con 0x7fe8f415b2f0 2023-10-16T12:50:26.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.021+0000 7fe8f9365640 1 -- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 msgr2=0x7fe8d806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/176900405 2023-10-16T12:50:26.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.021+0000 7fe8f9365640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 0x7fe8d806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:50:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.022+0000 7fe8e9ffb640 1 -- 172.21.15.60:0/1632555165 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+17066 (secure 0 0 0) 0x7fe8ec07a120 con 0x7fe8f415b2f0 2023-10-16T12:50:26.026 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:26.026 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":100,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:20.071860+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1468249085":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2269656792":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6832/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3435889746":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2501905996":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/1930765872":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6833/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3359683879":"2023-10-17T12:50:20.071781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:26.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.025+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 msgr2=0x7fe8d806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:50:26.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.025+0000 7fe8fa367640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 0x7fe8d806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.025+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 msgr2=0x7fe8f415b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.025+0000 7fe8fa367640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 0x7fe8f415b7a0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00a8d0 tx=0x7fe8ec001cb0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 shutdown_connections 2023-10-16T12:50:26.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:6832/176900405,v1:172.21.15.144:6833/176900405] conn(0x7fe8d806afc0 0x7fe8d806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8f415b2f0 0x7fe8f415b7a0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8f4157e00 0x7fe8f41e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 --2- 172.21.15.60:0/1632555165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8e0004d00 0x7fe8f4157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 >> 172.21.15.60:0/1632555165 conn(0x7fe8f40cc230 msgr2=0x7fe8f415d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:26.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 shutdown_connections 2023-10-16T12:50:26.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.026+0000 7fe8fa367640 1 -- 172.21.15.60:0/1632555165 wait complete. 2023-10-16T12:50:26.046 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:50:03.305764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:50:04.385189+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:50:26.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:50:26.069 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.066+0000 7f8c60e2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:50:26.126 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.123+0000 7f8c60e2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:50:26.240 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.237+0000 7f8c60e2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:50:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.269+0000 7f1f8359e640 1 Processor -- start 2023-10-16T12:50:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.269+0000 7f1f8359e640 1 -- start start 2023-10-16T12:50:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.271+0000 7f1f8359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f840cfc40 0x7f1f840d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.271+0000 7f1f8359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840d0970 con 0x7f1f840b1ca0 2023-10-16T12:50:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.271+0000 7f1f8359e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840d0ea0 con 0x7f1f840b2080 2023-10-16T12:50:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.271+0000 7f1f8359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840d0bc0 con 0x7f1f840cfc40 2023-10-16T12:50:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.271+0000 7f1f8259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1f840b1ca0 0x7f1f840ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49988/0 (socket says 172.21.15.60:49988) 2023-10-16T12:50:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.271+0000 7f1f8259c640 1 -- 172.21.15.60:0/3866533913 learned_addr learned my addr 172.21.15.60:0/3866533913 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f82d9d640 1 --2- 172.21.15.60:0/3866533913 >> v2:172.21.15.60:3300/0 conn(0x7f1f840cfc40 0x7f1f840d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f8159a640 1 -- 172.21.15.60:0/3866533913 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276465081 0 0) 0x7f1f840d0970 con 0x7f1f840b1ca0 2023-10-16T12:50:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f8159a640 1 -- 172.21.15.60:0/3866533913 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c003520 con 0x7f1f840b1ca0 2023-10-16T12:50:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f8159a640 1 -- 172.21.15.60:0/3866533913 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604899248 0 0) 0x7f1f840d0ea0 con 0x7f1f840b2080 2023-10-16T12:50:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f82d9d640 1 -- 172.21.15.60:0/3866533913 >> v1:172.21.15.144:6789/0 conn(0x7f1f840b2080 legacy=0x7f1f840b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f82d9d640 1 -- 172.21.15.60:0/3866533913 >> v1:172.21.15.60:6789/0 conn(0x7f1f840b1ca0 legacy=0x7f1f840ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.272+0000 7f1f82d9d640 1 -- 172.21.15.60:0/3866533913 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840cd940 con 0x7f1f840cfc40 2023-10-16T12:50:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.273+0000 7f1f82d9d640 1 --2- 172.21.15.60:0/3866533913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840d36b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009d90 tx=0x7f1f7c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434a5e1501a53cb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8159a640 1 -- 172.21.15.60:0/3866533913 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c02cd80 con 0x7f1f840cfc40 2023-10-16T12:50:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8159a640 1 -- 172.21.15.60:0/3866533913 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c034440 con 0x7f1f840cfc40 2023-10-16T12:50:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8159a640 1 -- 172.21.15.60:0/3866533913 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0347b0 con 0x7f1f840cfc40 2023-10-16T12:50:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8359e640 1 -- 172.21.15.60:0/3866533913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 msgr2=0x7f1f840d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8359e640 1 --2- 172.21.15.60:0/3866533913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840d36b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009d90 tx=0x7f1f7c02bfd0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8359e640 1 -- 172.21.15.60:0/3866533913 shutdown_connections 2023-10-16T12:50:26.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8359e640 1 --2- 172.21.15.60:0/3866533913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840d36b0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.274+0000 7f1f8359e640 1 -- 172.21.15.60:0/3866533913 >> 172.21.15.60:0/3866533913 conn(0x7f1f840d2b90 msgr2=0x7f1f840d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.275+0000 7f1f8359e640 1 -- 172.21.15.60:0/3866533913 shutdown_connections 2023-10-16T12:50:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.275+0000 7f1f8359e640 1 -- 172.21.15.60:0/3866533913 wait complete. 2023-10-16T12:50:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.276+0000 7f1f8359e640 1 Processor -- start 2023-10-16T12:50:26.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.276+0000 7f1f8359e640 1 -- start start 2023-10-16T12:50:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f840b1ca0 0x7f1f841550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f840b2080 0x7f1f840cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840cc340 con 0x7f1f840cfc40 2023-10-16T12:50:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840ccca0 con 0x7f1f840b2080 2023-10-16T12:50:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840cb6c0 con 0x7f1f840b1ca0 2023-10-16T12:50:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f82d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f81d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f840b2080 0x7f1f840cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f8259c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f840b1ca0 0x7f1f841550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f82d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53450/0 (socket says 172.21.15.60:53450) 2023-10-16T12:50:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.278+0000 7f1f82d9d640 1 -- 172.21.15.60:0/1715257535 learned_addr learned my addr 172.21.15.60:0/1715257535 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f81d9b640 1 -- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f840b1ca0 msgr2=0x7f1f841550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f81d9b640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f840b1ca0 0x7f1f841550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f81d9b640 1 -- 172.21.15.60:0/1715257535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 msgr2=0x7f1f840c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f81d9b640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f81d9b640 1 -- 172.21.15.60:0/1715257535 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c009a70 con 0x7f1f840b2080 2023-10-16T12:50:26.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f82d9d640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:50:26.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f81d9b640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f840b2080 0x7f1f840cc760 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c410 tx=0x7f1f7800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:26.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780035f0 con 0x7f1f840b2080 2023-10-16T12:50:26.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f78003750 con 0x7f1f840b2080 2023-10-16T12:50:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f840cb9a0 con 0x7f1f840b2080 2023-10-16T12:50:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78003b40 con 0x7f1f840b2080 2023-10-16T12:50:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.279+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f840cbf60 con 0x7f1f840b2080 2023-10-16T12:50:26.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.281+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f840cc480 con 0x7f1f840b2080 2023-10-16T12:50:26.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.283+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 41) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1f78003ca0 con 0x7f1f840b2080 2023-10-16T12:50:26.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.284+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1f78088900 con 0x7f1f840b2080 2023-10-16T12:50:26.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.289+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f840cc480 con 0x7f1f840b2080 2023-10-16T12:50:26.322 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:26.319+0000 7feb16812040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:50:26.326 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.323+0000 7f8be5a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.326 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.323+0000 7f8be5a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.326 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.323+0000 7f8be5a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.326 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.323+0000 7f8be5a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.327 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.323+0000 7f8be5a23640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.327 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.324+0000 7f8be2a1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.327 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.324+0000 7f8be2a1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.327 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.324+0000 7f8be2a1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.328 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.324+0000 7f8be2a1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.328 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:50:26.324+0000 7f8be2a1d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:50:26.382 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:26.380+0000 7feb16812040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:50:26.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.487+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f840cc480 con 0x7f1f840b2080 2023-10-16T12:50:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.488+0000 7f1f72ffd640 1 -- 172.21.15.60:0/1715257535 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f840cc480 con 0x7f1f840b2080 2023-10-16T12:50:26.490 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:50:26.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f840b2080 msgr2=0x7f1f840cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f840b2080 0x7f1f840cc760 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c410 tx=0x7f1f7800c810 comp rx=0 tx=0).stop 2023-10-16T12:50:26.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 shutdown_connections 2023-10-16T12:50:26.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f840cfc40 0x7f1f840c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f840b2080 0x7f1f840cc760 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 --2- 172.21.15.60:0/1715257535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f840b1ca0 0x7f1f841550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 >> 172.21.15.60:0/1715257535 conn(0x7f1f840d2b90 msgr2=0x7f1f8415ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:26.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 shutdown_connections 2023-10-16T12:50:26.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.491+0000 7f1f8359e640 1 -- 172.21.15.60:0/1715257535 wait complete. 2023-10-16T12:50:26.502 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:50:26.500+0000 7feb16812040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:50:26.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:50:26.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.720+0000 7fb49ad27640 1 Processor -- start 2023-10-16T12:50:26.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.721+0000 7fb49ad27640 1 -- start start 2023-10-16T12:50:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.722+0000 7fb49ad27640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb49406c040 0x7fb49406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.722+0000 7fb49ad27640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4940ad970 0x7fb49415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.722+0000 7fb49ad27640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb49415ebc0 0x7fb49415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.722+0000 7fb49ad27640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49406c960 con 0x7fb49406c040 2023-10-16T12:50:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.722+0000 7fb49ad27640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49415b9d0 con 0x7fb49415ebc0 2023-10-16T12:50:26.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.722+0000 7fb49ad27640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49415bcb0 con 0x7fb4940ad970 2023-10-16T12:50:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb49406c040 0x7fb49406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb49406c040 0x7fb49406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53458/0 (socket says 172.21.15.60:53458) 2023-10-16T12:50:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 -- 172.21.15.60:0/993298121 learned_addr learned my addr 172.21.15.60:0/993298121 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499524640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4940ad970 0x7fb49415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb49a526640 1 --2- 172.21.15.60:0/993298121 >> v2:172.21.15.144:3300/0 conn(0x7fb49415ebc0 0x7fb49415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 -- 172.21.15.60:0/993298121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4940ad970 msgr2=0x7fb49415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4940ad970 0x7fb49415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 -- 172.21.15.60:0/993298121 >> v2:172.21.15.144:3300/0 conn(0x7fb49415ebc0 msgr2=0x7fb49415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 --2- 172.21.15.60:0/993298121 >> v2:172.21.15.144:3300/0 conn(0x7fb49415ebc0 0x7fb49415f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499d25640 1 -- 172.21.15.60:0/993298121 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49415bf90 con 0x7fb49406c040 2023-10-16T12:50:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb499524640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4940ad970 0x7fb49415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.723+0000 7fb49a526640 1 --2- 172.21.15.60:0/993298121 >> v2:172.21.15.144:3300/0 conn(0x7fb49415ebc0 0x7fb49415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.724+0000 7fb499d25640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49406c040 0x7fb49406c420 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb490009870 tx=0x7fb49002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a9b5afc307f0da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:26.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb498d23640 1 -- 172.21.15.60:0/993298121 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490031640 con 0x7fb49406c040 2023-10-16T12:50:26.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb498d23640 1 -- 172.21.15.60:0/993298121 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4900317a0 con 0x7fb49406c040 2023-10-16T12:50:26.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb498d23640 1 -- 172.21.15.60:0/993298121 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490031b30 con 0x7fb49406c040 2023-10-16T12:50:26.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 -- 172.21.15.60:0/993298121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49406c040 msgr2=0x7fb49406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49406c040 0x7fb49406c420 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb490009870 tx=0x7fb49002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 -- 172.21.15.60:0/993298121 shutdown_connections 2023-10-16T12:50:26.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 --2- 172.21.15.60:0/993298121 >> v2:172.21.15.144:3300/0 conn(0x7fb49415ebc0 0x7fb49415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4940ad970 0x7fb49415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 --2- 172.21.15.60:0/993298121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49406c040 0x7fb49406c420 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.725+0000 7fb49ad27640 1 -- 172.21.15.60:0/993298121 >> 172.21.15.60:0/993298121 conn(0x7fb4940ce6a0 msgr2=0x7fb4940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:26.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.726+0000 7fb49ad27640 1 -- 172.21.15.60:0/993298121 shutdown_connections 2023-10-16T12:50:26.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.726+0000 7fb49ad27640 1 -- 172.21.15.60:0/993298121 wait complete. 2023-10-16T12:50:26.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.727+0000 7fb49ad27640 1 Processor -- start 2023-10-16T12:50:26.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.728+0000 7fb49ad27640 1 -- start start 2023-10-16T12:50:26.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.728+0000 7fb49ad27640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb49406c040 0x7fb4940ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.728+0000 7fb49ad27640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4940ad970 0x7fb494154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49ad27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 0x7fb49415c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:26.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49ad27640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940c7570 con 0x7fb49415ebc0 2023-10-16T12:50:26.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49ad27640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940c6960 con 0x7fb4940ad970 2023-10-16T12:50:26.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49ad27640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940c6c40 con 0x7fb49406c040 2023-10-16T12:50:26.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 0x7fb49415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 0x7fb49415c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53472/0 (socket says 172.21.15.60:53472) 2023-10-16T12:50:26.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 -- 172.21.15.60:0/2095728204 learned_addr learned my addr 172.21.15.60:0/2095728204 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb499524640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4940ad970 0x7fb494154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb499d25640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb49406c040 0x7fb4940ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 -- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb49406c040 msgr2=0x7fb4940ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb49406c040 0x7fb4940ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 -- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4940ad970 msgr2=0x7fb494154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4940ad970 0x7fb494154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb49a526640 1 -- 172.21.15.60:0/2095728204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900094d0 con 0x7fb49415ebc0 2023-10-16T12:50:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb499524640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4940ad970 0x7fb494154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.729+0000 7fb499d25640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb49406c040 0x7fb4940ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.730+0000 7fb49a526640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 0x7fb49415c940 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00cef0 tx=0x7fb48c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.730+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c00d840 con 0x7fb49415ebc0 2023-10-16T12:50:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.730+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c00d9c0 con 0x7fb49415ebc0 2023-10-16T12:50:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.730+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c00dd70 con 0x7fb49415ebc0 2023-10-16T12:50:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.730+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4940c6f20 con 0x7fb49415ebc0 2023-10-16T12:50:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.730+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49415e0d0 con 0x7fb49415ebc0 2023-10-16T12:50:26.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.731+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49406b330 con 0x7fb49415ebc0 2023-10-16T12:50:26.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.732+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 41) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb48c01e3b0 con 0x7fb49415ebc0 2023-10-16T12:50:26.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.733+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb48c089db0 con 0x7fb49415ebc0 2023-10-16T12:50:26.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.738+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb48c0550c0 con 0x7fb49415ebc0 2023-10-16T12:50:26.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:26.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:26.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.942+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb49404e9b0 con 0x7fb49415ebc0 2023-10-16T12:50:26.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.943+0000 7fb48a7fc640 1 -- 172.21.15.60:0/2095728204 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb48c01a020 con 0x7fb49415ebc0 2023-10-16T12:50:26.946 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:50:26.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 msgr2=0x7fb49415c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:26.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 0x7fb49415c940 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00cef0 tx=0x7fb48c00a710 comp rx=0 tx=0).stop 2023-10-16T12:50:26.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 shutdown_connections 2023-10-16T12:50:26.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb49415ebc0 0x7fb49415c940 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4940ad970 0x7fb494154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 --2- 172.21.15.60:0/2095728204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb49406c040 0x7fb4940ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:26.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 >> 172.21.15.60:0/2095728204 conn(0x7fb4940ce6a0 msgr2=0x7fb4940b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:26.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 shutdown_connections 2023-10-16T12:50:26.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:26.946+0000 7fb49ad27640 1 -- 172.21.15.60:0/2095728204 wait complete. 2023-10-16T12:50:26.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:50:27.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.246+0000 7fb7daf87640 1 Processor -- start 2023-10-16T12:50:27.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.247+0000 7fb7daf87640 1 -- start start 2023-10-16T12:50:27.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.248+0000 7fb7daf87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.248+0000 7fb7daf87640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7d40d0be0 0x7fb7d40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.248+0000 7fb7daf87640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d415d150 con 0x7fb7d415cd70 2023-10-16T12:50:27.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.248+0000 7fb7daf87640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d40d1180 con 0x7fb7d40d0be0 2023-10-16T12:50:27.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.248+0000 7fb7daf87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40d1460 con 0x7fb7d40b8480 2023-10-16T12:50:27.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.249+0000 7fb7d9784640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.249+0000 7fb7da786640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb7d40d0be0 0x7fb7d40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.249+0000 7fb7d9f85640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb7d415cd70 0x7fb7d40d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:50648/0 (socket says 172.21.15.60:50648) 2023-10-16T12:50:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.249+0000 7fb7d9f85640 1 -- 172.21.15.60:0/3663126207 learned_addr learned my addr 172.21.15.60:0/3663126207 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.249+0000 7fb7d9784640 1 --2- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:56784/0 (socket says 172.21.15.60:56784) 2023-10-16T12:50:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.250+0000 7fb7da786640 1 -- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 msgr2=0x7fb7d40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.250+0000 7fb7da786640 1 --2- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.250+0000 7fb7da786640 1 -- 172.21.15.60:0/3663126207 >> v1:172.21.15.144:6789/0 conn(0x7fb7d415cd70 legacy=0x7fb7d40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.250+0000 7fb7da786640 1 -- 172.21.15.60:0/3663126207 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d40d1740 con 0x7fb7d40d0be0 2023-10-16T12:50:27.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.251+0000 7fb7da786640 1 --2- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 0x7fb7d40cdd00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009d90 tx=0x7fb7cc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=387b994e132df7c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:27.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.254+0000 7fb7d8f83640 1 -- 172.21.15.60:0/3663126207 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc02cc50 con 0x7fb7d40d0be0 2023-10-16T12:50:27.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.254+0000 7fb7d8f83640 1 -- 172.21.15.60:0/3663126207 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc02cdb0 con 0x7fb7d40d0be0 2023-10-16T12:50:27.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.254+0000 7fb7d8f83640 1 -- 172.21.15.60:0/3663126207 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc034660 con 0x7fb7d40d0be0 2023-10-16T12:50:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 -- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 msgr2=0x7fb7d40cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 --2- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 0x7fb7d40cdd00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009d90 tx=0x7fb7cc02bea0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 -- 172.21.15.60:0/3663126207 shutdown_connections 2023-10-16T12:50:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 --2- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 0x7fb7d40cdd00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 --2- 172.21.15.60:0/3663126207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 -- 172.21.15.60:0/3663126207 >> 172.21.15.60:0/3663126207 conn(0x7fb7d40cf570 msgr2=0x7fb7d40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 -- 172.21.15.60:0/3663126207 shutdown_connections 2023-10-16T12:50:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.255+0000 7fb7daf87640 1 -- 172.21.15.60:0/3663126207 wait complete. 2023-10-16T12:50:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.257+0000 7fb7daf87640 1 Processor -- start 2023-10-16T12:50:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.258+0000 7fb7daf87640 1 -- start start 2023-10-16T12:50:27.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.259+0000 7fb7daf87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.259+0000 7fb7daf87640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 0x7fb7d415d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.259+0000 7fb7daf87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 0x7fb7d415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.259+0000 7fb7daf87640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4151f10 con 0x7fb7d415cd70 2023-10-16T12:50:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.259+0000 7fb7daf87640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40cafb0 con 0x7fb7d40d0be0 2023-10-16T12:50:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.259+0000 7fb7daf87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40ca100 con 0x7fb7d40b8480 2023-10-16T12:50:27.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 0x7fb7d415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 0x7fb7d415d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53480/0 (socket says 172.21.15.60:53480) 2023-10-16T12:50:27.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 -- 172.21.15.60:0/1271455765 learned_addr learned my addr 172.21.15.60:0/1271455765 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:27.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 -- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 msgr2=0x7fb7d40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 -- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 msgr2=0x7fb7d415d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 0x7fb7d415d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.260+0000 7fb7da786640 1 -- 172.21.15.60:0/1271455765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc009a70 con 0x7fb7d415cd70 2023-10-16T12:50:27.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.261+0000 7fb7da786640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 0x7fb7d415d8a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009d90 tx=0x7fb7cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.261+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc040650 con 0x7fb7d415cd70 2023-10-16T12:50:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.262+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d40ca3a0 con 0x7fb7d415cd70 2023-10-16T12:50:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.262+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0407b0 con 0x7fb7d415cd70 2023-10-16T12:50:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.262+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc040b60 con 0x7fb7d415cd70 2023-10-16T12:50:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.262+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d40ca960 con 0x7fb7d415cd70 2023-10-16T12:50:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.263+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d404e9b0 con 0x7fb7d415cd70 2023-10-16T12:50:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.265+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 42) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb7cc040cc0 con 0x7fb7d415cd70 2023-10-16T12:50:27.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.266+0000 7fb7ca7fc640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb7b806acf0 0x7fb7b806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.266+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb7cc0ac2c0 con 0x7fb7d415cd70 2023-10-16T12:50:27.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.270+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb7cc0772b0 con 0x7fb7d415cd70 2023-10-16T12:50:27.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.275+0000 7fb7d9f85640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb7b806acf0 0x7fb7b806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.280+0000 7fb7d9f85640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb7b806acf0 0x7fb7b806d1b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00015f0 tx=0x7fb7d0007a40 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.466+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb7d40cb120 con 0x7fb7d415cd70 2023-10-16T12:50:27.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.467+0000 7fb7ca7fc640 1 -- 172.21.15.60:0/1271455765 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb7cc009450 con 0x7fb7d415cd70 2023-10-16T12:50:27.470 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:50:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.470+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb7b806acf0 msgr2=0x7fb7b806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.470+0000 7fb7daf87640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb7b806acf0 0x7fb7b806d1b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00015f0 tx=0x7fb7d0007a40 comp rx=0 tx=0).stop 2023-10-16T12:50:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.470+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 msgr2=0x7fb7d415d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.470+0000 7fb7daf87640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 0x7fb7d415d8a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009d90 tx=0x7fb7cc004a40 comp rx=0 tx=0).stop 2023-10-16T12:50:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 shutdown_connections 2023-10-16T12:50:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb7b806acf0 0x7fb7b806d1b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7d415cd70 0x7fb7d415d8a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7d40d0be0 0x7fb7d415d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 --2- 172.21.15.60:0/1271455765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb7d40b8480 0x7fb7d40c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 >> 172.21.15.60:0/1271455765 conn(0x7fb7d40cf570 msgr2=0x7fb7d40d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 shutdown_connections 2023-10-16T12:50:27.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.471+0000 7fb7daf87640 1 -- 172.21.15.60:0/1271455765 wait complete. 2023-10-16T12:50:27.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:27.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.707+0000 7fbec30e7640 1 Processor -- start 2023-10-16T12:50:27.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.708+0000 7fbec30e7640 1 -- start start 2023-10-16T12:50:27.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.709+0000 7fbec30e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbebc0b0b20 0x7fbebc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.709+0000 7fbec30e7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc15d3f0 0x7fbebc15d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.709+0000 7fbec30e7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc15e3e0 con 0x7fbebc0cd0c0 2023-10-16T12:50:27.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.709+0000 7fbec30e7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc0cd4a0 con 0x7fbebc0b0b20 2023-10-16T12:50:27.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.709+0000 7fbec30e7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc15e100 con 0x7fbebc15d3f0 2023-10-16T12:50:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbebc0b0b20 0x7fbebc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbebc0b0b20 0x7fbebc1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53486/0 (socket says 172.21.15.60:53486) 2023-10-16T12:50:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 -- 172.21.15.60:0/2925075665 learned_addr learned my addr 172.21.15.60:0/2925075665 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbec1684640 1 --2- 172.21.15.60:0/2925075665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc15d3f0 0x7fbebc15d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 -- 172.21.15.60:0/2925075665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc15d3f0 msgr2=0x7fbebc15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 --2- 172.21.15.60:0/2925075665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc15d3f0 0x7fbebc15d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 -- 172.21.15.60:0/2925075665 >> v1:172.21.15.144:6789/0 conn(0x7fbebc0cd0c0 legacy=0x7fbebc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.710+0000 7fbebbfff640 1 -- 172.21.15.60:0/2925075665 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc160b10 con 0x7fbebc0b0b20 2023-10-16T12:50:27.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.711+0000 7fbebbfff640 1 --2- 172.21.15.60:0/2925075665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc1615f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009870 tx=0x7fbeb002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eac08f946539beb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:27.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbebb7fe640 1 -- 172.21.15.60:0/2925075665 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0031640 con 0x7fbebc0b0b20 2023-10-16T12:50:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbebb7fe640 1 -- 172.21.15.60:0/2925075665 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb00317a0 con 0x7fbebc0b0b20 2023-10-16T12:50:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbebb7fe640 1 -- 172.21.15.60:0/2925075665 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0031b30 con 0x7fbebc0b0b20 2023-10-16T12:50:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 -- 172.21.15.60:0/2925075665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 msgr2=0x7fbebc1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 --2- 172.21.15.60:0/2925075665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc1615f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009870 tx=0x7fbeb002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 -- 172.21.15.60:0/2925075665 shutdown_connections 2023-10-16T12:50:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 --2- 172.21.15.60:0/2925075665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc15d3f0 0x7fbebc15d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 --2- 172.21.15.60:0/2925075665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc1615f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 -- 172.21.15.60:0/2925075665 >> 172.21.15.60:0/2925075665 conn(0x7fbebc0cb410 msgr2=0x7fbebc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:27.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 -- 172.21.15.60:0/2925075665 shutdown_connections 2023-10-16T12:50:27.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.712+0000 7fbec30e7640 1 -- 172.21.15.60:0/2925075665 wait complete. 2023-10-16T12:50:27.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.714+0000 7fbec30e7640 1 Processor -- start 2023-10-16T12:50:27.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.714+0000 7fbec30e7640 1 -- start start 2023-10-16T12:50:27.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec30e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec30e7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc0cd0c0 0x7fbebc0c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec30e7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbebc15d3f0 0x7fbebc0ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec30e7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0c7480 con 0x7fbebc0b0b20 2023-10-16T12:50:27.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec30e7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0c61b0 con 0x7fbebc15d3f0 2023-10-16T12:50:27.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec30e7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0c63e0 con 0x7fbebc0cd0c0 2023-10-16T12:50:27.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.715+0000 7fbec0e83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53498/0 (socket says 172.21.15.60:53498) 2023-10-16T12:50:27.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 -- 172.21.15.60:0/3690883518 learned_addr learned my addr 172.21.15.60:0/3690883518 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:27.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 -- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc0cd0c0 msgr2=0x7fbebc0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc0cd0c0 0x7fbebc0c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 -- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbebc15d3f0 msgr2=0x7fbebc0ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbebc15d3f0 0x7fbebc0ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 -- 172.21.15.60:0/3690883518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00094d0 con 0x7fbebc0b0b20 2023-10-16T12:50:27.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbec0e83640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc154a40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00c760 tx=0x7fbeac00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:27.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.716+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac003b70 con 0x7fbebc0b0b20 2023-10-16T12:50:27.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.717+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac003cf0 con 0x7fbebc0b0b20 2023-10-16T12:50:27.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.717+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc0c66c0 con 0x7fbebc0b0b20 2023-10-16T12:50:27.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.717+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac00aa90 con 0x7fbebc0b0b20 2023-10-16T12:50:27.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.717+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc0c6c80 con 0x7fbebc0b0b20 2023-10-16T12:50:27.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.718+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc04e160 con 0x7fbebc0b0b20 2023-10-16T12:50:27.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.719+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 42) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbeac00abf0 con 0x7fbebc0b0b20 2023-10-16T12:50:27.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.720+0000 7fbeb97fa640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbea806acf0 0x7fbea806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:27.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.721+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbeac089400 con 0x7fbebc0b0b20 2023-10-16T12:50:27.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.721+0000 7fbebbfff640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbea806acf0 0x7fbea806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:27.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.722+0000 7fbebbfff640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbea806acf0 0x7fbea806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0036000 tx=0x7fbeb0030000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:27.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.725+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbeac054470 con 0x7fbebc0b0b20 2023-10-16T12:50:27.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.964+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbebc0cc730 con 0x7fbebc0b0b20 2023-10-16T12:50:27.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.965+0000 7fbeb97fa640 1 -- 172.21.15.60:0/3690883518 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fbeac003e60 con 0x7fbebc0b0b20 2023-10-16T12:50:27.968 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:27.968 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:05.354835+0000","modified":"2023-10-16T12:50:08.431520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/4244730882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4244730882},{"type":"v1","addr":"172.21.15.60:6835","nonce":4244730882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:50:27.968 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2023-10-16T12:50:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbea806acf0 msgr2=0x7fbea806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbea806acf0 0x7fbea806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0036000 tx=0x7fbeb0030000 comp rx=0 tx=0).stop 2023-10-16T12:50:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 msgr2=0x7fbebc154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc154a40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00c760 tx=0x7fbeac00cc20 comp rx=0 tx=0).stop 2023-10-16T12:50:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 shutdown_connections 2023-10-16T12:50:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbea806acf0 0x7fbea806d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbebc0cd0c0 0x7fbebc0c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbebc15d3f0 0x7fbebc0ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 --2- 172.21.15.60:0/3690883518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbebc0b0b20 0x7fbebc154a40 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 >> 172.21.15.60:0/3690883518 conn(0x7fbebc0cb410 msgr2=0x7fbebc06c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 shutdown_connections 2023-10-16T12:50:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:27.968+0000 7fbec30e7640 1 -- 172.21.15.60:0/3690883518 wait complete. 2023-10-16T12:50:27.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.216+0000 7fd92997a640 1 Processor -- start 2023-10-16T12:50:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.216+0000 7fd92997a640 1 -- start start 2023-10-16T12:50:28.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.218+0000 7fd92997a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9240ad970 0x7fd9240cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.218+0000 7fd92997a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92415b9d0 con 0x7fd92406c040 2023-10-16T12:50:28.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.218+0000 7fd92997a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92415bcb0 con 0x7fd9240ccfb0 2023-10-16T12:50:28.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.218+0000 7fd92997a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92415bf90 con 0x7fd9240ad970 2023-10-16T12:50:28.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.218+0000 7fd928978640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd92406c040 0x7fd92406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49996/0 (socket says 172.21.15.60:49996) 2023-10-16T12:50:28.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.218+0000 7fd928978640 1 -- 172.21.15.60:0/1026655858 learned_addr learned my addr 172.21.15.60:0/1026655858 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd923fff640 1 --2- 172.21.15.60:0/1026655858 >> v2:172.21.15.144:3300/0 conn(0x7fd9240ad970 0x7fd9240cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157199583 0 0) 0x7fd92415b9d0 con 0x7fd92406c040 2023-10-16T12:50:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd914003520 con 0x7fd92406c040 2023-10-16T12:50:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2594876613 0 0) 0x7fd914003520 con 0x7fd92406c040 2023-10-16T12:50:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 >> v2:172.21.15.144:3300/0 conn(0x7fd9240ad970 msgr2=0x7fd9240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 --2- 172.21.15.60:0/1026655858 >> v2:172.21.15.144:3300/0 conn(0x7fd9240ad970 0x7fd9240cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 >> v1:172.21.15.144:6789/0 conn(0x7fd9240ccfb0 legacy=0x7fd9240cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240cc270 con 0x7fd92406c040 2023-10-16T12:50:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd918001670 con 0x7fd92406c040 2023-10-16T12:50:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 >> v1:172.21.15.60:6789/0 conn(0x7fd92406c040 legacy=0x7fd92406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd914003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9140047c0 0x7fd914004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd914005180 0x7fd91400fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92415bf90 con 0x7fd914003970 2023-10-16T12:50:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd914010430 con 0x7fd9140047c0 2023-10-16T12:50:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.219+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9140105f0 con 0x7fd914005180 2023-10-16T12:50:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.220+0000 7fd928978640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd914003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.220+0000 7fd923fff640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9140047c0 0x7fd914004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.220+0000 7fd929179640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd914005180 0x7fd91400fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.220+0000 7fd928978640 1 -- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd914005180 msgr2=0x7fd91400fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.221+0000 7fd928978640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd914005180 0x7fd91400fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.221+0000 7fd928978640 1 -- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9140047c0 msgr2=0x7fd914004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.221+0000 7fd928978640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9140047c0 0x7fd914004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.221+0000 7fd928978640 1 -- 172.21.15.60:0/1026655858 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914003520 con 0x7fd914003970 2023-10-16T12:50:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.221+0000 7fd929179640 1 -- 172.21.15.60:0/1026655858 reap_dead start 2023-10-16T12:50:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.222+0000 7fd928978640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd914003d90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c240 tx=0x7fd9140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a615d9feb0e0f69e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.222+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918005ad0 con 0x7fd914003970 2023-10-16T12:50:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.222+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918005c30 con 0x7fd914003970 2023-10-16T12:50:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.222+0000 7fd9237fe640 1 -- 172.21.15.60:0/1026655858 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91802caf0 con 0x7fd914003970 2023-10-16T12:50:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 -- 172.21.15.60:0/1026655858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 msgr2=0x7fd914003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd914003d90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c240 tx=0x7fd9140057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 -- 172.21.15.60:0/1026655858 shutdown_connections 2023-10-16T12:50:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 --2- 172.21.15.60:0/1026655858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd914003d90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 -- 172.21.15.60:0/1026655858 >> 172.21.15.60:0/1026655858 conn(0x7fd9240ce6a0 msgr2=0x7fd9240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 -- 172.21.15.60:0/1026655858 shutdown_connections 2023-10-16T12:50:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.223+0000 7fd92997a640 1 -- 172.21.15.60:0/1026655858 wait complete. 2023-10-16T12:50:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.225+0000 7fd92997a640 1 Processor -- start 2023-10-16T12:50:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.225+0000 7fd92997a640 1 -- start start 2023-10-16T12:50:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.226+0000 7fd92997a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd92415e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.226+0000 7fd92997a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9240a8480 0x7fd92415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.226+0000 7fd92997a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd92415ef30 0x7fd9240b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.226+0000 7fd92997a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92415de40 con 0x7fd914003970 2023-10-16T12:50:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.226+0000 7fd92997a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92415a610 con 0x7fd9240a8480 2023-10-16T12:50:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.226+0000 7fd92997a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240a9460 con 0x7fd92415ef30 2023-10-16T12:50:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd92415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd92415e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53516/0 (socket says 172.21.15.60:53516) 2023-10-16T12:50:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 -- 172.21.15.60:0/2363718753 learned_addr learned my addr 172.21.15.60:0/2363718753 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd923fff640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9240a8480 0x7fd92415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd929179640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd92415ef30 0x7fd9240b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 -- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd92415ef30 msgr2=0x7fd9240b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd92415ef30 0x7fd9240b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 -- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9240a8480 msgr2=0x7fd92415a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9240a8480 0x7fd92415a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 -- 172.21.15.60:0/2363718753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240a8ea0 con 0x7fd914003970 2023-10-16T12:50:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd929179640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd92415ef30 0x7fd9240b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.227+0000 7fd928978640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd92415e260 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c240 tx=0x7fd918001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.228+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91803c040 con 0x7fd914003970 2023-10-16T12:50:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.228+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91802ccc0 con 0x7fd914003970 2023-10-16T12:50:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.228+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918007270 con 0x7fd914003970 2023-10-16T12:50:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.228+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9240a9180 con 0x7fd914003970 2023-10-16T12:50:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.228+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92415f3b0 con 0x7fd914003970 2023-10-16T12:50:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.229+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92415e0b0 con 0x7fd914003970 2023-10-16T12:50:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.230+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 42) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd918032070 con 0x7fd914003970 2023-10-16T12:50:28.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.231+0000 7fd9217fa640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd90806acc0 0x7fd90806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.232+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd9180aa730 con 0x7fd914003970 2023-10-16T12:50:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.232+0000 7fd923fff640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd90806acc0 0x7fd90806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.233+0000 7fd923fff640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd90806acc0 0x7fd90806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd92415b360 tx=0x7fd910008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.236+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9180757a0 con 0x7fd914003970 2023-10-16T12:50:28.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.277+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 43) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd918075b70 con 0x7fd914003970 2023-10-16T12:50:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.442+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9240b1030 con 0x7fd914003970 2023-10-16T12:50:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.444+0000 7fd9217fa640 1 -- 172.21.15.60:0/2363718753 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+17572 (secure 0 0 0) 0x7fd918046090 con 0x7fd914003970 2023-10-16T12:50:28.447 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:28.448 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":101,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:26.244014+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/734835368":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/2831755110":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1942560574":"2023-10-17T12:50:26.243946+0000","172.21.15.144:6832/176900405":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1135964389":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/668026912":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1468249085":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2269656792":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6832/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6833/176900405":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/3435889746":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2501905996":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/1930765872":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2188695119":"2023-10-17T12:50:26.243946+0000","172.21.15.144:6833/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3359683879":"2023-10-17T12:50:20.071781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:28.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd90806acc0 msgr2=0x7fd90806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd90806acc0 0x7fd90806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd92415b360 tx=0x7fd910008040 comp rx=0 tx=0).stop 2023-10-16T12:50:28.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 msgr2=0x7fd92415e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd92415e260 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c240 tx=0x7fd918001a00 comp rx=0 tx=0).stop 2023-10-16T12:50:28.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 shutdown_connections 2023-10-16T12:50:28.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd90806acc0 0x7fd90806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd92415ef30 0x7fd9240b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9240a8480 0x7fd92415a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 --2- 172.21.15.60:0/2363718753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd914003970 0x7fd92415e260 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.447+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 >> 172.21.15.60:0/2363718753 conn(0x7fd9240ce6a0 msgr2=0x7fd9240c6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:28.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.448+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 shutdown_connections 2023-10-16T12:50:28.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.448+0000 7fd92997a640 1 -- 172.21.15.60:0/2363718753 wait complete. 2023-10-16T12:50:28.470 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:50:28.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.693+0000 7f6132547640 1 Processor -- start 2023-10-16T12:50:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.693+0000 7f6132547640 1 -- start start 2023-10-16T12:50:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.695+0000 7f6132547640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f612c15c9f0 0x7f612c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.695+0000 7f6132547640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c160a90 con 0x7f612c15dde0 2023-10-16T12:50:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.695+0000 7f6132547640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0cd900 con 0x7f612c0c9230 2023-10-16T12:50:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.695+0000 7f6132547640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c0cdec0 con 0x7f612c15c9f0 2023-10-16T12:50:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.695+0000 7f6131d46640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f612c15dde0 0x7f612c0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50012/0 (socket says 172.21.15.60:50012) 2023-10-16T12:50:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.695+0000 7f6131d46640 1 -- 172.21.15.60:0/3689040444 learned_addr learned my addr 172.21.15.60:0/3689040444 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6131545640 1 --2- 172.21.15.60:0/3689040444 >> v2:172.21.15.144:3300/0 conn(0x7f612c15c9f0 0x7f612c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982389271 0 0) 0x7f612c160a90 con 0x7f612c15dde0 2023-10-16T12:50:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6118003520 con 0x7f612c15dde0 2023-10-16T12:50:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220782434 0 0) 0x7f6118003520 con 0x7f612c15dde0 2023-10-16T12:50:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 >> v2:172.21.15.144:3300/0 conn(0x7f612c15c9f0 msgr2=0x7f612c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 --2- 172.21.15.60:0/3689040444 >> v2:172.21.15.144:3300/0 conn(0x7f612c15c9f0 0x7f612c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 >> v1:172.21.15.144:6789/0 conn(0x7f612c0c9230 legacy=0x7f612c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0cdbe0 con 0x7f612c15dde0 2023-10-16T12:50:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f61240019f0 con 0x7f612c15dde0 2023-10-16T12:50:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 >> v1:172.21.15.60:6789/0 conn(0x7f612c15dde0 legacy=0x7f612c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6118003970 0x7f6118003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f6118004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6118005180 0x7f611800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611800bff0 con 0x7f61180047c0 2023-10-16T12:50:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611800c160 con 0x7f6118003970 2023-10-16T12:50:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.696+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611800c320 con 0x7f6118005180 2023-10-16T12:50:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6130d44640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f6118004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6131545640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6118003970 0x7f6118003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6130d44640 1 -- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6118005180 msgr2=0x7f611800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6130d44640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6118005180 0x7f611800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6130d44640 1 -- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6118003970 msgr2=0x7f6118003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6130d44640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6118003970 0x7f6118003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6130d44640 1 -- 172.21.15.60:0/3689040444 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6118003520 con 0x7f61180047c0 2023-10-16T12:50:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6131d46640 1 -- 172.21.15.60:0/3689040444 reap_dead start 2023-10-16T12:50:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.697+0000 7f6131545640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6118003970 0x7f6118003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.698+0000 7f6130d44640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f6118004c40 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f611c001750 tx=0x7f61180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0ace140097430e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.698+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c0052d0 con 0x7f61180047c0 2023-10-16T12:50:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.698+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c02e050 con 0x7f61180047c0 2023-10-16T12:50:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.698+0000 7f6123fff640 1 -- 172.21.15.60:0/3689040444 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c003830 con 0x7f61180047c0 2023-10-16T12:50:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 -- 172.21.15.60:0/3689040444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 msgr2=0x7f6118004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f6118004c40 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f611c001750 tx=0x7f61180057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 -- 172.21.15.60:0/3689040444 shutdown_connections 2023-10-16T12:50:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 --2- 172.21.15.60:0/3689040444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f6118004c40 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 -- 172.21.15.60:0/3689040444 >> 172.21.15.60:0/3689040444 conn(0x7f612c0ce6a0 msgr2=0x7f612c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 -- 172.21.15.60:0/3689040444 shutdown_connections 2023-10-16T12:50:28.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.699+0000 7f6132547640 1 -- 172.21.15.60:0/3689040444 wait complete. 2023-10-16T12:50:28.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.701+0000 7f6132547640 1 Processor -- start 2023-10-16T12:50:28.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.701+0000 7f6132547640 1 -- start start 2023-10-16T12:50:28.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.702+0000 7f6132547640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f612c157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.702+0000 7f6132547640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c1583e0 0x7f612c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.702+0000 7f6132547640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f612c1508a0 0x7f612c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.702+0000 7f6132547640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c159500 con 0x7f61180047c0 2023-10-16T12:50:28.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.702+0000 7f6132547640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c158880 con 0x7f612c1583e0 2023-10-16T12:50:28.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.702+0000 7f6132547640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c158b60 con 0x7f612c1508a0 2023-10-16T12:50:28.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f612c157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f612c157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53536/0 (socket says 172.21.15.60:53536) 2023-10-16T12:50:28.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 -- 172.21.15.60:0/2698758043 learned_addr learned my addr 172.21.15.60:0/2698758043 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:28.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6130d44640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c1583e0 0x7f612c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 -- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f612c1508a0 msgr2=0x7f612c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131d46640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f612c1508a0 0x7f612c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f612c1508a0 0x7f612c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 -- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c1583e0 msgr2=0x7f612c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c1583e0 0x7f612c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 -- 172.21.15.60:0/2698758043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c158e40 con 0x7f61180047c0 2023-10-16T12:50:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6130d44640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c1583e0 0x7f612c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131d46640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f612c1508a0 0x7f612c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:28.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.703+0000 7f6131545640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f612c157ea0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f61280107c0 tx=0x7f6128010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.704+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612800eb60 con 0x7f61180047c0 2023-10-16T12:50:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.704+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6128010ed0 con 0x7f61180047c0 2023-10-16T12:50:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.704+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612801c400 con 0x7f61180047c0 2023-10-16T12:50:28.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.704+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c159120 con 0x7f61180047c0 2023-10-16T12:50:28.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.704+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c1510e0 con 0x7f61180047c0 2023-10-16T12:50:28.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.705+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c161670 con 0x7f61180047c0 2023-10-16T12:50:28.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.706+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 43) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f612801c560 con 0x7f61180047c0 2023-10-16T12:50:28.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.707+0000 7f6121ffb640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f611006af30 0x7f611006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:28.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.707+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6128089e90 con 0x7f61180047c0 2023-10-16T12:50:28.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.708+0000 7f6130d44640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f611006af30 0x7f611006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:28.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.709+0000 7f6130d44640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f611006af30 0x7f611006d3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f611c033000 tx=0x7f611c009030 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:28.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.711+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6128054c50 con 0x7f61180047c0 2023-10-16T12:50:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.952+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f612c04e9b0 con 0x7f61180047c0 2023-10-16T12:50:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.953+0000 7f6121ffb640 1 -- 172.21.15.60:0/2698758043 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f6128058270 con 0x7f61180047c0 2023-10-16T12:50:28.956 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:28.957 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:05.354835+0000","modified":"2023-10-16T12:50:08.431520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/4244730882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4244730882},{"type":"v1","addr":"172.21.15.60:6835","nonce":4244730882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:50:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2023-10-16T12:50:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f611006af30 msgr2=0x7f611006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f611006af30 0x7f611006d3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f611c033000 tx=0x7f611c009030 comp rx=0 tx=0).stop 2023-10-16T12:50:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 msgr2=0x7f612c157ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f612c157ea0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f61280107c0 tx=0x7f6128010c80 comp rx=0 tx=0).stop 2023-10-16T12:50:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 shutdown_connections 2023-10-16T12:50:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f611006af30 0x7f611006d3f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f612c1508a0 0x7f612c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f612c1583e0 0x7f612c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 --2- 172.21.15.60:0/2698758043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180047c0 0x7f612c157ea0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 >> 172.21.15.60:0/2698758043 conn(0x7f612c0ce6a0 msgr2=0x7f612c0ca6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 shutdown_connections 2023-10-16T12:50:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:28.957+0000 7f6132547640 1 -- 172.21.15.60:0/2698758043 wait complete. 2023-10-16T12:50:28.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:29.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.204+0000 7f0105e44640 1 Processor -- start 2023-10-16T12:50:29.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.204+0000 7f0105e44640 1 -- start start 2023-10-16T12:50:29.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.206+0000 7f0105e44640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f010015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.206+0000 7f0105e44640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010015ed60 con 0x7f010015aaa0 2023-10-16T12:50:29.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.206+0000 7f0105e44640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01000707e0 con 0x7f01000ce390 2023-10-16T12:50:29.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.206+0000 7f0105e44640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010006b810 con 0x7f010015ae80 2023-10-16T12:50:29.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.206+0000 7f00ff7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f010015aaa0 0x7f01000cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44628/0 (socket says 172.21.15.60:44628) 2023-10-16T12:50:29.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.206+0000 7f00ff7fe640 1 -- 172.21.15.60:0/1660974165 learned_addr learned my addr 172.21.15.60:0/1660974165 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:29.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f010015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750705968 0 0) 0x7f010015ed60 con 0x7f010015aaa0 2023-10-16T12:50:29.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ec003520 con 0x7f010015aaa0 2023-10-16T12:50:29.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 138826622 0 0) 0x7f00ec003520 con 0x7f010015aaa0 2023-10-16T12:50:29.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 msgr2=0x7f010015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:29.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f010015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 >> v1:172.21.15.144:6789/0 conn(0x7f01000ce390 legacy=0x7f0100070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010015ef90 con 0x7f010015aaa0 2023-10-16T12:50:29.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f00f00019f0 con 0x7f010015aaa0 2023-10-16T12:50:29.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f010015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:29.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 >> v1:172.21.15.60:6789/0 conn(0x7f010015aaa0 legacy=0x7f01000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f00ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f00ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ec004d00 0x7f00ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010006b810 con 0x7f00ec003f90 2023-10-16T12:50:29.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec00bae0 con 0x7f00ec004d00 2023-10-16T12:50:29.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.207+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec00bca0 con 0x7f010015ae80 2023-10-16T12:50:29.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f00ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f00ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00fffff640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ec004d00 0x7f00ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00f6ffd640 1 -- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 msgr2=0x7f00ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:29.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f00ec0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00f6ffd640 1 -- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ec004d00 msgr2=0x7f00ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ec004d00 0x7f00ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00f6ffd640 1 -- 172.21.15.60:0/1660974165 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec003520 con 0x7f00ec003f90 2023-10-16T12:50:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ae80 0x7f00ec0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.208+0000 7f00fffff640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ec004d00 0x7f00ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:29.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.209+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f00ec0047c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f00e80098a0 tx=0x7f00ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaae5c480fa347ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:29.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e802d040 con 0x7f00ec003f90 2023-10-16T12:50:29.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e8004230 con 0x7f00ec003f90 2023-10-16T12:50:29.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f00feffd640 1 -- 172.21.15.60:0/1660974165 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8003610 con 0x7f00ec003f90 2023-10-16T12:50:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f0105e44640 1 -- 172.21.15.60:0/1660974165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 msgr2=0x7f00ec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f0105e44640 1 --2- 172.21.15.60:0/1660974165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f00ec0047c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f00e80098a0 tx=0x7f00ec0057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f00fffff640 1 -- 172.21.15.60:0/1660974165 reap_dead start 2023-10-16T12:50:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f0105e44640 1 -- 172.21.15.60:0/1660974165 shutdown_connections 2023-10-16T12:50:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.210+0000 7f0105e44640 1 -- 172.21.15.60:0/1660974165 >> 172.21.15.60:0/1660974165 conn(0x7f01000cb410 msgr2=0x7f01000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:29.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.211+0000 7f0105e44640 1 -- 172.21.15.60:0/1660974165 shutdown_connections 2023-10-16T12:50:29.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.211+0000 7f0105e44640 1 -- 172.21.15.60:0/1660974165 wait complete. 2023-10-16T12:50:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.212+0000 7f0105e44640 1 Processor -- start 2023-10-16T12:50:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.213+0000 7f0105e44640 1 -- start start 2023-10-16T12:50:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f0105e44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f010014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f0105e44640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010014ac70 0x7f010014d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f0105e44640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010014d790 0x7f010014b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f0105e44640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010014dc20 con 0x7f00ec003f90 2023-10-16T12:50:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f0105e44640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010014e0d0 con 0x7f010014ac70 2023-10-16T12:50:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f0105e44640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010014bda0 con 0x7f010014d790 2023-10-16T12:50:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f010014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f010014a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51662/0 (socket says 172.21.15.60:51662) 2023-10-16T12:50:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 -- 172.21.15.60:0/3417163258 learned_addr learned my addr 172.21.15.60:0/3417163258 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010014ac70 0x7f010014d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00fffff640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010014d790 0x7f010014b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 -- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010014d790 msgr2=0x7f010014b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010014d790 0x7f010014b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 -- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010014ac70 msgr2=0x7f010014d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010014ac70 0x7f010014d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.214+0000 7f00ff7fe640 1 -- 172.21.15.60:0/3417163258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010014c080 con 0x7f00ec003f90 2023-10-16T12:50:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.215+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f010014a730 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f00f000dfc0 tx=0x7f00f000b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.215+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0004d60 con 0x7f00ec003f90 2023-10-16T12:50:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.215+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f00070d0 con 0x7f00ec003f90 2023-10-16T12:50:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.215+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0007460 con 0x7f00ec003f90 2023-10-16T12:50:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.215+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010014c360 con 0x7f00ec003f90 2023-10-16T12:50:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.215+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010014c920 con 0x7f00ec003f90 2023-10-16T12:50:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.217+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010004e160 con 0x7f00ec003f90 2023-10-16T12:50:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.218+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 43) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f00f00075c0 con 0x7f00ec003f90 2023-10-16T12:50:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.219+0000 7f00fcff9640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f00e006af30 0x7f00e006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.219+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f00f0089b30 con 0x7f00ec003f90 2023-10-16T12:50:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.219+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f00e006af30 0x7f00e006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.221+0000 7f00f6ffd640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f00e006af30 0x7f00e006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00e8035000 tx=0x7f00e8031000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:29.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.223+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00f00547c0 con 0x7f00ec003f90 2023-10-16T12:50:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.465+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01000cdc60 con 0x7f00ec003f90 2023-10-16T12:50:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.467+0000 7f00fcff9640 1 -- 172.21.15.60:0/3417163258 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f00f0019090 con 0x7f00ec003f90 2023-10-16T12:50:29.470 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:29.470 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:05.354835+0000","modified":"2023-10-16T12:50:08.431520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/4244730882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4244730882},{"type":"v1","addr":"172.21.15.60:6835","nonce":4244730882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:50:29.470 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2023-10-16T12:50:29.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f00e006af30 msgr2=0x7f00e006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f00e006af30 0x7f00e006d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00e8035000 tx=0x7f00e8031000 comp rx=0 tx=0).stop 2023-10-16T12:50:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 msgr2=0x7f010014a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f010014a730 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f00f000dfc0 tx=0x7f00f000b560 comp rx=0 tx=0).stop 2023-10-16T12:50:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 shutdown_connections 2023-10-16T12:50:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f00e006af30 0x7f00e006d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010014d790 0x7f010014b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010014ac70 0x7f010014d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 --2- 172.21.15.60:0/3417163258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ec003f90 0x7f010014a730 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 >> 172.21.15.60:0/3417163258 conn(0x7f01000cb410 msgr2=0x7f01000acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 shutdown_connections 2023-10-16T12:50:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.470+0000 7f0105e44640 1 -- 172.21.15.60:0/3417163258 wait complete. 2023-10-16T12:50:29.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:29.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.719+0000 7f74ddc21640 1 Processor -- start 2023-10-16T12:50:29.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.720+0000 7f74ddc21640 1 -- start start 2023-10-16T12:50:29.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.721+0000 7f74ddc21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 0x7f74d80b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.721+0000 7f74ddc21640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d80d15a0 con 0x7f74d80d2640 2023-10-16T12:50:29.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.721+0000 7f74ddc21640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d80d1880 con 0x7f74d815cbe0 2023-10-16T12:50:29.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.721+0000 7f74ddc21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d80d1b60 con 0x7f74d815cfc0 2023-10-16T12:50:29.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.721+0000 7f74dd420640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f74d80d2640 0x7f74d80b5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44638/0 (socket says 172.21.15.60:44638) 2023-10-16T12:50:29.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.721+0000 7f74dd420640 1 -- 172.21.15.60:0/1831924477 learned_addr learned my addr 172.21.15.60:0/1831924477 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:29.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d7fff640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 0x7f74d80b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169880263 0 0) 0x7f74d80d15a0 con 0x7f74d80d2640 2023-10-16T12:50:29.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c8003520 con 0x7f74d80d2640 2023-10-16T12:50:29.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656505605 0 0) 0x7f74c8003520 con 0x7f74d80d2640 2023-10-16T12:50:29.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 msgr2=0x7f74d80b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:29.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 0x7f74d80b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 >> v1:172.21.15.144:6789/0 conn(0x7f74d815cbe0 legacy=0x7f74d80d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d80d1e40 con 0x7f74d80d2640 2023-10-16T12:50:29.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f74d00019f0 con 0x7f74d80d2640 2023-10-16T12:50:29.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 >> v1:172.21.15.60:6789/0 conn(0x7f74d80d2640 legacy=0x7f74d80b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80039a0 0x7f74c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c80047c0 0x7f74c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 0x7f74c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d80d1b60 con 0x7f74c80047c0 2023-10-16T12:50:29.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c800bac0 con 0x7f74c80039a0 2023-10-16T12:50:29.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.722+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c800bc80 con 0x7f74d815cfc0 2023-10-16T12:50:29.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74d7fff640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c80047c0 0x7f74c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74dcc1f640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80039a0 0x7f74c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74dd420640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 0x7f74c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74d7fff640 1 -- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 msgr2=0x7f74c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74d7fff640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815cfc0 0x7f74c8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74d7fff640 1 -- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80039a0 msgr2=0x7f74c8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74d7fff640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80039a0 0x7f74c8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.723+0000 7f74d7fff640 1 -- 172.21.15.60:0/1831924477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c8003520 con 0x7f74c80047c0 2023-10-16T12:50:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.724+0000 7f74d7fff640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c80047c0 0x7f74c8004c40 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009a90 tx=0x7f74c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7347fd2d2b4ea7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:29.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c402c660 con 0x7f74c80047c0 2023-10-16T12:50:29.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c402c7c0 con 0x7f74c80047c0 2023-10-16T12:50:29.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74d77fe640 1 -- 172.21.15.60:0/1831924477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c402cb50 con 0x7f74c80047c0 2023-10-16T12:50:29.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74ddc21640 1 -- 172.21.15.60:0/1831924477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c80047c0 msgr2=0x7f74c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74ddc21640 1 --2- 172.21.15.60:0/1831924477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c80047c0 0x7f74c8004c40 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009a90 tx=0x7f74c80057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74dd420640 1 -- 172.21.15.60:0/1831924477 reap_dead start 2023-10-16T12:50:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74ddc21640 1 -- 172.21.15.60:0/1831924477 shutdown_connections 2023-10-16T12:50:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74ddc21640 1 -- 172.21.15.60:0/1831924477 >> 172.21.15.60:0/1831924477 conn(0x7f74d80cf570 msgr2=0x7f74d80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74ddc21640 1 -- 172.21.15.60:0/1831924477 shutdown_connections 2023-10-16T12:50:29.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.725+0000 7f74ddc21640 1 -- 172.21.15.60:0/1831924477 wait complete. 2023-10-16T12:50:29.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.727+0000 7f74ddc21640 1 Processor -- start 2023-10-16T12:50:29.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.727+0000 7f74ddc21640 1 -- start start 2023-10-16T12:50:29.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.728+0000 7f74ddc21640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80047c0 0x7f74d81f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.728+0000 7f74ddc21640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 0x7f74d81f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.728+0000 7f74ddc21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815a3f0 0x7f74d815a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.728+0000 7f74ddc21640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d815adb0 con 0x7f74d81f55a0 2023-10-16T12:50:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.728+0000 7f74ddc21640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d81f34f0 con 0x7f74c80047c0 2023-10-16T12:50:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.728+0000 7f74ddc21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d815b270 con 0x7f74d815a3f0 2023-10-16T12:50:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 0x7f74d81f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 0x7f74d81f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51680/0 (socket says 172.21.15.60:51680) 2023-10-16T12:50:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 -- 172.21.15.60:0/2414844322 learned_addr learned my addr 172.21.15.60:0/2414844322 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74dcc1f640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80047c0 0x7f74d81f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74dd420640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815a3f0 0x7f74d815a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 -- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815a3f0 msgr2=0x7f74d815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815a3f0 0x7f74d815a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 -- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80047c0 msgr2=0x7f74d81f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80047c0 0x7f74d81f36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 -- 172.21.15.60:0/2414844322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d815f6e0 con 0x7f74d81f55a0 2023-10-16T12:50:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.729+0000 7f74d7fff640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 0x7f74d81f2fb0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f74c4037040 tx=0x7f74c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.730+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4003810 con 0x7f74d81f55a0 2023-10-16T12:50:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.730+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4003970 con 0x7f74d81f55a0 2023-10-16T12:50:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.730+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4003ce0 con 0x7f74d81f55a0 2023-10-16T12:50:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.730+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d815f9c0 con 0x7f74d81f55a0 2023-10-16T12:50:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.730+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d815ff80 con 0x7f74d81f55a0 2023-10-16T12:50:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.731+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d8160110 con 0x7f74d81f55a0 2023-10-16T12:50:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.732+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 43) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f74c4034440 con 0x7f74d81f55a0 2023-10-16T12:50:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.733+0000 7f74d57fa640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f74bc06af00 0x7f74bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.733+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f74c40ab7f0 con 0x7f74d81f55a0 2023-10-16T12:50:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.734+0000 7f74dcc1f640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f74bc06af00 0x7f74bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:29.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.735+0000 7f74dcc1f640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f74bc06af00 0x7f74bc06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f74cc007fd0 tx=0x7f74cc009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:29.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.737+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74c40767d0 con 0x7f74d81f55a0 2023-10-16T12:50:29.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.940+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74d80d11e0 con 0x7f74d81f55a0 2023-10-16T12:50:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.943+0000 7f74d57fa640 1 -- 172.21.15.60:0/2414844322 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+17572 (secure 0 0 0) 0x7f74c40044e0 con 0x7f74d81f55a0 2023-10-16T12:50:29.946 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:29.947 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":101,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:26.244014+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:03.305764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:04.385189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/734835368":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/2831755110":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1942560574":"2023-10-17T12:50:26.243946+0000","172.21.15.144:6832/176900405":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1135964389":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/668026912":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1468249085":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2269656792":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6832/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6833/176900405":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/3435889746":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2501905996":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/1930765872":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2188695119":"2023-10-17T12:50:26.243946+0000","172.21.15.144:6833/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3359683879":"2023-10-17T12:50:20.071781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.946+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f74bc06af00 msgr2=0x7f74bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.946+0000 7f74ddc21640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f74bc06af00 0x7f74bc06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f74cc007fd0 tx=0x7f74cc009040 comp rx=0 tx=0).stop 2023-10-16T12:50:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.946+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 msgr2=0x7f74d81f2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.946+0000 7f74ddc21640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 0x7f74d81f2fb0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f74c4037040 tx=0x7f74c4004730 comp rx=0 tx=0).stop 2023-10-16T12:50:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 shutdown_connections 2023-10-16T12:50:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f74bc06af00 0x7f74bc06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74d815a3f0 0x7f74d815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74d81f55a0 0x7f74d81f2fb0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 --2- 172.21.15.60:0/2414844322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c80047c0 0x7f74d81f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 >> 172.21.15.60:0/2414844322 conn(0x7f74d80cf570 msgr2=0x7f74d80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 shutdown_connections 2023-10-16T12:50:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:29.947+0000 7f74ddc21640 1 -- 172.21.15.60:0/2414844322 wait complete. 2023-10-16T12:50:29.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:50:30.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.195+0000 7fd26ad11640 1 Processor -- start 2023-10-16T12:50:30.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.195+0000 7fd26ad11640 1 -- start start 2023-10-16T12:50:30.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.196+0000 7fd26ad11640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd264070b60 0x7fd26415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.196+0000 7fd26ad11640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd26406c100 0x7fd26406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.196+0000 7fd26ad11640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26415aef0 con 0x7fd264159950 2023-10-16T12:50:30.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.196+0000 7fd26ad11640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26415e490 con 0x7fd26406c100 2023-10-16T12:50:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.196+0000 7fd26ad11640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26415d910 con 0x7fd264070b60 2023-10-16T12:50:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd269d0f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd264159950 0x7fd264070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44650/0 (socket says 172.21.15.60:44650) 2023-10-16T12:50:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd269d0f640 1 -- 172.21.15.60:0/2117890479 learned_addr learned my addr 172.21.15.60:0/2117890479 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26a510640 1 --2- 172.21.15.60:0/2117890479 >> v2:172.21.15.60:3300/0 conn(0x7fd26406c100 0x7fd26406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26950e640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd264070b60 0x7fd26415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd268d0d640 1 -- 172.21.15.60:0/2117890479 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371238862 0 0) 0x7fd254001490 con 0x7fd264159950 2023-10-16T12:50:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd268d0d640 1 -- 172.21.15.60:0/2117890479 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24c003520 con 0x7fd264159950 2023-10-16T12:50:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26a510640 1 -- 172.21.15.60:0/2117890479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd264070b60 msgr2=0x7fd26415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26a510640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd264070b60 0x7fd26415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26a510640 1 -- 172.21.15.60:0/2117890479 >> v1:172.21.15.60:6789/0 conn(0x7fd264159950 legacy=0x7fd264070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26a510640 1 -- 172.21.15.60:0/2117890479 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26415dbf0 con 0x7fd26406c100 2023-10-16T12:50:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.197+0000 7fd26950e640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd264070b60 0x7fd26415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.199+0000 7fd26a510640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26406c100 0x7fd26406c550 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0097a0 tx=0x7fd25c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f036427d33b8f7aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.199+0000 7fd268d0d640 1 -- 172.21.15.60:0/2117890479 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c031780 con 0x7fd26406c100 2023-10-16T12:50:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.199+0000 7fd268d0d640 1 -- 172.21.15.60:0/2117890479 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c0318e0 con 0x7fd26406c100 2023-10-16T12:50:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.199+0000 7fd268d0d640 1 -- 172.21.15.60:0/2117890479 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c031c70 con 0x7fd26406c100 2023-10-16T12:50:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.199+0000 7fd26ad11640 1 -- 172.21.15.60:0/2117890479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26406c100 msgr2=0x7fd26406c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.199+0000 7fd26ad11640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26406c100 0x7fd26406c550 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0097a0 tx=0x7fd25c02b820 comp rx=0 tx=0).stop 2023-10-16T12:50:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.200+0000 7fd26ad11640 1 -- 172.21.15.60:0/2117890479 shutdown_connections 2023-10-16T12:50:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.200+0000 7fd26ad11640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26406c100 0x7fd26406c550 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.200+0000 7fd26ad11640 1 --2- 172.21.15.60:0/2117890479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd264070b60 0x7fd26415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.200+0000 7fd26ad11640 1 -- 172.21.15.60:0/2117890479 >> 172.21.15.60:0/2117890479 conn(0x7fd2640ce6a0 msgr2=0x7fd2640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.200+0000 7fd26ad11640 1 -- 172.21.15.60:0/2117890479 shutdown_connections 2023-10-16T12:50:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.200+0000 7fd26ad11640 1 -- 172.21.15.60:0/2117890479 wait complete. 2023-10-16T12:50:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.201+0000 7fd26ad11640 1 Processor -- start 2023-10-16T12:50:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.202+0000 7fd26ad11640 1 -- start start 2023-10-16T12:50:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.202+0000 7fd26ad11640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26406c100 0x7fd264155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.202+0000 7fd26ad11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 0x7fd2640c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.202+0000 7fd26ad11640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd264159950 0x7fd26414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26ad11640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd264156fa0 con 0x7fd264070b60 2023-10-16T12:50:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26ad11640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640c31f0 con 0x7fd264159950 2023-10-16T12:50:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26ad11640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd264156700 con 0x7fd26406c100 2023-10-16T12:50:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 0x7fd2640c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 0x7fd2640c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51702/0 (socket says 172.21.15.60:51702) 2023-10-16T12:50:30.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 -- 172.21.15.60:0/386791325 learned_addr learned my addr 172.21.15.60:0/386791325 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:30.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd269d0f640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26406c100 0x7fd264155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 -- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26406c100 msgr2=0x7fd264155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26406c100 0x7fd264155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26a510640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd264159950 0x7fd26414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 -- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd264159950 msgr2=0x7fd26414cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd264159950 0x7fd26414cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.203+0000 7fd26950e640 1 -- 172.21.15.60:0/386791325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c009400 con 0x7fd264070b60 2023-10-16T12:50:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.204+0000 7fd26950e640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 0x7fd2640c7cc0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd25800c7f0 tx=0x7fd25800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.204+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258007a30 con 0x7fd264070b60 2023-10-16T12:50:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.204+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd258007b90 con 0x7fd264070b60 2023-10-16T12:50:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.204+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800ce40 con 0x7fd264070b60 2023-10-16T12:50:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.204+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2641569e0 con 0x7fd264070b60 2023-10-16T12:50:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.204+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2640c3470 con 0x7fd264070b60 2023-10-16T12:50:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.205+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2640c3690 con 0x7fd264070b60 2023-10-16T12:50:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.206+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 43) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd2580113f0 con 0x7fd264070b60 2023-10-16T12:50:30.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.207+0000 7fd2627fc640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd25006af50 0x7fd25006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.207+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd258089bf0 con 0x7fd264070b60 2023-10-16T12:50:30.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.208+0000 7fd269d0f640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd25006af50 0x7fd25006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.209+0000 7fd269d0f640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd25006af50 0x7fd25006d410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2540016f0 tx=0x7fd254009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.213+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd258054900 con 0x7fd264070b60 2023-10-16T12:50:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.459+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd26404e9b0 con 0x7fd264070b60 2023-10-16T12:50:30.462 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:50:30.460+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:50:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.464+0000 7fd2627fc640 1 -- 172.21.15.60:0/386791325 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd258057f20 con 0x7fd264070b60 2023-10-16T12:50:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:50:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd25006af50 msgr2=0x7fd25006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd25006af50 0x7fd25006d410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2540016f0 tx=0x7fd254009040 comp rx=0 tx=0).stop 2023-10-16T12:50:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 msgr2=0x7fd2640c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 0x7fd2640c7cc0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd25800c7f0 tx=0x7fd25800cbf0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 shutdown_connections 2023-10-16T12:50:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fd25006af50 0x7fd25006d410 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd264159950 0x7fd26414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd264070b60 0x7fd2640c7cc0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 --2- 172.21.15.60:0/386791325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26406c100 0x7fd264155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 >> 172.21.15.60:0/386791325 conn(0x7fd2640ce6a0 msgr2=0x7fd264070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.467+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 shutdown_connections 2023-10-16T12:50:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.468+0000 7fd26ad11640 1 -- 172.21.15.60:0/386791325 wait complete. 2023-10-16T12:50:30.477 INFO:tasks.ceph.mds.a.smithi060.stderr: -2649> 2023-10-16T12:50:11.856+0000 7f5ad2dba640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:50:30.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:50:30.547 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:50:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.689+0000 7f1aea947640 1 Processor -- start 2023-10-16T12:50:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.690+0000 7f1aea947640 1 -- start start 2023-10-16T12:50:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.691+0000 7f1aea947640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ae415d150 0x7f1ae40ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.691+0000 7f1aea947640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae406a4b0 0x7f1ae4160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.691+0000 7f1aea947640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae40ac890 con 0x7f1ae4160b90 2023-10-16T12:50:30.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.691+0000 7f1aea947640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae415d530 con 0x7f1ae406a4b0 2023-10-16T12:50:30.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.691+0000 7f1aea947640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae4161650 con 0x7f1ae415d150 2023-10-16T12:50:30.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae406a4b0 0x7f1ae4160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae406a4b0 0x7f1ae4160fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51724/0 (socket says 172.21.15.60:51724) 2023-10-16T12:50:30.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 -- 172.21.15.60:0/2909326098 learned_addr learned my addr 172.21.15.60:0/2909326098 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:30.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1ae4160b90 0x7f1ae4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44670/0 (socket says 172.21.15.60:44670) 2023-10-16T12:50:30.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae3fff640 1 --2- 172.21.15.60:0/2909326098 >> v2:172.21.15.144:3300/0 conn(0x7f1ae415d150 0x7f1ae40ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 -- 172.21.15.60:0/2909326098 >> v2:172.21.15.144:3300/0 conn(0x7f1ae415d150 msgr2=0x7f1ae40ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 --2- 172.21.15.60:0/2909326098 >> v2:172.21.15.144:3300/0 conn(0x7f1ae415d150 0x7f1ae40ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 -- 172.21.15.60:0/2909326098 >> v1:172.21.15.60:6789/0 conn(0x7f1ae4160b90 legacy=0x7f1ae4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae8ee4640 1 -- 172.21.15.60:0/2909326098 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4161930 con 0x7f1ae406a4b0 2023-10-16T12:50:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae3fff640 1 --2- 172.21.15.60:0/2909326098 >> v2:172.21.15.144:3300/0 conn(0x7f1ae415d150 0x7f1ae40ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.692+0000 7f1ae2ffd640 1 -- 172.21.15.60:0/2909326098 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635808112 0 0) 0x7f1ae40ac890 con 0x7f1ae4160b90 2023-10-16T12:50:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.694+0000 7f1ae8ee4640 1 --2- 172.21.15.60:0/2909326098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae406a4b0 0x7f1ae4160fe0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40b15a0 tx=0x7f1adc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c3791ea57ea9a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.694+0000 7f1ae2ffd640 1 -- 172.21.15.60:0/2909326098 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc031870 con 0x7f1ae406a4b0 2023-10-16T12:50:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.694+0000 7f1ae2ffd640 1 -- 172.21.15.60:0/2909326098 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc0319d0 con 0x7f1ae406a4b0 2023-10-16T12:50:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.694+0000 7f1ae2ffd640 1 -- 172.21.15.60:0/2909326098 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc031d60 con 0x7f1ae406a4b0 2023-10-16T12:50:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 -- 172.21.15.60:0/2909326098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae406a4b0 msgr2=0x7f1ae4160fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 --2- 172.21.15.60:0/2909326098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae406a4b0 0x7f1ae4160fe0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40b15a0 tx=0x7f1adc02b910 comp rx=0 tx=0).stop 2023-10-16T12:50:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 -- 172.21.15.60:0/2909326098 shutdown_connections 2023-10-16T12:50:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 --2- 172.21.15.60:0/2909326098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae406a4b0 0x7f1ae4160fe0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 --2- 172.21.15.60:0/2909326098 >> v2:172.21.15.144:3300/0 conn(0x7f1ae415d150 0x7f1ae40ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 -- 172.21.15.60:0/2909326098 >> 172.21.15.60:0/2909326098 conn(0x7f1ae40ce9b0 msgr2=0x7f1ae40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 -- 172.21.15.60:0/2909326098 shutdown_connections 2023-10-16T12:50:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.695+0000 7f1aea947640 1 -- 172.21.15.60:0/2909326098 wait complete. 2023-10-16T12:50:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.697+0000 7f1aea947640 1 Processor -- start 2023-10-16T12:50:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.697+0000 7f1aea947640 1 -- start start 2023-10-16T12:50:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1aea947640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ae406a4b0 0x7f1ae41f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1aea947640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 0x7f1ae414ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1aea947640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4160b90 0x7f1ae415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1aea947640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae415d690 con 0x7f1ae415d150 2023-10-16T12:50:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1aea947640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae415e210 con 0x7f1ae4160b90 2023-10-16T12:50:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1aea947640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41f2660 con 0x7f1ae406a4b0 2023-10-16T12:50:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 0x7f1ae414ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 0x7f1ae414ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51728/0 (socket says 172.21.15.60:51728) 2023-10-16T12:50:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1ae37fe640 1 -- 172.21.15.60:0/2825291736 learned_addr learned my addr 172.21.15.60:0/2825291736 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.698+0000 7f1ae3fff640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ae406a4b0 0x7f1ae41f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae37fe640 1 -- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ae406a4b0 msgr2=0x7f1ae41f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae8ee4640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4160b90 0x7f1ae415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae37fe640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ae406a4b0 0x7f1ae41f2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae37fe640 1 -- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4160b90 msgr2=0x7f1ae415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae37fe640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4160b90 0x7f1ae415e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae37fe640 1 -- 172.21.15.60:0/2825291736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc0094d0 con 0x7f1ae415d150 2023-10-16T12:50:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae3fff640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ae406a4b0 0x7f1ae41f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae37fe640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 0x7f1ae414ca90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00c460 tx=0x7f1acc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0074e0 con 0x7f1ae415d150 2023-10-16T12:50:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1acc007660 con 0x7f1ae415d150 2023-10-16T12:50:30.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc004a20 con 0x7f1ae415d150 2023-10-16T12:50:30.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae415d8e0 con 0x7f1ae415d150 2023-10-16T12:50:30.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.699+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae415dea0 con 0x7f1ae415d150 2023-10-16T12:50:30.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.700+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae404e160 con 0x7f1ae415d150 2023-10-16T12:50:30.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.701+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 43) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1acc004b80 con 0x7f1ae415d150 2023-10-16T12:50:30.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.702+0000 7f1ae0ff9640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f1ac406af00 0x7f1ac406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.702+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f1acc088b70 con 0x7f1ae415d150 2023-10-16T12:50:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.703+0000 7f1ae3fff640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f1ac406af00 0x7f1ac406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.704+0000 7f1ae3fff640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f1ac406af00 0x7f1ac406d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009540 tx=0x7f1ad400b040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.707+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1acc0538f0 con 0x7f1ae415d150 2023-10-16T12:50:30.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.809+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1acc053cc0 con 0x7f1ae415d150 2023-10-16T12:50:30.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:30.947+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1ae40ade10 con 0x7f1ae415d150 2023-10-16T12:50:31.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.467+0000 7f1ae0ff9640 1 -- 172.21.15.60:0/2825291736 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1acc072a50 con 0x7f1ae415d150 2023-10-16T12:50:31.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f1ac406af00 msgr2=0x7f1ac406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f1ac406af00 0x7f1ac406d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009540 tx=0x7f1ad400b040 comp rx=0 tx=0).stop 2023-10-16T12:50:31.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 msgr2=0x7f1ae414ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 0x7f1ae414ca90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00c460 tx=0x7f1acc00c920 comp rx=0 tx=0).stop 2023-10-16T12:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 shutdown_connections 2023-10-16T12:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f1ac406af00 0x7f1ac406d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4160b90 0x7f1ae415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae415d150 0x7f1ae414ca90 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 --2- 172.21.15.60:0/2825291736 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ae406a4b0 0x7f1ae41f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 >> 172.21.15.60:0/2825291736 conn(0x7f1ae40ce9b0 msgr2=0x7f1ae40cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.471+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 shutdown_connections 2023-10-16T12:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.472+0000 7f1aea947640 1 -- 172.21.15.60:0/2825291736 wait complete. 2023-10-16T12:50:31.496 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:50:31.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.725+0000 7f57eaef8640 1 Processor -- start 2023-10-16T12:50:31.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.725+0000 7f57eaef8640 1 -- start start 2023-10-16T12:50:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.726+0000 7f57eaef8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57e415dde0 0x7f57e40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.727+0000 7f57eaef8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e4160a90 con 0x7f57e415c9f0 2023-10-16T12:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.727+0000 7f57eaef8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40cd900 con 0x7f57e4070f20 2023-10-16T12:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.727+0000 7f57eaef8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e4071350 con 0x7f57e415dde0 2023-10-16T12:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.727+0000 7f57e9ef6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f57e415c9f0 0x7f57e40c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44678/0 (socket says 172.21.15.60:44678) 2023-10-16T12:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.727+0000 7f57e9ef6640 1 -- 172.21.15.60:0/3769020867 learned_addr learned my addr 172.21.15.60:0/3769020867 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:31.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.727+0000 7f57e96f5640 1 --2- 172.21.15.60:0/3769020867 >> v2:172.21.15.144:3300/0 conn(0x7f57e415dde0 0x7f57e40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067741433 0 0) 0x7f57e4160a90 con 0x7f57e415c9f0 2023-10-16T12:50:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d0003520 con 0x7f57e415c9f0 2023-10-16T12:50:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650081517 0 0) 0x7f57d0003520 con 0x7f57e415c9f0 2023-10-16T12:50:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 >> v2:172.21.15.144:3300/0 conn(0x7f57e415dde0 msgr2=0x7f57e40709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 --2- 172.21.15.60:0/3769020867 >> v2:172.21.15.144:3300/0 conn(0x7f57e415dde0 0x7f57e40709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 >> v1:172.21.15.144:6789/0 conn(0x7f57e4070f20 legacy=0x7f57e40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40cdbe0 con 0x7f57e415c9f0 2023-10-16T12:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f57e0001670 con 0x7f57e415c9f0 2023-10-16T12:50:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 >> v1:172.21.15.60:6789/0 conn(0x7f57e415c9f0 legacy=0x7f57e40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0003970 0x7f57d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57d00047c0 0x7f57d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57d0005180 0x7f57d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e4071350 con 0x7f57d0005180 2023-10-16T12:50:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d000c020 con 0x7f57d0003970 2023-10-16T12:50:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.728+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d000c1e0 con 0x7f57d00047c0 2023-10-16T12:50:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57d0005180 0x7f57d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57e9ef6640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0003970 0x7f57d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57e96f5640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57d00047c0 0x7f57d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57d00047c0 msgr2=0x7f57d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57d00047c0 0x7f57d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0003970 msgr2=0x7f57d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:31.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0003970 0x7f57d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3769020867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d0003520 con 0x7f57d0005180 2023-10-16T12:50:31.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.729+0000 7f57e96f5640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57d00047c0 0x7f57d0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:31.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.730+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3769020867 reap_dead start 2023-10-16T12:50:31.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.730+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57d0005180 0x7f57d000b980 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f57dc001e50 tx=0x7f57d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a432b7976a9e0c43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.730+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc003b70 con 0x7f57d0005180 2023-10-16T12:50:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.730+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc003d10 con 0x7f57d0005180 2023-10-16T12:50:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.730+0000 7f57e8ef4640 1 -- 172.21.15.60:0/3769020867 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc005310 con 0x7f57d0005180 2023-10-16T12:50:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 -- 172.21.15.60:0/3769020867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57d0005180 msgr2=0x7f57d000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57d0005180 0x7f57d000b980 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f57dc001e50 tx=0x7f57d00057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 -- 172.21.15.60:0/3769020867 shutdown_connections 2023-10-16T12:50:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3769020867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57d0005180 0x7f57d000b980 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 -- 172.21.15.60:0/3769020867 >> 172.21.15.60:0/3769020867 conn(0x7f57e40ce6a0 msgr2=0x7f57e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 -- 172.21.15.60:0/3769020867 shutdown_connections 2023-10-16T12:50:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.731+0000 7f57eaef8640 1 -- 172.21.15.60:0/3769020867 wait complete. 2023-10-16T12:50:31.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.733+0000 7f57eaef8640 1 Processor -- start 2023-10-16T12:50:31.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.733+0000 7f57eaef8640 1 -- start start 2023-10-16T12:50:31.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57eaef8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0005180 0x7f57e4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57eaef8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57e4153f40 0x7f57e406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57eaef8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 0x7f57e406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57eaef8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415f8f0 con 0x7f57e40c8060 2023-10-16T12:50:31.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57eaef8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415ec70 con 0x7f57d0005180 2023-10-16T12:50:31.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57eaef8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e415ef50 con 0x7f57e4153f40 2023-10-16T12:50:31.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57ea6f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 0x7f57e406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57ea6f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 0x7f57e406bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51752/0 (socket says 172.21.15.60:51752) 2023-10-16T12:50:31.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3861757522 learned_addr learned my addr 172.21.15.60:0/3861757522 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:31.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57e9ef6640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0005180 0x7f57e4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.734+0000 7f57e96f5640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57e4153f40 0x7f57e406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57e4153f40 msgr2=0x7f57e406ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57e4153f40 0x7f57e406ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0005180 msgr2=0x7f57e4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:31.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0005180 0x7f57e4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:31.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57ea6f7640 1 -- 172.21.15.60:0/3861757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e415f230 con 0x7f57e40c8060 2023-10-16T12:50:31.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57ea6f7640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 0x7f57e406bfc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f57dc001f70 tx=0x7f57dc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:31.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc006d50 con 0x7f57e40c8060 2023-10-16T12:50:31.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc003b80 con 0x7f57e40c8060 2023-10-16T12:50:31.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0396c0 con 0x7f57e40c8060 2023-10-16T12:50:31.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.735+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e415f510 con 0x7f57e40c8060 2023-10-16T12:50:31.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.736+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e406a4c0 con 0x7f57e40c8060 2023-10-16T12:50:31.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.737+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e404e9b0 con 0x7f57e40c8060 2023-10-16T12:50:31.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.738+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f57dc039820 con 0x7f57e40c8060 2023-10-16T12:50:31.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.739+0000 7f57da7fc640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f57c806af90 0x7f57c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:31.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.739+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f57dc0ab970 con 0x7f57e40c8060 2023-10-16T12:50:31.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.739+0000 7f57e9ef6640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f57c806af90 0x7f57c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:31.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.740+0000 7f57e9ef6640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f57c806af90 0x7f57c806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57e0000f60 tx=0x7f57e0009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:31.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.743+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57dc076590 con 0x7f57e40c8060 2023-10-16T12:50:31.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:31.938+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57e40adac0 con 0x7f57e40c8060 2023-10-16T12:50:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.478+0000 7f57da7fc640 1 -- 172.21.15.60:0/3861757522 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 181+0+0 (secure 0 0 0) 0x7f57dc079bb0 con 0x7f57e40c8060 2023-10-16T12:50:32.481 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:50:32.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f57c806af90 msgr2=0x7f57c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:32.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f57c806af90 0x7f57c806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57e0000f60 tx=0x7f57e0009040 comp rx=0 tx=0).stop 2023-10-16T12:50:32.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 msgr2=0x7f57e406bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 0x7f57e406bfc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f57dc001f70 tx=0x7f57dc0014a0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 shutdown_connections 2023-10-16T12:50:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f57c806af90 0x7f57c806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57e4153f40 0x7f57e406ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57e40c8060 0x7f57e406bfc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 --2- 172.21.15.60:0/3861757522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57d0005180 0x7f57e4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.481+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 >> 172.21.15.60:0/3861757522 conn(0x7f57e40ce6a0 msgr2=0x7f57e40af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.482+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 shutdown_connections 2023-10-16T12:50:32.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.482+0000 7f57eaef8640 1 -- 172.21.15.60:0/3861757522 wait complete. 2023-10-16T12:50:32.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:50:32.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.731+0000 7fbf3c9e2640 1 Processor -- start 2023-10-16T12:50:32.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.731+0000 7fbf3c9e2640 1 -- start start 2023-10-16T12:50:32.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf3c9e2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf38159950 0x7fbf3815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf3c9e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf380708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf3c9e2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbf3806c100 0x7fbf3806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf3c9e2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3815e490 con 0x7fbf38159950 2023-10-16T12:50:32.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf3c9e2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf38070df0 con 0x7fbf3806c100 2023-10-16T12:50:32.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf3c9e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3815aef0 con 0x7fbf38070450 2023-10-16T12:50:32.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf377fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf38159950 0x7fbf3815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:32.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf377fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf38159950 0x7fbf3815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51762/0 (socket says 172.21.15.60:51762) 2023-10-16T12:50:32.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.733+0000 7fbf377fe640 1 -- 172.21.15.60:0/1679244910 learned_addr learned my addr 172.21.15.60:0/1679244910 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:32.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.734+0000 7fbf377fe640 1 -- 172.21.15.60:0/1679244910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 msgr2=0x7fbf380708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:32.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.734+0000 7fbf377fe640 1 --2- 172.21.15.60:0/1679244910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf380708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.734+0000 7fbf377fe640 1 -- 172.21.15.60:0/1679244910 >> v2:172.21.15.144:3300/0 conn(0x7fbf3806c100 msgr2=0x7fbf3806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:32.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.734+0000 7fbf377fe640 1 --2- 172.21.15.60:0/1679244910 >> v2:172.21.15.144:3300/0 conn(0x7fbf3806c100 0x7fbf3806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.734+0000 7fbf377fe640 1 -- 172.21.15.60:0/1679244910 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3815d910 con 0x7fbf38159950 2023-10-16T12:50:32.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.735+0000 7fbf377fe640 1 --2- 172.21.15.60:0/1679244910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf38159950 0x7fbf3815a9b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fbf380b0120 tx=0x7fbf2802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432f438f1b3cffd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:32.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.735+0000 7fbf367fc640 1 -- 172.21.15.60:0/1679244910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf280317c0 con 0x7fbf38159950 2023-10-16T12:50:32.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.735+0000 7fbf367fc640 1 -- 172.21.15.60:0/1679244910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf28031920 con 0x7fbf38159950 2023-10-16T12:50:32.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.735+0000 7fbf367fc640 1 -- 172.21.15.60:0/1679244910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28031cb0 con 0x7fbf38159950 2023-10-16T12:50:32.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/1679244910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf38159950 msgr2=0x7fbf3815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:32.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/1679244910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf38159950 0x7fbf3815a9b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fbf380b0120 tx=0x7fbf2802b860 comp rx=0 tx=0).stop 2023-10-16T12:50:32.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/1679244910 shutdown_connections 2023-10-16T12:50:32.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/1679244910 >> v2:172.21.15.144:3300/0 conn(0x7fbf3806c100 0x7fbf3806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/1679244910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf380708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/1679244910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf38159950 0x7fbf3815a9b0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/1679244910 >> 172.21.15.60:0/1679244910 conn(0x7fbf380ce6a0 msgr2=0x7fbf380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:32.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/1679244910 shutdown_connections 2023-10-16T12:50:32.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.736+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/1679244910 wait complete. 2023-10-16T12:50:32.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.738+0000 7fbf3c9e2640 1 Processor -- start 2023-10-16T12:50:32.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.738+0000 7fbf3c9e2640 1 -- start start 2023-10-16T12:50:32.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf3c9e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 0x7fbf380c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf3c9e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf38158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf3c9e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf38159950 0x7fbf3814cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf3c9e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c7d50 con 0x7fbf3806c100 2023-10-16T12:50:32.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf3c9e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c8030 con 0x7fbf38159950 2023-10-16T12:50:32.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf3c9e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c8310 con 0x7fbf38070450 2023-10-16T12:50:32.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.739+0000 7fbf377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 0x7fbf380c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:32.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 0x7fbf380c4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51776/0 (socket says 172.21.15.60:51776) 2023-10-16T12:50:32.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 -- 172.21.15.60:0/3717911905 learned_addr learned my addr 172.21.15.60:0/3717911905 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:32.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf36ffd640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf38158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:32.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf37fff640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf38159950 0x7fbf3814cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:32.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 -- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 msgr2=0x7fbf38158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:32.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf38158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 -- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf38159950 msgr2=0x7fbf3814cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:32.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf38159950 0x7fbf3814cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:32.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 -- 172.21.15.60:0/3717911905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280094d0 con 0x7fbf3806c100 2023-10-16T12:50:32.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf37fff640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf38159950 0x7fbf3814cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:32.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.740+0000 7fbf377fe640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 0x7fbf380c4cc0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fbf28000c00 tx=0x7fbf280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:32.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.741+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28032860 con 0x7fbf3806c100 2023-10-16T12:50:32.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.741+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf280329e0 con 0x7fbf3806c100 2023-10-16T12:50:32.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.741+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28032d90 con 0x7fbf3806c100 2023-10-16T12:50:32.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.741+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf380c85f0 con 0x7fbf3806c100 2023-10-16T12:50:32.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.741+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf38158b90 con 0x7fbf3806c100 2023-10-16T12:50:32.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.742+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf38158d80 con 0x7fbf3806c100 2023-10-16T12:50:32.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.743+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf2804d3f0 con 0x7fbf3806c100 2023-10-16T12:50:32.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.744+0000 7fbf1ffff640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbf2406af90 0x7fbf2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:32.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.744+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6045+0+0 (secure 0 0 0) 0x7fbf280ab8d0 con 0x7fbf3806c100 2023-10-16T12:50:32.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.745+0000 7fbf36ffd640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbf2406af90 0x7fbf2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:32.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.746+0000 7fbf36ffd640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbf2406af90 0x7fbf2406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbf38165880 tx=0x7fbf2c008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:32.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.748+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf28076680 con 0x7fbf3806c100 2023-10-16T12:50:32.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:32.949+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf3804e9b0 con 0x7fbf3806c100 2023-10-16T12:50:33.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.490+0000 7fbf1ffff640 1 -- 172.21.15.60:0/3717911905 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbf28079ca0 con 0x7fbf3806c100 2023-10-16T12:50:33.493 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:50:33.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbf2406af90 msgr2=0x7fbf2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbf2406af90 0x7fbf2406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbf38165880 tx=0x7fbf2c008040 comp rx=0 tx=0).stop 2023-10-16T12:50:33.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 msgr2=0x7fbf380c4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 0x7fbf380c4cc0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fbf28000c00 tx=0x7fbf280046c0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 shutdown_connections 2023-10-16T12:50:33.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbf2406af90 0x7fbf2406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf38070450 0x7fbf38158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf38159950 0x7fbf3814cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 --2- 172.21.15.60:0/3717911905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf3806c100 0x7fbf380c4cc0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.494+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 >> 172.21.15.60:0/3717911905 conn(0x7fbf380ce6a0 msgr2=0x7fbf3815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:33.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.495+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 shutdown_connections 2023-10-16T12:50:33.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.495+0000 7fbf3c9e2640 1 -- 172.21.15.60:0/3717911905 wait complete. 2023-10-16T12:50:33.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:50:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.743+0000 7f5743fff640 1 Processor -- start 2023-10-16T12:50:33.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.743+0000 7f5743fff640 1 -- start start 2023-10-16T12:50:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.744+0000 7f5743fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57440b1ca0 0x7f57440d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.745+0000 7f5743fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440d0970 con 0x7f57440b2080 2023-10-16T12:50:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.745+0000 7f5743fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440d0ea0 con 0x7f57440cfc40 2023-10-16T12:50:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.745+0000 7f5743fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57440d0bc0 con 0x7f57440b1ca0 2023-10-16T12:50:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.745+0000 7f57427fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f57440b2080 0x7f57440ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44692/0 (socket says 172.21.15.60:44692) 2023-10-16T12:50:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.745+0000 7f57427fc640 1 -- 172.21.15.60:0/596996304 learned_addr learned my addr 172.21.15.60:0/596996304 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.745+0000 7f5742ffd640 1 --2- 172.21.15.60:0/596996304 >> v2:172.21.15.144:3300/0 conn(0x7f57440b1ca0 0x7f57440d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772321269 0 0) 0x7f57440d0970 con 0x7f57440b2080 2023-10-16T12:50:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5730003520 con 0x7f57440b2080 2023-10-16T12:50:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618726585 0 0) 0x7f5730003520 con 0x7f57440b2080 2023-10-16T12:50:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 >> v2:172.21.15.144:3300/0 conn(0x7f57440b1ca0 msgr2=0x7f57440d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 --2- 172.21.15.60:0/596996304 >> v2:172.21.15.144:3300/0 conn(0x7f57440b1ca0 0x7f57440d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 >> v1:172.21.15.144:6789/0 conn(0x7f57440cfc40 legacy=0x7f57440b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440cd940 con 0x7f57440b2080 2023-10-16T12:50:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f572c001670 con 0x7f57440b2080 2023-10-16T12:50:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 >> v1:172.21.15.60:6789/0 conn(0x7f57440b2080 legacy=0x7f57440ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5730003970 0x7f5730003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57300047c0 0x7f5730004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5730005180 0x7f573000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440d0bc0 con 0x7f5730003970 2023-10-16T12:50:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573000c030 con 0x7f57300047c0 2023-10-16T12:50:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573000c1f0 con 0x7f5730005180 2023-10-16T12:50:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.746+0000 7f5742ffd640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5730003970 0x7f5730003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f57427fc640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57300047c0 0x7f5730004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f5742ffd640 1 -- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5730005180 msgr2=0x7f573000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f5742ffd640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5730005180 0x7f573000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f5742ffd640 1 -- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57300047c0 msgr2=0x7f5730004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f5742ffd640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57300047c0 0x7f5730004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f5742ffd640 1 -- 172.21.15.60:0/596996304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730003520 con 0x7f5730003970 2023-10-16T12:50:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.747+0000 7f57437fe640 1 -- 172.21.15.60:0/596996304 reap_dead start 2023-10-16T12:50:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.748+0000 7f5742ffd640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5730003970 0x7f5730003d90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f5734009fc0 tx=0x7f57300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2163abf3a7cdf46f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.748+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573402cc20 con 0x7f5730003970 2023-10-16T12:50:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.748+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573400f040 con 0x7f5730003970 2023-10-16T12:50:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.748+0000 7f5741ffb640 1 -- 172.21.15.60:0/596996304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734034440 con 0x7f5730003970 2023-10-16T12:50:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 -- 172.21.15.60:0/596996304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5730003970 msgr2=0x7f5730003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5730003970 0x7f5730003d90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f5734009fc0 tx=0x7f57300057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 -- 172.21.15.60:0/596996304 shutdown_connections 2023-10-16T12:50:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 --2- 172.21.15.60:0/596996304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5730003970 0x7f5730003d90 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 -- 172.21.15.60:0/596996304 >> 172.21.15.60:0/596996304 conn(0x7f57440d2b90 msgr2=0x7f57440d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 -- 172.21.15.60:0/596996304 shutdown_connections 2023-10-16T12:50:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.749+0000 7f5743fff640 1 -- 172.21.15.60:0/596996304 wait complete. 2023-10-16T12:50:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.752+0000 7f5743fff640 1 Processor -- start 2023-10-16T12:50:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.752+0000 7f5743fff640 1 -- start start 2023-10-16T12:50:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f5743fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5730003970 0x7f57441e7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f5743fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57441f2e90 0x7f57440b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f5743fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 0x7f57441571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f5743fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5744157720 con 0x7f5744156d50 2023-10-16T12:50:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f5743fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57441e5ac0 con 0x7f5730003970 2023-10-16T12:50:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f5743fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57441e5da0 con 0x7f57441f2e90 2023-10-16T12:50:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f57437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 0x7f57441571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.753+0000 7f57437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 0x7f57441571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51790/0 (socket says 172.21.15.60:51790) 2023-10-16T12:50:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 -- 172.21.15.60:0/3983877226 learned_addr learned my addr 172.21.15.60:0/3983877226 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57427fc640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57441f2e90 0x7f57440b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f5742ffd640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5730003970 0x7f57441e7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 -- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57441f2e90 msgr2=0x7f57440b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57441f2e90 0x7f57440b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 -- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5730003970 msgr2=0x7f57441e7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5730003970 0x7f57441e7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 -- 172.21.15.60:0/3983877226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57441e6080 con 0x7f5744156d50 2023-10-16T12:50:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.754+0000 7f57437fe640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 0x7f57441571e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f57440d0020 tx=0x7f573c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.755+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c004220 con 0x7f5744156d50 2023-10-16T12:50:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.755+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c0043a0 con 0x7f5744156d50 2023-10-16T12:50:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.755+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c007310 con 0x7f5744156d50 2023-10-16T12:50:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.755+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57441e6360 con 0x7f5744156d50 2023-10-16T12:50:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.755+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57441e7890 con 0x7f5744156d50 2023-10-16T12:50:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.756+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574404e9b0 con 0x7f5744156d50 2023-10-16T12:50:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.757+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f573c007470 con 0x7f5744156d50 2023-10-16T12:50:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.758+0000 7f573b7fe640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f572406af90 0x7f572406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.758+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f573c088740 con 0x7f5744156d50 2023-10-16T12:50:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.759+0000 7f5742ffd640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f572406af90 0x7f572406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.760+0000 7f5742ffd640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f572406af90 0x7f572406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5734036000 tx=0x7f573402e000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.763+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f573c053730 con 0x7f5744156d50 2023-10-16T12:50:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.959+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5744073a40 con 0x7f5744156d50 2023-10-16T12:50:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.965+0000 7f573b7fe640 1 -- 172.21.15.60:0/3983877226 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f573c056d50 con 0x7f5744156d50 2023-10-16T12:50:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.967+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f572406af90 msgr2=0x7f572406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.967+0000 7f5743fff640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f572406af90 0x7f572406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5734036000 tx=0x7f573402e000 comp rx=0 tx=0).stop 2023-10-16T12:50:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.967+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 msgr2=0x7f57441571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.967+0000 7f5743fff640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 0x7f57441571e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f57440d0020 tx=0x7f573c00c820 comp rx=0 tx=0).stop 2023-10-16T12:50:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 shutdown_connections 2023-10-16T12:50:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f572406af90 0x7f572406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57441f2e90 0x7f57440b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5744156d50 0x7f57441571e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 --2- 172.21.15.60:0/3983877226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5730003970 0x7f57441e7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 >> 172.21.15.60:0/3983877226 conn(0x7f57440d2b90 msgr2=0x7f57441594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 shutdown_connections 2023-10-16T12:50:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:33.968+0000 7f5743fff640 1 -- 172.21.15.60:0/3983877226 wait complete. 2023-10-16T12:50:33.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:50:34.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.207+0000 7f920e49b640 1 Processor -- start 2023-10-16T12:50:34.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.208+0000 7f920e49b640 1 -- start start 2023-10-16T12:50:34.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.209+0000 7f920e49b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f920806b660 0x7f920806ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.209+0000 7f920e49b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f92081428c0 0x7f92080c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f920e49b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92080c9ae0 con 0x7f92080c9690 2023-10-16T12:50:34.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f920e49b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92080cc6b0 con 0x7f92081428c0 2023-10-16T12:50:34.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f920e49b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92081599b0 con 0x7f920806b660 2023-10-16T12:50:34.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f92081428c0 0x7f92080c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f92081428c0 0x7f92080c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51796/0 (socket says 172.21.15.60:51796) 2023-10-16T12:50:34.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 -- 172.21.15.60:0/936485676 learned_addr learned my addr 172.21.15.60:0/936485676 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:34.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f9207fff640 1 --2- 172.21.15.60:0/936485676 >> v2:172.21.15.144:3300/0 conn(0x7f920806b660 0x7f920806ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 -- 172.21.15.60:0/936485676 >> v2:172.21.15.144:3300/0 conn(0x7f920806b660 msgr2=0x7f920806ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:34.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 --2- 172.21.15.60:0/936485676 >> v2:172.21.15.144:3300/0 conn(0x7f920806b660 0x7f920806ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 -- 172.21.15.60:0/936485676 >> v1:172.21.15.144:6789/0 conn(0x7f92080c9690 legacy=0x7f920815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:34.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.210+0000 7f91ff7fe640 1 -- 172.21.15.60:0/936485676 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920806c290 con 0x7f92081428c0 2023-10-16T12:50:34.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.212+0000 7f91ff7fe640 1 --2- 172.21.15.60:0/936485676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92081428c0 0x7f92080c9150 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f92080c9120 tx=0x7f91f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c392b8369cb3d1d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:34.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.212+0000 7f92077fe640 1 -- 172.21.15.60:0/936485676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f002ca80 con 0x7f92081428c0 2023-10-16T12:50:34.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.212+0000 7f92077fe640 1 -- 172.21.15.60:0/936485676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f002cbe0 con 0x7f92081428c0 2023-10-16T12:50:34.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.212+0000 7f92077fe640 1 -- 172.21.15.60:0/936485676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f00343f0 con 0x7f92081428c0 2023-10-16T12:50:34.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 -- 172.21.15.60:0/936485676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92081428c0 msgr2=0x7f92080c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 --2- 172.21.15.60:0/936485676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92081428c0 0x7f92080c9150 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f92080c9120 tx=0x7f91f002bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 -- 172.21.15.60:0/936485676 shutdown_connections 2023-10-16T12:50:34.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 --2- 172.21.15.60:0/936485676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92081428c0 0x7f92080c9150 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 --2- 172.21.15.60:0/936485676 >> v2:172.21.15.144:3300/0 conn(0x7f920806b660 0x7f920806ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 -- 172.21.15.60:0/936485676 >> 172.21.15.60:0/936485676 conn(0x7f92080ce9b0 msgr2=0x7f92080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:34.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 -- 172.21.15.60:0/936485676 shutdown_connections 2023-10-16T12:50:34.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.213+0000 7f920e49b640 1 -- 172.21.15.60:0/936485676 wait complete. 2023-10-16T12:50:34.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.215+0000 7f920e49b640 1 Processor -- start 2023-10-16T12:50:34.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.215+0000 7f920e49b640 1 -- start start 2023-10-16T12:50:34.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920e49b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f920806b660 0x7f92080c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920e49b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 0x7f920814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920e49b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92081428c0 0x7f920814b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920e49b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920815bd50 con 0x7f92080c9690 2023-10-16T12:50:34.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920e49b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920814c870 con 0x7f92081428c0 2023-10-16T12:50:34.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920e49b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920814cb50 con 0x7f920806b660 2023-10-16T12:50:34.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f91ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 0x7f920814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f91ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 0x7f920814ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51802/0 (socket says 172.21.15.60:51802) 2023-10-16T12:50:34.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f91ff7fe640 1 -- 172.21.15.60:0/2554234612 learned_addr learned my addr 172.21.15.60:0/2554234612 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:34.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f9207fff640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f920806b660 0x7f92080c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f920ca38640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92081428c0 0x7f920814b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.216+0000 7f91ff7fe640 1 -- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f920806b660 msgr2=0x7f92080c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f91ff7fe640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f920806b660 0x7f92080c40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f91ff7fe640 1 -- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92081428c0 msgr2=0x7f920814b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f91ff7fe640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92081428c0 0x7f920814b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f91ff7fe640 1 -- 172.21.15.60:0/2554234612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0009b30 con 0x7f92080c9690 2023-10-16T12:50:34.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f920ca38640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92081428c0 0x7f920814b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:34.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f91ff7fe640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 0x7f920814ae50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f91f0000f80 tx=0x7f91f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:34.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0009510 con 0x7f92080c9690 2023-10-16T12:50:34.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f003e450 con 0x7f92080c9690 2023-10-16T12:50:34.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f003e800 con 0x7f92080c9690 2023-10-16T12:50:34.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920814ce30 con 0x7f92080c9690 2023-10-16T12:50:34.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.217+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92080c4600 con 0x7f92080c9690 2023-10-16T12:50:34.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.219+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920804e160 con 0x7f92080c9690 2023-10-16T12:50:34.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.220+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f91f002cd50 con 0x7f92080c9690 2023-10-16T12:50:34.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.221+0000 7f92057fa640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f91e806af90 0x7f91e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.221+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f91f00ac1d0 con 0x7f92080c9690 2023-10-16T12:50:34.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.221+0000 7f9207fff640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f91e806af90 0x7f91e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.222+0000 7f9207fff640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f91e806af90 0x7f91e806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f91f8006fd0 tx=0x7f91f8008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:34.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.225+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91f0077120 con 0x7f92080c9690 2023-10-16T12:50:34.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.432+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f92080c9ae0 con 0x7f92080c9690 2023-10-16T12:50:34.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.438+0000 7f92057fa640 1 -- 172.21.15.60:0/2554234612 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f92080c9ae0 con 0x7f92080c9690 2023-10-16T12:50:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f91e806af90 msgr2=0x7f91e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f91e806af90 0x7f91e806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f91f8006fd0 tx=0x7f91f8008040 comp rx=0 tx=0).stop 2023-10-16T12:50:34.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 msgr2=0x7f920814ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 0x7f920814ae50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f91f0000f80 tx=0x7f91f0004a40 comp rx=0 tx=0).stop 2023-10-16T12:50:34.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 shutdown_connections 2023-10-16T12:50:34.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f91e806af90 0x7f91e806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92081428c0 0x7f920814b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92080c9690 0x7f920814ae50 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 --2- 172.21.15.60:0/2554234612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f920806b660 0x7f92080c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 >> 172.21.15.60:0/2554234612 conn(0x7f92080ce9b0 msgr2=0x7f92080cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:34.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 shutdown_connections 2023-10-16T12:50:34.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.441+0000 7f920e49b640 1 -- 172.21.15.60:0/2554234612 wait complete. 2023-10-16T12:50:34.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:50:34.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.683+0000 7f0fd31b3640 1 Processor -- start 2023-10-16T12:50:34.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.683+0000 7f0fd31b3640 1 -- start start 2023-10-16T12:50:34.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.684+0000 7f0fd31b3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc15c9f0 0x7f0fcc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd31b3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0fcc15dde0 0x7f0fcc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd31b3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc160a90 con 0x7f0fcc0c9230 2023-10-16T12:50:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd31b3640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fcc0cdec0 con 0x7f0fcc15dde0 2023-10-16T12:50:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd31b3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0cd900 con 0x7f0fcc15c9f0 2023-10-16T12:50:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd19b0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0fcc0c9230 0x7f0fcc0709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44702/0 (socket says 172.21.15.60:44702) 2023-10-16T12:50:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd19b0640 1 -- 172.21.15.60:0/3280292817 learned_addr learned my addr 172.21.15.60:0/3280292817 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3280292817 >> v2:172.21.15.144:3300/0 conn(0x7f0fcc15dde0 0x7f0fcc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.685+0000 7f0fd21b1640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc15c9f0 0x7f0fcc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89263487 0 0) 0x7f0fcc160a90 con 0x7f0fcc0c9230 2023-10-16T12:50:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb8003520 con 0x7f0fcc0c9230 2023-10-16T12:50:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784819380 0 0) 0x7f0fb8003520 con 0x7f0fcc0c9230 2023-10-16T12:50:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc15c9f0 msgr2=0x7f0fcc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc15c9f0 0x7f0fcc0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 >> v2:172.21.15.144:3300/0 conn(0x7f0fcc15dde0 msgr2=0x7f0fcc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 --2- 172.21.15.60:0/3280292817 >> v2:172.21.15.144:3300/0 conn(0x7f0fcc15dde0 0x7f0fcc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc0cdbe0 con 0x7f0fcc0c9230 2023-10-16T12:50:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0fbc001670 con 0x7f0fcc0c9230 2023-10-16T12:50:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 >> v1:172.21.15.60:6789/0 conn(0x7f0fcc0c9230 legacy=0x7f0fcc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fb8003970 0x7f0fb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004360 0x7f0fb8004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fb8004eb0 0x7f0fb800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0cd900 con 0x7f0fb8004eb0 2023-10-16T12:50:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb800bec0 con 0x7f0fb8004360 2023-10-16T12:50:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.686+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb800c000 con 0x7f0fb8003970 2023-10-16T12:50:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fb8004eb0 0x7f0fb800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd19b0640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004360 0x7f0fb8004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd21b1640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fb8003970 0x7f0fb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fb8003970 msgr2=0x7f0fb8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fb8003970 0x7f0fb8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004360 msgr2=0x7f0fb8004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004360 0x7f0fb8004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.687+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3280292817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8003520 con 0x7f0fb8004eb0 2023-10-16T12:50:34.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.688+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fb8004eb0 0x7f0fb800b980 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009f90 tx=0x7f0fb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a9ffdb9f13f1bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:34.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc402cc20 con 0x7f0fb8004eb0 2023-10-16T12:50:34.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc400f040 con 0x7f0fb8004eb0 2023-10-16T12:50:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd11af640 1 -- 172.21.15.60:0/3280292817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4034440 con 0x7f0fb8004eb0 2023-10-16T12:50:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3280292817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fb8004eb0 msgr2=0x7f0fb800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3280292817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fb8004eb0 0x7f0fb800b980 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009f90 tx=0x7f0fb80057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3280292817 reap_dead start 2023-10-16T12:50:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3280292817 shutdown_connections 2023-10-16T12:50:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.689+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3280292817 >> 172.21.15.60:0/3280292817 conn(0x7f0fcc0ce6a0 msgr2=0x7f0fcc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.690+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3280292817 shutdown_connections 2023-10-16T12:50:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.690+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3280292817 wait complete. 2023-10-16T12:50:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.691+0000 7f0fd31b3640 1 Processor -- start 2023-10-16T12:50:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.692+0000 7f0fd31b3640 1 -- start start 2023-10-16T12:50:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.692+0000 7f0fd31b3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004eb0 0x7f0fcc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.692+0000 7f0fd31b3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc153b80 0x7f0fcc0cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd31b3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 0x7f0fcc146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd31b3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc06c770 con 0x7f0fcc145fd0 2023-10-16T12:50:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd31b3640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc146990 con 0x7f0fb8004eb0 2023-10-16T12:50:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd31b3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc06bdd0 con 0x7f0fcc153b80 2023-10-16T12:50:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 0x7f0fcc146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 0x7f0fcc146450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51810/0 (socket says 172.21.15.60:51810) 2023-10-16T12:50:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3375156306 learned_addr learned my addr 172.21.15.60:0/3375156306 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd21b1640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004eb0 0x7f0fcc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd19b0640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc153b80 0x7f0fcc0cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc153b80 msgr2=0x7f0fcc0cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc153b80 0x7f0fcc0cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004eb0 msgr2=0x7f0fcc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004eb0 0x7f0fcc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd29b2640 1 -- 172.21.15.60:0/3375156306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc06c0b0 con 0x7f0fcc145fd0 2023-10-16T12:50:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.693+0000 7f0fd21b1640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004eb0 0x7f0fcc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.694+0000 7f0fd29b2640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 0x7f0fcc146450 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4037040 tx=0x7f0fc4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.694+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4034b50 con 0x7f0fcc145fd0 2023-10-16T12:50:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.694+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc4034cb0 con 0x7f0fcc145fd0 2023-10-16T12:50:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.694+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc06c390 con 0x7f0fcc145fd0 2023-10-16T12:50:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.694+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4003bd0 con 0x7f0fcc145fd0 2023-10-16T12:50:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.694+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc06cb90 con 0x7f0fcc145fd0 2023-10-16T12:50:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.695+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc04e9b0 con 0x7f0fcc145fd0 2023-10-16T12:50:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.696+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0fc4047460 con 0x7f0fcc145fd0 2023-10-16T12:50:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.698+0000 7f0fc2ffd640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f0fb006af90 0x7f0fb006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.698+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f0fc40ab650 con 0x7f0fcc145fd0 2023-10-16T12:50:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.699+0000 7f0fd21b1640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f0fb006af90 0x7f0fb006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.700+0000 7f0fd21b1640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f0fb006af90 0x7f0fb006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0ad4e0 tx=0x7f0fc8008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.702+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fc4076630 con 0x7f0fcc145fd0 2023-10-16T12:50:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.905+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0fcc0c9010 con 0x7f0fcc145fd0 2023-10-16T12:50:34.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.910+0000 7f0fc2ffd640 1 -- 172.21.15.60:0/3375156306 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0fc4079c50 con 0x7f0fcc145fd0 2023-10-16T12:50:34.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.913+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f0fb006af90 msgr2=0x7f0fb006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.913+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f0fb006af90 0x7f0fb006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0ad4e0 tx=0x7f0fc8008040 comp rx=0 tx=0).stop 2023-10-16T12:50:34.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.913+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 msgr2=0x7f0fcc146450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.913+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 0x7f0fcc146450 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4037040 tx=0x7f0fc4009750 comp rx=0 tx=0).stop 2023-10-16T12:50:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 shutdown_connections 2023-10-16T12:50:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f0fb006af90 0x7f0fb006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0fcc153b80 0x7f0fcc0cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fcc145fd0 0x7f0fcc146450 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 --2- 172.21.15.60:0/3375156306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fb8004eb0 0x7f0fcc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 >> 172.21.15.60:0/3375156306 conn(0x7f0fcc0ce6a0 msgr2=0x7f0fcc0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 shutdown_connections 2023-10-16T12:50:34.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:34.914+0000 7f0fd31b3640 1 -- 172.21.15.60:0/3375156306 wait complete. 2023-10-16T12:50:34.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T12:50:35.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.155+0000 7fc85b59e640 1 Processor -- start 2023-10-16T12:50:35.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.155+0000 7fc85b59e640 1 -- start start 2023-10-16T12:50:35.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c159950 0x7fc85c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:35.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc85c06c810 0x7fc85c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:35.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85c070df0 con 0x7fc85c070450 2023-10-16T12:50:35.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85c15e490 con 0x7fc85c06c810 2023-10-16T12:50:35.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c15aef0 con 0x7fc85c159950 2023-10-16T12:50:35.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc85c06c810 0x7fc85c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:35.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc85c06c810 0x7fc85c0708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51816/0 (socket says 172.21.15.60:51816) 2023-10-16T12:50:35.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc85ad9d640 1 -- 172.21.15.60:0/1593869228 learned_addr learned my addr 172.21.15.60:0/1593869228 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:35.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.157+0000 7fc851d9b640 1 --1- 172.21.15.60:0/1593869228 >> v1:172.21.15.60:6789/0 conn(0x7fc85c070450 0x7fc85c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44716/0 (socket says 172.21.15.60:44716) 2023-10-16T12:50:35.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.158+0000 7fc85a59c640 1 --2- 172.21.15.60:0/1593869228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c159950 0x7fc85c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:35.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.158+0000 7fc85ad9d640 1 -- 172.21.15.60:0/1593869228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c159950 msgr2=0x7fc85c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:35.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.158+0000 7fc85ad9d640 1 --2- 172.21.15.60:0/1593869228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c159950 0x7fc85c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.158+0000 7fc85ad9d640 1 -- 172.21.15.60:0/1593869228 >> v1:172.21.15.60:6789/0 conn(0x7fc85c070450 legacy=0x7fc85c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:35.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.158+0000 7fc85ad9d640 1 -- 172.21.15.60:0/1593869228 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c06cc80 con 0x7fc85c06c810 2023-10-16T12:50:35.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.159+0000 7fc85ad9d640 1 --2- 172.21.15.60:0/1593869228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c06c810 0x7fc85c0708b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc85400d150 tx=0x7fc85402e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71026bc7e7dc3bfa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:35.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.159+0000 7fc859d9b640 1 -- 172.21.15.60:0/1593869228 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85402ea70 con 0x7fc85c06c810 2023-10-16T12:50:35.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.159+0000 7fc859d9b640 1 -- 172.21.15.60:0/1593869228 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85402ebd0 con 0x7fc85c06c810 2023-10-16T12:50:35.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.159+0000 7fc859d9b640 1 -- 172.21.15.60:0/1593869228 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854009600 con 0x7fc85c06c810 2023-10-16T12:50:35.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 -- 172.21.15.60:0/1593869228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c06c810 msgr2=0x7fc85c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:35.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 --2- 172.21.15.60:0/1593869228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c06c810 0x7fc85c0708b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc85400d150 tx=0x7fc85402e0d0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 -- 172.21.15.60:0/1593869228 shutdown_connections 2023-10-16T12:50:35.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 --2- 172.21.15.60:0/1593869228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c06c810 0x7fc85c0708b0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 --2- 172.21.15.60:0/1593869228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c159950 0x7fc85c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 -- 172.21.15.60:0/1593869228 >> 172.21.15.60:0/1593869228 conn(0x7fc85c0ce6a0 msgr2=0x7fc85c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:35.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 -- 172.21.15.60:0/1593869228 shutdown_connections 2023-10-16T12:50:35.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.160+0000 7fc85b59e640 1 -- 172.21.15.60:0/1593869228 wait complete. 2023-10-16T12:50:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.162+0000 7fc85b59e640 1 Processor -- start 2023-10-16T12:50:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.162+0000 7fc85b59e640 1 -- start start 2023-10-16T12:50:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc85b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c06c810 0x7fc85c0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:35.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc85b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 0x7fc85c0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:35.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc85b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc85c159950 0x7fc85c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:35.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc85b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0b3890 con 0x7fc85c070450 2023-10-16T12:50:35.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc85b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0b2c80 con 0x7fc85c159950 2023-10-16T12:50:35.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc85b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0b2f60 con 0x7fc85c06c810 2023-10-16T12:50:35.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.163+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 0x7fc85c0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:35.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 0x7fc85c0acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51828/0 (socket says 172.21.15.60:51828) 2023-10-16T12:50:35.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 -- 172.21.15.60:0/2203440506 learned_addr learned my addr 172.21.15.60:0/2203440506 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:35.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 -- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c06c810 msgr2=0x7fc85c0b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:35.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c06c810 0x7fc85c0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 -- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc85c159950 msgr2=0x7fc85c0b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:35.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc85c159950 0x7fc85c0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 -- 172.21.15.60:0/2203440506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85400bd40 con 0x7fc85c070450 2023-10-16T12:50:35.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.164+0000 7fc851d9b640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 0x7fc85c0acb00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc84400c3e0 tx=0x7fc84400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:35.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.165+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8440074e0 con 0x7fc85c070450 2023-10-16T12:50:35.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.165+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc844007640 con 0x7fc85c070450 2023-10-16T12:50:35.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.165+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8440049e0 con 0x7fc85c070450 2023-10-16T12:50:35.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.165+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c0b3240 con 0x7fc85c070450 2023-10-16T12:50:35.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.165+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c0ad040 con 0x7fc85c070450 2023-10-16T12:50:35.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.166+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c0ad200 con 0x7fc85c070450 2023-10-16T12:50:35.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.167+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc844004b40 con 0x7fc85c070450 2023-10-16T12:50:35.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.168+0000 7fc8537fe640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc83c06af90 0x7fc83c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:35.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.168+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc844089270 con 0x7fc85c070450 2023-10-16T12:50:35.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.169+0000 7fc85a59c640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc83c06af90 0x7fc83c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:35.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.170+0000 7fc85a59c640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc83c06af90 0x7fc83c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0098f0 tx=0x7fc84c008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:35.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.172+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc844054130 con 0x7fc85c070450 2023-10-16T12:50:35.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.440+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fc85c0b39d0 con 0x7fc85c070450 2023-10-16T12:50:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.813+0000 7fc8537fe640 1 -- 172.21.15.60:0/2203440506 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v641) v1 ==== 166+0+0 (secure 0 0 0) 0x7fc85c0b39d0 con 0x7fc85c070450 2023-10-16T12:50:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.816+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc83c06af90 msgr2=0x7fc83c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.816+0000 7fc85b59e640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc83c06af90 0x7fc83c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0098f0 tx=0x7fc84c008040 comp rx=0 tx=0).stop 2023-10-16T12:50:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.816+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 msgr2=0x7fc85c0acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.816+0000 7fc85b59e640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 0x7fc85c0acb00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc84400c3e0 tx=0x7fc84400c8a0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 shutdown_connections 2023-10-16T12:50:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc83c06af90 0x7fc83c06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc85c159950 0x7fc85c0b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc85c070450 0x7fc85c0acb00 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 --2- 172.21.15.60:0/2203440506 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc85c06c810 0x7fc85c0b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 >> 172.21.15.60:0/2203440506 conn(0x7fc85c0ce6a0 msgr2=0x7fc85c0c97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 shutdown_connections 2023-10-16T12:50:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:35.817+0000 7fc85b59e640 1 -- 172.21.15.60:0/2203440506 wait complete. 2023-10-16T12:50:35.843 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-16T12:50:35.844 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-16T12:50:35.844 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:50:35.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.075+0000 7fbb0fd33640 1 Processor -- start 2023-10-16T12:50:36.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.075+0000 7fbb0fd33640 1 -- start start 2023-10-16T12:50:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.077+0000 7fbb0fd33640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbb080ce390 0x7fbb0815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.077+0000 7fbb0fd33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0815ed60 con 0x7fbb0815ae80 2023-10-16T12:50:36.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.077+0000 7fbb0fd33640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080707e0 con 0x7fbb0815aaa0 2023-10-16T12:50:36.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.077+0000 7fbb0fd33640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0806b810 con 0x7fbb080ce390 2023-10-16T12:50:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.078+0000 7fbb0d2ce640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbb0815ae80 0x7fbb08070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44726/0 (socket says 172.21.15.60:44726) 2023-10-16T12:50:36.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.078+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/1829711240 learned_addr learned my addr 172.21.15.60:0/1829711240 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.078+0000 7fbb0e2d0640 1 --2- 172.21.15.60:0/1829711240 >> v2:172.21.15.144:3300/0 conn(0x7fbb080ce390 0x7fbb0815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.078+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944169546 0 0) 0x7fbb0815ed60 con 0x7fbb0815ae80 2023-10-16T12:50:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.078+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf4003520 con 0x7fbb0815ae80 2023-10-16T12:50:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473854783 0 0) 0x7fbaf4003520 con 0x7fbb0815ae80 2023-10-16T12:50:36.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 >> v2:172.21.15.144:3300/0 conn(0x7fbb080ce390 msgr2=0x7fbb0815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 --2- 172.21.15.60:0/1829711240 >> v2:172.21.15.144:3300/0 conn(0x7fbb080ce390 0x7fbb0815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 >> v1:172.21.15.144:6789/0 conn(0x7fbb0815aaa0 legacy=0x7fbb080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0815ef90 con 0x7fbb0815ae80 2023-10-16T12:50:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fbaf8001670 con 0x7fbb0815ae80 2023-10-16T12:50:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 >> v1:172.21.15.60:6789/0 conn(0x7fbb0815ae80 legacy=0x7fbb08070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4003970 0x7fbaf4003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaf4004370 0x7fbaf4004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbaf4005040 0x7fbaf400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0806b810 con 0x7fbaf4004370 2023-10-16T12:50:36.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf400bfa0 con 0x7fbaf4005040 2023-10-16T12:50:36.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf400c160 con 0x7fbaf4003970 2023-10-16T12:50:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaf4004370 0x7fbaf4004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.079+0000 7fbb0dacf640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4003970 0x7fbaf4003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0e2d0640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbaf4005040 0x7fbaf400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4003970 msgr2=0x7fbaf4003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4003970 0x7fbaf4003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbaf4005040 msgr2=0x7fbaf400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbaf4005040 0x7fbaf400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/1829711240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf4003520 con 0x7fbaf4004370 2023-10-16T12:50:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.080+0000 7fbb0e2d0640 1 -- 172.21.15.60:0/1829711240 reap_dead start 2023-10-16T12:50:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.081+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaf4004370 0x7fbaf4004840 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fbb0815b2f0 tx=0x7fbaf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647b6c22b40498c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8030dd0 con 0x7fbaf4004370 2023-10-16T12:50:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf80058d0 con 0x7fbaf4004370 2023-10-16T12:50:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0cacd640 1 -- 172.21.15.60:0/1829711240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8005cc0 con 0x7fbaf4004370 2023-10-16T12:50:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0fd33640 1 -- 172.21.15.60:0/1829711240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaf4004370 msgr2=0x7fbaf4004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaf4004370 0x7fbaf4004840 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fbb0815b2f0 tx=0x7fbaf40057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0fd33640 1 -- 172.21.15.60:0/1829711240 shutdown_connections 2023-10-16T12:50:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.082+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/1829711240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbaf4004370 0x7fbaf4004840 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.083+0000 7fbb0fd33640 1 -- 172.21.15.60:0/1829711240 >> 172.21.15.60:0/1829711240 conn(0x7fbb080cb410 msgr2=0x7fbb080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.083+0000 7fbb0fd33640 1 -- 172.21.15.60:0/1829711240 shutdown_connections 2023-10-16T12:50:36.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.083+0000 7fbb0fd33640 1 -- 172.21.15.60:0/1829711240 wait complete. 2023-10-16T12:50:36.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.084+0000 7fbb0fd33640 1 Processor -- start 2023-10-16T12:50:36.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.085+0000 7fbb0fd33640 1 -- start start 2023-10-16T12:50:36.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0fd33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4004370 0x7fbb081f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0fd33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 0x7fbb080ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0fd33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb080af570 0x7fbb080af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0fd33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080aff30 con 0x7fbb080c60e0 2023-10-16T12:50:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0fd33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb081f1db0 con 0x7fbb080af570 2023-10-16T12:50:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0fd33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb081f2090 con 0x7fbaf4004370 2023-10-16T12:50:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 0x7fbb080ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 0x7fbb080ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51852/0 (socket says 172.21.15.60:51852) 2023-10-16T12:50:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/2003512807 learned_addr learned my addr 172.21.15.60:0/2003512807 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0dacf640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4004370 0x7fbb081f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4004370 msgr2=0x7fbb081f17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4004370 0x7fbb081f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb080af570 msgr2=0x7fbb080af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb080af570 0x7fbb080af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.086+0000 7fbb0d2ce640 1 -- 172.21.15.60:0/2003512807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb081f2370 con 0x7fbb080c60e0 2023-10-16T12:50:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.087+0000 7fbb0d2ce640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 0x7fbb080ca2b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8030770 tx=0x7fbaf8032070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.087+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8040040 con 0x7fbb080c60e0 2023-10-16T12:50:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.087+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8005230 con 0x7fbb080c60e0 2023-10-16T12:50:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.087+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80055a0 con 0x7fbb080c60e0 2023-10-16T12:50:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.087+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb081f2650 con 0x7fbb080c60e0 2023-10-16T12:50:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.087+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb080ca7f0 con 0x7fbb080c60e0 2023-10-16T12:50:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.088+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0804e160 con 0x7fbb080c60e0 2023-10-16T12:50:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.089+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbaf8036070 con 0x7fbb080c60e0 2023-10-16T12:50:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.090+0000 7fbafe7fc640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbaec06afc0 0x7fbaec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.090+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fbaf80af250 con 0x7fbb080c60e0 2023-10-16T12:50:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.091+0000 7fbb0dacf640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbaec06afc0 0x7fbaec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.092+0000 7fbb0dacf640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbaec06afc0 0x7fbaec06d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb08069740 tx=0x7fbb04009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.095+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbaf807a1a0 con 0x7fbb080c60e0 2023-10-16T12:50:36.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.296+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb080cb000 con 0x7fbb080c60e0 2023-10-16T12:50:36.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.298+0000 7fbafe7fc640 1 -- 172.21.15.60:0/2003512807 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+13992 (secure 0 0 0) 0x7fbaf807d7c0 con 0x7fbb080c60e0 2023-10-16T12:50:36.300 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:36.301 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:33.479641+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/4244730882":"2023-10-17T12:50:30.461577+0000","172.21.15.144:0/734835368":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/2831755110":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1942560574":"2023-10-17T12:50:26.243946+0000","172.21.15.144:6832/176900405":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1135964389":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/668026912":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/1468249085":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2269656792":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6832/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:6833/176900405":"2023-10-17T12:50:26.243946+0000","172.21.15.144:0/3435889746":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/2501905996":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/1930765872":"2023-10-17T12:50:20.071781+0000","172.21.15.60:6834/4244730882":"2023-10-17T12:50:30.461577+0000","172.21.15.144:0/2188695119":"2023-10-17T12:50:26.243946+0000","172.21.15.144:6833/1915751278":"2023-10-17T12:50:20.071781+0000","172.21.15.144:0/3359683879":"2023-10-17T12:50:20.071781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:36.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.300+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbaec06afc0 msgr2=0x7fbaec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.300+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbaec06afc0 0x7fbaec06d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb08069740 tx=0x7fbb04009040 comp rx=0 tx=0).stop 2023-10-16T12:50:36.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.300+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 msgr2=0x7fbb080ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.300+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 0x7fbb080ca2b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8030770 tx=0x7fbaf8032070 comp rx=0 tx=0).stop 2023-10-16T12:50:36.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 shutdown_connections 2023-10-16T12:50:36.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fbaec06afc0 0x7fbaec06d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb080af570 0x7fbb080af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb080c60e0 0x7fbb080ca2b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 --2- 172.21.15.60:0/2003512807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbaf4004370 0x7fbb081f17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 >> 172.21.15.60:0/2003512807 conn(0x7fbb080cb410 msgr2=0x7fbb080702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:36.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 shutdown_connections 2023-10-16T12:50:36.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.301+0000 7fbb0fd33640 1 -- 172.21.15.60:0/2003512807 wait complete. 2023-10-16T12:50:36.324 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:50:36.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:50:36.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.561+0000 7f907342d640 1 Processor -- start 2023-10-16T12:50:36.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.561+0000 7f907342d640 1 -- start start 2023-10-16T12:50:36.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f907342d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f906c06a4b0 0x7f906c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f907342d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f906c0c9ca0 0x7f906c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f907342d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906c0cb030 con 0x7f906c15ad70 2023-10-16T12:50:36.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f907342d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906c0cae30 con 0x7f906c06a4b0 2023-10-16T12:50:36.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f907342d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906c0c91c0 con 0x7f906c0c9ca0 2023-10-16T12:50:36.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f90709c8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f906c06a4b0 0x7f906c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f90709c8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f906c06a4b0 0x7f906c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51856/0 (socket says 172.21.15.60:51856) 2023-10-16T12:50:36.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f90709c8640 1 -- 172.21.15.60:0/700752154 learned_addr learned my addr 172.21.15.60:0/700752154 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:36.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f90719ca640 1 --2- 172.21.15.60:0/700752154 >> v2:172.21.15.144:3300/0 conn(0x7f906c0c9ca0 0x7f906c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f90709c8640 1 -- 172.21.15.60:0/700752154 >> v2:172.21.15.144:3300/0 conn(0x7f906c0c9ca0 msgr2=0x7f906c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:36.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.563+0000 7f90709c8640 1 --2- 172.21.15.60:0/700752154 >> v2:172.21.15.144:3300/0 conn(0x7f906c0c9ca0 0x7f906c0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.564+0000 7f90709c8640 1 -- 172.21.15.60:0/700752154 >> v1:172.21.15.144:6789/0 conn(0x7f906c15ad70 legacy=0x7f906c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:36.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.564+0000 7f90709c8640 1 -- 172.21.15.60:0/700752154 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c0c94a0 con 0x7f906c06a4b0 2023-10-16T12:50:36.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.565+0000 7f90709c8640 1 --2- 172.21.15.60:0/700752154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0ca3b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f906c06f270 tx=0x7f905c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f8b07a72325c22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:36.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.565+0000 7f9063fff640 1 -- 172.21.15.60:0/700752154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c031870 con 0x7f906c06a4b0 2023-10-16T12:50:36.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.565+0000 7f9063fff640 1 -- 172.21.15.60:0/700752154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c0319d0 con 0x7f906c06a4b0 2023-10-16T12:50:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.565+0000 7f9063fff640 1 -- 172.21.15.60:0/700752154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c031d60 con 0x7f906c06a4b0 2023-10-16T12:50:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 -- 172.21.15.60:0/700752154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 msgr2=0x7f906c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 --2- 172.21.15.60:0/700752154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0ca3b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f906c06f270 tx=0x7f905c02b910 comp rx=0 tx=0).stop 2023-10-16T12:50:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 -- 172.21.15.60:0/700752154 shutdown_connections 2023-10-16T12:50:36.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 --2- 172.21.15.60:0/700752154 >> v2:172.21.15.144:3300/0 conn(0x7f906c0c9ca0 0x7f906c0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 --2- 172.21.15.60:0/700752154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0ca3b0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 -- 172.21.15.60:0/700752154 >> 172.21.15.60:0/700752154 conn(0x7f906c0ce9b0 msgr2=0x7f906c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:36.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 -- 172.21.15.60:0/700752154 shutdown_connections 2023-10-16T12:50:36.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.566+0000 7f907342d640 1 -- 172.21.15.60:0/700752154 wait complete. 2023-10-16T12:50:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.568+0000 7f907342d640 1 Processor -- start 2023-10-16T12:50:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.568+0000 7f907342d640 1 -- start start 2023-10-16T12:50:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f907342d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f907342d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c0c9ca0 0x7f906c0c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f907342d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f906c15ad70 0x7f906c0c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f907342d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c0c3a80 con 0x7f906c06a4b0 2023-10-16T12:50:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f907342d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c0c2bd0 con 0x7f906c0c9ca0 2023-10-16T12:50:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f907342d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906c0c56c0 con 0x7f906c15ad70 2023-10-16T12:50:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51868/0 (socket says 172.21.15.60:51868) 2023-10-16T12:50:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 -- 172.21.15.60:0/599500897 learned_addr learned my addr 172.21.15.60:0/599500897 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 -- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f906c15ad70 msgr2=0x7f906c0c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f906c15ad70 0x7f906c0c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90709c8640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c0c9ca0 0x7f906c0c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 -- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c0c9ca0 msgr2=0x7f906c0c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.569+0000 7f90711c9640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c0c9ca0 0x7f906c0c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f90711c9640 1 -- 172.21.15.60:0/599500897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c0094d0 con 0x7f906c06a4b0 2023-10-16T12:50:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f90711c9640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0c2690 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f906800b420 tx=0x7f906800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068011830 con 0x7f906c06a4b0 2023-10-16T12:50:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068011990 con 0x7f906c06a4b0 2023-10-16T12:50:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068011d40 con 0x7f906c06a4b0 2023-10-16T12:50:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906c0c3ea0 con 0x7f906c06a4b0 2023-10-16T12:50:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.570+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906c0c30e0 con 0x7f906c06a4b0 2023-10-16T12:50:36.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.572+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906c0c3bc0 con 0x7f906c06a4b0 2023-10-16T12:50:36.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.572+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f906801b460 con 0x7f906c06a4b0 2023-10-16T12:50:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.573+0000 7f9061ffb640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f905006af90 0x7f905006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.573+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f90680892a0 con 0x7f906c06a4b0 2023-10-16T12:50:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.574+0000 7f90709c8640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f905006af90 0x7f905006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.575+0000 7f90709c8640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f905006af90 0x7f905006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f905c036000 tx=0x7f905c03c040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:36.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.578+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9068054280 con 0x7f906c06a4b0 2023-10-16T12:50:36.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.772+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f906c04e160 con 0x7f906c06a4b0 2023-10-16T12:50:36.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.773+0000 7f9061ffb640 1 -- 172.21.15.60:0/599500897 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90680578a0 con 0x7f906c06a4b0 2023-10-16T12:50:36.776 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:50:36.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f905006af90 msgr2=0x7f905006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f905006af90 0x7f905006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f905c036000 tx=0x7f905c03c040 comp rx=0 tx=0).stop 2023-10-16T12:50:36.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 msgr2=0x7f906c0c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:36.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0c2690 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f906800b420 tx=0x7f906800b8e0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 shutdown_connections 2023-10-16T12:50:36.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f905006af90 0x7f905006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f906c15ad70 0x7f906c0c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f906c0c9ca0 0x7f906c0c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 --2- 172.21.15.60:0/599500897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f906c06a4b0 0x7f906c0c2690 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:36.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 >> 172.21.15.60:0/599500897 conn(0x7f906c0ce9b0 msgr2=0x7f906c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:36.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 shutdown_connections 2023-10-16T12:50:36.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:36.776+0000 7f907342d640 1 -- 172.21.15.60:0/599500897 wait complete. 2023-10-16T12:50:36.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T12:50:37.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.029+0000 7fc7a1ada640 1 Processor -- start 2023-10-16T12:50:37.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.029+0000 7fc7a1ada640 1 -- start start 2023-10-16T12:50:37.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.031+0000 7fc7a1ada640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc79c0cd930 0x7fc79c164240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.031+0000 7fc7a1ada640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c159790 con 0x7fc79c1593b0 2023-10-16T12:50:37.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.031+0000 7fc7a1ada640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c0c9b20 con 0x7fc79c0caa10 2023-10-16T12:50:37.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.031+0000 7fc7a1ada640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c070370 con 0x7fc79c0cd930 2023-10-16T12:50:37.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.031+0000 7fc79b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc79c1593b0 0x7fc79c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44736/0 (socket says 172.21.15.60:44736) 2023-10-16T12:50:37.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.031+0000 7fc79b7fe640 1 -- 172.21.15.60:0/842400890 learned_addr learned my addr 172.21.15.60:0/842400890 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79bfff640 1 --2- 172.21.15.60:0/842400890 >> v2:172.21.15.144:3300/0 conn(0x7fc79c0cd930 0x7fc79c164240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084084853 0 0) 0x7fc79c159790 con 0x7fc79c1593b0 2023-10-16T12:50:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc788003520 con 0x7fc79c1593b0 2023-10-16T12:50:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183642415 0 0) 0x7fc788003520 con 0x7fc79c1593b0 2023-10-16T12:50:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 >> v2:172.21.15.144:3300/0 conn(0x7fc79c0cd930 msgr2=0x7fc79c164240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 --2- 172.21.15.60:0/842400890 >> v2:172.21.15.144:3300/0 conn(0x7fc79c0cd930 0x7fc79c164240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 >> v1:172.21.15.144:6789/0 conn(0x7fc79c0caa10 legacy=0x7fc79c06fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c0c9e00 con 0x7fc79c1593b0 2023-10-16T12:50:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc78c001670 con 0x7fc79c1593b0 2023-10-16T12:50:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 >> v1:172.21.15.60:6789/0 conn(0x7fc79c1593b0 legacy=0x7fc79c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc788003970 0x7fc788003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7880047c0 0x7fc788004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7880052a0 0x7fc78800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c070370 con 0x7fc7880052a0 2023-10-16T12:50:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78800e380 con 0x7fc7880047c0 2023-10-16T12:50:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.032+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78800e540 con 0x7fc788003970 2023-10-16T12:50:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79bfff640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7880052a0 0x7fc78800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc792ffd640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7880047c0 0x7fc788004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79b7fe640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc788003970 0x7fc788003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79bfff640 1 -- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc788003970 msgr2=0x7fc788003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79bfff640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc788003970 0x7fc788003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79bfff640 1 -- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7880047c0 msgr2=0x7fc788004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79bfff640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7880047c0 0x7fc788004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79bfff640 1 -- 172.21.15.60:0/842400890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc788003520 con 0x7fc7880052a0 2023-10-16T12:50:37.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.033+0000 7fc79b7fe640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc788003970 0x7fc788003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:37.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.034+0000 7fc79bfff640 1 -- 172.21.15.60:0/842400890 reap_dead start 2023-10-16T12:50:37.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.034+0000 7fc79bfff640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7880052a0 0x7fc78800db90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0cddc0 tx=0x7fc7880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16df8e5b027a5f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:37.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79402cbd0 con 0x7fc7880052a0 2023-10-16T12:50:37.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79400f040 con 0x7fc7880052a0 2023-10-16T12:50:37.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc79affd640 1 -- 172.21.15.60:0/842400890 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794003990 con 0x7fc7880052a0 2023-10-16T12:50:37.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 -- 172.21.15.60:0/842400890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7880052a0 msgr2=0x7fc78800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7880052a0 0x7fc78800db90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0cddc0 tx=0x7fc7880057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 -- 172.21.15.60:0/842400890 shutdown_connections 2023-10-16T12:50:37.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/842400890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7880052a0 0x7fc78800db90 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 -- 172.21.15.60:0/842400890 >> 172.21.15.60:0/842400890 conn(0x7fc79c0cb410 msgr2=0x7fc79c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:37.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 -- 172.21.15.60:0/842400890 shutdown_connections 2023-10-16T12:50:37.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.035+0000 7fc7a1ada640 1 -- 172.21.15.60:0/842400890 wait complete. 2023-10-16T12:50:37.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.037+0000 7fc7a1ada640 1 Processor -- start 2023-10-16T12:50:37.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.037+0000 7fc7a1ada640 1 -- start start 2023-10-16T12:50:37.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.038+0000 7fc7a1ada640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7880052a0 0x7fc79c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.038+0000 7fc7a1ada640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 0x7fc79c15b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.038+0000 7fc7a1ada640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc79c0ac890 0x7fc79c0acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.038+0000 7fc7a1ada640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0ad250 con 0x7fc79c15b420 2023-10-16T12:50:37.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.038+0000 7fc7a1ada640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c15bdc0 con 0x7fc79c0ac890 2023-10-16T12:50:37.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.038+0000 7fc7a1ada640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0c5a60 con 0x7fc7880052a0 2023-10-16T12:50:37.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 0x7fc79c15b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 0x7fc79c15b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51898/0 (socket says 172.21.15.60:51898) 2023-10-16T12:50:37.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 -- 172.21.15.60:0/2994928169 learned_addr learned my addr 172.21.15.60:0/2994928169 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:37.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc79b7fe640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7880052a0 0x7fc79c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 -- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7880052a0 msgr2=0x7fc79c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc79bfff640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc79c0ac890 0x7fc79c0acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7880052a0 0x7fc79c0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 -- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc79c0ac890 msgr2=0x7fc79c0acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc79c0ac890 0x7fc79c0acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 -- 172.21.15.60:0/2994928169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c0c86b0 con 0x7fc79c15b420 2023-10-16T12:50:37.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.039+0000 7fc792ffd640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 0x7fc79c15b880 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc78400e530 tx=0x7fc78400e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:37.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.040+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7840052e0 con 0x7fc79c15b420 2023-10-16T12:50:37.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.040+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc784005440 con 0x7fc79c15b420 2023-10-16T12:50:37.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.040+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784003810 con 0x7fc79c15b420 2023-10-16T12:50:37.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.040+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c15bfe0 con 0x7fc79c15b420 2023-10-16T12:50:37.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.040+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c0c5f90 con 0x7fc79c15b420 2023-10-16T12:50:37.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.041+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c0cf830 con 0x7fc79c15b420 2023-10-16T12:50:37.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.042+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc784003970 con 0x7fc79c15b420 2023-10-16T12:50:37.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.043+0000 7fc798ff9640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc77c06af90 0x7fc77c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:37.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.044+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fc7840894a0 con 0x7fc79c15b420 2023-10-16T12:50:37.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.044+0000 7fc79b7fe640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc77c06af90 0x7fc77c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:37.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.046+0000 7fc79b7fe640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc77c06af90 0x7fc77c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc78c001200 tx=0x7fc78c009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:37.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.048+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc784054400 con 0x7fc79c15b420 2023-10-16T12:50:37.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.316+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fc79c04e160 con 0x7fc79c15b420 2023-10-16T12:50:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.826+0000 7fc798ff9640 1 -- 172.21.15.60:0/2994928169 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v643) v1 ==== 186+0+0 (secure 0 0 0) 0x7fc784057a20 con 0x7fc79c15b420 2023-10-16T12:50:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc77c06af90 msgr2=0x7fc77c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc77c06af90 0x7fc77c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc78c001200 tx=0x7fc78c009040 comp rx=0 tx=0).stop 2023-10-16T12:50:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 msgr2=0x7fc79c15b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 0x7fc79c15b880 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc78400e530 tx=0x7fc78400e9f0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 shutdown_connections 2023-10-16T12:50:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc77c06af90 0x7fc77c06d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc79c0ac890 0x7fc79c0acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79c15b420 0x7fc79c15b880 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 --2- 172.21.15.60:0/2994928169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7880052a0 0x7fc79c0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 >> 172.21.15.60:0/2994928169 conn(0x7fc79c0cb410 msgr2=0x7fc79c0aea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 shutdown_connections 2023-10-16T12:50:37.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:37.830+0000 7fc7a1ada640 1 -- 172.21.15.60:0/2994928169 wait complete. 2023-10-16T12:50:37.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:50:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.079+0000 7f26bec44640 1 Processor -- start 2023-10-16T12:50:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.079+0000 7f26bec44640 1 -- start start 2023-10-16T12:50:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.080+0000 7f26bec44640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f26b815d330 0x7f26b81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26bec44640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b80d0890 con 0x7f26b806c7d0 2023-10-16T12:50:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26bec44640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b80d05b0 con 0x7f26b80661c0 2023-10-16T12:50:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26bec44640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b806b1e0 con 0x7f26b815d330 2023-10-16T12:50:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26b7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f26b806c7d0 0x7f26b8065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44742/0 (socket says 172.21.15.60:44742) 2023-10-16T12:50:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26b7fff640 1 -- 172.21.15.60:0/2340879997 learned_addr learned my addr 172.21.15.60:0/2340879997 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26bc9e0640 1 --2- 172.21.15.60:0/2340879997 >> v2:172.21.15.144:3300/0 conn(0x7f26b815d330 0x7f26b81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332902180 0 0) 0x7f26b80d0890 con 0x7f26b806c7d0 2023-10-16T12:50:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.081+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a8003520 con 0x7f26b806c7d0 2023-10-16T12:50:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458060548 0 0) 0x7f26a8003520 con 0x7f26b806c7d0 2023-10-16T12:50:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 >> v2:172.21.15.144:3300/0 conn(0x7f26b815d330 msgr2=0x7f26b81618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 --2- 172.21.15.60:0/2340879997 >> v2:172.21.15.144:3300/0 conn(0x7f26b815d330 0x7f26b81618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 >> v1:172.21.15.144:6789/0 conn(0x7f26b80661c0 legacy=0x7f26b80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8160dd0 con 0x7f26b806c7d0 2023-10-16T12:50:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f26a4001670 con 0x7f26b806c7d0 2023-10-16T12:50:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 >> v1:172.21.15.60:6789/0 conn(0x7f26b806c7d0 legacy=0x7f26b8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80039c0 0x7f26a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26a80047c0 0x7f26a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26a80051d0 0x7f26a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b806b1e0 con 0x7f26a80051d0 2023-10-16T12:50:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a800c120 con 0x7f26a80039c0 2023-10-16T12:50:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a800c2e0 con 0x7f26a80047c0 2023-10-16T12:50:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.082+0000 7f26bd1e1640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26a80051d0 0x7f26a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26bc9e0640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80039c0 0x7f26a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26b7fff640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26a80047c0 0x7f26a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26bd1e1640 1 -- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26a80047c0 msgr2=0x7f26a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26bd1e1640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26a80047c0 0x7f26a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26bd1e1640 1 -- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80039c0 msgr2=0x7f26a8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26bd1e1640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80039c0 0x7f26a8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26bd1e1640 1 -- 172.21.15.60:0/2340879997 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a8003520 con 0x7f26a80051d0 2023-10-16T12:50:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.083+0000 7f26b7fff640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26a80047c0 0x7f26a8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.084+0000 7f26bd1e1640 1 -- 172.21.15.60:0/2340879997 reap_dead start 2023-10-16T12:50:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.084+0000 7f26bd1e1640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26a80051d0 0x7f26a800b980 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f26b0001e20 tx=0x7f26a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a55484d20cc0b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.084+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0003b70 con 0x7f26a80051d0 2023-10-16T12:50:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.084+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b0003cd0 con 0x7f26a80051d0 2023-10-16T12:50:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.084+0000 7f26b77fe640 1 -- 172.21.15.60:0/2340879997 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0005310 con 0x7f26a80051d0 2023-10-16T12:50:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 -- 172.21.15.60:0/2340879997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26a80051d0 msgr2=0x7f26a800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26a80051d0 0x7f26a800b980 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f26b0001e20 tx=0x7f26a80057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 -- 172.21.15.60:0/2340879997 shutdown_connections 2023-10-16T12:50:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 --2- 172.21.15.60:0/2340879997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26a80051d0 0x7f26a800b980 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 -- 172.21.15.60:0/2340879997 >> 172.21.15.60:0/2340879997 conn(0x7f26b80d0ab0 msgr2=0x7f26b80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 -- 172.21.15.60:0/2340879997 shutdown_connections 2023-10-16T12:50:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.085+0000 7f26bec44640 1 -- 172.21.15.60:0/2340879997 wait complete. 2023-10-16T12:50:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.087+0000 7f26bec44640 1 Processor -- start 2023-10-16T12:50:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.087+0000 7f26bec44640 1 -- start start 2023-10-16T12:50:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bec44640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80051d0 0x7f26b815bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bec44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 0x7f26b80cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bec44640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b80ae770 0x7f26b80aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bec44640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b81f2b50 con 0x7f26b8159db0 2023-10-16T12:50:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bec44640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80af130 con 0x7f26a80051d0 2023-10-16T12:50:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bec44640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80cc9d0 con 0x7f26b80ae770 2023-10-16T12:50:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 0x7f26b80cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 0x7f26b80cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51910/0 (socket says 172.21.15.60:51910) 2023-10-16T12:50:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 -- 172.21.15.60:0/88980716 learned_addr learned my addr 172.21.15.60:0/88980716 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bc9e0640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80051d0 0x7f26b815bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26bd1e1640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b80ae770 0x7f26b80aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 -- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b80ae770 msgr2=0x7f26b80aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b80ae770 0x7f26b80aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 -- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80051d0 msgr2=0x7f26b815bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80051d0 0x7f26b815bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.088+0000 7f26b7fff640 1 -- 172.21.15.60:0/88980716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b815a320 con 0x7f26b8159db0 2023-10-16T12:50:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26bc9e0640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80051d0 0x7f26b815bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26b7fff640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 0x7f26b80cc490 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f26a400d7a0 tx=0x7f26a400dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40072d0 con 0x7f26b8159db0 2023-10-16T12:50:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4007430 con 0x7f26b8159db0 2023-10-16T12:50:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40047e0 con 0x7f26b8159db0 2023-10-16T12:50:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b815a600 con 0x7f26b8159db0 2023-10-16T12:50:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.089+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b815abc0 con 0x7f26b8159db0 2023-10-16T12:50:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.090+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b81f2cc0 con 0x7f26b8159db0 2023-10-16T12:50:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.091+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f26a4004940 con 0x7f26b8159db0 2023-10-16T12:50:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.093+0000 7f26b57fa640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f269c06afc0 0x7f269c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.093+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f26a40890a0 con 0x7f26b8159db0 2023-10-16T12:50:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.094+0000 7f26bc9e0640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f269c06afc0 0x7f269c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.095+0000 7f26bc9e0640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f269c06afc0 0x7f269c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f26b80b2db0 tx=0x7f26ac008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:38.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.097+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26b81f2cc0 con 0x7f26b8159db0 2023-10-16T12:50:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.300+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f26b8161390 con 0x7f26b8159db0 2023-10-16T12:50:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.307+0000 7f26b57fa640 1 -- 172.21.15.60:0/88980716 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f26a4053f70 con 0x7f26b8159db0 2023-10-16T12:50:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f269c06afc0 msgr2=0x7f269c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f269c06afc0 0x7f269c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f26b80b2db0 tx=0x7f26ac008040 comp rx=0 tx=0).stop 2023-10-16T12:50:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 msgr2=0x7f26b80cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:38.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 0x7f26b80cc490 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f26a400d7a0 tx=0x7f26a400dba0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 shutdown_connections 2023-10-16T12:50:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f269c06afc0 0x7f269c06d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b80ae770 0x7f26b80aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b8159db0 0x7f26b80cc490 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 --2- 172.21.15.60:0/88980716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26a80051d0 0x7f26b815bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:38.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 >> 172.21.15.60:0/88980716 conn(0x7f26b80d0ab0 msgr2=0x7f26b815c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:38.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.309+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 shutdown_connections 2023-10-16T12:50:38.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:38.310+0000 7f26bec44640 1 -- 172.21.15.60:0/88980716 wait complete. 2023-10-16T12:50:38.331 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:50:38.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:50:38.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:50:38.572 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:50:38.573 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:50:38.573 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:38.637 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:50:38.638 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:50:38.638 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:38.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:50:38.696 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:50:38.697 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:38.762 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:50:38.762 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:50:38.762 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:50:38.821 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:50:38.822 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:50:38.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:39.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.055+0000 7f14a914e640 1 Processor -- start 2023-10-16T12:50:39.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.055+0000 7f14a914e640 1 -- start start 2023-10-16T12:50:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a914e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14a406c040 0x7f14a406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a914e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 0x7f14a415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a914e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a406c960 con 0x7f14a415ebc0 2023-10-16T12:50:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a914e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a415b9d0 con 0x7f14a406c040 2023-10-16T12:50:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a914e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415bcb0 con 0x7f14a40ad970 2023-10-16T12:50:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 0x7f14a415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 0x7f14a415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:46646/0 (socket says 172.21.15.60:46646) 2023-10-16T12:50:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14a406c040 0x7f14a406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:39.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 -- 172.21.15.60:0/2169100018 learned_addr learned my addr 172.21.15.60:0/2169100018 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:39.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 -- 172.21.15.60:0/2169100018 >> v2:172.21.15.144:3300/0 conn(0x7f14a406c040 msgr2=0x7f14a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 --2- 172.21.15.60:0/2169100018 >> v2:172.21.15.144:3300/0 conn(0x7f14a406c040 0x7f14a406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 -- 172.21.15.60:0/2169100018 >> v1:172.21.15.144:6789/0 conn(0x7f14a415ebc0 legacy=0x7f14a40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.057+0000 7f14a37fe640 1 -- 172.21.15.60:0/2169100018 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a415bf90 con 0x7f14a40ad970 2023-10-16T12:50:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.059+0000 7f14a37fe640 1 --2- 172.21.15.60:0/2169100018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 0x7f14a415e680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14980098a0 tx=0x7f149802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0b7dd486b65770a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:39.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.059+0000 7f14a2ffd640 1 -- 172.21.15.60:0/2169100018 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498030840 con 0x7f14a40ad970 2023-10-16T12:50:39.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.059+0000 7f14a2ffd640 1 -- 172.21.15.60:0/2169100018 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14980309a0 con 0x7f14a40ad970 2023-10-16T12:50:39.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.059+0000 7f14a2ffd640 1 -- 172.21.15.60:0/2169100018 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498030d30 con 0x7f14a40ad970 2023-10-16T12:50:39.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 -- 172.21.15.60:0/2169100018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 msgr2=0x7f14a415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:39.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 --2- 172.21.15.60:0/2169100018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 0x7f14a415e680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14980098a0 tx=0x7f149802b980 comp rx=0 tx=0).stop 2023-10-16T12:50:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 -- 172.21.15.60:0/2169100018 shutdown_connections 2023-10-16T12:50:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 --2- 172.21.15.60:0/2169100018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a40ad970 0x7f14a415e680 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 --2- 172.21.15.60:0/2169100018 >> v2:172.21.15.144:3300/0 conn(0x7f14a406c040 0x7f14a406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 -- 172.21.15.60:0/2169100018 >> 172.21.15.60:0/2169100018 conn(0x7f14a40ce6a0 msgr2=0x7f14a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 -- 172.21.15.60:0/2169100018 shutdown_connections 2023-10-16T12:50:39.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.060+0000 7f14a914e640 1 -- 172.21.15.60:0/2169100018 wait complete. 2023-10-16T12:50:39.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 Processor -- start 2023-10-16T12:50:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 -- start start 2023-10-16T12:50:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14a406c040 0x7f14a40b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 0x7f14a415d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a415ebc0 0x7f14a415d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a4149830 con 0x7f14a40ad970 2023-10-16T12:50:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40ae0c0 con 0x7f14a406c040 2023-10-16T12:50:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.063+0000 7f14a914e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40ae960 con 0x7f14a415ebc0 2023-10-16T12:50:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 0x7f14a415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 0x7f14a415d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51382/0 (socket says 172.21.15.60:51382) 2023-10-16T12:50:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 -- 172.21.15.60:0/3522007490 learned_addr learned my addr 172.21.15.60:0/3522007490 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a3fff640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14a406c040 0x7f14a40b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a894d640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a415ebc0 0x7f14a415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 -- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a415ebc0 msgr2=0x7f14a415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a415ebc0 0x7f14a415d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 -- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14a406c040 msgr2=0x7f14a40b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14a406c040 0x7f14a40b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a37fe640 1 -- 172.21.15.60:0/3522007490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980094d0 con 0x7f14a40ad970 2023-10-16T12:50:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.064+0000 7f14a894d640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a415ebc0 0x7f14a415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.065+0000 7f14a37fe640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 0x7f14a415d350 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f14980099c0 tx=0x7f1498004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.065+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498039a90 con 0x7f14a40ad970 2023-10-16T12:50:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.065+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1498039c10 con 0x7f14a40ad970 2023-10-16T12:50:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.065+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498043410 con 0x7f14a40ad970 2023-10-16T12:50:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.065+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a40c76b0 con 0x7f14a40ad970 2023-10-16T12:50:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.065+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a40ae5f0 con 0x7f14a40ad970 2023-10-16T12:50:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.066+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a406b330 con 0x7f14a40ad970 2023-10-16T12:50:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.067+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f149804c3b0 con 0x7f14a40ad970 2023-10-16T12:50:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.068+0000 7f14a0ff9640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f149006af90 0x7f149006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.068+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f14980abd50 con 0x7f14a40ad970 2023-10-16T12:50:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.069+0000 7f14a3fff640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f149006af90 0x7f149006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.070+0000 7f14a3fff640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f149006af90 0x7f149006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f14a40c8100 tx=0x7f1494008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.073+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1498076b90 con 0x7f14a40ad970 2023-10-16T12:50:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.306+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14a404e9b0 con 0x7f14a40ad970 2023-10-16T12:50:39.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.307+0000 7f14a0ff9640 1 -- 172.21.15.60:0/3522007490 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f1498048090 con 0x7f14a40ad970 2023-10-16T12:50:39.310 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:39.310 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5558,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2608524245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2608524245},{"type":"v1","addr":"172.21.15.144:6835","nonce":2608524245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":5561,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/3590720178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":3590720178},{"type":"v1","addr":"172.21.15.144:6837","nonce":3590720178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5567,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2458991037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2458991037},{"type":"v1","addr":"172.21.15.60:6833","nonce":2458991037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":7269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2401112190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2401112190},{"type":"v1","addr":"172.21.15.60:6835","nonce":2401112190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[]} 2023-10-16T12:50:39.311 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 45 2023-10-16T12:50:39.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f149006af90 msgr2=0x7f149006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:39.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f149006af90 0x7f149006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f14a40c8100 tx=0x7f1494008040 comp rx=0 tx=0).stop 2023-10-16T12:50:39.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 msgr2=0x7f14a415d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:39.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 0x7f14a415d350 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f14980099c0 tx=0x7f1498004730 comp rx=0 tx=0).stop 2023-10-16T12:50:39.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 shutdown_connections 2023-10-16T12:50:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f149006af90 0x7f149006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14a415ebc0 0x7f14a415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14a40ad970 0x7f14a415d350 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 --2- 172.21.15.60:0/3522007490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14a406c040 0x7f14a40b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:39.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 >> 172.21.15.60:0/3522007490 conn(0x7f14a40ce6a0 msgr2=0x7f14a40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:39.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 shutdown_connections 2023-10-16T12:50:39.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:39.311+0000 7f14a914e640 1 -- 172.21.15.60:0/3522007490 wait complete. 2023-10-16T12:50:39.334 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:50:39.334 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:50:39.334 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:50:39.335 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:50:39.335 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:50:39.335 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:50:39.335 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:50:39.336 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:50:39.336 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:50:39.336 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:50:39.336 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:50:39.337 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:50:39.337 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:50:39.337 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:50:39.337 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:50:39.338 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:50:39.353 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:50:39.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:50:39.356 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:50:39.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:50:39.471 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:50:39.472 DEBUG:teuthology.parallel:result is None 2023-10-16T12:50:39.478 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:50:39.479 DEBUG:teuthology.parallel:result is None 2023-10-16T12:50:39.648 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:50:39.649 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:50:45.458 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:50:45.459 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:50:45.460 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:50:45.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:50:45.468 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:50:45.469 DEBUG:teuthology.parallel:result is None 2023-10-16T12:50:45.475 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:50:45.475 DEBUG:teuthology.parallel:result is None 2023-10-16T12:50:45.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:50:45.632 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:50:45.634 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:50:45.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.715+0000 7fa6c48a2640 1 Processor -- start 2023-10-16T12:50:45.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.715+0000 7fa6c48a2640 1 -- start start 2023-10-16T12:50:45.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.717+0000 7fa6c48a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 0x7fa6c00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.717+0000 7fa6c48a2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c0070360 con 0x7fa6c0159e90 2023-10-16T12:50:45.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.717+0000 7fa6c48a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c00cd1f0 con 0x7fa6c006c140 2023-10-16T12:50:45.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.717+0000 7fa6c48a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015d890 con 0x7fa6c0159ab0 2023-10-16T12:50:45.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.717+0000 7fa6b7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa6c0159e90 0x7fa6c006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44608/0 (socket says 172.21.15.60:44608) 2023-10-16T12:50:45.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.717+0000 7fa6b7fff640 1 -- 172.21.15.60:0/3477077270 learned_addr learned my addr 172.21.15.60:0/3477077270 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:45.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6be59c640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 0x7fa6c00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964254755 0 0) 0x7fa6c0070360 con 0x7fa6c0159e90 2023-10-16T12:50:45.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac005730 con 0x7fa6c0159e90 2023-10-16T12:50:45.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928262741 0 0) 0x7fa6c00cd1f0 con 0x7fa6c006c140 2023-10-16T12:50:45.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0070360 con 0x7fa6c006c140 2023-10-16T12:50:45.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2260669225 0 0) 0x7fa6ac005730 con 0x7fa6c0159e90 2023-10-16T12:50:45.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 msgr2=0x7fa6c00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:45.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 0x7fa6c00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:45.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 >> v1:172.21.15.144:6789/0 conn(0x7fa6c006c140 legacy=0x7fa6c015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:45.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c00cd4d0 con 0x7fa6c0159e90 2023-10-16T12:50:45.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa6a8001670 con 0x7fa6c0159e90 2023-10-16T12:50:45.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 >> v1:172.21.15.60:6789/0 conn(0x7fa6c0159e90 legacy=0x7fa6c006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:45.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 0x7fa6ac005d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.718+0000 7fa6bdd9b640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6ac0062f0 0x7fa6ac0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6bdd9b640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6ac006f10 0x7fa6ac007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015d890 con 0x7fa6ac0062f0 2023-10-16T12:50:45.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac00ddf0 con 0x7fa6ac006f10 2023-10-16T12:50:45.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac00dfb0 con 0x7fa6c0159ab0 2023-10-16T12:50:45.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6b7fff640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6ac0062f0 0x7fa6ac0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6bed9d640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6ac006f10 0x7fa6ac007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6be59c640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 0x7fa6ac005d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6b7fff640 1 -- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 msgr2=0x7fa6ac005d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:45.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6b7fff640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6c0159ab0 0x7fa6ac005d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:45.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6b7fff640 1 -- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6ac006f10 msgr2=0x7fa6ac007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:45.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6b7fff640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6ac006f10 0x7fa6ac007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:45.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6b7fff640 1 -- 172.21.15.60:0/3477077270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac005730 con 0x7fa6ac0062f0 2023-10-16T12:50:45.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.719+0000 7fa6bed9d640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6ac006f10 0x7fa6ac007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.721+0000 7fa6b7fff640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6ac0062f0 0x7fa6ac0069d0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8007fa0 tx=0x7fa6ac007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4848f478ba31a513 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.721+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8005ad0 con 0x7fa6ac0062f0 2023-10-16T12:50:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.721+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8005c50 con 0x7fa6ac0062f0 2023-10-16T12:50:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.721+0000 7fa6bdd9b640 1 -- 172.21.15.60:0/3477077270 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8007070 con 0x7fa6ac0062f0 2023-10-16T12:50:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.721+0000 7fa6c48a2640 1 -- 172.21.15.60:0/3477077270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6ac0062f0 msgr2=0x7fa6ac0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.721+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/3477077270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6ac0062f0 0x7fa6ac0069d0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8007fa0 tx=0x7fa6ac007a00 comp rx=0 tx=0).stop 2023-10-16T12:50:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.722+0000 7fa6bed9d640 1 -- 172.21.15.60:0/3477077270 reap_dead start 2023-10-16T12:50:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.722+0000 7fa6c48a2640 1 -- 172.21.15.60:0/3477077270 shutdown_connections 2023-10-16T12:50:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.722+0000 7fa6c48a2640 1 -- 172.21.15.60:0/3477077270 >> 172.21.15.60:0/3477077270 conn(0x7fa6c00ce9b0 msgr2=0x7fa6c00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.722+0000 7fa6c48a2640 1 -- 172.21.15.60:0/3477077270 shutdown_connections 2023-10-16T12:50:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.722+0000 7fa6c48a2640 1 -- 172.21.15.60:0/3477077270 wait complete. 2023-10-16T12:50:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.724+0000 7fa6c48a2640 1 Processor -- start 2023-10-16T12:50:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.724+0000 7fa6c48a2640 1 -- start start 2023-10-16T12:50:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6c48a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6ac0062f0 0x7fa6c00c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6c48a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c00acb00 0x7fa6c0146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6c48a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 0x7fa6c01468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6c48a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0146e10 con 0x7fa6c00acf60 2023-10-16T12:50:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6c48a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c00c4db0 con 0x7fa6c00acb00 2023-10-16T12:50:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6c48a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0143790 con 0x7fa6ac0062f0 2023-10-16T12:50:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 0x7fa6c01468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 0x7fa6c01468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51414/0 (socket says 172.21.15.60:51414) 2023-10-16T12:50:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 -- 172.21.15.60:0/929850666 learned_addr learned my addr 172.21.15.60:0/929850666 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6be59c640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6ac0062f0 0x7fa6c00c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6b7fff640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c00acb00 0x7fa6c0146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 -- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6ac0062f0 msgr2=0x7fa6c00c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6ac0062f0 0x7fa6c00c5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 -- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c00acb00 msgr2=0x7fa6c0146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c00acb00 0x7fa6c0146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6bed9d640 1 -- 172.21.15.60:0/929850666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c00c4480 con 0x7fa6c00acf60 2023-10-16T12:50:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.725+0000 7fa6b7fff640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c00acb00 0x7fa6c0146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.726+0000 7fa6bed9d640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 0x7fa6c01468d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa6c00c24b0 tx=0x7fa6b800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.726+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8017070 con 0x7fa6c00acf60 2023-10-16T12:50:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.726+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b8006b50 con 0x7fa6c00acf60 2023-10-16T12:50:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.726+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c00c4760 con 0x7fa6c00acf60 2023-10-16T12:50:45.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.726+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8006d00 con 0x7fa6c00acf60 2023-10-16T12:50:45.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.726+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c01429e0 con 0x7fa6c00acf60 2023-10-16T12:50:45.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.727+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c004e160 con 0x7fa6c00acf60 2023-10-16T12:50:45.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.728+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa6b8003970 con 0x7fa6c00acf60 2023-10-16T12:50:45.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.729+0000 7fa6b6ffd640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fa6a006afc0 0x7fa6a006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:45.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.729+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fa6b8088f60 con 0x7fa6c00acf60 2023-10-16T12:50:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.730+0000 7fa6be59c640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fa6a006afc0 0x7fa6a006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.731+0000 7fa6be59c640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fa6a006afc0 0x7fa6a006d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009870 tx=0x7fa6b0008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.734+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6b8053e20 con 0x7fa6c00acf60 2023-10-16T12:50:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:45.952+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa6c00c4ef0 con 0x7fa6c00acf60 2023-10-16T12:50:46.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.901+0000 7fa6b6ffd640 1 -- 172.21.15.60:0/929850666 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v109) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa6b8006e60 con 0x7fa6c00acf60 2023-10-16T12:50:46.904 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:50:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.905+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fa6a006afc0 msgr2=0x7fa6a006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.905+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fa6a006afc0 0x7fa6a006d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009870 tx=0x7fa6b0008040 comp rx=0 tx=0).stop 2023-10-16T12:50:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.905+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 msgr2=0x7fa6c01468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:46.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.905+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 0x7fa6c01468d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa6c00c24b0 tx=0x7fa6b800eb90 comp rx=0 tx=0).stop 2023-10-16T12:50:46.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 shutdown_connections 2023-10-16T12:50:46.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fa6a006afc0 0x7fa6a006d480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:46.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6c00acf60 0x7fa6c01468d0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:46.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6c00acb00 0x7fa6c0146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:46.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 --2- 172.21.15.60:0/929850666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6ac0062f0 0x7fa6c00c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:46.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 >> 172.21.15.60:0/929850666 conn(0x7fa6c00ce9b0 msgr2=0x7fa6c00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:46.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 shutdown_connections 2023-10-16T12:50:46.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:46.906+0000 7fa6c48a2640 1 -- 172.21.15.60:0/929850666 wait complete. 2023-10-16T12:50:46.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:50:47.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.147+0000 7f93bf59e640 1 Processor -- start 2023-10-16T12:50:47.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.147+0000 7f93bf59e640 1 -- start start 2023-10-16T12:50:47.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93bf59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93c006c040 0x7f93c006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93bf59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93c00ad970 0x7f93c015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93bf59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c006c960 con 0x7f93c015ebc0 2023-10-16T12:50:47.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93bf59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c015b9d0 con 0x7f93c00ad970 2023-10-16T12:50:47.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93bf59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c015bcb0 con 0x7f93c006c040 2023-10-16T12:50:47.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93c00ad970 0x7f93c015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93c00ad970 0x7f93c015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51416/0 (socket says 172.21.15.60:51416) 2023-10-16T12:50:47.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 -- 172.21.15.60:0/1172731639 learned_addr learned my addr 172.21.15.60:0/1172731639 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:47.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 -- 172.21.15.60:0/1172731639 >> v2:172.21.15.144:3300/0 conn(0x7f93c006c040 msgr2=0x7f93c006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:47.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93be59c640 1 --2- 172.21.15.60:0/1172731639 >> v2:172.21.15.144:3300/0 conn(0x7f93c006c040 0x7f93c006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 --2- 172.21.15.60:0/1172731639 >> v2:172.21.15.144:3300/0 conn(0x7f93c006c040 0x7f93c006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 -- 172.21.15.60:0/1172731639 >> v1:172.21.15.144:6789/0 conn(0x7f93c015ebc0 legacy=0x7f93c00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:47.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.149+0000 7f93b7fff640 1 -- 172.21.15.60:0/1172731639 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c015bf90 con 0x7f93c00ad970 2023-10-16T12:50:47.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.151+0000 7f93b7fff640 1 --2- 172.21.15.60:0/1172731639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c015e680 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f93c00cf350 tx=0x7f93a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1f34808ee5bbc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:47.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.151+0000 7f93bdd9b640 1 -- 172.21.15.60:0/1172731639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8031870 con 0x7f93c00ad970 2023-10-16T12:50:47.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.151+0000 7f93bdd9b640 1 -- 172.21.15.60:0/1172731639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a80319d0 con 0x7f93c00ad970 2023-10-16T12:50:47.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.151+0000 7f93bdd9b640 1 -- 172.21.15.60:0/1172731639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8031d60 con 0x7f93c00ad970 2023-10-16T12:50:47.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 -- 172.21.15.60:0/1172731639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 msgr2=0x7f93c015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1172731639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c015e680 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f93c00cf350 tx=0x7f93a802b910 comp rx=0 tx=0).stop 2023-10-16T12:50:47.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 -- 172.21.15.60:0/1172731639 shutdown_connections 2023-10-16T12:50:47.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1172731639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c015e680 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1172731639 >> v2:172.21.15.144:3300/0 conn(0x7f93c006c040 0x7f93c006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 -- 172.21.15.60:0/1172731639 >> 172.21.15.60:0/1172731639 conn(0x7f93c00ce6a0 msgr2=0x7f93c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:47.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 -- 172.21.15.60:0/1172731639 shutdown_connections 2023-10-16T12:50:47.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.152+0000 7f93bf59e640 1 -- 172.21.15.60:0/1172731639 wait complete. 2023-10-16T12:50:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.154+0000 7f93bf59e640 1 Processor -- start 2023-10-16T12:50:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.154+0000 7f93bf59e640 1 -- start start 2023-10-16T12:50:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93bf59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93c006c040 0x7f93c014dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93bf59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c00c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93bf59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93c015ebc0 0x7f93c00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93bf59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c00ca5d0 con 0x7f93c00ad970 2023-10-16T12:50:47.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93bf59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c00c99c0 con 0x7f93c006c040 2023-10-16T12:50:47.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93bf59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c00c9ca0 con 0x7f93c015ebc0 2023-10-16T12:50:47.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c00c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c00c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51424/0 (socket says 172.21.15.60:51424) 2023-10-16T12:50:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93b7fff640 1 -- 172.21.15.60:0/1282203638 learned_addr learned my addr 172.21.15.60:0/1282203638 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.155+0000 7f93be59c640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93c006c040 0x7f93c014dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93bed9d640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93c015ebc0 0x7f93c00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b7fff640 1 -- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93c015ebc0 msgr2=0x7f93c00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b7fff640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93c015ebc0 0x7f93c00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b7fff640 1 -- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93c006c040 msgr2=0x7f93c014dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b7fff640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93c006c040 0x7f93c014dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b7fff640 1 -- 172.21.15.60:0/1282203638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a80094d0 con 0x7f93c00ad970 2023-10-16T12:50:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93bed9d640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93c015ebc0 0x7f93c00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:47.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93be59c640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93c006c040 0x7f93c014dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:47.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b7fff640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c00c9050 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f93a8000c00 tx=0x7f93a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:47.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8031840 con 0x7f93c00ad970 2023-10-16T12:50:47.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a80443f0 con 0x7f93c00ad970 2023-10-16T12:50:47.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.156+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a80447a0 con 0x7f93c00ad970 2023-10-16T12:50:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.157+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c00c9f80 con 0x7f93c00ad970 2023-10-16T12:50:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.157+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c014e000 con 0x7f93c00ad970 2023-10-16T12:50:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.158+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c006b330 con 0x7f93c00ad970 2023-10-16T12:50:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.159+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93a804d450 con 0x7f93c00ad970 2023-10-16T12:50:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.160+0000 7f93b6ffd640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f93a006af90 0x7f93a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.160+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f93a802d040 con 0x7f93c00ad970 2023-10-16T12:50:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.160+0000 7f93be59c640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f93a006af90 0x7f93a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.162+0000 7f93be59c640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f93a006af90 0x7f93a006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93b0006fd0 tx=0x7f93b0008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:47.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.164+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93a8076e70 con 0x7f93c00ad970 2023-10-16T12:50:47.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.428+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f93c00c9de0 con 0x7f93c00ad970 2023-10-16T12:50:47.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.429+0000 7f93b6ffd640 1 -- 172.21.15.60:0/1282203638 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f93c00c9de0 con 0x7f93c00ad970 2023-10-16T12:50:47.430 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:47.431 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:50:47.431 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:50:47.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.431 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:50:47.431 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:50:47.432 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.432 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.432 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:47.432 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:47.432 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.433 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.433 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:50:47.433 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:50:47.434 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.434 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.434 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:47.434 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:47.434 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.435 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.435 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:50:47.435 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:50:47.435 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.436 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.436 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:47.436 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:47.436 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.436 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.437 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.437 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:50:47.437 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:50:47.437 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.438 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.438 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:47.438 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:47.438 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.438 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.439 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:50:47.439 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:50:47.439 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.439 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.440 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.440 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.441 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:50:47.441 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:50:47.441 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.441 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.441 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.441 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.442 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.442 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:50:47.442 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:50:47.443 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.443 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.443 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.443 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.443 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.444 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:50:47.444 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:50:47.444 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.444 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.445 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.445 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.445 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.446 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:50:47.446 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:50:47.446 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.446 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.446 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.447 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.447 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.447 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:50:47.448 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:50:47.448 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.448 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.448 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.448 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.449 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:50:47.449 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:50:47.449 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.450 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.450 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.450 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.450 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.451 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:50:47.451 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:50:47.451 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.451 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:50:47.452 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:50:47.452 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.452 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.453 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:50:47.453 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:50:47.453 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.453 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.453 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:50:47.454 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:50:47.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.454 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:50:47.454 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:50:47.455 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.455 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.455 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:50:47.455 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:50:47.455 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:50:47.456 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.456 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:50:47.456 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:50:47.457 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.457 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.457 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:50:47.457 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:50:47.457 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:50:47.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.458 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:50:47.458 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:50:47.459 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.459 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:50:47.459 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:50:47.459 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:50:47.459 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:50:47.460 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.460 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:50:47.460 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:50:47.460 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.461 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:50:47.461 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:50:47.461 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:50:47.461 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.461 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.462 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:50:47.462 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:50:47.462 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.463 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:50:47.463 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:50:47.463 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.463 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:50:47.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:47.464 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:50:47.464 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:50:47.464 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:50:47.464 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:50:47.464 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:50:47.465 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:50:47.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:47.465 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:50:47.466 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:50:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f93a006af90 msgr2=0x7f93a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f93a006af90 0x7f93a006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93b0006fd0 tx=0x7f93b0008040 comp rx=0 tx=0).stop 2023-10-16T12:50:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 msgr2=0x7f93c00c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c00c9050 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f93a8000c00 tx=0x7f93a80046c0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 shutdown_connections 2023-10-16T12:50:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f93a006af90 0x7f93a006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93c015ebc0 0x7f93c00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93c00ad970 0x7f93c00c9050 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 --2- 172.21.15.60:0/1282203638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93c006c040 0x7f93c014dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 >> 172.21.15.60:0/1282203638 conn(0x7f93c00ce6a0 msgr2=0x7f93c015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.431+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 shutdown_connections 2023-10-16T12:50:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.432+0000 7f93bf59e640 1 -- 172.21.15.60:0/1282203638 wait complete. 2023-10-16T12:50:47.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.700+0000 7fe6a884d640 1 Processor -- start 2023-10-16T12:50:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.701+0000 7fe6a884d640 1 -- start start 2023-10-16T12:50:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.702+0000 7fe6a884d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 0x7fe6a406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.702+0000 7fe6a884d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a415f4a0 con 0x7fe6a40cda80 2023-10-16T12:50:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.702+0000 7fe6a884d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a406c960 con 0x7fe6a40ad970 2023-10-16T12:50:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.702+0000 7fe6a884d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a415b9d0 con 0x7fe6a406c040 2023-10-16T12:50:47.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe6a40cda80 0x7fe6a415ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44624/0 (socket says 172.21.15.60:44624) 2023-10-16T12:50:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a3fff640 1 -- 172.21.15.60:0/3063042340 learned_addr learned my addr 172.21.15.60:0/3063042340 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 0x7fe6a406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856641028 0 0) 0x7fe6a415f4a0 con 0x7fe6a40cda80 2023-10-16T12:50:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe68c003520 con 0x7fe6a40cda80 2023-10-16T12:50:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1808941684 0 0) 0x7fe68c003520 con 0x7fe6a40cda80 2023-10-16T12:50:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 msgr2=0x7fe6a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 0x7fe6a406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 >> v1:172.21.15.144:6789/0 conn(0x7fe6a40ad970 legacy=0x7fe6a415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a415bcb0 con 0x7fe6a40cda80 2023-10-16T12:50:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe69c001670 con 0x7fe6a40cda80 2023-10-16T12:50:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 >> v1:172.21.15.60:6789/0 conn(0x7fe6a40cda80 legacy=0x7fe6a415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.703+0000 7fe6a27fc640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68c0039a0 0x7fe68c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a27fc640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0047c0 0x7fe68c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a27fc640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 0x7fe68c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a415b9d0 con 0x7fe68c0039a0 2023-10-16T12:50:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c00ba60 con 0x7fe68c0047c0 2023-10-16T12:50:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c00bc20 con 0x7fe6a406c040 2023-10-16T12:50:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68c0039a0 0x7fe68c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a37fe640 1 -- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 msgr2=0x7fe68c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a406c040 0x7fe68c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a37fe640 1 -- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0047c0 msgr2=0x7fe68c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a2ffd640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0047c0 0x7fe68c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0047c0 0x7fe68c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a37fe640 1 -- 172.21.15.60:0/3063042340 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c003520 con 0x7fe68c0039a0 2023-10-16T12:50:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.704+0000 7fe6a2ffd640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0047c0 0x7fe68c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.705+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68c0039a0 0x7fe68c003dc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe694009a90 tx=0x7fe68c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34acfe554027fe8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69402c6b0 con 0x7fe68c0039a0 2023-10-16T12:50:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69402c810 con 0x7fe68c0039a0 2023-10-16T12:50:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a27fc640 1 -- 172.21.15.60:0/3063042340 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69402cba0 con 0x7fe68c0039a0 2023-10-16T12:50:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a884d640 1 -- 172.21.15.60:0/3063042340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68c0039a0 msgr2=0x7fe68c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3063042340 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68c0039a0 0x7fe68c003dc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe694009a90 tx=0x7fe68c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a3fff640 1 -- 172.21.15.60:0/3063042340 reap_dead start 2023-10-16T12:50:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a884d640 1 -- 172.21.15.60:0/3063042340 shutdown_connections 2023-10-16T12:50:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a884d640 1 -- 172.21.15.60:0/3063042340 >> 172.21.15.60:0/3063042340 conn(0x7fe6a40ce6a0 msgr2=0x7fe6a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a884d640 1 -- 172.21.15.60:0/3063042340 shutdown_connections 2023-10-16T12:50:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.706+0000 7fe6a884d640 1 -- 172.21.15.60:0/3063042340 wait complete. 2023-10-16T12:50:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.708+0000 7fe6a884d640 1 Processor -- start 2023-10-16T12:50:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.708+0000 7fe6a884d640 1 -- start start 2023-10-16T12:50:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a884d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0039a0 0x7fe6a40acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a884d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 0x7fe6a413fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a884d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a41f2a80 0x7fe6a41f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a884d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a4140430 con 0x7fe6a413f740 2023-10-16T12:50:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a884d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a4140710 con 0x7fe68c0039a0 2023-10-16T12:50:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a884d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a41409f0 con 0x7fe6a41f2a80 2023-10-16T12:50:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 0x7fe6a413fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 0x7fe6a413fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51440/0 (socket says 172.21.15.60:51440) 2023-10-16T12:50:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.709+0000 7fe6a2ffd640 1 -- 172.21.15.60:0/3927609501 learned_addr learned my addr 172.21.15.60:0/3927609501 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a3fff640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a41f2a80 0x7fe6a41f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a2ffd640 1 -- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a41f2a80 msgr2=0x7fe6a41f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a2ffd640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a41f2a80 0x7fe6a41f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0039a0 0x7fe6a40acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a2ffd640 1 -- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0039a0 msgr2=0x7fe6a40acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a2ffd640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0039a0 0x7fe6a40acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a2ffd640 1 -- 172.21.15.60:0/3927609501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a4140cd0 con 0x7fe6a413f740 2023-10-16T12:50:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0039a0 0x7fe6a40acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe6a2ffd640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 0x7fe6a413fbc0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fe69800db10 tx=0x7fe6980020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.710+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698003da0 con 0x7fe6a413f740 2023-10-16T12:50:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.711+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe698006b50 con 0x7fe6a413f740 2023-10-16T12:50:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.711+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69800b740 con 0x7fe6a413f740 2023-10-16T12:50:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.711+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a4140fb0 con 0x7fe6a413f740 2023-10-16T12:50:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.711+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a4140100 con 0x7fe6a413f740 2023-10-16T12:50:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.712+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a406b330 con 0x7fe6a413f740 2023-10-16T12:50:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.713+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe69800b8a0 con 0x7fe6a413f740 2023-10-16T12:50:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.714+0000 7fe68bfff640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fe69006af90 0x7fe69006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.714+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe698088b80 con 0x7fe6a413f740 2023-10-16T12:50:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.714+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fe69006af90 0x7fe69006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.716+0000 7fe6a37fe640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fe69006af90 0x7fe69006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe694036000 tx=0x7fe694030000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.718+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe698053d70 con 0x7fe6a413f740 2023-10-16T12:50:47.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.914+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6a40cc520 con 0x7fe6a413f740 2023-10-16T12:50:47.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.916+0000 7fe68bfff640 1 -- 172.21.15.60:0/3927609501 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+12847 (secure 0 0 0) 0x7fe698018090 con 0x7fe6a413f740 2023-10-16T12:50:47.917 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:47.918 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":109,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:46.892542+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.918+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fe69006af90 msgr2=0x7fe69006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.918+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fe69006af90 0x7fe69006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe694036000 tx=0x7fe694030000 comp rx=0 tx=0).stop 2023-10-16T12:50:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 msgr2=0x7fe6a413fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 0x7fe6a413fbc0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fe69800db10 tx=0x7fe6980020c0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 shutdown_connections 2023-10-16T12:50:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fe69006af90 0x7fe69006d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6a41f2a80 0x7fe6a41f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6a413f740 0x7fe6a413fbc0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 --2- 172.21.15.60:0/3927609501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe68c0039a0 0x7fe6a40acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 >> 172.21.15.60:0/3927609501 conn(0x7fe6a40ce6a0 msgr2=0x7fe6a40cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 shutdown_connections 2023-10-16T12:50:47.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:47.919+0000 7fe6a884d640 1 -- 172.21.15.60:0/3927609501 wait complete. 2023-10-16T12:50:47.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:50:47.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:50:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.169+0000 7fb365fff640 1 Processor -- start 2023-10-16T12:50:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.169+0000 7fb365fff640 1 -- start start 2023-10-16T12:50:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.171+0000 7fb365fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb36015cc00 0x7fb3600ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.171+0000 7fb365fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb3600ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.171+0000 7fb365fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3600ce480 con 0x7fb3600ad670 2023-10-16T12:50:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.171+0000 7fb365fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3600ca950 con 0x7fb36015cc00 2023-10-16T12:50:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.171+0000 7fb365fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600cd7b0 con 0x7fb3600cd260 2023-10-16T12:50:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb3657fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3600ad670 0x7fb3600c8cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44640/0 (socket says 172.21.15.60:44640) 2023-10-16T12:50:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb3657fe640 1 -- 172.21.15.60:0/997590803 learned_addr learned my addr 172.21.15.60:0/997590803 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb364ffd640 1 --2- 172.21.15.60:0/997590803 >> v2:172.21.15.144:3300/0 conn(0x7fb36015cc00 0x7fb3600ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35ffff640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb3600ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950313851 0 0) 0x7fb3600ce480 con 0x7fb3600ad670 2023-10-16T12:50:48.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb350003520 con 0x7fb3600ad670 2023-10-16T12:50:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355526898 0 0) 0x7fb350003520 con 0x7fb3600ad670 2023-10-16T12:50:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 msgr2=0x7fb3600ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb3600ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 >> v2:172.21.15.144:3300/0 conn(0x7fb36015cc00 msgr2=0x7fb3600ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 --2- 172.21.15.60:0/997590803 >> v2:172.21.15.144:3300/0 conn(0x7fb36015cc00 0x7fb3600ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.172+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600cda90 con 0x7fb3600ad670 2023-10-16T12:50:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb3580019f0 con 0x7fb3600ad670 2023-10-16T12:50:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 >> v1:172.21.15.60:6789/0 conn(0x7fb3600ad670 legacy=0x7fb3600c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb350003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb350003f40 0x7fb3500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb350004d00 0x7fb350005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600ca950 con 0x7fb350004d00 2023-10-16T12:50:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600cd7b0 con 0x7fb350003f40 2023-10-16T12:50:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35000b980 con 0x7fb3600cd260 2023-10-16T12:50:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb3657fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb350004d00 0x7fb350005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb364ffd640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb350003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb35ffff640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb350003f40 0x7fb3500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb3657fe640 1 -- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 msgr2=0x7fb350003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb3657fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb350003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb3657fe640 1 -- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb350003f40 msgr2=0x7fb3500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb3657fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb350003f40 0x7fb3500047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb3657fe640 1 -- 172.21.15.60:0/997590803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350003520 con 0x7fb350004d00 2023-10-16T12:50:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.173+0000 7fb364ffd640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3600cd260 0x7fb350003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.174+0000 7fb3657fe640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb350004d00 0x7fb350005180 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb35800a760 tx=0x7fb3500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f717baa511623249 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.175+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358034630 con 0x7fb350004d00 2023-10-16T12:50:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.175+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35802f070 con 0x7fb350004d00 2023-10-16T12:50:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.175+0000 7fb35f7fe640 1 -- 172.21.15.60:0/997590803 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358034a30 con 0x7fb350004d00 2023-10-16T12:50:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.175+0000 7fb365fff640 1 -- 172.21.15.60:0/997590803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb350004d00 msgr2=0x7fb350005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.175+0000 7fb365fff640 1 --2- 172.21.15.60:0/997590803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb350004d00 0x7fb350005180 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb35800a760 tx=0x7fb3500057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.175+0000 7fb3657fe640 1 -- 172.21.15.60:0/997590803 reap_dead start 2023-10-16T12:50:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.176+0000 7fb365fff640 1 -- 172.21.15.60:0/997590803 shutdown_connections 2023-10-16T12:50:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.176+0000 7fb365fff640 1 -- 172.21.15.60:0/997590803 >> 172.21.15.60:0/997590803 conn(0x7fb3600b0170 msgr2=0x7fb3600b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.176+0000 7fb365fff640 1 -- 172.21.15.60:0/997590803 shutdown_connections 2023-10-16T12:50:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.176+0000 7fb365fff640 1 -- 172.21.15.60:0/997590803 wait complete. 2023-10-16T12:50:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.177+0000 7fb365fff640 1 Processor -- start 2023-10-16T12:50:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 -- start start 2023-10-16T12:50:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb350004d00 0x7fb3600c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb36015f530 0x7fb36006b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 0x7fb36015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600c5e60 con 0x7fb36015e030 2023-10-16T12:50:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36006ad20 con 0x7fb36015f530 2023-10-16T12:50:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.178+0000 7fb365fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36006a140 con 0x7fb350004d00 2023-10-16T12:50:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 0x7fb36015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 0x7fb36015e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51454/0 (socket says 172.21.15.60:51454) 2023-10-16T12:50:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 -- 172.21.15.60:0/3357727475 learned_addr learned my addr 172.21.15.60:0/3357727475 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb364ffd640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb350004d00 0x7fb3600c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb35ffff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb36015f530 0x7fb36006b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 -- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb350004d00 msgr2=0x7fb3600c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb350004d00 0x7fb3600c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 -- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb36015f530 msgr2=0x7fb36006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb36015f530 0x7fb36006b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.179+0000 7fb3657fe640 1 -- 172.21.15.60:0/3357727475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36006a3f0 con 0x7fb36015e030 2023-10-16T12:50:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.180+0000 7fb3657fe640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 0x7fb36015e4b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb35802cb30 tx=0x7fb358001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.180+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358041440 con 0x7fb36015e030 2023-10-16T12:50:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.180+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3580415a0 con 0x7fb36015e030 2023-10-16T12:50:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.180+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358041910 con 0x7fb36015e030 2023-10-16T12:50:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.180+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36006a6d0 con 0x7fb36015e030 2023-10-16T12:50:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.180+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36006b6f0 con 0x7fb36015e030 2023-10-16T12:50:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.181+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36004e9b0 con 0x7fb36015e030 2023-10-16T12:50:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.182+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb358031030 con 0x7fb36015e030 2023-10-16T12:50:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.183+0000 7fb35d7fa640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb34406af90 0x7fb34406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.183+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb3580ab4a0 con 0x7fb36015e030 2023-10-16T12:50:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.184+0000 7fb364ffd640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb34406af90 0x7fb34406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.185+0000 7fb364ffd640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb34406af90 0x7fb34406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb3600c7010 tx=0x7fb354008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.188+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3580765e0 con 0x7fb36015e030 2023-10-16T12:50:48.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.382+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb36006d320 con 0x7fb36015e030 2023-10-16T12:50:48.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.383+0000 7fb35d7fa640 1 -- 172.21.15.60:0/3357727475 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb358047020 con 0x7fb36015e030 2023-10-16T12:50:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:50:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.385+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb34406af90 msgr2=0x7fb34406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.385+0000 7fb365fff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb34406af90 0x7fb34406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb3600c7010 tx=0x7fb354008040 comp rx=0 tx=0).stop 2023-10-16T12:50:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.385+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 msgr2=0x7fb36015e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.385+0000 7fb365fff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 0x7fb36015e4b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb35802cb30 tx=0x7fb358001cb0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 shutdown_connections 2023-10-16T12:50:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fb34406af90 0x7fb34406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36015e030 0x7fb36015e4b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb36015f530 0x7fb36006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 --2- 172.21.15.60:0/3357727475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb350004d00 0x7fb3600c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 >> 172.21.15.60:0/3357727475 conn(0x7fb3600b0170 msgr2=0x7fb3600adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 shutdown_connections 2023-10-16T12:50:48.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.386+0000 7fb365fff640 1 -- 172.21.15.60:0/3357727475 wait complete. 2023-10-16T12:50:48.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:50:48.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.620+0000 7f2951661640 1 Processor -- start 2023-10-16T12:50:48.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.620+0000 7f2951661640 1 -- start start 2023-10-16T12:50:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.621+0000 7f2951661640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f294c15d190 0x7f294c0d38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.621+0000 7f2951661640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0c68d0 con 0x7f294c0d1a70 2023-10-16T12:50:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f2951661640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0ceb10 con 0x7f294c0ce3a0 2023-10-16T12:50:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f2951661640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294c0c6b20 con 0x7f294c15d190 2023-10-16T12:50:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f294c15d190 0x7f294c0d38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f294c0d1a70 0x7f294c0b3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44642/0 (socket says 172.21.15.60:44642) 2023-10-16T12:50:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f294c15d190 0x7f294c0d38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51466/0 (socket says 172.21.15.60:51466) 2023-10-16T12:50:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294b7fe640 1 -- 172.21.15.60:0/242533084 learned_addr learned my addr 172.21.15.60:0/242533084 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294affd640 1 -- 172.21.15.60:0/242533084 >> v1:172.21.15.144:6789/0 conn(0x7f294c0ce3a0 legacy=0x7f294c0d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294affd640 1 -- 172.21.15.60:0/242533084 >> v1:172.21.15.60:6789/0 conn(0x7f294c0d1a70 legacy=0x7f294c0b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f294affd640 1 -- 172.21.15.60:0/242533084 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c0c6e00 con 0x7f294c15d190 2023-10-16T12:50:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.622+0000 7f2949ffb640 1 -- 172.21.15.60:0/242533084 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387824317 0 0) 0x7f294c0c68d0 con 0x7f294c0d1a70 2023-10-16T12:50:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.624+0000 7f294affd640 1 --2- 172.21.15.60:0/242533084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0d38a0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f293c009d90 tx=0x7f293c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7b25e6a0014a5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.624+0000 7f2949ffb640 1 -- 172.21.15.60:0/242533084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c02cd00 con 0x7f294c15d190 2023-10-16T12:50:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.624+0000 7f2949ffb640 1 -- 172.21.15.60:0/242533084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293c02ce60 con 0x7f294c15d190 2023-10-16T12:50:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.624+0000 7f2949ffb640 1 -- 172.21.15.60:0/242533084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c034660 con 0x7f294c15d190 2023-10-16T12:50:48.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.624+0000 7f2951661640 1 -- 172.21.15.60:0/242533084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 msgr2=0x7f294c0d38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.624+0000 7f2951661640 1 --2- 172.21.15.60:0/242533084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0d38a0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f293c009d90 tx=0x7f293c02bf50 comp rx=0 tx=0).stop 2023-10-16T12:50:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.625+0000 7f2951661640 1 -- 172.21.15.60:0/242533084 shutdown_connections 2023-10-16T12:50:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.625+0000 7f2951661640 1 --2- 172.21.15.60:0/242533084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0d38a0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.625+0000 7f2951661640 1 -- 172.21.15.60:0/242533084 >> 172.21.15.60:0/242533084 conn(0x7f294c0d2d80 msgr2=0x7f294c0d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.625+0000 7f2951661640 1 -- 172.21.15.60:0/242533084 shutdown_connections 2023-10-16T12:50:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.625+0000 7f2951661640 1 -- 172.21.15.60:0/242533084 wait complete. 2023-10-16T12:50:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.626+0000 7f2951661640 1 Processor -- start 2023-10-16T12:50:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 -- start start 2023-10-16T12:50:48.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f294c0ce3a0 0x7f294c152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f294c0d1a70 0x7f294c0af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c1527c0 con 0x7f294c15d190 2023-10-16T12:50:48.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0cb730 con 0x7f294c0d1a70 2023-10-16T12:50:48.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.627+0000 7f2951661640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0a0990 con 0x7f294c0ce3a0 2023-10-16T12:50:48.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51472/0 (socket says 172.21.15.60:51472) 2023-10-16T12:50:48.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 -- 172.21.15.60:0/1074387254 learned_addr learned my addr 172.21.15.60:0/1074387254 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:48.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294affd640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f294c0ce3a0 0x7f294c152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294a7fc640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f294c0d1a70 0x7f294c0af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 -- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f294c0ce3a0 msgr2=0x7f294c152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f294c0ce3a0 0x7f294c152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 -- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f294c0d1a70 msgr2=0x7f294c0af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f294c0d1a70 0x7f294c0af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.628+0000 7f294b7fe640 1 -- 172.21.15.60:0/1074387254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c009a70 con 0x7f294c15d190 2023-10-16T12:50:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f294affd640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f294c0ce3a0 0x7f294c152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f294b7fe640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0b7b70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f294400c450 tx=0x7f294400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29440074e0 con 0x7f294c15d190 2023-10-16T12:50:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2944007660 con 0x7f294c15d190 2023-10-16T12:50:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944004a20 con 0x7f294c15d190 2023-10-16T12:50:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c0cb9f0 con 0x7f294c15d190 2023-10-16T12:50:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.629+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c0cbfb0 con 0x7f294c15d190 2023-10-16T12:50:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.631+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c0ce7e0 con 0x7f294c15d190 2023-10-16T12:50:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.631+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2944004b80 con 0x7f294c15d190 2023-10-16T12:50:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.632+0000 7f29437fe640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f292c06af90 0x7f292c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.632+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2944089270 con 0x7f294c15d190 2023-10-16T12:50:48.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.633+0000 7f294affd640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f292c06af90 0x7f292c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:48.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.634+0000 7f294affd640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f292c06af90 0x7f292c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f293c036000 tx=0x7f293c030000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:48.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.637+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f294c0ce7e0 con 0x7f294c15d190 2023-10-16T12:50:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.834+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f294c0ce7e0 con 0x7f294c15d190 2023-10-16T12:50:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.835+0000 7f29437fe640 1 -- 172.21.15.60:0/1074387254 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v109) v1 ==== 84+0+65 (secure 0 0 0) 0x7f294c0ce7e0 con 0x7f294c15d190 2023-10-16T12:50:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:48.837 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:50:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:50:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:50:48.838 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:50:48.838 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:50:48.838 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:50:48.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f292c06af90 msgr2=0x7f292c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f292c06af90 0x7f292c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f293c036000 tx=0x7f293c030000 comp rx=0 tx=0).stop 2023-10-16T12:50:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 msgr2=0x7f294c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0b7b70 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f294400c450 tx=0x7f294400c910 comp rx=0 tx=0).stop 2023-10-16T12:50:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 shutdown_connections 2023-10-16T12:50:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f292c06af90 0x7f292c06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f294c15d190 0x7f294c0b7b70 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f294c0d1a70 0x7f294c0af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 --2- 172.21.15.60:0/1074387254 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f294c0ce3a0 0x7f294c152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 >> 172.21.15.60:0/1074387254 conn(0x7f294c0d2d80 msgr2=0x7f294c0c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 shutdown_connections 2023-10-16T12:50:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:48.838+0000 7f2951661640 1 -- 172.21.15.60:0/1074387254 wait complete. 2023-10-16T12:50:48.857 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:50:48.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:50:49.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.086+0000 7f7721840640 1 Processor -- start 2023-10-16T12:50:49.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.086+0000 7f7721840640 1 -- start start 2023-10-16T12:50:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.087+0000 7f7721840640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f771c15ad80 0x7f771c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.087+0000 7f7721840640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f771c0cae40 0x7f771c06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.087+0000 7f7721840640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c0ad7f0 0x7f771c06b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.087+0000 7f7721840640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c0c8e70 con 0x7f771c0cae40 2023-10-16T12:50:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.087+0000 7f7721840640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c0b4170 con 0x7f771c15ad80 2023-10-16T12:50:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.087+0000 7f7721840640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0adde0 con 0x7f771c0ad7f0 2023-10-16T12:50:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f771c0cae40 0x7f771c06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f771c0cae40 0x7f771c06d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41142/0 (socket says 172.21.15.60:41142) 2023-10-16T12:50:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 -- 172.21.15.60:0/4097943865 learned_addr learned my addr 172.21.15.60:0/4097943865 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f772103f640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c0ad7f0 0x7f771c06b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f772083e640 1 --2- 172.21.15.60:0/4097943865 >> v2:172.21.15.144:3300/0 conn(0x7f771c15ad80 0x7f771c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 -- 172.21.15.60:0/4097943865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c0ad7f0 msgr2=0x7f771c06b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c0ad7f0 0x7f771c06b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 -- 172.21.15.60:0/4097943865 >> v2:172.21.15.144:3300/0 conn(0x7f771c15ad80 msgr2=0x7f771c06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.088+0000 7f771bfff640 1 --2- 172.21.15.60:0/4097943865 >> v2:172.21.15.144:3300/0 conn(0x7f771c15ad80 0x7f771c06cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.089+0000 7f771bfff640 1 -- 172.21.15.60:0/4097943865 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c0b4410 con 0x7f771c0cae40 2023-10-16T12:50:49.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.089+0000 7f772103f640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c0ad7f0 0x7f771c06b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.090+0000 7f771bfff640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c06d0b0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f771c070710 tx=0x7f771002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9b903c09c7bb8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.090+0000 7f771b7fe640 1 -- 172.21.15.60:0/4097943865 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771002ca80 con 0x7f771c0cae40 2023-10-16T12:50:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f771b7fe640 1 -- 172.21.15.60:0/4097943865 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771002cbe0 con 0x7f771c0cae40 2023-10-16T12:50:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f771b7fe640 1 -- 172.21.15.60:0/4097943865 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100343f0 con 0x7f771c0cae40 2023-10-16T12:50:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 -- 172.21.15.60:0/4097943865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 msgr2=0x7f771c06d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c06d0b0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f771c070710 tx=0x7f771002bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 -- 172.21.15.60:0/4097943865 shutdown_connections 2023-10-16T12:50:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c0ad7f0 0x7f771c06b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 --2- 172.21.15.60:0/4097943865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c06d0b0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 --2- 172.21.15.60:0/4097943865 >> v2:172.21.15.144:3300/0 conn(0x7f771c15ad80 0x7f771c06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 -- 172.21.15.60:0/4097943865 >> 172.21.15.60:0/4097943865 conn(0x7f771c0ce820 msgr2=0x7f771c0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 -- 172.21.15.60:0/4097943865 shutdown_connections 2023-10-16T12:50:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.091+0000 7f7721840640 1 -- 172.21.15.60:0/4097943865 wait complete. 2023-10-16T12:50:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.093+0000 7f7721840640 1 Processor -- start 2023-10-16T12:50:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.093+0000 7f7721840640 1 -- start start 2023-10-16T12:50:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f7721840640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c0ad7f0 0x7f771c15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f7721840640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f7721840640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15ad80 0x7f771c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f7721840640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0b4650 con 0x7f771c0cae40 2023-10-16T12:50:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f7721840640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0c6300 con 0x7f771c0ad7f0 2023-10-16T12:50:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f7721840640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0c6550 con 0x7f771c15ad80 2023-10-16T12:50:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f771bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f771bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41158/0 (socket says 172.21.15.60:41158) 2023-10-16T12:50:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.094+0000 7f771bfff640 1 -- 172.21.15.60:0/822960584 learned_addr learned my addr 172.21.15.60:0/822960584 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f772083e640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c0ad7f0 0x7f771c15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f771bfff640 1 -- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15ad80 msgr2=0x7f771c0af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f771bfff640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15ad80 0x7f771c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f771bfff640 1 -- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c0ad7f0 msgr2=0x7f771c15d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f771bfff640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c0ad7f0 0x7f771c15d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f771bfff640 1 -- 172.21.15.60:0/822960584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710009b30 con 0x7f771c0cae40 2023-10-16T12:50:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.095+0000 7f771bfff640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c150060 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7710000f80 tx=0x7f7710004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.096+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710009510 con 0x7f771c0cae40 2023-10-16T12:50:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.096+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771003e450 con 0x7f771c0cae40 2023-10-16T12:50:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.096+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771003e800 con 0x7f771c0cae40 2023-10-16T12:50:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.096+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c0c6830 con 0x7f771c0cae40 2023-10-16T12:50:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.096+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c0c6df0 con 0x7f771c0cae40 2023-10-16T12:50:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.097+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c04e9b0 con 0x7f771c0cae40 2023-10-16T12:50:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.098+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f771002cd50 con 0x7f771c0cae40 2023-10-16T12:50:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.099+0000 7f77197fa640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f770806af90 0x7f770806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.099+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f77100ac0e0 con 0x7f771c0cae40 2023-10-16T12:50:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.099+0000 7f772083e640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f770806af90 0x7f770806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.101+0000 7f772083e640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f770806af90 0x7f770806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f770c006fd0 tx=0x7f770c008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.103+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7710077330 con 0x7f771c0cae40 2023-10-16T12:50:49.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.298+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f771c06d680 con 0x7f771c0cae40 2023-10-16T12:50:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.911+0000 7f77197fa640 1 -- 172.21.15.60:0/822960584 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v110) v1 ==== 138+0+0 (secure 0 0 0) 0x7f771007a950 con 0x7f771c0cae40 2023-10-16T12:50:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:50:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.914+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f770806af90 msgr2=0x7f770806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.914+0000 7f7721840640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f770806af90 0x7f770806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f770c006fd0 tx=0x7f770c008040 comp rx=0 tx=0).stop 2023-10-16T12:50:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.914+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 msgr2=0x7f771c150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.914+0000 7f7721840640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c150060 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7710000f80 tx=0x7f7710004a40 comp rx=0 tx=0).stop 2023-10-16T12:50:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 shutdown_connections 2023-10-16T12:50:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f770806af90 0x7f770806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15ad80 0x7f771c0af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c0cae40 0x7f771c150060 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 --2- 172.21.15.60:0/822960584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c0ad7f0 0x7f771c15d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 >> 172.21.15.60:0/822960584 conn(0x7f771c0ce820 msgr2=0x7f771c0cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 shutdown_connections 2023-10-16T12:50:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:49.915+0000 7f7721840640 1 -- 172.21.15.60:0/822960584 wait complete. 2023-10-16T12:50:49.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:50:50.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.166+0000 7f6259e2e640 1 Processor -- start 2023-10-16T12:50:50.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.166+0000 7f6259e2e640 1 -- start start 2023-10-16T12:50:50.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.167+0000 7f6259e2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f6254161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.167+0000 7f6259e2e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f62540b1690 0x7f62540b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.167+0000 7f6259e2e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62540b4ab0 0x7f625406bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.167+0000 7f6259e2e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62540b4f00 con 0x7f62540b4ab0 2023-10-16T12:50:50.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.167+0000 7f6259e2e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62541616c0 con 0x7f62540b1690 2023-10-16T12:50:50.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.167+0000 7f6259e2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62540cdbf0 con 0x7f625415cb50 2023-10-16T12:50:50.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.168+0000 7f625962d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62540b4ab0 0x7f625406bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.168+0000 7f625962d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62540b4ab0 0x7f625406bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41168/0 (socket says 172.21.15.60:41168) 2023-10-16T12:50:50.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.168+0000 7f625962d640 1 -- 172.21.15.60:0/3114178296 learned_addr learned my addr 172.21.15.60:0/3114178296 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:50.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.168+0000 7f6258e2c640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f6254161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.168+0000 7f624bfff640 1 --2- 172.21.15.60:0/3114178296 >> v2:172.21.15.144:3300/0 conn(0x7f62540b1690 0x7f62540b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f625962d640 1 -- 172.21.15.60:0/3114178296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 msgr2=0x7f6254161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f625962d640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f6254161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f625962d640 1 -- 172.21.15.60:0/3114178296 >> v2:172.21.15.144:3300/0 conn(0x7f62540b1690 msgr2=0x7f62540b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f625962d640 1 --2- 172.21.15.60:0/3114178296 >> v2:172.21.15.144:3300/0 conn(0x7f62540b1690 0x7f62540b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f625962d640 1 -- 172.21.15.60:0/3114178296 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62540cde90 con 0x7f62540b4ab0 2023-10-16T12:50:50.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f624bfff640 1 --2- 172.21.15.60:0/3114178296 >> v2:172.21.15.144:3300/0 conn(0x7f62540b1690 0x7f62540b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:50.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.169+0000 7f6258e2c640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f6254161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:50.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.170+0000 7f625962d640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b4ab0 0x7f625406bbd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f62540724e0 tx=0x7f624c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9384cac71828d562 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:50.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.170+0000 7f624b7fe640 1 -- 172.21.15.60:0/3114178296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c031520 con 0x7f62540b4ab0 2023-10-16T12:50:50.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.170+0000 7f624b7fe640 1 -- 172.21.15.60:0/3114178296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c031680 con 0x7f62540b4ab0 2023-10-16T12:50:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.170+0000 7f624b7fe640 1 -- 172.21.15.60:0/3114178296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c031a10 con 0x7f62540b4ab0 2023-10-16T12:50:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 -- 172.21.15.60:0/3114178296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b4ab0 msgr2=0x7f625406bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b4ab0 0x7f625406bbd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f62540724e0 tx=0x7f624c02b7c0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 -- 172.21.15.60:0/3114178296 shutdown_connections 2023-10-16T12:50:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b4ab0 0x7f625406bbd0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 --2- 172.21.15.60:0/3114178296 >> v2:172.21.15.144:3300/0 conn(0x7f62540b1690 0x7f62540b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 --2- 172.21.15.60:0/3114178296 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f6254161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 -- 172.21.15.60:0/3114178296 >> 172.21.15.60:0/3114178296 conn(0x7f62540cf570 msgr2=0x7f62540cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:50.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 -- 172.21.15.60:0/3114178296 shutdown_connections 2023-10-16T12:50:50.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.171+0000 7f6259e2e640 1 -- 172.21.15.60:0/3114178296 wait complete. 2023-10-16T12:50:50.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.173+0000 7f6259e2e640 1 Processor -- start 2023-10-16T12:50:50.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.173+0000 7f6259e2e640 1 -- start start 2023-10-16T12:50:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6259e2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 0x7f62541f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6259e2e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62540b4ab0 0x7f62541f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6259e2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f62540b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6259e2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62541f6a50 con 0x7f62540b1690 2023-10-16T12:50:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6259e2e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62540cc880 con 0x7f62540b4ab0 2023-10-16T12:50:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6259e2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62540c7ea0 con 0x7f625415cb50 2023-10-16T12:50:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6258e2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 0x7f62541f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6258e2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 0x7f62541f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41176/0 (socket says 172.21.15.60:41176) 2023-10-16T12:50:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.174+0000 7f6258e2c640 1 -- 172.21.15.60:0/2980234274 learned_addr learned my addr 172.21.15.60:0/2980234274 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f625962d640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f62540b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f624bfff640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62540b4ab0 0x7f62541f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f6258e2c640 1 -- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 msgr2=0x7f62540b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f6258e2c640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f62540b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f6258e2c640 1 -- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62540b4ab0 msgr2=0x7f62541f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f6258e2c640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62540b4ab0 0x7f62541f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f6258e2c640 1 -- 172.21.15.60:0/2980234274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c0094b0 con 0x7f62540b1690 2023-10-16T12:50:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.175+0000 7f6258e2c640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 0x7f62541f6510 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f6250009fc0 tx=0x7f625000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.176+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250003b90 con 0x7f62540b1690 2023-10-16T12:50:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.176+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625000f040 con 0x7f62540b1690 2023-10-16T12:50:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.176+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250004030 con 0x7f62540b1690 2023-10-16T12:50:50.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.176+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62540c8180 con 0x7f62540b1690 2023-10-16T12:50:50.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.176+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62541f2340 con 0x7f62540b1690 2023-10-16T12:50:50.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.177+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62540c7d00 con 0x7f62540b1690 2023-10-16T12:50:50.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.178+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6250004190 con 0x7f62540b1690 2023-10-16T12:50:50.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.179+0000 7f62497fa640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f624006af90 0x7f624006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:50.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.179+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6250089f30 con 0x7f62540b1690 2023-10-16T12:50:50.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.180+0000 7f624bfff640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f624006af90 0x7f624006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:50.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.181+0000 7f624bfff640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f624006af90 0x7f624006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f62540ce940 tx=0x7f6244008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:50.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.184+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6250055130 con 0x7f62540b1690 2023-10-16T12:50:50.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.380+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f62540c7d00 con 0x7f62540b1690 2023-10-16T12:50:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.918+0000 7f62497fa640 1 -- 172.21.15.60:0/2980234274 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v111) v1 ==== 170+0+0 (secure 0 0 0) 0x7f62540c7d00 con 0x7f62540b1690 2023-10-16T12:50:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:50:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f624006af90 msgr2=0x7f624006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f624006af90 0x7f624006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f62540ce940 tx=0x7f6244008040 comp rx=0 tx=0).stop 2023-10-16T12:50:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 msgr2=0x7f62541f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 0x7f62541f6510 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f6250009fc0 tx=0x7f625000d560 comp rx=0 tx=0).stop 2023-10-16T12:50:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 shutdown_connections 2023-10-16T12:50:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f624006af90 0x7f624006d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f625415cb50 0x7f62540b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62540b4ab0 0x7f62541f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 --2- 172.21.15.60:0/2980234274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540b1690 0x7f62541f6510 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 >> 172.21.15.60:0/2980234274 conn(0x7f62540cf570 msgr2=0x7f62540b52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 shutdown_connections 2023-10-16T12:50:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:50.922+0000 7f6259e2e640 1 -- 172.21.15.60:0/2980234274 wait complete. 2023-10-16T12:50:50.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:50:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.153+0000 7f26e6b0a640 1 Processor -- start 2023-10-16T12:50:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.154+0000 7f26e6b0a640 1 -- start start 2023-10-16T12:50:51.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.155+0000 7f26e6b0a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26e00b8480 0x7f26e00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.155+0000 7f26e6b0a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f26e00d0be0 0x7f26e00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.155+0000 7f26e6b0a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e015d150 con 0x7f26e015cd70 2023-10-16T12:50:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.155+0000 7f26e6b0a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00d1180 con 0x7f26e00b8480 2023-10-16T12:50:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.155+0000 7f26e6b0a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00d1460 con 0x7f26e00d0be0 2023-10-16T12:50:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.156+0000 7f26e5307640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26e00b8480 0x7f26e00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.156+0000 7f26e5307640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26e00b8480 0x7f26e00cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41186/0 (socket says 172.21.15.60:41186) 2023-10-16T12:50:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.156+0000 7f26e5307640 1 -- 172.21.15.60:0/631733806 learned_addr learned my addr 172.21.15.60:0/631733806 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.156+0000 7f26e6309640 1 --2- 172.21.15.60:0/631733806 >> v2:172.21.15.144:3300/0 conn(0x7f26e00d0be0 0x7f26e00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.156+0000 7f26e5307640 1 -- 172.21.15.60:0/631733806 >> v2:172.21.15.144:3300/0 conn(0x7f26e00d0be0 msgr2=0x7f26e00cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.157+0000 7f26e5307640 1 --2- 172.21.15.60:0/631733806 >> v2:172.21.15.144:3300/0 conn(0x7f26e00d0be0 0x7f26e00cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.157+0000 7f26e5307640 1 -- 172.21.15.60:0/631733806 >> v1:172.21.15.144:6789/0 conn(0x7f26e015cd70 legacy=0x7f26e00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.157+0000 7f26e5307640 1 -- 172.21.15.60:0/631733806 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00d1740 con 0x7f26e00b8480 2023-10-16T12:50:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.158+0000 7f26e5307640 1 --2- 172.21.15.60:0/631733806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e00b8480 0x7f26e00cd7c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f26e00b8910 tx=0x7f26d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688586ab8a846388 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.158+0000 7f26e4b06640 1 -- 172.21.15.60:0/631733806 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0031870 con 0x7f26e00b8480 2023-10-16T12:50:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.158+0000 7f26e4b06640 1 -- 172.21.15.60:0/631733806 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d00319d0 con 0x7f26e00b8480 2023-10-16T12:50:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.158+0000 7f26e4b06640 1 -- 172.21.15.60:0/631733806 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0031d60 con 0x7f26e00b8480 2023-10-16T12:50:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 -- 172.21.15.60:0/631733806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e00b8480 msgr2=0x7f26e00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/631733806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e00b8480 0x7f26e00cd7c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f26e00b8910 tx=0x7f26d002b910 comp rx=0 tx=0).stop 2023-10-16T12:50:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 -- 172.21.15.60:0/631733806 shutdown_connections 2023-10-16T12:50:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/631733806 >> v2:172.21.15.144:3300/0 conn(0x7f26e00d0be0 0x7f26e00cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/631733806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e00b8480 0x7f26e00cd7c0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 -- 172.21.15.60:0/631733806 >> 172.21.15.60:0/631733806 conn(0x7f26e00cf570 msgr2=0x7f26e00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:51.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 -- 172.21.15.60:0/631733806 shutdown_connections 2023-10-16T12:50:51.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.159+0000 7f26e6b0a640 1 -- 172.21.15.60:0/631733806 wait complete. 2023-10-16T12:50:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.161+0000 7f26e6b0a640 1 Processor -- start 2023-10-16T12:50:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.161+0000 7f26e6b0a640 1 -- start start 2023-10-16T12:50:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6b0a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00b8480 0x7f26e00cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6b0a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e00d0be0 0x7f26e0150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6b0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 0x7f26e01f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6b0a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01f6170 con 0x7f26e015cd70 2023-10-16T12:50:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6b0a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01f6450 con 0x7f26e00d0be0 2023-10-16T12:50:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6b0a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01f6730 con 0x7f26e00b8480 2023-10-16T12:50:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 0x7f26e01f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 0x7f26e01f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41188/0 (socket says 172.21.15.60:41188) 2023-10-16T12:50:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e6309640 1 -- 172.21.15.60:0/3200808667 learned_addr learned my addr 172.21.15.60:0/3200808667 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e5b08640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00b8480 0x7f26e00cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.163+0000 7f26e5307640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e00d0be0 0x7f26e0150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6309640 1 -- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00b8480 msgr2=0x7f26e00cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6309640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00b8480 0x7f26e00cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6309640 1 -- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e00d0be0 msgr2=0x7f26e0150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6309640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e00d0be0 0x7f26e0150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6309640 1 -- 172.21.15.60:0/3200808667 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d00094d0 con 0x7f26e015cd70 2023-10-16T12:50:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e5307640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e00d0be0 0x7f26e0150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:51.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6309640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 0x7f26e01f5920 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f26d8007d90 tx=0x7f26d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8014420 con 0x7f26e015cd70 2023-10-16T12:50:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d80145a0 con 0x7f26e015cd70 2023-10-16T12:50:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8014950 con 0x7f26e015cd70 2023-10-16T12:50:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.164+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e015d3e0 con 0x7f26e015cd70 2023-10-16T12:50:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.165+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e01f5e60 con 0x7f26e015cd70 2023-10-16T12:50:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.166+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e015d900 con 0x7f26e015cd70 2023-10-16T12:50:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.168+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f26d8014ab0 con 0x7f26e015cd70 2023-10-16T12:50:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.169+0000 7f26d67fc640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f26c406af90 0x7f26c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:51.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.169+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f26d808ad90 con 0x7f26e015cd70 2023-10-16T12:50:51.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.170+0000 7f26e5b08640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f26c406af90 0x7f26c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.171+0000 7f26e5b08640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f26c406af90 0x7f26c406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26dc000f60 tx=0x7f26dc009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.173+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26d8055e00 con 0x7f26e015cd70 2023-10-16T12:50:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.412+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f26e015d900 con 0x7f26e015cd70 2023-10-16T12:50:51.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.960+0000 7f26d67fc640 1 -- 172.21.15.60:0/3200808667 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v49) v1 ==== 174+0+0 (secure 0 0 0) 0x7f26e015d900 con 0x7f26e015cd70 2023-10-16T12:50:51.964 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 12 and data pool 13 2023-10-16T12:50:51.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.971+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f26c406af90 msgr2=0x7f26c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:51.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.971+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f26c406af90 0x7f26c406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26dc000f60 tx=0x7f26dc009040 comp rx=0 tx=0).stop 2023-10-16T12:50:51.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.971+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 msgr2=0x7f26e01f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:51.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.971+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 0x7f26e01f5920 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f26d8007d90 tx=0x7f26d800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.972+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 shutdown_connections 2023-10-16T12:50:51.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.972+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f26c406af90 0x7f26c406d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.972+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26e015cd70 0x7f26e01f5920 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.972+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e00d0be0 0x7f26e0150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.972+0000 7f26e6b0a640 1 --2- 172.21.15.60:0/3200808667 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00b8480 0x7f26e00cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:51.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.972+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 >> 172.21.15.60:0/3200808667 conn(0x7f26e00cf570 msgr2=0x7f26e00b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:51.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.974+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 shutdown_connections 2023-10-16T12:50:51.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:51.976+0000 7f26e6b0a640 1 -- 172.21.15.60:0/3200808667 wait complete. 2023-10-16T12:50:52.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.198+0000 7f4398d41640 1 Processor -- start 2023-10-16T12:50:52.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.198+0000 7f4398d41640 1 -- start start 2023-10-16T12:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.199+0000 7f4398d41640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f439415aaa0 0x7f439415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.199+0000 7f4398d41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43940ce390 0x7f4394070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.199+0000 7f4398d41640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439415ed60 con 0x7f439415ae80 2023-10-16T12:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.199+0000 7f4398d41640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439406b810 con 0x7f439415aaa0 2023-10-16T12:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.199+0000 7f4398d41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43940707e0 con 0x7f43940ce390 2023-10-16T12:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f439415aaa0 0x7f439415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f439415aaa0 0x7f439415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41200/0 (socket says 172.21.15.60:41200) 2023-10-16T12:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 -- 172.21.15.60:0/3043880696 learned_addr learned my addr 172.21.15.60:0/3043880696 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f4392d9d640 1 --2- 172.21.15.60:0/3043880696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43940ce390 0x7f4394070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 -- 172.21.15.60:0/3043880696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43940ce390 msgr2=0x7f4394070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 --2- 172.21.15.60:0/3043880696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43940ce390 0x7f4394070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 -- 172.21.15.60:0/3043880696 >> v1:172.21.15.60:6789/0 conn(0x7f439415ae80 legacy=0x7f43940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f439259c640 1 -- 172.21.15.60:0/3043880696 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439415ef90 con 0x7f439415aaa0 2023-10-16T12:50:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.200+0000 7f4391d9b640 1 -- 172.21.15.60:0/3043880696 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561952227 0 0) 0x7f439415ed60 con 0x7f439415ae80 2023-10-16T12:50:52.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f439259c640 1 --2- 172.21.15.60:0/3043880696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f439415e820 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4384009870 tx=0x7f438402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d0e67bec57297e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:52.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4391d9b640 1 -- 172.21.15.60:0/3043880696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384030840 con 0x7f439415aaa0 2023-10-16T12:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4391d9b640 1 -- 172.21.15.60:0/3043880696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43840309a0 con 0x7f439415aaa0 2023-10-16T12:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4391d9b640 1 -- 172.21.15.60:0/3043880696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384030d30 con 0x7f439415aaa0 2023-10-16T12:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4398d41640 1 -- 172.21.15.60:0/3043880696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 msgr2=0x7f439415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4398d41640 1 --2- 172.21.15.60:0/3043880696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f439415e820 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4384009870 tx=0x7f438402b980 comp rx=0 tx=0).stop 2023-10-16T12:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4398d41640 1 -- 172.21.15.60:0/3043880696 shutdown_connections 2023-10-16T12:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4398d41640 1 --2- 172.21.15.60:0/3043880696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43940ce390 0x7f4394070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4398d41640 1 --2- 172.21.15.60:0/3043880696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f439415e820 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.202+0000 7f4398d41640 1 -- 172.21.15.60:0/3043880696 >> 172.21.15.60:0/3043880696 conn(0x7f43940cb410 msgr2=0x7f43940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.203+0000 7f4398d41640 1 -- 172.21.15.60:0/3043880696 shutdown_connections 2023-10-16T12:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.203+0000 7f4398d41640 1 -- 172.21.15.60:0/3043880696 wait complete. 2023-10-16T12:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.204+0000 7f4398d41640 1 Processor -- start 2023-10-16T12:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.205+0000 7f4398d41640 1 -- start start 2023-10-16T12:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.205+0000 7f4398d41640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43940ce390 0x7f43940c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f4398d41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f43940acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f4398d41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439415ae80 0x7f439414e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f4398d41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43940c78d0 con 0x7f439415aaa0 2023-10-16T12:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f4398d41640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43940c6cc0 con 0x7f43940ce390 2023-10-16T12:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f4398d41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43940c6fa0 con 0x7f439415ae80 2023-10-16T12:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f43940acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f43940acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41210/0 (socket says 172.21.15.60:41210) 2023-10-16T12:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 -- 172.21.15.60:0/1101898224 learned_addr learned my addr 172.21.15.60:0/1101898224 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f4392d9d640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439415ae80 0x7f439414e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f439259c640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43940ce390 0x7f43940c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 -- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439415ae80 msgr2=0x7f439414e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439415ae80 0x7f439414e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 -- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43940ce390 msgr2=0x7f43940c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43940ce390 0x7f43940c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f438bfff640 1 -- 172.21.15.60:0/1101898224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43840094d0 con 0x7f439415aaa0 2023-10-16T12:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.206+0000 7f439259c640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43940ce390 0x7f43940c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.207+0000 7f438bfff640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f43940acb00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c410 tx=0x7f437c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.207+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c0052d0 con 0x7f439415aaa0 2023-10-16T12:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.207+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c005450 con 0x7f439415aaa0 2023-10-16T12:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.207+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c003850 con 0x7f439415aaa0 2023-10-16T12:50:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.207+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43940c7280 con 0x7f439415aaa0 2023-10-16T12:50:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.207+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43940c69b0 con 0x7f439415aaa0 2023-10-16T12:50:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.208+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43940c7a40 con 0x7f439415aaa0 2023-10-16T12:50:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.210+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f437c0039b0 con 0x7f439415aaa0 2023-10-16T12:50:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.210+0000 7f438affd640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f437406af90 0x7f437406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.211+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f437c088e10 con 0x7f439415aaa0 2023-10-16T12:50:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.211+0000 7f439259c640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f437406af90 0x7f437406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.212+0000 7f439259c640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f437406af90 0x7f437406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4384036000 tx=0x7f438403b040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.215+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f437c08c030 con 0x7f439415aaa0 2023-10-16T12:50:52.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.409+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f439404e160 con 0x7f439415aaa0 2023-10-16T12:50:52.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.412+0000 7f438affd640 1 -- 172.21.15.60:0/1101898224 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16572 (secure 0 0 0) 0x7f437c017090 con 0x7f439415aaa0 2023-10-16T12:50:52.414 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:52.415 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":112,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:51.414027+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f437406af90 msgr2=0x7f437406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f437406af90 0x7f437406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4384036000 tx=0x7f438403b040 comp rx=0 tx=0).stop 2023-10-16T12:50:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 msgr2=0x7f43940acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f43940acb00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c410 tx=0x7f437c00c8d0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 shutdown_connections 2023-10-16T12:50:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f437406af90 0x7f437406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439415ae80 0x7f439414e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439415aaa0 0x7f43940acb00 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 --2- 172.21.15.60:0/1101898224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43940ce390 0x7f43940c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.415+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 >> 172.21.15.60:0/1101898224 conn(0x7f43940cb410 msgr2=0x7f43940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.416+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 shutdown_connections 2023-10-16T12:50:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.416+0000 7f4398d41640 1 -- 172.21.15.60:0/1101898224 wait complete. 2023-10-16T12:50:52.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:52.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.657+0000 7efce88ae640 1 Processor -- start 2023-10-16T12:50:52.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.657+0000 7efce88ae640 1 -- start start 2023-10-16T12:50:52.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.659+0000 7efce88ae640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 0x7efce40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.659+0000 7efce88ae640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce4160a90 con 0x7efce4070f20 2023-10-16T12:50:52.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.659+0000 7efce88ae640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40cd900 con 0x7efce415c9f0 2023-10-16T12:50:52.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.659+0000 7efce88ae640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce40cdec0 con 0x7efce415dde0 2023-10-16T12:50:52.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.659+0000 7efce359e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efce4070f20 0x7efce40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35328/0 (socket says 172.21.15.60:35328) 2023-10-16T12:50:52.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.659+0000 7efce359e640 1 -- 172.21.15.60:0/380832813 learned_addr learned my addr 172.21.15.60:0/380832813 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:52.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efcda59c640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 0x7efce40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120853265 0 0) 0x7efce4160a90 con 0x7efce4070f20 2023-10-16T12:50:52.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd0003520 con 0x7efce4070f20 2023-10-16T12:50:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274342968 0 0) 0x7efcd0003520 con 0x7efce4070f20 2023-10-16T12:50:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 msgr2=0x7efce40709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 0x7efce40709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 >> v1:172.21.15.144:6789/0 conn(0x7efce415c9f0 legacy=0x7efce40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40cdbe0 con 0x7efce4070f20 2023-10-16T12:50:52.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efcdc0019f0 con 0x7efce4070f20 2023-10-16T12:50:52.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 >> v1:172.21.15.60:6789/0 conn(0x7efce4070f20 legacy=0x7efce40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcd00039a0 0x7efcd0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 0x7efcd00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd0004d00 0x7efcd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd000b980 con 0x7efcd0004d00 2023-10-16T12:50:52.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd000bac0 con 0x7efcd00039a0 2023-10-16T12:50:52.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.660+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd000bc80 con 0x7efce415dde0 2023-10-16T12:50:52.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce359e640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd0004d00 0x7efcd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efcda59c640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 0x7efcd00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce2d9d640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcd00039a0 0x7efcd0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce359e640 1 -- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 msgr2=0x7efcd00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:52.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce359e640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efce415dde0 0x7efcd00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce359e640 1 -- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcd00039a0 msgr2=0x7efcd0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:52.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce359e640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcd00039a0 0x7efcd0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce359e640 1 -- 172.21.15.60:0/380832813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd0003520 con 0x7efcd0004d00 2023-10-16T12:50:52.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.661+0000 7efce2d9d640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcd00039a0 0x7efcd0003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:52.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.662+0000 7efce359e640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd0004d00 0x7efcd0005180 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00a830 tx=0x7efcd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f02cfcda7273f7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:52.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc02cea0 con 0x7efcd0004d00 2023-10-16T12:50:52.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc02f070 con 0x7efcd0004d00 2023-10-16T12:50:52.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce259c640 1 -- 172.21.15.60:0/380832813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc034890 con 0x7efcd0004d00 2023-10-16T12:50:52.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce88ae640 1 -- 172.21.15.60:0/380832813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd0004d00 msgr2=0x7efcd0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce88ae640 1 --2- 172.21.15.60:0/380832813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd0004d00 0x7efcd0005180 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00a830 tx=0x7efcd00057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce359e640 1 -- 172.21.15.60:0/380832813 reap_dead start 2023-10-16T12:50:52.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce88ae640 1 -- 172.21.15.60:0/380832813 shutdown_connections 2023-10-16T12:50:52.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce88ae640 1 -- 172.21.15.60:0/380832813 >> 172.21.15.60:0/380832813 conn(0x7efce40ce6a0 msgr2=0x7efce40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:52.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce88ae640 1 -- 172.21.15.60:0/380832813 shutdown_connections 2023-10-16T12:50:52.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.663+0000 7efce88ae640 1 -- 172.21.15.60:0/380832813 wait complete. 2023-10-16T12:50:52.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.665+0000 7efce88ae640 1 Processor -- start 2023-10-16T12:50:52.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.665+0000 7efce88ae640 1 -- start start 2023-10-16T12:50:52.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efce88ae640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcd0004d00 0x7efce4153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efce88ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 0x7efce40ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efce88ae640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efce406b060 0x7efce406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efce88ae640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce4154260 con 0x7efce40adf10 2023-10-16T12:50:52.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efce88ae640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce40ae8d0 con 0x7efce406b060 2023-10-16T12:50:52.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efce88ae640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce406ae40 con 0x7efcd0004d00 2023-10-16T12:50:52.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.666+0000 7efcda59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 0x7efce40ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 0x7efce40ae390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41238/0 (socket says 172.21.15.60:41238) 2023-10-16T12:50:52.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 -- 172.21.15.60:0/1317911829 learned_addr learned my addr 172.21.15.60:0/1317911829 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:52.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 -- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcd0004d00 msgr2=0x7efce4153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcd0004d00 0x7efce4153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 -- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efce406b060 msgr2=0x7efce406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efce406b060 0x7efce406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 -- 172.21.15.60:0/1317911829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce4069da0 con 0x7efce40adf10 2023-10-16T12:50:52.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcda59c640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 0x7efce40ae390 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7efccc011590 tx=0x7efccc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:52.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.667+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc004220 con 0x7efce40adf10 2023-10-16T12:50:52.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.668+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc004380 con 0x7efce40adf10 2023-10-16T12:50:52.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.668+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc015870 con 0x7efce40adf10 2023-10-16T12:50:52.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.668+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce406a080 con 0x7efce40adf10 2023-10-16T12:50:52.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.668+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce406a640 con 0x7efce40adf10 2023-10-16T12:50:52.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.669+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce404e9b0 con 0x7efce40adf10 2023-10-16T12:50:52.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.670+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efccc0159d0 con 0x7efce40adf10 2023-10-16T12:50:52.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.671+0000 7efcdbfff640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7efcc406afc0 0x7efcc406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:52.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.671+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efccc089930 con 0x7efce40adf10 2023-10-16T12:50:52.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.671+0000 7efce2d9d640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7efcc406afc0 0x7efcc406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:52.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.672+0000 7efce2d9d640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7efcc406afc0 0x7efcc406d480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efcd4000f60 tx=0x7efcd4009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:52.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.675+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efccc054830 con 0x7efce40adf10 2023-10-16T12:50:52.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.869+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efce40701b0 con 0x7efce40adf10 2023-10-16T12:50:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.871+0000 7efcdbfff640 1 -- 172.21.15.60:0/1317911829 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16572 (secure 0 0 0) 0x7efccc057e50 con 0x7efce40adf10 2023-10-16T12:50:52.873 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:52.874 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":112,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:51.414027+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7efcc406afc0 msgr2=0x7efcc406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7efcc406afc0 0x7efcc406d480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efcd4000f60 tx=0x7efcd4009040 comp rx=0 tx=0).stop 2023-10-16T12:50:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 msgr2=0x7efce40ae390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 0x7efce40ae390 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7efccc011590 tx=0x7efccc011a50 comp rx=0 tx=0).stop 2023-10-16T12:50:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 shutdown_connections 2023-10-16T12:50:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7efcc406afc0 0x7efcc406d480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efce406b060 0x7efce406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce40adf10 0x7efce40ae390 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 --2- 172.21.15.60:0/1317911829 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcd0004d00 0x7efce4153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 >> 172.21.15.60:0/1317911829 conn(0x7efce40ce6a0 msgr2=0x7efce40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 shutdown_connections 2023-10-16T12:50:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:52.874+0000 7efce88ae640 1 -- 172.21.15.60:0/1317911829 wait complete. 2023-10-16T12:50:52.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:50:53.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.112+0000 7f401ee8a640 1 Processor -- start 2023-10-16T12:50:53.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.113+0000 7f401ee8a640 1 -- start start 2023-10-16T12:50:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.114+0000 7f401ee8a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f401815c9f0 0x7f40180c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.114+0000 7f401ee8a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f401815dde0 0x7f40180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:53.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.114+0000 7f401ee8a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4018160a90 con 0x7f40180c9230 2023-10-16T12:50:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.114+0000 7f401ee8a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40180cdec0 con 0x7f401815dde0 2023-10-16T12:50:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.114+0000 7f401ee8a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40180cd900 con 0x7f401815c9f0 2023-10-16T12:50:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f401815dde0 0x7f40180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f401815dde0 0x7f40180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41250/0 (socket says 172.21.15.60:41250) 2023-10-16T12:50:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 -- 172.21.15.60:0/3101617542 learned_addr learned my addr 172.21.15.60:0/3101617542 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401de88640 1 --2- 172.21.15.60:0/3101617542 >> v2:172.21.15.144:3300/0 conn(0x7f401815c9f0 0x7f40180c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 -- 172.21.15.60:0/3101617542 >> v2:172.21.15.144:3300/0 conn(0x7f401815c9f0 msgr2=0x7f40180c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 --2- 172.21.15.60:0/3101617542 >> v2:172.21.15.144:3300/0 conn(0x7f401815c9f0 0x7f40180c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 -- 172.21.15.60:0/3101617542 >> v1:172.21.15.144:6789/0 conn(0x7f40180c9230 legacy=0x7f40180709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.115+0000 7f401e689640 1 -- 172.21.15.60:0/3101617542 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180cdbe0 con 0x7f401815dde0 2023-10-16T12:50:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.116+0000 7f401e689640 1 --2- 172.21.15.60:0/3101617542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815dde0 0x7f40180ac890 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f40180ad4e0 tx=0x7f401002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6674e9417312ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ce86640 1 -- 172.21.15.60:0/3101617542 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401002ca80 con 0x7f401815dde0 2023-10-16T12:50:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ce86640 1 -- 172.21.15.60:0/3101617542 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401002cbe0 con 0x7f401815dde0 2023-10-16T12:50:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ce86640 1 -- 172.21.15.60:0/3101617542 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40100343f0 con 0x7f401815dde0 2023-10-16T12:50:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 -- 172.21.15.60:0/3101617542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815dde0 msgr2=0x7f40180ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3101617542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815dde0 0x7f40180ac890 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f40180ad4e0 tx=0x7f401002bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 -- 172.21.15.60:0/3101617542 shutdown_connections 2023-10-16T12:50:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3101617542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815dde0 0x7f40180ac890 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3101617542 >> v2:172.21.15.144:3300/0 conn(0x7f401815c9f0 0x7f40180c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 -- 172.21.15.60:0/3101617542 >> 172.21.15.60:0/3101617542 conn(0x7f40180ce6a0 msgr2=0x7f40180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 -- 172.21.15.60:0/3101617542 shutdown_connections 2023-10-16T12:50:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.117+0000 7f401ee8a640 1 -- 172.21.15.60:0/3101617542 wait complete. 2023-10-16T12:50:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.119+0000 7f401ee8a640 1 Processor -- start 2023-10-16T12:50:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.119+0000 7f401ee8a640 1 -- start start 2023-10-16T12:50:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401ee8a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40180c9230 0x7f40181515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401ee8a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 0x7f40181535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401ee8a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401815dde0 0x7f40181457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401ee8a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180aed80 con 0x7f401815c9f0 2023-10-16T12:50:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401ee8a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181548f0 con 0x7f401815dde0 2023-10-16T12:50:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401ee8a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40181465e0 con 0x7f40180c9230 2023-10-16T12:50:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.120+0000 7f401d687640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 0x7f40181535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 0x7f40181535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41252/0 (socket says 172.21.15.60:41252) 2023-10-16T12:50:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 -- 172.21.15.60:0/3516432234 learned_addr learned my addr 172.21.15.60:0/3516432234 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 -- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40180c9230 msgr2=0x7f40181515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401de88640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40180c9230 0x7f40181515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401e689640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401815dde0 0x7f40181457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40180c9230 0x7f40181515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 -- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401815dde0 msgr2=0x7f40181457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401815dde0 0x7f40181457b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 -- 172.21.15.60:0/3516432234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4010009b30 con 0x7f401815c9f0 2023-10-16T12:50:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401e689640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401815dde0 0x7f40181457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.121+0000 7f401d687640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 0x7f40181535f0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f400800c440 tx=0x7f400800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.122+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008005480 con 0x7f401815c9f0 2023-10-16T12:50:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.122+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40080035f0 con 0x7f401815c9f0 2023-10-16T12:50:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.122+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080039c0 con 0x7f401815c9f0 2023-10-16T12:50:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.122+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4018154050 con 0x7f401815c9f0 2023-10-16T12:50:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.122+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018154610 con 0x7f401815c9f0 2023-10-16T12:50:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.123+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401804e9b0 con 0x7f401815c9f0 2023-10-16T12:50:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.124+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4008003b20 con 0x7f401815c9f0 2023-10-16T12:50:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.125+0000 7f400e7fc640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f3ffc06af90 0x7f3ffc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.125+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40080889a0 con 0x7f401815c9f0 2023-10-16T12:50:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.126+0000 7f401de88640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f3ffc06af90 0x7f3ffc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.127+0000 7f401de88640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f3ffc06af90 0x7f3ffc06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4014006fd0 tx=0x7f4014008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.129+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4008053930 con 0x7f401815c9f0 2023-10-16T12:50:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.362+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f40180adac0 con 0x7f401815c9f0 2023-10-16T12:50:53.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.973+0000 7f400e7fc640 1 -- 172.21.15.60:0/3516432234 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v52) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4008017020 con 0x7f401815c9f0 2023-10-16T12:50:53.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f3ffc06af90 msgr2=0x7f3ffc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:53.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f3ffc06af90 0x7f3ffc06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4014006fd0 tx=0x7f4014008040 comp rx=0 tx=0).stop 2023-10-16T12:50:53.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 msgr2=0x7f40181535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:53.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 0x7f40181535f0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f400800c440 tx=0x7f400800c900 comp rx=0 tx=0).stop 2023-10-16T12:50:53.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 shutdown_connections 2023-10-16T12:50:53.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f3ffc06af90 0x7f3ffc06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f401815dde0 0x7f40181457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f401815c9f0 0x7f40181535f0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 --2- 172.21.15.60:0/3516432234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40180c9230 0x7f40181515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:53.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 >> 172.21.15.60:0/3516432234 conn(0x7f40180ce6a0 msgr2=0x7f401815a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:53.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.976+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 shutdown_connections 2023-10-16T12:50:53.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:53.977+0000 7f401ee8a640 1 -- 172.21.15.60:0/3516432234 wait complete. 2023-10-16T12:50:53.999 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:50:53.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.227+0000 7f098666d640 1 Processor -- start 2023-10-16T12:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.228+0000 7f098666d640 1 -- start start 2023-10-16T12:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.229+0000 7f098666d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09801630e0 0x7f09801654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.229+0000 7f098666d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098015ac60 con 0x7f09800ad4d0 2023-10-16T12:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f098666d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0980070030 con 0x7f0980153740 2023-10-16T12:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f098666d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0980070310 con 0x7f09801630e0 2023-10-16T12:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f0985e6c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09801630e0 0x7f09801654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f0985e6c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09801630e0 0x7f09801654d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41264/0 (socket says 172.21.15.60:41264) 2023-10-16T12:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f0985e6c640 1 -- 172.21.15.60:0/2526456739 learned_addr learned my addr 172.21.15.60:0/2526456739 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f0984e6a640 1 --1- 172.21.15.60:0/2526456739 >> v1:172.21.15.60:6789/0 conn(0x7f09800ad4d0 0x7f09800ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35342/0 (socket says 172.21.15.60:35342) 2023-10-16T12:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.230+0000 7f0985e6c640 1 -- 172.21.15.60:0/2526456739 >> v1:172.21.15.144:6789/0 conn(0x7f0980153740 legacy=0x7f098015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.231+0000 7f0985e6c640 1 -- 172.21.15.60:0/2526456739 >> v1:172.21.15.60:6789/0 conn(0x7f09800ad4d0 legacy=0x7f09800ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.231+0000 7f0985e6c640 1 -- 172.21.15.60:0/2526456739 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09800705f0 con 0x7f09801630e0 2023-10-16T12:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.232+0000 7f0985e6c640 1 --2- 172.21.15.60:0/2526456739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f09801654d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f0978009870 tx=0x7f097802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7fc6f333c1919b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.232+0000 7f0977fff640 1 -- 172.21.15.60:0/2526456739 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097802f850 con 0x7f09801630e0 2023-10-16T12:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.232+0000 7f0977fff640 1 -- 172.21.15.60:0/2526456739 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097802f9b0 con 0x7f09801630e0 2023-10-16T12:50:54.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.232+0000 7f0977fff640 1 -- 172.21.15.60:0/2526456739 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097802fd40 con 0x7f09801630e0 2023-10-16T12:50:54.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 -- 172.21.15.60:0/2526456739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 msgr2=0x7f09801654d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:54.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 --2- 172.21.15.60:0/2526456739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f09801654d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f0978009870 tx=0x7f097802ba30 comp rx=0 tx=0).stop 2023-10-16T12:50:54.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 -- 172.21.15.60:0/2526456739 shutdown_connections 2023-10-16T12:50:54.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 --2- 172.21.15.60:0/2526456739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f09801654d0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 -- 172.21.15.60:0/2526456739 >> 172.21.15.60:0/2526456739 conn(0x7f09800ce6a0 msgr2=0x7f09800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:54.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 -- 172.21.15.60:0/2526456739 shutdown_connections 2023-10-16T12:50:54.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.233+0000 7f098666d640 1 -- 172.21.15.60:0/2526456739 wait complete. 2023-10-16T12:50:54.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.235+0000 7f098666d640 1 Processor -- start 2023-10-16T12:50:54.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.235+0000 7f098666d640 1 -- start start 2023-10-16T12:50:54.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098666d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09800ad4d0 0x7f09800c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:54.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098666d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0980153740 0x7f09801520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:54.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098666d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f0980145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098666d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0980155590 con 0x7f09801630e0 2023-10-16T12:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098666d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0980145ec0 con 0x7f0980153740 2023-10-16T12:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098666d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09800c2fa0 con 0x7f09800ad4d0 2023-10-16T12:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f0985e6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f0980145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f0985e6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f0980145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41278/0 (socket says 172.21.15.60:41278) 2023-10-16T12:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f0985e6c640 1 -- 172.21.15.60:0/36704797 learned_addr learned my addr 172.21.15.60:0/36704797 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f0984e6a640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0980153740 0x7f09801520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f098566b640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09800ad4d0 0x7f09800c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.236+0000 7f0985e6c640 1 -- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09800ad4d0 msgr2=0x7f09800c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0985e6c640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09800ad4d0 0x7f09800c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0985e6c640 1 -- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0980153740 msgr2=0x7f09801520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:54.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0985e6c640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0980153740 0x7f09801520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0985e6c640 1 -- 172.21.15.60:0/36704797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780094d0 con 0x7f09801630e0 2023-10-16T12:50:54.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0984e6a640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0980153740 0x7f09801520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:54.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f098566b640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09800ad4d0 0x7f09800c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0985e6c640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f0980145980 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f0978009870 tx=0x7f0978004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097802f7e0 con 0x7f09801630e0 2023-10-16T12:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0978041400 con 0x7f09801630e0 2023-10-16T12:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09780417b0 con 0x7f09801630e0 2023-10-16T12:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0980154f40 con 0x7f09801630e0 2023-10-16T12:50:54.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.237+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0980151e10 con 0x7f09801630e0 2023-10-16T12:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.239+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0980155740 con 0x7f09801630e0 2023-10-16T12:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.240+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0978041910 con 0x7f09801630e0 2023-10-16T12:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.240+0000 7f0975ffb640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f096406af90 0x7f096406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.241+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f097803c070 con 0x7f09801630e0 2023-10-16T12:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.241+0000 7f098566b640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f096406af90 0x7f096406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.242+0000 7f098566b640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f096406af90 0x7f096406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f097c0012e0 tx=0x7f097c009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.245+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0980155740 con 0x7f09801630e0 2023-10-16T12:50:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.479+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f098006fc30 con 0x7f09801630e0 2023-10-16T12:50:54.858 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T12:50:54.857+0000 7f950a96f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:50:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.986+0000 7f0975ffb640 1 -- 172.21.15.60:0/36704797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v53) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09780775c0 con 0x7f09801630e0 2023-10-16T12:50:54.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f096406af90 msgr2=0x7f096406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f096406af90 0x7f096406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f097c0012e0 tx=0x7f097c009040 comp rx=0 tx=0).stop 2023-10-16T12:50:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 msgr2=0x7f0980145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f0980145980 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f0978009870 tx=0x7f0978004730 comp rx=0 tx=0).stop 2023-10-16T12:50:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 shutdown_connections 2023-10-16T12:50:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f096406af90 0x7f096406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09801630e0 0x7f0980145980 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0980153740 0x7f09801520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 --2- 172.21.15.60:0/36704797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09800ad4d0 0x7f09800c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 >> 172.21.15.60:0/36704797 conn(0x7f09800ce6a0 msgr2=0x7f0980158dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 shutdown_connections 2023-10-16T12:50:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:54.990+0000 7f098666d640 1 -- 172.21.15.60:0/36704797 wait complete. 2023-10-16T12:50:55.011 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:50:55.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.238+0000 7f35e1c67640 1 Processor -- start 2023-10-16T12:50:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.238+0000 7f35e1c67640 1 -- start start 2023-10-16T12:50:55.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.239+0000 7f35e1c67640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35dc0ad970 0x7f35dc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35e1c67640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc15b9d0 con 0x7f35dc06c040 2023-10-16T12:50:55.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35e1c67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc15bcb0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35e1c67640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc15bf90 con 0x7f35dc0ad970 2023-10-16T12:50:55.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35e0c65640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f35dc06c040 0x7f35dc06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35350/0 (socket says 172.21.15.60:35350) 2023-10-16T12:50:55.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35e0c65640 1 -- 172.21.15.60:0/1770359275 learned_addr learned my addr 172.21.15.60:0/1770359275 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35d3fff640 1 --2- 172.21.15.60:0/1770359275 >> v2:172.21.15.60:3300/0 conn(0x7f35dc0ad970 0x7f35dc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35d3fff640 1 -- 172.21.15.60:0/1770359275 >> v1:172.21.15.144:6789/0 conn(0x7f35dc0ccfb0 legacy=0x7f35dc0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35d3fff640 1 -- 172.21.15.60:0/1770359275 >> v1:172.21.15.60:6789/0 conn(0x7f35dc06c040 legacy=0x7f35dc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35d37fe640 1 -- 172.21.15.60:0/1770359275 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614900952 0 0) 0x7f35dc15b9d0 con 0x7f35dc06c040 2023-10-16T12:50:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.240+0000 7f35d3fff640 1 -- 172.21.15.60:0/1770359275 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0cc270 con 0x7f35dc0ad970 2023-10-16T12:50:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.241+0000 7f35d3fff640 1 --2- 172.21.15.60:0/1770359275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ad970 0x7f35dc0cca70 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0097d0 tx=0x7f35cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a276b27db3b201a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35d37fe640 1 -- 172.21.15.60:0/1770359275 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc031830 con 0x7f35dc0ad970 2023-10-16T12:50:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35d37fe640 1 -- 172.21.15.60:0/1770359275 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc031990 con 0x7f35dc0ad970 2023-10-16T12:50:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35d37fe640 1 -- 172.21.15.60:0/1770359275 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc031d20 con 0x7f35dc0ad970 2023-10-16T12:50:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35e1c67640 1 -- 172.21.15.60:0/1770359275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ad970 msgr2=0x7f35dc0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1770359275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ad970 0x7f35dc0cca70 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0097d0 tx=0x7f35cc02b8d0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35e1c67640 1 -- 172.21.15.60:0/1770359275 shutdown_connections 2023-10-16T12:50:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1770359275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ad970 0x7f35dc0cca70 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.242+0000 7f35e1c67640 1 -- 172.21.15.60:0/1770359275 >> 172.21.15.60:0/1770359275 conn(0x7f35dc0ce6a0 msgr2=0x7f35dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.243+0000 7f35e1c67640 1 -- 172.21.15.60:0/1770359275 shutdown_connections 2023-10-16T12:50:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.243+0000 7f35e1c67640 1 -- 172.21.15.60:0/1770359275 wait complete. 2023-10-16T12:50:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.244+0000 7f35e1c67640 1 Processor -- start 2023-10-16T12:50:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.244+0000 7f35e1c67640 1 -- start start 2023-10-16T12:50:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.245+0000 7f35e1c67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35dc06c040 0x7f35dc0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.245+0000 7f35e1c67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35dc0ad970 0x7f35dc159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.245+0000 7f35e1c67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 0x7f35dc0c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.245+0000 7f35e1c67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc0b4100 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.245+0000 7f35e1c67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc15a2c0 con 0x7f35dc06c040 2023-10-16T12:50:55.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.245+0000 7f35e1c67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc0b3e20 con 0x7f35dc0ad970 2023-10-16T12:50:55.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 0x7f35dc0c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 0x7f35dc0c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41284/0 (socket says 172.21.15.60:41284) 2023-10-16T12:50:55.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 -- 172.21.15.60:0/1737026355 learned_addr learned my addr 172.21.15.60:0/1737026355 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:55.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e0c65640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35dc06c040 0x7f35dc0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35d3fff640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35dc0ad970 0x7f35dc159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 -- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35dc0ad970 msgr2=0x7f35dc159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35dc0ad970 0x7f35dc159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 -- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35dc06c040 msgr2=0x7f35dc0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35dc06c040 0x7f35dc0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.246+0000 7f35e1466640 1 -- 172.21.15.60:0/1737026355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc009400 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.247+0000 7f35e1466640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 0x7f35dc0c6ab0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c780 tx=0x7f35d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:55.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.247+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4003d80 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.247+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d400ce90 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.247+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc0c62b0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.247+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d40052b0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.247+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc0c6ff0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.248+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35dc06b330 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.249+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35d4005410 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.250+0000 7f35d17fa640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f35c806af90 0x7f35c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.250+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35d4088fe0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.251+0000 7f35e0c65640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f35c806af90 0x7f35c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.252+0000 7f35e0c65640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f35c806af90 0x7f35c806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f35d8000f60 tx=0x7f35d8007c00 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:55.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.254+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35d4053f70 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.477+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35dc0c67d0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.478+0000 7f35d17fa640 1 -- 172.21.15.60:0/1737026355 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f35dc0c67d0 con 0x7f35dc0ccfb0 2023-10-16T12:50:55.480 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:55.481 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:51.414872+0000","modified":"2023-10-16T12:50:54.480441+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7353":{"gid":7353,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/3569389918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3569389918},{"type":"v1","addr":"172.21.15.60:6835","nonce":3569389918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:50:55.481 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 53 2023-10-16T12:50:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f35c806af90 msgr2=0x7f35c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f35c806af90 0x7f35c806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f35d8000f60 tx=0x7f35d8007c00 comp rx=0 tx=0).stop 2023-10-16T12:50:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 msgr2=0x7f35dc0c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 0x7f35dc0c6ab0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c780 tx=0x7f35d400cc40 comp rx=0 tx=0).stop 2023-10-16T12:50:55.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 shutdown_connections 2023-10-16T12:50:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f35c806af90 0x7f35c806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35dc0ad970 0x7f35dc159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35dc0ccfb0 0x7f35dc0c6ab0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 --2- 172.21.15.60:0/1737026355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35dc06c040 0x7f35dc0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 >> 172.21.15.60:0/1737026355 conn(0x7f35dc0ce6a0 msgr2=0x7f35dc0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:55.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 shutdown_connections 2023-10-16T12:50:55.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.481+0000 7f35e1c67640 1 -- 172.21.15.60:0/1737026355 wait complete. 2023-10-16T12:50:55.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:50:55.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.722+0000 7f74994e6640 1 Processor -- start 2023-10-16T12:50:55.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.722+0000 7f74994e6640 1 -- start start 2023-10-16T12:50:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.723+0000 7f74994e6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f749415cd70 0x7f74940d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.723+0000 7f74994e6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749415d150 con 0x7f74940cded0 2023-10-16T12:50:55.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.723+0000 7f74994e6640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74940d1180 con 0x7f74940b8480 2023-10-16T12:50:55.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.723+0000 7f74994e6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74940d1460 con 0x7f749415cd70 2023-10-16T12:50:55.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.724+0000 7f7493fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f749415cd70 0x7f74940d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.724+0000 7f7498ce5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f74940cded0 0x7f74940b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35360/0 (socket says 172.21.15.60:35360) 2023-10-16T12:50:55.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.724+0000 7f7498ce5640 1 -- 172.21.15.60:0/3865729502 learned_addr learned my addr 172.21.15.60:0/3865729502 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:55.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.724+0000 7f7493fff640 1 -- 172.21.15.60:0/3865729502 >> v1:172.21.15.144:6789/0 conn(0x7f74940b8480 legacy=0x7f74940cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:55.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.724+0000 7f7493fff640 1 -- 172.21.15.60:0/3865729502 >> v1:172.21.15.60:6789/0 conn(0x7f74940cded0 legacy=0x7f74940b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.724+0000 7f7493fff640 1 -- 172.21.15.60:0/3865729502 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74940d1740 con 0x7f749415cd70 2023-10-16T12:50:55.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.725+0000 7f7493fff640 1 --2- 172.21.15.60:0/3865729502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749415cd70 0x7f74940d2220 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7484009d90 tx=0x7f748402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4aa9fb60928f9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:55.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f7492ffd640 1 -- 172.21.15.60:0/3865729502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748402cc60 con 0x7f749415cd70 2023-10-16T12:50:55.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f7492ffd640 1 -- 172.21.15.60:0/3865729502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748402cdc0 con 0x7f749415cd70 2023-10-16T12:50:55.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f7492ffd640 1 -- 172.21.15.60:0/3865729502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484034660 con 0x7f749415cd70 2023-10-16T12:50:55.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f74994e6640 1 -- 172.21.15.60:0/3865729502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749415cd70 msgr2=0x7f74940d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f74994e6640 1 --2- 172.21.15.60:0/3865729502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749415cd70 0x7f74940d2220 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7484009d90 tx=0x7f748402beb0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f74994e6640 1 -- 172.21.15.60:0/3865729502 shutdown_connections 2023-10-16T12:50:55.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.726+0000 7f74994e6640 1 --2- 172.21.15.60:0/3865729502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749415cd70 0x7f74940d2220 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.727+0000 7f74994e6640 1 -- 172.21.15.60:0/3865729502 >> 172.21.15.60:0/3865729502 conn(0x7f74940cf570 msgr2=0x7f74940cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:55.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.727+0000 7f74994e6640 1 -- 172.21.15.60:0/3865729502 shutdown_connections 2023-10-16T12:50:55.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.727+0000 7f74994e6640 1 -- 172.21.15.60:0/3865729502 wait complete. 2023-10-16T12:50:55.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.728+0000 7f74994e6640 1 Processor -- start 2023-10-16T12:50:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.729+0000 7f74994e6640 1 -- start start 2023-10-16T12:50:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74994e6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74940b8480 0x7f74941f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74994e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 0x7f74941f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74994e6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749415cd70 0x7f749415e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74994e6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74940cad90 con 0x7f74940cded0 2023-10-16T12:50:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74994e6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74941f5e30 con 0x7f749415cd70 2023-10-16T12:50:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74994e6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74941f2f80 con 0x7f74940b8480 2023-10-16T12:50:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 0x7f74941f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 0x7f74941f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41300/0 (socket says 172.21.15.60:41300) 2023-10-16T12:50:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 -- 172.21.15.60:0/1167104787 learned_addr learned my addr 172.21.15.60:0/1167104787 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f7498ce5640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749415cd70 0x7f749415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 -- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74940b8480 msgr2=0x7f74941f58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f7493fff640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74940b8480 0x7f74941f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74940b8480 0x7f74941f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 -- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749415cd70 msgr2=0x7f749415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749415cd70 0x7f749415e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f74937fe640 1 -- 172.21.15.60:0/1167104787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484009a70 con 0x7f74940cded0 2023-10-16T12:50:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.730+0000 7f7498ce5640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749415cd70 0x7f749415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.731+0000 7f74937fe640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 0x7f74941f2a40 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f747c00c430 tx=0x7f747c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.731+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c011790 con 0x7f74940cded0 2023-10-16T12:50:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.731+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c0118f0 con 0x7f74940cded0 2023-10-16T12:50:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.731+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c011ca0 con 0x7f74940cded0 2023-10-16T12:50:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.731+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74941564d0 con 0x7f74940cded0 2023-10-16T12:50:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.731+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74940caab0 con 0x7f74940cded0 2023-10-16T12:50:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.732+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74940cbf00 con 0x7f74940cded0 2023-10-16T12:50:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.733+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f747c01c460 con 0x7f74940cded0 2023-10-16T12:50:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.734+0000 7f7490ff9640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f747406b010 0x7f747406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.734+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f747c0896e0 con 0x7f74940cded0 2023-10-16T12:50:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.735+0000 7f7493fff640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f747406b010 0x7f747406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.736+0000 7f7493fff640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f747406b010 0x7f747406d4d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7484036000 tx=0x7f7484030000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.739+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74940cbf00 con 0x7f74940cded0 2023-10-16T12:50:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.939+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74940cbf00 con 0x7f74940cded0 2023-10-16T12:50:55.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.942+0000 7f7490ff9640 1 -- 172.21.15.60:0/1167104787 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16559 (secure 0 0 0) 0x7f74940cbf00 con 0x7f74940cded0 2023-10-16T12:50:55.944 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:55.945 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":113,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:50:52.958571+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:50:55.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f747406b010 msgr2=0x7f747406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f747406b010 0x7f747406d4d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7484036000 tx=0x7f7484030000 comp rx=0 tx=0).stop 2023-10-16T12:50:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 msgr2=0x7f74941f2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:55.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 0x7f74941f2a40 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f747c00c430 tx=0x7f747c00c8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 shutdown_connections 2023-10-16T12:50:55.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f747406b010 0x7f747406d4d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749415cd70 0x7f749415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74940cded0 0x7f74941f2a40 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 --2- 172.21.15.60:0/1167104787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74940b8480 0x7f74941f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:55.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.945+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 >> 172.21.15.60:0/1167104787 conn(0x7f74940cf570 msgr2=0x7f74940b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:55.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.946+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 shutdown_connections 2023-10-16T12:50:55.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:55.946+0000 7f74994e6640 1 -- 172.21.15.60:0/1167104787 wait complete. 2023-10-16T12:50:55.967 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:50:55.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:50:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.194+0000 7ff9c3f0f640 1 Processor -- start 2023-10-16T12:50:56.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.194+0000 7ff9c3f0f640 1 -- start start 2023-10-16T12:50:56.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c3f0f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9bc159b20 0x7ff9bc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c3f0f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff9bc159f00 0x7ff9bc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c3f0f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c3f0f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9bc06f7f0 con 0x7ff9bc159b20 2023-10-16T12:50:56.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c3f0f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9bc06c5e0 con 0x7ff9bc159f00 2023-10-16T12:50:56.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c3f0f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc06c8c0 con 0x7ff9bc06c120 2023-10-16T12:50:56.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c1cab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9bc159b20 0x7ff9bc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c1cab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9bc159b20 0x7ff9bc06bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41310/0 (socket says 172.21.15.60:41310) 2023-10-16T12:50:56.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.196+0000 7ff9c1cab640 1 -- 172.21.15.60:0/1943781135 learned_addr learned my addr 172.21.15.60:0/1943781135 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:56.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c14aa640 1 --2- 172.21.15.60:0/1943781135 >> v2:172.21.15.144:3300/0 conn(0x7ff9bc159f00 0x7ff9bc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c24ac640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c1cab640 1 -- 172.21.15.60:0/1943781135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 msgr2=0x7ff9bc0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c1cab640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c1cab640 1 -- 172.21.15.60:0/1943781135 >> v2:172.21.15.144:3300/0 conn(0x7ff9bc159f00 msgr2=0x7ff9bc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c1cab640 1 --2- 172.21.15.60:0/1943781135 >> v2:172.21.15.144:3300/0 conn(0x7ff9bc159f00 0x7ff9bc0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c1cab640 1 -- 172.21.15.60:0/1943781135 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc06cba0 con 0x7ff9bc159b20 2023-10-16T12:50:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.197+0000 7ff9c24ac640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:56.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.198+0000 7ff9c1cab640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc06bbe0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc15d770 tx=0x7ff9b802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809ffd690cee4ad5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:56.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.198+0000 7ff9c0ca9640 1 -- 172.21.15.60:0/1943781135 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8031860 con 0x7ff9bc159b20 2023-10-16T12:50:56.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.198+0000 7ff9c0ca9640 1 -- 172.21.15.60:0/1943781135 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b80319c0 con 0x7ff9bc159b20 2023-10-16T12:50:56.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.198+0000 7ff9c0ca9640 1 -- 172.21.15.60:0/1943781135 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8031d50 con 0x7ff9bc159b20 2023-10-16T12:50:56.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/1943781135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 msgr2=0x7ff9bc06bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc06bbe0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc15d770 tx=0x7ff9b802b900 comp rx=0 tx=0).stop 2023-10-16T12:50:56.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/1943781135 shutdown_connections 2023-10-16T12:50:56.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/1943781135 >> v2:172.21.15.144:3300/0 conn(0x7ff9bc159f00 0x7ff9bc0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/1943781135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc06bbe0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/1943781135 >> 172.21.15.60:0/1943781135 conn(0x7ff9bc0cb410 msgr2=0x7ff9bc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:56.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/1943781135 shutdown_connections 2023-10-16T12:50:56.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.199+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/1943781135 wait complete. 2023-10-16T12:50:56.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.201+0000 7ff9c3f0f640 1 Processor -- start 2023-10-16T12:50:56.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.201+0000 7ff9c3f0f640 1 -- start start 2023-10-16T12:50:56.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.202+0000 7ff9c3f0f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.202+0000 7ff9c3f0f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.202+0000 7ff9c3f0f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9bc159f00 0x7ff9bc15ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.202+0000 7ff9c3f0f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc0c4a60 con 0x7ff9bc159b20 2023-10-16T12:50:56.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.202+0000 7ff9c3f0f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc0c4d40 con 0x7ff9bc159f00 2023-10-16T12:50:56.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.202+0000 7ff9c3f0f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc0c5020 con 0x7ff9bc06c120 2023-10-16T12:50:56.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc15c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41312/0 (socket says 172.21.15.60:41312) 2023-10-16T12:50:56.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 -- 172.21.15.60:0/4220735645 learned_addr learned my addr 172.21.15.60:0/4220735645 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:56.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c1cab640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c24ac640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9bc159f00 0x7ff9bc15ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 -- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 msgr2=0x7ff9bc0c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 -- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9bc159f00 msgr2=0x7ff9bc15ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9bc159f00 0x7ff9bc15ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.203+0000 7ff9c14aa640 1 -- 172.21.15.60:0/4220735645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b80094d0 con 0x7ff9bc159b20 2023-10-16T12:50:56.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.204+0000 7ff9c14aa640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc15c4e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00d7d0 tx=0x7ff9ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:56.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.204+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00b6c0 con 0x7ff9bc159b20 2023-10-16T12:50:56.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.204+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ac00b820 con 0x7ff9bc159b20 2023-10-16T12:50:56.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.204+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00bbb0 con 0x7ff9bc159b20 2023-10-16T12:50:56.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.204+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9bc0c5300 con 0x7ff9bc159b20 2023-10-16T12:50:56.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.204+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9bc0c46a0 con 0x7ff9bc159b20 2023-10-16T12:50:56.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.205+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9bc0c4830 con 0x7ff9bc159b20 2023-10-16T12:50:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.206+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9ac00bd10 con 0x7ff9bc159b20 2023-10-16T12:50:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.207+0000 7ff9b27fc640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7ff9a006af90 0x7ff9a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.207+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9ac089c00 con 0x7ff9bc159b20 2023-10-16T12:50:56.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.208+0000 7ff9c1cab640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7ff9a006af90 0x7ff9a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.209+0000 7ff9c1cab640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7ff9a006af90 0x7ff9a006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8036000 tx=0x7ff9b803c040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:56.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.212+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9ac054b00 con 0x7ff9bc159b20 2023-10-16T12:50:56.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.404+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 --> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9bc0c4eb0 con 0x7ff9a006af90 2023-10-16T12:50:56.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.434+0000 7ff9b27fc640 1 -- 172.21.15.60:0/4220735645 <== mgr.5705 v2:172.21.15.144:6832/3513615444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287168 (secure 0 0 0) 0x7ff9bc0c4eb0 con 0x7ff9a006af90 2023-10-16T12:50:56.440 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:50:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.440+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7ff9a006af90 msgr2=0x7ff9a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.440+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7ff9a006af90 0x7ff9a006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8036000 tx=0x7ff9b803c040 comp rx=0 tx=0).stop 2023-10-16T12:50:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.440+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 msgr2=0x7ff9bc15c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.440+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc15c4e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00d7d0 tx=0x7ff9ac00dc90 comp rx=0 tx=0).stop 2023-10-16T12:50:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.440+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 shutdown_connections 2023-10-16T12:50:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7ff9a006af90 0x7ff9a006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9bc159f00 0x7ff9bc15ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9bc159b20 0x7ff9bc15c4e0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 --2- 172.21.15.60:0/4220735645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9bc06c120 0x7ff9bc0c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 >> 172.21.15.60:0/4220735645 conn(0x7ff9bc0cb410 msgr2=0x7ff9bc0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 shutdown_connections 2023-10-16T12:50:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.441+0000 7ff9c3f0f640 1 -- 172.21.15.60:0/4220735645 wait complete. 2023-10-16T12:50:56.463 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T12:50:56.284069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":327,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330692,"kb_used_data":11856,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754644028,"statfs":{"total":773094113280,"available":772755484672,"internally_reserved":0,"allocated":12140544,"data_stored":6388456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":326356536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001904"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992506+0000","last_change":"2023-10-16T12:50:51.985499+0000","last_active":"2023-10-16T12:50:51.992506+0000","last_peered":"2023-10-16T12:50:51.992506+0000","last_clean":"2023-10-16T12:50:51.992506+0000","last_became_active":"2023-10-16T12:50:51.984856+0000","last_became_peered":"2023-10-16T12:50:51.984856+0000","last_unstale":"2023-10-16T12:50:51.992506+0000","last_undegraded":"2023-10-16T12:50:51.992506+0000","last_fullsized":"2023-10-16T12:50:51.992506+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:45.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984788+0000","last_change":"2023-10-16T12:50:51.979288+0000","last_active":"2023-10-16T12:50:51.984788+0000","last_peered":"2023-10-16T12:50:51.984788+0000","last_clean":"2023-10-16T12:50:51.984788+0000","last_became_active":"2023-10-16T12:50:51.978924+0000","last_became_peered":"2023-10-16T12:50:51.978924+0000","last_unstale":"2023-10-16T12:50:51.984788+0000","last_undegraded":"2023-10-16T12:50:51.984788+0000","last_fullsized":"2023-10-16T12:50:51.984788+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:08.207898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993777+0000","last_change":"2023-10-16T12:50:51.992352+0000","last_active":"2023-10-16T12:50:51.993777+0000","last_peered":"2023-10-16T12:50:51.993777+0000","last_clean":"2023-10-16T12:50:51.993777+0000","last_became_active":"2023-10-16T12:50:51.991858+0000","last_became_peered":"2023-10-16T12:50:51.991858+0000","last_unstale":"2023-10-16T12:50:51.993777+0000","last_undegraded":"2023-10-16T12:50:51.993777+0000","last_fullsized":"2023-10-16T12:50:51.993777+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:16.349362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995687+0000","last_change":"2023-10-16T12:50:51.989011+0000","last_active":"2023-10-16T12:50:51.995687+0000","last_peered":"2023-10-16T12:50:51.995687+0000","last_clean":"2023-10-16T12:50:51.995687+0000","last_became_active":"2023-10-16T12:50:51.988765+0000","last_became_peered":"2023-10-16T12:50:51.988765+0000","last_unstale":"2023-10-16T12:50:51.995687+0000","last_undegraded":"2023-10-16T12:50:51.995687+0000","last_fullsized":"2023-10-16T12:50:51.995687+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:06.346666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991230+0000","last_change":"2023-10-16T12:50:51.990368+0000","last_active":"2023-10-16T12:50:51.991230+0000","last_peered":"2023-10-16T12:50:51.991230+0000","last_clean":"2023-10-16T12:50:51.991230+0000","last_became_active":"2023-10-16T12:50:51.990043+0000","last_became_peered":"2023-10-16T12:50:51.990043+0000","last_unstale":"2023-10-16T12:50:51.991230+0000","last_undegraded":"2023-10-16T12:50:51.991230+0000","last_fullsized":"2023-10-16T12:50:51.991230+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:58.838268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.986570+0000","last_change":"2023-10-16T12:50:51.983430+0000","last_active":"2023-10-16T12:50:51.986570+0000","last_peered":"2023-10-16T12:50:51.986570+0000","last_clean":"2023-10-16T12:50:51.986570+0000","last_became_active":"2023-10-16T12:50:51.983148+0000","last_became_peered":"2023-10-16T12:50:51.983148+0000","last_unstale":"2023-10-16T12:50:51.986570+0000","last_undegraded":"2023-10-16T12:50:51.986570+0000","last_fullsized":"2023-10-16T12:50:51.986570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:43.803664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984997+0000","last_change":"2023-10-16T12:50:51.980807+0000","last_active":"2023-10-16T12:50:51.984997+0000","last_peered":"2023-10-16T12:50:51.984997+0000","last_clean":"2023-10-16T12:50:51.984997+0000","last_became_active":"2023-10-16T12:50:51.980523+0000","last_became_peered":"2023-10-16T12:50:51.980523+0000","last_unstale":"2023-10-16T12:50:51.984997+0000","last_undegraded":"2023-10-16T12:50:51.984997+0000","last_fullsized":"2023-10-16T12:50:51.984997+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:12.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.973686+0000","last_change":"2023-10-16T12:50:51.971728+0000","last_active":"2023-10-16T12:50:51.973686+0000","last_peered":"2023-10-16T12:50:51.973686+0000","last_clean":"2023-10-16T12:50:51.973686+0000","last_became_active":"2023-10-16T12:50:51.971426+0000","last_became_peered":"2023-10-16T12:50:51.971426+0000","last_unstale":"2023-10-16T12:50:51.973686+0000","last_undegraded":"2023-10-16T12:50:51.973686+0000","last_fullsized":"2023-10-16T12:50:51.973686+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:20.941851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977674+0000","last_change":"2023-10-16T12:50:51.974747+0000","last_active":"2023-10-16T12:50:51.977674+0000","last_peered":"2023-10-16T12:50:51.977674+0000","last_clean":"2023-10-16T12:50:51.977674+0000","last_became_active":"2023-10-16T12:50:51.974450+0000","last_became_peered":"2023-10-16T12:50:51.974450+0000","last_unstale":"2023-10-16T12:50:51.977674+0000","last_undegraded":"2023-10-16T12:50:51.977674+0000","last_fullsized":"2023-10-16T12:50:51.977674+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:28.869259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.983410+0000","last_change":"2023-10-16T12:50:51.980703+0000","last_active":"2023-10-16T12:50:51.983410+0000","last_peered":"2023-10-16T12:50:51.983410+0000","last_clean":"2023-10-16T12:50:51.983410+0000","last_became_active":"2023-10-16T12:50:51.975265+0000","last_became_peered":"2023-10-16T12:50:51.975265+0000","last_unstale":"2023-10-16T12:50:51.983410+0000","last_undegraded":"2023-10-16T12:50:51.983410+0000","last_fullsized":"2023-10-16T12:50:51.983410+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:00.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.986715+0000","last_change":"2023-10-16T12:50:51.982384+0000","last_active":"2023-10-16T12:50:51.986715+0000","last_peered":"2023-10-16T12:50:51.986715+0000","last_clean":"2023-10-16T12:50:51.986715+0000","last_became_active":"2023-10-16T12:50:51.982025+0000","last_became_peered":"2023-10-16T12:50:51.982025+0000","last_unstale":"2023-10-16T12:50:51.986715+0000","last_undegraded":"2023-10-16T12:50:51.986715+0000","last_fullsized":"2023-10-16T12:50:51.986715+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:24.702209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981677+0000","last_change":"2023-10-16T12:50:51.978401+0000","last_active":"2023-10-16T12:50:51.981677+0000","last_peered":"2023-10-16T12:50:51.981677+0000","last_clean":"2023-10-16T12:50:51.981677+0000","last_became_active":"2023-10-16T12:50:51.978142+0000","last_became_peered":"2023-10-16T12:50:51.978142+0000","last_unstale":"2023-10-16T12:50:51.981677+0000","last_undegraded":"2023-10-16T12:50:51.981677+0000","last_fullsized":"2023-10-16T12:50:51.981677+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:08.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981409+0000","last_change":"2023-10-16T12:50:51.979102+0000","last_active":"2023-10-16T12:50:51.981409+0000","last_peered":"2023-10-16T12:50:51.981409+0000","last_clean":"2023-10-16T12:50:51.981409+0000","last_became_active":"2023-10-16T12:50:51.978828+0000","last_became_peered":"2023-10-16T12:50:51.978828+0000","last_unstale":"2023-10-16T12:50:51.981409+0000","last_undegraded":"2023-10-16T12:50:51.981409+0000","last_fullsized":"2023-10-16T12:50:51.981409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:28.753302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995814+0000","last_change":"2023-10-16T12:50:51.989860+0000","last_active":"2023-10-16T12:50:51.995814+0000","last_peered":"2023-10-16T12:50:51.995814+0000","last_clean":"2023-10-16T12:50:51.995814+0000","last_became_active":"2023-10-16T12:50:51.989356+0000","last_became_peered":"2023-10-16T12:50:51.989356+0000","last_unstale":"2023-10-16T12:50:51.995814+0000","last_undegraded":"2023-10-16T12:50:51.995814+0000","last_fullsized":"2023-10-16T12:50:51.995814+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:10.845718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.980673+0000","last_change":"2023-10-16T12:50:51.979267+0000","last_active":"2023-10-16T12:50:51.980673+0000","last_peered":"2023-10-16T12:50:51.980673+0000","last_clean":"2023-10-16T12:50:51.980673+0000","last_became_active":"2023-10-16T12:50:51.978518+0000","last_became_peered":"2023-10-16T12:50:51.978518+0000","last_unstale":"2023-10-16T12:50:51.980673+0000","last_undegraded":"2023-10-16T12:50:51.980673+0000","last_fullsized":"2023-10-16T12:50:51.980673+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:22.239539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.976751+0000","last_change":"2023-10-16T12:50:51.971243+0000","last_active":"2023-10-16T12:50:51.976751+0000","last_peered":"2023-10-16T12:50:51.976751+0000","last_clean":"2023-10-16T12:50:51.976751+0000","last_became_active":"2023-10-16T12:50:51.970016+0000","last_became_peered":"2023-10-16T12:50:51.970016+0000","last_unstale":"2023-10-16T12:50:51.976751+0000","last_undegraded":"2023-10-16T12:50:51.976751+0000","last_fullsized":"2023-10-16T12:50:51.976751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:37.226580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981878+0000","last_change":"2023-10-16T12:50:51.981295+0000","last_active":"2023-10-16T12:50:51.981878+0000","last_peered":"2023-10-16T12:50:51.981878+0000","last_clean":"2023-10-16T12:50:51.981878+0000","last_became_active":"2023-10-16T12:50:51.980999+0000","last_became_peered":"2023-10-16T12:50:51.980999+0000","last_unstale":"2023-10-16T12:50:51.981878+0000","last_undegraded":"2023-10-16T12:50:51.981878+0000","last_fullsized":"2023-10-16T12:50:51.981878+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:58.055049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992949+0000","last_change":"2023-10-16T12:50:51.986647+0000","last_active":"2023-10-16T12:50:51.992949+0000","last_peered":"2023-10-16T12:50:51.992949+0000","last_clean":"2023-10-16T12:50:51.992949+0000","last_became_active":"2023-10-16T12:50:51.985262+0000","last_became_peered":"2023-10-16T12:50:51.985262+0000","last_unstale":"2023-10-16T12:50:51.992949+0000","last_undegraded":"2023-10-16T12:50:51.992949+0000","last_fullsized":"2023-10-16T12:50:51.992949+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:56.127519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981663+0000","last_change":"2023-10-16T12:50:51.975066+0000","last_active":"2023-10-16T12:50:51.981663+0000","last_peered":"2023-10-16T12:50:51.981663+0000","last_clean":"2023-10-16T12:50:51.981663+0000","last_became_active":"2023-10-16T12:50:51.974760+0000","last_became_peered":"2023-10-16T12:50:51.974760+0000","last_unstale":"2023-10-16T12:50:51.981663+0000","last_undegraded":"2023-10-16T12:50:51.981663+0000","last_fullsized":"2023-10-16T12:50:51.981663+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:39.777334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995662+0000","last_change":"2023-10-16T12:50:51.986719+0000","last_active":"2023-10-16T12:50:51.995662+0000","last_peered":"2023-10-16T12:50:51.995662+0000","last_clean":"2023-10-16T12:50:51.995662+0000","last_became_active":"2023-10-16T12:50:51.986369+0000","last_became_peered":"2023-10-16T12:50:51.986369+0000","last_unstale":"2023-10-16T12:50:51.995662+0000","last_undegraded":"2023-10-16T12:50:51.995662+0000","last_fullsized":"2023-10-16T12:50:51.995662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:22.655885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988528+0000","last_change":"2023-10-16T12:50:51.984511+0000","last_active":"2023-10-16T12:50:51.988528+0000","last_peered":"2023-10-16T12:50:51.988528+0000","last_clean":"2023-10-16T12:50:51.988528+0000","last_became_active":"2023-10-16T12:50:51.983895+0000","last_became_peered":"2023-10-16T12:50:51.983895+0000","last_unstale":"2023-10-16T12:50:51.988528+0000","last_undegraded":"2023-10-16T12:50:51.988528+0000","last_fullsized":"2023-10-16T12:50:51.988528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:42.291463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.983924+0000","last_change":"2023-10-16T12:50:51.977461+0000","last_active":"2023-10-16T12:50:51.983924+0000","last_peered":"2023-10-16T12:50:51.983924+0000","last_clean":"2023-10-16T12:50:51.983924+0000","last_became_active":"2023-10-16T12:50:51.977116+0000","last_became_peered":"2023-10-16T12:50:51.977116+0000","last_unstale":"2023-10-16T12:50:51.983924+0000","last_undegraded":"2023-10-16T12:50:51.983924+0000","last_fullsized":"2023-10-16T12:50:51.983924+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:16.572912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.982736+0000","last_change":"2023-10-16T12:50:51.980110+0000","last_active":"2023-10-16T12:50:51.982736+0000","last_peered":"2023-10-16T12:50:51.982736+0000","last_clean":"2023-10-16T12:50:51.982736+0000","last_became_active":"2023-10-16T12:50:51.976849+0000","last_became_peered":"2023-10-16T12:50:51.976849+0000","last_unstale":"2023-10-16T12:50:51.982736+0000","last_undegraded":"2023-10-16T12:50:51.982736+0000","last_fullsized":"2023-10-16T12:50:51.982736+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:04.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991607+0000","last_change":"2023-10-16T12:50:51.984966+0000","last_active":"2023-10-16T12:50:51.991607+0000","last_peered":"2023-10-16T12:50:51.991607+0000","last_clean":"2023-10-16T12:50:51.991607+0000","last_became_active":"2023-10-16T12:50:51.984106+0000","last_became_peered":"2023-10-16T12:50:51.984106+0000","last_unstale":"2023-10-16T12:50:51.991607+0000","last_undegraded":"2023-10-16T12:50:51.991607+0000","last_fullsized":"2023-10-16T12:50:51.991607+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:23.228050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984037+0000","last_change":"2023-10-16T12:50:51.977403+0000","last_active":"2023-10-16T12:50:51.984037+0000","last_peered":"2023-10-16T12:50:51.984037+0000","last_clean":"2023-10-16T12:50:51.984037+0000","last_became_active":"2023-10-16T12:50:51.975047+0000","last_became_peered":"2023-10-16T12:50:51.975047+0000","last_unstale":"2023-10-16T12:50:51.984037+0000","last_undegraded":"2023-10-16T12:50:51.984037+0000","last_fullsized":"2023-10-16T12:50:51.984037+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:37:24.318824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.989610+0000","last_change":"2023-10-16T12:50:51.988066+0000","last_active":"2023-10-16T12:50:51.989610+0000","last_peered":"2023-10-16T12:50:51.989610+0000","last_clean":"2023-10-16T12:50:51.989610+0000","last_became_active":"2023-10-16T12:50:51.986386+0000","last_became_peered":"2023-10-16T12:50:51.986386+0000","last_unstale":"2023-10-16T12:50:51.989610+0000","last_undegraded":"2023-10-16T12:50:51.989610+0000","last_fullsized":"2023-10-16T12:50:51.989610+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:22.239539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977494+0000","last_change":"2023-10-16T12:50:51.975391+0000","last_active":"2023-10-16T12:50:51.977494+0000","last_peered":"2023-10-16T12:50:51.977494+0000","last_clean":"2023-10-16T12:50:51.977494+0000","last_became_active":"2023-10-16T12:50:51.975118+0000","last_became_peered":"2023-10-16T12:50:51.975118+0000","last_unstale":"2023-10-16T12:50:51.977494+0000","last_undegraded":"2023-10-16T12:50:51.977494+0000","last_fullsized":"2023-10-16T12:50:51.977494+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:07.518531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993072+0000","last_change":"2023-10-16T12:50:51.989431+0000","last_active":"2023-10-16T12:50:51.993072+0000","last_peered":"2023-10-16T12:50:51.993072+0000","last_clean":"2023-10-16T12:50:51.993072+0000","last_became_active":"2023-10-16T12:50:51.988780+0000","last_became_peered":"2023-10-16T12:50:51.988780+0000","last_unstale":"2023-10-16T12:50:51.993072+0000","last_undegraded":"2023-10-16T12:50:51.993072+0000","last_fullsized":"2023-10-16T12:50:51.993072+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:45:40.616676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984022+0000","last_change":"2023-10-16T12:50:51.979142+0000","last_active":"2023-10-16T12:50:51.984022+0000","last_peered":"2023-10-16T12:50:51.984022+0000","last_clean":"2023-10-16T12:50:51.984022+0000","last_became_active":"2023-10-16T12:50:51.974676+0000","last_became_peered":"2023-10-16T12:50:51.974676+0000","last_unstale":"2023-10-16T12:50:51.984022+0000","last_undegraded":"2023-10-16T12:50:51.984022+0000","last_fullsized":"2023-10-16T12:50:51.984022+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:38.497935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.996961+0000","last_change":"2023-10-16T12:50:51.994200+0000","last_active":"2023-10-16T12:50:51.996961+0000","last_peered":"2023-10-16T12:50:51.996961+0000","last_clean":"2023-10-16T12:50:51.996961+0000","last_became_active":"2023-10-16T12:50:51.993880+0000","last_became_peered":"2023-10-16T12:50:51.993880+0000","last_unstale":"2023-10-16T12:50:51.996961+0000","last_undegraded":"2023-10-16T12:50:51.996961+0000","last_fullsized":"2023-10-16T12:50:51.996961+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:44.304350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978626+0000","last_change":"2023-10-16T12:50:51.977007+0000","last_active":"2023-10-16T12:50:51.978626+0000","last_peered":"2023-10-16T12:50:51.978626+0000","last_clean":"2023-10-16T12:50:51.978626+0000","last_became_active":"2023-10-16T12:50:51.976740+0000","last_became_peered":"2023-10-16T12:50:51.976740+0000","last_unstale":"2023-10-16T12:50:51.978626+0000","last_undegraded":"2023-10-16T12:50:51.978626+0000","last_fullsized":"2023-10-16T12:50:51.978626+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:40.127068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992256+0000","last_change":"2023-10-16T12:50:51.984287+0000","last_active":"2023-10-16T12:50:51.992256+0000","last_peered":"2023-10-16T12:50:51.992256+0000","last_clean":"2023-10-16T12:50:51.992256+0000","last_became_active":"2023-10-16T12:50:51.984018+0000","last_became_peered":"2023-10-16T12:50:51.984018+0000","last_unstale":"2023-10-16T12:50:51.992256+0000","last_undegraded":"2023-10-16T12:50:51.992256+0000","last_fullsized":"2023-10-16T12:50:51.992256+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:48.737941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977243+0000","last_change":"2023-10-16T12:50:51.973838+0000","last_active":"2023-10-16T12:50:51.977243+0000","last_peered":"2023-10-16T12:50:51.977243+0000","last_clean":"2023-10-16T12:50:51.977243+0000","last_became_active":"2023-10-16T12:50:51.973524+0000","last_became_peered":"2023-10-16T12:50:51.973524+0000","last_unstale":"2023-10-16T12:50:51.977243+0000","last_undegraded":"2023-10-16T12:50:51.977243+0000","last_fullsized":"2023-10-16T12:50:51.977243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:44.773337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"112'4","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.998076+0000","last_change":"2023-10-16T12:50:50.959454+0000","last_active":"2023-10-16T12:50:51.998076+0000","last_peered":"2023-10-16T12:50:51.998076+0000","last_clean":"2023-10-16T12:50:51.998076+0000","last_became_active":"2023-10-16T12:50:50.959131+0000","last_became_peered":"2023-10-16T12:50:50.959131+0000","last_unstale":"2023-10-16T12:50:51.998076+0000","last_undegraded":"2023-10-16T12:50:51.998076+0000","last_fullsized":"2023-10-16T12:50:51.998076+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:44.125691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992525+0000","last_change":"2023-10-16T12:50:51.985714+0000","last_active":"2023-10-16T12:50:51.992525+0000","last_peered":"2023-10-16T12:50:51.992525+0000","last_clean":"2023-10-16T12:50:51.992525+0000","last_became_active":"2023-10-16T12:50:51.985230+0000","last_became_peered":"2023-10-16T12:50:51.985230+0000","last_unstale":"2023-10-16T12:50:51.992525+0000","last_undegraded":"2023-10-16T12:50:51.992525+0000","last_fullsized":"2023-10-16T12:50:51.992525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:54:46.760198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993694+0000","last_change":"2023-10-16T12:50:50.946700+0000","last_active":"2023-10-16T12:50:51.993694+0000","last_peered":"2023-10-16T12:50:51.993694+0000","last_clean":"2023-10-16T12:50:51.993694+0000","last_became_active":"2023-10-16T12:50:50.946024+0000","last_became_peered":"2023-10-16T12:50:50.946024+0000","last_unstale":"2023-10-16T12:50:51.993694+0000","last_undegraded":"2023-10-16T12:50:51.993694+0000","last_fullsized":"2023-10-16T12:50:51.993694+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:23.489241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991133+0000","last_change":"2023-10-16T12:50:51.982523+0000","last_active":"2023-10-16T12:50:51.991133+0000","last_peered":"2023-10-16T12:50:51.991133+0000","last_clean":"2023-10-16T12:50:51.991133+0000","last_became_active":"2023-10-16T12:50:51.982164+0000","last_became_peered":"2023-10-16T12:50:51.982164+0000","last_unstale":"2023-10-16T12:50:51.991133+0000","last_undegraded":"2023-10-16T12:50:51.991133+0000","last_fullsized":"2023-10-16T12:50:51.991133+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:23.578588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997882+0000","last_change":"2023-10-16T12:50:50.959919+0000","last_active":"2023-10-16T12:50:51.997882+0000","last_peered":"2023-10-16T12:50:51.997882+0000","last_clean":"2023-10-16T12:50:51.997882+0000","last_became_active":"2023-10-16T12:50:50.959625+0000","last_became_peered":"2023-10-16T12:50:50.959625+0000","last_unstale":"2023-10-16T12:50:51.997882+0000","last_undegraded":"2023-10-16T12:50:51.997882+0000","last_fullsized":"2023-10-16T12:50:51.997882+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:05.335588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988363+0000","last_change":"2023-10-16T12:50:51.984319+0000","last_active":"2023-10-16T12:50:51.988363+0000","last_peered":"2023-10-16T12:50:51.988363+0000","last_clean":"2023-10-16T12:50:51.988363+0000","last_became_active":"2023-10-16T12:50:51.982993+0000","last_became_peered":"2023-10-16T12:50:51.982993+0000","last_unstale":"2023-10-16T12:50:51.988363+0000","last_undegraded":"2023-10-16T12:50:51.988363+0000","last_fullsized":"2023-10-16T12:50:51.988363+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:00.073808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997592+0000","last_change":"2023-10-16T12:50:50.956160+0000","last_active":"2023-10-16T12:50:51.997592+0000","last_peered":"2023-10-16T12:50:51.997592+0000","last_clean":"2023-10-16T12:50:51.997592+0000","last_became_active":"2023-10-16T12:50:50.955840+0000","last_became_peered":"2023-10-16T12:50:50.955840+0000","last_unstale":"2023-10-16T12:50:51.997592+0000","last_undegraded":"2023-10-16T12:50:51.997592+0000","last_fullsized":"2023-10-16T12:50:51.997592+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:58.285331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.996940+0000","last_change":"2023-10-16T12:50:51.993958+0000","last_active":"2023-10-16T12:50:51.996940+0000","last_peered":"2023-10-16T12:50:51.996940+0000","last_clean":"2023-10-16T12:50:51.996940+0000","last_became_active":"2023-10-16T12:50:51.993303+0000","last_became_peered":"2023-10-16T12:50:51.993303+0000","last_unstale":"2023-10-16T12:50:51.996940+0000","last_undegraded":"2023-10-16T12:50:51.996940+0000","last_fullsized":"2023-10-16T12:50:51.996940+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:53.675341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.999876+0000","last_change":"2023-10-16T12:50:50.946280+0000","last_active":"2023-10-16T12:50:51.999876+0000","last_peered":"2023-10-16T12:50:51.999876+0000","last_clean":"2023-10-16T12:50:51.999876+0000","last_became_active":"2023-10-16T12:50:50.945949+0000","last_became_peered":"2023-10-16T12:50:50.945949+0000","last_unstale":"2023-10-16T12:50:51.999876+0000","last_undegraded":"2023-10-16T12:50:51.999876+0000","last_fullsized":"2023-10-16T12:50:51.999876+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:30.429826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981595+0000","last_change":"2023-10-16T12:50:51.980675+0000","last_active":"2023-10-16T12:50:51.981595+0000","last_peered":"2023-10-16T12:50:51.981595+0000","last_clean":"2023-10-16T12:50:51.981595+0000","last_became_active":"2023-10-16T12:50:51.980387+0000","last_became_peered":"2023-10-16T12:50:51.980387+0000","last_unstale":"2023-10-16T12:50:51.981595+0000","last_undegraded":"2023-10-16T12:50:51.981595+0000","last_fullsized":"2023-10-16T12:50:51.981595+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:51.379827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993750+0000","last_change":"2023-10-16T12:50:50.946234+0000","last_active":"2023-10-16T12:50:51.993750+0000","last_peered":"2023-10-16T12:50:51.993750+0000","last_clean":"2023-10-16T12:50:51.993750+0000","last_became_active":"2023-10-16T12:50:50.945802+0000","last_became_peered":"2023-10-16T12:50:50.945802+0000","last_unstale":"2023-10-16T12:50:51.993750+0000","last_undegraded":"2023-10-16T12:50:51.993750+0000","last_fullsized":"2023-10-16T12:50:51.993750+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:48.628407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978858+0000","last_change":"2023-10-16T12:50:51.978129+0000","last_active":"2023-10-16T12:50:51.978858+0000","last_peered":"2023-10-16T12:50:51.978858+0000","last_clean":"2023-10-16T12:50:51.978858+0000","last_became_active":"2023-10-16T12:50:51.977327+0000","last_became_peered":"2023-10-16T12:50:51.977327+0000","last_unstale":"2023-10-16T12:50:51.978858+0000","last_undegraded":"2023-10-16T12:50:51.978858+0000","last_fullsized":"2023-10-16T12:50:51.978858+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:56.284790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.968463+0000","last_change":"2023-10-16T12:50:50.954237+0000","last_active":"2023-10-16T12:50:51.968463+0000","last_peered":"2023-10-16T12:50:51.968463+0000","last_clean":"2023-10-16T12:50:51.968463+0000","last_became_active":"2023-10-16T12:50:50.953891+0000","last_became_peered":"2023-10-16T12:50:50.953891+0000","last_unstale":"2023-10-16T12:50:51.968463+0000","last_undegraded":"2023-10-16T12:50:51.968463+0000","last_fullsized":"2023-10-16T12:50:51.968463+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:41.505644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.980652+0000","last_change":"2023-10-16T12:50:51.978767+0000","last_active":"2023-10-16T12:50:51.980652+0000","last_peered":"2023-10-16T12:50:51.980652+0000","last_clean":"2023-10-16T12:50:51.980652+0000","last_became_active":"2023-10-16T12:50:51.978431+0000","last_became_peered":"2023-10-16T12:50:51.978431+0000","last_unstale":"2023-10-16T12:50:51.980652+0000","last_undegraded":"2023-10-16T12:50:51.980652+0000","last_fullsized":"2023-10-16T12:50:51.980652+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:11.792351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.957001+0000","last_change":"2023-10-16T12:50:50.953976+0000","last_active":"2023-10-16T12:50:51.957001+0000","last_peered":"2023-10-16T12:50:51.957001+0000","last_clean":"2023-10-16T12:50:51.957001+0000","last_became_active":"2023-10-16T12:50:50.953686+0000","last_became_peered":"2023-10-16T12:50:50.953686+0000","last_unstale":"2023-10-16T12:50:51.957001+0000","last_undegraded":"2023-10-16T12:50:51.957001+0000","last_fullsized":"2023-10-16T12:50:51.957001+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:46.748740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978668+0000","last_change":"2023-10-16T12:50:51.974620+0000","last_active":"2023-10-16T12:50:51.978668+0000","last_peered":"2023-10-16T12:50:51.978668+0000","last_clean":"2023-10-16T12:50:51.978668+0000","last_became_active":"2023-10-16T12:50:51.974331+0000","last_became_peered":"2023-10-16T12:50:51.974331+0000","last_unstale":"2023-10-16T12:50:51.978668+0000","last_undegraded":"2023-10-16T12:50:51.978668+0000","last_fullsized":"2023-10-16T12:50:51.978668+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:43.451631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972935+0000","last_change":"2023-10-16T12:50:50.958264+0000","last_active":"2023-10-16T12:50:51.972935+0000","last_peered":"2023-10-16T12:50:51.972935+0000","last_clean":"2023-10-16T12:50:51.972935+0000","last_became_active":"2023-10-16T12:50:50.957548+0000","last_became_peered":"2023-10-16T12:50:50.957548+0000","last_unstale":"2023-10-16T12:50:51.972935+0000","last_undegraded":"2023-10-16T12:50:51.972935+0000","last_fullsized":"2023-10-16T12:50:51.972935+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:39.969803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.982586+0000","last_change":"2023-10-16T12:50:51.980928+0000","last_active":"2023-10-16T12:50:51.982586+0000","last_peered":"2023-10-16T12:50:51.982586+0000","last_clean":"2023-10-16T12:50:51.982586+0000","last_became_active":"2023-10-16T12:50:51.980630+0000","last_became_peered":"2023-10-16T12:50:51.980630+0000","last_unstale":"2023-10-16T12:50:51.982586+0000","last_undegraded":"2023-10-16T12:50:51.982586+0000","last_fullsized":"2023-10-16T12:50:51.982586+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:03.273600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972242+0000","last_change":"2023-10-16T12:50:50.956801+0000","last_active":"2023-10-16T12:50:51.972242+0000","last_peered":"2023-10-16T12:50:51.972242+0000","last_clean":"2023-10-16T12:50:51.972242+0000","last_became_active":"2023-10-16T12:50:50.954920+0000","last_became_peered":"2023-10-16T12:50:50.954920+0000","last_unstale":"2023-10-16T12:50:51.972242+0000","last_undegraded":"2023-10-16T12:50:51.972242+0000","last_fullsized":"2023-10-16T12:50:51.972242+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:42:27.027931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978062+0000","last_change":"2023-10-16T12:50:51.974468+0000","last_active":"2023-10-16T12:50:51.978062+0000","last_peered":"2023-10-16T12:50:51.978062+0000","last_clean":"2023-10-16T12:50:51.978062+0000","last_became_active":"2023-10-16T12:50:51.974154+0000","last_became_peered":"2023-10-16T12:50:51.974154+0000","last_unstale":"2023-10-16T12:50:51.978062+0000","last_undegraded":"2023-10-16T12:50:51.978062+0000","last_fullsized":"2023-10-16T12:50:51.978062+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:50.612126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"112'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:52.000677+0000","last_change":"2023-10-16T12:50:50.955048+0000","last_active":"2023-10-16T12:50:52.000677+0000","last_peered":"2023-10-16T12:50:52.000677+0000","last_clean":"2023-10-16T12:50:52.000677+0000","last_became_active":"2023-10-16T12:50:50.954649+0000","last_became_peered":"2023-10-16T12:50:50.954649+0000","last_unstale":"2023-10-16T12:50:52.000677+0000","last_undegraded":"2023-10-16T12:50:52.000677+0000","last_fullsized":"2023-10-16T12:50:52.000677+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:52.226837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988480+0000","last_change":"2023-10-16T12:50:51.985106+0000","last_active":"2023-10-16T12:50:51.988480+0000","last_peered":"2023-10-16T12:50:51.988480+0000","last_clean":"2023-10-16T12:50:51.988480+0000","last_became_active":"2023-10-16T12:50:51.984822+0000","last_became_peered":"2023-10-16T12:50:51.984822+0000","last_unstale":"2023-10-16T12:50:51.988480+0000","last_undegraded":"2023-10-16T12:50:51.988480+0000","last_fullsized":"2023-10-16T12:50:51.988480+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:10.767414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:52.002099+0000","last_change":"2023-10-16T12:50:50.960112+0000","last_active":"2023-10-16T12:50:52.002099+0000","last_peered":"2023-10-16T12:50:52.002099+0000","last_clean":"2023-10-16T12:50:52.002099+0000","last_became_active":"2023-10-16T12:50:50.959802+0000","last_became_peered":"2023-10-16T12:50:50.959802+0000","last_unstale":"2023-10-16T12:50:52.002099+0000","last_undegraded":"2023-10-16T12:50:52.002099+0000","last_fullsized":"2023-10-16T12:50:52.002099+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:49.936794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984796+0000","last_change":"2023-10-16T12:50:51.981691+0000","last_active":"2023-10-16T12:50:51.984796+0000","last_peered":"2023-10-16T12:50:51.984796+0000","last_clean":"2023-10-16T12:50:51.984796+0000","last_became_active":"2023-10-16T12:50:51.981368+0000","last_became_peered":"2023-10-16T12:50:51.981368+0000","last_unstale":"2023-10-16T12:50:51.984796+0000","last_undegraded":"2023-10-16T12:50:51.984796+0000","last_fullsized":"2023-10-16T12:50:51.984796+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:08.300310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:52.000344+0000","last_change":"2023-10-16T12:50:50.957084+0000","last_active":"2023-10-16T12:50:52.000344+0000","last_peered":"2023-10-16T12:50:52.000344+0000","last_clean":"2023-10-16T12:50:52.000344+0000","last_became_active":"2023-10-16T12:50:50.952131+0000","last_became_peered":"2023-10-16T12:50:50.952131+0000","last_unstale":"2023-10-16T12:50:52.000344+0000","last_undegraded":"2023-10-16T12:50:52.000344+0000","last_fullsized":"2023-10-16T12:50:52.000344+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:33:41.349606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.987620+0000","last_change":"2023-10-16T12:50:51.986007+0000","last_active":"2023-10-16T12:50:51.987620+0000","last_peered":"2023-10-16T12:50:51.987620+0000","last_clean":"2023-10-16T12:50:51.987620+0000","last_became_active":"2023-10-16T12:50:51.984975+0000","last_became_peered":"2023-10-16T12:50:51.984975+0000","last_unstale":"2023-10-16T12:50:51.987620+0000","last_undegraded":"2023-10-16T12:50:51.987620+0000","last_fullsized":"2023-10-16T12:50:51.987620+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:43.451631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.947351+0000","last_change":"2023-10-16T12:50:50.952961+0000","last_active":"2023-10-16T12:50:51.947351+0000","last_peered":"2023-10-16T12:50:51.947351+0000","last_clean":"2023-10-16T12:50:51.947351+0000","last_became_active":"2023-10-16T12:50:50.952677+0000","last_became_peered":"2023-10-16T12:50:50.952677+0000","last_unstale":"2023-10-16T12:50:51.947351+0000","last_undegraded":"2023-10-16T12:50:51.947351+0000","last_fullsized":"2023-10-16T12:50:51.947351+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:55.273712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997007+0000","last_change":"2023-10-16T12:50:51.992907+0000","last_active":"2023-10-16T12:50:51.997007+0000","last_peered":"2023-10-16T12:50:51.997007+0000","last_clean":"2023-10-16T12:50:51.997007+0000","last_became_active":"2023-10-16T12:50:51.992610+0000","last_became_peered":"2023-10-16T12:50:51.992610+0000","last_unstale":"2023-10-16T12:50:51.997007+0000","last_undegraded":"2023-10-16T12:50:51.997007+0000","last_fullsized":"2023-10-16T12:50:51.997007+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:06.576628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.956767+0000","last_change":"2023-10-16T12:50:50.950308+0000","last_active":"2023-10-16T12:50:51.956767+0000","last_peered":"2023-10-16T12:50:51.956767+0000","last_clean":"2023-10-16T12:50:51.956767+0000","last_became_active":"2023-10-16T12:50:50.948881+0000","last_became_peered":"2023-10-16T12:50:50.948881+0000","last_unstale":"2023-10-16T12:50:51.956767+0000","last_undegraded":"2023-10-16T12:50:51.956767+0000","last_fullsized":"2023-10-16T12:50:51.956767+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:38.167448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.983958+0000","last_change":"2023-10-16T12:50:51.978726+0000","last_active":"2023-10-16T12:50:51.983958+0000","last_peered":"2023-10-16T12:50:51.983958+0000","last_clean":"2023-10-16T12:50:51.983958+0000","last_became_active":"2023-10-16T12:50:51.976811+0000","last_became_peered":"2023-10-16T12:50:51.976811+0000","last_unstale":"2023-10-16T12:50:51.983958+0000","last_undegraded":"2023-10-16T12:50:51.983958+0000","last_fullsized":"2023-10-16T12:50:51.983958+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:04.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"112'8","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995215+0000","last_change":"2023-10-16T12:50:50.956360+0000","last_active":"2023-10-16T12:50:51.995215+0000","last_peered":"2023-10-16T12:50:51.995215+0000","last_clean":"2023-10-16T12:50:51.995215+0000","last_became_active":"2023-10-16T12:50:50.955503+0000","last_became_peered":"2023-10-16T12:50:50.955503+0000","last_unstale":"2023-10-16T12:50:51.995215+0000","last_undegraded":"2023-10-16T12:50:51.995215+0000","last_fullsized":"2023-10-16T12:50:51.995215+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:11.038097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977890+0000","last_change":"2023-10-16T12:50:51.976699+0000","last_active":"2023-10-16T12:50:51.977890+0000","last_peered":"2023-10-16T12:50:51.977890+0000","last_clean":"2023-10-16T12:50:51.977890+0000","last_became_active":"2023-10-16T12:50:51.976428+0000","last_became_peered":"2023-10-16T12:50:51.976428+0000","last_unstale":"2023-10-16T12:50:51.977890+0000","last_undegraded":"2023-10-16T12:50:51.977890+0000","last_fullsized":"2023-10-16T12:50:51.977890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:11.792351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992614+0000","last_change":"2023-10-16T12:50:50.956994+0000","last_active":"2023-10-16T12:50:51.992614+0000","last_peered":"2023-10-16T12:50:51.992614+0000","last_clean":"2023-10-16T12:50:51.992614+0000","last_became_active":"2023-10-16T12:50:50.956697+0000","last_became_peered":"2023-10-16T12:50:50.956697+0000","last_unstale":"2023-10-16T12:50:51.992614+0000","last_undegraded":"2023-10-16T12:50:51.992614+0000","last_fullsized":"2023-10-16T12:50:51.992614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:25.101451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991316+0000","last_change":"2023-10-16T12:50:51.982721+0000","last_active":"2023-10-16T12:50:51.991316+0000","last_peered":"2023-10-16T12:50:51.991316+0000","last_clean":"2023-10-16T12:50:51.991316+0000","last_became_active":"2023-10-16T12:50:51.982362+0000","last_became_peered":"2023-10-16T12:50:51.982362+0000","last_unstale":"2023-10-16T12:50:51.991316+0000","last_undegraded":"2023-10-16T12:50:51.991316+0000","last_fullsized":"2023-10-16T12:50:51.991316+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:47.455433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992120+0000","last_change":"2023-10-16T12:50:50.953349+0000","last_active":"2023-10-16T12:50:51.992120+0000","last_peered":"2023-10-16T12:50:51.992120+0000","last_clean":"2023-10-16T12:50:51.992120+0000","last_became_active":"2023-10-16T12:50:50.952853+0000","last_became_peered":"2023-10-16T12:50:50.952853+0000","last_unstale":"2023-10-16T12:50:51.992120+0000","last_undegraded":"2023-10-16T12:50:51.992120+0000","last_fullsized":"2023-10-16T12:50:51.992120+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:22.216972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.990581+0000","last_change":"2023-10-16T12:50:51.989278+0000","last_active":"2023-10-16T12:50:51.990581+0000","last_peered":"2023-10-16T12:50:51.990581+0000","last_clean":"2023-10-16T12:50:51.990581+0000","last_became_active":"2023-10-16T12:50:51.988832+0000","last_became_peered":"2023-10-16T12:50:51.988832+0000","last_unstale":"2023-10-16T12:50:51.990581+0000","last_undegraded":"2023-10-16T12:50:51.990581+0000","last_fullsized":"2023-10-16T12:50:51.990581+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:03.959532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972095+0000","last_change":"2023-10-16T12:50:50.952898+0000","last_active":"2023-10-16T12:50:51.972095+0000","last_peered":"2023-10-16T12:50:51.972095+0000","last_clean":"2023-10-16T12:50:51.972095+0000","last_became_active":"2023-10-16T12:50:50.952613+0000","last_became_peered":"2023-10-16T12:50:50.952613+0000","last_unstale":"2023-10-16T12:50:51.972095+0000","last_undegraded":"2023-10-16T12:50:51.972095+0000","last_fullsized":"2023-10-16T12:50:51.972095+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:17.475548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997051+0000","last_change":"2023-10-16T12:50:51.992330+0000","last_active":"2023-10-16T12:50:51.997051+0000","last_peered":"2023-10-16T12:50:51.997051+0000","last_clean":"2023-10-16T12:50:51.997051+0000","last_became_active":"2023-10-16T12:50:51.988283+0000","last_became_peered":"2023-10-16T12:50:51.988283+0000","last_unstale":"2023-10-16T12:50:51.997051+0000","last_undegraded":"2023-10-16T12:50:51.997051+0000","last_fullsized":"2023-10-16T12:50:51.997051+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:23.438935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.998703+0000","last_change":"2023-10-16T12:50:50.949855+0000","last_active":"2023-10-16T12:50:51.998703+0000","last_peered":"2023-10-16T12:50:51.998703+0000","last_clean":"2023-10-16T12:50:51.998703+0000","last_became_active":"2023-10-16T12:50:50.949584+0000","last_became_peered":"2023-10-16T12:50:50.949584+0000","last_unstale":"2023-10-16T12:50:51.998703+0000","last_undegraded":"2023-10-16T12:50:51.998703+0000","last_fullsized":"2023-10-16T12:50:51.998703+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:27.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972257+0000","last_change":"2023-10-16T12:50:51.968059+0000","last_active":"2023-10-16T12:50:51.972257+0000","last_peered":"2023-10-16T12:50:51.972257+0000","last_clean":"2023-10-16T12:50:51.972257+0000","last_became_active":"2023-10-16T12:50:51.967788+0000","last_became_peered":"2023-10-16T12:50:51.967788+0000","last_unstale":"2023-10-16T12:50:51.972257+0000","last_undegraded":"2023-10-16T12:50:51.972257+0000","last_fullsized":"2023-10-16T12:50:51.972257+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:28.708339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.975575+0000","last_change":"2023-10-16T12:50:50.948020+0000","last_active":"2023-10-16T12:50:51.975575+0000","last_peered":"2023-10-16T12:50:51.975575+0000","last_clean":"2023-10-16T12:50:51.975575+0000","last_became_active":"2023-10-16T12:50:50.947523+0000","last_became_peered":"2023-10-16T12:50:50.947523+0000","last_unstale":"2023-10-16T12:50:51.975575+0000","last_undegraded":"2023-10-16T12:50:51.975575+0000","last_fullsized":"2023-10-16T12:50:51.975575+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:51.877118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.985996+0000","last_change":"2023-10-16T12:50:51.983170+0000","last_active":"2023-10-16T12:50:51.985996+0000","last_peered":"2023-10-16T12:50:51.985996+0000","last_clean":"2023-10-16T12:50:51.985996+0000","last_became_active":"2023-10-16T12:50:51.978796+0000","last_became_peered":"2023-10-16T12:50:51.978796+0000","last_unstale":"2023-10-16T12:50:51.985996+0000","last_undegraded":"2023-10-16T12:50:51.985996+0000","last_fullsized":"2023-10-16T12:50:51.985996+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:18.486626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.966573+0000","last_change":"2023-10-16T12:50:50.956668+0000","last_active":"2023-10-16T12:50:51.966573+0000","last_peered":"2023-10-16T12:50:51.966573+0000","last_clean":"2023-10-16T12:50:51.966573+0000","last_became_active":"2023-10-16T12:50:50.956392+0000","last_became_peered":"2023-10-16T12:50:50.956392+0000","last_unstale":"2023-10-16T12:50:51.966573+0000","last_undegraded":"2023-10-16T12:50:51.966573+0000","last_fullsized":"2023-10-16T12:50:51.966573+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:58.285331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978279+0000","last_change":"2023-10-16T12:50:51.976802+0000","last_active":"2023-10-16T12:50:51.978279+0000","last_peered":"2023-10-16T12:50:51.978279+0000","last_clean":"2023-10-16T12:50:51.978279+0000","last_became_active":"2023-10-16T12:50:51.975837+0000","last_became_peered":"2023-10-16T12:50:51.975837+0000","last_unstale":"2023-10-16T12:50:51.978279+0000","last_undegraded":"2023-10-16T12:50:51.978279+0000","last_fullsized":"2023-10-16T12:50:51.978279+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:22.570835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.946865+0000","last_change":"2023-10-16T12:50:50.949104+0000","last_active":"2023-10-16T12:50:51.946865+0000","last_peered":"2023-10-16T12:50:51.946865+0000","last_clean":"2023-10-16T12:50:51.946865+0000","last_became_active":"2023-10-16T12:50:50.948829+0000","last_became_peered":"2023-10-16T12:50:50.948829+0000","last_unstale":"2023-10-16T12:50:51.946865+0000","last_undegraded":"2023-10-16T12:50:51.946865+0000","last_fullsized":"2023-10-16T12:50:51.946865+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:30.523367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.976497+0000","last_change":"2023-10-16T12:50:51.970928+0000","last_active":"2023-10-16T12:50:51.976497+0000","last_peered":"2023-10-16T12:50:51.976497+0000","last_clean":"2023-10-16T12:50:51.976497+0000","last_became_active":"2023-10-16T12:50:51.970641+0000","last_became_peered":"2023-10-16T12:50:51.970641+0000","last_unstale":"2023-10-16T12:50:51.976497+0000","last_undegraded":"2023-10-16T12:50:51.976497+0000","last_fullsized":"2023-10-16T12:50:51.976497+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:13.918437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991052+0000","last_change":"2023-10-16T12:50:50.951295+0000","last_active":"2023-10-16T12:50:51.991052+0000","last_peered":"2023-10-16T12:50:51.991052+0000","last_clean":"2023-10-16T12:50:51.991052+0000","last_became_active":"2023-10-16T12:50:50.950985+0000","last_became_peered":"2023-10-16T12:50:50.950985+0000","last_unstale":"2023-10-16T12:50:51.991052+0000","last_undegraded":"2023-10-16T12:50:51.991052+0000","last_fullsized":"2023-10-16T12:50:51.991052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:37.970636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993705+0000","last_change":"2023-10-16T12:50:51.992003+0000","last_active":"2023-10-16T12:50:51.993705+0000","last_peered":"2023-10-16T12:50:51.993705+0000","last_clean":"2023-10-16T12:50:51.993705+0000","last_became_active":"2023-10-16T12:50:51.991691+0000","last_became_peered":"2023-10-16T12:50:51.991691+0000","last_unstale":"2023-10-16T12:50:51.993705+0000","last_undegraded":"2023-10-16T12:50:51.993705+0000","last_fullsized":"2023-10-16T12:50:51.993705+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:34.671005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.964808+0000","last_change":"2023-10-16T12:50:50.948373+0000","last_active":"2023-10-16T12:50:51.964808+0000","last_peered":"2023-10-16T12:50:51.964808+0000","last_clean":"2023-10-16T12:50:51.964808+0000","last_became_active":"2023-10-16T12:50:50.948060+0000","last_became_peered":"2023-10-16T12:50:50.948060+0000","last_unstale":"2023-10-16T12:50:51.964808+0000","last_undegraded":"2023-10-16T12:50:51.964808+0000","last_fullsized":"2023-10-16T12:50:51.964808+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:12.644977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992862+0000","last_change":"2023-10-16T12:50:51.985400+0000","last_active":"2023-10-16T12:50:51.992862+0000","last_peered":"2023-10-16T12:50:51.992862+0000","last_clean":"2023-10-16T12:50:51.992862+0000","last_became_active":"2023-10-16T12:50:51.985122+0000","last_became_peered":"2023-10-16T12:50:51.985122+0000","last_unstale":"2023-10-16T12:50:51.992862+0000","last_undegraded":"2023-10-16T12:50:51.992862+0000","last_fullsized":"2023-10-16T12:50:51.992862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:34.945007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972003+0000","last_change":"2023-10-16T12:50:50.961570+0000","last_active":"2023-10-16T12:50:51.972003+0000","last_peered":"2023-10-16T12:50:51.972003+0000","last_clean":"2023-10-16T12:50:51.972003+0000","last_became_active":"2023-10-16T12:50:50.961278+0000","last_became_peered":"2023-10-16T12:50:50.961278+0000","last_unstale":"2023-10-16T12:50:51.972003+0000","last_undegraded":"2023-10-16T12:50:51.972003+0000","last_fullsized":"2023-10-16T12:50:51.972003+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:20.269854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978124+0000","last_change":"2023-10-16T12:50:51.975598+0000","last_active":"2023-10-16T12:50:51.978124+0000","last_peered":"2023-10-16T12:50:51.978124+0000","last_clean":"2023-10-16T12:50:51.978124+0000","last_became_active":"2023-10-16T12:50:51.974570+0000","last_became_peered":"2023-10-16T12:50:51.974570+0000","last_unstale":"2023-10-16T12:50:51.978124+0000","last_undegraded":"2023-10-16T12:50:51.978124+0000","last_fullsized":"2023-10-16T12:50:51.978124+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:32.342448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.971849+0000","last_change":"2023-10-16T12:50:50.959558+0000","last_active":"2023-10-16T12:50:51.971849+0000","last_peered":"2023-10-16T12:50:51.971849+0000","last_clean":"2023-10-16T12:50:51.971849+0000","last_became_active":"2023-10-16T12:50:50.959226+0000","last_became_peered":"2023-10-16T12:50:50.959226+0000","last_unstale":"2023-10-16T12:50:51.971849+0000","last_undegraded":"2023-10-16T12:50:51.971849+0000","last_fullsized":"2023-10-16T12:50:51.971849+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:24.521173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972584+0000","last_change":"2023-10-16T12:50:51.966258+0000","last_active":"2023-10-16T12:50:51.972584+0000","last_peered":"2023-10-16T12:50:51.972584+0000","last_clean":"2023-10-16T12:50:51.972584+0000","last_became_active":"2023-10-16T12:50:51.965482+0000","last_became_peered":"2023-10-16T12:50:51.965482+0000","last_unstale":"2023-10-16T12:50:51.972584+0000","last_undegraded":"2023-10-16T12:50:51.972584+0000","last_fullsized":"2023-10-16T12:50:51.972584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:02.458758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.970232+0000","last_change":"2023-10-16T12:50:50.951548+0000","last_active":"2023-10-16T12:50:51.970232+0000","last_peered":"2023-10-16T12:50:51.970232+0000","last_clean":"2023-10-16T12:50:51.970232+0000","last_became_active":"2023-10-16T12:50:50.951276+0000","last_became_peered":"2023-10-16T12:50:50.951276+0000","last_unstale":"2023-10-16T12:50:51.970232+0000","last_undegraded":"2023-10-16T12:50:51.970232+0000","last_fullsized":"2023-10-16T12:50:51.970232+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:41.280385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991409+0000","last_change":"2023-10-16T12:50:51.982927+0000","last_active":"2023-10-16T12:50:51.991409+0000","last_peered":"2023-10-16T12:50:51.991409+0000","last_clean":"2023-10-16T12:50:51.991409+0000","last_became_active":"2023-10-16T12:50:51.982582+0000","last_became_peered":"2023-10-16T12:50:51.982582+0000","last_unstale":"2023-10-16T12:50:51.991409+0000","last_undegraded":"2023-10-16T12:50:51.991409+0000","last_fullsized":"2023-10-16T12:50:51.991409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:08.422266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.959082+0000","last_change":"2023-10-16T12:50:50.946010+0000","last_active":"2023-10-16T12:50:51.959082+0000","last_peered":"2023-10-16T12:50:51.959082+0000","last_clean":"2023-10-16T12:50:51.959082+0000","last_became_active":"2023-10-16T12:50:50.945492+0000","last_became_peered":"2023-10-16T12:50:50.945492+0000","last_unstale":"2023-10-16T12:50:51.959082+0000","last_undegraded":"2023-10-16T12:50:51.959082+0000","last_fullsized":"2023-10-16T12:50:51.959082+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:16.960765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.994440+0000","last_change":"2023-10-16T12:50:51.993708+0000","last_active":"2023-10-16T12:50:51.994440+0000","last_peered":"2023-10-16T12:50:51.994440+0000","last_clean":"2023-10-16T12:50:51.994440+0000","last_became_active":"2023-10-16T12:50:51.993293+0000","last_became_peered":"2023-10-16T12:50:51.993293+0000","last_unstale":"2023-10-16T12:50:51.994440+0000","last_undegraded":"2023-10-16T12:50:51.994440+0000","last_fullsized":"2023-10-16T12:50:51.994440+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:39.020851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972259+0000","last_change":"2023-10-16T12:50:50.955193+0000","last_active":"2023-10-16T12:50:51.972259+0000","last_peered":"2023-10-16T12:50:51.972259+0000","last_clean":"2023-10-16T12:50:51.972259+0000","last_became_active":"2023-10-16T12:50:50.954789+0000","last_became_peered":"2023-10-16T12:50:50.954789+0000","last_unstale":"2023-10-16T12:50:51.972259+0000","last_undegraded":"2023-10-16T12:50:51.972259+0000","last_fullsized":"2023-10-16T12:50:51.972259+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:08.300625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992945+0000","last_change":"2023-10-16T12:50:51.988746+0000","last_active":"2023-10-16T12:50:51.992945+0000","last_peered":"2023-10-16T12:50:51.992945+0000","last_clean":"2023-10-16T12:50:51.992945+0000","last_became_active":"2023-10-16T12:50:51.988266+0000","last_became_peered":"2023-10-16T12:50:51.988266+0000","last_unstale":"2023-10-16T12:50:51.992945+0000","last_undegraded":"2023-10-16T12:50:51.992945+0000","last_fullsized":"2023-10-16T12:50:51.992945+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:09.389663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.961029+0000","last_change":"2023-10-16T12:50:50.952335+0000","last_active":"2023-10-16T12:50:51.961029+0000","last_peered":"2023-10-16T12:50:51.961029+0000","last_clean":"2023-10-16T12:50:51.961029+0000","last_became_active":"2023-10-16T12:50:50.951117+0000","last_became_peered":"2023-10-16T12:50:50.951117+0000","last_unstale":"2023-10-16T12:50:51.961029+0000","last_undegraded":"2023-10-16T12:50:51.961029+0000","last_fullsized":"2023-10-16T12:50:51.961029+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:27.665346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.990717+0000","last_change":"2023-10-16T12:50:51.982322+0000","last_active":"2023-10-16T12:50:51.990717+0000","last_peered":"2023-10-16T12:50:51.990717+0000","last_clean":"2023-10-16T12:50:51.990717+0000","last_became_active":"2023-10-16T12:50:51.982000+0000","last_became_peered":"2023-10-16T12:50:51.982000+0000","last_unstale":"2023-10-16T12:50:51.990717+0000","last_undegraded":"2023-10-16T12:50:51.990717+0000","last_fullsized":"2023-10-16T12:50:51.990717+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:41.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988304+0000","last_change":"2023-10-16T12:50:50.945112+0000","last_active":"2023-10-16T12:50:51.988304+0000","last_peered":"2023-10-16T12:50:51.988304+0000","last_clean":"2023-10-16T12:50:51.988304+0000","last_became_active":"2023-10-16T12:50:50.944779+0000","last_became_peered":"2023-10-16T12:50:50.944779+0000","last_unstale":"2023-10-16T12:50:51.988304+0000","last_undegraded":"2023-10-16T12:50:51.988304+0000","last_fullsized":"2023-10-16T12:50:51.988304+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:52.614970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'186","reported_seq":497,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:50:50.933348+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:50:50.933348+0000","last_peered":"2023-10-16T12:50:50.933348+0000","last_clean":"2023-10-16T12:50:50.933348+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:50:50.933348+0000","last_undegraded":"2023-10-16T12:50:50.933348+0000","last_fullsized":"2023-10-16T12:50:50.933348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542353,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1970176,"data_stored":1244618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2490000000000001,"15min":1.2949999999999999},"min":{"1min":0.86499999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":2.1240000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3100000000000001,"15min":1.3480000000000001},"min":{"1min":0.95599999999999996,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.8899999999999999},"last":1.0289999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1779999999999999,"15min":1.2090000000000001},"min":{"1min":0.71799999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":19.158999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1519999999999999,"15min":1.208},"min":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.413,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":5.5490000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.147,"15min":1.196},"min":{"1min":0.77500000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.506,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":2.855},{"interface":"front","average":{"1min":1.276,"5min":1.284,"15min":1.323},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6200000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.077}]},{"osd":3,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3,"15min":1.2989999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.706,"5min":1.847,"15min":1.847},"last":3.2759999999999998},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4019999999999999,"15min":1.403},"min":{"1min":0.96899999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.681,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":2.4369999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.179,"15min":1.1890000000000001},"min":{"1min":0.90000000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5589999999999999,"5min":1.95,"15min":1.95},"last":2.5710000000000002},{"interface":"front","average":{"1min":1.421,"5min":1.3540000000000001,"15min":1.333},"min":{"1min":0.84999999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.337,"15min":1.331},"min":{"1min":1.141,"5min":0.624,"15min":0.624},"max":{"1min":1.8700000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":5.0140000000000002},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5089999999999999,"15min":1.4690000000000001},"min":{"1min":1.0009999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9450000000000001,"5min":2.411,"15min":2.411},"last":2.1720000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.593,"15min":1.5640000000000001},"min":{"1min":1.05,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.0760000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5009999999999999,"15min":1.45},"min":{"1min":1.262,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.97,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":2.3170000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1384448,"data_stored":654874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.214,"15min":1.28},"min":{"1min":0.73599999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.952,"5min":1.952,"15min":3.7450000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2749999999999999,"15min":1.3340000000000001},"min":{"1min":0.77100000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6279999999999999,"5min":2.0270000000000001,"15min":3.819},"last":1.4119999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1850000000000001,"15min":1.1930000000000001},"min":{"1min":0.70699999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.8979999999999999},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.218,"15min":1.21},"min":{"1min":0.74399999999999999,"5min":0.625,"15min":0.625},"max":{"1min":1.611,"5min":1.821,"15min":3.8519999999999999},"last":1.3320000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1990000000000001,"15min":1.2190000000000001},"min":{"1min":0.66900000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.6279999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2130000000000001,"15min":1.272},"min":{"1min":0.86399999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5,"5min":1.829,"15min":3.9009999999999998},"last":1.3720000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2949999999999999,"15min":1.3260000000000001},"min":{"1min":0.85199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.73,"5min":1.9399999999999999,"15min":4.2380000000000004},"last":1.111},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.298,"15min":1.351},"min":{"1min":0.96299999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.694,"5min":2.012,"15min":4.4269999999999996},"last":1.2709999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.383,"15min":1.3460000000000001},"min":{"1min":0.996,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.798,"5min":2.0830000000000002,"15min":2.3029999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.94799999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.8799999999999999,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.5409999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3,"15min":1.3109999999999999},"min":{"1min":0.86399999999999999,"5min":0.755,"15min":0.755},"max":{"1min":1.9219999999999999,"5min":1.9810000000000001,"15min":2.0910000000000002},"last":1.573},{"interface":"front","average":{"1min":1.482,"5min":1.4670000000000001,"15min":1.4470000000000001},"min":{"1min":1.0720000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7769999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.661}]},{"osd":7,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4450000000000001,"15min":1.4430000000000001},"min":{"1min":0.91300000000000003,"5min":0.878,"15min":0.878},"max":{"1min":1.796,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.458,"15min":1.458},"min":{"1min":1.167,"5min":0.98699999999999999,"15min":0.96399999999999997},"max":{"1min":1.984,"5min":2.1230000000000002,"15min":2.2610000000000001},"last":1.514}]}]},{"osd":5,"up_from":14,"seq":60129542355,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.38200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.8480000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.119},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1779999999999999,"15min":1.1819999999999999},"min":{"1min":0.77800000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.645,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":0.69599999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.22,"15min":1.2549999999999999},"min":{"1min":0.69399999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.827},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1819999999999999,"15min":1.226},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.647,"5min":1.708,"15min":1.7889999999999999},"last":1.2869999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.26,"15min":1.2989999999999999},"min":{"1min":0.71599999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.591,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.444},{"interface":"front","average":{"1min":1.155,"5min":1.24,"15min":1.2669999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.554,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.22}]},{"osd":3,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.73099999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.363,"15min":1.3799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.929,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.0649999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1930000000000001,"15min":1.2050000000000001},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7430000000000001,"5min":1.996,"15min":1.996},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.361,"15min":1.3140000000000001},"min":{"1min":0.55000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.0910000000000002},"last":1.248}]},{"osd":6,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.321,"15min":1.3220000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8,"5min":1.927,"15min":1.927},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.3939999999999999,"15min":1.3799999999999999},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9470000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3620000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4470000000000001,"15min":1.448},"min":{"1min":0.64300000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.3900000000000001},"last":1.556},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5029999999999999},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.0209999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.524}]}]},{"osd":4,"up_from":14,"seq":60129542355,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58880,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312960,"statfs":{"total":96636764160,"available":96576471040,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1799999999999999,"15min":1.2010000000000001},"min":{"1min":0.65700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4359999999999999,"5min":1.766,"15min":2.2200000000000002},"last":1.264},{"interface":"front","average":{"1min":1.032,"5min":1.1120000000000001,"15min":1.151},"min":{"1min":0.73499999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.569,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.141,"15min":1.1899999999999999},"min":{"1min":0.69299999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5669999999999999,"5min":1.681,"15min":1.865},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.228,"15min":1.26},"min":{"1min":0.76500000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.532,"5min":1.724,"15min":1.9390000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2549999999999999,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.81899999999999995,"15min":0.753},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":1.208},{"interface":"front","average":{"1min":1.24,"5min":1.292,"15min":1.329},"min":{"1min":0.78800000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6379999999999999,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.46}]},{"osd":3,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2569999999999999,"15min":1.278},"min":{"1min":0.91600000000000004,"5min":0.80900000000000005,"15min":0.752},"max":{"1min":1.804,"5min":1.804,"15min":1.8100000000000001},"last":1.155},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.359,"15min":1.381},"min":{"1min":0.89800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.4139999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3420000000000001,"15min":1.3049999999999999},"min":{"1min":1.1699999999999999,"5min":0.86799999999999999,"15min":0.50900000000000001},"max":{"1min":1.917,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.379},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3220000000000001,"15min":1.3009999999999999},"min":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.53600000000000003},"max":{"1min":1.994,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.5640000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.407,"15min":1.391},"min":{"1min":0.92300000000000004,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.329},{"interface":"front","average":{"1min":1.569,"5min":1.488,"15min":1.4830000000000001},"min":{"1min":1.282,"5min":0.90400000000000003,"15min":0.66600000000000004},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.53}]},{"osd":7,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.4870000000000001,"15min":1.4790000000000001},"min":{"1min":1.0600000000000001,"5min":0.88600000000000001,"15min":0.60399999999999998},"max":{"1min":2.109,"5min":2.109,"15min":2.254},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.591,"15min":1.5960000000000001},"min":{"1min":1.026,"5min":0.94799999999999995,"15min":0.71299999999999997},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.6519999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542355,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.91700000000000004,"15min":0.95399999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.5649999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.1180000000000001,"15min":1.169},"min":{"1min":0.44900000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.244,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.2569999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.004,"15min":1.0369999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.337,"5min":1.758,"15min":1.758},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.034,"15min":1.0740000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.45600000000000002},"max":{"1min":1.133,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.95199999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.0229999999999999,"15min":1.0740000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.1639999999999999,"5min":1.7949999999999999,"15min":1.9510000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.2689999999999999,"15min":1.3540000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.359,"5min":1.8600000000000001,"15min":1.921},"last":1.5569999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4530000000000001,"15min":1.484},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.77600000000000002},"max":{"1min":1.806,"5min":1.956,"15min":1.974},"last":1.526},{"interface":"front","average":{"1min":1.236,"5min":1.391,"15min":1.4119999999999999},"min":{"1min":0.82699999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.607,"5min":1.8939999999999999,"15min":2.0539999999999998},"last":1.7010000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.363,"15min":1.389},"min":{"1min":0.83899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5329999999999999,"5min":2.028,"15min":2.0539999999999998},"last":1.623},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.4219999999999999,"15min":1.4670000000000001},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.80300000000000005},"max":{"1min":1.575,"5min":1.9990000000000001,"15min":2.3540000000000001},"last":1.407}]},{"osd":6,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.4139999999999999,"15min":1.4410000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.881,"5min":2.2989999999999999,"15min":2.3220000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.696,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.3520000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.512,"15min":1.5369999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6850000000000001,"5min":2.1339999999999999,"15min":2.431},"last":1.472},{"interface":"front","average":{"1min":1.381,"5min":1.5740000000000001,"15min":1.603},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.956,"5min":2.552,"15min":2.552},"last":1.5900000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542353,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1376256,"data_stored":654762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.88900000000000001,"15min":0.91800000000000004},"min":{"1min":0.44600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.413,"5min":1.742,"15min":1.7889999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0429999999999999,"15min":1.1040000000000001},"min":{"1min":0.69299999999999995,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.746,"5min":1.746,"15min":2.085},"last":1.091}]},{"osd":1,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.1240000000000001,"15min":1.1279999999999999},"min":{"1min":0.64100000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.677,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.125,"5min":1.129,"15min":1.2210000000000001},"min":{"1min":0.58599999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.8440000000000001,"5min":2.347,"15min":2.4289999999999998},"last":1.224}]},{"osd":3,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.006,"15min":1.079},"min":{"1min":0.498,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5860000000000001,"5min":1.6879999999999999,"15min":1.859},"last":1.016},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3140000000000001},"min":{"1min":0.52800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.972,"5min":2.1000000000000001,"15min":2.1970000000000001},"last":1.4870000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4099999999999999,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.2149999999999999,"5min":2.431,"15min":2.431},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.369,"15min":1.4450000000000001},"min":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.9430000000000001,"5min":2.0670000000000002,"15min":2.2530000000000001},"last":1.454}]},{"osd":5,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4350000000000001,"15min":1.502},"min":{"1min":0.99099999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.0390000000000001,"5min":2.23,"15min":2.3250000000000002},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3400000000000001,"15min":1.417},"min":{"1min":0.751,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.679,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.6259999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3839999999999999,"15min":1.4430000000000001},"min":{"1min":0.89400000000000002,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.7090000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.514,"15min":1.556},"min":{"1min":0.95399999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.3069999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.524,"15min":1.611},"min":{"1min":0.88600000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.0840000000000001,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.4770000000000001,"15min":1.542},"min":{"1min":0.877,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.819,"5min":2.573,"15min":2.573},"last":1.8480000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58876,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312964,"statfs":{"total":96636764160,"available":96576475136,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.91700000000000004,"15min":0.90800000000000003},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.609,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.874},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1,"15min":1.0880000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.8500000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.2130000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":0.98299999999999998},"min":{"1min":0.45900000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.49,"5min":1.8460000000000001,"15min":2.262},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1299999999999999,"15min":1.1799999999999999},"min":{"1min":0.46200000000000002,"5min":0.433,"15min":0.433},"max":{"1min":1.5309999999999999,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.0409999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2270000000000001,"15min":1.3149999999999999},"min":{"1min":0.47899999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.8460000000000001,"5min":1.988,"15min":2.2349999999999999},"last":1.379},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.161,"15min":1.212},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.6200000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.399}]},{"osd":4,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.2490000000000001,"15min":1.306},"min":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.7350000000000001,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.415,"15min":1.4390000000000001},"min":{"1min":0.71899999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9970000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.1699999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.357,"15min":1.429},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.8500000000000001,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.619},{"interface":"front","average":{"1min":1.304,"5min":1.3839999999999999,"15min":1.4299999999999999},"min":{"1min":0.85599999999999998,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":1.6679999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":0.95899999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4890000000000001,"15min":1.5289999999999999},"min":{"1min":0.77400000000000002,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.1320000000000001,"5min":2.1779999999999999,"15min":2.202},"last":1.292},{"interface":"front","average":{"1min":1.516,"5min":1.4710000000000001,"15min":1.52},"min":{"1min":0.71399999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.1640000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.7529999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5589999999999999,"15min":1.621},"min":{"1min":0.75900000000000001,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.2440000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.508,"15min":1.5569999999999999},"min":{"1min":0.68200000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.9139999999999999,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.026}]}]},{"osd":0,"up_from":14,"seq":60129542354,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37272,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334568,"statfs":{"total":96636764160,"available":96598597632,"internally_reserved":0,"allocated":1970176,"data_stored":1244582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.85299999999999998,"15min":0.89300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.655,"5min":1.655,"15min":2.5720000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.0509999999999999,"15min":1.089},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.439},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":2.1800000000000002},"last":0.57399999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.0660000000000001,"15min":1.127},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.579,"5min":1.8120000000000001,"15min":2.7650000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.1140000000000001,"15min":1.159},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.5880000000000001,"5min":1.952,"15min":1.952},"last":0.64700000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.016,"15min":1.0820000000000001},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.609,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.226,"15min":1.3009999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.696,"5min":1.8740000000000001,"15min":3.6579999999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2929999999999999,"15min":1.3300000000000001},"min":{"1min":0.71899999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.542,"5min":1.8660000000000001,"15min":2.6920000000000002},"last":1.121},{"interface":"front","average":{"1min":1.252,"5min":1.3799999999999999,"15min":1.4390000000000001},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.845,"5min":1.966,"15min":3.544},"last":0.90600000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.377,"15min":1.4219999999999999},"min":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8320000000000001,"5min":2.2229999999999999,"15min":2.2440000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3560000000000001,"15min":1.417},"min":{"1min":0.872,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.7490000000000001,"5min":2.02,"15min":2.8610000000000002},"last":1.1859999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.488,"15min":1.5289999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.9790000000000001,"5min":2.0750000000000002,"15min":3.7890000000000001},"last":1.034},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.4470000000000001,"15min":1.516},"min":{"1min":0.92000000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.95,"5min":2.145,"15min":2.363},"last":0.88}]},{"osd":7,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5620000000000001,"15min":1.6200000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.056,"5min":2.3490000000000002,"15min":2.4729999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.389,"5min":1.4990000000000001,"15min":1.552},"min":{"1min":0.81599999999999995,"5min":0.755,"15min":0.755},"max":{"1min":1.8560000000000001,"5min":2.2050000000000001,"15min":3.1989999999999998},"last":1.2370000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:50:56.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:50:56.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.737+0000 7f9a43fff640 1 Processor -- start 2023-10-16T12:50:56.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.737+0000 7f9a43fff640 1 -- start start 2023-10-16T12:50:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.739+0000 7f9a43fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 0x7f9a440cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.739+0000 7f9a43fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4415b9d0 con 0x7f9a440ccfb0 2023-10-16T12:50:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.739+0000 7f9a43fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4415bcb0 con 0x7f9a4406c040 2023-10-16T12:50:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.739+0000 7f9a43fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4415bf90 con 0x7f9a440ad970 2023-10-16T12:50:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.739+0000 7f9a437fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9a440ccfb0 0x7f9a440cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35364/0 (socket says 172.21.15.60:35364) 2023-10-16T12:50:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.739+0000 7f9a437fe640 1 -- 172.21.15.60:0/1676612812 learned_addr learned my addr 172.21.15.60:0/1676612812 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a427fc640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 0x7f9a440cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735547947 0 0) 0x7f9a4415b9d0 con 0x7f9a440ccfb0 2023-10-16T12:50:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c003520 con 0x7f9a440ccfb0 2023-10-16T12:50:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474065992 0 0) 0x7f9a2c003520 con 0x7f9a440ccfb0 2023-10-16T12:50:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 msgr2=0x7f9a440cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 0x7f9a440cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 >> v1:172.21.15.144:6789/0 conn(0x7f9a4406c040 legacy=0x7f9a4406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a440cc270 con 0x7f9a440ccfb0 2023-10-16T12:50:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9a3c0019f0 con 0x7f9a440ccfb0 2023-10-16T12:50:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 >> v1:172.21.15.60:6789/0 conn(0x7f9a440ccfb0 legacy=0x7f9a440cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c0039a0 0x7f9a2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 0x7f9a2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a2c004d00 0x7f9a2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4415bf90 con 0x7f9a2c004d00 2023-10-16T12:50:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c00ba60 con 0x7f9a2c0039a0 2023-10-16T12:50:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.740+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c00bc20 con 0x7f9a440ad970 2023-10-16T12:50:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a437fe640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a2c004d00 0x7f9a2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a427fc640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 0x7f9a2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a42ffd640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c0039a0 0x7f9a2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a437fe640 1 -- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 msgr2=0x7f9a2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a437fe640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a440ad970 0x7f9a2c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a437fe640 1 -- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c0039a0 msgr2=0x7f9a2c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a437fe640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c0039a0 0x7f9a2c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a437fe640 1 -- 172.21.15.60:0/1676612812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c003520 con 0x7f9a2c004d00 2023-10-16T12:50:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.741+0000 7f9a42ffd640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c0039a0 0x7f9a2c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.742+0000 7f9a437fe640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a2c004d00 0x7f9a2c005180 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00a830 tx=0x7f9a2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361458108af2a10a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0072d0 con 0x7f9a2c004d00 2023-10-16T12:50:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c02f070 con 0x7f9a2c004d00 2023-10-16T12:50:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a41ffb640 1 -- 172.21.15.60:0/1676612812 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0076b0 con 0x7f9a2c004d00 2023-10-16T12:50:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a43fff640 1 -- 172.21.15.60:0/1676612812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a2c004d00 msgr2=0x7f9a2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a43fff640 1 --2- 172.21.15.60:0/1676612812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a2c004d00 0x7f9a2c005180 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00a830 tx=0x7f9a2c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a437fe640 1 -- 172.21.15.60:0/1676612812 reap_dead start 2023-10-16T12:50:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a43fff640 1 -- 172.21.15.60:0/1676612812 shutdown_connections 2023-10-16T12:50:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a43fff640 1 -- 172.21.15.60:0/1676612812 >> 172.21.15.60:0/1676612812 conn(0x7f9a440ce6a0 msgr2=0x7f9a440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.743+0000 7f9a43fff640 1 -- 172.21.15.60:0/1676612812 shutdown_connections 2023-10-16T12:50:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.744+0000 7f9a43fff640 1 -- 172.21.15.60:0/1676612812 wait complete. 2023-10-16T12:50:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.745+0000 7f9a43fff640 1 Processor -- start 2023-10-16T12:50:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.745+0000 7f9a43fff640 1 -- start start 2023-10-16T12:50:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.746+0000 7f9a43fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c004d00 0x7f9a440ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.746+0000 7f9a43fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a44156560 0x7f9a440c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.746+0000 7f9a43fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 0x7f9a44150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.746+0000 7f9a43fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a44155550 con 0x7f9a440c7b50 2023-10-16T12:50:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.746+0000 7f9a43fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a441560d0 con 0x7f9a2c004d00 2023-10-16T12:50:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.746+0000 7f9a43fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a441557a0 con 0x7f9a44156560 2023-10-16T12:50:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 0x7f9a44150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 0x7f9a44150b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41342/0 (socket says 172.21.15.60:41342) 2023-10-16T12:50:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 -- 172.21.15.60:0/3165280827 learned_addr learned my addr 172.21.15.60:0/3165280827 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a427fc640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a44156560 0x7f9a440c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a42ffd640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c004d00 0x7f9a440ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 -- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a44156560 msgr2=0x7f9a440c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a44156560 0x7f9a440c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 -- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c004d00 msgr2=0x7f9a440ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c004d00 0x7f9a440ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 -- 172.21.15.60:0/3165280827 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a440cdcd0 con 0x7f9a440c7b50 2023-10-16T12:50:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a42ffd640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c004d00 0x7f9a440ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.747+0000 7f9a437fe640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 0x7f9a44150b80 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00a830 tx=0x7f9a3c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.748+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0074e0 con 0x7f9a440c7b50 2023-10-16T12:50:56.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.748+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c001ae0 con 0x7f9a440c7b50 2023-10-16T12:50:56.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.748+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c007840 con 0x7f9a440c7b50 2023-10-16T12:50:56.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.748+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a44155a80 con 0x7f9a440c7b50 2023-10-16T12:50:56.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.748+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a440c7fd0 con 0x7f9a440c7b50 2023-10-16T12:50:56.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.749+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4406b330 con 0x7f9a440c7b50 2023-10-16T12:50:56.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.750+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a3c031030 con 0x7f9a440c7b50 2023-10-16T12:50:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.751+0000 7f9a2bfff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f9a3006afc0 0x7f9a3006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.751+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a3c0ab3d0 con 0x7f9a440c7b50 2023-10-16T12:50:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.752+0000 7f9a42ffd640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f9a3006afc0 0x7f9a3006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.753+0000 7f9a42ffd640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f9a3006afc0 0x7f9a3006d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a340016f0 tx=0x7f9a34009040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.755+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a3c076250 con 0x7f9a440c7b50 2023-10-16T12:50:56.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.948+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 --> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a440c8250 con 0x7f9a3006afc0 2023-10-16T12:50:56.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:56.964 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:56.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.971+0000 7f9a2bfff640 1 -- 172.21.15.60:0/3165280827 <== mgr.5705 v2:172.21.15.144:6832/3513615444 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287168 (secure 0 0 0) 0x7f9a3400d040 con 0x7f9a3006afc0 2023-10-16T12:50:56.977 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:56.978 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:50:56.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f9a3006afc0 msgr2=0x7f9a3006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f9a3006afc0 0x7f9a3006d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a340016f0 tx=0x7f9a34009040 comp rx=0 tx=0).stop 2023-10-16T12:50:56.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 msgr2=0x7f9a44150b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:56.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 0x7f9a44150b80 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00a830 tx=0x7f9a3c001d80 comp rx=0 tx=0).stop 2023-10-16T12:50:56.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 shutdown_connections 2023-10-16T12:50:56.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f9a3006afc0 0x7f9a3006d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a44156560 0x7f9a440c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a440c7b50 0x7f9a44150b80 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 --2- 172.21.15.60:0/3165280827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a2c004d00 0x7f9a440ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 >> 172.21.15.60:0/3165280827 conn(0x7f9a440ce6a0 msgr2=0x7f9a4415c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 shutdown_connections 2023-10-16T12:50:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:56.977+0000 7f9a43fff640 1 -- 172.21.15.60:0/3165280827 wait complete. 2023-10-16T12:50:56.997 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T12:50:56.284069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":327,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330692,"kb_used_data":11856,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754644028,"statfs":{"total":773094113280,"available":772755484672,"internally_reserved":0,"allocated":12140544,"data_stored":6388456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":326356536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001904"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992506+0000","last_change":"2023-10-16T12:50:51.985499+0000","last_active":"2023-10-16T12:50:51.992506+0000","last_peered":"2023-10-16T12:50:51.992506+0000","last_clean":"2023-10-16T12:50:51.992506+0000","last_became_active":"2023-10-16T12:50:51.984856+0000","last_became_peered":"2023-10-16T12:50:51.984856+0000","last_unstale":"2023-10-16T12:50:51.992506+0000","last_undegraded":"2023-10-16T12:50:51.992506+0000","last_fullsized":"2023-10-16T12:50:51.992506+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:45.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984788+0000","last_change":"2023-10-16T12:50:51.979288+0000","last_active":"2023-10-16T12:50:51.984788+0000","last_peered":"2023-10-16T12:50:51.984788+0000","last_clean":"2023-10-16T12:50:51.984788+0000","last_became_active":"2023-10-16T12:50:51.978924+0000","last_became_peered":"2023-10-16T12:50:51.978924+0000","last_unstale":"2023-10-16T12:50:51.984788+0000","last_undegraded":"2023-10-16T12:50:51.984788+0000","last_fullsized":"2023-10-16T12:50:51.984788+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:08.207898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993777+0000","last_change":"2023-10-16T12:50:51.992352+0000","last_active":"2023-10-16T12:50:51.993777+0000","last_peered":"2023-10-16T12:50:51.993777+0000","last_clean":"2023-10-16T12:50:51.993777+0000","last_became_active":"2023-10-16T12:50:51.991858+0000","last_became_peered":"2023-10-16T12:50:51.991858+0000","last_unstale":"2023-10-16T12:50:51.993777+0000","last_undegraded":"2023-10-16T12:50:51.993777+0000","last_fullsized":"2023-10-16T12:50:51.993777+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:16.349362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995687+0000","last_change":"2023-10-16T12:50:51.989011+0000","last_active":"2023-10-16T12:50:51.995687+0000","last_peered":"2023-10-16T12:50:51.995687+0000","last_clean":"2023-10-16T12:50:51.995687+0000","last_became_active":"2023-10-16T12:50:51.988765+0000","last_became_peered":"2023-10-16T12:50:51.988765+0000","last_unstale":"2023-10-16T12:50:51.995687+0000","last_undegraded":"2023-10-16T12:50:51.995687+0000","last_fullsized":"2023-10-16T12:50:51.995687+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:06.346666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991230+0000","last_change":"2023-10-16T12:50:51.990368+0000","last_active":"2023-10-16T12:50:51.991230+0000","last_peered":"2023-10-16T12:50:51.991230+0000","last_clean":"2023-10-16T12:50:51.991230+0000","last_became_active":"2023-10-16T12:50:51.990043+0000","last_became_peered":"2023-10-16T12:50:51.990043+0000","last_unstale":"2023-10-16T12:50:51.991230+0000","last_undegraded":"2023-10-16T12:50:51.991230+0000","last_fullsized":"2023-10-16T12:50:51.991230+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:58.838268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.986570+0000","last_change":"2023-10-16T12:50:51.983430+0000","last_active":"2023-10-16T12:50:51.986570+0000","last_peered":"2023-10-16T12:50:51.986570+0000","last_clean":"2023-10-16T12:50:51.986570+0000","last_became_active":"2023-10-16T12:50:51.983148+0000","last_became_peered":"2023-10-16T12:50:51.983148+0000","last_unstale":"2023-10-16T12:50:51.986570+0000","last_undegraded":"2023-10-16T12:50:51.986570+0000","last_fullsized":"2023-10-16T12:50:51.986570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:43.803664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984997+0000","last_change":"2023-10-16T12:50:51.980807+0000","last_active":"2023-10-16T12:50:51.984997+0000","last_peered":"2023-10-16T12:50:51.984997+0000","last_clean":"2023-10-16T12:50:51.984997+0000","last_became_active":"2023-10-16T12:50:51.980523+0000","last_became_peered":"2023-10-16T12:50:51.980523+0000","last_unstale":"2023-10-16T12:50:51.984997+0000","last_undegraded":"2023-10-16T12:50:51.984997+0000","last_fullsized":"2023-10-16T12:50:51.984997+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:12.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.973686+0000","last_change":"2023-10-16T12:50:51.971728+0000","last_active":"2023-10-16T12:50:51.973686+0000","last_peered":"2023-10-16T12:50:51.973686+0000","last_clean":"2023-10-16T12:50:51.973686+0000","last_became_active":"2023-10-16T12:50:51.971426+0000","last_became_peered":"2023-10-16T12:50:51.971426+0000","last_unstale":"2023-10-16T12:50:51.973686+0000","last_undegraded":"2023-10-16T12:50:51.973686+0000","last_fullsized":"2023-10-16T12:50:51.973686+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:20.941851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977674+0000","last_change":"2023-10-16T12:50:51.974747+0000","last_active":"2023-10-16T12:50:51.977674+0000","last_peered":"2023-10-16T12:50:51.977674+0000","last_clean":"2023-10-16T12:50:51.977674+0000","last_became_active":"2023-10-16T12:50:51.974450+0000","last_became_peered":"2023-10-16T12:50:51.974450+0000","last_unstale":"2023-10-16T12:50:51.977674+0000","last_undegraded":"2023-10-16T12:50:51.977674+0000","last_fullsized":"2023-10-16T12:50:51.977674+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:28.869259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.983410+0000","last_change":"2023-10-16T12:50:51.980703+0000","last_active":"2023-10-16T12:50:51.983410+0000","last_peered":"2023-10-16T12:50:51.983410+0000","last_clean":"2023-10-16T12:50:51.983410+0000","last_became_active":"2023-10-16T12:50:51.975265+0000","last_became_peered":"2023-10-16T12:50:51.975265+0000","last_unstale":"2023-10-16T12:50:51.983410+0000","last_undegraded":"2023-10-16T12:50:51.983410+0000","last_fullsized":"2023-10-16T12:50:51.983410+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:00.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.986715+0000","last_change":"2023-10-16T12:50:51.982384+0000","last_active":"2023-10-16T12:50:51.986715+0000","last_peered":"2023-10-16T12:50:51.986715+0000","last_clean":"2023-10-16T12:50:51.986715+0000","last_became_active":"2023-10-16T12:50:51.982025+0000","last_became_peered":"2023-10-16T12:50:51.982025+0000","last_unstale":"2023-10-16T12:50:51.986715+0000","last_undegraded":"2023-10-16T12:50:51.986715+0000","last_fullsized":"2023-10-16T12:50:51.986715+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:24.702209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981677+0000","last_change":"2023-10-16T12:50:51.978401+0000","last_active":"2023-10-16T12:50:51.981677+0000","last_peered":"2023-10-16T12:50:51.981677+0000","last_clean":"2023-10-16T12:50:51.981677+0000","last_became_active":"2023-10-16T12:50:51.978142+0000","last_became_peered":"2023-10-16T12:50:51.978142+0000","last_unstale":"2023-10-16T12:50:51.981677+0000","last_undegraded":"2023-10-16T12:50:51.981677+0000","last_fullsized":"2023-10-16T12:50:51.981677+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:08.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981409+0000","last_change":"2023-10-16T12:50:51.979102+0000","last_active":"2023-10-16T12:50:51.981409+0000","last_peered":"2023-10-16T12:50:51.981409+0000","last_clean":"2023-10-16T12:50:51.981409+0000","last_became_active":"2023-10-16T12:50:51.978828+0000","last_became_peered":"2023-10-16T12:50:51.978828+0000","last_unstale":"2023-10-16T12:50:51.981409+0000","last_undegraded":"2023-10-16T12:50:51.981409+0000","last_fullsized":"2023-10-16T12:50:51.981409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:28.753302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995814+0000","last_change":"2023-10-16T12:50:51.989860+0000","last_active":"2023-10-16T12:50:51.995814+0000","last_peered":"2023-10-16T12:50:51.995814+0000","last_clean":"2023-10-16T12:50:51.995814+0000","last_became_active":"2023-10-16T12:50:51.989356+0000","last_became_peered":"2023-10-16T12:50:51.989356+0000","last_unstale":"2023-10-16T12:50:51.995814+0000","last_undegraded":"2023-10-16T12:50:51.995814+0000","last_fullsized":"2023-10-16T12:50:51.995814+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:10.845718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.980673+0000","last_change":"2023-10-16T12:50:51.979267+0000","last_active":"2023-10-16T12:50:51.980673+0000","last_peered":"2023-10-16T12:50:51.980673+0000","last_clean":"2023-10-16T12:50:51.980673+0000","last_became_active":"2023-10-16T12:50:51.978518+0000","last_became_peered":"2023-10-16T12:50:51.978518+0000","last_unstale":"2023-10-16T12:50:51.980673+0000","last_undegraded":"2023-10-16T12:50:51.980673+0000","last_fullsized":"2023-10-16T12:50:51.980673+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:22.239539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.976751+0000","last_change":"2023-10-16T12:50:51.971243+0000","last_active":"2023-10-16T12:50:51.976751+0000","last_peered":"2023-10-16T12:50:51.976751+0000","last_clean":"2023-10-16T12:50:51.976751+0000","last_became_active":"2023-10-16T12:50:51.970016+0000","last_became_peered":"2023-10-16T12:50:51.970016+0000","last_unstale":"2023-10-16T12:50:51.976751+0000","last_undegraded":"2023-10-16T12:50:51.976751+0000","last_fullsized":"2023-10-16T12:50:51.976751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:37.226580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981878+0000","last_change":"2023-10-16T12:50:51.981295+0000","last_active":"2023-10-16T12:50:51.981878+0000","last_peered":"2023-10-16T12:50:51.981878+0000","last_clean":"2023-10-16T12:50:51.981878+0000","last_became_active":"2023-10-16T12:50:51.980999+0000","last_became_peered":"2023-10-16T12:50:51.980999+0000","last_unstale":"2023-10-16T12:50:51.981878+0000","last_undegraded":"2023-10-16T12:50:51.981878+0000","last_fullsized":"2023-10-16T12:50:51.981878+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:58.055049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992949+0000","last_change":"2023-10-16T12:50:51.986647+0000","last_active":"2023-10-16T12:50:51.992949+0000","last_peered":"2023-10-16T12:50:51.992949+0000","last_clean":"2023-10-16T12:50:51.992949+0000","last_became_active":"2023-10-16T12:50:51.985262+0000","last_became_peered":"2023-10-16T12:50:51.985262+0000","last_unstale":"2023-10-16T12:50:51.992949+0000","last_undegraded":"2023-10-16T12:50:51.992949+0000","last_fullsized":"2023-10-16T12:50:51.992949+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:56.127519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981663+0000","last_change":"2023-10-16T12:50:51.975066+0000","last_active":"2023-10-16T12:50:51.981663+0000","last_peered":"2023-10-16T12:50:51.981663+0000","last_clean":"2023-10-16T12:50:51.981663+0000","last_became_active":"2023-10-16T12:50:51.974760+0000","last_became_peered":"2023-10-16T12:50:51.974760+0000","last_unstale":"2023-10-16T12:50:51.981663+0000","last_undegraded":"2023-10-16T12:50:51.981663+0000","last_fullsized":"2023-10-16T12:50:51.981663+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:39.777334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995662+0000","last_change":"2023-10-16T12:50:51.986719+0000","last_active":"2023-10-16T12:50:51.995662+0000","last_peered":"2023-10-16T12:50:51.995662+0000","last_clean":"2023-10-16T12:50:51.995662+0000","last_became_active":"2023-10-16T12:50:51.986369+0000","last_became_peered":"2023-10-16T12:50:51.986369+0000","last_unstale":"2023-10-16T12:50:51.995662+0000","last_undegraded":"2023-10-16T12:50:51.995662+0000","last_fullsized":"2023-10-16T12:50:51.995662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:22.655885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988528+0000","last_change":"2023-10-16T12:50:51.984511+0000","last_active":"2023-10-16T12:50:51.988528+0000","last_peered":"2023-10-16T12:50:51.988528+0000","last_clean":"2023-10-16T12:50:51.988528+0000","last_became_active":"2023-10-16T12:50:51.983895+0000","last_became_peered":"2023-10-16T12:50:51.983895+0000","last_unstale":"2023-10-16T12:50:51.988528+0000","last_undegraded":"2023-10-16T12:50:51.988528+0000","last_fullsized":"2023-10-16T12:50:51.988528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:42.291463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.983924+0000","last_change":"2023-10-16T12:50:51.977461+0000","last_active":"2023-10-16T12:50:51.983924+0000","last_peered":"2023-10-16T12:50:51.983924+0000","last_clean":"2023-10-16T12:50:51.983924+0000","last_became_active":"2023-10-16T12:50:51.977116+0000","last_became_peered":"2023-10-16T12:50:51.977116+0000","last_unstale":"2023-10-16T12:50:51.983924+0000","last_undegraded":"2023-10-16T12:50:51.983924+0000","last_fullsized":"2023-10-16T12:50:51.983924+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:16.572912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.982736+0000","last_change":"2023-10-16T12:50:51.980110+0000","last_active":"2023-10-16T12:50:51.982736+0000","last_peered":"2023-10-16T12:50:51.982736+0000","last_clean":"2023-10-16T12:50:51.982736+0000","last_became_active":"2023-10-16T12:50:51.976849+0000","last_became_peered":"2023-10-16T12:50:51.976849+0000","last_unstale":"2023-10-16T12:50:51.982736+0000","last_undegraded":"2023-10-16T12:50:51.982736+0000","last_fullsized":"2023-10-16T12:50:51.982736+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:04.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991607+0000","last_change":"2023-10-16T12:50:51.984966+0000","last_active":"2023-10-16T12:50:51.991607+0000","last_peered":"2023-10-16T12:50:51.991607+0000","last_clean":"2023-10-16T12:50:51.991607+0000","last_became_active":"2023-10-16T12:50:51.984106+0000","last_became_peered":"2023-10-16T12:50:51.984106+0000","last_unstale":"2023-10-16T12:50:51.991607+0000","last_undegraded":"2023-10-16T12:50:51.991607+0000","last_fullsized":"2023-10-16T12:50:51.991607+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:23.228050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984037+0000","last_change":"2023-10-16T12:50:51.977403+0000","last_active":"2023-10-16T12:50:51.984037+0000","last_peered":"2023-10-16T12:50:51.984037+0000","last_clean":"2023-10-16T12:50:51.984037+0000","last_became_active":"2023-10-16T12:50:51.975047+0000","last_became_peered":"2023-10-16T12:50:51.975047+0000","last_unstale":"2023-10-16T12:50:51.984037+0000","last_undegraded":"2023-10-16T12:50:51.984037+0000","last_fullsized":"2023-10-16T12:50:51.984037+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:37:24.318824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.989610+0000","last_change":"2023-10-16T12:50:51.988066+0000","last_active":"2023-10-16T12:50:51.989610+0000","last_peered":"2023-10-16T12:50:51.989610+0000","last_clean":"2023-10-16T12:50:51.989610+0000","last_became_active":"2023-10-16T12:50:51.986386+0000","last_became_peered":"2023-10-16T12:50:51.986386+0000","last_unstale":"2023-10-16T12:50:51.989610+0000","last_undegraded":"2023-10-16T12:50:51.989610+0000","last_fullsized":"2023-10-16T12:50:51.989610+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:19:22.239539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977494+0000","last_change":"2023-10-16T12:50:51.975391+0000","last_active":"2023-10-16T12:50:51.977494+0000","last_peered":"2023-10-16T12:50:51.977494+0000","last_clean":"2023-10-16T12:50:51.977494+0000","last_became_active":"2023-10-16T12:50:51.975118+0000","last_became_peered":"2023-10-16T12:50:51.975118+0000","last_unstale":"2023-10-16T12:50:51.977494+0000","last_undegraded":"2023-10-16T12:50:51.977494+0000","last_fullsized":"2023-10-16T12:50:51.977494+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:07.518531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993072+0000","last_change":"2023-10-16T12:50:51.989431+0000","last_active":"2023-10-16T12:50:51.993072+0000","last_peered":"2023-10-16T12:50:51.993072+0000","last_clean":"2023-10-16T12:50:51.993072+0000","last_became_active":"2023-10-16T12:50:51.988780+0000","last_became_peered":"2023-10-16T12:50:51.988780+0000","last_unstale":"2023-10-16T12:50:51.993072+0000","last_undegraded":"2023-10-16T12:50:51.993072+0000","last_fullsized":"2023-10-16T12:50:51.993072+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:45:40.616676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984022+0000","last_change":"2023-10-16T12:50:51.979142+0000","last_active":"2023-10-16T12:50:51.984022+0000","last_peered":"2023-10-16T12:50:51.984022+0000","last_clean":"2023-10-16T12:50:51.984022+0000","last_became_active":"2023-10-16T12:50:51.974676+0000","last_became_peered":"2023-10-16T12:50:51.974676+0000","last_unstale":"2023-10-16T12:50:51.984022+0000","last_undegraded":"2023-10-16T12:50:51.984022+0000","last_fullsized":"2023-10-16T12:50:51.984022+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:38.497935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.996961+0000","last_change":"2023-10-16T12:50:51.994200+0000","last_active":"2023-10-16T12:50:51.996961+0000","last_peered":"2023-10-16T12:50:51.996961+0000","last_clean":"2023-10-16T12:50:51.996961+0000","last_became_active":"2023-10-16T12:50:51.993880+0000","last_became_peered":"2023-10-16T12:50:51.993880+0000","last_unstale":"2023-10-16T12:50:51.996961+0000","last_undegraded":"2023-10-16T12:50:51.996961+0000","last_fullsized":"2023-10-16T12:50:51.996961+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:44.304350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978626+0000","last_change":"2023-10-16T12:50:51.977007+0000","last_active":"2023-10-16T12:50:51.978626+0000","last_peered":"2023-10-16T12:50:51.978626+0000","last_clean":"2023-10-16T12:50:51.978626+0000","last_became_active":"2023-10-16T12:50:51.976740+0000","last_became_peered":"2023-10-16T12:50:51.976740+0000","last_unstale":"2023-10-16T12:50:51.978626+0000","last_undegraded":"2023-10-16T12:50:51.978626+0000","last_fullsized":"2023-10-16T12:50:51.978626+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:40.127068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992256+0000","last_change":"2023-10-16T12:50:51.984287+0000","last_active":"2023-10-16T12:50:51.992256+0000","last_peered":"2023-10-16T12:50:51.992256+0000","last_clean":"2023-10-16T12:50:51.992256+0000","last_became_active":"2023-10-16T12:50:51.984018+0000","last_became_peered":"2023-10-16T12:50:51.984018+0000","last_unstale":"2023-10-16T12:50:51.992256+0000","last_undegraded":"2023-10-16T12:50:51.992256+0000","last_fullsized":"2023-10-16T12:50:51.992256+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:48.737941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977243+0000","last_change":"2023-10-16T12:50:51.973838+0000","last_active":"2023-10-16T12:50:51.977243+0000","last_peered":"2023-10-16T12:50:51.977243+0000","last_clean":"2023-10-16T12:50:51.977243+0000","last_became_active":"2023-10-16T12:50:51.973524+0000","last_became_peered":"2023-10-16T12:50:51.973524+0000","last_unstale":"2023-10-16T12:50:51.977243+0000","last_undegraded":"2023-10-16T12:50:51.977243+0000","last_fullsized":"2023-10-16T12:50:51.977243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:44.773337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"112'4","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.998076+0000","last_change":"2023-10-16T12:50:50.959454+0000","last_active":"2023-10-16T12:50:51.998076+0000","last_peered":"2023-10-16T12:50:51.998076+0000","last_clean":"2023-10-16T12:50:51.998076+0000","last_became_active":"2023-10-16T12:50:50.959131+0000","last_became_peered":"2023-10-16T12:50:50.959131+0000","last_unstale":"2023-10-16T12:50:51.998076+0000","last_undegraded":"2023-10-16T12:50:51.998076+0000","last_fullsized":"2023-10-16T12:50:51.998076+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:44.125691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992525+0000","last_change":"2023-10-16T12:50:51.985714+0000","last_active":"2023-10-16T12:50:51.992525+0000","last_peered":"2023-10-16T12:50:51.992525+0000","last_clean":"2023-10-16T12:50:51.992525+0000","last_became_active":"2023-10-16T12:50:51.985230+0000","last_became_peered":"2023-10-16T12:50:51.985230+0000","last_unstale":"2023-10-16T12:50:51.992525+0000","last_undegraded":"2023-10-16T12:50:51.992525+0000","last_fullsized":"2023-10-16T12:50:51.992525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:54:46.760198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993694+0000","last_change":"2023-10-16T12:50:50.946700+0000","last_active":"2023-10-16T12:50:51.993694+0000","last_peered":"2023-10-16T12:50:51.993694+0000","last_clean":"2023-10-16T12:50:51.993694+0000","last_became_active":"2023-10-16T12:50:50.946024+0000","last_became_peered":"2023-10-16T12:50:50.946024+0000","last_unstale":"2023-10-16T12:50:51.993694+0000","last_undegraded":"2023-10-16T12:50:51.993694+0000","last_fullsized":"2023-10-16T12:50:51.993694+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:23.489241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991133+0000","last_change":"2023-10-16T12:50:51.982523+0000","last_active":"2023-10-16T12:50:51.991133+0000","last_peered":"2023-10-16T12:50:51.991133+0000","last_clean":"2023-10-16T12:50:51.991133+0000","last_became_active":"2023-10-16T12:50:51.982164+0000","last_became_peered":"2023-10-16T12:50:51.982164+0000","last_unstale":"2023-10-16T12:50:51.991133+0000","last_undegraded":"2023-10-16T12:50:51.991133+0000","last_fullsized":"2023-10-16T12:50:51.991133+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:23.578588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997882+0000","last_change":"2023-10-16T12:50:50.959919+0000","last_active":"2023-10-16T12:50:51.997882+0000","last_peered":"2023-10-16T12:50:51.997882+0000","last_clean":"2023-10-16T12:50:51.997882+0000","last_became_active":"2023-10-16T12:50:50.959625+0000","last_became_peered":"2023-10-16T12:50:50.959625+0000","last_unstale":"2023-10-16T12:50:51.997882+0000","last_undegraded":"2023-10-16T12:50:51.997882+0000","last_fullsized":"2023-10-16T12:50:51.997882+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:05.335588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988363+0000","last_change":"2023-10-16T12:50:51.984319+0000","last_active":"2023-10-16T12:50:51.988363+0000","last_peered":"2023-10-16T12:50:51.988363+0000","last_clean":"2023-10-16T12:50:51.988363+0000","last_became_active":"2023-10-16T12:50:51.982993+0000","last_became_peered":"2023-10-16T12:50:51.982993+0000","last_unstale":"2023-10-16T12:50:51.988363+0000","last_undegraded":"2023-10-16T12:50:51.988363+0000","last_fullsized":"2023-10-16T12:50:51.988363+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:00.073808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997592+0000","last_change":"2023-10-16T12:50:50.956160+0000","last_active":"2023-10-16T12:50:51.997592+0000","last_peered":"2023-10-16T12:50:51.997592+0000","last_clean":"2023-10-16T12:50:51.997592+0000","last_became_active":"2023-10-16T12:50:50.955840+0000","last_became_peered":"2023-10-16T12:50:50.955840+0000","last_unstale":"2023-10-16T12:50:51.997592+0000","last_undegraded":"2023-10-16T12:50:51.997592+0000","last_fullsized":"2023-10-16T12:50:51.997592+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:58.285331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.996940+0000","last_change":"2023-10-16T12:50:51.993958+0000","last_active":"2023-10-16T12:50:51.996940+0000","last_peered":"2023-10-16T12:50:51.996940+0000","last_clean":"2023-10-16T12:50:51.996940+0000","last_became_active":"2023-10-16T12:50:51.993303+0000","last_became_peered":"2023-10-16T12:50:51.993303+0000","last_unstale":"2023-10-16T12:50:51.996940+0000","last_undegraded":"2023-10-16T12:50:51.996940+0000","last_fullsized":"2023-10-16T12:50:51.996940+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:53.675341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.999876+0000","last_change":"2023-10-16T12:50:50.946280+0000","last_active":"2023-10-16T12:50:51.999876+0000","last_peered":"2023-10-16T12:50:51.999876+0000","last_clean":"2023-10-16T12:50:51.999876+0000","last_became_active":"2023-10-16T12:50:50.945949+0000","last_became_peered":"2023-10-16T12:50:50.945949+0000","last_unstale":"2023-10-16T12:50:51.999876+0000","last_undegraded":"2023-10-16T12:50:51.999876+0000","last_fullsized":"2023-10-16T12:50:51.999876+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:30.429826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.981595+0000","last_change":"2023-10-16T12:50:51.980675+0000","last_active":"2023-10-16T12:50:51.981595+0000","last_peered":"2023-10-16T12:50:51.981595+0000","last_clean":"2023-10-16T12:50:51.981595+0000","last_became_active":"2023-10-16T12:50:51.980387+0000","last_became_peered":"2023-10-16T12:50:51.980387+0000","last_unstale":"2023-10-16T12:50:51.981595+0000","last_undegraded":"2023-10-16T12:50:51.981595+0000","last_fullsized":"2023-10-16T12:50:51.981595+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:51.379827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993750+0000","last_change":"2023-10-16T12:50:50.946234+0000","last_active":"2023-10-16T12:50:51.993750+0000","last_peered":"2023-10-16T12:50:51.993750+0000","last_clean":"2023-10-16T12:50:51.993750+0000","last_became_active":"2023-10-16T12:50:50.945802+0000","last_became_peered":"2023-10-16T12:50:50.945802+0000","last_unstale":"2023-10-16T12:50:51.993750+0000","last_undegraded":"2023-10-16T12:50:51.993750+0000","last_fullsized":"2023-10-16T12:50:51.993750+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:58:48.628407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978858+0000","last_change":"2023-10-16T12:50:51.978129+0000","last_active":"2023-10-16T12:50:51.978858+0000","last_peered":"2023-10-16T12:50:51.978858+0000","last_clean":"2023-10-16T12:50:51.978858+0000","last_became_active":"2023-10-16T12:50:51.977327+0000","last_became_peered":"2023-10-16T12:50:51.977327+0000","last_unstale":"2023-10-16T12:50:51.978858+0000","last_undegraded":"2023-10-16T12:50:51.978858+0000","last_fullsized":"2023-10-16T12:50:51.978858+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:56.284790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.968463+0000","last_change":"2023-10-16T12:50:50.954237+0000","last_active":"2023-10-16T12:50:51.968463+0000","last_peered":"2023-10-16T12:50:51.968463+0000","last_clean":"2023-10-16T12:50:51.968463+0000","last_became_active":"2023-10-16T12:50:50.953891+0000","last_became_peered":"2023-10-16T12:50:50.953891+0000","last_unstale":"2023-10-16T12:50:51.968463+0000","last_undegraded":"2023-10-16T12:50:51.968463+0000","last_fullsized":"2023-10-16T12:50:51.968463+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:41.505644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.980652+0000","last_change":"2023-10-16T12:50:51.978767+0000","last_active":"2023-10-16T12:50:51.980652+0000","last_peered":"2023-10-16T12:50:51.980652+0000","last_clean":"2023-10-16T12:50:51.980652+0000","last_became_active":"2023-10-16T12:50:51.978431+0000","last_became_peered":"2023-10-16T12:50:51.978431+0000","last_unstale":"2023-10-16T12:50:51.980652+0000","last_undegraded":"2023-10-16T12:50:51.980652+0000","last_fullsized":"2023-10-16T12:50:51.980652+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:11.792351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.957001+0000","last_change":"2023-10-16T12:50:50.953976+0000","last_active":"2023-10-16T12:50:51.957001+0000","last_peered":"2023-10-16T12:50:51.957001+0000","last_clean":"2023-10-16T12:50:51.957001+0000","last_became_active":"2023-10-16T12:50:50.953686+0000","last_became_peered":"2023-10-16T12:50:50.953686+0000","last_unstale":"2023-10-16T12:50:51.957001+0000","last_undegraded":"2023-10-16T12:50:51.957001+0000","last_fullsized":"2023-10-16T12:50:51.957001+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:46.748740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978668+0000","last_change":"2023-10-16T12:50:51.974620+0000","last_active":"2023-10-16T12:50:51.978668+0000","last_peered":"2023-10-16T12:50:51.978668+0000","last_clean":"2023-10-16T12:50:51.978668+0000","last_became_active":"2023-10-16T12:50:51.974331+0000","last_became_peered":"2023-10-16T12:50:51.974331+0000","last_unstale":"2023-10-16T12:50:51.978668+0000","last_undegraded":"2023-10-16T12:50:51.978668+0000","last_fullsized":"2023-10-16T12:50:51.978668+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:43.451631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972935+0000","last_change":"2023-10-16T12:50:50.958264+0000","last_active":"2023-10-16T12:50:51.972935+0000","last_peered":"2023-10-16T12:50:51.972935+0000","last_clean":"2023-10-16T12:50:51.972935+0000","last_became_active":"2023-10-16T12:50:50.957548+0000","last_became_peered":"2023-10-16T12:50:50.957548+0000","last_unstale":"2023-10-16T12:50:51.972935+0000","last_undegraded":"2023-10-16T12:50:51.972935+0000","last_fullsized":"2023-10-16T12:50:51.972935+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:39.969803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.982586+0000","last_change":"2023-10-16T12:50:51.980928+0000","last_active":"2023-10-16T12:50:51.982586+0000","last_peered":"2023-10-16T12:50:51.982586+0000","last_clean":"2023-10-16T12:50:51.982586+0000","last_became_active":"2023-10-16T12:50:51.980630+0000","last_became_peered":"2023-10-16T12:50:51.980630+0000","last_unstale":"2023-10-16T12:50:51.982586+0000","last_undegraded":"2023-10-16T12:50:51.982586+0000","last_fullsized":"2023-10-16T12:50:51.982586+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:21:03.273600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972242+0000","last_change":"2023-10-16T12:50:50.956801+0000","last_active":"2023-10-16T12:50:51.972242+0000","last_peered":"2023-10-16T12:50:51.972242+0000","last_clean":"2023-10-16T12:50:51.972242+0000","last_became_active":"2023-10-16T12:50:50.954920+0000","last_became_peered":"2023-10-16T12:50:50.954920+0000","last_unstale":"2023-10-16T12:50:51.972242+0000","last_undegraded":"2023-10-16T12:50:51.972242+0000","last_fullsized":"2023-10-16T12:50:51.972242+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:42:27.027931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978062+0000","last_change":"2023-10-16T12:50:51.974468+0000","last_active":"2023-10-16T12:50:51.978062+0000","last_peered":"2023-10-16T12:50:51.978062+0000","last_clean":"2023-10-16T12:50:51.978062+0000","last_became_active":"2023-10-16T12:50:51.974154+0000","last_became_peered":"2023-10-16T12:50:51.974154+0000","last_unstale":"2023-10-16T12:50:51.978062+0000","last_undegraded":"2023-10-16T12:50:51.978062+0000","last_fullsized":"2023-10-16T12:50:51.978062+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:50.612126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"112'3","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:52.000677+0000","last_change":"2023-10-16T12:50:50.955048+0000","last_active":"2023-10-16T12:50:52.000677+0000","last_peered":"2023-10-16T12:50:52.000677+0000","last_clean":"2023-10-16T12:50:52.000677+0000","last_became_active":"2023-10-16T12:50:50.954649+0000","last_became_peered":"2023-10-16T12:50:50.954649+0000","last_unstale":"2023-10-16T12:50:52.000677+0000","last_undegraded":"2023-10-16T12:50:52.000677+0000","last_fullsized":"2023-10-16T12:50:52.000677+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:52.226837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988480+0000","last_change":"2023-10-16T12:50:51.985106+0000","last_active":"2023-10-16T12:50:51.988480+0000","last_peered":"2023-10-16T12:50:51.988480+0000","last_clean":"2023-10-16T12:50:51.988480+0000","last_became_active":"2023-10-16T12:50:51.984822+0000","last_became_peered":"2023-10-16T12:50:51.984822+0000","last_unstale":"2023-10-16T12:50:51.988480+0000","last_undegraded":"2023-10-16T12:50:51.988480+0000","last_fullsized":"2023-10-16T12:50:51.988480+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:10.767414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:52.002099+0000","last_change":"2023-10-16T12:50:50.960112+0000","last_active":"2023-10-16T12:50:52.002099+0000","last_peered":"2023-10-16T12:50:52.002099+0000","last_clean":"2023-10-16T12:50:52.002099+0000","last_became_active":"2023-10-16T12:50:50.959802+0000","last_became_peered":"2023-10-16T12:50:50.959802+0000","last_unstale":"2023-10-16T12:50:52.002099+0000","last_undegraded":"2023-10-16T12:50:52.002099+0000","last_fullsized":"2023-10-16T12:50:52.002099+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:49.936794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.984796+0000","last_change":"2023-10-16T12:50:51.981691+0000","last_active":"2023-10-16T12:50:51.984796+0000","last_peered":"2023-10-16T12:50:51.984796+0000","last_clean":"2023-10-16T12:50:51.984796+0000","last_became_active":"2023-10-16T12:50:51.981368+0000","last_became_peered":"2023-10-16T12:50:51.981368+0000","last_unstale":"2023-10-16T12:50:51.984796+0000","last_undegraded":"2023-10-16T12:50:51.984796+0000","last_fullsized":"2023-10-16T12:50:51.984796+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:08.300310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:52.000344+0000","last_change":"2023-10-16T12:50:50.957084+0000","last_active":"2023-10-16T12:50:52.000344+0000","last_peered":"2023-10-16T12:50:52.000344+0000","last_clean":"2023-10-16T12:50:52.000344+0000","last_became_active":"2023-10-16T12:50:50.952131+0000","last_became_peered":"2023-10-16T12:50:50.952131+0000","last_unstale":"2023-10-16T12:50:52.000344+0000","last_undegraded":"2023-10-16T12:50:52.000344+0000","last_fullsized":"2023-10-16T12:50:52.000344+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:33:41.349606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.987620+0000","last_change":"2023-10-16T12:50:51.986007+0000","last_active":"2023-10-16T12:50:51.987620+0000","last_peered":"2023-10-16T12:50:51.987620+0000","last_clean":"2023-10-16T12:50:51.987620+0000","last_became_active":"2023-10-16T12:50:51.984975+0000","last_became_peered":"2023-10-16T12:50:51.984975+0000","last_unstale":"2023-10-16T12:50:51.987620+0000","last_undegraded":"2023-10-16T12:50:51.987620+0000","last_fullsized":"2023-10-16T12:50:51.987620+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:43.451631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.947351+0000","last_change":"2023-10-16T12:50:50.952961+0000","last_active":"2023-10-16T12:50:51.947351+0000","last_peered":"2023-10-16T12:50:51.947351+0000","last_clean":"2023-10-16T12:50:51.947351+0000","last_became_active":"2023-10-16T12:50:50.952677+0000","last_became_peered":"2023-10-16T12:50:50.952677+0000","last_unstale":"2023-10-16T12:50:51.947351+0000","last_undegraded":"2023-10-16T12:50:51.947351+0000","last_fullsized":"2023-10-16T12:50:51.947351+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:55.273712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997007+0000","last_change":"2023-10-16T12:50:51.992907+0000","last_active":"2023-10-16T12:50:51.997007+0000","last_peered":"2023-10-16T12:50:51.997007+0000","last_clean":"2023-10-16T12:50:51.997007+0000","last_became_active":"2023-10-16T12:50:51.992610+0000","last_became_peered":"2023-10-16T12:50:51.992610+0000","last_unstale":"2023-10-16T12:50:51.997007+0000","last_undegraded":"2023-10-16T12:50:51.997007+0000","last_fullsized":"2023-10-16T12:50:51.997007+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:06.576628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.956767+0000","last_change":"2023-10-16T12:50:50.950308+0000","last_active":"2023-10-16T12:50:51.956767+0000","last_peered":"2023-10-16T12:50:51.956767+0000","last_clean":"2023-10-16T12:50:51.956767+0000","last_became_active":"2023-10-16T12:50:50.948881+0000","last_became_peered":"2023-10-16T12:50:50.948881+0000","last_unstale":"2023-10-16T12:50:51.956767+0000","last_undegraded":"2023-10-16T12:50:51.956767+0000","last_fullsized":"2023-10-16T12:50:51.956767+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:38.167448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.983958+0000","last_change":"2023-10-16T12:50:51.978726+0000","last_active":"2023-10-16T12:50:51.983958+0000","last_peered":"2023-10-16T12:50:51.983958+0000","last_clean":"2023-10-16T12:50:51.983958+0000","last_became_active":"2023-10-16T12:50:51.976811+0000","last_became_peered":"2023-10-16T12:50:51.976811+0000","last_unstale":"2023-10-16T12:50:51.983958+0000","last_undegraded":"2023-10-16T12:50:51.983958+0000","last_fullsized":"2023-10-16T12:50:51.983958+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:04.923152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"112'8","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.995215+0000","last_change":"2023-10-16T12:50:50.956360+0000","last_active":"2023-10-16T12:50:51.995215+0000","last_peered":"2023-10-16T12:50:51.995215+0000","last_clean":"2023-10-16T12:50:51.995215+0000","last_became_active":"2023-10-16T12:50:50.955503+0000","last_became_peered":"2023-10-16T12:50:50.955503+0000","last_unstale":"2023-10-16T12:50:51.995215+0000","last_undegraded":"2023-10-16T12:50:51.995215+0000","last_fullsized":"2023-10-16T12:50:51.995215+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:11.038097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.977890+0000","last_change":"2023-10-16T12:50:51.976699+0000","last_active":"2023-10-16T12:50:51.977890+0000","last_peered":"2023-10-16T12:50:51.977890+0000","last_clean":"2023-10-16T12:50:51.977890+0000","last_became_active":"2023-10-16T12:50:51.976428+0000","last_became_peered":"2023-10-16T12:50:51.976428+0000","last_unstale":"2023-10-16T12:50:51.977890+0000","last_undegraded":"2023-10-16T12:50:51.977890+0000","last_fullsized":"2023-10-16T12:50:51.977890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:11.792351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992614+0000","last_change":"2023-10-16T12:50:50.956994+0000","last_active":"2023-10-16T12:50:51.992614+0000","last_peered":"2023-10-16T12:50:51.992614+0000","last_clean":"2023-10-16T12:50:51.992614+0000","last_became_active":"2023-10-16T12:50:50.956697+0000","last_became_peered":"2023-10-16T12:50:50.956697+0000","last_unstale":"2023-10-16T12:50:51.992614+0000","last_undegraded":"2023-10-16T12:50:51.992614+0000","last_fullsized":"2023-10-16T12:50:51.992614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:25.101451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991316+0000","last_change":"2023-10-16T12:50:51.982721+0000","last_active":"2023-10-16T12:50:51.991316+0000","last_peered":"2023-10-16T12:50:51.991316+0000","last_clean":"2023-10-16T12:50:51.991316+0000","last_became_active":"2023-10-16T12:50:51.982362+0000","last_became_peered":"2023-10-16T12:50:51.982362+0000","last_unstale":"2023-10-16T12:50:51.991316+0000","last_undegraded":"2023-10-16T12:50:51.991316+0000","last_fullsized":"2023-10-16T12:50:51.991316+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:47.455433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992120+0000","last_change":"2023-10-16T12:50:50.953349+0000","last_active":"2023-10-16T12:50:51.992120+0000","last_peered":"2023-10-16T12:50:51.992120+0000","last_clean":"2023-10-16T12:50:51.992120+0000","last_became_active":"2023-10-16T12:50:50.952853+0000","last_became_peered":"2023-10-16T12:50:50.952853+0000","last_unstale":"2023-10-16T12:50:51.992120+0000","last_undegraded":"2023-10-16T12:50:51.992120+0000","last_fullsized":"2023-10-16T12:50:51.992120+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:31:22.216972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.990581+0000","last_change":"2023-10-16T12:50:51.989278+0000","last_active":"2023-10-16T12:50:51.990581+0000","last_peered":"2023-10-16T12:50:51.990581+0000","last_clean":"2023-10-16T12:50:51.990581+0000","last_became_active":"2023-10-16T12:50:51.988832+0000","last_became_peered":"2023-10-16T12:50:51.988832+0000","last_unstale":"2023-10-16T12:50:51.990581+0000","last_undegraded":"2023-10-16T12:50:51.990581+0000","last_fullsized":"2023-10-16T12:50:51.990581+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:03.959532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972095+0000","last_change":"2023-10-16T12:50:50.952898+0000","last_active":"2023-10-16T12:50:51.972095+0000","last_peered":"2023-10-16T12:50:51.972095+0000","last_clean":"2023-10-16T12:50:51.972095+0000","last_became_active":"2023-10-16T12:50:50.952613+0000","last_became_peered":"2023-10-16T12:50:50.952613+0000","last_unstale":"2023-10-16T12:50:51.972095+0000","last_undegraded":"2023-10-16T12:50:51.972095+0000","last_fullsized":"2023-10-16T12:50:51.972095+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:17.475548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.997051+0000","last_change":"2023-10-16T12:50:51.992330+0000","last_active":"2023-10-16T12:50:51.997051+0000","last_peered":"2023-10-16T12:50:51.997051+0000","last_clean":"2023-10-16T12:50:51.997051+0000","last_became_active":"2023-10-16T12:50:51.988283+0000","last_became_peered":"2023-10-16T12:50:51.988283+0000","last_unstale":"2023-10-16T12:50:51.997051+0000","last_undegraded":"2023-10-16T12:50:51.997051+0000","last_fullsized":"2023-10-16T12:50:51.997051+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:23.438935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.998703+0000","last_change":"2023-10-16T12:50:50.949855+0000","last_active":"2023-10-16T12:50:51.998703+0000","last_peered":"2023-10-16T12:50:51.998703+0000","last_clean":"2023-10-16T12:50:51.998703+0000","last_became_active":"2023-10-16T12:50:50.949584+0000","last_became_peered":"2023-10-16T12:50:50.949584+0000","last_unstale":"2023-10-16T12:50:51.998703+0000","last_undegraded":"2023-10-16T12:50:51.998703+0000","last_fullsized":"2023-10-16T12:50:51.998703+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:27.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972257+0000","last_change":"2023-10-16T12:50:51.968059+0000","last_active":"2023-10-16T12:50:51.972257+0000","last_peered":"2023-10-16T12:50:51.972257+0000","last_clean":"2023-10-16T12:50:51.972257+0000","last_became_active":"2023-10-16T12:50:51.967788+0000","last_became_peered":"2023-10-16T12:50:51.967788+0000","last_unstale":"2023-10-16T12:50:51.972257+0000","last_undegraded":"2023-10-16T12:50:51.972257+0000","last_fullsized":"2023-10-16T12:50:51.972257+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:28.708339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.975575+0000","last_change":"2023-10-16T12:50:50.948020+0000","last_active":"2023-10-16T12:50:51.975575+0000","last_peered":"2023-10-16T12:50:51.975575+0000","last_clean":"2023-10-16T12:50:51.975575+0000","last_became_active":"2023-10-16T12:50:50.947523+0000","last_became_peered":"2023-10-16T12:50:50.947523+0000","last_unstale":"2023-10-16T12:50:51.975575+0000","last_undegraded":"2023-10-16T12:50:51.975575+0000","last_fullsized":"2023-10-16T12:50:51.975575+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:51.877118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.985996+0000","last_change":"2023-10-16T12:50:51.983170+0000","last_active":"2023-10-16T12:50:51.985996+0000","last_peered":"2023-10-16T12:50:51.985996+0000","last_clean":"2023-10-16T12:50:51.985996+0000","last_became_active":"2023-10-16T12:50:51.978796+0000","last_became_peered":"2023-10-16T12:50:51.978796+0000","last_unstale":"2023-10-16T12:50:51.985996+0000","last_undegraded":"2023-10-16T12:50:51.985996+0000","last_fullsized":"2023-10-16T12:50:51.985996+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:18.486626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.966573+0000","last_change":"2023-10-16T12:50:50.956668+0000","last_active":"2023-10-16T12:50:51.966573+0000","last_peered":"2023-10-16T12:50:51.966573+0000","last_clean":"2023-10-16T12:50:51.966573+0000","last_became_active":"2023-10-16T12:50:50.956392+0000","last_became_peered":"2023-10-16T12:50:50.956392+0000","last_unstale":"2023-10-16T12:50:51.966573+0000","last_undegraded":"2023-10-16T12:50:51.966573+0000","last_fullsized":"2023-10-16T12:50:51.966573+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:58.285331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978279+0000","last_change":"2023-10-16T12:50:51.976802+0000","last_active":"2023-10-16T12:50:51.978279+0000","last_peered":"2023-10-16T12:50:51.978279+0000","last_clean":"2023-10-16T12:50:51.978279+0000","last_became_active":"2023-10-16T12:50:51.975837+0000","last_became_peered":"2023-10-16T12:50:51.975837+0000","last_unstale":"2023-10-16T12:50:51.978279+0000","last_undegraded":"2023-10-16T12:50:51.978279+0000","last_fullsized":"2023-10-16T12:50:51.978279+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:22.570835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.946865+0000","last_change":"2023-10-16T12:50:50.949104+0000","last_active":"2023-10-16T12:50:51.946865+0000","last_peered":"2023-10-16T12:50:51.946865+0000","last_clean":"2023-10-16T12:50:51.946865+0000","last_became_active":"2023-10-16T12:50:50.948829+0000","last_became_peered":"2023-10-16T12:50:50.948829+0000","last_unstale":"2023-10-16T12:50:51.946865+0000","last_undegraded":"2023-10-16T12:50:51.946865+0000","last_fullsized":"2023-10-16T12:50:51.946865+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:30.523367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.976497+0000","last_change":"2023-10-16T12:50:51.970928+0000","last_active":"2023-10-16T12:50:51.976497+0000","last_peered":"2023-10-16T12:50:51.976497+0000","last_clean":"2023-10-16T12:50:51.976497+0000","last_became_active":"2023-10-16T12:50:51.970641+0000","last_became_peered":"2023-10-16T12:50:51.970641+0000","last_unstale":"2023-10-16T12:50:51.976497+0000","last_undegraded":"2023-10-16T12:50:51.976497+0000","last_fullsized":"2023-10-16T12:50:51.976497+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:13.918437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991052+0000","last_change":"2023-10-16T12:50:50.951295+0000","last_active":"2023-10-16T12:50:51.991052+0000","last_peered":"2023-10-16T12:50:51.991052+0000","last_clean":"2023-10-16T12:50:51.991052+0000","last_became_active":"2023-10-16T12:50:50.950985+0000","last_became_peered":"2023-10-16T12:50:50.950985+0000","last_unstale":"2023-10-16T12:50:51.991052+0000","last_undegraded":"2023-10-16T12:50:51.991052+0000","last_fullsized":"2023-10-16T12:50:51.991052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:37.970636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.993705+0000","last_change":"2023-10-16T12:50:51.992003+0000","last_active":"2023-10-16T12:50:51.993705+0000","last_peered":"2023-10-16T12:50:51.993705+0000","last_clean":"2023-10-16T12:50:51.993705+0000","last_became_active":"2023-10-16T12:50:51.991691+0000","last_became_peered":"2023-10-16T12:50:51.991691+0000","last_unstale":"2023-10-16T12:50:51.993705+0000","last_undegraded":"2023-10-16T12:50:51.993705+0000","last_fullsized":"2023-10-16T12:50:51.993705+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:34.671005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.964808+0000","last_change":"2023-10-16T12:50:50.948373+0000","last_active":"2023-10-16T12:50:51.964808+0000","last_peered":"2023-10-16T12:50:51.964808+0000","last_clean":"2023-10-16T12:50:51.964808+0000","last_became_active":"2023-10-16T12:50:50.948060+0000","last_became_peered":"2023-10-16T12:50:50.948060+0000","last_unstale":"2023-10-16T12:50:51.964808+0000","last_undegraded":"2023-10-16T12:50:51.964808+0000","last_fullsized":"2023-10-16T12:50:51.964808+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:12.644977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992862+0000","last_change":"2023-10-16T12:50:51.985400+0000","last_active":"2023-10-16T12:50:51.992862+0000","last_peered":"2023-10-16T12:50:51.992862+0000","last_clean":"2023-10-16T12:50:51.992862+0000","last_became_active":"2023-10-16T12:50:51.985122+0000","last_became_peered":"2023-10-16T12:50:51.985122+0000","last_unstale":"2023-10-16T12:50:51.992862+0000","last_undegraded":"2023-10-16T12:50:51.992862+0000","last_fullsized":"2023-10-16T12:50:51.992862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:34.945007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972003+0000","last_change":"2023-10-16T12:50:50.961570+0000","last_active":"2023-10-16T12:50:51.972003+0000","last_peered":"2023-10-16T12:50:51.972003+0000","last_clean":"2023-10-16T12:50:51.972003+0000","last_became_active":"2023-10-16T12:50:50.961278+0000","last_became_peered":"2023-10-16T12:50:50.961278+0000","last_unstale":"2023-10-16T12:50:51.972003+0000","last_undegraded":"2023-10-16T12:50:51.972003+0000","last_fullsized":"2023-10-16T12:50:51.972003+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:20.269854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.978124+0000","last_change":"2023-10-16T12:50:51.975598+0000","last_active":"2023-10-16T12:50:51.978124+0000","last_peered":"2023-10-16T12:50:51.978124+0000","last_clean":"2023-10-16T12:50:51.978124+0000","last_became_active":"2023-10-16T12:50:51.974570+0000","last_became_peered":"2023-10-16T12:50:51.974570+0000","last_unstale":"2023-10-16T12:50:51.978124+0000","last_undegraded":"2023-10-16T12:50:51.978124+0000","last_fullsized":"2023-10-16T12:50:51.978124+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:32.342448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.971849+0000","last_change":"2023-10-16T12:50:50.959558+0000","last_active":"2023-10-16T12:50:51.971849+0000","last_peered":"2023-10-16T12:50:51.971849+0000","last_clean":"2023-10-16T12:50:51.971849+0000","last_became_active":"2023-10-16T12:50:50.959226+0000","last_became_peered":"2023-10-16T12:50:50.959226+0000","last_unstale":"2023-10-16T12:50:51.971849+0000","last_undegraded":"2023-10-16T12:50:51.971849+0000","last_fullsized":"2023-10-16T12:50:51.971849+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:24.521173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972584+0000","last_change":"2023-10-16T12:50:51.966258+0000","last_active":"2023-10-16T12:50:51.972584+0000","last_peered":"2023-10-16T12:50:51.972584+0000","last_clean":"2023-10-16T12:50:51.972584+0000","last_became_active":"2023-10-16T12:50:51.965482+0000","last_became_peered":"2023-10-16T12:50:51.965482+0000","last_unstale":"2023-10-16T12:50:51.972584+0000","last_undegraded":"2023-10-16T12:50:51.972584+0000","last_fullsized":"2023-10-16T12:50:51.972584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:02.458758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.970232+0000","last_change":"2023-10-16T12:50:50.951548+0000","last_active":"2023-10-16T12:50:51.970232+0000","last_peered":"2023-10-16T12:50:51.970232+0000","last_clean":"2023-10-16T12:50:51.970232+0000","last_became_active":"2023-10-16T12:50:50.951276+0000","last_became_peered":"2023-10-16T12:50:50.951276+0000","last_unstale":"2023-10-16T12:50:51.970232+0000","last_undegraded":"2023-10-16T12:50:51.970232+0000","last_fullsized":"2023-10-16T12:50:51.970232+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:41.280385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.991409+0000","last_change":"2023-10-16T12:50:51.982927+0000","last_active":"2023-10-16T12:50:51.991409+0000","last_peered":"2023-10-16T12:50:51.991409+0000","last_clean":"2023-10-16T12:50:51.991409+0000","last_became_active":"2023-10-16T12:50:51.982582+0000","last_became_peered":"2023-10-16T12:50:51.982582+0000","last_unstale":"2023-10-16T12:50:51.991409+0000","last_undegraded":"2023-10-16T12:50:51.991409+0000","last_fullsized":"2023-10-16T12:50:51.991409+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:08.422266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.959082+0000","last_change":"2023-10-16T12:50:50.946010+0000","last_active":"2023-10-16T12:50:51.959082+0000","last_peered":"2023-10-16T12:50:51.959082+0000","last_clean":"2023-10-16T12:50:51.959082+0000","last_became_active":"2023-10-16T12:50:50.945492+0000","last_became_peered":"2023-10-16T12:50:50.945492+0000","last_unstale":"2023-10-16T12:50:51.959082+0000","last_undegraded":"2023-10-16T12:50:51.959082+0000","last_fullsized":"2023-10-16T12:50:51.959082+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:16.960765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.994440+0000","last_change":"2023-10-16T12:50:51.993708+0000","last_active":"2023-10-16T12:50:51.994440+0000","last_peered":"2023-10-16T12:50:51.994440+0000","last_clean":"2023-10-16T12:50:51.994440+0000","last_became_active":"2023-10-16T12:50:51.993293+0000","last_became_peered":"2023-10-16T12:50:51.993293+0000","last_unstale":"2023-10-16T12:50:51.994440+0000","last_undegraded":"2023-10-16T12:50:51.994440+0000","last_fullsized":"2023-10-16T12:50:51.994440+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:39.020851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.972259+0000","last_change":"2023-10-16T12:50:50.955193+0000","last_active":"2023-10-16T12:50:51.972259+0000","last_peered":"2023-10-16T12:50:51.972259+0000","last_clean":"2023-10-16T12:50:51.972259+0000","last_became_active":"2023-10-16T12:50:50.954789+0000","last_became_peered":"2023-10-16T12:50:50.954789+0000","last_unstale":"2023-10-16T12:50:51.972259+0000","last_undegraded":"2023-10-16T12:50:51.972259+0000","last_fullsized":"2023-10-16T12:50:51.972259+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:08.300625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.992945+0000","last_change":"2023-10-16T12:50:51.988746+0000","last_active":"2023-10-16T12:50:51.992945+0000","last_peered":"2023-10-16T12:50:51.992945+0000","last_clean":"2023-10-16T12:50:51.992945+0000","last_became_active":"2023-10-16T12:50:51.988266+0000","last_became_peered":"2023-10-16T12:50:51.988266+0000","last_unstale":"2023-10-16T12:50:51.992945+0000","last_undegraded":"2023-10-16T12:50:51.992945+0000","last_fullsized":"2023-10-16T12:50:51.992945+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:09.389663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.961029+0000","last_change":"2023-10-16T12:50:50.952335+0000","last_active":"2023-10-16T12:50:51.961029+0000","last_peered":"2023-10-16T12:50:51.961029+0000","last_clean":"2023-10-16T12:50:51.961029+0000","last_became_active":"2023-10-16T12:50:50.951117+0000","last_became_peered":"2023-10-16T12:50:50.951117+0000","last_unstale":"2023-10-16T12:50:51.961029+0000","last_undegraded":"2023-10-16T12:50:51.961029+0000","last_fullsized":"2023-10-16T12:50:51.961029+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:27.665346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.990717+0000","last_change":"2023-10-16T12:50:51.982322+0000","last_active":"2023-10-16T12:50:51.990717+0000","last_peered":"2023-10-16T12:50:51.990717+0000","last_clean":"2023-10-16T12:50:51.990717+0000","last_became_active":"2023-10-16T12:50:51.982000+0000","last_became_peered":"2023-10-16T12:50:51.982000+0000","last_unstale":"2023-10-16T12:50:51.990717+0000","last_undegraded":"2023-10-16T12:50:51.990717+0000","last_fullsized":"2023-10-16T12:50:51.990717+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:50.910840+0000","last_clean_scrub_stamp":"2023-10-16T12:50:50.910840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:41.210218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:50:51.988304+0000","last_change":"2023-10-16T12:50:50.945112+0000","last_active":"2023-10-16T12:50:51.988304+0000","last_peered":"2023-10-16T12:50:51.988304+0000","last_clean":"2023-10-16T12:50:51.988304+0000","last_became_active":"2023-10-16T12:50:50.944779+0000","last_became_peered":"2023-10-16T12:50:50.944779+0000","last_unstale":"2023-10-16T12:50:51.988304+0000","last_undegraded":"2023-10-16T12:50:51.988304+0000","last_fullsized":"2023-10-16T12:50:51.988304+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:50:49.899762+0000","last_clean_scrub_stamp":"2023-10-16T12:50:49.899762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:52.614970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'186","reported_seq":497,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:50:50.933348+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:50:50.933348+0000","last_peered":"2023-10-16T12:50:50.933348+0000","last_clean":"2023-10-16T12:50:50.933348+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:50:50.933348+0000","last_undegraded":"2023-10-16T12:50:50.933348+0000","last_fullsized":"2023-10-16T12:50:50.933348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":715,"num_read_kb":1418,"num_write":283,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542353,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1970176,"data_stored":1244618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2490000000000001,"15min":1.2949999999999999},"min":{"1min":0.86499999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":2.1240000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3100000000000001,"15min":1.3480000000000001},"min":{"1min":0.95599999999999996,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.8899999999999999},"last":1.0289999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1779999999999999,"15min":1.2090000000000001},"min":{"1min":0.71799999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":19.158999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1519999999999999,"15min":1.208},"min":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.413,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":5.5490000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.147,"15min":1.196},"min":{"1min":0.77500000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.506,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":2.855},{"interface":"front","average":{"1min":1.276,"5min":1.284,"15min":1.323},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6200000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.077}]},{"osd":3,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3,"15min":1.2989999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.706,"5min":1.847,"15min":1.847},"last":3.2759999999999998},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4019999999999999,"15min":1.403},"min":{"1min":0.96899999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.681,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":2.4369999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.179,"15min":1.1890000000000001},"min":{"1min":0.90000000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5589999999999999,"5min":1.95,"15min":1.95},"last":2.5710000000000002},{"interface":"front","average":{"1min":1.421,"5min":1.3540000000000001,"15min":1.333},"min":{"1min":0.84999999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.337,"15min":1.331},"min":{"1min":1.141,"5min":0.624,"15min":0.624},"max":{"1min":1.8700000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":5.0140000000000002},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5089999999999999,"15min":1.4690000000000001},"min":{"1min":1.0009999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9450000000000001,"5min":2.411,"15min":2.411},"last":2.1720000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.593,"15min":1.5640000000000001},"min":{"1min":1.05,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.0760000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5009999999999999,"15min":1.45},"min":{"1min":1.262,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.97,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":2.3170000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1384448,"data_stored":654874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.214,"15min":1.28},"min":{"1min":0.73599999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.952,"5min":1.952,"15min":3.7450000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2749999999999999,"15min":1.3340000000000001},"min":{"1min":0.77100000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6279999999999999,"5min":2.0270000000000001,"15min":3.819},"last":1.4119999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1850000000000001,"15min":1.1930000000000001},"min":{"1min":0.70699999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.8979999999999999},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.218,"15min":1.21},"min":{"1min":0.74399999999999999,"5min":0.625,"15min":0.625},"max":{"1min":1.611,"5min":1.821,"15min":3.8519999999999999},"last":1.3320000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1990000000000001,"15min":1.2190000000000001},"min":{"1min":0.66900000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.6279999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2130000000000001,"15min":1.272},"min":{"1min":0.86399999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.5,"5min":1.829,"15min":3.9009999999999998},"last":1.3720000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2949999999999999,"15min":1.3260000000000001},"min":{"1min":0.85199999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.73,"5min":1.9399999999999999,"15min":4.2380000000000004},"last":1.111},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.298,"15min":1.351},"min":{"1min":0.96299999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.694,"5min":2.012,"15min":4.4269999999999996},"last":1.2709999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.383,"15min":1.3460000000000001},"min":{"1min":0.996,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.798,"5min":2.0830000000000002,"15min":2.3029999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.94799999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.8799999999999999,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.5409999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3,"15min":1.3109999999999999},"min":{"1min":0.86399999999999999,"5min":0.755,"15min":0.755},"max":{"1min":1.9219999999999999,"5min":1.9810000000000001,"15min":2.0910000000000002},"last":1.573},{"interface":"front","average":{"1min":1.482,"5min":1.4670000000000001,"15min":1.4470000000000001},"min":{"1min":1.0720000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7769999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.661}]},{"osd":7,"last update":"Mon Oct 16 12:50:50 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4450000000000001,"15min":1.4430000000000001},"min":{"1min":0.91300000000000003,"5min":0.878,"15min":0.878},"max":{"1min":1.796,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.458,"15min":1.458},"min":{"1min":1.167,"5min":0.98699999999999999,"15min":0.96399999999999997},"max":{"1min":1.984,"5min":2.1230000000000002,"15min":2.2610000000000001},"last":1.514}]}]},{"osd":5,"up_from":14,"seq":60129542355,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.38200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.8480000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.119},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1779999999999999,"15min":1.1819999999999999},"min":{"1min":0.77800000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.645,"5min":1.6819999999999999,"15min":1.7689999999999999},"last":0.69599999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.22,"15min":1.2549999999999999},"min":{"1min":0.69399999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.827},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1819999999999999,"15min":1.226},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.647,"5min":1.708,"15min":1.7889999999999999},"last":1.2869999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.26,"15min":1.2989999999999999},"min":{"1min":0.71599999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.591,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.444},{"interface":"front","average":{"1min":1.155,"5min":1.24,"15min":1.2669999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.554,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.22}]},{"osd":3,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.73099999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.363,"15min":1.3799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.929,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.0649999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1930000000000001,"15min":1.2050000000000001},"min":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7430000000000001,"5min":1.996,"15min":1.996},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.361,"15min":1.3140000000000001},"min":{"1min":0.55000000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.0910000000000002},"last":1.248}]},{"osd":6,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.321,"15min":1.3220000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8,"5min":1.927,"15min":1.927},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.3939999999999999,"15min":1.3799999999999999},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9470000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3620000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:49:47 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4470000000000001,"15min":1.448},"min":{"1min":0.64300000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.3900000000000001},"last":1.556},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5029999999999999},"min":{"1min":0.66900000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.0209999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.524}]}]},{"osd":4,"up_from":14,"seq":60129542355,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58880,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312960,"statfs":{"total":96636764160,"available":96576471040,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1799999999999999,"15min":1.2010000000000001},"min":{"1min":0.65700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4359999999999999,"5min":1.766,"15min":2.2200000000000002},"last":1.264},{"interface":"front","average":{"1min":1.032,"5min":1.1120000000000001,"15min":1.151},"min":{"1min":0.73499999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.569,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.141,"15min":1.1899999999999999},"min":{"1min":0.69299999999999995,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5669999999999999,"5min":1.681,"15min":1.865},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.228,"15min":1.26},"min":{"1min":0.76500000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.532,"5min":1.724,"15min":1.9390000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2549999999999999,"15min":1.258},"min":{"1min":0.82999999999999996,"5min":0.81899999999999995,"15min":0.753},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":1.208},{"interface":"front","average":{"1min":1.24,"5min":1.292,"15min":1.329},"min":{"1min":0.78800000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6379999999999999,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.46}]},{"osd":3,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2569999999999999,"15min":1.278},"min":{"1min":0.91600000000000004,"5min":0.80900000000000005,"15min":0.752},"max":{"1min":1.804,"5min":1.804,"15min":1.8100000000000001},"last":1.155},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.359,"15min":1.381},"min":{"1min":0.89800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.4139999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3420000000000001,"15min":1.3049999999999999},"min":{"1min":1.1699999999999999,"5min":0.86799999999999999,"15min":0.50900000000000001},"max":{"1min":1.917,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.379},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3220000000000001,"15min":1.3009999999999999},"min":{"1min":0.78300000000000003,"5min":0.76400000000000001,"15min":0.53600000000000003},"max":{"1min":1.994,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.5640000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.407,"15min":1.391},"min":{"1min":0.92300000000000004,"5min":0.78500000000000003,"15min":0.57299999999999995},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.329},{"interface":"front","average":{"1min":1.569,"5min":1.488,"15min":1.4830000000000001},"min":{"1min":1.282,"5min":0.90400000000000003,"15min":0.66600000000000004},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.53}]},{"osd":7,"last update":"Mon Oct 16 12:49:53 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.4870000000000001,"15min":1.4790000000000001},"min":{"1min":1.0600000000000001,"5min":0.88600000000000001,"15min":0.60399999999999998},"max":{"1min":2.109,"5min":2.109,"15min":2.254},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.591,"15min":1.5960000000000001},"min":{"1min":1.026,"5min":0.94799999999999995,"15min":0.71299999999999997},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.6519999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542355,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.91700000000000004,"15min":0.95399999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34000000000000002},"max":{"1min":1.5649999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.1180000000000001,"15min":1.169},"min":{"1min":0.44900000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.244,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.2569999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.004,"15min":1.0369999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.337,"5min":1.758,"15min":1.758},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.034,"15min":1.0740000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.45600000000000002},"max":{"1min":1.133,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.95199999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.0229999999999999,"15min":1.0740000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.1639999999999999,"5min":1.7949999999999999,"15min":1.9510000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.2689999999999999,"15min":1.3540000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.359,"5min":1.8600000000000001,"15min":1.921},"last":1.5569999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4530000000000001,"15min":1.484},"min":{"1min":0.85699999999999998,"5min":0.80400000000000005,"15min":0.77600000000000002},"max":{"1min":1.806,"5min":1.956,"15min":1.974},"last":1.526},{"interface":"front","average":{"1min":1.236,"5min":1.391,"15min":1.4119999999999999},"min":{"1min":0.82699999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.607,"5min":1.8939999999999999,"15min":2.0539999999999998},"last":1.7010000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.363,"15min":1.389},"min":{"1min":0.83899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5329999999999999,"5min":2.028,"15min":2.0539999999999998},"last":1.623},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.4219999999999999,"15min":1.4670000000000001},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.80300000000000005},"max":{"1min":1.575,"5min":1.9990000000000001,"15min":2.3540000000000001},"last":1.407}]},{"osd":6,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.4139999999999999,"15min":1.4410000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.881,"5min":2.2989999999999999,"15min":2.3220000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.696,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.3520000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:50:02 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.512,"15min":1.5369999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6850000000000001,"5min":2.1339999999999999,"15min":2.431},"last":1.472},{"interface":"front","average":{"1min":1.381,"5min":1.5740000000000001,"15min":1.603},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.956,"5min":2.552,"15min":2.552},"last":1.5900000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542353,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1376256,"data_stored":654762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.88900000000000001,"15min":0.91800000000000004},"min":{"1min":0.44600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.413,"5min":1.742,"15min":1.7889999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0429999999999999,"15min":1.1040000000000001},"min":{"1min":0.69299999999999995,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.746,"5min":1.746,"15min":2.085},"last":1.091}]},{"osd":1,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.1240000000000001,"15min":1.1279999999999999},"min":{"1min":0.64100000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.677,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.125,"5min":1.129,"15min":1.2210000000000001},"min":{"1min":0.58599999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.8440000000000001,"5min":2.347,"15min":2.4289999999999998},"last":1.224}]},{"osd":3,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.006,"15min":1.079},"min":{"1min":0.498,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5860000000000001,"5min":1.6879999999999999,"15min":1.859},"last":1.016},{"interface":"front","average":{"1min":1.212,"5min":1.264,"15min":1.3140000000000001},"min":{"1min":0.52800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.972,"5min":2.1000000000000001,"15min":2.1970000000000001},"last":1.4870000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4099999999999999,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.2149999999999999,"5min":2.431,"15min":2.431},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.369,"15min":1.4450000000000001},"min":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.9430000000000001,"5min":2.0670000000000002,"15min":2.2530000000000001},"last":1.454}]},{"osd":5,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4350000000000001,"15min":1.502},"min":{"1min":0.99099999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.0390000000000001,"5min":2.23,"15min":2.3250000000000002},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3400000000000001,"15min":1.417},"min":{"1min":0.751,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.679,"5min":2.3639999999999999,"15min":2.4620000000000002},"last":1.6259999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3839999999999999,"15min":1.4430000000000001},"min":{"1min":0.89400000000000002,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.7090000000000001,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.514,"15min":1.556},"min":{"1min":0.95399999999999996,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.3069999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.524,"15min":1.611},"min":{"1min":0.88600000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.0840000000000001,"5min":2.4590000000000001,"15min":2.5339999999999998},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.4770000000000001,"15min":1.542},"min":{"1min":0.877,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.819,"5min":2.573,"15min":2.573},"last":1.8480000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542353,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58876,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312964,"statfs":{"total":96636764160,"available":96576475136,"internally_reserved":0,"allocated":1359872,"data_stored":647405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.91700000000000004,"15min":0.90800000000000003},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.609,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.874},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1,"15min":1.0880000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.8500000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.2130000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90600000000000003,"15min":0.98299999999999998},"min":{"1min":0.45900000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.49,"5min":1.8460000000000001,"15min":2.262},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1299999999999999,"15min":1.1799999999999999},"min":{"1min":0.46200000000000002,"5min":0.433,"15min":0.433},"max":{"1min":1.5309999999999999,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.0409999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2270000000000001,"15min":1.3149999999999999},"min":{"1min":0.47899999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.8460000000000001,"5min":1.988,"15min":2.2349999999999999},"last":1.379},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.161,"15min":1.212},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.6200000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.399}]},{"osd":4,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.2490000000000001,"15min":1.306},"min":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.7350000000000001,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.415,"15min":1.4390000000000001},"min":{"1min":0.71899999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9970000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.1699999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.357,"15min":1.429},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.8500000000000001,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.619},{"interface":"front","average":{"1min":1.304,"5min":1.3839999999999999,"15min":1.4299999999999999},"min":{"1min":0.85599999999999998,"5min":0.72699999999999998,"15min":0.70999999999999996},"max":{"1min":1.6679999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":0.95899999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4890000000000001,"15min":1.5289999999999999},"min":{"1min":0.77400000000000002,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.1320000000000001,"5min":2.1779999999999999,"15min":2.202},"last":1.292},{"interface":"front","average":{"1min":1.516,"5min":1.4710000000000001,"15min":1.52},"min":{"1min":0.71399999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.1640000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.7529999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:50:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5589999999999999,"15min":1.621},"min":{"1min":0.75900000000000001,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.2440000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.508,"15min":1.5569999999999999},"min":{"1min":0.68200000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.9139999999999999,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.026}]}]},{"osd":0,"up_from":14,"seq":60129542354,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37272,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334568,"statfs":{"total":96636764160,"available":96598597632,"internally_reserved":0,"allocated":1970176,"data_stored":1244582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.85299999999999998,"15min":0.89300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.655,"5min":1.655,"15min":2.5720000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.0509999999999999,"15min":1.089},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.439},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":2.1800000000000002},"last":0.57399999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.0660000000000001,"15min":1.127},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.579,"5min":1.8120000000000001,"15min":2.7650000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.1140000000000001,"15min":1.159},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.5880000000000001,"5min":1.952,"15min":1.952},"last":0.64700000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.016,"15min":1.0820000000000001},"min":{"1min":0.41899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.609,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.226,"15min":1.3009999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.696,"5min":1.8740000000000001,"15min":3.6579999999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2929999999999999,"15min":1.3300000000000001},"min":{"1min":0.71899999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.542,"5min":1.8660000000000001,"15min":2.6920000000000002},"last":1.121},{"interface":"front","average":{"1min":1.252,"5min":1.3799999999999999,"15min":1.4390000000000001},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.845,"5min":1.966,"15min":3.544},"last":0.90600000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.377,"15min":1.4219999999999999},"min":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8320000000000001,"5min":2.2229999999999999,"15min":2.2440000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3560000000000001,"15min":1.417},"min":{"1min":0.872,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.7490000000000001,"5min":2.02,"15min":2.8610000000000002},"last":1.1859999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.488,"15min":1.5289999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.9790000000000001,"5min":2.0750000000000002,"15min":3.7890000000000001},"last":1.034},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.4470000000000001,"15min":1.516},"min":{"1min":0.92000000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.95,"5min":2.145,"15min":2.363},"last":0.88}]},{"osd":7,"last update":"Mon Oct 16 12:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5620000000000001,"15min":1.6200000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.056,"5min":2.3490000000000002,"15min":2.4729999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.389,"5min":1.4990000000000001,"15min":1.552},"min":{"1min":0.81599999999999995,"5min":0.755,"15min":0.755},"max":{"1min":1.8560000000000001,"5min":2.2050000000000001,"15min":3.1989999999999998},"last":1.2370000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:50:57.003 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:50:57.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:50:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.264+0000 7f33893d7640 1 Processor -- start 2023-10-16T12:50:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.264+0000 7f33893d7640 1 -- start start 2023-10-16T12:50:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33893d7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f338415ac70 0x7f33840cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33893d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33840b4160 0x7f33840cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33893d7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3384069da0 0x7f338406a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33893d7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338406a770 con 0x7f3384069da0 2023-10-16T12:50:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33893d7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33840cc850 con 0x7f338415ac70 2023-10-16T12:50:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33893d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840ccb30 con 0x7f33840b4160 2023-10-16T12:50:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f3388bd6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3384069da0 0x7f338406a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f3388bd6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3384069da0 0x7f338406a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41346/0 (socket says 172.21.15.60:41346) 2023-10-16T12:50:57.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f3388bd6640 1 -- 172.21.15.60:0/448398415 learned_addr learned my addr 172.21.15.60:0/448398415 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f3383fff640 1 --2- 172.21.15.60:0/448398415 >> v2:172.21.15.144:3300/0 conn(0x7f338415ac70 0x7f33840cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f3388bd6640 1 -- 172.21.15.60:0/448398415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33840b4160 msgr2=0x7f33840cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.266+0000 7f33837fe640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33840b4160 0x7f33840cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.267+0000 7f3388bd6640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33840b4160 0x7f33840cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.267+0000 7f3388bd6640 1 -- 172.21.15.60:0/448398415 >> v2:172.21.15.144:3300/0 conn(0x7f338415ac70 msgr2=0x7f33840cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.267+0000 7f3388bd6640 1 --2- 172.21.15.60:0/448398415 >> v2:172.21.15.144:3300/0 conn(0x7f338415ac70 0x7f33840cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.267+0000 7f3388bd6640 1 -- 172.21.15.60:0/448398415 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840cce10 con 0x7f3384069da0 2023-10-16T12:50:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.267+0000 7f3383fff640 1 --2- 172.21.15.60:0/448398415 >> v2:172.21.15.144:3300/0 conn(0x7f338415ac70 0x7f33840cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.267+0000 7f33837fe640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33840b4160 0x7f33840cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.268+0000 7f3388bd6640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3384069da0 0x7f338406a230 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f33840b1290 tx=0x7f337c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e831e16ca4cc830 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.268+0000 7f3382ffd640 1 -- 172.21.15.60:0/448398415 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c02ca80 con 0x7f3384069da0 2023-10-16T12:50:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.268+0000 7f3382ffd640 1 -- 172.21.15.60:0/448398415 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c02cbe0 con 0x7f3384069da0 2023-10-16T12:50:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.268+0000 7f3382ffd640 1 -- 172.21.15.60:0/448398415 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c0343f0 con 0x7f3384069da0 2023-10-16T12:50:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 -- 172.21.15.60:0/448398415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3384069da0 msgr2=0x7f338406a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3384069da0 0x7f338406a230 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f33840b1290 tx=0x7f337c02bee0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 -- 172.21.15.60:0/448398415 shutdown_connections 2023-10-16T12:50:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3384069da0 0x7f338406a230 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 --2- 172.21.15.60:0/448398415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33840b4160 0x7f33840cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 --2- 172.21.15.60:0/448398415 >> v2:172.21.15.144:3300/0 conn(0x7f338415ac70 0x7f33840cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 -- 172.21.15.60:0/448398415 >> 172.21.15.60:0/448398415 conn(0x7f33840ce6a0 msgr2=0x7f33840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 -- 172.21.15.60:0/448398415 shutdown_connections 2023-10-16T12:50:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.269+0000 7f33893d7640 1 -- 172.21.15.60:0/448398415 wait complete. 2023-10-16T12:50:57.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.271+0000 7f33893d7640 1 Processor -- start 2023-10-16T12:50:57.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.271+0000 7f33893d7640 1 -- start start 2023-10-16T12:50:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33893d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3384069da0 0x7f33840b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33893d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 0x7f338415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33893d7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338415ac70 0x7f338415d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33893d7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840b2c10 con 0x7f33840b4160 2023-10-16T12:50:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33893d7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840c6920 con 0x7f338415ac70 2023-10-16T12:50:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33893d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840c6030 con 0x7f3384069da0 2023-10-16T12:50:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33837fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 0x7f338415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33837fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 0x7f338415d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41350/0 (socket says 172.21.15.60:41350) 2023-10-16T12:50:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f33837fe640 1 -- 172.21.15.60:0/2800716578 learned_addr learned my addr 172.21.15.60:0/2800716578 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.272+0000 7f3383fff640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3384069da0 0x7f33840b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f3388bd6640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338415ac70 0x7f338415d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f33837fe640 1 -- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3384069da0 msgr2=0x7f33840b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f33837fe640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3384069da0 0x7f33840b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f33837fe640 1 -- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338415ac70 msgr2=0x7f338415d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f33837fe640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338415ac70 0x7f338415d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f33837fe640 1 -- 172.21.15.60:0/2800716578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c009b30 con 0x7f33840b4160 2023-10-16T12:50:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f33837fe640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 0x7f338415d2d0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f337800c820 tx=0x7f337800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3378007a30 con 0x7f33840b4160 2023-10-16T12:50:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3378007b90 con 0x7f33840b4160 2023-10-16T12:50:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.273+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337800ce10 con 0x7f33840b4160 2023-10-16T12:50:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.274+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33840c62d0 con 0x7f33840b4160 2023-10-16T12:50:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.274+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338415de90 con 0x7f33840b4160 2023-10-16T12:50:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.275+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33840c6ab0 con 0x7f33840b4160 2023-10-16T12:50:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.276+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33780113f0 con 0x7f33840b4160 2023-10-16T12:50:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.277+0000 7f3380ff9640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f337006af90 0x7f337006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.277+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3378089aa0 con 0x7f33840b4160 2023-10-16T12:50:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.278+0000 7f3383fff640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f337006af90 0x7f337006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.279+0000 7f3383fff640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f337006af90 0x7f337006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f337400a990 tx=0x7f3374008040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.281+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3378054810 con 0x7f33840b4160 2023-10-16T12:50:57.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.548+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f338404e9b0 con 0x7f33840b4160 2023-10-16T12:50:57.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.553+0000 7f3380ff9640 1 -- 172.21.15.60:0/2800716578 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3378057e30 con 0x7f33840b4160 2023-10-16T12:50:57.555 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:50:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:50:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:50:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:50:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:50:57.556 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:50:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.556+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f337006af90 msgr2=0x7f337006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.556+0000 7f33893d7640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f337006af90 0x7f337006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f337400a990 tx=0x7f3374008040 comp rx=0 tx=0).stop 2023-10-16T12:50:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 msgr2=0x7f338415d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 0x7f338415d2d0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f337800c820 tx=0x7f337800cce0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 shutdown_connections 2023-10-16T12:50:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f337006af90 0x7f337006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f338415ac70 0x7f338415d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33840b4160 0x7f338415d2d0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 --2- 172.21.15.60:0/2800716578 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3384069da0 0x7f33840b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 >> 172.21.15.60:0/2800716578 conn(0x7f33840ce6a0 msgr2=0x7f33840cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 shutdown_connections 2023-10-16T12:50:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.557+0000 7f33893d7640 1 -- 172.21.15.60:0/2800716578 wait complete. 2023-10-16T12:50:57.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:50:57.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.796+0000 7facae651640 1 Processor -- start 2023-10-16T12:50:57.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.797+0000 7facae651640 1 -- start start 2023-10-16T12:50:57.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7facae651640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faca815aaa0 0x7faca815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7facae651640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faca815ae80 0x7faca80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7facae651640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca815ed60 con 0x7faca80ccaa0 2023-10-16T12:50:57.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7facae651640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca806b810 con 0x7faca815aaa0 2023-10-16T12:50:57.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7facae651640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca80707e0 con 0x7faca815ae80 2023-10-16T12:50:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7faca7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faca815aaa0 0x7faca815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7faca7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faca815aaa0 0x7faca815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41366/0 (socket says 172.21.15.60:41366) 2023-10-16T12:50:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.798+0000 7faca7fff640 1 -- 172.21.15.60:0/2097207504 learned_addr learned my addr 172.21.15.60:0/2097207504 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.799+0000 7faca7fff640 1 -- 172.21.15.60:0/2097207504 >> v2:172.21.15.144:3300/0 conn(0x7faca815ae80 msgr2=0x7faca80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.799+0000 7faca7fff640 1 --2- 172.21.15.60:0/2097207504 >> v2:172.21.15.144:3300/0 conn(0x7faca815ae80 0x7faca80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.799+0000 7faca7fff640 1 -- 172.21.15.60:0/2097207504 >> v1:172.21.15.144:6789/0 conn(0x7faca80ccaa0 legacy=0x7faca8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.799+0000 7faca7fff640 1 -- 172.21.15.60:0/2097207504 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca815ef90 con 0x7faca815aaa0 2023-10-16T12:50:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.800+0000 7faca7fff640 1 --2- 172.21.15.60:0/2097207504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca815e820 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7faca80c59c0 tx=0x7fac9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be8cb189139930b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.800+0000 7faca6ffd640 1 -- 172.21.15.60:0/2097207504 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac98031870 con 0x7faca815aaa0 2023-10-16T12:50:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.800+0000 7faca6ffd640 1 -- 172.21.15.60:0/2097207504 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac980319d0 con 0x7faca815aaa0 2023-10-16T12:50:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.800+0000 7faca6ffd640 1 -- 172.21.15.60:0/2097207504 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac98031d60 con 0x7faca815aaa0 2023-10-16T12:50:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 -- 172.21.15.60:0/2097207504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 msgr2=0x7faca815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 --2- 172.21.15.60:0/2097207504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca815e820 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7faca80c59c0 tx=0x7fac9802b910 comp rx=0 tx=0).stop 2023-10-16T12:50:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 -- 172.21.15.60:0/2097207504 shutdown_connections 2023-10-16T12:50:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 --2- 172.21.15.60:0/2097207504 >> v2:172.21.15.144:3300/0 conn(0x7faca815ae80 0x7faca80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 --2- 172.21.15.60:0/2097207504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca815e820 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 -- 172.21.15.60:0/2097207504 >> 172.21.15.60:0/2097207504 conn(0x7faca80cb410 msgr2=0x7faca80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 -- 172.21.15.60:0/2097207504 shutdown_connections 2023-10-16T12:50:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.801+0000 7facae651640 1 -- 172.21.15.60:0/2097207504 wait complete. 2023-10-16T12:50:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.803+0000 7facae651640 1 Processor -- start 2023-10-16T12:50:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.803+0000 7facae651640 1 -- start start 2023-10-16T12:50:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7facae651640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80ccaa0 0x7faca815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7facae651640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca8153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7facae651640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca815ae80 0x7faca80c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7facae651640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca8155ca0 con 0x7faca815aaa0 2023-10-16T12:50:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7facae651640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca80c4680 con 0x7faca80ccaa0 2023-10-16T12:50:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7facae651640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca80c37f0 con 0x7faca815ae80 2023-10-16T12:50:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7faca77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca8153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7faca77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca8153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41380/0 (socket says 172.21.15.60:41380) 2023-10-16T12:50:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7faca77fe640 1 -- 172.21.15.60:0/2654567895 learned_addr learned my addr 172.21.15.60:0/2654567895 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.804+0000 7faca7fff640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80ccaa0 0x7faca815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca77fe640 1 -- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca815ae80 msgr2=0x7faca80c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca77fe640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca815ae80 0x7faca80c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca77fe640 1 -- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80ccaa0 msgr2=0x7faca815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca77fe640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80ccaa0 0x7faca815dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca77fe640 1 -- 172.21.15.60:0/2654567895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac980094d0 con 0x7faca815aaa0 2023-10-16T12:50:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca77fe640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca8153bc0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fac9000b9c0 tx=0x7fac9000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9000cba0 con 0x7faca815aaa0 2023-10-16T12:50:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9000cd20 con 0x7faca815aaa0 2023-10-16T12:50:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90014680 con 0x7faca815aaa0 2023-10-16T12:50:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca80c4100 con 0x7faca815aaa0 2023-10-16T12:50:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.805+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca815b4c0 con 0x7faca815aaa0 2023-10-16T12:50:57.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.807+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca815b320 con 0x7faca815aaa0 2023-10-16T12:50:57.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.808+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac9001d3b0 con 0x7faca815aaa0 2023-10-16T12:50:57.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.809+0000 7faca4ff9640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fac8806afc0 0x7fac8806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:57.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.809+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac90089640 con 0x7faca815aaa0 2023-10-16T12:50:57.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.809+0000 7faca7fff640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fac8806afc0 0x7fac8806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:57.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.810+0000 7faca7fff640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fac8806afc0 0x7fac8806d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac98036000 tx=0x7fac9803c040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:57.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:57.813+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faca815b320 con 0x7faca815aaa0 2023-10-16T12:50:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.047+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faca815b320 con 0x7faca815aaa0 2023-10-16T12:50:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.048+0000 7faca4ff9640 1 -- 172.21.15.60:0/2654567895 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4915 (secure 0 0 0) 0x7faca815b320 con 0x7faca815aaa0 2023-10-16T12:50:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:50:58.051 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:51.414872+0000","modified":"2023-10-16T12:50:54.480441+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7353":{"gid":7353,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/3569389918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3569389918},{"type":"v1","addr":"172.21.15.60:6835","nonce":3569389918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:50:58.051 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 53 2023-10-16T12:50:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.051+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fac8806afc0 msgr2=0x7fac8806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.051+0000 7facae651640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fac8806afc0 0x7fac8806d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac98036000 tx=0x7fac9803c040 comp rx=0 tx=0).stop 2023-10-16T12:50:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.051+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 msgr2=0x7faca8153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.051+0000 7facae651640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca8153bc0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fac9000b9c0 tx=0x7fac9000be80 comp rx=0 tx=0).stop 2023-10-16T12:50:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 shutdown_connections 2023-10-16T12:50:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fac8806afc0 0x7fac8806d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faca815ae80 0x7faca80c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faca815aaa0 0x7faca8153bc0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 --2- 172.21.15.60:0/2654567895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faca80ccaa0 0x7faca815dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 >> 172.21.15.60:0/2654567895 conn(0x7faca80cb410 msgr2=0x7faca80b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 shutdown_connections 2023-10-16T12:50:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.052+0000 7facae651640 1 -- 172.21.15.60:0/2654567895 wait complete. 2023-10-16T12:50:58.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:50:51.414872+0000', 'modified': '2023-10-16T12:50:54.480441+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7353}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7353': {'gid': 7353, 'name': 'a', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.60:6835/3569389918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6833', 'nonce': 3569389918}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 3569389918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:50:58.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:50:58.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:50:58.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.715+0000 7f6e06a40640 1 Processor -- start 2023-10-16T12:50:58.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.715+0000 7f6e06a40640 1 -- start start 2023-10-16T12:50:58.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.716+0000 7f6e06a40640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e000c9060 0x7f6e0006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.716+0000 7f6e06a40640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e0006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e06a40640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e000ca560 con 0x7f6e000c9440 2023-10-16T12:50:58.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e06a40640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000c9950 con 0x7f6e000c9060 2023-10-16T12:50:58.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e06a40640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000c9c30 con 0x7f6e0006c9e0 2023-10-16T12:50:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e05a3e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e000c9060 0x7f6e0006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e05a3e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e000c9060 0x7f6e0006f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41390/0 (socket says 172.21.15.60:41390) 2023-10-16T12:50:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e05a3e640 1 -- 172.21.15.60:0/34272746 learned_addr learned my addr 172.21.15.60:0/34272746 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.717+0000 7f6e0523d640 1 --2- 172.21.15.60:0/34272746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e0006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.718+0000 7f6e05a3e640 1 -- 172.21.15.60:0/34272746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 msgr2=0x7f6e0006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.718+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/34272746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e0006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.718+0000 7f6e05a3e640 1 -- 172.21.15.60:0/34272746 >> v1:172.21.15.144:6789/0 conn(0x7f6e000c9440 legacy=0x7f6e00164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.718+0000 7f6e05a3e640 1 -- 172.21.15.60:0/34272746 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000c9f10 con 0x7f6e000c9060 2023-10-16T12:50:58.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.719+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/34272746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e0006f440 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151ce00660fc6cb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.719+0000 7f6e04a3c640 1 -- 172.21.15.60:0/34272746 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc031640 con 0x7f6e000c9060 2023-10-16T12:50:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.719+0000 7f6e04a3c640 1 -- 172.21.15.60:0/34272746 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc0317a0 con 0x7f6e000c9060 2023-10-16T12:50:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.719+0000 7f6e04a3c640 1 -- 172.21.15.60:0/34272746 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc031b30 con 0x7f6e000c9060 2023-10-16T12:50:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 -- 172.21.15.60:0/34272746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 msgr2=0x7f6e0006f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 --2- 172.21.15.60:0/34272746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e0006f440 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc02b8f0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 -- 172.21.15.60:0/34272746 shutdown_connections 2023-10-16T12:50:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 --2- 172.21.15.60:0/34272746 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e0006f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 --2- 172.21.15.60:0/34272746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e0006f440 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 -- 172.21.15.60:0/34272746 >> 172.21.15.60:0/34272746 conn(0x7f6e000ce6a0 msgr2=0x7f6e000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 -- 172.21.15.60:0/34272746 shutdown_connections 2023-10-16T12:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.720+0000 7f6e06a40640 1 -- 172.21.15.60:0/34272746 wait complete. 2023-10-16T12:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.722+0000 7f6e06a40640 1 Processor -- start 2023-10-16T12:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.722+0000 7f6e06a40640 1 -- start start 2023-10-16T12:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.723+0000 7f6e06a40640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e001fa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.723+0000 7f6e06a40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e001f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.723+0000 7f6e06a40640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e000c9440 0x7f6e00069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.723+0000 7f6e06a40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e001fac80 con 0x7f6e000c9060 2023-10-16T12:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.723+0000 7f6e06a40640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000c6d80 con 0x7f6e000c9440 2023-10-16T12:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.723+0000 7f6e06a40640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000c7060 con 0x7f6e0006c9e0 2023-10-16T12:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e001f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e001f6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41406/0 (socket says 172.21.15.60:41406) 2023-10-16T12:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 -- 172.21.15.60:0/1514757720 learned_addr learned my addr 172.21.15.60:0/1514757720 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e001fa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0623f640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e000c9440 0x7f6e00069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 -- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 msgr2=0x7f6e001fa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e001fa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 -- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e000c9440 msgr2=0x7f6e00069fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e000c9440 0x7f6e00069fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 -- 172.21.15.60:0/1514757720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc0094d0 con 0x7f6e000c9060 2023-10-16T12:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0623f640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e000c9440 0x7f6e00069fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.724+0000 7f6e0523d640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e001f6020 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f6df000d7d0 tx=0x7f6df000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.725+0000 7f6df67fc640 1 -- 172.21.15.60:0/1514757720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df000b6c0 con 0x7f6e000c9060 2023-10-16T12:50:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.725+0000 7f6df67fc640 1 -- 172.21.15.60:0/1514757720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df000b820 con 0x7f6e000c9060 2023-10-16T12:50:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.725+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e000c7340 con 0x7f6e000c9060 2023-10-16T12:50:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.725+0000 7f6df67fc640 1 -- 172.21.15.60:0/1514757720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df000bbb0 con 0x7f6e000c9060 2023-10-16T12:50:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.725+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e001f6560 con 0x7f6e000c9060 2023-10-16T12:50:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.729+0000 7f6df67fc640 1 -- 172.21.15.60:0/1514757720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6df000bd10 con 0x7f6e000c9060 2023-10-16T12:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.730+0000 7f6df67fc640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f6de406af90 0x7f6de406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.730+0000 7f6e0a835740 1 Processor -- start 2023-10-16T12:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.730+0000 7f6df67fc640 1 -- 172.21.15.60:0/1514757720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6df0089d60 con 0x7f6e000c9060 2023-10-16T12:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.730+0000 7f6e0a835740 1 -- start start 2023-10-16T12:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.730+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f6de406af90 0x7f6de406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462170 0x558e8d435070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d464f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 0x558e8d465440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0a835740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d3557a0 con 0x558e8d4355b0 2023-10-16T12:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d3a7740 con 0x558e8d462170 2023-10-16T12:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e8d3a4e30 con 0x558e8d462550 2023-10-16T12:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0623f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 0x558e8d465440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0623f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 0x558e8d465440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41420/0 (socket says 172.21.15.60:41420) 2023-10-16T12:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0623f640 1 -- 172.21.15.60:0/788094881 learned_addr learned my addr 172.21.15.60:0/788094881 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.731+0000 7f6e0523d640 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d464f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462170 0x558e8d435070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0623f640 1 -- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 msgr2=0x558e8d464f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0623f640 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d464f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0623f640 1 -- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462170 msgr2=0x558e8d435070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0623f640 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462170 0x558e8d435070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f6de406af90 0x7f6de406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc036000 tx=0x7f6dfc008030 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0623f640 1 -- 172.21.15.60:0/788094881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e8d364910 con 0x558e8d4355b0 2023-10-16T12:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0523d640 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d464f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6e0623f640 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 0x558e8d465440 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f6df8007f10 tx=0x7f6df800c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348e2469edb97dd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6de37fe640 1 -- 172.21.15.60:0/788094881 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800cea0 con 0x558e8d4355b0 2023-10-16T12:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6de37fe640 1 -- 172.21.15.60:0/788094881 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df800f830 con 0x558e8d4355b0 2023-10-16T12:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.732+0000 7f6de37fe640 1 -- 172.21.15.60:0/788094881 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800fbc0 con 0x558e8d4355b0 2023-10-16T12:50:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 -- 172.21.15.60:0/788094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 msgr2=0x558e8d465440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 0x558e8d465440 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f6df8007f10 tx=0x7f6df800c670 comp rx=0 tx=0).stop 2023-10-16T12:50:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 -- 172.21.15.60:0/788094881 shutdown_connections 2023-10-16T12:50:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d464f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d4355b0 0x558e8d465440 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 --2- 172.21.15.60:0/788094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462170 0x558e8d435070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 -- 172.21.15.60:0/788094881 >> 172.21.15.60:0/788094881 conn(0x558e8d437380 msgr2=0x558e8d47ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 -- 172.21.15.60:0/788094881 shutdown_connections 2023-10-16T12:50:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.733+0000 7f6e0a835740 1 -- 172.21.15.60:0/788094881 wait complete. 2023-10-16T12:50:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.734+0000 7f6e0a835740 1 -- start start 2023-10-16T12:50:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.734+0000 7f6e0a835740 1 Processor -- start 2023-10-16T12:50:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d478f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d465ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d466430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d3a0720 con 0x558e8d462170 2023-10-16T12:50:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d33e970 con 0x558e8d468760 2023-10-16T12:50:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e8d33e610 con 0x558e8d462550 2023-10-16T12:50:58.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e0a835740 10 client.0 _ll_register_callbacks cb 0x558e8d3609d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:50:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e05a3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d478f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e05a3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d478f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41436/0 (socket says 172.21.15.60:41436) 2023-10-16T12:50:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.735+0000 7f6e05a3e640 1 -- 172.21.15.60:0/4091398871 learned_addr learned my addr 172.21.15.60:0/4091398871 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e05a3e640 1 -- 172.21.15.60:0/4091398871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 msgr2=0x558e8d465ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d465ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e05a3e640 1 -- 172.21.15.60:0/4091398871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 msgr2=0x558e8d466430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d466430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e05a3e640 1 -- 172.21.15.60:0/4091398871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df8007b10 con 0x558e8d462170 2023-10-16T12:50:58.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d478f70 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009990 tx=0x7f6dfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1e6de5470aa293 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6de0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:50:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6de0ff9640 1 -- 172.21.15.60:0/4091398871 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc04b070 con 0x558e8d462170 2023-10-16T12:50:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6de0ff9640 1 -- 172.21.15.60:0/4091398871 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dfc046b40 con 0x558e8d462170 2023-10-16T12:50:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.736+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e8d3c7160 con 0x558e8d462170 2023-10-16T12:50:58.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6de0ff9640 1 -- 172.21.15.60:0/4091398871 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dfc046d00 con 0x558e8d462170 2023-10-16T12:50:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 10 client.7497 fetch_fsmap learned FSMap version 53 2023-10-16T12:50:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x558e8d39eda0 con 0x558e8d462170 2023-10-16T12:50:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6de0ff9640 1 -- 172.21.15.60:0/4091398871 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e8d3c7160 con 0x558e8d462170 2023-10-16T12:50:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6de0ff9640 1 -- 172.21.15.60:0/4091398871 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 53) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6dfc03a420 con 0x558e8d462170 2023-10-16T12:50:58.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 10 client.7497 fetch_fsmap finished waiting for FSMap version 53 2023-10-16T12:50:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 10 client.7497 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:50:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 20 client.7497 populate_metadata read hostname 'smithi060' 2023-10-16T12:50:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d466970 0x558e8d493b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 4 client.7497 mds_command: new command op to 7353 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:50:58.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.737+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 --> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e8d372cc0 con 0x558e8d466970 2023-10-16T12:50:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.738+0000 7f6e0523d640 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d466970 0x558e8d493b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.738+0000 7f6e0523d640 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d466970 0x558e8d493b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e757fe7ee6b11f5 server_cookie=ea8948bd0beb8058 in_seq=0 out_seq=0 2023-10-16T12:50:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.738+0000 7f6de0ff9640 10 client.7497 ms_handle_connect on v2:172.21.15.60:6833/3569389918 2023-10-16T12:50:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6de0ff9640 1 -- 172.21.15.60:0/4091398871 <== mds.0 v2:172.21.15.60:6833/3569389918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558e8d372cc0 con 0x558e8d466970 2023-10-16T12:50:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6de0ff9640 10 client.7497 handle_command_reply: tid=0 2023-10-16T12:50:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 client.7497 shutdown 2023-10-16T12:50:58.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 msgr2=0x558e8d478f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d478f70 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009990 tx=0x7f6dfc004730 comp rx=0 tx=0).stop 2023-10-16T12:50:58.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 shutdown_connections 2023-10-16T12:50:58.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d466970 0x558e8d493b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462550 0x558e8d465ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d466430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 --2- 172.21.15.60:0/4091398871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d478f70 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6de0ff9640 0 client.7497 ms_handle_reset on v2:172.21.15.60:6833/3569389918 2023-10-16T12:50:58.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.739+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 >> 172.21.15.60:0/4091398871 conn(0x558e8d4355b0 msgr2=0x558e8d439020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.740+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 shutdown_connections 2023-10-16T12:50:58.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.740+0000 7f6e0a835740 1 -- 172.21.15.60:0/4091398871 wait complete. 2023-10-16T12:50:58.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.740+0000 7f6e0a835740 20 client.7497 trim_cache size 0 max 16384 2023-10-16T12:50:58.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.757+0000 7f6e0a835740 1 Processor -- start 2023-10-16T12:50:58.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.757+0000 7f6e0a835740 1 -- start start 2023-10-16T12:50:58.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462170 0x558e8d467c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 0x558e8d467170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d4836e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0a835740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d104f20 con 0x558e8d462550 2023-10-16T12:50:58.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d4791a0 con 0x558e8d468760 2023-10-16T12:50:58.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e8d483220 con 0x558e8d462170 2023-10-16T12:50:58.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0523d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 0x558e8d467170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0523d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 0x558e8d467170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41446/0 (socket says 172.21.15.60:41446) 2023-10-16T12:50:58.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e0523d640 1 -- 172.21.15.60:0/3996709597 learned_addr learned my addr 172.21.15.60:0/3996709597 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.758+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462170 0x558e8d467c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0623f640 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d4836e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0523d640 1 -- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462170 msgr2=0x558e8d467c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0523d640 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462170 0x558e8d467c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0523d640 1 -- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 msgr2=0x558e8d4836e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0523d640 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d4836e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0523d640 1 -- 172.21.15.60:0/3996709597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e8d33db80 con 0x558e8d462550 2023-10-16T12:50:58.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0623f640 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d4836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:58.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6e0523d640 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 0x558e8d467170 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6df000dfd0 tx=0x7f6df0054cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6402fecc76ebf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6de3fff640 1 -- 172.21.15.60:0/3996709597 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0084070 con 0x558e8d462550 2023-10-16T12:50:58.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6de3fff640 1 -- 172.21.15.60:0/3996709597 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0058780 con 0x558e8d462550 2023-10-16T12:50:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.759+0000 7f6de3fff640 1 -- 172.21.15.60:0/3996709597 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0056d50 con 0x558e8d462550 2023-10-16T12:50:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 -- 172.21.15.60:0/3996709597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 msgr2=0x558e8d467170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 0x558e8d467170 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6df000dfd0 tx=0x7f6df0054cf0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 -- 172.21.15.60:0/3996709597 shutdown_connections 2023-10-16T12:50:58.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d468760 0x558e8d4836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462550 0x558e8d467170 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 --2- 172.21.15.60:0/3996709597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d462170 0x558e8d467c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 -- 172.21.15.60:0/3996709597 >> 172.21.15.60:0/3996709597 conn(0x558e8d4355b0 msgr2=0x558e8d465ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 -- 172.21.15.60:0/3996709597 shutdown_connections 2023-10-16T12:50:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.760+0000 7f6e0a835740 1 -- 172.21.15.60:0/3996709597 wait complete. 2023-10-16T12:50:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.761+0000 7f6e0a835740 1 -- start start 2023-10-16T12:50:58.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.761+0000 7f6e0a835740 1 Processor -- start 2023-10-16T12:50:58.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d43a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462550 0x558e8d4e8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d468760 0x558e8d48eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d48e430 con 0x558e8d462170 2023-10-16T12:50:58.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8d43aee0 con 0x558e8d462550 2023-10-16T12:50:58.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e8d4664c0 con 0x558e8d468760 2023-10-16T12:50:58.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e0a835740 10 client.0 _ll_register_callbacks cb 0x558e8d4789c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:50:58.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e05a3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d43a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e05a3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d43a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41456/0 (socket says 172.21.15.60:41456) 2023-10-16T12:50:58.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.762+0000 7f6e05a3e640 1 -- 172.21.15.60:0/440477216 learned_addr learned my addr 172.21.15.60:0/440477216 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:58.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e0523d640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462550 0x558e8d4e8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e0623f640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d468760 0x558e8d48eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e05a3e640 1 -- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d468760 msgr2=0x558e8d48eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d468760 0x558e8d48eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e05a3e640 1 -- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462550 msgr2=0x558e8d4e8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:58.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462550 0x558e8d4e8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e05a3e640 1 -- 172.21.15.60:0/440477216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e001f6560 con 0x558e8d462170 2023-10-16T12:50:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6e05a3e640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d43a6c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc031770 tx=0x7f6dfc03c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637367c04d74ebc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6de0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:50:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.763+0000 7f6de0ff9640 1 -- 172.21.15.60:0/440477216 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc046d40 con 0x558e8d462170 2023-10-16T12:50:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6de0ff9640 1 -- 172.21.15.60:0/440477216 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dfc031ca0 con 0x558e8d462170 2023-10-16T12:50:58.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6df800f830 con 0x558e8d462170 2023-10-16T12:50:58.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6de0ff9640 1 -- 172.21.15.60:0/440477216 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dfc046ea0 con 0x558e8d462170 2023-10-16T12:50:58.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 10 client.7503 fetch_fsmap learned FSMap version 53 2023-10-16T12:50:58.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x558e8d363170 con 0x558e8d462170 2023-10-16T12:50:58.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6de0ff9640 1 -- 172.21.15.60:0/440477216 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df800f830 con 0x558e8d462170 2023-10-16T12:50:58.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6de0ff9640 1 -- 172.21.15.60:0/440477216 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 53) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6dfc004030 con 0x558e8d462170 2023-10-16T12:50:58.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 10 client.7503 fetch_fsmap finished waiting for FSMap version 53 2023-10-16T12:50:58.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 10 client.7503 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:50:58.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 20 client.7503 populate_metadata read hostname 'smithi060' 2023-10-16T12:50:58.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d45c980 0x558e8d45cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:58.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 4 client.7503 mds_command: new command op to 7353 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:50:58.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.764+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 --> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558e8d399700 con 0x558e8d45c980 2023-10-16T12:50:58.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.765+0000 7f6e0523d640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d45c980 0x558e8d45cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:58.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.765+0000 7f6e0523d640 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d45c980 0x558e8d45cd60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3df597fa95ed2a3 server_cookie=50e883b6de9e117d in_seq=0 out_seq=0 2023-10-16T12:50:58.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.765+0000 7f6de0ff9640 10 client.7503 ms_handle_connect on v2:172.21.15.60:6833/3569389918 2023-10-16T12:50:58.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.765+0000 7f6de0ff9640 1 -- 172.21.15.60:0/440477216 <== mds.0 v2:172.21.15.60:6833/3569389918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x558e8d399700 con 0x558e8d45c980 2023-10-16T12:50:58.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.765+0000 7f6de0ff9640 10 client.7503 handle_command_reply: tid=0 2023-10-16T12:50:58.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.765+0000 7f6e0a835740 1 client.7503 shutdown 2023-10-16T12:50:58.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 msgr2=0x558e8d43a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d43a6c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc031770 tx=0x7f6dfc03c040 comp rx=0 tx=0).stop 2023-10-16T12:50:58.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 shutdown_connections 2023-10-16T12:50:58.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:6833/3569389918,v1:172.21.15.60:6835/3569389918] conn(0x558e8d45c980 0x558e8d45cd60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x558e8d468760 0x558e8d48eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558e8d462550 0x558e8d4e8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 --2- 172.21.15.60:0/440477216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558e8d462170 0x558e8d43a6c0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6de0ff9640 0 client.7503 ms_handle_reset on v2:172.21.15.60:6833/3569389918 2023-10-16T12:50:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 >> 172.21.15.60:0/440477216 conn(0x558e8d4355b0 msgr2=0x558e8d4836e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 shutdown_connections 2023-10-16T12:50:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 1 -- 172.21.15.60:0/440477216 wait complete. 2023-10-16T12:50:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e0a835740 20 client.7503 trim_cache size 0 max 16384 2023-10-16T12:50:58.780 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:50:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:50:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:50:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 7353, 2023-10-16T12:50:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:50:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:50:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:50:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8049715050000001, 2023-10-16T12:50:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 53, 2023-10-16T12:50:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 113, 2023-10-16T12:50:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 113, 2023-10-16T12:50:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 19.114015317 2023-10-16T12:50:58.783 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:50:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f6de406af90 msgr2=0x7f6de406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e06a40640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f6de406af90 0x7f6de406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc036000 tx=0x7f6dfc008030 comp rx=0 tx=0).stop 2023-10-16T12:50:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 msgr2=0x7f6e001f6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.766+0000 7f6e06a40640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e001f6020 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f6df000d7d0 tx=0x7f6df000dc90 comp rx=0 tx=0).stop 2023-10-16T12:50:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 shutdown_connections 2023-10-16T12:50:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f6de406af90 0x7f6de406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e000c9440 0x7f6e00069fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e000c9060 0x7f6e001f6020 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 --2- 172.21.15.60:0/1514757720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e0006c9e0 0x7f6e001fa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 >> 172.21.15.60:0/1514757720 conn(0x7f6e000ce6a0 msgr2=0x7f6e000b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 shutdown_connections 2023-10-16T12:50:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:58.767+0000 7f6e06a40640 1 -- 172.21.15.60:0/1514757720 wait complete. 2023-10-16T12:50:58.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:50:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.018+0000 7f991698e640 1 Processor -- start 2023-10-16T12:50:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.018+0000 7f991698e640 1 -- start start 2023-10-16T12:50:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991698e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f991015cdb0 0x7f99100cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991698e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99100acfa0 0x7f991006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991698e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99100c8cf0 con 0x7f99100c9a20 2023-10-16T12:50:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991698e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991015d190 con 0x7f991015cdb0 2023-10-16T12:50:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991698e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99100ccca0 con 0x7f99100acfa0 2023-10-16T12:50:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991598c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f991015cdb0 0x7f99100cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991518b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f99100c9a20 0x7f99100ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41526/0 (socket says 172.21.15.60:41526) 2023-10-16T12:50:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.020+0000 7f991518b640 1 -- 172.21.15.60:0/2161736782 learned_addr learned my addr 172.21.15.60:0/2161736782 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:50:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.021+0000 7f991618d640 1 --2- 172.21.15.60:0/2161736782 >> v2:172.21.15.144:3300/0 conn(0x7f99100acfa0 0x7f991006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.021+0000 7f991498a640 1 -- 172.21.15.60:0/2161736782 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752081688 0 0) 0x7f99100c8cf0 con 0x7f99100c9a20 2023-10-16T12:50:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.021+0000 7f991598c640 1 -- 172.21.15.60:0/2161736782 >> v2:172.21.15.144:3300/0 conn(0x7f99100acfa0 msgr2=0x7f991006fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:50:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.021+0000 7f991598c640 1 --2- 172.21.15.60:0/2161736782 >> v2:172.21.15.144:3300/0 conn(0x7f99100acfa0 0x7f991006fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.021+0000 7f991598c640 1 -- 172.21.15.60:0/2161736782 >> v1:172.21.15.60:6789/0 conn(0x7f99100c9a20 legacy=0x7f99100ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.021+0000 7f991598c640 1 -- 172.21.15.60:0/2161736782 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99100c8f40 con 0x7f991015cdb0 2023-10-16T12:50:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.022+0000 7f991598c640 1 --2- 172.21.15.60:0/2161736782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100cc270 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f99100c5700 tx=0x7f990c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31a9ec3c4f2a381 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.022+0000 7f991498a640 1 -- 172.21.15.60:0/2161736782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c02e070 con 0x7f991015cdb0 2023-10-16T12:50:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.022+0000 7f991498a640 1 -- 172.21.15.60:0/2161736782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c0315a0 con 0x7f991015cdb0 2023-10-16T12:50:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.022+0000 7f991498a640 1 -- 172.21.15.60:0/2161736782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c031930 con 0x7f991015cdb0 2023-10-16T12:50:59.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 -- 172.21.15.60:0/2161736782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 msgr2=0x7f99100cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 --2- 172.21.15.60:0/2161736782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100cc270 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f99100c5700 tx=0x7f990c02b840 comp rx=0 tx=0).stop 2023-10-16T12:50:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 -- 172.21.15.60:0/2161736782 shutdown_connections 2023-10-16T12:50:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 --2- 172.21.15.60:0/2161736782 >> v2:172.21.15.144:3300/0 conn(0x7f99100acfa0 0x7f991006fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 --2- 172.21.15.60:0/2161736782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100cc270 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 -- 172.21.15.60:0/2161736782 >> 172.21.15.60:0/2161736782 conn(0x7f99100b0a90 msgr2=0x7f99100ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 -- 172.21.15.60:0/2161736782 shutdown_connections 2023-10-16T12:50:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.023+0000 7f991698e640 1 -- 172.21.15.60:0/2161736782 wait complete. 2023-10-16T12:50:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.025+0000 7f991698e640 1 Processor -- start 2023-10-16T12:50:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.025+0000 7f991698e640 1 -- start start 2023-10-16T12:50:59.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991698e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99100acfa0 0x7f9910160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991698e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99100c9a20 0x7f99100c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991698e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991698e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99100c4900 con 0x7f991015cdb0 2023-10-16T12:50:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991698e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991014cd60 con 0x7f99100c9a20 2023-10-16T12:50:59.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991698e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9910146970 con 0x7f99100acfa0 2023-10-16T12:50:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991618d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991618d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100c28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53964/0 (socket says 172.21.15.60:53964) 2023-10-16T12:50:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991618d640 1 -- 172.21.15.60:0/2475519646 learned_addr learned my addr 172.21.15.60:0/2475519646 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:50:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.026+0000 7f991598c640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99100acfa0 0x7f9910160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:59.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991518b640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99100c9a20 0x7f99100c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991618d640 1 -- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99100acfa0 msgr2=0x7f9910160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991618d640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99100acfa0 0x7f9910160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991618d640 1 -- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99100c9a20 msgr2=0x7f99100c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991618d640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99100c9a20 0x7f99100c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991618d640 1 -- 172.21.15.60:0/2475519646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c009400 con 0x7f991015cdb0 2023-10-16T12:50:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991598c640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99100acfa0 0x7f9910160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991518b640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99100c9a20 0x7f99100c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:50:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991618d640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100c28c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9908007e40 tx=0x7f990800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908014620 con 0x7f991015cdb0 2023-10-16T12:50:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990800ee90 con 0x7f991015cdb0 2023-10-16T12:50:59.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99080149e0 con 0x7f991015cdb0 2023-10-16T12:50:59.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99100c4b10 con 0x7f991015cdb0 2023-10-16T12:50:59.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.027+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99100c50d0 con 0x7f991015cdb0 2023-10-16T12:50:59.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.028+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99100c2020 con 0x7f991015cdb0 2023-10-16T12:50:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.029+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9908014b40 con 0x7f991015cdb0 2023-10-16T12:50:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.030+0000 7f99067fc640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f98f406af90 0x7f98f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:50:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.030+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f990808a120 con 0x7f991015cdb0 2023-10-16T12:50:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.031+0000 7f991598c640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f98f406af90 0x7f98f406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:50:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.032+0000 7f991598c640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f98f406af90 0x7f98f406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f990c036000 tx=0x7f990c03d040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:50:59.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.035+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9908055150 con 0x7f991015cdb0 2023-10-16T12:50:59.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.272+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f991004e9b0 con 0x7f991015cdb0 2023-10-16T12:50:59.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.273+0000 7f99067fc640 1 -- 172.21.15.60:0/2475519646 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9908058770 con 0x7f991015cdb0 2023-10-16T12:50:59.274 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:50:59.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f98f406af90 msgr2=0x7f98f406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:59.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f98f406af90 0x7f98f406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f990c036000 tx=0x7f990c03d040 comp rx=0 tx=0).stop 2023-10-16T12:50:59.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 msgr2=0x7f99100c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:50:59.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100c28c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9908007e40 tx=0x7f990800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 shutdown_connections 2023-10-16T12:50:59.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f98f406af90 0x7f98f406d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f991015cdb0 0x7f99100c28c0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99100c9a20 0x7f99100c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 --2- 172.21.15.60:0/2475519646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f99100acfa0 0x7f9910160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:50:59.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 >> 172.21.15.60:0/2475519646 conn(0x7f99100b0a90 msgr2=0x7f99100af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:50:59.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 shutdown_connections 2023-10-16T12:50:59.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:50:59.275+0000 7f991698e640 1 -- 172.21.15.60:0/2475519646 wait complete. 2023-10-16T12:50:59.296 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:50:59.296 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:50:59.296 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:50:59.297 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:50:59.297 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:59.297 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:50:59.298 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:50:59.298 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:50:59.298 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:50:59.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:59.299 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:50:59.359 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:50:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:50:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:50:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:50:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.360 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:50:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:50:59.361 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:50:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:50:59.362 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:50:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:50:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:50:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41328sec preferred_lft 41328sec 2023-10-16T12:50:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:50:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.363 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:50:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:50:59.363 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:50:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:50:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:50:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:50:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.365 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:50:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:50:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:50:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.365 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:50:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:50:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:50:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:50:59.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:59.367 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:50:59.464 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:50:59.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:59.469 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:50:59.526 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:50:59.527 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:50:59.527 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:50:59.527 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:50:59.527 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:50:59.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:59.529 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:50:59.529 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:50:59.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:50:59.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:50:59.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:50:59.531 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:50:59.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:59.735 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:50:59.790 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:50:59.790 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:50:59.791 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:50:59.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:50:59.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:50:59.897 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:50:59.898 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:50:59.957 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:50:59.957 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:50:59.958 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:51:00.015 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:51:00.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:51:00.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:51:00.156 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:51:00.157 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:51:00.192 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:51:00.192 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:51:00.192 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:51:00.193 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:51:00.193 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:51:00.195 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:51:00.195 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:51:00.195 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:51:00.196 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:51:00.196 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:51:00.199 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:51:00.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:51:00.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:51:00.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:51:00.295 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:51:00.366 INFO:teuthology.orchestra.run.smithi060.stdout:client6460 2023-10-16T12:51:00.366 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:51:00.367 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:51:00.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:51:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.649+0000 7fcafb59e640 1 Processor -- start 2023-10-16T12:51:00.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.649+0000 7fcafb59e640 1 -- start start 2023-10-16T12:51:00.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafb59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcafc15c9f0 0x7fcafc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:00.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafb59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:00.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafb59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcafc160a90 con 0x7fcafc070f20 2023-10-16T12:51:00.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafb59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcafc0cdec0 con 0x7fcafc15c9f0 2023-10-16T12:51:00.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafb59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0cd900 con 0x7fcafc0c9230 2023-10-16T12:51:00.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcafc070f20 0x7fcafc0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41536/0 (socket says 172.21.15.60:41536) 2023-10-16T12:51:00.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcafad9d640 1 -- 172.21.15.60:0/199540781 learned_addr learned my addr 172.21.15.60:0/199540781 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:51:00.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.651+0000 7fcaf3fff640 1 --2- 172.21.15.60:0/199540781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:00.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcafa59c640 1 --2- 172.21.15.60:0/199540781 >> v2:172.21.15.144:3300/0 conn(0x7fcafc15c9f0 0x7fcafc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:00.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf9d9b640 1 -- 172.21.15.60:0/199540781 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965556343 0 0) 0x7fcafc160a90 con 0x7fcafc070f20 2023-10-16T12:51:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf9d9b640 1 -- 172.21.15.60:0/199540781 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae8003520 con 0x7fcafc070f20 2023-10-16T12:51:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf3fff640 1 -- 172.21.15.60:0/199540781 >> v2:172.21.15.144:3300/0 conn(0x7fcafc15c9f0 msgr2=0x7fcafc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf3fff640 1 --2- 172.21.15.60:0/199540781 >> v2:172.21.15.144:3300/0 conn(0x7fcafc15c9f0 0x7fcafc0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf3fff640 1 -- 172.21.15.60:0/199540781 >> v1:172.21.15.60:6789/0 conn(0x7fcafc070f20 legacy=0x7fcafc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf9d9b640 1 -- 172.21.15.60:0/199540781 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679323355 0 0) 0x7fcae8003520 con 0x7fcafc070f20 2023-10-16T12:51:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.652+0000 7fcaf3fff640 1 -- 172.21.15.60:0/199540781 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc0cdbe0 con 0x7fcafc0c9230 2023-10-16T12:51:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.654+0000 7fcaf3fff640 1 --2- 172.21.15.60:0/199540781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0709e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcae40097a0 tx=0x7fcae402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b61b003676d0a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.654+0000 7fcaf9d9b640 1 -- 172.21.15.60:0/199540781 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40019f0 con 0x7fcafc0c9230 2023-10-16T12:51:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.654+0000 7fcaf9d9b640 1 -- 172.21.15.60:0/199540781 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae402e050 con 0x7fcafc0c9230 2023-10-16T12:51:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.654+0000 7fcaf9d9b640 1 -- 172.21.15.60:0/199540781 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4030730 con 0x7fcafc0c9230 2023-10-16T12:51:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 -- 172.21.15.60:0/199540781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 msgr2=0x7fcafc0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 --2- 172.21.15.60:0/199540781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0709e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcae40097a0 tx=0x7fcae402b790 comp rx=0 tx=0).stop 2023-10-16T12:51:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 -- 172.21.15.60:0/199540781 shutdown_connections 2023-10-16T12:51:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 --2- 172.21.15.60:0/199540781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0709e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 --2- 172.21.15.60:0/199540781 >> v2:172.21.15.144:3300/0 conn(0x7fcafc15c9f0 0x7fcafc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 -- 172.21.15.60:0/199540781 >> 172.21.15.60:0/199540781 conn(0x7fcafc0ce6a0 msgr2=0x7fcafc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 -- 172.21.15.60:0/199540781 shutdown_connections 2023-10-16T12:51:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.655+0000 7fcafb59e640 1 -- 172.21.15.60:0/199540781 wait complete. 2023-10-16T12:51:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.657+0000 7fcafb59e640 1 Processor -- start 2023-10-16T12:51:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.657+0000 7fcafb59e640 1 -- start start 2023-10-16T12:51:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafb59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 0x7fcafc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafb59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc15c9f0 0x7fcafc159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:00.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafb59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0c6d50 con 0x7fcafc070f20 2023-10-16T12:51:00.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafb59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0c6fa0 con 0x7fcafc15c9f0 2023-10-16T12:51:00.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafb59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0c7280 con 0x7fcafc0c9230 2023-10-16T12:51:00.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 0x7fcafc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafa59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 0x7fcafc157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53978/0 (socket says 172.21.15.60:53978) 2023-10-16T12:51:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.658+0000 7fcafa59c640 1 -- 172.21.15.60:0/683126723 learned_addr learned my addr 172.21.15.60:0/683126723 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcaf3fff640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafad9d640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc15c9f0 0x7fcafc159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafa59c640 1 -- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 msgr2=0x7fcafc0c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafa59c640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafa59c640 1 -- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc15c9f0 msgr2=0x7fcafc159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafa59c640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc15c9f0 0x7fcafc159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafa59c640 1 -- 172.21.15.60:0/683126723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4009400 con 0x7fcafc070f20 2023-10-16T12:51:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcafa59c640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 0x7fcafc157c40 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00e870 tx=0x7fcaec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0094f0 con 0x7fcafc070f20 2023-10-16T12:51:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec00c940 con 0x7fcafc070f20 2023-10-16T12:51:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.659+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec00ccf0 con 0x7fcafc070f20 2023-10-16T12:51:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.660+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc0c7560 con 0x7fcafc070f20 2023-10-16T12:51:00.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.660+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc158180 con 0x7fcafc070f20 2023-10-16T12:51:00.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.661+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc04e9b0 con 0x7fcafc070f20 2023-10-16T12:51:00.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.662+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcaec013470 con 0x7fcafc070f20 2023-10-16T12:51:00.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.663+0000 7fcaf2ffd640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcadc06af90 0x7fcadc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:00.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.663+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcaec089950 con 0x7fcafc070f20 2023-10-16T12:51:00.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.664+0000 7fcaf3fff640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcadc06af90 0x7fcadc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:00.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.665+0000 7fcaf3fff640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcadc06af90 0x7fcadc06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcae4034000 tx=0x7fcae402f000 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:00.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.668+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcaec054850 con 0x7fcafc070f20 2023-10-16T12:51:00.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.890+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 --> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcafc0704c0 con 0x7fcadc06af90 2023-10-16T12:51:00.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.894+0000 7fcaf2ffd640 1 -- 172.21.15.60:0/683126723 <== mgr.5705 v2:172.21.15.144:6832/3513615444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcae4041070 con 0x7fcadc06af90 2023-10-16T12:51:00.895 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:51:00.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:51:00.896 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:51:00.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:51:00.896 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:51:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.896+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcadc06af90 msgr2=0x7fcadc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.896+0000 7fcafb59e640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcadc06af90 0x7fcadc06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcae4034000 tx=0x7fcae402f000 comp rx=0 tx=0).stop 2023-10-16T12:51:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.896+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 msgr2=0x7fcafc157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.896+0000 7fcafb59e640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 0x7fcafc157c40 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00e870 tx=0x7fcaec00ed30 comp rx=0 tx=0).stop 2023-10-16T12:51:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 shutdown_connections 2023-10-16T12:51:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcadc06af90 0x7fcadc06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcafc0c9230 0x7fcafc0c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcafc15c9f0 0x7fcafc159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 --2- 172.21.15.60:0/683126723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcafc070f20 0x7fcafc157c40 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 >> 172.21.15.60:0/683126723 conn(0x7fcafc0ce6a0 msgr2=0x7fcafc0c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 shutdown_connections 2023-10-16T12:51:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:00.897+0000 7fcafb59e640 1 -- 172.21.15.60:0/683126723 wait complete. 2023-10-16T12:51:00.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:51:01.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.149+0000 7f821a3cb640 1 Processor -- start 2023-10-16T12:51:01.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.149+0000 7f821a3cb640 1 -- start start 2023-10-16T12:51:01.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.150+0000 7f821a3cb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f821415c100 0x7f82140ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f821a3cb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82140cd4a0 con 0x7f8214153740 2023-10-16T12:51:01.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f821a3cb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821415ac60 con 0x7f82140ccb00 2023-10-16T12:51:01.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f821a3cb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8214070030 con 0x7f821415c100 2023-10-16T12:51:01.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f82193c9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8214153740 0x7f821415a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41552/0 (socket says 172.21.15.60:41552) 2023-10-16T12:51:01.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f82193c9640 1 -- 172.21.15.60:0/3308245098 learned_addr learned my addr 172.21.15.60:0/3308245098 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:51:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916521920 0 0) 0x7f82140cd4a0 con 0x7f8214153740 2023-10-16T12:51:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8200003520 con 0x7f8214153740 2023-10-16T12:51:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.151+0000 7f8219bca640 1 --2- 172.21.15.60:0/3308245098 >> v2:172.21.15.144:3300/0 conn(0x7f821415c100 0x7f82140ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963427727 0 0) 0x7f8200003520 con 0x7f8214153740 2023-10-16T12:51:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 >> v2:172.21.15.144:3300/0 conn(0x7f821415c100 msgr2=0x7f82140ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 --2- 172.21.15.60:0/3308245098 >> v2:172.21.15.144:3300/0 conn(0x7f821415c100 0x7f82140ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 >> v1:172.21.15.144:6789/0 conn(0x7f82140ccb00 legacy=0x7f82140ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8214070310 con 0x7f8214153740 2023-10-16T12:51:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f8210001670 con 0x7f8214153740 2023-10-16T12:51:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 >> v1:172.21.15.60:6789/0 conn(0x7f8214153740 legacy=0x7f821415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000047c0 0x7f8200004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8200005180 0x7f820000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8214070030 con 0x7f82000039c0 2023-10-16T12:51:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820000c080 con 0x7f82000047c0 2023-10-16T12:51:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820000c240 con 0x7f8200005180 2023-10-16T12:51:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.152+0000 7f82193c9640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f8218bc8640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000047c0 0x7f8200004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f82193c9640 1 -- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8200005180 msgr2=0x7f820000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f8219bca640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8200005180 0x7f820000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f82193c9640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8200005180 0x7f820000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f82193c9640 1 -- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000047c0 msgr2=0x7f8200004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f82193c9640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000047c0 0x7f8200004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f82193c9640 1 -- 172.21.15.60:0/3308245098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200003520 con 0x7f82000039c0 2023-10-16T12:51:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f8219bca640 1 -- 172.21.15.60:0/3308245098 reap_dead start 2023-10-16T12:51:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.153+0000 7f8218bc8640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000047c0 0x7f8200004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:51:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.154+0000 7f82193c9640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82000039c0 0x7f8200003de0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f821000c0e0 tx=0x7f82000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1203c02b206be18f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.154+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210005ad0 con 0x7f82000039c0 2023-10-16T12:51:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.155+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8210005c50 con 0x7f82000039c0 2023-10-16T12:51:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.155+0000 7f820bfff640 1 -- 172.21.15.60:0/3308245098 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210007070 con 0x7f82000039c0 2023-10-16T12:51:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.155+0000 7f821a3cb640 1 -- 172.21.15.60:0/3308245098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82000039c0 msgr2=0x7f8200003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.155+0000 7f821a3cb640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82000039c0 0x7f8200003de0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f821000c0e0 tx=0x7f82000057f0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.156+0000 7f821a3cb640 1 -- 172.21.15.60:0/3308245098 shutdown_connections 2023-10-16T12:51:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.156+0000 7f821a3cb640 1 --2- 172.21.15.60:0/3308245098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82000039c0 0x7f8200003de0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.156+0000 7f821a3cb640 1 -- 172.21.15.60:0/3308245098 >> 172.21.15.60:0/3308245098 conn(0x7f82140ce6a0 msgr2=0x7f82140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.156+0000 7f821a3cb640 1 -- 172.21.15.60:0/3308245098 shutdown_connections 2023-10-16T12:51:01.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.156+0000 7f821a3cb640 1 -- 172.21.15.60:0/3308245098 wait complete. 2023-10-16T12:51:01.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.157+0000 7f821a3cb640 1 Processor -- start 2023-10-16T12:51:01.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.158+0000 7f821a3cb640 1 -- start start 2023-10-16T12:51:01.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f821a3cb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000039c0 0x7f82141f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f821a3cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 0x7f82140b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f821a3cb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8214152830 0x7f82140b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f821a3cb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82140c5540 con 0x7f82141f24f0 2023-10-16T12:51:01.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f821a3cb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141f29d0 con 0x7f82000039c0 2023-10-16T12:51:01.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f821a3cb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82141f2cb0 con 0x7f8214152830 2023-10-16T12:51:01.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 0x7f82140b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 0x7f82140b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53990/0 (socket says 172.21.15.60:53990) 2023-10-16T12:51:01.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 -- 172.21.15.60:0/1900935398 learned_addr learned my addr 172.21.15.60:0/1900935398 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:01.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8219bca640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8214152830 0x7f82140b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 -- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8214152830 msgr2=0x7f82140b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:01.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8214152830 0x7f82140b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 -- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000039c0 msgr2=0x7f82141f1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:01.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000039c0 0x7f82141f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:01.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8218bc8640 1 -- 172.21.15.60:0/1900935398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82141f2f90 con 0x7f82141f24f0 2023-10-16T12:51:01.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.159+0000 7f8219bca640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8214152830 0x7f82140b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:51:01.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.160+0000 7f8218bc8640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 0x7f82140b2e50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f820400d400 tx=0x7f820400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.160+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820400b070 con 0x7f82141f24f0 2023-10-16T12:51:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.160+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040054e0 con 0x7f82141f24f0 2023-10-16T12:51:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.160+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204003850 con 0x7f82141f24f0 2023-10-16T12:51:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.160+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82141f3270 con 0x7f82141f24f0 2023-10-16T12:51:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.160+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8214152d80 con 0x7f82141f24f0 2023-10-16T12:51:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.162+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821406fc30 con 0x7f82141f24f0 2023-10-16T12:51:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.162+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 44) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f82040039b0 con 0x7f82141f24f0 2023-10-16T12:51:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.163+0000 7f8209ffb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f81f806afc0 0x7f81f806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.163+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8204089040 con 0x7f82141f24f0 2023-10-16T12:51:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.164+0000 7f82193c9640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f81f806afc0 0x7f81f806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.165+0000 7f82193c9640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f81f806afc0 0x7f81f806d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8210034000 tx=0x7f8210039040 comp rx=0 tx=0).ready entity=mgr.5705 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:01.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.168+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8204053f80 con 0x7f82141f24f0 2023-10-16T12:51:01.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:01.418+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f821404e9b0 con 0x7f82141f24f0 2023-10-16T12:51:02.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.015+0000 7f8209ffb640 1 -- 172.21.15.60:0/1900935398 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v45) v1 ==== 92+0+0 (secure 0 0 0) 0x7f82040575a0 con 0x7f82141f24f0 2023-10-16T12:51:02.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.020+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f81f806afc0 msgr2=0x7f81f806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.020+0000 7f821a3cb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f81f806afc0 0x7f81f806d480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8210034000 tx=0x7f8210039040 comp rx=0 tx=0).stop 2023-10-16T12:51:02.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.020+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 msgr2=0x7f82140b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.020+0000 7f821a3cb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 0x7f82140b2e50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f820400d400 tx=0x7f820400d8c0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.021+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 shutdown_connections 2023-10-16T12:51:02.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.021+0000 7f821a3cb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f81f806afc0 0x7f81f806d480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.021+0000 7f821a3cb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8214152830 0x7f82140b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.021+0000 7f821a3cb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82141f24f0 0x7f82140b2e50 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.021+0000 7f821a3cb640 1 --2- 172.21.15.60:0/1900935398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82000039c0 0x7f82141f1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.021+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 >> 172.21.15.60:0/1900935398 conn(0x7f82140ce6a0 msgr2=0x7f8214157af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.023+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 shutdown_connections 2023-10-16T12:51:02.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.023+0000 7f821a3cb640 1 -- 172.21.15.60:0/1900935398 wait complete. 2023-10-16T12:51:02.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:51:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.265+0000 7f5e09cde640 1 Processor -- start 2023-10-16T12:51:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.266+0000 7f5e09cde640 1 -- start start 2023-10-16T12:51:02.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.267+0000 7f5e09cde640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5e04065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.267+0000 7f5e09cde640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e04066420 con 0x7f5e04065ff0 2023-10-16T12:51:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.267+0000 7f5e09cde640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0406cee0 con 0x7f5e0415d330 2023-10-16T12:51:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.267+0000 7f5e09cde640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0406b1e0 con 0x7f5e041618b0 2023-10-16T12:51:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.267+0000 7f5e03fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e04065ff0 0x7f5e040ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41574/0 (socket says 172.21.15.60:41574) 2023-10-16T12:51:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.267+0000 7f5e03fff640 1 -- 172.21.15.60:0/876660796 learned_addr learned my addr 172.21.15.60:0/876660796 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:51:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e02ffd640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5e04065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424150272 0 0) 0x7f5e04066420 con 0x7f5e04065ff0 2023-10-16T12:51:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec003520 con 0x7f5e04065ff0 2023-10-16T12:51:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779039195 0 0) 0x7f5dec003520 con 0x7f5e04065ff0 2023-10-16T12:51:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 msgr2=0x7f5e04065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5e04065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 >> v1:172.21.15.144:6789/0 conn(0x7f5e0415d330 legacy=0x7f5e0406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e040d0890 con 0x7f5e04065ff0 2023-10-16T12:51:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5dfc0019f0 con 0x7f5e04065ff0 2023-10-16T12:51:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 >> v1:172.21.15.60:6789/0 conn(0x7f5e04065ff0 legacy=0x7f5e040ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5dec003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dec004350 0x7f5dec004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5dec005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0406b1e0 con 0x7f5dec0039a0 2023-10-16T12:51:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec00ba60 con 0x7f5dec004350 2023-10-16T12:51:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.268+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec00bc20 con 0x7f5e041618b0 2023-10-16T12:51:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e037fe640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5dec003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e02ffd640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dec004350 0x7f5dec004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e03fff640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5dec005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e037fe640 1 -- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 msgr2=0x7f5dec005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e037fe640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5dec005120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e037fe640 1 -- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dec004350 msgr2=0x7f5dec004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e037fe640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5dec004350 0x7f5dec004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e037fe640 1 -- 172.21.15.60:0/876660796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec003520 con 0x7f5dec0039a0 2023-10-16T12:51:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.269+0000 7f5e03fff640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e041618b0 0x7f5dec005120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:51:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.271+0000 7f5e037fe640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5dec003d80 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5df4001860 tx=0x7f5dec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dada04f3d391bb82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.271+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4033630 con 0x7f5dec0039a0 2023-10-16T12:51:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.271+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df40337b0 con 0x7f5dec0039a0 2023-10-16T12:51:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.271+0000 7f5e027fc640 1 -- 172.21.15.60:0/876660796 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4033b60 con 0x7f5dec0039a0 2023-10-16T12:51:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.271+0000 7f5e09cde640 1 -- 172.21.15.60:0/876660796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 msgr2=0x7f5dec003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.271+0000 7f5e09cde640 1 --2- 172.21.15.60:0/876660796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5dec003d80 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5df4001860 tx=0x7f5dec0057f0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.272+0000 7f5e03fff640 1 -- 172.21.15.60:0/876660796 reap_dead start 2023-10-16T12:51:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.272+0000 7f5e09cde640 1 -- 172.21.15.60:0/876660796 shutdown_connections 2023-10-16T12:51:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.272+0000 7f5e09cde640 1 -- 172.21.15.60:0/876660796 >> 172.21.15.60:0/876660796 conn(0x7f5e040d0ab0 msgr2=0x7f5e040d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.272+0000 7f5e09cde640 1 -- 172.21.15.60:0/876660796 shutdown_connections 2023-10-16T12:51:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.272+0000 7f5e09cde640 1 -- 172.21.15.60:0/876660796 wait complete. 2023-10-16T12:51:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.274+0000 7f5e09cde640 1 Processor -- start 2023-10-16T12:51:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.274+0000 7f5e09cde640 1 -- start start 2023-10-16T12:51:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e09cde640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5e041f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e09cde640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e041f3210 0x7f5e040ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e09cde640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e0415f820 0x7f5e0415fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e09cde640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e041f4a70 con 0x7f5dec0039a0 2023-10-16T12:51:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e09cde640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e041601e0 con 0x7f5e041f3210 2023-10-16T12:51:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e09cde640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e041f3df0 con 0x7f5e0415f820 2023-10-16T12:51:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5e041f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5e041f0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54010/0 (socket says 172.21.15.60:54010) 2023-10-16T12:51:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e037fe640 1 -- 172.21.15.60:0/3139039078 learned_addr learned my addr 172.21.15.60:0/3139039078 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e03fff640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e0415f820 0x7f5e0415fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.275+0000 7f5e02ffd640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e041f3210 0x7f5e040ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e037fe640 1 -- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e0415f820 msgr2=0x7f5e0415fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e037fe640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e0415f820 0x7f5e0415fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e037fe640 1 -- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e041f3210 msgr2=0x7f5e040ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e037fe640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e041f3210 0x7f5e040ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e037fe640 1 -- 172.21.15.60:0/3139039078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e041f40d0 con 0x7f5dec0039a0 2023-10-16T12:51:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e037fe640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5e041f0420 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5df4008fd0 tx=0x7f5df4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4007050 con 0x7f5dec0039a0 2023-10-16T12:51:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df40071b0 con 0x7f5dec0039a0 2023-10-16T12:51:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4007520 con 0x7f5dec0039a0 2023-10-16T12:51:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e041f43b0 con 0x7f5dec0039a0 2023-10-16T12:51:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.276+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0415ba90 con 0x7f5dec0039a0 2023-10-16T12:51:02.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.278+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0415c840 con 0x7f5dec0039a0 2023-10-16T12:51:02.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.279+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 45) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5df40067d0 con 0x7f5dec0039a0 2023-10-16T12:51:02.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.280+0000 7f5debfff640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 0x7f5df006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.280+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5df40ab8f0 con 0x7f5dec0039a0 2023-10-16T12:51:02.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.280+0000 7f5e02ffd640 1 -- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 msgr2=0x7f5df006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:02.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.280+0000 7f5e02ffd640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 0x7f5df006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:51:02.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.284+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5df4076770 con 0x7f5dec0039a0 2023-10-16T12:51:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.481+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5e0404e160 con 0x7f5dec0039a0 2023-10-16T12:51:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.481+0000 7f5e02ffd640 1 -- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 msgr2=0x7f5df006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.481+0000 7f5e02ffd640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 0x7f5df006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:51:02.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.485+0000 7f5debfff640 1 -- 172.21.15.60:0/3139039078 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5df402f0d0 con 0x7f5dec0039a0 2023-10-16T12:51:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 msgr2=0x7f5df006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:51:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 0x7f5df006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 msgr2=0x7f5e041f0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5e041f0420 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5df4008fd0 tx=0x7f5df4001230 comp rx=0 tx=0).stop 2023-10-16T12:51:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 shutdown_connections 2023-10-16T12:51:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7f5df006afe0 0x7f5df006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e0415f820 0x7f5e0415fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e041f3210 0x7f5e040ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 --2- 172.21.15.60:0/3139039078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5dec0039a0 0x7f5e041f0420 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 >> 172.21.15.60:0/3139039078 conn(0x7f5e040d0ab0 msgr2=0x7f5e0415e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 shutdown_connections 2023-10-16T12:51:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.488+0000 7f5e09cde640 1 -- 172.21.15.60:0/3139039078 wait complete. 2023-10-16T12:51:02.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:51:02.539 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:02.538+0000 7f4c42e41040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:51:02.566 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:02.565+0000 7f1a41a45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:51:02.592 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:02.591+0000 7f4c42e41040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:51:02.624 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:02.623+0000 7f1a41a45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:51:02.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.711+0000 7fc4fe8e7640 1 Processor -- start 2023-10-16T12:51:02.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.712+0000 7fc4fe8e7640 1 -- start start 2023-10-16T12:51:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.713+0000 7fc4fe8e7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f8070830 0x7fc4f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.713+0000 7fc4fe8e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4f80cf5f0 0x7fc4f806c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.713+0000 7fc4fe8e7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f8070c10 0x7fc4f80cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.713+0000 7fc4fe8e7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f8161740 con 0x7fc4f80cf5f0 2023-10-16T12:51:02.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.713+0000 7fc4fe8e7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f806c680 con 0x7fc4f8070c10 2023-10-16T12:51:02.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.713+0000 7fc4fe8e7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80cdbd0 con 0x7fc4f8070830 2023-10-16T12:51:02.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4f80cf5f0 0x7fc4f806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4f80cf5f0 0x7fc4f806c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54024/0 (socket says 172.21.15.60:54024) 2023-10-16T12:51:02.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 -- 172.21.15.60:0/3056126396 learned_addr learned my addr 172.21.15.60:0/3056126396 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:02.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4f7fff640 1 --2- 172.21.15.60:0/3056126396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f8070830 0x7fc4f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4fce84640 1 --2- 172.21.15.60:0/3056126396 >> v2:172.21.15.144:3300/0 conn(0x7fc4f8070c10 0x7fc4f80cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 -- 172.21.15.60:0/3056126396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f8070830 msgr2=0x7fc4f806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:02.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 --2- 172.21.15.60:0/3056126396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f8070830 0x7fc4f806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 -- 172.21.15.60:0/3056126396 >> v2:172.21.15.144:3300/0 conn(0x7fc4f8070c10 msgr2=0x7fc4f80cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:02.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 --2- 172.21.15.60:0/3056126396 >> v2:172.21.15.144:3300/0 conn(0x7fc4f8070c10 0x7fc4f80cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4effff640 1 -- 172.21.15.60:0/3056126396 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f806b0f0 con 0x7fc4f80cf5f0 2023-10-16T12:51:02.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.714+0000 7fc4fce84640 1 --2- 172.21.15.60:0/3056126396 >> v2:172.21.15.144:3300/0 conn(0x7fc4f8070c10 0x7fc4f80cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:51:02.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.716+0000 7fc4effff640 1 --2- 172.21.15.60:0/3056126396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f80cf5f0 0x7fc4f806c140 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc4f806d900 tx=0x7fc4e002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33144dac27d616d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:02.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.716+0000 7fc4f77fe640 1 -- 172.21.15.60:0/3056126396 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0031750 con 0x7fc4f80cf5f0 2023-10-16T12:51:02.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.716+0000 7fc4f77fe640 1 -- 172.21.15.60:0/3056126396 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e00318b0 con 0x7fc4f80cf5f0 2023-10-16T12:51:02.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.716+0000 7fc4f77fe640 1 -- 172.21.15.60:0/3056126396 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0031c40 con 0x7fc4f80cf5f0 2023-10-16T12:51:02.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/3056126396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f80cf5f0 msgr2=0x7fc4f806c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/3056126396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f80cf5f0 0x7fc4f806c140 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc4f806d900 tx=0x7fc4e002b7f0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/3056126396 shutdown_connections 2023-10-16T12:51:02.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/3056126396 >> v2:172.21.15.144:3300/0 conn(0x7fc4f8070c10 0x7fc4f80cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/3056126396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f80cf5f0 0x7fc4f806c140 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/3056126396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f8070830 0x7fc4f806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/3056126396 >> 172.21.15.60:0/3056126396 conn(0x7fc4f80cfac0 msgr2=0x7fc4f80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:02.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/3056126396 shutdown_connections 2023-10-16T12:51:02.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.717+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/3056126396 wait complete. 2023-10-16T12:51:02.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.719+0000 7fc4fe8e7640 1 Processor -- start 2023-10-16T12:51:02.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.719+0000 7fc4fe8e7640 1 -- start start 2023-10-16T12:51:02.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.719+0000 7fc4fe8e7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f8070830 0x7fc4f80ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4fe8e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 0x7fc4f80cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4fe8e7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f80cf5f0 0x7fc4f80b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4fe8e7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80ca2e0 con 0x7fc4f8070c10 2023-10-16T12:51:02.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4fe8e7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80c96d0 con 0x7fc4f8070830 2023-10-16T12:51:02.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4fe8e7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80c99b0 con 0x7fc4f80cf5f0 2023-10-16T12:51:02.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 0x7fc4f80cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 0x7fc4f80cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54038/0 (socket says 172.21.15.60:54038) 2023-10-16T12:51:02.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 -- 172.21.15.60:0/1159321628 learned_addr learned my addr 172.21.15.60:0/1159321628 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:02.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4f7fff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f8070830 0x7fc4f80ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4fce84640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f80cf5f0 0x7fc4f80b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:02.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 -- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f80cf5f0 msgr2=0x7fc4f80b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f80cf5f0 0x7fc4f80b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 -- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f8070830 msgr2=0x7fc4f80ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f8070830 0x7fc4f80ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4effff640 1 -- 172.21.15.60:0/1159321628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e00094d0 con 0x7fc4f8070c10 2023-10-16T12:51:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.720+0000 7fc4f7fff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f8070830 0x7fc4f80ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:51:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.721+0000 7fc4effff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 0x7fc4f80cacb0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0000c00 tx=0x7fc4e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.721+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0033c80 con 0x7fc4f8070c10 2023-10-16T12:51:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.721+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e0033e00 con 0x7fc4f8070c10 2023-10-16T12:51:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.721+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e003c640 con 0x7fc4f8070c10 2023-10-16T12:51:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.721+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f80c9c90 con 0x7fc4f8070c10 2023-10-16T12:51:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.721+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f8151320 con 0x7fc4f8070c10 2023-10-16T12:51:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.723+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f8151570 con 0x7fc4f8070c10 2023-10-16T12:51:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.724+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 45) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc4e003a040 con 0x7fc4f8070c10 2023-10-16T12:51:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.725+0000 7fc4f57fa640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 0x7fc4d806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.725+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4e00ab360 con 0x7fc4f8070c10 2023-10-16T12:51:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.725+0000 7fc4f7fff640 1 -- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 msgr2=0x7fc4d806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.725+0000 7fc4f7fff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 0x7fc4d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:51:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.730+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4e00762f0 con 0x7fc4f8070c10 2023-10-16T12:51:02.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.923+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc4f804e160 con 0x7fc4f8070c10 2023-10-16T12:51:02.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.925+0000 7fc4f7fff640 1 -- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 msgr2=0x7fc4d806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:02.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.925+0000 7fc4f7fff640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 0x7fc4d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:51:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.928+0000 7fc4f57fa640 1 -- 172.21.15.60:0/1159321628 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc4e0079910 con 0x7fc4f8070c10 2023-10-16T12:51:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.930+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 msgr2=0x7fc4d806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:51:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.930+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 0x7fc4d806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.930+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 msgr2=0x7fc4f80cacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.930+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 0x7fc4f80cacb0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0000c00 tx=0x7fc4e00046c0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 shutdown_connections 2023-10-16T12:51:02.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fc4d806afe0 0x7fc4d806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4f80cf5f0 0x7fc4f80b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4f8070c10 0x7fc4f80cacb0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 --2- 172.21.15.60:0/1159321628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f8070830 0x7fc4f80ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:02.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 >> 172.21.15.60:0/1159321628 conn(0x7fc4f80cfac0 msgr2=0x7fc4f80c6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:02.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 shutdown_connections 2023-10-16T12:51:02.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:02.931+0000 7fc4fe8e7640 1 -- 172.21.15.60:0/1159321628 wait complete. 2023-10-16T12:51:02.952 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-10-16T12:51:03.042 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.040+0000 7f4c42e41040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:51:03.094 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.092+0000 7f4c42e41040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:51:03.106 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.104+0000 7f1a41a45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:51:03.143 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.142+0000 7f4c42e41040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:51:03.161 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.159+0000 7f1a41a45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:51:03.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.169+0000 7fcfc5c83640 1 Processor -- start 2023-10-16T12:51:03.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.169+0000 7fcfc5c83640 1 -- start start 2023-10-16T12:51:03.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5c83640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcfc015cd70 0x7fcfc00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:03.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5c83640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcfc00b8480 0x7fcfc00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:03.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5c83640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc015d150 con 0x7fcfc00d0be0 2023-10-16T12:51:03.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5c83640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00d1180 con 0x7fcfc00b8480 2023-10-16T12:51:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5c83640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00d1460 con 0x7fcfc015cd70 2023-10-16T12:51:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5482640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcfc00d0be0 0x7fcfc00b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41580/0 (socket says 172.21.15.60:41580) 2023-10-16T12:51:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.171+0000 7fcfc5482640 1 -- 172.21.15.60:0/2856116320 learned_addr learned my addr 172.21.15.60:0/2856116320 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:51:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2856116320 >> v2:172.21.15.144:3300/0 conn(0x7fcfc015cd70 0x7fcfc00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbffff640 1 --2- 172.21.15.60:0/2856116320 >> v2:172.21.15.60:3300/0 conn(0x7fcfc00b8480 0x7fcfc00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:03.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbf7fe640 1 -- 172.21.15.60:0/2856116320 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811152582 0 0) 0x7fcfc015d150 con 0x7fcfc00d0be0 2023-10-16T12:51:03.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbf7fe640 1 -- 172.21.15.60:0/2856116320 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb0003520 con 0x7fcfc00d0be0 2023-10-16T12:51:03.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbffff640 1 -- 172.21.15.60:0/2856116320 >> v2:172.21.15.144:3300/0 conn(0x7fcfc015cd70 msgr2=0x7fcfc00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:03.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbffff640 1 --2- 172.21.15.60:0/2856116320 >> v2:172.21.15.144:3300/0 conn(0x7fcfc015cd70 0x7fcfc00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:03.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbffff640 1 -- 172.21.15.60:0/2856116320 >> v1:172.21.15.60:6789/0 conn(0x7fcfc00d0be0 legacy=0x7fcfc00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:03.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.172+0000 7fcfbffff640 1 -- 172.21.15.60:0/2856116320 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00d1740 con 0x7fcfc00b8480 2023-10-16T12:51:03.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.173+0000 7fcfbffff640 1 --2- 172.21.15.60:0/2856116320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc00b8480 0x7fcfc00cd7c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00b56a0 tx=0x7fcfac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6359d0622d9a107b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:03.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfbf7fe640 1 -- 172.21.15.60:0/2856116320 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac02e070 con 0x7fcfc00b8480 2023-10-16T12:51:03.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfbf7fe640 1 -- 172.21.15.60:0/2856116320 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfac0315a0 con 0x7fcfc00b8480 2023-10-16T12:51:03.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfbf7fe640 1 -- 172.21.15.60:0/2856116320 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac031930 con 0x7fcfc00b8480 2023-10-16T12:51:03.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2856116320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc00b8480 msgr2=0x7fcfc00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:03.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfc5c83640 1 --2- 172.21.15.60:0/2856116320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc00b8480 0x7fcfc00cd7c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00b56a0 tx=0x7fcfac02b840 comp rx=0 tx=0).stop 2023-10-16T12:51:03.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2856116320 shutdown_connections 2023-10-16T12:51:03.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfc5c83640 1 --2- 172.21.15.60:0/2856116320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc00b8480 0x7fcfc00cd7c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:03.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfc5c83640 1 --2- 172.21.15.60:0/2856116320 >> v2:172.21.15.144:3300/0 conn(0x7fcfc015cd70 0x7fcfc00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:03.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.174+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2856116320 >> 172.21.15.60:0/2856116320 conn(0x7fcfc00cf570 msgr2=0x7fcfc00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:03.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.175+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2856116320 shutdown_connections 2023-10-16T12:51:03.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.175+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2856116320 wait complete. 2023-10-16T12:51:03.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.176+0000 7fcfc5c83640 1 Processor -- start 2023-10-16T12:51:03.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.176+0000 7fcfc5c83640 1 -- start start 2023-10-16T12:51:03.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.177+0000 7fcfc5c83640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc00b8480 0x7fcfc0151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:03.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.177+0000 7fcfc5c83640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc00d0be0 0x7fcfc0159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:03.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.177+0000 7fcfc5c83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc015cd70 0x7fcfc015b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:03.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.177+0000 7fcfc5c83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc0152fe0 con 0x7fcfc015cd70 2023-10-16T12:51:03.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.177+0000 7fcfc5c83640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01532c0 con 0x7fcfc00d0be0 2023-10-16T12:51:03.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.177+0000 7fcfc5c83640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c9d60 con 0x7fcfc00b8480 2023-10-16T12:51:03.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc015cd70 0x7fcfc015b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:03.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc015cd70 0x7fcfc015b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54048/0 (socket says 172.21.15.60:54048) 2023-10-16T12:51:03.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 -- 172.21.15.60:0/2591022150 learned_addr learned my addr 172.21.15.60:0/2591022150 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:03.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc00b8480 0x7fcfc0151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc00b8480 msgr2=0x7fcfc0151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfc00b8480 0x7fcfc0151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc00d0be0 msgr2=0x7fcfc0159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfbffff640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc00d0be0 0x7fcfc0159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfc00d0be0 0x7fcfc0159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:03.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 -- 172.21.15.60:0/2591022150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac009400 con 0x7fcfc015cd70 2023-10-16T12:51:03.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.178+0000 7fcfc5482640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc015cd70 0x7fcfc015b4d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800caa0 tx=0x7fcfb800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:03.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.179+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb800ac00 con 0x7fcfc015cd70 2023-10-16T12:51:03.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.179+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb800ad60 con 0x7fcfc015cd70 2023-10-16T12:51:03.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.179+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc00ca040 con 0x7fcfc015cd70 2023-10-16T12:51:03.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.179+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8012620 con 0x7fcfc015cd70 2023-10-16T12:51:03.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.179+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc0152b70 con 0x7fcfc015cd70 2023-10-16T12:51:03.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.180+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc00c9bc0 con 0x7fcfc015cd70 2023-10-16T12:51:03.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.182+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 45) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fcfb8007070 con 0x7fcfc015cd70 2023-10-16T12:51:03.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.183+0000 7fcfbd7fa640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 0x7fcfa406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:03.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.183+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfb8089880 con 0x7fcfc015cd70 2023-10-16T12:51:03.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.183+0000 7fcfc4c81640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 msgr2=0x7fcfa406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:03.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.183+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 0x7fcfa406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:51:03.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.187+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcfc00c9bc0 con 0x7fcfc015cd70 2023-10-16T12:51:03.205 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.203+0000 7f4c42e41040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:51:03.217 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.215+0000 7f1a41a45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:51:03.254 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.252+0000 7f4c42e41040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:51:03.282 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.280+0000 7f1a41a45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:51:03.334 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.331+0000 7f1a41a45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:51:03.364 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.362+0000 7f4c42e41040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:51:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.384+0000 7fcfc4c81640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 msgr2=0x7fcfa406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:03.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.384+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 0x7fcfa406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:51:03.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.405+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 --> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfc00d1d00 con 0x7fcfa406afe0 2023-10-16T12:51:03.426 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.425+0000 7f4c42e41040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:51:03.453 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.449+0000 7f1a41a45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:51:03.475 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.474+0000 7f4c42e41040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:51:03.517 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.514+0000 7f1a41a45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:51:03.533 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.531+0000 7f4c42e41040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:51:03.568 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.566+0000 7f1a41a45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:51:03.583 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.581+0000 7f4c42e41040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:51:03.628 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.625+0000 7f1a41a45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:51:03.680 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:03.678+0000 7f1a41a45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:51:03.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.785+0000 7fcfc4c81640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 msgr2=0x7fcfa406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:03.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:03.785+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 0x7fcfa406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:51:03.962 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:03.960+0000 7f4c42e41040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:51:04.082 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.079+0000 7f1a41a45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:51:04.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.120+0000 7f4c42e41040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:51:04.230 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.228+0000 7f4c42e41040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:51:04.249 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.247+0000 7f1a41a45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:51:04.280 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.278+0000 7f4c42e41040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:51:04.363 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.360+0000 7f1a41a45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:51:04.387 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.385+0000 7f4c42e41040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:51:04.416 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.413+0000 7f1a41a45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:51:04.493 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.492+0000 7f4c42e41040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:51:04.529 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.526+0000 7f1a41a45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:51:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:04.586+0000 7fcfc4c81640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 msgr2=0x7fcfa406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3513615444 2023-10-16T12:51:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:04.586+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 0x7fcfa406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:51:04.641 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.639+0000 7f1a41a45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:51:04.650 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.648+0000 7f4c42e41040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:51:04.699 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.698+0000 7f4c42e41040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:51:04.807 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.804+0000 7f1a41a45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:51:04.818 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:04.816+0000 7f4c42e41040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:51:04.859 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.857+0000 7f1a41a45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:51:04.986 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:04.983+0000 7f1a41a45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:51:05.637 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:05.635+0000 7f4c42e41040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:51:05.865 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:05.862+0000 7f1a41a45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:51:05.930 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:05.928+0000 7f4c42e41040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:51:05.986 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:05.985+0000 7f4c42e41040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:51:06.099 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.098+0000 7f4c42e41040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:51:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:06.115+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 46) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fcfb8054b50 con 0x7fcfc015cd70 2023-10-16T12:51:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:06.115+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 msgr2=0x7fcfa406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:51:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:06.115+0000 7fcfbd7fa640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/3513615444,v1:172.21.15.144:6833/3513615444] conn(0x7fcfa406afe0 0x7fcfa406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:06.191 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:06.189+0000 7f1a41a45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:51:06.258 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:06.255+0000 7f1a41a45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:51:06.392 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:51:06.389+0000 7f1a41a45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:51:06.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bc21ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bc21ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bc21ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bc21ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bc21ae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bbf969640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bbf969640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bbf969640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.448 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bbf969640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:06.448 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:51:06.444+0000 7f4bbf969640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:51:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.129+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 47) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fcfb8075d50 con 0x7fcfc015cd70 2023-10-16T12:51:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.130+0000 7fcfbd7fa640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fcfa40358e0 0x7fcfa4074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.130+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 --> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfa4035cc0 con 0x7fcfa40358e0 2023-10-16T12:51:07.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.136+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fcfa40358e0 0x7fcfa4074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.145+0000 7fcfc4c81640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fcfa40358e0 0x7fcfa4074ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009500 tx=0x7fcfb4009190 comp rx=0 tx=0).ready entity=mgr.6487 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:07.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.338+0000 7fcfbd7fa640 1 -- 172.21.15.60:0/2591022150 <== mgr.6487 v2:172.21.15.144:6832/2022371982 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fcfa4035cc0 con 0x7fcfa40358e0 2023-10-16T12:51:07.341 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /bad-path 2023-10-16T12:51:07.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.342+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fcfa40358e0 msgr2=0x7fcfa4074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.342+0000 7fcfc5c83640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fcfa40358e0 0x7fcfa4074ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009500 tx=0x7fcfb4009190 comp rx=0 tx=0).stop 2023-10-16T12:51:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.342+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc015cd70 msgr2=0x7fcfc015b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.342+0000 7fcfc5c83640 1 --2- 172.21.15.60:0/2591022150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfc015cd70 0x7fcfc015b4d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800caa0 tx=0x7fcfb800cf60 comp rx=0 tx=0).stop 2023-10-16T12:51:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.342+0000 7fcfc5482640 1 -- 172.21.15.60:0/2591022150 reap_dead start 2023-10-16T12:51:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.343+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 shutdown_connections 2023-10-16T12:51:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.343+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 >> 172.21.15.60:0/2591022150 conn(0x7fcfc00cf570 msgr2=0x7fcfc00b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.343+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 shutdown_connections 2023-10-16T12:51:07.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.343+0000 7fcfc5c83640 1 -- 172.21.15.60:0/2591022150 wait complete. 2023-10-16T12:51:07.368 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-16T12:51:07.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T12:51:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.596+0000 7ffa74abb640 1 Processor -- start 2023-10-16T12:51:07.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.596+0000 7ffa74abb640 1 -- start start 2023-10-16T12:51:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa74abb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa700cd800 0x7ffa700cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa74abb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa7006aeb0 0x7ffa7006b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa74abb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700ce180 con 0x7ffa7015cba0 2023-10-16T12:51:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa74abb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa700d0500 con 0x7ffa7006aeb0 2023-10-16T12:51:07.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa74abb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa700d07c0 con 0x7ffa700cd800 2023-10-16T12:51:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa6ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa7006aeb0 0x7ffa7006b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa6ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa7006aeb0 0x7ffa7006b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54050/0 (socket says 172.21.15.60:54050) 2023-10-16T12:51:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.598+0000 7ffa6ffff640 1 -- 172.21.15.60:0/3060041263 learned_addr learned my addr 172.21.15.60:0/3060041263 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.599+0000 7ffa6ffff640 1 -- 172.21.15.60:0/3060041263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa700cd800 msgr2=0x7ffa700cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.599+0000 7ffa6ffff640 1 --2- 172.21.15.60:0/3060041263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa700cd800 0x7ffa700cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.599+0000 7ffa6ffff640 1 -- 172.21.15.60:0/3060041263 >> v1:172.21.15.144:6789/0 conn(0x7ffa7015cba0 legacy=0x7ffa70071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:51:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.599+0000 7ffa6ffff640 1 -- 172.21.15.60:0/3060041263 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa700d0aa0 con 0x7ffa7006aeb0 2023-10-16T12:51:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.600+0000 7ffa6ffff640 1 --2- 172.21.15.60:0/3060041263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa7006aeb0 0x7ffa7006b340 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009e50 tx=0x7ffa6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad04288d242c78a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:07.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.600+0000 7ffa6e7fc640 1 -- 172.21.15.60:0/3060041263 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6802ca70 con 0x7ffa7006aeb0 2023-10-16T12:51:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.600+0000 7ffa6e7fc640 1 -- 172.21.15.60:0/3060041263 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6802cbd0 con 0x7ffa7006aeb0 2023-10-16T12:51:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.600+0000 7ffa6e7fc640 1 -- 172.21.15.60:0/3060041263 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa680343f0 con 0x7ffa7006aeb0 2023-10-16T12:51:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 -- 172.21.15.60:0/3060041263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa7006aeb0 msgr2=0x7ffa7006b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 --2- 172.21.15.60:0/3060041263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa7006aeb0 0x7ffa7006b340 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009e50 tx=0x7ffa6802bed0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 -- 172.21.15.60:0/3060041263 shutdown_connections 2023-10-16T12:51:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 --2- 172.21.15.60:0/3060041263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa7006aeb0 0x7ffa7006b340 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 --2- 172.21.15.60:0/3060041263 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa700cd800 0x7ffa700cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 -- 172.21.15.60:0/3060041263 >> 172.21.15.60:0/3060041263 conn(0x7ffa70065ed0 msgr2=0x7ffa700662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 -- 172.21.15.60:0/3060041263 shutdown_connections 2023-10-16T12:51:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.601+0000 7ffa74abb640 1 -- 172.21.15.60:0/3060041263 wait complete. 2023-10-16T12:51:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.603+0000 7ffa74abb640 1 Processor -- start 2023-10-16T12:51:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.603+0000 7ffa74abb640 1 -- start start 2023-10-16T12:51:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa74abb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7006aeb0 0x7ffa7015d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa74abb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 0x7ffa70160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa74abb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa7015cba0 0x7ffa7014ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa74abb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa701605a0 con 0x7ffa700cd800 2023-10-16T12:51:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa74abb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7014f230 con 0x7ffa7015cba0 2023-10-16T12:51:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa74abb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7015e090 con 0x7ffa7006aeb0 2023-10-16T12:51:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa6effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 0x7ffa70160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa6effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 0x7ffa70160060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54062/0 (socket says 172.21.15.60:54062) 2023-10-16T12:51:07.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.604+0000 7ffa6effd640 1 -- 172.21.15.60:0/413166080 learned_addr learned my addr 172.21.15.60:0/413166080 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:51:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6f7fe640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7006aeb0 0x7ffa7015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6effd640 1 -- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7006aeb0 msgr2=0x7ffa7015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6ffff640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa7015cba0 0x7ffa7014ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6effd640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7006aeb0 0x7ffa7015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6effd640 1 -- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa7015cba0 msgr2=0x7ffa7014ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6effd640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa7015cba0 0x7ffa7014ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6effd640 1 -- 172.21.15.60:0/413166080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa68009b30 con 0x7ffa700cd800 2023-10-16T12:51:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6f7fe640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7006aeb0 0x7ffa7015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:51:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa6effd640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 0x7ffa70160060 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400b4a0 tx=0x7ffa6400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:07.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64010610 con 0x7ffa700cd800 2023-10-16T12:51:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64010770 con 0x7ffa700cd800 2023-10-16T12:51:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64010b20 con 0x7ffa700cd800 2023-10-16T12:51:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7015e370 con 0x7ffa700cd800 2023-10-16T12:51:07.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.605+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7015e930 con 0x7ffa700cd800 2023-10-16T12:51:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.607+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7004e9b0 con 0x7ffa700cd800 2023-10-16T12:51:07.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.607+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 47) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ffa64010c80 con 0x7ffa700cd800 2023-10-16T12:51:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.608+0000 7ffa57fff640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ffa5c06ad10 0x7ffa5c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:51:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.609+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ffa64089130 con 0x7ffa700cd800 2023-10-16T12:51:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.609+0000 7ffa6f7fe640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ffa5c06ad10 0x7ffa5c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:51:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.610+0000 7ffa6f7fe640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ffa5c06ad10 0x7ffa5c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000a830 tx=0x7ffa60008040 comp rx=0 tx=0).ready entity=mgr.6487 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:51:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.613+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa640542f0 con 0x7ffa700cd800 2023-10-16T12:51:07.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.832+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 --> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa700d1060 con 0x7ffa5c06ad10 2023-10-16T12:51:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.848+0000 7ffa57fff640 1 -- 172.21.15.60:0/413166080 <== mgr.6487 v2:172.21.15.144:6832/2022371982 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7ffa6000adb0 con 0x7ffa5c06ad10 2023-10-16T12:51:07.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:51:07.851 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:51:07", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T12:51:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ffa5c06ad10 msgr2=0x7ffa5c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ffa5c06ad10 0x7ffa5c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000a830 tx=0x7ffa60008040 comp rx=0 tx=0).stop 2023-10-16T12:51:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 msgr2=0x7ffa70160060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:51:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 0x7ffa70160060 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400b4a0 tx=0x7ffa6400b960 comp rx=0 tx=0).stop 2023-10-16T12:51:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 shutdown_connections 2023-10-16T12:51:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa7015cba0 0x7ffa7014ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa700cd800 0x7ffa70160060 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ffa5c06ad10 0x7ffa5c06d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 --2- 172.21.15.60:0/413166080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7006aeb0 0x7ffa7015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:51:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 >> 172.21.15.60:0/413166080 conn(0x7ffa70065ed0 msgr2=0x7ffa700662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:51:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 shutdown_connections 2023-10-16T12:51:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:51:07.851+0000 7ffa74abb640 1 -- 172.21.15.60:0/413166080 wait complete. 2023-10-16T12:51:07.875 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:51:07", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T12:51:27.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:27.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:57.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:57.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:00.365 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:00.363+0000 7f4bb214e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:00.365 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:00.363+0000 7f4bb214e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:00.365 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:00.363+0000 7f4bb214e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:00.366 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:00.363+0000 7f4bb214e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:00.366 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:00.363+0000 7f4bb214e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:07.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T12:52:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.114+0000 7fe35dc83640 1 Processor -- start 2023-10-16T12:52:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.114+0000 7fe35dc83640 1 -- start start 2023-10-16T12:52:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.115+0000 7fe35dc83640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe35806b660 0x7fe35806ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe35dc83640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3581428c0 0x7fe3580c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe35dc83640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580c9ae0 con 0x7fe3580c9690 2023-10-16T12:52:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe35dc83640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3580cc6b0 con 0x7fe3581428c0 2023-10-16T12:52:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe35dc83640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3581599b0 con 0x7fe35806b660 2023-10-16T12:52:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe356ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3581428c0 0x7fe3580c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe356ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3581428c0 0x7fe3580c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59802/0 (socket says 172.21.15.60:59802) 2023-10-16T12:52:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe356ffd640 1 -- 172.21.15.60:0/3232595945 learned_addr learned my addr 172.21.15.60:0/3232595945 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.116+0000 7fe3577fe640 1 --2- 172.21.15.60:0/3232595945 >> v2:172.21.15.144:3300/0 conn(0x7fe35806b660 0x7fe35806ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.117+0000 7fe356ffd640 1 -- 172.21.15.60:0/3232595945 >> v2:172.21.15.144:3300/0 conn(0x7fe35806b660 msgr2=0x7fe35806ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.117+0000 7fe356ffd640 1 --2- 172.21.15.60:0/3232595945 >> v2:172.21.15.144:3300/0 conn(0x7fe35806b660 0x7fe35806ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.117+0000 7fe356ffd640 1 -- 172.21.15.60:0/3232595945 >> v1:172.21.15.144:6789/0 conn(0x7fe3580c9690 legacy=0x7fe35815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.117+0000 7fe356ffd640 1 -- 172.21.15.60:0/3232595945 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35806c290 con 0x7fe3581428c0 2023-10-16T12:52:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.118+0000 7fe356ffd640 1 --2- 172.21.15.60:0/3232595945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3581428c0 0x7fe3580c9150 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fe3580c9120 tx=0x7fe34002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea60613d91a256d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.118+0000 7fe3567fc640 1 -- 172.21.15.60:0/3232595945 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340030860 con 0x7fe3581428c0 2023-10-16T12:52:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.118+0000 7fe3567fc640 1 -- 172.21.15.60:0/3232595945 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3400309c0 con 0x7fe3581428c0 2023-10-16T12:52:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.118+0000 7fe3567fc640 1 -- 172.21.15.60:0/3232595945 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340030d50 con 0x7fe3581428c0 2023-10-16T12:52:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 -- 172.21.15.60:0/3232595945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3581428c0 msgr2=0x7fe3580c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 --2- 172.21.15.60:0/3232595945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3581428c0 0x7fe3580c9150 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fe3580c9120 tx=0x7fe34002b9a0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 -- 172.21.15.60:0/3232595945 shutdown_connections 2023-10-16T12:52:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 --2- 172.21.15.60:0/3232595945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3581428c0 0x7fe3580c9150 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 --2- 172.21.15.60:0/3232595945 >> v2:172.21.15.144:3300/0 conn(0x7fe35806b660 0x7fe35806ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 -- 172.21.15.60:0/3232595945 >> 172.21.15.60:0/3232595945 conn(0x7fe3580ce9b0 msgr2=0x7fe3580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:08.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 -- 172.21.15.60:0/3232595945 shutdown_connections 2023-10-16T12:52:08.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.119+0000 7fe35dc83640 1 -- 172.21.15.60:0/3232595945 wait complete. 2023-10-16T12:52:08.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.121+0000 7fe35dc83640 1 Processor -- start 2023-10-16T12:52:08.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.121+0000 7fe35dc83640 1 -- start start 2023-10-16T12:52:08.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe35dc83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 0x7fe3580c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe35dc83640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3580c9690 0x7fe3580c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe35dc83640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3581428c0 0x7fe358153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe35dc83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358159680 con 0x7fe35806b660 2023-10-16T12:52:08.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe35dc83640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358158a70 con 0x7fe3581428c0 2023-10-16T12:52:08.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe35dc83640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe358158d50 con 0x7fe3580c9690 2023-10-16T12:52:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe3577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 0x7fe3580c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe3577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 0x7fe3580c4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59810/0 (socket says 172.21.15.60:59810) 2023-10-16T12:52:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.122+0000 7fe3577fe640 1 -- 172.21.15.60:0/2655201108 learned_addr learned my addr 172.21.15.60:0/2655201108 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe356ffd640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3580c9690 0x7fe3580c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe357fff640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3581428c0 0x7fe358153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe3577fe640 1 -- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3580c9690 msgr2=0x7fe3580c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe3577fe640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3580c9690 0x7fe3580c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe3577fe640 1 -- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3581428c0 msgr2=0x7fe358153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe3577fe640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3581428c0 0x7fe358153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe3577fe640 1 -- 172.21.15.60:0/2655201108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400094d0 con 0x7fe35806b660 2023-10-16T12:52:08.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe356ffd640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3580c9690 0x7fe3580c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:08.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe357fff640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3581428c0 0x7fe358153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:08.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe3577fe640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 0x7fe3580c4800 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fe348007e40 tx=0x7fe34800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:08.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348014620 con 0x7fe35806b660 2023-10-16T12:52:08.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34800ee90 con 0x7fe35806b660 2023-10-16T12:52:08.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480149c0 con 0x7fe35806b660 2023-10-16T12:52:08.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.123+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe358159030 con 0x7fe35806b660 2023-10-16T12:52:08.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.124+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe358154120 con 0x7fe35806b660 2023-10-16T12:52:08.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.125+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35804e160 con 0x7fe35806b660 2023-10-16T12:52:08.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.126+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 50) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe348014b20 con 0x7fe35806b660 2023-10-16T12:52:08.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.127+0000 7fe34ffff640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fe33806b100 0x7fe33806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.127+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fe34808a4b0 con 0x7fe35806b660 2023-10-16T12:52:08.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.128+0000 7fe356ffd640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fe33806b100 0x7fe33806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.129+0000 7fe356ffd640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fe33806b100 0x7fe33806d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe340036000 tx=0x7fe34003b040 comp rx=0 tx=0).ready entity=mgr.6487 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.132+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe348055140 con 0x7fe35806b660 2023-10-16T12:52:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.350+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 --> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe35806c850 con 0x7fe33806b100 2023-10-16T12:52:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.371+0000 7fe34ffff640 1 -- 172.21.15.60:0/2655201108 <== mgr.6487 v2:172.21.15.144:6832/2022371982 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7fe340030c30 con 0x7fe33806b100 2023-10-16T12:52:08.373 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:08.374 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:51:07", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T12:52:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.373+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fe33806b100 msgr2=0x7fe33806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.373+0000 7fe35dc83640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fe33806b100 0x7fe33806d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe340036000 tx=0x7fe34003b040 comp rx=0 tx=0).stop 2023-10-16T12:52:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.373+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 msgr2=0x7fe3580c4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.373+0000 7fe35dc83640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 0x7fe3580c4800 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fe348007e40 tx=0x7fe34800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 shutdown_connections 2023-10-16T12:52:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fe33806b100 0x7fe33806d5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3580c9690 0x7fe3580c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3581428c0 0x7fe358153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 --2- 172.21.15.60:0/2655201108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe35806b660 0x7fe3580c4800 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 >> 172.21.15.60:0/2655201108 conn(0x7fe3580ce9b0 msgr2=0x7fe3580ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 shutdown_connections 2023-10-16T12:52:08.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.374+0000 7fe35dc83640 1 -- 172.21.15.60:0/2655201108 wait complete. 2023-10-16T12:52:08.394 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:51:07", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T12:52:08.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-10-16T12:52:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.610+0000 7fd3264b4640 1 Processor -- start 2023-10-16T12:52:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.610+0000 7fd3264b4640 1 -- start start 2023-10-16T12:52:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd3264b4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3200ad970 0x7fd3200cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd3264b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32015b9d0 con 0x7fd32006c040 2023-10-16T12:52:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd3264b4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32015bcb0 con 0x7fd3200ccfb0 2023-10-16T12:52:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd3264b4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32015bf90 con 0x7fd3200ad970 2023-10-16T12:52:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd3254b2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd32006c040 0x7fd32006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43186/0 (socket says 172.21.15.60:43186) 2023-10-16T12:52:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd3254b2640 1 -- 172.21.15.60:0/3122361277 learned_addr learned my addr 172.21.15.60:0/3122361277 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.612+0000 7fd324cb1640 1 --2- 172.21.15.60:0/3122361277 >> v2:172.21.15.60:3300/0 conn(0x7fd3200ad970 0x7fd3200cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.613+0000 7fd317fff640 1 -- 172.21.15.60:0/3122361277 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003335518 0 0) 0x7fd32015b9d0 con 0x7fd32006c040 2023-10-16T12:52:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.613+0000 7fd317fff640 1 -- 172.21.15.60:0/3122361277 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd30c003520 con 0x7fd32006c040 2023-10-16T12:52:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.613+0000 7fd324cb1640 1 -- 172.21.15.60:0/3122361277 >> v1:172.21.15.144:6789/0 conn(0x7fd3200ccfb0 legacy=0x7fd3200cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.613+0000 7fd324cb1640 1 -- 172.21.15.60:0/3122361277 >> v1:172.21.15.60:6789/0 conn(0x7fd32006c040 legacy=0x7fd32006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.613+0000 7fd324cb1640 1 -- 172.21.15.60:0/3122361277 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200cc270 con 0x7fd3200ad970 2023-10-16T12:52:08.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.614+0000 7fd324cb1640 1 --2- 172.21.15.60:0/3122361277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ad970 0x7fd3200cca70 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd3100097d0 tx=0x7fd31002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85e997e33e7eaf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:08.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.614+0000 7fd317fff640 1 -- 172.21.15.60:0/3122361277 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31002d040 con 0x7fd3200ad970 2023-10-16T12:52:08.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.614+0000 7fd317fff640 1 -- 172.21.15.60:0/3122361277 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd310031810 con 0x7fd3200ad970 2023-10-16T12:52:08.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.614+0000 7fd317fff640 1 -- 172.21.15.60:0/3122361277 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310031b80 con 0x7fd3200ad970 2023-10-16T12:52:08.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 -- 172.21.15.60:0/3122361277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ad970 msgr2=0x7fd3200cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3122361277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ad970 0x7fd3200cca70 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd3100097d0 tx=0x7fd31002b8b0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 -- 172.21.15.60:0/3122361277 shutdown_connections 2023-10-16T12:52:08.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3122361277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ad970 0x7fd3200cca70 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 -- 172.21.15.60:0/3122361277 >> 172.21.15.60:0/3122361277 conn(0x7fd3200ce6a0 msgr2=0x7fd3200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 -- 172.21.15.60:0/3122361277 shutdown_connections 2023-10-16T12:52:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.615+0000 7fd3264b4640 1 -- 172.21.15.60:0/3122361277 wait complete. 2023-10-16T12:52:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.617+0000 7fd3264b4640 1 Processor -- start 2023-10-16T12:52:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.617+0000 7fd3264b4640 1 -- start start 2023-10-16T12:52:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.618+0000 7fd3264b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd32006c040 0x7fd320153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.618+0000 7fd3264b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3200ad970 0x7fd3200b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.618+0000 7fd3264b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 0x7fd3200b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.618+0000 7fd3264b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b36f0 con 0x7fd3200ccfb0 2023-10-16T12:52:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.618+0000 7fd3264b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b2d50 con 0x7fd32006c040 2023-10-16T12:52:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.618+0000 7fd3264b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b3030 con 0x7fd3200ad970 2023-10-16T12:52:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 0x7fd3200b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 0x7fd3200b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59830/0 (socket says 172.21.15.60:59830) 2023-10-16T12:52:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 -- 172.21.15.60:0/3054474425 learned_addr learned my addr 172.21.15.60:0/3054474425 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd3254b2640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd32006c040 0x7fd320153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd324cb1640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3200ad970 0x7fd3200b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 -- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3200ad970 msgr2=0x7fd3200b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3200ad970 0x7fd3200b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 -- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd32006c040 msgr2=0x7fd320153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd32006c040 0x7fd320153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 -- 172.21.15.60:0/3054474425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310009400 con 0x7fd3200ccfb0 2023-10-16T12:52:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.619+0000 7fd325cb3640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 0x7fd3200b2190 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd31800c780 tx=0x7fd31800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.620+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318004b60 con 0x7fd3200ccfb0 2023-10-16T12:52:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.620+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31800ce90 con 0x7fd3200ccfb0 2023-10-16T12:52:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.620+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31800a850 con 0x7fd3200ccfb0 2023-10-16T12:52:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.620+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3200b3310 con 0x7fd3200ccfb0 2023-10-16T12:52:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.620+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3200b2810 con 0x7fd3200ccfb0 2023-10-16T12:52:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.621+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32006b330 con 0x7fd3200ccfb0 2023-10-16T12:52:08.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.622+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 50) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd31800a9b0 con 0x7fd3200ccfb0 2023-10-16T12:52:08.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.623+0000 7fd315ffb640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd30406b100 0x7fd30406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:08.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.623+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fd318089dd0 con 0x7fd3200ccfb0 2023-10-16T12:52:08.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.624+0000 7fd3254b2640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd30406b100 0x7fd30406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.625+0000 7fd3254b2640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd30406b100 0x7fd30406d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd31c005880 tx=0x7fd31c009040 comp rx=0 tx=0).ready entity=mgr.6487 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.628+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3180549a0 con 0x7fd3200ccfb0 2023-10-16T12:52:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:08.843+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 --> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3200cdb70 con 0x7fd30406b100 2023-10-16T12:52:09.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.037+0000 7fd315ffb640 1 -- 172.21.15.60:0/3054474425 <== mgr.6487 v2:172.21.15.144:6832/2022371982 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fd31c001350 con 0x7fd30406b100 2023-10-16T12:52:09.040 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /bad-path 2023-10-16T12:52:09.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.040+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd30406b100 msgr2=0x7fd30406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd30406b100 0x7fd30406d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd31c005880 tx=0x7fd31c009040 comp rx=0 tx=0).stop 2023-10-16T12:52:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 msgr2=0x7fd3200b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 0x7fd3200b2190 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd31800c780 tx=0x7fd31800cc40 comp rx=0 tx=0).stop 2023-10-16T12:52:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 shutdown_connections 2023-10-16T12:52:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd30406b100 0x7fd30406d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3200ad970 0x7fd3200b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3200ccfb0 0x7fd3200b2190 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 --2- 172.21.15.60:0/3054474425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd32006c040 0x7fd320153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 >> 172.21.15.60:0/3054474425 conn(0x7fd3200ce6a0 msgr2=0x7fd32015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:09.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 shutdown_connections 2023-10-16T12:52:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.041+0000 7fd3264b4640 1 -- 172.21.15.60:0/3054474425 wait complete. 2023-10-16T12:52:09.065 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-16T12:52:09.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:52:09.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.289+0000 7fd2af7e9640 1 Processor -- start 2023-10-16T12:52:09.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.289+0000 7fd2af7e9640 1 -- start start 2023-10-16T12:52:09.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2af7e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2a80ad350 0x7fd2a8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:09.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2af7e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a806a730 con 0x7fd2a806a2e0 2023-10-16T12:52:09.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2af7e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a80ac890 con 0x7fd2a815d150 2023-10-16T12:52:09.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2af7e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a815d530 con 0x7fd2a80ad350 2023-10-16T12:52:09.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2acd84640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2a80ad350 0x7fd2a8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:09.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2acd84640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2a80ad350 0x7fd2a8069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34756/0 (socket says 172.21.15.60:34756) 2023-10-16T12:52:09.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2add86640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd2a806a2e0 0x7fd2a80c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50262/0 (socket says 172.21.15.60:50262) 2023-10-16T12:52:09.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.291+0000 7fd2acd84640 1 -- 172.21.15.60:0/12021912 learned_addr learned my addr 172.21.15.60:0/12021912 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.292+0000 7fd2acd84640 1 -- 172.21.15.60:0/12021912 >> v1:172.21.15.144:6789/0 conn(0x7fd2a815d150 legacy=0x7fd2a8160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.292+0000 7fd2acd84640 1 -- 172.21.15.60:0/12021912 >> v1:172.21.15.60:6789/0 conn(0x7fd2a806a2e0 legacy=0x7fd2a80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.292+0000 7fd2acd84640 1 -- 172.21.15.60:0/12021912 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a80ad7a0 con 0x7fd2a80ad350 2023-10-16T12:52:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.292+0000 7fd29ffff640 1 -- 172.21.15.60:0/12021912 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137701119 0 0) 0x7fd2a0001490 con 0x7fd2a806a2e0 2023-10-16T12:52:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.293+0000 7fd2acd84640 1 --2- 172.21.15.60:0/12021912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a80ad350 0x7fd2a8069da0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd298009870 tx=0x7fd29802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02915f1521e62bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.293+0000 7fd29ffff640 1 -- 172.21.15.60:0/12021912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29802f840 con 0x7fd2a80ad350 2023-10-16T12:52:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.293+0000 7fd29ffff640 1 -- 172.21.15.60:0/12021912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29802f9a0 con 0x7fd2a80ad350 2023-10-16T12:52:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.293+0000 7fd29ffff640 1 -- 172.21.15.60:0/12021912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29802fd30 con 0x7fd2a80ad350 2023-10-16T12:52:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 -- 172.21.15.60:0/12021912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a80ad350 msgr2=0x7fd2a8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/12021912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a80ad350 0x7fd2a8069da0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd298009870 tx=0x7fd29802ba20 comp rx=0 tx=0).stop 2023-10-16T12:52:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 -- 172.21.15.60:0/12021912 shutdown_connections 2023-10-16T12:52:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/12021912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a80ad350 0x7fd2a8069da0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 -- 172.21.15.60:0/12021912 >> 172.21.15.60:0/12021912 conn(0x7fd2a80ce9b0 msgr2=0x7fd2a80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 -- 172.21.15.60:0/12021912 shutdown_connections 2023-10-16T12:52:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.294+0000 7fd2af7e9640 1 -- 172.21.15.60:0/12021912 wait complete. 2023-10-16T12:52:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.296+0000 7fd2af7e9640 1 Processor -- start 2023-10-16T12:52:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.296+0000 7fd2af7e9640 1 -- start start 2023-10-16T12:52:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2af7e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 0x7fd2a815e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2af7e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2a80ad350 0x7fd2a80c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2af7e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2a815d150 0x7fd2a815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2af7e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815ec60 con 0x7fd2a806a2e0 2023-10-16T12:52:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2af7e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815c5a0 con 0x7fd2a815d150 2023-10-16T12:52:09.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2af7e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a815ee70 con 0x7fd2a80ad350 2023-10-16T12:52:09.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2ad585640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 0x7fd2a815e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:09.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2ad585640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 0x7fd2a815e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34766/0 (socket says 172.21.15.60:34766) 2023-10-16T12:52:09.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.297+0000 7fd2ad585640 1 -- 172.21.15.60:0/4154425261 learned_addr learned my addr 172.21.15.60:0/4154425261 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:09.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2acd84640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2a80ad350 0x7fd2a80c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:09.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2add86640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2a815d150 0x7fd2a815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:09.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2ad585640 1 -- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2a80ad350 msgr2=0x7fd2a80c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2ad585640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2a80ad350 0x7fd2a80c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2ad585640 1 -- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2a815d150 msgr2=0x7fd2a815b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:09.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2ad585640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2a815d150 0x7fd2a815b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2ad585640 1 -- 172.21.15.60:0/4154425261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980094d0 con 0x7fd2a806a2e0 2023-10-16T12:52:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2ad585640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 0x7fd2a815e720 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400c400 tx=0x7fd2a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40052d0 con 0x7fd2a806a2e0 2023-10-16T12:52:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4005430 con 0x7fd2a806a2e0 2023-10-16T12:52:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4003830 con 0x7fd2a806a2e0 2023-10-16T12:52:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a80c2a20 con 0x7fd2a806a2e0 2023-10-16T12:52:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.298+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a80b2d40 con 0x7fd2a806a2e0 2023-10-16T12:52:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.300+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a804e160 con 0x7fd2a806a2e0 2023-10-16T12:52:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.300+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 50) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd2a4003990 con 0x7fd2a806a2e0 2023-10-16T12:52:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.301+0000 7fd29dffb640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd28c06b100 0x7fd28c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.301+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fd2a4089180 con 0x7fd2a806a2e0 2023-10-16T12:52:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.302+0000 7fd2acd84640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd28c06b100 0x7fd28c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.303+0000 7fd2acd84640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd28c06b100 0x7fd28c06d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd298036000 tx=0x7fd298039040 comp rx=0 tx=0).ready entity=mgr.6487 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.306+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2a4053d80 con 0x7fd2a806a2e0 2023-10-16T12:52:09.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:09.558+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd2a80ace20 con 0x7fd2a806a2e0 2023-10-16T12:52:10.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.058+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v51) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd2a40573a0 con 0x7fd2a806a2e0 2023-10-16T12:52:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.064+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd28c06b100 msgr2=0x7fd28c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.064+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd28c06b100 0x7fd28c06d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd298036000 tx=0x7fd298039040 comp rx=0 tx=0).stop 2023-10-16T12:52:10.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.064+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 msgr2=0x7fd2a815e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:10.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.064+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 0x7fd2a815e720 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400c400 tx=0x7fd2a400c8c0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.064+0000 7fd29dffb640 1 -- 172.21.15.60:0/4154425261 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd2a4054150 con 0x7fd2a806a2e0 2023-10-16T12:52:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 shutdown_connections 2023-10-16T12:52:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fd28c06b100 0x7fd28c06d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2a80ad350 0x7fd2a80c24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2a815d150 0x7fd2a815b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 --2- 172.21.15.60:0/4154425261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2a806a2e0 0x7fd2a815e720 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 >> 172.21.15.60:0/4154425261 conn(0x7fd2a80ce9b0 msgr2=0x7fd2a806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.065+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 shutdown_connections 2023-10-16T12:52:10.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.066+0000 7fd2af7e9640 1 -- 172.21.15.60:0/4154425261 wait complete. 2023-10-16T12:52:10.097 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:52:10.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:52:10.215 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:52:10.304 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:52:10.305 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:52:10.305 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:10.361 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:52:10.361 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:10.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:10.591 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:10.588+0000 7f11a8e1d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:52:10.595 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:10.593+0000 7fb67e438040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:52:10.644 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:10.641+0000 7f11a8e1d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:52:10.653 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:10.650+0000 7fb67e438040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:52:10.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.663+0000 7fa390c61640 1 Processor -- start 2023-10-16T12:52:10.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.664+0000 7fa390c61640 1 -- start start 2023-10-16T12:52:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa390c61640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa38c0713e0 0x7fa38c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa390c61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c06b950 0x7fa38c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa390c61640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c070fb0 con 0x7fa38c15d330 2023-10-16T12:52:10.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa390c61640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c0cf790 con 0x7fa38c0713e0 2023-10-16T12:52:10.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa390c61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0cf4b0 con 0x7fa38c06b950 2023-10-16T12:52:10.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa383fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa38c0713e0 0x7fa38c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:10.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa383fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa38c0713e0 0x7fa38c06b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34782/0 (socket says 172.21.15.60:34782) 2023-10-16T12:52:10.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa383fff640 1 -- 172.21.15.60:0/204305044 learned_addr learned my addr 172.21.15.60:0/204305044 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.666+0000 7fa38ad9d640 1 --2- 172.21.15.60:0/204305044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c06b950 0x7fa38c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.667+0000 7fa383fff640 1 -- 172.21.15.60:0/204305044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c06b950 msgr2=0x7fa38c06fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.667+0000 7fa383fff640 1 --2- 172.21.15.60:0/204305044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c06b950 0x7fa38c06fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.667+0000 7fa383fff640 1 -- 172.21.15.60:0/204305044 >> v1:172.21.15.144:6789/0 conn(0x7fa38c15d330 legacy=0x7fa38c0c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.667+0000 7fa383fff640 1 -- 172.21.15.60:0/204305044 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0cc890 con 0x7fa38c0713e0 2023-10-16T12:52:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.668+0000 7fa383fff640 1 --2- 172.21.15.60:0/204305044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c06b410 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa374009870 tx=0x7fa37402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af889e91a6bce52 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.668+0000 7fa389d9b640 1 -- 172.21.15.60:0/204305044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374031640 con 0x7fa38c0713e0 2023-10-16T12:52:10.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa389d9b640 1 -- 172.21.15.60:0/204305044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3740317a0 con 0x7fa38c0713e0 2023-10-16T12:52:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa389d9b640 1 -- 172.21.15.60:0/204305044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374031b30 con 0x7fa38c0713e0 2023-10-16T12:52:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 -- 172.21.15.60:0/204305044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 msgr2=0x7fa38c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 --2- 172.21.15.60:0/204305044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c06b410 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa374009870 tx=0x7fa37402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 -- 172.21.15.60:0/204305044 shutdown_connections 2023-10-16T12:52:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 --2- 172.21.15.60:0/204305044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c06b950 0x7fa38c06fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 --2- 172.21.15.60:0/204305044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c06b410 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 -- 172.21.15.60:0/204305044 >> 172.21.15.60:0/204305044 conn(0x7fa38c0ce860 msgr2=0x7fa38c0cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 -- 172.21.15.60:0/204305044 shutdown_connections 2023-10-16T12:52:10.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.669+0000 7fa390c61640 1 -- 172.21.15.60:0/204305044 wait complete. 2023-10-16T12:52:10.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.671+0000 7fa390c61640 1 Processor -- start 2023-10-16T12:52:10.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.671+0000 7fa390c61640 1 -- start start 2023-10-16T12:52:10.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.672+0000 7fa390c61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c06b950 0x7fa38c15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:10.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.672+0000 7fa390c61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:10.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.672+0000 7fa390c61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c15d330 0x7fa38c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:10.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.672+0000 7fa390c61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c15a000 con 0x7fa38c0713e0 2023-10-16T12:52:10.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.672+0000 7fa390c61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c15a2e0 con 0x7fa38c06b950 2023-10-16T12:52:10.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.672+0000 7fa390c61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c15a5c0 con 0x7fa38c15d330 2023-10-16T12:52:10.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:10.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c0af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34786/0 (socket says 172.21.15.60:34786) 2023-10-16T12:52:10.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 -- 172.21.15.60:0/1662332070 learned_addr learned my addr 172.21.15.60:0/1662332070 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:10.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa38ad9d640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c15d330 0x7fa38c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:10.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa38a59c640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c06b950 0x7fa38c15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:10.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 -- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c15d330 msgr2=0x7fa38c151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:10.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c15d330 0x7fa38c151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 -- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c06b950 msgr2=0x7fa38c15b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:10.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c06b950 0x7fa38c15b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.673+0000 7fa383fff640 1 -- 172.21.15.60:0/1662332070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3740094d0 con 0x7fa38c0713e0 2023-10-16T12:52:10.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.674+0000 7fa383fff640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c0af020 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fa374000c00 tx=0x7fa3740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:10.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.674+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37403ac60 con 0x7fa38c0713e0 2023-10-16T12:52:10.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.674+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37403adc0 con 0x7fa38c0713e0 2023-10-16T12:52:10.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.674+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374044630 con 0x7fa38c0713e0 2023-10-16T12:52:10.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.674+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c15a8a0 con 0x7fa38c0713e0 2023-10-16T12:52:10.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.674+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c0af560 con 0x7fa38c0713e0 2023-10-16T12:52:10.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.675+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c0af6f0 con 0x7fa38c0713e0 2023-10-16T12:52:10.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.678+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa37402f050 con 0x7fa38c0713e0 2023-10-16T12:52:10.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.678+0000 7fa382ffd640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 0x7fa36c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:10.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.679+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fa3740abb10 con 0x7fa38c0713e0 2023-10-16T12:52:10.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.679+0000 7fa38a59c640 1 -- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 msgr2=0x7fa36c06d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:10.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.679+0000 7fa38a59c640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 0x7fa36c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:10.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.683+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3740766b0 con 0x7fa38c0713e0 2023-10-16T12:52:10.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.878+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa38c04e160 con 0x7fa38c0713e0 2023-10-16T12:52:10.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.880+0000 7fa38a59c640 1 -- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 msgr2=0x7fa36c06d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:10.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.880+0000 7fa38a59c640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 0x7fa36c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:10.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.882+0000 7fa382ffd640 1 -- 172.21.15.60:0/1662332070 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17117 (secure 0 0 0) 0x7fa374031e60 con 0x7fa38c0713e0 2023-10-16T12:52:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:10.885 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":116,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:08.022489+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.22","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2189557198":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1153724330":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1224606105":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/840292481":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6833/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1070434778":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6832/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/4048637185":"2023-10-17T12:51:06.102326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:10.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 msgr2=0x7fa36c06d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:10.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 0x7fa36c06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 msgr2=0x7fa38c0af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:10.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c0af020 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fa374000c00 tx=0x7fa3740046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 shutdown_connections 2023-10-16T12:52:10.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fa36c06b100 0x7fa36c06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa38c15d330 0x7fa38c151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa38c0713e0 0x7fa38c0af020 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 --2- 172.21.15.60:0/1662332070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c06b950 0x7fa38c15b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:10.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 >> 172.21.15.60:0/1662332070 conn(0x7fa38c0ce860 msgr2=0x7fa38c0af950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:10.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 shutdown_connections 2023-10-16T12:52:10.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:10.884+0000 7fa390c61640 1 -- 172.21.15.60:0/1662332070 wait complete. 2023-10-16T12:52:10.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:50:49.355185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:50:50.439078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:52:10.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:52:11.093 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.090+0000 7f11a8e1d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:52:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.116+0000 7f1dfcf7a640 1 Processor -- start 2023-10-16T12:52:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.116+0000 7f1dfcf7a640 1 -- start start 2023-10-16T12:52:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1dfcf7a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1dfcf7a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1df80b0ea0 0x7f1df80cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1dfcf7a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1df80b43c0 0x7f1df815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1dfcf7a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df806ca00 con 0x7f1df80b43c0 2023-10-16T12:52:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1dfcf7a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df806be30 con 0x7f1df80b0ea0 2023-10-16T12:52:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1dfcf7a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80ada10 con 0x7f1df80b3ec0 2023-10-16T12:52:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1df7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1df80b43c0 0x7f1df815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1df7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1df80b43c0 0x7f1df815a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34800/0 (socket says 172.21.15.60:34800) 2023-10-16T12:52:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1df7fff640 1 -- 172.21.15.60:0/127668764 learned_addr learned my addr 172.21.15.60:0/127668764 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1df6ffd640 1 --2- 172.21.15.60:0/127668764 >> v2:172.21.15.144:3300/0 conn(0x7f1df80b0ea0 0x7f1df80cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.118+0000 7f1df77fe640 1 --2- 172.21.15.60:0/127668764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.119+0000 7f1df7fff640 1 -- 172.21.15.60:0/127668764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 msgr2=0x7f1df80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.119+0000 7f1df7fff640 1 --2- 172.21.15.60:0/127668764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.119+0000 7f1df7fff640 1 -- 172.21.15.60:0/127668764 >> v2:172.21.15.144:3300/0 conn(0x7f1df80b0ea0 msgr2=0x7f1df80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:11.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.119+0000 7f1df7fff640 1 --2- 172.21.15.60:0/127668764 >> v2:172.21.15.144:3300/0 conn(0x7f1df80b0ea0 0x7f1df80cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.119+0000 7f1df7fff640 1 -- 172.21.15.60:0/127668764 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df806c060 con 0x7f1df80b43c0 2023-10-16T12:52:11.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.120+0000 7f1df7fff640 1 --2- 172.21.15.60:0/127668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b43c0 0x7f1df815a2f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1df806ef60 tx=0x7f1df002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d684a120e0cde312 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:11.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.121+0000 7f1df67fc640 1 -- 172.21.15.60:0/127668764 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df002c9e0 con 0x7f1df80b43c0 2023-10-16T12:52:11.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.121+0000 7f1df67fc640 1 -- 172.21.15.60:0/127668764 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df002cb40 con 0x7f1df80b43c0 2023-10-16T12:52:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.121+0000 7f1df67fc640 1 -- 172.21.15.60:0/127668764 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df00343f0 con 0x7f1df80b43c0 2023-10-16T12:52:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/127668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b43c0 msgr2=0x7f1df815a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/127668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b43c0 0x7f1df815a2f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1df806ef60 tx=0x7f1df002be40 comp rx=0 tx=0).stop 2023-10-16T12:52:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/127668764 shutdown_connections 2023-10-16T12:52:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/127668764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b43c0 0x7f1df815a2f0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/127668764 >> v2:172.21.15.144:3300/0 conn(0x7f1df80b0ea0 0x7f1df80cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/127668764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/127668764 >> 172.21.15.60:0/127668764 conn(0x7f1df80b0170 msgr2=0x7f1df80b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/127668764 shutdown_connections 2023-10-16T12:52:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.122+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/127668764 wait complete. 2023-10-16T12:52:11.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.124+0000 7f1dfcf7a640 1 Processor -- start 2023-10-16T12:52:11.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.124+0000 7f1dfcf7a640 1 -- start start 2023-10-16T12:52:11.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1dfcf7a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 0x7f1df815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1dfcf7a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1dfcf7a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1df80b43c0 0x7f1df80c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1dfcf7a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df815b510 con 0x7f1df80b0ea0 2023-10-16T12:52:11.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1dfcf7a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df81504a0 con 0x7f1df80b43c0 2023-10-16T12:52:11.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1dfcf7a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df815a900 con 0x7f1df80b3ec0 2023-10-16T12:52:11.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1df77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 0x7f1df815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1df77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 0x7f1df815b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34810/0 (socket says 172.21.15.60:34810) 2023-10-16T12:52:11.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1df77fe640 1 -- 172.21.15.60:0/2100211415 learned_addr learned my addr 172.21.15.60:0/2100211415 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:11.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1df6ffd640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1df7fff640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1df80b43c0 0x7f1df80c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.125+0000 7f1df77fe640 1 -- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 msgr2=0x7f1df80c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df77fe640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df77fe640 1 -- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1df80b43c0 msgr2=0x7f1df80c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df77fe640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1df80b43c0 0x7f1df80c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df77fe640 1 -- 172.21.15.60:0/2100211415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df0009b30 con 0x7f1df80b0ea0 2023-10-16T12:52:11.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df7fff640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1df80b43c0 0x7f1df80c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:11.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df6ffd640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:11.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1df77fe640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 0x7f1df815b9a0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1de800d5f0 tx=0x7f1de800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:11.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80125f0 con 0x7f1df80b0ea0 2023-10-16T12:52:11.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de8012750 con 0x7f1df80b0ea0 2023-10-16T12:52:11.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df815abe0 con 0x7f1df80b0ea0 2023-10-16T12:52:11.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8012b00 con 0x7f1df80b0ea0 2023-10-16T12:52:11.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.126+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df815b1a0 con 0x7f1df80b0ea0 2023-10-16T12:52:11.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.127+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df804e9b0 con 0x7f1df80b0ea0 2023-10-16T12:52:11.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.128+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1de8012c60 con 0x7f1df80b0ea0 2023-10-16T12:52:11.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.130+0000 7f1ddffff640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f1de406b0b0 0x7f1de406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.130+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f1de808a070 con 0x7f1df80b0ea0 2023-10-16T12:52:11.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.130+0000 7f1df6ffd640 1 -- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f1de406b0b0 msgr2=0x7f1de406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:11.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.130+0000 7f1df6ffd640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f1de406b0b0 0x7f1de406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:11.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.134+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1de80bcfe0 con 0x7f1df80b0ea0 2023-10-16T12:52:11.138 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.134+0000 7fb67e438040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:52:11.147 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.144+0000 7f11a8e1d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:52:11.192 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.189+0000 7fb67e438040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:52:11.197 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.194+0000 7f11a8e1d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:52:11.245 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.242+0000 7fb67e438040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:52:11.256 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.254+0000 7f11a8e1d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:52:11.305 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.302+0000 7f11a8e1d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:52:11.308 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.306+0000 7fb67e438040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:52:11.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.325+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1df806abd0 con 0x7f1df80b0ea0 2023-10-16T12:52:11.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.326+0000 7f1ddffff640 1 -- 172.21.15.60:0/2100211415 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1de8054c30 con 0x7f1df80b0ea0 2023-10-16T12:52:11.328 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:52:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f1de406b0b0 msgr2=0x7f1de406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f1de406b0b0 0x7f1de406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 msgr2=0x7f1df815b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 0x7f1df815b9a0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1de800d5f0 tx=0x7f1de800dab0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 shutdown_connections 2023-10-16T12:52:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f1de406b0b0 0x7f1de406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1df80b3ec0 0x7f1df80c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1df80b43c0 0x7f1df80c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 --2- 172.21.15.60:0/2100211415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df80b0ea0 0x7f1df815b9a0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 >> 172.21.15.60:0/2100211415 conn(0x7f1df80b0170 msgr2=0x7f1df80b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:11.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.328+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 shutdown_connections 2023-10-16T12:52:11.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.329+0000 7f1dfcf7a640 1 -- 172.21.15.60:0/2100211415 wait complete. 2023-10-16T12:52:11.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:52:11.362 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.360+0000 7fb67e438040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:52:11.416 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.413+0000 7f11a8e1d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:52:11.479 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.476+0000 7f11a8e1d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:11.482 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.479+0000 7fb67e438040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:52:11.528 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.525+0000 7f11a8e1d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:52:11.550 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.547+0000 7fb67e438040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.565+0000 7fbf81875640 1 Processor -- start 2023-10-16T12:52:11.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.565+0000 7fbf81875640 1 -- start start 2023-10-16T12:52:11.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf81875640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf7c06fa20 0x7fbf7c0ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf81875640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbf7c065cc0 0x7fbf7c0acc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf81875640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7c06d900 con 0x7fbf7c06d520 2023-10-16T12:52:11.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf81875640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7c15b1d0 con 0x7fbf7c06fa20 2023-10-16T12:52:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf81875640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7c0b24f0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf7bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf7c06fa20 0x7fbf7c0ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.567+0000 7fbf7bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf7c06fa20 0x7fbf7c0ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34820/0 (socket says 172.21.15.60:34820) 2023-10-16T12:52:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.568+0000 7fbf7bfff640 1 -- 172.21.15.60:0/3300506556 learned_addr learned my addr 172.21.15.60:0/3300506556 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.568+0000 7fbf81074640 1 --2- 172.21.15.60:0/3300506556 >> v2:172.21.15.144:3300/0 conn(0x7fbf7c065cc0 0x7fbf7c0acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.568+0000 7fbf7bfff640 1 -- 172.21.15.60:0/3300506556 >> v2:172.21.15.144:3300/0 conn(0x7fbf7c065cc0 msgr2=0x7fbf7c0acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:11.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.568+0000 7fbf7bfff640 1 --2- 172.21.15.60:0/3300506556 >> v2:172.21.15.144:3300/0 conn(0x7fbf7c065cc0 0x7fbf7c0acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.568+0000 7fbf7bfff640 1 -- 172.21.15.60:0/3300506556 >> v1:172.21.15.144:6789/0 conn(0x7fbf7c06d520 legacy=0x7fbf7c0b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.568+0000 7fbf7bfff640 1 -- 172.21.15.60:0/3300506556 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7c06feb0 con 0x7fbf7c06fa20 2023-10-16T12:52:11.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.569+0000 7fbf7bfff640 1 --2- 172.21.15.60:0/3300506556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c06fa20 0x7fbf7c0ac750 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0ae180 tx=0x7fbf6802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5735a02ea679baa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:11.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.569+0000 7fbf7b7fe640 1 -- 172.21.15.60:0/3300506556 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf680317e0 con 0x7fbf7c06fa20 2023-10-16T12:52:11.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.569+0000 7fbf7b7fe640 1 -- 172.21.15.60:0/3300506556 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68031940 con 0x7fbf7c06fa20 2023-10-16T12:52:11.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf7b7fe640 1 -- 172.21.15.60:0/3300506556 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68031cd0 con 0x7fbf7c06fa20 2023-10-16T12:52:11.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 -- 172.21.15.60:0/3300506556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c06fa20 msgr2=0x7fbf7c0ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 --2- 172.21.15.60:0/3300506556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c06fa20 0x7fbf7c0ac750 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0ae180 tx=0x7fbf6802b880 comp rx=0 tx=0).stop 2023-10-16T12:52:11.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 -- 172.21.15.60:0/3300506556 shutdown_connections 2023-10-16T12:52:11.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 --2- 172.21.15.60:0/3300506556 >> v2:172.21.15.144:3300/0 conn(0x7fbf7c065cc0 0x7fbf7c0acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 --2- 172.21.15.60:0/3300506556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c06fa20 0x7fbf7c0ac750 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 -- 172.21.15.60:0/3300506556 >> 172.21.15.60:0/3300506556 conn(0x7fbf7c0b0a80 msgr2=0x7fbf7c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:11.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.570+0000 7fbf81875640 1 -- 172.21.15.60:0/3300506556 shutdown_connections 2023-10-16T12:52:11.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.571+0000 7fbf81875640 1 -- 172.21.15.60:0/3300506556 wait complete. 2023-10-16T12:52:11.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.572+0000 7fbf81875640 1 Processor -- start 2023-10-16T12:52:11.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.572+0000 7fbf81875640 1 -- start start 2023-10-16T12:52:11.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf81875640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 0x7fbf7c1f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf81875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf7c06d520 0x7fbf7c15d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf81875640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf7c15da10 0x7fbf7c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf81875640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c1f5850 con 0x7fbf7c065cc0 2023-10-16T12:52:11.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf81875640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c06bfe0 con 0x7fbf7c15da10 2023-10-16T12:52:11.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf81875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c06c2c0 con 0x7fbf7c06d520 2023-10-16T12:52:11.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.573+0000 7fbf80873640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 0x7fbf7c1f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 0x7fbf7c1f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34822/0 (socket says 172.21.15.60:34822) 2023-10-16T12:52:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 -- 172.21.15.60:0/733453574 learned_addr learned my addr 172.21.15.60:0/733453574 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf7bfff640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf7c06d520 0x7fbf7c15d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf81074640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf7c15da10 0x7fbf7c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 -- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf7c06d520 msgr2=0x7fbf7c15d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf7c06d520 0x7fbf7c15d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 -- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf7c15da10 msgr2=0x7fbf7c15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf7c15da10 0x7fbf7c15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 -- 172.21.15.60:0/733453574 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680094d0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf81074640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf7c15da10 0x7fbf7c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf80873640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 0x7fbf7c1f5310 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000c450 tx=0x7fbf7000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.574+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf700052d0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.575+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf70005430 con 0x7fbf7c065cc0 2023-10-16T12:52:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.575+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7c06c5a0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.575+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70012870 con 0x7fbf7c065cc0 2023-10-16T12:52:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.575+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7c06cb60 con 0x7fbf7c065cc0 2023-10-16T12:52:11.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.576+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7c04e9b0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.577+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fbf700129d0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.578+0000 7fbf797fa640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 0x7fbf6006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:11.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.578+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fbf70089770 con 0x7fbf7c065cc0 2023-10-16T12:52:11.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.578+0000 7fbf7bfff640 1 -- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 msgr2=0x7fbf6006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:11.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.578+0000 7fbf7bfff640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 0x7fbf6006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:11.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.582+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf700542a0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.587 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.585+0000 7f11a8e1d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:52:11.604 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.601+0000 7fb67e438040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:52:11.637 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:11.635+0000 7f11a8e1d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:52:11.664 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.661+0000 7fb67e438040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:52:11.717 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:11.714+0000 7fb67e438040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:52:11.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.779+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf7c0ac530 con 0x7fbf7c065cc0 2023-10-16T12:52:11.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.779+0000 7fbf7bfff640 1 -- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 msgr2=0x7fbf6006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:11.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.779+0000 7fbf7bfff640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 0x7fbf6006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:11.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.780+0000 7fbf797fa640 1 -- 172.21.15.60:0/733453574 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf700578c0 con 0x7fbf7c065cc0 2023-10-16T12:52:11.783 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:52:11.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 msgr2=0x7fbf6006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 0x7fbf6006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 msgr2=0x7fbf7c1f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 0x7fbf7c1f5310 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000c450 tx=0x7fbf7000c910 comp rx=0 tx=0).stop 2023-10-16T12:52:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 shutdown_connections 2023-10-16T12:52:11.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fbf6006b0b0 0x7fbf6006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf7c06d520 0x7fbf7c15d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf7c15da10 0x7fbf7c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 --2- 172.21.15.60:0/733453574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf7c065cc0 0x7fbf7c1f5310 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 >> 172.21.15.60:0/733453574 conn(0x7fbf7c0b0a80 msgr2=0x7fbf7c0b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:11.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 shutdown_connections 2023-10-16T12:52:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:11.782+0000 7fbf81875640 1 -- 172.21.15.60:0/733453574 wait complete. 2023-10-16T12:52:11.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:52:12.022 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.019+0000 7f11a8e1d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:52:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.070+0000 7ff0e8ef2640 1 Processor -- start 2023-10-16T12:52:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.070+0000 7ff0e8ef2640 1 -- start start 2023-10-16T12:52:12.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e8ef2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0e415ad70 0x7ff0e4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e8ef2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e8ef2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e40cb030 con 0x7ff0e40c9ca0 2023-10-16T12:52:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e8ef2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e40c91c0 con 0x7ff0e415ad70 2023-10-16T12:52:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e8ef2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40c94a0 con 0x7ff0e40caac0 2023-10-16T12:52:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0e415ad70 0x7ff0e4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e1d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff0e40c9ca0 0x7ff0e40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50264/0 (socket says 172.21.15.60:50264) 2023-10-16T12:52:12.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e1d9b640 1 -- 172.21.15.60:0/1523235106 learned_addr learned my addr 172.21.15.60:0/1523235106 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e2d9d640 1 --2- 172.21.15.60:0/1523235106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e259c640 1 -- 172.21.15.60:0/1523235106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 msgr2=0x7ff0e406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.072+0000 7ff0e259c640 1 --2- 172.21.15.60:0/1523235106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.073+0000 7ff0e259c640 1 -- 172.21.15.60:0/1523235106 >> v1:172.21.15.60:6789/0 conn(0x7ff0e40c9ca0 legacy=0x7ff0e40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.073+0000 7ff0e259c640 1 -- 172.21.15.60:0/1523235106 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e40c9780 con 0x7ff0e415ad70 2023-10-16T12:52:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.074+0000 7ff0e259c640 1 --2- 172.21.15.60:0/1523235106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4069da0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009d90 tx=0x7ff0dc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15206cb6b3016939 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:12.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.074+0000 7ff0e159a640 1 -- 172.21.15.60:0/1523235106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc02cc60 con 0x7ff0e415ad70 2023-10-16T12:52:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.074+0000 7ff0e159a640 1 -- 172.21.15.60:0/1523235106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc02cdc0 con 0x7ff0e415ad70 2023-10-16T12:52:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.074+0000 7ff0e159a640 1 -- 172.21.15.60:0/1523235106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc034660 con 0x7ff0e415ad70 2023-10-16T12:52:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/1523235106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 msgr2=0x7ff0e4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/1523235106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4069da0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009d90 tx=0x7ff0dc02beb0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/1523235106 shutdown_connections 2023-10-16T12:52:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/1523235106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/1523235106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4069da0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/1523235106 >> 172.21.15.60:0/1523235106 conn(0x7ff0e40ce9b0 msgr2=0x7ff0e40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/1523235106 shutdown_connections 2023-10-16T12:52:12.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.075+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/1523235106 wait complete. 2023-10-16T12:52:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.077+0000 7ff0e8ef2640 1 Processor -- start 2023-10-16T12:52:12.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.077+0000 7ff0e8ef2640 1 -- start start 2023-10-16T12:52:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e8ef2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0e40c9ca0 0x7ff0e4155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e8ef2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e40c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e8ef2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e8ef2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e415bdf0 con 0x7ff0e415ad70 2023-10-16T12:52:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e8ef2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4156500 con 0x7ff0e40c9ca0 2023-10-16T12:52:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e8ef2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40c6360 con 0x7ff0e40caac0 2023-10-16T12:52:12.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34840/0 (socket says 172.21.15.60:34840) 2023-10-16T12:52:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 -- 172.21.15.60:0/3203750043 learned_addr learned my addr 172.21.15.60:0/3203750043 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e1d9b640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e40c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e259c640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0e40c9ca0 0x7ff0e4155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 -- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 msgr2=0x7ff0e40c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e40c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 -- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0e40c9ca0 msgr2=0x7ff0e4155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.078+0000 7ff0e2d9d640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0e40c9ca0 0x7ff0e4155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0e2d9d640 1 -- 172.21.15.60:0/3203750043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc009a70 con 0x7ff0e415ad70 2023-10-16T12:52:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0e2d9d640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4157750 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400d800 tx=0x7ff0d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:12.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d400b6c0 con 0x7ff0e415ad70 2023-10-16T12:52:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d400b840 con 0x7ff0e415ad70 2023-10-16T12:52:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d400bbf0 con 0x7ff0e415ad70 2023-10-16T12:52:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e40c6640 con 0x7ff0e415ad70 2023-10-16T12:52:12.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.079+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e40c6c00 con 0x7ff0e415ad70 2023-10-16T12:52:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.081+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e415bf30 con 0x7ff0e415ad70 2023-10-16T12:52:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.082+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff0d400bd50 con 0x7ff0e415ad70 2023-10-16T12:52:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.083+0000 7ff0daffd640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 0x7ff0c806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.083+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7ff0d408a0f0 con 0x7ff0e415ad70 2023-10-16T12:52:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.083+0000 7ff0e259c640 1 -- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 msgr2=0x7ff0c806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.083+0000 7ff0e259c640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 0x7ff0c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.087+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff0e415bf30 con 0x7ff0e415ad70 2023-10-16T12:52:12.129 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.127+0000 7fb67e438040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:52:12.182 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.179+0000 7f11a8e1d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:52:12.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.281+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0e415bf30 con 0x7ff0e415ad70 2023-10-16T12:52:12.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.281+0000 7ff0daffd640 1 -- 172.21.15.60:0/3203750043 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0e415bf30 con 0x7ff0e415ad70 2023-10-16T12:52:12.284 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:52:12.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e259c640 1 -- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 msgr2=0x7ff0c806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:12.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e259c640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 0x7ff0c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:12.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 msgr2=0x7ff0c806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:12.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 0x7ff0c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 msgr2=0x7ff0e4157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4157750 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400d800 tx=0x7ff0d400dcc0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 shutdown_connections 2023-10-16T12:52:12.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7ff0c806b0b0 0x7ff0c806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0e40caac0 0x7ff0e40c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e415ad70 0x7ff0e4157750 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 --2- 172.21.15.60:0/3203750043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0e40c9ca0 0x7ff0e4155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 >> 172.21.15.60:0/3203750043 conn(0x7ff0e40ce9b0 msgr2=0x7ff0e40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:12.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 shutdown_connections 2023-10-16T12:52:12.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.284+0000 7ff0e8ef2640 1 -- 172.21.15.60:0/3203750043 wait complete. 2023-10-16T12:52:12.290 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.286+0000 7f11a8e1d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:52:12.299 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.297+0000 7fb67e438040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:52:12.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:52:12.338 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.335+0000 7f11a8e1d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:52:12.414 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.412+0000 7fb67e438040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:52:12.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.442+0000 7f11a8e1d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:12.467 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.465+0000 7fb67e438040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:52:12.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.514+0000 7f69ea95e640 1 Processor -- start 2023-10-16T12:52:12.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.515+0000 7f69ea95e640 1 -- start start 2023-10-16T12:52:12.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.516+0000 7f69ea95e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e415cd70 0x7f69e40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.516+0000 7f69ea95e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f69e40cded0 0x7f69e40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.516+0000 7f69ea95e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e415d150 con 0x7f69e40b8480 2023-10-16T12:52:12.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.516+0000 7f69ea95e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e40d1180 con 0x7f69e40cded0 2023-10-16T12:52:12.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.516+0000 7f69ea95e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40d1460 con 0x7f69e415cd70 2023-10-16T12:52:12.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e915b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f69e40b8480 0x7f69e40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50270/0 (socket says 172.21.15.60:50270) 2023-10-16T12:52:12.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e915b640 1 -- 172.21.15.60:0/2274783837 learned_addr learned my addr 172.21.15.60:0/2274783837 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:12.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e995c640 1 --2- 172.21.15.60:0/2274783837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e415cd70 0x7f69e40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69ea15d640 1 --2- 172.21.15.60:0/2274783837 >> v2:172.21.15.144:3300/0 conn(0x7f69e40cded0 0x7f69e40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e895a640 1 -- 172.21.15.60:0/2274783837 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685244777 0 0) 0x7f69e415d150 con 0x7f69e40b8480 2023-10-16T12:52:12.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e895a640 1 -- 172.21.15.60:0/2274783837 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d0003520 con 0x7f69e40b8480 2023-10-16T12:52:12.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e995c640 1 -- 172.21.15.60:0/2274783837 >> v2:172.21.15.144:3300/0 conn(0x7f69e40cded0 msgr2=0x7f69e40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:12.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e995c640 1 --2- 172.21.15.60:0/2274783837 >> v2:172.21.15.144:3300/0 conn(0x7f69e40cded0 0x7f69e40b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e995c640 1 -- 172.21.15.60:0/2274783837 >> v1:172.21.15.60:6789/0 conn(0x7f69e40b8480 legacy=0x7f69e40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.517+0000 7f69e995c640 1 -- 172.21.15.60:0/2274783837 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e40d1740 con 0x7f69e415cd70 2023-10-16T12:52:12.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.519+0000 7f69e995c640 1 --2- 172.21.15.60:0/2274783837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e415cd70 0x7f69e40d2220 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69e00097a0 tx=0x7f69e002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e049a68085982bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:12.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.519+0000 7f69e895a640 1 -- 172.21.15.60:0/2274783837 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e002d040 con 0x7f69e415cd70 2023-10-16T12:52:12.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.519+0000 7f69e895a640 1 -- 172.21.15.60:0/2274783837 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e0031810 con 0x7f69e415cd70 2023-10-16T12:52:12.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.519+0000 7f69e895a640 1 -- 172.21.15.60:0/2274783837 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e0031b80 con 0x7f69e415cd70 2023-10-16T12:52:12.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 -- 172.21.15.60:0/2274783837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e415cd70 msgr2=0x7f69e40d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 --2- 172.21.15.60:0/2274783837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e415cd70 0x7f69e40d2220 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69e00097a0 tx=0x7f69e002b8b0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 -- 172.21.15.60:0/2274783837 shutdown_connections 2023-10-16T12:52:12.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 --2- 172.21.15.60:0/2274783837 >> v2:172.21.15.144:3300/0 conn(0x7f69e40cded0 0x7f69e40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 --2- 172.21.15.60:0/2274783837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e415cd70 0x7f69e40d2220 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 -- 172.21.15.60:0/2274783837 >> 172.21.15.60:0/2274783837 conn(0x7f69e40cf570 msgr2=0x7f69e40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:12.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 -- 172.21.15.60:0/2274783837 shutdown_connections 2023-10-16T12:52:12.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.520+0000 7f69ea95e640 1 -- 172.21.15.60:0/2274783837 wait complete. 2023-10-16T12:52:12.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.523+0000 7f69ea95e640 1 Processor -- start 2023-10-16T12:52:12.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.523+0000 7f69ea95e640 1 -- start start 2023-10-16T12:52:12.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea95e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69e40b8480 0x7f69e4151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea95e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e40cded0 0x7f69e41f9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea95e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 0x7f69e41f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea95e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e41f9ed0 con 0x7f69e415cd70 2023-10-16T12:52:12.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea95e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40c9d40 con 0x7f69e40b8480 2023-10-16T12:52:12.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea95e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e41fa100 con 0x7f69e40cded0 2023-10-16T12:52:12.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 0x7f69e41f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 0x7f69e41f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34852/0 (socket says 172.21.15.60:34852) 2023-10-16T12:52:12.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 -- 172.21.15.60:0/3321742778 learned_addr learned my addr 172.21.15.60:0/3321742778 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:12.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69e995c640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69e40b8480 0x7f69e4151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69e915b640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e40cded0 0x7f69e41f9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:12.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 -- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e40cded0 msgr2=0x7f69e41f9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e40cded0 0x7f69e41f9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 -- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69e40b8480 msgr2=0x7f69e4151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69e40b8480 0x7f69e4151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69ea15d640 1 -- 172.21.15.60:0/3321742778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0009400 con 0x7f69e415cd70 2023-10-16T12:52:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69e995c640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69e40b8480 0x7f69e4151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.525+0000 7f69e915b640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e40cded0 0x7f69e41f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.526+0000 7f69ea15d640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 0x7f69e41f5a60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00efc0 tx=0x7f69dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.526+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc009b10 con 0x7f69e415cd70 2023-10-16T12:52:12.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.526+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc00f040 con 0x7f69e415cd70 2023-10-16T12:52:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.526+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e41fa3e0 con 0x7f69e415cd70 2023-10-16T12:52:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.526+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc0146c0 con 0x7f69e415cd70 2023-10-16T12:52:12.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.526+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e41f5fa0 con 0x7f69e415cd70 2023-10-16T12:52:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.527+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e40c9f60 con 0x7f69e415cd70 2023-10-16T12:52:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.529+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f69dc014820 con 0x7f69e415cd70 2023-10-16T12:52:12.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.530+0000 7f69da7fc640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 0x7f69c806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.530+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f69dc08a820 con 0x7f69e415cd70 2023-10-16T12:52:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.530+0000 7f69e995c640 1 -- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 msgr2=0x7f69c806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:12.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.530+0000 7f69e995c640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 0x7f69c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:12.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.534+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69e40c9f60 con 0x7f69e415cd70 2023-10-16T12:52:12.553 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.550+0000 7f11a8e1d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:52:12.587 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.584+0000 7fb67e438040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:12.701 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.697+0000 7fb67e438040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:52:12.709 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.706+0000 7f11a8e1d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:12.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.731+0000 7f69e995c640 1 -- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 msgr2=0x7f69c806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:12.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.731+0000 7f69e995c640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 0x7f69c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.746+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f69e41fa900 con 0x7f69e415cd70 2023-10-16T12:52:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.747+0000 7f69da7fc640 1 -- 172.21.15.60:0/3321742778 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v116) v1 ==== 80+0+477 (secure 0 0 0) 0x7f69dc055240 con 0x7f69e415cd70 2023-10-16T12:52:12.749 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2189557198 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.750 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1153724330 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.750 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1224606105 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.750 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/840292481 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.750 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/3513615444 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.750 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1070434778 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.751 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/3513615444 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.751 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/4048637185 2023-10-17T12:51:06.102326+0000 2023-10-16T12:52:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T12:52:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 msgr2=0x7f69c806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 0x7f69c806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 msgr2=0x7f69e41f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 0x7f69e41f5a60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00efc0 tx=0x7f69dc00c490 comp rx=0 tx=0).stop 2023-10-16T12:52:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 shutdown_connections 2023-10-16T12:52:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f69c806b0b0 0x7f69c806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69e40cded0 0x7f69e41f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69e415cd70 0x7f69e41f5a60 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 --2- 172.21.15.60:0/3321742778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69e40b8480 0x7f69e4151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:12.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 >> 172.21.15.60:0/3321742778 conn(0x7f69e40cf570 msgr2=0x7f69e40b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:12.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 shutdown_connections 2023-10-16T12:52:12.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:12.749+0000 7f69ea95e640 1 -- 172.21.15.60:0/3321742778 wait complete. 2023-10-16T12:52:12.757 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.755+0000 7f11a8e1d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:12.769 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:52:12.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:12.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:52:12.857 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:52:12.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:12.858 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:52:12.870 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.868+0000 7fb67e438040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:12.876 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:12.873+0000 7f11a8e1d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:52:12.903 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:52:12.903 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:12.924 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:12.922+0000 7fb67e438040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:12.958 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:52:12.959 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:52:12.959 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:13.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:52:13.015 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:52:13.017 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:13.052 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:13.050+0000 7fb67e438040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:52:13.074 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:52:13.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.286+0000 7fdef8116640 1 Processor -- start 2023-10-16T12:52:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.287+0000 7fdef8116640 1 -- start start 2023-10-16T12:52:13.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.289+0000 7fdef8116640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdef00caa10 0x7fdef006fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef8116640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef0159790 con 0x7fdef00cc920 2023-10-16T12:52:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef8116640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef00c9b20 con 0x7fdef01593b0 2023-10-16T12:52:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef8116640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef00c9e00 con 0x7fdef00caa10 2023-10-16T12:52:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef56b1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdef00caa10 0x7fdef006fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef66b3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdef00cc920 0x7fdef00ccdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50284/0 (socket says 172.21.15.60:50284) 2023-10-16T12:52:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef66b3640 1 -- 172.21.15.60:0/3609004088 learned_addr learned my addr 172.21.15.60:0/3609004088 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef56b1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdef00caa10 0x7fdef006fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34854/0 (socket says 172.21.15.60:34854) 2023-10-16T12:52:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef56b1640 1 -- 172.21.15.60:0/3609004088 >> v1:172.21.15.144:6789/0 conn(0x7fdef01593b0 legacy=0x7fdef006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef56b1640 1 -- 172.21.15.60:0/3609004088 >> v1:172.21.15.60:6789/0 conn(0x7fdef00cc920 legacy=0x7fdef00ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.291+0000 7fdef56b1640 1 -- 172.21.15.60:0/3609004088 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef00ca0e0 con 0x7fdef00caa10 2023-10-16T12:52:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.290+0000 7fdef4eb0640 1 -- 172.21.15.60:0/3609004088 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181625099 0 0) 0x7fdef0159790 con 0x7fdef00cc920 2023-10-16T12:52:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.292+0000 7fdef56b1640 1 --2- 172.21.15.60:0/3609004088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef006fc60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fdee00097a0 tx=0x7fdee002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c5d344d24c6b6a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.292+0000 7fdef4eb0640 1 -- 172.21.15.60:0/3609004088 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee002e070 con 0x7fdef00caa10 2023-10-16T12:52:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.293+0000 7fdef4eb0640 1 -- 172.21.15.60:0/3609004088 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee00315a0 con 0x7fdef00caa10 2023-10-16T12:52:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.293+0000 7fdef4eb0640 1 -- 172.21.15.60:0/3609004088 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0031930 con 0x7fdef00caa10 2023-10-16T12:52:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.293+0000 7fdef8116640 1 -- 172.21.15.60:0/3609004088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 msgr2=0x7fdef006fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.293+0000 7fdef8116640 1 --2- 172.21.15.60:0/3609004088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef006fc60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fdee00097a0 tx=0x7fdee002b840 comp rx=0 tx=0).stop 2023-10-16T12:52:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.294+0000 7fdef8116640 1 -- 172.21.15.60:0/3609004088 shutdown_connections 2023-10-16T12:52:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.294+0000 7fdef8116640 1 --2- 172.21.15.60:0/3609004088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef006fc60 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.294+0000 7fdef8116640 1 -- 172.21.15.60:0/3609004088 >> 172.21.15.60:0/3609004088 conn(0x7fdef00cb410 msgr2=0x7fdef00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.294+0000 7fdef8116640 1 -- 172.21.15.60:0/3609004088 shutdown_connections 2023-10-16T12:52:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.294+0000 7fdef8116640 1 -- 172.21.15.60:0/3609004088 wait complete. 2023-10-16T12:52:13.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.295+0000 7fdef8116640 1 Processor -- start 2023-10-16T12:52:13.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.295+0000 7fdef8116640 1 -- start start 2023-10-16T12:52:13.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.296+0000 7fdef8116640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.296+0000 7fdef8116640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdef00cc920 0x7fdef00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.296+0000 7fdef8116640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdef01593b0 0x7fdef0143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.296+0000 7fdef8116640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00b2dd0 con 0x7fdef00caa10 2023-10-16T12:52:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.296+0000 7fdef8116640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00b2b10 con 0x7fdef00cc920 2023-10-16T12:52:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.296+0000 7fdef8116640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef01531b0 con 0x7fdef01593b0 2023-10-16T12:52:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34864/0 (socket says 172.21.15.60:34864) 2023-10-16T12:52:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 -- 172.21.15.60:0/3637860419 learned_addr learned my addr 172.21.15.60:0/3637860419 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef56b1640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdef00cc920 0x7fdef00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 -- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdef01593b0 msgr2=0x7fdef0143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdef01593b0 0x7fdef0143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 -- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdef00cc920 msgr2=0x7fdef00acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdef00cc920 0x7fdef00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 -- 172.21.15.60:0/3637860419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0009400 con 0x7fdef00caa10 2023-10-16T12:52:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.297+0000 7fdef5eb2640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef00ac890 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c450 tx=0x7fdeec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.298+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0052d0 con 0x7fdef00caa10 2023-10-16T12:52:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.298+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec005430 con 0x7fdef00caa10 2023-10-16T12:52:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.298+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec012870 con 0x7fdef00caa10 2023-10-16T12:52:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.298+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef0144800 con 0x7fdef00caa10 2023-10-16T12:52:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.298+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef0144240 con 0x7fdef00caa10 2023-10-16T12:52:13.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.299+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef004e160 con 0x7fdef00caa10 2023-10-16T12:52:13.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.301+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fdeec0129d0 con 0x7fdef00caa10 2023-10-16T12:52:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.301+0000 7fdee67fc640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 0x7fded406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.302+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fdeec08a250 con 0x7fdef00caa10 2023-10-16T12:52:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.302+0000 7fdef56b1640 1 -- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 msgr2=0x7fded406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.302+0000 7fdef56b1640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 0x7fded406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:13.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.308+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdeec054e00 con 0x7fdef00caa10 2023-10-16T12:52:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.503+0000 7fdef56b1640 1 -- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 msgr2=0x7fded406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.503+0000 7fdef56b1640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 0x7fded406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.540+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdef006b730 con 0x7fdef00caa10 2023-10-16T12:52:13.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.540+0000 7fdee67fc640 1 -- 172.21.15.60:0/3637860419 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fdeec019300 con 0x7fdef00caa10 2023-10-16T12:52:13.543 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:13.544 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:51.414872+0000","modified":"2023-10-16T12:50:54.480441+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7353":{"gid":7353,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/3569389918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3569389918},{"type":"v1","addr":"172.21.15.60:6835","nonce":3569389918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:52:13.544 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 53 2023-10-16T12:52:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 msgr2=0x7fded406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 0x7fded406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 msgr2=0x7fdef00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef00ac890 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c450 tx=0x7fdeec00c910 comp rx=0 tx=0).stop 2023-10-16T12:52:13.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 shutdown_connections 2023-10-16T12:52:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7fded406b0e0 0x7fded406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdef01593b0 0x7fdef0143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdef00cc920 0x7fdef00acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 --2- 172.21.15.60:0/3637860419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef00caa10 0x7fdef00ac890 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 >> 172.21.15.60:0/3637860419 conn(0x7fdef00cb410 msgr2=0x7fdef0155590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 shutdown_connections 2023-10-16T12:52:13.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.543+0000 7fdef8116640 1 -- 172.21.15.60:0/3637860419 wait complete. 2023-10-16T12:52:13.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:13.715 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:13.712+0000 7f11a8e1d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:52:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.764+0000 7f5d0ffff640 1 Processor -- start 2023-10-16T12:52:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.764+0000 7f5d0ffff640 1 -- start start 2023-10-16T12:52:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.765+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 0x7f5d1015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.765+0000 7f5d0ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5d100ccb00 0x7f5d100ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.765+0000 7f5d0ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100cd4a0 con 0x7f5d100ad4d0 2023-10-16T12:52:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.765+0000 7f5d0ffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1015ac60 con 0x7f5d100ccb00 2023-10-16T12:52:13.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.765+0000 7f5d0ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10070030 con 0x7f5d10153740 2023-10-16T12:52:13.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.766+0000 7f5d0effd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 0x7f5d1015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.766+0000 7f5d0effd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 0x7f5d1015a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:38398/0 (socket says 172.21.15.60:38398) 2023-10-16T12:52:13.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.766+0000 7f5d0effd640 1 -- 172.21.15.60:0/4189665106 learned_addr learned my addr 172.21.15.60:0/4189665106 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:13.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.766+0000 7f5d067fc640 1 --2- 172.21.15.60:0/4189665106 >> v2:172.21.15.144:3300/0 conn(0x7f5d100ccb00 0x7f5d100ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.767+0000 7f5d0effd640 1 -- 172.21.15.60:0/4189665106 >> v2:172.21.15.144:3300/0 conn(0x7f5d100ccb00 msgr2=0x7f5d100ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:13.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.767+0000 7f5d0effd640 1 --2- 172.21.15.60:0/4189665106 >> v2:172.21.15.144:3300/0 conn(0x7f5d100ccb00 0x7f5d100ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.767+0000 7f5d0effd640 1 -- 172.21.15.60:0/4189665106 >> v1:172.21.15.144:6789/0 conn(0x7f5d100ad4d0 legacy=0x7f5d100ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.767+0000 7f5d0effd640 1 -- 172.21.15.60:0/4189665106 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10070310 con 0x7f5d10153740 2023-10-16T12:52:13.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.768+0000 7f5d0effd640 1 --2- 172.21.15.60:0/4189665106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 0x7f5d1015a0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009870 tx=0x7f5d0002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f62a3856c30e686 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:13.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0e7fc640 1 -- 172.21.15.60:0/4189665106 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00031640 con 0x7f5d10153740 2023-10-16T12:52:13.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0e7fc640 1 -- 172.21.15.60:0/4189665106 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d000317a0 con 0x7f5d10153740 2023-10-16T12:52:13.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0e7fc640 1 -- 172.21.15.60:0/4189665106 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00031b30 con 0x7f5d10153740 2023-10-16T12:52:13.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 -- 172.21.15.60:0/4189665106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 msgr2=0x7f5d1015a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/4189665106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 0x7f5d1015a0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009870 tx=0x7f5d0002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 -- 172.21.15.60:0/4189665106 shutdown_connections 2023-10-16T12:52:13.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/4189665106 >> v2:172.21.15.144:3300/0 conn(0x7f5d100ccb00 0x7f5d100ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/4189665106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d10153740 0x7f5d1015a0e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 -- 172.21.15.60:0/4189665106 >> 172.21.15.60:0/4189665106 conn(0x7f5d100ce6a0 msgr2=0x7f5d100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:13.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.769+0000 7f5d0ffff640 1 -- 172.21.15.60:0/4189665106 shutdown_connections 2023-10-16T12:52:13.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.770+0000 7f5d0ffff640 1 -- 172.21.15.60:0/4189665106 wait complete. 2023-10-16T12:52:13.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.771+0000 7f5d0ffff640 1 Processor -- start 2023-10-16T12:52:13.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.771+0000 7f5d0ffff640 1 -- start start 2023-10-16T12:52:13.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.772+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d100ad4d0 0x7f5d10156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.772+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d100ccb00 0x7f5d100b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.772+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 0x7f5d100c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.772+0000 7f5d0ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10156270 con 0x7f5d10153740 2023-10-16T12:52:13.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.772+0000 7f5d0ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10147370 con 0x7f5d100ccb00 2023-10-16T12:52:13.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.772+0000 7f5d0ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100b4230 con 0x7f5d100ad4d0 2023-10-16T12:52:13.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 0x7f5d100c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 0x7f5d100c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34880/0 (socket says 172.21.15.60:34880) 2023-10-16T12:52:13.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 -- 172.21.15.60:0/469725291 learned_addr learned my addr 172.21.15.60:0/469725291 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:13.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0effd640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d100ad4d0 0x7f5d10156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d067fc640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d100ccb00 0x7f5d100b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:13.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 -- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d100ad4d0 msgr2=0x7f5d10156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d100ad4d0 0x7f5d10156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 -- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d100ccb00 msgr2=0x7f5d100b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d100ccb00 0x7f5d100b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d0f7fe640 1 -- 172.21.15.60:0/469725291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d000094d0 con 0x7f5d10153740 2023-10-16T12:52:13.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.773+0000 7f5d067fc640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d100ccb00 0x7f5d100b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:13.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.774+0000 7f5d0f7fe640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 0x7f5d100c41d0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800c6e0 tx=0x7f5d0800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:13.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.774+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08003b70 con 0x7f5d10153740 2023-10-16T12:52:13.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.774+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08003cf0 con 0x7f5d10153740 2023-10-16T12:52:13.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.774+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08005310 con 0x7f5d10153740 2023-10-16T12:52:13.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.774+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d100c4710 con 0x7f5d10153740 2023-10-16T12:52:13.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.774+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d10155bb0 con 0x7f5d10153740 2023-10-16T12:52:13.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.775+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d10156bd0 con 0x7f5d10153740 2023-10-16T12:52:13.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.777+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 51) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f5d08005470 con 0x7f5d10153740 2023-10-16T12:52:13.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.777+0000 7f5d07fff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f5cf006b0b0 0x7f5cf006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:13.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.778+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f5d080899d0 con 0x7f5d10153740 2023-10-16T12:52:13.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.778+0000 7f5d0effd640 1 -- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f5cf006b0b0 msgr2=0x7f5cf006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2022371982 2023-10-16T12:52:13.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.778+0000 7f5d0effd640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f5cf006b0b0 0x7f5cf006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:13.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.782+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d08054500 con 0x7f5d10153740 2023-10-16T12:52:13.906 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:13.903+0000 7fb67e438040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:52:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.974+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d100705d0 con 0x7f5d10153740 2023-10-16T12:52:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.976+0000 7f5d07fff640 1 -- 172.21.15.60:0/469725291 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17117 (secure 0 0 0) 0x7f5d100705d0 con 0x7f5d10153740 2023-10-16T12:52:13.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:13.979 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":116,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:08.022489+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.22","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2189557198":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1153724330":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1224606105":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/840292481":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6833/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1070434778":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6832/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/4048637185":"2023-10-17T12:51:06.102326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f5cf006b0b0 msgr2=0x7f5cf006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f5cf006b0b0 0x7f5cf006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 msgr2=0x7f5d100c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 0x7f5d100c41d0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800c6e0 tx=0x7f5d0800cba0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 shutdown_connections 2023-10-16T12:52:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d10153740 0x7f5d100c41d0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d100ccb00 0x7f5d100b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.978+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:6832/2022371982,v1:172.21.15.144:6833/2022371982] conn(0x7f5cf006b0b0 0x7f5cf006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.979+0000 7f5d0ffff640 1 --2- 172.21.15.60:0/469725291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5d100ad4d0 0x7f5d10156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.979+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 >> 172.21.15.60:0/469725291 conn(0x7f5d100ce6a0 msgr2=0x7f5d100ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.979+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 shutdown_connections 2023-10-16T12:52:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:13.979+0000 7f5d0ffff640 1 -- 172.21.15.60:0/469725291 wait complete. 2023-10-16T12:52:13.999 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:50:49.355185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:50:50.439078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:52:14.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:52:14.010 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.007+0000 7f11a8e1d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:52:14.066 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.063+0000 7f11a8e1d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:52:14.180 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.178+0000 7f11a8e1d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:52:14.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.219+0000 7fbe693ee640 1 Processor -- start 2023-10-16T12:52:14.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.219+0000 7fbe693ee640 1 -- start start 2023-10-16T12:52:14.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.220+0000 7fbe693ee640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.220+0000 7fbe693ee640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbe640c9ca0 0x7fbe6406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.220+0000 7fbe693ee640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe640cb030 con 0x7fbe640ca3b0 2023-10-16T12:52:14.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.220+0000 7fbe693ee640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe640c91c0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.220+0000 7fbe693ee640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c94a0 con 0x7fbe6415ad70 2023-10-16T12:52:14.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe627fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbe640c9ca0 0x7fbe6406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:38408/0 (socket says 172.21.15.60:38408) 2023-10-16T12:52:14.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1510457020 learned_addr learned my addr 172.21.15.60:0/1510457020 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:14.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1510457020 >> v2:172.21.15.60:3300/0 conn(0x7fbe640c9ca0 msgr2=0x7fbe6406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:14.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 --2- 172.21.15.60:0/1510457020 >> v2:172.21.15.60:3300/0 conn(0x7fbe640c9ca0 0x7fbe6406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1510457020 >> v1:172.21.15.60:6789/0 conn(0x7fbe640ca3b0 legacy=0x7fbe640ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.221+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1510457020 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640c9780 con 0x7fbe6415ad70 2023-10-16T12:52:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.222+0000 7fbe62ffd640 1 --2- 172.21.15.60:0/1510457020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 0x7fbe64069da0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009e50 tx=0x7fbe5402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c5a10e4e8aec1fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.223+0000 7fbe61ffb640 1 -- 172.21.15.60:0/1510457020 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5402cb00 con 0x7fbe6415ad70 2023-10-16T12:52:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.223+0000 7fbe61ffb640 1 -- 172.21.15.60:0/1510457020 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5402cc60 con 0x7fbe6415ad70 2023-10-16T12:52:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.223+0000 7fbe61ffb640 1 -- 172.21.15.60:0/1510457020 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe540343f0 con 0x7fbe6415ad70 2023-10-16T12:52:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.223+0000 7fbe693ee640 1 -- 172.21.15.60:0/1510457020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 msgr2=0x7fbe64069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.223+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1510457020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 0x7fbe64069da0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009e50 tx=0x7fbe5402bf60 comp rx=0 tx=0).stop 2023-10-16T12:52:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.224+0000 7fbe693ee640 1 -- 172.21.15.60:0/1510457020 shutdown_connections 2023-10-16T12:52:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.224+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1510457020 >> v2:172.21.15.60:3300/0 conn(0x7fbe640c9ca0 0x7fbe6406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.224+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1510457020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.224+0000 7fbe693ee640 1 -- 172.21.15.60:0/1510457020 >> 172.21.15.60:0/1510457020 conn(0x7fbe640ce9b0 msgr2=0x7fbe640cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:14.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.224+0000 7fbe693ee640 1 -- 172.21.15.60:0/1510457020 shutdown_connections 2023-10-16T12:52:14.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.224+0000 7fbe693ee640 1 -- 172.21.15.60:0/1510457020 wait complete. 2023-10-16T12:52:14.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.226+0000 7fbe693ee640 1 Processor -- start 2023-10-16T12:52:14.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.226+0000 7fbe693ee640 1 -- start start 2023-10-16T12:52:14.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe693ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 0x7fbe6415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe693ee640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe640ca3b0 0x7fbe640c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe693ee640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe693ee640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c7e50 con 0x7fbe640c9ca0 2023-10-16T12:52:14.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe693ee640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c8130 con 0x7fbe6415ad70 2023-10-16T12:52:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe693ee640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c8410 con 0x7fbe640ca3b0 2023-10-16T12:52:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 0x7fbe6415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 0x7fbe6415dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34910/0 (socket says 172.21.15.60:34910) 2023-10-16T12:52:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1964470843 learned_addr learned my addr 172.21.15.60:0/1964470843 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe627fc640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe640ca3b0 0x7fbe640c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe637fe640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe640ca3b0 msgr2=0x7fbe640c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe640ca3b0 0x7fbe640c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6415ad70 msgr2=0x7fbe64069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.227+0000 7fbe62ffd640 1 -- 172.21.15.60:0/1964470843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54009b30 con 0x7fbe640c9ca0 2023-10-16T12:52:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe637fe640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe62ffd640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 0x7fbe6415dd80 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fbe54000f80 tx=0x7fbe54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54040670 con 0x7fbe640c9ca0 2023-10-16T12:52:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe540407d0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54040b80 con 0x7fbe640c9ca0 2023-10-16T12:52:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640c86f0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.228+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe640c7a20 con 0x7fbe640c9ca0 2023-10-16T12:52:14.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.229+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6415bdd0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.231+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 52) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbe54040ce0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.232+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fbe540ac3f0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.240+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe540775a0 con 0x7fbe640c9ca0 2023-10-16T12:52:14.245 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:14.243+0000 7fb67e438040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:52:14.273 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112dd12640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.273 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112dd12640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.274 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112dd12640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.274 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112dd12640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.274 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112dd12640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.275 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112a50b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.275 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112a50b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.275 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112a50b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.275 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112a50b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.276 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:14.271+0000 7f112a50b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:14.312 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:14.309+0000 7fb67e438040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:52:14.436 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:14.433+0000 7fb67e438040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:52:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.439+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe6404e160 con 0x7fbe640c9ca0 2023-10-16T12:52:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.440+0000 7fbe5b7fe640 1 -- 172.21.15.60:0/1964470843 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe54009510 con 0x7fbe640c9ca0 2023-10-16T12:52:14.443 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:52:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 msgr2=0x7fbe6415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 0x7fbe6415dd80 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fbe54000f80 tx=0x7fbe54004a40 comp rx=0 tx=0).stop 2023-10-16T12:52:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 shutdown_connections 2023-10-16T12:52:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe640ca3b0 0x7fbe640c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe6415ad70 0x7fbe64069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 --2- 172.21.15.60:0/1964470843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe640c9ca0 0x7fbe6415dd80 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 >> 172.21.15.60:0/1964470843 conn(0x7fbe640ce9b0 msgr2=0x7fbe640ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 shutdown_connections 2023-10-16T12:52:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.443+0000 7fbe693ee640 1 -- 172.21.15.60:0/1964470843 wait complete. 2023-10-16T12:52:14.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:52:14.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.672+0000 7f5e3b656640 1 Processor -- start 2023-10-16T12:52:14.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.672+0000 7f5e3b656640 1 -- start start 2023-10-16T12:52:14.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.673+0000 7f5e3b656640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e3415af80 0x7f5e340cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.673+0000 7f5e3b656640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e34065ab0 0x7f5e34065f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.673+0000 7f5e3b656640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e34066450 con 0x7f5e3406faf0 2023-10-16T12:52:14.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.673+0000 7f5e3b656640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e340d0930 con 0x7f5e34065ab0 2023-10-16T12:52:14.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.673+0000 7f5e3b656640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3415eff0 con 0x7f5e3415af80 2023-10-16T12:52:14.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.674+0000 7f5e39bf3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e34065ab0 0x7f5e34065f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.674+0000 7f5e39bf3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e34065ab0 0x7f5e34065f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34924/0 (socket says 172.21.15.60:34924) 2023-10-16T12:52:14.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.674+0000 7f5e39bf3640 1 -- 172.21.15.60:0/3919744707 learned_addr learned my addr 172.21.15.60:0/3919744707 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:14.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.674+0000 7f5e393f2640 1 --2- 172.21.15.60:0/3919744707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e3415af80 0x7f5e340cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.675+0000 7f5e39bf3640 1 -- 172.21.15.60:0/3919744707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e3415af80 msgr2=0x7f5e340cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:14.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.675+0000 7f5e39bf3640 1 --2- 172.21.15.60:0/3919744707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e3415af80 0x7f5e340cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.675+0000 7f5e39bf3640 1 -- 172.21.15.60:0/3919744707 >> v1:172.21.15.144:6789/0 conn(0x7f5e3406faf0 legacy=0x7f5e3415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.675+0000 7f5e39bf3640 1 -- 172.21.15.60:0/3919744707 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3415ecc0 con 0x7f5e34065ab0 2023-10-16T12:52:14.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.676+0000 7f5e39bf3640 1 --2- 172.21.15.60:0/3919744707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e34065ab0 0x7f5e34065f10 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009e50 tx=0x7f5e2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd9095e868acf4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e2bfff640 1 -- 172.21.15.60:0/3919744707 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c02ca70 con 0x7f5e34065ab0 2023-10-16T12:52:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e2bfff640 1 -- 172.21.15.60:0/3919744707 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c02cbd0 con 0x7f5e34065ab0 2023-10-16T12:52:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e2bfff640 1 -- 172.21.15.60:0/3919744707 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0343f0 con 0x7f5e34065ab0 2023-10-16T12:52:14.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e3b656640 1 -- 172.21.15.60:0/3919744707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e34065ab0 msgr2=0x7f5e34065f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e3b656640 1 --2- 172.21.15.60:0/3919744707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e34065ab0 0x7f5e34065f10 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009e50 tx=0x7f5e2c02bed0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e3b656640 1 -- 172.21.15.60:0/3919744707 shutdown_connections 2023-10-16T12:52:14.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e3b656640 1 --2- 172.21.15.60:0/3919744707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e34065ab0 0x7f5e34065f10 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e3b656640 1 --2- 172.21.15.60:0/3919744707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e3415af80 0x7f5e340cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.677+0000 7f5e3b656640 1 -- 172.21.15.60:0/3919744707 >> 172.21.15.60:0/3919744707 conn(0x7f5e340d0ab0 msgr2=0x7f5e340d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:14.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.678+0000 7f5e3b656640 1 -- 172.21.15.60:0/3919744707 shutdown_connections 2023-10-16T12:52:14.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.678+0000 7f5e3b656640 1 -- 172.21.15.60:0/3919744707 wait complete. 2023-10-16T12:52:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.679+0000 7f5e3b656640 1 Processor -- start 2023-10-16T12:52:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.679+0000 7f5e3b656640 1 -- start start 2023-10-16T12:52:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.680+0000 7f5e3b656640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e34065ab0 0x7f5e3414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.680+0000 7f5e3b656640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 0x7f5e3415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.680+0000 7f5e3b656640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e3415af80 0x7f5e340c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.680+0000 7f5e3b656640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3415d800 con 0x7f5e3406faf0 2023-10-16T12:52:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.680+0000 7f5e3b656640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3415c430 con 0x7f5e3415af80 2023-10-16T12:52:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.680+0000 7f5e3b656640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3415ce60 con 0x7f5e34065ab0 2023-10-16T12:52:14.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 0x7f5e3415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 0x7f5e3415dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34938/0 (socket says 172.21.15.60:34938) 2023-10-16T12:52:14.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 -- 172.21.15.60:0/2598817156 learned_addr learned my addr 172.21.15.60:0/2598817156 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:14.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e39bf3640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e3415af80 0x7f5e340c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e393f2640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e34065ab0 0x7f5e3414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:14.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 -- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e34065ab0 msgr2=0x7f5e3414aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e34065ab0 0x7f5e3414aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 -- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e3415af80 msgr2=0x7f5e340c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e3415af80 0x7f5e340c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.681+0000 7f5e38bf1640 1 -- 172.21.15.60:0/2598817156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c009b30 con 0x7f5e3406faf0 2023-10-16T12:52:14.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.682+0000 7f5e38bf1640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 0x7f5e3415dc20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c400 tx=0x7f5e2400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.682+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240052d0 con 0x7f5e3406faf0 2023-10-16T12:52:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.682+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e24005430 con 0x7f5e3406faf0 2023-10-16T12:52:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.682+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24003830 con 0x7f5e3406faf0 2023-10-16T12:52:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.682+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3415d140 con 0x7f5e3406faf0 2023-10-16T12:52:14.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.682+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3414b430 con 0x7f5e3406faf0 2023-10-16T12:52:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.683+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3415d940 con 0x7f5e3406faf0 2023-10-16T12:52:14.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.684+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 52) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f5e24003990 con 0x7f5e3406faf0 2023-10-16T12:52:14.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.685+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f5e240889f0 con 0x7f5e3406faf0 2023-10-16T12:52:14.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.689+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e3415d940 con 0x7f5e3406faf0 2023-10-16T12:52:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.884+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e3415d940 con 0x7f5e3406faf0 2023-10-16T12:52:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.884+0000 7f5e29ffb640 1 -- 172.21.15.60:0/2598817156 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e3415d940 con 0x7f5e3406faf0 2023-10-16T12:52:14.887 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:52:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 msgr2=0x7f5e3415dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 0x7f5e3415dc20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c400 tx=0x7f5e2400c8c0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 shutdown_connections 2023-10-16T12:52:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e3415af80 0x7f5e340c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e3406faf0 0x7f5e3415dc20 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 --2- 172.21.15.60:0/2598817156 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e34065ab0 0x7f5e3414aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.887+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 >> 172.21.15.60:0/2598817156 conn(0x7f5e340d0ab0 msgr2=0x7f5e340ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.888+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 shutdown_connections 2023-10-16T12:52:14.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:14.888+0000 7f5e3b656640 1 -- 172.21.15.60:0/2598817156 wait complete. 2023-10-16T12:52:14.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:52:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.132+0000 7f58f08ff640 1 Processor -- start 2023-10-16T12:52:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.132+0000 7f58f08ff640 1 -- start start 2023-10-16T12:52:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58f08ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 0x7f58ec0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58f08ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec15b9d0 con 0x7f58ec0ccfb0 2023-10-16T12:52:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58f08ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec15bcb0 con 0x7f58ec06c040 2023-10-16T12:52:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58f08ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec15bf90 con 0x7f58ec0ad970 2023-10-16T12:52:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58ebfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f58ec0ccfb0 0x7f58ec0cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50310/0 (socket says 172.21.15.60:50310) 2023-10-16T12:52:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58ebfff640 1 -- 172.21.15.60:0/4011343980 learned_addr learned my addr 172.21.15.60:0/4011343980 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:15.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.134+0000 7f58e3fff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 0x7f58ec0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581966527 0 0) 0x7f58ec15b9d0 con 0x7f58ec0ccfb0 2023-10-16T12:52:15.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8003520 con 0x7f58ec0ccfb0 2023-10-16T12:52:15.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215218336 0 0) 0x7f58d8003520 con 0x7f58ec0ccfb0 2023-10-16T12:52:15.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 msgr2=0x7f58ec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:15.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 0x7f58ec0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 >> v1:172.21.15.144:6789/0 conn(0x7f58ec06c040 legacy=0x7f58ec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0cc270 con 0x7f58ec0ccfb0 2023-10-16T12:52:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f58e40019f0 con 0x7f58ec0ccfb0 2023-10-16T12:52:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 >> v1:172.21.15.60:6789/0 conn(0x7f58ec0ccfb0 legacy=0x7f58ec0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 0x7f58d80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58d8003f70 0x7f58d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8004d00 0x7f58d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec15bf90 con 0x7f58d8004d00 2023-10-16T12:52:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d800bac0 con 0x7f58d8003f70 2023-10-16T12:52:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.135+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d800bc80 con 0x7f58ec0ad970 2023-10-16T12:52:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58ebfff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8004d00 0x7f58d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58eb7fe640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 0x7f58d80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58e3fff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58d8003f70 0x7f58d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58ebfff640 1 -- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 msgr2=0x7f58d80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:15.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58ebfff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58ec0ad970 0x7f58d80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58ebfff640 1 -- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58d8003f70 msgr2=0x7f58d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:15.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58ebfff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58d8003f70 0x7f58d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.136+0000 7f58ebfff640 1 -- 172.21.15.60:0/4011343980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d8003520 con 0x7f58d8004d00 2023-10-16T12:52:15.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.137+0000 7f58ebfff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8004d00 0x7f58d8005180 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f58e400a760 tx=0x7f58d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18a58b7dd67349d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:15.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.137+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4034630 con 0x7f58d8004d00 2023-10-16T12:52:15.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e402f070 con 0x7f58d8004d00 2023-10-16T12:52:15.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58eaffd640 1 -- 172.21.15.60:0/4011343980 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e40349f0 con 0x7f58d8004d00 2023-10-16T12:52:15.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58f08ff640 1 -- 172.21.15.60:0/4011343980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8004d00 msgr2=0x7f58d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58f08ff640 1 --2- 172.21.15.60:0/4011343980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58d8004d00 0x7f58d8005180 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f58e400a760 tx=0x7f58d80057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58ebfff640 1 -- 172.21.15.60:0/4011343980 reap_dead start 2023-10-16T12:52:15.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58f08ff640 1 -- 172.21.15.60:0/4011343980 shutdown_connections 2023-10-16T12:52:15.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58f08ff640 1 -- 172.21.15.60:0/4011343980 >> 172.21.15.60:0/4011343980 conn(0x7f58ec0ce6a0 msgr2=0x7f58ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:15.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58f08ff640 1 -- 172.21.15.60:0/4011343980 shutdown_connections 2023-10-16T12:52:15.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.138+0000 7f58f08ff640 1 -- 172.21.15.60:0/4011343980 wait complete. 2023-10-16T12:52:15.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.140+0000 7f58f08ff640 1 Processor -- start 2023-10-16T12:52:15.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.140+0000 7f58f08ff640 1 -- start start 2023-10-16T12:52:15.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.141+0000 7f58f08ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58d8004d00 0x7f58ec0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.141+0000 7f58f08ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 0x7f58ec153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.141+0000 7f58f08ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58ec159e10 0x7f58ec15a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.141+0000 7f58f08ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec15a7d0 con 0x7f58ec0c21d0 2023-10-16T12:52:15.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.141+0000 7f58f08ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0c8cf0 con 0x7f58ec159e10 2023-10-16T12:52:15.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.141+0000 7f58f08ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec153c80 con 0x7f58d8004d00 2023-10-16T12:52:15.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 0x7f58ec153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 0x7f58ec153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34962/0 (socket says 172.21.15.60:34962) 2023-10-16T12:52:15.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 -- 172.21.15.60:0/1447610517 learned_addr learned my addr 172.21.15.60:0/1447610517 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:15.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58ebfff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58ec159e10 0x7f58ec15a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 -- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58d8004d00 msgr2=0x7f58ec0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58d8004d00 0x7f58ec0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 -- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58ec159e10 msgr2=0x7f58ec15a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58ec159e10 0x7f58ec15a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 -- 172.21.15.60:0/1447610517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0cdcd0 con 0x7f58ec0c21d0 2023-10-16T12:52:15.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.142+0000 7f58e3fff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 0x7f58ec153740 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f58d4011630 tx=0x7f58d4011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:15.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.143+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400f920 con 0x7f58ec0c21d0 2023-10-16T12:52:15.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.143+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d400fa80 con 0x7f58ec0c21d0 2023-10-16T12:52:15.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.143+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400fdf0 con 0x7f58ec0c21d0 2023-10-16T12:52:15.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.143+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec0c2940 con 0x7f58ec0c21d0 2023-10-16T12:52:15.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.143+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec0c2f00 con 0x7f58ec0c21d0 2023-10-16T12:52:15.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.144+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec06b330 con 0x7f58ec0c21d0 2023-10-16T12:52:15.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.145+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 52) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f58d4015460 con 0x7f58ec0c21d0 2023-10-16T12:52:15.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.147+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f58d4089990 con 0x7f58ec0c21d0 2023-10-16T12:52:15.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.152+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58d4054aa0 con 0x7f58ec0c21d0 2023-10-16T12:52:15.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.212+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 53) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f58d4054550 con 0x7f58ec0c21d0 2023-10-16T12:52:15.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.212+0000 7f58e8ff9640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f58cc070690 0x7f58cc072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.213+0000 7f58eb7fe640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f58cc070690 0x7f58cc072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.221+0000 7f58eb7fe640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f58cc070690 0x7f58cc072a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007fd0 tx=0x7f58dc009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.358+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58ec04e9b0 con 0x7f58ec0c21d0 2023-10-16T12:52:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.359+0000 7f58e8ff9640 1 -- 172.21.15.60:0/1447610517 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58d40862a0 con 0x7f58ec0c21d0 2023-10-16T12:52:15.362 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:52:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.361+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f58cc070690 msgr2=0x7f58cc072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.361+0000 7f58f08ff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f58cc070690 0x7f58cc072a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007fd0 tx=0x7f58dc009040 comp rx=0 tx=0).stop 2023-10-16T12:52:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 msgr2=0x7f58ec153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 0x7f58ec153740 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f58d4011630 tx=0x7f58d4011af0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 shutdown_connections 2023-10-16T12:52:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f58cc070690 0x7f58cc072a80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58ec159e10 0x7f58ec15a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58ec0c21d0 0x7f58ec153740 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 --2- 172.21.15.60:0/1447610517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58d8004d00 0x7f58ec0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 >> 172.21.15.60:0/1447610517 conn(0x7f58ec0ce6a0 msgr2=0x7f58ec0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 shutdown_connections 2023-10-16T12:52:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.362+0000 7f58f08ff640 1 -- 172.21.15.60:0/1447610517 wait complete. 2023-10-16T12:52:15.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.602+0000 7ff3d1e95640 1 Processor -- start 2023-10-16T12:52:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.602+0000 7ff3d1e95640 1 -- start start 2023-10-16T12:52:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.603+0000 7ff3d1e95640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 0x7ff3cc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.603+0000 7ff3d1e95640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3cc0ad970 0x7ff3cc15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.603+0000 7ff3d1e95640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3cc15ebc0 0x7ff3cc15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.603+0000 7ff3d1e95640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc06c960 con 0x7ff3cc0ad970 2023-10-16T12:52:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.603+0000 7ff3d1e95640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc15b9d0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.603+0000 7ff3d1e95640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc15bcb0 con 0x7ff3cc06c040 2023-10-16T12:52:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3cc0ad970 0x7ff3cc15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3cc0ad970 0x7ff3cc15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34970/0 (socket says 172.21.15.60:34970) 2023-10-16T12:52:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 -- 172.21.15.60:0/3873885682 learned_addr learned my addr 172.21.15.60:0/3873885682 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 -- 172.21.15.60:0/3873885682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 msgr2=0x7ff3cc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 --2- 172.21.15.60:0/3873885682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 0x7ff3cc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 -- 172.21.15.60:0/3873885682 >> v2:172.21.15.144:3300/0 conn(0x7ff3cc15ebc0 msgr2=0x7ff3cc15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 --2- 172.21.15.60:0/3873885682 >> v2:172.21.15.144:3300/0 conn(0x7ff3cc15ebc0 0x7ff3cc15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.604+0000 7ff3cbfff640 1 -- 172.21.15.60:0/3873885682 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc15bf90 con 0x7ff3cc0ad970 2023-10-16T12:52:15.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.605+0000 7ff3cbfff640 1 --2- 172.21.15.60:0/3873885682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc15e680 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009870 tx=0x7ff3c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa2ec88f224d45a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:15.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3cb7fe640 1 -- 172.21.15.60:0/3873885682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0031640 con 0x7ff3cc0ad970 2023-10-16T12:52:15.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3cb7fe640 1 -- 172.21.15.60:0/3873885682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c00317a0 con 0x7ff3cc0ad970 2023-10-16T12:52:15.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3cb7fe640 1 -- 172.21.15.60:0/3873885682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0031b30 con 0x7ff3cc0ad970 2023-10-16T12:52:15.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 -- 172.21.15.60:0/3873885682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc0ad970 msgr2=0x7ff3cc15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/3873885682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc15e680 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009870 tx=0x7ff3c002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 -- 172.21.15.60:0/3873885682 shutdown_connections 2023-10-16T12:52:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/3873885682 >> v2:172.21.15.144:3300/0 conn(0x7ff3cc15ebc0 0x7ff3cc15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/3873885682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc15e680 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/3873885682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 0x7ff3cc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 -- 172.21.15.60:0/3873885682 >> 172.21.15.60:0/3873885682 conn(0x7ff3cc0ce6a0 msgr2=0x7ff3cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.606+0000 7ff3d1e95640 1 -- 172.21.15.60:0/3873885682 shutdown_connections 2023-10-16T12:52:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.607+0000 7ff3d1e95640 1 -- 172.21.15.60:0/3873885682 wait complete. 2023-10-16T12:52:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.608+0000 7ff3d1e95640 1 Processor -- start 2023-10-16T12:52:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.608+0000 7ff3d1e95640 1 -- start start 2023-10-16T12:52:15.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.609+0000 7ff3d1e95640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 0x7ff3cc0c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.609+0000 7ff3d1e95640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc0afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.609+0000 7ff3d1e95640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 0x7ff3cc0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.609+0000 7ff3d1e95640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c81c0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.609+0000 7ff3d1e95640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c7c00 con 0x7ff3cc0ad970 2023-10-16T12:52:15.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.609+0000 7ff3d1e95640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c7ee0 con 0x7ff3cc06c040 2023-10-16T12:52:15.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 0x7ff3cc0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 0x7ff3cc0b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34986/0 (socket says 172.21.15.60:34986) 2023-10-16T12:52:15.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 -- 172.21.15.60:0/4019906308 learned_addr learned my addr 172.21.15.60:0/4019906308 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:15.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3cbfff640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc0afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 -- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 msgr2=0x7ff3cc0c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 0x7ff3cc0c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 -- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3cc0ad970 msgr2=0x7ff3cc0afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc0afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.610+0000 7ff3d1694640 1 -- 172.21.15.60:0/4019906308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00094d0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.611+0000 7ff3d1694640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 0x7ff3cc0b0430 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400b970 tx=0x7ff3c400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.611+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400cb40 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.611+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c400ccc0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.611+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4014680 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.611+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc0ae2a0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.611+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc1f23f0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.612+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc06b330 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.613+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 53) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff3c401d3b0 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.615+0000 7ff3c97fa640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ff3b806acf0 0x7ff3b806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.615+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7ff3c4089550 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.615+0000 7ff3d0e93640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ff3b806acf0 0x7ff3b806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.617+0000 7ff3d0e93640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ff3b806acf0 0x7ff3b806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009670 tx=0x7ff3bc006bf0 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.620+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3c4054570 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.855+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3cc0ae100 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.856+0000 7ff3c97fa640 1 -- 172.21.15.60:0/4019906308 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4915 (secure 0 0 0) 0x7ff3cc0ae100 con 0x7ff3cc15ebc0 2023-10-16T12:52:15.859 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:15.859 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:51.414872+0000","modified":"2023-10-16T12:50:54.480441+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7353":{"gid":7353,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/3569389918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3569389918},{"type":"v1","addr":"172.21.15.60:6835","nonce":3569389918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:52:15.859 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 53 2023-10-16T12:52:15.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ff3b806acf0 msgr2=0x7ff3b806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ff3b806acf0 0x7ff3b806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009670 tx=0x7ff3bc006bf0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 msgr2=0x7ff3cc0b0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 0x7ff3cc0b0430 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400b970 tx=0x7ff3c400be30 comp rx=0 tx=0).stop 2023-10-16T12:52:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 shutdown_connections 2023-10-16T12:52:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ff3b806acf0 0x7ff3b806d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3cc15ebc0 0x7ff3cc0b0430 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3cc0ad970 0x7ff3cc0afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 --2- 172.21.15.60:0/4019906308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3cc06c040 0x7ff3cc0c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.858+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 >> 172.21.15.60:0/4019906308 conn(0x7ff3cc0ce6a0 msgr2=0x7ff3cc0c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.859+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 shutdown_connections 2023-10-16T12:52:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:15.859+0000 7ff3d1e95640 1 -- 172.21.15.60:0/4019906308 wait complete. 2023-10-16T12:52:15.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.107+0000 7fdd5e747640 1 Processor -- start 2023-10-16T12:52:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.107+0000 7fdd5e747640 1 -- start start 2023-10-16T12:52:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.108+0000 7fdd5e747640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd580c9060 0x7fdd5806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.108+0000 7fdd5e747640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd5806c9e0 0x7fdd5806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.108+0000 7fdd5e747640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd580c9440 0x7fdd58164d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.108+0000 7fdd5e747640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580ca560 con 0x7fdd580c9440 2023-10-16T12:52:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.108+0000 7fdd5e747640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580c9950 con 0x7fdd5806c9e0 2023-10-16T12:52:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.108+0000 7fdd5e747640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c9c30 con 0x7fdd580c9060 2023-10-16T12:52:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.109+0000 7fdd5df46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd580c9440 0x7fdd58164d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.109+0000 7fdd5df46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd580c9440 0x7fdd58164d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34990/0 (socket says 172.21.15.60:34990) 2023-10-16T12:52:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.109+0000 7fdd5df46640 1 -- 172.21.15.60:0/92165405 learned_addr learned my addr 172.21.15.60:0/92165405 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.109+0000 7fdd5cf44640 1 --2- 172.21.15.60:0/92165405 >> v2:172.21.15.144:3300/0 conn(0x7fdd5806c9e0 0x7fdd5806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.109+0000 7fdd5d745640 1 --2- 172.21.15.60:0/92165405 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd580c9060 0x7fdd5806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.110+0000 7fdd5df46640 1 -- 172.21.15.60:0/92165405 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd580c9060 msgr2=0x7fdd5806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.110+0000 7fdd5df46640 1 --2- 172.21.15.60:0/92165405 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd580c9060 0x7fdd5806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.110+0000 7fdd5df46640 1 -- 172.21.15.60:0/92165405 >> v2:172.21.15.144:3300/0 conn(0x7fdd5806c9e0 msgr2=0x7fdd5806f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.110+0000 7fdd5df46640 1 --2- 172.21.15.60:0/92165405 >> v2:172.21.15.144:3300/0 conn(0x7fdd5806c9e0 0x7fdd5806f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.110+0000 7fdd5df46640 1 -- 172.21.15.60:0/92165405 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580c9f10 con 0x7fdd580c9440 2023-10-16T12:52:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.111+0000 7fdd5df46640 1 --2- 172.21.15.60:0/92165405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd58164d50 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fdd580719e0 tx=0x7fdd5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd210ad23419f99a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.112+0000 7fdd4bfff640 1 -- 172.21.15.60:0/92165405 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50031870 con 0x7fdd580c9440 2023-10-16T12:52:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.112+0000 7fdd4bfff640 1 -- 172.21.15.60:0/92165405 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd500319d0 con 0x7fdd580c9440 2023-10-16T12:52:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.112+0000 7fdd4bfff640 1 -- 172.21.15.60:0/92165405 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50031d60 con 0x7fdd580c9440 2023-10-16T12:52:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 -- 172.21.15.60:0/92165405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 msgr2=0x7fdd58164d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 --2- 172.21.15.60:0/92165405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd58164d50 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fdd580719e0 tx=0x7fdd5002b910 comp rx=0 tx=0).stop 2023-10-16T12:52:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 -- 172.21.15.60:0/92165405 shutdown_connections 2023-10-16T12:52:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 --2- 172.21.15.60:0/92165405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd58164d50 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 --2- 172.21.15.60:0/92165405 >> v2:172.21.15.144:3300/0 conn(0x7fdd5806c9e0 0x7fdd5806f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 --2- 172.21.15.60:0/92165405 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd580c9060 0x7fdd5806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 -- 172.21.15.60:0/92165405 >> 172.21.15.60:0/92165405 conn(0x7fdd580ce6a0 msgr2=0x7fdd580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 -- 172.21.15.60:0/92165405 shutdown_connections 2023-10-16T12:52:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.113+0000 7fdd5e747640 1 -- 172.21.15.60:0/92165405 wait complete. 2023-10-16T12:52:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.116+0000 7fdd5e747640 1 Processor -- start 2023-10-16T12:52:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.116+0000 7fdd5e747640 1 -- start start 2023-10-16T12:52:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5e747640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd5806c9e0 0x7fdd580c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5e747640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd580c9060 0x7fdd5815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5e747640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd580c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5e747640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c3df0 con 0x7fdd580c9440 2023-10-16T12:52:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5e747640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c47c0 con 0x7fdd580c9060 2023-10-16T12:52:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5e747640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c3530 con 0x7fdd5806c9e0 2023-10-16T12:52:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.117+0000 7fdd5df46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd580c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd580c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35002/0 (socket says 172.21.15.60:35002) 2023-10-16T12:52:16.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 -- 172.21.15.60:0/2113569531 learned_addr learned my addr 172.21.15.60:0/2113569531 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:16.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5cf44640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd580c9060 0x7fdd5815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 -- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd5806c9e0 msgr2=0x7fdd580c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd5806c9e0 0x7fdd580c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 -- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd580c9060 msgr2=0x7fdd5815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd580c9060 0x7fdd5815fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 -- 172.21.15.60:0/2113569531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500094d0 con 0x7fdd580c9440 2023-10-16T12:52:16.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5cf44640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd580c9060 0x7fdd5815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:16.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.118+0000 7fdd5df46640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd580c80a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fdd500090b0 tx=0x7fdd500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:16.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.119+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5003ac60 con 0x7fdd580c9440 2023-10-16T12:52:16.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.119+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5003ade0 con 0x7fdd580c9440 2023-10-16T12:52:16.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.119+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd58163ab0 con 0x7fdd580c9440 2023-10-16T12:52:16.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.119+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50044630 con 0x7fdd580c9440 2023-10-16T12:52:16.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.119+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd580c3a80 con 0x7fdd580c9440 2023-10-16T12:52:16.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.120+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5804e9b0 con 0x7fdd580c9440 2023-10-16T12:52:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.121+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 53) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fdd5004d450 con 0x7fdd580c9440 2023-10-16T12:52:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.122+0000 7fdd49ffb640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdd3c06ad40 0x7fdd3c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.123+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fdd5002d040 con 0x7fdd580c9440 2023-10-16T12:52:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.123+0000 7fdd5d745640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdd3c06ad40 0x7fdd3c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.124+0000 7fdd5d745640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdd3c06ad40 0x7fdd3c06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006a90 tx=0x7fdd4c006a20 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.129+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd50076aa0 con 0x7fdd580c9440 2023-10-16T12:52:16.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.222+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 54) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdd50076e70 con 0x7fdd580c9440 2023-10-16T12:52:16.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.337+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd580c3f30 con 0x7fdd580c9440 2023-10-16T12:52:16.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.338+0000 7fdd49ffb640 1 -- 172.21.15.60:0/2113569531 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+17749 (secure 0 0 0) 0x7fdd50090b10 con 0x7fdd580c9440 2023-10-16T12:52:16.341 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:16.342 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":117,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:14.184254+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.22","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2025887462":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/948877502":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/1510335921":"2023-10-17T12:52:14.184032+0000","172.21.15.144:6833/2022371982":"2023-10-17T12:52:14.184032+0000","172.21.15.144:6832/2022371982":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/337630602":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/671149719":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/3635246428":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/2189557198":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1153724330":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1224606105":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1935697927":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/840292481":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6833/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/737927288":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/1070434778":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6832/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/4048637185":"2023-10-17T12:51:06.102326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:16.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdd3c06ad40 msgr2=0x7fdd3c06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdd3c06ad40 0x7fdd3c06d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006a90 tx=0x7fdd4c006a20 comp rx=0 tx=0).stop 2023-10-16T12:52:16.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 msgr2=0x7fdd580c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd580c80a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fdd500090b0 tx=0x7fdd500046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 shutdown_connections 2023-10-16T12:52:16.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdd3c06ad40 0x7fdd3c06d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd580c9440 0x7fdd580c80a0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd580c9060 0x7fdd5815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 --2- 172.21.15.60:0/2113569531 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd5806c9e0 0x7fdd580c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.341+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 >> 172.21.15.60:0/2113569531 conn(0x7fdd580ce6a0 msgr2=0x7fdd580cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:16.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.342+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 shutdown_connections 2023-10-16T12:52:16.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.342+0000 7fdd5e747640 1 -- 172.21.15.60:0/2113569531 wait complete. 2023-10-16T12:52:16.365 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:52:16.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:16.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.593+0000 7f401070f640 1 Processor -- start 2023-10-16T12:52:16.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.593+0000 7f401070f640 1 -- start start 2023-10-16T12:52:16.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.594+0000 7f401070f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f400815aaa0 0x7f400815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.594+0000 7f401070f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40080ce390 0x7f4008070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.594+0000 7f401070f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400815ed60 con 0x7f400815ae80 2023-10-16T12:52:16.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.594+0000 7f401070f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400806b810 con 0x7f400815aaa0 2023-10-16T12:52:16.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.594+0000 7f401070f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080707e0 con 0x7f40080ce390 2023-10-16T12:52:16.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.594+0000 7f400e4ab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f400815aaa0 0x7f400815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400e4ab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f400815aaa0 0x7f400815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35006/0 (socket says 172.21.15.60:35006) 2023-10-16T12:52:16.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400e4ab640 1 -- 172.21.15.60:0/2335373120 learned_addr learned my addr 172.21.15.60:0/2335373120 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:16.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400ecac640 1 --2- 172.21.15.60:0/2335373120 >> v2:172.21.15.144:3300/0 conn(0x7f40080ce390 0x7f4008070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400e4ab640 1 -- 172.21.15.60:0/2335373120 >> v2:172.21.15.144:3300/0 conn(0x7f40080ce390 msgr2=0x7f4008070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:16.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400e4ab640 1 --2- 172.21.15.60:0/2335373120 >> v2:172.21.15.144:3300/0 conn(0x7f40080ce390 0x7f4008070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400e4ab640 1 -- 172.21.15.60:0/2335373120 >> v1:172.21.15.144:6789/0 conn(0x7f400815ae80 legacy=0x7f40080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:16.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400e4ab640 1 -- 172.21.15.60:0/2335373120 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400815ef90 con 0x7f400815aaa0 2023-10-16T12:52:16.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.595+0000 7f400ecac640 1 --2- 172.21.15.60:0/2335373120 >> v2:172.21.15.144:3300/0 conn(0x7f40080ce390 0x7f4008070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:16.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.596+0000 7f400e4ab640 1 --2- 172.21.15.60:0/2335373120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815aaa0 0x7f400815e820 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f40080c2490 tx=0x7f400402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7305b568ab7ca7ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:16.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.596+0000 7f400d4a9640 1 -- 172.21.15.60:0/2335373120 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004031870 con 0x7f400815aaa0 2023-10-16T12:52:16.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f400d4a9640 1 -- 172.21.15.60:0/2335373120 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40040319d0 con 0x7f400815aaa0 2023-10-16T12:52:16.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f400d4a9640 1 -- 172.21.15.60:0/2335373120 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004031d60 con 0x7f400815aaa0 2023-10-16T12:52:16.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 -- 172.21.15.60:0/2335373120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815aaa0 msgr2=0x7f400815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 --2- 172.21.15.60:0/2335373120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815aaa0 0x7f400815e820 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f40080c2490 tx=0x7f400402b910 comp rx=0 tx=0).stop 2023-10-16T12:52:16.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 -- 172.21.15.60:0/2335373120 shutdown_connections 2023-10-16T12:52:16.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 --2- 172.21.15.60:0/2335373120 >> v2:172.21.15.144:3300/0 conn(0x7f40080ce390 0x7f4008070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 --2- 172.21.15.60:0/2335373120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815aaa0 0x7f400815e820 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 -- 172.21.15.60:0/2335373120 >> 172.21.15.60:0/2335373120 conn(0x7f40080cb410 msgr2=0x7f40080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 -- 172.21.15.60:0/2335373120 shutdown_connections 2023-10-16T12:52:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.597+0000 7f401070f640 1 -- 172.21.15.60:0/2335373120 wait complete. 2023-10-16T12:52:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.599+0000 7f401070f640 1 Processor -- start 2023-10-16T12:52:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.599+0000 7f401070f640 1 -- start start 2023-10-16T12:52:16.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.600+0000 7f401070f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ce390 0x7f40081f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.600+0000 7f401070f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400815aaa0 0x7f40080ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.600+0000 7f401070f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 0x7f40081f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.600+0000 7f401070f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c65a0 con 0x7f400815ae80 2023-10-16T12:52:16.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.600+0000 7f401070f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c6880 con 0x7f40080ce390 2023-10-16T12:52:16.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.600+0000 7f401070f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c6b60 con 0x7f400815aaa0 2023-10-16T12:52:16.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 0x7f40081f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 0x7f40081f6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35016/0 (socket says 172.21.15.60:35016) 2023-10-16T12:52:16.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 -- 172.21.15.60:0/964521166 learned_addr learned my addr 172.21.15.60:0/964521166 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:16.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400e4ab640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ce390 0x7f40081f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 -- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400815aaa0 msgr2=0x7f40080ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400815aaa0 0x7f40080ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 -- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ce390 msgr2=0x7f40081f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ce390 0x7f40081f2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.601+0000 7f400ecac640 1 -- 172.21.15.60:0/964521166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40040094d0 con 0x7f400815ae80 2023-10-16T12:52:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.602+0000 7f400ecac640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 0x7f40081f6a70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f400000cef0 tx=0x7f400000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.602+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400000d840 con 0x7f400815ae80 2023-10-16T12:52:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.602+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400000d9c0 con 0x7f400815ae80 2023-10-16T12:52:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.602+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40080c6e40 con 0x7f400815ae80 2023-10-16T12:52:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.602+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400000dd70 con 0x7f400815ae80 2023-10-16T12:52:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.602+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40080ca700 con 0x7f400815ae80 2023-10-16T12:52:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.603+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400815e820 con 0x7f400815ae80 2023-10-16T12:52:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.604+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 54) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f400001e3b0 con 0x7f400815ae80 2023-10-16T12:52:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.605+0000 7f3ffeffd640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f3fec06af00 0x7f3fec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.605+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f400008a440 con 0x7f400815ae80 2023-10-16T12:52:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.606+0000 7f400e4ab640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f3fec06af00 0x7f3fec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.607+0000 7f400e4ab640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f3fec06af00 0x7f3fec06d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4004036000 tx=0x7f400403c040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.610+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4000055250 con 0x7f400815ae80 2023-10-16T12:52:16.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.845+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f400804e160 con 0x7f400815ae80 2023-10-16T12:52:16.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.846+0000 7f3ffeffd640 1 -- 172.21.15.60:0/964521166 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f4000058870 con 0x7f400815ae80 2023-10-16T12:52:16.849 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:16.849 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:51.414872+0000","modified":"2023-10-16T12:50:54.480441+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7353":{"gid":7353,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/3569389918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3569389918},{"type":"v1","addr":"172.21.15.60:6835","nonce":3569389918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:52:16.850 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 53 2023-10-16T12:52:16.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f3fec06af00 msgr2=0x7f3fec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f3fec06af00 0x7f3fec06d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4004036000 tx=0x7f400403c040 comp rx=0 tx=0).stop 2023-10-16T12:52:16.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 msgr2=0x7f40081f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:16.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 0x7f40081f6a70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f400000cef0 tx=0x7f400000a710 comp rx=0 tx=0).stop 2023-10-16T12:52:16.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 shutdown_connections 2023-10-16T12:52:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f3fec06af00 0x7f3fec06d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400815aaa0 0x7f40080ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400815ae80 0x7f40081f6a70 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 --2- 172.21.15.60:0/964521166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ce390 0x7f40081f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 >> 172.21.15.60:0/964521166 conn(0x7f40080cb410 msgr2=0x7f40080b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 shutdown_connections 2023-10-16T12:52:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:16.849+0000 7f401070f640 1 -- 172.21.15.60:0/964521166 wait complete. 2023-10-16T12:52:16.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:17.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.097+0000 7f11511ff640 1 Processor -- start 2023-10-16T12:52:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.097+0000 7f11511ff640 1 -- start start 2023-10-16T12:52:17.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.098+0000 7f11511ff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f114c070b60 0x7f114c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.098+0000 7f11511ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 0x7f114c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f11511ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c15aef0 con 0x7f114c159950 2023-10-16T12:52:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f11511ff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c15e490 con 0x7f114c070b60 2023-10-16T12:52:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f11511ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15d910 con 0x7f114c06c100 2023-10-16T12:52:17.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f114bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f114c159950 0x7f114c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50322/0 (socket says 172.21.15.60:50322) 2023-10-16T12:52:17.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f114bfff640 1 -- 172.21.15.60:0/3978101881 learned_addr learned my addr 172.21.15.60:0/3978101881 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:17.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f11509fe640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 0x7f114c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f114b7fe640 1 --2- 172.21.15.60:0/3978101881 >> v2:172.21.15.144:3300/0 conn(0x7f114c070b60 0x7f114c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.099+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764784029 0 0) 0x7f113c001490 con 0x7f114c159950 2023-10-16T12:52:17.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1134005730 con 0x7f114c159950 2023-10-16T12:52:17.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688006980 0 0) 0x7f1134005730 con 0x7f114c159950 2023-10-16T12:52:17.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 msgr2=0x7f114c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:17.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 0x7f114c06c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 >> v2:172.21.15.144:3300/0 conn(0x7f114c070b60 msgr2=0x7f114c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 --2- 172.21.15.60:0/3978101881 >> v2:172.21.15.144:3300/0 conn(0x7f114c070b60 0x7f114c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c15dbf0 con 0x7f114c159950 2023-10-16T12:52:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f113c001b80 con 0x7f114c159950 2023-10-16T12:52:17.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 >> v1:172.21.15.60:6789/0 conn(0x7f114c159950 legacy=0x7f114c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 0x7f1134005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1134006100 0x7f11340069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1134006f10 0x7f1134007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15e490 con 0x7f1134006f10 2023-10-16T12:52:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15d910 con 0x7f1134006100 2023-10-16T12:52:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113400dbe0 con 0x7f114c06c100 2023-10-16T12:52:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.100+0000 7f11509fe640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1134006f10 0x7f1134007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f114bfff640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 0x7f1134005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f114b7fe640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1134006100 0x7f11340069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f11509fe640 1 -- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 msgr2=0x7f1134005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f11509fe640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f114c06c100 0x7f1134005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f11509fe640 1 -- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1134006100 msgr2=0x7f11340069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:17.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f11509fe640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1134006100 0x7f11340069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.101+0000 7f11509fe640 1 -- 172.21.15.60:0/3978101881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134005730 con 0x7f1134006f10 2023-10-16T12:52:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.102+0000 7f11509fe640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1134006f10 0x7f1134007390 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f1144009af0 tx=0x7f1134007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29841e2d1ba26de4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.102+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114402c6b0 con 0x7f1134006f10 2023-10-16T12:52:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.102+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114402c810 con 0x7f1134006f10 2023-10-16T12:52:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.102+0000 7f114affd640 1 -- 172.21.15.60:0/3978101881 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114402cba0 con 0x7f1134006f10 2023-10-16T12:52:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11511ff640 1 -- 172.21.15.60:0/3978101881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1134006f10 msgr2=0x7f1134007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11511ff640 1 --2- 172.21.15.60:0/3978101881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1134006f10 0x7f1134007390 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f1144009af0 tx=0x7f1134007a00 comp rx=0 tx=0).stop 2023-10-16T12:52:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11509fe640 1 -- 172.21.15.60:0/3978101881 reap_dead start 2023-10-16T12:52:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11511ff640 1 -- 172.21.15.60:0/3978101881 shutdown_connections 2023-10-16T12:52:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11511ff640 1 -- 172.21.15.60:0/3978101881 >> 172.21.15.60:0/3978101881 conn(0x7f114c0ce6a0 msgr2=0x7f114c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11511ff640 1 -- 172.21.15.60:0/3978101881 shutdown_connections 2023-10-16T12:52:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.103+0000 7f11511ff640 1 -- 172.21.15.60:0/3978101881 wait complete. 2023-10-16T12:52:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.105+0000 7f11511ff640 1 Processor -- start 2023-10-16T12:52:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.105+0000 7f11511ff640 1 -- start start 2023-10-16T12:52:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.106+0000 7f11511ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1134006f10 0x7f114c1549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.106+0000 7f11511ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 0x7f114c0ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.106+0000 7f11511ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f114c0ae760 0x7f114c06ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.106+0000 7f11511ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15cec0 con 0x7f114c0b0cb0 2023-10-16T12:52:17.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.106+0000 7f11511ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15c2b0 con 0x7f114c0ae760 2023-10-16T12:52:17.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.106+0000 7f11511ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c15c590 con 0x7f1134006f10 2023-10-16T12:52:17.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 0x7f114c0ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 0x7f114c0ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35020/0 (socket says 172.21.15.60:35020) 2023-10-16T12:52:17.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 -- 172.21.15.60:0/3211028021 learned_addr learned my addr 172.21.15.60:0/3211028021 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:17.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114bfff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1134006f10 0x7f114c1549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f11509fe640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f114c0ae760 0x7f114c06ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 -- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1134006f10 msgr2=0x7f114c1549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1134006f10 0x7f114c1549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 -- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f114c0ae760 msgr2=0x7f114c06ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f114c0ae760 0x7f114c06ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f114b7fe640 1 -- 172.21.15.60:0/3211028021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c15c870 con 0x7f114c0b0cb0 2023-10-16T12:52:17.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.107+0000 7f11509fe640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f114c0ae760 0x7f114c06ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.108+0000 7f114b7fe640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 0x7f114c0ae220 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f1140011600 tx=0x7f1140011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.108+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114000f920 con 0x7f114c0b0cb0 2023-10-16T12:52:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.108+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114000fa80 con 0x7f114c0b0cb0 2023-10-16T12:52:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.108+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114000fdf0 con 0x7f114c0b0cb0 2023-10-16T12:52:17.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.108+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c15cb50 con 0x7f114c0b0cb0 2023-10-16T12:52:17.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.108+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c06b320 con 0x7f114c0b0cb0 2023-10-16T12:52:17.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.109+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c04e9b0 con 0x7f114c0b0cb0 2023-10-16T12:52:17.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.110+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 54) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1140015460 con 0x7f114c0b0cb0 2023-10-16T12:52:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.112+0000 7f1148ff9640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f113806af00 0x7f113806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.112+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f1140089f10 con 0x7f114c0b0cb0 2023-10-16T12:52:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.112+0000 7f114bfff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f113806af00 0x7f113806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.114+0000 7f114bfff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f113806af00 0x7f113806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f113c005e70 tx=0x7f113c009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:17.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.117+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1140054bd0 con 0x7f114c0b0cb0 2023-10-16T12:52:17.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.359+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f114c0c88c0 con 0x7f114c0b0cb0 2023-10-16T12:52:17.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.361+0000 7f1148ff9640 1 -- 172.21.15.60:0/3211028021 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f1140022090 con 0x7f114c0b0cb0 2023-10-16T12:52:17.364 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:17.364 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:50:51.414872+0000","modified":"2023-10-16T12:50:54.480441+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7353":{"gid":7353,"name":"a","rank":0,"incarnation":49,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/3569389918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3569389918},{"type":"v1","addr":"172.21.15.60:6835","nonce":3569389918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:52:17.365 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 53 2023-10-16T12:52:17.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f113806af00 msgr2=0x7f113806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f113806af00 0x7f113806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f113c005e70 tx=0x7f113c009040 comp rx=0 tx=0).stop 2023-10-16T12:52:17.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 msgr2=0x7f114c0ae220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 0x7f114c0ae220 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f1140011600 tx=0x7f1140011ac0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 shutdown_connections 2023-10-16T12:52:17.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f113806af00 0x7f113806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f114c0ae760 0x7f114c06ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f114c0b0cb0 0x7f114c0ae220 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 --2- 172.21.15.60:0/3211028021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1134006f10 0x7f114c1549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.363+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 >> 172.21.15.60:0/3211028021 conn(0x7f114c0ce6a0 msgr2=0x7f114c06c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.364+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 shutdown_connections 2023-10-16T12:52:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.364+0000 7f11511ff640 1 -- 172.21.15.60:0/3211028021 wait complete. 2023-10-16T12:52:17.386 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.603+0000 7f37cfae7640 1 Processor -- start 2023-10-16T12:52:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.604+0000 7f37cfae7640 1 -- start start 2023-10-16T12:52:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.605+0000 7f37cfae7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37c815e260 0x7f37c80ca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.605+0000 7f37cfae7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37c80cb0f0 0x7f37c806d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.605+0000 7f37cfae7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c815d810 con 0x7f37c80ceec0 2023-10-16T12:52:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cfae7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c815d660 con 0x7f37c815e260 2023-10-16T12:52:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cfae7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80cab50 con 0x7f37c80cb0f0 2023-10-16T12:52:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd883640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f37c80ceec0 0x7f37c80cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50324/0 (socket says 172.21.15.60:50324) 2023-10-16T12:52:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd883640 1 -- 172.21.15.60:0/100674778 learned_addr learned my addr 172.21.15.60:0/100674778 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:17.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd082640 1 --2- 172.21.15.60:0/100674778 >> v2:172.21.15.60:3300/0 conn(0x7f37c815e260 0x7f37c80ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37ce084640 1 --2- 172.21.15.60:0/100674778 >> v2:172.21.15.144:3300/0 conn(0x7f37c80cb0f0 0x7f37c806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd082640 1 -- 172.21.15.60:0/100674778 >> v2:172.21.15.144:3300/0 conn(0x7f37c80cb0f0 msgr2=0x7f37c806d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd082640 1 --2- 172.21.15.60:0/100674778 >> v2:172.21.15.144:3300/0 conn(0x7f37c80cb0f0 0x7f37c806d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd082640 1 -- 172.21.15.60:0/100674778 >> v1:172.21.15.60:6789/0 conn(0x7f37c80ceec0 legacy=0x7f37c80cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.606+0000 7f37cd082640 1 -- 172.21.15.60:0/100674778 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c815e6c0 con 0x7f37c815e260 2023-10-16T12:52:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.608+0000 7f37cd082640 1 --2- 172.21.15.60:0/100674778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80ca610 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f37c8070230 tx=0x7f37b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b992f7e9db22a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.608+0000 7f37cc881640 1 -- 172.21.15.60:0/100674778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b802e070 con 0x7f37c815e260 2023-10-16T12:52:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.608+0000 7f37cc881640 1 -- 172.21.15.60:0/100674778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b80315a0 con 0x7f37c815e260 2023-10-16T12:52:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.608+0000 7f37cc881640 1 -- 172.21.15.60:0/100674778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8031930 con 0x7f37c815e260 2023-10-16T12:52:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 -- 172.21.15.60:0/100674778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 msgr2=0x7f37c80ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 --2- 172.21.15.60:0/100674778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80ca610 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f37c8070230 tx=0x7f37b802b840 comp rx=0 tx=0).stop 2023-10-16T12:52:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 -- 172.21.15.60:0/100674778 shutdown_connections 2023-10-16T12:52:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 --2- 172.21.15.60:0/100674778 >> v2:172.21.15.144:3300/0 conn(0x7f37c80cb0f0 0x7f37c806d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 --2- 172.21.15.60:0/100674778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80ca610 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 -- 172.21.15.60:0/100674778 >> 172.21.15.60:0/100674778 conn(0x7f37c80cfaa0 msgr2=0x7f37c80cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 -- 172.21.15.60:0/100674778 shutdown_connections 2023-10-16T12:52:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.609+0000 7f37cfae7640 1 -- 172.21.15.60:0/100674778 wait complete. 2023-10-16T12:52:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.611+0000 7f37cfae7640 1 Processor -- start 2023-10-16T12:52:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.611+0000 7f37cfae7640 1 -- start start 2023-10-16T12:52:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cfae7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37c80cb0f0 0x7f37c80c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cfae7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37c80ceec0 0x7f37c80b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cfae7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cfae7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c81f0ef0 con 0x7f37c815e260 2023-10-16T12:52:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cfae7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80b4700 con 0x7f37c80cb0f0 2023-10-16T12:52:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cfae7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80b49e0 con 0x7f37c80ceec0 2023-10-16T12:52:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37ce084640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37ce084640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35040/0 (socket says 172.21.15.60:35040) 2023-10-16T12:52:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37ce084640 1 -- 172.21.15.60:0/1146695099 learned_addr learned my addr 172.21.15.60:0/1146695099 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cd883640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37c80cb0f0 0x7f37c80c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37cd082640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37c80ceec0 0x7f37c80b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.612+0000 7f37ce084640 1 -- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37c80ceec0 msgr2=0x7f37c80b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37ce084640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37c80ceec0 0x7f37c80b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37ce084640 1 -- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37c80cb0f0 msgr2=0x7f37c80c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37ce084640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37c80cb0f0 0x7f37c80c5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37ce084640 1 -- 172.21.15.60:0/1146695099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b8009400 con 0x7f37c815e260 2023-10-16T12:52:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37cd883640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37c80cb0f0 0x7f37c80c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37cd082640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37c80ceec0 0x7f37c80b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37ce084640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80b3e70 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f37c0007e40 tx=0x7f37c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0014620 con 0x7f37c815e260 2023-10-16T12:52:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c000ee90 con 0x7f37c815e260 2023-10-16T12:52:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c00149e0 con 0x7f37c815e260 2023-10-16T12:52:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c80b4cc0 con 0x7f37c815e260 2023-10-16T12:52:17.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.613+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c80b43b0 con 0x7f37c815e260 2023-10-16T12:52:17.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.614+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c81f1360 con 0x7f37c815e260 2023-10-16T12:52:17.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.616+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 54) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f37c0014b40 con 0x7f37c815e260 2023-10-16T12:52:17.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.616+0000 7f37be7fc640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f37ac06af50 0x7f37ac06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:17.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.616+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f37c008a390 con 0x7f37c815e260 2023-10-16T12:52:17.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.617+0000 7f37cd883640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f37ac06af50 0x7f37ac06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:17.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.618+0000 7f37cd883640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f37ac06af50 0x7f37ac06d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37c4000f60 tx=0x7f37c4009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:17.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.621+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f37c81f1360 con 0x7f37c815e260 2023-10-16T12:52:17.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.817+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37c81f1360 con 0x7f37c815e260 2023-10-16T12:52:17.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.821+0000 7f37be7fc640 1 -- 172.21.15.60:0/1146695099 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+17749 (secure 0 0 0) 0x7f37c81f1360 con 0x7f37c815e260 2023-10-16T12:52:17.824 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:17.824 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":117,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:14.184254+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:50:49.355185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-16T12:50:50.439078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.22","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2025887462":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/948877502":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/1510335921":"2023-10-17T12:52:14.184032+0000","172.21.15.144:6833/2022371982":"2023-10-17T12:52:14.184032+0000","172.21.15.144:6832/2022371982":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/337630602":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/671149719":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/3635246428":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/2189557198":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1153724330":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1224606105":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1935697927":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/840292481":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6833/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/737927288":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/1070434778":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6832/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/4048637185":"2023-10-17T12:51:06.102326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.823+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f37ac06af50 msgr2=0x7f37ac06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.823+0000 7f37cfae7640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f37ac06af50 0x7f37ac06d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f37c4000f60 tx=0x7f37c4009040 comp rx=0 tx=0).stop 2023-10-16T12:52:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.823+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 msgr2=0x7f37c80b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.823+0000 7f37cfae7640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80b3e70 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f37c0007e40 tx=0x7f37c000e4b0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 shutdown_connections 2023-10-16T12:52:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f37ac06af50 0x7f37ac06d410 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37c80ceec0 0x7f37c80b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37c815e260 0x7f37c80b3e70 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 --2- 172.21.15.60:0/1146695099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37c80cb0f0 0x7f37c80c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:17.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 >> 172.21.15.60:0/1146695099 conn(0x7f37c80cfaa0 msgr2=0x7f37c80c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:17.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 shutdown_connections 2023-10-16T12:52:17.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:17.824+0000 7f37cfae7640 1 -- 172.21.15.60:0/1146695099 wait complete. 2023-10-16T12:52:17.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:52:18.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.069+0000 7f232f59e640 1 Processor -- start 2023-10-16T12:52:18.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.069+0000 7f232f59e640 1 -- start start 2023-10-16T12:52:18.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.070+0000 7f232f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300b8480 0x7f23300cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f23300d0be0 0x7f23300cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233015d150 con 0x7f233015cd70 2023-10-16T12:52:18.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232f59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23300d1180 con 0x7f23300d0be0 2023-10-16T12:52:18.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23300d1460 con 0x7f23300b8480 2023-10-16T12:52:18.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232e59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f233015cd70 0x7f23300d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50332/0 (socket says 172.21.15.60:50332) 2023-10-16T12:52:18.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232e59c640 1 -- 172.21.15.60:0/1876532219 learned_addr learned my addr 172.21.15.60:0/1876532219 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:18.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232ed9d640 1 --2- 172.21.15.60:0/1876532219 >> v2:172.21.15.60:3300/0 conn(0x7f23300d0be0 0x7f23300cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232dd9b640 1 --2- 172.21.15.60:0/1876532219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300b8480 0x7f23300cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232ed9d640 1 -- 172.21.15.60:0/1876532219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300b8480 msgr2=0x7f23300cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:18.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232ed9d640 1 --2- 172.21.15.60:0/1876532219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300b8480 0x7f23300cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232ed9d640 1 -- 172.21.15.60:0/1876532219 >> v1:172.21.15.60:6789/0 conn(0x7f233015cd70 legacy=0x7f23300d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.071+0000 7f232ed9d640 1 -- 172.21.15.60:0/1876532219 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23300d1740 con 0x7f23300d0be0 2023-10-16T12:52:18.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.073+0000 7f232ed9d640 1 --2- 172.21.15.60:0/1876532219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23300d0be0 0x7f23300cdd00 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f23280097a0 tx=0x7f232802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619eeaac0a078619 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:18.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.073+0000 7f232d59a640 1 -- 172.21.15.60:0/1876532219 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232802d040 con 0x7f23300d0be0 2023-10-16T12:52:18.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.073+0000 7f232d59a640 1 -- 172.21.15.60:0/1876532219 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2328031810 con 0x7f23300d0be0 2023-10-16T12:52:18.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.073+0000 7f232d59a640 1 -- 172.21.15.60:0/1876532219 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328031ba0 con 0x7f23300d0be0 2023-10-16T12:52:18.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.073+0000 7f232f59e640 1 -- 172.21.15.60:0/1876532219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23300d0be0 msgr2=0x7f23300cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.073+0000 7f232f59e640 1 --2- 172.21.15.60:0/1876532219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23300d0be0 0x7f23300cdd00 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f23280097a0 tx=0x7f232802b8b0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.074+0000 7f232f59e640 1 -- 172.21.15.60:0/1876532219 shutdown_connections 2023-10-16T12:52:18.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.074+0000 7f232f59e640 1 --2- 172.21.15.60:0/1876532219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23300d0be0 0x7f23300cdd00 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.074+0000 7f232f59e640 1 --2- 172.21.15.60:0/1876532219 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300b8480 0x7f23300cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.074+0000 7f232f59e640 1 -- 172.21.15.60:0/1876532219 >> 172.21.15.60:0/1876532219 conn(0x7f23300cf570 msgr2=0x7f23300cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:18.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.074+0000 7f232f59e640 1 -- 172.21.15.60:0/1876532219 shutdown_connections 2023-10-16T12:52:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.074+0000 7f232f59e640 1 -- 172.21.15.60:0/1876532219 wait complete. 2023-10-16T12:52:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.075+0000 7f232f59e640 1 Processor -- start 2023-10-16T12:52:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.076+0000 7f232f59e640 1 -- start start 2023-10-16T12:52:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23300b8480 0x7f23301f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300d0be0 0x7f233015e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 0x7f233015e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23300c9070 con 0x7f233015cd70 2023-10-16T12:52:18.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23301f22f0 con 0x7f23300b8480 2023-10-16T12:52:18.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23300c8740 con 0x7f23300d0be0 2023-10-16T12:52:18.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 0x7f233015e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 0x7f233015e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35056/0 (socket says 172.21.15.60:35056) 2023-10-16T12:52:18.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232ed9d640 1 -- 172.21.15.60:0/1460374112 learned_addr learned my addr 172.21.15.60:0/1460374112 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:18.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232dd9b640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300d0be0 0x7f233015e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.077+0000 7f232e59c640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23300b8480 0x7f23301f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232ed9d640 1 -- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300d0be0 msgr2=0x7f233015e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232ed9d640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300d0be0 0x7f233015e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232ed9d640 1 -- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23300b8480 msgr2=0x7f23301f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232ed9d640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23300b8480 0x7f23301f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232ed9d640 1 -- 172.21.15.60:0/1460374112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2328009400 con 0x7f233015cd70 2023-10-16T12:52:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232ed9d640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 0x7f233015e630 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f232802b8f0 tx=0x7f23280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328006e10 con 0x7f233015cd70 2023-10-16T12:52:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232803ace0 con 0x7f233015cd70 2023-10-16T12:52:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328044610 con 0x7f233015cd70 2023-10-16T12:52:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.078+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23300c8a20 con 0x7f233015cd70 2023-10-16T12:52:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.079+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23300c8430 con 0x7f233015cd70 2023-10-16T12:52:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.080+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23300c91e0 con 0x7f233015cd70 2023-10-16T12:52:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.081+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 54) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2328044770 con 0x7f233015cd70 2023-10-16T12:52:18.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.081+0000 7f2326ffd640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f231006af00 0x7f231006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.082+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f23280abc80 con 0x7f233015cd70 2023-10-16T12:52:18.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.082+0000 7f232e59c640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f231006af00 0x7f231006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.083+0000 7f232e59c640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f231006af00 0x7f231006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2320005c00 tx=0x7f2320009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:18.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.086+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23280b0070 con 0x7f233015cd70 2023-10-16T12:52:18.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.244+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2328076d00 con 0x7f233015cd70 2023-10-16T12:52:18.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.327+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f233004e9b0 con 0x7f233015cd70 2023-10-16T12:52:18.331 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:52:18.328+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:52:18.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.333+0000 7f2326ffd640 1 -- 172.21.15.60:0/1460374112 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) v1 ==== 161+0+0 (secure 0 0 0) 0x7f232807cfd0 con 0x7f233015cd70 2023-10-16T12:52:18.335 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:52:18.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f231006af00 msgr2=0x7f231006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f231006af00 0x7f231006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2320005c00 tx=0x7f2320009040 comp rx=0 tx=0).stop 2023-10-16T12:52:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 msgr2=0x7f233015e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 0x7f233015e630 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f232802b8f0 tx=0x7f23280046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 shutdown_connections 2023-10-16T12:52:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f231006af00 0x7f231006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f23300d0be0 0x7f233015e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f233015cd70 0x7f233015e630 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 --2- 172.21.15.60:0/1460374112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23300b8480 0x7f23301f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.336+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 >> 172.21.15.60:0/1460374112 conn(0x7f23300cf570 msgr2=0x7f23300cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:18.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.337+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 shutdown_connections 2023-10-16T12:52:18.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.337+0000 7f232f59e640 1 -- 172.21.15.60:0/1460374112 wait complete. 2023-10-16T12:52:18.361 INFO:tasks.ceph.mds.a.smithi060.stderr: -7029> 2023-10-16T12:50:54.857+0000 7f950a96f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:52:18.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:52:18.428 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:52:18.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.566+0000 7f44982ec640 1 Processor -- start 2023-10-16T12:52:18.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.567+0000 7f44982ec640 1 -- start start 2023-10-16T12:52:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.568+0000 7f44982ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4490159e90 0x7f44900ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.568+0000 7f44982ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f449006c140 0x7f449006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.568+0000 7f44982ec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4490070360 con 0x7f4490159ab0 2023-10-16T12:52:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.568+0000 7f44982ec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449006cb10 con 0x7f4490159e90 2023-10-16T12:52:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.568+0000 7f44982ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449015d890 con 0x7f449006c140 2023-10-16T12:52:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.568+0000 7f4495887640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4490159e90 0x7f44900ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4495887640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4490159e90 0x7f44900ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35084/0 (socket says 172.21.15.60:35084) 2023-10-16T12:52:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4495887640 1 -- 172.21.15.60:0/4138873689 learned_addr learned my addr 172.21.15.60:0/4138873689 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4496889640 1 --2- 172.21.15.60:0/4138873689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f449006c140 0x7f449006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4495887640 1 -- 172.21.15.60:0/4138873689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f449006c140 msgr2=0x7f449006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:18.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4495887640 1 --2- 172.21.15.60:0/4138873689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f449006c140 0x7f449006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4495887640 1 -- 172.21.15.60:0/4138873689 >> v1:172.21.15.144:6789/0 conn(0x7f4490159ab0 legacy=0x7f449006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.569+0000 7f4495887640 1 -- 172.21.15.60:0/4138873689 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900cd1f0 con 0x7f4490159e90 2023-10-16T12:52:18.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.570+0000 7f4495887640 1 --2- 172.21.15.60:0/4138873689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159e90 0x7f44900ce370 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f4484009870 tx=0x7f448402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ee0a0a67f2ac04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:18.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f4495086640 1 -- 172.21.15.60:0/4138873689 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840317c0 con 0x7f4490159e90 2023-10-16T12:52:18.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f4495086640 1 -- 172.21.15.60:0/4138873689 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4484031920 con 0x7f4490159e90 2023-10-16T12:52:18.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f4495086640 1 -- 172.21.15.60:0/4138873689 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484031cb0 con 0x7f4490159e90 2023-10-16T12:52:18.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 -- 172.21.15.60:0/4138873689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159e90 msgr2=0x7f44900ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 --2- 172.21.15.60:0/4138873689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159e90 0x7f44900ce370 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f4484009870 tx=0x7f448402b860 comp rx=0 tx=0).stop 2023-10-16T12:52:18.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 -- 172.21.15.60:0/4138873689 shutdown_connections 2023-10-16T12:52:18.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 --2- 172.21.15.60:0/4138873689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f449006c140 0x7f449006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 --2- 172.21.15.60:0/4138873689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159e90 0x7f44900ce370 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 -- 172.21.15.60:0/4138873689 >> 172.21.15.60:0/4138873689 conn(0x7f44900ce9b0 msgr2=0x7f44900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:18.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 -- 172.21.15.60:0/4138873689 shutdown_connections 2023-10-16T12:52:18.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.571+0000 7f44982ec640 1 -- 172.21.15.60:0/4138873689 wait complete. 2023-10-16T12:52:18.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.573+0000 7f44982ec640 1 Processor -- start 2023-10-16T12:52:18.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.573+0000 7f44982ec640 1 -- start start 2023-10-16T12:52:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.574+0000 7f44982ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f449006c140 0x7f44900c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.574+0000 7f44982ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 0x7f44901475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.574+0000 7f44982ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4490159e90 0x7f44900c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.574+0000 7f44982ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449015b4b0 con 0x7f4490159ab0 2023-10-16T12:52:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.574+0000 7f44982ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44900c22a0 con 0x7f449006c140 2023-10-16T12:52:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.574+0000 7f44982ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449015b700 con 0x7f4490159e90 2023-10-16T12:52:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 0x7f44901475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 0x7f44901475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35098/0 (socket says 172.21.15.60:35098) 2023-10-16T12:52:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 -- 172.21.15.60:0/1696898863 learned_addr learned my addr 172.21.15.60:0/1696898863 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4496088640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f449006c140 0x7f44900c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4496889640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4490159e90 0x7f44900c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 -- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4490159e90 msgr2=0x7f44900c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4490159e90 0x7f44900c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 -- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f449006c140 msgr2=0x7f44900c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:18.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f449006c140 0x7f44900c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:18.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 -- 172.21.15.60:0/1696898863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44840094d0 con 0x7f4490159ab0 2023-10-16T12:52:18.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4496088640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f449006c140 0x7f44900c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:18.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.575+0000 7f4495887640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 0x7f44901475f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f448402b8a0 tx=0x7f44840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:18.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.576+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484032860 con 0x7f4490159ab0 2023-10-16T12:52:18.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.576+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4484032a00 con 0x7f4490159ab0 2023-10-16T12:52:18.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.576+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484032db0 con 0x7f4490159ab0 2023-10-16T12:52:18.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.576+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449015b9e0 con 0x7f4490159ab0 2023-10-16T12:52:18.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.576+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449015bfa0 con 0x7f4490159ab0 2023-10-16T12:52:18.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.577+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449015b840 con 0x7f4490159ab0 2023-10-16T12:52:18.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.578+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f448404e450 con 0x7f4490159ab0 2023-10-16T12:52:18.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.579+0000 7f448effd640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f447c06af90 0x7f447c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:18.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.579+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f44840abdd0 con 0x7f4490159ab0 2023-10-16T12:52:18.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.580+0000 7f4496088640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f447c06af90 0x7f447c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:18.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.581+0000 7f4496088640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f447c06af90 0x7f447c06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4480000f60 tx=0x7f4480009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:18.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.583+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f449015b840 con 0x7f4490159ab0 2023-10-16T12:52:18.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:18.817+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f449015b840 con 0x7f4490159ab0 2023-10-16T12:52:19.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.336+0000 7f448effd640 1 -- 172.21.15.60:0/1696898863 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f449015b840 con 0x7f4490159ab0 2023-10-16T12:52:19.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.339+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f447c06af90 msgr2=0x7f447c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:19.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.339+0000 7f44982ec640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f447c06af90 0x7f447c06d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4480000f60 tx=0x7f4480009040 comp rx=0 tx=0).stop 2023-10-16T12:52:19.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.339+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 msgr2=0x7f44901475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:19.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.339+0000 7f44982ec640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 0x7f44901475f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f448402b8a0 tx=0x7f44840046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 shutdown_connections 2023-10-16T12:52:19.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f447c06af90 0x7f447c06d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4490159e90 0x7f44900c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4490159ab0 0x7f44901475f0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 --2- 172.21.15.60:0/1696898863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f449006c140 0x7f44900c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 >> 172.21.15.60:0/1696898863 conn(0x7f44900ce9b0 msgr2=0x7f44900b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:19.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 shutdown_connections 2023-10-16T12:52:19.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.340+0000 7f44982ec640 1 -- 172.21.15.60:0/1696898863 wait complete. 2023-10-16T12:52:19.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:52:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.581+0000 7f68f0ce6640 1 Processor -- start 2023-10-16T12:52:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.581+0000 7f68f0ce6640 1 -- start start 2023-10-16T12:52:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.582+0000 7f68f0ce6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68f0ce6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68ec0ad4d0 0x7f68ec0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68f0ce6640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec15ac60 con 0x7f68ec15cd50 2023-10-16T12:52:19.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68f0ce6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ec070030 con 0x7f68ec0ad4d0 2023-10-16T12:52:19.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68f0ce6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec070310 con 0x7f68ec153740 2023-10-16T12:52:19.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68e3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68ec0ad4d0 0x7f68ec0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:19.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68e3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68ec0ad4d0 0x7f68ec0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49752/0 (socket says 172.21.15.60:49752) 2023-10-16T12:52:19.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68e3fff640 1 -- 172.21.15.60:0/2228404993 learned_addr learned my addr 172.21.15.60:0/2228404993 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:19.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.583+0000 7f68eb7fe640 1 --2- 172.21.15.60:0/2228404993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.584+0000 7f68e3fff640 1 -- 172.21.15.60:0/2228404993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 msgr2=0x7f68ec15a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.584+0000 7f68e3fff640 1 --2- 172.21.15.60:0/2228404993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec15a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.584+0000 7f68e3fff640 1 -- 172.21.15.60:0/2228404993 >> v1:172.21.15.144:6789/0 conn(0x7f68ec15cd50 legacy=0x7f68ec15d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.584+0000 7f68e3fff640 1 -- 172.21.15.60:0/2228404993 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0705f0 con 0x7f68ec0ad4d0 2023-10-16T12:52:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.585+0000 7f68e3fff640 1 --2- 172.21.15.60:0/2228404993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec0ad930 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009870 tx=0x7f68d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb4c111150f3777 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.585+0000 7f68eaffd640 1 -- 172.21.15.60:0/2228404993 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4031640 con 0x7f68ec0ad4d0 2023-10-16T12:52:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.585+0000 7f68eaffd640 1 -- 172.21.15.60:0/2228404993 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40317a0 con 0x7f68ec0ad4d0 2023-10-16T12:52:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.585+0000 7f68eaffd640 1 -- 172.21.15.60:0/2228404993 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4031b30 con 0x7f68ec0ad4d0 2023-10-16T12:52:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 -- 172.21.15.60:0/2228404993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec0ad4d0 msgr2=0x7f68ec0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/2228404993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec0ad930 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009870 tx=0x7f68d402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 -- 172.21.15.60:0/2228404993 shutdown_connections 2023-10-16T12:52:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/2228404993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec0ad930 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/2228404993 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 -- 172.21.15.60:0/2228404993 >> 172.21.15.60:0/2228404993 conn(0x7f68ec0ce6a0 msgr2=0x7f68ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 -- 172.21.15.60:0/2228404993 shutdown_connections 2023-10-16T12:52:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.586+0000 7f68f0ce6640 1 -- 172.21.15.60:0/2228404993 wait complete. 2023-10-16T12:52:19.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.588+0000 7f68f0ce6640 1 Processor -- start 2023-10-16T12:52:19.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.588+0000 7f68f0ce6640 1 -- start start 2023-10-16T12:52:19.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.589+0000 7f68f0ce6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:19.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.589+0000 7f68f0ce6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec0b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:19.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.589+0000 7f68f0ce6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 0x7f68ec152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:19.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.589+0000 7f68f0ce6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec153b20 con 0x7f68ec15cd50 2023-10-16T12:52:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.589+0000 7f68f0ce6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec153e00 con 0x7f68ec0ad4d0 2023-10-16T12:52:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.589+0000 7f68f0ce6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec1540e0 con 0x7f68ec153740 2023-10-16T12:52:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 0x7f68ec152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 0x7f68ec152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49760/0 (socket says 172.21.15.60:49760) 2023-10-16T12:52:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 -- 172.21.15.60:0/393319861 learned_addr learned my addr 172.21.15.60:0/393319861 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68eb7fe640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68e3fff640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec0b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 -- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 msgr2=0x7f68ec0b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec0b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 -- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68ec0ad4d0 msgr2=0x7f68ec155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.590+0000 7f68ebfff640 1 -- 172.21.15.60:0/393319861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40094d0 con 0x7f68ec15cd50 2023-10-16T12:52:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.591+0000 7f68ebfff640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 0x7f68ec152670 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f68e400b970 tx=0x7f68e400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.591+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e400cb40 con 0x7f68ec15cd50 2023-10-16T12:52:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.591+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e400cca0 con 0x7f68ec15cd50 2023-10-16T12:52:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.591+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec1543c0 con 0x7f68ec15cd50 2023-10-16T12:52:19.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.591+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4014680 con 0x7f68ec15cd50 2023-10-16T12:52:19.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.591+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec154bc0 con 0x7f68ec15cd50 2023-10-16T12:52:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.592+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec1548e0 con 0x7f68ec15cd50 2023-10-16T12:52:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.593+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f68e401d3b0 con 0x7f68ec15cd50 2023-10-16T12:52:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.595+0000 7f68e8ff9640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f68cc06af90 0x7f68cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.595+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f68e40899f0 con 0x7f68ec15cd50 2023-10-16T12:52:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.596+0000 7f68eb7fe640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f68cc06af90 0x7f68cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.597+0000 7f68eb7fe640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f68cc06af90 0x7f68cc06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f68dc006fd0 tx=0x7f68dc008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:19.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.599+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f68e4054570 con 0x7f68ec15cd50 2023-10-16T12:52:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:19.793+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f68ec06fc30 con 0x7f68ec15cd50 2023-10-16T12:52:20.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.339+0000 7f68e8ff9640 1 -- 172.21.15.60:0/393319861 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) v1 ==== 181+0+0 (secure 0 0 0) 0x7f68e4057b90 con 0x7f68ec15cd50 2023-10-16T12:52:20.343 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:52:20.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.343+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f68cc06af90 msgr2=0x7f68cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:20.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.343+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f68cc06af90 0x7f68cc06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f68dc006fd0 tx=0x7f68dc008040 comp rx=0 tx=0).stop 2023-10-16T12:52:20.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.343+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 msgr2=0x7f68ec152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:20.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.343+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 0x7f68ec152670 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f68e400b970 tx=0x7f68e400be30 comp rx=0 tx=0).stop 2023-10-16T12:52:20.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.343+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 shutdown_connections 2023-10-16T12:52:20.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.343+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f68cc06af90 0x7f68cc06d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.344+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68ec153740 0x7f68ec0b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.344+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68ec15cd50 0x7f68ec152670 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.344+0000 7f68f0ce6640 1 --2- 172.21.15.60:0/393319861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68ec0ad4d0 0x7f68ec155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.344+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 >> 172.21.15.60:0/393319861 conn(0x7f68ec0ce6a0 msgr2=0x7f68ec158770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:20.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.344+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 shutdown_connections 2023-10-16T12:52:20.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.345+0000 7f68f0ce6640 1 -- 172.21.15.60:0/393319861 wait complete. 2023-10-16T12:52:20.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:52:20.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.601+0000 7f59247fc640 1 Processor -- start 2023-10-16T12:52:20.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.601+0000 7f59247fc640 1 -- start start 2023-10-16T12:52:20.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.602+0000 7f59247fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f591c0cac20 0x7f591c0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:20.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.602+0000 7f59247fc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c15b310 con 0x7f591c15ac10 2023-10-16T12:52:20.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.602+0000 7f59247fc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c06f640 con 0x7f591c0b0360 2023-10-16T12:52:20.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.602+0000 7f59247fc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c15f800 con 0x7f591c0cac20 2023-10-16T12:52:20.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922d99640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f591c0cac20 0x7f591c0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:20.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922598640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f591c15ac10 0x7f591c06c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50708/0 (socket says 172.21.15.60:50708) 2023-10-16T12:52:20.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922d99640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f591c0cac20 0x7f591c0b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49770/0 (socket says 172.21.15.60:49770) 2023-10-16T12:52:20.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922598640 1 -- 172.21.15.60:0/2031296836 learned_addr learned my addr 172.21.15.60:0/2031296836 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:20.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922d99640 1 -- 172.21.15.60:0/2031296836 >> v1:172.21.15.144:6789/0 conn(0x7f591c0b0360 legacy=0x7f591c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:20.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922d99640 1 -- 172.21.15.60:0/2031296836 >> v1:172.21.15.60:6789/0 conn(0x7f591c15ac10 legacy=0x7f591c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:20.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5921596640 1 -- 172.21.15.60:0/2031296836 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341265459 0 0) 0x7f591c15b310 con 0x7f591c15ac10 2023-10-16T12:52:20.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.603+0000 7f5922d99640 1 -- 172.21.15.60:0/2031296836 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c15b5b0 con 0x7f591c0cac20 2023-10-16T12:52:20.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.605+0000 7f5922d99640 1 --2- 172.21.15.60:0/2031296836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0cac20 0x7f591c0b07f0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5914009d90 tx=0x7f591402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4824f3d5c3760cd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:20.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.605+0000 7f5921596640 1 -- 172.21.15.60:0/2031296836 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591402cbd0 con 0x7f591c0cac20 2023-10-16T12:52:20.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.605+0000 7f5921596640 1 -- 172.21.15.60:0/2031296836 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591402cd30 con 0x7f591c0cac20 2023-10-16T12:52:20.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.605+0000 7f5921596640 1 -- 172.21.15.60:0/2031296836 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591402cea0 con 0x7f591c0cac20 2023-10-16T12:52:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 -- 172.21.15.60:0/2031296836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0cac20 msgr2=0x7f591c0b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 --2- 172.21.15.60:0/2031296836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0cac20 0x7f591c0b07f0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5914009d90 tx=0x7f591402be20 comp rx=0 tx=0).stop 2023-10-16T12:52:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 -- 172.21.15.60:0/2031296836 shutdown_connections 2023-10-16T12:52:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 --2- 172.21.15.60:0/2031296836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0cac20 0x7f591c0b07f0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 -- 172.21.15.60:0/2031296836 >> 172.21.15.60:0/2031296836 conn(0x7f591c0cb360 msgr2=0x7f591c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 -- 172.21.15.60:0/2031296836 shutdown_connections 2023-10-16T12:52:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.606+0000 7f59247fc640 1 -- 172.21.15.60:0/2031296836 wait complete. 2023-10-16T12:52:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.608+0000 7f59247fc640 1 Processor -- start 2023-10-16T12:52:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.608+0000 7f59247fc640 1 -- start start 2023-10-16T12:52:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f59247fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 0x7f591c155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f59247fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f591c0cac20 0x7f591c0c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f59247fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f591c15ac10 0x7f591c1f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f59247fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0c6620 con 0x7f591c0b0360 2023-10-16T12:52:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f59247fc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0c5fa0 con 0x7f591c0cac20 2023-10-16T12:52:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f59247fc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c1f63c0 con 0x7f591c15ac10 2023-10-16T12:52:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f5922598640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 0x7f591c155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f5922598640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 0x7f591c155d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49784/0 (socket says 172.21.15.60:49784) 2023-10-16T12:52:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.609+0000 7f5922598640 1 -- 172.21.15.60:0/3615242237 learned_addr learned my addr 172.21.15.60:0/3615242237 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5922598640 1 -- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f591c15ac10 msgr2=0x7f591c1f5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5922598640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f591c15ac10 0x7f591c1f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5922598640 1 -- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f591c0cac20 msgr2=0x7f591c0c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5922598640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f591c0cac20 0x7f591c0c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5922598640 1 -- 172.21.15.60:0/3615242237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914009a70 con 0x7f591c0b0360 2023-10-16T12:52:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5922598640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 0x7f591c155d60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f591800c430 tx=0x7f591800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59180074e0 con 0x7f591c0b0360 2023-10-16T12:52:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5918007660 con 0x7f591c0b0360 2023-10-16T12:52:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918004a20 con 0x7f591c0b0360 2023-10-16T12:52:20.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.610+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c1f66a0 con 0x7f591c0b0360 2023-10-16T12:52:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.611+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c1f25a0 con 0x7f591c0b0360 2023-10-16T12:52:20.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.612+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c0c6760 con 0x7f591c0b0360 2023-10-16T12:52:20.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.613+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5918004b80 con 0x7f591c0b0360 2023-10-16T12:52:20.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.614+0000 7f5912ffd640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f590006af90 0x7f590006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.614+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f59180897b0 con 0x7f591c0b0360 2023-10-16T12:52:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.615+0000 7f5921d97640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f590006af90 0x7f590006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:20.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.616+0000 7f5921d97640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f590006af90 0x7f590006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f591c0af590 tx=0x7f590c0079b0 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:20.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.619+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f591c0c6760 con 0x7f591c0b0360 2023-10-16T12:52:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:20.814+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f591c0c6760 con 0x7f591c0b0360 2023-10-16T12:52:21.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.354+0000 7f5912ffd640 1 -- 172.21.15.60:0/3615242237 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) v1 ==== 169+0+0 (secure 0 0 0) 0x7f59180545a0 con 0x7f591c0b0360 2023-10-16T12:52:21.358 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:52:21.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f590006af90 msgr2=0x7f590006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f590006af90 0x7f590006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f591c0af590 tx=0x7f590c0079b0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 msgr2=0x7f591c155d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 0x7f591c155d60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f591800c430 tx=0x7f591800c8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 shutdown_connections 2023-10-16T12:52:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f590006af90 0x7f590006d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f591c15ac10 0x7f591c1f5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f591c0cac20 0x7f591c0c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 --2- 172.21.15.60:0/3615242237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f591c0b0360 0x7f591c155d60 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.358+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 >> 172.21.15.60:0/3615242237 conn(0x7f591c0cb360 msgr2=0x7f591c0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.359+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 shutdown_connections 2023-10-16T12:52:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.359+0000 7f59247fc640 1 -- 172.21.15.60:0/3615242237 wait complete. 2023-10-16T12:52:21.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:52:21.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.597+0000 7f1b93916640 1 Processor -- start 2023-10-16T12:52:21.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.597+0000 7f1b93916640 1 -- start start 2023-10-16T12:52:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.598+0000 7f1b93916640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1b8c15aaa0 0x7f1b8c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.598+0000 7f1b93916640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1b8c15ae80 0x7f1b8c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.598+0000 7f1b93916640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c0ccaa0 0x7f1b8c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b93916640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c0707e0 con 0x7f1b8c15aaa0 2023-10-16T12:52:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b93916640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c06b7a0 con 0x7f1b8c15ae80 2023-10-16T12:52:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b93916640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c15ef20 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b916b2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1b8c15aaa0 0x7f1b8c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b916b2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1b8c15aaa0 0x7f1b8c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49786/0 (socket says 172.21.15.60:49786) 2023-10-16T12:52:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b916b2640 1 -- 172.21.15.60:0/3923637777 learned_addr learned my addr 172.21.15.60:0/3923637777 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b91eb3640 1 --2- 172.21.15.60:0/3923637777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c0ccaa0 0x7f1b8c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b90eb1640 1 --2- 172.21.15.60:0/3923637777 >> v2:172.21.15.144:3300/0 conn(0x7f1b8c15ae80 0x7f1b8c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b916b2640 1 -- 172.21.15.60:0/3923637777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c0ccaa0 msgr2=0x7f1b8c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:21.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b916b2640 1 --2- 172.21.15.60:0/3923637777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c0ccaa0 0x7f1b8c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.599+0000 7f1b916b2640 1 -- 172.21.15.60:0/3923637777 >> v2:172.21.15.144:3300/0 conn(0x7f1b8c15ae80 msgr2=0x7f1b8c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:21.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.600+0000 7f1b916b2640 1 --2- 172.21.15.60:0/3923637777 >> v2:172.21.15.144:3300/0 conn(0x7f1b8c15ae80 0x7f1b8c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.600+0000 7f1b916b2640 1 -- 172.21.15.60:0/3923637777 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c15f200 con 0x7f1b8c15aaa0 2023-10-16T12:52:21.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.600+0000 7f1b90eb1640 1 --2- 172.21.15.60:0/3923637777 >> v2:172.21.15.144:3300/0 conn(0x7f1b8c15ae80 0x7f1b8c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:21.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.601+0000 7f1b916b2640 1 --2- 172.21.15.60:0/3923637777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c15e820 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0b45e0 tx=0x7f1b8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0013f3d83f11a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.601+0000 7f1b7ffff640 1 -- 172.21.15.60:0/3923637777 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b880317e0 con 0x7f1b8c15aaa0 2023-10-16T12:52:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.601+0000 7f1b7ffff640 1 -- 172.21.15.60:0/3923637777 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88031940 con 0x7f1b8c15aaa0 2023-10-16T12:52:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.601+0000 7f1b7ffff640 1 -- 172.21.15.60:0/3923637777 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88031cd0 con 0x7f1b8c15aaa0 2023-10-16T12:52:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 -- 172.21.15.60:0/3923637777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c15aaa0 msgr2=0x7f1b8c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 --2- 172.21.15.60:0/3923637777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c15e820 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0b45e0 tx=0x7f1b8802b880 comp rx=0 tx=0).stop 2023-10-16T12:52:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 -- 172.21.15.60:0/3923637777 shutdown_connections 2023-10-16T12:52:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 --2- 172.21.15.60:0/3923637777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c0ccaa0 0x7f1b8c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 --2- 172.21.15.60:0/3923637777 >> v2:172.21.15.144:3300/0 conn(0x7f1b8c15ae80 0x7f1b8c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 --2- 172.21.15.60:0/3923637777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c15e820 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 -- 172.21.15.60:0/3923637777 >> 172.21.15.60:0/3923637777 conn(0x7f1b8c0cb410 msgr2=0x7f1b8c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 -- 172.21.15.60:0/3923637777 shutdown_connections 2023-10-16T12:52:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.602+0000 7f1b93916640 1 -- 172.21.15.60:0/3923637777 wait complete. 2023-10-16T12:52:21.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.604+0000 7f1b93916640 1 Processor -- start 2023-10-16T12:52:21.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.604+0000 7f1b93916640 1 -- start start 2023-10-16T12:52:21.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b93916640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 0x7f1b8c0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b93916640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c0c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b93916640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c15ae80 0x7f1b8c0c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b93916640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0c6080 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b93916640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0c4810 con 0x7f1b8c15aaa0 2023-10-16T12:52:21.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b93916640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0c4af0 con 0x7f1b8c15ae80 2023-10-16T12:52:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 0x7f1b8c0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 0x7f1b8c0c4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49800/0 (socket says 172.21.15.60:49800) 2023-10-16T12:52:21.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 -- 172.21.15.60:0/1335404892 learned_addr learned my addr 172.21.15.60:0/1335404892 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 -- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c15ae80 msgr2=0x7f1b8c0c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c15ae80 0x7f1b8c0c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b90eb1640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c0c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 -- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b8c15aaa0 msgr2=0x7f1b8c0c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c0c5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.605+0000 7f1b916b2640 1 -- 172.21.15.60:0/1335404892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880094d0 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.606+0000 7f1b916b2640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 0x7f1b8c0c4180 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f1b88000c00 tx=0x7f1b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:21.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.606+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88032900 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.606+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88032a60 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.606+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88032e10 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.606+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c0c4dd0 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.606+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c15c2b0 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.607+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c0ce310 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.608+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1b8804e450 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.609+0000 7f1b7dffb640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f1b7006af90 0x7f1b7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:21.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.609+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f1b880abc50 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.610+0000 7f1b90eb1640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f1b7006af90 0x7f1b7006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:21.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.611+0000 7f1b90eb1640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f1b7006af90 0x7f1b7006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c15be20 tx=0x7f1b80008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:21.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.614+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b88076ae0 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.822+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1b8c04e160 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.828+0000 7f1b7dffb640 1 -- 172.21.15.60:0/1335404892 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1b8807a100 con 0x7f1b8c0ccaa0 2023-10-16T12:52:21.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f1b7006af90 msgr2=0x7f1b7006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f1b7006af90 0x7f1b7006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c15be20 tx=0x7f1b80008040 comp rx=0 tx=0).stop 2023-10-16T12:52:21.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 msgr2=0x7f1b8c0c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:21.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 0x7f1b8c0c4180 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f1b88000c00 tx=0x7f1b880046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 shutdown_connections 2023-10-16T12:52:21.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f1b7006af90 0x7f1b7006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b8c15ae80 0x7f1b8c0c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b8c15aaa0 0x7f1b8c0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 --2- 172.21.15.60:0/1335404892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b8c0ccaa0 0x7f1b8c0c4180 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:21.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 >> 172.21.15.60:0/1335404892 conn(0x7f1b8c0cb410 msgr2=0x7f1b8c0b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 shutdown_connections 2023-10-16T12:52:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:21.831+0000 7f1b93916640 1 -- 172.21.15.60:0/1335404892 wait complete. 2023-10-16T12:52:21.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:52:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.080+0000 7fc1ef59e640 1 Processor -- start 2023-10-16T12:52:22.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.080+0000 7fc1ef59e640 1 -- start start 2023-10-16T12:52:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.081+0000 7fc1ef59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 0x7fc1f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1ef59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f015f4a0 con 0x7fc1f00ad970 2023-10-16T12:52:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1ef59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f006c960 con 0x7fc1f00cda80 2023-10-16T12:52:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1ef59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f015b9d0 con 0x7fc1f006c040 2023-10-16T12:52:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1edd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc1f00ad970 0x7fc1f015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50722/0 (socket says 172.21.15.60:50722) 2023-10-16T12:52:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1edd9b640 1 -- 172.21.15.60:0/1344334991 learned_addr learned my addr 172.21.15.60:0/1344334991 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1ee59c640 1 --2- 172.21.15.60:0/1344334991 >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 0x7fc1f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363847645 0 0) 0x7fc1f015f4a0 con 0x7fc1f00ad970 2023-10-16T12:52:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.082+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1dc003520 con 0x7fc1f00ad970 2023-10-16T12:52:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747664373 0 0) 0x7fc1dc003520 con 0x7fc1f00ad970 2023-10-16T12:52:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 msgr2=0x7fc1f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 --2- 172.21.15.60:0/1344334991 >> v2:172.21.15.144:3300/0 conn(0x7fc1f006c040 0x7fc1f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 >> v1:172.21.15.144:6789/0 conn(0x7fc1f00cda80 legacy=0x7fc1f015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f015bcb0 con 0x7fc1f00ad970 2023-10-16T12:52:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc1d8001670 con 0x7fc1f00ad970 2023-10-16T12:52:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 >> v1:172.21.15.60:6789/0 conn(0x7fc1f00ad970 legacy=0x7fc1f015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003970 0x7fc1dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc005180 0x7fc1dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f015b9d0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc00c020 con 0x7fc1dc005180 2023-10-16T12:52:22.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc00c1e0 con 0x7fc1dc003970 2023-10-16T12:52:22.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.083+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1eed9d640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc005180 0x7fc1dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1ee59c640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003970 0x7fc1dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1edd9b640 1 -- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003970 msgr2=0x7fc1dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:22.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1dc003970 0x7fc1dc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1edd9b640 1 -- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc005180 msgr2=0x7fc1dc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:22.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1dc005180 0x7fc1dc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1edd9b640 1 -- 172.21.15.60:0/1344334991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc003520 con 0x7fc1dc0047c0 2023-10-16T12:52:22.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.084+0000 7fc1eed9d640 1 -- 172.21.15.60:0/1344334991 reap_dead start 2023-10-16T12:52:22.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.085+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1dc004c40 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c210 tx=0x7fc1dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cc8b6c427c3faf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:22.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.085+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8005ad0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.085+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d8005c30 con 0x7fc1dc0047c0 2023-10-16T12:52:22.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.085+0000 7fc1ed59a640 1 -- 172.21.15.60:0/1344334991 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d802cad0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 -- 172.21.15.60:0/1344334991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 msgr2=0x7fc1dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1dc004c40 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c210 tx=0x7fc1dc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 -- 172.21.15.60:0/1344334991 shutdown_connections 2023-10-16T12:52:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/1344334991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1dc004c40 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 -- 172.21.15.60:0/1344334991 >> 172.21.15.60:0/1344334991 conn(0x7fc1f00ce6a0 msgr2=0x7fc1f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 -- 172.21.15.60:0/1344334991 shutdown_connections 2023-10-16T12:52:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.086+0000 7fc1ef59e640 1 -- 172.21.15.60:0/1344334991 wait complete. 2023-10-16T12:52:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.088+0000 7fc1ef59e640 1 Processor -- start 2023-10-16T12:52:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.088+0000 7fc1ef59e640 1 -- start start 2023-10-16T12:52:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ef59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1f014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ef59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00b3520 0x7fc1f00b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ef59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f014fbd0 0x7fc1f015d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ef59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00b2a70 con 0x7fc1dc0047c0 2023-10-16T12:52:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ef59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f014fa10 con 0x7fc1f014fbd0 2023-10-16T12:52:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ef59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00c7a80 con 0x7fc1f00b3520 2023-10-16T12:52:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ee59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1f014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ee59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1f014f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49826/0 (socket says 172.21.15.60:49826) 2023-10-16T12:52:22.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1ee59c640 1 -- 172.21.15.60:0/3604620584 learned_addr learned my addr 172.21.15.60:0/3604620584 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:22.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.089+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00b3520 0x7fc1f00b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ee59c640 1 -- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00b3520 msgr2=0x7fc1f00b1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ee59c640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00b3520 0x7fc1f00b1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1eed9d640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f014fbd0 0x7fc1f015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ee59c640 1 -- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f014fbd0 msgr2=0x7fc1f015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ee59c640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f014fbd0 0x7fc1f015d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ee59c640 1 -- 172.21.15.60:0/3604620584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00b21d0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00b3520 0x7fc1f00b1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:22.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1eed9d640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f014fbd0 0x7fc1f015d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:22.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ee59c640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1f014f4d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009f30 tx=0x7fc1e00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:22.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000e030 con 0x7fc1dc0047c0 2023-10-16T12:52:22.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0011ea0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00166b0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f00b24b0 con 0x7fc1dc0047c0 2023-10-16T12:52:22.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.090+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f00c5910 con 0x7fc1dc0047c0 2023-10-16T12:52:22.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.092+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f006b330 con 0x7fc1dc0047c0 2023-10-16T12:52:22.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.093+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc1e0016810 con 0x7fc1dc0047c0 2023-10-16T12:52:22.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.094+0000 7fc1e6ffd640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc1d006afc0 0x7fc1d006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.094+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fc1e000f070 con 0x7fc1dc0047c0 2023-10-16T12:52:22.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.095+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc1d006afc0 0x7fc1d006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.096+0000 7fc1edd9b640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc1d006afc0 0x7fc1d006d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8034000 tx=0x7fc1d8009030 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:22.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.099+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc1e0055600 con 0x7fc1dc0047c0 2023-10-16T12:52:22.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.294+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc1f00c5750 con 0x7fc1dc0047c0 2023-10-16T12:52:22.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.301+0000 7fc1e6ffd640 1 -- 172.21.15.60:0/3604620584 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc1e0058c20 con 0x7fc1dc0047c0 2023-10-16T12:52:22.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc1d006afc0 msgr2=0x7fc1d006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc1d006afc0 0x7fc1d006d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8034000 tx=0x7fc1d8009030 comp rx=0 tx=0).stop 2023-10-16T12:52:22.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 msgr2=0x7fc1f014f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1f014f4d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009f30 tx=0x7fc1e00115a0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 shutdown_connections 2023-10-16T12:52:22.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc1d006afc0 0x7fc1d006d480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1f00b3520 0x7fc1f00b1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1f014fbd0 0x7fc1f015d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 --2- 172.21.15.60:0/3604620584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0047c0 0x7fc1f014f4d0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 >> 172.21.15.60:0/3604620584 conn(0x7fc1f00ce6a0 msgr2=0x7fc1f015c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:22.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 shutdown_connections 2023-10-16T12:52:22.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.304+0000 7fc1ef59e640 1 -- 172.21.15.60:0/3604620584 wait complete. 2023-10-16T12:52:22.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:52:22.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.552+0000 7ffa7b467640 1 Processor -- start 2023-10-16T12:52:22.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.552+0000 7ffa7b467640 1 -- start start 2023-10-16T12:52:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7b467640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa7406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7b467640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa740cda80 0x7ffa7415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7b467640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7406c960 con 0x7ffa740ad970 2023-10-16T12:52:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7b467640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa7415b9d0 con 0x7ffa740cda80 2023-10-16T12:52:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7b467640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7415bcb0 con 0x7ffa7406c040 2023-10-16T12:52:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa740cda80 0x7ffa7415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa740cda80 0x7ffa7415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49840/0 (socket says 172.21.15.60:49840) 2023-10-16T12:52:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa79c64640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffa740ad970 0x7ffa7415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50728/0 (socket says 172.21.15.60:50728) 2023-10-16T12:52:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 -- 172.21.15.60:0/4079748516 learned_addr learned my addr 172.21.15.60:0/4079748516 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7a465640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa7406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 -- 172.21.15.60:0/4079748516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 msgr2=0x7ffa7406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa7406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 -- 172.21.15.60:0/4079748516 >> v1:172.21.15.60:6789/0 conn(0x7ffa740ad970 legacy=0x7ffa7415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7ac66640 1 -- 172.21.15.60:0/4079748516 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7415bf90 con 0x7ffa740cda80 2023-10-16T12:52:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa7a465640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa7406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.554+0000 7ffa79463640 1 -- 172.21.15.60:0/4079748516 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910410876 0 0) 0x7ffa7406c960 con 0x7ffa740ad970 2023-10-16T12:52:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.555+0000 7ffa7ac66640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740cda80 0x7ffa7415ed90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009e50 tx=0x7ffa6c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3370464f529b937 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:22.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa79463640 1 -- 172.21.15.60:0/4079748516 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c02cb00 con 0x7ffa740cda80 2023-10-16T12:52:22.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa79463640 1 -- 172.21.15.60:0/4079748516 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6c02cc60 con 0x7ffa740cda80 2023-10-16T12:52:22.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa79463640 1 -- 172.21.15.60:0/4079748516 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c0343f0 con 0x7ffa740cda80 2023-10-16T12:52:22.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 -- 172.21.15.60:0/4079748516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740cda80 msgr2=0x7ffa7415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740cda80 0x7ffa7415ed90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009e50 tx=0x7ffa6c02bf60 comp rx=0 tx=0).stop 2023-10-16T12:52:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 -- 172.21.15.60:0/4079748516 shutdown_connections 2023-10-16T12:52:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740cda80 0x7ffa7415ed90 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 --2- 172.21.15.60:0/4079748516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa7406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 -- 172.21.15.60:0/4079748516 >> 172.21.15.60:0/4079748516 conn(0x7ffa740ce6a0 msgr2=0x7ffa740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 -- 172.21.15.60:0/4079748516 shutdown_connections 2023-10-16T12:52:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.556+0000 7ffa7b467640 1 -- 172.21.15.60:0/4079748516 wait complete. 2023-10-16T12:52:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 Processor -- start 2023-10-16T12:52:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 -- start start 2023-10-16T12:52:22.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa741f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 0x7ffa740c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa740cda80 0x7ffa741461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74158ee0 con 0x7ffa740ad970 2023-10-16T12:52:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740b4070 con 0x7ffa740cda80 2023-10-16T12:52:22.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.559+0000 7ffa7b467640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74149470 con 0x7ffa7406c040 2023-10-16T12:52:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 0x7ffa740c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 0x7ffa740c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49854/0 (socket says 172.21.15.60:49854) 2023-10-16T12:52:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 -- 172.21.15.60:0/1278246974 learned_addr learned my addr 172.21.15.60:0/1278246974 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa7ac66640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa740cda80 0x7ffa741461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 -- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 msgr2=0x7ffa741f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa7a465640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa741f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa741f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 -- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa740cda80 msgr2=0x7ffa741461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa740cda80 0x7ffa741461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 -- 172.21.15.60:0/1278246974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c009b30 con 0x7ffa740ad970 2023-10-16T12:52:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa7ac66640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa740cda80 0x7ffa741461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.560+0000 7ffa79c64640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 0x7ffa740c9670 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400c430 tx=0x7ffa6400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.561+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640074e0 con 0x7ffa740ad970 2023-10-16T12:52:22.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.561+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64007660 con 0x7ffa740ad970 2023-10-16T12:52:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.561+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64004a20 con 0x7ffa740ad970 2023-10-16T12:52:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.561+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa74149750 con 0x7ffa740ad970 2023-10-16T12:52:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.561+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa74149d10 con 0x7ffa740ad970 2023-10-16T12:52:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.562+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7406b330 con 0x7ffa740ad970 2023-10-16T12:52:22.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.563+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa64004b80 con 0x7ffa740ad970 2023-10-16T12:52:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.565+0000 7ffa6affd640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ffa5806af90 0x7ffa5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:22.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.565+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7ffa640895e0 con 0x7ffa740ad970 2023-10-16T12:52:22.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.565+0000 7ffa7a465640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ffa5806af90 0x7ffa5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:22.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.566+0000 7ffa7a465640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ffa5806af90 0x7ffa5806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffa700096d0 tx=0x7ffa70008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:22.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.569+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa64054560 con 0x7ffa740ad970 2023-10-16T12:52:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.769+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ffa7404e9b0 con 0x7ffa740ad970 2023-10-16T12:52:22.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.774+0000 7ffa6affd640 1 -- 172.21.15.60:0/1278246974 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7ffa64057b80 con 0x7ffa740ad970 2023-10-16T12:52:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ffa5806af90 msgr2=0x7ffa5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ffa5806af90 0x7ffa5806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffa700096d0 tx=0x7ffa70008040 comp rx=0 tx=0).stop 2023-10-16T12:52:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 msgr2=0x7ffa740c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 0x7ffa740c9670 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400c430 tx=0x7ffa6400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 shutdown_connections 2023-10-16T12:52:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7ffa5806af90 0x7ffa5806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa740cda80 0x7ffa741461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740ad970 0x7ffa740c9670 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 --2- 172.21.15.60:0/1278246974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa7406c040 0x7ffa741f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 >> 172.21.15.60:0/1278246974 conn(0x7ffa740ce6a0 msgr2=0x7ffa74069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 shutdown_connections 2023-10-16T12:52:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.777+0000 7ffa7b467640 1 -- 172.21.15.60:0/1278246974 wait complete. 2023-10-16T12:52:22.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T12:52:23.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.999+0000 7f897de83640 1 Processor -- start 2023-10-16T12:52:23.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:22.999+0000 7f897de83640 1 -- start start 2023-10-16T12:52:23.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.001+0000 7f897de83640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89780b0b20 0x7f89781615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.001+0000 7f897de83640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f897815d3f0 0x7f897815d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.001+0000 7f897de83640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897815e3e0 con 0x7f89780cd0c0 2023-10-16T12:52:23.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.001+0000 7f897de83640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89780cd4a0 con 0x7f89780b0b20 2023-10-16T12:52:23.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.001+0000 7f897de83640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f897815e100 con 0x7f897815d3f0 2023-10-16T12:52:23.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.001+0000 7f89777fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f89780cd0c0 0x7f89780b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50740/0 (socket says 172.21.15.60:50740) 2023-10-16T12:52:23.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f89777fe640 1 -- 172.21.15.60:0/2057720653 learned_addr learned my addr 172.21.15.60:0/2057720653 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:23.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f896ffff640 1 --2- 172.21.15.60:0/2057720653 >> v2:172.21.15.60:3300/0 conn(0x7f89780b0b20 0x7f89781615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f8977fff640 1 --2- 172.21.15.60:0/2057720653 >> v2:172.21.15.144:3300/0 conn(0x7f897815d3f0 0x7f897815d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f8976ffd640 1 -- 172.21.15.60:0/2057720653 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565026300 0 0) 0x7f897815e3e0 con 0x7f89780cd0c0 2023-10-16T12:52:23.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f8976ffd640 1 -- 172.21.15.60:0/2057720653 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8964003520 con 0x7f89780cd0c0 2023-10-16T12:52:23.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f896ffff640 1 -- 172.21.15.60:0/2057720653 >> v2:172.21.15.144:3300/0 conn(0x7f897815d3f0 msgr2=0x7f897815d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:23.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f896ffff640 1 --2- 172.21.15.60:0/2057720653 >> v2:172.21.15.144:3300/0 conn(0x7f897815d3f0 0x7f897815d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f896ffff640 1 -- 172.21.15.60:0/2057720653 >> v1:172.21.15.60:6789/0 conn(0x7f89780cd0c0 legacy=0x7f89780b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.002+0000 7f896ffff640 1 -- 172.21.15.60:0/2057720653 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8978160b10 con 0x7f89780b0b20 2023-10-16T12:52:23.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.004+0000 7f896ffff640 1 --2- 172.21.15.60:0/2057720653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780b0b20 0x7f89781615f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f897806da60 tx=0x7f896002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c774e5ea79857d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:23.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.004+0000 7f8976ffd640 1 -- 172.21.15.60:0/2057720653 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896002e070 con 0x7f89780b0b20 2023-10-16T12:52:23.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.004+0000 7f8976ffd640 1 -- 172.21.15.60:0/2057720653 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89600315a0 con 0x7f89780b0b20 2023-10-16T12:52:23.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.004+0000 7f8976ffd640 1 -- 172.21.15.60:0/2057720653 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960031930 con 0x7f89780b0b20 2023-10-16T12:52:23.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.004+0000 7f897de83640 1 -- 172.21.15.60:0/2057720653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780b0b20 msgr2=0x7f89781615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.004+0000 7f897de83640 1 --2- 172.21.15.60:0/2057720653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780b0b20 0x7f89781615f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f897806da60 tx=0x7f896002b840 comp rx=0 tx=0).stop 2023-10-16T12:52:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.005+0000 7f897de83640 1 -- 172.21.15.60:0/2057720653 shutdown_connections 2023-10-16T12:52:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.005+0000 7f897de83640 1 --2- 172.21.15.60:0/2057720653 >> v2:172.21.15.144:3300/0 conn(0x7f897815d3f0 0x7f897815d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.005+0000 7f897de83640 1 --2- 172.21.15.60:0/2057720653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780b0b20 0x7f89781615f0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.005+0000 7f897de83640 1 -- 172.21.15.60:0/2057720653 >> 172.21.15.60:0/2057720653 conn(0x7f89780cb410 msgr2=0x7f89780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.005+0000 7f897de83640 1 -- 172.21.15.60:0/2057720653 shutdown_connections 2023-10-16T12:52:23.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.005+0000 7f897de83640 1 -- 172.21.15.60:0/2057720653 wait complete. 2023-10-16T12:52:23.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.007+0000 7f897de83640 1 Processor -- start 2023-10-16T12:52:23.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.007+0000 7f897de83640 1 -- start start 2023-10-16T12:52:23.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f897de83640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89780b0b20 0x7f897806cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f897de83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 0x7f897806d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f897de83640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f897815d3f0 0x7f89780b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f897de83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89780c3530 con 0x7f89780cd0c0 2023-10-16T12:52:23.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f897de83640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89780c2920 con 0x7f89780b0b20 2023-10-16T12:52:23.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f897de83640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89780c2c00 con 0x7f897815d3f0 2023-10-16T12:52:23.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f896ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 0x7f897806d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f896ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 0x7f897806d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49868/0 (socket says 172.21.15.60:49868) 2023-10-16T12:52:23.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f896ffff640 1 -- 172.21.15.60:0/1424553329 learned_addr learned my addr 172.21.15.60:0/1424553329 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f89777fe640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89780b0b20 0x7f897806cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f8977fff640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f897815d3f0 0x7f89780b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f896ffff640 1 -- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f897815d3f0 msgr2=0x7f89780b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.008+0000 7f896ffff640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f897815d3f0 0x7f89780b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f896ffff640 1 -- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89780b0b20 msgr2=0x7f897806cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f896ffff640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89780b0b20 0x7f897806cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f896ffff640 1 -- 172.21.15.60:0/1424553329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8960009400 con 0x7f89780cd0c0 2023-10-16T12:52:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f896ffff640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 0x7f897806d130 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f8960009890 tx=0x7f89600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960032840 con 0x7f89780cd0c0 2023-10-16T12:52:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8960031f00 con 0x7f89780cd0c0 2023-10-16T12:52:23.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89780c2ee0 con 0x7f89780cd0c0 2023-10-16T12:52:23.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960032be0 con 0x7f89780cd0c0 2023-10-16T12:52:23.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.009+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89780b42e0 con 0x7f89780cd0c0 2023-10-16T12:52:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.011+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897804e160 con 0x7f89780cd0c0 2023-10-16T12:52:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.012+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8960032d40 con 0x7f89780cd0c0 2023-10-16T12:52:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.013+0000 7f8974ff9640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f895806afc0 0x7f895806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.013+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f89600ab6b0 con 0x7f89780cd0c0 2023-10-16T12:52:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.014+0000 7f89777fe640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f895806afc0 0x7f895806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.015+0000 7f89777fe640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f895806afc0 0x7f895806d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89680011e0 tx=0x7f8968009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.019+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8960076670 con 0x7f89780cd0c0 2023-10-16T12:52:23.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.291+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f89780b3860 con 0x7f89780cd0c0 2023-10-16T12:52:23.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.360+0000 7f8974ff9640 1 -- 172.21.15.60:0/1424553329 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v713) v1 ==== 183+0+0 (secure 0 0 0) 0x7f8960079c90 con 0x7f89780cd0c0 2023-10-16T12:52:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.363+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f895806afc0 msgr2=0x7f895806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.363+0000 7f897de83640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f895806afc0 0x7f895806d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89680011e0 tx=0x7f8968009040 comp rx=0 tx=0).stop 2023-10-16T12:52:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.363+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 msgr2=0x7f897806d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.363+0000 7f897de83640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 0x7f897806d130 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f8960009890 tx=0x7f89600046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 shutdown_connections 2023-10-16T12:52:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f895806afc0 0x7f895806d480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f897815d3f0 0x7f89780b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89780cd0c0 0x7f897806d130 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 --2- 172.21.15.60:0/1424553329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89780b0b20 0x7f897806cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 >> 172.21.15.60:0/1424553329 conn(0x7f89780cb410 msgr2=0x7f89780ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 shutdown_connections 2023-10-16T12:52:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.364+0000 7f897de83640 1 -- 172.21.15.60:0/1424553329 wait complete. 2023-10-16T12:52:23.387 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-16T12:52:23.387 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-16T12:52:23.387 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:52:23.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:23.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.614+0000 7fd9e75b0640 1 Processor -- start 2023-10-16T12:52:23.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.614+0000 7fd9e75b0640 1 -- start start 2023-10-16T12:52:23.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.615+0000 7fd9e75b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9e015aaa0 0x7fd9e015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.615+0000 7fd9e75b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e015ae80 0x7fd9e00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.615+0000 7fd9e75b0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e015ed60 con 0x7fd9e00ccaa0 2023-10-16T12:52:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.615+0000 7fd9e75b0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e006b810 con 0x7fd9e015aaa0 2023-10-16T12:52:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.615+0000 7fd9e75b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00707e0 con 0x7fd9e015ae80 2023-10-16T12:52:23.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.616+0000 7fd9e534c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9e015aaa0 0x7fd9e015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.616+0000 7fd9e4b4b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e015ae80 0x7fd9e00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.616+0000 7fd9e5b4d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd9e00ccaa0 0x7fd9e0070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:46744/0 (socket says 172.21.15.60:46744) 2023-10-16T12:52:23.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.616+0000 7fd9e5b4d640 1 -- 172.21.15.60:0/2610713213 learned_addr learned my addr 172.21.15.60:0/2610713213 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:23.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.617+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/2610713213 >> v2:172.21.15.144:3300/0 conn(0x7fd9e015aaa0 msgr2=0x7fd9e015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:23.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.617+0000 7fd9e4b4b640 1 --2- 172.21.15.60:0/2610713213 >> v2:172.21.15.144:3300/0 conn(0x7fd9e015aaa0 0x7fd9e015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.617+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/2610713213 >> v1:172.21.15.144:6789/0 conn(0x7fd9e00ccaa0 legacy=0x7fd9e0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.617+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/2610713213 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e015ef90 con 0x7fd9e015ae80 2023-10-16T12:52:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.619+0000 7fd9e4b4b640 1 --2- 172.21.15.60:0/2610713213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e015ae80 0x7fd9e00cc560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7868e5388968 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9d7fff640 1 -- 172.21.15.60:0/2610713213 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d002d040 con 0x7fd9e015ae80 2023-10-16T12:52:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9d7fff640 1 -- 172.21.15.60:0/2610713213 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d0031810 con 0x7fd9e015ae80 2023-10-16T12:52:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9d7fff640 1 -- 172.21.15.60:0/2610713213 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0031ba0 con 0x7fd9e015ae80 2023-10-16T12:52:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 -- 172.21.15.60:0/2610713213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e015ae80 msgr2=0x7fd9e00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/2610713213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e015ae80 0x7fd9e00cc560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b8b0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 -- 172.21.15.60:0/2610713213 shutdown_connections 2023-10-16T12:52:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/2610713213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e015ae80 0x7fd9e00cc560 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/2610713213 >> v2:172.21.15.144:3300/0 conn(0x7fd9e015aaa0 0x7fd9e015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 -- 172.21.15.60:0/2610713213 >> 172.21.15.60:0/2610713213 conn(0x7fd9e00cb410 msgr2=0x7fd9e00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 -- 172.21.15.60:0/2610713213 shutdown_connections 2023-10-16T12:52:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.620+0000 7fd9e75b0640 1 -- 172.21.15.60:0/2610713213 wait complete. 2023-10-16T12:52:23.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.621+0000 7fd9e75b0640 1 Processor -- start 2023-10-16T12:52:23.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 -- start start 2023-10-16T12:52:23.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e00ccaa0 0x7fd9e0159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 0x7fd9e00c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9e015ae80 0x7fd9e00c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015a420 con 0x7fd9e015aaa0 2023-10-16T12:52:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015d3f0 con 0x7fd9e015ae80 2023-10-16T12:52:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.622+0000 7fd9e75b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015d6d0 con 0x7fd9e00ccaa0 2023-10-16T12:52:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 0x7fd9e00c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 0x7fd9e00c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49874/0 (socket says 172.21.15.60:49874) 2023-10-16T12:52:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/1957736696 learned_addr learned my addr 172.21.15.60:0/1957736696 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e5b4d640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9e015ae80 0x7fd9e00c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e534c640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e00ccaa0 0x7fd9e0159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e00ccaa0 msgr2=0x7fd9e0159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e00ccaa0 0x7fd9e0159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9e015ae80 msgr2=0x7fd9e00c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9e015ae80 0x7fd9e00c3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.623+0000 7fd9e4b4b640 1 -- 172.21.15.60:0/1957736696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0009400 con 0x7fd9e015aaa0 2023-10-16T12:52:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.624+0000 7fd9e4b4b640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 0x7fd9e00c3a90 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098c0 tx=0x7fd9d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.624+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0006e10 con 0x7fd9e015aaa0 2023-10-16T12:52:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.624+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d003ad60 con 0x7fd9e015aaa0 2023-10-16T12:52:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.624+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0044610 con 0x7fd9e015aaa0 2023-10-16T12:52:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.624+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e015d9b0 con 0x7fd9e015aaa0 2023-10-16T12:52:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.624+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e015df70 con 0x7fd9e015aaa0 2023-10-16T12:52:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.625+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e004e160 con 0x7fd9e015aaa0 2023-10-16T12:52:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.626+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd9d004d450 con 0x7fd9e015aaa0 2023-10-16T12:52:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.627+0000 7fd9d5ffb640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd9c406af90 0x7fd9c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.627+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7fd9d00abaf0 con 0x7fd9e015aaa0 2023-10-16T12:52:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.628+0000 7fd9e534c640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd9c406af90 0x7fd9c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.629+0000 7fd9e534c640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd9c406af90 0x7fd9c406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd9e01f5ee0 tx=0x7fd9dc008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.632+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9d00769e0 con 0x7fd9e015aaa0 2023-10-16T12:52:23.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.833+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9e00b06b0 con 0x7fd9e015aaa0 2023-10-16T12:52:23.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.837+0000 7fd9d5ffb640 1 -- 172.21.15.60:0/1957736696 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+14125 (secure 0 0 0) 0x7fd9d00abe60 con 0x7fd9e015aaa0 2023-10-16T12:52:23.839 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:23.839 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":120,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:21.341359+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/3569389918":"2023-10-17T12:52:18.329187+0000","172.21.15.60:6833/3569389918":"2023-10-17T12:52:18.329187+0000","172.21.15.144:0/2025887462":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/948877502":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/1510335921":"2023-10-17T12:52:14.184032+0000","172.21.15.144:6833/2022371982":"2023-10-17T12:52:14.184032+0000","172.21.15.144:6832/2022371982":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/337630602":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/671149719":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/3635246428":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/2189557198":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1153724330":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1224606105":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/1935697927":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/840292481":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6833/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/737927288":"2023-10-17T12:52:14.184032+0000","172.21.15.144:0/1070434778":"2023-10-17T12:51:06.102326+0000","172.21.15.144:6832/3513615444":"2023-10-17T12:51:06.102326+0000","172.21.15.144:0/4048637185":"2023-10-17T12:51:06.102326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:23.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd9c406af90 msgr2=0x7fd9c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd9c406af90 0x7fd9c406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd9e01f5ee0 tx=0x7fd9dc008040 comp rx=0 tx=0).stop 2023-10-16T12:52:23.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 msgr2=0x7fd9e00c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:23.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 0x7fd9e00c3a90 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098c0 tx=0x7fd9d00046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 shutdown_connections 2023-10-16T12:52:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd9c406af90 0x7fd9c406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9e015ae80 0x7fd9e00c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9e015aaa0 0x7fd9e00c3a90 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 --2- 172.21.15.60:0/1957736696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9e00ccaa0 0x7fd9e0159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 >> 172.21.15.60:0/1957736696 conn(0x7fd9e00cb410 msgr2=0x7fd9e00b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.839+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 shutdown_connections 2023-10-16T12:52:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:23.841+0000 7fd9e75b0640 1 -- 172.21.15.60:0/1957736696 wait complete. 2023-10-16T12:52:23.861 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:52:23.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:52:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.089+0000 7f5a1b59e640 1 Processor -- start 2023-10-16T12:52:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.089+0000 7f5a1b59e640 1 -- start start 2023-10-16T12:52:24.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a1b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a1c070f30 0x7f5a1c06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a1b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c0cd260 con 0x7f5a1c159ad0 2023-10-16T12:52:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a1b59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c15de30 con 0x7f5a1c06b3e0 2023-10-16T12:52:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a1b59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1c15e110 con 0x7f5a1c070f30 2023-10-16T12:52:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a1a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5a1c159ad0 0x7f5a1c15d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50746/0 (socket says 172.21.15.60:50746) 2023-10-16T12:52:24.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a1a59c640 1 -- 172.21.15.60:0/1365591877 learned_addr learned my addr 172.21.15.60:0/1365591877 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.091+0000 7f5a13fff640 1 --2- 172.21.15.60:0/1365591877 >> v2:172.21.15.144:3300/0 conn(0x7f5a1c070f30 0x7f5a1c06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845406831 0 0) 0x7f5a1c0cd260 con 0x7f5a1c159ad0 2023-10-16T12:52:24.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a08005730 con 0x7f5a1c159ad0 2023-10-16T12:52:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219341443 0 0) 0x7f5a08005730 con 0x7f5a1c159ad0 2023-10-16T12:52:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 >> v2:172.21.15.144:3300/0 conn(0x7f5a1c070f30 msgr2=0x7f5a1c06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 --2- 172.21.15.60:0/1365591877 >> v2:172.21.15.144:3300/0 conn(0x7f5a1c070f30 0x7f5a1c06aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 >> v1:172.21.15.144:6789/0 conn(0x7f5a1c06b3e0 legacy=0x7f5a1c0c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c15e3f0 con 0x7f5a1c159ad0 2023-10-16T12:52:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5a0c001670 con 0x7f5a1c159ad0 2023-10-16T12:52:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 >> v1:172.21.15.60:6789/0 conn(0x7f5a1c159ad0 legacy=0x7f5a1c15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a08005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a080069d0 0x7f5a08006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a08007390 0x7f5a0800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c15e110 con 0x7f5a08005bd0 2023-10-16T12:52:24.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0800e290 con 0x7f5a080069d0 2023-10-16T12:52:24.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.092+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0800e450 con 0x7f5a08007390 2023-10-16T12:52:24.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a08005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a13fff640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a080069d0 0x7f5a08006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1ad9d640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a08007390 0x7f5a0800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1a59c640 1 -- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a08007390 msgr2=0x7f5a0800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:24.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a08007390 0x7f5a0800db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1a59c640 1 -- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a080069d0 msgr2=0x7f5a08006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:24.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a080069d0 0x7f5a08006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1a59c640 1 -- 172.21.15.60:0/1365591877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a08005730 con 0x7f5a08005bd0 2023-10-16T12:52:24.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.093+0000 7f5a1ad9d640 1 -- 172.21.15.60:0/1365591877 reap_dead start 2023-10-16T12:52:24.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.095+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a08005ff0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c0e0 tx=0x7f5a08007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91d1a62c6edc939 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:24.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.095+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c005ad0 con 0x7f5a08005bd0 2023-10-16T12:52:24.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.095+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c005c30 con 0x7f5a08005bd0 2023-10-16T12:52:24.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.095+0000 7f5a19d9b640 1 -- 172.21.15.60:0/1365591877 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c02c7d0 con 0x7f5a08005bd0 2023-10-16T12:52:24.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 -- 172.21.15.60:0/1365591877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 msgr2=0x7f5a08005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a08005ff0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c0e0 tx=0x7f5a08007a00 comp rx=0 tx=0).stop 2023-10-16T12:52:24.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 -- 172.21.15.60:0/1365591877 shutdown_connections 2023-10-16T12:52:24.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/1365591877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a08005ff0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 -- 172.21.15.60:0/1365591877 >> 172.21.15.60:0/1365591877 conn(0x7f5a1c0b0170 msgr2=0x7f5a1c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:24.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 -- 172.21.15.60:0/1365591877 shutdown_connections 2023-10-16T12:52:24.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.096+0000 7f5a1b59e640 1 -- 172.21.15.60:0/1365591877 wait complete. 2023-10-16T12:52:24.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.098+0000 7f5a1b59e640 1 Processor -- start 2023-10-16T12:52:24.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.100+0000 7f5a1b59e640 1 -- start start 2023-10-16T12:52:24.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a1c0ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a1c15ab30 0x7f5a1c14cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a1c1575c0 0x7f5a1c157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c14d120 con 0x7f5a08005bd0 2023-10-16T12:52:24.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c157f80 con 0x7f5a1c15ab30 2023-10-16T12:52:24.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0c8ab0 con 0x7f5a1c1575c0 2023-10-16T12:52:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a1c0ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a1c0ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49890/0 (socket says 172.21.15.60:49890) 2023-10-16T12:52:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 -- 172.21.15.60:0/2868357832 learned_addr learned my addr 172.21.15.60:0/2868357832 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:24.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a13fff640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a1c15ab30 0x7f5a1c14cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 -- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a1c1575c0 msgr2=0x7f5a1c157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a1c1575c0 0x7f5a1c157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 -- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a1c15ab30 msgr2=0x7f5a1c14cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a1c15ab30 0x7f5a1c14cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 -- 172.21.15.60:0/2868357832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c15afb0 con 0x7f5a08005bd0 2023-10-16T12:52:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.101+0000 7f5a1a59c640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a1c0ae240 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c200 tx=0x7f5a1c15e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.103+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c03c070 con 0x7f5a08005bd0 2023-10-16T12:52:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.103+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c005e70 con 0x7f5a08005bd0 2023-10-16T12:52:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.103+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c005a40 con 0x7f5a08005bd0 2023-10-16T12:52:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.103+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c1f5980 con 0x7f5a08005bd0 2023-10-16T12:52:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.103+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c1f5f40 con 0x7f5a08005bd0 2023-10-16T12:52:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.103+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c04e9b0 con 0x7f5a08005bd0 2023-10-16T12:52:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.104+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a0c032070 con 0x7f5a08005bd0 2023-10-16T12:52:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.105+0000 7f5a12ffd640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f59fc06af90 0x7f59fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.106+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f5a0c0aa830 con 0x7f5a08005bd0 2023-10-16T12:52:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.106+0000 7f5a13fff640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f59fc06af90 0x7f59fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.107+0000 7f5a13fff640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f59fc06af90 0x7f59fc06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0acd10 tx=0x7f5a04008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.110+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a0c0757b0 con 0x7f5a08005bd0 2023-10-16T12:52:24.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.318+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a1c0b4600 con 0x7f5a08005bd0 2023-10-16T12:52:24.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.320+0000 7f5a12ffd640 1 -- 172.21.15.60:0/2868357832 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a0c078dd0 con 0x7f5a08005bd0 2023-10-16T12:52:24.322 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:52:24.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f59fc06af90 msgr2=0x7f59fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f59fc06af90 0x7f59fc06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0acd10 tx=0x7f5a04008040 comp rx=0 tx=0).stop 2023-10-16T12:52:24.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 msgr2=0x7f5a1c0ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a1c0ae240 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c200 tx=0x7f5a1c15e6d0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 shutdown_connections 2023-10-16T12:52:24.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f59fc06af90 0x7f59fc06d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a1c1575c0 0x7f5a1c157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a1c15ab30 0x7f5a1c14cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 --2- 172.21.15.60:0/2868357832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a08005bd0 0x7f5a1c0ae240 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 >> 172.21.15.60:0/2868357832 conn(0x7f5a1c0b0170 msgr2=0x7f5a1c156aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:24.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 shutdown_connections 2023-10-16T12:52:24.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.324+0000 7f5a1b59e640 1 -- 172.21.15.60:0/2868357832 wait complete. 2023-10-16T12:52:24.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T12:52:24.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.557+0000 7f6f5ffff640 1 Processor -- start 2023-10-16T12:52:24.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.557+0000 7f6f5ffff640 1 -- start start 2023-10-16T12:52:24.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f6015aad0 0x7f6f600cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f60070d70 0x7f6f6006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f600b4620 con 0x7f6f600712c0 2023-10-16T12:52:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f600b4340 con 0x7f6f60070d70 2023-10-16T12:52:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5ffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6006f520 con 0x7f6f6015aad0 2023-10-16T12:52:24.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5e7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f60070d70 0x7f6f6006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5e7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f60070d70 0x7f6f6006bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49902/0 (socket says 172.21.15.60:49902) 2023-10-16T12:52:24.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.559+0000 7f6f5e7fc640 1 -- 172.21.15.60:0/2236193992 learned_addr learned my addr 172.21.15.60:0/2236193992 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:24.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.560+0000 7f6f5effd640 1 --2- 172.21.15.60:0/2236193992 >> v2:172.21.15.144:3300/0 conn(0x7f6f6015aad0 0x7f6f600cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.560+0000 7f6f5e7fc640 1 -- 172.21.15.60:0/2236193992 >> v2:172.21.15.144:3300/0 conn(0x7f6f6015aad0 msgr2=0x7f6f600cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.560+0000 7f6f5e7fc640 1 --2- 172.21.15.60:0/2236193992 >> v2:172.21.15.144:3300/0 conn(0x7f6f6015aad0 0x7f6f600cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.560+0000 7f6f5e7fc640 1 -- 172.21.15.60:0/2236193992 >> v1:172.21.15.144:6789/0 conn(0x7f6f600712c0 legacy=0x7f6f60070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:24.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.560+0000 7f6f5e7fc640 1 -- 172.21.15.60:0/2236193992 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6006f800 con 0x7f6f60070d70 2023-10-16T12:52:24.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.561+0000 7f6f5e7fc640 1 --2- 172.21.15.60:0/2236193992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f6006bf40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6f600cf4f0 tx=0x7f6f5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb29b54271d239e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:24.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.561+0000 7f6f5dffb640 1 -- 172.21.15.60:0/2236193992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54030860 con 0x7f6f60070d70 2023-10-16T12:52:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.561+0000 7f6f5dffb640 1 -- 172.21.15.60:0/2236193992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f540309c0 con 0x7f6f60070d70 2023-10-16T12:52:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.561+0000 7f6f5dffb640 1 -- 172.21.15.60:0/2236193992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54030d50 con 0x7f6f60070d70 2023-10-16T12:52:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 -- 172.21.15.60:0/2236193992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 msgr2=0x7f6f6006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/2236193992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f6006bf40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6f600cf4f0 tx=0x7f6f5402b9a0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 -- 172.21.15.60:0/2236193992 shutdown_connections 2023-10-16T12:52:24.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/2236193992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f6006bf40 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/2236193992 >> v2:172.21.15.144:3300/0 conn(0x7f6f6015aad0 0x7f6f600cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 -- 172.21.15.60:0/2236193992 >> 172.21.15.60:0/2236193992 conn(0x7f6f600b0a90 msgr2=0x7f6f600ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:24.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 -- 172.21.15.60:0/2236193992 shutdown_connections 2023-10-16T12:52:24.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.562+0000 7f6f5ffff640 1 -- 172.21.15.60:0/2236193992 wait complete. 2023-10-16T12:52:24.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.564+0000 7f6f5ffff640 1 Processor -- start 2023-10-16T12:52:24.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.564+0000 7f6f5ffff640 1 -- start start 2023-10-16T12:52:24.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f60148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600712c0 0x7f6f60149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6015aad0 0x7f6f60144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f601434d0 con 0x7f6f60070d70 2023-10-16T12:52:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f601456e0 con 0x7f6f600712c0 2023-10-16T12:52:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f601459c0 con 0x7f6f6015aad0 2023-10-16T12:52:24.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f60148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f60148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49908/0 (socket says 172.21.15.60:49908) 2023-10-16T12:52:24.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.565+0000 7f6f5effd640 1 -- 172.21.15.60:0/3230028032 learned_addr learned my addr 172.21.15.60:0/3230028032 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5e7fc640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600712c0 0x7f6f60149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5effd640 1 -- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6015aad0 msgr2=0x7f6f60144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5effd640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6015aad0 0x7f6f60144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5effd640 1 -- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600712c0 msgr2=0x7f6f60149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5effd640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600712c0 0x7f6f60149570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5effd640 1 -- 172.21.15.60:0/3230028032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f540094d0 con 0x7f6f60070d70 2023-10-16T12:52:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f5effd640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f60148fb0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000d5f0 tx=0x7f6f5000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.566+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500125f0 con 0x7f6f60070d70 2023-10-16T12:52:24.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.567+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50012750 con 0x7f6f60070d70 2023-10-16T12:52:24.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.567+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50012b00 con 0x7f6f60070d70 2023-10-16T12:52:24.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.567+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f60145ca0 con 0x7f6f60070d70 2023-10-16T12:52:24.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.567+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f60146260 con 0x7f6f60070d70 2023-10-16T12:52:24.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.568+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6004e9b0 con 0x7f6f60070d70 2023-10-16T12:52:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.569+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6f50012c60 con 0x7f6f60070d70 2023-10-16T12:52:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.570+0000 7f6f477fe640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f6f4c06afc0 0x7f6f4c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.570+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f6f50089cd0 con 0x7f6f60070d70 2023-10-16T12:52:24.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.571+0000 7f6f5e7fc640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f6f4c06afc0 0x7f6f4c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.572+0000 7f6f5e7fc640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f6f4c06afc0 0x7f6f4c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6f54036000 tx=0x7f6f5403b040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.575+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f50054bc0 con 0x7f6f60070d70 2023-10-16T12:52:24.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:24.845+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f6f600cf520 con 0x7f6f60070d70 2023-10-16T12:52:25.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.376+0000 7f6f477fe640 1 -- 172.21.15.60:0/3230028032 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v715) v1 ==== 151+0+0 (secure 0 0 0) 0x7f6f500581e0 con 0x7f6f60070d70 2023-10-16T12:52:25.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f6f4c06afc0 msgr2=0x7f6f4c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f6f4c06afc0 0x7f6f4c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6f54036000 tx=0x7f6f5403b040 comp rx=0 tx=0).stop 2023-10-16T12:52:25.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 msgr2=0x7f6f60148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f60148fb0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000d5f0 tx=0x7f6f5000dab0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 shutdown_connections 2023-10-16T12:52:25.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f6f4c06afc0 0x7f6f4c06d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6015aad0 0x7f6f60144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600712c0 0x7f6f60149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 --2- 172.21.15.60:0/3230028032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f60070d70 0x7f6f60148fb0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 >> 172.21.15.60:0/3230028032 conn(0x7f6f600b0a90 msgr2=0x7f6f600ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:25.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 shutdown_connections 2023-10-16T12:52:25.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.380+0000 7f6f5ffff640 1 -- 172.21.15.60:0/3230028032 wait complete. 2023-10-16T12:52:25.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:52:25.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.636+0000 7f32595e2640 1 Processor -- start 2023-10-16T12:52:25.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.636+0000 7f32595e2640 1 -- start start 2023-10-16T12:52:25.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.637+0000 7f32595e2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f325415c9f0 0x7f32540c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:25.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.637+0000 7f32595e2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32540c9230 0x7f32540709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:25.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.637+0000 7f32595e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3254160a90 con 0x7f3254070f20 2023-10-16T12:52:25.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.637+0000 7f32595e2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32540cdec0 con 0x7f32540c9230 2023-10-16T12:52:25.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.637+0000 7f32595e2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32540cd900 con 0x7f325415c9f0 2023-10-16T12:52:25.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f32537fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32540c9230 0x7f32540709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:25.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f32537fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32540c9230 0x7f32540709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49914/0 (socket says 172.21.15.60:49914) 2023-10-16T12:52:25.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f32537fe640 1 -- 172.21.15.60:0/2627417718 learned_addr learned my addr 172.21.15.60:0/2627417718 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:25.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f3258de1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3254070f20 0x7f32540ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50762/0 (socket says 172.21.15.60:50762) 2023-10-16T12:52:25.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f3253fff640 1 --2- 172.21.15.60:0/2627417718 >> v2:172.21.15.144:3300/0 conn(0x7f325415c9f0 0x7f32540c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:25.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f32537fe640 1 -- 172.21.15.60:0/2627417718 >> v2:172.21.15.144:3300/0 conn(0x7f325415c9f0 msgr2=0x7f32540c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:25.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f32537fe640 1 --2- 172.21.15.60:0/2627417718 >> v2:172.21.15.144:3300/0 conn(0x7f325415c9f0 0x7f32540c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.638+0000 7f32537fe640 1 -- 172.21.15.60:0/2627417718 >> v1:172.21.15.60:6789/0 conn(0x7f3254070f20 legacy=0x7f32540ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.639+0000 7f32537fe640 1 -- 172.21.15.60:0/2627417718 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540cdbe0 con 0x7f32540c9230 2023-10-16T12:52:25.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.640+0000 7f32537fe640 1 --2- 172.21.15.60:0/2627417718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32540c9230 0x7f32540709e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f32540ad4e0 tx=0x7f323c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22d7412438bc8ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:25.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.640+0000 7f3252ffd640 1 -- 172.21.15.60:0/2627417718 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c030860 con 0x7f32540c9230 2023-10-16T12:52:25.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.640+0000 7f3252ffd640 1 -- 172.21.15.60:0/2627417718 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c0309c0 con 0x7f32540c9230 2023-10-16T12:52:25.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.640+0000 7f3252ffd640 1 -- 172.21.15.60:0/2627417718 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c030d50 con 0x7f32540c9230 2023-10-16T12:52:25.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 -- 172.21.15.60:0/2627417718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32540c9230 msgr2=0x7f32540709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 --2- 172.21.15.60:0/2627417718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32540c9230 0x7f32540709e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f32540ad4e0 tx=0x7f323c02b9a0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 -- 172.21.15.60:0/2627417718 shutdown_connections 2023-10-16T12:52:25.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 --2- 172.21.15.60:0/2627417718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32540c9230 0x7f32540709e0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 --2- 172.21.15.60:0/2627417718 >> v2:172.21.15.144:3300/0 conn(0x7f325415c9f0 0x7f32540c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 -- 172.21.15.60:0/2627417718 >> 172.21.15.60:0/2627417718 conn(0x7f32540ce6a0 msgr2=0x7f32540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:25.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 -- 172.21.15.60:0/2627417718 shutdown_connections 2023-10-16T12:52:25.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.641+0000 7f32595e2640 1 -- 172.21.15.60:0/2627417718 wait complete. 2023-10-16T12:52:25.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.643+0000 7f32595e2640 1 Processor -- start 2023-10-16T12:52:25.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.643+0000 7f32595e2640 1 -- start start 2023-10-16T12:52:25.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f32595e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 0x7f3254151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:25.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f32595e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32540c9230 0x7f3254150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:25.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f32595e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325415c9f0 0x7f325406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:25.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f32595e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540c3230 con 0x7f3254070f20 2023-10-16T12:52:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f32595e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540c26b0 con 0x7f32540c9230 2023-10-16T12:52:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f32595e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32540c2990 con 0x7f325415c9f0 2023-10-16T12:52:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 0x7f3254151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:25.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 0x7f3254151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49920/0 (socket says 172.21.15.60:49920) 2023-10-16T12:52:25.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 -- 172.21.15.60:0/607877647 learned_addr learned my addr 172.21.15.60:0/607877647 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 -- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325415c9f0 msgr2=0x7f325406b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325415c9f0 0x7f325406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 -- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32540c9230 msgr2=0x7f3254150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32540c9230 0x7f3254150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.644+0000 7f3253fff640 1 -- 172.21.15.60:0/607877647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0094d0 con 0x7f3254070f20 2023-10-16T12:52:25.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.645+0000 7f3253fff640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 0x7f3254151480 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f3244007e40 tx=0x7f324400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:25.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.645+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244014620 con 0x7f3254070f20 2023-10-16T12:52:25.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.645+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324400ee90 con 0x7f3254070f20 2023-10-16T12:52:25.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.645+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32440149c0 con 0x7f3254070f20 2023-10-16T12:52:25.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.645+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32540c2c70 con 0x7f3254070f20 2023-10-16T12:52:25.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.645+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32541510a0 con 0x7f3254070f20 2023-10-16T12:52:25.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.646+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325404e9b0 con 0x7f3254070f20 2023-10-16T12:52:25.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.647+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3244014b20 con 0x7f3254070f20 2023-10-16T12:52:25.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.648+0000 7f3250ff9640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f323406af90 0x7f323406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:25.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.648+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f324408a170 con 0x7f3254070f20 2023-10-16T12:52:25.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.649+0000 7f32537fe640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f323406af90 0x7f323406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:25.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.650+0000 7f32537fe640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f323406af90 0x7f323406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f323c036000 tx=0x7f323c03b040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:25.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.653+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3244055060 con 0x7f3254070f20 2023-10-16T12:52:25.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.847+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3254069da0 con 0x7f3254070f20 2023-10-16T12:52:25.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.854+0000 7f3250ff9640 1 -- 172.21.15.60:0/607877647 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3244058680 con 0x7f3254070f20 2023-10-16T12:52:25.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f323406af90 msgr2=0x7f323406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f323406af90 0x7f323406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f323c036000 tx=0x7f323c03b040 comp rx=0 tx=0).stop 2023-10-16T12:52:25.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 msgr2=0x7f3254151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 0x7f3254151480 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f3244007e40 tx=0x7f324400e4b0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 shutdown_connections 2023-10-16T12:52:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f323406af90 0x7f323406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325415c9f0 0x7f325406b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32540c9230 0x7f3254150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 --2- 172.21.15.60:0/607877647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3254070f20 0x7f3254151480 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:25.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.857+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 >> 172.21.15.60:0/607877647 conn(0x7f32540ce6a0 msgr2=0x7f32540c15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:25.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.858+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 shutdown_connections 2023-10-16T12:52:25.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:25.858+0000 7f32595e2640 1 -- 172.21.15.60:0/607877647 wait complete. 2023-10-16T12:52:25.880 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:52:25.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:52:26.009 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:52:26.143 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:52:26.144 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:52:26.144 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:26.209 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:52:26.210 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:52:26.210 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:26.270 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:52:26.270 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:52:26.271 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:26.336 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:52:26.336 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:52:26.337 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:26.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:52:26.400 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:52:26.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:26.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.634+0000 7efd9ffff640 1 Processor -- start 2023-10-16T12:52:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.634+0000 7efd9ffff640 1 -- start start 2023-10-16T12:52:26.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.636+0000 7efd9ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efda015cbe0 0x7efda00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.636+0000 7efd9ffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00d15a0 con 0x7efda00b4c80 2023-10-16T12:52:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.636+0000 7efd9ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00d1880 con 0x7efda015cfc0 2023-10-16T12:52:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.636+0000 7efd9ffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00d1b60 con 0x7efda015cbe0 2023-10-16T12:52:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.636+0000 7efd9f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efda00b4c80 0x7efda00b5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50764/0 (socket says 172.21.15.60:50764) 2023-10-16T12:52:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.636+0000 7efd9f7fe640 1 -- 172.21.15.60:0/1950436439 learned_addr learned my addr 172.21.15.60:0/1950436439 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:26.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9effd640 1 --2- 172.21.15.60:0/1950436439 >> v2:172.21.15.144:3300/0 conn(0x7efda015cbe0 0x7efda00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962538599 0 0) 0x7efda00d15a0 con 0x7efda00b4c80 2023-10-16T12:52:26.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c003520 con 0x7efda00b4c80 2023-10-16T12:52:26.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833772820 0 0) 0x7efd8c003520 con 0x7efda00b4c80 2023-10-16T12:52:26.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 >> v2:172.21.15.144:3300/0 conn(0x7efda015cbe0 msgr2=0x7efda00d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:26.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 --2- 172.21.15.60:0/1950436439 >> v2:172.21.15.144:3300/0 conn(0x7efda015cbe0 0x7efda00d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 >> v1:172.21.15.144:6789/0 conn(0x7efda015cfc0 legacy=0x7efda00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:26.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00d1e40 con 0x7efda00b4c80 2023-10-16T12:52:26.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efd98001670 con 0x7efda00b4c80 2023-10-16T12:52:26.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 >> v1:172.21.15.60:6789/0 conn(0x7efda00b4c80 legacy=0x7efda00b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:26.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd8c0039c0 0x7efd8c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0047c0 0x7efd8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd8c005180 0x7efd8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda00d1b60 con 0x7efd8c0039c0 2023-10-16T12:52:26.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00c050 con 0x7efd8c0047c0 2023-10-16T12:52:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.637+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00c210 con 0x7efd8c005180 2023-10-16T12:52:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9effd640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd8c0039c0 0x7efd8c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9e7fc640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0047c0 0x7efd8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9effd640 1 -- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd8c005180 msgr2=0x7efd8c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9f7fe640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd8c005180 0x7efd8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9effd640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd8c005180 0x7efd8c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9effd640 1 -- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0047c0 msgr2=0x7efd8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9effd640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0047c0 0x7efd8c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9effd640 1 -- 172.21.15.60:0/1950436439 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c003520 con 0x7efd8c0039c0 2023-10-16T12:52:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9f7fe640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd8c005180 0x7efd8c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.638+0000 7efd9f7fe640 1 -- 172.21.15.60:0/1950436439 reap_dead start 2023-10-16T12:52:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.639+0000 7efd9effd640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd8c0039c0 0x7efd8c003dc0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7efd88009ac0 tx=0x7efd8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5861c8b19622891 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8802c6b0 con 0x7efd8c0039c0 2023-10-16T12:52:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8802c810 con 0x7efd8c0039c0 2023-10-16T12:52:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9dffb640 1 -- 172.21.15.60:0/1950436439 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8802cba0 con 0x7efd8c0039c0 2023-10-16T12:52:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9ffff640 1 -- 172.21.15.60:0/1950436439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd8c0039c0 msgr2=0x7efd8c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9ffff640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd8c0039c0 0x7efd8c003dc0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7efd88009ac0 tx=0x7efd8c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9ffff640 1 -- 172.21.15.60:0/1950436439 shutdown_connections 2023-10-16T12:52:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9ffff640 1 --2- 172.21.15.60:0/1950436439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd8c0039c0 0x7efd8c003dc0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.640+0000 7efd9ffff640 1 -- 172.21.15.60:0/1950436439 >> 172.21.15.60:0/1950436439 conn(0x7efda00cf570 msgr2=0x7efda00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.641+0000 7efd9ffff640 1 -- 172.21.15.60:0/1950436439 shutdown_connections 2023-10-16T12:52:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.641+0000 7efd9ffff640 1 -- 172.21.15.60:0/1950436439 wait complete. 2023-10-16T12:52:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.642+0000 7efd9ffff640 1 Processor -- start 2023-10-16T12:52:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.643+0000 7efd9ffff640 1 -- start start 2023-10-16T12:52:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.643+0000 7efd9ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0039c0 0x7efda00caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efda0152080 0x7efda00cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 0x7efda015d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda00cdf50 con 0x7efda00cdad0 2023-10-16T12:52:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda00cb420 con 0x7efd8c0039c0 2023-10-16T12:52:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda015f880 con 0x7efda0152080 2023-10-16T12:52:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 0x7efda015d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 0x7efda015d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49946/0 (socket says 172.21.15.60:49946) 2023-10-16T12:52:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 -- 172.21.15.60:0/715227510 learned_addr learned my addr 172.21.15.60:0/715227510 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9e7fc640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efda0152080 0x7efda00cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9effd640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0039c0 0x7efda00caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 -- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efda0152080 msgr2=0x7efda00cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efda0152080 0x7efda00cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 -- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0039c0 msgr2=0x7efda00caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0039c0 0x7efda00caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.644+0000 7efd9f7fe640 1 -- 172.21.15.60:0/715227510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda015ec70 con 0x7efda00cdad0 2023-10-16T12:52:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.645+0000 7efd9f7fe640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 0x7efda015d3d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7efda00b5670 tx=0x7efd9800d880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:26.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.645+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98007440 con 0x7efda00cdad0 2023-10-16T12:52:26.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.645+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd980075a0 con 0x7efda00cdad0 2023-10-16T12:52:26.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.645+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98004a20 con 0x7efda00cdad0 2023-10-16T12:52:26.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.645+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda015ef50 con 0x7efda00cdad0 2023-10-16T12:52:26.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.645+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda015f510 con 0x7efda00cdad0 2023-10-16T12:52:26.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.646+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda015f6a0 con 0x7efda00cdad0 2023-10-16T12:52:26.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.647+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd98004b80 con 0x7efda00cdad0 2023-10-16T12:52:26.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.648+0000 7efd97fff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7efd8006afc0 0x7efd8006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.648+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7efd980892c0 con 0x7efda00cdad0 2023-10-16T12:52:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.649+0000 7efd9effd640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7efd8006afc0 0x7efd8006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.650+0000 7efd9effd640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7efd8006afc0 0x7efd8006d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efd88036000 tx=0x7efd88030000 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.653+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd980540a0 con 0x7efda00cdad0 2023-10-16T12:52:26.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.888+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efda004e9b0 con 0x7efda00cdad0 2023-10-16T12:52:26.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.889+0000 7efd97fff640 1 -- 172.21.15.60:0/715227510 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+3696 (secure 0 0 0) 0x7efd9801ae80 con 0x7efda00cdad0 2023-10-16T12:52:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:26.891 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5822,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/63371843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":63371843},{"type":"v1","addr":"172.21.15.144:6837","nonce":63371843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/2616157791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2616157791},{"type":"v1","addr":"172.21.15.144:6835","nonce":2616157791}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7350,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1853865568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1853865568},{"type":"v1","addr":"172.21.15.60:6834","nonce":1853865568}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/258258281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":258258281},{"type":"v1","addr":"172.21.15.60:6835","nonce":258258281}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[]} 2023-10-16T12:52:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 55 2023-10-16T12:52:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7efd8006afc0 msgr2=0x7efd8006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7efd8006afc0 0x7efd8006d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efd88036000 tx=0x7efd88030000 comp rx=0 tx=0).stop 2023-10-16T12:52:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 msgr2=0x7efda015d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 0x7efda015d3d0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7efda00b5670 tx=0x7efd9800d880 comp rx=0 tx=0).stop 2023-10-16T12:52:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 shutdown_connections 2023-10-16T12:52:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7efd8006afc0 0x7efd8006d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efda0152080 0x7efda00cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efda00cdad0 0x7efda015d3d0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 --2- 172.21.15.60:0/715227510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd8c0039c0 0x7efda00caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 >> 172.21.15.60:0/715227510 conn(0x7efda00cf570 msgr2=0x7efda01e6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 shutdown_connections 2023-10-16T12:52:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:26.892+0000 7efd9ffff640 1 -- 172.21.15.60:0/715227510 wait complete. 2023-10-16T12:52:26.914 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:52:26.914 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:52:26.914 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:52:26.915 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:52:26.915 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:52:26.915 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:52:26.916 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:52:26.916 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:52:26.916 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:52:26.916 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:52:26.917 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:52:26.917 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:52:26.917 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:52:26.918 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:52:26.918 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:52:26.918 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:52:26.933 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:52:26.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:52:26.936 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:52:26.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:52:26.940 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:52:26.940 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:52:26.945 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:52:26.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:52:26.984 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:52:26.985 DEBUG:teuthology.parallel:result is None 2023-10-16T12:52:26.992 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:52:26.992 DEBUG:teuthology.parallel:result is None 2023-10-16T12:52:26.993 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:52:26.994 DEBUG:teuthology.parallel:result is None 2023-10-16T12:52:27.002 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:52:27.002 DEBUG:teuthology.parallel:result is None 2023-10-16T12:52:27.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:52:27.152 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:52:27.156 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:52:27.158 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:52:27.159 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:52:27.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.198+0000 7f0deb076640 1 Processor -- start 2023-10-16T12:52:27.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.198+0000 7f0deb076640 1 -- start start 2023-10-16T12:52:27.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0deb076640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:27.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0deb076640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0de40cd260 0x7f0de40cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:27.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0deb076640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de40b5d90 con 0x7f0de415d410 2023-10-16T12:52:27.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0deb076640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de40d2b50 con 0x7f0de40cd260 2023-10-16T12:52:27.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0deb076640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4160fc0 con 0x7f0de415d030 2023-10-16T12:52:27.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0de9613640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0de40cd260 0x7f0de40cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:27.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0de3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0de415d410 0x7f0de4161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50786/0 (socket says 172.21.15.60:50786) 2023-10-16T12:52:27.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0de3fff640 1 -- 172.21.15.60:0/3176065890 learned_addr learned my addr 172.21.15.60:0/3176065890 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:27.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.199+0000 7f0de8e12640 1 --2- 172.21.15.60:0/3176065890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:27.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.200+0000 7f0de9613640 1 -- 172.21.15.60:0/3176065890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 msgr2=0x7f0de40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:27.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.200+0000 7f0de9613640 1 --2- 172.21.15.60:0/3176065890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.200+0000 7f0de9613640 1 -- 172.21.15.60:0/3176065890 >> v1:172.21.15.60:6789/0 conn(0x7f0de415d410 legacy=0x7f0de4161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:27.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.200+0000 7f0de37fe640 1 -- 172.21.15.60:0/3176065890 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26814454 0 0) 0x7f0de40b5d90 con 0x7f0de415d410 2023-10-16T12:52:27.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.200+0000 7f0de9613640 1 -- 172.21.15.60:0/3176065890 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de40d25d0 con 0x7f0de40cd260 2023-10-16T12:52:27.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.201+0000 7f0de9613640 1 --2- 172.21.15.60:0/3176065890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de40cd6f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc009d90 tx=0x7f0ddc02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a28b37ee3edbc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:27.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0de37fe640 1 -- 172.21.15.60:0/3176065890 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc02cce0 con 0x7f0de40cd260 2023-10-16T12:52:27.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0de37fe640 1 -- 172.21.15.60:0/3176065890 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ddc02ce40 con 0x7f0de40cd260 2023-10-16T12:52:27.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0de37fe640 1 -- 172.21.15.60:0/3176065890 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc034660 con 0x7f0de40cd260 2023-10-16T12:52:27.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 -- 172.21.15.60:0/3176065890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 msgr2=0x7f0de40cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:27.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 --2- 172.21.15.60:0/3176065890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de40cd6f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc009d90 tx=0x7f0ddc02bf30 comp rx=0 tx=0).stop 2023-10-16T12:52:27.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 -- 172.21.15.60:0/3176065890 shutdown_connections 2023-10-16T12:52:27.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 --2- 172.21.15.60:0/3176065890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de40cd6f0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 --2- 172.21.15.60:0/3176065890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 -- 172.21.15.60:0/3176065890 >> 172.21.15.60:0/3176065890 conn(0x7f0de40cf720 msgr2=0x7f0de40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:27.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 -- 172.21.15.60:0/3176065890 shutdown_connections 2023-10-16T12:52:27.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.202+0000 7f0deb076640 1 -- 172.21.15.60:0/3176065890 wait complete. 2023-10-16T12:52:27.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.204+0000 7f0deb076640 1 Processor -- start 2023-10-16T12:52:27.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.204+0000 7f0deb076640 1 -- start start 2023-10-16T12:52:27.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0deb076640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de4159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:27.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0deb076640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0deb076640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de415d410 0x7f0de40cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0deb076640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4066500 con 0x7f0de40cd260 2023-10-16T12:52:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0deb076640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40c8940 con 0x7f0de415d410 2023-10-16T12:52:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0deb076640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de40c8c20 con 0x7f0de415d030 2023-10-16T12:52:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.205+0000 7f0de8e12640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de4159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:27.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de4159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49978/0 (socket says 172.21.15.60:49978) 2023-10-16T12:52:27.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 -- 172.21.15.60:0/346391410 learned_addr learned my addr 172.21.15.60:0/346391410 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:27.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 -- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 msgr2=0x7f0de40cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:27.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de3fff640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:27.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de9613640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de415d410 0x7f0de40cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:27.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 -- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de415d410 msgr2=0x7f0de40cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:27.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de415d410 0x7f0de40cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 -- 172.21.15.60:0/346391410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc009a70 con 0x7f0de40cd260 2023-10-16T12:52:27.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de3fff640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de9613640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de415d410 0x7f0de40cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.206+0000 7f0de8e12640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de4159d20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9b0 tx=0x7f0dd800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.207+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8009670 con 0x7f0de40cd260 2023-10-16T12:52:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.207+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd800c9a0 con 0x7f0de40cd260 2023-10-16T12:52:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.207+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de40c8f00 con 0x7f0de40cd260 2023-10-16T12:52:27.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.207+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800cd50 con 0x7f0de40cd260 2023-10-16T12:52:27.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.207+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de4066120 con 0x7f0de40cd260 2023-10-16T12:52:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.208+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de404e160 con 0x7f0de40cd260 2023-10-16T12:52:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.209+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0dd8013470 con 0x7f0de40cd260 2023-10-16T12:52:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.210+0000 7f0de17fa640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f0dc806af90 0x7f0dc806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:27.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.210+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5717+0+0 (secure 0 0 0) 0x7f0dd8089a90 con 0x7f0de40cd260 2023-10-16T12:52:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.210+0000 7f0de3fff640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f0dc806af90 0x7f0dc806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.211+0000 7f0de3fff640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f0dc806af90 0x7f0dc806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0de40c97b0 tx=0x7f0dd0009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:27.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.214+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0dd8054980 con 0x7f0de40cd260 2023-10-16T12:52:27.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.422+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0de40c9aa0 con 0x7f0de40cd260 2023-10-16T12:52:27.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:27.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:27.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.880+0000 7f0de17fa640 1 -- 172.21.15.60:0/346391410 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v122) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0dd8017080 con 0x7f0de40cd260 2023-10-16T12:52:27.882 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:52:27.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.883+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f0dc806af90 msgr2=0x7f0dc806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:27.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f0dc806af90 0x7f0dc806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0de40c97b0 tx=0x7f0dd0009040 comp rx=0 tx=0).stop 2023-10-16T12:52:27.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 msgr2=0x7f0de4159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:27.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de4159d20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9b0 tx=0x7f0dd800ee70 comp rx=0 tx=0).stop 2023-10-16T12:52:27.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 shutdown_connections 2023-10-16T12:52:27.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f0dc806af90 0x7f0dc806d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0de415d030 0x7f0de40cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0de415d410 0x7f0de40cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 --2- 172.21.15.60:0/346391410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0de40cd260 0x7f0de4159d20 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:27.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 >> 172.21.15.60:0/346391410 conn(0x7f0de40cf720 msgr2=0x7f0de40b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:27.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 shutdown_connections 2023-10-16T12:52:27.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:27.884+0000 7f0deb076640 1 -- 172.21.15.60:0/346391410 wait complete. 2023-10-16T12:52:27.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:52:28.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.122+0000 7f331ca75640 1 Processor -- start 2023-10-16T12:52:28.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.123+0000 7f331ca75640 1 -- start start 2023-10-16T12:52:28.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.124+0000 7f331ca75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 0x7f331806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.124+0000 7f331ca75640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331815f4a0 con 0x7f33180cda80 2023-10-16T12:52:28.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.124+0000 7f331ca75640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331806c960 con 0x7f33180ad970 2023-10-16T12:52:28.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.124+0000 7f331ca75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331815b9d0 con 0x7f331806c040 2023-10-16T12:52:28.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.125+0000 7f3317fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f33180cda80 0x7f331815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50788/0 (socket says 172.21.15.60:50788) 2023-10-16T12:52:28.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.125+0000 7f3317fff640 1 -- 172.21.15.60:0/1648854379 learned_addr learned my addr 172.21.15.60:0/1648854379 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:28.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.125+0000 7f33177fe640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 0x7f331806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.125+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592940554 0 0) 0x7f331815f4a0 con 0x7f33180cda80 2023-10-16T12:52:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.125+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003520 con 0x7f33180cda80 2023-10-16T12:52:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193971407 0 0) 0x7f3300003520 con 0x7f33180cda80 2023-10-16T12:52:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 msgr2=0x7f331806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 0x7f331806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 >> v1:172.21.15.144:6789/0 conn(0x7f33180ad970 legacy=0x7f331815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331815bcb0 con 0x7f33180cda80 2023-10-16T12:52:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f33100019f0 con 0x7f33180cda80 2023-10-16T12:52:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 >> v1:172.21.15.60:6789/0 conn(0x7f33180cda80 legacy=0x7f331815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33000039a0 0x7f3300003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 0x7f33000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3300004d00 0x7f3300005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331815b9d0 con 0x7f3300004d00 2023-10-16T12:52:28.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330000ba30 con 0x7f33000039a0 2023-10-16T12:52:28.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330000bbf0 con 0x7f331806c040 2023-10-16T12:52:28.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.126+0000 7f3317fff640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3300004d00 0x7f3300005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f33177fe640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33000039a0 0x7f3300003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f3316ffd640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 0x7f33000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f3317fff640 1 -- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 msgr2=0x7f33000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:28.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f3317fff640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f331806c040 0x7f33000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f3317fff640 1 -- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33000039a0 msgr2=0x7f3300003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:28.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f3317fff640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33000039a0 0x7f3300003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.127+0000 7f3317fff640 1 -- 172.21.15.60:0/1648854379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300003520 con 0x7f3300004d00 2023-10-16T12:52:28.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.128+0000 7f3317fff640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3300004d00 0x7f3300005180 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f331000a800 tx=0x7f33000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c5de481ff27971 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:28.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.128+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100072d0 con 0x7f3300004d00 2023-10-16T12:52:28.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.128+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331002f070 con 0x7f3300004d00 2023-10-16T12:52:28.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.128+0000 7f33167fc640 1 -- 172.21.15.60:0/1648854379 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100076b0 con 0x7f3300004d00 2023-10-16T12:52:28.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f331ca75640 1 -- 172.21.15.60:0/1648854379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3300004d00 msgr2=0x7f3300005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f331ca75640 1 --2- 172.21.15.60:0/1648854379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3300004d00 0x7f3300005180 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f331000a800 tx=0x7f33000057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f3317fff640 1 -- 172.21.15.60:0/1648854379 reap_dead start 2023-10-16T12:52:28.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f331ca75640 1 -- 172.21.15.60:0/1648854379 shutdown_connections 2023-10-16T12:52:28.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f331ca75640 1 -- 172.21.15.60:0/1648854379 >> 172.21.15.60:0/1648854379 conn(0x7f33180ce6a0 msgr2=0x7f33180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:28.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f331ca75640 1 -- 172.21.15.60:0/1648854379 shutdown_connections 2023-10-16T12:52:28.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.129+0000 7f331ca75640 1 -- 172.21.15.60:0/1648854379 wait complete. 2023-10-16T12:52:28.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.131+0000 7f331ca75640 1 Processor -- start 2023-10-16T12:52:28.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.131+0000 7f331ca75640 1 -- start start 2023-10-16T12:52:28.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f331ca75640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004d00 0x7f3318152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f331ca75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 0x7f3318146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f331ca75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33180c9ec0 0x7f3318146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f331ca75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318154e90 con 0x7f33180c9a60 2023-10-16T12:52:28.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f331ca75640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318155170 con 0x7f3300004d00 2023-10-16T12:52:28.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f331ca75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318155450 con 0x7f33180c9ec0 2023-10-16T12:52:28.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3316ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 0x7f3318146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3316ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 0x7f3318146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50006/0 (socket says 172.21.15.60:50006) 2023-10-16T12:52:28.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3316ffd640 1 -- 172.21.15.60:0/3261941379 learned_addr learned my addr 172.21.15.60:0/3261941379 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:28.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f33177fe640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004d00 0x7f3318152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3317fff640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33180c9ec0 0x7f3318146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3316ffd640 1 -- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33180c9ec0 msgr2=0x7f3318146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3316ffd640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33180c9ec0 0x7f3318146a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.132+0000 7f3316ffd640 1 -- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004d00 msgr2=0x7f3318152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f3316ffd640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004d00 0x7f3318152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f3316ffd640 1 -- 172.21.15.60:0/3261941379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318155730 con 0x7f33180c9a60 2023-10-16T12:52:28.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f33177fe640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004d00 0x7f3318152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:28.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f3316ffd640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 0x7f3318146520 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f330c00e550 tx=0x7f330c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:28.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c0052e0 con 0x7f33180c9a60 2023-10-16T12:52:28.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c005440 con 0x7f33180c9a60 2023-10-16T12:52:28.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c003810 con 0x7f33180c9a60 2023-10-16T12:52:28.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3318155a10 con 0x7f33180c9a60 2023-10-16T12:52:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.133+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331814efc0 con 0x7f33180c9a60 2023-10-16T12:52:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.135+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331806b330 con 0x7f33180c9a60 2023-10-16T12:52:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.135+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f330c003970 con 0x7f33180c9a60 2023-10-16T12:52:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.136+0000 7f32fffff640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f330406afc0 0x7f330406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.136+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f330c089220 con 0x7f33180c9a60 2023-10-16T12:52:28.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.137+0000 7f33177fe640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f330406afc0 0x7f330406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.138+0000 7f33177fe640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f330406afc0 0x7f330406d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3308009a00 tx=0x7f3308008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:28.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.141+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f330c054480 con 0x7f33180c9a60 2023-10-16T12:52:28.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.407+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f331804e9b0 con 0x7f33180c9a60 2023-10-16T12:52:28.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.408+0000 7f32fffff640 1 -- 172.21.15.60:0/3261941379 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f330c057aa0 con 0x7f33180c9a60 2023-10-16T12:52:28.410 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:28.410 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:52:28.410 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:52:28.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.411 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:52:28.411 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:52:28.411 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.411 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.411 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:52:28.412 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:52:28.412 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.412 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.413 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:52:28.413 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:52:28.413 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.413 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.413 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:52:28.414 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:52:28.414 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.415 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:52:28.415 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:52:28.415 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.415 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.416 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:52:28.416 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:52:28.416 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.416 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.417 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:52:28.417 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:52:28.417 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.417 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.417 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:52:28.418 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:52:28.418 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.418 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.419 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:52:28.419 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:52:28.419 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.419 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.419 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.420 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.420 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.420 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:52:28.421 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:52:28.421 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.421 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.421 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.421 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.422 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.422 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:52:28.422 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:52:28.423 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.423 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.423 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.423 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.423 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.424 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:52:28.424 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:52:28.424 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.425 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.425 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.425 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.425 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.426 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:52:28.426 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:52:28.426 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.426 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.427 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.428 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:52:28.428 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:52:28.428 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.428 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.428 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.428 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.429 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.429 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:52:28.429 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:52:28.430 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.430 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.430 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.430 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.431 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:52:28.431 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:52:28.431 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.432 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:52:28.432 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:52:28.432 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.432 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.433 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:52:28.433 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:52:28.433 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.433 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.434 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:52:28.434 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:52:28.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.435 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:52:28.435 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:52:28.435 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.435 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.435 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:52:28.436 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:52:28.436 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:52:28.436 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.437 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:52:28.437 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:52:28.437 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.437 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.437 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:52:28.438 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:52:28.438 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:52:28.438 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.439 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:52:28.439 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:52:28.439 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.439 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:52:28.439 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:52:28.440 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:52:28.440 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:52:28.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.441 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:52:28.441 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:52:28.441 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.441 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:52:28.441 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:52:28.441 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:52:28.442 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.442 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.442 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:52:28.443 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:52:28.443 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.443 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:52:28.443 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:52:28.443 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.444 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:52:28.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:28.444 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:52:28.444 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:52:28.445 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:52:28.445 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:52:28.445 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:52:28.445 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:52:28.445 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:28.446 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:52:28.446 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:52:28.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.410+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f330406afc0 msgr2=0x7f330406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.410+0000 7f331ca75640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f330406afc0 0x7f330406d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3308009a00 tx=0x7f3308008040 comp rx=0 tx=0).stop 2023-10-16T12:52:28.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.410+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 msgr2=0x7f3318146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.410+0000 7f331ca75640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 0x7f3318146520 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f330c00e550 tx=0x7f330c00ea10 comp rx=0 tx=0).stop 2023-10-16T12:52:28.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 shutdown_connections 2023-10-16T12:52:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f330406afc0 0x7f330406d480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33180c9ec0 0x7f3318146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33180c9a60 0x7f3318146520 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 --2- 172.21.15.60:0/3261941379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004d00 0x7f3318152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 >> 172.21.15.60:0/3261941379 conn(0x7f33180ce6a0 msgr2=0x7f33180ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:28.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 shutdown_connections 2023-10-16T12:52:28.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.411+0000 7f331ca75640 1 -- 172.21.15.60:0/3261941379 wait complete. 2023-10-16T12:52:28.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:28.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.674+0000 7f608d999640 1 Processor -- start 2023-10-16T12:52:28.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.674+0000 7f608d999640 1 -- start start 2023-10-16T12:52:28.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.675+0000 7f608d999640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6088070b60 0x7f608815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.675+0000 7f608d999640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608815aef0 con 0x7f6088159950 2023-10-16T12:52:28.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.675+0000 7f608d999640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608815d910 con 0x7f608806c100 2023-10-16T12:52:28.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.675+0000 7f608d999640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608815e490 con 0x7f6088070b60 2023-10-16T12:52:28.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.676+0000 7f6087fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6088070b60 0x7f608815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.676+0000 7f608c997640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6088159950 0x7f6088070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50794/0 (socket says 172.21.15.60:50794) 2023-10-16T12:52:28.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.676+0000 7f608c997640 1 -- 172.21.15.60:0/411832376 learned_addr learned my addr 172.21.15.60:0/411832376 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:28.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.676+0000 7f60877fe640 1 -- 172.21.15.60:0/411832376 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236318005 0 0) 0x7f607c001110 con 0x7f6088159950 2023-10-16T12:52:28.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.676+0000 7f60877fe640 1 -- 172.21.15.60:0/411832376 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078003520 con 0x7f6088159950 2023-10-16T12:52:28.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.677+0000 7f6087fff640 1 -- 172.21.15.60:0/411832376 >> v1:172.21.15.144:6789/0 conn(0x7f608806c100 legacy=0x7f608806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.677+0000 7f6087fff640 1 -- 172.21.15.60:0/411832376 >> v1:172.21.15.60:6789/0 conn(0x7f6088159950 legacy=0x7f6088070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.677+0000 7f6087fff640 1 -- 172.21.15.60:0/411832376 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608815dbf0 con 0x7f6088070b60 2023-10-16T12:52:28.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.678+0000 7f6087fff640 1 --2- 172.21.15.60:0/411832376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088070b60 0x7f608815a9b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f60740097a0 tx=0x7f607402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d8e9c244542ef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:28.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.678+0000 7f60877fe640 1 -- 172.21.15.60:0/411832376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074031830 con 0x7f6088070b60 2023-10-16T12:52:28.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f60877fe640 1 -- 172.21.15.60:0/411832376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6074031990 con 0x7f6088070b60 2023-10-16T12:52:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f60877fe640 1 -- 172.21.15.60:0/411832376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074031d20 con 0x7f6088070b60 2023-10-16T12:52:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 -- 172.21.15.60:0/411832376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088070b60 msgr2=0x7f608815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 --2- 172.21.15.60:0/411832376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088070b60 0x7f608815a9b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f60740097a0 tx=0x7f607402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 -- 172.21.15.60:0/411832376 shutdown_connections 2023-10-16T12:52:28.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 --2- 172.21.15.60:0/411832376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088070b60 0x7f608815a9b0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 -- 172.21.15.60:0/411832376 >> 172.21.15.60:0/411832376 conn(0x7f60880ce6a0 msgr2=0x7f60880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 -- 172.21.15.60:0/411832376 shutdown_connections 2023-10-16T12:52:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.679+0000 7f608d999640 1 -- 172.21.15.60:0/411832376 wait complete. 2023-10-16T12:52:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.681+0000 7f608d999640 1 Processor -- start 2023-10-16T12:52:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.681+0000 7f608d999640 1 -- start start 2023-10-16T12:52:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d999640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f608806c100 0x7f6088151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d999640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6088070b60 0x7f60880c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d999640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 0x7f60881e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d999640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6088152fc0 con 0x7f6088159950 2023-10-16T12:52:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d999640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60881e5280 con 0x7f6088070b60 2023-10-16T12:52:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d999640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60881e5560 con 0x7f608806c100 2023-10-16T12:52:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d198640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 0x7f60881e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d198640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 0x7f60881e5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50026/0 (socket says 172.21.15.60:50026) 2023-10-16T12:52:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.682+0000 7f608d198640 1 -- 172.21.15.60:0/2866798078 learned_addr learned my addr 172.21.15.60:0/2866798078 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f6087fff640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6088070b60 0x7f60880c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d198640 1 -- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f608806c100 msgr2=0x7f6088151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d198640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f608806c100 0x7f6088151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d198640 1 -- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6088070b60 msgr2=0x7f60880c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d198640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6088070b60 0x7f60880c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d198640 1 -- 172.21.15.60:0/2866798078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6074009400 con 0x7f6088159950 2023-10-16T12:52:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f6087fff640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6088070b60 0x7f60880c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d198640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 0x7f60881e5e10 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f608000c7a0 tx=0x7f608000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080003b70 con 0x7f6088159950 2023-10-16T12:52:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6080003cf0 con 0x7f6088159950 2023-10-16T12:52:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080003ea0 con 0x7f6088159950 2023-10-16T12:52:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.683+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60880c2e80 con 0x7f6088159950 2023-10-16T12:52:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.684+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60880c2b90 con 0x7f6088159950 2023-10-16T12:52:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.685+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60880c33a0 con 0x7f6088159950 2023-10-16T12:52:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.686+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f608000aa90 con 0x7f6088159950 2023-10-16T12:52:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.687+0000 7f60857fa640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f606c06af90 0x7f606c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.687+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f60800892b0 con 0x7f6088159950 2023-10-16T12:52:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.687+0000 7f608c997640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f606c06af90 0x7f606c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.688+0000 7f608c997640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f606c06af90 0x7f606c06d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f607c005880 tx=0x7f607c001150 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.691+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60880c33a0 con 0x7f6088159950 2023-10-16T12:52:28.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.888+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60880c33a0 con 0x7f6088159950 2023-10-16T12:52:28.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.890+0000 7f60857fa640 1 -- 172.21.15.60:0/2866798078 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+12855 (secure 0 0 0) 0x7f60880c33a0 con 0x7f6088159950 2023-10-16T12:52:28.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:28.892 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":124,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:27.898346+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:28.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f606c06af90 msgr2=0x7f606c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f606c06af90 0x7f606c06d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f607c005880 tx=0x7f607c001150 comp rx=0 tx=0).stop 2023-10-16T12:52:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 msgr2=0x7f60881e5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 0x7f60881e5e10 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f608000c7a0 tx=0x7f608000cc60 comp rx=0 tx=0).stop 2023-10-16T12:52:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 shutdown_connections 2023-10-16T12:52:28.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f606c06af90 0x7f606c06d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6088159950 0x7f60881e5e10 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6088070b60 0x7f60880c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 --2- 172.21.15.60:0/2866798078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f608806c100 0x7f6088151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 >> 172.21.15.60:0/2866798078 conn(0x7f60880ce6a0 msgr2=0x7f6088070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:28.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 shutdown_connections 2023-10-16T12:52:28.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:28.893+0000 7f608d999640 1 -- 172.21.15.60:0/2866798078 wait complete. 2023-10-16T12:52:28.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:52:28.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:52:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.144+0000 7f404df47640 1 Processor -- start 2023-10-16T12:52:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.144+0000 7f404df47640 1 -- start start 2023-10-16T12:52:29.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.146+0000 7f404df47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 0x7f40480cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.146+0000 7f404df47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40480b43c0 0x7f40480cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.146+0000 7f404df47640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404806be30 con 0x7f40480b0ea0 2023-10-16T12:52:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.146+0000 7f404df47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404806ca00 con 0x7f40480b43c0 2023-10-16T12:52:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.146+0000 7f404df47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40480ada10 con 0x7f40480b3ec0 2023-10-16T12:52:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.147+0000 7f404d746640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40480b43c0 0x7f40480cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.147+0000 7f404cf45640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 0x7f40480cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.147+0000 7f404d746640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40480b43c0 0x7f40480cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49712/0 (socket says 172.21.15.60:49712) 2023-10-16T12:52:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.147+0000 7f4047fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f40480b0ea0 0x7f404815a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:37862/0 (socket says 172.21.15.60:37862) 2023-10-16T12:52:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.147+0000 7f404d746640 1 -- 172.21.15.60:0/3412886124 learned_addr learned my addr 172.21.15.60:0/3412886124 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.147+0000 7f404cf45640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 0x7f40480cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:49690/0 (socket says 172.21.15.60:49690) 2023-10-16T12:52:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.148+0000 7f404d746640 1 -- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 msgr2=0x7f40480cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.148+0000 7f404d746640 1 --2- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 0x7f40480cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.148+0000 7f404d746640 1 -- 172.21.15.60:0/3412886124 >> v1:172.21.15.144:6789/0 conn(0x7f40480b0ea0 legacy=0x7f404815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.148+0000 7f404d746640 1 -- 172.21.15.60:0/3412886124 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404806c060 con 0x7f40480b43c0 2023-10-16T12:52:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.148+0000 7f40477fe640 1 -- 172.21.15.60:0/3412886124 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994778413 0 0) 0x7f404806be30 con 0x7f40480b0ea0 2023-10-16T12:52:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.148+0000 7f404cf45640 1 --2- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 0x7f40480cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:52:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.150+0000 7f404d746640 1 --2- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480cd7a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4040009e50 tx=0x7f404002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9d80ffd48b0a73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.150+0000 7f40477fe640 1 -- 172.21.15.60:0/3412886124 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404002ca80 con 0x7f40480b43c0 2023-10-16T12:52:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f40477fe640 1 -- 172.21.15.60:0/3412886124 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404002cbe0 con 0x7f40480b43c0 2023-10-16T12:52:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f40477fe640 1 -- 172.21.15.60:0/3412886124 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40400343f0 con 0x7f40480b43c0 2023-10-16T12:52:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 -- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 msgr2=0x7f40480cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 --2- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480cd7a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4040009e50 tx=0x7f404002bee0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 -- 172.21.15.60:0/3412886124 shutdown_connections 2023-10-16T12:52:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 --2- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480cd7a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 --2- 172.21.15.60:0/3412886124 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b3ec0 0x7f40480cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 -- 172.21.15.60:0/3412886124 >> 172.21.15.60:0/3412886124 conn(0x7f40480b0170 msgr2=0x7f40480b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 -- 172.21.15.60:0/3412886124 shutdown_connections 2023-10-16T12:52:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.151+0000 7f404df47640 1 -- 172.21.15.60:0/3412886124 wait complete. 2023-10-16T12:52:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.153+0000 7f404df47640 1 Processor -- start 2023-10-16T12:52:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.153+0000 7f404df47640 1 -- start start 2023-10-16T12:52:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.154+0000 7f404df47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b0ea0 0x7f404814fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.154+0000 7f404df47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 0x7f40480c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.154+0000 7f404df47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.154+0000 7f404df47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40481500e0 con 0x7f40480b3ec0 2023-10-16T12:52:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.154+0000 7f404df47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40480c7090 con 0x7f40480b43c0 2023-10-16T12:52:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.154+0000 7f404df47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40480c6480 con 0x7f40480b0ea0 2023-10-16T12:52:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 0x7f40480c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 0x7f40480c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37968/0 (socket says 172.21.15.60:37968) 2023-10-16T12:52:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 -- 172.21.15.60:0/2208299489 learned_addr learned my addr 172.21.15.60:0/2208299489 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f404cf45640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b0ea0 0x7f404814fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 -- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b0ea0 msgr2=0x7f404814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f404d746640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b0ea0 0x7f404814fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 -- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 msgr2=0x7f40480c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f4047fff640 1 -- 172.21.15.60:0/2208299489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c009400 con 0x7f40480b3ec0 2023-10-16T12:52:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f404cf45640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b0ea0 0x7f404814fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.155+0000 7f404d746640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.156+0000 7f4047fff640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 0x7f40480c7dd0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f403400c430 tx=0x7f403400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.156+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340074e0 con 0x7f40480b3ec0 2023-10-16T12:52:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.156+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034007660 con 0x7f40480b3ec0 2023-10-16T12:52:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.156+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4040009b30 con 0x7f40480b3ec0 2023-10-16T12:52:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.156+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034004a20 con 0x7f40480b3ec0 2023-10-16T12:52:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.156+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40480c6a40 con 0x7f40480b3ec0 2023-10-16T12:52:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.157+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404804e9b0 con 0x7f40480b3ec0 2023-10-16T12:52:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.158+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4034004b80 con 0x7f40480b3ec0 2023-10-16T12:52:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.159+0000 7f40457fa640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f402c06af90 0x7f402c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.159+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4034089260 con 0x7f40480b3ec0 2023-10-16T12:52:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.160+0000 7f404cf45640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f402c06af90 0x7f402c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.161+0000 7f404cf45640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f402c06af90 0x7f402c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f403c009c50 tx=0x7f403c008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.164+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4034054540 con 0x7f40480b3ec0 2023-10-16T12:52:29.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.361+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40480ac670 con 0x7f40480b3ec0 2023-10-16T12:52:29.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.362+0000 7f40457fa640 1 -- 172.21.15.60:0/2208299489 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4034087e60 con 0x7f40480b3ec0 2023-10-16T12:52:29.364 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:52:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f402c06af90 msgr2=0x7f402c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f402c06af90 0x7f402c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f403c009c50 tx=0x7f403c008040 comp rx=0 tx=0).stop 2023-10-16T12:52:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 msgr2=0x7f40480c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 0x7f40480c7dd0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f403400c430 tx=0x7f403400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 shutdown_connections 2023-10-16T12:52:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f402c06af90 0x7f402c06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40480b43c0 0x7f40480c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40480b3ec0 0x7f40480c7dd0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 --2- 172.21.15.60:0/2208299489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40480b0ea0 0x7f404814fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 >> 172.21.15.60:0/2208299489 conn(0x7f40480b0170 msgr2=0x7f40480adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:29.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 shutdown_connections 2023-10-16T12:52:29.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.365+0000 7f404df47640 1 -- 172.21.15.60:0/2208299489 wait complete. 2023-10-16T12:52:29.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:52:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.619+0000 7f1876dc0640 1 Processor -- start 2023-10-16T12:52:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.619+0000 7f1876dc0640 1 -- start start 2023-10-16T12:52:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1876dc0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 0x7f18700b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1876dc0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f187015d880 0x7f18700b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1876dc0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187006f7f0 con 0x7f1870159b20 2023-10-16T12:52:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1876dc0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187006c5e0 con 0x7f187015d880 2023-10-16T12:52:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1876dc0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187006c8c0 con 0x7f1870159f00 2023-10-16T12:52:29.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1874b5c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1870159b20 0x7f187006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46716/0 (socket says 172.21.15.60:46716) 2023-10-16T12:52:29.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.621+0000 7f1874b5c640 1 -- 172.21.15.60:0/2921623821 learned_addr learned my addr 172.21.15.60:0/2921623821 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273051760 0 0) 0x7f187006f7f0 con 0x7f1870159b20 2023-10-16T12:52:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f1867fff640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 0x7f18700b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1858003520 con 0x7f1870159b20 2023-10-16T12:52:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f187535d640 1 --2- 172.21.15.60:0/2921623821 >> v2:172.21.15.144:3300/0 conn(0x7f187015d880 0x7f18700b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865664861 0 0) 0x7f1858003520 con 0x7f1870159b20 2023-10-16T12:52:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 msgr2=0x7f18700b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 0x7f18700b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 >> v2:172.21.15.144:3300/0 conn(0x7f187015d880 msgr2=0x7f18700b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 --2- 172.21.15.60:0/2921623821 >> v2:172.21.15.144:3300/0 conn(0x7f187015d880 0x7f18700b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187006cba0 con 0x7f1870159b20 2023-10-16T12:52:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f186c001670 con 0x7f1870159b20 2023-10-16T12:52:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.622+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 >> v1:172.21.15.60:6789/0 conn(0x7f1870159b20 legacy=0x7f187006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f18677fe640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 0x7f1858003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f18677fe640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1858003fa0 0x7f18580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f18677fe640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858004d00 0x7f1858005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187006c5e0 con 0x7f1858004d00 2023-10-16T12:52:29.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187006c8c0 con 0x7f1858003fa0 2023-10-16T12:52:29.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185800ba80 con 0x7f1870159f00 2023-10-16T12:52:29.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f187535d640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858004d00 0x7f1858005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f1867fff640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1858003fa0 0x7f18580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f1874b5c640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 0x7f1858003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f187535d640 1 -- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 msgr2=0x7f1858003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:29.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f187535d640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1870159f00 0x7f1858003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f187535d640 1 -- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1858003fa0 msgr2=0x7f18580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f187535d640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1858003fa0 0x7f18580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.623+0000 7f187535d640 1 -- 172.21.15.60:0/2921623821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858003520 con 0x7f1858004d00 2023-10-16T12:52:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.624+0000 7f187535d640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858004d00 0x7f1858005180 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f1868007360 tx=0x7f18580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01747823055e993 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186802ccf0 con 0x7f1858004d00 2023-10-16T12:52:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186802ce50 con 0x7f1858004d00 2023-10-16T12:52:29.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f18677fe640 1 -- 172.21.15.60:0/2921623821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868034660 con 0x7f1858004d00 2023-10-16T12:52:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f1876dc0640 1 -- 172.21.15.60:0/2921623821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858004d00 msgr2=0x7f1858005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f1876dc0640 1 --2- 172.21.15.60:0/2921623821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1858004d00 0x7f1858005180 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f1868007360 tx=0x7f18580057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f187535d640 1 -- 172.21.15.60:0/2921623821 reap_dead start 2023-10-16T12:52:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f1876dc0640 1 -- 172.21.15.60:0/2921623821 shutdown_connections 2023-10-16T12:52:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f1876dc0640 1 -- 172.21.15.60:0/2921623821 >> 172.21.15.60:0/2921623821 conn(0x7f18700cb410 msgr2=0x7f18700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f1876dc0640 1 -- 172.21.15.60:0/2921623821 shutdown_connections 2023-10-16T12:52:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.625+0000 7f1876dc0640 1 -- 172.21.15.60:0/2921623821 wait complete. 2023-10-16T12:52:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.627+0000 7f1876dc0640 1 Processor -- start 2023-10-16T12:52:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.627+0000 7f1876dc0640 1 -- start start 2023-10-16T12:52:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.628+0000 7f1876dc0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 0x7f18700c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.628+0000 7f1876dc0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18700c44b0 0x7f18700b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.628+0000 7f1876dc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18700b0950 0x7f187014f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.628+0000 7f1876dc0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700b0dd0 con 0x7f187014cc30 2023-10-16T12:52:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.628+0000 7f1876dc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700c2e50 con 0x7f18700b0950 2023-10-16T12:52:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.628+0000 7f1876dc0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18700c2560 con 0x7f18700c44b0 2023-10-16T12:52:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 0x7f18700c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 0x7f18700c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37986/0 (socket says 172.21.15.60:37986) 2023-10-16T12:52:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 -- 172.21.15.60:0/141212417 learned_addr learned my addr 172.21.15.60:0/141212417 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 -- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18700c44b0 msgr2=0x7f18700b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f187535d640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18700b0950 0x7f187014f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1867fff640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18700c44b0 0x7f18700b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18700c44b0 0x7f18700b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 -- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18700b0950 msgr2=0x7f187014f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18700b0950 0x7f187014f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 -- 172.21.15.60:0/141212417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700c2800 con 0x7f187014cc30 2023-10-16T12:52:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.629+0000 7f1874b5c640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 0x7f18700c3f70 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f186c00eaf0 tx=0x7f186c005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.630+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c00c990 con 0x7f187014cc30 2023-10-16T12:52:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.630+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c00caf0 con 0x7f187014cc30 2023-10-16T12:52:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.630+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c00cea0 con 0x7f187014cc30 2023-10-16T12:52:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.630+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18700c2ae0 con 0x7f187014cc30 2023-10-16T12:52:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.630+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18700c3990 con 0x7f187014cc30 2023-10-16T12:52:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.631+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187006c270 con 0x7f187014cc30 2023-10-16T12:52:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.632+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f186c01d460 con 0x7f187014cc30 2023-10-16T12:52:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.633+0000 7f18657fa640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f185c06afc0 0x7f185c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.633+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f186c089540 con 0x7f187014cc30 2023-10-16T12:52:29.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.634+0000 7f1867fff640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f185c06afc0 0x7f185c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:29.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.635+0000 7f1867fff640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f185c06afc0 0x7f185c06d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18700b1080 tx=0x7f1860008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:29.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.638+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f186c0548c0 con 0x7f187014cc30 2023-10-16T12:52:29.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.835+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f187004e160 con 0x7f187014cc30 2023-10-16T12:52:29.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.836+0000 7f18657fa640 1 -- 172.21.15.60:0/141212417 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+65 (secure 0 0 0) 0x7f186c057ee0 con 0x7f187014cc30 2023-10-16T12:52:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:29.838 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:52:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:52:29.839 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:52:29.839 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:29.839 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:52:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f185c06afc0 msgr2=0x7f185c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f185c06afc0 0x7f185c06d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18700b1080 tx=0x7f1860008040 comp rx=0 tx=0).stop 2023-10-16T12:52:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 msgr2=0x7f18700c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 0x7f18700c3f70 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f186c00eaf0 tx=0x7f186c005e10 comp rx=0 tx=0).stop 2023-10-16T12:52:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 shutdown_connections 2023-10-16T12:52:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f185c06afc0 0x7f185c06d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18700c44b0 0x7f18700b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18700b0950 0x7f187014f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 --2- 172.21.15.60:0/141212417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f187014cc30 0x7f18700c3f70 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 >> 172.21.15.60:0/141212417 conn(0x7f18700cb410 msgr2=0x7f18700aed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 shutdown_connections 2023-10-16T12:52:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:29.839+0000 7f1876dc0640 1 -- 172.21.15.60:0/141212417 wait complete. 2023-10-16T12:52:29.860 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:52:29.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:52:30.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.090+0000 7fd74a327640 1 Processor -- start 2023-10-16T12:52:30.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.090+0000 7fd74a327640 1 -- start start 2023-10-16T12:52:30.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.092+0000 7fd74a327640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 0x7fd744167c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.092+0000 7fd74a327640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7440cc480 con 0x7fd7440c9880 2023-10-16T12:52:30.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.092+0000 7fd74a327640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7440c9cc0 con 0x7fd7440ac890 2023-10-16T12:52:30.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.092+0000 7fd74a327640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440adfe0 con 0x7fd7440cd920 2023-10-16T12:52:30.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.092+0000 7fd748b24640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd7440c9880 0x7fd74415d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46722/0 (socket says 172.21.15.60:46722) 2023-10-16T12:52:30.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.092+0000 7fd748b24640 1 -- 172.21.15.60:0/375020912 learned_addr learned my addr 172.21.15.60:0/375020912 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd749b26640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 0x7fd744167c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232508676 0 0) 0x7fd7440cc480 con 0x7fd7440c9880 2023-10-16T12:52:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd730003520 con 0x7fd7440c9880 2023-10-16T12:52:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577345875 0 0) 0x7fd730003520 con 0x7fd7440c9880 2023-10-16T12:52:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 msgr2=0x7fd744167c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 0x7fd744167c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 >> v1:172.21.15.144:6789/0 conn(0x7fd7440ac890 legacy=0x7fd7440acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74415d8c0 con 0x7fd7440c9880 2023-10-16T12:52:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd734001670 con 0x7fd7440c9880 2023-10-16T12:52:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.093+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 >> v1:172.21.15.60:6789/0 conn(0x7fd7440c9880 legacy=0x7fd74415d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd73bfff640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 0x7fd730003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd73bfff640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd730003f90 0x7fd7300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd73bfff640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd730005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440adfe0 con 0x7fd730004d00 2023-10-16T12:52:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73000bae0 con 0x7fd730003f90 2023-10-16T12:52:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73000bca0 con 0x7fd7440cd920 2023-10-16T12:52:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749b26640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd730005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749325640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 0x7fd730003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd748b24640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd730003f90 0x7fd7300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749b26640 1 -- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 msgr2=0x7fd730003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749b26640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7440cd920 0x7fd730003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749b26640 1 -- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd730003f90 msgr2=0x7fd7300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749b26640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd730003f90 0x7fd7300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.094+0000 7fd749b26640 1 -- 172.21.15.60:0/375020912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd730003520 con 0x7fd730004d00 2023-10-16T12:52:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd749b26640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd730005180 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009f90 tx=0x7fd7300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a862d2f6be87d8aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c02cbd0 con 0x7fd730004d00 2023-10-16T12:52:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c00f040 con 0x7fd730004d00 2023-10-16T12:52:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd73bfff640 1 -- 172.21.15.60:0/375020912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c034440 con 0x7fd730004d00 2023-10-16T12:52:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd74a327640 1 -- 172.21.15.60:0/375020912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 msgr2=0x7fd730005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd74a327640 1 --2- 172.21.15.60:0/375020912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd730005180 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009f90 tx=0x7fd7300057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd749b26640 1 -- 172.21.15.60:0/375020912 reap_dead start 2023-10-16T12:52:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd74a327640 1 -- 172.21.15.60:0/375020912 shutdown_connections 2023-10-16T12:52:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd74a327640 1 -- 172.21.15.60:0/375020912 >> 172.21.15.60:0/375020912 conn(0x7fd7440ce820 msgr2=0x7fd7440cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.096+0000 7fd74a327640 1 -- 172.21.15.60:0/375020912 shutdown_connections 2023-10-16T12:52:30.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.097+0000 7fd74a327640 1 -- 172.21.15.60:0/375020912 wait complete. 2023-10-16T12:52:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.098+0000 7fd74a327640 1 Processor -- start 2023-10-16T12:52:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.098+0000 7fd74a327640 1 -- start start 2023-10-16T12:52:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.099+0000 7fd74a327640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd7440c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.099+0000 7fd74a327640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7441e83e0 0x7fd7440707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.099+0000 7fd74a327640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd744158910 0x7fd744158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.099+0000 7fd74a327640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7441592d0 con 0x7fd730004d00 2023-10-16T12:52:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.099+0000 7fd74a327640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd744159fc0 con 0x7fd7441e83e0 2023-10-16T12:52:30.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.099+0000 7fd74a327640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74415a2a0 con 0x7fd744158910 2023-10-16T12:52:30.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd7440c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd7440c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38004/0 (socket says 172.21.15.60:38004) 2023-10-16T12:52:30.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 -- 172.21.15.60:0/3142113675 learned_addr learned my addr 172.21.15.60:0/3142113675 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:30.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd748b24640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7441e83e0 0x7fd7440707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749b26640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd744158910 0x7fd744158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 -- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd744158910 msgr2=0x7fd744158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd744158910 0x7fd744158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 -- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7441e83e0 msgr2=0x7fd7440707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7441e83e0 0x7fd7440707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.100+0000 7fd749325640 1 -- 172.21.15.60:0/3142113675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74415a580 con 0x7fd730004d00 2023-10-16T12:52:30.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd749b26640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd744158910 0x7fd744158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:30.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd748b24640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7441e83e0 0x7fd7440707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:30.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd749325640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd7440c7840 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e550 tx=0x7fd74000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:30.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7400052e0 con 0x7fd730004d00 2023-10-16T12:52:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd740005440 con 0x7fd730004d00 2023-10-16T12:52:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74415a860 con 0x7fd730004d00 2023-10-16T12:52:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740003810 con 0x7fd730004d00 2023-10-16T12:52:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.101+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7440c7d80 con 0x7fd730004d00 2023-10-16T12:52:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.102+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74404e9b0 con 0x7fd730004d00 2023-10-16T12:52:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.103+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd740003970 con 0x7fd730004d00 2023-10-16T12:52:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.104+0000 7fd739ffb640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd72806afc0 0x7fd72806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.104+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd740089160 con 0x7fd730004d00 2023-10-16T12:52:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.105+0000 7fd748b24640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd72806afc0 0x7fd72806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.106+0000 7fd748b24640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd72806afc0 0x7fd72806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd7440c8af0 tx=0x7fd734009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.109+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7400544e0 con 0x7fd730004d00 2023-10-16T12:52:30.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.308+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd7440c5210 con 0x7fd730004d00 2023-10-16T12:52:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.899+0000 7fd739ffb640 1 -- 172.21.15.60:0/3142113675 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v125) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd740057b00 con 0x7fd730004d00 2023-10-16T12:52:30.901 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:52:30.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd72806afc0 msgr2=0x7fd72806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd72806afc0 0x7fd72806d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd7440c8af0 tx=0x7fd734009040 comp rx=0 tx=0).stop 2023-10-16T12:52:30.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 msgr2=0x7fd7440c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:30.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd7440c7840 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e550 tx=0x7fd74000ea10 comp rx=0 tx=0).stop 2023-10-16T12:52:30.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 shutdown_connections 2023-10-16T12:52:30.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fd72806afc0 0x7fd72806d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd744158910 0x7fd744158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd7441e83e0 0x7fd7440707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 --2- 172.21.15.60:0/3142113675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd730004d00 0x7fd7440c7840 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:30.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.904+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 >> 172.21.15.60:0/3142113675 conn(0x7fd7440ce820 msgr2=0x7fd7440cb110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:30.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.905+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 shutdown_connections 2023-10-16T12:52:30.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:30.905+0000 7fd74a327640 1 -- 172.21.15.60:0/3142113675 wait complete. 2023-10-16T12:52:30.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:52:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.152+0000 7f6200e2e640 1 Processor -- start 2023-10-16T12:52:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.153+0000 7f6200e2e640 1 -- start start 2023-10-16T12:52:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.154+0000 7f6200e2e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61fc0717f0 0x7f61fc0abd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.154+0000 7f6200e2e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0b4d80 con 0x7f61fc06d5a0 2023-10-16T12:52:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f6200e2e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc15e0a0 con 0x7f61fc0b4530 2023-10-16T12:52:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f6200e2e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61fc159fb0 con 0x7f61fc0717f0 2023-10-16T12:52:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f61fb7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61fc06d5a0 0x7f61fc0ae180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46726/0 (socket says 172.21.15.60:46726) 2023-10-16T12:52:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f61fb7fe640 1 -- 172.21.15.60:0/828126983 learned_addr learned my addr 172.21.15.60:0/828126983 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f61fbfff640 1 --2- 172.21.15.60:0/828126983 >> v2:172.21.15.60:3300/0 conn(0x7f61fc0717f0 0x7f61fc0abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:31.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f61faffd640 1 -- 172.21.15.60:0/828126983 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368689785 0 0) 0x7f61fc0b4d80 con 0x7f61fc06d5a0 2023-10-16T12:52:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.155+0000 7f61faffd640 1 -- 172.21.15.60:0/828126983 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e8005730 con 0x7f61fc06d5a0 2023-10-16T12:52:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.156+0000 7f61fbfff640 1 -- 172.21.15.60:0/828126983 >> v1:172.21.15.144:6789/0 conn(0x7f61fc0b4530 legacy=0x7f61fc0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.156+0000 7f61fbfff640 1 -- 172.21.15.60:0/828126983 >> v1:172.21.15.60:6789/0 conn(0x7f61fc06d5a0 legacy=0x7f61fc0ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.156+0000 7f61fbfff640 1 -- 172.21.15.60:0/828126983 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc15e380 con 0x7f61fc0717f0 2023-10-16T12:52:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.157+0000 7f61fbfff640 1 --2- 172.21.15.60:0/828126983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc0abd40 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f61f40097a0 tx=0x7f61f402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce6b0fda742ebc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.157+0000 7f61faffd640 1 -- 172.21.15.60:0/828126983 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4030610 con 0x7f61fc0717f0 2023-10-16T12:52:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.157+0000 7f61faffd640 1 -- 172.21.15.60:0/828126983 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4030770 con 0x7f61fc0717f0 2023-10-16T12:52:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.157+0000 7f61faffd640 1 -- 172.21.15.60:0/828126983 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4030b00 con 0x7f61fc0717f0 2023-10-16T12:52:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 -- 172.21.15.60:0/828126983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 msgr2=0x7f61fc0abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 --2- 172.21.15.60:0/828126983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc0abd40 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f61f40097a0 tx=0x7f61f402b960 comp rx=0 tx=0).stop 2023-10-16T12:52:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 -- 172.21.15.60:0/828126983 shutdown_connections 2023-10-16T12:52:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 --2- 172.21.15.60:0/828126983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc0abd40 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 -- 172.21.15.60:0/828126983 >> 172.21.15.60:0/828126983 conn(0x7f61fc0b0a80 msgr2=0x7f61fc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 -- 172.21.15.60:0/828126983 shutdown_connections 2023-10-16T12:52:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.158+0000 7f6200e2e640 1 -- 172.21.15.60:0/828126983 wait complete. 2023-10-16T12:52:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.160+0000 7f6200e2e640 1 Processor -- start 2023-10-16T12:52:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.160+0000 7f6200e2e640 1 -- start start 2023-10-16T12:52:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f6200e2e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61fc06d5a0 0x7f61fc06b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f6200e2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f6200e2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61fc0b4530 0x7f61fc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:31.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f6200e2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc06b390 con 0x7f61fc0717f0 2023-10-16T12:52:31.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f6200e2e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc06bd60 con 0x7f61fc06d5a0 2023-10-16T12:52:31.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f6200e2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc06a780 con 0x7f61fc0b4530 2023-10-16T12:52:31.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.161+0000 7f61f2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:31.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc1514a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38030/0 (socket says 172.21.15.60:38030) 2023-10-16T12:52:31.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2641434454 learned_addr learned my addr 172.21.15.60:0/2641434454 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:31.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61fb7fe640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61fc06d5a0 0x7f61fc06b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:31.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61fbfff640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61fc0b4530 0x7f61fc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:31.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61fc0b4530 msgr2=0x7f61fc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61fc0b4530 0x7f61fc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61fc06d5a0 msgr2=0x7f61fc06b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61fc06d5a0 0x7f61fc06b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2641434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f4009400 con 0x7f61fc0717f0 2023-10-16T12:52:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.162+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc1514a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f61e400c430 tx=0x7f61e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:31.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.163+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e40052d0 con 0x7f61fc0717f0 2023-10-16T12:52:31.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.163+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e4005430 con 0x7f61fc0717f0 2023-10-16T12:52:31.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.163+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc06aa60 con 0x7f61fc0717f0 2023-10-16T12:52:31.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.163+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4003830 con 0x7f61fc0717f0 2023-10-16T12:52:31.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.163+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc06b020 con 0x7f61fc0717f0 2023-10-16T12:52:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.164+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc06b540 con 0x7f61fc0717f0 2023-10-16T12:52:31.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.166+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f61e4003990 con 0x7f61fc0717f0 2023-10-16T12:52:31.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.167+0000 7f61f8ff9640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61dc06afc0 0x7f61dc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:31.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.167+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f61e4088e30 con 0x7f61fc0717f0 2023-10-16T12:52:31.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.168+0000 7f61fb7fe640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61dc06afc0 0x7f61dc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:31.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.169+0000 7f61fb7fe640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61dc06afc0 0x7f61dc06d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f61ec001560 tx=0x7f61ec009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:31.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.171+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61e4053f90 con 0x7f61fc0717f0 2023-10-16T12:52:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.365+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f61fc06b540 con 0x7f61fc0717f0 2023-10-16T12:52:31.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.907+0000 7f61f8ff9640 1 -- 172.21.15.60:0/2641434454 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v126) v1 ==== 170+0+0 (secure 0 0 0) 0x7f61fc06b540 con 0x7f61fc0717f0 2023-10-16T12:52:31.909 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:52:31.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.911+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61dc06afc0 msgr2=0x7f61dc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.911+0000 7f6200e2e640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61dc06afc0 0x7f61dc06d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f61ec001560 tx=0x7f61ec009040 comp rx=0 tx=0).stop 2023-10-16T12:52:31.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.911+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 msgr2=0x7f61fc1514a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:31.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.911+0000 7f6200e2e640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc1514a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f61e400c430 tx=0x7f61e400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 shutdown_connections 2023-10-16T12:52:31.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61dc06afc0 0x7f61dc06d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61fc0b4530 0x7f61fc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61fc0717f0 0x7f61fc1514a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 --2- 172.21.15.60:0/2641434454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61fc06d5a0 0x7f61fc06b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:31.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 >> 172.21.15.60:0/2641434454 conn(0x7f61fc0b0a80 msgr2=0x7f61fc0708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:31.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 shutdown_connections 2023-10-16T12:52:31.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:31.913+0000 7f6200e2e640 1 -- 172.21.15.60:0/2641434454 wait complete. 2023-10-16T12:52:31.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:52:32.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.140+0000 7f6432987640 1 Processor -- start 2023-10-16T12:52:32.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.140+0000 7f6432987640 1 -- start start 2023-10-16T12:52:32.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6432987640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f642c159950 0x7f642c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:32.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6432987640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c070450 0x7f642c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:32.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6432987640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c070df0 con 0x7f642c06c100 2023-10-16T12:52:32.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6432987640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c15e490 con 0x7f642c159950 2023-10-16T12:52:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6432987640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c15aef0 con 0x7f642c070450 2023-10-16T12:52:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6431985640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f642c159950 0x7f642c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6431985640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f642c159950 0x7f642c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38034/0 (socket says 172.21.15.60:38034) 2023-10-16T12:52:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6432186640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f642c06c100 0x7f642c06c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46736/0 (socket says 172.21.15.60:46736) 2023-10-16T12:52:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.142+0000 7f6431985640 1 -- 172.21.15.60:0/142318485 learned_addr learned my addr 172.21.15.60:0/142318485 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:32.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.143+0000 7f6431184640 1 --2- 172.21.15.60:0/142318485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c070450 0x7f642c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.143+0000 7f6431985640 1 -- 172.21.15.60:0/142318485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c070450 msgr2=0x7f642c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.143+0000 7f6431985640 1 --2- 172.21.15.60:0/142318485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c070450 0x7f642c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.143+0000 7f6431985640 1 -- 172.21.15.60:0/142318485 >> v1:172.21.15.60:6789/0 conn(0x7f642c06c100 legacy=0x7f642c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:32.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.143+0000 7f6430983640 1 -- 172.21.15.60:0/142318485 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646658527 0 0) 0x7f642c070df0 con 0x7f642c06c100 2023-10-16T12:52:32.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.143+0000 7f6431985640 1 -- 172.21.15.60:0/142318485 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c06cc80 con 0x7f642c159950 2023-10-16T12:52:32.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.144+0000 7f6431985640 1 --2- 172.21.15.60:0/142318485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15a9b0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f6428009870 tx=0x7f642802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9209dab3434ed5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:32.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.144+0000 7f6430983640 1 -- 172.21.15.60:0/142318485 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428031640 con 0x7f642c159950 2023-10-16T12:52:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.144+0000 7f6430983640 1 -- 172.21.15.60:0/142318485 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64280317a0 con 0x7f642c159950 2023-10-16T12:52:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.144+0000 7f6430983640 1 -- 172.21.15.60:0/142318485 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428031b30 con 0x7f642c159950 2023-10-16T12:52:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 -- 172.21.15.60:0/142318485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 msgr2=0x7f642c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 --2- 172.21.15.60:0/142318485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15a9b0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f6428009870 tx=0x7f642802b8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 -- 172.21.15.60:0/142318485 shutdown_connections 2023-10-16T12:52:32.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 --2- 172.21.15.60:0/142318485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c070450 0x7f642c0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 --2- 172.21.15.60:0/142318485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15a9b0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 -- 172.21.15.60:0/142318485 >> 172.21.15.60:0/142318485 conn(0x7f642c0ce6a0 msgr2=0x7f642c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:32.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 -- 172.21.15.60:0/142318485 shutdown_connections 2023-10-16T12:52:32.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.145+0000 7f6432987640 1 -- 172.21.15.60:0/142318485 wait complete. 2023-10-16T12:52:32.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.147+0000 7f6432987640 1 Processor -- start 2023-10-16T12:52:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.147+0000 7f6432987640 1 -- start start 2023-10-16T12:52:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432987640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c06c100 0x7f642c155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432987640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f642c070450 0x7f642c157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432987640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432987640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0c21b0 con 0x7f642c159950 2023-10-16T12:52:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432987640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c156080 con 0x7f642c070450 2023-10-16T12:52:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432987640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c157b80 con 0x7f642c06c100 2023-10-16T12:52:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.148+0000 7f6432186640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38038/0 (socket says 172.21.15.60:38038) 2023-10-16T12:52:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 -- 172.21.15.60:0/4106603148 learned_addr learned my addr 172.21.15.60:0/4106603148 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6431184640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f642c070450 0x7f642c157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6431985640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c06c100 0x7f642c155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 -- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c06c100 msgr2=0x7f642c155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c06c100 0x7f642c155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 -- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f642c070450 msgr2=0x7f642c157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f642c070450 0x7f642c157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 -- 172.21.15.60:0/4106603148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64280094d0 con 0x7f642c159950 2023-10-16T12:52:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6431184640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f642c070450 0x7f642c157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.149+0000 7f6432186640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15c210 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f642400c700 tx=0x7f642400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.150+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642400ce10 con 0x7f642c159950 2023-10-16T12:52:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.150+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642400a640 con 0x7f642c159950 2023-10-16T12:52:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.150+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c157e60 con 0x7f642c159950 2023-10-16T12:52:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.150+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642400aa30 con 0x7f642c159950 2023-10-16T12:52:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.150+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c158420 con 0x7f642c159950 2023-10-16T12:52:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.151+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c0c2340 con 0x7f642c159950 2023-10-16T12:52:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.153+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f642400ab90 con 0x7f642c159950 2023-10-16T12:52:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.154+0000 7f64227fc640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f641006afc0 0x7f641006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.154+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6424089010 con 0x7f642c159950 2023-10-16T12:52:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.154+0000 7f6431985640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f641006afc0 0x7f641006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.155+0000 7f6431985640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f641006afc0 0x7f641006d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6428036000 tx=0x7f6428008030 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.157+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6424053ff0 con 0x7f642c159950 2023-10-16T12:52:32.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.394+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f642c0c22f0 con 0x7f642c159950 2023-10-16T12:52:32.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.954+0000 7f64227fc640 1 -- 172.21.15.60:0/4106603148 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v58) v1 ==== 174+0+0 (secure 0 0 0) 0x7f64240180c0 con 0x7f642c159950 2023-10-16T12:52:32.957 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 14 and data pool 15 2023-10-16T12:52:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.970+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f641006afc0 msgr2=0x7f641006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.970+0000 7f6432987640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f641006afc0 0x7f641006d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6428036000 tx=0x7f6428008030 comp rx=0 tx=0).stop 2023-10-16T12:52:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.970+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 msgr2=0x7f642c15c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.970+0000 7f6432987640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15c210 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f642400c700 tx=0x7f642400cbc0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.971+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 shutdown_connections 2023-10-16T12:52:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.971+0000 7f6432987640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f641006afc0 0x7f641006d480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.971+0000 7f6432987640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f642c159950 0x7f642c15c210 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.971+0000 7f6432987640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f642c070450 0x7f642c157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.971+0000 7f6432987640 1 --2- 172.21.15.60:0/4106603148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f642c06c100 0x7f642c155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.971+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 >> 172.21.15.60:0/4106603148 conn(0x7f642c0ce6a0 msgr2=0x7f642c06b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.973+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 shutdown_connections 2023-10-16T12:52:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:32.973+0000 7f6432987640 1 -- 172.21.15.60:0/4106603148 wait complete. 2023-10-16T12:52:33.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.184+0000 7f3301044640 1 Processor -- start 2023-10-16T12:52:33.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.184+0000 7f3301044640 1 -- start start 2023-10-16T12:52:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f3301044640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32fc0cd0c0 0x7f32fc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f3301044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0b0410 0x7f32fc0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f3301044640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32fc0b0db0 con 0x7f32fc15d3f0 2023-10-16T12:52:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f3301044640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32fc15e3e0 con 0x7f32fc0cd0c0 2023-10-16T12:52:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f3301044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0cd4a0 con 0x7f32fc0b0410 2023-10-16T12:52:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f32fb59e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f32fc15d3f0 0x7f32fc15d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46752/0 (socket says 172.21.15.60:46752) 2023-10-16T12:52:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f32fad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f32fc0cd0c0 0x7f32fc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.186+0000 7f32fb59e640 1 -- 172.21.15.60:0/3773725961 learned_addr learned my addr 172.21.15.60:0/3773725961 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32fa59c640 1 -- 172.21.15.60:0/3773725961 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249032565 0 0) 0x7f32fc0b0db0 con 0x7f32fc15d3f0 2023-10-16T12:52:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32f3fff640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0b0410 0x7f32fc0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32fa59c640 1 -- 172.21.15.60:0/3773725961 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e8003520 con 0x7f32fc15d3f0 2023-10-16T12:52:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32fad9d640 1 -- 172.21.15.60:0/3773725961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0b0410 msgr2=0x7f32fc0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32fad9d640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0b0410 0x7f32fc0b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32fad9d640 1 -- 172.21.15.60:0/3773725961 >> v1:172.21.15.60:6789/0 conn(0x7f32fc15d3f0 legacy=0x7f32fc15d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32fad9d640 1 -- 172.21.15.60:0/3773725961 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc15e100 con 0x7f32fc0cd0c0 2023-10-16T12:52:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.187+0000 7f32f3fff640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0b0410 0x7f32fc0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.188+0000 7f32fad9d640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc0cd0c0 0x7f32fc1615f0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0097a0 tx=0x7f32ec02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f29cc3d459ae0ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.188+0000 7f32fa59c640 1 -- 172.21.15.60:0/3773725961 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec031780 con 0x7f32fc0cd0c0 2023-10-16T12:52:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.188+0000 7f32fa59c640 1 -- 172.21.15.60:0/3773725961 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec0318e0 con 0x7f32fc0cd0c0 2023-10-16T12:52:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.188+0000 7f32fa59c640 1 -- 172.21.15.60:0/3773725961 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec031c70 con 0x7f32fc0cd0c0 2023-10-16T12:52:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 -- 172.21.15.60:0/3773725961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc0cd0c0 msgr2=0x7f32fc1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc0cd0c0 0x7f32fc1615f0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0097a0 tx=0x7f32ec02b820 comp rx=0 tx=0).stop 2023-10-16T12:52:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 -- 172.21.15.60:0/3773725961 shutdown_connections 2023-10-16T12:52:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0b0410 0x7f32fc0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 --2- 172.21.15.60:0/3773725961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc0cd0c0 0x7f32fc1615f0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 -- 172.21.15.60:0/3773725961 >> 172.21.15.60:0/3773725961 conn(0x7f32fc0cb410 msgr2=0x7f32fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:33.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 -- 172.21.15.60:0/3773725961 shutdown_connections 2023-10-16T12:52:33.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.189+0000 7f3301044640 1 -- 172.21.15.60:0/3773725961 wait complete. 2023-10-16T12:52:33.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.191+0000 7f3301044640 1 Processor -- start 2023-10-16T12:52:33.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.191+0000 7f3301044640 1 -- start start 2023-10-16T12:52:33.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f3301044640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32fc0b0410 0x7f32fc1f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f3301044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0cd0c0 0x7f32fc0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f3301044640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 0x7f32fc15f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f3301044640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc14c3e0 con 0x7f32fc15d3f0 2023-10-16T12:52:33.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f3301044640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc15f2d0 con 0x7f32fc0b0410 2023-10-16T12:52:33.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f3301044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0b3db0 con 0x7f32fc0cd0c0 2023-10-16T12:52:33.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 0x7f32fc15f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 0x7f32fc15f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38058/0 (socket says 172.21.15.60:38058) 2023-10-16T12:52:33.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 -- 172.21.15.60:0/3371340010 learned_addr learned my addr 172.21.15.60:0/3371340010 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:33.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 -- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0cd0c0 msgr2=0x7f32fc0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0cd0c0 0x7f32fc0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 -- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32fc0b0410 msgr2=0x7f32fc1f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32fc0b0410 0x7f32fc1f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.192+0000 7f32fb59e640 1 -- 172.21.15.60:0/3371340010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec009400 con 0x7f32fc15d3f0 2023-10-16T12:52:33.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.193+0000 7f32fb59e640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 0x7f32fc15f760 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f32f400cb40 tx=0x7f32f400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:33.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.193+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f400aca0 con 0x7f32fc15d3f0 2023-10-16T12:52:33.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.193+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f400ae00 con 0x7f32fc15d3f0 2023-10-16T12:52:33.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.193+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4012620 con 0x7f32fc15d3f0 2023-10-16T12:52:33.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.193+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc0b4090 con 0x7f32fc15d3f0 2023-10-16T12:52:33.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.193+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc15ef10 con 0x7f32fc15d3f0 2023-10-16T12:52:33.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.194+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc04e160 con 0x7f32fc15d3f0 2023-10-16T12:52:33.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.195+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f32f4007070 con 0x7f32fc15d3f0 2023-10-16T12:52:33.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.196+0000 7f32f37fe640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f32dc06af90 0x7f32dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.196+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32f40897f0 con 0x7f32fc15d3f0 2023-10-16T12:52:33.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.197+0000 7f32fad9d640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f32dc06af90 0x7f32dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.198+0000 7f32fad9d640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f32dc06af90 0x7f32dc06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32ec036000 tx=0x7f32ec030000 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:33.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.201+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32f4054790 con 0x7f32fc15d3f0 2023-10-16T12:52:33.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.396+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32fc15f440 con 0x7f32fc15d3f0 2023-10-16T12:52:33.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.398+0000 7f32f37fe640 1 -- 172.21.15.60:0/3371340010 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16520 (secure 0 0 0) 0x7f32f4017020 con 0x7f32fc15d3f0 2023-10-16T12:52:33.400 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:33.401 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":127,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:32.395134+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:33.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f32dc06af90 msgr2=0x7f32dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f32dc06af90 0x7f32dc06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32ec036000 tx=0x7f32ec030000 comp rx=0 tx=0).stop 2023-10-16T12:52:33.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 msgr2=0x7f32fc15f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 0x7f32fc15f760 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f32f400cb40 tx=0x7f32f400a060 comp rx=0 tx=0).stop 2023-10-16T12:52:33.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 shutdown_connections 2023-10-16T12:52:33.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f32dc06af90 0x7f32dc06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32fc0cd0c0 0x7f32fc0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32fc15d3f0 0x7f32fc15f760 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 --2- 172.21.15.60:0/3371340010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32fc0b0410 0x7f32fc1f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.401+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 >> 172.21.15.60:0/3371340010 conn(0x7f32fc0cb410 msgr2=0x7f32fc15c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.402+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 shutdown_connections 2023-10-16T12:52:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.402+0000 7f3301044640 1 -- 172.21.15.60:0/3371340010 wait complete. 2023-10-16T12:52:33.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:33.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.639+0000 7f78e01a5640 1 Processor -- start 2023-10-16T12:52:33.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.639+0000 7f78e01a5640 1 -- start start 2023-10-16T12:52:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.640+0000 7f78e01a5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78d80ce390 0x7f78d815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.640+0000 7f78e01a5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d815ed60 con 0x7f78d815aaa0 2023-10-16T12:52:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.640+0000 7f78e01a5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80707e0 con 0x7f78d815ae80 2023-10-16T12:52:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.640+0000 7f78e01a5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d806b810 con 0x7f78d80ce390 2023-10-16T12:52:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.641+0000 7f78de742640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78d80ce390 0x7f78d815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.641+0000 7f78de742640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78d80ce390 0x7f78d815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38062/0 (socket says 172.21.15.60:38062) 2023-10-16T12:52:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.641+0000 7f78de742640 1 -- 172.21.15.60:0/2613861219 learned_addr learned my addr 172.21.15.60:0/2613861219 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:33.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.641+0000 7f78de742640 1 -- 172.21.15.60:0/2613861219 >> v1:172.21.15.144:6789/0 conn(0x7f78d815ae80 legacy=0x7f78d8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:33.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.641+0000 7f78de742640 1 -- 172.21.15.60:0/2613861219 >> v1:172.21.15.60:6789/0 conn(0x7f78d815aaa0 legacy=0x7f78d80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.641+0000 7f78de742640 1 -- 172.21.15.60:0/2613861219 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d815ef90 con 0x7f78d80ce390 2023-10-16T12:52:33.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.642+0000 7f78de742640 1 --2- 172.21.15.60:0/2613861219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d80ce390 0x7f78d815e820 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009e50 tx=0x7f78d002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4a9451637542bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:33.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78dcf3f640 1 -- 172.21.15.60:0/2613861219 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d002cb10 con 0x7f78d80ce390 2023-10-16T12:52:33.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78dcf3f640 1 -- 172.21.15.60:0/2613861219 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d002cc70 con 0x7f78d80ce390 2023-10-16T12:52:33.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78dcf3f640 1 -- 172.21.15.60:0/2613861219 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d00343f0 con 0x7f78d80ce390 2023-10-16T12:52:33.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 -- 172.21.15.60:0/2613861219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d80ce390 msgr2=0x7f78d815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 --2- 172.21.15.60:0/2613861219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d80ce390 0x7f78d815e820 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009e50 tx=0x7f78d002bf70 comp rx=0 tx=0).stop 2023-10-16T12:52:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 -- 172.21.15.60:0/2613861219 shutdown_connections 2023-10-16T12:52:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 --2- 172.21.15.60:0/2613861219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d80ce390 0x7f78d815e820 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 -- 172.21.15.60:0/2613861219 >> 172.21.15.60:0/2613861219 conn(0x7f78d80cb410 msgr2=0x7f78d80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 -- 172.21.15.60:0/2613861219 shutdown_connections 2023-10-16T12:52:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.643+0000 7f78e01a5640 1 -- 172.21.15.60:0/2613861219 wait complete. 2023-10-16T12:52:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.645+0000 7f78e01a5640 1 Processor -- start 2023-10-16T12:52:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.645+0000 7f78e01a5640 1 -- start start 2023-10-16T12:52:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78e01a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78d80ce390 0x7f78d80c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78e01a5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78d815aaa0 0x7f78d80ae230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78e01a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 0x7f78d80ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78e01a5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8148150 con 0x7f78d815ae80 2023-10-16T12:52:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78e01a5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8147540 con 0x7f78d815aaa0 2023-10-16T12:52:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78e01a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8147820 con 0x7f78d80ce390 2023-10-16T12:52:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.646+0000 7f78de742640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 0x7f78d80ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 0x7f78d80ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38070/0 (socket says 172.21.15.60:38070) 2023-10-16T12:52:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 -- 172.21.15.60:0/3443524771 learned_addr learned my addr 172.21.15.60:0/3443524771 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 -- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78d80ce390 msgr2=0x7f78d80c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78d80ce390 0x7f78d80c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 -- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78d815aaa0 msgr2=0x7f78d80ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78d815aaa0 0x7f78d80ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 -- 172.21.15.60:0/3443524771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009b30 con 0x7f78d815ae80 2023-10-16T12:52:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78de742640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 0x7f78d80ae770 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f78d0036040 tx=0x7f78d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.647+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d003e450 con 0x7f78d815ae80 2023-10-16T12:52:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.648+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d003e5d0 con 0x7f78d815ae80 2023-10-16T12:52:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.648+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d003e980 con 0x7f78d815ae80 2023-10-16T12:52:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.648+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d8147b00 con 0x7f78d815ae80 2023-10-16T12:52:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.648+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d80c5110 con 0x7f78d815ae80 2023-10-16T12:52:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.649+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d804e160 con 0x7f78d815ae80 2023-10-16T12:52:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.650+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f78d003eae0 con 0x7f78d815ae80 2023-10-16T12:52:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.650+0000 7f78ce7fc640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f78bc06afc0 0x7f78bc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.651+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78d00ac350 con 0x7f78d815ae80 2023-10-16T12:52:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.651+0000 7f78ddf41640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f78bc06afc0 0x7f78bc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.652+0000 7f78ddf41640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f78bc06afc0 0x7f78bc06d480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78d4000f60 tx=0x7f78d40079b0 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.655+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78d00773a0 con 0x7f78d815ae80 2023-10-16T12:52:33.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.854+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78d80cdc60 con 0x7f78d815ae80 2023-10-16T12:52:33.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.856+0000 7f78ce7fc640 1 -- 172.21.15.60:0/3443524771 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16520 (secure 0 0 0) 0x7f78d007a9c0 con 0x7f78d815ae80 2023-10-16T12:52:33.858 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:33.858 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":127,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:32.395134+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f78bc06afc0 msgr2=0x7f78bc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f78bc06afc0 0x7f78bc06d480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78d4000f60 tx=0x7f78d40079b0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 msgr2=0x7f78d80ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:33.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 0x7f78d80ae770 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f78d0036040 tx=0x7f78d0004a40 comp rx=0 tx=0).stop 2023-10-16T12:52:33.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 shutdown_connections 2023-10-16T12:52:33.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f78bc06afc0 0x7f78bc06d480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78d815ae80 0x7f78d80ae770 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78d815aaa0 0x7f78d80ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 --2- 172.21.15.60:0/3443524771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78d80ce390 0x7f78d80c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:33.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 >> 172.21.15.60:0/3443524771 conn(0x7f78d80cb410 msgr2=0x7f78d80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:33.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.859+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 shutdown_connections 2023-10-16T12:52:33.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:33.860+0000 7f78e01a5640 1 -- 172.21.15.60:0/3443524771 wait complete. 2023-10-16T12:52:33.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:52:34.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.100+0000 7f09c8e59640 1 Processor -- start 2023-10-16T12:52:34.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.100+0000 7f09c8e59640 1 -- start start 2023-10-16T12:52:34.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.101+0000 7f09c8e59640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09c406fb50 0x7f09c40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:34.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c8e59640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09c406c9e0 0x7f09c40ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:34.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c8e59640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40ca560 con 0x7f09c40c9060 2023-10-16T12:52:34.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c8e59640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c40c9950 con 0x7f09c406fb50 2023-10-16T12:52:34.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c8e59640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c40c9c30 con 0x7f09c406c9e0 2023-10-16T12:52:34.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c2ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09c406fb50 0x7f09c40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:34.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c2ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09c406fb50 0x7f09c40cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38082/0 (socket says 172.21.15.60:38082) 2023-10-16T12:52:34.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c2ffd640 1 -- 172.21.15.60:0/1882113754 learned_addr learned my addr 172.21.15.60:0/1882113754 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:34.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c37fe640 1 --1- 172.21.15.60:0/1882113754 >> v1:172.21.15.60:6789/0 conn(0x7f09c40c9060 0x7f09c406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46762/0 (socket says 172.21.15.60:46762) 2023-10-16T12:52:34.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c2ffd640 1 -- 172.21.15.60:0/1882113754 >> v2:172.21.15.144:3300/0 conn(0x7f09c406c9e0 msgr2=0x7f09c40ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:34.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c3fff640 1 --2- 172.21.15.60:0/1882113754 >> v2:172.21.15.144:3300/0 conn(0x7f09c406c9e0 0x7f09c40ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:34.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c2ffd640 1 --2- 172.21.15.60:0/1882113754 >> v2:172.21.15.144:3300/0 conn(0x7f09c406c9e0 0x7f09c40ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.102+0000 7f09c2ffd640 1 -- 172.21.15.60:0/1882113754 >> v1:172.21.15.60:6789/0 conn(0x7f09c40c9060 legacy=0x7f09c406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:34.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.103+0000 7f09c2ffd640 1 -- 172.21.15.60:0/1882113754 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40c9f10 con 0x7f09c406fb50 2023-10-16T12:52:34.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.103+0000 7f09c27fc640 1 -- 172.21.15.60:0/1882113754 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668840271 0 0) 0x7f09c40ca560 con 0x7f09c40c9060 2023-10-16T12:52:34.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.103+0000 7f09c3fff640 1 --2- 172.21.15.60:0/1882113754 >> v2:172.21.15.144:3300/0 conn(0x7f09c406c9e0 0x7f09c40ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.104+0000 7f09c2ffd640 1 --2- 172.21.15.60:0/1882113754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40cc8a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f09c40ce550 tx=0x7f09ac02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21647b0d1203de4f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.104+0000 7f09c27fc640 1 -- 172.21.15.60:0/1882113754 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac030880 con 0x7f09c406fb50 2023-10-16T12:52:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.104+0000 7f09c27fc640 1 -- 172.21.15.60:0/1882113754 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac0309e0 con 0x7f09c406fb50 2023-10-16T12:52:34.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.104+0000 7f09c27fc640 1 -- 172.21.15.60:0/1882113754 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac030d70 con 0x7f09c406fb50 2023-10-16T12:52:34.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 -- 172.21.15.60:0/1882113754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 msgr2=0x7f09c40cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 --2- 172.21.15.60:0/1882113754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40cc8a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f09c40ce550 tx=0x7f09ac02b9c0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 -- 172.21.15.60:0/1882113754 shutdown_connections 2023-10-16T12:52:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 --2- 172.21.15.60:0/1882113754 >> v2:172.21.15.144:3300/0 conn(0x7f09c406c9e0 0x7f09c40ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 --2- 172.21.15.60:0/1882113754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40cc8a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 -- 172.21.15.60:0/1882113754 >> 172.21.15.60:0/1882113754 conn(0x7f09c40ce6a0 msgr2=0x7f09c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 -- 172.21.15.60:0/1882113754 shutdown_connections 2023-10-16T12:52:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.105+0000 7f09c8e59640 1 -- 172.21.15.60:0/1882113754 wait complete. 2023-10-16T12:52:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.107+0000 7f09c8e59640 1 Processor -- start 2023-10-16T12:52:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.107+0000 7f09c8e59640 1 -- start start 2023-10-16T12:52:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c8e59640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c406c9e0 0x7f09c40c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c8e59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c8e59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09c40c9060 0x7f09c4152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c8e59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c40c7ad0 con 0x7f09c406fb50 2023-10-16T12:52:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c8e59640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c40b3af0 con 0x7f09c406c9e0 2023-10-16T12:52:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c8e59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c41530c0 con 0x7f09c40c9060 2023-10-16T12:52:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38096/0 (socket says 172.21.15.60:38096) 2023-10-16T12:52:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c2ffd640 1 -- 172.21.15.60:0/498339788 learned_addr learned my addr 172.21.15.60:0/498339788 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c37fe640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c406c9e0 0x7f09c40c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c3fff640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09c40c9060 0x7f09c4152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c2ffd640 1 -- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09c40c9060 msgr2=0x7f09c4152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.108+0000 7f09c2ffd640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09c40c9060 0x7f09c4152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c2ffd640 1 -- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c406c9e0 msgr2=0x7f09c40c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:34.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c2ffd640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c406c9e0 0x7f09c40c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c2ffd640 1 -- 172.21.15.60:0/498339788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0094d0 con 0x7f09c406fb50 2023-10-16T12:52:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c37fe640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c406c9e0 0x7f09c40c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c2ffd640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40b35b0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f09ac000c00 tx=0x7f09ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac030850 con 0x7f09c406fb50 2023-10-16T12:52:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac043400 con 0x7f09c406fb50 2023-10-16T12:52:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac0437b0 con 0x7f09c406fb50 2023-10-16T12:52:34.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c40c6e50 con 0x7f09c406fb50 2023-10-16T12:52:34.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.109+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c40c7410 con 0x7f09c406fb50 2023-10-16T12:52:34.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.110+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c404e9b0 con 0x7f09c406fb50 2023-10-16T12:52:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.111+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f09ac043910 con 0x7f09c406fb50 2023-10-16T12:52:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.112+0000 7f09bbfff640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f09a406af90 0x7f09a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.113+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09ac03e070 con 0x7f09c406fb50 2023-10-16T12:52:34.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.113+0000 7f09c37fe640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f09a406af90 0x7f09a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:34.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.114+0000 7f09c37fe640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f09a406af90 0x7f09a406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f09b4007fd0 tx=0x7f09b4009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:34.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.117+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09ac077db0 con 0x7f09c406fb50 2023-10-16T12:52:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.352+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f09c41601e0 con 0x7f09c406fb50 2023-10-16T12:52:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.974+0000 7f09bbfff640 1 -- 172.21.15.60:0/498339788 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7f09ac0340b0 con 0x7f09c406fb50 2023-10-16T12:52:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f09a406af90 msgr2=0x7f09a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f09a406af90 0x7f09a406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f09b4007fd0 tx=0x7f09b4009040 comp rx=0 tx=0).stop 2023-10-16T12:52:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 msgr2=0x7f09c40b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40b35b0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f09ac000c00 tx=0x7f09ac004730 comp rx=0 tx=0).stop 2023-10-16T12:52:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 shutdown_connections 2023-10-16T12:52:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f09a406af90 0x7f09a406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09c40c9060 0x7f09c4152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09c406fb50 0x7f09c40b35b0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 --2- 172.21.15.60:0/498339788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c406c9e0 0x7f09c40c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 >> 172.21.15.60:0/498339788 conn(0x7f09c40ce6a0 msgr2=0x7f09c406b780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 shutdown_connections 2023-10-16T12:52:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:34.978+0000 7f09c8e59640 1 -- 172.21.15.60:0/498339788 wait complete. 2023-10-16T12:52:35.003 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:52:35.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:52:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.229+0000 7fc3b4fff640 1 Processor -- start 2023-10-16T12:52:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.229+0000 7fc3b4fff640 1 -- start start 2023-10-16T12:52:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3b4fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3b015c9f0 0x7fc3b00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3b4fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00c9230 0x7fc3b00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3b4fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3b0070f20 0x7fc3b00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3b4fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b00cdec0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3b4fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0160a90 con 0x7fc3b0070f20 2023-10-16T12:52:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3b4fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00cd900 con 0x7fc3b00c9230 2023-10-16T12:52:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3af7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3b015c9f0 0x7fc3b00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:35.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3af7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3b015c9f0 0x7fc3b00c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38104/0 (socket says 172.21.15.60:38104) 2023-10-16T12:52:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3af7fe640 1 -- 172.21.15.60:0/84639118 learned_addr learned my addr 172.21.15.60:0/84639118 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3a7fff640 1 --2- 172.21.15.60:0/84639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00c9230 0x7fc3b00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.231+0000 7fc3affff640 1 --2- 172.21.15.60:0/84639118 >> v2:172.21.15.144:3300/0 conn(0x7fc3b0070f20 0x7fc3b00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.232+0000 7fc3af7fe640 1 -- 172.21.15.60:0/84639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00c9230 msgr2=0x7fc3b00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:35.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.232+0000 7fc3af7fe640 1 --2- 172.21.15.60:0/84639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00c9230 0x7fc3b00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.232+0000 7fc3af7fe640 1 -- 172.21.15.60:0/84639118 >> v2:172.21.15.144:3300/0 conn(0x7fc3b0070f20 msgr2=0x7fc3b00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.232+0000 7fc3af7fe640 1 --2- 172.21.15.60:0/84639118 >> v2:172.21.15.144:3300/0 conn(0x7fc3b0070f20 0x7fc3b00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.232+0000 7fc3af7fe640 1 -- 172.21.15.60:0/84639118 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00cdbe0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.233+0000 7fc3af7fe640 1 --2- 172.21.15.60:0/84639118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b00c8cf0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00c8c80 tx=0x7fc3a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d16898590e3c2a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.233+0000 7fc3aeffd640 1 -- 172.21.15.60:0/84639118 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031870 con 0x7fc3b015c9f0 2023-10-16T12:52:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.233+0000 7fc3aeffd640 1 -- 172.21.15.60:0/84639118 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a00319d0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.233+0000 7fc3aeffd640 1 -- 172.21.15.60:0/84639118 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031d60 con 0x7fc3b015c9f0 2023-10-16T12:52:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 -- 172.21.15.60:0/84639118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 msgr2=0x7fc3b00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/84639118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b00c8cf0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00c8c80 tx=0x7fc3a002b910 comp rx=0 tx=0).stop 2023-10-16T12:52:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 -- 172.21.15.60:0/84639118 shutdown_connections 2023-10-16T12:52:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/84639118 >> v2:172.21.15.144:3300/0 conn(0x7fc3b0070f20 0x7fc3b00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/84639118 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b00c9230 0x7fc3b00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/84639118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b00c8cf0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 -- 172.21.15.60:0/84639118 >> 172.21.15.60:0/84639118 conn(0x7fc3b00ce6a0 msgr2=0x7fc3b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 -- 172.21.15.60:0/84639118 shutdown_connections 2023-10-16T12:52:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.234+0000 7fc3b4fff640 1 -- 172.21.15.60:0/84639118 wait complete. 2023-10-16T12:52:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.236+0000 7fc3b4fff640 1 Processor -- start 2023-10-16T12:52:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.236+0000 7fc3b4fff640 1 -- start start 2023-10-16T12:52:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3b4fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b0070f20 0x7fc3b00c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3b4fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00c9230 0x7fc3b015fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3b4fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b014e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3b4fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b015ff50 con 0x7fc3b015c9f0 2023-10-16T12:52:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3b4fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00c2860 con 0x7fc3b00c9230 2023-10-16T12:52:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3b4fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00c4a90 con 0x7fc3b0070f20 2023-10-16T12:52:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b014e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b014e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38106/0 (socket says 172.21.15.60:38106) 2023-10-16T12:52:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.237+0000 7fc3affff640 1 -- 172.21.15.60:0/2633342252 learned_addr learned my addr 172.21.15.60:0/2633342252 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3affff640 1 -- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b0070f20 msgr2=0x7fc3b00c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3affff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b0070f20 0x7fc3b00c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3affff640 1 -- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00c9230 msgr2=0x7fc3b015fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3affff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00c9230 0x7fc3b015fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3affff640 1 -- 172.21.15.60:0/2633342252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a00094d0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3affff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b014e980 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8007e40 tx=0x7fc3a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8014620 con 0x7fc3b015c9f0 2023-10-16T12:52:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a800ee90 con 0x7fc3b015c9f0 2023-10-16T12:52:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a80149e0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b00c4d70 con 0x7fc3b015c9f0 2023-10-16T12:52:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.238+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b014eec0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.240+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b00c90a0 con 0x7fc3b015c9f0 2023-10-16T12:52:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.241+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc3a8014b40 con 0x7fc3b015c9f0 2023-10-16T12:52:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.242+0000 7fc3acff9640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc39006af90 0x7fc39006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.242+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3a808a120 con 0x7fc3b015c9f0 2023-10-16T12:52:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.242+0000 7fc3af7fe640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc39006af90 0x7fc39006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.244+0000 7fc3af7fe640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc39006af90 0x7fc39006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036000 tx=0x7fc3a0030000 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.246+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3a8055150 con 0x7fc3b015c9f0 2023-10-16T12:52:35.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.482+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc3b015d310 con 0x7fc3b015c9f0 2023-10-16T12:52:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.977+0000 7fc3acff9640 1 -- 172.21.15.60:0/2633342252 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v62) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc3a8058770 con 0x7fc3b015c9f0 2023-10-16T12:52:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc39006af90 msgr2=0x7fc39006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc39006af90 0x7fc39006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036000 tx=0x7fc3a0030000 comp rx=0 tx=0).stop 2023-10-16T12:52:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 msgr2=0x7fc3b014e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b014e980 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8007e40 tx=0x7fc3a800e4b0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 shutdown_connections 2023-10-16T12:52:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc39006af90 0x7fc39006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3b015c9f0 0x7fc3b014e980 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3b00c9230 0x7fc3b015fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 --2- 172.21.15.60:0/2633342252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3b0070f20 0x7fc3b00c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 >> 172.21.15.60:0/2633342252 conn(0x7fc3b00ce6a0 msgr2=0x7fc3b00cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.981+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 shutdown_connections 2023-10-16T12:52:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:35.982+0000 7fc3b4fff640 1 -- 172.21.15.60:0/2633342252 wait complete. 2023-10-16T12:52:36.003 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:52:36.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:36.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.226+0000 7febfcb8c640 1 Processor -- start 2023-10-16T12:52:36.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.226+0000 7febfcb8c640 1 -- start start 2023-10-16T12:52:36.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.228+0000 7febfcb8c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.228+0000 7febfcb8c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.228+0000 7febfcb8c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf815b9d0 con 0x7febf806c040 2023-10-16T12:52:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.228+0000 7febfcb8c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf815bcb0 con 0x7febf80ccfb0 2023-10-16T12:52:36.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.228+0000 7febfcb8c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf815bf90 con 0x7febf80ad970 2023-10-16T12:52:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.228+0000 7febf7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf6ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49858/0 (socket says 172.21.15.60:49858) 2023-10-16T12:52:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf7fff640 1 -- 172.21.15.60:0/1461799187 learned_addr learned my addr 172.21.15.60:0/1461799187 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf77fe640 1 --1- 172.21.15.60:0/1461799187 >> v1:172.21.15.144:6789/0 conn(0x7febf806c040 0x7febf806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:37904/0 (socket says 172.21.15.60:37904) 2023-10-16T12:52:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf7fff640 1 -- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf80ad970 msgr2=0x7febf80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf7fff640 1 --2- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf7fff640 1 -- 172.21.15.60:0/1461799187 >> v1:172.21.15.144:6789/0 conn(0x7febf806c040 legacy=0x7febf806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.229+0000 7febf7fff640 1 -- 172.21.15.60:0/1461799187 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80cc270 con 0x7febf80ccfb0 2023-10-16T12:52:36.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.230+0000 7febf6ffd640 1 --2- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:52:36.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.231+0000 7febf7fff640 1 --2- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80cd440 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7febf0009e80 tx=0x7febf002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7358ccba48705bf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:36.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.231+0000 7febf67fc640 1 -- 172.21.15.60:0/1461799187 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf002caf0 con 0x7febf80ccfb0 2023-10-16T12:52:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.231+0000 7febf67fc640 1 -- 172.21.15.60:0/1461799187 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf002cc50 con 0x7febf80ccfb0 2023-10-16T12:52:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.231+0000 7febf67fc640 1 -- 172.21.15.60:0/1461799187 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf00343f0 con 0x7febf80ccfb0 2023-10-16T12:52:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 -- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 msgr2=0x7febf80cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 --2- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80cd440 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7febf0009e80 tx=0x7febf002bf50 comp rx=0 tx=0).stop 2023-10-16T12:52:36.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 -- 172.21.15.60:0/1461799187 shutdown_connections 2023-10-16T12:52:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 --2- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 --2- 172.21.15.60:0/1461799187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 -- 172.21.15.60:0/1461799187 >> 172.21.15.60:0/1461799187 conn(0x7febf80ce6a0 msgr2=0x7febf80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 -- 172.21.15.60:0/1461799187 shutdown_connections 2023-10-16T12:52:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.232+0000 7febfcb8c640 1 -- 172.21.15.60:0/1461799187 wait complete. 2023-10-16T12:52:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.234+0000 7febfcb8c640 1 Processor -- start 2023-10-16T12:52:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.234+0000 7febfcb8c640 1 -- start start 2023-10-16T12:52:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febfcb8c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf806c040 0x7febf8156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febfcb8c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 0x7febf80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febfcb8c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febfcb8c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf81571e0 con 0x7febf80ad970 2023-10-16T12:52:36.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febfcb8c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80ca350 con 0x7febf80ccfb0 2023-10-16T12:52:36.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febfcb8c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80cad20 con 0x7febf806c040 2023-10-16T12:52:36.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.235+0000 7febf6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 0x7febf80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 0x7febf80ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38122/0 (socket says 172.21.15.60:38122) 2023-10-16T12:52:36.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 -- 172.21.15.60:0/357259639 learned_addr learned my addr 172.21.15.60:0/357259639 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf7fff640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf77fe640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf806c040 0x7febf8156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 -- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf806c040 msgr2=0x7febf8156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf806c040 0x7febf8156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 -- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 msgr2=0x7febf80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 -- 172.21.15.60:0/357259639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec009400 con 0x7febf80ad970 2023-10-16T12:52:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf7fff640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf77fe640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf806c040 0x7febf8156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.236+0000 7febf6ffd640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 0x7febf80ca7e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7febec00ebe0 tx=0x7febec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.237+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec009760 con 0x7febf80ad970 2023-10-16T12:52:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.237+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec00cc30 con 0x7febf80ad970 2023-10-16T12:52:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.237+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf0009b30 con 0x7febf80ad970 2023-10-16T12:52:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.237+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00eee0 con 0x7febf80ad970 2023-10-16T12:52:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.237+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf80c9a20 con 0x7febf80ad970 2023-10-16T12:52:36.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.238+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf806b330 con 0x7febf80ad970 2023-10-16T12:52:36.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.239+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7febec013470 con 0x7febf80ad970 2023-10-16T12:52:36.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.240+0000 7febdffff640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7febe406afc0 0x7febe406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.240+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febec089e00 con 0x7febf80ad970 2023-10-16T12:52:36.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.241+0000 7febf77fe640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7febe406afc0 0x7febe406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.242+0000 7febf77fe640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7febe406afc0 0x7febe406d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febe8000f60 tx=0x7febe8009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:36.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.245+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febec054d90 con 0x7febf80ad970 2023-10-16T12:52:36.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.482+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febf80ca4c0 con 0x7febf80ad970 2023-10-16T12:52:36.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.484+0000 7febdffff640 1 -- 172.21.15.60:0/357259639 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4921 (secure 0 0 0) 0x7febec0583b0 con 0x7febf80ad970 2023-10-16T12:52:36.486 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:36.486 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:52:32.396002+0000","modified":"2023-10-16T12:52:35.483028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7866":{"gid":7866,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2146889084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2146889084},{"type":"v1","addr":"172.21.15.60:6833","nonce":2146889084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:52:36.486 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2023-10-16T12:52:36.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7febe406afc0 msgr2=0x7febe406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7febe406afc0 0x7febe406d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febe8000f60 tx=0x7febe8009040 comp rx=0 tx=0).stop 2023-10-16T12:52:36.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 msgr2=0x7febf80ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 0x7febf80ca7e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7febec00ebe0 tx=0x7febec00c520 comp rx=0 tx=0).stop 2023-10-16T12:52:36.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 shutdown_connections 2023-10-16T12:52:36.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7febe406afc0 0x7febe406d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febf80ccfb0 0x7febf80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febf80ad970 0x7febf80ca7e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 --2- 172.21.15.60:0/357259639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7febf806c040 0x7febf8156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 >> 172.21.15.60:0/357259639 conn(0x7febf80ce6a0 msgr2=0x7febf815ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:36.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 shutdown_connections 2023-10-16T12:52:36.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.487+0000 7febfcb8c640 1 -- 172.21.15.60:0/357259639 wait complete. 2023-10-16T12:52:36.508 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:52:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.728+0000 7fc1ac86e640 1 Processor -- start 2023-10-16T12:52:36.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.728+0000 7fc1ac86e640 1 -- start start 2023-10-16T12:52:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.730+0000 7fc1ac86e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 0x7fc1a80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.730+0000 7fc1ac86e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1a80ccfb0 0x7fc1a80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.730+0000 7fc1ac86e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a815b9d0 con 0x7fc1a806c040 2023-10-16T12:52:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.730+0000 7fc1ac86e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a815bcb0 con 0x7fc1a80ccfb0 2023-10-16T12:52:36.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.730+0000 7fc1ac86e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a815bf90 con 0x7fc1a80ad970 2023-10-16T12:52:36.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.731+0000 7fc1a6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc1a806c040 0x7fc1a806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46772/0 (socket says 172.21.15.60:46772) 2023-10-16T12:52:36.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.731+0000 7fc1a6d9d640 1 -- 172.21.15.60:0/702274843 learned_addr learned my addr 172.21.15.60:0/702274843 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:36.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.731+0000 7fc1a759e640 1 --2- 172.21.15.60:0/702274843 >> v2:172.21.15.144:3300/0 conn(0x7fc1a80ccfb0 0x7fc1a80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.731+0000 7fc1a659c640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 0x7fc1a80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.731+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148730498 0 0) 0x7fc1a815b9d0 con 0x7fc1a806c040 2023-10-16T12:52:36.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.731+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc190003520 con 0x7fc1a806c040 2023-10-16T12:52:36.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877433975 0 0) 0x7fc190003520 con 0x7fc1a806c040 2023-10-16T12:52:36.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 msgr2=0x7fc1a80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 0x7fc1a80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 >> v2:172.21.15.144:3300/0 conn(0x7fc1a80ccfb0 msgr2=0x7fc1a80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 --2- 172.21.15.60:0/702274843 >> v2:172.21.15.144:3300/0 conn(0x7fc1a80ccfb0 0x7fc1a80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a80cc270 con 0x7fc1a806c040 2023-10-16T12:52:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc198001670 con 0x7fc1a806c040 2023-10-16T12:52:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 >> v1:172.21.15.60:6789/0 conn(0x7fc1a806c040 legacy=0x7fc1a806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 0x7fc190003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190003f40 0x7fc1900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc190004d00 0x7fc190005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a815bcb0 con 0x7fc190004d00 2023-10-16T12:52:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a815bf90 con 0x7fc190003f40 2023-10-16T12:52:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19000b980 con 0x7fc1a80ad970 2023-10-16T12:52:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.732+0000 7fc1a759e640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc190004d00 0x7fc190005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a6d9d640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 0x7fc190003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a659c640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190003f40 0x7fc1900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a759e640 1 -- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 msgr2=0x7fc190003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a759e640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80ad970 0x7fc190003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a759e640 1 -- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190003f40 msgr2=0x7fc1900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a759e640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190003f40 0x7fc1900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a759e640 1 -- 172.21.15.60:0/702274843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc190003520 con 0x7fc190004d00 2023-10-16T12:52:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.733+0000 7fc1a659c640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190003f40 0x7fc1900047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.734+0000 7fc1a759e640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc190004d00 0x7fc190005180 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e170 tx=0x7fc1900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1f20428c852223 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.734+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a002cc20 con 0x7fc190004d00 2023-10-16T12:52:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.734+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a000f040 con 0x7fc190004d00 2023-10-16T12:52:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.734+0000 7fc1a5d9b640 1 -- 172.21.15.60:0/702274843 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0034440 con 0x7fc190004d00 2023-10-16T12:52:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1ac86e640 1 -- 172.21.15.60:0/702274843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc190004d00 msgr2=0x7fc190005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/702274843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc190004d00 0x7fc190005180 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e170 tx=0x7fc1900057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1a759e640 1 -- 172.21.15.60:0/702274843 reap_dead start 2023-10-16T12:52:36.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1ac86e640 1 -- 172.21.15.60:0/702274843 shutdown_connections 2023-10-16T12:52:36.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1ac86e640 1 -- 172.21.15.60:0/702274843 >> 172.21.15.60:0/702274843 conn(0x7fc1a80ce6a0 msgr2=0x7fc1a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:36.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1ac86e640 1 -- 172.21.15.60:0/702274843 shutdown_connections 2023-10-16T12:52:36.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.735+0000 7fc1ac86e640 1 -- 172.21.15.60:0/702274843 wait complete. 2023-10-16T12:52:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.737+0000 7fc1ac86e640 1 Processor -- start 2023-10-16T12:52:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.737+0000 7fc1ac86e640 1 -- start start 2023-10-16T12:52:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1ac86e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190004d00 0x7fc1a80c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1ac86e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 0x7fc1a80c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1ac86e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80c81a0 0x7fc1a813eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1ac86e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80c8630 con 0x7fc1a8159770 2023-10-16T12:52:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1ac86e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a81f2760 con 0x7fc190004d00 2023-10-16T12:52:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1ac86e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a81586d0 con 0x7fc1a80c81a0 2023-10-16T12:52:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 0x7fc1a80c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1a659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 0x7fc1a80c7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38140/0 (socket says 172.21.15.60:38140) 2023-10-16T12:52:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1a659c640 1 -- 172.21.15.60:0/3131950616 learned_addr learned my addr 172.21.15.60:0/3131950616 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.738+0000 7fc1a6d9d640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190004d00 0x7fc1a80c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a759e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80c81a0 0x7fc1a813eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a659c640 1 -- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80c81a0 msgr2=0x7fc1a813eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a659c640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80c81a0 0x7fc1a813eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a659c640 1 -- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190004d00 msgr2=0x7fc1a80c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a659c640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190004d00 0x7fc1a80c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a659c640 1 -- 172.21.15.60:0/3131950616 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a81589b0 con 0x7fc1a8159770 2023-10-16T12:52:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1a659c640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 0x7fc1a80c7c60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fc19c010890 tx=0x7fc19c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c00eb00 con 0x7fc1a8159770 2023-10-16T12:52:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c00ec60 con 0x7fc1a8159770 2023-10-16T12:52:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c01c400 con 0x7fc1a8159770 2023-10-16T12:52:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a8158c90 con 0x7fc1a8159770 2023-10-16T12:52:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.739+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a8159250 con 0x7fc1a8159770 2023-10-16T12:52:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.741+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a806b330 con 0x7fc1a8159770 2023-10-16T12:52:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.742+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc19c01c560 con 0x7fc1a8159770 2023-10-16T12:52:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.743+0000 7fc18f7fe640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc19406af90 0x7fc19406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.743+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc19c089ce0 con 0x7fc1a8159770 2023-10-16T12:52:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.744+0000 7fc1a6d9d640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc19406af90 0x7fc19406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.745+0000 7fc1a6d9d640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc19406af90 0x7fc19406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc198001bc0 tx=0x7fc198009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.748+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc19c054ca0 con 0x7fc1a8159770 2023-10-16T12:52:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.945+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1a804e9b0 con 0x7fc1a8159770 2023-10-16T12:52:36.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.947+0000 7fc18f7fe640 1 -- 172.21.15.60:0/3131950616 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16507 (secure 0 0 0) 0x7fc19c021020 con 0x7fc1a8159770 2023-10-16T12:52:36.949 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:36.949 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:33.953552+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:52:36.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc19406af90 msgr2=0x7fc19406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc19406af90 0x7fc19406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc198001bc0 tx=0x7fc198009040 comp rx=0 tx=0).stop 2023-10-16T12:52:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 msgr2=0x7fc1a80c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 0x7fc1a80c7c60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fc19c010890 tx=0x7fc19c010d50 comp rx=0 tx=0).stop 2023-10-16T12:52:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 shutdown_connections 2023-10-16T12:52:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fc19406af90 0x7fc19406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc1a80c81a0 0x7fc1a813eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a8159770 0x7fc1a80c7c60 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 --2- 172.21.15.60:0/3131950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc190004d00 0x7fc1a80c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:36.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 >> 172.21.15.60:0/3131950616 conn(0x7fc1a80ce6a0 msgr2=0x7fc1a806a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:36.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 shutdown_connections 2023-10-16T12:52:36.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:36.950+0000 7fc1ac86e640 1 -- 172.21.15.60:0/3131950616 wait complete. 2023-10-16T12:52:36.972 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:52:36.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:52:37.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.196+0000 7f61f156a640 1 Processor -- start 2023-10-16T12:52:37.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.197+0000 7f61f156a640 1 -- start start 2023-10-16T12:52:37.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.198+0000 7f61f156a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 0x7f61ec0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.198+0000 7f61f156a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec160a90 con 0x7f61ec15dde0 2023-10-16T12:52:37.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.198+0000 7f61f156a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec0cd900 con 0x7f61ec0c9230 2023-10-16T12:52:37.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.198+0000 7f61f156a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0cdec0 con 0x7f61ec15c9f0 2023-10-16T12:52:37.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.198+0000 7f61f0d69640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61ec15dde0 0x7f61ec0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46786/0 (socket says 172.21.15.60:46786) 2023-10-16T12:52:37.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61f0d69640 1 -- 172.21.15.60:0/2021584309 learned_addr learned my addr 172.21.15.60:0/2021584309 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:37.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61ebfff640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 0x7f61ec0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397571643 0 0) 0x7f61ec160a90 con 0x7f61ec15dde0 2023-10-16T12:52:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d8003520 con 0x7f61ec15dde0 2023-10-16T12:52:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735662884 0 0) 0x7f61d8003520 con 0x7f61ec15dde0 2023-10-16T12:52:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 msgr2=0x7f61ec0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:37.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 0x7f61ec0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 >> v1:172.21.15.144:6789/0 conn(0x7f61ec0c9230 legacy=0x7f61ec0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec0cdbe0 con 0x7f61ec15dde0 2023-10-16T12:52:37.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.199+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f61e40019f0 con 0x7f61ec15dde0 2023-10-16T12:52:37.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 >> v1:172.21.15.60:6789/0 conn(0x7f61ec15dde0 legacy=0x7f61ec0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61d80039a0 0x7f61d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 0x7f61d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61d8004d00 0x7f61d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d800ba30 con 0x7f61d8004d00 2023-10-16T12:52:37.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d800bba0 con 0x7f61d80039a0 2023-10-16T12:52:37.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d800bd60 con 0x7f61ec15c9f0 2023-10-16T12:52:37.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61f0d69640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61d8004d00 0x7f61d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61ebfff640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61d80039a0 0x7f61d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61f0d69640 1 -- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 msgr2=0x7f61d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:37.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61f0d69640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61ec15c9f0 0x7f61d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61f0d69640 1 -- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61d80039a0 msgr2=0x7f61d8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61f0d69640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61d80039a0 0x7f61d8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61f0d69640 1 -- 172.21.15.60:0/2021584309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d8003520 con 0x7f61d8004d00 2023-10-16T12:52:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.200+0000 7f61ebfff640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61d80039a0 0x7f61d8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61f0d69640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61d8004d00 0x7f61d8005180 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f61e4002230 tx=0x7f61d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae29855b9746817 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4034630 con 0x7f61d8004d00 2023-10-16T12:52:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e402f070 con 0x7f61d8004d00 2023-10-16T12:52:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61eb7fe640 1 -- 172.21.15.60:0/2021584309 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4034a10 con 0x7f61d8004d00 2023-10-16T12:52:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61f156a640 1 -- 172.21.15.60:0/2021584309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61d8004d00 msgr2=0x7f61d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61f156a640 1 --2- 172.21.15.60:0/2021584309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61d8004d00 0x7f61d8005180 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f61e4002230 tx=0x7f61d80057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61f0d69640 1 -- 172.21.15.60:0/2021584309 reap_dead start 2023-10-16T12:52:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61f156a640 1 -- 172.21.15.60:0/2021584309 shutdown_connections 2023-10-16T12:52:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.202+0000 7f61f156a640 1 -- 172.21.15.60:0/2021584309 >> 172.21.15.60:0/2021584309 conn(0x7f61ec0ce6a0 msgr2=0x7f61ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.203+0000 7f61f156a640 1 -- 172.21.15.60:0/2021584309 shutdown_connections 2023-10-16T12:52:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.203+0000 7f61f156a640 1 -- 172.21.15.60:0/2021584309 wait complete. 2023-10-16T12:52:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.204+0000 7f61f156a640 1 Processor -- start 2023-10-16T12:52:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.205+0000 7f61f156a640 1 -- start start 2023-10-16T12:52:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.205+0000 7f61f156a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61d8004d00 0x7f61ec1f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.205+0000 7f61f156a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 0x7f61ec1f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.205+0000 7f61f156a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61ec1f5d10 0x7f61ec0c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61f156a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec1f61c0 con 0x7f61ec1f64d0 2023-10-16T12:52:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61f156a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0c5c10 con 0x7f61ec1f5d10 2023-10-16T12:52:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61f156a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec1f20c0 con 0x7f61d8004d00 2023-10-16T12:52:37.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 0x7f61ec1f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 0x7f61ec1f57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38152/0 (socket says 172.21.15.60:38152) 2023-10-16T12:52:37.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 -- 172.21.15.60:0/3988159226 learned_addr learned my addr 172.21.15.60:0/3988159226 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:37.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61ebfff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61d8004d00 0x7f61ec1f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 -- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61d8004d00 msgr2=0x7f61ec1f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61d8004d00 0x7f61ec1f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 -- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61ec1f5d10 msgr2=0x7f61ec0c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61ec1f5d10 0x7f61ec0c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61e3fff640 1 -- 172.21.15.60:0/3988159226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec06b710 con 0x7f61ec1f64d0 2023-10-16T12:52:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.206+0000 7f61ebfff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61d8004d00 0x7f61ec1f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.207+0000 7f61e3fff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 0x7f61ec1f57d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f61d400c430 tx=0x7f61d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.207+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40052d0 con 0x7f61ec1f64d0 2023-10-16T12:52:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.207+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d4005430 con 0x7f61ec1f64d0 2023-10-16T12:52:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.207+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4003830 con 0x7f61ec1f64d0 2023-10-16T12:52:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.207+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec06b9f0 con 0x7f61ec1f64d0 2023-10-16T12:52:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.207+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec06bcd0 con 0x7f61ec1f64d0 2023-10-16T12:52:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.208+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec0c5da0 con 0x7f61ec1f64d0 2023-10-16T12:52:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.209+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f61d4003990 con 0x7f61ec1f64d0 2023-10-16T12:52:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.210+0000 7f61e97fa640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61cc06afc0 0x7f61cc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.210+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61d4088830 con 0x7f61ec1f64d0 2023-10-16T12:52:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.211+0000 7f61ebfff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61cc06afc0 0x7f61cc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.212+0000 7f61ebfff640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61cc06afc0 0x7f61cc06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009ac0 tx=0x7f61dc008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.215+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61d4053860 con 0x7f61ec1f64d0 2023-10-16T12:52:37.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.406+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 --> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61ec0704c0 con 0x7f61cc06afc0 2023-10-16T12:52:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.435+0000 7f61e97fa640 1 -- 172.21.15.60:0/3988159226 <== mgr.6574 v2:172.21.15.144:6832/3337897861 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287290 (secure 0 0 0) 0x7f61dc011980 con 0x7f61cc06afc0 2023-10-16T12:52:37.441 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:52:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.440+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61cc06afc0 msgr2=0x7f61cc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.440+0000 7f61f156a640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61cc06afc0 0x7f61cc06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009ac0 tx=0x7f61dc008040 comp rx=0 tx=0).stop 2023-10-16T12:52:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 msgr2=0x7f61ec1f57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 0x7f61ec1f57d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f61d400c430 tx=0x7f61d400c8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 shutdown_connections 2023-10-16T12:52:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f61cc06afc0 0x7f61cc06d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61ec1f5d10 0x7f61ec0c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ec1f64d0 0x7f61ec1f57d0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 --2- 172.21.15.60:0/3988159226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61d8004d00 0x7f61ec1f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 >> 172.21.15.60:0/3988159226 conn(0x7f61ec0ce6a0 msgr2=0x7f61ec0cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 shutdown_connections 2023-10-16T12:52:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.441+0000 7f61f156a640 1 -- 172.21.15.60:0/3988159226 wait complete. 2023-10-16T12:52:37.462 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:52:36.224612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":367,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":332348,"kb_used_data":13520,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754642372,"statfs":{"total":773094113280,"available":772753788928,"internally_reserved":0,"allocated":13844480,"data_stored":7204452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":326356536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.322121"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968970+0000","last_change":"2023-10-16T12:52:32.982080+0000","last_active":"2023-10-16T12:52:33.968970+0000","last_peered":"2023-10-16T12:52:33.968970+0000","last_clean":"2023-10-16T12:52:33.968970+0000","last_became_active":"2023-10-16T12:52:32.981780+0000","last_became_peered":"2023-10-16T12:52:32.981780+0000","last_unstale":"2023-10-16T12:52:33.968970+0000","last_undegraded":"2023-10-16T12:52:33.968970+0000","last_fullsized":"2023-10-16T12:52:33.968970+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:43.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690416+0000","last_change":"2023-10-16T12:52:32.983006+0000","last_active":"2023-10-16T12:52:34.690416+0000","last_peered":"2023-10-16T12:52:34.690416+0000","last_clean":"2023-10-16T12:52:34.690416+0000","last_became_active":"2023-10-16T12:52:32.972755+0000","last_became_peered":"2023-10-16T12:52:32.972755+0000","last_unstale":"2023-10-16T12:52:34.690416+0000","last_undegraded":"2023-10-16T12:52:34.690416+0000","last_fullsized":"2023-10-16T12:52:34.690416+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:19.066290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969043+0000","last_change":"2023-10-16T12:52:32.982515+0000","last_active":"2023-10-16T12:52:33.969043+0000","last_peered":"2023-10-16T12:52:33.969043+0000","last_clean":"2023-10-16T12:52:33.969043+0000","last_became_active":"2023-10-16T12:52:32.980783+0000","last_became_peered":"2023-10-16T12:52:32.980783+0000","last_unstale":"2023-10-16T12:52:33.969043+0000","last_undegraded":"2023-10-16T12:52:33.969043+0000","last_fullsized":"2023-10-16T12:52:33.969043+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:31.936481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577214+0000","last_change":"2023-10-16T12:52:32.969157+0000","last_active":"2023-10-16T12:52:34.577214+0000","last_peered":"2023-10-16T12:52:34.577214+0000","last_clean":"2023-10-16T12:52:34.577214+0000","last_became_active":"2023-10-16T12:52:32.968851+0000","last_became_peered":"2023-10-16T12:52:32.968851+0000","last_unstale":"2023-10-16T12:52:34.577214+0000","last_undegraded":"2023-10-16T12:52:34.577214+0000","last_fullsized":"2023-10-16T12:52:34.577214+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:30.313878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971344+0000","last_change":"2023-10-16T12:52:32.970626+0000","last_active":"2023-10-16T12:52:33.971344+0000","last_peered":"2023-10-16T12:52:33.971344+0000","last_clean":"2023-10-16T12:52:33.971344+0000","last_became_active":"2023-10-16T12:52:32.970238+0000","last_became_peered":"2023-10-16T12:52:32.970238+0000","last_unstale":"2023-10-16T12:52:33.971344+0000","last_undegraded":"2023-10-16T12:52:33.971344+0000","last_fullsized":"2023-10-16T12:52:33.971344+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:10.822211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689693+0000","last_change":"2023-10-16T12:52:32.961451+0000","last_active":"2023-10-16T12:52:34.689693+0000","last_peered":"2023-10-16T12:52:34.689693+0000","last_clean":"2023-10-16T12:52:34.689693+0000","last_became_active":"2023-10-16T12:52:32.961114+0000","last_became_peered":"2023-10-16T12:52:32.961114+0000","last_unstale":"2023-10-16T12:52:34.689693+0000","last_undegraded":"2023-10-16T12:52:34.689693+0000","last_fullsized":"2023-10-16T12:52:34.689693+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:24.418432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689093+0000","last_change":"2023-10-16T12:52:32.968318+0000","last_active":"2023-10-16T12:52:34.689093+0000","last_peered":"2023-10-16T12:52:34.689093+0000","last_clean":"2023-10-16T12:52:34.689093+0000","last_became_active":"2023-10-16T12:52:32.967344+0000","last_became_peered":"2023-10-16T12:52:32.967344+0000","last_unstale":"2023-10-16T12:52:34.689093+0000","last_undegraded":"2023-10-16T12:52:34.689093+0000","last_fullsized":"2023-10-16T12:52:34.689093+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:14.904662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689800+0000","last_change":"2023-10-16T12:52:32.964609+0000","last_active":"2023-10-16T12:52:34.689800+0000","last_peered":"2023-10-16T12:52:34.689800+0000","last_clean":"2023-10-16T12:52:34.689800+0000","last_became_active":"2023-10-16T12:52:32.964325+0000","last_became_peered":"2023-10-16T12:52:32.964325+0000","last_unstale":"2023-10-16T12:52:34.689800+0000","last_undegraded":"2023-10-16T12:52:34.689800+0000","last_fullsized":"2023-10-16T12:52:34.689800+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:29.118326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691526+0000","last_change":"2023-10-16T12:52:32.986124+0000","last_active":"2023-10-16T12:52:34.691526+0000","last_peered":"2023-10-16T12:52:34.691526+0000","last_clean":"2023-10-16T12:52:34.691526+0000","last_became_active":"2023-10-16T12:52:32.985825+0000","last_became_peered":"2023-10-16T12:52:32.985825+0000","last_unstale":"2023-10-16T12:52:34.691526+0000","last_undegraded":"2023-10-16T12:52:34.691526+0000","last_fullsized":"2023-10-16T12:52:34.691526+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:22.340901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972028+0000","last_change":"2023-10-16T12:52:32.971929+0000","last_active":"2023-10-16T12:52:33.972028+0000","last_peered":"2023-10-16T12:52:33.972028+0000","last_clean":"2023-10-16T12:52:33.972028+0000","last_became_active":"2023-10-16T12:52:32.971617+0000","last_became_peered":"2023-10-16T12:52:32.971617+0000","last_unstale":"2023-10-16T12:52:33.972028+0000","last_undegraded":"2023-10-16T12:52:33.972028+0000","last_fullsized":"2023-10-16T12:52:33.972028+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:58.405850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689289+0000","last_change":"2023-10-16T12:52:32.964556+0000","last_active":"2023-10-16T12:52:34.689289+0000","last_peered":"2023-10-16T12:52:34.689289+0000","last_clean":"2023-10-16T12:52:34.689289+0000","last_became_active":"2023-10-16T12:52:32.964260+0000","last_became_peered":"2023-10-16T12:52:32.964260+0000","last_unstale":"2023-10-16T12:52:34.689289+0000","last_undegraded":"2023-10-16T12:52:34.689289+0000","last_fullsized":"2023-10-16T12:52:34.689289+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:49.681741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691153+0000","last_change":"2023-10-16T12:52:32.967708+0000","last_active":"2023-10-16T12:52:34.691153+0000","last_peered":"2023-10-16T12:52:34.691153+0000","last_clean":"2023-10-16T12:52:34.691153+0000","last_became_active":"2023-10-16T12:52:32.967413+0000","last_became_peered":"2023-10-16T12:52:32.967413+0000","last_unstale":"2023-10-16T12:52:34.691153+0000","last_undegraded":"2023-10-16T12:52:34.691153+0000","last_fullsized":"2023-10-16T12:52:34.691153+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:29:54.151963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576592+0000","last_change":"2023-10-16T12:52:32.964947+0000","last_active":"2023-10-16T12:52:34.576592+0000","last_peered":"2023-10-16T12:52:34.576592+0000","last_clean":"2023-10-16T12:52:34.576592+0000","last_became_active":"2023-10-16T12:52:32.964618+0000","last_became_peered":"2023-10-16T12:52:32.964618+0000","last_unstale":"2023-10-16T12:52:34.576592+0000","last_undegraded":"2023-10-16T12:52:34.576592+0000","last_fullsized":"2023-10-16T12:52:34.576592+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:40.320909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690395+0000","last_change":"2023-10-16T12:52:32.956714+0000","last_active":"2023-10-16T12:52:34.690395+0000","last_peered":"2023-10-16T12:52:34.690395+0000","last_clean":"2023-10-16T12:52:34.690395+0000","last_became_active":"2023-10-16T12:52:32.956394+0000","last_became_peered":"2023-10-16T12:52:32.956394+0000","last_unstale":"2023-10-16T12:52:34.690395+0000","last_undegraded":"2023-10-16T12:52:34.690395+0000","last_fullsized":"2023-10-16T12:52:34.690395+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:19.066290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576428+0000","last_change":"2023-10-16T12:52:32.981843+0000","last_active":"2023-10-16T12:52:34.576428+0000","last_peered":"2023-10-16T12:52:34.576428+0000","last_clean":"2023-10-16T12:52:34.576428+0000","last_became_active":"2023-10-16T12:52:32.981580+0000","last_became_peered":"2023-10-16T12:52:32.981580+0000","last_unstale":"2023-10-16T12:52:34.576428+0000","last_undegraded":"2023-10-16T12:52:34.576428+0000","last_fullsized":"2023-10-16T12:52:34.576428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:36.244079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689000+0000","last_change":"2023-10-16T12:52:32.966307+0000","last_active":"2023-10-16T12:52:34.689000+0000","last_peered":"2023-10-16T12:52:34.689000+0000","last_clean":"2023-10-16T12:52:34.689000+0000","last_became_active":"2023-10-16T12:52:32.965562+0000","last_became_peered":"2023-10-16T12:52:32.965562+0000","last_unstale":"2023-10-16T12:52:34.689000+0000","last_undegraded":"2023-10-16T12:52:34.689000+0000","last_fullsized":"2023-10-16T12:52:34.689000+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:47.691249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690640+0000","last_change":"2023-10-16T12:52:32.987134+0000","last_active":"2023-10-16T12:52:34.690640+0000","last_peered":"2023-10-16T12:52:34.690640+0000","last_clean":"2023-10-16T12:52:34.690640+0000","last_became_active":"2023-10-16T12:52:32.986824+0000","last_became_peered":"2023-10-16T12:52:32.986824+0000","last_unstale":"2023-10-16T12:52:34.690640+0000","last_undegraded":"2023-10-16T12:52:34.690640+0000","last_fullsized":"2023-10-16T12:52:34.690640+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:42.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577288+0000","last_change":"2023-10-16T12:52:32.962930+0000","last_active":"2023-10-16T12:52:34.577288+0000","last_peered":"2023-10-16T12:52:34.577288+0000","last_clean":"2023-10-16T12:52:34.577288+0000","last_became_active":"2023-10-16T12:52:32.962650+0000","last_became_peered":"2023-10-16T12:52:32.962650+0000","last_unstale":"2023-10-16T12:52:34.577288+0000","last_undegraded":"2023-10-16T12:52:34.577288+0000","last_fullsized":"2023-10-16T12:52:34.577288+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:54.737606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692175+0000","last_change":"2023-10-16T12:52:32.975694+0000","last_active":"2023-10-16T12:52:34.692175+0000","last_peered":"2023-10-16T12:52:34.692175+0000","last_clean":"2023-10-16T12:52:34.692175+0000","last_became_active":"2023-10-16T12:52:32.975352+0000","last_became_peered":"2023-10-16T12:52:32.975352+0000","last_unstale":"2023-10-16T12:52:34.692175+0000","last_undegraded":"2023-10-16T12:52:34.692175+0000","last_fullsized":"2023-10-16T12:52:34.692175+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:44.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691507+0000","last_change":"2023-10-16T12:52:32.975781+0000","last_active":"2023-10-16T12:52:34.691507+0000","last_peered":"2023-10-16T12:52:34.691507+0000","last_clean":"2023-10-16T12:52:34.691507+0000","last_became_active":"2023-10-16T12:52:32.974496+0000","last_became_peered":"2023-10-16T12:52:32.974496+0000","last_unstale":"2023-10-16T12:52:34.691507+0000","last_undegraded":"2023-10-16T12:52:34.691507+0000","last_fullsized":"2023-10-16T12:52:34.691507+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:20.952208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691741+0000","last_change":"2023-10-16T12:52:32.975559+0000","last_active":"2023-10-16T12:52:34.691741+0000","last_peered":"2023-10-16T12:52:34.691741+0000","last_clean":"2023-10-16T12:52:34.691741+0000","last_became_active":"2023-10-16T12:52:32.975268+0000","last_became_peered":"2023-10-16T12:52:32.975268+0000","last_unstale":"2023-10-16T12:52:34.691741+0000","last_undegraded":"2023-10-16T12:52:34.691741+0000","last_fullsized":"2023-10-16T12:52:34.691741+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:01.930459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691673+0000","last_change":"2023-10-16T12:52:32.970933+0000","last_active":"2023-10-16T12:52:34.691673+0000","last_peered":"2023-10-16T12:52:34.691673+0000","last_clean":"2023-10-16T12:52:34.691673+0000","last_became_active":"2023-10-16T12:52:32.970666+0000","last_became_peered":"2023-10-16T12:52:32.970666+0000","last_unstale":"2023-10-16T12:52:34.691673+0000","last_undegraded":"2023-10-16T12:52:34.691673+0000","last_fullsized":"2023-10-16T12:52:34.691673+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:01.930459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689829+0000","last_change":"2023-10-16T12:52:32.970781+0000","last_active":"2023-10-16T12:52:34.689829+0000","last_peered":"2023-10-16T12:52:34.689829+0000","last_clean":"2023-10-16T12:52:34.689829+0000","last_became_active":"2023-10-16T12:52:32.970469+0000","last_became_peered":"2023-10-16T12:52:32.970469+0000","last_unstale":"2023-10-16T12:52:34.689829+0000","last_undegraded":"2023-10-16T12:52:34.689829+0000","last_fullsized":"2023-10-16T12:52:34.689829+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:21.452261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689406+0000","last_change":"2023-10-16T12:52:32.971507+0000","last_active":"2023-10-16T12:52:34.689406+0000","last_peered":"2023-10-16T12:52:34.689406+0000","last_clean":"2023-10-16T12:52:34.689406+0000","last_became_active":"2023-10-16T12:52:32.970904+0000","last_became_peered":"2023-10-16T12:52:32.970904+0000","last_unstale":"2023-10-16T12:52:34.689406+0000","last_undegraded":"2023-10-16T12:52:34.689406+0000","last_fullsized":"2023-10-16T12:52:34.689406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:26.693581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972406+0000","last_change":"2023-10-16T12:52:32.965139+0000","last_active":"2023-10-16T12:52:33.972406+0000","last_peered":"2023-10-16T12:52:33.972406+0000","last_clean":"2023-10-16T12:52:33.972406+0000","last_became_active":"2023-10-16T12:52:32.964855+0000","last_became_peered":"2023-10-16T12:52:32.964855+0000","last_unstale":"2023-10-16T12:52:33.972406+0000","last_undegraded":"2023-10-16T12:52:33.972406+0000","last_fullsized":"2023-10-16T12:52:33.972406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:08:09.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690764+0000","last_change":"2023-10-16T12:52:32.983256+0000","last_active":"2023-10-16T12:52:34.690764+0000","last_peered":"2023-10-16T12:52:34.690764+0000","last_clean":"2023-10-16T12:52:34.690764+0000","last_became_active":"2023-10-16T12:52:32.982687+0000","last_became_peered":"2023-10-16T12:52:32.982687+0000","last_unstale":"2023-10-16T12:52:34.690764+0000","last_undegraded":"2023-10-16T12:52:34.690764+0000","last_fullsized":"2023-10-16T12:52:34.690764+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:20.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689535+0000","last_change":"2023-10-16T12:52:32.956515+0000","last_active":"2023-10-16T12:52:34.689535+0000","last_peered":"2023-10-16T12:52:34.689535+0000","last_clean":"2023-10-16T12:52:34.689535+0000","last_became_active":"2023-10-16T12:52:32.955440+0000","last_became_peered":"2023-10-16T12:52:32.955440+0000","last_unstale":"2023-10-16T12:52:34.689535+0000","last_undegraded":"2023-10-16T12:52:34.689535+0000","last_fullsized":"2023-10-16T12:52:34.689535+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:14.201658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971565+0000","last_change":"2023-10-16T12:52:32.969204+0000","last_active":"2023-10-16T12:52:33.971565+0000","last_peered":"2023-10-16T12:52:33.971565+0000","last_clean":"2023-10-16T12:52:33.971565+0000","last_became_active":"2023-10-16T12:52:32.968911+0000","last_became_peered":"2023-10-16T12:52:32.968911+0000","last_unstale":"2023-10-16T12:52:33.971565+0000","last_undegraded":"2023-10-16T12:52:33.971565+0000","last_fullsized":"2023-10-16T12:52:33.971565+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:10.594986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690967+0000","last_change":"2023-10-16T12:52:32.977462+0000","last_active":"2023-10-16T12:52:34.690967+0000","last_peered":"2023-10-16T12:52:34.690967+0000","last_clean":"2023-10-16T12:52:34.690967+0000","last_became_active":"2023-10-16T12:52:32.977182+0000","last_became_peered":"2023-10-16T12:52:32.977182+0000","last_unstale":"2023-10-16T12:52:34.690967+0000","last_undegraded":"2023-10-16T12:52:34.690967+0000","last_fullsized":"2023-10-16T12:52:34.690967+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:42.073865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690781+0000","last_change":"2023-10-16T12:52:32.976653+0000","last_active":"2023-10-16T12:52:34.690781+0000","last_peered":"2023-10-16T12:52:34.690781+0000","last_clean":"2023-10-16T12:52:34.690781+0000","last_became_active":"2023-10-16T12:52:32.975283+0000","last_became_peered":"2023-10-16T12:52:32.975283+0000","last_unstale":"2023-10-16T12:52:34.690781+0000","last_undegraded":"2023-10-16T12:52:34.690781+0000","last_fullsized":"2023-10-16T12:52:34.690781+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:42.073865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576709+0000","last_change":"2023-10-16T12:52:32.968204+0000","last_active":"2023-10-16T12:52:34.576709+0000","last_peered":"2023-10-16T12:52:34.576709+0000","last_clean":"2023-10-16T12:52:34.576709+0000","last_became_active":"2023-10-16T12:52:32.967837+0000","last_became_peered":"2023-10-16T12:52:32.967837+0000","last_unstale":"2023-10-16T12:52:34.576709+0000","last_undegraded":"2023-10-16T12:52:34.576709+0000","last_fullsized":"2023-10-16T12:52:34.576709+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:09.915933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.967920+0000","last_change":"2023-10-16T12:52:32.976622+0000","last_active":"2023-10-16T12:52:33.967920+0000","last_peered":"2023-10-16T12:52:33.967920+0000","last_clean":"2023-10-16T12:52:33.967920+0000","last_became_active":"2023-10-16T12:52:32.976213+0000","last_became_peered":"2023-10-16T12:52:32.976213+0000","last_unstale":"2023-10-16T12:52:33.967920+0000","last_undegraded":"2023-10-16T12:52:33.967920+0000","last_fullsized":"2023-10-16T12:52:33.967920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:09.857867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692364+0000","last_change":"2023-10-16T12:52:32.958259+0000","last_active":"2023-10-16T12:52:34.692364+0000","last_peered":"2023-10-16T12:52:34.692364+0000","last_clean":"2023-10-16T12:52:34.692364+0000","last_became_active":"2023-10-16T12:52:32.957948+0000","last_became_peered":"2023-10-16T12:52:32.957948+0000","last_unstale":"2023-10-16T12:52:34.692364+0000","last_undegraded":"2023-10-16T12:52:34.692364+0000","last_fullsized":"2023-10-16T12:52:34.692364+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:09.857867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"127'4","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690887+0000","last_change":"2023-10-16T12:52:31.941706+0000","last_active":"2023-10-16T12:52:34.690887+0000","last_peered":"2023-10-16T12:52:34.690887+0000","last_clean":"2023-10-16T12:52:34.690887+0000","last_became_active":"2023-10-16T12:52:31.941434+0000","last_became_peered":"2023-10-16T12:52:31.941434+0000","last_unstale":"2023-10-16T12:52:34.690887+0000","last_undegraded":"2023-10-16T12:52:34.690887+0000","last_fullsized":"2023-10-16T12:52:34.690887+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:53.480734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577540+0000","last_change":"2023-10-16T12:52:32.973277+0000","last_active":"2023-10-16T12:52:34.577540+0000","last_peered":"2023-10-16T12:52:34.577540+0000","last_clean":"2023-10-16T12:52:34.577540+0000","last_became_active":"2023-10-16T12:52:32.972994+0000","last_became_peered":"2023-10-16T12:52:32.972994+0000","last_unstale":"2023-10-16T12:52:34.577540+0000","last_undegraded":"2023-10-16T12:52:34.577540+0000","last_fullsized":"2023-10-16T12:52:34.577540+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:48.415965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689643+0000","last_change":"2023-10-16T12:52:31.947702+0000","last_active":"2023-10-16T12:52:34.689643+0000","last_peered":"2023-10-16T12:52:34.689643+0000","last_clean":"2023-10-16T12:52:34.689643+0000","last_became_active":"2023-10-16T12:52:31.947275+0000","last_became_peered":"2023-10-16T12:52:31.947275+0000","last_unstale":"2023-10-16T12:52:34.689643+0000","last_undegraded":"2023-10-16T12:52:34.689643+0000","last_fullsized":"2023-10-16T12:52:34.689643+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:45.370230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969009+0000","last_change":"2023-10-16T12:52:32.981451+0000","last_active":"2023-10-16T12:52:33.969009+0000","last_peered":"2023-10-16T12:52:33.969009+0000","last_clean":"2023-10-16T12:52:33.969009+0000","last_became_active":"2023-10-16T12:52:32.981168+0000","last_became_peered":"2023-10-16T12:52:32.981168+0000","last_unstale":"2023-10-16T12:52:33.969009+0000","last_undegraded":"2023-10-16T12:52:33.969009+0000","last_fullsized":"2023-10-16T12:52:33.969009+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:20.167134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690759+0000","last_change":"2023-10-16T12:52:31.948780+0000","last_active":"2023-10-16T12:52:34.690759+0000","last_peered":"2023-10-16T12:52:34.690759+0000","last_clean":"2023-10-16T12:52:34.690759+0000","last_became_active":"2023-10-16T12:52:31.948475+0000","last_became_peered":"2023-10-16T12:52:31.948475+0000","last_unstale":"2023-10-16T12:52:34.690759+0000","last_undegraded":"2023-10-16T12:52:34.690759+0000","last_fullsized":"2023-10-16T12:52:34.690759+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:59.552949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691532+0000","last_change":"2023-10-16T12:52:32.971887+0000","last_active":"2023-10-16T12:52:34.691532+0000","last_peered":"2023-10-16T12:52:34.691532+0000","last_clean":"2023-10-16T12:52:34.691532+0000","last_became_active":"2023-10-16T12:52:32.970322+0000","last_became_peered":"2023-10-16T12:52:32.970322+0000","last_unstale":"2023-10-16T12:52:34.691532+0000","last_undegraded":"2023-10-16T12:52:34.691532+0000","last_fullsized":"2023-10-16T12:52:34.691532+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:55.063149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690039+0000","last_change":"2023-10-16T12:52:31.935360+0000","last_active":"2023-10-16T12:52:34.690039+0000","last_peered":"2023-10-16T12:52:34.690039+0000","last_clean":"2023-10-16T12:52:34.690039+0000","last_became_active":"2023-10-16T12:52:31.932585+0000","last_became_peered":"2023-10-16T12:52:31.932585+0000","last_unstale":"2023-10-16T12:52:34.690039+0000","last_undegraded":"2023-10-16T12:52:34.690039+0000","last_fullsized":"2023-10-16T12:52:34.690039+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:09.940167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973406+0000","last_change":"2023-10-16T12:52:32.966332+0000","last_active":"2023-10-16T12:52:33.973406+0000","last_peered":"2023-10-16T12:52:33.973406+0000","last_clean":"2023-10-16T12:52:33.973406+0000","last_became_active":"2023-10-16T12:52:32.966050+0000","last_became_peered":"2023-10-16T12:52:32.966050+0000","last_unstale":"2023-10-16T12:52:33.973406+0000","last_undegraded":"2023-10-16T12:52:33.973406+0000","last_fullsized":"2023-10-16T12:52:33.973406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:33.889940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968551+0000","last_change":"2023-10-16T12:52:31.936086+0000","last_active":"2023-10-16T12:52:33.968551+0000","last_peered":"2023-10-16T12:52:33.968551+0000","last_clean":"2023-10-16T12:52:33.968551+0000","last_became_active":"2023-10-16T12:52:31.935759+0000","last_became_peered":"2023-10-16T12:52:31.935759+0000","last_unstale":"2023-10-16T12:52:33.968551+0000","last_undegraded":"2023-10-16T12:52:33.968551+0000","last_fullsized":"2023-10-16T12:52:33.968551+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:08.683475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689147+0000","last_change":"2023-10-16T12:52:32.953315+0000","last_active":"2023-10-16T12:52:34.689147+0000","last_peered":"2023-10-16T12:52:34.689147+0000","last_clean":"2023-10-16T12:52:34.689147+0000","last_became_active":"2023-10-16T12:52:32.952608+0000","last_became_peered":"2023-10-16T12:52:32.952608+0000","last_unstale":"2023-10-16T12:52:34.689147+0000","last_undegraded":"2023-10-16T12:52:34.689147+0000","last_fullsized":"2023-10-16T12:52:34.689147+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:35.485882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691920+0000","last_change":"2023-10-16T12:52:31.936889+0000","last_active":"2023-10-16T12:52:34.691920+0000","last_peered":"2023-10-16T12:52:34.691920+0000","last_clean":"2023-10-16T12:52:34.691920+0000","last_became_active":"2023-10-16T12:52:31.935613+0000","last_became_peered":"2023-10-16T12:52:31.935613+0000","last_unstale":"2023-10-16T12:52:34.691920+0000","last_undegraded":"2023-10-16T12:52:34.691920+0000","last_fullsized":"2023-10-16T12:52:34.691920+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:48.089984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576808+0000","last_change":"2023-10-16T12:52:32.981362+0000","last_active":"2023-10-16T12:52:34.576808+0000","last_peered":"2023-10-16T12:52:34.576808+0000","last_clean":"2023-10-16T12:52:34.576808+0000","last_became_active":"2023-10-16T12:52:32.980936+0000","last_became_peered":"2023-10-16T12:52:32.980936+0000","last_unstale":"2023-10-16T12:52:34.576808+0000","last_undegraded":"2023-10-16T12:52:34.576808+0000","last_fullsized":"2023-10-16T12:52:34.576808+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:41.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971514+0000","last_change":"2023-10-16T12:52:31.937779+0000","last_active":"2023-10-16T12:52:33.971514+0000","last_peered":"2023-10-16T12:52:33.971514+0000","last_clean":"2023-10-16T12:52:33.971514+0000","last_became_active":"2023-10-16T12:52:31.937466+0000","last_became_peered":"2023-10-16T12:52:31.937466+0000","last_unstale":"2023-10-16T12:52:33.971514+0000","last_undegraded":"2023-10-16T12:52:33.971514+0000","last_fullsized":"2023-10-16T12:52:33.971514+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:55.501105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690571+0000","last_change":"2023-10-16T12:52:32.970104+0000","last_active":"2023-10-16T12:52:34.690571+0000","last_peered":"2023-10-16T12:52:34.690571+0000","last_clean":"2023-10-16T12:52:34.690571+0000","last_became_active":"2023-10-16T12:52:32.969413+0000","last_became_peered":"2023-10-16T12:52:32.969413+0000","last_unstale":"2023-10-16T12:52:34.690571+0000","last_undegraded":"2023-10-16T12:52:34.690571+0000","last_fullsized":"2023-10-16T12:52:34.690571+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:06.795464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968030+0000","last_change":"2023-10-16T12:52:31.939413+0000","last_active":"2023-10-16T12:52:33.968030+0000","last_peered":"2023-10-16T12:52:33.968030+0000","last_clean":"2023-10-16T12:52:33.968030+0000","last_became_active":"2023-10-16T12:52:31.938945+0000","last_became_peered":"2023-10-16T12:52:31.938945+0000","last_unstale":"2023-10-16T12:52:33.968030+0000","last_undegraded":"2023-10-16T12:52:33.968030+0000","last_fullsized":"2023-10-16T12:52:33.968030+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:05.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691121+0000","last_change":"2023-10-16T12:52:32.970316+0000","last_active":"2023-10-16T12:52:34.691121+0000","last_peered":"2023-10-16T12:52:34.691121+0000","last_clean":"2023-10-16T12:52:34.691121+0000","last_became_active":"2023-10-16T12:52:32.970024+0000","last_became_peered":"2023-10-16T12:52:32.970024+0000","last_unstale":"2023-10-16T12:52:34.691121+0000","last_undegraded":"2023-10-16T12:52:34.691121+0000","last_fullsized":"2023-10-16T12:52:34.691121+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:09.227980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969264+0000","last_change":"2023-10-16T12:52:31.937580+0000","last_active":"2023-10-16T12:52:33.969264+0000","last_peered":"2023-10-16T12:52:33.969264+0000","last_clean":"2023-10-16T12:52:33.969264+0000","last_became_active":"2023-10-16T12:52:31.937263+0000","last_became_peered":"2023-10-16T12:52:31.937263+0000","last_unstale":"2023-10-16T12:52:33.969264+0000","last_undegraded":"2023-10-16T12:52:33.969264+0000","last_fullsized":"2023-10-16T12:52:33.969264+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:43.248737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691739+0000","last_change":"2023-10-16T12:52:32.969791+0000","last_active":"2023-10-16T12:52:34.691739+0000","last_peered":"2023-10-16T12:52:34.691739+0000","last_clean":"2023-10-16T12:52:34.691739+0000","last_became_active":"2023-10-16T12:52:32.969481+0000","last_became_peered":"2023-10-16T12:52:32.969481+0000","last_unstale":"2023-10-16T12:52:34.691739+0000","last_undegraded":"2023-10-16T12:52:34.691739+0000","last_fullsized":"2023-10-16T12:52:34.691739+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:22.385025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577207+0000","last_change":"2023-10-16T12:52:31.940500+0000","last_active":"2023-10-16T12:52:34.577207+0000","last_peered":"2023-10-16T12:52:34.577207+0000","last_clean":"2023-10-16T12:52:34.577207+0000","last_became_active":"2023-10-16T12:52:31.940186+0000","last_became_peered":"2023-10-16T12:52:31.940186+0000","last_unstale":"2023-10-16T12:52:34.577207+0000","last_undegraded":"2023-10-16T12:52:34.577207+0000","last_fullsized":"2023-10-16T12:52:34.577207+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:45.370230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972281+0000","last_change":"2023-10-16T12:52:32.968381+0000","last_active":"2023-10-16T12:52:33.972281+0000","last_peered":"2023-10-16T12:52:33.972281+0000","last_clean":"2023-10-16T12:52:33.972281+0000","last_became_active":"2023-10-16T12:52:32.967991+0000","last_became_peered":"2023-10-16T12:52:32.967991+0000","last_unstale":"2023-10-16T12:52:33.972281+0000","last_undegraded":"2023-10-16T12:52:33.972281+0000","last_fullsized":"2023-10-16T12:52:33.972281+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:32.346811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"127'3","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690031+0000","last_change":"2023-10-16T12:52:31.947894+0000","last_active":"2023-10-16T12:52:34.690031+0000","last_peered":"2023-10-16T12:52:34.690031+0000","last_clean":"2023-10-16T12:52:34.690031+0000","last_became_active":"2023-10-16T12:52:31.947401+0000","last_became_peered":"2023-10-16T12:52:31.947401+0000","last_unstale":"2023-10-16T12:52:34.690031+0000","last_undegraded":"2023-10-16T12:52:34.690031+0000","last_fullsized":"2023-10-16T12:52:34.690031+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:40.667980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689775+0000","last_change":"2023-10-16T12:52:32.967239+0000","last_active":"2023-10-16T12:52:34.689775+0000","last_peered":"2023-10-16T12:52:34.689775+0000","last_clean":"2023-10-16T12:52:34.689775+0000","last_became_active":"2023-10-16T12:52:32.966895+0000","last_became_peered":"2023-10-16T12:52:32.966895+0000","last_unstale":"2023-10-16T12:52:34.689775+0000","last_undegraded":"2023-10-16T12:52:34.689775+0000","last_fullsized":"2023-10-16T12:52:34.689775+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:41.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"127'6","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691699+0000","last_change":"2023-10-16T12:52:31.944329+0000","last_active":"2023-10-16T12:52:34.691699+0000","last_peered":"2023-10-16T12:52:34.691699+0000","last_clean":"2023-10-16T12:52:34.691699+0000","last_became_active":"2023-10-16T12:52:31.943418+0000","last_became_peered":"2023-10-16T12:52:31.943418+0000","last_unstale":"2023-10-16T12:52:34.691699+0000","last_undegraded":"2023-10-16T12:52:34.691699+0000","last_fullsized":"2023-10-16T12:52:34.691699+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:34.122017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576797+0000","last_change":"2023-10-16T12:52:32.964151+0000","last_active":"2023-10-16T12:52:34.576797+0000","last_peered":"2023-10-16T12:52:34.576797+0000","last_clean":"2023-10-16T12:52:34.576797+0000","last_became_active":"2023-10-16T12:52:32.962164+0000","last_became_peered":"2023-10-16T12:52:32.962164+0000","last_unstale":"2023-10-16T12:52:34.576797+0000","last_undegraded":"2023-10-16T12:52:34.576797+0000","last_fullsized":"2023-10-16T12:52:34.576797+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:29.826473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692269+0000","last_change":"2023-10-16T12:52:31.950744+0000","last_active":"2023-10-16T12:52:34.692269+0000","last_peered":"2023-10-16T12:52:34.692269+0000","last_clean":"2023-10-16T12:52:34.692269+0000","last_became_active":"2023-10-16T12:52:31.950465+0000","last_became_peered":"2023-10-16T12:52:31.950465+0000","last_unstale":"2023-10-16T12:52:34.692269+0000","last_undegraded":"2023-10-16T12:52:34.692269+0000","last_fullsized":"2023-10-16T12:52:34.692269+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:55.286130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968436+0000","last_change":"2023-10-16T12:52:32.981804+0000","last_active":"2023-10-16T12:52:33.968436+0000","last_peered":"2023-10-16T12:52:33.968436+0000","last_clean":"2023-10-16T12:52:33.968436+0000","last_became_active":"2023-10-16T12:52:32.981513+0000","last_became_peered":"2023-10-16T12:52:32.981513+0000","last_unstale":"2023-10-16T12:52:33.968436+0000","last_undegraded":"2023-10-16T12:52:33.968436+0000","last_fullsized":"2023-10-16T12:52:33.968436+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:03.559443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691547+0000","last_change":"2023-10-16T12:52:31.945301+0000","last_active":"2023-10-16T12:52:34.691547+0000","last_peered":"2023-10-16T12:52:34.691547+0000","last_clean":"2023-10-16T12:52:34.691547+0000","last_became_active":"2023-10-16T12:52:31.944963+0000","last_became_peered":"2023-10-16T12:52:31.944963+0000","last_unstale":"2023-10-16T12:52:34.691547+0000","last_undegraded":"2023-10-16T12:52:34.691547+0000","last_fullsized":"2023-10-16T12:52:34.691547+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:28.104854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690473+0000","last_change":"2023-10-16T12:52:32.969973+0000","last_active":"2023-10-16T12:52:34.690473+0000","last_peered":"2023-10-16T12:52:34.690473+0000","last_clean":"2023-10-16T12:52:34.690473+0000","last_became_active":"2023-10-16T12:52:32.969264+0000","last_became_peered":"2023-10-16T12:52:32.969264+0000","last_unstale":"2023-10-16T12:52:34.690473+0000","last_undegraded":"2023-10-16T12:52:34.690473+0000","last_fullsized":"2023-10-16T12:52:34.690473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:42.604065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690641+0000","last_change":"2023-10-16T12:52:31.938306+0000","last_active":"2023-10-16T12:52:34.690641+0000","last_peered":"2023-10-16T12:52:34.690641+0000","last_clean":"2023-10-16T12:52:34.690641+0000","last_became_active":"2023-10-16T12:52:31.938003+0000","last_became_peered":"2023-10-16T12:52:31.938003+0000","last_unstale":"2023-10-16T12:52:34.690641+0000","last_undegraded":"2023-10-16T12:52:34.690641+0000","last_fullsized":"2023-10-16T12:52:34.690641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:14.646141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690295+0000","last_change":"2023-10-16T12:52:32.972461+0000","last_active":"2023-10-16T12:52:34.690295+0000","last_peered":"2023-10-16T12:52:34.690295+0000","last_clean":"2023-10-16T12:52:34.690295+0000","last_became_active":"2023-10-16T12:52:32.972151+0000","last_became_peered":"2023-10-16T12:52:32.972151+0000","last_unstale":"2023-10-16T12:52:34.690295+0000","last_undegraded":"2023-10-16T12:52:34.690295+0000","last_fullsized":"2023-10-16T12:52:34.690295+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:21.452261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"127'8","reported_seq":20,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968094+0000","last_change":"2023-10-16T12:52:31.942622+0000","last_active":"2023-10-16T12:52:33.968094+0000","last_peered":"2023-10-16T12:52:33.968094+0000","last_clean":"2023-10-16T12:52:33.968094+0000","last_became_active":"2023-10-16T12:52:31.942275+0000","last_became_peered":"2023-10-16T12:52:31.942275+0000","last_unstale":"2023-10-16T12:52:33.968094+0000","last_undegraded":"2023-10-16T12:52:33.968094+0000","last_fullsized":"2023-10-16T12:52:33.968094+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:00.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972475+0000","last_change":"2023-10-16T12:52:32.969301+0000","last_active":"2023-10-16T12:52:33.972475+0000","last_peered":"2023-10-16T12:52:33.972475+0000","last_clean":"2023-10-16T12:52:33.972475+0000","last_became_active":"2023-10-16T12:52:32.968949+0000","last_became_peered":"2023-10-16T12:52:32.968949+0000","last_unstale":"2023-10-16T12:52:33.972475+0000","last_undegraded":"2023-10-16T12:52:33.972475+0000","last_fullsized":"2023-10-16T12:52:33.972475+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:02.417224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691189+0000","last_change":"2023-10-16T12:52:31.950767+0000","last_active":"2023-10-16T12:52:34.691189+0000","last_peered":"2023-10-16T12:52:34.691189+0000","last_clean":"2023-10-16T12:52:34.691189+0000","last_became_active":"2023-10-16T12:52:31.950478+0000","last_became_peered":"2023-10-16T12:52:31.950478+0000","last_unstale":"2023-10-16T12:52:34.691189+0000","last_undegraded":"2023-10-16T12:52:34.691189+0000","last_fullsized":"2023-10-16T12:52:34.691189+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:03.632195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973389+0000","last_change":"2023-10-16T12:52:32.966261+0000","last_active":"2023-10-16T12:52:33.973389+0000","last_peered":"2023-10-16T12:52:33.973389+0000","last_clean":"2023-10-16T12:52:33.973389+0000","last_became_active":"2023-10-16T12:52:32.965297+0000","last_became_peered":"2023-10-16T12:52:32.965297+0000","last_unstale":"2023-10-16T12:52:33.973389+0000","last_undegraded":"2023-10-16T12:52:33.973389+0000","last_fullsized":"2023-10-16T12:52:33.973389+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:26.855080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971768+0000","last_change":"2023-10-16T12:52:31.942569+0000","last_active":"2023-10-16T12:52:33.971768+0000","last_peered":"2023-10-16T12:52:33.971768+0000","last_clean":"2023-10-16T12:52:33.971768+0000","last_became_active":"2023-10-16T12:52:31.942087+0000","last_became_peered":"2023-10-16T12:52:31.942087+0000","last_unstale":"2023-10-16T12:52:33.971768+0000","last_undegraded":"2023-10-16T12:52:33.971768+0000","last_fullsized":"2023-10-16T12:52:33.971768+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:56.117211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689904+0000","last_change":"2023-10-16T12:52:32.963120+0000","last_active":"2023-10-16T12:52:34.689904+0000","last_peered":"2023-10-16T12:52:34.689904+0000","last_clean":"2023-10-16T12:52:34.689904+0000","last_became_active":"2023-10-16T12:52:32.962826+0000","last_became_peered":"2023-10-16T12:52:32.962826+0000","last_unstale":"2023-10-16T12:52:34.689904+0000","last_undegraded":"2023-10-16T12:52:34.689904+0000","last_fullsized":"2023-10-16T12:52:34.689904+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:49.040358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690311+0000","last_change":"2023-10-16T12:52:31.932569+0000","last_active":"2023-10-16T12:52:34.690311+0000","last_peered":"2023-10-16T12:52:34.690311+0000","last_clean":"2023-10-16T12:52:34.690311+0000","last_became_active":"2023-10-16T12:52:31.932229+0000","last_became_peered":"2023-10-16T12:52:31.932229+0000","last_unstale":"2023-10-16T12:52:34.690311+0000","last_undegraded":"2023-10-16T12:52:34.690311+0000","last_fullsized":"2023-10-16T12:52:34.690311+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:53.510824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691317+0000","last_change":"2023-10-16T12:52:32.966941+0000","last_active":"2023-10-16T12:52:34.691317+0000","last_peered":"2023-10-16T12:52:34.691317+0000","last_clean":"2023-10-16T12:52:34.691317+0000","last_became_active":"2023-10-16T12:52:32.966672+0000","last_became_peered":"2023-10-16T12:52:32.966672+0000","last_unstale":"2023-10-16T12:52:34.691317+0000","last_undegraded":"2023-10-16T12:52:34.691317+0000","last_fullsized":"2023-10-16T12:52:34.691317+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:09.227980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691256+0000","last_change":"2023-10-16T12:52:31.948545+0000","last_active":"2023-10-16T12:52:34.691256+0000","last_peered":"2023-10-16T12:52:34.691256+0000","last_clean":"2023-10-16T12:52:34.691256+0000","last_became_active":"2023-10-16T12:52:31.947915+0000","last_became_peered":"2023-10-16T12:52:31.947915+0000","last_unstale":"2023-10-16T12:52:34.691256+0000","last_undegraded":"2023-10-16T12:52:34.691256+0000","last_fullsized":"2023-10-16T12:52:34.691256+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:05.355539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691849+0000","last_change":"2023-10-16T12:52:32.966143+0000","last_active":"2023-10-16T12:52:34.691849+0000","last_peered":"2023-10-16T12:52:34.691849+0000","last_clean":"2023-10-16T12:52:34.691849+0000","last_became_active":"2023-10-16T12:52:32.965876+0000","last_became_peered":"2023-10-16T12:52:32.965876+0000","last_unstale":"2023-10-16T12:52:34.691849+0000","last_undegraded":"2023-10-16T12:52:34.691849+0000","last_fullsized":"2023-10-16T12:52:34.691849+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:09.696947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973396+0000","last_change":"2023-10-16T12:52:31.939737+0000","last_active":"2023-10-16T12:52:33.973396+0000","last_peered":"2023-10-16T12:52:33.973396+0000","last_clean":"2023-10-16T12:52:33.973396+0000","last_became_active":"2023-10-16T12:52:31.938728+0000","last_became_peered":"2023-10-16T12:52:31.938728+0000","last_unstale":"2023-10-16T12:52:33.973396+0000","last_undegraded":"2023-10-16T12:52:33.973396+0000","last_fullsized":"2023-10-16T12:52:33.973396+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:08:30.161261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690588+0000","last_change":"2023-10-16T12:52:32.976105+0000","last_active":"2023-10-16T12:52:34.690588+0000","last_peered":"2023-10-16T12:52:34.690588+0000","last_clean":"2023-10-16T12:52:34.690588+0000","last_became_active":"2023-10-16T12:52:32.975799+0000","last_became_peered":"2023-10-16T12:52:32.975799+0000","last_unstale":"2023-10-16T12:52:34.690588+0000","last_undegraded":"2023-10-16T12:52:34.690588+0000","last_fullsized":"2023-10-16T12:52:34.690588+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:42.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973161+0000","last_change":"2023-10-16T12:52:31.939521+0000","last_active":"2023-10-16T12:52:33.973161+0000","last_peered":"2023-10-16T12:52:33.973161+0000","last_clean":"2023-10-16T12:52:33.973161+0000","last_became_active":"2023-10-16T12:52:31.939185+0000","last_became_peered":"2023-10-16T12:52:31.939185+0000","last_unstale":"2023-10-16T12:52:33.973161+0000","last_undegraded":"2023-10-16T12:52:33.973161+0000","last_fullsized":"2023-10-16T12:52:33.973161+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:16.678621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.970918+0000","last_change":"2023-10-16T12:52:32.974264+0000","last_active":"2023-10-16T12:52:33.970918+0000","last_peered":"2023-10-16T12:52:33.970918+0000","last_clean":"2023-10-16T12:52:33.970918+0000","last_became_active":"2023-10-16T12:52:32.973914+0000","last_became_peered":"2023-10-16T12:52:32.973914+0000","last_unstale":"2023-10-16T12:52:33.970918+0000","last_undegraded":"2023-10-16T12:52:33.970918+0000","last_fullsized":"2023-10-16T12:52:33.970918+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:15.734992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689653+0000","last_change":"2023-10-16T12:52:31.939466+0000","last_active":"2023-10-16T12:52:34.689653+0000","last_peered":"2023-10-16T12:52:34.689653+0000","last_clean":"2023-10-16T12:52:34.689653+0000","last_became_active":"2023-10-16T12:52:31.938997+0000","last_became_peered":"2023-10-16T12:52:31.938997+0000","last_unstale":"2023-10-16T12:52:34.689653+0000","last_undegraded":"2023-10-16T12:52:34.689653+0000","last_fullsized":"2023-10-16T12:52:34.689653+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:38.030185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691263+0000","last_change":"2023-10-16T12:52:32.976260+0000","last_active":"2023-10-16T12:52:34.691263+0000","last_peered":"2023-10-16T12:52:34.691263+0000","last_clean":"2023-10-16T12:52:34.691263+0000","last_became_active":"2023-10-16T12:52:32.972361+0000","last_became_peered":"2023-10-16T12:52:32.972361+0000","last_unstale":"2023-10-16T12:52:34.691263+0000","last_undegraded":"2023-10-16T12:52:34.691263+0000","last_fullsized":"2023-10-16T12:52:34.691263+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:06.496645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973984+0000","last_change":"2023-10-16T12:52:31.936338+0000","last_active":"2023-10-16T12:52:33.973984+0000","last_peered":"2023-10-16T12:52:33.973984+0000","last_clean":"2023-10-16T12:52:33.973984+0000","last_became_active":"2023-10-16T12:52:31.935886+0000","last_became_peered":"2023-10-16T12:52:31.935886+0000","last_unstale":"2023-10-16T12:52:33.973984+0000","last_undegraded":"2023-10-16T12:52:33.973984+0000","last_fullsized":"2023-10-16T12:52:33.973984+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:17.236374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969139+0000","last_change":"2023-10-16T12:52:32.976047+0000","last_active":"2023-10-16T12:52:33.969139+0000","last_peered":"2023-10-16T12:52:33.969139+0000","last_clean":"2023-10-16T12:52:33.969139+0000","last_became_active":"2023-10-16T12:52:32.972474+0000","last_became_peered":"2023-10-16T12:52:32.972474+0000","last_unstale":"2023-10-16T12:52:33.969139+0000","last_undegraded":"2023-10-16T12:52:33.969139+0000","last_fullsized":"2023-10-16T12:52:33.969139+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:49:00.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691392+0000","last_change":"2023-10-16T12:52:31.950985+0000","last_active":"2023-10-16T12:52:34.691392+0000","last_peered":"2023-10-16T12:52:34.691392+0000","last_clean":"2023-10-16T12:52:34.691392+0000","last_became_active":"2023-10-16T12:52:31.950666+0000","last_became_peered":"2023-10-16T12:52:31.950666+0000","last_unstale":"2023-10-16T12:52:34.691392+0000","last_undegraded":"2023-10-16T12:52:34.691392+0000","last_fullsized":"2023-10-16T12:52:34.691392+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:28.836570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968854+0000","last_change":"2023-10-16T12:52:32.978260+0000","last_active":"2023-10-16T12:52:33.968854+0000","last_peered":"2023-10-16T12:52:33.968854+0000","last_clean":"2023-10-16T12:52:33.968854+0000","last_became_active":"2023-10-16T12:52:32.977948+0000","last_became_peered":"2023-10-16T12:52:32.977948+0000","last_unstale":"2023-10-16T12:52:33.968854+0000","last_undegraded":"2023-10-16T12:52:33.968854+0000","last_fullsized":"2023-10-16T12:52:33.968854+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:24.792272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690470+0000","last_change":"2023-10-16T12:52:31.929524+0000","last_active":"2023-10-16T12:52:34.690470+0000","last_peered":"2023-10-16T12:52:34.690470+0000","last_clean":"2023-10-16T12:52:34.690470+0000","last_became_active":"2023-10-16T12:52:31.929153+0000","last_became_peered":"2023-10-16T12:52:31.929153+0000","last_unstale":"2023-10-16T12:52:34.690470+0000","last_undegraded":"2023-10-16T12:52:34.690470+0000","last_fullsized":"2023-10-16T12:52:34.690470+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:40.033574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971497+0000","last_change":"2023-10-16T12:52:32.971626+0000","last_active":"2023-10-16T12:52:33.971497+0000","last_peered":"2023-10-16T12:52:33.971497+0000","last_clean":"2023-10-16T12:52:33.971497+0000","last_became_active":"2023-10-16T12:52:32.971335+0000","last_became_peered":"2023-10-16T12:52:32.971335+0000","last_unstale":"2023-10-16T12:52:33.971497+0000","last_undegraded":"2023-10-16T12:52:33.971497+0000","last_fullsized":"2023-10-16T12:52:33.971497+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:33.569175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971950+0000","last_change":"2023-10-16T12:52:31.944196+0000","last_active":"2023-10-16T12:52:33.971950+0000","last_peered":"2023-10-16T12:52:33.971950+0000","last_clean":"2023-10-16T12:52:33.971950+0000","last_became_active":"2023-10-16T12:52:31.943052+0000","last_became_peered":"2023-10-16T12:52:31.943052+0000","last_unstale":"2023-10-16T12:52:33.971950+0000","last_undegraded":"2023-10-16T12:52:33.971950+0000","last_fullsized":"2023-10-16T12:52:33.971950+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:36.799355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691420+0000","last_change":"2023-10-16T12:52:32.973396+0000","last_active":"2023-10-16T12:52:34.691420+0000","last_peered":"2023-10-16T12:52:34.691420+0000","last_clean":"2023-10-16T12:52:34.691420+0000","last_became_active":"2023-10-16T12:52:32.973024+0000","last_became_peered":"2023-10-16T12:52:32.973024+0000","last_unstale":"2023-10-16T12:52:34.691420+0000","last_undegraded":"2023-10-16T12:52:34.691420+0000","last_fullsized":"2023-10-16T12:52:34.691420+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:06.496645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691493+0000","last_change":"2023-10-16T12:52:31.945999+0000","last_active":"2023-10-16T12:52:34.691493+0000","last_peered":"2023-10-16T12:52:34.691493+0000","last_clean":"2023-10-16T12:52:34.691493+0000","last_became_active":"2023-10-16T12:52:31.943208+0000","last_became_peered":"2023-10-16T12:52:31.943208+0000","last_unstale":"2023-10-16T12:52:34.691493+0000","last_undegraded":"2023-10-16T12:52:34.691493+0000","last_fullsized":"2023-10-16T12:52:34.691493+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:39.007201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691118+0000","last_change":"2023-10-16T12:52:32.961721+0000","last_active":"2023-10-16T12:52:34.691118+0000","last_peered":"2023-10-16T12:52:34.691118+0000","last_clean":"2023-10-16T12:52:34.691118+0000","last_became_active":"2023-10-16T12:52:32.960328+0000","last_became_peered":"2023-10-16T12:52:32.960328+0000","last_unstale":"2023-10-16T12:52:34.691118+0000","last_undegraded":"2023-10-16T12:52:34.691118+0000","last_fullsized":"2023-10-16T12:52:34.691118+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:20.915196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.970741+0000","last_change":"2023-10-16T12:52:31.935888+0000","last_active":"2023-10-16T12:52:33.970741+0000","last_peered":"2023-10-16T12:52:33.970741+0000","last_clean":"2023-10-16T12:52:33.970741+0000","last_became_active":"2023-10-16T12:52:31.935571+0000","last_became_peered":"2023-10-16T12:52:31.935571+0000","last_unstale":"2023-10-16T12:52:33.970741+0000","last_undegraded":"2023-10-16T12:52:33.970741+0000","last_fullsized":"2023-10-16T12:52:33.970741+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:01.015817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690761+0000","last_change":"2023-10-16T12:52:32.964954+0000","last_active":"2023-10-16T12:52:34.690761+0000","last_peered":"2023-10-16T12:52:34.690761+0000","last_clean":"2023-10-16T12:52:34.690761+0000","last_became_active":"2023-10-16T12:52:32.964399+0000","last_became_peered":"2023-10-16T12:52:32.964399+0000","last_unstale":"2023-10-16T12:52:34.690761+0000","last_undegraded":"2023-10-16T12:52:34.690761+0000","last_fullsized":"2023-10-16T12:52:34.690761+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:20.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576946+0000","last_change":"2023-10-16T12:52:31.947943+0000","last_active":"2023-10-16T12:52:34.576946+0000","last_peered":"2023-10-16T12:52:34.576946+0000","last_clean":"2023-10-16T12:52:34.576946+0000","last_became_active":"2023-10-16T12:52:31.947034+0000","last_became_peered":"2023-10-16T12:52:31.947034+0000","last_unstale":"2023-10-16T12:52:34.576946+0000","last_undegraded":"2023-10-16T12:52:34.576946+0000","last_fullsized":"2023-10-16T12:52:34.576946+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:04.855667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971277+0000","last_change":"2023-10-16T12:52:32.973703+0000","last_active":"2023-10-16T12:52:33.971277+0000","last_peered":"2023-10-16T12:52:33.971277+0000","last_clean":"2023-10-16T12:52:33.971277+0000","last_became_active":"2023-10-16T12:52:32.972589+0000","last_became_peered":"2023-10-16T12:52:32.972589+0000","last_unstale":"2023-10-16T12:52:33.971277+0000","last_undegraded":"2023-10-16T12:52:33.971277+0000","last_fullsized":"2023-10-16T12:52:33.971277+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:08:00.066774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690828+0000","last_change":"2023-10-16T12:52:31.937242+0000","last_active":"2023-10-16T12:52:34.690828+0000","last_peered":"2023-10-16T12:52:34.690828+0000","last_clean":"2023-10-16T12:52:34.690828+0000","last_became_active":"2023-10-16T12:52:31.936928+0000","last_became_peered":"2023-10-16T12:52:31.936928+0000","last_unstale":"2023-10-16T12:52:34.690828+0000","last_undegraded":"2023-10-16T12:52:34.690828+0000","last_fullsized":"2023-10-16T12:52:34.690828+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:23.992548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692501+0000","last_change":"2023-10-16T12:52:32.972474+0000","last_active":"2023-10-16T12:52:34.692501+0000","last_peered":"2023-10-16T12:52:34.692501+0000","last_clean":"2023-10-16T12:52:34.692501+0000","last_became_active":"2023-10-16T12:52:32.972141+0000","last_became_peered":"2023-10-16T12:52:32.972141+0000","last_unstale":"2023-10-16T12:52:34.692501+0000","last_undegraded":"2023-10-16T12:52:34.692501+0000","last_fullsized":"2023-10-16T12:52:34.692501+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:03.559443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689405+0000","last_change":"2023-10-16T12:52:31.948754+0000","last_active":"2023-10-16T12:52:34.689405+0000","last_peered":"2023-10-16T12:52:34.689405+0000","last_clean":"2023-10-16T12:52:34.689405+0000","last_became_active":"2023-10-16T12:52:31.948465+0000","last_became_peered":"2023-10-16T12:52:31.948465+0000","last_unstale":"2023-10-16T12:52:34.689405+0000","last_undegraded":"2023-10-16T12:52:34.689405+0000","last_fullsized":"2023-10-16T12:52:34.689405+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:53.724134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"117'208","reported_seq":589,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:52:32.945566+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:52:32.945566+0000","last_peered":"2023-10-16T12:52:32.945566+0000","last_clean":"2023-10-16T12:52:32.945566+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:52:32.945566+0000","last_undegraded":"2023-10-16T12:52:32.945566+0000","last_fullsized":"2023-10-16T12:52:32.945566+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1338117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2569999999999999,"15min":1.3029999999999999},"min":{"1min":0.68799999999999994,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.835},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3080000000000001,"15min":1.3560000000000001},"min":{"1min":0.81000000000000005,"5min":0.69399999999999995,"15min":0.63200000000000001},"max":{"1min":1.613,"5min":1.7729999999999999,"15min":1.8899999999999999},"last":1.46}]},{"osd":1,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.3440000000000001,"15min":1.2649999999999999},"min":{"1min":0.90600000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":19.158999999999999,"5min":19.158999999999999,"15min":19.158999999999999},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2070000000000001,"15min":1.2250000000000001},"min":{"1min":0.745,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":5.5490000000000004,"5min":5.5490000000000004,"15min":5.5490000000000004},"last":1.6739999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1539999999999999,"15min":1.2},"min":{"1min":0.83099999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.272,"15min":1.3180000000000001},"min":{"1min":0.85599999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7170000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.595}]},{"osd":3,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.339,"15min":1.3109999999999999},"min":{"1min":0.90800000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.2759999999999998,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.3839999999999999,"15min":1.403},"min":{"1min":1.0089999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.4369999999999998,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.54}]},{"osd":4,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1799999999999999,"15min":1.1899999999999999},"min":{"1min":0.77200000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.282,"5min":1.3460000000000001,"15min":1.335},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.831,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.345}]},{"osd":5,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.3480000000000001,"15min":1.339},"min":{"1min":0.875,"5min":0.624,"15min":0.624},"max":{"1min":5.0140000000000002,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.512,"15min":1.4750000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.1720000000000002,"5min":2.411,"15min":2.411},"last":1.631}]},{"osd":6,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5940000000000001,"15min":1.5680000000000001},"min":{"1min":0.83299999999999996,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.141,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.478,"15min":1.452},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3170000000000002,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.292}]}]},{"osd":6,"up_from":14,"seq":60129542373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1585152,"data_stored":754589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2210000000000001,"15min":1.2789999999999999},"min":{"1min":0.70899999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.881,"5min":1.952,"15min":3.7450000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2769999999999999,"15min":1.3320000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7470000000000001,"5min":2.0270000000000001,"15min":3.819},"last":1.482}]},{"osd":1,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1819999999999999,"15min":1.1899999999999999},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.216,"15min":1.22},"min":{"1min":0.90400000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":3.8519999999999999},"last":0.96699999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.1950000000000001,"15min":1.2190000000000001},"min":{"1min":0.94499999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.752,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.234,"15min":1.2729999999999999},"min":{"1min":0.96599999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":3.9009999999999998},"last":0.93600000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.302,"15min":1.3280000000000001},"min":{"1min":0.86299999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.7190000000000001,"5min":1.9399999999999999,"15min":4.2380000000000004},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3180000000000001,"15min":1.351},"min":{"1min":0.92300000000000004,"5min":0.82199999999999995,"15min":0.69599999999999995},"max":{"1min":1.754,"5min":1.806,"15min":4.4269999999999996},"last":1.6319999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3859999999999999,"15min":1.3420000000000001},"min":{"1min":0.94599999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.829,"5min":2.0609999999999999,"15min":2.3029999999999999},"last":1.415},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.47,"15min":1.47},"min":{"1min":0.89600000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.7090000000000001,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.784}]},{"osd":5,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.3400000000000001,"15min":1.3129999999999999},"min":{"1min":0.997,"5min":0.755,"15min":0.755},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.0910000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4770000000000001,"15min":1.4470000000000001},"min":{"1min":1.147,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7869999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.5309999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4610000000000001,"15min":1.446},"min":{"1min":1.0229999999999999,"5min":0.878,"15min":0.878},"max":{"1min":1.841,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.4910000000000001,"15min":1.466},"min":{"1min":1.244,"5min":1.0640000000000001,"15min":0.96399999999999997},"max":{"1min":2.052,"5min":2.1230000000000002,"15min":2.2610000000000001},"last":1.742}]}]},{"osd":5,"up_from":14,"seq":60129542374,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":754007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1000000000000001,"15min":1.1579999999999999},"min":{"1min":0.79900000000000004,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.4870000000000001,"5min":1.8480000000000001,"15min":2.2690000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1759999999999999,"15min":1.1779999999999999},"min":{"1min":0.72499999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.6060000000000001,"5min":1.6579999999999999,"15min":1.7689999999999999},"last":1.365}]},{"osd":1,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.272,"15min":1.26},"min":{"1min":0.89100000000000001,"5min":0.56699999999999995,"15min":0.41299999999999998},"max":{"1min":1.7829999999999999,"5min":1.8080000000000001,"15min":1.8089999999999999},"last":1.079},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1859999999999999,"15min":1.2190000000000001},"min":{"1min":0.89000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.7889999999999999},"last":0.95599999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.274,"15min":1.2929999999999999},"min":{"1min":0.79300000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.9710000000000001,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.21,"15min":1.266},"min":{"1min":0.94399999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.2370000000000001},"last":1.0169999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2969999999999999,"15min":1.3089999999999999},"min":{"1min":0.96699999999999997,"5min":0.73099999999999998,"15min":0.66000000000000003},"max":{"1min":1.5680000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.232},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.381,"15min":1.3879999999999999},"min":{"1min":1.1120000000000001,"5min":0.62,"15min":0.62},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.2160000000000002},"last":1.6419999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.1970000000000001,"15min":1.2050000000000001},"min":{"1min":0.88600000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.589},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3580000000000001,"15min":1.321},"min":{"1min":0.78900000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.9510000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.5509999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3500000000000001,"15min":1.3340000000000001},"min":{"1min":0.95699999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":1.2170000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.415}]},{"osd":7,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.448,"15min":1.4470000000000001},"min":{"1min":1.145,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.5249999999999999,"15min":1.512},"min":{"1min":1.167,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.772}]}]},{"osd":4,"up_from":14,"seq":60129542374,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59108,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312732,"statfs":{"total":96636764160,"available":96576237568,"internally_reserved":0,"allocated":1597440,"data_stored":755673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1259999999999999,"15min":1.1919999999999999},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.696,"5min":1.766,"15min":2.2200000000000002},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.105,"15min":1.1539999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.1220000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1240000000000001,"15min":1.177},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.732,"5min":1.7410000000000001,"15min":1.8400000000000001},"last":1.171},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.167,"15min":1.24},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.4770000000000001,"5min":1.724,"15min":1.9390000000000001},"last":0.78300000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.234,"15min":1.252},"min":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.651,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.194,"5min":1.2430000000000001,"15min":1.3129999999999999},"min":{"1min":0.76300000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.571,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.198}]},{"osd":3,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.2230000000000001,"15min":1.2649999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.3089999999999999,"15min":1.3580000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7669999999999999,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.0660000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3400000000000001,"15min":1.2989999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.669,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.292,"15min":1.2849999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.851,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.262}]},{"osd":6,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3700000000000001,"15min":1.3680000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.841,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4590000000000001,"15min":1.472},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.744,"5min":2.165,"15min":2.165},"last":1.603}]},{"osd":7,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4490000000000001,"15min":1.466},"min":{"1min":0.83299999999999996,"5min":0.67600000000000005,"15min":0.60399999999999998},"max":{"1min":1.8340000000000001,"5min":2.109,"15min":2.109},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.573,"15min":1.581},"min":{"1min":0.80900000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9379999999999999,"5min":2.423,"15min":2.423},"last":1.655}]}]},{"osd":3,"up_from":14,"seq":60129542374,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":755571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.93100000000000005,"15min":0.96699999999999997},"min":{"1min":0.71999999999999997,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":1.4430000000000001,"5min":1.5649999999999999,"15min":2.0419999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1060000000000001,"15min":1.171},"min":{"1min":0.93799999999999994,"5min":0.44900000000000001,"15min":0.34799999999999998},"max":{"1min":1.5189999999999999,"5min":1.5800000000000001,"15min":2.2749999999999999},"last":0.81399999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.98699999999999999,"15min":1.0209999999999999},"min":{"1min":0.81799999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.758},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.0269999999999999,"15min":1.0780000000000001},"min":{"1min":0.82099999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.784,"5min":1.784,"15min":2.4580000000000002},"last":0.99099999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.002,"15min":1.0600000000000001},"min":{"1min":0.76600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.679,"5min":1.679,"15min":1.9510000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.546,"5min":1.2689999999999999,"15min":1.353},"min":{"1min":1.1319999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.139}]},{"osd":4,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.4119999999999999,"15min":1.4750000000000001},"min":{"1min":0.98699999999999999,"5min":0.78700000000000003,"15min":0.77600000000000002},"max":{"1min":1.726,"5min":1.9239999999999999,"15min":1.956},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.369,"15min":1.4059999999999999},"min":{"1min":1.0109999999999999,"5min":0.80500000000000005,"15min":0.64800000000000002},"max":{"1min":1.8220000000000001,"5min":1.8939999999999999,"15min":2.0099999999999998},"last":1.2470000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.3280000000000001,"15min":1.3779999999999999},"min":{"1min":1.056,"5min":0.70299999999999996,"15min":0.65900000000000003},"max":{"1min":1.893,"5min":2.028,"15min":2.0539999999999998},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4279999999999999,"15min":1.466},"min":{"1min":1.1160000000000001,"5min":0.877,"15min":0.877},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.3540000000000001},"last":1.1910000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.405,"15min":1.4390000000000001},"min":{"1min":1.0860000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.998,"5min":1.998,"15min":2.3220000000000001},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.3779999999999999,"15min":1.4159999999999999},"min":{"1min":1.1200000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2,"5min":2.093,"15min":2.3740000000000001},"last":1.042}]},{"osd":7,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.5149999999999999,"15min":1.538},"min":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.157,"5min":2.157,"15min":2.431},"last":1.069},{"interface":"front","average":{"1min":1.651,"5min":1.536,"15min":1.6000000000000001},"min":{"1min":1.3380000000000001,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.552},"last":1.4139999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28556,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343284,"statfs":{"total":96636764160,"available":96607522816,"internally_reserved":0,"allocated":1568768,"data_stored":747671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.877,"15min":0.92200000000000004},"min":{"1min":0.55700000000000005,"5min":0.315,"15min":0.315},"max":{"1min":1.508,"5min":1.508,"15min":1.7889999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.034,"15min":1.101},"min":{"1min":0.70399999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.643,"5min":1.746,"15min":2.085},"last":0.67600000000000005}]},{"osd":1,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.72299999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6599999999999999,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.1479999999999999,"15min":1.226},"min":{"1min":0.96299999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.0510000000000002,"5min":2.347,"15min":2.4289999999999998},"last":1.1100000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.0229999999999999,"15min":1.0820000000000001},"min":{"1min":0.76600000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5649999999999999,"5min":1.5860000000000001,"15min":1.859},"last":1.3},{"interface":"front","average":{"1min":1.405,"5min":1.2729999999999999,"15min":1.3240000000000001},"min":{"1min":0.89100000000000001,"5min":0.434,"15min":0.434},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":1.2509999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.411,"15min":1.46},"min":{"1min":1.1100000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.105,"5min":2.431,"15min":2.431},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.3500000000000001,"15min":1.4319999999999999},"min":{"1min":0.84699999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6459999999999999,"5min":2.0670000000000002,"15min":2.2530000000000001},"last":1.468}]},{"osd":5,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":1.1559999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.9350000000000001,"5min":2.117,"15min":2.3250000000000002},"last":1.363},{"interface":"front","average":{"1min":1.446,"5min":1.3440000000000001,"15min":1.4199999999999999},"min":{"1min":0.995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.8080000000000001,"5min":1.923,"15min":2.4620000000000002},"last":1.504}]},{"osd":6,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.3979999999999999,"15min":1.4470000000000001},"min":{"1min":1.026,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.9950000000000001,"5min":2.181,"15min":2.5129999999999999},"last":1.421},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.508,"15min":1.5620000000000001},"min":{"1min":1.2669999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.298,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.167}]},{"osd":7,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.522,"15min":1.6080000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.931,"5min":2.202,"15min":2.5339999999999998},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.4790000000000001,"15min":1.544},"min":{"1min":1.101,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.573},"last":1.5629999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542372,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59088,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312752,"statfs":{"total":96636764160,"available":96576258048,"internally_reserved":0,"allocated":1576960,"data_stored":753937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91700000000000004,"15min":0.91000000000000003},"min":{"1min":0.439,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.345,"5min":1.6359999999999999,"15min":2.0840000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.997,"15min":1.0760000000000001},"min":{"1min":0.47399999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.5669999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.2829999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.92900000000000005,"15min":0.98299999999999998},"min":{"1min":0.56100000000000005,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":1.694,"5min":1.694,"15min":2.262},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.135,"15min":1.1870000000000001},"min":{"1min":0.66900000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.754,"5min":1.9910000000000001,"15min":2.2210000000000001},"last":1.6639999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2350000000000001,"15min":1.302},"min":{"1min":0.63600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.151,"15min":1.2130000000000001},"min":{"1min":0.59799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.7230000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.712}]},{"osd":4,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2569999999999999,"15min":1.302},"min":{"1min":0.71399999999999997,"5min":0.59599999999999997,"15min":0.54600000000000004},"max":{"1min":1.7230000000000001,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.429,"15min":1.4410000000000001},"min":{"1min":0.92700000000000005,"5min":0.67900000000000005,"15min":0.60599999999999998},"max":{"1min":2.214,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.4550000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.365,"15min":1.423},"min":{"1min":1.1060000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.6679999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.9410000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":0.999,"5min":0.753,"15min":0.70999999999999996},"max":{"1min":2.0089999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.2190000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5009999999999999,"15min":1.528},"min":{"1min":0.878,"5min":0.72099999999999997,"15min":0.68999999999999995},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":1.861},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4770000000000001,"15min":1.514},"min":{"1min":0.92500000000000004,"5min":0.71399999999999997,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.798}]},{"osd":7,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5669999999999999,"15min":1.6180000000000001},"min":{"1min":0.97899999999999998,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.9570000000000001,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5369999999999999,"15min":1.5609999999999999},"min":{"1min":1.119,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.1280000000000001,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.371}]}]},{"osd":0,"up_from":14,"seq":60129542373,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37464,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334376,"statfs":{"total":96636764160,"available":96598401024,"internally_reserved":0,"allocated":2174976,"data_stored":1344887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.88700000000000001},"min":{"1min":0.65400000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.4430000000000001,"5min":1.655,"15min":2.5720000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.222,"5min":1.0549999999999999,"15min":1.0960000000000001},"min":{"1min":0.77300000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.708,"5min":1.8979999999999999,"15min":2.1800000000000002},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.101,"15min":1.135},"min":{"1min":0.88,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7250000000000001,"5min":1.9339999999999999,"15min":2.7650000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.109,"15min":1.1699999999999999},"min":{"1min":0.91400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.623,"5min":1.788,"15min":1.952},"last":1.1719999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.0049999999999999,"15min":1.0900000000000001},"min":{"1min":0.78200000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.585,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.216,"15min":1.2929999999999999},"min":{"1min":0.91500000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.7110000000000001,"5min":1.9399999999999999,"15min":3.6579999999999999},"last":1.073}]},{"osd":4,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.272,"15min":1.337},"min":{"1min":0.93000000000000005,"5min":0.71499999999999997,"15min":0.56499999999999995},"max":{"1min":1.6839999999999999,"5min":1.8779999999999999,"15min":2.6920000000000002},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3879999999999999,"15min":1.4399999999999999},"min":{"1min":1.177,"5min":0.625,"15min":0.59699999999999998},"max":{"1min":1.6870000000000001,"5min":1.966,"15min":3.544},"last":1.444}]},{"osd":5,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.383,"15min":1.4279999999999999},"min":{"1min":1.232,"5min":0.74299999999999999,"15min":0.72499999999999998},"max":{"1min":1.9630000000000001,"5min":2.024,"15min":2.2440000000000002},"last":1.393},{"interface":"front","average":{"1min":1.397,"5min":1.353,"15min":1.419},"min":{"1min":1.1799999999999999,"5min":0.65400000000000003,"15min":0.58999999999999997},"max":{"1min":1.714,"5min":1.978,"15min":2.8610000000000002},"last":1.351}]},{"osd":6,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.482,"15min":1.5349999999999999},"min":{"1min":1.2230000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9239999999999999,"5min":2.125,"15min":3.7890000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.423,"15min":1.5169999999999999},"min":{"1min":1.1779999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.8839999999999999,"5min":2.0299999999999998,"15min":2.363},"last":1.4610000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.5469999999999999,"15min":1.6279999999999999},"min":{"1min":1.4279999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0249999999999999,"5min":2.1080000000000001,"15min":2.4729999999999999},"last":1.788},{"interface":"front","average":{"1min":1.496,"5min":1.4890000000000001,"15min":1.5569999999999999},"min":{"1min":1.157,"5min":0.79800000000000004,"15min":0.755},"max":{"1min":1.851,"5min":2.3290000000000002,"15min":3.1989999999999998},"last":1.7569999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:52:37.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:52:37.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.735+0000 7f806937a640 1 Processor -- start 2023-10-16T12:52:37.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.735+0000 7f806937a640 1 -- start start 2023-10-16T12:52:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f806937a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80640c9060 0x7f806406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f806937a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f806406c9e0 0x7f806406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f806937a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80640ca560 con 0x7f80640c9440 2023-10-16T12:52:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f806937a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80640c9950 con 0x7f806406c9e0 2023-10-16T12:52:37.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f806937a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80640c9c30 con 0x7f80640c9060 2023-10-16T12:52:37.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f80637fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f806406c9e0 0x7f806406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f8068b79640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f80640c9440 0x7f8064164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46790/0 (socket says 172.21.15.60:46790) 2023-10-16T12:52:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.737+0000 7f8068b79640 1 -- 172.21.15.60:0/1694423973 learned_addr learned my addr 172.21.15.60:0/1694423973 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.738+0000 7f8063fff640 1 --2- 172.21.15.60:0/1694423973 >> v2:172.21.15.144:3300/0 conn(0x7f80640c9060 0x7f806406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.738+0000 7f80637fe640 1 -- 172.21.15.60:0/1694423973 >> v2:172.21.15.144:3300/0 conn(0x7f80640c9060 msgr2=0x7f806406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:37.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.738+0000 7f80637fe640 1 --2- 172.21.15.60:0/1694423973 >> v2:172.21.15.144:3300/0 conn(0x7f80640c9060 0x7f806406f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.738+0000 7f80637fe640 1 -- 172.21.15.60:0/1694423973 >> v1:172.21.15.60:6789/0 conn(0x7f80640c9440 legacy=0x7f8064164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.738+0000 7f80637fe640 1 -- 172.21.15.60:0/1694423973 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640c9f10 con 0x7f806406c9e0 2023-10-16T12:52:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.738+0000 7f8063fff640 1 --2- 172.21.15.60:0/1694423973 >> v2:172.21.15.144:3300/0 conn(0x7f80640c9060 0x7f806406f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.740+0000 7f80637fe640 1 --2- 172.21.15.60:0/1694423973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806406c9e0 0x7f806406f980 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f806406d490 tx=0x7f805802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1b3966a4416bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.740+0000 7f8062ffd640 1 -- 172.21.15.60:0/1694423973 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058031830 con 0x7f806406c9e0 2023-10-16T12:52:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.740+0000 7f8062ffd640 1 -- 172.21.15.60:0/1694423973 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058031990 con 0x7f806406c9e0 2023-10-16T12:52:37.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.740+0000 7f8062ffd640 1 -- 172.21.15.60:0/1694423973 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058031d20 con 0x7f806406c9e0 2023-10-16T12:52:37.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.740+0000 7f806937a640 1 -- 172.21.15.60:0/1694423973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806406c9e0 msgr2=0x7f806406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.740+0000 7f806937a640 1 --2- 172.21.15.60:0/1694423973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806406c9e0 0x7f806406f980 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f806406d490 tx=0x7f805802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.741+0000 7f806937a640 1 -- 172.21.15.60:0/1694423973 shutdown_connections 2023-10-16T12:52:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.741+0000 7f806937a640 1 --2- 172.21.15.60:0/1694423973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806406c9e0 0x7f806406f980 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.741+0000 7f806937a640 1 --2- 172.21.15.60:0/1694423973 >> v2:172.21.15.144:3300/0 conn(0x7f80640c9060 0x7f806406f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.741+0000 7f806937a640 1 -- 172.21.15.60:0/1694423973 >> 172.21.15.60:0/1694423973 conn(0x7f80640ce6a0 msgr2=0x7f80640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:37.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.741+0000 7f806937a640 1 -- 172.21.15.60:0/1694423973 shutdown_connections 2023-10-16T12:52:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.741+0000 7f806937a640 1 -- 172.21.15.60:0/1694423973 wait complete. 2023-10-16T12:52:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.742+0000 7f806937a640 1 Processor -- start 2023-10-16T12:52:37.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.743+0000 7f806937a640 1 -- start start 2023-10-16T12:52:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.743+0000 7f806937a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f806406c9e0 0x7f8064163820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f806937a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80640c9060 0x7f806415d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f806937a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 0x7f80640c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f806937a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806415ded0 con 0x7f80640c9440 2023-10-16T12:52:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f806937a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80640c5cb0 con 0x7f80640c9060 2023-10-16T12:52:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f806937a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80640c5f50 con 0x7f806406c9e0 2023-10-16T12:52:37.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 0x7f80640c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 0x7f80640c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38178/0 (socket says 172.21.15.60:38178) 2023-10-16T12:52:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 -- 172.21.15.60:0/3312370351 learned_addr learned my addr 172.21.15.60:0/3312370351 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f80637fe640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80640c9060 0x7f806415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8063fff640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f806406c9e0 0x7f8064163820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 -- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f806406c9e0 msgr2=0x7f8064163820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f806406c9e0 0x7f8064163820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 -- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80640c9060 msgr2=0x7f806415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80640c9060 0x7f806415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f8068b79640 1 -- 172.21.15.60:0/3312370351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8058009400 con 0x7f80640c9440 2023-10-16T12:52:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.744+0000 7f80637fe640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80640c9060 0x7f806415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.745+0000 7f8068b79640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 0x7f80640c5770 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f805c00c340 tx=0x7f805c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.745+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c005ce0 con 0x7f80640c9440 2023-10-16T12:52:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.745+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c005e40 con 0x7f80640c9440 2023-10-16T12:52:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.745+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0072d0 con 0x7f80640c9440 2023-10-16T12:52:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.745+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80640c6230 con 0x7f80640c9440 2023-10-16T12:52:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.745+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80640c67f0 con 0x7f80640c9440 2023-10-16T12:52:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.746+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806404e9b0 con 0x7f80640c9440 2023-10-16T12:52:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.747+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f805c0074f0 con 0x7f80640c9440 2023-10-16T12:52:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.748+0000 7f8060ff9640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f805006af90 0x7f805006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.748+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f805c088ce0 con 0x7f80640c9440 2023-10-16T12:52:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.749+0000 7f8063fff640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f805006af90 0x7f805006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.750+0000 7f8063fff640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f805006af90 0x7f805006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8054003fd0 tx=0x7f80540039f0 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.753+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f805c053c00 con 0x7f80640c9440 2023-10-16T12:52:37.879 INFO:tasks.ceph.mds.c.smithi060.stderr:2023-10-16T12:52:37.878+0000 7fcf601bf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:52:37.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.948+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 --> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80640c1f10 con 0x7f805006af90 2023-10-16T12:52:37.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.971+0000 7f8060ff9640 1 -- 172.21.15.60:0/3312370351 <== mgr.6574 v2:172.21.15.144:6832/3337897861 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287290 (secure 0 0 0) 0x7f8054004780 con 0x7f805006af90 2023-10-16T12:52:37.974 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:37.978 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:52:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f805006af90 msgr2=0x7f805006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f805006af90 0x7f805006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8054003fd0 tx=0x7f80540039f0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 msgr2=0x7f80640c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 0x7f80640c5770 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f805c00c340 tx=0x7f805c00c800 comp rx=0 tx=0).stop 2023-10-16T12:52:37.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 shutdown_connections 2023-10-16T12:52:37.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f805006af90 0x7f805006d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80640c9440 0x7f80640c5770 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80640c9060 0x7f806415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 --2- 172.21.15.60:0/3312370351 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f806406c9e0 0x7f8064163820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.979+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 >> 172.21.15.60:0/3312370351 conn(0x7f80640ce6a0 msgr2=0x7f80640af900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.980+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 shutdown_connections 2023-10-16T12:52:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:37.980+0000 7f806937a640 1 -- 172.21.15.60:0/3312370351 wait complete. 2023-10-16T12:52:38.005 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:52:36.224612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":367,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":332348,"kb_used_data":13520,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754642372,"statfs":{"total":773094113280,"available":772753788928,"internally_reserved":0,"allocated":13844480,"data_stored":7204452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":326356536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.322121"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968970+0000","last_change":"2023-10-16T12:52:32.982080+0000","last_active":"2023-10-16T12:52:33.968970+0000","last_peered":"2023-10-16T12:52:33.968970+0000","last_clean":"2023-10-16T12:52:33.968970+0000","last_became_active":"2023-10-16T12:52:32.981780+0000","last_became_peered":"2023-10-16T12:52:32.981780+0000","last_unstale":"2023-10-16T12:52:33.968970+0000","last_undegraded":"2023-10-16T12:52:33.968970+0000","last_fullsized":"2023-10-16T12:52:33.968970+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:43.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690416+0000","last_change":"2023-10-16T12:52:32.983006+0000","last_active":"2023-10-16T12:52:34.690416+0000","last_peered":"2023-10-16T12:52:34.690416+0000","last_clean":"2023-10-16T12:52:34.690416+0000","last_became_active":"2023-10-16T12:52:32.972755+0000","last_became_peered":"2023-10-16T12:52:32.972755+0000","last_unstale":"2023-10-16T12:52:34.690416+0000","last_undegraded":"2023-10-16T12:52:34.690416+0000","last_fullsized":"2023-10-16T12:52:34.690416+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:19.066290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969043+0000","last_change":"2023-10-16T12:52:32.982515+0000","last_active":"2023-10-16T12:52:33.969043+0000","last_peered":"2023-10-16T12:52:33.969043+0000","last_clean":"2023-10-16T12:52:33.969043+0000","last_became_active":"2023-10-16T12:52:32.980783+0000","last_became_peered":"2023-10-16T12:52:32.980783+0000","last_unstale":"2023-10-16T12:52:33.969043+0000","last_undegraded":"2023-10-16T12:52:33.969043+0000","last_fullsized":"2023-10-16T12:52:33.969043+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:31.936481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577214+0000","last_change":"2023-10-16T12:52:32.969157+0000","last_active":"2023-10-16T12:52:34.577214+0000","last_peered":"2023-10-16T12:52:34.577214+0000","last_clean":"2023-10-16T12:52:34.577214+0000","last_became_active":"2023-10-16T12:52:32.968851+0000","last_became_peered":"2023-10-16T12:52:32.968851+0000","last_unstale":"2023-10-16T12:52:34.577214+0000","last_undegraded":"2023-10-16T12:52:34.577214+0000","last_fullsized":"2023-10-16T12:52:34.577214+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:30.313878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971344+0000","last_change":"2023-10-16T12:52:32.970626+0000","last_active":"2023-10-16T12:52:33.971344+0000","last_peered":"2023-10-16T12:52:33.971344+0000","last_clean":"2023-10-16T12:52:33.971344+0000","last_became_active":"2023-10-16T12:52:32.970238+0000","last_became_peered":"2023-10-16T12:52:32.970238+0000","last_unstale":"2023-10-16T12:52:33.971344+0000","last_undegraded":"2023-10-16T12:52:33.971344+0000","last_fullsized":"2023-10-16T12:52:33.971344+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:10.822211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689693+0000","last_change":"2023-10-16T12:52:32.961451+0000","last_active":"2023-10-16T12:52:34.689693+0000","last_peered":"2023-10-16T12:52:34.689693+0000","last_clean":"2023-10-16T12:52:34.689693+0000","last_became_active":"2023-10-16T12:52:32.961114+0000","last_became_peered":"2023-10-16T12:52:32.961114+0000","last_unstale":"2023-10-16T12:52:34.689693+0000","last_undegraded":"2023-10-16T12:52:34.689693+0000","last_fullsized":"2023-10-16T12:52:34.689693+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:24.418432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689093+0000","last_change":"2023-10-16T12:52:32.968318+0000","last_active":"2023-10-16T12:52:34.689093+0000","last_peered":"2023-10-16T12:52:34.689093+0000","last_clean":"2023-10-16T12:52:34.689093+0000","last_became_active":"2023-10-16T12:52:32.967344+0000","last_became_peered":"2023-10-16T12:52:32.967344+0000","last_unstale":"2023-10-16T12:52:34.689093+0000","last_undegraded":"2023-10-16T12:52:34.689093+0000","last_fullsized":"2023-10-16T12:52:34.689093+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:14.904662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689800+0000","last_change":"2023-10-16T12:52:32.964609+0000","last_active":"2023-10-16T12:52:34.689800+0000","last_peered":"2023-10-16T12:52:34.689800+0000","last_clean":"2023-10-16T12:52:34.689800+0000","last_became_active":"2023-10-16T12:52:32.964325+0000","last_became_peered":"2023-10-16T12:52:32.964325+0000","last_unstale":"2023-10-16T12:52:34.689800+0000","last_undegraded":"2023-10-16T12:52:34.689800+0000","last_fullsized":"2023-10-16T12:52:34.689800+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:29.118326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691526+0000","last_change":"2023-10-16T12:52:32.986124+0000","last_active":"2023-10-16T12:52:34.691526+0000","last_peered":"2023-10-16T12:52:34.691526+0000","last_clean":"2023-10-16T12:52:34.691526+0000","last_became_active":"2023-10-16T12:52:32.985825+0000","last_became_peered":"2023-10-16T12:52:32.985825+0000","last_unstale":"2023-10-16T12:52:34.691526+0000","last_undegraded":"2023-10-16T12:52:34.691526+0000","last_fullsized":"2023-10-16T12:52:34.691526+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:22.340901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972028+0000","last_change":"2023-10-16T12:52:32.971929+0000","last_active":"2023-10-16T12:52:33.972028+0000","last_peered":"2023-10-16T12:52:33.972028+0000","last_clean":"2023-10-16T12:52:33.972028+0000","last_became_active":"2023-10-16T12:52:32.971617+0000","last_became_peered":"2023-10-16T12:52:32.971617+0000","last_unstale":"2023-10-16T12:52:33.972028+0000","last_undegraded":"2023-10-16T12:52:33.972028+0000","last_fullsized":"2023-10-16T12:52:33.972028+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:58.405850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689289+0000","last_change":"2023-10-16T12:52:32.964556+0000","last_active":"2023-10-16T12:52:34.689289+0000","last_peered":"2023-10-16T12:52:34.689289+0000","last_clean":"2023-10-16T12:52:34.689289+0000","last_became_active":"2023-10-16T12:52:32.964260+0000","last_became_peered":"2023-10-16T12:52:32.964260+0000","last_unstale":"2023-10-16T12:52:34.689289+0000","last_undegraded":"2023-10-16T12:52:34.689289+0000","last_fullsized":"2023-10-16T12:52:34.689289+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:49.681741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691153+0000","last_change":"2023-10-16T12:52:32.967708+0000","last_active":"2023-10-16T12:52:34.691153+0000","last_peered":"2023-10-16T12:52:34.691153+0000","last_clean":"2023-10-16T12:52:34.691153+0000","last_became_active":"2023-10-16T12:52:32.967413+0000","last_became_peered":"2023-10-16T12:52:32.967413+0000","last_unstale":"2023-10-16T12:52:34.691153+0000","last_undegraded":"2023-10-16T12:52:34.691153+0000","last_fullsized":"2023-10-16T12:52:34.691153+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:29:54.151963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576592+0000","last_change":"2023-10-16T12:52:32.964947+0000","last_active":"2023-10-16T12:52:34.576592+0000","last_peered":"2023-10-16T12:52:34.576592+0000","last_clean":"2023-10-16T12:52:34.576592+0000","last_became_active":"2023-10-16T12:52:32.964618+0000","last_became_peered":"2023-10-16T12:52:32.964618+0000","last_unstale":"2023-10-16T12:52:34.576592+0000","last_undegraded":"2023-10-16T12:52:34.576592+0000","last_fullsized":"2023-10-16T12:52:34.576592+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:40.320909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690395+0000","last_change":"2023-10-16T12:52:32.956714+0000","last_active":"2023-10-16T12:52:34.690395+0000","last_peered":"2023-10-16T12:52:34.690395+0000","last_clean":"2023-10-16T12:52:34.690395+0000","last_became_active":"2023-10-16T12:52:32.956394+0000","last_became_peered":"2023-10-16T12:52:32.956394+0000","last_unstale":"2023-10-16T12:52:34.690395+0000","last_undegraded":"2023-10-16T12:52:34.690395+0000","last_fullsized":"2023-10-16T12:52:34.690395+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:19.066290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576428+0000","last_change":"2023-10-16T12:52:32.981843+0000","last_active":"2023-10-16T12:52:34.576428+0000","last_peered":"2023-10-16T12:52:34.576428+0000","last_clean":"2023-10-16T12:52:34.576428+0000","last_became_active":"2023-10-16T12:52:32.981580+0000","last_became_peered":"2023-10-16T12:52:32.981580+0000","last_unstale":"2023-10-16T12:52:34.576428+0000","last_undegraded":"2023-10-16T12:52:34.576428+0000","last_fullsized":"2023-10-16T12:52:34.576428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:36.244079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689000+0000","last_change":"2023-10-16T12:52:32.966307+0000","last_active":"2023-10-16T12:52:34.689000+0000","last_peered":"2023-10-16T12:52:34.689000+0000","last_clean":"2023-10-16T12:52:34.689000+0000","last_became_active":"2023-10-16T12:52:32.965562+0000","last_became_peered":"2023-10-16T12:52:32.965562+0000","last_unstale":"2023-10-16T12:52:34.689000+0000","last_undegraded":"2023-10-16T12:52:34.689000+0000","last_fullsized":"2023-10-16T12:52:34.689000+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:00:47.691249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690640+0000","last_change":"2023-10-16T12:52:32.987134+0000","last_active":"2023-10-16T12:52:34.690640+0000","last_peered":"2023-10-16T12:52:34.690640+0000","last_clean":"2023-10-16T12:52:34.690640+0000","last_became_active":"2023-10-16T12:52:32.986824+0000","last_became_peered":"2023-10-16T12:52:32.986824+0000","last_unstale":"2023-10-16T12:52:34.690640+0000","last_undegraded":"2023-10-16T12:52:34.690640+0000","last_fullsized":"2023-10-16T12:52:34.690640+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:42.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577288+0000","last_change":"2023-10-16T12:52:32.962930+0000","last_active":"2023-10-16T12:52:34.577288+0000","last_peered":"2023-10-16T12:52:34.577288+0000","last_clean":"2023-10-16T12:52:34.577288+0000","last_became_active":"2023-10-16T12:52:32.962650+0000","last_became_peered":"2023-10-16T12:52:32.962650+0000","last_unstale":"2023-10-16T12:52:34.577288+0000","last_undegraded":"2023-10-16T12:52:34.577288+0000","last_fullsized":"2023-10-16T12:52:34.577288+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:54.737606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692175+0000","last_change":"2023-10-16T12:52:32.975694+0000","last_active":"2023-10-16T12:52:34.692175+0000","last_peered":"2023-10-16T12:52:34.692175+0000","last_clean":"2023-10-16T12:52:34.692175+0000","last_became_active":"2023-10-16T12:52:32.975352+0000","last_became_peered":"2023-10-16T12:52:32.975352+0000","last_unstale":"2023-10-16T12:52:34.692175+0000","last_undegraded":"2023-10-16T12:52:34.692175+0000","last_fullsized":"2023-10-16T12:52:34.692175+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:44.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691507+0000","last_change":"2023-10-16T12:52:32.975781+0000","last_active":"2023-10-16T12:52:34.691507+0000","last_peered":"2023-10-16T12:52:34.691507+0000","last_clean":"2023-10-16T12:52:34.691507+0000","last_became_active":"2023-10-16T12:52:32.974496+0000","last_became_peered":"2023-10-16T12:52:32.974496+0000","last_unstale":"2023-10-16T12:52:34.691507+0000","last_undegraded":"2023-10-16T12:52:34.691507+0000","last_fullsized":"2023-10-16T12:52:34.691507+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:20.952208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691741+0000","last_change":"2023-10-16T12:52:32.975559+0000","last_active":"2023-10-16T12:52:34.691741+0000","last_peered":"2023-10-16T12:52:34.691741+0000","last_clean":"2023-10-16T12:52:34.691741+0000","last_became_active":"2023-10-16T12:52:32.975268+0000","last_became_peered":"2023-10-16T12:52:32.975268+0000","last_unstale":"2023-10-16T12:52:34.691741+0000","last_undegraded":"2023-10-16T12:52:34.691741+0000","last_fullsized":"2023-10-16T12:52:34.691741+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:01.930459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691673+0000","last_change":"2023-10-16T12:52:32.970933+0000","last_active":"2023-10-16T12:52:34.691673+0000","last_peered":"2023-10-16T12:52:34.691673+0000","last_clean":"2023-10-16T12:52:34.691673+0000","last_became_active":"2023-10-16T12:52:32.970666+0000","last_became_peered":"2023-10-16T12:52:32.970666+0000","last_unstale":"2023-10-16T12:52:34.691673+0000","last_undegraded":"2023-10-16T12:52:34.691673+0000","last_fullsized":"2023-10-16T12:52:34.691673+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:01.930459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689829+0000","last_change":"2023-10-16T12:52:32.970781+0000","last_active":"2023-10-16T12:52:34.689829+0000","last_peered":"2023-10-16T12:52:34.689829+0000","last_clean":"2023-10-16T12:52:34.689829+0000","last_became_active":"2023-10-16T12:52:32.970469+0000","last_became_peered":"2023-10-16T12:52:32.970469+0000","last_unstale":"2023-10-16T12:52:34.689829+0000","last_undegraded":"2023-10-16T12:52:34.689829+0000","last_fullsized":"2023-10-16T12:52:34.689829+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:21.452261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689406+0000","last_change":"2023-10-16T12:52:32.971507+0000","last_active":"2023-10-16T12:52:34.689406+0000","last_peered":"2023-10-16T12:52:34.689406+0000","last_clean":"2023-10-16T12:52:34.689406+0000","last_became_active":"2023-10-16T12:52:32.970904+0000","last_became_peered":"2023-10-16T12:52:32.970904+0000","last_unstale":"2023-10-16T12:52:34.689406+0000","last_undegraded":"2023-10-16T12:52:34.689406+0000","last_fullsized":"2023-10-16T12:52:34.689406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:26.693581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972406+0000","last_change":"2023-10-16T12:52:32.965139+0000","last_active":"2023-10-16T12:52:33.972406+0000","last_peered":"2023-10-16T12:52:33.972406+0000","last_clean":"2023-10-16T12:52:33.972406+0000","last_became_active":"2023-10-16T12:52:32.964855+0000","last_became_peered":"2023-10-16T12:52:32.964855+0000","last_unstale":"2023-10-16T12:52:33.972406+0000","last_undegraded":"2023-10-16T12:52:33.972406+0000","last_fullsized":"2023-10-16T12:52:33.972406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:08:09.238308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690764+0000","last_change":"2023-10-16T12:52:32.983256+0000","last_active":"2023-10-16T12:52:34.690764+0000","last_peered":"2023-10-16T12:52:34.690764+0000","last_clean":"2023-10-16T12:52:34.690764+0000","last_became_active":"2023-10-16T12:52:32.982687+0000","last_became_peered":"2023-10-16T12:52:32.982687+0000","last_unstale":"2023-10-16T12:52:34.690764+0000","last_undegraded":"2023-10-16T12:52:34.690764+0000","last_fullsized":"2023-10-16T12:52:34.690764+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:20.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689535+0000","last_change":"2023-10-16T12:52:32.956515+0000","last_active":"2023-10-16T12:52:34.689535+0000","last_peered":"2023-10-16T12:52:34.689535+0000","last_clean":"2023-10-16T12:52:34.689535+0000","last_became_active":"2023-10-16T12:52:32.955440+0000","last_became_peered":"2023-10-16T12:52:32.955440+0000","last_unstale":"2023-10-16T12:52:34.689535+0000","last_undegraded":"2023-10-16T12:52:34.689535+0000","last_fullsized":"2023-10-16T12:52:34.689535+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:14.201658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971565+0000","last_change":"2023-10-16T12:52:32.969204+0000","last_active":"2023-10-16T12:52:33.971565+0000","last_peered":"2023-10-16T12:52:33.971565+0000","last_clean":"2023-10-16T12:52:33.971565+0000","last_became_active":"2023-10-16T12:52:32.968911+0000","last_became_peered":"2023-10-16T12:52:32.968911+0000","last_unstale":"2023-10-16T12:52:33.971565+0000","last_undegraded":"2023-10-16T12:52:33.971565+0000","last_fullsized":"2023-10-16T12:52:33.971565+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:10.594986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690967+0000","last_change":"2023-10-16T12:52:32.977462+0000","last_active":"2023-10-16T12:52:34.690967+0000","last_peered":"2023-10-16T12:52:34.690967+0000","last_clean":"2023-10-16T12:52:34.690967+0000","last_became_active":"2023-10-16T12:52:32.977182+0000","last_became_peered":"2023-10-16T12:52:32.977182+0000","last_unstale":"2023-10-16T12:52:34.690967+0000","last_undegraded":"2023-10-16T12:52:34.690967+0000","last_fullsized":"2023-10-16T12:52:34.690967+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:42.073865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690781+0000","last_change":"2023-10-16T12:52:32.976653+0000","last_active":"2023-10-16T12:52:34.690781+0000","last_peered":"2023-10-16T12:52:34.690781+0000","last_clean":"2023-10-16T12:52:34.690781+0000","last_became_active":"2023-10-16T12:52:32.975283+0000","last_became_peered":"2023-10-16T12:52:32.975283+0000","last_unstale":"2023-10-16T12:52:34.690781+0000","last_undegraded":"2023-10-16T12:52:34.690781+0000","last_fullsized":"2023-10-16T12:52:34.690781+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:42.073865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576709+0000","last_change":"2023-10-16T12:52:32.968204+0000","last_active":"2023-10-16T12:52:34.576709+0000","last_peered":"2023-10-16T12:52:34.576709+0000","last_clean":"2023-10-16T12:52:34.576709+0000","last_became_active":"2023-10-16T12:52:32.967837+0000","last_became_peered":"2023-10-16T12:52:32.967837+0000","last_unstale":"2023-10-16T12:52:34.576709+0000","last_undegraded":"2023-10-16T12:52:34.576709+0000","last_fullsized":"2023-10-16T12:52:34.576709+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:09.915933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.967920+0000","last_change":"2023-10-16T12:52:32.976622+0000","last_active":"2023-10-16T12:52:33.967920+0000","last_peered":"2023-10-16T12:52:33.967920+0000","last_clean":"2023-10-16T12:52:33.967920+0000","last_became_active":"2023-10-16T12:52:32.976213+0000","last_became_peered":"2023-10-16T12:52:32.976213+0000","last_unstale":"2023-10-16T12:52:33.967920+0000","last_undegraded":"2023-10-16T12:52:33.967920+0000","last_fullsized":"2023-10-16T12:52:33.967920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:09.857867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692364+0000","last_change":"2023-10-16T12:52:32.958259+0000","last_active":"2023-10-16T12:52:34.692364+0000","last_peered":"2023-10-16T12:52:34.692364+0000","last_clean":"2023-10-16T12:52:34.692364+0000","last_became_active":"2023-10-16T12:52:32.957948+0000","last_became_peered":"2023-10-16T12:52:32.957948+0000","last_unstale":"2023-10-16T12:52:34.692364+0000","last_undegraded":"2023-10-16T12:52:34.692364+0000","last_fullsized":"2023-10-16T12:52:34.692364+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:09.857867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"127'4","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690887+0000","last_change":"2023-10-16T12:52:31.941706+0000","last_active":"2023-10-16T12:52:34.690887+0000","last_peered":"2023-10-16T12:52:34.690887+0000","last_clean":"2023-10-16T12:52:34.690887+0000","last_became_active":"2023-10-16T12:52:31.941434+0000","last_became_peered":"2023-10-16T12:52:31.941434+0000","last_unstale":"2023-10-16T12:52:34.690887+0000","last_undegraded":"2023-10-16T12:52:34.690887+0000","last_fullsized":"2023-10-16T12:52:34.690887+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:53.480734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577540+0000","last_change":"2023-10-16T12:52:32.973277+0000","last_active":"2023-10-16T12:52:34.577540+0000","last_peered":"2023-10-16T12:52:34.577540+0000","last_clean":"2023-10-16T12:52:34.577540+0000","last_became_active":"2023-10-16T12:52:32.972994+0000","last_became_peered":"2023-10-16T12:52:32.972994+0000","last_unstale":"2023-10-16T12:52:34.577540+0000","last_undegraded":"2023-10-16T12:52:34.577540+0000","last_fullsized":"2023-10-16T12:52:34.577540+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:48.415965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689643+0000","last_change":"2023-10-16T12:52:31.947702+0000","last_active":"2023-10-16T12:52:34.689643+0000","last_peered":"2023-10-16T12:52:34.689643+0000","last_clean":"2023-10-16T12:52:34.689643+0000","last_became_active":"2023-10-16T12:52:31.947275+0000","last_became_peered":"2023-10-16T12:52:31.947275+0000","last_unstale":"2023-10-16T12:52:34.689643+0000","last_undegraded":"2023-10-16T12:52:34.689643+0000","last_fullsized":"2023-10-16T12:52:34.689643+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:45.370230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969009+0000","last_change":"2023-10-16T12:52:32.981451+0000","last_active":"2023-10-16T12:52:33.969009+0000","last_peered":"2023-10-16T12:52:33.969009+0000","last_clean":"2023-10-16T12:52:33.969009+0000","last_became_active":"2023-10-16T12:52:32.981168+0000","last_became_peered":"2023-10-16T12:52:32.981168+0000","last_unstale":"2023-10-16T12:52:33.969009+0000","last_undegraded":"2023-10-16T12:52:33.969009+0000","last_fullsized":"2023-10-16T12:52:33.969009+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:20.167134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690759+0000","last_change":"2023-10-16T12:52:31.948780+0000","last_active":"2023-10-16T12:52:34.690759+0000","last_peered":"2023-10-16T12:52:34.690759+0000","last_clean":"2023-10-16T12:52:34.690759+0000","last_became_active":"2023-10-16T12:52:31.948475+0000","last_became_peered":"2023-10-16T12:52:31.948475+0000","last_unstale":"2023-10-16T12:52:34.690759+0000","last_undegraded":"2023-10-16T12:52:34.690759+0000","last_fullsized":"2023-10-16T12:52:34.690759+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:44:59.552949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691532+0000","last_change":"2023-10-16T12:52:32.971887+0000","last_active":"2023-10-16T12:52:34.691532+0000","last_peered":"2023-10-16T12:52:34.691532+0000","last_clean":"2023-10-16T12:52:34.691532+0000","last_became_active":"2023-10-16T12:52:32.970322+0000","last_became_peered":"2023-10-16T12:52:32.970322+0000","last_unstale":"2023-10-16T12:52:34.691532+0000","last_undegraded":"2023-10-16T12:52:34.691532+0000","last_fullsized":"2023-10-16T12:52:34.691532+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:55.063149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690039+0000","last_change":"2023-10-16T12:52:31.935360+0000","last_active":"2023-10-16T12:52:34.690039+0000","last_peered":"2023-10-16T12:52:34.690039+0000","last_clean":"2023-10-16T12:52:34.690039+0000","last_became_active":"2023-10-16T12:52:31.932585+0000","last_became_peered":"2023-10-16T12:52:31.932585+0000","last_unstale":"2023-10-16T12:52:34.690039+0000","last_undegraded":"2023-10-16T12:52:34.690039+0000","last_fullsized":"2023-10-16T12:52:34.690039+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:09.940167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973406+0000","last_change":"2023-10-16T12:52:32.966332+0000","last_active":"2023-10-16T12:52:33.973406+0000","last_peered":"2023-10-16T12:52:33.973406+0000","last_clean":"2023-10-16T12:52:33.973406+0000","last_became_active":"2023-10-16T12:52:32.966050+0000","last_became_peered":"2023-10-16T12:52:32.966050+0000","last_unstale":"2023-10-16T12:52:33.973406+0000","last_undegraded":"2023-10-16T12:52:33.973406+0000","last_fullsized":"2023-10-16T12:52:33.973406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:33.889940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968551+0000","last_change":"2023-10-16T12:52:31.936086+0000","last_active":"2023-10-16T12:52:33.968551+0000","last_peered":"2023-10-16T12:52:33.968551+0000","last_clean":"2023-10-16T12:52:33.968551+0000","last_became_active":"2023-10-16T12:52:31.935759+0000","last_became_peered":"2023-10-16T12:52:31.935759+0000","last_unstale":"2023-10-16T12:52:33.968551+0000","last_undegraded":"2023-10-16T12:52:33.968551+0000","last_fullsized":"2023-10-16T12:52:33.968551+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:08.683475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689147+0000","last_change":"2023-10-16T12:52:32.953315+0000","last_active":"2023-10-16T12:52:34.689147+0000","last_peered":"2023-10-16T12:52:34.689147+0000","last_clean":"2023-10-16T12:52:34.689147+0000","last_became_active":"2023-10-16T12:52:32.952608+0000","last_became_peered":"2023-10-16T12:52:32.952608+0000","last_unstale":"2023-10-16T12:52:34.689147+0000","last_undegraded":"2023-10-16T12:52:34.689147+0000","last_fullsized":"2023-10-16T12:52:34.689147+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:35.485882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691920+0000","last_change":"2023-10-16T12:52:31.936889+0000","last_active":"2023-10-16T12:52:34.691920+0000","last_peered":"2023-10-16T12:52:34.691920+0000","last_clean":"2023-10-16T12:52:34.691920+0000","last_became_active":"2023-10-16T12:52:31.935613+0000","last_became_peered":"2023-10-16T12:52:31.935613+0000","last_unstale":"2023-10-16T12:52:34.691920+0000","last_undegraded":"2023-10-16T12:52:34.691920+0000","last_fullsized":"2023-10-16T12:52:34.691920+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:48.089984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576808+0000","last_change":"2023-10-16T12:52:32.981362+0000","last_active":"2023-10-16T12:52:34.576808+0000","last_peered":"2023-10-16T12:52:34.576808+0000","last_clean":"2023-10-16T12:52:34.576808+0000","last_became_active":"2023-10-16T12:52:32.980936+0000","last_became_peered":"2023-10-16T12:52:32.980936+0000","last_unstale":"2023-10-16T12:52:34.576808+0000","last_undegraded":"2023-10-16T12:52:34.576808+0000","last_fullsized":"2023-10-16T12:52:34.576808+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:41.681452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971514+0000","last_change":"2023-10-16T12:52:31.937779+0000","last_active":"2023-10-16T12:52:33.971514+0000","last_peered":"2023-10-16T12:52:33.971514+0000","last_clean":"2023-10-16T12:52:33.971514+0000","last_became_active":"2023-10-16T12:52:31.937466+0000","last_became_peered":"2023-10-16T12:52:31.937466+0000","last_unstale":"2023-10-16T12:52:33.971514+0000","last_undegraded":"2023-10-16T12:52:33.971514+0000","last_fullsized":"2023-10-16T12:52:33.971514+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:55.501105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690571+0000","last_change":"2023-10-16T12:52:32.970104+0000","last_active":"2023-10-16T12:52:34.690571+0000","last_peered":"2023-10-16T12:52:34.690571+0000","last_clean":"2023-10-16T12:52:34.690571+0000","last_became_active":"2023-10-16T12:52:32.969413+0000","last_became_peered":"2023-10-16T12:52:32.969413+0000","last_unstale":"2023-10-16T12:52:34.690571+0000","last_undegraded":"2023-10-16T12:52:34.690571+0000","last_fullsized":"2023-10-16T12:52:34.690571+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:06.795464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968030+0000","last_change":"2023-10-16T12:52:31.939413+0000","last_active":"2023-10-16T12:52:33.968030+0000","last_peered":"2023-10-16T12:52:33.968030+0000","last_clean":"2023-10-16T12:52:33.968030+0000","last_became_active":"2023-10-16T12:52:31.938945+0000","last_became_peered":"2023-10-16T12:52:31.938945+0000","last_unstale":"2023-10-16T12:52:33.968030+0000","last_undegraded":"2023-10-16T12:52:33.968030+0000","last_fullsized":"2023-10-16T12:52:33.968030+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:05.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691121+0000","last_change":"2023-10-16T12:52:32.970316+0000","last_active":"2023-10-16T12:52:34.691121+0000","last_peered":"2023-10-16T12:52:34.691121+0000","last_clean":"2023-10-16T12:52:34.691121+0000","last_became_active":"2023-10-16T12:52:32.970024+0000","last_became_peered":"2023-10-16T12:52:32.970024+0000","last_unstale":"2023-10-16T12:52:34.691121+0000","last_undegraded":"2023-10-16T12:52:34.691121+0000","last_fullsized":"2023-10-16T12:52:34.691121+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:09.227980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969264+0000","last_change":"2023-10-16T12:52:31.937580+0000","last_active":"2023-10-16T12:52:33.969264+0000","last_peered":"2023-10-16T12:52:33.969264+0000","last_clean":"2023-10-16T12:52:33.969264+0000","last_became_active":"2023-10-16T12:52:31.937263+0000","last_became_peered":"2023-10-16T12:52:31.937263+0000","last_unstale":"2023-10-16T12:52:33.969264+0000","last_undegraded":"2023-10-16T12:52:33.969264+0000","last_fullsized":"2023-10-16T12:52:33.969264+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:43.248737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691739+0000","last_change":"2023-10-16T12:52:32.969791+0000","last_active":"2023-10-16T12:52:34.691739+0000","last_peered":"2023-10-16T12:52:34.691739+0000","last_clean":"2023-10-16T12:52:34.691739+0000","last_became_active":"2023-10-16T12:52:32.969481+0000","last_became_peered":"2023-10-16T12:52:32.969481+0000","last_unstale":"2023-10-16T12:52:34.691739+0000","last_undegraded":"2023-10-16T12:52:34.691739+0000","last_fullsized":"2023-10-16T12:52:34.691739+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:22.385025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.577207+0000","last_change":"2023-10-16T12:52:31.940500+0000","last_active":"2023-10-16T12:52:34.577207+0000","last_peered":"2023-10-16T12:52:34.577207+0000","last_clean":"2023-10-16T12:52:34.577207+0000","last_became_active":"2023-10-16T12:52:31.940186+0000","last_became_peered":"2023-10-16T12:52:31.940186+0000","last_unstale":"2023-10-16T12:52:34.577207+0000","last_undegraded":"2023-10-16T12:52:34.577207+0000","last_fullsized":"2023-10-16T12:52:34.577207+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:45.370230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972281+0000","last_change":"2023-10-16T12:52:32.968381+0000","last_active":"2023-10-16T12:52:33.972281+0000","last_peered":"2023-10-16T12:52:33.972281+0000","last_clean":"2023-10-16T12:52:33.972281+0000","last_became_active":"2023-10-16T12:52:32.967991+0000","last_became_peered":"2023-10-16T12:52:32.967991+0000","last_unstale":"2023-10-16T12:52:33.972281+0000","last_undegraded":"2023-10-16T12:52:33.972281+0000","last_fullsized":"2023-10-16T12:52:33.972281+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:32.346811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"127'3","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690031+0000","last_change":"2023-10-16T12:52:31.947894+0000","last_active":"2023-10-16T12:52:34.690031+0000","last_peered":"2023-10-16T12:52:34.690031+0000","last_clean":"2023-10-16T12:52:34.690031+0000","last_became_active":"2023-10-16T12:52:31.947401+0000","last_became_peered":"2023-10-16T12:52:31.947401+0000","last_unstale":"2023-10-16T12:52:34.690031+0000","last_undegraded":"2023-10-16T12:52:34.690031+0000","last_fullsized":"2023-10-16T12:52:34.690031+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:40.667980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689775+0000","last_change":"2023-10-16T12:52:32.967239+0000","last_active":"2023-10-16T12:52:34.689775+0000","last_peered":"2023-10-16T12:52:34.689775+0000","last_clean":"2023-10-16T12:52:34.689775+0000","last_became_active":"2023-10-16T12:52:32.966895+0000","last_became_peered":"2023-10-16T12:52:32.966895+0000","last_unstale":"2023-10-16T12:52:34.689775+0000","last_undegraded":"2023-10-16T12:52:34.689775+0000","last_fullsized":"2023-10-16T12:52:34.689775+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:32:41.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"127'6","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691699+0000","last_change":"2023-10-16T12:52:31.944329+0000","last_active":"2023-10-16T12:52:34.691699+0000","last_peered":"2023-10-16T12:52:34.691699+0000","last_clean":"2023-10-16T12:52:34.691699+0000","last_became_active":"2023-10-16T12:52:31.943418+0000","last_became_peered":"2023-10-16T12:52:31.943418+0000","last_unstale":"2023-10-16T12:52:34.691699+0000","last_undegraded":"2023-10-16T12:52:34.691699+0000","last_fullsized":"2023-10-16T12:52:34.691699+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:34.122017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576797+0000","last_change":"2023-10-16T12:52:32.964151+0000","last_active":"2023-10-16T12:52:34.576797+0000","last_peered":"2023-10-16T12:52:34.576797+0000","last_clean":"2023-10-16T12:52:34.576797+0000","last_became_active":"2023-10-16T12:52:32.962164+0000","last_became_peered":"2023-10-16T12:52:32.962164+0000","last_unstale":"2023-10-16T12:52:34.576797+0000","last_undegraded":"2023-10-16T12:52:34.576797+0000","last_fullsized":"2023-10-16T12:52:34.576797+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:29.826473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692269+0000","last_change":"2023-10-16T12:52:31.950744+0000","last_active":"2023-10-16T12:52:34.692269+0000","last_peered":"2023-10-16T12:52:34.692269+0000","last_clean":"2023-10-16T12:52:34.692269+0000","last_became_active":"2023-10-16T12:52:31.950465+0000","last_became_peered":"2023-10-16T12:52:31.950465+0000","last_unstale":"2023-10-16T12:52:34.692269+0000","last_undegraded":"2023-10-16T12:52:34.692269+0000","last_fullsized":"2023-10-16T12:52:34.692269+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:55.286130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968436+0000","last_change":"2023-10-16T12:52:32.981804+0000","last_active":"2023-10-16T12:52:33.968436+0000","last_peered":"2023-10-16T12:52:33.968436+0000","last_clean":"2023-10-16T12:52:33.968436+0000","last_became_active":"2023-10-16T12:52:32.981513+0000","last_became_peered":"2023-10-16T12:52:32.981513+0000","last_unstale":"2023-10-16T12:52:33.968436+0000","last_undegraded":"2023-10-16T12:52:33.968436+0000","last_fullsized":"2023-10-16T12:52:33.968436+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:03.559443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691547+0000","last_change":"2023-10-16T12:52:31.945301+0000","last_active":"2023-10-16T12:52:34.691547+0000","last_peered":"2023-10-16T12:52:34.691547+0000","last_clean":"2023-10-16T12:52:34.691547+0000","last_became_active":"2023-10-16T12:52:31.944963+0000","last_became_peered":"2023-10-16T12:52:31.944963+0000","last_unstale":"2023-10-16T12:52:34.691547+0000","last_undegraded":"2023-10-16T12:52:34.691547+0000","last_fullsized":"2023-10-16T12:52:34.691547+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:28.104854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690473+0000","last_change":"2023-10-16T12:52:32.969973+0000","last_active":"2023-10-16T12:52:34.690473+0000","last_peered":"2023-10-16T12:52:34.690473+0000","last_clean":"2023-10-16T12:52:34.690473+0000","last_became_active":"2023-10-16T12:52:32.969264+0000","last_became_peered":"2023-10-16T12:52:32.969264+0000","last_unstale":"2023-10-16T12:52:34.690473+0000","last_undegraded":"2023-10-16T12:52:34.690473+0000","last_fullsized":"2023-10-16T12:52:34.690473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:42.604065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690641+0000","last_change":"2023-10-16T12:52:31.938306+0000","last_active":"2023-10-16T12:52:34.690641+0000","last_peered":"2023-10-16T12:52:34.690641+0000","last_clean":"2023-10-16T12:52:34.690641+0000","last_became_active":"2023-10-16T12:52:31.938003+0000","last_became_peered":"2023-10-16T12:52:31.938003+0000","last_unstale":"2023-10-16T12:52:34.690641+0000","last_undegraded":"2023-10-16T12:52:34.690641+0000","last_fullsized":"2023-10-16T12:52:34.690641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:14.646141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690295+0000","last_change":"2023-10-16T12:52:32.972461+0000","last_active":"2023-10-16T12:52:34.690295+0000","last_peered":"2023-10-16T12:52:34.690295+0000","last_clean":"2023-10-16T12:52:34.690295+0000","last_became_active":"2023-10-16T12:52:32.972151+0000","last_became_peered":"2023-10-16T12:52:32.972151+0000","last_unstale":"2023-10-16T12:52:34.690295+0000","last_undegraded":"2023-10-16T12:52:34.690295+0000","last_fullsized":"2023-10-16T12:52:34.690295+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:21.452261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"127'8","reported_seq":20,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968094+0000","last_change":"2023-10-16T12:52:31.942622+0000","last_active":"2023-10-16T12:52:33.968094+0000","last_peered":"2023-10-16T12:52:33.968094+0000","last_clean":"2023-10-16T12:52:33.968094+0000","last_became_active":"2023-10-16T12:52:31.942275+0000","last_became_peered":"2023-10-16T12:52:31.942275+0000","last_unstale":"2023-10-16T12:52:33.968094+0000","last_undegraded":"2023-10-16T12:52:33.968094+0000","last_fullsized":"2023-10-16T12:52:33.968094+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:00.916987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.972475+0000","last_change":"2023-10-16T12:52:32.969301+0000","last_active":"2023-10-16T12:52:33.972475+0000","last_peered":"2023-10-16T12:52:33.972475+0000","last_clean":"2023-10-16T12:52:33.972475+0000","last_became_active":"2023-10-16T12:52:32.968949+0000","last_became_peered":"2023-10-16T12:52:32.968949+0000","last_unstale":"2023-10-16T12:52:33.972475+0000","last_undegraded":"2023-10-16T12:52:33.972475+0000","last_fullsized":"2023-10-16T12:52:33.972475+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:02.417224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691189+0000","last_change":"2023-10-16T12:52:31.950767+0000","last_active":"2023-10-16T12:52:34.691189+0000","last_peered":"2023-10-16T12:52:34.691189+0000","last_clean":"2023-10-16T12:52:34.691189+0000","last_became_active":"2023-10-16T12:52:31.950478+0000","last_became_peered":"2023-10-16T12:52:31.950478+0000","last_unstale":"2023-10-16T12:52:34.691189+0000","last_undegraded":"2023-10-16T12:52:34.691189+0000","last_fullsized":"2023-10-16T12:52:34.691189+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:51:03.632195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973389+0000","last_change":"2023-10-16T12:52:32.966261+0000","last_active":"2023-10-16T12:52:33.973389+0000","last_peered":"2023-10-16T12:52:33.973389+0000","last_clean":"2023-10-16T12:52:33.973389+0000","last_became_active":"2023-10-16T12:52:32.965297+0000","last_became_peered":"2023-10-16T12:52:32.965297+0000","last_unstale":"2023-10-16T12:52:33.973389+0000","last_undegraded":"2023-10-16T12:52:33.973389+0000","last_fullsized":"2023-10-16T12:52:33.973389+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:26.855080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971768+0000","last_change":"2023-10-16T12:52:31.942569+0000","last_active":"2023-10-16T12:52:33.971768+0000","last_peered":"2023-10-16T12:52:33.971768+0000","last_clean":"2023-10-16T12:52:33.971768+0000","last_became_active":"2023-10-16T12:52:31.942087+0000","last_became_peered":"2023-10-16T12:52:31.942087+0000","last_unstale":"2023-10-16T12:52:33.971768+0000","last_undegraded":"2023-10-16T12:52:33.971768+0000","last_fullsized":"2023-10-16T12:52:33.971768+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:56.117211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689904+0000","last_change":"2023-10-16T12:52:32.963120+0000","last_active":"2023-10-16T12:52:34.689904+0000","last_peered":"2023-10-16T12:52:34.689904+0000","last_clean":"2023-10-16T12:52:34.689904+0000","last_became_active":"2023-10-16T12:52:32.962826+0000","last_became_peered":"2023-10-16T12:52:32.962826+0000","last_unstale":"2023-10-16T12:52:34.689904+0000","last_undegraded":"2023-10-16T12:52:34.689904+0000","last_fullsized":"2023-10-16T12:52:34.689904+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:49.040358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690311+0000","last_change":"2023-10-16T12:52:31.932569+0000","last_active":"2023-10-16T12:52:34.690311+0000","last_peered":"2023-10-16T12:52:34.690311+0000","last_clean":"2023-10-16T12:52:34.690311+0000","last_became_active":"2023-10-16T12:52:31.932229+0000","last_became_peered":"2023-10-16T12:52:31.932229+0000","last_unstale":"2023-10-16T12:52:34.690311+0000","last_undegraded":"2023-10-16T12:52:34.690311+0000","last_fullsized":"2023-10-16T12:52:34.690311+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:53.510824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691317+0000","last_change":"2023-10-16T12:52:32.966941+0000","last_active":"2023-10-16T12:52:34.691317+0000","last_peered":"2023-10-16T12:52:34.691317+0000","last_clean":"2023-10-16T12:52:34.691317+0000","last_became_active":"2023-10-16T12:52:32.966672+0000","last_became_peered":"2023-10-16T12:52:32.966672+0000","last_unstale":"2023-10-16T12:52:34.691317+0000","last_undegraded":"2023-10-16T12:52:34.691317+0000","last_fullsized":"2023-10-16T12:52:34.691317+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:49:09.227980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691256+0000","last_change":"2023-10-16T12:52:31.948545+0000","last_active":"2023-10-16T12:52:34.691256+0000","last_peered":"2023-10-16T12:52:34.691256+0000","last_clean":"2023-10-16T12:52:34.691256+0000","last_became_active":"2023-10-16T12:52:31.947915+0000","last_became_peered":"2023-10-16T12:52:31.947915+0000","last_unstale":"2023-10-16T12:52:34.691256+0000","last_undegraded":"2023-10-16T12:52:34.691256+0000","last_fullsized":"2023-10-16T12:52:34.691256+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:05.355539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691849+0000","last_change":"2023-10-16T12:52:32.966143+0000","last_active":"2023-10-16T12:52:34.691849+0000","last_peered":"2023-10-16T12:52:34.691849+0000","last_clean":"2023-10-16T12:52:34.691849+0000","last_became_active":"2023-10-16T12:52:32.965876+0000","last_became_peered":"2023-10-16T12:52:32.965876+0000","last_unstale":"2023-10-16T12:52:34.691849+0000","last_undegraded":"2023-10-16T12:52:34.691849+0000","last_fullsized":"2023-10-16T12:52:34.691849+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:09.696947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973396+0000","last_change":"2023-10-16T12:52:31.939737+0000","last_active":"2023-10-16T12:52:33.973396+0000","last_peered":"2023-10-16T12:52:33.973396+0000","last_clean":"2023-10-16T12:52:33.973396+0000","last_became_active":"2023-10-16T12:52:31.938728+0000","last_became_peered":"2023-10-16T12:52:31.938728+0000","last_unstale":"2023-10-16T12:52:33.973396+0000","last_undegraded":"2023-10-16T12:52:33.973396+0000","last_fullsized":"2023-10-16T12:52:33.973396+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:08:30.161261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690588+0000","last_change":"2023-10-16T12:52:32.976105+0000","last_active":"2023-10-16T12:52:34.690588+0000","last_peered":"2023-10-16T12:52:34.690588+0000","last_clean":"2023-10-16T12:52:34.690588+0000","last_became_active":"2023-10-16T12:52:32.975799+0000","last_became_peered":"2023-10-16T12:52:32.975799+0000","last_unstale":"2023-10-16T12:52:34.690588+0000","last_undegraded":"2023-10-16T12:52:34.690588+0000","last_fullsized":"2023-10-16T12:52:34.690588+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:42.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973161+0000","last_change":"2023-10-16T12:52:31.939521+0000","last_active":"2023-10-16T12:52:33.973161+0000","last_peered":"2023-10-16T12:52:33.973161+0000","last_clean":"2023-10-16T12:52:33.973161+0000","last_became_active":"2023-10-16T12:52:31.939185+0000","last_became_peered":"2023-10-16T12:52:31.939185+0000","last_unstale":"2023-10-16T12:52:33.973161+0000","last_undegraded":"2023-10-16T12:52:33.973161+0000","last_fullsized":"2023-10-16T12:52:33.973161+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:16.678621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.970918+0000","last_change":"2023-10-16T12:52:32.974264+0000","last_active":"2023-10-16T12:52:33.970918+0000","last_peered":"2023-10-16T12:52:33.970918+0000","last_clean":"2023-10-16T12:52:33.970918+0000","last_became_active":"2023-10-16T12:52:32.973914+0000","last_became_peered":"2023-10-16T12:52:32.973914+0000","last_unstale":"2023-10-16T12:52:33.970918+0000","last_undegraded":"2023-10-16T12:52:33.970918+0000","last_fullsized":"2023-10-16T12:52:33.970918+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:15.734992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689653+0000","last_change":"2023-10-16T12:52:31.939466+0000","last_active":"2023-10-16T12:52:34.689653+0000","last_peered":"2023-10-16T12:52:34.689653+0000","last_clean":"2023-10-16T12:52:34.689653+0000","last_became_active":"2023-10-16T12:52:31.938997+0000","last_became_peered":"2023-10-16T12:52:31.938997+0000","last_unstale":"2023-10-16T12:52:34.689653+0000","last_undegraded":"2023-10-16T12:52:34.689653+0000","last_fullsized":"2023-10-16T12:52:34.689653+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:38.030185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691263+0000","last_change":"2023-10-16T12:52:32.976260+0000","last_active":"2023-10-16T12:52:34.691263+0000","last_peered":"2023-10-16T12:52:34.691263+0000","last_clean":"2023-10-16T12:52:34.691263+0000","last_became_active":"2023-10-16T12:52:32.972361+0000","last_became_peered":"2023-10-16T12:52:32.972361+0000","last_unstale":"2023-10-16T12:52:34.691263+0000","last_undegraded":"2023-10-16T12:52:34.691263+0000","last_fullsized":"2023-10-16T12:52:34.691263+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:06.496645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"127'2","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.973984+0000","last_change":"2023-10-16T12:52:31.936338+0000","last_active":"2023-10-16T12:52:33.973984+0000","last_peered":"2023-10-16T12:52:33.973984+0000","last_clean":"2023-10-16T12:52:33.973984+0000","last_became_active":"2023-10-16T12:52:31.935886+0000","last_became_peered":"2023-10-16T12:52:31.935886+0000","last_unstale":"2023-10-16T12:52:33.973984+0000","last_undegraded":"2023-10-16T12:52:33.973984+0000","last_fullsized":"2023-10-16T12:52:33.973984+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:17.236374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.969139+0000","last_change":"2023-10-16T12:52:32.976047+0000","last_active":"2023-10-16T12:52:33.969139+0000","last_peered":"2023-10-16T12:52:33.969139+0000","last_clean":"2023-10-16T12:52:33.969139+0000","last_became_active":"2023-10-16T12:52:32.972474+0000","last_became_peered":"2023-10-16T12:52:32.972474+0000","last_unstale":"2023-10-16T12:52:33.969139+0000","last_undegraded":"2023-10-16T12:52:33.969139+0000","last_fullsized":"2023-10-16T12:52:33.969139+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:49:00.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691392+0000","last_change":"2023-10-16T12:52:31.950985+0000","last_active":"2023-10-16T12:52:34.691392+0000","last_peered":"2023-10-16T12:52:34.691392+0000","last_clean":"2023-10-16T12:52:34.691392+0000","last_became_active":"2023-10-16T12:52:31.950666+0000","last_became_peered":"2023-10-16T12:52:31.950666+0000","last_unstale":"2023-10-16T12:52:34.691392+0000","last_undegraded":"2023-10-16T12:52:34.691392+0000","last_fullsized":"2023-10-16T12:52:34.691392+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:28.836570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.968854+0000","last_change":"2023-10-16T12:52:32.978260+0000","last_active":"2023-10-16T12:52:33.968854+0000","last_peered":"2023-10-16T12:52:33.968854+0000","last_clean":"2023-10-16T12:52:33.968854+0000","last_became_active":"2023-10-16T12:52:32.977948+0000","last_became_peered":"2023-10-16T12:52:32.977948+0000","last_unstale":"2023-10-16T12:52:33.968854+0000","last_undegraded":"2023-10-16T12:52:33.968854+0000","last_fullsized":"2023-10-16T12:52:33.968854+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:24.792272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690470+0000","last_change":"2023-10-16T12:52:31.929524+0000","last_active":"2023-10-16T12:52:34.690470+0000","last_peered":"2023-10-16T12:52:34.690470+0000","last_clean":"2023-10-16T12:52:34.690470+0000","last_became_active":"2023-10-16T12:52:31.929153+0000","last_became_peered":"2023-10-16T12:52:31.929153+0000","last_unstale":"2023-10-16T12:52:34.690470+0000","last_undegraded":"2023-10-16T12:52:34.690470+0000","last_fullsized":"2023-10-16T12:52:34.690470+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:40.033574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971497+0000","last_change":"2023-10-16T12:52:32.971626+0000","last_active":"2023-10-16T12:52:33.971497+0000","last_peered":"2023-10-16T12:52:33.971497+0000","last_clean":"2023-10-16T12:52:33.971497+0000","last_became_active":"2023-10-16T12:52:32.971335+0000","last_became_peered":"2023-10-16T12:52:32.971335+0000","last_unstale":"2023-10-16T12:52:33.971497+0000","last_undegraded":"2023-10-16T12:52:33.971497+0000","last_fullsized":"2023-10-16T12:52:33.971497+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:33.569175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971950+0000","last_change":"2023-10-16T12:52:31.944196+0000","last_active":"2023-10-16T12:52:33.971950+0000","last_peered":"2023-10-16T12:52:33.971950+0000","last_clean":"2023-10-16T12:52:33.971950+0000","last_became_active":"2023-10-16T12:52:31.943052+0000","last_became_peered":"2023-10-16T12:52:31.943052+0000","last_unstale":"2023-10-16T12:52:33.971950+0000","last_undegraded":"2023-10-16T12:52:33.971950+0000","last_fullsized":"2023-10-16T12:52:33.971950+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:36.799355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691420+0000","last_change":"2023-10-16T12:52:32.973396+0000","last_active":"2023-10-16T12:52:34.691420+0000","last_peered":"2023-10-16T12:52:34.691420+0000","last_clean":"2023-10-16T12:52:34.691420+0000","last_became_active":"2023-10-16T12:52:32.973024+0000","last_became_peered":"2023-10-16T12:52:32.973024+0000","last_unstale":"2023-10-16T12:52:34.691420+0000","last_undegraded":"2023-10-16T12:52:34.691420+0000","last_fullsized":"2023-10-16T12:52:34.691420+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:06.496645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691493+0000","last_change":"2023-10-16T12:52:31.945999+0000","last_active":"2023-10-16T12:52:34.691493+0000","last_peered":"2023-10-16T12:52:34.691493+0000","last_clean":"2023-10-16T12:52:34.691493+0000","last_became_active":"2023-10-16T12:52:31.943208+0000","last_became_peered":"2023-10-16T12:52:31.943208+0000","last_unstale":"2023-10-16T12:52:34.691493+0000","last_undegraded":"2023-10-16T12:52:34.691493+0000","last_fullsized":"2023-10-16T12:52:34.691493+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:39.007201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.691118+0000","last_change":"2023-10-16T12:52:32.961721+0000","last_active":"2023-10-16T12:52:34.691118+0000","last_peered":"2023-10-16T12:52:34.691118+0000","last_clean":"2023-10-16T12:52:34.691118+0000","last_became_active":"2023-10-16T12:52:32.960328+0000","last_became_peered":"2023-10-16T12:52:32.960328+0000","last_unstale":"2023-10-16T12:52:34.691118+0000","last_undegraded":"2023-10-16T12:52:34.691118+0000","last_fullsized":"2023-10-16T12:52:34.691118+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:20.915196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.970741+0000","last_change":"2023-10-16T12:52:31.935888+0000","last_active":"2023-10-16T12:52:33.970741+0000","last_peered":"2023-10-16T12:52:33.970741+0000","last_clean":"2023-10-16T12:52:33.970741+0000","last_became_active":"2023-10-16T12:52:31.935571+0000","last_became_peered":"2023-10-16T12:52:31.935571+0000","last_unstale":"2023-10-16T12:52:33.970741+0000","last_undegraded":"2023-10-16T12:52:33.970741+0000","last_fullsized":"2023-10-16T12:52:33.970741+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:01.015817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690761+0000","last_change":"2023-10-16T12:52:32.964954+0000","last_active":"2023-10-16T12:52:34.690761+0000","last_peered":"2023-10-16T12:52:34.690761+0000","last_clean":"2023-10-16T12:52:34.690761+0000","last_became_active":"2023-10-16T12:52:32.964399+0000","last_became_peered":"2023-10-16T12:52:32.964399+0000","last_unstale":"2023-10-16T12:52:34.690761+0000","last_undegraded":"2023-10-16T12:52:34.690761+0000","last_fullsized":"2023-10-16T12:52:34.690761+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:20:20.180905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.576946+0000","last_change":"2023-10-16T12:52:31.947943+0000","last_active":"2023-10-16T12:52:34.576946+0000","last_peered":"2023-10-16T12:52:34.576946+0000","last_clean":"2023-10-16T12:52:34.576946+0000","last_became_active":"2023-10-16T12:52:31.947034+0000","last_became_peered":"2023-10-16T12:52:31.947034+0000","last_unstale":"2023-10-16T12:52:34.576946+0000","last_undegraded":"2023-10-16T12:52:34.576946+0000","last_fullsized":"2023-10-16T12:52:34.576946+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:04.855667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:33.971277+0000","last_change":"2023-10-16T12:52:32.973703+0000","last_active":"2023-10-16T12:52:33.971277+0000","last_peered":"2023-10-16T12:52:33.971277+0000","last_clean":"2023-10-16T12:52:33.971277+0000","last_became_active":"2023-10-16T12:52:32.972589+0000","last_became_peered":"2023-10-16T12:52:32.972589+0000","last_unstale":"2023-10-16T12:52:33.971277+0000","last_undegraded":"2023-10-16T12:52:33.971277+0000","last_fullsized":"2023-10-16T12:52:33.971277+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:08:00.066774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.690828+0000","last_change":"2023-10-16T12:52:31.937242+0000","last_active":"2023-10-16T12:52:34.690828+0000","last_peered":"2023-10-16T12:52:34.690828+0000","last_clean":"2023-10-16T12:52:34.690828+0000","last_became_active":"2023-10-16T12:52:31.936928+0000","last_became_peered":"2023-10-16T12:52:31.936928+0000","last_unstale":"2023-10-16T12:52:34.690828+0000","last_undegraded":"2023-10-16T12:52:34.690828+0000","last_fullsized":"2023-10-16T12:52:34.690828+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:23.992548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.692501+0000","last_change":"2023-10-16T12:52:32.972474+0000","last_active":"2023-10-16T12:52:34.692501+0000","last_peered":"2023-10-16T12:52:34.692501+0000","last_clean":"2023-10-16T12:52:34.692501+0000","last_became_active":"2023-10-16T12:52:32.972141+0000","last_became_peered":"2023-10-16T12:52:32.972141+0000","last_unstale":"2023-10-16T12:52:34.692501+0000","last_undegraded":"2023-10-16T12:52:34.692501+0000","last_fullsized":"2023-10-16T12:52:34.692501+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:31.899449+0000","last_clean_scrub_stamp":"2023-10-16T12:52:31.899449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:03.559443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"127'1","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-10-16T12:52:34.689405+0000","last_change":"2023-10-16T12:52:31.948754+0000","last_active":"2023-10-16T12:52:34.689405+0000","last_peered":"2023-10-16T12:52:34.689405+0000","last_clean":"2023-10-16T12:52:34.689405+0000","last_became_active":"2023-10-16T12:52:31.948465+0000","last_became_peered":"2023-10-16T12:52:31.948465+0000","last_unstale":"2023-10-16T12:52:34.689405+0000","last_undegraded":"2023-10-16T12:52:34.689405+0000","last_fullsized":"2023-10-16T12:52:34.689405+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:52:30.885977+0000","last_clean_scrub_stamp":"2023-10-16T12:52:30.885977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:55:53.724134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"117'208","reported_seq":589,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:52:32.945566+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:52:32.945566+0000","last_peered":"2023-10-16T12:52:32.945566+0000","last_clean":"2023-10-16T12:52:32.945566+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:52:32.945566+0000","last_undegraded":"2023-10-16T12:52:32.945566+0000","last_fullsized":"2023-10-16T12:52:32.945566+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":1830,"num_write":323,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33304,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338536,"statfs":{"total":96636764160,"available":96602660864,"internally_reserved":0,"allocated":2166784,"data_stored":1338117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.2569999999999999,"15min":1.3029999999999999},"min":{"1min":0.68799999999999994,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.835},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3080000000000001,"15min":1.3560000000000001},"min":{"1min":0.81000000000000005,"5min":0.69399999999999995,"15min":0.63200000000000001},"max":{"1min":1.613,"5min":1.7729999999999999,"15min":1.8899999999999999},"last":1.46}]},{"osd":1,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.3440000000000001,"15min":1.2649999999999999},"min":{"1min":0.90600000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":19.158999999999999,"5min":19.158999999999999,"15min":19.158999999999999},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2070000000000001,"15min":1.2250000000000001},"min":{"1min":0.745,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":5.5490000000000004,"5min":5.5490000000000004,"15min":5.5490000000000004},"last":1.6739999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1539999999999999,"15min":1.2},"min":{"1min":0.83099999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.272,"15min":1.3180000000000001},"min":{"1min":0.85599999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7170000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.595}]},{"osd":3,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.339,"15min":1.3109999999999999},"min":{"1min":0.90800000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.2759999999999998,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.3839999999999999,"15min":1.403},"min":{"1min":1.0089999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.4369999999999998,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.54}]},{"osd":4,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1799999999999999,"15min":1.1899999999999999},"min":{"1min":0.77200000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.282,"5min":1.3460000000000001,"15min":1.335},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.831,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.345}]},{"osd":5,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.3480000000000001,"15min":1.339},"min":{"1min":0.875,"5min":0.624,"15min":0.624},"max":{"1min":5.0140000000000002,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.512,"15min":1.4750000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.1720000000000002,"5min":2.411,"15min":2.411},"last":1.631}]},{"osd":6,"last update":"Mon Oct 16 12:51:42 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5940000000000001,"15min":1.5680000000000001},"min":{"1min":0.83299999999999996,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.141,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.478,"15min":1.452},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3170000000000002,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.292}]}]},{"osd":6,"up_from":14,"seq":60129542373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1585152,"data_stored":754589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2210000000000001,"15min":1.2789999999999999},"min":{"1min":0.70899999999999996,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.881,"5min":1.952,"15min":3.7450000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2769999999999999,"15min":1.3320000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7470000000000001,"5min":2.0270000000000001,"15min":3.819},"last":1.482}]},{"osd":1,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1819999999999999,"15min":1.1899999999999999},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.216,"15min":1.22},"min":{"1min":0.90400000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":3.8519999999999999},"last":0.96699999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.1950000000000001,"15min":1.2190000000000001},"min":{"1min":0.94499999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.752,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.234,"15min":1.2729999999999999},"min":{"1min":0.96599999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":3.9009999999999998},"last":0.93600000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.302,"15min":1.3280000000000001},"min":{"1min":0.86299999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.7190000000000001,"5min":1.9399999999999999,"15min":4.2380000000000004},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3180000000000001,"15min":1.351},"min":{"1min":0.92300000000000004,"5min":0.82199999999999995,"15min":0.69599999999999995},"max":{"1min":1.754,"5min":1.806,"15min":4.4269999999999996},"last":1.6319999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3859999999999999,"15min":1.3420000000000001},"min":{"1min":0.94599999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.829,"5min":2.0609999999999999,"15min":2.3029999999999999},"last":1.415},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.47,"15min":1.47},"min":{"1min":0.89600000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.7090000000000001,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.784}]},{"osd":5,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.3400000000000001,"15min":1.3129999999999999},"min":{"1min":0.997,"5min":0.755,"15min":0.755},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.0910000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4770000000000001,"15min":1.4470000000000001},"min":{"1min":1.147,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.7869999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.5309999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:51:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4610000000000001,"15min":1.446},"min":{"1min":1.0229999999999999,"5min":0.878,"15min":0.878},"max":{"1min":1.841,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.4910000000000001,"15min":1.466},"min":{"1min":1.244,"5min":1.0640000000000001,"15min":0.96399999999999997},"max":{"1min":2.052,"5min":2.1230000000000002,"15min":2.2610000000000001},"last":1.742}]}]},{"osd":5,"up_from":14,"seq":60129542374,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":754007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1000000000000001,"15min":1.1579999999999999},"min":{"1min":0.79900000000000004,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.4870000000000001,"5min":1.8480000000000001,"15min":2.2690000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1759999999999999,"15min":1.1779999999999999},"min":{"1min":0.72499999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.6060000000000001,"5min":1.6579999999999999,"15min":1.7689999999999999},"last":1.365}]},{"osd":1,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.272,"15min":1.26},"min":{"1min":0.89100000000000001,"5min":0.56699999999999995,"15min":0.41299999999999998},"max":{"1min":1.7829999999999999,"5min":1.8080000000000001,"15min":1.8089999999999999},"last":1.079},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1859999999999999,"15min":1.2190000000000001},"min":{"1min":0.89000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.7889999999999999},"last":0.95599999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.274,"15min":1.2929999999999999},"min":{"1min":0.79300000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.9710000000000001,"5min":2.0059999999999998,"15min":2.0259999999999998},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.21,"15min":1.266},"min":{"1min":0.94399999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.2370000000000001},"last":1.0169999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2969999999999999,"15min":1.3089999999999999},"min":{"1min":0.96699999999999997,"5min":0.73099999999999998,"15min":0.66000000000000003},"max":{"1min":1.5680000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.232},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.381,"15min":1.3879999999999999},"min":{"1min":1.1120000000000001,"5min":0.62,"15min":0.62},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.2160000000000002},"last":1.6419999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.1970000000000001,"15min":1.2050000000000001},"min":{"1min":0.88600000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.589},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3580000000000001,"15min":1.321},"min":{"1min":0.78900000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.9510000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.5509999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3500000000000001,"15min":1.3340000000000001},"min":{"1min":0.95699999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":1.2170000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.415}]},{"osd":7,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.448,"15min":1.4470000000000001},"min":{"1min":1.145,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.5249999999999999,"15min":1.512},"min":{"1min":1.167,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.772}]}]},{"osd":4,"up_from":14,"seq":60129542374,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59108,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312732,"statfs":{"total":96636764160,"available":96576237568,"internally_reserved":0,"allocated":1597440,"data_stored":755673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1259999999999999,"15min":1.1919999999999999},"min":{"1min":0.61299999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.696,"5min":1.766,"15min":2.2200000000000002},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.105,"15min":1.1539999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.1220000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1240000000000001,"15min":1.177},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.732,"5min":1.7410000000000001,"15min":1.8400000000000001},"last":1.171},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.167,"15min":1.24},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.4770000000000001,"5min":1.724,"15min":1.9390000000000001},"last":0.78300000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.234,"15min":1.252},"min":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.651,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.194,"5min":1.2430000000000001,"15min":1.3129999999999999},"min":{"1min":0.76300000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.571,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.198}]},{"osd":3,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.2230000000000001,"15min":1.2649999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.3089999999999999,"15min":1.3580000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7669999999999999,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.0660000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3400000000000001,"15min":1.2989999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.669,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.292,"15min":1.2849999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.851,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.262}]},{"osd":6,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3700000000000001,"15min":1.3680000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.841,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4590000000000001,"15min":1.472},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.744,"5min":2.165,"15min":2.165},"last":1.603}]},{"osd":7,"last update":"Mon Oct 16 12:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4490000000000001,"15min":1.466},"min":{"1min":0.83299999999999996,"5min":0.67600000000000005,"15min":0.60399999999999998},"max":{"1min":1.8340000000000001,"5min":2.109,"15min":2.109},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.573,"15min":1.581},"min":{"1min":0.80900000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9379999999999999,"5min":2.423,"15min":2.423},"last":1.655}]}]},{"osd":3,"up_from":14,"seq":60129542374,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":755571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.93100000000000005,"15min":0.96699999999999997},"min":{"1min":0.71999999999999997,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":1.4430000000000001,"5min":1.5649999999999999,"15min":2.0419999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1060000000000001,"15min":1.171},"min":{"1min":0.93799999999999994,"5min":0.44900000000000001,"15min":0.34799999999999998},"max":{"1min":1.5189999999999999,"5min":1.5800000000000001,"15min":2.2749999999999999},"last":0.81399999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.98699999999999999,"15min":1.0209999999999999},"min":{"1min":0.81799999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.758},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.0269999999999999,"15min":1.0780000000000001},"min":{"1min":0.82099999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.784,"5min":1.784,"15min":2.4580000000000002},"last":0.99099999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.002,"15min":1.0600000000000001},"min":{"1min":0.76600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.679,"5min":1.679,"15min":1.9510000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.546,"5min":1.2689999999999999,"15min":1.353},"min":{"1min":1.1319999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.139}]},{"osd":4,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.4119999999999999,"15min":1.4750000000000001},"min":{"1min":0.98699999999999999,"5min":0.78700000000000003,"15min":0.77600000000000002},"max":{"1min":1.726,"5min":1.9239999999999999,"15min":1.956},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.369,"15min":1.4059999999999999},"min":{"1min":1.0109999999999999,"5min":0.80500000000000005,"15min":0.64800000000000002},"max":{"1min":1.8220000000000001,"5min":1.8939999999999999,"15min":2.0099999999999998},"last":1.2470000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.3280000000000001,"15min":1.3779999999999999},"min":{"1min":1.056,"5min":0.70299999999999996,"15min":0.65900000000000003},"max":{"1min":1.893,"5min":2.028,"15min":2.0539999999999998},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4279999999999999,"15min":1.466},"min":{"1min":1.1160000000000001,"5min":0.877,"15min":0.877},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.3540000000000001},"last":1.1910000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.405,"15min":1.4390000000000001},"min":{"1min":1.0860000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.998,"5min":1.998,"15min":2.3220000000000001},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.3779999999999999,"15min":1.4159999999999999},"min":{"1min":1.1200000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2,"5min":2.093,"15min":2.3740000000000001},"last":1.042}]},{"osd":7,"last update":"Mon Oct 16 12:52:04 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.5149999999999999,"15min":1.538},"min":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.157,"5min":2.157,"15min":2.431},"last":1.069},{"interface":"front","average":{"1min":1.651,"5min":1.536,"15min":1.6000000000000001},"min":{"1min":1.3380000000000001,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.552},"last":1.4139999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28556,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343284,"statfs":{"total":96636764160,"available":96607522816,"internally_reserved":0,"allocated":1568768,"data_stored":747671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.877,"15min":0.92200000000000004},"min":{"1min":0.55700000000000005,"5min":0.315,"15min":0.315},"max":{"1min":1.508,"5min":1.508,"15min":1.7889999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.034,"15min":1.101},"min":{"1min":0.70399999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.643,"5min":1.746,"15min":2.085},"last":0.67600000000000005}]},{"osd":1,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.72299999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6599999999999999,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.1479999999999999,"15min":1.226},"min":{"1min":0.96299999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.0510000000000002,"5min":2.347,"15min":2.4289999999999998},"last":1.1100000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.0229999999999999,"15min":1.0820000000000001},"min":{"1min":0.76600000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5649999999999999,"5min":1.5860000000000001,"15min":1.859},"last":1.3},{"interface":"front","average":{"1min":1.405,"5min":1.2729999999999999,"15min":1.3240000000000001},"min":{"1min":0.89100000000000001,"5min":0.434,"15min":0.434},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":1.2509999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.411,"15min":1.46},"min":{"1min":1.1100000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.105,"5min":2.431,"15min":2.431},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.3500000000000001,"15min":1.4319999999999999},"min":{"1min":0.84699999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6459999999999999,"5min":2.0670000000000002,"15min":2.2530000000000001},"last":1.468}]},{"osd":5,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":1.1559999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.9350000000000001,"5min":2.117,"15min":2.3250000000000002},"last":1.363},{"interface":"front","average":{"1min":1.446,"5min":1.3440000000000001,"15min":1.4199999999999999},"min":{"1min":0.995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.8080000000000001,"5min":1.923,"15min":2.4620000000000002},"last":1.504}]},{"osd":6,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.3979999999999999,"15min":1.4470000000000001},"min":{"1min":1.026,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.9950000000000001,"5min":2.181,"15min":2.5129999999999999},"last":1.421},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.508,"15min":1.5620000000000001},"min":{"1min":1.2669999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.298,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.167}]},{"osd":7,"last update":"Mon Oct 16 12:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.522,"15min":1.6080000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.931,"5min":2.202,"15min":2.5339999999999998},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.4790000000000001,"15min":1.544},"min":{"1min":1.101,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.573},"last":1.5629999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542372,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59088,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312752,"statfs":{"total":96636764160,"available":96576258048,"internally_reserved":0,"allocated":1576960,"data_stored":753937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.91700000000000004,"15min":0.91000000000000003},"min":{"1min":0.439,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.345,"5min":1.6359999999999999,"15min":2.0840000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.997,"15min":1.0760000000000001},"min":{"1min":0.47399999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.5669999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.2829999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.92900000000000005,"15min":0.98299999999999998},"min":{"1min":0.56100000000000005,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":1.694,"5min":1.694,"15min":2.262},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.135,"15min":1.1870000000000001},"min":{"1min":0.66900000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.754,"5min":1.9910000000000001,"15min":2.2210000000000001},"last":1.6639999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2350000000000001,"15min":1.302},"min":{"1min":0.63600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.151,"15min":1.2130000000000001},"min":{"1min":0.59799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.7230000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.712}]},{"osd":4,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2569999999999999,"15min":1.302},"min":{"1min":0.71399999999999997,"5min":0.59599999999999997,"15min":0.54600000000000004},"max":{"1min":1.7230000000000001,"5min":2.1030000000000002,"15min":2.1030000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.429,"15min":1.4410000000000001},"min":{"1min":0.92700000000000005,"5min":0.67900000000000005,"15min":0.60599999999999998},"max":{"1min":2.214,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.4550000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.365,"15min":1.423},"min":{"1min":1.1060000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.6679999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.9410000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.399,"15min":1.4330000000000001},"min":{"1min":0.999,"5min":0.753,"15min":0.70999999999999996},"max":{"1min":2.0089999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.2190000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5009999999999999,"15min":1.528},"min":{"1min":0.878,"5min":0.72099999999999997,"15min":0.68999999999999995},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":1.861},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4770000000000001,"15min":1.514},"min":{"1min":0.92500000000000004,"5min":0.71399999999999997,"15min":0.70899999999999996},"max":{"1min":1.867,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.798}]},{"osd":7,"last update":"Mon Oct 16 12:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5669999999999999,"15min":1.6180000000000001},"min":{"1min":0.97899999999999998,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.9570000000000001,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5369999999999999,"15min":1.5609999999999999},"min":{"1min":1.119,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.1280000000000001,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.371}]}]},{"osd":0,"up_from":14,"seq":60129542373,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37464,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334376,"statfs":{"total":96636764160,"available":96598401024,"internally_reserved":0,"allocated":2174976,"data_stored":1344887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.88700000000000001},"min":{"1min":0.65400000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.4430000000000001,"5min":1.655,"15min":2.5720000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.222,"5min":1.0549999999999999,"15min":1.0960000000000001},"min":{"1min":0.77300000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.708,"5min":1.8979999999999999,"15min":2.1800000000000002},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.101,"15min":1.135},"min":{"1min":0.88,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7250000000000001,"5min":1.9339999999999999,"15min":2.7650000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.109,"15min":1.1699999999999999},"min":{"1min":0.91400000000000003,"5min":0.496,"15min":0.496},"max":{"1min":1.623,"5min":1.788,"15min":1.952},"last":1.1719999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.0049999999999999,"15min":1.0900000000000001},"min":{"1min":0.78200000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.585,"5min":1.9570000000000001,"15min":3.0339999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.216,"15min":1.2929999999999999},"min":{"1min":0.91500000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.7110000000000001,"5min":1.9399999999999999,"15min":3.6579999999999999},"last":1.073}]},{"osd":4,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.272,"15min":1.337},"min":{"1min":0.93000000000000005,"5min":0.71499999999999997,"15min":0.56499999999999995},"max":{"1min":1.6839999999999999,"5min":1.8779999999999999,"15min":2.6920000000000002},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3879999999999999,"15min":1.4399999999999999},"min":{"1min":1.177,"5min":0.625,"15min":0.59699999999999998},"max":{"1min":1.6870000000000001,"5min":1.966,"15min":3.544},"last":1.444}]},{"osd":5,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.383,"15min":1.4279999999999999},"min":{"1min":1.232,"5min":0.74299999999999999,"15min":0.72499999999999998},"max":{"1min":1.9630000000000001,"5min":2.024,"15min":2.2440000000000002},"last":1.393},{"interface":"front","average":{"1min":1.397,"5min":1.353,"15min":1.419},"min":{"1min":1.1799999999999999,"5min":0.65400000000000003,"15min":0.58999999999999997},"max":{"1min":1.714,"5min":1.978,"15min":2.8610000000000002},"last":1.351}]},{"osd":6,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.482,"15min":1.5349999999999999},"min":{"1min":1.2230000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9239999999999999,"5min":2.125,"15min":3.7890000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.423,"15min":1.5169999999999999},"min":{"1min":1.1779999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.8839999999999999,"5min":2.0299999999999998,"15min":2.363},"last":1.4610000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.5469999999999999,"15min":1.6279999999999999},"min":{"1min":1.4279999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0249999999999999,"5min":2.1080000000000001,"15min":2.4729999999999999},"last":1.788},{"interface":"front","average":{"1min":1.496,"5min":1.4890000000000001,"15min":1.5569999999999999},"min":{"1min":1.157,"5min":0.79800000000000004,"15min":0.755},"max":{"1min":1.851,"5min":2.3290000000000002,"15min":3.1989999999999998},"last":1.7569999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:52:38.011 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:52:38.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:52:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.289+0000 7f0682708640 1 Processor -- start 2023-10-16T12:52:38.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.289+0000 7f0682708640 1 -- start start 2023-10-16T12:52:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.291+0000 7f0682708640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f067c0ccfc0 0x7f067c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.291+0000 7f0682708640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c15e3c0 con 0x7f067c1615d0 2023-10-16T12:52:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.291+0000 7f0682708640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c15e0e0 con 0x7f067c15db00 2023-10-16T12:52:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.291+0000 7f0682708640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c160af0 con 0x7f067c0ccfc0 2023-10-16T12:52:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.291+0000 7f067bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f067c0ccfc0 0x7f067c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f067c1615d0 0x7f067c15d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46804/0 (socket says 172.21.15.60:46804) 2023-10-16T12:52:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067b7fe640 1 -- 172.21.15.60:0/2285437879 learned_addr learned my addr 172.21.15.60:0/2285437879 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067affd640 1 -- 172.21.15.60:0/2285437879 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364354905 0 0) 0x7f067c15e3c0 con 0x7f067c1615d0 2023-10-16T12:52:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067affd640 1 -- 172.21.15.60:0/2285437879 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664003520 con 0x7f067c1615d0 2023-10-16T12:52:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067bfff640 1 -- 172.21.15.60:0/2285437879 >> v1:172.21.15.144:6789/0 conn(0x7f067c15db00 legacy=0x7f067c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067bfff640 1 -- 172.21.15.60:0/2285437879 >> v1:172.21.15.60:6789/0 conn(0x7f067c1615d0 legacy=0x7f067c15d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.292+0000 7f067bfff640 1 -- 172.21.15.60:0/2285437879 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c160dd0 con 0x7f067c0ccfc0 2023-10-16T12:52:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.293+0000 7f067bfff640 1 --2- 172.21.15.60:0/2285437879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c0ccfc0 0x7f067c0b0410 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f066c0097a0 tx=0x7f066c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00d801d2c780d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.294+0000 7f067affd640 1 -- 172.21.15.60:0/2285437879 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c02e070 con 0x7f067c0ccfc0 2023-10-16T12:52:38.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.294+0000 7f067affd640 1 -- 172.21.15.60:0/2285437879 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c0315a0 con 0x7f067c0ccfc0 2023-10-16T12:52:38.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.294+0000 7f067affd640 1 -- 172.21.15.60:0/2285437879 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c031930 con 0x7f067c0ccfc0 2023-10-16T12:52:38.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 -- 172.21.15.60:0/2285437879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c0ccfc0 msgr2=0x7f067c0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 --2- 172.21.15.60:0/2285437879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c0ccfc0 0x7f067c0b0410 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f066c0097a0 tx=0x7f066c02b840 comp rx=0 tx=0).stop 2023-10-16T12:52:38.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 -- 172.21.15.60:0/2285437879 shutdown_connections 2023-10-16T12:52:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 --2- 172.21.15.60:0/2285437879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c0ccfc0 0x7f067c0b0410 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 -- 172.21.15.60:0/2285437879 >> 172.21.15.60:0/2285437879 conn(0x7f067c0cb410 msgr2=0x7f067c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 -- 172.21.15.60:0/2285437879 shutdown_connections 2023-10-16T12:52:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.295+0000 7f0682708640 1 -- 172.21.15.60:0/2285437879 wait complete. 2023-10-16T12:52:38.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.297+0000 7f0682708640 1 Processor -- start 2023-10-16T12:52:38.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.297+0000 7f0682708640 1 -- start start 2023-10-16T12:52:38.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0682708640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f067c15db00 0x7f067c156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0682708640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 0x7f067c1e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0682708640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f067c1e8ea0 0x7f067c1f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0682708640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c1f7460 con 0x7f067c1615d0 2023-10-16T12:52:38.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0682708640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c1e9330 con 0x7f067c15db00 2023-10-16T12:52:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0682708640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c0c91a0 con 0x7f067c1e8ea0 2023-10-16T12:52:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f067b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 0x7f067c1e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f067b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 0x7f067c1e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38200/0 (socket says 172.21.15.60:38200) 2023-10-16T12:52:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f067b7fe640 1 -- 172.21.15.60:0/4027292162 learned_addr learned my addr 172.21.15.60:0/4027292162 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:38.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f067bfff640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f067c15db00 0x7f067c156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.298+0000 7f0680ca5640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f067c1e8ea0 0x7f067c1f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f067b7fe640 1 -- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f067c1e8ea0 msgr2=0x7f067c1f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f067b7fe640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f067c1e8ea0 0x7f067c1f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f067b7fe640 1 -- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f067c15db00 msgr2=0x7f067c156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f067b7fe640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f067c15db00 0x7f067c156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f067b7fe640 1 -- 172.21.15.60:0/4027292162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c009400 con 0x7f067c1615d0 2023-10-16T12:52:38.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f0680ca5640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f067c1e8ea0 0x7f067c1f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:38.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f067b7fe640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 0x7f067c1e8960 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f067000c440 tx=0x7f067000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:38.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670005ce0 con 0x7f067c1615d0 2023-10-16T12:52:38.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0670005e40 con 0x7f067c1615d0 2023-10-16T12:52:38.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c1f7180 con 0x7f067c1615d0 2023-10-16T12:52:38.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700072b0 con 0x7f067c1615d0 2023-10-16T12:52:38.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.299+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c0c9700 con 0x7f067c1615d0 2023-10-16T12:52:38.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.301+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c04e160 con 0x7f067c1615d0 2023-10-16T12:52:38.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.302+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0670007410 con 0x7f067c1615d0 2023-10-16T12:52:38.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.302+0000 7f0678ff9640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f066806afc0 0x7f066806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.303+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0670088d20 con 0x7f067c1615d0 2023-10-16T12:52:38.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.303+0000 7f067bfff640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f066806afc0 0x7f066806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.304+0000 7f067bfff640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f066806afc0 0x7f066806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f066c036000 tx=0x7f066c03d040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:38.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.307+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0670053ce0 con 0x7f067c1615d0 2023-10-16T12:52:38.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.575+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f067c15fee0 con 0x7f067c1615d0 2023-10-16T12:52:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.580+0000 7f0678ff9640 1 -- 172.21.15.60:0/4027292162 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0670017020 con 0x7f067c1615d0 2023-10-16T12:52:38.582 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:52:38.582 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:52:38.582 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:52:38.582 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:52:38.582 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:52:38.583 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:52:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.582+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f066806afc0 msgr2=0x7f066806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.582+0000 7f0682708640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f066806afc0 0x7f066806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f066c036000 tx=0x7f066c03d040 comp rx=0 tx=0).stop 2023-10-16T12:52:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 msgr2=0x7f067c1e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 0x7f067c1e8960 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f067000c440 tx=0x7f067000c900 comp rx=0 tx=0).stop 2023-10-16T12:52:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 shutdown_connections 2023-10-16T12:52:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f066806afc0 0x7f066806d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f067c1e8ea0 0x7f067c1f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f067c1615d0 0x7f067c1e8960 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 --2- 172.21.15.60:0/4027292162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f067c15db00 0x7f067c156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 >> 172.21.15.60:0/4027292162 conn(0x7f067c0cb410 msgr2=0x7f067c06f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 shutdown_connections 2023-10-16T12:52:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.583+0000 7f0682708640 1 -- 172.21.15.60:0/4027292162 wait complete. 2023-10-16T12:52:38.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:52:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.828+0000 7fdb91e7e640 1 Processor -- start 2023-10-16T12:52:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.829+0000 7fdb91e7e640 1 -- start start 2023-10-16T12:52:38.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.830+0000 7fdb91e7e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.830+0000 7fdb91e7e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb8c0cded0 0x7fdb8c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.830+0000 7fdb91e7e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c15d150 con 0x7fdb8c0b8480 2023-10-16T12:52:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.830+0000 7fdb91e7e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c0d1180 con 0x7fdb8c0cded0 2023-10-16T12:52:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.830+0000 7fdb91e7e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0d1460 con 0x7fdb8c15cd70 2023-10-16T12:52:38.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.830+0000 7fdb9167d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb8c0cded0 0x7fdb8c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb9167d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb8c0cded0 0x7fdb8c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38210/0 (socket says 172.21.15.60:38210) 2023-10-16T12:52:38.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb9167d640 1 -- 172.21.15.60:0/3730746476 learned_addr learned my addr 172.21.15.60:0/3730746476 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:38.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb90e7c640 1 --2- 172.21.15.60:0/3730746476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb9167d640 1 -- 172.21.15.60:0/3730746476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 msgr2=0x7fdb8c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:38.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb9167d640 1 --2- 172.21.15.60:0/3730746476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb9167d640 1 -- 172.21.15.60:0/3730746476 >> v1:172.21.15.144:6789/0 conn(0x7fdb8c0b8480 legacy=0x7fdb8c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:38.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.831+0000 7fdb9167d640 1 -- 172.21.15.60:0/3730746476 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c0d1740 con 0x7fdb8c0cded0 2023-10-16T12:52:38.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.832+0000 7fdb9167d640 1 --2- 172.21.15.60:0/3730746476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0cded0 0x7fdb8c0b5c90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009870 tx=0x7fdb8402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d232f6a6d709a6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb8b7fe640 1 -- 172.21.15.60:0/3730746476 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84031640 con 0x7fdb8c0cded0 2023-10-16T12:52:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb8b7fe640 1 -- 172.21.15.60:0/3730746476 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb840317a0 con 0x7fdb8c0cded0 2023-10-16T12:52:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb8b7fe640 1 -- 172.21.15.60:0/3730746476 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84031b30 con 0x7fdb8c0cded0 2023-10-16T12:52:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb91e7e640 1 -- 172.21.15.60:0/3730746476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0cded0 msgr2=0x7fdb8c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/3730746476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0cded0 0x7fdb8c0b5c90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009870 tx=0x7fdb8402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb91e7e640 1 -- 172.21.15.60:0/3730746476 shutdown_connections 2023-10-16T12:52:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/3730746476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0cded0 0x7fdb8c0b5c90 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/3730746476 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.833+0000 7fdb91e7e640 1 -- 172.21.15.60:0/3730746476 >> 172.21.15.60:0/3730746476 conn(0x7fdb8c0cf570 msgr2=0x7fdb8c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.834+0000 7fdb91e7e640 1 -- 172.21.15.60:0/3730746476 shutdown_connections 2023-10-16T12:52:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.834+0000 7fdb91e7e640 1 -- 172.21.15.60:0/3730746476 wait complete. 2023-10-16T12:52:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.835+0000 7fdb91e7e640 1 Processor -- start 2023-10-16T12:52:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.836+0000 7fdb91e7e640 1 -- start start 2023-10-16T12:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.836+0000 7fdb91e7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 0x7fdb8c0c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb91e7e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c0cded0 0x7fdb8c15fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb91e7e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb91e7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0c6c20 con 0x7fdb8c0b8480 2023-10-16T12:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb91e7e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0c8660 con 0x7fdb8c0cded0 2023-10-16T12:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb91e7e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0c6560 con 0x7fdb8c15cd70 2023-10-16T12:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 0x7fdb8c0c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 0x7fdb8c0c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38220/0 (socket says 172.21.15.60:38220) 2023-10-16T12:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 -- 172.21.15.60:0/1009127425 learned_addr learned my addr 172.21.15.60:0/1009127425 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb9167d640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 -- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 msgr2=0x7fdb8c0c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 -- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c0cded0 msgr2=0x7fdb8c15fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c0cded0 0x7fdb8c15fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb90e7c640 1 -- 172.21.15.60:0/1009127425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb840094d0 con 0x7fdb8c0b8480 2023-10-16T12:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.837+0000 7fdb9167d640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.838+0000 7fdb90e7c640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 0x7fdb8c0c7040 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c007e40 tx=0x7fdb7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.838+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c014620 con 0x7fdb8c0b8480 2023-10-16T12:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.838+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c00ee90 con 0x7fdb8c0b8480 2023-10-16T12:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.838+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0149c0 con 0x7fdb8c0b8480 2023-10-16T12:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.838+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c0c6840 con 0x7fdb8c0b8480 2023-10-16T12:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.838+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c0c8940 con 0x7fdb8c0b8480 2023-10-16T12:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.840+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8c0c6e90 con 0x7fdb8c0b8480 2023-10-16T12:52:38.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.841+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb7c014b20 con 0x7fdb8c0b8480 2023-10-16T12:52:38.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.841+0000 7fdb897fa640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdb7806af90 0x7fdb7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:38.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.842+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb7c08a150 con 0x7fdb8c0b8480 2023-10-16T12:52:38.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.842+0000 7fdb8bfff640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdb7806af90 0x7fdb7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:38.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.843+0000 7fdb8bfff640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdb7806af90 0x7fdb7806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0b2c60 tx=0x7fdb800079b0 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:38.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:38.846+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb7c055060 con 0x7fdb8c0b8480 2023-10-16T12:52:39.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.086+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb8c0d1a00 con 0x7fdb8c0b8480 2023-10-16T12:52:39.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.087+0000 7fdb897fa640 1 -- 172.21.15.60:0/1009127425 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fdb8c0d1a00 con 0x7fdb8c0b8480 2023-10-16T12:52:39.089 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:39.089 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:52:32.396002+0000","modified":"2023-10-16T12:52:35.483028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7866":{"gid":7866,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2146889084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2146889084},{"type":"v1","addr":"172.21.15.60:6833","nonce":2146889084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:52:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2023-10-16T12:52:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdb7806af90 msgr2=0x7fdb7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdb7806af90 0x7fdb7806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0b2c60 tx=0x7fdb800079b0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 msgr2=0x7fdb8c0c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 0x7fdb8c0c7040 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c007e40 tx=0x7fdb7c00e4b0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 shutdown_connections 2023-10-16T12:52:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fdb7806af90 0x7fdb7806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb8c15cd70 0x7fdb8c0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb8c0cded0 0x7fdb8c15fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 --2- 172.21.15.60:0/1009127425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb8c0b8480 0x7fdb8c0c7040 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.090+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 >> 172.21.15.60:0/1009127425 conn(0x7fdb8c0cf570 msgr2=0x7fdb8c15bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.091+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 shutdown_connections 2023-10-16T12:52:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.091+0000 7fdb91e7e640 1 -- 172.21.15.60:0/1009127425 wait complete. 2023-10-16T12:52:39.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:52:32.396002+0000', 'modified': '2023-10-16T12:52:35.483028+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7866}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7866': {'gid': 7866, 'name': 'c', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2146889084', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2146889084}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2146889084}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:52:39.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:52:39.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T12:52:39.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.753+0000 7f926cad1640 1 Processor -- start 2023-10-16T12:52:39.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.753+0000 7f926cad1640 1 -- start start 2023-10-16T12:52:39.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.755+0000 7f926cad1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.755+0000 7f926cad1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.755+0000 7f926cad1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926815d150 con 0x7f926815cd70 2023-10-16T12:52:39.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.755+0000 7f926cad1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92680d1180 con 0x7f92680d0be0 2023-10-16T12:52:39.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.755+0000 7f926cad1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92680d1460 con 0x7f92680b8480 2023-10-16T12:52:39.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f92677fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f926815cd70 0x7f92680d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42840/0 (socket says 172.21.15.60:42840) 2023-10-16T12:52:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f92677fe640 1 -- 172.21.15.60:0/1458693178 learned_addr learned my addr 172.21.15.60:0/1458693178 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f925ffff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9267fff640 1 --2- 172.21.15.60:0/1458693178 >> v2:172.21.15.144:3300/0 conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009341092 0 0) 0x7f926815d150 con 0x7f926815cd70 2023-10-16T12:52:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9254003520 con 0x7f926815cd70 2023-10-16T12:52:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2742206163 0 0) 0x7f9254003520 con 0x7f926815cd70 2023-10-16T12:52:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 msgr2=0x7f92680cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 >> v2:172.21.15.144:3300/0 conn(0x7f92680d0be0 msgr2=0x7f92680cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.756+0000 7f9266ffd640 1 --2- 172.21.15.60:0/1458693178 >> v2:172.21.15.144:3300/0 conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680d1740 con 0x7f926815cd70 2023-10-16T12:52:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9258001670 con 0x7f926815cd70 2023-10-16T12:52:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 >> v1:172.21.15.60:6789/0 conn(0x7f926815cd70 legacy=0x7f92680d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9254003970 0x7f9254003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92540047c0 0x7f9254004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f9254005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680d1180 con 0x7f92540047c0 2023-10-16T12:52:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680d1460 con 0x7f9254003970 2023-10-16T12:52:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925400b980 con 0x7f92680b8480 2023-10-16T12:52:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f925ffff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92540047c0 0x7f9254004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f92677fe640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9254003970 0x7f9254003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f9267fff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f9254005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f925ffff640 1 -- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 msgr2=0x7f9254005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f925ffff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f9254005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f925ffff640 1 -- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9254003970 msgr2=0x7f9254003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.757+0000 7f925ffff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9254003970 0x7f9254003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.758+0000 7f925ffff640 1 -- 172.21.15.60:0/1458693178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9254003520 con 0x7f92540047c0 2023-10-16T12:52:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.758+0000 7f9267fff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f92680b8480 0x7f9254005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.758+0000 7f92677fe640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9254003970 0x7f9254003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.759+0000 7f925ffff640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92540047c0 0x7f9254004c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9250009ac0 tx=0x7f92540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc95ccf5cb3241ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.759+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925002c6b0 con 0x7f92540047c0 2023-10-16T12:52:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.759+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925002c810 con 0x7f92540047c0 2023-10-16T12:52:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.759+0000 7f9266ffd640 1 -- 172.21.15.60:0/1458693178 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925002cba0 con 0x7f92540047c0 2023-10-16T12:52:39.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f926cad1640 1 -- 172.21.15.60:0/1458693178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92540047c0 msgr2=0x7f9254004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f926cad1640 1 --2- 172.21.15.60:0/1458693178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92540047c0 0x7f9254004c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9250009ac0 tx=0x7f92540057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f9267fff640 1 -- 172.21.15.60:0/1458693178 reap_dead start 2023-10-16T12:52:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f926cad1640 1 -- 172.21.15.60:0/1458693178 shutdown_connections 2023-10-16T12:52:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f926cad1640 1 -- 172.21.15.60:0/1458693178 >> 172.21.15.60:0/1458693178 conn(0x7f92680cf570 msgr2=0x7f92680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f926cad1640 1 -- 172.21.15.60:0/1458693178 shutdown_connections 2023-10-16T12:52:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.760+0000 7f926cad1640 1 -- 172.21.15.60:0/1458693178 wait complete. 2023-10-16T12:52:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.762+0000 7f926cad1640 1 Processor -- start 2023-10-16T12:52:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.762+0000 7f926cad1640 1 -- start start 2023-10-16T12:52:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f926cad1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92540047c0 0x7f9268156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f926cad1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 0x7f9268159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f926cad1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926815a3d0 0x7f9268157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f926cad1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92681f2170 con 0x7f92681528f0 2023-10-16T12:52:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f926cad1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926815a860 con 0x7f92540047c0 2023-10-16T12:52:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f926cad1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92681568b0 con 0x7f926815a3d0 2023-10-16T12:52:39.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f925ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 0x7f9268159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f925ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 0x7f9268159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58346/0 (socket says 172.21.15.60:58346) 2023-10-16T12:52:39.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.763+0000 7f925ffff640 1 -- 172.21.15.60:0/745961187 learned_addr learned my addr 172.21.15.60:0/745961187 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:39.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f92677fe640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92540047c0 0x7f9268156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f9267fff640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926815a3d0 0x7f9268157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f925ffff640 1 -- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926815a3d0 msgr2=0x7f9268157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f925ffff640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926815a3d0 0x7f9268157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f925ffff640 1 -- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92540047c0 msgr2=0x7f9268156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f925ffff640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92540047c0 0x7f9268156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f925ffff640 1 -- 172.21.15.60:0/745961187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680ca5d0 con 0x7f92681528f0 2023-10-16T12:52:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f925ffff640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 0x7f9268159e90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9250000c00 tx=0x7f9250004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.764+0000 7f9264ff9640 1 -- 172.21.15.60:0/745961187 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250003610 con 0x7f92681528f0 2023-10-16T12:52:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.765+0000 7f9264ff9640 1 -- 172.21.15.60:0/745961187 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250003790 con 0x7f92681528f0 2023-10-16T12:52:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.765+0000 7f9264ff9640 1 -- 172.21.15.60:0/745961187 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250003b40 con 0x7f92681528f0 2023-10-16T12:52:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.765+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92680ccad0 con 0x7f92681528f0 2023-10-16T12:52:39.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.765+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92681593a0 con 0x7f92681528f0 2023-10-16T12:52:39.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.767+0000 7f9264ff9640 1 -- 172.21.15.60:0/745961187 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f925003b070 con 0x7f92681528f0 2023-10-16T12:52:39.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.768+0000 7f9264ff9640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f924806af90 0x7f924806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.768+0000 7f9264ff9640 1 -- 172.21.15.60:0/745961187 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92500aad40 con 0x7f92681528f0 2023-10-16T12:52:39.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.769+0000 7f92677fe640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f924806af90 0x7f924806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.769+0000 7f92709a2740 1 Processor -- start 2023-10-16T12:52:39.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.769+0000 7f92709a2740 1 -- start start 2023-10-16T12:52:39.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92677fe640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f924806af90 0x7f924806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f926815aa50 tx=0x7f9258009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d72070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92709a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 0x562720da2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92709a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720c927a0 con 0x562720d725b0 2023-10-16T12:52:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720ce4740 con 0x562720d9f170 2023-10-16T12:52:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562720ce1e30 con 0x562720d9f550 2023-10-16T12:52:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 0x562720da2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 0x562720da2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58358/0 (socket says 172.21.15.60:58358) 2023-10-16T12:52:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f9267fff640 1 -- 172.21.15.60:0/2388245653 learned_addr learned my addr 172.21.15.60:0/2388245653 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f925ffff640 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.770+0000 7f92677fe640 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d72070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f9267fff640 1 -- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 msgr2=0x562720da1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f9267fff640 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f9267fff640 1 -- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 msgr2=0x562720d72070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f9267fff640 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d72070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f9267fff640 1 -- 172.21.15.60:0/2388245653 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562720ca1910 con 0x562720d725b0 2023-10-16T12:52:39.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f9267fff640 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 0x562720da2440 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f92680b24f0 tx=0x7f9260009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225c4774273daa13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f925d7fa640 1 -- 172.21.15.60:0/2388245653 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260009bf0 con 0x562720d725b0 2023-10-16T12:52:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f925d7fa640 1 -- 172.21.15.60:0/2388245653 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926000f070 con 0x562720d725b0 2023-10-16T12:52:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.771+0000 7f925d7fa640 1 -- 172.21.15.60:0/2388245653 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92600153f0 con 0x562720d725b0 2023-10-16T12:52:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 -- 172.21.15.60:0/2388245653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 msgr2=0x562720da2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 0x562720da2440 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f92680b24f0 tx=0x7f9260009750 comp rx=0 tx=0).stop 2023-10-16T12:52:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 -- 172.21.15.60:0/2388245653 shutdown_connections 2023-10-16T12:52:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720d725b0 0x562720da2440 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 --2- 172.21.15.60:0/2388245653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d72070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 -- 172.21.15.60:0/2388245653 >> 172.21.15.60:0/2388245653 conn(0x562720d74380 msgr2=0x562720dbcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 -- 172.21.15.60:0/2388245653 shutdown_connections 2023-10-16T12:52:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.772+0000 7f92709a2740 1 -- 172.21.15.60:0/2388245653 wait complete. 2023-10-16T12:52:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.773+0000 7f92709a2740 1 -- start start 2023-10-16T12:52:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.773+0000 7f92709a2740 1 Processor -- start 2023-10-16T12:52:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d8ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d8f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720cdd720 con 0x562720da5650 2023-10-16T12:52:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720c7b970 con 0x562720d9f170 2023-10-16T12:52:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562720c7b610 con 0x562720d9f550 2023-10-16T12:52:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f92709a2740 10 client.0 _ll_register_callbacks cb 0x562720c9d9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:52:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d8f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d8f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58364/0 (socket says 172.21.15.60:58364) 2023-10-16T12:52:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.774+0000 7f9267fff640 1 -- 172.21.15.60:0/1250523958 learned_addr learned my addr 172.21.15.60:0/1250523958 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f92677fe640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d8ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f925ffff640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f9267fff640 1 -- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 msgr2=0x562720da5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f9267fff640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f9267fff640 1 -- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 msgr2=0x562720d8ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f9267fff640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d8ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f9267fff640 1 -- 172.21.15.60:0/1250523958 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9260009d50 con 0x562720da5650 2023-10-16T12:52:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f9267fff640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d8f5f0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9260017000 tx=0x7f9260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670e7edc351cf18c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f92467fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:52:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f92467fc640 1 -- 172.21.15.60:0/1250523958 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260009bf0 con 0x562720da5650 2023-10-16T12:52:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f92467fc640 1 -- 172.21.15.60:0/1250523958 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9260027670 con 0x562720da5650 2023-10-16T12:52:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.775+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562720d04160 con 0x562720da5650 2023-10-16T12:52:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92467fc640 1 -- 172.21.15.60:0/1250523958 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9260027830 con 0x562720da5650 2023-10-16T12:52:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 10 client.8016 fetch_fsmap learned FSMap version 62 2023-10-16T12:52:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x562720cdbda0 con 0x562720da5650 2023-10-16T12:52:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92467fc640 1 -- 172.21.15.60:0/1250523958 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562720d04160 con 0x562720da5650 2023-10-16T12:52:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92467fc640 1 -- 172.21.15.60:0/1250523958 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 62) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9260014070 con 0x562720da5650 2023-10-16T12:52:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 10 client.8016 fetch_fsmap finished waiting for FSMap version 62 2023-10-16T12:52:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 10 client.8016 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T12:52:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 20 client.8016 populate_metadata read hostname 'smithi060' 2023-10-16T12:52:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720d8efb0 0x562720da3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 4 client.8016 mds_command: new command op to 7866 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:52:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.776+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 --> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562720cafcc0 con 0x562720d8efb0 2023-10-16T12:52:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.777+0000 7f92677fe640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720d8efb0 0x562720da3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.777+0000 7f92677fe640 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720d8efb0 0x562720da3f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e07ccbe6c087416 server_cookie=c17d3fdc85e5a233 in_seq=0 out_seq=0 2023-10-16T12:52:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.777+0000 7f92467fc640 10 client.8016 ms_handle_connect on v2:172.21.15.60:6832/2146889084 2023-10-16T12:52:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.778+0000 7f92467fc640 1 -- 172.21.15.60:0/1250523958 <== mds.0 v2:172.21.15.60:6832/2146889084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562720cafcc0 con 0x562720d8efb0 2023-10-16T12:52:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.778+0000 7f92467fc640 10 client.8016 handle_command_reply: tid=0 2023-10-16T12:52:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.778+0000 7f92709a2740 1 client.8016 shutdown 2023-10-16T12:52:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.778+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 msgr2=0x562720d8f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.778+0000 7f92709a2740 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d8f5f0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9260017000 tx=0x7f9260004a40 comp rx=0 tx=0).stop 2023-10-16T12:52:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.778+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 shutdown_connections 2023-10-16T12:52:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720d8efb0 0x562720da3f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f550 0x562720da5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d8f5f0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 --2- 172.21.15.60:0/1250523958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f170 0x562720d8ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92467fc640 0 client.8016 ms_handle_reset on v2:172.21.15.60:6832/2146889084 2023-10-16T12:52:39.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 >> 172.21.15.60:0/1250523958 conn(0x562720d725b0 msgr2=0x562720da1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 shutdown_connections 2023-10-16T12:52:39.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 1 -- 172.21.15.60:0/1250523958 wait complete. 2023-10-16T12:52:39.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.779+0000 7f92709a2740 20 client.8016 trim_cache size 0 max 16384 2023-10-16T12:52:39.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.796+0000 7f92709a2740 1 Processor -- start 2023-10-16T12:52:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.796+0000 7f92709a2740 1 -- start start 2023-10-16T12:52:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720db76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720d779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92709a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d77f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92709a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720a41f20 con 0x562720da5650 2023-10-16T12:52:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720db8780 con 0x562720d9f550 2023-10-16T12:52:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562720d78440 con 0x562720d9f170 2023-10-16T12:52:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d77f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d77f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58378/0 (socket says 172.21.15.60:58378) 2023-10-16T12:52:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 -- 172.21.15.60:0/1328699006 learned_addr learned my addr 172.21.15.60:0/1328699006 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f92677fe640 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720db76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 -- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 msgr2=0x562720db76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720db76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 -- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 msgr2=0x562720d779c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720d779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.797+0000 7f9267fff640 1 -- 172.21.15.60:0/1328699006 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562720c7ab80 con 0x562720da5650 2023-10-16T12:52:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.798+0000 7f9267fff640 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d77f00 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f926000b010 tx=0x7f9260020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124476865071ee2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.798+0000 7f925dffb640 1 -- 172.21.15.60:0/1328699006 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926002b040 con 0x562720da5650 2023-10-16T12:52:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.798+0000 7f925dffb640 1 -- 172.21.15.60:0/1328699006 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9260015ab0 con 0x562720da5650 2023-10-16T12:52:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.798+0000 7f925dffb640 1 -- 172.21.15.60:0/1328699006 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92600276b0 con 0x562720da5650 2023-10-16T12:52:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 -- 172.21.15.60:0/1328699006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 msgr2=0x562720d77f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d77f00 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f926000b010 tx=0x7f9260020070 comp rx=0 tx=0).stop 2023-10-16T12:52:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 -- 172.21.15.60:0/1328699006 shutdown_connections 2023-10-16T12:52:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720d77f00 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720d779c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 --2- 172.21.15.60:0/1328699006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720db76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 -- 172.21.15.60:0/1328699006 >> 172.21.15.60:0/1328699006 conn(0x562720d725b0 msgr2=0x562720da4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 -- 172.21.15.60:0/1328699006 shutdown_connections 2023-10-16T12:52:39.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.799+0000 7f92709a2740 1 -- 172.21.15.60:0/1328699006 wait complete. 2023-10-16T12:52:39.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.800+0000 7f92709a2740 1 -- start start 2023-10-16T12:52:39.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.800+0000 7f92709a2740 1 Processor -- start 2023-10-16T12:52:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720e24c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720da1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720dbe290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720dbdf60 con 0x562720da5650 2023-10-16T12:52:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x562720d77c70 con 0x562720d9f550 2023-10-16T12:52:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x562720d73270 con 0x562720d9f170 2023-10-16T12:52:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92709a2740 10 client.0 _ll_register_callbacks cb 0x7f9260009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:52:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720dbe290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f9267fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720dbe290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58388/0 (socket says 172.21.15.60:58388) 2023-10-16T12:52:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f9267fff640 1 -- 172.21.15.60:0/2535348292 learned_addr learned my addr 172.21.15.60:0/2535348292 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.801+0000 7f92677fe640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720e24c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f925ffff640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720da1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f9267fff640 1 -- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 msgr2=0x562720e24c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f9267fff640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720e24c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f9267fff640 1 -- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 msgr2=0x562720da1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:39.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f9267fff640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720da1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f9267fff640 1 -- 172.21.15.60:0/2535348292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562720cdbda0 con 0x562720da5650 2023-10-16T12:52:39.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f9267fff640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720dbe290 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f9260027d30 tx=0x7f926000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d9c5e2cd39d6d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:39.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f92467fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:52:39.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f92467fc640 1 -- 172.21.15.60:0/2535348292 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260034660 con 0x562720da5650 2023-10-16T12:52:39.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f92467fc640 1 -- 172.21.15.60:0/2535348292 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92600347c0 con 0x562720da5650 2023-10-16T12:52:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.802+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f926000f070 con 0x562720da5650 2023-10-16T12:52:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92467fc640 1 -- 172.21.15.60:0/2535348292 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9260034980 con 0x562720da5650 2023-10-16T12:52:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 10 client.8022 fetch_fsmap learned FSMap version 62 2023-10-16T12:52:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x562720ca0170 con 0x562720da5650 2023-10-16T12:52:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92467fc640 1 -- 172.21.15.60:0/2535348292 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926000f070 con 0x562720da5650 2023-10-16T12:52:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92467fc640 1 -- 172.21.15.60:0/2535348292 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 62) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f926003e070 con 0x562720da5650 2023-10-16T12:52:39.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 10 client.8022 fetch_fsmap finished waiting for FSMap version 62 2023-10-16T12:52:39.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 10 client.8022 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T12:52:39.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 20 client.8022 populate_metadata read hostname 'smithi060' 2023-10-16T12:52:39.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720dbe7d0 0x562720dbebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:39.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 4 client.8022 mds_command: new command op to 7866 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:52:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.803+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 --> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562720cd6700 con 0x562720dbe7d0 2023-10-16T12:52:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.804+0000 7f92677fe640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720dbe7d0 0x562720dbebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.804+0000 7f92677fe640 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720dbe7d0 0x562720dbebb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a09740f7d64e4 server_cookie=b67cbe64157efd48 in_seq=0 out_seq=0 2023-10-16T12:52:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.804+0000 7f92467fc640 10 client.8022 ms_handle_connect on v2:172.21.15.60:6832/2146889084 2023-10-16T12:52:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.804+0000 7f92467fc640 1 -- 172.21.15.60:0/2535348292 <== mds.0 v2:172.21.15.60:6832/2146889084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x562720cd6700 con 0x562720dbe7d0 2023-10-16T12:52:39.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.804+0000 7f92467fc640 10 client.8022 handle_command_reply: tid=0 2023-10-16T12:52:39.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.804+0000 7f92709a2740 1 client.8022 shutdown 2023-10-16T12:52:39.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 msgr2=0x562720dbe290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720dbe290 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f9260027d30 tx=0x7f926000b040 comp rx=0 tx=0).stop 2023-10-16T12:52:39.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 shutdown_connections 2023-10-16T12:52:39.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:6832/2146889084,v1:172.21.15.60:6833/2146889084] conn(0x562720dbe7d0 0x562720dbebb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562720da5650 0x562720dbe290 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x562720d9f550 0x562720da1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 --2- 172.21.15.60:0/2535348292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x562720d9f170 0x562720e24c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92467fc640 0 client.8022 ms_handle_reset on v2:172.21.15.60:6832/2146889084 2023-10-16T12:52:39.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 >> 172.21.15.60:0/2535348292 conn(0x562720d725b0 msgr2=0x562720d8cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 shutdown_connections 2023-10-16T12:52:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 1 -- 172.21.15.60:0/2535348292 wait complete. 2023-10-16T12:52:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.805+0000 7f92709a2740 20 client.8022 trim_cache size 0 max 16384 2023-10-16T12:52:39.821 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:52:39.822 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:52:39.822 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:52:39.822 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 7866, 2023-10-16T12:52:39.822 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:52:39.822 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:52:39.823 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:52:39.823 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8467888749999997, 2023-10-16T12:52:39.823 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 62, 2023-10-16T12:52:39.823 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 128, 2023-10-16T12:52:39.823 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 128, 2023-10-16T12:52:39.824 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.648597669000001 2023-10-16T12:52:39.824 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:52:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f924806af90 msgr2=0x7f924806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f924806af90 0x7f924806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f926815aa50 tx=0x7f9258009040 comp rx=0 tx=0).stop 2023-10-16T12:52:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 msgr2=0x7f9268159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 0x7f9268159e90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9250000c00 tx=0x7f9250004730 comp rx=0 tx=0).stop 2023-10-16T12:52:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 shutdown_connections 2023-10-16T12:52:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f924806af90 0x7f924806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926815a3d0 0x7f9268157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92681528f0 0x7f9268159e90 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 --2- 172.21.15.60:0/745961187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92540047c0 0x7f9268156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 >> 172.21.15.60:0/745961187 conn(0x7f92680cf570 msgr2=0x7f92680cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:39.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 shutdown_connections 2023-10-16T12:52:39.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:39.806+0000 7f926cad1640 1 -- 172.21.15.60:0/745961187 wait complete. 2023-10-16T12:52:39.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:52:40.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.051+0000 7fbffc21f640 1 Processor -- start 2023-10-16T12:52:40.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.051+0000 7fbffc21f640 1 -- start start 2023-10-16T12:52:40.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbffc21f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:40.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbffc21f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbff40ce390 0x7fbff4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbffc21f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff415ed60 con 0x7fbff415ae80 2023-10-16T12:52:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbffc21f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff406b810 con 0x7fbff40ce390 2023-10-16T12:52:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbffc21f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40707e0 con 0x7fbff415aaa0 2023-10-16T12:52:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbff97ba640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbff415ae80 0x7fbff40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42850/0 (socket says 172.21.15.60:42850) 2023-10-16T12:52:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbff97ba640 1 -- 172.21.15.60:0/3222748870 learned_addr learned my addr 172.21.15.60:0/3222748870 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.053+0000 7fbffa7bc640 1 --2- 172.21.15.60:0/3222748870 >> v2:172.21.15.60:3300/0 conn(0x7fbff40ce390 0x7fbff4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:40.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbff9fbb640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbffa7bc640 1 -- 172.21.15.60:0/3222748870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 msgr2=0x7fbff415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbffa7bc640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbffa7bc640 1 -- 172.21.15.60:0/3222748870 >> v1:172.21.15.60:6789/0 conn(0x7fbff415ae80 legacy=0x7fbff40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbff8fb9640 1 -- 172.21.15.60:0/3222748870 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568519297 0 0) 0x7fbff415ed60 con 0x7fbff415ae80 2023-10-16T12:52:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbffa7bc640 1 -- 172.21.15.60:0/3222748870 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff415ef90 con 0x7fbff40ce390 2023-10-16T12:52:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.054+0000 7fbff9fbb640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.055+0000 7fbffa7bc640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff40ce390 0x7fbff4070f50 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009d90 tx=0x7fbfec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4438b556bd6716f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.055+0000 7fbff8fb9640 1 -- 172.21.15.60:0/3222748870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec02cc50 con 0x7fbff40ce390 2023-10-16T12:52:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.055+0000 7fbff8fb9640 1 -- 172.21.15.60:0/3222748870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec02cdb0 con 0x7fbff40ce390 2023-10-16T12:52:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.055+0000 7fbff8fb9640 1 -- 172.21.15.60:0/3222748870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec034660 con 0x7fbff40ce390 2023-10-16T12:52:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 -- 172.21.15.60:0/3222748870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff40ce390 msgr2=0x7fbff4070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff40ce390 0x7fbff4070f50 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009d90 tx=0x7fbfec02bea0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 -- 172.21.15.60:0/3222748870 shutdown_connections 2023-10-16T12:52:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff40ce390 0x7fbff4070f50 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 --2- 172.21.15.60:0/3222748870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 -- 172.21.15.60:0/3222748870 >> 172.21.15.60:0/3222748870 conn(0x7fbff40cb410 msgr2=0x7fbff40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 -- 172.21.15.60:0/3222748870 shutdown_connections 2023-10-16T12:52:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.056+0000 7fbffc21f640 1 -- 172.21.15.60:0/3222748870 wait complete. 2023-10-16T12:52:40.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.058+0000 7fbffc21f640 1 Processor -- start 2023-10-16T12:52:40.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.058+0000 7fbffc21f640 1 -- start start 2023-10-16T12:52:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffc21f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbff40ce390 0x7fbff40c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffc21f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffc21f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 0x7fbff40c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffc21f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40b0df0 con 0x7fbff415ae80 2023-10-16T12:52:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffc21f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c6d80 con 0x7fbff40ce390 2023-10-16T12:52:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffc21f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c7060 con 0x7fbff415aaa0 2023-10-16T12:52:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffa7bc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 0x7fbff40c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffa7bc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 0x7fbff40c2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58400/0 (socket says 172.21.15.60:58400) 2023-10-16T12:52:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.059+0000 7fbffa7bc640 1 -- 172.21.15.60:0/2154384140 learned_addr learned my addr 172.21.15.60:0/2154384140 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbff9fbb640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbff40ce390 0x7fbff40c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbff97ba640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffa7bc640 1 -- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 msgr2=0x7fbff41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffa7bc640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffa7bc640 1 -- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbff40ce390 msgr2=0x7fbff40c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffa7bc640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbff40ce390 0x7fbff40c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffa7bc640 1 -- 172.21.15.60:0/2154384140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec009a70 con 0x7fbff415ae80 2023-10-16T12:52:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffa7bc640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 0x7fbff40c2780 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009d90 tx=0x7fbfec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec040650 con 0x7fbff415ae80 2023-10-16T12:52:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec0407d0 con 0x7fbff415ae80 2023-10-16T12:52:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec040b80 con 0x7fbff415ae80 2023-10-16T12:52:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.060+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff40c7340 con 0x7fbff415ae80 2023-10-16T12:52:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.061+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff4157bf0 con 0x7fbff415ae80 2023-10-16T12:52:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.062+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff40c6be0 con 0x7fbff415ae80 2023-10-16T12:52:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.063+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbfec040ce0 con 0x7fbff415ae80 2023-10-16T12:52:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.064+0000 7fbfea7fc640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fbfd806af90 0x7fbfd806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.064+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfec0ac5f0 con 0x7fbff415ae80 2023-10-16T12:52:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.064+0000 7fbff9fbb640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fbfd806af90 0x7fbfd806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.065+0000 7fbff9fbb640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fbfd806af90 0x7fbfd806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbff40cb1c0 tx=0x7fbff0008040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.069+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbff40c6be0 con 0x7fbff415ae80 2023-10-16T12:52:40.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.305+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbff40c6be0 con 0x7fbff415ae80 2023-10-16T12:52:40.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.306+0000 7fbfea7fc640 1 -- 172.21.15.60:0/2154384140 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbff40c6be0 con 0x7fbff415ae80 2023-10-16T12:52:40.308 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:52:40.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.308+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fbfd806af90 msgr2=0x7fbfd806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:40.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.308+0000 7fbffc21f640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fbfd806af90 0x7fbfd806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbff40cb1c0 tx=0x7fbff0008040 comp rx=0 tx=0).stop 2023-10-16T12:52:40.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.308+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 msgr2=0x7fbff40c2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:40.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.308+0000 7fbffc21f640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 0x7fbff40c2780 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009d90 tx=0x7fbfec004a40 comp rx=0 tx=0).stop 2023-10-16T12:52:40.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 shutdown_connections 2023-10-16T12:52:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7fbfd806af90 0x7fbfd806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbff415aaa0 0x7fbff41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbff415ae80 0x7fbff40c2780 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 --2- 172.21.15.60:0/2154384140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbff40ce390 0x7fbff40c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:40.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 >> 172.21.15.60:0/2154384140 conn(0x7fbff40cb410 msgr2=0x7fbff40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:40.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 shutdown_connections 2023-10-16T12:52:40.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:40.309+0000 7fbffc21f640 1 -- 172.21.15.60:0/2154384140 wait complete. 2023-10-16T12:52:40.331 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:52:40.331 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:52:40.331 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:52:40.332 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:52:40.332 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:52:40.332 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:52:40.333 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:52:40.333 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:52:40.333 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:52:40.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:40.334 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:52:40.391 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:52:40.392 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:52:40.392 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:52:40.392 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.392 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:52:40.393 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.393 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:52:40.393 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:52:40.393 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:52:40.393 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:52:40.394 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:52:40.394 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:52:40.394 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:52:40.394 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:52:40.394 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:52:40.395 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:52:40.395 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41227sec preferred_lft 41227sec 2023-10-16T12:52:40.395 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:52:40.395 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.395 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:52:40.396 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:52:40.396 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:52:40.396 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:52:40.396 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:52:40.396 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:52:40.397 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.397 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:52:40.397 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.397 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:52:40.397 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:52:40.398 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:52:40.398 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.398 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:52:40.398 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:52:40.398 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:52:40.399 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:52:40.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:40.400 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:52:40.498 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:52:40.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:40.503 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:52:40.563 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:52:40.564 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:52:40.564 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:52:40.564 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:52:40.564 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:52:40.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:40.566 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:52:40.566 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:52:40.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:52:40.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:52:40.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:52:40.568 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:52:40.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:40.777 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:52:40.835 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:52:40.835 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:52:40.836 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:52:40.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:40.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:52:40.941 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:52:40.941 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:52:41.004 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:52:41.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:52:41.005 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:52:41.056 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:52:41.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:41.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:52:41.197 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:52:41.198 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:52:41.233 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:52:41.233 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:52:41.234 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:52:41.234 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:52:41.234 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:52:41.238 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:52:41.238 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:52:41.238 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:52:41.238 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:52:41.239 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:52:41.242 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:52:41.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:41.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:52:41.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:41.342 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:52:41.411 INFO:teuthology.orchestra.run.smithi060.stdout:client6164 2023-10-16T12:52:41.411 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:52:41.412 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:52:41.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:52:41.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.698+0000 7f75cf490640 1 Processor -- start 2023-10-16T12:52:41.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.699+0000 7f75cf490640 1 -- start start 2023-10-16T12:52:41.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.700+0000 7f75cf490640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 0x7f75c80ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.700+0000 7f75cf490640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c80cb030 con 0x7f75c815ad70 2023-10-16T12:52:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.700+0000 7f75cf490640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c80c91c0 con 0x7f75c806a4b0 2023-10-16T12:52:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.700+0000 7f75cf490640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c94a0 con 0x7f75c80caac0 2023-10-16T12:52:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.700+0000 7f75cd22c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75c815ad70 0x7f75c8069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42852/0 (socket says 172.21.15.60:42852) 2023-10-16T12:52:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.700+0000 7f75cd22c640 1 -- 172.21.15.60:0/1219497379 learned_addr learned my addr 172.21.15.60:0/1219497379 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:41.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75cda2d640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 0x7f75c80ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306903762 0 0) 0x7f75c80cb030 con 0x7f75c815ad70 2023-10-16T12:52:41.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b4003520 con 0x7f75c815ad70 2023-10-16T12:52:41.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44177542 0 0) 0x7f75b4003520 con 0x7f75c815ad70 2023-10-16T12:52:41.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 msgr2=0x7f75c80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:41.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 0x7f75c80ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 >> v1:172.21.15.144:6789/0 conn(0x7f75c806a4b0 legacy=0x7f75c80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80c9780 con 0x7f75c815ad70 2023-10-16T12:52:41.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f75c4001670 con 0x7f75c815ad70 2023-10-16T12:52:41.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 >> v1:172.21.15.60:6789/0 conn(0x7f75c815ad70 legacy=0x7f75c8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 0x7f75b40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004020 0x7f75b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c94a0 con 0x7f75b4004020 2023-10-16T12:52:41.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b400bbc0 con 0x7f75b4004d00 2023-10-16T12:52:41.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.701+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b400bd80 con 0x7f75c80caac0 2023-10-16T12:52:41.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cca2b640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004020 0x7f75b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cda2d640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cd22c640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 0x7f75b40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cca2b640 1 -- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 msgr2=0x7f75b40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:41.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cca2b640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c80caac0 0x7f75b40039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cca2b640 1 -- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b4004d00 msgr2=0x7f75b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:41.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cca2b640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cca2b640 1 -- 172.21.15.60:0/1219497379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4003520 con 0x7f75b4004020 2023-10-16T12:52:41.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.702+0000 7f75cda2d640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:41.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.703+0000 7f75cca2b640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004020 0x7f75b40047c0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f75b8001750 tx=0x7f75b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32db7d8b04db310 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:41.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.703+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80052d0 con 0x7f75b4004020 2023-10-16T12:52:41.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.703+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b802e050 con 0x7f75b4004020 2023-10-16T12:52:41.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.703+0000 7f75bffff640 1 -- 172.21.15.60:0/1219497379 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8003830 con 0x7f75b4004020 2023-10-16T12:52:41.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cf490640 1 -- 172.21.15.60:0/1219497379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004020 msgr2=0x7f75b40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cf490640 1 --2- 172.21.15.60:0/1219497379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004020 0x7f75b40047c0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f75b8001750 tx=0x7f75b40057f0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cda2d640 1 -- 172.21.15.60:0/1219497379 reap_dead start 2023-10-16T12:52:41.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cf490640 1 -- 172.21.15.60:0/1219497379 shutdown_connections 2023-10-16T12:52:41.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cf490640 1 -- 172.21.15.60:0/1219497379 >> 172.21.15.60:0/1219497379 conn(0x7f75c80ce9b0 msgr2=0x7f75c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:41.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cf490640 1 -- 172.21.15.60:0/1219497379 shutdown_connections 2023-10-16T12:52:41.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.704+0000 7f75cf490640 1 -- 172.21.15.60:0/1219497379 wait complete. 2023-10-16T12:52:41.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.706+0000 7f75cf490640 1 Processor -- start 2023-10-16T12:52:41.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.706+0000 7f75cf490640 1 -- start start 2023-10-16T12:52:41.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cf490640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75b4004020 0x7f75c81f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cf490640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 0x7f75c80c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cf490640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c814d690 0x7f75c814db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cf490640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80c50d0 con 0x7f75c814e3b0 2023-10-16T12:52:41.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cf490640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8151550 con 0x7f75c814d690 2023-10-16T12:52:41.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cf490640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c814e060 con 0x7f75b4004020 2023-10-16T12:52:41.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cca2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 0x7f75c80c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.707+0000 7f75cca2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 0x7f75c80c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58422/0 (socket says 172.21.15.60:58422) 2023-10-16T12:52:41.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 -- 172.21.15.60:0/2820802959 learned_addr learned my addr 172.21.15.60:0/2820802959 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:41.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cda2d640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c814d690 0x7f75c814db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cd22c640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75b4004020 0x7f75c81f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 -- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75b4004020 msgr2=0x7f75c81f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75b4004020 0x7f75c81f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 -- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c814d690 msgr2=0x7f75c814db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c814d690 0x7f75c814db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 -- 172.21.15.60:0/2820802959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c814b860 con 0x7f75c814e3b0 2023-10-16T12:52:41.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cda2d640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c814d690 0x7f75c814db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:41.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.708+0000 7f75cca2b640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 0x7f75c80c3eb0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f75b8008fd0 tx=0x7f75b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:41.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.709+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80076c0 con 0x7f75c814e3b0 2023-10-16T12:52:41.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.709+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8007a30 con 0x7f75c814e3b0 2023-10-16T12:52:41.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.709+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8006c00 con 0x7f75c814e3b0 2023-10-16T12:52:41.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.709+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c80c4830 con 0x7f75c814e3b0 2023-10-16T12:52:41.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.709+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c80c4df0 con 0x7f75c814e3b0 2023-10-16T12:52:41.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.710+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c814b630 con 0x7f75c814e3b0 2023-10-16T12:52:41.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.711+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75b80035f0 con 0x7f75c814e3b0 2023-10-16T12:52:41.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.712+0000 7f75bdffb640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f75ac06af90 0x7f75ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:41.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.712+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75b80ab5f0 con 0x7f75c814e3b0 2023-10-16T12:52:41.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.713+0000 7f75cd22c640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f75ac06af90 0x7f75ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:41.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.714+0000 7f75cd22c640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f75ac06af90 0x7f75ac06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000ff0 tx=0x7f75c4009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:41.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.717+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75c814b630 con 0x7f75c814e3b0 2023-10-16T12:52:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.946+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 --> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c80c9ca0 con 0x7f75ac06af90 2023-10-16T12:52:41.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.950+0000 7f75bdffb640 1 -- 172.21.15.60:0/2820802959 <== mgr.6574 v2:172.21.15.144:6832/3337897861 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f75c400d040 con 0x7f75ac06af90 2023-10-16T12:52:41.951 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:52:41.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:52:41.952 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:52:41.952 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:52:41.952 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:52:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f75ac06af90 msgr2=0x7f75ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f75ac06af90 0x7f75ac06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000ff0 tx=0x7f75c4009040 comp rx=0 tx=0).stop 2023-10-16T12:52:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 msgr2=0x7f75c80c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 0x7f75c80c3eb0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f75b8008fd0 tx=0x7f75b8001120 comp rx=0 tx=0).stop 2023-10-16T12:52:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 shutdown_connections 2023-10-16T12:52:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f75ac06af90 0x7f75ac06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c814d690 0x7f75c814db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c814e3b0 0x7f75c80c3eb0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 --2- 172.21.15.60:0/2820802959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75b4004020 0x7f75c81f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.952+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 >> 172.21.15.60:0/2820802959 conn(0x7f75c80ce9b0 msgr2=0x7f75c8159fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.953+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 shutdown_connections 2023-10-16T12:52:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:41.953+0000 7f75cf490640 1 -- 172.21.15.60:0/2820802959 wait complete. 2023-10-16T12:52:41.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:52:42.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.189+0000 7f5215ac2640 1 Processor -- start 2023-10-16T12:52:42.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.189+0000 7f5215ac2640 1 -- start start 2023-10-16T12:52:42.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f5215ac2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f521006a4b0 0x7f52100ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:42.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f5215ac2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100cb030 con 0x7f521015ad70 2023-10-16T12:52:42.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f5215ac2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100c91c0 con 0x7f52100c9ca0 2023-10-16T12:52:42.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f5215ac2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100c94a0 con 0x7f521006a4b0 2023-10-16T12:52:42.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f520f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f521015ad70 0x7f5210069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42866/0 (socket says 172.21.15.60:42866) 2023-10-16T12:52:42.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f520f7fe640 1 -- 172.21.15.60:0/2559073813 learned_addr learned my addr 172.21.15.60:0/2559073813 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:42.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.191+0000 7f520effd640 1 --2- 172.21.15.60:0/2559073813 >> v2:172.21.15.60:3300/0 conn(0x7f521006a4b0 0x7f52100ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:42.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.192+0000 7f520e7fc640 1 -- 172.21.15.60:0/2559073813 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138470287 0 0) 0x7f52100cb030 con 0x7f521015ad70 2023-10-16T12:52:42.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.192+0000 7f520e7fc640 1 -- 172.21.15.60:0/2559073813 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f8003520 con 0x7f521015ad70 2023-10-16T12:52:42.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.192+0000 7f520effd640 1 -- 172.21.15.60:0/2559073813 >> v1:172.21.15.144:6789/0 conn(0x7f52100c9ca0 legacy=0x7f52100ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:42.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.192+0000 7f520effd640 1 -- 172.21.15.60:0/2559073813 >> v1:172.21.15.60:6789/0 conn(0x7f521015ad70 legacy=0x7f5210069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:42.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.192+0000 7f520effd640 1 -- 172.21.15.60:0/2559073813 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100c9780 con 0x7f521006a4b0 2023-10-16T12:52:42.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.193+0000 7f520effd640 1 --2- 172.21.15.60:0/2559073813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521006a4b0 0x7f52100ca3b0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f52040097a0 tx=0x7f520402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd997a7a5818f41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:42.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.193+0000 7f520e7fc640 1 -- 172.21.15.60:0/2559073813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204031830 con 0x7f521006a4b0 2023-10-16T12:52:42.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.193+0000 7f520e7fc640 1 -- 172.21.15.60:0/2559073813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5204031990 con 0x7f521006a4b0 2023-10-16T12:52:42.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.193+0000 7f520e7fc640 1 -- 172.21.15.60:0/2559073813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204031d20 con 0x7f521006a4b0 2023-10-16T12:52:42.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 -- 172.21.15.60:0/2559073813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521006a4b0 msgr2=0x7f52100ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:42.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 --2- 172.21.15.60:0/2559073813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521006a4b0 0x7f52100ca3b0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f52040097a0 tx=0x7f520402b8d0 comp rx=0 tx=0).stop 2023-10-16T12:52:42.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 -- 172.21.15.60:0/2559073813 shutdown_connections 2023-10-16T12:52:42.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 --2- 172.21.15.60:0/2559073813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521006a4b0 0x7f52100ca3b0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:42.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 -- 172.21.15.60:0/2559073813 >> 172.21.15.60:0/2559073813 conn(0x7f52100ce9b0 msgr2=0x7f52100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:42.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 -- 172.21.15.60:0/2559073813 shutdown_connections 2023-10-16T12:52:42.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.194+0000 7f5215ac2640 1 -- 172.21.15.60:0/2559073813 wait complete. 2023-10-16T12:52:42.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.196+0000 7f5215ac2640 1 Processor -- start 2023-10-16T12:52:42.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.196+0000 7f5215ac2640 1 -- start start 2023-10-16T12:52:42.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f5215ac2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521006a4b0 0x7f5210158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:42.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f5215ac2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 0x7f52100ad110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:42.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f5215ac2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f521015ad70 0x7f52100ad650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:42.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f5215ac2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100cce90 con 0x7f52100c9ca0 2023-10-16T12:52:42.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f5215ac2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100cc580 con 0x7f521015ad70 2023-10-16T12:52:42.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f5215ac2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5210158970 con 0x7f521006a4b0 2023-10-16T12:52:42.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f520effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 0x7f52100ad110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:42.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f520effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 0x7f52100ad110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58438/0 (socket says 172.21.15.60:58438) 2023-10-16T12:52:42.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.197+0000 7f520effd640 1 -- 172.21.15.60:0/985921089 learned_addr learned my addr 172.21.15.60:0/985921089 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:42.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520f7fe640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521006a4b0 0x7f5210158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:42.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520ffff640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f521015ad70 0x7f52100ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:42.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520effd640 1 -- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521006a4b0 msgr2=0x7f5210158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:42.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520effd640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521006a4b0 0x7f5210158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:42.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520effd640 1 -- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f521015ad70 msgr2=0x7f52100ad650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:42.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520effd640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f521015ad70 0x7f52100ad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:42.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520effd640 1 -- 172.21.15.60:0/985921089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5204009400 con 0x7f52100c9ca0 2023-10-16T12:52:42.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.198+0000 7f520effd640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 0x7f52100ad110 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f52040098c0 tx=0x7f52040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:42.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.199+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520403a6f0 con 0x7f52100c9ca0 2023-10-16T12:52:42.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.199+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520403a870 con 0x7f52100c9ca0 2023-10-16T12:52:42.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.199+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520403ac20 con 0x7f52100c9ca0 2023-10-16T12:52:42.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.199+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52100c82f0 con 0x7f52100c9ca0 2023-10-16T12:52:42.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.199+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52100ccab0 con 0x7f52100c9ca0 2023-10-16T12:52:42.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.200+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52100ccfd0 con 0x7f52100c9ca0 2023-10-16T12:52:42.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.201+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 55) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5204030380 con 0x7f52100c9ca0 2023-10-16T12:52:42.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.202+0000 7f51f7fff640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f51fc06af90 0x7f51fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:42.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.202+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52040ab230 con 0x7f52100c9ca0 2023-10-16T12:52:42.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.203+0000 7f520f7fe640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f51fc06af90 0x7f51fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:42.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.204+0000 7f520f7fe640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f51fc06af90 0x7f51fc06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52000011e0 tx=0x7f5200009040 comp rx=0 tx=0).ready entity=mgr.6574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:42.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.206+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52100ccfd0 con 0x7f52100c9ca0 2023-10-16T12:52:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:42.450+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f52100ccc40 con 0x7f52100c9ca0 2023-10-16T12:52:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.038+0000 7f51f7fff640 1 -- 172.21.15.60:0/985921089 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v56) v1 ==== 92+0+0 (secure 0 0 0) 0x7f52040760b0 con 0x7f52100c9ca0 2023-10-16T12:52:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.042+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f51fc06af90 msgr2=0x7f51fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.042+0000 7f5215ac2640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f51fc06af90 0x7f51fc06d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52000011e0 tx=0x7f5200009040 comp rx=0 tx=0).stop 2023-10-16T12:52:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 msgr2=0x7f52100ad110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 0x7f52100ad110 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f52040098c0 tx=0x7f52040046c0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 shutdown_connections 2023-10-16T12:52:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f51fc06af90 0x7f51fc06d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f521015ad70 0x7f52100ad650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100c9ca0 0x7f52100ad110 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 --2- 172.21.15.60:0/985921089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521006a4b0 0x7f5210158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 >> 172.21.15.60:0/985921089 conn(0x7f52100ce9b0 msgr2=0x7f52100afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.043+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 shutdown_connections 2023-10-16T12:52:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.044+0000 7f5215ac2640 1 -- 172.21.15.60:0/985921089 wait complete. 2023-10-16T12:52:43.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:52:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.282+0000 7f2c6a91d640 1 Processor -- start 2023-10-16T12:52:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.282+0000 7f2c6a91d640 1 -- start start 2023-10-16T12:52:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c6a91d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c6415d330 0x7f2c641618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c6a91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2c6406c7d0 0x7f2c64065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c6a91d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406cc80 0x7f2c64065ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c6a91d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6406b1e0 con 0x7f2c6415d330 2023-10-16T12:52:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c6a91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c640d0890 con 0x7f2c6406c7d0 2023-10-16T12:52:43.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c6a91d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c640d05b0 con 0x7f2c6406cc80 2023-10-16T12:52:43.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c63fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c6415d330 0x7f2c641618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c63fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c6415d330 0x7f2c641618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58468/0 (socket says 172.21.15.60:58468) 2023-10-16T12:52:43.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.283+0000 7f2c63fff640 1 -- 172.21.15.60:0/3138940516 learned_addr learned my addr 172.21.15.60:0/3138940516 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:43.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c68eba640 1 --2- 172.21.15.60:0/3138940516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406cc80 0x7f2c64065ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c637fe640 1 --2- 172.21.15.60:0/3138940516 >> v2:172.21.15.144:3300/0 conn(0x7f2c6406c7d0 0x7f2c64065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c63fff640 1 -- 172.21.15.60:0/3138940516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406cc80 msgr2=0x7f2c64065ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:43.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c63fff640 1 --2- 172.21.15.60:0/3138940516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406cc80 0x7f2c64065ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c63fff640 1 -- 172.21.15.60:0/3138940516 >> v2:172.21.15.144:3300/0 conn(0x7f2c6406c7d0 msgr2=0x7f2c64065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:43.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c63fff640 1 --2- 172.21.15.60:0/3138940516 >> v2:172.21.15.144:3300/0 conn(0x7f2c6406c7d0 0x7f2c64065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c63fff640 1 -- 172.21.15.60:0/3138940516 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64160dd0 con 0x7f2c6415d330 2023-10-16T12:52:43.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.284+0000 7f2c637fe640 1 --2- 172.21.15.60:0/3138940516 >> v2:172.21.15.144:3300/0 conn(0x7f2c6406c7d0 0x7f2c64065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:43.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.285+0000 7f2c63fff640 1 --2- 172.21.15.60:0/3138940516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c641618b0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f2c640b6bc0 tx=0x7f2c5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa64c4f2d21dc31 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:43.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.285+0000 7f2c62ffd640 1 -- 172.21.15.60:0/3138940516 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5402c9e0 con 0x7f2c6415d330 2023-10-16T12:52:43.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.285+0000 7f2c62ffd640 1 -- 172.21.15.60:0/3138940516 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5402cb40 con 0x7f2c6415d330 2023-10-16T12:52:43.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.285+0000 7f2c62ffd640 1 -- 172.21.15.60:0/3138940516 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c540343f0 con 0x7f2c6415d330 2023-10-16T12:52:43.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3138940516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 msgr2=0x7f2c641618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3138940516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c641618b0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f2c640b6bc0 tx=0x7f2c5402be40 comp rx=0 tx=0).stop 2023-10-16T12:52:43.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3138940516 shutdown_connections 2023-10-16T12:52:43.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3138940516 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406cc80 0x7f2c64065ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3138940516 >> v2:172.21.15.144:3300/0 conn(0x7f2c6406c7d0 0x7f2c64065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3138940516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c641618b0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3138940516 >> 172.21.15.60:0/3138940516 conn(0x7f2c640d0ab0 msgr2=0x7f2c640d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:43.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3138940516 shutdown_connections 2023-10-16T12:52:43.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.286+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3138940516 wait complete. 2023-10-16T12:52:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.288+0000 7f2c6a91d640 1 Processor -- start 2023-10-16T12:52:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.288+0000 7f2c6a91d640 1 -- start start 2023-10-16T12:52:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.289+0000 7f2c6a91d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406c7d0 0x7f2c641e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.289+0000 7f2c6a91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c6406cc80 0x7f2c640af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.289+0000 7f2c6a91d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c640acf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.289+0000 7f2c6a91d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c641e91c0 con 0x7f2c6415d330 2023-10-16T12:52:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.289+0000 7f2c6a91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c641e87f0 con 0x7f2c6406cc80 2023-10-16T12:52:43.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.289+0000 7f2c6a91d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c640af840 con 0x7f2c6406c7d0 2023-10-16T12:52:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c640acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c640acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58482/0 (socket says 172.21.15.60:58482) 2023-10-16T12:52:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 -- 172.21.15.60:0/3503759823 learned_addr learned my addr 172.21.15.60:0/3503759823 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c637fe640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c6406cc80 0x7f2c640af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c63fff640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406c7d0 0x7f2c641e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 -- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406c7d0 msgr2=0x7f2c641e8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406c7d0 0x7f2c641e8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 -- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c6406cc80 msgr2=0x7f2c640af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c6406cc80 0x7f2c640af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.290+0000 7f2c68eba640 1 -- 172.21.15.60:0/3503759823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54009b30 con 0x7f2c6415d330 2023-10-16T12:52:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.291+0000 7f2c68eba640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c640acf60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00d5a0 tx=0x7f2c5c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.291+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0125f0 con 0x7f2c6415d330 2023-10-16T12:52:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.291+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c012770 con 0x7f2c6415d330 2023-10-16T12:52:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.291+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c012b20 con 0x7f2c6415d330 2023-10-16T12:52:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.291+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c641e7be0 con 0x7f2c6415d330 2023-10-16T12:52:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.292+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c641e81a0 con 0x7f2c6415d330 2023-10-16T12:52:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.293+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c641e8980 con 0x7f2c6415d330 2023-10-16T12:52:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.294+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 56) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2c5c012c80 con 0x7f2c6415d330 2023-10-16T12:52:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.295+0000 7f2c60ff9640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 0x7f2c5006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.295+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c5c089aa0 con 0x7f2c6415d330 2023-10-16T12:52:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.295+0000 7f2c63fff640 1 -- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 msgr2=0x7f2c5006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:43.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.295+0000 7f2c63fff640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 0x7f2c5006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:43.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.299+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2c5c054a60 con 0x7f2c6415d330 2023-10-16T12:52:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.496+0000 7f2c63fff640 1 -- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 msgr2=0x7f2c5006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.496+0000 7f2c63fff640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 0x7f2c5006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.496+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2c6404e160 con 0x7f2c6415d330 2023-10-16T12:52:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.503+0000 7f2c60ff9640 1 -- 172.21.15.60:0/3503759823 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2c5c058080 con 0x7f2c6415d330 2023-10-16T12:52:43.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.505+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 msgr2=0x7f2c5006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:43.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.505+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 0x7f2c5006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.505+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 msgr2=0x7f2c640acf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.505+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c640acf60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00d5a0 tx=0x7f2c5c00da60 comp rx=0 tx=0).stop 2023-10-16T12:52:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 shutdown_connections 2023-10-16T12:52:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2c5006afe0 0x7f2c5006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415d330 0x7f2c640acf60 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c6406cc80 0x7f2c640af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 --2- 172.21.15.60:0/3503759823 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c6406c7d0 0x7f2c641e8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 >> 172.21.15.60:0/3503759823 conn(0x7f2c640d0ab0 msgr2=0x7f2c640c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 shutdown_connections 2023-10-16T12:52:43.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.506+0000 7f2c6a91d640 1 -- 172.21.15.60:0/3503759823 wait complete. 2023-10-16T12:52:43.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:52:43.561 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:43.559+0000 7fbd8ec58040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:52:43.614 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:43.613+0000 7fbd8ec58040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:52:43.644 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:43.642+0000 7f1079a52040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:52:43.706 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:43.704+0000 7f1079a52040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:52:43.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.738+0000 7feef5e51640 1 Processor -- start 2023-10-16T12:52:43.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.739+0000 7feef5e51640 1 -- start start 2023-10-16T12:52:43.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.740+0000 7feef5e51640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feef00aeef0 0x7feef015f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.740+0000 7feef5e51640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef006c140 con 0x7feef00cbf20 2023-10-16T12:52:43.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.740+0000 7feef5e51640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00af360 con 0x7feef015aee0 2023-10-16T12:52:43.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.740+0000 7feef5e51640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef015b2c0 con 0x7feef00aeef0 2023-10-16T12:52:43.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feeeffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feef00aeef0 0x7feef015f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feee7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feef00cbf20 0x7feef015e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42876/0 (socket says 172.21.15.60:42876) 2023-10-16T12:52:43.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feee7fff640 1 -- 172.21.15.60:0/3140318857 learned_addr learned my addr 172.21.15.60:0/3140318857 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:52:43.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feeeffff640 1 -- 172.21.15.60:0/3140318857 >> v1:172.21.15.144:6789/0 conn(0x7feef015aee0 legacy=0x7feef00cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feeeffff640 1 -- 172.21.15.60:0/3140318857 >> v1:172.21.15.60:6789/0 conn(0x7feef00cbf20 legacy=0x7feef015e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feeeffff640 1 -- 172.21.15.60:0/3140318857 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef006b270 con 0x7feef00aeef0 2023-10-16T12:52:43.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.741+0000 7feeeeffd640 1 -- 172.21.15.60:0/3140318857 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628600535 0 0) 0x7feef006c140 con 0x7feef00cbf20 2023-10-16T12:52:43.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.743+0000 7feeeffff640 1 --2- 172.21.15.60:0/3140318857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00aeef0 0x7feef015f0c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7feee8009d90 tx=0x7feee802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ff9c0468bd3545 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:43.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.743+0000 7feeeeffd640 1 -- 172.21.15.60:0/3140318857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee802cc60 con 0x7feef00aeef0 2023-10-16T12:52:43.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feeeeffd640 1 -- 172.21.15.60:0/3140318857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee802cdc0 con 0x7feef00aeef0 2023-10-16T12:52:43.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feeeeffd640 1 -- 172.21.15.60:0/3140318857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8034660 con 0x7feef00aeef0 2023-10-16T12:52:43.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 -- 172.21.15.60:0/3140318857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00aeef0 msgr2=0x7feef015f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 --2- 172.21.15.60:0/3140318857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00aeef0 0x7feef015f0c0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7feee8009d90 tx=0x7feee802beb0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 -- 172.21.15.60:0/3140318857 shutdown_connections 2023-10-16T12:52:43.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 --2- 172.21.15.60:0/3140318857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00aeef0 0x7feef015f0c0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 -- 172.21.15.60:0/3140318857 >> 172.21.15.60:0/3140318857 conn(0x7feef00cfac0 msgr2=0x7feef00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:43.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 -- 172.21.15.60:0/3140318857 shutdown_connections 2023-10-16T12:52:43.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.744+0000 7feef5e51640 1 -- 172.21.15.60:0/3140318857 wait complete. 2023-10-16T12:52:43.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.746+0000 7feef5e51640 1 Processor -- start 2023-10-16T12:52:43.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.746+0000 7feef5e51640 1 -- start start 2023-10-16T12:52:43.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.747+0000 7feef5e51640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef00aeef0 0x7feef0158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.747+0000 7feef5e51640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cbf20 0x7feef01e4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.747+0000 7feef5e51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 0x7feef01f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.747+0000 7feef5e51640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef01597f0 con 0x7feef015aee0 2023-10-16T12:52:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.747+0000 7feef5e51640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef00c6d30 con 0x7feef00cbf20 2023-10-16T12:52:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.747+0000 7feef5e51640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef00c7010 con 0x7feef00aeef0 2023-10-16T12:52:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 0x7feef01f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 0x7feef01f2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58500/0 (socket says 172.21.15.60:58500) 2023-10-16T12:52:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 -- 172.21.15.60:0/3952717387 learned_addr learned my addr 172.21.15.60:0/3952717387 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feee7fff640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cbf20 0x7feef01e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeef7fe640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef00aeef0 0x7feef0158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 -- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef00aeef0 msgr2=0x7feef0158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef00aeef0 0x7feef0158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 -- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cbf20 msgr2=0x7feef01e4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cbf20 0x7feef01e4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feeeffff640 1 -- 172.21.15.60:0/3952717387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee8009a70 con 0x7feef015aee0 2023-10-16T12:52:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.748+0000 7feee7fff640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cbf20 0x7feef01e4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:52:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.749+0000 7feeeffff640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 0x7feef01f2300 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7feee8009d90 tx=0x7feee8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.749+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8040450 con 0x7feef015aee0 2023-10-16T12:52:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.749+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee80405d0 con 0x7feef015aee0 2023-10-16T12:52:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.749+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef00c72f0 con 0x7feef015aee0 2023-10-16T12:52:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.749+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8040980 con 0x7feef015aee0 2023-10-16T12:52:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.749+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef00c78b0 con 0x7feef015aee0 2023-10-16T12:52:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.751+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef004e160 con 0x7feef015aee0 2023-10-16T12:52:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.752+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 56) v1 ==== 89348+0+0 (secure 0 0 0) 0x7feee8040ae0 con 0x7feef015aee0 2023-10-16T12:52:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.753+0000 7feeecff9640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 0x7feed006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:43.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.754+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feee80ac360 con 0x7feef015aee0 2023-10-16T12:52:43.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.754+0000 7feeef7fe640 1 -- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 msgr2=0x7feed006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:43.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.754+0000 7feeef7fe640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 0x7feed006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.758+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feee80773b0 con 0x7feef015aee0 2023-10-16T12:52:43.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.955+0000 7feeef7fe640 1 -- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 msgr2=0x7feed006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:43.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.955+0000 7feeef7fe640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 0x7feed006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:43.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.956+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7feef0070560 con 0x7feef015aee0 2023-10-16T12:52:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.960+0000 7feeecff9640 1 -- 172.21.15.60:0/3952717387 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7feee807a9d0 con 0x7feef015aee0 2023-10-16T12:52:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 msgr2=0x7feed006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 0x7feed006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 msgr2=0x7feef01f2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 0x7feef01f2300 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7feee8009d90 tx=0x7feee8004a40 comp rx=0 tx=0).stop 2023-10-16T12:52:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 shutdown_connections 2023-10-16T12:52:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7feed006afe0 0x7feed006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef015aee0 0x7feef01f2300 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cbf20 0x7feef01e4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 --2- 172.21.15.60:0/3952717387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef00aeef0 0x7feef0158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 >> 172.21.15.60:0/3952717387 conn(0x7feef00cfac0 msgr2=0x7feef006d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 shutdown_connections 2023-10-16T12:52:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:43.963+0000 7feef5e51640 1 -- 172.21.15.60:0/3952717387 wait complete. 2023-10-16T12:52:43.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:43.985 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:52:44.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T12:52:44.061 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.059+0000 7fbd8ec58040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:52:44.113 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.111+0000 7fbd8ec58040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:52:44.162 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.160+0000 7fbd8ec58040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:52:44.196 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.194+0000 7f1079a52040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:52:44.223 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.221+0000 7fbd8ec58040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:52:44.250 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.249+0000 7f1079a52040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:52:44.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.252+0000 7f2b9fabb640 1 Processor -- start 2023-10-16T12:52:44.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.253+0000 7f2b9fabb640 1 -- start start 2023-10-16T12:52:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.254+0000 7f2b9fabb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b98070830 0x7f2b9806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.254+0000 7f2b9fabb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2b98070c10 0x7f2b980cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.254+0000 7f2b9fabb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b98161740 con 0x7f2b980cf5f0 2023-10-16T12:52:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.254+0000 7f2b9fabb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b980cdbd0 con 0x7f2b98070830 2023-10-16T12:52:44.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.254+0000 7f2b9fabb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9806b0f0 con 0x7f2b98070c10 2023-10-16T12:52:44.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b98070830 0x7f2b9806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:44.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b98070830 0x7f2b9806d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58510/0 (socket says 172.21.15.60:58510) 2023-10-16T12:52:44.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 -- 172.21.15.60:0/4048315743 learned_addr learned my addr 172.21.15.60:0/4048315743 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:44.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 -- 172.21.15.60:0/4048315743 >> v2:172.21.15.144:3300/0 conn(0x7f2b98070c10 msgr2=0x7f2b980cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:44.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9e058640 1 --2- 172.21.15.60:0/4048315743 >> v2:172.21.15.144:3300/0 conn(0x7f2b98070c10 0x7f2b980cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:44.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 --2- 172.21.15.60:0/4048315743 >> v2:172.21.15.144:3300/0 conn(0x7f2b98070c10 0x7f2b980cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 -- 172.21.15.60:0/4048315743 >> v1:172.21.15.144:6789/0 conn(0x7f2b980cf5f0 legacy=0x7f2b9806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9d857640 1 -- 172.21.15.60:0/4048315743 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b981619b0 con 0x7f2b98070830 2023-10-16T12:52:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.255+0000 7f2b9e058640 1 --2- 172.21.15.60:0/4048315743 >> v2:172.21.15.144:3300/0 conn(0x7f2b98070c10 0x7f2b980cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.256+0000 7f2b9d857640 1 --2- 172.21.15.60:0/4048315743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b98070830 0x7f2b9806d3c0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f2b980ceba0 tx=0x7f2b9402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e447a40c25ab4bc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.256+0000 7f2b9c855640 1 -- 172.21.15.60:0/4048315743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94031750 con 0x7f2b98070830 2023-10-16T12:52:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9c855640 1 -- 172.21.15.60:0/4048315743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b940318b0 con 0x7f2b98070830 2023-10-16T12:52:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9c855640 1 -- 172.21.15.60:0/4048315743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94031c40 con 0x7f2b98070830 2023-10-16T12:52:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 -- 172.21.15.60:0/4048315743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b98070830 msgr2=0x7f2b9806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 --2- 172.21.15.60:0/4048315743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b98070830 0x7f2b9806d3c0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f2b980ceba0 tx=0x7f2b9402b7f0 comp rx=0 tx=0).stop 2023-10-16T12:52:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 -- 172.21.15.60:0/4048315743 shutdown_connections 2023-10-16T12:52:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 --2- 172.21.15.60:0/4048315743 >> v2:172.21.15.144:3300/0 conn(0x7f2b98070c10 0x7f2b980cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 --2- 172.21.15.60:0/4048315743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b98070830 0x7f2b9806d3c0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 -- 172.21.15.60:0/4048315743 >> 172.21.15.60:0/4048315743 conn(0x7f2b980cfac0 msgr2=0x7f2b980cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 -- 172.21.15.60:0/4048315743 shutdown_connections 2023-10-16T12:52:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.257+0000 7f2b9fabb640 1 -- 172.21.15.60:0/4048315743 wait complete. 2023-10-16T12:52:44.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.259+0000 7f2b9fabb640 1 Processor -- start 2023-10-16T12:52:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.259+0000 7f2b9fabb640 1 -- start start 2023-10-16T12:52:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9fabb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b98070830 0x7f2b981f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9fabb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b98070c10 0x7f2b981f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9fabb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b980cf5f0 0x7f2b980c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9fabb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980c5ef0 con 0x7f2b980cf5f0 2023-10-16T12:52:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9fabb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b981f2780 con 0x7f2b98070830 2023-10-16T12:52:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9fabb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9814b1b0 con 0x7f2b98070c10 2023-10-16T12:52:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9e058640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b980cf5f0 0x7f2b980c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.260+0000 7f2b9e058640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b980cf5f0 0x7f2b980c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58518/0 (socket says 172.21.15.60:58518) 2023-10-16T12:52:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 -- 172.21.15.60:0/3823996456 learned_addr learned my addr 172.21.15.60:0/3823996456 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b98070830 0x7f2b981f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b98070c10 msgr2=0x7f2b981f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b98070c10 0x7f2b981f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b98070830 msgr2=0x7f2b981f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b98070830 0x7f2b981f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 -- 172.21.15.60:0/3823996456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b940094d0 con 0x7f2b980cf5f0 2023-10-16T12:52:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b98070830 0x7f2b981f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.261+0000 7f2b9e058640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b980cf5f0 0x7f2b980c7420 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000cef0 tx=0x7f2b9000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.262+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000d840 con 0x7f2b980cf5f0 2023-10-16T12:52:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.262+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9000d9c0 con 0x7f2b980cf5f0 2023-10-16T12:52:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.262+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000dd70 con 0x7f2b980cf5f0 2023-10-16T12:52:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.262+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b981f6d90 con 0x7f2b980cf5f0 2023-10-16T12:52:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.262+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b981f6ab0 con 0x7f2b980cf5f0 2023-10-16T12:52:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.263+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9814b340 con 0x7f2b980cf5f0 2023-10-16T12:52:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.264+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 56) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2b9001e3b0 con 0x7f2b980cf5f0 2023-10-16T12:52:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.265+0000 7f2b8e7fc640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 0x7f2b7c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.265+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b9008a2c0 con 0x7f2b980cf5f0 2023-10-16T12:52:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.265+0000 7f2b9d857640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 msgr2=0x7f2b7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.265+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 0x7f2b7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:52:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.269+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b90055310 con 0x7f2b980cf5f0 2023-10-16T12:52:44.272 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.270+0000 7fbd8ec58040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:52:44.305 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.304+0000 7f1079a52040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:52:44.369 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.367+0000 7f1079a52040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:52:44.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.381+0000 7fbd8ec58040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:52:44.421 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.419+0000 7f1079a52040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:52:44.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.443+0000 7fbd8ec58040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.466+0000 7f2b9d857640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 msgr2=0x7f2b7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:44.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.466+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 0x7f2b7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:52:44.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.487+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 --> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b980cf060 con 0x7f2b7c06b010 2023-10-16T12:52:44.493 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.491+0000 7fbd8ec58040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:52:44.537 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.535+0000 7f1079a52040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:52:44.551 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.549+0000 7fbd8ec58040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:52:44.600 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.598+0000 7fbd8ec58040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:52:44.602 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.600+0000 7f1079a52040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:44.655 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.654+0000 7f1079a52040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:52:44.714 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.712+0000 7f1079a52040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:52:44.766 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:44.764+0000 7f1079a52040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:52:44.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.867+0000 7f2b9d857640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 msgr2=0x7f2b7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:44.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:44.867+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 0x7f2b7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:52:44.978 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:44.976+0000 7fbd8ec58040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:52:45.136 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.134+0000 7fbd8ec58040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:52:45.164 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.163+0000 7f1079a52040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:52:45.245 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.243+0000 7fbd8ec58040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:52:45.295 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.294+0000 7fbd8ec58040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:52:45.332 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.331+0000 7f1079a52040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:52:45.403 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.401+0000 7fbd8ec58040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:45.445 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.443+0000 7f1079a52040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:52:45.497 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.495+0000 7f1079a52040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:52:45.508 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.507+0000 7fbd8ec58040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:52:45.609 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.608+0000 7f1079a52040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:45.665 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.663+0000 7fbd8ec58040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:45.668+0000 7f2b9d857640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 msgr2=0x7f2b7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3337897861 2023-10-16T12:52:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:45.668+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 0x7f2b7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:52:45.714 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.712+0000 7fbd8ec58040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:45.720 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.718+0000 7f1079a52040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:52:45.832 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:45.830+0000 7fbd8ec58040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:52:45.883 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.881+0000 7f1079a52040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:52:45.934 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:45.932+0000 7f1079a52040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:52:46.059 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:46.058+0000 7f1079a52040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:52:46.653 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:46.651+0000 7fbd8ec58040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:52:46.938 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:46.936+0000 7f1079a52040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:52:46.948 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:46.946+0000 7fbd8ec58040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:52:47.005 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.003+0000 7fbd8ec58040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:52:47.117 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.115+0000 7fbd8ec58040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:52:47.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:47.136+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 57) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f2b900556e0 con 0x7f2b980cf5f0 2023-10-16T12:52:47.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:47.136+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 msgr2=0x7f2b7c06d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:52:47.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:47.136+0000 7f2b8e7fc640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3337897861,v1:172.21.15.144:6833/3337897861] conn(0x7f2b7c06b010 0x7f2b7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:47.264 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:47.261+0000 7f1079a52040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:52:47.329 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:47.327+0000 7f1079a52040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:52:47.458 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.455+0000 7fbd0eee1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.458 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.455+0000 7fbd0eee1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.458 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0eee1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.459 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0eee1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.459 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0eee1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.459 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0ae99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.459 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0ae99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.459 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0ae99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.460 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0ae99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.460 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:52:47.456+0000 7fbd0ae99640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:52:47.460 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:52:47.455+0000 7f1079a52040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:52:48.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.146+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 58) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2b90088740 con 0x7f2b980cf5f0 2023-10-16T12:52:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.147+0000 7f2b8e7fc640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f2b7c0358b0 0x7f2b7c074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.147+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 --> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b7c035c90 con 0x7f2b7c0358b0 2023-10-16T12:52:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.152+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f2b7c0358b0 0x7f2b7c074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.154+0000 7f2b9d857640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f2b7c0358b0 0x7f2b7c074ed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2b94035000 tx=0x7f2b94008030 comp rx=0 tx=0).ready entity=mgr.6173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:48.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.353+0000 7f2b8e7fc640 1 -- 172.21.15.60:0/3823996456 <== mgr.6173 v2:172.21.15.144:6832/3420535028 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2b7c035c90 con 0x7f2b7c0358b0 2023-10-16T12:52:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1 2023-10-16T12:52:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f2b7c0358b0 msgr2=0x7f2b7c074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9fabb640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f2b7c0358b0 0x7f2b7c074ed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2b94035000 tx=0x7f2b94008030 comp rx=0 tx=0).stop 2023-10-16T12:52:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b980cf5f0 msgr2=0x7f2b980c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9fabb640 1 --2- 172.21.15.60:0/3823996456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b980cf5f0 0x7f2b980c7420 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000cef0 tx=0x7f2b9000a710 comp rx=0 tx=0).stop 2023-10-16T12:52:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9e058640 1 -- 172.21.15.60:0/3823996456 reap_dead start 2023-10-16T12:52:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 shutdown_connections 2023-10-16T12:52:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.356+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 >> 172.21.15.60:0/3823996456 conn(0x7f2b980cfac0 msgr2=0x7f2b980cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.357+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 shutdown_connections 2023-10-16T12:52:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.357+0000 7f2b9fabb640 1 -- 172.21.15.60:0/3823996456 wait complete. 2023-10-16T12:52:48.379 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T12:52:48.380 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-12_53_00 in ~12s... 2023-10-16T12:52:48.380 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T12:52:48.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T12:52:48.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.587+0000 7fca464bb640 1 Processor -- start 2023-10-16T12:52:48.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.587+0000 7fca464bb640 1 -- start start 2023-10-16T12:52:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.588+0000 7fca464bb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca4015c9f0 0x7fca400c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.588+0000 7fca464bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015dde0 0x7fca400ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.588+0000 7fca464bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca40160a90 con 0x7fca400c9230 2023-10-16T12:52:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.588+0000 7fca464bb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca400cdec0 con 0x7fca4015c9f0 2023-10-16T12:52:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.588+0000 7fca464bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca400cd900 con 0x7fca4015dde0 2023-10-16T12:52:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca4015c9f0 0x7fca400c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca4015c9f0 0x7fca400c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58534/0 (socket says 172.21.15.60:58534) 2023-10-16T12:52:48.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 -- 172.21.15.60:0/342312869 learned_addr learned my addr 172.21.15.60:0/342312869 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 -- 172.21.15.60:0/342312869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015dde0 msgr2=0x7fca400ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 --2- 172.21.15.60:0/342312869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015dde0 0x7fca400ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 -- 172.21.15.60:0/342312869 >> v1:172.21.15.144:6789/0 conn(0x7fca400c9230 legacy=0x7fca400709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:52:48.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.589+0000 7fca454b9640 1 -- 172.21.15.60:0/342312869 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca400cdbe0 con 0x7fca4015c9f0 2023-10-16T12:52:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.590+0000 7fca454b9640 1 --2- 172.21.15.60:0/342312869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015c9f0 0x7fca400c8cf0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fca30009e50 tx=0x7fca3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f6c763df5f2059 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca3ffff640 1 -- 172.21.15.60:0/342312869 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3002ca80 con 0x7fca4015c9f0 2023-10-16T12:52:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca3ffff640 1 -- 172.21.15.60:0/342312869 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3002cbe0 con 0x7fca4015c9f0 2023-10-16T12:52:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca3ffff640 1 -- 172.21.15.60:0/342312869 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca300343f0 con 0x7fca4015c9f0 2023-10-16T12:52:48.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 -- 172.21.15.60:0/342312869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015c9f0 msgr2=0x7fca400c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 --2- 172.21.15.60:0/342312869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015c9f0 0x7fca400c8cf0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fca30009e50 tx=0x7fca3002bee0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 -- 172.21.15.60:0/342312869 shutdown_connections 2023-10-16T12:52:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 --2- 172.21.15.60:0/342312869 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015dde0 0x7fca400ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 --2- 172.21.15.60:0/342312869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015c9f0 0x7fca400c8cf0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 -- 172.21.15.60:0/342312869 >> 172.21.15.60:0/342312869 conn(0x7fca400ce6a0 msgr2=0x7fca400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 -- 172.21.15.60:0/342312869 shutdown_connections 2023-10-16T12:52:48.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.591+0000 7fca464bb640 1 -- 172.21.15.60:0/342312869 wait complete. 2023-10-16T12:52:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.594+0000 7fca464bb640 1 Processor -- start 2023-10-16T12:52:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.594+0000 7fca464bb640 1 -- start start 2023-10-16T12:52:48.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca464bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca400c9230 0x7fca401f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca464bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015c9f0 0x7fca400b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca464bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 0x7fca40158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca464bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4006aac0 con 0x7fca4015dde0 2023-10-16T12:52:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca464bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40069e40 con 0x7fca400c9230 2023-10-16T12:52:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca464bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4006a120 con 0x7fca4015c9f0 2023-10-16T12:52:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca45cba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 0x7fca40158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca45cba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 0x7fca40158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58536/0 (socket says 172.21.15.60:58536) 2023-10-16T12:52:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca45cba640 1 -- 172.21.15.60:0/4067816342 learned_addr learned my addr 172.21.15.60:0/4067816342 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:52:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca454b9640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca400c9230 0x7fca401f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca44cb8640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015c9f0 0x7fca400b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca45cba640 1 -- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015c9f0 msgr2=0x7fca400b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca45cba640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015c9f0 0x7fca400b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.595+0000 7fca45cba640 1 -- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca400c9230 msgr2=0x7fca401f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca45cba640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca400c9230 0x7fca401f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca45cba640 1 -- 172.21.15.60:0/4067816342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30009b30 con 0x7fca4015dde0 2023-10-16T12:52:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca44cb8640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015c9f0 0x7fca400b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:52:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca45cba640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 0x7fca40158d70 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fca3800b410 tx=0x7fca3800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38011830 con 0x7fca4015dde0 2023-10-16T12:52:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38011990 con 0x7fca4015dde0 2023-10-16T12:52:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38011d40 con 0x7fca4015dde0 2023-10-16T12:52:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4006a400 con 0x7fca4015dde0 2023-10-16T12:52:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.596+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca401f2340 con 0x7fca4015dde0 2023-10-16T12:52:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.597+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4006ac00 con 0x7fca4015dde0 2023-10-16T12:52:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.598+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 58) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fca3801a400 con 0x7fca4015dde0 2023-10-16T12:52:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.599+0000 7fca3dffb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fca2c06ad10 0x7fca2c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:52:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.599+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fca380895c0 con 0x7fca4015dde0 2023-10-16T12:52:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.600+0000 7fca454b9640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fca2c06ad10 0x7fca2c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:52:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.601+0000 7fca454b9640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fca2c06ad10 0x7fca2c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fca30035000 tx=0x7fca30030000 comp rx=0 tx=0).ready entity=mgr.6173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:52:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.604+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca4006ac00 con 0x7fca4015dde0 2023-10-16T12:52:48.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.809+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 --> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca400704c0 con 0x7fca2c06ad10 2023-10-16T12:52:48.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.826+0000 7fca3dffb640 1 -- 172.21.15.60:0/4067816342 <== mgr.6173 v2:172.21.15.144:6832/3420535028 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fca300440c0 con 0x7fca2c06ad10 2023-10-16T12:52:48.829 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:52:48.829 INFO:teuthology.orchestra.run.smithi060.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T12:52:48.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fca2c06ad10 msgr2=0x7fca2c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fca2c06ad10 0x7fca2c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fca30035000 tx=0x7fca30030000 comp rx=0 tx=0).stop 2023-10-16T12:52:48.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 msgr2=0x7fca40158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:52:48.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 0x7fca40158d70 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fca3800b410 tx=0x7fca3800b810 comp rx=0 tx=0).stop 2023-10-16T12:52:48.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 shutdown_connections 2023-10-16T12:52:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fca2c06ad10 0x7fca2c06d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca4015c9f0 0x7fca400b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca4015dde0 0x7fca40158d70 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 --2- 172.21.15.60:0/4067816342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca400c9230 0x7fca401f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:52:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 >> 172.21.15.60:0/4067816342 conn(0x7fca400ce6a0 msgr2=0x7fca400ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:52:48.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.828+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 shutdown_connections 2023-10-16T12:52:48.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:52:48.829+0000 7fca464bb640 1 -- 172.21.15.60:0/4067816342 wait complete. 2023-10-16T12:52:48.847 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T12:52:48.848 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-10-16T12:52:48.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:48.849 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:48.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:48.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:48.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:49.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:49.905 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:49.972 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:49.973 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:49.973 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:50.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:50.975 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:51.041 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:51.041 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:51.042 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:52.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:52.043 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:52.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:52.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:52.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:53.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:53.113 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:53.182 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:53.182 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:53.183 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:54.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:54.184 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:54.248 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:54.248 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:54.248 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:55.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:55.251 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:55.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:55.319 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:55.320 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:56.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:56.323 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:56.392 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:56.393 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:56.393 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:57.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:57.395 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:57.461 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:57.462 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:57.462 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:57.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:58.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:58.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:58.464 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:58.532 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:58.533 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:58.533 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:52:59.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:52:59.535 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:52:59.604 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:52:59.604 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:52:59.605 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:53:00.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:00.379+0000 7fbcfe680640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:00.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:00.379+0000 7fbcfe680640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:00.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:00.379+0000 7fbcfe680640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:00.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:00.379+0000 7fbcfe680640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:00.383 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:00.379+0000 7fbcfe680640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:00.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:00.607 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:53:00.670 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_53_00_UTC 2023-10-16T12:53:00.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_53_00_UTC'} 2023-10-16T12:53:00.672 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-12_53_00_UTC'} 2023-10-16T12:53:00.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:53:00.673 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-12_53_00_UTC'] 2023-10-16T12:53:00.673 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697460768.3800366, now = 1697460780.6736367, timo = 12 2023-10-16T12:53:01.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T12:53:01.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.908+0000 7f56cd8d7640 1 Processor -- start 2023-10-16T12:53:01.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.908+0000 7f56cd8d7640 1 -- start start 2023-10-16T12:53:01.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56cd8d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c806a4b0 0x7f56c80ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:01.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56cd8d7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56c80c9ca0 0x7f56c80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:01.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56cd8d7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c80cb030 con 0x7f56c815ad70 2023-10-16T12:53:01.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56cd8d7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c80cae30 con 0x7f56c80c9ca0 2023-10-16T12:53:01.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56cd8d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80c91c0 con 0x7f56c806a4b0 2023-10-16T12:53:01.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56c77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56c80c9ca0 0x7f56c80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:01.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56c77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56c80c9ca0 0x7f56c80ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43912/0 (socket says 172.21.15.60:43912) 2023-10-16T12:53:01.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56c77fe640 1 -- 172.21.15.60:0/3827706045 learned_addr learned my addr 172.21.15.60:0/3827706045 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:01.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.910+0000 7f56bffff640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c806a4b0 0x7f56c80ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:01.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.911+0000 7f56c77fe640 1 -- 172.21.15.60:0/3827706045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c806a4b0 msgr2=0x7f56c80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:01.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.911+0000 7f56c77fe640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c806a4b0 0x7f56c80ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.911+0000 7f56c77fe640 1 -- 172.21.15.60:0/3827706045 >> v1:172.21.15.144:6789/0 conn(0x7f56c815ad70 legacy=0x7f56c8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.911+0000 7f56c77fe640 1 -- 172.21.15.60:0/3827706045 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c80c94a0 con 0x7f56c80c9ca0 2023-10-16T12:53:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.911+0000 7f56bffff640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c806a4b0 0x7f56c80ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.912+0000 7f56c77fe640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c80c9ca0 0x7f56c80ca8f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009e50 tx=0x7f56c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21d819475d8e442 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.912+0000 7f56c67fc640 1 -- 172.21.15.60:0/3827706045 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c002ca80 con 0x7f56c80c9ca0 2023-10-16T12:53:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.912+0000 7f56c67fc640 1 -- 172.21.15.60:0/3827706045 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c002cbe0 con 0x7f56c80c9ca0 2023-10-16T12:53:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.912+0000 7f56c67fc640 1 -- 172.21.15.60:0/3827706045 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c00343f0 con 0x7f56c80c9ca0 2023-10-16T12:53:01.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 -- 172.21.15.60:0/3827706045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c80c9ca0 msgr2=0x7f56c80ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c80c9ca0 0x7f56c80ca8f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009e50 tx=0x7f56c002bee0 comp rx=0 tx=0).stop 2023-10-16T12:53:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 -- 172.21.15.60:0/3827706045 shutdown_connections 2023-10-16T12:53:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c80c9ca0 0x7f56c80ca8f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/3827706045 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c806a4b0 0x7f56c80ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 -- 172.21.15.60:0/3827706045 >> 172.21.15.60:0/3827706045 conn(0x7f56c80ce9b0 msgr2=0x7f56c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 -- 172.21.15.60:0/3827706045 shutdown_connections 2023-10-16T12:53:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.913+0000 7f56cd8d7640 1 -- 172.21.15.60:0/3827706045 wait complete. 2023-10-16T12:53:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.915+0000 7f56cd8d7640 1 Processor -- start 2023-10-16T12:53:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.915+0000 7f56cd8d7640 1 -- start start 2023-10-16T12:53:01.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56cd8d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 0x7f56c8136650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:01.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56cd8d7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c80c9ca0 0x7f56c8136b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:01.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56cd8d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c815ad70 0x7f56c8152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:01.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56cd8d7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8150b50 con 0x7f56c806a4b0 2023-10-16T12:53:01.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56cd8d7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8150e30 con 0x7f56c80c9ca0 2023-10-16T12:53:01.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56cd8d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80c63a0 con 0x7f56c815ad70 2023-10-16T12:53:01.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56c6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 0x7f56c8136650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:01.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56c6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 0x7f56c8136650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43918/0 (socket says 172.21.15.60:43918) 2023-10-16T12:53:01.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.916+0000 7f56c6ffd640 1 -- 172.21.15.60:0/2910396071 learned_addr learned my addr 172.21.15.60:0/2910396071 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:01.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c77fe640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c815ad70 0x7f56c8152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:01.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56bffff640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c80c9ca0 0x7f56c8136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:01.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c6ffd640 1 -- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c815ad70 msgr2=0x7f56c8152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:01.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c6ffd640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c815ad70 0x7f56c8152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:01.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c6ffd640 1 -- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c80c9ca0 msgr2=0x7f56c8136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:01.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c6ffd640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c80c9ca0 0x7f56c8136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:01.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c6ffd640 1 -- 172.21.15.60:0/2910396071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c0009b30 con 0x7f56c806a4b0 2023-10-16T12:53:01.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c6ffd640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 0x7f56c8136650 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b3f0 tx=0x7f56b800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:01.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.917+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8011830 con 0x7f56c806a4b0 2023-10-16T12:53:01.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.918+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b8011990 con 0x7f56c806a4b0 2023-10-16T12:53:01.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.918+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8011d40 con 0x7f56c806a4b0 2023-10-16T12:53:01.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.918+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c80c6680 con 0x7f56c806a4b0 2023-10-16T12:53:01.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.918+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c81370d0 con 0x7f56c806a4b0 2023-10-16T12:53:01.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.919+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c8151e80 con 0x7f56c806a4b0 2023-10-16T12:53:01.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.920+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 61) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f56b801a400 con 0x7f56c806a4b0 2023-10-16T12:53:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.921+0000 7f56c4ff9640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f56a806b100 0x7f56a806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.921+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f56b8089ac0 con 0x7f56c806a4b0 2023-10-16T12:53:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.922+0000 7f56bffff640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f56a806b100 0x7f56a806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:01.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.923+0000 7f56bffff640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f56a806b100 0x7f56a806d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f56c8150a80 tx=0x7f56b0008040 comp rx=0 tx=0).ready entity=mgr.6173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:01.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:01.926+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56b80546b0 con 0x7f56c806a4b0 2023-10-16T12:53:02.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.146+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 --> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f56c80c9780 con 0x7f56a806b100 2023-10-16T12:53:02.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.352+0000 7f56c4ff9640 1 -- 172.21.15.60:0/2910396071 <== mgr.6173 v2:172.21.15.144:6832/3420535028 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f56b000a660 con 0x7f56a806b100 2023-10-16T12:53:02.356 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path snap_test_dir1 2023-10-16T12:53:02.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f56a806b100 msgr2=0x7f56a806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:02.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f56a806b100 0x7f56a806d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f56c8150a80 tx=0x7f56b0008040 comp rx=0 tx=0).stop 2023-10-16T12:53:02.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 msgr2=0x7f56c8136650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:02.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 0x7f56c8136650 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b3f0 tx=0x7f56b800b8b0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 shutdown_connections 2023-10-16T12:53:02.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f56a806b100 0x7f56a806d5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f56c815ad70 0x7f56c8152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c80c9ca0 0x7f56c8136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 --2- 172.21.15.60:0/2910396071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c806a4b0 0x7f56c8136650 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 >> 172.21.15.60:0/2910396071 conn(0x7f56c80ce9b0 msgr2=0x7f56c815d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:02.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 shutdown_connections 2023-10-16T12:53:02.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.356+0000 7f56cd8d7640 1 -- 172.21.15.60:0/2910396071 wait complete. 2023-10-16T12:53:02.379 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T12:53:02.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:02.380 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:53:02.436 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_53_00_UTC 2023-10-16T12:53:02.437 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_53_00_UTC 2023-10-16T12:53:02.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:02.438 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_53_00_UTC) 2023-10-16T12:53:02.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:02.504 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T12:53:02.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:53:02.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.801+0000 7f780759e640 1 Processor -- start 2023-10-16T12:53:02.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.802+0000 7f780759e640 1 -- start start 2023-10-16T12:53:02.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.803+0000 7f780759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:02.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.803+0000 7f780759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f780815dde0 0x7f78080ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:02.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.803+0000 7f780759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7808160a90 con 0x7f78080c9230 2023-10-16T12:53:02.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.803+0000 7f780759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78080cdec0 con 0x7f780815dde0 2023-10-16T12:53:02.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.803+0000 7f780759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080cd900 con 0x7f780815c9f0 2023-10-16T12:53:02.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f780815dde0 0x7f78080ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:02.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f780815dde0 0x7f78080ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43932/0 (socket says 172.21.15.60:43932) 2023-10-16T12:53:02.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 -- 172.21.15.60:0/3716938798 learned_addr learned my addr 172.21.15.60:0/3716938798 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:02.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f780659c640 1 --2- 172.21.15.60:0/3716938798 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:02.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 -- 172.21.15.60:0/3716938798 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 msgr2=0x7f78080c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:02.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 --2- 172.21.15.60:0/3716938798 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 -- 172.21.15.60:0/3716938798 >> v1:172.21.15.144:6789/0 conn(0x7f78080c9230 legacy=0x7f78080709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:02.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.804+0000 7f7806d9d640 1 -- 172.21.15.60:0/3716938798 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78080cdbe0 con 0x7f780815dde0 2023-10-16T12:53:02.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.806+0000 7f7806d9d640 1 --2- 172.21.15.60:0/3716938798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080ac890 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fafe1ad29655a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:02.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.806+0000 7f7805d9b640 1 -- 172.21.15.60:0/3716938798 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780002ca70 con 0x7f780815dde0 2023-10-16T12:53:02.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.806+0000 7f7805d9b640 1 -- 172.21.15.60:0/3716938798 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780002cbd0 con 0x7f780815dde0 2023-10-16T12:53:02.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.806+0000 7f7805d9b640 1 -- 172.21.15.60:0/3716938798 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78000343f0 con 0x7f780815dde0 2023-10-16T12:53:02.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 -- 172.21.15.60:0/3716938798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 msgr2=0x7f78080ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:02.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 --2- 172.21.15.60:0/3716938798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080ac890 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780002bed0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 -- 172.21.15.60:0/3716938798 shutdown_connections 2023-10-16T12:53:02.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 --2- 172.21.15.60:0/3716938798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080ac890 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 --2- 172.21.15.60:0/3716938798 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 -- 172.21.15.60:0/3716938798 >> 172.21.15.60:0/3716938798 conn(0x7f78080ce6a0 msgr2=0x7f78080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:02.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 -- 172.21.15.60:0/3716938798 shutdown_connections 2023-10-16T12:53:02.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.807+0000 7f780759e640 1 -- 172.21.15.60:0/3716938798 wait complete. 2023-10-16T12:53:02.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.809+0000 7f780759e640 1 Processor -- start 2023-10-16T12:53:02.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.809+0000 7f780759e640 1 -- start start 2023-10-16T12:53:02.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.810+0000 7f780759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78080c9230 0x7f78080c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:02.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.810+0000 7f780759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:02.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.810+0000 7f780759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:02.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.810+0000 7f780759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080c86e0 con 0x7f780815dde0 2023-10-16T12:53:02.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.810+0000 7f780759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080c32c0 con 0x7f78080c9230 2023-10-16T12:53:02.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.810+0000 7f780759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080c35a0 con 0x7f780815c9f0 2023-10-16T12:53:02.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:02.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43940/0 (socket says 172.21.15.60:43940) 2023-10-16T12:53:02.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 -- 172.21.15.60:0/3959885693 learned_addr learned my addr 172.21.15.60:0/3959885693 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:02.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f77fffff640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:02.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f780659c640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78080c9230 0x7f78080c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:02.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 -- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 msgr2=0x7f78080c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:02.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 -- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78080c9230 msgr2=0x7f78080c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78080c9230 0x7f78080c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.811+0000 7f7806d9d640 1 -- 172.21.15.60:0/3959885693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7800009b30 con 0x7f780815dde0 2023-10-16T12:53:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.812+0000 7f7806d9d640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080c5cb0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f7800036040 tx=0x7f7800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:02.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.812+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800009510 con 0x7f780815dde0 2023-10-16T12:53:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.812+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780003e450 con 0x7f780815dde0 2023-10-16T12:53:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.812+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78080c3880 con 0x7f780815dde0 2023-10-16T12:53:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.812+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780003e7e0 con 0x7f780815dde0 2023-10-16T12:53:02.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.812+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78080c89b0 con 0x7f780815dde0 2023-10-16T12:53:02.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.813+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78080c3da0 con 0x7f780815dde0 2023-10-16T12:53:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.814+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 61) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f780003e940 con 0x7f780815dde0 2023-10-16T12:53:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.815+0000 7f77feffd640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f77e806b100 0x7f77e806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.815+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f78000ac9a0 con 0x7f780815dde0 2023-10-16T12:53:02.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.816+0000 7f780659c640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f77e806b100 0x7f77e806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:02.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.817+0000 7f780659c640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f77e806b100 0x7f77e806d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006fd0 tx=0x7f77f8008040 comp rx=0 tx=0).ready entity=mgr.6173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:02.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:02.820+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7800077590 con 0x7f780815dde0 2023-10-16T12:53:03.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.068+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f780804e9b0 con 0x7f780815dde0 2023-10-16T12:53:03.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.911+0000 7f77feffd640 1 -- 172.21.15.60:0/3959885693 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v62) v1 ==== 93+0+0 (secure 0 0 0) 0x7f780007abb0 con 0x7f780815dde0 2023-10-16T12:53:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.914+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f77e806b100 msgr2=0x7f77e806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.914+0000 7f780759e640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f77e806b100 0x7f77e806d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006fd0 tx=0x7f77f8008040 comp rx=0 tx=0).stop 2023-10-16T12:53:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.914+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 msgr2=0x7f78080c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.914+0000 7f780759e640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080c5cb0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f7800036040 tx=0x7f7800004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.916+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 shutdown_connections 2023-10-16T12:53:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.916+0000 7f780759e640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f77e806b100 0x7f77e806d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.916+0000 7f780759e640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f780815c9f0 0x7f78080c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.916+0000 7f780759e640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780815dde0 0x7f78080c5cb0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.916+0000 7f780759e640 1 --2- 172.21.15.60:0/3959885693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78080c9230 0x7f78080c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.916+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 >> 172.21.15.60:0/3959885693 conn(0x7f78080ce6a0 msgr2=0x7f780815e700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.917+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 shutdown_connections 2023-10-16T12:53:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:03.917+0000 7f780759e640 1 -- 172.21.15.60:0/3959885693 wait complete. 2023-10-16T12:53:03.944 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:53:03.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:53:04.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:53:04.137 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:53:04.138 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:04.138 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:04.190 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:04.191 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:04.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:04.443 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:04.442+0000 7f4990a2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:53:04.490 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:04.487+0000 7f2ee1a45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:53:04.496 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:04.495+0000 7f4990a2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:53:04.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.494+0000 7fb7062c9640 1 Processor -- start 2023-10-16T12:53:04.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.495+0000 7fb7062c9640 1 -- start start 2023-10-16T12:53:04.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.496+0000 7fb7062c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb70015aaa0 0x7fb70015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.496+0000 7fb7062c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70015ed60 con 0x7fb7000ce390 2023-10-16T12:53:04.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.496+0000 7fb7062c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7000707e0 con 0x7fb70015ae80 2023-10-16T12:53:04.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.496+0000 7fb7062c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70006b810 con 0x7fb70015aaa0 2023-10-16T12:53:04.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.497+0000 7fb704866640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb7000ce390 0x7fb700070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53026/0 (socket says 172.21.15.60:53026) 2023-10-16T12:53:04.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.497+0000 7fb704866640 1 -- 172.21.15.60:0/3567643021 learned_addr learned my addr 172.21.15.60:0/3567643021 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:04.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.497+0000 7fb6fffff640 1 --2- 172.21.15.60:0/3567643021 >> v2:172.21.15.144:3300/0 conn(0x7fb70015aaa0 0x7fb70015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.497+0000 7fb6ff7fe640 1 -- 172.21.15.60:0/3567643021 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170643994 0 0) 0x7fb70015ed60 con 0x7fb7000ce390 2023-10-16T12:53:04.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.497+0000 7fb6ff7fe640 1 -- 172.21.15.60:0/3567643021 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ec003520 con 0x7fb7000ce390 2023-10-16T12:53:04.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.498+0000 7fb6fffff640 1 -- 172.21.15.60:0/3567643021 >> v1:172.21.15.144:6789/0 conn(0x7fb70015ae80 legacy=0x7fb7000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.498+0000 7fb6fffff640 1 -- 172.21.15.60:0/3567643021 >> v1:172.21.15.60:6789/0 conn(0x7fb7000ce390 legacy=0x7fb700070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.498+0000 7fb6fffff640 1 -- 172.21.15.60:0/3567643021 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70015ef90 con 0x7fb70015aaa0 2023-10-16T12:53:04.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.499+0000 7fb6fffff640 1 --2- 172.21.15.60:0/3567643021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb70015aaa0 0x7fb70015e820 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00097a0 tx=0x7fb6f002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=573734eb760b2ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb6ff7fe640 1 -- 172.21.15.60:0/3567643021 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0030610 con 0x7fb70015aaa0 2023-10-16T12:53:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb6ff7fe640 1 -- 172.21.15.60:0/3567643021 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f0030770 con 0x7fb70015aaa0 2023-10-16T12:53:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb6ff7fe640 1 -- 172.21.15.60:0/3567643021 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0030b00 con 0x7fb70015aaa0 2023-10-16T12:53:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb7062c9640 1 -- 172.21.15.60:0/3567643021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb70015aaa0 msgr2=0x7fb70015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb7062c9640 1 --2- 172.21.15.60:0/3567643021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb70015aaa0 0x7fb70015e820 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00097a0 tx=0x7fb6f002b960 comp rx=0 tx=0).stop 2023-10-16T12:53:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb7062c9640 1 -- 172.21.15.60:0/3567643021 shutdown_connections 2023-10-16T12:53:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb7062c9640 1 --2- 172.21.15.60:0/3567643021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb70015aaa0 0x7fb70015e820 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.500+0000 7fb7062c9640 1 -- 172.21.15.60:0/3567643021 >> 172.21.15.60:0/3567643021 conn(0x7fb7000cb410 msgr2=0x7fb7000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.501+0000 7fb7062c9640 1 -- 172.21.15.60:0/3567643021 shutdown_connections 2023-10-16T12:53:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.501+0000 7fb7062c9640 1 -- 172.21.15.60:0/3567643021 wait complete. 2023-10-16T12:53:04.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.502+0000 7fb7062c9640 1 Processor -- start 2023-10-16T12:53:04.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.502+0000 7fb7062c9640 1 -- start start 2023-10-16T12:53:04.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.503+0000 7fb7062c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7000ce390 0x7fb700157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.503+0000 7fb7062c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 0x7fb7000c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.503+0000 7fb7062c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb70015ae80 0x7fb7000c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.503+0000 7fb7062c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7001582d0 con 0x7fb70015aaa0 2023-10-16T12:53:04.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.503+0000 7fb7062c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7000c3740 con 0x7fb7000ce390 2023-10-16T12:53:04.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.503+0000 7fb7062c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb700156fd0 con 0x7fb70015ae80 2023-10-16T12:53:04.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 0x7fb7000c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 0x7fb7000c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43960/0 (socket says 172.21.15.60:43960) 2023-10-16T12:53:04.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 -- 172.21.15.60:0/1773681394 learned_addr learned my addr 172.21.15.60:0/1773681394 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:04.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6fffff640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7000ce390 0x7fb700157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb704866640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb70015ae80 0x7fb7000c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 -- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb70015ae80 msgr2=0x7fb7000c3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb70015ae80 0x7fb7000c3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 -- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7000ce390 msgr2=0x7fb700157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7000ce390 0x7fb700157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb6f7fff640 1 -- 172.21.15.60:0/1773681394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f0009400 con 0x7fb70015aaa0 2023-10-16T12:53:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.504+0000 7fb704866640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb70015ae80 0x7fb7000c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.505+0000 7fb6f7fff640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 0x7fb7000c2ea0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c3c0 tx=0x7fb6e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.505+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80052d0 con 0x7fb70015aaa0 2023-10-16T12:53:04.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.505+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e8005450 con 0x7fb70015aaa0 2023-10-16T12:53:04.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.505+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7001572b0 con 0x7fb70015aaa0 2023-10-16T12:53:04.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.505+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8003830 con 0x7fb70015aaa0 2023-10-16T12:53:04.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.505+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb700157870 con 0x7fb70015aaa0 2023-10-16T12:53:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.507+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb700157a90 con 0x7fb70015aaa0 2023-10-16T12:53:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.507+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb6e8003990 con 0x7fb70015aaa0 2023-10-16T12:53:04.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.509+0000 7fb6fd7fa640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 0x7fb6e006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.509+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fb6e8088fb0 con 0x7fb70015aaa0 2023-10-16T12:53:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.509+0000 7fb6fffff640 1 -- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 msgr2=0x7fb6e006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:04.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.509+0000 7fb6fffff640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 0x7fb6e006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:04.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.513+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb6e80bbfe0 con 0x7fb70015aaa0 2023-10-16T12:53:04.552 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:04.550+0000 7f2ee1a45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:53:04.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.710+0000 7fb6fffff640 1 -- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 msgr2=0x7fb6e006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.710+0000 7fb6fffff640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 0x7fb6e006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:04.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.710+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb70004e160 con 0x7fb70015aaa0 2023-10-16T12:53:04.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.712+0000 7fb6fd7fa640 1 -- 172.21.15.60:0/1773681394 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+17015 (secure 0 0 0) 0x7fb6e8017090 con 0x7fb70015aaa0 2023-10-16T12:53:04.715 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:04.715 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":129,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:47.120212+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3005566427":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/945398958":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/1880380841":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6833/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/4085361535":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/3615970143":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/2155057237":"2023-10-17T12:52:47.120136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:04.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 msgr2=0x7fb6e006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 0x7fb6e006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 msgr2=0x7fb7000c2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 0x7fb7000c2ea0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c3c0 tx=0x7fb6e800c880 comp rx=0 tx=0).stop 2023-10-16T12:53:04.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 shutdown_connections 2023-10-16T12:53:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fb6e006b0b0 0x7fb6e006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb70015ae80 0x7fb7000c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb70015aaa0 0x7fb7000c2ea0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 --2- 172.21.15.60:0/1773681394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7000ce390 0x7fb700157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 >> 172.21.15.60:0/1773681394 conn(0x7fb7000cb410 msgr2=0x7fb7000b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 shutdown_connections 2023-10-16T12:53:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.715+0000 7fb7062c9640 1 -- 172.21.15.60:0/1773681394 wait complete. 2023-10-16T12:53:04.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:52:30.363329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:52:31.424370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:53:04.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:53:04.943 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:04.942+0000 7f4990a2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:53:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.943+0000 7fab85685640 1 Processor -- start 2023-10-16T12:53:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.943+0000 7fab85685640 1 -- start start 2023-10-16T12:53:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab85685640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab800cad80 0x7fab8015dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab85685640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800b4c00 con 0x7fab80159c00 2023-10-16T12:53:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab85685640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800c9fc0 con 0x7fab800b3530 2023-10-16T12:53:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab85685640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab800ca2a0 con 0x7fab800cad80 2023-10-16T12:53:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab7ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fab80159c00 0x7fab8015d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53030/0 (socket says 172.21.15.60:53030) 2023-10-16T12:53:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab7ffff640 1 -- 172.21.15.60:0/3252979291 learned_addr learned my addr 172.21.15.60:0/3252979291 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab777fe640 1 --2- 172.21.15.60:0/3252979291 >> v2:172.21.15.60:3300/0 conn(0x7fab800cad80 0x7fab8015dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab7f7fe640 1 -- 172.21.15.60:0/3252979291 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289842413 0 0) 0x7fab800b4c00 con 0x7fab80159c00 2023-10-16T12:53:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.945+0000 7fab7f7fe640 1 -- 172.21.15.60:0/3252979291 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6c005730 con 0x7fab80159c00 2023-10-16T12:53:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.946+0000 7fab777fe640 1 -- 172.21.15.60:0/3252979291 >> v1:172.21.15.144:6789/0 conn(0x7fab800b3530 legacy=0x7fab800b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.946+0000 7fab777fe640 1 -- 172.21.15.60:0/3252979291 >> v1:172.21.15.60:6789/0 conn(0x7fab80159c00 legacy=0x7fab8015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.946+0000 7fab777fe640 1 -- 172.21.15.60:0/3252979291 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab800ca580 con 0x7fab800cad80 2023-10-16T12:53:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.947+0000 7fab777fe640 1 --2- 172.21.15.60:0/3252979291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8015dd00 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fab680097a0 tx=0x7fab6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34bb09d6f441912 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.947+0000 7fab7f7fe640 1 -- 172.21.15.60:0/3252979291 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031830 con 0x7fab800cad80 2023-10-16T12:53:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.947+0000 7fab7f7fe640 1 -- 172.21.15.60:0/3252979291 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab68031990 con 0x7fab800cad80 2023-10-16T12:53:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.947+0000 7fab7f7fe640 1 -- 172.21.15.60:0/3252979291 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68031d00 con 0x7fab800cad80 2023-10-16T12:53:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 -- 172.21.15.60:0/3252979291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 msgr2=0x7fab8015dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 --2- 172.21.15.60:0/3252979291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8015dd00 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fab680097a0 tx=0x7fab6802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 -- 172.21.15.60:0/3252979291 shutdown_connections 2023-10-16T12:53:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 --2- 172.21.15.60:0/3252979291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8015dd00 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 -- 172.21.15.60:0/3252979291 >> 172.21.15.60:0/3252979291 conn(0x7fab800b0a50 msgr2=0x7fab800b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 -- 172.21.15.60:0/3252979291 shutdown_connections 2023-10-16T12:53:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.948+0000 7fab85685640 1 -- 172.21.15.60:0/3252979291 wait complete. 2023-10-16T12:53:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.950+0000 7fab85685640 1 Processor -- start 2023-10-16T12:53:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.950+0000 7fab85685640 1 -- start start 2023-10-16T12:53:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab85685640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab800b3530 0x7fab8009ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab85685640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab85685640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab80159c00 0x7fab800c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab85685640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8009a900 con 0x7fab800cad80 2023-10-16T12:53:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab85685640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80099a10 con 0x7fab800b3530 2023-10-16T12:53:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab85685640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab80099cf0 con 0x7fab80159c00 2023-10-16T12:53:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab777fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab777fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8009b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43986/0 (socket says 172.21.15.60:43986) 2023-10-16T12:53:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab777fe640 1 -- 172.21.15.60:0/1345968007 learned_addr learned my addr 172.21.15.60:0/1345968007 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab7ffff640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab800b3530 0x7fab8009ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab84e84640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab80159c00 0x7fab800c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.951+0000 7fab777fe640 1 -- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab80159c00 msgr2=0x7fab800c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab777fe640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab80159c00 0x7fab800c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab777fe640 1 -- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab800b3530 msgr2=0x7fab8009ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab777fe640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab800b3530 0x7fab8009ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab777fe640 1 -- 172.21.15.60:0/1345968007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009400 con 0x7fab800cad80 2023-10-16T12:53:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab777fe640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8009b8e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fab6802b910 tx=0x7fab680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab680317b0 con 0x7fab800cad80 2023-10-16T12:53:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab68032e10 con 0x7fab800cad80 2023-10-16T12:53:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68003810 con 0x7fab800cad80 2023-10-16T12:53:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.952+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab80099fd0 con 0x7fab800cad80 2023-10-16T12:53:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.953+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8009a590 con 0x7fab800cad80 2023-10-16T12:53:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.954+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8009aa70 con 0x7fab800cad80 2023-10-16T12:53:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.956+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fab68003970 con 0x7fab800cad80 2023-10-16T12:53:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.957+0000 7fab7d7fa640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 0x7fab6006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:04.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.957+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fab680ab540 con 0x7fab800cad80 2023-10-16T12:53:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.957+0000 7fab7ffff640 1 -- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 msgr2=0x7fab6006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:04.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.957+0000 7fab7ffff640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 0x7fab6006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:04.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:04.966+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab68076160 con 0x7fab800cad80 2023-10-16T12:53:05.000 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:04.999+0000 7f4990a2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:53:05.047 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.044+0000 7f2ee1a45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:53:05.053 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.051+0000 7f4990a2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:53:05.101 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.099+0000 7f2ee1a45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:53:05.113 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.112+0000 7f4990a2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:53:05.154 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.151+0000 7f2ee1a45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:53:05.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.158+0000 7fab7ffff640 1 -- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 msgr2=0x7fab6006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:05.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.158+0000 7fab7ffff640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 0x7fab6006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:05.162 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.160+0000 7f4990a2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:53:05.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.162+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab8004e9b0 con 0x7fab800cad80 2023-10-16T12:53:05.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.163+0000 7fab7d7fa640 1 -- 172.21.15.60:0/1345968007 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab68035020 con 0x7fab800cad80 2023-10-16T12:53:05.165 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:53:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.165+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 msgr2=0x7fab6006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.165+0000 7fab85685640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 0x7fab6006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.165+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 msgr2=0x7fab8009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.165+0000 7fab85685640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8009b8e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fab6802b910 tx=0x7fab680046c0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 shutdown_connections 2023-10-16T12:53:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fab6006b0b0 0x7fab6006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab80159c00 0x7fab800c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab800cad80 0x7fab8009b8e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 --2- 172.21.15.60:0/1345968007 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab800b3530 0x7fab8009ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 >> 172.21.15.60:0/1345968007 conn(0x7fab800b0a50 msgr2=0x7fab800ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 shutdown_connections 2023-10-16T12:53:05.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.166+0000 7fab85685640 1 -- 172.21.15.60:0/1345968007 wait complete. 2023-10-16T12:53:05.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:53:05.219 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.217+0000 7f2ee1a45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:53:05.272 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.269+0000 7f2ee1a45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:53:05.273 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.271+0000 7f4990a2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:53:05.335 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.333+0000 7f4990a2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:05.384 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.382+0000 7f4990a2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:53:05.389 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.387+0000 7f2ee1a45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:53:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.397+0000 7fd3fb511640 1 Processor -- start 2023-10-16T12:53:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.397+0000 7fd3fb511640 1 -- start start 2023-10-16T12:53:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fb511640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3f415cd70 0x7fd3f40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fb511640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd3f40cded0 0x7fd3f40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fb511640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f415d150 con 0x7fd3f40b8480 2023-10-16T12:53:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fb511640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f40d1180 con 0x7fd3f415cd70 2023-10-16T12:53:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fb511640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f40d1460 con 0x7fd3f40cded0 2023-10-16T12:53:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fa50f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3f415cd70 0x7fd3f40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3f9d0e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd3f40b8480 0x7fd3f40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53044/0 (socket says 172.21.15.60:53044) 2023-10-16T12:53:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3f9d0e640 1 -- 172.21.15.60:0/3124555350 learned_addr learned my addr 172.21.15.60:0/3124555350 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fad10640 1 --2- 172.21.15.60:0/3124555350 >> v2:172.21.15.144:3300/0 conn(0x7fd3f40cded0 0x7fd3f40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.399+0000 7fd3fa50f640 1 -- 172.21.15.60:0/3124555350 >> v2:172.21.15.144:3300/0 conn(0x7fd3f40cded0 msgr2=0x7fd3f40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.400+0000 7fd3fa50f640 1 --2- 172.21.15.60:0/3124555350 >> v2:172.21.15.144:3300/0 conn(0x7fd3f40cded0 0x7fd3f40b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.400+0000 7fd3fa50f640 1 -- 172.21.15.60:0/3124555350 >> v1:172.21.15.60:6789/0 conn(0x7fd3f40b8480 legacy=0x7fd3f40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.400+0000 7fd3f950d640 1 -- 172.21.15.60:0/3124555350 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532537278 0 0) 0x7fd3f415d150 con 0x7fd3f40b8480 2023-10-16T12:53:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.400+0000 7fd3fa50f640 1 -- 172.21.15.60:0/3124555350 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40d1740 con 0x7fd3f415cd70 2023-10-16T12:53:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.401+0000 7fd3fa50f640 1 --2- 172.21.15.60:0/3124555350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f40d2220 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40b8910 tx=0x7fd3f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0ca61f28ad97dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.401+0000 7fd3f950d640 1 -- 172.21.15.60:0/3124555350 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0031830 con 0x7fd3f415cd70 2023-10-16T12:53:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.401+0000 7fd3f950d640 1 -- 172.21.15.60:0/3124555350 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f0031990 con 0x7fd3f415cd70 2023-10-16T12:53:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.401+0000 7fd3f950d640 1 -- 172.21.15.60:0/3124555350 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0031d20 con 0x7fd3f415cd70 2023-10-16T12:53:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 -- 172.21.15.60:0/3124555350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 msgr2=0x7fd3f40d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 --2- 172.21.15.60:0/3124555350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f40d2220 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40b8910 tx=0x7fd3f002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 -- 172.21.15.60:0/3124555350 shutdown_connections 2023-10-16T12:53:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 --2- 172.21.15.60:0/3124555350 >> v2:172.21.15.144:3300/0 conn(0x7fd3f40cded0 0x7fd3f40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 --2- 172.21.15.60:0/3124555350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f40d2220 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 -- 172.21.15.60:0/3124555350 >> 172.21.15.60:0/3124555350 conn(0x7fd3f40cf570 msgr2=0x7fd3f40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:05.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 -- 172.21.15.60:0/3124555350 shutdown_connections 2023-10-16T12:53:05.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.402+0000 7fd3fb511640 1 -- 172.21.15.60:0/3124555350 wait complete. 2023-10-16T12:53:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.404+0000 7fd3fb511640 1 Processor -- start 2023-10-16T12:53:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.404+0000 7fd3fb511640 1 -- start start 2023-10-16T12:53:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fb511640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3f40b8480 0x7fd3f4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fb511640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3f40cded0 0x7fd3f40a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fb511640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f415e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fb511640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4149630 con 0x7fd3f415cd70 2023-10-16T12:53:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fb511640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40b2370 con 0x7fd3f40cded0 2023-10-16T12:53:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fb511640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40a9a30 con 0x7fd3f40b8480 2023-10-16T12:53:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fad10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f415e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fad10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f415e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44018/0 (socket says 172.21.15.60:44018) 2023-10-16T12:53:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3fad10640 1 -- 172.21.15.60:0/1377194740 learned_addr learned my addr 172.21.15.60:0/1377194740 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.405+0000 7fd3f9d0e640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3f40cded0 0x7fd3f40a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fa50f640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3f40b8480 0x7fd3f4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fad10640 1 -- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3f40b8480 msgr2=0x7fd3f4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fad10640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3f40b8480 0x7fd3f4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fad10640 1 -- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3f40cded0 msgr2=0x7fd3f40a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fad10640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3f40cded0 0x7fd3f40a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fad10640 1 -- 172.21.15.60:0/1377194740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0009400 con 0x7fd3f415cd70 2023-10-16T12:53:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fad10640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f415e8f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec007d90 tx=0x7fd3ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec014420 con 0x7fd3f415cd70 2023-10-16T12:53:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec0145a0 con 0x7fd3f415cd70 2023-10-16T12:53:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec014950 con 0x7fd3f415cd70 2023-10-16T12:53:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f414c470 con 0x7fd3f415cd70 2023-10-16T12:53:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.406+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f4149b60 con 0x7fd3f415cd70 2023-10-16T12:53:05.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.408+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f404e9b0 con 0x7fd3f415cd70 2023-10-16T12:53:05.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.409+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd3ec014ab0 con 0x7fd3f415cd70 2023-10-16T12:53:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.409+0000 7fd3eaffd640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 0x7fd3d806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.409+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd3ec08a400 con 0x7fd3f415cd70 2023-10-16T12:53:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.410+0000 7fd3fa50f640 1 -- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 msgr2=0x7fd3d806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:05.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.410+0000 7fd3fa50f640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 0x7fd3d806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:05.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.418+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3ec055150 con 0x7fd3f415cd70 2023-10-16T12:53:05.442 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.440+0000 7f4990a2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:53:05.472 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.469+0000 7f2ee1a45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:05.492 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.490+0000 7f4990a2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:53:05.524 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.522+0000 7f2ee1a45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:53:05.583 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.581+0000 7f2ee1a45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:53:05.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.611+0000 7fd3fa50f640 1 -- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 msgr2=0x7fd3d806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:05.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.611+0000 7fd3fa50f640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 0x7fd3d806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:05.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.626+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3f415f7e0 con 0x7fd3f415cd70 2023-10-16T12:53:05.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.627+0000 7fd3eaffd640 1 -- 172.21.15.60:0/1377194740 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3ec058770 con 0x7fd3f415cd70 2023-10-16T12:53:05.629 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:53:05.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.629+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 msgr2=0x7fd3d806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:05.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.629+0000 7fd3fb511640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 0x7fd3d806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.629+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 msgr2=0x7fd3f415e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.629+0000 7fd3fb511640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f415e8f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec007d90 tx=0x7fd3ec00e4b0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 shutdown_connections 2023-10-16T12:53:05.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fd3d806b0b0 0x7fd3d806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f415cd70 0x7fd3f415e8f0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3f40cded0 0x7fd3f40a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 --2- 172.21.15.60:0/1377194740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd3f40b8480 0x7fd3f4148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 >> 172.21.15.60:0/1377194740 conn(0x7fd3f40cf570 msgr2=0x7fd3f40b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:05.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 shutdown_connections 2023-10-16T12:53:05.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.630+0000 7fd3fb511640 1 -- 172.21.15.60:0/1377194740 wait complete. 2023-10-16T12:53:05.636 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:05.634+0000 7f2ee1a45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:53:05.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:53:05.870 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:05.868+0000 7f4990a2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:53:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.904+0000 7f09e4c64640 1 Processor -- start 2023-10-16T12:53:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.904+0000 7f09e4c64640 1 -- start start 2023-10-16T12:53:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.905+0000 7f09e4c64640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09e015d030 0x7f09e00d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.905+0000 7f09e4c64640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d410 0x7f09e0161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.905+0000 7f09e4c64640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e00b5d90 con 0x7f09e0161990 2023-10-16T12:53:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.905+0000 7f09e4c64640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00d2b50 con 0x7f09e015d030 2023-10-16T12:53:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.905+0000 7f09e4c64640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e0160fc0 con 0x7f09e015d410 2023-10-16T12:53:05.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09de59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f09e015d030 0x7f09e00d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09ded9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f09e0161990 0x7f09e00cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53048/0 (socket says 172.21.15.60:53048) 2023-10-16T12:53:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09ded9d640 1 -- 172.21.15.60:0/1861997819 learned_addr learned my addr 172.21.15.60:0/1861997819 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09d7fff640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d410 0x7f09e0161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09de59c640 1 -- 172.21.15.60:0/1861997819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d410 msgr2=0x7f09e0161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09de59c640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d410 0x7f09e0161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09de59c640 1 -- 172.21.15.60:0/1861997819 >> v1:172.21.15.60:6789/0 conn(0x7f09e0161990 legacy=0x7f09e00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09de59c640 1 -- 172.21.15.60:0/1861997819 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00d25d0 con 0x7f09e015d030 2023-10-16T12:53:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09d7fff640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d410 0x7f09e0161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.906+0000 7f09ddd9b640 1 -- 172.21.15.60:0/1861997819 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745596799 0 0) 0x7f09e00b5d90 con 0x7f09e0161990 2023-10-16T12:53:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.907+0000 7f09de59c640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d030 0x7f09e00d1630 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097a0 tx=0x7f09d002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ddc40ac36f12e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09ddd9b640 1 -- 172.21.15.60:0/1861997819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0030800 con 0x7f09e015d030 2023-10-16T12:53:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09ddd9b640 1 -- 172.21.15.60:0/1861997819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d0030960 con 0x7f09e015d030 2023-10-16T12:53:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09ddd9b640 1 -- 172.21.15.60:0/1861997819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0030cf0 con 0x7f09e015d030 2023-10-16T12:53:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09e4c64640 1 -- 172.21.15.60:0/1861997819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d030 msgr2=0x7f09e00d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09e4c64640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d030 0x7f09e00d1630 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097a0 tx=0x7f09d002b940 comp rx=0 tx=0).stop 2023-10-16T12:53:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09e4c64640 1 -- 172.21.15.60:0/1861997819 shutdown_connections 2023-10-16T12:53:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09e4c64640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d410 0x7f09e0161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09e4c64640 1 --2- 172.21.15.60:0/1861997819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d030 0x7f09e00d1630 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.908+0000 7f09e4c64640 1 -- 172.21.15.60:0/1861997819 >> 172.21.15.60:0/1861997819 conn(0x7f09e00cf720 msgr2=0x7f09e00cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.909+0000 7f09e4c64640 1 -- 172.21.15.60:0/1861997819 shutdown_connections 2023-10-16T12:53:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.909+0000 7f09e4c64640 1 -- 172.21.15.60:0/1861997819 wait complete. 2023-10-16T12:53:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.910+0000 7f09e4c64640 1 Processor -- start 2023-10-16T12:53:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.911+0000 7f09e4c64640 1 -- start start 2023-10-16T12:53:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09e4c64640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d030 0x7f09e0147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09e4c64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 0x7f09e0157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09e4c64640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09e0161990 0x7f09e01f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09e4c64640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e014b190 con 0x7f09e015d410 2023-10-16T12:53:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09e4c64640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01f6d40 con 0x7f09e0161990 2023-10-16T12:53:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09e4c64640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01f6a60 con 0x7f09e015d030 2023-10-16T12:53:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 0x7f09e0157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 0x7f09e0157230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44028/0 (socket says 172.21.15.60:44028) 2023-10-16T12:53:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 -- 172.21.15.60:0/802019974 learned_addr learned my addr 172.21.15.60:0/802019974 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09de59c640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d030 0x7f09e0147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09ded9d640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09e0161990 0x7f09e01f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:05.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 -- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d030 msgr2=0x7f09e0147950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d030 0x7f09e0147950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 -- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09e0161990 msgr2=0x7f09e01f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09e0161990 0x7f09e01f5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09d7fff640 1 -- 172.21.15.60:0/802019974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d0009400 con 0x7f09e015d410 2023-10-16T12:53:05.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.912+0000 7f09ded9d640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09e0161990 0x7f09e01f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.913+0000 7f09d7fff640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 0x7f09e0157230 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f09c800c7f0 tx=0x7f09c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.913+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8007a30 con 0x7f09e015d410 2023-10-16T12:53:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.913+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c8007b90 con 0x7f09e015d410 2023-10-16T12:53:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.913+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c800ce40 con 0x7f09e015d410 2023-10-16T12:53:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.913+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e0146e70 con 0x7f09e015d410 2023-10-16T12:53:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.913+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e0157770 con 0x7f09e015d410 2023-10-16T12:53:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.915+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e0157960 con 0x7f09e015d410 2023-10-16T12:53:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.917+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f09c80113f0 con 0x7f09e015d410 2023-10-16T12:53:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.917+0000 7f09d6ffd640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f09c006b0e0 0x7f09c006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.918+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f09c8089e00 con 0x7f09e015d410 2023-10-16T12:53:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.918+0000 7f09de59c640 1 -- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f09c006b0e0 msgr2=0x7f09c006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.918+0000 7f09de59c640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f09c006b0e0 0x7f09c006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:05.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:05.922+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09c80548e0 con 0x7f09e015d410 2023-10-16T12:53:06.030 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.029+0000 7f4990a2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:53:06.040 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.038+0000 7f2ee1a45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:53:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.115+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09e0147670 con 0x7f09e015d410 2023-10-16T12:53:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.116+0000 7f09d6ffd640 1 -- 172.21.15.60:0/802019974 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09e0147670 con 0x7f09e015d410 2023-10-16T12:53:06.119 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:53:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.118+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f09c006b0e0 msgr2=0x7f09c006d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.118+0000 7f09e4c64640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f09c006b0e0 0x7f09c006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.118+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 msgr2=0x7f09e0157230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.118+0000 7f09e4c64640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 0x7f09e0157230 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f09c800c7f0 tx=0x7f09c800cbf0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 shutdown_connections 2023-10-16T12:53:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f09c006b0e0 0x7f09c006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09e0161990 0x7f09e01f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e015d410 0x7f09e0157230 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 --2- 172.21.15.60:0/802019974 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f09e015d030 0x7f09e0147950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 >> 172.21.15.60:0/802019974 conn(0x7f09e00cf720 msgr2=0x7f09e00b80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 shutdown_connections 2023-10-16T12:53:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.119+0000 7f09e4c64640 1 -- 172.21.15.60:0/802019974 wait complete. 2023-10-16T12:53:06.138 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.135+0000 7f4990a2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:53:06.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:53:06.189 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.186+0000 7f4990a2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:53:06.210 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.208+0000 7f2ee1a45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:53:06.295 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.293+0000 7f4990a2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:06.325 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.323+0000 7f2ee1a45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:53:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.343+0000 7ff8fb101640 1 Processor -- start 2023-10-16T12:53:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.343+0000 7ff8fb101640 1 -- start start 2023-10-16T12:53:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.345+0000 7ff8fb101640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8f40b3420 0x7ff8f40b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.345+0000 7ff8fb101640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40d3980 con 0x7ff8f40cfaf0 2023-10-16T12:53:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.345+0000 7ff8fb101640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40ce630 con 0x7ff8f40cf710 2023-10-16T12:53:06.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.345+0000 7ff8fb101640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f40d3c60 con 0x7ff8f40b3420 2023-10-16T12:53:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.345+0000 7ff8f969e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff8f40cfaf0 0x7ff8f40b6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53058/0 (socket says 172.21.15.60:53058) 2023-10-16T12:53:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.345+0000 7ff8f969e640 1 -- 172.21.15.60:0/2389989380 learned_addr learned my addr 172.21.15.60:0/2389989380 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8ebfff640 1 --2- 172.21.15.60:0/2389989380 >> v2:172.21.15.144:3300/0 conn(0x7ff8f40b3420 0x7ff8f40b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133086069 0 0) 0x7ff8f40d3980 con 0x7ff8f40cfaf0 2023-10-16T12:53:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc003520 con 0x7ff8f40cfaf0 2023-10-16T12:53:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3676903790 0 0) 0x7ff8dc003520 con 0x7ff8f40cfaf0 2023-10-16T12:53:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 >> v2:172.21.15.144:3300/0 conn(0x7ff8f40b3420 msgr2=0x7ff8f40b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 --2- 172.21.15.60:0/2389989380 >> v2:172.21.15.144:3300/0 conn(0x7ff8f40b3420 0x7ff8f40b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 >> v1:172.21.15.144:6789/0 conn(0x7ff8f40cf710 legacy=0x7ff8f40b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40ce910 con 0x7ff8f40cfaf0 2023-10-16T12:53:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff8ec0019f0 con 0x7ff8f40cfaf0 2023-10-16T12:53:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 >> v1:172.21.15.60:6789/0 conn(0x7ff8f40cfaf0 legacy=0x7ff8f40b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc0047c0 0x7ff8dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.346+0000 7ff8eb7fe640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8dc0051d0 0x7ff8dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40d3c60 con 0x7ff8dc0039c0 2023-10-16T12:53:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00c120 con 0x7ff8dc0047c0 2023-10-16T12:53:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00c2e0 con 0x7ff8dc0051d0 2023-10-16T12:53:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8ebfff640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc0047c0 0x7ff8dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8dc0051d0 msgr2=0x7ff8dc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8dc0051d0 0x7ff8dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc0047c0 msgr2=0x7ff8dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8dc0047c0 0x7ff8dc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2389989380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc003520 con 0x7ff8dc0039c0 2023-10-16T12:53:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.347+0000 7ff8f969e640 1 -- 172.21.15.60:0/2389989380 reap_dead start 2023-10-16T12:53:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.348+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8dc003de0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0007fd0 tx=0x7ff8dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f479d26101e304f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.348+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00052d0 con 0x7ff8dc0039c0 2023-10-16T12:53:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.348+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f002e050 con 0x7ff8dc0039c0 2023-10-16T12:53:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.348+0000 7ff8eb7fe640 1 -- 172.21.15.60:0/2389989380 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0003640 con 0x7ff8dc0039c0 2023-10-16T12:53:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 -- 172.21.15.60:0/2389989380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 msgr2=0x7ff8dc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8dc003de0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0007fd0 tx=0x7ff8dc0057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 -- 172.21.15.60:0/2389989380 shutdown_connections 2023-10-16T12:53:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2389989380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8dc003de0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 -- 172.21.15.60:0/2389989380 >> 172.21.15.60:0/2389989380 conn(0x7ff8f40d2cc0 msgr2=0x7ff8f40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 -- 172.21.15.60:0/2389989380 shutdown_connections 2023-10-16T12:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.349+0000 7ff8fb101640 1 -- 172.21.15.60:0/2389989380 wait complete. 2023-10-16T12:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.351+0000 7ff8fb101640 1 Processor -- start 2023-10-16T12:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.352+0000 7ff8fb101640 1 -- start start 2023-10-16T12:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8fb101640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8f41f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8fb101640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f40b1000 0x7ff8f415b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8fb101640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f40b1480 0x7ff8f415b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8fb101640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f415e9b0 con 0x7ff8dc0039c0 2023-10-16T12:53:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8fb101640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f415dda0 con 0x7ff8f40b1480 2023-10-16T12:53:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8fb101640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f415e080 con 0x7ff8f40b1000 2023-10-16T12:53:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8f8e9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8f41f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8f8e9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8f41f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44044/0 (socket says 172.21.15.60:44044) 2023-10-16T12:53:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2589756103 learned_addr learned my addr 172.21.15.60:0/2589756103 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f40b1000 msgr2=0x7ff8f415b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f40b1000 0x7ff8f415b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.353+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f40b1480 msgr2=0x7ff8f415b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f40b1480 0x7ff8f415b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8f8e9d640 1 -- 172.21.15.60:0/2589756103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f415e360 con 0x7ff8dc0039c0 2023-10-16T12:53:06.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8f8e9d640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8f41f3380 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ff8f002bbb0 tx=0x7ff8f0001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:06.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0006ea0 con 0x7ff8dc0039c0 2023-10-16T12:53:06.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f0007a80 con 0x7ff8dc0039c0 2023-10-16T12:53:06.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f415e640 con 0x7ff8dc0039c0 2023-10-16T12:53:06.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0007450 con 0x7ff8dc0039c0 2023-10-16T12:53:06.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.354+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f41f38c0 con 0x7ff8dc0039c0 2023-10-16T12:53:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.356+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f404e160 con 0x7ff8dc0039c0 2023-10-16T12:53:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.357+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff8f00075b0 con 0x7ff8dc0039c0 2023-10-16T12:53:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.358+0000 7ff8e97fa640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 0x7ff8e006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.358+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff8f00ab790 con 0x7ff8dc0039c0 2023-10-16T12:53:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.358+0000 7ff8ebfff640 1 -- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 msgr2=0x7ff8e006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.358+0000 7ff8ebfff640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 0x7ff8e006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.362+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8f0076450 con 0x7ff8dc0039c0 2023-10-16T12:53:06.381 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.378+0000 7f2ee1a45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:53:06.403 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.400+0000 7f4990a2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:53:06.494 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.492+0000 7f2ee1a45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:06.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.555+0000 7f4990a2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:06.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.559+0000 7ff8ebfff640 1 -- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 msgr2=0x7ff8e006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.559+0000 7ff8ebfff640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 0x7ff8e006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.571+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff8f4071840 con 0x7ff8dc0039c0 2023-10-16T12:53:06.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.572+0000 7ff8e97fa640 1 -- 172.21.15.60:0/2589756103 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v129) v1 ==== 80+0+477 (secure 0 0 0) 0x7ff8f0079a70 con 0x7ff8dc0039c0 2023-10-16T12:53:06.575 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3005566427 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.576 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/945398958 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.576 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1880380841 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.576 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/3337897861 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.576 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/3337897861 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.577 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/4085361535 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.578 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3615970143 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.578 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2155057237 2023-10-17T12:52:47.120136+0000 2023-10-16T12:53:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T12:53:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.575+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 msgr2=0x7ff8e006d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.575+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 0x7ff8e006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.575+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 msgr2=0x7ff8f41f3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.575+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8f41f3380 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ff8f002bbb0 tx=0x7ff8f0001170 comp rx=0 tx=0).stop 2023-10-16T12:53:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 shutdown_connections 2023-10-16T12:53:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7ff8e006b0e0 0x7ff8e006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8f40b1000 0x7ff8f415b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8f40b1480 0x7ff8f415b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 --2- 172.21.15.60:0/2589756103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8dc0039c0 0x7ff8f41f3380 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 >> 172.21.15.60:0/2589756103 conn(0x7ff8f40d2cc0 msgr2=0x7ff8f40d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 shutdown_connections 2023-10-16T12:53:06.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:06.576+0000 7ff8fb101640 1 -- 172.21.15.60:0/2589756103 wait complete. 2023-10-16T12:53:06.597 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:53:06.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:06.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:53:06.608 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.605+0000 7f4990a2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:06.609 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.606+0000 7f2ee1a45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:53:06.699 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:53:06.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:06.700 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:53:06.729 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:06.725+0000 7f4990a2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:53:06.778 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.775+0000 7f2ee1a45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:06.791 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:53:06.792 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:06.829 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.827+0000 7f2ee1a45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:06.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:53:06.846 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:06.846 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:06.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:53:06.904 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:53:06.904 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:06.956 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:06.953+0000 7f2ee1a45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:53:06.958 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:53:06.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:07.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.168+0000 7fabd759e640 1 Processor -- start 2023-10-16T12:53:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.168+0000 7fabd759e640 1 -- start start 2023-10-16T12:53:07.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fabd815ac80 0x7fabd80718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 0x7fabd80cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd80cca60 con 0x7fabd80cd870 2023-10-16T12:53:07.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd80ccd40 con 0x7fabd815ac80 2023-10-16T12:53:07.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd80cd020 con 0x7fabd80cdc50 2023-10-16T12:53:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd6d9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 0x7fabd80cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd5d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fabd815ac80 0x7fabd80718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd6d9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 0x7fabd80cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:54218/0 (socket says 172.21.15.60:54218) 2023-10-16T12:53:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.170+0000 7fabd6d9d640 1 -- 172.21.15.60:0/3637589563 learned_addr learned my addr 172.21.15.60:0/3637589563 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.171+0000 7fabd6d9d640 1 -- 172.21.15.60:0/3637589563 >> v2:172.21.15.144:3300/0 conn(0x7fabd815ac80 msgr2=0x7fabd80718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.171+0000 7fabd6d9d640 1 --2- 172.21.15.60:0/3637589563 >> v2:172.21.15.144:3300/0 conn(0x7fabd815ac80 0x7fabd80718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.171+0000 7fabd6d9d640 1 -- 172.21.15.60:0/3637589563 >> v1:172.21.15.144:6789/0 conn(0x7fabd80cd870 legacy=0x7fabd80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.171+0000 7fabd6d9d640 1 -- 172.21.15.60:0/3637589563 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd80ce4b0 con 0x7fabd80cdc50 2023-10-16T12:53:07.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.171+0000 7fabd5d9b640 1 --2- 172.21.15.60:0/3637589563 >> v2:172.21.15.144:3300/0 conn(0x7fabd815ac80 0x7fabd80718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:53:07.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.172+0000 7fabd6d9d640 1 --2- 172.21.15.60:0/3637589563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 0x7fabd80cfab0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009e80 tx=0x7fabd002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0aa3a82ad2873a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:07.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.173+0000 7fabd559a640 1 -- 172.21.15.60:0/3637589563 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd002ca50 con 0x7fabd80cdc50 2023-10-16T12:53:07.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.173+0000 7fabd559a640 1 -- 172.21.15.60:0/3637589563 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd002cbb0 con 0x7fabd80cdc50 2023-10-16T12:53:07.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.173+0000 7fabd559a640 1 -- 172.21.15.60:0/3637589563 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd00343f0 con 0x7fabd80cdc50 2023-10-16T12:53:07.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 -- 172.21.15.60:0/3637589563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 msgr2=0x7fabd80cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 --2- 172.21.15.60:0/3637589563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 0x7fabd80cfab0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009e80 tx=0x7fabd002beb0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 -- 172.21.15.60:0/3637589563 shutdown_connections 2023-10-16T12:53:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 --2- 172.21.15.60:0/3637589563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cdc50 0x7fabd80cfab0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 --2- 172.21.15.60:0/3637589563 >> v2:172.21.15.144:3300/0 conn(0x7fabd815ac80 0x7fabd80718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 -- 172.21.15.60:0/3637589563 >> 172.21.15.60:0/3637589563 conn(0x7fabd8065ed0 msgr2=0x7fabd80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 -- 172.21.15.60:0/3637589563 shutdown_connections 2023-10-16T12:53:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.174+0000 7fabd759e640 1 -- 172.21.15.60:0/3637589563 wait complete. 2023-10-16T12:53:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.176+0000 7fabd759e640 1 Processor -- start 2023-10-16T12:53:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.177+0000 7fabd759e640 1 -- start start 2023-10-16T12:53:07.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cd870 0x7fabd80c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 0x7fabd80aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabd815ac80 0x7fabd8151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd80c6570 con 0x7fabd815ac80 2023-10-16T12:53:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd80c67c0 con 0x7fabd80cdc50 2023-10-16T12:53:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd80c6aa0 con 0x7fabd80cd870 2023-10-16T12:53:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabd815ac80 0x7fabd8151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd5d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 0x7fabd80aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd659c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cd870 0x7fabd80c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd5d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 0x7fabd80aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:59414/0 (socket says 172.21.15.60:59414) 2023-10-16T12:53:07.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd5d9b640 1 -- 172.21.15.60:0/3691658709 learned_addr learned my addr 172.21.15.60:0/3691658709 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:07.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.178+0000 7fabd6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabd815ac80 0x7fabd8151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44046/0 (socket says 172.21.15.60:44046) 2023-10-16T12:53:07.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.179+0000 7fabd5d9b640 1 -- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cd870 msgr2=0x7fabd80c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.179+0000 7fabd5d9b640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cd870 0x7fabd80c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.179+0000 7fabd5d9b640 1 -- 172.21.15.60:0/3691658709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabd815ac80 msgr2=0x7fabd8151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.179+0000 7fabd5d9b640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabd815ac80 0x7fabd8151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.179+0000 7fabd5d9b640 1 -- 172.21.15.60:0/3691658709 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc009400 con 0x7fabd80cdc50 2023-10-16T12:53:07.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.179+0000 7fabd5d9b640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 0x7fabd80aece0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9b0 tx=0x7fabcc00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:07.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.180+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00ca00 con 0x7fabd80cdc50 2023-10-16T12:53:07.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.180+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc00cb60 con 0x7fabd80cdc50 2023-10-16T12:53:07.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.180+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd0009b30 con 0x7fabd80cdc50 2023-10-16T12:53:07.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.180+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc013470 con 0x7fabd80cdc50 2023-10-16T12:53:07.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.180+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd80c7060 con 0x7fabd80cdc50 2023-10-16T12:53:07.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.181+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd80cf8f0 con 0x7fabd80cdc50 2023-10-16T12:53:07.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.183+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fabcc004020 con 0x7fabd80cdc50 2023-10-16T12:53:07.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.184+0000 7fabc6ffd640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 0x7fabb806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.184+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fabcc089ee0 con 0x7fabd80cdc50 2023-10-16T12:53:07.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.184+0000 7fabd659c640 1 -- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 msgr2=0x7fabb806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:07.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.185+0000 7fabd659c640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 0x7fabb806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:07.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.194+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fabcc0b6fe0 con 0x7fabd80cdc50 2023-10-16T12:53:07.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.386+0000 7fabd659c640 1 -- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 msgr2=0x7fabb806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:07.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.386+0000 7fabd659c640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 0x7fabb806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.430+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd80c6900 con 0x7fabd80cdc50 2023-10-16T12:53:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.431+0000 7fabc6ffd640 1 -- 172.21.15.60:0/3691658709 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fabd80c6900 con 0x7fabd80cdc50 2023-10-16T12:53:07.434 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:07.434 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:52:32.396002+0000","modified":"2023-10-16T12:52:35.483028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7866":{"gid":7866,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2146889084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2146889084},{"type":"v1","addr":"172.21.15.60:6833","nonce":2146889084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:53:07.435 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2023-10-16T12:53:07.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.433+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 msgr2=0x7fabb806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.433+0000 7fabd759e640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 0x7fabb806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 msgr2=0x7fabd80aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 0x7fabd80aece0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9b0 tx=0x7fabcc00ee70 comp rx=0 tx=0).stop 2023-10-16T12:53:07.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 shutdown_connections 2023-10-16T12:53:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7fabb806b0b0 0x7fabb806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabd815ac80 0x7fabd8151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabd80cdc50 0x7fabd80aece0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 --2- 172.21.15.60:0/3691658709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fabd80cd870 0x7fabd80c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 >> 172.21.15.60:0/3691658709 conn(0x7fabd8065ed0 msgr2=0x7fabd806d550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 shutdown_connections 2023-10-16T12:53:07.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.434+0000 7fabd759e640 1 -- 172.21.15.60:0/3691658709 wait complete. 2023-10-16T12:53:07.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:07.590 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:07.587+0000 7f4990a2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:53:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.646+0000 7f0e7759e640 1 Processor -- start 2023-10-16T12:53:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.646+0000 7f0e7759e640 1 -- start start 2023-10-16T12:53:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.647+0000 7f0e7759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e7815cbe0 0x7f0e780d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.647+0000 7f0e7759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e780d15a0 con 0x7f0e7815cfc0 2023-10-16T12:53:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.647+0000 7f0e7759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e780d1880 con 0x7f0e780b4c80 2023-10-16T12:53:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.647+0000 7f0e7759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e780d1b60 con 0x7f0e7815cbe0 2023-10-16T12:53:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.648+0000 7f0e75d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0e7815cfc0 0x7f0e780b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53066/0 (socket says 172.21.15.60:53066) 2023-10-16T12:53:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.648+0000 7f0e75d9b640 1 -- 172.21.15.60:0/2496089607 learned_addr learned my addr 172.21.15.60:0/2496089607 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.648+0000 7f0e7659c640 1 --2- 172.21.15.60:0/2496089607 >> v2:172.21.15.144:3300/0 conn(0x7f0e7815cbe0 0x7f0e780d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.648+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333787216 0 0) 0x7f0e780d15a0 con 0x7f0e7815cfc0 2023-10-16T12:53:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.648+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e64003520 con 0x7f0e7815cfc0 2023-10-16T12:53:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.648+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 525140595 0 0) 0x7f0e64003520 con 0x7f0e7815cfc0 2023-10-16T12:53:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 >> v2:172.21.15.144:3300/0 conn(0x7f0e7815cbe0 msgr2=0x7f0e780d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 --2- 172.21.15.60:0/2496089607 >> v2:172.21.15.144:3300/0 conn(0x7f0e7815cbe0 0x7f0e780d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 >> v1:172.21.15.144:6789/0 conn(0x7f0e780b4c80 legacy=0x7f0e780b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e780d1e40 con 0x7f0e7815cfc0 2023-10-16T12:53:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0e60001670 con 0x7f0e7815cfc0 2023-10-16T12:53:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 >> v1:172.21.15.60:6789/0 conn(0x7f0e7815cfc0 legacy=0x7f0e780b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e640039c0 0x7f0e64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e640047c0 0x7f0e64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e64005180 0x7f0e6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e780d1b60 con 0x7f0e640047c0 2023-10-16T12:53:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6400c080 con 0x7f0e64005180 2023-10-16T12:53:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6400c240 con 0x7f0e640039c0 2023-10-16T12:53:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e75d9b640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e640047c0 0x7f0e64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e76d9d640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e64005180 0x7f0e6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e75d9b640 1 -- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e640039c0 msgr2=0x7f0e64003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.649+0000 7f0e7659c640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e640039c0 0x7f0e64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.650+0000 7f0e75d9b640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e640039c0 0x7f0e64003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.650+0000 7f0e75d9b640 1 -- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e64005180 msgr2=0x7f0e6400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.650+0000 7f0e75d9b640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e64005180 0x7f0e6400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.650+0000 7f0e75d9b640 1 -- 172.21.15.60:0/2496089607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e64003520 con 0x7f0e640047c0 2023-10-16T12:53:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.650+0000 7f0e76d9d640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e64005180 0x7f0e6400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:07.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.650+0000 7f0e76d9d640 1 -- 172.21.15.60:0/2496089607 reap_dead start 2023-10-16T12:53:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.651+0000 7f0e75d9b640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e640047c0 0x7f0e64004c40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0e60007fb0 tx=0x7f0e640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2badf72aa6af03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.651+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60005ad0 con 0x7f0e640047c0 2023-10-16T12:53:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.651+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e60005c30 con 0x7f0e640047c0 2023-10-16T12:53:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.651+0000 7f0e7559a640 1 -- 172.21.15.60:0/2496089607 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6002c7c0 con 0x7f0e640047c0 2023-10-16T12:53:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 -- 172.21.15.60:0/2496089607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e640047c0 msgr2=0x7f0e64004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e640047c0 0x7f0e64004c40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0e60007fb0 tx=0x7f0e640057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 -- 172.21.15.60:0/2496089607 shutdown_connections 2023-10-16T12:53:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 --2- 172.21.15.60:0/2496089607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e640047c0 0x7f0e64004c40 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 -- 172.21.15.60:0/2496089607 >> 172.21.15.60:0/2496089607 conn(0x7f0e780cf570 msgr2=0x7f0e780cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 -- 172.21.15.60:0/2496089607 shutdown_connections 2023-10-16T12:53:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.652+0000 7f0e7759e640 1 -- 172.21.15.60:0/2496089607 wait complete. 2023-10-16T12:53:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.654+0000 7f0e7759e640 1 Processor -- start 2023-10-16T12:53:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.654+0000 7f0e7759e640 1 -- start start 2023-10-16T12:53:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e640047c0 0x7f0e7814a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e7814cc90 0x7f0e7814d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 0x7f0e7814b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7814bbf0 con 0x7f0e7814b130 2023-10-16T12:53:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7814bed0 con 0x7f0e640047c0 2023-10-16T12:53:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7814c1b0 con 0x7f0e7814cc90 2023-10-16T12:53:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e76d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 0x7f0e7814b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e76d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 0x7f0e7814b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44058/0 (socket says 172.21.15.60:44058) 2023-10-16T12:53:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e76d9d640 1 -- 172.21.15.60:0/983747016 learned_addr learned my addr 172.21.15.60:0/983747016 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.655+0000 7f0e7659c640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e640047c0 0x7f0e7814a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e75d9b640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e7814cc90 0x7f0e7814d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e76d9d640 1 -- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e7814cc90 msgr2=0x7f0e7814d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e76d9d640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e7814cc90 0x7f0e7814d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e76d9d640 1 -- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e640047c0 msgr2=0x7f0e7814a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e76d9d640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e640047c0 0x7f0e7814a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e76d9d640 1 -- 172.21.15.60:0/983747016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7814c490 con 0x7f0e7814b130 2023-10-16T12:53:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e75d9b640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e7814cc90 0x7f0e7814d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e76d9d640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 0x7f0e7814b5b0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f0e780d3b20 tx=0x7f0e7000d8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7000b070 con 0x7f0e7814b130 2023-10-16T12:53:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70003e60 con 0x7f0e7814b130 2023-10-16T12:53:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7814c770 con 0x7f0e7814b130 2023-10-16T12:53:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70005310 con 0x7f0e7814b130 2023-10-16T12:53:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.656+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7814add0 con 0x7f0e7814b130 2023-10-16T12:53:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.657+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7814c9b0 con 0x7f0e7814b130 2023-10-16T12:53:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.658+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 62) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0e70005470 con 0x7f0e7814b130 2023-10-16T12:53:07.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.659+0000 7f0e6effd640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f0e5806b0e0 0x7f0e5806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.659+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0e70089580 con 0x7f0e7814b130 2023-10-16T12:53:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.660+0000 7f0e7659c640 1 -- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f0e5806b0e0 msgr2=0x7f0e5806d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3420535028 2023-10-16T12:53:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.660+0000 7f0e7659c640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f0e5806b0e0 0x7f0e5806d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.664+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e7814c9b0 con 0x7f0e7814b130 2023-10-16T12:53:07.817 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:07.814+0000 7f2ee1a45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:53:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.857+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e7814c9b0 con 0x7f0e7814b130 2023-10-16T12:53:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.858+0000 7f0e6effd640 1 -- 172.21.15.60:0/983747016 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+17015 (secure 0 0 0) 0x7f0e7814c9b0 con 0x7f0e7814b130 2023-10-16T12:53:07.861 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:07.861 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":129,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:52:47.120212+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3005566427":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/945398958":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/1880380841":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6833/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/4085361535":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/3615970143":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/2155057237":"2023-10-17T12:52:47.120136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f0e5806b0e0 msgr2=0x7f0e5806d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f0e5806b0e0 0x7f0e5806d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 msgr2=0x7f0e7814b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 0x7f0e7814b5b0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f0e780d3b20 tx=0x7f0e7000d8e0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 shutdown_connections 2023-10-16T12:53:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:6832/3420535028,v1:172.21.15.144:6833/3420535028] conn(0x7f0e5806b0e0 0x7f0e5806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e7814cc90 0x7f0e7814d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e7814b130 0x7f0e7814b5b0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 --2- 172.21.15.60:0/983747016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e640047c0 0x7f0e7814a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 >> 172.21.15.60:0/983747016 conn(0x7f0e780cf570 msgr2=0x7f0e780b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 shutdown_connections 2023-10-16T12:53:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:07.861+0000 7f0e7759e640 1 -- 172.21.15.60:0/983747016 wait complete. 2023-10-16T12:53:07.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:52:30.363329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:52:31.424370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:53:07.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:53:07.885 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:07.882+0000 7f4990a2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:53:07.950 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:07.948+0000 7f4990a2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:53:08.065 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.063+0000 7f4990a2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:53:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.092+0000 7f169a1fb640 1 Processor -- start 2023-10-16T12:53:08.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.092+0000 7f169a1fb640 1 -- start start 2023-10-16T12:53:08.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.094+0000 7f169a1fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16940b2080 0x7f16940d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.094+0000 7f169a1fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940cfc40 0x7f16940b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.094+0000 7f169a1fb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16940d0970 con 0x7f16940b1ca0 2023-10-16T12:53:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.094+0000 7f169a1fb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16940d0ea0 con 0x7f16940b2080 2023-10-16T12:53:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.094+0000 7f169a1fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940d0bc0 con 0x7f16940cfc40 2023-10-16T12:53:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.094+0000 7f16989f8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16940b2080 0x7f16940d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16991f9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f16940b1ca0 0x7f16940ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:38614/0 (socket says 172.21.15.60:38614) 2023-10-16T12:53:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16991f9640 1 -- 172.21.15.60:0/1231086211 learned_addr learned my addr 172.21.15.60:0/1231086211 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16999fa640 1 --2- 172.21.15.60:0/1231086211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940cfc40 0x7f16940b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16989f8640 1 -- 172.21.15.60:0/1231086211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940cfc40 msgr2=0x7f16940b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16989f8640 1 --2- 172.21.15.60:0/1231086211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940cfc40 0x7f16940b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16989f8640 1 -- 172.21.15.60:0/1231086211 >> v1:172.21.15.144:6789/0 conn(0x7f16940b1ca0 legacy=0x7f16940ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.095+0000 7f16989f8640 1 -- 172.21.15.60:0/1231086211 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16940cd940 con 0x7f16940b2080 2023-10-16T12:53:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.098+0000 7f16989f8640 1 --2- 172.21.15.60:0/1231086211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b2080 0x7f16940d36b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f16840097a0 tx=0x7f168402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1517425effecea7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.098+0000 7f168bfff640 1 -- 172.21.15.60:0/1231086211 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684030800 con 0x7f16940b2080 2023-10-16T12:53:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.099+0000 7f168bfff640 1 -- 172.21.15.60:0/1231086211 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684030960 con 0x7f16940b2080 2023-10-16T12:53:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.099+0000 7f168bfff640 1 -- 172.21.15.60:0/1231086211 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684030cf0 con 0x7f16940b2080 2023-10-16T12:53:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.099+0000 7f169a1fb640 1 -- 172.21.15.60:0/1231086211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b2080 msgr2=0x7f16940d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.099+0000 7f169a1fb640 1 --2- 172.21.15.60:0/1231086211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b2080 0x7f16940d36b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f16840097a0 tx=0x7f168402b940 comp rx=0 tx=0).stop 2023-10-16T12:53:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.100+0000 7f169a1fb640 1 -- 172.21.15.60:0/1231086211 shutdown_connections 2023-10-16T12:53:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.100+0000 7f169a1fb640 1 --2- 172.21.15.60:0/1231086211 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940cfc40 0x7f16940b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.100+0000 7f169a1fb640 1 --2- 172.21.15.60:0/1231086211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b2080 0x7f16940d36b0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.100+0000 7f169a1fb640 1 -- 172.21.15.60:0/1231086211 >> 172.21.15.60:0/1231086211 conn(0x7f16940d2b90 msgr2=0x7f16940d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.100+0000 7f169a1fb640 1 -- 172.21.15.60:0/1231086211 shutdown_connections 2023-10-16T12:53:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.100+0000 7f169a1fb640 1 -- 172.21.15.60:0/1231086211 wait complete. 2023-10-16T12:53:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.102+0000 7f169a1fb640 1 Processor -- start 2023-10-16T12:53:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.102+0000 7f169a1fb640 1 -- start start 2023-10-16T12:53:08.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f169a1fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 0x7f169415df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f169a1fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940b2080 0x7f16941e7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f169a1fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16940cfc40 0x7f16940b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f169a1fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940c6860 con 0x7f16940b1ca0 2023-10-16T12:53:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f169a1fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940c6b40 con 0x7f16940cfc40 2023-10-16T12:53:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f169a1fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940c6e20 con 0x7f16940b2080 2023-10-16T12:53:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f16991f9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 0x7f169415df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f16991f9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 0x7f169415df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44084/0 (socket says 172.21.15.60:44084) 2023-10-16T12:53:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 -- 172.21.15.60:0/975132453 learned_addr learned my addr 172.21.15.60:0/975132453 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.103+0000 7f16989f8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940b2080 0x7f16941e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16999fa640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16940cfc40 0x7f16940b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 -- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940b2080 msgr2=0x7f16941e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940b2080 0x7f16941e7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 -- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16940cfc40 msgr2=0x7f16940b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16940cfc40 0x7f16940b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 -- 172.21.15.60:0/975132453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1684009400 con 0x7f16940b1ca0 2023-10-16T12:53:08.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.104+0000 7f16991f9640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 0x7f169415df20 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f169000c480 tx=0x7f169000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.105+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690005ce0 con 0x7f16940b1ca0 2023-10-16T12:53:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.105+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690005e40 con 0x7f16940b1ca0 2023-10-16T12:53:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.105+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16900072f0 con 0x7f16940b1ca0 2023-10-16T12:53:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.105+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16940c7100 con 0x7f16940b1ca0 2023-10-16T12:53:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.105+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16940b7ef0 con 0x7f16940b1ca0 2023-10-16T12:53:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.106+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16940c7620 con 0x7f16940b1ca0 2023-10-16T12:53:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.108+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 63) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1690007510 con 0x7f16940b1ca0 2023-10-16T12:53:08.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.109+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f1690088b70 con 0x7f16940b1ca0 2023-10-16T12:53:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.113+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1690053ea0 con 0x7f16940b1ca0 2023-10-16T12:53:08.154 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:08.152+0000 7f2ee1a45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:53:08.155 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.153+0000 7f4913e0a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.155 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.153+0000 7f4913e0a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.156 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.153+0000 7f4913e0a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.156 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.153+0000 7f4913e0a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.156 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.153+0000 7f4913e0a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.157 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.154+0000 7f4910e04640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.157 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.154+0000 7f4910e04640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.157 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.154+0000 7f4910e04640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.157 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.154+0000 7f4910e04640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.157 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:08.154+0000 7f4910e04640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:08.217 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:08.215+0000 7f2ee1a45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:53:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.318+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f16940c7620 con 0x7f16940b1ca0 2023-10-16T12:53:08.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.319+0000 7f1689ffb640 1 -- 172.21.15.60:0/975132453 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f16940c7620 con 0x7f16940b1ca0 2023-10-16T12:53:08.322 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:53:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 msgr2=0x7f169415df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 0x7f169415df20 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f169000c480 tx=0x7f169000c940 comp rx=0 tx=0).stop 2023-10-16T12:53:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 shutdown_connections 2023-10-16T12:53:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16940b2080 0x7f16941e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16940cfc40 0x7f16940b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 --2- 172.21.15.60:0/975132453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940b1ca0 0x7f169415df20 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.322+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 >> 172.21.15.60:0/975132453 conn(0x7f16940d2b90 msgr2=0x7f16940d1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.323+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 shutdown_connections 2023-10-16T12:53:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.323+0000 7f169a1fb640 1 -- 172.21.15.60:0/975132453 wait complete. 2023-10-16T12:53:08.339 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:08.337+0000 7f2ee1a45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:53:08.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:53:08.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.561+0000 7f74bd81c640 1 Processor -- start 2023-10-16T12:53:08.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.561+0000 7f74bd81c640 1 -- start start 2023-10-16T12:53:08.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.563+0000 7f74bd81c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 0x7f74b8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.563+0000 7f74bd81c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b80caec0 con 0x7f74b806a2e0 2023-10-16T12:53:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.563+0000 7f74bd81c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b80cb140 con 0x7f74b80addb0 2023-10-16T12:53:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.563+0000 7f74bd81c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8161a50 con 0x7f74b80caa80 2023-10-16T12:53:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f74b806a2e0 0x7f74b81650d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53078/0 (socket says 172.21.15.60:53078) 2023-10-16T12:53:08.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b77fe640 1 -- 172.21.15.60:0/2267331554 learned_addr learned my addr 172.21.15.60:0/2267331554 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:08.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b67fc640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 0x7f74b8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863740367 0 0) 0x7f74b80caec0 con 0x7f74b806a2e0 2023-10-16T12:53:08.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a4003520 con 0x7f74b806a2e0 2023-10-16T12:53:08.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731875036 0 0) 0x7f74a4003520 con 0x7f74b806a2e0 2023-10-16T12:53:08.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 msgr2=0x7f74b8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:08.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b5ffb640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 0x7f74b8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.564+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 >> v1:172.21.15.144:6789/0 conn(0x7f74b80addb0 legacy=0x7f74b80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b806b030 con 0x7f74b806a2e0 2023-10-16T12:53:08.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f74b0001670 con 0x7f74b806a2e0 2023-10-16T12:53:08.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 >> v1:172.21.15.60:6789/0 conn(0x7f74b806a2e0 legacy=0x7f74b81650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a40039a0 0x7f74a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 0x7f74a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a4004d00 0x7f74a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8161a50 con 0x7f74a40039a0 2023-10-16T12:53:08.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a400ba90 con 0x7f74a4004d00 2023-10-16T12:53:08.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a400bc50 con 0x7f74b80caa80 2023-10-16T12:53:08.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b6ffd640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a40039a0 0x7f74a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b6ffd640 1 -- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 msgr2=0x7f74a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:08.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b6ffd640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80caa80 0x7f74a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b6ffd640 1 -- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a4004d00 msgr2=0x7f74a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:08.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b77fe640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a4004d00 0x7f74a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.565+0000 7f74b6ffd640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a4004d00 0x7f74a4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.566+0000 7f74b6ffd640 1 -- 172.21.15.60:0/2267331554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a4003520 con 0x7f74a40039a0 2023-10-16T12:53:08.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74b6ffd640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a40039a0 0x7f74a4003dc0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f74a8001e20 tx=0x7f74a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99eca20495518c3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:08.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8003b70 con 0x7f74a40039a0 2023-10-16T12:53:08.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8003cd0 con 0x7f74a40039a0 2023-10-16T12:53:08.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74b5ffb640 1 -- 172.21.15.60:0/2267331554 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8005310 con 0x7f74a40039a0 2023-10-16T12:53:08.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74bd81c640 1 -- 172.21.15.60:0/2267331554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a40039a0 msgr2=0x7f74a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74bd81c640 1 --2- 172.21.15.60:0/2267331554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a40039a0 0x7f74a4003dc0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f74a8001e20 tx=0x7f74a40057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.567+0000 7f74b77fe640 1 -- 172.21.15.60:0/2267331554 reap_dead start 2023-10-16T12:53:08.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.568+0000 7f74bd81c640 1 -- 172.21.15.60:0/2267331554 shutdown_connections 2023-10-16T12:53:08.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.568+0000 7f74bd81c640 1 -- 172.21.15.60:0/2267331554 >> 172.21.15.60:0/2267331554 conn(0x7f74b80ce9b0 msgr2=0x7f74b80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:08.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.568+0000 7f74bd81c640 1 -- 172.21.15.60:0/2267331554 shutdown_connections 2023-10-16T12:53:08.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.568+0000 7f74bd81c640 1 -- 172.21.15.60:0/2267331554 wait complete. 2023-10-16T12:53:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.569+0000 7f74bd81c640 1 Processor -- start 2023-10-16T12:53:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.570+0000 7f74bd81c640 1 -- start start 2023-10-16T12:53:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74bd81c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a40039a0 0x7f74b815f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74bd81c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80c3200 0x7f74b80b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74bd81c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 0x7f74b80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74bd81c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80b1020 con 0x7f74b80b0ba0 2023-10-16T12:53:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74bd81c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b815f7b0 con 0x7f74a40039a0 2023-10-16T12:53:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74bd81c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80c6ad0 con 0x7f74b80c3200 2023-10-16T12:53:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 0x7f74b80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 0x7f74b80b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44094/0 (socket says 172.21.15.60:44094) 2023-10-16T12:53:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 -- 172.21.15.60:0/1872444114 learned_addr learned my addr 172.21.15.60:0/1872444114 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 -- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80c3200 msgr2=0x7f74b80b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80c3200 0x7f74b80b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 -- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a40039a0 msgr2=0x7f74b815f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b6ffd640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a40039a0 0x7f74b815f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.571+0000 7f74b77fe640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a40039a0 0x7f74b815f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74b77fe640 1 -- 172.21.15.60:0/1872444114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b80c6db0 con 0x7f74b80b0ba0 2023-10-16T12:53:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74b6ffd640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a40039a0 0x7f74b815f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74b77fe640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 0x7f74b80b3510 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f74b806af00 tx=0x7f74b000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0007230 con 0x7f74b80b0ba0 2023-10-16T12:53:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0007390 con 0x7f74b80b0ba0 2023-10-16T12:53:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b00047e0 con 0x7f74b80b0ba0 2023-10-16T12:53:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b80c7090 con 0x7f74b80b0ba0 2023-10-16T12:53:08.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.572+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b80c7650 con 0x7f74b80b0ba0 2023-10-16T12:53:08.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.573+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b804e160 con 0x7f74b80b0ba0 2023-10-16T12:53:08.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.574+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 63) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f74b0004940 con 0x7f74b80b0ba0 2023-10-16T12:53:08.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.575+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f74b0089170 con 0x7f74b80b0ba0 2023-10-16T12:53:08.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.579+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74b00541d0 con 0x7f74b80b0ba0 2023-10-16T12:53:08.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.774+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f74b80b32f0 con 0x7f74b80b0ba0 2023-10-16T12:53:08.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.775+0000 7f74affff640 1 -- 172.21.15.60:0/1872444114 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7f74b00577f0 con 0x7f74b80b0ba0 2023-10-16T12:53:08.778 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:53:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 msgr2=0x7f74b80b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 0x7f74b80b3510 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f74b806af00 tx=0x7f74b000dc00 comp rx=0 tx=0).stop 2023-10-16T12:53:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 shutdown_connections 2023-10-16T12:53:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74b80c3200 0x7f74b80b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74b80b0ba0 0x7f74b80b3510 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 --2- 172.21.15.60:0/1872444114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74a40039a0 0x7f74b815f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 >> 172.21.15.60:0/1872444114 conn(0x7f74b80ce9b0 msgr2=0x7f74b8163ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:08.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 shutdown_connections 2023-10-16T12:53:08.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:08.778+0000 7f74bd81c640 1 -- 172.21.15.60:0/1872444114 wait complete. 2023-10-16T12:53:08.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:53:09.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.016+0000 7f548987e640 1 Processor -- start 2023-10-16T12:53:09.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.016+0000 7f548987e640 1 -- start start 2023-10-16T12:53:09.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f548987e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 0x7f54840d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f548987e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548415d150 con 0x7f54840cded0 2023-10-16T12:53:09.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f548987e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54840d1180 con 0x7f54840b8480 2023-10-16T12:53:09.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f548987e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54840d1460 con 0x7f548415cd70 2023-10-16T12:53:09.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f548907d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f54840cded0 0x7f54840b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45550/0 (socket says 172.21.15.60:45550) 2023-10-16T12:53:09.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f548907d640 1 -- 172.21.15.60:0/3791120903 learned_addr learned my addr 172.21.15.60:0/3791120903 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:09.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.018+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957657915 0 0) 0x7f548415d150 con 0x7f54840cded0 2023-10-16T12:53:09.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5474003520 con 0x7f54840cded0 2023-10-16T12:53:09.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f548887c640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 0x7f54840d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 499168830 0 0) 0x7f5474003520 con 0x7f54840cded0 2023-10-16T12:53:09.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 msgr2=0x7f54840d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 0x7f54840d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 >> v1:172.21.15.144:6789/0 conn(0x7f54840b8480 legacy=0x7f54840cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54840d1740 con 0x7f54840cded0 2023-10-16T12:53:09.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f547c0019f0 con 0x7f54840cded0 2023-10-16T12:53:09.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 >> v1:172.21.15.60:6789/0 conn(0x7f54840cded0 legacy=0x7f54840b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 0x7f54740039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5474004020 0x7f54740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004d00 0x7f5474005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54840d1460 con 0x7f5474004020 2023-10-16T12:53:09.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547400bbc0 con 0x7f5474004d00 2023-10-16T12:53:09.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547400bd80 con 0x7f548415cd70 2023-10-16T12:53:09.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.019+0000 7f5483fff640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5474004020 0x7f54740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f548907d640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004d00 0x7f5474005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f548887c640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 0x7f54740039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f5483fff640 1 -- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 msgr2=0x7f54740039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f5483fff640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f548415cd70 0x7f54740039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f5483fff640 1 -- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004d00 msgr2=0x7f5474005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f5483fff640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004d00 0x7f5474005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f5483fff640 1 -- 172.21.15.60:0/3791120903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5474003520 con 0x7f5474004020 2023-10-16T12:53:09.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.020+0000 7f548907d640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004d00 0x7f5474005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:09.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.021+0000 7f5483fff640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5474004020 0x7f54740047c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f5470001750 tx=0x7f54740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d76027e876425ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:09.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.021+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54700052d0 con 0x7f5474004020 2023-10-16T12:53:09.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.021+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547002e050 con 0x7f5474004020 2023-10-16T12:53:09.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.021+0000 7f54837fe640 1 -- 172.21.15.60:0/3791120903 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470003830 con 0x7f5474004020 2023-10-16T12:53:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548987e640 1 -- 172.21.15.60:0/3791120903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5474004020 msgr2=0x7f54740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548987e640 1 --2- 172.21.15.60:0/3791120903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5474004020 0x7f54740047c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f5470001750 tx=0x7f54740057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548907d640 1 -- 172.21.15.60:0/3791120903 reap_dead start 2023-10-16T12:53:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548987e640 1 -- 172.21.15.60:0/3791120903 shutdown_connections 2023-10-16T12:53:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548987e640 1 -- 172.21.15.60:0/3791120903 >> 172.21.15.60:0/3791120903 conn(0x7f54840cf570 msgr2=0x7f54840cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548987e640 1 -- 172.21.15.60:0/3791120903 shutdown_connections 2023-10-16T12:53:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.022+0000 7f548987e640 1 -- 172.21.15.60:0/3791120903 wait complete. 2023-10-16T12:53:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.025+0000 7f548987e640 1 Processor -- start 2023-10-16T12:53:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.025+0000 7f548987e640 1 -- start start 2023-10-16T12:53:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548987e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004020 0x7f54841f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548987e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54840cbbd0 0x7f54840cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548987e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 0x7f548414bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548987e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54840cc570 con 0x7f548414b7e0 2023-10-16T12:53:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548987e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548414d640 con 0x7f5474004020 2023-10-16T12:53:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548987e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548414cd30 con 0x7f54840cbbd0 2023-10-16T12:53:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548907d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 0x7f548414bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548907d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 0x7f548414bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39518/0 (socket says 172.21.15.60:39518) 2023-10-16T12:53:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.026+0000 7f548907d640 1 -- 172.21.15.60:0/696593112 learned_addr learned my addr 172.21.15.60:0/696593112 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548887c640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004020 0x7f54841f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548907d640 1 -- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54840cbbd0 msgr2=0x7f54840cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548907d640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54840cbbd0 0x7f54840cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548907d640 1 -- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004020 msgr2=0x7f54841f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548907d640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004020 0x7f54841f1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548907d640 1 -- 172.21.15.60:0/696593112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548414c700 con 0x7f548414b7e0 2023-10-16T12:53:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548907d640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 0x7f548414bc40 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f548408b0b0 tx=0x7f547c00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c00b8d0 con 0x7f548414b7e0 2023-10-16T12:53:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c00ba50 con 0x7f548414b7e0 2023-10-16T12:53:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548414cff0 con 0x7f548414b7e0 2023-10-16T12:53:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.027+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c00be00 con 0x7f548414b7e0 2023-10-16T12:53:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.028+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54841f21d0 con 0x7f548414b7e0 2023-10-16T12:53:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.029+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548414d7d0 con 0x7f548414b7e0 2023-10-16T12:53:09.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.030+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 63) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f547c007110 con 0x7f548414b7e0 2023-10-16T12:53:09.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.030+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f547c089880 con 0x7f548414b7e0 2023-10-16T12:53:09.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.035+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f547c054970 con 0x7f548414b7e0 2023-10-16T12:53:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.090+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 64) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f547c08bd00 con 0x7f548414b7e0 2023-10-16T12:53:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.091+0000 7f54817fa640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f5468070690 0x7f5468072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.092+0000 7f548887c640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f5468070690 0x7f5468072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.097+0000 7f548887c640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f5468070690 0x7f5468072a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5478006fd0 tx=0x7f5478008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:09.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.237+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f548404e9b0 con 0x7f548414b7e0 2023-10-16T12:53:09.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.238+0000 7f54817fa640 1 -- 172.21.15.60:0/696593112 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7f547c074040 con 0x7f548414b7e0 2023-10-16T12:53:09.240 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:53:09.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.240+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f5468070690 msgr2=0x7f5468072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.240+0000 7f548987e640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f5468070690 0x7f5468072a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5478006fd0 tx=0x7f5478008040 comp rx=0 tx=0).stop 2023-10-16T12:53:09.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.240+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 msgr2=0x7f548414bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.240+0000 7f548987e640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 0x7f548414bc40 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f548408b0b0 tx=0x7f547c00b0d0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 shutdown_connections 2023-10-16T12:53:09.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f5468070690 0x7f5468072a80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54840cbbd0 0x7f54840cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f548414b7e0 0x7f548414bc40 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 --2- 172.21.15.60:0/696593112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5474004020 0x7f54841f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 >> 172.21.15.60:0/696593112 conn(0x7f54840cf570 msgr2=0x7f54840b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:09.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 shutdown_connections 2023-10-16T12:53:09.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.241+0000 7f548987e640 1 -- 172.21.15.60:0/696593112 wait complete. 2023-10-16T12:53:09.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.486+0000 7f816e46d640 1 Processor -- start 2023-10-16T12:53:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.487+0000 7f816e46d640 1 -- start start 2023-10-16T12:53:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.488+0000 7f816e46d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f816815c9f0 0x7f81680c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.488+0000 7f816e46d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f816815dde0 0x7f81680ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.488+0000 7f816e46d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8168160a90 con 0x7f81680c9230 2023-10-16T12:53:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.488+0000 7f816e46d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680cdec0 con 0x7f816815dde0 2023-10-16T12:53:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.488+0000 7f816e46d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680cd900 con 0x7f816815c9f0 2023-10-16T12:53:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f816815dde0 0x7f81680ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f816815dde0 0x7f81680ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39522/0 (socket says 172.21.15.60:39522) 2023-10-16T12:53:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 -- 172.21.15.60:0/2118997986 learned_addr learned my addr 172.21.15.60:0/2118997986 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:09.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816d46b640 1 --2- 172.21.15.60:0/2118997986 >> v2:172.21.15.144:3300/0 conn(0x7f816815c9f0 0x7f81680c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 -- 172.21.15.60:0/2118997986 >> v2:172.21.15.144:3300/0 conn(0x7f816815c9f0 msgr2=0x7f81680c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 --2- 172.21.15.60:0/2118997986 >> v2:172.21.15.144:3300/0 conn(0x7f816815c9f0 0x7f81680c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 -- 172.21.15.60:0/2118997986 >> v1:172.21.15.144:6789/0 conn(0x7f81680c9230 legacy=0x7f81680709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.489+0000 7f816dc6c640 1 -- 172.21.15.60:0/2118997986 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680cdbe0 con 0x7f816815dde0 2023-10-16T12:53:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.490+0000 7f816dc6c640 1 --2- 172.21.15.60:0/2118997986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815dde0 0x7f81680ac890 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f81680ad4e0 tx=0x7f816002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed666848071cddc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.491+0000 7f815ffff640 1 -- 172.21.15.60:0/2118997986 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816002ca80 con 0x7f816815dde0 2023-10-16T12:53:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.491+0000 7f815ffff640 1 -- 172.21.15.60:0/2118997986 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816002cbe0 con 0x7f816815dde0 2023-10-16T12:53:09.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.491+0000 7f815ffff640 1 -- 172.21.15.60:0/2118997986 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600343f0 con 0x7f816815dde0 2023-10-16T12:53:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.491+0000 7f816e46d640 1 -- 172.21.15.60:0/2118997986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815dde0 msgr2=0x7f81680ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.491+0000 7f816e46d640 1 --2- 172.21.15.60:0/2118997986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815dde0 0x7f81680ac890 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f81680ad4e0 tx=0x7f816002bee0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.492+0000 7f816e46d640 1 -- 172.21.15.60:0/2118997986 shutdown_connections 2023-10-16T12:53:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.492+0000 7f816e46d640 1 --2- 172.21.15.60:0/2118997986 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815dde0 0x7f81680ac890 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.492+0000 7f816e46d640 1 --2- 172.21.15.60:0/2118997986 >> v2:172.21.15.144:3300/0 conn(0x7f816815c9f0 0x7f81680c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.492+0000 7f816e46d640 1 -- 172.21.15.60:0/2118997986 >> 172.21.15.60:0/2118997986 conn(0x7f81680ce6a0 msgr2=0x7f81680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.492+0000 7f816e46d640 1 -- 172.21.15.60:0/2118997986 shutdown_connections 2023-10-16T12:53:09.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.492+0000 7f816e46d640 1 -- 172.21.15.60:0/2118997986 wait complete. 2023-10-16T12:53:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.493+0000 7f816e46d640 1 Processor -- start 2023-10-16T12:53:09.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.494+0000 7f816e46d640 1 -- start start 2023-10-16T12:53:09.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.494+0000 7f816e46d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81680c9230 0x7f81680c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.494+0000 7f816e46d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 0x7f81680b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816e46d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f816815dde0 0x7f81680ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816e46d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81680c19e0 con 0x7f816815c9f0 2023-10-16T12:53:09.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816e46d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81680c78c0 con 0x7f816815dde0 2023-10-16T12:53:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816e46d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8168154420 con 0x7f81680c9230 2023-10-16T12:53:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 0x7f81680b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 0x7f81680b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39530/0 (socket says 172.21.15.60:39530) 2023-10-16T12:53:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 -- 172.21.15.60:0/1353984069 learned_addr learned my addr 172.21.15.60:0/1353984069 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:09.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816dc6c640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f816815dde0 0x7f81680ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816d46b640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81680c9230 0x7f81680c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 -- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81680c9230 msgr2=0x7f81680c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81680c9230 0x7f81680c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 -- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f816815dde0 msgr2=0x7f81680ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f816815dde0 0x7f81680ae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.495+0000 7f816cc6a640 1 -- 172.21.15.60:0/1353984069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8160009b30 con 0x7f816815c9f0 2023-10-16T12:53:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.496+0000 7f816cc6a640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 0x7f81680b0850 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f815800c430 tx=0x7f815800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.496+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158011790 con 0x7f816815c9f0 2023-10-16T12:53:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.496+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81580118f0 con 0x7f816815c9f0 2023-10-16T12:53:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.496+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158011ce0 con 0x7f816815c9f0 2023-10-16T12:53:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.496+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8168154700 con 0x7f816815c9f0 2023-10-16T12:53:09.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.496+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81680c7550 con 0x7f816815c9f0 2023-10-16T12:53:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.498+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816804e9b0 con 0x7f816815c9f0 2023-10-16T12:53:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.499+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 64) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f815801c460 con 0x7f816815c9f0 2023-10-16T12:53:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.499+0000 7f815dffb640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f814c06acf0 0x7f814c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.500+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f8158089550 con 0x7f816815c9f0 2023-10-16T12:53:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.500+0000 7f816d46b640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f814c06acf0 0x7f814c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.501+0000 7f816d46b640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f814c06acf0 0x7f814c06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8164006fd0 tx=0x7f8164008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.504+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8158054550 con 0x7f816815c9f0 2023-10-16T12:53:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.736+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81680adac0 con 0x7f816815c9f0 2023-10-16T12:53:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.738+0000 7f815dffb640 1 -- 172.21.15.60:0/1353984069 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f8158057b70 con 0x7f816815c9f0 2023-10-16T12:53:09.741 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:09.741 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:52:32.396002+0000","modified":"2023-10-16T12:52:35.483028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7866":{"gid":7866,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2146889084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2146889084},{"type":"v1","addr":"172.21.15.60:6833","nonce":2146889084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:53:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2023-10-16T12:53:09.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.740+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f814c06acf0 msgr2=0x7f814c06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.740+0000 7f816e46d640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f814c06acf0 0x7f814c06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8164006fd0 tx=0x7f8164008040 comp rx=0 tx=0).stop 2023-10-16T12:53:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.740+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 msgr2=0x7f81680b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.740+0000 7f816e46d640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 0x7f81680b0850 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f815800c430 tx=0x7f815800c8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 shutdown_connections 2023-10-16T12:53:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f814c06acf0 0x7f814c06d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f816815dde0 0x7f81680ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816815c9f0 0x7f81680b0850 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 --2- 172.21.15.60:0/1353984069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81680c9230 0x7f81680c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 >> 172.21.15.60:0/1353984069 conn(0x7f81680ce6a0 msgr2=0x7f81680708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 shutdown_connections 2023-10-16T12:53:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.741+0000 7f816e46d640 1 -- 172.21.15.60:0/1353984069 wait complete. 2023-10-16T12:53:09.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.983+0000 7f9ac3ae7640 1 Processor -- start 2023-10-16T12:53:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.984+0000 7f9ac3ae7640 1 -- start start 2023-10-16T12:53:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.985+0000 7f9ac3ae7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9abc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.985+0000 7f9ac3ae7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9abc0ce390 0x7f9abc070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.985+0000 7f9ac3ae7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc15ed60 con 0x7f9abc15ae80 2023-10-16T12:53:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.985+0000 7f9ac3ae7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc06b810 con 0x7f9abc0ce390 2023-10-16T12:53:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.985+0000 7f9ac3ae7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0707e0 con 0x7f9abc15aaa0 2023-10-16T12:53:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.986+0000 7f9ac1082640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9abc15ae80 0x7f9abc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45562/0 (socket says 172.21.15.60:45562) 2023-10-16T12:53:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.986+0000 7f9ac1082640 1 -- 172.21.15.60:0/2592034520 learned_addr learned my addr 172.21.15.60:0/2592034520 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:09.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.986+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9abc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.986+0000 7f9ac2084640 1 --2- 172.21.15.60:0/2592034520 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0ce390 0x7f9abc070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:09.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.986+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512830158 0 0) 0x7f9abc15ed60 con 0x7f9abc15ae80 2023-10-16T12:53:09.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.986+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8003520 con 0x7f9abc15ae80 2023-10-16T12:53:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924082539 0 0) 0x7f9aa8003520 con 0x7f9abc15ae80 2023-10-16T12:53:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 msgr2=0x7f9abc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9abc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0ce390 msgr2=0x7f9abc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 --2- 172.21.15.60:0/2592034520 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0ce390 0x7f9abc070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc15ef90 con 0x7f9abc15ae80 2023-10-16T12:53:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9aac001670 con 0x7f9abc15ae80 2023-10-16T12:53:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 >> v1:172.21.15.60:6789/0 conn(0x7f9abc15ae80 legacy=0x7f9abc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8003970 0x7f9aa8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9aa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc06b810 con 0x7f9aa8003970 2023-10-16T12:53:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0707e0 con 0x7f9aa8004d00 2023-10-16T12:53:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800b980 con 0x7f9abc15aaa0 2023-10-16T12:53:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.987+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8003970 0x7f9aa8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac2084640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1883640 1 -- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 msgr2=0x7f9aa80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1082640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9aa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9aa80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1883640 1 -- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9aa8004d00 msgr2=0x7f9aa8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1883640 1 -- 172.21.15.60:0/2592034520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8003520 con 0x7f9aa8003970 2023-10-16T12:53:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac2084640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.988+0000 7f9ac1082640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9abc15aaa0 0x7f9aa80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.989+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8003970 0x7f9aa8003d90 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009ac0 tx=0x7f9aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da47bad3cd6ef4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.989+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802c6d0 con 0x7f9aa8003970 2023-10-16T12:53:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.989+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab802c830 con 0x7f9aa8003970 2023-10-16T12:53:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.989+0000 7f9ac0881640 1 -- 172.21.15.60:0/2592034520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802cbc0 con 0x7f9aa8003970 2023-10-16T12:53:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2592034520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8003970 msgr2=0x7f9aa8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2592034520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9aa8003970 0x7f9aa8003d90 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009ac0 tx=0x7f9aa80057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac2084640 1 -- 172.21.15.60:0/2592034520 reap_dead start 2023-10-16T12:53:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2592034520 shutdown_connections 2023-10-16T12:53:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2592034520 >> 172.21.15.60:0/2592034520 conn(0x7f9abc0cb410 msgr2=0x7f9abc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2592034520 shutdown_connections 2023-10-16T12:53:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.990+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2592034520 wait complete. 2023-10-16T12:53:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.992+0000 7f9ac3ae7640 1 Processor -- start 2023-10-16T12:53:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.992+0000 7f9ac3ae7640 1 -- start start 2023-10-16T12:53:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac3ae7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9aa8003970 0x7f9abc15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac3ae7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 0x7f9abc158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac3ae7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc157520 0x7f9abc1579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac3ae7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc157ef0 con 0x7f9abc0c4590 2023-10-16T12:53:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac3ae7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc15c360 con 0x7f9abc157520 2023-10-16T12:53:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac3ae7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc15c640 con 0x7f9aa8003970 2023-10-16T12:53:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.993+0000 7f9ac1082640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 0x7f9abc158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 0x7f9abc158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39544/0 (socket says 172.21.15.60:39544) 2023-10-16T12:53:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 -- 172.21.15.60:0/2344228305 learned_addr learned my addr 172.21.15.60:0/2344228305 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9aa8003970 0x7f9abc15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac2084640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc157520 0x7f9abc1579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 -- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9aa8003970 msgr2=0x7f9abc15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9aa8003970 0x7f9abc15bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 -- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc157520 msgr2=0x7f9abc1579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc157520 0x7f9abc1579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 -- 172.21.15.60:0/2344228305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc15c920 con 0x7f9abc0c4590 2023-10-16T12:53:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.994+0000 7f9ac1082640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 0x7f9abc158fa0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00d7f0 tx=0x7f9aac00dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.995+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0074e0 con 0x7f9abc0c4590 2023-10-16T12:53:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.995+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac007640 con 0x7f9abc0c4590 2023-10-16T12:53:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.995+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac004a00 con 0x7f9abc0c4590 2023-10-16T12:53:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.995+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc15cc00 con 0x7f9abc0c4590 2023-10-16T12:53:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.995+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc0c49f0 con 0x7f9abc0c4590 2023-10-16T12:53:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.996+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc15f530 con 0x7f9abc0c4590 2023-10-16T12:53:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.997+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 64) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9aac004b60 con 0x7f9abc0c4590 2023-10-16T12:53:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.998+0000 7f9ab27fc640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f9aa006acc0 0x7f9aa006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.998+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f9aac088fa0 con 0x7f9abc0c4590 2023-10-16T12:53:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:09.999+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f9aa006acc0 0x7f9aa006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.000+0000 7f9ac1883640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f9aa006acc0 0x7f9aa006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8036000 tx=0x7f9ab8030000 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.003+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9abc15f530 con 0x7f9abc0c4590 2023-10-16T12:53:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.204+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9abc15f530 con 0x7f9abc0c4590 2023-10-16T12:53:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.206+0000 7f9ab27fc640 1 -- 172.21.15.60:0/2344228305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17716 (secure 0 0 0) 0x7f9abc15f530 con 0x7f9abc0c4590 2023-10-16T12:53:10.208 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:10.209 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":131,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:08.068321+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/3420535028":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/2503609014":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/1605553329":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3076854692":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/132154623":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/277155730":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3226844042":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/1223431809":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/2573756770":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3005566427":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/945398958":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/1880380841":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6833/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/4085361535":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3420535028":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3615970143":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/2155057237":"2023-10-17T12:52:47.120136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:10.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f9aa006acc0 msgr2=0x7f9aa006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f9aa006acc0 0x7f9aa006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8036000 tx=0x7f9ab8030000 comp rx=0 tx=0).stop 2023-10-16T12:53:10.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 msgr2=0x7f9abc158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 0x7f9abc158fa0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00d7f0 tx=0x7f9aac00dcb0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 shutdown_connections 2023-10-16T12:53:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f9aa006acc0 0x7f9aa006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc157520 0x7f9abc1579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9abc0c4590 0x7f9abc158fa0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.208+0000 7f9ac3ae7640 1 --2- 172.21.15.60:0/2344228305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9aa8003970 0x7f9abc15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.209+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 >> 172.21.15.60:0/2344228305 conn(0x7f9abc0cb410 msgr2=0x7f9abc0c1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.209+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 shutdown_connections 2023-10-16T12:53:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.209+0000 7f9ac3ae7640 1 -- 172.21.15.60:0/2344228305 wait complete. 2023-10-16T12:53:10.231 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:53:10.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.445+0000 7fe64ef16640 1 Processor -- start 2023-10-16T12:53:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.446+0000 7fe64ef16640 1 -- start start 2023-10-16T12:53:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.447+0000 7fe64ef16640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe64806c040 0x7fe64806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.447+0000 7fe64ef16640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64815f4a0 con 0x7fe6480cda80 2023-10-16T12:53:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.447+0000 7fe64ef16640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64806c960 con 0x7fe6480ad970 2023-10-16T12:53:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.447+0000 7fe64ef16640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64815b9d0 con 0x7fe64806c040 2023-10-16T12:53:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.447+0000 7fe64df14640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe64806c040 0x7fe64806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64e715640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe6480cda80 0x7fe64815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45570/0 (socket says 172.21.15.60:45570) 2023-10-16T12:53:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64df14640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe64806c040 0x7fe64806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39548/0 (socket says 172.21.15.60:39548) 2023-10-16T12:53:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64e715640 1 -- 172.21.15.60:0/3385361424 learned_addr learned my addr 172.21.15.60:0/3385361424 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:10.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64df14640 1 -- 172.21.15.60:0/3385361424 >> v1:172.21.15.144:6789/0 conn(0x7fe6480ad970 legacy=0x7fe64815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:10.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64df14640 1 -- 172.21.15.60:0/3385361424 >> v1:172.21.15.60:6789/0 conn(0x7fe6480cda80 legacy=0x7fe64815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64df14640 1 -- 172.21.15.60:0/3385361424 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64815bcb0 con 0x7fe64806c040 2023-10-16T12:53:10.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.448+0000 7fe64cf12640 1 -- 172.21.15.60:0/3385361424 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240963588 0 0) 0x7fe64815f4a0 con 0x7fe6480cda80 2023-10-16T12:53:10.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.449+0000 7fe64df14640 1 --2- 172.21.15.60:0/3385361424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64806c420 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe644009d90 tx=0x7fe64402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719052272dc37101 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64cf12640 1 -- 172.21.15.60:0/3385361424 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64402cc60 con 0x7fe64806c040 2023-10-16T12:53:10.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64cf12640 1 -- 172.21.15.60:0/3385361424 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64402cdc0 con 0x7fe64806c040 2023-10-16T12:53:10.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64cf12640 1 -- 172.21.15.60:0/3385361424 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644034660 con 0x7fe64806c040 2023-10-16T12:53:10.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 -- 172.21.15.60:0/3385361424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 msgr2=0x7fe64806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3385361424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64806c420 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe644009d90 tx=0x7fe64402beb0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 -- 172.21.15.60:0/3385361424 shutdown_connections 2023-10-16T12:53:10.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3385361424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64806c420 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 -- 172.21.15.60:0/3385361424 >> 172.21.15.60:0/3385361424 conn(0x7fe6480ce6a0 msgr2=0x7fe6480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:10.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 -- 172.21.15.60:0/3385361424 shutdown_connections 2023-10-16T12:53:10.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.450+0000 7fe64ef16640 1 -- 172.21.15.60:0/3385361424 wait complete. 2023-10-16T12:53:10.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.452+0000 7fe64ef16640 1 Processor -- start 2023-10-16T12:53:10.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.452+0000 7fe64ef16640 1 -- start start 2023-10-16T12:53:10.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64ef16640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64814fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64ef16640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6480ad970 0x7fe6480b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64ef16640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6480cda80 0x7fe64815e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64ef16640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648150420 con 0x7fe64806c040 2023-10-16T12:53:10.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64ef16640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480b23c0 con 0x7fe6480cda80 2023-10-16T12:53:10.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64ef16640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480b26a0 con 0x7fe6480ad970 2023-10-16T12:53:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64df14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64814fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64df14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64814fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39562/0 (socket says 172.21.15.60:39562) 2023-10-16T12:53:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.453+0000 7fe64df14640 1 -- 172.21.15.60:0/3252790709 learned_addr learned my addr 172.21.15.60:0/3252790709 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64df14640 1 -- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6480ad970 msgr2=0x7fe6480b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64df14640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6480ad970 0x7fe6480b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64df14640 1 -- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6480cda80 msgr2=0x7fe64815e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64df14640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6480cda80 0x7fe64815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64df14640 1 -- 172.21.15.60:0/3252790709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe644009a70 con 0x7fe64806c040 2023-10-16T12:53:10.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64df14640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64814fee0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe644037040 tx=0x7fe644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64403e610 con 0x7fe64806c040 2023-10-16T12:53:10.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64403e770 con 0x7fe64806c040 2023-10-16T12:53:10.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.454+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6480b2980 con 0x7fe64806c040 2023-10-16T12:53:10.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.455+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64403eb20 con 0x7fe64806c040 2023-10-16T12:53:10.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.455+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6480b2f40 con 0x7fe64806c040 2023-10-16T12:53:10.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.456+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64806b330 con 0x7fe64806c040 2023-10-16T12:53:10.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.457+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 64) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe64403b050 con 0x7fe64806c040 2023-10-16T12:53:10.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.458+0000 7fe63e7fc640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe62c06acc0 0x7fe62c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.458+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fe6440ac0b0 con 0x7fe64806c040 2023-10-16T12:53:10.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.458+0000 7fe64d713640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe62c06acc0 0x7fe62c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.460+0000 7fe64d713640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe62c06acc0 0x7fe62c06d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe6480c8c40 tx=0x7fe6380079d0 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.462+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6440771d0 con 0x7fe64806c040 2023-10-16T12:53:10.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.687+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe64804e9b0 con 0x7fe64806c040 2023-10-16T12:53:10.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.688+0000 7fe63e7fc640 1 -- 172.21.15.60:0/3252790709 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fe644039090 con 0x7fe64806c040 2023-10-16T12:53:10.691 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:10.691 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:52:32.396002+0000","modified":"2023-10-16T12:52:35.483028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7866":{"gid":7866,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2146889084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2146889084},{"type":"v1","addr":"172.21.15.60:6833","nonce":2146889084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:53:10.692 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2023-10-16T12:53:10.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe62c06acc0 msgr2=0x7fe62c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe62c06acc0 0x7fe62c06d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe6480c8c40 tx=0x7fe6380079d0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 msgr2=0x7fe64814fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64814fee0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe644037040 tx=0x7fe644004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:10.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 shutdown_connections 2023-10-16T12:53:10.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe62c06acc0 0x7fe62c06d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6480ad970 0x7fe6480b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6480cda80 0x7fe64815e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 --2- 172.21.15.60:0/3252790709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe64806c040 0x7fe64814fee0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 >> 172.21.15.60:0/3252790709 conn(0x7fe6480ce6a0 msgr2=0x7fe64815ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:10.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 shutdown_connections 2023-10-16T12:53:10.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.691+0000 7fe64ef16640 1 -- 172.21.15.60:0/3252790709 wait complete. 2023-10-16T12:53:10.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:10.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.929+0000 7f32d4b67640 1 Processor -- start 2023-10-16T12:53:10.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.929+0000 7f32d4b67640 1 -- start start 2023-10-16T12:53:10.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.930+0000 7f32d4b67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0070b60 0x7f32d015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.930+0000 7f32d4b67640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 0x7f32d006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.930+0000 7f32d4b67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d015aef0 con 0x7f32d0159950 2023-10-16T12:53:10.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.930+0000 7f32d4b67640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d015e490 con 0x7f32d006c100 2023-10-16T12:53:10.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.930+0000 7f32d4b67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d015d910 con 0x7f32d0070b60 2023-10-16T12:53:10.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.931+0000 7f32cffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 0x7f32d006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.931+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0070b60 0x7f32d015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.931+0000 7f32cffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 0x7f32d006c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:36904/0 (socket says 172.21.15.60:36904) 2023-10-16T12:53:10.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.931+0000 7f32cffff640 1 -- 172.21.15.60:0/2186262941 learned_addr learned my addr 172.21.15.60:0/2186262941 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:10.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.931+0000 7f32cf7fe640 1 --1- 172.21.15.60:0/2186262941 >> v1:172.21.15.144:6789/0 conn(0x7f32d0159950 0x7f32d0070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:60460/0 (socket says 172.21.15.60:60460) 2023-10-16T12:53:10.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.932+0000 7f32c7fff640 1 -- 172.21.15.60:0/2186262941 >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 msgr2=0x7f32d006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:10.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.932+0000 7f32c7fff640 1 --2- 172.21.15.60:0/2186262941 >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 0x7f32d006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.932+0000 7f32c7fff640 1 -- 172.21.15.60:0/2186262941 >> v1:172.21.15.144:6789/0 conn(0x7f32d0159950 legacy=0x7f32d0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.933+0000 7f32c7fff640 1 -- 172.21.15.60:0/2186262941 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d015dbf0 con 0x7f32d0070b60 2023-10-16T12:53:10.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.932+0000 7f32ceffd640 1 -- 172.21.15.60:0/2186262941 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899844389 0 0) 0x7f32c0001110 con 0x7f32d0159950 2023-10-16T12:53:10.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.933+0000 7f32cffff640 1 --2- 172.21.15.60:0/2186262941 >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 0x7f32d006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:53:10.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.934+0000 7f32c7fff640 1 --2- 172.21.15.60:0/2186262941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0070b60 0x7f32d015a9b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32b80097a0 tx=0x7f32b802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df44752a91c254f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32ceffd640 1 -- 172.21.15.60:0/2186262941 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8031780 con 0x7f32d0070b60 2023-10-16T12:53:10.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32ceffd640 1 -- 172.21.15.60:0/2186262941 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b80318e0 con 0x7f32d0070b60 2023-10-16T12:53:10.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32ceffd640 1 -- 172.21.15.60:0/2186262941 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8031c70 con 0x7f32d0070b60 2023-10-16T12:53:10.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32d4b67640 1 -- 172.21.15.60:0/2186262941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0070b60 msgr2=0x7f32d015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32d4b67640 1 --2- 172.21.15.60:0/2186262941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0070b60 0x7f32d015a9b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32b80097a0 tx=0x7f32b802b820 comp rx=0 tx=0).stop 2023-10-16T12:53:10.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32d4b67640 1 -- 172.21.15.60:0/2186262941 shutdown_connections 2023-10-16T12:53:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32d4b67640 1 --2- 172.21.15.60:0/2186262941 >> v2:172.21.15.144:3300/0 conn(0x7f32d006c100 0x7f32d006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32d4b67640 1 --2- 172.21.15.60:0/2186262941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0070b60 0x7f32d015a9b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.935+0000 7f32d4b67640 1 -- 172.21.15.60:0/2186262941 >> 172.21.15.60:0/2186262941 conn(0x7f32d00ce6a0 msgr2=0x7f32d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.936+0000 7f32d4b67640 1 -- 172.21.15.60:0/2186262941 shutdown_connections 2023-10-16T12:53:10.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.936+0000 7f32d4b67640 1 -- 172.21.15.60:0/2186262941 wait complete. 2023-10-16T12:53:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.937+0000 7f32d4b67640 1 Processor -- start 2023-10-16T12:53:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.937+0000 7f32d4b67640 1 -- start start 2023-10-16T12:53:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.938+0000 7f32d4b67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 0x7f32d014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.938+0000 7f32d4b67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32d0070b60 0x7f32d01543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.938+0000 7f32d4b67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0159950 0x7f32d0154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.938+0000 7f32d4b67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c21b0 con 0x7f32d006c100 2023-10-16T12:53:10.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.938+0000 7f32d4b67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d014f3a0 con 0x7f32d0070b60 2023-10-16T12:53:10.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.938+0000 7f32d4b67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d0152ea0 con 0x7f32d0159950 2023-10-16T12:53:10.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 0x7f32d014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 0x7f32d014ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39572/0 (socket says 172.21.15.60:39572) 2023-10-16T12:53:10.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 -- 172.21.15.60:0/867012142 learned_addr learned my addr 172.21.15.60:0/867012142 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:10.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 -- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0159950 msgr2=0x7f32d0154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cffff640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0159950 0x7f32d0154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0159950 0x7f32d0154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 -- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32d0070b60 msgr2=0x7f32d01543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:10.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32c7fff640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32d0070b60 0x7f32d01543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32d0070b60 0x7f32d01543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 -- 172.21.15.60:0/867012142 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c800bd40 con 0x7f32d006c100 2023-10-16T12:53:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.939+0000 7f32cf7fe640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 0x7f32d014ee60 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f32c000c430 tx=0x7f32c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.940+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c00074e0 con 0x7f32d006c100 2023-10-16T12:53:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.940+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c0007660 con 0x7f32d006c100 2023-10-16T12:53:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.940+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b8009400 con 0x7f32d006c100 2023-10-16T12:53:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.940+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0004a20 con 0x7f32d006c100 2023-10-16T12:53:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.940+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d0153460 con 0x7f32d006c100 2023-10-16T12:53:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.941+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d004e9b0 con 0x7f32d006c100 2023-10-16T12:53:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.942+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 65) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f32c0004b80 con 0x7f32d006c100 2023-10-16T12:53:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.943+0000 7f32ccff9640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f32b006af00 0x7f32b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.943+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f32c0089760 con 0x7f32d006c100 2023-10-16T12:53:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.943+0000 7f32c7fff640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f32b006af00 0x7f32b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.945+0000 7f32c7fff640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f32b006af00 0x7f32b006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32b8036000 tx=0x7f32b803d040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:10.947+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32c0054400 con 0x7f32d006c100 2023-10-16T12:53:11.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.187+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d00b4620 con 0x7f32d006c100 2023-10-16T12:53:11.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.188+0000 7f32ccff9640 1 -- 172.21.15.60:0/867012142 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f32c0057a20 con 0x7f32d006c100 2023-10-16T12:53:11.191 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:11.191 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:52:32.396002+0000","modified":"2023-10-16T12:52:35.483028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7866":{"gid":7866,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2146889084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2146889084},{"type":"v1","addr":"172.21.15.60:6833","nonce":2146889084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:53:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2023-10-16T12:53:11.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.190+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f32b006af00 msgr2=0x7f32b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.190+0000 7f32d4b67640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f32b006af00 0x7f32b006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32b8036000 tx=0x7f32b803d040 comp rx=0 tx=0).stop 2023-10-16T12:53:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 msgr2=0x7f32d014ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 0x7f32d014ee60 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f32c000c430 tx=0x7f32c000c8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 shutdown_connections 2023-10-16T12:53:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f32b006af00 0x7f32b006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f32d0159950 0x7f32d0154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32d0070b60 0x7f32d01543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 --2- 172.21.15.60:0/867012142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32d006c100 0x7f32d014ee60 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 >> 172.21.15.60:0/867012142 conn(0x7f32d00ce6a0 msgr2=0x7f32d0158c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 shutdown_connections 2023-10-16T12:53:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.191+0000 7f32d4b67640 1 -- 172.21.15.60:0/867012142 wait complete. 2023-10-16T12:53:11.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:11.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.438+0000 7fc2c5867640 1 Processor -- start 2023-10-16T12:53:11.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.438+0000 7fc2c5867640 1 -- start start 2023-10-16T12:53:11.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2c5867640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2c5867640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2c00cda80 0x7fc2c015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2c5867640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c006c960 con 0x7fc2c00ad970 2023-10-16T12:53:11.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2c5867640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c015b9d0 con 0x7fc2c00cda80 2023-10-16T12:53:11.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2c5867640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c015bcb0 con 0x7fc2c006c040 2023-10-16T12:53:11.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2c5066640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2c00cda80 0x7fc2c015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2bffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc2c00ad970 0x7fc2c015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45574/0 (socket says 172.21.15.60:45574) 2023-10-16T12:53:11.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.440+0000 7fc2bffff640 1 -- 172.21.15.60:0/2700815801 learned_addr learned my addr 172.21.15.60:0/2700815801 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:11.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.441+0000 7fc2c4865640 1 --2- 172.21.15.60:0/2700815801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.441+0000 7fc2c5066640 1 -- 172.21.15.60:0/2700815801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 msgr2=0x7fc2c006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:11.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.441+0000 7fc2c5066640 1 --2- 172.21.15.60:0/2700815801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.441+0000 7fc2c5066640 1 -- 172.21.15.60:0/2700815801 >> v1:172.21.15.60:6789/0 conn(0x7fc2c00ad970 legacy=0x7fc2c015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.441+0000 7fc2c5066640 1 -- 172.21.15.60:0/2700815801 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c015bf90 con 0x7fc2c00cda80 2023-10-16T12:53:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.441+0000 7fc2bf7fe640 1 -- 172.21.15.60:0/2700815801 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426371519 0 0) 0x7fc2c006c960 con 0x7fc2c00ad970 2023-10-16T12:53:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.442+0000 7fc2c5066640 1 --2- 172.21.15.60:0/2700815801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c015ed90 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009d90 tx=0x7fc2b802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819ee3f94698a17a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.442+0000 7fc2bf7fe640 1 -- 172.21.15.60:0/2700815801 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b802cc60 con 0x7fc2c00cda80 2023-10-16T12:53:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.442+0000 7fc2bf7fe640 1 -- 172.21.15.60:0/2700815801 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b802cdc0 con 0x7fc2c00cda80 2023-10-16T12:53:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.442+0000 7fc2bf7fe640 1 -- 172.21.15.60:0/2700815801 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8034660 con 0x7fc2c00cda80 2023-10-16T12:53:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 -- 172.21.15.60:0/2700815801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 msgr2=0x7fc2c015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2700815801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c015ed90 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009d90 tx=0x7fc2b802beb0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 -- 172.21.15.60:0/2700815801 shutdown_connections 2023-10-16T12:53:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2700815801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c015ed90 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2700815801 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 -- 172.21.15.60:0/2700815801 >> 172.21.15.60:0/2700815801 conn(0x7fc2c00ce6a0 msgr2=0x7fc2c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 -- 172.21.15.60:0/2700815801 shutdown_connections 2023-10-16T12:53:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.443+0000 7fc2c5867640 1 -- 172.21.15.60:0/2700815801 wait complete. 2023-10-16T12:53:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.445+0000 7fc2c5867640 1 Processor -- start 2023-10-16T12:53:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.445+0000 7fc2c5867640 1 -- start start 2023-10-16T12:53:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.446+0000 7fc2c5867640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c014ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.446+0000 7fc2c5867640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c00ad970 0x7fc2c00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.446+0000 7fc2c5867640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.446+0000 7fc2c5867640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c0158690 con 0x7fc2c00cda80 2023-10-16T12:53:11.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.446+0000 7fc2c5867640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00c8eb0 con 0x7fc2c00ad970 2023-10-16T12:53:11.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.446+0000 7fc2c5867640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c00c9190 con 0x7fc2c006c040 2023-10-16T12:53:11.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39580/0 (socket says 172.21.15.60:39580) 2023-10-16T12:53:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 -- 172.21.15.60:0/2035789863 learned_addr learned my addr 172.21.15.60:0/2035789863 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c4865640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c014ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 -- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 msgr2=0x7fc2c014ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c014ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2bffff640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c00ad970 0x7fc2c00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 -- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c00ad970 msgr2=0x7fc2c00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c00ad970 0x7fc2c00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.447+0000 7fc2c5066640 1 -- 172.21.15.60:0/2035789863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b8009a70 con 0x7fc2c00cda80 2023-10-16T12:53:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.448+0000 7fc2c5066640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c00acdd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009d90 tx=0x7fc2b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.448+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b803e610 con 0x7fc2c00cda80 2023-10-16T12:53:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.448+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b803e790 con 0x7fc2c00cda80 2023-10-16T12:53:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.448+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b803eb40 con 0x7fc2c00cda80 2023-10-16T12:53:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.448+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c00c9470 con 0x7fc2c00cda80 2023-10-16T12:53:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.448+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c01579d0 con 0x7fc2c00cda80 2023-10-16T12:53:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.449+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c006b330 con 0x7fc2c00cda80 2023-10-16T12:53:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.450+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 65) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc2b803b050 con 0x7fc2c00cda80 2023-10-16T12:53:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.451+0000 7fc2bd7fa640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc2a406af50 0x7fc2a406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.451+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7fc2b80ac370 con 0x7fc2c00cda80 2023-10-16T12:53:11.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.452+0000 7fc2c4865640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc2a406af50 0x7fc2a406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.453+0000 7fc2c4865640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc2a406af50 0x7fc2a406d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4006fd0 tx=0x7fc2b4008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:11.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.455+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc2b80771b0 con 0x7fc2c00cda80 2023-10-16T12:53:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.654+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2c004e9b0 con 0x7fc2c00cda80 2023-10-16T12:53:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.656+0000 7fc2bd7fa640 1 -- 172.21.15.60:0/2035789863 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17716 (secure 0 0 0) 0x7fc2b807a7d0 con 0x7fc2c00cda80 2023-10-16T12:53:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:11.659 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":131,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:08.068321+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:52:30.363329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-16T12:52:31.424370+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/3420535028":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/2503609014":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/1605553329":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3076854692":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/132154623":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/277155730":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3226844042":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/1223431809":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/2573756770":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3005566427":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/945398958":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/1880380841":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6833/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/4085361535":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3420535028":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3615970143":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/2155057237":"2023-10-17T12:52:47.120136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc2a406af50 msgr2=0x7fc2a406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc2a406af50 0x7fc2a406d410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4006fd0 tx=0x7fc2b4008040 comp rx=0 tx=0).stop 2023-10-16T12:53:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 msgr2=0x7fc2c00acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c00acdd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009d90 tx=0x7fc2b8004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 shutdown_connections 2023-10-16T12:53:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc2a406af50 0x7fc2a406d410 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00cda80 0x7fc2c00acdd0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c00ad970 0x7fc2c00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 --2- 172.21.15.60:0/2035789863 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c006c040 0x7fc2c014ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.659+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 >> 172.21.15.60:0/2035789863 conn(0x7fc2c00ce6a0 msgr2=0x7fc2c015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:11.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.660+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 shutdown_connections 2023-10-16T12:53:11.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.660+0000 7fc2c5867640 1 -- 172.21.15.60:0/2035789863 wait complete. 2023-10-16T12:53:11.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:53:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.908+0000 7f1ccd34e640 1 Processor -- start 2023-10-16T12:53:11.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.908+0000 7f1ccd34e640 1 -- start start 2023-10-16T12:53:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1ccd34e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cc8159950 0x7f1cc815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1ccd34e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 0x7f1cc80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1ccd34e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc8070df0 con 0x7f1cc806c100 2023-10-16T12:53:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1ccd34e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc815e490 con 0x7f1cc8159950 2023-10-16T12:53:11.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1ccd34e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc815aef0 con 0x7f1cc8070450 2023-10-16T12:53:11.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1cccb4d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1cc806c100 0x7f1cc806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45584/0 (socket says 172.21.15.60:45584) 2023-10-16T12:53:11.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.910+0000 7f1cccb4d640 1 -- 172.21.15.60:0/804726519 learned_addr learned my addr 172.21.15.60:0/804726519 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc7fff640 1 --2- 172.21.15.60:0/804726519 >> v2:172.21.15.144:3300/0 conn(0x7f1cc8159950 0x7f1cc815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cbf7fe640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 0x7f1cc80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927222124 0 0) 0x7f1cc8070df0 con 0x7f1cc806c100 2023-10-16T12:53:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb4005730 con 0x7f1cc806c100 2023-10-16T12:53:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091093953 0 0) 0x7f1cb4005730 con 0x7f1cc806c100 2023-10-16T12:53:11.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 msgr2=0x7f1cc80708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:11.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 0x7f1cc80708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 >> v2:172.21.15.144:3300/0 conn(0x7f1cc8159950 msgr2=0x7f1cc815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:11.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.911+0000 7f1cc77fe640 1 --2- 172.21.15.60:0/804726519 >> v2:172.21.15.144:3300/0 conn(0x7f1cc8159950 0x7f1cc815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc806cc80 con 0x7f1cc806c100 2023-10-16T12:53:11.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1cc0001670 con 0x7f1cc806c100 2023-10-16T12:53:11.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 >> v1:172.21.15.60:6789/0 conn(0x7f1cc806c100 legacy=0x7f1cc806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb4005b80 0x7f1cb4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 0x7f1cb40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cb4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc815e490 con 0x7f1cb4006f10 2023-10-16T12:53:11.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc815aef0 con 0x7f1cb4005b80 2023-10-16T12:53:11.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb400dc10 con 0x7f1cc8070450 2023-10-16T12:53:11.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cccb4d640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cb4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cc7fff640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb4005b80 0x7f1cb4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.912+0000 7f1cbf7fe640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 0x7f1cb40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.913+0000 7f1cccb4d640 1 -- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 msgr2=0x7f1cb40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:11.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.913+0000 7f1cccb4d640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc8070450 0x7f1cb40069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.913+0000 7f1cccb4d640 1 -- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb4005b80 msgr2=0x7f1cb4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:11.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.913+0000 7f1cccb4d640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb4005b80 0x7f1cb4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.913+0000 7f1cccb4d640 1 -- 172.21.15.60:0/804726519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb4005730 con 0x7f1cb4006f10 2023-10-16T12:53:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.914+0000 7f1cccb4d640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cb4007390 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000c110 tx=0x7f1cb4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9bfcaea36818ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.914+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0005ad0 con 0x7f1cb4006f10 2023-10-16T12:53:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.914+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc0005c50 con 0x7f1cb4006f10 2023-10-16T12:53:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.914+0000 7f1cc77fe640 1 -- 172.21.15.60:0/804726519 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc002c830 con 0x7f1cb4006f10 2023-10-16T12:53:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1ccd34e640 1 -- 172.21.15.60:0/804726519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 msgr2=0x7f1cb4007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/804726519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cb4007390 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000c110 tx=0x7f1cb4007a00 comp rx=0 tx=0).stop 2023-10-16T12:53:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1cccb4d640 1 -- 172.21.15.60:0/804726519 reap_dead start 2023-10-16T12:53:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1ccd34e640 1 -- 172.21.15.60:0/804726519 shutdown_connections 2023-10-16T12:53:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1ccd34e640 1 -- 172.21.15.60:0/804726519 >> 172.21.15.60:0/804726519 conn(0x7f1cc80ce6a0 msgr2=0x7f1cc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1ccd34e640 1 -- 172.21.15.60:0/804726519 shutdown_connections 2023-10-16T12:53:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.915+0000 7f1ccd34e640 1 -- 172.21.15.60:0/804726519 wait complete. 2023-10-16T12:53:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.917+0000 7f1ccd34e640 1 Processor -- start 2023-10-16T12:53:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.917+0000 7f1ccd34e640 1 -- start start 2023-10-16T12:53:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1ccd34e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cc8145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1ccd34e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc8146500 0x7f1cc80c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1ccd34e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc80c9840 0x7f1cc81485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1ccd34e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c9cc0 con 0x7f1cb4006f10 2023-10-16T12:53:11.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1ccd34e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c7770 con 0x7f1cc8146500 2023-10-16T12:53:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1ccd34e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8149370 con 0x7f1cc80c9840 2023-10-16T12:53:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1cc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cc8145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1cc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cc8145dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39592/0 (socket says 172.21.15.60:39592) 2023-10-16T12:53:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.918+0000 7f1cc7fff640 1 -- 172.21.15.60:0/240151615 learned_addr learned my addr 172.21.15.60:0/240151615 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cbf7fe640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc8146500 0x7f1cc80c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cccb4d640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc80c9840 0x7f1cc81485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cc7fff640 1 -- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc80c9840 msgr2=0x7f1cc81485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cc7fff640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc80c9840 0x7f1cc81485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cc7fff640 1 -- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc8146500 msgr2=0x7f1cc80c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cc7fff640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc8146500 0x7f1cc80c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cc7fff640 1 -- 172.21.15.60:0/240151615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80c7120 con 0x7f1cb4006f10 2023-10-16T12:53:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cbf7fe640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc8146500 0x7f1cc80c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.919+0000 7f1cc7fff640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cc8145dd0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8007ea0 tx=0x7f1cb8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.920+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800e030 con 0x7f1cb4006f10 2023-10-16T12:53:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.920+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8016470 con 0x7f1cb4006f10 2023-10-16T12:53:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.920+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8016800 con 0x7f1cb4006f10 2023-10-16T12:53:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.920+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc80c7400 con 0x7f1cb4006f10 2023-10-16T12:53:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.920+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc8148af0 con 0x7f1cb4006f10 2023-10-16T12:53:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.921+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc804e9b0 con 0x7f1cb4006f10 2023-10-16T12:53:11.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.922+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 65) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1cb8016960 con 0x7f1cb4006f10 2023-10-16T12:53:11.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.923+0000 7f1cc57fa640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1ca806af00 0x7f1ca806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:11.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.924+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6511+0+0 (secure 0 0 0) 0x7f1cb800f070 con 0x7f1cb4006f10 2023-10-16T12:53:11.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.924+0000 7f1cbf7fe640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1ca806af00 0x7f1ca806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:11.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.925+0000 7f1cbf7fe640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1ca806af00 0x7f1ca806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80adce0 tx=0x7f1cb0008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:11.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:11.928+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cb80556e0 con 0x7f1cb4006f10 2023-10-16T12:53:12.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.126+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1cb8055ab0 con 0x7f1cb4006f10 2023-10-16T12:53:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.177+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1cc80b4620 con 0x7f1cb4006f10 2023-10-16T12:53:12.181 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:53:12.179+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:53:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.182+0000 7f1cc57fa640 1 -- 172.21.15.60:0/240151615 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1cb8087290 con 0x7f1cb4006f10 2023-10-16T12:53:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:53:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.185+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1ca806af00 msgr2=0x7f1ca806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.185+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1ca806af00 0x7f1ca806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80adce0 tx=0x7f1cb0008040 comp rx=0 tx=0).stop 2023-10-16T12:53:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.185+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 msgr2=0x7f1cc8145dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.185+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cc8145dd0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8007ea0 tx=0x7f1cb8007f90 comp rx=0 tx=0).stop 2023-10-16T12:53:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.186+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 shutdown_connections 2023-10-16T12:53:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.186+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1ca806af00 0x7f1ca806d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.186+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cc80c9840 0x7f1cc81485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.186+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc8146500 0x7f1cc80c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.186+0000 7f1ccd34e640 1 --2- 172.21.15.60:0/240151615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cb4006f10 0x7f1cc8145dd0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.186+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 >> 172.21.15.60:0/240151615 conn(0x7f1cc80ce6a0 msgr2=0x7f1cc80cdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.187+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 shutdown_connections 2023-10-16T12:53:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.187+0000 7f1ccd34e640 1 -- 172.21.15.60:0/240151615 wait complete. 2023-10-16T12:53:12.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:53:12.222 INFO:tasks.ceph.mds.c.smithi060.stderr: -6397> 2023-10-16T12:52:37.878+0000 7fcf601bf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:53:12.289 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:53:12.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.413+0000 7f26b8e83640 1 Processor -- start 2023-10-16T12:53:12.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.413+0000 7f26b8e83640 1 -- start start 2023-10-16T12:53:12.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b8e83640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26b4159e90 0x7f26b40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b8e83640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b406c140 0x7f26b406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b8e83640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4070360 con 0x7f26b4159ab0 2023-10-16T12:53:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b8e83640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b406cb10 con 0x7f26b4159e90 2023-10-16T12:53:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b8e83640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415d890 con 0x7f26b406c140 2023-10-16T12:53:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26abfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26b4159e90 0x7f26b40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f26b4159ab0 0x7f26b406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45592/0 (socket says 172.21.15.60:45592) 2023-10-16T12:53:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.415+0000 7f26b259c640 1 -- 172.21.15.60:0/1403902720 learned_addr learned my addr 172.21.15.60:0/1403902720 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.416+0000 7f26b2d9d640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b406c140 0x7f26b406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.416+0000 7f26abfff640 1 -- 172.21.15.60:0/1403902720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b406c140 msgr2=0x7f26b406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.416+0000 7f26abfff640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b406c140 0x7f26b406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.416+0000 7f26abfff640 1 -- 172.21.15.60:0/1403902720 >> v1:172.21.15.60:6789/0 conn(0x7f26b4159ab0 legacy=0x7f26b406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.416+0000 7f26abfff640 1 -- 172.21.15.60:0/1403902720 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40cd1f0 con 0x7f26b4159e90 2023-10-16T12:53:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.416+0000 7f26b2d9d640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b406c140 0x7f26b406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.417+0000 7f26abfff640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b4159e90 0x7f26b40ce370 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f269c0097a0 tx=0x7f269c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277258f86a958d18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.417+0000 7f26b1d9b640 1 -- 172.21.15.60:0/1403902720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c0316f0 con 0x7f26b4159e90 2023-10-16T12:53:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b1d9b640 1 -- 172.21.15.60:0/1403902720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c031850 con 0x7f26b4159e90 2023-10-16T12:53:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b1d9b640 1 -- 172.21.15.60:0/1403902720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c031be0 con 0x7f26b4159e90 2023-10-16T12:53:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 -- 172.21.15.60:0/1403902720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b4159e90 msgr2=0x7f26b40ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b4159e90 0x7f26b40ce370 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f269c0097a0 tx=0x7f269c02b790 comp rx=0 tx=0).stop 2023-10-16T12:53:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 -- 172.21.15.60:0/1403902720 shutdown_connections 2023-10-16T12:53:12.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b406c140 0x7f26b406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 --2- 172.21.15.60:0/1403902720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b4159e90 0x7f26b40ce370 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 -- 172.21.15.60:0/1403902720 >> 172.21.15.60:0/1403902720 conn(0x7f26b40ce9b0 msgr2=0x7f26b40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 -- 172.21.15.60:0/1403902720 shutdown_connections 2023-10-16T12:53:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.418+0000 7f26b8e83640 1 -- 172.21.15.60:0/1403902720 wait complete. 2023-10-16T12:53:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.420+0000 7f26b8e83640 1 Processor -- start 2023-10-16T12:53:12.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.420+0000 7f26b8e83640 1 -- start start 2023-10-16T12:53:12.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.421+0000 7f26b8e83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 0x7f26b4158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:12.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.421+0000 7f26b8e83640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4159ab0 0x7f26b40c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:12.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.421+0000 7f26b8e83640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4159e90 0x7f26b415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:12.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.421+0000 7f26b8e83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4158740 con 0x7f26b406c140 2023-10-16T12:53:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.421+0000 7f26b8e83640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40c7490 con 0x7f26b4159ab0 2023-10-16T12:53:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.421+0000 7f26b8e83640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40c7770 con 0x7f26b4159e90 2023-10-16T12:53:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 0x7f26b4158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 0x7f26b4158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39634/0 (socket says 172.21.15.60:39634) 2023-10-16T12:53:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 -- 172.21.15.60:0/3360538914 learned_addr learned my addr 172.21.15.60:0/3360538914 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b2d9d640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4159e90 0x7f26b415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26abfff640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4159ab0 0x7f26b40c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 -- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4159e90 msgr2=0x7f26b415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4159e90 0x7f26b415b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 -- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4159ab0 msgr2=0x7f26b40c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:12.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4159ab0 0x7f26b40c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:12.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 -- 172.21.15.60:0/3360538914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c009400 con 0x7f26b406c140 2023-10-16T12:53:12.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b2d9d640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4159e90 0x7f26b415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:12.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.422+0000 7f26b259c640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 0x7f26b4158200 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c480 tx=0x7f26a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.423+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4005ce0 con 0x7f26b406c140 2023-10-16T12:53:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.423+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4005e40 con 0x7f26b406c140 2023-10-16T12:53:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.423+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a40072f0 con 0x7f26b406c140 2023-10-16T12:53:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.423+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b40c7a50 con 0x7f26b406c140 2023-10-16T12:53:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.423+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b40c8010 con 0x7f26b406c140 2023-10-16T12:53:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.424+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b404e160 con 0x7f26b406c140 2023-10-16T12:53:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.426+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f26a4007510 con 0x7f26b406c140 2023-10-16T12:53:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.426+0000 7f26aaffd640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f269406af90 0x7f269406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.427+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6597+0+0 (secure 0 0 0) 0x7f26a4088fe0 con 0x7f26b406c140 2023-10-16T12:53:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.427+0000 7f26abfff640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f269406af90 0x7f269406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.429+0000 7f26abfff640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f269406af90 0x7f269406d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f269c036000 tx=0x7f269c008030 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.431+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26a4053d00 con 0x7f26b406c140 2023-10-16T12:53:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:12.668+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f26b40ccb60 con 0x7f26b406c140 2023-10-16T12:53:13.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.186+0000 7f26aaffd640 1 -- 172.21.15.60:0/3360538914 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f26a4057320 con 0x7f26b406c140 2023-10-16T12:53:13.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f269406af90 msgr2=0x7f269406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f269406af90 0x7f269406d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f269c036000 tx=0x7f269c008030 comp rx=0 tx=0).stop 2023-10-16T12:53:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 msgr2=0x7f26b4158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 0x7f26b4158200 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c480 tx=0x7f26a400c940 comp rx=0 tx=0).stop 2023-10-16T12:53:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 shutdown_connections 2023-10-16T12:53:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f269406af90 0x7f269406d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26b4159e90 0x7f26b415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26b4159ab0 0x7f26b40c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 --2- 172.21.15.60:0/3360538914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26b406c140 0x7f26b4158200 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 >> 172.21.15.60:0/3360538914 conn(0x7f26b40ce9b0 msgr2=0x7f26b40b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 shutdown_connections 2023-10-16T12:53:13.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.190+0000 7f26b8e83640 1 -- 172.21.15.60:0/3360538914 wait complete. 2023-10-16T12:53:13.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:53:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.497+0000 7fe3c30c2640 1 Processor -- start 2023-10-16T12:53:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.497+0000 7fe3c30c2640 1 -- start start 2023-10-16T12:53:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.498+0000 7fe3c30c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3bc15af90 0x7fe3bc0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.498+0000 7fe3c30c2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe3bc0cd320 0x7fe3bc0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.498+0000 7fe3c30c2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc0cb890 con 0x7fe3bc0cf3d0 2023-10-16T12:53:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c30c2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cef80 con 0x7fe3bc15af90 2023-10-16T12:53:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c30c2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cbac0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3bc15af90 0x7fe3bc0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3bc15af90 0x7fe3bc0ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39648/0 (socket says 172.21.15.60:39648) 2023-10-16T12:53:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/619177196 learned_addr learned my addr 172.21.15.60:0/619177196 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:13.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3bbfff640 1 --2- 172.21.15.60:0/619177196 >> v2:172.21.15.144:3300/0 conn(0x7fe3bc0cd320 0x7fe3bc0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:13.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/619177196 >> v2:172.21.15.144:3300/0 conn(0x7fe3bc0cd320 msgr2=0x7fe3bc0cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 --2- 172.21.15.60:0/619177196 >> v2:172.21.15.144:3300/0 conn(0x7fe3bc0cd320 0x7fe3bc0cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/619177196 >> v1:172.21.15.144:6789/0 conn(0x7fe3bc0cf3d0 legacy=0x7fe3bc0afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/619177196 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0cdcc0 con 0x7fe3bc15af90 2023-10-16T12:53:13.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.499+0000 7fe3bbfff640 1 --2- 172.21.15.60:0/619177196 >> v2:172.21.15.144:3300/0 conn(0x7fe3bc0cd320 0x7fe3bc0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.500+0000 7fe3c0e5e640 1 --2- 172.21.15.60:0/619177196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc15af90 0x7fe3bc0ccb60 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc070ea0 tx=0x7fe3ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6ec8ef14de00b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.501+0000 7fe3bb7fe640 1 -- 172.21.15.60:0/619177196 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac031870 con 0x7fe3bc15af90 2023-10-16T12:53:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.501+0000 7fe3bb7fe640 1 -- 172.21.15.60:0/619177196 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac0319d0 con 0x7fe3bc15af90 2023-10-16T12:53:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.501+0000 7fe3bb7fe640 1 -- 172.21.15.60:0/619177196 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac031d60 con 0x7fe3bc15af90 2023-10-16T12:53:13.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.501+0000 7fe3c30c2640 1 -- 172.21.15.60:0/619177196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc15af90 msgr2=0x7fe3bc0ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:13.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.501+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/619177196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc15af90 0x7fe3bc0ccb60 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc070ea0 tx=0x7fe3ac02b910 comp rx=0 tx=0).stop 2023-10-16T12:53:13.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.502+0000 7fe3c30c2640 1 -- 172.21.15.60:0/619177196 shutdown_connections 2023-10-16T12:53:13.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.502+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/619177196 >> v2:172.21.15.144:3300/0 conn(0x7fe3bc0cd320 0x7fe3bc0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.502+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/619177196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc15af90 0x7fe3bc0ccb60 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.502+0000 7fe3c30c2640 1 -- 172.21.15.60:0/619177196 >> 172.21.15.60:0/619177196 conn(0x7fe3bc099e30 msgr2=0x7fe3bc0d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:13.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.502+0000 7fe3c30c2640 1 -- 172.21.15.60:0/619177196 shutdown_connections 2023-10-16T12:53:13.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.502+0000 7fe3c30c2640 1 -- 172.21.15.60:0/619177196 wait complete. 2023-10-16T12:53:13.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.503+0000 7fe3c30c2640 1 Processor -- start 2023-10-16T12:53:13.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.504+0000 7fe3c30c2640 1 -- start start 2023-10-16T12:53:13.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.504+0000 7fe3c30c2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 0x7fe3bc06c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:13.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.504+0000 7fe3c30c2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3bc0cf3d0 0x7fe3bc0c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c30c2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3bc15af90 0x7fe3bc0c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c30c2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc06c030 con 0x7fe3bc0cd320 2023-10-16T12:53:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c30c2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0ca470 con 0x7fe3bc0cf3d0 2023-10-16T12:53:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c30c2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1539f0 con 0x7fe3bc15af90 2023-10-16T12:53:13.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 0x7fe3bc06c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:13.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 0x7fe3bc06c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39654/0 (socket says 172.21.15.60:39654) 2023-10-16T12:53:13.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/361177966 learned_addr learned my addr 172.21.15.60:0/361177966 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:13.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3bbfff640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3bc0cf3d0 0x7fe3bc0c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:13.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c165f640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3bc15af90 0x7fe3bc0c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:13.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3bc15af90 msgr2=0x7fe3bc0c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:13.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3bc15af90 0x7fe3bc0c5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3bc0cf3d0 msgr2=0x7fe3bc0c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:13.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3bc0cf3d0 0x7fe3bc0c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:13.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.505+0000 7fe3c0e5e640 1 -- 172.21.15.60:0/361177966 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac0094d0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.506+0000 7fe3c0e5e640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 0x7fe3bc06c450 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac000c00 tx=0x7fe3ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:13.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.506+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac031860 con 0x7fe3bc0cd320 2023-10-16T12:53:13.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.506+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac0443f0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.506+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0447a0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.506+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc06b3b0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.506+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc06b970 con 0x7fe3bc0cd320 2023-10-16T12:53:13.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.507+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc06c2a0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.509+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe3ac044900 con 0x7fe3bc0cd320 2023-10-16T12:53:13.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.510+0000 7fe3b97fa640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe3a806af90 0x7fe3a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:13.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.510+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6597+0+0 (secure 0 0 0) 0x7fe3ac02d040 con 0x7fe3bc0cd320 2023-10-16T12:53:13.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.510+0000 7fe3bbfff640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe3a806af90 0x7fe3a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:13.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.511+0000 7fe3bbfff640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe3a806af90 0x7fe3a806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0007510 tx=0x7fe3b00074a0 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:13.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.514+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3ac0770d0 con 0x7fe3bc0cd320 2023-10-16T12:53:13.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:13.711+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe3bc04e160 con 0x7fe3bc0cd320 2023-10-16T12:53:14.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.189+0000 7fe3b97fa640 1 -- 172.21.15.60:0/361177966 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe3ac07a6f0 con 0x7fe3bc0cd320 2023-10-16T12:53:14.192 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:53:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.194+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe3a806af90 msgr2=0x7fe3a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.194+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe3a806af90 0x7fe3a806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0007510 tx=0x7fe3b00074a0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.194+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 msgr2=0x7fe3bc06c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.194+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 0x7fe3bc06c450 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac000c00 tx=0x7fe3ac0046c0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 shutdown_connections 2023-10-16T12:53:14.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe3a806af90 0x7fe3a806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3bc15af90 0x7fe3bc0c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3bc0cf3d0 0x7fe3bc0c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 --2- 172.21.15.60:0/361177966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3bc0cd320 0x7fe3bc06c450 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 >> 172.21.15.60:0/361177966 conn(0x7fe3bc099e30 msgr2=0x7fe3bc06e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.195+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 shutdown_connections 2023-10-16T12:53:14.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.196+0000 7fe3c30c2640 1 -- 172.21.15.60:0/361177966 wait complete. 2023-10-16T12:53:14.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:53:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.453+0000 7f86da733640 1 Processor -- start 2023-10-16T12:53:14.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.453+0000 7f86da733640 1 -- start start 2023-10-16T12:53:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.454+0000 7f86da733640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f86d40b0360 0x7f86d40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.454+0000 7f86da733640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86d40ca510 0x7f86d40ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.454+0000 7f86da733640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d415b310 con 0x7f86d415ac10 2023-10-16T12:53:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.454+0000 7f86da733640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d415f800 con 0x7f86d40ca510 2023-10-16T12:53:14.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.454+0000 7f86da733640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d40b0d30 con 0x7f86d40b0360 2023-10-16T12:53:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86d415ac10 0x7f86d406c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45598/0 (socket says 172.21.15.60:45598) 2023-10-16T12:53:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d3fff640 1 -- 172.21.15.60:0/3853662506 learned_addr learned my addr 172.21.15.60:0/3853662506 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d8cd0640 1 --2- 172.21.15.60:0/3853662506 >> v2:172.21.15.60:3300/0 conn(0x7f86d40ca510 0x7f86d40ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d37fe640 1 -- 172.21.15.60:0/3853662506 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095763854 0 0) 0x7f86d415b310 con 0x7f86d415ac10 2023-10-16T12:53:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d37fe640 1 -- 172.21.15.60:0/3853662506 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0003520 con 0x7f86d415ac10 2023-10-16T12:53:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d8cd0640 1 -- 172.21.15.60:0/3853662506 >> v2:172.21.15.144:3300/0 conn(0x7f86d40b0360 msgr2=0x7f86d40b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d8cd0640 1 --2- 172.21.15.60:0/3853662506 >> v2:172.21.15.144:3300/0 conn(0x7f86d40b0360 0x7f86d40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d8cd0640 1 -- 172.21.15.60:0/3853662506 >> v1:172.21.15.60:6789/0 conn(0x7f86d415ac10 legacy=0x7f86d406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.455+0000 7f86d8cd0640 1 -- 172.21.15.60:0/3853662506 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d406f640 con 0x7f86d40ca510 2023-10-16T12:53:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.456+0000 7f86d8cd0640 1 --2- 172.21.15.60:0/3853662506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40ca510 0x7f86d40ca920 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f86d40ae2b0 tx=0x7f86cc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc65b3a120716a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86d37fe640 1 -- 172.21.15.60:0/3853662506 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc02cbe0 con 0x7f86d40ca510 2023-10-16T12:53:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86d37fe640 1 -- 172.21.15.60:0/3853662506 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc02cd40 con 0x7f86d40ca510 2023-10-16T12:53:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86d37fe640 1 -- 172.21.15.60:0/3853662506 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc034660 con 0x7f86d40ca510 2023-10-16T12:53:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 -- 172.21.15.60:0/3853662506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40ca510 msgr2=0x7f86d40ca920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 --2- 172.21.15.60:0/3853662506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40ca510 0x7f86d40ca920 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f86d40ae2b0 tx=0x7f86cc02be30 comp rx=0 tx=0).stop 2023-10-16T12:53:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 -- 172.21.15.60:0/3853662506 shutdown_connections 2023-10-16T12:53:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 --2- 172.21.15.60:0/3853662506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40ca510 0x7f86d40ca920 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 --2- 172.21.15.60:0/3853662506 >> v2:172.21.15.144:3300/0 conn(0x7f86d40b0360 0x7f86d40b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 -- 172.21.15.60:0/3853662506 >> 172.21.15.60:0/3853662506 conn(0x7f86d40cb360 msgr2=0x7f86d40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 -- 172.21.15.60:0/3853662506 shutdown_connections 2023-10-16T12:53:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.457+0000 7f86da733640 1 -- 172.21.15.60:0/3853662506 wait complete. 2023-10-16T12:53:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.459+0000 7f86da733640 1 Processor -- start 2023-10-16T12:53:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.459+0000 7f86da733640 1 -- start start 2023-10-16T12:53:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.460+0000 7f86da733640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 0x7f86d406b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.460+0000 7f86da733640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86d40ca510 0x7f86d40b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.460+0000 7f86da733640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86d415ac10 0x7f86d40b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.460+0000 7f86da733640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4157f90 con 0x7f86d40b0360 2023-10-16T12:53:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.460+0000 7f86da733640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4156c10 con 0x7f86d40ca510 2023-10-16T12:53:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.460+0000 7f86da733640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4156ef0 con 0x7f86d415ac10 2023-10-16T12:53:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 0x7f86d406b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 0x7f86d406b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39666/0 (socket says 172.21.15.60:39666) 2023-10-16T12:53:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 -- 172.21.15.60:0/928156593 learned_addr learned my addr 172.21.15.60:0/928156593 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 -- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86d415ac10 msgr2=0x7f86d40b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86d415ac10 0x7f86d40b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 -- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86d40ca510 msgr2=0x7f86d40b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86d40ca510 0x7f86d40b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 -- 172.21.15.60:0/928156593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc009a70 con 0x7f86d40b0360 2023-10-16T12:53:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d3fff640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 0x7f86d406b230 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f86c400c500 tx=0x7f86c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.461+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c4005ce0 con 0x7f86d40b0360 2023-10-16T12:53:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.462+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c4005e40 con 0x7f86d40b0360 2023-10-16T12:53:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.462+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c4007330 con 0x7f86d40b0360 2023-10-16T12:53:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.462+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d41571d0 con 0x7f86d40b0360 2023-10-16T12:53:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.462+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d4157790 con 0x7f86d40b0360 2023-10-16T12:53:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.463+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d404e160 con 0x7f86d40b0360 2023-10-16T12:53:14.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.464+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f86c4007550 con 0x7f86d40b0360 2023-10-16T12:53:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.465+0000 7f86d17fa640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f86b406af90 0x7f86b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.465+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f86c4088fe0 con 0x7f86d40b0360 2023-10-16T12:53:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.465+0000 7f86cbfff640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f86b406af90 0x7f86b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.467+0000 7f86cbfff640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f86b406af90 0x7f86b406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009670 tx=0x7f86bc006bf0 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.469+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f86c4053e90 con 0x7f86d40b0360 2023-10-16T12:53:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.667+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86d406cea0 con 0x7f86d40b0360 2023-10-16T12:53:14.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.877+0000 7f86d17fa640 1 -- 172.21.15.60:0/928156593 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 169+0+0 (secure 0 0 0) 0x7f86c4017020 con 0x7f86d40b0360 2023-10-16T12:53:14.880 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:53:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.881+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f86b406af90 msgr2=0x7f86b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.881+0000 7f86da733640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f86b406af90 0x7f86b406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009670 tx=0x7f86bc006bf0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.881+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 msgr2=0x7f86d406b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:14.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.881+0000 7f86da733640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 0x7f86d406b230 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f86c400c500 tx=0x7f86c400c9c0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.882+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 shutdown_connections 2023-10-16T12:53:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.882+0000 7f86da733640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f86b406af90 0x7f86b406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.882+0000 7f86da733640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f86d415ac10 0x7f86d40b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.882+0000 7f86da733640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86d40ca510 0x7f86d40b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.882+0000 7f86da733640 1 --2- 172.21.15.60:0/928156593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86d40b0360 0x7f86d406b230 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.882+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 >> 172.21.15.60:0/928156593 conn(0x7f86d40cb360 msgr2=0x7f86d406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:14.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.884+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 shutdown_connections 2023-10-16T12:53:14.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:14.884+0000 7f86da733640 1 -- 172.21.15.60:0/928156593 wait complete. 2023-10-16T12:53:14.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:53:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.118+0000 7f940bc21640 1 Processor -- start 2023-10-16T12:53:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.119+0000 7f940bc21640 1 -- start start 2023-10-16T12:53:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.120+0000 7f940bc21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.120+0000 7f940bc21640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94040adfc0 0x7f94040b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.120+0000 7f940bc21640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94040cc600 con 0x7f9404070fd0 2023-10-16T12:53:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.120+0000 7f940bc21640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9404071470 con 0x7f94040adfc0 2023-10-16T12:53:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.120+0000 7f940bc21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94040cc870 con 0x7f94040adbe0 2023-10-16T12:53:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94040adfc0 0x7f94040b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94040adfc0 0x7f94040b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39670/0 (socket says 172.21.15.60:39670) 2023-10-16T12:53:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 -- 172.21.15.60:0/2644505178 learned_addr learned my addr 172.21.15.60:0/2644505178 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f94099bd640 1 --2- 172.21.15.60:0/2644505178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 -- 172.21.15.60:0/2644505178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 msgr2=0x7f94040cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 --2- 172.21.15.60:0/2644505178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 -- 172.21.15.60:0/2644505178 >> v1:172.21.15.144:6789/0 conn(0x7f9404070fd0 legacy=0x7f940406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.121+0000 7f940a1be640 1 -- 172.21.15.60:0/2644505178 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94040ccb50 con 0x7f94040adfc0 2023-10-16T12:53:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.123+0000 7f940a1be640 1 --2- 172.21.15.60:0/2644505178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040b0410 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009e50 tx=0x7f93fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1411fa6300c05d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.123+0000 7f94089bb640 1 -- 172.21.15.60:0/2644505178 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc02cb00 con 0x7f94040adfc0 2023-10-16T12:53:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.123+0000 7f94089bb640 1 -- 172.21.15.60:0/2644505178 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc02cc60 con 0x7f94040adfc0 2023-10-16T12:53:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.123+0000 7f94089bb640 1 -- 172.21.15.60:0/2644505178 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0343f0 con 0x7f94040adfc0 2023-10-16T12:53:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.123+0000 7f940bc21640 1 -- 172.21.15.60:0/2644505178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 msgr2=0x7f94040b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.123+0000 7f940bc21640 1 --2- 172.21.15.60:0/2644505178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040b0410 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009e50 tx=0x7f93fc02bf60 comp rx=0 tx=0).stop 2023-10-16T12:53:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.124+0000 7f940bc21640 1 -- 172.21.15.60:0/2644505178 shutdown_connections 2023-10-16T12:53:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.124+0000 7f940bc21640 1 --2- 172.21.15.60:0/2644505178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040b0410 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.124+0000 7f940bc21640 1 --2- 172.21.15.60:0/2644505178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.124+0000 7f940bc21640 1 -- 172.21.15.60:0/2644505178 >> 172.21.15.60:0/2644505178 conn(0x7f94040cb410 msgr2=0x7f94040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.124+0000 7f940bc21640 1 -- 172.21.15.60:0/2644505178 shutdown_connections 2023-10-16T12:53:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.124+0000 7f940bc21640 1 -- 172.21.15.60:0/2644505178 wait complete. 2023-10-16T12:53:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.125+0000 7f940bc21640 1 Processor -- start 2023-10-16T12:53:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.126+0000 7f940bc21640 1 -- start start 2023-10-16T12:53:15.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940bc21640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9404070fd0 0x7f940415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940bc21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940bc21640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940bc21640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041640a0 con 0x7f94040adfc0 2023-10-16T12:53:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940bc21640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9404164860 con 0x7f9404070fd0 2023-10-16T12:53:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940bc21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94041642a0 con 0x7f94040adbe0 2023-10-16T12:53:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940a1be640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940a1be640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39684/0 (socket says 172.21.15.60:39684) 2023-10-16T12:53:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.127+0000 7f940a1be640 1 -- 172.21.15.60:0/3822277038 learned_addr learned my addr 172.21.15.60:0/3822277038 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f94091bc640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f940a1be640 1 -- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 msgr2=0x7f94040b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f940a1be640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f940a1be640 1 -- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9404070fd0 msgr2=0x7f940415c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f940a1be640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9404070fd0 0x7f940415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f940a1be640 1 -- 172.21.15.60:0/3822277038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc009b30 con 0x7f94040adfc0 2023-10-16T12:53:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.128+0000 7f940a1be640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040afbd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009e50 tx=0x7f93fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.129+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc040670 con 0x7f94040adfc0 2023-10-16T12:53:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.129+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc0407d0 con 0x7f94040adfc0 2023-10-16T12:53:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.129+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc040b80 con 0x7f94040adfc0 2023-10-16T12:53:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.129+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9404164580 con 0x7f94040adfc0 2023-10-16T12:53:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.129+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9404161d70 con 0x7f94040adfc0 2023-10-16T12:53:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.131+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940404e160 con 0x7f94040adfc0 2023-10-16T12:53:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.133+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93fc040ce0 con 0x7f94040adfc0 2023-10-16T12:53:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.135+0000 7f93f67fc640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f93e806af90 0x7f93e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.135+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f93fc0ac810 con 0x7f94040adfc0 2023-10-16T12:53:15.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.136+0000 7f94099bd640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f93e806af90 0x7f93e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.137+0000 7f94099bd640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f93e806af90 0x7f93e806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94000098f0 tx=0x7f9400008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.140+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93fc077770 con 0x7f94040adfc0 2023-10-16T12:53:15.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.339+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f94040b4910 con 0x7f94040adfc0 2023-10-16T12:53:15.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.346+0000 7f93f67fc640 1 -- 172.21.15.60:0/3822277038 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7f93fc009510 con 0x7f94040adfc0 2023-10-16T12:53:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f93e806af90 msgr2=0x7f93e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f93e806af90 0x7f93e806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94000098f0 tx=0x7f9400008040 comp rx=0 tx=0).stop 2023-10-16T12:53:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 msgr2=0x7f94040afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040afbd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009e50 tx=0x7f93fc004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 shutdown_connections 2023-10-16T12:53:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f93e806af90 0x7f93e806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94040adbe0 0x7f94040b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94040adfc0 0x7f94040afbd0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 --2- 172.21.15.60:0/3822277038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9404070fd0 0x7f940415c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.349+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 >> 172.21.15.60:0/3822277038 conn(0x7f94040cb410 msgr2=0x7f940406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.350+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 shutdown_connections 2023-10-16T12:53:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.350+0000 7f940bc21640 1 -- 172.21.15.60:0/3822277038 wait complete. 2023-10-16T12:53:15.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:53:15.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.593+0000 7f886c61c640 1 Processor -- start 2023-10-16T12:53:15.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.593+0000 7f886c61c640 1 -- start start 2023-10-16T12:53:15.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f886c61c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f886406a4b0 0x7f88640ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f886c61c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640cb030 con 0x7f88640c9ca0 2023-10-16T12:53:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f886c61c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640c91c0 con 0x7f886415ad70 2023-10-16T12:53:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f886c61c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88640c94a0 con 0x7f886406a4b0 2023-10-16T12:53:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f8869bb7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f886406a4b0 0x7f88640ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f886abb9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f88640c9ca0 0x7f88640ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45604/0 (socket says 172.21.15.60:45604) 2023-10-16T12:53:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.595+0000 7f886abb9640 1 -- 172.21.15.60:0/1940977165 learned_addr learned my addr 172.21.15.60:0/1940977165 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.596+0000 7f8869bb7640 1 -- 172.21.15.60:0/1940977165 >> v1:172.21.15.144:6789/0 conn(0x7f886415ad70 legacy=0x7f8864069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.596+0000 7f8869bb7640 1 -- 172.21.15.60:0/1940977165 >> v1:172.21.15.60:6789/0 conn(0x7f88640c9ca0 legacy=0x7f88640ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.596+0000 7f8869bb7640 1 -- 172.21.15.60:0/1940977165 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88640c9780 con 0x7f886406a4b0 2023-10-16T12:53:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.597+0000 7f8869bb7640 1 --2- 172.21.15.60:0/1940977165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ca3b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f88540097a0 tx=0x7f885402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38037705f710b46 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.597+0000 7f88693b6640 1 -- 172.21.15.60:0/1940977165 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854030610 con 0x7f886406a4b0 2023-10-16T12:53:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.597+0000 7f88693b6640 1 -- 172.21.15.60:0/1940977165 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8854030770 con 0x7f886406a4b0 2023-10-16T12:53:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.597+0000 7f88693b6640 1 -- 172.21.15.60:0/1940977165 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854030b00 con 0x7f886406a4b0 2023-10-16T12:53:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 -- 172.21.15.60:0/1940977165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 msgr2=0x7f88640ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 --2- 172.21.15.60:0/1940977165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ca3b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f88540097a0 tx=0x7f885402b960 comp rx=0 tx=0).stop 2023-10-16T12:53:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 -- 172.21.15.60:0/1940977165 shutdown_connections 2023-10-16T12:53:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 --2- 172.21.15.60:0/1940977165 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ca3b0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 -- 172.21.15.60:0/1940977165 >> 172.21.15.60:0/1940977165 conn(0x7f88640ce9b0 msgr2=0x7f88640cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 -- 172.21.15.60:0/1940977165 shutdown_connections 2023-10-16T12:53:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.598+0000 7f886c61c640 1 -- 172.21.15.60:0/1940977165 wait complete. 2023-10-16T12:53:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.600+0000 7f886c61c640 1 Processor -- start 2023-10-16T12:53:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.600+0000 7f886c61c640 1 -- start start 2023-10-16T12:53:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886c61c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886c61c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88640c9ca0 0x7f88640cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886c61c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886415ad70 0x7f88640c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886c61c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886415ccd0 con 0x7f886406a4b0 2023-10-16T12:53:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886c61c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88640ce610 con 0x7f88640c9ca0 2023-10-16T12:53:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886c61c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886415c650 con 0x7f886415ad70 2023-10-16T12:53:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886a3b8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886a3b8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39694/0 (socket says 172.21.15.60:39694) 2023-10-16T12:53:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.601+0000 7f886a3b8640 1 -- 172.21.15.60:0/2861692207 learned_addr learned my addr 172.21.15.60:0/2861692207 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f8869bb7640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88640c9ca0 0x7f88640cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886abb9640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886415ad70 0x7f88640c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886a3b8640 1 -- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886415ad70 msgr2=0x7f88640c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886a3b8640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886415ad70 0x7f88640c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886a3b8640 1 -- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88640c9ca0 msgr2=0x7f88640cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886a3b8640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88640c9ca0 0x7f88640cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886a3b8640 1 -- 172.21.15.60:0/2861692207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8854009400 con 0x7f886406a4b0 2023-10-16T12:53:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886abb9640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886415ad70 0x7f88640c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f886a3b8640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ac890 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f886000c430 tx=0x7f886000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88600052d0 con 0x7f886406a4b0 2023-10-16T12:53:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8860005430 con 0x7f886406a4b0 2023-10-16T12:53:15.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.602+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860003830 con 0x7f886406a4b0 2023-10-16T12:53:15.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.603+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88640ce330 con 0x7f886406a4b0 2023-10-16T12:53:15.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.603+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88640ccfd0 con 0x7f886406a4b0 2023-10-16T12:53:15.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.604+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886415c4b0 con 0x7f886406a4b0 2023-10-16T12:53:15.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.605+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8860003990 con 0x7f886406a4b0 2023-10-16T12:53:15.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.606+0000 7f885affd640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f884806af90 0x7f884806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:15.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.606+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f8860088d30 con 0x7f886406a4b0 2023-10-16T12:53:15.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.606+0000 7f8869bb7640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f884806af90 0x7f884806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:15.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.608+0000 7f8869bb7640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f884806af90 0x7f884806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8854036000 tx=0x7f8854008030 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:15.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.610+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f886415c4b0 con 0x7f886406a4b0 2023-10-16T12:53:15.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.807+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f886415c4b0 con 0x7f886406a4b0 2023-10-16T12:53:15.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.813+0000 7f885affd640 1 -- 172.21.15.60:0/2861692207 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f886415c4b0 con 0x7f886406a4b0 2023-10-16T12:53:15.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f884806af90 msgr2=0x7f884806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f884806af90 0x7f884806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8854036000 tx=0x7f8854008030 comp rx=0 tx=0).stop 2023-10-16T12:53:15.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 msgr2=0x7f88640ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:15.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ac890 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f886000c430 tx=0x7f886000c8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 shutdown_connections 2023-10-16T12:53:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f884806af90 0x7f884806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f886415ad70 0x7f88640c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88640c9ca0 0x7f88640cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 --2- 172.21.15.60:0/2861692207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886406a4b0 0x7f88640ac890 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:15.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.815+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 >> 172.21.15.60:0/2861692207 conn(0x7f88640ce9b0 msgr2=0x7f886406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:15.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.816+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 shutdown_connections 2023-10-16T12:53:15.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:15.816+0000 7f886c61c640 1 -- 172.21.15.60:0/2861692207 wait complete. 2023-10-16T12:53:15.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:53:16.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.065+0000 7f4d45f99640 1 Processor -- start 2023-10-16T12:53:16.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.066+0000 7f4d45f99640 1 -- start start 2023-10-16T12:53:16.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.067+0000 7f4d45f99640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d4006c040 0x7f4d4006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.067+0000 7f4d45f99640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4015f4a0 con 0x7f4d400ad970 2023-10-16T12:53:16.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.067+0000 7f4d45f99640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4006c960 con 0x7f4d400cda80 2023-10-16T12:53:16.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.067+0000 7f4d45f99640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4015b9d0 con 0x7f4d4006c040 2023-10-16T12:53:16.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.068+0000 7f4d37fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4d400ad970 0x7f4d4015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45606/0 (socket says 172.21.15.60:45606) 2023-10-16T12:53:16.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.068+0000 7f4d37fff640 1 -- 172.21.15.60:0/1763538579 learned_addr learned my addr 172.21.15.60:0/1763538579 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:16.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.068+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1763538579 >> v2:172.21.15.144:3300/0 conn(0x7f4d4006c040 0x7f4d4006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.068+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227614258 0 0) 0x7f4d4015f4a0 con 0x7f4d400ad970 2023-10-16T12:53:16.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.068+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d28003520 con 0x7f4d400ad970 2023-10-16T12:53:16.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1582587929 0 0) 0x7f4d28003520 con 0x7f4d400ad970 2023-10-16T12:53:16.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 >> v2:172.21.15.144:3300/0 conn(0x7f4d4006c040 msgr2=0x7f4d4006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:16.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 --2- 172.21.15.60:0/1763538579 >> v2:172.21.15.144:3300/0 conn(0x7f4d4006c040 0x7f4d4006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 >> v1:172.21.15.144:6789/0 conn(0x7f4d400cda80 legacy=0x7f4d4015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4015bcb0 con 0x7f4d400ad970 2023-10-16T12:53:16.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4d30001670 con 0x7f4d400ad970 2023-10-16T12:53:16.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 >> v1:172.21.15.60:6789/0 conn(0x7f4d400ad970 legacy=0x7f4d4015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d280047c0 0x7f4d28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d28005180 0x7f4d2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4015b9d0 con 0x7f4d280039c0 2023-10-16T12:53:16.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2800c080 con 0x7f4d280047c0 2023-10-16T12:53:16.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2800c240 con 0x7f4d28005180 2023-10-16T12:53:16.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.069+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.070+0000 7f4d44f97640 1 -- 172.21.15.60:0/1763538579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d28005180 msgr2=0x7f4d2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:16.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.070+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d28005180 0x7f4d2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.070+0000 7f4d44f97640 1 -- 172.21.15.60:0/1763538579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d280047c0 msgr2=0x7f4d28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:16.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.070+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d280047c0 0x7f4d28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.070+0000 7f4d44f97640 1 -- 172.21.15.60:0/1763538579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28003520 con 0x7f4d280039c0 2023-10-16T12:53:16.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.070+0000 7f4d45798640 1 -- 172.21.15.60:0/1763538579 reap_dead start 2023-10-16T12:53:16.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.071+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d28003de0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009ac0 tx=0x7f4d280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a9e4068acea4ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c02c6b0 con 0x7f4d280039c0 2023-10-16T12:53:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c02c810 con 0x7f4d280039c0 2023-10-16T12:53:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d377fe640 1 -- 172.21.15.60:0/1763538579 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c02cba0 con 0x7f4d280039c0 2023-10-16T12:53:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 -- 172.21.15.60:0/1763538579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 msgr2=0x7f4d28003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d28003de0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009ac0 tx=0x7f4d280057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 -- 172.21.15.60:0/1763538579 shutdown_connections 2023-10-16T12:53:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1763538579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d28003de0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 -- 172.21.15.60:0/1763538579 >> 172.21.15.60:0/1763538579 conn(0x7f4d400ce6a0 msgr2=0x7f4d400ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 -- 172.21.15.60:0/1763538579 shutdown_connections 2023-10-16T12:53:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.072+0000 7f4d45f99640 1 -- 172.21.15.60:0/1763538579 wait complete. 2023-10-16T12:53:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.074+0000 7f4d45f99640 1 Processor -- start 2023-10-16T12:53:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.074+0000 7f4d45f99640 1 -- start start 2023-10-16T12:53:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.075+0000 7f4d45f99640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d400ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.075+0000 7f4d45f99640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d400b2b10 0x7f4d4014cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.075+0000 7f4d45f99640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d400b2f90 0x7f4d4014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.075+0000 7f4d45f99640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d400ae390 con 0x7f4d280039c0 2023-10-16T12:53:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.075+0000 7f4d45f99640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d400c5b50 con 0x7f4d400b2b10 2023-10-16T12:53:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.075+0000 7f4d45f99640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d400c5e30 con 0x7f4d400b2f90 2023-10-16T12:53:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d400ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d400ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39720/0 (socket says 172.21.15.60:39720) 2023-10-16T12:53:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 -- 172.21.15.60:0/1454824434 learned_addr learned my addr 172.21.15.60:0/1454824434 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:16.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d37fff640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d400b2b10 0x7f4d4014cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 -- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d400b2f90 msgr2=0x7f4d4014d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d45798640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d400b2f90 0x7f4d4014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d400b2f90 0x7f4d4014d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 -- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d400b2b10 msgr2=0x7f4d4014cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d400b2b10 0x7f4d4014cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d44f97640 1 -- 172.21.15.60:0/1454824434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400c6110 con 0x7f4d280039c0 2023-10-16T12:53:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.076+0000 7f4d37fff640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d400b2b10 0x7f4d4014cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:16.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.077+0000 7f4d44f97640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d400ade50 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009ac0 tx=0x7f4d3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.077+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c003810 con 0x7f4d280039c0 2023-10-16T12:53:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.077+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c003970 con 0x7f4d280039c0 2023-10-16T12:53:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.077+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c003d00 con 0x7f4d280039c0 2023-10-16T12:53:16.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.077+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d400c63f0 con 0x7f4d280039c0 2023-10-16T12:53:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.077+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4014de80 con 0x7f4d280039c0 2023-10-16T12:53:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.078+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4006b330 con 0x7f4d280039c0 2023-10-16T12:53:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.079+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4d3c040450 con 0x7f4d280039c0 2023-10-16T12:53:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.080+0000 7f4d357fa640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4d2c06afc0 0x7f4d2c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.080+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f4d3c0ab6b0 con 0x7f4d280039c0 2023-10-16T12:53:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.081+0000 7f4d37fff640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4d2c06afc0 0x7f4d2c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.082+0000 7f4d37fff640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4d2c06afc0 0x7f4d2c06d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d300017a0 tx=0x7f4d30009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.085+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d3c076610 con 0x7f4d280039c0 2023-10-16T12:53:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.296+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f4d4004e9b0 con 0x7f4d280039c0 2023-10-16T12:53:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.303+0000 7f4d357fa640 1 -- 172.21.15.60:0/1454824434 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4d3c003e60 con 0x7f4d280039c0 2023-10-16T12:53:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4d2c06afc0 msgr2=0x7f4d2c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4d2c06afc0 0x7f4d2c06d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d300017a0 tx=0x7f4d30009040 comp rx=0 tx=0).stop 2023-10-16T12:53:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 msgr2=0x7f4d400ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d400ade50 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009ac0 tx=0x7f4d3c004730 comp rx=0 tx=0).stop 2023-10-16T12:53:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 shutdown_connections 2023-10-16T12:53:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4d2c06afc0 0x7f4d2c06d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d400b2f90 0x7f4d4014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d400b2b10 0x7f4d4014cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 --2- 172.21.15.60:0/1454824434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d280039c0 0x7f4d400ade50 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 >> 172.21.15.60:0/1454824434 conn(0x7f4d400ce6a0 msgr2=0x7f4d400c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 shutdown_connections 2023-10-16T12:53:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.306+0000 7f4d45f99640 1 -- 172.21.15.60:0/1454824434 wait complete. 2023-10-16T12:53:16.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T12:53:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.547+0000 7f8b9ed43640 1 Processor -- start 2023-10-16T12:53:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.547+0000 7f8b9ed43640 1 -- start start 2023-10-16T12:53:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.548+0000 7f8b9ed43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9815cd70 0x7f8b980d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.548+0000 7f8b9ed43640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b980b8480 0x7f8b980cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.548+0000 7f8b9ed43640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9815d150 con 0x7f8b980d0be0 2023-10-16T12:53:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.548+0000 7f8b9ed43640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b980d1180 con 0x7f8b980b8480 2023-10-16T12:53:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.548+0000 7f8b9ed43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980d1460 con 0x7f8b9815cd70 2023-10-16T12:53:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.549+0000 7f8b9d540640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b980b8480 0x7f8b980cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.549+0000 7f8b9d540640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b980b8480 0x7f8b980cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39734/0 (socket says 172.21.15.60:39734) 2023-10-16T12:53:16.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.549+0000 7f8b9d540640 1 -- 172.21.15.60:0/3249078585 learned_addr learned my addr 172.21.15.60:0/3249078585 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:16.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.549+0000 7f8b9dd41640 1 --2- 172.21.15.60:0/3249078585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9815cd70 0x7f8b980d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.550+0000 7f8b9d540640 1 -- 172.21.15.60:0/3249078585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9815cd70 msgr2=0x7f8b980d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:16.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.550+0000 7f8b9d540640 1 --2- 172.21.15.60:0/3249078585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9815cd70 0x7f8b980d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.550+0000 7f8b9d540640 1 -- 172.21.15.60:0/3249078585 >> v1:172.21.15.144:6789/0 conn(0x7f8b980d0be0 legacy=0x7f8b980b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.550+0000 7f8b9d540640 1 -- 172.21.15.60:0/3249078585 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980d1740 con 0x7f8b980b8480 2023-10-16T12:53:16.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.551+0000 7f8b9d540640 1 --2- 172.21.15.60:0/3249078585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b980cd7c0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009870 tx=0x7f8b8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ec44729f55ea5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:16.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.551+0000 7f8b9cd3f640 1 -- 172.21.15.60:0/3249078585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88030840 con 0x7f8b980b8480 2023-10-16T12:53:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.551+0000 7f8b9cd3f640 1 -- 172.21.15.60:0/3249078585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b880309a0 con 0x7f8b980b8480 2023-10-16T12:53:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.551+0000 7f8b9cd3f640 1 -- 172.21.15.60:0/3249078585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88030d30 con 0x7f8b980b8480 2023-10-16T12:53:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3249078585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 msgr2=0x7f8b980cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3249078585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b980cd7c0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009870 tx=0x7f8b8802b980 comp rx=0 tx=0).stop 2023-10-16T12:53:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3249078585 shutdown_connections 2023-10-16T12:53:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3249078585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b980cd7c0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3249078585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b9815cd70 0x7f8b980d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3249078585 >> 172.21.15.60:0/3249078585 conn(0x7f8b980cf570 msgr2=0x7f8b980cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:16.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3249078585 shutdown_connections 2023-10-16T12:53:16.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.552+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3249078585 wait complete. 2023-10-16T12:53:16.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.554+0000 7f8b9ed43640 1 Processor -- start 2023-10-16T12:53:16.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.554+0000 7f8b9ed43640 1 -- start start 2023-10-16T12:53:16.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9ed43640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b9815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9ed43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b980d0be0 0x7f8b980d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9ed43640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9815cd70 0x7f8b981f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9ed43640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b981f6770 con 0x7f8b980b8480 2023-10-16T12:53:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9ed43640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b981f5b60 con 0x7f8b9815cd70 2023-10-16T12:53:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9ed43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b981f5e40 con 0x7f8b980d0be0 2023-10-16T12:53:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9dd41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b9815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9dd41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b9815fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39740/0 (socket says 172.21.15.60:39740) 2023-10-16T12:53:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.555+0000 7f8b9dd41640 1 -- 172.21.15.60:0/3745746451 learned_addr learned my addr 172.21.15.60:0/3745746451 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9d540640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b980d0be0 0x7f8b980d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9e542640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9815cd70 0x7f8b981f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9dd41640 1 -- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b980d0be0 msgr2=0x7f8b980d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9dd41640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b980d0be0 0x7f8b980d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9dd41640 1 -- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9815cd70 msgr2=0x7f8b981f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9dd41640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9815cd70 0x7f8b981f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9dd41640 1 -- 172.21.15.60:0/3745746451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b880094d0 con 0x7f8b980b8480 2023-10-16T12:53:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9e542640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9815cd70 0x7f8b981f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9d540640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b980d0be0 0x7f8b980d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9dd41640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b9815fd40 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400d800 tx=0x7f8b9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400b6c0 con 0x7f8b980b8480 2023-10-16T12:53:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9400b820 con 0x7f8b980b8480 2023-10-16T12:53:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b981f6120 con 0x7f8b980b8480 2023-10-16T12:53:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.556+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400bbb0 con 0x7f8b980b8480 2023-10-16T12:53:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.557+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98160280 con 0x7f8b980b8480 2023-10-16T12:53:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.558+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9804e9b0 con 0x7f8b980b8480 2023-10-16T12:53:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.559+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b9400bd10 con 0x7f8b980b8480 2023-10-16T12:53:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.560+0000 7f8b8e7fc640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f8b7c06af90 0x7f8b7c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.560+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f8b94089cd0 con 0x7f8b980b8480 2023-10-16T12:53:16.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.561+0000 7f8b9d540640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f8b7c06af90 0x7f8b7c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:16.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.562+0000 7f8b9d540640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f8b7c06af90 0x7f8b7c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8b88036000 tx=0x7f8b8803b040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:16.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.565+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b94054cd0 con 0x7f8b980b8480 2023-10-16T12:53:16.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:16.834+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f8b980d2470 con 0x7f8b980b8480 2023-10-16T12:53:17.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.198+0000 7f8b8e7fc640 1 -- 172.21.15.60:0/3745746451 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v757) v1 ==== 148+0+0 (secure 0 0 0) 0x7f8b940582f0 con 0x7f8b980b8480 2023-10-16T12:53:17.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.201+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f8b7c06af90 msgr2=0x7f8b7c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f8b7c06af90 0x7f8b7c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8b88036000 tx=0x7f8b8803b040 comp rx=0 tx=0).stop 2023-10-16T12:53:17.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 msgr2=0x7f8b9815fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b9815fd40 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400d800 tx=0x7f8b9400dcc0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 shutdown_connections 2023-10-16T12:53:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f8b7c06af90 0x7f8b7c06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b980d0be0 0x7f8b980d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b9815cd70 0x7f8b981f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 --2- 172.21.15.60:0/3745746451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b980b8480 0x7f8b9815fd40 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 >> 172.21.15.60:0/3745746451 conn(0x7f8b980cf570 msgr2=0x7f8b980b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:17.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 shutdown_connections 2023-10-16T12:53:17.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.202+0000 7f8b9ed43640 1 -- 172.21.15.60:0/3745746451 wait complete. 2023-10-16T12:53:17.227 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-16T12:53:17.227 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:53:17.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:17.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.458+0000 7f03e8044640 1 Processor -- start 2023-10-16T12:53:17.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.458+0000 7f03e8044640 1 -- start start 2023-10-16T12:53:17.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e8044640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03e015ae80 0x7f03e015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e8044640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f03e00ce390 0x7f03e0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e8044640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e015ed60 con 0x7f03e015aaa0 2023-10-16T12:53:17.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e8044640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e006b810 con 0x7f03e015ae80 2023-10-16T12:53:17.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e8044640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e00707e0 con 0x7f03e00ce390 2023-10-16T12:53:17.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e55df640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03e015ae80 0x7f03e015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e55df640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03e015ae80 0x7f03e015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39748/0 (socket says 172.21.15.60:39748) 2023-10-16T12:53:17.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e55df640 1 -- 172.21.15.60:0/1954486335 learned_addr learned my addr 172.21.15.60:0/1954486335 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:17.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.460+0000 7f03e65e1640 1 --2- 172.21.15.60:0/1954486335 >> v2:172.21.15.144:3300/0 conn(0x7f03e00ce390 0x7f03e0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.461+0000 7f03e55df640 1 -- 172.21.15.60:0/1954486335 >> v2:172.21.15.144:3300/0 conn(0x7f03e00ce390 msgr2=0x7f03e0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:17.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.461+0000 7f03e55df640 1 --2- 172.21.15.60:0/1954486335 >> v2:172.21.15.144:3300/0 conn(0x7f03e00ce390 0x7f03e0070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.461+0000 7f03e55df640 1 -- 172.21.15.60:0/1954486335 >> v1:172.21.15.144:6789/0 conn(0x7f03e015aaa0 legacy=0x7f03e00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:17.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.461+0000 7f03e55df640 1 -- 172.21.15.60:0/1954486335 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e015ef90 con 0x7f03e015ae80 2023-10-16T12:53:17.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.462+0000 7f03e55df640 1 --2- 172.21.15.60:0/1954486335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015ae80 0x7f03e015e820 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03e00c2490 tx=0x7f03d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1210162473fb683b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.462+0000 7f03e4dde640 1 -- 172.21.15.60:0/1954486335 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0031870 con 0x7f03e015ae80 2023-10-16T12:53:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.462+0000 7f03e4dde640 1 -- 172.21.15.60:0/1954486335 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d00319d0 con 0x7f03e015ae80 2023-10-16T12:53:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.462+0000 7f03e4dde640 1 -- 172.21.15.60:0/1954486335 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0031d60 con 0x7f03e015ae80 2023-10-16T12:53:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 -- 172.21.15.60:0/1954486335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015ae80 msgr2=0x7f03e015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 --2- 172.21.15.60:0/1954486335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015ae80 0x7f03e015e820 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03e00c2490 tx=0x7f03d002b910 comp rx=0 tx=0).stop 2023-10-16T12:53:17.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 -- 172.21.15.60:0/1954486335 shutdown_connections 2023-10-16T12:53:17.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 --2- 172.21.15.60:0/1954486335 >> v2:172.21.15.144:3300/0 conn(0x7f03e00ce390 0x7f03e0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 --2- 172.21.15.60:0/1954486335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015ae80 0x7f03e015e820 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 -- 172.21.15.60:0/1954486335 >> 172.21.15.60:0/1954486335 conn(0x7f03e00cb410 msgr2=0x7f03e00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:17.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 -- 172.21.15.60:0/1954486335 shutdown_connections 2023-10-16T12:53:17.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.463+0000 7f03e8044640 1 -- 172.21.15.60:0/1954486335 wait complete. 2023-10-16T12:53:17.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.465+0000 7f03e8044640 1 Processor -- start 2023-10-16T12:53:17.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.465+0000 7f03e8044640 1 -- start start 2023-10-16T12:53:17.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e8044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03e00ce390 0x7f03e0151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e8044640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 0x7f03e00c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e8044640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e015ae80 0x7f03e006cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e8044640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00c9110 con 0x7f03e015aaa0 2023-10-16T12:53:17.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e8044640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00c93f0 con 0x7f03e015ae80 2023-10-16T12:53:17.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e8044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00c96d0 con 0x7f03e00ce390 2023-10-16T12:53:17.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e55df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 0x7f03e00c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e55df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 0x7f03e00c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39758/0 (socket says 172.21.15.60:39758) 2023-10-16T12:53:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.466+0000 7f03e55df640 1 -- 172.21.15.60:0/3327278555 learned_addr learned my addr 172.21.15.60:0/3327278555 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e5de0640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03e00ce390 0x7f03e0151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e65e1640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e015ae80 0x7f03e006cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e55df640 1 -- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03e00ce390 msgr2=0x7f03e0151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e55df640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03e00ce390 0x7f03e0151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e55df640 1 -- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e015ae80 msgr2=0x7f03e006cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e55df640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e015ae80 0x7f03e006cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e55df640 1 -- 172.21.15.60:0/3327278555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00094d0 con 0x7f03e015aaa0 2023-10-16T12:53:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e65e1640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e015ae80 0x7f03e006cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03e55df640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 0x7f03e00c9ed0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f03d00090b0 tx=0x7f03d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.467+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d003adf0 con 0x7f03e015aaa0 2023-10-16T12:53:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.468+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d00443f0 con 0x7f03e015aaa0 2023-10-16T12:53:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.468+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d00447a0 con 0x7f03e015aaa0 2023-10-16T12:53:17.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.468+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e00c99b0 con 0x7f03e015aaa0 2023-10-16T12:53:17.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.468+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e0151740 con 0x7f03e015aaa0 2023-10-16T12:53:17.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.469+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e00c9ba0 con 0x7f03e015aaa0 2023-10-16T12:53:17.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.470+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03d004d450 con 0x7f03e015aaa0 2023-10-16T12:53:17.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.471+0000 7f03d67fc640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f03c406af90 0x7f03c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.471+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f03d002d040 con 0x7f03e015aaa0 2023-10-16T12:53:17.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.472+0000 7f03e5de0640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f03c406af90 0x7f03c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.473+0000 7f03e5de0640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f03c406af90 0x7f03c406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03dc007fd0 tx=0x7f03dc009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:17.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.476+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03d0076de0 con 0x7f03e015aaa0 2023-10-16T12:53:17.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.673+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03e00c9250 con 0x7f03e015aaa0 2023-10-16T12:53:17.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.675+0000 7f03d67fc640 1 -- 172.21.15.60:0/3327278555 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14171 (secure 0 0 0) 0x7f03e00c9250 con 0x7f03e015aaa0 2023-10-16T12:53:17.678 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:17.678 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:14.867906+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6832/2146889084":"2023-10-17T12:53:12.179853+0000","172.21.15.144:6833/3420535028":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/2503609014":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/1605553329":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3076854692":"2023-10-17T12:53:08.068261+0000","172.21.15.60:6833/2146889084":"2023-10-17T12:53:12.179853+0000","172.21.15.144:0/132154623":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/277155730":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3226844042":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/1223431809":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/2573756770":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3005566427":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/945398958":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/1880380841":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6833/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3337897861":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/4085361535":"2023-10-17T12:52:47.120136+0000","172.21.15.144:6832/3420535028":"2023-10-17T12:53:08.068261+0000","172.21.15.144:0/3615970143":"2023-10-17T12:52:47.120136+0000","172.21.15.144:0/2155057237":"2023-10-17T12:52:47.120136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f03c406af90 msgr2=0x7f03c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f03c406af90 0x7f03c406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03dc007fd0 tx=0x7f03dc009040 comp rx=0 tx=0).stop 2023-10-16T12:53:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 msgr2=0x7f03e00c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 0x7f03e00c9ed0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f03d00090b0 tx=0x7f03d00046c0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 shutdown_connections 2023-10-16T12:53:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f03c406af90 0x7f03c406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e015ae80 0x7f03e006cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03e015aaa0 0x7f03e00c9ed0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 --2- 172.21.15.60:0/3327278555 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03e00ce390 0x7f03e0151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 >> 172.21.15.60:0/3327278555 conn(0x7f03e00cb410 msgr2=0x7f03e015a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 shutdown_connections 2023-10-16T12:53:17.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.678+0000 7f03e8044640 1 -- 172.21.15.60:0/3327278555 wait complete. 2023-10-16T12:53:17.700 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:53:17.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:53:17.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.930+0000 7f1b38b97640 1 Processor -- start 2023-10-16T12:53:17.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.930+0000 7f1b38b97640 1 -- start start 2023-10-16T12:53:17.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b38b97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 0x7f1b340c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b38b97640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340cc6b0 con 0x7f1b3406b660 2023-10-16T12:53:17.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b38b97640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b341599b0 con 0x7f1b340c9690 2023-10-16T12:53:17.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b38b97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340c9b20 con 0x7f1b341428c0 2023-10-16T12:53:17.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b3259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1b3406b660 0x7f1b3406ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45616/0 (socket says 172.21.15.60:45616) 2023-10-16T12:53:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b3259c640 1 -- 172.21.15.60:0/2568874813 learned_addr learned my addr 172.21.15.60:0/2568874813 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b31d9b640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 0x7f1b340c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460604159 0 0) 0x7f1b340cc6b0 con 0x7f1b3406b660 2023-10-16T12:53:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.932+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c003520 con 0x7f1b3406b660 2023-10-16T12:53:17.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2937867764 0 0) 0x7f1b1c003520 con 0x7f1b3406b660 2023-10-16T12:53:17.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 msgr2=0x7f1b340c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:17.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 0x7f1b340c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 >> v1:172.21.15.144:6789/0 conn(0x7f1b340c9690 legacy=0x7f1b340ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3406c290 con 0x7f1b3406b660 2023-10-16T12:53:17.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1b24001670 con 0x7f1b3406b660 2023-10-16T12:53:17.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 >> v1:172.21.15.60:6789/0 conn(0x7f1b3406b660 legacy=0x7f1b3406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b1c0039a0 0x7f1b1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 0x7f1b1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340c9b20 con 0x7f1b1c0039a0 2023-10-16T12:53:17.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c00ba30 con 0x7f1b1c0047c0 2023-10-16T12:53:17.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c00de00 con 0x7f1b341428c0 2023-10-16T12:53:17.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.933+0000 7f1b3259c640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b1c0039a0 0x7f1b1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b31d9b640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b32d9d640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 0x7f1b1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b3259c640 1 -- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 msgr2=0x7f1b1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:17.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b3259c640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b341428c0 0x7f1b1c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b3259c640 1 -- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0047c0 msgr2=0x7f1b1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:17.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b3259c640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.934+0000 7f1b3259c640 1 -- 172.21.15.60:0/2568874813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c003520 con 0x7f1b1c0039a0 2023-10-16T12:53:17.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.935+0000 7f1b3259c640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b1c0039a0 0x7f1b1c003dc0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c210 tx=0x7f1b1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad71a3d549e7f3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:17.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24005ad0 con 0x7f1b1c0039a0 2023-10-16T12:53:17.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24005c50 con 0x7f1b1c0039a0 2023-10-16T12:53:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b3159a640 1 -- 172.21.15.60:0/2568874813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2402c870 con 0x7f1b1c0039a0 2023-10-16T12:53:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b38b97640 1 -- 172.21.15.60:0/2568874813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b1c0039a0 msgr2=0x7f1b1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b38b97640 1 --2- 172.21.15.60:0/2568874813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b1c0039a0 0x7f1b1c003dc0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c210 tx=0x7f1b1c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b32d9d640 1 -- 172.21.15.60:0/2568874813 reap_dead start 2023-10-16T12:53:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b38b97640 1 -- 172.21.15.60:0/2568874813 shutdown_connections 2023-10-16T12:53:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.936+0000 7f1b38b97640 1 -- 172.21.15.60:0/2568874813 >> 172.21.15.60:0/2568874813 conn(0x7f1b340ce9b0 msgr2=0x7f1b340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:17.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.937+0000 7f1b38b97640 1 -- 172.21.15.60:0/2568874813 shutdown_connections 2023-10-16T12:53:17.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.937+0000 7f1b38b97640 1 -- 172.21.15.60:0/2568874813 wait complete. 2023-10-16T12:53:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.938+0000 7f1b38b97640 1 Processor -- start 2023-10-16T12:53:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 -- start start 2023-10-16T12:53:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0039a0 0x7f1b340c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b340b1fe0 0x7f1b340b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 0x7f1b3415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b341568f0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3415ac30 con 0x7f1b1c0039a0 2023-10-16T12:53:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.939+0000 7f1b38b97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3415af10 con 0x7f1b340b1fe0 2023-10-16T12:53:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 0x7f1b3415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 0x7f1b3415be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39768/0 (socket says 172.21.15.60:39768) 2023-10-16T12:53:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 -- 172.21.15.60:0/3544082790 learned_addr learned my addr 172.21.15.60:0/3544082790 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:17.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b3259c640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0039a0 0x7f1b340c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b31d9b640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b340b1fe0 0x7f1b340b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 -- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b340b1fe0 msgr2=0x7f1b340b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b340b1fe0 0x7f1b340b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 -- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0039a0 msgr2=0x7f1b340c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:17.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0039a0 0x7f1b340c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:17.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.940+0000 7f1b32d9d640 1 -- 172.21.15.60:0/3544082790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3415b1f0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.941+0000 7f1b32d9d640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 0x7f1b3415be70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f1b3406d980 tx=0x7f1b2c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:17.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.941+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c003b70 con 0x7f1b3415b9f0 2023-10-16T12:53:17.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.941+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c003cd0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.941+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3415b4d0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.941+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0052b0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.941+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b340b2cd0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.942+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3404e160 con 0x7f1b3415b9f0 2023-10-16T12:53:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.943+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1b2c005410 con 0x7f1b3415b9f0 2023-10-16T12:53:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.944+0000 7f1b22ffd640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1b1406afc0 0x7f1b1406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:17.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.944+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f1b2c0893a0 con 0x7f1b3415b9f0 2023-10-16T12:53:17.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.945+0000 7f1b3259c640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1b1406afc0 0x7f1b1406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:17.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.946+0000 7f1b3259c640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1b1406afc0 0x7f1b1406d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1b24034000 tx=0x7f1b24009030 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:17.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:17.949+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b2c054160 con 0x7f1b3415b9f0 2023-10-16T12:53:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.145+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b3406ac60 con 0x7f1b3415b9f0 2023-10-16T12:53:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.146+0000 7f1b22ffd640 1 -- 172.21.15.60:0/3544082790 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b2c057780 con 0x7f1b3415b9f0 2023-10-16T12:53:18.149 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:53:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1b1406afc0 msgr2=0x7f1b1406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1b1406afc0 0x7f1b1406d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1b24034000 tx=0x7f1b24009030 comp rx=0 tx=0).stop 2023-10-16T12:53:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 msgr2=0x7f1b3415be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 0x7f1b3415be70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f1b3406d980 tx=0x7f1b2c00dc00 comp rx=0 tx=0).stop 2023-10-16T12:53:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 shutdown_connections 2023-10-16T12:53:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f1b1406afc0 0x7f1b1406d480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b340b1fe0 0x7f1b340b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b3415b9f0 0x7f1b3415be70 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 --2- 172.21.15.60:0/3544082790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b1c0039a0 0x7f1b340c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 >> 172.21.15.60:0/3544082790 conn(0x7f1b340ce9b0 msgr2=0x7f1b340ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 shutdown_connections 2023-10-16T12:53:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.149+0000 7f1b38b97640 1 -- 172.21.15.60:0/3544082790 wait complete. 2023-10-16T12:53:18.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T12:53:18.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.390+0000 7f680ff89640 1 Processor -- start 2023-10-16T12:53:18.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.390+0000 7f680ff89640 1 -- start start 2023-10-16T12:53:18.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680ff89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f680815ae80 0x7f680815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680ff89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68080ce390 0x7f6808070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680d524640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f680815ae80 0x7f680815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680d524640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f680815ae80 0x7f680815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39772/0 (socket says 172.21.15.60:39772) 2023-10-16T12:53:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680d524640 1 -- 172.21.15.60:0/2760594563 learned_addr learned my addr 172.21.15.60:0/2760594563 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680ff89640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680815ed60 con 0x7f680815aaa0 2023-10-16T12:53:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680806b810 con 0x7f680815ae80 2023-10-16T12:53:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.392+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68080707e0 con 0x7f68080ce390 2023-10-16T12:53:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.393+0000 7f680e526640 1 --2- 172.21.15.60:0/2760594563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68080ce390 0x7f6808070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:18.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.393+0000 7f680d524640 1 -- 172.21.15.60:0/2760594563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68080ce390 msgr2=0x7f6808070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:18.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.393+0000 7f680d524640 1 --2- 172.21.15.60:0/2760594563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68080ce390 0x7f6808070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.393+0000 7f680d524640 1 -- 172.21.15.60:0/2760594563 >> v1:172.21.15.144:6789/0 conn(0x7f680815aaa0 legacy=0x7f68080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:18.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.393+0000 7f680d524640 1 -- 172.21.15.60:0/2760594563 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680815ef90 con 0x7f680815ae80 2023-10-16T12:53:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.394+0000 7f680d524640 1 --2- 172.21.15.60:0/2760594563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680815ae80 0x7f680815e820 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f68080c59c0 tx=0x7f67f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c10cb8bf521775 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.394+0000 7f680cd23640 1 -- 172.21.15.60:0/2760594563 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8031830 con 0x7f680815ae80 2023-10-16T12:53:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.395+0000 7f680cd23640 1 -- 172.21.15.60:0/2760594563 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f8031990 con 0x7f680815ae80 2023-10-16T12:53:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.395+0000 7f680cd23640 1 -- 172.21.15.60:0/2760594563 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8031d20 con 0x7f680815ae80 2023-10-16T12:53:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.395+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680815ae80 msgr2=0x7f680815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.395+0000 7f680ff89640 1 --2- 172.21.15.60:0/2760594563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680815ae80 0x7f680815e820 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f68080c59c0 tx=0x7f67f802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.396+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 shutdown_connections 2023-10-16T12:53:18.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.396+0000 7f680ff89640 1 --2- 172.21.15.60:0/2760594563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f68080ce390 0x7f6808070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.396+0000 7f680ff89640 1 --2- 172.21.15.60:0/2760594563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680815ae80 0x7f680815e820 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.396+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 >> 172.21.15.60:0/2760594563 conn(0x7f68080cb410 msgr2=0x7f68080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:18.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.396+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 shutdown_connections 2023-10-16T12:53:18.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.396+0000 7f680ff89640 1 -- 172.21.15.60:0/2760594563 wait complete. 2023-10-16T12:53:18.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.398+0000 7f680ff89640 1 Processor -- start 2023-10-16T12:53:18.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.398+0000 7f680ff89640 1 -- start start 2023-10-16T12:53:18.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680ff89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 0x7f68081f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:18.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680ff89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f680815aaa0 0x7f68080c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:18.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680ff89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f680815ae80 0x7f68080b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:18.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680ff89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6808158860 con 0x7f68080ce390 2023-10-16T12:53:18.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680ff89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6808159680 con 0x7f680815aaa0 2023-10-16T12:53:18.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680ff89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68081f2900 con 0x7f680815ae80 2023-10-16T12:53:18.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680dd25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 0x7f68081f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:18.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680dd25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 0x7f68081f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39784/0 (socket says 172.21.15.60:39784) 2023-10-16T12:53:18.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680dd25640 1 -- 172.21.15.60:0/1533864482 learned_addr learned my addr 172.21.15.60:0/1533864482 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680d524640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f680815aaa0 0x7f68080c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.400+0000 7f680e526640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f680815ae80 0x7f68080b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680dd25640 1 -- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f680815ae80 msgr2=0x7f68080b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680dd25640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f680815ae80 0x7f68080b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680dd25640 1 -- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f680815aaa0 msgr2=0x7f68080c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680dd25640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f680815aaa0 0x7f68080c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680dd25640 1 -- 172.21.15.60:0/1533864482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f80094d0 con 0x7f68080ce390 2023-10-16T12:53:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680dd25640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 0x7f68081f23c0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f680400c7c0 tx=0x7f680400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680400aa50 con 0x7f68080ce390 2023-10-16T12:53:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680400abb0 con 0x7f68080ce390 2023-10-16T12:53:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804012420 con 0x7f68080ce390 2023-10-16T12:53:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.401+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68080c5040 con 0x7f68080ce390 2023-10-16T12:53:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.402+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6808158d50 con 0x7f68080ce390 2023-10-16T12:53:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.403+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680804e160 con 0x7f68080ce390 2023-10-16T12:53:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.404+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6804012580 con 0x7f68080ce390 2023-10-16T12:53:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.404+0000 7f67fe7fc640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f67ec06afc0 0x7f67ec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.405+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f68040896d0 con 0x7f68080ce390 2023-10-16T12:53:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.405+0000 7f680d524640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f67ec06afc0 0x7f67ec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.406+0000 7f680d524640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f67ec06afc0 0x7f67ec06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f67f8036000 tx=0x7f67f803c040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:18.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.412+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6804054630 con 0x7f68080ce390 2023-10-16T12:53:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:18.681+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f68080cdc60 con 0x7f68080ce390 2023-10-16T12:53:19.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.212+0000 7f67fe7fc640 1 -- 172.21.15.60:0/1533864482 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v759) v1 ==== 163+0+0 (secure 0 0 0) 0x7f6804057c50 con 0x7f68080ce390 2023-10-16T12:53:19.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.215+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f67ec06afc0 msgr2=0x7f67ec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.215+0000 7f680ff89640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f67ec06afc0 0x7f67ec06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f67f8036000 tx=0x7f67f803c040 comp rx=0 tx=0).stop 2023-10-16T12:53:19.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.215+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 msgr2=0x7f68081f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.215+0000 7f680ff89640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 0x7f68081f23c0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f680400c7c0 tx=0x7f680400cc80 comp rx=0 tx=0).stop 2023-10-16T12:53:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 shutdown_connections 2023-10-16T12:53:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f67ec06afc0 0x7f67ec06d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f680815ae80 0x7f68080b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f680815aaa0 0x7f68080c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 --2- 172.21.15.60:0/1533864482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68080ce390 0x7f68081f23c0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 >> 172.21.15.60:0/1533864482 conn(0x7f68080cb410 msgr2=0x7f680806f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 shutdown_connections 2023-10-16T12:53:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.216+0000 7f680ff89640 1 -- 172.21.15.60:0/1533864482 wait complete. 2023-10-16T12:53:19.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:53:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.469+0000 7fc4cfc7c640 1 Processor -- start 2023-10-16T12:53:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.469+0000 7fc4cfc7c640 1 -- start start 2023-10-16T12:53:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.470+0000 7fc4cfc7c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4c8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.470+0000 7fc4cfc7c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c80cb030 con 0x7fc4c80c9ca0 2023-10-16T12:53:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.470+0000 7fc4cfc7c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c80c91c0 con 0x7fc4c80caac0 2023-10-16T12:53:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.470+0000 7fc4cfc7c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80c94a0 con 0x7fc4c815ad70 2023-10-16T12:53:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.471+0000 7fc4cd217640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc4c80c9ca0 0x7fc4c80ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51928/0 (socket says 172.21.15.60:51928) 2023-10-16T12:53:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.471+0000 7fc4cd217640 1 -- 172.21.15.60:0/571241668 learned_addr learned my addr 172.21.15.60:0/571241668 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.471+0000 7fc4cda18640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4c8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.471+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207702491 0 0) 0x7fc4c80cb030 con 0x7fc4c80c9ca0 2023-10-16T12:53:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b4003520 con 0x7fc4c80c9ca0 2023-10-16T12:53:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 330633993 0 0) 0x7fc4b4003520 con 0x7fc4c80c9ca0 2023-10-16T12:53:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 msgr2=0x7fc4c8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4c8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 >> v1:172.21.15.144:6789/0 conn(0x7fc4c80caac0 legacy=0x7fc4c80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c80c9780 con 0x7fc4c80c9ca0 2023-10-16T12:53:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc4b8001670 con 0x7fc4c80c9ca0 2023-10-16T12:53:19.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 >> v1:172.21.15.60:6789/0 conn(0x7fc4c80c9ca0 legacy=0x7fc4c80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40039a0 0x7fc4b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b4004d00 0x7fc4b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80c94a0 con 0x7fc4b40039a0 2023-10-16T12:53:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b400ba60 con 0x7fc4b4004d00 2023-10-16T12:53:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b400bc20 con 0x7fc4c815ad70 2023-10-16T12:53:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.472+0000 7fc4cda18640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40039a0 0x7fc4b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cd217640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cda18640 1 -- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 msgr2=0x7fc4b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cda18640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4ce219640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b4004d00 0x7fc4b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cda18640 1 -- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b4004d00 msgr2=0x7fc4b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cda18640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b4004d00 0x7fc4b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cda18640 1 -- 172.21.15.60:0/571241668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4003520 con 0x7fc4b40039a0 2023-10-16T12:53:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4cd217640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4c815ad70 0x7fc4b40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.473+0000 7fc4ce219640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4b4004d00 0x7fc4b4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.474+0000 7fc4cda18640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40039a0 0x7fc4b4003dc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009f90 tx=0x7fc4b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8201cd35808a02dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.474+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c402cbd0 con 0x7fc4b40039a0 2023-10-16T12:53:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.474+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c400f040 con 0x7fc4b40039a0 2023-10-16T12:53:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.474+0000 7fc4cca16640 1 -- 172.21.15.60:0/571241668 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4034440 con 0x7fc4b40039a0 2023-10-16T12:53:19.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/571241668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40039a0 msgr2=0x7fc4b4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/571241668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4b40039a0 0x7fc4b4003dc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009f90 tx=0x7fc4b40057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4ce219640 1 -- 172.21.15.60:0/571241668 reap_dead start 2023-10-16T12:53:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/571241668 shutdown_connections 2023-10-16T12:53:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/571241668 >> 172.21.15.60:0/571241668 conn(0x7fc4c80ce9b0 msgr2=0x7fc4c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/571241668 shutdown_connections 2023-10-16T12:53:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.475+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/571241668 wait complete. 2023-10-16T12:53:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.477+0000 7fc4cfc7c640 1 Processor -- start 2023-10-16T12:53:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.477+0000 7fc4cfc7c640 1 -- start start 2023-10-16T12:53:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cfc7c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40039a0 0x7fc4c80c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cfc7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 0x7fc4c80c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cfc7c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c8151e10 0x7fc4c80c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cfc7c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80c6180 con 0x7fc4c80c7220 2023-10-16T12:53:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cfc7c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80c6460 con 0x7fc4c8151e10 2023-10-16T12:53:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cfc7c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80c6740 con 0x7fc4b40039a0 2023-10-16T12:53:19.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cd217640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 0x7fc4c80c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cd217640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 0x7fc4c80c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46526/0 (socket says 172.21.15.60:46526) 2023-10-16T12:53:19.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.478+0000 7fc4cd217640 1 -- 172.21.15.60:0/1669666428 learned_addr learned my addr 172.21.15.60:0/1669666428 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:19.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4ce219640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c8151e10 0x7fc4c80c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cda18640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40039a0 0x7fc4c80c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cd217640 1 -- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40039a0 msgr2=0x7fc4c80c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cd217640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40039a0 0x7fc4c80c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cd217640 1 -- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c8151e10 msgr2=0x7fc4c80c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cd217640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c8151e10 0x7fc4c80c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cd217640 1 -- 172.21.15.60:0/1669666428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c80c6a20 con 0x7fc4c80c7220 2023-10-16T12:53:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4ce219640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c8151e10 0x7fc4c80c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cd217640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 0x7fc4c80c8670 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800d8e0 tx=0x7fc4b800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b80074e0 con 0x7fc4c80c7220 2023-10-16T12:53:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b8007640 con 0x7fc4c80c7220 2023-10-16T12:53:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b80049e0 con 0x7fc4c80c7220 2023-10-16T12:53:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c80c6d00 con 0x7fc4c80c7220 2023-10-16T12:53:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.479+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c80c4a40 con 0x7fc4c80c7220 2023-10-16T12:53:19.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.481+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c80c6e90 con 0x7fc4c80c7220 2023-10-16T12:53:19.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.482+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc4b8004b40 con 0x7fc4c80c7220 2023-10-16T12:53:19.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.482+0000 7fc4be7fc640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc4ac06af90 0x7fc4ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:19.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.483+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fc4b8089350 con 0x7fc4c80c7220 2023-10-16T12:53:19.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.483+0000 7fc4cda18640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc4ac06af90 0x7fc4ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:19.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.484+0000 7fc4cda18640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc4ac06af90 0x7fc4ac06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4036000 tx=0x7fc4c402e000 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:19.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.487+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4b8057ba0 con 0x7fc4c80c7220 2023-10-16T12:53:19.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.684+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc4c804e160 con 0x7fc4c80c7220 2023-10-16T12:53:19.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.691+0000 7fc4be7fc640 1 -- 172.21.15.60:0/1669666428 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc4b8057730 con 0x7fc4c80c7220 2023-10-16T12:53:19.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.693+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc4ac06af90 msgr2=0x7fc4ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.693+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc4ac06af90 0x7fc4ac06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4036000 tx=0x7fc4c402e000 comp rx=0 tx=0).stop 2023-10-16T12:53:19.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.693+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 msgr2=0x7fc4c80c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:19.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.693+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 0x7fc4c80c8670 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800d8e0 tx=0x7fc4b800dda0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 shutdown_connections 2023-10-16T12:53:19.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc4ac06af90 0x7fc4ac06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4c8151e10 0x7fc4c80c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4c80c7220 0x7fc4c80c8670 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 --2- 172.21.15.60:0/1669666428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4b40039a0 0x7fc4c80c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:19.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 >> 172.21.15.60:0/1669666428 conn(0x7fc4c80ce9b0 msgr2=0x7fc4c80b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:19.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 shutdown_connections 2023-10-16T12:53:19.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:19.694+0000 7fc4cfc7c640 1 -- 172.21.15.60:0/1669666428 wait complete. 2023-10-16T12:53:19.715 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:53:19.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:53:19.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:53:19.944 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:53:19.944 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:19.945 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:20.008 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:53:20.008 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:53:20.009 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:20.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:53:20.068 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:20.068 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:20.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:53:20.131 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:53:20.131 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:20.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:53:20.192 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:53:20.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:20.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.427+0000 7fa3b5be2640 1 Processor -- start 2023-10-16T12:53:20.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.427+0000 7fa3b5be2640 1 -- start start 2023-10-16T12:53:20.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.428+0000 7fa3b5be2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3b00ad7a0 0x7fa3b0070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.428+0000 7fa3b5be2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b0070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.428+0000 7fa3b5be2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b00ae1a0 con 0x7fa3b0160c80 2023-10-16T12:53:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.428+0000 7fa3b5be2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b00ae480 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.428+0000 7fa3b5be2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00ae760 con 0x7fa3b00adc30 2023-10-16T12:53:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.429+0000 7fa3affff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3b00ad7a0 0x7fa3b0070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.429+0000 7fa3b53e1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b0070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.429+0000 7fa3affff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3b00ad7a0 0x7fa3b0070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:52988/0 (socket says 172.21.15.60:52988) 2023-10-16T12:53:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.429+0000 7fa3affff640 1 -- 172.21.15.60:0/3597665799 learned_addr learned my addr 172.21.15.60:0/3597665799 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:20.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.430+0000 7fa3affff640 1 -- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 msgr2=0x7fa3b0070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:20.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.430+0000 7fa3affff640 1 --2- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b0070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.430+0000 7fa3affff640 1 -- 172.21.15.60:0/3597665799 >> v1:172.21.15.144:6789/0 conn(0x7fa3b0160c80 legacy=0x7fa3b0161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:20.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.430+0000 7fa3affff640 1 -- 172.21.15.60:0/3597665799 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b00aea40 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.430+0000 7fa3b53e1640 1 --2- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b0070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:53:20.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.431+0000 7fa3affff640 1 --2- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b0070240 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009870 tx=0x7fa3a402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b16065fd9a1b7a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:20.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.432+0000 7fa3af7fe640 1 -- 172.21.15.60:0/3597665799 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4030840 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.432+0000 7fa3af7fe640 1 -- 172.21.15.60:0/3597665799 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a40309a0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.432+0000 7fa3af7fe640 1 -- 172.21.15.60:0/3597665799 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4030d30 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.432+0000 7fa3b5be2640 1 -- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b00ad7a0 msgr2=0x7fa3b0070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:20.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.432+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b0070240 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009870 tx=0x7fa3a402b980 comp rx=0 tx=0).stop 2023-10-16T12:53:20.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.433+0000 7fa3b5be2640 1 -- 172.21.15.60:0/3597665799 shutdown_connections 2023-10-16T12:53:20.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.433+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b0070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.433+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/3597665799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b0070240 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.433+0000 7fa3b5be2640 1 -- 172.21.15.60:0/3597665799 >> 172.21.15.60:0/3597665799 conn(0x7fa3b00cc230 msgr2=0x7fa3b00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:20.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.433+0000 7fa3b5be2640 1 -- 172.21.15.60:0/3597665799 shutdown_connections 2023-10-16T12:53:20.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.433+0000 7fa3b5be2640 1 -- 172.21.15.60:0/3597665799 wait complete. 2023-10-16T12:53:20.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.435+0000 7fa3b5be2640 1 Processor -- start 2023-10-16T12:53:20.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.435+0000 7fa3b5be2640 1 -- start start 2023-10-16T12:53:20.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b5be2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:20.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b5be2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:20.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b5be2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b0160c80 0x7fa3b0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:20.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b5be2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b01f66b0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b5be2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b01f5930 con 0x7fa3b0160c80 2023-10-16T12:53:20.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b5be2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b0156d30 con 0x7fa3b00adc30 2023-10-16T12:53:20.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b4be0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:20.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b4be0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b01564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46534/0 (socket says 172.21.15.60:46534) 2023-10-16T12:53:20.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b4be0640 1 -- 172.21.15.60:0/2011399081 learned_addr learned my addr 172.21.15.60:0/2011399081 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:20.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3affff640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:20.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.436+0000 7fa3b53e1640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b0160c80 0x7fa3b0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:20.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b4be0640 1 -- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 msgr2=0x7fa3b015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:20.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b4be0640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b4be0640 1 -- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b0160c80 msgr2=0x7fa3b0154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:20.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b4be0640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b0160c80 0x7fa3b0154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b4be0640 1 -- 172.21.15.60:0/2011399081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8009400 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b4be0640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b01564c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000c730 tx=0x7fa3a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:20.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0006d50 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a000ce40 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0003bb0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.437+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a40094d0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.438+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b01f5eb0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.439+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b004e9b0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.440+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa3a0003d10 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.441+0000 7fa3ad7fa640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fa39c06af90 0x7fa39c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:20.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.441+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7fa3a00893b0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.441+0000 7fa3affff640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fa39c06af90 0x7fa39c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:20.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.443+0000 7fa3affff640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fa39c06af90 0x7fa39c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4036000 tx=0x7fa3a4008030 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:20.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.446+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3a0057da0 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.681+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3b00b5730 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.682+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/2011399081 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fa3a005ce60 con 0x7fa3b00ad7a0 2023-10-16T12:53:20.685 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:20.685 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":64,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6092,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/2457053502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":2457053502},{"type":"v1","addr":"172.21.15.144:6836","nonce":2457053502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4129191374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4129191374},{"type":"v1","addr":"172.21.15.60:6835","nonce":4129191374}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6098,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/4196545990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":4196545990},{"type":"v1","addr":"172.21.15.144:6837","nonce":4196545990}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":8283,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2756700584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2756700584},{"type":"v1","addr":"172.21.15.60:6833","nonce":2756700584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64}],"filesystems":[]} 2023-10-16T12:53:20.686 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 64 2023-10-16T12:53:20.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fa39c06af90 msgr2=0x7fa39c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:20.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fa39c06af90 0x7fa39c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4036000 tx=0x7fa3a4008030 comp rx=0 tx=0).stop 2023-10-16T12:53:20.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 msgr2=0x7fa3b01564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:20.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b01564c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000c730 tx=0x7fa3a000cbf0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 shutdown_connections 2023-10-16T12:53:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fa39c06af90 0x7fa39c06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3b00adc30 0x7fa3b015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b0160c80 0x7fa3b0154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 --2- 172.21.15.60:0/2011399081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00ad7a0 0x7fa3b01564c0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 >> 172.21.15.60:0/2011399081 conn(0x7fa3b00cc230 msgr2=0x7fa3b00c37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:20.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 shutdown_connections 2023-10-16T12:53:20.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.685+0000 7fa3b5be2640 1 -- 172.21.15.60:0/2011399081 wait complete. 2023-10-16T12:53:20.709 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:53:20.709 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:53:20.709 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:53:20.710 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:53:20.710 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:53:20.711 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:53:20.711 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:53:20.711 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:53:20.711 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:53:20.712 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:53:20.712 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:53:20.713 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:53:20.713 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:53:20.713 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:53:20.714 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:53:20.714 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:53:20.727 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:53:20.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:53:20.729 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:53:20.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:53:20.735 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:53:20.735 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:53:20.736 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:53:20.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:53:20.777 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:53:20.778 DEBUG:teuthology.parallel:result is None 2023-10-16T12:53:20.784 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:53:20.785 DEBUG:teuthology.parallel:result is None 2023-10-16T12:53:20.785 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:53:20.786 DEBUG:teuthology.parallel:result is None 2023-10-16T12:53:20.791 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:53:20.792 DEBUG:teuthology.parallel:result is None 2023-10-16T12:53:20.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:53:20.943 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:53:20.944 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:53:20.952 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:53:20.953 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:53:20.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.996+0000 7f51308af640 1 Processor -- start 2023-10-16T12:53:20.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.996+0000 7f51308af640 1 -- start start 2023-10-16T12:53:21.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f51308af640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f512c15ad70 0x7f512c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:21.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f51308af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:21.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f51308af640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c0cb030 con 0x7f512c0ca3b0 2023-10-16T12:53:21.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f51308af640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c0c91c0 con 0x7f512c15ad70 2023-10-16T12:53:21.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f51308af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0c94a0 con 0x7f512c0c9ca0 2023-10-16T12:53:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f512a59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f512c15ad70 0x7f512c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f512a59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f512c15ad70 0x7f512c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46546/0 (socket says 172.21.15.60:46546) 2023-10-16T12:53:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f512a59c640 1 -- 172.21.15.60:0/111876612 learned_addr learned my addr 172.21.15.60:0/111876612 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f512ad9d640 1 --1- 172.21.15.60:0/111876612 >> v1:172.21.15.60:6789/0 conn(0x7f512c0ca3b0 0x7f512c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51952/0 (socket says 172.21.15.60:51952) 2023-10-16T12:53:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.998+0000 7f5123fff640 1 --2- 172.21.15.60:0/111876612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:21.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.999+0000 7f512a59c640 1 -- 172.21.15.60:0/111876612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 msgr2=0x7f512c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:21.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.999+0000 7f512a59c640 1 --2- 172.21.15.60:0/111876612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.999+0000 7f512a59c640 1 -- 172.21.15.60:0/111876612 >> v1:172.21.15.60:6789/0 conn(0x7f512c0ca3b0 legacy=0x7f512c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.999+0000 7f512a59c640 1 -- 172.21.15.60:0/111876612 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c0c9780 con 0x7f512c15ad70 2023-10-16T12:53:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:20.999+0000 7f5129d9b640 1 -- 172.21.15.60:0/111876612 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679205988 0 0) 0x7f512c0cb030 con 0x7f512c0ca3b0 2023-10-16T12:53:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.000+0000 7f512a59c640 1 --2- 172.21.15.60:0/111876612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c069da0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f511c009870 tx=0x7f511c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3db11e4977ddf3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.000+0000 7f5129d9b640 1 -- 172.21.15.60:0/111876612 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c030840 con 0x7f512c15ad70 2023-10-16T12:53:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.000+0000 7f5129d9b640 1 -- 172.21.15.60:0/111876612 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511c0309a0 con 0x7f512c15ad70 2023-10-16T12:53:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.000+0000 7f5129d9b640 1 -- 172.21.15.60:0/111876612 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c030d30 con 0x7f512c15ad70 2023-10-16T12:53:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 -- 172.21.15.60:0/111876612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 msgr2=0x7f512c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 --2- 172.21.15.60:0/111876612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c069da0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f511c009870 tx=0x7f511c02b980 comp rx=0 tx=0).stop 2023-10-16T12:53:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 -- 172.21.15.60:0/111876612 shutdown_connections 2023-10-16T12:53:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 --2- 172.21.15.60:0/111876612 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 --2- 172.21.15.60:0/111876612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c069da0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 -- 172.21.15.60:0/111876612 >> 172.21.15.60:0/111876612 conn(0x7f512c0ce9b0 msgr2=0x7f512c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:21.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 -- 172.21.15.60:0/111876612 shutdown_connections 2023-10-16T12:53:21.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.001+0000 7f51308af640 1 -- 172.21.15.60:0/111876612 wait complete. 2023-10-16T12:53:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.003+0000 7f51308af640 1 Processor -- start 2023-10-16T12:53:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.003+0000 7f51308af640 1 -- start start 2023-10-16T12:53:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f51308af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f51308af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f512c0ca3b0 0x7f512c150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f51308af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c1f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f51308af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0cca00 con 0x7f512c15ad70 2023-10-16T12:53:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f51308af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0ccce0 con 0x7f512c0ca3b0 2023-10-16T12:53:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f51308af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0ccfc0 con 0x7f512c0c9ca0 2023-10-16T12:53:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f512ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c1f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f512ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c1f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46554/0 (socket says 172.21.15.60:46554) 2023-10-16T12:53:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f512ad9d640 1 -- 172.21.15.60:0/4177152225 learned_addr learned my addr 172.21.15.60:0/4177152225 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f5123fff640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f512c0ca3b0 0x7f512c150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f512a59c640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.004+0000 7f512ad9d640 1 -- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 msgr2=0x7f512c157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f512ad9d640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f512ad9d640 1 -- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f512c0ca3b0 msgr2=0x7f512c150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f512ad9d640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f512c0ca3b0 0x7f512c150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f512ad9d640 1 -- 172.21.15.60:0/4177152225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c0094d0 con 0x7f512c15ad70 2023-10-16T12:53:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f512ad9d640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c1f2ad0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f512400c7a0 tx=0x7f512400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512400a640 con 0x7f512c15ad70 2023-10-16T12:53:21.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512400a7a0 con 0x7f512c15ad70 2023-10-16T12:53:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512400abd0 con 0x7f512c15ad70 2023-10-16T12:53:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512c0cd2a0 con 0x7f512c15ad70 2023-10-16T12:53:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.005+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512c151280 con 0x7f512c15ad70 2023-10-16T12:53:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.007+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512c0cd7c0 con 0x7f512c15ad70 2023-10-16T12:53:21.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.008+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f512400ad30 con 0x7f512c15ad70 2023-10-16T12:53:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.008+0000 7f5122ffd640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f510c06af90 0x7f510c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.008+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5745+0+0 (secure 0 0 0) 0x7f5124088d60 con 0x7f512c15ad70 2023-10-16T12:53:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.009+0000 7f512a59c640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f510c06af90 0x7f510c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.010+0000 7f512a59c640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f510c06af90 0x7f510c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f511c036000 tx=0x7f511c008030 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:21.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.013+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f512c0cd7c0 con 0x7f512c15ad70 2023-10-16T12:53:21.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:21.218+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f512c0cd7c0 con 0x7f512c15ad70 2023-10-16T12:53:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.249+0000 7f5122ffd640 1 -- 172.21.15.60:0/4177152225 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v139) v1 ==== 97+0+0 (secure 0 0 0) 0x7f512c0cd7c0 con 0x7f512c15ad70 2023-10-16T12:53:22.252 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:53:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.253+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f510c06af90 msgr2=0x7f510c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.253+0000 7f51308af640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f510c06af90 0x7f510c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f511c036000 tx=0x7f511c008030 comp rx=0 tx=0).stop 2023-10-16T12:53:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.253+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 msgr2=0x7f512c1f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.253+0000 7f51308af640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c1f2ad0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f512400c7a0 tx=0x7f512400cc60 comp rx=0 tx=0).stop 2023-10-16T12:53:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 shutdown_connections 2023-10-16T12:53:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f510c06af90 0x7f510c06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c15ad70 0x7f512c1f2ad0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f512c0ca3b0 0x7f512c150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 --2- 172.21.15.60:0/4177152225 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f512c0c9ca0 0x7f512c157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 >> 172.21.15.60:0/4177152225 conn(0x7f512c0ce9b0 msgr2=0x7f512c0aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 shutdown_connections 2023-10-16T12:53:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.254+0000 7f51308af640 1 -- 172.21.15.60:0/4177152225 wait complete. 2023-10-16T12:53:22.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:53:22.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.499+0000 7effbf1ff640 1 Processor -- start 2023-10-16T12:53:22.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.499+0000 7effbf1ff640 1 -- start start 2023-10-16T12:53:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.500+0000 7effbf1ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effb80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.500+0000 7effbf1ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb815b9d0 con 0x7effb806c040 2023-10-16T12:53:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.500+0000 7effbf1ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb815bcb0 con 0x7effb80ad970 2023-10-16T12:53:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.500+0000 7effbf1ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb815b7d0 con 0x7effb815b320 2023-10-16T12:53:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.500+0000 7effbe1fd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7effb806c040 0x7effb806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51954/0 (socket says 172.21.15.60:51954) 2023-10-16T12:53:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.500+0000 7effbe1fd640 1 -- 172.21.15.60:0/2267730069 learned_addr learned my addr 172.21.15.60:0/2267730069 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:22.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473801026 0 0) 0x7effb815b9d0 con 0x7effb806c040 2023-10-16T12:53:22.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa4003520 con 0x7effb806c040 2023-10-16T12:53:22.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbe9fe640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effb80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339125626 0 0) 0x7effa4003520 con 0x7effb806c040 2023-10-16T12:53:22.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 msgr2=0x7effb80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:22.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effb80cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 >> v1:172.21.15.144:6789/0 conn(0x7effb80ad970 legacy=0x7effb80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb815bf90 con 0x7effb806c040 2023-10-16T12:53:22.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7effb40019f0 con 0x7effb806c040 2023-10-16T12:53:22.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbe9fe640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effb80cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:22.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.501+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 >> v1:172.21.15.60:6789/0 conn(0x7effb806c040 legacy=0x7effb806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd1fb640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effa4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd1fb640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effa40047c0 0x7effa4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd1fb640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effa40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb815b7d0 con 0x7effa40039a0 2023-10-16T12:53:22.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400bb10 con 0x7effa40047c0 2023-10-16T12:53:22.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400bcd0 con 0x7effb815b320 2023-10-16T12:53:22.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe1fd640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effa4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe9fe640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effa40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbd9fc640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effa40047c0 0x7effa4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe1fd640 1 -- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 msgr2=0x7effa40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:22.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe1fd640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb815b320 0x7effa40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe1fd640 1 -- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effa40047c0 msgr2=0x7effa4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:22.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe1fd640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effa40047c0 0x7effa4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.502+0000 7effbe1fd640 1 -- 172.21.15.60:0/2267730069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4003520 con 0x7effa40039a0 2023-10-16T12:53:22.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.503+0000 7effbe1fd640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effa4003dc0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7effb400a760 tx=0x7effa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4347c4742c477124 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:22.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.504+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4034630 con 0x7effa40039a0 2023-10-16T12:53:22.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.504+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb402f070 con 0x7effa40039a0 2023-10-16T12:53:22.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.504+0000 7effbd1fb640 1 -- 172.21.15.60:0/2267730069 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb40349d0 con 0x7effa40039a0 2023-10-16T12:53:22.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.504+0000 7effbf1ff640 1 -- 172.21.15.60:0/2267730069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 msgr2=0x7effa4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.504+0000 7effbf1ff640 1 --2- 172.21.15.60:0/2267730069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effa4003dc0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7effb400a760 tx=0x7effa40057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.504+0000 7effbe9fe640 1 -- 172.21.15.60:0/2267730069 reap_dead start 2023-10-16T12:53:22.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.505+0000 7effbf1ff640 1 -- 172.21.15.60:0/2267730069 shutdown_connections 2023-10-16T12:53:22.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.505+0000 7effbf1ff640 1 -- 172.21.15.60:0/2267730069 >> 172.21.15.60:0/2267730069 conn(0x7effb80ce6a0 msgr2=0x7effb80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:22.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.505+0000 7effbf1ff640 1 -- 172.21.15.60:0/2267730069 shutdown_connections 2023-10-16T12:53:22.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.505+0000 7effbf1ff640 1 -- 172.21.15.60:0/2267730069 wait complete. 2023-10-16T12:53:22.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.506+0000 7effbf1ff640 1 Processor -- start 2023-10-16T12:53:22.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.506+0000 7effbf1ff640 1 -- start start 2023-10-16T12:53:22.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.507+0000 7effbf1ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effb8152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.507+0000 7effbf1ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80c21d0 0x7effb80c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.507+0000 7effbf1ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb80c6fe0 0x7effb8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.507+0000 7effbf1ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb80c3270 con 0x7effa40039a0 2023-10-16T12:53:22.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.507+0000 7effbf1ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb80c3c40 con 0x7effb80c21d0 2023-10-16T12:53:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.507+0000 7effbf1ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb80c2660 con 0x7effb80c6fe0 2023-10-16T12:53:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effb8152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effb8152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46574/0 (socket says 172.21.15.60:46574) 2023-10-16T12:53:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 -- 172.21.15.60:0/796385862 learned_addr learned my addr 172.21.15.60:0/796385862 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:22.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbd9fc640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80c21d0 0x7effb80c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe9fe640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb80c6fe0 0x7effb8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 -- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb80c6fe0 msgr2=0x7effb8157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb80c6fe0 0x7effb8157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 -- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80c21d0 msgr2=0x7effb80c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80c21d0 0x7effb80c3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe1fd640 1 -- 172.21.15.60:0/796385862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80c2940 con 0x7effa40039a0 2023-10-16T12:53:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbe9fe640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb80c6fe0 0x7effb8157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.508+0000 7effbd9fc640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80c21d0 0x7effb80c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:22.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.509+0000 7effbe1fd640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effb8152af0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7effb400a880 tx=0x7effb4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:22.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.509+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4043070 con 0x7effa40039a0 2023-10-16T12:53:22.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.509+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4034430 con 0x7effa40039a0 2023-10-16T12:53:22.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.509+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb80c2c20 con 0x7effa40039a0 2023-10-16T12:53:22.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.509+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb40347a0 con 0x7effa40039a0 2023-10-16T12:53:22.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.509+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb80c7470 con 0x7effa40039a0 2023-10-16T12:53:22.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.510+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb806b330 con 0x7effa40039a0 2023-10-16T12:53:22.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.511+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7effb4031030 con 0x7effa40039a0 2023-10-16T12:53:22.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.512+0000 7effaeffd640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff9c06af90 0x7eff9c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:22.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.512+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4857+0+0 (secure 0 0 0) 0x7effb40ab270 con 0x7effa40039a0 2023-10-16T12:53:22.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.513+0000 7effbd9fc640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff9c06af90 0x7eff9c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:22.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.514+0000 7effbd9fc640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff9c06af90 0x7eff9c06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7effb80b11b0 tx=0x7effa8009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:22.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.517+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7effb40764e0 con 0x7effa40039a0 2023-10-16T12:53:22.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.788+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7effb806b330 con 0x7effa40039a0 2023-10-16T12:53:22.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.790+0000 7effaeffd640 1 -- 172.21.15.60:0/796385862 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+5304 (secure 0 0 0) 0x7effb4032e60 con 0x7effa40039a0 2023-10-16T12:53:22.793 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:22.794 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:53:22.794 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:53:22.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.794 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:53:22.794 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:53:22.795 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.795 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.795 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:53:22.795 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:53:22.795 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:53:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:53:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:53:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:53:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:53:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:53:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:53:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:53:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:53:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:53:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:53:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:53:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:53:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:53:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:53:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:53:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:53:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:53:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:53:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:53:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:53:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:53:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:53:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:53:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:53:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:53:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:53:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:53:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:53:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:53:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:53:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:53:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:53:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:53:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:53:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:53:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:53:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:53:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:53:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:53:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:53:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:53:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:53:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:53:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:53:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:53:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:53:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:53:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:53:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:53:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.824 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:53:22.824 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:53:22.824 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.825 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:53:22.825 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:53:22.825 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:53:22.825 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:53:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:53:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:53:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:53:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:53:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:53:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:53:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:53:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:53:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:53:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:53:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:53:22.830 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:53:22.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff9c06af90 msgr2=0x7eff9c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff9c06af90 0x7eff9c06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7effb80b11b0 tx=0x7effa8009040 comp rx=0 tx=0).stop 2023-10-16T12:53:22.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 msgr2=0x7effb8152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:22.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effb8152af0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7effb400a880 tx=0x7effb4001cb0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 shutdown_connections 2023-10-16T12:53:22.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff9c06af90 0x7eff9c06d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7effb80c6fe0 0x7effb8157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effb80c21d0 0x7effb80c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 --2- 172.21.15.60:0/796385862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effa40039a0 0x7effb8152af0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:22.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 >> 172.21.15.60:0/796385862 conn(0x7effb80ce6a0 msgr2=0x7effb806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:22.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.792+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 shutdown_connections 2023-10-16T12:53:22.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:22.793+0000 7effbf1ff640 1 -- 172.21.15.60:0/796385862 wait complete. 2023-10-16T12:53:22.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:23.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.065+0000 7f69599ff640 1 Processor -- start 2023-10-16T12:53:23.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.066+0000 7f69599ff640 1 -- start start 2023-10-16T12:53:23.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.067+0000 7f69599ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69540cded0 0x7f69540b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.067+0000 7f69599ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695415d150 con 0x7f695415cd70 2023-10-16T12:53:23.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.067+0000 7f69599ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540d1180 con 0x7f69540b8480 2023-10-16T12:53:23.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.067+0000 7f69599ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69540d1460 con 0x7f69540cded0 2023-10-16T12:53:23.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.067+0000 7f69589fd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f695415cd70 0x7f69540d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51968/0 (socket says 172.21.15.60:51968) 2023-10-16T12:53:23.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.067+0000 7f69589fd640 1 -- 172.21.15.60:0/3741390324 learned_addr learned my addr 172.21.15.60:0/3741390324 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:23.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.068+0000 7f69591fe640 1 --2- 172.21.15.60:0/3741390324 >> v2:172.21.15.60:3300/0 conn(0x7f69540cded0 0x7f69540b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.068+0000 7f69537fe640 1 -- 172.21.15.60:0/3741390324 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591886858 0 0) 0x7f695415d150 con 0x7f695415cd70 2023-10-16T12:53:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.068+0000 7f69537fe640 1 -- 172.21.15.60:0/3741390324 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944003520 con 0x7f695415cd70 2023-10-16T12:53:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.068+0000 7f69591fe640 1 -- 172.21.15.60:0/3741390324 >> v1:172.21.15.144:6789/0 conn(0x7f69540b8480 legacy=0x7f69540cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.068+0000 7f69591fe640 1 -- 172.21.15.60:0/3741390324 >> v1:172.21.15.60:6789/0 conn(0x7f695415cd70 legacy=0x7f69540d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.068+0000 7f69591fe640 1 -- 172.21.15.60:0/3741390324 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69540d1740 con 0x7f69540cded0 2023-10-16T12:53:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.069+0000 7f69591fe640 1 --2- 172.21.15.60:0/3741390324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540cded0 0x7f69540b5c90 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f694c009d90 tx=0x7f694c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5f0d2deb39c0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.069+0000 7f69537fe640 1 -- 172.21.15.60:0/3741390324 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c02cc60 con 0x7f69540cded0 2023-10-16T12:53:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69537fe640 1 -- 172.21.15.60:0/3741390324 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c02cdc0 con 0x7f69540cded0 2023-10-16T12:53:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69537fe640 1 -- 172.21.15.60:0/3741390324 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c034660 con 0x7f69540cded0 2023-10-16T12:53:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 -- 172.21.15.60:0/3741390324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540cded0 msgr2=0x7f69540b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 --2- 172.21.15.60:0/3741390324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540cded0 0x7f69540b5c90 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f694c009d90 tx=0x7f694c02beb0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 -- 172.21.15.60:0/3741390324 shutdown_connections 2023-10-16T12:53:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 --2- 172.21.15.60:0/3741390324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540cded0 0x7f69540b5c90 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 -- 172.21.15.60:0/3741390324 >> 172.21.15.60:0/3741390324 conn(0x7f69540cf570 msgr2=0x7f69540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 -- 172.21.15.60:0/3741390324 shutdown_connections 2023-10-16T12:53:23.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.070+0000 7f69599ff640 1 -- 172.21.15.60:0/3741390324 wait complete. 2023-10-16T12:53:23.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.072+0000 7f69599ff640 1 Processor -- start 2023-10-16T12:53:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.072+0000 7f69599ff640 1 -- start start 2023-10-16T12:53:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.073+0000 7f69599ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 0x7f695415fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.073+0000 7f69599ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69540cded0 0x7f6954156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.073+0000 7f69599ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695415cd70 0x7f69541f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.073+0000 7f69599ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69540b6ff0 con 0x7f69540b8480 2023-10-16T12:53:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.073+0000 7f69599ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69540b6700 con 0x7f695415cd70 2023-10-16T12:53:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.073+0000 7f69599ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69540b69a0 con 0x7f69540cded0 2023-10-16T12:53:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 0x7f695415fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 0x7f695415fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46578/0 (socket says 172.21.15.60:46578) 2023-10-16T12:53:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 -- 172.21.15.60:0/2006098702 learned_addr learned my addr 172.21.15.60:0/2006098702 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69591fe640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695415cd70 0x7f69541f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f6953fff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69540cded0 0x7f6954156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 -- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69540cded0 msgr2=0x7f6954156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69540cded0 0x7f6954156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 -- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695415cd70 msgr2=0x7f69541f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695415cd70 0x7f69541f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 -- 172.21.15.60:0/2006098702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c009a70 con 0x7f69540b8480 2023-10-16T12:53:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.074+0000 7f69589fd640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 0x7f695415fc50 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f694800c410 tx=0x7f694800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.075+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948005ce0 con 0x7f69540b8480 2023-10-16T12:53:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.075+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948005e40 con 0x7f69540b8480 2023-10-16T12:53:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.075+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69480072f0 con 0x7f69540b8480 2023-10-16T12:53:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.075+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69540b6c80 con 0x7f69540b8480 2023-10-16T12:53:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.075+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69541f2990 con 0x7f69540b8480 2023-10-16T12:53:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.076+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69541f2b80 con 0x7f69540b8480 2023-10-16T12:53:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.077+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6948007450 con 0x7f69540b8480 2023-10-16T12:53:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.078+0000 7f69517fa640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f693806afc0 0x7f693806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.078+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6948088a90 con 0x7f69540b8480 2023-10-16T12:53:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.079+0000 7f6953fff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f693806afc0 0x7f693806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.080+0000 7f6953fff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f693806afc0 0x7f693806d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f69541f6800 tx=0x7f6940009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:23.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.083+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69480578c0 con 0x7f69540b8480 2023-10-16T12:53:23.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.281+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f695404e9b0 con 0x7f69540b8480 2023-10-16T12:53:23.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.283+0000 7f69517fa640 1 -- 172.21.15.60:0/2006098702 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+12855 (secure 0 0 0) 0x7f6948017090 con 0x7f69540b8480 2023-10-16T12:53:23.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:23.287 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":139,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:22.246899+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:23.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f693806afc0 msgr2=0x7f693806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f693806afc0 0x7f693806d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f69541f6800 tx=0x7f6940009040 comp rx=0 tx=0).stop 2023-10-16T12:53:23.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 msgr2=0x7f695415fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 0x7f695415fc50 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f694800c410 tx=0x7f694800c8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 shutdown_connections 2023-10-16T12:53:23.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f693806afc0 0x7f693806d480 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69540cded0 0x7f6954156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695415cd70 0x7f69541f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 --2- 172.21.15.60:0/2006098702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69540b8480 0x7f695415fc50 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 >> 172.21.15.60:0/2006098702 conn(0x7f69540cf570 msgr2=0x7f695415be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:23.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 shutdown_connections 2023-10-16T12:53:23.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.286+0000 7f69599ff640 1 -- 172.21.15.60:0/2006098702 wait complete. 2023-10-16T12:53:23.310 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:53:23.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:53:23.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.528+0000 7fd43b94e640 1 Processor -- start 2023-10-16T12:53:23.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.528+0000 7fd43b94e640 1 -- start start 2023-10-16T12:53:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd43b94e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4340b3420 0x7fd4340b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd43b94e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4340d3980 con 0x7fd4340cf710 2023-10-16T12:53:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd43b94e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4340ce630 con 0x7fd4340cfaf0 2023-10-16T12:53:23.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd43b94e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4340d3c60 con 0x7fd4340b3420 2023-10-16T12:53:23.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd4396ea640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd4340cf710 0x7fd4340b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51978/0 (socket says 172.21.15.60:51978) 2023-10-16T12:53:23.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd4396ea640 1 -- 172.21.15.60:0/46219030 learned_addr learned my addr 172.21.15.60:0/46219030 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:23.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd438ee9640 1 --2- 172.21.15.60:0/46219030 >> v2:172.21.15.60:3300/0 conn(0x7fd4340b3420 0x7fd4340b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.530+0000 7fd42bfff640 1 -- 172.21.15.60:0/46219030 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671244829 0 0) 0x7fd4340d3980 con 0x7fd4340cf710 2023-10-16T12:53:23.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.531+0000 7fd42bfff640 1 -- 172.21.15.60:0/46219030 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd420003520 con 0x7fd4340cf710 2023-10-16T12:53:23.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.531+0000 7fd438ee9640 1 -- 172.21.15.60:0/46219030 >> v1:172.21.15.144:6789/0 conn(0x7fd4340cfaf0 legacy=0x7fd4340b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:23.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.531+0000 7fd438ee9640 1 -- 172.21.15.60:0/46219030 >> v1:172.21.15.60:6789/0 conn(0x7fd4340cf710 legacy=0x7fd4340b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.531+0000 7fd438ee9640 1 -- 172.21.15.60:0/46219030 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4340ce910 con 0x7fd4340b3420 2023-10-16T12:53:23.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.532+0000 7fd438ee9640 1 --2- 172.21.15.60:0/46219030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4340b5cc0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fd4240097a0 tx=0x7fd42402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa327472042ebab9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:23.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.532+0000 7fd42bfff640 1 -- 172.21.15.60:0/46219030 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42402e070 con 0x7fd4340b3420 2023-10-16T12:53:23.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.532+0000 7fd42bfff640 1 -- 172.21.15.60:0/46219030 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4240315a0 con 0x7fd4340b3420 2023-10-16T12:53:23.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.532+0000 7fd42bfff640 1 -- 172.21.15.60:0/46219030 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424031930 con 0x7fd4340b3420 2023-10-16T12:53:23.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 -- 172.21.15.60:0/46219030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 msgr2=0x7fd4340b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 --2- 172.21.15.60:0/46219030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4340b5cc0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fd4240097a0 tx=0x7fd42402b840 comp rx=0 tx=0).stop 2023-10-16T12:53:23.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 -- 172.21.15.60:0/46219030 shutdown_connections 2023-10-16T12:53:23.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 --2- 172.21.15.60:0/46219030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4340b5cc0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 -- 172.21.15.60:0/46219030 >> 172.21.15.60:0/46219030 conn(0x7fd4340d2cc0 msgr2=0x7fd4340d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:23.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 -- 172.21.15.60:0/46219030 shutdown_connections 2023-10-16T12:53:23.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.533+0000 7fd43b94e640 1 -- 172.21.15.60:0/46219030 wait complete. 2023-10-16T12:53:23.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.535+0000 7fd43b94e640 1 Processor -- start 2023-10-16T12:53:23.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.535+0000 7fd43b94e640 1 -- start start 2023-10-16T12:53:23.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd43b94e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4341f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd43b94e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4340cf710 0x7fd43414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd43b94e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4340cfaf0 0x7fd43406b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd43b94e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4341f35a0 con 0x7fd4340b3420 2023-10-16T12:53:23.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd43b94e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43414d460 con 0x7fd4340cf710 2023-10-16T12:53:23.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd43b94e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43414fd40 con 0x7fd4340cfaf0 2023-10-16T12:53:23.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd4396ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4341f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd4396ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4341f3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46606/0 (socket says 172.21.15.60:46606) 2023-10-16T12:53:23.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.536+0000 7fd4396ea640 1 -- 172.21.15.60:0/2741617848 learned_addr learned my addr 172.21.15.60:0/2741617848 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:23.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd438ee9640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4340cf710 0x7fd43414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd4396ea640 1 -- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4340cfaf0 msgr2=0x7fd43406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd439eeb640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4340cfaf0 0x7fd43406b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd4396ea640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4340cfaf0 0x7fd43406b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd4396ea640 1 -- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4340cf710 msgr2=0x7fd43414f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd4396ea640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4340cf710 0x7fd43414f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd4396ea640 1 -- 172.21.15.60:0/2741617848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424009400 con 0x7fd4340b3420 2023-10-16T12:53:23.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd438ee9640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4340cf710 0x7fd43414f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:23.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd439eeb640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4340cfaf0 0x7fd43406b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:23.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd4396ea640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4341f3060 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd430001260 tx=0x7fd43000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:23.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430007480 con 0x7fd4340b3420 2023-10-16T12:53:23.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4300075e0 con 0x7fd4340b3420 2023-10-16T12:53:23.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4300059c0 con 0x7fd4340b3420 2023-10-16T12:53:23.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd434150020 con 0x7fd4340b3420 2023-10-16T12:53:23.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.537+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43414d700 con 0x7fd4340b3420 2023-10-16T12:53:23.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.539+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43414d8f0 con 0x7fd4340b3420 2023-10-16T12:53:23.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.540+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd430005b20 con 0x7fd4340b3420 2023-10-16T12:53:23.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.541+0000 7fd429ffb640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd41806af90 0x7fd41806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.541+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd430088a60 con 0x7fd4340b3420 2023-10-16T12:53:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.542+0000 7fd438ee9640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd41806af90 0x7fd41806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.543+0000 7fd438ee9640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd41806af90 0x7fd41806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd424036000 tx=0x7fd42403d040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:23.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.545+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd430053d40 con 0x7fd4340b3420 2023-10-16T12:53:23.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.743+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd43404e160 con 0x7fd4340b3420 2023-10-16T12:53:23.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.743+0000 7fd429ffb640 1 -- 172.21.15.60:0/2741617848 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd430017070 con 0x7fd4340b3420 2023-10-16T12:53:23.747 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:53:23.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.746+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd41806af90 msgr2=0x7fd41806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.746+0000 7fd43b94e640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd41806af90 0x7fd41806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd424036000 tx=0x7fd42403d040 comp rx=0 tx=0).stop 2023-10-16T12:53:23.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.746+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 msgr2=0x7fd4341f3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:23.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.746+0000 7fd43b94e640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4341f3060 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd430001260 tx=0x7fd43000c960 comp rx=0 tx=0).stop 2023-10-16T12:53:23.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 shutdown_connections 2023-10-16T12:53:23.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd41806af90 0x7fd41806d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4340cfaf0 0x7fd43406b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4340cf710 0x7fd43414f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 --2- 172.21.15.60:0/2741617848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4340b3420 0x7fd4341f3060 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 >> 172.21.15.60:0/2741617848 conn(0x7fd4340d2cc0 msgr2=0x7fd434160140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 shutdown_connections 2023-10-16T12:53:23.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.747+0000 7fd43b94e640 1 -- 172.21.15.60:0/2741617848 wait complete. 2023-10-16T12:53:23.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:53:24.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:23.999+0000 7f784882e640 1 Processor -- start 2023-10-16T12:53:24.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.000+0000 7f784882e640 1 -- start start 2023-10-16T12:53:24.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.001+0000 7f784882e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844070b60 0x7f784415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.001+0000 7f784882e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f784406c100 0x7f784406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.001+0000 7f784882e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784415aef0 con 0x7f7844159950 2023-10-16T12:53:24.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.001+0000 7f784882e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784415e490 con 0x7f784406c100 2023-10-16T12:53:24.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.001+0000 7f784882e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784415d910 con 0x7f7844070b60 2023-10-16T12:53:24.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f784406c100 0x7f784406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f784406c100 0x7f784406c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46610/0 (socket says 172.21.15.60:46610) 2023-10-16T12:53:24.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 -- 172.21.15.60:0/4287580701 learned_addr learned my addr 172.21.15.60:0/4287580701 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:24.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f783bfff640 1 --2- 172.21.15.60:0/4287580701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844070b60 0x7f784415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 -- 172.21.15.60:0/4287580701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844070b60 msgr2=0x7f784415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:24.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 --2- 172.21.15.60:0/4287580701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844070b60 0x7f784415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 -- 172.21.15.60:0/4287580701 >> v1:172.21.15.144:6789/0 conn(0x7f7844159950 legacy=0x7f7844070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.002+0000 7f7843fff640 1 -- 172.21.15.60:0/4287580701 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784415dbf0 con 0x7f784406c100 2023-10-16T12:53:24.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.004+0000 7f7843fff640 1 --2- 172.21.15.60:0/4287580701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f784406c550 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f783c009e50 tx=0x7f783c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18561807972f2348 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:24.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.004+0000 7f7842ffd640 1 -- 172.21.15.60:0/4287580701 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c02ca70 con 0x7f784406c100 2023-10-16T12:53:24.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.004+0000 7f7842ffd640 1 -- 172.21.15.60:0/4287580701 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c02cbd0 con 0x7f784406c100 2023-10-16T12:53:24.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.004+0000 7f7842ffd640 1 -- 172.21.15.60:0/4287580701 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c0343f0 con 0x7f784406c100 2023-10-16T12:53:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 -- 172.21.15.60:0/4287580701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 msgr2=0x7f784406c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 --2- 172.21.15.60:0/4287580701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f784406c550 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f783c009e50 tx=0x7f783c02bed0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 -- 172.21.15.60:0/4287580701 shutdown_connections 2023-10-16T12:53:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 --2- 172.21.15.60:0/4287580701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f784406c550 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 --2- 172.21.15.60:0/4287580701 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844070b60 0x7f784415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 -- 172.21.15.60:0/4287580701 >> 172.21.15.60:0/4287580701 conn(0x7f78440ce6a0 msgr2=0x7f78440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 -- 172.21.15.60:0/4287580701 shutdown_connections 2023-10-16T12:53:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.005+0000 7f784882e640 1 -- 172.21.15.60:0/4287580701 wait complete. 2023-10-16T12:53:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.007+0000 7f784882e640 1 Processor -- start 2023-10-16T12:53:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.008+0000 7f784882e640 1 -- start start 2023-10-16T12:53:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f784882e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f78440af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f784882e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7844070b60 0x7f7844153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f784882e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844159950 0x7f7844151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f784882e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78440c9300 con 0x7f784406c100 2023-10-16T12:53:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f784882e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7844154600 con 0x7f7844070b60 2023-10-16T12:53:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f784882e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78441548e0 con 0x7f7844159950 2023-10-16T12:53:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f78437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f78440af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f78437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f78440af620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46626/0 (socket says 172.21.15.60:46626) 2023-10-16T12:53:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.009+0000 7f78437fe640 1 -- 172.21.15.60:0/2086213465 learned_addr learned my addr 172.21.15.60:0/2086213465 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f783bfff640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7844070b60 0x7f7844153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f78437fe640 1 -- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844159950 msgr2=0x7f7844151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f7843fff640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844159950 0x7f7844151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f78437fe640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844159950 0x7f7844151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f78437fe640 1 -- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7844070b60 msgr2=0x7f7844153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f78437fe640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7844070b60 0x7f7844153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f78437fe640 1 -- 172.21.15.60:0/2086213465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c009b30 con 0x7f784406c100 2023-10-16T12:53:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f783bfff640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7844070b60 0x7f7844153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f78437fe640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f78440af620 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f783400c400 tx=0x7f783400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:24.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78340052d0 con 0x7f784406c100 2023-10-16T12:53:24.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7834005430 con 0x7f784406c100 2023-10-16T12:53:24.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7834003830 con 0x7f784406c100 2023-10-16T12:53:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.010+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7844154bc0 con 0x7f784406c100 2023-10-16T12:53:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.011+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7844154280 con 0x7f784406c100 2023-10-16T12:53:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.012+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784404e9b0 con 0x7f784406c100 2023-10-16T12:53:24.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.013+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7834003990 con 0x7f784406c100 2023-10-16T12:53:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.014+0000 7f7840ff9640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f782406af90 0x7f782406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.014+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f78340888e0 con 0x7f784406c100 2023-10-16T12:53:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.014+0000 7f783bfff640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f782406af90 0x7f782406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.016+0000 7f783bfff640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f782406af90 0x7f782406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f78440afb60 tx=0x7f782c008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.018+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7834053b30 con 0x7f784406c100 2023-10-16T12:53:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.220+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f784415aef0 con 0x7f784406c100 2023-10-16T12:53:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.221+0000 7f7840ff9640 1 -- 172.21.15.60:0/2086213465 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v139) v1 ==== 84+0+65 (secure 0 0 0) 0x7f784415aef0 con 0x7f784406c100 2023-10-16T12:53:24.223 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:24.224 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:53:24.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:24.224 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:53:24.224 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:53:24.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:53:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.223+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f782406af90 msgr2=0x7f782406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f782406af90 0x7f782406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f78440afb60 tx=0x7f782c008040 comp rx=0 tx=0).stop 2023-10-16T12:53:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 msgr2=0x7f78440af620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f78440af620 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f783400c400 tx=0x7f783400c8c0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 shutdown_connections 2023-10-16T12:53:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f782406af90 0x7f782406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7844159950 0x7f7844151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7844070b60 0x7f7844153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 --2- 172.21.15.60:0/2086213465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784406c100 0x7f78440af620 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 >> 172.21.15.60:0/2086213465 conn(0x7f78440ce6a0 msgr2=0x7f784406b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 shutdown_connections 2023-10-16T12:53:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.224+0000 7f784882e640 1 -- 172.21.15.60:0/2086213465 wait complete. 2023-10-16T12:53:24.246 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:53:24.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:53:24.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.487+0000 7ff24e16c640 1 Processor -- start 2023-10-16T12:53:24.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.487+0000 7ff24e16c640 1 -- start start 2023-10-16T12:53:24.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.489+0000 7ff24e16c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff2481598b0 0x7ff2480ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.489+0000 7ff24e16c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2480cd5c0 con 0x7ff248159c90 2023-10-16T12:53:24.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.489+0000 7ff24e16c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24815aed0 con 0x7ff2480b1b80 2023-10-16T12:53:24.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.489+0000 7ff24e16c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2480cd860 con 0x7ff2481598b0 2023-10-16T12:53:24.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.489+0000 7ff246ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff248159c90 0x7ff2480b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51980/0 (socket says 172.21.15.60:51980) 2023-10-16T12:53:24.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.489+0000 7ff246ffd640 1 -- 172.21.15.60:0/2568894573 learned_addr learned my addr 172.21.15.60:0/2568894573 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:24.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665307777 0 0) 0x7ff2480cd5c0 con 0x7ff248159c90 2023-10-16T12:53:24.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2477fe640 1 --2- 172.21.15.60:0/2568894573 >> v2:172.21.15.144:3300/0 conn(0x7ff2481598b0 0x7ff2480ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22c003520 con 0x7ff248159c90 2023-10-16T12:53:24.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3936712758 0 0) 0x7ff22c003520 con 0x7ff248159c90 2023-10-16T12:53:24.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 >> v2:172.21.15.144:3300/0 conn(0x7ff2481598b0 msgr2=0x7ff2480ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:24.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 --2- 172.21.15.60:0/2568894573 >> v2:172.21.15.144:3300/0 conn(0x7ff2481598b0 0x7ff2480ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 >> v1:172.21.15.144:6789/0 conn(0x7ff2480b1b80 legacy=0x7ff24815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2480cdb40 con 0x7ff248159c90 2023-10-16T12:53:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff234001670 con 0x7ff248159c90 2023-10-16T12:53:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 >> v1:172.21.15.60:6789/0 conn(0x7ff248159c90 legacy=0x7ff2480b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff22c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22c0047c0 0x7ff22c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.490+0000 7ff2467fc640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff22c0052a0 0x7ff22c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2480cd860 con 0x7ff22c0039e0 2023-10-16T12:53:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c00c1f0 con 0x7ff22c0047c0 2023-10-16T12:53:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c00c3b0 con 0x7ff22c0052a0 2023-10-16T12:53:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 --2- 172.21.15.60:0/2568894573 >> v2:172.21.15.144:3300/0 conn(0x7ff2481598b0 0x7ff2480ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff22c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 -- 172.21.15.60:0/2568894573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff22c0052a0 msgr2=0x7ff22c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff22c0052a0 0x7ff22c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 -- 172.21.15.60:0/2568894573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22c0047c0 msgr2=0x7ff22c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22c0047c0 0x7ff22c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff2477fe640 1 -- 172.21.15.60:0/2568894573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c003520 con 0x7ff22c0039e0 2023-10-16T12:53:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.491+0000 7ff247fff640 1 -- 172.21.15.60:0/2568894573 reap_dead start 2023-10-16T12:53:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.493+0000 7ff2477fe640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff22c003e00 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7ff238006fd0 tx=0x7ff22c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db40fbcba1aa3c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.493+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380305f0 con 0x7ff22c0039e0 2023-10-16T12:53:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.493+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff238030750 con 0x7ff22c0039e0 2023-10-16T12:53:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.493+0000 7ff2467fc640 1 -- 172.21.15.60:0/2568894573 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238030ac0 con 0x7ff22c0039e0 2023-10-16T12:53:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.493+0000 7ff24e16c640 1 -- 172.21.15.60:0/2568894573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 msgr2=0x7ff22c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.493+0000 7ff24e16c640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff22c003e00 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7ff238006fd0 tx=0x7ff22c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.494+0000 7ff24e16c640 1 -- 172.21.15.60:0/2568894573 shutdown_connections 2023-10-16T12:53:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.494+0000 7ff24e16c640 1 --2- 172.21.15.60:0/2568894573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff22c003e00 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.494+0000 7ff24e16c640 1 -- 172.21.15.60:0/2568894573 >> 172.21.15.60:0/2568894573 conn(0x7ff2480cc470 msgr2=0x7ff2480cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.494+0000 7ff24e16c640 1 -- 172.21.15.60:0/2568894573 shutdown_connections 2023-10-16T12:53:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.494+0000 7ff24e16c640 1 -- 172.21.15.60:0/2568894573 wait complete. 2023-10-16T12:53:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.495+0000 7ff24e16c640 1 Processor -- start 2023-10-16T12:53:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.496+0000 7ff24e16c640 1 -- start start 2023-10-16T12:53:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff24e16c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff2481f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff24e16c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2481f62c0 0x7ff2481f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff24e16c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff24814fc40 0x7ff2481500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff24e16c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2481f6c60 con 0x7ff22c0039e0 2023-10-16T12:53:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff24e16c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2481514a0 con 0x7ff24814fc40 2023-10-16T12:53:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff24e16c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff248150920 con 0x7ff2481f62c0 2023-10-16T12:53:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff2481f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff2481f3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46636/0 (socket says 172.21.15.60:46636) 2023-10-16T12:53:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 -- 172.21.15.60:0/392253947 learned_addr learned my addr 172.21.15.60:0/392253947 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff246ffd640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2481f62c0 0x7ff2481f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff247fff640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff24814fc40 0x7ff2481500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 -- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2481f62c0 msgr2=0x7ff2481f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2481f62c0 0x7ff2481f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 -- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff24814fc40 msgr2=0x7ff2481500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.497+0000 7ff2477fe640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff24814fc40 0x7ff2481500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff2477fe640 1 -- 172.21.15.60:0/392253947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff248150c00 con 0x7ff22c0039e0 2023-10-16T12:53:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff246ffd640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2481f62c0 0x7ff2481f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff2477fe640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff2481f3a00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff23802be80 tx=0x7ff238004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380399e0 con 0x7ff22c0039e0 2023-10-16T12:53:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23803f040 con 0x7ff22c0039e0 2023-10-16T12:53:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238039da0 con 0x7ff22c0039e0 2023-10-16T12:53:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff248150ee0 con 0x7ff22c0039e0 2023-10-16T12:53:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.498+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff248150610 con 0x7ff22c0039e0 2023-10-16T12:53:24.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.500+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2480af1c0 con 0x7ff22c0039e0 2023-10-16T12:53:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.500+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff23804c400 con 0x7ff22c0039e0 2023-10-16T12:53:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.501+0000 7ff23ffff640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff23006af90 0x7ff23006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.501+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff2380ab880 con 0x7ff22c0039e0 2023-10-16T12:53:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.502+0000 7ff246ffd640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff23006af90 0x7ff23006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:24.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.503+0000 7ff246ffd640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff23006af90 0x7ff23006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff2340011e0 tx=0x7ff234009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:24.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.506+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff238076a50 con 0x7ff22c0039e0 2023-10-16T12:53:24.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:24.708+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff2481f6c60 con 0x7ff22c0039e0 2023-10-16T12:53:25.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.276+0000 7ff23ffff640 1 -- 172.21.15.60:0/392253947 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v140) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff2380340b0 con 0x7ff22c0039e0 2023-10-16T12:53:25.279 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:53:25.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.279+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff23006af90 msgr2=0x7ff23006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:25.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.279+0000 7ff24e16c640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff23006af90 0x7ff23006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff2340011e0 tx=0x7ff234009040 comp rx=0 tx=0).stop 2023-10-16T12:53:25.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 msgr2=0x7ff2481f3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:25.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff2481f3a00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff23802be80 tx=0x7ff238004730 comp rx=0 tx=0).stop 2023-10-16T12:53:25.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 shutdown_connections 2023-10-16T12:53:25.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff23006af90 0x7ff23006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2481f62c0 0x7ff2481f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff24814fc40 0x7ff2481500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 --2- 172.21.15.60:0/392253947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c0039e0 0x7ff2481f3a00 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 >> 172.21.15.60:0/392253947 conn(0x7ff2480cc470 msgr2=0x7ff2480c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:25.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 shutdown_connections 2023-10-16T12:53:25.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.280+0000 7ff24e16c640 1 -- 172.21.15.60:0/392253947 wait complete. 2023-10-16T12:53:25.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:53:25.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.532+0000 7eff8af67640 1 Processor -- start 2023-10-16T12:53:25.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.533+0000 7eff8af67640 1 -- start start 2023-10-16T12:53:25.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.534+0000 7eff8af67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 0x7eff840cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:25.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.534+0000 7eff8af67640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7eff840b4160 0x7eff840cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:25.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.534+0000 7eff8af67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff840cc850 con 0x7eff8415ac70 2023-10-16T12:53:25.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.534+0000 7eff8af67640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff840ccb30 con 0x7eff840b4160 2023-10-16T12:53:25.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.534+0000 7eff8af67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff840cce10 con 0x7eff8406a4b0 2023-10-16T12:53:25.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.535+0000 7eff89764640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 0x7eff840cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:25.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.535+0000 7eff89764640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 0x7eff840cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:35032/0 (socket says 172.21.15.60:35032) 2023-10-16T12:53:25.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.535+0000 7eff89764640 1 -- 172.21.15.60:0/4105986307 learned_addr learned my addr 172.21.15.60:0/4105986307 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:25.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.535+0000 7eff8a766640 1 --2- 172.21.15.60:0/4105986307 >> v2:172.21.15.144:3300/0 conn(0x7eff840b4160 0x7eff840cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:25.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.536+0000 7eff88f63640 1 -- 172.21.15.60:0/4105986307 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029976098 0 0) 0x7eff840cc850 con 0x7eff8415ac70 2023-10-16T12:53:25.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.536+0000 7eff89764640 1 -- 172.21.15.60:0/4105986307 >> v2:172.21.15.144:3300/0 conn(0x7eff840b4160 msgr2=0x7eff840cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:25.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.536+0000 7eff89764640 1 --2- 172.21.15.60:0/4105986307 >> v2:172.21.15.144:3300/0 conn(0x7eff840b4160 0x7eff840cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.536+0000 7eff89764640 1 -- 172.21.15.60:0/4105986307 >> v1:172.21.15.144:6789/0 conn(0x7eff8415ac70 legacy=0x7eff84069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:25.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.536+0000 7eff89764640 1 -- 172.21.15.60:0/4105986307 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff840cd0f0 con 0x7eff8406a4b0 2023-10-16T12:53:25.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.537+0000 7eff89764640 1 --2- 172.21.15.60:0/4105986307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 0x7eff840cc270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff74009870 tx=0x7eff7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a1e181f930cccb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:25.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.538+0000 7eff88f63640 1 -- 172.21.15.60:0/4105986307 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74030840 con 0x7eff8406a4b0 2023-10-16T12:53:25.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.538+0000 7eff88f63640 1 -- 172.21.15.60:0/4105986307 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff740309a0 con 0x7eff8406a4b0 2023-10-16T12:53:25.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.538+0000 7eff88f63640 1 -- 172.21.15.60:0/4105986307 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74030d30 con 0x7eff8406a4b0 2023-10-16T12:53:25.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 -- 172.21.15.60:0/4105986307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 msgr2=0x7eff840cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:25.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 --2- 172.21.15.60:0/4105986307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 0x7eff840cc270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff74009870 tx=0x7eff7402b980 comp rx=0 tx=0).stop 2023-10-16T12:53:25.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 -- 172.21.15.60:0/4105986307 shutdown_connections 2023-10-16T12:53:25.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 --2- 172.21.15.60:0/4105986307 >> v2:172.21.15.144:3300/0 conn(0x7eff840b4160 0x7eff840cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 --2- 172.21.15.60:0/4105986307 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff8406a4b0 0x7eff840cc270 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 -- 172.21.15.60:0/4105986307 >> 172.21.15.60:0/4105986307 conn(0x7eff840ce6a0 msgr2=0x7eff840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:25.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 -- 172.21.15.60:0/4105986307 shutdown_connections 2023-10-16T12:53:25.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.539+0000 7eff8af67640 1 -- 172.21.15.60:0/4105986307 wait complete. 2023-10-16T12:53:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.541+0000 7eff8af67640 1 Processor -- start 2023-10-16T12:53:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.541+0000 7eff8af67640 1 -- start start 2023-10-16T12:53:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.542+0000 7eff8af67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 0x7eff840c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.542+0000 7eff8af67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff840b4160 0x7eff840c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.542+0000 7eff8af67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff8415ac70 0x7eff841ed500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.542+0000 7eff8af67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff841551c0 con 0x7eff8406a4b0 2023-10-16T12:53:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.542+0000 7eff8af67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff841554a0 con 0x7eff8415ac70 2023-10-16T12:53:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.542+0000 7eff8af67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff84155780 con 0x7eff840b4160 2023-10-16T12:53:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 0x7eff840c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 0x7eff840c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46638/0 (socket says 172.21.15.60:46638) 2023-10-16T12:53:25.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 -- 172.21.15.60:0/2351601509 learned_addr learned my addr 172.21.15.60:0/2351601509 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:25.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89764640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff840b4160 0x7eff840c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 -- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff840b4160 msgr2=0x7eff840c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff840b4160 0x7eff840c7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff8a766640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff8415ac70 0x7eff841ed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 -- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff8415ac70 msgr2=0x7eff841ed500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff8415ac70 0x7eff841ed500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 -- 172.21.15.60:0/2351601509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff740094d0 con 0x7eff8406a4b0 2023-10-16T12:53:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.543+0000 7eff89f65640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 0x7eff840c8f70 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7eff8000c410 tx=0x7eff8000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.544+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff800035f0 con 0x7eff8406a4b0 2023-10-16T12:53:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.544+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff80003770 con 0x7eff8406a4b0 2023-10-16T12:53:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.544+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80003b20 con 0x7eff8406a4b0 2023-10-16T12:53:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.544+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff84155a60 con 0x7eff8406a4b0 2023-10-16T12:53:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.544+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff840c94b0 con 0x7eff8406a4b0 2023-10-16T12:53:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.545+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff840c9640 con 0x7eff8406a4b0 2023-10-16T12:53:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.547+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7eff80003c80 con 0x7eff8406a4b0 2023-10-16T12:53:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.549+0000 7eff7a7fc640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff6806af90 0x7eff6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.549+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5227+0+0 (secure 0 0 0) 0x7eff80088f80 con 0x7eff8406a4b0 2023-10-16T12:53:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.550+0000 7eff89764640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff6806af90 0x7eff6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.551+0000 7eff89764640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff6806af90 0x7eff6806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff74036000 tx=0x7eff7403b040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.553+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7eff800541e0 con 0x7eff8406a4b0 2023-10-16T12:53:25.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:25.761+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7eff8404e9b0 con 0x7eff8406a4b0 2023-10-16T12:53:26.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.293+0000 7eff7a7fc640 1 -- 172.21.15.60:0/2351601509 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v141) v1 ==== 170+0+0 (secure 0 0 0) 0x7eff800170d0 con 0x7eff8406a4b0 2023-10-16T12:53:26.296 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:53:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff6806af90 msgr2=0x7eff6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff6806af90 0x7eff6806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff74036000 tx=0x7eff7403b040 comp rx=0 tx=0).stop 2023-10-16T12:53:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 msgr2=0x7eff840c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 0x7eff840c8f70 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7eff8000c410 tx=0x7eff8000c8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 shutdown_connections 2023-10-16T12:53:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7eff6806af90 0x7eff6806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7eff840b4160 0x7eff840c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff8415ac70 0x7eff841ed500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 --2- 172.21.15.60:0/2351601509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff8406a4b0 0x7eff840c8f70 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 >> 172.21.15.60:0/2351601509 conn(0x7eff840ce6a0 msgr2=0x7eff84069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.297+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 shutdown_connections 2023-10-16T12:53:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.298+0000 7eff8af67640 1 -- 172.21.15.60:0/2351601509 wait complete. 2023-10-16T12:53:26.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:53:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.524+0000 7fb9df676640 1 Processor -- start 2023-10-16T12:53:26.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.524+0000 7fb9df676640 1 -- start start 2023-10-16T12:53:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.525+0000 7fb9df676640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d815ce00 0x7fb9d8160980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.525+0000 7fb9df676640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9d806ea40 0x7fb9d8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.525+0000 7fb9df676640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9d806a2e0 0x7fb9d8070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.525+0000 7fb9df676640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d80cdc60 con 0x7fb9d806a2e0 2023-10-16T12:53:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.525+0000 7fb9df676640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d806a720 con 0x7fb9d806ea40 2023-10-16T12:53:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.525+0000 7fb9df676640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d81618b0 con 0x7fb9d815ce00 2023-10-16T12:53:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9d806a2e0 0x7fb9d8070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9d806a2e0 0x7fb9d8070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46640/0 (socket says 172.21.15.60:46640) 2023-10-16T12:53:26.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 -- 172.21.15.60:0/69226754 learned_addr learned my addr 172.21.15.60:0/69226754 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9dd412640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d815ce00 0x7fb9d8160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 -- 172.21.15.60:0/69226754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d815ce00 msgr2=0x7fb9d8160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d815ce00 0x7fb9d8160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9dcc11640 1 --2- 172.21.15.60:0/69226754 >> v2:172.21.15.144:3300/0 conn(0x7fb9d806ea40 0x7fb9d8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 -- 172.21.15.60:0/69226754 >> v2:172.21.15.144:3300/0 conn(0x7fb9d806ea40 msgr2=0x7fb9d8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 --2- 172.21.15.60:0/69226754 >> v2:172.21.15.144:3300/0 conn(0x7fb9d806ea40 0x7fb9d8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9ddc13640 1 -- 172.21.15.60:0/69226754 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8161010 con 0x7fb9d806a2e0 2023-10-16T12:53:26.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9dd412640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d815ce00 0x7fb9d8160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.526+0000 7fb9dcc11640 1 --2- 172.21.15.60:0/69226754 >> v2:172.21.15.144:3300/0 conn(0x7fb9d806ea40 0x7fb9d8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.527+0000 7fb9ddc13640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d8070660 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8070190 tx=0x7fb9d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23530d6335dc2f26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.528+0000 7fb9d7fff640 1 -- 172.21.15.60:0/69226754 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00317e0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.528+0000 7fb9d7fff640 1 -- 172.21.15.60:0/69226754 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0031940 con 0x7fb9d806a2e0 2023-10-16T12:53:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.528+0000 7fb9d7fff640 1 -- 172.21.15.60:0/69226754 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0031cd0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.528+0000 7fb9df676640 1 -- 172.21.15.60:0/69226754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 msgr2=0x7fb9d8070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.528+0000 7fb9df676640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d8070660 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8070190 tx=0x7fb9d002b880 comp rx=0 tx=0).stop 2023-10-16T12:53:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 -- 172.21.15.60:0/69226754 shutdown_connections 2023-10-16T12:53:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d8070660 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 --2- 172.21.15.60:0/69226754 >> v2:172.21.15.144:3300/0 conn(0x7fb9d806ea40 0x7fb9d8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 --2- 172.21.15.60:0/69226754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d815ce00 0x7fb9d8160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 -- 172.21.15.60:0/69226754 >> 172.21.15.60:0/69226754 conn(0x7fb9d80cc460 msgr2=0x7fb9d80cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:26.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 -- 172.21.15.60:0/69226754 shutdown_connections 2023-10-16T12:53:26.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.529+0000 7fb9df676640 1 -- 172.21.15.60:0/69226754 wait complete. 2023-10-16T12:53:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.531+0000 7fb9df676640 1 Processor -- start 2023-10-16T12:53:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.531+0000 7fb9df676640 1 -- start start 2023-10-16T12:53:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9df676640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d81f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9df676640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d806ea40 0x7fb9d80c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9df676640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9d815ce00 0x7fb9d80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9df676640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d814a640 con 0x7fb9d806a2e0 2023-10-16T12:53:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9df676640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d814a920 con 0x7fb9d815ce00 2023-10-16T12:53:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9df676640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80c3530 con 0x7fb9d806ea40 2023-10-16T12:53:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9dd412640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d81f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9dd412640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d81f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46646/0 (socket says 172.21.15.60:46646) 2023-10-16T12:53:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9dd412640 1 -- 172.21.15.60:0/4161430585 learned_addr learned my addr 172.21.15.60:0/4161430585 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9dd412640 1 -- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d806ea40 msgr2=0x7fb9d80c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.532+0000 7fb9dcc11640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d806ea40 0x7fb9d80c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9dd412640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d806ea40 0x7fb9d80c3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9ddc13640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9d815ce00 0x7fb9d80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9dd412640 1 -- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9d815ce00 msgr2=0x7fb9d80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:26.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9dd412640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9d815ce00 0x7fb9d80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9dd412640 1 -- 172.21.15.60:0/4161430585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d00094d0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9dd412640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d81f4fd0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800cf90 tx=0x7fb9c800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c800d840 con 0x7fb9d806a2e0 2023-10-16T12:53:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c800d9a0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d80c37d0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c800dd30 con 0x7fb9d806a2e0 2023-10-16T12:53:26.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.533+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d81f5510 con 0x7fb9d806a2e0 2023-10-16T12:53:26.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.534+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d81615b0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.536+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb9c8015440 con 0x7fb9d806a2e0 2023-10-16T12:53:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.537+0000 7fb9d5ffb640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fb9c406af90 0x7fb9c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.537+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb9c808a0b0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.537+0000 7fb9dcc11640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fb9c406af90 0x7fb9c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.538+0000 7fb9dcc11640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fb9c406af90 0x7fb9c406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006fd0 tx=0x7fb9cc008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.541+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9d81615b0 con 0x7fb9d806a2e0 2023-10-16T12:53:26.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:26.779+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb9d81615b0 con 0x7fb9d806a2e0 2023-10-16T12:53:27.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.320+0000 7fb9d5ffb640 1 -- 172.21.15.60:0/4161430585 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v66) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb9d81615b0 con 0x7fb9d806a2e0 2023-10-16T12:53:27.324 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 16 and data pool 17 2023-10-16T12:53:27.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.333+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fb9c406af90 msgr2=0x7fb9c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.333+0000 7fb9df676640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fb9c406af90 0x7fb9c406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006fd0 tx=0x7fb9cc008040 comp rx=0 tx=0).stop 2023-10-16T12:53:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.333+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 msgr2=0x7fb9d81f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.333+0000 7fb9df676640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d81f4fd0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800cf90 tx=0x7fb9c800a7d0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.334+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 shutdown_connections 2023-10-16T12:53:27.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.334+0000 7fb9df676640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fb9c406af90 0x7fb9c406d450 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006fd0 tx=0x7fb9cc008040 comp rx=0 tx=0).stop 2023-10-16T12:53:27.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.334+0000 7fb9df676640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9d806ea40 0x7fb9d80c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.334+0000 7fb9df676640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9d815ce00 0x7fb9d80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.334+0000 7fb9df676640 1 --2- 172.21.15.60:0/4161430585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9d806a2e0 0x7fb9d81f4fd0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.334+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 >> 172.21.15.60:0/4161430585 conn(0x7fb9d80cc460 msgr2=0x7fb9d806dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:27.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.338+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 shutdown_connections 2023-10-16T12:53:27.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.339+0000 7fb9df676640 1 -- 172.21.15.60:0/4161430585 wait complete. 2023-10-16T12:53:27.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:27.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.553+0000 7fe9f5c2e640 1 Processor -- start 2023-10-16T12:53:27.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.554+0000 7fe9f5c2e640 1 -- start start 2023-10-16T12:53:27.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.555+0000 7fe9f5c2e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe9f015cd70 0x7fe9f00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.555+0000 7fe9f5c2e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9f00cded0 0x7fe9f00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.555+0000 7fe9f5c2e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f015d150 con 0x7fe9f00b8480 2023-10-16T12:53:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.555+0000 7fe9f5c2e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00d1180 con 0x7fe9f00cded0 2023-10-16T12:53:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.555+0000 7fe9f5c2e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00d1460 con 0x7fe9f015cd70 2023-10-16T12:53:27.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9effff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe9f00b8480 0x7fe9f00cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51992/0 (socket says 172.21.15.60:51992) 2023-10-16T12:53:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9effff640 1 -- 172.21.15.60:0/3933998788 learned_addr learned my addr 172.21.15.60:0/3933998788 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9f542d640 1 --2- 172.21.15.60:0/3933998788 >> v2:172.21.15.60:3300/0 conn(0x7fe9f00cded0 0x7fe9f00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9f4c2c640 1 --2- 172.21.15.60:0/3933998788 >> v2:172.21.15.144:3300/0 conn(0x7fe9f015cd70 0x7fe9f00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:27.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9ef7fe640 1 -- 172.21.15.60:0/3933998788 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290553994 0 0) 0x7fe9f015d150 con 0x7fe9f00b8480 2023-10-16T12:53:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9ef7fe640 1 -- 172.21.15.60:0/3933998788 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc003520 con 0x7fe9f00b8480 2023-10-16T12:53:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9f542d640 1 -- 172.21.15.60:0/3933998788 >> v2:172.21.15.144:3300/0 conn(0x7fe9f015cd70 msgr2=0x7fe9f00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9f542d640 1 --2- 172.21.15.60:0/3933998788 >> v2:172.21.15.144:3300/0 conn(0x7fe9f015cd70 0x7fe9f00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9f542d640 1 -- 172.21.15.60:0/3933998788 >> v1:172.21.15.60:6789/0 conn(0x7fe9f00b8480 legacy=0x7fe9f00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.556+0000 7fe9f542d640 1 -- 172.21.15.60:0/3933998788 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00d1740 con 0x7fe9f00cded0 2023-10-16T12:53:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.557+0000 7fe9f542d640 1 --2- 172.21.15.60:0/3933998788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00b5c90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00b56a0 tx=0x7fe9e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489def89cf3beb54 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.558+0000 7fe9ef7fe640 1 -- 172.21.15.60:0/3933998788 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8031830 con 0x7fe9f00cded0 2023-10-16T12:53:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.558+0000 7fe9ef7fe640 1 -- 172.21.15.60:0/3933998788 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e8031990 con 0x7fe9f00cded0 2023-10-16T12:53:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.558+0000 7fe9ef7fe640 1 -- 172.21.15.60:0/3933998788 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8031d20 con 0x7fe9f00cded0 2023-10-16T12:53:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.558+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/3933998788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 msgr2=0x7fe9f00b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.558+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/3933998788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00b5c90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00b56a0 tx=0x7fe9e802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.559+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/3933998788 shutdown_connections 2023-10-16T12:53:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.559+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/3933998788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00b5c90 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.559+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/3933998788 >> v2:172.21.15.144:3300/0 conn(0x7fe9f015cd70 0x7fe9f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.559+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/3933998788 >> 172.21.15.60:0/3933998788 conn(0x7fe9f00cf570 msgr2=0x7fe9f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:27.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.559+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/3933998788 shutdown_connections 2023-10-16T12:53:27.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.559+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/3933998788 wait complete. 2023-10-16T12:53:27.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.560+0000 7fe9f5c2e640 1 Processor -- start 2023-10-16T12:53:27.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.561+0000 7fe9f5c2e640 1 -- start start 2023-10-16T12:53:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f5c2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f00b8480 0x7fe9f00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:27.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f5c2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f5c2e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f015cd70 0x7fe9f015a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f5c2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00ab110 con 0x7fe9f00cded0 2023-10-16T12:53:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f5c2e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00cab40 con 0x7fe9f015cd70 2023-10-16T12:53:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f5c2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00ca2a0 con 0x7fe9f00b8480 2023-10-16T12:53:27.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00aabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46672/0 (socket says 172.21.15.60:46672) 2023-10-16T12:53:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9effff640 1 -- 172.21.15.60:0/928513875 learned_addr learned my addr 172.21.15.60:0/928513875 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f542d640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f015cd70 0x7fe9f015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9f4c2c640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f00b8480 0x7fe9f00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9effff640 1 -- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f00b8480 msgr2=0x7fe9f00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.562+0000 7fe9effff640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f00b8480 0x7fe9f00c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9effff640 1 -- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f015cd70 msgr2=0x7fe9f015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9effff640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f015cd70 0x7fe9f015a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9effff640 1 -- 172.21.15.60:0/928513875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e8009400 con 0x7fe9f00cded0 2023-10-16T12:53:27.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9f4c2c640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f00b8480 0x7fe9f00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9effff640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00aabd0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800c500 tx=0x7fe9d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8005ce0 con 0x7fe9f00cded0 2023-10-16T12:53:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8005e40 con 0x7fe9f00cded0 2023-10-16T12:53:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8007330 con 0x7fe9f00cded0 2023-10-16T12:53:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f00ca580 con 0x7fe9f00cded0 2023-10-16T12:53:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.563+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f00ac6c0 con 0x7fe9f00cded0 2023-10-16T12:53:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.564+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f004e9b0 con 0x7fe9f00cded0 2023-10-16T12:53:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.566+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe9d8007550 con 0x7fe9f00cded0 2023-10-16T12:53:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.566+0000 7fe9ed7fa640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe9d006af90 0x7fe9d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.567+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9d8088e80 con 0x7fe9f00cded0 2023-10-16T12:53:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.567+0000 7fe9f4c2c640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe9d006af90 0x7fe9d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.568+0000 7fe9f4c2c640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe9d006af90 0x7fe9d006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0007440 tx=0x7fe9e000b040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:27.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.571+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9d8053ed0 con 0x7fe9f00cded0 2023-10-16T12:53:27.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.767+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9f00c9310 con 0x7fe9f00cded0 2023-10-16T12:53:27.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.769+0000 7fe9ed7fa640 1 -- 172.21.15.60:0/928513875 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16528 (secure 0 0 0) 0x7fe9d8017020 con 0x7fe9f00cded0 2023-10-16T12:53:27.772 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:27.772 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":142,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:26.781162+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:27.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe9d006af90 msgr2=0x7fe9d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe9d006af90 0x7fe9d006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0007440 tx=0x7fe9e000b040 comp rx=0 tx=0).stop 2023-10-16T12:53:27.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 msgr2=0x7fe9f00aabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:27.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00aabd0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800c500 tx=0x7fe9d800c9c0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 shutdown_connections 2023-10-16T12:53:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe9d006af90 0x7fe9d006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f015cd70 0x7fe9f015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f00cded0 0x7fe9f00aabd0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 --2- 172.21.15.60:0/928513875 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f00b8480 0x7fe9f00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.771+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 >> 172.21.15.60:0/928513875 conn(0x7fe9f00cf570 msgr2=0x7fe9f00b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:27.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.772+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 shutdown_connections 2023-10-16T12:53:27.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:27.772+0000 7fe9f5c2e640 1 -- 172.21.15.60:0/928513875 wait complete. 2023-10-16T12:53:27.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:28.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.010+0000 7f3f505d5640 1 Processor -- start 2023-10-16T12:53:28.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.010+0000 7f3f505d5640 1 -- start start 2023-10-16T12:53:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f505d5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f48159b20 0x7f3f4806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f505d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f480b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f505d5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4806f7f0 con 0x7f3f4806c120 2023-10-16T12:53:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f505d5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4806c5e0 con 0x7f3f48159b20 2023-10-16T12:53:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f505d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4806c8c0 con 0x7f3f48159f00 2023-10-16T12:53:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f4e371640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f48159b20 0x7f3f4806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f4e371640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f48159b20 0x7f3f4806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46688/0 (socket says 172.21.15.60:46688) 2023-10-16T12:53:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.012+0000 7f3f4e371640 1 -- 172.21.15.60:0/3267791637 learned_addr learned my addr 172.21.15.60:0/3267791637 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.013+0000 7f3f4db70640 1 --2- 172.21.15.60:0/3267791637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f480b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.013+0000 7f3f4e371640 1 -- 172.21.15.60:0/3267791637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 msgr2=0x7f3f480b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:28.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.013+0000 7f3f4e371640 1 --2- 172.21.15.60:0/3267791637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f480b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.013+0000 7f3f4e371640 1 -- 172.21.15.60:0/3267791637 >> v1:172.21.15.144:6789/0 conn(0x7f3f4806c120 legacy=0x7f3f480cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:28.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.013+0000 7f3f4e371640 1 -- 172.21.15.60:0/3267791637 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4806cba0 con 0x7f3f48159b20 2023-10-16T12:53:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.014+0000 7f3f4e371640 1 --2- 172.21.15.60:0/3267791637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f4806bbe0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab488f25a467a4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.014+0000 7f3f4d36f640 1 -- 172.21.15.60:0/3267791637 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44031640 con 0x7f3f48159b20 2023-10-16T12:53:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.014+0000 7f3f4d36f640 1 -- 172.21.15.60:0/3267791637 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f440317a0 con 0x7f3f48159b20 2023-10-16T12:53:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.014+0000 7f3f4d36f640 1 -- 172.21.15.60:0/3267791637 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44031b30 con 0x7f3f48159b20 2023-10-16T12:53:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 -- 172.21.15.60:0/3267791637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 msgr2=0x7f3f4806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 --2- 172.21.15.60:0/3267791637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f4806bbe0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f4402b8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 -- 172.21.15.60:0/3267791637 shutdown_connections 2023-10-16T12:53:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 --2- 172.21.15.60:0/3267791637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f480b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 --2- 172.21.15.60:0/3267791637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f4806bbe0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 -- 172.21.15.60:0/3267791637 >> 172.21.15.60:0/3267791637 conn(0x7f3f480cb410 msgr2=0x7f3f480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 -- 172.21.15.60:0/3267791637 shutdown_connections 2023-10-16T12:53:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.015+0000 7f3f505d5640 1 -- 172.21.15.60:0/3267791637 wait complete. 2023-10-16T12:53:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.017+0000 7f3f505d5640 1 Processor -- start 2023-10-16T12:53:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.017+0000 7f3f505d5640 1 -- start start 2023-10-16T12:53:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f505d5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806c120 0x7f3f48150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f505d5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f480c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f505d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f48156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f505d5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480b2e40 con 0x7f3f48159b20 2023-10-16T12:53:28.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f505d5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480b21c0 con 0x7f3f4806c120 2023-10-16T12:53:28.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f505d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480b24a0 con 0x7f3f48159f00 2023-10-16T12:53:28.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f4db70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f480c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f4db70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f480c4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46692/0 (socket says 172.21.15.60:46692) 2023-10-16T12:53:28.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.018+0000 7f3f4db70640 1 -- 172.21.15.60:0/2497836062 learned_addr learned my addr 172.21.15.60:0/2497836062 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:28.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4e371640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806c120 0x7f3f48150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4eb72640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f48156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4db70640 1 -- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 msgr2=0x7f3f48156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4db70640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f48156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4db70640 1 -- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806c120 msgr2=0x7f3f48150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4db70640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806c120 0x7f3f48150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4db70640 1 -- 172.21.15.60:0/2497836062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f440094d0 con 0x7f3f48159b20 2023-10-16T12:53:28.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.019+0000 7f3f4db70640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f480c4960 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800d5c0 tx=0x7f3f3800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:28.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.020+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f380125f0 con 0x7f3f48159b20 2023-10-16T12:53:28.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.020+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f38012750 con 0x7f3f48159b20 2023-10-16T12:53:28.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.020+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38012b00 con 0x7f3f48159b20 2023-10-16T12:53:28.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.020+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f480b2780 con 0x7f3f48159b20 2023-10-16T12:53:28.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.020+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f48151520 con 0x7f3f48159b20 2023-10-16T12:53:28.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.021+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f481516e0 con 0x7f3f48159b20 2023-10-16T12:53:28.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.022+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f38012c60 con 0x7f3f48159b20 2023-10-16T12:53:28.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.023+0000 7f3f3effd640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f3f2c06af90 0x7f3f2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.023+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f38089c00 con 0x7f3f48159b20 2023-10-16T12:53:28.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.024+0000 7f3f4e371640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f3f2c06af90 0x7f3f2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.025+0000 7f3f4e371640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f3f2c06af90 0x7f3f2c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f44036000 tx=0x7f3f4403c040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:28.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.027+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f38054b90 con 0x7f3f48159b20 2023-10-16T12:53:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.217+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f4804e160 con 0x7f3f48159b20 2023-10-16T12:53:28.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.219+0000 7f3f3effd640 1 -- 172.21.15.60:0/2497836062 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16528 (secure 0 0 0) 0x7f3f380581b0 con 0x7f3f48159b20 2023-10-16T12:53:28.222 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:28.222 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":142,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:26.781162+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f3f2c06af90 msgr2=0x7f3f2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f3f2c06af90 0x7f3f2c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f44036000 tx=0x7f3f4403c040 comp rx=0 tx=0).stop 2023-10-16T12:53:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 msgr2=0x7f3f480c4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f480c4960 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800d5c0 tx=0x7f3f3800da80 comp rx=0 tx=0).stop 2023-10-16T12:53:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 shutdown_connections 2023-10-16T12:53:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f3f2c06af90 0x7f3f2c06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f48159f00 0x7f3f48156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f48159b20 0x7f3f480c4960 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 --2- 172.21.15.60:0/2497836062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806c120 0x7f3f48150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.221+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 >> 172.21.15.60:0/2497836062 conn(0x7f3f480cb410 msgr2=0x7f3f480cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.222+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 shutdown_connections 2023-10-16T12:53:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.222+0000 7f3f505d5640 1 -- 172.21.15.60:0/2497836062 wait complete. 2023-10-16T12:53:28.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:53:28.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:28.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:28.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.454+0000 7f43debc4640 1 Processor -- start 2023-10-16T12:53:28.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.454+0000 7f43debc4640 1 -- start start 2023-10-16T12:53:28.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.455+0000 7f43debc4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f43d8159950 0x7f43d815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43debc4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43d806c810 0x7f43d80708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43debc4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d815aef0 con 0x7f43d8070450 2023-10-16T12:53:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43debc4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d8070e20 con 0x7f43d806c810 2023-10-16T12:53:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43debc4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d815e490 con 0x7f43d8159950 2023-10-16T12:53:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43d806c810 0x7f43d80708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43d806c810 0x7f43d80708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46708/0 (socket says 172.21.15.60:46708) 2023-10-16T12:53:28.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 -- 172.21.15.60:0/2938780672 learned_addr learned my addr 172.21.15.60:0/2938780672 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:28.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 -- 172.21.15.60:0/2938780672 >> v2:172.21.15.144:3300/0 conn(0x7f43d8159950 msgr2=0x7f43d815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:28.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 --2- 172.21.15.60:0/2938780672 >> v2:172.21.15.144:3300/0 conn(0x7f43d8159950 0x7f43d815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 -- 172.21.15.60:0/2938780672 >> v1:172.21.15.144:6789/0 conn(0x7f43d8070450 legacy=0x7f43d806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.456+0000 7f43de3c3640 1 -- 172.21.15.60:0/2938780672 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d815d910 con 0x7f43d806c810 2023-10-16T12:53:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.458+0000 7f43de3c3640 1 --2- 172.21.15.60:0/2938780672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d806c810 0x7f43d80708e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f43d80ca950 tx=0x7f43d002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2fe76cc747fa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.458+0000 7f43dcbc0640 1 -- 172.21.15.60:0/2938780672 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d002c9f0 con 0x7f43d806c810 2023-10-16T12:53:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.458+0000 7f43dcbc0640 1 -- 172.21.15.60:0/2938780672 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d002cb50 con 0x7f43d806c810 2023-10-16T12:53:28.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.458+0000 7f43dcbc0640 1 -- 172.21.15.60:0/2938780672 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00343f0 con 0x7f43d806c810 2023-10-16T12:53:28.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.458+0000 7f43debc4640 1 -- 172.21.15.60:0/2938780672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d806c810 msgr2=0x7f43d80708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.458+0000 7f43debc4640 1 --2- 172.21.15.60:0/2938780672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d806c810 0x7f43d80708e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f43d80ca950 tx=0x7f43d002be50 comp rx=0 tx=0).stop 2023-10-16T12:53:28.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.459+0000 7f43debc4640 1 -- 172.21.15.60:0/2938780672 shutdown_connections 2023-10-16T12:53:28.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.459+0000 7f43debc4640 1 --2- 172.21.15.60:0/2938780672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d806c810 0x7f43d80708e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.459+0000 7f43debc4640 1 --2- 172.21.15.60:0/2938780672 >> v2:172.21.15.144:3300/0 conn(0x7f43d8159950 0x7f43d815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.459+0000 7f43debc4640 1 -- 172.21.15.60:0/2938780672 >> 172.21.15.60:0/2938780672 conn(0x7f43d80ce6a0 msgr2=0x7f43d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:28.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.459+0000 7f43debc4640 1 -- 172.21.15.60:0/2938780672 shutdown_connections 2023-10-16T12:53:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.459+0000 7f43debc4640 1 -- 172.21.15.60:0/2938780672 wait complete. 2023-10-16T12:53:28.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.460+0000 7f43debc4640 1 Processor -- start 2023-10-16T12:53:28.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 -- start start 2023-10-16T12:53:28.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d806c810 0x7f43d80c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 0x7f43d8155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159950 0x7f43d80c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80c7ae0 con 0x7f43d8070450 2023-10-16T12:53:28.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80c6500 con 0x7f43d8159950 2023-10-16T12:53:28.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.461+0000 7f43debc4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80c67e0 con 0x7f43d806c810 2023-10-16T12:53:28.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 0x7f43d8155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 0x7f43d8155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46720/0 (socket says 172.21.15.60:46720) 2023-10-16T12:53:28.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 -- 172.21.15.60:0/558947688 learned_addr learned my addr 172.21.15.60:0/558947688 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:28.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43ddbc2640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d806c810 0x7f43d80c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 -- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d806c810 msgr2=0x7f43d80c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d806c810 0x7f43d80c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 -- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159950 msgr2=0x7f43d80c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43de3c3640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159950 0x7f43d80c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159950 0x7f43d80c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.462+0000 7f43dd3c1640 1 -- 172.21.15.60:0/558947688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0009b30 con 0x7f43d8070450 2023-10-16T12:53:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.463+0000 7f43dd3c1640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 0x7f43d8155fb0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f43c800c410 tx=0x7f43c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:28.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.463+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80035f0 con 0x7f43d8070450 2023-10-16T12:53:28.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.463+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8003750 con 0x7f43d8070450 2023-10-16T12:53:28.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.463+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d80c6ac0 con 0x7f43d8070450 2023-10-16T12:53:28.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.463+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8003b00 con 0x7f43d8070450 2023-10-16T12:53:28.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.463+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d80c7080 con 0x7f43d8070450 2023-10-16T12:53:28.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.464+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d804e9b0 con 0x7f43d8070450 2023-10-16T12:53:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.466+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43c8003c60 con 0x7f43d8070450 2023-10-16T12:53:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.467+0000 7f43ce7fc640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f43bc06af90 0x7f43bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:28.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.467+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43c8088940 con 0x7f43d8070450 2023-10-16T12:53:28.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.467+0000 7f43ddbc2640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f43bc06af90 0x7f43bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:28.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.468+0000 7f43ddbc2640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f43bc06af90 0x7f43bc06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43d80c5ad0 tx=0x7f43d4008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:28.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.471+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43c8053970 con 0x7f43d8070450 2023-10-16T12:53:28.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:28.711+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f43d815dea0 con 0x7f43d8070450 2023-10-16T12:53:29.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.331+0000 7f43ce7fc640 1 -- 172.21.15.60:0/558947688 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) v1 ==== 118+0+0 (secure 0 0 0) 0x7f43c8056f90 con 0x7f43d8070450 2023-10-16T12:53:29.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f43bc06af90 msgr2=0x7f43bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f43bc06af90 0x7f43bc06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43d80c5ad0 tx=0x7f43d4008040 comp rx=0 tx=0).stop 2023-10-16T12:53:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 msgr2=0x7f43d8155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 0x7f43d8155fb0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f43c800c410 tx=0x7f43c800c8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 shutdown_connections 2023-10-16T12:53:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f43bc06af90 0x7f43bc06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159950 0x7f43d80c2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8070450 0x7f43d8155fb0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 --2- 172.21.15.60:0/558947688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d806c810 0x7f43d80c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.334+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 >> 172.21.15.60:0/558947688 conn(0x7f43d80ce6a0 msgr2=0x7f43d806b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.335+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 shutdown_connections 2023-10-16T12:53:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.335+0000 7f43debc4640 1 -- 172.21.15.60:0/558947688 wait complete. 2023-10-16T12:53:29.358 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:53:29.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:53:29.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.587+0000 7f23a1341640 1 Processor -- start 2023-10-16T12:53:29.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.587+0000 7f23a1341640 1 -- start start 2023-10-16T12:53:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f23a1341640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 0x7f239c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f23a1341640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f239c06a4b0 0x7f239c1650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f23a1341640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0cb140 con 0x7f239c0c9000 2023-10-16T12:53:29.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f23a1341640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239c161a50 con 0x7f239c06a4b0 2023-10-16T12:53:29.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f23a1341640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c06b030 con 0x7f239c0addb0 2023-10-16T12:53:29.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f239a7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f239c0c9000 0x7f239c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44778/0 (socket says 172.21.15.60:44778) 2023-10-16T12:53:29.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f239a7fc640 1 -- 172.21.15.60:0/2593450266 learned_addr learned my addr 172.21.15.60:0/2593450266 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:29.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f239affd640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 0x7f239c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.589+0000 7f239b7fe640 1 --2- 172.21.15.60:0/2593450266 >> v2:172.21.15.144:3300/0 conn(0x7f239c06a4b0 0x7f239c1650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311440675 0 0) 0x7f239c0cb140 con 0x7f239c0c9000 2023-10-16T12:53:29.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2384003520 con 0x7f239c0c9000 2023-10-16T12:53:29.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 483955728 0 0) 0x7f2384003520 con 0x7f239c0c9000 2023-10-16T12:53:29.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 msgr2=0x7f239c0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:29.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 0x7f239c0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 >> v2:172.21.15.144:3300/0 conn(0x7f239c06a4b0 msgr2=0x7f239c1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:29.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 --2- 172.21.15.60:0/2593450266 >> v2:172.21.15.144:3300/0 conn(0x7f239c06a4b0 0x7f239c1650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c06b310 con 0x7f239c0c9000 2023-10-16T12:53:29.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2390001670 con 0x7f239c0c9000 2023-10-16T12:53:29.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 >> v1:172.21.15.60:6789/0 conn(0x7f239c0c9000 legacy=0x7f239c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:29.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384003970 0x7f2384003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 0x7f23840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2384004d00 0x7f2384005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c161a50 con 0x7f2384004d00 2023-10-16T12:53:29.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c06b030 con 0x7f2384003970 2023-10-16T12:53:29.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.590+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238400b980 con 0x7f239c0addb0 2023-10-16T12:53:29.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239b7fe640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2384004d00 0x7f2384005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239a7fc640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 0x7f23840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239affd640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384003970 0x7f2384003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239b7fe640 1 -- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 msgr2=0x7f23840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239b7fe640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c0addb0 0x7f23840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239b7fe640 1 -- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384003970 msgr2=0x7f2384003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239b7fe640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384003970 0x7f2384003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239b7fe640 1 -- 172.21.15.60:0/2593450266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384003520 con 0x7f2384004d00 2023-10-16T12:53:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.591+0000 7f239affd640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384003970 0x7f2384003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.592+0000 7f239b7fe640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2384004d00 0x7f2384005180 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f239c06d830 tx=0x7f23840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2447bfe0b3ee00e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.592+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239402cbd0 con 0x7f2384004d00 2023-10-16T12:53:29.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.592+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239400f040 con 0x7f2384004d00 2023-10-16T12:53:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.592+0000 7f2399ffb640 1 -- 172.21.15.60:0/2593450266 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394034440 con 0x7f2384004d00 2023-10-16T12:53:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f23a1341640 1 -- 172.21.15.60:0/2593450266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2384004d00 msgr2=0x7f2384005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f23a1341640 1 --2- 172.21.15.60:0/2593450266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2384004d00 0x7f2384005180 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f239c06d830 tx=0x7f23840057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f239b7fe640 1 -- 172.21.15.60:0/2593450266 reap_dead start 2023-10-16T12:53:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f23a1341640 1 -- 172.21.15.60:0/2593450266 shutdown_connections 2023-10-16T12:53:29.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f23a1341640 1 -- 172.21.15.60:0/2593450266 >> 172.21.15.60:0/2593450266 conn(0x7f239c0ce9b0 msgr2=0x7f239c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f23a1341640 1 -- 172.21.15.60:0/2593450266 shutdown_connections 2023-10-16T12:53:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.593+0000 7f23a1341640 1 -- 172.21.15.60:0/2593450266 wait complete. 2023-10-16T12:53:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.595+0000 7f23a1341640 1 Processor -- start 2023-10-16T12:53:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.595+0000 7f23a1341640 1 -- start start 2023-10-16T12:53:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f23a1341640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384004d00 0x7f239c1f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f23a1341640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 0x7f239c0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f23a1341640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c1f6d50 0x7f239c1f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f23a1341640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c1f7aa0 con 0x7f239c0c81b0 2023-10-16T12:53:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f23a1341640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0c8b70 con 0x7f2384004d00 2023-10-16T12:53:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f23a1341640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0c4450 con 0x7f239c1f6d50 2023-10-16T12:53:29.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f239a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 0x7f239c0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f239a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 0x7f239c0c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44318/0 (socket says 172.21.15.60:44318) 2023-10-16T12:53:29.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.596+0000 7f239a7fc640 1 -- 172.21.15.60:0/4219549376 learned_addr learned my addr 172.21.15.60:0/4219549376 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:29.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239a7fc640 1 -- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c1f6d50 msgr2=0x7f239c1f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:29.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239a7fc640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c1f6d50 0x7f239c1f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239affd640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384004d00 0x7f239c1f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239a7fc640 1 -- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384004d00 msgr2=0x7f239c1f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239a7fc640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384004d00 0x7f239c1f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239a7fc640 1 -- 172.21.15.60:0/4219549376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0c3690 con 0x7f239c0c81b0 2023-10-16T12:53:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f239a7fc640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 0x7f239c0c8630 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2390001eb0 tx=0x7f239000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239000ee50 con 0x7f239c0c81b0 2023-10-16T12:53:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239000c8d0 con 0x7f239c0c81b0 2023-10-16T12:53:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239000cc80 con 0x7f239c0c81b0 2023-10-16T12:53:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.597+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c0c3970 con 0x7f239c0c81b0 2023-10-16T12:53:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.598+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c0c3f30 con 0x7f239c0c81b0 2023-10-16T12:53:29.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.599+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c0afa40 con 0x7f239c0c81b0 2023-10-16T12:53:29.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.600+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f239001d460 con 0x7f239c0c81b0 2023-10-16T12:53:29.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.600+0000 7f23837fe640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f238806af90 0x7f238806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:29.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.601+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2390089780 con 0x7f239c0c81b0 2023-10-16T12:53:29.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.601+0000 7f239affd640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f238806af90 0x7f238806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:29.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.602+0000 7f239affd640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f238806af90 0x7f238806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f239c0c4fd0 tx=0x7f238c008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:29.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.605+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2390054710 con 0x7f239c0c81b0 2023-10-16T12:53:29.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:29.831+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f239c04e160 con 0x7f239c0c81b0 2023-10-16T12:53:30.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.334+0000 7f23837fe640 1 -- 172.21.15.60:0/4219549376 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2390057d30 con 0x7f239c0c81b0 2023-10-16T12:53:30.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f238806af90 msgr2=0x7f238806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f238806af90 0x7f238806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f239c0c4fd0 tx=0x7f238c008040 comp rx=0 tx=0).stop 2023-10-16T12:53:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 msgr2=0x7f239c0c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 0x7f239c0c8630 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2390001eb0 tx=0x7f239000ec00 comp rx=0 tx=0).stop 2023-10-16T12:53:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 shutdown_connections 2023-10-16T12:53:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f238806af90 0x7f238806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f239c1f6d50 0x7f239c1f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f239c0c81b0 0x7f239c0c8630 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 --2- 172.21.15.60:0/4219549376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2384004d00 0x7f239c1f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 >> 172.21.15.60:0/4219549376 conn(0x7f239c0ce9b0 msgr2=0x7f239c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 shutdown_connections 2023-10-16T12:53:30.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.338+0000 7f23a1341640 1 -- 172.21.15.60:0/4219549376 wait complete. 2023-10-16T12:53:30.362 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:53:30.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.590+0000 7f072759e640 1 Processor -- start 2023-10-16T12:53:30.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.590+0000 7f072759e640 1 -- start start 2023-10-16T12:53:30.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f072806c040 0x7f072806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07280ad970 0x7f072815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f072815f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072806c960 con 0x7f072806c040 2023-10-16T12:53:30.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072815b9d0 con 0x7f07280ad970 2023-10-16T12:53:30.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072815bcb0 con 0x7f072815ebc0 2023-10-16T12:53:30.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f072806c040 0x7f072806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f072806c040 0x7f072806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44326/0 (socket says 172.21.15.60:44326) 2023-10-16T12:53:30.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f072659c640 1 -- 172.21.15.60:0/76393898 learned_addr learned my addr 172.21.15.60:0/76393898 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:30.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.592+0000 7f0726d9d640 1 --2- 172.21.15.60:0/76393898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f072815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.593+0000 7f071ffff640 1 --2- 172.21.15.60:0/76393898 >> v2:172.21.15.144:3300/0 conn(0x7f07280ad970 0x7f072815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.593+0000 7f072659c640 1 -- 172.21.15.60:0/76393898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 msgr2=0x7f072815f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.593+0000 7f072659c640 1 --2- 172.21.15.60:0/76393898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f072815f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.593+0000 7f072659c640 1 -- 172.21.15.60:0/76393898 >> v2:172.21.15.144:3300/0 conn(0x7f07280ad970 msgr2=0x7f072815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.593+0000 7f072659c640 1 --2- 172.21.15.60:0/76393898 >> v2:172.21.15.144:3300/0 conn(0x7f07280ad970 0x7f072815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.593+0000 7f072659c640 1 -- 172.21.15.60:0/76393898 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072815bf90 con 0x7f072806c040 2023-10-16T12:53:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.594+0000 7f072659c640 1 --2- 172.21.15.60:0/76393898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f072806c420 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f07280cb060 tx=0x7f071802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a40d487c3bb240 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.594+0000 7f0725d9b640 1 -- 172.21.15.60:0/76393898 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718031860 con 0x7f072806c040 2023-10-16T12:53:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.594+0000 7f0725d9b640 1 -- 172.21.15.60:0/76393898 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07180319c0 con 0x7f072806c040 2023-10-16T12:53:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.594+0000 7f0725d9b640 1 -- 172.21.15.60:0/76393898 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718031d50 con 0x7f072806c040 2023-10-16T12:53:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 -- 172.21.15.60:0/76393898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 msgr2=0x7f072806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 --2- 172.21.15.60:0/76393898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f072806c420 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f07280cb060 tx=0x7f071802b900 comp rx=0 tx=0).stop 2023-10-16T12:53:30.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 -- 172.21.15.60:0/76393898 shutdown_connections 2023-10-16T12:53:30.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 --2- 172.21.15.60:0/76393898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f072815f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 --2- 172.21.15.60:0/76393898 >> v2:172.21.15.144:3300/0 conn(0x7f07280ad970 0x7f072815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 --2- 172.21.15.60:0/76393898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f072806c420 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 -- 172.21.15.60:0/76393898 >> 172.21.15.60:0/76393898 conn(0x7f07280ce6a0 msgr2=0x7f07280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 -- 172.21.15.60:0/76393898 shutdown_connections 2023-10-16T12:53:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.595+0000 7f072759e640 1 -- 172.21.15.60:0/76393898 wait complete. 2023-10-16T12:53:30.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.597+0000 7f072759e640 1 Processor -- start 2023-10-16T12:53:30.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.597+0000 7f072759e640 1 -- start start 2023-10-16T12:53:30.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f0728157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07280ad970 0x7f072814ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f07280c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07280b10a0 con 0x7f072806c040 2023-10-16T12:53:30.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07280c6af0 con 0x7f07280ad970 2023-10-16T12:53:30.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07280c6dd0 con 0x7f072815ebc0 2023-10-16T12:53:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f0728157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f0728157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44334/0 (socket says 172.21.15.60:44334) 2023-10-16T12:53:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.598+0000 7f072659c640 1 -- 172.21.15.60:0/2863776815 learned_addr learned my addr 172.21.15.60:0/2863776815 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f0726d9d640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f07280c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f071ffff640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07280ad970 0x7f072814ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f072659c640 1 -- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 msgr2=0x7f07280c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f072659c640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f07280c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f072659c640 1 -- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07280ad970 msgr2=0x7f072814ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f072659c640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07280ad970 0x7f072814ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f072659c640 1 -- 172.21.15.60:0/2863776815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180094d0 con 0x7f072806c040 2023-10-16T12:53:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f071ffff640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07280ad970 0x7f072814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f0726d9d640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f07280c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.599+0000 7f072659c640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f0728157e90 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f07180097c0 tx=0x7f07180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.600+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071803ac60 con 0x7f072806c040 2023-10-16T12:53:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.600+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071803adc0 con 0x7f072806c040 2023-10-16T12:53:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.600+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718044610 con 0x7f072806c040 2023-10-16T12:53:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.600+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07280c70b0 con 0x7f072806c040 2023-10-16T12:53:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.600+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07280c7670 con 0x7f072806c040 2023-10-16T12:53:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.601+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072806b330 con 0x7f072806c040 2023-10-16T12:53:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.602+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f071804d450 con 0x7f072806c040 2023-10-16T12:53:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.603+0000 7f071effd640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f070806af90 0x7f070806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.603+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07180abe10 con 0x7f072806c040 2023-10-16T12:53:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.604+0000 7f071ffff640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f070806af90 0x7f070806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:30.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.605+0000 7f071ffff640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f070806af90 0x7f070806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f072806b8c0 tx=0x7f0710008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:30.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.609+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07180af030 con 0x7f072806c040 2023-10-16T12:53:30.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.838+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f072804e9b0 con 0x7f072806c040 2023-10-16T12:53:30.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.839+0000 7f071effd640 1 -- 172.21.15.60:0/2863776815 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f0718035070 con 0x7f072806c040 2023-10-16T12:53:30.842 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:30.842 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:53:26.782026+0000","modified":"2023-10-16T12:53:29.833032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7048},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7048":{"gid":7048,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/3032734613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3032734613},{"type":"v1","addr":"172.21.15.60:6834","nonce":3032734613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:53:30.843 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 70 2023-10-16T12:53:30.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f070806af90 msgr2=0x7f070806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f070806af90 0x7f070806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f072806b8c0 tx=0x7f0710008040 comp rx=0 tx=0).stop 2023-10-16T12:53:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 msgr2=0x7f0728157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f0728157e90 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f07180097c0 tx=0x7f07180046c0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 shutdown_connections 2023-10-16T12:53:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f070806af90 0x7f070806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072815ebc0 0x7f07280c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07280ad970 0x7f072814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 --2- 172.21.15.60:0/2863776815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f072806c040 0x7f0728157e90 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 >> 172.21.15.60:0/2863776815 conn(0x7f07280ce6a0 msgr2=0x7f0728159ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.842+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 shutdown_connections 2023-10-16T12:53:30.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:30.843+0000 7f072759e640 1 -- 172.21.15.60:0/2863776815 wait complete. 2023-10-16T12:53:30.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.090+0000 7fe186ab4640 1 Processor -- start 2023-10-16T12:53:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.090+0000 7fe186ab4640 1 -- start start 2023-10-16T12:53:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.091+0000 7fe186ab4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.091+0000 7fe186ab4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe18015d880 0x7fe1800b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.091+0000 7fe186ab4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18006f7f0 con 0x7fe180159b20 2023-10-16T12:53:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.091+0000 7fe186ab4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18006c5e0 con 0x7fe18015d880 2023-10-16T12:53:31.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.091+0000 7fe186ab4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18006c8c0 con 0x7fe180159f00 2023-10-16T12:53:31.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe184850640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe180159b20 0x7fe18006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44782/0 (socket says 172.21.15.60:44782) 2023-10-16T12:53:31.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe184850640 1 -- 172.21.15.60:0/1126660826 learned_addr learned my addr 172.21.15.60:0/1126660826 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:31.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe185051640 1 --2- 172.21.15.60:0/1126660826 >> v2:172.21.15.60:3300/0 conn(0x7fe18015d880 0x7fe1800b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe17ffff640 1 --2- 172.21.15.60:0/1126660826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe185051640 1 -- 172.21.15.60:0/1126660826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 msgr2=0x7fe1800b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe17f7fe640 1 -- 172.21.15.60:0/1126660826 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538001456 0 0) 0x7fe18006f7f0 con 0x7fe180159b20 2023-10-16T12:53:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe185051640 1 --2- 172.21.15.60:0/1126660826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe185051640 1 -- 172.21.15.60:0/1126660826 >> v1:172.21.15.60:6789/0 conn(0x7fe180159b20 legacy=0x7fe18006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.092+0000 7fe185051640 1 -- 172.21.15.60:0/1126660826 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18006cba0 con 0x7fe18015d880 2023-10-16T12:53:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.093+0000 7fe185051640 1 --2- 172.21.15.60:0/1126660826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe18015d880 0x7fe1800b0950 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe1780097a0 tx=0x7fe17802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160a869841eb49d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe17f7fe640 1 -- 172.21.15.60:0/1126660826 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178031830 con 0x7fe18015d880 2023-10-16T12:53:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe17f7fe640 1 -- 172.21.15.60:0/1126660826 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178031990 con 0x7fe18015d880 2023-10-16T12:53:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe17f7fe640 1 -- 172.21.15.60:0/1126660826 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178031d20 con 0x7fe18015d880 2023-10-16T12:53:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 -- 172.21.15.60:0/1126660826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe18015d880 msgr2=0x7fe1800b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 --2- 172.21.15.60:0/1126660826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe18015d880 0x7fe1800b0950 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe1780097a0 tx=0x7fe17802b8d0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 -- 172.21.15.60:0/1126660826 shutdown_connections 2023-10-16T12:53:31.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 --2- 172.21.15.60:0/1126660826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe18015d880 0x7fe1800b0950 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 --2- 172.21.15.60:0/1126660826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 -- 172.21.15.60:0/1126660826 >> 172.21.15.60:0/1126660826 conn(0x7fe1800cb410 msgr2=0x7fe1800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:31.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.094+0000 7fe186ab4640 1 -- 172.21.15.60:0/1126660826 shutdown_connections 2023-10-16T12:53:31.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.095+0000 7fe186ab4640 1 -- 172.21.15.60:0/1126660826 wait complete. 2023-10-16T12:53:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.096+0000 7fe186ab4640 1 Processor -- start 2023-10-16T12:53:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.096+0000 7fe186ab4640 1 -- start start 2023-10-16T12:53:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe186ab4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 0x7fe1800c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe186ab4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe186ab4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe18015d880 0x7fe1800b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe186ab4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800c7090 con 0x7fe180159b20 2023-10-16T12:53:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe186ab4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800c7370 con 0x7fe18015d880 2023-10-16T12:53:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe186ab4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800c7650 con 0x7fe180159f00 2023-10-16T12:53:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.097+0000 7fe184850640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 0x7fe1800c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 0x7fe1800c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44354/0 (socket says 172.21.15.60:44354) 2023-10-16T12:53:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 -- 172.21.15.60:0/2362602317 learned_addr learned my addr 172.21.15.60:0/2362602317 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 -- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 msgr2=0x7fe1800c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 -- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe18015d880 msgr2=0x7fe1800b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe18015d880 0x7fe1800b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 -- 172.21.15.60:0/2362602317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178009400 con 0x7fe180159b20 2023-10-16T12:53:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe184850640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 0x7fe1800c24c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fe17000c7c0 tx=0x7fe17000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.098+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17000a640 con 0x7fe180159b20 2023-10-16T12:53:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.099+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17000a7c0 con 0x7fe180159b20 2023-10-16T12:53:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.099+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17000ab70 con 0x7fe180159b20 2023-10-16T12:53:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.099+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1800c7930 con 0x7fe180159b20 2023-10-16T12:53:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.099+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1800c7ef0 con 0x7fe180159b20 2023-10-16T12:53:31.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.100+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1800c8100 con 0x7fe180159b20 2023-10-16T12:53:31.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.101+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe17000acd0 con 0x7fe180159b20 2023-10-16T12:53:31.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.102+0000 7fe17d7fa640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe16c06af90 0x7fe16c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.102+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe170088e00 con 0x7fe180159b20 2023-10-16T12:53:31.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.103+0000 7fe17ffff640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe16c06af90 0x7fe16c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.104+0000 7fe17ffff640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe16c06af90 0x7fe16c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1801f40f0 tx=0x7fe174008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.106+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe170053d10 con 0x7fe180159b20 2023-10-16T12:53:31.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.294+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe18004e160 con 0x7fe180159b20 2023-10-16T12:53:31.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.297+0000 7fe17d7fa640 1 -- 172.21.15.60:0/2362602317 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16515 (secure 0 0 0) 0x7fe170018030 con 0x7fe180159b20 2023-10-16T12:53:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:31.301 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":143,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:28.320758+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:31.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.299+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe16c06af90 msgr2=0x7fe16c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.299+0000 7fe186ab4640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe16c06af90 0x7fe16c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1801f40f0 tx=0x7fe174008040 comp rx=0 tx=0).stop 2023-10-16T12:53:31.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.299+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 msgr2=0x7fe1800c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.299+0000 7fe186ab4640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 0x7fe1800c24c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fe17000c7c0 tx=0x7fe17000cc80 comp rx=0 tx=0).stop 2023-10-16T12:53:31.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 shutdown_connections 2023-10-16T12:53:31.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe16c06af90 0x7fe16c06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe180159f00 0x7fe1800c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe18015d880 0x7fe1800b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 --2- 172.21.15.60:0/2362602317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe180159b20 0x7fe1800c24c0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 >> 172.21.15.60:0/2362602317 conn(0x7fe1800cb410 msgr2=0x7fe18006bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 shutdown_connections 2023-10-16T12:53:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.300+0000 7fe186ab4640 1 -- 172.21.15.60:0/2362602317 wait complete. 2023-10-16T12:53:31.321 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:53:31.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:53:31.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.534+0000 7f4625615640 1 Processor -- start 2023-10-16T12:53:31.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.535+0000 7f4625615640 1 -- start start 2023-10-16T12:53:31.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4625615640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f462015ae80 0x7f462015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4625615640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ce390 0x7f4620070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4625615640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462015ed60 con 0x7f462015aaa0 2023-10-16T12:53:31.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4625615640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462006b810 con 0x7f462015ae80 2023-10-16T12:53:31.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4625615640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46200707e0 con 0x7f46200ce390 2023-10-16T12:53:31.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4617fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f462015ae80 0x7f462015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4617fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f462015ae80 0x7f462015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44364/0 (socket says 172.21.15.60:44364) 2023-10-16T12:53:31.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f4617fff640 1 -- 172.21.15.60:0/3368793972 learned_addr learned my addr 172.21.15.60:0/3368793972 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:31.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.537+0000 7f461effd640 1 --1- 172.21.15.60:0/3368793972 >> v1:172.21.15.60:6789/0 conn(0x7f462015aaa0 0x7f46200cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44794/0 (socket says 172.21.15.60:44794) 2023-10-16T12:53:31.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f461f7fe640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ce390 0x7f4620070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f4617fff640 1 -- 172.21.15.60:0/3368793972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ce390 msgr2=0x7f4620070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:31.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f4617fff640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ce390 0x7f4620070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f4617fff640 1 -- 172.21.15.60:0/3368793972 >> v1:172.21.15.60:6789/0 conn(0x7f462015aaa0 legacy=0x7f46200cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f4617fff640 1 -- 172.21.15.60:0/3368793972 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462015ef90 con 0x7f462015ae80 2023-10-16T12:53:31.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f461f7fe640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ce390 0x7f4620070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:31.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.538+0000 7f461e7fc640 1 -- 172.21.15.60:0/3368793972 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010021693 0 0) 0x7f462015ed60 con 0x7f462015aaa0 2023-10-16T12:53:31.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.539+0000 7f4617fff640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462015e820 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4608009870 tx=0x7f460802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be497ddc1ca9079e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.539+0000 7f461e7fc640 1 -- 172.21.15.60:0/3368793972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4608030840 con 0x7f462015ae80 2023-10-16T12:53:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.539+0000 7f461e7fc640 1 -- 172.21.15.60:0/3368793972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46080309a0 con 0x7f462015ae80 2023-10-16T12:53:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.539+0000 7f461e7fc640 1 -- 172.21.15.60:0/3368793972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4608030d30 con 0x7f462015ae80 2023-10-16T12:53:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 -- 172.21.15.60:0/3368793972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 msgr2=0x7f462015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462015e820 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4608009870 tx=0x7f460802b980 comp rx=0 tx=0).stop 2023-10-16T12:53:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 -- 172.21.15.60:0/3368793972 shutdown_connections 2023-10-16T12:53:31.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f46200ce390 0x7f4620070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 --2- 172.21.15.60:0/3368793972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462015e820 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 -- 172.21.15.60:0/3368793972 >> 172.21.15.60:0/3368793972 conn(0x7f46200cb410 msgr2=0x7f46200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:31.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 -- 172.21.15.60:0/3368793972 shutdown_connections 2023-10-16T12:53:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.540+0000 7f4625615640 1 -- 172.21.15.60:0/3368793972 wait complete. 2023-10-16T12:53:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.542+0000 7f4625615640 1 Processor -- start 2023-10-16T12:53:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.542+0000 7f4625615640 1 -- start start 2023-10-16T12:53:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4625615640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46200ce390 0x7f462014fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4625615640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f462015aaa0 0x7f46200caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4625615640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462006cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4625615640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200c7a80 con 0x7f462015ae80 2023-10-16T12:53:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4625615640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200c7e60 con 0x7f46200ce390 2023-10-16T12:53:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4625615640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46200c82f0 con 0x7f462015aaa0 2023-10-16T12:53:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f461f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462006cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f461f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462006cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44368/0 (socket says 172.21.15.60:44368) 2023-10-16T12:53:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f461f7fe640 1 -- 172.21.15.60:0/1171530030 learned_addr learned my addr 172.21.15.60:0/1171530030 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.543+0000 7f4617fff640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f462015aaa0 0x7f46200caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461effd640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46200ce390 0x7f462014fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461f7fe640 1 -- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f462015aaa0 msgr2=0x7f46200caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461f7fe640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f462015aaa0 0x7f46200caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461f7fe640 1 -- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46200ce390 msgr2=0x7f462014fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461f7fe640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46200ce390 0x7f462014fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461f7fe640 1 -- 172.21.15.60:0/1171530030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46080094d0 con 0x7f462015ae80 2023-10-16T12:53:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f461f7fe640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462006cc80 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f461800cf20 tx=0x7f461800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:31.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461800d840 con 0x7f462015ae80 2023-10-16T12:53:31.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461800d9a0 con 0x7f462015ae80 2023-10-16T12:53:31.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461800dd50 con 0x7f462015ae80 2023-10-16T12:53:31.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.544+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46200c2860 con 0x7f462015ae80 2023-10-16T12:53:31.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.545+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46200c77a0 con 0x7f462015ae80 2023-10-16T12:53:31.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.546+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46200c8010 con 0x7f462015ae80 2023-10-16T12:53:31.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.547+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f461801e3b0 con 0x7f462015ae80 2023-10-16T12:53:31.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.548+0000 7f46177fe640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f460006af90 0x7f460006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.548+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f461808a260 con 0x7f462015ae80 2023-10-16T12:53:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.549+0000 7f461effd640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f460006af90 0x7f460006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.549+0000 7f461effd640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f460006af90 0x7f460006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4610000f60 tx=0x7f4610007b90 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:31.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.552+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46200c8010 con 0x7f462015ae80 2023-10-16T12:53:31.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.745+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 --> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46200c7c50 con 0x7f460006af90 2023-10-16T12:53:31.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.774+0000 7f46177fe640 1 -- 172.21.15.60:0/1171530030 <== mgr.6886 v2:172.21.15.144:6832/682650079 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287236 (secure 0 0 0) 0x7f46100071f0 con 0x7f460006af90 2023-10-16T12:53:31.780 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:31.782 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:53:31.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f460006af90 msgr2=0x7f460006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f460006af90 0x7f460006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4610000f60 tx=0x7f4610007b90 comp rx=0 tx=0).stop 2023-10-16T12:53:31.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 msgr2=0x7f462006cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:31.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462006cc80 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f461800cf20 tx=0x7f461800a7d0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 shutdown_connections 2023-10-16T12:53:31.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f460006af90 0x7f460006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f462015aaa0 0x7f46200caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f462015ae80 0x7f462006cc80 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 --2- 172.21.15.60:0/1171530030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46200ce390 0x7f462014fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:31.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 >> 172.21.15.60:0/1171530030 conn(0x7f46200cb410 msgr2=0x7f46200700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:31.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.779+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 shutdown_connections 2023-10-16T12:53:31.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:31.780+0000 7f4625615640 1 -- 172.21.15.60:0/1171530030 wait complete. 2023-10-16T12:53:31.801 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T12:53:30.110876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":407,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356036,"kb_used_data":14988,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754618684,"statfs":{"total":773094113280,"available":772729532416,"internally_reserved":0,"allocated":15347712,"data_stored":7946680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":349097528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.831872"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051574+0000","last_change":"2023-10-16T12:53:27.345574+0000","last_active":"2023-10-16T12:53:29.051574+0000","last_peered":"2023-10-16T12:53:29.051574+0000","last_clean":"2023-10-16T12:53:29.051574+0000","last_became_active":"2023-10-16T12:53:27.344341+0000","last_became_peered":"2023-10-16T12:53:27.344341+0000","last_unstale":"2023-10-16T12:53:29.051574+0000","last_undegraded":"2023-10-16T12:53:29.051574+0000","last_fullsized":"2023-10-16T12:53:29.051574+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:50.132329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335699+0000","last_change":"2023-10-16T12:53:27.330679+0000","last_active":"2023-10-16T12:53:28.335699+0000","last_peered":"2023-10-16T12:53:28.335699+0000","last_clean":"2023-10-16T12:53:28.335699+0000","last_became_active":"2023-10-16T12:53:27.330378+0000","last_became_peered":"2023-10-16T12:53:27.330378+0000","last_unstale":"2023-10-16T12:53:28.335699+0000","last_undegraded":"2023-10-16T12:53:28.335699+0000","last_fullsized":"2023-10-16T12:53:28.335699+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:22.932698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335718+0000","last_change":"2023-10-16T12:53:27.337785+0000","last_active":"2023-10-16T12:53:28.335718+0000","last_peered":"2023-10-16T12:53:28.335718+0000","last_clean":"2023-10-16T12:53:28.335718+0000","last_became_active":"2023-10-16T12:53:27.336305+0000","last_became_peered":"2023-10-16T12:53:27.336305+0000","last_unstale":"2023-10-16T12:53:28.335718+0000","last_undegraded":"2023-10-16T12:53:28.335718+0000","last_fullsized":"2023-10-16T12:53:28.335718+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:23.132944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050680+0000","last_change":"2023-10-16T12:53:27.346793+0000","last_active":"2023-10-16T12:53:29.050680+0000","last_peered":"2023-10-16T12:53:29.050680+0000","last_clean":"2023-10-16T12:53:29.050680+0000","last_became_active":"2023-10-16T12:53:27.346470+0000","last_became_peered":"2023-10-16T12:53:27.346470+0000","last_unstale":"2023-10-16T12:53:29.050680+0000","last_undegraded":"2023-10-16T12:53:29.050680+0000","last_fullsized":"2023-10-16T12:53:29.050680+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:38.066820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334803+0000","last_change":"2023-10-16T12:53:27.339652+0000","last_active":"2023-10-16T12:53:28.334803+0000","last_peered":"2023-10-16T12:53:28.334803+0000","last_clean":"2023-10-16T12:53:28.334803+0000","last_became_active":"2023-10-16T12:53:27.339303+0000","last_became_peered":"2023-10-16T12:53:27.339303+0000","last_unstale":"2023-10-16T12:53:28.334803+0000","last_undegraded":"2023-10-16T12:53:28.334803+0000","last_fullsized":"2023-10-16T12:53:28.334803+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:51.189014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604677+0000","last_change":"2023-10-16T12:53:27.332054+0000","last_active":"2023-10-16T12:53:28.604677+0000","last_peered":"2023-10-16T12:53:28.604677+0000","last_clean":"2023-10-16T12:53:28.604677+0000","last_became_active":"2023-10-16T12:53:27.331737+0000","last_became_peered":"2023-10-16T12:53:27.331737+0000","last_unstale":"2023-10-16T12:53:28.604677+0000","last_undegraded":"2023-10-16T12:53:28.604677+0000","last_fullsized":"2023-10-16T12:53:28.604677+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:46.675807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050886+0000","last_change":"2023-10-16T12:53:27.322436+0000","last_active":"2023-10-16T12:53:29.050886+0000","last_peered":"2023-10-16T12:53:29.050886+0000","last_clean":"2023-10-16T12:53:29.050886+0000","last_became_active":"2023-10-16T12:53:27.322103+0000","last_became_peered":"2023-10-16T12:53:27.322103+0000","last_unstale":"2023-10-16T12:53:29.050886+0000","last_undegraded":"2023-10-16T12:53:29.050886+0000","last_fullsized":"2023-10-16T12:53:29.050886+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:21.072014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335616+0000","last_change":"2023-10-16T12:53:27.325679+0000","last_active":"2023-10-16T12:53:28.335616+0000","last_peered":"2023-10-16T12:53:28.335616+0000","last_clean":"2023-10-16T12:53:28.335616+0000","last_became_active":"2023-10-16T12:53:27.325058+0000","last_became_peered":"2023-10-16T12:53:27.325058+0000","last_unstale":"2023-10-16T12:53:28.335616+0000","last_undegraded":"2023-10-16T12:53:28.335616+0000","last_fullsized":"2023-10-16T12:53:28.335616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:14.229391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051538+0000","last_change":"2023-10-16T12:53:27.349228+0000","last_active":"2023-10-16T12:53:29.051538+0000","last_peered":"2023-10-16T12:53:29.051538+0000","last_clean":"2023-10-16T12:53:29.051538+0000","last_became_active":"2023-10-16T12:53:27.348976+0000","last_became_peered":"2023-10-16T12:53:27.348976+0000","last_unstale":"2023-10-16T12:53:29.051538+0000","last_undegraded":"2023-10-16T12:53:29.051538+0000","last_fullsized":"2023-10-16T12:53:29.051538+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:16.198856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050962+0000","last_change":"2023-10-16T12:53:27.340712+0000","last_active":"2023-10-16T12:53:29.050962+0000","last_peered":"2023-10-16T12:53:29.050962+0000","last_clean":"2023-10-16T12:53:29.050962+0000","last_became_active":"2023-10-16T12:53:27.337374+0000","last_became_peered":"2023-10-16T12:53:27.337374+0000","last_unstale":"2023-10-16T12:53:29.050962+0000","last_undegraded":"2023-10-16T12:53:29.050962+0000","last_fullsized":"2023-10-16T12:53:29.050962+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:04.993657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335357+0000","last_change":"2023-10-16T12:53:27.337354+0000","last_active":"2023-10-16T12:53:28.335357+0000","last_peered":"2023-10-16T12:53:28.335357+0000","last_clean":"2023-10-16T12:53:28.335357+0000","last_became_active":"2023-10-16T12:53:27.337081+0000","last_became_peered":"2023-10-16T12:53:27.337081+0000","last_unstale":"2023-10-16T12:53:28.335357+0000","last_undegraded":"2023-10-16T12:53:28.335357+0000","last_fullsized":"2023-10-16T12:53:28.335357+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:49.116039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050970+0000","last_change":"2023-10-16T12:53:27.336650+0000","last_active":"2023-10-16T12:53:29.050970+0000","last_peered":"2023-10-16T12:53:29.050970+0000","last_clean":"2023-10-16T12:53:29.050970+0000","last_became_active":"2023-10-16T12:53:27.336341+0000","last_became_peered":"2023-10-16T12:53:27.336341+0000","last_unstale":"2023-10-16T12:53:29.050970+0000","last_undegraded":"2023-10-16T12:53:29.050970+0000","last_fullsized":"2023-10-16T12:53:29.050970+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:01.014572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335870+0000","last_change":"2023-10-16T12:53:27.339484+0000","last_active":"2023-10-16T12:53:28.335870+0000","last_peered":"2023-10-16T12:53:28.335870+0000","last_clean":"2023-10-16T12:53:28.335870+0000","last_became_active":"2023-10-16T12:53:27.339206+0000","last_became_peered":"2023-10-16T12:53:27.339206+0000","last_unstale":"2023-10-16T12:53:28.335870+0000","last_undegraded":"2023-10-16T12:53:28.335870+0000","last_fullsized":"2023-10-16T12:53:28.335870+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:30.995925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336364+0000","last_change":"2023-10-16T12:53:27.320397+0000","last_active":"2023-10-16T12:53:28.336364+0000","last_peered":"2023-10-16T12:53:28.336364+0000","last_clean":"2023-10-16T12:53:28.336364+0000","last_became_active":"2023-10-16T12:53:27.319827+0000","last_became_peered":"2023-10-16T12:53:27.319827+0000","last_unstale":"2023-10-16T12:53:28.336364+0000","last_undegraded":"2023-10-16T12:53:28.336364+0000","last_fullsized":"2023-10-16T12:53:28.336364+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:05.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605154+0000","last_change":"2023-10-16T12:53:27.325300+0000","last_active":"2023-10-16T12:53:28.605154+0000","last_peered":"2023-10-16T12:53:28.605154+0000","last_clean":"2023-10-16T12:53:28.605154+0000","last_became_active":"2023-10-16T12:53:27.324687+0000","last_became_peered":"2023-10-16T12:53:27.324687+0000","last_unstale":"2023-10-16T12:53:28.605154+0000","last_undegraded":"2023-10-16T12:53:28.605154+0000","last_fullsized":"2023-10-16T12:53:28.605154+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:33.012375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050265+0000","last_change":"2023-10-16T12:53:27.331540+0000","last_active":"2023-10-16T12:53:29.050265+0000","last_peered":"2023-10-16T12:53:29.050265+0000","last_clean":"2023-10-16T12:53:29.050265+0000","last_became_active":"2023-10-16T12:53:27.331237+0000","last_became_peered":"2023-10-16T12:53:27.331237+0000","last_unstale":"2023-10-16T12:53:29.050265+0000","last_undegraded":"2023-10-16T12:53:29.050265+0000","last_fullsized":"2023-10-16T12:53:29.050265+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:21.233513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335925+0000","last_change":"2023-10-16T12:53:27.341766+0000","last_active":"2023-10-16T12:53:28.335925+0000","last_peered":"2023-10-16T12:53:28.335925+0000","last_clean":"2023-10-16T12:53:28.335925+0000","last_became_active":"2023-10-16T12:53:27.341470+0000","last_became_peered":"2023-10-16T12:53:27.341470+0000","last_unstale":"2023-10-16T12:53:28.335925+0000","last_undegraded":"2023-10-16T12:53:28.335925+0000","last_fullsized":"2023-10-16T12:53:28.335925+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:34.908703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051517+0000","last_change":"2023-10-16T12:53:27.333817+0000","last_active":"2023-10-16T12:53:29.051517+0000","last_peered":"2023-10-16T12:53:29.051517+0000","last_clean":"2023-10-16T12:53:29.051517+0000","last_became_active":"2023-10-16T12:53:27.333546+0000","last_became_peered":"2023-10-16T12:53:27.333546+0000","last_unstale":"2023-10-16T12:53:29.051517+0000","last_undegraded":"2023-10-16T12:53:29.051517+0000","last_fullsized":"2023-10-16T12:53:29.051517+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:47.422545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335878+0000","last_change":"2023-10-16T12:53:27.328374+0000","last_active":"2023-10-16T12:53:28.335878+0000","last_peered":"2023-10-16T12:53:28.335878+0000","last_clean":"2023-10-16T12:53:28.335878+0000","last_became_active":"2023-10-16T12:53:27.326977+0000","last_became_peered":"2023-10-16T12:53:27.326977+0000","last_unstale":"2023-10-16T12:53:28.335878+0000","last_undegraded":"2023-10-16T12:53:28.335878+0000","last_fullsized":"2023-10-16T12:53:28.335878+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:45.236649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336217+0000","last_change":"2023-10-16T12:53:27.333119+0000","last_active":"2023-10-16T12:53:28.336217+0000","last_peered":"2023-10-16T12:53:28.336217+0000","last_clean":"2023-10-16T12:53:28.336217+0000","last_became_active":"2023-10-16T12:53:27.332819+0000","last_became_peered":"2023-10-16T12:53:27.332819+0000","last_unstale":"2023-10-16T12:53:28.336217+0000","last_undegraded":"2023-10-16T12:53:28.336217+0000","last_fullsized":"2023-10-16T12:53:28.336217+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:29.864314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335391+0000","last_change":"2023-10-16T12:53:27.340007+0000","last_active":"2023-10-16T12:53:28.335391+0000","last_peered":"2023-10-16T12:53:28.335391+0000","last_clean":"2023-10-16T12:53:28.335391+0000","last_became_active":"2023-10-16T12:53:27.339442+0000","last_became_peered":"2023-10-16T12:53:27.339442+0000","last_unstale":"2023-10-16T12:53:28.335391+0000","last_undegraded":"2023-10-16T12:53:28.335391+0000","last_fullsized":"2023-10-16T12:53:28.335391+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:42.794398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331416+0000","last_change":"2023-10-16T12:53:27.321043+0000","last_active":"2023-10-16T12:53:28.331416+0000","last_peered":"2023-10-16T12:53:28.331416+0000","last_clean":"2023-10-16T12:53:28.331416+0000","last_became_active":"2023-10-16T12:53:27.316935+0000","last_became_peered":"2023-10-16T12:53:27.316935+0000","last_unstale":"2023-10-16T12:53:28.331416+0000","last_undegraded":"2023-10-16T12:53:28.331416+0000","last_fullsized":"2023-10-16T12:53:28.331416+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:03.488589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334499+0000","last_change":"2023-10-16T12:53:27.332186+0000","last_active":"2023-10-16T12:53:28.334499+0000","last_peered":"2023-10-16T12:53:28.334499+0000","last_clean":"2023-10-16T12:53:28.334499+0000","last_became_active":"2023-10-16T12:53:27.331925+0000","last_became_peered":"2023-10-16T12:53:27.331925+0000","last_unstale":"2023-10-16T12:53:28.334499+0000","last_undegraded":"2023-10-16T12:53:28.334499+0000","last_fullsized":"2023-10-16T12:53:28.334499+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:04.294366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331687+0000","last_change":"2023-10-16T12:53:27.322639+0000","last_active":"2023-10-16T12:53:28.331687+0000","last_peered":"2023-10-16T12:53:28.331687+0000","last_clean":"2023-10-16T12:53:28.331687+0000","last_became_active":"2023-10-16T12:53:27.322323+0000","last_became_peered":"2023-10-16T12:53:27.322323+0000","last_unstale":"2023-10-16T12:53:28.331687+0000","last_undegraded":"2023-10-16T12:53:28.331687+0000","last_fullsized":"2023-10-16T12:53:28.331687+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:35.930393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336179+0000","last_change":"2023-10-16T12:53:27.338716+0000","last_active":"2023-10-16T12:53:28.336179+0000","last_peered":"2023-10-16T12:53:28.336179+0000","last_clean":"2023-10-16T12:53:28.336179+0000","last_became_active":"2023-10-16T12:53:27.338346+0000","last_became_peered":"2023-10-16T12:53:27.338346+0000","last_unstale":"2023-10-16T12:53:28.336179+0000","last_undegraded":"2023-10-16T12:53:28.336179+0000","last_fullsized":"2023-10-16T12:53:28.336179+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:42.069682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051306+0000","last_change":"2023-10-16T12:53:27.342504+0000","last_active":"2023-10-16T12:53:29.051306+0000","last_peered":"2023-10-16T12:53:29.051306+0000","last_clean":"2023-10-16T12:53:29.051306+0000","last_became_active":"2023-10-16T12:53:27.341754+0000","last_became_peered":"2023-10-16T12:53:27.341754+0000","last_unstale":"2023-10-16T12:53:29.051306+0000","last_undegraded":"2023-10-16T12:53:29.051306+0000","last_fullsized":"2023-10-16T12:53:29.051306+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:01.116602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605561+0000","last_change":"2023-10-16T12:53:27.343903+0000","last_active":"2023-10-16T12:53:28.605561+0000","last_peered":"2023-10-16T12:53:28.605561+0000","last_clean":"2023-10-16T12:53:28.605561+0000","last_became_active":"2023-10-16T12:53:27.343069+0000","last_became_peered":"2023-10-16T12:53:27.343069+0000","last_unstale":"2023-10-16T12:53:28.605561+0000","last_undegraded":"2023-10-16T12:53:28.605561+0000","last_fullsized":"2023-10-16T12:53:28.605561+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:25.553166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332452+0000","last_change":"2023-10-16T12:53:27.340160+0000","last_active":"2023-10-16T12:53:28.332452+0000","last_peered":"2023-10-16T12:53:28.332452+0000","last_clean":"2023-10-16T12:53:28.332452+0000","last_became_active":"2023-10-16T12:53:27.339680+0000","last_became_peered":"2023-10-16T12:53:27.339680+0000","last_unstale":"2023-10-16T12:53:28.332452+0000","last_undegraded":"2023-10-16T12:53:28.332452+0000","last_fullsized":"2023-10-16T12:53:28.332452+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:14.229391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334940+0000","last_change":"2023-10-16T12:53:27.332974+0000","last_active":"2023-10-16T12:53:28.334940+0000","last_peered":"2023-10-16T12:53:28.334940+0000","last_clean":"2023-10-16T12:53:28.334940+0000","last_became_active":"2023-10-16T12:53:27.332531+0000","last_became_peered":"2023-10-16T12:53:27.332531+0000","last_unstale":"2023-10-16T12:53:28.334940+0000","last_undegraded":"2023-10-16T12:53:28.334940+0000","last_fullsized":"2023-10-16T12:53:28.334940+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:36.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052318+0000","last_change":"2023-10-16T12:53:27.332884+0000","last_active":"2023-10-16T12:53:29.052318+0000","last_peered":"2023-10-16T12:53:29.052318+0000","last_clean":"2023-10-16T12:53:29.052318+0000","last_became_active":"2023-10-16T12:53:27.331591+0000","last_became_peered":"2023-10-16T12:53:27.331591+0000","last_unstale":"2023-10-16T12:53:29.052318+0000","last_undegraded":"2023-10-16T12:53:29.052318+0000","last_fullsized":"2023-10-16T12:53:29.052318+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:36.096973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050397+0000","last_change":"2023-10-16T12:53:27.333474+0000","last_active":"2023-10-16T12:53:29.050397+0000","last_peered":"2023-10-16T12:53:29.050397+0000","last_clean":"2023-10-16T12:53:29.050397+0000","last_became_active":"2023-10-16T12:53:27.333132+0000","last_became_peered":"2023-10-16T12:53:27.333132+0000","last_unstale":"2023-10-16T12:53:29.050397+0000","last_undegraded":"2023-10-16T12:53:29.050397+0000","last_fullsized":"2023-10-16T12:53:29.050397+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:28.268373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604479+0000","last_change":"2023-10-16T12:53:27.331337+0000","last_active":"2023-10-16T12:53:28.604479+0000","last_peered":"2023-10-16T12:53:28.604479+0000","last_clean":"2023-10-16T12:53:28.604479+0000","last_became_active":"2023-10-16T12:53:27.330944+0000","last_became_peered":"2023-10-16T12:53:27.330944+0000","last_unstale":"2023-10-16T12:53:28.604479+0000","last_undegraded":"2023-10-16T12:53:28.604479+0000","last_fullsized":"2023-10-16T12:53:28.604479+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:27.638445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605348+0000","last_change":"2023-10-16T12:53:27.328318+0000","last_active":"2023-10-16T12:53:28.605348+0000","last_peered":"2023-10-16T12:53:28.605348+0000","last_clean":"2023-10-16T12:53:28.605348+0000","last_became_active":"2023-10-16T12:53:27.328007+0000","last_became_peered":"2023-10-16T12:53:27.328007+0000","last_unstale":"2023-10-16T12:53:28.605348+0000","last_undegraded":"2023-10-16T12:53:28.605348+0000","last_fullsized":"2023-10-16T12:53:28.605348+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:24.518288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"142'4","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051705+0000","last_change":"2023-10-16T12:53:26.329697+0000","last_active":"2023-10-16T12:53:29.051705+0000","last_peered":"2023-10-16T12:53:29.051705+0000","last_clean":"2023-10-16T12:53:29.051705+0000","last_became_active":"2023-10-16T12:53:26.328260+0000","last_became_peered":"2023-10-16T12:53:26.328260+0000","last_unstale":"2023-10-16T12:53:29.051705+0000","last_undegraded":"2023-10-16T12:53:29.051705+0000","last_fullsized":"2023-10-16T12:53:29.051705+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:08.270888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052531+0000","last_change":"2023-10-16T12:53:27.334601+0000","last_active":"2023-10-16T12:53:29.052531+0000","last_peered":"2023-10-16T12:53:29.052531+0000","last_clean":"2023-10-16T12:53:29.052531+0000","last_became_active":"2023-10-16T12:53:27.330656+0000","last_became_peered":"2023-10-16T12:53:27.330656+0000","last_unstale":"2023-10-16T12:53:29.052531+0000","last_undegraded":"2023-10-16T12:53:29.052531+0000","last_fullsized":"2023-10-16T12:53:29.052531+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:28.134464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336447+0000","last_change":"2023-10-16T12:53:26.332967+0000","last_active":"2023-10-16T12:53:28.336447+0000","last_peered":"2023-10-16T12:53:28.336447+0000","last_clean":"2023-10-16T12:53:28.336447+0000","last_became_active":"2023-10-16T12:53:26.332651+0000","last_became_peered":"2023-10-16T12:53:26.332651+0000","last_unstale":"2023-10-16T12:53:28.336447+0000","last_undegraded":"2023-10-16T12:53:28.336447+0000","last_fullsized":"2023-10-16T12:53:28.336447+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:20.572277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605111+0000","last_change":"2023-10-16T12:53:27.322154+0000","last_active":"2023-10-16T12:53:28.605111+0000","last_peered":"2023-10-16T12:53:28.605111+0000","last_clean":"2023-10-16T12:53:28.605111+0000","last_became_active":"2023-10-16T12:53:27.321670+0000","last_became_peered":"2023-10-16T12:53:27.321670+0000","last_unstale":"2023-10-16T12:53:28.605111+0000","last_undegraded":"2023-10-16T12:53:28.605111+0000","last_fullsized":"2023-10-16T12:53:28.605111+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:51.189014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334977+0000","last_change":"2023-10-16T12:53:26.318315+0000","last_active":"2023-10-16T12:53:28.334977+0000","last_peered":"2023-10-16T12:53:28.334977+0000","last_clean":"2023-10-16T12:53:28.334977+0000","last_became_active":"2023-10-16T12:53:26.317997+0000","last_became_peered":"2023-10-16T12:53:26.317997+0000","last_unstale":"2023-10-16T12:53:28.334977+0000","last_undegraded":"2023-10-16T12:53:28.334977+0000","last_fullsized":"2023-10-16T12:53:28.334977+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:18.848088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336010+0000","last_change":"2023-10-16T12:53:27.327212+0000","last_active":"2023-10-16T12:53:28.336010+0000","last_peered":"2023-10-16T12:53:28.336010+0000","last_clean":"2023-10-16T12:53:28.336010+0000","last_became_active":"2023-10-16T12:53:27.326353+0000","last_became_peered":"2023-10-16T12:53:27.326353+0000","last_unstale":"2023-10-16T12:53:28.336010+0000","last_undegraded":"2023-10-16T12:53:28.336010+0000","last_fullsized":"2023-10-16T12:53:28.336010+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:42.888155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334811+0000","last_change":"2023-10-16T12:53:26.321761+0000","last_active":"2023-10-16T12:53:28.334811+0000","last_peered":"2023-10-16T12:53:28.334811+0000","last_clean":"2023-10-16T12:53:28.334811+0000","last_became_active":"2023-10-16T12:53:26.321456+0000","last_became_peered":"2023-10-16T12:53:26.321456+0000","last_unstale":"2023-10-16T12:53:28.334811+0000","last_undegraded":"2023-10-16T12:53:28.334811+0000","last_fullsized":"2023-10-16T12:53:28.334811+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:36.980313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050118+0000","last_change":"2023-10-16T12:53:27.338660+0000","last_active":"2023-10-16T12:53:29.050118+0000","last_peered":"2023-10-16T12:53:29.050118+0000","last_clean":"2023-10-16T12:53:29.050118+0000","last_became_active":"2023-10-16T12:53:27.338343+0000","last_became_peered":"2023-10-16T12:53:27.338343+0000","last_unstale":"2023-10-16T12:53:29.050118+0000","last_undegraded":"2023-10-16T12:53:29.050118+0000","last_fullsized":"2023-10-16T12:53:29.050118+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:25.553166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050486+0000","last_change":"2023-10-16T12:53:26.330036+0000","last_active":"2023-10-16T12:53:29.050486+0000","last_peered":"2023-10-16T12:53:29.050486+0000","last_clean":"2023-10-16T12:53:29.050486+0000","last_became_active":"2023-10-16T12:53:26.329693+0000","last_became_peered":"2023-10-16T12:53:26.329693+0000","last_unstale":"2023-10-16T12:53:29.050486+0000","last_undegraded":"2023-10-16T12:53:29.050486+0000","last_fullsized":"2023-10-16T12:53:29.050486+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:48.103832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335103+0000","last_change":"2023-10-16T12:53:27.333790+0000","last_active":"2023-10-16T12:53:28.335103+0000","last_peered":"2023-10-16T12:53:28.335103+0000","last_clean":"2023-10-16T12:53:28.335103+0000","last_became_active":"2023-10-16T12:53:27.333425+0000","last_became_peered":"2023-10-16T12:53:27.333425+0000","last_unstale":"2023-10-16T12:53:28.335103+0000","last_undegraded":"2023-10-16T12:53:28.335103+0000","last_fullsized":"2023-10-16T12:53:28.335103+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:24.692311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331611+0000","last_change":"2023-10-16T12:53:26.322834+0000","last_active":"2023-10-16T12:53:28.331611+0000","last_peered":"2023-10-16T12:53:28.331611+0000","last_clean":"2023-10-16T12:53:28.331611+0000","last_became_active":"2023-10-16T12:53:26.322529+0000","last_became_peered":"2023-10-16T12:53:26.322529+0000","last_unstale":"2023-10-16T12:53:28.331611+0000","last_undegraded":"2023-10-16T12:53:28.331611+0000","last_fullsized":"2023-10-16T12:53:28.331611+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:20.351796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050101+0000","last_change":"2023-10-16T12:53:27.328519+0000","last_active":"2023-10-16T12:53:29.050101+0000","last_peered":"2023-10-16T12:53:29.050101+0000","last_clean":"2023-10-16T12:53:29.050101+0000","last_became_active":"2023-10-16T12:53:27.327557+0000","last_became_peered":"2023-10-16T12:53:27.327557+0000","last_unstale":"2023-10-16T12:53:29.050101+0000","last_undegraded":"2023-10-16T12:53:29.050101+0000","last_fullsized":"2023-10-16T12:53:29.050101+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:12.070525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334449+0000","last_change":"2023-10-16T12:53:26.324360+0000","last_active":"2023-10-16T12:53:28.334449+0000","last_peered":"2023-10-16T12:53:28.334449+0000","last_clean":"2023-10-16T12:53:28.334449+0000","last_became_active":"2023-10-16T12:53:26.324078+0000","last_became_peered":"2023-10-16T12:53:26.324078+0000","last_unstale":"2023-10-16T12:53:28.334449+0000","last_undegraded":"2023-10-16T12:53:28.334449+0000","last_fullsized":"2023-10-16T12:53:28.334449+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:23.507818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050645+0000","last_change":"2023-10-16T12:53:27.332056+0000","last_active":"2023-10-16T12:53:29.050645+0000","last_peered":"2023-10-16T12:53:29.050645+0000","last_clean":"2023-10-16T12:53:29.050645+0000","last_became_active":"2023-10-16T12:53:27.331358+0000","last_became_peered":"2023-10-16T12:53:27.331358+0000","last_unstale":"2023-10-16T12:53:29.050645+0000","last_undegraded":"2023-10-16T12:53:29.050645+0000","last_fullsized":"2023-10-16T12:53:29.050645+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:44.060174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050419+0000","last_change":"2023-10-16T12:53:26.330844+0000","last_active":"2023-10-16T12:53:29.050419+0000","last_peered":"2023-10-16T12:53:29.050419+0000","last_clean":"2023-10-16T12:53:29.050419+0000","last_became_active":"2023-10-16T12:53:26.330044+0000","last_became_peered":"2023-10-16T12:53:26.330044+0000","last_unstale":"2023-10-16T12:53:29.050419+0000","last_undegraded":"2023-10-16T12:53:29.050419+0000","last_fullsized":"2023-10-16T12:53:29.050419+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:14.318434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051512+0000","last_change":"2023-10-16T12:53:27.328794+0000","last_active":"2023-10-16T12:53:29.051512+0000","last_peered":"2023-10-16T12:53:29.051512+0000","last_clean":"2023-10-16T12:53:29.051512+0000","last_became_active":"2023-10-16T12:53:27.328448+0000","last_became_peered":"2023-10-16T12:53:27.328448+0000","last_unstale":"2023-10-16T12:53:29.051512+0000","last_undegraded":"2023-10-16T12:53:29.051512+0000","last_fullsized":"2023-10-16T12:53:29.051512+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:08.580090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335028+0000","last_change":"2023-10-16T12:53:26.325980+0000","last_active":"2023-10-16T12:53:28.335028+0000","last_peered":"2023-10-16T12:53:28.335028+0000","last_clean":"2023-10-16T12:53:28.335028+0000","last_became_active":"2023-10-16T12:53:26.324482+0000","last_became_peered":"2023-10-16T12:53:26.324482+0000","last_unstale":"2023-10-16T12:53:28.335028+0000","last_undegraded":"2023-10-16T12:53:28.335028+0000","last_fullsized":"2023-10-16T12:53:28.335028+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:05.040015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052650+0000","last_change":"2023-10-16T12:53:27.336572+0000","last_active":"2023-10-16T12:53:29.052650+0000","last_peered":"2023-10-16T12:53:29.052650+0000","last_clean":"2023-10-16T12:53:29.052650+0000","last_became_active":"2023-10-16T12:53:27.336288+0000","last_became_peered":"2023-10-16T12:53:27.336288+0000","last_unstale":"2023-10-16T12:53:29.052650+0000","last_undegraded":"2023-10-16T12:53:29.052650+0000","last_fullsized":"2023-10-16T12:53:29.052650+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:58.627231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331928+0000","last_change":"2023-10-16T12:53:26.329852+0000","last_active":"2023-10-16T12:53:28.331928+0000","last_peered":"2023-10-16T12:53:28.331928+0000","last_clean":"2023-10-16T12:53:28.331928+0000","last_became_active":"2023-10-16T12:53:26.329561+0000","last_became_peered":"2023-10-16T12:53:26.329561+0000","last_unstale":"2023-10-16T12:53:28.331928+0000","last_undegraded":"2023-10-16T12:53:28.331928+0000","last_fullsized":"2023-10-16T12:53:28.331928+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:52.917224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332077+0000","last_change":"2023-10-16T12:53:27.339947+0000","last_active":"2023-10-16T12:53:28.332077+0000","last_peered":"2023-10-16T12:53:28.332077+0000","last_clean":"2023-10-16T12:53:28.332077+0000","last_became_active":"2023-10-16T12:53:27.339307+0000","last_became_peered":"2023-10-16T12:53:27.339307+0000","last_unstale":"2023-10-16T12:53:28.332077+0000","last_undegraded":"2023-10-16T12:53:28.332077+0000","last_fullsized":"2023-10-16T12:53:28.332077+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:05.373869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"142'3","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051496+0000","last_change":"2023-10-16T12:53:26.331155+0000","last_active":"2023-10-16T12:53:29.051496+0000","last_peered":"2023-10-16T12:53:29.051496+0000","last_clean":"2023-10-16T12:53:29.051496+0000","last_became_active":"2023-10-16T12:53:26.330750+0000","last_became_peered":"2023-10-16T12:53:26.330750+0000","last_unstale":"2023-10-16T12:53:29.051496+0000","last_undegraded":"2023-10-16T12:53:29.051496+0000","last_fullsized":"2023-10-16T12:53:29.051496+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:45.663599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051861+0000","last_change":"2023-10-16T12:53:27.329942+0000","last_active":"2023-10-16T12:53:29.051861+0000","last_peered":"2023-10-16T12:53:29.051861+0000","last_clean":"2023-10-16T12:53:29.051861+0000","last_became_active":"2023-10-16T12:53:27.329618+0000","last_became_peered":"2023-10-16T12:53:27.329618+0000","last_unstale":"2023-10-16T12:53:29.051861+0000","last_undegraded":"2023-10-16T12:53:29.051861+0000","last_fullsized":"2023-10-16T12:53:29.051861+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:18.796864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"142'6","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051271+0000","last_change":"2023-10-16T12:53:26.332212+0000","last_active":"2023-10-16T12:53:29.051271+0000","last_peered":"2023-10-16T12:53:29.051271+0000","last_clean":"2023-10-16T12:53:29.051271+0000","last_became_active":"2023-10-16T12:53:26.331931+0000","last_became_peered":"2023-10-16T12:53:26.331931+0000","last_unstale":"2023-10-16T12:53:29.051271+0000","last_undegraded":"2023-10-16T12:53:29.051271+0000","last_fullsized":"2023-10-16T12:53:29.051271+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:26.626238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331556+0000","last_change":"2023-10-16T12:53:27.336965+0000","last_active":"2023-10-16T12:53:28.331556+0000","last_peered":"2023-10-16T12:53:28.331556+0000","last_clean":"2023-10-16T12:53:28.331556+0000","last_became_active":"2023-10-16T12:53:27.336608+0000","last_became_peered":"2023-10-16T12:53:27.336608+0000","last_unstale":"2023-10-16T12:53:28.331556+0000","last_undegraded":"2023-10-16T12:53:28.331556+0000","last_fullsized":"2023-10-16T12:53:28.331556+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:16.313311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604808+0000","last_change":"2023-10-16T12:53:26.327068+0000","last_active":"2023-10-16T12:53:28.604808+0000","last_peered":"2023-10-16T12:53:28.604808+0000","last_clean":"2023-10-16T12:53:28.604808+0000","last_became_active":"2023-10-16T12:53:26.326806+0000","last_became_peered":"2023-10-16T12:53:26.326806+0000","last_unstale":"2023-10-16T12:53:28.604808+0000","last_undegraded":"2023-10-16T12:53:28.604808+0000","last_fullsized":"2023-10-16T12:53:28.604808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:19.202881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331502+0000","last_change":"2023-10-16T12:53:27.333771+0000","last_active":"2023-10-16T12:53:28.331502+0000","last_peered":"2023-10-16T12:53:28.331502+0000","last_clean":"2023-10-16T12:53:28.331502+0000","last_became_active":"2023-10-16T12:53:27.327719+0000","last_became_peered":"2023-10-16T12:53:27.327719+0000","last_unstale":"2023-10-16T12:53:28.331502+0000","last_undegraded":"2023-10-16T12:53:28.331502+0000","last_fullsized":"2023-10-16T12:53:28.331502+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.802458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052188+0000","last_change":"2023-10-16T12:53:26.331629+0000","last_active":"2023-10-16T12:53:29.052188+0000","last_peered":"2023-10-16T12:53:29.052188+0000","last_clean":"2023-10-16T12:53:29.052188+0000","last_became_active":"2023-10-16T12:53:26.331114+0000","last_became_peered":"2023-10-16T12:53:26.331114+0000","last_unstale":"2023-10-16T12:53:29.052188+0000","last_undegraded":"2023-10-16T12:53:29.052188+0000","last_fullsized":"2023-10-16T12:53:29.052188+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:58.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331800+0000","last_change":"2023-10-16T12:53:27.338968+0000","last_active":"2023-10-16T12:53:28.331800+0000","last_peered":"2023-10-16T12:53:28.331800+0000","last_clean":"2023-10-16T12:53:28.331800+0000","last_became_active":"2023-10-16T12:53:27.338699+0000","last_became_peered":"2023-10-16T12:53:27.338699+0000","last_unstale":"2023-10-16T12:53:28.331800+0000","last_undegraded":"2023-10-16T12:53:28.331800+0000","last_fullsized":"2023-10-16T12:53:28.331800+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:55.705002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"142'8","reported_seq":20,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334493+0000","last_change":"2023-10-16T12:53:26.323856+0000","last_active":"2023-10-16T12:53:28.334493+0000","last_peered":"2023-10-16T12:53:28.334493+0000","last_clean":"2023-10-16T12:53:28.334493+0000","last_became_active":"2023-10-16T12:53:26.323543+0000","last_became_peered":"2023-10-16T12:53:26.323543+0000","last_unstale":"2023-10-16T12:53:28.334493+0000","last_undegraded":"2023-10-16T12:53:28.334493+0000","last_fullsized":"2023-10-16T12:53:28.334493+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:58.885993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335138+0000","last_change":"2023-10-16T12:53:27.337936+0000","last_active":"2023-10-16T12:53:28.335138+0000","last_peered":"2023-10-16T12:53:28.335138+0000","last_clean":"2023-10-16T12:53:28.335138+0000","last_became_active":"2023-10-16T12:53:27.337636+0000","last_became_peered":"2023-10-16T12:53:27.337636+0000","last_unstale":"2023-10-16T12:53:28.335138+0000","last_undegraded":"2023-10-16T12:53:28.335138+0000","last_fullsized":"2023-10-16T12:53:28.335138+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:33.012375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052045+0000","last_change":"2023-10-16T12:53:26.324486+0000","last_active":"2023-10-16T12:53:29.052045+0000","last_peered":"2023-10-16T12:53:29.052045+0000","last_clean":"2023-10-16T12:53:29.052045+0000","last_became_active":"2023-10-16T12:53:26.324210+0000","last_became_peered":"2023-10-16T12:53:26.324210+0000","last_unstale":"2023-10-16T12:53:29.052045+0000","last_undegraded":"2023-10-16T12:53:29.052045+0000","last_fullsized":"2023-10-16T12:53:29.052045+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:32.000168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605452+0000","last_change":"2023-10-16T12:53:27.341475+0000","last_active":"2023-10-16T12:53:28.605452+0000","last_peered":"2023-10-16T12:53:28.605452+0000","last_clean":"2023-10-16T12:53:28.605452+0000","last_became_active":"2023-10-16T12:53:27.341143+0000","last_became_peered":"2023-10-16T12:53:27.341143+0000","last_unstale":"2023-10-16T12:53:28.605452+0000","last_undegraded":"2023-10-16T12:53:28.605452+0000","last_fullsized":"2023-10-16T12:53:28.605452+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:12.070525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335124+0000","last_change":"2023-10-16T12:53:26.319389+0000","last_active":"2023-10-16T12:53:28.335124+0000","last_peered":"2023-10-16T12:53:28.335124+0000","last_clean":"2023-10-16T12:53:28.335124+0000","last_became_active":"2023-10-16T12:53:26.319048+0000","last_became_peered":"2023-10-16T12:53:26.319048+0000","last_unstale":"2023-10-16T12:53:28.335124+0000","last_undegraded":"2023-10-16T12:53:28.335124+0000","last_fullsized":"2023-10-16T12:53:28.335124+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:26.626238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.337121+0000","last_change":"2023-10-16T12:53:27.341295+0000","last_active":"2023-10-16T12:53:28.337121+0000","last_peered":"2023-10-16T12:53:28.337121+0000","last_clean":"2023-10-16T12:53:28.337121+0000","last_became_active":"2023-10-16T12:53:27.340978+0000","last_became_peered":"2023-10-16T12:53:27.340978+0000","last_unstale":"2023-10-16T12:53:28.337121+0000","last_undegraded":"2023-10-16T12:53:28.337121+0000","last_fullsized":"2023-10-16T12:53:28.337121+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:48.136247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052272+0000","last_change":"2023-10-16T12:53:26.323329+0000","last_active":"2023-10-16T12:53:29.052272+0000","last_peered":"2023-10-16T12:53:29.052272+0000","last_clean":"2023-10-16T12:53:29.052272+0000","last_became_active":"2023-10-16T12:53:26.322383+0000","last_became_peered":"2023-10-16T12:53:26.322383+0000","last_unstale":"2023-10-16T12:53:29.052272+0000","last_undegraded":"2023-10-16T12:53:29.052272+0000","last_fullsized":"2023-10-16T12:53:29.052272+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:05.679613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336493+0000","last_change":"2023-10-16T12:53:27.331450+0000","last_active":"2023-10-16T12:53:28.336493+0000","last_peered":"2023-10-16T12:53:28.336493+0000","last_clean":"2023-10-16T12:53:28.336493+0000","last_became_active":"2023-10-16T12:53:27.330380+0000","last_became_peered":"2023-10-16T12:53:27.330380+0000","last_unstale":"2023-10-16T12:53:28.336493+0000","last_undegraded":"2023-10-16T12:53:28.336493+0000","last_fullsized":"2023-10-16T12:53:28.336493+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:31.970824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050742+0000","last_change":"2023-10-16T12:53:26.325128+0000","last_active":"2023-10-16T12:53:29.050742+0000","last_peered":"2023-10-16T12:53:29.050742+0000","last_clean":"2023-10-16T12:53:29.050742+0000","last_became_active":"2023-10-16T12:53:26.323701+0000","last_became_peered":"2023-10-16T12:53:26.323701+0000","last_unstale":"2023-10-16T12:53:29.050742+0000","last_undegraded":"2023-10-16T12:53:29.050742+0000","last_fullsized":"2023-10-16T12:53:29.050742+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:35.047678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336771+0000","last_change":"2023-10-16T12:53:27.337914+0000","last_active":"2023-10-16T12:53:28.336771+0000","last_peered":"2023-10-16T12:53:28.336771+0000","last_clean":"2023-10-16T12:53:28.336771+0000","last_became_active":"2023-10-16T12:53:27.336096+0000","last_became_peered":"2023-10-16T12:53:27.336096+0000","last_unstale":"2023-10-16T12:53:28.336771+0000","last_undegraded":"2023-10-16T12:53:28.336771+0000","last_fullsized":"2023-10-16T12:53:28.336771+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:18.976350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604813+0000","last_change":"2023-10-16T12:53:26.313637+0000","last_active":"2023-10-16T12:53:28.604813+0000","last_peered":"2023-10-16T12:53:28.604813+0000","last_clean":"2023-10-16T12:53:28.604813+0000","last_became_active":"2023-10-16T12:53:26.313321+0000","last_became_peered":"2023-10-16T12:53:26.313321+0000","last_unstale":"2023-10-16T12:53:28.604813+0000","last_undegraded":"2023-10-16T12:53:28.604813+0000","last_fullsized":"2023-10-16T12:53:28.604813+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:05.040015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050131+0000","last_change":"2023-10-16T12:53:27.341847+0000","last_active":"2023-10-16T12:53:29.050131+0000","last_peered":"2023-10-16T12:53:29.050131+0000","last_clean":"2023-10-16T12:53:29.050131+0000","last_became_active":"2023-10-16T12:53:27.341572+0000","last_became_peered":"2023-10-16T12:53:27.341572+0000","last_unstale":"2023-10-16T12:53:29.050131+0000","last_undegraded":"2023-10-16T12:53:29.050131+0000","last_fullsized":"2023-10-16T12:53:29.050131+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:12.628278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334515+0000","last_change":"2023-10-16T12:53:26.321570+0000","last_active":"2023-10-16T12:53:28.334515+0000","last_peered":"2023-10-16T12:53:28.334515+0000","last_clean":"2023-10-16T12:53:28.334515+0000","last_became_active":"2023-10-16T12:53:26.321303+0000","last_became_peered":"2023-10-16T12:53:26.321303+0000","last_unstale":"2023-10-16T12:53:28.334515+0000","last_undegraded":"2023-10-16T12:53:28.334515+0000","last_fullsized":"2023-10-16T12:53:28.334515+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:36.179677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336152+0000","last_change":"2023-10-16T12:53:27.338969+0000","last_active":"2023-10-16T12:53:28.336152+0000","last_peered":"2023-10-16T12:53:28.336152+0000","last_clean":"2023-10-16T12:53:28.336152+0000","last_became_active":"2023-10-16T12:53:27.338711+0000","last_became_peered":"2023-10-16T12:53:27.338711+0000","last_unstale":"2023-10-16T12:53:28.336152+0000","last_undegraded":"2023-10-16T12:53:28.336152+0000","last_fullsized":"2023-10-16T12:53:28.336152+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:25.778923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604430+0000","last_change":"2023-10-16T12:53:26.320432+0000","last_active":"2023-10-16T12:53:28.604430+0000","last_peered":"2023-10-16T12:53:28.604430+0000","last_clean":"2023-10-16T12:53:28.604430+0000","last_became_active":"2023-10-16T12:53:26.320108+0000","last_became_peered":"2023-10-16T12:53:26.320108+0000","last_unstale":"2023-10-16T12:53:28.604430+0000","last_undegraded":"2023-10-16T12:53:28.604430+0000","last_fullsized":"2023-10-16T12:53:28.604430+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:36.980313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332147+0000","last_change":"2023-10-16T12:53:27.335791+0000","last_active":"2023-10-16T12:53:28.332147+0000","last_peered":"2023-10-16T12:53:28.332147+0000","last_clean":"2023-10-16T12:53:28.332147+0000","last_became_active":"2023-10-16T12:53:27.335489+0000","last_became_peered":"2023-10-16T12:53:27.335489+0000","last_unstale":"2023-10-16T12:53:28.332147+0000","last_undegraded":"2023-10-16T12:53:28.332147+0000","last_fullsized":"2023-10-16T12:53:28.332147+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:32.708283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050832+0000","last_change":"2023-10-16T12:53:26.330049+0000","last_active":"2023-10-16T12:53:29.050832+0000","last_peered":"2023-10-16T12:53:29.050832+0000","last_clean":"2023-10-16T12:53:29.050832+0000","last_became_active":"2023-10-16T12:53:26.329792+0000","last_became_peered":"2023-10-16T12:53:26.329792+0000","last_unstale":"2023-10-16T12:53:29.050832+0000","last_undegraded":"2023-10-16T12:53:29.050832+0000","last_fullsized":"2023-10-16T12:53:29.050832+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:27:15.707126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051576+0000","last_change":"2023-10-16T12:53:27.338277+0000","last_active":"2023-10-16T12:53:29.051576+0000","last_peered":"2023-10-16T12:53:29.051576+0000","last_clean":"2023-10-16T12:53:29.051576+0000","last_became_active":"2023-10-16T12:53:27.337928+0000","last_became_peered":"2023-10-16T12:53:27.337928+0000","last_unstale":"2023-10-16T12:53:29.051576+0000","last_undegraded":"2023-10-16T12:53:29.051576+0000","last_fullsized":"2023-10-16T12:53:29.051576+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:50.337210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331982+0000","last_change":"2023-10-16T12:53:26.333641+0000","last_active":"2023-10-16T12:53:28.331982+0000","last_peered":"2023-10-16T12:53:28.331982+0000","last_clean":"2023-10-16T12:53:28.331982+0000","last_became_active":"2023-10-16T12:53:26.333282+0000","last_became_peered":"2023-10-16T12:53:26.333282+0000","last_unstale":"2023-10-16T12:53:28.331982+0000","last_undegraded":"2023-10-16T12:53:28.331982+0000","last_fullsized":"2023-10-16T12:53:28.331982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:04.438933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051942+0000","last_change":"2023-10-16T12:53:27.345392+0000","last_active":"2023-10-16T12:53:29.051942+0000","last_peered":"2023-10-16T12:53:29.051942+0000","last_clean":"2023-10-16T12:53:29.051942+0000","last_became_active":"2023-10-16T12:53:27.343476+0000","last_became_peered":"2023-10-16T12:53:27.343476+0000","last_unstale":"2023-10-16T12:53:29.051942+0000","last_undegraded":"2023-10-16T12:53:29.051942+0000","last_fullsized":"2023-10-16T12:53:29.051942+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:33.971723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051758+0000","last_change":"2023-10-16T12:53:26.328849+0000","last_active":"2023-10-16T12:53:29.051758+0000","last_peered":"2023-10-16T12:53:29.051758+0000","last_clean":"2023-10-16T12:53:29.051758+0000","last_became_active":"2023-10-16T12:53:26.326676+0000","last_became_peered":"2023-10-16T12:53:26.326676+0000","last_unstale":"2023-10-16T12:53:29.051758+0000","last_undegraded":"2023-10-16T12:53:29.051758+0000","last_fullsized":"2023-10-16T12:53:29.051758+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:48.372259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051082+0000","last_change":"2023-10-16T12:53:27.334448+0000","last_active":"2023-10-16T12:53:29.051082+0000","last_peered":"2023-10-16T12:53:29.051082+0000","last_clean":"2023-10-16T12:53:29.051082+0000","last_became_active":"2023-10-16T12:53:27.334158+0000","last_became_peered":"2023-10-16T12:53:27.334158+0000","last_unstale":"2023-10-16T12:53:29.051082+0000","last_undegraded":"2023-10-16T12:53:29.051082+0000","last_fullsized":"2023-10-16T12:53:29.051082+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:41.302841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051067+0000","last_change":"2023-10-16T12:53:26.327304+0000","last_active":"2023-10-16T12:53:29.051067+0000","last_peered":"2023-10-16T12:53:29.051067+0000","last_clean":"2023-10-16T12:53:29.051067+0000","last_became_active":"2023-10-16T12:53:26.326184+0000","last_became_peered":"2023-10-16T12:53:26.326184+0000","last_unstale":"2023-10-16T12:53:29.051067+0000","last_undegraded":"2023-10-16T12:53:29.051067+0000","last_fullsized":"2023-10-16T12:53:29.051067+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:50.176807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052052+0000","last_change":"2023-10-16T12:53:27.343350+0000","last_active":"2023-10-16T12:53:29.052052+0000","last_peered":"2023-10-16T12:53:29.052052+0000","last_clean":"2023-10-16T12:53:29.052052+0000","last_became_active":"2023-10-16T12:53:27.342994+0000","last_became_peered":"2023-10-16T12:53:27.342994+0000","last_unstale":"2023-10-16T12:53:29.052052+0000","last_undegraded":"2023-10-16T12:53:29.052052+0000","last_fullsized":"2023-10-16T12:53:29.052052+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:38.391019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332165+0000","last_change":"2023-10-16T12:53:26.320506+0000","last_active":"2023-10-16T12:53:28.332165+0000","last_peered":"2023-10-16T12:53:28.332165+0000","last_clean":"2023-10-16T12:53:28.332165+0000","last_became_active":"2023-10-16T12:53:26.320244+0000","last_became_peered":"2023-10-16T12:53:26.320244+0000","last_unstale":"2023-10-16T12:53:28.332165+0000","last_undegraded":"2023-10-16T12:53:28.332165+0000","last_fullsized":"2023-10-16T12:53:28.332165+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:19.794914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334857+0000","last_change":"2023-10-16T12:53:27.334390+0000","last_active":"2023-10-16T12:53:28.334857+0000","last_peered":"2023-10-16T12:53:28.334857+0000","last_clean":"2023-10-16T12:53:28.334857+0000","last_became_active":"2023-10-16T12:53:27.334114+0000","last_became_peered":"2023-10-16T12:53:27.334114+0000","last_unstale":"2023-10-16T12:53:28.334857+0000","last_undegraded":"2023-10-16T12:53:28.334857+0000","last_fullsized":"2023-10-16T12:53:28.334857+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:24.204906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050755+0000","last_change":"2023-10-16T12:53:26.323111+0000","last_active":"2023-10-16T12:53:29.050755+0000","last_peered":"2023-10-16T12:53:29.050755+0000","last_clean":"2023-10-16T12:53:29.050755+0000","last_became_active":"2023-10-16T12:53:26.321661+0000","last_became_peered":"2023-10-16T12:53:26.321661+0000","last_unstale":"2023-10-16T12:53:29.050755+0000","last_undegraded":"2023-10-16T12:53:29.050755+0000","last_fullsized":"2023-10-16T12:53:29.050755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:41.782191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336217+0000","last_change":"2023-10-16T12:53:27.335997+0000","last_active":"2023-10-16T12:53:28.336217+0000","last_peered":"2023-10-16T12:53:28.336217+0000","last_clean":"2023-10-16T12:53:28.336217+0000","last_became_active":"2023-10-16T12:53:27.335727+0000","last_became_peered":"2023-10-16T12:53:27.335727+0000","last_unstale":"2023-10-16T12:53:28.336217+0000","last_undegraded":"2023-10-16T12:53:28.336217+0000","last_fullsized":"2023-10-16T12:53:28.336217+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:09.283095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334612+0000","last_change":"2023-10-16T12:53:26.325158+0000","last_active":"2023-10-16T12:53:28.334612+0000","last_peered":"2023-10-16T12:53:28.334612+0000","last_clean":"2023-10-16T12:53:28.334612+0000","last_became_active":"2023-10-16T12:53:26.323556+0000","last_became_peered":"2023-10-16T12:53:26.323556+0000","last_unstale":"2023-10-16T12:53:28.334612+0000","last_undegraded":"2023-10-16T12:53:28.334612+0000","last_fullsized":"2023-10-16T12:53:28.334612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:29.610305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335793+0000","last_change":"2023-10-16T12:53:27.323690+0000","last_active":"2023-10-16T12:53:28.335793+0000","last_peered":"2023-10-16T12:53:28.335793+0000","last_clean":"2023-10-16T12:53:28.335793+0000","last_became_active":"2023-10-16T12:53:27.323161+0000","last_became_peered":"2023-10-16T12:53:27.323161+0000","last_unstale":"2023-10-16T12:53:28.335793+0000","last_undegraded":"2023-10-16T12:53:28.335793+0000","last_fullsized":"2023-10-16T12:53:28.335793+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:03.587489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334625+0000","last_change":"2023-10-16T12:53:26.325356+0000","last_active":"2023-10-16T12:53:28.334625+0000","last_peered":"2023-10-16T12:53:28.334625+0000","last_clean":"2023-10-16T12:53:28.334625+0000","last_became_active":"2023-10-16T12:53:26.323762+0000","last_became_peered":"2023-10-16T12:53:26.323762+0000","last_unstale":"2023-10-16T12:53:28.334625+0000","last_undegraded":"2023-10-16T12:53:28.334625+0000","last_fullsized":"2023-10-16T12:53:28.334625+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:55.640614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051428+0000","last_change":"2023-10-16T12:53:27.332740+0000","last_active":"2023-10-16T12:53:29.051428+0000","last_peered":"2023-10-16T12:53:29.051428+0000","last_clean":"2023-10-16T12:53:29.051428+0000","last_became_active":"2023-10-16T12:53:27.332210+0000","last_became_peered":"2023-10-16T12:53:27.332210+0000","last_unstale":"2023-10-16T12:53:29.051428+0000","last_undegraded":"2023-10-16T12:53:29.051428+0000","last_fullsized":"2023-10-16T12:53:29.051428+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:11.719188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050738+0000","last_change":"2023-10-16T12:53:26.328402+0000","last_active":"2023-10-16T12:53:29.050738+0000","last_peered":"2023-10-16T12:53:29.050738+0000","last_clean":"2023-10-16T12:53:29.050738+0000","last_became_active":"2023-10-16T12:53:26.327462+0000","last_became_peered":"2023-10-16T12:53:26.327462+0000","last_unstale":"2023-10-16T12:53:29.050738+0000","last_undegraded":"2023-10-16T12:53:29.050738+0000","last_fullsized":"2023-10-16T12:53:29.050738+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:15.751250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331539+0000","last_change":"2023-10-16T12:53:27.327831+0000","last_active":"2023-10-16T12:53:28.331539+0000","last_peered":"2023-10-16T12:53:28.331539+0000","last_clean":"2023-10-16T12:53:28.331539+0000","last_became_active":"2023-10-16T12:53:27.327499+0000","last_became_peered":"2023-10-16T12:53:27.327499+0000","last_unstale":"2023-10-16T12:53:28.331539+0000","last_undegraded":"2023-10-16T12:53:28.331539+0000","last_fullsized":"2023-10-16T12:53:28.331539+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:21.584484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050915+0000","last_change":"2023-10-16T12:53:26.330806+0000","last_active":"2023-10-16T12:53:29.050915+0000","last_peered":"2023-10-16T12:53:29.050915+0000","last_clean":"2023-10-16T12:53:29.050915+0000","last_became_active":"2023-10-16T12:53:26.330515+0000","last_became_peered":"2023-10-16T12:53:26.330515+0000","last_unstale":"2023-10-16T12:53:29.050915+0000","last_undegraded":"2023-10-16T12:53:29.050915+0000","last_fullsized":"2023-10-16T12:53:29.050915+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:28.852107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"131'230","reported_seq":679,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:53:27.314270+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:53:27.314270+0000","last_peered":"2023-10-16T12:53:27.314270+0000","last_clean":"2023-10-16T12:53:27.314270+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:53:27.314270+0000","last_undegraded":"2023-10-16T12:53:27.314270+0000","last_fullsized":"2023-10-16T12:53:27.314270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33496,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338344,"statfs":{"total":96636764160,"available":96602464256,"internally_reserved":0,"allocated":2371584,"data_stored":1440062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.2889999999999999,"15min":1.3069999999999999},"min":{"1min":0.72499999999999998,"5min":0.68200000000000005,"15min":0.66100000000000003},"max":{"1min":1.835,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3300000000000001,"15min":1.3520000000000001},"min":{"1min":0.91000000000000003,"5min":0.76100000000000001,"15min":0.63200000000000001},"max":{"1min":1.804,"5min":1.804,"15min":1.8899999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.373,"15min":1.2689999999999999},"min":{"1min":0.69599999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.0840000000000001,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2170000000000001,"15min":1.224},"min":{"1min":0.79100000000000004,"5min":0.64800000000000002,"15min":0.64700000000000002},"max":{"1min":1.9239999999999999,"5min":5.5490000000000004,"15min":5.5490000000000004},"last":0.876}]},{"osd":2,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.179,"15min":1.1970000000000001},"min":{"1min":0.77900000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6220000000000001,"5min":2.855,"15min":2.855},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.262,"5min":1.278,"15min":1.319},"min":{"1min":0.81399999999999995,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6579999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.98799999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3560000000000001,"15min":1.3180000000000001},"min":{"1min":1.0740000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.724,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.371,"15min":1.391},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5720000000000001,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.0920000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1890000000000001,"15min":1.1859999999999999},"min":{"1min":0.81799999999999995,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4339999999999999,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.3700000000000001,"15min":1.345},"min":{"1min":1.036,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.8169999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.2230000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3560000000000001,"15min":1.337},"min":{"1min":0.871,"5min":0.624,"15min":0.624},"max":{"1min":1.772,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.5269999999999999,"15min":1.4870000000000001},"min":{"1min":1.034,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0270000000000001,"5min":2.411,"15min":2.411},"last":1.2989999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.613,"15min":1.5800000000000001},"min":{"1min":1.1759999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.194,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.341},{"interface":"front","average":{"1min":1.395,"5min":1.47,"15min":1.452},"min":{"1min":0.83399999999999996,"5min":0.75,"15min":0.75},"max":{"1min":1.752,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.2689999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542383,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1761280,"data_stored":842848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2390000000000001,"15min":1.282},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.7370000000000001,"5min":1.952,"15min":3.7450000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.331,"5min":1.302,"15min":1.329},"min":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":3.819},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.165,"15min":1.1850000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.5620000000000001,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.204,"5min":1.2150000000000001,"15min":1.214},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.625},"max":{"1min":1.887,"5min":1.9330000000000001,"15min":3.8519999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2110000000000001,"15min":1.21},"min":{"1min":0.80500000000000005,"5min":0.66900000000000004,"15min":0.627},"max":{"1min":1.5409999999999999,"5min":1.752,"15min":1.8899999999999999},"last":0.746},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.224,"15min":1.266},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5149999999999999,"5min":1.8520000000000001,"15min":3.9009999999999998},"last":0.90300000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.304,"15min":1.3260000000000001},"min":{"1min":0.92400000000000004,"5min":0.85199999999999998,"15min":0.80100000000000005},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":4.2380000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.367,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.85099999999999998,"5min":0.82199999999999995,"15min":0.69599999999999995},"max":{"1min":1.73,"5min":1.806,"15min":4.4269999999999996},"last":0.84499999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.375,"15min":1.3440000000000001},"min":{"1min":0.78100000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3029999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":1.107,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.843,"5min":1.9430000000000001,"15min":2.2189999999999999},"last":0.94799999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.3180000000000001,"15min":1.3049999999999999},"min":{"1min":0.92600000000000005,"5min":0.755,"15min":0.755},"max":{"1min":1.6899999999999999,"5min":1.9830000000000001,"15min":2.0910000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.369,"5min":1.458,"15min":1.4410000000000001},"min":{"1min":0.872,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9830000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.78300000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4610000000000001,"15min":1.446},"min":{"1min":1.0089999999999999,"5min":0.878,"15min":0.878},"max":{"1min":2.0470000000000002,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4890000000000001,"15min":1.468},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":0.96399999999999997},"max":{"1min":1.8049999999999999,"5min":2.052,"15min":2.2610000000000001},"last":0.97599999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":843382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.113,"15min":1.153},"min":{"1min":0.69599999999999995,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.538,"5min":1.8480000000000001,"15min":2.2690000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.181,"5min":1.179,"15min":1.1759999999999999},"min":{"1min":0.76700000000000002,"5min":0.65200000000000002,"15min":0.63300000000000001},"max":{"1min":1.631,"5min":1.6459999999999999,"15min":1.7689999999999999},"last":1.228}]},{"osd":1,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2869999999999999,"15min":1.262},"min":{"1min":0.77800000000000002,"5min":0.56699999999999995,"15min":0.41299999999999998},"max":{"1min":1.665,"5min":1.8080000000000001,"15min":1.8089999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.1859999999999999,"15min":1.2050000000000001},"min":{"1min":0.77000000000000002,"5min":0.46000000000000002,"15min":0.41599999999999998},"max":{"1min":1.6519999999999999,"5min":1.738,"15min":1.738},"last":0.95699999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.278,"15min":1.2849999999999999},"min":{"1min":0.86099999999999999,"5min":0.60799999999999998,"15min":0.60199999999999998},"max":{"1min":1.6060000000000001,"5min":1.9710000000000001,"15min":2.0259999999999998},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.224,"15min":1.272},"min":{"1min":0.94699999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6319999999999999,"5min":1.9870000000000001,"15min":2.2370000000000001},"last":1.1599999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3049999999999999,"15min":1.3069999999999999},"min":{"1min":1.02,"5min":0.73099999999999998,"15min":0.66000000000000003},"max":{"1min":1.474,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.3720000000000001,"15min":1.375},"min":{"1min":0.70999999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.6459999999999999,"5min":2.1150000000000002,"15min":2.2160000000000002},"last":1.601}]},{"osd":4,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.234,"15min":1.2090000000000001},"min":{"1min":0.87,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":1.7549999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.367,"15min":1.325},"min":{"1min":1.0700000000000001,"5min":0.54200000000000004,"15min":0.42999999999999999},"max":{"1min":1.786,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.458}]},{"osd":6,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3759999999999999,"15min":1.3380000000000001},"min":{"1min":0.78600000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8500000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.482},{"interface":"front","average":{"1min":1.474,"5min":1.4179999999999999,"15min":1.3879999999999999},"min":{"1min":1.0920000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0529999999999999,"5min":2.48,"15min":2.48},"last":1.645}]},{"osd":7,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.478,"15min":1.454},"min":{"1min":0.997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.9330000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.5349999999999999,"15min":1.5129999999999999},"min":{"1min":1.1279999999999999,"5min":0.64400000000000002,"15min":0.55300000000000005},"max":{"1min":2.1680000000000001,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.7}]}]},{"osd":4,"up_from":14,"seq":60129542384,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59276,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312564,"statfs":{"total":96636764160,"available":96576065536,"internally_reserved":0,"allocated":1769472,"data_stored":849078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.1539999999999999,"15min":1.194},"min":{"1min":0.622,"5min":0.503,"15min":0.503},"max":{"1min":3.0779999999999998,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.623},{"interface":"front","average":{"1min":1.165,"5min":1.1180000000000001,"15min":1.151},"min":{"1min":0.61199999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":1.1739999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.1759999999999999,"15min":1.1870000000000001},"min":{"1min":0.90800000000000003,"5min":0.38,"15min":0.38},"max":{"1min":3.3180000000000001,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":1.464},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.179,"15min":1.238},"min":{"1min":0.66400000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.5939999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":2.1579999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.2649999999999999,"15min":1.26},"min":{"1min":0.76800000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":4.3470000000000004,"5min":4.3470000000000004,"15min":4.3470000000000004},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.266,"15min":1.319},"min":{"1min":0.75600000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.4670000000000001,"5min":2.4670000000000001,"15min":2.4670000000000001},"last":1.9710000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.246,"15min":1.266},"min":{"1min":0.70899999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.8300000000000001,"5min":2.8300000000000001,"15min":2.8300000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.732,"5min":1.395,"15min":1.381},"min":{"1min":0.91700000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":8.9849999999999994,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":1.8520000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3460000000000001,"15min":1.306},"min":{"1min":0.69599999999999995,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6150000000000002,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":2.2280000000000002},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.347,"15min":1.3},"min":{"1min":0.98499999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.391,"5min":4.391,"15min":4.391},"last":1.7070000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.3899999999999999,"15min":1.3779999999999999},"min":{"1min":0.73899999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.4649999999999999,"5min":3.4649999999999999,"15min":3.4649999999999999},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.4870000000000001,"15min":1.476},"min":{"1min":0.874,"5min":0.502,"15min":0.502},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":2.395}]},{"osd":7,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.498,"15min":1.4730000000000001},"min":{"1min":1.2,"5min":0.67600000000000005,"15min":0.60399999999999998},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.728,"5min":1.605,"15min":1.5880000000000001},"min":{"1min":1.2010000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":2.4460000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542384,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1785856,"data_stored":850760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.92400000000000004,"15min":0.95999999999999996},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":1.609,"5min":1.609,"15min":2.0419999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.071,"15min":1.161},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.34799999999999998},"max":{"1min":1.5549999999999999,"5min":1.5800000000000001,"15min":2.2749999999999999},"last":1.661}]},{"osd":1,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.97899999999999998,"15min":1.0209999999999999},"min":{"1min":0.503,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5529999999999999,"5min":1.6240000000000001,"15min":1.758},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.004,"15min":1.069},"min":{"1min":0.58399999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2070000000000001,"5min":1.784,"15min":2.4580000000000002},"last":1.1259999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98899999999999999,"15min":1.0589999999999999},"min":{"1min":0.39900000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.746,"5min":1.746,"15min":1.9510000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.26,"15min":1.345},"min":{"1min":0.71499999999999997,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8180000000000001,"5min":2.044,"15min":2.044},"last":1.5880000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.411,"15min":1.4730000000000001},"min":{"1min":0.78900000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.905,"5min":1.9239999999999999,"15min":1.956},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.337,"15min":1.399},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.64800000000000002},"max":{"1min":1.706,"5min":1.869,"15min":2.0099999999999998},"last":1.3680000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2889999999999999,"15min":1.371},"min":{"1min":0.85499999999999998,"5min":0.70299999999999996,"15min":0.65900000000000003},"max":{"1min":1.754,"5min":1.9910000000000001,"15min":2.0539999999999998},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.429,"15min":1.466},"min":{"1min":1.0609999999999999,"5min":0.877,"15min":0.877},"max":{"1min":1.8560000000000001,"5min":2.0750000000000002,"15min":2.3540000000000001},"last":1.7190000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3979999999999999,"15min":1.4390000000000001},"min":{"1min":1.083,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.7929999999999999,"5min":1.998,"15min":2.3220000000000001},"last":1.536},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3500000000000001,"15min":1.4099999999999999},"min":{"1min":1.0009999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6319999999999999,"5min":2,"15min":2.3740000000000001},"last":1.512}]},{"osd":7,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4910000000000001,"15min":1.5329999999999999},"min":{"1min":0.94399999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.8720000000000001,"5min":2.157,"15min":2.431},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.522,"15min":1.5960000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.8540000000000001,"5min":2.1040000000000001,"15min":2.552},"last":1.77}]}]},{"osd":2,"up_from":14,"seq":60129542382,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46788,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325052,"statfs":{"total":96636764160,"available":96588853248,"internally_reserved":0,"allocated":1757184,"data_stored":842778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.86399999999999999,"15min":0.91800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.315},"max":{"1min":1.2709999999999999,"5min":1.508,"15min":1.7889999999999999},"last":0.629},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.0369999999999999,"15min":1.0800000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.41699999999999998},"max":{"1min":1.589,"5min":1.746,"15min":2.085},"last":0.97299999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.079,"15min":1.1299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.5149999999999999,"5min":1.677,"15min":2.5590000000000002},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.1779999999999999,"15min":1.2210000000000001},"min":{"1min":0.60599999999999998,"5min":0.56999999999999995,"15min":0.46500000000000002},"max":{"1min":1.73,"5min":2.347,"15min":2.4289999999999998},"last":1.0620000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0649999999999999,"15min":1.0840000000000001},"min":{"1min":0.54000000000000004,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":1.234},{"interface":"front","average":{"1min":1.123,"5min":1.26,"15min":1.3120000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.671,"5min":2.238,"15min":2.238},"last":0.82499999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3799999999999999,"15min":1.4490000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.8979999999999999,"5min":2.2149999999999999,"15min":2.431},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.3660000000000001,"15min":1.421},"min":{"1min":0.78400000000000003,"5min":0.75600000000000001,"15min":0.68999999999999995},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.1699999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.4650000000000001,"15min":1.504},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.8570000000000002,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.373,"15min":1.4179999999999999},"min":{"1min":0.755,"5min":0.751,"15min":0.52100000000000002},"max":{"1min":3.7759999999999998,"5min":3.7759999999999998,"15min":3.7759999999999998},"last":1.605}]},{"osd":6,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.4419999999999999,"15min":1.4450000000000001},"min":{"1min":0.89800000000000002,"5min":0.89400000000000002,"15min":0.59699999999999998},"max":{"1min":3.6120000000000001,"5min":3.6120000000000001,"15min":3.6120000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4810000000000001,"15min":1.5509999999999999},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.78000000000000003},"max":{"1min":1.8420000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.323}]},{"osd":7,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.548,"15min":1.6020000000000001},"min":{"1min":1.081,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4990000000000001,"15min":1.5389999999999999},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.56599999999999995},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":1.4810000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59280,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312560,"statfs":{"total":96636764160,"available":96576061440,"internally_reserved":0,"allocated":1765376,"data_stored":842902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.93500000000000005,"15min":0.90900000000000003},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2669999999999999,"5min":1.6359999999999999,"15min":2.0840000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0129999999999999,"15min":1.0700000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.5509999999999999,"5min":1.931,"15min":1.9770000000000001},"last":0.88200000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.95299999999999996,"15min":0.98399999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3280000000000001,"5min":1.694,"15min":2.262},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.135,"15min":1.175},"min":{"1min":0.57999999999999996,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":1.6639999999999999,"5min":1.9910000000000001,"15min":2.2210000000000001},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.272,"15min":1.298},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.46100000000000002},"max":{"1min":1.919,"5min":2.073,"15min":2.073},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1619999999999999,"15min":1.208},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.758,"5min":1.758,"15min":1.9199999999999999},"last":0.999}]},{"osd":4,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2410000000000001,"15min":1.294},"min":{"1min":0.60199999999999998,"5min":0.59599999999999997,"15min":0.54600000000000004},"max":{"1min":1.6200000000000001,"5min":2.032,"15min":2.1030000000000002},"last":1.429},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4259999999999999,"15min":1.4339999999999999},"min":{"1min":0.69499999999999995,"5min":0.67900000000000005,"15min":0.60599999999999998},"max":{"1min":2.3380000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.8560000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.393,"15min":1.4179999999999999},"min":{"1min":0.79700000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3939999999999999,"15min":1.429},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.786,"5min":2.1179999999999999,"15min":2.4350000000000001},"last":1.7410000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.85899999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"max":{"1min":2.4460000000000002,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.554},{"interface":"front","average":{"1min":1.577,"5min":1.494,"15min":1.5149999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.4390000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.5169999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.583,"15min":1.6100000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.669},{"interface":"front","average":{"1min":1.454,"5min":1.52,"15min":1.552},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8720000000000001,"5min":2.153,"15min":2.5710000000000002},"last":1.1850000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542383,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41820,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330020,"statfs":{"total":96636764160,"available":96593940480,"internally_reserved":0,"allocated":2371584,"data_stored":1434870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.879},"min":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.4470000000000001,"5min":1.655,"15min":1.6890000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.0409999999999999,"15min":1.079},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.752}]},{"osd":2,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0800000000000001,"15min":1.127},"min":{"1min":0.495,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.744,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.0680000000000001,"15min":1.151},"min":{"1min":0.497,"5min":0.497,"15min":0.496},"max":{"1min":1.518,"5min":1.788,"15min":1.952},"last":1.2410000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0069999999999999,"15min":1.079},"min":{"1min":0.435,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.9570000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.181,"15min":1.272},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.70399999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.2829999999999999,"15min":1.3320000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5579999999999998,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.3740000000000001,"15min":1.427},"min":{"1min":0.75,"5min":0.625,"15min":0.59699999999999998},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.173},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3500000000000001,"15min":1.415},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72499999999999998},"max":{"1min":1.8320000000000001,"5min":1.9630000000000001,"15min":2.2440000000000002},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.3500000000000001,"15min":1.407},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.58999999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.0270000000000001},"last":1.1970000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4470000000000001,"15min":1.516},"min":{"1min":0.65600000000000003,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.085,"5min":2.125,"15min":2.3519999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.397,"15min":1.5029999999999999},"min":{"1min":0.66900000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.8520000000000001,"5min":1.95,"15min":2.363},"last":1.073}]},{"osd":7,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.528,"15min":1.615},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.395,"5min":2.395,"15min":2.4729999999999999},"last":1.448},{"interface":"front","average":{"1min":1.482,"5min":1.4830000000000001,"15min":1.5449999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.1589999999999998,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:53:31.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:53:31.905 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T12:53:31.903+0000 7f4a46dfa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:53:32.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.070+0000 7f4160c9f640 1 Processor -- start 2023-10-16T12:53:32.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.070+0000 7f4160c9f640 1 -- start start 2023-10-16T12:53:32.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.072+0000 7f4160c9f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.072+0000 7f4160c9f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c15f4a0 con 0x7f415c0ad970 2023-10-16T12:53:32.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.072+0000 7f4160c9f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c06c960 con 0x7f415c0cda80 2023-10-16T12:53:32.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.072+0000 7f4160c9f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415c15b9d0 con 0x7f415c06c040 2023-10-16T12:53:32.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.072+0000 7f415affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f415c0ad970 0x7f415c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44806/0 (socket says 172.21.15.60:44806) 2023-10-16T12:53:32.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.072+0000 7f415affd640 1 -- 172.21.15.60:0/1417614941 learned_addr learned my addr 172.21.15.60:0/1417614941 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:32.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415b7fe640 1 --2- 172.21.15.60:0/1417614941 >> v2:172.21.15.144:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672509467 0 0) 0x7f415c15f4a0 con 0x7f415c0ad970 2023-10-16T12:53:32.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4148003520 con 0x7f415c0ad970 2023-10-16T12:53:32.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797898144 0 0) 0x7f4148003520 con 0x7f415c0ad970 2023-10-16T12:53:32.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 >> v2:172.21.15.144:3300/0 conn(0x7f415c06c040 msgr2=0x7f415c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 --2- 172.21.15.60:0/1417614941 >> v2:172.21.15.144:3300/0 conn(0x7f415c06c040 0x7f415c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 >> v1:172.21.15.144:6789/0 conn(0x7f415c0cda80 legacy=0x7f415c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c15bcb0 con 0x7f415c0ad970 2023-10-16T12:53:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4144001670 con 0x7f415c0ad970 2023-10-16T12:53:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 >> v1:172.21.15.60:6789/0 conn(0x7f415c0ad970 legacy=0x7f415c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4148003970 0x7f4148003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f4148004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4148005180 0x7f414800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c15b9d0 con 0x7f41480047c0 2023-10-16T12:53:32.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414800c020 con 0x7f4148003970 2023-10-16T12:53:32.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.073+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414800c1e0 con 0x7f4148005180 2023-10-16T12:53:32.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415affd640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f4148004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415affd640 1 -- 172.21.15.60:0/1417614941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4148005180 msgr2=0x7f414800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:32.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415affd640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4148005180 0x7f414800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415affd640 1 -- 172.21.15.60:0/1417614941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4148003970 msgr2=0x7f4148003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415affd640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4148003970 0x7f4148003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415affd640 1 -- 172.21.15.60:0/1417614941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4148003520 con 0x7f41480047c0 2023-10-16T12:53:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.074+0000 7f415bfff640 1 -- 172.21.15.60:0/1417614941 reap_dead start 2023-10-16T12:53:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.075+0000 7f415affd640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f4148004c40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f414400c210 tx=0x7f41480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2064402aabc2b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:32.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.075+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144005ad0 con 0x7f41480047c0 2023-10-16T12:53:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.075+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4144005c30 con 0x7f41480047c0 2023-10-16T12:53:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.075+0000 7f415a7fc640 1 -- 172.21.15.60:0/1417614941 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414402cad0 con 0x7f41480047c0 2023-10-16T12:53:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 -- 172.21.15.60:0/1417614941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 msgr2=0x7f4148004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f4148004c40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f414400c210 tx=0x7f41480057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 -- 172.21.15.60:0/1417614941 shutdown_connections 2023-10-16T12:53:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 --2- 172.21.15.60:0/1417614941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f4148004c40 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 -- 172.21.15.60:0/1417614941 >> 172.21.15.60:0/1417614941 conn(0x7f415c0ce6a0 msgr2=0x7f415c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 -- 172.21.15.60:0/1417614941 shutdown_connections 2023-10-16T12:53:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.076+0000 7f4160c9f640 1 -- 172.21.15.60:0/1417614941 wait complete. 2023-10-16T12:53:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.078+0000 7f4160c9f640 1 Processor -- start 2023-10-16T12:53:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.078+0000 7f4160c9f640 1 -- start start 2023-10-16T12:53:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f4160c9f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f415c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f4160c9f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c0c6180 0x7f415c0b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f4160c9f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f415c0ae640 0x7f415c0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f4160c9f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0c79b0 con 0x7f41480047c0 2023-10-16T12:53:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f4160c9f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0b0da0 con 0x7f415c0c6180 2023-10-16T12:53:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f4160c9f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0c6600 con 0x7f415c0ae640 2023-10-16T12:53:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f415b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f415c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f415b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f415c1f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44380/0 (socket says 172.21.15.60:44380) 2023-10-16T12:53:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.079+0000 7f415b7fe640 1 -- 172.21.15.60:0/949939652 learned_addr learned my addr 172.21.15.60:0/949939652 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415affd640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c0c6180 0x7f415c0b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415b7fe640 1 -- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f415c0ae640 msgr2=0x7f415c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415b7fe640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f415c0ae640 0x7f415c0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415b7fe640 1 -- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c0c6180 msgr2=0x7f415c0b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415b7fe640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c0c6180 0x7f415c0b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415b7fe640 1 -- 172.21.15.60:0/949939652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c1f6a70 con 0x7f41480047c0 2023-10-16T12:53:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f415b7fe640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f415c1f5b40 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f414c00e890 tx=0x7f414c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c00c940 con 0x7f41480047c0 2023-10-16T12:53:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c00cac0 con 0x7f41480047c0 2023-10-16T12:53:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c00ce70 con 0x7f41480047c0 2023-10-16T12:53:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.080+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c1f61d0 con 0x7f41480047c0 2023-10-16T12:53:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.081+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c1f6790 con 0x7f41480047c0 2023-10-16T12:53:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.082+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c06b330 con 0x7f41480047c0 2023-10-16T12:53:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.083+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f414c013470 con 0x7f41480047c0 2023-10-16T12:53:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.083+0000 7f4153fff640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f413c06afc0 0x7f413c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.084+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f414c0898d0 con 0x7f41480047c0 2023-10-16T12:53:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.084+0000 7f415affd640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f413c06afc0 0x7f413c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.085+0000 7f415affd640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f413c06afc0 0x7f413c06d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4144034000 tx=0x7f4144009030 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.088+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f414c0582f0 con 0x7f41480047c0 2023-10-16T12:53:32.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.271+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 --> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f415c0cc270 con 0x7f413c06afc0 2023-10-16T12:53:32.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.295+0000 7f4153fff640 1 -- 172.21.15.60:0/949939652 <== mgr.6886 v2:172.21.15.144:6832/682650079 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287236 (secure 0 0 0) 0x7f41440014f0 con 0x7f413c06afc0 2023-10-16T12:53:32.301 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:32.303 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:53:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.301+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f413c06afc0 msgr2=0x7f413c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.301+0000 7f4160c9f640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f413c06afc0 0x7f413c06d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4144034000 tx=0x7f4144009030 comp rx=0 tx=0).stop 2023-10-16T12:53:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 msgr2=0x7f415c1f5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f415c1f5b40 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f414c00e890 tx=0x7f414c00ed50 comp rx=0 tx=0).stop 2023-10-16T12:53:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 shutdown_connections 2023-10-16T12:53:32.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f413c06afc0 0x7f413c06d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f415c0ae640 0x7f415c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c0c6180 0x7f415c0b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 --2- 172.21.15.60:0/949939652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41480047c0 0x7f415c1f5b40 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 >> 172.21.15.60:0/949939652 conn(0x7f415c0ce6a0 msgr2=0x7f415c06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:32.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 shutdown_connections 2023-10-16T12:53:32.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.302+0000 7f4160c9f640 1 -- 172.21.15.60:0/949939652 wait complete. 2023-10-16T12:53:32.325 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T12:53:32.111845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":407,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356036,"kb_used_data":14988,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754618684,"statfs":{"total":773094113280,"available":772729532416,"internally_reserved":0,"allocated":15347712,"data_stored":7946680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":349097528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002261"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051574+0000","last_change":"2023-10-16T12:53:27.345574+0000","last_active":"2023-10-16T12:53:29.051574+0000","last_peered":"2023-10-16T12:53:29.051574+0000","last_clean":"2023-10-16T12:53:29.051574+0000","last_became_active":"2023-10-16T12:53:27.344341+0000","last_became_peered":"2023-10-16T12:53:27.344341+0000","last_unstale":"2023-10-16T12:53:29.051574+0000","last_undegraded":"2023-10-16T12:53:29.051574+0000","last_fullsized":"2023-10-16T12:53:29.051574+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:50.132329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335699+0000","last_change":"2023-10-16T12:53:27.330679+0000","last_active":"2023-10-16T12:53:28.335699+0000","last_peered":"2023-10-16T12:53:28.335699+0000","last_clean":"2023-10-16T12:53:28.335699+0000","last_became_active":"2023-10-16T12:53:27.330378+0000","last_became_peered":"2023-10-16T12:53:27.330378+0000","last_unstale":"2023-10-16T12:53:28.335699+0000","last_undegraded":"2023-10-16T12:53:28.335699+0000","last_fullsized":"2023-10-16T12:53:28.335699+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:22.932698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335718+0000","last_change":"2023-10-16T12:53:27.337785+0000","last_active":"2023-10-16T12:53:28.335718+0000","last_peered":"2023-10-16T12:53:28.335718+0000","last_clean":"2023-10-16T12:53:28.335718+0000","last_became_active":"2023-10-16T12:53:27.336305+0000","last_became_peered":"2023-10-16T12:53:27.336305+0000","last_unstale":"2023-10-16T12:53:28.335718+0000","last_undegraded":"2023-10-16T12:53:28.335718+0000","last_fullsized":"2023-10-16T12:53:28.335718+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:23.132944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050680+0000","last_change":"2023-10-16T12:53:27.346793+0000","last_active":"2023-10-16T12:53:29.050680+0000","last_peered":"2023-10-16T12:53:29.050680+0000","last_clean":"2023-10-16T12:53:29.050680+0000","last_became_active":"2023-10-16T12:53:27.346470+0000","last_became_peered":"2023-10-16T12:53:27.346470+0000","last_unstale":"2023-10-16T12:53:29.050680+0000","last_undegraded":"2023-10-16T12:53:29.050680+0000","last_fullsized":"2023-10-16T12:53:29.050680+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:38.066820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334803+0000","last_change":"2023-10-16T12:53:27.339652+0000","last_active":"2023-10-16T12:53:28.334803+0000","last_peered":"2023-10-16T12:53:28.334803+0000","last_clean":"2023-10-16T12:53:28.334803+0000","last_became_active":"2023-10-16T12:53:27.339303+0000","last_became_peered":"2023-10-16T12:53:27.339303+0000","last_unstale":"2023-10-16T12:53:28.334803+0000","last_undegraded":"2023-10-16T12:53:28.334803+0000","last_fullsized":"2023-10-16T12:53:28.334803+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:51.189014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604677+0000","last_change":"2023-10-16T12:53:27.332054+0000","last_active":"2023-10-16T12:53:28.604677+0000","last_peered":"2023-10-16T12:53:28.604677+0000","last_clean":"2023-10-16T12:53:28.604677+0000","last_became_active":"2023-10-16T12:53:27.331737+0000","last_became_peered":"2023-10-16T12:53:27.331737+0000","last_unstale":"2023-10-16T12:53:28.604677+0000","last_undegraded":"2023-10-16T12:53:28.604677+0000","last_fullsized":"2023-10-16T12:53:28.604677+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:46.675807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050886+0000","last_change":"2023-10-16T12:53:27.322436+0000","last_active":"2023-10-16T12:53:29.050886+0000","last_peered":"2023-10-16T12:53:29.050886+0000","last_clean":"2023-10-16T12:53:29.050886+0000","last_became_active":"2023-10-16T12:53:27.322103+0000","last_became_peered":"2023-10-16T12:53:27.322103+0000","last_unstale":"2023-10-16T12:53:29.050886+0000","last_undegraded":"2023-10-16T12:53:29.050886+0000","last_fullsized":"2023-10-16T12:53:29.050886+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:21.072014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335616+0000","last_change":"2023-10-16T12:53:27.325679+0000","last_active":"2023-10-16T12:53:28.335616+0000","last_peered":"2023-10-16T12:53:28.335616+0000","last_clean":"2023-10-16T12:53:28.335616+0000","last_became_active":"2023-10-16T12:53:27.325058+0000","last_became_peered":"2023-10-16T12:53:27.325058+0000","last_unstale":"2023-10-16T12:53:28.335616+0000","last_undegraded":"2023-10-16T12:53:28.335616+0000","last_fullsized":"2023-10-16T12:53:28.335616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:14.229391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051538+0000","last_change":"2023-10-16T12:53:27.349228+0000","last_active":"2023-10-16T12:53:29.051538+0000","last_peered":"2023-10-16T12:53:29.051538+0000","last_clean":"2023-10-16T12:53:29.051538+0000","last_became_active":"2023-10-16T12:53:27.348976+0000","last_became_peered":"2023-10-16T12:53:27.348976+0000","last_unstale":"2023-10-16T12:53:29.051538+0000","last_undegraded":"2023-10-16T12:53:29.051538+0000","last_fullsized":"2023-10-16T12:53:29.051538+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:16.198856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050962+0000","last_change":"2023-10-16T12:53:27.340712+0000","last_active":"2023-10-16T12:53:29.050962+0000","last_peered":"2023-10-16T12:53:29.050962+0000","last_clean":"2023-10-16T12:53:29.050962+0000","last_became_active":"2023-10-16T12:53:27.337374+0000","last_became_peered":"2023-10-16T12:53:27.337374+0000","last_unstale":"2023-10-16T12:53:29.050962+0000","last_undegraded":"2023-10-16T12:53:29.050962+0000","last_fullsized":"2023-10-16T12:53:29.050962+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:04.993657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335357+0000","last_change":"2023-10-16T12:53:27.337354+0000","last_active":"2023-10-16T12:53:28.335357+0000","last_peered":"2023-10-16T12:53:28.335357+0000","last_clean":"2023-10-16T12:53:28.335357+0000","last_became_active":"2023-10-16T12:53:27.337081+0000","last_became_peered":"2023-10-16T12:53:27.337081+0000","last_unstale":"2023-10-16T12:53:28.335357+0000","last_undegraded":"2023-10-16T12:53:28.335357+0000","last_fullsized":"2023-10-16T12:53:28.335357+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:49.116039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050970+0000","last_change":"2023-10-16T12:53:27.336650+0000","last_active":"2023-10-16T12:53:29.050970+0000","last_peered":"2023-10-16T12:53:29.050970+0000","last_clean":"2023-10-16T12:53:29.050970+0000","last_became_active":"2023-10-16T12:53:27.336341+0000","last_became_peered":"2023-10-16T12:53:27.336341+0000","last_unstale":"2023-10-16T12:53:29.050970+0000","last_undegraded":"2023-10-16T12:53:29.050970+0000","last_fullsized":"2023-10-16T12:53:29.050970+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:01.014572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335870+0000","last_change":"2023-10-16T12:53:27.339484+0000","last_active":"2023-10-16T12:53:28.335870+0000","last_peered":"2023-10-16T12:53:28.335870+0000","last_clean":"2023-10-16T12:53:28.335870+0000","last_became_active":"2023-10-16T12:53:27.339206+0000","last_became_peered":"2023-10-16T12:53:27.339206+0000","last_unstale":"2023-10-16T12:53:28.335870+0000","last_undegraded":"2023-10-16T12:53:28.335870+0000","last_fullsized":"2023-10-16T12:53:28.335870+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:30.995925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336364+0000","last_change":"2023-10-16T12:53:27.320397+0000","last_active":"2023-10-16T12:53:28.336364+0000","last_peered":"2023-10-16T12:53:28.336364+0000","last_clean":"2023-10-16T12:53:28.336364+0000","last_became_active":"2023-10-16T12:53:27.319827+0000","last_became_peered":"2023-10-16T12:53:27.319827+0000","last_unstale":"2023-10-16T12:53:28.336364+0000","last_undegraded":"2023-10-16T12:53:28.336364+0000","last_fullsized":"2023-10-16T12:53:28.336364+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:05.267379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605154+0000","last_change":"2023-10-16T12:53:27.325300+0000","last_active":"2023-10-16T12:53:28.605154+0000","last_peered":"2023-10-16T12:53:28.605154+0000","last_clean":"2023-10-16T12:53:28.605154+0000","last_became_active":"2023-10-16T12:53:27.324687+0000","last_became_peered":"2023-10-16T12:53:27.324687+0000","last_unstale":"2023-10-16T12:53:28.605154+0000","last_undegraded":"2023-10-16T12:53:28.605154+0000","last_fullsized":"2023-10-16T12:53:28.605154+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:33.012375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050265+0000","last_change":"2023-10-16T12:53:27.331540+0000","last_active":"2023-10-16T12:53:29.050265+0000","last_peered":"2023-10-16T12:53:29.050265+0000","last_clean":"2023-10-16T12:53:29.050265+0000","last_became_active":"2023-10-16T12:53:27.331237+0000","last_became_peered":"2023-10-16T12:53:27.331237+0000","last_unstale":"2023-10-16T12:53:29.050265+0000","last_undegraded":"2023-10-16T12:53:29.050265+0000","last_fullsized":"2023-10-16T12:53:29.050265+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:21.233513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335925+0000","last_change":"2023-10-16T12:53:27.341766+0000","last_active":"2023-10-16T12:53:28.335925+0000","last_peered":"2023-10-16T12:53:28.335925+0000","last_clean":"2023-10-16T12:53:28.335925+0000","last_became_active":"2023-10-16T12:53:27.341470+0000","last_became_peered":"2023-10-16T12:53:27.341470+0000","last_unstale":"2023-10-16T12:53:28.335925+0000","last_undegraded":"2023-10-16T12:53:28.335925+0000","last_fullsized":"2023-10-16T12:53:28.335925+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:34.908703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051517+0000","last_change":"2023-10-16T12:53:27.333817+0000","last_active":"2023-10-16T12:53:29.051517+0000","last_peered":"2023-10-16T12:53:29.051517+0000","last_clean":"2023-10-16T12:53:29.051517+0000","last_became_active":"2023-10-16T12:53:27.333546+0000","last_became_peered":"2023-10-16T12:53:27.333546+0000","last_unstale":"2023-10-16T12:53:29.051517+0000","last_undegraded":"2023-10-16T12:53:29.051517+0000","last_fullsized":"2023-10-16T12:53:29.051517+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:47.422545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335878+0000","last_change":"2023-10-16T12:53:27.328374+0000","last_active":"2023-10-16T12:53:28.335878+0000","last_peered":"2023-10-16T12:53:28.335878+0000","last_clean":"2023-10-16T12:53:28.335878+0000","last_became_active":"2023-10-16T12:53:27.326977+0000","last_became_peered":"2023-10-16T12:53:27.326977+0000","last_unstale":"2023-10-16T12:53:28.335878+0000","last_undegraded":"2023-10-16T12:53:28.335878+0000","last_fullsized":"2023-10-16T12:53:28.335878+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:45.236649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336217+0000","last_change":"2023-10-16T12:53:27.333119+0000","last_active":"2023-10-16T12:53:28.336217+0000","last_peered":"2023-10-16T12:53:28.336217+0000","last_clean":"2023-10-16T12:53:28.336217+0000","last_became_active":"2023-10-16T12:53:27.332819+0000","last_became_peered":"2023-10-16T12:53:27.332819+0000","last_unstale":"2023-10-16T12:53:28.336217+0000","last_undegraded":"2023-10-16T12:53:28.336217+0000","last_fullsized":"2023-10-16T12:53:28.336217+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:29.864314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335391+0000","last_change":"2023-10-16T12:53:27.340007+0000","last_active":"2023-10-16T12:53:28.335391+0000","last_peered":"2023-10-16T12:53:28.335391+0000","last_clean":"2023-10-16T12:53:28.335391+0000","last_became_active":"2023-10-16T12:53:27.339442+0000","last_became_peered":"2023-10-16T12:53:27.339442+0000","last_unstale":"2023-10-16T12:53:28.335391+0000","last_undegraded":"2023-10-16T12:53:28.335391+0000","last_fullsized":"2023-10-16T12:53:28.335391+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:42.794398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331416+0000","last_change":"2023-10-16T12:53:27.321043+0000","last_active":"2023-10-16T12:53:28.331416+0000","last_peered":"2023-10-16T12:53:28.331416+0000","last_clean":"2023-10-16T12:53:28.331416+0000","last_became_active":"2023-10-16T12:53:27.316935+0000","last_became_peered":"2023-10-16T12:53:27.316935+0000","last_unstale":"2023-10-16T12:53:28.331416+0000","last_undegraded":"2023-10-16T12:53:28.331416+0000","last_fullsized":"2023-10-16T12:53:28.331416+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:03.488589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334499+0000","last_change":"2023-10-16T12:53:27.332186+0000","last_active":"2023-10-16T12:53:28.334499+0000","last_peered":"2023-10-16T12:53:28.334499+0000","last_clean":"2023-10-16T12:53:28.334499+0000","last_became_active":"2023-10-16T12:53:27.331925+0000","last_became_peered":"2023-10-16T12:53:27.331925+0000","last_unstale":"2023-10-16T12:53:28.334499+0000","last_undegraded":"2023-10-16T12:53:28.334499+0000","last_fullsized":"2023-10-16T12:53:28.334499+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:04.294366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331687+0000","last_change":"2023-10-16T12:53:27.322639+0000","last_active":"2023-10-16T12:53:28.331687+0000","last_peered":"2023-10-16T12:53:28.331687+0000","last_clean":"2023-10-16T12:53:28.331687+0000","last_became_active":"2023-10-16T12:53:27.322323+0000","last_became_peered":"2023-10-16T12:53:27.322323+0000","last_unstale":"2023-10-16T12:53:28.331687+0000","last_undegraded":"2023-10-16T12:53:28.331687+0000","last_fullsized":"2023-10-16T12:53:28.331687+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:35.930393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336179+0000","last_change":"2023-10-16T12:53:27.338716+0000","last_active":"2023-10-16T12:53:28.336179+0000","last_peered":"2023-10-16T12:53:28.336179+0000","last_clean":"2023-10-16T12:53:28.336179+0000","last_became_active":"2023-10-16T12:53:27.338346+0000","last_became_peered":"2023-10-16T12:53:27.338346+0000","last_unstale":"2023-10-16T12:53:28.336179+0000","last_undegraded":"2023-10-16T12:53:28.336179+0000","last_fullsized":"2023-10-16T12:53:28.336179+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:42.069682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051306+0000","last_change":"2023-10-16T12:53:27.342504+0000","last_active":"2023-10-16T12:53:29.051306+0000","last_peered":"2023-10-16T12:53:29.051306+0000","last_clean":"2023-10-16T12:53:29.051306+0000","last_became_active":"2023-10-16T12:53:27.341754+0000","last_became_peered":"2023-10-16T12:53:27.341754+0000","last_unstale":"2023-10-16T12:53:29.051306+0000","last_undegraded":"2023-10-16T12:53:29.051306+0000","last_fullsized":"2023-10-16T12:53:29.051306+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:01.116602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605561+0000","last_change":"2023-10-16T12:53:27.343903+0000","last_active":"2023-10-16T12:53:28.605561+0000","last_peered":"2023-10-16T12:53:28.605561+0000","last_clean":"2023-10-16T12:53:28.605561+0000","last_became_active":"2023-10-16T12:53:27.343069+0000","last_became_peered":"2023-10-16T12:53:27.343069+0000","last_unstale":"2023-10-16T12:53:28.605561+0000","last_undegraded":"2023-10-16T12:53:28.605561+0000","last_fullsized":"2023-10-16T12:53:28.605561+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:25.553166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332452+0000","last_change":"2023-10-16T12:53:27.340160+0000","last_active":"2023-10-16T12:53:28.332452+0000","last_peered":"2023-10-16T12:53:28.332452+0000","last_clean":"2023-10-16T12:53:28.332452+0000","last_became_active":"2023-10-16T12:53:27.339680+0000","last_became_peered":"2023-10-16T12:53:27.339680+0000","last_unstale":"2023-10-16T12:53:28.332452+0000","last_undegraded":"2023-10-16T12:53:28.332452+0000","last_fullsized":"2023-10-16T12:53:28.332452+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:14.229391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334940+0000","last_change":"2023-10-16T12:53:27.332974+0000","last_active":"2023-10-16T12:53:28.334940+0000","last_peered":"2023-10-16T12:53:28.334940+0000","last_clean":"2023-10-16T12:53:28.334940+0000","last_became_active":"2023-10-16T12:53:27.332531+0000","last_became_peered":"2023-10-16T12:53:27.332531+0000","last_unstale":"2023-10-16T12:53:28.334940+0000","last_undegraded":"2023-10-16T12:53:28.334940+0000","last_fullsized":"2023-10-16T12:53:28.334940+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:36.059885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052318+0000","last_change":"2023-10-16T12:53:27.332884+0000","last_active":"2023-10-16T12:53:29.052318+0000","last_peered":"2023-10-16T12:53:29.052318+0000","last_clean":"2023-10-16T12:53:29.052318+0000","last_became_active":"2023-10-16T12:53:27.331591+0000","last_became_peered":"2023-10-16T12:53:27.331591+0000","last_unstale":"2023-10-16T12:53:29.052318+0000","last_undegraded":"2023-10-16T12:53:29.052318+0000","last_fullsized":"2023-10-16T12:53:29.052318+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:36.096973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050397+0000","last_change":"2023-10-16T12:53:27.333474+0000","last_active":"2023-10-16T12:53:29.050397+0000","last_peered":"2023-10-16T12:53:29.050397+0000","last_clean":"2023-10-16T12:53:29.050397+0000","last_became_active":"2023-10-16T12:53:27.333132+0000","last_became_peered":"2023-10-16T12:53:27.333132+0000","last_unstale":"2023-10-16T12:53:29.050397+0000","last_undegraded":"2023-10-16T12:53:29.050397+0000","last_fullsized":"2023-10-16T12:53:29.050397+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:28.268373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604479+0000","last_change":"2023-10-16T12:53:27.331337+0000","last_active":"2023-10-16T12:53:28.604479+0000","last_peered":"2023-10-16T12:53:28.604479+0000","last_clean":"2023-10-16T12:53:28.604479+0000","last_became_active":"2023-10-16T12:53:27.330944+0000","last_became_peered":"2023-10-16T12:53:27.330944+0000","last_unstale":"2023-10-16T12:53:28.604479+0000","last_undegraded":"2023-10-16T12:53:28.604479+0000","last_fullsized":"2023-10-16T12:53:28.604479+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:27.638445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605348+0000","last_change":"2023-10-16T12:53:27.328318+0000","last_active":"2023-10-16T12:53:28.605348+0000","last_peered":"2023-10-16T12:53:28.605348+0000","last_clean":"2023-10-16T12:53:28.605348+0000","last_became_active":"2023-10-16T12:53:27.328007+0000","last_became_peered":"2023-10-16T12:53:27.328007+0000","last_unstale":"2023-10-16T12:53:28.605348+0000","last_undegraded":"2023-10-16T12:53:28.605348+0000","last_fullsized":"2023-10-16T12:53:28.605348+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:24.518288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"142'4","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051705+0000","last_change":"2023-10-16T12:53:26.329697+0000","last_active":"2023-10-16T12:53:29.051705+0000","last_peered":"2023-10-16T12:53:29.051705+0000","last_clean":"2023-10-16T12:53:29.051705+0000","last_became_active":"2023-10-16T12:53:26.328260+0000","last_became_peered":"2023-10-16T12:53:26.328260+0000","last_unstale":"2023-10-16T12:53:29.051705+0000","last_undegraded":"2023-10-16T12:53:29.051705+0000","last_fullsized":"2023-10-16T12:53:29.051705+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:08.270888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052531+0000","last_change":"2023-10-16T12:53:27.334601+0000","last_active":"2023-10-16T12:53:29.052531+0000","last_peered":"2023-10-16T12:53:29.052531+0000","last_clean":"2023-10-16T12:53:29.052531+0000","last_became_active":"2023-10-16T12:53:27.330656+0000","last_became_peered":"2023-10-16T12:53:27.330656+0000","last_unstale":"2023-10-16T12:53:29.052531+0000","last_undegraded":"2023-10-16T12:53:29.052531+0000","last_fullsized":"2023-10-16T12:53:29.052531+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:28.134464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336447+0000","last_change":"2023-10-16T12:53:26.332967+0000","last_active":"2023-10-16T12:53:28.336447+0000","last_peered":"2023-10-16T12:53:28.336447+0000","last_clean":"2023-10-16T12:53:28.336447+0000","last_became_active":"2023-10-16T12:53:26.332651+0000","last_became_peered":"2023-10-16T12:53:26.332651+0000","last_unstale":"2023-10-16T12:53:28.336447+0000","last_undegraded":"2023-10-16T12:53:28.336447+0000","last_fullsized":"2023-10-16T12:53:28.336447+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:20.572277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605111+0000","last_change":"2023-10-16T12:53:27.322154+0000","last_active":"2023-10-16T12:53:28.605111+0000","last_peered":"2023-10-16T12:53:28.605111+0000","last_clean":"2023-10-16T12:53:28.605111+0000","last_became_active":"2023-10-16T12:53:27.321670+0000","last_became_peered":"2023-10-16T12:53:27.321670+0000","last_unstale":"2023-10-16T12:53:28.605111+0000","last_undegraded":"2023-10-16T12:53:28.605111+0000","last_fullsized":"2023-10-16T12:53:28.605111+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:51.189014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334977+0000","last_change":"2023-10-16T12:53:26.318315+0000","last_active":"2023-10-16T12:53:28.334977+0000","last_peered":"2023-10-16T12:53:28.334977+0000","last_clean":"2023-10-16T12:53:28.334977+0000","last_became_active":"2023-10-16T12:53:26.317997+0000","last_became_peered":"2023-10-16T12:53:26.317997+0000","last_unstale":"2023-10-16T12:53:28.334977+0000","last_undegraded":"2023-10-16T12:53:28.334977+0000","last_fullsized":"2023-10-16T12:53:28.334977+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:18.848088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336010+0000","last_change":"2023-10-16T12:53:27.327212+0000","last_active":"2023-10-16T12:53:28.336010+0000","last_peered":"2023-10-16T12:53:28.336010+0000","last_clean":"2023-10-16T12:53:28.336010+0000","last_became_active":"2023-10-16T12:53:27.326353+0000","last_became_peered":"2023-10-16T12:53:27.326353+0000","last_unstale":"2023-10-16T12:53:28.336010+0000","last_undegraded":"2023-10-16T12:53:28.336010+0000","last_fullsized":"2023-10-16T12:53:28.336010+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:42.888155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334811+0000","last_change":"2023-10-16T12:53:26.321761+0000","last_active":"2023-10-16T12:53:28.334811+0000","last_peered":"2023-10-16T12:53:28.334811+0000","last_clean":"2023-10-16T12:53:28.334811+0000","last_became_active":"2023-10-16T12:53:26.321456+0000","last_became_peered":"2023-10-16T12:53:26.321456+0000","last_unstale":"2023-10-16T12:53:28.334811+0000","last_undegraded":"2023-10-16T12:53:28.334811+0000","last_fullsized":"2023-10-16T12:53:28.334811+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:36.980313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050118+0000","last_change":"2023-10-16T12:53:27.338660+0000","last_active":"2023-10-16T12:53:29.050118+0000","last_peered":"2023-10-16T12:53:29.050118+0000","last_clean":"2023-10-16T12:53:29.050118+0000","last_became_active":"2023-10-16T12:53:27.338343+0000","last_became_peered":"2023-10-16T12:53:27.338343+0000","last_unstale":"2023-10-16T12:53:29.050118+0000","last_undegraded":"2023-10-16T12:53:29.050118+0000","last_fullsized":"2023-10-16T12:53:29.050118+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:25.553166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050486+0000","last_change":"2023-10-16T12:53:26.330036+0000","last_active":"2023-10-16T12:53:29.050486+0000","last_peered":"2023-10-16T12:53:29.050486+0000","last_clean":"2023-10-16T12:53:29.050486+0000","last_became_active":"2023-10-16T12:53:26.329693+0000","last_became_peered":"2023-10-16T12:53:26.329693+0000","last_unstale":"2023-10-16T12:53:29.050486+0000","last_undegraded":"2023-10-16T12:53:29.050486+0000","last_fullsized":"2023-10-16T12:53:29.050486+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:48.103832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335103+0000","last_change":"2023-10-16T12:53:27.333790+0000","last_active":"2023-10-16T12:53:28.335103+0000","last_peered":"2023-10-16T12:53:28.335103+0000","last_clean":"2023-10-16T12:53:28.335103+0000","last_became_active":"2023-10-16T12:53:27.333425+0000","last_became_peered":"2023-10-16T12:53:27.333425+0000","last_unstale":"2023-10-16T12:53:28.335103+0000","last_undegraded":"2023-10-16T12:53:28.335103+0000","last_fullsized":"2023-10-16T12:53:28.335103+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:24.692311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331611+0000","last_change":"2023-10-16T12:53:26.322834+0000","last_active":"2023-10-16T12:53:28.331611+0000","last_peered":"2023-10-16T12:53:28.331611+0000","last_clean":"2023-10-16T12:53:28.331611+0000","last_became_active":"2023-10-16T12:53:26.322529+0000","last_became_peered":"2023-10-16T12:53:26.322529+0000","last_unstale":"2023-10-16T12:53:28.331611+0000","last_undegraded":"2023-10-16T12:53:28.331611+0000","last_fullsized":"2023-10-16T12:53:28.331611+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:20.351796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050101+0000","last_change":"2023-10-16T12:53:27.328519+0000","last_active":"2023-10-16T12:53:29.050101+0000","last_peered":"2023-10-16T12:53:29.050101+0000","last_clean":"2023-10-16T12:53:29.050101+0000","last_became_active":"2023-10-16T12:53:27.327557+0000","last_became_peered":"2023-10-16T12:53:27.327557+0000","last_unstale":"2023-10-16T12:53:29.050101+0000","last_undegraded":"2023-10-16T12:53:29.050101+0000","last_fullsized":"2023-10-16T12:53:29.050101+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:12.070525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334449+0000","last_change":"2023-10-16T12:53:26.324360+0000","last_active":"2023-10-16T12:53:28.334449+0000","last_peered":"2023-10-16T12:53:28.334449+0000","last_clean":"2023-10-16T12:53:28.334449+0000","last_became_active":"2023-10-16T12:53:26.324078+0000","last_became_peered":"2023-10-16T12:53:26.324078+0000","last_unstale":"2023-10-16T12:53:28.334449+0000","last_undegraded":"2023-10-16T12:53:28.334449+0000","last_fullsized":"2023-10-16T12:53:28.334449+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:23.507818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050645+0000","last_change":"2023-10-16T12:53:27.332056+0000","last_active":"2023-10-16T12:53:29.050645+0000","last_peered":"2023-10-16T12:53:29.050645+0000","last_clean":"2023-10-16T12:53:29.050645+0000","last_became_active":"2023-10-16T12:53:27.331358+0000","last_became_peered":"2023-10-16T12:53:27.331358+0000","last_unstale":"2023-10-16T12:53:29.050645+0000","last_undegraded":"2023-10-16T12:53:29.050645+0000","last_fullsized":"2023-10-16T12:53:29.050645+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:44.060174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050419+0000","last_change":"2023-10-16T12:53:26.330844+0000","last_active":"2023-10-16T12:53:29.050419+0000","last_peered":"2023-10-16T12:53:29.050419+0000","last_clean":"2023-10-16T12:53:29.050419+0000","last_became_active":"2023-10-16T12:53:26.330044+0000","last_became_peered":"2023-10-16T12:53:26.330044+0000","last_unstale":"2023-10-16T12:53:29.050419+0000","last_undegraded":"2023-10-16T12:53:29.050419+0000","last_fullsized":"2023-10-16T12:53:29.050419+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:14.318434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051512+0000","last_change":"2023-10-16T12:53:27.328794+0000","last_active":"2023-10-16T12:53:29.051512+0000","last_peered":"2023-10-16T12:53:29.051512+0000","last_clean":"2023-10-16T12:53:29.051512+0000","last_became_active":"2023-10-16T12:53:27.328448+0000","last_became_peered":"2023-10-16T12:53:27.328448+0000","last_unstale":"2023-10-16T12:53:29.051512+0000","last_undegraded":"2023-10-16T12:53:29.051512+0000","last_fullsized":"2023-10-16T12:53:29.051512+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:08.580090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335028+0000","last_change":"2023-10-16T12:53:26.325980+0000","last_active":"2023-10-16T12:53:28.335028+0000","last_peered":"2023-10-16T12:53:28.335028+0000","last_clean":"2023-10-16T12:53:28.335028+0000","last_became_active":"2023-10-16T12:53:26.324482+0000","last_became_peered":"2023-10-16T12:53:26.324482+0000","last_unstale":"2023-10-16T12:53:28.335028+0000","last_undegraded":"2023-10-16T12:53:28.335028+0000","last_fullsized":"2023-10-16T12:53:28.335028+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:05.040015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052650+0000","last_change":"2023-10-16T12:53:27.336572+0000","last_active":"2023-10-16T12:53:29.052650+0000","last_peered":"2023-10-16T12:53:29.052650+0000","last_clean":"2023-10-16T12:53:29.052650+0000","last_became_active":"2023-10-16T12:53:27.336288+0000","last_became_peered":"2023-10-16T12:53:27.336288+0000","last_unstale":"2023-10-16T12:53:29.052650+0000","last_undegraded":"2023-10-16T12:53:29.052650+0000","last_fullsized":"2023-10-16T12:53:29.052650+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:58.627231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331928+0000","last_change":"2023-10-16T12:53:26.329852+0000","last_active":"2023-10-16T12:53:28.331928+0000","last_peered":"2023-10-16T12:53:28.331928+0000","last_clean":"2023-10-16T12:53:28.331928+0000","last_became_active":"2023-10-16T12:53:26.329561+0000","last_became_peered":"2023-10-16T12:53:26.329561+0000","last_unstale":"2023-10-16T12:53:28.331928+0000","last_undegraded":"2023-10-16T12:53:28.331928+0000","last_fullsized":"2023-10-16T12:53:28.331928+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:52.917224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332077+0000","last_change":"2023-10-16T12:53:27.339947+0000","last_active":"2023-10-16T12:53:28.332077+0000","last_peered":"2023-10-16T12:53:28.332077+0000","last_clean":"2023-10-16T12:53:28.332077+0000","last_became_active":"2023-10-16T12:53:27.339307+0000","last_became_peered":"2023-10-16T12:53:27.339307+0000","last_unstale":"2023-10-16T12:53:28.332077+0000","last_undegraded":"2023-10-16T12:53:28.332077+0000","last_fullsized":"2023-10-16T12:53:28.332077+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:05.373869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"142'3","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051496+0000","last_change":"2023-10-16T12:53:26.331155+0000","last_active":"2023-10-16T12:53:29.051496+0000","last_peered":"2023-10-16T12:53:29.051496+0000","last_clean":"2023-10-16T12:53:29.051496+0000","last_became_active":"2023-10-16T12:53:26.330750+0000","last_became_peered":"2023-10-16T12:53:26.330750+0000","last_unstale":"2023-10-16T12:53:29.051496+0000","last_undegraded":"2023-10-16T12:53:29.051496+0000","last_fullsized":"2023-10-16T12:53:29.051496+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:45.663599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051861+0000","last_change":"2023-10-16T12:53:27.329942+0000","last_active":"2023-10-16T12:53:29.051861+0000","last_peered":"2023-10-16T12:53:29.051861+0000","last_clean":"2023-10-16T12:53:29.051861+0000","last_became_active":"2023-10-16T12:53:27.329618+0000","last_became_peered":"2023-10-16T12:53:27.329618+0000","last_unstale":"2023-10-16T12:53:29.051861+0000","last_undegraded":"2023-10-16T12:53:29.051861+0000","last_fullsized":"2023-10-16T12:53:29.051861+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:18.796864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"142'6","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051271+0000","last_change":"2023-10-16T12:53:26.332212+0000","last_active":"2023-10-16T12:53:29.051271+0000","last_peered":"2023-10-16T12:53:29.051271+0000","last_clean":"2023-10-16T12:53:29.051271+0000","last_became_active":"2023-10-16T12:53:26.331931+0000","last_became_peered":"2023-10-16T12:53:26.331931+0000","last_unstale":"2023-10-16T12:53:29.051271+0000","last_undegraded":"2023-10-16T12:53:29.051271+0000","last_fullsized":"2023-10-16T12:53:29.051271+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:26.626238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331556+0000","last_change":"2023-10-16T12:53:27.336965+0000","last_active":"2023-10-16T12:53:28.331556+0000","last_peered":"2023-10-16T12:53:28.331556+0000","last_clean":"2023-10-16T12:53:28.331556+0000","last_became_active":"2023-10-16T12:53:27.336608+0000","last_became_peered":"2023-10-16T12:53:27.336608+0000","last_unstale":"2023-10-16T12:53:28.331556+0000","last_undegraded":"2023-10-16T12:53:28.331556+0000","last_fullsized":"2023-10-16T12:53:28.331556+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:16.313311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604808+0000","last_change":"2023-10-16T12:53:26.327068+0000","last_active":"2023-10-16T12:53:28.604808+0000","last_peered":"2023-10-16T12:53:28.604808+0000","last_clean":"2023-10-16T12:53:28.604808+0000","last_became_active":"2023-10-16T12:53:26.326806+0000","last_became_peered":"2023-10-16T12:53:26.326806+0000","last_unstale":"2023-10-16T12:53:28.604808+0000","last_undegraded":"2023-10-16T12:53:28.604808+0000","last_fullsized":"2023-10-16T12:53:28.604808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:19.202881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331502+0000","last_change":"2023-10-16T12:53:27.333771+0000","last_active":"2023-10-16T12:53:28.331502+0000","last_peered":"2023-10-16T12:53:28.331502+0000","last_clean":"2023-10-16T12:53:28.331502+0000","last_became_active":"2023-10-16T12:53:27.327719+0000","last_became_peered":"2023-10-16T12:53:27.327719+0000","last_unstale":"2023-10-16T12:53:28.331502+0000","last_undegraded":"2023-10-16T12:53:28.331502+0000","last_fullsized":"2023-10-16T12:53:28.331502+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:01.802458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052188+0000","last_change":"2023-10-16T12:53:26.331629+0000","last_active":"2023-10-16T12:53:29.052188+0000","last_peered":"2023-10-16T12:53:29.052188+0000","last_clean":"2023-10-16T12:53:29.052188+0000","last_became_active":"2023-10-16T12:53:26.331114+0000","last_became_peered":"2023-10-16T12:53:26.331114+0000","last_unstale":"2023-10-16T12:53:29.052188+0000","last_undegraded":"2023-10-16T12:53:29.052188+0000","last_fullsized":"2023-10-16T12:53:29.052188+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:58.061463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331800+0000","last_change":"2023-10-16T12:53:27.338968+0000","last_active":"2023-10-16T12:53:28.331800+0000","last_peered":"2023-10-16T12:53:28.331800+0000","last_clean":"2023-10-16T12:53:28.331800+0000","last_became_active":"2023-10-16T12:53:27.338699+0000","last_became_peered":"2023-10-16T12:53:27.338699+0000","last_unstale":"2023-10-16T12:53:28.331800+0000","last_undegraded":"2023-10-16T12:53:28.331800+0000","last_fullsized":"2023-10-16T12:53:28.331800+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:55.705002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"142'8","reported_seq":20,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334493+0000","last_change":"2023-10-16T12:53:26.323856+0000","last_active":"2023-10-16T12:53:28.334493+0000","last_peered":"2023-10-16T12:53:28.334493+0000","last_clean":"2023-10-16T12:53:28.334493+0000","last_became_active":"2023-10-16T12:53:26.323543+0000","last_became_peered":"2023-10-16T12:53:26.323543+0000","last_unstale":"2023-10-16T12:53:28.334493+0000","last_undegraded":"2023-10-16T12:53:28.334493+0000","last_fullsized":"2023-10-16T12:53:28.334493+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:58.885993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335138+0000","last_change":"2023-10-16T12:53:27.337936+0000","last_active":"2023-10-16T12:53:28.335138+0000","last_peered":"2023-10-16T12:53:28.335138+0000","last_clean":"2023-10-16T12:53:28.335138+0000","last_became_active":"2023-10-16T12:53:27.337636+0000","last_became_peered":"2023-10-16T12:53:27.337636+0000","last_unstale":"2023-10-16T12:53:28.335138+0000","last_undegraded":"2023-10-16T12:53:28.335138+0000","last_fullsized":"2023-10-16T12:53:28.335138+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:33.012375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052045+0000","last_change":"2023-10-16T12:53:26.324486+0000","last_active":"2023-10-16T12:53:29.052045+0000","last_peered":"2023-10-16T12:53:29.052045+0000","last_clean":"2023-10-16T12:53:29.052045+0000","last_became_active":"2023-10-16T12:53:26.324210+0000","last_became_peered":"2023-10-16T12:53:26.324210+0000","last_unstale":"2023-10-16T12:53:29.052045+0000","last_undegraded":"2023-10-16T12:53:29.052045+0000","last_fullsized":"2023-10-16T12:53:29.052045+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:32.000168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.605452+0000","last_change":"2023-10-16T12:53:27.341475+0000","last_active":"2023-10-16T12:53:28.605452+0000","last_peered":"2023-10-16T12:53:28.605452+0000","last_clean":"2023-10-16T12:53:28.605452+0000","last_became_active":"2023-10-16T12:53:27.341143+0000","last_became_peered":"2023-10-16T12:53:27.341143+0000","last_unstale":"2023-10-16T12:53:28.605452+0000","last_undegraded":"2023-10-16T12:53:28.605452+0000","last_fullsized":"2023-10-16T12:53:28.605452+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:12.070525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335124+0000","last_change":"2023-10-16T12:53:26.319389+0000","last_active":"2023-10-16T12:53:28.335124+0000","last_peered":"2023-10-16T12:53:28.335124+0000","last_clean":"2023-10-16T12:53:28.335124+0000","last_became_active":"2023-10-16T12:53:26.319048+0000","last_became_peered":"2023-10-16T12:53:26.319048+0000","last_unstale":"2023-10-16T12:53:28.335124+0000","last_undegraded":"2023-10-16T12:53:28.335124+0000","last_fullsized":"2023-10-16T12:53:28.335124+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:26.626238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.337121+0000","last_change":"2023-10-16T12:53:27.341295+0000","last_active":"2023-10-16T12:53:28.337121+0000","last_peered":"2023-10-16T12:53:28.337121+0000","last_clean":"2023-10-16T12:53:28.337121+0000","last_became_active":"2023-10-16T12:53:27.340978+0000","last_became_peered":"2023-10-16T12:53:27.340978+0000","last_unstale":"2023-10-16T12:53:28.337121+0000","last_undegraded":"2023-10-16T12:53:28.337121+0000","last_fullsized":"2023-10-16T12:53:28.337121+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:48.136247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"142'1","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052272+0000","last_change":"2023-10-16T12:53:26.323329+0000","last_active":"2023-10-16T12:53:29.052272+0000","last_peered":"2023-10-16T12:53:29.052272+0000","last_clean":"2023-10-16T12:53:29.052272+0000","last_became_active":"2023-10-16T12:53:26.322383+0000","last_became_peered":"2023-10-16T12:53:26.322383+0000","last_unstale":"2023-10-16T12:53:29.052272+0000","last_undegraded":"2023-10-16T12:53:29.052272+0000","last_fullsized":"2023-10-16T12:53:29.052272+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:05.679613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336493+0000","last_change":"2023-10-16T12:53:27.331450+0000","last_active":"2023-10-16T12:53:28.336493+0000","last_peered":"2023-10-16T12:53:28.336493+0000","last_clean":"2023-10-16T12:53:28.336493+0000","last_became_active":"2023-10-16T12:53:27.330380+0000","last_became_peered":"2023-10-16T12:53:27.330380+0000","last_unstale":"2023-10-16T12:53:28.336493+0000","last_undegraded":"2023-10-16T12:53:28.336493+0000","last_fullsized":"2023-10-16T12:53:28.336493+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:23:31.970824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050742+0000","last_change":"2023-10-16T12:53:26.325128+0000","last_active":"2023-10-16T12:53:29.050742+0000","last_peered":"2023-10-16T12:53:29.050742+0000","last_clean":"2023-10-16T12:53:29.050742+0000","last_became_active":"2023-10-16T12:53:26.323701+0000","last_became_peered":"2023-10-16T12:53:26.323701+0000","last_unstale":"2023-10-16T12:53:29.050742+0000","last_undegraded":"2023-10-16T12:53:29.050742+0000","last_fullsized":"2023-10-16T12:53:29.050742+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:35.047678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336771+0000","last_change":"2023-10-16T12:53:27.337914+0000","last_active":"2023-10-16T12:53:28.336771+0000","last_peered":"2023-10-16T12:53:28.336771+0000","last_clean":"2023-10-16T12:53:28.336771+0000","last_became_active":"2023-10-16T12:53:27.336096+0000","last_became_peered":"2023-10-16T12:53:27.336096+0000","last_unstale":"2023-10-16T12:53:28.336771+0000","last_undegraded":"2023-10-16T12:53:28.336771+0000","last_fullsized":"2023-10-16T12:53:28.336771+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:18.976350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604813+0000","last_change":"2023-10-16T12:53:26.313637+0000","last_active":"2023-10-16T12:53:28.604813+0000","last_peered":"2023-10-16T12:53:28.604813+0000","last_clean":"2023-10-16T12:53:28.604813+0000","last_became_active":"2023-10-16T12:53:26.313321+0000","last_became_peered":"2023-10-16T12:53:26.313321+0000","last_unstale":"2023-10-16T12:53:28.604813+0000","last_undegraded":"2023-10-16T12:53:28.604813+0000","last_fullsized":"2023-10-16T12:53:28.604813+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:05.040015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050131+0000","last_change":"2023-10-16T12:53:27.341847+0000","last_active":"2023-10-16T12:53:29.050131+0000","last_peered":"2023-10-16T12:53:29.050131+0000","last_clean":"2023-10-16T12:53:29.050131+0000","last_became_active":"2023-10-16T12:53:27.341572+0000","last_became_peered":"2023-10-16T12:53:27.341572+0000","last_unstale":"2023-10-16T12:53:29.050131+0000","last_undegraded":"2023-10-16T12:53:29.050131+0000","last_fullsized":"2023-10-16T12:53:29.050131+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:12.628278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334515+0000","last_change":"2023-10-16T12:53:26.321570+0000","last_active":"2023-10-16T12:53:28.334515+0000","last_peered":"2023-10-16T12:53:28.334515+0000","last_clean":"2023-10-16T12:53:28.334515+0000","last_became_active":"2023-10-16T12:53:26.321303+0000","last_became_peered":"2023-10-16T12:53:26.321303+0000","last_unstale":"2023-10-16T12:53:28.334515+0000","last_undegraded":"2023-10-16T12:53:28.334515+0000","last_fullsized":"2023-10-16T12:53:28.334515+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:36.179677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336152+0000","last_change":"2023-10-16T12:53:27.338969+0000","last_active":"2023-10-16T12:53:28.336152+0000","last_peered":"2023-10-16T12:53:28.336152+0000","last_clean":"2023-10-16T12:53:28.336152+0000","last_became_active":"2023-10-16T12:53:27.338711+0000","last_became_peered":"2023-10-16T12:53:27.338711+0000","last_unstale":"2023-10-16T12:53:28.336152+0000","last_undegraded":"2023-10-16T12:53:28.336152+0000","last_fullsized":"2023-10-16T12:53:28.336152+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:25.778923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.604430+0000","last_change":"2023-10-16T12:53:26.320432+0000","last_active":"2023-10-16T12:53:28.604430+0000","last_peered":"2023-10-16T12:53:28.604430+0000","last_clean":"2023-10-16T12:53:28.604430+0000","last_became_active":"2023-10-16T12:53:26.320108+0000","last_became_peered":"2023-10-16T12:53:26.320108+0000","last_unstale":"2023-10-16T12:53:28.604430+0000","last_undegraded":"2023-10-16T12:53:28.604430+0000","last_fullsized":"2023-10-16T12:53:28.604430+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:36.980313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332147+0000","last_change":"2023-10-16T12:53:27.335791+0000","last_active":"2023-10-16T12:53:28.332147+0000","last_peered":"2023-10-16T12:53:28.332147+0000","last_clean":"2023-10-16T12:53:28.332147+0000","last_became_active":"2023-10-16T12:53:27.335489+0000","last_became_peered":"2023-10-16T12:53:27.335489+0000","last_unstale":"2023-10-16T12:53:28.332147+0000","last_undegraded":"2023-10-16T12:53:28.332147+0000","last_fullsized":"2023-10-16T12:53:28.332147+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:28:32.708283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050832+0000","last_change":"2023-10-16T12:53:26.330049+0000","last_active":"2023-10-16T12:53:29.050832+0000","last_peered":"2023-10-16T12:53:29.050832+0000","last_clean":"2023-10-16T12:53:29.050832+0000","last_became_active":"2023-10-16T12:53:26.329792+0000","last_became_peered":"2023-10-16T12:53:26.329792+0000","last_unstale":"2023-10-16T12:53:29.050832+0000","last_undegraded":"2023-10-16T12:53:29.050832+0000","last_fullsized":"2023-10-16T12:53:29.050832+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:27:15.707126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051576+0000","last_change":"2023-10-16T12:53:27.338277+0000","last_active":"2023-10-16T12:53:29.051576+0000","last_peered":"2023-10-16T12:53:29.051576+0000","last_clean":"2023-10-16T12:53:29.051576+0000","last_became_active":"2023-10-16T12:53:27.337928+0000","last_became_peered":"2023-10-16T12:53:27.337928+0000","last_unstale":"2023-10-16T12:53:29.051576+0000","last_undegraded":"2023-10-16T12:53:29.051576+0000","last_fullsized":"2023-10-16T12:53:29.051576+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:50.337210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"142'2","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331982+0000","last_change":"2023-10-16T12:53:26.333641+0000","last_active":"2023-10-16T12:53:28.331982+0000","last_peered":"2023-10-16T12:53:28.331982+0000","last_clean":"2023-10-16T12:53:28.331982+0000","last_became_active":"2023-10-16T12:53:26.333282+0000","last_became_peered":"2023-10-16T12:53:26.333282+0000","last_unstale":"2023-10-16T12:53:28.331982+0000","last_undegraded":"2023-10-16T12:53:28.331982+0000","last_fullsized":"2023-10-16T12:53:28.331982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:04.438933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051942+0000","last_change":"2023-10-16T12:53:27.345392+0000","last_active":"2023-10-16T12:53:29.051942+0000","last_peered":"2023-10-16T12:53:29.051942+0000","last_clean":"2023-10-16T12:53:29.051942+0000","last_became_active":"2023-10-16T12:53:27.343476+0000","last_became_peered":"2023-10-16T12:53:27.343476+0000","last_unstale":"2023-10-16T12:53:29.051942+0000","last_undegraded":"2023-10-16T12:53:29.051942+0000","last_fullsized":"2023-10-16T12:53:29.051942+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:33.971723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051758+0000","last_change":"2023-10-16T12:53:26.328849+0000","last_active":"2023-10-16T12:53:29.051758+0000","last_peered":"2023-10-16T12:53:29.051758+0000","last_clean":"2023-10-16T12:53:29.051758+0000","last_became_active":"2023-10-16T12:53:26.326676+0000","last_became_peered":"2023-10-16T12:53:26.326676+0000","last_unstale":"2023-10-16T12:53:29.051758+0000","last_undegraded":"2023-10-16T12:53:29.051758+0000","last_fullsized":"2023-10-16T12:53:29.051758+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:48.372259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051082+0000","last_change":"2023-10-16T12:53:27.334448+0000","last_active":"2023-10-16T12:53:29.051082+0000","last_peered":"2023-10-16T12:53:29.051082+0000","last_clean":"2023-10-16T12:53:29.051082+0000","last_became_active":"2023-10-16T12:53:27.334158+0000","last_became_peered":"2023-10-16T12:53:27.334158+0000","last_unstale":"2023-10-16T12:53:29.051082+0000","last_undegraded":"2023-10-16T12:53:29.051082+0000","last_fullsized":"2023-10-16T12:53:29.051082+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:41.302841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051067+0000","last_change":"2023-10-16T12:53:26.327304+0000","last_active":"2023-10-16T12:53:29.051067+0000","last_peered":"2023-10-16T12:53:29.051067+0000","last_clean":"2023-10-16T12:53:29.051067+0000","last_became_active":"2023-10-16T12:53:26.326184+0000","last_became_peered":"2023-10-16T12:53:26.326184+0000","last_unstale":"2023-10-16T12:53:29.051067+0000","last_undegraded":"2023-10-16T12:53:29.051067+0000","last_fullsized":"2023-10-16T12:53:29.051067+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:56:50.176807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.052052+0000","last_change":"2023-10-16T12:53:27.343350+0000","last_active":"2023-10-16T12:53:29.052052+0000","last_peered":"2023-10-16T12:53:29.052052+0000","last_clean":"2023-10-16T12:53:29.052052+0000","last_became_active":"2023-10-16T12:53:27.342994+0000","last_became_peered":"2023-10-16T12:53:27.342994+0000","last_unstale":"2023-10-16T12:53:29.052052+0000","last_undegraded":"2023-10-16T12:53:29.052052+0000","last_fullsized":"2023-10-16T12:53:29.052052+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:38.391019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.332165+0000","last_change":"2023-10-16T12:53:26.320506+0000","last_active":"2023-10-16T12:53:28.332165+0000","last_peered":"2023-10-16T12:53:28.332165+0000","last_clean":"2023-10-16T12:53:28.332165+0000","last_became_active":"2023-10-16T12:53:26.320244+0000","last_became_peered":"2023-10-16T12:53:26.320244+0000","last_unstale":"2023-10-16T12:53:28.332165+0000","last_undegraded":"2023-10-16T12:53:28.332165+0000","last_fullsized":"2023-10-16T12:53:28.332165+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:19.794914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334857+0000","last_change":"2023-10-16T12:53:27.334390+0000","last_active":"2023-10-16T12:53:28.334857+0000","last_peered":"2023-10-16T12:53:28.334857+0000","last_clean":"2023-10-16T12:53:28.334857+0000","last_became_active":"2023-10-16T12:53:27.334114+0000","last_became_peered":"2023-10-16T12:53:27.334114+0000","last_unstale":"2023-10-16T12:53:28.334857+0000","last_undegraded":"2023-10-16T12:53:28.334857+0000","last_fullsized":"2023-10-16T12:53:28.334857+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:24.204906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050755+0000","last_change":"2023-10-16T12:53:26.323111+0000","last_active":"2023-10-16T12:53:29.050755+0000","last_peered":"2023-10-16T12:53:29.050755+0000","last_clean":"2023-10-16T12:53:29.050755+0000","last_became_active":"2023-10-16T12:53:26.321661+0000","last_became_peered":"2023-10-16T12:53:26.321661+0000","last_unstale":"2023-10-16T12:53:29.050755+0000","last_undegraded":"2023-10-16T12:53:29.050755+0000","last_fullsized":"2023-10-16T12:53:29.050755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:41.782191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.336217+0000","last_change":"2023-10-16T12:53:27.335997+0000","last_active":"2023-10-16T12:53:28.336217+0000","last_peered":"2023-10-16T12:53:28.336217+0000","last_clean":"2023-10-16T12:53:28.336217+0000","last_became_active":"2023-10-16T12:53:27.335727+0000","last_became_peered":"2023-10-16T12:53:27.335727+0000","last_unstale":"2023-10-16T12:53:28.336217+0000","last_undegraded":"2023-10-16T12:53:28.336217+0000","last_fullsized":"2023-10-16T12:53:28.336217+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:09.283095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334612+0000","last_change":"2023-10-16T12:53:26.325158+0000","last_active":"2023-10-16T12:53:28.334612+0000","last_peered":"2023-10-16T12:53:28.334612+0000","last_clean":"2023-10-16T12:53:28.334612+0000","last_became_active":"2023-10-16T12:53:26.323556+0000","last_became_peered":"2023-10-16T12:53:26.323556+0000","last_unstale":"2023-10-16T12:53:28.334612+0000","last_undegraded":"2023-10-16T12:53:28.334612+0000","last_fullsized":"2023-10-16T12:53:28.334612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:29.610305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.335793+0000","last_change":"2023-10-16T12:53:27.323690+0000","last_active":"2023-10-16T12:53:28.335793+0000","last_peered":"2023-10-16T12:53:28.335793+0000","last_clean":"2023-10-16T12:53:28.335793+0000","last_became_active":"2023-10-16T12:53:27.323161+0000","last_became_peered":"2023-10-16T12:53:27.323161+0000","last_unstale":"2023-10-16T12:53:28.335793+0000","last_undegraded":"2023-10-16T12:53:28.335793+0000","last_fullsized":"2023-10-16T12:53:28.335793+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:03.587489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.334625+0000","last_change":"2023-10-16T12:53:26.325356+0000","last_active":"2023-10-16T12:53:28.334625+0000","last_peered":"2023-10-16T12:53:28.334625+0000","last_clean":"2023-10-16T12:53:28.334625+0000","last_became_active":"2023-10-16T12:53:26.323762+0000","last_became_peered":"2023-10-16T12:53:26.323762+0000","last_unstale":"2023-10-16T12:53:28.334625+0000","last_undegraded":"2023-10-16T12:53:28.334625+0000","last_fullsized":"2023-10-16T12:53:28.334625+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:55.640614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.051428+0000","last_change":"2023-10-16T12:53:27.332740+0000","last_active":"2023-10-16T12:53:29.051428+0000","last_peered":"2023-10-16T12:53:29.051428+0000","last_clean":"2023-10-16T12:53:29.051428+0000","last_became_active":"2023-10-16T12:53:27.332210+0000","last_became_peered":"2023-10-16T12:53:27.332210+0000","last_unstale":"2023-10-16T12:53:29.051428+0000","last_undegraded":"2023-10-16T12:53:29.051428+0000","last_fullsized":"2023-10-16T12:53:29.051428+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:11.719188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050738+0000","last_change":"2023-10-16T12:53:26.328402+0000","last_active":"2023-10-16T12:53:29.050738+0000","last_peered":"2023-10-16T12:53:29.050738+0000","last_clean":"2023-10-16T12:53:29.050738+0000","last_became_active":"2023-10-16T12:53:26.327462+0000","last_became_peered":"2023-10-16T12:53:26.327462+0000","last_unstale":"2023-10-16T12:53:29.050738+0000","last_undegraded":"2023-10-16T12:53:29.050738+0000","last_fullsized":"2023-10-16T12:53:29.050738+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:44:15.751250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:28.331539+0000","last_change":"2023-10-16T12:53:27.327831+0000","last_active":"2023-10-16T12:53:28.331539+0000","last_peered":"2023-10-16T12:53:28.331539+0000","last_clean":"2023-10-16T12:53:28.331539+0000","last_became_active":"2023-10-16T12:53:27.327499+0000","last_became_peered":"2023-10-16T12:53:27.327499+0000","last_unstale":"2023-10-16T12:53:28.331539+0000","last_undegraded":"2023-10-16T12:53:28.331539+0000","last_fullsized":"2023-10-16T12:53:28.331539+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:26.277881+0000","last_clean_scrub_stamp":"2023-10-16T12:53:26.277881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:21.584484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-16T12:53:29.050915+0000","last_change":"2023-10-16T12:53:26.330806+0000","last_active":"2023-10-16T12:53:29.050915+0000","last_peered":"2023-10-16T12:53:29.050915+0000","last_clean":"2023-10-16T12:53:29.050915+0000","last_became_active":"2023-10-16T12:53:26.330515+0000","last_became_peered":"2023-10-16T12:53:26.330515+0000","last_unstale":"2023-10-16T12:53:29.050915+0000","last_undegraded":"2023-10-16T12:53:29.050915+0000","last_fullsized":"2023-10-16T12:53:29.050915+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:53:25.265674+0000","last_clean_scrub_stamp":"2023-10-16T12:53:25.265674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:28.852107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"131'230","reported_seq":679,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:53:27.314270+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:53:27.314270+0000","last_peered":"2023-10-16T12:53:27.314270+0000","last_clean":"2023-10-16T12:53:27.314270+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:53:27.314270+0000","last_undegraded":"2023-10-16T12:53:27.314270+0000","last_fullsized":"2023-10-16T12:53:27.314270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1099,"num_read_kb":2242,"num_write":363,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33496,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338344,"statfs":{"total":96636764160,"available":96602464256,"internally_reserved":0,"allocated":2371584,"data_stored":1440062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.2889999999999999,"15min":1.3069999999999999},"min":{"1min":0.72499999999999998,"5min":0.68200000000000005,"15min":0.66100000000000003},"max":{"1min":1.835,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3300000000000001,"15min":1.3520000000000001},"min":{"1min":0.91000000000000003,"5min":0.76100000000000001,"15min":0.63200000000000001},"max":{"1min":1.804,"5min":1.804,"15min":1.8899999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.373,"15min":1.2689999999999999},"min":{"1min":0.69599999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.0840000000000001,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2170000000000001,"15min":1.224},"min":{"1min":0.79100000000000004,"5min":0.64800000000000002,"15min":0.64700000000000002},"max":{"1min":1.9239999999999999,"5min":5.5490000000000004,"15min":5.5490000000000004},"last":0.876}]},{"osd":2,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.179,"15min":1.1970000000000001},"min":{"1min":0.77900000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6220000000000001,"5min":2.855,"15min":2.855},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.262,"5min":1.278,"15min":1.319},"min":{"1min":0.81399999999999995,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6579999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.98799999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.3560000000000001,"15min":1.3180000000000001},"min":{"1min":1.0740000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.724,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.371,"15min":1.391},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5720000000000001,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.0920000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1890000000000001,"15min":1.1859999999999999},"min":{"1min":0.81799999999999995,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4339999999999999,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.3700000000000001,"15min":1.345},"min":{"1min":1.036,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.8169999999999999,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.2230000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3560000000000001,"15min":1.337},"min":{"1min":0.871,"5min":0.624,"15min":0.624},"max":{"1min":1.772,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.589,"5min":1.5269999999999999,"15min":1.4870000000000001},"min":{"1min":1.034,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0270000000000001,"5min":2.411,"15min":2.411},"last":1.2989999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:52:44 2023","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.613,"15min":1.5800000000000001},"min":{"1min":1.1759999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.194,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.341},{"interface":"front","average":{"1min":1.395,"5min":1.47,"15min":1.452},"min":{"1min":0.83399999999999996,"5min":0.75,"15min":0.75},"max":{"1min":1.752,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.2689999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542383,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1761280,"data_stored":842848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2390000000000001,"15min":1.282},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.7370000000000001,"5min":1.952,"15min":3.7450000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.331,"5min":1.302,"15min":1.329},"min":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":3.819},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.165,"15min":1.1850000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.5620000000000001,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.204,"5min":1.2150000000000001,"15min":1.214},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.625},"max":{"1min":1.887,"5min":1.9330000000000001,"15min":3.8519999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2110000000000001,"15min":1.21},"min":{"1min":0.80500000000000005,"5min":0.66900000000000004,"15min":0.627},"max":{"1min":1.5409999999999999,"5min":1.752,"15min":1.8899999999999999},"last":0.746},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.224,"15min":1.266},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5149999999999999,"5min":1.8520000000000001,"15min":3.9009999999999998},"last":0.90300000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.304,"15min":1.3260000000000001},"min":{"1min":0.92400000000000004,"5min":0.85199999999999998,"15min":0.80100000000000005},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":4.2380000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.367,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.85099999999999998,"5min":0.82199999999999995,"15min":0.69599999999999995},"max":{"1min":1.73,"5min":1.806,"15min":4.4269999999999996},"last":0.84499999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.375,"15min":1.3440000000000001},"min":{"1min":0.78100000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.3029999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":1.107,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.843,"5min":1.9430000000000001,"15min":2.2189999999999999},"last":0.94799999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.3180000000000001,"15min":1.3049999999999999},"min":{"1min":0.92600000000000005,"5min":0.755,"15min":0.755},"max":{"1min":1.6899999999999999,"5min":1.9830000000000001,"15min":2.0910000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.369,"5min":1.458,"15min":1.4410000000000001},"min":{"1min":0.872,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9830000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.78300000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4610000000000001,"15min":1.446},"min":{"1min":1.0089999999999999,"5min":0.878,"15min":0.878},"max":{"1min":2.0470000000000002,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4890000000000001,"15min":1.468},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":0.96399999999999997},"max":{"1min":1.8049999999999999,"5min":2.052,"15min":2.2610000000000001},"last":0.97599999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":843382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.113,"15min":1.153},"min":{"1min":0.69599999999999995,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.538,"5min":1.8480000000000001,"15min":2.2690000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.181,"5min":1.179,"15min":1.1759999999999999},"min":{"1min":0.76700000000000002,"5min":0.65200000000000002,"15min":0.63300000000000001},"max":{"1min":1.631,"5min":1.6459999999999999,"15min":1.7689999999999999},"last":1.228}]},{"osd":1,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2869999999999999,"15min":1.262},"min":{"1min":0.77800000000000002,"5min":0.56699999999999995,"15min":0.41299999999999998},"max":{"1min":1.665,"5min":1.8080000000000001,"15min":1.8089999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.1859999999999999,"15min":1.2050000000000001},"min":{"1min":0.77000000000000002,"5min":0.46000000000000002,"15min":0.41599999999999998},"max":{"1min":1.6519999999999999,"5min":1.738,"15min":1.738},"last":0.95699999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.278,"15min":1.2849999999999999},"min":{"1min":0.86099999999999999,"5min":0.60799999999999998,"15min":0.60199999999999998},"max":{"1min":1.6060000000000001,"5min":1.9710000000000001,"15min":2.0259999999999998},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.224,"15min":1.272},"min":{"1min":0.94699999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6319999999999999,"5min":1.9870000000000001,"15min":2.2370000000000001},"last":1.1599999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3049999999999999,"15min":1.3069999999999999},"min":{"1min":1.02,"5min":0.73099999999999998,"15min":0.66000000000000003},"max":{"1min":1.474,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.3720000000000001,"15min":1.375},"min":{"1min":0.70999999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.6459999999999999,"5min":2.1150000000000002,"15min":2.2160000000000002},"last":1.601}]},{"osd":4,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.234,"15min":1.2090000000000001},"min":{"1min":0.87,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":1.7549999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.367,"15min":1.325},"min":{"1min":1.0700000000000001,"5min":0.54200000000000004,"15min":0.42999999999999999},"max":{"1min":1.786,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.458}]},{"osd":6,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3759999999999999,"15min":1.3380000000000001},"min":{"1min":0.78600000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.8500000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.482},{"interface":"front","average":{"1min":1.474,"5min":1.4179999999999999,"15min":1.3879999999999999},"min":{"1min":1.0920000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0529999999999999,"5min":2.48,"15min":2.48},"last":1.645}]},{"osd":7,"last update":"Mon Oct 16 12:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.478,"15min":1.454},"min":{"1min":0.997,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.9330000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.548,"5min":1.5349999999999999,"15min":1.5129999999999999},"min":{"1min":1.1279999999999999,"5min":0.64400000000000002,"15min":0.55300000000000005},"max":{"1min":2.1680000000000001,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.7}]}]},{"osd":4,"up_from":14,"seq":60129542384,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59276,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312564,"statfs":{"total":96636764160,"available":96576065536,"internally_reserved":0,"allocated":1769472,"data_stored":849078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.1539999999999999,"15min":1.194},"min":{"1min":0.622,"5min":0.503,"15min":0.503},"max":{"1min":3.0779999999999998,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.623},{"interface":"front","average":{"1min":1.165,"5min":1.1180000000000001,"15min":1.151},"min":{"1min":0.61199999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":1.1739999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.1759999999999999,"15min":1.1870000000000001},"min":{"1min":0.90800000000000003,"5min":0.38,"15min":0.38},"max":{"1min":3.3180000000000001,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":1.464},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.179,"15min":1.238},"min":{"1min":0.66400000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.5939999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":2.1579999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.2649999999999999,"15min":1.26},"min":{"1min":0.76800000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":4.3470000000000004,"5min":4.3470000000000004,"15min":4.3470000000000004},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.266,"15min":1.319},"min":{"1min":0.75600000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.4670000000000001,"5min":2.4670000000000001,"15min":2.4670000000000001},"last":1.9710000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.246,"15min":1.266},"min":{"1min":0.70899999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.8300000000000001,"5min":2.8300000000000001,"15min":2.8300000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.732,"5min":1.395,"15min":1.381},"min":{"1min":0.91700000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":8.9849999999999994,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":1.8520000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3460000000000001,"15min":1.306},"min":{"1min":0.69599999999999995,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6150000000000002,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":2.2280000000000002},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.347,"15min":1.3},"min":{"1min":0.98499999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.391,"5min":4.391,"15min":4.391},"last":1.7070000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.3899999999999999,"15min":1.3779999999999999},"min":{"1min":0.73899999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.4649999999999999,"5min":3.4649999999999999,"15min":3.4649999999999999},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.4870000000000001,"15min":1.476},"min":{"1min":0.874,"5min":0.502,"15min":0.502},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":2.395}]},{"osd":7,"last update":"Mon Oct 16 12:52:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.498,"15min":1.4730000000000001},"min":{"1min":1.2,"5min":0.67600000000000005,"15min":0.60399999999999998},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":3.9369999999999998},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.728,"5min":1.605,"15min":1.5880000000000001},"min":{"1min":1.2010000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":2.4460000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542384,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1785856,"data_stored":850760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.92400000000000004,"15min":0.95999999999999996},"min":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":1.609,"5min":1.609,"15min":2.0419999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.071,"15min":1.161},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.34799999999999998},"max":{"1min":1.5549999999999999,"5min":1.5800000000000001,"15min":2.2749999999999999},"last":1.661}]},{"osd":1,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.97899999999999998,"15min":1.0209999999999999},"min":{"1min":0.503,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5529999999999999,"5min":1.6240000000000001,"15min":1.758},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.004,"15min":1.069},"min":{"1min":0.58399999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.2070000000000001,"5min":1.784,"15min":2.4580000000000002},"last":1.1259999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98899999999999999,"15min":1.0589999999999999},"min":{"1min":0.39900000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.746,"5min":1.746,"15min":1.9510000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.26,"15min":1.345},"min":{"1min":0.71499999999999997,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8180000000000001,"5min":2.044,"15min":2.044},"last":1.5880000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.411,"15min":1.4730000000000001},"min":{"1min":0.78900000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.905,"5min":1.9239999999999999,"15min":1.956},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.337,"15min":1.399},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.64800000000000002},"max":{"1min":1.706,"5min":1.869,"15min":2.0099999999999998},"last":1.3680000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2889999999999999,"15min":1.371},"min":{"1min":0.85499999999999998,"5min":0.70299999999999996,"15min":0.65900000000000003},"max":{"1min":1.754,"5min":1.9910000000000001,"15min":2.0539999999999998},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.429,"15min":1.466},"min":{"1min":1.0609999999999999,"5min":0.877,"15min":0.877},"max":{"1min":1.8560000000000001,"5min":2.0750000000000002,"15min":2.3540000000000001},"last":1.7190000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3979999999999999,"15min":1.4390000000000001},"min":{"1min":1.083,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.7929999999999999,"5min":1.998,"15min":2.3220000000000001},"last":1.536},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3500000000000001,"15min":1.4099999999999999},"min":{"1min":1.0009999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6319999999999999,"5min":2,"15min":2.3740000000000001},"last":1.512}]},{"osd":7,"last update":"Mon Oct 16 12:53:05 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4910000000000001,"15min":1.5329999999999999},"min":{"1min":0.94399999999999995,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.8720000000000001,"5min":2.157,"15min":2.431},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.522,"15min":1.5960000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.8540000000000001,"5min":2.1040000000000001,"15min":2.552},"last":1.77}]}]},{"osd":2,"up_from":14,"seq":60129542382,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46788,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325052,"statfs":{"total":96636764160,"available":96588853248,"internally_reserved":0,"allocated":1757184,"data_stored":842778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.86399999999999999,"15min":0.91800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.315},"max":{"1min":1.2709999999999999,"5min":1.508,"15min":1.7889999999999999},"last":0.629},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.0369999999999999,"15min":1.0800000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.41699999999999998},"max":{"1min":1.589,"5min":1.746,"15min":2.085},"last":0.97299999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.079,"15min":1.1299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.5149999999999999,"5min":1.677,"15min":2.5590000000000002},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.1779999999999999,"15min":1.2210000000000001},"min":{"1min":0.60599999999999998,"5min":0.56999999999999995,"15min":0.46500000000000002},"max":{"1min":1.73,"5min":2.347,"15min":2.4289999999999998},"last":1.0620000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0649999999999999,"15min":1.0840000000000001},"min":{"1min":0.54000000000000004,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":1.234},{"interface":"front","average":{"1min":1.123,"5min":1.26,"15min":1.3120000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.671,"5min":2.238,"15min":2.238},"last":0.82499999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3799999999999999,"15min":1.4490000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.8979999999999999,"5min":2.2149999999999999,"15min":2.431},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.3660000000000001,"15min":1.421},"min":{"1min":0.78400000000000003,"5min":0.75600000000000001,"15min":0.68999999999999995},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.1699999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.4650000000000001,"15min":1.504},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.8570000000000002,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.373,"15min":1.4179999999999999},"min":{"1min":0.755,"5min":0.751,"15min":0.52100000000000002},"max":{"1min":3.7759999999999998,"5min":3.7759999999999998,"15min":3.7759999999999998},"last":1.605}]},{"osd":6,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.4419999999999999,"15min":1.4450000000000001},"min":{"1min":0.89800000000000002,"5min":0.89400000000000002,"15min":0.59699999999999998},"max":{"1min":3.6120000000000001,"5min":3.6120000000000001,"15min":3.6120000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4810000000000001,"15min":1.5509999999999999},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.78000000000000003},"max":{"1min":1.8420000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.323}]},{"osd":7,"last update":"Mon Oct 16 12:52:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.548,"15min":1.6020000000000001},"min":{"1min":1.081,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4990000000000001,"15min":1.5389999999999999},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.56599999999999995},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":1.4810000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59280,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312560,"statfs":{"total":96636764160,"available":96576061440,"internally_reserved":0,"allocated":1765376,"data_stored":842902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.93500000000000005,"15min":0.90900000000000003},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2669999999999999,"5min":1.6359999999999999,"15min":2.0840000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0129999999999999,"15min":1.0700000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.5509999999999999,"5min":1.931,"15min":1.9770000000000001},"last":0.88200000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.95299999999999996,"15min":0.98399999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3280000000000001,"5min":1.694,"15min":2.262},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.135,"15min":1.175},"min":{"1min":0.57999999999999996,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":1.6639999999999999,"5min":1.9910000000000001,"15min":2.2210000000000001},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.272,"15min":1.298},"min":{"1min":0.53600000000000003,"5min":0.47899999999999998,"15min":0.46100000000000002},"max":{"1min":1.919,"5min":2.073,"15min":2.073},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1619999999999999,"15min":1.208},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.758,"5min":1.758,"15min":1.9199999999999999},"last":0.999}]},{"osd":4,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2410000000000001,"15min":1.294},"min":{"1min":0.60199999999999998,"5min":0.59599999999999997,"15min":0.54600000000000004},"max":{"1min":1.6200000000000001,"5min":2.032,"15min":2.1030000000000002},"last":1.429},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4259999999999999,"15min":1.4339999999999999},"min":{"1min":0.69499999999999995,"5min":0.67900000000000005,"15min":0.60599999999999998},"max":{"1min":2.3380000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.8560000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.393,"15min":1.4179999999999999},"min":{"1min":0.79700000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3939999999999999,"15min":1.429},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.786,"5min":2.1179999999999999,"15min":2.4350000000000001},"last":1.7410000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.85899999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"max":{"1min":2.4460000000000002,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.554},{"interface":"front","average":{"1min":1.577,"5min":1.494,"15min":1.5149999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.4390000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.5169999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.583,"15min":1.6100000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.669},{"interface":"front","average":{"1min":1.454,"5min":1.52,"15min":1.552},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8720000000000001,"5min":2.153,"15min":2.5710000000000002},"last":1.1850000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542383,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41820,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330020,"statfs":{"total":96636764160,"available":96593940480,"internally_reserved":0,"allocated":2371584,"data_stored":1434870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83299999999999996,"15min":0.879},"min":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.4470000000000001,"5min":1.655,"15min":1.6890000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.0409999999999999,"15min":1.079},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.4590000000000001,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.752}]},{"osd":2,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0800000000000001,"15min":1.127},"min":{"1min":0.495,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.744,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.0680000000000001,"15min":1.151},"min":{"1min":0.497,"5min":0.497,"15min":0.496},"max":{"1min":1.518,"5min":1.788,"15min":1.952},"last":1.2410000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0069999999999999,"15min":1.079},"min":{"1min":0.435,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.9570000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.181,"15min":1.272},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.70399999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.2829999999999999,"15min":1.3320000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5579999999999998,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.3740000000000001,"15min":1.427},"min":{"1min":0.75,"5min":0.625,"15min":0.59699999999999998},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.173},"last":1.0269999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3500000000000001,"15min":1.415},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72499999999999998},"max":{"1min":1.8320000000000001,"5min":1.9630000000000001,"15min":2.2440000000000002},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.3500000000000001,"15min":1.407},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.58999999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.0270000000000001},"last":1.1970000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4470000000000001,"15min":1.516},"min":{"1min":0.65600000000000003,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.085,"5min":2.125,"15min":2.3519999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.397,"15min":1.5029999999999999},"min":{"1min":0.66900000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.8520000000000001,"5min":1.95,"15min":2.363},"last":1.073}]},{"osd":7,"last update":"Mon Oct 16 12:53:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.528,"15min":1.615},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.395,"5min":2.395,"15min":2.4729999999999999},"last":1.448},{"interface":"front","average":{"1min":1.482,"5min":1.4830000000000001,"15min":1.5449999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.1589999999999998,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.339}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:53:32.331 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:53:32.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:53:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.602+0000 7fc765440640 1 Processor -- start 2023-10-16T12:53:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.602+0000 7fc765440640 1 -- start start 2023-10-16T12:53:32.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc765440640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc76015ac70 0x7fc7600cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc765440640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc76006a4b0 0x7fc7600cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc765440640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7600cc850 con 0x7fc7600b4160 2023-10-16T12:53:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc765440640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7600ccb30 con 0x7fc76006a4b0 2023-10-16T12:53:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc765440640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7600cce10 con 0x7fc76015ac70 2023-10-16T12:53:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc764c3f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc76006a4b0 0x7fc7600cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc764c3f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc76006a4b0 0x7fc7600cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44382/0 (socket says 172.21.15.60:44382) 2023-10-16T12:53:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.604+0000 7fc764c3f640 1 -- 172.21.15.60:0/3972801231 learned_addr learned my addr 172.21.15.60:0/3972801231 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.605+0000 7fc75ffff640 1 --2- 172.21.15.60:0/3972801231 >> v2:172.21.15.144:3300/0 conn(0x7fc76015ac70 0x7fc7600cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.605+0000 7fc764c3f640 1 -- 172.21.15.60:0/3972801231 >> v2:172.21.15.144:3300/0 conn(0x7fc76015ac70 msgr2=0x7fc7600cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.605+0000 7fc764c3f640 1 --2- 172.21.15.60:0/3972801231 >> v2:172.21.15.144:3300/0 conn(0x7fc76015ac70 0x7fc7600cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.605+0000 7fc764c3f640 1 -- 172.21.15.60:0/3972801231 >> v1:172.21.15.60:6789/0 conn(0x7fc7600b4160 legacy=0x7fc760069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.605+0000 7fc764c3f640 1 -- 172.21.15.60:0/3972801231 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7600cd0f0 con 0x7fc76006a4b0 2023-10-16T12:53:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.606+0000 7fc764c3f640 1 --2- 172.21.15.60:0/3972801231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76006a4b0 0x7fc7600cd8f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fc7600b1290 tx=0x7fc75802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c98358ec778c2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.606+0000 7fc75effd640 1 -- 172.21.15.60:0/3972801231 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75802cb10 con 0x7fc76006a4b0 2023-10-16T12:53:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.606+0000 7fc75effd640 1 -- 172.21.15.60:0/3972801231 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75802cc70 con 0x7fc76006a4b0 2023-10-16T12:53:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.606+0000 7fc75effd640 1 -- 172.21.15.60:0/3972801231 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7580343f0 con 0x7fc76006a4b0 2023-10-16T12:53:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 -- 172.21.15.60:0/3972801231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76006a4b0 msgr2=0x7fc7600cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 --2- 172.21.15.60:0/3972801231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76006a4b0 0x7fc7600cd8f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fc7600b1290 tx=0x7fc75802bf70 comp rx=0 tx=0).stop 2023-10-16T12:53:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 -- 172.21.15.60:0/3972801231 shutdown_connections 2023-10-16T12:53:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 --2- 172.21.15.60:0/3972801231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76006a4b0 0x7fc7600cd8f0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 --2- 172.21.15.60:0/3972801231 >> v2:172.21.15.144:3300/0 conn(0x7fc76015ac70 0x7fc7600cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 -- 172.21.15.60:0/3972801231 >> 172.21.15.60:0/3972801231 conn(0x7fc7600ce6a0 msgr2=0x7fc7600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:32.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 -- 172.21.15.60:0/3972801231 shutdown_connections 2023-10-16T12:53:32.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.607+0000 7fc765440640 1 -- 172.21.15.60:0/3972801231 wait complete. 2023-10-16T12:53:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.609+0000 7fc765440640 1 Processor -- start 2023-10-16T12:53:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.609+0000 7fc765440640 1 -- start start 2023-10-16T12:53:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.610+0000 7fc765440640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76006a4b0 0x7fc7601532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.610+0000 7fc765440640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7600b4160 0x7fc7600aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.610+0000 7fc765440640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 0x7fc7601544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.610+0000 7fc765440640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600c9f90 con 0x7fc76015ac70 2023-10-16T12:53:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.610+0000 7fc765440640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760155000 con 0x7fc76006a4b0 2023-10-16T12:53:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.610+0000 7fc765440640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600ca580 con 0x7fc7600b4160 2023-10-16T12:53:32.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 0x7fc7601544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 0x7fc7601544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44398/0 (socket says 172.21.15.60:44398) 2023-10-16T12:53:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 -- 172.21.15.60:0/1932227762 learned_addr learned my addr 172.21.15.60:0/1932227762 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc75f7fe640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7600b4160 0x7fc7600aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 -- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7600b4160 msgr2=0x7fc7600aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc75ffff640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76006a4b0 0x7fc7601532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7600b4160 0x7fc7600aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 -- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76006a4b0 msgr2=0x7fc7601532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76006a4b0 0x7fc7601532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc764c3f640 1 -- 172.21.15.60:0/1932227762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758009b30 con 0x7fc76015ac70 2023-10-16T12:53:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc75f7fe640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7600b4160 0x7fc7600aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.611+0000 7fc75ffff640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76006a4b0 0x7fc7601532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.612+0000 7fc764c3f640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 0x7fc7601544b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc758009f40 tx=0x7fc758004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.612+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758009510 con 0x7fc76015ac70 2023-10-16T12:53:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.612+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75803e450 con 0x7fc76015ac70 2023-10-16T12:53:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.612+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7600cad10 con 0x7fc76015ac70 2023-10-16T12:53:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.612+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75803e800 con 0x7fc76015ac70 2023-10-16T12:53:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.612+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7600b12b0 con 0x7fc76015ac70 2023-10-16T12:53:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.613+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76004e9b0 con 0x7fc76015ac70 2023-10-16T12:53:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.614+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc75803e960 con 0x7fc76015ac70 2023-10-16T12:53:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.615+0000 7fc75cff9640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc74c06af90 0x7fc74c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.615+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7580ac5d0 con 0x7fc76015ac70 2023-10-16T12:53:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.616+0000 7fc75ffff640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc74c06af90 0x7fc74c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.617+0000 7fc75ffff640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc74c06af90 0x7fc74c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc750006fd0 tx=0x7fc750008040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.620+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7580775f0 con 0x7fc76015ac70 2023-10-16T12:53:32.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.887+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc7600ce300 con 0x7fc76015ac70 2023-10-16T12:53:32.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.893+0000 7fc75cff9640 1 -- 172.21.15.60:0/1932227762 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc75807ac10 con 0x7fc76015ac70 2023-10-16T12:53:32.895 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:53:32.896 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:53:32.896 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:53:32.896 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:53:32.896 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:53:32.897 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:53:32.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.895+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc74c06af90 msgr2=0x7fc74c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.895+0000 7fc765440640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc74c06af90 0x7fc74c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc750006fd0 tx=0x7fc750008040 comp rx=0 tx=0).stop 2023-10-16T12:53:32.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.895+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 msgr2=0x7fc7601544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:32.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.895+0000 7fc765440640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 0x7fc7601544b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc758009f40 tx=0x7fc758004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:32.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 shutdown_connections 2023-10-16T12:53:32.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc74c06af90 0x7fc74c06d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7600b4160 0x7fc7600aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc76015ac70 0x7fc7601544b0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 --2- 172.21.15.60:0/1932227762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76006a4b0 0x7fc7601532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:32.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 >> 172.21.15.60:0/1932227762 conn(0x7fc7600ce6a0 msgr2=0x7fc760070550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:32.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 shutdown_connections 2023-10-16T12:53:32.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:32.896+0000 7fc765440640 1 -- 172.21.15.60:0/1932227762 wait complete. 2023-10-16T12:53:32.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:33.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.134+0000 7fd58a71a640 1 Processor -- start 2023-10-16T12:53:33.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.134+0000 7fd58a71a640 1 -- start start 2023-10-16T12:53:33.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd58a71a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd584070b60 0x7fd58415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd58a71a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd58406c100 0x7fd58406c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd58a71a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58415aef0 con 0x7fd584159950 2023-10-16T12:53:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd58a71a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58415e490 con 0x7fd58406c100 2023-10-16T12:53:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd58a71a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58415d910 con 0x7fd584070b60 2023-10-16T12:53:33.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd589f19640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd58406c100 0x7fd58406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.135+0000 7fd589718640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd584159950 0x7fd584070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44816/0 (socket says 172.21.15.60:44816) 2023-10-16T12:53:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd589718640 1 -- 172.21.15.60:0/3845119940 learned_addr learned my addr 172.21.15.60:0/3845119940 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd588f17640 1 --2- 172.21.15.60:0/3845119940 >> v2:172.21.15.144:3300/0 conn(0x7fd584070b60 0x7fd58415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd589f19640 1 -- 172.21.15.60:0/3845119940 >> v2:172.21.15.144:3300/0 conn(0x7fd584070b60 msgr2=0x7fd58415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd589f19640 1 --2- 172.21.15.60:0/3845119940 >> v2:172.21.15.144:3300/0 conn(0x7fd584070b60 0x7fd58415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd589f19640 1 -- 172.21.15.60:0/3845119940 >> v1:172.21.15.60:6789/0 conn(0x7fd584159950 legacy=0x7fd584070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd589f19640 1 -- 172.21.15.60:0/3845119940 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58415dbf0 con 0x7fd58406c100 2023-10-16T12:53:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.136+0000 7fd57bfff640 1 -- 172.21.15.60:0/3845119940 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452075917 0 0) 0x7fd580001110 con 0x7fd584159950 2023-10-16T12:53:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.137+0000 7fd589f19640 1 --2- 172.21.15.60:0/3845119940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd58406c100 0x7fd58406c510 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd5840716a0 tx=0x7fd57c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c935ea04c307cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.137+0000 7fd57bfff640 1 -- 172.21.15.60:0/3845119940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c02cbf0 con 0x7fd58406c100 2023-10-16T12:53:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.137+0000 7fd57bfff640 1 -- 172.21.15.60:0/3845119940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c02cd50 con 0x7fd58406c100 2023-10-16T12:53:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd57bfff640 1 -- 172.21.15.60:0/3845119940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c034660 con 0x7fd58406c100 2023-10-16T12:53:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 -- 172.21.15.60:0/3845119940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd58406c100 msgr2=0x7fd58406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 --2- 172.21.15.60:0/3845119940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd58406c100 0x7fd58406c510 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd5840716a0 tx=0x7fd57c02be40 comp rx=0 tx=0).stop 2023-10-16T12:53:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 -- 172.21.15.60:0/3845119940 shutdown_connections 2023-10-16T12:53:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 --2- 172.21.15.60:0/3845119940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd58406c100 0x7fd58406c510 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 --2- 172.21.15.60:0/3845119940 >> v2:172.21.15.144:3300/0 conn(0x7fd584070b60 0x7fd58415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 -- 172.21.15.60:0/3845119940 >> 172.21.15.60:0/3845119940 conn(0x7fd5840ce6a0 msgr2=0x7fd5840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 -- 172.21.15.60:0/3845119940 shutdown_connections 2023-10-16T12:53:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.138+0000 7fd58a71a640 1 -- 172.21.15.60:0/3845119940 wait complete. 2023-10-16T12:53:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.140+0000 7fd58a71a640 1 Processor -- start 2023-10-16T12:53:33.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.140+0000 7fd58a71a640 1 -- start start 2023-10-16T12:53:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd58a71a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58406c100 0x7fd584151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd58a71a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd584070b60 0x7fd5840aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd58a71a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 0x7fd584153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:33.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd58a71a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840ae9a0 con 0x7fd584159950 2023-10-16T12:53:33.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd58a71a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd584151d20 con 0x7fd584070b60 2023-10-16T12:53:33.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd58a71a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840ae070 con 0x7fd58406c100 2023-10-16T12:53:33.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd589f19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 0x7fd584153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:33.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd589f19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 0x7fd584153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44410/0 (socket says 172.21.15.60:44410) 2023-10-16T12:53:33.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.141+0000 7fd589f19640 1 -- 172.21.15.60:0/209174055 learned_addr learned my addr 172.21.15.60:0/209174055 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589f19640 1 -- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58406c100 msgr2=0x7fd584151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589718640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58406c100 0x7fd584151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd588f17640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd584070b60 0x7fd5840aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589f19640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58406c100 0x7fd584151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589f19640 1 -- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd584070b60 msgr2=0x7fd5840aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589f19640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd584070b60 0x7fd5840aee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589f19640 1 -- 172.21.15.60:0/209174055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c009a70 con 0x7fd584159950 2023-10-16T12:53:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd589f19640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 0x7fd584153df0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009e80 tx=0x7fd57c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c040650 con 0x7fd584159950 2023-10-16T12:53:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c0407d0 con 0x7fd584159950 2023-10-16T12:53:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c040b80 con 0x7fd584159950 2023-10-16T12:53:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.142+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5840ae350 con 0x7fd584159950 2023-10-16T12:53:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.143+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5841519c0 con 0x7fd584159950 2023-10-16T12:53:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.144+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5840aeb10 con 0x7fd584159950 2023-10-16T12:53:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.145+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd57c040ce0 con 0x7fd584159950 2023-10-16T12:53:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.146+0000 7fd579ffb640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd56806af90 0x7fd56806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.146+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd57c0ac5c0 con 0x7fd584159950 2023-10-16T12:53:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.147+0000 7fd589718640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd56806af90 0x7fd56806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.148+0000 7fd589718640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd56806af90 0x7fd56806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd580000f60 tx=0x7fd580009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:33.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.151+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd57c077560 con 0x7fd584159950 2023-10-16T12:53:33.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.388+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd58404e9b0 con 0x7fd584159950 2023-10-16T12:53:33.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.389+0000 7fd579ffb640 1 -- 172.21.15.60:0/209174055 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd57c07ab80 con 0x7fd584159950 2023-10-16T12:53:33.392 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:33.392 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:53:26.782026+0000","modified":"2023-10-16T12:53:29.833032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7048},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7048":{"gid":7048,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/3032734613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3032734613},{"type":"v1","addr":"172.21.15.60:6834","nonce":3032734613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:53:33.392 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 70 2023-10-16T12:53:33.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.391+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd56806af90 msgr2=0x7fd56806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:33.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.391+0000 7fd58a71a640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd56806af90 0x7fd56806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd580000f60 tx=0x7fd580009040 comp rx=0 tx=0).stop 2023-10-16T12:53:33.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.391+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 msgr2=0x7fd584153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:33.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.391+0000 7fd58a71a640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 0x7fd584153df0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009e80 tx=0x7fd57c004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:33.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 shutdown_connections 2023-10-16T12:53:33.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fd56806af90 0x7fd56806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd584159950 0x7fd584153df0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd584070b60 0x7fd5840aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 --2- 172.21.15.60:0/209174055 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd58406c100 0x7fd584151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:33.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 >> 172.21.15.60:0/209174055 conn(0x7fd5840ce6a0 msgr2=0x7fd584070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:33.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 shutdown_connections 2023-10-16T12:53:33.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:33.392+0000 7fd58a71a640 1 -- 172.21.15.60:0/209174055 wait complete. 2023-10-16T12:53:33.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:53:26.782026+0000', 'modified': '2023-10-16T12:53:29.833032+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7048}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7048': {'gid': 7048, 'name': 'a', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6834/3032734613', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3032734613}, {'type': 'v1', 'addr': '172.21.15.60:6834', 'nonce': 3032734613}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:53:33.817 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:53:33.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:53:34.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.053+0000 7f771e7cb640 1 Processor -- start 2023-10-16T12:53:34.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.053+0000 7f771e7cb640 1 -- start start 2023-10-16T12:53:34.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.054+0000 7f771e7cb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77180ad970 0x7f77180cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771e7cb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77180ccfb0 0x7f77180cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771e7cb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771815b9d0 con 0x7f771806c040 2023-10-16T12:53:34.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771e7cb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771815bcb0 con 0x7f77180ad970 2023-10-16T12:53:34.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771e7cb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771815bf90 con 0x7f77180ccfb0 2023-10-16T12:53:34.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771cfc8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77180ad970 0x7f77180cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771cfc8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77180ad970 0x7f77180cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44424/0 (socket says 172.21.15.60:44424) 2023-10-16T12:53:34.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771cfc8640 1 -- 172.21.15.60:0/1859248728 learned_addr learned my addr 172.21.15.60:0/1859248728 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.055+0000 7f771dfca640 1 --2- 172.21.15.60:0/1859248728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77180ccfb0 0x7f77180cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.056+0000 7f771cfc8640 1 -- 172.21.15.60:0/1859248728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77180ccfb0 msgr2=0x7f77180cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.056+0000 7f771cfc8640 1 --2- 172.21.15.60:0/1859248728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77180ccfb0 0x7f77180cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.056+0000 7f771cfc8640 1 -- 172.21.15.60:0/1859248728 >> v1:172.21.15.60:6789/0 conn(0x7f771806c040 legacy=0x7f771806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.056+0000 7f770ffff640 1 -- 172.21.15.60:0/1859248728 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846595820 0 0) 0x7f771815b9d0 con 0x7f771806c040 2023-10-16T12:53:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.056+0000 7f771cfc8640 1 -- 172.21.15.60:0/1859248728 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180cc270 con 0x7f77180ad970 2023-10-16T12:53:34.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.057+0000 7f771cfc8640 1 --2- 172.21.15.60:0/1859248728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f77180cca70 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f7708009870 tx=0x7f770802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8dc555b8f623ff1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.057+0000 7f770ffff640 1 -- 172.21.15.60:0/1859248728 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708030840 con 0x7f77180ad970 2023-10-16T12:53:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.057+0000 7f770ffff640 1 -- 172.21.15.60:0/1859248728 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77080309a0 con 0x7f77180ad970 2023-10-16T12:53:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.057+0000 7f770ffff640 1 -- 172.21.15.60:0/1859248728 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708030d30 con 0x7f77180ad970 2023-10-16T12:53:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 -- 172.21.15.60:0/1859248728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 msgr2=0x7f77180cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1859248728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f77180cca70 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f7708009870 tx=0x7f770802b980 comp rx=0 tx=0).stop 2023-10-16T12:53:34.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 -- 172.21.15.60:0/1859248728 shutdown_connections 2023-10-16T12:53:34.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1859248728 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77180ccfb0 0x7f77180cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1859248728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f77180cca70 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 -- 172.21.15.60:0/1859248728 >> 172.21.15.60:0/1859248728 conn(0x7f77180ce6a0 msgr2=0x7f77180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 -- 172.21.15.60:0/1859248728 shutdown_connections 2023-10-16T12:53:34.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.058+0000 7f771e7cb640 1 -- 172.21.15.60:0/1859248728 wait complete. 2023-10-16T12:53:34.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.060+0000 7f771e7cb640 1 Processor -- start 2023-10-16T12:53:34.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.060+0000 7f771e7cb640 1 -- start start 2023-10-16T12:53:34.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771e7cb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771806c040 0x7f7718156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771e7cb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f7718155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771e7cb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77180ccfb0 0x7f77180c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771e7cb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77180c5770 con 0x7f77180ad970 2023-10-16T12:53:34.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771e7cb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77180c59c0 con 0x7f77180ccfb0 2023-10-16T12:53:34.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771e7cb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77180c5ca0 con 0x7f771806c040 2023-10-16T12:53:34.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771cfc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f7718155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771cfc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f7718155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44434/0 (socket says 172.21.15.60:44434) 2023-10-16T12:53:34.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771cfc8640 1 -- 172.21.15.60:0/1395724901 learned_addr learned my addr 172.21.15.60:0/1395724901 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.061+0000 7f771d7c9640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771806c040 0x7f7718156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771dfca640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77180ccfb0 0x7f77180c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771cfc8640 1 -- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771806c040 msgr2=0x7f7718156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771cfc8640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771806c040 0x7f7718156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771cfc8640 1 -- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77180ccfb0 msgr2=0x7f77180c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771cfc8640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77180ccfb0 0x7f77180c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771cfc8640 1 -- 172.21.15.60:0/1395724901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080094d0 con 0x7f77180ad970 2023-10-16T12:53:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771cfc8640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f7718155fd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f77080097c0 tx=0x7f7708004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f770dffb640 1 -- 172.21.15.60:0/1395724901 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770803e040 con 0x7f77180ad970 2023-10-16T12:53:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f770dffb640 1 -- 172.21.15.60:0/1395724901 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708039c10 con 0x7f77180ad970 2023-10-16T12:53:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.062+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77180c5f80 con 0x7f77180ad970 2023-10-16T12:53:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.063+0000 7f770dffb640 1 -- 172.21.15.60:0/1395724901 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708004020 con 0x7f77180ad970 2023-10-16T12:53:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.063+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77180c6540 con 0x7f77180ad970 2023-10-16T12:53:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.065+0000 7f770dffb640 1 -- 172.21.15.60:0/1395724901 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7708004180 con 0x7f77180ad970 2023-10-16T12:53:34.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.066+0000 7f770dffb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f76fc06af90 0x7f76fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.066+0000 7f770dffb640 1 -- 172.21.15.60:0/1395724901 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77080abde0 con 0x7f77180ad970 2023-10-16T12:53:34.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.067+0000 7f771d7c9640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f76fc06af90 0x7f76fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.067+0000 7f77225c3740 1 Processor -- start 2023-10-16T12:53:34.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.067+0000 7f77225c3740 1 -- start start 2023-10-16T12:53:34.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f771d7c9640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f76fc06af90 0x7f76fc06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7714000f60 tx=0x7f7714009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74caf5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f77225c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 0x55b74cb24f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74caf55b0 0x55b74cb25440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f77225c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74ca157a0 con 0x55b74cb22550 2023-10-16T12:53:34.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74ca67740 con 0x55b74cb22170 2023-10-16T12:53:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b74ca64e30 con 0x55b74caf55b0 2023-10-16T12:53:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f771cfc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 0x55b74cb24f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f771cfc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 0x55b74cb24f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44448/0 (socket says 172.21.15.60:44448) 2023-10-16T12:53:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.068+0000 7f771cfc8640 1 -- 172.21.15.60:0/479022930 learned_addr learned my addr 172.21.15.60:0/479022930 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771d7c9640 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74caf5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771cfc8640 1 -- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74caf55b0 msgr2=0x55b74cb25440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771dfca640 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74caf55b0 0x55b74cb25440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771cfc8640 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74caf55b0 0x55b74cb25440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771cfc8640 1 -- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 msgr2=0x55b74caf5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771cfc8640 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74caf5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771cfc8640 1 -- 172.21.15.60:0/479022930 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b74ca24910 con 0x55b74cb22550 2023-10-16T12:53:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771d7c9640 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74caf5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f771cfc8640 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 0x55b74cb24f00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f770807a9e0 tx=0x7f7708004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49899012a6a7350 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f7702ffd640 1 -- 172.21.15.60:0/479022930 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770807eba0 con 0x55b74cb22550 2023-10-16T12:53:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f7702ffd640 1 -- 172.21.15.60:0/479022930 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708082d50 con 0x55b74cb22550 2023-10-16T12:53:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.069+0000 7f7702ffd640 1 -- 172.21.15.60:0/479022930 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708077160 con 0x55b74cb22550 2023-10-16T12:53:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 -- 172.21.15.60:0/479022930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 msgr2=0x55b74cb24f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 0x55b74cb24f00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f770807a9e0 tx=0x7f7708004430 comp rx=0 tx=0).stop 2023-10-16T12:53:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 -- 172.21.15.60:0/479022930 shutdown_connections 2023-10-16T12:53:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74caf55b0 0x55b74cb25440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb22550 0x55b74cb24f00 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 --2- 172.21.15.60:0/479022930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74caf5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 -- 172.21.15.60:0/479022930 >> 172.21.15.60:0/479022930 conn(0x55b74caf7380 msgr2=0x55b74cb3ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 -- 172.21.15.60:0/479022930 shutdown_connections 2023-10-16T12:53:34.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.070+0000 7f77225c3740 1 -- 172.21.15.60:0/479022930 wait complete. 2023-10-16T12:53:34.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.071+0000 7f77225c3740 1 -- start start 2023-10-16T12:53:34.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.071+0000 7f77225c3740 1 Processor -- start 2023-10-16T12:53:34.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cb164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb3a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb3fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74ca60720 con 0x55b74cb28160 2023-10-16T12:53:34.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74c9fe970 con 0x55b74cb22550 2023-10-16T12:53:34.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b74c9fe610 con 0x55b74cb22170 2023-10-16T12:53:34.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.072+0000 7f77225c3740 10 client.0 _ll_register_callbacks cb 0x55b74ca209d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:53:34.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb3fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb3fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44462/0 (socket says 172.21.15.60:44462) 2023-10-16T12:53:34.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 -- 172.21.15.60:0/2206279637 learned_addr learned my addr 172.21.15.60:0/2206279637 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771d7c9640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cb164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771cfc8640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb3a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 -- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 msgr2=0x55b74cb164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cb164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 -- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 msgr2=0x55b74cb3a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb3a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.073+0000 7f771dfca640 1 -- 172.21.15.60:0/2206279637 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77180c6540 con 0x55b74cb28160 2023-10-16T12:53:34.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f771dfca640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb3fe90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7710007a30 tx=0x7f7710007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef884f77fe3743d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f76f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:53:34.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f76f7fff640 1 -- 172.21.15.60:0/2206279637 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100109e0 con 0x55b74cb28160 2023-10-16T12:53:34.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f76f7fff640 1 -- 172.21.15.60:0/2206279637 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7710010b40 con 0x55b74cb28160 2023-10-16T12:53:34.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b74ca87160 con 0x55b74cb28160 2023-10-16T12:53:34.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f76f7fff640 1 -- 172.21.15.60:0/2206279637 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77100098d0 con 0x55b74cb28160 2023-10-16T12:53:34.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f77225c3740 10 client.8532 fetch_fsmap learned FSMap version 70 2023-10-16T12:53:34.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55b74ca5eda0 con 0x55b74cb28160 2023-10-16T12:53:34.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.074+0000 7f76f7fff640 1 -- 172.21.15.60:0/2206279637 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b74ca87160 con 0x55b74cb28160 2023-10-16T12:53:34.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f76f7fff640 1 -- 172.21.15.60:0/2206279637 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 70) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7710010d00 con 0x55b74cb28160 2023-10-16T12:53:34.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f77225c3740 10 client.8532 fetch_fsmap finished waiting for FSMap version 70 2023-10-16T12:53:34.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f77225c3740 10 client.8532 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:53:34.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f77225c3740 20 client.8532 populate_metadata read hostname 'smithi060' 2023-10-16T12:53:34.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f77225c3740 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cb3ac80 0x55b74cb3d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f77225c3740 4 client.8532 mds_command: new command op to 7048 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:53:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 --> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b74ca32cc0 con 0x55b74cb3ac80 2023-10-16T12:53:34.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.075+0000 7f771d7c9640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cb3ac80 0x55b74cb3d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.076+0000 7f771d7c9640 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cb3ac80 0x55b74cb3d5a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a15dd565a95ecf9 server_cookie=a65de796a4a9acd8 in_seq=0 out_seq=0 2023-10-16T12:53:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.076+0000 7f76f7fff640 10 client.8532 ms_handle_connect on v2:172.21.15.60:6832/3032734613 2023-10-16T12:53:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.076+0000 7f76f7fff640 1 -- 172.21.15.60:0/2206279637 <== mds.0 v2:172.21.15.60:6832/3032734613 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b74ca32cc0 con 0x55b74cb3ac80 2023-10-16T12:53:34.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.076+0000 7f76f7fff640 10 client.8532 handle_command_reply: tid=0 2023-10-16T12:53:34.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.076+0000 7f77225c3740 1 client.8532 shutdown 2023-10-16T12:53:34.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 msgr2=0x55b74cb3fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb3fe90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7710007a30 tx=0x7f7710007ef0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 shutdown_connections 2023-10-16T12:53:34.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cb3ac80 0x55b74cb3d5a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb3fe90 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb3a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 --2- 172.21.15.60:0/2206279637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cb164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f76f7fff640 0 client.8532 ms_handle_reset on v2:172.21.15.60:6832/3032734613 2023-10-16T12:53:34.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 >> 172.21.15.60:0/2206279637 conn(0x55b74caf55b0 msgr2=0x55b74cb39d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 shutdown_connections 2023-10-16T12:53:34.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 1 -- 172.21.15.60:0/2206279637 wait complete. 2023-10-16T12:53:34.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.077+0000 7f77225c3740 20 client.8532 trim_cache size 0 max 16384 2023-10-16T12:53:34.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.094+0000 7f77225c3740 1 Processor -- start 2023-10-16T12:53:34.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.094+0000 7f77225c3740 1 -- start start 2023-10-16T12:53:34.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74cb3d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22550 0x55b74cb25530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f77225c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb24b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f77225c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74c7c4f20 con 0x55b74cb28160 2023-10-16T12:53:34.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74ca065c0 con 0x55b74cb22170 2023-10-16T12:53:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b74cb203c0 con 0x55b74cb22550 2023-10-16T12:53:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.095+0000 7f771dfca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb24b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771d7c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74cb3d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb24b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44466/0 (socket says 172.21.15.60:44466) 2023-10-16T12:53:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 -- 172.21.15.60:0/3652676381 learned_addr learned my addr 172.21.15.60:0/3652676381 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771cfc8640 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22550 0x55b74cb25530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 -- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22550 msgr2=0x55b74cb25530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22550 0x55b74cb25530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 -- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 msgr2=0x55b74cb3d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74cb3d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 -- 172.21.15.60:0/3652676381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b74c9fdb80 con 0x55b74cb28160 2023-10-16T12:53:34.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.096+0000 7f771dfca640 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb24b30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7710017570 tx=0x7f7710016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c495a72de7d1f5c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77037fe640 1 -- 172.21.15.60:0/3652676381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100175f0 con 0x55b74cb28160 2023-10-16T12:53:34.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77037fe640 1 -- 172.21.15.60:0/3652676381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77100097f0 con 0x55b74cb28160 2023-10-16T12:53:34.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77037fe640 1 -- 172.21.15.60:0/3652676381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710009070 con 0x55b74cb28160 2023-10-16T12:53:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 -- 172.21.15.60:0/3652676381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 msgr2=0x55b74cb24b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb24b30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7710017570 tx=0x7f7710016000 comp rx=0 tx=0).stop 2023-10-16T12:53:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 -- 172.21.15.60:0/3652676381 shutdown_connections 2023-10-16T12:53:34.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22550 0x55b74cb25530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb24b30 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 --2- 172.21.15.60:0/3652676381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22170 0x55b74cb3d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 -- 172.21.15.60:0/3652676381 >> 172.21.15.60:0/3652676381 conn(0x55b74caf55b0 msgr2=0x55b74cb28ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.097+0000 7f77225c3740 1 -- 172.21.15.60:0/3652676381 shutdown_connections 2023-10-16T12:53:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.098+0000 7f77225c3740 1 -- 172.21.15.60:0/3652676381 wait complete. 2023-10-16T12:53:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.098+0000 7f77225c3740 1 -- start start 2023-10-16T12:53:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.098+0000 7f77225c3740 1 Processor -- start 2023-10-16T12:53:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.099+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cba01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.099+0000 7f77225c3740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb26260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.099+0000 7f77225c3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb267a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.099+0000 7f77225c3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74cba92d0 con 0x55b74cb28160 2023-10-16T12:53:34.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.099+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b74cba0930 con 0x55b74cb22550 2023-10-16T12:53:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.099+0000 7f77225c3740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b74cba06e0 con 0x55b74cb22170 2023-10-16T12:53:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f77225c3740 10 client.0 _ll_register_callbacks cb 0x7f7710009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:53:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb267a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb267a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44468/0 (socket says 172.21.15.60:44468) 2023-10-16T12:53:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 -- 172.21.15.60:0/3162979227 learned_addr learned my addr 172.21.15.60:0/3162979227 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771cfc8640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb26260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771d7c9640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cba01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 -- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 msgr2=0x55b74cba01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cba01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 -- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 msgr2=0x55b74cb26260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb26260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771dfca640 1 -- 172.21.15.60:0/3162979227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b74ca5eda0 con 0x55b74cb28160 2023-10-16T12:53:34.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.100+0000 7f771d7c9640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cba01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f771dfca640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb267a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f771002a040 tx=0x7f7710004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfd7a02ccb7396b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f76f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:53:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f76f7fff640 1 -- 172.21.15.60:0/3162979227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100109e0 con 0x55b74cb28160 2023-10-16T12:53:34.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f76f7fff640 1 -- 172.21.15.60:0/3162979227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7710010b40 con 0x55b74cb28160 2023-10-16T12:53:34.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7708082d50 con 0x55b74cb28160 2023-10-16T12:53:34.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f76f7fff640 1 -- 172.21.15.60:0/3162979227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7710010d00 con 0x55b74cb28160 2023-10-16T12:53:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f77225c3740 10 client.8538 fetch_fsmap learned FSMap version 70 2023-10-16T12:53:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.101+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55b74ca23170 con 0x55b74cb28160 2023-10-16T12:53:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f76f7fff640 1 -- 172.21.15.60:0/3162979227 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708082d50 con 0x55b74cb28160 2023-10-16T12:53:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f76f7fff640 1 -- 172.21.15.60:0/3162979227 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 70) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7710017880 con 0x55b74cb28160 2023-10-16T12:53:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f77225c3740 10 client.8538 fetch_fsmap finished waiting for FSMap version 70 2023-10-16T12:53:34.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f77225c3740 10 client.8538 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:53:34.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f77225c3740 20 client.8538 populate_metadata read hostname 'smithi060' 2023-10-16T12:53:34.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f77225c3740 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cba1c90 0x55b74cba2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f77225c3740 4 client.8538 mds_command: new command op to 7048 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:53:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 --> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b74ca59700 con 0x55b74cba1c90 2023-10-16T12:53:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.102+0000 7f771d7c9640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cba1c90 0x55b74cba2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f771d7c9640 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cba1c90 0x55b74cba2070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cad578b5de7d6728 server_cookie=e28b3d843cbe0eab in_seq=0 out_seq=0 2023-10-16T12:53:34.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f76f7fff640 10 client.8538 ms_handle_connect on v2:172.21.15.60:6832/3032734613 2023-10-16T12:53:34.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f76f7fff640 1 -- 172.21.15.60:0/3162979227 <== mds.0 v2:172.21.15.60:6832/3032734613 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b74ca59700 con 0x55b74cba1c90 2023-10-16T12:53:34.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f76f7fff640 10 client.8538 handle_command_reply: tid=0 2023-10-16T12:53:34.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 client.8538 shutdown 2023-10-16T12:53:34.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 msgr2=0x55b74cb267a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb267a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f771002a040 tx=0x7f7710004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:34.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 shutdown_connections 2023-10-16T12:53:34.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:6832/3032734613,v1:172.21.15.60:6834/3032734613] conn(0x55b74cba1c90 0x55b74cba2070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b74cb28160 0x55b74cb267a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55b74cb22550 0x55b74cb26260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 --2- 172.21.15.60:0/3162979227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55b74cb22170 0x55b74cba01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f76f7fff640 0 client.8538 ms_handle_reset on v2:172.21.15.60:6832/3032734613 2023-10-16T12:53:34.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.103+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 >> 172.21.15.60:0/3162979227 conn(0x55b74caf55b0 msgr2=0x55b74cb20a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 shutdown_connections 2023-10-16T12:53:34.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f77225c3740 1 -- 172.21.15.60:0/3162979227 wait complete. 2023-10-16T12:53:34.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f77225c3740 20 client.8538 trim_cache size 0 max 16384 2023-10-16T12:53:34.131 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:53:34.131 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:53:34.131 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:53:34.131 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 7048, 2023-10-16T12:53:34.131 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:53:34.132 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:53:34.132 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:53:34.132 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.7785322380000004, 2023-10-16T12:53:34.132 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 70, 2023-10-16T12:53:34.132 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 143, 2023-10-16T12:53:34.133 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 143, 2023-10-16T12:53:34.133 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.157499007 2023-10-16T12:53:34.133 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:53:34.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f76fc06af90 msgr2=0x7f76fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f76fc06af90 0x7f76fc06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7714000f60 tx=0x7f7714009040 comp rx=0 tx=0).stop 2023-10-16T12:53:34.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 msgr2=0x7f7718155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f7718155fd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f77080097c0 tx=0x7f7708004730 comp rx=0 tx=0).stop 2023-10-16T12:53:34.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 shutdown_connections 2023-10-16T12:53:34.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f76fc06af90 0x7f76fc06d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77180ccfb0 0x7f77180c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77180ad970 0x7f7718155fd0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 --2- 172.21.15.60:0/1395724901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771806c040 0x7f7718156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.104+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 >> 172.21.15.60:0/1395724901 conn(0x7f77180ce6a0 msgr2=0x7f77180cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.105+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 shutdown_connections 2023-10-16T12:53:34.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.105+0000 7f771e7cb640 1 -- 172.21.15.60:0/1395724901 wait complete. 2023-10-16T12:53:34.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:53:34.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.372+0000 7f4ced578640 1 Processor -- start 2023-10-16T12:53:34.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.372+0000 7f4ced578640 1 -- start start 2023-10-16T12:53:34.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.374+0000 7f4ced578640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.374+0000 7f4ced578640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4ce80b3e70 0x7f4ce80d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.374+0000 7f4ced578640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce80b3940 con 0x7f4ce80b3070 2023-10-16T12:53:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.374+0000 7f4ced578640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce80b45e0 con 0x7f4ce80b3e70 2023-10-16T12:53:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.374+0000 7f4ced578640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80c7880 con 0x7f4ce80cfc10 2023-10-16T12:53:34.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.375+0000 7f4ce67fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.375+0000 7f4ce67fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80d0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:35774/0 (socket says 172.21.15.60:35774) 2023-10-16T12:53:34.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.375+0000 7f4ce67fc640 1 -- 172.21.15.60:0/1469720303 learned_addr learned my addr 172.21.15.60:0/1469720303 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.375+0000 7f4ce77fe640 1 --2- 172.21.15.60:0/1469720303 >> v2:172.21.15.144:3300/0 conn(0x7f4ce80b3e70 0x7f4ce80d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.375+0000 7f4ce67fc640 1 -- 172.21.15.60:0/1469720303 >> v2:172.21.15.144:3300/0 conn(0x7f4ce80b3e70 msgr2=0x7f4ce80d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:34.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.376+0000 7f4ce67fc640 1 --2- 172.21.15.60:0/1469720303 >> v2:172.21.15.144:3300/0 conn(0x7f4ce80b3e70 0x7f4ce80d1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.376+0000 7f4ce67fc640 1 -- 172.21.15.60:0/1469720303 >> v1:172.21.15.144:6789/0 conn(0x7f4ce80b3070 legacy=0x7f4ce80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.376+0000 7f4ce67fc640 1 -- 172.21.15.60:0/1469720303 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80b3b90 con 0x7f4ce80cfc10 2023-10-16T12:53:34.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.376+0000 7f4ce77fe640 1 --2- 172.21.15.60:0/1469720303 >> v2:172.21.15.144:3300/0 conn(0x7f4ce80b3e70 0x7f4ce80d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:53:34.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.377+0000 7f4ce67fc640 1 --2- 172.21.15.60:0/1469720303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80d0b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009870 tx=0x7f4cdc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881a38c4c2a34d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.377+0000 7f4ce5ffb640 1 -- 172.21.15.60:0/1469720303 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc030840 con 0x7f4ce80cfc10 2023-10-16T12:53:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.377+0000 7f4ce5ffb640 1 -- 172.21.15.60:0/1469720303 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc0309a0 con 0x7f4ce80cfc10 2023-10-16T12:53:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.377+0000 7f4ce5ffb640 1 -- 172.21.15.60:0/1469720303 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc030d30 con 0x7f4ce80cfc10 2023-10-16T12:53:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 -- 172.21.15.60:0/1469720303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 msgr2=0x7f4ce80d0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 --2- 172.21.15.60:0/1469720303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80d0b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009870 tx=0x7f4cdc02b980 comp rx=0 tx=0).stop 2023-10-16T12:53:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 -- 172.21.15.60:0/1469720303 shutdown_connections 2023-10-16T12:53:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 --2- 172.21.15.60:0/1469720303 >> v2:172.21.15.144:3300/0 conn(0x7f4ce80b3e70 0x7f4ce80d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 --2- 172.21.15.60:0/1469720303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80d0b00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 -- 172.21.15.60:0/1469720303 >> 172.21.15.60:0/1469720303 conn(0x7f4ce80d2d20 msgr2=0x7f4ce80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 -- 172.21.15.60:0/1469720303 shutdown_connections 2023-10-16T12:53:34.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.378+0000 7f4ced578640 1 -- 172.21.15.60:0/1469720303 wait complete. 2023-10-16T12:53:34.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.380+0000 7f4ced578640 1 Processor -- start 2023-10-16T12:53:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.380+0000 7f4ced578640 1 -- start start 2023-10-16T12:53:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ced578640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 0x7f4ce8149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ced578640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ce80b3e70 0x7f4ce80b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ced578640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ced578640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81497b0 con 0x7f4ce80b3070 2023-10-16T12:53:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ced578640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80b22b0 con 0x7f4ce80b3e70 2023-10-16T12:53:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ced578640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81469b0 con 0x7f4ce80cfc10 2023-10-16T12:53:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ce6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 0x7f4ce8149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ce6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 0x7f4ce8149270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44472/0 (socket says 172.21.15.60:44472) 2023-10-16T12:53:34.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.381+0000 7f4ce6ffd640 1 -- 172.21.15.60:0/1442920465 learned_addr learned my addr 172.21.15.60:0/1442920465 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:34.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce77fe640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce67fc640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ce80b3e70 0x7f4ce80b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce6ffd640 1 -- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 msgr2=0x7f4ce80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce6ffd640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce6ffd640 1 -- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ce80b3e70 msgr2=0x7f4ce80b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce6ffd640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ce80b3e70 0x7f4ce80b1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce6ffd640 1 -- 172.21.15.60:0/1442920465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0009a70 con 0x7f4ce80b3070 2023-10-16T12:53:34.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce77fe640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:34.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ce6ffd640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 0x7f4ce8149270 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c450 tx=0x7f4cd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80052d0 con 0x7f4ce80b3070 2023-10-16T12:53:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd8005450 con 0x7f4ce80b3070 2023-10-16T12:53:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8003880 con 0x7f4ce80b3070 2023-10-16T12:53:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.382+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc0094d0 con 0x7f4ce80b3070 2023-10-16T12:53:34.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.383+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce8146f70 con 0x7f4ce80b3070 2023-10-16T12:53:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.384+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce804e160 con 0x7f4ce80b3070 2023-10-16T12:53:34.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.385+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4cd80039e0 con 0x7f4ce80b3070 2023-10-16T12:53:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.385+0000 7f4ccf7fe640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4cd406af90 0x7f4cd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.386+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cd8089370 con 0x7f4ce80b3070 2023-10-16T12:53:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.386+0000 7f4ce67fc640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4cd406af90 0x7f4cd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:34.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.387+0000 7f4ce67fc640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4cd406af90 0x7f4cd406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc036000 tx=0x7f4cdc03b040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:34.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.390+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4cd8054310 con 0x7f4ce80b3070 2023-10-16T12:53:34.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.624+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4ce80d25d0 con 0x7f4ce80b3070 2023-10-16T12:53:34.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.625+0000 7f4ccf7fe640 1 -- 172.21.15.60:0/1442920465 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4cd805ce60 con 0x7f4ce80b3070 2023-10-16T12:53:34.629 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:53:34.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4cd406af90 msgr2=0x7f4cd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4cd406af90 0x7f4cd406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc036000 tx=0x7f4cdc03b040 comp rx=0 tx=0).stop 2023-10-16T12:53:34.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 msgr2=0x7f4ce8149270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 0x7f4ce8149270 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c450 tx=0x7f4cd800c910 comp rx=0 tx=0).stop 2023-10-16T12:53:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 shutdown_connections 2023-10-16T12:53:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f4cd406af90 0x7f4cd406d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ce80cfc10 0x7f4ce80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ce80b3e70 0x7f4ce80b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 --2- 172.21.15.60:0/1442920465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ce80b3070 0x7f4ce8149270 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 >> 172.21.15.60:0/1442920465 conn(0x7f4ce80d2d20 msgr2=0x7f4ce80c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.629+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 shutdown_connections 2023-10-16T12:53:34.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:34.630+0000 7f4ced578640 1 -- 172.21.15.60:0/1442920465 wait complete. 2023-10-16T12:53:34.650 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:53:34.651 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:53:34.651 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:53:34.652 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:53:34.652 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:53:34.652 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:53:34.653 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:53:34.653 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:53:34.653 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:53:34.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:34.654 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:53:34.714 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:53:34.715 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:53:34.715 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:53:34.715 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.715 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:53:34.716 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.716 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:53:34.716 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:53:34.716 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:53:34.717 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:53:34.717 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:53:34.717 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:53:34.717 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:53:34.718 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:53:34.718 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:53:34.718 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:53:34.718 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41172sec preferred_lft 41172sec 2023-10-16T12:53:34.719 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:53:34.719 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.719 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:53:34.719 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:53:34.719 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:53:34.720 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:53:34.720 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:53:34.720 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:53:34.721 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.721 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:53:34.721 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.721 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:53:34.722 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:53:34.722 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:53:34.722 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.722 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:53:34.723 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:53:34.723 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:53:34.723 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:53:34.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:34.724 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:53:34.830 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:53:34.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:34.834 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:53:34.894 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:53:34.894 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:53:34.895 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:53:34.895 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:53:34.896 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:53:34.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:34.897 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:53:34.898 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:53:34.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:53:34.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:53:34.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:53:34.900 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:53:35.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:35.101 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:53:35.156 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:53:35.156 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:53:35.157 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:53:35.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:35.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:53:35.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:53:35.260 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:53:35.319 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:53:35.319 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:53:35.320 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:53:35.375 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:53:35.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:35.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:53:35.518 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:53:35.518 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:53:35.553 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:53:35.553 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:53:35.553 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:53:35.554 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:53:35.554 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:53:35.556 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:53:35.556 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:53:35.556 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:53:35.556 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:53:35.556 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:53:35.560 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:53:35.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:35.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:53:35.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:35.655 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:53:35.726 INFO:teuthology.orchestra.run.smithi060.stdout:client6437 2023-10-16T12:53:35.726 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:35.727 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:35.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:53:36.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:35.999+0000 7fe2167e2640 1 Processor -- start 2023-10-16T12:53:36.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.000+0000 7fe2167e2640 1 -- start start 2023-10-16T12:53:36.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.001+0000 7fe2167e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe2100ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.001+0000 7fe2167e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100cb030 con 0x7fe21006a4b0 2023-10-16T12:53:36.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.001+0000 7fe2167e2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100c91c0 con 0x7fe21015ad70 2023-10-16T12:53:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.001+0000 7fe2167e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100c94a0 con 0x7fe2100caac0 2023-10-16T12:53:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.002+0000 7fe20f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe21006a4b0 0x7fe2100ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44842/0 (socket says 172.21.15.60:44842) 2023-10-16T12:53:36.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.002+0000 7fe20f7fe640 1 -- 172.21.15.60:0/1302225154 learned_addr learned my addr 172.21.15.60:0/1302225154 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.002+0000 7fe214d7f640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe2100ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.002+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225662910 0 0) 0x7fe2100cb030 con 0x7fe21006a4b0 2023-10-16T12:53:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.002+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8003520 con 0x7fe21006a4b0 2023-10-16T12:53:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731116240 0 0) 0x7fe1f8003520 con 0x7fe21006a4b0 2023-10-16T12:53:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 msgr2=0x7fe2100ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe2100ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 >> v1:172.21.15.144:6789/0 conn(0x7fe21015ad70 legacy=0x7fe210069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100c9780 con 0x7fe21006a4b0 2023-10-16T12:53:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe204001670 con 0x7fe21006a4b0 2023-10-16T12:53:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 >> v1:172.21.15.60:6789/0 conn(0x7fe21006a4b0 legacy=0x7fe2100ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe1f8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f80047c0 0x7fe1f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe1f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100c94a0 con 0x7fe1f80039a0 2023-10-16T12:53:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f800ba30 con 0x7fe1f80047c0 2023-10-16T12:53:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f800bbf0 con 0x7fe2100caac0 2023-10-16T12:53:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.003+0000 7fe20ffff640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe1f8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20ffff640 1 -- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 msgr2=0x7fe1f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe214d7f640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe1f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f80047c0 0x7fe1f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20ffff640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe1f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20ffff640 1 -- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f80047c0 msgr2=0x7fe1f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20ffff640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f80047c0 0x7fe1f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20ffff640 1 -- 172.21.15.60:0/1302225154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8003520 con 0x7fe1f80039a0 2023-10-16T12:53:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f80047c0 0x7fe1f8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.004+0000 7fe214d7f640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100caac0 0x7fe1f8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.005+0000 7fe20ffff640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe1f8003dc0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fe200001e20 tx=0x7fe1f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4be2d6870384fe2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.005+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200003b70 con 0x7fe1f80039a0 2023-10-16T12:53:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.005+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe200003cd0 con 0x7fe1f80039a0 2023-10-16T12:53:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.005+0000 7fe20effd640 1 -- 172.21.15.60:0/1302225154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200005310 con 0x7fe1f80039a0 2023-10-16T12:53:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe2167e2640 1 -- 172.21.15.60:0/1302225154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 msgr2=0x7fe1f8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe2167e2640 1 --2- 172.21.15.60:0/1302225154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe1f8003dc0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fe200001e20 tx=0x7fe1f80057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe214d7f640 1 -- 172.21.15.60:0/1302225154 reap_dead start 2023-10-16T12:53:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe2167e2640 1 -- 172.21.15.60:0/1302225154 shutdown_connections 2023-10-16T12:53:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe2167e2640 1 -- 172.21.15.60:0/1302225154 >> 172.21.15.60:0/1302225154 conn(0x7fe2100ce9b0 msgr2=0x7fe2100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe2167e2640 1 -- 172.21.15.60:0/1302225154 shutdown_connections 2023-10-16T12:53:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.006+0000 7fe2167e2640 1 -- 172.21.15.60:0/1302225154 wait complete. 2023-10-16T12:53:36.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.008+0000 7fe2167e2640 1 Processor -- start 2023-10-16T12:53:36.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.008+0000 7fe2167e2640 1 -- start start 2023-10-16T12:53:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe2167e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe2100c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe2167e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100c24a0 0x7fe21014d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe2167e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21014dbb0 0x7fe2100cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe2167e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21014e030 con 0x7fe1f80039a0 2023-10-16T12:53:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe2167e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21014c350 con 0x7fe21014dbb0 2023-10-16T12:53:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe2167e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100c48c0 con 0x7fe2100c24a0 2023-10-16T12:53:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe2100c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe2100c4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44492/0 (socket says 172.21.15.60:44492) 2023-10-16T12:53:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.009+0000 7fe20ffff640 1 -- 172.21.15.60:0/650776551 learned_addr learned my addr 172.21.15.60:0/650776551 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100c24a0 0x7fe21014d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe214d7f640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21014dbb0 0x7fe2100cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20ffff640 1 -- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100c24a0 msgr2=0x7fe21014d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20ffff640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100c24a0 0x7fe21014d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20ffff640 1 -- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21014dbb0 msgr2=0x7fe2100cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20ffff640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21014dbb0 0x7fe2100cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20ffff640 1 -- 172.21.15.60:0/650776551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21014b780 con 0x7fe1f80039a0 2023-10-16T12:53:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe214d7f640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21014dbb0 0x7fe2100cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20ffff640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe2100c4380 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fe200001f40 tx=0x7fe2000014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20003e070 con 0x7fe1f80039a0 2023-10-16T12:53:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.010+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe200006b50 con 0x7fe1f80039a0 2023-10-16T12:53:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.011+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21014ba20 con 0x7fe1f80039a0 2023-10-16T12:53:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.011+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200007030 con 0x7fe1f80039a0 2023-10-16T12:53:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.011+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21014bfe0 con 0x7fe1f80039a0 2023-10-16T12:53:36.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.012+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21014c170 con 0x7fe1f80039a0 2023-10-16T12:53:36.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.013+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe200007190 con 0x7fe1f80039a0 2023-10-16T12:53:36.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.014+0000 7fe20cff9640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe1fc06afc0 0x7fe1fc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.014+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2000ab3b0 con 0x7fe1f80039a0 2023-10-16T12:53:36.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.015+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe1fc06afc0 0x7fe1fc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.016+0000 7fe20f7fe640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe1fc06afc0 0x7fe1fc06d480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe2040017a0 tx=0x7fe204009040 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:36.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.019+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2000761a0 con 0x7fe1f80039a0 2023-10-16T12:53:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.239+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 --> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2100c9ca0 con 0x7fe1fc06afc0 2023-10-16T12:53:36.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.244+0000 7fe20cff9640 1 -- 172.21.15.60:0/650776551 <== mgr.6886 v2:172.21.15.144:6832/682650079 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe20400d040 con 0x7fe1fc06afc0 2023-10-16T12:53:36.246 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:53:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:53:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:53:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:53:36.247 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:53:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.246+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe1fc06afc0 msgr2=0x7fe1fc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.246+0000 7fe2167e2640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe1fc06afc0 0x7fe1fc06d480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe2040017a0 tx=0x7fe204009040 comp rx=0 tx=0).stop 2023-10-16T12:53:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.246+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 msgr2=0x7fe2100c4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.246+0000 7fe2167e2640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe2100c4380 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fe200001f40 tx=0x7fe2000014a0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.246+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 shutdown_connections 2023-10-16T12:53:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fe1fc06afc0 0x7fe1fc06d480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2100c24a0 0x7fe21014d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21014dbb0 0x7fe2100cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 --2- 172.21.15.60:0/650776551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f80039a0 0x7fe2100c4380 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 >> 172.21.15.60:0/650776551 conn(0x7fe2100ce9b0 msgr2=0x7fe2100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 shutdown_connections 2023-10-16T12:53:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.247+0000 7fe2167e2640 1 -- 172.21.15.60:0/650776551 wait complete. 2023-10-16T12:53:36.269 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:53:36.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.490+0000 7fbb7b54e640 1 Processor -- start 2023-10-16T12:53:36.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.490+0000 7fbb7b54e640 1 -- start start 2023-10-16T12:53:36.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.491+0000 7fbb7b54e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb741573e0 0x7fbb740ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.491+0000 7fbb7b54e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74070950 0x7fbb7415a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7b54e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740ca3a0 con 0x7fbb74159e70 2023-10-16T12:53:36.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7b54e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740cad70 con 0x7fbb741573e0 2023-10-16T12:53:36.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7b54e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740c9d70 con 0x7fbb74070950 2023-10-16T12:53:36.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7a54c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb741573e0 0x7fbb740ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7a54c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb741573e0 0x7fbb740ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44496/0 (socket says 172.21.15.60:44496) 2023-10-16T12:53:36.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb79d4b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbb74159e70 0x7fbb74070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44850/0 (socket says 172.21.15.60:44850) 2023-10-16T12:53:36.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7a54c640 1 -- 172.21.15.60:0/1871250077 learned_addr learned my addr 172.21.15.60:0/1871250077 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:36.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7a54c640 1 -- 172.21.15.60:0/1871250077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74070950 msgr2=0x7fbb7415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:36.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7ad4d640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74070950 0x7fbb7415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7a54c640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74070950 0x7fbb7415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.492+0000 7fbb7954a640 1 -- 172.21.15.60:0/1871250077 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253642116 0 0) 0x7fbb740ca3a0 con 0x7fbb74159e70 2023-10-16T12:53:36.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.493+0000 7fbb7a54c640 1 -- 172.21.15.60:0/1871250077 >> v1:172.21.15.60:6789/0 conn(0x7fbb74159e70 legacy=0x7fbb74070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.493+0000 7fbb7a54c640 1 -- 172.21.15.60:0/1871250077 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb741577c0 con 0x7fbb741573e0 2023-10-16T12:53:36.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.493+0000 7fbb7ad4d640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74070950 0x7fbb7415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.494+0000 7fbb7a54c640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb741573e0 0x7fbb740ca830 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000d150 tx=0x7fbb7002e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3285324ac3b7b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.494+0000 7fbb7954a640 1 -- 172.21.15.60:0/1871250077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7000b0d0 con 0x7fbb741573e0 2023-10-16T12:53:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.494+0000 7fbb7954a640 1 -- 172.21.15.60:0/1871250077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7000b230 con 0x7fbb741573e0 2023-10-16T12:53:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.494+0000 7fbb7954a640 1 -- 172.21.15.60:0/1871250077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70009600 con 0x7fbb741573e0 2023-10-16T12:53:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 -- 172.21.15.60:0/1871250077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb741573e0 msgr2=0x7fbb740ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb741573e0 0x7fbb740ca830 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000d150 tx=0x7fbb7002e160 comp rx=0 tx=0).stop 2023-10-16T12:53:36.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 -- 172.21.15.60:0/1871250077 shutdown_connections 2023-10-16T12:53:36.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74070950 0x7fbb7415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/1871250077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb741573e0 0x7fbb740ca830 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 -- 172.21.15.60:0/1871250077 >> 172.21.15.60:0/1871250077 conn(0x7fbb740cc230 msgr2=0x7fbb740cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:36.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 -- 172.21.15.60:0/1871250077 shutdown_connections 2023-10-16T12:53:36.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.495+0000 7fbb7b54e640 1 -- 172.21.15.60:0/1871250077 wait complete. 2023-10-16T12:53:36.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.497+0000 7fbb7b54e640 1 Processor -- start 2023-10-16T12:53:36.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.497+0000 7fbb7b54e640 1 -- start start 2023-10-16T12:53:36.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7b54e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 0x7fbb7414edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7b54e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb741573e0 0x7fbb740c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7b54e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74159e70 0x7fbb7415bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7b54e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740c8f80 con 0x7fbb74070950 2023-10-16T12:53:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7b54e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740c8370 con 0x7fbb741573e0 2023-10-16T12:53:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7b54e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740c8650 con 0x7fbb74159e70 2023-10-16T12:53:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7a54c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 0x7fbb7414edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7a54c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 0x7fbb7414edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44508/0 (socket says 172.21.15.60:44508) 2023-10-16T12:53:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.498+0000 7fbb7a54c640 1 -- 172.21.15.60:0/58856497 learned_addr learned my addr 172.21.15.60:0/58856497 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7a54c640 1 -- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74159e70 msgr2=0x7fbb7415bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7a54c640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74159e70 0x7fbb7415bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7a54c640 1 -- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb741573e0 msgr2=0x7fbb740c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:36.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7a54c640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb741573e0 0x7fbb740c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7a54c640 1 -- 172.21.15.60:0/58856497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7000bd40 con 0x7fbb74070950 2023-10-16T12:53:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7a54c640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 0x7fbb7414edb0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbb70000f80 tx=0x7fbb70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:36.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70036800 con 0x7fbb74070950 2023-10-16T12:53:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb70036980 con 0x7fbb74070950 2023-10-16T12:53:36.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70036d30 con 0x7fbb74070950 2023-10-16T12:53:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.499+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb740c8930 con 0x7fbb74070950 2023-10-16T12:53:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.500+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb740c7ed0 con 0x7fbb74070950 2023-10-16T12:53:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.501+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7404e9b0 con 0x7fbb74070950 2023-10-16T12:53:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.502+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 66) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbb7004a460 con 0x7fbb74070950 2023-10-16T12:53:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.502+0000 7fbb6affd640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fbb5806af90 0x7fbb5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.503+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb700accf0 con 0x7fbb74070950 2023-10-16T12:53:36.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.503+0000 7fbb79d4b640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fbb5806af90 0x7fbb5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.504+0000 7fbb79d4b640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fbb5806af90 0x7fbb5806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007fd0 tx=0x7fbb64007920 comp rx=0 tx=0).ready entity=mgr.6886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.509+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb70077bf0 con 0x7fbb74070950 2023-10-16T12:53:36.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:36.765+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbb740b5730 con 0x7fbb74070950 2023-10-16T12:53:37.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.382+0000 7fbb6affd640 1 -- 172.21.15.60:0/58856497 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v67) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbb7000b720 con 0x7fbb74070950 2023-10-16T12:53:37.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fbb5806af90 msgr2=0x7fbb5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fbb5806af90 0x7fbb5806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007fd0 tx=0x7fbb64007920 comp rx=0 tx=0).stop 2023-10-16T12:53:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 msgr2=0x7fbb7414edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 0x7fbb7414edb0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbb70000f80 tx=0x7fbb70004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 shutdown_connections 2023-10-16T12:53:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fbb5806af90 0x7fbb5806d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb74159e70 0x7fbb7415bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb741573e0 0x7fbb740c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 --2- 172.21.15.60:0/58856497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb74070950 0x7fbb7414edb0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.386+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 >> 172.21.15.60:0/58856497 conn(0x7fbb740cc230 msgr2=0x7fbb740ad3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.387+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 shutdown_connections 2023-10-16T12:53:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.387+0000 7fbb7b54e640 1 -- 172.21.15.60:0/58856497 wait complete. 2023-10-16T12:53:37.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:53:37.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.613+0000 7ff6f68ec640 1 Processor -- start 2023-10-16T12:53:37.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.613+0000 7ff6f68ec640 1 -- start start 2023-10-16T12:53:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.614+0000 7ff6f68ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f015ad70 0x7ff6f0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.614+0000 7ff6f68ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6f00caac0 0x7ff6f006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.614+0000 7ff6f68ec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00cb030 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f68ec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00c91c0 con 0x7ff6f00caac0 2023-10-16T12:53:37.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f68ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00c94a0 con 0x7ff6f015ad70 2023-10-16T12:53:37.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6f00caac0 0x7ff6f006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:37.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6f00caac0 0x7ff6f006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44526/0 (socket says 172.21.15.60:44526) 2023-10-16T12:53:37.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 -- 172.21.15.60:0/1822030666 learned_addr learned my addr 172.21.15.60:0/1822030666 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:37.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6effff640 1 --2- 172.21.15.60:0/1822030666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f015ad70 0x7ff6f0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:37.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 -- 172.21.15.60:0/1822030666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f015ad70 msgr2=0x7ff6f0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:37.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 --2- 172.21.15.60:0/1822030666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f015ad70 0x7ff6f0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 -- 172.21.15.60:0/1822030666 >> v1:172.21.15.144:6789/0 conn(0x7ff6f00c9ca0 legacy=0x7ff6f00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.615+0000 7ff6f4e89640 1 -- 172.21.15.60:0/1822030666 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00c9780 con 0x7ff6f00caac0 2023-10-16T12:53:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.617+0000 7ff6f4e89640 1 --2- 172.21.15.60:0/1822030666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00caac0 0x7ff6f006a2e0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009870 tx=0x7ff6e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4a0a7d9cc9033c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.617+0000 7ff6ef7fe640 1 -- 172.21.15.60:0/1822030666 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8031640 con 0x7ff6f00caac0 2023-10-16T12:53:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.617+0000 7ff6ef7fe640 1 -- 172.21.15.60:0/1822030666 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e80317a0 con 0x7ff6f00caac0 2023-10-16T12:53:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.617+0000 7ff6ef7fe640 1 -- 172.21.15.60:0/1822030666 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8031b30 con 0x7ff6f00caac0 2023-10-16T12:53:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 -- 172.21.15.60:0/1822030666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00caac0 msgr2=0x7ff6f006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/1822030666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00caac0 0x7ff6f006a2e0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009870 tx=0x7ff6e802b8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 -- 172.21.15.60:0/1822030666 shutdown_connections 2023-10-16T12:53:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/1822030666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00caac0 0x7ff6f006a2e0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/1822030666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f015ad70 0x7ff6f0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 -- 172.21.15.60:0/1822030666 >> 172.21.15.60:0/1822030666 conn(0x7ff6f00ce9b0 msgr2=0x7ff6f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:37.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 -- 172.21.15.60:0/1822030666 shutdown_connections 2023-10-16T12:53:37.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.618+0000 7ff6f68ec640 1 -- 172.21.15.60:0/1822030666 wait complete. 2023-10-16T12:53:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.621+0000 7ff6f68ec640 1 Processor -- start 2023-10-16T12:53:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.621+0000 7ff6f68ec640 1 -- start start 2023-10-16T12:53:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f68ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 0x7ff6f0157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f68ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f00caac0 0x7ff6f00b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f68ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f015ad70 0x7ff6f00b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f68ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00c2390 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f68ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00c2c30 con 0x7ff6f015ad70 2023-10-16T12:53:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f68ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f0158490 con 0x7ff6f00caac0 2023-10-16T12:53:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 0x7ff6f0157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 0x7ff6f0157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44540/0 (socket says 172.21.15.60:44540) 2023-10-16T12:53:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6effff640 1 -- 172.21.15.60:0/3868508278 learned_addr learned my addr 172.21.15.60:0/3868508278 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6f4e89640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f015ad70 0x7ff6f00b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6effff640 1 -- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f00caac0 msgr2=0x7ff6f00b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.623+0000 7ff6e77fe640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f00caac0 0x7ff6f00b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:37.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6effff640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f00caac0 0x7ff6f00b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6effff640 1 -- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f015ad70 msgr2=0x7ff6f00b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:37.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6effff640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f015ad70 0x7ff6f00b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6effff640 1 -- 172.21.15.60:0/3868508278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e80094d0 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6e77fe640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f00caac0 0x7ff6f00b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:37.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6f4e89640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f015ad70 0x7ff6f00b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:37.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6effff640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 0x7ff6f0157f50 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0007e70 tx=0x7ff6e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:37.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0014620 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e0014780 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0014b30 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.624+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f00b3920 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.625+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00c28c0 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.626+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f004e160 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.627+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 67) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff6e0014c90 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.628+0000 7ff6ed7fa640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 0x7ff6d006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:37.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.628+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6e008a250 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.629+0000 7ff6e77fe640 1 -- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 msgr2=0x7ff6d006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:37.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.629+0000 7ff6e77fe640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 0x7ff6d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:37.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.632+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6e0055150 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.830+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff6f00af9f0 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.830+0000 7ff6e77fe640 1 -- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 msgr2=0x7ff6d006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.830+0000 7ff6e77fe640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 0x7ff6d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:37.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.834+0000 7ff6ed7fa640 1 -- 172.21.15.60:0/3868508278 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff6e0058770 con 0x7ff6f00c9ca0 2023-10-16T12:53:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.836+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 msgr2=0x7ff6d006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.836+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 0x7ff6d006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.836+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 msgr2=0x7ff6f0157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.836+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 0x7ff6f0157f50 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0007e70 tx=0x7ff6e000e4b0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 shutdown_connections 2023-10-16T12:53:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7ff6d006afe0 0x7ff6d006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff6f00caac0 0x7ff6f00b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f015ad70 0x7ff6f00b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 --2- 172.21.15.60:0/3868508278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00c9ca0 0x7ff6f0157f50 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 >> 172.21.15.60:0/3868508278 conn(0x7ff6f00ce9b0 msgr2=0x7ff6f00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 shutdown_connections 2023-10-16T12:53:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:37.837+0000 7ff6f68ec640 1 -- 172.21.15.60:0/3868508278 wait complete. 2023-10-16T12:53:37.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:53:37.907 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:37.906+0000 7fa3c040d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:53:37.927 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:37.925+0000 7f470523b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:53:37.961 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:37.960+0000 7fa3c040d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:53:37.985 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:37.983+0000 7f470523b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:53:38.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.068+0000 7f7ea10fc640 1 Processor -- start 2023-10-16T12:53:38.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.068+0000 7f7ea10fc640 1 -- start start 2023-10-16T12:53:38.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea10fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 0x7f7e9c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea10fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7e9c0ad970 0x7f7e9c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea10fc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c06c960 con 0x7f7e9c15ebc0 2023-10-16T12:53:38.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea10fc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c15b9d0 con 0x7f7e9c0ad970 2023-10-16T12:53:38.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea10fc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c15bcb0 con 0x7f7e9c06c040 2023-10-16T12:53:38.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea08fb640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7e9c15ebc0 0x7f7e9c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44858/0 (socket says 172.21.15.60:44858) 2023-10-16T12:53:38.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7ea08fb640 1 -- 172.21.15.60:0/2846409864 learned_addr learned my addr 172.21.15.60:0/2846409864 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:38.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 0x7f7e9c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.070+0000 7f7e9b7fe640 1 --2- 172.21.15.60:0/2846409864 >> v2:172.21.15.144:3300/0 conn(0x7f7e9c0ad970 0x7f7e9c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454387718 0 0) 0x7f7e9c06c960 con 0x7f7e9c15ebc0 2023-10-16T12:53:38.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e84003520 con 0x7f7e9c15ebc0 2023-10-16T12:53:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592241184 0 0) 0x7f7e84003520 con 0x7f7e9c15ebc0 2023-10-16T12:53:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 msgr2=0x7f7e9c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 0x7f7e9c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 >> v2:172.21.15.144:3300/0 conn(0x7f7e9c0ad970 msgr2=0x7f7e9c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 --2- 172.21.15.60:0/2846409864 >> v2:172.21.15.144:3300/0 conn(0x7f7e9c0ad970 0x7f7e9c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c15bf90 con 0x7f7e9c15ebc0 2023-10-16T12:53:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7e940019f0 con 0x7f7e9c15ebc0 2023-10-16T12:53:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9b7fe640 1 --2- 172.21.15.60:0/2846409864 >> v2:172.21.15.144:3300/0 conn(0x7f7e9c0ad970 0x7f7e9c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 >> v1:172.21.15.60:6789/0 conn(0x7f7e9c15ebc0 legacy=0x7f7e9c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e84003970 0x7f7e84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 0x7f7e840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e84004d00 0x7f7e84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c15b9d0 con 0x7f7e84003970 2023-10-16T12:53:38.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c15bcb0 con 0x7f7e84004d00 2023-10-16T12:53:38.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.071+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8400ba00 con 0x7f7e9c06c040 2023-10-16T12:53:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e84003970 0x7f7e84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9b7fe640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 0x7f7e840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7ea08fb640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e84004d00 0x7f7e84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9bfff640 1 -- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 msgr2=0x7f7e840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e9c06c040 0x7f7e840047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9bfff640 1 -- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e84004d00 msgr2=0x7f7e84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e84004d00 0x7f7e84005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7e9bfff640 1 -- 172.21.15.60:0/2846409864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84003520 con 0x7f7e84003970 2023-10-16T12:53:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.072+0000 7f7ea08fb640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e84004d00 0x7f7e84005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.073+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e84003970 0x7f7e84003d90 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009ac0 tx=0x7f7e840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d5899c1d26f405 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.073+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c02c6b0 con 0x7f7e84003970 2023-10-16T12:53:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c02c810 con 0x7f7e84003970 2023-10-16T12:53:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7e9affd640 1 -- 172.21.15.60:0/2846409864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c02cba0 con 0x7f7e84003970 2023-10-16T12:53:38.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2846409864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e84003970 msgr2=0x7f7e84003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2846409864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e84003970 0x7f7e84003d90 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009ac0 tx=0x7f7e840057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea08fb640 1 -- 172.21.15.60:0/2846409864 reap_dead start 2023-10-16T12:53:38.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2846409864 shutdown_connections 2023-10-16T12:53:38.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2846409864 >> 172.21.15.60:0/2846409864 conn(0x7f7e9c0ce6a0 msgr2=0x7f7e9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:38.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2846409864 shutdown_connections 2023-10-16T12:53:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.074+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2846409864 wait complete. 2023-10-16T12:53:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.076+0000 7f7ea10fc640 1 Processor -- start 2023-10-16T12:53:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.076+0000 7f7ea10fc640 1 -- start start 2023-10-16T12:53:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.077+0000 7f7ea10fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e84003970 0x7f7e9c156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.077+0000 7f7ea10fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e9c156e70 0x7f7e9c0afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.077+0000 7f7ea10fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 0x7f7e9c0ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.077+0000 7f7ea10fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0b0720 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.077+0000 7f7ea10fc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0c8310 con 0x7f7e9c156e70 2023-10-16T12:53:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.077+0000 7f7ea10fc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0c7270 con 0x7f7e84003970 2023-10-16T12:53:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 0x7f7e9c0ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 0x7f7e9c0ae0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44560/0 (socket says 172.21.15.60:44560) 2023-10-16T12:53:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 -- 172.21.15.60:0/2357085475 learned_addr learned my addr 172.21.15.60:0/2357085475 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 -- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e84003970 msgr2=0x7f7e9c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e84003970 0x7f7e9c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 -- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e9c156e70 msgr2=0x7f7e9c0afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e9c156e70 0x7f7e9c0afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 -- 172.21.15.60:0/2357085475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c0c7550 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.078+0000 7f7ea08fb640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 0x7f7e9c0ae0a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0b1260 tx=0x7f7e9400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.079+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94004d60 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.079+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9400bb00 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.079+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9400beb0 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.079+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c0c7830 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.079+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c0c7df0 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.080+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c04e9b0 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.081+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 67) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7e94007070 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.082+0000 7f7e98ff9640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 0x7f7e8806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.082+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e94089c10 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.082+0000 7f7e9bfff640 1 -- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 msgr2=0x7f7e8806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.082+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 0x7f7e8806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.086+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e94054ab0 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.282+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7e9c06b2b0 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.283+0000 7f7e9bfff640 1 -- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 msgr2=0x7f7e8806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:38.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.283+0000 7f7e9bfff640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 0x7f7e8806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:38.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.287+0000 7f7e98ff9640 1 -- 172.21.15.60:0/2357085475 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7e940580d0 con 0x7f7e9c0b02a0 2023-10-16T12:53:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 msgr2=0x7f7e8806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:38.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 0x7f7e8806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 msgr2=0x7f7e9c0ae0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 0x7f7e9c0ae0a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0b1260 tx=0x7f7e9400b4a0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 shutdown_connections 2023-10-16T12:53:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7f7e8806afe0 0x7f7e8806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e9c0b02a0 0x7f7e9c0ae0a0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e9c156e70 0x7f7e9c0afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 --2- 172.21.15.60:0/2357085475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e84003970 0x7f7e9c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 >> 172.21.15.60:0/2357085475 conn(0x7f7e9c0ce6a0 msgr2=0x7f7e9c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 shutdown_connections 2023-10-16T12:53:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.290+0000 7f7ea10fc640 1 -- 172.21.15.60:0/2357085475 wait complete. 2023-10-16T12:53:38.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:38.318 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-10-16T12:53:38.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-10-16T12:53:38.412 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.411+0000 7fa3c040d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:53:38.465 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.463+0000 7fa3c040d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:53:38.475 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.473+0000 7f470523b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:53:38.516 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.514+0000 7fa3c040d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:53:38.531 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.529+0000 7f470523b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:53:38.576 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.575+0000 7fa3c040d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:53:38.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.577+0000 7fc84efe9640 1 Processor -- start 2023-10-16T12:53:38.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.577+0000 7fc84efe9640 1 -- start start 2023-10-16T12:53:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84efe9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8480b1ca0 0x7fc8480d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84efe9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 0x7fc8480b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84efe9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8480d0970 con 0x7fc8480b2080 2023-10-16T12:53:38.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84efe9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8480d0ea0 con 0x7fc8480b1ca0 2023-10-16T12:53:38.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84efe9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8480d0bc0 con 0x7fc8480cfc40 2023-10-16T12:53:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84d7e6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc8480b2080 0x7fc8480ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44874/0 (socket says 172.21.15.60:44874) 2023-10-16T12:53:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84d7e6640 1 -- 172.21.15.60:0/4201865661 learned_addr learned my addr 172.21.15.60:0/4201865661 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.579+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/4201865661 >> v2:172.21.15.144:3300/0 conn(0x7fc8480b1ca0 0x7fc8480d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84e7e8640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 0x7fc8480b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740388747 0 0) 0x7fc8480d0970 con 0x7fc8480b2080 2023-10-16T12:53:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc834003520 con 0x7fc8480b2080 2023-10-16T12:53:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 626185648 0 0) 0x7fc834003520 con 0x7fc8480b2080 2023-10-16T12:53:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 msgr2=0x7fc8480b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 0x7fc8480b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 >> v2:172.21.15.144:3300/0 conn(0x7fc8480b1ca0 msgr2=0x7fc8480d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 --2- 172.21.15.60:0/4201865661 >> v2:172.21.15.144:3300/0 conn(0x7fc8480b1ca0 0x7fc8480d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8480cd940 con 0x7fc8480b2080 2023-10-16T12:53:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc838001670 con 0x7fc8480b2080 2023-10-16T12:53:38.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 >> v1:172.21.15.60:6789/0 conn(0x7fc8480b2080 legacy=0x7fc8480ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 0x7fc834003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc834003ef0 0x7fc8340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc834004d00 0x7fc834005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8480d0ea0 con 0x7fc834003ef0 2023-10-16T12:53:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8480d0bc0 con 0x7fc834004d00 2023-10-16T12:53:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.580+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83400b9b0 con 0x7fc8480cfc40 2023-10-16T12:53:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.581+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc834003ef0 0x7fc8340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.581+0000 7fc84d7e6640 1 -- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 msgr2=0x7fc834003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.581+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8480cfc40 0x7fc834003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.581+0000 7fc84d7e6640 1 -- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc834004d00 msgr2=0x7fc834005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:38.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.581+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc834004d00 0x7fc834005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.581+0000 7fc84d7e6640 1 -- 172.21.15.60:0/4201865661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc834003520 con 0x7fc834003ef0 2023-10-16T12:53:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.582+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc834003ef0 0x7fc8340047c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fc83800c290 tx=0x7fc8340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37bcada3a9a3e8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.582+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838005ad0 con 0x7fc834003ef0 2023-10-16T12:53:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.582+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc838005c30 con 0x7fc834003ef0 2023-10-16T12:53:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.582+0000 7fc84cfe5640 1 -- 172.21.15.60:0/4201865661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83802c8d0 con 0x7fc834003ef0 2023-10-16T12:53:38.591 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.583+0000 7f470523b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:53:38.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84efe9640 1 -- 172.21.15.60:0/4201865661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc834003ef0 msgr2=0x7fc8340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84efe9640 1 --2- 172.21.15.60:0/4201865661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc834003ef0 0x7fc8340047c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fc83800c290 tx=0x7fc8340057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84e7e8640 1 -- 172.21.15.60:0/4201865661 reap_dead start 2023-10-16T12:53:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84efe9640 1 -- 172.21.15.60:0/4201865661 shutdown_connections 2023-10-16T12:53:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84efe9640 1 -- 172.21.15.60:0/4201865661 >> 172.21.15.60:0/4201865661 conn(0x7fc8480d2b90 msgr2=0x7fc8480d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84efe9640 1 -- 172.21.15.60:0/4201865661 shutdown_connections 2023-10-16T12:53:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.583+0000 7fc84efe9640 1 -- 172.21.15.60:0/4201865661 wait complete. 2023-10-16T12:53:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.585+0000 7fc84efe9640 1 Processor -- start 2023-10-16T12:53:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.585+0000 7fc84efe9640 1 -- start start 2023-10-16T12:53:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84efe9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc834003ef0 0x7fc84815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84efe9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8481e67f0 0x7fc84815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84efe9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc84815cd20 0x7fc8480cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84efe9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84815edc0 con 0x7fc8481e67f0 2023-10-16T12:53:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84efe9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84815f0a0 con 0x7fc84815cd20 2023-10-16T12:53:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84efe9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84815f380 con 0x7fc834003ef0 2023-10-16T12:53:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84d7e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8481e67f0 0x7fc84815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84d7e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8481e67f0 0x7fc84815c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44578/0 (socket says 172.21.15.60:44578) 2023-10-16T12:53:38.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.586+0000 7fc84d7e6640 1 -- 172.21.15.60:0/3572881411 learned_addr learned my addr 172.21.15.60:0/3572881411 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:38.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc834003ef0 0x7fc84815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84e7e8640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc84815cd20 0x7fc8480cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:38.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84d7e6640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc834003ef0 msgr2=0x7fc84815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc834003ef0 0x7fc84815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84d7e6640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc84815cd20 msgr2=0x7fc8480cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:38.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc84815cd20 0x7fc8480cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:38.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84d7e6640 1 -- 172.21.15.60:0/3572881411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84815f660 con 0x7fc8481e67f0 2023-10-16T12:53:38.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84d7e6640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8481e67f0 0x7fc84815c7e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fc83802c470 tx=0x7fc838000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:38.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83803c040 con 0x7fc8481e67f0 2023-10-16T12:53:38.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc838007290 con 0x7fc8481e67f0 2023-10-16T12:53:38.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc838007640 con 0x7fc8481e67f0 2023-10-16T12:53:38.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.587+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84815f940 con 0x7fc8481e67f0 2023-10-16T12:53:38.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.588+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8480c7700 con 0x7fc8481e67f0 2023-10-16T12:53:38.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.589+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8480c78f0 con 0x7fc8481e67f0 2023-10-16T12:53:38.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.590+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 67) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc838032070 con 0x7fc8481e67f0 2023-10-16T12:53:38.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.590+0000 7fc83e7fc640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 0x7fc82c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:38.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.590+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8380aacc0 con 0x7fc8481e67f0 2023-10-16T12:53:38.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.591+0000 7fc84dfe7640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 msgr2=0x7fc82c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:38.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.591+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 0x7fc82c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:38.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.595+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8380b1080 con 0x7fc8481e67f0 2023-10-16T12:53:38.627 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.625+0000 7fa3c040d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:53:38.651 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.649+0000 7f470523b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:53:38.704 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.701+0000 7f470523b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:53:38.737 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.735+0000 7fa3c040d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:53:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.792+0000 7fc84dfe7640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 msgr2=0x7fc82c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.792+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 0x7fc82c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:38.799 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.797+0000 7fa3c040d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:38.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:38.816+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 --> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc84815f210 con 0x7fc82c06b010 2023-10-16T12:53:38.821 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.818+0000 7f470523b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:53:38.847 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.846+0000 7fa3c040d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:53:38.887 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.884+0000 7f470523b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:38.905 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.903+0000 7fa3c040d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:53:38.938 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.936+0000 7f470523b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:53:38.955 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:38.953+0000 7fa3c040d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:53:38.998 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:38.995+0000 7f470523b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:53:39.050 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:39.048+0000 7f470523b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:53:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:39.193+0000 7fc84dfe7640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 msgr2=0x7fc82c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:39.193+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 0x7fc82c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:53:39.335 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:39.333+0000 7fa3c040d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:53:39.452 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:39.449+0000 7f470523b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:53:39.493 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:39.491+0000 7fa3c040d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:53:39.602 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:39.600+0000 7fa3c040d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:53:39.620 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:39.618+0000 7f470523b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:53:39.652 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:39.651+0000 7fa3c040d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:53:39.733 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:39.731+0000 7f470523b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:53:39.759 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:39.758+0000 7fa3c040d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:39.785 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:39.783+0000 7f470523b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:53:39.864 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:39.862+0000 7fa3c040d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:53:39.899 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:39.896+0000 7f470523b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:39.994+0000 7fc84dfe7640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 msgr2=0x7fc82c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/682650079 2023-10-16T12:53:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:39.994+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 0x7fc82c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:53:40.012 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:40.009+0000 7f470523b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:53:40.022 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:40.021+0000 7fa3c040d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:40.072 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:40.071+0000 7fa3c040d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:40.177 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:40.174+0000 7f470523b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:40.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:40.189+0000 7fa3c040d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:53:40.229 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:40.226+0000 7f470523b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:40.353 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:40.351+0000 7f470523b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:53:41.016 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.015+0000 7fa3c040d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:53:41.241 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:41.238+0000 7f470523b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:53:41.311 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.309+0000 7fa3c040d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:53:41.367 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.366+0000 7fa3c040d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:53:41.481 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.479+0000 7fa3c040d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:53:41.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:41.498+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 68) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fc838076020 con 0x7fc8481e67f0 2023-10-16T12:53:41.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:41.499+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 msgr2=0x7fc82c06d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:41.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:41.499+0000 7fc83e7fc640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/682650079,v1:172.21.15.144:6833/682650079] conn(0x7fc82c06b010 0x7fc82c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:41.559 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:41.556+0000 7f470523b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:53:41.624 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:41.622+0000 7f470523b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:53:41.750 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:41.747+0000 7f470523b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:53:41.759 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.757+0000 7fa33cbc3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.757+0000 7fa33cbc3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.757+0000 7fa33cbc3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.757+0000 7fa33cbc3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.760 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.757+0000 7fa33cbc3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.758+0000 7fa33f408640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.758+0000 7fa33f408640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.758+0000 7fa33f408640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.758+0000 7fa33f408640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:41.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:41.758+0000 7fa33f408640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:42.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.511+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 69) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc83808d660 con 0x7fc8481e67f0 2023-10-16T12:53:42.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.512+0000 7fc83e7fc640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc82c035910 0x7fc82c074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:42.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.512+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc84815f210 con 0x7fc82c035910 2023-10-16T12:53:42.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.517+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc82c035910 0x7fc82c074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.532+0000 7fc84dfe7640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc82c035910 0x7fc82c074ed0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8440093d0 tx=0x7fc844008040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:42.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.707+0000 7fc83e7fc640 1 -- 172.21.15.60:0/3572881411 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fc84815f210 con 0x7fc82c035910 2023-10-16T12:53:42.710 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1/minutes 2023-10-16T12:53:42.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc82c035910 msgr2=0x7fc82c074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:42.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc82c035910 0x7fc82c074ed0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8440093d0 tx=0x7fc844008040 comp rx=0 tx=0).stop 2023-10-16T12:53:42.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8481e67f0 msgr2=0x7fc84815c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:42.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 --2- 172.21.15.60:0/3572881411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8481e67f0 0x7fc84815c7e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fc83802c470 tx=0x7fc838000fe0 comp rx=0 tx=0).stop 2023-10-16T12:53:42.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84e7e8640 1 -- 172.21.15.60:0/3572881411 reap_dead start 2023-10-16T12:53:42.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 shutdown_connections 2023-10-16T12:53:42.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 >> 172.21.15.60:0/3572881411 conn(0x7fc8480d2b90 msgr2=0x7fc8480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:42.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 shutdown_connections 2023-10-16T12:53:42.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:42.711+0000 7fc84efe9640 1 -- 172.21.15.60:0/3572881411 wait complete. 2023-10-16T12:53:42.734 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-10-16T12:53:42.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:42.735 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-10-16T12:53:42.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-10-16T12:53:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.027+0000 7f400cc14640 1 Processor -- start 2023-10-16T12:53:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.027+0000 7f400cc14640 1 -- start start 2023-10-16T12:53:43.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.028+0000 7f400cc14640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f400806c040 0x7f400806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.028+0000 7f400cc14640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40080cda80 0x7f400815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.028+0000 7f400cc14640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400806c960 con 0x7f40080ad970 2023-10-16T12:53:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.028+0000 7f400cc14640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400815b9d0 con 0x7f400806c040 2023-10-16T12:53:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.028+0000 7f400cc14640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400815bcb0 con 0x7f40080cda80 2023-10-16T12:53:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.028+0000 7f40077fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f400806c040 0x7f400806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f4006ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f40080ad970 0x7f400815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35028/0 (socket says 172.21.15.60:35028) 2023-10-16T12:53:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f4006ffd640 1 -- 172.21.15.60:0/203916962 learned_addr learned my addr 172.21.15.60:0/203916962 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f40077fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f400806c040 0x7f400806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56132/0 (socket says 172.21.15.60:56132) 2023-10-16T12:53:43.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f4007fff640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40080cda80 0x7f400815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f40077fe640 1 -- 172.21.15.60:0/203916962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40080cda80 msgr2=0x7f400815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f40077fe640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40080cda80 0x7f400815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f40077fe640 1 -- 172.21.15.60:0/203916962 >> v1:172.21.15.60:6789/0 conn(0x7f40080ad970 legacy=0x7f400815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f40077fe640 1 -- 172.21.15.60:0/203916962 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400815bf90 con 0x7f400806c040 2023-10-16T12:53:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f4007fff640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40080cda80 0x7f400815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.029+0000 7f40067fc640 1 -- 172.21.15.60:0/203916962 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230174327 0 0) 0x7f400806c960 con 0x7f40080ad970 2023-10-16T12:53:43.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.030+0000 7f40077fe640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400806c040 0x7f400806c420 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009d90 tx=0x7f3ff802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e2e49b9091fc79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:43.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.030+0000 7f40067fc640 1 -- 172.21.15.60:0/203916962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff802cc50 con 0x7f400806c040 2023-10-16T12:53:43.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f40067fc640 1 -- 172.21.15.60:0/203916962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff802cdb0 con 0x7f400806c040 2023-10-16T12:53:43.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f40067fc640 1 -- 172.21.15.60:0/203916962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8034660 con 0x7f400806c040 2023-10-16T12:53:43.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 -- 172.21.15.60:0/203916962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400806c040 msgr2=0x7f400806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400806c040 0x7f400806c420 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009d90 tx=0x7f3ff802bea0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 -- 172.21.15.60:0/203916962 shutdown_connections 2023-10-16T12:53:43.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40080cda80 0x7f400815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 --2- 172.21.15.60:0/203916962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f400806c040 0x7f400806c420 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 -- 172.21.15.60:0/203916962 >> 172.21.15.60:0/203916962 conn(0x7f40080ce6a0 msgr2=0x7f40080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 -- 172.21.15.60:0/203916962 shutdown_connections 2023-10-16T12:53:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.031+0000 7f400cc14640 1 -- 172.21.15.60:0/203916962 wait complete. 2023-10-16T12:53:43.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.033+0000 7f400cc14640 1 Processor -- start 2023-10-16T12:53:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.033+0000 7f400cc14640 1 -- start start 2023-10-16T12:53:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f400cc14640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400806c040 0x7f40080c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f400cc14640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ad970 0x7f40080c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f400cc14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 0x7f40080c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f400cc14640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080b12c0 con 0x7f40080cda80 2023-10-16T12:53:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f400cc14640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c6af0 con 0x7f40080ad970 2023-10-16T12:53:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f400cc14640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c8280 con 0x7f400806c040 2023-10-16T12:53:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f4007fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 0x7f40080c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.034+0000 7f4007fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 0x7f40080c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56144/0 (socket says 172.21.15.60:56144) 2023-10-16T12:53:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 -- 172.21.15.60:0/4187964983 learned_addr learned my addr 172.21.15.60:0/4187964983 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f40077fe640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400806c040 0x7f40080c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 -- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400806c040 msgr2=0x7f40080c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400806c040 0x7f40080c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4006ffd640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ad970 0x7f40080c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 -- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ad970 msgr2=0x7f40080c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ad970 0x7f40080c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 -- 172.21.15.60:0/4187964983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8009a70 con 0x7f40080cda80 2023-10-16T12:53:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f4007fff640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 0x7f40080c7be0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f400000c7f0 tx=0x7f400000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.035+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40000077c0 con 0x7f40080cda80 2023-10-16T12:53:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.036+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4000007920 con 0x7f40080cda80 2023-10-16T12:53:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.036+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4000007d10 con 0x7f40080cda80 2023-10-16T12:53:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.036+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40080b14d0 con 0x7f40080cda80 2023-10-16T12:53:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.036+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40080b1a90 con 0x7f40080cda80 2023-10-16T12:53:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.037+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400806b330 con 0x7f40080cda80 2023-10-16T12:53:43.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.038+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 69) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f40000113f0 con 0x7f40080cda80 2023-10-16T12:53:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.039+0000 7f3ffffff640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f3fe806ad10 0x7f3fe806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.039+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f4000089750 con 0x7f40080cda80 2023-10-16T12:53:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.039+0000 7f40077fe640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f3fe806ad10 0x7f3fe806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.040+0000 7f40077fe640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f3fe806ad10 0x7f3fe806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8036000 tx=0x7f3ff8030000 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.044+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40000549e0 con 0x7f40080cda80 2023-10-16T12:53:43.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.262+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f40080cc550 con 0x7f3fe806ad10 2023-10-16T12:53:43.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.441+0000 7f3ffffff640 1 -- 172.21.15.60:0/4187964983 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f3ff802ca50 con 0x7f3fe806ad10 2023-10-16T12:53:43.444 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1/hourly 2023-10-16T12:53:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f3fe806ad10 msgr2=0x7f3fe806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f3fe806ad10 0x7f3fe806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8036000 tx=0x7f3ff8030000 comp rx=0 tx=0).stop 2023-10-16T12:53:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 msgr2=0x7f40080c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 0x7f40080c7be0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f400000c7f0 tx=0x7f400000ccb0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 shutdown_connections 2023-10-16T12:53:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f3fe806ad10 0x7f3fe806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40080cda80 0x7f40080c7be0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40080ad970 0x7f40080c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 --2- 172.21.15.60:0/4187964983 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f400806c040 0x7f40080c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.444+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 >> 172.21.15.60:0/4187964983 conn(0x7f40080ce6a0 msgr2=0x7f400815e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.445+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 shutdown_connections 2023-10-16T12:53:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.445+0000 7f400cc14640 1 -- 172.21.15.60:0/4187964983 wait complete. 2023-10-16T12:53:43.468 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-10-16T12:53:43.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:43.468 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-10-16T12:53:43.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-10-16T12:53:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.757+0000 7f938eba9640 1 Processor -- start 2023-10-16T12:53:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.758+0000 7f938eba9640 1 -- start start 2023-10-16T12:53:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.759+0000 7f938eba9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f938815d150 0x7f93880ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.759+0000 7f938eba9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9388160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.759+0000 7f938eba9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880ac890 con 0x7f9388160b90 2023-10-16T12:53:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.759+0000 7f938eba9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938815d530 con 0x7f938815d150 2023-10-16T12:53:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.759+0000 7f938eba9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9388161650 con 0x7f938806a4b0 2023-10-16T12:53:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.760+0000 7f9387fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9388160b90 0x7f9388069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35040/0 (socket says 172.21.15.60:35040) 2023-10-16T12:53:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.760+0000 7f9387fff640 1 -- 172.21.15.60:0/1967654767 learned_addr learned my addr 172.21.15.60:0/1967654767 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:43.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.760+0000 7f938c945640 1 --2- 172.21.15.60:0/1967654767 >> v2:172.21.15.144:3300/0 conn(0x7f938815d150 0x7f93880ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.760+0000 7f938d146640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9388160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.760+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269489262 0 0) 0x7f93880ac890 con 0x7f9388160b90 2023-10-16T12:53:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.760+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9378003520 con 0x7f9388160b90 2023-10-16T12:53:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416762433 0 0) 0x7f9378003520 con 0x7f9388160b90 2023-10-16T12:53:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 msgr2=0x7f9388160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9388160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 >> v2:172.21.15.144:3300/0 conn(0x7f938815d150 msgr2=0x7f93880ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 --2- 172.21.15.60:0/1967654767 >> v2:172.21.15.144:3300/0 conn(0x7f938815d150 0x7f93880ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388161930 con 0x7f9388160b90 2023-10-16T12:53:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9374001670 con 0x7f9388160b90 2023-10-16T12:53:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 >> v1:172.21.15.60:6789/0 conn(0x7f9388160b90 legacy=0x7f9388069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9378003970 0x7f9378003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93780047c0 0x7f9378004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9378005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938815d530 con 0x7f93780047c0 2023-10-16T12:53:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9388161650 con 0x7f9378003970 2023-10-16T12:53:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937800b9b0 con 0x7f938806a4b0 2023-10-16T12:53:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.761+0000 7f9387fff640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93780047c0 0x7f9378004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f938c945640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9378003970 0x7f9378003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f938d146640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9378005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f9387fff640 1 -- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 msgr2=0x7f9378005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f9387fff640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9378005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f9387fff640 1 -- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9378003970 msgr2=0x7f9378003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f9387fff640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9378003970 0x7f9378003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f9387fff640 1 -- 172.21.15.60:0/1967654767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9378003520 con 0x7f93780047c0 2023-10-16T12:53:43.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.762+0000 7f938d146640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f938806a4b0 0x7f9378005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:43.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.763+0000 7f9387fff640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93780047c0 0x7f9378004c40 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f937400c240 tx=0x7f93780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd85eb2cf0116550 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:43.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.763+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937402caf0 con 0x7f93780047c0 2023-10-16T12:53:43.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.763+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937402cc50 con 0x7f93780047c0 2023-10-16T12:53:43.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.763+0000 7f93877fe640 1 -- 172.21.15.60:0/1967654767 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93740058d0 con 0x7f93780047c0 2023-10-16T12:53:43.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938eba9640 1 -- 172.21.15.60:0/1967654767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93780047c0 msgr2=0x7f9378004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938eba9640 1 --2- 172.21.15.60:0/1967654767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93780047c0 0x7f9378004c40 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f937400c240 tx=0x7f93780057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938d146640 1 -- 172.21.15.60:0/1967654767 reap_dead start 2023-10-16T12:53:43.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938eba9640 1 -- 172.21.15.60:0/1967654767 shutdown_connections 2023-10-16T12:53:43.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938eba9640 1 -- 172.21.15.60:0/1967654767 >> 172.21.15.60:0/1967654767 conn(0x7f93880ce9b0 msgr2=0x7f93880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:43.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938eba9640 1 -- 172.21.15.60:0/1967654767 shutdown_connections 2023-10-16T12:53:43.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.764+0000 7f938eba9640 1 -- 172.21.15.60:0/1967654767 wait complete. 2023-10-16T12:53:43.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.766+0000 7f938eba9640 1 Processor -- start 2023-10-16T12:53:43.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.766+0000 7f938eba9640 1 -- start start 2023-10-16T12:53:43.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.767+0000 7f938eba9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93780047c0 0x7f93881e7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.767+0000 7f938eba9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93880ae240 0x7f93881f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.767+0000 7f938eba9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 0x7f938815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.767+0000 7f938eba9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93881f6330 con 0x7f9388157d40 2023-10-16T12:53:43.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.767+0000 7f938eba9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938815e1e0 con 0x7f93780047c0 2023-10-16T12:53:43.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.767+0000 7f938eba9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938815e4c0 con 0x7f93880ae240 2023-10-16T12:53:43.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 0x7f938815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 0x7f938815a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56166/0 (socket says 172.21.15.60:56166) 2023-10-16T12:53:43.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 -- 172.21.15.60:0/3455630885 learned_addr learned my addr 172.21.15.60:0/3455630885 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:43.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f9387fff640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93880ae240 0x7f93881f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938c945640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93780047c0 0x7f93881e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 -- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93880ae240 msgr2=0x7f93881f5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93880ae240 0x7f93881f5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 -- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93780047c0 msgr2=0x7f93881e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:43.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93780047c0 0x7f93881e7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:43.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 -- 172.21.15.60:0/3455630885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93881f6560 con 0x7f9388157d40 2023-10-16T12:53:43.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938c945640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93780047c0 0x7f93881e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f9387fff640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93880ae240 0x7f93881f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.768+0000 7f938d146640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 0x7f938815a2d0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f93880cf870 tx=0x7f938000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.769+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380017400 con 0x7f9388157d40 2023-10-16T12:53:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.769+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9380011040 con 0x7f9388157d40 2023-10-16T12:53:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.769+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380017780 con 0x7f9388157d40 2023-10-16T12:53:43.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.769+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93881f6840 con 0x7f9388157d40 2023-10-16T12:53:43.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.769+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93881581c0 con 0x7f9388157d40 2023-10-16T12:53:43.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.770+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938804e160 con 0x7f9388157d40 2023-10-16T12:53:43.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.771+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 70) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f93800203b0 con 0x7f9388157d40 2023-10-16T12:53:43.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.773+0000 7f93857fa640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f936c06afe0 0x7f936c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:43.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.773+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f9380089c80 con 0x7f9388157d40 2023-10-16T12:53:43.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.773+0000 7f938c945640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f936c06afe0 0x7f936c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:43.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.774+0000 7f938c945640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f936c06afe0 0x7f936c06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f937c009a00 tx=0x7f937c008040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:43.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.777+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9380054910 con 0x7f9388157d40 2023-10-16T12:53:43.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:43.993+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f93881583b0 con 0x7f936c06afe0 2023-10-16T12:53:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.174+0000 7f93857fa640 1 -- 172.21.15.60:0/3455630885 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f937c009670 con 0x7f936c06afe0 2023-10-16T12:53:44.178 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1/daily 2023-10-16T12:53:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f936c06afe0 msgr2=0x7f936c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f936c06afe0 0x7f936c06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f937c009a00 tx=0x7f937c008040 comp rx=0 tx=0).stop 2023-10-16T12:53:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 msgr2=0x7f938815a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 0x7f938815a2d0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f93880cf870 tx=0x7f938000f4a0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 shutdown_connections 2023-10-16T12:53:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f936c06afe0 0x7f936c06d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93880ae240 0x7f93881f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9388157d40 0x7f938815a2d0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 --2- 172.21.15.60:0/3455630885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93780047c0 0x7f93881e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 >> 172.21.15.60:0/3455630885 conn(0x7f93880ce9b0 msgr2=0x7f938815aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:44.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 shutdown_connections 2023-10-16T12:53:44.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.178+0000 7f938eba9640 1 -- 172.21.15.60:0/3455630885 wait complete. 2023-10-16T12:53:44.201 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-10-16T12:53:44.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:44.202 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-10-16T12:53:44.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-10-16T12:53:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.502+0000 7ff1e1c67640 1 Processor -- start 2023-10-16T12:53:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.502+0000 7ff1e1c67640 1 -- start start 2023-10-16T12:53:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.504+0000 7ff1e1c67640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1dc0ad970 0x7ff1dc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.504+0000 7ff1e1c67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.504+0000 7ff1e1c67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc15b9d0 con 0x7ff1dc06c040 2023-10-16T12:53:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.504+0000 7ff1e1c67640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc15bcb0 con 0x7ff1dc0ad970 2023-10-16T12:53:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.504+0000 7ff1e1c67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc15bf90 con 0x7ff1dc0ccfb0 2023-10-16T12:53:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.504+0000 7ff1dbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1dc0ad970 0x7ff1dc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1dbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1dc0ad970 0x7ff1dc0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56168/0 (socket says 172.21.15.60:56168) 2023-10-16T12:53:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1534438230 learned_addr learned my addr 172.21.15.60:0/1534438230 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1534438230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc0ccfb0 msgr2=0x7ff1dc0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1e1466640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1dbfff640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1534438230 >> v1:172.21.15.144:6789/0 conn(0x7ff1dc06c040 legacy=0x7ff1dc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1534438230 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc0cc270 con 0x7ff1dc0ad970 2023-10-16T12:53:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.505+0000 7ff1e1466640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:44.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.506+0000 7ff1dbfff640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0cca70 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009870 tx=0x7ff1c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63247b20d38ee6d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:44.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.507+0000 7ff1db7fe640 1 -- 172.21.15.60:0/1534438230 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8031640 con 0x7ff1dc0ad970 2023-10-16T12:53:44.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.507+0000 7ff1db7fe640 1 -- 172.21.15.60:0/1534438230 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c80317a0 con 0x7ff1dc0ad970 2023-10-16T12:53:44.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.507+0000 7ff1db7fe640 1 -- 172.21.15.60:0/1534438230 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8031b30 con 0x7ff1dc0ad970 2023-10-16T12:53:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1534438230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 msgr2=0x7ff1dc0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0cca70 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009870 tx=0x7ff1c802b8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1534438230 shutdown_connections 2023-10-16T12:53:44.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1534438230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0cca70 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1534438230 >> 172.21.15.60:0/1534438230 conn(0x7ff1dc0ce6a0 msgr2=0x7ff1dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:44.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1534438230 shutdown_connections 2023-10-16T12:53:44.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.508+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1534438230 wait complete. 2023-10-16T12:53:44.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.510+0000 7ff1e1c67640 1 Processor -- start 2023-10-16T12:53:44.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.510+0000 7ff1e1c67640 1 -- start start 2023-10-16T12:53:44.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1e1c67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc06c040 0x7ff1dc0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1e1c67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1e1c67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1e1c67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0c6960 con 0x7ff1dc0ad970 2023-10-16T12:53:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1e1c67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc1571a0 con 0x7ff1dc0ccfb0 2023-10-16T12:53:44.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1e1c67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc0c5d50 con 0x7ff1dc06c040 2023-10-16T12:53:44.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:44.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56174/0 (socket says 172.21.15.60:56174) 2023-10-16T12:53:44.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.511+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1934261151 learned_addr learned my addr 172.21.15.60:0/1934261151 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:44.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1e1466640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:44.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1e0c65640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc06c040 0x7ff1dc0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:44.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc06c040 msgr2=0x7ff1dc0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1dbfff640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc06c040 0x7ff1dc0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1dc0ccfb0 msgr2=0x7ff1dc0c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1dbfff640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1dbfff640 1 -- 172.21.15.60:0/1934261151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c80094d0 con 0x7ff1dc0ad970 2023-10-16T12:53:44.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1e1466640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:44.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1dbfff640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0c8250 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097c0 tx=0x7ff1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:44.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c803ad20 con 0x7ff1dc0ad970 2023-10-16T12:53:44.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c803ae80 con 0x7ff1dc0ad970 2023-10-16T12:53:44.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8044630 con 0x7ff1dc0ad970 2023-10-16T12:53:44.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.512+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc0c6030 con 0x7ff1dc0ad970 2023-10-16T12:53:44.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.513+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc0c65f0 con 0x7ff1dc0ad970 2023-10-16T12:53:44.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.514+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc06b330 con 0x7ff1dc0ad970 2023-10-16T12:53:44.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.515+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 70) v1 ==== 89349+0+0 (secure 0 0 0) 0x7ff1c802f050 con 0x7ff1dc0ad970 2023-10-16T12:53:44.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.516+0000 7ff1d97fa640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ff1c006afe0 0x7ff1c006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:44.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.516+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff1c80abb30 con 0x7ff1dc0ad970 2023-10-16T12:53:44.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.516+0000 7ff1e0c65640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ff1c006afe0 0x7ff1c006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:44.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.517+0000 7ff1e0c65640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ff1c006afe0 0x7ff1c006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0b35b0 tx=0x7ff1d0009040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:44.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.520+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1c8076860 con 0x7ff1dc0ad970 2023-10-16T12:53:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.743+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1dc0cc550 con 0x7ff1c006afe0 2023-10-16T12:53:44.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.924+0000 7ff1d97fa640 1 -- 172.21.15.60:0/1934261151 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7ff1d0003980 con 0x7ff1c006afe0 2023-10-16T12:53:44.927 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1/weekly 2023-10-16T12:53:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ff1c006afe0 msgr2=0x7ff1c006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ff1c006afe0 0x7ff1c006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0b35b0 tx=0x7ff1d0009040 comp rx=0 tx=0).stop 2023-10-16T12:53:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 msgr2=0x7ff1dc0c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0c8250 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097c0 tx=0x7ff1c80046c0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 shutdown_connections 2023-10-16T12:53:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ff1c006afe0 0x7ff1c006d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1dc0ccfb0 0x7ff1dc0c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1dc0ad970 0x7ff1dc0c8250 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 --2- 172.21.15.60:0/1934261151 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1dc06c040 0x7ff1dc0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 >> 172.21.15.60:0/1934261151 conn(0x7ff1dc0ce6a0 msgr2=0x7ff1dc15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 shutdown_connections 2023-10-16T12:53:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:44.928+0000 7ff1e1c67640 1 -- 172.21.15.60:0/1934261151 wait complete. 2023-10-16T12:53:44.955 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-10-16T12:53:44.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:44.957 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-10-16T12:53:45.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-10-16T12:53:45.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.249+0000 7f973e72e640 1 Processor -- start 2023-10-16T12:53:45.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.250+0000 7f973e72e640 1 -- start start 2023-10-16T12:53:45.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.251+0000 7f973e72e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:45.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.251+0000 7f973e72e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f97380b43c0 0x7f97380cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:45.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.251+0000 7f973e72e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973806be30 con 0x7f97380b0ea0 2023-10-16T12:53:45.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.251+0000 7f973e72e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973806ca00 con 0x7f97380b43c0 2023-10-16T12:53:45.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.251+0000 7f973e72e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97380ada10 con 0x7f97380b3ec0 2023-10-16T12:53:45.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973cf2b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f97380b0ea0 0x7f973815a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35052/0 (socket says 172.21.15.60:35052) 2023-10-16T12:53:45.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973cf2b640 1 -- 172.21.15.60:0/497538068 learned_addr learned my addr 172.21.15.60:0/497538068 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:45.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973df2d640 1 --2- 172.21.15.60:0/497538068 >> v2:172.21.15.60:3300/0 conn(0x7f97380b43c0 0x7f97380cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973d72c640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f972bfff640 1 -- 172.21.15.60:0/497538068 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516292640 0 0) 0x7f973806be30 con 0x7f97380b0ea0 2023-10-16T12:53:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973df2d640 1 -- 172.21.15.60:0/497538068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 msgr2=0x7f97380cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973df2d640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973df2d640 1 -- 172.21.15.60:0/497538068 >> v1:172.21.15.60:6789/0 conn(0x7f97380b0ea0 legacy=0x7f973815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973df2d640 1 -- 172.21.15.60:0/497538068 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973806c060 con 0x7f97380b43c0 2023-10-16T12:53:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.252+0000 7f973d72c640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.253+0000 7f973df2d640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380cd7a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f9730009d90 tx=0x7f973002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184e1866fd101d5b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f972bfff640 1 -- 172.21.15.60:0/497538068 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973002cbc0 con 0x7f97380b43c0 2023-10-16T12:53:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f972bfff640 1 -- 172.21.15.60:0/497538068 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973002cd20 con 0x7f97380b43c0 2023-10-16T12:53:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f972bfff640 1 -- 172.21.15.60:0/497538068 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730034660 con 0x7f97380b43c0 2023-10-16T12:53:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f973e72e640 1 -- 172.21.15.60:0/497538068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 msgr2=0x7f97380cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f973e72e640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380cd7a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f9730009d90 tx=0x7f973002be10 comp rx=0 tx=0).stop 2023-10-16T12:53:45.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f973e72e640 1 -- 172.21.15.60:0/497538068 shutdown_connections 2023-10-16T12:53:45.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f973e72e640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380cd7a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f973e72e640 1 --2- 172.21.15.60:0/497538068 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.254+0000 7f973e72e640 1 -- 172.21.15.60:0/497538068 >> 172.21.15.60:0/497538068 conn(0x7f97380b0170 msgr2=0x7f97380b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:45.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.255+0000 7f973e72e640 1 -- 172.21.15.60:0/497538068 shutdown_connections 2023-10-16T12:53:45.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.255+0000 7f973e72e640 1 -- 172.21.15.60:0/497538068 wait complete. 2023-10-16T12:53:45.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.256+0000 7f973e72e640 1 Processor -- start 2023-10-16T12:53:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 -- start start 2023-10-16T12:53:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97380b0ea0 0x7f97381e3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380c3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97380c4bf0 con 0x7f97380b43c0 2023-10-16T12:53:45.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973815d060 con 0x7f97380b0ea0 2023-10-16T12:53:45.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.257+0000 7f973e72e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97381496e0 con 0x7f97380b3ec0 2023-10-16T12:53:45.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:45.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380c3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56198/0 (socket says 172.21.15.60:56198) 2023-10-16T12:53:45.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 -- 172.21.15.60:0/1330110859 learned_addr learned my addr 172.21.15.60:0/1330110859 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:45.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 -- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 msgr2=0x7f97380c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:45.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 -- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97380b0ea0 msgr2=0x7f97381e3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:45.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97380b0ea0 0x7f97381e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.258+0000 7f973df2d640 1 -- 172.21.15.60:0/1330110859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9730009a70 con 0x7f97380b43c0 2023-10-16T12:53:45.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.259+0000 7f973df2d640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380c3200 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f973002c420 tx=0x7f9730004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:45.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.259+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730040650 con 0x7f97380b43c0 2023-10-16T12:53:45.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.259+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973002ce90 con 0x7f97380b43c0 2023-10-16T12:53:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.259+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97300409f0 con 0x7f97380b43c0 2023-10-16T12:53:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.259+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9738149ca0 con 0x7f97380b43c0 2023-10-16T12:53:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.259+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97380c8680 con 0x7f97380b43c0 2023-10-16T12:53:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.260+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973804e9b0 con 0x7f97380b43c0 2023-10-16T12:53:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.261+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 70) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f9730040b50 con 0x7f97380b43c0 2023-10-16T12:53:45.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.263+0000 7f9729ffb640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f971c06afe0 0x7f971c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.263+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f97300ac790 con 0x7f97380b43c0 2023-10-16T12:53:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.264+0000 7f973d72c640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f971c06afe0 0x7f971c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.265+0000 7f973d72c640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f971c06afe0 0x7f971c06d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9734006fd0 tx=0x7f9734008040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.268+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9730077530 con 0x7f97380b43c0 2023-10-16T12:53:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.490+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f973806c620 con 0x7f971c06afe0 2023-10-16T12:53:45.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.675+0000 7f9729ffb640 1 -- 172.21.15.60:0/1330110859 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f973400ae00 con 0x7f971c06afe0 2023-10-16T12:53:45.677 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1/monthly 2023-10-16T12:53:45.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.678+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f971c06afe0 msgr2=0x7f971c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:45.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.678+0000 7f973e72e640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f971c06afe0 0x7f971c06d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9734006fd0 tx=0x7f9734008040 comp rx=0 tx=0).stop 2023-10-16T12:53:45.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.678+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 msgr2=0x7f97380c3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:45.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.678+0000 7f973e72e640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380c3200 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f973002c420 tx=0x7f9730004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:45.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 shutdown_connections 2023-10-16T12:53:45.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f971c06afe0 0x7f971c06d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f97380b3ec0 0x7f97380c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97380b43c0 0x7f97380c3200 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 --2- 172.21.15.60:0/1330110859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97380b0ea0 0x7f97381e3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 >> 172.21.15.60:0/1330110859 conn(0x7f97380b0170 msgr2=0x7f97380aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 shutdown_connections 2023-10-16T12:53:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:45.679+0000 7f973e72e640 1 -- 172.21.15.60:0/1330110859 wait complete. 2023-10-16T12:53:45.704 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-10-16T12:53:45.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:45.705 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-10-16T12:53:45.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-10-16T12:53:46.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.005+0000 7feffc5fc640 1 Processor -- start 2023-10-16T12:53:46.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.006+0000 7feffc5fc640 1 -- start start 2023-10-16T12:53:46.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.007+0000 7feffc5fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feff415ad70 0x7feff4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.007+0000 7feffc5fc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff40cb030 con 0x7feff40c9ca0 2023-10-16T12:53:46.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.007+0000 7feffc5fc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff40c91c0 con 0x7feff40caac0 2023-10-16T12:53:46.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.007+0000 7feffc5fc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff40c94a0 con 0x7feff415ad70 2023-10-16T12:53:46.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.007+0000 7feff9b97640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feff40c9ca0 0x7feff40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35064/0 (socket says 172.21.15.60:35064) 2023-10-16T12:53:46.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.007+0000 7feff9b97640 1 -- 172.21.15.60:0/1091816825 learned_addr learned my addr 172.21.15.60:0/1091816825 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feffa398640 1 --2- 172.21.15.60:0/1091816825 >> v2:172.21.15.144:3300/0 conn(0x7feff415ad70 0x7feff4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263946582 0 0) 0x7feff40cb030 con 0x7feff40c9ca0 2023-10-16T12:53:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe4003520 con 0x7feff40c9ca0 2023-10-16T12:53:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736488635 0 0) 0x7fefe4003520 con 0x7feff40c9ca0 2023-10-16T12:53:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 >> v2:172.21.15.144:3300/0 conn(0x7feff415ad70 msgr2=0x7feff4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 --2- 172.21.15.60:0/1091816825 >> v2:172.21.15.144:3300/0 conn(0x7feff415ad70 0x7feff4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 >> v1:172.21.15.144:6789/0 conn(0x7feff40caac0 legacy=0x7feff40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff40c9780 con 0x7feff40c9ca0 2023-10-16T12:53:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fefe0001670 con 0x7feff40c9ca0 2023-10-16T12:53:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 >> v1:172.21.15.60:6789/0 conn(0x7feff40c9ca0 legacy=0x7feff40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.008+0000 7feff9396640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe4003970 0x7fefe4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feff9396640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe40047c0 0x7fefe4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feff9396640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7fefe400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40c94a0 con 0x7fefe4005180 2023-10-16T12:53:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe400c000 con 0x7fefe40047c0 2023-10-16T12:53:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe400c1c0 con 0x7fefe4003970 2023-10-16T12:53:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffab99640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7fefe400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffa398640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe4003970 0x7fefe4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feff9b97640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe40047c0 0x7fefe4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffab99640 1 -- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe4003970 msgr2=0x7fefe4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffab99640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe4003970 0x7fefe4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffab99640 1 -- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe40047c0 msgr2=0x7fefe4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffab99640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe40047c0 0x7fefe4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.009+0000 7feffab99640 1 -- 172.21.15.60:0/1091816825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4003520 con 0x7fefe4005180 2023-10-16T12:53:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.010+0000 7feffab99640 1 -- 172.21.15.60:0/1091816825 reap_dead start 2023-10-16T12:53:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.011+0000 7feffab99640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7fefe400b980 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fefec003db0 tx=0x7fefe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0eb0312c6c886 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.011+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec007220 con 0x7fefe4005180 2023-10-16T12:53:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.011+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec02e050 con 0x7fefe4005180 2023-10-16T12:53:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.011+0000 7feff9396640 1 -- 172.21.15.60:0/1091816825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0075e0 con 0x7fefe4005180 2023-10-16T12:53:46.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 -- 172.21.15.60:0/1091816825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 msgr2=0x7fefe400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7fefe400b980 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fefec003db0 tx=0x7fefe40057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 -- 172.21.15.60:0/1091816825 shutdown_connections 2023-10-16T12:53:46.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 --2- 172.21.15.60:0/1091816825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7fefe400b980 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 -- 172.21.15.60:0/1091816825 >> 172.21.15.60:0/1091816825 conn(0x7feff40ce9b0 msgr2=0x7feff40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:46.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 -- 172.21.15.60:0/1091816825 shutdown_connections 2023-10-16T12:53:46.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.012+0000 7feffc5fc640 1 -- 172.21.15.60:0/1091816825 wait complete. 2023-10-16T12:53:46.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.014+0000 7feffc5fc640 1 Processor -- start 2023-10-16T12:53:46.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.014+0000 7feffc5fc640 1 -- start start 2023-10-16T12:53:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffc5fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7feff40cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffc5fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feff415d8d0 0x7feff41591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffc5fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feff41f2630 0x7feff41f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffc5fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40ccb80 con 0x7fefe4005180 2023-10-16T12:53:46.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffc5fc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40cce60 con 0x7feff415d8d0 2023-10-16T12:53:46.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffc5fc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff40cd140 con 0x7feff41f2630 2023-10-16T12:53:46.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7feff40cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7feff40cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56224/0 (socket says 172.21.15.60:56224) 2023-10-16T12:53:46.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 -- 172.21.15.60:0/2692104182 learned_addr learned my addr 172.21.15.60:0/2692104182 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:46.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feff9b97640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feff415d8d0 0x7feff41591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffab99640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feff41f2630 0x7feff41f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 -- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feff41f2630 msgr2=0x7feff41f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feff41f2630 0x7feff41f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 -- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feff415d8d0 msgr2=0x7feff41591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feff415d8d0 0x7feff41591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feffa398640 1 -- 172.21.15.60:0/2692104182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff40cd420 con 0x7fefe4005180 2023-10-16T12:53:46.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.015+0000 7feff9b97640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feff415d8d0 0x7feff41591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:46.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.016+0000 7feffa398640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7feff40cc580 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009fb0 tx=0x7fefe80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:46.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.016+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe800e030 con 0x7fefe4005180 2023-10-16T12:53:46.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.016+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe8016470 con 0x7fefe4005180 2023-10-16T12:53:46.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.016+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe8016820 con 0x7fefe4005180 2023-10-16T12:53:46.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.016+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff40cd700 con 0x7fefe4005180 2023-10-16T12:53:46.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.016+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff415dd10 con 0x7fefe4005180 2023-10-16T12:53:46.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.018+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff404e160 con 0x7fefe4005180 2023-10-16T12:53:46.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.018+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fefe8016980 con 0x7fefe4005180 2023-10-16T12:53:46.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.019+0000 7feff2ffd640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fefd806b070 0x7fefd806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.020+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fefe800f070 con 0x7fefe4005180 2023-10-16T12:53:46.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.020+0000 7feff9b97640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fefd806b070 0x7fefd806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.021+0000 7feff9b97640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fefd806b070 0x7fefd806d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feff40c91e0 tx=0x7fefe0000f10 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:46.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.024+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefe8055750 con 0x7fefe4005180 2023-10-16T12:53:46.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.239+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7feff40ccfd0 con 0x7fefd806b070 2023-10-16T12:53:46.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.420+0000 7feff2ffd640 1 -- 172.21.15.60:0/2692104182 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7feff40ccfd0 con 0x7fefd806b070 2023-10-16T12:53:46.423 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1/yearly 2023-10-16T12:53:46.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.423+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fefd806b070 msgr2=0x7fefd806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.423+0000 7feffc5fc640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fefd806b070 0x7fefd806d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feff40c91e0 tx=0x7fefe0000f10 comp rx=0 tx=0).stop 2023-10-16T12:53:46.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.423+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 msgr2=0x7feff40cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.423+0000 7feffc5fc640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7feff40cc580 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009fb0 tx=0x7fefe80115a0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 shutdown_connections 2023-10-16T12:53:46.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fefd806b070 0x7fefd806d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feff41f2630 0x7feff41f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feff415d8d0 0x7feff41591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 --2- 172.21.15.60:0/2692104182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe4005180 0x7feff40cc580 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 >> 172.21.15.60:0/2692104182 conn(0x7feff40ce9b0 msgr2=0x7feff40c5080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:46.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 shutdown_connections 2023-10-16T12:53:46.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.424+0000 7feffc5fc640 1 -- 172.21.15.60:0/2692104182 wait complete. 2023-10-16T12:53:46.448 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-10-16T12:53:46.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:46.449 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-10-16T12:53:46.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-10-16T12:53:46.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.745+0000 7f8548e89640 1 Processor -- start 2023-10-16T12:53:46.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.745+0000 7f8548e89640 1 -- start start 2023-10-16T12:53:46.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.747+0000 7f8548e89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 0x7f8544065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.747+0000 7f8548e89640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 0x7f85440b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.747+0000 7f8548e89640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85440cd790 con 0x7f85440b4f80 2023-10-16T12:53:46.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.747+0000 7f8548e89640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85440ce310 con 0x7f8544065ff0 2023-10-16T12:53:46.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.747+0000 7f8548e89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8544066430 con 0x7f85440b20f0 2023-10-16T12:53:46.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.748+0000 7f8541d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 0x7f85440b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.748+0000 7f854259c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 0x7f8544065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.748+0000 7f8541d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 0x7f85440b4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:34360/0 (socket says 172.21.15.60:34360) 2023-10-16T12:53:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.748+0000 7f8541d9b640 1 -- 172.21.15.60:0/3618979367 learned_addr learned my addr 172.21.15.60:0/3618979367 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.748+0000 7f854259c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 0x7f8544065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:39486/0 (socket says 172.21.15.60:39486) 2023-10-16T12:53:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.748+0000 7f854159a640 1 -- 172.21.15.60:0/3618979367 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302256752 0 0) 0x7f85440cd790 con 0x7f85440b4f80 2023-10-16T12:53:46.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.749+0000 7f854159a640 1 -- 172.21.15.60:0/3618979367 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8530003520 con 0x7f85440b4f80 2023-10-16T12:53:46.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.749+0000 7f854259c640 1 -- 172.21.15.60:0/3618979367 >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 msgr2=0x7f85440b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:46.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.749+0000 7f854259c640 1 --2- 172.21.15.60:0/3618979367 >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 0x7f85440b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.749+0000 7f854259c640 1 -- 172.21.15.60:0/3618979367 >> v1:172.21.15.144:6789/0 conn(0x7f85440b4f80 legacy=0x7f85440d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.749+0000 7f854259c640 1 -- 172.21.15.60:0/3618979367 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85440cd9e0 con 0x7f85440b20f0 2023-10-16T12:53:46.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.749+0000 7f8541d9b640 1 --2- 172.21.15.60:0/3618979367 >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 0x7f85440b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:53:46.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.750+0000 7f854259c640 1 --2- 172.21.15.60:0/3618979367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 0x7f8544065ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8534009d90 tx=0x7f853402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d88a20a346af8f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:46.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.751+0000 7f854159a640 1 -- 172.21.15.60:0/3618979367 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853402cbd0 con 0x7f85440b20f0 2023-10-16T12:53:46.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.751+0000 7f854159a640 1 -- 172.21.15.60:0/3618979367 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853402cd30 con 0x7f85440b20f0 2023-10-16T12:53:46.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.751+0000 7f854159a640 1 -- 172.21.15.60:0/3618979367 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853402cea0 con 0x7f85440b20f0 2023-10-16T12:53:46.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.751+0000 7f8548e89640 1 -- 172.21.15.60:0/3618979367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 msgr2=0x7f8544065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.751+0000 7f8548e89640 1 --2- 172.21.15.60:0/3618979367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 0x7f8544065ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8534009d90 tx=0x7f853402be20 comp rx=0 tx=0).stop 2023-10-16T12:53:46.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.752+0000 7f8548e89640 1 -- 172.21.15.60:0/3618979367 shutdown_connections 2023-10-16T12:53:46.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.752+0000 7f8548e89640 1 --2- 172.21.15.60:0/3618979367 >> v2:172.21.15.144:3300/0 conn(0x7f8544065ff0 0x7f85440b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.752+0000 7f8548e89640 1 --2- 172.21.15.60:0/3618979367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b20f0 0x7f8544065ab0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.752+0000 7f8548e89640 1 -- 172.21.15.60:0/3618979367 >> 172.21.15.60:0/3618979367 conn(0x7f85440d2d10 msgr2=0x7f85440d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:46.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.752+0000 7f8548e89640 1 -- 172.21.15.60:0/3618979367 shutdown_connections 2023-10-16T12:53:46.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.752+0000 7f8548e89640 1 -- 172.21.15.60:0/3618979367 wait complete. 2023-10-16T12:53:46.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.754+0000 7f8548e89640 1 Processor -- start 2023-10-16T12:53:46.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.754+0000 7f8548e89640 1 -- start start 2023-10-16T12:53:46.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8548e89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8544065ff0 0x7f85440aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8548e89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85440b20f0 0x7f8544150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8548e89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b4f80 0x7f854406a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8548e89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440ccc10 con 0x7f8544065ff0 2023-10-16T12:53:46.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8548e89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440cbb70 con 0x7f85440b20f0 2023-10-16T12:53:46.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8548e89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440cbe50 con 0x7f85440b4f80 2023-10-16T12:53:46.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8544065ff0 0x7f85440aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8544065ff0 0x7f85440aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56232/0 (socket says 172.21.15.60:56232) 2023-10-16T12:53:46.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 -- 172.21.15.60:0/708616234 learned_addr learned my addr 172.21.15.60:0/708616234 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:46.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f8542d9d640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b4f80 0x7f854406a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 -- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b4f80 msgr2=0x7f854406a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f85440b4f80 0x7f854406a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 -- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85440b20f0 msgr2=0x7f8544150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:46.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85440b20f0 0x7f8544150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:46.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.755+0000 7f854259c640 1 -- 172.21.15.60:0/708616234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0094d0 con 0x7f8544065ff0 2023-10-16T12:53:46.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.756+0000 7f854259c640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8544065ff0 0x7f85440aa0b0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8534000f80 tx=0x7f8534004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:46.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.756+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853402cea0 con 0x7f8544065ff0 2023-10-16T12:53:46.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.756+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534004880 con 0x7f8544065ff0 2023-10-16T12:53:46.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.756+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534034640 con 0x7f8544065ff0 2023-10-16T12:53:46.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.756+0000 7f8548e89640 1 -- 172.21.15.60:0/708616234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8534009a70 con 0x7f8544065ff0 2023-10-16T12:53:46.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.756+0000 7f8548e89640 1 -- 172.21.15.60:0/708616234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85440cc410 con 0x7f8544065ff0 2023-10-16T12:53:46.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.758+0000 7f8548e89640 1 -- 172.21.15.60:0/708616234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854404e160 con 0x7f8544065ff0 2023-10-16T12:53:46.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.758+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f853403b030 con 0x7f8544065ff0 2023-10-16T12:53:46.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.759+0000 7f853affd640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f852406b0c0 0x7f852406d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:46.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.759+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f85340abe70 con 0x7f8544065ff0 2023-10-16T12:53:46.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.760+0000 7f8541d9b640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f852406b0c0 0x7f852406d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:46.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.761+0000 7f8541d9b640 1 --2- 172.21.15.60:0/708616234 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f852406b0c0 0x7f852406d580 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f852c006f40 tx=0x7f852c006e90 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:46.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.764+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8534076bb0 con 0x7f8544065ff0 2023-10-16T12:53:46.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.984+0000 7f8548e89640 1 -- 172.21.15.60:0/708616234 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f85440cdfa0 con 0x7f852406b0c0 2023-10-16T12:53:46.990 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:46.988+0000 7fa346c17640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T12:53:46.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:46.988+0000 7f853affd640 1 -- 172.21.15.60:0/708616234 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f852c004020 con 0x7f852406b0c0 2023-10-16T12:53:46.991 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T12:53:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:53:47.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-10-16T12:53:47.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.220+0000 7f193759e640 1 Processor -- start 2023-10-16T12:53:47.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.220+0000 7f193759e640 1 -- start start 2023-10-16T12:53:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.221+0000 7f193759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1938159950 0x7f193815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.221+0000 7f193759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1938070450 0x7f19380708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.221+0000 7f193759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f193806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.221+0000 7f193759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1938070e20 con 0x7f1938159950 2023-10-16T12:53:47.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193815e490 con 0x7f1938070450 2023-10-16T12:53:47.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193815aef0 con 0x7f193806c100 2023-10-16T12:53:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1938159950 0x7f193815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1938159950 0x7f193815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56244/0 (socket says 172.21.15.60:56244) 2023-10-16T12:53:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193659c640 1 -- 172.21.15.60:0/754093455 learned_addr learned my addr 172.21.15.60:0/754093455 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f192ffff640 1 --2- 172.21.15.60:0/754093455 >> v2:172.21.15.144:3300/0 conn(0x7f1938070450 0x7f19380708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f1936d9d640 1 --2- 172.21.15.60:0/754093455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f193806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193659c640 1 -- 172.21.15.60:0/754093455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 msgr2=0x7f193806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193659c640 1 --2- 172.21.15.60:0/754093455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f193806c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.222+0000 7f193659c640 1 -- 172.21.15.60:0/754093455 >> v2:172.21.15.144:3300/0 conn(0x7f1938070450 msgr2=0x7f19380708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.223+0000 7f193659c640 1 --2- 172.21.15.60:0/754093455 >> v2:172.21.15.144:3300/0 conn(0x7f1938070450 0x7f19380708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.223+0000 7f193659c640 1 -- 172.21.15.60:0/754093455 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193815d910 con 0x7f1938159950 2023-10-16T12:53:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.224+0000 7f193659c640 1 --2- 172.21.15.60:0/754093455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f193815a9b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f19380cab70 tx=0x7f192802da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bc86571f410a9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:47.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.224+0000 7f1935d9b640 1 -- 172.21.15.60:0/754093455 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192802f830 con 0x7f1938159950 2023-10-16T12:53:47.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.224+0000 7f1935d9b640 1 -- 172.21.15.60:0/754093455 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192802f990 con 0x7f1938159950 2023-10-16T12:53:47.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.224+0000 7f1935d9b640 1 -- 172.21.15.60:0/754093455 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192802fd20 con 0x7f1938159950 2023-10-16T12:53:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 -- 172.21.15.60:0/754093455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 msgr2=0x7f193815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 --2- 172.21.15.60:0/754093455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f193815a9b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f19380cab70 tx=0x7f192802da70 comp rx=0 tx=0).stop 2023-10-16T12:53:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 -- 172.21.15.60:0/754093455 shutdown_connections 2023-10-16T12:53:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 --2- 172.21.15.60:0/754093455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f193806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 --2- 172.21.15.60:0/754093455 >> v2:172.21.15.144:3300/0 conn(0x7f1938070450 0x7f19380708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 --2- 172.21.15.60:0/754093455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f193815a9b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 -- 172.21.15.60:0/754093455 >> 172.21.15.60:0/754093455 conn(0x7f19380ce6a0 msgr2=0x7f19380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 -- 172.21.15.60:0/754093455 shutdown_connections 2023-10-16T12:53:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.225+0000 7f193759e640 1 -- 172.21.15.60:0/754093455 wait complete. 2023-10-16T12:53:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.227+0000 7f193759e640 1 Processor -- start 2023-10-16T12:53:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.227+0000 7f193759e640 1 -- start start 2023-10-16T12:53:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.228+0000 7f193759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f1938155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f193759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1938070450 0x7f19380b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f193759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f19380b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f193759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b1bd0 con 0x7f1938159950 2023-10-16T12:53:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f193759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b1eb0 con 0x7f1938070450 2023-10-16T12:53:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f193759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b2190 con 0x7f193806c100 2023-10-16T12:53:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f19380b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f19380b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56246/0 (socket says 172.21.15.60:56246) 2023-10-16T12:53:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 -- 172.21.15.60:0/542055327 learned_addr learned my addr 172.21.15.60:0/542055327 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f193659c640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f1938155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f192ffff640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1938070450 0x7f19380b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 -- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 msgr2=0x7f1938155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f193806c100 0x7f1938155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 -- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1938070450 msgr2=0x7f19380b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1938070450 0x7f19380b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f1936d9d640 1 -- 172.21.15.60:0/542055327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280094d0 con 0x7f1938159950 2023-10-16T12:53:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.229+0000 7f192ffff640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1938070450 0x7f19380b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.230+0000 7f1936d9d640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1938159950 0x7f19380b2c70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1930007e60 tx=0x7f193000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.230+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930014420 con 0x7f1938159950 2023-10-16T12:53:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.230+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19300145a0 con 0x7f1938159950 2023-10-16T12:53:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.230+0000 7f193759e640 1 -- 172.21.15.60:0/542055327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19380b2470 con 0x7f1938159950 2023-10-16T12:53:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.230+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930014950 con 0x7f1938159950 2023-10-16T12:53:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.230+0000 7f193759e640 1 -- 172.21.15.60:0/542055327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19380b1800 con 0x7f1938159950 2023-10-16T12:53:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.231+0000 7f193759e640 1 -- 172.21.15.60:0/542055327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19380b2990 con 0x7f1938159950 2023-10-16T12:53:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.232+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f1930014ab0 con 0x7f1938159950 2023-10-16T12:53:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.233+0000 7f192effd640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f191806b070 0x7f191806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.233+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f193008a550 con 0x7f1938159950 2023-10-16T12:53:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.234+0000 7f193659c640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f191806b070 0x7f191806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.235+0000 7f193659c640 1 --2- 172.21.15.60:0/542055327 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f191806b070 0x7f191806d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1928038000 tx=0x7f192803b040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:47.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.239+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19380b2990 con 0x7f1938159950 2023-10-16T12:53:47.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.455+0000 7f193759e640 1 -- 172.21.15.60:0/542055327 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f19380b19c0 con 0x7f191806b070 2023-10-16T12:53:47.460 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:47.458+0000 7fa346c17640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:47.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.458+0000 7f192effd640 1 -- 172.21.15.60:0/542055327 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f19380b19c0 con 0x7f191806b070 2023-10-16T12:53:47.461 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:47.475 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:53:47.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-10-16T12:53:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.705+0000 7fc7a4cec640 1 Processor -- start 2023-10-16T12:53:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.705+0000 7fc7a4cec640 1 -- start start 2023-10-16T12:53:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.706+0000 7fc7a4cec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a006c040 0x7fc7a006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.706+0000 7fc7a4cec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7a00cda80 0x7fc7a015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc7a4cec640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a006c960 con 0x7fc7a00ad970 2023-10-16T12:53:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc7a4cec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a015b9d0 con 0x7fc7a00cda80 2023-10-16T12:53:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc7a4cec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015bcb0 con 0x7fc7a006c040 2023-10-16T12:53:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc797fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc7a00ad970 0x7fc7a015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35078/0 (socket says 172.21.15.60:35078) 2023-10-16T12:53:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc797fff640 1 -- 172.21.15.60:0/2459104429 learned_addr learned my addr 172.21.15.60:0/2459104429 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc79ffff640 1 --2- 172.21.15.60:0/2459104429 >> v2:172.21.15.60:3300/0 conn(0x7fc7a00cda80 0x7fc7a015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc79f7fe640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a006c040 0x7fc7a006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc79effd640 1 -- 172.21.15.60:0/2459104429 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919106123 0 0) 0x7fc7a006c960 con 0x7fc7a00ad970 2023-10-16T12:53:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.707+0000 7fc79effd640 1 -- 172.21.15.60:0/2459104429 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c003520 con 0x7fc7a00ad970 2023-10-16T12:53:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.708+0000 7fc79ffff640 1 -- 172.21.15.60:0/2459104429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a006c040 msgr2=0x7fc7a006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.708+0000 7fc79ffff640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a006c040 0x7fc7a006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.708+0000 7fc79ffff640 1 -- 172.21.15.60:0/2459104429 >> v1:172.21.15.60:6789/0 conn(0x7fc7a00ad970 legacy=0x7fc7a015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.708+0000 7fc79ffff640 1 -- 172.21.15.60:0/2459104429 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a015bf90 con 0x7fc7a00cda80 2023-10-16T12:53:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.708+0000 7fc79f7fe640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a006c040 0x7fc7a006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.709+0000 7fc79ffff640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a00cda80 0x7fc7a015ed90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc798009d90 tx=0x7fc79802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2f4046e78f5753 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.709+0000 7fc79effd640 1 -- 172.21.15.60:0/2459104429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79802cc60 con 0x7fc7a00cda80 2023-10-16T12:53:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.709+0000 7fc79effd640 1 -- 172.21.15.60:0/2459104429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79802cdc0 con 0x7fc7a00cda80 2023-10-16T12:53:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.709+0000 7fc79effd640 1 -- 172.21.15.60:0/2459104429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798034640 con 0x7fc7a00cda80 2023-10-16T12:53:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2459104429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a00cda80 msgr2=0x7fc7a015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a00cda80 0x7fc7a015ed90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc798009d90 tx=0x7fc79802beb0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2459104429 shutdown_connections 2023-10-16T12:53:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a00cda80 0x7fc7a015ed90 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 --2- 172.21.15.60:0/2459104429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a006c040 0x7fc7a006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2459104429 >> 172.21.15.60:0/2459104429 conn(0x7fc7a00ce6a0 msgr2=0x7fc7a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2459104429 shutdown_connections 2023-10-16T12:53:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.710+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2459104429 wait complete. 2023-10-16T12:53:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.713+0000 7fc7a4cec640 1 Processor -- start 2023-10-16T12:53:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.713+0000 7fc7a4cec640 1 -- start start 2023-10-16T12:53:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc7a4cec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a006c040 0x7fc7a014fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc7a4cec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a00ad970 0x7fc7a01566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc7a4cec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a00cda80 0x7fc7a00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc7a4cec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a00c99c0 con 0x7fc7a006c040 2023-10-16T12:53:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc7a4cec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a00c9ca0 con 0x7fc7a00cda80 2023-10-16T12:53:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc7a4cec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a00c9f80 con 0x7fc7a00ad970 2023-10-16T12:53:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.714+0000 7fc79f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a006c040 0x7fc7a014fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a006c040 0x7fc7a014fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56256/0 (socket says 172.21.15.60:56256) 2023-10-16T12:53:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 -- 172.21.15.60:0/2258505078 learned_addr learned my addr 172.21.15.60:0/2258505078 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc797fff640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a00ad970 0x7fc7a01566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79ffff640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a00cda80 0x7fc7a00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 -- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a00ad970 msgr2=0x7fc7a01566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7a00ad970 0x7fc7a01566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 -- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a00cda80 msgr2=0x7fc7a00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7a00cda80 0x7fc7a00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.715+0000 7fc79f7fe640 1 -- 172.21.15.60:0/2258505078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798009a70 con 0x7fc7a006c040 2023-10-16T12:53:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.716+0000 7fc79f7fe640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7a006c040 0x7fc7a014fcc0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc79000d7d0 tx=0x7fc79000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.716+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790004030 con 0x7fc7a006c040 2023-10-16T12:53:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.716+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc790004190 con 0x7fc7a006c040 2023-10-16T12:53:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.716+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790004520 con 0x7fc7a006c040 2023-10-16T12:53:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.716+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2258505078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a00ca260 con 0x7fc7a006c040 2023-10-16T12:53:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.716+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2258505078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a0150200 con 0x7fc7a006c040 2023-10-16T12:53:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.717+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2258505078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a006b330 con 0x7fc7a006c040 2023-10-16T12:53:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.718+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fc7900096a0 con 0x7fc7a006c040 2023-10-16T12:53:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.719+0000 7fc79cff9640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc78006b070 0x7fc78006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.719+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc790089c10 con 0x7fc7a006c040 2023-10-16T12:53:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.720+0000 7fc797fff640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc78006b070 0x7fc78006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.721+0000 7fc797fff640 1 --2- 172.21.15.60:0/2258505078 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fc78006b070 0x7fc78006d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7a01541d0 tx=0x7fc788009040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.726+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc790054970 con 0x7fc7a006c040 2023-10-16T12:53:47.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.936+0000 7fc7a4cec640 1 -- 172.21.15.60:0/2258505078 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7a00cc550 con 0x7fc78006b070 2023-10-16T12:53:47.942 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:47.940+0000 7fa346c17640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:47.940+0000 7fc79cff9640 1 -- 172.21.15.60:0/2258505078 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc788001d00 con 0x7fc78006b070 2023-10-16T12:53:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:53:47.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-10-16T12:53:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.179+0000 7fcdb2567640 1 Processor -- start 2023-10-16T12:53:48.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.179+0000 7fcdb2567640 1 -- start start 2023-10-16T12:53:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.180+0000 7fcdb2567640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcdac0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.180+0000 7fcdb2567640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcdac0ccfb0 0x7fcdac0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.180+0000 7fcdb2567640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac15b9d0 con 0x7fcdac06c040 2023-10-16T12:53:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.180+0000 7fcdb2567640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac15bcb0 con 0x7fcdac0ccfb0 2023-10-16T12:53:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.180+0000 7fcdb2567640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac15bf90 con 0x7fcdac0ad970 2023-10-16T12:53:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.181+0000 7fcdb1565640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcdac06c040 0x7fcdac06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35082/0 (socket says 172.21.15.60:35082) 2023-10-16T12:53:48.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.181+0000 7fcdb1565640 1 -- 172.21.15.60:0/2195985918 learned_addr learned my addr 172.21.15.60:0/2195985918 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.181+0000 7fcdb1d66640 1 --2- 172.21.15.60:0/2195985918 >> v2:172.21.15.144:3300/0 conn(0x7fcdac0ccfb0 0x7fcdac0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.181+0000 7fcdb0d64640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcdac0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.181+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783064097 0 0) 0x7fcdac15b9d0 con 0x7fcdac06c040 2023-10-16T12:53:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.181+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd98003520 con 0x7fcdac06c040 2023-10-16T12:53:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1918114312 0 0) 0x7fcd98003520 con 0x7fcdac06c040 2023-10-16T12:53:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 msgr2=0x7fcdac0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcdac0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 >> v2:172.21.15.144:3300/0 conn(0x7fcdac0ccfb0 msgr2=0x7fcdac0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 --2- 172.21.15.60:0/2195985918 >> v2:172.21.15.144:3300/0 conn(0x7fcdac0ccfb0 0x7fcdac0cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0cc270 con 0x7fcdac06c040 2023-10-16T12:53:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcda80019f0 con 0x7fcdac06c040 2023-10-16T12:53:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 >> v1:172.21.15.60:6789/0 conn(0x7fcdac06c040 legacy=0x7fcdac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcd98003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd98003ef0 0x7fcd980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd98004d00 0x7fcd98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac15bcb0 con 0x7fcd98004d00 2023-10-16T12:53:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac15bf90 con 0x7fcd98003ef0 2023-10-16T12:53:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9800b980 con 0x7fcdac0ad970 2023-10-16T12:53:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.182+0000 7fcdb1d66640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd98004d00 0x7fcd98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb0d64640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd98003ef0 0x7fcd980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1565640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcd98003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1d66640 1 -- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 msgr2=0x7fcd98003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1d66640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcd98003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1d66640 1 -- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd98003ef0 msgr2=0x7fcd980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1d66640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd98003ef0 0x7fcd980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1d66640 1 -- 172.21.15.60:0/2195985918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98003520 con 0x7fcd98004d00 2023-10-16T12:53:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.183+0000 7fcdb1565640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcdac0ad970 0x7fcd98003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.184+0000 7fcdb1d66640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd98004d00 0x7fcd98005180 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009ac0 tx=0x7fcd980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca51740c29f10f7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.184+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda402c6d0 con 0x7fcd98004d00 2023-10-16T12:53:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.184+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda402c830 con 0x7fcd98004d00 2023-10-16T12:53:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.184+0000 7fcda3fff640 1 -- 172.21.15.60:0/2195985918 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda402cbc0 con 0x7fcd98004d00 2023-10-16T12:53:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb2567640 1 -- 172.21.15.60:0/2195985918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd98004d00 msgr2=0x7fcd98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb2567640 1 --2- 172.21.15.60:0/2195985918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd98004d00 0x7fcd98005180 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009ac0 tx=0x7fcd980057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb1d66640 1 -- 172.21.15.60:0/2195985918 reap_dead start 2023-10-16T12:53:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb2567640 1 -- 172.21.15.60:0/2195985918 shutdown_connections 2023-10-16T12:53:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb2567640 1 -- 172.21.15.60:0/2195985918 >> 172.21.15.60:0/2195985918 conn(0x7fcdac0ce6a0 msgr2=0x7fcdac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb2567640 1 -- 172.21.15.60:0/2195985918 shutdown_connections 2023-10-16T12:53:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.185+0000 7fcdb2567640 1 -- 172.21.15.60:0/2195985918 wait complete. 2023-10-16T12:53:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.187+0000 7fcdb2567640 1 Processor -- start 2023-10-16T12:53:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.187+0000 7fcdb2567640 1 -- start start 2023-10-16T12:53:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb2567640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd98004d00 0x7fcdac1f6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb2567640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdac150af0 0x7fcdac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb2567640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcdac0acdd0 0x7fcdac0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb2567640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0ad250 con 0x7fcdac150af0 2023-10-16T12:53:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb2567640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0c8cf0 con 0x7fcdac0acdd0 2023-10-16T12:53:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb2567640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1f6b90 con 0x7fcd98004d00 2023-10-16T12:53:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.188+0000 7fcdb0d64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdac150af0 0x7fcdac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdac150af0 0x7fcdac0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56284/0 (socket says 172.21.15.60:56284) 2023-10-16T12:53:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 -- 172.21.15.60:0/3280649161 learned_addr learned my addr 172.21.15.60:0/3280649161 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb1565640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd98004d00 0x7fcdac1f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb1d66640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcdac0acdd0 0x7fcdac0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 -- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd98004d00 msgr2=0x7fcdac1f6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd98004d00 0x7fcdac1f6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 -- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcdac0acdd0 msgr2=0x7fcdac0c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcdac0acdd0 0x7fcdac0c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 -- 172.21.15.60:0/3280649161 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0c8f40 con 0x7fcdac150af0 2023-10-16T12:53:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb1565640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcd98004d00 0x7fcdac1f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.189+0000 7fcdb0d64640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdac150af0 0x7fcdac0ac890 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c011570 tx=0x7fcd9c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.190+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c015930 con 0x7fcdac150af0 2023-10-16T12:53:48.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.190+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c015a90 con 0x7fcdac150af0 2023-10-16T12:53:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.190+0000 7fcdb2567640 1 -- 172.21.15.60:0/3280649161 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac0c9220 con 0x7fcdac150af0 2023-10-16T12:53:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.190+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c015e60 con 0x7fcdac150af0 2023-10-16T12:53:48.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.190+0000 7fcdb2567640 1 -- 172.21.15.60:0/3280649161 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac0c97e0 con 0x7fcdac150af0 2023-10-16T12:53:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.191+0000 7fcdb2567640 1 -- 172.21.15.60:0/3280649161 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac06b330 con 0x7fcdac150af0 2023-10-16T12:53:48.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.192+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fcd9c00f460 con 0x7fcdac150af0 2023-10-16T12:53:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.193+0000 7fcda1ffb640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcd9006b0a0 0x7fcd9006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.194+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fcd9c08a270 con 0x7fcdac150af0 2023-10-16T12:53:48.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.194+0000 7fcdb1565640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcd9006b0a0 0x7fcd9006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.195+0000 7fcdb1565640 1 --2- 172.21.15.60:0/3280649161 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcd9006b0a0 0x7fcd9006d560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcda8002230 tx=0x7fcda8009040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:48.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.198+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd9c054f30 con 0x7fcdac150af0 2023-10-16T12:53:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.422+0000 7fcdb2567640 1 -- 172.21.15.60:0/3280649161 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdac158b50 con 0x7fcd9006b0a0 2023-10-16T12:53:48.427 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:48.425+0000 7fa346c17640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.426+0000 7fcda1ffb640 1 -- 172.21.15.60:0/3280649161 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcdac158b50 con 0x7fcd9006b0a0 2023-10-16T12:53:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:53:48.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-10-16T12:53:48.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.676+0000 7f2fcbfff640 1 Processor -- start 2023-10-16T12:53:48.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.676+0000 7f2fcbfff640 1 -- start start 2023-10-16T12:53:48.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fcbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fcc070b60 0x7f2fcc15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fcbfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc15aef0 con 0x7f2fcc159950 2023-10-16T12:53:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fcbfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc15d910 con 0x7f2fcc06c100 2023-10-16T12:53:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fcbfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc15e490 con 0x7f2fcc070b60 2023-10-16T12:53:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fc3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fcc070b60 0x7f2fcc15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fcaffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2fcc159950 0x7f2fcc070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35096/0 (socket says 172.21.15.60:35096) 2023-10-16T12:53:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.678+0000 7f2fcaffd640 1 -- 172.21.15.60:0/659620861 learned_addr learned my addr 172.21.15.60:0/659620861 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.679+0000 7f2fc3fff640 1 -- 172.21.15.60:0/659620861 >> v1:172.21.15.144:6789/0 conn(0x7f2fcc06c100 legacy=0x7f2fcc06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.679+0000 7f2fc3fff640 1 -- 172.21.15.60:0/659620861 >> v1:172.21.15.60:6789/0 conn(0x7f2fcc159950 legacy=0x7f2fcc070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.679+0000 7f2fc3fff640 1 -- 172.21.15.60:0/659620861 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc15dbf0 con 0x7f2fcc070b60 2023-10-16T12:53:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.679+0000 7f2fca7fc640 1 -- 172.21.15.60:0/659620861 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557255160 0 0) 0x7f2fbc001110 con 0x7f2fcc159950 2023-10-16T12:53:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.680+0000 7f2fc3fff640 1 --2- 172.21.15.60:0/659620861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc070b60 0x7f2fcc15a9b0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40097a0 tx=0x7f2fb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc1395b1036813a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.680+0000 7f2fca7fc640 1 -- 172.21.15.60:0/659620861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb402e070 con 0x7f2fcc070b60 2023-10-16T12:53:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.680+0000 7f2fca7fc640 1 -- 172.21.15.60:0/659620861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb40315a0 con 0x7f2fcc070b60 2023-10-16T12:53:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.680+0000 7f2fca7fc640 1 -- 172.21.15.60:0/659620861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4031930 con 0x7f2fcc070b60 2023-10-16T12:53:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 -- 172.21.15.60:0/659620861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc070b60 msgr2=0x7f2fcc15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 --2- 172.21.15.60:0/659620861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc070b60 0x7f2fcc15a9b0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40097a0 tx=0x7f2fb402b840 comp rx=0 tx=0).stop 2023-10-16T12:53:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 -- 172.21.15.60:0/659620861 shutdown_connections 2023-10-16T12:53:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 --2- 172.21.15.60:0/659620861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc070b60 0x7f2fcc15a9b0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 -- 172.21.15.60:0/659620861 >> 172.21.15.60:0/659620861 conn(0x7f2fcc0ce6a0 msgr2=0x7f2fcc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 -- 172.21.15.60:0/659620861 shutdown_connections 2023-10-16T12:53:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.681+0000 7f2fcbfff640 1 -- 172.21.15.60:0/659620861 wait complete. 2023-10-16T12:53:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.683+0000 7f2fcbfff640 1 Processor -- start 2023-10-16T12:53:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.683+0000 7f2fcbfff640 1 -- start start 2023-10-16T12:53:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fcc06c100 0x7f2fcc06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fcc070b60 0x7f2fcc150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc159950 0x7f2fcc1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcbfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc06ca20 con 0x7f2fcc159950 2023-10-16T12:53:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcbfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15bb30 con 0x7f2fcc070b60 2023-10-16T12:53:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcbfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0cebd0 con 0x7f2fcc06c100 2023-10-16T12:53:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc159950 0x7f2fcc1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc159950 0x7f2fcc1551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56290/0 (socket says 172.21.15.60:56290) 2023-10-16T12:53:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.684+0000 7f2fcb7fe640 1 -- 172.21.15.60:0/911485730 learned_addr learned my addr 172.21.15.60:0/911485730 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcaffd640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fcc06c100 0x7f2fcc06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fc3fff640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fcc070b60 0x7f2fcc150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcb7fe640 1 -- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fcc06c100 msgr2=0x7f2fcc06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcb7fe640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fcc06c100 0x7f2fcc06c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcb7fe640 1 -- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fcc070b60 msgr2=0x7f2fcc150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcb7fe640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fcc070b60 0x7f2fcc150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcb7fe640 1 -- 172.21.15.60:0/911485730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4009400 con 0x7f2fcc159950 2023-10-16T12:53:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.685+0000 7f2fcb7fe640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fcc159950 0x7f2fcc1551e0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c780 tx=0x7f2fc400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.686+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4004b60 con 0x7f2fcc159950 2023-10-16T12:53:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.686+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc400ce90 con 0x7f2fcc159950 2023-10-16T12:53:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.686+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400a850 con 0x7f2fcc159950 2023-10-16T12:53:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.686+0000 7f2fcbfff640 1 -- 172.21.15.60:0/911485730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc0ceeb0 con 0x7f2fcc159950 2023-10-16T12:53:48.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.686+0000 7f2fcbfff640 1 -- 172.21.15.60:0/911485730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc15c060 con 0x7f2fcc159950 2023-10-16T12:53:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.687+0000 7f2fcbfff640 1 -- 172.21.15.60:0/911485730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc04e9b0 con 0x7f2fcc159950 2023-10-16T12:53:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.688+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f2fc400a9b0 con 0x7f2fcc159950 2023-10-16T12:53:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.689+0000 7f2fc8ff9640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.689+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2fc4089d50 con 0x7f2fcc159950 2023-10-16T12:53:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.690+0000 7f2fcaffd640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:48.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.691+0000 7f2fcaffd640 1 --2- 172.21.15.60:0/911485730 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f2fac06b070 0x7f2fac06d530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc007c00 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:48.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.694+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fc4054980 con 0x7f2fcc159950 2023-10-16T12:53:48.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.914+0000 7f2fcbfff640 1 -- 172.21.15.60:0/911485730 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fcc15e1b0 con 0x7f2fac06b070 2023-10-16T12:53:48.919 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:48.917+0000 7fa346c17640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T12:53:48.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:48.917+0000 7f2fc8ff9640 1 -- 172.21.15.60:0/911485730 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7f2fbc005c80 con 0x7f2fac06b070 2023-10-16T12:53:48.920 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T12:53:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:53:48.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-10-16T12:53:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.175+0000 7f1b00105640 1 Processor -- start 2023-10-16T12:53:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.175+0000 7f1b00105640 1 -- start start 2023-10-16T12:53:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.176+0000 7f1b00105640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1af80d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.177+0000 7f1b00105640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af80b5d90 con 0x7f1af80cd260 2023-10-16T12:53:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.177+0000 7f1b00105640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af8160fc0 con 0x7f1af815d410 2023-10-16T12:53:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.177+0000 7f1b00105640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af80d2b50 con 0x7f1af815d030 2023-10-16T12:53:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.177+0000 7f1afe6a2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1af80cd260 0x7f1af80b4720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42290/0 (socket says 172.21.15.60:42290) 2023-10-16T12:53:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.177+0000 7f1afe6a2640 1 -- 172.21.15.60:0/669360406 learned_addr learned my addr 172.21.15.60:0/669360406 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.177+0000 7f1afdea1640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1af80d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456185447 0 0) 0x7f1af80b5d90 con 0x7f1af80cd260 2023-10-16T12:53:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae4003520 con 0x7f1af80cd260 2023-10-16T12:53:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442674711 0 0) 0x7f1ae4003520 con 0x7f1af80cd260 2023-10-16T12:53:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 msgr2=0x7f1af80d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1af80d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 >> v1:172.21.15.144:6789/0 conn(0x7f1af815d410 legacy=0x7f1af8161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af80d25d0 con 0x7f1af80cd260 2023-10-16T12:53:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1af0001670 con 0x7f1af80cd260 2023-10-16T12:53:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afdea1640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1af80d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:53:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 >> v1:172.21.15.60:6789/0 conn(0x7f1af80cd260 legacy=0x7f1af80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae40039a0 0x7f1ae4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1ae40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae4004d00 0x7f1ae4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae400bac0 con 0x7f1ae4004d00 2023-10-16T12:53:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae400bc00 con 0x7f1ae40039a0 2023-10-16T12:53:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.178+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae400bd70 con 0x7f1af815d030 2023-10-16T12:53:49.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae4004d00 0x7f1ae4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afdea1640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae40039a0 0x7f1ae4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afd6a0640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1ae40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afe6a2640 1 -- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 msgr2=0x7f1ae40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:49.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af815d030 0x7f1ae40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:49.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afe6a2640 1 -- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae40039a0 msgr2=0x7f1ae4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:49.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae40039a0 0x7f1ae4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:49.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.179+0000 7f1afe6a2640 1 -- 172.21.15.60:0/669360406 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af4009a70 con 0x7f1ae4004d00 2023-10-16T12:53:49.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.180+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae4004d00 0x7f1ae4005180 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c190 tx=0x7f1ae40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7ddd2d3893ac5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:49.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.180+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0005ad0 con 0x7f1ae4004d00 2023-10-16T12:53:49.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.180+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0005c30 con 0x7f1ae4004d00 2023-10-16T12:53:49.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.180+0000 7f1afce9f640 1 -- 172.21.15.60:0/669360406 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af002c8a0 con 0x7f1ae4004d00 2023-10-16T12:53:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1b00105640 1 -- 172.21.15.60:0/669360406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae4004d00 msgr2=0x7f1ae4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1b00105640 1 --2- 172.21.15.60:0/669360406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae4004d00 0x7f1ae4005180 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c190 tx=0x7f1ae40057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1afe6a2640 1 -- 172.21.15.60:0/669360406 reap_dead start 2023-10-16T12:53:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1b00105640 1 -- 172.21.15.60:0/669360406 shutdown_connections 2023-10-16T12:53:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1b00105640 1 -- 172.21.15.60:0/669360406 >> 172.21.15.60:0/669360406 conn(0x7f1af80cf720 msgr2=0x7f1af80cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1b00105640 1 -- 172.21.15.60:0/669360406 shutdown_connections 2023-10-16T12:53:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.181+0000 7f1b00105640 1 -- 172.21.15.60:0/669360406 wait complete. 2023-10-16T12:53:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.183+0000 7f1b00105640 1 Processor -- start 2023-10-16T12:53:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.183+0000 7f1b00105640 1 -- start start 2023-10-16T12:53:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1b00105640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4004d00 0x7f1af81f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1b00105640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af81f2ca0 0x7f1af8065d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1b00105640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1af8151a50 0x7f1af8151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1b00105640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af8152410 con 0x7f1af8151a50 2023-10-16T12:53:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1b00105640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af81f71e0 con 0x7f1ae4004d00 2023-10-16T12:53:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1b00105640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af815db80 con 0x7f1af81f2ca0 2023-10-16T12:53:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1afe6a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1af8151a50 0x7f1af8151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1afe6a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1af8151a50 0x7f1af8151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57482/0 (socket says 172.21.15.60:57482) 2023-10-16T12:53:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.184+0000 7f1afe6a2640 1 -- 172.21.15.60:0/4163963189 learned_addr learned my addr 172.21.15.60:0/4163963189 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afd6a0640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af81f2ca0 0x7f1af8065d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afdea1640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4004d00 0x7f1af81f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afe6a2640 1 -- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af81f2ca0 msgr2=0x7f1af8065d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1af81f2ca0 0x7f1af8065d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afe6a2640 1 -- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4004d00 msgr2=0x7f1af81f6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4004d00 0x7f1af81f6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afe6a2640 1 -- 172.21.15.60:0/4163963189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af81f3480 con 0x7f1af8151a50 2023-10-16T12:53:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afdea1640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ae4004d00 0x7f1af81f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1afe6a2640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1af8151a50 0x7f1af8151ed0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c190 tx=0x7f1af002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af003c040 con 0x7f1af8151a50 2023-10-16T12:53:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af002ccb0 con 0x7f1af8151a50 2023-10-16T12:53:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.185+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0007270 con 0x7f1af8151a50 2023-10-16T12:53:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.186+0000 7f1b00105640 1 -- 172.21.15.60:0/4163963189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af81f3760 con 0x7f1af8151a50 2023-10-16T12:53:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.186+0000 7f1b00105640 1 -- 172.21.15.60:0/4163963189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af81f3120 con 0x7f1af8151a50 2023-10-16T12:53:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.187+0000 7f1b00105640 1 -- 172.21.15.60:0/4163963189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af81f3c80 con 0x7f1af8151a50 2023-10-16T12:53:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.188+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f1af0032070 con 0x7f1af8151a50 2023-10-16T12:53:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.189+0000 7f1aee7fc640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f1adc06b0a0 0x7f1adc06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.189+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f1af00aa7d0 con 0x7f1af8151a50 2023-10-16T12:53:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.190+0000 7f1afdea1640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f1adc06b0a0 0x7f1adc06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.191+0000 7f1afdea1640 1 --2- 172.21.15.60:0/4163963189 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f1adc06b0a0 0x7f1adc06d560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1af40075e0 tx=0x7f1af400b040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.194+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1af0075400 con 0x7f1af8151a50 2023-10-16T12:53:49.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.417+0000 7f1b00105640 1 -- 172.21.15.60:0/4163963189 --> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f1af8160490 con 0x7f1adc06b0a0 2023-10-16T12:53:49.422 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:49.421+0000 7fa346c17640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:49.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:49.421+0000 7f1aee7fc640 1 -- 172.21.15.60:0/4163963189 <== mgr.7135 v2:172.21.15.144:6832/1139477310 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1af8160490 con 0x7f1adc06b0a0 2023-10-16T12:53:49.423 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:53:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:53:49.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.438 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-10-16T12:53:49.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.499 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-10-16T12:53:49.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.560 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-10-16T12:53:49.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.622 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-10-16T12:53:49.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.682 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-10-16T12:53:49.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.746 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-10-16T12:53:49.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:49.806 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-10-16T12:53:49.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:53:50.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.092+0000 7f6737fff640 1 Processor -- start 2023-10-16T12:53:50.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.092+0000 7f6737fff640 1 -- start start 2023-10-16T12:53:50.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6737fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 0x7f67380b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6737fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67380cd2f0 0x7f67380cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6737fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67380cac20 con 0x7f6738070240 2023-10-16T12:53:50.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6737fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67380c9d70 con 0x7f67380cd2f0 2023-10-16T12:53:50.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6737fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67380ca010 con 0x7f67380b3780 2023-10-16T12:53:50.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6736ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6738070240 0x7f6738070620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42294/0 (socket says 172.21.15.60:42294) 2023-10-16T12:53:50.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.094+0000 7f6736ffd640 1 -- 172.21.15.60:0/4126268403 learned_addr learned my addr 172.21.15.60:0/4126268403 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:50.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556190677 0 0) 0x7f67380cac20 con 0x7f6738070240 2023-10-16T12:53:50.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f67377fe640 1 --2- 172.21.15.60:0/4126268403 >> v2:172.21.15.144:3300/0 conn(0x7f67380cd2f0 0x7f67380cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6724003520 con 0x7f6738070240 2023-10-16T12:53:50.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f67367fc640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 0x7f67380b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655979780 0 0) 0x7f6724003520 con 0x7f6738070240 2023-10-16T12:53:50.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 msgr2=0x7f67380b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:50.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 0x7f67380b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 >> v2:172.21.15.144:3300/0 conn(0x7f67380cd2f0 msgr2=0x7f67380cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:50.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 --2- 172.21.15.60:0/4126268403 >> v2:172.21.15.144:3300/0 conn(0x7f67380cd2f0 0x7f67380cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67380ca2f0 con 0x7f6738070240 2023-10-16T12:53:50.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f6728001670 con 0x7f6738070240 2023-10-16T12:53:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 >> v1:172.21.15.60:6789/0 conn(0x7f6738070240 legacy=0x7f6738070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6724003970 0x7f6724003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 0x7f67240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6724004d00 0x7f6724005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380c9d70 con 0x7f6724004d00 2023-10-16T12:53:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380ca010 con 0x7f6724003970 2023-10-16T12:53:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.095+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672400ba00 con 0x7f67380b3780 2023-10-16T12:53:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67377fe640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6724004d00 0x7f6724005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67367fc640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 0x7f67240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f6736ffd640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6724003970 0x7f6724003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67377fe640 1 -- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 msgr2=0x7f67240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67377fe640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67380b3780 0x7f67240047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67377fe640 1 -- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6724003970 msgr2=0x7f6724003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67377fe640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6724003970 0x7f6724003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f67377fe640 1 -- 172.21.15.60:0/4126268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724003520 con 0x7f6724004d00 2023-10-16T12:53:50.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.096+0000 7f6736ffd640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6724003970 0x7f6724003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:50.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.097+0000 7f67377fe640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6724004d00 0x7f6724005180 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6730009ac0 tx=0x7f67240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2aa26c7170ac529 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:50.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.097+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673002c710 con 0x7f6724004d00 2023-10-16T12:53:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.097+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673002c870 con 0x7f6724004d00 2023-10-16T12:53:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.097+0000 7f6735ffb640 1 -- 172.21.15.60:0/4126268403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673002cc00 con 0x7f6724004d00 2023-10-16T12:53:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f6737fff640 1 -- 172.21.15.60:0/4126268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6724004d00 msgr2=0x7f6724005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f6737fff640 1 --2- 172.21.15.60:0/4126268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6724004d00 0x7f6724005180 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6730009ac0 tx=0x7f67240057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f67377fe640 1 -- 172.21.15.60:0/4126268403 reap_dead start 2023-10-16T12:53:50.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f6737fff640 1 -- 172.21.15.60:0/4126268403 shutdown_connections 2023-10-16T12:53:50.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f6737fff640 1 -- 172.21.15.60:0/4126268403 >> 172.21.15.60:0/4126268403 conn(0x7f67380cc230 msgr2=0x7f67380cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:50.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f6737fff640 1 -- 172.21.15.60:0/4126268403 shutdown_connections 2023-10-16T12:53:50.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.098+0000 7f6737fff640 1 -- 172.21.15.60:0/4126268403 wait complete. 2023-10-16T12:53:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.100+0000 7f6737fff640 1 Processor -- start 2023-10-16T12:53:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.100+0000 7f6737fff640 1 -- start start 2023-10-16T12:53:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.101+0000 7f6737fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6724004d00 0x7f673815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.101+0000 7f6737fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 0x7f67380c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.101+0000 7f6737fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67380c8560 0x7f673815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.101+0000 7f6737fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380c4ad0 con 0x7f67380c5e10 2023-10-16T12:53:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.101+0000 7f6737fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380c4d70 con 0x7f67380c8560 2023-10-16T12:53:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.101+0000 7f6737fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67380c5050 con 0x7f6724004d00 2023-10-16T12:53:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 0x7f67380c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 0x7f67380c4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57506/0 (socket says 172.21.15.60:57506) 2023-10-16T12:53:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 -- 172.21.15.60:0/57968605 learned_addr learned my addr 172.21.15.60:0/57968605 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67377fe640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67380c8560 0x7f673815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 -- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6724004d00 msgr2=0x7f673815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6724004d00 0x7f673815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 -- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67380c8560 msgr2=0x7f673815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67380c8560 0x7f673815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 -- 172.21.15.60:0/57968605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67380c5330 con 0x7f67380c5e10 2023-10-16T12:53:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67377fe640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67380c8560 0x7f673815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.102+0000 7f67367fc640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 0x7f67380c4590 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f6720007e60 tx=0x7f672000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.103+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200145f0 con 0x7f67380c5e10 2023-10-16T12:53:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.103+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720014750 con 0x7f67380c5e10 2023-10-16T12:53:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.103+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67380c5610 con 0x7f67380c5e10 2023-10-16T12:53:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.103+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720014ae0 con 0x7f67380c5e10 2023-10-16T12:53:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.103+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67380c89e0 con 0x7f67380c5e10 2023-10-16T12:53:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.104+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673804e9b0 con 0x7f67380c5e10 2023-10-16T12:53:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.105+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 71) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f6720014c40 con 0x7f67380c5e10 2023-10-16T12:53:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.106+0000 7f672ffff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f671806b070 0x7f671806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.106+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f672008a9c0 con 0x7f67380c5e10 2023-10-16T12:53:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.107+0000 7f6736ffd640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f671806b070 0x7f671806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:50.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.108+0000 7f6736ffd640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f671806b070 0x7f671806d530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f67280079f0 tx=0x7f6728009040 comp rx=0 tx=0).ready entity=mgr.7135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:50.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.112+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6720055580 con 0x7f67380c5e10 2023-10-16T12:53:50.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.361+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f67380b4fb0 con 0x7f67380c5e10 2023-10-16T12:53:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.944+0000 7f672ffff640 1 -- 172.21.15.60:0/57968605 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v72) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6720058ba0 con 0x7f67380c5e10 2023-10-16T12:53:50.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f671806b070 msgr2=0x7f671806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:50.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f671806b070 0x7f671806d530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f67280079f0 tx=0x7f6728009040 comp rx=0 tx=0).stop 2023-10-16T12:53:50.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 msgr2=0x7f67380c4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:50.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 0x7f67380c4590 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f6720007e60 tx=0x7f672000f520 comp rx=0 tx=0).stop 2023-10-16T12:53:50.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 shutdown_connections 2023-10-16T12:53:50.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f671806b070 0x7f671806d530 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67380c8560 0x7f673815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67380c5e10 0x7f67380c4590 secure :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f6720007e60 tx=0x7f672000f520 comp rx=0 tx=0).stop 2023-10-16T12:53:50.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 --2- 172.21.15.60:0/57968605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6724004d00 0x7f673815c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:50.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.948+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 >> 172.21.15.60:0/57968605 conn(0x7f67380cc230 msgr2=0x7f6738161db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.950+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 shutdown_connections 2023-10-16T12:53:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:50.950+0000 7f6737fff640 1 -- 172.21.15.60:0/57968605 wait complete. 2023-10-16T12:53:50.983 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:53:50.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:53:51.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:53:51.205 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:53:51.206 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:51.206 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:51.261 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:51.261 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:51.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:51.480 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:51.477+0000 7fa0a702b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:53:51.495 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:51.493+0000 7fad0663a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:53:51.533 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:51.531+0000 7fa0a702b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:53:51.553 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:51.550+0000 7fad0663a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:53:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.557+0000 7fb3f6528640 1 Processor -- start 2023-10-16T12:53:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.557+0000 7fb3f6528640 1 -- start start 2023-10-16T12:53:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.558+0000 7fb3f6528640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb3f015cd70 0x7fb3f00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.558+0000 7fb3f6528640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3f00b8480 0x7fb3f00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f6528640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f6528640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f015d150 con 0x7fb3f00b8480 2023-10-16T12:53:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f6528640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f00d1180 con 0x7fb3f015cd70 2023-10-16T12:53:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f6528640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00d1460 con 0x7fb3f00d0be0 2023-10-16T12:53:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3f00b8480 0x7fb3f00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3f00b8480 0x7fb3f00cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57520/0 (socket says 172.21.15.60:57520) 2023-10-16T12:53:51.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 -- 172.21.15.60:0/2352743518 learned_addr learned my addr 172.21.15.60:0/2352743518 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f5526640 1 --2- 172.21.15.60:0/2352743518 >> v2:172.21.15.144:3300/0 conn(0x7fb3f015cd70 0x7fb3f00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f5d27640 1 --2- 172.21.15.60:0/2352743518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 -- 172.21.15.60:0/2352743518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 msgr2=0x7fb3f00cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 --2- 172.21.15.60:0/2352743518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 -- 172.21.15.60:0/2352743518 >> v2:172.21.15.144:3300/0 conn(0x7fb3f015cd70 msgr2=0x7fb3f00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 --2- 172.21.15.60:0/2352743518 >> v2:172.21.15.144:3300/0 conn(0x7fb3f015cd70 0x7fb3f00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.559+0000 7fb3f4d25640 1 -- 172.21.15.60:0/2352743518 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f00d1740 con 0x7fb3f00b8480 2023-10-16T12:53:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.561+0000 7fb3f4d25640 1 --2- 172.21.15.60:0/2352743518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f00b8480 0x7fb3f00cd7c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00b56a0 tx=0x7fb3e402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34242d5b95d98ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.561+0000 7fb3effff640 1 -- 172.21.15.60:0/2352743518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4031860 con 0x7fb3f00b8480 2023-10-16T12:53:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.561+0000 7fb3effff640 1 -- 172.21.15.60:0/2352743518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e40319c0 con 0x7fb3f00b8480 2023-10-16T12:53:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.561+0000 7fb3effff640 1 -- 172.21.15.60:0/2352743518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4031d50 con 0x7fb3f00b8480 2023-10-16T12:53:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 -- 172.21.15.60:0/2352743518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f00b8480 msgr2=0x7fb3f00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 --2- 172.21.15.60:0/2352743518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f00b8480 0x7fb3f00cd7c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00b56a0 tx=0x7fb3e402b900 comp rx=0 tx=0).stop 2023-10-16T12:53:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 -- 172.21.15.60:0/2352743518 shutdown_connections 2023-10-16T12:53:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 --2- 172.21.15.60:0/2352743518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 --2- 172.21.15.60:0/2352743518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f00b8480 0x7fb3f00cd7c0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 --2- 172.21.15.60:0/2352743518 >> v2:172.21.15.144:3300/0 conn(0x7fb3f015cd70 0x7fb3f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 -- 172.21.15.60:0/2352743518 >> 172.21.15.60:0/2352743518 conn(0x7fb3f00cf570 msgr2=0x7fb3f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 -- 172.21.15.60:0/2352743518 shutdown_connections 2023-10-16T12:53:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.562+0000 7fb3f6528640 1 -- 172.21.15.60:0/2352743518 wait complete. 2023-10-16T12:53:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.564+0000 7fb3f6528640 1 Processor -- start 2023-10-16T12:53:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.564+0000 7fb3f6528640 1 -- start start 2023-10-16T12:53:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.565+0000 7fb3f6528640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f00b8480 0x7fb3f0150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.565+0000 7fb3f6528640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.565+0000 7fb3f6528640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 0x7fb3f00b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.565+0000 7fb3f6528640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01f3050 con 0x7fb3f015cd70 2023-10-16T12:53:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.565+0000 7fb3f6528640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01f3330 con 0x7fb3f00b8480 2023-10-16T12:53:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.565+0000 7fb3f6528640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f00cb0c0 con 0x7fb3f00d0be0 2023-10-16T12:53:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 0x7fb3f00b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 0x7fb3f00b4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57530/0 (socket says 172.21.15.60:57530) 2023-10-16T12:53:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 -- 172.21.15.60:0/3541216455 learned_addr learned my addr 172.21.15.60:0/3541216455 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f4d25640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5526640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f00b8480 0x7fb3f0150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 -- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 msgr2=0x7fb3f00b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 -- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f00b8480 msgr2=0x7fb3f0150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f00b8480 0x7fb3f0150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5d27640 1 -- 172.21.15.60:0/3541216455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40094d0 con 0x7fb3f015cd70 2023-10-16T12:53:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.566+0000 7fb3f5526640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f00b8480 0x7fb3f0150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.567+0000 7fb3f5d27640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 0x7fb3f00b4760 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800d580 tx=0x7fb3e800da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.567+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e80125f0 con 0x7fb3f015cd70 2023-10-16T12:53:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.567+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e8012770 con 0x7fb3f015cd70 2023-10-16T12:53:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.567+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f00cb3a0 con 0x7fb3f015cd70 2023-10-16T12:53:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.567+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8012b20 con 0x7fb3f015cd70 2023-10-16T12:53:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.567+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f01602f0 con 0x7fb3f015cd70 2023-10-16T12:53:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.568+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f00cb8c0 con 0x7fb3f015cd70 2023-10-16T12:53:51.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.569+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fb3e8012c80 con 0x7fb3f015cd70 2023-10-16T12:53:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.570+0000 7fb3edffb640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 0x7fb3dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.570+0000 7fb3f5526640 1 -- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 msgr2=0x7fb3dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.570+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fb3e8089f60 con 0x7fb3f015cd70 2023-10-16T12:53:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.570+0000 7fb3f5526640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 0x7fb3dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.574+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3f00cb8c0 con 0x7fb3f015cd70 2023-10-16T12:53:51.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.770+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3f00cb8c0 con 0x7fb3f015cd70 2023-10-16T12:53:51.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.771+0000 7fb3f5526640 1 -- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 msgr2=0x7fb3dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:51.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.771+0000 7fb3f5526640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 0x7fb3dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:51.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.772+0000 7fb3edffb640 1 -- 172.21.15.60:0/3541216455 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+17019 (secure 0 0 0) 0x7fb3f00cb8c0 con 0x7fb3f015cd70 2023-10-16T12:53:51.775 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:51.775 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:41.484660+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:6832/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/690478556":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/177518790":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2812310974":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1140080778":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1196951169":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/325928307":"2023-10-17T12:53:41.484587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:51.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 msgr2=0x7fb3dc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:51.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 0x7fb3dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 msgr2=0x7fb3f00b4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:51.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 0x7fb3f00b4760 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800d580 tx=0x7fb3e800da40 comp rx=0 tx=0).stop 2023-10-16T12:53:51.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 shutdown_connections 2023-10-16T12:53:51.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb3dc06b020 0x7fb3dc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f00d0be0 0x7fb3f00b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3f015cd70 0x7fb3f00b4760 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 --2- 172.21.15.60:0/3541216455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f00b8480 0x7fb3f0150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:51.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 >> 172.21.15.60:0/3541216455 conn(0x7fb3f00cf570 msgr2=0x7fb3f00b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:51.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 shutdown_connections 2023-10-16T12:53:51.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:51.775+0000 7fb3f6528640 1 -- 172.21.15.60:0/3541216455 wait complete. 2023-10-16T12:53:51.796 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:53:24.764434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:53:25.816764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:53:51.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:53:51.981 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:51.979+0000 7fa0a702b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:53:52.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.001+0000 7fcb2757d640 1 Processor -- start 2023-10-16T12:53:52.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.002+0000 7fcb2757d640 1 -- start start 2023-10-16T12:53:52.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.003+0000 7fcb2757d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb200ad350 0x7fcb20069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.003+0000 7fcb2757d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2006a730 con 0x7fcb2015d150 2023-10-16T12:53:52.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.003+0000 7fcb2757d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200ac890 con 0x7fcb2006a2e0 2023-10-16T12:53:52.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.003+0000 7fcb2757d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2015d530 con 0x7fcb200ad350 2023-10-16T12:53:52.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb25319640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcb2015d150 0x7fcb20160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42308/0 (socket says 172.21.15.60:42308) 2023-10-16T12:53:52.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb25319640 1 -- 172.21.15.60:0/1765640326 learned_addr learned my addr 172.21.15.60:0/1765640326 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb24b18640 1 --2- 172.21.15.60:0/1765640326 >> v2:172.21.15.144:3300/0 conn(0x7fcb200ad350 0x7fcb20069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286292117 0 0) 0x7fcb1c001110 con 0x7fcb2015d150 2023-10-16T12:53:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb0c003520 con 0x7fcb2015d150 2023-10-16T12:53:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700250460 0 0) 0x7fcb0c003520 con 0x7fcb2015d150 2023-10-16T12:53:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 >> v2:172.21.15.144:3300/0 conn(0x7fcb200ad350 msgr2=0x7fcb20069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.004+0000 7fcb13fff640 1 --2- 172.21.15.60:0/1765640326 >> v2:172.21.15.144:3300/0 conn(0x7fcb200ad350 0x7fcb20069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 >> v1:172.21.15.144:6789/0 conn(0x7fcb2006a2e0 legacy=0x7fcb200c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200ad7a0 con 0x7fcb2015d150 2023-10-16T12:53:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcb1c001800 con 0x7fcb2015d150 2023-10-16T12:53:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 >> v1:172.21.15.60:6789/0 conn(0x7fcb2015d150 legacy=0x7fcb20160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb0c0039c0 0x7fcb0c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c005180 0x7fcb0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2015d530 con 0x7fcb0c0047c0 2023-10-16T12:53:52.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00c080 con 0x7fcb0c005180 2023-10-16T12:53:52.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00c240 con 0x7fcb0c0039c0 2023-10-16T12:53:52.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb24b18640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb25319640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb0c0039c0 0x7fcb0c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.005+0000 7fcb25b1a640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c005180 0x7fcb0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.006+0000 7fcb24b18640 1 -- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb0c0039c0 msgr2=0x7fcb0c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:52.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.006+0000 7fcb24b18640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb0c0039c0 0x7fcb0c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.006+0000 7fcb24b18640 1 -- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c005180 msgr2=0x7fcb0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:52.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.006+0000 7fcb24b18640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c005180 0x7fcb0c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.006+0000 7fcb24b18640 1 -- 172.21.15.60:0/1765640326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c003520 con 0x7fcb0c0047c0 2023-10-16T12:53:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.006+0000 7fcb25b1a640 1 -- 172.21.15.60:0/1765640326 reap_dead start 2023-10-16T12:53:52.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.007+0000 7fcb24b18640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009ac0 tx=0x7fcb0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=544984877c1302b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:52.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.007+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1402c6b0 con 0x7fcb0c0047c0 2023-10-16T12:53:52.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1402c810 con 0x7fcb0c0047c0 2023-10-16T12:53:52.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb13fff640 1 -- 172.21.15.60:0/1765640326 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1402cba0 con 0x7fcb0c0047c0 2023-10-16T12:53:52.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb2757d640 1 -- 172.21.15.60:0/1765640326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c0047c0 msgr2=0x7fcb0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb2757d640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009ac0 tx=0x7fcb0c0057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb2757d640 1 -- 172.21.15.60:0/1765640326 shutdown_connections 2023-10-16T12:53:52.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb2757d640 1 --2- 172.21.15.60:0/1765640326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.008+0000 7fcb2757d640 1 -- 172.21.15.60:0/1765640326 >> 172.21.15.60:0/1765640326 conn(0x7fcb200ce9b0 msgr2=0x7fcb200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.009+0000 7fcb2757d640 1 -- 172.21.15.60:0/1765640326 shutdown_connections 2023-10-16T12:53:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.009+0000 7fcb2757d640 1 -- 172.21.15.60:0/1765640326 wait complete. 2023-10-16T12:53:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.010+0000 7fcb2757d640 1 Processor -- start 2023-10-16T12:53:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.010+0000 7fcb2757d640 1 -- start start 2023-10-16T12:53:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.011+0000 7fcb2757d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c0047c0 0x7fcb2015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.011+0000 7fcb2757d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb20156020 0x7fcb20155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.011+0000 7fcb2757d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 0x7fcb200707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.011+0000 7fcb2757d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20070d00 con 0x7fcb20070330 2023-10-16T12:53:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.011+0000 7fcb2757d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200c3cb0 con 0x7fcb0c0047c0 2023-10-16T12:53:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.011+0000 7fcb2757d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20155cd0 con 0x7fcb20156020 2023-10-16T12:53:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 0x7fcb200707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 0x7fcb200707c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57558/0 (socket says 172.21.15.60:57558) 2023-10-16T12:53:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 -- 172.21.15.60:0/3976539037 learned_addr learned my addr 172.21.15.60:0/3976539037 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:52.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb24b18640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb20156020 0x7fcb20155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25319640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c0047c0 0x7fcb2015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 -- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb20156020 msgr2=0x7fcb20155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb20156020 0x7fcb20155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 -- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c0047c0 msgr2=0x7fcb2015d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c0047c0 0x7fcb2015d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 -- 172.21.15.60:0/3976539037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2015e5b0 con 0x7fcb20070330 2023-10-16T12:53:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.012+0000 7fcb25b1a640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 0x7fcb200707c0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcb18007fd0 tx=0x7fcb1800e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.013+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18003b80 con 0x7fcb20070330 2023-10-16T12:53:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.013+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18003ce0 con 0x7fcb20070330 2023-10-16T12:53:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.013+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2015e890 con 0x7fcb20070330 2023-10-16T12:53:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.013+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18005310 con 0x7fcb20070330 2023-10-16T12:53:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.013+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2015ee50 con 0x7fcb20070330 2023-10-16T12:53:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.014+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb2004e160 con 0x7fcb20070330 2023-10-16T12:53:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.015+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fcb18005470 con 0x7fcb20070330 2023-10-16T12:53:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.017+0000 7fcb11ffb640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 0x7fcb0406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.017+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fcb18089390 con 0x7fcb20070330 2023-10-16T12:53:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.017+0000 7fcb25319640 1 -- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 msgr2=0x7fcb0406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.017+0000 7fcb25319640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 0x7fcb0406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.021+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb18053fd0 con 0x7fcb20070330 2023-10-16T12:53:52.035 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.032+0000 7fad0663a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:53:52.035 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.033+0000 7fa0a702b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:53:52.084 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.082+0000 7fa0a702b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:53:52.090 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.088+0000 7fad0663a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:53:52.143 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.141+0000 7fad0663a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:53:52.144 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.142+0000 7fa0a702b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:53:52.193 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.191+0000 7fa0a702b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:53:52.207 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.205+0000 7fad0663a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:53:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.216+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcb200acb20 con 0x7fcb20070330 2023-10-16T12:53:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.217+0000 7fcb11ffb640 1 -- 172.21.15.60:0/3976539037 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb180575f0 con 0x7fcb20070330 2023-10-16T12:53:52.220 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:53:52.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.218+0000 7fcb25319640 1 -- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 msgr2=0x7fcb0406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:52.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.218+0000 7fcb25319640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 0x7fcb0406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:52.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 msgr2=0x7fcb0406d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:52.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 0x7fcb0406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 msgr2=0x7fcb200707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 0x7fcb200707c0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcb18007fd0 tx=0x7fcb1800e9e0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 shutdown_connections 2023-10-16T12:53:52.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb0406b050 0x7fcb0406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb20156020 0x7fcb20155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb20070330 0x7fcb200707c0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 --2- 172.21.15.60:0/3976539037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c0047c0 0x7fcb2015d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 >> 172.21.15.60:0/3976539037 conn(0x7fcb200ce9b0 msgr2=0x7fcb2006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:52.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 shutdown_connections 2023-10-16T12:53:52.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.220+0000 7fcb2757d640 1 -- 172.21.15.60:0/3976539037 wait complete. 2023-10-16T12:53:52.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:53:52.260 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.258+0000 7fad0663a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:53:52.302 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.300+0000 7fa0a702b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:53:52.363 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.361+0000 7fa0a702b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:52.378 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.376+0000 7fad0663a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:53:52.414 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.412+0000 7fa0a702b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:53:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.435+0000 7fcb9f3d5640 1 Processor -- start 2023-10-16T12:53:52.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.436+0000 7fcb9f3d5640 1 -- start start 2023-10-16T12:53:52.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9f3d5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcb980cf710 0x7fcb980b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9f3d5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980d18c0 con 0x7fcb980cfaf0 2023-10-16T12:53:52.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9f3d5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980d3980 con 0x7fcb980b6cd0 2023-10-16T12:53:52.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9f3d5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb980d3c60 con 0x7fcb980cf710 2023-10-16T12:53:52.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9d171640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcb980cf710 0x7fcb980b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9c970640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcb980cfaf0 0x7fcb980d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42310/0 (socket says 172.21.15.60:42310) 2023-10-16T12:53:52.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.437+0000 7fcb9c970640 1 -- 172.21.15.60:0/2829341550 learned_addr learned my addr 172.21.15.60:0/2829341550 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:52.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.438+0000 7fcb9d171640 1 -- 172.21.15.60:0/2829341550 >> v1:172.21.15.144:6789/0 conn(0x7fcb980b6cd0 legacy=0x7fcb980d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:52.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.438+0000 7fcb9d171640 1 -- 172.21.15.60:0/2829341550 >> v1:172.21.15.60:6789/0 conn(0x7fcb980cfaf0 legacy=0x7fcb980d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.438+0000 7fcb9d171640 1 -- 172.21.15.60:0/2829341550 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980ce630 con 0x7fcb980cf710 2023-10-16T12:53:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.439+0000 7fcb9d171640 1 --2- 172.21.15.60:0/2829341550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb980b2d10 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009790 tx=0x7fcb9402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de002132bd0a25f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.439+0000 7fcb8ffff640 1 -- 172.21.15.60:0/2829341550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94031780 con 0x7fcb980cf710 2023-10-16T12:53:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.439+0000 7fcb8ffff640 1 -- 172.21.15.60:0/2829341550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb940318e0 con 0x7fcb980cf710 2023-10-16T12:53:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.439+0000 7fcb8ffff640 1 -- 172.21.15.60:0/2829341550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94031c70 con 0x7fcb980cf710 2023-10-16T12:53:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/2829341550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 msgr2=0x7fcb980b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/2829341550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb980b2d10 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009790 tx=0x7fcb9402b820 comp rx=0 tx=0).stop 2023-10-16T12:53:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/2829341550 shutdown_connections 2023-10-16T12:53:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/2829341550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb980b2d10 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/2829341550 >> 172.21.15.60:0/2829341550 conn(0x7fcb980d2cc0 msgr2=0x7fcb980d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/2829341550 shutdown_connections 2023-10-16T12:53:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.440+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/2829341550 wait complete. 2023-10-16T12:53:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.442+0000 7fcb9f3d5640 1 Processor -- start 2023-10-16T12:53:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.442+0000 7fcb9f3d5640 1 -- start start 2023-10-16T12:53:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9f3d5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb980b6cd0 0x7fcb980b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9f3d5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb981f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9f3d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb980cfaf0 0x7fcb981f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9f3d5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980b1cb0 con 0x7fcb980cf710 2023-10-16T12:53:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9f3d5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980b0e20 con 0x7fcb980b6cd0 2023-10-16T12:53:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9f3d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980b1030 con 0x7fcb980cfaf0 2023-10-16T12:53:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9c970640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb981f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9c970640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb981f21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57580/0 (socket says 172.21.15.60:57580) 2023-10-16T12:53:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9c970640 1 -- 172.21.15.60:0/4099697047 learned_addr learned my addr 172.21.15.60:0/4099697047 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.443+0000 7fcb9d171640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb980b6cd0 0x7fcb980b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9d972640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb980cfaf0 0x7fcb981f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9c970640 1 -- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb980cfaf0 msgr2=0x7fcb981f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9c970640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb980cfaf0 0x7fcb981f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9c970640 1 -- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb980b6cd0 msgr2=0x7fcb980b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9c970640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb980b6cd0 0x7fcb980b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9c970640 1 -- 172.21.15.60:0/4099697047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb940093a0 con 0x7fcb980cf710 2023-10-16T12:53:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb9c970640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb981f21e0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c480 tx=0x7fcb8800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.444+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88005ce0 con 0x7fcb980cf710 2023-10-16T12:53:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.445+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88005e40 con 0x7fcb980cf710 2023-10-16T12:53:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.445+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880072f0 con 0x7fcb980cf710 2023-10-16T12:53:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.445+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb980b1310 con 0x7fcb980cf710 2023-10-16T12:53:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.445+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb980b18d0 con 0x7fcb980cf710 2023-10-16T12:53:52.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.446+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb980b1df0 con 0x7fcb980cf710 2023-10-16T12:53:52.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.447+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fcb88007510 con 0x7fcb980cf710 2023-10-16T12:53:52.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.448+0000 7fcb8dffb640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 0x7fcb7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.448+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fcb88088dd0 con 0x7fcb980cf710 2023-10-16T12:53:52.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.448+0000 7fcb9d171640 1 -- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 msgr2=0x7fcb7c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:52.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.448+0000 7fcb9d171640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 0x7fcb7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:52.455 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.450+0000 7fad0663a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:52.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.452+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb980b1df0 con 0x7fcb980cf710 2023-10-16T12:53:52.471 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.470+0000 7fa0a702b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:53:52.505 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.503+0000 7fad0663a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:53:52.523 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.521+0000 7fa0a702b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:53:52.566 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.563+0000 7fad0663a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:53:52.619 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:52.616+0000 7fad0663a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:53:52.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.648+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb980b1df0 con 0x7fcb980cf710 2023-10-16T12:53:52.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.649+0000 7fcb9d171640 1 -- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 msgr2=0x7fcb7c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:52.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.649+0000 7fcb9d171640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 0x7fcb7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:52.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.649+0000 7fcb8dffb640 1 -- 172.21.15.60:0/4099697047 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb980b1df0 con 0x7fcb980cf710 2023-10-16T12:53:52.653 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:53:52.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 msgr2=0x7fcb7c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:52.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 0x7fcb7c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 msgr2=0x7fcb981f21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb981f21e0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c480 tx=0x7fcb8800c940 comp rx=0 tx=0).stop 2023-10-16T12:53:52.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 shutdown_connections 2023-10-16T12:53:52.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fcb7c06b020 0x7fcb7c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb980cfaf0 0x7fcb981f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb980cf710 0x7fcb981f21e0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 --2- 172.21.15.60:0/4099697047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb980b6cd0 0x7fcb980b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 >> 172.21.15.60:0/4099697047 conn(0x7fcb980d2cc0 msgr2=0x7fcb980b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:52.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 shutdown_connections 2023-10-16T12:53:52.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.652+0000 7fcb9f3d5640 1 -- 172.21.15.60:0/4099697047 wait complete. 2023-10-16T12:53:52.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:53:52.902 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:52.900+0000 7fa0a702b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:53:52.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.925+0000 7f95a894d640 1 Processor -- start 2023-10-16T12:53:52.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.925+0000 7f95a894d640 1 -- start start 2023-10-16T12:53:52.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.926+0000 7f95a894d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f95a406c040 0x7f95a406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.926+0000 7f95a894d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f95a40cda80 0x7f95a415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.926+0000 7f95a894d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a406c960 con 0x7f95a40ad970 2023-10-16T12:53:52.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a894d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a415b9d0 con 0x7f95a40cda80 2023-10-16T12:53:52.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a894d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a415bcb0 con 0x7f95a406c040 2023-10-16T12:53:52.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f95a40cda80 0x7f95a415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f95a40cda80 0x7f95a415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57582/0 (socket says 172.21.15.60:57582) 2023-10-16T12:53:52.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 -- 172.21.15.60:0/115278738 learned_addr learned my addr 172.21.15.60:0/115278738 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:52.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a2d9d640 1 --2- 172.21.15.60:0/115278738 >> v2:172.21.15.144:3300/0 conn(0x7f95a406c040 0x7f95a406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 -- 172.21.15.60:0/115278738 >> v2:172.21.15.144:3300/0 conn(0x7f95a406c040 msgr2=0x7f95a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:52.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 --2- 172.21.15.60:0/115278738 >> v2:172.21.15.144:3300/0 conn(0x7f95a406c040 0x7f95a406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 -- 172.21.15.60:0/115278738 >> v1:172.21.15.144:6789/0 conn(0x7f95a40ad970 legacy=0x7f95a415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:52.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.927+0000 7f95a359e640 1 -- 172.21.15.60:0/115278738 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a415bf90 con 0x7f95a40cda80 2023-10-16T12:53:52.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.928+0000 7f95a359e640 1 --2- 172.21.15.60:0/115278738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a40cda80 0x7f95a415ed90 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f95a40cf350 tx=0x7f959c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2257017472e6f80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:52.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.929+0000 7f95a259c640 1 -- 172.21.15.60:0/115278738 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c02ca80 con 0x7f95a40cda80 2023-10-16T12:53:52.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.929+0000 7f95a259c640 1 -- 172.21.15.60:0/115278738 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c02cbe0 con 0x7f95a40cda80 2023-10-16T12:53:52.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.929+0000 7f95a259c640 1 -- 172.21.15.60:0/115278738 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c0343f0 con 0x7f95a40cda80 2023-10-16T12:53:52.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 -- 172.21.15.60:0/115278738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a40cda80 msgr2=0x7f95a415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 --2- 172.21.15.60:0/115278738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a40cda80 0x7f95a415ed90 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f95a40cf350 tx=0x7f959c02bee0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 -- 172.21.15.60:0/115278738 shutdown_connections 2023-10-16T12:53:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 --2- 172.21.15.60:0/115278738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a40cda80 0x7f95a415ed90 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 --2- 172.21.15.60:0/115278738 >> v2:172.21.15.144:3300/0 conn(0x7f95a406c040 0x7f95a406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 -- 172.21.15.60:0/115278738 >> 172.21.15.60:0/115278738 conn(0x7f95a40ce6a0 msgr2=0x7f95a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:52.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 -- 172.21.15.60:0/115278738 shutdown_connections 2023-10-16T12:53:52.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.930+0000 7f95a894d640 1 -- 172.21.15.60:0/115278738 wait complete. 2023-10-16T12:53:52.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.932+0000 7f95a894d640 1 Processor -- start 2023-10-16T12:53:52.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.932+0000 7f95a894d640 1 -- start start 2023-10-16T12:53:52.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a894d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 0x7f95a41f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a894d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95a40ad970 0x7f95a415d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a894d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95a40cda80 0x7f95a40af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a894d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40c6420 con 0x7f95a406c040 2023-10-16T12:53:52.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a894d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40c5810 con 0x7f95a40ad970 2023-10-16T12:53:52.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a894d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40c5af0 con 0x7f95a40cda80 2023-10-16T12:53:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 0x7f95a41f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 0x7f95a41f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57586/0 (socket says 172.21.15.60:57586) 2023-10-16T12:53:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a2d9d640 1 -- 172.21.15.60:0/2557009239 learned_addr learned my addr 172.21.15.60:0/2557009239 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f959bfff640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95a40ad970 0x7f95a415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.933+0000 7f95a359e640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95a40cda80 0x7f95a40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a2d9d640 1 -- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95a40cda80 msgr2=0x7f95a40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a2d9d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95a40cda80 0x7f95a40af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a2d9d640 1 -- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95a40ad970 msgr2=0x7f95a415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:52.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a2d9d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95a40ad970 0x7f95a415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a2d9d640 1 -- 172.21.15.60:0/2557009239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c009b30 con 0x7f95a406c040 2023-10-16T12:53:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a2d9d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 0x7f95a41f2b30 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f959400d5f0 tx=0x7f959400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95940125f0 con 0x7f95a406c040 2023-10-16T12:53:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9594012750 con 0x7f95a406c040 2023-10-16T12:53:52.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a40c5dd0 con 0x7f95a406c040 2023-10-16T12:53:52.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.934+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594012b00 con 0x7f95a406c040 2023-10-16T12:53:52.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.935+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41f3070 con 0x7f95a406c040 2023-10-16T12:53:52.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.936+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a406b330 con 0x7f95a406c040 2023-10-16T12:53:52.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.938+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f9594012c60 con 0x7f95a406c040 2023-10-16T12:53:52.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.938+0000 7f959b7fe640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 0x7f958406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:52.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.939+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f9594089ee0 con 0x7f95a406c040 2023-10-16T12:53:52.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.939+0000 7f959bfff640 1 -- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 msgr2=0x7f958406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:52.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.939+0000 7f959bfff640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 0x7f958406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:52.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:52.943+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9594054b20 con 0x7f95a406c040 2023-10-16T12:53:53.030 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.028+0000 7fad0663a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:53:53.064 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.062+0000 7fa0a702b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:53:53.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.138+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95a41f3200 con 0x7f95a406c040 2023-10-16T12:53:53.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.139+0000 7f959b7fe640 1 -- 172.21.15.60:0/2557009239 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95a41f3200 con 0x7f95a406c040 2023-10-16T12:53:53.142 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:53:53.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.140+0000 7f959bfff640 1 -- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 msgr2=0x7f958406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:53.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.140+0000 7f959bfff640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 0x7f958406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:53.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 msgr2=0x7f958406d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:53.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 0x7f958406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 msgr2=0x7f95a41f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 0x7f95a41f2b30 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f959400d5f0 tx=0x7f959400dab0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 shutdown_connections 2023-10-16T12:53:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7f958406b050 0x7f958406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95a40cda80 0x7f95a40af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95a40ad970 0x7f95a415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 --2- 172.21.15.60:0/2557009239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95a406c040 0x7f95a41f2b30 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.141+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 >> 172.21.15.60:0/2557009239 conn(0x7f95a40ce6a0 msgr2=0x7f95a40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:53.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.142+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 shutdown_connections 2023-10-16T12:53:53.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.142+0000 7f95a894d640 1 -- 172.21.15.60:0/2557009239 wait complete. 2023-10-16T12:53:53.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:53:53.172 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.171+0000 7fa0a702b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:53:53.200 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.198+0000 7fad0663a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:53:53.222 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.220+0000 7fa0a702b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:53:53.316 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.314+0000 7fad0663a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:53:53.328 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.327+0000 7fa0a702b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:53.370 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.368+0000 7fad0663a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:53:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.370+0000 7ffa9e790640 1 Processor -- start 2023-10-16T12:53:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.371+0000 7ffa9e790640 1 -- start start 2023-10-16T12:53:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.372+0000 7ffa9e790640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa980b3420 0x7ffa980b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.372+0000 7ffa9e790640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cfaf0 0x7ffa980b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.372+0000 7ffa9e790640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa980d3980 con 0x7ffa980cf710 2023-10-16T12:53:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.372+0000 7ffa9e790640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa980d3c60 con 0x7ffa980b3420 2023-10-16T12:53:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.372+0000 7ffa9e790640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980ce630 con 0x7ffa980cfaf0 2023-10-16T12:53:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.373+0000 7ffa8ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa980b3420 0x7ffa980b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.373+0000 7ffa9cd2d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cfaf0 0x7ffa980b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.373+0000 7ffa8ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa980b3420 0x7ffa980b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:53032/0 (socket says 172.21.15.60:53032) 2023-10-16T12:53:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.373+0000 7ffa8ffff640 1 -- 172.21.15.60:0/2562976613 learned_addr learned my addr 172.21.15.60:0/2562976613 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.374+0000 7ffa9cd2d640 1 -- 172.21.15.60:0/2562976613 >> v2:172.21.15.144:3300/0 conn(0x7ffa980b3420 msgr2=0x7ffa980b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.374+0000 7ffa9cd2d640 1 --2- 172.21.15.60:0/2562976613 >> v2:172.21.15.144:3300/0 conn(0x7ffa980b3420 0x7ffa980b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.374+0000 7ffa9cd2d640 1 -- 172.21.15.60:0/2562976613 >> v1:172.21.15.144:6789/0 conn(0x7ffa980cf710 legacy=0x7ffa980b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.374+0000 7ffa9cd2d640 1 -- 172.21.15.60:0/2562976613 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa980ce910 con 0x7ffa980cfaf0 2023-10-16T12:53:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.375+0000 7ffa9cd2d640 1 --2- 172.21.15.60:0/2562976613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cfaf0 0x7ffa980b6200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009d90 tx=0x7ffa9002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b09c8281a408c048 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.375+0000 7ffa977fe640 1 -- 172.21.15.60:0/2562976613 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9002cbc0 con 0x7ffa980cfaf0 2023-10-16T12:53:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.375+0000 7ffa977fe640 1 -- 172.21.15.60:0/2562976613 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9002cd20 con 0x7ffa980cfaf0 2023-10-16T12:53:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.375+0000 7ffa977fe640 1 -- 172.21.15.60:0/2562976613 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90034660 con 0x7ffa980cfaf0 2023-10-16T12:53:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 -- 172.21.15.60:0/2562976613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cfaf0 msgr2=0x7ffa980b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 --2- 172.21.15.60:0/2562976613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cfaf0 0x7ffa980b6200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009d90 tx=0x7ffa9002be10 comp rx=0 tx=0).stop 2023-10-16T12:53:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 -- 172.21.15.60:0/2562976613 shutdown_connections 2023-10-16T12:53:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 --2- 172.21.15.60:0/2562976613 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cfaf0 0x7ffa980b6200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 --2- 172.21.15.60:0/2562976613 >> v2:172.21.15.144:3300/0 conn(0x7ffa980b3420 0x7ffa980b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 -- 172.21.15.60:0/2562976613 >> 172.21.15.60:0/2562976613 conn(0x7ffa980d2cc0 msgr2=0x7ffa980d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 -- 172.21.15.60:0/2562976613 shutdown_connections 2023-10-16T12:53:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.376+0000 7ffa9e790640 1 -- 172.21.15.60:0/2562976613 wait complete. 2023-10-16T12:53:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.378+0000 7ffa9e790640 1 Processor -- start 2023-10-16T12:53:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.378+0000 7ffa9e790640 1 -- start start 2023-10-16T12:53:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.379+0000 7ffa9e790640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 0x7ffa98153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.379+0000 7ffa9e790640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cf710 0x7ffa981f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.379+0000 7ffa9e790640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa980cfaf0 0x7ffa980b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.379+0000 7ffa9e790640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981f2720 con 0x7ffa980b3420 2023-10-16T12:53:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.379+0000 7ffa9e790640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9809eb90 con 0x7ffa980cfaf0 2023-10-16T12:53:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.379+0000 7ffa9e790640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98150070 con 0x7ffa980cf710 2023-10-16T12:53:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 0x7ffa98153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 0x7ffa98153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57596/0 (socket says 172.21.15.60:57596) 2023-10-16T12:53:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 -- 172.21.15.60:0/1955430014 learned_addr learned my addr 172.21.15.60:0/1955430014 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa8ffff640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cf710 0x7ffa981f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa9cd2d640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa980cfaf0 0x7ffa980b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 -- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cf710 msgr2=0x7ffa981f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cf710 0x7ffa981f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 -- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa980cfaf0 msgr2=0x7ffa980b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa980cfaf0 0x7ffa980b6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 -- 172.21.15.60:0/1955430014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90009a70 con 0x7ffa980b3420 2023-10-16T12:53:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.380+0000 7ffa97fff640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 0x7ffa98153620 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800c450 tx=0x7ffa8800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.381+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa880052d0 con 0x7ffa980b3420 2023-10-16T12:53:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.381+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88005450 con 0x7ffa980b3420 2023-10-16T12:53:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.381+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88003880 con 0x7ffa980b3420 2023-10-16T12:53:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.381+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa98150350 con 0x7ffa980b3420 2023-10-16T12:53:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.381+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa98150910 con 0x7ffa980b3420 2023-10-16T12:53:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.382+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9804e160 con 0x7ffa980b3420 2023-10-16T12:53:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.383+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7ffa880039e0 con 0x7ffa980b3420 2023-10-16T12:53:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.384+0000 7ffa957fa640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 0x7ffa7806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.384+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ffa88089610 con 0x7ffa980b3420 2023-10-16T12:53:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.384+0000 7ffa8ffff640 1 -- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 msgr2=0x7ffa7806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.384+0000 7ffa8ffff640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 0x7ffa7806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.388+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa880542d0 con 0x7ffa980b3420 2023-10-16T12:53:53.437 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.435+0000 7fa0a702b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:53:53.487 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.484+0000 7fad0663a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.585+0000 7ffa8ffff640 1 -- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 msgr2=0x7ffa7806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.585+0000 7ffa8ffff640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 0x7ffa7806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:53.593 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.591+0000 7fa0a702b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.596+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffa980ce2f0 con 0x7ffa980b3420 2023-10-16T12:53:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.597+0000 7ffa957fa640 1 -- 172.21.15.60:0/1955430014 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v144) v1 ==== 80+0+473 (secure 0 0 0) 0x7ffa880578f0 con 0x7ffa980b3420 2023-10-16T12:53:53.599 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/682650079 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/682650079 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/690478556 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/177518790 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.600 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2812310974 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.601 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1140080778 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.601 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1196951169 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.601 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/325928307 2023-10-17T12:53:41.484587+0000 2023-10-16T12:53:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T12:53:53.602 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.597+0000 7fad0663a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:53:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 msgr2=0x7ffa7806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 0x7ffa7806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 msgr2=0x7ffa98153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 0x7ffa98153620 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800c450 tx=0x7ffa8800c910 comp rx=0 tx=0).stop 2023-10-16T12:53:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 shutdown_connections 2023-10-16T12:53:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffa7806b050 0x7ffa7806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa980cf710 0x7ffa981f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa980cfaf0 0x7ffa980b6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.599+0000 7ffa9e790640 1 --2- 172.21.15.60:0/1955430014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa980b3420 0x7ffa98153620 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.600+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 >> 172.21.15.60:0/1955430014 conn(0x7ffa980d2cc0 msgr2=0x7ffa980b5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.600+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 shutdown_connections 2023-10-16T12:53:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:53.600+0000 7ffa9e790640 1 -- 172.21.15.60:0/1955430014 wait complete. 2023-10-16T12:53:53.621 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:53:53.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:53.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:53:53.641 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.639+0000 7fa0a702b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:53.759 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:53.757+0000 7fa0a702b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:53:53.764 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T12:53:53.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:53:53.765 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:53:53.773 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.771+0000 7fad0663a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:53:53.814 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:53:53.815 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:53.828 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.826+0000 7fad0663a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:53:53.871 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:53:53.872 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:53:53.872 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:53.928 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:53:53.929 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:53:53.929 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:53:53.955 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:53.953+0000 7fad0663a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:53:53.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:53:53.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.201+0000 7fb2861d5640 1 Processor -- start 2023-10-16T12:53:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.201+0000 7fb2861d5640 1 -- start start 2023-10-16T12:53:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb2861d5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb28015aaa0 0x7fb28015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb2861d5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb28015ae80 0x7fb2800cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb2861d5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28015ed60 con 0x7fb2800ccaa0 2023-10-16T12:53:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb2861d5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28006b810 con 0x7fb28015aaa0 2023-10-16T12:53:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb2861d5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2800707e0 con 0x7fb28015ae80 2023-10-16T12:53:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb27f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb28015aaa0 0x7fb28015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb27f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb28015aaa0 0x7fb28015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57608/0 (socket says 172.21.15.60:57608) 2023-10-16T12:53:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.202+0000 7fb27f7fe640 1 -- 172.21.15.60:0/3148889902 learned_addr learned my addr 172.21.15.60:0/3148889902 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.203+0000 7fb277fff640 1 --2- 172.21.15.60:0/3148889902 >> v2:172.21.15.144:3300/0 conn(0x7fb28015ae80 0x7fb2800cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.203+0000 7fb27f7fe640 1 -- 172.21.15.60:0/3148889902 >> v2:172.21.15.144:3300/0 conn(0x7fb28015ae80 msgr2=0x7fb2800cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.203+0000 7fb27f7fe640 1 --2- 172.21.15.60:0/3148889902 >> v2:172.21.15.144:3300/0 conn(0x7fb28015ae80 0x7fb2800cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.203+0000 7fb27f7fe640 1 -- 172.21.15.60:0/3148889902 >> v1:172.21.15.144:6789/0 conn(0x7fb2800ccaa0 legacy=0x7fb280070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.203+0000 7fb27f7fe640 1 -- 172.21.15.60:0/3148889902 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28015ef90 con 0x7fb28015aaa0 2023-10-16T12:53:54.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.204+0000 7fb27f7fe640 1 --2- 172.21.15.60:0/3148889902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb28015e820 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb2800c59c0 tx=0x7fb27002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdccafc59b1ff11d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.204+0000 7fb27effd640 1 -- 172.21.15.60:0/3148889902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270031870 con 0x7fb28015aaa0 2023-10-16T12:53:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.204+0000 7fb27effd640 1 -- 172.21.15.60:0/3148889902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2700319d0 con 0x7fb28015aaa0 2023-10-16T12:53:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.204+0000 7fb27effd640 1 -- 172.21.15.60:0/3148889902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270031d60 con 0x7fb28015aaa0 2023-10-16T12:53:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 -- 172.21.15.60:0/3148889902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 msgr2=0x7fb28015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 --2- 172.21.15.60:0/3148889902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb28015e820 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb2800c59c0 tx=0x7fb27002b910 comp rx=0 tx=0).stop 2023-10-16T12:53:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 -- 172.21.15.60:0/3148889902 shutdown_connections 2023-10-16T12:53:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 --2- 172.21.15.60:0/3148889902 >> v2:172.21.15.144:3300/0 conn(0x7fb28015ae80 0x7fb2800cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 --2- 172.21.15.60:0/3148889902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb28015e820 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 -- 172.21.15.60:0/3148889902 >> 172.21.15.60:0/3148889902 conn(0x7fb2800cb410 msgr2=0x7fb2800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 -- 172.21.15.60:0/3148889902 shutdown_connections 2023-10-16T12:53:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.205+0000 7fb2861d5640 1 -- 172.21.15.60:0/3148889902 wait complete. 2023-10-16T12:53:54.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.207+0000 7fb2861d5640 1 Processor -- start 2023-10-16T12:53:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.207+0000 7fb2861d5640 1 -- start start 2023-10-16T12:53:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb2861d5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2800ccaa0 0x7fb2800c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb2861d5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb2800c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb2861d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28015ae80 0x7fb2800c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb2861d5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28014ea90 con 0x7fb28015aaa0 2023-10-16T12:53:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb2861d5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28014daf0 con 0x7fb2800ccaa0 2023-10-16T12:53:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb2861d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28014ddd0 con 0x7fb28015ae80 2023-10-16T12:53:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb2800c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb2800c4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57616/0 (socket says 172.21.15.60:57616) 2023-10-16T12:53:54.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 -- 172.21.15.60:0/1483010635 learned_addr learned my addr 172.21.15.60:0/1483010635 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:54.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb27ffff640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28015ae80 0x7fb2800c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb27f7fe640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2800ccaa0 0x7fb2800c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 -- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28015ae80 msgr2=0x7fb2800c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28015ae80 0x7fb2800c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 -- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2800ccaa0 msgr2=0x7fb2800c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2800ccaa0 0x7fb2800c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb277fff640 1 -- 172.21.15.60:0/1483010635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2700094d0 con 0x7fb28015aaa0 2023-10-16T12:53:54.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb27f7fe640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2800ccaa0 0x7fb2800c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:54.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.208+0000 7fb27ffff640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28015ae80 0x7fb2800c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:53:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.209+0000 7fb277fff640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb2800c4690 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb26800d5c0 tx=0x7fb26800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.209+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680125f0 con 0x7fb28015aaa0 2023-10-16T12:53:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.209+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb268012750 con 0x7fb28015aaa0 2023-10-16T12:53:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.209+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28014e0b0 con 0x7fb28015aaa0 2023-10-16T12:53:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.209+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268012b00 con 0x7fb28015aaa0 2023-10-16T12:53:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.209+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb280159850 con 0x7fb28015aaa0 2023-10-16T12:53:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.210+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28014ec40 con 0x7fb28015aaa0 2023-10-16T12:53:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.211+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fb268012c60 con 0x7fb28015aaa0 2023-10-16T12:53:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.212+0000 7fb27cff9640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 0x7fb26006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.212+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fb268089fe0 con 0x7fb28015aaa0 2023-10-16T12:53:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.212+0000 7fb27f7fe640 1 -- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 msgr2=0x7fb26006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.212+0000 7fb27f7fe640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 0x7fb26006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.217+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb28014ec40 con 0x7fb28015aaa0 2023-10-16T12:53:54.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.413+0000 7fb27f7fe640 1 -- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 msgr2=0x7fb26006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:54.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.413+0000 7fb27f7fe640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 0x7fb26006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.447+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb280070600 con 0x7fb28015aaa0 2023-10-16T12:53:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.448+0000 7fb27cff9640 1 -- 172.21.15.60:0/1483010635 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fb268054c50 con 0x7fb28015aaa0 2023-10-16T12:53:54.451 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:54.451 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:53:26.782026+0000","modified":"2023-10-16T12:53:29.833032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7048},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7048":{"gid":7048,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/3032734613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3032734613},{"type":"v1","addr":"172.21.15.60:6834","nonce":3032734613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:53:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 70 2023-10-16T12:53:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 msgr2=0x7fb26006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 0x7fb26006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 msgr2=0x7fb2800c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb2800c4690 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb26800d5c0 tx=0x7fb26800da80 comp rx=0 tx=0).stop 2023-10-16T12:53:54.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 shutdown_connections 2023-10-16T12:53:54.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7fb26006b020 0x7fb26006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28015ae80 0x7fb2800c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28015aaa0 0x7fb2800c4690 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 --2- 172.21.15.60:0/1483010635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2800ccaa0 0x7fb2800c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 >> 172.21.15.60:0/1483010635 conn(0x7fb2800cb410 msgr2=0x7fb2800b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:54.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 shutdown_connections 2023-10-16T12:53:54.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.450+0000 7fb2861d5640 1 -- 172.21.15.60:0/1483010635 wait complete. 2023-10-16T12:53:54.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:54.598 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:54.597+0000 7fa0a702b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:53:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.659+0000 7ffbac6bb640 1 Processor -- start 2023-10-16T12:53:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.659+0000 7ffbac6bb640 1 -- start start 2023-10-16T12:53:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.660+0000 7ffbac6bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 0x7ffba40b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.660+0000 7ffbac6bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffba406fa90 0x7ffba40cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.660+0000 7ffbac6bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba415c1b0 con 0x7ffba415b0c0 2023-10-16T12:53:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.660+0000 7ffbac6bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba40ce2d0 con 0x7ffba406fa90 2023-10-16T12:53:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.660+0000 7ffbac6bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40b63d0 con 0x7ffba415ace0 2023-10-16T12:53:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.661+0000 7ffbaa457640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 0x7ffba40b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.661+0000 7ffbaa457640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 0x7ffba40b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:36726/0 (socket says 172.21.15.60:36726) 2023-10-16T12:53:54.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.661+0000 7ffbaa457640 1 -- 172.21.15.60:0/1834613615 learned_addr learned my addr 172.21.15.60:0/1834613615 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:54.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.661+0000 7ffbaac58640 1 --2- 172.21.15.60:0/1834613615 >> v2:172.21.15.144:3300/0 conn(0x7ffba406fa90 0x7ffba40cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.662+0000 7ffbaa457640 1 -- 172.21.15.60:0/1834613615 >> v2:172.21.15.144:3300/0 conn(0x7ffba406fa90 msgr2=0x7ffba40cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:54.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.662+0000 7ffbaa457640 1 --2- 172.21.15.60:0/1834613615 >> v2:172.21.15.144:3300/0 conn(0x7ffba406fa90 0x7ffba40cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.662+0000 7ffbaa457640 1 -- 172.21.15.60:0/1834613615 >> v1:172.21.15.144:6789/0 conn(0x7ffba415b0c0 legacy=0x7ffba415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.662+0000 7ffbaa457640 1 -- 172.21.15.60:0/1834613615 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba40b66b0 con 0x7ffba415ace0 2023-10-16T12:53:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.663+0000 7ffbaa457640 1 --2- 172.21.15.60:0/1834613615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 0x7ffba40b2f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009870 tx=0x7ffba002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=191715db10dbbdb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffba9455640 1 -- 172.21.15.60:0/1834613615 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0031640 con 0x7ffba415ace0 2023-10-16T12:53:54.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffba9455640 1 -- 172.21.15.60:0/1834613615 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba00317a0 con 0x7ffba415ace0 2023-10-16T12:53:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffba9455640 1 -- 172.21.15.60:0/1834613615 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0031b30 con 0x7ffba415ace0 2023-10-16T12:53:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffbac6bb640 1 -- 172.21.15.60:0/1834613615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 msgr2=0x7ffba40b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/1834613615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 0x7ffba40b2f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009870 tx=0x7ffba002b8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffbac6bb640 1 -- 172.21.15.60:0/1834613615 shutdown_connections 2023-10-16T12:53:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/1834613615 >> v2:172.21.15.144:3300/0 conn(0x7ffba406fa90 0x7ffba40cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/1834613615 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba415ace0 0x7ffba40b2f60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.664+0000 7ffbac6bb640 1 -- 172.21.15.60:0/1834613615 >> 172.21.15.60:0/1834613615 conn(0x7ffba406d700 msgr2=0x7ffba406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.665+0000 7ffbac6bb640 1 -- 172.21.15.60:0/1834613615 shutdown_connections 2023-10-16T12:53:54.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.665+0000 7ffbac6bb640 1 -- 172.21.15.60:0/1834613615 wait complete. 2023-10-16T12:53:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.666+0000 7ffbac6bb640 1 Processor -- start 2023-10-16T12:53:54.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.666+0000 7ffbac6bb640 1 -- start start 2023-10-16T12:53:54.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.667+0000 7ffbac6bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba406fa90 0x7ffba40afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.667+0000 7ffbac6bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 0x7ffba4159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.667+0000 7ffbac6bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba415b0c0 0x7ffba40c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.667+0000 7ffbac6bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba415a190 con 0x7ffba415ace0 2023-10-16T12:53:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.667+0000 7ffbac6bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40ca2b0 con 0x7ffba415b0c0 2023-10-16T12:53:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.667+0000 7ffbac6bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40c8e30 con 0x7ffba406fa90 2023-10-16T12:53:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 0x7ffba4159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 0x7ffba4159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57630/0 (socket says 172.21.15.60:57630) 2023-10-16T12:53:54.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 -- 172.21.15.60:0/807952908 learned_addr learned my addr 172.21.15.60:0/807952908 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:54.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffbaa457640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba406fa90 0x7ffba40afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffbaac58640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba415b0c0 0x7ffba40c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:54.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 -- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba406fa90 msgr2=0x7ffba40afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba406fa90 0x7ffba40afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 -- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba415b0c0 msgr2=0x7ffba40c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba415b0c0 0x7ffba40c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 -- 172.21.15.60:0/807952908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba00094d0 con 0x7ffba415ace0 2023-10-16T12:53:54.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffbaa457640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba406fa90 0x7ffba40afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:54.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffbaac58640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba415b0c0 0x7ffba40c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:54.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.668+0000 7ffba9c56640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 0x7ffba4159080 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400c3f0 tx=0x7ffb9400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:54.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.669+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb940052d0 con 0x7ffba415ace0 2023-10-16T12:53:54.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.669+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94005450 con 0x7ffba415ace0 2023-10-16T12:53:54.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.669+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba40c8870 con 0x7ffba415ace0 2023-10-16T12:53:54.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.669+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94007260 con 0x7ffba415ace0 2023-10-16T12:53:54.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.669+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba40ca530 con 0x7ffba415ace0 2023-10-16T12:53:54.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.670+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba404e160 con 0x7ffba415ace0 2023-10-16T12:53:54.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.671+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 72) v1 ==== 89388+0+0 (secure 0 0 0) 0x7ffb940073c0 con 0x7ffba415ace0 2023-10-16T12:53:54.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.672+0000 7ffb9affd640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 0x7ffb8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:54.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.672+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ffb94088e20 con 0x7ffba415ace0 2023-10-16T12:53:54.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.672+0000 7ffbaa457640 1 -- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 msgr2=0x7ffb8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:54.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.672+0000 7ffbaa457640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 0x7ffb8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:53:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.680+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb94053ae0 con 0x7ffba415ace0 2023-10-16T12:53:54.820 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:54.816+0000 7fad0663a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:53:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.873+0000 7ffbaa457640 1 -- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 msgr2=0x7ffb8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1139477310 2023-10-16T12:53:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.873+0000 7ffbaa457640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 0x7ffb8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:53:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.874+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffba40d03b0 con 0x7ffba415ace0 2023-10-16T12:53:54.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.876+0000 7ffb9affd640 1 -- 172.21.15.60:0/807952908 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+17019 (secure 0 0 0) 0x7ffb94057100 con 0x7ffba415ace0 2023-10-16T12:53:54.881 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:54.881 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:41.484660+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:6832/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/690478556":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/177518790":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2812310974":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1140080778":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1196951169":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/325928307":"2023-10-17T12:53:41.484587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:54.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 msgr2=0x7ffb8806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:53:54.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 0x7ffb8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 msgr2=0x7ffba4159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:54.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 0x7ffba4159080 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400c3f0 tx=0x7ffb9400c8b0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 shutdown_connections 2023-10-16T12:53:54.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:6832/1139477310,v1:172.21.15.144:6833/1139477310] conn(0x7ffb8806b020 0x7ffb8806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba415b0c0 0x7ffba40c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba415ace0 0x7ffba4159080 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 --2- 172.21.15.60:0/807952908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffba406fa90 0x7ffba40afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:54.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 >> 172.21.15.60:0/807952908 conn(0x7ffba406d700 msgr2=0x7ffba406dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:54.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 shutdown_connections 2023-10-16T12:53:54.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:54.880+0000 7ffbac6bb640 1 -- 172.21.15.60:0/807952908 wait complete. 2023-10-16T12:53:54.902 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:53:24.764434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:53:25.816764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:53:54.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:53:54.918 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:54.916+0000 7fa0a702b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:53:54.978 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:54.976+0000 7fa0a702b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:53:55.092 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.091+0000 7fa0a702b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:53:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.123+0000 7f2184409640 1 Processor -- start 2023-10-16T12:53:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.123+0000 7f2184409640 1 -- start start 2023-10-16T12:53:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.124+0000 7f2184409640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f217c15aee0 0x7f217c0aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.124+0000 7f2184409640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217c06c140 con 0x7f217c15e9b0 2023-10-16T12:53:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.124+0000 7f2184409640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217c15b2c0 con 0x7f217c0cb810 2023-10-16T12:53:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.124+0000 7f2184409640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217c06b270 con 0x7f217c15aee0 2023-10-16T12:53:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21829a6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f217c15e9b0 0x7f217c15ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42318/0 (socket says 172.21.15.60:42318) 2023-10-16T12:53:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21829a6640 1 -- 172.21.15.60:0/1920530947 learned_addr learned my addr 172.21.15.60:0/1920530947 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21819a4640 1 --1- 172.21.15.60:0/1920530947 >> v1:172.21.15.144:6789/0 conn(0x7f217c0cb810 0x7f217c0cbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:44234/0 (socket says 172.21.15.60:44234) 2023-10-16T12:53:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21821a5640 1 --2- 172.21.15.60:0/1920530947 >> v2:172.21.15.144:3300/0 conn(0x7f217c15aee0 0x7f217c0aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93015954 0 0) 0x7f217c06c140 con 0x7f217c15e9b0 2023-10-16T12:53:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2168003520 con 0x7f217c15e9b0 2023-10-16T12:53:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756155971 0 0) 0x7f217c15b2c0 con 0x7f217c0cb810 2023-10-16T12:53:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f217c06c140 con 0x7f217c0cb810 2023-10-16T12:53:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377552843 0 0) 0x7f2168003520 con 0x7f217c15e9b0 2023-10-16T12:53:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21821a5640 1 -- 172.21.15.60:0/1920530947 >> v1:172.21.15.144:6789/0 conn(0x7f217c0cb810 legacy=0x7f217c0cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.125+0000 7f21821a5640 1 -- 172.21.15.60:0/1920530947 >> v1:172.21.15.60:6789/0 conn(0x7f217c15e9b0 legacy=0x7f217c15ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.126+0000 7f21821a5640 1 -- 172.21.15.60:0/1920530947 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c06daf0 con 0x7f217c15aee0 2023-10-16T12:53:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.127+0000 7f21821a5640 1 --2- 172.21.15.60:0/1920530947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c15aee0 0x7f217c0aeef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f21780097a0 tx=0x7f217802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d41b4d35c86a5dab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.127+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21740019f0 con 0x7f217c15aee0 2023-10-16T12:53:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.127+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217802e050 con 0x7f217c15aee0 2023-10-16T12:53:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.127+0000 7f21811a3640 1 -- 172.21.15.60:0/1920530947 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178030710 con 0x7f217c15aee0 2023-10-16T12:53:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 -- 172.21.15.60:0/1920530947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c15aee0 msgr2=0x7f217c0aeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 --2- 172.21.15.60:0/1920530947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c15aee0 0x7f217c0aeef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f21780097a0 tx=0x7f217802b790 comp rx=0 tx=0).stop 2023-10-16T12:53:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 -- 172.21.15.60:0/1920530947 shutdown_connections 2023-10-16T12:53:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 --2- 172.21.15.60:0/1920530947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c15aee0 0x7f217c0aeef0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 -- 172.21.15.60:0/1920530947 >> 172.21.15.60:0/1920530947 conn(0x7f217c0cfac0 msgr2=0x7f217c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 -- 172.21.15.60:0/1920530947 shutdown_connections 2023-10-16T12:53:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.128+0000 7f2184409640 1 -- 172.21.15.60:0/1920530947 wait complete. 2023-10-16T12:53:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.130+0000 7f2184409640 1 Processor -- start 2023-10-16T12:53:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.130+0000 7f2184409640 1 -- start start 2023-10-16T12:53:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f2184409640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c0cb810 0x7f217c1f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f2184409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f217c15aee0 0x7f217c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f2184409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 0x7f217c1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f2184409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c1f5500 con 0x7f217c15e9b0 2023-10-16T12:53:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f2184409640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c1f5ac0 con 0x7f217c0cb810 2023-10-16T12:53:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f2184409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217c1f29d0 con 0x7f217c15aee0 2023-10-16T12:53:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.131+0000 7f21829a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 0x7f217c1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 0x7f217c1f1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57642/0 (socket says 172.21.15.60:57642) 2023-10-16T12:53:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 -- 172.21.15.60:0/3134972550 learned_addr learned my addr 172.21.15.60:0/3134972550 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21821a5640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c0cb810 0x7f217c1f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21819a4640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f217c15aee0 0x7f217c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 -- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f217c15aee0 msgr2=0x7f217c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f217c15aee0 0x7f217c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 -- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c0cb810 msgr2=0x7f217c1f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c0cb810 0x7f217c1f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 -- 172.21.15.60:0/3134972550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2178009400 con 0x7f217c15e9b0 2023-10-16T12:53:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.132+0000 7f21829a6640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 0x7f217c1f1f10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f217400cc50 tx=0x7f217400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.133+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217400ce60 con 0x7f217c15e9b0 2023-10-16T12:53:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.133+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2174012400 con 0x7f217c15e9b0 2023-10-16T12:53:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.133+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21740127b0 con 0x7f217c15e9b0 2023-10-16T12:53:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.133+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c1f2cb0 con 0x7f217c15e9b0 2023-10-16T12:53:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.133+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c0a8fa0 con 0x7f217c15e9b0 2023-10-16T12:53:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.134+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217c1f2ea0 con 0x7f217c15e9b0 2023-10-16T12:53:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.135+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 73) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f2174012910 con 0x7f217c15e9b0 2023-10-16T12:53:55.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.136+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2174089a60 con 0x7f217c15e9b0 2023-10-16T12:53:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.141+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f217c1f2ea0 con 0x7f217c15e9b0 2023-10-16T12:53:55.145 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:55.143+0000 7fad0663a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:53:55.190 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa028bdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.190 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa028bdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa028bdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa028bdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa028bdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa02bbe1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.191 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa02bbe1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.192 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa02bbe1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.192 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa02bbe1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.192 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:53:55.188+0000 7fa02bbe1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:53:55.210 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:55.207+0000 7fad0663a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:53:55.330 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:53:55.327+0000 7fad0663a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:53:55.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.341+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f217c1f2ea0 con 0x7f217c15e9b0 2023-10-16T12:53:55.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.342+0000 7f2172ffd640 1 -- 172.21.15.60:0/3134972550 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7f217c1f2ea0 con 0x7f217c15e9b0 2023-10-16T12:53:55.344 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:53:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 msgr2=0x7f217c1f1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 0x7f217c1f1f10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f217400cc50 tx=0x7f217400a430 comp rx=0 tx=0).stop 2023-10-16T12:53:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 shutdown_connections 2023-10-16T12:53:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f217c15aee0 0x7f217c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f217c15e9b0 0x7f217c1f1f10 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 --2- 172.21.15.60:0/3134972550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c0cb810 0x7f217c1f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.344+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 >> 172.21.15.60:0/3134972550 conn(0x7f217c0cfac0 msgr2=0x7f217c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.345+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 shutdown_connections 2023-10-16T12:53:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.345+0000 7f2184409640 1 -- 172.21.15.60:0/3134972550 wait complete. 2023-10-16T12:53:55.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:53:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.588+0000 7f96b7d5a640 1 Processor -- start 2023-10-16T12:53:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.588+0000 7f96b7d5a640 1 -- start start 2023-10-16T12:53:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.589+0000 7f96b7d5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b015ad70 0x7f96b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.589+0000 7f96b7d5a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96b00caac0 0x7f96b006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.589+0000 7f96b7d5a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00cb030 con 0x7f96b00c9ca0 2023-10-16T12:53:55.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.589+0000 7f96b7d5a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b00c91c0 con 0x7f96b00caac0 2023-10-16T12:53:55.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.589+0000 7f96b7d5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c94a0 con 0x7f96b015ad70 2023-10-16T12:53:55.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b52f5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f96b00c9ca0 0x7f96b00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42320/0 (socket says 172.21.15.60:42320) 2023-10-16T12:53:55.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b52f5640 1 -- 172.21.15.60:0/4071227994 learned_addr learned my addr 172.21.15.60:0/4071227994 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:55.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b62f7640 1 --2- 172.21.15.60:0/4071227994 >> v2:172.21.15.60:3300/0 conn(0x7f96b00caac0 0x7f96b006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b5af6640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b015ad70 0x7f96b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b62f7640 1 -- 172.21.15.60:0/4071227994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b015ad70 msgr2=0x7f96b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b62f7640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b015ad70 0x7f96b0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b62f7640 1 -- 172.21.15.60:0/4071227994 >> v1:172.21.15.60:6789/0 conn(0x7f96b00c9ca0 legacy=0x7f96b00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b62f7640 1 -- 172.21.15.60:0/4071227994 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00c9780 con 0x7f96b00caac0 2023-10-16T12:53:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.590+0000 7f96b5af6640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b015ad70 0x7f96b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.592+0000 7f96b62f7640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00caac0 0x7f96b006a2e0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009d90 tx=0x7f96a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d12cb5f45816a0d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.592+0000 7f96b4af4640 1 -- 172.21.15.60:0/4071227994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a802cc60 con 0x7f96b00caac0 2023-10-16T12:53:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.592+0000 7f96b4af4640 1 -- 172.21.15.60:0/4071227994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a802cdc0 con 0x7f96b00caac0 2023-10-16T12:53:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.592+0000 7f96b4af4640 1 -- 172.21.15.60:0/4071227994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8034660 con 0x7f96b00caac0 2023-10-16T12:53:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 -- 172.21.15.60:0/4071227994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00caac0 msgr2=0x7f96b006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00caac0 0x7f96b006a2e0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009d90 tx=0x7f96a802beb0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 -- 172.21.15.60:0/4071227994 shutdown_connections 2023-10-16T12:53:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00caac0 0x7f96b006a2e0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/4071227994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b015ad70 0x7f96b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 -- 172.21.15.60:0/4071227994 >> 172.21.15.60:0/4071227994 conn(0x7f96b00ce9b0 msgr2=0x7f96b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 -- 172.21.15.60:0/4071227994 shutdown_connections 2023-10-16T12:53:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.593+0000 7f96b7d5a640 1 -- 172.21.15.60:0/4071227994 wait complete. 2023-10-16T12:53:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.594+0000 7f96b7d5a640 1 Processor -- start 2023-10-16T12:53:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.595+0000 7f96b7d5a640 1 -- start start 2023-10-16T12:53:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b7d5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00c9ca0 0x7f96b01f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b7d5a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00caac0 0x7f96b01592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b7d5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 0x7f96b00b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b7d5a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c79a0 con 0x7f96b015ad70 2023-10-16T12:53:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b7d5a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c6ad0 con 0x7f96b00caac0 2023-10-16T12:53:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b7d5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c6d20 con 0x7f96b00c9ca0 2023-10-16T12:53:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 0x7f96b00b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 0x7f96b00b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57654/0 (socket says 172.21.15.60:57654) 2023-10-16T12:53:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 -- 172.21.15.60:0/2294682778 learned_addr learned my addr 172.21.15.60:0/2294682778 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 -- 172.21.15.60:0/2294682778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00c9ca0 msgr2=0x7f96b01f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00c9ca0 0x7f96b01f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 -- 172.21.15.60:0/2294682778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00caac0 msgr2=0x7f96b01592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00caac0 0x7f96b01592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.596+0000 7f96b62f7640 1 -- 172.21.15.60:0/2294682778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8009a70 con 0x7f96b015ad70 2023-10-16T12:53:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.597+0000 7f96b62f7640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 0x7f96b00b0480 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009d90 tx=0x7f96a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.597+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8040650 con 0x7f96b015ad70 2023-10-16T12:53:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.597+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a80407d0 con 0x7f96b015ad70 2023-10-16T12:53:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.597+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8040b80 con 0x7f96b015ad70 2023-10-16T12:53:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.597+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b00c7000 con 0x7f96b015ad70 2023-10-16T12:53:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.597+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b00c75c0 con 0x7f96b015ad70 2023-10-16T12:53:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.599+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b004e160 con 0x7f96b015ad70 2023-10-16T12:53:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.599+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 73) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f96a8040ce0 con 0x7f96b015ad70 2023-10-16T12:53:55.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.600+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f96a80ac340 con 0x7f96b015ad70 2023-10-16T12:53:55.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.605+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96a8077470 con 0x7f96b015ad70 2023-10-16T12:53:55.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.800+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96b00b09c0 con 0x7f96b015ad70 2023-10-16T12:53:55.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.801+0000 7f96a67fc640 1 -- 172.21.15.60:0/2294682778 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v145) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96a807aa90 con 0x7f96b015ad70 2023-10-16T12:53:55.803 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T12:53:55.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 msgr2=0x7f96b00b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:55.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 0x7f96b00b0480 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009d90 tx=0x7f96a8004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:55.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 shutdown_connections 2023-10-16T12:53:55.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b015ad70 0x7f96b00b0480 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00caac0 0x7f96b01592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 --2- 172.21.15.60:0/2294682778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00c9ca0 0x7f96b01f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:55.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 >> 172.21.15.60:0/2294682778 conn(0x7f96b00ce9b0 msgr2=0x7f96b015b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:55.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 shutdown_connections 2023-10-16T12:53:55.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:55.804+0000 7f96b7d5a640 1 -- 172.21.15.60:0/2294682778 wait complete. 2023-10-16T12:53:55.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:53:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.049+0000 7fb565044640 1 Processor -- start 2023-10-16T12:53:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.049+0000 7fb565044640 1 -- start start 2023-10-16T12:53:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.051+0000 7fb565044640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb56015ad70 0x7fb560069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.051+0000 7fb565044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb56006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.051+0000 7fb565044640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5600cb030 con 0x7fb5600c9ca0 2023-10-16T12:53:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.051+0000 7fb565044640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5600c91c0 con 0x7fb56015ad70 2023-10-16T12:53:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.051+0000 7fb565044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600c94a0 con 0x7fb5600caac0 2023-10-16T12:53:56.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.052+0000 7fb55f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb56006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.052+0000 7fb55f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb56006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:36784/0 (socket says 172.21.15.60:36784) 2023-10-16T12:53:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.052+0000 7fb55f59e640 1 -- 172.21.15.60:0/2490516152 learned_addr learned my addr 172.21.15.60:0/2490516152 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.052+0000 7fb557fff640 1 --1- 172.21.15.60:0/2490516152 >> v1:172.21.15.144:6789/0 conn(0x7fb5600c9ca0 0x7fb5600ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:44248/0 (socket says 172.21.15.60:44248) 2023-10-16T12:53:56.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.052+0000 7fb55ed9d640 1 --2- 172.21.15.60:0/2490516152 >> v2:172.21.15.144:3300/0 conn(0x7fb56015ad70 0x7fb560069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.053+0000 7fb55e59c640 1 -- 172.21.15.60:0/2490516152 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585544674 0 0) 0x7fb5600cb030 con 0x7fb5600c9ca0 2023-10-16T12:53:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.053+0000 7fb55e59c640 1 -- 172.21.15.60:0/2490516152 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54c003520 con 0x7fb5600c9ca0 2023-10-16T12:53:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.053+0000 7fb55f59e640 1 -- 172.21.15.60:0/2490516152 >> v2:172.21.15.144:3300/0 conn(0x7fb56015ad70 msgr2=0x7fb560069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.053+0000 7fb55f59e640 1 --2- 172.21.15.60:0/2490516152 >> v2:172.21.15.144:3300/0 conn(0x7fb56015ad70 0x7fb560069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.053+0000 7fb55f59e640 1 -- 172.21.15.60:0/2490516152 >> v1:172.21.15.144:6789/0 conn(0x7fb5600c9ca0 legacy=0x7fb5600ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.053+0000 7fb55f59e640 1 -- 172.21.15.60:0/2490516152 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600c9780 con 0x7fb5600caac0 2023-10-16T12:53:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.054+0000 7fb55f59e640 1 --2- 172.21.15.60:0/2490516152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb56006a2e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb558009e50 tx=0x7fb55802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0a57d5fbf012023 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb55e59c640 1 -- 172.21.15.60:0/2490516152 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55802cb00 con 0x7fb5600caac0 2023-10-16T12:53:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb55e59c640 1 -- 172.21.15.60:0/2490516152 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55802cc60 con 0x7fb5600caac0 2023-10-16T12:53:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb55e59c640 1 -- 172.21.15.60:0/2490516152 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580343f0 con 0x7fb5600caac0 2023-10-16T12:53:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb565044640 1 -- 172.21.15.60:0/2490516152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 msgr2=0x7fb56006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb565044640 1 --2- 172.21.15.60:0/2490516152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb56006a2e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb558009e50 tx=0x7fb55802bf60 comp rx=0 tx=0).stop 2023-10-16T12:53:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb565044640 1 -- 172.21.15.60:0/2490516152 shutdown_connections 2023-10-16T12:53:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb565044640 1 --2- 172.21.15.60:0/2490516152 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb56006a2e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb565044640 1 --2- 172.21.15.60:0/2490516152 >> v2:172.21.15.144:3300/0 conn(0x7fb56015ad70 0x7fb560069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.055+0000 7fb565044640 1 -- 172.21.15.60:0/2490516152 >> 172.21.15.60:0/2490516152 conn(0x7fb5600ce9b0 msgr2=0x7fb5600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.056+0000 7fb565044640 1 -- 172.21.15.60:0/2490516152 shutdown_connections 2023-10-16T12:53:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.056+0000 7fb565044640 1 -- 172.21.15.60:0/2490516152 wait complete. 2023-10-16T12:53:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.058+0000 7fb565044640 1 Processor -- start 2023-10-16T12:53:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.058+0000 7fb565044640 1 -- start start 2023-10-16T12:53:56.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb565044640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5600c9ca0 0x7fb5600c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb565044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb560158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb565044640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 0x7fb56015cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb565044640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56015c790 con 0x7fb56015ad70 2023-10-16T12:53:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb565044640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600c4970 con 0x7fb5600c9ca0 2023-10-16T12:53:56.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb565044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56015be60 con 0x7fb5600caac0 2023-10-16T12:53:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.059+0000 7fb55f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 0x7fb56015cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 0x7fb56015cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57656/0 (socket says 172.21.15.60:57656) 2023-10-16T12:53:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 -- 172.21.15.60:0/1997723330 learned_addr learned my addr 172.21.15.60:0/1997723330 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb557fff640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb560158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55ed9d640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5600c9ca0 0x7fb5600c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 -- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 msgr2=0x7fb560158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb560158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 -- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5600c9ca0 msgr2=0x7fb5600c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5600c9ca0 0x7fb5600c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 -- 172.21.15.60:0/1997723330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558009b30 con 0x7fb56015ad70 2023-10-16T12:53:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.060+0000 7fb55f59e640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 0x7fb56015cc20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb558009e50 tx=0x7fb558004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.061+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558009510 con 0x7fb56015ad70 2023-10-16T12:53:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.061+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb558003e10 con 0x7fb56015ad70 2023-10-16T12:53:56.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.061+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558004450 con 0x7fb56015ad70 2023-10-16T12:53:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.061+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56015c140 con 0x7fb56015ad70 2023-10-16T12:53:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.061+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb560159330 con 0x7fb56015ad70 2023-10-16T12:53:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.062+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56015c940 con 0x7fb56015ad70 2023-10-16T12:53:56.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.063+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 73) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb5580045b0 con 0x7fb56015ad70 2023-10-16T12:53:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.064+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fb5580abf10 con 0x7fb56015ad70 2023-10-16T12:53:56.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.068+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb56015c940 con 0x7fb56015ad70 2023-10-16T12:53:56.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.117+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 74) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb558076b80 con 0x7fb56015ad70 2023-10-16T12:53:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.117+0000 7fb5577fe640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb5400705e0 0x7fb5400729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.119+0000 7fb55ed9d640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb5400705e0 0x7fb5400729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.124+0000 7fb55ed9d640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb5400705e0 0x7fb5400729d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb550009970 tx=0x7fb550008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.272+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb56004e160 con 0x7fb56015ad70 2023-10-16T12:53:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.273+0000 7fb5577fe640 1 -- 172.21.15.60:0/1997723330 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v145) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb5580a5000 con 0x7fb56015ad70 2023-10-16T12:53:56.276 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T12:53:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb5400705e0 msgr2=0x7fb5400729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb5400705e0 0x7fb5400729d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb550009970 tx=0x7fb550008040 comp rx=0 tx=0).stop 2023-10-16T12:53:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 msgr2=0x7fb56015cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 0x7fb56015cc20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb558009e50 tx=0x7fb558004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 shutdown_connections 2023-10-16T12:53:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb5400705e0 0x7fb5400729d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5600caac0 0x7fb560158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56015ad70 0x7fb56015cc20 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 --2- 172.21.15.60:0/1997723330 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5600c9ca0 0x7fb5600c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 >> 172.21.15.60:0/1997723330 conn(0x7fb5600ce9b0 msgr2=0x7fb5600afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 shutdown_connections 2023-10-16T12:53:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.276+0000 7fb565044640 1 -- 172.21.15.60:0/1997723330 wait complete. 2023-10-16T12:53:56.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:56.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.528+0000 7f82e1640640 1 Processor -- start 2023-10-16T12:53:56.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.528+0000 7f82e1640640 1 -- start start 2023-10-16T12:53:56.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82e1640640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82dc15b320 0x7f82dc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82e1640640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc15b9d0 con 0x7f82dc0ad970 2023-10-16T12:53:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82e1640640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc15bcb0 con 0x7f82dc06c040 2023-10-16T12:53:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82e1640640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82dc15b7d0 con 0x7f82dc15b320 2023-10-16T12:53:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82e0e3f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82dc15b320 0x7f82dc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82db7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82dc0ad970 0x7f82dc0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42326/0 (socket says 172.21.15.60:42326) 2023-10-16T12:53:56.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.530+0000 7f82db7fe640 1 -- 172.21.15.60:0/2530873733 learned_addr learned my addr 172.21.15.60:0/2530873733 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:56.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.531+0000 7f82daffd640 1 -- 172.21.15.60:0/2530873733 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419054841 0 0) 0x7f82dc15b9d0 con 0x7f82dc0ad970 2023-10-16T12:53:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.531+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2530873733 >> v1:172.21.15.144:6789/0 conn(0x7f82dc06c040 legacy=0x7f82dc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.531+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2530873733 >> v1:172.21.15.60:6789/0 conn(0x7f82dc0ad970 legacy=0x7f82dc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.531+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2530873733 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc15bf90 con 0x7f82dc15b320 2023-10-16T12:53:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.532+0000 7f82e0e3f640 1 --2- 172.21.15.60:0/2530873733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0cd180 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009dc0 tx=0x7f82d402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972b7b6ff2be64fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.532+0000 7f82daffd640 1 -- 172.21.15.60:0/2530873733 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d402cc40 con 0x7f82dc15b320 2023-10-16T12:53:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.532+0000 7f82daffd640 1 -- 172.21.15.60:0/2530873733 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d402cda0 con 0x7f82dc15b320 2023-10-16T12:53:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.532+0000 7f82daffd640 1 -- 172.21.15.60:0/2530873733 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4034660 con 0x7f82dc15b320 2023-10-16T12:53:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 -- 172.21.15.60:0/2530873733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 msgr2=0x7f82dc0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 --2- 172.21.15.60:0/2530873733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0cd180 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009dc0 tx=0x7f82d402be90 comp rx=0 tx=0).stop 2023-10-16T12:53:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 -- 172.21.15.60:0/2530873733 shutdown_connections 2023-10-16T12:53:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 --2- 172.21.15.60:0/2530873733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0cd180 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 -- 172.21.15.60:0/2530873733 >> 172.21.15.60:0/2530873733 conn(0x7f82dc0ce6a0 msgr2=0x7f82dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:56.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 -- 172.21.15.60:0/2530873733 shutdown_connections 2023-10-16T12:53:56.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.533+0000 7f82e1640640 1 -- 172.21.15.60:0/2530873733 wait complete. 2023-10-16T12:53:56.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.535+0000 7f82e1640640 1 Processor -- start 2023-10-16T12:53:56.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.535+0000 7f82e1640640 1 -- start start 2023-10-16T12:53:56.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e1640640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82dc06c040 0x7f82dc0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e1640640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82dc0ad970 0x7f82dc15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e1640640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e1640640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0b1bc0 con 0x7f82dc15b320 2023-10-16T12:53:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e1640640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0b1ea0 con 0x7f82dc0ad970 2023-10-16T12:53:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e1640640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0b2180 con 0x7f82dc06c040 2023-10-16T12:53:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e0e3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e0e3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57664/0 (socket says 172.21.15.60:57664) 2023-10-16T12:53:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.536+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2462633222 learned_addr learned my addr 172.21.15.60:0/2462633222 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82db7fe640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82dc0ad970 0x7f82dc15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82dbfff640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82dc06c040 0x7f82dc0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82dc06c040 msgr2=0x7f82dc0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82e0e3f640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82dc06c040 0x7f82dc0b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82dc0ad970 msgr2=0x7f82dc15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82e0e3f640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82dc0ad970 0x7f82dc15d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82e0e3f640 1 -- 172.21.15.60:0/2462633222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d4009a70 con 0x7f82dc15b320 2023-10-16T12:53:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82e0e3f640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0c8cf0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009ee0 tx=0x7f82d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d403f420 con 0x7f82dc15b320 2023-10-16T12:53:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d403f580 con 0x7f82dc15b320 2023-10-16T12:53:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.537+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d403f910 con 0x7f82dc15b320 2023-10-16T12:53:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.538+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc0b2460 con 0x7f82dc15b320 2023-10-16T12:53:56.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.538+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc15dde0 con 0x7f82dc15b320 2023-10-16T12:53:56.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.539+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82dc06b330 con 0x7f82dc15b320 2023-10-16T12:53:56.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.540+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 74) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f82d403fa70 con 0x7f82dc15b320 2023-10-16T12:53:56.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.541+0000 7f82d8ff9640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f82c806acc0 0x7f82c806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:56.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.541+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f82d40abff0 con 0x7f82dc15b320 2023-10-16T12:53:56.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.542+0000 7f82dbfff640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f82c806acc0 0x7f82c806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:56.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.543+0000 7f82dbfff640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f82c806acc0 0x7f82c806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00a750 tx=0x7f82cc008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:56.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.545+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82d40defe0 con 0x7f82dc15b320 2023-10-16T12:53:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.768+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82dc04e9b0 con 0x7f82dc15b320 2023-10-16T12:53:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.769+0000 7f82d8ff9640 1 -- 172.21.15.60:0/2462633222 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f82d4077050 con 0x7f82dc15b320 2023-10-16T12:53:56.772 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:56.772 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:53:26.782026+0000","modified":"2023-10-16T12:53:29.833032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7048},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7048":{"gid":7048,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/3032734613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3032734613},{"type":"v1","addr":"172.21.15.60:6834","nonce":3032734613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:53:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 70 2023-10-16T12:53:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f82c806acc0 msgr2=0x7f82c806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f82c806acc0 0x7f82c806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00a750 tx=0x7f82cc008040 comp rx=0 tx=0).stop 2023-10-16T12:53:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 msgr2=0x7f82dc0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0c8cf0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009ee0 tx=0x7f82d4004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 shutdown_connections 2023-10-16T12:53:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f82c806acc0 0x7f82c806d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82dc15b320 0x7f82dc0c8cf0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82dc0ad970 0x7f82dc15d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 --2- 172.21.15.60:0/2462633222 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82dc06c040 0x7f82dc0b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 >> 172.21.15.60:0/2462633222 conn(0x7f82dc0ce6a0 msgr2=0x7f82dc0cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.771+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 shutdown_connections 2023-10-16T12:53:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:56.772+0000 7f82e1640640 1 -- 172.21.15.60:0/2462633222 wait complete. 2023-10-16T12:53:56.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:57.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.013+0000 7fe0486b4640 1 Processor -- start 2023-10-16T12:53:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.013+0000 7fe0486b4640 1 -- start start 2023-10-16T12:53:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe0486b4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe04015aaa0 0x7fe04015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe0486b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0400ce390 0x7fe040070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe0486b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04015ed60 con 0x7fe04015ae80 2023-10-16T12:53:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe0486b4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04006b810 con 0x7fe04015aaa0 2023-10-16T12:53:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe0486b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0400707e0 con 0x7fe0400ce390 2023-10-16T12:53:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe046450640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe04015aaa0 0x7fe04015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe046450640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe04015aaa0 0x7fe04015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57674/0 (socket says 172.21.15.60:57674) 2023-10-16T12:53:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe046450640 1 -- 172.21.15.60:0/3311338459 learned_addr learned my addr 172.21.15.60:0/3311338459 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.015+0000 7fe046c51640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0400ce390 0x7fe040070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.016+0000 7fe046450640 1 -- 172.21.15.60:0/3311338459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0400ce390 msgr2=0x7fe040070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.016+0000 7fe046450640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0400ce390 0x7fe040070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.016+0000 7fe046450640 1 -- 172.21.15.60:0/3311338459 >> v1:172.21.15.60:6789/0 conn(0x7fe04015ae80 legacy=0x7fe0400cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.016+0000 7fe046450640 1 -- 172.21.15.60:0/3311338459 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04015ef90 con 0x7fe04015aaa0 2023-10-16T12:53:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.016+0000 7fe046c51640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0400ce390 0x7fe040070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.016+0000 7fe04544e640 1 -- 172.21.15.60:0/3311338459 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923751262 0 0) 0x7fe04015ed60 con 0x7fe04015ae80 2023-10-16T12:53:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.017+0000 7fe046450640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe04015aaa0 0x7fe04015e820 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009870 tx=0x7fe03c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9562244a6a2bb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.017+0000 7fe04544e640 1 -- 172.21.15.60:0/3311338459 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c030840 con 0x7fe04015aaa0 2023-10-16T12:53:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.017+0000 7fe04544e640 1 -- 172.21.15.60:0/3311338459 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c0309a0 con 0x7fe04015aaa0 2023-10-16T12:53:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.017+0000 7fe04544e640 1 -- 172.21.15.60:0/3311338459 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c030d30 con 0x7fe04015aaa0 2023-10-16T12:53:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 -- 172.21.15.60:0/3311338459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe04015aaa0 msgr2=0x7fe04015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe04015aaa0 0x7fe04015e820 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009870 tx=0x7fe03c02b980 comp rx=0 tx=0).stop 2023-10-16T12:53:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 -- 172.21.15.60:0/3311338459 shutdown_connections 2023-10-16T12:53:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0400ce390 0x7fe040070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 --2- 172.21.15.60:0/3311338459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe04015aaa0 0x7fe04015e820 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 -- 172.21.15.60:0/3311338459 >> 172.21.15.60:0/3311338459 conn(0x7fe0400cb410 msgr2=0x7fe0400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 -- 172.21.15.60:0/3311338459 shutdown_connections 2023-10-16T12:53:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.018+0000 7fe0486b4640 1 -- 172.21.15.60:0/3311338459 wait complete. 2023-10-16T12:53:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.020+0000 7fe0486b4640 1 Processor -- start 2023-10-16T12:53:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.020+0000 7fe0486b4640 1 -- start start 2023-10-16T12:53:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe0486b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 0x7fe040159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe0486b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe04015aaa0 0x7fe0400c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe0486b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe04015ae80 0x7fe04015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe0486b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0400c3750 con 0x7fe0400ce390 2023-10-16T12:53:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe0486b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe040159b60 con 0x7fe04015aaa0 2023-10-16T12:53:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe0486b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0400c3f20 con 0x7fe04015ae80 2023-10-16T12:53:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe046450640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 0x7fe040159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.021+0000 7fe046450640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 0x7fe040159620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57684/0 (socket says 172.21.15.60:57684) 2023-10-16T12:53:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 -- 172.21.15.60:0/4171881830 learned_addr learned my addr 172.21.15.60:0/4171881830 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe045c4f640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe04015aaa0 0x7fe0400c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046c51640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe04015ae80 0x7fe04015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 -- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe04015ae80 msgr2=0x7fe04015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe04015ae80 0x7fe04015d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 -- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe04015aaa0 msgr2=0x7fe0400c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe04015aaa0 0x7fe0400c39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 -- 172.21.15.60:0/4171881830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0094d0 con 0x7fe0400ce390 2023-10-16T12:53:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.022+0000 7fe046450640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 0x7fe040159620 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009870 tx=0x7fe03c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.023+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0307f0 con 0x7fe0400ce390 2023-10-16T12:53:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.023+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c043400 con 0x7fe0400ce390 2023-10-16T12:53:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.023+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0437b0 con 0x7fe0400ce390 2023-10-16T12:53:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.023+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0401f2930 con 0x7fe0400ce390 2023-10-16T12:53:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.023+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0401f2ef0 con 0x7fe0400ce390 2023-10-16T12:53:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.024+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0400c9740 con 0x7fe0400ce390 2023-10-16T12:53:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.025+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 74) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe03c043910 con 0x7fe0400ce390 2023-10-16T12:53:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.026+0000 7fe036ffd640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe02406acc0 0x7fe02406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.026+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fe03c03e070 con 0x7fe0400ce390 2023-10-16T12:53:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.027+0000 7fe045c4f640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe02406acc0 0x7fe02406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.028+0000 7fe045c4f640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe02406acc0 0x7fe02406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe030007fd0 tx=0x7fe030009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.030+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe03c0778c0 con 0x7fe0400ce390 2023-10-16T12:53:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.130+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 75) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe03c077c90 con 0x7fe0400ce390 2023-10-16T12:53:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.235+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe04015f520 con 0x7fe0400ce390 2023-10-16T12:53:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.237+0000 7fe036ffd640 1 -- 172.21.15.60:0/4171881830 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+17590 (secure 0 0 0) 0x7fe03c07c8b0 con 0x7fe0400ce390 2023-10-16T12:53:57.240 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:57.241 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":145,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:55.095667+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3844965859":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/3473219507":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/119007135":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/1667410565":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/285791614":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6833/1139477310":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6832/1139477310":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/1161466656":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6833/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:6832/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/690478556":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/177518790":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2812310974":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1140080778":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1196951169":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2229533001":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/325928307":"2023-10-17T12:53:41.484587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe02406acc0 msgr2=0x7fe02406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe02406acc0 0x7fe02406d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe030007fd0 tx=0x7fe030009040 comp rx=0 tx=0).stop 2023-10-16T12:53:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 msgr2=0x7fe040159620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 0x7fe040159620 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009870 tx=0x7fe03c004730 comp rx=0 tx=0).stop 2023-10-16T12:53:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 shutdown_connections 2023-10-16T12:53:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe02406acc0 0x7fe02406d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe04015ae80 0x7fe04015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe04015aaa0 0x7fe0400c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 --2- 172.21.15.60:0/4171881830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0400ce390 0x7fe040159620 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 >> 172.21.15.60:0/4171881830 conn(0x7fe0400cb410 msgr2=0x7fe0400b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 shutdown_connections 2023-10-16T12:53:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.240+0000 7fe0486b4640 1 -- 172.21.15.60:0/4171881830 wait complete. 2023-10-16T12:53:57.262 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:53:57.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.479+0000 7f77a60d8640 1 Processor -- start 2023-10-16T12:53:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.479+0000 7f77a60d8640 1 -- start start 2023-10-16T12:53:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a60d8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f77a006c040 0x7f77a006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a60d8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77a00cda80 0x7f77a015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a60d8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a006c960 con 0x7f77a00ad970 2023-10-16T12:53:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a60d8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a015b9d0 con 0x7f77a00cda80 2023-10-16T12:53:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a60d8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a015bcb0 con 0x7f77a006c040 2023-10-16T12:53:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a58d7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77a00cda80 0x7f77a015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a58d7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77a00cda80 0x7f77a015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57700/0 (socket says 172.21.15.60:57700) 2023-10-16T12:53:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.481+0000 7f77a58d7640 1 -- 172.21.15.60:0/4205481251 learned_addr learned my addr 172.21.15.60:0/4205481251 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:57.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.482+0000 7f77a50d6640 1 --2- 172.21.15.60:0/4205481251 >> v2:172.21.15.144:3300/0 conn(0x7f77a006c040 0x7f77a006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.482+0000 7f77a58d7640 1 -- 172.21.15.60:0/4205481251 >> v2:172.21.15.144:3300/0 conn(0x7f77a006c040 msgr2=0x7f77a006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.482+0000 7f77a58d7640 1 --2- 172.21.15.60:0/4205481251 >> v2:172.21.15.144:3300/0 conn(0x7f77a006c040 0x7f77a006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.482+0000 7f77a58d7640 1 -- 172.21.15.60:0/4205481251 >> v1:172.21.15.60:6789/0 conn(0x7f77a00ad970 legacy=0x7f77a015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.482+0000 7f77a58d7640 1 -- 172.21.15.60:0/4205481251 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a015bf90 con 0x7f77a00cda80 2023-10-16T12:53:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.482+0000 7f77a50d6640 1 --2- 172.21.15.60:0/4205481251 >> v2:172.21.15.144:3300/0 conn(0x7f77a006c040 0x7f77a006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.483+0000 7f77a58d7640 1 --2- 172.21.15.60:0/4205481251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a015ed90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f77a00cf350 tx=0x7f779802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b817797139f0f955 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.483+0000 7f7797fff640 1 -- 172.21.15.60:0/4205481251 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779802cb10 con 0x7f77a00cda80 2023-10-16T12:53:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.483+0000 7f7797fff640 1 -- 172.21.15.60:0/4205481251 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779802cc70 con 0x7f77a00cda80 2023-10-16T12:53:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.483+0000 7f7797fff640 1 -- 172.21.15.60:0/4205481251 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77980343f0 con 0x7f77a00cda80 2023-10-16T12:53:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 -- 172.21.15.60:0/4205481251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 msgr2=0x7f77a015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 --2- 172.21.15.60:0/4205481251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a015ed90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f77a00cf350 tx=0x7f779802bf70 comp rx=0 tx=0).stop 2023-10-16T12:53:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 -- 172.21.15.60:0/4205481251 shutdown_connections 2023-10-16T12:53:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 --2- 172.21.15.60:0/4205481251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a015ed90 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 --2- 172.21.15.60:0/4205481251 >> v2:172.21.15.144:3300/0 conn(0x7f77a006c040 0x7f77a006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 -- 172.21.15.60:0/4205481251 >> 172.21.15.60:0/4205481251 conn(0x7f77a00ce6a0 msgr2=0x7f77a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 -- 172.21.15.60:0/4205481251 shutdown_connections 2023-10-16T12:53:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.484+0000 7f77a60d8640 1 -- 172.21.15.60:0/4205481251 wait complete. 2023-10-16T12:53:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.486+0000 7f77a60d8640 1 Processor -- start 2023-10-16T12:53:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.486+0000 7f77a60d8640 1 -- start start 2023-10-16T12:53:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a60d8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77a006c040 0x7f77a01efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a60d8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77a00ad970 0x7f77a00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a60d8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a00b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a60d8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a00c6d20 con 0x7f77a00cda80 2023-10-16T12:53:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a60d8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a00c6680 con 0x7f77a006c040 2023-10-16T12:53:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a60d8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a00c60c0 con 0x7f77a00ad970 2023-10-16T12:53:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a58d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a00b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a58d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a00b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57702/0 (socket says 172.21.15.60:57702) 2023-10-16T12:53:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.487+0000 7f77a58d7640 1 -- 172.21.15.60:0/2627454419 learned_addr learned my addr 172.21.15.60:0/2627454419 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a58d7640 1 -- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77a00ad970 msgr2=0x7f77a00c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a50d6640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77a006c040 0x7f77a01efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a48d5640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77a00ad970 0x7f77a00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a58d7640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77a00ad970 0x7f77a00c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a58d7640 1 -- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77a006c040 msgr2=0x7f77a01efe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a58d7640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77a006c040 0x7f77a01efe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a58d7640 1 -- 172.21.15.60:0/2627454419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798009b30 con 0x7f77a00cda80 2023-10-16T12:53:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a48d5640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77a00ad970 0x7f77a00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a50d6640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77a006c040 0x7f77a01efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a58d7640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a00b26a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f7798036040 tx=0x7f7798004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779803e450 con 0x7f77a00cda80 2023-10-16T12:53:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779803e5d0 con 0x7f77a00cda80 2023-10-16T12:53:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779803e980 con 0x7f77a00cda80 2023-10-16T12:53:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.488+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a00c63a0 con 0x7f77a00cda80 2023-10-16T12:53:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.489+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a01f0350 con 0x7f77a00cda80 2023-10-16T12:53:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.489+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a006b330 con 0x7f77a00cda80 2023-10-16T12:53:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.490+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 75) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f779803eae0 con 0x7f77a00cda80 2023-10-16T12:53:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.492+0000 7f7795ffb640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f778406af00 0x7f778406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.492+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f77980ac950 con 0x7f77a00cda80 2023-10-16T12:53:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.492+0000 7f77a50d6640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f778406af00 0x7f778406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.493+0000 7f77a50d6640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f778406af00 0x7f778406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.496+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7798077630 con 0x7f77a00cda80 2023-10-16T12:53:57.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.724+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77a004e9b0 con 0x7f77a00cda80 2023-10-16T12:53:57.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.725+0000 7f7795ffb640 1 -- 172.21.15.60:0/2627454419 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f779807ac50 con 0x7f77a00cda80 2023-10-16T12:53:57.728 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:57.728 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:53:26.782026+0000","modified":"2023-10-16T12:53:29.833032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7048},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7048":{"gid":7048,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/3032734613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3032734613},{"type":"v1","addr":"172.21.15.60:6834","nonce":3032734613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:53:57.728 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 70 2023-10-16T12:53:57.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.727+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f778406af00 msgr2=0x7f778406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f778406af00 0x7f778406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c008040 comp rx=0 tx=0).stop 2023-10-16T12:53:57.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 msgr2=0x7f77a00b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a00b26a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f7798036040 tx=0x7f7798004a40 comp rx=0 tx=0).stop 2023-10-16T12:53:57.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 shutdown_connections 2023-10-16T12:53:57.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f778406af00 0x7f778406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77a00ad970 0x7f77a00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77a00cda80 0x7f77a00b26a0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 --2- 172.21.15.60:0/2627454419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77a006c040 0x7f77a01efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 >> 172.21.15.60:0/2627454419 conn(0x7f77a00ce6a0 msgr2=0x7f77a015d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:57.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 shutdown_connections 2023-10-16T12:53:57.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.728+0000 7f77a60d8640 1 -- 172.21.15.60:0/2627454419 wait complete. 2023-10-16T12:53:57.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:53:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.969+0000 7fd89b527640 1 Processor -- start 2023-10-16T12:53:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.969+0000 7fd89b527640 1 -- start start 2023-10-16T12:53:57.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.971+0000 7fd89b527640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd89406c040 0x7fd89406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.971+0000 7fd89b527640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 0x7fd89415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.971+0000 7fd89b527640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89406c960 con 0x7fd89415ebc0 2023-10-16T12:53:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.971+0000 7fd89b527640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89415b9d0 con 0x7fd89406c040 2023-10-16T12:53:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.971+0000 7fd89b527640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89415bcb0 con 0x7fd8940ad970 2023-10-16T12:53:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.972+0000 7fd899d24640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 0x7fd89415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.972+0000 7fd899d24640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 0x7fd89415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:36826/0 (socket says 172.21.15.60:36826) 2023-10-16T12:53:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.972+0000 7fd899d24640 1 -- 172.21.15.60:0/922393877 learned_addr learned my addr 172.21.15.60:0/922393877 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.972+0000 7fd89a525640 1 --2- 172.21.15.60:0/922393877 >> v2:172.21.15.144:3300/0 conn(0x7fd89406c040 0x7fd89406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.973+0000 7fd899d24640 1 -- 172.21.15.60:0/922393877 >> v2:172.21.15.144:3300/0 conn(0x7fd89406c040 msgr2=0x7fd89406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.973+0000 7fd899d24640 1 --2- 172.21.15.60:0/922393877 >> v2:172.21.15.144:3300/0 conn(0x7fd89406c040 0x7fd89406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.973+0000 7fd899d24640 1 -- 172.21.15.60:0/922393877 >> v1:172.21.15.144:6789/0 conn(0x7fd89415ebc0 legacy=0x7fd8940cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.973+0000 7fd899d24640 1 -- 172.21.15.60:0/922393877 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89415bf90 con 0x7fd8940ad970 2023-10-16T12:53:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.974+0000 7fd899d24640 1 --2- 172.21.15.60:0/922393877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 0x7fd89415e680 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8840098a0 tx=0x7fd88402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff7bd9d2ba47c831 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd899523640 1 -- 172.21.15.60:0/922393877 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884030840 con 0x7fd8940ad970 2023-10-16T12:53:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd899523640 1 -- 172.21.15.60:0/922393877 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8840309a0 con 0x7fd8940ad970 2023-10-16T12:53:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd899523640 1 -- 172.21.15.60:0/922393877 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884030d30 con 0x7fd8940ad970 2023-10-16T12:53:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 -- 172.21.15.60:0/922393877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 msgr2=0x7fd89415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 --2- 172.21.15.60:0/922393877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 0x7fd89415e680 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8840098a0 tx=0x7fd88402b980 comp rx=0 tx=0).stop 2023-10-16T12:53:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 -- 172.21.15.60:0/922393877 shutdown_connections 2023-10-16T12:53:57.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 --2- 172.21.15.60:0/922393877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8940ad970 0x7fd89415e680 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 --2- 172.21.15.60:0/922393877 >> v2:172.21.15.144:3300/0 conn(0x7fd89406c040 0x7fd89406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 -- 172.21.15.60:0/922393877 >> 172.21.15.60:0/922393877 conn(0x7fd8940ce6a0 msgr2=0x7fd8940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:57.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 -- 172.21.15.60:0/922393877 shutdown_connections 2023-10-16T12:53:57.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.975+0000 7fd89b527640 1 -- 172.21.15.60:0/922393877 wait complete. 2023-10-16T12:53:57.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.977+0000 7fd89b527640 1 Processor -- start 2023-10-16T12:53:57.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.977+0000 7fd89b527640 1 -- start start 2023-10-16T12:53:57.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.978+0000 7fd89b527640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89406c040 0x7fd8940aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.978+0000 7fd89b527640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 0x7fd894153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.978+0000 7fd89b527640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89415ebc0 0x7fd8940b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.978+0000 7fd89b527640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940ae260 con 0x7fd8940ad970 2023-10-16T12:53:57.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.978+0000 7fd89b527640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940b0c20 con 0x7fd89406c040 2023-10-16T12:53:57.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.978+0000 7fd89b527640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940b0ec0 con 0x7fd89415ebc0 2023-10-16T12:53:57.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 0x7fd894153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 0x7fd894153d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57704/0 (socket says 172.21.15.60:57704) 2023-10-16T12:53:57.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 -- 172.21.15.60:0/188289042 learned_addr learned my addr 172.21.15.60:0/188289042 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:57.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd89ad26640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89415ebc0 0x7fd8940b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd89a525640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89406c040 0x7fd8940aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 -- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89415ebc0 msgr2=0x7fd8940b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89415ebc0 0x7fd8940b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 -- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89406c040 msgr2=0x7fd8940aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:57.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89406c040 0x7fd8940aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:57.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.979+0000 7fd899d24640 1 -- 172.21.15.60:0/188289042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8840094d0 con 0x7fd8940ad970 2023-10-16T12:53:57.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.980+0000 7fd899d24640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 0x7fd894153d30 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd884000c00 tx=0x7fd884004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.980+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884039a90 con 0x7fd8940ad970 2023-10-16T12:53:57.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.980+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884039bf0 con 0x7fd8940ad970 2023-10-16T12:53:57.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.980+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884043410 con 0x7fd8940ad970 2023-10-16T12:53:57.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.980+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8940ae500 con 0x7fd8940ad970 2023-10-16T12:53:57.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.980+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8940aeac0 con 0x7fd8940ad970 2023-10-16T12:53:57.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.981+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89406b330 con 0x7fd8940ad970 2023-10-16T12:53:57.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.982+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 75) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd88402f390 con 0x7fd8940ad970 2023-10-16T12:53:57.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.984+0000 7fd88affd640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd87806af00 0x7fd87806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:57.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.984+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd8840abbc0 con 0x7fd8940ad970 2023-10-16T12:53:57.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.984+0000 7fd89a525640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd87806af00 0x7fd87806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:57.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.985+0000 7fd89a525640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd87806af00 0x7fd87806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8900073f0 tx=0x7fd89000b040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:57.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:57.989+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd884076920 con 0x7fd8940ad970 2023-10-16T12:53:58.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.224+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd89404e9b0 con 0x7fd8940ad970 2023-10-16T12:53:58.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.225+0000 7fd88affd640 1 -- 172.21.15.60:0/188289042 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd884034090 con 0x7fd8940ad970 2023-10-16T12:53:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:58.228 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:53:26.782026+0000","modified":"2023-10-16T12:53:29.833032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7048},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7048":{"gid":7048,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/3032734613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3032734613},{"type":"v1","addr":"172.21.15.60:6834","nonce":3032734613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:53:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 70 2023-10-16T12:53:58.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd87806af00 msgr2=0x7fd87806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd87806af00 0x7fd87806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8900073f0 tx=0x7fd89000b040 comp rx=0 tx=0).stop 2023-10-16T12:53:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 msgr2=0x7fd894153d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 0x7fd894153d30 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd884000c00 tx=0x7fd884004730 comp rx=0 tx=0).stop 2023-10-16T12:53:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 shutdown_connections 2023-10-16T12:53:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd87806af00 0x7fd87806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89415ebc0 0x7fd8940b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8940ad970 0x7fd894153d30 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 --2- 172.21.15.60:0/188289042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89406c040 0x7fd8940aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 >> 172.21.15.60:0/188289042 conn(0x7fd8940ce6a0 msgr2=0x7fd8940ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:58.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 shutdown_connections 2023-10-16T12:53:58.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.228+0000 7fd89b527640 1 -- 172.21.15.60:0/188289042 wait complete. 2023-10-16T12:53:58.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:53:58.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.482+0000 7fd534e4d640 1 Processor -- start 2023-10-16T12:53:58.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.482+0000 7fd534e4d640 1 -- start start 2023-10-16T12:53:58.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd534e4d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd53015c9f0 0x7fd5300c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:58.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd534e4d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd530160a90 con 0x7fd53015dde0 2023-10-16T12:53:58.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd534e4d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300cd900 con 0x7fd5300c9230 2023-10-16T12:53:58.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd534e4d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300cdec0 con 0x7fd53015c9f0 2023-10-16T12:53:58.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd52f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd53015c9f0 0x7fd5300c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:58.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd52ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd53015dde0 0x7fd5300ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42360/0 (socket says 172.21.15.60:42360) 2023-10-16T12:53:58.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.484+0000 7fd52ffff640 1 -- 172.21.15.60:0/974137030 learned_addr learned my addr 172.21.15.60:0/974137030 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:58.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.485+0000 7fd52effd640 1 -- 172.21.15.60:0/974137030 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803145001 0 0) 0x7fd530160a90 con 0x7fd53015dde0 2023-10-16T12:53:58.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.485+0000 7fd52effd640 1 -- 172.21.15.60:0/974137030 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c003520 con 0x7fd53015dde0 2023-10-16T12:53:58.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.485+0000 7fd52f7fe640 1 -- 172.21.15.60:0/974137030 >> v1:172.21.15.144:6789/0 conn(0x7fd5300c9230 legacy=0x7fd5300709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.485+0000 7fd52f7fe640 1 -- 172.21.15.60:0/974137030 >> v1:172.21.15.60:6789/0 conn(0x7fd53015dde0 legacy=0x7fd5300ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.485+0000 7fd52f7fe640 1 -- 172.21.15.60:0/974137030 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300cdbe0 con 0x7fd53015c9f0 2023-10-16T12:53:58.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.486+0000 7fd52f7fe640 1 --2- 172.21.15.60:0/974137030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5300c8cf0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd5200097a0 tx=0x7fd52002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fc8bef6e9d5c6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.486+0000 7fd52effd640 1 -- 172.21.15.60:0/974137030 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520030610 con 0x7fd53015c9f0 2023-10-16T12:53:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.486+0000 7fd52effd640 1 -- 172.21.15.60:0/974137030 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd520030770 con 0x7fd53015c9f0 2023-10-16T12:53:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd52effd640 1 -- 172.21.15.60:0/974137030 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520030ae0 con 0x7fd53015c9f0 2023-10-16T12:53:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 -- 172.21.15.60:0/974137030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 msgr2=0x7fd5300c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 --2- 172.21.15.60:0/974137030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5300c8cf0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd5200097a0 tx=0x7fd52002b960 comp rx=0 tx=0).stop 2023-10-16T12:53:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 -- 172.21.15.60:0/974137030 shutdown_connections 2023-10-16T12:53:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 --2- 172.21.15.60:0/974137030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5300c8cf0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 -- 172.21.15.60:0/974137030 >> 172.21.15.60:0/974137030 conn(0x7fd5300ce6a0 msgr2=0x7fd5300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 -- 172.21.15.60:0/974137030 shutdown_connections 2023-10-16T12:53:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.487+0000 7fd534e4d640 1 -- 172.21.15.60:0/974137030 wait complete. 2023-10-16T12:53:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.489+0000 7fd534e4d640 1 Processor -- start 2023-10-16T12:53:58.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.489+0000 7fd534e4d640 1 -- start start 2023-10-16T12:53:58.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.490+0000 7fd534e4d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5300c9230 0x7fd53006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:58.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.490+0000 7fd534e4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5301f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:58.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.490+0000 7fd534e4d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd53015dde0 0x7fd5300c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:58.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.490+0000 7fd534e4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c7550 con 0x7fd53015c9f0 2023-10-16T12:53:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.490+0000 7fd534e4d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c66c0 con 0x7fd53015dde0 2023-10-16T12:53:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.490+0000 7fd534e4d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c6940 con 0x7fd5300c9230 2023-10-16T12:53:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5301f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5301f5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57724/0 (socket says 172.21.15.60:57724) 2023-10-16T12:53:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 -- 172.21.15.60:0/4088920765 learned_addr learned my addr 172.21.15.60:0/4088920765 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 -- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5300c9230 msgr2=0x7fd53006bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5300c9230 0x7fd53006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 -- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd53015dde0 msgr2=0x7fd5300c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd53015dde0 0x7fd5300c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 -- 172.21.15.60:0/4088920765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520009400 con 0x7fd53015c9f0 2023-10-16T12:53:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.491+0000 7fd527fff640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5301f5290 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c430 tx=0x7fd51800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.492+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180052d0 con 0x7fd53015c9f0 2023-10-16T12:53:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.492+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518005450 con 0x7fd53015c9f0 2023-10-16T12:53:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.492+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518007260 con 0x7fd53015c9f0 2023-10-16T12:53:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.492+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5300c6c20 con 0x7fd53015c9f0 2023-10-16T12:53:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.492+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5300c71e0 con 0x7fd53015c9f0 2023-10-16T12:53:58.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.493+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5300c73a0 con 0x7fd53015c9f0 2023-10-16T12:53:58.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.494+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 75) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd5180073c0 con 0x7fd53015c9f0 2023-10-16T12:53:58.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.495+0000 7fd52cff9640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd51006af00 0x7fd51006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:58.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.495+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd518088ef0 con 0x7fd53015c9f0 2023-10-16T12:53:58.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.496+0000 7fd52f7fe640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd51006af00 0x7fd51006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:58.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.497+0000 7fd52f7fe640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd51006af00 0x7fd51006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd520036000 tx=0x7fd520030000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:58.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.500+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd518053b40 con 0x7fd53015c9f0 2023-10-16T12:53:58.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:58.549 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.699+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd53004e9b0 con 0x7fd53015c9f0 2023-10-16T12:53:58.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.702+0000 7fd52cff9640 1 -- 172.21.15.60:0/4088920765 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+17590 (secure 0 0 0) 0x7fd518057160 con 0x7fd53015c9f0 2023-10-16T12:53:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:53:58.706 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":145,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:53:55.095667+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:53:24.764434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-16T12:53:25.816764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3844965859":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/3473219507":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/119007135":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/1667410565":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/285791614":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6833/1139477310":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6832/1139477310":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/1161466656":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6833/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:6832/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/690478556":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/177518790":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2812310974":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1140080778":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1196951169":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2229533001":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/325928307":"2023-10-17T12:53:41.484587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:53:58.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd51006af00 msgr2=0x7fd51006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd51006af00 0x7fd51006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd520036000 tx=0x7fd520030000 comp rx=0 tx=0).stop 2023-10-16T12:53:58.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 msgr2=0x7fd5301f5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:58.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5301f5290 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c430 tx=0x7fd51800c8f0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 shutdown_connections 2023-10-16T12:53:58.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd51006af00 0x7fd51006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd53015dde0 0x7fd5300c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd53015c9f0 0x7fd5301f5290 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 --2- 172.21.15.60:0/4088920765 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5300c9230 0x7fd53006bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:58.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.705+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 >> 172.21.15.60:0/4088920765 conn(0x7fd5300ce6a0 msgr2=0x7fd53015a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:58.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.706+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 shutdown_connections 2023-10-16T12:53:58.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.706+0000 7fd534e4d640 1 -- 172.21.15.60:0/4088920765 wait complete. 2023-10-16T12:53:58.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:53:59.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.998+0000 7fbe2ac60640 1 Processor -- start 2023-10-16T12:53:59.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:58.998+0000 7fbe2ac60640 1 -- start start 2023-10-16T12:53:59.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.000+0000 7fbe2ac60640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbe2415b320 0x7fbe240cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.000+0000 7fbe2ac60640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2415b9d0 con 0x7fbe240ad970 2023-10-16T12:53:59.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.000+0000 7fbe2ac60640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2415bcb0 con 0x7fbe2406c040 2023-10-16T12:53:59.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.000+0000 7fbe2ac60640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2415b7d0 con 0x7fbe2415b320 2023-10-16T12:53:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.000+0000 7fbe2945d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbe240ad970 0x7fbe240cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34000/0 (socket says 172.21.15.60:34000) 2023-10-16T12:53:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.000+0000 7fbe2945d640 1 -- 172.21.15.60:0/4041391723 learned_addr learned my addr 172.21.15.60:0/4041391723 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:53:59.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe2a45f640 1 --2- 172.21.15.60:0/4041391723 >> v2:172.21.15.144:3300/0 conn(0x7fbe2415b320 0x7fbe240cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50432715 0 0) 0x7fbe2415b9d0 con 0x7fbe240ad970 2023-10-16T12:53:59.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe10003520 con 0x7fbe240ad970 2023-10-16T12:53:59.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2564743229 0 0) 0x7fbe10003520 con 0x7fbe240ad970 2023-10-16T12:53:59.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 >> v2:172.21.15.144:3300/0 conn(0x7fbe2415b320 msgr2=0x7fbe240cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 --2- 172.21.15.60:0/4041391723 >> v2:172.21.15.144:3300/0 conn(0x7fbe2415b320 0x7fbe240cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 >> v1:172.21.15.144:6789/0 conn(0x7fbe2406c040 legacy=0x7fbe2406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2415bf90 con 0x7fbe240ad970 2023-10-16T12:53:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.001+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fbe14001670 con 0x7fbe240ad970 2023-10-16T12:53:59.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 >> v1:172.21.15.60:6789/0 conn(0x7fbe240ad970 legacy=0x7fbe240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe100039a0 0x7fbe10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100047c0 0x7fbe10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe10005180 0x7fbe1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2415b7d0 con 0x7fbe100039a0 2023-10-16T12:53:59.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1000c050 con 0x7fbe10005180 2023-10-16T12:53:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1000c210 con 0x7fbe100047c0 2023-10-16T12:53:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe100039a0 0x7fbe10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe2945d640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100047c0 0x7fbe10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe29c5e640 1 -- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100047c0 msgr2=0x7fbe10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:59.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe2a45f640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe10005180 0x7fbe1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100047c0 0x7fbe10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe29c5e640 1 -- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe10005180 msgr2=0x7fbe1000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.002+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe10005180 0x7fbe1000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.003+0000 7fbe29c5e640 1 -- 172.21.15.60:0/4041391723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10003520 con 0x7fbe100039a0 2023-10-16T12:53:59.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.003+0000 7fbe2945d640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100047c0 0x7fbe10004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:59.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.003+0000 7fbe2a45f640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe10005180 0x7fbe1000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:59.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.003+0000 7fbe2a45f640 1 -- 172.21.15.60:0/4041391723 reap_dead start 2023-10-16T12:53:59.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.004+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe100039a0 0x7fbe10003dc0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fbe20001750 tx=0x7fbe100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eda9371c4a25641 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:59.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.004+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe200052d0 con 0x7fbe100039a0 2023-10-16T12:53:59.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.004+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2002e050 con 0x7fbe100039a0 2023-10-16T12:53:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.004+0000 7fbe28c5c640 1 -- 172.21.15.60:0/4041391723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20003830 con 0x7fbe100039a0 2023-10-16T12:53:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.004+0000 7fbe2ac60640 1 -- 172.21.15.60:0/4041391723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe100039a0 msgr2=0x7fbe10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.004+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe100039a0 0x7fbe10003dc0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fbe20001750 tx=0x7fbe100057f0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.005+0000 7fbe2ac60640 1 -- 172.21.15.60:0/4041391723 shutdown_connections 2023-10-16T12:53:59.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.005+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/4041391723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe100039a0 0x7fbe10003dc0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.005+0000 7fbe2ac60640 1 -- 172.21.15.60:0/4041391723 >> 172.21.15.60:0/4041391723 conn(0x7fbe240ce6a0 msgr2=0x7fbe240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.005+0000 7fbe2ac60640 1 -- 172.21.15.60:0/4041391723 shutdown_connections 2023-10-16T12:53:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.005+0000 7fbe2ac60640 1 -- 172.21.15.60:0/4041391723 wait complete. 2023-10-16T12:53:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.007+0000 7fbe2ac60640 1 Processor -- start 2023-10-16T12:53:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.007+0000 7fbe2ac60640 1 -- start start 2023-10-16T12:53:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2ac60640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100039a0 0x7fbe240b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2ac60640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe24159200 0x7fbe2415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2ac60640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 0x7fbe241f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2ac60640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe241f4290 con 0x7fbe241f38d0 2023-10-16T12:53:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2ac60640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe241f1540 con 0x7fbe24159200 2023-10-16T12:53:59.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2ac60640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe240b40a0 con 0x7fbe100039a0 2023-10-16T12:53:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 0x7fbe241f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 0x7fbe241f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57868/0 (socket says 172.21.15.60:57868) 2023-10-16T12:53:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 -- 172.21.15.60:0/3569325988 learned_addr learned my addr 172.21.15.60:0/3569325988 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100039a0 0x7fbe240b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 -- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100039a0 msgr2=0x7fbe240b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100039a0 0x7fbe240b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 -- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe24159200 msgr2=0x7fbe2415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2945d640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe24159200 0x7fbe2415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.008+0000 7fbe2a45f640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe24159200 0x7fbe2415ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe2a45f640 1 -- 172.21.15.60:0/3569325988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2415d140 con 0x7fbe241f38d0 2023-10-16T12:53:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe2945d640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe24159200 0x7fbe2415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe2a45f640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 0x7fbe241f3d50 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fbe2415b700 tx=0x7fbe1c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c016420 con 0x7fbe241f38d0 2023-10-16T12:53:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c016580 con 0x7fbe241f38d0 2023-10-16T12:53:59.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c0168f0 con 0x7fbe241f38d0 2023-10-16T12:53:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2415d420 con 0x7fbe241f38d0 2023-10-16T12:53:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.009+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2415d700 con 0x7fbe241f38d0 2023-10-16T12:53:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.011+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2406b2e0 con 0x7fbe241f38d0 2023-10-16T12:53:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.012+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 75) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbe1c016a50 con 0x7fbe241f38d0 2023-10-16T12:53:59.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.012+0000 7fbe1a7fc640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe0806af30 0x7fbe0806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.013+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fbe1c08a880 con 0x7fbe241f38d0 2023-10-16T12:53:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.013+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe0806af30 0x7fbe0806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.014+0000 7fbe29c5e640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe0806af30 0x7fbe0806d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbe20033000 tx=0x7fbe20009030 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:59.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.017+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe1c055550 con 0x7fbe241f38d0 2023-10-16T12:53:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.152+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe1c055920 con 0x7fbe241f38d0 2023-10-16T12:53:59.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.260+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbe2404e9b0 con 0x7fbe241f38d0 2023-10-16T12:53:59.265 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T12:53:59.261+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:53:59.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.264+0000 7fbe1a7fc640 1 -- 172.21.15.60:0/3569325988 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbe1c088620 con 0x7fbe241f38d0 2023-10-16T12:53:59.267 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:53:59.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe0806af30 msgr2=0x7fbe0806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe0806af30 0x7fbe0806d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbe20033000 tx=0x7fbe20009030 comp rx=0 tx=0).stop 2023-10-16T12:53:59.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 msgr2=0x7fbe241f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 0x7fbe241f3d50 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fbe2415b700 tx=0x7fbe1c00f520 comp rx=0 tx=0).stop 2023-10-16T12:53:59.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 shutdown_connections 2023-10-16T12:53:59.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe0806af30 0x7fbe0806d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe241f38d0 0x7fbe241f3d50 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe24159200 0x7fbe2415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 --2- 172.21.15.60:0/3569325988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe100039a0 0x7fbe240b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.268+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 >> 172.21.15.60:0/3569325988 conn(0x7fbe240ce6a0 msgr2=0x7fbe240aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:59.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.269+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 shutdown_connections 2023-10-16T12:53:59.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.269+0000 7fbe2ac60640 1 -- 172.21.15.60:0/3569325988 wait complete. 2023-10-16T12:53:59.290 INFO:tasks.ceph.mds.a.smithi060.stderr: -8622> 2023-10-16T12:53:31.903+0000 7f4a46dfa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:53:59.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:53:59.359 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:53:59.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.491+0000 7f048f59e640 1 Processor -- start 2023-10-16T12:53:59.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.491+0000 7f048f59e640 1 -- start start 2023-10-16T12:53:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.493+0000 7f048f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f049006c9e0 0x7f04900ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.493+0000 7f048f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900ca560 con 0x7f049006fb50 2023-10-16T12:53:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.493+0000 7f048f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900c9950 con 0x7f04900c9060 2023-10-16T12:53:59.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.493+0000 7f048f59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04900c9c30 con 0x7f049006c9e0 2023-10-16T12:53:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.493+0000 7f048ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f049006c9e0 0x7f04900ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f049006c9e0 0x7f04900ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57888/0 (socket says 172.21.15.60:57888) 2023-10-16T12:53:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048ed9d640 1 -- 172.21.15.60:0/540122658 learned_addr learned my addr 172.21.15.60:0/540122658 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:59.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048dd9b640 1 --1- 172.21.15.60:0/540122658 >> v1:172.21.15.60:6789/0 conn(0x7f049006fb50 0x7f04900cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34002/0 (socket says 172.21.15.60:34002) 2023-10-16T12:53:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048ed9d640 1 -- 172.21.15.60:0/540122658 >> v1:172.21.15.144:6789/0 conn(0x7f04900c9060 legacy=0x7f049006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:53:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048ed9d640 1 -- 172.21.15.60:0/540122658 >> v1:172.21.15.60:6789/0 conn(0x7f049006fb50 legacy=0x7f04900cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048ed9d640 1 -- 172.21.15.60:0/540122658 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04900c9f10 con 0x7f049006c9e0 2023-10-16T12:53:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.494+0000 7f048d59a640 1 -- 172.21.15.60:0/540122658 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071760146 0 0) 0x7f04900ca560 con 0x7f049006fb50 2023-10-16T12:53:59.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.495+0000 7f048ed9d640 1 --2- 172.21.15.60:0/540122658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049006c9e0 0x7f04900ccfb0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f0488009e50 tx=0x7f048802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49eeaaae0055b2bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048d59a640 1 -- 172.21.15.60:0/540122658 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048802cb90 con 0x7f049006c9e0 2023-10-16T12:53:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048d59a640 1 -- 172.21.15.60:0/540122658 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048802ccf0 con 0x7f049006c9e0 2023-10-16T12:53:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048d59a640 1 -- 172.21.15.60:0/540122658 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488034610 con 0x7f049006c9e0 2023-10-16T12:53:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048f59e640 1 -- 172.21.15.60:0/540122658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049006c9e0 msgr2=0x7f04900ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048f59e640 1 --2- 172.21.15.60:0/540122658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049006c9e0 0x7f04900ccfb0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f0488009e50 tx=0x7f048802bff0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048f59e640 1 -- 172.21.15.60:0/540122658 shutdown_connections 2023-10-16T12:53:59.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048f59e640 1 --2- 172.21.15.60:0/540122658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049006c9e0 0x7f04900ccfb0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048f59e640 1 -- 172.21.15.60:0/540122658 >> 172.21.15.60:0/540122658 conn(0x7f04900ce6a0 msgr2=0x7f04900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:53:59.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.496+0000 7f048f59e640 1 -- 172.21.15.60:0/540122658 shutdown_connections 2023-10-16T12:53:59.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.497+0000 7f048f59e640 1 -- 172.21.15.60:0/540122658 wait complete. 2023-10-16T12:53:59.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.498+0000 7f048f59e640 1 Processor -- start 2023-10-16T12:53:59.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.498+0000 7f048f59e640 1 -- start start 2023-10-16T12:53:59.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.499+0000 7f048f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f049006c9e0 0x7f04900b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.499+0000 7f048f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f049006fb50 0x7f04901639c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.499+0000 7f048f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 0x7f04900c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.499+0000 7f048f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04900b4420 con 0x7f04900c9060 2023-10-16T12:53:59.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.499+0000 7f048f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490163f00 con 0x7f049006c9e0 2023-10-16T12:53:59.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.499+0000 7f048f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0490163530 con 0x7f049006fb50 2023-10-16T12:53:59.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 0x7f04900c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 0x7f04900c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57900/0 (socket says 172.21.15.60:57900) 2023-10-16T12:53:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 -- 172.21.15.60:0/3150089053 learned_addr learned my addr 172.21.15.60:0/3150089053 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:53:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048dd9b640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f049006fb50 0x7f04901639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048e59c640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f049006c9e0 0x7f04900b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 -- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f049006fb50 msgr2=0x7f04901639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f049006fb50 0x7f04901639c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 -- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f049006c9e0 msgr2=0x7f04900b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:53:59.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f049006c9e0 0x7f04900b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:53:59.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048ed9d640 1 -- 172.21.15.60:0/3150089053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488009b30 con 0x7f04900c9060 2023-10-16T12:53:59.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.500+0000 7f048e59c640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f049006c9e0 0x7f04900b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:53:59.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.501+0000 7f048ed9d640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 0x7f04900c49b0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f0488009e50 tx=0x7f0488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:59.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.501+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488003990 con 0x7f04900c9060 2023-10-16T12:53:59.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.501+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0488003b10 con 0x7f04900c9060 2023-10-16T12:53:59.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.501+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488003cc0 con 0x7f04900c9060 2023-10-16T12:53:59.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.501+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0490162920 con 0x7f04900c9060 2023-10-16T12:53:59.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.501+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0490162ee0 con 0x7f04900c9060 2023-10-16T12:53:59.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.503+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049004e9b0 con 0x7f04900c9060 2023-10-16T12:53:59.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.504+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f048803e450 con 0x7f04900c9060 2023-10-16T12:53:59.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.505+0000 7f047effd640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f047006af90 0x7f047006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:53:59.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.505+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f04880abf90 con 0x7f04900c9060 2023-10-16T12:53:59.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.505+0000 7f048e59c640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f047006af90 0x7f047006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:53:59.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.507+0000 7f048e59c640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f047006af90 0x7f047006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0480000f60 tx=0x7f0480009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:53:59.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.509+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0488076b60 con 0x7f04900c9060 2023-10-16T12:53:59.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:53:59.735+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0490163670 con 0x7f04900c9060 2023-10-16T12:54:00.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.267+0000 7f047effd640 1 -- 172.21.15.60:0/3150089053 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7f048802ce60 con 0x7f04900c9060 2023-10-16T12:54:00.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f047006af90 msgr2=0x7f047006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:00.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f047006af90 0x7f047006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0480000f60 tx=0x7f0480009040 comp rx=0 tx=0).stop 2023-10-16T12:54:00.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 msgr2=0x7f04900c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:00.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 0x7f04900c49b0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f0488009e50 tx=0x7f0488004a40 comp rx=0 tx=0).stop 2023-10-16T12:54:00.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 shutdown_connections 2023-10-16T12:54:00.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f047006af90 0x7f047006d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f049006fb50 0x7f04901639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04900c9060 0x7f04900c49b0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 --2- 172.21.15.60:0/3150089053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f049006c9e0 0x7f04900b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 >> 172.21.15.60:0/3150089053 conn(0x7f04900ce6a0 msgr2=0x7f04900cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:00.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 shutdown_connections 2023-10-16T12:54:00.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.272+0000 7f048f59e640 1 -- 172.21.15.60:0/3150089053 wait complete. 2023-10-16T12:54:00.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:54:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.511+0000 7f07e14df640 1 Processor -- start 2023-10-16T12:54:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.511+0000 7f07e14df640 1 -- start start 2023-10-16T12:54:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.513+0000 7f07e14df640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f07dc144060 0x7f07dc0b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.513+0000 7f07e14df640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07dc15ecb0 0x7f07dc06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.513+0000 7f07e14df640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07dc06f1d0 con 0x7f07dc144440 2023-10-16T12:54:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.513+0000 7f07e14df640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07dc0cd430 con 0x7f07dc144060 2023-10-16T12:54:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.513+0000 7f07e14df640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07dc0cd610 con 0x7f07dc15ecb0 2023-10-16T12:54:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f07dc144060 0x7f07dc0b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f07dc144060 0x7f07dc0b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57902/0 (socket says 172.21.15.60:57902) 2023-10-16T12:54:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 -- 172.21.15.60:0/2756601152 learned_addr learned my addr 172.21.15.60:0/2756601152 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:00.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 -- 172.21.15.60:0/2756601152 >> v2:172.21.15.144:3300/0 conn(0x7f07dc15ecb0 msgr2=0x7f07dc06f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:00.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07e0cde640 1 --2- 172.21.15.60:0/2756601152 >> v2:172.21.15.144:3300/0 conn(0x7f07dc15ecb0 0x7f07dc06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 --2- 172.21.15.60:0/2756601152 >> v2:172.21.15.144:3300/0 conn(0x7f07dc15ecb0 0x7f07dc06f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 -- 172.21.15.60:0/2756601152 >> v1:172.21.15.60:6789/0 conn(0x7f07dc144440 legacy=0x7f07dc15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07dbfff640 1 -- 172.21.15.60:0/2756601152 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc0cd8f0 con 0x7f07dc144060 2023-10-16T12:54:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.514+0000 7f07e0cde640 1 --2- 172.21.15.60:0/2756601152 >> v2:172.21.15.144:3300/0 conn(0x7f07dc15ecb0 0x7f07dc06f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.515+0000 7f07dbfff640 1 --2- 172.21.15.60:0/2756601152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc0b0a40 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f07dc15bfc0 tx=0x7f07cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baae5e57b0b11a1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07daffd640 1 -- 172.21.15.60:0/2756601152 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc031870 con 0x7f07dc144060 2023-10-16T12:54:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07daffd640 1 -- 172.21.15.60:0/2756601152 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc0319d0 con 0x7f07dc144060 2023-10-16T12:54:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07daffd640 1 -- 172.21.15.60:0/2756601152 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc031d60 con 0x7f07dc144060 2023-10-16T12:54:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07e14df640 1 -- 172.21.15.60:0/2756601152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 msgr2=0x7f07dc0b0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:00.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07e14df640 1 --2- 172.21.15.60:0/2756601152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc0b0a40 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f07dc15bfc0 tx=0x7f07cc02b910 comp rx=0 tx=0).stop 2023-10-16T12:54:00.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07e14df640 1 -- 172.21.15.60:0/2756601152 shutdown_connections 2023-10-16T12:54:00.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07e14df640 1 --2- 172.21.15.60:0/2756601152 >> v2:172.21.15.144:3300/0 conn(0x7f07dc15ecb0 0x7f07dc06f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07e14df640 1 --2- 172.21.15.60:0/2756601152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc0b0a40 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.516+0000 7f07e14df640 1 -- 172.21.15.60:0/2756601152 >> 172.21.15.60:0/2756601152 conn(0x7f07dc069da0 msgr2=0x7f07dc06a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:00.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.517+0000 7f07e14df640 1 -- 172.21.15.60:0/2756601152 shutdown_connections 2023-10-16T12:54:00.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.517+0000 7f07e14df640 1 -- 172.21.15.60:0/2756601152 wait complete. 2023-10-16T12:54:00.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.519+0000 7f07e14df640 1 Processor -- start 2023-10-16T12:54:00.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.519+0000 7f07e14df640 1 -- start start 2023-10-16T12:54:00.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07e14df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc13ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:00.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07e14df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07dc144440 0x7f07dc1f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:00.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07e14df640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07dc15ecb0 0x7f07dc1f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:00.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07e14df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc13b410 con 0x7f07dc144060 2023-10-16T12:54:00.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07e14df640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc14f5d0 con 0x7f07dc15ecb0 2023-10-16T12:54:00.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07e14df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc1f6510 con 0x7f07dc144440 2023-10-16T12:54:00.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc13ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:00.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07dbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc13ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57912/0 (socket says 172.21.15.60:57912) 2023-10-16T12:54:00.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.520+0000 7f07dbfff640 1 -- 172.21.15.60:0/710650682 learned_addr learned my addr 172.21.15.60:0/710650682 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:00.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07db7fe640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07dc144440 0x7f07dc1f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:00.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07e0cde640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07dc15ecb0 0x7f07dc1f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:00.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07dbfff640 1 -- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07dc144440 msgr2=0x7f07dc1f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:00.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07dbfff640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07dc144440 0x7f07dc1f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07dbfff640 1 -- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07dc15ecb0 msgr2=0x7f07dc1f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07dbfff640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07dc15ecb0 0x7f07dc1f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07dbfff640 1 -- 172.21.15.60:0/710650682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc0094d0 con 0x7f07dc144060 2023-10-16T12:54:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.521+0000 7f07dbfff640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc13ab50 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f07cc000c00 tx=0x7f07cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:00.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.522+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc031860 con 0x7f07dc144060 2023-10-16T12:54:00.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.522+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc0443f0 con 0x7f07dc144060 2023-10-16T12:54:00.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.522+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0447a0 con 0x7f07dc144060 2023-10-16T12:54:00.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.522+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07dc13b130 con 0x7f07dc144060 2023-10-16T12:54:00.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.522+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07dc14f260 con 0x7f07dc144060 2023-10-16T12:54:00.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.523+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07dc14ede0 con 0x7f07dc144060 2023-10-16T12:54:00.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.524+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f07cc044900 con 0x7f07dc144060 2023-10-16T12:54:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.526+0000 7f07d8ff9640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f07c806af90 0x7f07c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.526+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f07cc02d040 con 0x7f07dc144060 2023-10-16T12:54:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.527+0000 7f07db7fe640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f07c806af90 0x7f07c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.528+0000 7f07db7fe640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f07c806af90 0x7f07c806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0b37e0 tx=0x7f07d0009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:00.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.531+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07dc14ede0 con 0x7f07dc144060 2023-10-16T12:54:00.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:00.727+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f07dc14ede0 con 0x7f07dc144060 2023-10-16T12:54:01.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.280+0000 7f07d8ff9640 1 -- 172.21.15.60:0/710650682 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v147) v1 ==== 181+0+0 (secure 0 0 0) 0x7f07dc14ede0 con 0x7f07dc144060 2023-10-16T12:54:01.283 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:54:01.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.283+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f07c806af90 msgr2=0x7f07c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.283+0000 7f07e14df640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f07c806af90 0x7f07c806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0b37e0 tx=0x7f07d0009040 comp rx=0 tx=0).stop 2023-10-16T12:54:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 msgr2=0x7f07dc13ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc13ab50 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f07cc000c00 tx=0x7f07cc0046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 shutdown_connections 2023-10-16T12:54:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f07c806af90 0x7f07c806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07dc144440 0x7f07dc1f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07dc15ecb0 0x7f07dc1f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 --2- 172.21.15.60:0/710650682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07dc144060 0x7f07dc13ab50 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.284+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 >> 172.21.15.60:0/710650682 conn(0x7f07dc069da0 msgr2=0x7f07dc0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.285+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 shutdown_connections 2023-10-16T12:54:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.286+0000 7f07e14df640 1 -- 172.21.15.60:0/710650682 wait complete. 2023-10-16T12:54:01.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:54:01.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.521+0000 7f1e2f3ac640 1 Processor -- start 2023-10-16T12:54:01.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.521+0000 7f1e2f3ac640 1 -- start start 2023-10-16T12:54:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2f3ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 0x7f1e280cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2f3ac640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2815d150 con 0x7f1e2815cd70 2023-10-16T12:54:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2f3ac640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280d1180 con 0x7f1e280d0be0 2023-10-16T12:54:01.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2f3ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280d1460 con 0x7f1e280b8480 2023-10-16T12:54:01.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2e3aa640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1e2815cd70 0x7f1e280d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34028/0 (socket says 172.21.15.60:34028) 2023-10-16T12:54:01.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2e3aa640 1 -- 172.21.15.60:0/1022732049 learned_addr learned my addr 172.21.15.60:0/1022732049 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:01.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 0x7f1e280cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805071761 0 0) 0x7f1e2815d150 con 0x7f1e2815cd70 2023-10-16T12:54:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.523+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e14003520 con 0x7f1e2815cd70 2023-10-16T12:54:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 230647658 0 0) 0x7f1e14003520 con 0x7f1e2815cd70 2023-10-16T12:54:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 msgr2=0x7f1e280cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 0x7f1e280cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 >> v1:172.21.15.144:6789/0 conn(0x7f1e280d0be0 legacy=0x7f1e280b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280d1740 con 0x7f1e2815cd70 2023-10-16T12:54:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1e240019f0 con 0x7f1e2815cd70 2023-10-16T12:54:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 >> v1:172.21.15.60:6789/0 conn(0x7f1e2815cd70 legacy=0x7f1e280d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e140039a0 0x7f1e14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 0x7f1e14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280d1460 con 0x7f1e140047c0 2023-10-16T12:54:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1400ba60 con 0x7f1e140039a0 2023-10-16T12:54:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1400bc20 con 0x7f1e280b8480 2023-10-16T12:54:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.524+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2e3aa640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e140039a0 0x7f1e14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2dba9640 1 -- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 msgr2=0x7f1e14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e280b8480 0x7f1e14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2dba9640 1 -- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e140039a0 msgr2=0x7f1e14003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e140039a0 0x7f1e14003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2dba9640 1 -- 172.21.15.60:0/1022732049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e14003520 con 0x7f1e140047c0 2023-10-16T12:54:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.525+0000 7f1e2e3aa640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e140039a0 0x7f1e14003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.526+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e14004c40 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009a90 tx=0x7f1e140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792c1bbf18f486bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.526+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1802c690 con 0x7f1e140047c0 2023-10-16T12:54:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.526+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1802c7f0 con 0x7f1e140047c0 2023-10-16T12:54:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.526+0000 7f1e2d3a8640 1 -- 172.21.15.60:0/1022732049 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1802cb80 con 0x7f1e140047c0 2023-10-16T12:54:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1022732049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 msgr2=0x7f1e14004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1022732049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e14004c40 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009a90 tx=0x7f1e140057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2ebab640 1 -- 172.21.15.60:0/1022732049 reap_dead start 2023-10-16T12:54:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1022732049 shutdown_connections 2023-10-16T12:54:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1022732049 >> 172.21.15.60:0/1022732049 conn(0x7f1e280cf570 msgr2=0x7f1e280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1022732049 shutdown_connections 2023-10-16T12:54:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.527+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1022732049 wait complete. 2023-10-16T12:54:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.529+0000 7f1e2f3ac640 1 Processor -- start 2023-10-16T12:54:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.529+0000 7f1e2f3ac640 1 -- start start 2023-10-16T12:54:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2f3ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e281f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2f3ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e281f2240 0x7f1e280ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2f3ac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e281f4f20 0x7f1e280b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2f3ac640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280aad80 con 0x7f1e140047c0 2023-10-16T12:54:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2f3ac640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280a9620 con 0x7f1e281f4f20 2023-10-16T12:54:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2f3ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280ab5b0 con 0x7f1e281f2240 2023-10-16T12:54:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e281f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e281f57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57932/0 (socket says 172.21.15.60:57932) 2023-10-16T12:54:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 -- 172.21.15.60:0/1820590020 learned_addr learned my addr 172.21.15.60:0/1820590020 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 -- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e281f2240 msgr2=0x7f1e280ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2ebab640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e281f4f20 0x7f1e280b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e281f2240 0x7f1e280ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e281f2240 0x7f1e280ab070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 -- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e281f4f20 msgr2=0x7f1e280b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e281f4f20 0x7f1e280b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.530+0000 7f1e2e3aa640 1 -- 172.21.15.60:0/1820590020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e281f26c0 con 0x7f1e140047c0 2023-10-16T12:54:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.531+0000 7f1e2e3aa640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e281f57e0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f1e240021a0 tx=0x7f1e2400b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.531+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24004d60 con 0x7f1e140047c0 2023-10-16T12:54:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.531+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e240070d0 con 0x7f1e140047c0 2023-10-16T12:54:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.531+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24007480 con 0x7f1e140047c0 2023-10-16T12:54:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.531+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e281f5dc0 con 0x7f1e140047c0 2023-10-16T12:54:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.531+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e281f6380 con 0x7f1e140047c0 2023-10-16T12:54:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.532+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2804e9b0 con 0x7f1e140047c0 2023-10-16T12:54:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.533+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1e240075e0 con 0x7f1e140047c0 2023-10-16T12:54:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.534+0000 7f1e1effd640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1e0c06afc0 0x7f1e0c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.535+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6088+0+0 (secure 0 0 0) 0x7f1e24089a70 con 0x7f1e140047c0 2023-10-16T12:54:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.535+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1e0c06afc0 0x7f1e0c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.537+0000 7f1e2dba9640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1e0c06afc0 0x7f1e0c06d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e18036000 tx=0x7f1e18030000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.539+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e24054790 con 0x7f1e140047c0 2023-10-16T12:54:01.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:01.733+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1e280b5ea0 con 0x7f1e140047c0 2023-10-16T12:54:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.283+0000 7f1e1effd640 1 -- 172.21.15.60:0/1820590020 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v148) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1e24019020 con 0x7f1e140047c0 2023-10-16T12:54:02.287 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T12:54:02.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.287+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1e0c06afc0 msgr2=0x7f1e0c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.287+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1e0c06afc0 0x7f1e0c06d480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e18036000 tx=0x7f1e18030000 comp rx=0 tx=0).stop 2023-10-16T12:54:02.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.287+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 msgr2=0x7f1e281f57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.287+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e281f57e0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f1e240021a0 tx=0x7f1e2400b560 comp rx=0 tx=0).stop 2023-10-16T12:54:02.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 shutdown_connections 2023-10-16T12:54:02.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1e0c06afc0 0x7f1e0c06d480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e281f2240 0x7f1e280ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e281f4f20 0x7f1e280b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 --2- 172.21.15.60:0/1820590020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e140047c0 0x7f1e281f57e0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 >> 172.21.15.60:0/1820590020 conn(0x7f1e280cf570 msgr2=0x7f1e2806bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:02.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 shutdown_connections 2023-10-16T12:54:02.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.288+0000 7f1e2f3ac640 1 -- 172.21.15.60:0/1820590020 wait complete. 2023-10-16T12:54:02.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:54:02.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.529+0000 7fbad309f640 1 Processor -- start 2023-10-16T12:54:02.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.529+0000 7fbad309f640 1 -- start start 2023-10-16T12:54:02.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad309f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbacc159ad0 0x7fbacc070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad309f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 0x7fbacc06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad309f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0cd260 con 0x7fbacc06b3e0 2023-10-16T12:54:02.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad309f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbacc071470 con 0x7fbacc159ad0 2023-10-16T12:54:02.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad309f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc15de30 con 0x7fbacc15d650 2023-10-16T12:54:02.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad289e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbacc06b3e0 0x7fbacc0c9de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34040/0 (socket says 172.21.15.60:34040) 2023-10-16T12:54:02.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.531+0000 7fbad289e640 1 -- 172.21.15.60:0/484837624 learned_addr learned my addr 172.21.15.60:0/484837624 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:02.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908130008 0 0) 0x7fbacc0cd260 con 0x7fbacc06b3e0 2023-10-16T12:54:02.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8003520 con 0x7fbacc06b3e0 2023-10-16T12:54:02.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad189c640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 0x7fbacc06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad209d640 1 --2- 172.21.15.60:0/484837624 >> v2:172.21.15.144:3300/0 conn(0x7fbacc159ad0 0x7fbacc070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2322431347 0 0) 0x7fbab8003520 con 0x7fbacc06b3e0 2023-10-16T12:54:02.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 msgr2=0x7fbacc06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:02.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 0x7fbacc06aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 >> v2:172.21.15.144:3300/0 conn(0x7fbacc159ad0 msgr2=0x7fbacc070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:02.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 --2- 172.21.15.60:0/484837624 >> v2:172.21.15.144:3300/0 conn(0x7fbacc159ad0 0x7fbacc070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc15e110 con 0x7fbacc06b3e0 2023-10-16T12:54:02.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fbac40019f0 con 0x7fbacc06b3e0 2023-10-16T12:54:02.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 >> v1:172.21.15.60:6789/0 conn(0x7fbacc06b3e0 legacy=0x7fbacc0c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.532+0000 7fbad109b640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 0x7fbab8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc071470 con 0x7fbab8003970 2023-10-16T12:54:02.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc15de30 con 0x7fbab80047c0 2023-10-16T12:54:02.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab800dc90 con 0x7fbacc15d650 2023-10-16T12:54:02.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad209d640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad289e640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 0x7fbab8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad209d640 1 -- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 msgr2=0x7fbab8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:02.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad189c640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad209d640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbacc15d650 0x7fbab8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad209d640 1 -- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbab80047c0 msgr2=0x7fbab8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:02.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad209d640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad209d640 1 -- 172.21.15.60:0/484837624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8003520 con 0x7fbab8003970 2023-10-16T12:54:02.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.533+0000 7fbad189c640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbab80047c0 0x7fbab8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:02.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.534+0000 7fbad209d640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbac800ba00 tx=0x7fbab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5222da2693b61d7a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:02.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8032620 con 0x7fbab8003970 2023-10-16T12:54:02.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8032780 con 0x7fbab8003970 2023-10-16T12:54:02.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad109b640 1 -- 172.21.15.60:0/484837624 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8032b10 con 0x7fbab8003970 2023-10-16T12:54:02.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad309f640 1 -- 172.21.15.60:0/484837624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab8003970 msgr2=0x7fbab8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad309f640 1 --2- 172.21.15.60:0/484837624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab8003970 0x7fbab8003d90 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbac800ba00 tx=0x7fbab80057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad289e640 1 -- 172.21.15.60:0/484837624 reap_dead start 2023-10-16T12:54:02.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad309f640 1 -- 172.21.15.60:0/484837624 shutdown_connections 2023-10-16T12:54:02.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.535+0000 7fbad309f640 1 -- 172.21.15.60:0/484837624 >> 172.21.15.60:0/484837624 conn(0x7fbacc0b0170 msgr2=0x7fbacc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:02.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.536+0000 7fbad309f640 1 -- 172.21.15.60:0/484837624 shutdown_connections 2023-10-16T12:54:02.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.536+0000 7fbad309f640 1 -- 172.21.15.60:0/484837624 wait complete. 2023-10-16T12:54:02.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.537+0000 7fbad309f640 1 Processor -- start 2023-10-16T12:54:02.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.537+0000 7fbad309f640 1 -- start start 2023-10-16T12:54:02.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.538+0000 7fbad309f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbab8003970 0x7fbacc14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.538+0000 7fbad309f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 0x7fbacc1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.538+0000 7fbad309f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbacc153b00 0x7fbacc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.538+0000 7fbad309f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0b3650 con 0x7fbacc15a150 2023-10-16T12:54:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.538+0000 7fbad309f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc14af20 con 0x7fbacc153b00 2023-10-16T12:54:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.538+0000 7fbad309f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc1f5cc0 con 0x7fbab8003970 2023-10-16T12:54:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 0x7fbacc1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 0x7fbacc1535c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57950/0 (socket says 172.21.15.60:57950) 2023-10-16T12:54:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 -- 172.21.15.60:0/2731438539 learned_addr learned my addr 172.21.15.60:0/2731438539 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad289e640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbacc153b00 0x7fbacc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad209d640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbab8003970 0x7fbacc14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 -- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbab8003970 msgr2=0x7fbacc14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbab8003970 0x7fbacc14a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 -- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbacc153b00 msgr2=0x7fbacc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbacc153b00 0x7fbacc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.539+0000 7fbad189c640 1 -- 172.21.15.60:0/2731438539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc1f5fa0 con 0x7fbacc15a150 2023-10-16T12:54:02.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.540+0000 7fbad189c640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 0x7fbacc1535c0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00e7e0 tx=0x7fbabc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:02.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.540+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc00c6d0 con 0x7fbacc15a150 2023-10-16T12:54:02.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.540+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc00c830 con 0x7fbacc15a150 2023-10-16T12:54:02.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.540+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc1f6280 con 0x7fbacc15a150 2023-10-16T12:54:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.540+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc00cbe0 con 0x7fbacc15a150 2023-10-16T12:54:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.540+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc1f6840 con 0x7fbacc15a150 2023-10-16T12:54:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.541+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc04e9b0 con 0x7fbacc15a150 2023-10-16T12:54:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.542+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbabc00cd40 con 0x7fbacc15a150 2023-10-16T12:54:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.544+0000 7fbac2ffd640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbab006af90 0x7fbab006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.544+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7fbabc089630 con 0x7fbacc15a150 2023-10-16T12:54:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.544+0000 7fbad209d640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbab006af90 0x7fbab006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.545+0000 7fbad209d640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbab006af90 0x7fbab006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbab80057a0 tx=0x7fbac8030000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.548+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbabc0544e0 con 0x7fbacc15a150 2023-10-16T12:54:02.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.743+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbacc0b4600 con 0x7fbacc15a150 2023-10-16T12:54:02.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.750+0000 7fbac2ffd640 1 -- 172.21.15.60:0/2731438539 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbabc057b00 con 0x7fbacc15a150 2023-10-16T12:54:02.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.752+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbab006af90 msgr2=0x7fbab006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.752+0000 7fbad309f640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbab006af90 0x7fbab006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbab80057a0 tx=0x7fbac8030000 comp rx=0 tx=0).stop 2023-10-16T12:54:02.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.752+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 msgr2=0x7fbacc1535c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:02.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.752+0000 7fbad309f640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 0x7fbacc1535c0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00e7e0 tx=0x7fbabc00eca0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 shutdown_connections 2023-10-16T12:54:02.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbab006af90 0x7fbab006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbacc153b00 0x7fbacc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbacc15a150 0x7fbacc1535c0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 --2- 172.21.15.60:0/2731438539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbab8003970 0x7fbacc14a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 >> 172.21.15.60:0/2731438539 conn(0x7fbacc0b0170 msgr2=0x7fbacc0af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 shutdown_connections 2023-10-16T12:54:02.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:02.753+0000 7fbad309f640 1 -- 172.21.15.60:0/2731438539 wait complete. 2023-10-16T12:54:02.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:54:03.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.002+0000 7fb40a33c640 1 Processor -- start 2023-10-16T12:54:03.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.002+0000 7fb40a33c640 1 -- start start 2023-10-16T12:54:03.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40a33c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4040ad970 0x7fb4040cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40a33c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4040ccfb0 0x7fb4040cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40a33c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40415b9d0 con 0x7fb40406c040 2023-10-16T12:54:03.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40a33c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40415bcb0 con 0x7fb4040ad970 2023-10-16T12:54:03.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40a33c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40415bf90 con 0x7fb4040ccfb0 2023-10-16T12:54:03.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40933a640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb40406c040 0x7fb40406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34050/0 (socket says 172.21.15.60:34050) 2023-10-16T12:54:03.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.004+0000 7fb40933a640 1 -- 172.21.15.60:0/3139918000 learned_addr learned my addr 172.21.15.60:0/3139918000 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:03.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb408b39640 1 --2- 172.21.15.60:0/3139918000 >> v2:172.21.15.60:3300/0 conn(0x7fb4040ad970 0x7fb4040cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb409b3b640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4040ccfb0 0x7fb4040cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb408b39640 1 -- 172.21.15.60:0/3139918000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4040ccfb0 msgr2=0x7fb4040cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:03.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb408b39640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4040ccfb0 0x7fb4040cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb408b39640 1 -- 172.21.15.60:0/3139918000 >> v1:172.21.15.60:6789/0 conn(0x7fb40406c040 legacy=0x7fb40406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb408b39640 1 -- 172.21.15.60:0/3139918000 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4040cc270 con 0x7fb4040ad970 2023-10-16T12:54:03.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb3fbfff640 1 -- 172.21.15.60:0/3139918000 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605642132 0 0) 0x7fb40415b9d0 con 0x7fb40406c040 2023-10-16T12:54:03.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.005+0000 7fb409b3b640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4040ccfb0 0x7fb4040cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:03.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.006+0000 7fb408b39640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040cca70 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40097a0 tx=0x7fb3f402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d83e77b67b8cf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb3fbfff640 1 -- 172.21.15.60:0/3139918000 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f402d040 con 0x7fb4040ad970 2023-10-16T12:54:03.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb3fbfff640 1 -- 172.21.15.60:0/3139918000 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4031810 con 0x7fb4040ad970 2023-10-16T12:54:03.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb3fbfff640 1 -- 172.21.15.60:0/3139918000 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4031ba0 con 0x7fb4040ad970 2023-10-16T12:54:03.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb40a33c640 1 -- 172.21.15.60:0/3139918000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 msgr2=0x7fb4040cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040cca70 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40097a0 tx=0x7fb3f402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb40a33c640 1 -- 172.21.15.60:0/3139918000 shutdown_connections 2023-10-16T12:54:03.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4040ccfb0 0x7fb4040cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3139918000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040cca70 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.007+0000 7fb40a33c640 1 -- 172.21.15.60:0/3139918000 >> 172.21.15.60:0/3139918000 conn(0x7fb4040ce6a0 msgr2=0x7fb4040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:03.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.008+0000 7fb40a33c640 1 -- 172.21.15.60:0/3139918000 shutdown_connections 2023-10-16T12:54:03.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.008+0000 7fb40a33c640 1 -- 172.21.15.60:0/3139918000 wait complete. 2023-10-16T12:54:03.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.009+0000 7fb40a33c640 1 Processor -- start 2023-10-16T12:54:03.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.010+0000 7fb40a33c640 1 -- start start 2023-10-16T12:54:03.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40a33c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb40406c040 0x7fb404156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40a33c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40a33c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4040ccfb0 0x7fb4040af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40a33c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040c21b0 con 0x7fb4040ad970 2023-10-16T12:54:03.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40a33c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040afb90 con 0x7fb4040ccfb0 2023-10-16T12:54:03.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40a33c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040afe70 con 0x7fb40406c040 2023-10-16T12:54:03.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57966/0 (socket says 172.21.15.60:57966) 2023-10-16T12:54:03.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 -- 172.21.15.60:0/3070202838 learned_addr learned my addr 172.21.15.60:0/3070202838 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:03.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb40933a640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb40406c040 0x7fb404156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 -- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb40406c040 msgr2=0x7fb404156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb409b3b640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4040ccfb0 0x7fb4040af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb40406c040 0x7fb404156420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 -- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4040ccfb0 msgr2=0x7fb4040af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.011+0000 7fb408b39640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4040ccfb0 0x7fb4040af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb408b39640 1 -- 172.21.15.60:0/3070202838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4009400 con 0x7fb4040ad970 2023-10-16T12:54:03.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb408b39640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040aefd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40096f0 tx=0x7fb3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f403ad60 con 0x7fb4040ad970 2023-10-16T12:54:03.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4044400 con 0x7fb4040ad970 2023-10-16T12:54:03.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40447b0 con 0x7fb4040ad970 2023-10-16T12:54:03.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4040b0150 con 0x7fb4040ad970 2023-10-16T12:54:03.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.012+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4041571a0 con 0x7fb4040ad970 2023-10-16T12:54:03.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.014+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40406b330 con 0x7fb4040ad970 2023-10-16T12:54:03.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.015+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3f404d440 con 0x7fb4040ad970 2023-10-16T12:54:03.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.016+0000 7fb3f9ffb640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb3e806af90 0x7fb3e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.016+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7fb3f40abcb0 con 0x7fb4040ad970 2023-10-16T12:54:03.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.017+0000 7fb40933a640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb3e806af90 0x7fb3e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.018+0000 7fb40933a640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb3e806af90 0x7fb3e806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb400000f60 tx=0x7fb400009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.021+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3f4076b50 con 0x7fb4040ad970 2023-10-16T12:54:03.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.219+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb40406b330 con 0x7fb4040ad970 2023-10-16T12:54:03.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.224+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/3070202838 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb3f407a170 con 0x7fb4040ad970 2023-10-16T12:54:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb3e806af90 msgr2=0x7fb3e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb3e806af90 0x7fb3e806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb400000f60 tx=0x7fb400009040 comp rx=0 tx=0).stop 2023-10-16T12:54:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 msgr2=0x7fb4040aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040aefd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40096f0 tx=0x7fb3f40046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 shutdown_connections 2023-10-16T12:54:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb3e806af90 0x7fb3e806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4040ccfb0 0x7fb4040af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4040ad970 0x7fb4040aefd0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 --2- 172.21.15.60:0/3070202838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb40406c040 0x7fb404156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 >> 172.21.15.60:0/3070202838 conn(0x7fb4040ce6a0 msgr2=0x7fb4040ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 shutdown_connections 2023-10-16T12:54:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.227+0000 7fb40a33c640 1 -- 172.21.15.60:0/3070202838 wait complete. 2023-10-16T12:54:03.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:54:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.484+0000 7f4e29d0c640 1 Processor -- start 2023-10-16T12:54:03.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.485+0000 7f4e29d0c640 1 -- start start 2023-10-16T12:54:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e29d0c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e24069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e29d0c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e2406a2e0 0x7f4e240c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e29d0c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2406a6f0 con 0x7f4e2415d150 2023-10-16T12:54:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e29d0c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2415d530 con 0x7f4e2406a2e0 2023-10-16T12:54:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e29d0c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e240ac890 con 0x7f4e240ad350 2023-10-16T12:54:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e23fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e2406a2e0 0x7f4e240c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e237fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4e2415d150 0x7f4e24160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34062/0 (socket says 172.21.15.60:34062) 2023-10-16T12:54:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.486+0000 7f4e237fe640 1 -- 172.21.15.60:0/2069064357 learned_addr learned my addr 172.21.15.60:0/2069064357 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e22ffd640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e24069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e23fff640 1 -- 172.21.15.60:0/2069064357 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 msgr2=0x7f4e24069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e23fff640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e24069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e23fff640 1 -- 172.21.15.60:0/2069064357 >> v1:172.21.15.60:6789/0 conn(0x7f4e2415d150 legacy=0x7f4e24160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e23fff640 1 -- 172.21.15.60:0/2069064357 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24161650 con 0x7f4e2406a2e0 2023-10-16T12:54:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e227fc640 1 -- 172.21.15.60:0/2069064357 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489055384 0 0) 0x7f4e2406a6f0 con 0x7f4e2415d150 2023-10-16T12:54:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.487+0000 7f4e22ffd640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e24069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.488+0000 7f4e23fff640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2406a2e0 0x7f4e240c9000 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009d90 tx=0x7f4e1c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1c0424003a6d93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.488+0000 7f4e227fc640 1 -- 172.21.15.60:0/2069064357 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c02cc60 con 0x7f4e2406a2e0 2023-10-16T12:54:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e227fc640 1 -- 172.21.15.60:0/2069064357 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c02cdc0 con 0x7f4e2406a2e0 2023-10-16T12:54:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e227fc640 1 -- 172.21.15.60:0/2069064357 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c034660 con 0x7f4e2406a2e0 2023-10-16T12:54:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 -- 172.21.15.60:0/2069064357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2406a2e0 msgr2=0x7f4e240c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2406a2e0 0x7f4e240c9000 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009d90 tx=0x7f4e1c02beb0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 -- 172.21.15.60:0/2069064357 shutdown_connections 2023-10-16T12:54:03.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2406a2e0 0x7f4e240c9000 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/2069064357 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e24069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 -- 172.21.15.60:0/2069064357 >> 172.21.15.60:0/2069064357 conn(0x7f4e240ce9b0 msgr2=0x7f4e240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 -- 172.21.15.60:0/2069064357 shutdown_connections 2023-10-16T12:54:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.489+0000 7f4e29d0c640 1 -- 172.21.15.60:0/2069064357 wait complete. 2023-10-16T12:54:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.491+0000 7f4e29d0c640 1 Processor -- start 2023-10-16T12:54:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.491+0000 7f4e29d0c640 1 -- start start 2023-10-16T12:54:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.492+0000 7f4e29d0c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2406a2e0 0x7f4e241eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.492+0000 7f4e29d0c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e2415d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.492+0000 7f4e29d0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 0x7f4e240c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.492+0000 7f4e29d0c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2414f700 con 0x7f4e2415d150 2023-10-16T12:54:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.492+0000 7f4e29d0c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e241f0490 con 0x7f4e2406a2e0 2023-10-16T12:54:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.492+0000 7f4e29d0c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e240c3f50 con 0x7f4e240ad350 2023-10-16T12:54:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 0x7f4e240c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 0x7f4e240c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57980/0 (socket says 172.21.15.60:57980) 2023-10-16T12:54:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 -- 172.21.15.60:0/1769541561 learned_addr learned my addr 172.21.15.60:0/1769541561 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e237fe640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2406a2e0 0x7f4e241eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e22ffd640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e2415d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 -- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 msgr2=0x7f4e2415d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e2415d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 -- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2406a2e0 msgr2=0x7f4e241eff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2406a2e0 0x7f4e241eff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.493+0000 7f4e23fff640 1 -- 172.21.15.60:0/1769541561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c009a70 con 0x7f4e2415d150 2023-10-16T12:54:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.494+0000 7f4e23fff640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 0x7f4e240c24e0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009d90 tx=0x7f4e1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.494+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c003b90 con 0x7f4e2415d150 2023-10-16T12:54:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.494+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c003cf0 con 0x7f4e2415d150 2023-10-16T12:54:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.494+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c004270 con 0x7f4e2415d150 2023-10-16T12:54:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.494+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e240c2f80 con 0x7f4e2415d150 2023-10-16T12:54:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.494+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e240c41a0 con 0x7f4e2415d150 2023-10-16T12:54:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.495+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2404e160 con 0x7f4e2415d150 2023-10-16T12:54:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.496+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e1c03b050 con 0x7f4e2415d150 2023-10-16T12:54:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.497+0000 7f4e0bfff640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.497+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7f4e1c0abdf0 con 0x7f4e2415d150 2023-10-16T12:54:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.498+0000 7f4e237fe640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.499+0000 7f4e237fe640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4e1006af90 0x7f4e1006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e14000f60 tx=0x7f4e14007b50 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.501+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e1c076da0 con 0x7f4e2415d150 2023-10-16T12:54:03.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.695+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f4e240b1300 con 0x7f4e2415d150 2023-10-16T12:54:03.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.702+0000 7f4e0bfff640 1 -- 172.21.15.60:0/1769541561 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4e1c07a3c0 con 0x7f4e2415d150 2023-10-16T12:54:03.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4e1006af90 msgr2=0x7f4e1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4e1006af90 0x7f4e1006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e14000f60 tx=0x7f4e14007b50 comp rx=0 tx=0).stop 2023-10-16T12:54:03.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 msgr2=0x7f4e240c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 0x7f4e240c24e0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009d90 tx=0x7f4e1c004a40 comp rx=0 tx=0).stop 2023-10-16T12:54:03.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 shutdown_connections 2023-10-16T12:54:03.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e240ad350 0x7f4e2415d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e2415d150 0x7f4e240c24e0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 --2- 172.21.15.60:0/1769541561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2406a2e0 0x7f4e241eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 >> 172.21.15.60:0/1769541561 conn(0x7f4e240ce9b0 msgr2=0x7f4e241565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:03.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 shutdown_connections 2023-10-16T12:54:03.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.705+0000 7f4e29d0c640 1 -- 172.21.15.60:0/1769541561 wait complete. 2023-10-16T12:54:03.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T12:54:03.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.930+0000 7fec62409640 1 Processor -- start 2023-10-16T12:54:03.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.931+0000 7fec62409640 1 -- start start 2023-10-16T12:54:03.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.932+0000 7fec62409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15aaa0 0x7fec5c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.932+0000 7fec62409640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec5c0ce390 0x7fec5c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.932+0000 7fec62409640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5c15ed60 con 0x7fec5c15ae80 2023-10-16T12:54:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec62409640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c06b810 con 0x7fec5c0ce390 2023-10-16T12:54:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec62409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0707e0 con 0x7fec5c15aaa0 2023-10-16T12:54:03.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec5c0ce390 0x7fec5c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec5c0ce390 0x7fec5c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57986/0 (socket says 172.21.15.60:57986) 2023-10-16T12:54:03.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 -- 172.21.15.60:0/2962941896 learned_addr learned my addr 172.21.15.60:0/2962941896 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:03.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec5bfff640 1 --2- 172.21.15.60:0/2962941896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15aaa0 0x7fec5c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 -- 172.21.15.60:0/2962941896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15aaa0 msgr2=0x7fec5c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:03.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 --2- 172.21.15.60:0/2962941896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15aaa0 0x7fec5c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 -- 172.21.15.60:0/2962941896 >> v1:172.21.15.60:6789/0 conn(0x7fec5c15ae80 legacy=0x7fec5c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec5affd640 1 -- 172.21.15.60:0/2962941896 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979213897 0 0) 0x7fec5c15ed60 con 0x7fec5c15ae80 2023-10-16T12:54:03.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.933+0000 7fec609a6640 1 -- 172.21.15.60:0/2962941896 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c15ef90 con 0x7fec5c0ce390 2023-10-16T12:54:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.934+0000 7fec609a6640 1 --2- 172.21.15.60:0/2962941896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c0ce390 0x7fec5c070f50 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fec54009e50 tx=0x7fec5402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c06b1e426f9525 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec5affd640 1 -- 172.21.15.60:0/2962941896 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5402cb00 con 0x7fec5c0ce390 2023-10-16T12:54:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec5affd640 1 -- 172.21.15.60:0/2962941896 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5402cc60 con 0x7fec5c0ce390 2023-10-16T12:54:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec5affd640 1 -- 172.21.15.60:0/2962941896 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec540343f0 con 0x7fec5c0ce390 2023-10-16T12:54:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec62409640 1 -- 172.21.15.60:0/2962941896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c0ce390 msgr2=0x7fec5c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec62409640 1 --2- 172.21.15.60:0/2962941896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c0ce390 0x7fec5c070f50 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fec54009e50 tx=0x7fec5402bf60 comp rx=0 tx=0).stop 2023-10-16T12:54:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec62409640 1 -- 172.21.15.60:0/2962941896 shutdown_connections 2023-10-16T12:54:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec62409640 1 --2- 172.21.15.60:0/2962941896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c0ce390 0x7fec5c070f50 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec62409640 1 --2- 172.21.15.60:0/2962941896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15aaa0 0x7fec5c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.935+0000 7fec62409640 1 -- 172.21.15.60:0/2962941896 >> 172.21.15.60:0/2962941896 conn(0x7fec5c0cb410 msgr2=0x7fec5c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.936+0000 7fec62409640 1 -- 172.21.15.60:0/2962941896 shutdown_connections 2023-10-16T12:54:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.936+0000 7fec62409640 1 -- 172.21.15.60:0/2962941896 wait complete. 2023-10-16T12:54:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.937+0000 7fec62409640 1 Processor -- start 2023-10-16T12:54:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.938+0000 7fec62409640 1 -- start start 2023-10-16T12:54:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.938+0000 7fec62409640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec5c0ce390 0x7fec5c153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.938+0000 7fec62409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 0x7fec5c0acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec62409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15ae80 0x7fec5c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec62409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c153d40 con 0x7fec5c15aaa0 2023-10-16T12:54:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec62409640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1f5b10 con 0x7fec5c0ce390 2023-10-16T12:54:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec62409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1f5db0 con 0x7fec5c15ae80 2023-10-16T12:54:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 0x7fec5c0acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 0x7fec5c0acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57992/0 (socket says 172.21.15.60:57992) 2023-10-16T12:54:03.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 -- 172.21.15.60:0/1370444177 learned_addr learned my addr 172.21.15.60:0/1370444177 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:03.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 -- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15ae80 msgr2=0x7fec5c070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15ae80 0x7fec5c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 -- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec5c0ce390 msgr2=0x7fec5c153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:03.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec5c0ce390 0x7fec5c153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:03.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.939+0000 7fec5b7fe640 1 -- 172.21.15.60:0/1370444177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54009b30 con 0x7fec5c15aaa0 2023-10-16T12:54:03.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.940+0000 7fec5b7fe640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 0x7fec5c0acb10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fec5000c4b0 tx=0x7fec5000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.940+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50005480 con 0x7fec5c15aaa0 2023-10-16T12:54:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.940+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec500035f0 con 0x7fec5c15aaa0 2023-10-16T12:54:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.940+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c1f6090 con 0x7fec5c15aaa0 2023-10-16T12:54:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.940+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500039e0 con 0x7fec5c15aaa0 2023-10-16T12:54:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.940+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c1f6650 con 0x7fec5c15aaa0 2023-10-16T12:54:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.941+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5c0c1950 con 0x7fec5c15aaa0 2023-10-16T12:54:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.942+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec50003b40 con 0x7fec5c15aaa0 2023-10-16T12:54:03.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.943+0000 7fec58ff9640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fec4806af90 0x7fec4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:03.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.943+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7fec50088d50 con 0x7fec5c15aaa0 2023-10-16T12:54:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.944+0000 7fec5bfff640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fec4806af90 0x7fec4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.945+0000 7fec5bfff640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fec4806af90 0x7fec4806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0098f0 tx=0x7fec4c008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:03.947+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec5008c030 con 0x7fec5c15aaa0 2023-10-16T12:54:04.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.212+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fec5c04e160 con 0x7fec5c15aaa0 2023-10-16T12:54:04.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.308+0000 7fec58ff9640 1 -- 172.21.15.60:0/1370444177 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v800) v1 ==== 160+0+0 (secure 0 0 0) 0x7fec50017020 con 0x7fec5c15aaa0 2023-10-16T12:54:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.310+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fec4806af90 msgr2=0x7fec4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.310+0000 7fec62409640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fec4806af90 0x7fec4806d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0098f0 tx=0x7fec4c008040 comp rx=0 tx=0).stop 2023-10-16T12:54:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.310+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 msgr2=0x7fec5c0acb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.310+0000 7fec62409640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 0x7fec5c0acb10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fec5000c4b0 tx=0x7fec5000c970 comp rx=0 tx=0).stop 2023-10-16T12:54:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 shutdown_connections 2023-10-16T12:54:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fec4806af90 0x7fec4806d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec5c15ae80 0x7fec5c070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec5c15aaa0 0x7fec5c0acb10 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 --2- 172.21.15.60:0/1370444177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec5c0ce390 0x7fec5c153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 >> 172.21.15.60:0/1370444177 conn(0x7fec5c0cb410 msgr2=0x7fec5c0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 shutdown_connections 2023-10-16T12:54:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.311+0000 7fec62409640 1 -- 172.21.15.60:0/1370444177 wait complete. 2023-10-16T12:54:04.331 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-10-16T12:54:04.331 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-16T12:54:04.332 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:54:04.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:54:04.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.542+0000 7fc8f1f47640 1 Processor -- start 2023-10-16T12:54:04.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.542+0000 7fc8f1f47640 1 -- start start 2023-10-16T12:54:04.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8f1f47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc8ec0adb50 0x7fc8ec06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8f1f47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8ec070fd0 0x7fc8ec0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8f1f47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec0adf30 0x7fc8ec0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8f1f47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec071450 con 0x7fc8ec0adb50 2023-10-16T12:54:04.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8f1f47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec0cc600 con 0x7fc8ec070fd0 2023-10-16T12:54:04.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8f1f47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0cc8e0 con 0x7fc8ec0adf30 2023-10-16T12:54:04.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8eb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc8ec0adb50 0x7fc8ec06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:04.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8eb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc8ec0adb50 0x7fc8ec06f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58004/0 (socket says 172.21.15.60:58004) 2023-10-16T12:54:04.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.544+0000 7fc8eb7fe640 1 -- 172.21.15.60:0/2639527322 learned_addr learned my addr 172.21.15.60:0/2639527322 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:04.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.545+0000 7fc8eb7fe640 1 -- 172.21.15.60:0/2639527322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec0adf30 msgr2=0x7fc8ec0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:04.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.545+0000 7fc8eb7fe640 1 --2- 172.21.15.60:0/2639527322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec0adf30 0x7fc8ec0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.545+0000 7fc8eb7fe640 1 -- 172.21.15.60:0/2639527322 >> v2:172.21.15.144:3300/0 conn(0x7fc8ec070fd0 msgr2=0x7fc8ec0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:04.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.545+0000 7fc8eb7fe640 1 --2- 172.21.15.60:0/2639527322 >> v2:172.21.15.144:3300/0 conn(0x7fc8ec070fd0 0x7fc8ec0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.545+0000 7fc8eb7fe640 1 -- 172.21.15.60:0/2639527322 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0ccbc0 con 0x7fc8ec0adb50 2023-10-16T12:54:04.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.546+0000 7fc8eb7fe640 1 --2- 172.21.15.60:0/2639527322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec06f550 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0cc2b0 tx=0x7fc8dc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a02f19aa67d9b25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:04.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.546+0000 7fc8eaffd640 1 -- 172.21.15.60:0/2639527322 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc031860 con 0x7fc8ec0adb50 2023-10-16T12:54:04.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.546+0000 7fc8eaffd640 1 -- 172.21.15.60:0/2639527322 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc0319c0 con 0x7fc8ec0adb50 2023-10-16T12:54:04.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.546+0000 7fc8eaffd640 1 -- 172.21.15.60:0/2639527322 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc031d50 con 0x7fc8ec0adb50 2023-10-16T12:54:04.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 -- 172.21.15.60:0/2639527322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adb50 msgr2=0x7fc8ec06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/2639527322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec06f550 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0cc2b0 tx=0x7fc8dc02b900 comp rx=0 tx=0).stop 2023-10-16T12:54:04.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 -- 172.21.15.60:0/2639527322 shutdown_connections 2023-10-16T12:54:04.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/2639527322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec0adf30 0x7fc8ec0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/2639527322 >> v2:172.21.15.144:3300/0 conn(0x7fc8ec070fd0 0x7fc8ec0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/2639527322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec06f550 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 -- 172.21.15.60:0/2639527322 >> 172.21.15.60:0/2639527322 conn(0x7fc8ec0cb410 msgr2=0x7fc8ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:04.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 -- 172.21.15.60:0/2639527322 shutdown_connections 2023-10-16T12:54:04.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.547+0000 7fc8f1f47640 1 -- 172.21.15.60:0/2639527322 wait complete. 2023-10-16T12:54:04.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.549+0000 7fc8f1f47640 1 Processor -- start 2023-10-16T12:54:04.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.549+0000 7fc8f1f47640 1 -- start start 2023-10-16T12:54:04.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8f1f47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec070fd0 0x7fc8ec0c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8f1f47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8f1f47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 0x7fc8ec162c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8f1f47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec162800 con 0x7fc8ec0adf30 2023-10-16T12:54:04.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8f1f47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec161b80 con 0x7fc8ec0adb50 2023-10-16T12:54:04.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8f1f47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec161e60 con 0x7fc8ec070fd0 2023-10-16T12:54:04.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 0x7fc8ec162c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:04.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 0x7fc8ec162c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58020/0 (socket says 172.21.15.60:58020) 2023-10-16T12:54:04.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 -- 172.21.15.60:0/629569119 learned_addr learned my addr 172.21.15.60:0/629569119 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:04.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8e3fff640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:04.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8eb7fe640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec070fd0 0x7fc8ec0c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 -- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec070fd0 msgr2=0x7fc8ec0c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec070fd0 0x7fc8ec0c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 -- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8ec0adb50 msgr2=0x7fc8ec15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.550+0000 7fc8ebfff640 1 -- 172.21.15.60:0/629569119 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0094d0 con 0x7fc8ec0adf30 2023-10-16T12:54:04.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.551+0000 7fc8ebfff640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 0x7fc8ec162c20 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b970 tx=0x7fc8e400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:04.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.551+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400cb40 con 0x7fc8ec0adf30 2023-10-16T12:54:04.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.551+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e400ccc0 con 0x7fc8ec0adf30 2023-10-16T12:54:04.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.551+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4014680 con 0x7fc8ec0adf30 2023-10-16T12:54:04.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.551+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec162140 con 0x7fc8ec0adf30 2023-10-16T12:54:04.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.551+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec15b830 con 0x7fc8ec0adf30 2023-10-16T12:54:04.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.552+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec162940 con 0x7fc8ec0adf30 2023-10-16T12:54:04.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.553+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc8e401d3b0 con 0x7fc8ec0adf30 2023-10-16T12:54:04.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.554+0000 7fc8e8ff9640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:04.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.554+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7fc8e4089700 con 0x7fc8ec0adf30 2023-10-16T12:54:04.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.555+0000 7fc8eb7fe640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:04.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.556+0000 7fc8eb7fe640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fc8d006af90 0x7fc8d006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc036000 tx=0x7fc8dc03c040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:04.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.559+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8ec162940 con 0x7fc8ec0adf30 2023-10-16T12:54:04.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.760+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8ec162940 con 0x7fc8ec0adf30 2023-10-16T12:54:04.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.762+0000 7fc8e8ff9640 1 -- 172.21.15.60:0/629569119 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+14060 (secure 0 0 0) 0x7fc8ec162940 con 0x7fc8ec0adf30 2023-10-16T12:54:04.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:04.765 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":148,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:54:02.280954+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6834/3032734613":"2023-10-17T12:53:59.261984+0000","172.21.15.60:6832/3032734613":"2023-10-17T12:53:59.261984+0000","172.21.15.144:0/3844965859":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/3473219507":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/119007135":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/1667410565":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/285791614":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6833/1139477310":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6832/1139477310":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/1161466656":"2023-10-17T12:53:55.095584+0000","172.21.15.144:6833/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:6832/682650079":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/690478556":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/177518790":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2812310974":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1140080778":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/1196951169":"2023-10-17T12:53:41.484587+0000","172.21.15.144:0/2229533001":"2023-10-17T12:53:55.095584+0000","172.21.15.144:0/325928307":"2023-10-17T12:53:41.484587+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:54:04.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fc8d006af90 msgr2=0x7fc8d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fc8d006af90 0x7fc8d006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc036000 tx=0x7fc8dc03c040 comp rx=0 tx=0).stop 2023-10-16T12:54:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 msgr2=0x7fc8ec162c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 0x7fc8ec162c20 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b970 tx=0x7fc8e400be30 comp rx=0 tx=0).stop 2023-10-16T12:54:04.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 shutdown_connections 2023-10-16T12:54:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fc8d006af90 0x7fc8d006d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8ec0adf30 0x7fc8ec162c20 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8ec0adb50 0x7fc8ec15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 --2- 172.21.15.60:0/629569119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8ec070fd0 0x7fc8ec0c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:04.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 >> 172.21.15.60:0/629569119 conn(0x7fc8ec0cb410 msgr2=0x7fc8ec0ae690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 shutdown_connections 2023-10-16T12:54:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:04.765+0000 7fc8f1f47640 1 -- 172.21.15.60:0/629569119 wait complete. 2023-10-16T12:54:04.788 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:54:04.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:54:05.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.014+0000 7f5569e59640 1 Processor -- start 2023-10-16T12:54:05.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.015+0000 7f5569e59640 1 -- start start 2023-10-16T12:54:05.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.016+0000 7f5569e59640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f55640b3ec0 0x7f55640cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.016+0000 7f5569e59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 0x7f55640cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.016+0000 7f5569e59640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556406be30 con 0x7f55640b43c0 2023-10-16T12:54:05.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.016+0000 7f5569e59640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556406ca00 con 0x7f55640b3ec0 2023-10-16T12:54:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.016+0000 7f5569e59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55640ada10 con 0x7f55640b0ea0 2023-10-16T12:54:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.017+0000 7f5569658640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f55640b43c0 0x7f556415a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34088/0 (socket says 172.21.15.60:34088) 2023-10-16T12:54:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.017+0000 7f5569658640 1 -- 172.21.15.60:0/485729933 learned_addr learned my addr 172.21.15.60:0/485729933 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.017+0000 7f5563fff640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 0x7f55640cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.017+0000 7f5568e57640 1 --2- 172.21.15.60:0/485729933 >> v2:172.21.15.144:3300/0 conn(0x7f55640b3ec0 0x7f55640cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.017+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344146086 0 0) 0x7f556406be30 con 0x7f55640b43c0 2023-10-16T12:54:05.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.017+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5554003520 con 0x7f55640b43c0 2023-10-16T12:54:05.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555057308 0 0) 0x7f5554003520 con 0x7f55640b43c0 2023-10-16T12:54:05.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 msgr2=0x7f55640cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 0x7f55640cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 >> v2:172.21.15.144:3300/0 conn(0x7f55640b3ec0 msgr2=0x7f55640cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 --2- 172.21.15.60:0/485729933 >> v2:172.21.15.144:3300/0 conn(0x7f55640b3ec0 0x7f55640cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556406c060 con 0x7f55640b43c0 2023-10-16T12:54:05.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f555c0019f0 con 0x7f55640b43c0 2023-10-16T12:54:05.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 >> v1:172.21.15.60:6789/0 conn(0x7f55640b43c0 legacy=0x7f556415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 0x7f5554003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5554003f40 0x7f55540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554004d00 0x7f5554005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556406ca00 con 0x7f5554003f40 2023-10-16T12:54:05.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55640ada10 con 0x7f5554004d00 2023-10-16T12:54:05.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555400ba30 con 0x7f55640b0ea0 2023-10-16T12:54:05.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f5563fff640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5554003f40 0x7f55540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.018+0000 7f5568e57640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 0x7f5554003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5569658640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554004d00 0x7f5554005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5563fff640 1 -- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 msgr2=0x7f5554003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5563fff640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55640b0ea0 0x7f5554003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5563fff640 1 -- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554004d00 msgr2=0x7f5554005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5563fff640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554004d00 0x7f5554005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5563fff640 1 -- 172.21.15.60:0/485729933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5554003520 con 0x7f5554003f40 2023-10-16T12:54:05.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.019+0000 7f5569658640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554004d00 0x7f5554005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.020+0000 7f5563fff640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5554003f40 0x7f55540047c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5550009ac0 tx=0x7f55540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b28761fee26df89 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.020+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555002c6b0 con 0x7f5554003f40 2023-10-16T12:54:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.020+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555002c810 con 0x7f5554003f40 2023-10-16T12:54:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.020+0000 7f55637fe640 1 -- 172.21.15.60:0/485729933 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555002cba0 con 0x7f5554003f40 2023-10-16T12:54:05.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569e59640 1 -- 172.21.15.60:0/485729933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5554003f40 msgr2=0x7f55540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569e59640 1 --2- 172.21.15.60:0/485729933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5554003f40 0x7f55540047c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5550009ac0 tx=0x7f55540057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569658640 1 -- 172.21.15.60:0/485729933 reap_dead start 2023-10-16T12:54:05.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569e59640 1 -- 172.21.15.60:0/485729933 shutdown_connections 2023-10-16T12:54:05.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569e59640 1 -- 172.21.15.60:0/485729933 >> 172.21.15.60:0/485729933 conn(0x7f55640b0170 msgr2=0x7f55640b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:05.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569e59640 1 -- 172.21.15.60:0/485729933 shutdown_connections 2023-10-16T12:54:05.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.021+0000 7f5569e59640 1 -- 172.21.15.60:0/485729933 wait complete. 2023-10-16T12:54:05.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.023+0000 7f5569e59640 1 Processor -- start 2023-10-16T12:54:05.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.023+0000 7f5569e59640 1 -- start start 2023-10-16T12:54:05.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569e59640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554003f40 0x7f55640c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569e59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55641f2ec0 0x7f55641f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569e59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 0x7f55640ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569e59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55640c8640 con 0x7f5564153b70 2023-10-16T12:54:05.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569e59640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55641f5f40 con 0x7f5554003f40 2023-10-16T12:54:05.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569e59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55641f6910 con 0x7f55641f2ec0 2023-10-16T12:54:05.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569658640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 0x7f55640ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569658640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 0x7f55640ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58028/0 (socket says 172.21.15.60:58028) 2023-10-16T12:54:05.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.024+0000 7f5569658640 1 -- 172.21.15.60:0/27973689 learned_addr learned my addr 172.21.15.60:0/27973689 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:05.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5568e57640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554003f40 0x7f55640c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5563fff640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55641f2ec0 0x7f55641f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569658640 1 -- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55641f2ec0 msgr2=0x7f55641f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569658640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55641f2ec0 0x7f55641f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569658640 1 -- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554003f40 msgr2=0x7f55640c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569658640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554003f40 0x7f55640c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569658640 1 -- 172.21.15.60:0/27973689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55641f3340 con 0x7f5564153b70 2023-10-16T12:54:05.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5563fff640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55641f2ec0 0x7f55641f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:05.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569658640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 0x7f55640ac890 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f5564074310 tx=0x7f555c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:05.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c00bb20 con 0x7f5564153b70 2023-10-16T12:54:05.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c00bc80 con 0x7f5564153b70 2023-10-16T12:54:05.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55640c8080 con 0x7f5564153b70 2023-10-16T12:54:05.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.025+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c007070 con 0x7f5564153b70 2023-10-16T12:54:05.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.026+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55641f5bd0 con 0x7f5564153b70 2023-10-16T12:54:05.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.027+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55641f5d90 con 0x7f5564153b70 2023-10-16T12:54:05.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.028+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f555c0071d0 con 0x7f5564153b70 2023-10-16T12:54:05.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.029+0000 7f55617fa640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f554806afc0 0x7f554806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.029+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7f555c089c10 con 0x7f5564153b70 2023-10-16T12:54:05.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.029+0000 7f5568e57640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f554806afc0 0x7f554806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.030+0000 7f5568e57640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f554806afc0 0x7f554806d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f55640ce020 tx=0x7f5558008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:05.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.034+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f555c054ac0 con 0x7f5564153b70 2023-10-16T12:54:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.227+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55641f5a30 con 0x7f5564153b70 2023-10-16T12:54:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.228+0000 7f55617fa640 1 -- 172.21.15.60:0/27973689 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55641f5a30 con 0x7f5564153b70 2023-10-16T12:54:05.231 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:54:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.230+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f554806afc0 msgr2=0x7f554806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f554806afc0 0x7f554806d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f55640ce020 tx=0x7f5558008040 comp rx=0 tx=0).stop 2023-10-16T12:54:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 msgr2=0x7f55640ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 0x7f55640ac890 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f5564074310 tx=0x7f555c00b4a0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 shutdown_connections 2023-10-16T12:54:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f554806afc0 0x7f554806d480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55641f2ec0 0x7f55641f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5564153b70 0x7f55640ac890 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 --2- 172.21.15.60:0/27973689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5554003f40 0x7f55640c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 >> 172.21.15.60:0/27973689 conn(0x7f55640b0170 msgr2=0x7f55640af400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 shutdown_connections 2023-10-16T12:54:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.231+0000 7f5569e59640 1 -- 172.21.15.60:0/27973689 wait complete. 2023-10-16T12:54:05.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T12:54:05.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.478+0000 7f44dd02d640 1 Processor -- start 2023-10-16T12:54:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.478+0000 7f44dd02d640 1 -- start start 2023-10-16T12:54:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.479+0000 7f44dd02d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44d8159b20 0x7f44d806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44dd02d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 0x7f44d80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44dd02d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d806f7f0 con 0x7f44d806c120 2023-10-16T12:54:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44dd02d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d806c5e0 con 0x7f44d8159b20 2023-10-16T12:54:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44dd02d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d806c8c0 con 0x7f44d8159f00 2023-10-16T12:54:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44d759e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f44d806c120 0x7f44d80cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34104/0 (socket says 172.21.15.60:34104) 2023-10-16T12:54:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44d759e640 1 -- 172.21.15.60:0/4294657002 learned_addr learned my addr 172.21.15.60:0/4294657002 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:05.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/4294657002 >> v2:172.21.15.144:3300/0 conn(0x7f44d8159b20 0x7f44d806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.480+0000 7f44cffff640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 0x7f44d80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735322259 0 0) 0x7f44d806f7f0 con 0x7f44d806c120 2023-10-16T12:54:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c4003520 con 0x7f44d806c120 2023-10-16T12:54:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 35811952 0 0) 0x7f44c4003520 con 0x7f44d806c120 2023-10-16T12:54:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 msgr2=0x7f44d80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 0x7f44d80b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 >> v2:172.21.15.144:3300/0 conn(0x7f44d8159b20 msgr2=0x7f44d806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 --2- 172.21.15.60:0/4294657002 >> v2:172.21.15.144:3300/0 conn(0x7f44d8159b20 0x7f44d806bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d806cba0 con 0x7f44d806c120 2023-10-16T12:54:05.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f44d00019f0 con 0x7f44d806c120 2023-10-16T12:54:05.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 >> v1:172.21.15.60:6789/0 conn(0x7f44d806c120 legacy=0x7f44d80cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c4003970 0x7f44c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c40047c0 0x7f44c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 0x7f44c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d806c5e0 con 0x7f44c4003970 2023-10-16T12:54:05.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d806c8c0 con 0x7f44c40047c0 2023-10-16T12:54:05.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c400b980 con 0x7f44d8159f00 2023-10-16T12:54:05.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.481+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c4003970 0x7f44c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.482+0000 7f44cffff640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c40047c0 0x7f44c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.482+0000 7f44d6d9d640 1 -- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 msgr2=0x7f44c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.482+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d8159f00 0x7f44c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.482+0000 7f44d6d9d640 1 -- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c40047c0 msgr2=0x7f44c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:05.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.482+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c40047c0 0x7f44c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.482+0000 7f44d6d9d640 1 -- 172.21.15.60:0/4294657002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c4003520 con 0x7f44c4003970 2023-10-16T12:54:05.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.483+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c4003970 0x7f44c4003d90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009ac0 tx=0x7f44c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32212b1ef6a80aae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:05.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c802c6d0 con 0x7f44c4003970 2023-10-16T12:54:05.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c802c830 con 0x7f44c4003970 2023-10-16T12:54:05.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44d659c640 1 -- 172.21.15.60:0/4294657002 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c802cbc0 con 0x7f44c4003970 2023-10-16T12:54:05.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44dd02d640 1 -- 172.21.15.60:0/4294657002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c4003970 msgr2=0x7f44c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44dd02d640 1 --2- 172.21.15.60:0/4294657002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c4003970 0x7f44c4003d90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009ac0 tx=0x7f44c40057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44d759e640 1 -- 172.21.15.60:0/4294657002 reap_dead start 2023-10-16T12:54:05.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44dd02d640 1 -- 172.21.15.60:0/4294657002 shutdown_connections 2023-10-16T12:54:05.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44dd02d640 1 -- 172.21.15.60:0/4294657002 >> 172.21.15.60:0/4294657002 conn(0x7f44d80cb410 msgr2=0x7f44d80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:05.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44dd02d640 1 -- 172.21.15.60:0/4294657002 shutdown_connections 2023-10-16T12:54:05.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.484+0000 7f44dd02d640 1 -- 172.21.15.60:0/4294657002 wait complete. 2023-10-16T12:54:05.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.486+0000 7f44dd02d640 1 Processor -- start 2023-10-16T12:54:05.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.486+0000 7f44dd02d640 1 -- start start 2023-10-16T12:54:05.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.487+0000 7f44dd02d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c4003970 0x7f44d80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.487+0000 7f44dd02d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d80c5c90 0x7f44d8153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.487+0000 7f44dd02d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 0x7f44d80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.487+0000 7f44dd02d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80b2de0 con 0x7f44d80c6650 2023-10-16T12:54:05.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.487+0000 7f44dd02d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80c4ee0 con 0x7f44c4003970 2023-10-16T12:54:05.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.487+0000 7f44dd02d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80b1ce0 con 0x7f44d80c5c90 2023-10-16T12:54:05.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 0x7f44d80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 0x7f44d80c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58034/0 (socket says 172.21.15.60:58034) 2023-10-16T12:54:05.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 -- 172.21.15.60:0/862659035 learned_addr learned my addr 172.21.15.60:0/862659035 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:05.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c4003970 0x7f44d80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44cffff640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d80c5c90 0x7f44d8153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 -- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d80c5c90 msgr2=0x7f44d8153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d80c5c90 0x7f44d8153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 -- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c4003970 msgr2=0x7f44d80c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:05.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c4003970 0x7f44d80c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:05.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 -- 172.21.15.60:0/862659035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d80c7020 con 0x7f44d80c6650 2023-10-16T12:54:05.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.488+0000 7f44d759e640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 0x7f44d80c6ae0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f44d80c5650 tx=0x7f44d000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:05.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.489+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000a610 con 0x7f44d80c6650 2023-10-16T12:54:05.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.489+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d000a770 con 0x7f44d80c6650 2023-10-16T12:54:05.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.489+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000ab20 con 0x7f44d80c6650 2023-10-16T12:54:05.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.489+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d80b2740 con 0x7f44d80c6650 2023-10-16T12:54:05.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.489+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d80b2210 con 0x7f44d80c6650 2023-10-16T12:54:05.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.490+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d80b2f20 con 0x7f44d80c6650 2023-10-16T12:54:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.491+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f44d000ac80 con 0x7f44d80c6650 2023-10-16T12:54:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.492+0000 7f44cf7fe640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f44b806af80 0x7f44b806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.492+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7f44d0088f70 con 0x7f44d80c6650 2023-10-16T12:54:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.493+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f44b806af80 0x7f44b806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:05.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.494+0000 7f44d6d9d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f44b806af80 0x7f44b806d440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44c8036000 tx=0x7f44c8030000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:05.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.497+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44d80b2f20 con 0x7f44d80c6650 2023-10-16T12:54:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:05.766+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f44d80b2f20 con 0x7f44d80c6650 2023-10-16T12:54:06.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.313+0000 7f44cf7fe640 1 -- 172.21.15.60:0/862659035 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v802) v1 ==== 181+0+0 (secure 0 0 0) 0x7f44d80b2f20 con 0x7f44d80c6650 2023-10-16T12:54:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f44b806af80 msgr2=0x7f44b806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f44b806af80 0x7f44b806d440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44c8036000 tx=0x7f44c8030000 comp rx=0 tx=0).stop 2023-10-16T12:54:06.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 msgr2=0x7f44d80c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 0x7f44d80c6ae0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f44d80c5650 tx=0x7f44d000ccd0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 shutdown_connections 2023-10-16T12:54:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f44b806af80 0x7f44b806d440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d80c5c90 0x7f44d8153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44d80c6650 0x7f44d80c6ae0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 --2- 172.21.15.60:0/862659035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c4003970 0x7f44d80c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 >> 172.21.15.60:0/862659035 conn(0x7f44d80cb410 msgr2=0x7f44d80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.316+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 shutdown_connections 2023-10-16T12:54:06.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.317+0000 7f44dd02d640 1 -- 172.21.15.60:0/862659035 wait complete. 2023-10-16T12:54:06.339 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:54:06.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.557+0000 7f4a9f965640 1 Processor -- start 2023-10-16T12:54:06.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.557+0000 7f4a9f965640 1 -- start start 2023-10-16T12:54:06.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9f965640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a980ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9f965640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a98070360 con 0x7f4a9806c140 2023-10-16T12:54:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9f965640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980cd1f0 con 0x7f4a98159e90 2023-10-16T12:54:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9f965640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9815d890 con 0x7f4a98159ab0 2023-10-16T12:54:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9df02640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4a9806c140 0x7f4a9815dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34114/0 (socket says 172.21.15.60:34114) 2023-10-16T12:54:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9df02640 1 -- 172.21.15.60:0/829097164 learned_addr learned my addr 172.21.15.60:0/829097164 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a9d701640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a980ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330404104 0 0) 0x7f4a98070360 con 0x7f4a9806c140 2023-10-16T12:54:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.559+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a84005730 con 0x7f4a9806c140 2023-10-16T12:54:06.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4240413919 0 0) 0x7f4a84005730 con 0x7f4a9806c140 2023-10-16T12:54:06.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 msgr2=0x7f4a980ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:06.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a980ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 >> v1:172.21.15.144:6789/0 conn(0x7f4a98159e90 legacy=0x7f4a9806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980cd4d0 con 0x7f4a9806c140 2023-10-16T12:54:06.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4a900019f0 con 0x7f4a9806c140 2023-10-16T12:54:06.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 >> v1:172.21.15.60:6789/0 conn(0x7f4a9806c140 legacy=0x7f4a9815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a84005bb0 0x7f4a84005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840069d0 0x7f4a84006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a84007460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9815d890 con 0x7f4a84005bb0 2023-10-16T12:54:06.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8400dd50 con 0x7f4a840069d0 2023-10-16T12:54:06.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8400df10 con 0x7f4a98159ab0 2023-10-16T12:54:06.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a9d701640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a84005bb0 0x7f4a84005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.560+0000 7f4a9df02640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a84007460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9cf00640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840069d0 0x7f4a84006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9d701640 1 -- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 msgr2=0x7f4a84007460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9d701640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a84007460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9d701640 1 -- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840069d0 msgr2=0x7f4a84006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9d701640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a840069d0 0x7f4a84006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9d701640 1 -- 172.21.15.60:0/829097164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84005730 con 0x7f4a84005bb0 2023-10-16T12:54:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.561+0000 7f4a9df02640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a98159ab0 0x7f4a84007460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.562+0000 7f4a9d701640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a84005bb0 0x7f4a84005fd0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009ac0 tx=0x7f4a84007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a740ef9f93669b5b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.562+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9402c6b0 con 0x7f4a84005bb0 2023-10-16T12:54:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.562+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9402c810 con 0x7f4a84005bb0 2023-10-16T12:54:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.562+0000 7f4a8ffff640 1 -- 172.21.15.60:0/829097164 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9402cba0 con 0x7f4a84005bb0 2023-10-16T12:54:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.562+0000 7f4a9f965640 1 -- 172.21.15.60:0/829097164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a84005bb0 msgr2=0x7f4a84005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.562+0000 7f4a9f965640 1 --2- 172.21.15.60:0/829097164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a84005bb0 0x7f4a84005fd0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009ac0 tx=0x7f4a84007a00 comp rx=0 tx=0).stop 2023-10-16T12:54:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.563+0000 7f4a9df02640 1 -- 172.21.15.60:0/829097164 reap_dead start 2023-10-16T12:54:06.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.563+0000 7f4a9f965640 1 -- 172.21.15.60:0/829097164 shutdown_connections 2023-10-16T12:54:06.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.563+0000 7f4a9f965640 1 -- 172.21.15.60:0/829097164 >> 172.21.15.60:0/829097164 conn(0x7f4a980ce9b0 msgr2=0x7f4a980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:06.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.563+0000 7f4a9f965640 1 -- 172.21.15.60:0/829097164 shutdown_connections 2023-10-16T12:54:06.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.563+0000 7f4a9f965640 1 -- 172.21.15.60:0/829097164 wait complete. 2023-10-16T12:54:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.565+0000 7f4a9f965640 1 Processor -- start 2023-10-16T12:54:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.565+0000 7f4a9f965640 1 -- start start 2023-10-16T12:54:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9f965640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a98157a90 0x7f4a980c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9f965640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 0x7f4a980c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9f965640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9815c800 0x7f4a9815cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9f965640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980c4cd0 con 0x7f4a980c7a90 2023-10-16T12:54:06.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9f965640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980c8450 con 0x7f4a98157a90 2023-10-16T12:54:06.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9f965640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9815d1c0 con 0x7f4a9815c800 2023-10-16T12:54:06.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 0x7f4a980c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 0x7f4a980c7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58042/0 (socket says 172.21.15.60:58042) 2023-10-16T12:54:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 -- 172.21.15.60:0/4106716988 learned_addr learned my addr 172.21.15.60:0/4106716988 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 -- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9815c800 msgr2=0x7f4a9815cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9815c800 0x7f4a9815cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9d701640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a98157a90 0x7f4a980c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 -- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a98157a90 msgr2=0x7f4a980c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a98157a90 0x7f4a980c27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.566+0000 7f4a9cf00640 1 -- 172.21.15.60:0/4106716988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980b15d0 con 0x7f4a980c7a90 2023-10-16T12:54:06.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.567+0000 7f4a9cf00640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 0x7f4a980c7f10 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800d570 tx=0x7f4a8800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.567+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88005480 con 0x7f4a980c7a90 2023-10-16T12:54:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.567+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88003610 con 0x7f4a980c7a90 2023-10-16T12:54:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.567+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88003980 con 0x7f4a980c7a90 2023-10-16T12:54:06.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.567+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a980b2430 con 0x7f4a980c7a90 2023-10-16T12:54:06.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.567+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a980b1b00 con 0x7f4a980c7a90 2023-10-16T12:54:06.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.568+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a980b25c0 con 0x7f4a980c7a90 2023-10-16T12:54:06.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.569+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a88003ae0 con 0x7f4a980c7a90 2023-10-16T12:54:06.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.570+0000 7f4a8dffb640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4a7c06afc0 0x7f4a7c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:06.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.570+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7f4a880896f0 con 0x7f4a980c7a90 2023-10-16T12:54:06.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.571+0000 7f4a9d701640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4a7c06afc0 0x7f4a7c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:06.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.572+0000 7f4a9d701640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4a7c06afc0 0x7f4a7c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a94036000 tx=0x7f4a94030000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:06.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.574+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a88054590 con 0x7f4a980c7a90 2023-10-16T12:54:06.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.772+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4a9804e160 con 0x7f4a980c7a90 2023-10-16T12:54:06.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.778+0000 7f4a8dffb640 1 -- 172.21.15.60:0/4106716988 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4a88001230 con 0x7f4a980c7a90 2023-10-16T12:54:06.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.780+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4a7c06afc0 msgr2=0x7f4a7c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.780+0000 7f4a9f965640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4a7c06afc0 0x7f4a7c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a94036000 tx=0x7f4a94030000 comp rx=0 tx=0).stop 2023-10-16T12:54:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.780+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 msgr2=0x7f4a980c7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.780+0000 7f4a9f965640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 0x7f4a980c7f10 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800d570 tx=0x7f4a8800da30 comp rx=0 tx=0).stop 2023-10-16T12:54:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 shutdown_connections 2023-10-16T12:54:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f4a7c06afc0 0x7f4a7c06d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a9815c800 0x7f4a9815cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a980c7a90 0x7f4a980c7f10 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 --2- 172.21.15.60:0/4106716988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a98157a90 0x7f4a980c27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 >> 172.21.15.60:0/4106716988 conn(0x7f4a980ce9b0 msgr2=0x7f4a980af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 shutdown_connections 2023-10-16T12:54:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:06.781+0000 7f4a9f965640 1 -- 172.21.15.60:0/4106716988 wait complete. 2023-10-16T12:54:06.803 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:54:06.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:54:06.928 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:54:07.066 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:54:07.067 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:54:07.067 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:54:07.132 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:54:07.132 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:54:07.133 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:54:07.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:54:07.194 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:54:07.194 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:54:07.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T12:54:07.258 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T12:54:07.258 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:54:07.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T12:54:07.320 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:54:07.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:54:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.555+0000 7fe5fc882640 1 Processor -- start 2023-10-16T12:54:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.556+0000 7fe5fc882640 1 -- start start 2023-10-16T12:54:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.557+0000 7fe5fc882640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f806fa20 0x7fe5f80ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.557+0000 7fe5fc882640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f806d900 con 0x7fe5f806d520 2023-10-16T12:54:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.557+0000 7fe5fc882640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f80b24f0 con 0x7fe5f8065cc0 2023-10-16T12:54:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.557+0000 7fe5fc882640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f815b1d0 con 0x7fe5f806fa20 2023-10-16T12:54:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.558+0000 7fe5f6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe5f806d520 0x7fe5f80b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34120/0 (socket says 172.21.15.60:34120) 2023-10-16T12:54:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.558+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/4152470295 learned_addr learned my addr 172.21.15.60:0/4152470295 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.558+0000 7fe5ee59c640 1 --2- 172.21.15.60:0/4152470295 >> v2:172.21.15.144:3300/0 conn(0x7fe5f806fa20 0x7fe5f80ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.558+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189143998 0 0) 0x7fe5f806d900 con 0x7fe5f806d520 2023-10-16T12:54:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.558+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e4003520 con 0x7fe5f806d520 2023-10-16T12:54:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2836689116 0 0) 0x7fe5e4003520 con 0x7fe5f806d520 2023-10-16T12:54:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 >> v2:172.21.15.144:3300/0 conn(0x7fe5f806fa20 msgr2=0x7fe5f80ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 --2- 172.21.15.60:0/4152470295 >> v2:172.21.15.144:3300/0 conn(0x7fe5f806fa20 0x7fe5f80ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 >> v1:172.21.15.144:6789/0 conn(0x7fe5f8065cc0 legacy=0x7fe5f8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f806feb0 con 0x7fe5f806d520 2023-10-16T12:54:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe5e8001670 con 0x7fe5f806d520 2023-10-16T12:54:07.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 >> v1:172.21.15.60:6789/0 conn(0x7fe5f806d520 legacy=0x7fe5f80b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5e40047c0 0x7fe5e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5e40052a0 0x7fe5e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f815b1d0 con 0x7fe5e4003970 2023-10-16T12:54:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e400c170 con 0x7fe5e40047c0 2023-10-16T12:54:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e400c330 con 0x7fe5e40052a0 2023-10-16T12:54:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.559+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5ee59c640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5e40047c0 0x7fe5e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f759e640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5e40052a0 0x7fe5e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5e40052a0 msgr2=0x7fe5e400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5e40052a0 0x7fe5e400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5e40047c0 msgr2=0x7fe5e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5e40047c0 0x7fe5e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/4152470295 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4003520 con 0x7fe5e4003970 2023-10-16T12:54:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.560+0000 7fe5f759e640 1 -- 172.21.15.60:0/4152470295 reap_dead start 2023-10-16T12:54:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.561+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5e4003d90 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c210 tx=0x7fe5e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e4c2d8f1d5b0b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.561+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8005ad0 con 0x7fe5e4003970 2023-10-16T12:54:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.561+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e8005c30 con 0x7fe5e4003970 2023-10-16T12:54:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.561+0000 7fe5f659c640 1 -- 172.21.15.60:0/4152470295 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802cab0 con 0x7fe5e4003970 2023-10-16T12:54:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 -- 172.21.15.60:0/4152470295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 msgr2=0x7fe5e4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5e4003d90 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c210 tx=0x7fe5e40057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 -- 172.21.15.60:0/4152470295 shutdown_connections 2023-10-16T12:54:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 --2- 172.21.15.60:0/4152470295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5e4003d90 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 -- 172.21.15.60:0/4152470295 >> 172.21.15.60:0/4152470295 conn(0x7fe5f80b0a80 msgr2=0x7fe5f80b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 -- 172.21.15.60:0/4152470295 shutdown_connections 2023-10-16T12:54:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.562+0000 7fe5fc882640 1 -- 172.21.15.60:0/4152470295 wait complete. 2023-10-16T12:54:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.564+0000 7fe5fc882640 1 Processor -- start 2023-10-16T12:54:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.564+0000 7fe5fc882640 1 -- start start 2023-10-16T12:54:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5fc882640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5f80cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5fc882640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80ae8f0 0x7fe5f80aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5fc882640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80c60d0 0x7fe5f80c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5fc882640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80af2b0 con 0x7fe5e4003970 2023-10-16T12:54:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5fc882640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c7ec0 con 0x7fe5f80c60d0 2023-10-16T12:54:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5fc882640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c7a30 con 0x7fe5f80ae8f0 2023-10-16T12:54:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5f6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5f80cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5f6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5f80cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58066/0 (socket says 172.21.15.60:58066) 2023-10-16T12:54:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/3441828859 learned_addr learned my addr 172.21.15.60:0/3441828859 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5ee59c640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80ae8f0 0x7fe5f80aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.565+0000 7fe5f759e640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80c60d0 0x7fe5f80c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80ae8f0 msgr2=0x7fe5f80aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80ae8f0 0x7fe5f80aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80c60d0 msgr2=0x7fe5f80c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80c60d0 0x7fe5f80c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f6d9d640 1 -- 172.21.15.60:0/3441828859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80c6e20 con 0x7fe5e4003970 2023-10-16T12:54:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f759e640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80c60d0 0x7fe5f80c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5f6d9d640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5f80cb1a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c210 tx=0x7fe5e802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e803c040 con 0x7fe5e4003970 2023-10-16T12:54:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e802cc80 con 0x7fe5e4003970 2023-10-16T12:54:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8007070 con 0x7fe5e4003970 2023-10-16T12:54:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.566+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f80c7100 con 0x7fe5e4003970 2023-10-16T12:54:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.567+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f80c76c0 con 0x7fe5e4003970 2023-10-16T12:54:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.568+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f804e9b0 con 0x7fe5e4003970 2023-10-16T12:54:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.569+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe5e8032070 con 0x7fe5e4003970 2023-10-16T12:54:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.570+0000 7fe5effff640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5d806af90 0x7fe5d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.570+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7fe5e80aabb0 con 0x7fe5e4003970 2023-10-16T12:54:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.570+0000 7fe5ee59c640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5d806af90 0x7fe5d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.571+0000 7fe5ee59c640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5d806af90 0x7fe5d806d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80b0090 tx=0x7fe5e0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.574+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5e8075bf0 con 0x7fe5e4003970 2023-10-16T12:54:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.808+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5f80cec80 con 0x7fe5e4003970 2023-10-16T12:54:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.808+0000 7fe5effff640 1 -- 172.21.15.60:0/3441828859 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fe5e8079210 con 0x7fe5e4003970 2023-10-16T12:54:07.811 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:07.811 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6380,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3764120693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3764120693},{"type":"v1","addr":"172.21.15.60:6835","nonce":3764120693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3002524477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3002524477},{"type":"v1","addr":"172.21.15.144:6836","nonce":3002524477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1367074777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1367074777},{"type":"v1","addr":"172.21.15.144:6837","nonce":1367074777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8856,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/867331994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":867331994},{"type":"v1","addr":"172.21.15.60:6834","nonce":867331994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[]} 2023-10-16T12:54:07.812 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 72 2023-10-16T12:54:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.811+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5d806af90 msgr2=0x7fe5d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.811+0000 7fe5fc882640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5d806af90 0x7fe5d806d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80b0090 tx=0x7fe5e0008040 comp rx=0 tx=0).stop 2023-10-16T12:54:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.811+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 msgr2=0x7fe5f80cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.811+0000 7fe5fc882640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5f80cb1a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c210 tx=0x7fe5e802e070 comp rx=0 tx=0).stop 2023-10-16T12:54:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 shutdown_connections 2023-10-16T12:54:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5d806af90 0x7fe5d806d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe5f80ae8f0 0x7fe5f80aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f80c60d0 0x7fe5f80c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 --2- 172.21.15.60:0/3441828859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5e4003970 0x7fe5f80cb1a0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 >> 172.21.15.60:0/3441828859 conn(0x7fe5f80b0a80 msgr2=0x7fe5f80c1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 shutdown_connections 2023-10-16T12:54:07.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:07.812+0000 7fe5fc882640 1 -- 172.21.15.60:0/3441828859 wait complete. 2023-10-16T12:54:07.833 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:54:07.834 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:54:07.834 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:54:07.834 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:54:07.835 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:54:07.835 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:54:07.835 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:54:07.836 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:54:07.836 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:54:07.836 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:54:07.836 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:54:07.837 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:54:07.837 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:54:07.837 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:54:07.837 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:54:07.838 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:54:07.851 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:54:07.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:54:07.853 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:54:07.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:54:07.856 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:54:07.856 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:54:07.857 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:54:07.857 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:54:07.901 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:54:07.902 DEBUG:teuthology.parallel:result is None 2023-10-16T12:54:07.906 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:54:07.907 DEBUG:teuthology.parallel:result is None 2023-10-16T12:54:07.909 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:54:07.910 DEBUG:teuthology.parallel:result is None 2023-10-16T12:54:07.913 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:54:07.913 DEBUG:teuthology.parallel:result is None 2023-10-16T12:54:07.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:54:08.068 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T12:54:08.070 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T12:54:08.072 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T12:54:08.078 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T12:54:08.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.107+0000 7fbe50b01640 1 Processor -- start 2023-10-16T12:54:08.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.107+0000 7fbe50b01640 1 -- start start 2023-10-16T12:54:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe50b01640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe50b01640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0cb030 con 0x7fbe4c06a4b0 2023-10-16T12:54:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe50b01640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0c91c0 con 0x7fbe4c15ad70 2023-10-16T12:54:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe50b01640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c0c94a0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe43fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbe4c06a4b0 0x7fbe4c0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34140/0 (socket says 172.21.15.60:34140) 2023-10-16T12:54:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe43fff640 1 -- 172.21.15.60:0/725962238 learned_addr learned my addr 172.21.15.60:0/725962238 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:08.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe4ad9d640 1 --2- 172.21.15.60:0/725962238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.109+0000 7fbe49d9b640 1 -- 172.21.15.60:0/725962238 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789339376 0 0) 0x7fbe4c0cb030 con 0x7fbe4c06a4b0 2023-10-16T12:54:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.110+0000 7fbe49d9b640 1 -- 172.21.15.60:0/725962238 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38003520 con 0x7fbe4c06a4b0 2023-10-16T12:54:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.110+0000 7fbe4ad9d640 1 -- 172.21.15.60:0/725962238 >> v1:172.21.15.144:6789/0 conn(0x7fbe4c15ad70 legacy=0x7fbe4c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.110+0000 7fbe49d9b640 1 -- 172.21.15.60:0/725962238 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393071399 0 0) 0x7fbe4c0c91c0 con 0x7fbe4c15ad70 2023-10-16T12:54:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.110+0000 7fbe4ad9d640 1 -- 172.21.15.60:0/725962238 >> v1:172.21.15.60:6789/0 conn(0x7fbe4c06a4b0 legacy=0x7fbe4c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.110+0000 7fbe4ad9d640 1 -- 172.21.15.60:0/725962238 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c0c9780 con 0x7fbe4c0caac0 2023-10-16T12:54:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.111+0000 7fbe4ad9d640 1 --2- 172.21.15.60:0/725962238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c0ad4d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009dc0 tx=0x7fbe4402bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba938322eac19db0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.111+0000 7fbe49d9b640 1 -- 172.21.15.60:0/725962238 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4402ccc0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.111+0000 7fbe49d9b640 1 -- 172.21.15.60:0/725962238 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4402ce20 con 0x7fbe4c0caac0 2023-10-16T12:54:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.111+0000 7fbe49d9b640 1 -- 172.21.15.60:0/725962238 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44034640 con 0x7fbe4c0caac0 2023-10-16T12:54:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 -- 172.21.15.60:0/725962238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 msgr2=0x7fbe4c0ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 --2- 172.21.15.60:0/725962238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c0ad4d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009dc0 tx=0x7fbe4402bf10 comp rx=0 tx=0).stop 2023-10-16T12:54:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 -- 172.21.15.60:0/725962238 shutdown_connections 2023-10-16T12:54:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 --2- 172.21.15.60:0/725962238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c0ad4d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 -- 172.21.15.60:0/725962238 >> 172.21.15.60:0/725962238 conn(0x7fbe4c0ce9b0 msgr2=0x7fbe4c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 -- 172.21.15.60:0/725962238 shutdown_connections 2023-10-16T12:54:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.112+0000 7fbe50b01640 1 -- 172.21.15.60:0/725962238 wait complete. 2023-10-16T12:54:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.114+0000 7fbe50b01640 1 Processor -- start 2023-10-16T12:54:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.114+0000 7fbe50b01640 1 -- start start 2023-10-16T12:54:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe50b01640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c06a4b0 0x7fbe4c1f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe50b01640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c15d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:08.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe50b01640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe4c15ad70 0x7fbe4c0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe50b01640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c1f2380 con 0x7fbe4c06a4b0 2023-10-16T12:54:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe50b01640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c15daf0 con 0x7fbe4c15ad70 2023-10-16T12:54:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe50b01640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c15c7f0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe43fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c15d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe4a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c06a4b0 0x7fbe4c1f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe43fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c15d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:42640/0 (socket says 172.21.15.60:42640) 2023-10-16T12:54:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe43fff640 1 -- 172.21.15.60:0/2077455890 learned_addr learned my addr 172.21.15.60:0/2077455890 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.115+0000 7fbe4ad9d640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe4c15ad70 0x7fbe4c0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe43fff640 1 -- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe4c15ad70 msgr2=0x7fbe4c0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe43fff640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe4c15ad70 0x7fbe4c0b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe43fff640 1 -- 172.21.15.60:0/2077455890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c06a4b0 msgr2=0x7fbe4c1f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe43fff640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c06a4b0 0x7fbe4c1f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe43fff640 1 -- 172.21.15.60:0/2077455890 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44009a70 con 0x7fbe4c0caac0 2023-10-16T12:54:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe43fff640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c15d5b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbe3400c460 tx=0x7fbe3400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:08.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34005ce0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe34005e40 con 0x7fbe4c0caac0 2023-10-16T12:54:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34007290 con 0x7fbe4c0caac0 2023-10-16T12:54:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c15cad0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.116+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c15d090 con 0x7fbe4c0caac0 2023-10-16T12:54:08.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.117+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c15d2b0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.120+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe340073f0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.120+0000 7fbe42ffd640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe2c06af90 0x7fbe2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:08.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.120+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5674+0+0 (secure 0 0 0) 0x7fbe34088d20 con 0x7fbe4c0caac0 2023-10-16T12:54:08.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.120+0000 7fbe4a59c640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe2c06af90 0x7fbe2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:08.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.122+0000 7fbe4a59c640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe2c06af90 0x7fbe2c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c000f60 tx=0x7fbe3c009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:08.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.130+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe34053cf0 con 0x7fbe4c0caac0 2023-10-16T12:54:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.337+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbe4c15c930 con 0x7fbe4c0caac0 2023-10-16T12:54:08.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.808+0000 7fbe42ffd640 1 -- 172.21.15.60:0/2077455890 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v150) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbe4c15c930 con 0x7fbe4c0caac0 2023-10-16T12:54:08.811 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T12:54:08.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe2c06af90 msgr2=0x7fbe2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe2c06af90 0x7fbe2c06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c000f60 tx=0x7fbe3c009040 comp rx=0 tx=0).stop 2023-10-16T12:54:08.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 msgr2=0x7fbe4c15d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:08.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c15d5b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbe3400c460 tx=0x7fbe3400c920 comp rx=0 tx=0).stop 2023-10-16T12:54:08.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 shutdown_connections 2023-10-16T12:54:08.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbe2c06af90 0x7fbe2c06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbe4c0caac0 0x7fbe4c15d5b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe4c15ad70 0x7fbe4c0b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 --2- 172.21.15.60:0/2077455890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c06a4b0 0x7fbe4c1f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 >> 172.21.15.60:0/2077455890 conn(0x7fbe4c0ce9b0 msgr2=0x7fbe4c0c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 shutdown_connections 2023-10-16T12:54:08.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:08.812+0000 7fbe50b01640 1 -- 172.21.15.60:0/2077455890 wait complete. 2023-10-16T12:54:08.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:54:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.049+0000 7fbd53fff640 1 Processor -- start 2023-10-16T12:54:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.049+0000 7fbd53fff640 1 -- start start 2023-10-16T12:54:09.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd53fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd5415b320 0x7fbd540cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd53fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5415b9d0 con 0x7fbd540ad970 2023-10-16T12:54:09.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd53fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5415bcb0 con 0x7fbd5406c040 2023-10-16T12:54:09.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd53fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5415b7d0 con 0x7fbd5415b320 2023-10-16T12:54:09.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd527fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbd540ad970 0x7fbd540cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54224/0 (socket says 172.21.15.60:54224) 2023-10-16T12:54:09.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd527fc640 1 -- 172.21.15.60:0/600052450 learned_addr learned my addr 172.21.15.60:0/600052450 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:09.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd537fe640 1 --2- 172.21.15.60:0/600052450 >> v2:172.21.15.60:3300/0 conn(0x7fbd5415b320 0x7fbd540cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd51ffb640 1 -- 172.21.15.60:0/600052450 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868446437 0 0) 0x7fbd5415b9d0 con 0x7fbd540ad970 2023-10-16T12:54:09.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd51ffb640 1 -- 172.21.15.60:0/600052450 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c003520 con 0x7fbd540ad970 2023-10-16T12:54:09.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd537fe640 1 -- 172.21.15.60:0/600052450 >> v1:172.21.15.144:6789/0 conn(0x7fbd5406c040 legacy=0x7fbd5406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:09.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd537fe640 1 -- 172.21.15.60:0/600052450 >> v1:172.21.15.60:6789/0 conn(0x7fbd540ad970 legacy=0x7fbd540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.051+0000 7fbd537fe640 1 -- 172.21.15.60:0/600052450 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5415bf90 con 0x7fbd5415b320 2023-10-16T12:54:09.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.052+0000 7fbd537fe640 1 --2- 172.21.15.60:0/600052450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5415b320 0x7fbd540cd180 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009dc0 tx=0x7fbd4c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f7d20ee823c3f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:09.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.052+0000 7fbd51ffb640 1 -- 172.21.15.60:0/600052450 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c02cc40 con 0x7fbd5415b320 2023-10-16T12:54:09.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.052+0000 7fbd51ffb640 1 -- 172.21.15.60:0/600052450 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c02cda0 con 0x7fbd5415b320 2023-10-16T12:54:09.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.052+0000 7fbd51ffb640 1 -- 172.21.15.60:0/600052450 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c034660 con 0x7fbd5415b320 2023-10-16T12:54:09.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 -- 172.21.15.60:0/600052450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5415b320 msgr2=0x7fbd540cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 --2- 172.21.15.60:0/600052450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5415b320 0x7fbd540cd180 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009dc0 tx=0x7fbd4c02be90 comp rx=0 tx=0).stop 2023-10-16T12:54:09.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 -- 172.21.15.60:0/600052450 shutdown_connections 2023-10-16T12:54:09.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 --2- 172.21.15.60:0/600052450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5415b320 0x7fbd540cd180 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 -- 172.21.15.60:0/600052450 >> 172.21.15.60:0/600052450 conn(0x7fbd540ce6a0 msgr2=0x7fbd540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:09.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 -- 172.21.15.60:0/600052450 shutdown_connections 2023-10-16T12:54:09.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.053+0000 7fbd53fff640 1 -- 172.21.15.60:0/600052450 wait complete. 2023-10-16T12:54:09.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.056+0000 7fbd53fff640 1 Processor -- start 2023-10-16T12:54:09.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.057+0000 7fbd53fff640 1 -- start start 2023-10-16T12:54:09.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd53fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 0x7fbd540c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd53fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd540ad970 0x7fbd540c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd53fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd5415b320 0x7fbd540c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd53fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540c8660 con 0x7fbd5406c040 2023-10-16T12:54:09.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd53fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd54143110 con 0x7fbd5415b320 2023-10-16T12:54:09.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd53fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd54143f70 con 0x7fbd540ad970 2023-10-16T12:54:09.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 0x7fbd540c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 0x7fbd540c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39532/0 (socket says 172.21.15.60:39532) 2023-10-16T12:54:09.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 -- 172.21.15.60:0/912825763 learned_addr learned my addr 172.21.15.60:0/912825763 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:09.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd537fe640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd5415b320 0x7fbd540c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 -- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd540ad970 msgr2=0x7fbd540c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd527fc640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd540ad970 0x7fbd540c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd540ad970 0x7fbd540c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 -- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd5415b320 msgr2=0x7fbd540c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd5415b320 0x7fbd540c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 -- 172.21.15.60:0/912825763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c009a70 con 0x7fbd5406c040 2023-10-16T12:54:09.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd537fe640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd5415b320 0x7fbd540c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:09.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd52ffd640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 0x7fbd540c8120 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b3f0 tx=0x7fbd4400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:09.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.058+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44011830 con 0x7fbd5406c040 2023-10-16T12:54:09.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.059+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44011990 con 0x7fbd5406c040 2023-10-16T12:54:09.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.059+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44011d40 con 0x7fbd5406c040 2023-10-16T12:54:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.059+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd54143360 con 0x7fbd5406c040 2023-10-16T12:54:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.059+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd54143920 con 0x7fbd5406c040 2023-10-16T12:54:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.060+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5404e9b0 con 0x7fbd5406c040 2023-10-16T12:54:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.061+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbd4401a400 con 0x7fbd5406c040 2023-10-16T12:54:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.063+0000 7fbd3b7fe640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbd4006af90 0x7fbd4006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.063+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbd44089360 con 0x7fbd5406c040 2023-10-16T12:54:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.063+0000 7fbd527fc640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbd4006af90 0x7fbd4006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.064+0000 7fbd527fc640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbd4006af90 0x7fbd4006d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd54159e30 tx=0x7fbd48009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:09.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.068+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbd440545b0 con 0x7fbd5406c040 2023-10-16T12:54:09.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.337+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbd5406b2b0 con 0x7fbd5406c040 2023-10-16T12:54:09.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.338+0000 7fbd3b7fe640 1 -- 172.21.15.60:0/912825763 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbd44057bd0 con 0x7fbd5406c040 2023-10-16T12:54:09.340 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:09.340 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:54:09.340 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T12:54:09.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.341 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T12:54:09.341 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T12:54:09.341 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.341 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.341 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:54:09.342 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:54:09.342 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.343 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T12:54:09.343 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T12:54:09.343 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.343 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.343 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:54:09.344 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:54:09.344 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.344 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T12:54:09.345 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T12:54:09.345 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.345 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.345 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:54:09.345 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:54:09.346 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.346 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.346 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T12:54:09.347 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T12:54:09.347 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:54:09.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:54:09.348 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.348 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.348 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T12:54:09.348 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T12:54:09.349 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.349 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.349 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.349 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.350 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T12:54:09.350 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T12:54:09.350 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.351 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.351 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.351 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.351 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.352 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T12:54:09.352 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T12:54:09.352 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.352 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.352 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.353 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.353 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T12:54:09.353 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T12:54:09.354 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.354 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.354 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.354 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.354 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.355 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T12:54:09.355 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T12:54:09.355 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.356 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.356 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.356 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.357 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T12:54:09.357 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T12:54:09.357 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.357 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.357 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.358 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.358 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T12:54:09.359 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T12:54:09.359 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.359 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.359 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.360 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T12:54:09.360 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T12:54:09.360 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.361 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T12:54:09.361 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T12:54:09.361 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.361 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.362 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T12:54:09.362 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T12:54:09.362 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.362 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.362 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T12:54:09.363 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:54:09.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.363 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T12:54:09.364 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T12:54:09.364 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.364 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.364 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:54:09.364 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:54:09.364 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:54:09.365 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.365 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T12:54:09.365 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T12:54:09.366 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.366 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.366 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:54:09.366 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:54:09.366 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:54:09.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.367 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T12:54:09.367 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T12:54:09.367 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.368 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T12:54:09.368 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T12:54:09.368 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T12:54:09.368 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T12:54:09.368 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.369 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T12:54:09.369 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T12:54:09.369 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.369 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:54:09.370 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T12:54:09.370 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T12:54:09.370 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.371 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T12:54:09.371 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T12:54:09.371 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.371 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:54:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:54:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T12:54:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T12:54:09.373 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T12:54:09.373 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T12:54:09.373 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T12:54:09.373 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T12:54:09.373 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T12:54:09.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:09.374 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T12:54:09.374 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:54:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbd4006af90 msgr2=0x7fbd4006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbd4006af90 0x7fbd4006d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd54159e30 tx=0x7fbd48009040 comp rx=0 tx=0).stop 2023-10-16T12:54:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 msgr2=0x7fbd540c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 0x7fbd540c8120 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b3f0 tx=0x7fbd4400b8b0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 shutdown_connections 2023-10-16T12:54:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fbd4006af90 0x7fbd4006d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd540ad970 0x7fbd540c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd5415b320 0x7fbd540c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 --2- 172.21.15.60:0/912825763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd5406c040 0x7fbd540c8120 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 >> 172.21.15.60:0/912825763 conn(0x7fbd540ce6a0 msgr2=0x7fbd5415cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 shutdown_connections 2023-10-16T12:54:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.341+0000 7fbd53fff640 1 -- 172.21.15.60:0/912825763 wait complete. 2023-10-16T12:54:09.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:54:09.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.609+0000 7f27be867640 1 Processor -- start 2023-10-16T12:54:09.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.610+0000 7f27be867640 1 -- start start 2023-10-16T12:54:09.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.611+0000 7f27be867640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f27b80ad970 0x7f27b80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.611+0000 7f27be867640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b815b9d0 con 0x7f27b806c040 2023-10-16T12:54:09.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.611+0000 7f27be867640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b815bcb0 con 0x7f27b80ccfb0 2023-10-16T12:54:09.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.611+0000 7f27be867640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b815bf90 con 0x7f27b80ad970 2023-10-16T12:54:09.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bd865640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f27b806c040 0x7f27b806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54232/0 (socket says 172.21.15.60:54232) 2023-10-16T12:54:09.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bd865640 1 -- 172.21.15.60:0/213664080 learned_addr learned my addr 172.21.15.60:0/213664080 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:09.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386935531 0 0) 0x7f27b815b9d0 con 0x7f27b806c040 2023-10-16T12:54:09.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bd064640 1 --2- 172.21.15.60:0/213664080 >> v2:172.21.15.144:3300/0 conn(0x7f27b80ad970 0x7f27b80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a4003520 con 0x7f27b806c040 2023-10-16T12:54:09.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941211721 0 0) 0x7f27a4003520 con 0x7f27b806c040 2023-10-16T12:54:09.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 >> v2:172.21.15.144:3300/0 conn(0x7f27b80ad970 msgr2=0x7f27b80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:09.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.612+0000 7f27bc863640 1 --2- 172.21.15.60:0/213664080 >> v2:172.21.15.144:3300/0 conn(0x7f27b80ad970 0x7f27b80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 >> v1:172.21.15.144:6789/0 conn(0x7f27b80ccfb0 legacy=0x7f27b80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b80cc270 con 0x7f27b806c040 2023-10-16T12:54:09.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f27b4001670 con 0x7f27b806c040 2023-10-16T12:54:09.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 >> v1:172.21.15.60:6789/0 conn(0x7f27b806c040 legacy=0x7f27b806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27a40039c0 0x7f27a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40047c0 0x7f27a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27a40051d0 0x7f27a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b815bf90 con 0x7f27a40039c0 2023-10-16T12:54:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a400c120 con 0x7f27a40047c0 2023-10-16T12:54:09.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a400c2e0 con 0x7f27a40051d0 2023-10-16T12:54:09.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bd865640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27a40039c0 0x7f27a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bd064640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40047c0 0x7f27a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27be066640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27a40051d0 0x7f27a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.613+0000 7f27bd865640 1 -- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27a40051d0 msgr2=0x7f27a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:09.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.614+0000 7f27bd865640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27a40051d0 0x7f27a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.614+0000 7f27bd865640 1 -- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40047c0 msgr2=0x7f27a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.614+0000 7f27bd865640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40047c0 0x7f27a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.614+0000 7f27bd865640 1 -- 172.21.15.60:0/213664080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4003520 con 0x7f27a40039c0 2023-10-16T12:54:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.614+0000 7f27be066640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27a40051d0 0x7f27a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:09.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.614+0000 7f27be066640 1 -- 172.21.15.60:0/213664080 reap_dead start 2023-10-16T12:54:09.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.615+0000 7f27bd865640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27a40039c0 0x7f27a4003de0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f27b400c0e0 tx=0x7f27a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54df33cfc009c7c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:09.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.615+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4005ad0 con 0x7f27a40039c0 2023-10-16T12:54:09.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.615+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b4005c30 con 0x7f27a40039c0 2023-10-16T12:54:09.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.615+0000 7f27bc863640 1 -- 172.21.15.60:0/213664080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b402ca30 con 0x7f27a40039c0 2023-10-16T12:54:09.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.615+0000 7f27be867640 1 -- 172.21.15.60:0/213664080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27a40039c0 msgr2=0x7f27a4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.615+0000 7f27be867640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27a40039c0 0x7f27a4003de0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f27b400c0e0 tx=0x7f27a40057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.616+0000 7f27be867640 1 -- 172.21.15.60:0/213664080 shutdown_connections 2023-10-16T12:54:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.616+0000 7f27be867640 1 --2- 172.21.15.60:0/213664080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27a40039c0 0x7f27a4003de0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.616+0000 7f27be867640 1 -- 172.21.15.60:0/213664080 >> 172.21.15.60:0/213664080 conn(0x7f27b80ce6a0 msgr2=0x7f27b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.616+0000 7f27be867640 1 -- 172.21.15.60:0/213664080 shutdown_connections 2023-10-16T12:54:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.616+0000 7f27be867640 1 -- 172.21.15.60:0/213664080 wait complete. 2023-10-16T12:54:09.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.618+0000 7f27be867640 1 Processor -- start 2023-10-16T12:54:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.618+0000 7f27be867640 1 -- start start 2023-10-16T12:54:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be867640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40039c0 0x7f27b81f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be867640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27b815e920 0x7f27b815eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be867640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 0x7f27b80acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be867640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b815f3e0 con 0x7f27b80ac890 2023-10-16T12:54:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be867640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b80ad250 con 0x7f27a40039c0 2023-10-16T12:54:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be867640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b80c9c50 con 0x7f27b815e920 2023-10-16T12:54:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 0x7f27b80acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be066640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 0x7f27b80acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39546/0 (socket says 172.21.15.60:39546) 2023-10-16T12:54:09.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.619+0000 7f27be066640 1 -- 172.21.15.60:0/2549140758 learned_addr learned my addr 172.21.15.60:0/2549140758 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27bd865640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40039c0 0x7f27b81f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27bd064640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27b815e920 0x7f27b815eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be066640 1 -- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27b815e920 msgr2=0x7f27b815eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be066640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27b815e920 0x7f27b815eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be066640 1 -- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40039c0 msgr2=0x7f27b81f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be066640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40039c0 0x7f27b81f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be066640 1 -- 172.21.15.60:0/2549140758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b80c97c0 con 0x7f27b80ac890 2023-10-16T12:54:09.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be066640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 0x7f27b80acd10 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f27b80b1260 tx=0x7f27b000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:09.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0003b70 con 0x7f27b80ac890 2023-10-16T12:54:09.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0003cf0 con 0x7f27b80ac890 2023-10-16T12:54:09.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b80c8e90 con 0x7f27b80ac890 2023-10-16T12:54:09.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.620+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b00052b0 con 0x7f27b80ac890 2023-10-16T12:54:09.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.621+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b80c9450 con 0x7f27b80ac890 2023-10-16T12:54:09.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.622+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b806b330 con 0x7f27b80ac890 2023-10-16T12:54:09.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.623+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f27b0005410 con 0x7f27b80ac890 2023-10-16T12:54:09.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.624+0000 7f27ae7fc640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f279c06afc0 0x7f279c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:09.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.624+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f27b0088dc0 con 0x7f27b80ac890 2023-10-16T12:54:09.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.624+0000 7f27bd865640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f279c06afc0 0x7f279c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:09.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.625+0000 7f27bd865640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f279c06afc0 0x7f279c06d480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27b4034000 tx=0x7f27b4009030 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:09.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.628+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27b0053f00 con 0x7f27b80ac890 2023-10-16T12:54:09.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.828+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27b80c9950 con 0x7f27b80ac890 2023-10-16T12:54:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.830+0000 7f27ae7fc640 1 -- 172.21.15.60:0/2549140758 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+12855 (secure 0 0 0) 0x7f27b0018070 con 0x7f27b80ac890 2023-10-16T12:54:09.832 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:09.832 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:54:08.816679+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:54:09.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.832+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f279c06afc0 msgr2=0x7f279c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.832+0000 7f27be867640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f279c06afc0 0x7f279c06d480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27b4034000 tx=0x7f27b4009030 comp rx=0 tx=0).stop 2023-10-16T12:54:09.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 msgr2=0x7f27b80acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:09.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 0x7f27b80acd10 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f27b80b1260 tx=0x7f27b000dc00 comp rx=0 tx=0).stop 2023-10-16T12:54:09.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 shutdown_connections 2023-10-16T12:54:09.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f279c06afc0 0x7f279c06d480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27b815e920 0x7f27b815eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b80ac890 0x7f27b80acd10 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 --2- 172.21.15.60:0/2549140758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27a40039c0 0x7f27b81f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 >> 172.21.15.60:0/2549140758 conn(0x7f27b80ce6a0 msgr2=0x7f27b80ae230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 shutdown_connections 2023-10-16T12:54:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:09.833+0000 7f27be867640 1 -- 172.21.15.60:0/2549140758 wait complete. 2023-10-16T12:54:09.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:54:09.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:54:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.079+0000 7f1db8932640 1 Processor -- start 2023-10-16T12:54:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.079+0000 7f1db8932640 1 -- start start 2023-10-16T12:54:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.080+0000 7f1db8932640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1db4159ad0 0x7f1db4070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.080+0000 7f1db8932640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db40cd260 con 0x7f1db40c9de0 2023-10-16T12:54:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.080+0000 7f1db8932640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db415de30 con 0x7f1db415d650 2023-10-16T12:54:10.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.080+0000 7f1db8932640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db4071470 con 0x7f1db4159ad0 2023-10-16T12:54:10.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1db4159ad0 0x7f1db4070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1db4159ad0 0x7f1db4070f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39558/0 (socket says 172.21.15.60:39558) 2023-10-16T12:54:10.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1012068210 learned_addr learned my addr 172.21.15.60:0/1012068210 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:10.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db359e640 1 --1- 172.21.15.60:0/1012068210 >> v1:172.21.15.60:6789/0 conn(0x7f1db40c9de0 0x7f1db40cd9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54240/0 (socket says 172.21.15.60:54240) 2023-10-16T12:54:10.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1012068210 >> v1:172.21.15.144:6789/0 conn(0x7f1db415d650 legacy=0x7f1db406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1012068210 >> v1:172.21.15.60:6789/0 conn(0x7f1db40c9de0 legacy=0x7f1db40cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db259c640 1 -- 172.21.15.60:0/1012068210 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259617552 0 0) 0x7f1db40cd260 con 0x7f1db40c9de0 2023-10-16T12:54:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.081+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1012068210 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db415e110 con 0x7f1db4159ad0 2023-10-16T12:54:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.082+0000 7f1db2d9d640 1 --2- 172.21.15.60:0/1012068210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db4159ad0 0x7f1db4070f30 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009870 tx=0x7f1da402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef8b19236c2b3dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db259c640 1 -- 172.21.15.60:0/1012068210 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4030860 con 0x7f1db4159ad0 2023-10-16T12:54:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db259c640 1 -- 172.21.15.60:0/1012068210 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da40309c0 con 0x7f1db4159ad0 2023-10-16T12:54:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db259c640 1 -- 172.21.15.60:0/1012068210 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4030d50 con 0x7f1db4159ad0 2023-10-16T12:54:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db8932640 1 -- 172.21.15.60:0/1012068210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db4159ad0 msgr2=0x7f1db4070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db8932640 1 --2- 172.21.15.60:0/1012068210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db4159ad0 0x7f1db4070f30 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009870 tx=0x7f1da402b9a0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db8932640 1 -- 172.21.15.60:0/1012068210 shutdown_connections 2023-10-16T12:54:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.083+0000 7f1db8932640 1 --2- 172.21.15.60:0/1012068210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db4159ad0 0x7f1db4070f30 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.084+0000 7f1db8932640 1 -- 172.21.15.60:0/1012068210 >> 172.21.15.60:0/1012068210 conn(0x7f1db40b0170 msgr2=0x7f1db40b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.084+0000 7f1db8932640 1 -- 172.21.15.60:0/1012068210 shutdown_connections 2023-10-16T12:54:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.084+0000 7f1db8932640 1 -- 172.21.15.60:0/1012068210 wait complete. 2023-10-16T12:54:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.085+0000 7f1db8932640 1 Processor -- start 2023-10-16T12:54:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.086+0000 7f1db8932640 1 -- start start 2023-10-16T12:54:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db8932640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 0x7f1db40acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db8932640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1db4159ad0 0x7f1db414f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db8932640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1db415d650 0x7f1db414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db8932640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db4150220 con 0x7f1db40c9de0 2023-10-16T12:54:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db8932640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41eff20 con 0x7f1db4159ad0 2023-10-16T12:54:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db8932640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41f24d0 con 0x7f1db415d650 2023-10-16T12:54:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 0x7f1db40acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 0x7f1db40acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39566/0 (socket says 172.21.15.60:39566) 2023-10-16T12:54:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1264313842 learned_addr learned my addr 172.21.15.60:0/1264313842 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1db415d650 msgr2=0x7f1db414fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db359e640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1db415d650 0x7f1db414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1db2d9d640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1db415d650 0x7f1db414fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.087+0000 7f1daa59c640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1db4159ad0 0x7f1db414f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1db4159ad0 msgr2=0x7f1db414f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1db2d9d640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1db4159ad0 0x7f1db414f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1db2d9d640 1 -- 172.21.15.60:0/1264313842 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40094d0 con 0x7f1db40c9de0 2023-10-16T12:54:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1daa59c640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1db4159ad0 0x7f1db414f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1db2d9d640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 0x7f1db40acb00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f1da4000c00 tx=0x7f1da4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4030810 con 0x7f1db40c9de0 2023-10-16T12:54:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4043400 con 0x7f1db40c9de0 2023-10-16T12:54:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da40437b0 con 0x7f1db40c9de0 2023-10-16T12:54:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db41f6860 con 0x7f1db40c9de0 2023-10-16T12:54:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.088+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db41f2d10 con 0x7f1db40c9de0 2023-10-16T12:54:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.090+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db404e9b0 con 0x7f1db40c9de0 2023-10-16T12:54:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.090+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1da4043910 con 0x7f1db40c9de0 2023-10-16T12:54:10.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.091+0000 7f1dabfff640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1d9406af90 0x7f1d9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.092+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1da403e070 con 0x7f1db40c9de0 2023-10-16T12:54:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.092+0000 7f1daa59c640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1d9406af90 0x7f1d9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.093+0000 7f1daa59c640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1d9406af90 0x7f1d9406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1db40ae1e0 tx=0x7f1d9c0073c0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:10.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.096+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1da4077bc0 con 0x7f1db40c9de0 2023-10-16T12:54:10.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.297+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1db40b4600 con 0x7f1db40c9de0 2023-10-16T12:54:10.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.298+0000 7f1dabfff640 1 -- 172.21.15.60:0/1264313842 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1da407b1e0 con 0x7f1db40c9de0 2023-10-16T12:54:10.300 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T12:54:10.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1d9406af90 msgr2=0x7f1d9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1d9406af90 0x7f1d9406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1db40ae1e0 tx=0x7f1d9c0073c0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 msgr2=0x7f1db40acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 0x7f1db40acb00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f1da4000c00 tx=0x7f1da4004730 comp rx=0 tx=0).stop 2023-10-16T12:54:10.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 shutdown_connections 2023-10-16T12:54:10.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f1d9406af90 0x7f1d9406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1db415d650 0x7f1db414fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1db4159ad0 0x7f1db414f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 --2- 172.21.15.60:0/1264313842 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1db40c9de0 0x7f1db40acb00 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 >> 172.21.15.60:0/1264313842 conn(0x7f1db40b0170 msgr2=0x7f1db40ae450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:10.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.301+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 shutdown_connections 2023-10-16T12:54:10.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.302+0000 7f1db8932640 1 -- 172.21.15.60:0/1264313842 wait complete. 2023-10-16T12:54:10.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:54:10.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.554+0000 7f8067983640 1 Processor -- start 2023-10-16T12:54:10.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.555+0000 7f8067983640 1 -- start start 2023-10-16T12:54:10.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.556+0000 7f8067983640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80600ce690 0x7f806006adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.556+0000 7f8067983640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806006ffe0 con 0x7f806015c850 2023-10-16T12:54:10.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.556+0000 7f8067983640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80600cb9e0 con 0x7f80600c2870 2023-10-16T12:54:10.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.556+0000 7f8067983640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80600ccf40 con 0x7f80600ce690 2023-10-16T12:54:10.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f8064f1e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80600ce690 0x7f806006adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f8064f1e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80600ce690 0x7f806006adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39584/0 (socket says 172.21.15.60:39584) 2023-10-16T12:54:10.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f806571f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f806015c850 0x7f80600cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54248/0 (socket says 172.21.15.60:54248) 2023-10-16T12:54:10.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f8064f1e640 1 -- 172.21.15.60:0/2737840944 learned_addr learned my addr 172.21.15.60:0/2737840944 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:10.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f8064f1e640 1 -- 172.21.15.60:0/2737840944 >> v1:172.21.15.144:6789/0 conn(0x7f80600c2870 legacy=0x7f80600c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:10.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f8064f1e640 1 -- 172.21.15.60:0/2737840944 >> v1:172.21.15.60:6789/0 conn(0x7f806015c850 legacy=0x7f80600cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.557+0000 7f8064f1e640 1 -- 172.21.15.60:0/2737840944 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600cde10 con 0x7f80600ce690 2023-10-16T12:54:10.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.559+0000 7f8064f1e640 1 --2- 172.21.15.60:0/2737840944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80600ce690 0x7f806006adb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f8050009870 tx=0x7f805002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c082baee8ddd21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:10.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.559+0000 7f8057fff640 1 -- 172.21.15.60:0/2737840944 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80500317e0 con 0x7f80600ce690 2023-10-16T12:54:10.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.559+0000 7f8057fff640 1 -- 172.21.15.60:0/2737840944 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050031940 con 0x7f80600ce690 2023-10-16T12:54:10.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.559+0000 7f8057fff640 1 -- 172.21.15.60:0/2737840944 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050031cd0 con 0x7f80600ce690 2023-10-16T12:54:10.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 -- 172.21.15.60:0/2737840944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80600ce690 msgr2=0x7f806006adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 --2- 172.21.15.60:0/2737840944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80600ce690 0x7f806006adb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f8050009870 tx=0x7f805002b880 comp rx=0 tx=0).stop 2023-10-16T12:54:10.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 -- 172.21.15.60:0/2737840944 shutdown_connections 2023-10-16T12:54:10.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 --2- 172.21.15.60:0/2737840944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80600ce690 0x7f806006adb0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 -- 172.21.15.60:0/2737840944 >> 172.21.15.60:0/2737840944 conn(0x7f80600b0d30 msgr2=0x7f80600b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:10.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 -- 172.21.15.60:0/2737840944 shutdown_connections 2023-10-16T12:54:10.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.560+0000 7f8067983640 1 -- 172.21.15.60:0/2737840944 wait complete. 2023-10-16T12:54:10.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.562+0000 7f8067983640 1 Processor -- start 2023-10-16T12:54:10.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.562+0000 7f8067983640 1 -- start start 2023-10-16T12:54:10.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.563+0000 7f8067983640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80600c2870 0x7f80601532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.563+0000 7f8067983640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80600ce690 0x7f80600b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.563+0000 7f8067983640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 0x7f80600b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.563+0000 7f8067983640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80600b43a0 con 0x7f806015c850 2023-10-16T12:54:10.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.563+0000 7f8067983640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80600b03b0 con 0x7f80600c2870 2023-10-16T12:54:10.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.563+0000 7f8067983640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806015c4d0 con 0x7f80600ce690 2023-10-16T12:54:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 0x7f80600b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 0x7f80600b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39588/0 (socket says 172.21.15.60:39588) 2023-10-16T12:54:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 -- 172.21.15.60:0/2214913386 learned_addr learned my addr 172.21.15.60:0/2214913386 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f806571f640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80600c2870 0x7f80601532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8064f1e640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80600ce690 0x7f80600b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 -- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80600ce690 msgr2=0x7f80600b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80600ce690 0x7f80600b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 -- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80600c2870 msgr2=0x7f80601532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80600c2870 0x7f80601532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8065f20640 1 -- 172.21.15.60:0/2214913386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80500094d0 con 0x7f806015c850 2023-10-16T12:54:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f8064f1e640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80600ce690 0x7f80600b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.564+0000 7f806571f640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80600c2870 0x7f80601532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.565+0000 7f8065f20640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 0x7f80600b1d90 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f805800b970 tx=0x7f805800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.565+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805800cb40 con 0x7f806015c850 2023-10-16T12:54:10.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.565+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805800cca0 con 0x7f806015c850 2023-10-16T12:54:10.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.565+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058014680 con 0x7f806015c850 2023-10-16T12:54:10.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.565+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80600b3d50 con 0x7f806015c850 2023-10-16T12:54:10.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.565+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80600b3a00 con 0x7f806015c850 2023-10-16T12:54:10.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.566+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80600b4510 con 0x7f806015c850 2023-10-16T12:54:10.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.567+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f805801d3b0 con 0x7f806015c850 2023-10-16T12:54:10.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.568+0000 7f8055ffb640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f804406af90 0x7f804406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:10.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.568+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f80580893b0 con 0x7f806015c850 2023-10-16T12:54:10.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.569+0000 7f806571f640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f804406af90 0x7f804406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:10.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.570+0000 7f806571f640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f804406af90 0x7f804406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f805c000f60 tx=0x7f805c009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:10.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.573+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8058054570 con 0x7f806015c850 2023-10-16T12:54:10.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.771+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f806004e160 con 0x7f806015c850 2023-10-16T12:54:10.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.772+0000 7f8055ffb640 1 -- 172.21.15.60:0/2214913386 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8058057b90 con 0x7f806015c850 2023-10-16T12:54:10.774 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:10.774 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:54:10.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:10.775 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T12:54:10.775 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T12:54:10.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:10.775 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:54:10.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f804406af90 msgr2=0x7f804406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f804406af90 0x7f804406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f805c000f60 tx=0x7f805c009040 comp rx=0 tx=0).stop 2023-10-16T12:54:10.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 msgr2=0x7f80600b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:10.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 0x7f80600b1d90 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f805800b970 tx=0x7f805800be30 comp rx=0 tx=0).stop 2023-10-16T12:54:10.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 shutdown_connections 2023-10-16T12:54:10.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f804406af90 0x7f804406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f80600ce690 0x7f80600b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f806015c850 0x7f80600b1d90 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 --2- 172.21.15.60:0/2214913386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80600c2870 0x7f80601532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:10.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 >> 172.21.15.60:0/2214913386 conn(0x7f80600b0d30 msgr2=0x7f80601579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:10.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 shutdown_connections 2023-10-16T12:54:10.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:10.775+0000 7f8067983640 1 -- 172.21.15.60:0/2214913386 wait complete. 2023-10-16T12:54:10.797 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:54:10.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:54:11.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.031+0000 7fe8a359e640 1 Processor -- start 2023-10-16T12:54:11.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.031+0000 7fe8a359e640 1 -- start start 2023-10-16T12:54:11.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe8a359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe8a415ad80 0x7fe8a406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:11.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe8a359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe8a40cae40 0x7fe8a406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:11.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe8a359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a40c8e70 con 0x7fe8a40ad7f0 2023-10-16T12:54:11.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe8a359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a40b4170 con 0x7fe8a40cae40 2023-10-16T12:54:11.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe8a359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a40adde0 con 0x7fe8a415ad80 2023-10-16T12:54:11.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe899d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe8a40cae40 0x7fe8a406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:11.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe899d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe8a40cae40 0x7fe8a406d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39600/0 (socket says 172.21.15.60:39600) 2023-10-16T12:54:11.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.033+0000 7fe899d9b640 1 -- 172.21.15.60:0/2306876289 learned_addr learned my addr 172.21.15.60:0/2306876289 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:11.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.034+0000 7fe8a259c640 1 --2- 172.21.15.60:0/2306876289 >> v2:172.21.15.144:3300/0 conn(0x7fe8a415ad80 0x7fe8a406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:11.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.034+0000 7fe899d9b640 1 -- 172.21.15.60:0/2306876289 >> v2:172.21.15.144:3300/0 conn(0x7fe8a415ad80 msgr2=0x7fe8a406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:11.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.034+0000 7fe899d9b640 1 --2- 172.21.15.60:0/2306876289 >> v2:172.21.15.144:3300/0 conn(0x7fe8a415ad80 0x7fe8a406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.034+0000 7fe899d9b640 1 -- 172.21.15.60:0/2306876289 >> v1:172.21.15.60:6789/0 conn(0x7fe8a40ad7f0 legacy=0x7fe8a406b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.034+0000 7fe899d9b640 1 -- 172.21.15.60:0/2306876289 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a40b4410 con 0x7fe8a40cae40 2023-10-16T12:54:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.034+0000 7fe8a259c640 1 --2- 172.21.15.60:0/2306876289 >> v2:172.21.15.144:3300/0 conn(0x7fe8a415ad80 0x7fe8a406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.035+0000 7fe899d9b640 1 --2- 172.21.15.60:0/2306876289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a40cae40 0x7fe8a406d0b0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4070710 tx=0x7fe88c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572a6805e57e83be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.035+0000 7fe8a1d9b640 1 -- 172.21.15.60:0/2306876289 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c030860 con 0x7fe8a40cae40 2023-10-16T12:54:11.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a1d9b640 1 -- 172.21.15.60:0/2306876289 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c0309c0 con 0x7fe8a40cae40 2023-10-16T12:54:11.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a1d9b640 1 -- 172.21.15.60:0/2306876289 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c030d50 con 0x7fe8a40cae40 2023-10-16T12:54:11.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 -- 172.21.15.60:0/2306876289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a40cae40 msgr2=0x7fe8a406d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:11.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 --2- 172.21.15.60:0/2306876289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a40cae40 0x7fe8a406d0b0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4070710 tx=0x7fe88c02b9a0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 -- 172.21.15.60:0/2306876289 shutdown_connections 2023-10-16T12:54:11.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 --2- 172.21.15.60:0/2306876289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a40cae40 0x7fe8a406d0b0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 --2- 172.21.15.60:0/2306876289 >> v2:172.21.15.144:3300/0 conn(0x7fe8a415ad80 0x7fe8a406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 -- 172.21.15.60:0/2306876289 >> 172.21.15.60:0/2306876289 conn(0x7fe8a40ce820 msgr2=0x7fe8a40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:11.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 -- 172.21.15.60:0/2306876289 shutdown_connections 2023-10-16T12:54:11.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.036+0000 7fe8a359e640 1 -- 172.21.15.60:0/2306876289 wait complete. 2023-10-16T12:54:11.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.038+0000 7fe8a359e640 1 Processor -- start 2023-10-16T12:54:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.038+0000 7fe8a359e640 1 -- start start 2023-10-16T12:54:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8a40ad7f0 0x7fe8a40ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8a40cae40 0x7fe8a414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:11.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 0x7fe8a4149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a414a8b0 con 0x7fe8a415ad80 2023-10-16T12:54:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4149760 con 0x7fe8a40cae40 2023-10-16T12:54:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40aeda0 con 0x7fe8a40ad7f0 2023-10-16T12:54:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.039+0000 7fe8a2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 0x7fe8a4149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 0x7fe8a4149220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39602/0 (socket says 172.21.15.60:39602) 2023-10-16T12:54:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 -- 172.21.15.60:0/124284258 learned_addr learned my addr 172.21.15.60:0/124284258 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a259c640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8a40ad7f0 0x7fe8a40ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe899d9b640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8a40cae40 0x7fe8a414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 -- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8a40ad7f0 msgr2=0x7fe8a40ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8a40ad7f0 0x7fe8a40ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 -- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8a40cae40 msgr2=0x7fe8a414a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8a40cae40 0x7fe8a414a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 -- 172.21.15.60:0/124284258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c0094d0 con 0x7fe8a415ad80 2023-10-16T12:54:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe899d9b640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8a40cae40 0x7fe8a414a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.040+0000 7fe8a2d9d640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 0x7fe8a4149220 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00c730 tx=0x7fe89c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.041+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c003b70 con 0x7fe8a415ad80 2023-10-16T12:54:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.041+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c003cd0 con 0x7fe8a415ad80 2023-10-16T12:54:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.041+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c00aa90 con 0x7fe8a415ad80 2023-10-16T12:54:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.041+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a40af080 con 0x7fe8a415ad80 2023-10-16T12:54:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.041+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a40af640 con 0x7fe8a415ad80 2023-10-16T12:54:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.042+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a404e9b0 con 0x7fe8a415ad80 2023-10-16T12:54:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.043+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe89c00abf0 con 0x7fe8a415ad80 2023-10-16T12:54:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.044+0000 7fe89b7fe640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe88406af90 0x7fe88406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.044+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe89c0892b0 con 0x7fe8a415ad80 2023-10-16T12:54:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.044+0000 7fe8a259c640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe88406af90 0x7fe88406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.046+0000 7fe8a259c640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe88406af90 0x7fe88406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe89400a950 tx=0x7fe894008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.048+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe89c054590 con 0x7fe8a415ad80 2023-10-16T12:54:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.245+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe8a40b2a30 con 0x7fe8a415ad80 2023-10-16T12:54:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.818+0000 7fe89b7fe640 1 -- 172.21.15.60:0/124284258 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v153) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe89c057bb0 con 0x7fe8a415ad80 2023-10-16T12:54:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T12:54:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe88406af90 msgr2=0x7fe88406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe88406af90 0x7fe88406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe89400a950 tx=0x7fe894008040 comp rx=0 tx=0).stop 2023-10-16T12:54:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 msgr2=0x7fe8a4149220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:11.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 0x7fe8a4149220 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00c730 tx=0x7fe89c00cbf0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 shutdown_connections 2023-10-16T12:54:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe88406af90 0x7fe88406d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a415ad80 0x7fe8a4149220 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe8a40cae40 0x7fe8a414a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 --2- 172.21.15.60:0/124284258 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe8a40ad7f0 0x7fe8a40ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 >> 172.21.15.60:0/124284258 conn(0x7fe8a40ce820 msgr2=0x7fe8a40b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.822+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 shutdown_connections 2023-10-16T12:54:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:11.823+0000 7fe8a359e640 1 -- 172.21.15.60:0/124284258 wait complete. 2023-10-16T12:54:11.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:54:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.081+0000 7fe61b59e640 1 Processor -- start 2023-10-16T12:54:12.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.081+0000 7fe61b59e640 1 -- start start 2023-10-16T12:54:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.083+0000 7fe61b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe61c15cb50 0x7fe61c161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.083+0000 7fe61b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 0x7fe61c0d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.083+0000 7fe61b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61c1616c0 con 0x7fe61c0b1690 2023-10-16T12:54:12.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.083+0000 7fe61b59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61c0cdbf0 con 0x7fe61c15cb50 2023-10-16T12:54:12.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.083+0000 7fe61b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c0cde90 con 0x7fe61c0b4c80 2023-10-16T12:54:12.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.084+0000 7fe611d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe61c0b1690 0x7fe61c0b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54262/0 (socket says 172.21.15.60:54262) 2023-10-16T12:54:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.084+0000 7fe611d9b640 1 -- 172.21.15.60:0/1644348005 learned_addr learned my addr 172.21.15.60:0/1644348005 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.084+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1644348005 >> v2:172.21.15.144:3300/0 conn(0x7fe61c15cb50 0x7fe61c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.084+0000 7fe61ad9d640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 0x7fe61c0d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.084+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859142303 0 0) 0x7fe61c1616c0 con 0x7fe61c0b1690 2023-10-16T12:54:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.084+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe608003520 con 0x7fe61c0b1690 2023-10-16T12:54:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1293595909 0 0) 0x7fe608003520 con 0x7fe61c0b1690 2023-10-16T12:54:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 msgr2=0x7fe61c0d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 0x7fe61c0d1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 >> v2:172.21.15.144:3300/0 conn(0x7fe61c15cb50 msgr2=0x7fe61c161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 --2- 172.21.15.60:0/1644348005 >> v2:172.21.15.144:3300/0 conn(0x7fe61c15cb50 0x7fe61c161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c0ce170 con 0x7fe61c0b1690 2023-10-16T12:54:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe604001670 con 0x7fe61c0b1690 2023-10-16T12:54:12.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 >> v1:172.21.15.60:6789/0 conn(0x7fe61c0b1690 legacy=0x7fe61c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe608003970 0x7fe608003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 0x7fe6080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608004d00 0x7fe608005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c0cdbf0 con 0x7fe608003970 2023-10-16T12:54:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c0cde90 con 0x7fe608004d00 2023-10-16T12:54:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60800ba00 con 0x7fe61c0b4c80 2023-10-16T12:54:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.085+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe608003970 0x7fe608003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.086+0000 7fe611d9b640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 0x7fe6080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.086+0000 7fe61a59c640 1 -- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 msgr2=0x7fe6080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:12.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.086+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c0b4c80 0x7fe6080047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.086+0000 7fe61a59c640 1 -- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608004d00 msgr2=0x7fe608005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.086+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608004d00 0x7fe608005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.086+0000 7fe61a59c640 1 -- 172.21.15.60:0/1644348005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608003520 con 0x7fe608003970 2023-10-16T12:54:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.087+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe608003970 0x7fe608003d90 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009fc0 tx=0x7fe6080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782a24f2cd18cabf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.087+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c02cc60 con 0x7fe608003970 2023-10-16T12:54:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.087+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c00f040 con 0x7fe608003970 2023-10-16T12:54:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.087+0000 7fe619d9b640 1 -- 172.21.15.60:0/1644348005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c034440 con 0x7fe608003970 2023-10-16T12:54:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61b59e640 1 -- 172.21.15.60:0/1644348005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe608003970 msgr2=0x7fe608003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1644348005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe608003970 0x7fe608003d90 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009fc0 tx=0x7fe6080057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61ad9d640 1 -- 172.21.15.60:0/1644348005 reap_dead start 2023-10-16T12:54:12.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61b59e640 1 -- 172.21.15.60:0/1644348005 shutdown_connections 2023-10-16T12:54:12.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61b59e640 1 -- 172.21.15.60:0/1644348005 >> 172.21.15.60:0/1644348005 conn(0x7fe61c0cf570 msgr2=0x7fe61c0cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:12.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61b59e640 1 -- 172.21.15.60:0/1644348005 shutdown_connections 2023-10-16T12:54:12.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.088+0000 7fe61b59e640 1 -- 172.21.15.60:0/1644348005 wait complete. 2023-10-16T12:54:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.090+0000 7fe61b59e640 1 Processor -- start 2023-10-16T12:54:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.090+0000 7fe61b59e640 1 -- start start 2023-10-16T12:54:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608003970 0x7fe61c15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c155b90 0x7fe61c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 0x7fe61c0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c0cbcf0 con 0x7fe61c0cb330 2023-10-16T12:54:12.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c15d5a0 con 0x7fe608003970 2023-10-16T12:54:12.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c156c10 con 0x7fe61c155b90 2023-10-16T12:54:12.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 0x7fe61c0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 0x7fe61c0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39614/0 (socket says 172.21.15.60:39614) 2023-10-16T12:54:12.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.091+0000 7fe61ad9d640 1 -- 172.21.15.60:0/1685141756 learned_addr learned my addr 172.21.15.60:0/1685141756 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:12.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe611d9b640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c155b90 0x7fe61c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe61ad9d640 1 -- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c155b90 msgr2=0x7fe61c1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:12.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe61ad9d640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c155b90 0x7fe61c1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe61ad9d640 1 -- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608003970 msgr2=0x7fe61c15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:12.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe61ad9d640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608003970 0x7fe61c15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe61ad9d640 1 -- 172.21.15.60:0/1685141756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c156370 con 0x7fe61c0cb330 2023-10-16T12:54:12.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe61ad9d640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 0x7fe61c0cb7b0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0b59d0 tx=0x7fe61400eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:12.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.092+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140059e0 con 0x7fe61c0cb330 2023-10-16T12:54:12.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.093+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe614005b60 con 0x7fe61c0cb330 2023-10-16T12:54:12.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.093+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61400c680 con 0x7fe61c0cb330 2023-10-16T12:54:12.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.093+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61c156650 con 0x7fe61c0cb330 2023-10-16T12:54:12.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.093+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c15d7d0 con 0x7fe61c0cb330 2023-10-16T12:54:12.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.094+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61c04e9b0 con 0x7fe61c0cb330 2023-10-16T12:54:12.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.095+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe614004020 con 0x7fe61c0cb330 2023-10-16T12:54:12.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.096+0000 7fe6137fe640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5fc06afc0 0x7fe5fc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:12.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.096+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe614088c00 con 0x7fe61c0cb330 2023-10-16T12:54:12.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.097+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5fc06afc0 0x7fe5fc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:12.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.098+0000 7fe61a59c640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5fc06afc0 0x7fe5fc06d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe60c036000 tx=0x7fe60c02e070 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:12.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.100+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe614053d60 con 0x7fe61c0cb330 2023-10-16T12:54:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.302+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe61c0d0790 con 0x7fe61c0cb330 2023-10-16T12:54:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.825+0000 7fe6137fe640 1 -- 172.21.15.60:0/1685141756 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v154) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe614057380 con 0x7fe61c0cb330 2023-10-16T12:54:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T12:54:12.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5fc06afc0 msgr2=0x7fe5fc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:12.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5fc06afc0 0x7fe5fc06d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe60c036000 tx=0x7fe60c02e070 comp rx=0 tx=0).stop 2023-10-16T12:54:12.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 msgr2=0x7fe61c0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:12.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 0x7fe61c0cb7b0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0b59d0 tx=0x7fe61400eb40 comp rx=0 tx=0).stop 2023-10-16T12:54:12.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 shutdown_connections 2023-10-16T12:54:12.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fe5fc06afc0 0x7fe5fc06d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe61c155b90 0x7fe61c1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe61c0cb330 0x7fe61c0cb7b0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 --2- 172.21.15.60:0/1685141756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe608003970 0x7fe61c15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:12.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 >> 172.21.15.60:0/1685141756 conn(0x7fe61c0cf570 msgr2=0x7fe61c160dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:12.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 shutdown_connections 2023-10-16T12:54:12.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:12.829+0000 7fe61b59e640 1 -- 172.21.15.60:0/1685141756 wait complete. 2023-10-16T12:54:12.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:54:13.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.056+0000 7f71b8275640 1 Processor -- start 2023-10-16T12:54:13.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.056+0000 7f71b8275640 1 -- start start 2023-10-16T12:54:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b8275640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b8275640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71b006a2e0 0x7f71b00c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b8275640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b006a6f0 con 0x7f71b015d150 2023-10-16T12:54:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b8275640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b015d530 con 0x7f71b006a2e0 2023-10-16T12:54:13.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b8275640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00ac890 con 0x7f71b00ad350 2023-10-16T12:54:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b6812640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71b006a2e0 0x7f71b00c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b6812640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71b006a2e0 0x7f71b00c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39622/0 (socket says 172.21.15.60:39622) 2023-10-16T12:54:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b6812640 1 -- 172.21.15.60:0/2679070359 learned_addr learned my addr 172.21.15.60:0/2679070359 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.058+0000 7f71b5810640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:13.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.059+0000 7f71b6812640 1 -- 172.21.15.60:0/2679070359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 msgr2=0x7f71b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:13.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.059+0000 7f71b6812640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.059+0000 7f71b6812640 1 -- 172.21.15.60:0/2679070359 >> v1:172.21.15.144:6789/0 conn(0x7f71b015d150 legacy=0x7f71b0160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:13.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.059+0000 7f71b6812640 1 -- 172.21.15.60:0/2679070359 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b0161650 con 0x7f71b006a2e0 2023-10-16T12:54:13.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.059+0000 7f71b5810640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:54:13.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.060+0000 7f71b6812640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00c9000 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009e50 tx=0x7f71a802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a230df55e46c02c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:13.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.060+0000 7f71b500f640 1 -- 172.21.15.60:0/2679070359 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a802cb00 con 0x7f71b006a2e0 2023-10-16T12:54:13.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b500f640 1 -- 172.21.15.60:0/2679070359 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a802cc60 con 0x7f71b006a2e0 2023-10-16T12:54:13.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b500f640 1 -- 172.21.15.60:0/2679070359 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a80343f0 con 0x7f71b006a2e0 2023-10-16T12:54:13.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 -- 172.21.15.60:0/2679070359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 msgr2=0x7f71b00c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:13.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00c9000 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009e50 tx=0x7f71a802bf60 comp rx=0 tx=0).stop 2023-10-16T12:54:13.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 -- 172.21.15.60:0/2679070359 shutdown_connections 2023-10-16T12:54:13.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00c9000 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 --2- 172.21.15.60:0/2679070359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 -- 172.21.15.60:0/2679070359 >> 172.21.15.60:0/2679070359 conn(0x7f71b00ce9b0 msgr2=0x7f71b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:13.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 -- 172.21.15.60:0/2679070359 shutdown_connections 2023-10-16T12:54:13.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.061+0000 7f71b8275640 1 -- 172.21.15.60:0/2679070359 wait complete. 2023-10-16T12:54:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.063+0000 7f71b8275640 1 Processor -- start 2023-10-16T12:54:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.064+0000 7f71b8275640 1 -- start start 2023-10-16T12:54:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b8275640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:13.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b8275640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b8275640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b015d150 0x7f71b01f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b8275640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00c3200 con 0x7f71b006a2e0 2023-10-16T12:54:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b8275640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00c3aa0 con 0x7f71b015d150 2023-10-16T12:54:13.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b8275640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00b3df0 con 0x7f71b00ad350 2023-10-16T12:54:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b6011640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b6011640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39636/0 (socket says 172.21.15.60:39636) 2023-10-16T12:54:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b6011640 1 -- 172.21.15.60:0/2709536820 learned_addr learned my addr 172.21.15.60:0/2709536820 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:13.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b5810640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.065+0000 7f71b6812640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b015d150 0x7f71b01f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6011640 1 -- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 msgr2=0x7f71b014d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6011640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b014d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6011640 1 -- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b015d150 msgr2=0x7f71b01f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:13.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6011640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b015d150 0x7f71b01f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6011640 1 -- 172.21.15.60:0/2709536820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a8009b30 con 0x7f71b006a2e0 2023-10-16T12:54:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6812640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b015d150 0x7f71b01f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:13.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b6011640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00b0dc0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c400 tx=0x7f71ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac0052d0 con 0x7f71b006a2e0 2023-10-16T12:54:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac005430 con 0x7f71b006a2e0 2023-10-16T12:54:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b00c3450 con 0x7f71b006a2e0 2023-10-16T12:54:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac003830 con 0x7f71b006a2e0 2023-10-16T12:54:13.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.066+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b014dbe0 con 0x7f71b006a2e0 2023-10-16T12:54:13.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.067+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b004e160 con 0x7f71b006a2e0 2023-10-16T12:54:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.069+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f71ac003990 con 0x7f71b006a2e0 2023-10-16T12:54:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.070+0000 7f71a6ffd640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f719406afc0 0x7f719406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.070+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f71ac088fa0 con 0x7f71b006a2e0 2023-10-16T12:54:13.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.071+0000 7f71b5810640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f719406afc0 0x7f719406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.072+0000 7f71b5810640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f719406afc0 0x7f719406d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f71b00c2e50 tx=0x7f71a0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.075+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f71ac054110 con 0x7f71b006a2e0 2023-10-16T12:54:13.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.309+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f71b00ade10 con 0x7f71b006a2e0 2023-10-16T12:54:13.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.880+0000 7f71a6ffd640 1 -- 172.21.15.60:0/2709536820 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7f71ac057730 con 0x7f71b006a2e0 2023-10-16T12:54:13.890 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 18 and data pool 19 2023-10-16T12:54:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f719406afc0 msgr2=0x7f719406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:13.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f719406afc0 0x7f719406d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f71b00c2e50 tx=0x7f71a0008040 comp rx=0 tx=0).stop 2023-10-16T12:54:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 msgr2=0x7f71b00b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00b0dc0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c400 tx=0x7f71ac00c8c0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 shutdown_connections 2023-10-16T12:54:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f719406afc0 0x7f719406d480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71b00ad350 0x7f71b014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71b015d150 0x7f71b01f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 --2- 172.21.15.60:0/2709536820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71b006a2e0 0x7f71b00b0dc0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.892+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 >> 172.21.15.60:0/2709536820 conn(0x7f71b00ce9b0 msgr2=0x7f71b00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.895+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 shutdown_connections 2023-10-16T12:54:13.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:13.895+0000 7f71b8275640 1 -- 172.21.15.60:0/2709536820 wait complete. 2023-10-16T12:54:13.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:54:14.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.106+0000 7fcfbc30f640 1 Processor -- start 2023-10-16T12:54:14.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.106+0000 7fcfbc30f640 1 -- start start 2023-10-16T12:54:14.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.107+0000 7fcfbc30f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 0x7fcfb415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.107+0000 7fcfbc30f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb415ed60 con 0x7fcfb415ae80 2023-10-16T12:54:14.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.107+0000 7fcfbc30f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb40707e0 con 0x7fcfb415aaa0 2023-10-16T12:54:14.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.107+0000 7fcfbc30f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb406b810 con 0x7fcfb40ce390 2023-10-16T12:54:14.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.108+0000 7fcfb98aa640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcfb415ae80 0x7fcfb4070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54278/0 (socket says 172.21.15.60:54278) 2023-10-16T12:54:14.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.108+0000 7fcfb98aa640 1 -- 172.21.15.60:0/191996513 learned_addr learned my addr 172.21.15.60:0/191996513 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:14.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.108+0000 7fcfba8ac640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 0x7fcfb415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.108+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834854652 0 0) 0x7fcfb415ed60 con 0x7fcfb415ae80 2023-10-16T12:54:14.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.108+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa0003520 con 0x7fcfb415ae80 2023-10-16T12:54:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1578884830 0 0) 0x7fcfa0003520 con 0x7fcfb415ae80 2023-10-16T12:54:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 msgr2=0x7fcfb415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 0x7fcfb415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 >> v1:172.21.15.144:6789/0 conn(0x7fcfb415aaa0 legacy=0x7fcfb40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb415ef90 con 0x7fcfb415ae80 2023-10-16T12:54:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcfa8001690 con 0x7fcfb415ae80 2023-10-16T12:54:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 >> v1:172.21.15.60:6789/0 conn(0x7fcfb415ae80 legacy=0x7fcfb4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 0x7fcfa00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfa00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa0004d00 0x7fcfa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb406b810 con 0x7fcfa0003f70 2023-10-16T12:54:14.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa000bac0 con 0x7fcfa0004d00 2023-10-16T12:54:14.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa000bc80 con 0x7fcfb40ce390 2023-10-16T12:54:14.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.109+0000 7fcfb98aa640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfa00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfba0ab640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 0x7fcfa00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfb98aa640 1 -- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 msgr2=0x7fcfa00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:14.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfb98aa640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb40ce390 0x7fcfa00039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfba8ac640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa0004d00 0x7fcfa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfb98aa640 1 -- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa0004d00 msgr2=0x7fcfa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:14.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfb98aa640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfa0004d00 0x7fcfa0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.110+0000 7fcfb98aa640 1 -- 172.21.15.60:0/191996513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0003520 con 0x7fcfa0003f70 2023-10-16T12:54:14.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.111+0000 7fcfb98aa640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfa00047c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40c2490 tx=0x7fcfa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71a36df17ab6697 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:14.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.111+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8005ad0 con 0x7fcfa0003f70 2023-10-16T12:54:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.111+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa8005c30 con 0x7fcfa0003f70 2023-10-16T12:54:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.111+0000 7fcfb90a9640 1 -- 172.21.15.60:0/191996513 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8007070 con 0x7fcfa0003f70 2023-10-16T12:54:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfbc30f640 1 -- 172.21.15.60:0/191996513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 msgr2=0x7fcfa00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/191996513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfa00047c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40c2490 tx=0x7fcfa00057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfba8ac640 1 -- 172.21.15.60:0/191996513 reap_dead start 2023-10-16T12:54:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfbc30f640 1 -- 172.21.15.60:0/191996513 shutdown_connections 2023-10-16T12:54:14.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfbc30f640 1 -- 172.21.15.60:0/191996513 >> 172.21.15.60:0/191996513 conn(0x7fcfb40cb410 msgr2=0x7fcfb40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:14.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfbc30f640 1 -- 172.21.15.60:0/191996513 shutdown_connections 2023-10-16T12:54:14.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.112+0000 7fcfbc30f640 1 -- 172.21.15.60:0/191996513 wait complete. 2023-10-16T12:54:14.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.114+0000 7fcfbc30f640 1 Processor -- start 2023-10-16T12:54:14.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.114+0000 7fcfbc30f640 1 -- start start 2023-10-16T12:54:14.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfbc30f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfb40c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfbc30f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfbc30f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb415e060 0x7fcfb40af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfbc30f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40c4700 con 0x7fcfa0003f70 2023-10-16T12:54:14.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfbc30f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb415c450 con 0x7fcfb415dbe0 2023-10-16T12:54:14.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfbc30f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb415d750 con 0x7fcfb415e060 2023-10-16T12:54:14.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfb40c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfb40c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39652/0 (socket says 172.21.15.60:39652) 2023-10-16T12:54:14.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 -- 172.21.15.60:0/665897239 learned_addr learned my addr 172.21.15.60:0/665897239 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:14.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 -- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb415e060 msgr2=0x7fcfb40af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb415e060 0x7fcfb40af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 -- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfb415dbe0 msgr2=0x7fcfb40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.115+0000 7fcfba0ab640 1 -- 172.21.15.60:0/665897239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb415d140 con 0x7fcfa0003f70 2023-10-16T12:54:14.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.116+0000 7fcfba0ab640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfb40c41c0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000d570 tx=0x7fcfb000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:14.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.116+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0003610 con 0x7fcfa0003f70 2023-10-16T12:54:14.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.116+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb0003770 con 0x7fcfa0003f70 2023-10-16T12:54:14.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.116+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0003ae0 con 0x7fcfa0003f70 2023-10-16T12:54:14.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.116+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb415c170 con 0x7fcfa0003f70 2023-10-16T12:54:14.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.116+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb40c8e20 con 0x7fcfa0003f70 2023-10-16T12:54:14.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.117+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb415cfa0 con 0x7fcfa0003f70 2023-10-16T12:54:14.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.118+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcfb0003c40 con 0x7fcfa0003f70 2023-10-16T12:54:14.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.120+0000 7fcfa6ffd640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fcf9806af90 0x7fcf9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.120+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfb0089760 con 0x7fcfa0003f70 2023-10-16T12:54:14.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.120+0000 7fcfb98aa640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fcf9806af90 0x7fcf9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.121+0000 7fcfb98aa640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fcf9806af90 0x7fcf9806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8034000 tx=0x7fcfa8007000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:14.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.124+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcfb415cfa0 con 0x7fcfa0003f70 2023-10-16T12:54:14.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.311+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfb415cfa0 con 0x7fcfa0003f70 2023-10-16T12:54:14.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.313+0000 7fcfa6ffd640 1 -- 172.21.15.60:0/665897239 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16580 (secure 0 0 0) 0x7fcfb415cfa0 con 0x7fcfa0003f70 2023-10-16T12:54:14.315 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:14.316 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":155,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:54:13.310039+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:54:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fcf9806af90 msgr2=0x7fcf9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fcf9806af90 0x7fcf9806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8034000 tx=0x7fcfa8007000 comp rx=0 tx=0).stop 2023-10-16T12:54:14.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 msgr2=0x7fcfb40c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfb40c41c0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000d570 tx=0x7fcfb000da30 comp rx=0 tx=0).stop 2023-10-16T12:54:14.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 shutdown_connections 2023-10-16T12:54:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fcf9806af90 0x7fcf9806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfb415e060 0x7fcfb40af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 --2- 172.21.15.60:0/665897239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa0003f70 0x7fcfb40c41c0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 >> 172.21.15.60:0/665897239 conn(0x7fcfb40cb410 msgr2=0x7fcfb406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 shutdown_connections 2023-10-16T12:54:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.316+0000 7fcfbc30f640 1 -- 172.21.15.60:0/665897239 wait complete. 2023-10-16T12:54:14.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:54:14.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.556+0000 7f792759e640 1 Processor -- start 2023-10-16T12:54:14.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.556+0000 7f792759e640 1 -- start start 2023-10-16T12:54:14.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.557+0000 7f792759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f792806fb50 0x7f79280cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.557+0000 7f792759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79280ca560 con 0x7f79280c9060 2023-10-16T12:54:14.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.557+0000 7f792759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79280c9950 con 0x7f792806c9e0 2023-10-16T12:54:14.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.557+0000 7f792759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280c9c30 con 0x7f792806fb50 2023-10-16T12:54:14.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f79280c9060 0x7f792806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54290/0 (socket says 172.21.15.60:54290) 2023-10-16T12:54:14.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792659c640 1 -- 172.21.15.60:0/3454848247 learned_addr learned my addr 172.21.15.60:0/3454848247 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:14.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f7925d9b640 1 --2- 172.21.15.60:0/3454848247 >> v2:172.21.15.144:3300/0 conn(0x7f792806fb50 0x7f79280cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716976748 0 0) 0x7f79280ca560 con 0x7f79280c9060 2023-10-16T12:54:14.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7910003520 con 0x7f79280c9060 2023-10-16T12:54:14.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129066098 0 0) 0x7f7910003520 con 0x7f79280c9060 2023-10-16T12:54:14.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 >> v2:172.21.15.144:3300/0 conn(0x7f792806fb50 msgr2=0x7f79280cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:14.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 --2- 172.21.15.60:0/3454848247 >> v2:172.21.15.144:3300/0 conn(0x7f792806fb50 0x7f79280cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 >> v1:172.21.15.144:6789/0 conn(0x7f792806c9e0 legacy=0x7f79280ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.558+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280c9f10 con 0x7f79280c9060 2023-10-16T12:54:14.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f79180019f0 con 0x7f79280c9060 2023-10-16T12:54:14.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 >> v1:172.21.15.60:6789/0 conn(0x7f79280c9060 legacy=0x7f792806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910003970 0x7f7910003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79100047c0 0x7f7910004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7910005180 0x7f791000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280c9c30 con 0x7f7910005180 2023-10-16T12:54:14.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791000c020 con 0x7f79100047c0 2023-10-16T12:54:14.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791000c1e0 con 0x7f7910003970 2023-10-16T12:54:14.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f7926d9d640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7910005180 0x7f791000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f7925d9b640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79100047c0 0x7f7910004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.559+0000 7f792659c640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910003970 0x7f7910003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.560+0000 7f7926d9d640 1 -- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910003970 msgr2=0x7f7910003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:14.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.560+0000 7f7926d9d640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910003970 0x7f7910003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.560+0000 7f7926d9d640 1 -- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79100047c0 msgr2=0x7f7910004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:14.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.560+0000 7f7926d9d640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79100047c0 0x7f7910004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.560+0000 7f7926d9d640 1 -- 172.21.15.60:0/3454848247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7910003520 con 0x7f7910005180 2023-10-16T12:54:14.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.560+0000 7f7925d9b640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79100047c0 0x7f7910004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:54:14.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f7926d9d640 1 -- 172.21.15.60:0/3454848247 reap_dead start 2023-10-16T12:54:14.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f7926d9d640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7910005180 0x7f791000b980 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f7920001720 tx=0x7f79100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da4b97c1c8a1c9d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:14.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79200052d0 con 0x7f7910005180 2023-10-16T12:54:14.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792002e050 con 0x7f7910005180 2023-10-16T12:54:14.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f792559a640 1 -- 172.21.15.60:0/3454848247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79200035f0 con 0x7f7910005180 2023-10-16T12:54:14.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f792759e640 1 -- 172.21.15.60:0/3454848247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7910005180 msgr2=0x7f791000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.561+0000 7f792759e640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7910005180 0x7f791000b980 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f7920001720 tx=0x7f79100057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.562+0000 7f792759e640 1 -- 172.21.15.60:0/3454848247 shutdown_connections 2023-10-16T12:54:14.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.562+0000 7f792759e640 1 --2- 172.21.15.60:0/3454848247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7910005180 0x7f791000b980 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.562+0000 7f792759e640 1 -- 172.21.15.60:0/3454848247 >> 172.21.15.60:0/3454848247 conn(0x7f79280ce6a0 msgr2=0x7f79280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:14.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.562+0000 7f792759e640 1 -- 172.21.15.60:0/3454848247 shutdown_connections 2023-10-16T12:54:14.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.562+0000 7f792759e640 1 -- 172.21.15.60:0/3454848247 wait complete. 2023-10-16T12:54:14.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.564+0000 7f792759e640 1 Processor -- start 2023-10-16T12:54:14.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.564+0000 7f792759e640 1 -- start start 2023-10-16T12:54:14.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910005180 0x7f7928150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f792815a1e0 0x7f79280aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 0x7f79280c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79281511a0 con 0x7f79280aef50 2023-10-16T12:54:14.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280af3d0 con 0x7f792815a1e0 2023-10-16T12:54:14.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79281579f0 con 0x7f7910005180 2023-10-16T12:54:14.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 0x7f79280c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 0x7f79280c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39672/0 (socket says 172.21.15.60:39672) 2023-10-16T12:54:14.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 -- 172.21.15.60:0/1520826135 learned_addr learned my addr 172.21.15.60:0/1520826135 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:14.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f792659c640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910005180 0x7f7928150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 -- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910005180 msgr2=0x7f7928150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7925d9b640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f792815a1e0 0x7f79280aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910005180 0x7f7928150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 -- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f792815a1e0 msgr2=0x7f79280aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f792815a1e0 0x7f79280aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.565+0000 7f7926d9d640 1 -- 172.21.15.60:0/1520826135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7928158850 con 0x7f79280aef50 2023-10-16T12:54:14.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f792659c640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910005180 0x7f7928150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:14.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f7925d9b640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f792815a1e0 0x7f79280aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:14.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f7926d9d640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 0x7f79280c40a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f7920001840 tx=0x7f7920001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:14.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920007a30 con 0x7f79280aef50 2023-10-16T12:54:14.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7920038040 con 0x7f79280aef50 2023-10-16T12:54:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920007250 con 0x7f79280aef50 2023-10-16T12:54:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7928157c40 con 0x7f79280aef50 2023-10-16T12:54:14.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.566+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7928158200 con 0x7f79280aef50 2023-10-16T12:54:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.567+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792804e9b0 con 0x7f79280aef50 2023-10-16T12:54:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.568+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79200073b0 con 0x7f79280aef50 2023-10-16T12:54:14.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.570+0000 7f7916ffd640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f790806afe0 0x7f790806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.570+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79200aa330 con 0x7f79280aef50 2023-10-16T12:54:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.571+0000 7f792659c640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f790806afe0 0x7f790806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:14.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.572+0000 7f792659c640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f790806afe0 0x7f790806d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7918001580 tx=0x7f79180014f0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:14.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.575+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7920075460 con 0x7f79280aef50 2023-10-16T12:54:14.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.773+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79280b4620 con 0x7f79280aef50 2023-10-16T12:54:14.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.774+0000 7f7916ffd640 1 -- 172.21.15.60:0/1520826135 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f7920078a80 con 0x7f79280aef50 2023-10-16T12:54:14.776 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:14.777 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":155,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:54:13.310039+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:54:14.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f790806afe0 msgr2=0x7f790806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f790806afe0 0x7f790806d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7918001580 tx=0x7f79180014f0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 msgr2=0x7f79280c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:14.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 0x7f79280c40a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f7920001840 tx=0x7f7920001120 comp rx=0 tx=0).stop 2023-10-16T12:54:14.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 shutdown_connections 2023-10-16T12:54:14.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f790806afe0 0x7f790806d4a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79280aef50 0x7f79280c40a0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f792815a1e0 0x7f79280aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 --2- 172.21.15.60:0/1520826135 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7910005180 0x7f7928150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:14.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.777+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 >> 172.21.15.60:0/1520826135 conn(0x7f79280ce6a0 msgr2=0x7f792806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:14.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.778+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 shutdown_connections 2023-10-16T12:54:14.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:14.778+0000 7f792759e640 1 -- 172.21.15.60:0/1520826135 wait complete. 2023-10-16T12:54:14.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:54:15.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.031+0000 7f75c3fff640 1 Processor -- start 2023-10-16T12:54:15.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.031+0000 7f75c3fff640 1 -- start start 2023-10-16T12:54:15.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75c3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75c40b8480 0x7f75c40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75c3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f75c40d0be0 0x7f75c40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75c3fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c415d150 con 0x7f75c40b8480 2023-10-16T12:54:15.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75c3fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c40d1180 con 0x7f75c40d0be0 2023-10-16T12:54:15.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75c3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40d1460 con 0x7f75c415cd70 2023-10-16T12:54:15.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75ba7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75c40b8480 0x7f75c40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75ba7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75c40b8480 0x7f75c40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39674/0 (socket says 172.21.15.60:39674) 2023-10-16T12:54:15.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.033+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1629211129 learned_addr learned my addr 172.21.15.60:0/1629211129 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:15.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75c2ffd640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1629211129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 msgr2=0x7f75c40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75c37fe640 1 --2- 172.21.15.60:0/1629211129 >> v2:172.21.15.144:3300/0 conn(0x7f75c40d0be0 0x7f75c40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75ba7fc640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c40d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1629211129 >> v2:172.21.15.144:3300/0 conn(0x7f75c40d0be0 msgr2=0x7f75c40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75ba7fc640 1 --2- 172.21.15.60:0/1629211129 >> v2:172.21.15.144:3300/0 conn(0x7f75c40d0be0 0x7f75c40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1629211129 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c40d1740 con 0x7f75c40b8480 2023-10-16T12:54:15.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75c2ffd640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:15.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.034+0000 7f75c37fe640 1 --2- 172.21.15.60:0/1629211129 >> v2:172.21.15.144:3300/0 conn(0x7f75c40d0be0 0x7f75c40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:15.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.035+0000 7f75ba7fc640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40b8480 0x7f75c40cd7c0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f75c40b2520 tx=0x7f75ac02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8062c49d272b7b9e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:15.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.035+0000 7f75c27fc640 1 -- 172.21.15.60:0/1629211129 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac031860 con 0x7f75c40b8480 2023-10-16T12:54:15.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c27fc640 1 -- 172.21.15.60:0/1629211129 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac0319c0 con 0x7f75c40b8480 2023-10-16T12:54:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c27fc640 1 -- 172.21.15.60:0/1629211129 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac031d50 con 0x7f75c40b8480 2023-10-16T12:54:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 -- 172.21.15.60:0/1629211129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40b8480 msgr2=0x7f75c40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40b8480 0x7f75c40cd7c0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f75c40b2520 tx=0x7f75ac02b900 comp rx=0 tx=0).stop 2023-10-16T12:54:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 -- 172.21.15.60:0/1629211129 shutdown_connections 2023-10-16T12:54:15.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1629211129 >> v2:172.21.15.144:3300/0 conn(0x7f75c40d0be0 0x7f75c40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40b8480 0x7f75c40cd7c0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1629211129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 -- 172.21.15.60:0/1629211129 >> 172.21.15.60:0/1629211129 conn(0x7f75c40cf570 msgr2=0x7f75c40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:15.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 -- 172.21.15.60:0/1629211129 shutdown_connections 2023-10-16T12:54:15.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.036+0000 7f75c3fff640 1 -- 172.21.15.60:0/1629211129 wait complete. 2023-10-16T12:54:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.038+0000 7f75c3fff640 1 Processor -- start 2023-10-16T12:54:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.038+0000 7f75c3fff640 1 -- start start 2023-10-16T12:54:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75c3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c40b8480 0x7f75c40c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 0x7f75c4158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75c3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c4155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75c3fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40c85b0 con 0x7f75c40d0be0 2023-10-16T12:54:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75c3fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41f2060 con 0x7f75c40b8480 2023-10-16T12:54:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75c3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40c7930 con 0x7f75c415cd70 2023-10-16T12:54:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75ba7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 0x7f75c4158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75ba7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 0x7f75c4158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39676/0 (socket says 172.21.15.60:39676) 2023-10-16T12:54:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.039+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1351026190 learned_addr learned my addr 172.21.15.60:0/1351026190 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75c2ffd640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c40b8480 0x7f75c40c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75c37fe640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c4155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 msgr2=0x7f75c4155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75ba7fc640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c4155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c40b8480 msgr2=0x7f75c40c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75ba7fc640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c40b8480 0x7f75c40c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75ba7fc640 1 -- 172.21.15.60:0/1351026190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0094d0 con 0x7f75c40d0be0 2023-10-16T12:54:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75c2ffd640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c40b8480 0x7f75c40c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.040+0000 7f75ba7fc640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 0x7f75c4158af0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f75ac02b940 tx=0x7f75ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.041+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac031850 con 0x7f75c40d0be0 2023-10-16T12:54:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.041+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac0443f0 con 0x7f75c40d0be0 2023-10-16T12:54:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.041+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0447a0 con 0x7f75c40d0be0 2023-10-16T12:54:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.041+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c40c7c10 con 0x7f75c40d0be0 2023-10-16T12:54:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.041+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c40c81d0 con 0x7f75c40d0be0 2023-10-16T12:54:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.042+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c40d28f0 con 0x7f75c40d0be0 2023-10-16T12:54:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.043+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75ac04d450 con 0x7f75c40d0be0 2023-10-16T12:54:15.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.044+0000 7f75bbfff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f75a406af90 0x7f75a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:15.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.045+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75ac0abf50 con 0x7f75c40d0be0 2023-10-16T12:54:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.045+0000 7f75c2ffd640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f75a406af90 0x7f75a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.046+0000 7f75c2ffd640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f75a406af90 0x7f75a406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75b4006fd0 tx=0x7f75b4008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.049+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75ac076e70 con 0x7f75c40d0be0 2023-10-16T12:54:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.291+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f75c40c86f0 con 0x7f75c40d0be0 2023-10-16T12:54:15.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.908+0000 7f75bbfff640 1 -- 172.21.15.60:0/1351026190 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v78) v1 ==== 118+0+0 (secure 0 0 0) 0x7f75c40c86f0 con 0x7f75c40d0be0 2023-10-16T12:54:15.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f75a406af90 msgr2=0x7f75a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:15.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f75a406af90 0x7f75a406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75b4006fd0 tx=0x7f75b4008040 comp rx=0 tx=0).stop 2023-10-16T12:54:15.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 msgr2=0x7f75c4158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:15.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 0x7f75c4158af0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f75ac02b940 tx=0x7f75ac0046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 shutdown_connections 2023-10-16T12:54:15.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f75a406af90 0x7f75a406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75c415cd70 0x7f75c4155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75c40d0be0 0x7f75c4158af0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 --2- 172.21.15.60:0/1351026190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75c40b8480 0x7f75c40c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:15.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 >> 172.21.15.60:0/1351026190 conn(0x7f75c40cf570 msgr2=0x7f75c40cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:15.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 shutdown_connections 2023-10-16T12:54:15.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:15.913+0000 7f75c3fff640 1 -- 172.21.15.60:0/1351026190 wait complete. 2023-10-16T12:54:15.940 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:54:15.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:54:16.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.175+0000 7fb34bf41640 1 Processor -- start 2023-10-16T12:54:16.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.175+0000 7fb34bf41640 1 -- start start 2023-10-16T12:54:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb34bf41640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb344159ab0 0x7fb3440ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb34bf41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb34406c140 0x7fb34406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb34bf41640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb344070360 con 0x7fb344159e90 2023-10-16T12:54:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb34bf41640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34406cb10 con 0x7fb344159ab0 2023-10-16T12:54:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb34bf41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415d890 con 0x7fb34406c140 2023-10-16T12:54:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb349cdd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb344159ab0 0x7fb3440ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb349cdd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb344159ab0 0x7fb3440ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39690/0 (socket says 172.21.15.60:39690) 2023-10-16T12:54:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.177+0000 7fb349cdd640 1 -- 172.21.15.60:0/3156534362 learned_addr learned my addr 172.21.15.60:0/3156534362 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb349cdd640 1 -- 172.21.15.60:0/3156534362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb34406c140 msgr2=0x7fb34406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb34a4de640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb34406c140 0x7fb34406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb348cdb640 1 -- 172.21.15.60:0/3156534362 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973390066 0 0) 0x7fb344070360 con 0x7fb344159e90 2023-10-16T12:54:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb349cdd640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb34406c140 0x7fb34406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb349cdd640 1 -- 172.21.15.60:0/3156534362 >> v1:172.21.15.60:6789/0 conn(0x7fb344159e90 legacy=0x7fb34406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb349cdd640 1 -- 172.21.15.60:0/3156534362 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440cd1f0 con 0x7fb344159ab0 2023-10-16T12:54:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.178+0000 7fb34a4de640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb34406c140 0x7fb34406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.179+0000 7fb349cdd640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb3440ce370 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb34000bb10 tx=0x7fb34002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f848f039180d58d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:16.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.180+0000 7fb348cdb640 1 -- 172.21.15.60:0/3156534362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34002edf0 con 0x7fb344159ab0 2023-10-16T12:54:16.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.180+0000 7fb348cdb640 1 -- 172.21.15.60:0/3156534362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb340036440 con 0x7fb344159ab0 2023-10-16T12:54:16.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.180+0000 7fb348cdb640 1 -- 172.21.15.60:0/3156534362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3400367d0 con 0x7fb344159ab0 2023-10-16T12:54:16.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.180+0000 7fb34bf41640 1 -- 172.21.15.60:0/3156534362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 msgr2=0x7fb3440ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:16.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.180+0000 7fb34bf41640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb3440ce370 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb34000bb10 tx=0x7fb34002e040 comp rx=0 tx=0).stop 2023-10-16T12:54:16.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.181+0000 7fb34bf41640 1 -- 172.21.15.60:0/3156534362 shutdown_connections 2023-10-16T12:54:16.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.181+0000 7fb34bf41640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb34406c140 0x7fb34406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.181+0000 7fb34bf41640 1 --2- 172.21.15.60:0/3156534362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb3440ce370 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.181+0000 7fb34bf41640 1 -- 172.21.15.60:0/3156534362 >> 172.21.15.60:0/3156534362 conn(0x7fb3440ce9b0 msgr2=0x7fb3440cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:16.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.181+0000 7fb34bf41640 1 -- 172.21.15.60:0/3156534362 shutdown_connections 2023-10-16T12:54:16.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.181+0000 7fb34bf41640 1 -- 172.21.15.60:0/3156534362 wait complete. 2023-10-16T12:54:16.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.182+0000 7fb34bf41640 1 Processor -- start 2023-10-16T12:54:16.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.183+0000 7fb34bf41640 1 -- start start 2023-10-16T12:54:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.183+0000 7fb34bf41640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34406c140 0x7fb3441eed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.183+0000 7fb34bf41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb34415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.183+0000 7fb34bf41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb344159e90 0x7fb34415af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb34bf41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415cfb0 con 0x7fb344159ab0 2023-10-16T12:54:16.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb34bf41640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415d290 con 0x7fb34406c140 2023-10-16T12:54:16.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb34bf41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415d570 con 0x7fb344159e90 2023-10-16T12:54:16.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb34415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:16.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb34415a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39702/0 (socket says 172.21.15.60:39702) 2023-10-16T12:54:16.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 -- 172.21.15.60:0/86391415 learned_addr learned my addr 172.21.15.60:0/86391415 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:16.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 -- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb344159e90 msgr2=0x7fb34415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:16.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb34a4de640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb344159e90 0x7fb34415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:16.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb349cdd640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34406c140 0x7fb3441eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:16.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb344159e90 0x7fb34415af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 -- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34406c140 msgr2=0x7fb3441eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:16.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34406c140 0x7fb3441eed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb3494dc640 1 -- 172.21.15.60:0/86391415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34000b770 con 0x7fb344159ab0 2023-10-16T12:54:16.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.184+0000 7fb34a4de640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb344159e90 0x7fb34415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:16.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.185+0000 7fb3494dc640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb34415a330 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb33400c4b0 tx=0x7fb33400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:16.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.185+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334005480 con 0x7fb344159ab0 2023-10-16T12:54:16.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.185+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3340035f0 con 0x7fb344159ab0 2023-10-16T12:54:16.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.185+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3441ee550 con 0x7fb344159ab0 2023-10-16T12:54:16.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.185+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3340039e0 con 0x7fb344159ab0 2023-10-16T12:54:16.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.185+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3441ef290 con 0x7fb344159ab0 2023-10-16T12:54:16.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.186+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34404e160 con 0x7fb344159ab0 2023-10-16T12:54:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.189+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb334003b40 con 0x7fb344159ab0 2023-10-16T12:54:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.190+0000 7fb33a7fc640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb32806b010 0x7fb32806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.190+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb334088c20 con 0x7fb344159ab0 2023-10-16T12:54:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.190+0000 7fb349cdd640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb32806b010 0x7fb32806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.191+0000 7fb349cdd640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb32806b010 0x7fb32806d4d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb340038000 tx=0x7fb340032000 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:16.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.194+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb334053d30 con 0x7fb344159ab0 2023-10-16T12:54:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.430+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb3440ad4d0 con 0x7fb344159ab0 2023-10-16T12:54:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.918+0000 7fb33a7fc640 1 -- 172.21.15.60:0/86391415 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v79) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb334017070 con 0x7fb344159ab0 2023-10-16T12:54:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb32806b010 msgr2=0x7fb32806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb32806b010 0x7fb32806d4d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb340038000 tx=0x7fb340032000 comp rx=0 tx=0).stop 2023-10-16T12:54:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 msgr2=0x7fb34415a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb34415a330 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb33400c4b0 tx=0x7fb33400c970 comp rx=0 tx=0).stop 2023-10-16T12:54:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 shutdown_connections 2023-10-16T12:54:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fb32806b010 0x7fb32806d4d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb344159e90 0x7fb34415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb344159ab0 0x7fb34415a330 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 --2- 172.21.15.60:0/86391415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb34406c140 0x7fb3441eed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.922+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 >> 172.21.15.60:0/86391415 conn(0x7fb3440ce9b0 msgr2=0x7fb344070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.923+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 shutdown_connections 2023-10-16T12:54:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:16.923+0000 7fb34bf41640 1 -- 172.21.15.60:0/86391415 wait complete. 2023-10-16T12:54:16.946 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:54:16.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:54:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.172+0000 7f611bfff640 1 Processor -- start 2023-10-16T12:54:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.172+0000 7f611bfff640 1 -- start start 2023-10-16T12:54:17.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.174+0000 7f611bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.174+0000 7f611bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f611c070f20 0x7f611c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.174+0000 7f611bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611c160a90 con 0x7f611c15c9f0 2023-10-16T12:54:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.174+0000 7f611bfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611c0cdec0 con 0x7f611c070f20 2023-10-16T12:54:17.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.174+0000 7f611bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c0cd900 con 0x7f611c15dde0 2023-10-16T12:54:17.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.175+0000 7f6113fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.175+0000 7f611b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f611c070f20 0x7f611c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.175+0000 7f611b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f611c070f20 0x7f611c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:54706/0 (socket says 172.21.15.60:54706) 2023-10-16T12:54:17.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.175+0000 7f611b7fe640 1 -- 172.21.15.60:0/486262973 learned_addr learned my addr 172.21.15.60:0/486262973 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:17.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.175+0000 7f6113fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:38190/0 (socket says 172.21.15.60:38190) 2023-10-16T12:54:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.176+0000 7f611b7fe640 1 -- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 msgr2=0x7f611c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.176+0000 7f611b7fe640 1 --2- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.176+0000 7f611b7fe640 1 -- 172.21.15.60:0/486262973 >> v1:172.21.15.144:6789/0 conn(0x7f611c15c9f0 legacy=0x7f611c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.176+0000 7f611b7fe640 1 -- 172.21.15.60:0/486262973 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c0cdbe0 con 0x7f611c070f20 2023-10-16T12:54:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.178+0000 7f611b7fe640 1 --2- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c0ac890 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6114009e50 tx=0x7f611402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87773aac2b7f5037 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.178+0000 7f611a7fc640 1 -- 172.21.15.60:0/486262973 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611402caf0 con 0x7f611c070f20 2023-10-16T12:54:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.178+0000 7f611a7fc640 1 -- 172.21.15.60:0/486262973 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611402cc50 con 0x7f611c070f20 2023-10-16T12:54:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.178+0000 7f611a7fc640 1 -- 172.21.15.60:0/486262973 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61140343f0 con 0x7f611c070f20 2023-10-16T12:54:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 -- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 msgr2=0x7f611c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 --2- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c0ac890 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6114009e50 tx=0x7f611402bf50 comp rx=0 tx=0).stop 2023-10-16T12:54:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 -- 172.21.15.60:0/486262973 shutdown_connections 2023-10-16T12:54:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 --2- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c0ac890 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 --2- 172.21.15.60:0/486262973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 -- 172.21.15.60:0/486262973 >> 172.21.15.60:0/486262973 conn(0x7f611c0ce6a0 msgr2=0x7f611c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 -- 172.21.15.60:0/486262973 shutdown_connections 2023-10-16T12:54:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.179+0000 7f611bfff640 1 -- 172.21.15.60:0/486262973 wait complete. 2023-10-16T12:54:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.182+0000 7f611bfff640 1 Processor -- start 2023-10-16T12:54:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.182+0000 7f611bfff640 1 -- start start 2023-10-16T12:54:17.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f611bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c1f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f611bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 0x7f611c0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f611bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c06c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f611bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c1f2be0 con 0x7f611c15c9f0 2023-10-16T12:54:17.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f611bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c0c4c40 con 0x7f611c070f20 2023-10-16T12:54:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f611bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c06cce0 con 0x7f611c15dde0 2023-10-16T12:54:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f6113fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 0x7f611c0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f6113fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 0x7f611c0c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39718/0 (socket says 172.21.15.60:39718) 2023-10-16T12:54:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.183+0000 7f6113fff640 1 -- 172.21.15.60:0/3499484422 learned_addr learned my addr 172.21.15.60:0/3499484422 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:17.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f611affd640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c1f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f611b7fe640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c06c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f6113fff640 1 -- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 msgr2=0x7f611c06c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f6113fff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c06c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f6113fff640 1 -- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 msgr2=0x7f611c1f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f6113fff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c1f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f6113fff640 1 -- 172.21.15.60:0/3499484422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6114009b30 con 0x7f611c15c9f0 2023-10-16T12:54:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f611b7fe640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c06c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.184+0000 7f6113fff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 0x7f611c0c4700 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f610400e870 tx=0x7f610400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.185+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61040094f0 con 0x7f611c15c9f0 2023-10-16T12:54:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.185+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610400c940 con 0x7f611c15c9f0 2023-10-16T12:54:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.185+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400ccd0 con 0x7f611c15c9f0 2023-10-16T12:54:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.185+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c1f3690 con 0x7f611c15c9f0 2023-10-16T12:54:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.185+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c0c2390 con 0x7f611c15c9f0 2023-10-16T12:54:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.186+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c04e9b0 con 0x7f611c15c9f0 2023-10-16T12:54:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.187+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6104013470 con 0x7f611c15c9f0 2023-10-16T12:54:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.188+0000 7f61137fe640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60fc06afc0 0x7f60fc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.188+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61040897d0 con 0x7f611c15c9f0 2023-10-16T12:54:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.189+0000 7f611affd640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60fc06afc0 0x7f60fc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.190+0000 7f611affd640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60fc06afc0 0x7f60fc06d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f611c0caaf0 tx=0x7f610c009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.193+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6104054760 con 0x7f611c15c9f0 2023-10-16T12:54:17.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.433+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f611c15aca0 con 0x7f611c15c9f0 2023-10-16T12:54:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.434+0000 7f61137fe640 1 -- 172.21.15.60:0/3499484422 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f6104057d80 con 0x7f611c15c9f0 2023-10-16T12:54:17.436 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:17.436 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:54:13.310628+0000","modified":"2023-10-16T12:54:16.431585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8964":{"gid":8964,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3372982800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3372982800},{"type":"v1","addr":"172.21.15.60:6833","nonce":3372982800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T12:54:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 79 2023-10-16T12:54:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60fc06afc0 msgr2=0x7f60fc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60fc06afc0 0x7f60fc06d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f611c0caaf0 tx=0x7f610c009040 comp rx=0 tx=0).stop 2023-10-16T12:54:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 msgr2=0x7f611c0c4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 0x7f611c0c4700 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f610400e870 tx=0x7f610400ed30 comp rx=0 tx=0).stop 2023-10-16T12:54:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 shutdown_connections 2023-10-16T12:54:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60fc06afc0 0x7f60fc06d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f611c15dde0 0x7f611c06c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f611c15c9f0 0x7f611c0c4700 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 --2- 172.21.15.60:0/3499484422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f611c070f20 0x7f611c1f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 >> 172.21.15.60:0/3499484422 conn(0x7f611c0ce6a0 msgr2=0x7f611c0c15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 shutdown_connections 2023-10-16T12:54:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.437+0000 7f611bfff640 1 -- 172.21.15.60:0/3499484422 wait complete. 2023-10-16T12:54:17.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:54:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.678+0000 7f7ed6bd5640 1 Processor -- start 2023-10-16T12:54:17.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.679+0000 7f7ed6bd5640 1 -- start start 2023-10-16T12:54:17.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.680+0000 7f7ed6bd5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7ed01593b0 0x7f7ed006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.680+0000 7f7ed6bd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed006fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.680+0000 7f7ed6bd5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed0159790 con 0x7f7ed006ffd0 2023-10-16T12:54:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.680+0000 7f7ed6bd5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed00c9b20 con 0x7f7ed01593b0 2023-10-16T12:54:17.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.680+0000 7f7ed6bd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00c9e00 con 0x7f7ed00caa10 2023-10-16T12:54:17.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.681+0000 7f7ed4971640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7ed01593b0 0x7f7ed006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.681+0000 7f7ed4971640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7ed01593b0 0x7f7ed006f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:54732/0 (socket says 172.21.15.60:54732) 2023-10-16T12:54:17.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.681+0000 7f7ed4971640 1 -- 172.21.15.60:0/851755758 learned_addr learned my addr 172.21.15.60:0/851755758 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:17.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.681+0000 7f7ecffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed006fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.682+0000 7f7ed4971640 1 -- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 msgr2=0x7f7ed006fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:17.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.682+0000 7f7ed4971640 1 --2- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed006fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.682+0000 7f7ed4971640 1 -- 172.21.15.60:0/851755758 >> v1:172.21.15.144:6789/0 conn(0x7f7ed006ffd0 legacy=0x7f7ed015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.682+0000 7f7ed4971640 1 -- 172.21.15.60:0/851755758 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed00ca0e0 con 0x7f7ed01593b0 2023-10-16T12:54:17.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.683+0000 7f7ed4971640 1 --2- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed006f550 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009870 tx=0x7f7ec402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8544a58d26915600 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:17.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.684+0000 7f7ecf7fe640 1 -- 172.21.15.60:0/851755758 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40317c0 con 0x7f7ed01593b0 2023-10-16T12:54:17.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.684+0000 7f7ecf7fe640 1 -- 172.21.15.60:0/851755758 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec4031920 con 0x7f7ed01593b0 2023-10-16T12:54:17.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.684+0000 7f7ecf7fe640 1 -- 172.21.15.60:0/851755758 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4031cb0 con 0x7f7ed01593b0 2023-10-16T12:54:17.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.684+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 msgr2=0x7f7ed006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.684+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed006f550 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009870 tx=0x7f7ec402b860 comp rx=0 tx=0).stop 2023-10-16T12:54:17.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.685+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/851755758 shutdown_connections 2023-10-16T12:54:17.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.685+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed006fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.685+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/851755758 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed006f550 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.685+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/851755758 >> 172.21.15.60:0/851755758 conn(0x7f7ed00cb410 msgr2=0x7f7ed00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:17.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.685+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/851755758 shutdown_connections 2023-10-16T12:54:17.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.685+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/851755758 wait complete. 2023-10-16T12:54:17.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.687+0000 7f7ed6bd5640 1 Processor -- start 2023-10-16T12:54:17.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.687+0000 7f7ed6bd5640 1 -- start start 2023-10-16T12:54:17.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed6bd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 0x7f7ed00b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed6bd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed6bd5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed0153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed6bd5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00b1a70 con 0x7f7ed006ffd0 2023-10-16T12:54:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed6bd5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed014d3c0 con 0x7f7ed01593b0 2023-10-16T12:54:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed6bd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00b4370 con 0x7f7ed00caa10 2023-10-16T12:54:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed4971640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 0x7f7ed00b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed4971640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 0x7f7ed00b1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39732/0 (socket says 172.21.15.60:39732) 2023-10-16T12:54:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.689+0000 7f7ed4971640 1 -- 172.21.15.60:0/905020093 learned_addr learned my addr 172.21.15.60:0/905020093 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:17.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ecffff640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed5172640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed0153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed4971640 1 -- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 msgr2=0x7f7ed014ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed4971640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed014ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed4971640 1 -- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 msgr2=0x7f7ed0153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed4971640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed0153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed4971640 1 -- 172.21.15.60:0/905020093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec40094d0 con 0x7f7ed006ffd0 2023-10-16T12:54:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed5172640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed0153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed4971640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 0x7f7ed00b1530 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4000c00 tx=0x7f7ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4031f10 con 0x7f7ed006ffd0 2023-10-16T12:54:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec4032860 con 0x7f7ed006ffd0 2023-10-16T12:54:17.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4032c10 con 0x7f7ed006ffd0 2023-10-16T12:54:17.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.690+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed00c6ca0 con 0x7f7ed006ffd0 2023-10-16T12:54:17.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.691+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed00c7260 con 0x7f7ed006ffd0 2023-10-16T12:54:17.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.692+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed004e160 con 0x7f7ed006ffd0 2023-10-16T12:54:17.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.693+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7ec404e450 con 0x7f7ed006ffd0 2023-10-16T12:54:17.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.694+0000 7f7ecd7fa640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7eb406af90 0x7f7eb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:17.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.694+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ec40ab950 con 0x7f7ed006ffd0 2023-10-16T12:54:17.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.694+0000 7f7ecffff640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7eb406af90 0x7f7eb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:17.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.696+0000 7f7ecffff640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7eb406af90 0x7f7eb406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ed01555f0 tx=0x7f7ebc008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:17.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.698+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7ec40769a0 con 0x7f7ed006ffd0 2023-10-16T12:54:17.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.893+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ed00b4910 con 0x7f7ed006ffd0 2023-10-16T12:54:17.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.894+0000 7f7ecd7fa640 1 -- 172.21.15.60:0/905020093 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) v1 ==== 74+0+16567 (secure 0 0 0) 0x7f7ec4079fc0 con 0x7f7ed006ffd0 2023-10-16T12:54:17.896 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:17.897 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":156,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T12:54:14.856111+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:54:17.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.897+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7eb406af90 msgr2=0x7f7eb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.897+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7eb406af90 0x7f7eb406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ed01555f0 tx=0x7f7ebc008040 comp rx=0 tx=0).stop 2023-10-16T12:54:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.897+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 msgr2=0x7f7ed00b1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.897+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 0x7f7ed00b1530 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4000c00 tx=0x7f7ec40046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 shutdown_connections 2023-10-16T12:54:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7eb406af90 0x7f7eb406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ed00caa10 0x7f7ed014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed01593b0 0x7f7ed0153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 --2- 172.21.15.60:0/905020093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ed006ffd0 0x7f7ed00b1530 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 >> 172.21.15.60:0/905020093 conn(0x7f7ed00cb410 msgr2=0x7f7ed00ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 shutdown_connections 2023-10-16T12:54:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:17.898+0000 7f7ed6bd5640 1 -- 172.21.15.60:0/905020093 wait complete. 2023-10-16T12:54:17.918 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:54:17.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:54:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.149+0000 7f6107e0e640 1 Processor -- start 2023-10-16T12:54:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.149+0000 7f6107e0e640 1 -- start start 2023-10-16T12:54:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f6107e0e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f610015ae80 0x7f610015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f6107e0e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61000ce390 0x7f6100070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f6107e0e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610015ed60 con 0x7f610015aaa0 2023-10-16T12:54:18.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f6107e0e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610006b810 con 0x7f61000ce390 2023-10-16T12:54:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f6107e0e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61000707e0 con 0x7f610015ae80 2023-10-16T12:54:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f61063ab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61000ce390 0x7f6100070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f61063ab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61000ce390 0x7f6100070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39736/0 (socket says 172.21.15.60:39736) 2023-10-16T12:54:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.151+0000 7f61063ab640 1 -- 172.21.15.60:0/3267239893 learned_addr learned my addr 172.21.15.60:0/3267239893 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.152+0000 7f61053a9640 1 --2- 172.21.15.60:0/3267239893 >> v2:172.21.15.144:3300/0 conn(0x7f610015ae80 0x7f610015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.152+0000 7f61063ab640 1 -- 172.21.15.60:0/3267239893 >> v2:172.21.15.144:3300/0 conn(0x7f610015ae80 msgr2=0x7f610015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:18.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.152+0000 7f61063ab640 1 --2- 172.21.15.60:0/3267239893 >> v2:172.21.15.144:3300/0 conn(0x7f610015ae80 0x7f610015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.152+0000 7f61063ab640 1 -- 172.21.15.60:0/3267239893 >> v1:172.21.15.144:6789/0 conn(0x7f610015aaa0 legacy=0x7f61000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:18.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.152+0000 7f61063ab640 1 -- 172.21.15.60:0/3267239893 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610015ef90 con 0x7f61000ce390 2023-10-16T12:54:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.153+0000 7f61063ab640 1 --2- 172.21.15.60:0/3267239893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100070f50 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f61000c59c0 tx=0x7f60f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e06e4e9b52350b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.153+0000 7f6104ba8640 1 -- 172.21.15.60:0/3267239893 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f802ca80 con 0x7f61000ce390 2023-10-16T12:54:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.153+0000 7f6104ba8640 1 -- 172.21.15.60:0/3267239893 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f802cbe0 con 0x7f61000ce390 2023-10-16T12:54:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.153+0000 7f6104ba8640 1 -- 172.21.15.60:0/3267239893 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80343f0 con 0x7f61000ce390 2023-10-16T12:54:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 -- 172.21.15.60:0/3267239893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 msgr2=0x7f6100070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 --2- 172.21.15.60:0/3267239893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100070f50 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f61000c59c0 tx=0x7f60f802bee0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 -- 172.21.15.60:0/3267239893 shutdown_connections 2023-10-16T12:54:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 --2- 172.21.15.60:0/3267239893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100070f50 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 --2- 172.21.15.60:0/3267239893 >> v2:172.21.15.144:3300/0 conn(0x7f610015ae80 0x7f610015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 -- 172.21.15.60:0/3267239893 >> 172.21.15.60:0/3267239893 conn(0x7f61000cb410 msgr2=0x7f61000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:18.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 -- 172.21.15.60:0/3267239893 shutdown_connections 2023-10-16T12:54:18.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.154+0000 7f6107e0e640 1 -- 172.21.15.60:0/3267239893 wait complete. 2023-10-16T12:54:18.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.156+0000 7f6107e0e640 1 Processor -- start 2023-10-16T12:54:18.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.156+0000 7f6107e0e640 1 -- start start 2023-10-16T12:54:18.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6107e0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6107e0e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f610015aaa0 0x7f610015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6107e0e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f610015ae80 0x7f610015b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6107e0e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6100150f30 con 0x7f61000ce390 2023-10-16T12:54:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6107e0e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61000c4e10 con 0x7f610015aaa0 2023-10-16T12:54:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6107e0e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610015bda0 con 0x7f610015ae80 2023-10-16T12:54:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.157+0000 7f6105baa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39750/0 (socket says 172.21.15.60:39750) 2023-10-16T12:54:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 -- 172.21.15.60:0/2134358901 learned_addr learned my addr 172.21.15.60:0/2134358901 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f61053a9640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f610015aaa0 0x7f610015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 -- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f610015ae80 msgr2=0x7f610015b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f610015ae80 0x7f610015b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 -- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f610015aaa0 msgr2=0x7f610015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f610015aaa0 0x7f610015b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 -- 172.21.15.60:0/2134358901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8009b30 con 0x7f61000ce390 2023-10-16T12:54:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f61053a9640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f610015aaa0 0x7f610015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.158+0000 7f6105baa640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100151350 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00c430 tx=0x7f60fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.159+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60fc011790 con 0x7f61000ce390 2023-10-16T12:54:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.159+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60fc0118f0 con 0x7f61000ce390 2023-10-16T12:54:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.159+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60fc011ce0 con 0x7f61000ce390 2023-10-16T12:54:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.159+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610015a1a0 con 0x7f61000ce390 2023-10-16T12:54:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.159+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6100150b50 con 0x7f61000ce390 2023-10-16T12:54:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.160+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610004e160 con 0x7f61000ce390 2023-10-16T12:54:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.161+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f60fc01c460 con 0x7f61000ce390 2023-10-16T12:54:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.162+0000 7f60f67fc640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60e406afc0 0x7f60e406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.162+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60fc089730 con 0x7f61000ce390 2023-10-16T12:54:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.163+0000 7f61053a9640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60e406afc0 0x7f60e406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.164+0000 7f61053a9640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60e406afc0 0x7f60e406d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f60f0006fd0 tx=0x7f60f0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:18.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.166+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60fc0546c0 con 0x7f61000ce390 2023-10-16T12:54:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.357+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 --> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f610015f550 con 0x7f60e406afc0 2023-10-16T12:54:18.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.386+0000 7f60f67fc640 1 -- 172.21.15.60:0/2134358901 <== mgr.7225 v2:172.21.15.144:6832/3094169155 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287279 (secure 0 0 0) 0x7f60f0009530 con 0x7f60e406afc0 2023-10-16T12:54:18.388 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:18.393 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:54:18.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60e406afc0 msgr2=0x7f60e406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60e406afc0 0x7f60e406d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f60f0006fd0 tx=0x7f60f0008040 comp rx=0 tx=0).stop 2023-10-16T12:54:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 msgr2=0x7f6100151350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100151350 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00c430 tx=0x7f60fc00c8f0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 shutdown_connections 2023-10-16T12:54:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f60e406afc0 0x7f60e406d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f610015ae80 0x7f610015b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f610015aaa0 0x7f610015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 --2- 172.21.15.60:0/2134358901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61000ce390 0x7f6100151350 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.393+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 >> 172.21.15.60:0/2134358901 conn(0x7f61000cb410 msgr2=0x7f610006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.394+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 shutdown_connections 2023-10-16T12:54:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.394+0000 7f6107e0e640 1 -- 172.21.15.60:0/2134358901 wait complete. 2023-10-16T12:54:18.417 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:54:17.143675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":447,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":375432,"kb_used_data":16336,"kb_used_omap":12,"kb_used_meta":358963,"kb_avail":754599288,"statfs":{"total":773094113280,"available":772709670912,"internally_reserved":0,"allocated":16728064,"data_stored":8608324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":367578680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.322611"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549447+0000","last_change":"2023-10-16T12:54:13.901564+0000","last_active":"2023-10-16T12:54:15.549447+0000","last_peered":"2023-10-16T12:54:15.549447+0000","last_clean":"2023-10-16T12:54:15.549447+0000","last_became_active":"2023-10-16T12:54:13.901280+0000","last_became_peered":"2023-10-16T12:54:13.901280+0000","last_unstale":"2023-10-16T12:54:15.549447+0000","last_undegraded":"2023-10-16T12:54:15.549447+0000","last_fullsized":"2023-10-16T12:54:15.549447+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:55.120581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874077+0000","last_change":"2023-10-16T12:54:13.883701+0000","last_active":"2023-10-16T12:54:14.874077+0000","last_peered":"2023-10-16T12:54:14.874077+0000","last_clean":"2023-10-16T12:54:14.874077+0000","last_became_active":"2023-10-16T12:54:13.883115+0000","last_became_peered":"2023-10-16T12:54:13.883115+0000","last_unstale":"2023-10-16T12:54:14.874077+0000","last_undegraded":"2023-10-16T12:54:14.874077+0000","last_fullsized":"2023-10-16T12:54:14.874077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:58.472483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663319+0000","last_change":"2023-10-16T12:54:13.900901+0000","last_active":"2023-10-16T12:54:15.663319+0000","last_peered":"2023-10-16T12:54:15.663319+0000","last_clean":"2023-10-16T12:54:15.663319+0000","last_became_active":"2023-10-16T12:54:13.900597+0000","last_became_peered":"2023-10-16T12:54:13.900597+0000","last_unstale":"2023-10-16T12:54:15.663319+0000","last_undegraded":"2023-10-16T12:54:15.663319+0000","last_fullsized":"2023-10-16T12:54:15.663319+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:36.213936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548666+0000","last_change":"2023-10-16T12:54:13.896257+0000","last_active":"2023-10-16T12:54:15.548666+0000","last_peered":"2023-10-16T12:54:15.548666+0000","last_clean":"2023-10-16T12:54:15.548666+0000","last_became_active":"2023-10-16T12:54:13.895980+0000","last_became_peered":"2023-10-16T12:54:13.895980+0000","last_unstale":"2023-10-16T12:54:15.548666+0000","last_undegraded":"2023-10-16T12:54:15.548666+0000","last_fullsized":"2023-10-16T12:54:15.548666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:55.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634082+0000","last_change":"2023-10-16T12:54:13.893083+0000","last_active":"2023-10-16T12:54:15.634082+0000","last_peered":"2023-10-16T12:54:15.634082+0000","last_clean":"2023-10-16T12:54:15.634082+0000","last_became_active":"2023-10-16T12:54:13.891680+0000","last_became_peered":"2023-10-16T12:54:13.891680+0000","last_unstale":"2023-10-16T12:54:15.634082+0000","last_undegraded":"2023-10-16T12:54:15.634082+0000","last_fullsized":"2023-10-16T12:54:15.634082+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:24.019225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875851+0000","last_change":"2023-10-16T12:54:13.898617+0000","last_active":"2023-10-16T12:54:14.875851+0000","last_peered":"2023-10-16T12:54:14.875851+0000","last_clean":"2023-10-16T12:54:14.875851+0000","last_became_active":"2023-10-16T12:54:13.898278+0000","last_became_peered":"2023-10-16T12:54:13.898278+0000","last_unstale":"2023-10-16T12:54:14.875851+0000","last_undegraded":"2023-10-16T12:54:14.875851+0000","last_fullsized":"2023-10-16T12:54:14.875851+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:34.209338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873239+0000","last_change":"2023-10-16T12:54:13.900560+0000","last_active":"2023-10-16T12:54:14.873239+0000","last_peered":"2023-10-16T12:54:14.873239+0000","last_clean":"2023-10-16T12:54:14.873239+0000","last_became_active":"2023-10-16T12:54:13.899414+0000","last_became_peered":"2023-10-16T12:54:13.899414+0000","last_unstale":"2023-10-16T12:54:14.873239+0000","last_undegraded":"2023-10-16T12:54:14.873239+0000","last_fullsized":"2023-10-16T12:54:14.873239+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:04.318061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662893+0000","last_change":"2023-10-16T12:54:13.906051+0000","last_active":"2023-10-16T12:54:15.662893+0000","last_peered":"2023-10-16T12:54:15.662893+0000","last_clean":"2023-10-16T12:54:15.662893+0000","last_became_active":"2023-10-16T12:54:13.905105+0000","last_became_peered":"2023-10-16T12:54:13.905105+0000","last_unstale":"2023-10-16T12:54:15.662893+0000","last_undegraded":"2023-10-16T12:54:15.662893+0000","last_fullsized":"2023-10-16T12:54:15.662893+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.110743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663773+0000","last_change":"2023-10-16T12:54:13.897694+0000","last_active":"2023-10-16T12:54:15.663773+0000","last_peered":"2023-10-16T12:54:15.663773+0000","last_clean":"2023-10-16T12:54:15.663773+0000","last_became_active":"2023-10-16T12:54:13.897352+0000","last_became_peered":"2023-10-16T12:54:13.897352+0000","last_unstale":"2023-10-16T12:54:15.663773+0000","last_undegraded":"2023-10-16T12:54:15.663773+0000","last_fullsized":"2023-10-16T12:54:15.663773+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:07.179112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874280+0000","last_change":"2023-10-16T12:54:13.886309+0000","last_active":"2023-10-16T12:54:14.874280+0000","last_peered":"2023-10-16T12:54:14.874280+0000","last_clean":"2023-10-16T12:54:14.874280+0000","last_became_active":"2023-10-16T12:54:13.885820+0000","last_became_peered":"2023-10-16T12:54:13.885820+0000","last_unstale":"2023-10-16T12:54:14.874280+0000","last_undegraded":"2023-10-16T12:54:14.874280+0000","last_fullsized":"2023-10-16T12:54:14.874280+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:06.720936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662692+0000","last_change":"2023-10-16T12:54:13.882659+0000","last_active":"2023-10-16T12:54:15.662692+0000","last_peered":"2023-10-16T12:54:15.662692+0000","last_clean":"2023-10-16T12:54:15.662692+0000","last_became_active":"2023-10-16T12:54:13.882342+0000","last_became_peered":"2023-10-16T12:54:13.882342+0000","last_unstale":"2023-10-16T12:54:15.662692+0000","last_undegraded":"2023-10-16T12:54:15.662692+0000","last_fullsized":"2023-10-16T12:54:15.662692+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:50.778642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663458+0000","last_change":"2023-10-16T12:54:13.901151+0000","last_active":"2023-10-16T12:54:15.663458+0000","last_peered":"2023-10-16T12:54:15.663458+0000","last_clean":"2023-10-16T12:54:15.663458+0000","last_became_active":"2023-10-16T12:54:13.900852+0000","last_became_peered":"2023-10-16T12:54:13.900852+0000","last_unstale":"2023-10-16T12:54:15.663458+0000","last_undegraded":"2023-10-16T12:54:15.663458+0000","last_fullsized":"2023-10-16T12:54:15.663458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:49.700950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548584+0000","last_change":"2023-10-16T12:54:13.894794+0000","last_active":"2023-10-16T12:54:15.548584+0000","last_peered":"2023-10-16T12:54:15.548584+0000","last_clean":"2023-10-16T12:54:15.548584+0000","last_became_active":"2023-10-16T12:54:13.891670+0000","last_became_peered":"2023-10-16T12:54:13.891670+0000","last_unstale":"2023-10-16T12:54:15.548584+0000","last_undegraded":"2023-10-16T12:54:15.548584+0000","last_fullsized":"2023-10-16T12:54:15.548584+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:21.449193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873036+0000","last_change":"2023-10-16T12:54:13.899263+0000","last_active":"2023-10-16T12:54:14.873036+0000","last_peered":"2023-10-16T12:54:14.873036+0000","last_clean":"2023-10-16T12:54:14.873036+0000","last_became_active":"2023-10-16T12:54:13.898523+0000","last_became_peered":"2023-10-16T12:54:13.898523+0000","last_unstale":"2023-10-16T12:54:14.873036+0000","last_undegraded":"2023-10-16T12:54:14.873036+0000","last_fullsized":"2023-10-16T12:54:14.873036+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:50.778642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548333+0000","last_change":"2023-10-16T12:54:13.901168+0000","last_active":"2023-10-16T12:54:15.548333+0000","last_peered":"2023-10-16T12:54:15.548333+0000","last_clean":"2023-10-16T12:54:15.548333+0000","last_became_active":"2023-10-16T12:54:13.900886+0000","last_became_peered":"2023-10-16T12:54:13.900886+0000","last_unstale":"2023-10-16T12:54:15.548333+0000","last_undegraded":"2023-10-16T12:54:15.548333+0000","last_fullsized":"2023-10-16T12:54:15.548333+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:35.656529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664360+0000","last_change":"2023-10-16T12:54:13.911663+0000","last_active":"2023-10-16T12:54:15.664360+0000","last_peered":"2023-10-16T12:54:15.664360+0000","last_clean":"2023-10-16T12:54:15.664360+0000","last_became_active":"2023-10-16T12:54:13.911378+0000","last_became_peered":"2023-10-16T12:54:13.911378+0000","last_unstale":"2023-10-16T12:54:15.664360+0000","last_undegraded":"2023-10-16T12:54:15.664360+0000","last_fullsized":"2023-10-16T12:54:15.664360+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:50.801289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876321+0000","last_change":"2023-10-16T12:54:13.883297+0000","last_active":"2023-10-16T12:54:14.876321+0000","last_peered":"2023-10-16T12:54:14.876321+0000","last_clean":"2023-10-16T12:54:14.876321+0000","last_became_active":"2023-10-16T12:54:13.882182+0000","last_became_peered":"2023-10-16T12:54:13.882182+0000","last_unstale":"2023-10-16T12:54:14.876321+0000","last_undegraded":"2023-10-16T12:54:14.876321+0000","last_fullsized":"2023-10-16T12:54:14.876321+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:07.904494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665181+0000","last_change":"2023-10-16T12:54:13.902055+0000","last_active":"2023-10-16T12:54:15.665181+0000","last_peered":"2023-10-16T12:54:15.665181+0000","last_clean":"2023-10-16T12:54:15.665181+0000","last_became_active":"2023-10-16T12:54:13.901404+0000","last_became_peered":"2023-10-16T12:54:13.901404+0000","last_unstale":"2023-10-16T12:54:15.665181+0000","last_undegraded":"2023-10-16T12:54:15.665181+0000","last_fullsized":"2023-10-16T12:54:15.665181+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:35.225378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663095+0000","last_change":"2023-10-16T12:54:13.873645+0000","last_active":"2023-10-16T12:54:15.663095+0000","last_peered":"2023-10-16T12:54:15.663095+0000","last_clean":"2023-10-16T12:54:15.663095+0000","last_became_active":"2023-10-16T12:54:13.872856+0000","last_became_peered":"2023-10-16T12:54:13.872856+0000","last_unstale":"2023-10-16T12:54:15.663095+0000","last_undegraded":"2023-10-16T12:54:15.663095+0000","last_fullsized":"2023-10-16T12:54:15.663095+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:12.558438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664821+0000","last_change":"2023-10-16T12:54:13.906174+0000","last_active":"2023-10-16T12:54:15.664821+0000","last_peered":"2023-10-16T12:54:15.664821+0000","last_clean":"2023-10-16T12:54:15.664821+0000","last_became_active":"2023-10-16T12:54:13.905712+0000","last_became_peered":"2023-10-16T12:54:13.905712+0000","last_unstale":"2023-10-16T12:54:15.664821+0000","last_undegraded":"2023-10-16T12:54:15.664821+0000","last_fullsized":"2023-10-16T12:54:15.664821+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:50.302351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634708+0000","last_change":"2023-10-16T12:54:13.891370+0000","last_active":"2023-10-16T12:54:15.634708+0000","last_peered":"2023-10-16T12:54:15.634708+0000","last_clean":"2023-10-16T12:54:15.634708+0000","last_became_active":"2023-10-16T12:54:13.890881+0000","last_became_peered":"2023-10-16T12:54:13.890881+0000","last_unstale":"2023-10-16T12:54:15.634708+0000","last_undegraded":"2023-10-16T12:54:15.634708+0000","last_fullsized":"2023-10-16T12:54:15.634708+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:45.080091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662626+0000","last_change":"2023-10-16T12:54:13.890994+0000","last_active":"2023-10-16T12:54:15.662626+0000","last_peered":"2023-10-16T12:54:15.662626+0000","last_clean":"2023-10-16T12:54:15.662626+0000","last_became_active":"2023-10-16T12:54:13.890705+0000","last_became_peered":"2023-10-16T12:54:13.890705+0000","last_unstale":"2023-10-16T12:54:15.662626+0000","last_undegraded":"2023-10-16T12:54:15.662626+0000","last_fullsized":"2023-10-16T12:54:15.662626+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:29.959281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663093+0000","last_change":"2023-10-16T12:54:13.897894+0000","last_active":"2023-10-16T12:54:15.663093+0000","last_peered":"2023-10-16T12:54:15.663093+0000","last_clean":"2023-10-16T12:54:15.663093+0000","last_became_active":"2023-10-16T12:54:13.897125+0000","last_became_peered":"2023-10-16T12:54:13.897125+0000","last_unstale":"2023-10-16T12:54:15.663093+0000","last_undegraded":"2023-10-16T12:54:15.663093+0000","last_fullsized":"2023-10-16T12:54:15.663093+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:56.549726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662151+0000","last_change":"2023-10-16T12:54:13.879276+0000","last_active":"2023-10-16T12:54:15.662151+0000","last_peered":"2023-10-16T12:54:15.662151+0000","last_clean":"2023-10-16T12:54:15.662151+0000","last_became_active":"2023-10-16T12:54:13.878472+0000","last_became_peered":"2023-10-16T12:54:13.878472+0000","last_unstale":"2023-10-16T12:54:15.662151+0000","last_undegraded":"2023-10-16T12:54:15.662151+0000","last_fullsized":"2023-10-16T12:54:15.662151+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:22.637464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874138+0000","last_change":"2023-10-16T12:54:13.890994+0000","last_active":"2023-10-16T12:54:14.874138+0000","last_peered":"2023-10-16T12:54:14.874138+0000","last_clean":"2023-10-16T12:54:14.874138+0000","last_became_active":"2023-10-16T12:54:13.890723+0000","last_became_peered":"2023-10-16T12:54:13.890723+0000","last_unstale":"2023-10-16T12:54:14.874138+0000","last_undegraded":"2023-10-16T12:54:14.874138+0000","last_fullsized":"2023-10-16T12:54:14.874138+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:25.643113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665248+0000","last_change":"2023-10-16T12:54:13.913358+0000","last_active":"2023-10-16T12:54:15.665248+0000","last_peered":"2023-10-16T12:54:15.665248+0000","last_clean":"2023-10-16T12:54:15.665248+0000","last_became_active":"2023-10-16T12:54:13.913059+0000","last_became_peered":"2023-10-16T12:54:13.913059+0000","last_unstale":"2023-10-16T12:54:15.665248+0000","last_undegraded":"2023-10-16T12:54:15.665248+0000","last_fullsized":"2023-10-16T12:54:15.665248+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:06.637997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548713+0000","last_change":"2023-10-16T12:54:13.901807+0000","last_active":"2023-10-16T12:54:15.548713+0000","last_peered":"2023-10-16T12:54:15.548713+0000","last_clean":"2023-10-16T12:54:15.548713+0000","last_became_active":"2023-10-16T12:54:13.901457+0000","last_became_peered":"2023-10-16T12:54:13.901457+0000","last_unstale":"2023-10-16T12:54:15.548713+0000","last_undegraded":"2023-10-16T12:54:15.548713+0000","last_fullsized":"2023-10-16T12:54:15.548713+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:16.404805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874766+0000","last_change":"2023-10-16T12:54:13.900895+0000","last_active":"2023-10-16T12:54:14.874766+0000","last_peered":"2023-10-16T12:54:14.874766+0000","last_clean":"2023-10-16T12:54:14.874766+0000","last_became_active":"2023-10-16T12:54:13.899915+0000","last_became_peered":"2023-10-16T12:54:13.899915+0000","last_unstale":"2023-10-16T12:54:14.874766+0000","last_undegraded":"2023-10-16T12:54:14.874766+0000","last_fullsized":"2023-10-16T12:54:14.874766+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:44.510594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548580+0000","last_change":"2023-10-16T12:54:13.903111+0000","last_active":"2023-10-16T12:54:15.548580+0000","last_peered":"2023-10-16T12:54:15.548580+0000","last_clean":"2023-10-16T12:54:15.548580+0000","last_became_active":"2023-10-16T12:54:13.902830+0000","last_became_peered":"2023-10-16T12:54:13.902830+0000","last_unstale":"2023-10-16T12:54:15.548580+0000","last_undegraded":"2023-10-16T12:54:15.548580+0000","last_fullsized":"2023-10-16T12:54:15.548580+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:28.610173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548488+0000","last_change":"2023-10-16T12:54:13.901689+0000","last_active":"2023-10-16T12:54:15.548488+0000","last_peered":"2023-10-16T12:54:15.548488+0000","last_clean":"2023-10-16T12:54:15.548488+0000","last_became_active":"2023-10-16T12:54:13.901104+0000","last_became_peered":"2023-10-16T12:54:13.901104+0000","last_unstale":"2023-10-16T12:54:15.548488+0000","last_undegraded":"2023-10-16T12:54:15.548488+0000","last_fullsized":"2023-10-16T12:54:15.548488+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:29.334888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633980+0000","last_change":"2023-10-16T12:54:13.898231+0000","last_active":"2023-10-16T12:54:15.633980+0000","last_peered":"2023-10-16T12:54:15.633980+0000","last_clean":"2023-10-16T12:54:15.633980+0000","last_became_active":"2023-10-16T12:54:13.897823+0000","last_became_peered":"2023-10-16T12:54:13.897823+0000","last_unstale":"2023-10-16T12:54:15.633980+0000","last_undegraded":"2023-10-16T12:54:15.633980+0000","last_fullsized":"2023-10-16T12:54:15.633980+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:02.469574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873155+0000","last_change":"2023-10-16T12:54:13.909632+0000","last_active":"2023-10-16T12:54:14.873155+0000","last_peered":"2023-10-16T12:54:14.873155+0000","last_clean":"2023-10-16T12:54:14.873155+0000","last_became_active":"2023-10-16T12:54:13.909332+0000","last_became_peered":"2023-10-16T12:54:13.909332+0000","last_unstale":"2023-10-16T12:54:14.873155+0000","last_undegraded":"2023-10-16T12:54:14.873155+0000","last_fullsized":"2023-10-16T12:54:14.873155+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:02.670596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665519+0000","last_change":"2023-10-16T12:54:13.901868+0000","last_active":"2023-10-16T12:54:15.665519+0000","last_peered":"2023-10-16T12:54:15.665519+0000","last_clean":"2023-10-16T12:54:15.665519+0000","last_became_active":"2023-10-16T12:54:13.901005+0000","last_became_peered":"2023-10-16T12:54:13.901005+0000","last_unstale":"2023-10-16T12:54:15.665519+0000","last_undegraded":"2023-10-16T12:54:15.665519+0000","last_fullsized":"2023-10-16T12:54:15.665519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:43.000800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"155'4","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634597+0000","last_change":"2023-10-16T12:54:12.856409+0000","last_active":"2023-10-16T12:54:15.634597+0000","last_peered":"2023-10-16T12:54:15.634597+0000","last_clean":"2023-10-16T12:54:15.634597+0000","last_became_active":"2023-10-16T12:54:12.856105+0000","last_became_peered":"2023-10-16T12:54:12.856105+0000","last_unstale":"2023-10-16T12:54:15.634597+0000","last_undegraded":"2023-10-16T12:54:15.634597+0000","last_fullsized":"2023-10-16T12:54:15.634597+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:56.068424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663779+0000","last_change":"2023-10-16T12:54:13.894968+0000","last_active":"2023-10-16T12:54:15.663779+0000","last_peered":"2023-10-16T12:54:15.663779+0000","last_clean":"2023-10-16T12:54:15.663779+0000","last_became_active":"2023-10-16T12:54:13.894700+0000","last_became_peered":"2023-10-16T12:54:13.894700+0000","last_unstale":"2023-10-16T12:54:15.663779+0000","last_undegraded":"2023-10-16T12:54:15.663779+0000","last_fullsized":"2023-10-16T12:54:15.663779+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:20.017229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875435+0000","last_change":"2023-10-16T12:54:12.869100+0000","last_active":"2023-10-16T12:54:14.875435+0000","last_peered":"2023-10-16T12:54:14.875435+0000","last_clean":"2023-10-16T12:54:14.875435+0000","last_became_active":"2023-10-16T12:54:12.868772+0000","last_became_peered":"2023-10-16T12:54:12.868772+0000","last_unstale":"2023-10-16T12:54:14.875435+0000","last_undegraded":"2023-10-16T12:54:14.875435+0000","last_fullsized":"2023-10-16T12:54:14.875435+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:06.889175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634661+0000","last_change":"2023-10-16T12:54:13.897603+0000","last_active":"2023-10-16T12:54:15.634661+0000","last_peered":"2023-10-16T12:54:15.634661+0000","last_clean":"2023-10-16T12:54:15.634661+0000","last_became_active":"2023-10-16T12:54:13.897169+0000","last_became_peered":"2023-10-16T12:54:13.897169+0000","last_unstale":"2023-10-16T12:54:15.634661+0000","last_undegraded":"2023-10-16T12:54:15.634661+0000","last_fullsized":"2023-10-16T12:54:15.634661+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:44.549528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663046+0000","last_change":"2023-10-16T12:54:12.856792+0000","last_active":"2023-10-16T12:54:15.663046+0000","last_peered":"2023-10-16T12:54:15.663046+0000","last_clean":"2023-10-16T12:54:15.663046+0000","last_became_active":"2023-10-16T12:54:12.856513+0000","last_became_peered":"2023-10-16T12:54:12.856513+0000","last_unstale":"2023-10-16T12:54:15.663046+0000","last_undegraded":"2023-10-16T12:54:15.663046+0000","last_fullsized":"2023-10-16T12:54:15.663046+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:54.105261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634123+0000","last_change":"2023-10-16T12:54:13.898862+0000","last_active":"2023-10-16T12:54:15.634123+0000","last_peered":"2023-10-16T12:54:15.634123+0000","last_clean":"2023-10-16T12:54:15.634123+0000","last_became_active":"2023-10-16T12:54:13.898488+0000","last_became_peered":"2023-10-16T12:54:13.898488+0000","last_unstale":"2023-10-16T12:54:15.634123+0000","last_undegraded":"2023-10-16T12:54:15.634123+0000","last_fullsized":"2023-10-16T12:54:15.634123+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:04.718383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873742+0000","last_change":"2023-10-16T12:54:12.860475+0000","last_active":"2023-10-16T12:54:14.873742+0000","last_peered":"2023-10-16T12:54:14.873742+0000","last_clean":"2023-10-16T12:54:14.873742+0000","last_became_active":"2023-10-16T12:54:12.860067+0000","last_became_peered":"2023-10-16T12:54:12.860067+0000","last_unstale":"2023-10-16T12:54:14.873742+0000","last_undegraded":"2023-10-16T12:54:14.873742+0000","last_fullsized":"2023-10-16T12:54:14.873742+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:15.389485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664693+0000","last_change":"2023-10-16T12:54:13.902386+0000","last_active":"2023-10-16T12:54:15.664693+0000","last_peered":"2023-10-16T12:54:15.664693+0000","last_clean":"2023-10-16T12:54:15.664693+0000","last_became_active":"2023-10-16T12:54:13.900860+0000","last_became_peered":"2023-10-16T12:54:13.900860+0000","last_unstale":"2023-10-16T12:54:15.664693+0000","last_undegraded":"2023-10-16T12:54:15.664693+0000","last_fullsized":"2023-10-16T12:54:15.664693+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:00.945054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873279+0000","last_change":"2023-10-16T12:54:12.863695+0000","last_active":"2023-10-16T12:54:14.873279+0000","last_peered":"2023-10-16T12:54:14.873279+0000","last_clean":"2023-10-16T12:54:14.873279+0000","last_became_active":"2023-10-16T12:54:12.863438+0000","last_became_peered":"2023-10-16T12:54:12.863438+0000","last_unstale":"2023-10-16T12:54:14.873279+0000","last_undegraded":"2023-10-16T12:54:14.873279+0000","last_fullsized":"2023-10-16T12:54:14.873279+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:28.319569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876394+0000","last_change":"2023-10-16T12:54:13.878036+0000","last_active":"2023-10-16T12:54:14.876394+0000","last_peered":"2023-10-16T12:54:14.876394+0000","last_clean":"2023-10-16T12:54:14.876394+0000","last_became_active":"2023-10-16T12:54:13.877114+0000","last_became_peered":"2023-10-16T12:54:13.877114+0000","last_unstale":"2023-10-16T12:54:14.876394+0000","last_undegraded":"2023-10-16T12:54:14.876394+0000","last_fullsized":"2023-10-16T12:54:14.876394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:51.991631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664039+0000","last_change":"2023-10-16T12:54:12.849710+0000","last_active":"2023-10-16T12:54:15.664039+0000","last_peered":"2023-10-16T12:54:15.664039+0000","last_clean":"2023-10-16T12:54:15.664039+0000","last_became_active":"2023-10-16T12:54:12.846509+0000","last_became_peered":"2023-10-16T12:54:12.846509+0000","last_unstale":"2023-10-16T12:54:15.664039+0000","last_undegraded":"2023-10-16T12:54:15.664039+0000","last_fullsized":"2023-10-16T12:54:15.664039+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:59.579472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663247+0000","last_change":"2023-10-16T12:54:13.902131+0000","last_active":"2023-10-16T12:54:15.663247+0000","last_peered":"2023-10-16T12:54:15.663247+0000","last_clean":"2023-10-16T12:54:15.663247+0000","last_became_active":"2023-10-16T12:54:13.901829+0000","last_became_peered":"2023-10-16T12:54:13.901829+0000","last_unstale":"2023-10-16T12:54:15.663247+0000","last_undegraded":"2023-10-16T12:54:15.663247+0000","last_fullsized":"2023-10-16T12:54:15.663247+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:05.688392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873403+0000","last_change":"2023-10-16T12:54:12.849187+0000","last_active":"2023-10-16T12:54:14.873403+0000","last_peered":"2023-10-16T12:54:14.873403+0000","last_clean":"2023-10-16T12:54:14.873403+0000","last_became_active":"2023-10-16T12:54:12.848583+0000","last_became_peered":"2023-10-16T12:54:12.848583+0000","last_unstale":"2023-10-16T12:54:14.873403+0000","last_undegraded":"2023-10-16T12:54:14.873403+0000","last_fullsized":"2023-10-16T12:54:14.873403+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:05.000460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872669+0000","last_change":"2023-10-16T12:54:13.899751+0000","last_active":"2023-10-16T12:54:14.872669+0000","last_peered":"2023-10-16T12:54:14.872669+0000","last_clean":"2023-10-16T12:54:14.872669+0000","last_became_active":"2023-10-16T12:54:13.898612+0000","last_became_peered":"2023-10-16T12:54:13.898612+0000","last_unstale":"2023-10-16T12:54:14.872669+0000","last_undegraded":"2023-10-16T12:54:14.872669+0000","last_fullsized":"2023-10-16T12:54:14.872669+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:29.959281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872667+0000","last_change":"2023-10-16T12:54:12.867244+0000","last_active":"2023-10-16T12:54:14.872667+0000","last_peered":"2023-10-16T12:54:14.872667+0000","last_clean":"2023-10-16T12:54:14.872667+0000","last_became_active":"2023-10-16T12:54:12.866979+0000","last_became_peered":"2023-10-16T12:54:12.866979+0000","last_unstale":"2023-10-16T12:54:14.872667+0000","last_undegraded":"2023-10-16T12:54:14.872667+0000","last_fullsized":"2023-10-16T12:54:14.872667+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:54.808266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549358+0000","last_change":"2023-10-16T12:54:13.897060+0000","last_active":"2023-10-16T12:54:15.549358+0000","last_peered":"2023-10-16T12:54:15.549358+0000","last_clean":"2023-10-16T12:54:15.549358+0000","last_became_active":"2023-10-16T12:54:13.896796+0000","last_became_peered":"2023-10-16T12:54:13.896796+0000","last_unstale":"2023-10-16T12:54:15.549358+0000","last_undegraded":"2023-10-16T12:54:15.549358+0000","last_fullsized":"2023-10-16T12:54:15.549358+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:07.612504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875889+0000","last_change":"2023-10-16T12:54:12.866455+0000","last_active":"2023-10-16T12:54:14.875889+0000","last_peered":"2023-10-16T12:54:14.875889+0000","last_clean":"2023-10-16T12:54:14.875889+0000","last_became_active":"2023-10-16T12:54:12.866129+0000","last_became_peered":"2023-10-16T12:54:12.866129+0000","last_unstale":"2023-10-16T12:54:14.875889+0000","last_undegraded":"2023-10-16T12:54:14.875889+0000","last_fullsized":"2023-10-16T12:54:14.875889+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:50.976311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875187+0000","last_change":"2023-10-16T12:54:13.902533+0000","last_active":"2023-10-16T12:54:14.875187+0000","last_peered":"2023-10-16T12:54:14.875187+0000","last_clean":"2023-10-16T12:54:14.875187+0000","last_became_active":"2023-10-16T12:54:13.902261+0000","last_became_peered":"2023-10-16T12:54:13.902261+0000","last_unstale":"2023-10-16T12:54:14.875187+0000","last_undegraded":"2023-10-16T12:54:14.875187+0000","last_fullsized":"2023-10-16T12:54:14.875187+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:07.347612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665431+0000","last_change":"2023-10-16T12:54:12.863681+0000","last_active":"2023-10-16T12:54:15.665431+0000","last_peered":"2023-10-16T12:54:15.665431+0000","last_clean":"2023-10-16T12:54:15.665431+0000","last_became_active":"2023-10-16T12:54:12.863383+0000","last_became_peered":"2023-10-16T12:54:12.863383+0000","last_unstale":"2023-10-16T12:54:15.665431+0000","last_undegraded":"2023-10-16T12:54:15.665431+0000","last_fullsized":"2023-10-16T12:54:15.665431+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:43.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634785+0000","last_change":"2023-10-16T12:54:13.903493+0000","last_active":"2023-10-16T12:54:15.634785+0000","last_peered":"2023-10-16T12:54:15.634785+0000","last_clean":"2023-10-16T12:54:15.634785+0000","last_became_active":"2023-10-16T12:54:13.901959+0000","last_became_peered":"2023-10-16T12:54:13.901959+0000","last_unstale":"2023-10-16T12:54:15.634785+0000","last_undegraded":"2023-10-16T12:54:15.634785+0000","last_fullsized":"2023-10-16T12:54:15.634785+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:39.219237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"155'3","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874812+0000","last_change":"2023-10-16T12:54:12.860597+0000","last_active":"2023-10-16T12:54:14.874812+0000","last_peered":"2023-10-16T12:54:14.874812+0000","last_clean":"2023-10-16T12:54:14.874812+0000","last_became_active":"2023-10-16T12:54:12.860293+0000","last_became_peered":"2023-10-16T12:54:12.860293+0000","last_unstale":"2023-10-16T12:54:14.874812+0000","last_undegraded":"2023-10-16T12:54:14.874812+0000","last_fullsized":"2023-10-16T12:54:14.874812+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:17.402238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634211+0000","last_change":"2023-10-16T12:54:13.903346+0000","last_active":"2023-10-16T12:54:15.634211+0000","last_peered":"2023-10-16T12:54:15.634211+0000","last_clean":"2023-10-16T12:54:15.634211+0000","last_became_active":"2023-10-16T12:54:13.903076+0000","last_became_peered":"2023-10-16T12:54:13.903076+0000","last_unstale":"2023-10-16T12:54:15.634211+0000","last_undegraded":"2023-10-16T12:54:15.634211+0000","last_fullsized":"2023-10-16T12:54:15.634211+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:46.018150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"155'6","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663660+0000","last_change":"2023-10-16T12:54:12.861976+0000","last_active":"2023-10-16T12:54:15.663660+0000","last_peered":"2023-10-16T12:54:15.663660+0000","last_clean":"2023-10-16T12:54:15.663660+0000","last_became_active":"2023-10-16T12:54:12.861378+0000","last_became_peered":"2023-10-16T12:54:12.861378+0000","last_unstale":"2023-10-16T12:54:15.663660+0000","last_undegraded":"2023-10-16T12:54:15.663660+0000","last_fullsized":"2023-10-16T12:54:15.663660+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:04.148703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548943+0000","last_change":"2023-10-16T12:54:13.895425+0000","last_active":"2023-10-16T12:54:15.548943+0000","last_peered":"2023-10-16T12:54:15.548943+0000","last_clean":"2023-10-16T12:54:15.548943+0000","last_became_active":"2023-10-16T12:54:13.891264+0000","last_became_peered":"2023-10-16T12:54:13.891264+0000","last_unstale":"2023-10-16T12:54:15.548943+0000","last_undegraded":"2023-10-16T12:54:15.548943+0000","last_fullsized":"2023-10-16T12:54:15.548943+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:30.600664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548857+0000","last_change":"2023-10-16T12:54:12.847293+0000","last_active":"2023-10-16T12:54:15.548857+0000","last_peered":"2023-10-16T12:54:15.548857+0000","last_clean":"2023-10-16T12:54:15.548857+0000","last_became_active":"2023-10-16T12:54:12.846550+0000","last_became_peered":"2023-10-16T12:54:12.846550+0000","last_unstale":"2023-10-16T12:54:15.548857+0000","last_undegraded":"2023-10-16T12:54:15.548857+0000","last_fullsized":"2023-10-16T12:54:15.548857+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:20.224512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872639+0000","last_change":"2023-10-16T12:54:13.897999+0000","last_active":"2023-10-16T12:54:14.872639+0000","last_peered":"2023-10-16T12:54:14.872639+0000","last_clean":"2023-10-16T12:54:14.872639+0000","last_became_active":"2023-10-16T12:54:13.896966+0000","last_became_peered":"2023-10-16T12:54:13.896966+0000","last_unstale":"2023-10-16T12:54:14.872639+0000","last_undegraded":"2023-10-16T12:54:14.872639+0000","last_fullsized":"2023-10-16T12:54:14.872639+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:05.337355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872786+0000","last_change":"2023-10-16T12:54:12.862503+0000","last_active":"2023-10-16T12:54:14.872786+0000","last_peered":"2023-10-16T12:54:14.872786+0000","last_clean":"2023-10-16T12:54:14.872786+0000","last_became_active":"2023-10-16T12:54:12.862227+0000","last_became_peered":"2023-10-16T12:54:12.862227+0000","last_unstale":"2023-10-16T12:54:14.872786+0000","last_undegraded":"2023-10-16T12:54:14.872786+0000","last_fullsized":"2023-10-16T12:54:14.872786+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:20.433874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663063+0000","last_change":"2023-10-16T12:54:13.901125+0000","last_active":"2023-10-16T12:54:15.663063+0000","last_peered":"2023-10-16T12:54:15.663063+0000","last_clean":"2023-10-16T12:54:15.663063+0000","last_became_active":"2023-10-16T12:54:13.900799+0000","last_became_peered":"2023-10-16T12:54:13.900799+0000","last_unstale":"2023-10-16T12:54:15.663063+0000","last_undegraded":"2023-10-16T12:54:15.663063+0000","last_fullsized":"2023-10-16T12:54:15.663063+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:37.513352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"155'8","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663299+0000","last_change":"2023-10-16T12:54:12.862441+0000","last_active":"2023-10-16T12:54:15.663299+0000","last_peered":"2023-10-16T12:54:15.663299+0000","last_clean":"2023-10-16T12:54:15.663299+0000","last_became_active":"2023-10-16T12:54:12.862131+0000","last_became_peered":"2023-10-16T12:54:12.862131+0000","last_unstale":"2023-10-16T12:54:15.663299+0000","last_undegraded":"2023-10-16T12:54:15.663299+0000","last_fullsized":"2023-10-16T12:54:15.663299+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:16.982789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663401+0000","last_change":"2023-10-16T12:54:13.907633+0000","last_active":"2023-10-16T12:54:15.663401+0000","last_peered":"2023-10-16T12:54:15.663401+0000","last_clean":"2023-10-16T12:54:15.663401+0000","last_became_active":"2023-10-16T12:54:13.907320+0000","last_became_peered":"2023-10-16T12:54:13.907320+0000","last_unstale":"2023-10-16T12:54:15.663401+0000","last_undegraded":"2023-10-16T12:54:15.663401+0000","last_fullsized":"2023-10-16T12:54:15.663401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:28.250703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876096+0000","last_change":"2023-10-16T12:54:12.862311+0000","last_active":"2023-10-16T12:54:14.876096+0000","last_peered":"2023-10-16T12:54:14.876096+0000","last_clean":"2023-10-16T12:54:14.876096+0000","last_became_active":"2023-10-16T12:54:12.861788+0000","last_became_peered":"2023-10-16T12:54:12.861788+0000","last_unstale":"2023-10-16T12:54:14.876096+0000","last_undegraded":"2023-10-16T12:54:14.876096+0000","last_fullsized":"2023-10-16T12:54:14.876096+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:39.454603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875456+0000","last_change":"2023-10-16T12:54:13.881336+0000","last_active":"2023-10-16T12:54:14.875456+0000","last_peered":"2023-10-16T12:54:14.875456+0000","last_clean":"2023-10-16T12:54:14.875456+0000","last_became_active":"2023-10-16T12:54:13.880741+0000","last_became_peered":"2023-10-16T12:54:13.880741+0000","last_unstale":"2023-10-16T12:54:14.875456+0000","last_undegraded":"2023-10-16T12:54:14.875456+0000","last_fullsized":"2023-10-16T12:54:14.875456+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:22.470884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663013+0000","last_change":"2023-10-16T12:54:12.860998+0000","last_active":"2023-10-16T12:54:15.663013+0000","last_peered":"2023-10-16T12:54:15.663013+0000","last_clean":"2023-10-16T12:54:15.663013+0000","last_became_active":"2023-10-16T12:54:12.859512+0000","last_became_peered":"2023-10-16T12:54:12.859512+0000","last_unstale":"2023-10-16T12:54:15.663013+0000","last_undegraded":"2023-10-16T12:54:15.663013+0000","last_fullsized":"2023-10-16T12:54:15.663013+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:06.597185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665533+0000","last_change":"2023-10-16T12:54:13.910041+0000","last_active":"2023-10-16T12:54:15.665533+0000","last_peered":"2023-10-16T12:54:15.665533+0000","last_clean":"2023-10-16T12:54:15.665533+0000","last_became_active":"2023-10-16T12:54:13.909760+0000","last_became_peered":"2023-10-16T12:54:13.909760+0000","last_unstale":"2023-10-16T12:54:15.665533+0000","last_undegraded":"2023-10-16T12:54:15.665533+0000","last_fullsized":"2023-10-16T12:54:15.665533+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:57.083744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876447+0000","last_change":"2023-10-16T12:54:12.844521+0000","last_active":"2023-10-16T12:54:14.876447+0000","last_peered":"2023-10-16T12:54:14.876447+0000","last_clean":"2023-10-16T12:54:14.876447+0000","last_became_active":"2023-10-16T12:54:12.841965+0000","last_became_peered":"2023-10-16T12:54:12.841965+0000","last_unstale":"2023-10-16T12:54:14.876447+0000","last_undegraded":"2023-10-16T12:54:14.876447+0000","last_fullsized":"2023-10-16T12:54:14.876447+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:54.671940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874063+0000","last_change":"2023-10-16T12:54:13.900467+0000","last_active":"2023-10-16T12:54:14.874063+0000","last_peered":"2023-10-16T12:54:14.874063+0000","last_clean":"2023-10-16T12:54:14.874063+0000","last_became_active":"2023-10-16T12:54:13.898704+0000","last_became_peered":"2023-10-16T12:54:13.898704+0000","last_unstale":"2023-10-16T12:54:14.874063+0000","last_undegraded":"2023-10-16T12:54:14.874063+0000","last_fullsized":"2023-10-16T12:54:14.874063+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:53.813728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874708+0000","last_change":"2023-10-16T12:54:12.860177+0000","last_active":"2023-10-16T12:54:14.874708+0000","last_peered":"2023-10-16T12:54:14.874708+0000","last_clean":"2023-10-16T12:54:14.874708+0000","last_became_active":"2023-10-16T12:54:12.859521+0000","last_became_peered":"2023-10-16T12:54:12.859521+0000","last_unstale":"2023-10-16T12:54:14.874708+0000","last_undegraded":"2023-10-16T12:54:14.874708+0000","last_fullsized":"2023-10-16T12:54:14.874708+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:44.660490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548499+0000","last_change":"2023-10-16T12:54:13.896656+0000","last_active":"2023-10-16T12:54:15.548499+0000","last_peered":"2023-10-16T12:54:15.548499+0000","last_clean":"2023-10-16T12:54:15.548499+0000","last_became_active":"2023-10-16T12:54:13.896393+0000","last_became_peered":"2023-10-16T12:54:13.896393+0000","last_unstale":"2023-10-16T12:54:15.548499+0000","last_undegraded":"2023-10-16T12:54:15.548499+0000","last_fullsized":"2023-10-16T12:54:15.548499+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:09.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875173+0000","last_change":"2023-10-16T12:54:12.866461+0000","last_active":"2023-10-16T12:54:14.875173+0000","last_peered":"2023-10-16T12:54:14.875173+0000","last_clean":"2023-10-16T12:54:14.875173+0000","last_became_active":"2023-10-16T12:54:12.866126+0000","last_became_peered":"2023-10-16T12:54:12.866126+0000","last_unstale":"2023-10-16T12:54:14.875173+0000","last_undegraded":"2023-10-16T12:54:14.875173+0000","last_fullsized":"2023-10-16T12:54:14.875173+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:01.655277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876625+0000","last_change":"2023-10-16T12:54:13.893747+0000","last_active":"2023-10-16T12:54:14.876625+0000","last_peered":"2023-10-16T12:54:14.876625+0000","last_clean":"2023-10-16T12:54:14.876625+0000","last_became_active":"2023-10-16T12:54:13.893447+0000","last_became_peered":"2023-10-16T12:54:13.893447+0000","last_unstale":"2023-10-16T12:54:14.876625+0000","last_undegraded":"2023-10-16T12:54:14.876625+0000","last_fullsized":"2023-10-16T12:54:14.876625+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:40.469922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664337+0000","last_change":"2023-10-16T12:54:12.858930+0000","last_active":"2023-10-16T12:54:15.664337+0000","last_peered":"2023-10-16T12:54:15.664337+0000","last_clean":"2023-10-16T12:54:15.664337+0000","last_became_active":"2023-10-16T12:54:12.858384+0000","last_became_peered":"2023-10-16T12:54:12.858384+0000","last_unstale":"2023-10-16T12:54:15.664337+0000","last_undegraded":"2023-10-16T12:54:15.664337+0000","last_fullsized":"2023-10-16T12:54:15.664337+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:57.116437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872513+0000","last_change":"2023-10-16T12:54:13.889068+0000","last_active":"2023-10-16T12:54:14.872513+0000","last_peered":"2023-10-16T12:54:14.872513+0000","last_clean":"2023-10-16T12:54:14.872513+0000","last_became_active":"2023-10-16T12:54:13.884669+0000","last_became_peered":"2023-10-16T12:54:13.884669+0000","last_unstale":"2023-10-16T12:54:14.872513+0000","last_undegraded":"2023-10-16T12:54:14.872513+0000","last_fullsized":"2023-10-16T12:54:14.872513+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:55.120581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664039+0000","last_change":"2023-10-16T12:54:12.860130+0000","last_active":"2023-10-16T12:54:15.664039+0000","last_peered":"2023-10-16T12:54:15.664039+0000","last_clean":"2023-10-16T12:54:15.664039+0000","last_became_active":"2023-10-16T12:54:12.859824+0000","last_became_peered":"2023-10-16T12:54:12.859824+0000","last_unstale":"2023-10-16T12:54:15.664039+0000","last_undegraded":"2023-10-16T12:54:15.664039+0000","last_fullsized":"2023-10-16T12:54:15.664039+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:09.544063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633666+0000","last_change":"2023-10-16T12:54:13.898363+0000","last_active":"2023-10-16T12:54:15.633666+0000","last_peered":"2023-10-16T12:54:15.633666+0000","last_clean":"2023-10-16T12:54:15.633666+0000","last_became_active":"2023-10-16T12:54:13.897793+0000","last_became_peered":"2023-10-16T12:54:13.897793+0000","last_unstale":"2023-10-16T12:54:15.633666+0000","last_undegraded":"2023-10-16T12:54:15.633666+0000","last_fullsized":"2023-10-16T12:54:15.633666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:50.613200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662106+0000","last_change":"2023-10-16T12:54:12.861854+0000","last_active":"2023-10-16T12:54:15.662106+0000","last_peered":"2023-10-16T12:54:15.662106+0000","last_clean":"2023-10-16T12:54:15.662106+0000","last_became_active":"2023-10-16T12:54:12.859335+0000","last_became_peered":"2023-10-16T12:54:12.859335+0000","last_unstale":"2023-10-16T12:54:15.662106+0000","last_undegraded":"2023-10-16T12:54:15.662106+0000","last_fullsized":"2023-10-16T12:54:15.662106+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:08.457869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876752+0000","last_change":"2023-10-16T12:54:13.881587+0000","last_active":"2023-10-16T12:54:14.876752+0000","last_peered":"2023-10-16T12:54:14.876752+0000","last_clean":"2023-10-16T12:54:14.876752+0000","last_became_active":"2023-10-16T12:54:13.879074+0000","last_became_peered":"2023-10-16T12:54:13.879074+0000","last_unstale":"2023-10-16T12:54:14.876752+0000","last_undegraded":"2023-10-16T12:54:14.876752+0000","last_fullsized":"2023-10-16T12:54:14.876752+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:50.206683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549669+0000","last_change":"2023-10-16T12:54:12.861573+0000","last_active":"2023-10-16T12:54:15.549669+0000","last_peered":"2023-10-16T12:54:15.549669+0000","last_clean":"2023-10-16T12:54:15.549669+0000","last_became_active":"2023-10-16T12:54:12.861304+0000","last_became_peered":"2023-10-16T12:54:12.861304+0000","last_unstale":"2023-10-16T12:54:15.549669+0000","last_undegraded":"2023-10-16T12:54:15.549669+0000","last_fullsized":"2023-10-16T12:54:15.549669+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:09.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549628+0000","last_change":"2023-10-16T12:54:13.906250+0000","last_active":"2023-10-16T12:54:15.549628+0000","last_peered":"2023-10-16T12:54:15.549628+0000","last_clean":"2023-10-16T12:54:15.549628+0000","last_became_active":"2023-10-16T12:54:13.905848+0000","last_became_peered":"2023-10-16T12:54:13.905848+0000","last_unstale":"2023-10-16T12:54:15.549628+0000","last_undegraded":"2023-10-16T12:54:15.549628+0000","last_fullsized":"2023-10-16T12:54:15.549628+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:05.337355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633988+0000","last_change":"2023-10-16T12:54:12.855590+0000","last_active":"2023-10-16T12:54:15.633988+0000","last_peered":"2023-10-16T12:54:15.633988+0000","last_clean":"2023-10-16T12:54:15.633988+0000","last_became_active":"2023-10-16T12:54:12.855237+0000","last_became_peered":"2023-10-16T12:54:12.855237+0000","last_unstale":"2023-10-16T12:54:15.633988+0000","last_undegraded":"2023-10-16T12:54:15.633988+0000","last_fullsized":"2023-10-16T12:54:15.633988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:34.210059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873643+0000","last_change":"2023-10-16T12:54:13.893769+0000","last_active":"2023-10-16T12:54:14.873643+0000","last_peered":"2023-10-16T12:54:14.873643+0000","last_clean":"2023-10-16T12:54:14.873643+0000","last_became_active":"2023-10-16T12:54:13.893424+0000","last_became_peered":"2023-10-16T12:54:13.893424+0000","last_unstale":"2023-10-16T12:54:14.873643+0000","last_undegraded":"2023-10-16T12:54:14.873643+0000","last_fullsized":"2023-10-16T12:54:14.873643+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:12.558438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874056+0000","last_change":"2023-10-16T12:54:12.866418+0000","last_active":"2023-10-16T12:54:14.874056+0000","last_peered":"2023-10-16T12:54:14.874056+0000","last_clean":"2023-10-16T12:54:14.874056+0000","last_became_active":"2023-10-16T12:54:12.865909+0000","last_became_peered":"2023-10-16T12:54:12.865909+0000","last_unstale":"2023-10-16T12:54:14.874056+0000","last_undegraded":"2023-10-16T12:54:14.874056+0000","last_fullsized":"2023-10-16T12:54:14.874056+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:09.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876298+0000","last_change":"2023-10-16T12:54:13.894982+0000","last_active":"2023-10-16T12:54:14.876298+0000","last_peered":"2023-10-16T12:54:14.876298+0000","last_clean":"2023-10-16T12:54:14.876298+0000","last_became_active":"2023-10-16T12:54:13.894710+0000","last_became_peered":"2023-10-16T12:54:13.894710+0000","last_unstale":"2023-10-16T12:54:14.876298+0000","last_undegraded":"2023-10-16T12:54:14.876298+0000","last_fullsized":"2023-10-16T12:54:14.876298+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:45.675810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662634+0000","last_change":"2023-10-16T12:54:12.856151+0000","last_active":"2023-10-16T12:54:15.662634+0000","last_peered":"2023-10-16T12:54:15.662634+0000","last_clean":"2023-10-16T12:54:15.662634+0000","last_became_active":"2023-10-16T12:54:12.855854+0000","last_became_peered":"2023-10-16T12:54:12.855854+0000","last_unstale":"2023-10-16T12:54:15.662634+0000","last_undegraded":"2023-10-16T12:54:15.662634+0000","last_fullsized":"2023-10-16T12:54:15.662634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:29.585345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633744+0000","last_change":"2023-10-16T12:54:13.894010+0000","last_active":"2023-10-16T12:54:15.633744+0000","last_peered":"2023-10-16T12:54:15.633744+0000","last_clean":"2023-10-16T12:54:15.633744+0000","last_became_active":"2023-10-16T12:54:13.892969+0000","last_became_peered":"2023-10-16T12:54:13.892969+0000","last_unstale":"2023-10-16T12:54:15.633744+0000","last_undegraded":"2023-10-16T12:54:15.633744+0000","last_fullsized":"2023-10-16T12:54:15.633744+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:18.885935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873579+0000","last_change":"2023-10-16T12:54:12.862944+0000","last_active":"2023-10-16T12:54:14.873579+0000","last_peered":"2023-10-16T12:54:14.873579+0000","last_clean":"2023-10-16T12:54:14.873579+0000","last_became_active":"2023-10-16T12:54:12.862588+0000","last_became_peered":"2023-10-16T12:54:12.862588+0000","last_unstale":"2023-10-16T12:54:14.873579+0000","last_undegraded":"2023-10-16T12:54:14.873579+0000","last_fullsized":"2023-10-16T12:54:14.873579+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:29.585345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874941+0000","last_change":"2023-10-16T12:54:13.895152+0000","last_active":"2023-10-16T12:54:14.874941+0000","last_peered":"2023-10-16T12:54:14.874941+0000","last_clean":"2023-10-16T12:54:14.874941+0000","last_became_active":"2023-10-16T12:54:13.894850+0000","last_became_peered":"2023-10-16T12:54:13.894850+0000","last_unstale":"2023-10-16T12:54:14.874941+0000","last_undegraded":"2023-10-16T12:54:14.874941+0000","last_fullsized":"2023-10-16T12:54:14.874941+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:07:50.289014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633880+0000","last_change":"2023-10-16T12:54:12.857963+0000","last_active":"2023-10-16T12:54:15.633880+0000","last_peered":"2023-10-16T12:54:15.633880+0000","last_clean":"2023-10-16T12:54:15.633880+0000","last_became_active":"2023-10-16T12:54:12.857683+0000","last_became_peered":"2023-10-16T12:54:12.857683+0000","last_unstale":"2023-10-16T12:54:15.633880+0000","last_undegraded":"2023-10-16T12:54:15.633880+0000","last_fullsized":"2023-10-16T12:54:15.633880+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:36.261813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664587+0000","last_change":"2023-10-16T12:54:13.906861+0000","last_active":"2023-10-16T12:54:15.664587+0000","last_peered":"2023-10-16T12:54:15.664587+0000","last_clean":"2023-10-16T12:54:15.664587+0000","last_became_active":"2023-10-16T12:54:13.905013+0000","last_became_peered":"2023-10-16T12:54:13.905013+0000","last_unstale":"2023-10-16T12:54:15.664587+0000","last_undegraded":"2023-10-16T12:54:15.664587+0000","last_fullsized":"2023-10-16T12:54:15.664587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:56.235900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664317+0000","last_change":"2023-10-16T12:54:12.857638+0000","last_active":"2023-10-16T12:54:15.664317+0000","last_peered":"2023-10-16T12:54:15.664317+0000","last_clean":"2023-10-16T12:54:15.664317+0000","last_became_active":"2023-10-16T12:54:12.857295+0000","last_became_peered":"2023-10-16T12:54:12.857295+0000","last_unstale":"2023-10-16T12:54:15.664317+0000","last_undegraded":"2023-10-16T12:54:15.664317+0000","last_fullsized":"2023-10-16T12:54:15.664317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.689317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872668+0000","last_change":"2023-10-16T12:54:13.892390+0000","last_active":"2023-10-16T12:54:14.872668+0000","last_peered":"2023-10-16T12:54:14.872668+0000","last_clean":"2023-10-16T12:54:14.872668+0000","last_became_active":"2023-10-16T12:54:13.892116+0000","last_became_peered":"2023-10-16T12:54:13.892116+0000","last_unstale":"2023-10-16T12:54:14.872668+0000","last_undegraded":"2023-10-16T12:54:14.872668+0000","last_fullsized":"2023-10-16T12:54:14.872668+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:22.637464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662524+0000","last_change":"2023-10-16T12:54:12.854987+0000","last_active":"2023-10-16T12:54:15.662524+0000","last_peered":"2023-10-16T12:54:15.662524+0000","last_clean":"2023-10-16T12:54:15.662524+0000","last_became_active":"2023-10-16T12:54:12.853280+0000","last_became_peered":"2023-10-16T12:54:12.853280+0000","last_unstale":"2023-10-16T12:54:15.662524+0000","last_undegraded":"2023-10-16T12:54:15.662524+0000","last_fullsized":"2023-10-16T12:54:15.662524+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:54.808266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633666+0000","last_change":"2023-10-16T12:54:13.903906+0000","last_active":"2023-10-16T12:54:15.633666+0000","last_peered":"2023-10-16T12:54:15.633666+0000","last_clean":"2023-10-16T12:54:15.633666+0000","last_became_active":"2023-10-16T12:54:13.902720+0000","last_became_peered":"2023-10-16T12:54:13.902720+0000","last_unstale":"2023-10-16T12:54:15.633666+0000","last_undegraded":"2023-10-16T12:54:15.633666+0000","last_fullsized":"2023-10-16T12:54:15.633666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:40.959584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634333+0000","last_change":"2023-10-16T12:54:12.864817+0000","last_active":"2023-10-16T12:54:15.634333+0000","last_peered":"2023-10-16T12:54:15.634333+0000","last_clean":"2023-10-16T12:54:15.634333+0000","last_became_active":"2023-10-16T12:54:12.864438+0000","last_became_peered":"2023-10-16T12:54:12.864438+0000","last_unstale":"2023-10-16T12:54:15.634333+0000","last_undegraded":"2023-10-16T12:54:15.634333+0000","last_fullsized":"2023-10-16T12:54:15.634333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:41.985480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"145'252","reported_seq":770,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:54:13.878475+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:54:13.878475+0000","last_peered":"2023-10-16T12:54:13.878475+0000","last_clean":"2023-10-16T12:54:13.878475+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:54:13.878475+0000","last_undegraded":"2023-10-16T12:54:13.878475+0000","last_fullsized":"2023-10-16T12:54:13.878475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542390,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33668,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338172,"statfs":{"total":96636764160,"available":96602288128,"internally_reserved":0,"allocated":2543616,"data_stored":1515739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.278,"15min":1.3009999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7809999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.022},{"interface":"front","average":{"1min":1.218,"5min":1.2969999999999999,"15min":1.3400000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.63200000000000001},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8899999999999999},"last":1.1359999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.3720000000000001,"15min":1.266},"min":{"1min":0.83899999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.647,"5min":19.158999999999999,"15min":19.158999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.222,"15min":1.2190000000000001},"min":{"1min":0.81299999999999994,"5min":0.64800000000000002,"15min":0.64700000000000002},"max":{"1min":1.5249999999999999,"5min":5.5490000000000004,"15min":5.5490000000000004},"last":1.633}]},{"osd":2,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.135,"15min":1.181},"min":{"1min":0.70099999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.4830000000000001,"5min":2.855,"15min":2.855},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2789999999999999,"15min":1.3140000000000001},"min":{"1min":0.85599999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.8009999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.538}]},{"osd":3,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.343,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6919999999999999,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.361,"15min":1.3859999999999999},"min":{"1min":1.0920000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.823,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.2629999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2090000000000001,"15min":1.1910000000000001},"min":{"1min":0.77400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.0670000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.391,"15min":1.365},"min":{"1min":0.96399999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.597}]},{"osd":5,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3700000000000001,"15min":1.3420000000000001},"min":{"1min":1.01,"5min":0.624,"15min":0.624},"max":{"1min":2.0529999999999999,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.5569999999999999,"15min":1.498},"min":{"1min":1.2989999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.3170000000000002,"5min":2.411,"15min":2.411},"last":1.8009999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.6339999999999999,"15min":1.591},"min":{"1min":1.341,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.234,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.712},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.492,"15min":1.462},"min":{"1min":1.0469999999999999,"5min":0.75,"15min":0.75},"max":{"1min":1.996,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.7549999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542391,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1949696,"data_stored":931033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.2090000000000001,"15min":1.258},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6060000000000001,"5min":1.952,"15min":1.952},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.254,"15min":1.306},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.589,"5min":1.9399999999999999,"15min":2.0270000000000001},"last":0.89600000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1220000000000001,"15min":1.177},"min":{"1min":0.68000000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.5129999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.204,"15min":1.2050000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7949999999999999,"5min":1.9330000000000001,"15min":2.0459999999999998},"last":0.83799999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2030000000000001,"15min":1.2130000000000001},"min":{"1min":0.746,"5min":0.66900000000000004,"15min":0.627},"max":{"1min":1.631,"5min":1.752,"15min":1.8899999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.206,"15min":1.254},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.482,"5min":1.8520000000000001,"15min":1.9970000000000001},"last":0.70899999999999996}]},{"osd":3,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2849999999999999,"15min":1.3080000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.9359999999999999,"5min":2.0510000000000002,"15min":2.3900000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.329,"15min":1.333},"min":{"1min":0.84499999999999997,"5min":0.82199999999999995,"15min":0.69599999999999995},"max":{"1min":1.637,"5min":1.806,"15min":2.012},"last":1.143}]},{"osd":4,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3799999999999999,"15min":1.349},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71099999999999997},"max":{"1min":2.0920000000000001,"5min":2.0990000000000002,"15min":2.3029999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4710000000000001,"15min":1.468},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.77300000000000002},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.2250000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3129999999999999,"15min":1.3089999999999999},"min":{"1min":0.81899999999999995,"5min":0.755,"15min":0.755},"max":{"1min":1.873,"5min":1.9830000000000001,"15min":2.0910000000000002},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.351,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.78300000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.157,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.1859999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4470000000000001,"15min":1.4370000000000001},"min":{"1min":1.0009999999999999,"5min":0.91300000000000003,"15min":0.878},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.494,"15min":1.47},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.96399999999999997},"max":{"1min":2.214,"5min":2.214,"15min":2.2610000000000001},"last":1.099}]}]},{"osd":5,"up_from":14,"seq":60129542392,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":929533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0980000000000001,"15min":1.1439999999999999},"min":{"1min":0.64000000000000001,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.621,"5min":1.8480000000000001,"15min":2.2690000000000001},"last":1.345},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1779999999999999,"15min":1.1739999999999999},"min":{"1min":0.84099999999999997,"5min":0.65200000000000002,"15min":0.63300000000000001},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":1.403}]},{"osd":1,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.256,"15min":1.25},"min":{"1min":0.67200000000000004,"5min":0.56699999999999995,"15min":0.41299999999999998},"max":{"1min":1.6120000000000001,"5min":1.8080000000000001,"15min":1.8089999999999999},"last":1.478},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.1970000000000001,"15min":1.2070000000000001},"min":{"1min":0.95699999999999996,"5min":0.46000000000000002,"15min":0.41599999999999998},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.264,"15min":1.274},"min":{"1min":0.89200000000000002,"5min":0.60799999999999998,"15min":0.60199999999999998},"max":{"1min":1.881,"5min":1.9710000000000001,"15min":2.0059999999999998},"last":1.099},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.204,"15min":1.26},"min":{"1min":0.93500000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4610000000000001,"5min":1.9870000000000001,"15min":2.2370000000000001},"last":1.5069999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2809999999999999,"15min":1.304},"min":{"1min":0.78400000000000003,"5min":0.73099999999999998,"15min":0.66000000000000003},"max":{"1min":1.776,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3500000000000001,"15min":1.361},"min":{"1min":0.89700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.601,"5min":2.1150000000000002,"15min":2.2160000000000002},"last":1.444}]},{"osd":4,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.25,"15min":1.2090000000000001},"min":{"1min":0.83999999999999997,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":1.5269999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.137},{"interface":"front","average":{"1min":1.321,"5min":1.3560000000000001,"15min":1.321},"min":{"1min":0.69899999999999995,"5min":0.54200000000000004,"15min":0.42999999999999999},"max":{"1min":1.6220000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.2410000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.381,"15min":1.3440000000000001},"min":{"1min":1.014,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6830000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.421,"15min":1.389},"min":{"1min":1.046,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.734,"5min":2.48,"15min":2.48},"last":1.5900000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4690000000000001,"15min":1.4550000000000001},"min":{"1min":1.1739999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.718,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.536},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5249999999999999,"15min":1.5109999999999999},"min":{"1min":1.115,"5min":0.64400000000000002,"15min":0.55300000000000005},"max":{"1min":1.98,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5589999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542392,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59448,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312392,"statfs":{"total":96636764160,"available":96575889408,"internally_reserved":0,"allocated":1941504,"data_stored":929491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1200000000000001,"15min":1.1870000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.623,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1100000000000001,"15min":1.1439999999999999},"min":{"1min":0.66900000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.4370000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":1.4890000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.175,"15min":1.177},"min":{"1min":0.69499999999999995,"5min":0.38,"15min":0.38},"max":{"1min":1.6100000000000001,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.173,"15min":1.2390000000000001},"min":{"1min":0.73599999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.1579999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":1.671}]},{"osd":2,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.70299999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.623,"5min":4.3470000000000004,"15min":4.3470000000000004},"last":1.714},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2749999999999999,"15min":1.321},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9710000000000001,"5min":2.4670000000000001,"15min":2.4670000000000001},"last":1.5469999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2330000000000001,"15min":1.258},"min":{"1min":0.91000000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6559999999999999,"5min":2.8300000000000001,"15min":2.8300000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3640000000000001,"15min":1.3740000000000001},"min":{"1min":0.97199999999999998,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.8520000000000001,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":1.605}]},{"osd":5,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.355,"15min":1.3080000000000001},"min":{"1min":0.84499999999999997,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.2280000000000002,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.349,"15min":1.3029999999999999},"min":{"1min":0.98199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.851,"5min":4.391,"15min":4.391},"last":0.997}]},{"osd":6,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.395,"15min":1.375},"min":{"1min":1.0189999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.0600000000000001,"5min":3.4649999999999999,"15min":3.4649999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.4950000000000001,"15min":1.476},"min":{"1min":0.92000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.395,"5min":2.504,"15min":2.504},"last":1.355}]},{"osd":7,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.496,"15min":1.47},"min":{"1min":0.84099999999999997,"5min":0.67600000000000005,"15min":0.60399999999999998},"max":{"1min":2.0230000000000001,"5min":3.9369999999999998,"15min":3.9369999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5960000000000001,"15min":1.5860000000000001},"min":{"1min":1.131,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4460000000000002,"5min":3.262,"15min":3.262},"last":1.8160000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542392,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41388,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330452,"statfs":{"total":96636764160,"available":96594382848,"internally_reserved":0,"allocated":1933312,"data_stored":923281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.90200000000000002,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":1.3160000000000001,"5min":1.609,"15min":2.0419999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.02,"15min":1.141},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34799999999999998},"max":{"1min":1.661,"5min":1.661,"15min":2.2749999999999999},"last":1.0720000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.96599999999999997,"15min":1.0129999999999999},"min":{"1min":0.52700000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.96099999999999997,"15min":1.056},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.6899999999999999,"5min":1.784,"15min":2.4580000000000002},"last":1.2809999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.93100000000000005,"15min":1.044},"min":{"1min":0.56399999999999995,"5min":0.38,"15min":0.38},"max":{"1min":1.361,"5min":1.746,"15min":1.9510000000000001},"last":0.747},{"interface":"front","average":{"1min":1.145,"5min":1.216,"15min":1.3280000000000001},"min":{"1min":0.67300000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.865,"5min":2.044,"15min":2.044},"last":1.3160000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3620000000000001,"15min":1.462},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.294,"15min":1.389},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.677,"5min":1.8220000000000001,"15min":2.0099999999999998},"last":1.528}]},{"osd":5,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.258,"15min":1.3560000000000001},"min":{"1min":0.90200000000000002,"5min":0.70299999999999996,"15min":0.65900000000000003},"max":{"1min":1.5820000000000001,"5min":1.893,"15min":2.0539999999999998},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.399,"15min":1.456},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0049999999999999,"5min":2.0750000000000002,"15min":2.3540000000000001},"last":1.6299999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.365,"15min":1.4370000000000001},"min":{"1min":0.80700000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":1.696},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.3100000000000001,"15min":1.397},"min":{"1min":0.80200000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8320000000000001,"5min":2,"15min":2.3740000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4610000000000001,"15min":1.5289999999999999},"min":{"1min":0.88500000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.081,"5min":2.157,"15min":2.431},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4770000000000001,"15min":1.583},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.5760000000000001,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.585}]}]},{"osd":2,"up_from":14,"seq":60129542390,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46972,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324868,"statfs":{"total":96636764160,"available":96588664832,"internally_reserved":0,"allocated":1945600,"data_stored":930037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.876,"15min":0.91200000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.7889999999999999},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0209999999999999,"15min":1.0720000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5429999999999999,"5min":1.746,"15min":2.085},"last":0.71799999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.069,"15min":1.1180000000000001},"min":{"1min":0.442,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.591,"5min":1.677,"15min":2.5590000000000002},"last":1.155},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1539999999999999,"15min":1.2110000000000001},"min":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.46500000000000002},"max":{"1min":1.655,"5min":2.0510000000000002,"15min":2.4289999999999998},"last":1.232}]},{"osd":3,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0409999999999999,"15min":1.0760000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3740000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":1.46},{"interface":"front","average":{"1min":1.179,"5min":1.2230000000000001,"15min":1.3009999999999999},"min":{"1min":0.59799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.6020000000000001,"5min":2.238,"15min":2.238},"last":1.54}]},{"osd":4,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.367,"15min":1.4370000000000001},"min":{"1min":0.91700000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.734,"5min":2.2149999999999999,"15min":2.431},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.3580000000000001,"15min":1.411},"min":{"1min":0.874,"5min":0.75600000000000001,"15min":0.68999999999999995},"max":{"1min":1.859,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.484}]},{"osd":5,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.458,"15min":1.4950000000000001},"min":{"1min":0.79900000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.875,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.988},{"interface":"front","average":{"1min":1.294,"5min":1.3600000000000001,"15min":1.3999999999999999},"min":{"1min":0.875,"5min":0.751,"15min":0.52100000000000002},"max":{"1min":1.754,"5min":3.7759999999999998,"15min":3.7759999999999998},"last":1.7769999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.4430000000000001},"min":{"1min":0.90400000000000003,"5min":0.89400000000000002,"15min":0.59699999999999998},"max":{"1min":1.806,"5min":3.6120000000000001,"15min":3.6120000000000001},"last":1.657},{"interface":"front","average":{"1min":1.446,"5min":1.47,"15min":1.5389999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.78000000000000003},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":1.6819999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.534,"15min":1.5900000000000001},"min":{"1min":0.91900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8580000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4730000000000001,"15min":1.5269999999999999},"min":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.56599999999999995},"max":{"1min":1.913,"5min":3.73,"15min":3.73},"last":1.5629999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542390,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59448,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312392,"statfs":{"total":96636764160,"available":96575889408,"internally_reserved":0,"allocated":1937408,"data_stored":929421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.89800000000000002,"15min":0.90100000000000002},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.357,"5min":1.6359999999999999,"15min":2.0840000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.96199999999999997,"15min":1.054},"min":{"1min":0.43099999999999999,"5min":0.31,"15min":0.31},"max":{"1min":1.351,"5min":1.8500000000000001,"15min":1.9770000000000001},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.97599999999999998},"min":{"1min":0.501,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.5740000000000001,"5min":1.694,"15min":2.262},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1020000000000001,"15min":1.171},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.696,"5min":1.9910000000000001,"15min":2.2210000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.208,"15min":1.2909999999999999},"min":{"1min":0.499,"5min":0.47899999999999998,"15min":0.46100000000000002},"max":{"1min":1.8500000000000001,"5min":2.073,"15min":2.073},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.113,"5min":1.1299999999999999,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.597,"5min":1.758,"15min":1.9199999999999999},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2829999999999999},"min":{"1min":0.71399999999999997,"5min":0.59599999999999997,"15min":0.54600000000000004},"max":{"1min":1.536,"5min":1.7350000000000001,"15min":2.1030000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3979999999999999,"15min":1.4319999999999999},"min":{"1min":0.77600000000000002,"5min":0.67900000000000005,"15min":0.60599999999999998},"max":{"1min":1.9750000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.0760000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3759999999999999,"15min":1.413},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0649999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3640000000000001,"15min":1.4259999999999999},"min":{"1min":0.755,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0049999999999999,"5min":2.1179999999999999,"15min":2.4350000000000001},"last":1.349}]},{"osd":6,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.478,"15min":1.52},"min":{"1min":0.749,"5min":0.72099999999999997,"15min":0.68999999999999995},"max":{"1min":2.0800000000000001,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4670000000000001,"15min":1.508},"min":{"1min":0.76800000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.2029999999999998,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":1.0229999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2749999999999999,"5min":2.383,"15min":2.383},"last":1.109},{"interface":"front","average":{"1min":1.504,"5min":1.49,"15min":1.548},"min":{"1min":0.80700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.0539999999999998,"5min":2.1280000000000001,"15min":2.5710000000000002},"last":1.1539999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542391,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60024,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311816,"statfs":{"total":96636764160,"available":96575299584,"internally_reserved":0,"allocated":2531328,"data_stored":1519789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.81399999999999995,"15min":0.871},"min":{"1min":0.39000000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.085,"5min":1.655,"15min":1.6890000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.98899999999999999,"15min":1.0680000000000001},"min":{"1min":0.61099999999999999,"5min":0.441,"15min":0.441},"max":{"1min":1.601,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.506}]},{"osd":2,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.042,"15min":1.105},"min":{"1min":0.57799999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.304,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0209999999999999,"15min":1.135},"min":{"1min":0.5,"5min":0.497,"15min":0.496},"max":{"1min":1.47,"5min":1.788,"15min":1.952},"last":1.694}]},{"osd":3,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96599999999999997,"15min":1.0660000000000001},"min":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":1.516,"5min":1.8660000000000001,"15min":1.9570000000000001},"last":1},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.115,"15min":1.244},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.756,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":1.587}]},{"osd":4,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.224,"15min":1.3140000000000001},"min":{"1min":0.55500000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.597,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.337,"15min":1.409},"min":{"1min":0.85499999999999998,"5min":0.625,"15min":0.59699999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.173},"last":1.458}]},{"osd":5,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.331,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7130000000000001,"5min":1.9630000000000001,"15min":2.2440000000000002},"last":1.909},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.298,"15min":1.3879999999999999},"min":{"1min":0.66400000000000003,"5min":0.63100000000000001,"15min":0.58999999999999997},"max":{"1min":1.6319999999999999,"5min":1.9970000000000001,"15min":2.0270000000000001},"last":1.4179999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.419,"15min":1.496},"min":{"1min":0.86699999999999999,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0670000000000002,"5min":2.125,"15min":2.3519999999999999},"last":2.0539999999999998},{"interface":"front","average":{"1min":1.286,"5min":1.3580000000000001,"15min":1.48},"min":{"1min":0.67600000000000005,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.8340000000000001,"5min":1.95,"15min":2.363},"last":2.1110000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.496,"15min":1.5960000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0019999999999998,"5min":2.395,"15min":2.4729999999999999},"last":2.379},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.4350000000000001,"15min":1.528},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.734,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.7509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:54:18.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:54:18.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.691+0000 7f146ffff640 1 Processor -- start 2023-10-16T12:54:18.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.691+0000 7f146ffff640 1 -- start start 2023-10-16T12:54:18.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.693+0000 7f146ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f147006c040 0x7f147006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.693+0000 7f146ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f147015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.693+0000 7f146ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147006c960 con 0x7f147015ebc0 2023-10-16T12:54:18.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.693+0000 7f146ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147015b9d0 con 0x7f147006c040 2023-10-16T12:54:18.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.693+0000 7f146ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147015bcb0 con 0x7f14700ad970 2023-10-16T12:54:18.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f146effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f147006c040 0x7f147006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f146effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f147006c040 0x7f147006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39760/0 (socket says 172.21.15.60:39760) 2023-10-16T12:54:18.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f146effd640 1 -- 172.21.15.60:0/1074503760 learned_addr learned my addr 172.21.15.60:0/1074503760 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:18.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f1467fff640 1 --2- 172.21.15.60:0/1074503760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f147015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f146effd640 1 -- 172.21.15.60:0/1074503760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 msgr2=0x7f147015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:18.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f146effd640 1 --2- 172.21.15.60:0/1074503760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f147015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.694+0000 7f146effd640 1 -- 172.21.15.60:0/1074503760 >> v1:172.21.15.144:6789/0 conn(0x7f147015ebc0 legacy=0x7f14700cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.695+0000 7f146effd640 1 -- 172.21.15.60:0/1074503760 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147015bf90 con 0x7f147006c040 2023-10-16T12:54:18.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.696+0000 7f146effd640 1 --2- 172.21.15.60:0/1074503760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147006c040 0x7f147006c420 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f14600098a0 tx=0x7f146002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c002e148fd5672 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:18.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.696+0000 7f146e7fc640 1 -- 172.21.15.60:0/1074503760 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460030840 con 0x7f147006c040 2023-10-16T12:54:18.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146e7fc640 1 -- 172.21.15.60:0/1074503760 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14600309a0 con 0x7f147006c040 2023-10-16T12:54:18.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146e7fc640 1 -- 172.21.15.60:0/1074503760 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460030d30 con 0x7f147006c040 2023-10-16T12:54:18.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 -- 172.21.15.60:0/1074503760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147006c040 msgr2=0x7f147006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 --2- 172.21.15.60:0/1074503760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147006c040 0x7f147006c420 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f14600098a0 tx=0x7f146002b980 comp rx=0 tx=0).stop 2023-10-16T12:54:18.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 -- 172.21.15.60:0/1074503760 shutdown_connections 2023-10-16T12:54:18.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 --2- 172.21.15.60:0/1074503760 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f147015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 --2- 172.21.15.60:0/1074503760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147006c040 0x7f147006c420 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 -- 172.21.15.60:0/1074503760 >> 172.21.15.60:0/1074503760 conn(0x7f14700ce6a0 msgr2=0x7f14700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:18.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 -- 172.21.15.60:0/1074503760 shutdown_connections 2023-10-16T12:54:18.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.697+0000 7f146ffff640 1 -- 172.21.15.60:0/1074503760 wait complete. 2023-10-16T12:54:18.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.699+0000 7f146ffff640 1 Processor -- start 2023-10-16T12:54:18.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.699+0000 7f146ffff640 1 -- start start 2023-10-16T12:54:18.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.700+0000 7f146ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f147006c040 0x7f1470153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.700+0000 7f146ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f14700c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.700+0000 7f146ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 0x7f14700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.700+0000 7f146ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700c7960 con 0x7f147015ebc0 2023-10-16T12:54:18.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.700+0000 7f146ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14700c9c40 con 0x7f147006c040 2023-10-16T12:54:18.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.700+0000 7f146ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700c6d50 con 0x7f14700ad970 2023-10-16T12:54:18.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 0x7f14700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 0x7f14700ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39766/0 (socket says 172.21.15.60:39766) 2023-10-16T12:54:18.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 -- 172.21.15.60:0/994104790 learned_addr learned my addr 172.21.15.60:0/994104790 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:18.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146effd640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f147006c040 0x7f1470153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f1467fff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f14700c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 -- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 msgr2=0x7f14700c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f14700c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 -- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f147006c040 msgr2=0x7f1470153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f147006c040 0x7f1470153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f146f7fe640 1 -- 172.21.15.60:0/994104790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600094d0 con 0x7f147015ebc0 2023-10-16T12:54:18.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.701+0000 7f1467fff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f14700c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:18.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.702+0000 7f146f7fe640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 0x7f14700ac890 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f146800c7a0 tx=0x7f146800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:18.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.702+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468003b70 con 0x7f147015ebc0 2023-10-16T12:54:18.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.702+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468003cf0 con 0x7f147015ebc0 2023-10-16T12:54:18.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.702+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468003ea0 con 0x7f147015ebc0 2023-10-16T12:54:18.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.702+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14700c7030 con 0x7f147015ebc0 2023-10-16T12:54:18.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.702+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14700c75f0 con 0x7f147015ebc0 2023-10-16T12:54:18.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.703+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14700c77b0 con 0x7f147015ebc0 2023-10-16T12:54:18.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.704+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f146800aa90 con 0x7f147015ebc0 2023-10-16T12:54:18.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.705+0000 7f14677fe640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f145006af90 0x7f145006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:18.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.705+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14680895d0 con 0x7f147015ebc0 2023-10-16T12:54:18.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.706+0000 7f146effd640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f145006af90 0x7f145006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:18.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.707+0000 7f146effd640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f145006af90 0x7f145006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1460036000 tx=0x7f146003b040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:18.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.709+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1468054560 con 0x7f147015ebc0 2023-10-16T12:54:18.815 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T12:54:18.813+0000 7f7e6117b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:54:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.903+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 --> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14700cc550 con 0x7f145006af90 2023-10-16T12:54:18.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.926+0000 7f14677fe640 1 -- 172.21.15.60:0/994104790 <== mgr.7225 v2:172.21.15.144:6832/3094169155 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287279 (secure 0 0 0) 0x7f1460030b90 con 0x7f145006af90 2023-10-16T12:54:18.931 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:18.933 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T12:54:18.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f145006af90 msgr2=0x7f145006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f145006af90 0x7f145006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1460036000 tx=0x7f146003b040 comp rx=0 tx=0).stop 2023-10-16T12:54:18.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 msgr2=0x7f14700ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:18.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 0x7f14700ac890 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f146800c7a0 tx=0x7f146800cc60 comp rx=0 tx=0).stop 2023-10-16T12:54:18.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 shutdown_connections 2023-10-16T12:54:18.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f145006af90 0x7f145006d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14700ad970 0x7f14700c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f147015ebc0 0x7f14700ac890 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 --2- 172.21.15.60:0/994104790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f147006c040 0x7f1470153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:18.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.932+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 >> 172.21.15.60:0/994104790 conn(0x7f14700ce6a0 msgr2=0x7f14700cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:18.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.933+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 shutdown_connections 2023-10-16T12:54:18.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:18.933+0000 7f146ffff640 1 -- 172.21.15.60:0/994104790 wait complete. 2023-10-16T12:54:18.955 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T12:54:17.143675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":447,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":375432,"kb_used_data":16336,"kb_used_omap":12,"kb_used_meta":358963,"kb_avail":754599288,"statfs":{"total":773094113280,"available":772709670912,"internally_reserved":0,"allocated":16728064,"data_stored":8608324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":367578680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.322611"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549447+0000","last_change":"2023-10-16T12:54:13.901564+0000","last_active":"2023-10-16T12:54:15.549447+0000","last_peered":"2023-10-16T12:54:15.549447+0000","last_clean":"2023-10-16T12:54:15.549447+0000","last_became_active":"2023-10-16T12:54:13.901280+0000","last_became_peered":"2023-10-16T12:54:13.901280+0000","last_unstale":"2023-10-16T12:54:15.549447+0000","last_undegraded":"2023-10-16T12:54:15.549447+0000","last_fullsized":"2023-10-16T12:54:15.549447+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:55.120581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874077+0000","last_change":"2023-10-16T12:54:13.883701+0000","last_active":"2023-10-16T12:54:14.874077+0000","last_peered":"2023-10-16T12:54:14.874077+0000","last_clean":"2023-10-16T12:54:14.874077+0000","last_became_active":"2023-10-16T12:54:13.883115+0000","last_became_peered":"2023-10-16T12:54:13.883115+0000","last_unstale":"2023-10-16T12:54:14.874077+0000","last_undegraded":"2023-10-16T12:54:14.874077+0000","last_fullsized":"2023-10-16T12:54:14.874077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:58.472483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663319+0000","last_change":"2023-10-16T12:54:13.900901+0000","last_active":"2023-10-16T12:54:15.663319+0000","last_peered":"2023-10-16T12:54:15.663319+0000","last_clean":"2023-10-16T12:54:15.663319+0000","last_became_active":"2023-10-16T12:54:13.900597+0000","last_became_peered":"2023-10-16T12:54:13.900597+0000","last_unstale":"2023-10-16T12:54:15.663319+0000","last_undegraded":"2023-10-16T12:54:15.663319+0000","last_fullsized":"2023-10-16T12:54:15.663319+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:36.213936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548666+0000","last_change":"2023-10-16T12:54:13.896257+0000","last_active":"2023-10-16T12:54:15.548666+0000","last_peered":"2023-10-16T12:54:15.548666+0000","last_clean":"2023-10-16T12:54:15.548666+0000","last_became_active":"2023-10-16T12:54:13.895980+0000","last_became_peered":"2023-10-16T12:54:13.895980+0000","last_unstale":"2023-10-16T12:54:15.548666+0000","last_undegraded":"2023-10-16T12:54:15.548666+0000","last_fullsized":"2023-10-16T12:54:15.548666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:55.823585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634082+0000","last_change":"2023-10-16T12:54:13.893083+0000","last_active":"2023-10-16T12:54:15.634082+0000","last_peered":"2023-10-16T12:54:15.634082+0000","last_clean":"2023-10-16T12:54:15.634082+0000","last_became_active":"2023-10-16T12:54:13.891680+0000","last_became_peered":"2023-10-16T12:54:13.891680+0000","last_unstale":"2023-10-16T12:54:15.634082+0000","last_undegraded":"2023-10-16T12:54:15.634082+0000","last_fullsized":"2023-10-16T12:54:15.634082+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:24.019225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875851+0000","last_change":"2023-10-16T12:54:13.898617+0000","last_active":"2023-10-16T12:54:14.875851+0000","last_peered":"2023-10-16T12:54:14.875851+0000","last_clean":"2023-10-16T12:54:14.875851+0000","last_became_active":"2023-10-16T12:54:13.898278+0000","last_became_peered":"2023-10-16T12:54:13.898278+0000","last_unstale":"2023-10-16T12:54:14.875851+0000","last_undegraded":"2023-10-16T12:54:14.875851+0000","last_fullsized":"2023-10-16T12:54:14.875851+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:41:34.209338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873239+0000","last_change":"2023-10-16T12:54:13.900560+0000","last_active":"2023-10-16T12:54:14.873239+0000","last_peered":"2023-10-16T12:54:14.873239+0000","last_clean":"2023-10-16T12:54:14.873239+0000","last_became_active":"2023-10-16T12:54:13.899414+0000","last_became_peered":"2023-10-16T12:54:13.899414+0000","last_unstale":"2023-10-16T12:54:14.873239+0000","last_undegraded":"2023-10-16T12:54:14.873239+0000","last_fullsized":"2023-10-16T12:54:14.873239+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:04.318061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662893+0000","last_change":"2023-10-16T12:54:13.906051+0000","last_active":"2023-10-16T12:54:15.662893+0000","last_peered":"2023-10-16T12:54:15.662893+0000","last_clean":"2023-10-16T12:54:15.662893+0000","last_became_active":"2023-10-16T12:54:13.905105+0000","last_became_peered":"2023-10-16T12:54:13.905105+0000","last_unstale":"2023-10-16T12:54:15.662893+0000","last_undegraded":"2023-10-16T12:54:15.662893+0000","last_fullsized":"2023-10-16T12:54:15.662893+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:44.110743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663773+0000","last_change":"2023-10-16T12:54:13.897694+0000","last_active":"2023-10-16T12:54:15.663773+0000","last_peered":"2023-10-16T12:54:15.663773+0000","last_clean":"2023-10-16T12:54:15.663773+0000","last_became_active":"2023-10-16T12:54:13.897352+0000","last_became_peered":"2023-10-16T12:54:13.897352+0000","last_unstale":"2023-10-16T12:54:15.663773+0000","last_undegraded":"2023-10-16T12:54:15.663773+0000","last_fullsized":"2023-10-16T12:54:15.663773+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:07.179112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874280+0000","last_change":"2023-10-16T12:54:13.886309+0000","last_active":"2023-10-16T12:54:14.874280+0000","last_peered":"2023-10-16T12:54:14.874280+0000","last_clean":"2023-10-16T12:54:14.874280+0000","last_became_active":"2023-10-16T12:54:13.885820+0000","last_became_peered":"2023-10-16T12:54:13.885820+0000","last_unstale":"2023-10-16T12:54:14.874280+0000","last_undegraded":"2023-10-16T12:54:14.874280+0000","last_fullsized":"2023-10-16T12:54:14.874280+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:06.720936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662692+0000","last_change":"2023-10-16T12:54:13.882659+0000","last_active":"2023-10-16T12:54:15.662692+0000","last_peered":"2023-10-16T12:54:15.662692+0000","last_clean":"2023-10-16T12:54:15.662692+0000","last_became_active":"2023-10-16T12:54:13.882342+0000","last_became_peered":"2023-10-16T12:54:13.882342+0000","last_unstale":"2023-10-16T12:54:15.662692+0000","last_undegraded":"2023-10-16T12:54:15.662692+0000","last_fullsized":"2023-10-16T12:54:15.662692+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:50.778642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663458+0000","last_change":"2023-10-16T12:54:13.901151+0000","last_active":"2023-10-16T12:54:15.663458+0000","last_peered":"2023-10-16T12:54:15.663458+0000","last_clean":"2023-10-16T12:54:15.663458+0000","last_became_active":"2023-10-16T12:54:13.900852+0000","last_became_peered":"2023-10-16T12:54:13.900852+0000","last_unstale":"2023-10-16T12:54:15.663458+0000","last_undegraded":"2023-10-16T12:54:15.663458+0000","last_fullsized":"2023-10-16T12:54:15.663458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:49.700950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548584+0000","last_change":"2023-10-16T12:54:13.894794+0000","last_active":"2023-10-16T12:54:15.548584+0000","last_peered":"2023-10-16T12:54:15.548584+0000","last_clean":"2023-10-16T12:54:15.548584+0000","last_became_active":"2023-10-16T12:54:13.891670+0000","last_became_peered":"2023-10-16T12:54:13.891670+0000","last_unstale":"2023-10-16T12:54:15.548584+0000","last_undegraded":"2023-10-16T12:54:15.548584+0000","last_fullsized":"2023-10-16T12:54:15.548584+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:21.449193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873036+0000","last_change":"2023-10-16T12:54:13.899263+0000","last_active":"2023-10-16T12:54:14.873036+0000","last_peered":"2023-10-16T12:54:14.873036+0000","last_clean":"2023-10-16T12:54:14.873036+0000","last_became_active":"2023-10-16T12:54:13.898523+0000","last_became_peered":"2023-10-16T12:54:13.898523+0000","last_unstale":"2023-10-16T12:54:14.873036+0000","last_undegraded":"2023-10-16T12:54:14.873036+0000","last_fullsized":"2023-10-16T12:54:14.873036+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:50.778642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548333+0000","last_change":"2023-10-16T12:54:13.901168+0000","last_active":"2023-10-16T12:54:15.548333+0000","last_peered":"2023-10-16T12:54:15.548333+0000","last_clean":"2023-10-16T12:54:15.548333+0000","last_became_active":"2023-10-16T12:54:13.900886+0000","last_became_peered":"2023-10-16T12:54:13.900886+0000","last_unstale":"2023-10-16T12:54:15.548333+0000","last_undegraded":"2023-10-16T12:54:15.548333+0000","last_fullsized":"2023-10-16T12:54:15.548333+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:57:35.656529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664360+0000","last_change":"2023-10-16T12:54:13.911663+0000","last_active":"2023-10-16T12:54:15.664360+0000","last_peered":"2023-10-16T12:54:15.664360+0000","last_clean":"2023-10-16T12:54:15.664360+0000","last_became_active":"2023-10-16T12:54:13.911378+0000","last_became_peered":"2023-10-16T12:54:13.911378+0000","last_unstale":"2023-10-16T12:54:15.664360+0000","last_undegraded":"2023-10-16T12:54:15.664360+0000","last_fullsized":"2023-10-16T12:54:15.664360+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:50.801289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876321+0000","last_change":"2023-10-16T12:54:13.883297+0000","last_active":"2023-10-16T12:54:14.876321+0000","last_peered":"2023-10-16T12:54:14.876321+0000","last_clean":"2023-10-16T12:54:14.876321+0000","last_became_active":"2023-10-16T12:54:13.882182+0000","last_became_peered":"2023-10-16T12:54:13.882182+0000","last_unstale":"2023-10-16T12:54:14.876321+0000","last_undegraded":"2023-10-16T12:54:14.876321+0000","last_fullsized":"2023-10-16T12:54:14.876321+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:07.904494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665181+0000","last_change":"2023-10-16T12:54:13.902055+0000","last_active":"2023-10-16T12:54:15.665181+0000","last_peered":"2023-10-16T12:54:15.665181+0000","last_clean":"2023-10-16T12:54:15.665181+0000","last_became_active":"2023-10-16T12:54:13.901404+0000","last_became_peered":"2023-10-16T12:54:13.901404+0000","last_unstale":"2023-10-16T12:54:15.665181+0000","last_undegraded":"2023-10-16T12:54:15.665181+0000","last_fullsized":"2023-10-16T12:54:15.665181+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:35.225378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663095+0000","last_change":"2023-10-16T12:54:13.873645+0000","last_active":"2023-10-16T12:54:15.663095+0000","last_peered":"2023-10-16T12:54:15.663095+0000","last_clean":"2023-10-16T12:54:15.663095+0000","last_became_active":"2023-10-16T12:54:13.872856+0000","last_became_peered":"2023-10-16T12:54:13.872856+0000","last_unstale":"2023-10-16T12:54:15.663095+0000","last_undegraded":"2023-10-16T12:54:15.663095+0000","last_fullsized":"2023-10-16T12:54:15.663095+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:12.558438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664821+0000","last_change":"2023-10-16T12:54:13.906174+0000","last_active":"2023-10-16T12:54:15.664821+0000","last_peered":"2023-10-16T12:54:15.664821+0000","last_clean":"2023-10-16T12:54:15.664821+0000","last_became_active":"2023-10-16T12:54:13.905712+0000","last_became_peered":"2023-10-16T12:54:13.905712+0000","last_unstale":"2023-10-16T12:54:15.664821+0000","last_undegraded":"2023-10-16T12:54:15.664821+0000","last_fullsized":"2023-10-16T12:54:15.664821+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:50.302351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634708+0000","last_change":"2023-10-16T12:54:13.891370+0000","last_active":"2023-10-16T12:54:15.634708+0000","last_peered":"2023-10-16T12:54:15.634708+0000","last_clean":"2023-10-16T12:54:15.634708+0000","last_became_active":"2023-10-16T12:54:13.890881+0000","last_became_peered":"2023-10-16T12:54:13.890881+0000","last_unstale":"2023-10-16T12:54:15.634708+0000","last_undegraded":"2023-10-16T12:54:15.634708+0000","last_fullsized":"2023-10-16T12:54:15.634708+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:45.080091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662626+0000","last_change":"2023-10-16T12:54:13.890994+0000","last_active":"2023-10-16T12:54:15.662626+0000","last_peered":"2023-10-16T12:54:15.662626+0000","last_clean":"2023-10-16T12:54:15.662626+0000","last_became_active":"2023-10-16T12:54:13.890705+0000","last_became_peered":"2023-10-16T12:54:13.890705+0000","last_unstale":"2023-10-16T12:54:15.662626+0000","last_undegraded":"2023-10-16T12:54:15.662626+0000","last_fullsized":"2023-10-16T12:54:15.662626+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:29.959281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663093+0000","last_change":"2023-10-16T12:54:13.897894+0000","last_active":"2023-10-16T12:54:15.663093+0000","last_peered":"2023-10-16T12:54:15.663093+0000","last_clean":"2023-10-16T12:54:15.663093+0000","last_became_active":"2023-10-16T12:54:13.897125+0000","last_became_peered":"2023-10-16T12:54:13.897125+0000","last_unstale":"2023-10-16T12:54:15.663093+0000","last_undegraded":"2023-10-16T12:54:15.663093+0000","last_fullsized":"2023-10-16T12:54:15.663093+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:56.549726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662151+0000","last_change":"2023-10-16T12:54:13.879276+0000","last_active":"2023-10-16T12:54:15.662151+0000","last_peered":"2023-10-16T12:54:15.662151+0000","last_clean":"2023-10-16T12:54:15.662151+0000","last_became_active":"2023-10-16T12:54:13.878472+0000","last_became_peered":"2023-10-16T12:54:13.878472+0000","last_unstale":"2023-10-16T12:54:15.662151+0000","last_undegraded":"2023-10-16T12:54:15.662151+0000","last_fullsized":"2023-10-16T12:54:15.662151+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:22.637464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874138+0000","last_change":"2023-10-16T12:54:13.890994+0000","last_active":"2023-10-16T12:54:14.874138+0000","last_peered":"2023-10-16T12:54:14.874138+0000","last_clean":"2023-10-16T12:54:14.874138+0000","last_became_active":"2023-10-16T12:54:13.890723+0000","last_became_peered":"2023-10-16T12:54:13.890723+0000","last_unstale":"2023-10-16T12:54:14.874138+0000","last_undegraded":"2023-10-16T12:54:14.874138+0000","last_fullsized":"2023-10-16T12:54:14.874138+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:25.643113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665248+0000","last_change":"2023-10-16T12:54:13.913358+0000","last_active":"2023-10-16T12:54:15.665248+0000","last_peered":"2023-10-16T12:54:15.665248+0000","last_clean":"2023-10-16T12:54:15.665248+0000","last_became_active":"2023-10-16T12:54:13.913059+0000","last_became_peered":"2023-10-16T12:54:13.913059+0000","last_unstale":"2023-10-16T12:54:15.665248+0000","last_undegraded":"2023-10-16T12:54:15.665248+0000","last_fullsized":"2023-10-16T12:54:15.665248+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:06.637997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548713+0000","last_change":"2023-10-16T12:54:13.901807+0000","last_active":"2023-10-16T12:54:15.548713+0000","last_peered":"2023-10-16T12:54:15.548713+0000","last_clean":"2023-10-16T12:54:15.548713+0000","last_became_active":"2023-10-16T12:54:13.901457+0000","last_became_peered":"2023-10-16T12:54:13.901457+0000","last_unstale":"2023-10-16T12:54:15.548713+0000","last_undegraded":"2023-10-16T12:54:15.548713+0000","last_fullsized":"2023-10-16T12:54:15.548713+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:16.404805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874766+0000","last_change":"2023-10-16T12:54:13.900895+0000","last_active":"2023-10-16T12:54:14.874766+0000","last_peered":"2023-10-16T12:54:14.874766+0000","last_clean":"2023-10-16T12:54:14.874766+0000","last_became_active":"2023-10-16T12:54:13.899915+0000","last_became_peered":"2023-10-16T12:54:13.899915+0000","last_unstale":"2023-10-16T12:54:14.874766+0000","last_undegraded":"2023-10-16T12:54:14.874766+0000","last_fullsized":"2023-10-16T12:54:14.874766+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:44.510594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548580+0000","last_change":"2023-10-16T12:54:13.903111+0000","last_active":"2023-10-16T12:54:15.548580+0000","last_peered":"2023-10-16T12:54:15.548580+0000","last_clean":"2023-10-16T12:54:15.548580+0000","last_became_active":"2023-10-16T12:54:13.902830+0000","last_became_peered":"2023-10-16T12:54:13.902830+0000","last_unstale":"2023-10-16T12:54:15.548580+0000","last_undegraded":"2023-10-16T12:54:15.548580+0000","last_fullsized":"2023-10-16T12:54:15.548580+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:28.610173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548488+0000","last_change":"2023-10-16T12:54:13.901689+0000","last_active":"2023-10-16T12:54:15.548488+0000","last_peered":"2023-10-16T12:54:15.548488+0000","last_clean":"2023-10-16T12:54:15.548488+0000","last_became_active":"2023-10-16T12:54:13.901104+0000","last_became_peered":"2023-10-16T12:54:13.901104+0000","last_unstale":"2023-10-16T12:54:15.548488+0000","last_undegraded":"2023-10-16T12:54:15.548488+0000","last_fullsized":"2023-10-16T12:54:15.548488+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:29.334888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633980+0000","last_change":"2023-10-16T12:54:13.898231+0000","last_active":"2023-10-16T12:54:15.633980+0000","last_peered":"2023-10-16T12:54:15.633980+0000","last_clean":"2023-10-16T12:54:15.633980+0000","last_became_active":"2023-10-16T12:54:13.897823+0000","last_became_peered":"2023-10-16T12:54:13.897823+0000","last_unstale":"2023-10-16T12:54:15.633980+0000","last_undegraded":"2023-10-16T12:54:15.633980+0000","last_fullsized":"2023-10-16T12:54:15.633980+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:02.469574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873155+0000","last_change":"2023-10-16T12:54:13.909632+0000","last_active":"2023-10-16T12:54:14.873155+0000","last_peered":"2023-10-16T12:54:14.873155+0000","last_clean":"2023-10-16T12:54:14.873155+0000","last_became_active":"2023-10-16T12:54:13.909332+0000","last_became_peered":"2023-10-16T12:54:13.909332+0000","last_unstale":"2023-10-16T12:54:14.873155+0000","last_undegraded":"2023-10-16T12:54:14.873155+0000","last_fullsized":"2023-10-16T12:54:14.873155+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:02.670596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665519+0000","last_change":"2023-10-16T12:54:13.901868+0000","last_active":"2023-10-16T12:54:15.665519+0000","last_peered":"2023-10-16T12:54:15.665519+0000","last_clean":"2023-10-16T12:54:15.665519+0000","last_became_active":"2023-10-16T12:54:13.901005+0000","last_became_peered":"2023-10-16T12:54:13.901005+0000","last_unstale":"2023-10-16T12:54:15.665519+0000","last_undegraded":"2023-10-16T12:54:15.665519+0000","last_fullsized":"2023-10-16T12:54:15.665519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:43.000800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"155'4","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634597+0000","last_change":"2023-10-16T12:54:12.856409+0000","last_active":"2023-10-16T12:54:15.634597+0000","last_peered":"2023-10-16T12:54:15.634597+0000","last_clean":"2023-10-16T12:54:15.634597+0000","last_became_active":"2023-10-16T12:54:12.856105+0000","last_became_peered":"2023-10-16T12:54:12.856105+0000","last_unstale":"2023-10-16T12:54:15.634597+0000","last_undegraded":"2023-10-16T12:54:15.634597+0000","last_fullsized":"2023-10-16T12:54:15.634597+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:56.068424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663779+0000","last_change":"2023-10-16T12:54:13.894968+0000","last_active":"2023-10-16T12:54:15.663779+0000","last_peered":"2023-10-16T12:54:15.663779+0000","last_clean":"2023-10-16T12:54:15.663779+0000","last_became_active":"2023-10-16T12:54:13.894700+0000","last_became_peered":"2023-10-16T12:54:13.894700+0000","last_unstale":"2023-10-16T12:54:15.663779+0000","last_undegraded":"2023-10-16T12:54:15.663779+0000","last_fullsized":"2023-10-16T12:54:15.663779+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:20.017229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875435+0000","last_change":"2023-10-16T12:54:12.869100+0000","last_active":"2023-10-16T12:54:14.875435+0000","last_peered":"2023-10-16T12:54:14.875435+0000","last_clean":"2023-10-16T12:54:14.875435+0000","last_became_active":"2023-10-16T12:54:12.868772+0000","last_became_peered":"2023-10-16T12:54:12.868772+0000","last_unstale":"2023-10-16T12:54:14.875435+0000","last_undegraded":"2023-10-16T12:54:14.875435+0000","last_fullsized":"2023-10-16T12:54:14.875435+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:06.889175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634661+0000","last_change":"2023-10-16T12:54:13.897603+0000","last_active":"2023-10-16T12:54:15.634661+0000","last_peered":"2023-10-16T12:54:15.634661+0000","last_clean":"2023-10-16T12:54:15.634661+0000","last_became_active":"2023-10-16T12:54:13.897169+0000","last_became_peered":"2023-10-16T12:54:13.897169+0000","last_unstale":"2023-10-16T12:54:15.634661+0000","last_undegraded":"2023-10-16T12:54:15.634661+0000","last_fullsized":"2023-10-16T12:54:15.634661+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:44.549528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663046+0000","last_change":"2023-10-16T12:54:12.856792+0000","last_active":"2023-10-16T12:54:15.663046+0000","last_peered":"2023-10-16T12:54:15.663046+0000","last_clean":"2023-10-16T12:54:15.663046+0000","last_became_active":"2023-10-16T12:54:12.856513+0000","last_became_peered":"2023-10-16T12:54:12.856513+0000","last_unstale":"2023-10-16T12:54:15.663046+0000","last_undegraded":"2023-10-16T12:54:15.663046+0000","last_fullsized":"2023-10-16T12:54:15.663046+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:54.105261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634123+0000","last_change":"2023-10-16T12:54:13.898862+0000","last_active":"2023-10-16T12:54:15.634123+0000","last_peered":"2023-10-16T12:54:15.634123+0000","last_clean":"2023-10-16T12:54:15.634123+0000","last_became_active":"2023-10-16T12:54:13.898488+0000","last_became_peered":"2023-10-16T12:54:13.898488+0000","last_unstale":"2023-10-16T12:54:15.634123+0000","last_undegraded":"2023-10-16T12:54:15.634123+0000","last_fullsized":"2023-10-16T12:54:15.634123+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:04.718383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873742+0000","last_change":"2023-10-16T12:54:12.860475+0000","last_active":"2023-10-16T12:54:14.873742+0000","last_peered":"2023-10-16T12:54:14.873742+0000","last_clean":"2023-10-16T12:54:14.873742+0000","last_became_active":"2023-10-16T12:54:12.860067+0000","last_became_peered":"2023-10-16T12:54:12.860067+0000","last_unstale":"2023-10-16T12:54:14.873742+0000","last_undegraded":"2023-10-16T12:54:14.873742+0000","last_fullsized":"2023-10-16T12:54:14.873742+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:15.389485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664693+0000","last_change":"2023-10-16T12:54:13.902386+0000","last_active":"2023-10-16T12:54:15.664693+0000","last_peered":"2023-10-16T12:54:15.664693+0000","last_clean":"2023-10-16T12:54:15.664693+0000","last_became_active":"2023-10-16T12:54:13.900860+0000","last_became_peered":"2023-10-16T12:54:13.900860+0000","last_unstale":"2023-10-16T12:54:15.664693+0000","last_undegraded":"2023-10-16T12:54:15.664693+0000","last_fullsized":"2023-10-16T12:54:15.664693+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:00.945054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873279+0000","last_change":"2023-10-16T12:54:12.863695+0000","last_active":"2023-10-16T12:54:14.873279+0000","last_peered":"2023-10-16T12:54:14.873279+0000","last_clean":"2023-10-16T12:54:14.873279+0000","last_became_active":"2023-10-16T12:54:12.863438+0000","last_became_peered":"2023-10-16T12:54:12.863438+0000","last_unstale":"2023-10-16T12:54:14.873279+0000","last_undegraded":"2023-10-16T12:54:14.873279+0000","last_fullsized":"2023-10-16T12:54:14.873279+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:28.319569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876394+0000","last_change":"2023-10-16T12:54:13.878036+0000","last_active":"2023-10-16T12:54:14.876394+0000","last_peered":"2023-10-16T12:54:14.876394+0000","last_clean":"2023-10-16T12:54:14.876394+0000","last_became_active":"2023-10-16T12:54:13.877114+0000","last_became_peered":"2023-10-16T12:54:13.877114+0000","last_unstale":"2023-10-16T12:54:14.876394+0000","last_undegraded":"2023-10-16T12:54:14.876394+0000","last_fullsized":"2023-10-16T12:54:14.876394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:51.991631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664039+0000","last_change":"2023-10-16T12:54:12.849710+0000","last_active":"2023-10-16T12:54:15.664039+0000","last_peered":"2023-10-16T12:54:15.664039+0000","last_clean":"2023-10-16T12:54:15.664039+0000","last_became_active":"2023-10-16T12:54:12.846509+0000","last_became_peered":"2023-10-16T12:54:12.846509+0000","last_unstale":"2023-10-16T12:54:15.664039+0000","last_undegraded":"2023-10-16T12:54:15.664039+0000","last_fullsized":"2023-10-16T12:54:15.664039+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:59.579472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663247+0000","last_change":"2023-10-16T12:54:13.902131+0000","last_active":"2023-10-16T12:54:15.663247+0000","last_peered":"2023-10-16T12:54:15.663247+0000","last_clean":"2023-10-16T12:54:15.663247+0000","last_became_active":"2023-10-16T12:54:13.901829+0000","last_became_peered":"2023-10-16T12:54:13.901829+0000","last_unstale":"2023-10-16T12:54:15.663247+0000","last_undegraded":"2023-10-16T12:54:15.663247+0000","last_fullsized":"2023-10-16T12:54:15.663247+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:05.688392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873403+0000","last_change":"2023-10-16T12:54:12.849187+0000","last_active":"2023-10-16T12:54:14.873403+0000","last_peered":"2023-10-16T12:54:14.873403+0000","last_clean":"2023-10-16T12:54:14.873403+0000","last_became_active":"2023-10-16T12:54:12.848583+0000","last_became_peered":"2023-10-16T12:54:12.848583+0000","last_unstale":"2023-10-16T12:54:14.873403+0000","last_undegraded":"2023-10-16T12:54:14.873403+0000","last_fullsized":"2023-10-16T12:54:14.873403+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:05.000460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872669+0000","last_change":"2023-10-16T12:54:13.899751+0000","last_active":"2023-10-16T12:54:14.872669+0000","last_peered":"2023-10-16T12:54:14.872669+0000","last_clean":"2023-10-16T12:54:14.872669+0000","last_became_active":"2023-10-16T12:54:13.898612+0000","last_became_peered":"2023-10-16T12:54:13.898612+0000","last_unstale":"2023-10-16T12:54:14.872669+0000","last_undegraded":"2023-10-16T12:54:14.872669+0000","last_fullsized":"2023-10-16T12:54:14.872669+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:29.959281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872667+0000","last_change":"2023-10-16T12:54:12.867244+0000","last_active":"2023-10-16T12:54:14.872667+0000","last_peered":"2023-10-16T12:54:14.872667+0000","last_clean":"2023-10-16T12:54:14.872667+0000","last_became_active":"2023-10-16T12:54:12.866979+0000","last_became_peered":"2023-10-16T12:54:12.866979+0000","last_unstale":"2023-10-16T12:54:14.872667+0000","last_undegraded":"2023-10-16T12:54:14.872667+0000","last_fullsized":"2023-10-16T12:54:14.872667+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:54.808266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549358+0000","last_change":"2023-10-16T12:54:13.897060+0000","last_active":"2023-10-16T12:54:15.549358+0000","last_peered":"2023-10-16T12:54:15.549358+0000","last_clean":"2023-10-16T12:54:15.549358+0000","last_became_active":"2023-10-16T12:54:13.896796+0000","last_became_peered":"2023-10-16T12:54:13.896796+0000","last_unstale":"2023-10-16T12:54:15.549358+0000","last_undegraded":"2023-10-16T12:54:15.549358+0000","last_fullsized":"2023-10-16T12:54:15.549358+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:07.612504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875889+0000","last_change":"2023-10-16T12:54:12.866455+0000","last_active":"2023-10-16T12:54:14.875889+0000","last_peered":"2023-10-16T12:54:14.875889+0000","last_clean":"2023-10-16T12:54:14.875889+0000","last_became_active":"2023-10-16T12:54:12.866129+0000","last_became_peered":"2023-10-16T12:54:12.866129+0000","last_unstale":"2023-10-16T12:54:14.875889+0000","last_undegraded":"2023-10-16T12:54:14.875889+0000","last_fullsized":"2023-10-16T12:54:14.875889+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:09:50.976311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875187+0000","last_change":"2023-10-16T12:54:13.902533+0000","last_active":"2023-10-16T12:54:14.875187+0000","last_peered":"2023-10-16T12:54:14.875187+0000","last_clean":"2023-10-16T12:54:14.875187+0000","last_became_active":"2023-10-16T12:54:13.902261+0000","last_became_peered":"2023-10-16T12:54:13.902261+0000","last_unstale":"2023-10-16T12:54:14.875187+0000","last_undegraded":"2023-10-16T12:54:14.875187+0000","last_fullsized":"2023-10-16T12:54:14.875187+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:07.347612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665431+0000","last_change":"2023-10-16T12:54:12.863681+0000","last_active":"2023-10-16T12:54:15.665431+0000","last_peered":"2023-10-16T12:54:15.665431+0000","last_clean":"2023-10-16T12:54:15.665431+0000","last_became_active":"2023-10-16T12:54:12.863383+0000","last_became_peered":"2023-10-16T12:54:12.863383+0000","last_unstale":"2023-10-16T12:54:15.665431+0000","last_undegraded":"2023-10-16T12:54:15.665431+0000","last_fullsized":"2023-10-16T12:54:15.665431+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:43.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634785+0000","last_change":"2023-10-16T12:54:13.903493+0000","last_active":"2023-10-16T12:54:15.634785+0000","last_peered":"2023-10-16T12:54:15.634785+0000","last_clean":"2023-10-16T12:54:15.634785+0000","last_became_active":"2023-10-16T12:54:13.901959+0000","last_became_peered":"2023-10-16T12:54:13.901959+0000","last_unstale":"2023-10-16T12:54:15.634785+0000","last_undegraded":"2023-10-16T12:54:15.634785+0000","last_fullsized":"2023-10-16T12:54:15.634785+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:39.219237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"155'3","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874812+0000","last_change":"2023-10-16T12:54:12.860597+0000","last_active":"2023-10-16T12:54:14.874812+0000","last_peered":"2023-10-16T12:54:14.874812+0000","last_clean":"2023-10-16T12:54:14.874812+0000","last_became_active":"2023-10-16T12:54:12.860293+0000","last_became_peered":"2023-10-16T12:54:12.860293+0000","last_unstale":"2023-10-16T12:54:14.874812+0000","last_undegraded":"2023-10-16T12:54:14.874812+0000","last_fullsized":"2023-10-16T12:54:14.874812+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:17.402238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634211+0000","last_change":"2023-10-16T12:54:13.903346+0000","last_active":"2023-10-16T12:54:15.634211+0000","last_peered":"2023-10-16T12:54:15.634211+0000","last_clean":"2023-10-16T12:54:15.634211+0000","last_became_active":"2023-10-16T12:54:13.903076+0000","last_became_peered":"2023-10-16T12:54:13.903076+0000","last_unstale":"2023-10-16T12:54:15.634211+0000","last_undegraded":"2023-10-16T12:54:15.634211+0000","last_fullsized":"2023-10-16T12:54:15.634211+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:46.018150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"155'6","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663660+0000","last_change":"2023-10-16T12:54:12.861976+0000","last_active":"2023-10-16T12:54:15.663660+0000","last_peered":"2023-10-16T12:54:15.663660+0000","last_clean":"2023-10-16T12:54:15.663660+0000","last_became_active":"2023-10-16T12:54:12.861378+0000","last_became_peered":"2023-10-16T12:54:12.861378+0000","last_unstale":"2023-10-16T12:54:15.663660+0000","last_undegraded":"2023-10-16T12:54:15.663660+0000","last_fullsized":"2023-10-16T12:54:15.663660+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:04.148703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548943+0000","last_change":"2023-10-16T12:54:13.895425+0000","last_active":"2023-10-16T12:54:15.548943+0000","last_peered":"2023-10-16T12:54:15.548943+0000","last_clean":"2023-10-16T12:54:15.548943+0000","last_became_active":"2023-10-16T12:54:13.891264+0000","last_became_peered":"2023-10-16T12:54:13.891264+0000","last_unstale":"2023-10-16T12:54:15.548943+0000","last_undegraded":"2023-10-16T12:54:15.548943+0000","last_fullsized":"2023-10-16T12:54:15.548943+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:30.600664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548857+0000","last_change":"2023-10-16T12:54:12.847293+0000","last_active":"2023-10-16T12:54:15.548857+0000","last_peered":"2023-10-16T12:54:15.548857+0000","last_clean":"2023-10-16T12:54:15.548857+0000","last_became_active":"2023-10-16T12:54:12.846550+0000","last_became_peered":"2023-10-16T12:54:12.846550+0000","last_unstale":"2023-10-16T12:54:15.548857+0000","last_undegraded":"2023-10-16T12:54:15.548857+0000","last_fullsized":"2023-10-16T12:54:15.548857+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:20.224512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872639+0000","last_change":"2023-10-16T12:54:13.897999+0000","last_active":"2023-10-16T12:54:14.872639+0000","last_peered":"2023-10-16T12:54:14.872639+0000","last_clean":"2023-10-16T12:54:14.872639+0000","last_became_active":"2023-10-16T12:54:13.896966+0000","last_became_peered":"2023-10-16T12:54:13.896966+0000","last_unstale":"2023-10-16T12:54:14.872639+0000","last_undegraded":"2023-10-16T12:54:14.872639+0000","last_fullsized":"2023-10-16T12:54:14.872639+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:05.337355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872786+0000","last_change":"2023-10-16T12:54:12.862503+0000","last_active":"2023-10-16T12:54:14.872786+0000","last_peered":"2023-10-16T12:54:14.872786+0000","last_clean":"2023-10-16T12:54:14.872786+0000","last_became_active":"2023-10-16T12:54:12.862227+0000","last_became_peered":"2023-10-16T12:54:12.862227+0000","last_unstale":"2023-10-16T12:54:14.872786+0000","last_undegraded":"2023-10-16T12:54:14.872786+0000","last_fullsized":"2023-10-16T12:54:14.872786+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:20.433874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663063+0000","last_change":"2023-10-16T12:54:13.901125+0000","last_active":"2023-10-16T12:54:15.663063+0000","last_peered":"2023-10-16T12:54:15.663063+0000","last_clean":"2023-10-16T12:54:15.663063+0000","last_became_active":"2023-10-16T12:54:13.900799+0000","last_became_peered":"2023-10-16T12:54:13.900799+0000","last_unstale":"2023-10-16T12:54:15.663063+0000","last_undegraded":"2023-10-16T12:54:15.663063+0000","last_fullsized":"2023-10-16T12:54:15.663063+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:37.513352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"155'8","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663299+0000","last_change":"2023-10-16T12:54:12.862441+0000","last_active":"2023-10-16T12:54:15.663299+0000","last_peered":"2023-10-16T12:54:15.663299+0000","last_clean":"2023-10-16T12:54:15.663299+0000","last_became_active":"2023-10-16T12:54:12.862131+0000","last_became_peered":"2023-10-16T12:54:12.862131+0000","last_unstale":"2023-10-16T12:54:15.663299+0000","last_undegraded":"2023-10-16T12:54:15.663299+0000","last_fullsized":"2023-10-16T12:54:15.663299+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:16.982789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663401+0000","last_change":"2023-10-16T12:54:13.907633+0000","last_active":"2023-10-16T12:54:15.663401+0000","last_peered":"2023-10-16T12:54:15.663401+0000","last_clean":"2023-10-16T12:54:15.663401+0000","last_became_active":"2023-10-16T12:54:13.907320+0000","last_became_peered":"2023-10-16T12:54:13.907320+0000","last_unstale":"2023-10-16T12:54:15.663401+0000","last_undegraded":"2023-10-16T12:54:15.663401+0000","last_fullsized":"2023-10-16T12:54:15.663401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:28.250703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876096+0000","last_change":"2023-10-16T12:54:12.862311+0000","last_active":"2023-10-16T12:54:14.876096+0000","last_peered":"2023-10-16T12:54:14.876096+0000","last_clean":"2023-10-16T12:54:14.876096+0000","last_became_active":"2023-10-16T12:54:12.861788+0000","last_became_peered":"2023-10-16T12:54:12.861788+0000","last_unstale":"2023-10-16T12:54:14.876096+0000","last_undegraded":"2023-10-16T12:54:14.876096+0000","last_fullsized":"2023-10-16T12:54:14.876096+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:39.454603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875456+0000","last_change":"2023-10-16T12:54:13.881336+0000","last_active":"2023-10-16T12:54:14.875456+0000","last_peered":"2023-10-16T12:54:14.875456+0000","last_clean":"2023-10-16T12:54:14.875456+0000","last_became_active":"2023-10-16T12:54:13.880741+0000","last_became_peered":"2023-10-16T12:54:13.880741+0000","last_unstale":"2023-10-16T12:54:14.875456+0000","last_undegraded":"2023-10-16T12:54:14.875456+0000","last_fullsized":"2023-10-16T12:54:14.875456+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:22.470884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.663013+0000","last_change":"2023-10-16T12:54:12.860998+0000","last_active":"2023-10-16T12:54:15.663013+0000","last_peered":"2023-10-16T12:54:15.663013+0000","last_clean":"2023-10-16T12:54:15.663013+0000","last_became_active":"2023-10-16T12:54:12.859512+0000","last_became_peered":"2023-10-16T12:54:12.859512+0000","last_unstale":"2023-10-16T12:54:15.663013+0000","last_undegraded":"2023-10-16T12:54:15.663013+0000","last_fullsized":"2023-10-16T12:54:15.663013+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:06.597185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.665533+0000","last_change":"2023-10-16T12:54:13.910041+0000","last_active":"2023-10-16T12:54:15.665533+0000","last_peered":"2023-10-16T12:54:15.665533+0000","last_clean":"2023-10-16T12:54:15.665533+0000","last_became_active":"2023-10-16T12:54:13.909760+0000","last_became_peered":"2023-10-16T12:54:13.909760+0000","last_unstale":"2023-10-16T12:54:15.665533+0000","last_undegraded":"2023-10-16T12:54:15.665533+0000","last_fullsized":"2023-10-16T12:54:15.665533+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:57.083744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"155'1","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876447+0000","last_change":"2023-10-16T12:54:12.844521+0000","last_active":"2023-10-16T12:54:14.876447+0000","last_peered":"2023-10-16T12:54:14.876447+0000","last_clean":"2023-10-16T12:54:14.876447+0000","last_became_active":"2023-10-16T12:54:12.841965+0000","last_became_peered":"2023-10-16T12:54:12.841965+0000","last_unstale":"2023-10-16T12:54:14.876447+0000","last_undegraded":"2023-10-16T12:54:14.876447+0000","last_fullsized":"2023-10-16T12:54:14.876447+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:06:54.671940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874063+0000","last_change":"2023-10-16T12:54:13.900467+0000","last_active":"2023-10-16T12:54:14.874063+0000","last_peered":"2023-10-16T12:54:14.874063+0000","last_clean":"2023-10-16T12:54:14.874063+0000","last_became_active":"2023-10-16T12:54:13.898704+0000","last_became_peered":"2023-10-16T12:54:13.898704+0000","last_unstale":"2023-10-16T12:54:14.874063+0000","last_undegraded":"2023-10-16T12:54:14.874063+0000","last_fullsized":"2023-10-16T12:54:14.874063+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:53.813728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874708+0000","last_change":"2023-10-16T12:54:12.860177+0000","last_active":"2023-10-16T12:54:14.874708+0000","last_peered":"2023-10-16T12:54:14.874708+0000","last_clean":"2023-10-16T12:54:14.874708+0000","last_became_active":"2023-10-16T12:54:12.859521+0000","last_became_peered":"2023-10-16T12:54:12.859521+0000","last_unstale":"2023-10-16T12:54:14.874708+0000","last_undegraded":"2023-10-16T12:54:14.874708+0000","last_fullsized":"2023-10-16T12:54:14.874708+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:44.660490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.548499+0000","last_change":"2023-10-16T12:54:13.896656+0000","last_active":"2023-10-16T12:54:15.548499+0000","last_peered":"2023-10-16T12:54:15.548499+0000","last_clean":"2023-10-16T12:54:15.548499+0000","last_became_active":"2023-10-16T12:54:13.896393+0000","last_became_peered":"2023-10-16T12:54:13.896393+0000","last_unstale":"2023-10-16T12:54:15.548499+0000","last_undegraded":"2023-10-16T12:54:15.548499+0000","last_fullsized":"2023-10-16T12:54:15.548499+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:09.473188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.875173+0000","last_change":"2023-10-16T12:54:12.866461+0000","last_active":"2023-10-16T12:54:14.875173+0000","last_peered":"2023-10-16T12:54:14.875173+0000","last_clean":"2023-10-16T12:54:14.875173+0000","last_became_active":"2023-10-16T12:54:12.866126+0000","last_became_peered":"2023-10-16T12:54:12.866126+0000","last_unstale":"2023-10-16T12:54:14.875173+0000","last_undegraded":"2023-10-16T12:54:14.875173+0000","last_fullsized":"2023-10-16T12:54:14.875173+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:01.655277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876625+0000","last_change":"2023-10-16T12:54:13.893747+0000","last_active":"2023-10-16T12:54:14.876625+0000","last_peered":"2023-10-16T12:54:14.876625+0000","last_clean":"2023-10-16T12:54:14.876625+0000","last_became_active":"2023-10-16T12:54:13.893447+0000","last_became_peered":"2023-10-16T12:54:13.893447+0000","last_unstale":"2023-10-16T12:54:14.876625+0000","last_undegraded":"2023-10-16T12:54:14.876625+0000","last_fullsized":"2023-10-16T12:54:14.876625+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:40.469922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664337+0000","last_change":"2023-10-16T12:54:12.858930+0000","last_active":"2023-10-16T12:54:15.664337+0000","last_peered":"2023-10-16T12:54:15.664337+0000","last_clean":"2023-10-16T12:54:15.664337+0000","last_became_active":"2023-10-16T12:54:12.858384+0000","last_became_peered":"2023-10-16T12:54:12.858384+0000","last_unstale":"2023-10-16T12:54:15.664337+0000","last_undegraded":"2023-10-16T12:54:15.664337+0000","last_fullsized":"2023-10-16T12:54:15.664337+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:57.116437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872513+0000","last_change":"2023-10-16T12:54:13.889068+0000","last_active":"2023-10-16T12:54:14.872513+0000","last_peered":"2023-10-16T12:54:14.872513+0000","last_clean":"2023-10-16T12:54:14.872513+0000","last_became_active":"2023-10-16T12:54:13.884669+0000","last_became_peered":"2023-10-16T12:54:13.884669+0000","last_unstale":"2023-10-16T12:54:14.872513+0000","last_undegraded":"2023-10-16T12:54:14.872513+0000","last_fullsized":"2023-10-16T12:54:14.872513+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:55.120581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664039+0000","last_change":"2023-10-16T12:54:12.860130+0000","last_active":"2023-10-16T12:54:15.664039+0000","last_peered":"2023-10-16T12:54:15.664039+0000","last_clean":"2023-10-16T12:54:15.664039+0000","last_became_active":"2023-10-16T12:54:12.859824+0000","last_became_peered":"2023-10-16T12:54:12.859824+0000","last_unstale":"2023-10-16T12:54:15.664039+0000","last_undegraded":"2023-10-16T12:54:15.664039+0000","last_fullsized":"2023-10-16T12:54:15.664039+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:09.544063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633666+0000","last_change":"2023-10-16T12:54:13.898363+0000","last_active":"2023-10-16T12:54:15.633666+0000","last_peered":"2023-10-16T12:54:15.633666+0000","last_clean":"2023-10-16T12:54:15.633666+0000","last_became_active":"2023-10-16T12:54:13.897793+0000","last_became_peered":"2023-10-16T12:54:13.897793+0000","last_unstale":"2023-10-16T12:54:15.633666+0000","last_undegraded":"2023-10-16T12:54:15.633666+0000","last_fullsized":"2023-10-16T12:54:15.633666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:50.613200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662106+0000","last_change":"2023-10-16T12:54:12.861854+0000","last_active":"2023-10-16T12:54:15.662106+0000","last_peered":"2023-10-16T12:54:15.662106+0000","last_clean":"2023-10-16T12:54:15.662106+0000","last_became_active":"2023-10-16T12:54:12.859335+0000","last_became_peered":"2023-10-16T12:54:12.859335+0000","last_unstale":"2023-10-16T12:54:15.662106+0000","last_undegraded":"2023-10-16T12:54:15.662106+0000","last_fullsized":"2023-10-16T12:54:15.662106+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:08.457869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876752+0000","last_change":"2023-10-16T12:54:13.881587+0000","last_active":"2023-10-16T12:54:14.876752+0000","last_peered":"2023-10-16T12:54:14.876752+0000","last_clean":"2023-10-16T12:54:14.876752+0000","last_became_active":"2023-10-16T12:54:13.879074+0000","last_became_peered":"2023-10-16T12:54:13.879074+0000","last_unstale":"2023-10-16T12:54:14.876752+0000","last_undegraded":"2023-10-16T12:54:14.876752+0000","last_fullsized":"2023-10-16T12:54:14.876752+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:50.206683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549669+0000","last_change":"2023-10-16T12:54:12.861573+0000","last_active":"2023-10-16T12:54:15.549669+0000","last_peered":"2023-10-16T12:54:15.549669+0000","last_clean":"2023-10-16T12:54:15.549669+0000","last_became_active":"2023-10-16T12:54:12.861304+0000","last_became_peered":"2023-10-16T12:54:12.861304+0000","last_unstale":"2023-10-16T12:54:15.549669+0000","last_undegraded":"2023-10-16T12:54:15.549669+0000","last_fullsized":"2023-10-16T12:54:15.549669+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:09.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.549628+0000","last_change":"2023-10-16T12:54:13.906250+0000","last_active":"2023-10-16T12:54:15.549628+0000","last_peered":"2023-10-16T12:54:15.549628+0000","last_clean":"2023-10-16T12:54:15.549628+0000","last_became_active":"2023-10-16T12:54:13.905848+0000","last_became_peered":"2023-10-16T12:54:13.905848+0000","last_unstale":"2023-10-16T12:54:15.549628+0000","last_undegraded":"2023-10-16T12:54:15.549628+0000","last_fullsized":"2023-10-16T12:54:15.549628+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:05.337355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633988+0000","last_change":"2023-10-16T12:54:12.855590+0000","last_active":"2023-10-16T12:54:15.633988+0000","last_peered":"2023-10-16T12:54:15.633988+0000","last_clean":"2023-10-16T12:54:15.633988+0000","last_became_active":"2023-10-16T12:54:12.855237+0000","last_became_peered":"2023-10-16T12:54:12.855237+0000","last_unstale":"2023-10-16T12:54:15.633988+0000","last_undegraded":"2023-10-16T12:54:15.633988+0000","last_fullsized":"2023-10-16T12:54:15.633988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:34.210059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873643+0000","last_change":"2023-10-16T12:54:13.893769+0000","last_active":"2023-10-16T12:54:14.873643+0000","last_peered":"2023-10-16T12:54:14.873643+0000","last_clean":"2023-10-16T12:54:14.873643+0000","last_became_active":"2023-10-16T12:54:13.893424+0000","last_became_peered":"2023-10-16T12:54:13.893424+0000","last_unstale":"2023-10-16T12:54:14.873643+0000","last_undegraded":"2023-10-16T12:54:14.873643+0000","last_fullsized":"2023-10-16T12:54:14.873643+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:12.558438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874056+0000","last_change":"2023-10-16T12:54:12.866418+0000","last_active":"2023-10-16T12:54:14.874056+0000","last_peered":"2023-10-16T12:54:14.874056+0000","last_clean":"2023-10-16T12:54:14.874056+0000","last_became_active":"2023-10-16T12:54:12.865909+0000","last_became_peered":"2023-10-16T12:54:12.865909+0000","last_unstale":"2023-10-16T12:54:14.874056+0000","last_undegraded":"2023-10-16T12:54:14.874056+0000","last_fullsized":"2023-10-16T12:54:14.874056+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:09.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.876298+0000","last_change":"2023-10-16T12:54:13.894982+0000","last_active":"2023-10-16T12:54:14.876298+0000","last_peered":"2023-10-16T12:54:14.876298+0000","last_clean":"2023-10-16T12:54:14.876298+0000","last_became_active":"2023-10-16T12:54:13.894710+0000","last_became_peered":"2023-10-16T12:54:13.894710+0000","last_unstale":"2023-10-16T12:54:14.876298+0000","last_undegraded":"2023-10-16T12:54:14.876298+0000","last_fullsized":"2023-10-16T12:54:14.876298+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:45.675810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662634+0000","last_change":"2023-10-16T12:54:12.856151+0000","last_active":"2023-10-16T12:54:15.662634+0000","last_peered":"2023-10-16T12:54:15.662634+0000","last_clean":"2023-10-16T12:54:15.662634+0000","last_became_active":"2023-10-16T12:54:12.855854+0000","last_became_peered":"2023-10-16T12:54:12.855854+0000","last_unstale":"2023-10-16T12:54:15.662634+0000","last_undegraded":"2023-10-16T12:54:15.662634+0000","last_fullsized":"2023-10-16T12:54:15.662634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:29.585345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633744+0000","last_change":"2023-10-16T12:54:13.894010+0000","last_active":"2023-10-16T12:54:15.633744+0000","last_peered":"2023-10-16T12:54:15.633744+0000","last_clean":"2023-10-16T12:54:15.633744+0000","last_became_active":"2023-10-16T12:54:13.892969+0000","last_became_peered":"2023-10-16T12:54:13.892969+0000","last_unstale":"2023-10-16T12:54:15.633744+0000","last_undegraded":"2023-10-16T12:54:15.633744+0000","last_fullsized":"2023-10-16T12:54:15.633744+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:18.885935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.873579+0000","last_change":"2023-10-16T12:54:12.862944+0000","last_active":"2023-10-16T12:54:14.873579+0000","last_peered":"2023-10-16T12:54:14.873579+0000","last_clean":"2023-10-16T12:54:14.873579+0000","last_became_active":"2023-10-16T12:54:12.862588+0000","last_became_peered":"2023-10-16T12:54:12.862588+0000","last_unstale":"2023-10-16T12:54:14.873579+0000","last_undegraded":"2023-10-16T12:54:14.873579+0000","last_fullsized":"2023-10-16T12:54:14.873579+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:29.585345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.874941+0000","last_change":"2023-10-16T12:54:13.895152+0000","last_active":"2023-10-16T12:54:14.874941+0000","last_peered":"2023-10-16T12:54:14.874941+0000","last_clean":"2023-10-16T12:54:14.874941+0000","last_became_active":"2023-10-16T12:54:13.894850+0000","last_became_peered":"2023-10-16T12:54:13.894850+0000","last_unstale":"2023-10-16T12:54:14.874941+0000","last_undegraded":"2023-10-16T12:54:14.874941+0000","last_fullsized":"2023-10-16T12:54:14.874941+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:07:50.289014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633880+0000","last_change":"2023-10-16T12:54:12.857963+0000","last_active":"2023-10-16T12:54:15.633880+0000","last_peered":"2023-10-16T12:54:15.633880+0000","last_clean":"2023-10-16T12:54:15.633880+0000","last_became_active":"2023-10-16T12:54:12.857683+0000","last_became_peered":"2023-10-16T12:54:12.857683+0000","last_unstale":"2023-10-16T12:54:15.633880+0000","last_undegraded":"2023-10-16T12:54:15.633880+0000","last_fullsized":"2023-10-16T12:54:15.633880+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:36.261813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664587+0000","last_change":"2023-10-16T12:54:13.906861+0000","last_active":"2023-10-16T12:54:15.664587+0000","last_peered":"2023-10-16T12:54:15.664587+0000","last_clean":"2023-10-16T12:54:15.664587+0000","last_became_active":"2023-10-16T12:54:13.905013+0000","last_became_peered":"2023-10-16T12:54:13.905013+0000","last_unstale":"2023-10-16T12:54:15.664587+0000","last_undegraded":"2023-10-16T12:54:15.664587+0000","last_fullsized":"2023-10-16T12:54:15.664587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:33:56.235900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.664317+0000","last_change":"2023-10-16T12:54:12.857638+0000","last_active":"2023-10-16T12:54:15.664317+0000","last_peered":"2023-10-16T12:54:15.664317+0000","last_clean":"2023-10-16T12:54:15.664317+0000","last_became_active":"2023-10-16T12:54:12.857295+0000","last_became_peered":"2023-10-16T12:54:12.857295+0000","last_unstale":"2023-10-16T12:54:15.664317+0000","last_undegraded":"2023-10-16T12:54:15.664317+0000","last_fullsized":"2023-10-16T12:54:15.664317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.689317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:14.872668+0000","last_change":"2023-10-16T12:54:13.892390+0000","last_active":"2023-10-16T12:54:14.872668+0000","last_peered":"2023-10-16T12:54:14.872668+0000","last_clean":"2023-10-16T12:54:14.872668+0000","last_became_active":"2023-10-16T12:54:13.892116+0000","last_became_peered":"2023-10-16T12:54:13.892116+0000","last_unstale":"2023-10-16T12:54:14.872668+0000","last_undegraded":"2023-10-16T12:54:14.872668+0000","last_fullsized":"2023-10-16T12:54:14.872668+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:22.637464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.662524+0000","last_change":"2023-10-16T12:54:12.854987+0000","last_active":"2023-10-16T12:54:15.662524+0000","last_peered":"2023-10-16T12:54:15.662524+0000","last_clean":"2023-10-16T12:54:15.662524+0000","last_became_active":"2023-10-16T12:54:12.853280+0000","last_became_peered":"2023-10-16T12:54:12.853280+0000","last_unstale":"2023-10-16T12:54:15.662524+0000","last_undegraded":"2023-10-16T12:54:15.662524+0000","last_fullsized":"2023-10-16T12:54:15.662524+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:54.808266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.633666+0000","last_change":"2023-10-16T12:54:13.903906+0000","last_active":"2023-10-16T12:54:15.633666+0000","last_peered":"2023-10-16T12:54:15.633666+0000","last_clean":"2023-10-16T12:54:15.633666+0000","last_became_active":"2023-10-16T12:54:13.902720+0000","last_became_peered":"2023-10-16T12:54:13.902720+0000","last_unstale":"2023-10-16T12:54:15.633666+0000","last_undegraded":"2023-10-16T12:54:15.633666+0000","last_fullsized":"2023-10-16T12:54:15.633666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:12.818372+0000","last_clean_scrub_stamp":"2023-10-16T12:54:12.818372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:40.959584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"155'2","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-16T12:54:15.634333+0000","last_change":"2023-10-16T12:54:12.864817+0000","last_active":"2023-10-16T12:54:15.634333+0000","last_peered":"2023-10-16T12:54:15.634333+0000","last_clean":"2023-10-16T12:54:15.634333+0000","last_became_active":"2023-10-16T12:54:12.864438+0000","last_became_peered":"2023-10-16T12:54:12.864438+0000","last_unstale":"2023-10-16T12:54:15.634333+0000","last_undegraded":"2023-10-16T12:54:15.634333+0000","last_fullsized":"2023-10-16T12:54:15.634333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:54:11.803052+0000","last_clean_scrub_stamp":"2023-10-16T12:54:11.803052+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:41.985480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"145'252","reported_seq":770,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:54:13.878475+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T12:54:13.878475+0000","last_peered":"2023-10-16T12:54:13.878475+0000","last_clean":"2023-10-16T12:54:13.878475+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T12:54:13.878475+0000","last_undegraded":"2023-10-16T12:54:13.878475+0000","last_fullsized":"2023-10-16T12:54:13.878475+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":2654,"num_write":403,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542390,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33668,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338172,"statfs":{"total":96636764160,"available":96602288128,"internally_reserved":0,"allocated":2543616,"data_stored":1515739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.278,"15min":1.3009999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.7809999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.022},{"interface":"front","average":{"1min":1.218,"5min":1.2969999999999999,"15min":1.3400000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.63200000000000001},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8899999999999999},"last":1.1359999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.3720000000000001,"15min":1.266},"min":{"1min":0.83899999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.647,"5min":19.158999999999999,"15min":19.158999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.222,"15min":1.2190000000000001},"min":{"1min":0.81299999999999994,"5min":0.64800000000000002,"15min":0.64700000000000002},"max":{"1min":1.5249999999999999,"5min":5.5490000000000004,"15min":5.5490000000000004},"last":1.633}]},{"osd":2,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.135,"15min":1.181},"min":{"1min":0.70099999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.4830000000000001,"5min":2.855,"15min":2.855},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2789999999999999,"15min":1.3140000000000001},"min":{"1min":0.85599999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.8009999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.538}]},{"osd":3,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.343,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6919999999999999,"5min":3.2759999999999998,"15min":3.2759999999999998},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.361,"15min":1.3859999999999999},"min":{"1min":1.0920000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.823,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.2629999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2090000000000001,"15min":1.1910000000000001},"min":{"1min":0.77400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.0670000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.391,"15min":1.365},"min":{"1min":0.96399999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.597}]},{"osd":5,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3700000000000001,"15min":1.3420000000000001},"min":{"1min":1.01,"5min":0.624,"15min":0.624},"max":{"1min":2.0529999999999999,"5min":5.0140000000000002,"15min":5.0140000000000002},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.5569999999999999,"15min":1.498},"min":{"1min":1.2989999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.3170000000000002,"5min":2.411,"15min":2.411},"last":1.8009999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.6339999999999999,"15min":1.591},"min":{"1min":1.341,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.234,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.712},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.492,"15min":1.462},"min":{"1min":1.0469999999999999,"5min":0.75,"15min":0.75},"max":{"1min":1.996,"5min":3.1360000000000001,"15min":3.1360000000000001},"last":1.7549999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542391,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1949696,"data_stored":931033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.2090000000000001,"15min":1.258},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6060000000000001,"5min":1.952,"15min":1.952},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.254,"15min":1.306},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.589,"5min":1.9399999999999999,"15min":2.0270000000000001},"last":0.89600000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1220000000000001,"15min":1.177},"min":{"1min":0.68000000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.5129999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.204,"15min":1.2050000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7949999999999999,"5min":1.9330000000000001,"15min":2.0459999999999998},"last":0.83799999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2030000000000001,"15min":1.2130000000000001},"min":{"1min":0.746,"5min":0.66900000000000004,"15min":0.627},"max":{"1min":1.631,"5min":1.752,"15min":1.8899999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.206,"15min":1.254},"min":{"1min":0.89300000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.482,"5min":1.8520000000000001,"15min":1.9970000000000001},"last":0.70899999999999996}]},{"osd":3,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2849999999999999,"15min":1.3080000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.9359999999999999,"5min":2.0510000000000002,"15min":2.3900000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.329,"15min":1.333},"min":{"1min":0.84499999999999997,"5min":0.82199999999999995,"15min":0.69599999999999995},"max":{"1min":1.637,"5min":1.806,"15min":2.012},"last":1.143}]},{"osd":4,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3799999999999999,"15min":1.349},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71099999999999997},"max":{"1min":2.0920000000000001,"5min":2.0990000000000002,"15min":2.3029999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4710000000000001,"15min":1.468},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.77300000000000002},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.2250000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3129999999999999,"15min":1.3089999999999999},"min":{"1min":0.81899999999999995,"5min":0.755,"15min":0.755},"max":{"1min":1.873,"5min":1.9830000000000001,"15min":2.0910000000000002},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.351,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.78300000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":2.157,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.1859999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4470000000000001,"15min":1.4370000000000001},"min":{"1min":1.0009999999999999,"5min":0.91300000000000003,"15min":0.878},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.494,"15min":1.47},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.96399999999999997},"max":{"1min":2.214,"5min":2.214,"15min":2.2610000000000001},"last":1.099}]}]},{"osd":5,"up_from":14,"seq":60129542392,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":929533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0980000000000001,"15min":1.1439999999999999},"min":{"1min":0.64000000000000001,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.621,"5min":1.8480000000000001,"15min":2.2690000000000001},"last":1.345},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1779999999999999,"15min":1.1739999999999999},"min":{"1min":0.84099999999999997,"5min":0.65200000000000002,"15min":0.63300000000000001},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":1.403}]},{"osd":1,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.256,"15min":1.25},"min":{"1min":0.67200000000000004,"5min":0.56699999999999995,"15min":0.41299999999999998},"max":{"1min":1.6120000000000001,"5min":1.8080000000000001,"15min":1.8089999999999999},"last":1.478},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.1970000000000001,"15min":1.2070000000000001},"min":{"1min":0.95699999999999996,"5min":0.46000000000000002,"15min":0.41599999999999998},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.264,"15min":1.274},"min":{"1min":0.89200000000000002,"5min":0.60799999999999998,"15min":0.60199999999999998},"max":{"1min":1.881,"5min":1.9710000000000001,"15min":2.0059999999999998},"last":1.099},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.204,"15min":1.26},"min":{"1min":0.93500000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4610000000000001,"5min":1.9870000000000001,"15min":2.2370000000000001},"last":1.5069999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2809999999999999,"15min":1.304},"min":{"1min":0.78400000000000003,"5min":0.73099999999999998,"15min":0.66000000000000003},"max":{"1min":1.776,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3500000000000001,"15min":1.361},"min":{"1min":0.89700000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.601,"5min":2.1150000000000002,"15min":2.2160000000000002},"last":1.444}]},{"osd":4,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.25,"15min":1.2090000000000001},"min":{"1min":0.83999999999999997,"5min":0.40200000000000002,"15min":0.39100000000000001},"max":{"1min":1.5269999999999999,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.137},{"interface":"front","average":{"1min":1.321,"5min":1.3560000000000001,"15min":1.321},"min":{"1min":0.69899999999999995,"5min":0.54200000000000004,"15min":0.42999999999999999},"max":{"1min":1.6220000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.2410000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.381,"15min":1.3440000000000001},"min":{"1min":1.014,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6830000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.421,"15min":1.389},"min":{"1min":1.046,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.734,"5min":2.48,"15min":2.48},"last":1.5900000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4690000000000001,"15min":1.4550000000000001},"min":{"1min":1.1739999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.718,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.536},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5249999999999999,"15min":1.5109999999999999},"min":{"1min":1.115,"5min":0.64400000000000002,"15min":0.55300000000000005},"max":{"1min":1.98,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5589999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542392,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59448,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312392,"statfs":{"total":96636764160,"available":96575889408,"internally_reserved":0,"allocated":1941504,"data_stored":929491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1200000000000001,"15min":1.1870000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.623,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1100000000000001,"15min":1.1439999999999999},"min":{"1min":0.66900000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.4370000000000001,"5min":2.9060000000000001,"15min":2.9060000000000001},"last":1.4890000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.175,"15min":1.177},"min":{"1min":0.69499999999999995,"5min":0.38,"15min":0.38},"max":{"1min":1.6100000000000001,"5min":3.3180000000000001,"15min":3.3180000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.173,"15min":1.2390000000000001},"min":{"1min":0.73599999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.1579999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":1.671}]},{"osd":2,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.70299999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.623,"5min":4.3470000000000004,"15min":4.3470000000000004},"last":1.714},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2749999999999999,"15min":1.321},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9710000000000001,"5min":2.4670000000000001,"15min":2.4670000000000001},"last":1.5469999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2330000000000001,"15min":1.258},"min":{"1min":0.91000000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6559999999999999,"5min":2.8300000000000001,"15min":2.8300000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3640000000000001,"15min":1.3740000000000001},"min":{"1min":0.97199999999999998,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.8520000000000001,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":1.605}]},{"osd":5,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.355,"15min":1.3080000000000001},"min":{"1min":0.84499999999999997,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.2280000000000002,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.349,"15min":1.3029999999999999},"min":{"1min":0.98199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.851,"5min":4.391,"15min":4.391},"last":0.997}]},{"osd":6,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.395,"15min":1.375},"min":{"1min":1.0189999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.0600000000000001,"5min":3.4649999999999999,"15min":3.4649999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.4950000000000001,"15min":1.476},"min":{"1min":0.92000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.395,"5min":2.504,"15min":2.504},"last":1.355}]},{"osd":7,"last update":"Mon Oct 16 12:54:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.496,"15min":1.47},"min":{"1min":0.84099999999999997,"5min":0.67600000000000005,"15min":0.60399999999999998},"max":{"1min":2.0230000000000001,"5min":3.9369999999999998,"15min":3.9369999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5960000000000001,"15min":1.5860000000000001},"min":{"1min":1.131,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4460000000000002,"5min":3.262,"15min":3.262},"last":1.8160000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542392,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41388,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330452,"statfs":{"total":96636764160,"available":96594382848,"internally_reserved":0,"allocated":1933312,"data_stored":923281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.90200000000000002,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":1.3160000000000001,"5min":1.609,"15min":2.0419999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.02,"15min":1.141},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34799999999999998},"max":{"1min":1.661,"5min":1.661,"15min":2.2749999999999999},"last":1.0720000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.96599999999999997,"15min":1.0129999999999999},"min":{"1min":0.52700000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.96099999999999997,"15min":1.056},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.6899999999999999,"5min":1.784,"15min":2.4580000000000002},"last":1.2809999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.93100000000000005,"15min":1.044},"min":{"1min":0.56399999999999995,"5min":0.38,"15min":0.38},"max":{"1min":1.361,"5min":1.746,"15min":1.9510000000000001},"last":0.747},{"interface":"front","average":{"1min":1.145,"5min":1.216,"15min":1.3280000000000001},"min":{"1min":0.67300000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.865,"5min":2.044,"15min":2.044},"last":1.3160000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3620000000000001,"15min":1.462},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.294,"15min":1.389},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.677,"5min":1.8220000000000001,"15min":2.0099999999999998},"last":1.528}]},{"osd":5,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.258,"15min":1.3560000000000001},"min":{"1min":0.90200000000000002,"5min":0.70299999999999996,"15min":0.65900000000000003},"max":{"1min":1.5820000000000001,"5min":1.893,"15min":2.0539999999999998},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.399,"15min":1.456},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0049999999999999,"5min":2.0750000000000002,"15min":2.3540000000000001},"last":1.6299999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.365,"15min":1.4370000000000001},"min":{"1min":0.80700000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":1.696},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.3100000000000001,"15min":1.397},"min":{"1min":0.80200000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8320000000000001,"5min":2,"15min":2.3740000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:54:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4610000000000001,"15min":1.5289999999999999},"min":{"1min":0.88500000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.081,"5min":2.157,"15min":2.431},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4770000000000001,"15min":1.583},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.5760000000000001,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.585}]}]},{"osd":2,"up_from":14,"seq":60129542390,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46972,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324868,"statfs":{"total":96636764160,"available":96588664832,"internally_reserved":0,"allocated":1945600,"data_stored":930037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.876,"15min":0.91200000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.7889999999999999},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0209999999999999,"15min":1.0720000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5429999999999999,"5min":1.746,"15min":2.085},"last":0.71799999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.069,"15min":1.1180000000000001},"min":{"1min":0.442,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.591,"5min":1.677,"15min":2.5590000000000002},"last":1.155},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1539999999999999,"15min":1.2110000000000001},"min":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.46500000000000002},"max":{"1min":1.655,"5min":2.0510000000000002,"15min":2.4289999999999998},"last":1.232}]},{"osd":3,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0409999999999999,"15min":1.0760000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3740000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":1.46},{"interface":"front","average":{"1min":1.179,"5min":1.2230000000000001,"15min":1.3009999999999999},"min":{"1min":0.59799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.6020000000000001,"5min":2.238,"15min":2.238},"last":1.54}]},{"osd":4,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.367,"15min":1.4370000000000001},"min":{"1min":0.91700000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.734,"5min":2.2149999999999999,"15min":2.431},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.3580000000000001,"15min":1.411},"min":{"1min":0.874,"5min":0.75600000000000001,"15min":0.68999999999999995},"max":{"1min":1.859,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.484}]},{"osd":5,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.458,"15min":1.4950000000000001},"min":{"1min":0.79900000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.875,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.988},{"interface":"front","average":{"1min":1.294,"5min":1.3600000000000001,"15min":1.3999999999999999},"min":{"1min":0.875,"5min":0.751,"15min":0.52100000000000002},"max":{"1min":1.754,"5min":3.7759999999999998,"15min":3.7759999999999998},"last":1.7769999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4350000000000001,"15min":1.4430000000000001},"min":{"1min":0.90400000000000003,"5min":0.89400000000000002,"15min":0.59699999999999998},"max":{"1min":1.806,"5min":3.6120000000000001,"15min":3.6120000000000001},"last":1.657},{"interface":"front","average":{"1min":1.446,"5min":1.47,"15min":1.5389999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.78000000000000003},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":1.6819999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:53:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.534,"15min":1.5900000000000001},"min":{"1min":0.91900000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8580000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4730000000000001,"15min":1.5269999999999999},"min":{"1min":0.88900000000000001,"5min":0.81499999999999995,"15min":0.56599999999999995},"max":{"1min":1.913,"5min":3.73,"15min":3.73},"last":1.5629999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542390,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59448,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312392,"statfs":{"total":96636764160,"available":96575889408,"internally_reserved":0,"allocated":1937408,"data_stored":929421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.89800000000000002,"15min":0.90100000000000002},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.357,"5min":1.6359999999999999,"15min":2.0840000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.96199999999999997,"15min":1.054},"min":{"1min":0.43099999999999999,"5min":0.31,"15min":0.31},"max":{"1min":1.351,"5min":1.8500000000000001,"15min":1.9770000000000001},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.97599999999999998},"min":{"1min":0.501,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.5740000000000001,"5min":1.694,"15min":2.262},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1020000000000001,"15min":1.171},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.696,"5min":1.9910000000000001,"15min":2.2210000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.208,"15min":1.2909999999999999},"min":{"1min":0.499,"5min":0.47899999999999998,"15min":0.46100000000000002},"max":{"1min":1.8500000000000001,"5min":2.073,"15min":2.073},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.113,"5min":1.1299999999999999,"15min":1.2030000000000001},"min":{"1min":0.72199999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.597,"5min":1.758,"15min":1.9199999999999999},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2829999999999999},"min":{"1min":0.71399999999999997,"5min":0.59599999999999997,"15min":0.54600000000000004},"max":{"1min":1.536,"5min":1.7350000000000001,"15min":2.1030000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3979999999999999,"15min":1.4319999999999999},"min":{"1min":0.77600000000000002,"5min":0.67900000000000005,"15min":0.60599999999999998},"max":{"1min":1.9750000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.0760000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3759999999999999,"15min":1.413},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0649999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3640000000000001,"15min":1.4259999999999999},"min":{"1min":0.755,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.0049999999999999,"5min":2.1179999999999999,"15min":2.4350000000000001},"last":1.349}]},{"osd":6,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.478,"15min":1.52},"min":{"1min":0.749,"5min":0.72099999999999997,"15min":0.68999999999999995},"max":{"1min":2.0800000000000001,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4670000000000001,"15min":1.508},"min":{"1min":0.76800000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.2029999999999998,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Mon Oct 16 12:53:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":1.0229999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2749999999999999,"5min":2.383,"15min":2.383},"last":1.109},{"interface":"front","average":{"1min":1.504,"5min":1.49,"15min":1.548},"min":{"1min":0.80700000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.0539999999999998,"5min":2.1280000000000001,"15min":2.5710000000000002},"last":1.1539999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542391,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60024,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311816,"statfs":{"total":96636764160,"available":96575299584,"internally_reserved":0,"allocated":2531328,"data_stored":1519789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.81399999999999995,"15min":0.871},"min":{"1min":0.39000000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.085,"5min":1.655,"15min":1.6890000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.98899999999999999,"15min":1.0680000000000001},"min":{"1min":0.61099999999999999,"5min":0.441,"15min":0.441},"max":{"1min":1.601,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.506}]},{"osd":2,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.042,"15min":1.105},"min":{"1min":0.57799999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.304,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0209999999999999,"15min":1.135},"min":{"1min":0.5,"5min":0.497,"15min":0.496},"max":{"1min":1.47,"5min":1.788,"15min":1.952},"last":1.694}]},{"osd":3,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96599999999999997,"15min":1.0660000000000001},"min":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":1.516,"5min":1.8660000000000001,"15min":1.9570000000000001},"last":1},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.115,"15min":1.244},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.756,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":1.587}]},{"osd":4,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.224,"15min":1.3140000000000001},"min":{"1min":0.55500000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.597,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.337,"15min":1.409},"min":{"1min":0.85499999999999998,"5min":0.625,"15min":0.59699999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.173},"last":1.458}]},{"osd":5,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.331,"15min":1.3999999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7130000000000001,"5min":1.9630000000000001,"15min":2.2440000000000002},"last":1.909},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.298,"15min":1.3879999999999999},"min":{"1min":0.66400000000000003,"5min":0.63100000000000001,"15min":0.58999999999999997},"max":{"1min":1.6319999999999999,"5min":1.9970000000000001,"15min":2.0270000000000001},"last":1.4179999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.419,"15min":1.496},"min":{"1min":0.86699999999999999,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0670000000000002,"5min":2.125,"15min":2.3519999999999999},"last":2.0539999999999998},{"interface":"front","average":{"1min":1.286,"5min":1.3580000000000001,"15min":1.48},"min":{"1min":0.67600000000000005,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.8340000000000001,"5min":1.95,"15min":2.363},"last":2.1110000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:54:03 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.496,"15min":1.5960000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0019999999999998,"5min":2.395,"15min":2.4729999999999999},"last":2.379},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.4350000000000001,"15min":1.528},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.734,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.7509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:54:18.961 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:54:18.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:54:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.237+0000 7f739fe61640 1 Processor -- start 2023-10-16T12:54:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.237+0000 7f739fe61640 1 -- start start 2023-10-16T12:54:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739fe61640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73981593b0 0x7f739806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739fe61640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f73980cc8d0 0x7f73980ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739fe61640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980ca5d0 con 0x7f73980caa10 2023-10-16T12:54:19.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739fe61640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7398159790 con 0x7f73980cc8d0 2023-10-16T12:54:19.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739fe61640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73980c9a90 con 0x7f73981593b0 2023-10-16T12:54:19.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739e3fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f73980cc8d0 0x7f73980ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739e3fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f73980cc8d0 0x7f73980ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52708/0 (socket says 172.21.15.60:52708) 2023-10-16T12:54:19.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739e3fe640 1 -- 172.21.15.60:0/2132899742 learned_addr learned my addr 172.21.15.60:0/2132899742 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:19.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739dbfd640 1 --2- 172.21.15.60:0/2132899742 >> v2:172.21.15.144:3300/0 conn(0x7f73981593b0 0x7f739806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.239+0000 7f739e3fe640 1 -- 172.21.15.60:0/2132899742 >> v2:172.21.15.144:3300/0 conn(0x7f73981593b0 msgr2=0x7f739806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:19.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.240+0000 7f739e3fe640 1 --2- 172.21.15.60:0/2132899742 >> v2:172.21.15.144:3300/0 conn(0x7f73981593b0 0x7f739806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.240+0000 7f739e3fe640 1 -- 172.21.15.60:0/2132899742 >> v1:172.21.15.60:6789/0 conn(0x7f73980caa10 legacy=0x7f739806fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.240+0000 7f739e3fe640 1 -- 172.21.15.60:0/2132899742 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980c9d30 con 0x7f73980cc8d0 2023-10-16T12:54:19.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.240+0000 7f739dbfd640 1 --2- 172.21.15.60:0/2132899742 >> v2:172.21.15.144:3300/0 conn(0x7f73981593b0 0x7f739806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:19.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.241+0000 7f739e3fe640 1 --2- 172.21.15.60:0/2132899742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73980cc8d0 0x7f73980ccd60 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f73980ce8f0 tx=0x7f739002cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b07f388e1edeb2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:19.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.241+0000 7f739cbfb640 1 -- 172.21.15.60:0/2132899742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739002dba0 con 0x7f73980cc8d0 2023-10-16T12:54:19.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.241+0000 7f739cbfb640 1 -- 172.21.15.60:0/2132899742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739002dd00 con 0x7f73980cc8d0 2023-10-16T12:54:19.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.241+0000 7f739cbfb640 1 -- 172.21.15.60:0/2132899742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7390035660 con 0x7f73980cc8d0 2023-10-16T12:54:19.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 -- 172.21.15.60:0/2132899742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73980cc8d0 msgr2=0x7f73980ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 --2- 172.21.15.60:0/2132899742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73980cc8d0 0x7f73980ccd60 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f73980ce8f0 tx=0x7f739002cdf0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 -- 172.21.15.60:0/2132899742 shutdown_connections 2023-10-16T12:54:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 --2- 172.21.15.60:0/2132899742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73980cc8d0 0x7f73980ccd60 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 --2- 172.21.15.60:0/2132899742 >> v2:172.21.15.144:3300/0 conn(0x7f73981593b0 0x7f739806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 -- 172.21.15.60:0/2132899742 >> 172.21.15.60:0/2132899742 conn(0x7f73980cb410 msgr2=0x7f73980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:19.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 -- 172.21.15.60:0/2132899742 shutdown_connections 2023-10-16T12:54:19.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.242+0000 7f739fe61640 1 -- 172.21.15.60:0/2132899742 wait complete. 2023-10-16T12:54:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.244+0000 7f739fe61640 1 Processor -- start 2023-10-16T12:54:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.244+0000 7f739fe61640 1 -- start start 2023-10-16T12:54:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739fe61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73980caa10 0x7f7398154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739fe61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73980cc8d0 0x7f73980c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739fe61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 0x7f73980af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739fe61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73980b1590 con 0x7f73981593b0 2023-10-16T12:54:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739fe61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398154cc0 con 0x7f73980caa10 2023-10-16T12:54:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739fe61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980c83e0 con 0x7f73980cc8d0 2023-10-16T12:54:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 0x7f73980af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 0x7f73980af0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52714/0 (socket says 172.21.15.60:52714) 2023-10-16T12:54:19.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 -- 172.21.15.60:0/2545066063 learned_addr learned my addr 172.21.15.60:0/2545066063 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:19.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 -- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73980cc8d0 msgr2=0x7f73980c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73980cc8d0 0x7f73980c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 -- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73980caa10 msgr2=0x7f7398154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73980caa10 0x7f7398154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.245+0000 7f739e3fe640 1 -- 172.21.15.60:0/2545066063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739000a670 con 0x7f73981593b0 2023-10-16T12:54:19.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.246+0000 7f739e3fe640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 0x7f73980af0d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f739002ce30 tx=0x7f73900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:19.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.246+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739003d600 con 0x7f73981593b0 2023-10-16T12:54:19.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.246+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739003d780 con 0x7f73981593b0 2023-10-16T12:54:19.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.246+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739003db30 con 0x7f73981593b0 2023-10-16T12:54:19.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.246+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73980c86c0 con 0x7f73981593b0 2023-10-16T12:54:19.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.246+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73980b1ac0 con 0x7f73981593b0 2023-10-16T12:54:19.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.247+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73980c8900 con 0x7f73981593b0 2023-10-16T12:54:19.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.248+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f739003dc90 con 0x7f73981593b0 2023-10-16T12:54:19.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.250+0000 7f738e7fc640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f737c06af90 0x7f737c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.250+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73900ab540 con 0x7f73981593b0 2023-10-16T12:54:19.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.251+0000 7f739dbfd640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f737c06af90 0x7f737c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.252+0000 7f739dbfd640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f737c06af90 0x7f737c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f739815c160 tx=0x7f73940083d0 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:19.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.255+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73900764e0 con 0x7f73981593b0 2023-10-16T12:54:19.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.523+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f73980c9fe0 con 0x7f73981593b0 2023-10-16T12:54:19.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.528+0000 7f738e7fc640 1 -- 172.21.15.60:0/2545066063 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7390079b00 con 0x7f73981593b0 2023-10-16T12:54:19.530 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T12:54:19.530 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T12:54:19.531 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T12:54:19.531 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T12:54:19.531 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:54:19.531 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T12:54:19.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.531+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f737c06af90 msgr2=0x7f737c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.531+0000 7f739fe61640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f737c06af90 0x7f737c06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f739815c160 tx=0x7f73940083d0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.531+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 msgr2=0x7f73980af0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.531+0000 7f739fe61640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 0x7f73980af0d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f739002ce30 tx=0x7f73900046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 shutdown_connections 2023-10-16T12:54:19.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f737c06af90 0x7f737c06d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73980cc8d0 0x7f73980c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73981593b0 0x7f73980af0d0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 --2- 172.21.15.60:0/2545066063 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73980caa10 0x7f7398154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 >> 172.21.15.60:0/2545066063 conn(0x7f73980cb410 msgr2=0x7f73980cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:19.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 shutdown_connections 2023-10-16T12:54:19.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.532+0000 7f739fe61640 1 -- 172.21.15.60:0/2545066063 wait complete. 2023-10-16T12:54:19.552 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:54:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.780+0000 7fd4c5aaf640 1 Processor -- start 2023-10-16T12:54:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.781+0000 7fd4c5aaf640 1 -- start start 2023-10-16T12:54:19.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.782+0000 7fd4c5aaf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4c015ae80 0x7fd4c015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.782+0000 7fd4c5aaf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.782+0000 7fd4c5aaf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c015ed60 con 0x7fd4c015aaa0 2023-10-16T12:54:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.782+0000 7fd4c5aaf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c006b810 con 0x7fd4c015ae80 2023-10-16T12:54:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.782+0000 7fd4c5aaf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00707e0 con 0x7fd4c00ce390 2023-10-16T12:54:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4b7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4c015ae80 0x7fd4c015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4bf7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd4c015aaa0 0x7fd4c00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56096/0 (socket says 172.21.15.60:56096) 2023-10-16T12:54:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4bf7fe640 1 -- 172.21.15.60:0/2952273678 learned_addr learned my addr 172.21.15.60:0/2952273678 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4bffff640 1 --2- 172.21.15.60:0/2952273678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2952273678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 msgr2=0x7fd4c0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4b7fff640 1 --2- 172.21.15.60:0/2952273678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2952273678 >> v1:172.21.15.60:6789/0 conn(0x7fd4c015aaa0 legacy=0x7fd4c00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.783+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2952273678 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c015ef90 con 0x7fd4c015ae80 2023-10-16T12:54:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.785+0000 7fd4b7fff640 1 --2- 172.21.15.60:0/2952273678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015ae80 0x7fd4c015e820 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80097a0 tx=0x7fd4a802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f5d5676f43ff9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.785+0000 7fd4beffd640 1 -- 172.21.15.60:0/2952273678 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a802d040 con 0x7fd4c015ae80 2023-10-16T12:54:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.785+0000 7fd4beffd640 1 -- 172.21.15.60:0/2952273678 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a8031810 con 0x7fd4c015ae80 2023-10-16T12:54:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.785+0000 7fd4beffd640 1 -- 172.21.15.60:0/2952273678 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8031ba0 con 0x7fd4c015ae80 2023-10-16T12:54:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.785+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2952273678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015ae80 msgr2=0x7fd4c015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.785+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2952273678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015ae80 0x7fd4c015e820 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80097a0 tx=0x7fd4a802b8b0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.786+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2952273678 shutdown_connections 2023-10-16T12:54:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.786+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2952273678 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.786+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2952273678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015ae80 0x7fd4c015e820 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.786+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2952273678 >> 172.21.15.60:0/2952273678 conn(0x7fd4c00cb410 msgr2=0x7fd4c00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.786+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2952273678 shutdown_connections 2023-10-16T12:54:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.786+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2952273678 wait complete. 2023-10-16T12:54:19.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.788+0000 7fd4c5aaf640 1 Processor -- start 2023-10-16T12:54:19.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.788+0000 7fd4c5aaf640 1 -- start start 2023-10-16T12:54:19.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4c5aaf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c00b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4c5aaf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 0x7fd4c00acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4c5aaf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4c015ae80 0x7fd4c0155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4c5aaf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c015b2b0 con 0x7fd4c015aaa0 2023-10-16T12:54:19.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4c5aaf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00c4010 con 0x7fd4c015ae80 2023-10-16T12:54:19.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4c5aaf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00c3490 con 0x7fd4c00ce390 2023-10-16T12:54:19.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 0x7fd4c00acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 0x7fd4c00acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52740/0 (socket says 172.21.15.60:52740) 2023-10-16T12:54:19.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2263410622 learned_addr learned my addr 172.21.15.60:0/2263410622 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:19.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4bf7fe640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c00b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 msgr2=0x7fd4c00b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4b7fff640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c00b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4c015ae80 msgr2=0x7fd4c0155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:19.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.789+0000 7fd4bffff640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4c015ae80 0x7fd4c0155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4b7fff640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4c015ae80 0x7fd4c0155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4b7fff640 1 -- 172.21.15.60:0/2263410622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8009400 con 0x7fd4c015aaa0 2023-10-16T12:54:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4b7fff640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 0x7fd4c00acb00 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8000c00 tx=0x7fd4a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a803ad60 con 0x7fd4c015aaa0 2023-10-16T12:54:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a80443f0 con 0x7fd4c015aaa0 2023-10-16T12:54:19.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c00c3770 con 0x7fd4c015aaa0 2023-10-16T12:54:19.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a80447a0 con 0x7fd4c015aaa0 2023-10-16T12:54:19.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.790+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c00c3d30 con 0x7fd4c015aaa0 2023-10-16T12:54:19.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.792+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c004e160 con 0x7fd4c015aaa0 2023-10-16T12:54:19.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.792+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd4a804d450 con 0x7fd4c015aaa0 2023-10-16T12:54:19.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.793+0000 7fd4bcff9640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd4a006af90 0x7fd4a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:19.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.793+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4a80abb60 con 0x7fd4c015aaa0 2023-10-16T12:54:19.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.794+0000 7fd4bf7fe640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd4a006af90 0x7fd4a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:19.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.795+0000 7fd4bf7fe640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd4a006af90 0x7fd4a006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0000f60 tx=0x7fd4b0009040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:19.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:19.798+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4a80ade60 con 0x7fd4c015aaa0 2023-10-16T12:54:20.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.041+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4c00717e0 con 0x7fd4c015aaa0 2023-10-16T12:54:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.042+0000 7fd4bcff9640 1 -- 172.21.15.60:0/2263410622 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fd4a8076950 con 0x7fd4c015aaa0 2023-10-16T12:54:20.044 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:20.045 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:54:13.310628+0000","modified":"2023-10-16T12:54:16.431585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8964":{"gid":8964,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3372982800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3372982800},{"type":"v1","addr":"172.21.15.60:6833","nonce":3372982800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T12:54:20.045 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 79 2023-10-16T12:54:20.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd4a006af90 msgr2=0x7fd4a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd4a006af90 0x7fd4a006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0000f60 tx=0x7fd4b0009040 comp rx=0 tx=0).stop 2023-10-16T12:54:20.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 msgr2=0x7fd4c00acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 0x7fd4c00acb00 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8000c00 tx=0x7fd4a80046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 shutdown_connections 2023-10-16T12:54:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fd4a006af90 0x7fd4a006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4c015ae80 0x7fd4c0155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4c015aaa0 0x7fd4c00acb00 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 --2- 172.21.15.60:0/2263410622 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4c00ce390 0x7fd4c00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 >> 172.21.15.60:0/2263410622 conn(0x7fd4c00cb410 msgr2=0x7fd4c00b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 shutdown_connections 2023-10-16T12:54:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.045+0000 7fd4c5aaf640 1 -- 172.21.15.60:0/2263410622 wait complete. 2023-10-16T12:54:20.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:54:13.310628+0000', 'modified': '2023-10-16T12:54:16.431585+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8964}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8964': {'gid': 8964, 'name': 'a', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3372982800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3372982800}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3372982800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:54:20.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:54:20.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:54:20.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.706+0000 7f7a27fff640 1 Processor -- start 2023-10-16T12:54:20.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.706+0000 7f7a27fff640 1 -- start start 2023-10-16T12:54:20.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.707+0000 7f7a27fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a2806c040 0x7f7a2806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.707+0000 7f7a27fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7a280cda80 0x7f7a2815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a27fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2806c960 con 0x7f7a280ad970 2023-10-16T12:54:20.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a27fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2815b9d0 con 0x7f7a2806c040 2023-10-16T12:54:20.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a27fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2815bcb0 con 0x7f7a280cda80 2023-10-16T12:54:20.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a26ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a2806c040 0x7f7a2806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a267fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7a280ad970 0x7f7a2815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56112/0 (socket says 172.21.15.60:56112) 2023-10-16T12:54:20.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a267fc640 1 -- 172.21.15.60:0/1932392203 learned_addr learned my addr 172.21.15.60:0/1932392203 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:20.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a26ffd640 1 -- 172.21.15.60:0/1932392203 >> v2:172.21.15.144:3300/0 conn(0x7f7a280cda80 msgr2=0x7f7a2815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1932392203 >> v2:172.21.15.144:3300/0 conn(0x7f7a280cda80 0x7f7a2815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a26ffd640 1 -- 172.21.15.60:0/1932392203 >> v1:172.21.15.60:6789/0 conn(0x7f7a280ad970 legacy=0x7f7a2815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.708+0000 7f7a26ffd640 1 -- 172.21.15.60:0/1932392203 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2815bf90 con 0x7f7a2806c040 2023-10-16T12:54:20.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.709+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1932392203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2806c040 0x7f7a2806c420 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7a280cf300 tx=0x7f7a1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7eccf97d8ea5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.710+0000 7f7a25ffb640 1 -- 172.21.15.60:0/1932392203 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10031830 con 0x7f7a2806c040 2023-10-16T12:54:20.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.710+0000 7f7a25ffb640 1 -- 172.21.15.60:0/1932392203 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10031990 con 0x7f7a2806c040 2023-10-16T12:54:20.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.710+0000 7f7a25ffb640 1 -- 172.21.15.60:0/1932392203 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10031d20 con 0x7f7a2806c040 2023-10-16T12:54:20.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.710+0000 7f7a27fff640 1 -- 172.21.15.60:0/1932392203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2806c040 msgr2=0x7f7a2806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.710+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1932392203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2806c040 0x7f7a2806c420 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7a280cf300 tx=0x7f7a1002b8d0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.711+0000 7f7a27fff640 1 -- 172.21.15.60:0/1932392203 shutdown_connections 2023-10-16T12:54:20.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.711+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1932392203 >> v2:172.21.15.144:3300/0 conn(0x7f7a280cda80 0x7f7a2815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.711+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1932392203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2806c040 0x7f7a2806c420 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.711+0000 7f7a27fff640 1 -- 172.21.15.60:0/1932392203 >> 172.21.15.60:0/1932392203 conn(0x7f7a280ce6a0 msgr2=0x7f7a280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.711+0000 7f7a27fff640 1 -- 172.21.15.60:0/1932392203 shutdown_connections 2023-10-16T12:54:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.711+0000 7f7a27fff640 1 -- 172.21.15.60:0/1932392203 wait complete. 2023-10-16T12:54:20.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.712+0000 7f7a27fff640 1 Processor -- start 2023-10-16T12:54:20.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.713+0000 7f7a27fff640 1 -- start start 2023-10-16T12:54:20.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2806c040 0x7f7a28156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a280ad970 0x7f7a280c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 0x7f7a280c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a27fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280c7260 con 0x7f7a280cda80 2023-10-16T12:54:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a27fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280c6dd0 con 0x7f7a2806c040 2023-10-16T12:54:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a27fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28156e40 con 0x7f7a280ad970 2023-10-16T12:54:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a277fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 0x7f7a280c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a277fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 0x7f7a280c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52760/0 (socket says 172.21.15.60:52760) 2023-10-16T12:54:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a277fe640 1 -- 172.21.15.60:0/1382495030 learned_addr learned my addr 172.21.15.60:0/1382495030 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2806c040 0x7f7a28156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a267fc640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a280ad970 0x7f7a280c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a277fe640 1 -- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a280ad970 msgr2=0x7f7a280c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a277fe640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a280ad970 0x7f7a280c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.714+0000 7f7a277fe640 1 -- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2806c040 msgr2=0x7f7a28156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a277fe640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2806c040 0x7f7a28156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a277fe640 1 -- 172.21.15.60:0/1382495030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10009400 con 0x7f7a280cda80 2023-10-16T12:54:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2806c040 0x7f7a28156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:54:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a277fe640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 0x7f7a280c7af0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000b410 tx=0x7f7a2000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a1f7fe640 1 -- 172.21.15.60:0/1382495030 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20011830 con 0x7f7a280cda80 2023-10-16T12:54:20.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a1f7fe640 1 -- 172.21.15.60:0/1382495030 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a200119b0 con 0x7f7a280cda80 2023-10-16T12:54:20.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a1f7fe640 1 -- 172.21.15.60:0/1382495030 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20011d60 con 0x7f7a280cda80 2023-10-16T12:54:20.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a280c61c0 con 0x7f7a280cda80 2023-10-16T12:54:20.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.715+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a280c6780 con 0x7f7a280cda80 2023-10-16T12:54:20.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.717+0000 7f7a1f7fe640 1 -- 172.21.15.60:0/1382495030 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a2001a400 con 0x7f7a280cda80 2023-10-16T12:54:20.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.719+0000 7f7a1f7fe640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7a0806af90 0x7f7a0806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.719+0000 7f7a1f7fe640 1 -- 172.21.15.60:0/1382495030 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a20089980 con 0x7f7a280cda80 2023-10-16T12:54:20.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.720+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7a0806af90 0x7f7a0806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.720+0000 7f7a303dd740 1 Processor -- start 2023-10-16T12:54:20.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.720+0000 7f7a303dd740 1 -- start start 2023-10-16T12:54:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7a0806af90 0x7f7a0806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a10036000 tx=0x7f7a1003c040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce75cf0 0x55ed3ce738b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 0x55ed3ce2d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce2d760 0x55ed3ce57f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a303dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cd487a0 con 0x55ed3ce55530 2023-10-16T12:54:20.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cd9a740 con 0x55ed3ce2d760 2023-10-16T12:54:20.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cd97e30 con 0x55ed3ce75cf0 2023-10-16T12:54:20.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.721+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 0x55ed3ce2d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 0x55ed3ce2d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52772/0 (socket says 172.21.15.60:52772) 2023-10-16T12:54:20.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 -- 172.21.15.60:0/4173203115 learned_addr learned my addr 172.21.15.60:0/4173203115 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:20.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce75cf0 0x55ed3ce738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 -- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce75cf0 msgr2=0x55ed3ce738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a277fe640 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce2d760 0x55ed3ce57f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce75cf0 0x55ed3ce738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 -- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce2d760 msgr2=0x55ed3ce57f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce2d760 0x55ed3ce57f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 -- 172.21.15.60:0/4173203115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed3cd57910 con 0x55ed3ce55530 2023-10-16T12:54:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.722+0000 7f7a267fc640 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 0x55ed3ce2d220 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7a280ade20 tx=0x7f7a1800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac44e2250bbb196 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a1cff9640 1 -- 172.21.15.60:0/4173203115 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1800ee50 con 0x55ed3ce55530 2023-10-16T12:54:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a1cff9640 1 -- 172.21.15.60:0/4173203115 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a18014480 con 0x55ed3ce55530 2023-10-16T12:54:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a1cff9640 1 -- 172.21.15.60:0/4173203115 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180147f0 con 0x55ed3ce55530 2023-10-16T12:54:20.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 -- 172.21.15.60:0/4173203115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 msgr2=0x55ed3ce2d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 0x55ed3ce2d220 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7a280ade20 tx=0x7f7a1800e530 comp rx=0 tx=0).stop 2023-10-16T12:54:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 -- 172.21.15.60:0/4173203115 shutdown_connections 2023-10-16T12:54:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce2d760 0x55ed3ce57f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce55530 0x55ed3ce2d220 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 --2- 172.21.15.60:0/4173203115 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce75cf0 0x55ed3ce738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 -- 172.21.15.60:0/4173203115 >> 172.21.15.60:0/4173203115 conn(0x55ed3ce2a380 msgr2=0x55ed3ce72f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 -- 172.21.15.60:0/4173203115 shutdown_connections 2023-10-16T12:54:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.723+0000 7f7a303dd740 1 -- 172.21.15.60:0/4173203115 wait complete. 2023-10-16T12:54:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.724+0000 7f7a303dd740 1 -- start start 2023-10-16T12:54:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.724+0000 7f7a303dd740 1 Processor -- start 2023-10-16T12:54:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce50180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce59070 0x55ed3ce594a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cd93720 con 0x55ed3ce75cf0 2023-10-16T12:54:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cd31970 con 0x55ed3ce59070 2023-10-16T12:54:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cd31610 con 0x55ed3ce55530 2023-10-16T12:54:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.725+0000 7f7a303dd740 10 client.0 _ll_register_callbacks cb 0x55ed3cd539d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:54:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52778/0 (socket says 172.21.15.60:52778) 2023-10-16T12:54:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 -- 172.21.15.60:0/12141973 learned_addr learned my addr 172.21.15.60:0/12141973 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 -- 172.21.15.60:0/12141973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 msgr2=0x55ed3ce50180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce50180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 -- 172.21.15.60:0/12141973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce59070 msgr2=0x55ed3ce594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce59070 0x55ed3ce594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 -- 172.21.15.60:0/12141973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1800f040 con 0x55ed3ce75cf0 2023-10-16T12:54:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a267fc640 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58b30 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f7a18013000 tx=0x7f7a18001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe23ca4546dddf69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.726+0000 7f7a05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:54:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a05ffb640 1 -- 172.21.15.60:0/12141973 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18014bb0 con 0x55ed3ce75cf0 2023-10-16T12:54:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed3cdba160 con 0x55ed3ce75cf0 2023-10-16T12:54:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a05ffb640 1 -- 172.21.15.60:0/12141973 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a18014d10 con 0x55ed3ce75cf0 2023-10-16T12:54:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a05ffb640 1 -- 172.21.15.60:0/12141973 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a180070d0 con 0x55ed3ce75cf0 2023-10-16T12:54:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 10 client.9111 fetch_fsmap learned FSMap version 79 2023-10-16T12:54:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x55ed3cd91da0 con 0x55ed3ce75cf0 2023-10-16T12:54:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a05ffb640 1 -- 172.21.15.60:0/12141973 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ed3cdba160 con 0x55ed3ce75cf0 2023-10-16T12:54:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a05ffb640 1 -- 172.21.15.60:0/12141973 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 79) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7a18007230 con 0x55ed3ce75cf0 2023-10-16T12:54:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 10 client.9111 fetch_fsmap finished waiting for FSMap version 79 2023-10-16T12:54:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 10 client.9111 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:54:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 20 client.9111 populate_metadata read hostname 'smithi060' 2023-10-16T12:54:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3ce51360 0x55ed3ce56530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 4 client.9111 mds_command: new command op to 8964 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:54:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.727+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 --> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed3cd65cc0 con 0x55ed3ce51360 2023-10-16T12:54:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.728+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3ce51360 0x55ed3ce56530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.728+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3ce51360 0x55ed3ce56530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ef75fe54994ecba server_cookie=54abf9167a2de174 in_seq=0 out_seq=0 2023-10-16T12:54:20.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.728+0000 7f7a05ffb640 10 client.9111 ms_handle_connect on v2:172.21.15.60:6832/3372982800 2023-10-16T12:54:20.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a05ffb640 1 -- 172.21.15.60:0/12141973 <== mds.0 v2:172.21.15.60:6832/3372982800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ed3cd65cc0 con 0x55ed3ce51360 2023-10-16T12:54:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a05ffb640 10 client.9111 handle_command_reply: tid=0 2023-10-16T12:54:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 client.9111 shutdown 2023-10-16T12:54:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 msgr2=0x55ed3ce58b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58b30 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f7a18013000 tx=0x7f7a18001250 comp rx=0 tx=0).stop 2023-10-16T12:54:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 shutdown_connections 2023-10-16T12:54:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3ce51360 0x55ed3ce56530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce59070 0x55ed3ce594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58b30 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 --2- 172.21.15.60:0/12141973 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce50180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a05ffb640 0 client.9111 ms_handle_reset on v2:172.21.15.60:6832/3372982800 2023-10-16T12:54:20.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 >> 172.21.15.60:0/12141973 conn(0x55ed3ce2d760 msgr2=0x55ed3ce270f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.729+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 shutdown_connections 2023-10-16T12:54:20.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.730+0000 7f7a303dd740 1 -- 172.21.15.60:0/12141973 wait complete. 2023-10-16T12:54:20.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.730+0000 7f7a303dd740 20 client.9111 trim_cache size 0 max 16384 2023-10-16T12:54:20.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.746+0000 7f7a303dd740 1 Processor -- start 2023-10-16T12:54:20.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 -- start start 2023-10-16T12:54:20.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce278d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 0x55ed3ce57e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3caf7f20 con 0x55ed3ce59070 2023-10-16T12:54:20.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3ce2f3d0 con 0x55ed3ce75cf0 2023-10-16T12:54:20.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.747+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed3ce27e10 con 0x55ed3ce55530 2023-10-16T12:54:20.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 0x55ed3ce57e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 0x55ed3ce57e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52790/0 (socket says 172.21.15.60:52790) 2023-10-16T12:54:20.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 -- 172.21.15.60:0/1953217455 learned_addr learned my addr 172.21.15.60:0/1953217455 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:20.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce278d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a277fe640 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 -- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 msgr2=0x55ed3ce278d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce278d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 -- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce75cf0 msgr2=0x55ed3ce58390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.748+0000 7f7a267fc640 1 -- 172.21.15.60:0/1953217455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed3cd30b80 con 0x55ed3ce59070 2023-10-16T12:54:20.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.749+0000 7f7a267fc640 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 0x55ed3ce57e50 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f7a18005c00 tx=0x7f7a1801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa158e4d5229bf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.749+0000 7f7a1d7fa640 1 -- 172.21.15.60:0/1953217455 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18007a50 con 0x55ed3ce59070 2023-10-16T12:54:20.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.749+0000 7f7a1d7fa640 1 -- 172.21.15.60:0/1953217455 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a180011c0 con 0x55ed3ce59070 2023-10-16T12:54:20.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.749+0000 7f7a1d7fa640 1 -- 172.21.15.60:0/1953217455 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180070d0 con 0x55ed3ce59070 2023-10-16T12:54:20.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 -- 172.21.15.60:0/1953217455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 msgr2=0x55ed3ce57e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 0x55ed3ce57e50 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f7a18005c00 tx=0x7f7a1801f000 comp rx=0 tx=0).stop 2023-10-16T12:54:20.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 -- 172.21.15.60:0/1953217455 shutdown_connections 2023-10-16T12:54:20.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce58390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce59070 0x55ed3ce57e50 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 --2- 172.21.15.60:0/1953217455 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce278d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 -- 172.21.15.60:0/1953217455 >> 172.21.15.60:0/1953217455 conn(0x55ed3ce2d760 msgr2=0x55ed3ce71220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 -- 172.21.15.60:0/1953217455 shutdown_connections 2023-10-16T12:54:20.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.750+0000 7f7a303dd740 1 -- 172.21.15.60:0/1953217455 wait complete. 2023-10-16T12:54:20.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.751+0000 7f7a303dd740 1 -- start start 2023-10-16T12:54:20.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.751+0000 7f7a303dd740 1 Processor -- start 2023-10-16T12:54:20.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce56f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce69120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce54fc0 0x55ed3ce69660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3cedfa00 con 0x55ed3ce75cf0 2023-10-16T12:54:20.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed3ce767c0 con 0x55ed3ce54fc0 2023-10-16T12:54:20.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed3ce27960 con 0x55ed3ce55530 2023-10-16T12:54:20.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a303dd740 10 client.0 _ll_register_callbacks cb 0x55ed3ce2baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:54:20.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce69120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce69120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52792/0 (socket says 172.21.15.60:52792) 2023-10-16T12:54:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.752+0000 7f7a267fc640 1 -- 172.21.15.60:0/194517502 learned_addr learned my addr 172.21.15.60:0/194517502 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a267fc640 1 -- 172.21.15.60:0/194517502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 msgr2=0x55ed3ce56f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a267fc640 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce56f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a267fc640 1 -- 172.21.15.60:0/194517502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce54fc0 msgr2=0x55ed3ce69660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:20.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a267fc640 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce54fc0 0x55ed3ce69660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a267fc640 1 -- 172.21.15.60:0/194517502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed3cd91da0 con 0x55ed3ce75cf0 2023-10-16T12:54:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a267fc640 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce69120 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800ea30 tx=0x7f7a18009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861400487def8388 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T12:54:20.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.753+0000 7f7a05ffb640 1 -- 172.21.15.60:0/194517502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18014bb0 con 0x55ed3ce75cf0 2023-10-16T12:54:20.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a05ffb640 1 -- 172.21.15.60:0/194517502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a18014d10 con 0x55ed3ce75cf0 2023-10-16T12:54:20.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a18014480 con 0x55ed3ce75cf0 2023-10-16T12:54:20.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a05ffb640 1 -- 172.21.15.60:0/194517502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a1803b460 con 0x55ed3ce75cf0 2023-10-16T12:54:20.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 10 client.9117 fetch_fsmap learned FSMap version 79 2023-10-16T12:54:20.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x55ed3cd56170 con 0x55ed3ce75cf0 2023-10-16T12:54:20.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a05ffb640 1 -- 172.21.15.60:0/194517502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a18014480 con 0x55ed3ce75cf0 2023-10-16T12:54:20.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a05ffb640 1 -- 172.21.15.60:0/194517502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 79) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7a18039040 con 0x55ed3ce75cf0 2023-10-16T12:54:20.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 10 client.9117 fetch_fsmap finished waiting for FSMap version 79 2023-10-16T12:54:20.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 10 client.9117 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:54:20.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 20 client.9117 populate_metadata read hostname 'smithi060' 2023-10-16T12:54:20.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3cee0050 0x55ed3cee0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:20.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 4 client.9117 mds_command: new command op to 8964 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:54:20.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.754+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 --> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ed3cd8c700 con 0x55ed3cee0050 2023-10-16T12:54:20.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.755+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3cee0050 0x55ed3cee0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:20.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.755+0000 7f7a26ffd640 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3cee0050 0x55ed3cee0430 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1788afb9bfa3624 server_cookie=b31c17f326638d8 in_seq=0 out_seq=0 2023-10-16T12:54:20.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.755+0000 7f7a05ffb640 10 client.9117 ms_handle_connect on v2:172.21.15.60:6832/3372982800 2023-10-16T12:54:20.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.755+0000 7f7a05ffb640 1 -- 172.21.15.60:0/194517502 <== mds.0 v2:172.21.15.60:6832/3372982800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55ed3cd8c700 con 0x55ed3cee0050 2023-10-16T12:54:20.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.755+0000 7f7a05ffb640 10 client.9117 handle_command_reply: tid=0 2023-10-16T12:54:20.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.755+0000 7f7a303dd740 1 client.9117 shutdown 2023-10-16T12:54:20.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 msgr2=0x55ed3ce69120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce69120 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800ea30 tx=0x7f7a18009040 comp rx=0 tx=0).stop 2023-10-16T12:54:20.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 shutdown_connections 2023-10-16T12:54:20.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:6832/3372982800,v1:172.21.15.60:6833/3372982800] conn(0x55ed3cee0050 0x55ed3cee0430 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ed3ce54fc0 0x55ed3ce69660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ed3ce75cf0 0x55ed3ce69120 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a05ffb640 0 client.9117 ms_handle_reset on v2:172.21.15.60:6832/3372982800 2023-10-16T12:54:20.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 --2- 172.21.15.60:0/194517502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55ed3ce55530 0x55ed3ce56f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 >> 172.21.15.60:0/194517502 conn(0x55ed3ce2d760 msgr2=0x55ed3ce57e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 shutdown_connections 2023-10-16T12:54:20.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 1 -- 172.21.15.60:0/194517502 wait complete. 2023-10-16T12:54:20.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.756+0000 7f7a303dd740 20 client.9117 trim_cache size 0 max 16384 2023-10-16T12:54:20.788 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T12:54:20.789 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T12:54:20.789 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T12:54:20.789 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 8964, 2023-10-16T12:54:20.790 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T12:54:20.790 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T12:54:20.791 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T12:54:20.791 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8688250589999997, 2023-10-16T12:54:20.791 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 79, 2023-10-16T12:54:20.791 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 156, 2023-10-16T12:54:20.791 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 156, 2023-10-16T12:54:20.792 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.683757324 2023-10-16T12:54:20.792 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T12:54:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7a0806af90 msgr2=0x7f7a0806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7a0806af90 0x7f7a0806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a10036000 tx=0x7f7a1003c040 comp rx=0 tx=0).stop 2023-10-16T12:54:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 msgr2=0x7f7a280c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 0x7f7a280c7af0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000b410 tx=0x7f7a2000b8d0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 shutdown_connections 2023-10-16T12:54:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f7a0806af90 0x7f7a0806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a280ad970 0x7f7a280c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a280cda80 0x7f7a280c7af0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 --2- 172.21.15.60:0/1382495030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2806c040 0x7f7a28156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 >> 172.21.15.60:0/1382495030 conn(0x7f7a280ce6a0 msgr2=0x7f7a2815e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 shutdown_connections 2023-10-16T12:54:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:20.757+0000 7f7a27fff640 1 -- 172.21.15.60:0/1382495030 wait complete. 2023-10-16T12:54:20.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:54:21.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.030+0000 7ff7fe494640 1 Processor -- start 2023-10-16T12:54:21.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.030+0000 7ff7fe494640 1 -- start start 2023-10-16T12:54:21.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fe494640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7f80cfc10 0x7ff7f80d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:21.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fe494640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7f80b3e70 0x7ff7f80d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:21.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fe494640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80b3940 con 0x7ff7f80b3070 2023-10-16T12:54:21.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fe494640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80b45e0 con 0x7ff7f80b3e70 2023-10-16T12:54:21.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fe494640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80c7880 con 0x7ff7f80cfc10 2023-10-16T12:54:21.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fca31640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7f80b3e70 0x7ff7f80d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:21.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fca31640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7f80b3e70 0x7ff7f80d1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52806/0 (socket says 172.21.15.60:52806) 2023-10-16T12:54:21.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7fca31640 1 -- 172.21.15.60:0/3460368347 learned_addr learned my addr 172.21.15.60:0/3460368347 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:21.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.032+0000 7ff7ef7fe640 1 --2- 172.21.15.60:0/3460368347 >> v2:172.21.15.144:3300/0 conn(0x7ff7f80cfc10 0x7ff7f80d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:21.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.033+0000 7ff7fca31640 1 -- 172.21.15.60:0/3460368347 >> v2:172.21.15.144:3300/0 conn(0x7ff7f80cfc10 msgr2=0x7ff7f80d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:21.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.033+0000 7ff7fca31640 1 --2- 172.21.15.60:0/3460368347 >> v2:172.21.15.144:3300/0 conn(0x7ff7f80cfc10 0x7ff7f80d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.033+0000 7ff7fca31640 1 -- 172.21.15.60:0/3460368347 >> v1:172.21.15.144:6789/0 conn(0x7ff7f80b3070 legacy=0x7ff7f80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:21.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.033+0000 7ff7fca31640 1 -- 172.21.15.60:0/3460368347 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f80b3b90 con 0x7ff7f80b3e70 2023-10-16T12:54:21.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.034+0000 7ff7fca31640 1 --2- 172.21.15.60:0/3460368347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3e70 0x7ff7f80d1040 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80b58e0 tx=0x7ff7f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d9d2c687d44202 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:21.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.034+0000 7ff7f77fe640 1 -- 172.21.15.60:0/3460368347 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f002ca80 con 0x7ff7f80b3e70 2023-10-16T12:54:21.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.034+0000 7ff7f77fe640 1 -- 172.21.15.60:0/3460368347 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f002cbe0 con 0x7ff7f80b3e70 2023-10-16T12:54:21.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.034+0000 7ff7f77fe640 1 -- 172.21.15.60:0/3460368347 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f00343f0 con 0x7ff7f80b3e70 2023-10-16T12:54:21.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 -- 172.21.15.60:0/3460368347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3e70 msgr2=0x7ff7f80d1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:21.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 --2- 172.21.15.60:0/3460368347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3e70 0x7ff7f80d1040 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80b58e0 tx=0x7ff7f002bee0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 -- 172.21.15.60:0/3460368347 shutdown_connections 2023-10-16T12:54:21.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 --2- 172.21.15.60:0/3460368347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3e70 0x7ff7f80d1040 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 --2- 172.21.15.60:0/3460368347 >> v2:172.21.15.144:3300/0 conn(0x7ff7f80cfc10 0x7ff7f80d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 -- 172.21.15.60:0/3460368347 >> 172.21.15.60:0/3460368347 conn(0x7ff7f80d2d20 msgr2=0x7ff7f80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:21.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 -- 172.21.15.60:0/3460368347 shutdown_connections 2023-10-16T12:54:21.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.035+0000 7ff7fe494640 1 -- 172.21.15.60:0/3460368347 wait complete. 2023-10-16T12:54:21.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.037+0000 7ff7fe494640 1 Processor -- start 2023-10-16T12:54:21.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.037+0000 7ff7fe494640 1 -- start start 2023-10-16T12:54:21.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7fe494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 0x7ff7f8154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:21.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7fe494640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7f80b3e70 0x7ff7f80c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:21.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7fe494640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f80cfc10 0x7ff7f80c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:21.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7fe494640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b5520 con 0x7ff7f80b3070 2023-10-16T12:54:21.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7fe494640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b4930 con 0x7ff7f80cfc10 2023-10-16T12:54:21.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7fe494640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b4bf0 con 0x7ff7f80b3e70 2023-10-16T12:54:21.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 0x7ff7f8154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:21.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 0x7ff7f8154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52820/0 (socket says 172.21.15.60:52820) 2023-10-16T12:54:21.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.038+0000 7ff7f7fff640 1 -- 172.21.15.60:0/986107935 learned_addr learned my addr 172.21.15.60:0/986107935 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:21.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7fca31640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f80cfc10 0x7ff7f80c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:21.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7ef7fe640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7f80b3e70 0x7ff7f80c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:21.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7f7fff640 1 -- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7f80b3e70 msgr2=0x7ff7f80c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:21.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7f7fff640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7f80b3e70 0x7ff7f80c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7f7fff640 1 -- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f80cfc10 msgr2=0x7ff7f80c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:21.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7f7fff640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f80cfc10 0x7ff7f80c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7f7fff640 1 -- 172.21.15.60:0/986107935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0009b30 con 0x7ff7f80b3070 2023-10-16T12:54:21.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7fca31640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f80cfc10 0x7ff7f80c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:21.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.039+0000 7ff7f7fff640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 0x7ff7f8154460 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8004640 tx=0x7ff7e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:21.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.040+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8011790 con 0x7ff7f80b3070 2023-10-16T12:54:21.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.040+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e80118f0 con 0x7ff7f80b3070 2023-10-16T12:54:21.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.040+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8011ce0 con 0x7ff7f80b3070 2023-10-16T12:54:21.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.040+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f80b4ed0 con 0x7ff7f80b3070 2023-10-16T12:54:21.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.040+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f80b59b0 con 0x7ff7f80b3070 2023-10-16T12:54:21.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.041+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f804e160 con 0x7ff7f80b3070 2023-10-16T12:54:21.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.042+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff7e8012420 con 0x7ff7f80b3070 2023-10-16T12:54:21.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.043+0000 7ff7f57fa640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff7d806af90 0x7ff7d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:21.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.043+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7e8089400 con 0x7ff7f80b3070 2023-10-16T12:54:21.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.044+0000 7ff7ef7fe640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff7d806af90 0x7ff7d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:21.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.045+0000 7ff7ef7fe640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff7d806af90 0x7ff7d806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0006fd0 tx=0x7ff7e0008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.048+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff7e8054390 con 0x7ff7f80b3070 2023-10-16T12:54:21.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.290+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff7f80cdcd0 con 0x7ff7f80b3070 2023-10-16T12:54:21.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.291+0000 7ff7f57fa640 1 -- 172.21.15.60:0/986107935 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v79) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff7e80579b0 con 0x7ff7f80b3070 2023-10-16T12:54:21.293 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:54:21.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff7d806af90 msgr2=0x7ff7d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:21.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff7d806af90 0x7ff7d806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0006fd0 tx=0x7ff7e0008040 comp rx=0 tx=0).stop 2023-10-16T12:54:21.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 msgr2=0x7ff7f8154460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:21.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 0x7ff7f8154460 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8004640 tx=0x7ff7e800c8e0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 shutdown_connections 2023-10-16T12:54:21.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff7d806af90 0x7ff7d806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7f80b3e70 0x7ff7f80c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f80cfc10 0x7ff7f80c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 --2- 172.21.15.60:0/986107935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7f80b3070 0x7ff7f8154460 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:21.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 >> 172.21.15.60:0/986107935 conn(0x7ff7f80d2d20 msgr2=0x7ff7f80d3100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:21.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.294+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 shutdown_connections 2023-10-16T12:54:21.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:21.295+0000 7ff7fe494640 1 -- 172.21.15.60:0/986107935 wait complete. 2023-10-16T12:54:21.316 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:54:21.317 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:54:21.317 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T12:54:21.317 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:54:21.318 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:54:21.318 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:54:21.318 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:54:21.319 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:54:21.319 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:54:21.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:21.320 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T12:54:21.377 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:54:21.378 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:54:21.378 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:54:21.378 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.379 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T12:54:21.379 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.379 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:54:21.379 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T12:54:21.379 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T12:54:21.380 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:54:21.380 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T12:54:21.380 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T12:54:21.380 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:54:21.380 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:54:21.381 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T12:54:21.381 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:54:21.381 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 41126sec preferred_lft 41126sec 2023-10-16T12:54:21.381 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T12:54:21.381 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.382 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:54:21.382 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:54:21.382 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T12:54:21.382 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:54:21.382 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T12:54:21.383 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:54:21.383 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.383 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T12:54:21.383 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.383 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:54:21.384 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:54:21.384 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T12:54:21.384 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.384 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:54:21.384 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:54:21.385 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T12:54:21.385 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:54:21.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:21.387 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:54:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T12:54:21.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:21.494 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T12:54:21.558 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T12:54:21.559 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:54:21.559 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:54:21.559 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:54:21.559 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:54:21.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:21.560 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T12:54:21.561 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T12:54:21.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:54:21.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:54:21.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:54:21.562 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T12:54:21.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:21.764 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T12:54:21.823 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:54:21.823 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:54:21.824 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:54:21.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:21.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T12:54:21.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:54:21.934 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:54:21.991 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:54:21.992 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:54:21.992 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:54:22.046 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T12:54:22.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:22.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:54:22.195 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:54:22.195 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:54:22.234 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T12:54:22.235 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T12:54:22.235 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T12:54:22.235 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T12:54:22.235 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T12:54:22.237 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T12:54:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T12:54:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T12:54:22.237 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T12:54:22.238 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T12:54:22.241 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:54:22.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:22.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:54:22.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:22.328 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:54:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:client7459 2023-10-16T12:54:22.402 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T12:54:22.402 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:54:22.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:54:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.689+0000 7fed6a27c640 1 Processor -- start 2023-10-16T12:54:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.690+0000 7fed6a27c640 1 -- start start 2023-10-16T12:54:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.691+0000 7fed6a27c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed640ad350 0x7fed64069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.691+0000 7fed6a27c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6406a2e0 0x7fed640c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.691+0000 7fed6a27c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6406a730 con 0x7fed6415d150 2023-10-16T12:54:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.691+0000 7fed6a27c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6415d530 con 0x7fed640ad350 2023-10-16T12:54:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.691+0000 7fed6a27c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640ac890 con 0x7fed6406a2e0 2023-10-16T12:54:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.692+0000 7fed637fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed640ad350 0x7fed64069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.692+0000 7fed637fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed640ad350 0x7fed64069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55020/0 (socket says 172.21.15.60:55020) 2023-10-16T12:54:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.692+0000 7fed637fe640 1 -- 172.21.15.60:0/845100628 learned_addr learned my addr 172.21.15.60:0/845100628 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.692+0000 7fed68819640 1 --2- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6406a2e0 0x7fed640c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:22.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.692+0000 7fed63fff640 1 --1- 172.21.15.60:0/845100628 >> v1:172.21.15.144:6789/0 conn(0x7fed6415d150 0x7fed64160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:32926/0 (socket says 172.21.15.60:32926) 2023-10-16T12:54:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.693+0000 7fed637fe640 1 -- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6406a2e0 msgr2=0x7fed640c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.693+0000 7fed637fe640 1 --2- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6406a2e0 0x7fed640c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.693+0000 7fed637fe640 1 -- 172.21.15.60:0/845100628 >> v1:172.21.15.144:6789/0 conn(0x7fed6415d150 legacy=0x7fed64160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:22.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.693+0000 7fed637fe640 1 -- 172.21.15.60:0/845100628 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed640ad7a0 con 0x7fed640ad350 2023-10-16T12:54:22.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.694+0000 7fed637fe640 1 --2- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed640ad350 0x7fed64069da0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fed58009870 tx=0x7fed5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0abbba952af7271 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed62ffd640 1 -- 172.21.15.60:0/845100628 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58030840 con 0x7fed640ad350 2023-10-16T12:54:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed62ffd640 1 -- 172.21.15.60:0/845100628 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed580309a0 con 0x7fed640ad350 2023-10-16T12:54:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed62ffd640 1 -- 172.21.15.60:0/845100628 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58030d30 con 0x7fed640ad350 2023-10-16T12:54:22.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed6a27c640 1 -- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed640ad350 msgr2=0x7fed64069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed6a27c640 1 --2- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed640ad350 0x7fed64069da0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fed58009870 tx=0x7fed5802b980 comp rx=0 tx=0).stop 2023-10-16T12:54:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed6a27c640 1 -- 172.21.15.60:0/845100628 shutdown_connections 2023-10-16T12:54:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed6a27c640 1 --2- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6406a2e0 0x7fed640c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed6a27c640 1 --2- 172.21.15.60:0/845100628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed640ad350 0x7fed64069da0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.695+0000 7fed6a27c640 1 -- 172.21.15.60:0/845100628 >> 172.21.15.60:0/845100628 conn(0x7fed640ce9b0 msgr2=0x7fed640cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:22.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.696+0000 7fed6a27c640 1 -- 172.21.15.60:0/845100628 shutdown_connections 2023-10-16T12:54:22.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.696+0000 7fed6a27c640 1 -- 172.21.15.60:0/845100628 wait complete. 2023-10-16T12:54:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.697+0000 7fed6a27c640 1 Processor -- start 2023-10-16T12:54:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.698+0000 7fed6a27c640 1 -- start start 2023-10-16T12:54:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.698+0000 7fed6a27c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed6406a2e0 0x7fed64158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.698+0000 7fed6a27c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 0x7fed640af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.698+0000 7fed6a27c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6415d150 0x7fed640af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.698+0000 7fed6a27c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed640aee30 con 0x7fed640ad350 2023-10-16T12:54:22.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed6a27c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed640ae240 con 0x7fed6406a2e0 2023-10-16T12:54:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed6a27c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640ae490 con 0x7fed6415d150 2023-10-16T12:54:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 0x7fed640af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 0x7fed640af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52830/0 (socket says 172.21.15.60:52830) 2023-10-16T12:54:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 -- 172.21.15.60:0/679780782 learned_addr learned my addr 172.21.15.60:0/679780782 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:22.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed63fff640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed6406a2e0 0x7fed64158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed68819640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6415d150 0x7fed640af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 -- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6415d150 msgr2=0x7fed640af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6415d150 0x7fed640af790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 -- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed6406a2e0 msgr2=0x7fed64158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed6406a2e0 0x7fed64158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed637fe640 1 -- 172.21.15.60:0/679780782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed580094d0 con 0x7fed640ad350 2023-10-16T12:54:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.699+0000 7fed68819640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6415d150 0x7fed640af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:54:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.700+0000 7fed637fe640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 0x7fed640af250 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fed58000c00 tx=0x7fed58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.700+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58039c90 con 0x7fed640ad350 2023-10-16T12:54:22.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.700+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed58039df0 con 0x7fed640ad350 2023-10-16T12:54:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.700+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58043640 con 0x7fed640ad350 2023-10-16T12:54:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.700+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed640ae770 con 0x7fed640ad350 2023-10-16T12:54:22.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.700+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed64158fa0 con 0x7fed640ad350 2023-10-16T12:54:22.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.701+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6404e160 con 0x7fed640ad350 2023-10-16T12:54:22.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.702+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed580437a0 con 0x7fed640ad350 2023-10-16T12:54:22.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.704+0000 7fed60ff9640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fed5006af90 0x7fed5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:22.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.704+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed5803e070 con 0x7fed640ad350 2023-10-16T12:54:22.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.705+0000 7fed63fff640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fed5006af90 0x7fed5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:22.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.706+0000 7fed63fff640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fed5006af90 0x7fed5006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fed54007ef0 tx=0x7fed54007e80 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:22.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.709+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed58077750 con 0x7fed640ad350 2023-10-16T12:54:22.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.930+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 --> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fed64159130 con 0x7fed5006af90 2023-10-16T12:54:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.934+0000 7fed60ff9640 1 -- 172.21.15.60:0/679780782 <== mgr.7225 v2:172.21.15.144:6832/3094169155 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fed64159130 con 0x7fed5006af90 2023-10-16T12:54:22.935 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T12:54:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T12:54:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T12:54:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T12:54:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T12:54:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.936+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fed5006af90 msgr2=0x7fed5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:22.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.936+0000 7fed6a27c640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fed5006af90 0x7fed5006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fed54007ef0 tx=0x7fed54007e80 comp rx=0 tx=0).stop 2023-10-16T12:54:22.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.936+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 msgr2=0x7fed640af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:22.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.936+0000 7fed6a27c640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 0x7fed640af250 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fed58000c00 tx=0x7fed58004730 comp rx=0 tx=0).stop 2023-10-16T12:54:22.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 shutdown_connections 2023-10-16T12:54:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7fed5006af90 0x7fed5006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed6415d150 0x7fed640af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed640ad350 0x7fed640af250 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 --2- 172.21.15.60:0/679780782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed6406a2e0 0x7fed64158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:22.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 >> 172.21.15.60:0/679780782 conn(0x7fed640ce9b0 msgr2=0x7fed640c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:22.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 shutdown_connections 2023-10-16T12:54:22.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:22.937+0000 7fed6a27c640 1 -- 172.21.15.60:0/679780782 wait complete. 2023-10-16T12:54:22.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:54:23.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.186+0000 7f5f0874e640 1 Processor -- start 2023-10-16T12:54:23.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.186+0000 7f5f0874e640 1 -- start start 2023-10-16T12:54:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f0874e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f0015ad70 0x7f5f00069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f0874e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 0x7f5f0006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f0874e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000cb030 con 0x7f5f000c9ca0 2023-10-16T12:54:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f0874e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f000c91c0 con 0x7f5f0015ad70 2023-10-16T12:54:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f0874e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000c94a0 con 0x7f5f000caac0 2023-10-16T12:54:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f05ce9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5f000c9ca0 0x7f5f000ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56128/0 (socket says 172.21.15.60:56128) 2023-10-16T12:54:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.188+0000 7f5f05ce9640 1 -- 172.21.15.60:0/569600696 learned_addr learned my addr 172.21.15.60:0/569600696 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 0x7f5f0006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f064ea640 1 --2- 172.21.15.60:0/569600696 >> v2:172.21.15.144:3300/0 conn(0x7f5f0015ad70 0x7f5f00069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441075946 0 0) 0x7f5f000cb030 con 0x7f5f000c9ca0 2023-10-16T12:54:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eec003520 con 0x7f5f000c9ca0 2023-10-16T12:54:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400181151 0 0) 0x7f5eec003520 con 0x7f5f000c9ca0 2023-10-16T12:54:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 msgr2=0x7f5f0006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 0x7f5f0006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 >> v2:172.21.15.144:3300/0 conn(0x7f5f0015ad70 msgr2=0x7f5f00069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 --2- 172.21.15.60:0/569600696 >> v2:172.21.15.144:3300/0 conn(0x7f5f0015ad70 0x7f5f00069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000c9780 con 0x7f5f000c9ca0 2023-10-16T12:54:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5ef0001670 con 0x7f5f000c9ca0 2023-10-16T12:54:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.189+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 >> v1:172.21.15.60:6789/0 conn(0x7f5f000c9ca0 legacy=0x7f5f000ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f054e8640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 0x7f5eec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f054e8640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5eec003f40 0x7f5eec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f054e8640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5eec004d00 0x7f5eec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000c91c0 con 0x7f5eec004d00 2023-10-16T12:54:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000c94a0 con 0x7f5eec003f40 2023-10-16T12:54:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec00b980 con 0x7f5f000caac0 2023-10-16T12:54:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5eec004d00 0x7f5eec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f05ce9640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5eec003f40 0x7f5eec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f064ea640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 0x7f5eec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f06ceb640 1 -- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 msgr2=0x7f5eec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f000caac0 0x7f5eec003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f06ceb640 1 -- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5eec003f40 msgr2=0x7f5eec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5eec003f40 0x7f5eec0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.190+0000 7f5f06ceb640 1 -- 172.21.15.60:0/569600696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec003520 con 0x7f5eec004d00 2023-10-16T12:54:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.191+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5eec004d00 0x7f5eec005180 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800e170 tx=0x7f5eec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c946e1e9743cc0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef802cbf0 con 0x7f5eec004d00 2023-10-16T12:54:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef800f040 con 0x7f5eec004d00 2023-10-16T12:54:23.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f054e8640 1 -- 172.21.15.60:0/569600696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8034440 con 0x7f5eec004d00 2023-10-16T12:54:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f0874e640 1 -- 172.21.15.60:0/569600696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5eec004d00 msgr2=0x7f5eec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f0874e640 1 --2- 172.21.15.60:0/569600696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5eec004d00 0x7f5eec005180 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800e170 tx=0x7f5eec0057f0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f06ceb640 1 -- 172.21.15.60:0/569600696 reap_dead start 2023-10-16T12:54:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f0874e640 1 -- 172.21.15.60:0/569600696 shutdown_connections 2023-10-16T12:54:23.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.192+0000 7f5f0874e640 1 -- 172.21.15.60:0/569600696 >> 172.21.15.60:0/569600696 conn(0x7f5f000ce9b0 msgr2=0x7f5f000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.193+0000 7f5f0874e640 1 -- 172.21.15.60:0/569600696 shutdown_connections 2023-10-16T12:54:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.193+0000 7f5f0874e640 1 -- 172.21.15.60:0/569600696 wait complete. 2023-10-16T12:54:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.195+0000 7f5f0874e640 1 Processor -- start 2023-10-16T12:54:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.195+0000 7f5f0874e640 1 -- start start 2023-10-16T12:54:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f0874e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f00155ea0 0x7f5f000b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f0874e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f000af1e0 0x7f5f0006b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f0874e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 0x7f5f000ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f0874e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000c3a60 con 0x7f5f0006b570 2023-10-16T12:54:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f0874e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0006ba00 con 0x7f5f000af1e0 2023-10-16T12:54:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f0874e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000af670 con 0x7f5f00155ea0 2023-10-16T12:54:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f06ceb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 0x7f5f000ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f06ceb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 0x7f5f000ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52848/0 (socket says 172.21.15.60:52848) 2023-10-16T12:54:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f06ceb640 1 -- 172.21.15.60:0/4087952251 learned_addr learned my addr 172.21.15.60:0/4087952251 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f064ea640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f00155ea0 0x7f5f000b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.196+0000 7f5f05ce9640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f000af1e0 0x7f5f0006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f06ceb640 1 -- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f00155ea0 msgr2=0x7f5f000b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f00155ea0 0x7f5f000b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f06ceb640 1 -- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f000af1e0 msgr2=0x7f5f0006b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f000af1e0 0x7f5f0006b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f06ceb640 1 -- 172.21.15.60:0/4087952251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000c30c0 con 0x7f5f0006b570 2023-10-16T12:54:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f06ceb640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 0x7f5f000ad8e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8037040 tx=0x7f5ef8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8003b90 con 0x7f5f0006b570 2023-10-16T12:54:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef8003d10 con 0x7f5f0006b570 2023-10-16T12:54:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f000c33a0 con 0x7f5f0006b570 2023-10-16T12:54:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8034ac0 con 0x7f5f0006b570 2023-10-16T12:54:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.197+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f00156280 con 0x7f5f0006b570 2023-10-16T12:54:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.199+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0004e160 con 0x7f5f0006b570 2023-10-16T12:54:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.200+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 76) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ef8034c20 con 0x7f5f0006b570 2023-10-16T12:54:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.201+0000 7f5ef6ffd640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f5ee406af90 0x7f5ee406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.201+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ef80aae10 con 0x7f5f0006b570 2023-10-16T12:54:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.201+0000 7f5f064ea640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f5ee406af90 0x7f5ee406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.203+0000 7f5f064ea640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f5ee406af90 0x7f5ee406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009a90 tx=0x7f5efc008040 comp rx=0 tx=0).ready entity=mgr.7225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:23.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.205+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ef8075ef0 con 0x7f5f0006b570 2023-10-16T12:54:23.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.457+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5f000c3ba0 con 0x7f5f0006b570 2023-10-16T12:54:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.969+0000 7f5ef6ffd640 1 -- 172.21.15.60:0/4087952251 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v77) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5ef80450a0 con 0x7f5f0006b570 2023-10-16T12:54:23.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f5ee406af90 msgr2=0x7f5ee406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:23.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f5ee406af90 0x7f5ee406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009a90 tx=0x7f5efc008040 comp rx=0 tx=0).stop 2023-10-16T12:54:23.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 msgr2=0x7f5f000ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:23.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 0x7f5f000ad8e0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8037040 tx=0x7f5ef8009750 comp rx=0 tx=0).stop 2023-10-16T12:54:23.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 shutdown_connections 2023-10-16T12:54:23.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f5ee406af90 0x7f5ee406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f0006b570 0x7f5f000ad8e0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f000af1e0 0x7f5f0006b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 --2- 172.21.15.60:0/4087952251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f00155ea0 0x7f5f000b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:23.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.973+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 >> 172.21.15.60:0/4087952251 conn(0x7f5f000ce9b0 msgr2=0x7f5f000b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:23.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.975+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 shutdown_connections 2023-10-16T12:54:23.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:23.977+0000 7f5f0874e640 1 -- 172.21.15.60:0/4087952251 wait complete. 2023-10-16T12:54:24.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:54:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.224+0000 7f86957dd640 1 Processor -- start 2023-10-16T12:54:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.224+0000 7f86957dd640 1 -- start start 2023-10-16T12:54:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f86957dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f869015aaa0 0x7f86900b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f86957dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86900ca8a0 0x7f86900b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f86957dd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86900cad00 con 0x7f8690070240 2023-10-16T12:54:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f86957dd640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900b2d00 con 0x7f86900ca8a0 2023-10-16T12:54:24.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f86957dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900b2fe0 con 0x7f869015aaa0 2023-10-16T12:54:24.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f868f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86900ca8a0 0x7f86900b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f868f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86900ca8a0 0x7f86900b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52874/0 (socket says 172.21.15.60:52874) 2023-10-16T12:54:24.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.226+0000 7f868f7fe640 1 -- 172.21.15.60:0/41133489 learned_addr learned my addr 172.21.15.60:0/41133489 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:24.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.227+0000 7f868ffff640 1 --2- 172.21.15.60:0/41133489 >> v2:172.21.15.144:3300/0 conn(0x7f869015aaa0 0x7f86900b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.227+0000 7f868f7fe640 1 -- 172.21.15.60:0/41133489 >> v2:172.21.15.144:3300/0 conn(0x7f869015aaa0 msgr2=0x7f86900b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:24.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.227+0000 7f868f7fe640 1 --2- 172.21.15.60:0/41133489 >> v2:172.21.15.144:3300/0 conn(0x7f869015aaa0 0x7f86900b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.227+0000 7f868f7fe640 1 -- 172.21.15.60:0/41133489 >> v1:172.21.15.144:6789/0 conn(0x7f8690070240 legacy=0x7f86900c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.227+0000 7f868f7fe640 1 -- 172.21.15.60:0/41133489 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86900b32c0 con 0x7f86900ca8a0 2023-10-16T12:54:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.228+0000 7f868f7fe640 1 --2- 172.21.15.60:0/41133489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86900b3da0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f86900ccef0 tx=0x7f868402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe59767acdb7e86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.228+0000 7f868effd640 1 -- 172.21.15.60:0/41133489 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684031870 con 0x7f86900ca8a0 2023-10-16T12:54:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.228+0000 7f868effd640 1 -- 172.21.15.60:0/41133489 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86840319d0 con 0x7f86900ca8a0 2023-10-16T12:54:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.228+0000 7f868effd640 1 -- 172.21.15.60:0/41133489 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684031d60 con 0x7f86900ca8a0 2023-10-16T12:54:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 -- 172.21.15.60:0/41133489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 msgr2=0x7f86900b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 --2- 172.21.15.60:0/41133489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86900b3da0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f86900ccef0 tx=0x7f868402b910 comp rx=0 tx=0).stop 2023-10-16T12:54:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 -- 172.21.15.60:0/41133489 shutdown_connections 2023-10-16T12:54:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 --2- 172.21.15.60:0/41133489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86900b3da0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 --2- 172.21.15.60:0/41133489 >> v2:172.21.15.144:3300/0 conn(0x7f869015aaa0 0x7f86900b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 -- 172.21.15.60:0/41133489 >> 172.21.15.60:0/41133489 conn(0x7f86900cc230 msgr2=0x7f86900cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 -- 172.21.15.60:0/41133489 shutdown_connections 2023-10-16T12:54:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.229+0000 7f86957dd640 1 -- 172.21.15.60:0/41133489 wait complete. 2023-10-16T12:54:24.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.231+0000 7f86957dd640 1 Processor -- start 2023-10-16T12:54:24.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.231+0000 7f86957dd640 1 -- start start 2023-10-16T12:54:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f86957dd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8690070240 0x7f86901544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f86957dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86901f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f86957dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f869015aaa0 0x7f86900b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f86957dd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690150590 con 0x7f86900ca8a0 2023-10-16T12:54:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f86957dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901f21e0 con 0x7f869015aaa0 2023-10-16T12:54:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f86957dd640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86900cf120 con 0x7f8690070240 2023-10-16T12:54:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f868f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86901f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f868f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86901f1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52884/0 (socket says 172.21.15.60:52884) 2023-10-16T12:54:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.232+0000 7f868f7fe640 1 -- 172.21.15.60:0/877038682 learned_addr learned my addr 172.21.15.60:0/877038682 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868ffff640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8690070240 0x7f86901544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f8694fdc640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f869015aaa0 0x7f86900b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868f7fe640 1 -- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8690070240 msgr2=0x7f86901544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868f7fe640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8690070240 0x7f86901544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868f7fe640 1 -- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f869015aaa0 msgr2=0x7f86900b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868f7fe640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f869015aaa0 0x7f86900b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868f7fe640 1 -- 172.21.15.60:0/877038682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86840094d0 con 0x7f86900ca8a0 2023-10-16T12:54:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.233+0000 7f868f7fe640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86901f1ca0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f8684000c00 tx=0x7f86840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.234+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868403abe0 con 0x7f86900ca8a0 2023-10-16T12:54:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.234+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868403ad40 con 0x7f86900ca8a0 2023-10-16T12:54:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.234+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684044630 con 0x7f86900ca8a0 2023-10-16T12:54:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.234+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86900ad440 con 0x7f86900ca8a0 2023-10-16T12:54:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.234+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86900ada00 con 0x7f86900ca8a0 2023-10-16T12:54:24.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.235+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869004e9b0 con 0x7f86900ca8a0 2023-10-16T12:54:24.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.236+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 77) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f868404d450 con 0x7f86900ca8a0 2023-10-16T12:54:24.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.237+0000 7f868cff9640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 0x7f867c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.237+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f868402d040 con 0x7f86900ca8a0 2023-10-16T12:54:24.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.237+0000 7f868ffff640 1 -- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 msgr2=0x7f867c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:24.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.237+0000 7f868ffff640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 0x7f867c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:54:24.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.241+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8684076d60 con 0x7f86900ca8a0 2023-10-16T12:54:24.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.437+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f86900ad230 con 0x7f86900ca8a0 2023-10-16T12:54:24.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.438+0000 7f868ffff640 1 -- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 msgr2=0x7f867c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:24.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.438+0000 7f868ffff640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 0x7f867c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:54:24.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.442+0000 7f868cff9640 1 -- 172.21.15.60:0/877038682 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f868407a380 con 0x7f86900ca8a0 2023-10-16T12:54:24.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 msgr2=0x7f867c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:54:24.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 0x7f867c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 msgr2=0x7f86901f1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86901f1ca0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f8684000c00 tx=0x7f86840046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 shutdown_connections 2023-10-16T12:54:24.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f867c06afe0 0x7f867c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f869015aaa0 0x7f86900b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86900ca8a0 0x7f86901f1ca0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 --2- 172.21.15.60:0/877038682 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8690070240 0x7f86901544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 >> 172.21.15.60:0/877038682 conn(0x7f86900cc230 msgr2=0x7f869015bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:24.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 shutdown_connections 2023-10-16T12:54:24.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.445+0000 7f86957dd640 1 -- 172.21.15.60:0/877038682 wait complete. 2023-10-16T12:54:24.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:54:24.522 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:24.519+0000 7fd9e323a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:54:24.543 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:24.541+0000 7f68f4432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:54:24.575 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:24.572+0000 7fd9e323a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:54:24.602 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:24.601+0000 7f68f4432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:54:24.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.681+0000 7f100db11640 1 Processor -- start 2023-10-16T12:54:24.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.681+0000 7f100db11640 1 -- start start 2023-10-16T12:54:24.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100db11640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f10080d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100db11640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10080b8480 0x7f10080cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100db11640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100815d150 con 0x7f10080d0be0 2023-10-16T12:54:24.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100db11640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10080d1180 con 0x7f10080b8480 2023-10-16T12:54:24.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100db11640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d1460 con 0x7f100815cd70 2023-10-16T12:54:24.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f1007fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10080b8480 0x7f10080cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100d310640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f10080d0be0 0x7f10080b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56130/0 (socket says 172.21.15.60:56130) 2023-10-16T12:54:24.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100d310640 1 -- 172.21.15.60:0/3418916821 learned_addr learned my addr 172.21.15.60:0/3418916821 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:24.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f100cb0f640 1 --2- 172.21.15.60:0/3418916821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f10080d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f1007fff640 1 -- 172.21.15.60:0/3418916821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 msgr2=0x7f10080d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:24.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f1007fff640 1 --2- 172.21.15.60:0/3418916821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f10080d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f1007fff640 1 -- 172.21.15.60:0/3418916821 >> v1:172.21.15.60:6789/0 conn(0x7f10080d0be0 legacy=0x7f10080b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f1007fff640 1 -- 172.21.15.60:0/3418916821 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080d1740 con 0x7f10080b8480 2023-10-16T12:54:24.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.683+0000 7f10077fe640 1 -- 172.21.15.60:0/3418916821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060470804 0 0) 0x7f100815d150 con 0x7f10080d0be0 2023-10-16T12:54:24.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.685+0000 7f1007fff640 1 --2- 172.21.15.60:0/3418916821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080b8480 0x7f10080cd7c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40097a0 tx=0x7f0ff402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231d5943ab29ff54 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:24.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.685+0000 7f10077fe640 1 -- 172.21.15.60:0/3418916821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff402d040 con 0x7f10080b8480 2023-10-16T12:54:24.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.685+0000 7f10077fe640 1 -- 172.21.15.60:0/3418916821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff4031810 con 0x7f10080b8480 2023-10-16T12:54:24.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.685+0000 7f10077fe640 1 -- 172.21.15.60:0/3418916821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4031ba0 con 0x7f10080b8480 2023-10-16T12:54:24.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 -- 172.21.15.60:0/3418916821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080b8480 msgr2=0x7f10080cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 --2- 172.21.15.60:0/3418916821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080b8480 0x7f10080cd7c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40097a0 tx=0x7f0ff402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 -- 172.21.15.60:0/3418916821 shutdown_connections 2023-10-16T12:54:24.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 --2- 172.21.15.60:0/3418916821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080b8480 0x7f10080cd7c0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 --2- 172.21.15.60:0/3418916821 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f10080d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 -- 172.21.15.60:0/3418916821 >> 172.21.15.60:0/3418916821 conn(0x7f10080cf570 msgr2=0x7f10080cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:24.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 -- 172.21.15.60:0/3418916821 shutdown_connections 2023-10-16T12:54:24.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.686+0000 7f100db11640 1 -- 172.21.15.60:0/3418916821 wait complete. 2023-10-16T12:54:24.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.688+0000 7f100db11640 1 Processor -- start 2023-10-16T12:54:24.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.688+0000 7f100db11640 1 -- start start 2023-10-16T12:54:24.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100db11640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10080b8480 0x7f100815d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100db11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 0x7f100806bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100db11640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f100806c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100db11640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080b4d00 con 0x7f10080d0be0 2023-10-16T12:54:24.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100db11640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008160350 con 0x7f10080b8480 2023-10-16T12:54:24.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100db11640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10081572b0 con 0x7f100815cd70 2023-10-16T12:54:24.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f1007fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 0x7f100806bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f1007fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 0x7f100806bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52896/0 (socket says 172.21.15.60:52896) 2023-10-16T12:54:24.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f1007fff640 1 -- 172.21.15.60:0/3341681726 learned_addr learned my addr 172.21.15.60:0/3341681726 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:24.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100cb0f640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10080b8480 0x7f100815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.689+0000 7f100d310640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f100806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:24.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f1007fff640 1 -- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 msgr2=0x7f100806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f1007fff640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f100806c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f1007fff640 1 -- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10080b8480 msgr2=0x7f100815d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f1007fff640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10080b8480 0x7f100815d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f1007fff640 1 -- 172.21.15.60:0/3341681726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4009400 con 0x7f10080d0be0 2023-10-16T12:54:24.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f1007fff640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 0x7f100806bbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4000c00 tx=0x7f0ff40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:24.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff403ad60 con 0x7f10080d0be0 2023-10-16T12:54:24.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff40443f0 con 0x7f10080d0be0 2023-10-16T12:54:24.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1008160070 con 0x7f10080d0be0 2023-10-16T12:54:24.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.690+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff40447a0 con 0x7f10080d0be0 2023-10-16T12:54:24.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.691+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1008157830 con 0x7f10080d0be0 2023-10-16T12:54:24.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.692+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1008157a20 con 0x7f10080d0be0 2023-10-16T12:54:24.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.694+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 77) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0ff404d450 con 0x7f10080d0be0 2023-10-16T12:54:24.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.695+0000 7f10057fa640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 0x7f0fec06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:24.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.695+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ff40abac0 con 0x7f10080d0be0 2023-10-16T12:54:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.695+0000 7f100cb0f640 1 -- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 msgr2=0x7f0fec06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.695+0000 7f100cb0f640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 0x7f0fec06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:54:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.699+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ff40769e0 con 0x7f10080d0be0 2023-10-16T12:54:24.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.893+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f100804e9b0 con 0x7f10080d0be0 2023-10-16T12:54:24.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.896+0000 7f100cb0f640 1 -- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 msgr2=0x7f0fec06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:24.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.896+0000 7f100cb0f640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 0x7f0fec06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:54:24.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.899+0000 7f10057fa640 1 -- 172.21.15.60:0/3341681726 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0ff407a000 con 0x7f10080d0be0 2023-10-16T12:54:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 msgr2=0x7f0fec06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:54:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 0x7f0fec06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 msgr2=0x7f100806bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 0x7f100806bbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4000c00 tx=0x7f0ff40046c0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 shutdown_connections 2023-10-16T12:54:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7f0fec06afe0 0x7f0fec06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f100815cd70 0x7f100806c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10080d0be0 0x7f100806bbd0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 --2- 172.21.15.60:0/3341681726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10080b8480 0x7f100815d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 >> 172.21.15.60:0/3341681726 conn(0x7f10080cf570 msgr2=0x7f10081e7910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 shutdown_connections 2023-10-16T12:54:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:24.902+0000 7f100db11640 1 -- 172.21.15.60:0/3341681726 wait complete. 2023-10-16T12:54:24.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:54:24.923 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:54:24.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-10-16T12:54:25.023 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.021+0000 7fd9e323a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:54:25.075 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.072+0000 7fd9e323a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:54:25.085 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.083+0000 7f68f4432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:54:25.125 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.122+0000 7fd9e323a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:54:25.140 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.139+0000 7f68f4432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:54:25.184 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.181+0000 7fd9e323a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:54:25.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.191+0000 7ff655147640 1 Processor -- start 2023-10-16T12:54:25.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.192+0000 7ff655147640 1 -- start start 2023-10-16T12:54:25.194 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.192+0000 7f68f4432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:54:25.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff655147640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff650159ab0 0x7ff6500ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:25.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff655147640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff650159e90 0x7ff65006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:25.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff655147640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6500cd1f0 con 0x7ff65006c140 2023-10-16T12:54:25.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff655147640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff65015d890 con 0x7ff650159ab0 2023-10-16T12:54:25.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff655147640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6500cd4d0 con 0x7ff650159e90 2023-10-16T12:54:25.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff64ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff650159ab0 0x7ff6500ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:25.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff64ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff650159ab0 0x7ff6500ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52900/0 (socket says 172.21.15.60:52900) 2023-10-16T12:54:25.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff64f59e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff65006c140 0x7ff65015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56140/0 (socket says 172.21.15.60:56140) 2023-10-16T12:54:25.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.193+0000 7ff64ed9d640 1 -- 172.21.15.60:0/3925437775 learned_addr learned my addr 172.21.15.60:0/3925437775 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:25.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.194+0000 7ff64659c640 1 --2- 172.21.15.60:0/3925437775 >> v2:172.21.15.144:3300/0 conn(0x7ff650159e90 0x7ff65006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:25.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.194+0000 7ff64ed9d640 1 -- 172.21.15.60:0/3925437775 >> v2:172.21.15.144:3300/0 conn(0x7ff650159e90 msgr2=0x7ff65006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:25.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.194+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/3925437775 >> v2:172.21.15.144:3300/0 conn(0x7ff650159e90 0x7ff65006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:25.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.194+0000 7ff64ed9d640 1 -- 172.21.15.60:0/3925437775 >> v1:172.21.15.60:6789/0 conn(0x7ff65006c140 legacy=0x7ff65015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:25.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.194+0000 7ff64ed9d640 1 -- 172.21.15.60:0/3925437775 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6500cd7b0 con 0x7ff650159ab0 2023-10-16T12:54:25.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.194+0000 7ff64e59c640 1 -- 172.21.15.60:0/3925437775 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203596170 0 0) 0x7ff6500cd1f0 con 0x7ff65006c140 2023-10-16T12:54:25.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.195+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/3925437775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159ab0 0x7ff6500ce370 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7ff65006d900 tx=0x7ff64002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64656058e59cd24e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:25.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.195+0000 7ff64e59c640 1 -- 172.21.15.60:0/3925437775 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6400317e0 con 0x7ff650159ab0 2023-10-16T12:54:25.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.195+0000 7ff64e59c640 1 -- 172.21.15.60:0/3925437775 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff640031940 con 0x7ff650159ab0 2023-10-16T12:54:25.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.195+0000 7ff64e59c640 1 -- 172.21.15.60:0/3925437775 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff640031cd0 con 0x7ff650159ab0 2023-10-16T12:54:25.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 -- 172.21.15.60:0/3925437775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159ab0 msgr2=0x7ff6500ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:25.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 --2- 172.21.15.60:0/3925437775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159ab0 0x7ff6500ce370 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7ff65006d900 tx=0x7ff64002b880 comp rx=0 tx=0).stop 2023-10-16T12:54:25.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 -- 172.21.15.60:0/3925437775 shutdown_connections 2023-10-16T12:54:25.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 --2- 172.21.15.60:0/3925437775 >> v2:172.21.15.144:3300/0 conn(0x7ff650159e90 0x7ff65006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:25.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 --2- 172.21.15.60:0/3925437775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159ab0 0x7ff6500ce370 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:25.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 -- 172.21.15.60:0/3925437775 >> 172.21.15.60:0/3925437775 conn(0x7ff6500ce9b0 msgr2=0x7ff6500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:25.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 -- 172.21.15.60:0/3925437775 shutdown_connections 2023-10-16T12:54:25.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.196+0000 7ff655147640 1 -- 172.21.15.60:0/3925437775 wait complete. 2023-10-16T12:54:25.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.198+0000 7ff655147640 1 Processor -- start 2023-10-16T12:54:25.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.198+0000 7ff655147640 1 -- start start 2023-10-16T12:54:25.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff655147640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff65006c140 0x7ff650150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:25.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff655147640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff650159ab0 0x7ff65015c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:25.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff655147640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159e90 0x7ff650151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:25.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff655147640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500b22f0 con 0x7ff650159e90 2023-10-16T12:54:25.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff655147640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500b1670 con 0x7ff65006c140 2023-10-16T12:54:25.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff655147640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500b1950 con 0x7ff650159ab0 2023-10-16T12:54:25.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff64f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159e90 0x7ff650151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:25.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.199+0000 7ff64f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159e90 0x7ff650151840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52912/0 (socket says 172.21.15.60:52912) 2023-10-16T12:54:25.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 -- 172.21.15.60:0/1434874447 learned_addr learned my addr 172.21.15.60:0/1434874447 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:25.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff65006c140 0x7ff650150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:25.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64659c640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff650159ab0 0x7ff65015c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:25.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff650159ab0 msgr2=0x7ff65015c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:25.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff650159ab0 0x7ff65015c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:25.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff65006c140 msgr2=0x7ff650150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:25.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff65006c140 0x7ff650150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:25.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 -- 172.21.15.60:0/1434874447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400094d0 con 0x7ff650159e90 2023-10-16T12:54:25.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.200+0000 7ff64f59e640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159e90 0x7ff650151840 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7ff64800c7a0 tx=0x7ff64800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:25.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.201+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64800a640 con 0x7ff650159e90 2023-10-16T12:54:25.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.201+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64800a7a0 con 0x7ff650159e90 2023-10-16T12:54:25.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.201+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6500b1c30 con 0x7ff650159e90 2023-10-16T12:54:25.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.201+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64800abd0 con 0x7ff650159e90 2023-10-16T12:54:25.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.201+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff650150c00 con 0x7ff650159e90 2023-10-16T12:54:25.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.202+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6500adac0 con 0x7ff650159e90 2023-10-16T12:54:25.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.203+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 77) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff64800ad30 con 0x7ff650159e90 2023-10-16T12:54:25.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.204+0000 7ff647fff640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 0x7ff63006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:25.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.204+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff648088ff0 con 0x7ff650159e90 2023-10-16T12:54:25.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.204+0000 7ff64ed9d640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 msgr2=0x7ff63006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:25.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.204+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 0x7ff63006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:54:25.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.208+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6480bbfe0 con 0x7ff650159e90 2023-10-16T12:54:25.233 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.230+0000 7fd9e323a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:54:25.261 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.259+0000 7f68f4432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:54:25.313 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.311+0000 7f68f4432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:54:25.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.339+0000 7fd9e323a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:54:25.402 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.400+0000 7fd9e323a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:54:25.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.405+0000 7ff64ed9d640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 msgr2=0x7ff63006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.405+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 0x7ff63006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:54:25.430 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.428+0000 7f68f4432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:54:25.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.432+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 --> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6500b1ac0 con 0x7ff63006afe0 2023-10-16T12:54:25.450 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.448+0000 7fd9e323a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:54:25.496 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.494+0000 7f68f4432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:54:25.507 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.504+0000 7fd9e323a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:54:25.547 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.546+0000 7f68f4432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:54:25.557 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.554+0000 7fd9e323a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:54:25.606 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.605+0000 7f68f4432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:54:25.659 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:25.657+0000 7f68f4432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:54:25.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.806+0000 7ff64ed9d640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 msgr2=0x7ff63006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:25.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:25.806+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 0x7ff63006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:54:25.931 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:25.928+0000 7fd9e323a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:54:26.059 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.058+0000 7f68f4432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:54:26.089 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.086+0000 7fd9e323a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:54:26.194 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.192+0000 7fd9e323a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:54:26.227 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.226+0000 7f68f4432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:54:26.244 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.242+0000 7fd9e323a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:54:26.341 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.339+0000 7f68f4432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:54:26.350 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.347+0000 7fd9e323a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:54:26.393 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.391+0000 7f68f4432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:54:26.455 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.452+0000 7fd9e323a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:54:26.506 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.504+0000 7f68f4432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:54:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:26.607+0000 7ff64ed9d640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 msgr2=0x7ff63006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3094169155 2023-10-16T12:54:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:26.607+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 0x7ff63006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:54:26.610 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.607+0000 7fd9e323a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:54:26.617 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.616+0000 7f68f4432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:54:26.658 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.655+0000 7fd9e323a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:54:26.777 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:26.773+0000 7fd9e323a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:54:26.782 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.780+0000 7f68f4432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:54:26.834 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.832+0000 7f68f4432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:54:26.959 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:26.958+0000 7f68f4432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:54:27.603 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:27.601+0000 7fd9e323a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:54:27.835 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:27.833+0000 7f68f4432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:54:27.895 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:27.892+0000 7fd9e323a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:54:27.951 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:27.948+0000 7fd9e323a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:54:28.065 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.062+0000 7fd9e323a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:54:28.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:28.084+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 78) v1 ==== 88957+0+0 (secure 0 0 0) 0x7ff648054370 con 0x7ff650159e90 2023-10-16T12:54:28.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:28.085+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 msgr2=0x7ff63006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:54:28.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:28.085+0000 7ff647fff640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/3094169155,v1:172.21.15.144:6833/3094169155] conn(0x7ff63006afe0 0x7ff63006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:28.156 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:28.154+0000 7f68f4432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:54:28.218 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:28.217+0000 7f68f4432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:54:28.336 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd96153b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.336 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd96153b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.336 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd96153b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.336 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd96153b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd96153b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd95fcf8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd95fcf8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd95fcf8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.337 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd95fcf8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.338 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:54:28.333+0000 7fd95fcf8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:54:28.345 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T12:54:28.343+0000 7f68f4432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:54:28.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:28.701 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:29.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.097+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 79) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff64805fad0 con 0x7ff650159e90 2023-10-16T12:54:29.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.098+0000 7ff647fff640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7ff6300358e0 0x7ff630074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.098+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 --> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6500b1ac0 con 0x7ff6300358e0 2023-10-16T12:54:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.102+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7ff6300358e0 0x7ff630074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:29.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.114+0000 7ff64ed9d640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7ff6300358e0 0x7ff630074ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff640036000 tx=0x7ff64003d040 comp rx=0 tx=0).ready entity=mgr.6731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:29.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.284+0000 7ff647fff640 1 -- 172.21.15.60:0/1434874447 <== mgr.6731 v2:172.21.15.144:6832/110255757 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff6500b1ac0 con 0x7ff6300358e0 2023-10-16T12:54:29.286 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1 2023-10-16T12:54:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.287+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7ff6300358e0 msgr2=0x7ff630074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.287+0000 7ff655147640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7ff6300358e0 0x7ff630074ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff640036000 tx=0x7ff64003d040 comp rx=0 tx=0).stop 2023-10-16T12:54:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.287+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159e90 msgr2=0x7ff650151840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.287+0000 7ff655147640 1 --2- 172.21.15.60:0/1434874447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff650159e90 0x7ff650151840 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7ff64800c7a0 tx=0x7ff64800cc60 comp rx=0 tx=0).stop 2023-10-16T12:54:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.287+0000 7ff64f59e640 1 -- 172.21.15.60:0/1434874447 reap_dead start 2023-10-16T12:54:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.288+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 shutdown_connections 2023-10-16T12:54:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.288+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 >> 172.21.15.60:0/1434874447 conn(0x7ff6500ce9b0 msgr2=0x7ff65006b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.288+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 shutdown_connections 2023-10-16T12:54:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.288+0000 7ff655147640 1 -- 172.21.15.60:0/1434874447 wait complete. 2023-10-16T12:54:29.311 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-16T12:54:29.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-10-16T12:54:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.539+0000 7fc649264640 1 Processor -- start 2023-10-16T12:54:29.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.540+0000 7fc649264640 1 -- start start 2023-10-16T12:54:29.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.541+0000 7fc649264640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc64415cf50 0x7fc6440d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.541+0000 7fc649264640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b8a10 0x7fc6440d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc649264640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64415d330 con 0x7fc6440b4750 2023-10-16T12:54:29.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc649264640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6440d1680 con 0x7fc64415cf50 2023-10-16T12:54:29.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc649264640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6440d1960 con 0x7fc6440b8a10 2023-10-16T12:54:29.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc642ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc64415cf50 0x7fc6440d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:29.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc6437fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc6440b4750 0x7fc64406bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45722/0 (socket says 172.21.15.60:45722) 2023-10-16T12:54:29.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc642ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc64415cf50 0x7fc6440d2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43852/0 (socket says 172.21.15.60:43852) 2023-10-16T12:54:29.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc6437fe640 1 -- 172.21.15.60:0/1703188569 learned_addr learned my addr 172.21.15.60:0/1703188569 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T12:54:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc63bfff640 1 --2- 172.21.15.60:0/1703188569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b8a10 0x7fc6440d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc642ffd640 1 -- 172.21.15.60:0/1703188569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b8a10 msgr2=0x7fc6440d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc642ffd640 1 --2- 172.21.15.60:0/1703188569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b8a10 0x7fc6440d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc642ffd640 1 -- 172.21.15.60:0/1703188569 >> v1:172.21.15.60:6789/0 conn(0x7fc6440b4750 legacy=0x7fc64406bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc642ffd640 1 -- 172.21.15.60:0/1703188569 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6440d1c40 con 0x7fc64415cf50 2023-10-16T12:54:29.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.542+0000 7fc6427fc640 1 -- 172.21.15.60:0/1703188569 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783055391 0 0) 0x7fc64415d330 con 0x7fc6440b4750 2023-10-16T12:54:29.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.544+0000 7fc642ffd640 1 --2- 172.21.15.60:0/1703188569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc64415cf50 0x7fc6440d2720 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097a0 tx=0x7fc63402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781f9588e2a81810 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.544+0000 7fc6427fc640 1 -- 172.21.15.60:0/1703188569 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63402d040 con 0x7fc64415cf50 2023-10-16T12:54:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.544+0000 7fc6427fc640 1 -- 172.21.15.60:0/1703188569 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc634031810 con 0x7fc64415cf50 2023-10-16T12:54:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.544+0000 7fc6427fc640 1 -- 172.21.15.60:0/1703188569 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634031ba0 con 0x7fc64415cf50 2023-10-16T12:54:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.544+0000 7fc649264640 1 -- 172.21.15.60:0/1703188569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc64415cf50 msgr2=0x7fc6440d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.544+0000 7fc649264640 1 --2- 172.21.15.60:0/1703188569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc64415cf50 0x7fc6440d2720 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097a0 tx=0x7fc63402b8b0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.545+0000 7fc649264640 1 -- 172.21.15.60:0/1703188569 shutdown_connections 2023-10-16T12:54:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.545+0000 7fc649264640 1 --2- 172.21.15.60:0/1703188569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b8a10 0x7fc6440d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.545+0000 7fc649264640 1 --2- 172.21.15.60:0/1703188569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc64415cf50 0x7fc6440d2720 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.545+0000 7fc649264640 1 -- 172.21.15.60:0/1703188569 >> 172.21.15.60:0/1703188569 conn(0x7fc6440cf750 msgr2=0x7fc6440cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.545+0000 7fc649264640 1 -- 172.21.15.60:0/1703188569 shutdown_connections 2023-10-16T12:54:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.545+0000 7fc649264640 1 -- 172.21.15.60:0/1703188569 wait complete. 2023-10-16T12:54:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.546+0000 7fc649264640 1 Processor -- start 2023-10-16T12:54:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 -- start start 2023-10-16T12:54:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b4750 0x7fc6440cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 0x7fc6441567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc64415cf50 0x7fc644151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc644159bd0 con 0x7fc6440b8a10 2023-10-16T12:54:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc644159000 con 0x7fc64415cf50 2023-10-16T12:54:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.547+0000 7fc649264640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6441592a0 con 0x7fc6440b4750 2023-10-16T12:54:29.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 0x7fc6441567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:29.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 0x7fc6441567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43866/0 (socket says 172.21.15.60:43866) 2023-10-16T12:54:29.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 -- 172.21.15.60:0/2370956704 learned_addr learned my addr 172.21.15.60:0/2370956704 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:29.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 -- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b4750 msgr2=0x7fc6440cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b4750 0x7fc6440cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 -- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc64415cf50 msgr2=0x7fc644151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc64415cf50 0x7fc644151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.548+0000 7fc63bfff640 1 -- 172.21.15.60:0/2370956704 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634009400 con 0x7fc6440b8a10 2023-10-16T12:54:29.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.549+0000 7fc63bfff640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 0x7fc6441567b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00d800 tx=0x7fc62c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.549+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00b6c0 con 0x7fc6440b8a10 2023-10-16T12:54:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.549+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c00b820 con 0x7fc6440b8a10 2023-10-16T12:54:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.549+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc644159580 con 0x7fc6440b8a10 2023-10-16T12:54:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.549+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00bbb0 con 0x7fc6440b8a10 2023-10-16T12:54:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.549+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6440cc510 con 0x7fc6440b8a10 2023-10-16T12:54:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.550+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc644072350 con 0x7fc6440b8a10 2023-10-16T12:54:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.551+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 79) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc62c00bd10 con 0x7fc6440b8a10 2023-10-16T12:54:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.552+0000 7fc63b7fe640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fc62406ad10 0x7fc62406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.552+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc62c089a20 con 0x7fc6440b8a10 2023-10-16T12:54:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.553+0000 7fc642ffd640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fc62406ad10 0x7fc62406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.554+0000 7fc642ffd640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fc62406ad10 0x7fc62406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc634036000 tx=0x7fc634030000 comp rx=0 tx=0).ready entity=mgr.6731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.557+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc62c054be0 con 0x7fc6440b8a10 2023-10-16T12:54:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.769+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 --> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6440d2200 con 0x7fc62406ad10 2023-10-16T12:54:29.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.939+0000 7fc63b7fe640 1 -- 172.21.15.60:0/2370956704 <== mgr.6731 v2:172.21.15.144:6832/110255757 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7fc634031a80 con 0x7fc62406ad10 2023-10-16T12:54:29.941 INFO:teuthology.orchestra.run.smithi060.stdout:Retention added to path /snap_test_dir1 2023-10-16T12:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fc62406ad10 msgr2=0x7fc62406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fc62406ad10 0x7fc62406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc634036000 tx=0x7fc634030000 comp rx=0 tx=0).stop 2023-10-16T12:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 msgr2=0x7fc6441567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 0x7fc6441567b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00d800 tx=0x7fc62c00dcc0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 shutdown_connections 2023-10-16T12:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fc62406ad10 0x7fc62406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc64415cf50 0x7fc644151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6440b8a10 0x7fc6441567b0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 --2- 172.21.15.60:0/2370956704 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6440b4750 0x7fc6440cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.942+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 >> 172.21.15.60:0/2370956704 conn(0x7fc6440cf750 msgr2=0x7fc64406bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.943+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 shutdown_connections 2023-10-16T12:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:29.943+0000 7fc649264640 1 -- 172.21.15.60:0/2370956704 wait complete. 2023-10-16T12:54:29.966 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-16T12:54:29.966 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T12:54:29.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-10-16T12:54:30.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.192+0000 7f034f18a640 1 Processor -- start 2023-10-16T12:54:30.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.193+0000 7f034f18a640 1 -- start start 2023-10-16T12:54:30.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.194+0000 7f034f18a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f034806a4b0 0x7f03480ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:30.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.194+0000 7f034f18a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03480c9ca0 0x7f03480ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:30.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.194+0000 7f034f18a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03480cb030 con 0x7f034815ad70 2023-10-16T12:54:30.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.194+0000 7f034f18a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03480cae30 con 0x7f034806a4b0 2023-10-16T12:54:30.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.194+0000 7f034f18a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03480c91c0 con 0x7f03480c9ca0 2023-10-16T12:54:30.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f034806a4b0 0x7f03480ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:30.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f034806a4b0 0x7f03480ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43880/0 (socket says 172.21.15.60:43880) 2023-10-16T12:54:30.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 -- 172.21.15.60:0/3603485129 learned_addr learned my addr 172.21.15.60:0/3603485129 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:30.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f034cf26640 1 --1- 172.21.15.60:0/3603485129 >> v1:172.21.15.60:6789/0 conn(0x7f034815ad70 0x7f0348069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45730/0 (socket says 172.21.15.60:45730) 2023-10-16T12:54:30.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f034d727640 1 --2- 172.21.15.60:0/3603485129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03480c9ca0 0x7f03480ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 -- 172.21.15.60:0/3603485129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03480c9ca0 msgr2=0x7f03480ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:54:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 --2- 172.21.15.60:0/3603485129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03480c9ca0 0x7f03480ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 -- 172.21.15.60:0/3603485129 >> v1:172.21.15.60:6789/0 conn(0x7f034815ad70 legacy=0x7f0348069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:30.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.195+0000 7f033ffff640 1 -- 172.21.15.60:0/3603485129 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03480c94a0 con 0x7f034806a4b0 2023-10-16T12:54:30.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.197+0000 7f033ffff640 1 --2- 172.21.15.60:0/3603485129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806a4b0 0x7f03480ca3b0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f0338009870 tx=0x7f033802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15381b96d8569611 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:30.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.197+0000 7f033f7fe640 1 -- 172.21.15.60:0/3603485129 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338030840 con 0x7f034806a4b0 2023-10-16T12:54:30.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f033f7fe640 1 -- 172.21.15.60:0/3603485129 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03380309a0 con 0x7f034806a4b0 2023-10-16T12:54:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f033f7fe640 1 -- 172.21.15.60:0/3603485129 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338030d30 con 0x7f034806a4b0 2023-10-16T12:54:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 -- 172.21.15.60:0/3603485129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806a4b0 msgr2=0x7f03480ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 --2- 172.21.15.60:0/3603485129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806a4b0 0x7f03480ca3b0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f0338009870 tx=0x7f033802b980 comp rx=0 tx=0).stop 2023-10-16T12:54:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 -- 172.21.15.60:0/3603485129 shutdown_connections 2023-10-16T12:54:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 --2- 172.21.15.60:0/3603485129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03480c9ca0 0x7f03480ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 --2- 172.21.15.60:0/3603485129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806a4b0 0x7f03480ca3b0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 -- 172.21.15.60:0/3603485129 >> 172.21.15.60:0/3603485129 conn(0x7f03480ce9b0 msgr2=0x7f03480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.198+0000 7f034f18a640 1 -- 172.21.15.60:0/3603485129 shutdown_connections 2023-10-16T12:54:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.199+0000 7f034f18a640 1 -- 172.21.15.60:0/3603485129 wait complete. 2023-10-16T12:54:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.201+0000 7f034f18a640 1 Processor -- start 2023-10-16T12:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.201+0000 7f034f18a640 1 -- start start 2023-10-16T12:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034f18a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806a4b0 0x7f0348159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034f18a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 0x7f03480c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034f18a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034815ad70 0x7f03481f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034f18a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481f1230 con 0x7f03480c9ca0 2023-10-16T12:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034f18a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03480c70a0 con 0x7f034815ad70 2023-10-16T12:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034f18a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03481f0c70 con 0x7f034806a4b0 2023-10-16T12:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f033ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 0x7f03480c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f033ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 0x7f03480c7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43888/0 (socket says 172.21.15.60:43888) 2023-10-16T12:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f033ffff640 1 -- 172.21.15.60:0/788132199 learned_addr learned my addr 172.21.15.60:0/788132199 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T12:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034cf26640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806a4b0 0x7f0348159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.203+0000 7f034d727640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034815ad70 0x7f03481f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.204+0000 7f033ffff640 1 -- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806a4b0 msgr2=0x7f0348159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.204+0000 7f033ffff640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806a4b0 0x7f0348159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.204+0000 7f033ffff640 1 -- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034815ad70 msgr2=0x7f03481f02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.204+0000 7f033ffff640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034815ad70 0x7f03481f02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.204+0000 7f033ffff640 1 -- 172.21.15.60:0/788132199 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03380094d0 con 0x7f03480c9ca0 2023-10-16T12:54:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.204+0000 7f033ffff640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 0x7f03480c7530 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f0338000c00 tx=0x7f0338004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.205+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338039c90 con 0x7f03480c9ca0 2023-10-16T12:54:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.205+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0338039e10 con 0x7f03480c9ca0 2023-10-16T12:54:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.205+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03481f0f50 con 0x7f03480c9ca0 2023-10-16T12:54:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.205+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338043640 con 0x7f03480c9ca0 2023-10-16T12:54:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.205+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034815a310 con 0x7f03480c9ca0 2023-10-16T12:54:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.206+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03480b13b0 con 0x7f03480c9ca0 2023-10-16T12:54:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.207+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 79) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f03380437a0 con 0x7f03480c9ca0 2023-10-16T12:54:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.208+0000 7f033d7fa640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f033406ad10 0x7f033406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:54:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.208+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f033803e070 con 0x7f03480c9ca0 2023-10-16T12:54:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.209+0000 7f034cf26640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f033406ad10 0x7f033406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:54:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.210+0000 7f034cf26640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f033406ad10 0x7f033406d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0344007fd0 tx=0x7f0344001630 comp rx=0 tx=0).ready entity=mgr.6731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:54:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.214+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03380775f0 con 0x7f03480c9ca0 2023-10-16T12:54:30.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.430+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 --> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03480c9780 con 0x7f033406ad10 2023-10-16T12:54:30.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.447+0000 7f033d7fa640 1 -- 172.21.15.60:0/788132199 <== mgr.6731 v2:172.21.15.144:6832/110255757 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f0344004e20 con 0x7f033406ad10 2023-10-16T12:54:30.449 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T12:54:30.449 INFO:teuthology.orchestra.run.smithi060.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T12:54:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f033406ad10 msgr2=0x7f033406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f033406ad10 0x7f033406d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0344007fd0 tx=0x7f0344001630 comp rx=0 tx=0).stop 2023-10-16T12:54:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 msgr2=0x7f03480c7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:54:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 0x7f03480c7530 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f0338000c00 tx=0x7f0338004730 comp rx=0 tx=0).stop 2023-10-16T12:54:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 shutdown_connections 2023-10-16T12:54:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f033406ad10 0x7f033406d1d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034815ad70 0x7f03481f02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03480c9ca0 0x7f03480c7530 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 --2- 172.21.15.60:0/788132199 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806a4b0 0x7f0348159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:54:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.449+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 >> 172.21.15.60:0/788132199 conn(0x7f03480ce9b0 msgr2=0x7f03480ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:54:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.450+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 shutdown_connections 2023-10-16T12:54:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T12:54:30.450+0000 7f034f18a640 1 -- 172.21.15.60:0/788132199 wait complete. 2023-10-16T12:54:30.472 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T12:54:30.472 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-10-16T12:54:58.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:58.884 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:00.965 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:55:00.963+0000 7fd951cdc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:55:00.966 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:55:00.963+0000 7fd951cdc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:55:00.966 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:55:00.963+0000 7fd951cdc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:55:00.967 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:55:00.963+0000 7fd951cdc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:55:00.967 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T12:55:00.963+0000 7fd951cdc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:55:29.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:29.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:59.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:59.323 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:29.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:29.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:59.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:59.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:57:29.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:57:29.910 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:00.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:00.269 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:30.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:30.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:00.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:00.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:31.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:31.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:01.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:01.206 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:31.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:31.497 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:01.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:01.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:31.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:31.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:02.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:02.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:32.552 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:32.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:02.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:02.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:32.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:32.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:03.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:03.058 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:33.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:33.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:03.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:03.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:33.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:33.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:03.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:03.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:34.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:34.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:04.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:04.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:34.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:34.365 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:04.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:04.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:34.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:34.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:04.952 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:04.962 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:35.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:35.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:05.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:05.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:35.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:35.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:05.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:05.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:35.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:35.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:05.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:05.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:36.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:36.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:06.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:06.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:36.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:36.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:07.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:07.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:37.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:37.215 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:07.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:07.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:37.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:37.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:07.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:07.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:38.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:38.098 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:08.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:08.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:38.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:38.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:08.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:08.568 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:38.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:38.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:08.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:08.862 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:39.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:39.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:09.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:09.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:39.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:39.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:09.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:09.515 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:39.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:39.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:09.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:09.820 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:40.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:40.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:10.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:10.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:40.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:40.417 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:24:10.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:24:10.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:24:40.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:24:40.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:11.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:11.147 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:41.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:41.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:11.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:11.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:41.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:41.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:12.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:12.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:42.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:42.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:12.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:12.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:42.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:42.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:13.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:13.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:43.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:43.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:13.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:13.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:43.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:43.773 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:13.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:13.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:44.064 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:44.074 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:14.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:14.230 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:44.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:44.443 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:14.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:14.601 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:44.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:44.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:14.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:14.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:45.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:45.261 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:15.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:15.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:45.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:45.563 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:15.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:15.710 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:45.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:45.870 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:16.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:16.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:46.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:46.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:16.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:16.318 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:46.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:46.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:16.616 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:16.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:46.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:46.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:16.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:16.982 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:47.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:47.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:17.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:17.279 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:47.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:47.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:17.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:17.580 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:47.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:47.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:17.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:17.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:48.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:48.034 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:18.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:18.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:48.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:48.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:18.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:18.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:48.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:48.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:18.849 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:18.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:49.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:49.013 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:19.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:19.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:49.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:49.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:19.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:19.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:49.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:49.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:19.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:19.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:50.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:50.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:16.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:16.474 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:50:16.546 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_56_00_UTC 2023-10-16T13:50:16.546 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_57_00_UTC 2023-10-16T13:50:16.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_58_00_UTC 2023-10-16T13:50:16.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_59_00_UTC 2023-10-16T13:50:16.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_00_00_UTC 2023-10-16T13:50:16.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_01_01_UTC 2023-10-16T13:50:16.547 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_02_00_UTC 2023-10-16T13:50:16.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_03_00_UTC 2023-10-16T13:50:16.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_04_00_UTC 2023-10-16T13:50:16.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_05_00_UTC 2023-10-16T13:50:16.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_06_00_UTC 2023-10-16T13:50:16.548 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_07_00_UTC 2023-10-16T13:50:16.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_08_00_UTC 2023-10-16T13:50:16.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_09_00_UTC 2023-10-16T13:50:16.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_10_00_UTC 2023-10-16T13:50:16.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_11_00_UTC 2023-10-16T13:50:16.549 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_12_00_UTC 2023-10-16T13:50:16.550 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_13_00_UTC 2023-10-16T13:50:16.550 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_14_00_UTC 2023-10-16T13:50:16.550 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_15_00_UTC 2023-10-16T13:50:16.550 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_16_00_UTC 2023-10-16T13:50:16.551 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_17_00_UTC 2023-10-16T13:50:16.551 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_18_00_UTC 2023-10-16T13:50:16.551 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_19_00_UTC 2023-10-16T13:50:16.551 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_20_00_UTC 2023-10-16T13:50:16.551 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_21_00_UTC 2023-10-16T13:50:16.551 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_22_00_UTC 2023-10-16T13:50:16.552 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_23_00_UTC 2023-10-16T13:50:16.552 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_24_00_UTC 2023-10-16T13:50:16.552 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_25_00_UTC 2023-10-16T13:50:16.552 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_26_00_UTC 2023-10-16T13:50:16.552 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_27_00_UTC 2023-10-16T13:50:16.553 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_28_00_UTC 2023-10-16T13:50:16.553 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:50:16.553 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:50:16.553 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:50:16.553 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:50:16.554 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:50:16.554 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:50:16.554 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:50:16.554 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:50:16.554 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:50:16.555 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_38_01_UTC 2023-10-16T13:50:16.555 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:50:16.555 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_40_00_UTC 2023-10-16T13:50:16.555 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_41_00_UTC 2023-10-16T13:50:16.555 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_42_00_UTC 2023-10-16T13:50:16.556 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_43_01_UTC 2023-10-16T13:50:16.556 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_44_00_UTC 2023-10-16T13:50:16.556 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:50:16.556 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_46_00_UTC 2023-10-16T13:50:16.556 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_47_00_UTC 2023-10-16T13:50:16.557 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_48_01_UTC 2023-10-16T13:50:16.557 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_49_00_UTC 2023-10-16T13:50:16.557 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_50_00_UTC 2023-10-16T13:50:16.558 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_56_00_UTC', 'scheduled-2023-10-16-12_57_00_UTC', 'scheduled-2023-10-16-12_58_00_UTC', 'scheduled-2023-10-16-12_59_00_UTC', 'scheduled-2023-10-16-13_00_00_UTC', 'scheduled-2023-10-16-13_01_01_UTC', 'scheduled-2023-10-16-13_02_00_UTC', 'scheduled-2023-10-16-13_03_00_UTC', 'scheduled-2023-10-16-13_04_00_UTC', 'scheduled-2023-10-16-13_05_00_UTC', 'scheduled-2023-10-16-13_06_00_UTC', 'scheduled-2023-10-16-13_07_00_UTC', 'scheduled-2023-10-16-13_08_00_UTC', 'scheduled-2023-10-16-13_09_00_UTC', 'scheduled-2023-10-16-13_10_00_UTC', 'scheduled-2023-10-16-13_11_00_UTC', 'scheduled-2023-10-16-13_12_00_UTC', 'scheduled-2023-10-16-13_13_00_UTC', 'scheduled-2023-10-16-13_14_00_UTC', 'scheduled-2023-10-16-13_15_00_UTC', 'scheduled-2023-10-16-13_16_00_UTC', 'scheduled-2023-10-16-13_17_00_UTC', 'scheduled-2023-10-16-13_18_00_UTC', 'scheduled-2023-10-16-13_19_00_UTC', 'scheduled-2023-10-16-13_20_00_UTC', 'scheduled-2023-10-16-13_21_00_UTC', 'scheduled-2023-10-16-13_22_00_UTC', 'scheduled-2023-10-16-13_23_00_UTC', 'scheduled-2023-10-16-13_24_00_UTC', 'scheduled-2023-10-16-13_25_00_UTC', 'scheduled-2023-10-16-13_26_00_UTC', 'scheduled-2023-10-16-13_27_00_UTC', 'scheduled-2023-10-16-13_28_00_UTC', 'scheduled-2023-10-16-13_29_00_UTC', 'scheduled-2023-10-16-13_30_00_UTC', 'scheduled-2023-10-16-13_31_00_UTC', 'scheduled-2023-10-16-13_32_00_UTC', 'scheduled-2023-10-16-13_33_00_UTC', 'scheduled-2023-10-16-13_34_00_UTC', 'scheduled-2023-10-16-13_35_00_UTC', 'scheduled-2023-10-16-13_36_00_UTC', 'scheduled-2023-10-16-13_37_00_UTC', 'scheduled-2023-10-16-13_38_01_UTC', 'scheduled-2023-10-16-13_39_00_UTC', 'scheduled-2023-10-16-13_40_00_UTC', 'scheduled-2023-10-16-13_41_00_UTC', 'scheduled-2023-10-16-13_42_00_UTC', 'scheduled-2023-10-16-13_43_01_UTC', 'scheduled-2023-10-16-13_44_00_UTC', 'scheduled-2023-10-16-13_45_00_UTC', 'scheduled-2023-10-16-13_46_00_UTC', 'scheduled-2023-10-16-13_47_00_UTC', 'scheduled-2023-10-16-13_48_01_UTC', 'scheduled-2023-10-16-13_49_00_UTC', 'scheduled-2023-10-16-13_50_00_UTC'] 2023-10-16T13:50:16.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-10-16T13:50:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.796+0000 7f3ef0600640 1 Processor -- start 2023-10-16T13:50:16.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.797+0000 7f3ef0600640 1 -- start start 2023-10-16T13:50:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.798+0000 7f3ef0600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ee815d150 0x7f3ee80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.798+0000 7f3ef0600640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8160b90 0x7f3ee8160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.798+0000 7f3ef0600640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80ac890 con 0x7f3ee8069da0 2023-10-16T13:50:16.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.798+0000 7f3ef0600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee815d530 con 0x7f3ee815d150 2023-10-16T13:50:16.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.798+0000 7f3ef0600640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8161650 con 0x7f3ee8160b90 2023-10-16T13:50:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.798+0000 7f3eee39c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ee815d150 0x7f3ee80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eee39c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ee815d150 0x7f3ee80ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47750/0 (socket says 172.21.15.60:47750) 2023-10-16T13:50:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eee39c640 1 -- 172.21.15.60:0/4005555794 learned_addr learned my addr 172.21.15.60:0/4005555794 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eee39c640 1 -- 172.21.15.60:0/4005555794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8160b90 msgr2=0x7f3ee8160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eedb9b640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8160b90 0x7f3ee8160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eee39c640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8160b90 0x7f3ee8160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eee39c640 1 -- 172.21.15.60:0/4005555794 >> v1:172.21.15.144:6789/0 conn(0x7f3ee8069da0 legacy=0x7f3ee806a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eee39c640 1 -- 172.21.15.60:0/4005555794 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee8161930 con 0x7f3ee815d150 2023-10-16T13:50:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.799+0000 7f3eedb9b640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8160b90 0x7f3ee8160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.801+0000 7f3eee39c640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee815d150 0x7f3ee80ad350 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009870 tx=0x7f3ee402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750996de3cfb381d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.801+0000 7f3eed39a640 1 -- 172.21.15.60:0/4005555794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4031640 con 0x7f3ee815d150 2023-10-16T13:50:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.801+0000 7f3eed39a640 1 -- 172.21.15.60:0/4005555794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee40317a0 con 0x7f3ee815d150 2023-10-16T13:50:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.801+0000 7f3eed39a640 1 -- 172.21.15.60:0/4005555794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4031b30 con 0x7f3ee815d150 2023-10-16T13:50:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 -- 172.21.15.60:0/4005555794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee815d150 msgr2=0x7f3ee80ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee815d150 0x7f3ee80ad350 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009870 tx=0x7f3ee402b8f0 comp rx=0 tx=0).stop 2023-10-16T13:50:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 -- 172.21.15.60:0/4005555794 shutdown_connections 2023-10-16T13:50:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8160b90 0x7f3ee8160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 --2- 172.21.15.60:0/4005555794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee815d150 0x7f3ee80ad350 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 -- 172.21.15.60:0/4005555794 >> 172.21.15.60:0/4005555794 conn(0x7f3ee80ce9b0 msgr2=0x7f3ee80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 -- 172.21.15.60:0/4005555794 shutdown_connections 2023-10-16T13:50:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.802+0000 7f3ef0600640 1 -- 172.21.15.60:0/4005555794 wait complete. 2023-10-16T13:50:16.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.804+0000 7f3ef0600640 1 Processor -- start 2023-10-16T13:50:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.804+0000 7f3ef0600640 1 -- start start 2023-10-16T13:50:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.805+0000 7f3ef0600640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8069da0 0x7f3ee80c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.805+0000 7f3ef0600640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee815d150 0x7f3ee80c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.805+0000 7f3ef0600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 0x7f3ee80c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.805+0000 7f3ef0600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8159970 con 0x7f3ee8160b90 2023-10-16T13:50:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.805+0000 7f3ef0600640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80ae9c0 con 0x7f3ee815d150 2023-10-16T13:50:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.805+0000 7f3ef0600640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80c38d0 con 0x7f3ee8069da0 2023-10-16T13:50:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 0x7f3ee80c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 0x7f3ee80c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47766/0 (socket says 172.21.15.60:47766) 2023-10-16T13:50:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 -- 172.21.15.60:0/2064746074 learned_addr learned my addr 172.21.15.60:0/2064746074 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eee39c640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8069da0 0x7f3ee80c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eedb9b640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee815d150 0x7f3ee80c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 -- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8069da0 msgr2=0x7f3ee80c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8069da0 0x7f3ee80c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 -- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee815d150 msgr2=0x7f3ee80c2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee815d150 0x7f3ee80c2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.806+0000 7f3eeeb9d640 1 -- 172.21.15.60:0/2064746074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40094d0 con 0x7f3ee8160b90 2023-10-16T13:50:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.807+0000 7f3eeeb9d640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 0x7f3ee80c4090 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000c780 tx=0x7f3ee000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.807+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0004b60 con 0x7f3ee8160b90 2023-10-16T13:50:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.807+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee000ce90 con 0x7f3ee8160b90 2023-10-16T13:50:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.807+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee80c35f0 con 0x7f3ee8160b90 2023-10-16T13:50:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.807+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee000a850 con 0x7f3ee8160b90 2023-10-16T13:50:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.807+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee80ae650 con 0x7f3ee8160b90 2023-10-16T13:50:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.808+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee80aeb50 con 0x7f3ee8160b90 2023-10-16T13:50:16.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.810+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 83) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f3ee000a9b0 con 0x7f3ee8160b90 2023-10-16T13:50:16.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.811+0000 7f3edeffd640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f3ecc06afe0 0x7f3ecc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:16.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.811+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6155+0+0 (secure 0 0 0) 0x7f3ee0089b80 con 0x7f3ee8160b90 2023-10-16T13:50:16.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.811+0000 7f3eee39c640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f3ecc06afe0 0x7f3ecc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:16.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.813+0000 7f3eee39c640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f3ecc06afe0 0x7f3ecc06d4a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4036000 tx=0x7f3ee403c040 comp rx=0 tx=0).ready entity=mgr.6731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:16.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:16.816+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ee0054950 con 0x7f3ee8160b90 2023-10-16T13:50:17.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.034+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 --> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ee80ace30 con 0x7f3ecc06afe0 2023-10-16T13:50:17.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.055+0000 7f3edeffd640 1 -- 172.21.15.60:0/2064746074 <== mgr.6731 v2:172.21.15.144:6832/110255757 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7f3ee40315e0 con 0x7f3ecc06afe0 2023-10-16T13:50:17.058 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:17.058 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:54:29", "first": "2023-10-16T12:55:00", "last": "2023-10-16T13:50:00", "last_pruned": "2023-10-16T13:50:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T13:50:17.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f3ecc06afe0 msgr2=0x7f3ecc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f3ecc06afe0 0x7f3ecc06d4a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4036000 tx=0x7f3ee403c040 comp rx=0 tx=0).stop 2023-10-16T13:50:17.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 msgr2=0x7f3ee80c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 0x7f3ee80c4090 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000c780 tx=0x7f3ee000cc40 comp rx=0 tx=0).stop 2023-10-16T13:50:17.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 shutdown_connections 2023-10-16T13:50:17.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f3ecc06afe0 0x7f3ecc06d4a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee8160b90 0x7f3ee80c4090 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ee815d150 0x7f3ee80c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 --2- 172.21.15.60:0/2064746074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ee8069da0 0x7f3ee80c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.057+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 >> 172.21.15.60:0/2064746074 conn(0x7f3ee80ce9b0 msgr2=0x7f3ee80ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:17.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.058+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 shutdown_connections 2023-10-16T13:50:17.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.058+0000 7f3ef0600640 1 -- 172.21.15.60:0/2064746074 wait complete. 2023-10-16T13:50:17.080 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:54:29", "first": "2023-10-16T12:55:00", "last": "2023-10-16T13:50:00", "last_pruned": "2023-10-16T13:50:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T13:50:17.080 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:54:29', 'first': '2023-10-16T12:55:00', 'last': '2023-10-16T13:50:00', 'last_pruned': '2023-10-16T13:50:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-16T13:50:17.080 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-16T13:50:17.081 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-16T13:50:17.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-10-16T13:50:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.309+0000 7fe9fb002640 1 Processor -- start 2023-10-16T13:50:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.309+0000 7fe9fb002640 1 -- start start 2023-10-16T13:50:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.310+0000 7fe9fb002640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9f415ad70 0x7fe9f4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.310+0000 7fe9fb002640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe9f40c9ca0 0x7fe9f406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9fb002640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f40cb030 con 0x7fe9f40ca3b0 2023-10-16T13:50:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9fb002640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f40c91c0 con 0x7fe9f415ad70 2023-10-16T13:50:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9fb002640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f40c94a0 con 0x7fe9f40c9ca0 2023-10-16T13:50:17.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9f8d9e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9f415ad70 0x7fe9f4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:17.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9f8d9e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9f415ad70 0x7fe9f4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47774/0 (socket says 172.21.15.60:47774) 2023-10-16T13:50:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9f8d9e640 1 -- 172.21.15.60:0/161921164 learned_addr learned my addr 172.21.15.60:0/161921164 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.311+0000 7fe9f3fff640 1 --2- 172.21.15.60:0/161921164 >> v2:172.21.15.144:3300/0 conn(0x7fe9f40c9ca0 0x7fe9f406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.312+0000 7fe9f8d9e640 1 -- 172.21.15.60:0/161921164 >> v2:172.21.15.144:3300/0 conn(0x7fe9f40c9ca0 msgr2=0x7fe9f406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.312+0000 7fe9f8d9e640 1 --2- 172.21.15.60:0/161921164 >> v2:172.21.15.144:3300/0 conn(0x7fe9f40c9ca0 0x7fe9f406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.312+0000 7fe9f8d9e640 1 -- 172.21.15.60:0/161921164 >> v1:172.21.15.60:6789/0 conn(0x7fe9f40ca3b0 legacy=0x7fe9f40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.312+0000 7fe9f8d9e640 1 -- 172.21.15.60:0/161921164 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f40c9780 con 0x7fe9f415ad70 2023-10-16T13:50:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.313+0000 7fe9f8d9e640 1 --2- 172.21.15.60:0/161921164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f4069da0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40ce870 tx=0x7fe9e802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91251a16f115b013 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.313+0000 7fe9f37fe640 1 -- 172.21.15.60:0/161921164 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8030870 con 0x7fe9f415ad70 2023-10-16T13:50:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.313+0000 7fe9f37fe640 1 -- 172.21.15.60:0/161921164 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e80309d0 con 0x7fe9f415ad70 2023-10-16T13:50:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.313+0000 7fe9f37fe640 1 -- 172.21.15.60:0/161921164 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8030d60 con 0x7fe9f415ad70 2023-10-16T13:50:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 -- 172.21.15.60:0/161921164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 msgr2=0x7fe9f4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 --2- 172.21.15.60:0/161921164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f4069da0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40ce870 tx=0x7fe9e802b9b0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 -- 172.21.15.60:0/161921164 shutdown_connections 2023-10-16T13:50:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 --2- 172.21.15.60:0/161921164 >> v2:172.21.15.144:3300/0 conn(0x7fe9f40c9ca0 0x7fe9f406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 --2- 172.21.15.60:0/161921164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f4069da0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 -- 172.21.15.60:0/161921164 >> 172.21.15.60:0/161921164 conn(0x7fe9f40ce9b0 msgr2=0x7fe9f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:17.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 -- 172.21.15.60:0/161921164 shutdown_connections 2023-10-16T13:50:17.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.314+0000 7fe9fb002640 1 -- 172.21.15.60:0/161921164 wait complete. 2023-10-16T13:50:17.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.316+0000 7fe9fb002640 1 Processor -- start 2023-10-16T13:50:17.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.316+0000 7fe9fb002640 1 -- start start 2023-10-16T13:50:17.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9fb002640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40c9ca0 0x7fe9f40b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:17.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9fb002640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f40ca3b0 0x7fe9f40c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:17.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9fb002640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f40c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:17.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9fb002640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f40c5cb0 con 0x7fe9f415ad70 2023-10-16T13:50:17.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9fb002640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f41f69e0 con 0x7fe9f40c9ca0 2023-10-16T13:50:17.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9fb002640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f41f6470 con 0x7fe9f40ca3b0 2023-10-16T13:50:17.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.317+0000 7fe9f959f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f40c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:17.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f40c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47778/0 (socket says 172.21.15.60:47778) 2023-10-16T13:50:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 -- 172.21.15.60:0/59268082 learned_addr learned my addr 172.21.15.60:0/59268082 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f8d9e640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40c9ca0 0x7fe9f40b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 -- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f40ca3b0 msgr2=0x7fe9f40c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f40ca3b0 0x7fe9f40c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 -- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40c9ca0 msgr2=0x7fe9f40b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40c9ca0 0x7fe9f40b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 -- 172.21.15.60:0/59268082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80094d0 con 0x7fe9f415ad70 2023-10-16T13:50:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.318+0000 7fe9f959f640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f40c6f10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00c7a0 tx=0x7fe9ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:17.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.319+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec003b70 con 0x7fe9f415ad70 2023-10-16T13:50:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.319+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ec003cf0 con 0x7fe9f415ad70 2023-10-16T13:50:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.319+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec00aa90 con 0x7fe9f415ad70 2023-10-16T13:50:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.319+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f41f6700 con 0x7fe9f415ad70 2023-10-16T13:50:17.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.319+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f415b860 con 0x7fe9f415ad70 2023-10-16T13:50:17.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.320+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f404e160 con 0x7fe9f415ad70 2023-10-16T13:50:17.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.321+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 83) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fe9ec00abf0 con 0x7fe9f415ad70 2023-10-16T13:50:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.322+0000 7fe9f17fa640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fe9d806afe0 0x7fe9d806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.322+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6155+0+0 (secure 0 0 0) 0x7fe9ec0897f0 con 0x7fe9f415ad70 2023-10-16T13:50:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.322+0000 7fe9f8d9e640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fe9d806afe0 0x7fe9d806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:17.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.324+0000 7fe9f8d9e640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fe9d806afe0 0x7fe9d806d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8036000 tx=0x7fe9e803b040 comp rx=0 tx=0).ready entity=mgr.6731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.326+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9ec0545c0 con 0x7fe9f415ad70 2023-10-16T13:50:17.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.550+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 --> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9f40ad4d0 con 0x7fe9d806afe0 2023-10-16T13:50:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.759+0000 7fe9f17fa640 1 -- 172.21.15.60:0/59268082 <== mgr.6731 v2:172.21.15.144:6832/110255757 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7fe9e8030bc0 con 0x7fe9d806afe0 2023-10-16T13:50:17.763 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1 2023-10-16T13:50:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.762+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fe9d806afe0 msgr2=0x7fe9d806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.762+0000 7fe9fb002640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fe9d806afe0 0x7fe9d806d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8036000 tx=0x7fe9e803b040 comp rx=0 tx=0).stop 2023-10-16T13:50:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 msgr2=0x7fe9f40c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f40c6f10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00c7a0 tx=0x7fe9ec00cc60 comp rx=0 tx=0).stop 2023-10-16T13:50:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 shutdown_connections 2023-10-16T13:50:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fe9d806afe0 0x7fe9d806d4a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9f40ca3b0 0x7fe9f40c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9f415ad70 0x7fe9f40c6f10 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 --2- 172.21.15.60:0/59268082 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9f40c9ca0 0x7fe9f40b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 >> 172.21.15.60:0/59268082 conn(0x7fe9f40ce9b0 msgr2=0x7fe9f40acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 shutdown_connections 2023-10-16T13:50:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:17.763+0000 7fe9fb002640 1 -- 172.21.15.60:0/59268082 wait complete. 2023-10-16T13:50:17.788 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-16T13:50:17.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:17.789 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:50:17.850 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_56_00_UTC 2023-10-16T13:50:17.850 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_57_00_UTC 2023-10-16T13:50:17.850 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_58_00_UTC 2023-10-16T13:50:17.851 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-12_59_00_UTC 2023-10-16T13:50:17.851 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_00_00_UTC 2023-10-16T13:50:17.851 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_01_01_UTC 2023-10-16T13:50:17.851 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_02_00_UTC 2023-10-16T13:50:17.851 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_03_00_UTC 2023-10-16T13:50:17.851 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_04_00_UTC 2023-10-16T13:50:17.852 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_05_00_UTC 2023-10-16T13:50:17.852 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_06_00_UTC 2023-10-16T13:50:17.852 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_07_00_UTC 2023-10-16T13:50:17.852 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_08_00_UTC 2023-10-16T13:50:17.852 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_09_00_UTC 2023-10-16T13:50:17.853 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_10_00_UTC 2023-10-16T13:50:17.853 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_11_00_UTC 2023-10-16T13:50:17.853 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_12_00_UTC 2023-10-16T13:50:17.853 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_13_00_UTC 2023-10-16T13:50:17.853 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_14_00_UTC 2023-10-16T13:50:17.853 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_15_00_UTC 2023-10-16T13:50:17.854 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_16_00_UTC 2023-10-16T13:50:17.854 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_17_00_UTC 2023-10-16T13:50:17.854 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_18_00_UTC 2023-10-16T13:50:17.854 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_19_00_UTC 2023-10-16T13:50:17.854 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_20_00_UTC 2023-10-16T13:50:17.855 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_21_00_UTC 2023-10-16T13:50:17.855 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_22_00_UTC 2023-10-16T13:50:17.855 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_23_00_UTC 2023-10-16T13:50:17.855 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_24_00_UTC 2023-10-16T13:50:17.855 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_25_00_UTC 2023-10-16T13:50:17.856 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_26_00_UTC 2023-10-16T13:50:17.856 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_27_00_UTC 2023-10-16T13:50:17.856 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_28_00_UTC 2023-10-16T13:50:17.856 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:50:17.856 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:50:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:50:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:50:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:50:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:50:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:50:17.857 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:50:17.858 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:50:17.858 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_38_01_UTC 2023-10-16T13:50:17.858 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:50:17.858 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_40_00_UTC 2023-10-16T13:50:17.858 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_41_00_UTC 2023-10-16T13:50:17.859 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_42_00_UTC 2023-10-16T13:50:17.859 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_43_01_UTC 2023-10-16T13:50:17.859 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_44_00_UTC 2023-10-16T13:50:17.859 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:50:17.859 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_46_00_UTC 2023-10-16T13:50:17.860 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_47_00_UTC 2023-10-16T13:50:17.860 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_48_01_UTC 2023-10-16T13:50:17.860 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_49_00_UTC 2023-10-16T13:50:17.860 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_50_00_UTC 2023-10-16T13:50:17.861 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_56_00_UTC 2023-10-16T13:50:17.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:17.862 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_56_00_UTC) 2023-10-16T13:50:17.939 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_57_00_UTC 2023-10-16T13:50:17.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:17.940 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_57_00_UTC) 2023-10-16T13:50:18.016 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_58_00_UTC 2023-10-16T13:50:18.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.017 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_58_00_UTC) 2023-10-16T13:50:18.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_59_00_UTC 2023-10-16T13:50:18.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.095 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_59_00_UTC) 2023-10-16T13:50:18.167 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_00_00_UTC 2023-10-16T13:50:18.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.168 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_00_00_UTC) 2023-10-16T13:50:18.248 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_01_01_UTC 2023-10-16T13:50:18.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.249 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_01_01_UTC) 2023-10-16T13:50:18.328 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_02_00_UTC 2023-10-16T13:50:18.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.329 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_02_00_UTC) 2023-10-16T13:50:18.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_03_00_UTC 2023-10-16T13:50:18.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.404 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_03_00_UTC) 2023-10-16T13:50:18.482 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_04_00_UTC 2023-10-16T13:50:18.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.483 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_04_00_UTC) 2023-10-16T13:50:18.574 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_05_00_UTC 2023-10-16T13:50:18.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.575 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_05_00_UTC) 2023-10-16T13:50:18.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_06_00_UTC 2023-10-16T13:50:18.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.655 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_06_00_UTC) 2023-10-16T13:50:18.732 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_07_00_UTC 2023-10-16T13:50:18.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.733 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_07_00_UTC) 2023-10-16T13:50:18.802 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_08_00_UTC 2023-10-16T13:50:18.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.803 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_08_00_UTC) 2023-10-16T13:50:18.866 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_09_00_UTC 2023-10-16T13:50:18.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.867 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_09_00_UTC) 2023-10-16T13:50:18.944 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_10_00_UTC 2023-10-16T13:50:18.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:18.945 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_10_00_UTC) 2023-10-16T13:50:19.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_11_00_UTC 2023-10-16T13:50:19.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.025 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_11_00_UTC) 2023-10-16T13:50:19.107 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_12_00_UTC 2023-10-16T13:50:19.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.108 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_12_00_UTC) 2023-10-16T13:50:19.174 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_13_00_UTC 2023-10-16T13:50:19.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.175 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_13_00_UTC) 2023-10-16T13:50:19.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_14_00_UTC 2023-10-16T13:50:19.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.239 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_14_00_UTC) 2023-10-16T13:50:19.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_15_00_UTC 2023-10-16T13:50:19.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.308 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_15_00_UTC) 2023-10-16T13:50:19.381 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_16_00_UTC 2023-10-16T13:50:19.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.382 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_16_00_UTC) 2023-10-16T13:50:19.459 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_17_00_UTC 2023-10-16T13:50:19.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.460 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_17_00_UTC) 2023-10-16T13:50:19.548 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_18_00_UTC 2023-10-16T13:50:19.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.549 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_18_00_UTC) 2023-10-16T13:50:19.619 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_19_00_UTC 2023-10-16T13:50:19.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.619 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_19_00_UTC) 2023-10-16T13:50:19.688 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_20_00_UTC 2023-10-16T13:50:19.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.689 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_20_00_UTC) 2023-10-16T13:50:19.767 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_21_00_UTC 2023-10-16T13:50:19.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.768 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_21_00_UTC) 2023-10-16T13:50:19.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_22_00_UTC 2023-10-16T13:50:19.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.832 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_22_00_UTC) 2023-10-16T13:50:19.900 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_23_00_UTC 2023-10-16T13:50:19.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.900 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_23_00_UTC) 2023-10-16T13:50:19.971 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_24_00_UTC 2023-10-16T13:50:19.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:19.972 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_24_00_UTC) 2023-10-16T13:50:20.045 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_25_00_UTC 2023-10-16T13:50:20.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.045 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_25_00_UTC) 2023-10-16T13:50:20.106 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_26_00_UTC 2023-10-16T13:50:20.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.107 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_26_00_UTC) 2023-10-16T13:50:20.171 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_27_00_UTC 2023-10-16T13:50:20.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.172 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_27_00_UTC) 2023-10-16T13:50:20.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:20.227 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:20.241 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_28_00_UTC 2023-10-16T13:50:20.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.241 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_28_00_UTC) 2023-10-16T13:50:20.350 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:50:20.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.351 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_29_00_UTC) 2023-10-16T13:50:20.463 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:50:20.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.463 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_30_00_UTC) 2023-10-16T13:50:20.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:50:20.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.574 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_31_00_UTC) 2023-10-16T13:50:20.634 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:50:20.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.635 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_32_00_UTC) 2023-10-16T13:50:20.684 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:50:20.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.685 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_33_00_UTC) 2023-10-16T13:50:20.738 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:50:20.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.739 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_34_00_UTC) 2023-10-16T13:50:20.806 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:50:20.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.807 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_35_00_UTC) 2023-10-16T13:50:20.891 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:50:20.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.892 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_36_00_UTC) 2023-10-16T13:50:20.940 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:50:20.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:20.941 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_37_00_UTC) 2023-10-16T13:50:21.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_38_01_UTC 2023-10-16T13:50:21.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.041 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_38_01_UTC) 2023-10-16T13:50:21.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:50:21.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.103 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_39_00_UTC) 2023-10-16T13:50:21.168 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_40_00_UTC 2023-10-16T13:50:21.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.169 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_40_00_UTC) 2023-10-16T13:50:21.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_41_00_UTC 2023-10-16T13:50:21.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.233 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_41_00_UTC) 2023-10-16T13:50:21.299 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_42_00_UTC 2023-10-16T13:50:21.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.300 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_42_00_UTC) 2023-10-16T13:50:21.362 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_43_01_UTC 2023-10-16T13:50:21.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.363 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_43_01_UTC) 2023-10-16T13:50:21.429 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_44_00_UTC 2023-10-16T13:50:21.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.430 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_44_00_UTC) 2023-10-16T13:50:21.502 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:50:21.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.503 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_45_00_UTC) 2023-10-16T13:50:21.587 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_46_00_UTC 2023-10-16T13:50:21.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.588 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_46_00_UTC) 2023-10-16T13:50:21.676 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_47_00_UTC 2023-10-16T13:50:21.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.677 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_47_00_UTC) 2023-10-16T13:50:21.734 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_48_01_UTC 2023-10-16T13:50:21.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.734 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_48_01_UTC) 2023-10-16T13:50:21.781 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_49_00_UTC 2023-10-16T13:50:21.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.781 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_49_00_UTC) 2023-10-16T13:50:21.887 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_50_00_UTC 2023-10-16T13:50:21.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.888 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_50_00_UTC) 2023-10-16T13:50:21.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:21.953 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T13:50:22.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:50:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.241+0000 7f0006b47640 1 Processor -- start 2023-10-16T13:50:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.241+0000 7f0006b47640 1 -- start start 2023-10-16T13:50:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.243+0000 7f0006b47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00000ad970 0x7f00000cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.243+0000 7f0006b47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ccfb0 0x7f00000cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.243+0000 7f0006b47640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000015b9d0 con 0x7f000006c040 2023-10-16T13:50:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.243+0000 7f0006b47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000015bcb0 con 0x7f00000ad970 2023-10-16T13:50:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.243+0000 7f0006b47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000015bf90 con 0x7f00000ccfb0 2023-10-16T13:50:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.244+0000 7f0006346640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ccfb0 0x7f00000cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.244+0000 7f0005344640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00000ad970 0x7f00000cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.244+0000 7f0005b45640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f000006c040 0x7f000006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:54688/0 (socket says 172.21.15.60:54688) 2023-10-16T13:50:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.244+0000 7f0005b45640 1 -- 172.21.15.60:0/4018083481 learned_addr learned my addr 172.21.15.60:0/4018083481 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.244+0000 7f0004b43640 1 -- 172.21.15.60:0/4018083481 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875133240 0 0) 0x7f000015b9d0 con 0x7f000006c040 2023-10-16T13:50:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.245+0000 7f0004b43640 1 -- 172.21.15.60:0/4018083481 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec003520 con 0x7f000006c040 2023-10-16T13:50:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.245+0000 7f0005344640 1 -- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ccfb0 msgr2=0x7f00000cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.245+0000 7f0005344640 1 --2- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ccfb0 0x7f00000cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.245+0000 7f0005344640 1 -- 172.21.15.60:0/4018083481 >> v1:172.21.15.144:6789/0 conn(0x7f000006c040 legacy=0x7f000006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.245+0000 7f0005344640 1 -- 172.21.15.60:0/4018083481 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00000cc270 con 0x7f00000ad970 2023-10-16T13:50:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.245+0000 7f0006346640 1 --2- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ccfb0 0x7f00000cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:50:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.246+0000 7f0005344640 1 --2- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00000ad970 0x7f00000cca70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7efff00097a0 tx=0x7efff002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8141b606fd268327 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0004b43640 1 -- 172.21.15.60:0/4018083481 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff002d040 con 0x7f00000ad970 2023-10-16T13:50:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0004b43640 1 -- 172.21.15.60:0/4018083481 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0031810 con 0x7f00000ad970 2023-10-16T13:50:22.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0004b43640 1 -- 172.21.15.60:0/4018083481 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0031b80 con 0x7f00000ad970 2023-10-16T13:50:22.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 -- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00000ad970 msgr2=0x7f00000cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:22.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 --2- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00000ad970 0x7f00000cca70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7efff00097a0 tx=0x7efff002b8b0 comp rx=0 tx=0).stop 2023-10-16T13:50:22.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 -- 172.21.15.60:0/4018083481 shutdown_connections 2023-10-16T13:50:22.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 --2- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ccfb0 0x7f00000cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:22.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 --2- 172.21.15.60:0/4018083481 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00000ad970 0x7f00000cca70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:22.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 -- 172.21.15.60:0/4018083481 >> 172.21.15.60:0/4018083481 conn(0x7f00000ce6a0 msgr2=0x7f00000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:22.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 -- 172.21.15.60:0/4018083481 shutdown_connections 2023-10-16T13:50:22.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.247+0000 7f0006b47640 1 -- 172.21.15.60:0/4018083481 wait complete. 2023-10-16T13:50:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.249+0000 7f0006b47640 1 Processor -- start 2023-10-16T13:50:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.250+0000 7f0006b47640 1 -- start start 2023-10-16T13:50:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006b47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f000006c040 0x7f00001535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006b47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ad970 0x7f00001e4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006b47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 0x7f00001e53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006b47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0000153af0 con 0x7f00000ccfb0 2023-10-16T13:50:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006b47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0000153f50 con 0x7f000006c040 2023-10-16T13:50:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006b47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00001e5a00 con 0x7f00000ad970 2023-10-16T13:50:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006346640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 0x7f00001e53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006346640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 0x7f00001e53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53476/0 (socket says 172.21.15.60:53476) 2023-10-16T13:50:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.251+0000 7f0006346640 1 -- 172.21.15.60:0/4248384977 learned_addr learned my addr 172.21.15.60:0/4248384977 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0005344640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ad970 0x7f00001e4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0005b45640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f000006c040 0x7f00001535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006346640 1 -- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ad970 msgr2=0x7f00001e4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006346640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ad970 0x7f00001e4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006346640 1 -- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f000006c040 msgr2=0x7f00001535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:22.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006346640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f000006c040 0x7f00001535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:22.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006346640 1 -- 172.21.15.60:0/4248384977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff8009a70 con 0x7f00000ccfb0 2023-10-16T13:50:22.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006346640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 0x7f00001e53a0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7efff800efc0 tx=0x7efff800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:22.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff8009cd0 con 0x7f00000ccfb0 2023-10-16T13:50:22.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff0009400 con 0x7f00000ccfb0 2023-10-16T13:50:22.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.252+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff800f040 con 0x7f00000ccfb0 2023-10-16T13:50:22.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.253+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff8003990 con 0x7f00000ccfb0 2023-10-16T13:50:22.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.253+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00000c3180 con 0x7f00000ccfb0 2023-10-16T13:50:22.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.254+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000006b330 con 0x7f00000ccfb0 2023-10-16T13:50:22.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.255+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 83) v1 ==== 89360+0+0 (secure 0 0 0) 0x7efff8014480 con 0x7f00000ccfb0 2023-10-16T13:50:22.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.256+0000 7efff67fc640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7effe406afe0 0x7effe406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:22.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.256+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6199+0+0 (secure 0 0 0) 0x7efff808a730 con 0x7f00000ccfb0 2023-10-16T13:50:22.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.256+0000 7f0005b45640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7effe406afe0 0x7effe406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:22.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.257+0000 7f0005b45640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7effe406afe0 0x7effe406d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efffc007fd0 tx=0x7efffc009040 comp rx=0 tx=0).ready entity=mgr.6731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:22.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.261+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efff8055340 con 0x7f00000ccfb0 2023-10-16T13:50:22.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:22.530+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f000004e9b0 con 0x7f00000ccfb0 2023-10-16T13:50:23.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.537+0000 7efff67fc640 1 -- 172.21.15.60:0/4248384977 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v84) v1 ==== 93+0+0 (secure 0 0 0) 0x7efff8058960 con 0x7f00000ccfb0 2023-10-16T13:50:23.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.543+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7effe406afe0 msgr2=0x7effe406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.544+0000 7f0006b47640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7effe406afe0 0x7effe406d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efffc007fd0 tx=0x7efffc009040 comp rx=0 tx=0).stop 2023-10-16T13:50:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.544+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 msgr2=0x7f00001e53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:23.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.544+0000 7f0006b47640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 0x7f00001e53a0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7efff800efc0 tx=0x7efff800c490 comp rx=0 tx=0).stop 2023-10-16T13:50:23.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 shutdown_connections 2023-10-16T13:50:23.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7effe406afe0 0x7effe406d4a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:23.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00000ad970 0x7f00001e4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:23.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00000ccfb0 0x7f00001e53a0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:23.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 --2- 172.21.15.60:0/4248384977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f000006c040 0x7f00001535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:23.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 >> 172.21.15.60:0/4248384977 conn(0x7f00000ce6a0 msgr2=0x7f000015dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:23.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 shutdown_connections 2023-10-16T13:50:23.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:23.545+0000 7f0006b47640 1 -- 172.21.15.60:0/4248384977 wait complete. 2023-10-16T13:50:23.576 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:50:23.576 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:50:23.677 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:50:23.771 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:50:23.771 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:50:23.772 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:23.828 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:50:23.828 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:23.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:24.124 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.121+0000 7fa69880d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:50:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.125+0000 7fda925fb640 1 Processor -- start 2023-10-16T13:50:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.125+0000 7fda925fb640 1 -- start start 2023-10-16T13:50:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.126+0000 7fda925fb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda8c159b20 0x7fda8c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.126+0000 7fda925fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.126+0000 7fda925fb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c06f7f0 con 0x7fda8c159f00 2023-10-16T13:50:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.126+0000 7fda925fb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c06c5e0 con 0x7fda8c159b20 2023-10-16T13:50:24.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.126+0000 7fda925fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c06c8c0 con 0x7fda8c0b0b20 2023-10-16T13:50:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.127+0000 7fda8bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda8c159b20 0x7fda8c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.127+0000 7fda8bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda8c159b20 0x7fda8c06bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:47642/0 (socket says 172.21.15.60:47642) 2023-10-16T13:50:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.127+0000 7fda8bfff640 1 -- 172.21.15.60:0/1839308084 learned_addr learned my addr 172.21.15.60:0/1839308084 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.127+0000 7fda90b98640 1 --2- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.128+0000 7fda8bfff640 1 -- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 msgr2=0x7fda8c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.128+0000 7fda8bfff640 1 --2- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.128+0000 7fda8bfff640 1 -- 172.21.15.60:0/1839308084 >> v1:172.21.15.144:6789/0 conn(0x7fda8c159f00 legacy=0x7fda8c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.128+0000 7fda8bfff640 1 -- 172.21.15.60:0/1839308084 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c06cba0 con 0x7fda8c159b20 2023-10-16T13:50:24.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.129+0000 7fda8bfff640 1 --2- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159b20 0x7fda8c06bbe0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009870 tx=0x7fda7c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c21f2fb8140f29 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:24.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.129+0000 7fda8affd640 1 -- 172.21.15.60:0/1839308084 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c030840 con 0x7fda8c159b20 2023-10-16T13:50:24.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.129+0000 7fda8affd640 1 -- 172.21.15.60:0/1839308084 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c0309a0 con 0x7fda8c159b20 2023-10-16T13:50:24.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.129+0000 7fda8affd640 1 -- 172.21.15.60:0/1839308084 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c030d30 con 0x7fda8c159b20 2023-10-16T13:50:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 -- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159b20 msgr2=0x7fda8c06bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 --2- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159b20 0x7fda8c06bbe0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009870 tx=0x7fda7c02b980 comp rx=0 tx=0).stop 2023-10-16T13:50:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 -- 172.21.15.60:0/1839308084 shutdown_connections 2023-10-16T13:50:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 --2- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c06fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 --2- 172.21.15.60:0/1839308084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159b20 0x7fda8c06bbe0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 -- 172.21.15.60:0/1839308084 >> 172.21.15.60:0/1839308084 conn(0x7fda8c0cb410 msgr2=0x7fda8c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:24.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 -- 172.21.15.60:0/1839308084 shutdown_connections 2023-10-16T13:50:24.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.130+0000 7fda925fb640 1 -- 172.21.15.60:0/1839308084 wait complete. 2023-10-16T13:50:24.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.132+0000 7fda925fb640 1 Processor -- start 2023-10-16T13:50:24.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.132+0000 7fda925fb640 1 -- start start 2023-10-16T13:50:24.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.133+0000 7fda925fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda925fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 0x7fda8c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda925fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159f00 0x7fda8c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda925fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0c6470 con 0x7fda8c159b20 2023-10-16T13:50:24.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda925fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0c72d0 con 0x7fda8c159f00 2023-10-16T13:50:24.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda925fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0c7760 con 0x7fda8c0b0b20 2023-10-16T13:50:24.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda8b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 0x7fda8c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda8b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 0x7fda8c15a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53490/0 (socket says 172.21.15.60:53490) 2023-10-16T13:50:24.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda8b7fe640 1 -- 172.21.15.60:0/2144721776 learned_addr learned my addr 172.21.15.60:0/2144721776 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:24.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda90b98640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159f00 0x7fda8c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda8bfff640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda8b7fe640 1 -- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 msgr2=0x7fda8c158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.134+0000 7fda8b7fe640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda8b7fe640 1 -- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159f00 msgr2=0x7fda8c15a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda8b7fe640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159f00 0x7fda8c15a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda8b7fe640 1 -- 172.21.15.60:0/2144721776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0094d0 con 0x7fda8c159b20 2023-10-16T13:50:24.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda90b98640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159f00 0x7fda8c15a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:24.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda8b7fe640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 0x7fda8c15a310 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fda8000c430 tx=0x7fda8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:24.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda800052d0 con 0x7fda8c159b20 2023-10-16T13:50:24.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda80005430 con 0x7fda8c159b20 2023-10-16T13:50:24.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80003830 con 0x7fda8c159b20 2023-10-16T13:50:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c0c89f0 con 0x7fda8c159b20 2023-10-16T13:50:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.135+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c0c69a0 con 0x7fda8c159b20 2023-10-16T13:50:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.137+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c04e160 con 0x7fda8c159b20 2023-10-16T13:50:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.138+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fda80003990 con 0x7fda8c159b20 2023-10-16T13:50:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.139+0000 7fda88ff9640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 0x7fda7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.139+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fda80088f70 con 0x7fda8c159b20 2023-10-16T13:50:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.139+0000 7fda8bfff640 1 -- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 msgr2=0x7fda7806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.139+0000 7fda8bfff640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 0x7fda7806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.143+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda800bbfe0 con 0x7fda8c159b20 2023-10-16T13:50:24.151 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.148+0000 7ff6006da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:50:24.184 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.180+0000 7fa69880d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:50:24.215 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.212+0000 7ff6006da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:50:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.340+0000 7fda8bfff640 1 -- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 msgr2=0x7fda7806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.340+0000 7fda8bfff640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 0x7fda7806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:24.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.343+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda8c0c9610 con 0x7fda8c159b20 2023-10-16T13:50:24.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.344+0000 7fda88ff9640 1 -- 172.21.15.60:0/2144721776 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+17338 (secure 0 0 0) 0x7fda80017090 con 0x7fda8c159b20 2023-10-16T13:50:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:24.350 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":167,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:23.497052+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":167,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.30","mappings":[{"from":2,"to":0}]},{"pgid":"19.32","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3666412233":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/362473654":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1686539411":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6832/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/4230715707":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1256386612":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1071036703":"2023-10-17T12:54:28.068770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":51,"length":6},{"begin":107,"length":6}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:24.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 msgr2=0x7fda7806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:24.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 0x7fda7806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 msgr2=0x7fda8c15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 0x7fda8c15a310 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fda8000c430 tx=0x7fda8000c8f0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 shutdown_connections 2023-10-16T13:50:24.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fda7806af90 0x7fda7806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda8c159f00 0x7fda8c15a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda8c159b20 0x7fda8c15a310 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 --2- 172.21.15.60:0/2144721776 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda8c0b0b20 0x7fda8c158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 >> 172.21.15.60:0/2144721776 conn(0x7fda8c0cb410 msgr2=0x7fda8c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 shutdown_connections 2023-10-16T13:50:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.347+0000 7fda925fb640 1 -- 172.21.15.60:0/2144721776 wait complete. 2023-10-16T13:50:24.369 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:54:11.305180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:54:12.361464+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 167, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:50:24.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:50:24.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.592+0000 7f2b1182d640 1 Processor -- start 2023-10-16T13:50:24.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.592+0000 7f2b1182d640 1 -- start start 2023-10-16T13:50:24.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.593+0000 7f2b1182d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 0x7f2b0c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.593+0000 7f2b1182d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c0cb030 con 0x7f2b0c15ad70 2023-10-16T13:50:24.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.593+0000 7f2b1182d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c0c91c0 con 0x7f2b0c06a4b0 2023-10-16T13:50:24.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.593+0000 7f2b1182d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c0c94a0 con 0x7f2b0c0caac0 2023-10-16T13:50:24.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b0affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2b0c15ad70 0x7f2b0c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54202/0 (socket says 172.21.15.60:54202) 2023-10-16T13:50:24.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b0affd640 1 -- 172.21.15.60:0/398350544 learned_addr learned my addr 172.21.15.60:0/398350544 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:24.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b0b7fe640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 0x7f2b0c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357645719 0 0) 0x7f2b0c0cb030 con 0x7f2b0c15ad70 2023-10-16T13:50:24.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af4003520 con 0x7f2b0c15ad70 2023-10-16T13:50:24.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065058074 0 0) 0x7f2b0c0c91c0 con 0x7f2b0c06a4b0 2023-10-16T13:50:24.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b0c0cb030 con 0x7f2b0c06a4b0 2023-10-16T13:50:24.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 632872372 0 0) 0x7f2af4003520 con 0x7f2b0c15ad70 2023-10-16T13:50:24.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 msgr2=0x7f2b0c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:24.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 0x7f2b0c0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 >> v1:172.21.15.144:6789/0 conn(0x7f2b0c06a4b0 legacy=0x7f2b0c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c0c9780 con 0x7f2b0c15ad70 2023-10-16T13:50:24.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.594+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2afc001670 con 0x7f2b0c15ad70 2023-10-16T13:50:24.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 >> v1:172.21.15.60:6789/0 conn(0x7f2b0c15ad70 legacy=0x7f2b0c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af4003b30 0x7f2af4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2af4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 0x7f2af4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c0c94a0 con 0x7f2af40047c0 2023-10-16T13:50:24.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af400ba30 con 0x7f2af4003b30 2023-10-16T13:50:24.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af400bbf0 con 0x7f2b0c0caac0 2023-10-16T13:50:24.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2af4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0affd640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af4003b30 0x7f2af4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0b7fe640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 0x7f2af4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0a7fc640 1 -- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 msgr2=0x7f2af4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:24.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0caac0 0x7f2af4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0a7fc640 1 -- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af4003b30 msgr2=0x7f2af4003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af4003b30 0x7f2af4003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.595+0000 7f2b0a7fc640 1 -- 172.21.15.60:0/398350544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af4003520 con 0x7f2af40047c0 2023-10-16T13:50:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.597+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2af4004c40 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2b000018d0 tx=0x7f2af40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59c94e01f7576d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.597+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b000052d0 con 0x7f2af40047c0 2023-10-16T13:50:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.597+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b00005430 con 0x7f2af40047c0 2023-10-16T13:50:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.597+0000 7f2b09ffb640 1 -- 172.21.15.60:0/398350544 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00003810 con 0x7f2af40047c0 2023-10-16T13:50:24.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.597+0000 7f2b1182d640 1 -- 172.21.15.60:0/398350544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 msgr2=0x7f2af4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.598+0000 7f2b1182d640 1 --2- 172.21.15.60:0/398350544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2af4004c40 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2b000018d0 tx=0x7f2af40057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.598+0000 7f2b0b7fe640 1 -- 172.21.15.60:0/398350544 reap_dead start 2023-10-16T13:50:24.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.598+0000 7f2b1182d640 1 -- 172.21.15.60:0/398350544 shutdown_connections 2023-10-16T13:50:24.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.598+0000 7f2b1182d640 1 -- 172.21.15.60:0/398350544 >> 172.21.15.60:0/398350544 conn(0x7f2b0c0ce9b0 msgr2=0x7f2b0c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:24.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.598+0000 7f2b1182d640 1 -- 172.21.15.60:0/398350544 shutdown_connections 2023-10-16T13:50:24.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.598+0000 7f2b1182d640 1 -- 172.21.15.60:0/398350544 wait complete. 2023-10-16T13:50:24.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.600+0000 7f2b1182d640 1 Processor -- start 2023-10-16T13:50:24.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.600+0000 7f2b1182d640 1 -- start start 2023-10-16T13:50:24.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b1182d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2b0c0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b1182d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0b0d70 0x7f2b0c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b1182d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b0c158b00 0x7f2b0c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b1182d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c158fb0 con 0x7f2af40047c0 2023-10-16T13:50:24.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b1182d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c0c5390 con 0x7f2b0c158b00 2023-10-16T13:50:24.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b1182d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c0c27f0 con 0x7f2b0c0b0d70 2023-10-16T13:50:24.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b0affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2b0c0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b0affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2b0c0c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53516/0 (socket says 172.21.15.60:53516) 2023-10-16T13:50:24.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b0affd640 1 -- 172.21.15.60:0/2937027807 learned_addr learned my addr 172.21.15.60:0/2937027807 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:24.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0b0d70 0x7f2b0c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.601+0000 7f2b0b7fe640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b0c158b00 0x7f2b0c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:24.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0affd640 1 -- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0b0d70 msgr2=0x7f2b0c1585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0affd640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0b0d70 0x7f2b0c1585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0affd640 1 -- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b0c158b00 msgr2=0x7f2b0c151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0affd640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b0c158b00 0x7f2b0c151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0affd640 1 -- 172.21.15.60:0/2937027807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c0c2ad0 con 0x7f2af40047c0 2023-10-16T13:50:24.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0b0d70 0x7f2b0c1585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:24.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0b7fe640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b0c158b00 0x7f2b0c151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:24.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b0affd640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2b0c0c4e50 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f2afc001d10 tx=0x7f2afc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:24.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc00f700 con 0x7f2af40047c0 2023-10-16T13:50:24.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2afc00f860 con 0x7f2af40047c0 2023-10-16T13:50:24.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc00fc10 con 0x7f2af40047c0 2023-10-16T13:50:24.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0c0c2db0 con 0x7f2af40047c0 2023-10-16T13:50:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.602+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0c0cc580 con 0x7f2af40047c0 2023-10-16T13:50:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.604+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0c04e160 con 0x7f2af40047c0 2023-10-16T13:50:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.606+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2afc00fd70 con 0x7f2af40047c0 2023-10-16T13:50:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.606+0000 7f2af3fff640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f2af806afc0 0x7f2af806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:24.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.607+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f2afc0899f0 con 0x7f2af40047c0 2023-10-16T13:50:24.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.607+0000 7f2b0a7fc640 1 -- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f2af806afc0 msgr2=0x7f2af806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:24.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.607+0000 7f2b0a7fc640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f2af806afc0 0x7f2af806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:24.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.611+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2afc0547a0 con 0x7f2af40047c0 2023-10-16T13:50:24.651 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.648+0000 7fa69880d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:50:24.702 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.699+0000 7fa69880d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:50:24.724 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.721+0000 7ff6006da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:50:24.752 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.748+0000 7fa69880d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:50:24.779 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.775+0000 7ff6006da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:50:24.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.804+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b0c0cc740 con 0x7f2af40047c0 2023-10-16T13:50:24.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.805+0000 7f2af3fff640 1 -- 172.21.15.60:0/2937027807 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2afc057dc0 con 0x7f2af40047c0 2023-10-16T13:50:24.808 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:50:24.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f2af806afc0 msgr2=0x7f2af806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f2af806afc0 0x7f2af806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 msgr2=0x7f2b0c0c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2b0c0c4e50 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f2afc001d10 tx=0x7f2afc007430 comp rx=0 tx=0).stop 2023-10-16T13:50:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 shutdown_connections 2023-10-16T13:50:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f2af806afc0 0x7f2af806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b0c0b0d70 0x7f2b0c1585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b0c158b00 0x7f2b0c151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 --2- 172.21.15.60:0/2937027807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af40047c0 0x7f2b0c0c4e50 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:24.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 >> 172.21.15.60:0/2937027807 conn(0x7f2b0c0ce9b0 msgr2=0x7f2b0c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:24.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 shutdown_connections 2023-10-16T13:50:24.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:24.807+0000 7f2b1182d640 1 -- 172.21.15.60:0/2937027807 wait complete. 2023-10-16T13:50:24.813 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.808+0000 7fa69880d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:50:24.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:50:24.836 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.831+0000 7ff6006da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:50:24.860 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.857+0000 7fa69880d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:50:24.900 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.896+0000 7ff6006da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:50:24.952 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:24.949+0000 7ff6006da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:50:24.970 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:24.966+0000 7fa69880d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:50:25.033 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.029+0000 7fa69880d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:50:25.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.036+0000 7f5167fff640 1 Processor -- start 2023-10-16T13:50:25.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.036+0000 7f5167fff640 1 -- start start 2023-10-16T13:50:25.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.038+0000 7f5167fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f516815aaa0 0x7f51680b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.038+0000 7f5167fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f51680b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.038+0000 7f5167fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51680cad00 con 0x7f5168070240 2023-10-16T13:50:25.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.038+0000 7f5167fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51680b2d00 con 0x7f516815aaa0 2023-10-16T13:50:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.038+0000 7f5167fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51680b2fe0 con 0x7f51680ca8a0 2023-10-16T13:50:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.039+0000 7f51677fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5168070240 0x7f51680c67f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54210/0 (socket says 172.21.15.60:54210) 2023-10-16T13:50:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.039+0000 7f51677fe640 1 -- 172.21.15.60:0/552583643 learned_addr learned my addr 172.21.15.60:0/552583643 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.039+0000 7f5166ffd640 1 --2- 172.21.15.60:0/552583643 >> v2:172.21.15.144:3300/0 conn(0x7f516815aaa0 0x7f51680b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.039+0000 7f515ffff640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f51680b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.039+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680817892 0 0) 0x7f51680cad00 con 0x7f5168070240 2023-10-16T13:50:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.039+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5154003520 con 0x7f5168070240 2023-10-16T13:50:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1405348370 0 0) 0x7f5154003520 con 0x7f5168070240 2023-10-16T13:50:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 msgr2=0x7f51680b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f51680b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 >> v2:172.21.15.144:3300/0 conn(0x7f516815aaa0 msgr2=0x7f51680b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 --2- 172.21.15.60:0/552583643 >> v2:172.21.15.144:3300/0 conn(0x7f516815aaa0 0x7f51680b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680b32c0 con 0x7f5168070240 2023-10-16T13:50:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f51600019f0 con 0x7f5168070240 2023-10-16T13:50:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 >> v1:172.21.15.60:6789/0 conn(0x7f5168070240 legacy=0x7f51680c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5154003970 0x7f5154003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f5154004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f5154005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51680b2d00 con 0x7f51540047c0 2023-10-16T13:50:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51680b2fe0 con 0x7f5154003970 2023-10-16T13:50:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515400b980 con 0x7f51680ca8a0 2023-10-16T13:50:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f515ffff640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f5154004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f5166ffd640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5154003970 0x7f5154003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f51677fe640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f5154005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f515ffff640 1 -- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 msgr2=0x7f5154005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f515ffff640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f5154005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f515ffff640 1 -- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5154003970 msgr2=0x7f5154003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f515ffff640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5154003970 0x7f5154003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.040+0000 7f515ffff640 1 -- 172.21.15.60:0/552583643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154003520 con 0x7f51540047c0 2023-10-16T13:50:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.041+0000 7f51677fe640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f51680ca8a0 0x7f5154005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.042+0000 7f515ffff640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f5154004c40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f5150009ac0 tx=0x7f51540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abec2d1e8d5e63db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.042+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515002c6b0 con 0x7f51540047c0 2023-10-16T13:50:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.042+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515002c810 con 0x7f51540047c0 2023-10-16T13:50:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.042+0000 7f51667fc640 1 -- 172.21.15.60:0/552583643 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515002cba0 con 0x7f51540047c0 2023-10-16T13:50:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f5167fff640 1 -- 172.21.15.60:0/552583643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 msgr2=0x7f5154004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f5167fff640 1 --2- 172.21.15.60:0/552583643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f5154004c40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f5150009ac0 tx=0x7f51540057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f51677fe640 1 -- 172.21.15.60:0/552583643 reap_dead start 2023-10-16T13:50:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f5167fff640 1 -- 172.21.15.60:0/552583643 shutdown_connections 2023-10-16T13:50:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f5167fff640 1 -- 172.21.15.60:0/552583643 >> 172.21.15.60:0/552583643 conn(0x7f51680cc230 msgr2=0x7f51680cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f5167fff640 1 -- 172.21.15.60:0/552583643 shutdown_connections 2023-10-16T13:50:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.043+0000 7f5167fff640 1 -- 172.21.15.60:0/552583643 wait complete. 2023-10-16T13:50:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.045+0000 7f5167fff640 1 Processor -- start 2023-10-16T13:50:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.045+0000 7f5167fff640 1 -- start start 2023-10-16T13:50:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5167fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f51680afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5167fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f516806d830 0x7f516814bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5167fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51680b1870 0x7f51680b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5167fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51680b2230 con 0x7f51540047c0 2023-10-16T13:50:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5167fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5168069da0 con 0x7f51680b1870 2023-10-16T13:50:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5167fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516806dcb0 con 0x7f516806d830 2023-10-16T13:50:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f51680afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f51680afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53536/0 (socket says 172.21.15.60:53536) 2023-10-16T13:50:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 -- 172.21.15.60:0/3883235238 learned_addr learned my addr 172.21.15.60:0/3883235238 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f51677fe640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51680b1870 0x7f51680b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f515ffff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f516806d830 0x7f516814bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 -- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f516806d830 msgr2=0x7f516814bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f516806d830 0x7f516814bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 -- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51680b1870 msgr2=0x7f51680b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51680b1870 0x7f51680b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f5166ffd640 1 -- 172.21.15.60:0/3883235238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680b02b0 con 0x7f51540047c0 2023-10-16T13:50:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.046+0000 7f51677fe640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51680b1870 0x7f51680b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.047+0000 7f5166ffd640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f51680afc20 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f5158011700 tx=0x7f5158011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.047+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800f9c0 con 0x7f51540047c0 2023-10-16T13:50:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.047+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515800fb20 con 0x7f51540047c0 2023-10-16T13:50:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.047+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800fe90 con 0x7f51540047c0 2023-10-16T13:50:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.047+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51680b0590 con 0x7f51540047c0 2023-10-16T13:50:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.047+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51680b0b50 con 0x7f51540047c0 2023-10-16T13:50:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.048+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516804e9b0 con 0x7f51540047c0 2023-10-16T13:50:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.049+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5158015460 con 0x7f51540047c0 2023-10-16T13:50:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.051+0000 7f515f7fe640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 0x7f514806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.051+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f5158089ec0 con 0x7f51540047c0 2023-10-16T13:50:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.051+0000 7f515ffff640 1 -- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 msgr2=0x7f514806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.051+0000 7f515ffff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 0x7f514806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.055+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5158054c60 con 0x7f51540047c0 2023-10-16T13:50:25.072 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.070+0000 7ff6006da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:50:25.083 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.079+0000 7fa69880d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:50:25.139 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.136+0000 7ff6006da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:50:25.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.135+0000 7fa69880d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:50:25.189 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.186+0000 7fa69880d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:50:25.191 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.188+0000 7ff6006da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:50:25.250 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.247+0000 7ff6006da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:50:25.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.252+0000 7f515ffff640 1 -- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 msgr2=0x7f514806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:25.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.252+0000 7f515ffff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 0x7f514806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:25.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.252+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5168071580 con 0x7f51540047c0 2023-10-16T13:50:25.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.253+0000 7f515f7fe640 1 -- 172.21.15.60:0/3883235238 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v168) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5158022020 con 0x7f51540047c0 2023-10-16T13:50:25.256 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T13:50:25.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.255+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 msgr2=0x7f514806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:25.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.255+0000 7f5167fff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 0x7f514806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 msgr2=0x7f51680afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f51680afc20 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f5158011700 tx=0x7f5158011bc0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 shutdown_connections 2023-10-16T13:50:25.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f514806afc0 0x7f514806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f516806d830 0x7f516814bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51680b1870 0x7f51680b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 --2- 172.21.15.60:0/3883235238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51540047c0 0x7f51680afc20 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 >> 172.21.15.60:0/3883235238 conn(0x7f51680cc230 msgr2=0x7f516815d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:25.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 shutdown_connections 2023-10-16T13:50:25.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.256+0000 7f5167fff640 1 -- 172.21.15.60:0/3883235238 wait complete. 2023-10-16T13:50:25.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:50:25.306 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.303+0000 7ff6006da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:50:25.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.484+0000 7f5ef7c09640 1 Processor -- start 2023-10-16T13:50:25.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.484+0000 7f5ef7c09640 1 -- start start 2023-10-16T13:50:25.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.485+0000 7f5ef7c09640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ef015ad70 0x7f5ef0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.485+0000 7f5ef7c09640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00cb030 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.485+0000 7f5ef7c09640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00c91c0 con 0x7f5ef00caac0 2023-10-16T13:50:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.485+0000 7f5ef7c09640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00c94a0 con 0x7f5ef015ad70 2023-10-16T13:50:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef59a5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ef015ad70 0x7f5ef0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef59a5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ef015ad70 0x7f5ef0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53552/0 (socket says 172.21.15.60:53552) 2023-10-16T13:50:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef59a5640 1 -- 172.21.15.60:0/659766201 learned_addr learned my addr 172.21.15.60:0/659766201 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:25.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef51a4640 1 --1- 172.21.15.60:0/659766201 >> v1:172.21.15.60:6789/0 conn(0x7f5ef00c9ca0 0x7f5ef00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54216/0 (socket says 172.21.15.60:54216) 2023-10-16T13:50:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef59a5640 1 -- 172.21.15.60:0/659766201 >> v1:172.21.15.144:6789/0 conn(0x7f5ef00caac0 legacy=0x7f5ef00ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef59a5640 1 -- 172.21.15.60:0/659766201 >> v1:172.21.15.60:6789/0 conn(0x7f5ef00c9ca0 legacy=0x7f5ef00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef59a5640 1 -- 172.21.15.60:0/659766201 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00c9780 con 0x7f5ef015ad70 2023-10-16T13:50:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.486+0000 7f5ef49a3640 1 -- 172.21.15.60:0/659766201 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144279817 0 0) 0x7f5ef00cb030 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.487+0000 7f5ef59a5640 1 --2- 172.21.15.60:0/659766201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef015ad70 0x7f5ef0069da0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009870 tx=0x7f5eec02ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42aebf7b3280958d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:25.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef49a3640 1 -- 172.21.15.60:0/659766201 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec02f860 con 0x7f5ef015ad70 2023-10-16T13:50:25.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef49a3640 1 -- 172.21.15.60:0/659766201 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec02f9c0 con 0x7f5ef015ad70 2023-10-16T13:50:25.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef49a3640 1 -- 172.21.15.60:0/659766201 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec02fd50 con 0x7f5ef015ad70 2023-10-16T13:50:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef7c09640 1 -- 172.21.15.60:0/659766201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef015ad70 msgr2=0x7f5ef0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/659766201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef015ad70 0x7f5ef0069da0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009870 tx=0x7f5eec02ba40 comp rx=0 tx=0).stop 2023-10-16T13:50:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef7c09640 1 -- 172.21.15.60:0/659766201 shutdown_connections 2023-10-16T13:50:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/659766201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef015ad70 0x7f5ef0069da0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.488+0000 7f5ef7c09640 1 -- 172.21.15.60:0/659766201 >> 172.21.15.60:0/659766201 conn(0x7f5ef00ce9b0 msgr2=0x7f5ef00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.489+0000 7f5ef7c09640 1 -- 172.21.15.60:0/659766201 shutdown_connections 2023-10-16T13:50:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.489+0000 7f5ef7c09640 1 -- 172.21.15.60:0/659766201 wait complete. 2023-10-16T13:50:25.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.490+0000 7f5ef7c09640 1 Processor -- start 2023-10-16T13:50:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.491+0000 7f5ef7c09640 1 -- start start 2023-10-16T13:50:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef7c09640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 0x7f5ef014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef7c09640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef00caac0 0x7f5ef00aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef7c09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef015ad70 0x7f5ef00ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef7c09640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00a9260 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef7c09640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00a9540 con 0x7f5ef015ad70 2023-10-16T13:50:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef7c09640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00a9820 con 0x7f5ef00caac0 2023-10-16T13:50:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef59a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 0x7f5ef014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef59a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 0x7f5ef014bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53554/0 (socket says 172.21.15.60:53554) 2023-10-16T13:50:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef59a5640 1 -- 172.21.15.60:0/2270166960 learned_addr learned my addr 172.21.15.60:0/2270166960 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:25.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef59a5640 1 -- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef00caac0 msgr2=0x7f5ef00aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef59a5640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef00caac0 0x7f5ef00aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef59a5640 1 -- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef015ad70 msgr2=0x7f5ef00ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.492+0000 7f5ef61a6640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef015ad70 0x7f5ef00ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ef59a5640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef015ad70 0x7f5ef00ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ef59a5640 1 -- 172.21.15.60:0/2270166960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec0094d0 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ef61a6640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef015ad70 0x7f5ef00ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ef59a5640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 0x7f5ef014bcf0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f5eec000c00 tx=0x7f5eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec03b040 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec037d70 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec004270 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef00a9b00 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.493+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef014c230 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.495+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef004e160 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.496+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5eec0043d0 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.497+0000 7f5ee67fc640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 0x7f5ed406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.497+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f5eec0acb00 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.497+0000 7f5ef51a4640 1 -- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 msgr2=0x7f5ed406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:25.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.497+0000 7f5ef51a4640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 0x7f5ed406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:25.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.505+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5eec0aee60 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.569 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.565+0000 7fa69880d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:50:25.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.698+0000 7f5ef51a4640 1 -- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 msgr2=0x7f5ed406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:25.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.698+0000 7f5ef51a4640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 0x7f5ed406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:25.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.701+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ef00a9960 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.702+0000 7f5ee67fc640 1 -- 172.21.15.60:0/2270166960 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ef00a9960 con 0x7f5ef00c9ca0 2023-10-16T13:50:25.705 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T13:50:25.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 msgr2=0x7f5ed406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:25.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 0x7f5ed406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 msgr2=0x7f5ef014bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 0x7f5ef014bcf0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f5eec000c00 tx=0x7f5eec0046c0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 shutdown_connections 2023-10-16T13:50:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f5ed406af90 0x7f5ed406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef00caac0 0x7f5ef00aaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef015ad70 0x7f5ef00ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 --2- 172.21.15.60:0/2270166960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef00c9ca0 0x7f5ef014bcf0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 >> 172.21.15.60:0/2270166960 conn(0x7f5ef00ce9b0 msgr2=0x7f5ef0070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 shutdown_connections 2023-10-16T13:50:25.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.705+0000 7f5ef7c09640 1 -- 172.21.15.60:0/2270166960 wait complete. 2023-10-16T13:50:25.715 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.712+0000 7ff6006da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:50:25.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:50:25.727 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.724+0000 7fa69880d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:50:25.834 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.831+0000 7fa69880d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:50:25.883 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.880+0000 7fa69880d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:50:25.885 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.882+0000 7ff6006da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:50:25.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.976+0000 7fce5e5d5640 1 Processor -- start 2023-10-16T13:50:25.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.977+0000 7fce5e5d5640 1 -- start start 2023-10-16T13:50:25.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.978+0000 7fce5e5d5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fce580661c0 0x7fce580b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.978+0000 7fce5e5d5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580cd790 con 0x7fce580b20f0 2023-10-16T13:50:25.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.978+0000 7fce5e5d5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580ce310 con 0x7fce580b4f80 2023-10-16T13:50:25.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.978+0000 7fce5e5d5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce580cd9e0 con 0x7fce580661c0 2023-10-16T13:50:25.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.978+0000 7fce57fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fce580b20f0 0x7fce58065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54230/0 (socket says 172.21.15.60:54230) 2023-10-16T13:50:25.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.978+0000 7fce57fff640 1 -- 172.21.15.60:0/3483591546 learned_addr learned my addr 172.21.15.60:0/3483591546 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:25.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce4ffff640 1 --2- 172.21.15.60:0/3483591546 >> v2:172.21.15.144:3300/0 conn(0x7fce580661c0 0x7fce580b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850399561 0 0) 0x7fce580cd790 con 0x7fce580b20f0 2023-10-16T13:50:25.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce44003520 con 0x7fce580b20f0 2023-10-16T13:50:25.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 569811140 0 0) 0x7fce44003520 con 0x7fce580b20f0 2023-10-16T13:50:25.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 >> v2:172.21.15.144:3300/0 conn(0x7fce580661c0 msgr2=0x7fce580b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 --2- 172.21.15.60:0/3483591546 >> v2:172.21.15.144:3300/0 conn(0x7fce580661c0 0x7fce580b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 >> v1:172.21.15.144:6789/0 conn(0x7fce580b4f80 legacy=0x7fce5815c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580cdcc0 con 0x7fce580b20f0 2023-10-16T13:50:25.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.979+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fce48001670 con 0x7fce580b20f0 2023-10-16T13:50:25.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 >> v1:172.21.15.60:6789/0 conn(0x7fce580b20f0 legacy=0x7fce58065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce44003970 0x7fce44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce440047c0 0x7fce44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580cd9e0 con 0x7fce44005180 2023-10-16T13:50:25.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4400c120 con 0x7fce44003970 2023-10-16T13:50:25.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4400c2e0 con 0x7fce440047c0 2023-10-16T13:50:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce5cb72640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce57fff640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce44003970 0x7fce44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce4ffff640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce440047c0 0x7fce44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce5cb72640 1 -- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce440047c0 msgr2=0x7fce44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce5cb72640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce440047c0 0x7fce44004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce5cb72640 1 -- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce44003970 msgr2=0x7fce44003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:25.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce5cb72640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce44003970 0x7fce44003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.980+0000 7fce5cb72640 1 -- 172.21.15.60:0/3483591546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce44003520 con 0x7fce44005180 2023-10-16T13:50:25.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.981+0000 7fce5cb72640 1 -- 172.21.15.60:0/3483591546 reap_dead start 2023-10-16T13:50:25.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.981+0000 7fce5cb72640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce4400b980 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fce50001e50 tx=0x7fce440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c80aa5b7461734 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:25.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50003b70 con 0x7fce44005180 2023-10-16T13:50:25.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce50003cf0 con 0x7fce44005180 2023-10-16T13:50:25.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce577fe640 1 -- 172.21.15.60:0/3483591546 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50005310 con 0x7fce44005180 2023-10-16T13:50:25.991 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:25.987+0000 7fa69880d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:50:25.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce5e5d5640 1 -- 172.21.15.60:0/3483591546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 msgr2=0x7fce4400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce4400b980 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fce50001e50 tx=0x7fce440057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce5e5d5640 1 -- 172.21.15.60:0/3483591546 shutdown_connections 2023-10-16T13:50:25.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/3483591546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce4400b980 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce5e5d5640 1 -- 172.21.15.60:0/3483591546 >> 172.21.15.60:0/3483591546 conn(0x7fce580d2d10 msgr2=0x7fce580d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:25.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.982+0000 7fce5e5d5640 1 -- 172.21.15.60:0/3483591546 shutdown_connections 2023-10-16T13:50:25.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.983+0000 7fce5e5d5640 1 -- 172.21.15.60:0/3483591546 wait complete. 2023-10-16T13:50:25.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.984+0000 7fce5e5d5640 1 Processor -- start 2023-10-16T13:50:25.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.984+0000 7fce5e5d5640 1 -- start start 2023-10-16T13:50:25.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.985+0000 7fce5e5d5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce580c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.985+0000 7fce5e5d5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce580cada0 0x7fce580d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.985+0000 7fce5e5d5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce580cb220 0x7fce580d1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:25.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.985+0000 7fce5e5d5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580cb780 con 0x7fce44005180 2023-10-16T13:50:25.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.985+0000 7fce5e5d5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580cba60 con 0x7fce580cb220 2023-10-16T13:50:25.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.985+0000 7fce5e5d5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580cbd40 con 0x7fce580cada0 2023-10-16T13:50:25.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce580c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce580c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53568/0 (socket says 172.21.15.60:53568) 2023-10-16T13:50:25.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 -- 172.21.15.60:0/905247605 learned_addr learned my addr 172.21.15.60:0/905247605 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:25.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce5cb72640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce580cb220 0x7fce580d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce4ffff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce580cada0 0x7fce580d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 -- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce580cada0 msgr2=0x7fce580d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce580cada0 0x7fce580d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 -- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce580cb220 msgr2=0x7fce580d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce580cb220 0x7fce580d1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce57fff640 1 -- 172.21.15.60:0/905247605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580cc020 con 0x7fce44005180 2023-10-16T13:50:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.986+0000 7fce4ffff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce580cada0 0x7fce580d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:25.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.987+0000 7fce57fff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce580c8990 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fce48001050 tx=0x7fce4800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:25.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.987+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4800ee60 con 0x7fce44005180 2023-10-16T13:50:25.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.987+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4800c870 con 0x7fce44005180 2023-10-16T13:50:25.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.987+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4800cc20 con 0x7fce44005180 2023-10-16T13:50:25.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.987+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580cc300 con 0x7fce44005180 2023-10-16T13:50:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.987+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce580b6c00 con 0x7fce44005180 2023-10-16T13:50:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.988+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5804e160 con 0x7fce44005180 2023-10-16T13:50:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.989+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fce4801d460 con 0x7fce44005180 2023-10-16T13:50:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.990+0000 7fce557fa640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 0x7fce3806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.990+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fce480898a0 con 0x7fce44005180 2023-10-16T13:50:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.990+0000 7fce4ffff640 1 -- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 msgr2=0x7fce3806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.991+0000 7fce4ffff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 0x7fce3806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:26.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:25.994+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce480545b0 con 0x7fce44005180 2023-10-16T13:50:26.002 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:25.999+0000 7ff6006da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:50:26.055 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:26.052+0000 7ff6006da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:50:26.098 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:26.094+0000 7fa69880d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:50:26.169 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:26.166+0000 7ff6006da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:50:26.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.192+0000 7fce4ffff640 1 -- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 msgr2=0x7fce3806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:26.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.192+0000 7fce4ffff640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 0x7fce3806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:26.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.201+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fce58067fc0 con 0x7fce44005180 2023-10-16T13:50:26.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.202+0000 7fce557fa640 1 -- 172.21.15.60:0/905247605 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v168) v1 ==== 80+0+477 (secure 0 0 0) 0x7fce48057bd0 con 0x7fce44005180 2023-10-16T13:50:26.205 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3666412233 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.206 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/362473654 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.206 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1686539411 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.206 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/3094169155 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.206 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/3094169155 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.207 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/4230715707 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.207 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1256386612 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.207 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1071036703 2023-10-17T12:54:28.068770+0000 2023-10-16T13:50:26.208 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T13:50:26.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.204+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 msgr2=0x7fce3806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:26.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.204+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 0x7fce3806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.204+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 msgr2=0x7fce580c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:26.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.204+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce580c8990 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fce48001050 tx=0x7fce4800ec10 comp rx=0 tx=0).stop 2023-10-16T13:50:26.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 shutdown_connections 2023-10-16T13:50:26.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fce3806afc0 0x7fce3806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce580cada0 0x7fce580d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce580cb220 0x7fce580d1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 --2- 172.21.15.60:0/905247605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce44005180 0x7fce580c8990 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 >> 172.21.15.60:0/905247605 conn(0x7fce580d2d10 msgr2=0x7fce5815f520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 shutdown_connections 2023-10-16T13:50:26.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.205+0000 7fce5e5d5640 1 -- 172.21.15.60:0/905247605 wait complete. 2023-10-16T13:50:26.225 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:50:26.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:26.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:50:26.253 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:26.249+0000 7fa69880d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:50:26.284 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:26.281+0000 7ff6006da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:50:26.310 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:26.307+0000 7fa69880d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:50:26.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T13:50:26.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:26.341 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:50:26.384 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:50:26.385 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:26.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:50:26.435 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:50:26.436 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:26.443 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:26.440+0000 7fa69880d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:50:26.465 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:26.462+0000 7ff6006da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:50:26.490 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T13:50:26.491 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:50:26.491 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:26.517 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:26.515+0000 7ff6006da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:50:26.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:50:26.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:26.646 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:26.643+0000 7ff6006da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:50:26.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.756+0000 7fef68451640 1 Processor -- start 2023-10-16T13:50:26.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.756+0000 7fef68451640 1 -- start start 2023-10-16T13:50:26.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.757+0000 7fef68451640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef6015ae80 0x7fef6015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.757+0000 7fef68451640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef600ce390 0x7fef60070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.757+0000 7fef68451640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6015ed60 con 0x7fef6015aaa0 2023-10-16T13:50:26.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.757+0000 7fef68451640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6006b810 con 0x7fef6015ae80 2023-10-16T13:50:26.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.757+0000 7fef68451640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef600707e0 con 0x7fef600ce390 2023-10-16T13:50:26.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef661ed640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fef6015aaa0 0x7fef600cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54240/0 (socket says 172.21.15.60:54240) 2023-10-16T13:50:26.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef661ed640 1 -- 172.21.15.60:0/933134977 learned_addr learned my addr 172.21.15.60:0/933134977 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:26.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef659ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef6015ae80 0x7fef6015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:26.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef669ee640 1 --2- 172.21.15.60:0/933134977 >> v2:172.21.15.144:3300/0 conn(0x7fef600ce390 0x7fef60070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:26.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef659ec640 1 -- 172.21.15.60:0/933134977 >> v2:172.21.15.144:3300/0 conn(0x7fef600ce390 msgr2=0x7fef60070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:26.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef659ec640 1 --2- 172.21.15.60:0/933134977 >> v2:172.21.15.144:3300/0 conn(0x7fef600ce390 0x7fef60070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef659ec640 1 -- 172.21.15.60:0/933134977 >> v1:172.21.15.60:6789/0 conn(0x7fef6015aaa0 legacy=0x7fef600cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:26.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.758+0000 7fef659ec640 1 -- 172.21.15.60:0/933134977 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6015ef90 con 0x7fef6015ae80 2023-10-16T13:50:26.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.759+0000 7fef659ec640 1 --2- 172.21.15.60:0/933134977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015ae80 0x7fef6015e820 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fef600c2490 tx=0x7fef5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52825a388dda5910 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:26.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef651eb640 1 -- 172.21.15.60:0/933134977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50031830 con 0x7fef6015ae80 2023-10-16T13:50:26.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef651eb640 1 -- 172.21.15.60:0/933134977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50031990 con 0x7fef6015ae80 2023-10-16T13:50:26.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef651eb640 1 -- 172.21.15.60:0/933134977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50031d20 con 0x7fef6015ae80 2023-10-16T13:50:26.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 -- 172.21.15.60:0/933134977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015ae80 msgr2=0x7fef6015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:26.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 --2- 172.21.15.60:0/933134977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015ae80 0x7fef6015e820 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fef600c2490 tx=0x7fef5002b8d0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 -- 172.21.15.60:0/933134977 shutdown_connections 2023-10-16T13:50:26.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 --2- 172.21.15.60:0/933134977 >> v2:172.21.15.144:3300/0 conn(0x7fef600ce390 0x7fef60070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 --2- 172.21.15.60:0/933134977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015ae80 0x7fef6015e820 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 -- 172.21.15.60:0/933134977 >> 172.21.15.60:0/933134977 conn(0x7fef600cb410 msgr2=0x7fef600cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:26.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 -- 172.21.15.60:0/933134977 shutdown_connections 2023-10-16T13:50:26.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.760+0000 7fef68451640 1 -- 172.21.15.60:0/933134977 wait complete. 2023-10-16T13:50:26.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.762+0000 7fef68451640 1 Processor -- start 2023-10-16T13:50:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.762+0000 7fef68451640 1 -- start start 2023-10-16T13:50:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef68451640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef600ce390 0x7fef60155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef68451640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 0x7fef600af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef68451640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef6015ae80 0x7fef600af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef68451640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600c5d80 con 0x7fef6015aaa0 2023-10-16T13:50:26.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef68451640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef60156370 con 0x7fef600ce390 2023-10-16T13:50:26.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef68451640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef600c63e0 con 0x7fef6015ae80 2023-10-16T13:50:26.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef659ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 0x7fef600af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:26.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef659ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 0x7fef600af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53584/0 (socket says 172.21.15.60:53584) 2023-10-16T13:50:26.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.763+0000 7fef659ec640 1 -- 172.21.15.60:0/1357971260 learned_addr learned my addr 172.21.15.60:0/1357971260 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:26.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef661ed640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef600ce390 0x7fef60155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:26.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef669ee640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef6015ae80 0x7fef600af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:26.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef659ec640 1 -- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef6015ae80 msgr2=0x7fef600af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:26.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef659ec640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef6015ae80 0x7fef600af8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef659ec640 1 -- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef600ce390 msgr2=0x7fef60155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:26.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef659ec640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef600ce390 0x7fef60155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:26.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef659ec640 1 -- 172.21.15.60:0/1357971260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50009400 con 0x7fef6015aaa0 2023-10-16T13:50:26.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.764+0000 7fef659ec640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 0x7fef600af3b0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fef50009890 tx=0x7fef500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:26.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.765+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5003aa60 con 0x7fef6015aaa0 2023-10-16T13:50:26.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.765+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5003abe0 con 0x7fef6015aaa0 2023-10-16T13:50:26.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.765+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef500423f0 con 0x7fef6015aaa0 2023-10-16T13:50:26.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.765+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6015b9a0 con 0x7fef6015aaa0 2023-10-16T13:50:26.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.765+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6015b350 con 0x7fef6015aaa0 2023-10-16T13:50:26.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.766+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6004e160 con 0x7fef6015aaa0 2023-10-16T13:50:26.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.767+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fef5004b3b0 con 0x7fef6015aaa0 2023-10-16T13:50:26.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.768+0000 7fef56ffd640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 0x7fef4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:26.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.768+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fef500ab9e0 con 0x7fef6015aaa0 2023-10-16T13:50:26.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.768+0000 7fef661ed640 1 -- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 msgr2=0x7fef4406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:26.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.768+0000 7fef661ed640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 0x7fef4406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:26.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.772+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef50076700 con 0x7fef6015aaa0 2023-10-16T13:50:26.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.969+0000 7fef661ed640 1 -- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 msgr2=0x7fef4406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:26.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:26.969+0000 7fef661ed640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 0x7fef4406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:27.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.120+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef600cdc60 con 0x7fef6015aaa0 2023-10-16T13:50:27.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.121+0000 7fef56ffd640 1 -- 172.21.15.60:0/1357971260 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fef50047090 con 0x7fef6015aaa0 2023-10-16T13:50:27.124 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:27.124 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:54:13.310628+0000","modified":"2023-10-16T12:54:16.431585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8964":{"gid":8964,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3372982800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3372982800},{"type":"v1","addr":"172.21.15.60:6833","nonce":3372982800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:50:27.125 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 79 2023-10-16T13:50:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 msgr2=0x7fef4406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 0x7fef4406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 msgr2=0x7fef600af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 0x7fef600af3b0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fef50009890 tx=0x7fef500046c0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 shutdown_connections 2023-10-16T13:50:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7fef4406af90 0x7fef4406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef6015ae80 0x7fef600af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef6015aaa0 0x7fef600af3b0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 --2- 172.21.15.60:0/1357971260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef600ce390 0x7fef60155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 >> 172.21.15.60:0/1357971260 conn(0x7fef600cb410 msgr2=0x7fef600b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:27.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.123+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 shutdown_connections 2023-10-16T13:50:27.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.124+0000 7fef68451640 1 -- 172.21.15.60:0/1357971260 wait complete. 2023-10-16T13:50:27.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:27.282 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.279+0000 7fa69880d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:50:27.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.343+0000 7f88587e3640 1 Processor -- start 2023-10-16T13:50:27.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.344+0000 7f88587e3640 1 -- start start 2023-10-16T13:50:27.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.345+0000 7f88587e3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 0x7f885015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.345+0000 7f88587e3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885015ed60 con 0x7f88500ce390 2023-10-16T13:50:27.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.345+0000 7f88587e3640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500707e0 con 0x7f885015aaa0 2023-10-16T13:50:27.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.345+0000 7f88587e3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885006b810 con 0x7f885015ae80 2023-10-16T13:50:27.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f8856d80640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f88500ce390 0x7f8850070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54256/0 (socket says 172.21.15.60:54256) 2023-10-16T13:50:27.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f8856d80640 1 -- 172.21.15.60:0/2944067169 learned_addr learned my addr 172.21.15.60:0/2944067169 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:27.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f8855d7e640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 0x7f885015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931370147 0 0) 0x7f885015ed60 con 0x7f88500ce390 2023-10-16T13:50:27.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f883c003520 con 0x7f88500ce390 2023-10-16T13:50:27.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074987102 0 0) 0x7f883c003520 con 0x7f88500ce390 2023-10-16T13:50:27.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 msgr2=0x7f885015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:27.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 0x7f885015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 >> v1:172.21.15.144:6789/0 conn(0x7f885015aaa0 legacy=0x7f88500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.346+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885015ef90 con 0x7f88500ce390 2023-10-16T13:50:27.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f88480019f0 con 0x7f88500ce390 2023-10-16T13:50:27.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 >> v1:172.21.15.60:6789/0 conn(0x7f88500ce390 legacy=0x7f8850070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 0x7f883c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f883c004020 0x7f883c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f883c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885006b810 con 0x7f883c004d00 2023-10-16T13:50:27.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c00bbc0 con 0x7f883c004020 2023-10-16T13:50:27.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c00bd80 con 0x7f885015ae80 2023-10-16T13:50:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f8856d80640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f883c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f885657f640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 0x7f883c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f8855d7e640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f883c004020 0x7f883c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f8856d80640 1 -- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 msgr2=0x7f883c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:27.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f8856d80640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015ae80 0x7f883c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f8856d80640 1 -- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f883c004020 msgr2=0x7f883c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.347+0000 7f8856d80640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f883c004020 0x7f883c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.348+0000 7f8856d80640 1 -- 172.21.15.60:0/2944067169 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c003520 con 0x7f883c004d00 2023-10-16T13:50:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.348+0000 7f8855d7e640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f883c004020 0x7f883c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:27.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.349+0000 7f8856d80640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f883c005180 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f884800a760 tx=0x7f883c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa33cd6c551bb9a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:27.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.349+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848034630 con 0x7f883c004d00 2023-10-16T13:50:27.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.349+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884802f070 con 0x7f883c004d00 2023-10-16T13:50:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.349+0000 7f885557d640 1 -- 172.21.15.60:0/2944067169 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480349f0 con 0x7f883c004d00 2023-10-16T13:50:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f88587e3640 1 -- 172.21.15.60:0/2944067169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 msgr2=0x7f883c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f88587e3640 1 --2- 172.21.15.60:0/2944067169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f883c005180 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f884800a760 tx=0x7f883c0057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f8856d80640 1 -- 172.21.15.60:0/2944067169 reap_dead start 2023-10-16T13:50:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f88587e3640 1 -- 172.21.15.60:0/2944067169 shutdown_connections 2023-10-16T13:50:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f88587e3640 1 -- 172.21.15.60:0/2944067169 >> 172.21.15.60:0/2944067169 conn(0x7f88500cb410 msgr2=0x7f88500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f88587e3640 1 -- 172.21.15.60:0/2944067169 shutdown_connections 2023-10-16T13:50:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.350+0000 7f88587e3640 1 -- 172.21.15.60:0/2944067169 wait complete. 2023-10-16T13:50:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.352+0000 7f88587e3640 1 Processor -- start 2023-10-16T13:50:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.352+0000 7f88587e3640 1 -- start start 2023-10-16T13:50:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f88587e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f885014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f88587e3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015d6c0 0x7f88501efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f88587e3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88501f39f0 0x7f88501f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f88587e3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885015e3f0 con 0x7f883c004d00 2023-10-16T13:50:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f88587e3640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8850144860 con 0x7f88501f39f0 2023-10-16T13:50:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f88587e3640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8850155a00 con 0x7f885015d6c0 2023-10-16T13:50:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.353+0000 7f885657f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f885014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f885014a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53592/0 (socket says 172.21.15.60:53592) 2023-10-16T13:50:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 -- 172.21.15.60:0/2662903603 learned_addr learned my addr 172.21.15.60:0/2662903603 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f8855d7e640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015d6c0 0x7f88501efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f8856d80640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88501f39f0 0x7f88501f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 -- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015d6c0 msgr2=0x7f88501efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015d6c0 0x7f88501efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 -- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88501f39f0 msgr2=0x7f88501f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88501f39f0 0x7f88501f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 -- 172.21.15.60:0/2662903603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885015de30 con 0x7f883c004d00 2023-10-16T13:50:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f8856d80640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88501f39f0 0x7f88501f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.354+0000 7f885657f640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f885014a970 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e580 tx=0x7f884c00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.355+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c0052e0 con 0x7f883c004d00 2023-10-16T13:50:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.355+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c005440 con 0x7f883c004d00 2023-10-16T13:50:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.355+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c003810 con 0x7f883c004d00 2023-10-16T13:50:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.355+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885015e110 con 0x7f883c004d00 2023-10-16T13:50:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.355+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885015db40 con 0x7f883c004d00 2023-10-16T13:50:27.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.356+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850155860 con 0x7f883c004d00 2023-10-16T13:50:27.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.357+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 84) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f884c003970 con 0x7f883c004d00 2023-10-16T13:50:27.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.358+0000 7f8842ffd640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 0x7f883406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.358+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6199+0+0 (secure 0 0 0) 0x7f884c089740 con 0x7f883c004d00 2023-10-16T13:50:27.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.358+0000 7f8855d7e640 1 -- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 msgr2=0x7f883406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:27.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.358+0000 7f8855d7e640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 0x7f883406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:27.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.362+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8850155860 con 0x7f883c004d00 2023-10-16T13:50:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.559+0000 7f8855d7e640 1 -- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 msgr2=0x7f883406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/110255757 2023-10-16T13:50:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.559+0000 7f8855d7e640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 0x7f883406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.563+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8850155860 con 0x7f883c004d00 2023-10-16T13:50:27.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.565+0000 7f8842ffd640 1 -- 172.21.15.60:0/2662903603 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+17336 (secure 0 0 0) 0x7f8850155860 con 0x7f883c004d00 2023-10-16T13:50:27.570 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:27.570 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":169,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:26.284122+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.30","mappings":[{"from":2,"to":0}]},{"pgid":"19.32","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3666412233":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/362473654":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1686539411":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6832/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/4230715707":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1256386612":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1071036703":"2023-10-17T12:54:28.068770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":6,"length":52},{"begin":62,"length":52}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":3,"length":3},{"begin":59,"length":3}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 msgr2=0x7f883406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 0x7f883406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 msgr2=0x7f885014a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f885014a970 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e580 tx=0x7f884c00ea40 comp rx=0 tx=0).stop 2023-10-16T13:50:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 shutdown_connections 2023-10-16T13:50:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:6832/110255757,v1:172.21.15.144:6833/110255757] conn(0x7f883406af90 0x7f883406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f885015d6c0 0x7f88501efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88501f39f0 0x7f88501f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 --2- 172.21.15.60:0/2662903603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f883c004d00 0x7f885014a970 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 >> 172.21.15.60:0/2662903603 conn(0x7f88500cb410 msgr2=0x7f88500b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 shutdown_connections 2023-10-16T13:50:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.568+0000 7f88587e3640 1 -- 172.21.15.60:0/2662903603 wait complete. 2023-10-16T13:50:27.580 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.576+0000 7fa69880d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:50:27.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:54:11.305180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:54:12.361464+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 168, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:50:27.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:50:27.612 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:27.609+0000 7ff6006da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:50:27.635 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.632+0000 7fa69880d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:50:27.748 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.745+0000 7fa69880d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:50:27.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.797+0000 7f7ef531a640 1 Processor -- start 2023-10-16T13:50:27.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.797+0000 7f7ef531a640 1 -- start start 2023-10-16T13:50:27.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7ef531a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef015ad10 0x7f7ef00b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7ef531a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef0070c40 con 0x7f7ef0065cc0 2023-10-16T13:50:27.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7ef531a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef006e4c0 con 0x7f7ef00ce250 2023-10-16T13:50:27.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7ef531a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0066100 con 0x7f7ef015ad10 2023-10-16T13:50:27.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7ef4b19640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7ef00ce250 0x7f7ef00710d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:54752/0 (socket says 172.21.15.60:54752) 2023-10-16T13:50:27.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7ef4b19640 1 -- 172.21.15.60:0/3181387083 learned_addr learned my addr 172.21.15.60:0/3181387083 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:27.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.799+0000 7f7eeffff640 1 --2- 172.21.15.60:0/3181387083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef015ad10 0x7f7ef00b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.800+0000 7f7eeffff640 1 -- 172.21.15.60:0/3181387083 >> v1:172.21.15.144:6789/0 conn(0x7f7ef00ce250 legacy=0x7f7ef00710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.800+0000 7f7eeffff640 1 -- 172.21.15.60:0/3181387083 >> v1:172.21.15.60:6789/0 conn(0x7f7ef0065cc0 legacy=0x7f7ef00cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.800+0000 7f7eeeffd640 1 -- 172.21.15.60:0/3181387083 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668360820 0 0) 0x7f7ef006e4c0 con 0x7f7ef00ce250 2023-10-16T13:50:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.800+0000 7f7eeffff640 1 -- 172.21.15.60:0/3181387083 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef00b49c0 con 0x7f7ef015ad10 2023-10-16T13:50:27.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.801+0000 7f7eeffff640 1 --2- 172.21.15.60:0/3181387083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef015ad10 0x7f7ef00b1db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00097d0 tx=0x7f7ee002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f992cfaec4ba67b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.801+0000 7f7eeeffd640 1 -- 172.21.15.60:0/3181387083 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0030800 con 0x7f7ef015ad10 2023-10-16T13:50:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.801+0000 7f7eeeffd640 1 -- 172.21.15.60:0/3181387083 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee0030960 con 0x7f7ef015ad10 2023-10-16T13:50:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.801+0000 7f7eeeffd640 1 -- 172.21.15.60:0/3181387083 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0030cf0 con 0x7f7ef015ad10 2023-10-16T13:50:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 -- 172.21.15.60:0/3181387083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef015ad10 msgr2=0x7f7ef00b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 --2- 172.21.15.60:0/3181387083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef015ad10 0x7f7ef00b1db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00097d0 tx=0x7f7ee002b940 comp rx=0 tx=0).stop 2023-10-16T13:50:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 -- 172.21.15.60:0/3181387083 shutdown_connections 2023-10-16T13:50:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 --2- 172.21.15.60:0/3181387083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef015ad10 0x7f7ef00b1db0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 -- 172.21.15.60:0/3181387083 >> 172.21.15.60:0/3181387083 conn(0x7f7ef00b0a50 msgr2=0x7f7ef00b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 -- 172.21.15.60:0/3181387083 shutdown_connections 2023-10-16T13:50:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.802+0000 7f7ef531a640 1 -- 172.21.15.60:0/3181387083 wait complete. 2023-10-16T13:50:27.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.804+0000 7f7ef531a640 1 Processor -- start 2023-10-16T13:50:27.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.804+0000 7f7ef531a640 1 -- start start 2023-10-16T13:50:27.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef531a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 0x7f7ef014c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef531a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef00ce250 0x7f7ef01520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef531a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef015ad10 0x7f7ef0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:27.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef531a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef009bb50 con 0x7f7ef0065cc0 2023-10-16T13:50:27.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef531a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef009ad40 con 0x7f7ef015ad10 2023-10-16T13:50:27.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef531a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0099da0 con 0x7f7ef00ce250 2023-10-16T13:50:27.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7eeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 0x7f7ef014c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7eeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 0x7f7ef014c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53594/0 (socket says 172.21.15.60:53594) 2023-10-16T13:50:27.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7eeffff640 1 -- 172.21.15.60:0/3314108619 learned_addr learned my addr 172.21.15.60:0/3314108619 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:27.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7ef4b19640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef015ad10 0x7f7ef0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.805+0000 7f7eef7fe640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef00ce250 0x7f7ef01520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:27.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eeffff640 1 -- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef00ce250 msgr2=0x7f7ef01520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eeffff640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef00ce250 0x7f7ef01520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eeffff640 1 -- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef015ad10 msgr2=0x7f7ef0152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eeffff640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef015ad10 0x7f7ef0152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eeffff640 1 -- 172.21.15.60:0/3314108619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee0009400 con 0x7f7ef0065cc0 2023-10-16T13:50:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eeffff640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 0x7f7ef014c570 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f7ee002bf50 tx=0x7f7ee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:27.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee003e070 con 0x7f7ef0065cc0 2023-10-16T13:50:27.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee0039a60 con 0x7f7ef0065cc0 2023-10-16T13:50:27.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef009b260 con 0x7f7ef0065cc0 2023-10-16T13:50:27.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.806+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0039e10 con 0x7f7ef0065cc0 2023-10-16T13:50:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.807+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef014f530 con 0x7f7ef0065cc0 2023-10-16T13:50:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.808+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef004e9b0 con 0x7f7ef0065cc0 2023-10-16T13:50:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.809+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 85) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7ee004b400 con 0x7f7ef0065cc0 2023-10-16T13:50:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.810+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f7ee00ab8c0 con 0x7f7ef0065cc0 2023-10-16T13:50:27.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:27.813+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7ee0076ad0 con 0x7f7ef0065cc0 2023-10-16T13:50:27.839 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.836+0000 7fa61a6c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.840 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.836+0000 7fa61a6c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.840 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.836+0000 7fa61a6c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.840 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.836+0000 7fa61a6c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.840 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.836+0000 7fa61a6c3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.841 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.837+0000 7fa61d6c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.841 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.837+0000 7fa61d6c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.841 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.837+0000 7fa61d6c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.837+0000 7fa61d6c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.842 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:27.837+0000 7fa61d6c9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:50:27.945 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:27.942+0000 7ff6006da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:50:28.004 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:28.001+0000 7ff6006da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:50:28.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.015+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ef00c9ee0 con 0x7f7ef0065cc0 2023-10-16T13:50:28.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.016+0000 7f7eecff9640 1 -- 172.21.15.60:0/3314108619 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v170) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ee007a0f0 con 0x7f7ef0065cc0 2023-10-16T13:50:28.019 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:50:28.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 msgr2=0x7f7ef014c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 0x7f7ef014c570 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f7ee002bf50 tx=0x7f7ee00046c0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 shutdown_connections 2023-10-16T13:50:28.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7ef00ce250 0x7f7ef01520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ef015ad10 0x7f7ef0152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 --2- 172.21.15.60:0/3314108619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ef0065cc0 0x7f7ef014c570 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.018+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 >> 172.21.15.60:0/3314108619 conn(0x7f7ef00b0a50 msgr2=0x7f7ef006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.019+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 shutdown_connections 2023-10-16T13:50:28.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.019+0000 7f7ef531a640 1 -- 172.21.15.60:0/3314108619 wait complete. 2023-10-16T13:50:28.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:50:28.124 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:28.121+0000 7ff6006da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:50:28.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.294+0000 7fd9a0ebb640 1 Processor -- start 2023-10-16T13:50:28.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.295+0000 7fd9a0ebb640 1 -- start start 2023-10-16T13:50:28.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.296+0000 7fd9a0ebb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd99c15ad70 0x7fd99c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.296+0000 7fd9a0ebb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0cb030 con 0x7fd99c0c9ca0 2023-10-16T13:50:28.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.296+0000 7fd9a0ebb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0c91c0 con 0x7fd99c0caac0 2023-10-16T13:50:28.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.296+0000 7fd9a0ebb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99c0c94a0 con 0x7fd99c15ad70 2023-10-16T13:50:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.296+0000 7fd999d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd99c0c9ca0 0x7fd99c0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54268/0 (socket says 172.21.15.60:54268) 2023-10-16T13:50:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.296+0000 7fd999d9b640 1 -- 172.21.15.60:0/246271508 learned_addr learned my addr 172.21.15.60:0/246271508 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99a59c640 1 --2- 172.21.15.60:0/246271508 >> v2:172.21.15.144:3300/0 conn(0x7fd99c15ad70 0x7fd99c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086897241 0 0) 0x7fd99c0cb030 con 0x7fd99c0c9ca0 2023-10-16T13:50:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984003520 con 0x7fd99c0c9ca0 2023-10-16T13:50:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2470795864 0 0) 0x7fd984003520 con 0x7fd99c0c9ca0 2023-10-16T13:50:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 >> v2:172.21.15.144:3300/0 conn(0x7fd99c15ad70 msgr2=0x7fd99c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 --2- 172.21.15.60:0/246271508 >> v2:172.21.15.144:3300/0 conn(0x7fd99c15ad70 0x7fd99c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 >> v1:172.21.15.144:6789/0 conn(0x7fd99c0caac0 legacy=0x7fd99c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c0c9780 con 0x7fd99c0c9ca0 2023-10-16T13:50:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.297+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd990001670 con 0x7fd99c0c9ca0 2023-10-16T13:50:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 >> v1:172.21.15.60:6789/0 conn(0x7fd99c0c9ca0 legacy=0x7fd99c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9840039c0 0x7fd984003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9840047c0 0x7fd984004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd984005180 0x7fd98400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0c94a0 con 0x7fd984005180 2023-10-16T13:50:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98400c080 con 0x7fd9840047c0 2023-10-16T13:50:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98400c240 con 0x7fd9840039c0 2023-10-16T13:50:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99ad9d640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd984005180 0x7fd98400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99ad9d640 1 -- 172.21.15.60:0/246271508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9840039c0 msgr2=0x7fd984003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99ad9d640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd9840039c0 0x7fd984003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99ad9d640 1 -- 172.21.15.60:0/246271508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9840047c0 msgr2=0x7fd984004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99ad9d640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9840047c0 0x7fd984004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.298+0000 7fd99ad9d640 1 -- 172.21.15.60:0/246271508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984003520 con 0x7fd984005180 2023-10-16T13:50:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.299+0000 7fd99ad9d640 1 -- 172.21.15.60:0/246271508 reap_dead start 2023-10-16T13:50:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.299+0000 7fd99ad9d640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd984005180 0x7fd98400b980 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fd994001e50 tx=0x7fd9840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1968fdb3922c8c1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994003b70 con 0x7fd984005180 2023-10-16T13:50:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994003d10 con 0x7fd984005180 2023-10-16T13:50:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd99959a640 1 -- 172.21.15.60:0/246271508 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994005310 con 0x7fd984005180 2023-10-16T13:50:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/246271508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd984005180 msgr2=0x7fd98400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd984005180 0x7fd98400b980 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fd994001e50 tx=0x7fd9840057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/246271508 shutdown_connections 2023-10-16T13:50:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 --2- 172.21.15.60:0/246271508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd984005180 0x7fd98400b980 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/246271508 >> 172.21.15.60:0/246271508 conn(0x7fd99c0ce9b0 msgr2=0x7fd99c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/246271508 shutdown_connections 2023-10-16T13:50:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.300+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/246271508 wait complete. 2023-10-16T13:50:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.302+0000 7fd9a0ebb640 1 Processor -- start 2023-10-16T13:50:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.302+0000 7fd9a0ebb640 1 -- start start 2023-10-16T13:50:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd9a0ebb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd984005180 0x7fd99c0aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd9a0ebb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 0x7fd99c0ad370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd9a0ebb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd99c0ad8b0 0x7fd99c0b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd9a0ebb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0ac890 con 0x7fd99c157ee0 2023-10-16T13:50:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd9a0ebb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0cddd0 con 0x7fd984005180 2023-10-16T13:50:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd9a0ebb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0c8f00 con 0x7fd99c0ad8b0 2023-10-16T13:50:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.303+0000 7fd999d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 0x7fd99c0ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 0x7fd99c0ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53608/0 (socket says 172.21.15.60:53608) 2023-10-16T13:50:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 -- 172.21.15.60:0/3995521004 learned_addr learned my addr 172.21.15.60:0/3995521004 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd99a59c640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd984005180 0x7fd99c0aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd99ad9d640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd99c0ad8b0 0x7fd99c0b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 -- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd99c0ad8b0 msgr2=0x7fd99c0b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd99c0ad8b0 0x7fd99c0b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 -- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd984005180 msgr2=0x7fd99c0aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd984005180 0x7fd99c0aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 -- 172.21.15.60:0/3995521004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c0cdb60 con 0x7fd99c157ee0 2023-10-16T13:50:28.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.304+0000 7fd999d9b640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 0x7fd99c0ad370 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fd99000c730 tx=0x7fd99000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:28.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.305+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9900072d0 con 0x7fd99c157ee0 2023-10-16T13:50:28.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.305+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd990007450 con 0x7fd99c157ee0 2023-10-16T13:50:28.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.305+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9900047e0 con 0x7fd99c157ee0 2023-10-16T13:50:28.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.305+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c15d780 con 0x7fd99c157ee0 2023-10-16T13:50:28.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.305+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c0acae0 con 0x7fd99c157ee0 2023-10-16T13:50:28.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.306+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99c15dca0 con 0x7fd99c157ee0 2023-10-16T13:50:28.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.307+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 85) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd990004940 con 0x7fd99c157ee0 2023-10-16T13:50:28.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.308+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fd9900887b0 con 0x7fd99c157ee0 2023-10-16T13:50:28.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.312+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd99c15dca0 con 0x7fd99c157ee0 2023-10-16T13:50:28.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.503+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd99c15dca0 con 0x7fd99c157ee0 2023-10-16T13:50:28.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.504+0000 7fd98affd640 1 -- 172.21.15.60:0/3995521004 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v170) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd990018020 con 0x7fd99c157ee0 2023-10-16T13:50:28.507 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T13:50:28.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 msgr2=0x7fd99c0ad370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 0x7fd99c0ad370 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fd99000c730 tx=0x7fd99000cbf0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 shutdown_connections 2023-10-16T13:50:28.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd99c0ad8b0 0x7fd99c0b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd99c157ee0 0x7fd99c0ad370 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 --2- 172.21.15.60:0/3995521004 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd984005180 0x7fd99c0aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.506+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 >> 172.21.15.60:0/3995521004 conn(0x7fd99c0ce9b0 msgr2=0x7fd99c0c5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.507+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 shutdown_connections 2023-10-16T13:50:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.507+0000 7fd9a0ebb640 1 -- 172.21.15.60:0/3995521004 wait complete. 2023-10-16T13:50:28.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:50:28.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.806+0000 7f03501dc640 1 Processor -- start 2023-10-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.806+0000 7f03501dc640 1 -- start start 2023-10-16T13:50:28.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f03501dc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f034806b8c0 0x7f034806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f03501dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034815d130 0x7f03480cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f03501dc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03480cd390 con 0x7f034806b1a0 2023-10-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f03501dc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034806c1e0 con 0x7f034806b8c0 2023-10-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f03501dc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03480cc2f0 con 0x7f034815d130 2023-10-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f034df78640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f034806b8c0 0x7f034806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f034df78640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f034806b8c0 0x7f034806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53610/0 (socket says 172.21.15.60:53610) 2023-10-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.808+0000 7f034df78640 1 -- 172.21.15.60:0/3745085356 learned_addr learned my addr 172.21.15.60:0/3745085356 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034d777640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034815d130 0x7f03480cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034df78640 1 -- 172.21.15.60:0/3745085356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034815d130 msgr2=0x7f03480cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034df78640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034815d130 0x7f03480cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034df78640 1 -- 172.21.15.60:0/3745085356 >> v1:172.21.15.60:6789/0 conn(0x7f034806b1a0 legacy=0x7f03480cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034df78640 1 -- 172.21.15.60:0/3745085356 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03480cc5d0 con 0x7f034806b8c0 2023-10-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034cf76640 1 -- 172.21.15.60:0/3745085356 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34571924 0 0) 0x7f03480cd390 con 0x7f034806b1a0 2023-10-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.809+0000 7f034d777640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034815d130 0x7f03480cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.810+0000 7f034df78640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806b8c0 0x7f034806bca0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f0344009860 tx=0x7f034402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50db412591f10c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.810+0000 7f034cf76640 1 -- 172.21.15.60:0/3745085356 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344030840 con 0x7f034806b8c0 2023-10-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.810+0000 7f034cf76640 1 -- 172.21.15.60:0/3745085356 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03440309a0 con 0x7f034806b8c0 2023-10-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.810+0000 7f034cf76640 1 -- 172.21.15.60:0/3745085356 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344030d30 con 0x7f034806b8c0 2023-10-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 -- 172.21.15.60:0/3745085356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806b8c0 msgr2=0x7f034806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806b8c0 0x7f034806bca0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f0344009860 tx=0x7f034402b980 comp rx=0 tx=0).stop 2023-10-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 -- 172.21.15.60:0/3745085356 shutdown_connections 2023-10-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034815d130 0x7f03480cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 --2- 172.21.15.60:0/3745085356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034806b8c0 0x7f034806bca0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 -- 172.21.15.60:0/3745085356 >> 172.21.15.60:0/3745085356 conn(0x7f03480d0ad0 msgr2=0x7f03480d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 -- 172.21.15.60:0/3745085356 shutdown_connections 2023-10-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.811+0000 7f03501dc640 1 -- 172.21.15.60:0/3745085356 wait complete. 2023-10-16T13:50:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.813+0000 7f03501dc640 1 Processor -- start 2023-10-16T13:50:28.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.813+0000 7f03501dc640 1 -- start start 2023-10-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f03501dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806b1a0 0x7f03480c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f03501dc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034806b8c0 0x7f03480c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f03501dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 0x7f03480c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f03501dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03480c8520 con 0x7f034815d130 2023-10-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f03501dc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481597d0 con 0x7f034806b8c0 2023-10-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f03501dc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03480c87c0 con 0x7f034806b1a0 2023-10-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 0x7f03480c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 0x7f03480c5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53616/0 (socket says 172.21.15.60:53616) 2023-10-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 -- 172.21.15.60:0/262805404 learned_addr learned my addr 172.21.15.60:0/262805404 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 -- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806b1a0 msgr2=0x7f03480c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034d777640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034806b8c0 0x7f03480c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034df78640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806b1a0 0x7f03480c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806b1a0 0x7f03480c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 -- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034806b8c0 msgr2=0x7f03480c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034806b8c0 0x7f03480c56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034e779640 1 -- 172.21.15.60:0/262805404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0344009470 con 0x7f034815d130 2023-10-16T13:50:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034d777640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034806b8c0 0x7f03480c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.814+0000 7f034df78640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806b1a0 0x7f03480c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.815+0000 7f034e779640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 0x7f03480c5c10 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f034000c7c0 tx=0x7f034000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:28.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.815+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340003da0 con 0x7f034815d130 2023-10-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.815+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0340005070 con 0x7f034815d130 2023-10-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.815+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340005420 con 0x7f034815d130 2023-10-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.815+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03480c8aa0 con 0x7f034815d130 2023-10-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.815+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03480c9060 con 0x7f034815d130 2023-10-16T13:50:28.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.817+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034804e160 con 0x7f034815d130 2023-10-16T13:50:28.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.817+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 86) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0340005580 con 0x7f034815d130 2023-10-16T13:50:28.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.819+0000 7f033e7fc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f032c06acc0 0x7f032c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:28.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.819+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f0340088de0 con 0x7f034815d130 2023-10-16T13:50:28.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.820+0000 7f034df78640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f032c06acc0 0x7f032c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:28.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.821+0000 7f034df78640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f032c06acc0 0x7f032c06d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0344036000 tx=0x7f034403b040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:28.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:28.824+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0340053eb0 con 0x7f034815d130 2023-10-16T13:50:29.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.027+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03480c4320 con 0x7f034815d130 2023-10-16T13:50:29.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.028+0000 7f033e7fc640 1 -- 172.21.15.60:0/262805404 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v170) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0340017070 con 0x7f034815d130 2023-10-16T13:50:29.031 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f032c06acc0 msgr2=0x7f032c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f032c06acc0 0x7f032c06d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0344036000 tx=0x7f034403b040 comp rx=0 tx=0).stop 2023-10-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 msgr2=0x7f03480c5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 0x7f03480c5c10 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f034000c7c0 tx=0x7f034000cc80 comp rx=0 tx=0).stop 2023-10-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 shutdown_connections 2023-10-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f032c06acc0 0x7f032c06d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f034815d130 0x7f03480c5c10 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f034806b8c0 0x7f03480c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 --2- 172.21.15.60:0/262805404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f034806b1a0 0x7f03480c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.031+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 >> 172.21.15.60:0/262805404 conn(0x7f03480d0ad0 msgr2=0x7f03480ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:29.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.032+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 shutdown_connections 2023-10-16T13:50:29.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.032+0000 7f03501dc640 1 -- 172.21.15.60:0/262805404 wait complete. 2023-10-16T13:50:29.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:29.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.280+0000 7f29f2c7e640 1 Processor -- start 2023-10-16T13:50:29.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.280+0000 7f29f2c7e640 1 -- start start 2023-10-16T13:50:29.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.282+0000 7f29f2c7e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f29ec0ad970 0x7f29ec0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.282+0000 7f29f2c7e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec15b9d0 con 0x7f29ec06c040 2023-10-16T13:50:29.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.282+0000 7f29f2c7e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec15bcb0 con 0x7f29ec0ccfb0 2023-10-16T13:50:29.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.282+0000 7f29f2c7e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ec15bf90 con 0x7f29ec0ad970 2023-10-16T13:50:29.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.282+0000 7f29f1c7c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f29ec06c040 0x7f29ec06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41282/0 (socket says 172.21.15.60:41282) 2023-10-16T13:50:29.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.282+0000 7f29f1c7c640 1 -- 172.21.15.60:0/77118104 learned_addr learned my addr 172.21.15.60:0/77118104 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:29.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f147b640 1 --2- 172.21.15.60:0/77118104 >> v2:172.21.15.144:3300/0 conn(0x7f29ec0ad970 0x7f29ec0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498424148 0 0) 0x7f29ec15b9d0 con 0x7f29ec06c040 2023-10-16T13:50:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d8003520 con 0x7f29ec06c040 2023-10-16T13:50:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753941399 0 0) 0x7f29d8003520 con 0x7f29ec06c040 2023-10-16T13:50:29.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 >> v2:172.21.15.144:3300/0 conn(0x7f29ec0ad970 msgr2=0x7f29ec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 --2- 172.21.15.60:0/77118104 >> v2:172.21.15.144:3300/0 conn(0x7f29ec0ad970 0x7f29ec0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 >> v1:172.21.15.144:6789/0 conn(0x7f29ec0ccfb0 legacy=0x7f29ec0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec0cc270 con 0x7f29ec06c040 2023-10-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f29e8001670 con 0x7f29ec06c040 2023-10-16T13:50:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f147b640 1 --2- 172.21.15.60:0/77118104 >> v2:172.21.15.144:3300/0 conn(0x7f29ec0ad970 0x7f29ec0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 >> v1:172.21.15.60:6789/0 conn(0x7f29ec06c040 legacy=0x7f29ec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8003970 0x7f29d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d80047c0 0x7f29d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d8005180 0x7f29d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec15bf90 con 0x7f29d8005180 2023-10-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d800c0d0 con 0x7f29d8003970 2023-10-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.283+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d800c290 con 0x7f29d80047c0 2023-10-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.284+0000 7f29f247d640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d8005180 0x7f29d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.284+0000 7f29f247d640 1 -- 172.21.15.60:0/77118104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d80047c0 msgr2=0x7f29d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.284+0000 7f29f247d640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d80047c0 0x7f29d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.284+0000 7f29f247d640 1 -- 172.21.15.60:0/77118104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8003970 msgr2=0x7f29d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.284+0000 7f29f247d640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8003970 0x7f29d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.284+0000 7f29f247d640 1 -- 172.21.15.60:0/77118104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8003520 con 0x7f29d8005180 2023-10-16T13:50:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.285+0000 7f29f247d640 1 -- 172.21.15.60:0/77118104 reap_dead start 2023-10-16T13:50:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.285+0000 7f29f247d640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d8005180 0x7f29d800b980 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f29e4001e50 tx=0x7f29d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe51892e3cc849f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.285+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4003b70 con 0x7f29d8005180 2023-10-16T13:50:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e4003cf0 con 0x7f29d8005180 2023-10-16T13:50:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f0c7a640 1 -- 172.21.15.60:0/77118104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4005310 con 0x7f29d8005180 2023-10-16T13:50:29.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 -- 172.21.15.60:0/77118104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d8005180 msgr2=0x7f29d800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d8005180 0x7f29d800b980 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f29e4001e50 tx=0x7f29d80057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 -- 172.21.15.60:0/77118104 shutdown_connections 2023-10-16T13:50:29.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/77118104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d8005180 0x7f29d800b980 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 -- 172.21.15.60:0/77118104 >> 172.21.15.60:0/77118104 conn(0x7f29ec0ce6a0 msgr2=0x7f29ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:29.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 -- 172.21.15.60:0/77118104 shutdown_connections 2023-10-16T13:50:29.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.286+0000 7f29f2c7e640 1 -- 172.21.15.60:0/77118104 wait complete. 2023-10-16T13:50:29.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.288+0000 7f29f2c7e640 1 Processor -- start 2023-10-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.288+0000 7f29f2c7e640 1 -- start start 2023-10-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f2c7e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8005180 0x7f29ec15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f2c7e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ec15d6f0 0x7f29ec1f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f2c7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 0x7f29ec0c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f2c7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec0ca2d0 con 0x7f29ec0c98e0 2023-10-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f2c7e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec1f3060 con 0x7f29d8005180 2023-10-16T13:50:29.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f2c7e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec1f65b0 con 0x7f29ec15d6f0 2023-10-16T13:50:29.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.289+0000 7f29f247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 0x7f29ec0c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 0x7f29ec0c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49008/0 (socket says 172.21.15.60:49008) 2023-10-16T13:50:29.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 -- 172.21.15.60:0/90115876 learned_addr learned my addr 172.21.15.60:0/90115876 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:29.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f147b640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ec15d6f0 0x7f29ec1f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 -- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ec15d6f0 msgr2=0x7f29ec1f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ec15d6f0 0x7f29ec1f2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 -- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8005180 msgr2=0x7f29ec15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8005180 0x7f29ec15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 -- 172.21.15.60:0/90115876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec15db70 con 0x7f29ec0c98e0 2023-10-16T13:50:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29f247d640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 0x7f29ec0c9d90 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c060 tx=0x7f29e40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.290+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4003b80 con 0x7f29ec0c98e0 2023-10-16T13:50:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.291+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e4006b50 con 0x7f29ec0c98e0 2023-10-16T13:50:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.291+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4007030 con 0x7f29ec0c98e0 2023-10-16T13:50:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.291+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec1f5a30 con 0x7f29ec0c98e0 2023-10-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.291+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec1f5ff0 con 0x7f29ec0c98e0 2023-10-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.292+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ec06b330 con 0x7f29ec0c98e0 2023-10-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.293+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 86) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f29e4007190 con 0x7f29ec0c98e0 2023-10-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.294+0000 7f29e27fc640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f29d006acc0 0x7f29d006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.294+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f29e40ab600 con 0x7f29ec0c98e0 2023-10-16T13:50:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.295+0000 7f29f1c7c640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f29d006acc0 0x7f29d006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.296+0000 7f29f1c7c640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f29d006acc0 0x7f29d006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f29e8000f60 tx=0x7f29e80015c0 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:29.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.299+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29e4076570 con 0x7f29ec0c98e0 2023-10-16T13:50:29.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.533+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29ec0ca2d0 con 0x7f29ec0c98e0 2023-10-16T13:50:29.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.534+0000 7f29e27fc640 1 -- 172.21.15.60:0/90115876 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f29ec0ca2d0 con 0x7f29ec0c98e0 2023-10-16T13:50:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:54:13.310628+0000","modified":"2023-10-16T12:54:16.431585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8964":{"gid":8964,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3372982800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3372982800},{"type":"v1","addr":"172.21.15.60:6833","nonce":3372982800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 79 2023-10-16T13:50:29.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f29d006acc0 msgr2=0x7f29d006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f29d006acc0 0x7f29d006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f29e8000f60 tx=0x7f29e80015c0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 msgr2=0x7f29ec0c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 0x7f29ec0c9d90 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c060 tx=0x7f29e40014a0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 shutdown_connections 2023-10-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f29d006acc0 0x7f29d006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ec15d6f0 0x7f29ec1f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ec0c98e0 0x7f29ec0c9d90 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 --2- 172.21.15.60:0/90115876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d8005180 0x7f29ec15e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 >> 172.21.15.60:0/90115876 conn(0x7f29ec0ce6a0 msgr2=0x7f29ec0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:29.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 shutdown_connections 2023-10-16T13:50:29.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.537+0000 7f29f2c7e640 1 -- 172.21.15.60:0/90115876 wait complete. 2023-10-16T13:50:29.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:29.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.769+0000 7fb2ad533640 1 Processor -- start 2023-10-16T13:50:29.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.769+0000 7fb2ad533640 1 -- start start 2023-10-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2ad533640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2a806a4b0 0x7fb2a806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2ad533640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2a80aeb50 0x7fb2a806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2ad533640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a815b430 con 0x7fb2a815b050 2023-10-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2ad533640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a80d0620 con 0x7fb2a806a4b0 2023-10-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2ad533640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a80d0900 con 0x7fb2a80aeb50 2023-10-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2a67fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2a806a4b0 0x7fb2a806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2a67fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2a806a4b0 0x7fb2a806c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49022/0 (socket says 172.21.15.60:49022) 2023-10-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.771+0000 7fb2a67fc640 1 -- 172.21.15.60:0/2216771227 learned_addr learned my addr 172.21.15.60:0/2216771227 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.772+0000 7fb2a77fe640 1 --2- 172.21.15.60:0/2216771227 >> v2:172.21.15.144:3300/0 conn(0x7fb2a80aeb50 0x7fb2a806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.772+0000 7fb2a67fc640 1 -- 172.21.15.60:0/2216771227 >> v2:172.21.15.144:3300/0 conn(0x7fb2a80aeb50 msgr2=0x7fb2a806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:29.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.772+0000 7fb2a67fc640 1 --2- 172.21.15.60:0/2216771227 >> v2:172.21.15.144:3300/0 conn(0x7fb2a80aeb50 0x7fb2a806cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.772+0000 7fb2a67fc640 1 -- 172.21.15.60:0/2216771227 >> v1:172.21.15.144:6789/0 conn(0x7fb2a815b050 legacy=0x7fb2a8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:29.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.772+0000 7fb2a67fc640 1 -- 172.21.15.60:0/2216771227 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a80b5680 con 0x7fb2a806a4b0 2023-10-16T13:50:29.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.772+0000 7fb2a77fe640 1 --2- 172.21.15.60:0/2216771227 >> v2:172.21.15.144:3300/0 conn(0x7fb2a80aeb50 0x7fb2a806cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:29.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.773+0000 7fb2a67fc640 1 --2- 172.21.15.60:0/2216771227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a806c6c0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb2a806c0e0 tx=0x7fb29c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773ecbdc9a7a85cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:29.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.778+0000 7fb2a5ffb640 1 -- 172.21.15.60:0/2216771227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c030860 con 0x7fb2a806a4b0 2023-10-16T13:50:29.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.778+0000 7fb2a5ffb640 1 -- 172.21.15.60:0/2216771227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29c0309c0 con 0x7fb2a806a4b0 2023-10-16T13:50:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.778+0000 7fb2a5ffb640 1 -- 172.21.15.60:0/2216771227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c030d50 con 0x7fb2a806a4b0 2023-10-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 -- 172.21.15.60:0/2216771227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 msgr2=0x7fb2a806c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 --2- 172.21.15.60:0/2216771227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a806c6c0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb2a806c0e0 tx=0x7fb29c02b9a0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 -- 172.21.15.60:0/2216771227 shutdown_connections 2023-10-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 --2- 172.21.15.60:0/2216771227 >> v2:172.21.15.144:3300/0 conn(0x7fb2a80aeb50 0x7fb2a806cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 --2- 172.21.15.60:0/2216771227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a806c6c0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 -- 172.21.15.60:0/2216771227 >> 172.21.15.60:0/2216771227 conn(0x7fb2a80cfb00 msgr2=0x7fb2a80cff10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:29.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 -- 172.21.15.60:0/2216771227 shutdown_connections 2023-10-16T13:50:29.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.779+0000 7fb2ad533640 1 -- 172.21.15.60:0/2216771227 wait complete. 2023-10-16T13:50:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.781+0000 7fb2ad533640 1 Processor -- start 2023-10-16T13:50:29.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.781+0000 7fb2ad533640 1 -- start start 2023-10-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2ad533640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a80c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2ad533640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2a80aeb50 0x7fb2a80c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2ad533640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2a815b050 0x7fb2a815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2ad533640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80c39a0 con 0x7fb2a806a4b0 2023-10-16T13:50:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2ad533640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80c3c80 con 0x7fb2a80aeb50 2023-10-16T13:50:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2ad533640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80c3f60 con 0x7fb2a815b050 2023-10-16T13:50:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a80c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2a6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a80c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49028/0 (socket says 172.21.15.60:49028) 2023-10-16T13:50:29.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.782+0000 7fb2a6ffd640 1 -- 172.21.15.60:0/4028538260 learned_addr learned my addr 172.21.15.60:0/4028538260 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2a6ffd640 1 -- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2a815b050 msgr2=0x7fb2a815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2a6ffd640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2a815b050 0x7fb2a815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2a6ffd640 1 -- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2a80aeb50 msgr2=0x7fb2a80c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2a6ffd640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2a80aeb50 0x7fb2a80c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2a6ffd640 1 -- 172.21.15.60:0/4028538260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0094d0 con 0x7fb2a806a4b0 2023-10-16T13:50:29.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2a6ffd640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a80c6000 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fb29800b3e0 tx=0x7fb29800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:29.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298011830 con 0x7fb2a806a4b0 2023-10-16T13:50:29.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2980119b0 con 0x7fb2a806a4b0 2023-10-16T13:50:29.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298011d60 con 0x7fb2a806a4b0 2023-10-16T13:50:29.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.783+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a80c4240 con 0x7fb2a806a4b0 2023-10-16T13:50:29.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.784+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a80c3630 con 0x7fb2a806a4b0 2023-10-16T13:50:29.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.785+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a806dcb0 con 0x7fb2a806a4b0 2023-10-16T13:50:29.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.785+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 87) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb29801a400 con 0x7fb2a806a4b0 2023-10-16T13:50:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.786+0000 7fb28f7fe640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fb29406af30 0x7fb29406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.786+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fb2980899f0 con 0x7fb2a806a4b0 2023-10-16T13:50:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.788+0000 7fb2a67fc640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fb29406af30 0x7fb29406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:29.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.789+0000 7fb2a67fc640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fb29406af30 0x7fb29406d3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb29c036000 tx=0x7fb29c03b040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:29.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.791+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb29808be60 con 0x7fb2a806a4b0 2023-10-16T13:50:29.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.990+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a804e160 con 0x7fb2a806a4b0 2023-10-16T13:50:29.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.993+0000 7fb28f7fe640 1 -- 172.21.15.60:0/4028538260 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+17775 (secure 0 0 0) 0x7fb298015090 con 0x7fb2a806a4b0 2023-10-16T13:50:29.996 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:29.997 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":170,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:27.750456+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.30","mappings":[{"from":2,"to":0}]},{"pgid":"19.32","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1674745327":"2023-10-17T13:50:27.750339+0000","172.21.15.144:6832/110255757":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/1615247958":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/1009875684":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/931153185":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/3784462343":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/3666412233":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/362473654":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1686539411":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/4019169897":"2023-10-17T13:50:27.750339+0000","172.21.15.144:6832/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/110255757":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/4230715707":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1256386612":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1071036703":"2023-10-17T12:54:28.068770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":19,"snaps":[{"begin":6,"length":52},{"begin":62,"length":52}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fb29406af30 msgr2=0x7fb29406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fb29406af30 0x7fb29406d3f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb29c036000 tx=0x7fb29c03b040 comp rx=0 tx=0).stop 2023-10-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 msgr2=0x7fb2a80c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a80c6000 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fb29800b3e0 tx=0x7fb29800b8a0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 shutdown_connections 2023-10-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fb29406af30 0x7fb29406d3f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2a815b050 0x7fb2a815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2a80aeb50 0x7fb2a80c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 --2- 172.21.15.60:0/4028538260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2a806a4b0 0x7fb2a80c6000 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 >> 172.21.15.60:0/4028538260 conn(0x7fb2a80cfb00 msgr2=0x7fb2a80b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:30.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 shutdown_connections 2023-10-16T13:50:30.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:29.996+0000 7fb2ad533640 1 -- 172.21.15.60:0/4028538260 wait complete. 2023-10-16T13:50:30.017 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:50:30.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.231+0000 7fafa484a640 1 Processor -- start 2023-10-16T13:50:30.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.232+0000 7fafa484a640 1 -- start start 2023-10-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.233+0000 7fafa484a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.233+0000 7fafa484a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fafa015cfc0 0x7fafa00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.233+0000 7fafa484a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fafa00d1090 0x7fafa00b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.233+0000 7fafa484a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa00d15a0 con 0x7fafa00d1090 2023-10-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.233+0000 7fafa484a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa00d1880 con 0x7fafa015cfc0 2023-10-16T13:50:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.233+0000 7fafa484a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00d1b60 con 0x7fafa015cbe0 2023-10-16T13:50:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fafa00d1090 0x7fafa00b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fafa00d1090 0x7fafa00b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49034/0 (socket says 172.21.15.60:49034) 2023-10-16T13:50:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 -- 172.21.15.60:0/3104058904 learned_addr learned my addr 172.21.15.60:0/3104058904 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 -- 172.21.15.60:0/3104058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 msgr2=0x7fafa00d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 --2- 172.21.15.60:0/3104058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 -- 172.21.15.60:0/3104058904 >> v2:172.21.15.144:3300/0 conn(0x7fafa015cfc0 msgr2=0x7fafa00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 --2- 172.21.15.60:0/3104058904 >> v2:172.21.15.144:3300/0 conn(0x7fafa015cfc0 0x7fafa00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.234+0000 7faf9f59e640 1 -- 172.21.15.60:0/3104058904 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00d1e40 con 0x7fafa00d1090 2023-10-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.235+0000 7faf9f59e640 1 --2- 172.21.15.60:0/3104058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00b4ab0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fafa00724e0 tx=0x7faf9802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec70b1b003ad7af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.236+0000 7faf9e59c640 1 -- 172.21.15.60:0/3104058904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9802cb00 con 0x7fafa00d1090 2023-10-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.236+0000 7faf9e59c640 1 -- 172.21.15.60:0/3104058904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9802cc60 con 0x7fafa00d1090 2023-10-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.236+0000 7faf9e59c640 1 -- 172.21.15.60:0/3104058904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf980343f0 con 0x7fafa00d1090 2023-10-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.236+0000 7fafa484a640 1 -- 172.21.15.60:0/3104058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 msgr2=0x7fafa00b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.236+0000 7fafa484a640 1 --2- 172.21.15.60:0/3104058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00b4ab0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fafa00724e0 tx=0x7faf9802bf60 comp rx=0 tx=0).stop 2023-10-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 -- 172.21.15.60:0/3104058904 shutdown_connections 2023-10-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 --2- 172.21.15.60:0/3104058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00b4ab0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 --2- 172.21.15.60:0/3104058904 >> v2:172.21.15.144:3300/0 conn(0x7fafa015cfc0 0x7fafa00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 --2- 172.21.15.60:0/3104058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 -- 172.21.15.60:0/3104058904 >> 172.21.15.60:0/3104058904 conn(0x7fafa00cf570 msgr2=0x7fafa00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:30.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 -- 172.21.15.60:0/3104058904 shutdown_connections 2023-10-16T13:50:30.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.237+0000 7fafa484a640 1 -- 172.21.15.60:0/3104058904 wait complete. 2023-10-16T13:50:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.239+0000 7fafa484a640 1 Processor -- start 2023-10-16T13:50:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.239+0000 7fafa484a640 1 -- start start 2023-10-16T13:50:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7fafa484a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7fafa484a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7fafa484a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa015cfc0 0x7fafa00cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7fafa484a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa015f8e0 con 0x7fafa00d1090 2023-10-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7fafa484a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa015ef40 con 0x7fafa015cfc0 2023-10-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7fafa484a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa015f220 con 0x7fafa015cbe0 2023-10-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7faf9ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7faf9ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49046/0 (socket says 172.21.15.60:49046) 2023-10-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7faf9ed9d640 1 -- 172.21.15.60:0/2629463037 learned_addr learned my addr 172.21.15.60:0/2629463037 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7faf9ed9d640 1 -- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 msgr2=0x7fafa00cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.240+0000 7faf97fff640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9f59e640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa015cfc0 0x7fafa00cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9ed9d640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9ed9d640 1 -- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa015cfc0 msgr2=0x7fafa00cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9ed9d640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa015cfc0 0x7fafa00cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9ed9d640 1 -- 172.21.15.60:0/2629463037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf98009b30 con 0x7fafa00d1090 2023-10-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf97fff640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9f59e640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa015cfc0 0x7fafa00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf9ed9d640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00c6620 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7faf9000b4a0 tx=0x7faf9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90010610 con 0x7fafa00d1090 2023-10-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90010770 con 0x7fafa00d1090 2023-10-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90010b20 con 0x7fafa00d1090 2023-10-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.241+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa015f500 con 0x7fafa00d1090 2023-10-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.242+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa00c6d30 con 0x7fafa00d1090 2023-10-16T13:50:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.243+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa00d2100 con 0x7fafa00d1090 2023-10-16T13:50:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.244+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 87) v1 ==== 89275+0+0 (secure 0 0 0) 0x7faf90010c80 con 0x7fafa00d1090 2023-10-16T13:50:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.245+0000 7faf977fe640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7faf8006af00 0x7faf8006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.245+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7faf90089640 con 0x7fafa00d1090 2023-10-16T13:50:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.245+0000 7faf97fff640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7faf8006af00 0x7faf8006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.247+0000 7faf97fff640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7faf8006af00 0x7faf8006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fafa0152910 tx=0x7faf88008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:30.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.249+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafa00d2100 con 0x7fafa00d1090 2023-10-16T13:50:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.484+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa00d2100 con 0x7fafa00d1090 2023-10-16T13:50:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.485+0000 7faf977fe640 1 -- 172.21.15.60:0/2629463037 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fafa00d2100 con 0x7fafa00d1090 2023-10-16T13:50:30.489 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:30.489 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:54:13.310628+0000","modified":"2023-10-16T12:54:16.431585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8964":{"gid":8964,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3372982800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3372982800},{"type":"v1","addr":"172.21.15.60:6833","nonce":3372982800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:50:30.490 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 79 2023-10-16T13:50:30.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7faf8006af00 msgr2=0x7faf8006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7faf8006af00 0x7faf8006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fafa0152910 tx=0x7faf88008040 comp rx=0 tx=0).stop 2023-10-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 msgr2=0x7fafa00c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00c6620 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7faf9000b4a0 tx=0x7faf9000b960 comp rx=0 tx=0).stop 2023-10-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 shutdown_connections 2023-10-16T13:50:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7faf8006af00 0x7faf8006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafa015cbe0 0x7fafa00cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa015cfc0 0x7fafa00cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 --2- 172.21.15.60:0/2629463037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafa00d1090 0x7fafa00c6620 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 >> 172.21.15.60:0/2629463037 conn(0x7fafa00cf570 msgr2=0x7fafa00cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:30.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 shutdown_connections 2023-10-16T13:50:30.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.488+0000 7fafa484a640 1 -- 172.21.15.60:0/2629463037 wait complete. 2023-10-16T13:50:30.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.731+0000 7ff5aacad640 1 Processor -- start 2023-10-16T13:50:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.732+0000 7ff5aacad640 1 -- start start 2023-10-16T13:50:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.733+0000 7ff5aacad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a415c9f0 0x7ff5a40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.733+0000 7ff5aacad640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5a40c9230 0x7ff5a40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.733+0000 7ff5aacad640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a4160a90 con 0x7ff5a4070f20 2023-10-16T13:50:30.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.733+0000 7ff5aacad640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a40cdec0 con 0x7ff5a40c9230 2023-10-16T13:50:30.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.733+0000 7ff5aacad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40cd900 con 0x7ff5a415c9f0 2023-10-16T13:50:30.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5aa4ac640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff5a4070f20 0x7ff5a40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41288/0 (socket says 172.21.15.60:41288) 2023-10-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5aa4ac640 1 -- 172.21.15.60:0/1921604181 learned_addr learned my addr 172.21.15.60:0/1921604181 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5a94aa640 1 --2- 172.21.15.60:0/1921604181 >> v2:172.21.15.60:3300/0 conn(0x7ff5a40c9230 0x7ff5a40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5a9cab640 1 --2- 172.21.15.60:0/1921604181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a415c9f0 0x7ff5a40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5a94aa640 1 -- 172.21.15.60:0/1921604181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a415c9f0 msgr2=0x7ff5a40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5a94aa640 1 --2- 172.21.15.60:0/1921604181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a415c9f0 0x7ff5a40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5a94aa640 1 -- 172.21.15.60:0/1921604181 >> v1:172.21.15.60:6789/0 conn(0x7ff5a4070f20 legacy=0x7ff5a40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.734+0000 7ff5a94aa640 1 -- 172.21.15.60:0/1921604181 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a40cdbe0 con 0x7ff5a40c9230 2023-10-16T13:50:30.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.735+0000 7ff5a94aa640 1 --2- 172.21.15.60:0/1921604181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a40c9230 0x7ff5a40709e0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ff5940097a0 tx=0x7ff59402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf54dd11bbe252c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.736+0000 7ff5a8ca9640 1 -- 172.21.15.60:0/1921604181 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59402d040 con 0x7ff5a40c9230 2023-10-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.736+0000 7ff5a8ca9640 1 -- 172.21.15.60:0/1921604181 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff594031810 con 0x7ff5a40c9230 2023-10-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.736+0000 7ff5a8ca9640 1 -- 172.21.15.60:0/1921604181 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594031ba0 con 0x7ff5a40c9230 2023-10-16T13:50:30.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.736+0000 7ff5aacad640 1 -- 172.21.15.60:0/1921604181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a40c9230 msgr2=0x7ff5a40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.736+0000 7ff5aacad640 1 --2- 172.21.15.60:0/1921604181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a40c9230 0x7ff5a40709e0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ff5940097a0 tx=0x7ff59402b8b0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.737+0000 7ff5aacad640 1 -- 172.21.15.60:0/1921604181 shutdown_connections 2023-10-16T13:50:30.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.737+0000 7ff5aacad640 1 --2- 172.21.15.60:0/1921604181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a40c9230 0x7ff5a40709e0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.737+0000 7ff5aacad640 1 --2- 172.21.15.60:0/1921604181 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a415c9f0 0x7ff5a40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.737+0000 7ff5aacad640 1 -- 172.21.15.60:0/1921604181 >> 172.21.15.60:0/1921604181 conn(0x7ff5a40ce6a0 msgr2=0x7ff5a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.737+0000 7ff5aacad640 1 -- 172.21.15.60:0/1921604181 shutdown_connections 2023-10-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.737+0000 7ff5aacad640 1 -- 172.21.15.60:0/1921604181 wait complete. 2023-10-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.739+0000 7ff5aacad640 1 Processor -- start 2023-10-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.739+0000 7ff5aacad640 1 -- start start 2023-10-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.740+0000 7ff5aacad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a4070f20 0x7ff5a406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.740+0000 7ff5aacad640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5a40c9230 0x7ff5a415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.740+0000 7ff5aacad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 0x7ff5a415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.740+0000 7ff5aacad640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40c3220 con 0x7ff5a415c9f0 2023-10-16T13:50:30.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.740+0000 7ff5aacad640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40c3da0 con 0x7ff5a40c9230 2023-10-16T13:50:30.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.740+0000 7ff5aacad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40c3470 con 0x7ff5a4070f20 2023-10-16T13:50:30.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 0x7ff5a415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 0x7ff5a415abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49060/0 (socket says 172.21.15.60:49060) 2023-10-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 -- 172.21.15.60:0/3827236807 learned_addr learned my addr 172.21.15.60:0/3827236807 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5a94aa640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5a40c9230 0x7ff5a415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5a9cab640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a4070f20 0x7ff5a406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 -- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a4070f20 msgr2=0x7ff5a406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a4070f20 0x7ff5a406c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 -- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5a40c9230 msgr2=0x7ff5a415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:30.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5a40c9230 0x7ff5a415a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:30.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5aa4ac640 1 -- 172.21.15.60:0/3827236807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594009400 con 0x7ff5a415c9f0 2023-10-16T13:50:30.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.741+0000 7ff5a9cab640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a4070f20 0x7ff5a406c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:30.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.742+0000 7ff5aa4ac640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 0x7ff5a415abf0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00cbf0 tx=0x7ff59c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:30.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.742+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c00ad40 con 0x7ff5a415c9f0 2023-10-16T13:50:30.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.742+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c012400 con 0x7ff5a415c9f0 2023-10-16T13:50:30.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.742+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a40c3750 con 0x7ff5a415c9f0 2023-10-16T13:50:30.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.742+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c00aea0 con 0x7ff5a415c9f0 2023-10-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.742+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a406ca50 con 0x7ff5a415c9f0 2023-10-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.744+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a404e9b0 con 0x7ff5a415c9f0 2023-10-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.745+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 87) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff59c007070 con 0x7ff5a415c9f0 2023-10-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.746+0000 7ff59a7fc640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff58806af00 0x7ff58806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.746+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7ff59c089860 con 0x7ff5a415c9f0 2023-10-16T13:50:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.747+0000 7ff5a9cab640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff58806af00 0x7ff58806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:30.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.748+0000 7ff5a9cab640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff58806af00 0x7ff58806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0006fd0 tx=0x7ff5a0008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:30.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.751+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff59c054640 con 0x7ff5a415c9f0 2023-10-16T13:50:30.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.992+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5a40c3ee0 con 0x7ff5a415c9f0 2023-10-16T13:50:30.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.994+0000 7ff59a7fc640 1 -- 172.21.15.60:0/3827236807 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4918 (secure 0 0 0) 0x7ff59c057c60 con 0x7ff5a415c9f0 2023-10-16T13:50:30.997 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:30.997 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:54:13.310628+0000","modified":"2023-10-16T12:54:16.431585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8964":{"gid":8964,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3372982800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3372982800},{"type":"v1","addr":"172.21.15.60:6833","nonce":3372982800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:50:30.998 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 79 2023-10-16T13:50:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff58806af00 msgr2=0x7ff58806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff58806af00 0x7ff58806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0006fd0 tx=0x7ff5a0008040 comp rx=0 tx=0).stop 2023-10-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 msgr2=0x7ff5a415abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 0x7ff5a415abf0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00cbf0 tx=0x7ff59c00a430 comp rx=0 tx=0).stop 2023-10-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 shutdown_connections 2023-10-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff58806af00 0x7ff58806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5a415c9f0 0x7ff5a415abf0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5a40c9230 0x7ff5a415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 --2- 172.21.15.60:0/3827236807 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5a4070f20 0x7ff5a406c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 >> 172.21.15.60:0/3827236807 conn(0x7ff5a40ce6a0 msgr2=0x7ff5a4160f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 shutdown_connections 2023-10-16T13:50:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:30.998+0000 7ff5aacad640 1 -- 172.21.15.60:0/3827236807 wait complete. 2023-10-16T13:50:31.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:31.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.245+0000 7f4a854f8640 1 Processor -- start 2023-10-16T13:50:31.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.245+0000 7f4a854f8640 1 -- start start 2023-10-16T13:50:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.246+0000 7f4a854f8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a80070950 0x7f4a8006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a854f8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a800ca3a0 con 0x7f4a801573e0 2023-10-16T13:50:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a854f8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a800c9d70 con 0x7f4a800ca830 2023-10-16T13:50:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a854f8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a801577c0 con 0x7f4a80070950 2023-10-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a84cf7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a80070950 0x7f4a8006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a84cf7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a80070950 0x7f4a8006bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49076/0 (socket says 172.21.15.60:49076) 2023-10-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a7ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4a801573e0 0x7f4a80159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41292/0 (socket says 172.21.15.60:41292) 2023-10-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a84cf7640 1 -- 172.21.15.60:0/2444381894 learned_addr learned my addr 172.21.15.60:0/2444381894 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:31.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a84cf7640 1 -- 172.21.15.60:0/2444381894 >> v1:172.21.15.144:6789/0 conn(0x7f4a800ca830 legacy=0x7f4a80070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a84cf7640 1 -- 172.21.15.60:0/2444381894 >> v1:172.21.15.60:6789/0 conn(0x7f4a801573e0 legacy=0x7f4a80159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.247+0000 7f4a84cf7640 1 -- 172.21.15.60:0/2444381894 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8015a9a0 con 0x7f4a80070950 2023-10-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.249+0000 7f4a84cf7640 1 --2- 172.21.15.60:0/2444381894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8006bdc0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f4a780098a0 tx=0x7f4a7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458cfcfcc037a46d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.249+0000 7f4a7effd640 1 -- 172.21.15.60:0/2444381894 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78030840 con 0x7f4a80070950 2023-10-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.249+0000 7f4a7effd640 1 -- 172.21.15.60:0/2444381894 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a780309a0 con 0x7f4a80070950 2023-10-16T13:50:31.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.249+0000 7f4a7effd640 1 -- 172.21.15.60:0/2444381894 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78030d30 con 0x7f4a80070950 2023-10-16T13:50:31.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.249+0000 7f4a854f8640 1 -- 172.21.15.60:0/2444381894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 msgr2=0x7f4a8006bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.249+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2444381894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8006bdc0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f4a780098a0 tx=0x7f4a7802b980 comp rx=0 tx=0).stop 2023-10-16T13:50:31.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.250+0000 7f4a854f8640 1 -- 172.21.15.60:0/2444381894 shutdown_connections 2023-10-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.250+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2444381894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8006bdc0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.250+0000 7f4a854f8640 1 -- 172.21.15.60:0/2444381894 >> 172.21.15.60:0/2444381894 conn(0x7f4a800cc230 msgr2=0x7f4a800cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.250+0000 7f4a854f8640 1 -- 172.21.15.60:0/2444381894 shutdown_connections 2023-10-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.250+0000 7f4a854f8640 1 -- 172.21.15.60:0/2444381894 wait complete. 2023-10-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.251+0000 7f4a854f8640 1 Processor -- start 2023-10-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.252+0000 7f4a854f8640 1 -- start start 2023-10-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a854f8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a854f8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a800ca830 0x7f4a800ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a854f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a801573e0 0x7f4a8015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a854f8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a801559d0 con 0x7f4a80070950 2023-10-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a854f8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80155cb0 con 0x7f4a801573e0 2023-10-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a854f8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80155f90 con 0x7f4a800ca830 2023-10-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8014ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49082/0 (socket says 172.21.15.60:49082) 2023-10-16T13:50:31.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 -- 172.21.15.60:0/2683271649 learned_addr learned my addr 172.21.15.60:0/2683271649 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:31.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 -- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a800ca830 msgr2=0x7f4a800ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a800ca830 0x7f4a800ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 -- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a801573e0 msgr2=0x7f4a8015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a801573e0 0x7f4a8015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.253+0000 7f4a7ffff640 1 -- 172.21.15.60:0/2683271649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780094d0 con 0x7f4a80070950 2023-10-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.254+0000 7f4a7ffff640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8014ef60 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000c800 tx=0x7f4a7000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.254+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000a640 con 0x7f4a80070950 2023-10-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.254+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7000a7a0 con 0x7f4a80070950 2023-10-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.254+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000ab50 con 0x7f4a80070950 2023-10-16T13:50:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.254+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a80156270 con 0x7f4a80070950 2023-10-16T13:50:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.254+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8014f4a0 con 0x7f4a80070950 2023-10-16T13:50:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.255+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a800cacc0 con 0x7f4a80070950 2023-10-16T13:50:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.256+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 87) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4a7000acb0 con 0x7f4a80070950 2023-10-16T13:50:31.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.257+0000 7f4a7cff9640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4a6c06af00 0x7f4a6c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.257+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f4a70089110 con 0x7f4a80070950 2023-10-16T13:50:31.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.258+0000 7f4a7f7fe640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4a6c06af00 0x7f4a6c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.259+0000 7f4a7f7fe640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4a6c06af00 0x7f4a6c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a74006fd0 tx=0x7f4a74006bc0 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:31.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.262+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a800cacc0 con 0x7f4a80070950 2023-10-16T13:50:31.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.466+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a80156790 con 0x7f4a80070950 2023-10-16T13:50:31.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.468+0000 7f4a7cff9640 1 -- 172.21.15.60:0/2683271649 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+17775 (secure 0 0 0) 0x7f4a80156790 con 0x7f4a80070950 2023-10-16T13:50:31.472 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:31.473 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":171,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:31.286885+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:54:11.305180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-16T12:54:12.361464+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.30","mappings":[{"from":2,"to":0}]},{"pgid":"19.32","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1674745327":"2023-10-17T13:50:27.750339+0000","172.21.15.144:6832/110255757":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/1615247958":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/1009875684":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/931153185":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/3784462343":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/3666412233":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/362473654":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1686539411":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/4019169897":"2023-10-17T13:50:27.750339+0000","172.21.15.144:6832/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/110255757":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/4230715707":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1256386612":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1071036703":"2023-10-17T12:54:28.068770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":6,"length":52},{"begin":62,"length":52}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4a6c06af00 msgr2=0x7f4a6c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4a6c06af00 0x7f4a6c06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a74006fd0 tx=0x7f4a74006bc0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 msgr2=0x7f4a8014ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8014ef60 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000c800 tx=0x7f4a7000ccc0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 shutdown_connections 2023-10-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4a6c06af00 0x7f4a6c06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.471+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4a800ca830 0x7f4a800ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.472+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a801573e0 0x7f4a8015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.472+0000 7f4a854f8640 1 --2- 172.21.15.60:0/2683271649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a80070950 0x7f4a8014ef60 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.472+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 >> 172.21.15.60:0/2683271649 conn(0x7f4a800cc230 msgr2=0x7f4a8015afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.472+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 shutdown_connections 2023-10-16T13:50:31.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.472+0000 7f4a854f8640 1 -- 172.21.15.60:0/2683271649 wait complete. 2023-10-16T13:50:31.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:50:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.715+0000 7f9504761640 1 Processor -- start 2023-10-16T13:50:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.716+0000 7f9504761640 1 -- start start 2023-10-16T13:50:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.717+0000 7f9504761640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94fc06c850 0x7f94fc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.717+0000 7f9504761640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94fc070360 con 0x7f94fc159e90 2023-10-16T13:50:31.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.717+0000 7f9504761640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94fc0cd1f0 con 0x7f94fc159ab0 2023-10-16T13:50:31.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.717+0000 7f9504761640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94fc15d890 con 0x7f94fc06c850 2023-10-16T13:50:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f9501cfc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f94fc159e90 0x7f94fc06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41306/0 (socket says 172.21.15.60:41306) 2023-10-16T13:50:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f9501cfc640 1 -- 172.21.15.60:0/2758582056 learned_addr learned my addr 172.21.15.60:0/2758582056 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f9502cfe640 1 --2- 172.21.15.60:0/2758582056 >> v2:172.21.15.60:3300/0 conn(0x7f94fc06c850 0x7f94fc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f9502cfe640 1 -- 172.21.15.60:0/2758582056 >> v1:172.21.15.144:6789/0 conn(0x7f94fc159ab0 legacy=0x7f94fc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f9502cfe640 1 -- 172.21.15.60:0/2758582056 >> v1:172.21.15.60:6789/0 conn(0x7f94fc159e90 legacy=0x7f94fc06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f95014fb640 1 -- 172.21.15.60:0/2758582056 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259940306 0 0) 0x7f94fc070360 con 0x7f94fc159e90 2023-10-16T13:50:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.718+0000 7f9502cfe640 1 -- 172.21.15.60:0/2758582056 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc0cd4d0 con 0x7f94fc06c850 2023-10-16T13:50:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.719+0000 7f9502cfe640 1 --2- 172.21.15.60:0/2758582056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0ce370 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009d90 tx=0x7f94f402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cfba803c7f1360 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f95014fb640 1 -- 172.21.15.60:0/2758582056 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f402cb40 con 0x7f94fc06c850 2023-10-16T13:50:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f95014fb640 1 -- 172.21.15.60:0/2758582056 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f402cca0 con 0x7f94fc06c850 2023-10-16T13:50:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f95014fb640 1 -- 172.21.15.60:0/2758582056 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4034660 con 0x7f94fc06c850 2023-10-16T13:50:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f9504761640 1 -- 172.21.15.60:0/2758582056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 msgr2=0x7f94fc0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f9504761640 1 --2- 172.21.15.60:0/2758582056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0ce370 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009d90 tx=0x7f94f402bd90 comp rx=0 tx=0).stop 2023-10-16T13:50:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f9504761640 1 -- 172.21.15.60:0/2758582056 shutdown_connections 2023-10-16T13:50:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f9504761640 1 --2- 172.21.15.60:0/2758582056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0ce370 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.720+0000 7f9504761640 1 -- 172.21.15.60:0/2758582056 >> 172.21.15.60:0/2758582056 conn(0x7f94fc0ce9b0 msgr2=0x7f94fc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.721+0000 7f9504761640 1 -- 172.21.15.60:0/2758582056 shutdown_connections 2023-10-16T13:50:31.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.721+0000 7f9504761640 1 -- 172.21.15.60:0/2758582056 wait complete. 2023-10-16T13:50:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.722+0000 7f9504761640 1 Processor -- start 2023-10-16T13:50:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.723+0000 7f9504761640 1 -- start start 2023-10-16T13:50:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9504761640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9504761640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94fc159ab0 0x7f94fc0c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9504761640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94fc159e90 0x7f94fc0c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9504761640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc13e720 con 0x7f94fc06c850 2023-10-16T13:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9504761640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc140f60 con 0x7f94fc159e90 2023-10-16T13:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9504761640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc140c80 con 0x7f94fc159ab0 2023-10-16T13:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0aeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49094/0 (socket says 172.21.15.60:49094) 2023-10-16T13:50:31.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 -- 172.21.15.60:0/4205849363 learned_addr learned my addr 172.21.15.60:0/4205849363 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:31.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 -- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94fc159ab0 msgr2=0x7f94fc0c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9501cfc640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94fc159ab0 0x7f94fc0c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94fc159ab0 0x7f94fc0c0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f9502cfe640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94fc159e90 0x7f94fc0c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 -- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94fc159e90 msgr2=0x7f94fc0c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.724+0000 7f95024fd640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94fc159e90 0x7f94fc0c14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f95024fd640 1 -- 172.21.15.60:0/4205849363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4009a70 con 0x7f94fc06c850 2023-10-16T13:50:31.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f95024fd640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0aeea0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c410 tx=0x7f94f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:31.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f80035f0 con 0x7f94fc06c850 2023-10-16T13:50:31.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f8003750 con 0x7f94fc06c850 2023-10-16T13:50:31.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8003b40 con 0x7f94fc06c850 2023-10-16T13:50:31.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc13e0d0 con 0x7f94fc06c850 2023-10-16T13:50:31.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.725+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc0af3e0 con 0x7f94fc06c850 2023-10-16T13:50:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.726+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc066de0 con 0x7f94fc06c850 2023-10-16T13:50:31.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.728+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 87) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f94f8003ca0 con 0x7f94fc06c850 2023-10-16T13:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.729+0000 7f94f2ffd640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f94e006af30 0x7f94e006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.729+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f94f8088bb0 con 0x7f94fc06c850 2023-10-16T13:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.729+0000 7f9501cfc640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f94e006af30 0x7f94e006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:31.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.731+0000 7f9501cfc640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f94e006af30 0x7f94e006d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f94fc1f3a60 tx=0x7f94ec009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.733+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94f80538e0 con 0x7f94fc06c850 2023-10-16T13:50:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.971+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f94fc0cda60 con 0x7f94fc06c850 2023-10-16T13:50:31.975 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T13:50:31.972+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:50:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.976+0000 7f94f2ffd640 1 -- 172.21.15.60:0/4205849363 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v80) v1 ==== 161+0+0 (secure 0 0 0) 0x7f94f8056f00 con 0x7f94fc06c850 2023-10-16T13:50:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:50:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f94e006af30 msgr2=0x7f94e006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f94e006af30 0x7f94e006d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f94fc1f3a60 tx=0x7f94ec009040 comp rx=0 tx=0).stop 2023-10-16T13:50:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 msgr2=0x7f94fc0aeea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0aeea0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c410 tx=0x7f94f800c8d0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 shutdown_connections 2023-10-16T13:50:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f94e006af30 0x7f94e006d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94fc159ab0 0x7f94fc0c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94fc159e90 0x7f94fc0c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 --2- 172.21.15.60:0/4205849363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94fc06c850 0x7f94fc0aeea0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.979+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 >> 172.21.15.60:0/4205849363 conn(0x7f94fc0ce9b0 msgr2=0x7f94fc0c1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.980+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 shutdown_connections 2023-10-16T13:50:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:31.980+0000 7f9504761640 1 -- 172.21.15.60:0/4205849363 wait complete. 2023-10-16T13:50:32.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:50:32.106 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T13:50:32.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.197+0000 7f9166328640 1 Processor -- start 2023-10-16T13:50:32.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.197+0000 7f9166328640 1 -- start start 2023-10-16T13:50:32.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.198+0000 7f9166328640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 0x7f916006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.198+0000 7f9166328640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916015f4a0 con 0x7f91600ad970 2023-10-16T13:50:32.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.198+0000 7f9166328640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916006c960 con 0x7f91600cda80 2023-10-16T13:50:32.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.198+0000 7f9166328640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916015b9d0 con 0x7f916006c040 2023-10-16T13:50:32.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9164b25640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f91600ad970 0x7f916015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41334/0 (socket says 172.21.15.60:41334) 2023-10-16T13:50:32.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9164b25640 1 -- 172.21.15.60:0/488576131 learned_addr learned my addr 172.21.15.60:0/488576131 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:32.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9165326640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 0x7f916006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116562877 0 0) 0x7f916015f4a0 con 0x7f91600ad970 2023-10-16T13:50:32.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f914c003520 con 0x7f91600ad970 2023-10-16T13:50:32.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 131098924 0 0) 0x7f914c003520 con 0x7f91600ad970 2023-10-16T13:50:32.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 msgr2=0x7f916006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:32.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.199+0000 7f9157fff640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 0x7f916006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 >> v1:172.21.15.144:6789/0 conn(0x7f91600cda80 legacy=0x7f916015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916015bcb0 con 0x7f91600ad970 2023-10-16T13:50:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9150001670 con 0x7f91600ad970 2023-10-16T13:50:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 >> v1:172.21.15.60:6789/0 conn(0x7f91600ad970 legacy=0x7f916015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f914c0039a0 0x7f914c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 0x7f914c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f914c004d00 0x7f914c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916015b9d0 con 0x7f914c004d00 2023-10-16T13:50:32.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c00bb10 con 0x7f914c0039a0 2023-10-16T13:50:32.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c00bca0 con 0x7f916006c040 2023-10-16T13:50:32.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165b27640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f914c004d00 0x7f914c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9164b25640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 0x7f914c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165326640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f914c0039a0 0x7f914c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165b27640 1 -- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 msgr2=0x7f914c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:32.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165b27640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f916006c040 0x7f914c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165b27640 1 -- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f914c0039a0 msgr2=0x7f914c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:32.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165b27640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f914c0039a0 0x7f914c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.200+0000 7f9165b27640 1 -- 172.21.15.60:0/488576131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c003520 con 0x7f914c004d00 2023-10-16T13:50:32.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.201+0000 7f9165b27640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f914c004d00 0x7f914c005180 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f9158001e80 tx=0x7f914c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2134fb4bb9d5e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:32.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158003b70 con 0x7f914c004d00 2023-10-16T13:50:32.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158003cd0 con 0x7f914c004d00 2023-10-16T13:50:32.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9157fff640 1 -- 172.21.15.60:0/488576131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158005310 con 0x7f914c004d00 2023-10-16T13:50:32.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9166328640 1 -- 172.21.15.60:0/488576131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f914c004d00 msgr2=0x7f914c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9166328640 1 --2- 172.21.15.60:0/488576131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f914c004d00 0x7f914c005180 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f9158001e80 tx=0x7f914c0057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9165b27640 1 -- 172.21.15.60:0/488576131 reap_dead start 2023-10-16T13:50:32.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9166328640 1 -- 172.21.15.60:0/488576131 shutdown_connections 2023-10-16T13:50:32.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9166328640 1 -- 172.21.15.60:0/488576131 >> 172.21.15.60:0/488576131 conn(0x7f91600ce6a0 msgr2=0x7f91600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:32.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9166328640 1 -- 172.21.15.60:0/488576131 shutdown_connections 2023-10-16T13:50:32.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.202+0000 7f9166328640 1 -- 172.21.15.60:0/488576131 wait complete. 2023-10-16T13:50:32.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.204+0000 7f9166328640 1 Processor -- start 2023-10-16T13:50:32.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.204+0000 7f9166328640 1 -- start start 2023-10-16T13:50:32.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9166328640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f914c004d00 0x7f91600c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9166328640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91600c67c0 0x7f91600b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9166328640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 0x7f91600ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9166328640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9160142f00 con 0x7f91601f56e0 2023-10-16T13:50:32.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9166328640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91600c6cc0 con 0x7f91600c67c0 2023-10-16T13:50:32.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9166328640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91600c6fa0 con 0x7f914c004d00 2023-10-16T13:50:32.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.205+0000 7f9165b27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 0x7f91600ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 0x7f91600ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49128/0 (socket says 172.21.15.60:49128) 2023-10-16T13:50:32.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 -- 172.21.15.60:0/2320592022 learned_addr learned my addr 172.21.15.60:0/2320592022 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:32.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9164b25640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91600c67c0 0x7f91600b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 -- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f914c004d00 msgr2=0x7f91600c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f914c004d00 0x7f91600c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 -- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91600c67c0 msgr2=0x7f91600b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91600c67c0 0x7f91600b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 -- 172.21.15.60:0/2320592022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91600c7280 con 0x7f91601f56e0 2023-10-16T13:50:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9164b25640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91600c67c0 0x7f91600b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.206+0000 7f9165b27640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 0x7f91600ae100 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f9158005040 tx=0x7f91580014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.207+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915803e070 con 0x7f91601f56e0 2023-10-16T13:50:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.207+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158003e40 con 0x7f91601f56e0 2023-10-16T13:50:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.207+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158006d50 con 0x7f91601f56e0 2023-10-16T13:50:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.207+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91600c7560 con 0x7f91601f56e0 2023-10-16T13:50:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.207+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9160155a00 con 0x7f91601f56e0 2023-10-16T13:50:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.208+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916006b330 con 0x7f91601f56e0 2023-10-16T13:50:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.209+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 87) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9158007030 con 0x7f91601f56e0 2023-10-16T13:50:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.210+0000 7f9155ffb640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f914406af00 0x7f914406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.210+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6541+0+0 (secure 0 0 0) 0x7f91580aadf0 con 0x7f91601f56e0 2023-10-16T13:50:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.210+0000 7f9165326640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f914406af00 0x7f914406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.212+0000 7f9165326640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f914406af00 0x7f914406d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f91600b1830 tx=0x7f915c00b2e0 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.214+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91580759f0 con 0x7f91601f56e0 2023-10-16T13:50:32.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.313+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9158075dc0 con 0x7f91601f56e0 2023-10-16T13:50:32.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.458+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9160142f00 con 0x7f91601f56e0 2023-10-16T13:50:32.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.979+0000 7f9155ffb640 1 -- 172.21.15.60:0/2320592022 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v81) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9158038540 con 0x7f91601f56e0 2023-10-16T13:50:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f914406af00 msgr2=0x7f914406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f914406af00 0x7f914406d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f91600b1830 tx=0x7f915c00b2e0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 msgr2=0x7f91600ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 0x7f91600ae100 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f9158005040 tx=0x7f91580014a0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 shutdown_connections 2023-10-16T13:50:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f914406af00 0x7f914406d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91601f56e0 0x7f91600ae100 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91600c67c0 0x7f91600b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 --2- 172.21.15.60:0/2320592022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f914c004d00 0x7f91600c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 >> 172.21.15.60:0/2320592022 conn(0x7f91600ce6a0 msgr2=0x7f91600702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 shutdown_connections 2023-10-16T13:50:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:32.983+0000 7f9166328640 1 -- 172.21.15.60:0/2320592022 wait complete. 2023-10-16T13:50:33.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:50:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.230+0000 7fd4e277c640 1 Processor -- start 2023-10-16T13:50:33.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.231+0000 7fd4e277c640 1 -- start start 2023-10-16T13:50:33.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.232+0000 7fd4e277c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd4dc15cf50 0x7fd4dc0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.232+0000 7fd4e277c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4dc0b8a10 0x7fd4dc0d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.232+0000 7fd4e277c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc15d330 con 0x7fd4dc0b4750 2023-10-16T13:50:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.232+0000 7fd4e277c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0d1680 con 0x7fd4dc0b8a10 2023-10-16T13:50:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.232+0000 7fd4e277c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0d1960 con 0x7fd4dc15cf50 2023-10-16T13:50:33.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.232+0000 7fd4db7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4dc0b8a10 0x7fd4dc0d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:33.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4e0d19640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd4dc0b4750 0x7fd4dc0b4b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41344/0 (socket says 172.21.15.60:41344) 2023-10-16T13:50:33.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4e0d19640 1 -- 172.21.15.60:0/3575430962 learned_addr learned my addr 172.21.15.60:0/3575430962 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4dbfff640 1 --2- 172.21.15.60:0/3575430962 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc15cf50 0x7fd4dc0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4daffd640 1 -- 172.21.15.60:0/3575430962 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908260759 0 0) 0x7fd4dc15d330 con 0x7fd4dc0b4750 2023-10-16T13:50:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4daffd640 1 -- 172.21.15.60:0/3575430962 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c4003520 con 0x7fd4dc0b4750 2023-10-16T13:50:33.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4db7fe640 1 -- 172.21.15.60:0/3575430962 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc15cf50 msgr2=0x7fd4dc0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4db7fe640 1 --2- 172.21.15.60:0/3575430962 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc15cf50 0x7fd4dc0d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4db7fe640 1 -- 172.21.15.60:0/3575430962 >> v1:172.21.15.60:6789/0 conn(0x7fd4dc0b4750 legacy=0x7fd4dc0b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.233+0000 7fd4db7fe640 1 -- 172.21.15.60:0/3575430962 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0d1c40 con 0x7fd4dc0b8a10 2023-10-16T13:50:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.235+0000 7fd4db7fe640 1 --2- 172.21.15.60:0/3575430962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc0b8a10 0x7fd4dc0d0d10 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0cd520 tx=0x7fd4d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325b7888dd1fa98e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:33.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.235+0000 7fd4daffd640 1 -- 172.21.15.60:0/3575430962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0031830 con 0x7fd4dc0b8a10 2023-10-16T13:50:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.235+0000 7fd4daffd640 1 -- 172.21.15.60:0/3575430962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d0031990 con 0x7fd4dc0b8a10 2023-10-16T13:50:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.235+0000 7fd4daffd640 1 -- 172.21.15.60:0/3575430962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0031d20 con 0x7fd4dc0b8a10 2023-10-16T13:50:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.235+0000 7fd4e277c640 1 -- 172.21.15.60:0/3575430962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc0b8a10 msgr2=0x7fd4dc0d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 --2- 172.21.15.60:0/3575430962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc0b8a10 0x7fd4dc0d0d10 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0cd520 tx=0x7fd4d002b8d0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 -- 172.21.15.60:0/3575430962 shutdown_connections 2023-10-16T13:50:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 --2- 172.21.15.60:0/3575430962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc0b8a10 0x7fd4dc0d0d10 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 --2- 172.21.15.60:0/3575430962 >> v2:172.21.15.144:3300/0 conn(0x7fd4dc15cf50 0x7fd4dc0d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 -- 172.21.15.60:0/3575430962 >> 172.21.15.60:0/3575430962 conn(0x7fd4dc0cf750 msgr2=0x7fd4dc0cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 -- 172.21.15.60:0/3575430962 shutdown_connections 2023-10-16T13:50:33.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.236+0000 7fd4e277c640 1 -- 172.21.15.60:0/3575430962 wait complete. 2023-10-16T13:50:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.238+0000 7fd4e277c640 1 Processor -- start 2023-10-16T13:50:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.238+0000 7fd4e277c640 1 -- start start 2023-10-16T13:50:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e277c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc0b4750 0x7fd4dc154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e277c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc0b8a10 0x7fd4dc15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e277c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 0x7fd4dc157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e277c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc1603b0 con 0x7fd4dc15cf50 2023-10-16T13:50:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e277c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc15f5f0 con 0x7fd4dc0b4750 2023-10-16T13:50:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e277c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc15f870 con 0x7fd4dc0b8a10 2023-10-16T13:50:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e0d19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 0x7fd4dc157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e0d19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 0x7fd4dc157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49148/0 (socket says 172.21.15.60:49148) 2023-10-16T13:50:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4e0d19640 1 -- 172.21.15.60:0/2558009571 learned_addr learned my addr 172.21.15.60:0/2558009571 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4dbfff640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc0b4750 0x7fd4dc154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.239+0000 7fd4db7fe640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc0b8a10 0x7fd4dc15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e0d19640 1 -- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc0b8a10 msgr2=0x7fd4dc15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e0d19640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc0b8a10 0x7fd4dc15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e0d19640 1 -- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc0b4750 msgr2=0x7fd4dc154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e0d19640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc0b4750 0x7fd4dc154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e0d19640 1 -- 172.21.15.60:0/2558009571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0009400 con 0x7fd4dc15cf50 2023-10-16T13:50:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e0d19640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 0x7fd4dc157190 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400cb40 tx=0x7fd4d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400aca0 con 0x7fd4dc15cf50 2023-10-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d400ae00 con 0x7fd4dc15cf50 2023-10-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc15fb10 con 0x7fd4dc15cf50 2023-10-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4012620 con 0x7fd4dc15cf50 2023-10-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.240+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc1600d0 con 0x7fd4dc15cf50 2023-10-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.241+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc1f5580 con 0x7fd4dc15cf50 2023-10-16T13:50:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.243+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd4d4007070 con 0x7fd4dc15cf50 2023-10-16T13:50:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.244+0000 7fd4d8ff9640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd4c806af90 0x7fd4c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:33.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.244+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fd4d4089b30 con 0x7fd4dc15cf50 2023-10-16T13:50:33.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.244+0000 7fd4dbfff640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd4c806af90 0x7fd4c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:33.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.245+0000 7fd4dbfff640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd4c806af90 0x7fd4c806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0b8760 tx=0x7fd4cc008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:33.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.248+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4dc1f5580 con 0x7fd4dc15cf50 2023-10-16T13:50:33.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.446+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd4dc1f5580 con 0x7fd4dc15cf50 2023-10-16T13:50:33.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.983+0000 7fd4d8ff9640 1 -- 172.21.15.60:0/2558009571 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v173) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd4dc1f5580 con 0x7fd4dc15cf50 2023-10-16T13:50:33.987 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:50:33.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd4c806af90 msgr2=0x7fd4c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:33.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd4c806af90 0x7fd4c806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0b8760 tx=0x7fd4cc008040 comp rx=0 tx=0).stop 2023-10-16T13:50:33.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 msgr2=0x7fd4dc157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:33.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 0x7fd4dc157190 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400cb40 tx=0x7fd4d400a060 comp rx=0 tx=0).stop 2023-10-16T13:50:33.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 shutdown_connections 2023-10-16T13:50:33.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd4c806af90 0x7fd4c806d450 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0b8760 tx=0x7fd4cc008040 comp rx=0 tx=0).stop 2023-10-16T13:50:33.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4dc0b8a10 0x7fd4dc15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4dc15cf50 0x7fd4dc157190 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 --2- 172.21.15.60:0/2558009571 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4dc0b4750 0x7fd4dc154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:33.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.989+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 >> 172.21.15.60:0/2558009571 conn(0x7fd4dc0cf750 msgr2=0x7fd4dc0cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:33.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.991+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 shutdown_connections 2023-10-16T13:50:33.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:33.991+0000 7fd4e277c640 1 -- 172.21.15.60:0/2558009571 wait complete. 2023-10-16T13:50:34.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:50:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.241+0000 7f521fccf640 1 Processor -- start 2023-10-16T13:50:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.241+0000 7f521fccf640 1 -- start start 2023-10-16T13:50:34.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521fccf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f521815ac10 0x7f521806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:34.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521fccf640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f52180b0360 0x7f52180b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:34.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521fccf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521815b310 con 0x7f52180ca510 2023-10-16T13:50:34.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521fccf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521815f800 con 0x7f521815ac10 2023-10-16T13:50:34.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521fccf640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52180b0d30 con 0x7f52180b0360 2023-10-16T13:50:34.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521da6b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f521815ac10 0x7f521806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:34.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521e26c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f52180ca510 0x7f52180ca920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41354/0 (socket says 172.21.15.60:41354) 2023-10-16T13:50:34.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521e26c640 1 -- 172.21.15.60:0/3766651055 learned_addr learned my addr 172.21.15.60:0/3766651055 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:34.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521da6b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f521815ac10 0x7f521806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49158/0 (socket says 172.21.15.60:49158) 2023-10-16T13:50:34.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.243+0000 7f521d26a640 1 --2- 172.21.15.60:0/3766651055 >> v2:172.21.15.144:3300/0 conn(0x7f52180b0360 0x7f52180b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:34.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.244+0000 7f521da6b640 1 -- 172.21.15.60:0/3766651055 >> v2:172.21.15.144:3300/0 conn(0x7f52180b0360 msgr2=0x7f52180b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:34.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.244+0000 7f521da6b640 1 --2- 172.21.15.60:0/3766651055 >> v2:172.21.15.144:3300/0 conn(0x7f52180b0360 0x7f52180b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:34.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.244+0000 7f521da6b640 1 -- 172.21.15.60:0/3766651055 >> v1:172.21.15.60:6789/0 conn(0x7f52180ca510 legacy=0x7f52180ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:34.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.244+0000 7f521da6b640 1 -- 172.21.15.60:0/3766651055 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521806f640 con 0x7f521815ac10 2023-10-16T13:50:34.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.244+0000 7f521d26a640 1 --2- 172.21.15.60:0/3766651055 >> v2:172.21.15.144:3300/0 conn(0x7f52180b0360 0x7f52180b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:34.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.245+0000 7f521da6b640 1 --2- 172.21.15.60:0/3766651055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521815ac10 0x7f521806c730 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f52180ae2b0 tx=0x7f521402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a03078bb1dfddd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.245+0000 7f521ca69640 1 -- 172.21.15.60:0/3766651055 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521402e070 con 0x7f521815ac10 2023-10-16T13:50:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.245+0000 7f521ca69640 1 -- 172.21.15.60:0/3766651055 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52140315a0 con 0x7f521815ac10 2023-10-16T13:50:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.245+0000 7f521ca69640 1 -- 172.21.15.60:0/3766651055 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214031930 con 0x7f521815ac10 2023-10-16T13:50:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 -- 172.21.15.60:0/3766651055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521815ac10 msgr2=0x7f521806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 --2- 172.21.15.60:0/3766651055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521815ac10 0x7f521806c730 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f52180ae2b0 tx=0x7f521402b840 comp rx=0 tx=0).stop 2023-10-16T13:50:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 -- 172.21.15.60:0/3766651055 shutdown_connections 2023-10-16T13:50:34.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 --2- 172.21.15.60:0/3766651055 >> v2:172.21.15.144:3300/0 conn(0x7f52180b0360 0x7f52180b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:34.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 --2- 172.21.15.60:0/3766651055 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521815ac10 0x7f521806c730 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:34.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 -- 172.21.15.60:0/3766651055 >> 172.21.15.60:0/3766651055 conn(0x7f52180cb360 msgr2=0x7f52180cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:34.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 -- 172.21.15.60:0/3766651055 shutdown_connections 2023-10-16T13:50:34.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.246+0000 7f521fccf640 1 -- 172.21.15.60:0/3766651055 wait complete. 2023-10-16T13:50:34.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.248+0000 7f521fccf640 1 Processor -- start 2023-10-16T13:50:34.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.248+0000 7f521fccf640 1 -- start start 2023-10-16T13:50:34.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521fccf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52180b0360 0x7f5218150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:34.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521fccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 0x7f52181f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:34.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521fccf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521815ac10 0x7f52181f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:34.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521fccf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52181f45c0 con 0x7f52180ca510 2023-10-16T13:50:34.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521fccf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52181f48a0 con 0x7f52180b0360 2023-10-16T13:50:34.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521fccf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180b24f0 con 0x7f521815ac10 2023-10-16T13:50:34.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521d26a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 0x7f52181f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:34.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521d26a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 0x7f52181f3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49174/0 (socket says 172.21.15.60:49174) 2023-10-16T13:50:34.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.249+0000 7f521d26a640 1 -- 172.21.15.60:0/3598016497 learned_addr learned my addr 172.21.15.60:0/3598016497 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:34.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521da6b640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52180b0360 0x7f5218150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:34.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521e26c640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521815ac10 0x7f52181f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:34.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521d26a640 1 -- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521815ac10 msgr2=0x7f52181f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:34.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521d26a640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521815ac10 0x7f52181f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:34.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521d26a640 1 -- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52180b0360 msgr2=0x7f5218150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521d26a640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52180b0360 0x7f5218150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521d26a640 1 -- 172.21.15.60:0/3598016497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214009400 con 0x7f52180ca510 2023-10-16T13:50:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521da6b640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52180b0360 0x7f5218150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.250+0000 7f521d26a640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 0x7f52181f3a50 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f520800c7f0 tx=0x7f520800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:34.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.251+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208007820 con 0x7f52180ca510 2023-10-16T13:50:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.251+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208007980 con 0x7f52180ca510 2023-10-16T13:50:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.251+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52180b27d0 con 0x7f52180ca510 2023-10-16T13:50:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.251+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208007cf0 con 0x7f52180ca510 2023-10-16T13:50:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.251+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5218157680 con 0x7f52180ca510 2023-10-16T13:50:34.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.252+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521804e160 con 0x7f52180ca510 2023-10-16T13:50:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.253+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52080113f0 con 0x7f52180ca510 2023-10-16T13:50:34.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.254+0000 7f520e7fc640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f51fc06af90 0x7f51fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.254+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6103+0+0 (secure 0 0 0) 0x7f5208089d50 con 0x7f52180ca510 2023-10-16T13:50:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.255+0000 7f521da6b640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f51fc06af90 0x7f51fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.256+0000 7f521da6b640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f51fc06af90 0x7f51fc06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403d040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:34.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.258+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5208054a40 con 0x7f52180ca510 2023-10-16T13:50:34.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:34.454+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f521806c2e0 con 0x7f52180ca510 2023-10-16T13:50:35.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.002+0000 7f520e7fc640 1 -- 172.21.15.60:0/3598016497 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v174) v1 ==== 169+0+0 (secure 0 0 0) 0x7f520801f0d0 con 0x7f52180ca510 2023-10-16T13:50:35.005 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T13:50:35.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f51fc06af90 msgr2=0x7f51fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f51fc06af90 0x7f51fc06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403d040 comp rx=0 tx=0).stop 2023-10-16T13:50:35.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 msgr2=0x7f52181f3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 0x7f52181f3a50 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f520800c7f0 tx=0x7f520800ccb0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 shutdown_connections 2023-10-16T13:50:35.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f51fc06af90 0x7f51fc06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.005+0000 7f521fccf640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f521815ac10 0x7f52181f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.006+0000 7f521fccf640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52180ca510 0x7f52181f3a50 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.006+0000 7f521fccf640 1 --2- 172.21.15.60:0/3598016497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52180b0360 0x7f5218150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.006+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 >> 172.21.15.60:0/3598016497 conn(0x7f52180cb360 msgr2=0x7f521806c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:35.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.006+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 shutdown_connections 2023-10-16T13:50:35.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.006+0000 7f521fccf640 1 -- 172.21.15.60:0/3598016497 wait complete. 2023-10-16T13:50:35.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:50:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.238+0000 7f70899ce640 1 Processor -- start 2023-10-16T13:50:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.238+0000 7f70899ce640 1 -- start start 2023-10-16T13:50:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70899ce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7084159f00 0x7f70840b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70899ce640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708406f890 con 0x7f7084159b20 2023-10-16T13:50:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70899ce640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708406c650 con 0x7f708415d8c0 2023-10-16T13:50:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70899ce640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708406c930 con 0x7f7084159f00 2023-10-16T13:50:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70827fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7084159f00 0x7f70840b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70827fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7084159f00 0x7f70840b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49176/0 (socket says 172.21.15.60:49176) 2023-10-16T13:50:35.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70827fc640 1 -- 172.21.15.60:0/2886530332 learned_addr learned my addr 172.21.15.60:0/2886530332 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.240+0000 7f70827fc640 1 -- 172.21.15.60:0/2886530332 >> v1:172.21.15.144:6789/0 conn(0x7f708415d8c0 legacy=0x7f708406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.241+0000 7f70827fc640 1 -- 172.21.15.60:0/2886530332 >> v1:172.21.15.60:6789/0 conn(0x7f7084159b20 legacy=0x7f708406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.241+0000 7f70827fc640 1 -- 172.21.15.60:0/2886530332 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708406cc10 con 0x7f7084159f00 2023-10-16T13:50:35.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.241+0000 7f7081ffb640 1 -- 172.21.15.60:0/2886530332 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015525715 0 0) 0x7f708406f890 con 0x7f7084159b20 2023-10-16T13:50:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.242+0000 7f70827fc640 1 --2- 172.21.15.60:0/2886530332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159f00 0x7f70840b0410 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f706c00d150 tx=0x7f706c02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d84d7ba82cd3887 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.242+0000 7f7081ffb640 1 -- 172.21.15.60:0/2886530332 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c00b0d0 con 0x7f7084159f00 2023-10-16T13:50:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.242+0000 7f7081ffb640 1 -- 172.21.15.60:0/2886530332 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c00b230 con 0x7f7084159f00 2023-10-16T13:50:35.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.242+0000 7f7081ffb640 1 -- 172.21.15.60:0/2886530332 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c009600 con 0x7f7084159f00 2023-10-16T13:50:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 -- 172.21.15.60:0/2886530332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159f00 msgr2=0x7f70840b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 --2- 172.21.15.60:0/2886530332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159f00 0x7f70840b0410 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f706c00d150 tx=0x7f706c02e180 comp rx=0 tx=0).stop 2023-10-16T13:50:35.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 -- 172.21.15.60:0/2886530332 shutdown_connections 2023-10-16T13:50:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 --2- 172.21.15.60:0/2886530332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159f00 0x7f70840b0410 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 -- 172.21.15.60:0/2886530332 >> 172.21.15.60:0/2886530332 conn(0x7f70840cb410 msgr2=0x7f70840cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 -- 172.21.15.60:0/2886530332 shutdown_connections 2023-10-16T13:50:35.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.243+0000 7f70899ce640 1 -- 172.21.15.60:0/2886530332 wait complete. 2023-10-16T13:50:35.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.245+0000 7f70899ce640 1 Processor -- start 2023-10-16T13:50:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.245+0000 7f70899ce640 1 -- start start 2023-10-16T13:50:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f70899ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 0x7f70840b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f70899ce640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7084159f00 0x7f7084158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f70899ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f708415d8c0 0x7f708415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f70899ce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708415c3b0 con 0x7f7084159b20 2023-10-16T13:50:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f70899ce640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708415bdf0 con 0x7f7084159f00 2023-10-16T13:50:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f70899ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708415c0d0 con 0x7f708415d8c0 2023-10-16T13:50:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.246+0000 7f7082ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 0x7f70840b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 0x7f70840b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49188/0 (socket says 172.21.15.60:49188) 2023-10-16T13:50:35.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 -- 172.21.15.60:0/2054499723 learned_addr learned my addr 172.21.15.60:0/2054499723 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:35.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f70827fc640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7084159f00 0x7f7084158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f70837fe640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f708415d8c0 0x7f708415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 -- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f708415d8c0 msgr2=0x7f708415b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f708415d8c0 0x7f708415b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 -- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7084159f00 msgr2=0x7f7084158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7084159f00 0x7f7084158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 -- 172.21.15.60:0/2054499723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c00bd40 con 0x7f7084159b20 2023-10-16T13:50:35.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.247+0000 7f7082ffd640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 0x7f70840b3c20 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f707400c4b0 tx=0x7f707400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:35.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.248+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074005480 con 0x7f7084159b20 2023-10-16T13:50:35.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.248+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70740035f0 con 0x7f7084159b20 2023-10-16T13:50:35.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.248+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70740039a0 con 0x7f7084159b20 2023-10-16T13:50:35.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.248+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70840b3420 con 0x7f7084159b20 2023-10-16T13:50:35.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.248+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708415ce10 con 0x7f7084159b20 2023-10-16T13:50:35.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.249+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70840b3280 con 0x7f7084159b20 2023-10-16T13:50:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.250+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7074003b00 con 0x7f7084159b20 2023-10-16T13:50:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.251+0000 7f707bfff640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f706406af90 0x7f706406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.251+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7074088da0 con 0x7f7084159b20 2023-10-16T13:50:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.252+0000 7f70827fc640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f706406af90 0x7f706406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.253+0000 7f70827fc640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f706406af90 0x7f706406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f706c037000 tx=0x7f706c032000 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:35.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.255+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70840b3280 con 0x7f7084159b20 2023-10-16T13:50:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.454+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f70840b3280 con 0x7f7084159b20 2023-10-16T13:50:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.462+0000 7f707bfff640 1 -- 172.21.15.60:0/2054499723 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f70840b3280 con 0x7f7084159b20 2023-10-16T13:50:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f706406af90 msgr2=0x7f706406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f706406af90 0x7f706406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f706c037000 tx=0x7f706c032000 comp rx=0 tx=0).stop 2023-10-16T13:50:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 msgr2=0x7f70840b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 0x7f70840b3c20 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f707400c4b0 tx=0x7f707400c970 comp rx=0 tx=0).stop 2023-10-16T13:50:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 shutdown_connections 2023-10-16T13:50:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f706406af90 0x7f706406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f708415d8c0 0x7f708415b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7084159f00 0x7f7084158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 --2- 172.21.15.60:0/2054499723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7084159b20 0x7f70840b3c20 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.464+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 >> 172.21.15.60:0/2054499723 conn(0x7f70840cb410 msgr2=0x7f70841567c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.465+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 shutdown_connections 2023-10-16T13:50:35.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.465+0000 7f70899ce640 1 -- 172.21.15.60:0/2054499723 wait complete. 2023-10-16T13:50:35.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:50:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.714+0000 7f4db88c6640 1 Processor -- start 2023-10-16T13:50:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.715+0000 7f4db88c6640 1 -- start start 2023-10-16T13:50:35.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.716+0000 7f4db88c6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4db415ad70 0x7f4db4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.716+0000 7f4db88c6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4db40caac0 0x7f4db406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.716+0000 7f4db88c6640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db40cb030 con 0x7f4db40c9ca0 2023-10-16T13:50:35.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.716+0000 7f4db88c6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40c91c0 con 0x7f4db415ad70 2023-10-16T13:50:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.716+0000 7f4db88c6640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40c94a0 con 0x7f4db40caac0 2023-10-16T13:50:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4db415ad70 0x7f4db4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4db415ad70 0x7f4db4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49200/0 (socket says 172.21.15.60:49200) 2023-10-16T13:50:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 -- 172.21.15.60:0/2739174142 learned_addr learned my addr 172.21.15.60:0/2739174142 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 -- 172.21.15.60:0/2739174142 >> v2:172.21.15.144:3300/0 conn(0x7f4db40caac0 msgr2=0x7f4db406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 --2- 172.21.15.60:0/2739174142 >> v2:172.21.15.144:3300/0 conn(0x7f4db40caac0 0x7f4db406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 -- 172.21.15.60:0/2739174142 >> v1:172.21.15.144:6789/0 conn(0x7f4db40c9ca0 legacy=0x7f4db40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.717+0000 7f4db259c640 1 -- 172.21.15.60:0/2739174142 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40c9780 con 0x7f4db415ad70 2023-10-16T13:50:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.718+0000 7f4db259c640 1 --2- 172.21.15.60:0/2739174142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db415ad70 0x7f4db4069da0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f4db406d090 tx=0x7f4da402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee6833c83cca595 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db1d9b640 1 -- 172.21.15.60:0/2739174142 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4031870 con 0x7f4db415ad70 2023-10-16T13:50:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db1d9b640 1 -- 172.21.15.60:0/2739174142 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40319d0 con 0x7f4db415ad70 2023-10-16T13:50:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db1d9b640 1 -- 172.21.15.60:0/2739174142 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4031d60 con 0x7f4db415ad70 2023-10-16T13:50:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 -- 172.21.15.60:0/2739174142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db415ad70 msgr2=0x7f4db4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 --2- 172.21.15.60:0/2739174142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db415ad70 0x7f4db4069da0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f4db406d090 tx=0x7f4da402b910 comp rx=0 tx=0).stop 2023-10-16T13:50:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 -- 172.21.15.60:0/2739174142 shutdown_connections 2023-10-16T13:50:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 --2- 172.21.15.60:0/2739174142 >> v2:172.21.15.144:3300/0 conn(0x7f4db40caac0 0x7f4db406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 --2- 172.21.15.60:0/2739174142 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db415ad70 0x7f4db4069da0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 -- 172.21.15.60:0/2739174142 >> 172.21.15.60:0/2739174142 conn(0x7f4db40ce9b0 msgr2=0x7f4db40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 -- 172.21.15.60:0/2739174142 shutdown_connections 2023-10-16T13:50:35.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.719+0000 7f4db88c6640 1 -- 172.21.15.60:0/2739174142 wait complete. 2023-10-16T13:50:35.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.721+0000 7f4db88c6640 1 Processor -- start 2023-10-16T13:50:35.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.721+0000 7f4db88c6640 1 -- start start 2023-10-16T13:50:35.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4db88c6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4db40c9ca0 0x7f4db41f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4db88c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 0x7f4db40c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4db88c6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4db415ad70 0x7f4db40c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4db88c6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41f32c0 con 0x7f4db40caac0 2023-10-16T13:50:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4db88c6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41f6840 con 0x7f4db415ad70 2023-10-16T13:50:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4db88c6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41f2740 con 0x7f4db40c9ca0 2023-10-16T13:50:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.722+0000 7f4da9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 0x7f4db40c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 0x7f4db40c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49206/0 (socket says 172.21.15.60:49206) 2023-10-16T13:50:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 -- 172.21.15.60:0/3822808369 learned_addr learned my addr 172.21.15.60:0/3822808369 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4db259c640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4db40c9ca0 0x7f4db41f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 -- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4db40c9ca0 msgr2=0x7f4db41f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4db40c9ca0 0x7f4db41f6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 -- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4db415ad70 msgr2=0x7f4db40c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4db415ad70 0x7f4db40c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 -- 172.21.15.60:0/3822808369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da40094d0 con 0x7f4db40caac0 2023-10-16T13:50:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.723+0000 7f4da9d9b640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 0x7f4db40c7850 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00c410 tx=0x7f4d9c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.724+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0035f0 con 0x7f4db40caac0 2023-10-16T13:50:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.724+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c003750 con 0x7f4db40caac0 2023-10-16T13:50:35.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.724+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c003b40 con 0x7f4db40caac0 2023-10-16T13:50:35.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.724+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db41f2a20 con 0x7f4db40caac0 2023-10-16T13:50:35.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.724+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db41f2fe0 con 0x7f4db40caac0 2023-10-16T13:50:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.725+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40c5b10 con 0x7f4db40caac0 2023-10-16T13:50:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.726+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4d9c003ca0 con 0x7f4db40caac0 2023-10-16T13:50:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.727+0000 7f4dab7fe640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4d9406af90 0x7f4d9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.727+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f4d9c088960 con 0x7f4db40caac0 2023-10-16T13:50:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.728+0000 7f4db259c640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4d9406af90 0x7f4d9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.729+0000 7f4db259c640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4d9406af90 0x7f4d9406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4da4036000 tx=0x7f4da403c040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.731+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4db40c5b10 con 0x7f4db40caac0 2023-10-16T13:50:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.927+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4db40c5b10 con 0x7f4db40caac0 2023-10-16T13:50:35.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.933+0000 7f4dab7fe640 1 -- 172.21.15.60:0/3822808369 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4db40c5b10 con 0x7f4db40caac0 2023-10-16T13:50:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4d9406af90 msgr2=0x7f4d9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4d9406af90 0x7f4d9406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4da4036000 tx=0x7f4da403c040 comp rx=0 tx=0).stop 2023-10-16T13:50:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 msgr2=0x7f4db40c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 0x7f4db40c7850 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00c410 tx=0x7f4d9c00c8d0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 shutdown_connections 2023-10-16T13:50:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f4d9406af90 0x7f4d9406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4db415ad70 0x7f4db40c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4db40caac0 0x7f4db40c7850 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 --2- 172.21.15.60:0/3822808369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4db40c9ca0 0x7f4db41f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 >> 172.21.15.60:0/3822808369 conn(0x7f4db40ce9b0 msgr2=0x7f4db40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 shutdown_connections 2023-10-16T13:50:35.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:35.936+0000 7f4db88c6640 1 -- 172.21.15.60:0/3822808369 wait complete. 2023-10-16T13:50:35.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:50:36.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.164+0000 7fcc18ee6640 1 Processor -- start 2023-10-16T13:50:36.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.164+0000 7fcc18ee6640 1 -- start start 2023-10-16T13:50:36.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc18ee6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415c9f0 0x7fcc140c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc18ee6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcc1415dde0 0x7fcc140ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc18ee6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc14160a90 con 0x7fcc140c9230 2023-10-16T13:50:36.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc18ee6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc140cdec0 con 0x7fcc1415dde0 2023-10-16T13:50:36.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc18ee6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140cd900 con 0x7fcc1415c9f0 2023-10-16T13:50:36.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc13fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcc1415dde0 0x7fcc140ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc0bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcc140c9230 0x7fcc140709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41370/0 (socket says 172.21.15.60:41370) 2023-10-16T13:50:36.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.166+0000 7fcc0bfff640 1 -- 172.21.15.60:0/3513996291 learned_addr learned my addr 172.21.15.60:0/3513996291 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:36.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc137fe640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415c9f0 0x7fcc140c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc13fff640 1 -- 172.21.15.60:0/3513996291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415c9f0 msgr2=0x7fcc140c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:36.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc13fff640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415c9f0 0x7fcc140c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc13fff640 1 -- 172.21.15.60:0/3513996291 >> v1:172.21.15.60:6789/0 conn(0x7fcc140c9230 legacy=0x7fcc140709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc13fff640 1 -- 172.21.15.60:0/3513996291 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc140cdbe0 con 0x7fcc1415dde0 2023-10-16T13:50:36.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc12ffd640 1 -- 172.21.15.60:0/3513996291 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179501702 0 0) 0x7fcc14160a90 con 0x7fcc140c9230 2023-10-16T13:50:36.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.167+0000 7fcc137fe640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415c9f0 0x7fcc140c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:36.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.168+0000 7fcc13fff640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc1415dde0 0x7fcc140ac890 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e6a6b8e5fd9817 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:36.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.168+0000 7fcc12ffd640 1 -- 172.21.15.60:0/3513996291 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c02cc60 con 0x7fcc1415dde0 2023-10-16T13:50:36.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.168+0000 7fcc12ffd640 1 -- 172.21.15.60:0/3513996291 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c02cdc0 con 0x7fcc1415dde0 2023-10-16T13:50:36.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.168+0000 7fcc12ffd640 1 -- 172.21.15.60:0/3513996291 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c034660 con 0x7fcc1415dde0 2023-10-16T13:50:36.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.169+0000 7fcc18ee6640 1 -- 172.21.15.60:0/3513996291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc1415dde0 msgr2=0x7fcc140ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.169+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc1415dde0 0x7fcc140ac890 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02beb0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.169+0000 7fcc18ee6640 1 -- 172.21.15.60:0/3513996291 shutdown_connections 2023-10-16T13:50:36.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.169+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc1415dde0 0x7fcc140ac890 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.169+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/3513996291 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415c9f0 0x7fcc140c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.169+0000 7fcc18ee6640 1 -- 172.21.15.60:0/3513996291 >> 172.21.15.60:0/3513996291 conn(0x7fcc140ce6a0 msgr2=0x7fcc140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:36.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.170+0000 7fcc18ee6640 1 -- 172.21.15.60:0/3513996291 shutdown_connections 2023-10-16T13:50:36.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.170+0000 7fcc18ee6640 1 -- 172.21.15.60:0/3513996291 wait complete. 2023-10-16T13:50:36.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.172+0000 7fcc18ee6640 1 Processor -- start 2023-10-16T13:50:36.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.172+0000 7fcc18ee6640 1 -- start start 2023-10-16T13:50:36.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.173+0000 7fcc18ee6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 0x7fcc140c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.173+0000 7fcc18ee6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc1415c9f0 0x7fcc140c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc18ee6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415dde0 0x7fcc141462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc18ee6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc141f3050 con 0x7fcc140c9230 2023-10-16T13:50:36.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc18ee6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140c1f10 con 0x7fcc1415c9f0 2023-10-16T13:50:36.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc18ee6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140c3e40 con 0x7fcc1415dde0 2023-10-16T13:50:36.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 0x7fcc140c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 0x7fcc140c19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49224/0 (socket says 172.21.15.60:49224) 2023-10-16T13:50:36.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 -- 172.21.15.60:0/304934300 learned_addr learned my addr 172.21.15.60:0/304934300 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:36.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 -- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415dde0 msgr2=0x7fcc141462d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415dde0 0x7fcc141462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 -- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc1415c9f0 msgr2=0x7fcc140c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc1415c9f0 0x7fcc140c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.174+0000 7fcc137fe640 1 -- 172.21.15.60:0/304934300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c009a70 con 0x7fcc140c9230 2023-10-16T13:50:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.175+0000 7fcc137fe640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 0x7fcc140c19d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c850 tx=0x7fcc0400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.175+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04007a30 con 0x7fcc140c9230 2023-10-16T13:50:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.175+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc04007b90 con 0x7fcc140c9230 2023-10-16T13:50:36.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.175+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400cea0 con 0x7fcc140c9230 2023-10-16T13:50:36.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.175+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc141f2d70 con 0x7fcc140c9230 2023-10-16T13:50:36.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.175+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc140c43a0 con 0x7fcc140c9230 2023-10-16T13:50:36.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.176+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1404e9b0 con 0x7fcc140c9230 2023-10-16T13:50:36.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.177+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc040113f0 con 0x7fcc140c9230 2023-10-16T13:50:36.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.178+0000 7fcc10ff9640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcbf406af90 0x7fcbf406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.178+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fcc04089ab0 con 0x7fcc140c9230 2023-10-16T13:50:36.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.179+0000 7fcc0bfff640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcbf406af90 0x7fcbf406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.180+0000 7fcc0bfff640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcbf406af90 0x7fcbf406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc007fd0 tx=0x7fcbfc007920 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:36.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.183+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc04054870 con 0x7fcc140c9230 2023-10-16T13:50:36.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.380+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fcc140adac0 con 0x7fcc140c9230 2023-10-16T13:50:36.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.386+0000 7fcc10ff9640 1 -- 172.21.15.60:0/304934300 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7fcc04057e90 con 0x7fcc140c9230 2023-10-16T13:50:36.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.389+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcbf406af90 msgr2=0x7fcbf406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.389+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcbf406af90 0x7fcbf406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc007fd0 tx=0x7fcbfc007920 comp rx=0 tx=0).stop 2023-10-16T13:50:36.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.389+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 msgr2=0x7fcc140c19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.389+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 0x7fcc140c19d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c850 tx=0x7fcc0400cc50 comp rx=0 tx=0).stop 2023-10-16T13:50:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 shutdown_connections 2023-10-16T13:50:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcbf406af90 0x7fcbf406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc1415dde0 0x7fcc141462d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc1415c9f0 0x7fcc140c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 --2- 172.21.15.60:0/304934300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc140c9230 0x7fcc140c19d0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 >> 172.21.15.60:0/304934300 conn(0x7fcc140ce6a0 msgr2=0x7fcc1415fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:36.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 shutdown_connections 2023-10-16T13:50:36.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.390+0000 7fcc18ee6640 1 -- 172.21.15.60:0/304934300 wait complete. 2023-10-16T13:50:36.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T13:50:36.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.631+0000 7f5519cd1640 1 Processor -- start 2023-10-16T13:50:36.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.631+0000 7f5519cd1640 1 -- start start 2023-10-16T13:50:36.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.632+0000 7f5519cd1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f551415abf0 0x7f55140afce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.632+0000 7f5519cd1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551406f900 0x7f55140b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.632+0000 7f5519cd1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f55140b28b0 0x7f55140ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5519cd1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55140b2d40 con 0x7f551415abf0 2023-10-16T13:50:36.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5519cd1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551406fd60 con 0x7f55140b28b0 2023-10-16T13:50:36.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5519cd1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55140cf6e0 con 0x7f551406f900 2023-10-16T13:50:36.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f551415abf0 0x7f55140afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f551415abf0 0x7f55140afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49234/0 (socket says 172.21.15.60:49234) 2023-10-16T13:50:36.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 -- 172.21.15.60:0/562594126 learned_addr learned my addr 172.21.15.60:0/562594126 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 -- 172.21.15.60:0/562594126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551406f900 msgr2=0x7f55140b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 --2- 172.21.15.60:0/562594126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551406f900 0x7f55140b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 -- 172.21.15.60:0/562594126 >> v2:172.21.15.144:3300/0 conn(0x7f55140b28b0 msgr2=0x7f55140ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f55194d0640 1 --2- 172.21.15.60:0/562594126 >> v2:172.21.15.144:3300/0 conn(0x7f55140b28b0 0x7f55140ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 --2- 172.21.15.60:0/562594126 >> v2:172.21.15.144:3300/0 conn(0x7f55140b28b0 0x7f55140ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.633+0000 7f5518ccf640 1 -- 172.21.15.60:0/562594126 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551415afd0 con 0x7f551415abf0 2023-10-16T13:50:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.635+0000 7f5518ccf640 1 --2- 172.21.15.60:0/562594126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551415abf0 0x7f55140afce0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5514073b30 tx=0x7f551002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de53ea073509c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.635+0000 7f550b7fe640 1 -- 172.21.15.60:0/562594126 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100317d0 con 0x7f551415abf0 2023-10-16T13:50:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.635+0000 7f550b7fe640 1 -- 172.21.15.60:0/562594126 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5510031930 con 0x7f551415abf0 2023-10-16T13:50:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.635+0000 7f550b7fe640 1 -- 172.21.15.60:0/562594126 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5510031cc0 con 0x7f551415abf0 2023-10-16T13:50:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 -- 172.21.15.60:0/562594126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551415abf0 msgr2=0x7f55140afce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 --2- 172.21.15.60:0/562594126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551415abf0 0x7f55140afce0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5514073b30 tx=0x7f551002b870 comp rx=0 tx=0).stop 2023-10-16T13:50:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 -- 172.21.15.60:0/562594126 shutdown_connections 2023-10-16T13:50:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 --2- 172.21.15.60:0/562594126 >> v2:172.21.15.144:3300/0 conn(0x7f55140b28b0 0x7f55140ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 --2- 172.21.15.60:0/562594126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551406f900 0x7f55140b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 --2- 172.21.15.60:0/562594126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551415abf0 0x7f55140afce0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 -- 172.21.15.60:0/562594126 >> 172.21.15.60:0/562594126 conn(0x7f55140cd370 msgr2=0x7f55140cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 -- 172.21.15.60:0/562594126 shutdown_connections 2023-10-16T13:50:36.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.636+0000 7f5519cd1640 1 -- 172.21.15.60:0/562594126 wait complete. 2023-10-16T13:50:36.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.638+0000 7f5519cd1640 1 Processor -- start 2023-10-16T13:50:36.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.638+0000 7f5519cd1640 1 -- start start 2023-10-16T13:50:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5519cd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 0x7f551415d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5519cd1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140b28b0 0x7f55140b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5519cd1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415abf0 0x7f55141f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5519cd1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551415d770 con 0x7f551406f900 2023-10-16T13:50:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5519cd1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55140b5df0 con 0x7f55140b28b0 2023-10-16T13:50:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5519cd1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55141f6610 con 0x7f551415abf0 2023-10-16T13:50:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5518ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 0x7f551415d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5518ccf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 0x7f551415d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49240/0 (socket says 172.21.15.60:49240) 2023-10-16T13:50:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.639+0000 7f5518ccf640 1 -- 172.21.15.60:0/788899079 learned_addr learned my addr 172.21.15.60:0/788899079 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f550bfff640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140b28b0 0x7f55140b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f55194d0640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415abf0 0x7f55141f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5518ccf640 1 -- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415abf0 msgr2=0x7f55141f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5518ccf640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415abf0 0x7f55141f6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5518ccf640 1 -- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140b28b0 msgr2=0x7f55140b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5518ccf640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140b28b0 0x7f55140b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5518ccf640 1 -- 172.21.15.60:0/788899079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100094d0 con 0x7f551406f900 2023-10-16T13:50:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f550bfff640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140b28b0 0x7f55140b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5518ccf640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 0x7f551415d230 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f5510000c00 tx=0x7f55100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100328f0 con 0x7f551406f900 2023-10-16T13:50:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5510032a70 con 0x7f551406f900 2023-10-16T13:50:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5510032e20 con 0x7f551406f900 2023-10-16T13:50:36.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.640+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551415d960 con 0x7f551406f900 2023-10-16T13:50:36.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.641+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55141f68b0 con 0x7f551406f900 2023-10-16T13:50:36.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.642+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551404e9b0 con 0x7f551406f900 2023-10-16T13:50:36.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.643+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5510030070 con 0x7f551406f900 2023-10-16T13:50:36.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.644+0000 7f55097fa640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f550006af90 0x7f550006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:36.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.644+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f55100ab860 con 0x7f551406f900 2023-10-16T13:50:36.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.645+0000 7f550bfff640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f550006af90 0x7f550006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:36.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.646+0000 7f550bfff640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f550006af90 0x7f550006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f55141f9a50 tx=0x7f5504008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:36.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.649+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5510076880 con 0x7f551406f900 2023-10-16T13:50:36.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:36.925+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f55140b3020 con 0x7f551406f900 2023-10-16T13:50:37.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.295+0000 7f55097fa640 1 -- 172.21.15.60:0/788899079 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2521) v1 ==== 178+0+0 (secure 0 0 0) 0x7f5510079ea0 con 0x7f551406f900 2023-10-16T13:50:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f550006af90 msgr2=0x7f550006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f550006af90 0x7f550006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f55141f9a50 tx=0x7f5504008040 comp rx=0 tx=0).stop 2023-10-16T13:50:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 msgr2=0x7f551415d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 0x7f551415d230 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f5510000c00 tx=0x7f55100046c0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 shutdown_connections 2023-10-16T13:50:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f550006af90 0x7f550006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415abf0 0x7f55141f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140b28b0 0x7f55140b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 --2- 172.21.15.60:0/788899079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f551406f900 0x7f551415d230 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 >> 172.21.15.60:0/788899079 conn(0x7f55140cd370 msgr2=0x7f55140cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 shutdown_connections 2023-10-16T13:50:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.299+0000 7f5519cd1640 1 -- 172.21.15.60:0/788899079 wait complete. 2023-10-16T13:50:37.324 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-16T13:50:37.325 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-16T13:50:37.325 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T13:50:37.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:37.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.551+0000 7fbb9b347640 1 Processor -- start 2023-10-16T13:50:37.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.552+0000 7fbb9b347640 1 -- start start 2023-10-16T13:50:37.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.553+0000 7fbb9b347640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.553+0000 7fbb9b347640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbb940ad970 0x7fbb9415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.553+0000 7fbb9b347640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb9415ebc0 0x7fbb9415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.553+0000 7fbb9b347640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9406c960 con 0x7fbb9415ebc0 2023-10-16T13:50:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.553+0000 7fbb9b347640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9415b9d0 con 0x7fbb940ad970 2023-10-16T13:50:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.553+0000 7fbb9b347640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9415bcb0 con 0x7fbb9406c040 2023-10-16T13:50:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb9415ebc0 0x7fbb9415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb9415ebc0 0x7fbb9415f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49248/0 (socket says 172.21.15.60:49248) 2023-10-16T13:50:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 -- 172.21.15.60:0/2548089105 learned_addr learned my addr 172.21.15.60:0/2548089105 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:37.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9a345640 1 --2- 172.21.15.60:0/2548089105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb99b44640 1 --2- 172.21.15.60:0/2548089105 >> v2:172.21.15.144:3300/0 conn(0x7fbb940ad970 0x7fbb9415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 -- 172.21.15.60:0/2548089105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 msgr2=0x7fbb9406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:37.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 --2- 172.21.15.60:0/2548089105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 -- 172.21.15.60:0/2548089105 >> v2:172.21.15.144:3300/0 conn(0x7fbb940ad970 msgr2=0x7fbb9415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:37.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 --2- 172.21.15.60:0/2548089105 >> v2:172.21.15.144:3300/0 conn(0x7fbb940ad970 0x7fbb9415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.554+0000 7fbb9ab46640 1 -- 172.21.15.60:0/2548089105 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9415bf90 con 0x7fbb9415ebc0 2023-10-16T13:50:37.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.555+0000 7fbb9ab46640 1 --2- 172.21.15.60:0/2548089105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb9415ebc0 0x7fbb9415f050 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009e50 tx=0x7fbb8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a0648653d04767 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:37.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb99343640 1 -- 172.21.15.60:0/2548089105 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c02ca70 con 0x7fbb9415ebc0 2023-10-16T13:50:37.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb99343640 1 -- 172.21.15.60:0/2548089105 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c02cbd0 con 0x7fbb9415ebc0 2023-10-16T13:50:37.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb99343640 1 -- 172.21.15.60:0/2548089105 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0343f0 con 0x7fbb9415ebc0 2023-10-16T13:50:37.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 -- 172.21.15.60:0/2548089105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb9415ebc0 msgr2=0x7fbb9415f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 --2- 172.21.15.60:0/2548089105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb9415ebc0 0x7fbb9415f050 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009e50 tx=0x7fbb8c02bed0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 -- 172.21.15.60:0/2548089105 shutdown_connections 2023-10-16T13:50:37.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 --2- 172.21.15.60:0/2548089105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb9415ebc0 0x7fbb9415f050 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 --2- 172.21.15.60:0/2548089105 >> v2:172.21.15.144:3300/0 conn(0x7fbb940ad970 0x7fbb9415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 --2- 172.21.15.60:0/2548089105 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.556+0000 7fbb9b347640 1 -- 172.21.15.60:0/2548089105 >> 172.21.15.60:0/2548089105 conn(0x7fbb940ce6a0 msgr2=0x7fbb940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:37.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.557+0000 7fbb9b347640 1 -- 172.21.15.60:0/2548089105 shutdown_connections 2023-10-16T13:50:37.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.557+0000 7fbb9b347640 1 -- 172.21.15.60:0/2548089105 wait complete. 2023-10-16T13:50:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.558+0000 7fbb9b347640 1 Processor -- start 2023-10-16T13:50:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.559+0000 7fbb9b347640 1 -- start start 2023-10-16T13:50:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9b347640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9b347640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 0x7fbb940c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9b347640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb9415ebc0 0x7fbb940c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9b347640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb940c8170 con 0x7fbb940ad970 2023-10-16T13:50:37.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9b347640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb940c7880 con 0x7fbb9415ebc0 2023-10-16T13:50:37.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9b347640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb941fa9f0 con 0x7fbb9406c040 2023-10-16T13:50:37.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb99b44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 0x7fbb940c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb99b44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 0x7fbb940c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49256/0 (socket says 172.21.15.60:49256) 2023-10-16T13:50:37.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb99b44640 1 -- 172.21.15.60:0/210270834 learned_addr learned my addr 172.21.15.60:0/210270834 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:37.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9a345640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.560+0000 7fbb9ab46640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb9415ebc0 0x7fbb940c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb99b44640 1 -- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 msgr2=0x7fbb9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb99b44640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9414dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb99b44640 1 -- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb9415ebc0 msgr2=0x7fbb940c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb99b44640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb9415ebc0 0x7fbb940c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb99b44640 1 -- 172.21.15.60:0/210270834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c009b30 con 0x7fbb940ad970 2023-10-16T13:50:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb9a345640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9414dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:37.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb99b44640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 0x7fbb940c7340 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400d5f0 tx=0x7fbb8400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb840125f0 con 0x7fbb940ad970 2023-10-16T13:50:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb84012750 con 0x7fbb940ad970 2023-10-16T13:50:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84012b00 con 0x7fbb940ad970 2023-10-16T13:50:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.561+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb941facd0 con 0x7fbb940ad970 2023-10-16T13:50:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.562+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb940c6b40 con 0x7fbb940ad970 2023-10-16T13:50:37.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.563+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9406b330 con 0x7fbb940ad970 2023-10-16T13:50:37.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.564+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbb84012c60 con 0x7fbb940ad970 2023-10-16T13:50:37.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.565+0000 7fbb8affd640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbb7806af90 0x7fbb7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:37.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.565+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fbb84089bd0 con 0x7fbb940ad970 2023-10-16T13:50:37.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.565+0000 7fbb9a345640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbb7806af90 0x7fbb7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:37.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.567+0000 7fbb9a345640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbb7806af90 0x7fbb7806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb9406a0d0 tx=0x7fbb90008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:37.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.570+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb84054bc0 con 0x7fbb940ad970 2023-10-16T13:50:37.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.764+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb9404e9b0 con 0x7fbb940ad970 2023-10-16T13:50:37.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.766+0000 7fbb8affd640 1 -- 172.21.15.60:0/210270834 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+14000 (secure 0 0 0) 0x7fbb840581e0 con 0x7fbb940ad970 2023-10-16T13:50:37.769 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:37.770 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":174,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:34.999266+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6833/3372982800":"2023-10-17T13:50:31.973474+0000","172.21.15.144:0/1674745327":"2023-10-17T13:50:27.750339+0000","172.21.15.144:6832/110255757":"2023-10-17T13:50:27.750339+0000","172.21.15.60:6832/3372982800":"2023-10-17T13:50:31.973474+0000","172.21.15.144:0/1615247958":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/1009875684":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/931153185":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/3784462343":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/3666412233":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/362473654":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1686539411":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/4019169897":"2023-10-17T13:50:27.750339+0000","172.21.15.144:6832/3094169155":"2023-10-17T12:54:28.068770+0000","172.21.15.144:6833/110255757":"2023-10-17T13:50:27.750339+0000","172.21.15.144:0/4230715707":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1256386612":"2023-10-17T12:54:28.068770+0000","172.21.15.144:0/1071036703":"2023-10-17T12:54:28.068770+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:37.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbb7806af90 msgr2=0x7fbb7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbb7806af90 0x7fbb7806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb9406a0d0 tx=0x7fbb90008040 comp rx=0 tx=0).stop 2023-10-16T13:50:37.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 msgr2=0x7fbb940c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:37.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 0x7fbb940c7340 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400d5f0 tx=0x7fbb8400dab0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 shutdown_connections 2023-10-16T13:50:37.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbb7806af90 0x7fbb7806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb9415ebc0 0x7fbb940c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb940ad970 0x7fbb940c7340 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 --2- 172.21.15.60:0/210270834 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb9406c040 0x7fbb9414dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:37.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 >> 172.21.15.60:0/210270834 conn(0x7fbb940ce6a0 msgr2=0x7fbb940c89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:37.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 shutdown_connections 2023-10-16T13:50:37.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:37.769+0000 7fbb9b347640 1 -- 172.21.15.60:0/210270834 wait complete. 2023-10-16T13:50:37.790 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:50:37.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:50:38.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.018+0000 7fcfd95ac640 1 Processor -- start 2023-10-16T13:50:38.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.018+0000 7fcfd95ac640 1 -- start start 2023-10-16T13:50:38.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.019+0000 7fcfd95ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b1ca0 0x7fcfd40d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.019+0000 7fcfd95ac640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcfd40cfc40 0x7fcfd40b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd95ac640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd40d0970 con 0x7fcfd40b2080 2023-10-16T13:50:38.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd95ac640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40d0ea0 con 0x7fcfd40cfc40 2023-10-16T13:50:38.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd95ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40d0bc0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd8dab640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcfd40cfc40 0x7fcfd40b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcfd40b2080 0x7fcfd40ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41380/0 (socket says 172.21.15.60:41380) 2023-10-16T13:50:38.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd37fe640 1 -- 172.21.15.60:0/3008564292 learned_addr learned my addr 172.21.15.60:0/3008564292 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:38.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.020+0000 7fcfd3fff640 1 --2- 172.21.15.60:0/3008564292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b1ca0 0x7fcfd40d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.021+0000 7fcfd2ffd640 1 -- 172.21.15.60:0/3008564292 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975056131 0 0) 0x7fcfd40d0970 con 0x7fcfd40b2080 2023-10-16T13:50:38.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.021+0000 7fcfd2ffd640 1 -- 172.21.15.60:0/3008564292 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfbc003520 con 0x7fcfd40b2080 2023-10-16T13:50:38.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.021+0000 7fcfd8dab640 1 -- 172.21.15.60:0/3008564292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b1ca0 msgr2=0x7fcfd40d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:38.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.021+0000 7fcfd8dab640 1 --2- 172.21.15.60:0/3008564292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b1ca0 0x7fcfd40d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.021+0000 7fcfd8dab640 1 -- 172.21.15.60:0/3008564292 >> v1:172.21.15.60:6789/0 conn(0x7fcfd40b2080 legacy=0x7fcfd40ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.021+0000 7fcfd8dab640 1 -- 172.21.15.60:0/3008564292 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd40cd940 con 0x7fcfd40cfc40 2023-10-16T13:50:38.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.022+0000 7fcfd8dab640 1 --2- 172.21.15.60:0/3008564292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40b24c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0097a0 tx=0x7fcfcc02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afed4db3498447b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:38.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.022+0000 7fcfd2ffd640 1 -- 172.21.15.60:0/3008564292 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc030800 con 0x7fcfd40cfc40 2023-10-16T13:50:38.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.022+0000 7fcfd2ffd640 1 -- 172.21.15.60:0/3008564292 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfcc030960 con 0x7fcfd40cfc40 2023-10-16T13:50:38.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.022+0000 7fcfd2ffd640 1 -- 172.21.15.60:0/3008564292 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc030cd0 con 0x7fcfd40cfc40 2023-10-16T13:50:38.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3008564292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40cfc40 msgr2=0x7fcfd40b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3008564292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40b24c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0097a0 tx=0x7fcfcc02b940 comp rx=0 tx=0).stop 2023-10-16T13:50:38.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3008564292 shutdown_connections 2023-10-16T13:50:38.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3008564292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40b24c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3008564292 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b1ca0 0x7fcfd40d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3008564292 >> 172.21.15.60:0/3008564292 conn(0x7fcfd40d2b90 msgr2=0x7fcfd40d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:38.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3008564292 shutdown_connections 2023-10-16T13:50:38.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.023+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3008564292 wait complete. 2023-10-16T13:50:38.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.025+0000 7fcfd95ac640 1 Processor -- start 2023-10-16T13:50:38.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.025+0000 7fcfd95ac640 1 -- start start 2023-10-16T13:50:38.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd95ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 0x7fcfd40cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd95ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b2080 0x7fcfd40c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd95ac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd95ac640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c70d0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd95ac640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c95b0 con 0x7fcfd40cfc40 2023-10-16T13:50:38.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd95ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c7320 con 0x7fcfd40b2080 2023-10-16T13:50:38.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 0x7fcfd40cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 0x7fcfd40cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49266/0 (socket says 172.21.15.60:49266) 2023-10-16T13:50:38.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.026+0000 7fcfd3fff640 1 -- 172.21.15.60:0/3361454428 learned_addr learned my addr 172.21.15.60:0/3361454428 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:38.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd37fe640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b2080 0x7fcfd40c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd3fff640 1 -- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b2080 msgr2=0x7fcfd40c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd8dab640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd3fff640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b2080 0x7fcfd40c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd3fff640 1 -- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfd40cfc40 msgr2=0x7fcfd40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd3fff640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd3fff640 1 -- 172.21.15.60:0/3361454428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc009400 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd37fe640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b2080 0x7fcfd40c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:38.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd3fff640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 0x7fcfd40cae30 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4007e40 tx=0x7fcfc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:38.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4014620 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc400ee90 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40149c0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.027+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd40c7600 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.028+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd40c7bc0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.029+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd40c7d80 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.030+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcfc4014b20 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.031+0000 7fcfd0ff9640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcfc006af90 0x7fcfc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.031+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fcfc408a030 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.031+0000 7fcfd37fe640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcfc006af90 0x7fcfc006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.032+0000 7fcfd37fe640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcfc006af90 0x7fcfc006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcfd406b330 tx=0x7fcfc8009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:38.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.036+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcfc40550b0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.231+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfd404e9b0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.232+0000 7fcfd0ff9640 1 -- 172.21.15.60:0/3361454428 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v174) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfc40586d0 con 0x7fcfd40b1ca0 2023-10-16T13:50:38.236 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:50:38.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcfc006af90 msgr2=0x7fcfc006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcfc006af90 0x7fcfc006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcfd406b330 tx=0x7fcfc8009040 comp rx=0 tx=0).stop 2023-10-16T13:50:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 msgr2=0x7fcfd40cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 0x7fcfd40cae30 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4007e40 tx=0x7fcfc400e4b0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 shutdown_connections 2023-10-16T13:50:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fcfc006af90 0x7fcfc006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcfd40b2080 0x7fcfd40c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfd40cfc40 0x7fcfd40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 --2- 172.21.15.60:0/3361454428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfd40b1ca0 0x7fcfd40cae30 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 >> 172.21.15.60:0/3361454428 conn(0x7fcfd40d2b90 msgr2=0x7fcfd415ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 shutdown_connections 2023-10-16T13:50:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.235+0000 7fcfd95ac640 1 -- 172.21.15.60:0/3361454428 wait complete. 2023-10-16T13:50:38.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T13:50:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.479+0000 7ff58d5d2640 1 Processor -- start 2023-10-16T13:50:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.480+0000 7ff58d5d2640 1 -- start start 2023-10-16T13:50:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.481+0000 7ff58d5d2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff58806c9e0 0x7ff5880ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.481+0000 7ff58d5d2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5880ca560 con 0x7ff58806fb50 2023-10-16T13:50:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.481+0000 7ff58d5d2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5880c9950 con 0x7ff5880c9060 2023-10-16T13:50:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.481+0000 7ff58d5d2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5880c9c30 con 0x7ff58806c9e0 2023-10-16T13:50:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff58cdd1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff58806c9e0 0x7ff5880ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff57ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff58806fb50 0x7ff5880cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41382/0 (socket says 172.21.15.60:41382) 2023-10-16T13:50:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff57ffff640 1 -- 172.21.15.60:0/1765898932 learned_addr learned my addr 172.21.15.60:0/1765898932 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff58cdd1640 1 -- 172.21.15.60:0/1765898932 >> v1:172.21.15.144:6789/0 conn(0x7ff5880c9060 legacy=0x7ff58806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff58cdd1640 1 -- 172.21.15.60:0/1765898932 >> v1:172.21.15.60:6789/0 conn(0x7ff58806fb50 legacy=0x7ff5880cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff58cdd1640 1 -- 172.21.15.60:0/1765898932 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5880c9f10 con 0x7ff58806c9e0 2023-10-16T13:50:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.482+0000 7ff5877fe640 1 -- 172.21.15.60:0/1765898932 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673898898 0 0) 0x7ff5880ca560 con 0x7ff58806fb50 2023-10-16T13:50:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.483+0000 7ff58cdd1640 1 --2- 172.21.15.60:0/1765898932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58806c9e0 0x7ff5880ccfb0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff58002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9350f50a02cd558b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.484+0000 7ff5877fe640 1 -- 172.21.15.60:0/1765898932 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58002cc60 con 0x7ff58806c9e0 2023-10-16T13:50:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.484+0000 7ff5877fe640 1 -- 172.21.15.60:0/1765898932 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58002cdc0 con 0x7ff58806c9e0 2023-10-16T13:50:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.484+0000 7ff5877fe640 1 -- 172.21.15.60:0/1765898932 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580034660 con 0x7ff58806c9e0 2023-10-16T13:50:38.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.484+0000 7ff58d5d2640 1 -- 172.21.15.60:0/1765898932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58806c9e0 msgr2=0x7ff5880ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.484+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/1765898932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58806c9e0 0x7ff5880ccfb0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff58002beb0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.485+0000 7ff58d5d2640 1 -- 172.21.15.60:0/1765898932 shutdown_connections 2023-10-16T13:50:38.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.485+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/1765898932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff58806c9e0 0x7ff5880ccfb0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.485+0000 7ff58d5d2640 1 -- 172.21.15.60:0/1765898932 >> 172.21.15.60:0/1765898932 conn(0x7ff5880ce6a0 msgr2=0x7ff5880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.485+0000 7ff58d5d2640 1 -- 172.21.15.60:0/1765898932 shutdown_connections 2023-10-16T13:50:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.485+0000 7ff58d5d2640 1 -- 172.21.15.60:0/1765898932 wait complete. 2023-10-16T13:50:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.487+0000 7ff58d5d2640 1 Processor -- start 2023-10-16T13:50:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.487+0000 7ff58d5d2640 1 -- start start 2023-10-16T13:50:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58d5d2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58806c9e0 0x7ff588154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58d5d2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff58806fb50 0x7ff5880ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58d5d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 0x7ff5880cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58d5d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880b42b0 con 0x7ff5880c9060 2023-10-16T13:50:38.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58d5d2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880c2090 con 0x7ff58806fb50 2023-10-16T13:50:38.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58d5d2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff588162160 con 0x7ff58806c9e0 2023-10-16T13:50:38.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58cdd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 0x7ff5880cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58cdd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 0x7ff5880cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49280/0 (socket says 172.21.15.60:49280) 2023-10-16T13:50:38.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff58cdd1640 1 -- 172.21.15.60:0/308475803 learned_addr learned my addr 172.21.15.60:0/308475803 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:38.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff57ffff640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff58806fb50 0x7ff5880ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.488+0000 7ff587fff640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58806c9e0 0x7ff588154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58cdd1640 1 -- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58806c9e0 msgr2=0x7ff588154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58cdd1640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58806c9e0 0x7ff588154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58cdd1640 1 -- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff58806fb50 msgr2=0x7ff5880ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:38.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58cdd1640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff58806fb50 0x7ff5880ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:38.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58cdd1640 1 -- 172.21.15.60:0/308475803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580009a70 con 0x7ff5880c9060 2023-10-16T13:50:38.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff57ffff640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff58806fb50 0x7ff5880ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:38.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff587fff640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58806c9e0 0x7ff588154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:38.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58cdd1640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 0x7ff5880cd060 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:38.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58003e610 con 0x7ff5880c9060 2023-10-16T13:50:38.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58003e790 con 0x7ff5880c9060 2023-10-16T13:50:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58003eb40 con 0x7ff5880c9060 2023-10-16T13:50:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.489+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff588162440 con 0x7ff5880c9060 2023-10-16T13:50:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.490+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff588162c40 con 0x7ff5880c9060 2023-10-16T13:50:38.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.491+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58804e9b0 con 0x7ff5880c9060 2023-10-16T13:50:38.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.492+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff58003b050 con 0x7ff5880c9060 2023-10-16T13:50:38.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.492+0000 7ff5857fa640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff56806afc0 0x7ff56806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.493+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ff5800ac1c0 con 0x7ff5880c9060 2023-10-16T13:50:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.493+0000 7ff587fff640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff56806afc0 0x7ff56806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.494+0000 7ff587fff640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff56806afc0 0x7ff56806d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff57800a830 tx=0x7ff578008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:38.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.497+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5800771a0 con 0x7ff5880c9060 2023-10-16T13:50:38.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:38.769+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7ff5880b4620 con 0x7ff5880c9060 2023-10-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.310+0000 7ff5857fa640 1 -- 172.21.15.60:0/308475803 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2523) v1 ==== 177+0+0 (secure 0 0 0) 0x7ff58007a7c0 con 0x7ff5880c9060 2023-10-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.314+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff56806afc0 msgr2=0x7ff56806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.314+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff56806afc0 0x7ff56806d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff57800a830 tx=0x7ff578008040 comp rx=0 tx=0).stop 2023-10-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.314+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 msgr2=0x7ff5880cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.314+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 0x7ff5880cd060 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff580004a40 comp rx=0 tx=0).stop 2023-10-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 shutdown_connections 2023-10-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff56806afc0 0x7ff56806d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5880c9060 0x7ff5880cd060 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff58806fb50 0x7ff5880ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 --2- 172.21.15.60:0/308475803 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff58806c9e0 0x7ff588154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 >> 172.21.15.60:0/308475803 conn(0x7ff5880ce6a0 msgr2=0x7ff5880cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 shutdown_connections 2023-10-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.315+0000 7ff58d5d2640 1 -- 172.21.15.60:0/308475803 wait complete. 2023-10-16T13:50:39.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:50:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.579+0000 7fdb4c37d640 1 Processor -- start 2023-10-16T13:50:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.579+0000 7fdb4c37d640 1 -- start start 2023-10-16T13:50:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.580+0000 7fdb4c37d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb440ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4c37d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb440c9ca0 0x7fdb440ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4c37d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb440cb030 con 0x7fdb4415ad70 2023-10-16T13:50:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4c37d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb440cae30 con 0x7fdb440c9ca0 2023-10-16T13:50:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4c37d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb440c91c0 con 0x7fdb4406a4b0 2023-10-16T13:50:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb49918640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb440ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4a91a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb440c9ca0 0x7fdb440ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4a91a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb440c9ca0 0x7fdb440ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:35598/0 (socket says 172.21.15.60:35598) 2023-10-16T13:50:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.581+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2699499913 learned_addr learned my addr 172.21.15.60:0/2699499913 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.582+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 msgr2=0x7fdb440ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.582+0000 7fdb4a91a640 1 --2- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb440ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.582+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2699499913 >> v1:172.21.15.144:6789/0 conn(0x7fdb4415ad70 legacy=0x7fdb44069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.582+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2699499913 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb440c94a0 con 0x7fdb440c9ca0 2023-10-16T13:50:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.584+0000 7fdb4a91a640 1 --2- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb440ca8f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009870 tx=0x7fdb3c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad70ad124b253df2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.584+0000 7fdb49117640 1 -- 172.21.15.60:0/2699499913 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c030840 con 0x7fdb440c9ca0 2023-10-16T13:50:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.584+0000 7fdb49117640 1 -- 172.21.15.60:0/2699499913 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3c0309a0 con 0x7fdb440c9ca0 2023-10-16T13:50:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.584+0000 7fdb49117640 1 -- 172.21.15.60:0/2699499913 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c030d30 con 0x7fdb440c9ca0 2023-10-16T13:50:39.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 msgr2=0x7fdb440ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb440ca8f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009870 tx=0x7fdb3c02b980 comp rx=0 tx=0).stop 2023-10-16T13:50:39.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2699499913 shutdown_connections 2023-10-16T13:50:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb440ca8f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2699499913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb440ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2699499913 >> 172.21.15.60:0/2699499913 conn(0x7fdb440ce9b0 msgr2=0x7fdb440cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2699499913 shutdown_connections 2023-10-16T13:50:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.585+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2699499913 wait complete. 2023-10-16T13:50:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.587+0000 7fdb4c37d640 1 Processor -- start 2023-10-16T13:50:39.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.587+0000 7fdb4c37d640 1 -- start start 2023-10-16T13:50:39.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4c37d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb44159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:39.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4c37d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb4415c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:39.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4c37d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 0x7fdb440c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4c37d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb440c5b20 con 0x7fdb4415ad70 2023-10-16T13:50:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4c37d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4415cd10 con 0x7fdb440c9ca0 2023-10-16T13:50:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4c37d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4415cff0 con 0x7fdb4406a4b0 2023-10-16T13:50:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4a91a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 0x7fdb440c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4a91a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 0x7fdb440c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35778/0 (socket says 172.21.15.60:35778) 2023-10-16T13:50:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.588+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2611706524 learned_addr learned my addr 172.21.15.60:0/2611706524 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb49918640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb4415c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a119640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb44159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 msgr2=0x7fdb44159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a91a640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb44159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 msgr2=0x7fdb4415c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a91a640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb4415c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a91a640 1 -- 172.21.15.60:0/2611706524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0094d0 con 0x7fdb4415ad70 2023-10-16T13:50:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb4a91a640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 0x7fdb440c7b80 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c02b9c0 tx=0x7fdb3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.589+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c039c90 con 0x7fdb4415ad70 2023-10-16T13:50:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.590+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3c039df0 con 0x7fdb4415ad70 2023-10-16T13:50:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.590+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb440c5d90 con 0x7fdb4415ad70 2023-10-16T13:50:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.590+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c043640 con 0x7fdb4415ad70 2023-10-16T13:50:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.590+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4415e210 con 0x7fdb4415ad70 2023-10-16T13:50:39.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.591+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4404e160 con 0x7fdb4415ad70 2023-10-16T13:50:39.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.592+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb3c0437a0 con 0x7fdb4415ad70 2023-10-16T13:50:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.593+0000 7fdb42ffd640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fdb2806af90 0x7fdb2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.593+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fdb3c03e070 con 0x7fdb4415ad70 2023-10-16T13:50:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.593+0000 7fdb4a119640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fdb2806af90 0x7fdb2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.594+0000 7fdb4a119640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fdb2806af90 0x7fdb2806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdb380012e0 tx=0x7fdb38009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.597+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb3c0777e0 con 0x7fdb4415ad70 2023-10-16T13:50:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.792+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdb440ce550 con 0x7fdb4415ad70 2023-10-16T13:50:39.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.797+0000 7fdb42ffd640 1 -- 172.21.15.60:0/2611706524 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdb3c07ae00 con 0x7fdb4415ad70 2023-10-16T13:50:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.799+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fdb2806af90 msgr2=0x7fdb2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.799+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fdb2806af90 0x7fdb2806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdb380012e0 tx=0x7fdb38009040 comp rx=0 tx=0).stop 2023-10-16T13:50:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 msgr2=0x7fdb440c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 0x7fdb440c7b80 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c02b9c0 tx=0x7fdb3c004730 comp rx=0 tx=0).stop 2023-10-16T13:50:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 shutdown_connections 2023-10-16T13:50:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fdb2806af90 0x7fdb2806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb4415ad70 0x7fdb440c7b80 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb440c9ca0 0x7fdb4415c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 --2- 172.21.15.60:0/2611706524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb4406a4b0 0x7fdb44159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 >> 172.21.15.60:0/2611706524 conn(0x7fdb440ce9b0 msgr2=0x7fdb440ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 shutdown_connections 2023-10-16T13:50:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:39.800+0000 7fdb4c37d640 1 -- 172.21.15.60:0/2611706524 wait complete. 2023-10-16T13:50:39.823 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:50:39.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:50:39.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:50:40.069 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:50:40.070 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:50:40.070 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:40.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T13:50:40.137 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:50:40.137 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:40.197 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:50:40.197 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:50:40.198 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:40.262 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T13:50:40.263 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:50:40.263 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:40.325 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:50:40.325 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:50:40.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.560+0000 7f926bfff640 1 Processor -- start 2023-10-16T13:50:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.561+0000 7f926bfff640 1 -- start start 2023-10-16T13:50:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.562+0000 7f926bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f926c15cfc0 0x7f926c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.562+0000 7f926bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f926c0d2640 0x7f926c0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:40.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.562+0000 7f926bfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0d15a0 con 0x7f926c15cbe0 2023-10-16T13:50:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926bfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0d1880 con 0x7f926c0d2640 2023-10-16T13:50:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926bfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0d1b60 con 0x7f926c15cfc0 2023-10-16T13:50:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f926c15cbe0 0x7f926c0d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39434/0 (socket says 172.21.15.60:39434) 2023-10-16T13:50:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926affd640 1 -- 172.21.15.60:0/1773796079 learned_addr learned my addr 172.21.15.60:0/1773796079 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926b7fe640 1 --2- 172.21.15.60:0/1773796079 >> v2:172.21.15.60:3300/0 conn(0x7f926c0d2640 0x7f926c0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f92627fc640 1 --2- 172.21.15.60:0/1773796079 >> v2:172.21.15.144:3300/0 conn(0x7f926c15cfc0 0x7f926c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:40.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926a7fc640 1 -- 172.21.15.60:0/1773796079 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934543936 0 0) 0x7f926c0d15a0 con 0x7f926c15cbe0 2023-10-16T13:50:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.563+0000 7f926a7fc640 1 -- 172.21.15.60:0/1773796079 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9258003520 con 0x7f926c15cbe0 2023-10-16T13:50:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.564+0000 7f926b7fe640 1 -- 172.21.15.60:0/1773796079 >> v2:172.21.15.144:3300/0 conn(0x7f926c15cfc0 msgr2=0x7f926c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.564+0000 7f926b7fe640 1 --2- 172.21.15.60:0/1773796079 >> v2:172.21.15.144:3300/0 conn(0x7f926c15cfc0 0x7f926c0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.564+0000 7f926b7fe640 1 -- 172.21.15.60:0/1773796079 >> v1:172.21.15.60:6789/0 conn(0x7f926c15cbe0 legacy=0x7f926c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:40.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.564+0000 7f926b7fe640 1 -- 172.21.15.60:0/1773796079 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0d1e40 con 0x7f926c0d2640 2023-10-16T13:50:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.564+0000 7f92627fc640 1 --2- 172.21.15.60:0/1773796079 >> v2:172.21.15.144:3300/0 conn(0x7f926c15cfc0 0x7f926c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.565+0000 7f926b7fe640 1 --2- 172.21.15.60:0/1773796079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c0d2640 0x7f926c0b4ab0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f926c0cf4b0 tx=0x7f926402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8803bb973a5848 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.565+0000 7f926a7fc640 1 -- 172.21.15.60:0/1773796079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264030610 con 0x7f926c0d2640 2023-10-16T13:50:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.565+0000 7f926a7fc640 1 -- 172.21.15.60:0/1773796079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9264030770 con 0x7f926c0d2640 2023-10-16T13:50:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.565+0000 7f926a7fc640 1 -- 172.21.15.60:0/1773796079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264030b00 con 0x7f926c0d2640 2023-10-16T13:50:40.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 -- 172.21.15.60:0/1773796079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c0d2640 msgr2=0x7f926c0b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 --2- 172.21.15.60:0/1773796079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c0d2640 0x7f926c0b4ab0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f926c0cf4b0 tx=0x7f926402b960 comp rx=0 tx=0).stop 2023-10-16T13:50:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 -- 172.21.15.60:0/1773796079 shutdown_connections 2023-10-16T13:50:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 --2- 172.21.15.60:0/1773796079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c0d2640 0x7f926c0b4ab0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 --2- 172.21.15.60:0/1773796079 >> v2:172.21.15.144:3300/0 conn(0x7f926c15cfc0 0x7f926c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 -- 172.21.15.60:0/1773796079 >> 172.21.15.60:0/1773796079 conn(0x7f926c0cf570 msgr2=0x7f926c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 -- 172.21.15.60:0/1773796079 shutdown_connections 2023-10-16T13:50:40.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.566+0000 7f926bfff640 1 -- 172.21.15.60:0/1773796079 wait complete. 2023-10-16T13:50:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.568+0000 7f926bfff640 1 Processor -- start 2023-10-16T13:50:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.568+0000 7f926bfff640 1 -- start start 2023-10-16T13:50:40.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926c0d2640 0x7f926c1f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 0x7f926c0cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f926c15cfc0 0x7f926c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0aabc0 con 0x7f926c15cbe0 2023-10-16T13:50:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0aaea0 con 0x7f926c15cfc0 2023-10-16T13:50:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0ab180 con 0x7f926c0d2640 2023-10-16T13:50:40.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 0x7f926c0cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 0x7f926c0cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35792/0 (socket says 172.21.15.60:35792) 2023-10-16T13:50:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 -- 172.21.15.60:0/579543502 learned_addr learned my addr 172.21.15.60:0/579543502 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926affd640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926c0d2640 0x7f926c1f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 -- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926c0d2640 msgr2=0x7f926c1f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:40.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f926b7fe640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f926c15cfc0 0x7f926c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926c0d2640 0x7f926c1f1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 -- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f926c15cfc0 msgr2=0x7f926c0cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f926c15cfc0 0x7f926c0cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.569+0000 7f92627fc640 1 -- 172.21.15.60:0/579543502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264009400 con 0x7f926c15cbe0 2023-10-16T13:50:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f926b7fe640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f926c15cfc0 0x7f926c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f92627fc640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 0x7f926c0cbb80 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f925400d800 tx=0x7f925400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254004030 con 0x7f926c15cbe0 2023-10-16T13:50:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9254004190 con 0x7f926c15cbe0 2023-10-16T13:50:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925400b6c0 con 0x7f926c15cbe0 2023-10-16T13:50:40.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c0ab460 con 0x7f926c15cbe0 2023-10-16T13:50:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.570+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c1f1dc0 con 0x7f926c15cbe0 2023-10-16T13:50:40.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.571+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c0d38b0 con 0x7f926c15cbe0 2023-10-16T13:50:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.572+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f925400b820 con 0x7f926c15cbe0 2023-10-16T13:50:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.573+0000 7f9268ff9640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f924c06afc0 0x7f924c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.573+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f9254089970 con 0x7f926c15cbe0 2023-10-16T13:50:40.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.574+0000 7f926affd640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f924c06afc0 0x7f924c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.575+0000 7f926affd640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f924c06afc0 0x7f924c06d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f925c001560 tx=0x7f925c009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:40.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.578+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9254054830 con 0x7f926c15cbe0 2023-10-16T13:50:40.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.813+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f926c0d11e0 con 0x7f926c15cbe0 2023-10-16T13:50:40.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.815+0000 7f9268ff9640 1 -- 172.21.15.60:0/579543502 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f9254057e50 con 0x7f926c15cbe0 2023-10-16T13:50:40.818 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:40.818 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/665569261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":665569261},{"type":"v1","addr":"172.21.15.144:6837","nonce":665569261}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/3103270551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3103270551},{"type":"v1","addr":"172.21.15.144:6835","nonce":3103270551}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":7396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1416960995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1416960995},{"type":"v1","addr":"172.21.15.60:6835","nonce":1416960995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":9396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2363318527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2363318527},{"type":"v1","addr":"172.21.15.60:6833","nonce":2363318527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[]} 2023-10-16T13:50:40.819 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 81 2023-10-16T13:50:40.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f924c06afc0 msgr2=0x7f924c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:40.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f924c06afc0 0x7f924c06d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f925c001560 tx=0x7f925c009040 comp rx=0 tx=0).stop 2023-10-16T13:50:40.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 msgr2=0x7f926c0cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:40.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 0x7f926c0cbb80 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f925400d800 tx=0x7f925400dcc0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 shutdown_connections 2023-10-16T13:50:40.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f924c06afc0 0x7f924c06d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f926c15cfc0 0x7f926c0cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f926c15cbe0 0x7f926c0cbb80 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 --2- 172.21.15.60:0/579543502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f926c0d2640 0x7f926c1f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:40.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.817+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 >> 172.21.15.60:0/579543502 conn(0x7f926c0cf570 msgr2=0x7f926c15c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:40.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.818+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 shutdown_connections 2023-10-16T13:50:40.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:40.818+0000 7f926bfff640 1 -- 172.21.15.60:0/579543502 wait complete. 2023-10-16T13:50:40.841 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:50:40.842 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:50:40.842 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:50:40.842 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:40.843 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:50:40.843 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:50:40.843 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:50:40.843 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:40.844 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:50:40.844 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:50:40.844 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:50:40.845 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:40.845 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:50:40.845 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:50:40.845 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:50:40.846 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:40.861 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:50:40.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:50:40.865 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:50:40.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:50:40.867 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:50:40.868 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:50:40.868 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:50:40.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:50:40.914 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:50:40.914 DEBUG:teuthology.parallel:result is None 2023-10-16T13:50:40.917 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:50:40.917 DEBUG:teuthology.parallel:result is None 2023-10-16T13:50:40.922 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:50:40.922 DEBUG:teuthology.parallel:result is None 2023-10-16T13:50:40.924 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:50:40.924 DEBUG:teuthology.parallel:result is None 2023-10-16T13:50:40.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:50:41.078 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T13:50:41.079 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T13:50:41.084 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T13:50:41.085 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T13:50:41.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.113+0000 7f1a20e56640 1 Processor -- start 2023-10-16T13:50:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.113+0000 7f1a20e56640 1 -- start start 2023-10-16T13:50:41.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a20e56640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a1c06a4b0 0x7f1a1c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:41.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a20e56640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c0c9ca0 0x7f1a1c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:41.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a20e56640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0cb030 con 0x7f1a1c15ad70 2023-10-16T13:50:41.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a20e56640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c0cae30 con 0x7f1a1c06a4b0 2023-10-16T13:50:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a20e56640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0c91c0 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a1c06a4b0 0x7f1a1c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a1c06a4b0 0x7f1a1c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35826/0 (socket says 172.21.15.60:35826) 2023-10-16T13:50:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 -- 172.21.15.60:0/992651517 learned_addr learned my addr 172.21.15.60:0/992651517 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a1ad9d640 1 --2- 172.21.15.60:0/992651517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c0c9ca0 0x7f1a1c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 -- 172.21.15.60:0/992651517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c0c9ca0 msgr2=0x7f1a1c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 --2- 172.21.15.60:0/992651517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c0c9ca0 0x7f1a1c0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 -- 172.21.15.60:0/992651517 >> v1:172.21.15.144:6789/0 conn(0x7f1a1c15ad70 legacy=0x7f1a1c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a13fff640 1 -- 172.21.15.60:0/992651517 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0c94a0 con 0x7f1a1c06a4b0 2023-10-16T13:50:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.115+0000 7f1a19d9b640 1 -- 172.21.15.60:0/992651517 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338360011 0 0) 0x7f1a1c0cb030 con 0x7f1a1c15ad70 2023-10-16T13:50:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.117+0000 7f1a13fff640 1 --2- 172.21.15.60:0/992651517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c06a4b0 0x7f1a1c0ca3b0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009870 tx=0x7f1a0402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc373b95ebb368e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.117+0000 7f1a19d9b640 1 -- 172.21.15.60:0/992651517 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04030840 con 0x7f1a1c06a4b0 2023-10-16T13:50:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.117+0000 7f1a19d9b640 1 -- 172.21.15.60:0/992651517 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a040309a0 con 0x7f1a1c06a4b0 2023-10-16T13:50:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.117+0000 7f1a19d9b640 1 -- 172.21.15.60:0/992651517 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04030d30 con 0x7f1a1c06a4b0 2023-10-16T13:50:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 -- 172.21.15.60:0/992651517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c06a4b0 msgr2=0x7f1a1c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 --2- 172.21.15.60:0/992651517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c06a4b0 0x7f1a1c0ca3b0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009870 tx=0x7f1a0402b980 comp rx=0 tx=0).stop 2023-10-16T13:50:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 -- 172.21.15.60:0/992651517 shutdown_connections 2023-10-16T13:50:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 --2- 172.21.15.60:0/992651517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c0c9ca0 0x7f1a1c0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 --2- 172.21.15.60:0/992651517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c06a4b0 0x7f1a1c0ca3b0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 -- 172.21.15.60:0/992651517 >> 172.21.15.60:0/992651517 conn(0x7f1a1c0ce9b0 msgr2=0x7f1a1c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 -- 172.21.15.60:0/992651517 shutdown_connections 2023-10-16T13:50:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.118+0000 7f1a20e56640 1 -- 172.21.15.60:0/992651517 wait complete. 2023-10-16T13:50:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.119+0000 7f1a20e56640 1 Processor -- start 2023-10-16T13:50:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.120+0000 7f1a20e56640 1 -- start start 2023-10-16T13:50:41.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a20e56640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c06a4b0 0x7f1a1c0af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:41.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a20e56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 0x7f1a1c1f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:41.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a20e56640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a1c15ad70 0x7f1a1c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:41.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a20e56640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0acc70 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a20e56640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0acf50 con 0x7f1a1c15ad70 2023-10-16T13:50:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a20e56640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0ad230 con 0x7f1a1c06a4b0 2023-10-16T13:50:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 0x7f1a1c1f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 0x7f1a1c1f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35834/0 (socket says 172.21.15.60:35834) 2023-10-16T13:50:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 -- 172.21.15.60:0/2139094881 learned_addr learned my addr 172.21.15.60:0/2139094881 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a1ad9d640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a1c15ad70 0x7f1a1c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a1a59c640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c06a4b0 0x7f1a1c0af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 -- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c06a4b0 msgr2=0x7f1a1c0af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c06a4b0 0x7f1a1c0af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 -- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a1c15ad70 msgr2=0x7f1a1c1f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a1c15ad70 0x7f1a1c1f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.121+0000 7f1a13fff640 1 -- 172.21.15.60:0/2139094881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a040094d0 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.122+0000 7f1a13fff640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 0x7f1a1c1f5a60 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f1a04000c00 tx=0x7f1a04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.122+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04039c90 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.122+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04039df0 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.122+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04043640 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.122+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c0ad510 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.122+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c0ac890 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.123+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c0ada30 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.124+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1a040437a0 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.125+0000 7f1a12ffd640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f19fc06af90 0x7f19fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.125+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f1a0403e070 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.126+0000 7f1a1a59c640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f19fc06af90 0x7f19fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.126+0000 7f1a1a59c640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f19fc06af90 0x7f19fc06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c000f60 tx=0x7f1a0c009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:41.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.130+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a1c0ada30 con 0x7f1a1c0c9ca0 2023-10-16T13:50:41.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:41.344+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1a1c0ada30 con 0x7f1a1c0c9ca0 2023-10-16T13:50:42.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.333+0000 7f1a12ffd640 1 -- 172.21.15.60:0/2139094881 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v179) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1a1c0ada30 con 0x7f1a1c0c9ca0 2023-10-16T13:50:42.337 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T13:50:42.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.336+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f19fc06af90 msgr2=0x7f19fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.336+0000 7f1a20e56640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f19fc06af90 0x7f19fc06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c000f60 tx=0x7f1a0c009040 comp rx=0 tx=0).stop 2023-10-16T13:50:42.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.336+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 msgr2=0x7f1a1c1f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.336+0000 7f1a20e56640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 0x7f1a1c1f5a60 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f1a04000c00 tx=0x7f1a04004730 comp rx=0 tx=0).stop 2023-10-16T13:50:42.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 shutdown_connections 2023-10-16T13:50:42.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f19fc06af90 0x7f19fc06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a1c15ad70 0x7f1a1c1f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a1c0c9ca0 0x7f1a1c1f5a60 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 --2- 172.21.15.60:0/2139094881 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a1c06a4b0 0x7f1a1c0af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 >> 172.21.15.60:0/2139094881 conn(0x7f1a1c0ce9b0 msgr2=0x7f1a1c0cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:42.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 shutdown_connections 2023-10-16T13:50:42.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.337+0000 7f1a20e56640 1 -- 172.21.15.60:0/2139094881 wait complete. 2023-10-16T13:50:42.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:50:42.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.587+0000 7f75d0cdf640 1 Processor -- start 2023-10-16T13:50:42.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.587+0000 7f75d0cdf640 1 -- start start 2023-10-16T13:50:42.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.589+0000 7f75d0cdf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75cc0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.589+0000 7f75d0cdf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc160a90 con 0x7f75cc15c9f0 2023-10-16T13:50:42.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.589+0000 7f75d0cdf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0cd900 con 0x7f75cc15dde0 2023-10-16T13:50:42.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.589+0000 7f75d0cdf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0cdec0 con 0x7f75cc0ac890 2023-10-16T13:50:42.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.589+0000 7f75cb7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75cc15c9f0 0x7f75cc0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39464/0 (socket says 172.21.15.60:39464) 2023-10-16T13:50:42.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.589+0000 7f75cb7fe640 1 -- 172.21.15.60:0/651074171 learned_addr learned my addr 172.21.15.60:0/651074171 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:42.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75cbfff640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75cc0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:42.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183207175 0 0) 0x7f75cc160a90 con 0x7f75cc15c9f0 2023-10-16T13:50:42.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b4003520 con 0x7f75cc15c9f0 2023-10-16T13:50:42.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750164783 0 0) 0x7f75b4003520 con 0x7f75cc15c9f0 2023-10-16T13:50:42.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 msgr2=0x7f75cc0acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:42.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75cc0acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 >> v1:172.21.15.144:6789/0 conn(0x7f75cc15dde0 legacy=0x7f75cc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0cdbe0 con 0x7f75cc15c9f0 2023-10-16T13:50:42.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f75bc001670 con 0x7f75cc15c9f0 2023-10-16T13:50:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.590+0000 7f75cbfff640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75cc0acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:50:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 >> v1:172.21.15.60:6789/0 conn(0x7f75cc15c9f0 legacy=0x7f75cc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b40039a0 0x7f75b4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b400b980 con 0x7f75b4004d00 2023-10-16T13:50:42.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b400bac0 con 0x7f75b40039a0 2023-10-16T13:50:42.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b400bc30 con 0x7f75cc0ac890 2023-10-16T13:50:42.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cbfff640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:42.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75caffd640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:42.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cb7fe640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b40039a0 0x7f75b4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:42.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cbfff640 1 -- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 msgr2=0x7f75b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:42.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cbfff640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc0ac890 0x7f75b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cbfff640 1 -- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b40039a0 msgr2=0x7f75b4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:42.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cbfff640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b40039a0 0x7f75b4003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.591+0000 7f75cbfff640 1 -- 172.21.15.60:0/651074171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c4009a70 con 0x7f75b4004d00 2023-10-16T13:50:42.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.592+0000 7f75cb7fe640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b40039a0 0x7f75b4003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:42.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75cbfff640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f75c400e350 tx=0x7f75b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6aef263b49eba05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:42.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c402cd70 con 0x7f75b4004d00 2023-10-16T13:50:42.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c400f040 con 0x7f75b4004d00 2023-10-16T13:50:42.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75ca7fc640 1 -- 172.21.15.60:0/651074171 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4034640 con 0x7f75b4004d00 2023-10-16T13:50:42.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75d0cdf640 1 -- 172.21.15.60:0/651074171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 msgr2=0x7f75b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/651074171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75b4005180 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f75c400e350 tx=0x7f75b40057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.593+0000 7f75cbfff640 1 -- 172.21.15.60:0/651074171 reap_dead start 2023-10-16T13:50:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.594+0000 7f75d0cdf640 1 -- 172.21.15.60:0/651074171 shutdown_connections 2023-10-16T13:50:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.594+0000 7f75d0cdf640 1 -- 172.21.15.60:0/651074171 >> 172.21.15.60:0/651074171 conn(0x7f75cc0ce6a0 msgr2=0x7f75cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.594+0000 7f75d0cdf640 1 -- 172.21.15.60:0/651074171 shutdown_connections 2023-10-16T13:50:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.594+0000 7f75d0cdf640 1 -- 172.21.15.60:0/651074171 wait complete. 2023-10-16T13:50:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.595+0000 7f75d0cdf640 1 Processor -- start 2023-10-16T13:50:42.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.595+0000 7f75d0cdf640 1 -- start start 2023-10-16T13:50:42.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.596+0000 7f75d0cdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75cc158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.596+0000 7f75d0cdf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc15f400 0x7f75cc15f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.596+0000 7f75d0cdf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75cc0c3c80 0x7f75cc0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.596+0000 7f75d0cdf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc15fdc0 con 0x7f75b4004d00 2023-10-16T13:50:42.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.596+0000 7f75d0cdf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc159190 con 0x7f75cc0c3c80 2023-10-16T13:50:42.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.596+0000 7f75d0cdf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0ae1a0 con 0x7f75cc15f400 2023-10-16T13:50:42.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75cc158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:42.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75cc158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35854/0 (socket says 172.21.15.60:35854) 2023-10-16T13:50:42.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 -- 172.21.15.60:0/371181069 learned_addr learned my addr 172.21.15.60:0/371181069 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:42.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 -- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc15f400 msgr2=0x7f75cc15f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc15f400 0x7f75cc15f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 -- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75cc0c3c80 msgr2=0x7f75cc0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75cc0c3c80 0x7f75cc0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 -- 172.21.15.60:0/371181069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0ae410 con 0x7f75b4004d00 2023-10-16T13:50:42.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.597+0000 7f75cb7fe640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75cc158c50 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00d8b0 tx=0x7f75bc00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:42.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.598+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0074e0 con 0x7f75b4004d00 2023-10-16T13:50:42.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.598+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc007640 con 0x7f75b4004d00 2023-10-16T13:50:42.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.598+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0049e0 con 0x7f75b4004d00 2023-10-16T13:50:42.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.598+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc0ae6f0 con 0x7f75b4004d00 2023-10-16T13:50:42.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.598+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc0c4a40 con 0x7f75b4004d00 2023-10-16T13:50:42.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.599+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc04e9b0 con 0x7f75b4004d00 2023-10-16T13:50:42.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.600+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75bc004b40 con 0x7f75b4004d00 2023-10-16T13:50:42.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.601+0000 7f75b3fff640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f75b806af90 0x7f75b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:42.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.601+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f75bc088eb0 con 0x7f75b4004d00 2023-10-16T13:50:42.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.601+0000 7f75caffd640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f75b806af90 0x7f75b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:42.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.603+0000 7f75caffd640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f75b806af90 0x7f75b806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75c0007fd0 tx=0x7f75c0009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:42.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.605+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75bc054080 con 0x7f75b4004d00 2023-10-16T13:50:42.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.874+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f75cc0adac0 con 0x7f75b4004d00 2023-10-16T13:50:42.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.875+0000 7f75b3fff640 1 -- 172.21.15.60:0/371181069 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f75bc019090 con 0x7f75b4004d00 2023-10-16T13:50:42.879 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:42.879 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T13:50:42.879 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T13:50:42.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T13:50:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T13:50:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:50:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:50:42.881 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.881 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.881 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T13:50:42.882 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T13:50:42.882 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.882 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.882 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:50:42.882 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:50:42.883 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.883 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.883 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T13:50:42.884 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T13:50:42.884 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.884 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.884 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:50:42.884 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:50:42.884 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.885 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.885 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T13:50:42.885 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T13:50:42.885 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.886 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.886 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:50:42.886 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:50:42.886 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.887 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T13:50:42.887 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T13:50:42.887 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.887 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.888 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.888 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.888 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.889 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T13:50:42.889 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T13:50:42.889 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.889 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.889 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.889 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.890 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.890 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T13:50:42.890 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T13:50:42.890 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.891 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.891 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.891 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.892 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T13:50:42.892 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T13:50:42.892 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.892 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.892 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.893 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.893 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T13:50:42.893 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.896 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.896 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T13:50:42.896 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T13:50:42.897 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.897 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.897 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.897 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.898 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T13:50:42.898 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T13:50:42.898 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.898 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:50:42.898 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:50:42.899 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.899 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.899 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.902 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T13:50:42.902 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T13:50:42.902 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T13:50:42.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.903 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T13:50:42.903 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T13:50:42.903 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.903 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.903 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T13:50:42.904 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T13:50:42.905 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.905 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:50:42.905 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T13:50:42.905 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T13:50:42.905 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T13:50:42.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.906 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T13:50:42.906 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T13:50:42.906 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.908 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T13:50:42.908 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T13:50:42.908 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.908 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T13:50:42.908 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T13:50:42.910 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:50:42.910 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T13:50:42.910 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T13:50:42.910 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:50:42.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:42.911 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T13:50:42.911 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T13:50:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f75b806af90 msgr2=0x7f75b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f75b806af90 0x7f75b806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75c0007fd0 tx=0x7f75c0009040 comp rx=0 tx=0).stop 2023-10-16T13:50:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 msgr2=0x7f75cc158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75cc158c50 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00d8b0 tx=0x7f75bc00dd70 comp rx=0 tx=0).stop 2023-10-16T13:50:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 shutdown_connections 2023-10-16T13:50:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f75b806af90 0x7f75b806d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75cc15f400 0x7f75cc15f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75cc0c3c80 0x7f75cc0c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 --2- 172.21.15.60:0/371181069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75b4004d00 0x7f75cc158c50 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 >> 172.21.15.60:0/371181069 conn(0x7f75cc0ce6a0 msgr2=0x7f75cc06bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 shutdown_connections 2023-10-16T13:50:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:42.878+0000 7f75d0cdf640 1 -- 172.21.15.60:0/371181069 wait complete. 2023-10-16T13:50:42.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:43.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.139+0000 7fc3fe07c640 1 Processor -- start 2023-10-16T13:50:43.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.139+0000 7fc3fe07c640 1 -- start start 2023-10-16T13:50:43.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3fe07c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc3f806c3f0 0x7fc3f806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3fe07c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3f815e540 0x7fc3f80b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3fe07c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80cd630 con 0x7fc3f815e0c0 2023-10-16T13:50:43.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3fe07c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f80ce1b0 con 0x7fc3f815e540 2023-10-16T13:50:43.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3fe07c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f806cd10 con 0x7fc3f806c3f0 2023-10-16T13:50:43.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3f7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3f815e540 0x7fc3f80b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3f7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3f815e540 0x7fc3f80b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35860/0 (socket says 172.21.15.60:35860) 2023-10-16T13:50:43.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3f7fff640 1 -- 172.21.15.60:0/673235578 learned_addr learned my addr 172.21.15.60:0/673235578 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:43.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.141+0000 7fc3f77fe640 1 --2- 172.21.15.60:0/673235578 >> v2:172.21.15.144:3300/0 conn(0x7fc3f806c3f0 0x7fc3f806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.142+0000 7fc3f7fff640 1 -- 172.21.15.60:0/673235578 >> v2:172.21.15.144:3300/0 conn(0x7fc3f806c3f0 msgr2=0x7fc3f806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:43.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.142+0000 7fc3f7fff640 1 --2- 172.21.15.60:0/673235578 >> v2:172.21.15.144:3300/0 conn(0x7fc3f806c3f0 0x7fc3f806c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.142+0000 7fc3f7fff640 1 -- 172.21.15.60:0/673235578 >> v1:172.21.15.144:6789/0 conn(0x7fc3f815e0c0 legacy=0x7fc3f80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:43.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.142+0000 7fc3f7fff640 1 -- 172.21.15.60:0/673235578 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80d2170 con 0x7fc3f815e540 2023-10-16T13:50:43.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.144+0000 7fc3f7fff640 1 --2- 172.21.15.60:0/673235578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e540 0x7fc3f80b1710 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80d3d00 tx=0x7fc3f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3703f174ebb3668d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:43.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.144+0000 7fc3f67fc640 1 -- 172.21.15.60:0/673235578 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0031870 con 0x7fc3f815e540 2023-10-16T13:50:43.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.144+0000 7fc3f67fc640 1 -- 172.21.15.60:0/673235578 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f00319d0 con 0x7fc3f815e540 2023-10-16T13:50:43.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.144+0000 7fc3f67fc640 1 -- 172.21.15.60:0/673235578 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0031d60 con 0x7fc3f815e540 2023-10-16T13:50:43.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 -- 172.21.15.60:0/673235578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e540 msgr2=0x7fc3f80b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/673235578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e540 0x7fc3f80b1710 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80d3d00 tx=0x7fc3f002b910 comp rx=0 tx=0).stop 2023-10-16T13:50:43.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 -- 172.21.15.60:0/673235578 shutdown_connections 2023-10-16T13:50:43.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/673235578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e540 0x7fc3f80b1710 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/673235578 >> v2:172.21.15.144:3300/0 conn(0x7fc3f806c3f0 0x7fc3f806c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 -- 172.21.15.60:0/673235578 >> 172.21.15.60:0/673235578 conn(0x7fc3f80d2cb0 msgr2=0x7fc3f80d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:43.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 -- 172.21.15.60:0/673235578 shutdown_connections 2023-10-16T13:50:43.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.145+0000 7fc3fe07c640 1 -- 172.21.15.60:0/673235578 wait complete. 2023-10-16T13:50:43.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.147+0000 7fc3fe07c640 1 Processor -- start 2023-10-16T13:50:43.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.147+0000 7fc3fe07c640 1 -- start start 2023-10-16T13:50:43.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3fe07c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f806c3f0 0x7fc3f81f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3fe07c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 0x7fc3f815a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3fe07c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3f815e540 0x7fc3f815a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3fe07c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f81f5ef0 con 0x7fc3f815e0c0 2023-10-16T13:50:43.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3fe07c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8159820 con 0x7fc3f806c3f0 2023-10-16T13:50:43.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3fe07c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8158c10 con 0x7fc3f815e540 2023-10-16T13:50:43.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3f6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 0x7fc3f815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3f6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 0x7fc3f815a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35866/0 (socket says 172.21.15.60:35866) 2023-10-16T13:50:43.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.148+0000 7fc3f6ffd640 1 -- 172.21.15.60:0/2254477329 learned_addr learned my addr 172.21.15.60:0/2254477329 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:43.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f77fe640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f806c3f0 0x7fc3f81f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f7fff640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3f815e540 0x7fc3f815a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f6ffd640 1 -- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3f815e540 msgr2=0x7fc3f815a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f6ffd640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3f815e540 0x7fc3f815a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f6ffd640 1 -- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f806c3f0 msgr2=0x7fc3f81f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f6ffd640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f806c3f0 0x7fc3f81f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f6ffd640 1 -- 172.21.15.60:0/2254477329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f00094d0 con 0x7fc3f815e0c0 2023-10-16T13:50:43.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f77fe640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f806c3f0 0x7fc3f81f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:43.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f7fff640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3f815e540 0x7fc3f815a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:43.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3f6ffd640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 0x7fc3f815a010 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c7c0 tx=0x7fc3e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:43.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800aa50 con 0x7fc3f815e0c0 2023-10-16T13:50:43.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e800abb0 con 0x7fc3f815e0c0 2023-10-16T13:50:43.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f8158ef0 con 0x7fc3f815e0c0 2023-10-16T13:50:43.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.149+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8012420 con 0x7fc3f815e0c0 2023-10-16T13:50:43.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.150+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f81594b0 con 0x7fc3f815e0c0 2023-10-16T13:50:43.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.151+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f80cdb40 con 0x7fc3f815e0c0 2023-10-16T13:50:43.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.152+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc3e8012580 con 0x7fc3f815e0c0 2023-10-16T13:50:43.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.153+0000 7fc3effff640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fc3d806af90 0x7fc3d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.153+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3e80896e0 con 0x7fc3f815e0c0 2023-10-16T13:50:43.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.154+0000 7fc3f77fe640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fc3d806af90 0x7fc3d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.155+0000 7fc3f77fe640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fc3d806af90 0x7fc3d806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0006fd0 tx=0x7fc3e0008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:43.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.157+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3f80cdb40 con 0x7fc3f815e0c0 2023-10-16T13:50:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.353+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3f80cdb40 con 0x7fc3f815e0c0 2023-10-16T13:50:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.356+0000 7fc3effff640 1 -- 172.21.15.60:0/2254477329 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+12855 (secure 0 0 0) 0x7fc3f80cdb40 con 0x7fc3f815e0c0 2023-10-16T13:50:43.359 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:43.360 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":179,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:42.330212+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:43.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fc3d806af90 msgr2=0x7fc3d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fc3d806af90 0x7fc3d806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0006fd0 tx=0x7fc3e0008040 comp rx=0 tx=0).stop 2023-10-16T13:50:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 msgr2=0x7fc3f815a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 0x7fc3f815a010 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c7c0 tx=0x7fc3e800cc80 comp rx=0 tx=0).stop 2023-10-16T13:50:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 shutdown_connections 2023-10-16T13:50:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fc3d806af90 0x7fc3d806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc3f815e540 0x7fc3f815a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3f815e0c0 0x7fc3f815a010 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 --2- 172.21.15.60:0/2254477329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3f806c3f0 0x7fc3f81f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 >> 172.21.15.60:0/2254477329 conn(0x7fc3f80d2cb0 msgr2=0x7fc3f8157010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 shutdown_connections 2023-10-16T13:50:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.359+0000 7fc3fe07c640 1 -- 172.21.15.60:0/2254477329 wait complete. 2023-10-16T13:50:43.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:50:43.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:50:43.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.600+0000 7fed8ee02640 1 Processor -- start 2023-10-16T13:50:43.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.600+0000 7fed8ee02640 1 -- start start 2023-10-16T13:50:43.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8ee02640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fed880b3ec0 0x7fed880cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8ee02640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b43c0 0x7fed880cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8ee02640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8806be30 con 0x7fed880b0ea0 2023-10-16T13:50:43.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8ee02640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8806ca00 con 0x7fed880b3ec0 2023-10-16T13:50:43.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8ee02640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed880ada10 con 0x7fed880b43c0 2023-10-16T13:50:43.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8de00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fed880b3ec0 0x7fed880cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8d5ff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fed880b0ea0 0x7fed8815a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39474/0 (socket says 172.21.15.60:39474) 2023-10-16T13:50:43.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.602+0000 7fed8d5ff640 1 -- 172.21.15.60:0/1969884147 learned_addr learned my addr 172.21.15.60:0/1969884147 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:43.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8e601640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b43c0 0x7fed880cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8de00640 1 -- 172.21.15.60:0/1969884147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b43c0 msgr2=0x7fed880cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8cdfe640 1 -- 172.21.15.60:0/1969884147 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750514414 0 0) 0x7fed8806be30 con 0x7fed880b0ea0 2023-10-16T13:50:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8de00640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b43c0 0x7fed880cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8de00640 1 -- 172.21.15.60:0/1969884147 >> v1:172.21.15.60:6789/0 conn(0x7fed880b0ea0 legacy=0x7fed8815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8de00640 1 -- 172.21.15.60:0/1969884147 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8806c060 con 0x7fed880b3ec0 2023-10-16T13:50:43.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.603+0000 7fed8e601640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b43c0 0x7fed880cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:43.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.604+0000 7fed8de00640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b3ec0 0x7fed880cd260 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fed840097a0 tx=0x7fed8402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed6ec66c51b68d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:43.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.604+0000 7fed8cdfe640 1 -- 172.21.15.60:0/1969884147 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84031780 con 0x7fed880b3ec0 2023-10-16T13:50:43.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8cdfe640 1 -- 172.21.15.60:0/1969884147 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed840318e0 con 0x7fed880b3ec0 2023-10-16T13:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8cdfe640 1 -- 172.21.15.60:0/1969884147 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84031c70 con 0x7fed880b3ec0 2023-10-16T13:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 -- 172.21.15.60:0/1969884147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b3ec0 msgr2=0x7fed880cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b3ec0 0x7fed880cd260 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fed840097a0 tx=0x7fed8402b820 comp rx=0 tx=0).stop 2023-10-16T13:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 -- 172.21.15.60:0/1969884147 shutdown_connections 2023-10-16T13:50:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b43c0 0x7fed880cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 --2- 172.21.15.60:0/1969884147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b3ec0 0x7fed880cd260 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 -- 172.21.15.60:0/1969884147 >> 172.21.15.60:0/1969884147 conn(0x7fed880b0170 msgr2=0x7fed880b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 -- 172.21.15.60:0/1969884147 shutdown_connections 2023-10-16T13:50:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.605+0000 7fed8ee02640 1 -- 172.21.15.60:0/1969884147 wait complete. 2023-10-16T13:50:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.608+0000 7fed8ee02640 1 Processor -- start 2023-10-16T13:50:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.608+0000 7fed8ee02640 1 -- start start 2023-10-16T13:50:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8ee02640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b0ea0 0x7fed880c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8ee02640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed880b3ec0 0x7fed881f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8ee02640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 0x7fed880af140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8ee02640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880aecb0 con 0x7fed880b43c0 2023-10-16T13:50:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8ee02640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880ae0c0 con 0x7fed880b3ec0 2023-10-16T13:50:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8ee02640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed880ae380 con 0x7fed880b0ea0 2023-10-16T13:50:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.609+0000 7fed8e601640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 0x7fed880af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 0x7fed880af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35892/0 (socket says 172.21.15.60:35892) 2023-10-16T13:50:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 -- 172.21.15.60:0/529877344 learned_addr learned my addr 172.21.15.60:0/529877344 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8de00640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b0ea0 0x7fed880c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8d5ff640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed880b3ec0 0x7fed881f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 -- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b0ea0 msgr2=0x7fed880c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b0ea0 0x7fed880c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 -- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed880b3ec0 msgr2=0x7fed881f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed880b3ec0 0x7fed881f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 -- 172.21.15.60:0/529877344 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84009400 con 0x7fed880b43c0 2023-10-16T13:50:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.610+0000 7fed8e601640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 0x7fed880af140 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fed8000cea0 tx=0x7fed8000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.611+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80003b90 con 0x7fed880b43c0 2023-10-16T13:50:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.611+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80003d10 con 0x7fed880b43c0 2023-10-16T13:50:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.611+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80004260 con 0x7fed880b43c0 2023-10-16T13:50:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.611+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed880ae660 con 0x7fed880b43c0 2023-10-16T13:50:43.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.611+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed880c5430 con 0x7fed880b43c0 2023-10-16T13:50:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.612+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed880aee60 con 0x7fed880b43c0 2023-10-16T13:50:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.613+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed800043c0 con 0x7fed880b43c0 2023-10-16T13:50:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.614+0000 7fed7e7fc640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fed6c06afc0 0x7fed6c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.615+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fed80089db0 con 0x7fed880b43c0 2023-10-16T13:50:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.615+0000 7fed8de00640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fed6c06afc0 0x7fed6c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.617+0000 7fed8de00640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fed6c06afc0 0x7fed6c06d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fed84036000 tx=0x7fed8403d040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.619+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed880aee60 con 0x7fed880b43c0 2023-10-16T13:50:43.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.816+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed8804e9b0 con 0x7fed880b43c0 2023-10-16T13:50:43.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.817+0000 7fed7e7fc640 1 -- 172.21.15.60:0/529877344 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed8008e070 con 0x7fed880b43c0 2023-10-16T13:50:43.821 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:50:43.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fed6c06afc0 msgr2=0x7fed6c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fed6c06afc0 0x7fed6c06d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fed84036000 tx=0x7fed8403d040 comp rx=0 tx=0).stop 2023-10-16T13:50:43.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 msgr2=0x7fed880af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:43.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 0x7fed880af140 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fed8000cea0 tx=0x7fed8000cf90 comp rx=0 tx=0).stop 2023-10-16T13:50:43.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 shutdown_connections 2023-10-16T13:50:43.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fed6c06afc0 0x7fed6c06d480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed880b43c0 0x7fed880af140 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed880b3ec0 0x7fed881f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 --2- 172.21.15.60:0/529877344 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed880b0ea0 0x7fed880c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.820+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 >> 172.21.15.60:0/529877344 conn(0x7fed880b0170 msgr2=0x7fed880b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.821+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 shutdown_connections 2023-10-16T13:50:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:43.821+0000 7fed8ee02640 1 -- 172.21.15.60:0/529877344 wait complete. 2023-10-16T13:50:43.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:50:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.062+0000 7f9d8f3ff640 1 Processor -- start 2023-10-16T13:50:44.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.062+0000 7f9d8f3ff640 1 -- start start 2023-10-16T13:50:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8f3ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9d88070b60 0x7f9d8815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8f3ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8815aef0 con 0x7f9d88159950 2023-10-16T13:50:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8f3ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8815d910 con 0x7f9d8806c100 2023-10-16T13:50:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8f3ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8815e490 con 0x7f9d88070b60 2023-10-16T13:50:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8e3fd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9d88159950 0x7f9d88070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39476/0 (socket says 172.21.15.60:39476) 2023-10-16T13:50:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8e3fd640 1 -- 172.21.15.60:0/962105384 learned_addr learned my addr 172.21.15.60:0/962105384 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8dbfc640 1 --2- 172.21.15.60:0/962105384 >> v2:172.21.15.60:3300/0 conn(0x7f9d88070b60 0x7f9d8815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8d3fb640 1 -- 172.21.15.60:0/962105384 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745084666 0 0) 0x7f9d84001110 con 0x7f9d88159950 2023-10-16T13:50:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.064+0000 7f9d8d3fb640 1 -- 172.21.15.60:0/962105384 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d74003520 con 0x7f9d88159950 2023-10-16T13:50:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.065+0000 7f9d8dbfc640 1 -- 172.21.15.60:0/962105384 >> v1:172.21.15.144:6789/0 conn(0x7f9d8806c100 legacy=0x7f9d8806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.065+0000 7f9d8dbfc640 1 -- 172.21.15.60:0/962105384 >> v1:172.21.15.60:6789/0 conn(0x7f9d88159950 legacy=0x7f9d88070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.065+0000 7f9d8dbfc640 1 -- 172.21.15.60:0/962105384 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8815dbf0 con 0x7f9d88070b60 2023-10-16T13:50:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.066+0000 7f9d8dbfc640 1 --2- 172.21.15.60:0/962105384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88070b60 0x7f9d8815a9b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f9d780097a0 tx=0x7f9d7802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcc970d4e717c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.066+0000 7f9d8d3fb640 1 -- 172.21.15.60:0/962105384 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7802e070 con 0x7f9d88070b60 2023-10-16T13:50:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.066+0000 7f9d8d3fb640 1 -- 172.21.15.60:0/962105384 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d780315a0 con 0x7f9d88070b60 2023-10-16T13:50:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.066+0000 7f9d8d3fb640 1 -- 172.21.15.60:0/962105384 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d78031930 con 0x7f9d88070b60 2023-10-16T13:50:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/962105384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88070b60 msgr2=0x7f9d8815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/962105384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88070b60 0x7f9d8815a9b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f9d780097a0 tx=0x7f9d7802b840 comp rx=0 tx=0).stop 2023-10-16T13:50:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/962105384 shutdown_connections 2023-10-16T13:50:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/962105384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88070b60 0x7f9d8815a9b0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/962105384 >> 172.21.15.60:0/962105384 conn(0x7f9d880ce6a0 msgr2=0x7f9d880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/962105384 shutdown_connections 2023-10-16T13:50:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.067+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/962105384 wait complete. 2023-10-16T13:50:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.069+0000 7f9d8f3ff640 1 Processor -- start 2023-10-16T13:50:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.069+0000 7f9d8f3ff640 1 -- start start 2023-10-16T13:50:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8f3ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8806c100 0x7f9d880c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8f3ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88070b60 0x7f9d88156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8f3ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 0x7f9d880af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8f3ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d880c3d00 con 0x7f9d88159950 2023-10-16T13:50:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8f3ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88157490 con 0x7f9d8806c100 2023-10-16T13:50:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8f3ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d880c2720 con 0x7f9d88070b60 2023-10-16T13:50:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8ebfe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 0x7f9d880af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8ebfe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 0x7f9d880af4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35906/0 (socket says 172.21.15.60:35906) 2023-10-16T13:50:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.070+0000 7f9d8ebfe640 1 -- 172.21.15.60:0/4154849414 learned_addr learned my addr 172.21.15.60:0/4154849414 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8dbfc640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88070b60 0x7f9d88156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8e3fd640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8806c100 0x7f9d880c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8ebfe640 1 -- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88070b60 msgr2=0x7f9d88156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8ebfe640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88070b60 0x7f9d88156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8ebfe640 1 -- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8806c100 msgr2=0x7f9d880c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8ebfe640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8806c100 0x7f9d880c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8ebfe640 1 -- 172.21.15.60:0/4154849414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78009400 con 0x7f9d88159950 2023-10-16T13:50:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8e3fd640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8806c100 0x7f9d880c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d8ebfe640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 0x7f9d880af4d0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c780 tx=0x7f9d8000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.071+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80004b60 con 0x7f9d88159950 2023-10-16T13:50:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.072+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8000ce90 con 0x7f9d88159950 2023-10-16T13:50:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.072+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000a850 con 0x7f9d88159950 2023-10-16T13:50:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.072+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d880c2a00 con 0x7f9d88159950 2023-10-16T13:50:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.072+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d880c2fc0 con 0x7f9d88159950 2023-10-16T13:50:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.073+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8804e9b0 con 0x7f9d88159950 2023-10-16T13:50:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.074+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9d8000a9b0 con 0x7f9d88159950 2023-10-16T13:50:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.075+0000 7f9d7effd640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9d6c06af90 0x7f9d6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.075+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9d800896c0 con 0x7f9d88159950 2023-10-16T13:50:44.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.076+0000 7f9d8e3fd640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9d6c06af90 0x7f9d6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.077+0000 7f9d8e3fd640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9d6c06af90 0x7f9d6c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d84001370 tx=0x7f9d84009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:44.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.080+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d80054910 con 0x7f9d88159950 2023-10-16T13:50:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.280+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9d880cd9f0 con 0x7f9d88159950 2023-10-16T13:50:44.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.281+0000 7f9d7effd640 1 -- 172.21.15.60:0/4154849414 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v179) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9d80057f30 con 0x7f9d88159950 2023-10-16T13:50:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:44.285 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T13:50:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T13:50:44.285 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T13:50:44.286 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:44.286 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T13:50:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9d6c06af90 msgr2=0x7f9d6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9d6c06af90 0x7f9d6c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d84001370 tx=0x7f9d84009040 comp rx=0 tx=0).stop 2023-10-16T13:50:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 msgr2=0x7f9d880af4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 0x7f9d880af4d0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c780 tx=0x7f9d8000cc40 comp rx=0 tx=0).stop 2023-10-16T13:50:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 shutdown_connections 2023-10-16T13:50:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9d6c06af90 0x7f9d6c06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d88070b60 0x7f9d88156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d88159950 0x7f9d880af4d0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 --2- 172.21.15.60:0/4154849414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d8806c100 0x7f9d880c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 >> 172.21.15.60:0/4154849414 conn(0x7f9d880ce6a0 msgr2=0x7f9d88070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 shutdown_connections 2023-10-16T13:50:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.284+0000 7f9d8f3ff640 1 -- 172.21.15.60:0/4154849414 wait complete. 2023-10-16T13:50:44.306 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:50:44.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:50:44.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.527+0000 7f9ce9bdd640 1 Processor -- start 2023-10-16T13:50:44.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.527+0000 7f9ce9bdd640 1 -- start start 2023-10-16T13:50:44.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce9bdd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9ce406d280 0x7f9ce40c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce9bdd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9ce40cae40 0x7f9ce40c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce9bdd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce40c8e70 con 0x7f9ce415ad80 2023-10-16T13:50:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce9bdd640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40b4170 con 0x7f9ce406d280 2023-10-16T13:50:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce9bdd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40adde0 con 0x7f9ce40cae40 2023-10-16T13:50:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9ce406d280 0x7f9ce40c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9ce406d280 0x7f9ce40c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35916/0 (socket says 172.21.15.60:35916) 2023-10-16T13:50:44.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.529+0000 7f9ce3fff640 1 -- 172.21.15.60:0/2562949664 learned_addr learned my addr 172.21.15.60:0/2562949664 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:44.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.530+0000 7f9ce93dc640 1 --2- 172.21.15.60:0/2562949664 >> v2:172.21.15.144:3300/0 conn(0x7f9ce40cae40 0x7f9ce40c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.530+0000 7f9ce3fff640 1 -- 172.21.15.60:0/2562949664 >> v2:172.21.15.144:3300/0 conn(0x7f9ce40cae40 msgr2=0x7f9ce40c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:44.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.530+0000 7f9ce3fff640 1 --2- 172.21.15.60:0/2562949664 >> v2:172.21.15.144:3300/0 conn(0x7f9ce40cae40 0x7f9ce40c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.530+0000 7f9ce3fff640 1 -- 172.21.15.60:0/2562949664 >> v1:172.21.15.144:6789/0 conn(0x7f9ce415ad80 legacy=0x7f9ce406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:44.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.530+0000 7f9ce3fff640 1 -- 172.21.15.60:0/2562949664 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40b4410 con 0x7f9ce406d280 2023-10-16T13:50:44.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.531+0000 7f9ce3fff640 1 --2- 172.21.15.60:0/2562949664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce40c9300 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40cadc0 tx=0x7f9cd002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50f60acb4ed3fec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:44.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce37fe640 1 -- 172.21.15.60:0/2562949664 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0031870 con 0x7f9ce406d280 2023-10-16T13:50:44.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce37fe640 1 -- 172.21.15.60:0/2562949664 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd00319d0 con 0x7f9ce406d280 2023-10-16T13:50:44.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce37fe640 1 -- 172.21.15.60:0/2562949664 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0031d60 con 0x7f9ce406d280 2023-10-16T13:50:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2562949664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 msgr2=0x7f9ce40c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2562949664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce40c9300 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40cadc0 tx=0x7f9cd002b910 comp rx=0 tx=0).stop 2023-10-16T13:50:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2562949664 shutdown_connections 2023-10-16T13:50:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2562949664 >> v2:172.21.15.144:3300/0 conn(0x7f9ce40cae40 0x7f9ce40c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2562949664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce40c9300 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.532+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2562949664 >> 172.21.15.60:0/2562949664 conn(0x7f9ce40ce820 msgr2=0x7f9ce40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.533+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2562949664 shutdown_connections 2023-10-16T13:50:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.533+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2562949664 wait complete. 2023-10-16T13:50:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.535+0000 7f9ce9bdd640 1 Processor -- start 2023-10-16T13:50:44.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.535+0000 7f9ce9bdd640 1 -- start start 2023-10-16T13:50:44.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce9bdd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce41ef670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce9bdd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce40cae40 0x7f9ce4158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce9bdd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce415ad80 0x7f9ce40b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce9bdd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40c3e70 con 0x7f9ce406d280 2023-10-16T13:50:44.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce9bdd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4159190 con 0x7f9ce40cae40 2023-10-16T13:50:44.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce9bdd640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4159470 con 0x7f9ce415ad80 2023-10-16T13:50:44.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce41ef670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce41ef670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35928/0 (socket says 172.21.15.60:35928) 2023-10-16T13:50:44.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 -- 172.21.15.60:0/2796651466 learned_addr learned my addr 172.21.15.60:0/2796651466 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:44.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce3fff640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce40cae40 0x7f9ce4158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce93dc640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce415ad80 0x7f9ce40b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 -- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce415ad80 msgr2=0x7f9ce40b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce415ad80 0x7f9ce40b25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 -- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce40cae40 msgr2=0x7f9ce4158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:44.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.537+0000 7f9ce8bdb640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce40cae40 0x7f9ce4158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:44.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce8bdb640 1 -- 172.21.15.60:0/2796651466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd00094d0 con 0x7f9ce406d280 2023-10-16T13:50:44.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce3fff640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce40cae40 0x7f9ce4158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:44.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce8bdb640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce41ef670 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b3f0 tx=0x7f9cd800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:44.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8011830 con 0x7f9ce406d280 2023-10-16T13:50:44.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd8011990 con 0x7f9ce406d280 2023-10-16T13:50:44.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8011d40 con 0x7f9ce406d280 2023-10-16T13:50:44.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4159750 con 0x7f9ce406d280 2023-10-16T13:50:44.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.538+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce41f33e0 con 0x7f9ce406d280 2023-10-16T13:50:44.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.540+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4159940 con 0x7f9ce406d280 2023-10-16T13:50:44.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.541+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9cd801a400 con 0x7f9ce406d280 2023-10-16T13:50:44.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.542+0000 7f9ce17fa640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9cc806b010 0x7f9cc806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:44.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.542+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9cd80890d0 con 0x7f9ce406d280 2023-10-16T13:50:44.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.542+0000 7f9ce3fff640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9cc806b010 0x7f9cc806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:44.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.544+0000 7f9ce3fff640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9cc806b010 0x7f9cc806d4d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0036000 tx=0x7f9cd003c040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:44.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.547+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9cd8054530 con 0x7f9ce406d280 2023-10-16T13:50:44.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:44.741+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9ce404e9b0 con 0x7f9ce406d280 2023-10-16T13:50:45.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.374+0000 7f9ce17fa640 1 -- 172.21.15.60:0/2796651466 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v180) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9cd8057b50 con 0x7f9ce406d280 2023-10-16T13:50:45.378 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T13:50:45.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.378+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9cc806b010 msgr2=0x7f9cc806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.378+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9cc806b010 0x7f9cc806d4d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0036000 tx=0x7f9cd003c040 comp rx=0 tx=0).stop 2023-10-16T13:50:45.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.378+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 msgr2=0x7f9ce41ef670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.378+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce41ef670 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b3f0 tx=0x7f9cd800b8b0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 shutdown_connections 2023-10-16T13:50:45.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f9cc806b010 0x7f9cc806d4d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9ce415ad80 0x7f9ce40b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ce40cae40 0x7f9ce4158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 --2- 172.21.15.60:0/2796651466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ce406d280 0x7f9ce41ef670 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 >> 172.21.15.60:0/2796651466 conn(0x7f9ce40ce820 msgr2=0x7f9ce40c9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:45.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 shutdown_connections 2023-10-16T13:50:45.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.379+0000 7f9ce9bdd640 1 -- 172.21.15.60:0/2796651466 wait complete. 2023-10-16T13:50:45.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:50:45.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.623+0000 7f1bdc9db640 1 Processor -- start 2023-10-16T13:50:45.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.624+0000 7f1bdc9db640 1 -- start start 2023-10-16T13:50:45.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.625+0000 7f1bdc9db640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd8065ab0 0x7f1bd80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.625+0000 7f1bdc9db640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd80acd70 con 0x7f1bd806d220 2023-10-16T13:50:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.625+0000 7f1bdc9db640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd80ac4a0 con 0x7f1bd80ce280 2023-10-16T13:50:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.625+0000 7f1bdc9db640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd8065f40 con 0x7f1bd8065ab0 2023-10-16T13:50:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.625+0000 7f1bd6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1bd806d220 0x7f1bd80cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39486/0 (socket says 172.21.15.60:39486) 2023-10-16T13:50:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.625+0000 7f1bd6d9d640 1 -- 172.21.15.60:0/1576412936 learned_addr learned my addr 172.21.15.60:0/1576412936 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd8065ab0 0x7f1bd80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143313934 0 0) 0x7f1bd80acd70 con 0x7f1bd806d220 2023-10-16T13:50:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4003520 con 0x7f1bd806d220 2023-10-16T13:50:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003337606 0 0) 0x7f1bc4003520 con 0x7f1bd806d220 2023-10-16T13:50:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd8065ab0 msgr2=0x7f1bd80c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd8065ab0 0x7f1bd80c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 >> v1:172.21.15.144:6789/0 conn(0x7f1bd80ce280 legacy=0x7f1bd80ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd80ac720 con 0x7f1bd806d220 2023-10-16T13:50:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1bc8001670 con 0x7f1bd806d220 2023-10-16T13:50:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.626+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 >> v1:172.21.15.60:6789/0 conn(0x7f1bd806d220 legacy=0x7f1bd80cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd659c640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bc40039a0 0x7f1bc4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd659c640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc4004300 0x7f1bc40047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd659c640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004fe0 0x7f1bc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd8065f40 con 0x7f1bc4004300 2023-10-16T13:50:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc400bec0 con 0x7f1bc4004fe0 2023-10-16T13:50:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc400c030 con 0x7f1bc40039a0 2023-10-16T13:50:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc4004300 0x7f1bc40047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bd6d9d640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bc40039a0 0x7f1bc4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bcffff640 1 -- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bc40039a0 msgr2=0x7f1bc4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bc40039a0 0x7f1bc4003d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bcffff640 1 -- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004fe0 msgr2=0x7f1bc400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004fe0 0x7f1bc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.627+0000 7f1bcffff640 1 -- 172.21.15.60:0/1576412936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc4003520 con 0x7f1bc4004300 2023-10-16T13:50:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.629+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc4004300 0x7f1bc40047e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009ac0 tx=0x7f1bc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb803f47f377840b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.629+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc002c6b0 con 0x7f1bc4004300 2023-10-16T13:50:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.629+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc002c810 con 0x7f1bc4004300 2023-10-16T13:50:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.629+0000 7f1bd659c640 1 -- 172.21.15.60:0/1576412936 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc002cba0 con 0x7f1bc4004300 2023-10-16T13:50:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1576412936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc4004300 msgr2=0x7f1bc40047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1576412936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc4004300 0x7f1bc40047e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009ac0 tx=0x7f1bc40057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bd759e640 1 -- 172.21.15.60:0/1576412936 reap_dead start 2023-10-16T13:50:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1576412936 shutdown_connections 2023-10-16T13:50:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1576412936 >> 172.21.15.60:0/1576412936 conn(0x7f1bd80b0a80 msgr2=0x7f1bd80b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1576412936 shutdown_connections 2023-10-16T13:50:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.630+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1576412936 wait complete. 2023-10-16T13:50:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.632+0000 7f1bdc9db640 1 Processor -- start 2023-10-16T13:50:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.632+0000 7f1bdc9db640 1 -- start start 2023-10-16T13:50:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bdc9db640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004300 0x7f1bd815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bdc9db640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd80b1ff0 0x7f1bd814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bdc9db640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 0x7f1bd814b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bdc9db640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd80b32b0 con 0x7f1bd815ee80 2023-10-16T13:50:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bdc9db640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd80b4c00 con 0x7f1bc4004300 2023-10-16T13:50:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bdc9db640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd80b2480 con 0x7f1bd80b1ff0 2023-10-16T13:50:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.633+0000 7f1bd759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 0x7f1bd814b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 0x7f1bd814b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35942/0 (socket says 172.21.15.60:35942) 2023-10-16T13:50:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 -- 172.21.15.60:0/1279533267 learned_addr learned my addr 172.21.15.60:0/1279533267 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd6d9d640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004300 0x7f1bd815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd80b1ff0 0x7f1bd814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 -- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd80b1ff0 msgr2=0x7f1bd814aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd80b1ff0 0x7f1bd814aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 -- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004300 msgr2=0x7f1bd815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004300 0x7f1bd815d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bd759e640 1 -- 172.21.15.60:0/1279533267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd80b2980 con 0x7f1bd815ee80 2023-10-16T13:50:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.634+0000 7f1bcffff640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd80b1ff0 0x7f1bd814aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.635+0000 7f1bd759e640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 0x7f1bd814b010 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80c6010 tx=0x7f1bd000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.635+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0005d40 con 0x7f1bd815ee80 2023-10-16T13:50:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.635+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd000ced0 con 0x7f1bd815ee80 2023-10-16T13:50:45.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.635+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd000a850 con 0x7f1bd815ee80 2023-10-16T13:50:45.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.635+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd80b2c60 con 0x7f1bd815ee80 2023-10-16T13:50:45.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.635+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd815d9b0 con 0x7f1bd815ee80 2023-10-16T13:50:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.636+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd8067f50 con 0x7f1bd815ee80 2023-10-16T13:50:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.638+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1bd0003970 con 0x7f1bd815ee80 2023-10-16T13:50:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.639+0000 7f1bcf7fe640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f1bb806afc0 0x7f1bb806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.639+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f1bd0088900 con 0x7f1bd815ee80 2023-10-16T13:50:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.640+0000 7f1bd6d9d640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f1bb806afc0 0x7f1bb806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.641+0000 7f1bd6d9d640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f1bb806afc0 0x7f1bb806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8004c60 tx=0x7f1bc8010610 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.644+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1bd0053b10 con 0x7f1bd815ee80 2023-10-16T13:50:45.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:45.844+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1bd804e9b0 con 0x7f1bd815ee80 2023-10-16T13:50:46.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.380+0000 7f1bcf7fe640 1 -- 172.21.15.60:0/1279533267 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v181) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1bd00170d0 con 0x7f1bd815ee80 2023-10-16T13:50:46.384 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T13:50:46.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.384+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f1bb806afc0 msgr2=0x7f1bb806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:46.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.384+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f1bb806afc0 0x7f1bb806d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8004c60 tx=0x7f1bc8010610 comp rx=0 tx=0).stop 2023-10-16T13:50:46.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 msgr2=0x7f1bd814b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:46.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 0x7f1bd814b010 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1bd80c6010 tx=0x7f1bd000cb90 comp rx=0 tx=0).stop 2023-10-16T13:50:46.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 shutdown_connections 2023-10-16T13:50:46.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f1bb806afc0 0x7f1bb806d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bd80b1ff0 0x7f1bd814aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bd815ee80 0x7f1bd814b010 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 --2- 172.21.15.60:0/1279533267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bc4004300 0x7f1bd815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.385+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 >> 172.21.15.60:0/1279533267 conn(0x7f1bd80b0a80 msgr2=0x7f1bd815aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:46.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.386+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 shutdown_connections 2023-10-16T13:50:46.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.386+0000 7f1bdc9db640 1 -- 172.21.15.60:0/1279533267 wait complete. 2023-10-16T13:50:46.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:50:46.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.615+0000 7fd49620e640 1 Processor -- start 2023-10-16T13:50:46.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.615+0000 7fd49620e640 1 -- start start 2023-10-16T13:50:46.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.616+0000 7fd49620e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 0x7fd4900aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.616+0000 7fd49620e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd49015e9b0 0x7fd49015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.616+0000 7fd49620e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49006c140 con 0x7fd4900cb810 2023-10-16T13:50:46.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.616+0000 7fd49620e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49015b2c0 con 0x7fd49015e9b0 2023-10-16T13:50:46.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.616+0000 7fd49620e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49006b270 con 0x7fd49015aee0 2023-10-16T13:50:46.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48effd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd4900cb810 0x7fd4900cbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39494/0 (socket says 172.21.15.60:39494) 2023-10-16T13:50:46.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48effd640 1 -- 172.21.15.60:0/1541179502 learned_addr learned my addr 172.21.15.60:0/1541179502 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48f7fe640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 0x7fd4900aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48ffff640 1 --2- 172.21.15.60:0/1541179502 >> v2:172.21.15.144:3300/0 conn(0x7fd49015e9b0 0x7fd49015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524025994 0 0) 0x7fd49006c140 con 0x7fd4900cb810 2023-10-16T13:50:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c003520 con 0x7fd4900cb810 2023-10-16T13:50:46.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.617+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1479405619 0 0) 0x7fd47c003520 con 0x7fd4900cb810 2023-10-16T13:50:46.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 msgr2=0x7fd4900aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:46.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 0x7fd4900aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 >> v2:172.21.15.144:3300/0 conn(0x7fd49015e9b0 msgr2=0x7fd49015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:46.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 --2- 172.21.15.60:0/1541179502 >> v2:172.21.15.144:3300/0 conn(0x7fd49015e9b0 0x7fd49015ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49006daf0 con 0x7fd4900cb810 2023-10-16T13:50:46.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd478001670 con 0x7fd4900cb810 2023-10-16T13:50:46.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 >> v1:172.21.15.60:6789/0 conn(0x7fd4900cb810 legacy=0x7fd4900cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:46.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c003970 0x7fd47c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 0x7fd47c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd47c004d00 0x7fd47c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49015b2c0 con 0x7fd47c004d00 2023-10-16T13:50:46.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49006b270 con 0x7fd47c003970 2023-10-16T13:50:46.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c00b980 con 0x7fd49015aee0 2023-10-16T13:50:46.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48ffff640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd47c004d00 0x7fd47c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48effd640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 0x7fd47c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.618+0000 7fd48f7fe640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c003970 0x7fd47c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.619+0000 7fd48ffff640 1 -- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 msgr2=0x7fd47c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:46.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.619+0000 7fd48ffff640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49015aee0 0x7fd47c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.619+0000 7fd48ffff640 1 -- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c003970 msgr2=0x7fd47c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:46.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.619+0000 7fd48ffff640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c003970 0x7fd47c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.619+0000 7fd48ffff640 1 -- 172.21.15.60:0/1541179502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c003520 con 0x7fd47c004d00 2023-10-16T13:50:46.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.620+0000 7fd48ffff640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd47c004d00 0x7fd47c005180 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fd488009ac0 tx=0x7fd47c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce192944f367a0d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:46.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.620+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48802c710 con 0x7fd47c004d00 2023-10-16T13:50:46.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.620+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48802c870 con 0x7fd47c004d00 2023-10-16T13:50:46.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.620+0000 7fd48e7fc640 1 -- 172.21.15.60:0/1541179502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48802cc00 con 0x7fd47c004d00 2023-10-16T13:50:46.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd49620e640 1 -- 172.21.15.60:0/1541179502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd47c004d00 msgr2=0x7fd47c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:46.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd49620e640 1 --2- 172.21.15.60:0/1541179502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd47c004d00 0x7fd47c005180 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fd488009ac0 tx=0x7fd47c0057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd48ffff640 1 -- 172.21.15.60:0/1541179502 reap_dead start 2023-10-16T13:50:46.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd49620e640 1 -- 172.21.15.60:0/1541179502 shutdown_connections 2023-10-16T13:50:46.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd49620e640 1 -- 172.21.15.60:0/1541179502 >> 172.21.15.60:0/1541179502 conn(0x7fd4900cfac0 msgr2=0x7fd4900cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:46.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd49620e640 1 -- 172.21.15.60:0/1541179502 shutdown_connections 2023-10-16T13:50:46.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.621+0000 7fd49620e640 1 -- 172.21.15.60:0/1541179502 wait complete. 2023-10-16T13:50:46.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.623+0000 7fd49620e640 1 Processor -- start 2023-10-16T13:50:46.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.623+0000 7fd49620e640 1 -- start start 2023-10-16T13:50:46.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd49620e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c004d00 0x7fd4900c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd49620e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 0x7fd4900c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd49620e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49014c330 0x7fd4900c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd49620e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900c6b90 con 0x7fd49014beb0 2023-10-16T13:50:46.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd49620e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900c6de0 con 0x7fd47c004d00 2023-10-16T13:50:46.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd49620e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900c70c0 con 0x7fd49014c330 2023-10-16T13:50:46.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd48effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 0x7fd4900c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd48effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 0x7fd4900c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35950/0 (socket says 172.21.15.60:35950) 2023-10-16T13:50:46.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.624+0000 7fd48effd640 1 -- 172.21.15.60:0/2890989028 learned_addr learned my addr 172.21.15.60:0/2890989028 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:46.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48f7fe640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c004d00 0x7fd4900c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48ffff640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49014c330 0x7fd4900c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48effd640 1 -- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49014c330 msgr2=0x7fd4900c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:46.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48effd640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49014c330 0x7fd4900c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48effd640 1 -- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c004d00 msgr2=0x7fd4900c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:46.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48effd640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c004d00 0x7fd4900c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:46.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48effd640 1 -- 172.21.15.60:0/2890989028 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900c73a0 con 0x7fd49014beb0 2023-10-16T13:50:46.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd48effd640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 0x7fd4900c8c30 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fd47800e8c0 tx=0x7fd47800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:46.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478005320 con 0x7fd49014beb0 2023-10-16T13:50:46.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4780054a0 con 0x7fd49014beb0 2023-10-16T13:50:46.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.625+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780047e0 con 0x7fd49014beb0 2023-10-16T13:50:46.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.626+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4900c7680 con 0x7fd49014beb0 2023-10-16T13:50:46.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.626+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4900ac890 con 0x7fd49014beb0 2023-10-16T13:50:46.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.627+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49004e160 con 0x7fd49014beb0 2023-10-16T13:50:46.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.628+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd478004940 con 0x7fd49014beb0 2023-10-16T13:50:46.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.629+0000 7fd487fff640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd47006af90 0x7fd47006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:46.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.629+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd4780897c0 con 0x7fd49014beb0 2023-10-16T13:50:46.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.630+0000 7fd48f7fe640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd47006af90 0x7fd47006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:46.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.631+0000 7fd48f7fe640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd47006af90 0x7fd47006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd480009fd0 tx=0x7fd48000b040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:46.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.633+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4780547a0 con 0x7fd49014beb0 2023-10-16T13:50:46.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:46.864+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd49006d4f0 con 0x7fd49014beb0 2023-10-16T13:50:47.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.402+0000 7fd487fff640 1 -- 172.21.15.60:0/2890989028 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v83) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd478019020 con 0x7fd49014beb0 2023-10-16T13:50:47.406 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 20 and data pool 21 2023-10-16T13:50:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd47006af90 msgr2=0x7fd47006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd47006af90 0x7fd47006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd480009fd0 tx=0x7fd48000b040 comp rx=0 tx=0).stop 2023-10-16T13:50:47.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 msgr2=0x7fd4900c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 0x7fd4900c8c30 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fd47800e8c0 tx=0x7fd47800ed80 comp rx=0 tx=0).stop 2023-10-16T13:50:47.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 shutdown_connections 2023-10-16T13:50:47.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fd47006af90 0x7fd47006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd49014c330 0x7fd4900c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd49014beb0 0x7fd4900c8c30 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 --2- 172.21.15.60:0/2890989028 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd47c004d00 0x7fd4900c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 >> 172.21.15.60:0/2890989028 conn(0x7fd4900cfac0 msgr2=0x7fd49015a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 shutdown_connections 2023-10-16T13:50:47.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.408+0000 7fd49620e640 1 -- 172.21.15.60:0/2890989028 wait complete. 2023-10-16T13:50:47.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.633+0000 7f65505e9640 1 Processor -- start 2023-10-16T13:50:47.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.633+0000 7f65505e9640 1 -- start start 2023-10-16T13:50:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.634+0000 7f65505e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f654815aaa0 0x7f654815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f65505e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654815ed60 con 0x7f65480ce390 2023-10-16T13:50:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f65505e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65480707e0 con 0x7f654815ae80 2023-10-16T13:50:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f65505e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654806b810 con 0x7f654815aaa0 2023-10-16T13:50:47.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f654eb86640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f65480ce390 0x7f6548070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39498/0 (socket says 172.21.15.60:39498) 2023-10-16T13:50:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f654eb86640 1 -- 172.21.15.60:0/1737029802 learned_addr learned my addr 172.21.15.60:0/1737029802 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f654e385640 1 --2- 172.21.15.60:0/1737029802 >> v2:172.21.15.144:3300/0 conn(0x7f654815aaa0 0x7f654815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.635+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28838901 0 0) 0x7f654815ed60 con 0x7f65480ce390 2023-10-16T13:50:47.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6534003520 con 0x7f65480ce390 2023-10-16T13:50:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 995267939 0 0) 0x7f6534003520 con 0x7f65480ce390 2023-10-16T13:50:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 >> v2:172.21.15.144:3300/0 conn(0x7f654815aaa0 msgr2=0x7f654815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 --2- 172.21.15.60:0/1737029802 >> v2:172.21.15.144:3300/0 conn(0x7f654815aaa0 0x7f654815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 >> v1:172.21.15.144:6789/0 conn(0x7f654815ae80 legacy=0x7f65480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654815ef90 con 0x7f65480ce390 2023-10-16T13:50:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f65400019f0 con 0x7f65480ce390 2023-10-16T13:50:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 >> v1:172.21.15.60:6789/0 conn(0x7f65480ce390 legacy=0x7f6548070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534003970 0x7f6534003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65340047c0 0x7f6534004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6534005180 0x7f653400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654806b810 con 0x7f6534005180 2023-10-16T13:50:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653400c020 con 0x7f65340047c0 2023-10-16T13:50:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653400c1e0 con 0x7f6534003970 2023-10-16T13:50:47.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.636+0000 7f654eb86640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6534005180 0x7f653400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654e385640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534003970 0x7f6534003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654db84640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65340047c0 0x7f6534004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654eb86640 1 -- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534003970 msgr2=0x7f6534003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:47.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654eb86640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534003970 0x7f6534003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654eb86640 1 -- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65340047c0 msgr2=0x7f6534004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654eb86640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65340047c0 0x7f6534004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.637+0000 7f654eb86640 1 -- 172.21.15.60:0/1737029802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534003520 con 0x7f6534005180 2023-10-16T13:50:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.638+0000 7f654eb86640 1 -- 172.21.15.60:0/1737029802 reap_dead start 2023-10-16T13:50:47.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.638+0000 7f654eb86640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6534005180 0x7f653400b980 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f654000a820 tx=0x7f65340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b3ed541a7643c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.638+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400072d0 con 0x7f6534005180 2023-10-16T13:50:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.638+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654002f070 con 0x7f6534005180 2023-10-16T13:50:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.638+0000 7f654d383640 1 -- 172.21.15.60:0/1737029802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540007690 con 0x7f6534005180 2023-10-16T13:50:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 -- 172.21.15.60:0/1737029802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6534005180 msgr2=0x7f653400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6534005180 0x7f653400b980 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f654000a820 tx=0x7f65340057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 -- 172.21.15.60:0/1737029802 shutdown_connections 2023-10-16T13:50:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 --2- 172.21.15.60:0/1737029802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6534005180 0x7f653400b980 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 -- 172.21.15.60:0/1737029802 >> 172.21.15.60:0/1737029802 conn(0x7f65480cb410 msgr2=0x7f65480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 -- 172.21.15.60:0/1737029802 shutdown_connections 2023-10-16T13:50:47.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.639+0000 7f65505e9640 1 -- 172.21.15.60:0/1737029802 wait complete. 2023-10-16T13:50:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.641+0000 7f65505e9640 1 Processor -- start 2023-10-16T13:50:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.641+0000 7f65505e9640 1 -- start start 2023-10-16T13:50:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f65505e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534005180 0x7f65480a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f65505e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 0x7f65480c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f65505e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f654814d7f0 0x7f654814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f65505e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480c8440 con 0x7f65480c6b90 2023-10-16T13:50:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f65505e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480c78c0 con 0x7f654814d7f0 2023-10-16T13:50:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f65505e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65480c7ba0 con 0x7f6534005180 2023-10-16T13:50:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f654db84640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 0x7f65480c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f654db84640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 0x7f65480c7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35974/0 (socket says 172.21.15.60:35974) 2023-10-16T13:50:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f654db84640 1 -- 172.21.15.60:0/1970893198 learned_addr learned my addr 172.21.15.60:0/1970893198 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f654e385640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534005180 0x7f65480a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.642+0000 7f654eb86640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f654814d7f0 0x7f654814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654db84640 1 -- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534005180 msgr2=0x7f65480a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654db84640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534005180 0x7f65480a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654db84640 1 -- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f654814d7f0 msgr2=0x7f654814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654db84640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f654814d7f0 0x7f654814dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654db84640 1 -- 172.21.15.60:0/1970893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65480c7e80 con 0x7f65480c6b90 2023-10-16T13:50:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654e385640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534005180 0x7f65480a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:47.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654eb86640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f654814d7f0 0x7f654814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f654db84640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 0x7f65480c7010 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f653800d480 tx=0x7f653800d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538005480 con 0x7f65480c6b90 2023-10-16T13:50:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6538003610 con 0x7f65480c6b90 2023-10-16T13:50:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65380039e0 con 0x7f65480c6b90 2023-10-16T13:50:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65480c8160 con 0x7f65480c6b90 2023-10-16T13:50:47.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.643+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654814e1c0 con 0x7f65480c6b90 2023-10-16T13:50:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.644+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654804e160 con 0x7f65480c6b90 2023-10-16T13:50:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.645+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6538003b40 con 0x7f65480c6b90 2023-10-16T13:50:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.647+0000 7f653effd640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f652c06b010 0x7f652c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.647+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65380897b0 con 0x7f65480c6b90 2023-10-16T13:50:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.648+0000 7f654e385640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f652c06b010 0x7f652c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:47.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.649+0000 7f654e385640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f652c06b010 0x7f652c06d4d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f65440099e0 tx=0x7f6544008040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:47.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.651+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65380546e0 con 0x7f65480c6b90 2023-10-16T13:50:47.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.847+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65480cbaa0 con 0x7f65480c6b90 2023-10-16T13:50:47.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.850+0000 7f653effd640 1 -- 172.21.15.60:0/1970893198 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f6538057d00 con 0x7f65480c6b90 2023-10-16T13:50:47.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:47.854 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":182,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:46.866386+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f652c06b010 msgr2=0x7f652c06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f652c06b010 0x7f652c06d4d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f65440099e0 tx=0x7f6544008040 comp rx=0 tx=0).stop 2023-10-16T13:50:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 msgr2=0x7f65480c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 0x7f65480c7010 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f653800d480 tx=0x7f653800d940 comp rx=0 tx=0).stop 2023-10-16T13:50:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 shutdown_connections 2023-10-16T13:50:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f652c06b010 0x7f652c06d4d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f654814d7f0 0x7f654814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65480c6b90 0x7f65480c7010 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 --2- 172.21.15.60:0/1970893198 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6534005180 0x7f65480a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 >> 172.21.15.60:0/1970893198 conn(0x7f65480cb410 msgr2=0x7f65480b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 shutdown_connections 2023-10-16T13:50:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:47.853+0000 7f65505e9640 1 -- 172.21.15.60:0/1970893198 wait complete. 2023-10-16T13:50:47.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.101+0000 7ffa4652e640 1 Processor -- start 2023-10-16T13:50:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.101+0000 7ffa4652e640 1 -- start start 2023-10-16T13:50:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa4652e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa40144440 0x7ffa4015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa4652e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa4015f1b0 0x7ffa4015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa4652e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4006f1d0 con 0x7ffa40144060 2023-10-16T13:50:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa4652e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa400cd430 con 0x7ffa4015f1b0 2023-10-16T13:50:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa4652e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa400cd610 con 0x7ffa40144440 2023-10-16T13:50:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa45d2d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa4015f1b0 0x7ffa4015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa45d2d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa4015f1b0 0x7ffa4015eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35976/0 (socket says 172.21.15.60:35976) 2023-10-16T13:50:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.103+0000 7ffa45d2d640 1 -- 172.21.15.60:0/951151794 learned_addr learned my addr 172.21.15.60:0/951151794 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.104+0000 7ffa44d2b640 1 --2- 172.21.15.60:0/951151794 >> v2:172.21.15.144:3300/0 conn(0x7ffa40144440 0x7ffa4015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.104+0000 7ffa45d2d640 1 -- 172.21.15.60:0/951151794 >> v2:172.21.15.144:3300/0 conn(0x7ffa40144440 msgr2=0x7ffa4015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:48.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.104+0000 7ffa45d2d640 1 --2- 172.21.15.60:0/951151794 >> v2:172.21.15.144:3300/0 conn(0x7ffa40144440 0x7ffa4015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.104+0000 7ffa45d2d640 1 -- 172.21.15.60:0/951151794 >> v1:172.21.15.144:6789/0 conn(0x7ffa40144060 legacy=0x7ffa400b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.104+0000 7ffa45d2d640 1 -- 172.21.15.60:0/951151794 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400cd8f0 con 0x7ffa4015f1b0 2023-10-16T13:50:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.105+0000 7ffa45d2d640 1 --2- 172.21.15.60:0/951151794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4015f1b0 0x7ffa4015eae0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7ffa400713f0 tx=0x7ffa3802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1ef8929a2888ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.105+0000 7ffa37fff640 1 -- 172.21.15.60:0/951151794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3802c9f0 con 0x7ffa4015f1b0 2023-10-16T13:50:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.105+0000 7ffa37fff640 1 -- 172.21.15.60:0/951151794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3802cb50 con 0x7ffa4015f1b0 2023-10-16T13:50:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.105+0000 7ffa37fff640 1 -- 172.21.15.60:0/951151794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380343f0 con 0x7ffa4015f1b0 2023-10-16T13:50:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 -- 172.21.15.60:0/951151794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4015f1b0 msgr2=0x7ffa4015eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 --2- 172.21.15.60:0/951151794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4015f1b0 0x7ffa4015eae0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7ffa400713f0 tx=0x7ffa3802be50 comp rx=0 tx=0).stop 2023-10-16T13:50:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 -- 172.21.15.60:0/951151794 shutdown_connections 2023-10-16T13:50:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 --2- 172.21.15.60:0/951151794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa4015f1b0 0x7ffa4015eae0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 --2- 172.21.15.60:0/951151794 >> v2:172.21.15.144:3300/0 conn(0x7ffa40144440 0x7ffa4015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 -- 172.21.15.60:0/951151794 >> 172.21.15.60:0/951151794 conn(0x7ffa40069da0 msgr2=0x7ffa4006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 -- 172.21.15.60:0/951151794 shutdown_connections 2023-10-16T13:50:48.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.106+0000 7ffa4652e640 1 -- 172.21.15.60:0/951151794 wait complete. 2023-10-16T13:50:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.108+0000 7ffa4652e640 1 Processor -- start 2023-10-16T13:50:48.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.109+0000 7ffa4652e640 1 -- start start 2023-10-16T13:50:48.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4652e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa40144060 0x7ffa401595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4652e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 0x7ffa4013ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4652e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa4015f1b0 0x7ffa4006b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4652e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa401f34a0 con 0x7ffa40144440 2023-10-16T13:50:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4652e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa401f2490 con 0x7ffa4015f1b0 2023-10-16T13:50:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4652e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa401f26e0 con 0x7ffa40144060 2023-10-16T13:50:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa44d2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 0x7ffa4013ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa44d2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 0x7ffa4013ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35982/0 (socket says 172.21.15.60:35982) 2023-10-16T13:50:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa44d2b640 1 -- 172.21.15.60:0/2112538209 learned_addr learned my addr 172.21.15.60:0/2112538209 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa4552c640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa40144060 0x7ffa401595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.110+0000 7ffa45d2d640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa4015f1b0 0x7ffa4006b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa44d2b640 1 -- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa40144060 msgr2=0x7ffa401595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa44d2b640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa40144060 0x7ffa401595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa44d2b640 1 -- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa4015f1b0 msgr2=0x7ffa4006b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa44d2b640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa4015f1b0 0x7ffa4006b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa44d2b640 1 -- 172.21.15.60:0/2112538209 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38009b30 con 0x7ffa40144440 2023-10-16T13:50:48.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa44d2b640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 0x7ffa4013ab50 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000d800 tx=0x7ffa3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:48.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30004030 con 0x7ffa40144440 2023-10-16T13:50:48.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa30004190 con 0x7ffa40144440 2023-10-16T13:50:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa401f29c0 con 0x7ffa40144440 2023-10-16T13:50:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3000b6c0 con 0x7ffa40144440 2023-10-16T13:50:48.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.111+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa401f2f80 con 0x7ffa40144440 2023-10-16T13:50:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.113+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa400674f0 con 0x7ffa40144440 2023-10-16T13:50:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.113+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa3000b820 con 0x7ffa40144440 2023-10-16T13:50:48.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.114+0000 7ffa35ffb640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ffa2406af90 0x7ffa2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.114+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa30089a90 con 0x7ffa40144440 2023-10-16T13:50:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.115+0000 7ffa4552c640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ffa2406af90 0x7ffa2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.116+0000 7ffa4552c640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ffa2406af90 0x7ffa2406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c004ea0 tx=0x7ffa3c009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.118+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa30054990 con 0x7ffa40144440 2023-10-16T13:50:48.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.317+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4015a930 con 0x7ffa40144440 2023-10-16T13:50:48.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.319+0000 7ffa35ffb640 1 -- 172.21.15.60:0/2112538209 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16580 (secure 0 0 0) 0x7ffa30057fb0 con 0x7ffa40144440 2023-10-16T13:50:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":182,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:46.866386+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ffa2406af90 msgr2=0x7ffa2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ffa2406af90 0x7ffa2406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c004ea0 tx=0x7ffa3c009040 comp rx=0 tx=0).stop 2023-10-16T13:50:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 msgr2=0x7ffa4013ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 0x7ffa4013ab50 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000d800 tx=0x7ffa3000dcc0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 shutdown_connections 2023-10-16T13:50:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ffa2406af90 0x7ffa2406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa4015f1b0 0x7ffa4006b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa40144440 0x7ffa4013ab50 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 --2- 172.21.15.60:0/2112538209 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa40144060 0x7ffa401595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 >> 172.21.15.60:0/2112538209 conn(0x7ffa40069da0 msgr2=0x7ffa400afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 shutdown_connections 2023-10-16T13:50:48.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.322+0000 7ffa4652e640 1 -- 172.21.15.60:0/2112538209 wait complete. 2023-10-16T13:50:48.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:50:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.566+0000 7fa60cbd5640 1 Processor -- start 2023-10-16T13:50:48.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.567+0000 7fa60cbd5640 1 -- start start 2023-10-16T13:50:48.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.568+0000 7fa60cbd5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa60815d150 0x7fa6080ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.568+0000 7fa60cbd5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080ac890 con 0x7fa60806a4b0 2023-10-16T13:50:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.568+0000 7fa60cbd5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa608161650 con 0x7fa608160b90 2023-10-16T13:50:48.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.568+0000 7fa60cbd5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60815d530 con 0x7fa60815d150 2023-10-16T13:50:48.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.569+0000 7fa606d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa60806a4b0 0x7fa6080c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39504/0 (socket says 172.21.15.60:39504) 2023-10-16T13:50:48.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.569+0000 7fa606d9d640 1 -- 172.21.15.60:0/1796029396 learned_addr learned my addr 172.21.15.60:0/1796029396 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:48.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.569+0000 7fa60659c640 1 --2- 172.21.15.60:0/1796029396 >> v2:172.21.15.144:3300/0 conn(0x7fa60815d150 0x7fa6080ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.569+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502654642 0 0) 0x7fa6080ac890 con 0x7fa60806a4b0 2023-10-16T13:50:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.569+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f0003520 con 0x7fa60806a4b0 2023-10-16T13:50:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016397354 0 0) 0x7fa5f0003520 con 0x7fa60806a4b0 2023-10-16T13:50:48.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 >> v2:172.21.15.144:3300/0 conn(0x7fa60815d150 msgr2=0x7fa6080ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 --2- 172.21.15.60:0/1796029396 >> v2:172.21.15.144:3300/0 conn(0x7fa60815d150 0x7fa6080ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 >> v1:172.21.15.144:6789/0 conn(0x7fa608160b90 legacy=0x7fa608069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa608161930 con 0x7fa60806a4b0 2023-10-16T13:50:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa600001670 con 0x7fa60806a4b0 2023-10-16T13:50:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 >> v1:172.21.15.60:6789/0 conn(0x7fa60806a4b0 legacy=0x7fa6080c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa5f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f00047c0 0x7fa5f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f0005180 0x7fa5f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60815d530 con 0x7fa5f00039c0 2023-10-16T13:50:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f000c080 con 0x7fa5f00047c0 2023-10-16T13:50:48.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f000c240 con 0x7fa5f0005180 2023-10-16T13:50:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa60659c640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa5f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.570+0000 7fa605d9b640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f00047c0 0x7fa5f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa606d9d640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f0005180 0x7fa5f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa60659c640 1 -- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f0005180 msgr2=0x7fa5f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:48.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa60659c640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f0005180 0x7fa5f000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa60659c640 1 -- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f00047c0 msgr2=0x7fa5f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa60659c640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f00047c0 0x7fa5f0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa60659c640 1 -- 172.21.15.60:0/1796029396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0003520 con 0x7fa5f00039c0 2023-10-16T13:50:48.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.571+0000 7fa606d9d640 1 -- 172.21.15.60:0/1796029396 reap_dead start 2023-10-16T13:50:48.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.572+0000 7fa60659c640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa5f0003de0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009ac0 tx=0x7fa5f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7fe0eef714c5c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:48.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.572+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f802c6b0 con 0x7fa5f00039c0 2023-10-16T13:50:48.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.572+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f802c810 con 0x7fa5f00039c0 2023-10-16T13:50:48.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.572+0000 7fa60559a640 1 -- 172.21.15.60:0/1796029396 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f802cba0 con 0x7fa5f00039c0 2023-10-16T13:50:48.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 -- 172.21.15.60:0/1796029396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 msgr2=0x7fa5f0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa5f0003de0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009ac0 tx=0x7fa5f00057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 -- 172.21.15.60:0/1796029396 shutdown_connections 2023-10-16T13:50:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/1796029396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa5f0003de0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 -- 172.21.15.60:0/1796029396 >> 172.21.15.60:0/1796029396 conn(0x7fa6080ce9b0 msgr2=0x7fa6080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 -- 172.21.15.60:0/1796029396 shutdown_connections 2023-10-16T13:50:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.573+0000 7fa60cbd5640 1 -- 172.21.15.60:0/1796029396 wait complete. 2023-10-16T13:50:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.575+0000 7fa60cbd5640 1 Processor -- start 2023-10-16T13:50:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.575+0000 7fa60cbd5640 1 -- start start 2023-10-16T13:50:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60cbd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa6080ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60cbd5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa608157390 0x7fa6080c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60cbd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6080c68b0 0x7fa6080c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60cbd5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60815e720 con 0x7fa5f00039c0 2023-10-16T13:50:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60cbd5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60815ea00 con 0x7fa608157390 2023-10-16T13:50:48.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60cbd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60815ece0 con 0x7fa6080c68b0 2023-10-16T13:50:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa6080ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa6080ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36002/0 (socket says 172.21.15.60:36002) 2023-10-16T13:50:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.576+0000 7fa60659c640 1 -- 172.21.15.60:0/3989716271 learned_addr learned my addr 172.21.15.60:0/3989716271 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa605d9b640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa608157390 0x7fa6080c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa606d9d640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6080c68b0 0x7fa6080c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60659c640 1 -- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6080c68b0 msgr2=0x7fa6080c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60659c640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6080c68b0 0x7fa6080c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60659c640 1 -- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa608157390 msgr2=0x7fa6080c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60659c640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa608157390 0x7fa6080c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60659c640 1 -- 172.21.15.60:0/3989716271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60815efc0 con 0x7fa5f00039c0 2023-10-16T13:50:48.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60659c640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa6080ae240 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fa5f802c250 tx=0x7fa5f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8003810 con 0x7fa5f00039c0 2023-10-16T13:50:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f8003970 con 0x7fa5f00039c0 2023-10-16T13:50:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8003d00 con 0x7fa5f00039c0 2023-10-16T13:50:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.577+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60815f2a0 con 0x7fa5f00039c0 2023-10-16T13:50:48.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.578+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6081582a0 con 0x7fa5f00039c0 2023-10-16T13:50:48.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.579+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60804e160 con 0x7fa5f00039c0 2023-10-16T13:50:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.580+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa5f8034440 con 0x7fa5f00039c0 2023-10-16T13:50:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.581+0000 7fa5f6ffd640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fa5e806afc0 0x7fa5e806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.581+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5f80ab970 con 0x7fa5f00039c0 2023-10-16T13:50:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.582+0000 7fa605d9b640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fa5e806afc0 0x7fa5e806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:48.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.583+0000 7fa605d9b640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fa5e806afc0 0x7fa5e806d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc003fb0 tx=0x7fa5fc009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:48.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.586+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5f80769c0 con 0x7fa5f00039c0 2023-10-16T13:50:48.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:48.824+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa6080cd360 con 0x7fa5f00039c0 2023-10-16T13:50:49.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.433+0000 7fa5f6ffd640 1 -- 172.21.15.60:0/3989716271 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v86) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa5f804c0b0 con 0x7fa5f00039c0 2023-10-16T13:50:49.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fa5e806afc0 msgr2=0x7fa5e806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fa5e806afc0 0x7fa5e806d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc003fb0 tx=0x7fa5fc009040 comp rx=0 tx=0).stop 2023-10-16T13:50:49.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 msgr2=0x7fa6080ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa6080ae240 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fa5f802c250 tx=0x7fa5f8004730 comp rx=0 tx=0).stop 2023-10-16T13:50:49.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 shutdown_connections 2023-10-16T13:50:49.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fa5e806afc0 0x7fa5e806d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa6080c68b0 0x7fa6080c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa608157390 0x7fa6080c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 --2- 172.21.15.60:0/3989716271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00039c0 0x7fa6080ae240 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 >> 172.21.15.60:0/3989716271 conn(0x7fa6080ce9b0 msgr2=0x7fa60806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:49.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 shutdown_connections 2023-10-16T13:50:49.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.437+0000 7fa60cbd5640 1 -- 172.21.15.60:0/3989716271 wait complete. 2023-10-16T13:50:49.462 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:50:49.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:50:49.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.684+0000 7f6e71f7a640 1 Processor -- start 2023-10-16T13:50:49.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.684+0000 7f6e71f7a640 1 -- start start 2023-10-16T13:50:49.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e71f7a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e6c0cded0 0x7f6e6c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:49.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e71f7a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c15d150 con 0x7f6e6c0b8480 2023-10-16T13:50:49.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e71f7a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c0d1180 con 0x7f6e6c15cd70 2023-10-16T13:50:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e71f7a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c0d1460 con 0x7f6e6c0cded0 2023-10-16T13:50:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e6bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6e6c0b8480 0x7f6e6c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39686/0 (socket says 172.21.15.60:39686) 2023-10-16T13:50:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e71779640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e6c0cded0 0x7f6e6c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.686+0000 7f6e6bfff640 1 -- 172.21.15.60:0/4286366042 learned_addr learned my addr 172.21.15.60:0/4286366042 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.687+0000 7f6e6b7fe640 1 -- 172.21.15.60:0/4286366042 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128808427 0 0) 0x7f6e6c15d150 con 0x7f6e6c0b8480 2023-10-16T13:50:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.687+0000 7f6e71779640 1 -- 172.21.15.60:0/4286366042 >> v1:172.21.15.144:6789/0 conn(0x7f6e6c15cd70 legacy=0x7f6e6c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.687+0000 7f6e71779640 1 -- 172.21.15.60:0/4286366042 >> v1:172.21.15.60:6789/0 conn(0x7f6e6c0b8480 legacy=0x7f6e6c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.687+0000 7f6e71779640 1 -- 172.21.15.60:0/4286366042 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0d1740 con 0x7f6e6c0cded0 2023-10-16T13:50:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.688+0000 7f6e71779640 1 --2- 172.21.15.60:0/4286366042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c0cded0 0x7f6e6c0b5c90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f6e640097a0 tx=0x7f6e6402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc7ba80262c1a3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.688+0000 7f6e6b7fe640 1 -- 172.21.15.60:0/4286366042 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64030610 con 0x7f6e6c0cded0 2023-10-16T13:50:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.688+0000 7f6e6b7fe640 1 -- 172.21.15.60:0/4286366042 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e64030770 con 0x7f6e6c0cded0 2023-10-16T13:50:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.688+0000 7f6e6b7fe640 1 -- 172.21.15.60:0/4286366042 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64030b00 con 0x7f6e6c0cded0 2023-10-16T13:50:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 -- 172.21.15.60:0/4286366042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c0cded0 msgr2=0x7f6e6c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/4286366042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c0cded0 0x7f6e6c0b5c90 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f6e640097a0 tx=0x7f6e6402b960 comp rx=0 tx=0).stop 2023-10-16T13:50:49.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 -- 172.21.15.60:0/4286366042 shutdown_connections 2023-10-16T13:50:49.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/4286366042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c0cded0 0x7f6e6c0b5c90 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 -- 172.21.15.60:0/4286366042 >> 172.21.15.60:0/4286366042 conn(0x7f6e6c0cf570 msgr2=0x7f6e6c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:49.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 -- 172.21.15.60:0/4286366042 shutdown_connections 2023-10-16T13:50:49.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.689+0000 7f6e71f7a640 1 -- 172.21.15.60:0/4286366042 wait complete. 2023-10-16T13:50:49.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.691+0000 7f6e71f7a640 1 Processor -- start 2023-10-16T13:50:49.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.691+0000 7f6e71f7a640 1 -- start start 2023-10-16T13:50:49.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71f7a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e6c0b8480 0x7f6e6c158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:49.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71f7a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6c0cded0 0x7f6e6c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:49.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71f7a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 0x7f6e6c0b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:49.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71f7a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0aa600 con 0x7f6e6c15cd70 2023-10-16T13:50:49.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71f7a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0ca8e0 con 0x7f6e6c0cded0 2023-10-16T13:50:49.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71f7a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c1f6880 con 0x7f6e6c0b8480 2023-10-16T13:50:49.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.692+0000 7f6e71779640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 0x7f6e6c0b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:49.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 0x7f6e6c0b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53654/0 (socket says 172.21.15.60:53654) 2023-10-16T13:50:49.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 -- 172.21.15.60:0/1931877970 learned_addr learned my addr 172.21.15.60:0/1931877970 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:49.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 -- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e6c0b8480 msgr2=0x7f6e6c158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e6c0b8480 0x7f6e6c158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 -- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6c0cded0 msgr2=0x7f6e6c0b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:49.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6c0cded0 0x7f6e6c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:49.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 -- 172.21.15.60:0/1931877970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64009400 con 0x7f6e6c15cd70 2023-10-16T13:50:49.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.693+0000 7f6e71779640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 0x7f6e6c0b6730 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f6e6402bf70 tx=0x7f6e64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:49.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.694+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6403e040 con 0x7f6e6c15cd70 2023-10-16T13:50:49.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.694+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e64039c10 con 0x7f6e6c15cd70 2023-10-16T13:50:49.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.694+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64030e10 con 0x7f6e6c15cd70 2023-10-16T13:50:49.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.694+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c0caaf0 con 0x7f6e6c15cd70 2023-10-16T13:50:49.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.694+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c0cb0b0 con 0x7f6e6c15cd70 2023-10-16T13:50:49.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.695+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c0b1be0 con 0x7f6e6c15cd70 2023-10-16T13:50:49.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.696+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e640423e0 con 0x7f6e6c15cd70 2023-10-16T13:50:49.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.697+0000 7f6e697fa640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6e5806af90 0x7f6e5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:49.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.697+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e640abbb0 con 0x7f6e6c15cd70 2023-10-16T13:50:49.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.698+0000 7f6e70f78640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6e5806af90 0x7f6e5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:49.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.699+0000 7f6e70f78640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6e5806af90 0x7f6e5806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0012e0 tx=0x7f6e5c007d30 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:49.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.701+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e64076ad0 con 0x7f6e6c15cd70 2023-10-16T13:50:49.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:49.936+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6e6c04e9b0 con 0x7f6e6c15cd70 2023-10-16T13:50:50.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:50.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.446+0000 7f6e697fa640 1 -- 172.21.15.60:0/1931877970 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v87) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6e6407a0f0 con 0x7f6e6c15cd70 2023-10-16T13:50:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6e5806af90 msgr2=0x7f6e5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6e5806af90 0x7f6e5806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0012e0 tx=0x7f6e5c007d30 comp rx=0 tx=0).stop 2023-10-16T13:50:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 msgr2=0x7f6e6c0b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 0x7f6e6c0b6730 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f6e6402bf70 tx=0x7f6e64004730 comp rx=0 tx=0).stop 2023-10-16T13:50:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 shutdown_connections 2023-10-16T13:50:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6e5806af90 0x7f6e5806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6c15cd70 0x7f6e6c0b6730 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6c0cded0 0x7f6e6c0b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 --2- 172.21.15.60:0/1931877970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e6c0b8480 0x7f6e6c158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.449+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 >> 172.21.15.60:0/1931877970 conn(0x7f6e6c0cf570 msgr2=0x7f6e6c0cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.450+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 shutdown_connections 2023-10-16T13:50:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.450+0000 7f6e71f7a640 1 -- 172.21.15.60:0/1931877970 wait complete. 2023-10-16T13:50:50.474 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:50:50.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:50.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.689+0000 7f87b79b0640 1 Processor -- start 2023-10-16T13:50:50.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.689+0000 7f87b79b0640 1 -- start start 2023-10-16T13:50:50.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b79b0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87b0159f00 0x7f87b00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:50.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b79b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f87b015d880 0x7f87b00b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:50.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b79b0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b006f7f0 con 0x7f87b0159b20 2023-10-16T13:50:50.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b79b0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b006c5e0 con 0x7f87b0159f00 2023-10-16T13:50:50.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b79b0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b006c8c0 con 0x7f87b015d880 2023-10-16T13:50:50.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b574c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f87b0159b20 0x7f87b006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39702/0 (socket says 172.21.15.60:39702) 2023-10-16T13:50:50.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.691+0000 7f87b574c640 1 -- 172.21.15.60:0/594171136 learned_addr learned my addr 172.21.15.60:0/594171136 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:50.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b4f4b640 1 --2- 172.21.15.60:0/594171136 >> v2:172.21.15.60:3300/0 conn(0x7f87b0159f00 0x7f87b00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:50.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b5f4d640 1 --2- 172.21.15.60:0/594171136 >> v2:172.21.15.144:3300/0 conn(0x7f87b015d880 0x7f87b00b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:50.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87a7fff640 1 -- 172.21.15.60:0/594171136 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521236336 0 0) 0x7f87b006f7f0 con 0x7f87b0159b20 2023-10-16T13:50:50.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87a7fff640 1 -- 172.21.15.60:0/594171136 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879c003520 con 0x7f87b0159b20 2023-10-16T13:50:50.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b4f4b640 1 -- 172.21.15.60:0/594171136 >> v2:172.21.15.144:3300/0 conn(0x7f87b015d880 msgr2=0x7f87b00b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:50.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b4f4b640 1 --2- 172.21.15.60:0/594171136 >> v2:172.21.15.144:3300/0 conn(0x7f87b015d880 0x7f87b00b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b4f4b640 1 -- 172.21.15.60:0/594171136 >> v1:172.21.15.60:6789/0 conn(0x7f87b0159b20 legacy=0x7f87b006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b4f4b640 1 -- 172.21.15.60:0/594171136 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b006cba0 con 0x7f87b0159f00 2023-10-16T13:50:50.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.692+0000 7f87b5f4d640 1 --2- 172.21.15.60:0/594171136 >> v2:172.21.15.144:3300/0 conn(0x7f87b015d880 0x7f87b00b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:50.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.693+0000 7f87b4f4b640 1 --2- 172.21.15.60:0/594171136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b0159f00 0x7f87b00b0410 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f87b00c8bb0 tx=0x7f87a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df6be8d4cd7da91 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:50.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.693+0000 7f87a7fff640 1 -- 172.21.15.60:0/594171136 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0031830 con 0x7f87b0159f00 2023-10-16T13:50:50.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.693+0000 7f87a7fff640 1 -- 172.21.15.60:0/594171136 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a0031990 con 0x7f87b0159f00 2023-10-16T13:50:50.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.693+0000 7f87a7fff640 1 -- 172.21.15.60:0/594171136 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0031d20 con 0x7f87b0159f00 2023-10-16T13:50:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 -- 172.21.15.60:0/594171136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b0159f00 msgr2=0x7f87b00b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 --2- 172.21.15.60:0/594171136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b0159f00 0x7f87b00b0410 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f87b00c8bb0 tx=0x7f87a002b8d0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 -- 172.21.15.60:0/594171136 shutdown_connections 2023-10-16T13:50:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 --2- 172.21.15.60:0/594171136 >> v2:172.21.15.144:3300/0 conn(0x7f87b015d880 0x7f87b00b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 --2- 172.21.15.60:0/594171136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b0159f00 0x7f87b00b0410 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 -- 172.21.15.60:0/594171136 >> 172.21.15.60:0/594171136 conn(0x7f87b00cb410 msgr2=0x7f87b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 -- 172.21.15.60:0/594171136 shutdown_connections 2023-10-16T13:50:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.694+0000 7f87b79b0640 1 -- 172.21.15.60:0/594171136 wait complete. 2023-10-16T13:50:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.696+0000 7f87b79b0640 1 Processor -- start 2023-10-16T13:50:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.696+0000 7f87b79b0640 1 -- start start 2023-10-16T13:50:50.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b79b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b0159b20 0x7f87b014f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:50.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b79b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b0159f00 0x7f87b00c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:50.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b79b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 0x7f87b00c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:50.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b79b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b015bce0 con 0x7f87b015d880 2023-10-16T13:50:50.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b79b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0150460 con 0x7f87b0159b20 2023-10-16T13:50:50.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b79b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b014f9c0 con 0x7f87b0159f00 2023-10-16T13:50:50.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b5f4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 0x7f87b00c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:50.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b5f4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 0x7f87b00c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53672/0 (socket says 172.21.15.60:53672) 2023-10-16T13:50:50.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.697+0000 7f87b5f4d640 1 -- 172.21.15.60:0/3937195920 learned_addr learned my addr 172.21.15.60:0/3937195920 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:50.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b574c640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b0159b20 0x7f87b014f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b4f4b640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b0159f00 0x7f87b00c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b5f4d640 1 -- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b0159f00 msgr2=0x7f87b00c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b5f4d640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b0159f00 0x7f87b00c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b5f4d640 1 -- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b0159b20 msgr2=0x7f87b014f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b5f4d640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b0159b20 0x7f87b014f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b5f4d640 1 -- 172.21.15.60:0/3937195920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0009400 con 0x7f87b015d880 2023-10-16T13:50:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b5f4d640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 0x7f87b00c24c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f87a800cef0 tx=0x7f87a800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800d840 con 0x7f87b015d880 2023-10-16T13:50:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a800d9c0 con 0x7f87b015d880 2023-10-16T13:50:50.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800dd70 con 0x7f87b015d880 2023-10-16T13:50:50.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.698+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b015b0d0 con 0x7f87b015d880 2023-10-16T13:50:50.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.699+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b015b690 con 0x7f87b015d880 2023-10-16T13:50:50.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.700+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b015be50 con 0x7f87b015d880 2023-10-16T13:50:50.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.701+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87a801e3b0 con 0x7f87b015d880 2023-10-16T13:50:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.702+0000 7f87a5ffb640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f879406afc0 0x7f879406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.702+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87a808a220 con 0x7f87b015d880 2023-10-16T13:50:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.703+0000 7f87b574c640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f879406afc0 0x7f879406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.704+0000 7f87b574c640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f879406afc0 0x7f879406d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0011e0 tx=0x7f87ac009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.706+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87a8055290 con 0x7f87b015d880 2023-10-16T13:50:50.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.946+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b004e160 con 0x7f87b015d880 2023-10-16T13:50:50.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.947+0000 7f87a5ffb640 1 -- 172.21.15.60:0/3937195920 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f87a80588b0 con 0x7f87b015d880 2023-10-16T13:50:50.951 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:50.951 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:50:46.866966+0000","modified":"2023-10-16T13:50:49.938566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/691868089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":691868089},{"type":"v1","addr":"172.21.15.60:6834","nonce":691868089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:50:50.951 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 87 2023-10-16T13:50:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f879406afc0 msgr2=0x7f879406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f879406afc0 0x7f879406d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0011e0 tx=0x7f87ac009040 comp rx=0 tx=0).stop 2023-10-16T13:50:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 msgr2=0x7f87b00c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:50.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 0x7f87b00c24c0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f87a800cef0 tx=0x7f87a800a710 comp rx=0 tx=0).stop 2023-10-16T13:50:50.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 shutdown_connections 2023-10-16T13:50:50.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f879406afc0 0x7f879406d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87b0159f00 0x7f87b00c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87b015d880 0x7f87b00c24c0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 --2- 172.21.15.60:0/3937195920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87b0159b20 0x7f87b014f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:50.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.950+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 >> 172.21.15.60:0/3937195920 conn(0x7f87b00cb410 msgr2=0x7f87b00cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:50.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.951+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 shutdown_connections 2023-10-16T13:50:50.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:50.951+0000 7f87b79b0640 1 -- 172.21.15.60:0/3937195920 wait complete. 2023-10-16T13:50:50.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:50:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.190+0000 7f3b9b8cf640 1 Processor -- start 2023-10-16T13:50:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.190+0000 7f3b9b8cf640 1 -- start start 2023-10-16T13:50:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b9b8cf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 0x7f3b9406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b9b8cf640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b94159f00 0x7f3b940b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b9b8cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9406f7f0 con 0x7f3b9406c120 2023-10-16T13:50:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b9b8cf640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9406c5e0 con 0x7f3b94159f00 2023-10-16T13:50:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b9b8cf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9406c8c0 con 0x7f3b94159b20 2023-10-16T13:50:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b99e6c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3b9406c120 0x7f3b940cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39718/0 (socket says 172.21.15.60:39718) 2023-10-16T13:50:51.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b99e6c640 1 -- 172.21.15.60:0/1442079077 learned_addr learned my addr 172.21.15.60:0/1442079077 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:51.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b9966b640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 0x7f3b9406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b98e6a640 1 --2- 172.21.15.60:0/1442079077 >> v2:172.21.15.144:3300/0 conn(0x7f3b94159f00 0x7f3b940b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.192+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619935962 0 0) 0x7f3b9406f7f0 con 0x7f3b9406c120 2023-10-16T13:50:51.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b80003520 con 0x7f3b9406c120 2023-10-16T13:50:51.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1897222052 0 0) 0x7f3b80003520 con 0x7f3b9406c120 2023-10-16T13:50:51.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 msgr2=0x7f3b9406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 0x7f3b9406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 >> v2:172.21.15.144:3300/0 conn(0x7f3b94159f00 msgr2=0x7f3b940b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 --2- 172.21.15.60:0/1442079077 >> v2:172.21.15.144:3300/0 conn(0x7f3b94159f00 0x7f3b940b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9406cba0 con 0x7f3b9406c120 2023-10-16T13:50:51.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f3b8c0019f0 con 0x7f3b9406c120 2023-10-16T13:50:51.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 >> v1:172.21.15.60:6789/0 conn(0x7f3b9406c120 legacy=0x7f3b940cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 0x7f3b800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80004d00 0x7f3b80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9406c5e0 con 0x7f3b80003970 2023-10-16T13:50:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9406c8c0 con 0x7f3b80004d00 2023-10-16T13:50:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8000b980 con 0x7f3b94159b20 2023-10-16T13:50:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.193+0000 7f3b9966b640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b98e6a640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 0x7f3b800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b9966b640 1 -- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 msgr2=0x7f3b800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b9966b640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b94159b20 0x7f3b800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b99e6c640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80004d00 0x7f3b80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b9966b640 1 -- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80004d00 msgr2=0x7f3b80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b9966b640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80004d00 0x7f3b80005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.194+0000 7f3b9966b640 1 -- 172.21.15.60:0/1442079077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b80003520 con 0x7f3b80003970 2023-10-16T13:50:51.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.195+0000 7f3b9966b640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009af0 tx=0x7f3b800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5f1cba7f3374d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:51.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.195+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9002c6b0 con 0x7f3b80003970 2023-10-16T13:50:51.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.195+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9002c810 con 0x7f3b80003970 2023-10-16T13:50:51.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.195+0000 7f3b8bfff640 1 -- 172.21.15.60:0/1442079077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9002cba0 con 0x7f3b80003970 2023-10-16T13:50:51.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/1442079077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b80003970 msgr2=0x7f3b80003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/1442079077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b80003970 0x7f3b80003d90 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009af0 tx=0x7f3b800057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b99e6c640 1 -- 172.21.15.60:0/1442079077 reap_dead start 2023-10-16T13:50:51.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/1442079077 shutdown_connections 2023-10-16T13:50:51.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/1442079077 >> 172.21.15.60:0/1442079077 conn(0x7f3b940cb410 msgr2=0x7f3b940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:51.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/1442079077 shutdown_connections 2023-10-16T13:50:51.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.196+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/1442079077 wait complete. 2023-10-16T13:50:51.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.198+0000 7f3b9b8cf640 1 Processor -- start 2023-10-16T13:50:51.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.198+0000 7f3b9b8cf640 1 -- start start 2023-10-16T13:50:51.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b9b8cf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80003970 0x7f3b941508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b9b8cf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b9414e920 0x7f3b940c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b9b8cf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 0x7f3b940c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b9b8cf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9414eda0 con 0x7f3b940c5a60 2023-10-16T13:50:51.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b9b8cf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940c6420 con 0x7f3b80003970 2023-10-16T13:50:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b9b8cf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940c4dc0 con 0x7f3b9414e920 2023-10-16T13:50:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b99e6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 0x7f3b940c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.199+0000 7f3b99e6c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 0x7f3b940c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53700/0 (socket says 172.21.15.60:53700) 2023-10-16T13:50:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 -- 172.21.15.60:0/731186327 learned_addr learned my addr 172.21.15.60:0/731186327 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b9966b640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80003970 0x7f3b941508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b98e6a640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b9414e920 0x7f3b940c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 -- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b9414e920 msgr2=0x7f3b940c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b9414e920 0x7f3b940c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 -- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80003970 msgr2=0x7f3b941508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80003970 0x7f3b941508f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 -- 172.21.15.60:0/731186327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9414e760 con 0x7f3b940c5a60 2023-10-16T13:50:51.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b99e6c640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 0x7f3b940c5ee0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0012e0 tx=0x7f3b8c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:51.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00bb80 con 0x7f3b940c5a60 2023-10-16T13:50:51.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.200+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c00bce0 con 0x7f3b940c5a60 2023-10-16T13:50:51.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.201+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b941511a0 con 0x7f3b940c5a60 2023-10-16T13:50:51.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.201+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c007330 con 0x7f3b940c5a60 2023-10-16T13:50:51.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.201+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b940c8960 con 0x7f3b940c5a60 2023-10-16T13:50:51.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.202+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b940af420 con 0x7f3b940c5a60 2023-10-16T13:50:51.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.203+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3b8c007490 con 0x7f3b940c5a60 2023-10-16T13:50:51.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.204+0000 7f3b89ffb640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f3b7806af90 0x7f3b7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.205+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b8c089b70 con 0x7f3b940c5a60 2023-10-16T13:50:51.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.205+0000 7f3b9966b640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f3b7806af90 0x7f3b7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.206+0000 7f3b9966b640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f3b7806af90 0x7f3b7806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b90036000 tx=0x7f3b90030000 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:51.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.209+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b940af420 con 0x7f3b940c5a60 2023-10-16T13:50:51.346 INFO:tasks.ceph.mds.c.smithi060.stderr:2023-10-16T13:50:51.343+0000 7fc74c264640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:50:51.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.408+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b940af420 con 0x7f3b940c5a60 2023-10-16T13:50:51.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.411+0000 7f3b89ffb640 1 -- 172.21.15.60:0/731186327 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+16567 (secure 0 0 0) 0x7f3b940af420 con 0x7f3b940c5a60 2023-10-16T13:50:51.414 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:51.415 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":183,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:50:48.421518+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:50:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.413+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f3b7806af90 msgr2=0x7f3b7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.413+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f3b7806af90 0x7f3b7806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b90036000 tx=0x7f3b90030000 comp rx=0 tx=0).stop 2023-10-16T13:50:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.413+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 msgr2=0x7f3b940c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.413+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 0x7f3b940c5ee0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0012e0 tx=0x7f3b8c00b4a0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 shutdown_connections 2023-10-16T13:50:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f3b7806af90 0x7f3b7806d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3b9414e920 0x7f3b940c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b940c5a60 0x7f3b940c5ee0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 --2- 172.21.15.60:0/731186327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b80003970 0x7f3b941508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 >> 172.21.15.60:0/731186327 conn(0x7f3b940cb410 msgr2=0x7f3b940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 shutdown_connections 2023-10-16T13:50:51.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.414+0000 7f3b9b8cf640 1 -- 172.21.15.60:0/731186327 wait complete. 2023-10-16T13:50:51.438 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:50:51.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:50:51.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.651+0000 7f5f3c8ba640 1 Processor -- start 2023-10-16T13:50:51.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.652+0000 7f5f3c8ba640 1 -- start start 2023-10-16T13:50:51.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.653+0000 7f5f3c8ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f380ad970 0x7f5f380cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.653+0000 7f5f3c8ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3815b9d0 con 0x7f5f3806c040 2023-10-16T13:50:51.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.653+0000 7f5f3c8ba640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3815bcb0 con 0x7f5f380ccfb0 2023-10-16T13:50:51.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.653+0000 7f5f3c8ba640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3815bf90 con 0x7f5f380ad970 2023-10-16T13:50:51.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.653+0000 7f5f36d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5f3806c040 0x7f5f3806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39724/0 (socket says 172.21.15.60:39724) 2023-10-16T13:50:51.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.653+0000 7f5f36d9d640 1 -- 172.21.15.60:0/383888371 learned_addr learned my addr 172.21.15.60:0/383888371 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:51.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f2ffff640 1 --2- 172.21.15.60:0/383888371 >> v2:172.21.15.144:3300/0 conn(0x7f5f380ad970 0x7f5f380cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11612218 0 0) 0x7f5f3815b9d0 con 0x7f5f3806c040 2023-10-16T13:50:51.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f24003520 con 0x7f5f3806c040 2023-10-16T13:50:51.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2261128794 0 0) 0x7f5f24003520 con 0x7f5f3806c040 2023-10-16T13:50:51.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 >> v2:172.21.15.144:3300/0 conn(0x7f5f380ad970 msgr2=0x7f5f380cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 --2- 172.21.15.60:0/383888371 >> v2:172.21.15.144:3300/0 conn(0x7f5f380ad970 0x7f5f380cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 >> v1:172.21.15.144:6789/0 conn(0x7f5f380ccfb0 legacy=0x7f5f380cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380cc270 con 0x7f5f3806c040 2023-10-16T13:50:51.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5f28001670 con 0x7f5f3806c040 2023-10-16T13:50:51.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.654+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 >> v1:172.21.15.60:6789/0 conn(0x7f5f3806c040 legacy=0x7f5f3806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3659c640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f24003970 0x7f5f24003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3659c640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f240047c0 0x7f5f24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3659c640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24005180 0x7f5f2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3815bf90 con 0x7f5f24003970 2023-10-16T13:50:51.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2400c0b0 con 0x7f5f24005180 2023-10-16T13:50:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2400c270 con 0x7f5f240047c0 2023-10-16T13:50:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f24003970 0x7f5f24003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3759e640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24005180 0x7f5f2400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f2ffff640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f240047c0 0x7f5f24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f36d9d640 1 -- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f240047c0 msgr2=0x7f5f24004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f240047c0 0x7f5f24004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f36d9d640 1 -- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24005180 msgr2=0x7f5f2400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:51.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24005180 0x7f5f2400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f36d9d640 1 -- 172.21.15.60:0/383888371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24003520 con 0x7f5f24003970 2023-10-16T13:50:51.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3759e640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24005180 0x7f5f2400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:51.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.655+0000 7f5f3759e640 1 -- 172.21.15.60:0/383888371 reap_dead start 2023-10-16T13:50:51.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f24003970 0x7f5f24003d70 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5f28001eb0 tx=0x7f5f240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4901ac7575bd4bf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:51.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28005ad0 con 0x7f5f24003970 2023-10-16T13:50:51.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f28005c50 con 0x7f5f24003970 2023-10-16T13:50:51.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f3659c640 1 -- 172.21.15.60:0/383888371 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2802c8b0 con 0x7f5f24003970 2023-10-16T13:50:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/383888371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f24003970 msgr2=0x7f5f24003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f24003970 0x7f5f24003d70 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5f28001eb0 tx=0x7f5f240057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.657+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/383888371 shutdown_connections 2023-10-16T13:50:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.658+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/383888371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f24003970 0x7f5f24003d70 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.658+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/383888371 >> 172.21.15.60:0/383888371 conn(0x7f5f380ce6a0 msgr2=0x7f5f380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.658+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/383888371 shutdown_connections 2023-10-16T13:50:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.658+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/383888371 wait complete. 2023-10-16T13:50:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.659+0000 7f5f3c8ba640 1 Processor -- start 2023-10-16T13:50:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.660+0000 7f5f3c8ba640 1 -- start start 2023-10-16T13:50:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.660+0000 7f5f3c8ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24003970 0x7f5f380c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f3c8ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 0x7f5f3815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f3c8ba640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f380adf00 0x7f5f380ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f3c8ba640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380af540 con 0x7f5f3815a850 2023-10-16T13:50:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f3c8ba640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3815b210 con 0x7f5f24003970 2023-10-16T13:50:51.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f3c8ba640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380aeca0 con 0x7f5f380adf00 2023-10-16T13:50:51.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 0x7f5f3815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 0x7f5f3815acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53710/0 (socket says 172.21.15.60:53710) 2023-10-16T13:50:51.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 -- 172.21.15.60:0/1600242121 learned_addr learned my addr 172.21.15.60:0/1600242121 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:51.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f3759e640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f380adf00 0x7f5f380ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 -- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f380adf00 msgr2=0x7f5f380ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24003970 0x7f5f380c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f380adf00 0x7f5f380ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 -- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24003970 msgr2=0x7f5f380c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24003970 0x7f5f380c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.661+0000 7f5f2ffff640 1 -- 172.21.15.60:0/1600242121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380aef80 con 0x7f5f3815a850 2023-10-16T13:50:51.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.662+0000 7f5f2ffff640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 0x7f5f3815acd0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000e510 tx=0x7f5f2000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:51.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.662+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20013610 con 0x7f5f3815a850 2023-10-16T13:50:51.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.662+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f20013770 con 0x7f5f3815a850 2023-10-16T13:50:51.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.662+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20013b20 con 0x7f5f3815a850 2023-10-16T13:50:51.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.662+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f380af260 con 0x7f5f3815a850 2023-10-16T13:50:51.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.662+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f380ae8f0 con 0x7f5f3815a850 2023-10-16T13:50:51.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.664+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3806b330 con 0x7f5f3815a850 2023-10-16T13:50:51.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.665+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f20013c80 con 0x7f5f3815a850 2023-10-16T13:50:51.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.665+0000 7f5f2f7fe640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f5f1806af90 0x7f5f1806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:51.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.665+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f200893e0 con 0x7f5f3815a850 2023-10-16T13:50:51.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.666+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f5f1806af90 0x7f5f1806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:51.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.667+0000 7f5f36d9d640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f5f1806af90 0x7f5f1806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5f28034000 tx=0x7f5f28009030 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:51.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.670+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f20054270 con 0x7f5f3815a850 2023-10-16T13:50:51.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.865+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 --> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f380cc550 con 0x7f5f1806af90 2023-10-16T13:50:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.895+0000 7f5f2f7fe640 1 -- 172.21.15.60:0/1600242121 <== mgr.7588 v2:172.21.15.144:6832/2817709727 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286385 (secure 0 0 0) 0x7f5f28005dc0 con 0x7f5f1806af90 2023-10-16T13:50:51.902 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:51.904 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T13:50:51.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f5f1806af90 msgr2=0x7f5f1806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f5f1806af90 0x7f5f1806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5f28034000 tx=0x7f5f28009030 comp rx=0 tx=0).stop 2023-10-16T13:50:51.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 msgr2=0x7f5f3815acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:51.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 0x7f5f3815acd0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000e510 tx=0x7f5f2000e9d0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 shutdown_connections 2023-10-16T13:50:51.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f5f1806af90 0x7f5f1806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f380adf00 0x7f5f380ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f3815a850 0x7f5f3815acd0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 --2- 172.21.15.60:0/1600242121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f24003970 0x7f5f380c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:51.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 >> 172.21.15.60:0/1600242121 conn(0x7f5f380ce6a0 msgr2=0x7f5f380c2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:51.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 shutdown_connections 2023-10-16T13:50:51.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:51.901+0000 7f5f3c8ba640 1 -- 172.21.15.60:0/1600242121 wait complete. 2023-10-16T13:50:51.925 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T13:50:51.793461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":487,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397488,"kb_used_data":18920,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754577232,"statfs":{"total":773094113280,"available":772687085568,"internally_reserved":0,"allocated":19374080,"data_stored":9974556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":387501624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002141"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.447571+0000","last_change":"2023-10-16T13:50:47.446902+0000","last_active":"2023-10-16T13:50:47.447571+0000","last_peered":"2023-10-16T13:50:47.447571+0000","last_clean":"2023-10-16T13:50:47.447571+0000","last_became_active":"2023-10-16T13:50:47.446622+0000","last_became_peered":"2023-10-16T13:50:47.446622+0000","last_unstale":"2023-10-16T13:50:47.447571+0000","last_undegraded":"2023-10-16T13:50:47.447571+0000","last_fullsized":"2023-10-16T13:50:47.447571+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:52.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435042+0000","last_change":"2023-10-16T13:50:47.430670+0000","last_active":"2023-10-16T13:50:47.435042+0000","last_peered":"2023-10-16T13:50:47.435042+0000","last_clean":"2023-10-16T13:50:47.435042+0000","last_became_active":"2023-10-16T13:50:47.430400+0000","last_became_peered":"2023-10-16T13:50:47.430400+0000","last_unstale":"2023-10-16T13:50:47.435042+0000","last_undegraded":"2023-10-16T13:50:47.435042+0000","last_fullsized":"2023-10-16T13:50:47.435042+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:45.392056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441000+0000","last_change":"2023-10-16T13:50:47.438393+0000","last_active":"2023-10-16T13:50:47.441000+0000","last_peered":"2023-10-16T13:50:47.441000+0000","last_clean":"2023-10-16T13:50:47.441000+0000","last_became_active":"2023-10-16T13:50:47.437861+0000","last_became_peered":"2023-10-16T13:50:47.437861+0000","last_unstale":"2023-10-16T13:50:47.441000+0000","last_undegraded":"2023-10-16T13:50:47.441000+0000","last_fullsized":"2023-10-16T13:50:47.441000+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:14.867925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440316+0000","last_change":"2023-10-16T13:50:47.443315+0000","last_active":"2023-10-16T13:50:48.440316+0000","last_peered":"2023-10-16T13:50:48.440316+0000","last_clean":"2023-10-16T13:50:48.440316+0000","last_became_active":"2023-10-16T13:50:47.443044+0000","last_became_peered":"2023-10-16T13:50:47.443044+0000","last_unstale":"2023-10-16T13:50:48.440316+0000","last_undegraded":"2023-10-16T13:50:48.440316+0000","last_fullsized":"2023-10-16T13:50:48.440316+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:29.644314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441722+0000","last_change":"2023-10-16T13:50:47.453828+0000","last_active":"2023-10-16T13:50:48.441722+0000","last_peered":"2023-10-16T13:50:48.441722+0000","last_clean":"2023-10-16T13:50:48.441722+0000","last_became_active":"2023-10-16T13:50:47.453513+0000","last_became_peered":"2023-10-16T13:50:47.453513+0000","last_unstale":"2023-10-16T13:50:48.441722+0000","last_undegraded":"2023-10-16T13:50:48.441722+0000","last_fullsized":"2023-10-16T13:50:48.441722+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:26.600063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.447174+0000","last_change":"2023-10-16T13:50:47.445657+0000","last_active":"2023-10-16T13:50:47.447174+0000","last_peered":"2023-10-16T13:50:47.447174+0000","last_clean":"2023-10-16T13:50:47.447174+0000","last_became_active":"2023-10-16T13:50:47.445367+0000","last_became_peered":"2023-10-16T13:50:47.445367+0000","last_unstale":"2023-10-16T13:50:47.447174+0000","last_undegraded":"2023-10-16T13:50:47.447174+0000","last_fullsized":"2023-10-16T13:50:47.447174+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:55.999228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433687+0000","last_change":"2023-10-16T13:50:47.424390+0000","last_active":"2023-10-16T13:50:48.433687+0000","last_peered":"2023-10-16T13:50:48.433687+0000","last_clean":"2023-10-16T13:50:48.433687+0000","last_became_active":"2023-10-16T13:50:47.423963+0000","last_became_peered":"2023-10-16T13:50:47.423963+0000","last_unstale":"2023-10-16T13:50:48.433687+0000","last_undegraded":"2023-10-16T13:50:48.433687+0000","last_fullsized":"2023-10-16T13:50:48.433687+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:20.823584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.439800+0000","last_change":"2023-10-16T13:50:47.435814+0000","last_active":"2023-10-16T13:50:47.439800+0000","last_peered":"2023-10-16T13:50:47.439800+0000","last_clean":"2023-10-16T13:50:47.439800+0000","last_became_active":"2023-10-16T13:50:47.435459+0000","last_became_peered":"2023-10-16T13:50:47.435459+0000","last_unstale":"2023-10-16T13:50:47.439800+0000","last_undegraded":"2023-10-16T13:50:47.439800+0000","last_fullsized":"2023-10-16T13:50:47.439800+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:46.633552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.431767+0000","last_change":"2023-10-16T13:50:47.429997+0000","last_active":"2023-10-16T13:50:47.431767+0000","last_peered":"2023-10-16T13:50:47.431767+0000","last_clean":"2023-10-16T13:50:47.431767+0000","last_became_active":"2023-10-16T13:50:47.428900+0000","last_became_peered":"2023-10-16T13:50:47.428900+0000","last_unstale":"2023-10-16T13:50:47.431767+0000","last_undegraded":"2023-10-16T13:50:47.431767+0000","last_fullsized":"2023-10-16T13:50:47.431767+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:41.964971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456367+0000","last_change":"2023-10-16T13:50:47.454181+0000","last_active":"2023-10-16T13:50:47.456367+0000","last_peered":"2023-10-16T13:50:47.456367+0000","last_clean":"2023-10-16T13:50:47.456367+0000","last_became_active":"2023-10-16T13:50:47.453881+0000","last_became_peered":"2023-10-16T13:50:47.453881+0000","last_unstale":"2023-10-16T13:50:47.456367+0000","last_undegraded":"2023-10-16T13:50:47.456367+0000","last_fullsized":"2023-10-16T13:50:47.456367+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:12.077989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.447358+0000","last_change":"2023-10-16T13:50:47.446252+0000","last_active":"2023-10-16T13:50:47.447358+0000","last_peered":"2023-10-16T13:50:47.447358+0000","last_clean":"2023-10-16T13:50:47.447358+0000","last_became_active":"2023-10-16T13:50:47.445986+0000","last_became_peered":"2023-10-16T13:50:47.445986+0000","last_unstale":"2023-10-16T13:50:47.447358+0000","last_undegraded":"2023-10-16T13:50:47.447358+0000","last_fullsized":"2023-10-16T13:50:47.447358+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:56.632400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.454157+0000","last_change":"2023-10-16T13:50:47.449625+0000","last_active":"2023-10-16T13:50:47.454157+0000","last_peered":"2023-10-16T13:50:47.454157+0000","last_clean":"2023-10-16T13:50:47.454157+0000","last_became_active":"2023-10-16T13:50:47.449301+0000","last_became_peered":"2023-10-16T13:50:47.449301+0000","last_unstale":"2023-10-16T13:50:47.454157+0000","last_undegraded":"2023-10-16T13:50:47.454157+0000","last_fullsized":"2023-10-16T13:50:47.454157+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.650281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.442081+0000","last_change":"2023-10-16T13:50:47.453578+0000","last_active":"2023-10-16T13:50:48.442081+0000","last_peered":"2023-10-16T13:50:48.442081+0000","last_clean":"2023-10-16T13:50:48.442081+0000","last_became_active":"2023-10-16T13:50:47.453211+0000","last_became_peered":"2023-10-16T13:50:47.453211+0000","last_unstale":"2023-10-16T13:50:48.442081+0000","last_undegraded":"2023-10-16T13:50:48.442081+0000","last_fullsized":"2023-10-16T13:50:48.442081+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:19.603609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.439640+0000","last_change":"2023-10-16T13:50:47.446745+0000","last_active":"2023-10-16T13:50:48.439640+0000","last_peered":"2023-10-16T13:50:48.439640+0000","last_clean":"2023-10-16T13:50:48.439640+0000","last_became_active":"2023-10-16T13:50:47.446274+0000","last_became_peered":"2023-10-16T13:50:47.446274+0000","last_unstale":"2023-10-16T13:50:48.439640+0000","last_undegraded":"2023-10-16T13:50:48.439640+0000","last_fullsized":"2023-10-16T13:50:48.439640+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:35.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455590+0000","last_change":"2023-10-16T13:50:47.452319+0000","last_active":"2023-10-16T13:50:47.455590+0000","last_peered":"2023-10-16T13:50:47.455590+0000","last_clean":"2023-10-16T13:50:47.455590+0000","last_became_active":"2023-10-16T13:50:47.452037+0000","last_became_peered":"2023-10-16T13:50:47.452037+0000","last_unstale":"2023-10-16T13:50:47.455590+0000","last_undegraded":"2023-10-16T13:50:47.455590+0000","last_fullsized":"2023-10-16T13:50:47.455590+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:57.437624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482704+0000","last_change":"2023-10-16T13:50:47.424611+0000","last_active":"2023-10-16T13:50:48.482704+0000","last_peered":"2023-10-16T13:50:48.482704+0000","last_clean":"2023-10-16T13:50:48.482704+0000","last_became_active":"2023-10-16T13:50:47.424164+0000","last_became_peered":"2023-10-16T13:50:47.424164+0000","last_unstale":"2023-10-16T13:50:48.482704+0000","last_undegraded":"2023-10-16T13:50:48.482704+0000","last_fullsized":"2023-10-16T13:50:48.482704+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:35.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455016+0000","last_change":"2023-10-16T13:50:47.449973+0000","last_active":"2023-10-16T13:50:47.455016+0000","last_peered":"2023-10-16T13:50:47.455016+0000","last_clean":"2023-10-16T13:50:47.455016+0000","last_became_active":"2023-10-16T13:50:47.449698+0000","last_became_peered":"2023-10-16T13:50:47.449698+0000","last_unstale":"2023-10-16T13:50:47.455016+0000","last_undegraded":"2023-10-16T13:50:47.455016+0000","last_fullsized":"2023-10-16T13:50:47.455016+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:11.213997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.448815+0000","last_change":"2023-10-16T13:50:47.444990+0000","last_active":"2023-10-16T13:50:47.448815+0000","last_peered":"2023-10-16T13:50:47.448815+0000","last_clean":"2023-10-16T13:50:47.448815+0000","last_became_active":"2023-10-16T13:50:47.444310+0000","last_became_peered":"2023-10-16T13:50:47.444310+0000","last_unstale":"2023-10-16T13:50:47.448815+0000","last_undegraded":"2023-10-16T13:50:47.448815+0000","last_fullsized":"2023-10-16T13:50:47.448815+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:25.416341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441026+0000","last_change":"2023-10-16T13:50:47.437739+0000","last_active":"2023-10-16T13:50:47.441026+0000","last_peered":"2023-10-16T13:50:47.441026+0000","last_clean":"2023-10-16T13:50:47.441026+0000","last_became_active":"2023-10-16T13:50:47.437260+0000","last_became_peered":"2023-10-16T13:50:47.437260+0000","last_unstale":"2023-10-16T13:50:47.441026+0000","last_undegraded":"2023-10-16T13:50:47.441026+0000","last_fullsized":"2023-10-16T13:50:47.441026+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:47.622962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482337+0000","last_change":"2023-10-16T13:50:47.442073+0000","last_active":"2023-10-16T13:50:48.482337+0000","last_peered":"2023-10-16T13:50:48.482337+0000","last_clean":"2023-10-16T13:50:48.482337+0000","last_became_active":"2023-10-16T13:50:47.436922+0000","last_became_peered":"2023-10-16T13:50:47.436922+0000","last_unstale":"2023-10-16T13:50:48.482337+0000","last_undegraded":"2023-10-16T13:50:48.482337+0000","last_fullsized":"2023-10-16T13:50:48.482337+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:49.466210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456296+0000","last_change":"2023-10-16T13:50:47.453756+0000","last_active":"2023-10-16T13:50:47.456296+0000","last_peered":"2023-10-16T13:50:47.456296+0000","last_clean":"2023-10-16T13:50:47.456296+0000","last_became_active":"2023-10-16T13:50:47.453463+0000","last_became_peered":"2023-10-16T13:50:47.453463+0000","last_unstale":"2023-10-16T13:50:47.456296+0000","last_undegraded":"2023-10-16T13:50:47.456296+0000","last_fullsized":"2023-10-16T13:50:47.456296+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:57.649795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440033+0000","last_change":"2023-10-16T13:50:47.433921+0000","last_active":"2023-10-16T13:50:48.440033+0000","last_peered":"2023-10-16T13:50:48.440033+0000","last_clean":"2023-10-16T13:50:48.440033+0000","last_became_active":"2023-10-16T13:50:47.433348+0000","last_became_peered":"2023-10-16T13:50:47.433348+0000","last_unstale":"2023-10-16T13:50:48.440033+0000","last_undegraded":"2023-10-16T13:50:48.440033+0000","last_fullsized":"2023-10-16T13:50:48.440033+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:55.414291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.439317+0000","last_change":"2023-10-16T13:50:47.421293+0000","last_active":"2023-10-16T13:50:48.439317+0000","last_peered":"2023-10-16T13:50:48.439317+0000","last_clean":"2023-10-16T13:50:48.439317+0000","last_became_active":"2023-10-16T13:50:47.420179+0000","last_became_peered":"2023-10-16T13:50:47.420179+0000","last_unstale":"2023-10-16T13:50:48.439317+0000","last_undegraded":"2023-10-16T13:50:48.439317+0000","last_fullsized":"2023-10-16T13:50:48.439317+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:47.435807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.453949+0000","last_change":"2023-10-16T13:50:47.448898+0000","last_active":"2023-10-16T13:50:47.453949+0000","last_peered":"2023-10-16T13:50:47.453949+0000","last_clean":"2023-10-16T13:50:47.453949+0000","last_became_active":"2023-10-16T13:50:47.447351+0000","last_became_peered":"2023-10-16T13:50:47.447351+0000","last_unstale":"2023-10-16T13:50:47.453949+0000","last_undegraded":"2023-10-16T13:50:47.453949+0000","last_fullsized":"2023-10-16T13:50:47.453949+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:43.237019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482708+0000","last_change":"2023-10-16T13:50:47.429796+0000","last_active":"2023-10-16T13:50:48.482708+0000","last_peered":"2023-10-16T13:50:48.482708+0000","last_clean":"2023-10-16T13:50:48.482708+0000","last_became_active":"2023-10-16T13:50:47.429542+0000","last_became_peered":"2023-10-16T13:50:47.429542+0000","last_unstale":"2023-10-16T13:50:48.482708+0000","last_undegraded":"2023-10-16T13:50:48.482708+0000","last_fullsized":"2023-10-16T13:50:48.482708+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:29.691226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.444231+0000","last_change":"2023-10-16T13:50:47.442555+0000","last_active":"2023-10-16T13:50:47.444231+0000","last_peered":"2023-10-16T13:50:47.444231+0000","last_clean":"2023-10-16T13:50:47.444231+0000","last_became_active":"2023-10-16T13:50:47.442026+0000","last_became_peered":"2023-10-16T13:50:47.442026+0000","last_unstale":"2023-10-16T13:50:47.444231+0000","last_undegraded":"2023-10-16T13:50:47.444231+0000","last_fullsized":"2023-10-16T13:50:47.444231+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:35.290078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482289+0000","last_change":"2023-10-16T13:50:47.442882+0000","last_active":"2023-10-16T13:50:48.482289+0000","last_peered":"2023-10-16T13:50:48.482289+0000","last_clean":"2023-10-16T13:50:48.482289+0000","last_became_active":"2023-10-16T13:50:47.442614+0000","last_became_peered":"2023-10-16T13:50:47.442614+0000","last_unstale":"2023-10-16T13:50:48.482289+0000","last_undegraded":"2023-10-16T13:50:48.482289+0000","last_fullsized":"2023-10-16T13:50:48.482289+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:48.324511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433745+0000","last_change":"2023-10-16T13:50:47.440701+0000","last_active":"2023-10-16T13:50:48.433745+0000","last_peered":"2023-10-16T13:50:48.433745+0000","last_clean":"2023-10-16T13:50:48.433745+0000","last_became_active":"2023-10-16T13:50:47.440330+0000","last_became_peered":"2023-10-16T13:50:47.440330+0000","last_unstale":"2023-10-16T13:50:48.433745+0000","last_undegraded":"2023-10-16T13:50:48.433745+0000","last_fullsized":"2023-10-16T13:50:48.433745+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:02.008233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.440928+0000","last_change":"2023-10-16T13:50:47.437603+0000","last_active":"2023-10-16T13:50:47.440928+0000","last_peered":"2023-10-16T13:50:47.440928+0000","last_clean":"2023-10-16T13:50:47.440928+0000","last_became_active":"2023-10-16T13:50:47.437310+0000","last_became_peered":"2023-10-16T13:50:47.437310+0000","last_unstale":"2023-10-16T13:50:47.440928+0000","last_undegraded":"2023-10-16T13:50:47.440928+0000","last_fullsized":"2023-10-16T13:50:47.440928+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:06.229083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440715+0000","last_change":"2023-10-16T13:50:47.433770+0000","last_active":"2023-10-16T13:50:48.440715+0000","last_peered":"2023-10-16T13:50:48.440715+0000","last_clean":"2023-10-16T13:50:48.440715+0000","last_became_active":"2023-10-16T13:50:47.432991+0000","last_became_peered":"2023-10-16T13:50:47.432991+0000","last_unstale":"2023-10-16T13:50:48.440715+0000","last_undegraded":"2023-10-16T13:50:48.440715+0000","last_fullsized":"2023-10-16T13:50:48.440715+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:02.919787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441669+0000","last_change":"2023-10-16T13:50:47.438082+0000","last_active":"2023-10-16T13:50:47.441669+0000","last_peered":"2023-10-16T13:50:47.441669+0000","last_clean":"2023-10-16T13:50:47.441669+0000","last_became_active":"2023-10-16T13:50:47.437634+0000","last_became_peered":"2023-10-16T13:50:47.437634+0000","last_unstale":"2023-10-16T13:50:47.441669+0000","last_undegraded":"2023-10-16T13:50:47.441669+0000","last_fullsized":"2023-10-16T13:50:47.441669+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.613016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482838+0000","last_change":"2023-10-16T13:50:47.436391+0000","last_active":"2023-10-16T13:50:48.482838+0000","last_peered":"2023-10-16T13:50:48.482838+0000","last_clean":"2023-10-16T13:50:48.482838+0000","last_became_active":"2023-10-16T13:50:47.436110+0000","last_became_peered":"2023-10-16T13:50:47.436110+0000","last_unstale":"2023-10-16T13:50:48.482838+0000","last_undegraded":"2023-10-16T13:50:48.482838+0000","last_fullsized":"2023-10-16T13:50:48.482838+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:29.644314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456388+0000","last_change":"2023-10-16T13:50:47.455166+0000","last_active":"2023-10-16T13:50:47.456388+0000","last_peered":"2023-10-16T13:50:47.456388+0000","last_clean":"2023-10-16T13:50:47.456388+0000","last_became_active":"2023-10-16T13:50:47.454848+0000","last_became_peered":"2023-10-16T13:50:47.454848+0000","last_unstale":"2023-10-16T13:50:47.456388+0000","last_undegraded":"2023-10-16T13:50:47.456388+0000","last_fullsized":"2023-10-16T13:50:47.456388+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:12.196989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"182'4","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.450806+0000","last_change":"2023-10-16T13:50:46.421041+0000","last_active":"2023-10-16T13:50:47.450806+0000","last_peered":"2023-10-16T13:50:47.450806+0000","last_clean":"2023-10-16T13:50:47.450806+0000","last_became_active":"2023-10-16T13:50:46.420682+0000","last_became_peered":"2023-10-16T13:50:46.420682+0000","last_unstale":"2023-10-16T13:50:47.450806+0000","last_undegraded":"2023-10-16T13:50:47.450806+0000","last_fullsized":"2023-10-16T13:50:47.450806+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:42.597620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435079+0000","last_change":"2023-10-16T13:50:47.430600+0000","last_active":"2023-10-16T13:50:47.435079+0000","last_peered":"2023-10-16T13:50:47.435079+0000","last_clean":"2023-10-16T13:50:47.435079+0000","last_became_active":"2023-10-16T13:50:47.430299+0000","last_became_peered":"2023-10-16T13:50:47.430299+0000","last_unstale":"2023-10-16T13:50:47.435079+0000","last_undegraded":"2023-10-16T13:50:47.435079+0000","last_fullsized":"2023-10-16T13:50:47.435079+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:06.877456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"182'1","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452079+0000","last_change":"2023-10-16T13:50:46.414025+0000","last_active":"2023-10-16T13:50:47.452079+0000","last_peered":"2023-10-16T13:50:47.452079+0000","last_clean":"2023-10-16T13:50:47.452079+0000","last_became_active":"2023-10-16T13:50:46.413741+0000","last_became_peered":"2023-10-16T13:50:46.413741+0000","last_unstale":"2023-10-16T13:50:47.452079+0000","last_undegraded":"2023-10-16T13:50:47.452079+0000","last_fullsized":"2023-10-16T13:50:47.452079+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:30.795279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.445773+0000","last_change":"2023-10-16T13:50:47.442976+0000","last_active":"2023-10-16T13:50:47.445773+0000","last_peered":"2023-10-16T13:50:47.445773+0000","last_clean":"2023-10-16T13:50:47.445773+0000","last_became_active":"2023-10-16T13:50:47.442580+0000","last_became_peered":"2023-10-16T13:50:47.442580+0000","last_unstale":"2023-10-16T13:50:47.445773+0000","last_undegraded":"2023-10-16T13:50:47.445773+0000","last_fullsized":"2023-10-16T13:50:47.445773+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:22.258673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"182'2","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440520+0000","last_change":"2023-10-16T13:50:46.416787+0000","last_active":"2023-10-16T13:50:48.440520+0000","last_peered":"2023-10-16T13:50:48.440520+0000","last_clean":"2023-10-16T13:50:48.440520+0000","last_became_active":"2023-10-16T13:50:46.414901+0000","last_became_peered":"2023-10-16T13:50:46.414901+0000","last_unstale":"2023-10-16T13:50:48.440520+0000","last_undegraded":"2023-10-16T13:50:48.440520+0000","last_fullsized":"2023-10-16T13:50:48.440520+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:15.493786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456006+0000","last_change":"2023-10-16T13:50:47.453838+0000","last_active":"2023-10-16T13:50:47.456006+0000","last_peered":"2023-10-16T13:50:47.456006+0000","last_clean":"2023-10-16T13:50:47.456006+0000","last_became_active":"2023-10-16T13:50:47.453577+0000","last_became_peered":"2023-10-16T13:50:47.453577+0000","last_unstale":"2023-10-16T13:50:47.456006+0000","last_undegraded":"2023-10-16T13:50:47.456006+0000","last_fullsized":"2023-10-16T13:50:47.456006+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:57.869284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"182'2","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440366+0000","last_change":"2023-10-16T13:50:46.416865+0000","last_active":"2023-10-16T13:50:48.440366+0000","last_peered":"2023-10-16T13:50:48.440366+0000","last_clean":"2023-10-16T13:50:48.440366+0000","last_became_active":"2023-10-16T13:50:46.414286+0000","last_became_peered":"2023-10-16T13:50:46.414286+0000","last_unstale":"2023-10-16T13:50:48.440366+0000","last_undegraded":"2023-10-16T13:50:48.440366+0000","last_fullsized":"2023-10-16T13:50:48.440366+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:39.848223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.442252+0000","last_change":"2023-10-16T13:50:47.439022+0000","last_active":"2023-10-16T13:50:47.442252+0000","last_peered":"2023-10-16T13:50:47.442252+0000","last_clean":"2023-10-16T13:50:47.442252+0000","last_became_active":"2023-10-16T13:50:47.438750+0000","last_became_peered":"2023-10-16T13:50:47.438750+0000","last_unstale":"2023-10-16T13:50:47.442252+0000","last_undegraded":"2023-10-16T13:50:47.442252+0000","last_fullsized":"2023-10-16T13:50:47.442252+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:46.587219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482336+0000","last_change":"2023-10-16T13:50:46.416541+0000","last_active":"2023-10-16T13:50:48.482336+0000","last_peered":"2023-10-16T13:50:48.482336+0000","last_clean":"2023-10-16T13:50:48.482336+0000","last_became_active":"2023-10-16T13:50:46.415280+0000","last_became_peered":"2023-10-16T13:50:46.415280+0000","last_unstale":"2023-10-16T13:50:48.482336+0000","last_undegraded":"2023-10-16T13:50:48.482336+0000","last_fullsized":"2023-10-16T13:50:48.482336+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:15.036950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.439877+0000","last_change":"2023-10-16T13:50:47.442655+0000","last_active":"2023-10-16T13:50:48.439877+0000","last_peered":"2023-10-16T13:50:48.439877+0000","last_clean":"2023-10-16T13:50:48.439877+0000","last_became_active":"2023-10-16T13:50:47.442336+0000","last_became_peered":"2023-10-16T13:50:47.442336+0000","last_unstale":"2023-10-16T13:50:48.439877+0000","last_undegraded":"2023-10-16T13:50:48.439877+0000","last_fullsized":"2023-10-16T13:50:48.439877+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:29.691226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441746+0000","last_change":"2023-10-16T13:50:46.418773+0000","last_active":"2023-10-16T13:50:48.441746+0000","last_peered":"2023-10-16T13:50:48.441746+0000","last_clean":"2023-10-16T13:50:48.441746+0000","last_became_active":"2023-10-16T13:50:46.418289+0000","last_became_peered":"2023-10-16T13:50:46.418289+0000","last_unstale":"2023-10-16T13:50:48.441746+0000","last_undegraded":"2023-10-16T13:50:48.441746+0000","last_fullsized":"2023-10-16T13:50:48.441746+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:39.445331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441660+0000","last_change":"2023-10-16T13:50:47.437974+0000","last_active":"2023-10-16T13:50:47.441660+0000","last_peered":"2023-10-16T13:50:47.441660+0000","last_clean":"2023-10-16T13:50:47.441660+0000","last_became_active":"2023-10-16T13:50:47.437695+0000","last_became_peered":"2023-10-16T13:50:47.437695+0000","last_unstale":"2023-10-16T13:50:47.441660+0000","last_undegraded":"2023-10-16T13:50:47.441660+0000","last_fullsized":"2023-10-16T13:50:47.441660+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:15.216018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482124+0000","last_change":"2023-10-16T13:50:46.407191+0000","last_active":"2023-10-16T13:50:48.482124+0000","last_peered":"2023-10-16T13:50:48.482124+0000","last_clean":"2023-10-16T13:50:48.482124+0000","last_became_active":"2023-10-16T13:50:46.406915+0000","last_became_peered":"2023-10-16T13:50:46.406915+0000","last_unstale":"2023-10-16T13:50:48.482124+0000","last_undegraded":"2023-10-16T13:50:48.482124+0000","last_fullsized":"2023-10-16T13:50:48.482124+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:15.493786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441619+0000","last_change":"2023-10-16T13:50:47.435775+0000","last_active":"2023-10-16T13:50:48.441619+0000","last_peered":"2023-10-16T13:50:48.441619+0000","last_clean":"2023-10-16T13:50:48.441619+0000","last_became_active":"2023-10-16T13:50:47.435437+0000","last_became_peered":"2023-10-16T13:50:47.435437+0000","last_unstale":"2023-10-16T13:50:48.441619+0000","last_undegraded":"2023-10-16T13:50:48.441619+0000","last_fullsized":"2023-10-16T13:50:48.441619+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:46.187873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.438866+0000","last_change":"2023-10-16T13:50:46.421579+0000","last_active":"2023-10-16T13:50:48.438866+0000","last_peered":"2023-10-16T13:50:48.438866+0000","last_clean":"2023-10-16T13:50:48.438866+0000","last_became_active":"2023-10-16T13:50:46.420836+0000","last_became_peered":"2023-10-16T13:50:46.420836+0000","last_unstale":"2023-10-16T13:50:48.438866+0000","last_undegraded":"2023-10-16T13:50:48.438866+0000","last_fullsized":"2023-10-16T13:50:48.438866+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:39.153337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.443028+0000","last_change":"2023-10-16T13:50:47.450407+0000","last_active":"2023-10-16T13:50:48.443028+0000","last_peered":"2023-10-16T13:50:48.443028+0000","last_clean":"2023-10-16T13:50:48.443028+0000","last_became_active":"2023-10-16T13:50:47.446352+0000","last_became_peered":"2023-10-16T13:50:47.446352+0000","last_unstale":"2023-10-16T13:50:48.443028+0000","last_undegraded":"2023-10-16T13:50:48.443028+0000","last_fullsized":"2023-10-16T13:50:48.443028+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:46.399833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.420815+0000","last_change":"2023-10-16T13:50:46.399594+0000","last_active":"2023-10-16T13:50:47.420815+0000","last_peered":"2023-10-16T13:50:47.420815+0000","last_clean":"2023-10-16T13:50:47.420815+0000","last_became_active":"2023-10-16T13:50:46.399300+0000","last_became_peered":"2023-10-16T13:50:46.399300+0000","last_unstale":"2023-10-16T13:50:47.420815+0000","last_undegraded":"2023-10-16T13:50:47.420815+0000","last_fullsized":"2023-10-16T13:50:47.420815+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:49.741068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.454992+0000","last_change":"2023-10-16T13:50:47.450141+0000","last_active":"2023-10-16T13:50:47.454992+0000","last_peered":"2023-10-16T13:50:47.454992+0000","last_clean":"2023-10-16T13:50:47.454992+0000","last_became_active":"2023-10-16T13:50:47.447000+0000","last_became_peered":"2023-10-16T13:50:47.447000+0000","last_unstale":"2023-10-16T13:50:47.454992+0000","last_undegraded":"2023-10-16T13:50:47.454992+0000","last_fullsized":"2023-10-16T13:50:47.454992+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:32.823753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435576+0000","last_change":"2023-10-16T13:50:46.422730+0000","last_active":"2023-10-16T13:50:47.435576+0000","last_peered":"2023-10-16T13:50:47.435576+0000","last_clean":"2023-10-16T13:50:47.435576+0000","last_became_active":"2023-10-16T13:50:46.422306+0000","last_became_peered":"2023-10-16T13:50:46.422306+0000","last_unstale":"2023-10-16T13:50:47.435576+0000","last_undegraded":"2023-10-16T13:50:47.435576+0000","last_fullsized":"2023-10-16T13:50:47.435576+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:38.217439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482611+0000","last_change":"2023-10-16T13:50:47.434906+0000","last_active":"2023-10-16T13:50:48.482611+0000","last_peered":"2023-10-16T13:50:48.482611+0000","last_clean":"2023-10-16T13:50:48.482611+0000","last_became_active":"2023-10-16T13:50:47.432077+0000","last_became_peered":"2023-10-16T13:50:47.432077+0000","last_unstale":"2023-10-16T13:50:48.482611+0000","last_undegraded":"2023-10-16T13:50:48.482611+0000","last_fullsized":"2023-10-16T13:50:48.482611+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:47.435807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"182'6","reported_seq":18,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433990+0000","last_change":"2023-10-16T13:50:46.408951+0000","last_active":"2023-10-16T13:50:48.433990+0000","last_peered":"2023-10-16T13:50:48.433990+0000","last_clean":"2023-10-16T13:50:48.433990+0000","last_became_active":"2023-10-16T13:50:46.408403+0000","last_became_peered":"2023-10-16T13:50:46.408403+0000","last_unstale":"2023-10-16T13:50:48.433990+0000","last_undegraded":"2023-10-16T13:50:48.433990+0000","last_fullsized":"2023-10-16T13:50:48.433990+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:41.001687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452765+0000","last_change":"2023-10-16T13:50:47.449520+0000","last_active":"2023-10-16T13:50:47.452765+0000","last_peered":"2023-10-16T13:50:47.452765+0000","last_clean":"2023-10-16T13:50:47.452765+0000","last_became_active":"2023-10-16T13:50:47.448963+0000","last_became_peered":"2023-10-16T13:50:47.448963+0000","last_unstale":"2023-10-16T13:50:47.452765+0000","last_undegraded":"2023-10-16T13:50:47.452765+0000","last_fullsized":"2023-10-16T13:50:47.452765+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:33.170205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"182'2","reported_seq":13,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.443960+0000","last_change":"2023-10-16T13:50:46.410822+0000","last_active":"2023-10-16T13:50:47.443960+0000","last_peered":"2023-10-16T13:50:47.443960+0000","last_clean":"2023-10-16T13:50:47.443960+0000","last_became_active":"2023-10-16T13:50:46.409436+0000","last_became_peered":"2023-10-16T13:50:46.409436+0000","last_unstale":"2023-10-16T13:50:47.443960+0000","last_undegraded":"2023-10-16T13:50:47.443960+0000","last_fullsized":"2023-10-16T13:50:47.443960+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:37.935030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435105+0000","last_change":"2023-10-16T13:50:47.430121+0000","last_active":"2023-10-16T13:50:47.435105+0000","last_peered":"2023-10-16T13:50:47.435105+0000","last_clean":"2023-10-16T13:50:47.435105+0000","last_became_active":"2023-10-16T13:50:47.429825+0000","last_became_peered":"2023-10-16T13:50:47.429825+0000","last_unstale":"2023-10-16T13:50:47.435105+0000","last_undegraded":"2023-10-16T13:50:47.435105+0000","last_fullsized":"2023-10-16T13:50:47.435105+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:09.347181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.427115+0000","last_change":"2023-10-16T13:50:46.408032+0000","last_active":"2023-10-16T13:50:47.427115+0000","last_peered":"2023-10-16T13:50:47.427115+0000","last_clean":"2023-10-16T13:50:47.427115+0000","last_became_active":"2023-10-16T13:50:46.407687+0000","last_became_peered":"2023-10-16T13:50:46.407687+0000","last_unstale":"2023-10-16T13:50:47.427115+0000","last_undegraded":"2023-10-16T13:50:47.427115+0000","last_fullsized":"2023-10-16T13:50:47.427115+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:15.319979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441624+0000","last_change":"2023-10-16T13:50:47.442856+0000","last_active":"2023-10-16T13:50:48.441624+0000","last_peered":"2023-10-16T13:50:48.441624+0000","last_clean":"2023-10-16T13:50:48.441624+0000","last_became_active":"2023-10-16T13:50:47.442527+0000","last_became_peered":"2023-10-16T13:50:47.442527+0000","last_unstale":"2023-10-16T13:50:48.441624+0000","last_undegraded":"2023-10-16T13:50:48.441624+0000","last_fullsized":"2023-10-16T13:50:48.441624+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:33.822906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.426296+0000","last_change":"2023-10-16T13:50:46.407144+0000","last_active":"2023-10-16T13:50:47.426296+0000","last_peered":"2023-10-16T13:50:47.426296+0000","last_clean":"2023-10-16T13:50:47.426296+0000","last_became_active":"2023-10-16T13:50:46.406806+0000","last_became_peered":"2023-10-16T13:50:46.406806+0000","last_unstale":"2023-10-16T13:50:47.426296+0000","last_undegraded":"2023-10-16T13:50:47.426296+0000","last_fullsized":"2023-10-16T13:50:47.426296+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:15.493786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452144+0000","last_change":"2023-10-16T13:50:47.447843+0000","last_active":"2023-10-16T13:50:47.452144+0000","last_peered":"2023-10-16T13:50:47.452144+0000","last_clean":"2023-10-16T13:50:47.452144+0000","last_became_active":"2023-10-16T13:50:47.447547+0000","last_became_peered":"2023-10-16T13:50:47.447547+0000","last_unstale":"2023-10-16T13:50:47.452144+0000","last_undegraded":"2023-10-16T13:50:47.452144+0000","last_fullsized":"2023-10-16T13:50:47.452144+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:28.352075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"182'8","reported_seq":20,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482945+0000","last_change":"2023-10-16T13:50:46.400068+0000","last_active":"2023-10-16T13:50:48.482945+0000","last_peered":"2023-10-16T13:50:48.482945+0000","last_clean":"2023-10-16T13:50:48.482945+0000","last_became_active":"2023-10-16T13:50:46.398885+0000","last_became_peered":"2023-10-16T13:50:46.398885+0000","last_unstale":"2023-10-16T13:50:48.482945+0000","last_undegraded":"2023-10-16T13:50:48.482945+0000","last_fullsized":"2023-10-16T13:50:48.482945+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:34.274026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455153+0000","last_change":"2023-10-16T13:50:47.450466+0000","last_active":"2023-10-16T13:50:47.455153+0000","last_peered":"2023-10-16T13:50:47.455153+0000","last_clean":"2023-10-16T13:50:47.455153+0000","last_became_active":"2023-10-16T13:50:47.447062+0000","last_became_peered":"2023-10-16T13:50:47.447062+0000","last_unstale":"2023-10-16T13:50:47.455153+0000","last_undegraded":"2023-10-16T13:50:47.455153+0000","last_fullsized":"2023-10-16T13:50:47.455153+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:28.413676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.442775+0000","last_change":"2023-10-16T13:50:46.413917+0000","last_active":"2023-10-16T13:50:48.442775+0000","last_peered":"2023-10-16T13:50:48.442775+0000","last_clean":"2023-10-16T13:50:48.442775+0000","last_became_active":"2023-10-16T13:50:46.413618+0000","last_became_peered":"2023-10-16T13:50:46.413618+0000","last_unstale":"2023-10-16T13:50:48.442775+0000","last_undegraded":"2023-10-16T13:50:48.442775+0000","last_fullsized":"2023-10-16T13:50:48.442775+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:14.420011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482721+0000","last_change":"2023-10-16T13:50:47.448728+0000","last_active":"2023-10-16T13:50:48.482721+0000","last_peered":"2023-10-16T13:50:48.482721+0000","last_clean":"2023-10-16T13:50:48.482721+0000","last_became_active":"2023-10-16T13:50:47.448399+0000","last_became_peered":"2023-10-16T13:50:47.448399+0000","last_unstale":"2023-10-16T13:50:48.482721+0000","last_undegraded":"2023-10-16T13:50:48.482721+0000","last_fullsized":"2023-10-16T13:50:48.482721+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:55.414291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"182'2","reported_seq":13,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.444664+0000","last_change":"2023-10-16T13:50:46.412005+0000","last_active":"2023-10-16T13:50:47.444664+0000","last_peered":"2023-10-16T13:50:47.444664+0000","last_clean":"2023-10-16T13:50:47.444664+0000","last_became_active":"2023-10-16T13:50:46.411635+0000","last_became_peered":"2023-10-16T13:50:46.411635+0000","last_unstale":"2023-10-16T13:50:47.444664+0000","last_undegraded":"2023-10-16T13:50:47.444664+0000","last_fullsized":"2023-10-16T13:50:47.444664+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:41.253896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452594+0000","last_change":"2023-10-16T13:50:47.449400+0000","last_active":"2023-10-16T13:50:47.452594+0000","last_peered":"2023-10-16T13:50:47.452594+0000","last_clean":"2023-10-16T13:50:47.452594+0000","last_became_active":"2023-10-16T13:50:47.449076+0000","last_became_peered":"2023-10-16T13:50:47.449076+0000","last_unstale":"2023-10-16T13:50:47.452594+0000","last_undegraded":"2023-10-16T13:50:47.452594+0000","last_fullsized":"2023-10-16T13:50:47.452594+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:08.089747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482752+0000","last_change":"2023-10-16T13:50:46.418621+0000","last_active":"2023-10-16T13:50:48.482752+0000","last_peered":"2023-10-16T13:50:48.482752+0000","last_clean":"2023-10-16T13:50:48.482752+0000","last_became_active":"2023-10-16T13:50:46.418059+0000","last_became_peered":"2023-10-16T13:50:46.418059+0000","last_unstale":"2023-10-16T13:50:48.482752+0000","last_undegraded":"2023-10-16T13:50:48.482752+0000","last_fullsized":"2023-10-16T13:50:48.482752+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:39.153337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441527+0000","last_change":"2023-10-16T13:50:47.439032+0000","last_active":"2023-10-16T13:50:47.441527+0000","last_peered":"2023-10-16T13:50:47.441527+0000","last_clean":"2023-10-16T13:50:47.441527+0000","last_became_active":"2023-10-16T13:50:47.438774+0000","last_became_peered":"2023-10-16T13:50:47.438774+0000","last_unstale":"2023-10-16T13:50:47.441527+0000","last_undegraded":"2023-10-16T13:50:47.441527+0000","last_fullsized":"2023-10-16T13:50:47.441527+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:56.415282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.420453+0000","last_change":"2023-10-16T13:50:46.401553+0000","last_active":"2023-10-16T13:50:47.420453+0000","last_peered":"2023-10-16T13:50:47.420453+0000","last_clean":"2023-10-16T13:50:47.420453+0000","last_became_active":"2023-10-16T13:50:46.400652+0000","last_became_peered":"2023-10-16T13:50:46.400652+0000","last_unstale":"2023-10-16T13:50:47.420453+0000","last_undegraded":"2023-10-16T13:50:47.420453+0000","last_fullsized":"2023-10-16T13:50:47.420453+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:14.814624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440900+0000","last_change":"2023-10-16T13:50:47.433610+0000","last_active":"2023-10-16T13:50:48.440900+0000","last_peered":"2023-10-16T13:50:48.440900+0000","last_clean":"2023-10-16T13:50:48.440900+0000","last_became_active":"2023-10-16T13:50:47.432591+0000","last_became_peered":"2023-10-16T13:50:47.432591+0000","last_unstale":"2023-10-16T13:50:48.440900+0000","last_undegraded":"2023-10-16T13:50:48.440900+0000","last_fullsized":"2023-10-16T13:50:48.440900+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:55.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"182'2","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433973+0000","last_change":"2023-10-16T13:50:46.413858+0000","last_active":"2023-10-16T13:50:48.433973+0000","last_peered":"2023-10-16T13:50:48.433973+0000","last_clean":"2023-10-16T13:50:48.433973+0000","last_became_active":"2023-10-16T13:50:46.413462+0000","last_became_peered":"2023-10-16T13:50:46.413462+0000","last_unstale":"2023-10-16T13:50:48.433973+0000","last_undegraded":"2023-10-16T13:50:48.433973+0000","last_fullsized":"2023-10-16T13:50:48.433973+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:56:27.628143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433843+0000","last_change":"2023-10-16T13:50:47.423159+0000","last_active":"2023-10-16T13:50:48.433843+0000","last_peered":"2023-10-16T13:50:48.433843+0000","last_clean":"2023-10-16T13:50:48.433843+0000","last_became_active":"2023-10-16T13:50:47.422652+0000","last_became_peered":"2023-10-16T13:50:47.422652+0000","last_unstale":"2023-10-16T13:50:48.433843+0000","last_undegraded":"2023-10-16T13:50:48.433843+0000","last_fullsized":"2023-10-16T13:50:48.433843+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:17.998148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.428505+0000","last_change":"2023-10-16T13:50:46.417773+0000","last_active":"2023-10-16T13:50:47.428505+0000","last_peered":"2023-10-16T13:50:47.428505+0000","last_clean":"2023-10-16T13:50:47.428505+0000","last_became_active":"2023-10-16T13:50:46.417402+0000","last_became_peered":"2023-10-16T13:50:46.417402+0000","last_unstale":"2023-10-16T13:50:47.428505+0000","last_undegraded":"2023-10-16T13:50:47.428505+0000","last_fullsized":"2023-10-16T13:50:47.428505+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:06.931829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.438819+0000","last_change":"2023-10-16T13:50:47.418864+0000","last_active":"2023-10-16T13:50:48.438819+0000","last_peered":"2023-10-16T13:50:48.438819+0000","last_clean":"2023-10-16T13:50:48.438819+0000","last_became_active":"2023-10-16T13:50:47.417722+0000","last_became_peered":"2023-10-16T13:50:47.417722+0000","last_unstale":"2023-10-16T13:50:48.438819+0000","last_undegraded":"2023-10-16T13:50:48.438819+0000","last_fullsized":"2023-10-16T13:50:48.438819+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:49.466210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.429793+0000","last_change":"2023-10-16T13:50:46.416881+0000","last_active":"2023-10-16T13:50:47.429793+0000","last_peered":"2023-10-16T13:50:47.429793+0000","last_clean":"2023-10-16T13:50:47.429793+0000","last_became_active":"2023-10-16T13:50:46.416573+0000","last_became_peered":"2023-10-16T13:50:46.416573+0000","last_unstale":"2023-10-16T13:50:47.429793+0000","last_undegraded":"2023-10-16T13:50:47.429793+0000","last_fullsized":"2023-10-16T13:50:47.429793+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:11.247133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433549+0000","last_change":"2023-10-16T13:50:47.422780+0000","last_active":"2023-10-16T13:50:48.433549+0000","last_peered":"2023-10-16T13:50:48.433549+0000","last_clean":"2023-10-16T13:50:48.433549+0000","last_became_active":"2023-10-16T13:50:47.422418+0000","last_became_peered":"2023-10-16T13:50:47.422418+0000","last_unstale":"2023-10-16T13:50:48.433549+0000","last_undegraded":"2023-10-16T13:50:48.433549+0000","last_fullsized":"2023-10-16T13:50:48.433549+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:54.282098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.424763+0000","last_change":"2023-10-16T13:50:46.406731+0000","last_active":"2023-10-16T13:50:47.424763+0000","last_peered":"2023-10-16T13:50:47.424763+0000","last_clean":"2023-10-16T13:50:47.424763+0000","last_became_active":"2023-10-16T13:50:46.406451+0000","last_became_peered":"2023-10-16T13:50:46.406451+0000","last_unstale":"2023-10-16T13:50:47.424763+0000","last_undegraded":"2023-10-16T13:50:47.424763+0000","last_fullsized":"2023-10-16T13:50:47.424763+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:17.905413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.443207+0000","last_change":"2023-10-16T13:50:47.449297+0000","last_active":"2023-10-16T13:50:48.443207+0000","last_peered":"2023-10-16T13:50:48.443207+0000","last_clean":"2023-10-16T13:50:48.443207+0000","last_became_active":"2023-10-16T13:50:47.447338+0000","last_became_peered":"2023-10-16T13:50:47.447338+0000","last_unstale":"2023-10-16T13:50:48.443207+0000","last_undegraded":"2023-10-16T13:50:48.443207+0000","last_fullsized":"2023-10-16T13:50:48.443207+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:09.153279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"182'2","reported_seq":13,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.450581+0000","last_change":"2023-10-16T13:50:46.421826+0000","last_active":"2023-10-16T13:50:47.450581+0000","last_peered":"2023-10-16T13:50:47.450581+0000","last_clean":"2023-10-16T13:50:47.450581+0000","last_became_active":"2023-10-16T13:50:46.421397+0000","last_became_peered":"2023-10-16T13:50:46.421397+0000","last_unstale":"2023-10-16T13:50:47.450581+0000","last_undegraded":"2023-10-16T13:50:47.450581+0000","last_fullsized":"2023-10-16T13:50:47.450581+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:31.564366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.454127+0000","last_change":"2023-10-16T13:50:47.452419+0000","last_active":"2023-10-16T13:50:47.454127+0000","last_peered":"2023-10-16T13:50:47.454127+0000","last_clean":"2023-10-16T13:50:47.454127+0000","last_became_active":"2023-10-16T13:50:47.451831+0000","last_became_peered":"2023-10-16T13:50:47.451831+0000","last_unstale":"2023-10-16T13:50:47.454127+0000","last_undegraded":"2023-10-16T13:50:47.454127+0000","last_fullsized":"2023-10-16T13:50:47.454127+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:50.067196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.425789+0000","last_change":"2023-10-16T13:50:46.415644+0000","last_active":"2023-10-16T13:50:47.425789+0000","last_peered":"2023-10-16T13:50:47.425789+0000","last_clean":"2023-10-16T13:50:47.425789+0000","last_became_active":"2023-10-16T13:50:46.414611+0000","last_became_peered":"2023-10-16T13:50:46.414611+0000","last_unstale":"2023-10-16T13:50:47.425789+0000","last_undegraded":"2023-10-16T13:50:47.425789+0000","last_fullsized":"2023-10-16T13:50:47.425789+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:39.153337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.430449+0000","last_change":"2023-10-16T13:50:47.428380+0000","last_active":"2023-10-16T13:50:47.430449+0000","last_peered":"2023-10-16T13:50:47.430449+0000","last_clean":"2023-10-16T13:50:47.430449+0000","last_became_active":"2023-10-16T13:50:47.428022+0000","last_became_peered":"2023-10-16T13:50:47.428022+0000","last_unstale":"2023-10-16T13:50:47.430449+0000","last_undegraded":"2023-10-16T13:50:47.430449+0000","last_fullsized":"2023-10-16T13:50:47.430449+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:00.922242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.426929+0000","last_change":"2023-10-16T13:50:46.399487+0000","last_active":"2023-10-16T13:50:47.426929+0000","last_peered":"2023-10-16T13:50:47.426929+0000","last_clean":"2023-10-16T13:50:47.426929+0000","last_became_active":"2023-10-16T13:50:46.399200+0000","last_became_peered":"2023-10-16T13:50:46.399200+0000","last_unstale":"2023-10-16T13:50:47.426929+0000","last_undegraded":"2023-10-16T13:50:47.426929+0000","last_fullsized":"2023-10-16T13:50:47.426929+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:47.308458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.446796+0000","last_change":"2023-10-16T13:50:47.443207+0000","last_active":"2023-10-16T13:50:47.446796+0000","last_peered":"2023-10-16T13:50:47.446796+0000","last_clean":"2023-10-16T13:50:47.446796+0000","last_became_active":"2023-10-16T13:50:47.442928+0000","last_became_peered":"2023-10-16T13:50:47.442928+0000","last_unstale":"2023-10-16T13:50:47.446796+0000","last_undegraded":"2023-10-16T13:50:47.446796+0000","last_fullsized":"2023-10-16T13:50:47.446796+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:58.450740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.426318+0000","last_change":"2023-10-16T13:50:46.407783+0000","last_active":"2023-10-16T13:50:47.426318+0000","last_peered":"2023-10-16T13:50:47.426318+0000","last_clean":"2023-10-16T13:50:47.426318+0000","last_became_active":"2023-10-16T13:50:46.405811+0000","last_became_peered":"2023-10-16T13:50:46.405811+0000","last_unstale":"2023-10-16T13:50:47.426318+0000","last_undegraded":"2023-10-16T13:50:47.426318+0000","last_fullsized":"2023-10-16T13:50:47.426318+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:39.848223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482237+0000","last_change":"2023-10-16T13:50:47.443439+0000","last_active":"2023-10-16T13:50:48.482237+0000","last_peered":"2023-10-16T13:50:48.482237+0000","last_clean":"2023-10-16T13:50:48.482237+0000","last_became_active":"2023-10-16T13:50:47.443158+0000","last_became_peered":"2023-10-16T13:50:47.443158+0000","last_unstale":"2023-10-16T13:50:48.482237+0000","last_undegraded":"2023-10-16T13:50:48.482237+0000","last_fullsized":"2023-10-16T13:50:48.482237+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:47.622962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.434037+0000","last_change":"2023-10-16T13:50:46.416371+0000","last_active":"2023-10-16T13:50:48.434037+0000","last_peered":"2023-10-16T13:50:48.434037+0000","last_clean":"2023-10-16T13:50:48.434037+0000","last_became_active":"2023-10-16T13:50:46.415632+0000","last_became_peered":"2023-10-16T13:50:46.415632+0000","last_unstale":"2023-10-16T13:50:48.434037+0000","last_undegraded":"2023-10-16T13:50:48.434037+0000","last_fullsized":"2023-10-16T13:50:48.434037+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:37.640302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.442046+0000","last_change":"2023-10-16T13:50:47.440695+0000","last_active":"2023-10-16T13:50:48.442046+0000","last_peered":"2023-10-16T13:50:48.442046+0000","last_clean":"2023-10-16T13:50:48.442046+0000","last_became_active":"2023-10-16T13:50:47.440155+0000","last_became_peered":"2023-10-16T13:50:47.440155+0000","last_unstale":"2023-10-16T13:50:48.442046+0000","last_undegraded":"2023-10-16T13:50:48.442046+0000","last_fullsized":"2023-10-16T13:50:48.442046+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:40.246993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.428682+0000","last_change":"2023-10-16T13:50:46.419576+0000","last_active":"2023-10-16T13:50:47.428682+0000","last_peered":"2023-10-16T13:50:47.428682+0000","last_clean":"2023-10-16T13:50:47.428682+0000","last_became_active":"2023-10-16T13:50:46.419260+0000","last_became_peered":"2023-10-16T13:50:46.419260+0000","last_unstale":"2023-10-16T13:50:47.428682+0000","last_undegraded":"2023-10-16T13:50:47.428682+0000","last_fullsized":"2023-10-16T13:50:47.428682+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:00.287093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455841+0000","last_change":"2023-10-16T13:50:47.452620+0000","last_active":"2023-10-16T13:50:47.455841+0000","last_peered":"2023-10-16T13:50:47.455841+0000","last_clean":"2023-10-16T13:50:47.455841+0000","last_became_active":"2023-10-16T13:50:47.452219+0000","last_became_peered":"2023-10-16T13:50:47.452219+0000","last_unstale":"2023-10-16T13:50:47.455841+0000","last_undegraded":"2023-10-16T13:50:47.455841+0000","last_fullsized":"2023-10-16T13:50:47.455841+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:09.851640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.431013+0000","last_change":"2023-10-16T13:50:46.409440+0000","last_active":"2023-10-16T13:50:47.431013+0000","last_peered":"2023-10-16T13:50:47.431013+0000","last_clean":"2023-10-16T13:50:47.431013+0000","last_became_active":"2023-10-16T13:50:46.408811+0000","last_became_peered":"2023-10-16T13:50:46.408811+0000","last_unstale":"2023-10-16T13:50:47.431013+0000","last_undegraded":"2023-10-16T13:50:47.431013+0000","last_fullsized":"2023-10-16T13:50:47.431013+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:20.281225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441970+0000","last_change":"2023-10-16T13:50:47.429854+0000","last_active":"2023-10-16T13:50:48.441970+0000","last_peered":"2023-10-16T13:50:48.441970+0000","last_clean":"2023-10-16T13:50:48.441970+0000","last_became_active":"2023-10-16T13:50:47.429546+0000","last_became_peered":"2023-10-16T13:50:47.429546+0000","last_unstale":"2023-10-16T13:50:48.441970+0000","last_undegraded":"2023-10-16T13:50:48.441970+0000","last_fullsized":"2023-10-16T13:50:48.441970+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:54.353532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.420391+0000","last_change":"2023-10-16T13:50:46.414842+0000","last_active":"2023-10-16T13:50:47.420391+0000","last_peered":"2023-10-16T13:50:47.420391+0000","last_clean":"2023-10-16T13:50:47.420391+0000","last_became_active":"2023-10-16T13:50:46.414517+0000","last_became_peered":"2023-10-16T13:50:46.414517+0000","last_unstale":"2023-10-16T13:50:47.420391+0000","last_undegraded":"2023-10-16T13:50:47.420391+0000","last_fullsized":"2023-10-16T13:50:47.420391+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:52.012069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.434076+0000","last_change":"2023-10-16T13:50:47.423316+0000","last_active":"2023-10-16T13:50:48.434076+0000","last_peered":"2023-10-16T13:50:48.434076+0000","last_clean":"2023-10-16T13:50:48.434076+0000","last_became_active":"2023-10-16T13:50:47.423013+0000","last_became_peered":"2023-10-16T13:50:47.423013+0000","last_unstale":"2023-10-16T13:50:48.434076+0000","last_undegraded":"2023-10-16T13:50:48.434076+0000","last_fullsized":"2023-10-16T13:50:48.434076+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:54.540727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"182'3","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.444802+0000","last_change":"2023-10-16T13:50:46.406671+0000","last_active":"2023-10-16T13:50:47.444802+0000","last_peered":"2023-10-16T13:50:47.444802+0000","last_clean":"2023-10-16T13:50:47.444802+0000","last_became_active":"2023-10-16T13:50:46.406328+0000","last_became_peered":"2023-10-16T13:50:46.406328+0000","last_unstale":"2023-10-16T13:50:47.444802+0000","last_undegraded":"2023-10-16T13:50:47.444802+0000","last_fullsized":"2023-10-16T13:50:47.444802+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:46.887415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"170'274","reported_seq":899,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-16T13:50:46.413474+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T13:50:46.413474+0000","last_peered":"2023-10-16T13:50:46.413474+0000","last_clean":"2023-10-16T13:50:46.413474+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T13:50:46.413474+0000","last_undegraded":"2023-10-16T13:50:46.413474+0000","last_fullsized":"2023-10-16T13:50:46.413474+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543067,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47872,"kb_used_data":2800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2867200,"data_stored":1689645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3320000000000001,"15min":1.327},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.73999999999999999},"max":{"1min":1.7729999999999999,"5min":1.835,"15min":1.976},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.423,"15min":1.407},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.302,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.294}]},{"osd":1,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1779999999999999,"15min":1.218},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6279999999999999,"5min":1.637,"15min":1.8100000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.321,"15min":1.28},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.67900000000000005},"max":{"1min":1.901,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.6160000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.276,"15min":1.274},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.5529999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3440000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7450000000000001,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.718}]},{"osd":3,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.399,"15min":1.385},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.8120000000000001,"5min":1.871,"15min":2.1070000000000002},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4710000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.84999999999999998},"max":{"1min":2.093,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.7829999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3,"15min":1.2609999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.976,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.494},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.446,"15min":1.399},"min":{"1min":0.96699999999999997,"5min":0.84399999999999997,"15min":0.81599999999999995},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.1829999999999998},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.4379999999999999,"15min":1.399},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.1880000000000002,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":1.0389999999999999,"5min":1.0369999999999999,"15min":0.94199999999999995},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":2.9700000000000002},"last":1.6699999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.637,"15min":1.6180000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.2229999999999999,"5min":2.306,"15min":2.306},"last":1.74},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.532,"15min":1.524},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85199999999999998},"max":{"1min":1.8919999999999999,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.8560000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543068,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2277376,"data_stored":1098771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.349,"15min":1.3420000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.69599999999999995},"max":{"1min":1.748,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.8049999999999999,"5min":1.8169999999999999,"15min":1.9990000000000001},"last":1.476}]},{"osd":1,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.208,"15min":1.23},"min":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.82099999999999995},"max":{"1min":1.6299999999999999,"5min":1.6319999999999999,"15min":1.8280000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2270000000000001,"15min":1.232},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6890000000000001,"5min":1.776,"15min":1.788},"last":1.224}]},{"osd":2,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2090000000000001,"15min":1.254},"min":{"1min":0.89900000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6990000000000001,"5min":1.897,"15min":2.0350000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.325,"15min":1.3240000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.72,"5min":1.77,"15min":1.859},"last":1.675}]},{"osd":3,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.3759999999999999},"min":{"1min":0.93999999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.764,"5min":1.946,"15min":1.9850000000000001},"last":1.127},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4039999999999999,"15min":1.399},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.8200000000000001,"5min":1.8400000000000001,"15min":1.9159999999999999},"last":1.8029999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.395,"15min":1.3779999999999999},"min":{"1min":1.1020000000000001,"5min":0.86799999999999999,"15min":0.79600000000000004},"max":{"1min":1.7529999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.542,"15min":1.5369999999999999},"min":{"1min":1.3440000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.02,"5min":2.02,"15min":2.1219999999999999},"last":1.891}]},{"osd":5,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.359,"15min":1.331},"min":{"1min":0.876,"5min":0.86899999999999999,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":1.903,"15min":2.2669999999999999},"last":1.546},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.498,"15min":1.472},"min":{"1min":0.91500000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.895,"5min":1.9690000000000001,"15min":2.0510000000000002},"last":1.2969999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.4810000000000001},"min":{"1min":1.2190000000000001,"5min":0.88700000000000001,"15min":0.87},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.173},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.548,"15min":1.512},"min":{"1min":1.1779999999999999,"5min":1.0129999999999999,"15min":0.92500000000000004},"max":{"1min":1.923,"5min":1.9850000000000001,"15min":2.1419999999999999},"last":1.8440000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543069,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2269184,"data_stored":1098647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2390000000000001,"15min":1.218},"min":{"1min":0.96699999999999997,"5min":0.72399999999999998,"15min":0.70099999999999996},"max":{"1min":1.6990000000000001,"5min":1.78,"15min":1.821},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.2689999999999999,"15min":1.2769999999999999},"min":{"1min":0.96599999999999997,"5min":0.83399999999999996,"15min":0.745},"max":{"1min":1.8,"5min":1.839,"15min":1.9610000000000001},"last":1.046}]},{"osd":1,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3089999999999999,"15min":1.3160000000000001},"min":{"1min":0.92900000000000005,"5min":0.89200000000000002,"15min":0.82799999999999996},"max":{"1min":1.669,"5min":1.992,"15min":2.1640000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.256,"15min":1.272},"min":{"1min":1.024,"5min":0.84999999999999998,"15min":0.76000000000000001},"max":{"1min":1.694,"5min":1.754,"15min":2.431},"last":1.165}]},{"osd":2,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3460000000000001,"15min":1.3540000000000001},"min":{"1min":1.1619999999999999,"5min":0.94099999999999995,"15min":0.90200000000000002},"max":{"1min":1.6890000000000001,"5min":1.8029999999999999,"15min":2.0209999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3440000000000001,"15min":1.3300000000000001},"min":{"1min":1.1000000000000001,"5min":0.89800000000000002,"15min":0.876},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.93500000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.373,"15min":1.3979999999999999},"min":{"1min":1.03,"5min":0.98299999999999998,"15min":0.95499999999999996},"max":{"1min":1.746,"5min":2.0990000000000002,"15min":2.5289999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.4670000000000001,"15min":1.4490000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.93899999999999995},"max":{"1min":1.7589999999999999,"5min":1.952,"15min":2.242},"last":1.4239999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.3440000000000001,"15min":1.2909999999999999},"min":{"1min":0.88900000000000001,"5min":0.82399999999999995,"15min":0.69399999999999995},"max":{"1min":1.782,"5min":1.8,"15min":2.3679999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.446,"15min":1.423},"min":{"1min":0.93999999999999995,"5min":0.81599999999999995,"15min":0.80300000000000005},"max":{"1min":1.853,"5min":2.2530000000000001,"15min":2.4020000000000001},"last":1.4690000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4319999999999999,"15min":1.383},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.88},"max":{"1min":1.883,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.579},{"interface":"front","average":{"1min":1.528,"5min":1.494,"15min":1.48},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":0.97299999999999998},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.3460000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.532},"min":{"1min":1.1459999999999999,"5min":0.96999999999999997,"15min":0.93300000000000005},"max":{"1min":1.96,"5min":2.371,"15min":2.6320000000000001},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.617,"15min":1.5980000000000001},"min":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0089999999999999},"max":{"1min":1.9470000000000001,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.522}]}]},{"osd":4,"up_from":14,"seq":60129543069,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55608,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316232,"statfs":{"total":96636764160,"available":96579821568,"internally_reserved":0,"allocated":2265088,"data_stored":1098625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2450000000000001,"15min":1.234},"min":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.69499999999999995},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.194,"15min":1.2170000000000001},"min":{"1min":0.86899999999999999,"5min":0.82399999999999995,"15min":0.77900000000000003},"max":{"1min":1.5049999999999999,"5min":1.645,"15min":2.073},"last":0.999}]},{"osd":1,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.27,"15min":1.2490000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.82999999999999996},"max":{"1min":1.5549999999999999,"5min":1.8049999999999999,"15min":1.8069999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.161,"5min":1.29,"15min":1.3080000000000001},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.83399999999999996},"max":{"1min":1.6080000000000001,"5min":1.7729999999999999,"15min":1.887},"last":0.67200000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3069999999999999,"15min":1.2989999999999999},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.88400000000000001},"max":{"1min":1.651,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.081},{"interface":"front","average":{"1min":1.282,"5min":1.3799999999999999,"15min":1.411},"min":{"1min":1.0449999999999999,"5min":1.0149999999999999,"15min":0.97599999999999998},"max":{"1min":1.528,"5min":1.861,"15min":2.1840000000000002},"last":0.88800000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.3029999999999999,"15min":1.3100000000000001},"min":{"1min":0.97999999999999998,"5min":0.90000000000000002,"15min":0.84899999999999998},"max":{"1min":1.7430000000000001,"5min":1.7909999999999999,"15min":2.052},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.403,"5min":1.4119999999999999,"15min":1.4299999999999999},"min":{"1min":1.135,"5min":0.995,"15min":0.90600000000000003},"max":{"1min":1.6479999999999999,"5min":2.294,"15min":2.294},"last":1.2130000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3460000000000001,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.089},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.339,"5min":1.3320000000000001,"15min":1.325},"min":{"1min":0.871,"5min":0.871,"15min":0.752},"max":{"1min":1.8979999999999999,"5min":2.2360000000000002,"15min":2.4430000000000001},"last":1.147}]},{"osd":6,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3640000000000001,"15min":1.3859999999999999},"min":{"1min":0.85199999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.8109999999999999,"5min":2.121,"15min":2.383},"last":1.371},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5029999999999999,"15min":1.512},"min":{"1min":1.069,"5min":1.0269999999999999,"15min":0.89000000000000001},"max":{"1min":2.1030000000000002,"5min":2.1030000000000002,"15min":2.1549999999999998},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4490000000000001,"15min":1.4910000000000001},"min":{"1min":1.036,"5min":0.92800000000000005,"15min":0.92600000000000005},"max":{"1min":1.9650000000000001,"5min":2.3570000000000002,"15min":2.6459999999999999},"last":1.274},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.6240000000000001,"15min":1.6399999999999999},"min":{"1min":1.417,"5min":1.103,"15min":1.004},"max":{"1min":2.0179999999999998,"5min":2.4049999999999998,"15min":2.6030000000000002},"last":1.3140000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51448,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320392,"statfs":{"total":96636764160,"available":96584081408,"internally_reserved":0,"allocated":2269184,"data_stored":1098715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97099999999999997,"15min":0.97999999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.04,"5min":2.04,"15min":3.1000000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.242,"15min":1.2390000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.74099999999999999},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":2.714},"last":0.90400000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.133,"15min":1.1020000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.73299999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.143,"15min":1.139},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"max":{"1min":1.5529999999999999,"5min":1.879,"15min":1.879},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.1919999999999999,"15min":1.206},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.234,"5min":2.234,"15min":2.782},"last":0.627},{"interface":"front","average":{"1min":1.5,"5min":1.468,"15min":1.48},"min":{"1min":1.242,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.609},"last":0.84199999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.5009999999999999,"15min":1.524},"min":{"1min":1.28,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":1.8160000000000001,"5min":1.8979999999999999,"15min":2.2330000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.4379999999999999,"15min":1.472},"min":{"1min":1.022,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.1000000000000001},"last":1.222}]},{"osd":5,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.4530000000000001,"15min":1.4650000000000001},"min":{"1min":1.137,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.0990000000000002,"5min":2.1110000000000002,"15min":2.306},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.615,"5min":1.5649999999999999,"15min":1.5369999999999999},"min":{"1min":1.2350000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.2559999999999998,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.325}]},{"osd":6,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.0900000000000001,"5min":1.0489999999999999,"15min":1.032},"max":{"1min":2.028,"5min":2.1389999999999998,"15min":2.4430000000000001},"last":1.129},{"interface":"front","average":{"1min":1.532,"5min":1.516,"15min":1.506},"min":{"1min":1.177,"5min":1.1339999999999999,"15min":1.056},"max":{"1min":2.1779999999999999,"5min":2.2519999999999998,"15min":2.387},"last":1.1040000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.649,"15min":1.6240000000000001},"min":{"1min":1.3240000000000001,"5min":1.121,"15min":1.099},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.3340000000000001},"last":1.431},{"interface":"front","average":{"1min":1.756,"5min":1.6990000000000001,"15min":1.6839999999999999},"min":{"1min":1.2969999999999999,"5min":1.252,"15min":1.252},"max":{"1min":2.3119999999999998,"5min":2.387,"15min":2.387},"last":1.175}]}]},{"osd":2,"up_from":14,"seq":60129543067,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51456,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320384,"statfs":{"total":96636764160,"available":96584073216,"internally_reserved":0,"allocated":2281472,"data_stored":1100307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3480000000000001,"5min":1.732,"15min":2.1499999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.125,"15min":1.159},"min":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.367,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.141,"15min":1.151},"min":{"1min":0.75,"5min":0.75,"15min":0.72199999999999998},"max":{"1min":1.532,"5min":1.956,"15min":2.4550000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.2809999999999999,"15min":1.286},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.754,"5min":1.804,"15min":2.3490000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1339999999999999,"15min":1.1519999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.806,"5min":1.849,"15min":1.982},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.345,"5min":1.359,"15min":1.379},"min":{"1min":0.98799999999999999,"5min":0.91200000000000003,"15min":0.90700000000000003},"max":{"1min":1.7090000000000001,"5min":1.855,"15min":2.7970000000000002},"last":1.2949999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4950000000000001,"15min":1.5069999999999999},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.054},"max":{"1min":1.9430000000000001,"5min":2.1429999999999998,"15min":2.6589999999999998},"last":1.444},{"interface":"front","average":{"1min":1.452,"5min":1.492,"15min":1.514},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":1.762,"5min":2.0510000000000002,"15min":2.2629999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.575},"min":{"1min":1.129,"5min":1.129,"15min":1.056},"max":{"1min":1.921,"5min":2.101,"15min":2.5569999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.5009999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.7490000000000001,"5min":2.2650000000000001,"15min":2.899},"last":1.2350000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5549999999999999},"min":{"1min":1.0760000000000001,"5min":1.032,"15min":1.032},"max":{"1min":1.843,"5min":2.4300000000000002,"15min":3.0339999999999998},"last":1.389},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6080000000000001,"15min":1.637},"min":{"1min":1.145,"5min":1.145,"15min":1.145},"max":{"1min":1.982,"5min":2.0779999999999998,"15min":3.0790000000000002},"last":1.5069999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7050000000000001,"15min":1.6919999999999999},"min":{"1min":1.24,"5min":1.127,"15min":1.127},"max":{"1min":2.1880000000000002,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.651,"5min":1.643,"15min":1.661},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2429999999999999,"5min":2.5529999999999999,"15min":3.1320000000000001},"last":1.331}]}]},{"osd":1,"up_from":14,"seq":60129543067,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47284,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324556,"statfs":{"total":96636764160,"available":96588345344,"internally_reserved":0,"allocated":2269184,"data_stored":1099207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.73799999999999999,"5min":0.69999999999999996,"15min":0.497},"max":{"1min":1.2729999999999999,"5min":1.637,"15min":1.6970000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2,"15min":1.1910000000000001},"min":{"1min":0.85499999999999998,"5min":0.80100000000000005,"15min":0.56699999999999995},"max":{"1min":1.766,"5min":1.8,"15min":1.8089999999999999},"last":1.2270000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.052,"15min":1.038},"min":{"1min":0.81599999999999995,"5min":0.81499999999999995,"15min":0.74399999999999999},"max":{"1min":1.71,"5min":1.71,"15min":1.8069999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.254,"15min":1.26},"min":{"1min":0.873,"5min":0.873,"15min":0.85299999999999998},"max":{"1min":1.992,"5min":1.992,"15min":1.994},"last":1.149}]},{"osd":3,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.628},"max":{"1min":1.7729999999999999,"5min":1.883,"15min":2.226},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.29,"15min":1.292},"min":{"1min":0.92500000000000004,"5min":0.85099999999999998,"15min":0.68100000000000005},"max":{"1min":1.585,"5min":2.1269999999999998,"15min":2.2610000000000001},"last":1.1160000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3879999999999999,"15min":1.3500000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.86199999999999999},"max":{"1min":1.867,"5min":1.923,"15min":2.1539999999999999},"last":1.081},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5189999999999999,"15min":1.516},"min":{"1min":1.1930000000000001,"5min":1.0880000000000001,"15min":0.71899999999999997},"max":{"1min":1.974,"5min":2.0459999999999998,"15min":2.2240000000000002},"last":0.997}]},{"osd":5,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.51,"15min":1.5069999999999999},"min":{"1min":1.161,"5min":1.1080000000000001,"15min":0.84199999999999997},"max":{"1min":2.016,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4810000000000001,"15min":1.4690000000000001},"min":{"1min":1.1379999999999999,"5min":1.071,"15min":0.92300000000000004},"max":{"1min":1.9159999999999999,"5min":1.9970000000000001,"15min":2.028},"last":1.522}]},{"osd":6,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.6419999999999999,"15min":1.617},"min":{"1min":1.256,"5min":1.1830000000000001,"15min":0.877},"max":{"1min":2.0670000000000002,"5min":2.363,"15min":2.3879999999999999},"last":1.484},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6180000000000001,"15min":1.611},"min":{"1min":1.2529999999999999,"5min":1.1879999999999999,"15min":0.76900000000000002},"max":{"1min":2.105,"5min":2.306,"15min":2.306},"last":1.954}]},{"osd":7,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.736,"15min":1.708},"min":{"1min":1.2969999999999999,"5min":1.242,"15min":0.95499999999999996},"max":{"1min":2.0920000000000001,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":2.04},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6319999999999999,"15min":1.6180000000000001},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.0779999999999998,"5min":2.1040000000000001,"15min":2.3140000000000001},"last":1.6559999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543068,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60364,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311476,"statfs":{"total":96636764160,"available":96574951424,"internally_reserved":0,"allocated":2875392,"data_stored":1690639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.93700000000000006,"15min":0.93100000000000005},"min":{"1min":0.754,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1759999999999999,"15min":1.161},"min":{"1min":0.72299999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.6830000000000001,"5min":1.835,"15min":1.841},"last":0.79600000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1859999999999999,"15min":1.171},"min":{"1min":0.79400000000000004,"5min":0.76500000000000001,"15min":0.74399999999999999},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.9830000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2589999999999999,"15min":1.256},"min":{"1min":0.89900000000000002,"5min":0.86599999999999999,"15min":0.83999999999999997},"max":{"1min":1.863,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.079}]},{"osd":3,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1639999999999999,"15min":1.1259999999999999},"min":{"1min":0.85799999999999998,"5min":0.80900000000000005,"15min":0.72999999999999998},"max":{"1min":2.3340000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.397,"15min":1.3779999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.871},"max":{"1min":2.2450000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.613}]},{"osd":4,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4179999999999999,"15min":1.3979999999999999},"min":{"1min":1.022,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":1.7330000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.321},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5309999999999999,"15min":1.5129999999999999},"min":{"1min":1.248,"5min":1.1140000000000001,"15min":1.048},"max":{"1min":1.9430000000000001,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":1.3520000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.538,"15min":1.528},"min":{"1min":1.321,"5min":1.1359999999999999,"15min":1.04},"max":{"1min":2.0070000000000001,"5min":2.1560000000000001,"15min":2.2349999999999999},"last":1.22},{"interface":"front","average":{"1min":1.482,"5min":1.488,"15min":1.454},"min":{"1min":1.1200000000000001,"5min":1.0549999999999999,"15min":0.97599999999999998},"max":{"1min":2.0880000000000001,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.431}]},{"osd":6,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.659,"15min":1.625},"min":{"1min":1.327,"5min":1.1240000000000001,"15min":1.1240000000000001},"max":{"1min":2.2610000000000001,"5min":4.0270000000000001,"15min":4.0270000000000001},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6120000000000001,"15min":1.611},"min":{"1min":1.204,"5min":1.1870000000000001,"15min":1.163},"max":{"1min":2.0920000000000001,"5min":2.254,"15min":2.254},"last":1.5649999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.7290000000000001,"15min":1.7130000000000001},"min":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.202},"max":{"1min":2.1989999999999998,"5min":2.7149999999999999,"15min":2.7149999999999999},"last":1.649},{"interface":"front","average":{"1min":1.714,"5min":1.6519999999999999,"15min":1.6220000000000001},"min":{"1min":1.3280000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":2.286,"5min":3.3010000000000002,"15min":3.3010000000000002},"last":1.698}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:50:51.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:50:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.195+0000 7ff9d919a640 1 Processor -- start 2023-10-16T13:50:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.195+0000 7ff9d919a640 1 -- start start 2023-10-16T13:50:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.196+0000 7ff9d919a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9d40c9060 0x7ff9d406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.196+0000 7ff9d919a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.196+0000 7ff9d919a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff9d40c9440 0x7ff9d4164d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.196+0000 7ff9d919a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d40ca560 con 0x7ff9d40c9060 2023-10-16T13:50:52.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.196+0000 7ff9d919a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d40c9950 con 0x7ff9d40c9440 2023-10-16T13:50:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.196+0000 7ff9d919a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40c9c30 con 0x7ff9d406c9e0 2023-10-16T13:50:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9d40c9060 0x7ff9d406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff9d40c9060 0x7ff9d406f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53726/0 (socket says 172.21.15.60:53726) 2023-10-16T13:50:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 -- 172.21.15.60:0/1573485628 learned_addr learned my addr 172.21.15.60:0/1573485628 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d37fe640 1 --2- 172.21.15.60:0/1573485628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d8999640 1 --2- 172.21.15.60:0/1573485628 >> v2:172.21.15.144:3300/0 conn(0x7ff9d40c9440 0x7ff9d4164d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 -- 172.21.15.60:0/1573485628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 msgr2=0x7ff9d406f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/1573485628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d406f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 -- 172.21.15.60:0/1573485628 >> v2:172.21.15.144:3300/0 conn(0x7ff9d40c9440 msgr2=0x7ff9d4164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/1573485628 >> v2:172.21.15.144:3300/0 conn(0x7ff9d40c9440 0x7ff9d4164d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d3fff640 1 -- 172.21.15.60:0/1573485628 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d40c9f10 con 0x7ff9d40c9060 2023-10-16T13:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.197+0000 7ff9d8999640 1 --2- 172.21.15.60:0/1573485628 >> v2:172.21.15.144:3300/0 conn(0x7ff9d40c9440 0x7ff9d4164d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.198+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/1573485628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d406f440 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40ca4d0 tx=0x7ff9c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27117a893cf6825 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:52.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.199+0000 7ff9d2ffd640 1 -- 172.21.15.60:0/1573485628 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c402ca70 con 0x7ff9d40c9060 2023-10-16T13:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.199+0000 7ff9d2ffd640 1 -- 172.21.15.60:0/1573485628 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c402cbd0 con 0x7ff9d40c9060 2023-10-16T13:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.199+0000 7ff9d2ffd640 1 -- 172.21.15.60:0/1573485628 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40343f0 con 0x7ff9d40c9060 2023-10-16T13:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.199+0000 7ff9d919a640 1 -- 172.21.15.60:0/1573485628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 msgr2=0x7ff9d406f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.199+0000 7ff9d919a640 1 --2- 172.21.15.60:0/1573485628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d406f440 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40ca4d0 tx=0x7ff9c402bed0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 -- 172.21.15.60:0/1573485628 shutdown_connections 2023-10-16T13:50:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 --2- 172.21.15.60:0/1573485628 >> v2:172.21.15.144:3300/0 conn(0x7ff9d40c9440 0x7ff9d4164d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 --2- 172.21.15.60:0/1573485628 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d406f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 --2- 172.21.15.60:0/1573485628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d406f440 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 -- 172.21.15.60:0/1573485628 >> 172.21.15.60:0/1573485628 conn(0x7ff9d40ce6a0 msgr2=0x7ff9d40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 -- 172.21.15.60:0/1573485628 shutdown_connections 2023-10-16T13:50:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.200+0000 7ff9d919a640 1 -- 172.21.15.60:0/1573485628 wait complete. 2023-10-16T13:50:52.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.201+0000 7ff9d919a640 1 Processor -- start 2023-10-16T13:50:52.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 -- start start 2023-10-16T13:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d40cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9d40c9440 0x7ff9d40cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d41f71d0 con 0x7ff9d40c9060 2023-10-16T13:50:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d41fac70 con 0x7ff9d40c9440 2023-10-16T13:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.202+0000 7ff9d919a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d41f6b80 con 0x7ff9d406c9e0 2023-10-16T13:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d40cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d40cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53728/0 (socket says 172.21.15.60:53728) 2023-10-16T13:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 -- 172.21.15.60:0/3441419113 learned_addr learned my addr 172.21.15.60:0/3441419113 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 -- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 msgr2=0x7ff9d40b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d40b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d8999640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9d40c9440 0x7ff9d40cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 -- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9d40c9440 msgr2=0x7ff9d40cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9d40c9440 0x7ff9d40cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d37fe640 1 -- 172.21.15.60:0/3441419113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c4009b30 con 0x7ff9d40c9060 2023-10-16T13:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d40b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.203+0000 7ff9d8999640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9d40c9440 0x7ff9d40cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.204+0000 7ff9d37fe640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d40cea80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00d7d0 tx=0x7ff9bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.204+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0096a0 con 0x7ff9d40c9060 2023-10-16T13:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.204+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc009800 con 0x7ff9d40c9060 2023-10-16T13:50:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.204+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc009bb0 con 0x7ff9d40c9060 2023-10-16T13:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.204+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d41f6e60 con 0x7ff9d40c9060 2023-10-16T13:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.204+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d41f6830 con 0x7ff9d40c9060 2023-10-16T13:50:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.205+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d41f7340 con 0x7ff9d40c9060 2023-10-16T13:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.206+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9bc009d10 con 0x7ff9d40c9060 2023-10-16T13:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.207+0000 7ff9d0ff9640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff9b406afc0 0x7ff9b406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.207+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9bc089c60 con 0x7ff9d40c9060 2023-10-16T13:50:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.208+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff9b406afc0 0x7ff9b406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.209+0000 7ff9d3fff640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff9b406afc0 0x7ff9b406d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4035000 tx=0x7ff9c4030000 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:52.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.212+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9bc054c20 con 0x7ff9d40c9060 2023-10-16T13:50:52.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.404+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 --> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9d40c1f10 con 0x7ff9b406afc0 2023-10-16T13:50:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.427+0000 7ff9d0ff9640 1 -- 172.21.15.60:0/3441419113 <== mgr.7588 v2:172.21.15.144:6832/2817709727 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286385 (secure 0 0 0) 0x7ff9c40343f0 con 0x7ff9b406afc0 2023-10-16T13:50:52.432 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:52.435 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T13:50:52.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff9b406afc0 msgr2=0x7ff9b406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff9b406afc0 0x7ff9b406d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4035000 tx=0x7ff9c4030000 comp rx=0 tx=0).stop 2023-10-16T13:50:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 msgr2=0x7ff9d40cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d40cea80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00d7d0 tx=0x7ff9bc00dc90 comp rx=0 tx=0).stop 2023-10-16T13:50:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 shutdown_connections 2023-10-16T13:50:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7ff9b406afc0 0x7ff9b406d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9d40c9440 0x7ff9d40cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9d40c9060 0x7ff9d40cea80 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 --2- 172.21.15.60:0/3441419113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff9d406c9e0 0x7ff9d40b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 >> 172.21.15.60:0/3441419113 conn(0x7ff9d40ce6a0 msgr2=0x7ff9d40cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 shutdown_connections 2023-10-16T13:50:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.433+0000 7ff9d919a640 1 -- 172.21.15.60:0/3441419113 wait complete. 2023-10-16T13:50:52.454 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-16T13:50:51.793461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":487,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397488,"kb_used_data":18920,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754577232,"statfs":{"total":773094113280,"available":772687085568,"internally_reserved":0,"allocated":19374080,"data_stored":9974556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":387501624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002141"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.447571+0000","last_change":"2023-10-16T13:50:47.446902+0000","last_active":"2023-10-16T13:50:47.447571+0000","last_peered":"2023-10-16T13:50:47.447571+0000","last_clean":"2023-10-16T13:50:47.447571+0000","last_became_active":"2023-10-16T13:50:47.446622+0000","last_became_peered":"2023-10-16T13:50:47.446622+0000","last_unstale":"2023-10-16T13:50:47.447571+0000","last_undegraded":"2023-10-16T13:50:47.447571+0000","last_fullsized":"2023-10-16T13:50:47.447571+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:52.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435042+0000","last_change":"2023-10-16T13:50:47.430670+0000","last_active":"2023-10-16T13:50:47.435042+0000","last_peered":"2023-10-16T13:50:47.435042+0000","last_clean":"2023-10-16T13:50:47.435042+0000","last_became_active":"2023-10-16T13:50:47.430400+0000","last_became_peered":"2023-10-16T13:50:47.430400+0000","last_unstale":"2023-10-16T13:50:47.435042+0000","last_undegraded":"2023-10-16T13:50:47.435042+0000","last_fullsized":"2023-10-16T13:50:47.435042+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:45.392056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441000+0000","last_change":"2023-10-16T13:50:47.438393+0000","last_active":"2023-10-16T13:50:47.441000+0000","last_peered":"2023-10-16T13:50:47.441000+0000","last_clean":"2023-10-16T13:50:47.441000+0000","last_became_active":"2023-10-16T13:50:47.437861+0000","last_became_peered":"2023-10-16T13:50:47.437861+0000","last_unstale":"2023-10-16T13:50:47.441000+0000","last_undegraded":"2023-10-16T13:50:47.441000+0000","last_fullsized":"2023-10-16T13:50:47.441000+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:14.867925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440316+0000","last_change":"2023-10-16T13:50:47.443315+0000","last_active":"2023-10-16T13:50:48.440316+0000","last_peered":"2023-10-16T13:50:48.440316+0000","last_clean":"2023-10-16T13:50:48.440316+0000","last_became_active":"2023-10-16T13:50:47.443044+0000","last_became_peered":"2023-10-16T13:50:47.443044+0000","last_unstale":"2023-10-16T13:50:48.440316+0000","last_undegraded":"2023-10-16T13:50:48.440316+0000","last_fullsized":"2023-10-16T13:50:48.440316+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:29.644314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441722+0000","last_change":"2023-10-16T13:50:47.453828+0000","last_active":"2023-10-16T13:50:48.441722+0000","last_peered":"2023-10-16T13:50:48.441722+0000","last_clean":"2023-10-16T13:50:48.441722+0000","last_became_active":"2023-10-16T13:50:47.453513+0000","last_became_peered":"2023-10-16T13:50:47.453513+0000","last_unstale":"2023-10-16T13:50:48.441722+0000","last_undegraded":"2023-10-16T13:50:48.441722+0000","last_fullsized":"2023-10-16T13:50:48.441722+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:26.600063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.447174+0000","last_change":"2023-10-16T13:50:47.445657+0000","last_active":"2023-10-16T13:50:47.447174+0000","last_peered":"2023-10-16T13:50:47.447174+0000","last_clean":"2023-10-16T13:50:47.447174+0000","last_became_active":"2023-10-16T13:50:47.445367+0000","last_became_peered":"2023-10-16T13:50:47.445367+0000","last_unstale":"2023-10-16T13:50:47.447174+0000","last_undegraded":"2023-10-16T13:50:47.447174+0000","last_fullsized":"2023-10-16T13:50:47.447174+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:55.999228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433687+0000","last_change":"2023-10-16T13:50:47.424390+0000","last_active":"2023-10-16T13:50:48.433687+0000","last_peered":"2023-10-16T13:50:48.433687+0000","last_clean":"2023-10-16T13:50:48.433687+0000","last_became_active":"2023-10-16T13:50:47.423963+0000","last_became_peered":"2023-10-16T13:50:47.423963+0000","last_unstale":"2023-10-16T13:50:48.433687+0000","last_undegraded":"2023-10-16T13:50:48.433687+0000","last_fullsized":"2023-10-16T13:50:48.433687+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:20.823584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.439800+0000","last_change":"2023-10-16T13:50:47.435814+0000","last_active":"2023-10-16T13:50:47.439800+0000","last_peered":"2023-10-16T13:50:47.439800+0000","last_clean":"2023-10-16T13:50:47.439800+0000","last_became_active":"2023-10-16T13:50:47.435459+0000","last_became_peered":"2023-10-16T13:50:47.435459+0000","last_unstale":"2023-10-16T13:50:47.439800+0000","last_undegraded":"2023-10-16T13:50:47.439800+0000","last_fullsized":"2023-10-16T13:50:47.439800+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:46.633552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.431767+0000","last_change":"2023-10-16T13:50:47.429997+0000","last_active":"2023-10-16T13:50:47.431767+0000","last_peered":"2023-10-16T13:50:47.431767+0000","last_clean":"2023-10-16T13:50:47.431767+0000","last_became_active":"2023-10-16T13:50:47.428900+0000","last_became_peered":"2023-10-16T13:50:47.428900+0000","last_unstale":"2023-10-16T13:50:47.431767+0000","last_undegraded":"2023-10-16T13:50:47.431767+0000","last_fullsized":"2023-10-16T13:50:47.431767+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:41.964971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456367+0000","last_change":"2023-10-16T13:50:47.454181+0000","last_active":"2023-10-16T13:50:47.456367+0000","last_peered":"2023-10-16T13:50:47.456367+0000","last_clean":"2023-10-16T13:50:47.456367+0000","last_became_active":"2023-10-16T13:50:47.453881+0000","last_became_peered":"2023-10-16T13:50:47.453881+0000","last_unstale":"2023-10-16T13:50:47.456367+0000","last_undegraded":"2023-10-16T13:50:47.456367+0000","last_fullsized":"2023-10-16T13:50:47.456367+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:12.077989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.447358+0000","last_change":"2023-10-16T13:50:47.446252+0000","last_active":"2023-10-16T13:50:47.447358+0000","last_peered":"2023-10-16T13:50:47.447358+0000","last_clean":"2023-10-16T13:50:47.447358+0000","last_became_active":"2023-10-16T13:50:47.445986+0000","last_became_peered":"2023-10-16T13:50:47.445986+0000","last_unstale":"2023-10-16T13:50:47.447358+0000","last_undegraded":"2023-10-16T13:50:47.447358+0000","last_fullsized":"2023-10-16T13:50:47.447358+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:56.632400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.454157+0000","last_change":"2023-10-16T13:50:47.449625+0000","last_active":"2023-10-16T13:50:47.454157+0000","last_peered":"2023-10-16T13:50:47.454157+0000","last_clean":"2023-10-16T13:50:47.454157+0000","last_became_active":"2023-10-16T13:50:47.449301+0000","last_became_peered":"2023-10-16T13:50:47.449301+0000","last_unstale":"2023-10-16T13:50:47.454157+0000","last_undegraded":"2023-10-16T13:50:47.454157+0000","last_fullsized":"2023-10-16T13:50:47.454157+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.650281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.442081+0000","last_change":"2023-10-16T13:50:47.453578+0000","last_active":"2023-10-16T13:50:48.442081+0000","last_peered":"2023-10-16T13:50:48.442081+0000","last_clean":"2023-10-16T13:50:48.442081+0000","last_became_active":"2023-10-16T13:50:47.453211+0000","last_became_peered":"2023-10-16T13:50:47.453211+0000","last_unstale":"2023-10-16T13:50:48.442081+0000","last_undegraded":"2023-10-16T13:50:48.442081+0000","last_fullsized":"2023-10-16T13:50:48.442081+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:19.603609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.439640+0000","last_change":"2023-10-16T13:50:47.446745+0000","last_active":"2023-10-16T13:50:48.439640+0000","last_peered":"2023-10-16T13:50:48.439640+0000","last_clean":"2023-10-16T13:50:48.439640+0000","last_became_active":"2023-10-16T13:50:47.446274+0000","last_became_peered":"2023-10-16T13:50:47.446274+0000","last_unstale":"2023-10-16T13:50:48.439640+0000","last_undegraded":"2023-10-16T13:50:48.439640+0000","last_fullsized":"2023-10-16T13:50:48.439640+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:35.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455590+0000","last_change":"2023-10-16T13:50:47.452319+0000","last_active":"2023-10-16T13:50:47.455590+0000","last_peered":"2023-10-16T13:50:47.455590+0000","last_clean":"2023-10-16T13:50:47.455590+0000","last_became_active":"2023-10-16T13:50:47.452037+0000","last_became_peered":"2023-10-16T13:50:47.452037+0000","last_unstale":"2023-10-16T13:50:47.455590+0000","last_undegraded":"2023-10-16T13:50:47.455590+0000","last_fullsized":"2023-10-16T13:50:47.455590+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:57.437624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482704+0000","last_change":"2023-10-16T13:50:47.424611+0000","last_active":"2023-10-16T13:50:48.482704+0000","last_peered":"2023-10-16T13:50:48.482704+0000","last_clean":"2023-10-16T13:50:48.482704+0000","last_became_active":"2023-10-16T13:50:47.424164+0000","last_became_peered":"2023-10-16T13:50:47.424164+0000","last_unstale":"2023-10-16T13:50:48.482704+0000","last_undegraded":"2023-10-16T13:50:48.482704+0000","last_fullsized":"2023-10-16T13:50:48.482704+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:35.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455016+0000","last_change":"2023-10-16T13:50:47.449973+0000","last_active":"2023-10-16T13:50:47.455016+0000","last_peered":"2023-10-16T13:50:47.455016+0000","last_clean":"2023-10-16T13:50:47.455016+0000","last_became_active":"2023-10-16T13:50:47.449698+0000","last_became_peered":"2023-10-16T13:50:47.449698+0000","last_unstale":"2023-10-16T13:50:47.455016+0000","last_undegraded":"2023-10-16T13:50:47.455016+0000","last_fullsized":"2023-10-16T13:50:47.455016+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:11.213997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.448815+0000","last_change":"2023-10-16T13:50:47.444990+0000","last_active":"2023-10-16T13:50:47.448815+0000","last_peered":"2023-10-16T13:50:47.448815+0000","last_clean":"2023-10-16T13:50:47.448815+0000","last_became_active":"2023-10-16T13:50:47.444310+0000","last_became_peered":"2023-10-16T13:50:47.444310+0000","last_unstale":"2023-10-16T13:50:47.448815+0000","last_undegraded":"2023-10-16T13:50:47.448815+0000","last_fullsized":"2023-10-16T13:50:47.448815+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:25.416341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441026+0000","last_change":"2023-10-16T13:50:47.437739+0000","last_active":"2023-10-16T13:50:47.441026+0000","last_peered":"2023-10-16T13:50:47.441026+0000","last_clean":"2023-10-16T13:50:47.441026+0000","last_became_active":"2023-10-16T13:50:47.437260+0000","last_became_peered":"2023-10-16T13:50:47.437260+0000","last_unstale":"2023-10-16T13:50:47.441026+0000","last_undegraded":"2023-10-16T13:50:47.441026+0000","last_fullsized":"2023-10-16T13:50:47.441026+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:47.622962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482337+0000","last_change":"2023-10-16T13:50:47.442073+0000","last_active":"2023-10-16T13:50:48.482337+0000","last_peered":"2023-10-16T13:50:48.482337+0000","last_clean":"2023-10-16T13:50:48.482337+0000","last_became_active":"2023-10-16T13:50:47.436922+0000","last_became_peered":"2023-10-16T13:50:47.436922+0000","last_unstale":"2023-10-16T13:50:48.482337+0000","last_undegraded":"2023-10-16T13:50:48.482337+0000","last_fullsized":"2023-10-16T13:50:48.482337+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:49.466210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456296+0000","last_change":"2023-10-16T13:50:47.453756+0000","last_active":"2023-10-16T13:50:47.456296+0000","last_peered":"2023-10-16T13:50:47.456296+0000","last_clean":"2023-10-16T13:50:47.456296+0000","last_became_active":"2023-10-16T13:50:47.453463+0000","last_became_peered":"2023-10-16T13:50:47.453463+0000","last_unstale":"2023-10-16T13:50:47.456296+0000","last_undegraded":"2023-10-16T13:50:47.456296+0000","last_fullsized":"2023-10-16T13:50:47.456296+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:57.649795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440033+0000","last_change":"2023-10-16T13:50:47.433921+0000","last_active":"2023-10-16T13:50:48.440033+0000","last_peered":"2023-10-16T13:50:48.440033+0000","last_clean":"2023-10-16T13:50:48.440033+0000","last_became_active":"2023-10-16T13:50:47.433348+0000","last_became_peered":"2023-10-16T13:50:47.433348+0000","last_unstale":"2023-10-16T13:50:48.440033+0000","last_undegraded":"2023-10-16T13:50:48.440033+0000","last_fullsized":"2023-10-16T13:50:48.440033+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:55.414291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.439317+0000","last_change":"2023-10-16T13:50:47.421293+0000","last_active":"2023-10-16T13:50:48.439317+0000","last_peered":"2023-10-16T13:50:48.439317+0000","last_clean":"2023-10-16T13:50:48.439317+0000","last_became_active":"2023-10-16T13:50:47.420179+0000","last_became_peered":"2023-10-16T13:50:47.420179+0000","last_unstale":"2023-10-16T13:50:48.439317+0000","last_undegraded":"2023-10-16T13:50:48.439317+0000","last_fullsized":"2023-10-16T13:50:48.439317+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:47.435807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.453949+0000","last_change":"2023-10-16T13:50:47.448898+0000","last_active":"2023-10-16T13:50:47.453949+0000","last_peered":"2023-10-16T13:50:47.453949+0000","last_clean":"2023-10-16T13:50:47.453949+0000","last_became_active":"2023-10-16T13:50:47.447351+0000","last_became_peered":"2023-10-16T13:50:47.447351+0000","last_unstale":"2023-10-16T13:50:47.453949+0000","last_undegraded":"2023-10-16T13:50:47.453949+0000","last_fullsized":"2023-10-16T13:50:47.453949+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:43.237019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482708+0000","last_change":"2023-10-16T13:50:47.429796+0000","last_active":"2023-10-16T13:50:48.482708+0000","last_peered":"2023-10-16T13:50:48.482708+0000","last_clean":"2023-10-16T13:50:48.482708+0000","last_became_active":"2023-10-16T13:50:47.429542+0000","last_became_peered":"2023-10-16T13:50:47.429542+0000","last_unstale":"2023-10-16T13:50:48.482708+0000","last_undegraded":"2023-10-16T13:50:48.482708+0000","last_fullsized":"2023-10-16T13:50:48.482708+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:29.691226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.444231+0000","last_change":"2023-10-16T13:50:47.442555+0000","last_active":"2023-10-16T13:50:47.444231+0000","last_peered":"2023-10-16T13:50:47.444231+0000","last_clean":"2023-10-16T13:50:47.444231+0000","last_became_active":"2023-10-16T13:50:47.442026+0000","last_became_peered":"2023-10-16T13:50:47.442026+0000","last_unstale":"2023-10-16T13:50:47.444231+0000","last_undegraded":"2023-10-16T13:50:47.444231+0000","last_fullsized":"2023-10-16T13:50:47.444231+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:35.290078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482289+0000","last_change":"2023-10-16T13:50:47.442882+0000","last_active":"2023-10-16T13:50:48.482289+0000","last_peered":"2023-10-16T13:50:48.482289+0000","last_clean":"2023-10-16T13:50:48.482289+0000","last_became_active":"2023-10-16T13:50:47.442614+0000","last_became_peered":"2023-10-16T13:50:47.442614+0000","last_unstale":"2023-10-16T13:50:48.482289+0000","last_undegraded":"2023-10-16T13:50:48.482289+0000","last_fullsized":"2023-10-16T13:50:48.482289+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:48.324511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433745+0000","last_change":"2023-10-16T13:50:47.440701+0000","last_active":"2023-10-16T13:50:48.433745+0000","last_peered":"2023-10-16T13:50:48.433745+0000","last_clean":"2023-10-16T13:50:48.433745+0000","last_became_active":"2023-10-16T13:50:47.440330+0000","last_became_peered":"2023-10-16T13:50:47.440330+0000","last_unstale":"2023-10-16T13:50:48.433745+0000","last_undegraded":"2023-10-16T13:50:48.433745+0000","last_fullsized":"2023-10-16T13:50:48.433745+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:02.008233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.440928+0000","last_change":"2023-10-16T13:50:47.437603+0000","last_active":"2023-10-16T13:50:47.440928+0000","last_peered":"2023-10-16T13:50:47.440928+0000","last_clean":"2023-10-16T13:50:47.440928+0000","last_became_active":"2023-10-16T13:50:47.437310+0000","last_became_peered":"2023-10-16T13:50:47.437310+0000","last_unstale":"2023-10-16T13:50:47.440928+0000","last_undegraded":"2023-10-16T13:50:47.440928+0000","last_fullsized":"2023-10-16T13:50:47.440928+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:06.229083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440715+0000","last_change":"2023-10-16T13:50:47.433770+0000","last_active":"2023-10-16T13:50:48.440715+0000","last_peered":"2023-10-16T13:50:48.440715+0000","last_clean":"2023-10-16T13:50:48.440715+0000","last_became_active":"2023-10-16T13:50:47.432991+0000","last_became_peered":"2023-10-16T13:50:47.432991+0000","last_unstale":"2023-10-16T13:50:48.440715+0000","last_undegraded":"2023-10-16T13:50:48.440715+0000","last_fullsized":"2023-10-16T13:50:48.440715+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:02.919787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441669+0000","last_change":"2023-10-16T13:50:47.438082+0000","last_active":"2023-10-16T13:50:47.441669+0000","last_peered":"2023-10-16T13:50:47.441669+0000","last_clean":"2023-10-16T13:50:47.441669+0000","last_became_active":"2023-10-16T13:50:47.437634+0000","last_became_peered":"2023-10-16T13:50:47.437634+0000","last_unstale":"2023-10-16T13:50:47.441669+0000","last_undegraded":"2023-10-16T13:50:47.441669+0000","last_fullsized":"2023-10-16T13:50:47.441669+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.613016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482838+0000","last_change":"2023-10-16T13:50:47.436391+0000","last_active":"2023-10-16T13:50:48.482838+0000","last_peered":"2023-10-16T13:50:48.482838+0000","last_clean":"2023-10-16T13:50:48.482838+0000","last_became_active":"2023-10-16T13:50:47.436110+0000","last_became_peered":"2023-10-16T13:50:47.436110+0000","last_unstale":"2023-10-16T13:50:48.482838+0000","last_undegraded":"2023-10-16T13:50:48.482838+0000","last_fullsized":"2023-10-16T13:50:48.482838+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:29.644314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456388+0000","last_change":"2023-10-16T13:50:47.455166+0000","last_active":"2023-10-16T13:50:47.456388+0000","last_peered":"2023-10-16T13:50:47.456388+0000","last_clean":"2023-10-16T13:50:47.456388+0000","last_became_active":"2023-10-16T13:50:47.454848+0000","last_became_peered":"2023-10-16T13:50:47.454848+0000","last_unstale":"2023-10-16T13:50:47.456388+0000","last_undegraded":"2023-10-16T13:50:47.456388+0000","last_fullsized":"2023-10-16T13:50:47.456388+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:12.196989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"182'4","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.450806+0000","last_change":"2023-10-16T13:50:46.421041+0000","last_active":"2023-10-16T13:50:47.450806+0000","last_peered":"2023-10-16T13:50:47.450806+0000","last_clean":"2023-10-16T13:50:47.450806+0000","last_became_active":"2023-10-16T13:50:46.420682+0000","last_became_peered":"2023-10-16T13:50:46.420682+0000","last_unstale":"2023-10-16T13:50:47.450806+0000","last_undegraded":"2023-10-16T13:50:47.450806+0000","last_fullsized":"2023-10-16T13:50:47.450806+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:42.597620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435079+0000","last_change":"2023-10-16T13:50:47.430600+0000","last_active":"2023-10-16T13:50:47.435079+0000","last_peered":"2023-10-16T13:50:47.435079+0000","last_clean":"2023-10-16T13:50:47.435079+0000","last_became_active":"2023-10-16T13:50:47.430299+0000","last_became_peered":"2023-10-16T13:50:47.430299+0000","last_unstale":"2023-10-16T13:50:47.435079+0000","last_undegraded":"2023-10-16T13:50:47.435079+0000","last_fullsized":"2023-10-16T13:50:47.435079+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:00:06.877456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"182'1","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452079+0000","last_change":"2023-10-16T13:50:46.414025+0000","last_active":"2023-10-16T13:50:47.452079+0000","last_peered":"2023-10-16T13:50:47.452079+0000","last_clean":"2023-10-16T13:50:47.452079+0000","last_became_active":"2023-10-16T13:50:46.413741+0000","last_became_peered":"2023-10-16T13:50:46.413741+0000","last_unstale":"2023-10-16T13:50:47.452079+0000","last_undegraded":"2023-10-16T13:50:47.452079+0000","last_fullsized":"2023-10-16T13:50:47.452079+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:30.795279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.445773+0000","last_change":"2023-10-16T13:50:47.442976+0000","last_active":"2023-10-16T13:50:47.445773+0000","last_peered":"2023-10-16T13:50:47.445773+0000","last_clean":"2023-10-16T13:50:47.445773+0000","last_became_active":"2023-10-16T13:50:47.442580+0000","last_became_peered":"2023-10-16T13:50:47.442580+0000","last_unstale":"2023-10-16T13:50:47.445773+0000","last_undegraded":"2023-10-16T13:50:47.445773+0000","last_fullsized":"2023-10-16T13:50:47.445773+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:22.258673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"182'2","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440520+0000","last_change":"2023-10-16T13:50:46.416787+0000","last_active":"2023-10-16T13:50:48.440520+0000","last_peered":"2023-10-16T13:50:48.440520+0000","last_clean":"2023-10-16T13:50:48.440520+0000","last_became_active":"2023-10-16T13:50:46.414901+0000","last_became_peered":"2023-10-16T13:50:46.414901+0000","last_unstale":"2023-10-16T13:50:48.440520+0000","last_undegraded":"2023-10-16T13:50:48.440520+0000","last_fullsized":"2023-10-16T13:50:48.440520+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:15.493786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.456006+0000","last_change":"2023-10-16T13:50:47.453838+0000","last_active":"2023-10-16T13:50:47.456006+0000","last_peered":"2023-10-16T13:50:47.456006+0000","last_clean":"2023-10-16T13:50:47.456006+0000","last_became_active":"2023-10-16T13:50:47.453577+0000","last_became_peered":"2023-10-16T13:50:47.453577+0000","last_unstale":"2023-10-16T13:50:47.456006+0000","last_undegraded":"2023-10-16T13:50:47.456006+0000","last_fullsized":"2023-10-16T13:50:47.456006+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:57.869284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"182'2","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440366+0000","last_change":"2023-10-16T13:50:46.416865+0000","last_active":"2023-10-16T13:50:48.440366+0000","last_peered":"2023-10-16T13:50:48.440366+0000","last_clean":"2023-10-16T13:50:48.440366+0000","last_became_active":"2023-10-16T13:50:46.414286+0000","last_became_peered":"2023-10-16T13:50:46.414286+0000","last_unstale":"2023-10-16T13:50:48.440366+0000","last_undegraded":"2023-10-16T13:50:48.440366+0000","last_fullsized":"2023-10-16T13:50:48.440366+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:39.848223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.442252+0000","last_change":"2023-10-16T13:50:47.439022+0000","last_active":"2023-10-16T13:50:47.442252+0000","last_peered":"2023-10-16T13:50:47.442252+0000","last_clean":"2023-10-16T13:50:47.442252+0000","last_became_active":"2023-10-16T13:50:47.438750+0000","last_became_peered":"2023-10-16T13:50:47.438750+0000","last_unstale":"2023-10-16T13:50:47.442252+0000","last_undegraded":"2023-10-16T13:50:47.442252+0000","last_fullsized":"2023-10-16T13:50:47.442252+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:46.587219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482336+0000","last_change":"2023-10-16T13:50:46.416541+0000","last_active":"2023-10-16T13:50:48.482336+0000","last_peered":"2023-10-16T13:50:48.482336+0000","last_clean":"2023-10-16T13:50:48.482336+0000","last_became_active":"2023-10-16T13:50:46.415280+0000","last_became_peered":"2023-10-16T13:50:46.415280+0000","last_unstale":"2023-10-16T13:50:48.482336+0000","last_undegraded":"2023-10-16T13:50:48.482336+0000","last_fullsized":"2023-10-16T13:50:48.482336+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:15.036950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.439877+0000","last_change":"2023-10-16T13:50:47.442655+0000","last_active":"2023-10-16T13:50:48.439877+0000","last_peered":"2023-10-16T13:50:48.439877+0000","last_clean":"2023-10-16T13:50:48.439877+0000","last_became_active":"2023-10-16T13:50:47.442336+0000","last_became_peered":"2023-10-16T13:50:47.442336+0000","last_unstale":"2023-10-16T13:50:48.439877+0000","last_undegraded":"2023-10-16T13:50:48.439877+0000","last_fullsized":"2023-10-16T13:50:48.439877+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:29.691226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441746+0000","last_change":"2023-10-16T13:50:46.418773+0000","last_active":"2023-10-16T13:50:48.441746+0000","last_peered":"2023-10-16T13:50:48.441746+0000","last_clean":"2023-10-16T13:50:48.441746+0000","last_became_active":"2023-10-16T13:50:46.418289+0000","last_became_peered":"2023-10-16T13:50:46.418289+0000","last_unstale":"2023-10-16T13:50:48.441746+0000","last_undegraded":"2023-10-16T13:50:48.441746+0000","last_fullsized":"2023-10-16T13:50:48.441746+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:39.445331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441660+0000","last_change":"2023-10-16T13:50:47.437974+0000","last_active":"2023-10-16T13:50:47.441660+0000","last_peered":"2023-10-16T13:50:47.441660+0000","last_clean":"2023-10-16T13:50:47.441660+0000","last_became_active":"2023-10-16T13:50:47.437695+0000","last_became_peered":"2023-10-16T13:50:47.437695+0000","last_unstale":"2023-10-16T13:50:47.441660+0000","last_undegraded":"2023-10-16T13:50:47.441660+0000","last_fullsized":"2023-10-16T13:50:47.441660+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:15.216018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482124+0000","last_change":"2023-10-16T13:50:46.407191+0000","last_active":"2023-10-16T13:50:48.482124+0000","last_peered":"2023-10-16T13:50:48.482124+0000","last_clean":"2023-10-16T13:50:48.482124+0000","last_became_active":"2023-10-16T13:50:46.406915+0000","last_became_peered":"2023-10-16T13:50:46.406915+0000","last_unstale":"2023-10-16T13:50:48.482124+0000","last_undegraded":"2023-10-16T13:50:48.482124+0000","last_fullsized":"2023-10-16T13:50:48.482124+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:15.493786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441619+0000","last_change":"2023-10-16T13:50:47.435775+0000","last_active":"2023-10-16T13:50:48.441619+0000","last_peered":"2023-10-16T13:50:48.441619+0000","last_clean":"2023-10-16T13:50:48.441619+0000","last_became_active":"2023-10-16T13:50:47.435437+0000","last_became_peered":"2023-10-16T13:50:47.435437+0000","last_unstale":"2023-10-16T13:50:48.441619+0000","last_undegraded":"2023-10-16T13:50:48.441619+0000","last_fullsized":"2023-10-16T13:50:48.441619+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:46.187873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.438866+0000","last_change":"2023-10-16T13:50:46.421579+0000","last_active":"2023-10-16T13:50:48.438866+0000","last_peered":"2023-10-16T13:50:48.438866+0000","last_clean":"2023-10-16T13:50:48.438866+0000","last_became_active":"2023-10-16T13:50:46.420836+0000","last_became_peered":"2023-10-16T13:50:46.420836+0000","last_unstale":"2023-10-16T13:50:48.438866+0000","last_undegraded":"2023-10-16T13:50:48.438866+0000","last_fullsized":"2023-10-16T13:50:48.438866+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:39.153337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.443028+0000","last_change":"2023-10-16T13:50:47.450407+0000","last_active":"2023-10-16T13:50:48.443028+0000","last_peered":"2023-10-16T13:50:48.443028+0000","last_clean":"2023-10-16T13:50:48.443028+0000","last_became_active":"2023-10-16T13:50:47.446352+0000","last_became_peered":"2023-10-16T13:50:47.446352+0000","last_unstale":"2023-10-16T13:50:48.443028+0000","last_undegraded":"2023-10-16T13:50:48.443028+0000","last_fullsized":"2023-10-16T13:50:48.443028+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:46.399833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.420815+0000","last_change":"2023-10-16T13:50:46.399594+0000","last_active":"2023-10-16T13:50:47.420815+0000","last_peered":"2023-10-16T13:50:47.420815+0000","last_clean":"2023-10-16T13:50:47.420815+0000","last_became_active":"2023-10-16T13:50:46.399300+0000","last_became_peered":"2023-10-16T13:50:46.399300+0000","last_unstale":"2023-10-16T13:50:47.420815+0000","last_undegraded":"2023-10-16T13:50:47.420815+0000","last_fullsized":"2023-10-16T13:50:47.420815+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:49.741068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.454992+0000","last_change":"2023-10-16T13:50:47.450141+0000","last_active":"2023-10-16T13:50:47.454992+0000","last_peered":"2023-10-16T13:50:47.454992+0000","last_clean":"2023-10-16T13:50:47.454992+0000","last_became_active":"2023-10-16T13:50:47.447000+0000","last_became_peered":"2023-10-16T13:50:47.447000+0000","last_unstale":"2023-10-16T13:50:47.454992+0000","last_undegraded":"2023-10-16T13:50:47.454992+0000","last_fullsized":"2023-10-16T13:50:47.454992+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:32.823753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435576+0000","last_change":"2023-10-16T13:50:46.422730+0000","last_active":"2023-10-16T13:50:47.435576+0000","last_peered":"2023-10-16T13:50:47.435576+0000","last_clean":"2023-10-16T13:50:47.435576+0000","last_became_active":"2023-10-16T13:50:46.422306+0000","last_became_peered":"2023-10-16T13:50:46.422306+0000","last_unstale":"2023-10-16T13:50:47.435576+0000","last_undegraded":"2023-10-16T13:50:47.435576+0000","last_fullsized":"2023-10-16T13:50:47.435576+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:38.217439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482611+0000","last_change":"2023-10-16T13:50:47.434906+0000","last_active":"2023-10-16T13:50:48.482611+0000","last_peered":"2023-10-16T13:50:48.482611+0000","last_clean":"2023-10-16T13:50:48.482611+0000","last_became_active":"2023-10-16T13:50:47.432077+0000","last_became_peered":"2023-10-16T13:50:47.432077+0000","last_unstale":"2023-10-16T13:50:48.482611+0000","last_undegraded":"2023-10-16T13:50:48.482611+0000","last_fullsized":"2023-10-16T13:50:48.482611+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:47.435807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"182'6","reported_seq":18,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433990+0000","last_change":"2023-10-16T13:50:46.408951+0000","last_active":"2023-10-16T13:50:48.433990+0000","last_peered":"2023-10-16T13:50:48.433990+0000","last_clean":"2023-10-16T13:50:48.433990+0000","last_became_active":"2023-10-16T13:50:46.408403+0000","last_became_peered":"2023-10-16T13:50:46.408403+0000","last_unstale":"2023-10-16T13:50:48.433990+0000","last_undegraded":"2023-10-16T13:50:48.433990+0000","last_fullsized":"2023-10-16T13:50:48.433990+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:41.001687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452765+0000","last_change":"2023-10-16T13:50:47.449520+0000","last_active":"2023-10-16T13:50:47.452765+0000","last_peered":"2023-10-16T13:50:47.452765+0000","last_clean":"2023-10-16T13:50:47.452765+0000","last_became_active":"2023-10-16T13:50:47.448963+0000","last_became_peered":"2023-10-16T13:50:47.448963+0000","last_unstale":"2023-10-16T13:50:47.452765+0000","last_undegraded":"2023-10-16T13:50:47.452765+0000","last_fullsized":"2023-10-16T13:50:47.452765+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:33.170205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"182'2","reported_seq":13,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.443960+0000","last_change":"2023-10-16T13:50:46.410822+0000","last_active":"2023-10-16T13:50:47.443960+0000","last_peered":"2023-10-16T13:50:47.443960+0000","last_clean":"2023-10-16T13:50:47.443960+0000","last_became_active":"2023-10-16T13:50:46.409436+0000","last_became_peered":"2023-10-16T13:50:46.409436+0000","last_unstale":"2023-10-16T13:50:47.443960+0000","last_undegraded":"2023-10-16T13:50:47.443960+0000","last_fullsized":"2023-10-16T13:50:47.443960+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:37.935030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.435105+0000","last_change":"2023-10-16T13:50:47.430121+0000","last_active":"2023-10-16T13:50:47.435105+0000","last_peered":"2023-10-16T13:50:47.435105+0000","last_clean":"2023-10-16T13:50:47.435105+0000","last_became_active":"2023-10-16T13:50:47.429825+0000","last_became_peered":"2023-10-16T13:50:47.429825+0000","last_unstale":"2023-10-16T13:50:47.435105+0000","last_undegraded":"2023-10-16T13:50:47.435105+0000","last_fullsized":"2023-10-16T13:50:47.435105+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:09.347181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.427115+0000","last_change":"2023-10-16T13:50:46.408032+0000","last_active":"2023-10-16T13:50:47.427115+0000","last_peered":"2023-10-16T13:50:47.427115+0000","last_clean":"2023-10-16T13:50:47.427115+0000","last_became_active":"2023-10-16T13:50:46.407687+0000","last_became_peered":"2023-10-16T13:50:46.407687+0000","last_unstale":"2023-10-16T13:50:47.427115+0000","last_undegraded":"2023-10-16T13:50:47.427115+0000","last_fullsized":"2023-10-16T13:50:47.427115+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:15.319979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441624+0000","last_change":"2023-10-16T13:50:47.442856+0000","last_active":"2023-10-16T13:50:48.441624+0000","last_peered":"2023-10-16T13:50:48.441624+0000","last_clean":"2023-10-16T13:50:48.441624+0000","last_became_active":"2023-10-16T13:50:47.442527+0000","last_became_peered":"2023-10-16T13:50:47.442527+0000","last_unstale":"2023-10-16T13:50:48.441624+0000","last_undegraded":"2023-10-16T13:50:48.441624+0000","last_fullsized":"2023-10-16T13:50:48.441624+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:09:33.822906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.426296+0000","last_change":"2023-10-16T13:50:46.407144+0000","last_active":"2023-10-16T13:50:47.426296+0000","last_peered":"2023-10-16T13:50:47.426296+0000","last_clean":"2023-10-16T13:50:47.426296+0000","last_became_active":"2023-10-16T13:50:46.406806+0000","last_became_peered":"2023-10-16T13:50:46.406806+0000","last_unstale":"2023-10-16T13:50:47.426296+0000","last_undegraded":"2023-10-16T13:50:47.426296+0000","last_fullsized":"2023-10-16T13:50:47.426296+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:15.493786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452144+0000","last_change":"2023-10-16T13:50:47.447843+0000","last_active":"2023-10-16T13:50:47.452144+0000","last_peered":"2023-10-16T13:50:47.452144+0000","last_clean":"2023-10-16T13:50:47.452144+0000","last_became_active":"2023-10-16T13:50:47.447547+0000","last_became_peered":"2023-10-16T13:50:47.447547+0000","last_unstale":"2023-10-16T13:50:47.452144+0000","last_undegraded":"2023-10-16T13:50:47.452144+0000","last_fullsized":"2023-10-16T13:50:47.452144+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:28.352075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"182'8","reported_seq":20,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482945+0000","last_change":"2023-10-16T13:50:46.400068+0000","last_active":"2023-10-16T13:50:48.482945+0000","last_peered":"2023-10-16T13:50:48.482945+0000","last_clean":"2023-10-16T13:50:48.482945+0000","last_became_active":"2023-10-16T13:50:46.398885+0000","last_became_peered":"2023-10-16T13:50:46.398885+0000","last_unstale":"2023-10-16T13:50:48.482945+0000","last_undegraded":"2023-10-16T13:50:48.482945+0000","last_fullsized":"2023-10-16T13:50:48.482945+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:34.274026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455153+0000","last_change":"2023-10-16T13:50:47.450466+0000","last_active":"2023-10-16T13:50:47.455153+0000","last_peered":"2023-10-16T13:50:47.455153+0000","last_clean":"2023-10-16T13:50:47.455153+0000","last_became_active":"2023-10-16T13:50:47.447062+0000","last_became_peered":"2023-10-16T13:50:47.447062+0000","last_unstale":"2023-10-16T13:50:47.455153+0000","last_undegraded":"2023-10-16T13:50:47.455153+0000","last_fullsized":"2023-10-16T13:50:47.455153+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:28.413676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.442775+0000","last_change":"2023-10-16T13:50:46.413917+0000","last_active":"2023-10-16T13:50:48.442775+0000","last_peered":"2023-10-16T13:50:48.442775+0000","last_clean":"2023-10-16T13:50:48.442775+0000","last_became_active":"2023-10-16T13:50:46.413618+0000","last_became_peered":"2023-10-16T13:50:46.413618+0000","last_unstale":"2023-10-16T13:50:48.442775+0000","last_undegraded":"2023-10-16T13:50:48.442775+0000","last_fullsized":"2023-10-16T13:50:48.442775+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:14.420011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482721+0000","last_change":"2023-10-16T13:50:47.448728+0000","last_active":"2023-10-16T13:50:48.482721+0000","last_peered":"2023-10-16T13:50:48.482721+0000","last_clean":"2023-10-16T13:50:48.482721+0000","last_became_active":"2023-10-16T13:50:47.448399+0000","last_became_peered":"2023-10-16T13:50:47.448399+0000","last_unstale":"2023-10-16T13:50:48.482721+0000","last_undegraded":"2023-10-16T13:50:48.482721+0000","last_fullsized":"2023-10-16T13:50:48.482721+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:55.414291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"182'2","reported_seq":13,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.444664+0000","last_change":"2023-10-16T13:50:46.412005+0000","last_active":"2023-10-16T13:50:47.444664+0000","last_peered":"2023-10-16T13:50:47.444664+0000","last_clean":"2023-10-16T13:50:47.444664+0000","last_became_active":"2023-10-16T13:50:46.411635+0000","last_became_peered":"2023-10-16T13:50:46.411635+0000","last_unstale":"2023-10-16T13:50:47.444664+0000","last_undegraded":"2023-10-16T13:50:47.444664+0000","last_fullsized":"2023-10-16T13:50:47.444664+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:41.253896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.452594+0000","last_change":"2023-10-16T13:50:47.449400+0000","last_active":"2023-10-16T13:50:47.452594+0000","last_peered":"2023-10-16T13:50:47.452594+0000","last_clean":"2023-10-16T13:50:47.452594+0000","last_became_active":"2023-10-16T13:50:47.449076+0000","last_became_peered":"2023-10-16T13:50:47.449076+0000","last_unstale":"2023-10-16T13:50:47.452594+0000","last_undegraded":"2023-10-16T13:50:47.452594+0000","last_fullsized":"2023-10-16T13:50:47.452594+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:08.089747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"182'1","reported_seq":13,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482752+0000","last_change":"2023-10-16T13:50:46.418621+0000","last_active":"2023-10-16T13:50:48.482752+0000","last_peered":"2023-10-16T13:50:48.482752+0000","last_clean":"2023-10-16T13:50:48.482752+0000","last_became_active":"2023-10-16T13:50:46.418059+0000","last_became_peered":"2023-10-16T13:50:46.418059+0000","last_unstale":"2023-10-16T13:50:48.482752+0000","last_undegraded":"2023-10-16T13:50:48.482752+0000","last_fullsized":"2023-10-16T13:50:48.482752+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:39.153337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.441527+0000","last_change":"2023-10-16T13:50:47.439032+0000","last_active":"2023-10-16T13:50:47.441527+0000","last_peered":"2023-10-16T13:50:47.441527+0000","last_clean":"2023-10-16T13:50:47.441527+0000","last_became_active":"2023-10-16T13:50:47.438774+0000","last_became_peered":"2023-10-16T13:50:47.438774+0000","last_unstale":"2023-10-16T13:50:47.441527+0000","last_undegraded":"2023-10-16T13:50:47.441527+0000","last_fullsized":"2023-10-16T13:50:47.441527+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:56.415282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.420453+0000","last_change":"2023-10-16T13:50:46.401553+0000","last_active":"2023-10-16T13:50:47.420453+0000","last_peered":"2023-10-16T13:50:47.420453+0000","last_clean":"2023-10-16T13:50:47.420453+0000","last_became_active":"2023-10-16T13:50:46.400652+0000","last_became_peered":"2023-10-16T13:50:46.400652+0000","last_unstale":"2023-10-16T13:50:47.420453+0000","last_undegraded":"2023-10-16T13:50:47.420453+0000","last_fullsized":"2023-10-16T13:50:47.420453+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:14.814624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.440900+0000","last_change":"2023-10-16T13:50:47.433610+0000","last_active":"2023-10-16T13:50:48.440900+0000","last_peered":"2023-10-16T13:50:48.440900+0000","last_clean":"2023-10-16T13:50:48.440900+0000","last_became_active":"2023-10-16T13:50:47.432591+0000","last_became_peered":"2023-10-16T13:50:47.432591+0000","last_unstale":"2023-10-16T13:50:48.440900+0000","last_undegraded":"2023-10-16T13:50:48.440900+0000","last_fullsized":"2023-10-16T13:50:48.440900+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:55.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"182'2","reported_seq":14,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433973+0000","last_change":"2023-10-16T13:50:46.413858+0000","last_active":"2023-10-16T13:50:48.433973+0000","last_peered":"2023-10-16T13:50:48.433973+0000","last_clean":"2023-10-16T13:50:48.433973+0000","last_became_active":"2023-10-16T13:50:46.413462+0000","last_became_peered":"2023-10-16T13:50:46.413462+0000","last_unstale":"2023-10-16T13:50:48.433973+0000","last_undegraded":"2023-10-16T13:50:48.433973+0000","last_fullsized":"2023-10-16T13:50:48.433973+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:56:27.628143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433843+0000","last_change":"2023-10-16T13:50:47.423159+0000","last_active":"2023-10-16T13:50:48.433843+0000","last_peered":"2023-10-16T13:50:48.433843+0000","last_clean":"2023-10-16T13:50:48.433843+0000","last_became_active":"2023-10-16T13:50:47.422652+0000","last_became_peered":"2023-10-16T13:50:47.422652+0000","last_unstale":"2023-10-16T13:50:48.433843+0000","last_undegraded":"2023-10-16T13:50:48.433843+0000","last_fullsized":"2023-10-16T13:50:48.433843+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:17.998148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.428505+0000","last_change":"2023-10-16T13:50:46.417773+0000","last_active":"2023-10-16T13:50:47.428505+0000","last_peered":"2023-10-16T13:50:47.428505+0000","last_clean":"2023-10-16T13:50:47.428505+0000","last_became_active":"2023-10-16T13:50:46.417402+0000","last_became_peered":"2023-10-16T13:50:46.417402+0000","last_unstale":"2023-10-16T13:50:47.428505+0000","last_undegraded":"2023-10-16T13:50:47.428505+0000","last_fullsized":"2023-10-16T13:50:47.428505+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:06.931829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.438819+0000","last_change":"2023-10-16T13:50:47.418864+0000","last_active":"2023-10-16T13:50:48.438819+0000","last_peered":"2023-10-16T13:50:48.438819+0000","last_clean":"2023-10-16T13:50:48.438819+0000","last_became_active":"2023-10-16T13:50:47.417722+0000","last_became_peered":"2023-10-16T13:50:47.417722+0000","last_unstale":"2023-10-16T13:50:48.438819+0000","last_undegraded":"2023-10-16T13:50:48.438819+0000","last_fullsized":"2023-10-16T13:50:48.438819+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:49.466210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.429793+0000","last_change":"2023-10-16T13:50:46.416881+0000","last_active":"2023-10-16T13:50:47.429793+0000","last_peered":"2023-10-16T13:50:47.429793+0000","last_clean":"2023-10-16T13:50:47.429793+0000","last_became_active":"2023-10-16T13:50:46.416573+0000","last_became_peered":"2023-10-16T13:50:46.416573+0000","last_unstale":"2023-10-16T13:50:47.429793+0000","last_undegraded":"2023-10-16T13:50:47.429793+0000","last_fullsized":"2023-10-16T13:50:47.429793+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:11.247133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.433549+0000","last_change":"2023-10-16T13:50:47.422780+0000","last_active":"2023-10-16T13:50:48.433549+0000","last_peered":"2023-10-16T13:50:48.433549+0000","last_clean":"2023-10-16T13:50:48.433549+0000","last_became_active":"2023-10-16T13:50:47.422418+0000","last_became_peered":"2023-10-16T13:50:47.422418+0000","last_unstale":"2023-10-16T13:50:48.433549+0000","last_undegraded":"2023-10-16T13:50:48.433549+0000","last_fullsized":"2023-10-16T13:50:48.433549+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:54.282098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.424763+0000","last_change":"2023-10-16T13:50:46.406731+0000","last_active":"2023-10-16T13:50:47.424763+0000","last_peered":"2023-10-16T13:50:47.424763+0000","last_clean":"2023-10-16T13:50:47.424763+0000","last_became_active":"2023-10-16T13:50:46.406451+0000","last_became_peered":"2023-10-16T13:50:46.406451+0000","last_unstale":"2023-10-16T13:50:47.424763+0000","last_undegraded":"2023-10-16T13:50:47.424763+0000","last_fullsized":"2023-10-16T13:50:47.424763+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:17.905413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.443207+0000","last_change":"2023-10-16T13:50:47.449297+0000","last_active":"2023-10-16T13:50:48.443207+0000","last_peered":"2023-10-16T13:50:48.443207+0000","last_clean":"2023-10-16T13:50:48.443207+0000","last_became_active":"2023-10-16T13:50:47.447338+0000","last_became_peered":"2023-10-16T13:50:47.447338+0000","last_unstale":"2023-10-16T13:50:48.443207+0000","last_undegraded":"2023-10-16T13:50:48.443207+0000","last_fullsized":"2023-10-16T13:50:48.443207+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:09.153279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"182'2","reported_seq":13,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.450581+0000","last_change":"2023-10-16T13:50:46.421826+0000","last_active":"2023-10-16T13:50:47.450581+0000","last_peered":"2023-10-16T13:50:47.450581+0000","last_clean":"2023-10-16T13:50:47.450581+0000","last_became_active":"2023-10-16T13:50:46.421397+0000","last_became_peered":"2023-10-16T13:50:46.421397+0000","last_unstale":"2023-10-16T13:50:47.450581+0000","last_undegraded":"2023-10-16T13:50:47.450581+0000","last_fullsized":"2023-10-16T13:50:47.450581+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:31.564366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.454127+0000","last_change":"2023-10-16T13:50:47.452419+0000","last_active":"2023-10-16T13:50:47.454127+0000","last_peered":"2023-10-16T13:50:47.454127+0000","last_clean":"2023-10-16T13:50:47.454127+0000","last_became_active":"2023-10-16T13:50:47.451831+0000","last_became_peered":"2023-10-16T13:50:47.451831+0000","last_unstale":"2023-10-16T13:50:47.454127+0000","last_undegraded":"2023-10-16T13:50:47.454127+0000","last_fullsized":"2023-10-16T13:50:47.454127+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:50.067196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.425789+0000","last_change":"2023-10-16T13:50:46.415644+0000","last_active":"2023-10-16T13:50:47.425789+0000","last_peered":"2023-10-16T13:50:47.425789+0000","last_clean":"2023-10-16T13:50:47.425789+0000","last_became_active":"2023-10-16T13:50:46.414611+0000","last_became_peered":"2023-10-16T13:50:46.414611+0000","last_unstale":"2023-10-16T13:50:47.425789+0000","last_undegraded":"2023-10-16T13:50:47.425789+0000","last_fullsized":"2023-10-16T13:50:47.425789+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:39.153337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.430449+0000","last_change":"2023-10-16T13:50:47.428380+0000","last_active":"2023-10-16T13:50:47.430449+0000","last_peered":"2023-10-16T13:50:47.430449+0000","last_clean":"2023-10-16T13:50:47.430449+0000","last_became_active":"2023-10-16T13:50:47.428022+0000","last_became_peered":"2023-10-16T13:50:47.428022+0000","last_unstale":"2023-10-16T13:50:47.430449+0000","last_undegraded":"2023-10-16T13:50:47.430449+0000","last_fullsized":"2023-10-16T13:50:47.430449+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:00.922242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.426929+0000","last_change":"2023-10-16T13:50:46.399487+0000","last_active":"2023-10-16T13:50:47.426929+0000","last_peered":"2023-10-16T13:50:47.426929+0000","last_clean":"2023-10-16T13:50:47.426929+0000","last_became_active":"2023-10-16T13:50:46.399200+0000","last_became_peered":"2023-10-16T13:50:46.399200+0000","last_unstale":"2023-10-16T13:50:47.426929+0000","last_undegraded":"2023-10-16T13:50:47.426929+0000","last_fullsized":"2023-10-16T13:50:47.426929+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:47.308458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.446796+0000","last_change":"2023-10-16T13:50:47.443207+0000","last_active":"2023-10-16T13:50:47.446796+0000","last_peered":"2023-10-16T13:50:47.446796+0000","last_clean":"2023-10-16T13:50:47.446796+0000","last_became_active":"2023-10-16T13:50:47.442928+0000","last_became_peered":"2023-10-16T13:50:47.442928+0000","last_unstale":"2023-10-16T13:50:47.446796+0000","last_undegraded":"2023-10-16T13:50:47.446796+0000","last_fullsized":"2023-10-16T13:50:47.446796+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:58.450740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.426318+0000","last_change":"2023-10-16T13:50:46.407783+0000","last_active":"2023-10-16T13:50:47.426318+0000","last_peered":"2023-10-16T13:50:47.426318+0000","last_clean":"2023-10-16T13:50:47.426318+0000","last_became_active":"2023-10-16T13:50:46.405811+0000","last_became_peered":"2023-10-16T13:50:46.405811+0000","last_unstale":"2023-10-16T13:50:47.426318+0000","last_undegraded":"2023-10-16T13:50:47.426318+0000","last_fullsized":"2023-10-16T13:50:47.426318+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:39.848223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.482237+0000","last_change":"2023-10-16T13:50:47.443439+0000","last_active":"2023-10-16T13:50:48.482237+0000","last_peered":"2023-10-16T13:50:48.482237+0000","last_clean":"2023-10-16T13:50:48.482237+0000","last_became_active":"2023-10-16T13:50:47.443158+0000","last_became_peered":"2023-10-16T13:50:47.443158+0000","last_unstale":"2023-10-16T13:50:48.482237+0000","last_undegraded":"2023-10-16T13:50:48.482237+0000","last_fullsized":"2023-10-16T13:50:48.482237+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:47.622962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.434037+0000","last_change":"2023-10-16T13:50:46.416371+0000","last_active":"2023-10-16T13:50:48.434037+0000","last_peered":"2023-10-16T13:50:48.434037+0000","last_clean":"2023-10-16T13:50:48.434037+0000","last_became_active":"2023-10-16T13:50:46.415632+0000","last_became_peered":"2023-10-16T13:50:46.415632+0000","last_unstale":"2023-10-16T13:50:48.434037+0000","last_undegraded":"2023-10-16T13:50:48.434037+0000","last_fullsized":"2023-10-16T13:50:48.434037+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:37.640302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.442046+0000","last_change":"2023-10-16T13:50:47.440695+0000","last_active":"2023-10-16T13:50:48.442046+0000","last_peered":"2023-10-16T13:50:48.442046+0000","last_clean":"2023-10-16T13:50:48.442046+0000","last_became_active":"2023-10-16T13:50:47.440155+0000","last_became_peered":"2023-10-16T13:50:47.440155+0000","last_unstale":"2023-10-16T13:50:48.442046+0000","last_undegraded":"2023-10-16T13:50:48.442046+0000","last_fullsized":"2023-10-16T13:50:48.442046+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:40.246993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.428682+0000","last_change":"2023-10-16T13:50:46.419576+0000","last_active":"2023-10-16T13:50:47.428682+0000","last_peered":"2023-10-16T13:50:47.428682+0000","last_clean":"2023-10-16T13:50:47.428682+0000","last_became_active":"2023-10-16T13:50:46.419260+0000","last_became_peered":"2023-10-16T13:50:46.419260+0000","last_unstale":"2023-10-16T13:50:47.428682+0000","last_undegraded":"2023-10-16T13:50:47.428682+0000","last_fullsized":"2023-10-16T13:50:47.428682+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:00.287093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.455841+0000","last_change":"2023-10-16T13:50:47.452620+0000","last_active":"2023-10-16T13:50:47.455841+0000","last_peered":"2023-10-16T13:50:47.455841+0000","last_clean":"2023-10-16T13:50:47.455841+0000","last_became_active":"2023-10-16T13:50:47.452219+0000","last_became_peered":"2023-10-16T13:50:47.452219+0000","last_unstale":"2023-10-16T13:50:47.455841+0000","last_undegraded":"2023-10-16T13:50:47.455841+0000","last_fullsized":"2023-10-16T13:50:47.455841+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:09.851640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.431013+0000","last_change":"2023-10-16T13:50:46.409440+0000","last_active":"2023-10-16T13:50:47.431013+0000","last_peered":"2023-10-16T13:50:47.431013+0000","last_clean":"2023-10-16T13:50:47.431013+0000","last_became_active":"2023-10-16T13:50:46.408811+0000","last_became_peered":"2023-10-16T13:50:46.408811+0000","last_unstale":"2023-10-16T13:50:47.431013+0000","last_undegraded":"2023-10-16T13:50:47.431013+0000","last_fullsized":"2023-10-16T13:50:47.431013+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:20.281225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.441970+0000","last_change":"2023-10-16T13:50:47.429854+0000","last_active":"2023-10-16T13:50:48.441970+0000","last_peered":"2023-10-16T13:50:48.441970+0000","last_clean":"2023-10-16T13:50:48.441970+0000","last_became_active":"2023-10-16T13:50:47.429546+0000","last_became_peered":"2023-10-16T13:50:47.429546+0000","last_unstale":"2023-10-16T13:50:48.441970+0000","last_undegraded":"2023-10-16T13:50:48.441970+0000","last_fullsized":"2023-10-16T13:50:48.441970+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:54.353532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.420391+0000","last_change":"2023-10-16T13:50:46.414842+0000","last_active":"2023-10-16T13:50:47.420391+0000","last_peered":"2023-10-16T13:50:47.420391+0000","last_clean":"2023-10-16T13:50:47.420391+0000","last_became_active":"2023-10-16T13:50:46.414517+0000","last_became_peered":"2023-10-16T13:50:46.414517+0000","last_unstale":"2023-10-16T13:50:47.420391+0000","last_undegraded":"2023-10-16T13:50:47.420391+0000","last_fullsized":"2023-10-16T13:50:47.420391+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:52.012069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-10-16T13:50:48.434076+0000","last_change":"2023-10-16T13:50:47.423316+0000","last_active":"2023-10-16T13:50:48.434076+0000","last_peered":"2023-10-16T13:50:48.434076+0000","last_clean":"2023-10-16T13:50:48.434076+0000","last_became_active":"2023-10-16T13:50:47.423013+0000","last_became_peered":"2023-10-16T13:50:47.423013+0000","last_unstale":"2023-10-16T13:50:48.434076+0000","last_undegraded":"2023-10-16T13:50:48.434076+0000","last_fullsized":"2023-10-16T13:50:48.434076+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:46.375048+0000","last_clean_scrub_stamp":"2023-10-16T13:50:46.375048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:54.540727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"182'3","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-10-16T13:50:47.444802+0000","last_change":"2023-10-16T13:50:46.406671+0000","last_active":"2023-10-16T13:50:47.444802+0000","last_peered":"2023-10-16T13:50:47.444802+0000","last_clean":"2023-10-16T13:50:47.444802+0000","last_became_active":"2023-10-16T13:50:46.406328+0000","last_became_peered":"2023-10-16T13:50:46.406328+0000","last_unstale":"2023-10-16T13:50:47.444802+0000","last_undegraded":"2023-10-16T13:50:47.444802+0000","last_fullsized":"2023-10-16T13:50:47.444802+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:50:45.358996+0000","last_clean_scrub_stamp":"2023-10-16T13:50:45.358996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:46.887415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"170'274","reported_seq":899,"reported_epoch":181,"state":"active+clean","last_fresh":"2023-10-16T13:50:46.413474+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T13:50:46.413474+0000","last_peered":"2023-10-16T13:50:46.413474+0000","last_clean":"2023-10-16T13:50:46.413474+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T13:50:46.413474+0000","last_undegraded":"2023-10-16T13:50:46.413474+0000","last_fullsized":"2023-10-16T13:50:46.413474+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1483,"num_read_kb":3066,"num_write":443,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543067,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47872,"kb_used_data":2800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2867200,"data_stored":1689645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3320000000000001,"15min":1.327},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.73999999999999999},"max":{"1min":1.7729999999999999,"5min":1.835,"15min":1.976},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.423,"15min":1.407},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.302,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.294}]},{"osd":1,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1779999999999999,"15min":1.218},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6279999999999999,"5min":1.637,"15min":1.8100000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.321,"15min":1.28},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.67900000000000005},"max":{"1min":1.901,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.6160000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.276,"15min":1.274},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.5529999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3440000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7450000000000001,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.718}]},{"osd":3,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.399,"15min":1.385},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.8120000000000001,"5min":1.871,"15min":2.1070000000000002},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4710000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.84999999999999998},"max":{"1min":2.093,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.7829999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3,"15min":1.2609999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.976,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.494},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.446,"15min":1.399},"min":{"1min":0.96699999999999997,"5min":0.84399999999999997,"15min":0.81599999999999995},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.1829999999999998},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.4379999999999999,"15min":1.399},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.1880000000000002,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":1.0389999999999999,"5min":1.0369999999999999,"15min":0.94199999999999995},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":2.9700000000000002},"last":1.6699999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.637,"15min":1.6180000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.2229999999999999,"5min":2.306,"15min":2.306},"last":1.74},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.532,"15min":1.524},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85199999999999998},"max":{"1min":1.8919999999999999,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.8560000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543068,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2277376,"data_stored":1098771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.349,"15min":1.3420000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.69599999999999995},"max":{"1min":1.748,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.8049999999999999,"5min":1.8169999999999999,"15min":1.9990000000000001},"last":1.476}]},{"osd":1,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.208,"15min":1.23},"min":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.82099999999999995},"max":{"1min":1.6299999999999999,"5min":1.6319999999999999,"15min":1.8280000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2270000000000001,"15min":1.232},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6890000000000001,"5min":1.776,"15min":1.788},"last":1.224}]},{"osd":2,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2090000000000001,"15min":1.254},"min":{"1min":0.89900000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6990000000000001,"5min":1.897,"15min":2.0350000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.325,"15min":1.3240000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.72,"5min":1.77,"15min":1.859},"last":1.675}]},{"osd":3,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.3759999999999999},"min":{"1min":0.93999999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.764,"5min":1.946,"15min":1.9850000000000001},"last":1.127},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4039999999999999,"15min":1.399},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.8200000000000001,"5min":1.8400000000000001,"15min":1.9159999999999999},"last":1.8029999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.395,"15min":1.3779999999999999},"min":{"1min":1.1020000000000001,"5min":0.86799999999999999,"15min":0.79600000000000004},"max":{"1min":1.7529999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.542,"15min":1.5369999999999999},"min":{"1min":1.3440000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.02,"5min":2.02,"15min":2.1219999999999999},"last":1.891}]},{"osd":5,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.359,"15min":1.331},"min":{"1min":0.876,"5min":0.86899999999999999,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":1.903,"15min":2.2669999999999999},"last":1.546},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.498,"15min":1.472},"min":{"1min":0.91500000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.895,"5min":1.9690000000000001,"15min":2.0510000000000002},"last":1.2969999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.4810000000000001},"min":{"1min":1.2190000000000001,"5min":0.88700000000000001,"15min":0.87},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.173},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.548,"15min":1.512},"min":{"1min":1.1779999999999999,"5min":1.0129999999999999,"15min":0.92500000000000004},"max":{"1min":1.923,"5min":1.9850000000000001,"15min":2.1419999999999999},"last":1.8440000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543069,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2269184,"data_stored":1098647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2390000000000001,"15min":1.218},"min":{"1min":0.96699999999999997,"5min":0.72399999999999998,"15min":0.70099999999999996},"max":{"1min":1.6990000000000001,"5min":1.78,"15min":1.821},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.2689999999999999,"15min":1.2769999999999999},"min":{"1min":0.96599999999999997,"5min":0.83399999999999996,"15min":0.745},"max":{"1min":1.8,"5min":1.839,"15min":1.9610000000000001},"last":1.046}]},{"osd":1,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3089999999999999,"15min":1.3160000000000001},"min":{"1min":0.92900000000000005,"5min":0.89200000000000002,"15min":0.82799999999999996},"max":{"1min":1.669,"5min":1.992,"15min":2.1640000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.256,"15min":1.272},"min":{"1min":1.024,"5min":0.84999999999999998,"15min":0.76000000000000001},"max":{"1min":1.694,"5min":1.754,"15min":2.431},"last":1.165}]},{"osd":2,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3460000000000001,"15min":1.3540000000000001},"min":{"1min":1.1619999999999999,"5min":0.94099999999999995,"15min":0.90200000000000002},"max":{"1min":1.6890000000000001,"5min":1.8029999999999999,"15min":2.0209999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3440000000000001,"15min":1.3300000000000001},"min":{"1min":1.1000000000000001,"5min":0.89800000000000002,"15min":0.876},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.93500000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.373,"15min":1.3979999999999999},"min":{"1min":1.03,"5min":0.98299999999999998,"15min":0.95499999999999996},"max":{"1min":1.746,"5min":2.0990000000000002,"15min":2.5289999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.4670000000000001,"15min":1.4490000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.93899999999999995},"max":{"1min":1.7589999999999999,"5min":1.952,"15min":2.242},"last":1.4239999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.3440000000000001,"15min":1.2909999999999999},"min":{"1min":0.88900000000000001,"5min":0.82399999999999995,"15min":0.69399999999999995},"max":{"1min":1.782,"5min":1.8,"15min":2.3679999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.446,"15min":1.423},"min":{"1min":0.93999999999999995,"5min":0.81599999999999995,"15min":0.80300000000000005},"max":{"1min":1.853,"5min":2.2530000000000001,"15min":2.4020000000000001},"last":1.4690000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.4319999999999999,"15min":1.383},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.88},"max":{"1min":1.883,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.579},{"interface":"front","average":{"1min":1.528,"5min":1.494,"15min":1.48},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":0.97299999999999998},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.3460000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:49:51 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.532},"min":{"1min":1.1459999999999999,"5min":0.96999999999999997,"15min":0.93300000000000005},"max":{"1min":1.96,"5min":2.371,"15min":2.6320000000000001},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.617,"15min":1.5980000000000001},"min":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0089999999999999},"max":{"1min":1.9470000000000001,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.522}]}]},{"osd":4,"up_from":14,"seq":60129543069,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55608,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316232,"statfs":{"total":96636764160,"available":96579821568,"internally_reserved":0,"allocated":2265088,"data_stored":1098625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2450000000000001,"15min":1.234},"min":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.69499999999999995},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.194,"15min":1.2170000000000001},"min":{"1min":0.86899999999999999,"5min":0.82399999999999995,"15min":0.77900000000000003},"max":{"1min":1.5049999999999999,"5min":1.645,"15min":2.073},"last":0.999}]},{"osd":1,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.27,"15min":1.2490000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.82999999999999996},"max":{"1min":1.5549999999999999,"5min":1.8049999999999999,"15min":1.8069999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.161,"5min":1.29,"15min":1.3080000000000001},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.83399999999999996},"max":{"1min":1.6080000000000001,"5min":1.7729999999999999,"15min":1.887},"last":0.67200000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3069999999999999,"15min":1.2989999999999999},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.88400000000000001},"max":{"1min":1.651,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.081},{"interface":"front","average":{"1min":1.282,"5min":1.3799999999999999,"15min":1.411},"min":{"1min":1.0449999999999999,"5min":1.0149999999999999,"15min":0.97599999999999998},"max":{"1min":1.528,"5min":1.861,"15min":2.1840000000000002},"last":0.88800000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.3029999999999999,"15min":1.3100000000000001},"min":{"1min":0.97999999999999998,"5min":0.90000000000000002,"15min":0.84899999999999998},"max":{"1min":1.7430000000000001,"5min":1.7909999999999999,"15min":2.052},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.403,"5min":1.4119999999999999,"15min":1.4299999999999999},"min":{"1min":1.135,"5min":0.995,"15min":0.90600000000000003},"max":{"1min":1.6479999999999999,"5min":2.294,"15min":2.294},"last":1.2130000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3460000000000001,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.089},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.339,"5min":1.3320000000000001,"15min":1.325},"min":{"1min":0.871,"5min":0.871,"15min":0.752},"max":{"1min":1.8979999999999999,"5min":2.2360000000000002,"15min":2.4430000000000001},"last":1.147}]},{"osd":6,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3640000000000001,"15min":1.3859999999999999},"min":{"1min":0.85199999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.8109999999999999,"5min":2.121,"15min":2.383},"last":1.371},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5029999999999999,"15min":1.512},"min":{"1min":1.069,"5min":1.0269999999999999,"15min":0.89000000000000001},"max":{"1min":2.1030000000000002,"5min":2.1030000000000002,"15min":2.1549999999999998},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:49:54 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4490000000000001,"15min":1.4910000000000001},"min":{"1min":1.036,"5min":0.92800000000000005,"15min":0.92600000000000005},"max":{"1min":1.9650000000000001,"5min":2.3570000000000002,"15min":2.6459999999999999},"last":1.274},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.6240000000000001,"15min":1.6399999999999999},"min":{"1min":1.417,"5min":1.103,"15min":1.004},"max":{"1min":2.0179999999999998,"5min":2.4049999999999998,"15min":2.6030000000000002},"last":1.3140000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51448,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320392,"statfs":{"total":96636764160,"available":96584081408,"internally_reserved":0,"allocated":2269184,"data_stored":1098715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97099999999999997,"15min":0.97999999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.04,"5min":2.04,"15min":3.1000000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.242,"15min":1.2390000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.74099999999999999},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":2.714},"last":0.90400000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.133,"15min":1.1020000000000001},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.73299999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.143,"15min":1.139},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"max":{"1min":1.5529999999999999,"5min":1.879,"15min":1.879},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.1919999999999999,"15min":1.206},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.234,"5min":2.234,"15min":2.782},"last":0.627},{"interface":"front","average":{"1min":1.5,"5min":1.468,"15min":1.48},"min":{"1min":1.242,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.609},"last":0.84199999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.5009999999999999,"15min":1.524},"min":{"1min":1.28,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":1.8160000000000001,"5min":1.8979999999999999,"15min":2.2330000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.4379999999999999,"15min":1.472},"min":{"1min":1.022,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.1000000000000001},"last":1.222}]},{"osd":5,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.4530000000000001,"15min":1.4650000000000001},"min":{"1min":1.137,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.0990000000000002,"5min":2.1110000000000002,"15min":2.306},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.615,"5min":1.5649999999999999,"15min":1.5369999999999999},"min":{"1min":1.2350000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.2559999999999998,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.325}]},{"osd":6,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.0900000000000001,"5min":1.0489999999999999,"15min":1.032},"max":{"1min":2.028,"5min":2.1389999999999998,"15min":2.4430000000000001},"last":1.129},{"interface":"front","average":{"1min":1.532,"5min":1.516,"15min":1.506},"min":{"1min":1.177,"5min":1.1339999999999999,"15min":1.056},"max":{"1min":2.1779999999999999,"5min":2.2519999999999998,"15min":2.387},"last":1.1040000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:50:03 2023","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.649,"15min":1.6240000000000001},"min":{"1min":1.3240000000000001,"5min":1.121,"15min":1.099},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.3340000000000001},"last":1.431},{"interface":"front","average":{"1min":1.756,"5min":1.6990000000000001,"15min":1.6839999999999999},"min":{"1min":1.2969999999999999,"5min":1.252,"15min":1.252},"max":{"1min":2.3119999999999998,"5min":2.387,"15min":2.387},"last":1.175}]}]},{"osd":2,"up_from":14,"seq":60129543067,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51456,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320384,"statfs":{"total":96636764160,"available":96584073216,"internally_reserved":0,"allocated":2281472,"data_stored":1100307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3480000000000001,"5min":1.732,"15min":2.1499999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.125,"15min":1.159},"min":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.367,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.141,"15min":1.151},"min":{"1min":0.75,"5min":0.75,"15min":0.72199999999999998},"max":{"1min":1.532,"5min":1.956,"15min":2.4550000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.2809999999999999,"15min":1.286},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.754,"5min":1.804,"15min":2.3490000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1339999999999999,"15min":1.1519999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.806,"5min":1.849,"15min":1.982},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.345,"5min":1.359,"15min":1.379},"min":{"1min":0.98799999999999999,"5min":0.91200000000000003,"15min":0.90700000000000003},"max":{"1min":1.7090000000000001,"5min":1.855,"15min":2.7970000000000002},"last":1.2949999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4950000000000001,"15min":1.5069999999999999},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.054},"max":{"1min":1.9430000000000001,"5min":2.1429999999999998,"15min":2.6589999999999998},"last":1.444},{"interface":"front","average":{"1min":1.452,"5min":1.492,"15min":1.514},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":1.762,"5min":2.0510000000000002,"15min":2.2629999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.575},"min":{"1min":1.129,"5min":1.129,"15min":1.056},"max":{"1min":1.921,"5min":2.101,"15min":2.5569999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.5009999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.7490000000000001,"5min":2.2650000000000001,"15min":2.899},"last":1.2350000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5549999999999999},"min":{"1min":1.0760000000000001,"5min":1.032,"15min":1.032},"max":{"1min":1.843,"5min":2.4300000000000002,"15min":3.0339999999999998},"last":1.389},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6080000000000001,"15min":1.637},"min":{"1min":1.145,"5min":1.145,"15min":1.145},"max":{"1min":1.982,"5min":2.0779999999999998,"15min":3.0790000000000002},"last":1.5069999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7050000000000001,"15min":1.6919999999999999},"min":{"1min":1.24,"5min":1.127,"15min":1.127},"max":{"1min":2.1880000000000002,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.651,"5min":1.643,"15min":1.661},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2429999999999999,"5min":2.5529999999999999,"15min":3.1320000000000001},"last":1.331}]}]},{"osd":1,"up_from":14,"seq":60129543067,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47284,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324556,"statfs":{"total":96636764160,"available":96588345344,"internally_reserved":0,"allocated":2269184,"data_stored":1099207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94699999999999995,"15min":0.94599999999999995},"min":{"1min":0.73799999999999999,"5min":0.69999999999999996,"15min":0.497},"max":{"1min":1.2729999999999999,"5min":1.637,"15min":1.6970000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2,"15min":1.1910000000000001},"min":{"1min":0.85499999999999998,"5min":0.80100000000000005,"15min":0.56699999999999995},"max":{"1min":1.766,"5min":1.8,"15min":1.8089999999999999},"last":1.2270000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.052,"15min":1.038},"min":{"1min":0.81599999999999995,"5min":0.81499999999999995,"15min":0.74399999999999999},"max":{"1min":1.71,"5min":1.71,"15min":1.8069999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.254,"15min":1.26},"min":{"1min":0.873,"5min":0.873,"15min":0.85299999999999998},"max":{"1min":1.992,"5min":1.992,"15min":1.994},"last":1.149}]},{"osd":3,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.628},"max":{"1min":1.7729999999999999,"5min":1.883,"15min":2.226},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.29,"15min":1.292},"min":{"1min":0.92500000000000004,"5min":0.85099999999999998,"15min":0.68100000000000005},"max":{"1min":1.585,"5min":2.1269999999999998,"15min":2.2610000000000001},"last":1.1160000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3879999999999999,"15min":1.3500000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.86199999999999999},"max":{"1min":1.867,"5min":1.923,"15min":2.1539999999999999},"last":1.081},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5189999999999999,"15min":1.516},"min":{"1min":1.1930000000000001,"5min":1.0880000000000001,"15min":0.71899999999999997},"max":{"1min":1.974,"5min":2.0459999999999998,"15min":2.2240000000000002},"last":0.997}]},{"osd":5,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.51,"15min":1.5069999999999999},"min":{"1min":1.161,"5min":1.1080000000000001,"15min":0.84199999999999997},"max":{"1min":2.016,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4810000000000001,"15min":1.4690000000000001},"min":{"1min":1.1379999999999999,"5min":1.071,"15min":0.92300000000000004},"max":{"1min":1.9159999999999999,"5min":1.9970000000000001,"15min":2.028},"last":1.522}]},{"osd":6,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.6419999999999999,"15min":1.617},"min":{"1min":1.256,"5min":1.1830000000000001,"15min":0.877},"max":{"1min":2.0670000000000002,"5min":2.363,"15min":2.3879999999999999},"last":1.484},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6180000000000001,"15min":1.611},"min":{"1min":1.2529999999999999,"5min":1.1879999999999999,"15min":0.76900000000000002},"max":{"1min":2.105,"5min":2.306,"15min":2.306},"last":1.954}]},{"osd":7,"last update":"Mon Oct 16 13:49:50 2023","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.736,"15min":1.708},"min":{"1min":1.2969999999999999,"5min":1.242,"15min":0.95499999999999996},"max":{"1min":2.0920000000000001,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":2.04},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6319999999999999,"15min":1.6180000000000001},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.0779999999999998,"5min":2.1040000000000001,"15min":2.3140000000000001},"last":1.6559999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543068,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60364,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311476,"statfs":{"total":96636764160,"available":96574951424,"internally_reserved":0,"allocated":2875392,"data_stored":1690639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.93700000000000006,"15min":0.93100000000000005},"min":{"1min":0.754,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.1759999999999999,"15min":1.161},"min":{"1min":0.72299999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.6830000000000001,"5min":1.835,"15min":1.841},"last":0.79600000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1859999999999999,"15min":1.171},"min":{"1min":0.79400000000000004,"5min":0.76500000000000001,"15min":0.74399999999999999},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.9830000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2589999999999999,"15min":1.256},"min":{"1min":0.89900000000000002,"5min":0.86599999999999999,"15min":0.83999999999999997},"max":{"1min":1.863,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.079}]},{"osd":3,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1639999999999999,"15min":1.1259999999999999},"min":{"1min":0.85799999999999998,"5min":0.80900000000000005,"15min":0.72999999999999998},"max":{"1min":2.3340000000000001,"5min":3.0350000000000001,"15min":3.0350000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.397,"15min":1.3779999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.871},"max":{"1min":2.2450000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.613}]},{"osd":4,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4179999999999999,"15min":1.3979999999999999},"min":{"1min":1.022,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":1.7330000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.321},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5309999999999999,"15min":1.5129999999999999},"min":{"1min":1.248,"5min":1.1140000000000001,"15min":1.048},"max":{"1min":1.9430000000000001,"5min":4.6159999999999997,"15min":4.6159999999999997},"last":1.3520000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.538,"15min":1.528},"min":{"1min":1.321,"5min":1.1359999999999999,"15min":1.04},"max":{"1min":2.0070000000000001,"5min":2.1560000000000001,"15min":2.2349999999999999},"last":1.22},{"interface":"front","average":{"1min":1.482,"5min":1.488,"15min":1.454},"min":{"1min":1.1200000000000001,"5min":1.0549999999999999,"15min":0.97599999999999998},"max":{"1min":2.0880000000000001,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.431}]},{"osd":6,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.659,"15min":1.625},"min":{"1min":1.327,"5min":1.1240000000000001,"15min":1.1240000000000001},"max":{"1min":2.2610000000000001,"5min":4.0270000000000001,"15min":4.0270000000000001},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6120000000000001,"15min":1.611},"min":{"1min":1.204,"5min":1.1870000000000001,"15min":1.163},"max":{"1min":2.0920000000000001,"5min":2.254,"15min":2.254},"last":1.5649999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:49:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.7290000000000001,"15min":1.7130000000000001},"min":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.202},"max":{"1min":2.1989999999999998,"5min":2.7149999999999999,"15min":2.7149999999999999},"last":1.649},{"interface":"front","average":{"1min":1.714,"5min":1.6519999999999999,"15min":1.6220000000000001},"min":{"1min":1.3280000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":2.286,"5min":3.3010000000000002,"15min":3.3010000000000002},"last":1.698}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:50:52.459 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:50:52.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:50:52.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.736+0000 7fbc9dd33640 1 Processor -- start 2023-10-16T13:50:52.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.736+0000 7fbc9dd33640 1 -- start start 2023-10-16T13:50:52.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc9dd33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc9815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc9dd33640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc980ce390 0x7fbc98070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc9dd33640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980707e0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc9dd33640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9806b7a0 con 0x7fbc980ce390 2023-10-16T13:50:52.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc9dd33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9815ef20 con 0x7fbc9815ae80 2023-10-16T13:50:52.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc8ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc9815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc97fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc980ce390 0x7fbc98070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc8ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc9815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:45364/0 (socket says 172.21.15.60:45364) 2023-10-16T13:50:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.738+0000 7fbc8ffff640 1 -- 172.21.15.60:0/3930914843 learned_addr learned my addr 172.21.15.60:0/3930914843 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.739+0000 7fbc97fff640 1 -- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 msgr2=0x7fbc9815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.739+0000 7fbc97fff640 1 --2- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc9815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.739+0000 7fbc97fff640 1 -- 172.21.15.60:0/3930914843 >> v1:172.21.15.144:6789/0 conn(0x7fbc9815aaa0 legacy=0x7fbc980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.739+0000 7fbc97fff640 1 -- 172.21.15.60:0/3930914843 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9815f200 con 0x7fbc980ce390 2023-10-16T13:50:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.739+0000 7fbc8ffff640 1 --2- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc9815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:50:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.740+0000 7fbc97fff640 1 --2- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc98070f50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009d90 tx=0x7fbc9002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc3579d644d1d058 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.741+0000 7fbc96ffd640 1 -- 172.21.15.60:0/3930914843 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9002cbc0 con 0x7fbc980ce390 2023-10-16T13:50:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.741+0000 7fbc96ffd640 1 -- 172.21.15.60:0/3930914843 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9002cd20 con 0x7fbc980ce390 2023-10-16T13:50:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.741+0000 7fbc96ffd640 1 -- 172.21.15.60:0/3930914843 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90034660 con 0x7fbc980ce390 2023-10-16T13:50:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 msgr2=0x7fbc98070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc98070f50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009d90 tx=0x7fbc9002be10 comp rx=0 tx=0).stop 2023-10-16T13:50:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3930914843 shutdown_connections 2023-10-16T13:50:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc98070f50 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3930914843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc9815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3930914843 >> 172.21.15.60:0/3930914843 conn(0x7fbc980cb410 msgr2=0x7fbc980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3930914843 shutdown_connections 2023-10-16T13:50:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.742+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3930914843 wait complete. 2023-10-16T13:50:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.745+0000 7fbc9dd33640 1 Processor -- start 2023-10-16T13:50:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.745+0000 7fbc9dd33640 1 -- start start 2023-10-16T13:50:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.746+0000 7fbc9dd33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc9814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.746+0000 7fbc9dd33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 0x7fbc980ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.746+0000 7fbc9dd33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc980acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.746+0000 7fbc9dd33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9815b6c0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.746+0000 7fbc9dd33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980c4c60 con 0x7fbc980ce390 2023-10-16T13:50:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.746+0000 7fbc9dd33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9815dc50 con 0x7fbc9815ae80 2023-10-16T13:50:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 0x7fbc980ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 0x7fbc980ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53734/0 (socket says 172.21.15.60:53734) 2023-10-16T13:50:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 -- 172.21.15.60:0/3933008233 learned_addr learned my addr 172.21.15.60:0/3933008233 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:52.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc977fe640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc9814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc97fff640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc980acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 -- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 msgr2=0x7fbc980acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc980acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 -- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 msgr2=0x7fbc9814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc9814dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 -- 172.21.15.60:0/3933008233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800094d0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc977fe640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc9814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.747+0000 7fbc8ffff640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 0x7fbc980ac890 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000e9e0 tx=0x7fbc8000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.748+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8000c9d0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.748+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8000cb30 con 0x7fbc9815aaa0 2023-10-16T13:50:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.748+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8000cea0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.748+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc90009a70 con 0x7fbc9815aaa0 2023-10-16T13:50:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.748+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc980c4ed0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.749+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9804e160 con 0x7fbc9815aaa0 2023-10-16T13:50:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.750+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbc80013470 con 0x7fbc9815aaa0 2023-10-16T13:50:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.752+0000 7fbc94ff9640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbc7806afc0 0x7fbc7806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.752+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc80089aa0 con 0x7fbc9815aaa0 2023-10-16T13:50:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.753+0000 7fbc977fe640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbc7806afc0 0x7fbc7806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:52.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.754+0000 7fbc977fe640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbc7806afc0 0x7fbc7806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc88000f60 tx=0x7fbc88009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:52.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:52.756+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbc800549b0 con 0x7fbc9815aaa0 2023-10-16T13:50:53.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.024+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbc980cdc60 con 0x7fbc9815aaa0 2023-10-16T13:50:53.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.029+0000 7fbc94ff9640 1 -- 172.21.15.60:0/3933008233 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v47) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbc800210e0 con 0x7fbc9815aaa0 2023-10-16T13:50:53.033 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T13:50:53.033 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T13:50:53.033 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T13:50:53.033 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T13:50:53.034 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:50:53.034 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T13:50:53.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbc7806afc0 msgr2=0x7fbc7806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbc7806afc0 0x7fbc7806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc88000f60 tx=0x7fbc88009040 comp rx=0 tx=0).stop 2023-10-16T13:50:53.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 msgr2=0x7fbc980ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 0x7fbc980ac890 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000e9e0 tx=0x7fbc8000eea0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 shutdown_connections 2023-10-16T13:50:53.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7fbc7806afc0 0x7fbc7806d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbc9815ae80 0x7fbc980acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc9815aaa0 0x7fbc980ac890 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 --2- 172.21.15.60:0/3933008233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc980ce390 0x7fbc9814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 >> 172.21.15.60:0/3933008233 conn(0x7fbc980cb410 msgr2=0x7fbc980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:53.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 shutdown_connections 2023-10-16T13:50:53.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.032+0000 7fbc9dd33640 1 -- 172.21.15.60:0/3933008233 wait complete. 2023-10-16T13:50:53.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:53.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.272+0000 7f411b127640 1 Processor -- start 2023-10-16T13:50:53.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.272+0000 7f411b127640 1 -- start start 2023-10-16T13:50:53.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.273+0000 7f411b127640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f411406c040 0x7f411406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:53.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.274+0000 7f411b127640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411415f4a0 con 0x7f41140ad970 2023-10-16T13:50:53.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.274+0000 7f411b127640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411406c960 con 0x7f41140cda80 2023-10-16T13:50:53.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.274+0000 7f411b127640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411415b9d0 con 0x7f411406c040 2023-10-16T13:50:53.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.274+0000 7f411a125640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f411406c040 0x7f411406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:53.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.274+0000 7f4119924640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f41140ad970 0x7f411415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39732/0 (socket says 172.21.15.60:39732) 2023-10-16T13:50:53.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.274+0000 7f4119924640 1 -- 172.21.15.60:0/1171287458 learned_addr learned my addr 172.21.15.60:0/1171287458 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:53.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.275+0000 7f411a125640 1 -- 172.21.15.60:0/1171287458 >> v1:172.21.15.144:6789/0 conn(0x7f41140cda80 legacy=0x7f411415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:53.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.275+0000 7f411a125640 1 -- 172.21.15.60:0/1171287458 >> v1:172.21.15.60:6789/0 conn(0x7f41140ad970 legacy=0x7f411415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.275+0000 7f411a125640 1 -- 172.21.15.60:0/1171287458 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411415bcb0 con 0x7f411406c040 2023-10-16T13:50:53.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.276+0000 7f411a125640 1 --2- 172.21.15.60:0/1171287458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f411406c040 0x7f411406c420 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d90 tx=0x7f411002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850ccddc830dab42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:53.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.276+0000 7f4119123640 1 -- 172.21.15.60:0/1171287458 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411002cc60 con 0x7f411406c040 2023-10-16T13:50:53.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.276+0000 7f4119123640 1 -- 172.21.15.60:0/1171287458 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411002cdc0 con 0x7f411406c040 2023-10-16T13:50:53.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.276+0000 7f4119123640 1 -- 172.21.15.60:0/1171287458 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4110034660 con 0x7f411406c040 2023-10-16T13:50:53.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 -- 172.21.15.60:0/1171287458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f411406c040 msgr2=0x7f411406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 --2- 172.21.15.60:0/1171287458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f411406c040 0x7f411406c420 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f4110009d90 tx=0x7f411002beb0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 -- 172.21.15.60:0/1171287458 shutdown_connections 2023-10-16T13:50:53.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 --2- 172.21.15.60:0/1171287458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f411406c040 0x7f411406c420 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 -- 172.21.15.60:0/1171287458 >> 172.21.15.60:0/1171287458 conn(0x7f41140ce6a0 msgr2=0x7f41140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:53.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 -- 172.21.15.60:0/1171287458 shutdown_connections 2023-10-16T13:50:53.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.277+0000 7f411b127640 1 -- 172.21.15.60:0/1171287458 wait complete. 2023-10-16T13:50:53.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.279+0000 7f411b127640 1 Processor -- start 2023-10-16T13:50:53.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.279+0000 7f411b127640 1 -- start start 2023-10-16T13:50:53.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f411b127640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f411406c040 0x7f4114153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:53.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f411b127640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 0x7f41140c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:53.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f411b127640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41140cda80 0x7f4114153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:53.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f411b127640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c8080 con 0x7f41140ad970 2023-10-16T13:50:53.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f411b127640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41141535a0 con 0x7f411406c040 2023-10-16T13:50:53.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f411b127640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c6320 con 0x7f41140cda80 2023-10-16T13:50:53.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f4119924640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 0x7f41140c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:53.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f4119924640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 0x7f41140c6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53750/0 (socket says 172.21.15.60:53750) 2023-10-16T13:50:53.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.280+0000 7f4119924640 1 -- 172.21.15.60:0/3760268815 learned_addr learned my addr 172.21.15.60:0/3760268815 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:53.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f411a125640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f411406c040 0x7f4114153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:53.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f4119924640 1 -- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41140cda80 msgr2=0x7f4114153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f4119924640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41140cda80 0x7f4114153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f4119924640 1 -- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f411406c040 msgr2=0x7f4114153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f4119924640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f411406c040 0x7f4114153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f4119924640 1 -- 172.21.15.60:0/3760268815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110009a70 con 0x7f41140ad970 2023-10-16T13:50:53.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f4119924640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 0x7f41140c6dc0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f410400c530 tx=0x7f410400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:53.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.281+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104005ce0 con 0x7f41140ad970 2023-10-16T13:50:53.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.282+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104005e40 con 0x7f41140ad970 2023-10-16T13:50:53.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.282+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41140c7da0 con 0x7f41140ad970 2023-10-16T13:50:53.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.282+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104007270 con 0x7f41140ad970 2023-10-16T13:50:53.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.282+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41140c68a0 con 0x7f41140ad970 2023-10-16T13:50:53.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.283+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411406b330 con 0x7f41140ad970 2023-10-16T13:50:53.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.284+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f41040073d0 con 0x7f41140ad970 2023-10-16T13:50:53.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.285+0000 7f410affd640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f40f806af90 0x7f40f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:53.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.285+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4104088c10 con 0x7f41140ad970 2023-10-16T13:50:53.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.286+0000 7f411a125640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f40f806af90 0x7f40f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:53.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.287+0000 7f411a125640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f40f806af90 0x7f40f806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4110036000 tx=0x7f4110030000 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:53.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.289+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4104053bc0 con 0x7f41140ad970 2023-10-16T13:50:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.529+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41140c6150 con 0x7f41140ad970 2023-10-16T13:50:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.530+0000 7f410affd640 1 -- 172.21.15.60:0/3760268815 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f4104017020 con 0x7f41140ad970 2023-10-16T13:50:53.534 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:50:53.534 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:50:46.866966+0000","modified":"2023-10-16T13:50:49.938566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/691868089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":691868089},{"type":"v1","addr":"172.21.15.60:6834","nonce":691868089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:50:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 87 2023-10-16T13:50:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f40f806af90 msgr2=0x7f40f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f40f806af90 0x7f40f806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4110036000 tx=0x7f4110030000 comp rx=0 tx=0).stop 2023-10-16T13:50:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 msgr2=0x7f41140c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 0x7f41140c6dc0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f410400c530 tx=0x7f410400c9f0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 shutdown_connections 2023-10-16T13:50:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f40f806af90 0x7f40f806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41140cda80 0x7f4114153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41140ad970 0x7f41140c6dc0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 --2- 172.21.15.60:0/3760268815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f411406c040 0x7f4114153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 >> 172.21.15.60:0/3760268815 conn(0x7f41140ce6a0 msgr2=0x7f41140ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 shutdown_connections 2023-10-16T13:50:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:53.533+0000 7f411b127640 1 -- 172.21.15.60:0/3760268815 wait complete. 2023-10-16T13:50:53.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:50:46.866966+0000', 'modified': '2023-10-16T13:50:49.938566+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9510': {'gid': 9510, 'name': 'c', 'rank': 0, 'incarnation': 83, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6834/691868089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 691868089}, {'type': 'v1', 'addr': '172.21.15.60:6834', 'nonce': 691868089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:50:53.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:50:53.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T13:50:54.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.195+0000 7f69b620d640 1 Processor -- start 2023-10-16T13:50:54.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.196+0000 7f69b620d640 1 -- start start 2023-10-16T13:50:54.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.197+0000 7f69b620d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f69b015cd70 0x7f69b00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.197+0000 7f69b620d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69b00cded0 0x7f69b00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.197+0000 7f69b620d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b015d150 con 0x7f69b00b8480 2023-10-16T13:50:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.198+0000 7f69b620d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b00d1180 con 0x7f69b00cded0 2023-10-16T13:50:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.198+0000 7f69b620d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b00d1460 con 0x7f69b015cd70 2023-10-16T13:50:54.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.198+0000 7f69b5a0c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69b00cded0 0x7f69b00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.198+0000 7f69b4a0a640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f69b00b8480 0x7f69b00cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39734/0 (socket says 172.21.15.60:39734) 2023-10-16T13:50:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.198+0000 7f69b4a0a640 1 -- 172.21.15.60:0/3180738425 learned_addr learned my addr 172.21.15.60:0/3180738425 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.198+0000 7f69b520b640 1 --2- 172.21.15.60:0/3180738425 >> v2:172.21.15.144:3300/0 conn(0x7f69b015cd70 0x7f69b00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.199+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3180738425 >> v2:172.21.15.144:3300/0 conn(0x7f69b015cd70 msgr2=0x7f69b00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.199+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3180738425 >> v2:172.21.15.144:3300/0 conn(0x7f69b015cd70 0x7f69b00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.199+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3180738425 >> v1:172.21.15.60:6789/0 conn(0x7f69b00b8480 legacy=0x7f69b00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.199+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3180738425 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b00d1740 con 0x7f69b00cded0 2023-10-16T13:50:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.199+0000 7f69a7fff640 1 -- 172.21.15.60:0/3180738425 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957193743 0 0) 0x7f69b015d150 con 0x7f69b00b8480 2023-10-16T13:50:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.200+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3180738425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00cded0 0x7f69b00b5c90 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f69b00b56a0 tx=0x7f69a802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b13316045d1153b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.200+0000 7f69a7fff640 1 -- 172.21.15.60:0/3180738425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a802cc70 con 0x7f69b00cded0 2023-10-16T13:50:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.200+0000 7f69a7fff640 1 -- 172.21.15.60:0/3180738425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a802cdd0 con 0x7f69b00cded0 2023-10-16T13:50:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.200+0000 7f69a7fff640 1 -- 172.21.15.60:0/3180738425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8034660 con 0x7f69b00cded0 2023-10-16T13:50:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 -- 172.21.15.60:0/3180738425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00cded0 msgr2=0x7f69b00b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 --2- 172.21.15.60:0/3180738425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00cded0 0x7f69b00b5c90 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f69b00b56a0 tx=0x7f69a802bec0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 -- 172.21.15.60:0/3180738425 shutdown_connections 2023-10-16T13:50:54.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 --2- 172.21.15.60:0/3180738425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00cded0 0x7f69b00b5c90 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 --2- 172.21.15.60:0/3180738425 >> v2:172.21.15.144:3300/0 conn(0x7f69b015cd70 0x7f69b00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 -- 172.21.15.60:0/3180738425 >> 172.21.15.60:0/3180738425 conn(0x7f69b00cf570 msgr2=0x7f69b00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 -- 172.21.15.60:0/3180738425 shutdown_connections 2023-10-16T13:50:54.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.201+0000 7f69b620d640 1 -- 172.21.15.60:0/3180738425 wait complete. 2023-10-16T13:50:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.203+0000 7f69b620d640 1 Processor -- start 2023-10-16T13:50:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.203+0000 7f69b620d640 1 -- start start 2023-10-16T13:50:54.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b620d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 0x7f69b015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b620d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69b00cded0 0x7f69b00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b620d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69b015cd70 0x7f69b015f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b620d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b00cc0d0 con 0x7f69b00b8480 2023-10-16T13:50:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b620d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b015a840 con 0x7f69b015cd70 2023-10-16T13:50:54.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b620d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b00cb4c0 con 0x7f69b00cded0 2023-10-16T13:50:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b520b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 0x7f69b015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b520b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 0x7f69b015a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53760/0 (socket says 172.21.15.60:53760) 2023-10-16T13:50:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.204+0000 7f69b520b640 1 -- 172.21.15.60:0/1850643475 learned_addr learned my addr 172.21.15.60:0/1850643475 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69b00cded0 0x7f69b00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69b015cd70 0x7f69b015f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b520b640 1 -- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69b00cded0 msgr2=0x7f69b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b520b640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69b00cded0 0x7f69b00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b520b640 1 -- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69b015cd70 msgr2=0x7f69b015f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b520b640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69b015cd70 0x7f69b015f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b520b640 1 -- 172.21.15.60:0/1850643475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009a70 con 0x7f69b00b8480 2023-10-16T13:50:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69b00cded0 0x7f69b00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69b015cd70 0x7f69b015f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b520b640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 0x7f69b015a300 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00e870 tx=0x7f69ac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69a5ffb640 1 -- 172.21.15.60:0/1850643475 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0094f0 con 0x7f69b00b8480 2023-10-16T13:50:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69a5ffb640 1 -- 172.21.15.60:0/1850643475 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00c940 con 0x7f69b00b8480 2023-10-16T13:50:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69a5ffb640 1 -- 172.21.15.60:0/1850643475 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00ccf0 con 0x7f69b00b8480 2023-10-16T13:50:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.205+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b00cb7a0 con 0x7f69b00b8480 2023-10-16T13:50:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.206+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b00cbd60 con 0x7f69b00b8480 2023-10-16T13:50:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.209+0000 7f69a5ffb640 1 -- 172.21.15.60:0/1850643475 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69ac013470 con 0x7f69b00b8480 2023-10-16T13:50:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.210+0000 7f69a5ffb640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f699406af90 0x7f699406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.210+0000 7f69a5ffb640 1 -- 172.21.15.60:0/1850643475 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69ac089950 con 0x7f69b00b8480 2023-10-16T13:50:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.210+0000 7f69ba004740 1 Processor -- start 2023-10-16T13:50:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.210+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f699406af90 0x7f699406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.210+0000 7f69ba004740 1 -- start start 2023-10-16T13:50:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.211+0000 7f69ba004740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 0x55c58d1f8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.211+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225550 0x55c58d227f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.211+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d1f85b0 0x55c58d228440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.211+0000 7f69ba004740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d1187a0 con 0x55c58d225170 2023-10-16T13:50:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.211+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d16a740 con 0x55c58d225550 2023-10-16T13:50:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.211+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c58d167e30 con 0x55c58d1f85b0 2023-10-16T13:50:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f699406af90 0x7f699406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69b00d3560 tx=0x7f69a0009040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 0x55c58d1f8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 0x55c58d1f8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53764/0 (socket says 172.21.15.60:53764) 2023-10-16T13:50:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 -- 172.21.15.60:0/1549414740 learned_addr learned my addr 172.21.15.60:0/1549414740 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225550 0x55c58d227f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 -- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d1f85b0 msgr2=0x55c58d228440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d1f85b0 0x55c58d228440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 -- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225550 msgr2=0x55c58d227f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225550 0x55c58d227f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 -- 172.21.15.60:0/1549414740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c58d127910 con 0x55c58d225170 2023-10-16T13:50:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.212+0000 7f69b520b640 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 0x55c58d1f8070 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0584a0 tx=0x7f69ac0585d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea3f61a8258074 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f6992ffd640 1 -- 172.21.15.60:0/1549414740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac060810 con 0x55c58d225170 2023-10-16T13:50:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f6992ffd640 1 -- 172.21.15.60:0/1549414740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac087dd0 con 0x55c58d225170 2023-10-16T13:50:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f6992ffd640 1 -- 172.21.15.60:0/1549414740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac087f40 con 0x55c58d225170 2023-10-16T13:50:54.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 -- 172.21.15.60:0/1549414740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 msgr2=0x55c58d1f8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 0x55c58d1f8070 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0584a0 tx=0x7f69ac0585d0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 -- 172.21.15.60:0/1549414740 shutdown_connections 2023-10-16T13:50:54.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d1f85b0 0x55c58d228440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225550 0x55c58d227f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 --2- 172.21.15.60:0/1549414740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225170 0x55c58d1f8070 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.213+0000 7f69ba004740 1 -- 172.21.15.60:0/1549414740 >> 172.21.15.60:0/1549414740 conn(0x55c58d1fa380 msgr2=0x55c58d242f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.214+0000 7f69ba004740 1 -- 172.21.15.60:0/1549414740 shutdown_connections 2023-10-16T13:50:54.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.214+0000 7f69ba004740 1 -- 172.21.15.60:0/1549414740 wait complete. 2023-10-16T13:50:54.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.214+0000 7f69ba004740 1 -- start start 2023-10-16T13:50:54.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.215+0000 7f69ba004740 1 Processor -- start 2023-10-16T13:50:54.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d23d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d225550 0x55c58d23da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 0x55c58d216310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d163720 con 0x55c58d215e90 2023-10-16T13:50:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d101970 con 0x55c58d225170 2023-10-16T13:50:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c58d101610 con 0x55c58d225550 2023-10-16T13:50:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69ba004740 10 client.0 _ll_register_callbacks cb 0x55c58d1239d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:50:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69b5a0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 0x55c58d216310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69b5a0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 0x55c58d216310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53780/0 (socket says 172.21.15.60:53780) 2023-10-16T13:50:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69b5a0c640 1 -- 172.21.15.60:0/2690709249 learned_addr learned my addr 172.21.15.60:0/2690709249 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d225550 0x55c58d23da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.216+0000 7f69b520b640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d23d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69b5a0c640 1 -- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d225550 msgr2=0x55c58d23da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d225550 0x55c58d23da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69b5a0c640 1 -- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 msgr2=0x55c58d23d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d23d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69b5a0c640 1 -- 172.21.15.60:0/2690709249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b00cbd60 con 0x55c58d215e90 2023-10-16T13:50:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 0x55c58d216310 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f69a8037040 tx=0x7f69a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccbfaf154603bd94 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f698bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T13:50:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f698bfff640 1 -- 172.21.15.60:0/2690709249 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8034c70 con 0x55c58d215e90 2023-10-16T13:50:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f698bfff640 1 -- 172.21.15.60:0/2690709249 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69a8034dd0 con 0x55c58d215e90 2023-10-16T13:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.217+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c58d18a160 con 0x55c58d215e90 2023-10-16T13:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f698bfff640 1 -- 172.21.15.60:0/2690709249 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69a803e410 con 0x55c58d215e90 2023-10-16T13:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 10 client.9651 fetch_fsmap learned FSMap version 87 2023-10-16T13:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55c58d161da0 con 0x55c58d215e90 2023-10-16T13:50:54.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f698bfff640 1 -- 172.21.15.60:0/2690709249 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c58d18a160 con 0x55c58d215e90 2023-10-16T13:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f698bfff640 1 -- 172.21.15.60:0/2690709249 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 87) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69a803e570 con 0x55c58d215e90 2023-10-16T13:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 10 client.9651 fetch_fsmap finished waiting for FSMap version 87 2023-10-16T13:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 10 client.9651 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T13:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 20 client.9651 populate_metadata read hostname 'smithi060' 2023-10-16T13:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d22bdf0 0x55c58d23bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 4 client.9651 mds_command: new command op to 9510 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 --> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c58d135cc0 con 0x55c58d22bdf0 2023-10-16T13:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.218+0000 7f69b520b640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d22bdf0 0x55c58d23bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.219+0000 7f69b520b640 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d22bdf0 0x55c58d23bdd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad2e9d2ce9ea34d7 server_cookie=bc42f5b22f182b66 in_seq=0 out_seq=0 2023-10-16T13:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.219+0000 7f698bfff640 10 client.9651 ms_handle_connect on v2:172.21.15.60:6832/691868089 2023-10-16T13:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f698bfff640 1 -- 172.21.15.60:0/2690709249 <== mds.0 v2:172.21.15.60:6832/691868089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c58d135cc0 con 0x55c58d22bdf0 2023-10-16T13:50:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f698bfff640 10 client.9651 handle_command_reply: tid=0 2023-10-16T13:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 client.9651 shutdown 2023-10-16T13:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 msgr2=0x55c58d216310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 0x55c58d216310 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f69a8037040 tx=0x7f69a8004a40 comp rx=0 tx=0).stop 2023-10-16T13:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 shutdown_connections 2023-10-16T13:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d22bdf0 0x55c58d23bdd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d225550 0x55c58d23da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d215e90 0x55c58d216310 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 --2- 172.21.15.60:0/2690709249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d23d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 >> 172.21.15.60:0/2690709249 conn(0x55c58d1f85b0 msgr2=0x55c58d1fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f698bfff640 0 client.9651 ms_handle_reset on v2:172.21.15.60:6832/691868089 2023-10-16T13:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 shutdown_connections 2023-10-16T13:50:54.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 1 -- 172.21.15.60:0/2690709249 wait complete. 2023-10-16T13:50:54.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.220+0000 7f69ba004740 20 client.9651 trim_cache size 0 max 16384 2023-10-16T13:50:54.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.237+0000 7f69ba004740 1 Processor -- start 2023-10-16T13:50:54.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.237+0000 7f69ba004740 1 -- start start 2023-10-16T13:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d1f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69ba004740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d2440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69ba004740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58cec7f20 con 0x55c58d225550 2023-10-16T13:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d246910 con 0x55c58d225170 2023-10-16T13:50:54.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c58d265e00 con 0x55c58d215e90 2023-10-16T13:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.238+0000 7f69b5a0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d2440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d2440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53790/0 (socket says 172.21.15.60:53790) 2023-10-16T13:50:54.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3628648333 learned_addr learned my addr 172.21.15.60:0/3628648333 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:54.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d1f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b520b640 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 msgr2=0x55c58d2164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2164c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 msgr2=0x55c58d1f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d1f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3628648333 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c58d100b80 con 0x55c58d225550 2023-10-16T13:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d1f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d2440a0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f69a802bf00 tx=0x7f69a8041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504dd58e0ac8be04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.239+0000 7f69937fe640 1 -- 172.21.15.60:0/3628648333 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a803ebe0 con 0x55c58d225550 2023-10-16T13:50:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69937fe640 1 -- 172.21.15.60:0/3628648333 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a803e410 con 0x55c58d225550 2023-10-16T13:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69937fe640 1 -- 172.21.15.60:0/3628648333 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8034a70 con 0x55c58d225550 2023-10-16T13:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 -- 172.21.15.60:0/3628648333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 msgr2=0x55c58d2440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d2440a0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f69a802bf00 tx=0x7f69a8041000 comp rx=0 tx=0).stop 2023-10-16T13:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 -- 172.21.15.60:0/3628648333 shutdown_connections 2023-10-16T13:50:54.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d2440a0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d1f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 --2- 172.21.15.60:0/3628648333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 -- 172.21.15.60:0/3628648333 >> 172.21.15.60:0/3628648333 conn(0x55c58d1f85b0 msgr2=0x55c58d1fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 -- 172.21.15.60:0/3628648333 shutdown_connections 2023-10-16T13:50:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.240+0000 7f69ba004740 1 -- 172.21.15.60:0/3628648333 wait complete. 2023-10-16T13:50:54.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.241+0000 7f69ba004740 1 -- start start 2023-10-16T13:50:54.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.241+0000 7f69ba004740 1 Processor -- start 2023-10-16T13:50:54.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d228020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d228560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d2a5fb0 con 0x55c58d225550 2023-10-16T13:50:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58d215150 con 0x55c58d225170 2023-10-16T13:50:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c58d215640 con 0x55c58d215e90 2023-10-16T13:50:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.242+0000 7f69ba004740 10 client.0 _ll_register_callbacks cb 0x7f69b00741b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:50:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d228560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d228560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53802/0 (socket says 172.21.15.60:53802) 2023-10-16T13:50:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3268726217 learned_addr learned my addr 172.21.15.60:0/3268726217 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b4a0a640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d228020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b520b640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 msgr2=0x55c58d2440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 msgr2=0x55c58d228020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d228020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 -- 172.21.15.60:0/3268726217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c58d161da0 con 0x55c58d225550 2023-10-16T13:50:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b520b640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f69b5a0c640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d228560 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f69a802c4d0 tx=0x7f69a8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0ebf43b4258fa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.243+0000 7f698bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T13:50:54.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f698bfff640 1 -- 172.21.15.60:0/3268726217 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8034e80 con 0x55c58d225550 2023-10-16T13:50:54.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f698bfff640 1 -- 172.21.15.60:0/3268726217 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69a8003990 con 0x55c58d225550 2023-10-16T13:50:54.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69ac087dd0 con 0x55c58d225550 2023-10-16T13:50:54.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f698bfff640 1 -- 172.21.15.60:0/3268726217 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69a8003b50 con 0x55c58d225550 2023-10-16T13:50:54.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 10 client.9657 fetch_fsmap learned FSMap version 87 2023-10-16T13:50:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55c58d126170 con 0x55c58d225550 2023-10-16T13:50:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f698bfff640 1 -- 172.21.15.60:0/3268726217 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a8068020 con 0x55c58d225550 2023-10-16T13:50:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f698bfff640 1 -- 172.21.15.60:0/3268726217 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 87) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69a8003cb0 con 0x55c58d225550 2023-10-16T13:50:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 10 client.9657 fetch_fsmap finished waiting for FSMap version 87 2023-10-16T13:50:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 10 client.9657 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T13:50:54.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 20 client.9657 populate_metadata read hostname 'smithi060' 2023-10-16T13:50:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d2abaf0 0x55c58d2abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 4 client.9657 mds_command: new command op to 9510 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:50:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.244+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 --> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c58d15c700 con 0x55c58d2abaf0 2023-10-16T13:50:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.245+0000 7f69b520b640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d2abaf0 0x55c58d2abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.245+0000 7f69b520b640 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d2abaf0 0x55c58d2abed0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f02a2ead023c794 server_cookie=758ec3ed8988ebfb in_seq=0 out_seq=0 2023-10-16T13:50:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.245+0000 7f698bfff640 10 client.9657 ms_handle_connect on v2:172.21.15.60:6832/691868089 2023-10-16T13:50:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f698bfff640 1 -- 172.21.15.60:0/3268726217 <== mds.0 v2:172.21.15.60:6832/691868089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55c58d15c700 con 0x55c58d2abaf0 2023-10-16T13:50:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f698bfff640 10 client.9657 handle_command_reply: tid=0 2023-10-16T13:50:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 client.9657 shutdown 2023-10-16T13:50:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 msgr2=0x55c58d228560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d228560 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f69a802c4d0 tx=0x7f69a8030000 comp rx=0 tx=0).stop 2023-10-16T13:50:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 shutdown_connections 2023-10-16T13:50:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:6832/691868089,v1:172.21.15.60:6834/691868089] conn(0x55c58d2abaf0 0x55c58d2abed0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c58d225550 0x55c58d228560 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55c58d225170 0x55c58d228020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 --2- 172.21.15.60:0/3268726217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55c58d215e90 0x55c58d2440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f698bfff640 0 client.9657 ms_handle_reset on v2:172.21.15.60:6832/691868089 2023-10-16T13:50:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 >> 172.21.15.60:0/3268726217 conn(0x55c58d1f85b0 msgr2=0x55c58d25d080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 shutdown_connections 2023-10-16T13:50:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 1 -- 172.21.15.60:0/3268726217 wait complete. 2023-10-16T13:50:54.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.246+0000 7f69ba004740 20 client.9657 trim_cache size 0 max 16384 2023-10-16T13:50:54.262 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T13:50:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T13:50:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T13:50:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 9510, 2023-10-16T13:50:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T13:50:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T13:50:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T13:50:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8416972229999997, 2023-10-16T13:50:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 87, 2023-10-16T13:50:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 183, 2023-10-16T13:50:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 183, 2023-10-16T13:50:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.165355440000001 2023-10-16T13:50:54.264 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T13:50:54.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f699406af90 msgr2=0x7f699406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f699406af90 0x7f699406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69b00d3560 tx=0x7f69a0009040 comp rx=0 tx=0).stop 2023-10-16T13:50:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 msgr2=0x7f69b015a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 0x7f69b015a300 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00e870 tx=0x7f69ac00ed30 comp rx=0 tx=0).stop 2023-10-16T13:50:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 shutdown_connections 2023-10-16T13:50:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f699406af90 0x7f699406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69b00cded0 0x7f69b00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69b015cd70 0x7f69b015f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 --2- 172.21.15.60:0/1850643475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69b00b8480 0x7f69b015a300 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 >> 172.21.15.60:0/1850643475 conn(0x7f69b00cf570 msgr2=0x7f69b00b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 shutdown_connections 2023-10-16T13:50:54.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.247+0000 7f69b620d640 1 -- 172.21.15.60:0/1850643475 wait complete. 2023-10-16T13:50:54.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:50:54.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.493+0000 7f7aab27c640 1 Processor -- start 2023-10-16T13:50:54.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.493+0000 7f7aab27c640 1 -- start start 2023-10-16T13:50:54.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.495+0000 7f7aab27c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7aa415ae80 0x7f7aa415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.495+0000 7f7aab27c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa415ed60 con 0x7f7aa40ce390 2023-10-16T13:50:54.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.495+0000 7f7aab27c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa40707e0 con 0x7f7aa415aaa0 2023-10-16T13:50:54.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.495+0000 7f7aab27c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa406b810 con 0x7f7aa415ae80 2023-10-16T13:50:54.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.495+0000 7f7aa9819640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7aa40ce390 0x7f7aa4070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39740/0 (socket says 172.21.15.60:39740) 2023-10-16T13:50:54.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.495+0000 7f7aa9819640 1 -- 172.21.15.60:0/1659403387 learned_addr learned my addr 172.21.15.60:0/1659403387 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:54.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7aa8817640 1 --2- 172.21.15.60:0/1659403387 >> v2:172.21.15.144:3300/0 conn(0x7f7aa415ae80 0x7f7aa415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127166614 0 0) 0x7f7aa415ed60 con 0x7f7aa40ce390 2023-10-16T13:50:54.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a90003520 con 0x7f7aa40ce390 2023-10-16T13:50:54.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674580756 0 0) 0x7f7a90003520 con 0x7f7aa40ce390 2023-10-16T13:50:54.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 >> v2:172.21.15.144:3300/0 conn(0x7f7aa415ae80 msgr2=0x7f7aa415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 --2- 172.21.15.60:0/1659403387 >> v2:172.21.15.144:3300/0 conn(0x7f7aa415ae80 0x7f7aa415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 >> v1:172.21.15.144:6789/0 conn(0x7f7aa415aaa0 legacy=0x7f7aa40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa415ef90 con 0x7f7aa40ce390 2023-10-16T13:50:54.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7a9c0019f0 con 0x7f7aa40ce390 2023-10-16T13:50:54.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 >> v1:172.21.15.60:6789/0 conn(0x7f7aa40ce390 legacy=0x7f7aa4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a900039c0 0x7f7a90003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7a90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.496+0000 7f7a9bfff640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a90005180 0x7f7a9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa406b810 con 0x7f7a900047c0 2023-10-16T13:50:54.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9000c080 con 0x7f7a900039c0 2023-10-16T13:50:54.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9000c240 con 0x7f7a90005180 2023-10-16T13:50:54.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa8817640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7a90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa9819640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a90005180 0x7f7a9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa9018640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a900039c0 0x7f7a90003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa8817640 1 -- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a90005180 msgr2=0x7f7a9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa8817640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a90005180 0x7f7a9000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa8817640 1 -- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a900039c0 msgr2=0x7f7a90003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:54.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa8817640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a900039c0 0x7f7a90003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa8817640 1 -- 172.21.15.60:0/1659403387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90003520 con 0x7f7a900047c0 2023-10-16T13:50:54.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa9819640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a90005180 0x7f7a9000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:54.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.497+0000 7f7aa9819640 1 -- 172.21.15.60:0/1659403387 reap_dead start 2023-10-16T13:50:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7aa8817640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7a90004c40 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009a90 tx=0x7f7a900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c05d5f1163f4ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402c680 con 0x7f7a900047c0 2023-10-16T13:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9402c7e0 con 0x7f7a900047c0 2023-10-16T13:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7a9bfff640 1 -- 172.21.15.60:0/1659403387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402cb70 con 0x7f7a900047c0 2023-10-16T13:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7aab27c640 1 -- 172.21.15.60:0/1659403387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 msgr2=0x7f7a90004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7aab27c640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7a90004c40 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009a90 tx=0x7f7a900057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7aab27c640 1 -- 172.21.15.60:0/1659403387 shutdown_connections 2023-10-16T13:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7aab27c640 1 --2- 172.21.15.60:0/1659403387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7a90004c40 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.499+0000 7f7aab27c640 1 -- 172.21.15.60:0/1659403387 >> 172.21.15.60:0/1659403387 conn(0x7f7aa40cb410 msgr2=0x7f7aa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.500+0000 7f7aab27c640 1 -- 172.21.15.60:0/1659403387 shutdown_connections 2023-10-16T13:50:54.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.500+0000 7f7aab27c640 1 -- 172.21.15.60:0/1659403387 wait complete. 2023-10-16T13:50:54.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.501+0000 7f7aab27c640 1 Processor -- start 2023-10-16T13:50:54.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.502+0000 7f7aab27c640 1 -- start start 2023-10-16T13:50:54.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aab27c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7aa41eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aab27c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7aa415b6e0 0x7f7aa40c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aab27c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7aa41f32a0 0x7f7aa40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aab27c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c73b0 con 0x7f7a900047c0 2023-10-16T13:50:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aab27c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c7690 con 0x7f7aa41f32a0 2023-10-16T13:50:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aab27c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa41f2aa0 con 0x7f7aa415b6e0 2023-10-16T13:50:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7aa41eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7aa41eff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53806/0 (socket says 172.21.15.60:53806) 2023-10-16T13:50:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 -- 172.21.15.60:0/3476450570 learned_addr learned my addr 172.21.15.60:0/3476450570 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 -- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7aa415b6e0 msgr2=0x7f7aa40c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7aa415b6e0 0x7f7aa40c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 -- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7aa41f32a0 msgr2=0x7f7aa40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7aa41f32a0 0x7f7aa40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.503+0000 7f7aa9018640 1 -- 172.21.15.60:0/3476450570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa41f2d80 con 0x7f7a900047c0 2023-10-16T13:50:54.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.504+0000 7f7aa9018640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7aa41eff50 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e490 tx=0x7f7aa000e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.504+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00052e0 con 0x7f7a900047c0 2023-10-16T13:50:54.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.504+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0005440 con 0x7f7a900047c0 2023-10-16T13:50:54.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.504+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0003830 con 0x7f7a900047c0 2023-10-16T13:50:54.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.504+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40c7970 con 0x7f7a900047c0 2023-10-16T13:50:54.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.504+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa41f3720 con 0x7f7a900047c0 2023-10-16T13:50:54.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.505+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa41f2f10 con 0x7f7a900047c0 2023-10-16T13:50:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.506+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7aa0003990 con 0x7f7a900047c0 2023-10-16T13:50:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.507+0000 7f7a99ffb640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f7a8806afc0 0x7f7a8806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.507+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7aa00894d0 con 0x7f7a900047c0 2023-10-16T13:50:54.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.508+0000 7f7aa8817640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f7a8806afc0 0x7f7a8806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:54.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.510+0000 7f7aa8817640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f7a8806afc0 0x7f7a8806d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a94036000 tx=0x7f7a9402e000 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:54.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.512+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7aa0054470 con 0x7f7a900047c0 2023-10-16T13:50:54.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.745+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7aa404e160 con 0x7f7a900047c0 2023-10-16T13:50:54.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.746+0000 7f7a99ffb640 1 -- 172.21.15.60:0/3476450570 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7aa0057a90 con 0x7f7a900047c0 2023-10-16T13:50:54.749 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:50:54.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.748+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f7a8806afc0 msgr2=0x7f7a8806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.748+0000 7f7aab27c640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f7a8806afc0 0x7f7a8806d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a94036000 tx=0x7f7a9402e000 comp rx=0 tx=0).stop 2023-10-16T13:50:54.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.748+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 msgr2=0x7f7aa41eff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:54.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.748+0000 7f7aab27c640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7aa41eff50 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e490 tx=0x7f7aa000e950 comp rx=0 tx=0).stop 2023-10-16T13:50:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 shutdown_connections 2023-10-16T13:50:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f7a8806afc0 0x7f7a8806d480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7aa415b6e0 0x7f7aa40c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7aa41f32a0 0x7f7aa40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 --2- 172.21.15.60:0/3476450570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a900047c0 0x7f7aa41eff50 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 >> 172.21.15.60:0/3476450570 conn(0x7f7aa40cb410 msgr2=0x7f7aa406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:54.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 shutdown_connections 2023-10-16T13:50:54.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:54.749+0000 7f7aab27c640 1 -- 172.21.15.60:0/3476450570 wait complete. 2023-10-16T13:50:54.771 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:50:54.771 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:50:54.772 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T13:50:54.772 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:50:54.772 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:50:54.773 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:50:54.773 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:50:54.773 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:50:54.774 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:50:54.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:54.775 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T13:50:54.827 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:50:54.827 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:50:54.828 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:50:54.828 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.828 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T13:50:54.828 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.828 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:50:54.829 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T13:50:54.829 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T13:50:54.829 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:50:54.829 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T13:50:54.829 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T13:50:54.829 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:50:54.830 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:50:54.830 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T13:50:54.830 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:50:54.830 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 37732sec preferred_lft 37732sec 2023-10-16T13:50:54.831 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T13:50:54.831 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.831 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:50:54.831 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:50:54.831 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T13:50:54.831 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:50:54.832 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T13:50:54.832 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:50:54.832 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.832 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T13:50:54.832 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.832 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:50:54.833 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:50:54.833 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T13:50:54.833 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.833 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:50:54.833 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:50:54.833 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T13:50:54.834 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:50:54.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:54.835 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:50:54.932 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T13:50:54.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:54.937 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T13:50:54.998 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T13:50:54.998 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:50:54.998 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:50:54.999 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:50:54.999 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:50:54.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:55.000 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T13:50:55.000 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T13:50:55.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:50:55.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:50:55.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:50:55.001 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T13:50:55.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:55.211 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T13:50:55.266 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:50:55.267 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:50:55.267 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:50:55.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:55.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T13:50:55.375 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:50:55.375 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:50:55.439 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:50:55.439 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:50:55.440 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:50:55.494 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T13:50:55.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:55.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T13:50:55.641 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T13:50:55.642 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T13:50:55.673 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T13:50:55.674 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T13:50:55.674 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T13:50:55.674 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T13:50:55.674 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T13:50:55.676 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T13:50:55.676 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T13:50:55.676 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T13:50:55.676 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T13:50:55.676 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T13:50:55.679 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:50:55.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:55.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:50:55.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:55.771 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:50:55.844 INFO:teuthology.orchestra.run.smithi060.stdout:client9666 2023-10-16T13:50:55.844 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:50:55.844 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:50:55.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:50:56.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.127+0000 7f336f626640 1 Processor -- start 2023-10-16T13:50:56.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.127+0000 7f336f626640 1 -- start start 2023-10-16T13:50:56.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.128+0000 7f336f626640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f336815ad70 0x7f3368069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.128+0000 7f336f626640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33680caac0 0x7f336806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.128+0000 7f336f626640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33680cb030 con 0x7f33680c9ca0 2023-10-16T13:50:56.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.128+0000 7f336f626640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33680c91c0 con 0x7f336815ad70 2023-10-16T13:50:56.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.128+0000 7f336f626640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33680c94a0 con 0x7f33680caac0 2023-10-16T13:50:56.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.128+0000 7f336d3c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f336815ad70 0x7f3368069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.129+0000 7f336d3c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f336815ad70 0x7f3368069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53822/0 (socket says 172.21.15.60:53822) 2023-10-16T13:50:56.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.129+0000 7f336d3c2640 1 -- 172.21.15.60:0/3021064391 learned_addr learned my addr 172.21.15.60:0/3021064391 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:56.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.129+0000 7f336d3c2640 1 -- 172.21.15.60:0/3021064391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33680caac0 msgr2=0x7f336806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:56.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.129+0000 7f336d3c2640 1 --2- 172.21.15.60:0/3021064391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33680caac0 0x7f336806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.129+0000 7f336d3c2640 1 -- 172.21.15.60:0/3021064391 >> v1:172.21.15.144:6789/0 conn(0x7f33680c9ca0 legacy=0x7f33680ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:56.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.129+0000 7f336d3c2640 1 -- 172.21.15.60:0/3021064391 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680c9780 con 0x7f336815ad70 2023-10-16T13:50:56.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.131+0000 7f336d3c2640 1 --2- 172.21.15.60:0/3021064391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336815ad70 0x7f3368069da0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f3364009870 tx=0x7f336402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551ef9457483fc80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:56.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.131+0000 7f335ffff640 1 -- 172.21.15.60:0/3021064391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364031640 con 0x7f336815ad70 2023-10-16T13:50:56.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f335ffff640 1 -- 172.21.15.60:0/3021064391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33640317a0 con 0x7f336815ad70 2023-10-16T13:50:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f335ffff640 1 -- 172.21.15.60:0/3021064391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364031b30 con 0x7f336815ad70 2023-10-16T13:50:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 -- 172.21.15.60:0/3021064391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336815ad70 msgr2=0x7f3368069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 --2- 172.21.15.60:0/3021064391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336815ad70 0x7f3368069da0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f3364009870 tx=0x7f336402b8f0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 -- 172.21.15.60:0/3021064391 shutdown_connections 2023-10-16T13:50:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 --2- 172.21.15.60:0/3021064391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33680caac0 0x7f336806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 --2- 172.21.15.60:0/3021064391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f336815ad70 0x7f3368069da0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 -- 172.21.15.60:0/3021064391 >> 172.21.15.60:0/3021064391 conn(0x7f33680ce9b0 msgr2=0x7f33680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:56.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 -- 172.21.15.60:0/3021064391 shutdown_connections 2023-10-16T13:50:56.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.132+0000 7f336f626640 1 -- 172.21.15.60:0/3021064391 wait complete. 2023-10-16T13:50:56.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.133+0000 7f336f626640 1 Processor -- start 2023-10-16T13:50:56.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.133+0000 7f336f626640 1 -- start start 2023-10-16T13:50:56.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.134+0000 7f336f626640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33680c9ca0 0x7f33681f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.134+0000 7f336f626640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 0x7f33681f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.134+0000 7f336f626640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f336815ad70 0x7f3368151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.134+0000 7f336f626640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33681f66e0 con 0x7f33680caac0 2023-10-16T13:50:56.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.134+0000 7f336f626640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33680c5280 con 0x7f33680c9ca0 2023-10-16T13:50:56.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.134+0000 7f336f626640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33680c4fa0 con 0x7f336815ad70 2023-10-16T13:50:56.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 0x7f33681f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 0x7f33681f61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53836/0 (socket says 172.21.15.60:53836) 2023-10-16T13:50:56.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 -- 172.21.15.60:0/2147193538 learned_addr learned my addr 172.21.15.60:0/2147193538 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:56.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336d3c2640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33680c9ca0 0x7f33681f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336dbc3640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f336815ad70 0x7f3368151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 -- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f336815ad70 msgr2=0x7f3368151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f336815ad70 0x7f3368151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 -- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33680c9ca0 msgr2=0x7f33681f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33680c9ca0 0x7f33681f2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.135+0000 7f336cbc1640 1 -- 172.21.15.60:0/2147193538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640094d0 con 0x7f33680caac0 2023-10-16T13:50:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.136+0000 7f336cbc1640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 0x7f33681f61a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f335800c410 tx=0x7f335800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.136+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33580035f0 con 0x7f33680caac0 2023-10-16T13:50:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.136+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3358003750 con 0x7f33680caac0 2023-10-16T13:50:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.136+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358003b40 con 0x7f33680caac0 2023-10-16T13:50:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.136+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33681f22b0 con 0x7f33680caac0 2023-10-16T13:50:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.136+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33681f2870 con 0x7f33680caac0 2023-10-16T13:50:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.137+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33681f2a60 con 0x7f33680caac0 2023-10-16T13:50:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.140+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3358003ca0 con 0x7f33680caac0 2023-10-16T13:50:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.140+0000 7f335dffb640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f334c06af90 0x7f334c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.140+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3358088c20 con 0x7f33680caac0 2023-10-16T13:50:56.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.141+0000 7f336d3c2640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f334c06af90 0x7f334c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.142+0000 7f336d3c2640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f334c06af90 0x7f334c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3364036000 tx=0x7f336403c040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:56.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.143+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3358088e60 con 0x7f33680caac0 2023-10-16T13:50:56.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.366+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 --> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f33680ad4d0 con 0x7f334c06af90 2023-10-16T13:50:56.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.370+0000 7f335dffb640 1 -- 172.21.15.60:0/2147193538 <== mgr.7588 v2:172.21.15.144:6832/2817709727 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f33640315e0 con 0x7f334c06af90 2023-10-16T13:50:56.373 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T13:50:56.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:50:56.373 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T13:50:56.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:50:56.374 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T13:50:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f334c06af90 msgr2=0x7f334c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f334c06af90 0x7f334c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3364036000 tx=0x7f336403c040 comp rx=0 tx=0).stop 2023-10-16T13:50:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 msgr2=0x7f33681f61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 0x7f33681f61a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f335800c410 tx=0x7f335800c8d0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 shutdown_connections 2023-10-16T13:50:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f334c06af90 0x7f334c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f336815ad70 0x7f3368151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33680caac0 0x7f33681f61a0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 --2- 172.21.15.60:0/2147193538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33680c9ca0 0x7f33681f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 >> 172.21.15.60:0/2147193538 conn(0x7f33680ce9b0 msgr2=0x7f33680ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 shutdown_connections 2023-10-16T13:50:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.372+0000 7f336f626640 1 -- 172.21.15.60:0/2147193538 wait complete. 2023-10-16T13:50:56.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:50:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.624+0000 7f6fedee7640 1 Processor -- start 2023-10-16T13:50:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.624+0000 7f6fedee7640 1 -- start start 2023-10-16T13:50:56.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.625+0000 7f6fedee7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fe80caa80 0x7f6fe8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.625+0000 7f6fedee7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe806a2e0 0x7f6fe81650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.625+0000 7f6fedee7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe806a730 con 0x7f6fe80addb0 2023-10-16T13:50:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.625+0000 7f6fedee7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe8161a50 con 0x7f6fe80caa80 2023-10-16T13:50:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.625+0000 7f6fedee7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80caef0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.626+0000 7f6fe6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fe80caa80 0x7f6fe8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.626+0000 7f6fe7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe806a2e0 0x7f6fe81650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.626+0000 7f6fe6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fe80caa80 0x7f6fe8069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:47790/0 (socket says 172.21.15.60:47790) 2023-10-16T13:50:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.626+0000 7f6fe6ffd640 1 -- 172.21.15.60:0/3696841946 learned_addr learned my addr 172.21.15.60:0/3696841946 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.627+0000 7f6fe6ffd640 1 -- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe806a2e0 msgr2=0x7f6fe81650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.627+0000 7f6fe6ffd640 1 --2- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe806a2e0 0x7f6fe81650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.627+0000 7f6fe6ffd640 1 -- 172.21.15.60:0/3696841946 >> v1:172.21.15.144:6789/0 conn(0x7f6fe80addb0 legacy=0x7f6fe80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.627+0000 7f6fe6ffd640 1 -- 172.21.15.60:0/3696841946 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe80cb140 con 0x7f6fe80caa80 2023-10-16T13:50:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.628+0000 7f6fe6ffd640 1 --2- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe8069da0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009870 tx=0x7f6fd002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3762acaf81198aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.629+0000 7f6fe67fc640 1 -- 172.21.15.60:0/3696841946 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0030840 con 0x7f6fe80caa80 2023-10-16T13:50:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.629+0000 7f6fe67fc640 1 -- 172.21.15.60:0/3696841946 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd00309a0 con 0x7f6fe80caa80 2023-10-16T13:50:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.629+0000 7f6fe67fc640 1 -- 172.21.15.60:0/3696841946 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0030d30 con 0x7f6fe80caa80 2023-10-16T13:50:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 -- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 msgr2=0x7f6fe8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 --2- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe8069da0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009870 tx=0x7f6fd002b980 comp rx=0 tx=0).stop 2023-10-16T13:50:56.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 -- 172.21.15.60:0/3696841946 shutdown_connections 2023-10-16T13:50:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 --2- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe806a2e0 0x7f6fe81650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 --2- 172.21.15.60:0/3696841946 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe8069da0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 -- 172.21.15.60:0/3696841946 >> 172.21.15.60:0/3696841946 conn(0x7f6fe80ce9b0 msgr2=0x7f6fe80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 -- 172.21.15.60:0/3696841946 shutdown_connections 2023-10-16T13:50:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.631+0000 7f6fedee7640 1 -- 172.21.15.60:0/3696841946 wait complete. 2023-10-16T13:50:56.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.632+0000 7f6fedee7640 1 Processor -- start 2023-10-16T13:50:56.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.632+0000 7f6fedee7640 1 -- start start 2023-10-16T13:50:56.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fedee7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 0x7f6fe80b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fedee7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80addb0 0x7f6fe81f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fedee7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe80b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fedee7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80b24b0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fedee7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80c5b20 con 0x7f6fe80caa80 2023-10-16T13:50:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fedee7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80c5dc0 con 0x7f6fe80addb0 2023-10-16T13:50:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.633+0000 7f6fe77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 0x7f6fe80b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 0x7f6fe80b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53846/0 (socket says 172.21.15.60:53846) 2023-10-16T13:50:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 -- 172.21.15.60:0/954830640 learned_addr learned my addr 172.21.15.60:0/954830640 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe6ffd640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80addb0 0x7f6fe81f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe7fff640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe80b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 -- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80addb0 msgr2=0x7f6fe81f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80addb0 0x7f6fe81f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 -- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 msgr2=0x7f6fe80b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe80b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:56.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 -- 172.21.15.60:0/954830640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00094d0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe6ffd640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80addb0 0x7f6fe81f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:56.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.634+0000 7f6fe77fe640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 0x7f6fe80b1f70 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800c870 tx=0x7f6fd800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:56.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.635+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8003b70 con 0x7f6fe806a2e0 2023-10-16T13:50:56.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.635+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd8003cf0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.635+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe80c60a0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.635+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd800aa90 con 0x7f6fe806a2e0 2023-10-16T13:50:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.635+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe80b26e0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.636+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe80b2870 con 0x7f6fe806a2e0 2023-10-16T13:50:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.637+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 88) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6fd800abf0 con 0x7f6fe806a2e0 2023-10-16T13:50:56.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.638+0000 7f6fdffff640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6fc806af90 0x7f6fc806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:56.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.638+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fd8089590 con 0x7f6fe806a2e0 2023-10-16T13:50:56.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.639+0000 7f6fe6ffd640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6fc806af90 0x7f6fc806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:56.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.640+0000 7f6fe6ffd640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6fc806af90 0x7f6fc806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0036000 tx=0x7f6fd003b040 comp rx=0 tx=0).ready entity=mgr.7588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:56.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.643+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fd8054490 con 0x7f6fe806a2e0 2023-10-16T13:50:56.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:56.892+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6fe806a730 con 0x7f6fe806a2e0 2023-10-16T13:50:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.493+0000 7f6fdffff640 1 -- 172.21.15.60:0/954830640 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v89) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6fe806a730 con 0x7f6fe806a2e0 2023-10-16T13:50:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.497+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6fc806af90 msgr2=0x7f6fc806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.497+0000 7f6fedee7640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6fc806af90 0x7f6fc806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0036000 tx=0x7f6fd003b040 comp rx=0 tx=0).stop 2023-10-16T13:50:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.497+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 msgr2=0x7f6fe80b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.497+0000 7f6fedee7640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 0x7f6fe80b1f70 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800c870 tx=0x7f6fd800cd30 comp rx=0 tx=0).stop 2023-10-16T13:50:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 shutdown_connections 2023-10-16T13:50:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f6fc806af90 0x7f6fc806d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fe80addb0 0x7f6fe81f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fe80caa80 0x7f6fe80b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 --2- 172.21.15.60:0/954830640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fe806a2e0 0x7f6fe80b1f70 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 >> 172.21.15.60:0/954830640 conn(0x7f6fe80ce9b0 msgr2=0x7f6fe80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 shutdown_connections 2023-10-16T13:50:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.498+0000 7f6fedee7640 1 -- 172.21.15.60:0/954830640 wait complete. 2023-10-16T13:50:57.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:50:57.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.736+0000 7f1924f78640 1 Processor -- start 2023-10-16T13:50:57.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.736+0000 7f1924f78640 1 -- start start 2023-10-16T13:50:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.737+0000 7f1924f78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d030 0x7f19200d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.737+0000 7f1924f78640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19200b5d90 con 0x7f192015d410 2023-10-16T13:50:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.738+0000 7f1924f78640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1920160fc0 con 0x7f19200cd260 2023-10-16T13:50:57.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.738+0000 7f1924f78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19200d2b50 con 0x7f192015d030 2023-10-16T13:50:57.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.738+0000 7f1917fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f192015d410 0x7f1920161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39748/0 (socket says 172.21.15.60:39748) 2023-10-16T13:50:57.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.738+0000 7f1917fff640 1 -- 172.21.15.60:0/1540161889 learned_addr learned my addr 172.21.15.60:0/1540161889 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:57.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.738+0000 7f191e59c640 1 --2- 172.21.15.60:0/1540161889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d030 0x7f19200d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:57.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241507840 0 0) 0x7f19200b5d90 con 0x7f192015d410 2023-10-16T13:50:57.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f190c003520 con 0x7f192015d410 2023-10-16T13:50:57.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616731255 0 0) 0x7f1920160fc0 con 0x7f19200cd260 2023-10-16T13:50:57.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19200b5d90 con 0x7f19200cd260 2023-10-16T13:50:57.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 897923955 0 0) 0x7f190c003520 con 0x7f192015d410 2023-10-16T13:50:57.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191e59c640 1 -- 172.21.15.60:0/1540161889 >> v1:172.21.15.144:6789/0 conn(0x7f19200cd260 legacy=0x7f19200b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191e59c640 1 -- 172.21.15.60:0/1540161889 >> v1:172.21.15.60:6789/0 conn(0x7f192015d410 legacy=0x7f1920161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.739+0000 7f191e59c640 1 -- 172.21.15.60:0/1540161889 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19200d25d0 con 0x7f192015d030 2023-10-16T13:50:57.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.740+0000 7f191e59c640 1 --2- 172.21.15.60:0/1540161889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d030 0x7f19200d1630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f19100097d0 tx=0x7f191002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e676436322c42b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:57.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.741+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908001670 con 0x7f192015d030 2023-10-16T13:50:57.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.741+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910031580 con 0x7f192015d030 2023-10-16T13:50:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.741+0000 7f191dd9b640 1 -- 172.21.15.60:0/1540161889 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910031910 con 0x7f192015d030 2023-10-16T13:50:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 -- 172.21.15.60:0/1540161889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d030 msgr2=0x7f19200d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 --2- 172.21.15.60:0/1540161889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d030 0x7f19200d1630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f19100097d0 tx=0x7f191002b8a0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 -- 172.21.15.60:0/1540161889 shutdown_connections 2023-10-16T13:50:57.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 --2- 172.21.15.60:0/1540161889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d030 0x7f19200d1630 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 -- 172.21.15.60:0/1540161889 >> 172.21.15.60:0/1540161889 conn(0x7f19200cf720 msgr2=0x7f19200cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:57.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 -- 172.21.15.60:0/1540161889 shutdown_connections 2023-10-16T13:50:57.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.742+0000 7f1924f78640 1 -- 172.21.15.60:0/1540161889 wait complete. 2023-10-16T13:50:57.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.745+0000 7f1924f78640 1 Processor -- start 2023-10-16T13:50:57.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.745+0000 7f1924f78640 1 -- start start 2023-10-16T13:50:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1924f78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 0x7f1920159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1924f78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015d030 0x7f19200ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1924f78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d410 0x7f19200ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1924f78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1920158930 con 0x7f19200cd260 2023-10-16T13:50:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1924f78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1920150ef0 con 0x7f192015d030 2023-10-16T13:50:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1924f78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19201511d0 con 0x7f192015d410 2023-10-16T13:50:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f191e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 0x7f1920159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f191e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 0x7f1920159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53872/0 (socket says 172.21.15.60:53872) 2023-10-16T13:50:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f191e59c640 1 -- 172.21.15.60:0/3819362932 learned_addr learned my addr 172.21.15.60:0/3819362932 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.747+0000 7f1917fff640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015d030 0x7f19200ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191ed9d640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d410 0x7f19200ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191e59c640 1 -- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d410 msgr2=0x7f19200ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191e59c640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d410 0x7f19200ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191e59c640 1 -- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015d030 msgr2=0x7f19200ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191e59c640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015d030 0x7f19200ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191e59c640 1 -- 172.21.15.60:0/3819362932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1910009400 con 0x7f19200cd260 2023-10-16T13:50:57.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.748+0000 7f191e59c640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 0x7f1920159d40 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1910034040 tx=0x7f19100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:57.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.749+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910004220 con 0x7f19200cd260 2023-10-16T13:50:57.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.749+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910004380 con 0x7f19200cd260 2023-10-16T13:50:57.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.749+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19201514b0 con 0x7f19200cd260 2023-10-16T13:50:57.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.749+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910003850 con 0x7f19200cd260 2023-10-16T13:50:57.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.749+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192015a280 con 0x7f19200cd260 2023-10-16T13:50:57.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.750+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192004e160 con 0x7f19200cd260 2023-10-16T13:50:57.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.751+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 89) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1910043070 con 0x7f19200cd260 2023-10-16T13:50:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.752+0000 7f1916ffd640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 0x7f190006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.752+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19100ab600 con 0x7f19200cd260 2023-10-16T13:50:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.752+0000 7f1917fff640 1 -- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 msgr2=0x7f190006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:57.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.752+0000 7f1917fff640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 0x7f190006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.756+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19100765a0 con 0x7f19200cd260 2023-10-16T13:50:57.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.952+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f19200b8360 con 0x7f19200cd260 2023-10-16T13:50:57.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.953+0000 7f1917fff640 1 -- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 msgr2=0x7f190006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:57.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.953+0000 7f1917fff640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 0x7f190006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:57.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.959+0000 7f1916ffd640 1 -- 172.21.15.60:0/3819362932 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1910079bc0 con 0x7f19200cd260 2023-10-16T13:50:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.962+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 msgr2=0x7f190006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.962+0000 7f1924f78640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 0x7f190006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.962+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 msgr2=0x7f1920159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:57.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.962+0000 7f1924f78640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 0x7f1920159d40 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1910034040 tx=0x7f19100046c0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 shutdown_connections 2023-10-16T13:50:57.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f190006afe0 0x7f190006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015d410 0x7f19200ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015d030 0x7f19200ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 --2- 172.21.15.60:0/3819362932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19200cd260 0x7f1920159d40 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 >> 172.21.15.60:0/3819362932 conn(0x7f19200cf720 msgr2=0x7f192015c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 shutdown_connections 2023-10-16T13:50:57.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:57.963+0000 7f1924f78640 1 -- 172.21.15.60:0/3819362932 wait complete. 2023-10-16T13:50:57.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:50:58.028 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.025+0000 7f45be00f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:50:58.045 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.043+0000 7fb319837040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:50:58.081 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.079+0000 7f45be00f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:50:58.103 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.101+0000 7fb319837040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:50:58.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.200+0000 7f00fffff640 1 Processor -- start 2023-10-16T13:50:58.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.200+0000 7f00fffff640 1 -- start start 2023-10-16T13:50:58.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00fffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f010006c040 0x7f010006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00fffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f01000ad970 0x7f010015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00fffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f010015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00fffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010006c960 con 0x7f010006c040 2023-10-16T13:50:58.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00fffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010015b9d0 con 0x7f01000ad970 2023-10-16T13:50:58.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00fffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010015bcb0 con 0x7f010015ebc0 2023-10-16T13:50:58.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f010006c040 0x7f010006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f010006c040 0x7f010006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53874/0 (socket says 172.21.15.60:53874) 2023-10-16T13:50:58.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.202+0000 7f00feffd640 1 -- 172.21.15.60:0/3508594579 learned_addr learned my addr 172.21.15.60:0/3508594579 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00f7fff640 1 --2- 172.21.15.60:0/3508594579 >> v2:172.21.15.144:3300/0 conn(0x7f01000ad970 0x7f010015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/3508594579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f010015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00feffd640 1 -- 172.21.15.60:0/3508594579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 msgr2=0x7f010015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00feffd640 1 --2- 172.21.15.60:0/3508594579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f010015f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00feffd640 1 -- 172.21.15.60:0/3508594579 >> v2:172.21.15.144:3300/0 conn(0x7f01000ad970 msgr2=0x7f010015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:58.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00feffd640 1 --2- 172.21.15.60:0/3508594579 >> v2:172.21.15.144:3300/0 conn(0x7f01000ad970 0x7f010015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.203+0000 7f00feffd640 1 -- 172.21.15.60:0/3508594579 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010015bf90 con 0x7f010006c040 2023-10-16T13:50:58.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.204+0000 7f00feffd640 1 --2- 172.21.15.60:0/3508594579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f010006c420 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f01000cb060 tx=0x7f00f002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987ffc15930046af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:58.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.204+0000 7f00fe7fc640 1 -- 172.21.15.60:0/3508594579 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0031860 con 0x7f010006c040 2023-10-16T13:50:58.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.204+0000 7f00fe7fc640 1 -- 172.21.15.60:0/3508594579 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f00319c0 con 0x7f010006c040 2023-10-16T13:50:58.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.204+0000 7f00fe7fc640 1 -- 172.21.15.60:0/3508594579 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0031d50 con 0x7f010006c040 2023-10-16T13:50:58.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 -- 172.21.15.60:0/3508594579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 msgr2=0x7f010006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 --2- 172.21.15.60:0/3508594579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f010006c420 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f01000cb060 tx=0x7f00f002b900 comp rx=0 tx=0).stop 2023-10-16T13:50:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 -- 172.21.15.60:0/3508594579 shutdown_connections 2023-10-16T13:50:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 --2- 172.21.15.60:0/3508594579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f010015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 --2- 172.21.15.60:0/3508594579 >> v2:172.21.15.144:3300/0 conn(0x7f01000ad970 0x7f010015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 --2- 172.21.15.60:0/3508594579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f010006c420 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 -- 172.21.15.60:0/3508594579 >> 172.21.15.60:0/3508594579 conn(0x7f01000ce6a0 msgr2=0x7f01000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 -- 172.21.15.60:0/3508594579 shutdown_connections 2023-10-16T13:50:58.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.205+0000 7f00fffff640 1 -- 172.21.15.60:0/3508594579 wait complete. 2023-10-16T13:50:58.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.207+0000 7f00fffff640 1 Processor -- start 2023-10-16T13:50:58.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.207+0000 7f00fffff640 1 -- start start 2023-10-16T13:50:58.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f01001efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00fffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01000ad970 0x7f01001f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00fffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f01001f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00fffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01000c5dc0 con 0x7f010006c040 2023-10-16T13:50:58.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00fffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01001f32a0 con 0x7f01000ad970 2023-10-16T13:50:58.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00fffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01001f3580 con 0x7f010015ebc0 2023-10-16T13:50:58.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f01001efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00feffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f01001efd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53890/0 (socket says 172.21.15.60:53890) 2023-10-16T13:50:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00feffd640 1 -- 172.21.15.60:0/3846153992 learned_addr learned my addr 172.21.15.60:0/3846153992 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.208+0000 7f00f7fff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01000ad970 0x7f01001f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f01001f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00feffd640 1 -- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 msgr2=0x7f01001f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00feffd640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f01001f3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00feffd640 1 -- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01000ad970 msgr2=0x7f01001f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00feffd640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01000ad970 0x7f01001f3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00feffd640 1 -- 172.21.15.60:0/3846153992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f00094d0 con 0x7f010006c040 2023-10-16T13:50:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00ff7fe640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f01001f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:50:58.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00feffd640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f01001efd90 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f00f00097c0 tx=0x7f00f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f003ac60 con 0x7f010006c040 2023-10-16T13:50:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f003adc0 con 0x7f010006c040 2023-10-16T13:50:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01000c5770 con 0x7f010006c040 2023-10-16T13:50:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0044610 con 0x7f010006c040 2023-10-16T13:50:58.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.209+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01000c9f30 con 0x7f010006c040 2023-10-16T13:50:58.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.211+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010006b330 con 0x7f010006c040 2023-10-16T13:50:58.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.212+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 89) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f00f004d450 con 0x7f010006c040 2023-10-16T13:50:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.213+0000 7f00f77fe640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 0x7f00e006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.213+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00f00abe80 con 0x7f010006c040 2023-10-16T13:50:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.213+0000 7f00f7fff640 1 -- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 msgr2=0x7f00e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.213+0000 7f00f7fff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 0x7f00e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.217+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00f0076c70 con 0x7f010006c040 2023-10-16T13:50:58.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.409+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f010004e9b0 con 0x7f010006c040 2023-10-16T13:50:58.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.414+0000 7f00f7fff640 1 -- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 msgr2=0x7f00e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:58.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.414+0000 7f00f7fff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 0x7f00e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:58.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.415+0000 7f00f77fe640 1 -- 172.21.15.60:0/3846153992 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f00f0035020 con 0x7f010006c040 2023-10-16T13:50:58.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 msgr2=0x7f00e006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:50:58.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 0x7f00e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 msgr2=0x7f01001efd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f01001efd90 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f00f00097c0 tx=0x7f00f00046c0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 shutdown_connections 2023-10-16T13:50:58.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f00e006afe0 0x7f00e006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010015ebc0 0x7f01001f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01000ad970 0x7f01001f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 --2- 172.21.15.60:0/3846153992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f010006c040 0x7f01001efd90 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.417+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 >> 172.21.15.60:0/3846153992 conn(0x7f01000ce6a0 msgr2=0x7f010015de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:58.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.418+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 shutdown_connections 2023-10-16T13:50:58.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.418+0000 7f00fffff640 1 -- 172.21.15.60:0/3846153992 wait complete. 2023-10-16T13:50:58.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:50:58.439 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T13:50:58.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-10-16T13:50:58.527 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.525+0000 7f45be00f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:50:58.579 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.576+0000 7f45be00f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:50:58.590 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.588+0000 7fb319837040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:50:58.629 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.626+0000 7f45be00f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:50:58.645 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.643+0000 7fb319837040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:50:58.689 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.687+0000 7f45be00f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:50:58.700 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.698+0000 7fb319837040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:50:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.718+0000 7f88c4583640 1 Processor -- start 2023-10-16T13:50:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.718+0000 7f88c4583640 1 -- start start 2023-10-16T13:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c4583640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 0x7f88bc0aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c4583640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc06c140 con 0x7f88bc15aee0 2023-10-16T13:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c4583640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc15b2c0 con 0x7f88bc15e9b0 2023-10-16T13:50:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c4583640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc06b270 con 0x7f88bc0cbf20 2023-10-16T13:50:58.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c231f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f88bc15aee0 0x7f88bc0cb810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39764/0 (socket says 172.21.15.60:39764) 2023-10-16T13:50:58.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c231f640 1 -- 172.21.15.60:0/577502097 learned_addr learned my addr 172.21.15.60:0/577502097 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:50:58.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c1b1e640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 0x7f88bc0aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.720+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775291075 0 0) 0x7f88bc06c140 con 0x7f88bc15aee0 2023-10-16T13:50:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a8003520 con 0x7f88bc15aee0 2023-10-16T13:50:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389417301 0 0) 0x7f88a8003520 con 0x7f88bc15aee0 2023-10-16T13:50:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 msgr2=0x7f88bc0aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 0x7f88bc0aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 >> v1:172.21.15.144:6789/0 conn(0x7f88bc15e9b0 legacy=0x7f88bc15ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc06daf0 con 0x7f88bc15aee0 2023-10-16T13:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f88b8001670 con 0x7f88bc15aee0 2023-10-16T13:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 >> v1:172.21.15.60:6789/0 conn(0x7f88bc15aee0 legacy=0x7f88bc0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88a80039a0 0x7f88a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88a80047c0 0x7f88a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 0x7f88a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc06b270 con 0x7f88a80047c0 2023-10-16T13:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80054b0 con 0x7f88a80039a0 2023-10-16T13:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a8005670 con 0x7f88bc0cbf20 2023-10-16T13:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.721+0000 7f88c1b1e640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88a80047c0 0x7f88a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c231f640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88a80039a0 0x7f88a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c2b20640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 0x7f88a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c1b1e640 1 -- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 msgr2=0x7f88a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c1b1e640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88bc0cbf20 0x7f88a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c1b1e640 1 -- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88a80039a0 msgr2=0x7f88a8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c1b1e640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88a80039a0 0x7f88a8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c1b1e640 1 -- 172.21.15.60:0/577502097 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a8003520 con 0x7f88a80047c0 2023-10-16T13:50:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.722+0000 7f88c231f640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88a80039a0 0x7f88a8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.723+0000 7f88c1b1e640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88a80047c0 0x7f88a8004c40 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f88ac00ba30 tx=0x7f88a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7328672d767bda05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.723+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac032640 con 0x7f88a80047c0 2023-10-16T13:50:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.723+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac0327a0 con 0x7f88a80047c0 2023-10-16T13:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.723+0000 7f88c131d640 1 -- 172.21.15.60:0/577502097 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac032b30 con 0x7f88a80047c0 2023-10-16T13:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c4583640 1 -- 172.21.15.60:0/577502097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88a80047c0 msgr2=0x7f88a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c4583640 1 --2- 172.21.15.60:0/577502097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88a80047c0 0x7f88a8004c40 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f88ac00ba30 tx=0x7f88a80057f0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c2b20640 1 -- 172.21.15.60:0/577502097 reap_dead start 2023-10-16T13:50:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c4583640 1 -- 172.21.15.60:0/577502097 shutdown_connections 2023-10-16T13:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c4583640 1 -- 172.21.15.60:0/577502097 >> 172.21.15.60:0/577502097 conn(0x7f88bc0cfac0 msgr2=0x7f88bc0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c4583640 1 -- 172.21.15.60:0/577502097 shutdown_connections 2023-10-16T13:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.724+0000 7f88c4583640 1 -- 172.21.15.60:0/577502097 wait complete. 2023-10-16T13:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.726+0000 7f88c4583640 1 Processor -- start 2023-10-16T13:50:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.726+0000 7f88c4583640 1 -- start start 2023-10-16T13:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c4583640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88a80047c0 0x7f88bc154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c4583640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88bc15c160 0x7f88bc14ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c4583640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88bc155ab0 0x7f88bc0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c4583640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc1546a0 con 0x7f88bc155ab0 2023-10-16T13:50:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c4583640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc14efb0 con 0x7f88bc15c160 2023-10-16T13:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c4583640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc1530c0 con 0x7f88a80047c0 2023-10-16T13:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c2b20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88bc155ab0 0x7f88bc0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c2b20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88bc155ab0 0x7f88bc0b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53914/0 (socket says 172.21.15.60:53914) 2023-10-16T13:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.727+0000 7f88c2b20640 1 -- 172.21.15.60:0/1729103079 learned_addr learned my addr 172.21.15.60:0/1729103079 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:50:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88a80047c0 0x7f88bc154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c1b1e640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88bc15c160 0x7f88bc14ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c2b20640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88a80047c0 msgr2=0x7f88bc154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c2b20640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f88a80047c0 0x7f88bc154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c2b20640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88bc15c160 msgr2=0x7f88bc14ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c2b20640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f88bc15c160 0x7f88bc14ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c2b20640 1 -- 172.21.15.60:0/1729103079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc1533a0 con 0x7f88bc155ab0 2023-10-16T13:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c2b20640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88bc155ab0 0x7f88bc0b37b0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f88b4000f80 tx=0x7f88b400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b400eed0 con 0x7f88bc155ab0 2023-10-16T13:50:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b4003970 con 0x7f88bc155ab0 2023-10-16T13:50:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4003d00 con 0x7f88bc155ab0 2023-10-16T13:50:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc153680 con 0x7f88bc155ab0 2023-10-16T13:50:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.728+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc153c40 con 0x7f88bc155ab0 2023-10-16T13:50:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.730+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc04e160 con 0x7f88bc155ab0 2023-10-16T13:50:58.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.731+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 89) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f88b40050d0 con 0x7f88bc155ab0 2023-10-16T13:50:58.741 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.737+0000 7f45be00f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:50:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.731+0000 7f88b2ffd640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 0x7f88a006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.732+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88b4089570 con 0x7f88bc155ab0 2023-10-16T13:50:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.732+0000 7f88c231f640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 msgr2=0x7f88a006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:58.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.732+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 0x7f88a006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:50:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.735+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88b4054470 con 0x7f88bc155ab0 2023-10-16T13:50:58.767 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.765+0000 7fb319837040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:50:58.819 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.817+0000 7fb319837040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:50:58.850 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.848+0000 7f45be00f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:50:58.912 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.910+0000 7f45be00f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:50:58.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.933+0000 7f88c231f640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 msgr2=0x7f88a006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:58.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.933+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 0x7f88a006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:50:58.936 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.934+0000 7fb319837040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:50:58.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:58.951+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 --> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f88bc06b510 con 0x7f88a006b010 2023-10-16T13:50:58.961 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:58.959+0000 7f45be00f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:50:59.001 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:58.999+0000 7fb319837040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:50:59.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.016+0000 7f45be00f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:50:59.053 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.050+0000 7fb319837040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:50:59.069 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.067+0000 7f45be00f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:50:59.112 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.109+0000 7fb319837040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:50:59.164 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.162+0000 7fb319837040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:50:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:59.334+0000 7f88c231f640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 msgr2=0x7f88a006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:50:59.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:50:59.334+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 0x7f88a006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:50:59.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.444+0000 7f45be00f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:50:59.561 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.558+0000 7fb319837040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:50:59.603 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.600+0000 7f45be00f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:50:59.709 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.707+0000 7f45be00f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:50:59.728 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.725+0000 7fb319837040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:50:59.759 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.756+0000 7f45be00f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:50:59.841 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.838+0000 7fb319837040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:50:59.865 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.863+0000 7f45be00f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:50:59.893 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:50:59.891+0000 7fb319837040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:50:59.970 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:50:59.968+0000 7f45be00f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:51:00.005 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:00.003+0000 7fb319837040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:00.116 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:00.114+0000 7fb319837040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:51:00.126 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:00.123+0000 7f45be00f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:00.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:00.135+0000 7f88c231f640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 msgr2=0x7f88a006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2817709727 2023-10-16T13:51:00.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:00.135+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 0x7f88a006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:51:00.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:00.172+0000 7f45be00f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:00.280 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:00.277+0000 7fb319837040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:00.293 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:00.290+0000 7f45be00f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:51:00.331 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:00.329+0000 7fb319837040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:00.455 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:00.453+0000 7fb319837040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:51:01.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.129+0000 7f45be00f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:51:01.335 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:01.333+0000 7fb319837040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:51:01.426 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.424+0000 7f45be00f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:51:01.483 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.480+0000 7f45be00f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:51:01.595 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.593+0000 7f45be00f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:51:01.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:01.615+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 90) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f88b4054840 con 0x7f88bc155ab0 2023-10-16T13:51:01.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:01.615+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 msgr2=0x7f88a006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:01.615+0000 7f88b2ffd640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/2817709727,v1:172.21.15.144:6833/2817709727] conn(0x7f88a006b010 0x7f88a006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:01.650 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:01.648+0000 7fb319837040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:51:01.717 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:01.715+0000 7fb319837040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:51:01.850 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:01.848+0000 7fb319837040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:51:01.944 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.944 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.944 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.945 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.945 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.945 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453a6af640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.945 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453a6af640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453a6af640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453a6af640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:01.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:01.942+0000 7f453a6af640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:02.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.629+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 91) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f88b4012d90 con 0x7f88bc155ab0 2023-10-16T13:51:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.630+0000 7f88b2ffd640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f88a0035910 0x7f88a0074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.630+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 --> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f88b400eed0 con 0x7f88a0035910 2023-10-16T13:51:02.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.643+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f88a0035910 0x7f88a0074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:02.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.653+0000 7f88c231f640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f88a0035910 0x7f88a0074ed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f88b8001eb0 tx=0x7f88b8001e20 comp rx=0 tx=0).ready entity=mgr.6983 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:02.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.848+0000 7f88b2ffd640 1 -- 172.21.15.60:0/1729103079 <== mgr.6983 v2:172.21.15.144:6832/1295809784 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f88b400eed0 con 0x7f88a0035910 2023-10-16T13:51:02.851 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1 2023-10-16T13:51:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f88a0035910 msgr2=0x7f88a0074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f88a0035910 0x7f88a0074ed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f88b8001eb0 tx=0x7f88b8001e20 comp rx=0 tx=0).stop 2023-10-16T13:51:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88bc155ab0 msgr2=0x7f88bc0b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 --2- 172.21.15.60:0/1729103079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88bc155ab0 0x7f88bc0b37b0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f88b4000f80 tx=0x7f88b400eb90 comp rx=0 tx=0).stop 2023-10-16T13:51:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c2b20640 1 -- 172.21.15.60:0/1729103079 reap_dead start 2023-10-16T13:51:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 shutdown_connections 2023-10-16T13:51:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 >> 172.21.15.60:0/1729103079 conn(0x7f88bc0cfac0 msgr2=0x7f88bc0cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:02.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 shutdown_connections 2023-10-16T13:51:02.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:02.852+0000 7f88c4583640 1 -- 172.21.15.60:0/1729103079 wait complete. 2023-10-16T13:51:02.876 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T13:51:02.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T13:51:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.103+0000 7f6ea74a2640 1 Processor -- start 2023-10-16T13:51:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.104+0000 7f6ea74a2640 1 -- start start 2023-10-16T13:51:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.105+0000 7f6ea74a2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ea015ad70 0x7f6ea0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.105+0000 7f6ea74a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea74a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00cb030 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea74a2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea00c91c0 con 0x7f6ea015ad70 2023-10-16T13:51:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea74a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00c94a0 con 0x7f6ea00c9ca0 2023-10-16T13:51:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea523e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ea015ad70 0x7f6ea0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea523e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ea015ad70 0x7f6ea0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50870/0 (socket says 172.21.15.60:50870) 2023-10-16T13:51:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea523e640 1 -- 172.21.15.60:0/2624944726 learned_addr learned my addr 172.21.15.60:0/2624944726 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea4a3d640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.106+0000 7f6ea523e640 1 -- 172.21.15.60:0/2624944726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 msgr2=0x7f6ea006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.107+0000 7f6ea523e640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.107+0000 7f6ea523e640 1 -- 172.21.15.60:0/2624944726 >> v1:172.21.15.144:6789/0 conn(0x7f6ea00ca3b0 legacy=0x7f6ea00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.107+0000 7f6ea523e640 1 -- 172.21.15.60:0/2624944726 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00c9780 con 0x7f6ea015ad70 2023-10-16T13:51:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.107+0000 7f6ea4a3d640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.108+0000 7f6ea523e640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea015ad70 0x7f6ea0069da0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009e50 tx=0x7f6e9c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801dc02841f05285 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.108+0000 7f6e97fff640 1 -- 172.21.15.60:0/2624944726 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c02ca80 con 0x7f6ea015ad70 2023-10-16T13:51:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.108+0000 7f6e97fff640 1 -- 172.21.15.60:0/2624944726 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c02cbe0 con 0x7f6ea015ad70 2023-10-16T13:51:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.108+0000 7f6e97fff640 1 -- 172.21.15.60:0/2624944726 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0343f0 con 0x7f6ea015ad70 2023-10-16T13:51:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 -- 172.21.15.60:0/2624944726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea015ad70 msgr2=0x7f6ea0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea015ad70 0x7f6ea0069da0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009e50 tx=0x7f6e9c02bee0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 -- 172.21.15.60:0/2624944726 shutdown_connections 2023-10-16T13:51:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/2624944726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea015ad70 0x7f6ea0069da0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 -- 172.21.15.60:0/2624944726 >> 172.21.15.60:0/2624944726 conn(0x7f6ea00ce9b0 msgr2=0x7f6ea00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 -- 172.21.15.60:0/2624944726 shutdown_connections 2023-10-16T13:51:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.109+0000 7f6ea74a2640 1 -- 172.21.15.60:0/2624944726 wait complete. 2023-10-16T13:51:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.111+0000 7f6ea74a2640 1 Processor -- start 2023-10-16T13:51:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.111+0000 7f6ea74a2640 1 -- start start 2023-10-16T13:51:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea74a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea74a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 0x7f6ea0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea74a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ea015ad70 0x7f6ea006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea74a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea0157570 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea74a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea0157850 con 0x7f6ea015ad70 2023-10-16T13:51:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea74a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea0157b30 con 0x7f6ea00c9ca0 2023-10-16T13:51:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.112+0000 7f6ea4a3d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 0x7f6ea0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 0x7f6ea0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50884/0 (socket says 172.21.15.60:50884) 2023-10-16T13:51:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 -- 172.21.15.60:0/213252387 learned_addr learned my addr 172.21.15.60:0/213252387 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea523e640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 -- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 msgr2=0x7f6ea014bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea014bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 -- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ea015ad70 msgr2=0x7f6ea006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ea015ad70 0x7f6ea006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 -- 172.21.15.60:0/213252387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c009b30 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.113+0000 7f6ea4a3d640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 0x7f6ea0069da0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000c7f0 tx=0x7f6e9000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.114+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90007a30 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.114+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90007b90 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.114+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9000ce40 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.114+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea0157e10 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.114+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea0158330 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.115+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea0158050 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.116+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 91) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f6e900113f0 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.117+0000 7f6e95ffb640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f6e8406ad10 0x7f6e8406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.117+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6e90089610 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.118+0000 7f6ea523e640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f6e8406ad10 0x7f6e8406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.119+0000 7f6ea523e640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f6e8406ad10 0x7f6e8406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c035000 tx=0x7f6e9c030000 comp rx=0 tx=0).ready entity=mgr.6983 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:03.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.124+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ea0158050 con 0x7f6ea00ca3b0 2023-10-16T13:51:03.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.342+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 --> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ea01579c0 con 0x7f6e8406ad10 2023-10-16T13:51:03.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.536+0000 7f6e95ffb640 1 -- 172.21.15.60:0/213252387 <== mgr.6983 v2:172.21.15.144:6832/1295809784 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6ea01579c0 con 0x7f6e8406ad10 2023-10-16T13:51:03.539 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path snap_test_dir1 2023-10-16T13:51:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f6e8406ad10 msgr2=0x7f6e8406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f6e8406ad10 0x7f6e8406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c035000 tx=0x7f6e9c030000 comp rx=0 tx=0).stop 2023-10-16T13:51:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 msgr2=0x7f6ea0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 0x7f6ea0069da0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000c7f0 tx=0x7f6e9000cbf0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 shutdown_connections 2023-10-16T13:51:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f6e8406ad10 0x7f6e8406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ea015ad70 0x7f6ea006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ea00ca3b0 0x7f6ea0069da0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 --2- 172.21.15.60:0/213252387 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ea00c9ca0 0x7f6ea014bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 >> 172.21.15.60:0/213252387 conn(0x7f6ea00ce9b0 msgr2=0x7f6ea00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:03.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 shutdown_connections 2023-10-16T13:51:03.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.540+0000 7f6ea74a2640 1 -- 172.21.15.60:0/213252387 wait complete. 2023-10-16T13:51:03.565 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T13:51:03.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T13:51:03.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.793+0000 7fd706b7a640 1 Processor -- start 2023-10-16T13:51:03.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.793+0000 7fd706b7a640 1 -- start start 2023-10-16T13:51:03.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd706b7a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd700159950 0x7fd70015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd706b7a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd70006c810 0x7fd7000708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd706b7a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70015aef0 con 0x7fd700070450 2023-10-16T13:51:03.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd706b7a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd700070e20 con 0x7fd700159950 2023-10-16T13:51:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd706b7a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70015e490 con 0x7fd70006c810 2023-10-16T13:51:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd705b78640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd700159950 0x7fd70015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd705377640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd700070450 0x7fd70006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48960/0 (socket says 172.21.15.60:48960) 2023-10-16T13:51:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.795+0000 7fd705377640 1 -- 172.21.15.60:0/1068437692 learned_addr learned my addr 172.21.15.60:0/1068437692 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:03.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.796+0000 7fd705b78640 1 -- 172.21.15.60:0/1068437692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd70006c810 msgr2=0x7fd7000708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:03.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.796+0000 7fd705b78640 1 --2- 172.21.15.60:0/1068437692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd70006c810 0x7fd7000708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.796+0000 7fd705b78640 1 -- 172.21.15.60:0/1068437692 >> v1:172.21.15.60:6789/0 conn(0x7fd700070450 legacy=0x7fd70006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.796+0000 7fd705b78640 1 -- 172.21.15.60:0/1068437692 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70006cc60 con 0x7fd700159950 2023-10-16T13:51:03.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.797+0000 7fd705b78640 1 --2- 172.21.15.60:0/1068437692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd700159950 0x7fd70015a9b0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097a0 tx=0x7fd6fc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a42cffdc7c07c9b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:03.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.797+0000 7fd704b76640 1 -- 172.21.15.60:0/1068437692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc031780 con 0x7fd700159950 2023-10-16T13:51:03.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.797+0000 7fd704b76640 1 -- 172.21.15.60:0/1068437692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc0318e0 con 0x7fd700159950 2023-10-16T13:51:03.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.797+0000 7fd704b76640 1 -- 172.21.15.60:0/1068437692 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc031c70 con 0x7fd700159950 2023-10-16T13:51:03.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 -- 172.21.15.60:0/1068437692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd700159950 msgr2=0x7fd70015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 --2- 172.21.15.60:0/1068437692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd700159950 0x7fd70015a9b0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0097a0 tx=0x7fd6fc02b820 comp rx=0 tx=0).stop 2023-10-16T13:51:03.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 -- 172.21.15.60:0/1068437692 shutdown_connections 2023-10-16T13:51:03.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 --2- 172.21.15.60:0/1068437692 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd70006c810 0x7fd7000708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 --2- 172.21.15.60:0/1068437692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd700159950 0x7fd70015a9b0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 -- 172.21.15.60:0/1068437692 >> 172.21.15.60:0/1068437692 conn(0x7fd7000ce6a0 msgr2=0x7fd7000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:03.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 -- 172.21.15.60:0/1068437692 shutdown_connections 2023-10-16T13:51:03.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.798+0000 7fd706b7a640 1 -- 172.21.15.60:0/1068437692 wait complete. 2023-10-16T13:51:03.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.800+0000 7fd706b7a640 1 Processor -- start 2023-10-16T13:51:03.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.800+0000 7fd706b7a640 1 -- start start 2023-10-16T13:51:03.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd706b7a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd70006c810 0x7fd7000c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd706b7a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd700070450 0x7fd7001500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd706b7a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd700159950 0x7fd700154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd706b7a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7000c3bd0 con 0x7fd70006c810 2023-10-16T13:51:03.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd706b7a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7000c4190 con 0x7fd700159950 2023-10-16T13:51:03.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd706b7a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70015bb40 con 0x7fd700070450 2023-10-16T13:51:03.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd705b78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd70006c810 0x7fd7000c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd705b78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd70006c810 0x7fd7000c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50902/0 (socket says 172.21.15.60:50902) 2023-10-16T13:51:03.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.801+0000 7fd705b78640 1 -- 172.21.15.60:0/4193873333 learned_addr learned my addr 172.21.15.60:0/4193873333 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:03.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705377640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd700070450 0x7fd7001500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705b78640 1 -- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd700070450 msgr2=0x7fd7001500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd706379640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd700159950 0x7fd700154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705b78640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd700070450 0x7fd7001500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705b78640 1 -- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd700159950 msgr2=0x7fd700154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:03.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705b78640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd700159950 0x7fd700154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:03.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705b78640 1 -- 172.21.15.60:0/4193873333 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc009400 con 0x7fd70006c810 2023-10-16T13:51:03.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705377640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd700070450 0x7fd7001500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:03.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd705b78640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd70006c810 0x7fd7000c4c80 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc000c00 tx=0x7fd6fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:03.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc032960 con 0x7fd70006c810 2023-10-16T13:51:03.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc032ac0 con 0x7fd70006c810 2023-10-16T13:51:03.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.802+0000 7fd706b7a640 1 -- 172.21.15.60:0/4193873333 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70015bde0 con 0x7fd70006c810 2023-10-16T13:51:03.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.803+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc032e70 con 0x7fd70006c810 2023-10-16T13:51:03.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.803+0000 7fd706b7a640 1 -- 172.21.15.60:0/4193873333 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd700155b20 con 0x7fd70006c810 2023-10-16T13:51:03.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.804+0000 7fd706b7a640 1 -- 172.21.15.60:0/4193873333 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7000ca560 con 0x7fd70006c810 2023-10-16T13:51:03.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.805+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 92) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd6fc0453f0 con 0x7fd70006c810 2023-10-16T13:51:03.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.806+0000 7fd6f67fc640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fd6e406afe0 0x7fd6e406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:03.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.806+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd6fc0abe50 con 0x7fd70006c810 2023-10-16T13:51:03.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.806+0000 7fd705377640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fd6e406afe0 0x7fd6e406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:03.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.808+0000 7fd705377640 1 --2- 172.21.15.60:0/4193873333 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fd6e406afe0 0x7fd6e406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0007fd0 tx=0x7fd6f0009040 comp rx=0 tx=0).ready entity=mgr.6983 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:03.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:03.810+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6fc076ae0 con 0x7fd70006c810 2023-10-16T13:51:04.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.027+0000 7fd706b7a640 1 -- 172.21.15.60:0/4193873333 --> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd70015dbf0 con 0x7fd6e406afe0 2023-10-16T13:51:04.045 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:04.043+0000 7f4544703640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T13:51:04.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.044+0000 7fd6f67fc640 1 -- 172.21.15.60:0/4193873333 <== mgr.6983 v2:172.21.15.144:6832/1295809784 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fd6f0001d00 con 0x7fd6e406afe0 2023-10-16T13:51:04.046 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T13:51:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T13:51:04.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:04.067 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T13:51:04.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:51:04.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.361+0000 7ff8b969f640 1 Processor -- start 2023-10-16T13:51:04.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.362+0000 7ff8b969f640 1 -- start start 2023-10-16T13:51:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.363+0000 7ff8b969f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff8b40c9060 0x7ff8b406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.363+0000 7ff8b969f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 0x7ff8b406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.363+0000 7ff8b969f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40ca560 con 0x7ff8b406c9e0 2023-10-16T13:51:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.363+0000 7ff8b969f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b40c9950 con 0x7ff8b40c9060 2023-10-16T13:51:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.363+0000 7ff8b969f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40c9c30 con 0x7ff8b40c9440 2023-10-16T13:51:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.364+0000 7ff8b37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff8b406c9e0 0x7ff8b4164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48966/0 (socket says 172.21.15.60:48966) 2023-10-16T13:51:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.364+0000 7ff8b37fe640 1 -- 172.21.15.60:0/3711869149 learned_addr learned my addr 172.21.15.60:0/3711869149 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:04.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.364+0000 7ff8b8e9e640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 0x7ff8b406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.364+0000 7ff8b3fff640 1 --2- 172.21.15.60:0/3711869149 >> v2:172.21.15.144:3300/0 conn(0x7ff8b40c9060 0x7ff8b406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.364+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910669503 0 0) 0x7ff8b40ca560 con 0x7ff8b406c9e0 2023-10-16T13:51:04.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.364+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c003520 con 0x7ff8b406c9e0 2023-10-16T13:51:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2208099264 0 0) 0x7ff89c003520 con 0x7ff8b406c9e0 2023-10-16T13:51:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 msgr2=0x7ff8b406f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 0x7ff8b406f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 >> v2:172.21.15.144:3300/0 conn(0x7ff8b40c9060 msgr2=0x7ff8b406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 --2- 172.21.15.60:0/3711869149 >> v2:172.21.15.144:3300/0 conn(0x7ff8b40c9060 0x7ff8b406f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40c9f10 con 0x7ff8b406c9e0 2023-10-16T13:51:04.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff8a8001670 con 0x7ff8b406c9e0 2023-10-16T13:51:04.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 >> v1:172.21.15.60:6789/0 conn(0x7ff8b406c9e0 legacy=0x7ff8b4164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:04.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff89c003970 0x7ff89c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff89c0047c0 0x7ff89c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 0x7ff89c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40c9950 con 0x7ff89c0047c0 2023-10-16T13:51:04.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40c9c30 con 0x7ff89c003970 2023-10-16T13:51:04.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c00b980 con 0x7ff8b40c9440 2023-10-16T13:51:04.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.365+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff89c0047c0 0x7ff89c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b3fff640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff89c003970 0x7ff89c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b8e9e640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 0x7ff89c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b37fe640 1 -- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 msgr2=0x7ff89c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:04.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff8b40c9440 0x7ff89c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b37fe640 1 -- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff89c003970 msgr2=0x7ff89c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:04.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff89c003970 0x7ff89c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.366+0000 7ff8b37fe640 1 -- 172.21.15.60:0/3711869149 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c003520 con 0x7ff89c0047c0 2023-10-16T13:51:04.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.367+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff89c0047c0 0x7ff89c004c40 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7ff8b406cde0 tx=0x7ff89c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6511b7cd495a80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:04.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.367+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8005ad0 con 0x7ff89c0047c0 2023-10-16T13:51:04.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.367+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8005c50 con 0x7ff89c0047c0 2023-10-16T13:51:04.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.367+0000 7ff8b2ffd640 1 -- 172.21.15.60:0/3711869149 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a802c8d0 con 0x7ff89c0047c0 2023-10-16T13:51:04.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b969f640 1 -- 172.21.15.60:0/3711869149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff89c0047c0 msgr2=0x7ff89c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:04.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b969f640 1 --2- 172.21.15.60:0/3711869149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff89c0047c0 0x7ff89c004c40 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7ff8b406cde0 tx=0x7ff89c0057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b8e9e640 1 -- 172.21.15.60:0/3711869149 reap_dead start 2023-10-16T13:51:04.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b969f640 1 -- 172.21.15.60:0/3711869149 shutdown_connections 2023-10-16T13:51:04.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b969f640 1 -- 172.21.15.60:0/3711869149 >> 172.21.15.60:0/3711869149 conn(0x7ff8b40ce6a0 msgr2=0x7ff8b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:04.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b969f640 1 -- 172.21.15.60:0/3711869149 shutdown_connections 2023-10-16T13:51:04.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.368+0000 7ff8b969f640 1 -- 172.21.15.60:0/3711869149 wait complete. 2023-10-16T13:51:04.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.370+0000 7ff8b969f640 1 Processor -- start 2023-10-16T13:51:04.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.370+0000 7ff8b969f640 1 -- start start 2023-10-16T13:51:04.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b969f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff89c0047c0 0x7ff8b41f66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b969f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 0x7ff8b41633e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b969f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b4163920 0x7ff8b40ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b969f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40c5770 con 0x7ff8b41f9c80 2023-10-16T13:51:04.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b969f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b41f6c00 con 0x7ff8b4163920 2023-10-16T13:51:04.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b969f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b41fa110 con 0x7ff89c0047c0 2023-10-16T13:51:04.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 0x7ff8b41633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.371+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 0x7ff8b41633e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50918/0 (socket says 172.21.15.60:50918) 2023-10-16T13:51:04.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 -- 172.21.15.60:0/19612618 learned_addr learned my addr 172.21.15.60:0/19612618 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:04.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b3fff640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff89c0047c0 0x7ff8b41f66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 -- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff89c0047c0 msgr2=0x7ff8b41f66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:04.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b8e9e640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b4163920 0x7ff8b40ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff89c0047c0 0x7ff8b41f66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 -- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b4163920 msgr2=0x7ff8b40ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:04.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b4163920 0x7ff8b40ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 -- 172.21.15.60:0/19612618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b41fa3f0 con 0x7ff8b41f9c80 2023-10-16T13:51:04.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.372+0000 7ff8b37fe640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 0x7ff8b41633e0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c300 tx=0x7ff8a8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:04.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.373+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a802cce0 con 0x7ff8b41f9c80 2023-10-16T13:51:04.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.373+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a802ce40 con 0x7ff8b41f9c80 2023-10-16T13:51:04.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.373+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8007270 con 0x7ff8b41f9c80 2023-10-16T13:51:04.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.373+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b41fa6d0 con 0x7ff8b41f9c80 2023-10-16T13:51:04.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.373+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b41f9780 con 0x7ff8b41f9c80 2023-10-16T13:51:04.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.374+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b404e9b0 con 0x7ff8b41f9c80 2023-10-16T13:51:04.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.375+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 92) v1 ==== 89349+0+0 (secure 0 0 0) 0x7ff8a8032070 con 0x7ff8b41f9c80 2023-10-16T13:51:04.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.376+0000 7ff8b0ff9640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7ff8a006afe0 0x7ff8a006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:04.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.376+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff8a80ab0c0 con 0x7ff8b41f9c80 2023-10-16T13:51:04.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.377+0000 7ff8b3fff640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7ff8a006afe0 0x7ff8a006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:04.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.378+0000 7ff8b3fff640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7ff8a006afe0 0x7ff8a006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40098d0 tx=0x7ff8a4008040 comp rx=0 tx=0).ready entity=mgr.6983 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:04.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.381+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8a8075e60 con 0x7ff8b41f9c80 2023-10-16T13:51:04.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.629+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff8b406a720 con 0x7ff8b41f9c80 2023-10-16T13:51:04.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.649+0000 7ff8b0ff9640 1 -- 172.21.15.60:0/19612618 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v93) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff8a8079480 con 0x7ff8b41f9c80 2023-10-16T13:51:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7ff8a006afe0 msgr2=0x7ff8a006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7ff8a006afe0 0x7ff8a006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40098d0 tx=0x7ff8a4008040 comp rx=0 tx=0).stop 2023-10-16T13:51:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 msgr2=0x7ff8b41633e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 0x7ff8b41633e0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c300 tx=0x7ff8a8000fe0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 shutdown_connections 2023-10-16T13:51:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7ff8a006afe0 0x7ff8a006d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff8b4163920 0x7ff8b40ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b41f9c80 0x7ff8b41633e0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 --2- 172.21.15.60:0/19612618 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff89c0047c0 0x7ff8b41f66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.653+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 >> 172.21.15.60:0/19612618 conn(0x7ff8b40ce6a0 msgr2=0x7ff8b4069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.654+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 shutdown_connections 2023-10-16T13:51:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:04.654+0000 7ff8b969f640 1 -- 172.21.15.60:0/19612618 wait complete. 2023-10-16T13:51:04.681 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:51:04.681 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:51:04.798 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:51:04.892 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:51:04.892 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:51:04.893 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:04.948 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:51:04.948 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:04.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:05.203 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.200+0000 7f140fa13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:51:05.215 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.213+0000 7fcf7fa12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:51:05.256 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.254+0000 7f140fa13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:51:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.259+0000 7f861954e640 1 Processor -- start 2023-10-16T13:51:05.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.260+0000 7f861954e640 1 -- start start 2023-10-16T13:51:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f861954e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86140adb50 0x7f861406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f861954e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f86140b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f861954e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f86140adf30 0x7f86140b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f861954e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8614071450 con 0x7f86140adb50 2023-10-16T13:51:05.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f861954e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86140cc600 con 0x7f86140adf30 2023-10-16T13:51:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f861954e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86140cc8e0 con 0x7f8614070fd0 2023-10-16T13:51:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f8612ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86140adb50 0x7f861406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f8612ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86140adb50 0x7f861406f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50950/0 (socket says 172.21.15.60:50950) 2023-10-16T13:51:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f8612ffd640 1 -- 172.21.15.60:0/3093938789 learned_addr learned my addr 172.21.15.60:0/3093938789 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f86137fe640 1 --2- 172.21.15.60:0/3093938789 >> v2:172.21.15.144:3300/0 conn(0x7f86140adf30 0x7f86140b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.262+0000 7f860bfff640 1 --2- 172.21.15.60:0/3093938789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f86140b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.263+0000 7f8612ffd640 1 -- 172.21.15.60:0/3093938789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 msgr2=0x7f86140b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.263+0000 7f8612ffd640 1 --2- 172.21.15.60:0/3093938789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f86140b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.263+0000 7f8612ffd640 1 -- 172.21.15.60:0/3093938789 >> v2:172.21.15.144:3300/0 conn(0x7f86140adf30 msgr2=0x7f86140b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.263+0000 7f8612ffd640 1 --2- 172.21.15.60:0/3093938789 >> v2:172.21.15.144:3300/0 conn(0x7f86140adf30 0x7f86140b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.263+0000 7f8612ffd640 1 -- 172.21.15.60:0/3093938789 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86140ccbc0 con 0x7f86140adb50 2023-10-16T13:51:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.264+0000 7f8612ffd640 1 --2- 172.21.15.60:0/3093938789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adb50 0x7f861406f550 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f861406d840 tx=0x7f860402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601ae187092ef0d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.264+0000 7f86127fc640 1 -- 172.21.15.60:0/3093938789 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604031870 con 0x7f86140adb50 2023-10-16T13:51:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.264+0000 7f86127fc640 1 -- 172.21.15.60:0/3093938789 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86040319d0 con 0x7f86140adb50 2023-10-16T13:51:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.264+0000 7f86127fc640 1 -- 172.21.15.60:0/3093938789 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604031d60 con 0x7f86140adb50 2023-10-16T13:51:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 -- 172.21.15.60:0/3093938789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adb50 msgr2=0x7f861406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 --2- 172.21.15.60:0/3093938789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adb50 0x7f861406f550 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f861406d840 tx=0x7f860402b910 comp rx=0 tx=0).stop 2023-10-16T13:51:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 -- 172.21.15.60:0/3093938789 shutdown_connections 2023-10-16T13:51:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 --2- 172.21.15.60:0/3093938789 >> v2:172.21.15.144:3300/0 conn(0x7f86140adf30 0x7f86140b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 --2- 172.21.15.60:0/3093938789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f86140b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 --2- 172.21.15.60:0/3093938789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adb50 0x7f861406f550 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 -- 172.21.15.60:0/3093938789 >> 172.21.15.60:0/3093938789 conn(0x7f86140cb410 msgr2=0x7f86140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 -- 172.21.15.60:0/3093938789 shutdown_connections 2023-10-16T13:51:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.265+0000 7f861954e640 1 -- 172.21.15.60:0/3093938789 wait complete. 2023-10-16T13:51:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.267+0000 7f861954e640 1 Processor -- start 2023-10-16T13:51:05.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.267+0000 7f861954e640 1 -- start start 2023-10-16T13:51:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f861954e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f8614154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f861954e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86140adb50 0x7f86141641c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f861954e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 0x7f8614164700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f861954e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8614161aa0 con 0x7f86140adf30 2023-10-16T13:51:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f861954e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8614162060 con 0x7f86140adb50 2023-10-16T13:51:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f861954e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86140c4a60 con 0x7f8614070fd0 2023-10-16T13:51:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f86137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 0x7f8614164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f86137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 0x7f8614164700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50956/0 (socket says 172.21.15.60:50956) 2023-10-16T13:51:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f86137fe640 1 -- 172.21.15.60:0/530943769 learned_addr learned my addr 172.21.15.60:0/530943769 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f8612ffd640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f8614154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.268+0000 7f860bfff640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86140adb50 0x7f86141641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f86137fe640 1 -- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 msgr2=0x7f8614154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f86137fe640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f8614154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f86137fe640 1 -- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86140adb50 msgr2=0x7f86141641c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f86137fe640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86140adb50 0x7f86141641c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f86137fe640 1 -- 172.21.15.60:0/530943769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040094d0 con 0x7f86140adf30 2023-10-16T13:51:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f86137fe640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 0x7f8614164700 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f860c007e70 tx=0x7f860c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c014620 con 0x7f86140adf30 2023-10-16T13:51:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c0147a0 con 0x7f86140adf30 2023-10-16T13:51:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c014b50 con 0x7f86140adf30 2023-10-16T13:51:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.269+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86140c4d40 con 0x7f86140adf30 2023-10-16T13:51:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.270+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8614154770 con 0x7f86140adf30 2023-10-16T13:51:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.271+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861404e160 con 0x7f86140adf30 2023-10-16T13:51:05.277 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.271+0000 7fcf7fa12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:51:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.272+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f860c014cb0 con 0x7f86140adf30 2023-10-16T13:51:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.273+0000 7f860b7fe640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 0x7f85f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.273+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f860c08a240 con 0x7f86140adf30 2023-10-16T13:51:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.273+0000 7f8612ffd640 1 -- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 msgr2=0x7f85f406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:05.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.274+0000 7f8612ffd640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 0x7f85f406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:05.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.278+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f860c055070 con 0x7f86140adf30 2023-10-16T13:51:05.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.474+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86141621a0 con 0x7f86140adf30 2023-10-16T13:51:05.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.475+0000 7f8612ffd640 1 -- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 msgr2=0x7f85f406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:05.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.475+0000 7f8612ffd640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 0x7f85f406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:05.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.477+0000 7f860b7fe640 1 -- 172.21.15.60:0/530943769 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17076 (secure 0 0 0) 0x7f860c058690 con 0x7f86140adf30 2023-10-16T13:51:05.479 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:05.480 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":184,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:01.597787+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2455238630":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6833/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6832/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3372584586":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1775356785":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3925300811":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1821662703":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3430745922":"2023-10-17T13:51:01.597577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 msgr2=0x7f85f406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 0x7f85f406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 msgr2=0x7f8614164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 0x7f8614164700 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f860c007e70 tx=0x7f860c00e4b0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 shutdown_connections 2023-10-16T13:51:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f85f406af90 0x7f85f406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86140adf30 0x7f8614164700 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86140adb50 0x7f86141641c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 --2- 172.21.15.60:0/530943769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8614070fd0 0x7f8614154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 >> 172.21.15.60:0/530943769 conn(0x7f86140cb410 msgr2=0x7f8614160c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 shutdown_connections 2023-10-16T13:51:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.479+0000 7f861954e640 1 -- 172.21.15.60:0/530943769 wait complete. 2023-10-16T13:51:05.499 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:50:44.806197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:50:45.907641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:51:05.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:51:05.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.698+0000 7f7e871c0640 1 Processor -- start 2023-10-16T13:51:05.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.698+0000 7f7e871c0640 1 -- start start 2023-10-16T13:51:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.700+0000 7f7e871c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7e80070830 0x7f7e8006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.700+0000 7f7e871c0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e800cf5f0 0x7f7e8006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.700+0000 7f7e871c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e80161740 con 0x7f7e80070c10 2023-10-16T13:51:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.700+0000 7f7e871c0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8006c680 con 0x7f7e800cf5f0 2023-10-16T13:51:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.700+0000 7f7e871c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8006b0f0 con 0x7f7e80070830 2023-10-16T13:51:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e800cf5f0 0x7f7e8006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e800cf5f0 0x7f7e8006c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50960/0 (socket says 172.21.15.60:50960) 2023-10-16T13:51:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 -- 172.21.15.60:0/706114919 learned_addr learned my addr 172.21.15.60:0/706114919 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:05.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e84f5c640 1 --2- 172.21.15.60:0/706114919 >> v2:172.21.15.144:3300/0 conn(0x7f7e80070830 0x7f7e8006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 -- 172.21.15.60:0/706114919 >> v2:172.21.15.144:3300/0 conn(0x7f7e80070830 msgr2=0x7f7e8006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 --2- 172.21.15.60:0/706114919 >> v2:172.21.15.144:3300/0 conn(0x7f7e80070830 0x7f7e8006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 -- 172.21.15.60:0/706114919 >> v1:172.21.15.144:6789/0 conn(0x7f7e80070c10 legacy=0x7f7e800cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.701+0000 7f7e7ffff640 1 -- 172.21.15.60:0/706114919 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e801619b0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.702+0000 7f7e7ffff640 1 --2- 172.21.15.60:0/706114919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e8006c140 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7e8006c010 tx=0x7f7e6c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed8a7f9a89798b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.703+0000 7f7e7f7fe640 1 -- 172.21.15.60:0/706114919 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0317e0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.703+0000 7f7e7f7fe640 1 -- 172.21.15.60:0/706114919 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c031940 con 0x7f7e800cf5f0 2023-10-16T13:51:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.703+0000 7f7e7f7fe640 1 -- 172.21.15.60:0/706114919 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c031cd0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.703+0000 7f7e871c0640 1 -- 172.21.15.60:0/706114919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 msgr2=0x7f7e8006c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.703+0000 7f7e871c0640 1 --2- 172.21.15.60:0/706114919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e8006c140 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7e8006c010 tx=0x7f7e6c02b880 comp rx=0 tx=0).stop 2023-10-16T13:51:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.704+0000 7f7e871c0640 1 -- 172.21.15.60:0/706114919 shutdown_connections 2023-10-16T13:51:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.704+0000 7f7e871c0640 1 --2- 172.21.15.60:0/706114919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e8006c140 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.704+0000 7f7e871c0640 1 --2- 172.21.15.60:0/706114919 >> v2:172.21.15.144:3300/0 conn(0x7f7e80070830 0x7f7e8006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.704+0000 7f7e871c0640 1 -- 172.21.15.60:0/706114919 >> 172.21.15.60:0/706114919 conn(0x7f7e800cfac0 msgr2=0x7f7e800cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:05.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.704+0000 7f7e871c0640 1 -- 172.21.15.60:0/706114919 shutdown_connections 2023-10-16T13:51:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.704+0000 7f7e871c0640 1 -- 172.21.15.60:0/706114919 wait complete. 2023-10-16T13:51:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.705+0000 7f7e871c0640 1 Processor -- start 2023-10-16T13:51:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.706+0000 7f7e871c0640 1 -- start start 2023-10-16T13:51:05.709 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.705+0000 7f140fa13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:51:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e871c0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e80070830 0x7f7e8015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e871c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e80070c10 0x7f7e800b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e871c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e800c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e871c0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800b29f0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e871c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800af520 con 0x7f7e80070c10 2023-10-16T13:51:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e871c0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800b3b40 con 0x7f7e80070830 2023-10-16T13:51:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e800c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e800c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50976/0 (socket says 172.21.15.60:50976) 2023-10-16T13:51:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 -- 172.21.15.60:0/117780032 learned_addr learned my addr 172.21.15.60:0/117780032 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e7ffff640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e80070c10 0x7f7e800b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e84f5c640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e80070830 0x7f7e8015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 -- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e80070830 msgr2=0x7f7e8015dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e80070830 0x7f7e8015dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 -- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e80070c10 msgr2=0x7f7e800b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e80070c10 0x7f7e800b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.707+0000 7f7e8575d640 1 -- 172.21.15.60:0/117780032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0094d0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.708+0000 7f7e8575d640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e800c8e80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800c7a0 tx=0x7f7e7800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.708+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78003b70 con 0x7f7e800cf5f0 2023-10-16T13:51:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.708+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e78003cf0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.708+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78003ea0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.708+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e800b3e20 con 0x7f7e800cf5f0 2023-10-16T13:51:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.708+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e800b43e0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.709+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e800af6b0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.711+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7e7800aa90 con 0x7f7e800cf5f0 2023-10-16T13:51:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.712+0000 7f7e7d7fa640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 0x7f7e6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.712+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7e78089760 con 0x7f7e800cf5f0 2023-10-16T13:51:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.712+0000 7f7e84f5c640 1 -- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 msgr2=0x7f7e6406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:05.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.713+0000 7f7e84f5c640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 0x7f7e6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:05.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.716+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e78054590 con 0x7f7e800cf5f0 2023-10-16T13:51:05.754 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.752+0000 7fcf7fa12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:51:05.759 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.757+0000 7f140fa13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:51:05.810 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.807+0000 7f140fa13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:51:05.810 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.808+0000 7fcf7fa12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:51:05.863 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.861+0000 7fcf7fa12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:51:05.870 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.868+0000 7f140fa13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:51:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.914+0000 7f7e84f5c640 1 -- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 msgr2=0x7f7e6406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.914+0000 7f7e84f5c640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 0x7f7e6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.914+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e8004e160 con 0x7f7e800cf5f0 2023-10-16T13:51:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.915+0000 7f7e7d7fa640 1 -- 172.21.15.60:0/117780032 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e78057bb0 con 0x7f7e800cf5f0 2023-10-16T13:51:05.918 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:51:05.920 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:05.917+0000 7f140fa13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:51:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 msgr2=0x7f7e6406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 0x7f7e6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 msgr2=0x7f7e800c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e800c8e80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800c7a0 tx=0x7f7e7800cc60 comp rx=0 tx=0).stop 2023-10-16T13:51:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 shutdown_connections 2023-10-16T13:51:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f7e6406af90 0x7f7e6406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e800cf5f0 0x7f7e800c8e80 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e80070c10 0x7f7e800b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.917+0000 7f7e871c0640 1 --2- 172.21.15.60:0/117780032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7e80070830 0x7f7e8015dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.918+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 >> 172.21.15.60:0/117780032 conn(0x7f7e800cfac0 msgr2=0x7f7e800ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.918+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 shutdown_connections 2023-10-16T13:51:05.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:05.918+0000 7f7e871c0640 1 -- 172.21.15.60:0/117780032 wait complete. 2023-10-16T13:51:05.929 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.926+0000 7fcf7fa12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:51:05.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:51:05.981 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:05.979+0000 7fcf7fa12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:51:06.030 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.028+0000 7f140fa13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:51:06.092 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.090+0000 7f140fa13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:06.102 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.100+0000 7fcf7fa12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:51:06.142 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.139+0000 7f140fa13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:51:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.140+0000 7fa5f5a9f640 1 Processor -- start 2023-10-16T13:51:06.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.141+0000 7fa5f5a9f640 1 -- start start 2023-10-16T13:51:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f5a9f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5f015cd70 0x7fa5f00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f5a9f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f5a9f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f015d150 con 0x7fa5f00d0be0 2023-10-16T13:51:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f5a9f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f00d1180 con 0x7fa5f015cd70 2023-10-16T13:51:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f5a9f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f00d1460 con 0x7fa5f00b8480 2023-10-16T13:51:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f4a9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5f015cd70 0x7fa5f00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f4a9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5f015cd70 0x7fa5f00d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50986/0 (socket says 172.21.15.60:50986) 2023-10-16T13:51:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f529e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa5f00d0be0 0x7fa5f00b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48992/0 (socket says 172.21.15.60:48992) 2023-10-16T13:51:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.142+0000 7fa5f4a9d640 1 -- 172.21.15.60:0/587445855 learned_addr learned my addr 172.21.15.60:0/587445855 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5e7fff640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5f4a9d640 1 -- 172.21.15.60:0/587445855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 msgr2=0x7fa5f00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5f4a9d640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5f4a9d640 1 -- 172.21.15.60:0/587445855 >> v1:172.21.15.60:6789/0 conn(0x7fa5f00d0be0 legacy=0x7fa5f00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5f4a9d640 1 -- 172.21.15.60:0/587445855 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f00d1740 con 0x7fa5f015cd70 2023-10-16T13:51:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5e77fe640 1 -- 172.21.15.60:0/587445855 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091245134 0 0) 0x7fa5f015d150 con 0x7fa5f00d0be0 2023-10-16T13:51:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.143+0000 7fa5e7fff640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.144+0000 7fa5f4a9d640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f015cd70 0x7fa5f00d2220 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009e50 tx=0x7fa5e002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4921df5379139a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.144+0000 7fa5e77fe640 1 -- 172.21.15.60:0/587445855 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e002cb00 con 0x7fa5f015cd70 2023-10-16T13:51:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5e77fe640 1 -- 172.21.15.60:0/587445855 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e002cc60 con 0x7fa5f015cd70 2023-10-16T13:51:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5e77fe640 1 -- 172.21.15.60:0/587445855 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00343f0 con 0x7fa5f015cd70 2023-10-16T13:51:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/587445855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f015cd70 msgr2=0x7fa5f00d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f015cd70 0x7fa5f00d2220 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009e50 tx=0x7fa5e002bf60 comp rx=0 tx=0).stop 2023-10-16T13:51:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/587445855 shutdown_connections 2023-10-16T13:51:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/587445855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f015cd70 0x7fa5f00d2220 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/587445855 >> 172.21.15.60:0/587445855 conn(0x7fa5f00cf570 msgr2=0x7fa5f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/587445855 shutdown_connections 2023-10-16T13:51:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.145+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/587445855 wait complete. 2023-10-16T13:51:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.147+0000 7fa5f5a9f640 1 Processor -- start 2023-10-16T13:51:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.147+0000 7fa5f5a9f640 1 -- start start 2023-10-16T13:51:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5f5a9f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5f5a9f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 0x7fa5f0153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5f5a9f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f015cd70 0x7fa5f0159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5f5a9f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f0154d00 con 0x7fa5f00d0be0 2023-10-16T13:51:06.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5f5a9f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f014fb70 con 0x7fa5f015cd70 2023-10-16T13:51:06.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5f5a9f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f00cc790 con 0x7fa5f00b8480 2023-10-16T13:51:06.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.148+0000 7fa5e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 0x7fa5f0153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 0x7fa5f0153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50994/0 (socket says 172.21.15.60:50994) 2023-10-16T13:51:06.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 -- 172.21.15.60:0/2684729331 learned_addr learned my addr 172.21.15.60:0/2684729331 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5f529e640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f015cd70 0x7fa5f0159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5f4a9d640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 -- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 msgr2=0x7fa5f014f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f014f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 -- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f015cd70 msgr2=0x7fa5f0159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f015cd70 0x7fa5f0159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 -- 172.21.15.60:0/2684729331 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e0009b30 con 0x7fa5f00d0be0 2023-10-16T13:51:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5f529e640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f015cd70 0x7fa5f0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e7fff640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 0x7fa5f0153440 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00d7d0 tx=0x7fa5ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.149+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec004030 con 0x7fa5f00d0be0 2023-10-16T13:51:06.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.150+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec004190 con 0x7fa5f00d0be0 2023-10-16T13:51:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.150+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f0154a20 con 0x7fa5f00d0be0 2023-10-16T13:51:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.150+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec004520 con 0x7fa5f00d0be0 2023-10-16T13:51:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.150+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f0152c40 con 0x7fa5f00d0be0 2023-10-16T13:51:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.151+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f004e9b0 con 0x7fa5f00d0be0 2023-10-16T13:51:06.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.152+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa5ec00b6c0 con 0x7fa5f00d0be0 2023-10-16T13:51:06.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.153+0000 7fa5e57fa640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fa5dc06afc0 0x7fa5dc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.153+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa5ec089ab0 con 0x7fa5f00d0be0 2023-10-16T13:51:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.153+0000 7fa5f4a9d640 1 -- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fa5dc06afc0 msgr2=0x7fa5dc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.153+0000 7fa5f4a9d640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fa5dc06afc0 0x7fa5dc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:06.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.157+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5ec0548e0 con 0x7fa5f00d0be0 2023-10-16T13:51:06.171 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.169+0000 7fcf7fa12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:06.200 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.198+0000 7f140fa13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:51:06.224 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.222+0000 7fcf7fa12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:51:06.250 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.248+0000 7f140fa13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:51:06.285 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.283+0000 7fcf7fa12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:51:06.340 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.338+0000 7fcf7fa12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:51:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.350+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa5f015c3c0 con 0x7fa5f00d0be0 2023-10-16T13:51:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.350+0000 7fa5e57fa640 1 -- 172.21.15.60:0/2684729331 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v184) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa5ec057f00 con 0x7fa5f00d0be0 2023-10-16T13:51:06.352 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T13:51:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fa5dc06afc0 msgr2=0x7fa5dc06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fa5dc06afc0 0x7fa5dc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 msgr2=0x7fa5f0153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 0x7fa5f0153440 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00d7d0 tx=0x7fa5ec00dc90 comp rx=0 tx=0).stop 2023-10-16T13:51:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 shutdown_connections 2023-10-16T13:51:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fa5dc06afc0 0x7fa5dc06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5f015cd70 0x7fa5f0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f00d0be0 0x7fa5f0153440 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 --2- 172.21.15.60:0/2684729331 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5f00b8480 0x7fa5f014f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 >> 172.21.15.60:0/2684729331 conn(0x7fa5f00cf570 msgr2=0x7fa5f015b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 shutdown_connections 2023-10-16T13:51:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.353+0000 7fa5f5a9f640 1 -- 172.21.15.60:0/2684729331 wait complete. 2023-10-16T13:51:06.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:51:06.632 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.629+0000 7f140fa13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:51:06.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.633+0000 7fb2463f0640 1 Processor -- start 2023-10-16T13:51:06.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.633+0000 7fb2463f0640 1 -- start start 2023-10-16T13:51:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.635+0000 7fb2463f0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2400b3420 0x7fb2400b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.635+0000 7fb2463f0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.635+0000 7fb2463f0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2400d3980 con 0x7fb2400cf710 2023-10-16T13:51:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.635+0000 7fb2463f0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2400d3c60 con 0x7fb2400b3420 2023-10-16T13:51:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.635+0000 7fb2463f0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400ce630 con 0x7fb2400cfaf0 2023-10-16T13:51:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb24498d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2400b3420 0x7fb2400b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2400b3420 0x7fb2400b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55848/0 (socket says 172.21.15.60:55848) 2023-10-16T13:51:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 -- 172.21.15.60:0/376202039 learned_addr learned my addr 172.21.15.60:0/376202039 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 -- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 msgr2=0x7fb2400b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 --2- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 -- 172.21.15.60:0/376202039 >> v1:172.21.15.144:6789/0 conn(0x7fb2400cf710 legacy=0x7fb2400b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.636+0000 7fb237fff640 1 -- 172.21.15.60:0/376202039 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2400ce910 con 0x7fb2400b3420 2023-10-16T13:51:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.638+0000 7fb237fff640 1 --2- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400b3420 0x7fb2400b5cc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb228009870 tx=0x7fb22802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fa761358d2286cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.638+0000 7fb23f7fe640 1 -- 172.21.15.60:0/376202039 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228031640 con 0x7fb2400b3420 2023-10-16T13:51:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.638+0000 7fb23f7fe640 1 -- 172.21.15.60:0/376202039 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2280317a0 con 0x7fb2400b3420 2023-10-16T13:51:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.638+0000 7fb23f7fe640 1 -- 172.21.15.60:0/376202039 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228031b30 con 0x7fb2400b3420 2023-10-16T13:51:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 -- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400b3420 msgr2=0x7fb2400b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 --2- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400b3420 0x7fb2400b5cc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb228009870 tx=0x7fb22802b8f0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 -- 172.21.15.60:0/376202039 shutdown_connections 2023-10-16T13:51:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 --2- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 --2- 172.21.15.60:0/376202039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400b3420 0x7fb2400b5cc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 -- 172.21.15.60:0/376202039 >> 172.21.15.60:0/376202039 conn(0x7fb2400d2cc0 msgr2=0x7fb2400d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 -- 172.21.15.60:0/376202039 shutdown_connections 2023-10-16T13:51:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.639+0000 7fb2463f0640 1 -- 172.21.15.60:0/376202039 wait complete. 2023-10-16T13:51:06.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.640+0000 7fb2463f0640 1 Processor -- start 2023-10-16T13:51:06.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.641+0000 7fb2463f0640 1 -- start start 2023-10-16T13:51:06.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb2463f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 0x7fb2401f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb2463f0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400cf710 0x7fb2401f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb2463f0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb2463f0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400ca2d0 con 0x7fb2400b3420 2023-10-16T13:51:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb2463f0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400ca5b0 con 0x7fb2400cf710 2023-10-16T13:51:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb2463f0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400ca890 con 0x7fb2400cfaf0 2023-10-16T13:51:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 0x7fb2401f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 0x7fb2401f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51006/0 (socket says 172.21.15.60:51006) 2023-10-16T13:51:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 -- 172.21.15.60:0/2385847559 learned_addr learned my addr 172.21.15.60:0/2385847559 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb24498d640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb237fff640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400cf710 0x7fb2401f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 -- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 msgr2=0x7fb2400b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 -- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400cf710 msgr2=0x7fb2401f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400cf710 0x7fb2401f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.642+0000 7fb23ffff640 1 -- 172.21.15.60:0/2385847559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2280094d0 con 0x7fb2400b3420 2023-10-16T13:51:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.643+0000 7fb23ffff640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 0x7fb2401f31d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c3c0 tx=0x7fb23000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.643+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300052d0 con 0x7fb2400b3420 2023-10-16T13:51:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.643+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb230005470 con 0x7fb2400b3420 2023-10-16T13:51:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.643+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2400cab70 con 0x7fb2400b3420 2023-10-16T13:51:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.643+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230006a60 con 0x7fb2400b3420 2023-10-16T13:51:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.643+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401f5320 con 0x7fb2400b3420 2023-10-16T13:51:06.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.644+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24004e160 con 0x7fb2400b3420 2023-10-16T13:51:06.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.645+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb230006bc0 con 0x7fb2400b3420 2023-10-16T13:51:06.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.647+0000 7fb23d7fa640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fb22006af90 0x7fb22006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:06.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.647+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fb230088c30 con 0x7fb2400b3420 2023-10-16T13:51:06.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.647+0000 7fb237fff640 1 -- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fb22006af90 msgr2=0x7fb22006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:06.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.647+0000 7fb237fff640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fb22006af90 0x7fb22006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:06.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.651+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb230053a60 con 0x7fb2400b3420 2023-10-16T13:51:06.756 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.754+0000 7fcf7fa12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:51:06.793 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.790+0000 7f140fa13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:51:06.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.843+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2400cb090 con 0x7fb2400b3420 2023-10-16T13:51:06.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.844+0000 7fb23d7fa640 1 -- 172.21.15.60:0/2385847559 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v184) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb230057080 con 0x7fb2400b3420 2023-10-16T13:51:06.846 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T13:51:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fb22006af90 msgr2=0x7fb22006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fb22006af90 0x7fb22006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 msgr2=0x7fb2401f31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:06.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 0x7fb2401f31d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c3c0 tx=0x7fb23000c880 comp rx=0 tx=0).stop 2023-10-16T13:51:06.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 shutdown_connections 2023-10-16T13:51:06.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fb22006af90 0x7fb22006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2400cfaf0 0x7fb2400b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2400cf710 0x7fb2401f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 --2- 172.21.15.60:0/2385847559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2400b3420 0x7fb2401f31d0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:06.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.846+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 >> 172.21.15.60:0/2385847559 conn(0x7fb2400d2cc0 msgr2=0x7fb24006a410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:06.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.847+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 shutdown_connections 2023-10-16T13:51:06.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:06.847+0000 7fb2463f0640 1 -- 172.21.15.60:0/2385847559 wait complete. 2023-10-16T13:51:06.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:51:06.899 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.897+0000 7f140fa13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:51:06.928 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:06.926+0000 7fcf7fa12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:51:06.949 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:06.946+0000 7f140fa13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:51:07.047 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.044+0000 7fcf7fa12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:51:07.055 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:07.053+0000 7f140fa13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.083+0000 7f79609ef640 1 Processor -- start 2023-10-16T13:51:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.083+0000 7f79609ef640 1 -- start start 2023-10-16T13:51:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f79609ef640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f795c15cd70 0x7f795c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f79609ef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f795c0cded0 0x7f795c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f79609ef640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c15d150 con 0x7f795c0b8480 2023-10-16T13:51:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f79609ef640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795c0d1180 con 0x7f795c15cd70 2023-10-16T13:51:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f79609ef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795c0d1460 con 0x7f795c0cded0 2023-10-16T13:51:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f795ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f795c15cd70 0x7f795c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f795259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f795c0b8480 0x7f795c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48996/0 (socket says 172.21.15.60:48996) 2023-10-16T13:51:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.085+0000 7f795259c640 1 -- 172.21.15.60:0/348861985 learned_addr learned my addr 172.21.15.60:0/348861985 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.086+0000 7f795b59e640 1 --2- 172.21.15.60:0/348861985 >> v2:172.21.15.144:3300/0 conn(0x7f795c0cded0 0x7f795c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.086+0000 7f795ad9d640 1 -- 172.21.15.60:0/348861985 >> v2:172.21.15.144:3300/0 conn(0x7f795c0cded0 msgr2=0x7f795c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.086+0000 7f795ad9d640 1 --2- 172.21.15.60:0/348861985 >> v2:172.21.15.144:3300/0 conn(0x7f795c0cded0 0x7f795c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.086+0000 7f795a59c640 1 -- 172.21.15.60:0/348861985 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921536204 0 0) 0x7f795c15d150 con 0x7f795c0b8480 2023-10-16T13:51:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.086+0000 7f795ad9d640 1 -- 172.21.15.60:0/348861985 >> v1:172.21.15.60:6789/0 conn(0x7f795c0b8480 legacy=0x7f795c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.086+0000 7f795ad9d640 1 -- 172.21.15.60:0/348861985 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c0d1740 con 0x7f795c15cd70 2023-10-16T13:51:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.087+0000 7f795ad9d640 1 --2- 172.21.15.60:0/348861985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c15cd70 0x7f795c0d2220 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f795c0b8910 tx=0x7f794c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf220714497d4a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.087+0000 7f795a59c640 1 -- 172.21.15.60:0/348861985 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c02cc80 con 0x7f795c15cd70 2023-10-16T13:51:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.087+0000 7f795a59c640 1 -- 172.21.15.60:0/348861985 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c02cde0 con 0x7f795c15cd70 2023-10-16T13:51:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.087+0000 7f795a59c640 1 -- 172.21.15.60:0/348861985 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c034660 con 0x7f795c15cd70 2023-10-16T13:51:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 -- 172.21.15.60:0/348861985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c15cd70 msgr2=0x7f795c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 --2- 172.21.15.60:0/348861985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c15cd70 0x7f795c0d2220 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f795c0b8910 tx=0x7f794c02bed0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 -- 172.21.15.60:0/348861985 shutdown_connections 2023-10-16T13:51:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 --2- 172.21.15.60:0/348861985 >> v2:172.21.15.144:3300/0 conn(0x7f795c0cded0 0x7f795c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 --2- 172.21.15.60:0/348861985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c15cd70 0x7f795c0d2220 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 -- 172.21.15.60:0/348861985 >> 172.21.15.60:0/348861985 conn(0x7f795c0cf570 msgr2=0x7f795c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 -- 172.21.15.60:0/348861985 shutdown_connections 2023-10-16T13:51:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.088+0000 7f79609ef640 1 -- 172.21.15.60:0/348861985 wait complete. 2023-10-16T13:51:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.090+0000 7f79609ef640 1 Processor -- start 2023-10-16T13:51:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.090+0000 7f79609ef640 1 -- start start 2023-10-16T13:51:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f79609ef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f795c0b8480 0x7f795c1f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f79609ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 0x7f795c0a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f79609ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795c15cd70 0x7f795c0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f79609ef640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c1f3650 con 0x7f795c0cded0 2023-10-16T13:51:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f79609ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c1f2a20 con 0x7f795c15cd70 2023-10-16T13:51:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f79609ef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795c1f2d00 con 0x7f795c0b8480 2023-10-16T13:51:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f795259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 0x7f795c0a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f795259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 0x7f795c0a9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51020/0 (socket says 172.21.15.60:51020) 2023-10-16T13:51:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.091+0000 7f795259c640 1 -- 172.21.15.60:0/3816241544 learned_addr learned my addr 172.21.15.60:0/3816241544 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795b59e640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795c15cd70 0x7f795c0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795ad9d640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f795c0b8480 0x7f795c1f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795259c640 1 -- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f795c0b8480 msgr2=0x7f795c1f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795259c640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f795c0b8480 0x7f795c1f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795259c640 1 -- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795c15cd70 msgr2=0x7f795c0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795259c640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795c15cd70 0x7f795c0b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795259c640 1 -- 172.21.15.60:0/3816241544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c009a70 con 0x7f795c0cded0 2023-10-16T13:51:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795ad9d640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f795c0b8480 0x7f795c1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795b59e640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795c15cd70 0x7f795c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f795259c640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 0x7f795c0a9ab0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f794400c4d0 tx=0x7f794400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79440074e0 con 0x7f795c0cded0 2023-10-16T13:51:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7944007660 con 0x7f795c0cded0 2023-10-16T13:51:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c1f2fe0 con 0x7f795c0cded0 2023-10-16T13:51:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944004a20 con 0x7f795c0cded0 2023-10-16T13:51:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.092+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c1f32c0 con 0x7f795c0cded0 2023-10-16T13:51:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.093+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c04e9b0 con 0x7f795c0cded0 2023-10-16T13:51:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.095+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7944004b80 con 0x7f795c0cded0 2023-10-16T13:51:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.095+0000 7f7953fff640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 0x7f793c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.096+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7944088e60 con 0x7f795c0cded0 2023-10-16T13:51:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.096+0000 7f795ad9d640 1 -- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 msgr2=0x7f793c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.096+0000 7f795ad9d640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 0x7f793c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.100+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7944053ca0 con 0x7f795c0cded0 2023-10-16T13:51:07.103 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.100+0000 7fcf7fa12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:51:07.164 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:07.161+0000 7f140fa13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:51:07.217 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.215+0000 7fcf7fa12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:07.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.297+0000 7f795ad9d640 1 -- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 msgr2=0x7f793c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:07.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.297+0000 7f795ad9d640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 0x7f793c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.305+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f795c1f3790 con 0x7f795c0cded0 2023-10-16T13:51:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.305+0000 7f7953fff640 1 -- 172.21.15.60:0/3816241544 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v184) v1 ==== 80+0+478 (secure 0 0 0) 0x7f7944018020 con 0x7f795c0cded0 2023-10-16T13:51:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2455238630 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/2817709727 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/2817709727 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3372584586 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.308 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1775356785 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.309 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3925300811 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.309 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1821662703 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.309 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3430745922 2023-10-17T13:51:01.597577+0000 2023-10-16T13:51:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T13:51:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 msgr2=0x7f793c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 0x7f793c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 msgr2=0x7f795c0a9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 0x7f795c0a9ab0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f794400c4d0 tx=0x7f794400c990 comp rx=0 tx=0).stop 2023-10-16T13:51:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 shutdown_connections 2023-10-16T13:51:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f793c06af90 0x7f793c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f795c15cd70 0x7f795c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f795c0cded0 0x7f795c0a9ab0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 --2- 172.21.15.60:0/3816241544 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f795c0b8480 0x7f795c1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 >> 172.21.15.60:0/3816241544 conn(0x7f795c0cf570 msgr2=0x7f795c0b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 shutdown_connections 2023-10-16T13:51:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.308+0000 7f79609ef640 1 -- 172.21.15.60:0/3816241544 wait complete. 2023-10-16T13:51:07.319 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:07.317+0000 7f140fa13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:07.330 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:51:07.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:07.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:51:07.336 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.331+0000 7fcf7fa12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:51:07.369 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:07.366+0000 7f140fa13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:07.421 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T13:51:07.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:07.422 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:51:07.470 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:51:07.471 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:07.491 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:07.488+0000 7f140fa13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:51:07.508 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.506+0000 7fcf7fa12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:07.527 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:51:07.527 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:51:07.528 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:07.562 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.560+0000 7fcf7fa12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:07.582 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T13:51:07.582 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:51:07.583 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:07.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:51:07.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:07.692 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:07.690+0000 7fcf7fa12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:51:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.856+0000 7f07fe4ff640 1 Processor -- start 2023-10-16T13:51:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.856+0000 7f07fe4ff640 1 -- start start 2023-10-16T13:51:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.857+0000 7f07fe4ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 0x7f07f80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.858+0000 7f07fe4ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f815b9d0 con 0x7f07f80ad970 2023-10-16T13:51:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.858+0000 7f07fe4ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f815bcb0 con 0x7f07f806c040 2023-10-16T13:51:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.858+0000 7f07fe4ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f815b7d0 con 0x7f07f815b320 2023-10-16T13:51:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.858+0000 7f07fccfc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f07f80ad970 0x7f07f80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49010/0 (socket says 172.21.15.60:49010) 2023-10-16T13:51:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.858+0000 7f07fccfc640 1 -- 172.21.15.60:0/832146838 learned_addr learned my addr 172.21.15.60:0/832146838 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.858+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 0x7f07f80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221276254 0 0) 0x7f07f815b9d0 con 0x7f07f80ad970 2023-10-16T13:51:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e4003520 con 0x7f07f80ad970 2023-10-16T13:51:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 878796629 0 0) 0x7f07e4003520 con 0x7f07f80ad970 2023-10-16T13:51:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 msgr2=0x7f07f80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 0x7f07f80cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 >> v1:172.21.15.144:6789/0 conn(0x7f07f806c040 legacy=0x7f07f806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f815bf90 con 0x7f07f80ad970 2023-10-16T13:51:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f07e8001670 con 0x7f07f80ad970 2023-10-16T13:51:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 >> v1:172.21.15.60:6789/0 conn(0x7f07f80ad970 legacy=0x7f07f80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07e40039a0 0x7f07e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 0x7f07e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e4004d00 0x7f07e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f815b7d0 con 0x7f07e4004d00 2023-10-16T13:51:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e400ba60 con 0x7f07e40039a0 2023-10-16T13:51:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e400bc20 con 0x7f07f815b320 2023-10-16T13:51:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.859+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e4004d00 0x7f07e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fd4fd640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07e40039a0 0x7f07e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fccfc640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 0x7f07e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fdcfe640 1 -- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 msgr2=0x7f07e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07f815b320 0x7f07e40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fdcfe640 1 -- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07e40039a0 msgr2=0x7f07e4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07e40039a0 0x7f07e4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fdcfe640 1 -- 172.21.15.60:0/832146838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e4003520 con 0x7f07e4004d00 2023-10-16T13:51:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.860+0000 7f07fd4fd640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07e40039a0 0x7f07e4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.861+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e4004d00 0x7f07e4005180 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009a90 tx=0x7f07e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cb7ab0ac9cd1d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.861+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f002c680 con 0x7f07e4004d00 2023-10-16T13:51:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.861+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f002c7e0 con 0x7f07e4004d00 2023-10-16T13:51:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.861+0000 7f07effff640 1 -- 172.21.15.60:0/832146838 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f002cb70 con 0x7f07e4004d00 2023-10-16T13:51:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fe4ff640 1 -- 172.21.15.60:0/832146838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e4004d00 msgr2=0x7f07e4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/832146838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e4004d00 0x7f07e4005180 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009a90 tx=0x7f07e40057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fdcfe640 1 -- 172.21.15.60:0/832146838 reap_dead start 2023-10-16T13:51:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fe4ff640 1 -- 172.21.15.60:0/832146838 shutdown_connections 2023-10-16T13:51:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fe4ff640 1 -- 172.21.15.60:0/832146838 >> 172.21.15.60:0/832146838 conn(0x7f07f80ce6a0 msgr2=0x7f07f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fe4ff640 1 -- 172.21.15.60:0/832146838 shutdown_connections 2023-10-16T13:51:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.862+0000 7f07fe4ff640 1 -- 172.21.15.60:0/832146838 wait complete. 2023-10-16T13:51:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.864+0000 7f07fe4ff640 1 Processor -- start 2023-10-16T13:51:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.864+0000 7f07fe4ff640 1 -- start start 2023-10-16T13:51:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fe4ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07e4004d00 0x7f07f8159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fe4ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 0x7f07f81f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fe4ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07f80acef0 0x7f07f81f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fe4ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8158ad0 con 0x7f07f80aca70 2023-10-16T13:51:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fe4ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80c9fa0 con 0x7f07f80acef0 2023-10-16T13:51:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fe4ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8158230 con 0x7f07e4004d00 2023-10-16T13:51:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 0x7f07f81f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 0x7f07f81f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51040/0 (socket says 172.21.15.60:51040) 2023-10-16T13:51:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 -- 172.21.15.60:0/3999777445 learned_addr learned my addr 172.21.15.60:0/3999777445 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07f80acef0 0x7f07f81f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fd4fd640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07e4004d00 0x7f07f8159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 -- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07e4004d00 msgr2=0x7f07f8159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07e4004d00 0x7f07f8159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 -- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07f80acef0 msgr2=0x7f07f81f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07f80acef0 0x7f07f81f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fccfc640 1 -- 172.21.15.60:0/3999777445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f8158510 con 0x7f07f80aca70 2023-10-16T13:51:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fdcfe640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07f80acef0 0x7f07f81f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.865+0000 7f07fd4fd640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07e4004d00 0x7f07f8159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.866+0000 7f07fccfc640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 0x7f07f81f2710 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f07e8001050 tx=0x7f07e800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:07.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.866+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e800c8d0 con 0x7f07f80aca70 2023-10-16T13:51:07.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.866+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e800ca50 con 0x7f07f80aca70 2023-10-16T13:51:07.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.866+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e800ce00 con 0x7f07f80aca70 2023-10-16T13:51:07.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.866+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f81587f0 con 0x7f07f80aca70 2023-10-16T13:51:07.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.866+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f815a3e0 con 0x7f07f80aca70 2023-10-16T13:51:07.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.867+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f80ca130 con 0x7f07f80aca70 2023-10-16T13:51:07.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.868+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f07e801d460 con 0x7f07f80aca70 2023-10-16T13:51:07.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.869+0000 7f07edffb640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 0x7f07dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:07.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.869+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f07e80899f0 con 0x7f07f80aca70 2023-10-16T13:51:07.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.869+0000 7f07fd4fd640 1 -- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 msgr2=0x7f07dc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:07.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.869+0000 7f07fd4fd640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 0x7f07dc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:07.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:07.873+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07e8054820 con 0x7f07f80aca70 2023-10-16T13:51:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.070+0000 7f07fd4fd640 1 -- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 msgr2=0x7f07dc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.070+0000 7f07fd4fd640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 0x7f07dc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:08.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.108+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f804e9b0 con 0x7f07f80aca70 2023-10-16T13:51:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.109+0000 7f07edffb640 1 -- 172.21.15.60:0/3999777445 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f07e8057e40 con 0x7f07f80aca70 2023-10-16T13:51:08.111 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:08.112 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:50:46.866966+0000","modified":"2023-10-16T13:50:49.938566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/691868089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":691868089},{"type":"v1","addr":"172.21.15.60:6834","nonce":691868089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:51:08.112 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 87 2023-10-16T13:51:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 msgr2=0x7f07dc06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 0x7f07dc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 msgr2=0x7f07f81f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 0x7f07f81f2710 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f07e8001050 tx=0x7f07e800ed50 comp rx=0 tx=0).stop 2023-10-16T13:51:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 shutdown_connections 2023-10-16T13:51:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f07dc06af90 0x7f07dc06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07f80acef0 0x7f07f81f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f80aca70 0x7f07f81f2710 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 --2- 172.21.15.60:0/3999777445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07e4004d00 0x7f07f8159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 >> 172.21.15.60:0/3999777445 conn(0x7f07f80ce6a0 msgr2=0x7f07f80cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 shutdown_connections 2023-10-16T13:51:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.112+0000 7f07fe4ff640 1 -- 172.21.15.60:0/3999777445 wait complete. 2023-10-16T13:51:08.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.315+0000 7f4e3313a640 1 Processor -- start 2023-10-16T13:51:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.316+0000 7f4e3313a640 1 -- start start 2023-10-16T13:51:08.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e3313a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 0x7f4e2c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e3313a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4e2c0ce390 0x7f4e2c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e3313a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c15ed60 con 0x7f4e2c15ae80 2023-10-16T13:51:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e3313a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c06b810 con 0x7f4e2c0ce390 2023-10-16T13:51:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e3313a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c0707e0 con 0x7f4e2c15aaa0 2023-10-16T13:51:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e2bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4e2c15ae80 0x7f4e2c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49026/0 (socket says 172.21.15.60:49026) 2023-10-16T13:51:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.317+0000 7f4e2bfff640 1 -- 172.21.15.60:0/65732048 learned_addr learned my addr 172.21.15.60:0/65732048 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:08.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e30ed6640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 0x7f4e2c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e316d7640 1 --2- 172.21.15.60:0/65732048 >> v2:172.21.15.144:3300/0 conn(0x7f4e2c0ce390 0x7f4e2c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263872149 0 0) 0x7f4e2c15ed60 con 0x7f4e2c15ae80 2023-10-16T13:51:08.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c003520 con 0x7f4e2c15ae80 2023-10-16T13:51:08.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129467804 0 0) 0x7f4e1c003520 con 0x7f4e2c15ae80 2023-10-16T13:51:08.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 msgr2=0x7f4e2c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:08.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 0x7f4e2c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 >> v2:172.21.15.144:3300/0 conn(0x7f4e2c0ce390 msgr2=0x7f4e2c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:08.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 --2- 172.21.15.60:0/65732048 >> v2:172.21.15.144:3300/0 conn(0x7f4e2c0ce390 0x7f4e2c070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c15ef90 con 0x7f4e2c15ae80 2023-10-16T13:51:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4e18001670 con 0x7f4e2c15ae80 2023-10-16T13:51:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 >> v1:172.21.15.60:6789/0 conn(0x7f4e2c15ae80 legacy=0x7f4e2c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e1c003970 0x7f4e1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.318+0000 7f4e2b7fe640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2b7fe640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 0x7f4e1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c06b810 con 0x7f4e1c0047c0 2023-10-16T13:51:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c0707e0 con 0x7f4e1c003970 2023-10-16T13:51:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c00b9d0 con 0x7f4e2c15aaa0 2023-10-16T13:51:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e316d7640 1 --2- 172.21.15.60:0/65732048 >> v2:172.21.15.144:3300/0 conn(0x7f4e2c0ce390 0x7f4e2c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2bfff640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e30ed6640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e1c003970 0x7f4e1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2bfff640 1 -- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 msgr2=0x7f4e1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2bfff640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15aaa0 0x7f4e1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2bfff640 1 -- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e1c003970 msgr2=0x7f4e1c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2bfff640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e1c003970 0x7f4e1c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.319+0000 7f4e2bfff640 1 -- 172.21.15.60:0/65732048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c003520 con 0x7f4e1c0047c0 2023-10-16T13:51:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.320+0000 7f4e2bfff640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e1c004c40 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800c110 tx=0x7f4e1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31abdb994d108f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.321+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18005ad0 con 0x7f4e1c0047c0 2023-10-16T13:51:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.321+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e18005c50 con 0x7f4e1c0047c0 2023-10-16T13:51:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.321+0000 7f4e2b7fe640 1 -- 172.21.15.60:0/65732048 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1802c830 con 0x7f4e1c0047c0 2023-10-16T13:51:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.321+0000 7f4e3313a640 1 -- 172.21.15.60:0/65732048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 msgr2=0x7f4e1c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.321+0000 7f4e3313a640 1 --2- 172.21.15.60:0/65732048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e1c004c40 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800c110 tx=0x7f4e1c0057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.321+0000 7f4e316d7640 1 -- 172.21.15.60:0/65732048 reap_dead start 2023-10-16T13:51:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.322+0000 7f4e3313a640 1 -- 172.21.15.60:0/65732048 shutdown_connections 2023-10-16T13:51:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.322+0000 7f4e3313a640 1 -- 172.21.15.60:0/65732048 >> 172.21.15.60:0/65732048 conn(0x7f4e2c0cb410 msgr2=0x7f4e2c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.322+0000 7f4e3313a640 1 -- 172.21.15.60:0/65732048 shutdown_connections 2023-10-16T13:51:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.322+0000 7f4e3313a640 1 -- 172.21.15.60:0/65732048 wait complete. 2023-10-16T13:51:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.324+0000 7f4e3313a640 1 Processor -- start 2023-10-16T13:51:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.324+0000 7f4e3313a640 1 -- start start 2023-10-16T13:51:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e3313a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e2c1eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e3313a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2c1ef1b0 0x7f4e2c0aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e3313a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15d190 0x7f4e2c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e3313a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c15b5b0 con 0x7f4e1c0047c0 2023-10-16T13:51:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e3313a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c1eefb0 con 0x7f4e2c1ef1b0 2023-10-16T13:51:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e3313a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c15e360 con 0x7f4e2c15d190 2023-10-16T13:51:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e2c1eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e2c1eea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51058/0 (socket says 172.21.15.60:51058) 2023-10-16T13:51:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 -- 172.21.15.60:0/3188390883 learned_addr learned my addr 172.21.15.60:0/3188390883 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 -- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15d190 msgr2=0x7f4e2c0af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15d190 0x7f4e2c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 -- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2c1ef1b0 msgr2=0x7f4e2c0aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2c1ef1b0 0x7f4e2c0aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.325+0000 7f4e30ed6640 1 -- 172.21.15.60:0/3188390883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0acfc0 con 0x7f4e1c0047c0 2023-10-16T13:51:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.326+0000 7f4e30ed6640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e2c1eea70 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f4e200115b0 tx=0x7f4e20011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.326+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20015870 con 0x7f4e1c0047c0 2023-10-16T13:51:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.326+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e200159d0 con 0x7f4e1c0047c0 2023-10-16T13:51:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.326+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c15d750 con 0x7f4e1c0047c0 2023-10-16T13:51:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.326+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20015d40 con 0x7f4e1c0047c0 2023-10-16T13:51:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.326+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c15dd10 con 0x7f4e1c0047c0 2023-10-16T13:51:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.327+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c04e160 con 0x7f4e1c0047c0 2023-10-16T13:51:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.328+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f4e20013400 con 0x7f4e1c0047c0 2023-10-16T13:51:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.329+0000 7f4e297fa640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.329+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f4e20089db0 con 0x7f4e1c0047c0 2023-10-16T13:51:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.329+0000 7f4e2bfff640 1 -- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 msgr2=0x7f4e1006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.329+0000 7f4e2bfff640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:08.338 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.333+0000 7f140fa13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:51:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.339+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e2008e070 con 0x7f4e1c0047c0 2023-10-16T13:51:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.530+0000 7f4e2bfff640 1 -- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 msgr2=0x7f4e1006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.530+0000 7f4e2bfff640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:08.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.537+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e2c0cdc60 con 0x7f4e1c0047c0 2023-10-16T13:51:08.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.539+0000 7f4e297fa640 1 -- 172.21.15.60:0/3188390883 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17076 (secure 0 0 0) 0x7f4e20054ad0 con 0x7f4e1c0047c0 2023-10-16T13:51:08.542 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:08.542 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":184,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:01.597787+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2455238630":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6833/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6832/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3372584586":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1775356785":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3925300811":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1821662703":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3430745922":"2023-10-17T13:51:01.597577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 msgr2=0x7f4e1006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 msgr2=0x7f4e2c1eea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e2c1eea70 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f4e200115b0 tx=0x7f4e20011a70 comp rx=0 tx=0).stop 2023-10-16T13:51:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 shutdown_connections 2023-10-16T13:51:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e2c15d190 0x7f4e2c0af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e2c1ef1b0 0x7f4e2c0aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 --2- 172.21.15.60:0/3188390883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e1c0047c0 0x7f4e2c1eea70 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 >> 172.21.15.60:0/3188390883 conn(0x7f4e2c0cb410 msgr2=0x7f4e2c06fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 shutdown_connections 2023-10-16T13:51:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.542+0000 7f4e3313a640 1 -- 172.21.15.60:0/3188390883 wait complete. 2023-10-16T13:51:08.549 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:08.547+0000 7fcf7fa12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:51:08.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:50:44.806197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:50:45.907641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:51:08.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:51:08.632 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.629+0000 7f140fa13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:51:08.690 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.688+0000 7f140fa13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:51:08.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.770+0000 7fee3a178640 1 Processor -- start 2023-10-16T13:51:08.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.771+0000 7fee3a178640 1 -- start start 2023-10-16T13:51:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.772+0000 7fee3a178640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee34069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.772+0000 7fee3a178640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fee340caac0 0x7fee3406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.772+0000 7fee3a178640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340cb030 con 0x7fee340c9ca0 2023-10-16T13:51:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.772+0000 7fee3a178640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee340c91c0 con 0x7fee340caac0 2023-10-16T13:51:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.772+0000 7fee3a178640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee340c94a0 con 0x7fee3415ad70 2023-10-16T13:51:08.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.773+0000 7fee337fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee34069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.773+0000 7fee337fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee34069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:42488/0 (socket says 172.21.15.60:42488) 2023-10-16T13:51:08.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.773+0000 7fee337fe640 1 -- 172.21.15.60:0/2283433876 learned_addr learned my addr 172.21.15.60:0/2283433876 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:08.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.773+0000 7fee33fff640 1 --2- 172.21.15.60:0/2283433876 >> v2:172.21.15.144:3300/0 conn(0x7fee340caac0 0x7fee3406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.774+0000 7fee337fe640 1 -- 172.21.15.60:0/2283433876 >> v2:172.21.15.144:3300/0 conn(0x7fee340caac0 msgr2=0x7fee3406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:08.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.774+0000 7fee337fe640 1 --2- 172.21.15.60:0/2283433876 >> v2:172.21.15.144:3300/0 conn(0x7fee340caac0 0x7fee3406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.774+0000 7fee337fe640 1 -- 172.21.15.60:0/2283433876 >> v1:172.21.15.144:6789/0 conn(0x7fee340c9ca0 legacy=0x7fee340ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.774+0000 7fee337fe640 1 -- 172.21.15.60:0/2283433876 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340c9780 con 0x7fee3415ad70 2023-10-16T13:51:08.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.774+0000 7fee327fc640 1 -- 172.21.15.60:0/2283433876 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370476558 0 0) 0x7fee340cb030 con 0x7fee340c9ca0 2023-10-16T13:51:08.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.775+0000 7fee337fe640 1 --2- 172.21.15.60:0/2283433876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee34069da0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fee24009870 tx=0x7fee2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d3ca64a3ca1f9be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:08.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee327fc640 1 -- 172.21.15.60:0/2283433876 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24030840 con 0x7fee3415ad70 2023-10-16T13:51:08.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee327fc640 1 -- 172.21.15.60:0/2283433876 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee240309a0 con 0x7fee3415ad70 2023-10-16T13:51:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee327fc640 1 -- 172.21.15.60:0/2283433876 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24030d30 con 0x7fee3415ad70 2023-10-16T13:51:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 -- 172.21.15.60:0/2283433876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 msgr2=0x7fee34069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 --2- 172.21.15.60:0/2283433876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee34069da0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fee24009870 tx=0x7fee2402b980 comp rx=0 tx=0).stop 2023-10-16T13:51:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 -- 172.21.15.60:0/2283433876 shutdown_connections 2023-10-16T13:51:08.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 --2- 172.21.15.60:0/2283433876 >> v2:172.21.15.144:3300/0 conn(0x7fee340caac0 0x7fee3406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 --2- 172.21.15.60:0/2283433876 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee34069da0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 -- 172.21.15.60:0/2283433876 >> 172.21.15.60:0/2283433876 conn(0x7fee340ce9b0 msgr2=0x7fee340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 -- 172.21.15.60:0/2283433876 shutdown_connections 2023-10-16T13:51:08.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.776+0000 7fee3a178640 1 -- 172.21.15.60:0/2283433876 wait complete. 2023-10-16T13:51:08.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.778+0000 7fee3a178640 1 Processor -- start 2023-10-16T13:51:08.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.778+0000 7fee3a178640 1 -- start start 2023-10-16T13:51:08.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.779+0000 7fee3a178640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 0x7fee34149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.779+0000 7fee3a178640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee340caac0 0x7fee340c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.779+0000 7fee3a178640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee340c2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.779+0000 7fee3a178640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3414a420 con 0x7fee340c9ca0 2023-10-16T13:51:08.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.779+0000 7fee3a178640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee340ac1e0 con 0x7fee340caac0 2023-10-16T13:51:08.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.779+0000 7fee3a178640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee340ac4c0 con 0x7fee3415ad70 2023-10-16T13:51:08.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 0x7fee34149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee32ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee340caac0 0x7fee340c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 0x7fee34149ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51072/0 (socket says 172.21.15.60:51072) 2023-10-16T13:51:08.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 -- 172.21.15.60:0/4162624786 learned_addr learned my addr 172.21.15.60:0/4162624786 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:08.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee33fff640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee340c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:08.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 -- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 msgr2=0x7fee340c2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee340c2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 -- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee340caac0 msgr2=0x7fee340c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:08.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee340caac0 0x7fee340c1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 -- 172.21.15.60:0/4162624786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240094d0 con 0x7fee340c9ca0 2023-10-16T13:51:08.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.780+0000 7fee337fe640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 0x7fee34149ee0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fee2402b9c0 tx=0x7fee240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:08.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.781+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2403e070 con 0x7fee340c9ca0 2023-10-16T13:51:08.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.781+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee24039a10 con 0x7fee340c9ca0 2023-10-16T13:51:08.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.781+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24039de0 con 0x7fee340c9ca0 2023-10-16T13:51:08.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.781+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee340ac7a0 con 0x7fee340c9ca0 2023-10-16T13:51:08.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.781+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee340acd60 con 0x7fee340c9ca0 2023-10-16T13:51:08.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.782+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3404e160 con 0x7fee340c9ca0 2023-10-16T13:51:08.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.783+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 93) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fee2404b400 con 0x7fee340c9ca0 2023-10-16T13:51:08.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.784+0000 7fee2bfff640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fee1406afc0 0x7fee1406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:08.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.784+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fee240abf30 con 0x7fee340c9ca0 2023-10-16T13:51:08.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.785+0000 7fee32ffd640 1 -- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fee1406afc0 msgr2=0x7fee1406d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1295809784 2023-10-16T13:51:08.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.785+0000 7fee32ffd640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fee1406afc0 0x7fee1406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:08.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.789+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee24076bc0 con 0x7fee340c9ca0 2023-10-16T13:51:08.805 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.802+0000 7f140fa13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:51:08.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.820+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 94) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fee24076f90 con 0x7fee340c9ca0 2023-10-16T13:51:08.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.821+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fee1406afc0 msgr2=0x7fee1406d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:08.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.821+0000 7fee2bfff640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fee1406afc0 0x7fee1406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:08.876 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:08.874+0000 7fcf7fa12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:51:08.901 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.898+0000 7f1394a14640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.901 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.898+0000 7f1394a14640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.901 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.898+0000 7f1394a14640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.901 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.898+0000 7f1394a14640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.902 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.898+0000 7f1394a14640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.902 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.899+0000 7f139020b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.903 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.899+0000 7f139020b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.903 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.899+0000 7f139020b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.903 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.899+0000 7f139020b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.903 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:08.899+0000 7f139020b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:08.939 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:08.937+0000 7fcf7fa12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:51:08.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.993+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee3415c920 con 0x7fee340c9ca0 2023-10-16T13:51:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.995+0000 7fee2bfff640 1 -- 172.21.15.60:0/4162624786 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v185) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee24090940 con 0x7fee340c9ca0 2023-10-16T13:51:08.997 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:51:08.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 msgr2=0x7fee34149ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 0x7fee34149ee0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fee2402b9c0 tx=0x7fee240046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 shutdown_connections 2023-10-16T13:51:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:6832/1295809784,v1:172.21.15.144:6833/1295809784] conn(0x7fee1406afc0 0x7fee1406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee3415ad70 0x7fee340c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee340caac0 0x7fee340c1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 --2- 172.21.15.60:0/4162624786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee340c9ca0 0x7fee34149ee0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 >> 172.21.15.60:0/4162624786 conn(0x7fee340ce9b0 msgr2=0x7fee340c5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:09.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.997+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 shutdown_connections 2023-10-16T13:51:09.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:08.998+0000 7fee3a178640 1 -- 172.21.15.60:0/4162624786 wait complete. 2023-10-16T13:51:09.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:51:09.061 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:09.058+0000 7fcf7fa12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:51:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.218+0000 7f7894d16640 1 Processor -- start 2023-10-16T13:51:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.218+0000 7f7894d16640 1 -- start start 2023-10-16T13:51:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f7894d16640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890159950 0x7f789015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f7894d16640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7890070450 0x7f78900708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f7894d16640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f789006c100 0x7f789006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f7894d16640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789015e490 con 0x7f789006c100 2023-10-16T13:51:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f7894d16640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7890070df0 con 0x7f7890070450 2023-10-16T13:51:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f7894d16640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789015aef0 con 0x7f7890159950 2023-10-16T13:51:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f788ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f789006c100 0x7f789006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.220+0000 7f788ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f789006c100 0x7f789006c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40288/0 (socket says 172.21.15.60:40288) 2023-10-16T13:51:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788ffff640 1 -- 172.21.15.60:0/1523615486 learned_addr learned my addr 172.21.15.60:0/1523615486 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f7887fff640 1 --2- 172.21.15.60:0/1523615486 >> v2:172.21.15.144:3300/0 conn(0x7f7890070450 0x7f78900708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788f7fe640 1 --2- 172.21.15.60:0/1523615486 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890159950 0x7f789015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788ffff640 1 -- 172.21.15.60:0/1523615486 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890159950 msgr2=0x7f789015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788ffff640 1 --2- 172.21.15.60:0/1523615486 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890159950 0x7f789015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788ffff640 1 -- 172.21.15.60:0/1523615486 >> v2:172.21.15.144:3300/0 conn(0x7f7890070450 msgr2=0x7f78900708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788ffff640 1 --2- 172.21.15.60:0/1523615486 >> v2:172.21.15.144:3300/0 conn(0x7f7890070450 0x7f78900708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.221+0000 7f788ffff640 1 -- 172.21.15.60:0/1523615486 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789015d910 con 0x7f789006c100 2023-10-16T13:51:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.222+0000 7f788ffff640 1 --2- 172.21.15.60:0/1523615486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f789006c100 0x7f789006c570 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f78900719e0 tx=0x7f788802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c7e5f15674cd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.222+0000 7f788effd640 1 -- 172.21.15.60:0/1523615486 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788802c9e0 con 0x7f789006c100 2023-10-16T13:51:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.222+0000 7f788effd640 1 -- 172.21.15.60:0/1523615486 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788802cb40 con 0x7f789006c100 2023-10-16T13:51:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.222+0000 7f788effd640 1 -- 172.21.15.60:0/1523615486 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78880343f0 con 0x7f789006c100 2023-10-16T13:51:09.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 -- 172.21.15.60:0/1523615486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f789006c100 msgr2=0x7f789006c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 --2- 172.21.15.60:0/1523615486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f789006c100 0x7f789006c570 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f78900719e0 tx=0x7f788802be40 comp rx=0 tx=0).stop 2023-10-16T13:51:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 -- 172.21.15.60:0/1523615486 shutdown_connections 2023-10-16T13:51:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 --2- 172.21.15.60:0/1523615486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f789006c100 0x7f789006c570 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 --2- 172.21.15.60:0/1523615486 >> v2:172.21.15.144:3300/0 conn(0x7f7890070450 0x7f78900708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 --2- 172.21.15.60:0/1523615486 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890159950 0x7f789015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 -- 172.21.15.60:0/1523615486 >> 172.21.15.60:0/1523615486 conn(0x7f78900ce6a0 msgr2=0x7f78900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:09.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 -- 172.21.15.60:0/1523615486 shutdown_connections 2023-10-16T13:51:09.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.223+0000 7f7894d16640 1 -- 172.21.15.60:0/1523615486 wait complete. 2023-10-16T13:51:09.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.225+0000 7f7894d16640 1 Processor -- start 2023-10-16T13:51:09.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.225+0000 7f7894d16640 1 -- start start 2023-10-16T13:51:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f7894d16640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f789006c100 0x7f78901585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f7894d16640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890070450 0x7f78900c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f7894d16640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 0x7f78901f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f7894d16640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78901f3840 con 0x7f7890159950 2023-10-16T13:51:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f7894d16640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789015b5a0 con 0x7f789006c100 2023-10-16T13:51:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f7894d16640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7890158b00 con 0x7f7890070450 2023-10-16T13:51:09.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.226+0000 7f788ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 0x7f78901f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 0x7f78901f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40298/0 (socket says 172.21.15.60:40298) 2023-10-16T13:51:09.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 -- 172.21.15.60:0/4152280272 learned_addr learned my addr 172.21.15.60:0/4152280272 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:09.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f7887fff640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890070450 0x7f78900c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788f7fe640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f789006c100 0x7f78901585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 -- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890070450 msgr2=0x7f78900c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890070450 0x7f78900c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 -- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f789006c100 msgr2=0x7f78901585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f789006c100 0x7f78901585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 -- 172.21.15.60:0/4152280272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888009b30 con 0x7f7890159950 2023-10-16T13:51:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.227+0000 7f788ffff640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 0x7f78901f2830 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7888009e50 tx=0x7f7888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.228+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888009510 con 0x7f7890159950 2023-10-16T13:51:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.228+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788803e450 con 0x7f7890159950 2023-10-16T13:51:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.228+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788803e800 con 0x7f7890159950 2023-10-16T13:51:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.228+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78901f3280 con 0x7f7890159950 2023-10-16T13:51:09.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.228+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789015b230 con 0x7f7890159950 2023-10-16T13:51:09.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.229+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789015b090 con 0x7f7890159950 2023-10-16T13:51:09.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.230+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 94) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f788803e960 con 0x7f7890159950 2023-10-16T13:51:09.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.231+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f78880ac1a0 con 0x7f7890159950 2023-10-16T13:51:09.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.235+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f789015b090 con 0x7f7890159950 2023-10-16T13:51:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.433+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f789015b090 con 0x7f7890159950 2023-10-16T13:51:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.434+0000 7f788cff9640 1 -- 172.21.15.60:0/4152280272 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v185) v1 ==== 104+0+11 (secure 0 0 0) 0x7f789015b090 con 0x7f7890159950 2023-10-16T13:51:09.437 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T13:51:09.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 msgr2=0x7f78901f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 0x7f78901f2830 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7888009e50 tx=0x7f7888004a40 comp rx=0 tx=0).stop 2023-10-16T13:51:09.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 shutdown_connections 2023-10-16T13:51:09.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7890070450 0x7f78900c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7890159950 0x7f78901f2830 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 --2- 172.21.15.60:0/4152280272 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f789006c100 0x7f78901585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 >> 172.21.15.60:0/4152280272 conn(0x7f78900ce6a0 msgr2=0x7f78900cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 shutdown_connections 2023-10-16T13:51:09.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.437+0000 7f7894d16640 1 -- 172.21.15.60:0/4152280272 wait complete. 2023-10-16T13:51:09.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:51:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.735+0000 7f7690c0d640 1 Processor -- start 2023-10-16T13:51:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.736+0000 7f7690c0d640 1 -- start start 2023-10-16T13:51:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.737+0000 7f7690c0d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c0ca0a0 0x7f768c06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.737+0000 7f7690c0d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c15d910 con 0x7f768c159950 2023-10-16T13:51:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.737+0000 7f7690c0d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c15dbf0 con 0x7f768c070b60 2023-10-16T13:51:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.737+0000 7f7690c0d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c15e490 con 0x7f768c0ca0a0 2023-10-16T13:51:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.738+0000 7f768b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f768c159950 0x7f768c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56400/0 (socket says 172.21.15.60:56400) 2023-10-16T13:51:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.738+0000 7f768b7fe640 1 -- 172.21.15.60:0/1316615022 learned_addr learned my addr 172.21.15.60:0/1316615022 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:09.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.738+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735687832 0 0) 0x7f768c15d910 con 0x7f768c159950 2023-10-16T13:51:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.738+0000 7f768bfff640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c0ca0a0 0x7f768c06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.738+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7678003520 con 0x7f768c159950 2023-10-16T13:51:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4243216031 0 0) 0x7f7678003520 con 0x7f768c159950 2023-10-16T13:51:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c0ca0a0 msgr2=0x7f768c06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:09.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c0ca0a0 0x7f768c06c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 >> v1:172.21.15.144:6789/0 conn(0x7f768c070b60 legacy=0x7f768c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c15ded0 con 0x7f768c159950 2023-10-16T13:51:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f767c001670 con 0x7f768c159950 2023-10-16T13:51:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 >> v1:172.21.15.60:6789/0 conn(0x7f768c159950 legacy=0x7f768c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76780039a0 0x7f7678003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780047c0 0x7f7678004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678005250 0x7f767800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c15e490 con 0x7f76780039a0 2023-10-16T13:51:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767800c120 con 0x7f76780047c0 2023-10-16T13:51:09.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767800e4f0 con 0x7f7678005250 2023-10-16T13:51:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768bfff640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c0ca0a0 0x7f768c06c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76780039a0 0x7f7678003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768b7fe640 1 -- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678005250 msgr2=0x7f767800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7678005250 0x7f767800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768b7fe640 1 -- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780047c0 msgr2=0x7f7678004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:09.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f7683fff640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780047c0 0x7f7678004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.739+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780047c0 0x7f7678004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.740+0000 7f768b7fe640 1 -- 172.21.15.60:0/1316615022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678003520 con 0x7f76780039a0 2023-10-16T13:51:09.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76780039a0 0x7f7678003dc0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c0e0 tx=0x7f76780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a850709950e19110 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:09.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c005ad0 con 0x7f76780039a0 2023-10-16T13:51:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c005c30 con 0x7f76780039a0 2023-10-16T13:51:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f768affd640 1 -- 172.21.15.60:0/1316615022 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c02c810 con 0x7f76780039a0 2023-10-16T13:51:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f7690c0d640 1 -- 172.21.15.60:0/1316615022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76780039a0 msgr2=0x7f7678003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1316615022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76780039a0 0x7f7678003dc0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c0e0 tx=0x7f76780057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f768bfff640 1 -- 172.21.15.60:0/1316615022 reap_dead start 2023-10-16T13:51:09.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f7690c0d640 1 -- 172.21.15.60:0/1316615022 shutdown_connections 2023-10-16T13:51:09.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.741+0000 7f7690c0d640 1 -- 172.21.15.60:0/1316615022 >> 172.21.15.60:0/1316615022 conn(0x7f768c0ce6a0 msgr2=0x7f768c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:09.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.742+0000 7f7690c0d640 1 -- 172.21.15.60:0/1316615022 shutdown_connections 2023-10-16T13:51:09.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.742+0000 7f7690c0d640 1 -- 172.21.15.60:0/1316615022 wait complete. 2023-10-16T13:51:09.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.743+0000 7f7690c0d640 1 Processor -- start 2023-10-16T13:51:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.743+0000 7f7690c0d640 1 -- start start 2023-10-16T13:51:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.744+0000 7f7690c0d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780039a0 0x7f768c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.744+0000 7f7690c0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 0x7f768c158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.744+0000 7f7690c0d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c14d550 0x7f768c14d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.744+0000 7f7690c0d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c154f70 con 0x7f768c0c2a30 2023-10-16T13:51:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.744+0000 7f7690c0d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0c3110 con 0x7f76780039a0 2023-10-16T13:51:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.744+0000 7f7690c0d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0c2e70 con 0x7f768c14d550 2023-10-16T13:51:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 0x7f768c158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 0x7f768c158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40316/0 (socket says 172.21.15.60:40316) 2023-10-16T13:51:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 -- 172.21.15.60:0/1322621237 learned_addr learned my addr 172.21.15.60:0/1322621237 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780039a0 0x7f768c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 -- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c14d550 msgr2=0x7f768c14d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c14d550 0x7f768c14d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 -- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780039a0 msgr2=0x7f768c155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780039a0 0x7f768c155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 -- 172.21.15.60:0/1322621237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c155890 con 0x7f768c0c2a30 2023-10-16T13:51:09.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.745+0000 7f7683fff640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 0x7f768c158370 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f767400d560 tx=0x7f767400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:09.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.746+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674005480 con 0x7f768c0c2a30 2023-10-16T13:51:09.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.746+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7674003610 con 0x7f768c0c2a30 2023-10-16T13:51:09.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.746+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674003980 con 0x7f768c0c2a30 2023-10-16T13:51:09.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.746+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c155b70 con 0x7f768c0c2a30 2023-10-16T13:51:09.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.746+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c156130 con 0x7f768c0c2a30 2023-10-16T13:51:09.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.747+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c1562f0 con 0x7f768c0c2a30 2023-10-16T13:51:09.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.748+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 94) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7674003ae0 con 0x7f768c0c2a30 2023-10-16T13:51:09.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.749+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7674089230 con 0x7f768c0c2a30 2023-10-16T13:51:09.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.754+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7674054480 con 0x7f768c0c2a30 2023-10-16T13:51:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.829+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 95) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7674053f30 con 0x7f768c0c2a30 2023-10-16T13:51:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.830+0000 7f7688ff9640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f766c0705c0 0x7f766c0729b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.831+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f766c0705c0 0x7f766c0729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.836+0000 7f768b7fe640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f766c0705c0 0x7f766c0729b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f767c034000 tx=0x7f767c009030 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:09.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.955+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f768c06bf20 con 0x7f768c0c2a30 2023-10-16T13:51:09.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.956+0000 7f7688ff9640 1 -- 172.21.15.60:0/1322621237 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v185) v1 ==== 100+0+11 (secure 0 0 0) 0x7f76740875c0 con 0x7f768c0c2a30 2023-10-16T13:51:09.958 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T13:51:09.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.958+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f766c0705c0 msgr2=0x7f766c0729b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.958+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f766c0705c0 0x7f766c0729b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f767c034000 tx=0x7f767c009030 comp rx=0 tx=0).stop 2023-10-16T13:51:09.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 msgr2=0x7f768c158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 0x7f768c158370 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f767400d560 tx=0x7f767400da20 comp rx=0 tx=0).stop 2023-10-16T13:51:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 shutdown_connections 2023-10-16T13:51:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f766c0705c0 0x7f766c0729b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f768c14d550 0x7f768c14d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f768c0c2a30 0x7f768c158370 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 --2- 172.21.15.60:0/1322621237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76780039a0 0x7f768c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:09.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 >> 172.21.15.60:0/1322621237 conn(0x7f768c0ce6a0 msgr2=0x7f768c06f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:09.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 shutdown_connections 2023-10-16T13:51:09.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:09.959+0000 7f7690c0d640 1 -- 172.21.15.60:0/1322621237 wait complete. 2023-10-16T13:51:09.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.199+0000 7f7b27fff640 1 Processor -- start 2023-10-16T13:51:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.199+0000 7f7b27fff640 1 -- start start 2023-10-16T13:51:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b27fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b28159950 0x7f7b2815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28070450 0x7f7b280708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b27fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b2806c100 0x7f7b2806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b27fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2815e490 con 0x7f7b2806c100 2023-10-16T13:51:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b27fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b28070df0 con 0x7f7b28159950 2023-10-16T13:51:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b27fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815aef0 con 0x7f7b28070450 2023-10-16T13:51:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b277fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b2806c100 0x7f7b2806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b277fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b2806c100 0x7f7b2806c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40330/0 (socket says 172.21.15.60:40330) 2023-10-16T13:51:10.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.201+0000 7f7b277fe640 1 -- 172.21.15.60:0/294105106 learned_addr learned my addr 172.21.15.60:0/294105106 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b1ffff640 1 --2- 172.21.15.60:0/294105106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28070450 0x7f7b280708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b26ffd640 1 --2- 172.21.15.60:0/294105106 >> v2:172.21.15.144:3300/0 conn(0x7f7b28159950 0x7f7b2815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b277fe640 1 -- 172.21.15.60:0/294105106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28070450 msgr2=0x7f7b280708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b277fe640 1 --2- 172.21.15.60:0/294105106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28070450 0x7f7b280708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b277fe640 1 -- 172.21.15.60:0/294105106 >> v2:172.21.15.144:3300/0 conn(0x7f7b28159950 msgr2=0x7f7b2815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b277fe640 1 --2- 172.21.15.60:0/294105106 >> v2:172.21.15.144:3300/0 conn(0x7f7b28159950 0x7f7b2815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.202+0000 7f7b277fe640 1 -- 172.21.15.60:0/294105106 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2815d910 con 0x7f7b2806c100 2023-10-16T13:51:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.203+0000 7f7b277fe640 1 --2- 172.21.15.60:0/294105106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2806c100 0x7f7b2806c570 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f7b280ca950 tx=0x7f7b2002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d9474c9e4bc0df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.203+0000 7f7b267fc640 1 -- 172.21.15.60:0/294105106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2002c9e0 con 0x7f7b2806c100 2023-10-16T13:51:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.203+0000 7f7b267fc640 1 -- 172.21.15.60:0/294105106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2002cb40 con 0x7f7b2806c100 2023-10-16T13:51:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.203+0000 7f7b267fc640 1 -- 172.21.15.60:0/294105106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200343f0 con 0x7f7b2806c100 2023-10-16T13:51:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 -- 172.21.15.60:0/294105106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2806c100 msgr2=0x7f7b2806c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 --2- 172.21.15.60:0/294105106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2806c100 0x7f7b2806c570 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f7b280ca950 tx=0x7f7b2002be40 comp rx=0 tx=0).stop 2023-10-16T13:51:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 -- 172.21.15.60:0/294105106 shutdown_connections 2023-10-16T13:51:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 --2- 172.21.15.60:0/294105106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b2806c100 0x7f7b2806c570 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 --2- 172.21.15.60:0/294105106 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28070450 0x7f7b280708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 --2- 172.21.15.60:0/294105106 >> v2:172.21.15.144:3300/0 conn(0x7f7b28159950 0x7f7b2815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 -- 172.21.15.60:0/294105106 >> 172.21.15.60:0/294105106 conn(0x7f7b280ce6a0 msgr2=0x7f7b280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 -- 172.21.15.60:0/294105106 shutdown_connections 2023-10-16T13:51:10.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.204+0000 7f7b27fff640 1 -- 172.21.15.60:0/294105106 wait complete. 2023-10-16T13:51:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.206+0000 7f7b27fff640 1 Processor -- start 2023-10-16T13:51:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.206+0000 7f7b27fff640 1 -- start start 2023-10-16T13:51:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b2806c100 0x7f7b280cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 0x7f7b280cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b27fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28159950 0x7f7b2814bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b27fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2814fc30 con 0x7f7b28070450 2023-10-16T13:51:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b27fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2814d040 con 0x7f7b2806c100 2023-10-16T13:51:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b27fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2814d320 con 0x7f7b28159950 2023-10-16T13:51:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 0x7f7b280cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 0x7f7b280cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40346/0 (socket says 172.21.15.60:40346) 2023-10-16T13:51:10.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.207+0000 7f7b1ffff640 1 -- 172.21.15.60:0/2949980835 learned_addr learned my addr 172.21.15.60:0/2949980835 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:10.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b277fe640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28159950 0x7f7b2814bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b26ffd640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b2806c100 0x7f7b280cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1ffff640 1 -- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28159950 msgr2=0x7f7b2814bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1ffff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28159950 0x7f7b2814bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1ffff640 1 -- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b2806c100 msgr2=0x7f7b280cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1ffff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b2806c100 0x7f7b280cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1ffff640 1 -- 172.21.15.60:0/2949980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20009b30 con 0x7f7b28070450 2023-10-16T13:51:10.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b277fe640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28159950 0x7f7b2814bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:10.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b26ffd640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b2806c100 0x7f7b280cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:10.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1ffff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 0x7f7b280cd960 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000d5f0 tx=0x7f7b1000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:10.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b100125f0 con 0x7f7b28070450 2023-10-16T13:51:10.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b10012750 con 0x7f7b28070450 2023-10-16T13:51:10.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b10012b00 con 0x7f7b28070450 2023-10-16T13:51:10.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.208+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2814d600 con 0x7f7b28070450 2023-10-16T13:51:10.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.209+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2814e9f0 con 0x7f7b28070450 2023-10-16T13:51:10.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.210+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2804e9b0 con 0x7f7b28070450 2023-10-16T13:51:10.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.211+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 95) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7b10012c60 con 0x7f7b28070450 2023-10-16T13:51:10.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.211+0000 7f7b1f7fe640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7b0806ad10 0x7f7b0806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.212+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7b100899f0 con 0x7f7b28070450 2023-10-16T13:51:10.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.212+0000 7f7b26ffd640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7b0806ad10 0x7f7b0806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.213+0000 7f7b26ffd640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7b0806ad10 0x7f7b0806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b18006fd0 tx=0x7f7b18008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:10.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.216+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b10054ae0 con 0x7f7b28070450 2023-10-16T13:51:10.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.454+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b281656d0 con 0x7f7b28070450 2023-10-16T13:51:10.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.455+0000 7f7b1f7fe640 1 -- 172.21.15.60:0/2949980835 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f7b10058100 con 0x7f7b28070450 2023-10-16T13:51:10.457 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:10.457 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:50:46.866966+0000","modified":"2023-10-16T13:50:49.938566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/691868089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":691868089},{"type":"v1","addr":"172.21.15.60:6834","nonce":691868089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:51:10.458 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 87 2023-10-16T13:51:10.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7b0806ad10 msgr2=0x7f7b0806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7b0806ad10 0x7f7b0806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b18006fd0 tx=0x7f7b18008040 comp rx=0 tx=0).stop 2023-10-16T13:51:10.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 msgr2=0x7f7b280cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 0x7f7b280cd960 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000d5f0 tx=0x7f7b1000dab0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 shutdown_connections 2023-10-16T13:51:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7b0806ad10 0x7f7b0806d1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b28159950 0x7f7b2814bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b28070450 0x7f7b280cd960 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 --2- 172.21.15.60:0/2949980835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b2806c100 0x7f7b280cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.457+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 >> 172.21.15.60:0/2949980835 conn(0x7f7b280ce6a0 msgr2=0x7f7b280c9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.458+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 shutdown_connections 2023-10-16T13:51:10.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.458+0000 7f7b27fff640 1 -- 172.21.15.60:0/2949980835 wait complete. 2023-10-16T13:51:10.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:10.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.695+0000 7fc938eaa640 1 Processor -- start 2023-10-16T13:51:10.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.696+0000 7fc938eaa640 1 -- start start 2023-10-16T13:51:10.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.697+0000 7fc938eaa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc93406c040 0x7fc93406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.697+0000 7fc938eaa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc9340cda80 0x7fc93415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.697+0000 7fc938eaa640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93406c960 con 0x7fc9340ad970 2023-10-16T13:51:10.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.697+0000 7fc938eaa640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93415b9d0 con 0x7fc93406c040 2023-10-16T13:51:10.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.697+0000 7fc938eaa640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93415bcb0 con 0x7fc9340cda80 2023-10-16T13:51:10.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc93406c040 0x7fc93406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc93406c040 0x7fc93406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40358/0 (socket says 172.21.15.60:40358) 2023-10-16T13:51:10.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 -- 172.21.15.60:0/3705417749 learned_addr learned my addr 172.21.15.60:0/3705417749 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:10.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc933fff640 1 --2- 172.21.15.60:0/3705417749 >> v2:172.21.15.144:3300/0 conn(0x7fc9340cda80 0x7fc93415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 -- 172.21.15.60:0/3705417749 >> v2:172.21.15.144:3300/0 conn(0x7fc9340cda80 msgr2=0x7fc93415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:10.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 --2- 172.21.15.60:0/3705417749 >> v2:172.21.15.144:3300/0 conn(0x7fc9340cda80 0x7fc93415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 -- 172.21.15.60:0/3705417749 >> v1:172.21.15.144:6789/0 conn(0x7fc9340ad970 legacy=0x7fc93415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:10.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.698+0000 7fc9337fe640 1 -- 172.21.15.60:0/3705417749 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93415bf90 con 0x7fc93406c040 2023-10-16T13:51:10.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.699+0000 7fc9337fe640 1 --2- 172.21.15.60:0/3705417749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93406c040 0x7fc93406c420 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fc9340cf300 tx=0x7fc92402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9803497d9f905b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc932ffd640 1 -- 172.21.15.60:0/3705417749 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924031870 con 0x7fc93406c040 2023-10-16T13:51:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc932ffd640 1 -- 172.21.15.60:0/3705417749 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9240319d0 con 0x7fc93406c040 2023-10-16T13:51:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc932ffd640 1 -- 172.21.15.60:0/3705417749 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924031d60 con 0x7fc93406c040 2023-10-16T13:51:10.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 -- 172.21.15.60:0/3705417749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93406c040 msgr2=0x7fc93406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 --2- 172.21.15.60:0/3705417749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93406c040 0x7fc93406c420 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fc9340cf300 tx=0x7fc92402b910 comp rx=0 tx=0).stop 2023-10-16T13:51:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 -- 172.21.15.60:0/3705417749 shutdown_connections 2023-10-16T13:51:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 --2- 172.21.15.60:0/3705417749 >> v2:172.21.15.144:3300/0 conn(0x7fc9340cda80 0x7fc93415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 --2- 172.21.15.60:0/3705417749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc93406c040 0x7fc93406c420 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 -- 172.21.15.60:0/3705417749 >> 172.21.15.60:0/3705417749 conn(0x7fc9340ce6a0 msgr2=0x7fc9340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:10.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.700+0000 7fc938eaa640 1 -- 172.21.15.60:0/3705417749 shutdown_connections 2023-10-16T13:51:10.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.701+0000 7fc938eaa640 1 -- 172.21.15.60:0/3705417749 wait complete. 2023-10-16T13:51:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.702+0000 7fc938eaa640 1 Processor -- start 2023-10-16T13:51:10.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.702+0000 7fc938eaa640 1 -- start start 2023-10-16T13:51:10.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.703+0000 7fc938eaa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc93406c040 0x7fc934156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.703+0000 7fc938eaa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 0x7fc9340b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.703+0000 7fc938eaa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9340cda80 0x7fc9340ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.703+0000 7fc938eaa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340af0f0 con 0x7fc9340ad970 2023-10-16T13:51:10.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.703+0000 7fc938eaa640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340ae4e0 con 0x7fc93406c040 2023-10-16T13:51:10.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.703+0000 7fc938eaa640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340ae7c0 con 0x7fc9340cda80 2023-10-16T13:51:10.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 0x7fc9340b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 0x7fc9340b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40368/0 (socket says 172.21.15.60:40368) 2023-10-16T13:51:10.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 -- 172.21.15.60:0/1645530676 learned_addr learned my addr 172.21.15.60:0/1645530676 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:10.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 -- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9340cda80 msgr2=0x7fc9340ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9340cda80 0x7fc9340ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 -- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc93406c040 msgr2=0x7fc934156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc93406c040 0x7fc934156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 -- 172.21.15.60:0/1645530676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240094d0 con 0x7fc9340ad970 2023-10-16T13:51:10.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.704+0000 7fc92bfff640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 0x7fc9340b07f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c400 tx=0x7fc91c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.705+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c011790 con 0x7fc9340ad970 2023-10-16T13:51:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.705+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c011930 con 0x7fc9340ad970 2023-10-16T13:51:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.705+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c011ce0 con 0x7fc9340ad970 2023-10-16T13:51:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.705+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9340aeaa0 con 0x7fc9340ad970 2023-10-16T13:51:10.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.705+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9340af580 con 0x7fc9340ad970 2023-10-16T13:51:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.706+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93406b310 con 0x7fc9340ad970 2023-10-16T13:51:10.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.707+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 95) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc91c01c460 con 0x7fc9340ad970 2023-10-16T13:51:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.708+0000 7fc930ff9640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fc91406acf0 0x7fc91406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.708+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc91c089500 con 0x7fc9340ad970 2023-10-16T13:51:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.709+0000 7fc9337fe640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fc91406acf0 0x7fc91406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:10.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.710+0000 7fc9337fe640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fc91406acf0 0x7fc91406d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc924036000 tx=0x7fc924030000 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:10.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.712+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc91c054570 con 0x7fc9340ad970 2023-10-16T13:51:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.844+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 96) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc91c054940 con 0x7fc9340ad970 2023-10-16T13:51:10.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.916+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9340af230 con 0x7fc9340ad970 2023-10-16T13:51:10.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.919+0000 7fc930ff9640 1 -- 172.21.15.60:0/1645530676 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+17586 (secure 0 0 0) 0x7fc9340af230 con 0x7fc9340ad970 2023-10-16T13:51:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:10.922 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":185,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:08.807471+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/1295809784":"2023-10-17T13:51:08.807418+0000","172.21.15.144:6832/1295809784":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2083028711":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2443545815":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2621417165":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/4092785021":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/1556839457":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2455238630":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6833/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3778332831":"2023-10-17T13:51:08.807418+0000","172.21.15.144:6832/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3372584586":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1775356785":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3925300811":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1821662703":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3430745922":"2023-10-17T13:51:01.597577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:10.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fc91406acf0 msgr2=0x7fc91406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fc91406acf0 0x7fc91406d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc924036000 tx=0x7fc924030000 comp rx=0 tx=0).stop 2023-10-16T13:51:10.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 msgr2=0x7fc9340b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:10.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 0x7fc9340b07f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c400 tx=0x7fc91c00c8c0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 shutdown_connections 2023-10-16T13:51:10.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fc91406acf0 0x7fc91406d1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9340cda80 0x7fc9340ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9340ad970 0x7fc9340b07f0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 --2- 172.21.15.60:0/1645530676 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc93406c040 0x7fc934156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:10.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 >> 172.21.15.60:0/1645530676 conn(0x7fc9340ce6a0 msgr2=0x7fc93406f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:10.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 shutdown_connections 2023-10-16T13:51:10.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:10.922+0000 7fc938eaa640 1 -- 172.21.15.60:0/1645530676 wait complete. 2023-10-16T13:51:10.944 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:51:10.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:11.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.157+0000 7fcf63e1f640 1 Processor -- start 2023-10-16T13:51:11.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.157+0000 7fcf63e1f640 1 -- start start 2023-10-16T13:51:11.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf63e1f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcf5c0ce390 0x7fcf5c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf63e1f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5c15ed60 con 0x7fcf5c15ae80 2023-10-16T13:51:11.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf63e1f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5c0707e0 con 0x7fcf5c15aaa0 2023-10-16T13:51:11.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf63e1f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf5c06b810 con 0x7fcf5c0ce390 2023-10-16T13:51:11.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf613ba640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcf5c15ae80 0x7fcf5c070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56402/0 (socket says 172.21.15.60:56402) 2023-10-16T13:51:11.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf613ba640 1 -- 172.21.15.60:0/3427240089 learned_addr learned my addr 172.21.15.60:0/3427240089 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:11.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf623bc640 1 --2- 172.21.15.60:0/3427240089 >> v2:172.21.15.144:3300/0 conn(0x7fcf5c0ce390 0x7fcf5c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902357289 0 0) 0x7fcf5c15ed60 con 0x7fcf5c15ae80 2023-10-16T13:51:11.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.159+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf48003520 con 0x7fcf5c15ae80 2023-10-16T13:51:11.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744219643 0 0) 0x7fcf48003520 con 0x7fcf5c15ae80 2023-10-16T13:51:11.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 >> v2:172.21.15.144:3300/0 conn(0x7fcf5c0ce390 msgr2=0x7fcf5c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:11.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 --2- 172.21.15.60:0/3427240089 >> v2:172.21.15.144:3300/0 conn(0x7fcf5c0ce390 0x7fcf5c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 >> v1:172.21.15.144:6789/0 conn(0x7fcf5c15aaa0 legacy=0x7fcf5c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c15ef90 con 0x7fcf5c15ae80 2023-10-16T13:51:11.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcf4c001670 con 0x7fcf5c15ae80 2023-10-16T13:51:11.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 >> v1:172.21.15.60:6789/0 conn(0x7fcf5c15ae80 legacy=0x7fcf5c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf480047c0 0x7fcf48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf48005180 0x7fcf4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c06b810 con 0x7fcf480039c0 2023-10-16T13:51:11.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4800c080 con 0x7fcf480047c0 2023-10-16T13:51:11.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4800c240 con 0x7fcf48005180 2023-10-16T13:51:11.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.160+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf623bc640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf48005180 0x7fcf4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf61bbb640 1 -- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf48005180 msgr2=0x7fcf4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:11.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf48005180 0x7fcf4800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf61bbb640 1 -- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf480047c0 msgr2=0x7fcf48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:11.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf613ba640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf480047c0 0x7fcf48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf480047c0 0x7fcf48004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf61bbb640 1 -- 172.21.15.60:0/3427240089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf48003520 con 0x7fcf480039c0 2023-10-16T13:51:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.161+0000 7fcf623bc640 1 -- 172.21.15.60:0/3427240089 reap_dead start 2023-10-16T13:51:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.162+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fcf58001750 tx=0x7fcf480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758d6cccd9901b15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.162+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf580052d0 con 0x7fcf480039c0 2023-10-16T13:51:11.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.162+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5802e050 con 0x7fcf480039c0 2023-10-16T13:51:11.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.162+0000 7fcf60bb9640 1 -- 172.21.15.60:0/3427240089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58003830 con 0x7fcf480039c0 2023-10-16T13:51:11.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 -- 172.21.15.60:0/3427240089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 msgr2=0x7fcf48003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fcf58001750 tx=0x7fcf480057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 -- 172.21.15.60:0/3427240089 shutdown_connections 2023-10-16T13:51:11.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/3427240089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf48003de0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 -- 172.21.15.60:0/3427240089 >> 172.21.15.60:0/3427240089 conn(0x7fcf5c0cb410 msgr2=0x7fcf5c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:11.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 -- 172.21.15.60:0/3427240089 shutdown_connections 2023-10-16T13:51:11.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.163+0000 7fcf63e1f640 1 -- 172.21.15.60:0/3427240089 wait complete. 2023-10-16T13:51:11.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.165+0000 7fcf63e1f640 1 Processor -- start 2023-10-16T13:51:11.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.165+0000 7fcf63e1f640 1 -- start start 2023-10-16T13:51:11.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf63e1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf5c0c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf63e1f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf5c0c3290 0x7fcf5c0c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf63e1f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf5c0c6ff0 0x7fcf5c0c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf63e1f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c0c1a20 con 0x7fcf480039c0 2023-10-16T13:51:11.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf63e1f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c0c7c60 con 0x7fcf5c0c6ff0 2023-10-16T13:51:11.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf63e1f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c0c79b0 con 0x7fcf5c0c3290 2023-10-16T13:51:11.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf61bbb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf5c0c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf61bbb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf5c0c10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40380/0 (socket says 172.21.15.60:40380) 2023-10-16T13:51:11.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.166+0000 7fcf61bbb640 1 -- 172.21.15.60:0/2528380054 learned_addr learned my addr 172.21.15.60:0/2528380054 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:11.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf61bbb640 1 -- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf5c0c3290 msgr2=0x7fcf5c0c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf5c0c3290 0x7fcf5c0c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf61bbb640 1 -- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf5c0c6ff0 msgr2=0x7fcf5c0c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf5c0c6ff0 0x7fcf5c0c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf61bbb640 1 -- 172.21.15.60:0/2528380054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c0c7f00 con 0x7fcf480039c0 2023-10-16T13:51:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf61bbb640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf5c0c10b0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802bb80 tx=0x7fcf58001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf580076c0 con 0x7fcf480039c0 2023-10-16T13:51:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf58007a30 con 0x7fcf480039c0 2023-10-16T13:51:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58006c20 con 0x7fcf480039c0 2023-10-16T13:51:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.167+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5c0c81e0 con 0x7fcf480039c0 2023-10-16T13:51:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.168+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5c0c17d0 con 0x7fcf480039c0 2023-10-16T13:51:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.169+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5c0c83d0 con 0x7fcf480039c0 2023-10-16T13:51:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.170+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 96) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fcf580035f0 con 0x7fcf480039c0 2023-10-16T13:51:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.171+0000 7fcf527fc640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fcf4006af00 0x7fcf4006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.171+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcf580ab660 con 0x7fcf480039c0 2023-10-16T13:51:11.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.171+0000 7fcf613ba640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fcf4006af00 0x7fcf4006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.173+0000 7fcf613ba640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fcf4006af00 0x7fcf4006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c000c00 tx=0x7fcf4c009040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:11.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.175+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf58079d80 con 0x7fcf480039c0 2023-10-16T13:51:11.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.411+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf5c04e160 con 0x7fcf480039c0 2023-10-16T13:51:11.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.412+0000 7fcf527fc640 1 -- 172.21.15.60:0/2528380054 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fcf58079910 con 0x7fcf480039c0 2023-10-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:50:46.866966+0000","modified":"2023-10-16T13:50:49.938566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/691868089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":691868089},{"type":"v1","addr":"172.21.15.60:6834","nonce":691868089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 87 2023-10-16T13:51:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fcf4006af00 msgr2=0x7fcf4006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fcf4006af00 0x7fcf4006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c000c00 tx=0x7fcf4c009040 comp rx=0 tx=0).stop 2023-10-16T13:51:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 msgr2=0x7fcf5c0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf5c0c10b0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802bb80 tx=0x7fcf58001120 comp rx=0 tx=0).stop 2023-10-16T13:51:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 shutdown_connections 2023-10-16T13:51:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fcf4006af00 0x7fcf4006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcf5c0c3290 0x7fcf5c0c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf5c0c6ff0 0x7fcf5c0c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 --2- 172.21.15.60:0/2528380054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf480039c0 0x7fcf5c0c10b0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 >> 172.21.15.60:0/2528380054 conn(0x7fcf5c0cb410 msgr2=0x7fcf5c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 shutdown_connections 2023-10-16T13:51:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.415+0000 7fcf63e1f640 1 -- 172.21.15.60:0/2528380054 wait complete. 2023-10-16T13:51:11.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:11.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.664+0000 7f5a867ac640 1 Processor -- start 2023-10-16T13:51:11.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.664+0000 7f5a867ac640 1 -- start start 2023-10-16T13:51:11.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.665+0000 7f5a867ac640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a800ad970 0x7f5a800cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.665+0000 7f5a867ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.665+0000 7f5a867ac640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8015b9d0 con 0x7f5a8006c040 2023-10-16T13:51:11.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.665+0000 7f5a867ac640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8015bcb0 con 0x7f5a800ad970 2023-10-16T13:51:11.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.665+0000 7f5a867ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8015bf90 con 0x7f5a800ccfb0 2023-10-16T13:51:11.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.666+0000 7f5a85fab640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.666+0000 7f5a84fa9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a800ad970 0x7f5a800cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.666+0000 7f5a85fab640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53806/0 (socket says 172.21.15.60:53806) 2023-10-16T13:51:11.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.666+0000 7f5a85fab640 1 -- 172.21.15.60:0/3016157113 learned_addr learned my addr 172.21.15.60:0/3016157113 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:11.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.666+0000 7f5a84fa9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a800ad970 0x7f5a800cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:60990/0 (socket says 172.21.15.60:60990) 2023-10-16T13:51:11.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.667+0000 7f5a84fa9640 1 -- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 msgr2=0x7f5a800cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:11.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.667+0000 7f5a84fa9640 1 --2- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.667+0000 7f5a84fa9640 1 -- 172.21.15.60:0/3016157113 >> v1:172.21.15.144:6789/0 conn(0x7f5a8006c040 legacy=0x7f5a8006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.667+0000 7f5a84fa9640 1 -- 172.21.15.60:0/3016157113 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a800cc270 con 0x7f5a800ad970 2023-10-16T13:51:11.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.667+0000 7f5a85fab640 1 --2- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:51:11.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.669+0000 7f5a84fa9640 1 --2- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800cca70 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5a700097a0 tx=0x7f5a7002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2132f413aba12d7d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:11.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.669+0000 7f5a77fff640 1 -- 172.21.15.60:0/3016157113 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7002d040 con 0x7f5a800ad970 2023-10-16T13:51:11.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.669+0000 7f5a77fff640 1 -- 172.21.15.60:0/3016157113 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a70031810 con 0x7f5a800ad970 2023-10-16T13:51:11.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.669+0000 7f5a77fff640 1 -- 172.21.15.60:0/3016157113 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a70031ba0 con 0x7f5a800ad970 2023-10-16T13:51:11.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.670+0000 7f5a867ac640 1 -- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 msgr2=0x7f5a800cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.670+0000 7f5a867ac640 1 --2- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800cca70 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5a700097a0 tx=0x7f5a7002b8b0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.671+0000 7f5a867ac640 1 -- 172.21.15.60:0/3016157113 shutdown_connections 2023-10-16T13:51:11.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.671+0000 7f5a867ac640 1 --2- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.671+0000 7f5a867ac640 1 --2- 172.21.15.60:0/3016157113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800cca70 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.671+0000 7f5a867ac640 1 -- 172.21.15.60:0/3016157113 >> 172.21.15.60:0/3016157113 conn(0x7f5a800ce6a0 msgr2=0x7f5a800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:11.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.671+0000 7f5a867ac640 1 -- 172.21.15.60:0/3016157113 shutdown_connections 2023-10-16T13:51:11.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.671+0000 7f5a867ac640 1 -- 172.21.15.60:0/3016157113 wait complete. 2023-10-16T13:51:11.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.673+0000 7f5a867ac640 1 Processor -- start 2023-10-16T13:51:11.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.673+0000 7f5a867ac640 1 -- start start 2023-10-16T13:51:11.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.674+0000 7f5a867ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 0x7f5a80154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.674+0000 7f5a867ac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.674+0000 7f5a867ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.674+0000 7f5a867ac640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800c2dc0 con 0x7f5a8006c040 2023-10-16T13:51:11.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.674+0000 7f5a867ac640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800c21b0 con 0x7f5a800ad970 2023-10-16T13:51:11.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.674+0000 7f5a867ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800c2490 con 0x7f5a800ccfb0 2023-10-16T13:51:11.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 0x7f5a80154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 0x7f5a80154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40390/0 (socket says 172.21.15.60:40390) 2023-10-16T13:51:11.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 -- 172.21.15.60:0/4063877176 learned_addr learned my addr 172.21.15.60:0/4063877176 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:11.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a84fa9640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a85fab640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 -- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 msgr2=0x7f5a800c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800c4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 -- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 msgr2=0x7f5a800c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.675+0000 7f5a857aa640 1 -- 172.21.15.60:0/4063877176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a780094d0 con 0x7f5a8006c040 2023-10-16T13:51:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.676+0000 7f5a857aa640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 0x7f5a80154070 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00c870 tx=0x7f5a7c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.676+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c003b70 con 0x7f5a8006c040 2023-10-16T13:51:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.676+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c003cf0 con 0x7f5a8006c040 2023-10-16T13:51:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.676+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00aa90 con 0x7f5a8006c040 2023-10-16T13:51:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.676+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70009400 con 0x7f5a8006c040 2023-10-16T13:51:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.676+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a800c2a50 con 0x7f5a8006c040 2023-10-16T13:51:11.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.677+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8006b330 con 0x7f5a8006c040 2023-10-16T13:51:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.678+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 96) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5a7c00abf0 con 0x7f5a8006c040 2023-10-16T13:51:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.679+0000 7f5a75ffb640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5a6406af30 0x7f5a6406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.679+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5a7c089770 con 0x7f5a8006c040 2023-10-16T13:51:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.680+0000 7f5a84fa9640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5a6406af30 0x7f5a6406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.681+0000 7f5a84fa9640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5a6406af30 0x7f5a6406d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a70036000 tx=0x7f5a7003c040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.684+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a7c054580 con 0x7f5a8006c040 2023-10-16T13:51:11.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.921+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a8004e9b0 con 0x7f5a8006c040 2023-10-16T13:51:11.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.922+0000 7f5a75ffb640 1 -- 172.21.15.60:0/4063877176 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f5a7c003e60 con 0x7f5a8006c040 2023-10-16T13:51:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:11.925 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:50:46.866966+0000","modified":"2023-10-16T13:50:49.938566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9510":{"gid":9510,"name":"c","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/691868089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":691868089},{"type":"v1","addr":"172.21.15.60:6834","nonce":691868089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:51:11.925 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 87 2023-10-16T13:51:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5a6406af30 msgr2=0x7f5a6406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5a6406af30 0x7f5a6406d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a70036000 tx=0x7f5a7003c040 comp rx=0 tx=0).stop 2023-10-16T13:51:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 msgr2=0x7f5a80154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 0x7f5a80154070 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00c870 tx=0x7f5a7c00cd30 comp rx=0 tx=0).stop 2023-10-16T13:51:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 shutdown_connections 2023-10-16T13:51:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5a6406af30 0x7f5a6406d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a800ccfb0 0x7f5a800c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a800ad970 0x7f5a800c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 --2- 172.21.15.60:0/4063877176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8006c040 0x7f5a80154070 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 >> 172.21.15.60:0/4063877176 conn(0x7f5a800ce6a0 msgr2=0x7f5a8006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:11.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 shutdown_connections 2023-10-16T13:51:11.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:11.925+0000 7f5a867ac640 1 -- 172.21.15.60:0/4063877176 wait complete. 2023-10-16T13:51:11.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.174+0000 7efccebf0640 1 Processor -- start 2023-10-16T13:51:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.174+0000 7efccebf0640 1 -- start start 2023-10-16T13:51:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.175+0000 7efccebf0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 0x7efcc815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.175+0000 7efccebf0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efcc80ce390 0x7efcc8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.175+0000 7efccebf0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc815ed60 con 0x7efcc815ae80 2023-10-16T13:51:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.175+0000 7efccebf0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc806b810 con 0x7efcc80ce390 2023-10-16T13:51:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.175+0000 7efccebf0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80707e0 con 0x7efcc815aaa0 2023-10-16T13:51:12.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.176+0000 7efccc98c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 0x7efcc815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.176+0000 7efccd18d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efcc80ce390 0x7efcc8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.176+0000 7efccc98c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 0x7efcc815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53826/0 (socket says 172.21.15.60:53826) 2023-10-16T13:51:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.176+0000 7efccc98c640 1 -- 172.21.15.60:0/3234194586 learned_addr learned my addr 172.21.15.60:0/3234194586 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.177+0000 7efccd18d640 1 -- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 msgr2=0x7efcc815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.177+0000 7efccd18d640 1 --2- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 0x7efcc815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.177+0000 7efccd18d640 1 -- 172.21.15.60:0/3234194586 >> v1:172.21.15.144:6789/0 conn(0x7efcc815ae80 legacy=0x7efcc80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.177+0000 7efccd18d640 1 -- 172.21.15.60:0/3234194586 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc815ef90 con 0x7efcc80ce390 2023-10-16T13:51:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.177+0000 7efcc77fe640 1 -- 172.21.15.60:0/3234194586 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550714303 0 0) 0x7efcc815ed60 con 0x7efcc815ae80 2023-10-16T13:51:12.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.177+0000 7efccc98c640 1 --2- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 0x7efcc815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:51:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.178+0000 7efccd18d640 1 --2- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc80ce390 0x7efcc8070f50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efcc00097a0 tx=0x7efcc002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56fe07e52379d706 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efcc77fe640 1 -- 172.21.15.60:0/3234194586 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc002d040 con 0x7efcc80ce390 2023-10-16T13:51:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efcc77fe640 1 -- 172.21.15.60:0/3234194586 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc0031810 con 0x7efcc80ce390 2023-10-16T13:51:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efcc77fe640 1 -- 172.21.15.60:0/3234194586 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0031ba0 con 0x7efcc80ce390 2023-10-16T13:51:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efccebf0640 1 -- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc80ce390 msgr2=0x7efcc8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efccebf0640 1 --2- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc80ce390 0x7efcc8070f50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efcc00097a0 tx=0x7efcc002b8b0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efccebf0640 1 -- 172.21.15.60:0/3234194586 shutdown_connections 2023-10-16T13:51:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efccebf0640 1 --2- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc80ce390 0x7efcc8070f50 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efccebf0640 1 --2- 172.21.15.60:0/3234194586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc815aaa0 0x7efcc815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.179+0000 7efccebf0640 1 -- 172.21.15.60:0/3234194586 >> 172.21.15.60:0/3234194586 conn(0x7efcc80cb410 msgr2=0x7efcc80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.180+0000 7efccebf0640 1 -- 172.21.15.60:0/3234194586 shutdown_connections 2023-10-16T13:51:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.180+0000 7efccebf0640 1 -- 172.21.15.60:0/3234194586 wait complete. 2023-10-16T13:51:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.181+0000 7efccebf0640 1 Processor -- start 2023-10-16T13:51:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.181+0000 7efccebf0640 1 -- start start 2023-10-16T13:51:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.182+0000 7efccebf0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc80ce390 0x7efcc8151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.182+0000 7efccebf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 0x7efcc80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efccebf0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc815ae80 0x7efcc80b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efccebf0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc8151590 con 0x7efcc815aaa0 2023-10-16T13:51:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efccebf0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80aed60 con 0x7efcc815ae80 2023-10-16T13:51:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efccebf0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80af040 con 0x7efcc80ce390 2023-10-16T13:51:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 0x7efcc80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 0x7efcc80ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40400/0 (socket says 172.21.15.60:40400) 2023-10-16T13:51:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 -- 172.21.15.60:0/2375711662 learned_addr learned my addr 172.21.15.60:0/2375711662 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efccc98c640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc80ce390 0x7efcc8151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efccd18d640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc815ae80 0x7efcc80b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 -- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc80ce390 msgr2=0x7efcc8151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc80ce390 0x7efcc8151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 -- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc815ae80 msgr2=0x7efcc80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.183+0000 7efcc7fff640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc815ae80 0x7efcc80b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efcc7fff640 1 -- 172.21.15.60:0/2375711662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb8009b30 con 0x7efcc815aaa0 2023-10-16T13:51:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efcc7fff640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 0x7efcc80ae3f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00c410 tx=0x7efcbc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc0035f0 con 0x7efcc815aaa0 2023-10-16T13:51:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc003770 con 0x7efcc815aaa0 2023-10-16T13:51:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc0009400 con 0x7efcc815aaa0 2023-10-16T13:51:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc003b20 con 0x7efcc815aaa0 2023-10-16T13:51:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.184+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc8156cc0 con 0x7efcc815aaa0 2023-10-16T13:51:12.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.186+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc804e160 con 0x7efcc815aaa0 2023-10-16T13:51:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.186+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 96) v1 ==== 89275+0+0 (secure 0 0 0) 0x7efcbc003c80 con 0x7efcc815aaa0 2023-10-16T13:51:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.187+0000 7efcc57fa640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7efcb406af00 0x7efcb406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.188+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efcbc088ec0 con 0x7efcc815aaa0 2023-10-16T13:51:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.188+0000 7efccc98c640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7efcb406af00 0x7efcb406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.189+0000 7efccc98c640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7efcb406af00 0x7efcb406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efcc80b2880 tx=0x7efcb800b040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:12.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.192+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efcbc053d60 con 0x7efcc815aaa0 2023-10-16T13:51:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.391+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcc80cdc60 con 0x7efcc815aaa0 2023-10-16T13:51:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.395+0000 7efcc57fa640 1 -- 172.21.15.60:0/2375711662 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+17586 (secure 0 0 0) 0x7efcbc017020 con 0x7efcc815aaa0 2023-10-16T13:51:12.397 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:12.398 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":185,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:08.807471+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:50:44.806197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-16T13:50:45.907641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/1295809784":"2023-10-17T13:51:08.807418+0000","172.21.15.144:6832/1295809784":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2083028711":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2443545815":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2621417165":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/4092785021":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/1556839457":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2455238630":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6833/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3778332831":"2023-10-17T13:51:08.807418+0000","172.21.15.144:6832/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3372584586":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1775356785":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3925300811":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1821662703":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3430745922":"2023-10-17T13:51:01.597577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7efcb406af00 msgr2=0x7efcb406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7efcb406af00 0x7efcb406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efcc80b2880 tx=0x7efcb800b040 comp rx=0 tx=0).stop 2023-10-16T13:51:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 msgr2=0x7efcc80ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 0x7efcc80ae3f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00c410 tx=0x7efcbc00c8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 shutdown_connections 2023-10-16T13:51:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7efcb406af00 0x7efcb406d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efcc815ae80 0x7efcc80b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc815aaa0 0x7efcc80ae3f0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 --2- 172.21.15.60:0/2375711662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efcc80ce390 0x7efcc8151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.397+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 >> 172.21.15.60:0/2375711662 conn(0x7efcc80cb410 msgr2=0x7efcc8159dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.398+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 shutdown_connections 2023-10-16T13:51:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.398+0000 7efccebf0640 1 -- 172.21.15.60:0/2375711662 wait complete. 2023-10-16T13:51:12.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:51:12.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.648+0000 7f740249f640 1 Processor -- start 2023-10-16T13:51:12.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.648+0000 7f740249f640 1 -- start start 2023-10-16T13:51:12.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740249f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f73fc159950 0x7f73fc15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740249f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73fc070450 0x7f73fc0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740249f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc15aef0 con 0x7f73fc06c100 2023-10-16T13:51:12.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740249f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc070e20 con 0x7f73fc159950 2023-10-16T13:51:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740249f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc15e490 con 0x7f73fc070450 2023-10-16T13:51:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740149d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f73fc159950 0x7f73fc15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740149d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f73fc159950 0x7f73fc15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40416/0 (socket says 172.21.15.60:40416) 2023-10-16T13:51:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f740149d640 1 -- 172.21.15.60:0/2257376264 learned_addr learned my addr 172.21.15.60:0/2257376264 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.650+0000 7f7400c9c640 1 --2- 172.21.15.60:0/2257376264 >> v2:172.21.15.144:3300/0 conn(0x7f73fc070450 0x7f73fc0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.651+0000 7f740149d640 1 -- 172.21.15.60:0/2257376264 >> v2:172.21.15.144:3300/0 conn(0x7f73fc070450 msgr2=0x7f73fc0708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.651+0000 7f740149d640 1 --2- 172.21.15.60:0/2257376264 >> v2:172.21.15.144:3300/0 conn(0x7f73fc070450 0x7f73fc0708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.651+0000 7f740149d640 1 -- 172.21.15.60:0/2257376264 >> v1:172.21.15.144:6789/0 conn(0x7f73fc06c100 legacy=0x7f73fc06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.651+0000 7f740149d640 1 -- 172.21.15.60:0/2257376264 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc15d910 con 0x7f73fc159950 2023-10-16T13:51:12.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.651+0000 7f7400c9c640 1 --2- 172.21.15.60:0/2257376264 >> v2:172.21.15.144:3300/0 conn(0x7f73fc070450 0x7f73fc0708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.652+0000 7f740149d640 1 --2- 172.21.15.60:0/2257376264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc159950 0x7f73fc15a9b0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0cab70 tx=0x7f73f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0bd9396b95f803 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.652+0000 7f73f3fff640 1 -- 172.21.15.60:0/2257376264 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80317e0 con 0x7f73fc159950 2023-10-16T13:51:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.652+0000 7f73f3fff640 1 -- 172.21.15.60:0/2257376264 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f8031940 con 0x7f73fc159950 2023-10-16T13:51:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.652+0000 7f73f3fff640 1 -- 172.21.15.60:0/2257376264 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8031cd0 con 0x7f73fc159950 2023-10-16T13:51:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 -- 172.21.15.60:0/2257376264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc159950 msgr2=0x7f73fc15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 --2- 172.21.15.60:0/2257376264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc159950 0x7f73fc15a9b0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0cab70 tx=0x7f73f802b880 comp rx=0 tx=0).stop 2023-10-16T13:51:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 -- 172.21.15.60:0/2257376264 shutdown_connections 2023-10-16T13:51:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 --2- 172.21.15.60:0/2257376264 >> v2:172.21.15.144:3300/0 conn(0x7f73fc070450 0x7f73fc0708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 --2- 172.21.15.60:0/2257376264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc159950 0x7f73fc15a9b0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 -- 172.21.15.60:0/2257376264 >> 172.21.15.60:0/2257376264 conn(0x7f73fc0ce6a0 msgr2=0x7f73fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 -- 172.21.15.60:0/2257376264 shutdown_connections 2023-10-16T13:51:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.653+0000 7f740249f640 1 -- 172.21.15.60:0/2257376264 wait complete. 2023-10-16T13:51:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.655+0000 7f740249f640 1 Processor -- start 2023-10-16T13:51:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.655+0000 7f740249f640 1 -- start start 2023-10-16T13:51:12.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740249f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 0x7f73fc0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740249f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73fc070450 0x7f73fc0af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740249f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73fc159950 0x7f73fc1ed640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740249f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0c8a70 con 0x7f73fc06c100 2023-10-16T13:51:12.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740249f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0b1440 con 0x7f73fc070450 2023-10-16T13:51:12.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740249f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0b1700 con 0x7f73fc159950 2023-10-16T13:51:12.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.656+0000 7f740149d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 0x7f73fc0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 0x7f73fc0c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40422/0 (socket says 172.21.15.60:40422) 2023-10-16T13:51:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 -- 172.21.15.60:0/1544227932 learned_addr learned my addr 172.21.15.60:0/1544227932 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f7401c9e640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73fc159950 0x7f73fc1ed640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f7400c9c640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73fc070450 0x7f73fc0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 -- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73fc159950 msgr2=0x7f73fc1ed640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73fc159950 0x7f73fc1ed640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 -- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73fc070450 msgr2=0x7f73fc0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73fc070450 0x7f73fc0af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 -- 172.21.15.60:0/1544227932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80094d0 con 0x7f73fc06c100 2023-10-16T13:51:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.657+0000 7f740149d640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 0x7f73fc0c8530 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f73f802b8c0 tx=0x7f73f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.658+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80329c0 con 0x7f73fc06c100 2023-10-16T13:51:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.658+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f8032b40 con 0x7f73fc06c100 2023-10-16T13:51:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.658+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80453f0 con 0x7f73fc06c100 2023-10-16T13:51:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.658+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc0b19e0 con 0x7f73fc06c100 2023-10-16T13:51:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.658+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc0b1fa0 con 0x7f73fc06c100 2023-10-16T13:51:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.659+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc04e9b0 con 0x7f73fc06c100 2023-10-16T13:51:12.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.660+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 96) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f73f804d3f0 con 0x7f73fc06c100 2023-10-16T13:51:12.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.661+0000 7f73f1ffb640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f73e006af00 0x7f73e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.661+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f73f80ab8a0 con 0x7f73fc06c100 2023-10-16T13:51:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.662+0000 7f7400c9c640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f73e006af00 0x7f73e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.663+0000 7f7400c9c640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f73e006af00 0x7f73e006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f73ec006fd0 tx=0x7f73ec008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.665+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73f80764a0 con 0x7f73fc06c100 2023-10-16T13:51:12.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.863+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f73f8076870 con 0x7f73fc06c100 2023-10-16T13:51:12.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.908+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f73fc15aef0 con 0x7f73fc06c100 2023-10-16T13:51:12.912 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T13:51:12.910+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:51:12.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.913+0000 7f73f1ffb640 1 -- 172.21.15.60:0/1544227932 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) v1 ==== 161+0+0 (secure 0 0 0) 0x7f73fc15aef0 con 0x7f73fc06c100 2023-10-16T13:51:12.916 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:51:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.917+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f73e006af00 msgr2=0x7f73e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.917+0000 7f740249f640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f73e006af00 0x7f73e006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f73ec006fd0 tx=0x7f73ec008040 comp rx=0 tx=0).stop 2023-10-16T13:51:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.917+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 msgr2=0x7f73fc0c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.917+0000 7f740249f640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 0x7f73fc0c8530 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f73f802b8c0 tx=0x7f73f80046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 shutdown_connections 2023-10-16T13:51:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f73e006af00 0x7f73e006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73fc159950 0x7f73fc1ed640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73fc070450 0x7f73fc0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 --2- 172.21.15.60:0/1544227932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73fc06c100 0x7f73fc0c8530 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:12.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 >> 172.21.15.60:0/1544227932 conn(0x7f73fc0ce6a0 msgr2=0x7f73fc0ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:12.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 shutdown_connections 2023-10-16T13:51:12.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:12.918+0000 7f740249f640 1 -- 172.21.15.60:0/1544227932 wait complete. 2023-10-16T13:51:12.932 INFO:tasks.ceph.mds.c.smithi060.stderr: -3810> 2023-10-16T13:50:51.343+0000 7fc74c264640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:51:12.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:51:13.003 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T13:51:13.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.145+0000 7f07258bb640 1 Processor -- start 2023-10-16T13:51:13.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.145+0000 7f07258bb640 1 -- start start 2023-10-16T13:51:13.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.147+0000 7f07258bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 0x7f07200c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.147+0000 7f07258bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200ac890 con 0x7f072015d150 2023-10-16T13:51:13.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.147+0000 7f07258bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200ad7a0 con 0x7f07200ad350 2023-10-16T13:51:13.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.147+0000 7f07258bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072015d530 con 0x7f072006a4b0 2023-10-16T13:51:13.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.147+0000 7f071effd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f072015d150 0x7f0720160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56406/0 (socket says 172.21.15.60:56406) 2023-10-16T13:51:13.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.147+0000 7f071effd640 1 -- 172.21.15.60:0/3612124271 learned_addr learned my addr 172.21.15.60:0/3612124271 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:13.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071f7fe640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 0x7f07200c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:13.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795998654 0 0) 0x7f07200ac890 con 0x7f072015d150 2023-10-16T13:51:13.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0708003520 con 0x7f072015d150 2023-10-16T13:51:13.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 151634317 0 0) 0x7f0708003520 con 0x7f072015d150 2023-10-16T13:51:13.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 msgr2=0x7f07200c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:13.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 0x7f07200c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 >> v1:172.21.15.144:6789/0 conn(0x7f07200ad350 legacy=0x7f0720069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720161650 con 0x7f072015d150 2023-10-16T13:51:13.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0710001670 con 0x7f072015d150 2023-10-16T13:51:13.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 >> v1:172.21.15.60:6789/0 conn(0x7f072015d150 legacy=0x7f0720160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07080039a0 0x7f0708003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 0x7f07080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.148+0000 7f071dffb640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f0708005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072015d530 con 0x7f0708004d00 2023-10-16T13:51:13.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070800ba30 con 0x7f07080039a0 2023-10-16T13:51:13.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070800bbc0 con 0x7f072006a4b0 2023-10-16T13:51:13.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071f7fe640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f0708005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:13.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071f7fe640 1 -- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 msgr2=0x7f07080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:13.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071f7fe640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f072006a4b0 0x7f07080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071f7fe640 1 -- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07080039a0 msgr2=0x7f0708003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:13.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071effd640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07080039a0 0x7f0708003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:13.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071f7fe640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07080039a0 0x7f0708003d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.149+0000 7f071f7fe640 1 -- 172.21.15.60:0/3612124271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708003520 con 0x7f0708004d00 2023-10-16T13:51:13.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.150+0000 7f071f7fe640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f0708005180 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f071800e170 tx=0x7f07080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcf06651a214c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:13.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071802cc20 con 0x7f0708004d00 2023-10-16T13:51:13.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071800f040 con 0x7f0708004d00 2023-10-16T13:51:13.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f071dffb640 1 -- 172.21.15.60:0/3612124271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718034440 con 0x7f0708004d00 2023-10-16T13:51:13.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f07258bb640 1 -- 172.21.15.60:0/3612124271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 msgr2=0x7f0708005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f07258bb640 1 --2- 172.21.15.60:0/3612124271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f0708005180 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f071800e170 tx=0x7f07080057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f071f7fe640 1 -- 172.21.15.60:0/3612124271 reap_dead start 2023-10-16T13:51:13.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f07258bb640 1 -- 172.21.15.60:0/3612124271 shutdown_connections 2023-10-16T13:51:13.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f07258bb640 1 -- 172.21.15.60:0/3612124271 >> 172.21.15.60:0/3612124271 conn(0x7f07200ce9b0 msgr2=0x7f07200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:13.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f07258bb640 1 -- 172.21.15.60:0/3612124271 shutdown_connections 2023-10-16T13:51:13.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.151+0000 7f07258bb640 1 -- 172.21.15.60:0/3612124271 wait complete. 2023-10-16T13:51:13.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.153+0000 7f07258bb640 1 Processor -- start 2023-10-16T13:51:13.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.153+0000 7f07258bb640 1 -- start start 2023-10-16T13:51:13.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f07258bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f072014ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f07258bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f072015c3a0 0x7f07200aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f07258bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07200af160 0x7f07200c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f07258bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200afbf0 con 0x7f0708004d00 2023-10-16T13:51:13.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f07258bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200c6000 con 0x7f072015c3a0 2023-10-16T13:51:13.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f07258bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07200af630 con 0x7f07200af160 2023-10-16T13:51:13.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.154+0000 7f071effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f072014ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:13.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f072014ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40460/0 (socket says 172.21.15.60:40460) 2023-10-16T13:51:13.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 -- 172.21.15.60:0/661891535 learned_addr learned my addr 172.21.15.60:0/661891535 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:13.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 -- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07200af160 msgr2=0x7f07200c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07200af160 0x7f07200c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 -- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f072015c3a0 msgr2=0x7f07200aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f072015c3a0 0x7f07200aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 -- 172.21.15.60:0/661891535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200af910 con 0x7f0708004d00 2023-10-16T13:51:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.155+0000 7f071effd640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f072014ed50 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f071000d810 tx=0x7f071000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.156+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07100074e0 con 0x7f0708004d00 2023-10-16T13:51:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.156+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0710007640 con 0x7f0708004d00 2023-10-16T13:51:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.156+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710004a20 con 0x7f0708004d00 2023-10-16T13:51:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.156+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07200c5c20 con 0x7f0708004d00 2023-10-16T13:51:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.156+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07200c84a0 con 0x7f0708004d00 2023-10-16T13:51:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.157+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072004e160 con 0x7f0708004d00 2023-10-16T13:51:13.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.158+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f071000b5d0 con 0x7f0708004d00 2023-10-16T13:51:13.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.159+0000 7f0707fff640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f070c06afc0 0x7f070c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:13.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.159+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f0710089800 con 0x7f0708004d00 2023-10-16T13:51:13.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.160+0000 7f071e7fc640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f070c06afc0 0x7f070c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:13.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.161+0000 7f071e7fc640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f070c06afc0 0x7f070c06d480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07200afd50 tx=0x7f0714007950 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:13.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.163+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07100544a0 con 0x7f0708004d00 2023-10-16T13:51:13.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.402+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f07200ccf80 con 0x7f0708004d00 2023-10-16T13:51:13.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.917+0000 7f0707fff640 1 -- 172.21.15.60:0/661891535 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0710057ac0 con 0x7f0708004d00 2023-10-16T13:51:13.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.921+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f070c06afc0 msgr2=0x7f070c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.921+0000 7f07258bb640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f070c06afc0 0x7f070c06d480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07200afd50 tx=0x7f0714007950 comp rx=0 tx=0).stop 2023-10-16T13:51:13.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.921+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 msgr2=0x7f072014ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:13.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.921+0000 7f07258bb640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f072014ed50 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f071000d810 tx=0x7f071000dcd0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 shutdown_connections 2023-10-16T13:51:13.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f070c06afc0 0x7f070c06d480 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f07200af160 0x7f07200c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f072015c3a0 0x7f07200aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 --2- 172.21.15.60:0/661891535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0708004d00 0x7f072014ed50 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:13.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 >> 172.21.15.60:0/661891535 conn(0x7f07200ce9b0 msgr2=0x7f072006fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:13.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 shutdown_connections 2023-10-16T13:51:13.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:13.922+0000 7f07258bb640 1 -- 172.21.15.60:0/661891535 wait complete. 2023-10-16T13:51:13.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:51:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.168+0000 7f0879b78640 1 Processor -- start 2023-10-16T13:51:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.169+0000 7f0879b78640 1 -- start start 2023-10-16T13:51:14.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.170+0000 7f0879b78640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f087415ac10 0x7f087406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:14.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.170+0000 7f0879b78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f08740b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:14.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.170+0000 7f0879b78640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087415b310 con 0x7f08740b0360 2023-10-16T13:51:14.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.170+0000 7f0879b78640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087415f800 con 0x7f087415ac10 2023-10-16T13:51:14.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.170+0000 7f0879b78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08740b0d30 con 0x7f08740cac20 2023-10-16T13:51:14.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.171+0000 7f08737fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f087415ac10 0x7f087406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:14.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.171+0000 7f08737fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f087415ac10 0x7f087406c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:32848/0 (socket says 172.21.15.60:32848) 2023-10-16T13:51:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.171+0000 7f08737fe640 1 -- 172.21.15.60:0/2585229713 learned_addr learned my addr 172.21.15.60:0/2585229713 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.171+0000 7f0872ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f08740b0360 0x7f08740ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:54802/0 (socket says 172.21.15.60:54802) 2023-10-16T13:51:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.171+0000 7f0873fff640 1 --2- 172.21.15.60:0/2585229713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f08740b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.172+0000 7f08727fc640 1 -- 172.21.15.60:0/2585229713 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388736615 0 0) 0x7f087415b310 con 0x7f08740b0360 2023-10-16T13:51:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.172+0000 7f08727fc640 1 -- 172.21.15.60:0/2585229713 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0860003520 con 0x7f08740b0360 2023-10-16T13:51:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.172+0000 7f0873fff640 1 -- 172.21.15.60:0/2585229713 >> v2:172.21.15.144:3300/0 conn(0x7f087415ac10 msgr2=0x7f087406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.172+0000 7f0873fff640 1 --2- 172.21.15.60:0/2585229713 >> v2:172.21.15.144:3300/0 conn(0x7f087415ac10 0x7f087406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.172+0000 7f0873fff640 1 -- 172.21.15.60:0/2585229713 >> v1:172.21.15.144:6789/0 conn(0x7f08740b0360 legacy=0x7f08740ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.172+0000 7f0873fff640 1 -- 172.21.15.60:0/2585229713 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087406f640 con 0x7f08740cac20 2023-10-16T13:51:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.173+0000 7f0873fff640 1 --2- 172.21.15.60:0/2585229713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f08740b07f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f086c0097a0 tx=0x7f086c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5ebb616745391ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f08727fc640 1 -- 172.21.15.60:0/2585229713 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c02e070 con 0x7f08740cac20 2023-10-16T13:51:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f08727fc640 1 -- 172.21.15.60:0/2585229713 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c0315a0 con 0x7f08740cac20 2023-10-16T13:51:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f08727fc640 1 -- 172.21.15.60:0/2585229713 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c031930 con 0x7f08740cac20 2023-10-16T13:51:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f0879b78640 1 -- 172.21.15.60:0/2585229713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 msgr2=0x7f08740b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f0879b78640 1 --2- 172.21.15.60:0/2585229713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f08740b07f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f086c0097a0 tx=0x7f086c02b840 comp rx=0 tx=0).stop 2023-10-16T13:51:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f0879b78640 1 -- 172.21.15.60:0/2585229713 shutdown_connections 2023-10-16T13:51:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f0879b78640 1 --2- 172.21.15.60:0/2585229713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f08740b07f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f0879b78640 1 --2- 172.21.15.60:0/2585229713 >> v2:172.21.15.144:3300/0 conn(0x7f087415ac10 0x7f087406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.174+0000 7f0879b78640 1 -- 172.21.15.60:0/2585229713 >> 172.21.15.60:0/2585229713 conn(0x7f08740cb360 msgr2=0x7f08740cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.175+0000 7f0879b78640 1 -- 172.21.15.60:0/2585229713 shutdown_connections 2023-10-16T13:51:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.175+0000 7f0879b78640 1 -- 172.21.15.60:0/2585229713 wait complete. 2023-10-16T13:51:14.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.176+0000 7f0879b78640 1 Processor -- start 2023-10-16T13:51:14.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.177+0000 7f0879b78640 1 -- start start 2023-10-16T13:51:14.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.177+0000 7f0879b78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08740b0360 0x7f08741f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:14.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.177+0000 7f0879b78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f0874150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:14.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.177+0000 7f0879b78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 0x7f08740c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:14.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0879b78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08741512e0 con 0x7f087415ac10 2023-10-16T13:51:14.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0879b78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08740cc7c0 con 0x7f08740b0360 2023-10-16T13:51:14.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0879b78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08740c6890 con 0x7f08740cac20 2023-10-16T13:51:14.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 0x7f08740c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:14.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 0x7f08740c6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40474/0 (socket says 172.21.15.60:40474) 2023-10-16T13:51:14.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 -- 172.21.15.60:0/3663405994 learned_addr learned my addr 172.21.15.60:0/3663405994 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:14.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0872ffd640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f0874150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:14.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f08737fe640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08740b0360 0x7f08741f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:14.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 -- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 msgr2=0x7f0874150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:14.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f0874150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 -- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08740b0360 msgr2=0x7f08741f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:14.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08740b0360 0x7f08741f6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0873fff640 1 -- 172.21.15.60:0/3663405994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c009400 con 0x7f087415ac10 2023-10-16T13:51:14.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.178+0000 7f0872ffd640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f0874150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:14.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.179+0000 7f0873fff640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 0x7f08740c6350 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f086c0098c0 tx=0x7f086c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:14.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.179+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c032840 con 0x7f087415ac10 2023-10-16T13:51:14.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.179+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c0329a0 con 0x7f087415ac10 2023-10-16T13:51:14.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.179+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c032d30 con 0x7f087415ac10 2023-10-16T13:51:14.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.179+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08740c6b10 con 0x7f087415ac10 2023-10-16T13:51:14.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.179+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08740cca10 con 0x7f087415ac10 2023-10-16T13:51:14.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.180+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087404e160 con 0x7f087415ac10 2023-10-16T13:51:14.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.181+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f086c04e430 con 0x7f087415ac10 2023-10-16T13:51:14.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.182+0000 7f086bfff640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f085406b010 0x7f085406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:14.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.182+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f086c0aba40 con 0x7f087415ac10 2023-10-16T13:51:14.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.183+0000 7f08737fe640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f085406b010 0x7f085406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:14.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.184+0000 7f08737fe640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f085406b010 0x7f085406d4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0864009ce0 tx=0x7f086400b040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:14.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.186+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f086c076630 con 0x7f087415ac10 2023-10-16T13:51:14.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.382+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f087406c2e0 con 0x7f087415ac10 2023-10-16T13:51:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.921+0000 7f086bfff640 1 -- 172.21.15.60:0/3663405994 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v187) v1 ==== 181+0+0 (secure 0 0 0) 0x7f086c079c50 con 0x7f087415ac10 2023-10-16T13:51:14.923 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:51:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f085406b010 msgr2=0x7f085406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:14.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f085406b010 0x7f085406d4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0864009ce0 tx=0x7f086400b040 comp rx=0 tx=0).stop 2023-10-16T13:51:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 msgr2=0x7f08740c6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 0x7f08740c6350 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f086c0098c0 tx=0x7f086c0046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 shutdown_connections 2023-10-16T13:51:14.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f085406b010 0x7f085406d4d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08740cac20 0x7f0874150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f087415ac10 0x7f08740c6350 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 --2- 172.21.15.60:0/3663405994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08740b0360 0x7f08741f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:14.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.928+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 >> 172.21.15.60:0/3663405994 conn(0x7f08740cb360 msgr2=0x7f087406bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.929+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 shutdown_connections 2023-10-16T13:51:14.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:14.929+0000 7f0879b78640 1 -- 172.21.15.60:0/3663405994 wait complete. 2023-10-16T13:51:14.952 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:51:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.178+0000 7f209a9a9640 1 Processor -- start 2023-10-16T13:51:15.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.179+0000 7f209a9a9640 1 -- start start 2023-10-16T13:51:15.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f209a9a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20941599c0 0x7f20940c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f209a9a9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2094159da0 0x7f209415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f209a9a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f209406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f209a9a9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209415dd10 con 0x7f2094159da0 2023-10-16T13:51:15.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f209a9a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209415dfb0 con 0x7f20941599c0 2023-10-16T13:51:15.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f209a9a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209415e290 con 0x7f20940cce20 2023-10-16T13:51:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.180+0000 7f20937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2094159da0 0x7f209415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2094159da0 0x7f209415d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40490/0 (socket says 172.21.15.60:40490) 2023-10-16T13:51:15.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 -- 172.21.15.60:0/1027239336 learned_addr learned my addr 172.21.15.60:0/1027239336 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f2093fff640 1 --2- 172.21.15.60:0/1027239336 >> v2:172.21.15.144:3300/0 conn(0x7f20941599c0 0x7f20940c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f2098f46640 1 --2- 172.21.15.60:0/1027239336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f209406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 -- 172.21.15.60:0/1027239336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 msgr2=0x7f209406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 --2- 172.21.15.60:0/1027239336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f209406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 -- 172.21.15.60:0/1027239336 >> v2:172.21.15.144:3300/0 conn(0x7f20941599c0 msgr2=0x7f20940c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 --2- 172.21.15.60:0/1027239336 >> v2:172.21.15.144:3300/0 conn(0x7f20941599c0 0x7f20940c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.181+0000 7f20937fe640 1 -- 172.21.15.60:0/1027239336 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209415e570 con 0x7f2094159da0 2023-10-16T13:51:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.183+0000 7f20937fe640 1 --2- 172.21.15.60:0/1027239336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f209415d7d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f209406d730 tx=0x7f207c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d8910db6d47229 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.183+0000 7f2092ffd640 1 -- 172.21.15.60:0/1027239336 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c031860 con 0x7f2094159da0 2023-10-16T13:51:15.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.183+0000 7f2092ffd640 1 -- 172.21.15.60:0/1027239336 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c0319c0 con 0x7f2094159da0 2023-10-16T13:51:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.183+0000 7f2092ffd640 1 -- 172.21.15.60:0/1027239336 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c031d50 con 0x7f2094159da0 2023-10-16T13:51:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 -- 172.21.15.60:0/1027239336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 msgr2=0x7f209415d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 --2- 172.21.15.60:0/1027239336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f209415d7d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f209406d730 tx=0x7f207c02b900 comp rx=0 tx=0).stop 2023-10-16T13:51:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 -- 172.21.15.60:0/1027239336 shutdown_connections 2023-10-16T13:51:15.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 --2- 172.21.15.60:0/1027239336 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f209406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 --2- 172.21.15.60:0/1027239336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f209415d7d0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 --2- 172.21.15.60:0/1027239336 >> v2:172.21.15.144:3300/0 conn(0x7f20941599c0 0x7f20940c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 -- 172.21.15.60:0/1027239336 >> 172.21.15.60:0/1027239336 conn(0x7f20940cb360 msgr2=0x7f20940cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 -- 172.21.15.60:0/1027239336 shutdown_connections 2023-10-16T13:51:15.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.184+0000 7f209a9a9640 1 -- 172.21.15.60:0/1027239336 wait complete. 2023-10-16T13:51:15.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.186+0000 7f209a9a9640 1 Processor -- start 2023-10-16T13:51:15.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.186+0000 7f209a9a9640 1 -- start start 2023-10-16T13:51:15.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f209a9a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f20940c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f209a9a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20941599c0 0x7f20941ede50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f209a9a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f20940c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f209a9a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20940c7b20 con 0x7f2094159da0 2023-10-16T13:51:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f209a9a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20940c7f40 con 0x7f20941599c0 2023-10-16T13:51:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f209a9a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20940c6ea0 con 0x7f20940cce20 2023-10-16T13:51:15.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f2098f46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f20940c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f2098f46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f20940c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40506/0 (socket says 172.21.15.60:40506) 2023-10-16T13:51:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.187+0000 7f2098f46640 1 -- 172.21.15.60:0/4114707665 learned_addr learned my addr 172.21.15.60:0/4114707665 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2093fff640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f20940c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f20937fe640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20941599c0 0x7f20941ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2098f46640 1 -- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 msgr2=0x7f20940c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2098f46640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f20940c68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2098f46640 1 -- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20941599c0 msgr2=0x7f20941ede50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2098f46640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20941599c0 0x7f20941ede50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2098f46640 1 -- 172.21.15.60:0/4114707665 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c0094d0 con 0x7f2094159da0 2023-10-16T13:51:15.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2093fff640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f20940c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2098f46640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f20940c3340 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f208c00d5d0 tx=0x7f208c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c0125f0 con 0x7f2094159da0 2023-10-16T13:51:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c012770 con 0x7f2094159da0 2023-10-16T13:51:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c012b20 con 0x7f2094159da0 2023-10-16T13:51:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.188+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20940c7180 con 0x7f2094159da0 2023-10-16T13:51:15.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.189+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20940c7740 con 0x7f2094159da0 2023-10-16T13:51:15.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.190+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209404e160 con 0x7f2094159da0 2023-10-16T13:51:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.191+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f208c012c80 con 0x7f2094159da0 2023-10-16T13:51:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.192+0000 7f2090ff9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f207406af90 0x7f207406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.192+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6045+0+0 (secure 0 0 0) 0x7f208c089db0 con 0x7f2094159da0 2023-10-16T13:51:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.192+0000 7f2093fff640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f207406af90 0x7f207406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:15.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.194+0000 7f2093fff640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f207406af90 0x7f207406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2084009ed0 tx=0x7f208400b040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:15.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.196+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f208c054c10 con 0x7f2094159da0 2023-10-16T13:51:15.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.396+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f20940c5670 con 0x7f2094159da0 2023-10-16T13:51:15.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.936+0000 7f2090ff9640 1 -- 172.21.15.60:0/4114707665 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v188) v1 ==== 169+0+0 (secure 0 0 0) 0x7f208c058230 con 0x7f2094159da0 2023-10-16T13:51:15.939 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T13:51:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.941+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f207406af90 msgr2=0x7f207406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.941+0000 7f209a9a9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f207406af90 0x7f207406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2084009ed0 tx=0x7f208400b040 comp rx=0 tx=0).stop 2023-10-16T13:51:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.941+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 msgr2=0x7f20940c3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.941+0000 7f209a9a9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f20940c3340 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f208c00d5d0 tx=0x7f208c00da90 comp rx=0 tx=0).stop 2023-10-16T13:51:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 shutdown_connections 2023-10-16T13:51:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f207406af90 0x7f207406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159da0 0x7f20940c3340 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20941599c0 0x7f20941ede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 --2- 172.21.15.60:0/4114707665 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f20940cce20 0x7f20940c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 >> 172.21.15.60:0/4114707665 conn(0x7f20940cb360 msgr2=0x7f209406bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 shutdown_connections 2023-10-16T13:51:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:15.943+0000 7f209a9a9640 1 -- 172.21.15.60:0/4114707665 wait complete. 2023-10-16T13:51:15.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:51:16.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.192+0000 7f571e17c640 1 Processor -- start 2023-10-16T13:51:16.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.192+0000 7f571e17c640 1 -- start start 2023-10-16T13:51:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.193+0000 7f571e17c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f571815ae80 0x7f571815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.193+0000 7f571e17c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180707e0 con 0x7f57180ce390 2023-10-16T13:51:16.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.193+0000 7f571e17c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571806b7a0 con 0x7f571815aaa0 2023-10-16T13:51:16.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.193+0000 7f571e17c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571815ef20 con 0x7f571815ae80 2023-10-16T13:51:16.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.193+0000 7f5717fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f57180ce390 0x7f5718070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56410/0 (socket says 172.21.15.60:56410) 2023-10-16T13:51:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.193+0000 7f5717fff640 1 -- 172.21.15.60:0/1661958522 learned_addr learned my addr 172.21.15.60:0/1661958522 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f5716ffd640 1 --2- 172.21.15.60:0/1661958522 >> v2:172.21.15.144:3300/0 conn(0x7f571815ae80 0x7f571815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861888643 0 0) 0x7f57180707e0 con 0x7f57180ce390 2023-10-16T13:51:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700003520 con 0x7f57180ce390 2023-10-16T13:51:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 227886298 0 0) 0x7f5700003520 con 0x7f57180ce390 2023-10-16T13:51:16.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 >> v2:172.21.15.144:3300/0 conn(0x7f571815ae80 msgr2=0x7f571815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:16.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 --2- 172.21.15.60:0/1661958522 >> v2:172.21.15.144:3300/0 conn(0x7f571815ae80 0x7f571815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 >> v1:172.21.15.144:6789/0 conn(0x7f571815aaa0 legacy=0x7f57180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571815f200 con 0x7f57180ce390 2023-10-16T13:51:16.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f57100019f0 con 0x7f57180ce390 2023-10-16T13:51:16.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 >> v1:172.21.15.60:6789/0 conn(0x7f57180ce390 legacy=0x7f5718070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57000039c0 0x7f5700003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57000047c0 0x7f5700004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f570000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571815ef20 con 0x7f57000051d0 2023-10-16T13:51:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570000c120 con 0x7f57000039c0 2023-10-16T13:51:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.194+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570000c2e0 con 0x7f57000047c0 2023-10-16T13:51:16.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5717fff640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f570000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f57177fe640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57000039c0 0x7f5700003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5716ffd640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57000047c0 0x7f5700004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5717fff640 1 -- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57000047c0 msgr2=0x7f5700004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:16.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5717fff640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57000047c0 0x7f5700004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5717fff640 1 -- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57000039c0 msgr2=0x7f5700003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:16.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5717fff640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57000039c0 0x7f5700003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f5717fff640 1 -- 172.21.15.60:0/1661958522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5700003520 con 0x7f57000051d0 2023-10-16T13:51:16.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.195+0000 7f57177fe640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57000039c0 0x7f5700003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:16.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.196+0000 7f5717fff640 1 -- 172.21.15.60:0/1661958522 reap_dead start 2023-10-16T13:51:16.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.196+0000 7f5717fff640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f570000b980 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5710002230 tx=0x7f57000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41cdb299481008f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:16.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710034630 con 0x7f57000051d0 2023-10-16T13:51:16.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571002f070 con 0x7f57000051d0 2023-10-16T13:51:16.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f57167fc640 1 -- 172.21.15.60:0/1661958522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100349d0 con 0x7f57000051d0 2023-10-16T13:51:16.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f571e17c640 1 -- 172.21.15.60:0/1661958522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 msgr2=0x7f570000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f571e17c640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f570000b980 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5710002230 tx=0x7f57000057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f571e17c640 1 -- 172.21.15.60:0/1661958522 shutdown_connections 2023-10-16T13:51:16.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f571e17c640 1 --2- 172.21.15.60:0/1661958522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f570000b980 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f571e17c640 1 -- 172.21.15.60:0/1661958522 >> 172.21.15.60:0/1661958522 conn(0x7f57180cb410 msgr2=0x7f57180cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:16.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.197+0000 7f571e17c640 1 -- 172.21.15.60:0/1661958522 shutdown_connections 2023-10-16T13:51:16.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.198+0000 7f571e17c640 1 -- 172.21.15.60:0/1661958522 wait complete. 2023-10-16T13:51:16.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.199+0000 7f571e17c640 1 Processor -- start 2023-10-16T13:51:16.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.199+0000 7f571e17c640 1 -- start start 2023-10-16T13:51:16.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.200+0000 7f571e17c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f5718155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.200+0000 7f571e17c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57180c3f20 0x7f57180cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.200+0000 7f571e17c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f571815d360 0x7f571815d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.200+0000 7f571e17c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571815dd30 con 0x7f57000051d0 2023-10-16T13:51:16.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.200+0000 7f571e17c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718156660 con 0x7f57180c3f20 2023-10-16T13:51:16.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.200+0000 7f571e17c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180cd340 con 0x7f571815d360 2023-10-16T13:51:16.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f5718155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f5718155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40534/0 (socket says 172.21.15.60:40534) 2023-10-16T13:51:16.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 -- 172.21.15.60:0/2620584157 learned_addr learned my addr 172.21.15.60:0/2620584157 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:16.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 -- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f571815d360 msgr2=0x7f571815d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f571815d360 0x7f571815d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 -- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57180c3f20 msgr2=0x7f57180cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57180c3f20 0x7f57180cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 -- 172.21.15.60:0/2620584157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180c4420 con 0x7f57000051d0 2023-10-16T13:51:16.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.201+0000 7f57177fe640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f5718155120 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f570800e410 tx=0x7f570800e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:16.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.202+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57080052e0 con 0x7f57000051d0 2023-10-16T13:51:16.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.202+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708005440 con 0x7f57000051d0 2023-10-16T13:51:16.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.202+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708003830 con 0x7f57000051d0 2023-10-16T13:51:16.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.202+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57180c4700 con 0x7f57000051d0 2023-10-16T13:51:16.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.202+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57180c4cc0 con 0x7f57000051d0 2023-10-16T13:51:16.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.203+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57180ce200 con 0x7f57000051d0 2023-10-16T13:51:16.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.204+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5708003990 con 0x7f57000051d0 2023-10-16T13:51:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.205+0000 7f56fffff640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f570406af90 0x7f570406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.205+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f5708088e80 con 0x7f57000051d0 2023-10-16T13:51:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.205+0000 7f5716ffd640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f570406af90 0x7f570406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.207+0000 7f5716ffd640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f570406af90 0x7f570406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f571815cb90 tx=0x7f570c008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:16.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.209+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5708053f10 con 0x7f57000051d0 2023-10-16T13:51:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.406+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f571804e160 con 0x7f57000051d0 2023-10-16T13:51:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.413+0000 7f56fffff640 1 -- 172.21.15.60:0/2620584157 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5708019020 con 0x7f57000051d0 2023-10-16T13:51:16.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f570406af90 msgr2=0x7f570406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f570406af90 0x7f570406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f571815cb90 tx=0x7f570c008040 comp rx=0 tx=0).stop 2023-10-16T13:51:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 msgr2=0x7f5718155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f5718155120 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f570800e410 tx=0x7f570800e8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 shutdown_connections 2023-10-16T13:51:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f570406af90 0x7f570406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f571815d360 0x7f571815d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.416+0000 7f571e17c640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57180c3f20 0x7f57180cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.417+0000 7f571e17c640 1 --2- 172.21.15.60:0/2620584157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57000051d0 0x7f5718155120 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.417+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 >> 172.21.15.60:0/2620584157 conn(0x7f57180cb410 msgr2=0x7f571806f880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.417+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 shutdown_connections 2023-10-16T13:51:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.417+0000 7f571e17c640 1 -- 172.21.15.60:0/2620584157 wait complete. 2023-10-16T13:51:16.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:51:16.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.666+0000 7f5da3fff640 1 Processor -- start 2023-10-16T13:51:16.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.666+0000 7f5da3fff640 1 -- start start 2023-10-16T13:51:16.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5da40ad970 0x7f5da40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5da40ccfb0 0x7f5da40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da3fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da415b9d0 con 0x7f5da406c040 2023-10-16T13:51:16.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da3fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da415bcb0 con 0x7f5da40ccfb0 2023-10-16T13:51:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da3fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da415bf90 con 0x7f5da40ad970 2023-10-16T13:51:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5da40ccfb0 0x7f5da40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5da40ccfb0 0x7f5da40cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40538/0 (socket says 172.21.15.60:40538) 2023-10-16T13:51:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da37fe640 1 -- 172.21.15.60:0/3596659778 learned_addr learned my addr 172.21.15.60:0/3596659778 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.668+0000 7f5da27fc640 1 --2- 172.21.15.60:0/3596659778 >> v2:172.21.15.144:3300/0 conn(0x7f5da40ad970 0x7f5da40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.669+0000 7f5da37fe640 1 -- 172.21.15.60:0/3596659778 >> v2:172.21.15.144:3300/0 conn(0x7f5da40ad970 msgr2=0x7f5da40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.669+0000 7f5da37fe640 1 --2- 172.21.15.60:0/3596659778 >> v2:172.21.15.144:3300/0 conn(0x7f5da40ad970 0x7f5da40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.669+0000 7f5da37fe640 1 -- 172.21.15.60:0/3596659778 >> v1:172.21.15.144:6789/0 conn(0x7f5da406c040 legacy=0x7f5da406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.669+0000 7f5da37fe640 1 -- 172.21.15.60:0/3596659778 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40cc270 con 0x7f5da40ccfb0 2023-10-16T13:51:16.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.670+0000 7f5da37fe640 1 --2- 172.21.15.60:0/3596659778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da40ccfb0 0x7f5da40cd460 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009870 tx=0x7f5d9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0da11f6167115a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:16.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.670+0000 7f5da1ffb640 1 -- 172.21.15.60:0/3596659778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c031870 con 0x7f5da40ccfb0 2023-10-16T13:51:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.670+0000 7f5da1ffb640 1 -- 172.21.15.60:0/3596659778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c0319d0 con 0x7f5da40ccfb0 2023-10-16T13:51:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.670+0000 7f5da1ffb640 1 -- 172.21.15.60:0/3596659778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c031d60 con 0x7f5da40ccfb0 2023-10-16T13:51:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 -- 172.21.15.60:0/3596659778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da40ccfb0 msgr2=0x7f5da40cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 --2- 172.21.15.60:0/3596659778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da40ccfb0 0x7f5da40cd460 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009870 tx=0x7f5d9c02b910 comp rx=0 tx=0).stop 2023-10-16T13:51:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 -- 172.21.15.60:0/3596659778 shutdown_connections 2023-10-16T13:51:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 --2- 172.21.15.60:0/3596659778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da40ccfb0 0x7f5da40cd460 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 --2- 172.21.15.60:0/3596659778 >> v2:172.21.15.144:3300/0 conn(0x7f5da40ad970 0x7f5da40cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 -- 172.21.15.60:0/3596659778 >> 172.21.15.60:0/3596659778 conn(0x7f5da40ce6a0 msgr2=0x7f5da40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 -- 172.21.15.60:0/3596659778 shutdown_connections 2023-10-16T13:51:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.671+0000 7f5da3fff640 1 -- 172.21.15.60:0/3596659778 wait complete. 2023-10-16T13:51:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.673+0000 7f5da3fff640 1 Processor -- start 2023-10-16T13:51:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.673+0000 7f5da3fff640 1 -- start start 2023-10-16T13:51:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 0x7f5da40aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5da40ad970 0x7f5da41f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5da40ccfb0 0x7f5da40b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da3fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40c8cf0 con 0x7f5da406c040 2023-10-16T13:51:16.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da3fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40c8fd0 con 0x7f5da40ad970 2023-10-16T13:51:16.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40c92b0 con 0x7f5da40ccfb0 2023-10-16T13:51:16.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 0x7f5da40aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 0x7f5da40aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40544/0 (socket says 172.21.15.60:40544) 2023-10-16T13:51:16.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da2ffd640 1 -- 172.21.15.60:0/2345845477 learned_addr learned my addr 172.21.15.60:0/2345845477 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:16.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.674+0000 7f5da27fc640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5da40ad970 0x7f5da41f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da2ffd640 1 -- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5da40ccfb0 msgr2=0x7f5da40b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da2ffd640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5da40ccfb0 0x7f5da40b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da2ffd640 1 -- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5da40ad970 msgr2=0x7f5da41f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da2ffd640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5da40ad970 0x7f5da41f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da2ffd640 1 -- 172.21.15.60:0/2345845477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c0094d0 con 0x7f5da406c040 2023-10-16T13:51:16.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da27fc640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5da40ad970 0x7f5da41f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:16.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da2ffd640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 0x7f5da40aeb10 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400c7c0 tx=0x7f5d9400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:16.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9400aa50 con 0x7f5da406c040 2023-10-16T13:51:16.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9400abb0 con 0x7f5da406c040 2023-10-16T13:51:16.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94012420 con 0x7f5da406c040 2023-10-16T13:51:16.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da40c9590 con 0x7f5da406c040 2023-10-16T13:51:16.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.675+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da40af050 con 0x7f5da406c040 2023-10-16T13:51:16.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.677+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da406b330 con 0x7f5da406c040 2023-10-16T13:51:16.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.677+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d94012580 con 0x7f5da406c040 2023-10-16T13:51:16.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.678+0000 7f5d8b7fe640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5d9006af90 0x7f5d9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:16.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.678+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f5d94089a40 con 0x7f5da406c040 2023-10-16T13:51:16.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.679+0000 7f5da27fc640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5d9006af90 0x7f5d9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:16.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.680+0000 7f5da27fc640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5d9006af90 0x7f5d9006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5da415f080 tx=0x7f5d98008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:16.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.683+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d94054990 con 0x7f5da406c040 2023-10-16T13:51:16.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.878+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5da404e9b0 con 0x7f5da406c040 2023-10-16T13:51:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.884+0000 7f5d8b7fe640 1 -- 172.21.15.60:0/2345845477 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5d94057fb0 con 0x7f5da406c040 2023-10-16T13:51:16.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5d9006af90 msgr2=0x7f5d9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5d9006af90 0x7f5d9006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5da415f080 tx=0x7f5d98008040 comp rx=0 tx=0).stop 2023-10-16T13:51:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 msgr2=0x7f5da40aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 0x7f5da40aeb10 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400c7c0 tx=0x7f5d9400cc80 comp rx=0 tx=0).stop 2023-10-16T13:51:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 shutdown_connections 2023-10-16T13:51:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5d9006af90 0x7f5d9006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5da40ccfb0 0x7f5da40b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5da40ad970 0x7f5da41f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 --2- 172.21.15.60:0/2345845477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5da406c040 0x7f5da40aeb10 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 >> 172.21.15.60:0/2345845477 conn(0x7f5da40ce6a0 msgr2=0x7f5da40c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 shutdown_connections 2023-10-16T13:51:16.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:16.887+0000 7f5da3fff640 1 -- 172.21.15.60:0/2345845477 wait complete. 2023-10-16T13:51:16.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:51:17.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.129+0000 7fb85b7f5640 1 Processor -- start 2023-10-16T13:51:17.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.129+0000 7fb85b7f5640 1 -- start start 2023-10-16T13:51:17.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.130+0000 7fb85b7f5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540b3420 0x7fb8540b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb85b7f5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb8540cfaf0 0x7fb8540b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb85b7f5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8540d3980 con 0x7fb8540cf710 2023-10-16T13:51:17.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb85b7f5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8540d3c60 con 0x7fb8540cfaf0 2023-10-16T13:51:17.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb85b7f5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540ce630 con 0x7fb8540b3420 2023-10-16T13:51:17.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb858d90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540b3420 0x7fb8540b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb859d92640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb8540cfaf0 0x7fb8540b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb858d90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540b3420 0x7fb8540b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:53990/0 (socket says 172.21.15.60:53990) 2023-10-16T13:51:17.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.131+0000 7fb858d90640 1 -- 172.21.15.60:0/925040196 learned_addr learned my addr 172.21.15.60:0/925040196 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:17.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.132+0000 7fb859d92640 1 -- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540b3420 msgr2=0x7fb8540b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:17.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.132+0000 7fb859d92640 1 --2- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540b3420 0x7fb8540b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.132+0000 7fb859d92640 1 -- 172.21.15.60:0/925040196 >> v1:172.21.15.144:6789/0 conn(0x7fb8540cf710 legacy=0x7fb8540b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.132+0000 7fb859d92640 1 -- 172.21.15.60:0/925040196 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8540ce910 con 0x7fb8540cfaf0 2023-10-16T13:51:17.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.133+0000 7fb859d92640 1 --2- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb8540b6200 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009d90 tx=0x7fb84c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b47159072955174d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:17.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.134+0000 7fb84bfff640 1 -- 172.21.15.60:0/925040196 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c02cbc0 con 0x7fb8540cfaf0 2023-10-16T13:51:17.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.134+0000 7fb84bfff640 1 -- 172.21.15.60:0/925040196 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c02cd20 con 0x7fb8540cfaf0 2023-10-16T13:51:17.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.134+0000 7fb84bfff640 1 -- 172.21.15.60:0/925040196 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c034660 con 0x7fb8540cfaf0 2023-10-16T13:51:17.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 -- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 msgr2=0x7fb8540b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb8540b6200 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009d90 tx=0x7fb84c02be10 comp rx=0 tx=0).stop 2023-10-16T13:51:17.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 -- 172.21.15.60:0/925040196 shutdown_connections 2023-10-16T13:51:17.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb8540b6200 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/925040196 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540b3420 0x7fb8540b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 -- 172.21.15.60:0/925040196 >> 172.21.15.60:0/925040196 conn(0x7fb8540d2cc0 msgr2=0x7fb8540d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:17.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 -- 172.21.15.60:0/925040196 shutdown_connections 2023-10-16T13:51:17.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.135+0000 7fb85b7f5640 1 -- 172.21.15.60:0/925040196 wait complete. 2023-10-16T13:51:17.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.137+0000 7fb85b7f5640 1 Processor -- start 2023-10-16T13:51:17.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.138+0000 7fb85b7f5640 1 -- start start 2023-10-16T13:51:17.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb85b7f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 0x7fb8541f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb85b7f5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540cf710 0x7fb8541f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb85b7f5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb85409fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb85b7f5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb854155120 con 0x7fb8540b3420 2023-10-16T13:51:17.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb85b7f5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8540b2800 con 0x7fb8540cfaf0 2023-10-16T13:51:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb85b7f5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8541f6be0 con 0x7fb8540cf710 2023-10-16T13:51:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb859591640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 0x7fb8541f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb859591640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 0x7fb8541f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40556/0 (socket says 172.21.15.60:40556) 2023-10-16T13:51:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.139+0000 7fb859591640 1 -- 172.21.15.60:0/1783970605 learned_addr learned my addr 172.21.15.60:0/1783970605 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:17.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859d92640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb85409fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb858d90640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540cf710 0x7fb8541f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859591640 1 -- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540cf710 msgr2=0x7fb8541f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859591640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540cf710 0x7fb8541f1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859591640 1 -- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 msgr2=0x7fb85409fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859591640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb85409fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859591640 1 -- 172.21.15.60:0/1783970605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c009a70 con 0x7fb8540b3420 2023-10-16T13:51:17.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859d92640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb85409fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:17.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb859591640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 0x7fb8541f66a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c3c0 tx=0x7fb85000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:17.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8500052d0 con 0x7fb8540b3420 2023-10-16T13:51:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb850005470 con 0x7fb8540b3420 2023-10-16T13:51:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8540b1bf0 con 0x7fb8540b3420 2023-10-16T13:51:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850006a60 con 0x7fb8540b3420 2023-10-16T13:51:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.140+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8540b21b0 con 0x7fb8540b3420 2023-10-16T13:51:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.141+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85404e160 con 0x7fb8540b3420 2023-10-16T13:51:17.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.143+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb850006bc0 con 0x7fb8540b3420 2023-10-16T13:51:17.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.143+0000 7fb849ffb640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb83806af90 0x7fb83806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.144+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fb850088a80 con 0x7fb8540b3420 2023-10-16T13:51:17.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.144+0000 7fb858d90640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb83806af90 0x7fb83806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.146+0000 7fb858d90640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb83806af90 0x7fb83806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb8540cc520 tx=0x7fb844008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:17.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.148+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb850053a60 con 0x7fb8540b3420 2023-10-16T13:51:17.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.344+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fb854071920 con 0x7fb8540b3420 2023-10-16T13:51:17.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.349+0000 7fb849ffb640 1 -- 172.21.15.60:0/1783970605 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb850057080 con 0x7fb8540b3420 2023-10-16T13:51:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.352+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb83806af90 msgr2=0x7fb83806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.352+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb83806af90 0x7fb83806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb8540cc520 tx=0x7fb844008040 comp rx=0 tx=0).stop 2023-10-16T13:51:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.352+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 msgr2=0x7fb8541f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.352+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 0x7fb8541f66a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c3c0 tx=0x7fb85000c880 comp rx=0 tx=0).stop 2023-10-16T13:51:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 shutdown_connections 2023-10-16T13:51:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb83806af90 0x7fb83806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb8540cf710 0x7fb8541f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb8540cfaf0 0x7fb85409fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 --2- 172.21.15.60:0/1783970605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8540b3420 0x7fb8541f66a0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 >> 172.21.15.60:0/1783970605 conn(0x7fb8540d2cc0 msgr2=0x7fb8540cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 shutdown_connections 2023-10-16T13:51:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.353+0000 7fb85b7f5640 1 -- 172.21.15.60:0/1783970605 wait complete. 2023-10-16T13:51:17.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T13:51:17.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.597+0000 7f2ec335a640 1 Processor -- start 2023-10-16T13:51:17.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.597+0000 7f2ec335a640 1 -- start start 2023-10-16T13:51:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec335a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ebc0cd0c0 0x7f2ebc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec335a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc15e3e0 con 0x7f2ebc15db00 2023-10-16T13:51:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec335a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc0cd4a0 con 0x7f2ebc0b0410 2023-10-16T13:51:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec335a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc15e100 con 0x7f2ebc0cd0c0 2023-10-16T13:51:17.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec18f7640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2ebc15db00 0x7f2ebc06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56426/0 (socket says 172.21.15.60:56426) 2023-10-16T13:51:17.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec18f7640 1 -- 172.21.15.60:0/1022495103 learned_addr learned my addr 172.21.15.60:0/1022495103 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.599+0000 7f2ec10f6640 1 --2- 172.21.15.60:0/1022495103 >> v2:172.21.15.144:3300/0 conn(0x7f2ebc0cd0c0 0x7f2ebc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920597554 0 0) 0x7f2ebc15e3e0 con 0x7f2ebc15db00 2023-10-16T13:51:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea8003520 con 0x7f2ebc15db00 2023-10-16T13:51:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371590805 0 0) 0x7f2ea8003520 con 0x7f2ebc15db00 2023-10-16T13:51:17.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 >> v2:172.21.15.144:3300/0 conn(0x7f2ebc0cd0c0 msgr2=0x7f2ebc1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:17.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 --2- 172.21.15.60:0/1022495103 >> v2:172.21.15.144:3300/0 conn(0x7f2ebc0cd0c0 0x7f2ebc1615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 >> v1:172.21.15.144:6789/0 conn(0x7f2ebc0b0410 legacy=0x7f2ebc15d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc160b10 con 0x7f2ebc15db00 2023-10-16T13:51:17.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2eb40019f0 con 0x7f2ebc15db00 2023-10-16T13:51:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 >> v1:172.21.15.60:6789/0 conn(0x7f2ebc15db00 legacy=0x7f2ebc06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80039c0 0x7f2ea8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea80047c0 0x7f2ea8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ea80051d0 0x7f2ea800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc15e100 con 0x7f2ea80051d0 2023-10-16T13:51:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea800c120 con 0x7f2ea80047c0 2023-10-16T13:51:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.600+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea800c2e0 con 0x7f2ea80039c0 2023-10-16T13:51:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ea80051d0 0x7f2ea800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec08f5640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea80047c0 0x7f2ea8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec10f6640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80039c0 0x7f2ea8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec18f7640 1 -- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80039c0 msgr2=0x7f2ea8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80039c0 0x7f2ea8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec18f7640 1 -- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea80047c0 msgr2=0x7f2ea8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea80047c0 0x7f2ea8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec18f7640 1 -- 172.21.15.60:0/1022495103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8003520 con 0x7f2ea80051d0 2023-10-16T13:51:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.601+0000 7f2ec08f5640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea80047c0 0x7f2ea8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.602+0000 7f2ec18f7640 1 -- 172.21.15.60:0/1022495103 reap_dead start 2023-10-16T13:51:17.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.602+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ea80051d0 0x7f2ea800b980 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4002230 tx=0x7f2ea80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c44133a8186ea3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.602+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4034630 con 0x7f2ea80051d0 2023-10-16T13:51:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb402f070 con 0x7f2ea80051d0 2023-10-16T13:51:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2eb3fff640 1 -- 172.21.15.60:0/1022495103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb40349d0 con 0x7f2ea80051d0 2023-10-16T13:51:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 -- 172.21.15.60:0/1022495103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ea80051d0 msgr2=0x7f2ea800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ea80051d0 0x7f2ea800b980 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4002230 tx=0x7f2ea80057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 -- 172.21.15.60:0/1022495103 shutdown_connections 2023-10-16T13:51:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 --2- 172.21.15.60:0/1022495103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ea80051d0 0x7f2ea800b980 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 -- 172.21.15.60:0/1022495103 >> 172.21.15.60:0/1022495103 conn(0x7f2ebc0cb410 msgr2=0x7f2ebc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 -- 172.21.15.60:0/1022495103 shutdown_connections 2023-10-16T13:51:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.603+0000 7f2ec335a640 1 -- 172.21.15.60:0/1022495103 wait complete. 2023-10-16T13:51:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.605+0000 7f2ec335a640 1 Processor -- start 2023-10-16T13:51:17.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.605+0000 7f2ec335a640 1 -- start start 2023-10-16T13:51:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec335a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80051d0 0x7f2ebc1f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec335a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0b1860 0x7f2ebc0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec335a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 0x7f2ebc06cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec335a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0ca160 con 0x7f2ebc0b3120 2023-10-16T13:51:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec335a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc1f3460 con 0x7f2ebc0b1860 2023-10-16T13:51:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec335a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c98c0 con 0x7f2ea80051d0 2023-10-16T13:51:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.606+0000 7f2ec18f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 0x7f2ebc06cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 0x7f2ebc06cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40570/0 (socket says 172.21.15.60:40570) 2023-10-16T13:51:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 -- 172.21.15.60:0/979083830 learned_addr learned my addr 172.21.15.60:0/979083830 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec10f6640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80051d0 0x7f2ebc1f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec08f5640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0b1860 0x7f2ebc0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 -- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80051d0 msgr2=0x7f2ebc1f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80051d0 0x7f2ebc1f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 -- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0b1860 msgr2=0x7f2ebc0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0b1860 0x7f2ebc0c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 -- 172.21.15.60:0/979083830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0c9ba0 con 0x7f2ebc0b3120 2023-10-16T13:51:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec10f6640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80051d0 0x7f2ebc1f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.607+0000 7f2ec18f7640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 0x7f2ebc06cad0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4002230 tx=0x7f2eb4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.608+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4042070 con 0x7f2ebc0b3120 2023-10-16T13:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.608+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb40017f0 con 0x7f2ebc0b3120 2023-10-16T13:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.608+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4034430 con 0x7f2ebc0b3120 2023-10-16T13:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.608+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0c9e80 con 0x7f2ebc0b3120 2023-10-16T13:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.608+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc0c9500 con 0x7f2ebc0b3120 2023-10-16T13:51:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.609+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc154850 con 0x7f2ebc0b3120 2023-10-16T13:51:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.610+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2eb4031030 con 0x7f2ebc0b3120 2023-10-16T13:51:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.611+0000 7f2eb1ffb640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2ea006af90 0x7f2ea006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.611+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f2eb40ab010 con 0x7f2ebc0b3120 2023-10-16T13:51:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.612+0000 7f2ec10f6640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2ea006af90 0x7f2ea006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.613+0000 7f2ec10f6640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2ea006af90 0x7f2ea006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80099e0 tx=0x7f2eb8008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.615+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ebc154850 con 0x7f2ebc0b3120 2023-10-16T13:51:17.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.885+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f2ebc154c30 con 0x7f2ebc0b3120 2023-10-16T13:51:17.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.941+0000 7f2eb1ffb640 1 -- 172.21.15.60:0/979083830 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2556) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2ebc154c30 con 0x7f2ebc0b3120 2023-10-16T13:51:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2ea006af90 msgr2=0x7f2ea006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2ea006af90 0x7f2ea006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80099e0 tx=0x7f2eb8008040 comp rx=0 tx=0).stop 2023-10-16T13:51:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 msgr2=0x7f2ebc06cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 0x7f2ebc06cad0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4002230 tx=0x7f2eb4001cb0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 shutdown_connections 2023-10-16T13:51:17.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2ea006af90 0x7f2ea006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ebc0b3120 0x7f2ebc06cad0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ebc0b1860 0x7f2ebc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 --2- 172.21.15.60:0/979083830 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ea80051d0 0x7f2ebc1f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.944+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 >> 172.21.15.60:0/979083830 conn(0x7f2ebc0cb410 msgr2=0x7f2ebc15b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:17.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.945+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 shutdown_connections 2023-10-16T13:51:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:17.945+0000 7f2ec335a640 1 -- 172.21.15.60:0/979083830 wait complete. 2023-10-16T13:51:17.967 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-16T13:51:17.968 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-16T13:51:17.968 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T13:51:17.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:18.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.196+0000 7f204e733640 1 Processor -- start 2023-10-16T13:51:18.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.197+0000 7f204e733640 1 -- start start 2023-10-16T13:51:18.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.198+0000 7f204e733640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20481615f0 0x7f204815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.198+0000 7f204e733640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204815e3e0 con 0x7f20480b0b20 2023-10-16T13:51:18.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.198+0000 7f204e733640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20480cd4a0 con 0x7f20480cd0c0 2023-10-16T13:51:18.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.198+0000 7f204e733640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204815e100 con 0x7f20481615f0 2023-10-16T13:51:18.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.199+0000 7f20477fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f20480b0b20 0x7f204815d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56428/0 (socket says 172.21.15.60:56428) 2023-10-16T13:51:18.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.199+0000 7f20477fe640 1 -- 172.21.15.60:0/3381047777 learned_addr learned my addr 172.21.15.60:0/3381047777 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:18.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.199+0000 7f204ccd0640 1 --2- 172.21.15.60:0/3381047777 >> v2:172.21.15.144:3300/0 conn(0x7f20481615f0 0x7f204815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.199+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259899771 0 0) 0x7f204815e3e0 con 0x7f20480b0b20 2023-10-16T13:51:18.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.199+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030003520 con 0x7f20480b0b20 2023-10-16T13:51:18.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248184784 0 0) 0x7f2030003520 con 0x7f20480b0b20 2023-10-16T13:51:18.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 >> v2:172.21.15.144:3300/0 conn(0x7f20481615f0 msgr2=0x7f204815db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:18.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 --2- 172.21.15.60:0/3381047777 >> v2:172.21.15.144:3300/0 conn(0x7f20481615f0 0x7f204815db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 >> v1:172.21.15.144:6789/0 conn(0x7f20480cd0c0 legacy=0x7f20480b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2048160b10 con 0x7f20480b0b20 2023-10-16T13:51:18.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f203c001670 con 0x7f20480b0b20 2023-10-16T13:51:18.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 >> v1:172.21.15.60:6789/0 conn(0x7f20480b0b20 legacy=0x7f204815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2030003970 0x7f2030003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20300047c0 0x7f2030004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030005180 0x7f203000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204815e100 con 0x7f2030003970 2023-10-16T13:51:18.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203000c120 con 0x7f20300047c0 2023-10-16T13:51:18.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203000c2e0 con 0x7f2030005180 2023-10-16T13:51:18.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f2047fff640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2030003970 0x7f2030003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.200+0000 7f20477fe640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20300047c0 0x7f2030004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f204ccd0640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030005180 0x7f203000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f2047fff640 1 -- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030005180 msgr2=0x7f203000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f2047fff640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030005180 0x7f203000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f2047fff640 1 -- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20300047c0 msgr2=0x7f2030004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f2047fff640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20300047c0 0x7f2030004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f2047fff640 1 -- 172.21.15.60:0/3381047777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2030003520 con 0x7f2030003970 2023-10-16T13:51:18.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.201+0000 7f204ccd0640 1 -- 172.21.15.60:0/3381047777 reap_dead start 2023-10-16T13:51:18.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.202+0000 7f2047fff640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2030003970 0x7f2030003d90 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f2038001750 tx=0x7f20300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9d7194aaa2d799 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:18.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.202+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20380052d0 con 0x7f2030003970 2023-10-16T13:51:18.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.202+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203802e050 con 0x7f2030003970 2023-10-16T13:51:18.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.202+0000 7f2046ffd640 1 -- 172.21.15.60:0/3381047777 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038003830 con 0x7f2030003970 2023-10-16T13:51:18.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 -- 172.21.15.60:0/3381047777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2030003970 msgr2=0x7f2030003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2030003970 0x7f2030003d90 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f2038001750 tx=0x7f20300057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 -- 172.21.15.60:0/3381047777 shutdown_connections 2023-10-16T13:51:18.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 --2- 172.21.15.60:0/3381047777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2030003970 0x7f2030003d90 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 -- 172.21.15.60:0/3381047777 >> 172.21.15.60:0/3381047777 conn(0x7f20480cb410 msgr2=0x7f20480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:18.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 -- 172.21.15.60:0/3381047777 shutdown_connections 2023-10-16T13:51:18.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.203+0000 7f204e733640 1 -- 172.21.15.60:0/3381047777 wait complete. 2023-10-16T13:51:18.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.205+0000 7f204e733640 1 Processor -- start 2023-10-16T13:51:18.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.205+0000 7f204e733640 1 -- start start 2023-10-16T13:51:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204e733640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030003970 0x7f20481f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204e733640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20480c4320 0x7f20480b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204e733640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 0x7f204815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204e733640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20481f2840 con 0x7f20480b2450 2023-10-16T13:51:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204e733640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20481f25d0 con 0x7f20480c4320 2023-10-16T13:51:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204e733640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20480b28d0 con 0x7f2030003970 2023-10-16T13:51:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204ccd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 0x7f204815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204ccd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 0x7f204815f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40592/0 (socket says 172.21.15.60:40592) 2023-10-16T13:51:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f204ccd0640 1 -- 172.21.15.60:0/4025382003 learned_addr learned my addr 172.21.15.60:0/4025382003 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.206+0000 7f2047fff640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030003970 0x7f20481f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f20477fe640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20480c4320 0x7f20480b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204ccd0640 1 -- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030003970 msgr2=0x7f20481f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204ccd0640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030003970 0x7f20481f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204ccd0640 1 -- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20480c4320 msgr2=0x7f20480b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204ccd0640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20480c4320 0x7f20480b1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204ccd0640 1 -- 172.21.15.60:0/4025382003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20480b2bb0 con 0x7f20480b2450 2023-10-16T13:51:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204ccd0640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 0x7f204815f5e0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f20480c8f70 tx=0x7f20400115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:18.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204000e070 con 0x7f20480b2450 2023-10-16T13:51:18.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040011de0 con 0x7f20480b2450 2023-10-16T13:51:18.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040003990 con 0x7f20480b2450 2023-10-16T13:51:18.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.207+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20480b2e90 con 0x7f20480b2450 2023-10-16T13:51:18.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.208+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204815fb20 con 0x7f20480b2450 2023-10-16T13:51:18.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.209+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20481f2a10 con 0x7f20480b2450 2023-10-16T13:51:18.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.210+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2040003af0 con 0x7f20480b2450 2023-10-16T13:51:18.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.211+0000 7f2044ff9640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f203406af90 0x7f203406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.211+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f204008a010 con 0x7f20480b2450 2023-10-16T13:51:18.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.211+0000 7f2047fff640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f203406af90 0x7f203406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.213+0000 7f2047fff640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f203406af90 0x7f203406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2038033000 tx=0x7f2038009030 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:18.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.215+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20481f2a10 con 0x7f20480b2450 2023-10-16T13:51:18.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.413+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20481f2a10 con 0x7f20480b2450 2023-10-16T13:51:18.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.415+0000 7f2044ff9640 1 -- 172.21.15.60:0/4025382003 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+14002 (secure 0 0 0) 0x7f20481f2a10 con 0x7f20480b2450 2023-10-16T13:51:18.417 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:18.417 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":188,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:15.932834+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6834/691868089":"2023-10-17T13:51:12.909426+0000","172.21.15.60:6832/691868089":"2023-10-17T13:51:12.909426+0000","172.21.15.144:6833/1295809784":"2023-10-17T13:51:08.807418+0000","172.21.15.144:6832/1295809784":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2083028711":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2443545815":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2621417165":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/4092785021":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/1556839457":"2023-10-17T13:51:08.807418+0000","172.21.15.144:0/2455238630":"2023-10-17T13:51:01.597577+0000","172.21.15.144:6833/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3778332831":"2023-10-17T13:51:08.807418+0000","172.21.15.144:6832/2817709727":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3372584586":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1775356785":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3925300811":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/1821662703":"2023-10-17T13:51:01.597577+0000","172.21.15.144:0/3430745922":"2023-10-17T13:51:01.597577+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:18.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.417+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f203406af90 msgr2=0x7f203406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.417+0000 7f204e733640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f203406af90 0x7f203406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2038033000 tx=0x7f2038009030 comp rx=0 tx=0).stop 2023-10-16T13:51:18.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.417+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 msgr2=0x7f204815f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.417+0000 7f204e733640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 0x7f204815f5e0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f20480c8f70 tx=0x7f20400115a0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 shutdown_connections 2023-10-16T13:51:18.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f203406af90 0x7f203406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20480b2450 0x7f204815f5e0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20480c4320 0x7f20480b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 --2- 172.21.15.60:0/4025382003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2030003970 0x7f20481f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 >> 172.21.15.60:0/4025382003 conn(0x7f20480cb410 msgr2=0x7f20481558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:18.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 shutdown_connections 2023-10-16T13:51:18.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.418+0000 7f204e733640 1 -- 172.21.15.60:0/4025382003 wait complete. 2023-10-16T13:51:18.440 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:51:18.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:51:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.674+0000 7f94c781f640 1 Processor -- start 2023-10-16T13:51:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.674+0000 7f94c781f640 1 -- start start 2023-10-16T13:51:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c781f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94c01618b0 0x7f94c0065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c781f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f94c0065ff0 0x7f94c00ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c781f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c006cee0 con 0x7f94c015d330 2023-10-16T13:51:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c781f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c006b1e0 con 0x7f94c01618b0 2023-10-16T13:51:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c781f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c00663d0 con 0x7f94c0065ff0 2023-10-16T13:51:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c4dba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94c01618b0 0x7f94c0065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c4dba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94c01618b0 0x7f94c0065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40600/0 (socket says 172.21.15.60:40600) 2023-10-16T13:51:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.675+0000 7f94c4dba640 1 -- 172.21.15.60:0/629648097 learned_addr learned my addr 172.21.15.60:0/629648097 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.676+0000 7f94c4dba640 1 -- 172.21.15.60:0/629648097 >> v2:172.21.15.144:3300/0 conn(0x7f94c0065ff0 msgr2=0x7f94c00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.676+0000 7f94c4dba640 1 --2- 172.21.15.60:0/629648097 >> v2:172.21.15.144:3300/0 conn(0x7f94c0065ff0 0x7f94c00ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.676+0000 7f94c4dba640 1 -- 172.21.15.60:0/629648097 >> v1:172.21.15.144:6789/0 conn(0x7f94c015d330 legacy=0x7f94c006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.676+0000 7f94c4dba640 1 -- 172.21.15.60:0/629648097 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c00d0890 con 0x7f94c01618b0 2023-10-16T13:51:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.677+0000 7f94c4dba640 1 --2- 172.21.15.60:0/629648097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c0065ab0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f94c006f340 tx=0x7f94b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0b4d6b8fe52f99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.677+0000 7f94b7fff640 1 -- 172.21.15.60:0/629648097 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b00317e0 con 0x7f94c01618b0 2023-10-16T13:51:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.677+0000 7f94b7fff640 1 -- 172.21.15.60:0/629648097 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0031940 con 0x7f94c01618b0 2023-10-16T13:51:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.677+0000 7f94b7fff640 1 -- 172.21.15.60:0/629648097 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0031cd0 con 0x7f94c01618b0 2023-10-16T13:51:18.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 -- 172.21.15.60:0/629648097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 msgr2=0x7f94c0065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 --2- 172.21.15.60:0/629648097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c0065ab0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f94c006f340 tx=0x7f94b002b880 comp rx=0 tx=0).stop 2023-10-16T13:51:18.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 -- 172.21.15.60:0/629648097 shutdown_connections 2023-10-16T13:51:18.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 --2- 172.21.15.60:0/629648097 >> v2:172.21.15.144:3300/0 conn(0x7f94c0065ff0 0x7f94c00ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 --2- 172.21.15.60:0/629648097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c0065ab0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 -- 172.21.15.60:0/629648097 >> 172.21.15.60:0/629648097 conn(0x7f94c00d0ab0 msgr2=0x7f94c00d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 -- 172.21.15.60:0/629648097 shutdown_connections 2023-10-16T13:51:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.678+0000 7f94c781f640 1 -- 172.21.15.60:0/629648097 wait complete. 2023-10-16T13:51:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.680+0000 7f94c781f640 1 Processor -- start 2023-10-16T13:51:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.680+0000 7f94c781f640 1 -- start start 2023-10-16T13:51:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c781f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c0065ff0 0x7f94c01f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c781f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015d330 0x7f94c0158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c781f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c015f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c781f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0160640 con 0x7f94c01618b0 2023-10-16T13:51:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c781f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c015fda0 con 0x7f94c0065ff0 2023-10-16T13:51:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c781f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0160080 con 0x7f94c015d330 2023-10-16T13:51:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c5dbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c5dbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c015f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40602/0 (socket says 172.21.15.60:40602) 2023-10-16T13:51:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.681+0000 7f94c5dbc640 1 -- 172.21.15.60:0/1341533092 learned_addr learned my addr 172.21.15.60:0/1341533092 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c55bb640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c0065ff0 0x7f94c01f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c4dba640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015d330 0x7f94c0158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c5dbc640 1 -- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015d330 msgr2=0x7f94c0158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c5dbc640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015d330 0x7f94c0158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c5dbc640 1 -- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c0065ff0 msgr2=0x7f94c01f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c5dbc640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c0065ff0 0x7f94c01f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c5dbc640 1 -- 172.21.15.60:0/1341533092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00094d0 con 0x7f94c01618b0 2023-10-16T13:51:18.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.682+0000 7f94c5dbc640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c015f550 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f94b800b410 tx=0x7f94b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:18.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.683+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8011830 con 0x7f94c01618b0 2023-10-16T13:51:18.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.683+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b80119b0 con 0x7f94c01618b0 2023-10-16T13:51:18.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.683+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8011d60 con 0x7f94c01618b0 2023-10-16T13:51:18.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.683+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c0160360 con 0x7f94c01618b0 2023-10-16T13:51:18.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.683+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c015fa90 con 0x7f94c01618b0 2023-10-16T13:51:18.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.684+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c015da80 con 0x7f94c01618b0 2023-10-16T13:51:18.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.685+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f94b801a400 con 0x7f94c01618b0 2023-10-16T13:51:18.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.686+0000 7f94b5ffb640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f94a406af90 0x7f94a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:18.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.686+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f94b8089860 con 0x7f94c01618b0 2023-10-16T13:51:18.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.687+0000 7f94c55bb640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f94a406af90 0x7f94a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:18.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.688+0000 7f94c55bb640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f94a406af90 0x7f94a406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f94c00afa40 tx=0x7f94bc0094f0 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:18.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.690+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94c015da80 con 0x7f94c01618b0 2023-10-16T13:51:18.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.885+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94c015da80 con 0x7f94c01618b0 2023-10-16T13:51:18.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.885+0000 7f94b5ffb640 1 -- 172.21.15.60:0/1341533092 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94c015da80 con 0x7f94c01618b0 2023-10-16T13:51:18.888 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:51:18.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f94a406af90 msgr2=0x7f94a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f94a406af90 0x7f94a406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f94c00afa40 tx=0x7f94bc0094f0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 msgr2=0x7f94c015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:18.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c015f550 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f94b800b410 tx=0x7f94b800b8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 shutdown_connections 2023-10-16T13:51:18.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f94a406af90 0x7f94a406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015d330 0x7f94c0158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c01618b0 0x7f94c015f550 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 --2- 172.21.15.60:0/1341533092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c0065ff0 0x7f94c01f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:18.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 >> 172.21.15.60:0/1341533092 conn(0x7f94c00d0ab0 msgr2=0x7f94c00ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:18.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 shutdown_connections 2023-10-16T13:51:18.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:18.888+0000 7f94c781f640 1 -- 172.21.15.60:0/1341533092 wait complete. 2023-10-16T13:51:18.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T13:51:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.129+0000 7f4e5554e640 1 Processor -- start 2023-10-16T13:51:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.129+0000 7f4e5554e640 1 -- start start 2023-10-16T13:51:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.130+0000 7f4e5554e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e50159b20 0x7f4e5006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.130+0000 7f4e5554e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4e50159f00 0x7f4e500b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e5554e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5006f7f0 con 0x7f4e5006c120 2023-10-16T13:51:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e5554e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5006c5e0 con 0x7f4e50159b20 2023-10-16T13:51:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e5554e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5006c8c0 con 0x7f4e50159f00 2023-10-16T13:51:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e4f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4e5006c120 0x7f4e500cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47946/0 (socket says 172.21.15.60:47946) 2023-10-16T13:51:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e4f7fe640 1 -- 172.21.15.60:0/3502556353 learned_addr learned my addr 172.21.15.60:0/3502556353 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e4effd640 1 --2- 172.21.15.60:0/3502556353 >> v2:172.21.15.60:3300/0 conn(0x7f4e50159b20 0x7f4e5006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.131+0000 7f4e47fff640 1 --2- 172.21.15.60:0/3502556353 >> v2:172.21.15.144:3300/0 conn(0x7f4e50159f00 0x7f4e500b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.132+0000 7f4e4e7fc640 1 -- 172.21.15.60:0/3502556353 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209342806 0 0) 0x7f4e5006f7f0 con 0x7f4e5006c120 2023-10-16T13:51:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.132+0000 7f4e4e7fc640 1 -- 172.21.15.60:0/3502556353 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3c003520 con 0x7f4e5006c120 2023-10-16T13:51:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.132+0000 7f4e4effd640 1 -- 172.21.15.60:0/3502556353 >> v2:172.21.15.144:3300/0 conn(0x7f4e50159f00 msgr2=0x7f4e500b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.132+0000 7f4e4effd640 1 --2- 172.21.15.60:0/3502556353 >> v2:172.21.15.144:3300/0 conn(0x7f4e50159f00 0x7f4e500b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.132+0000 7f4e4effd640 1 -- 172.21.15.60:0/3502556353 >> v1:172.21.15.60:6789/0 conn(0x7f4e5006c120 legacy=0x7f4e500cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.132+0000 7f4e4effd640 1 -- 172.21.15.60:0/3502556353 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5006cba0 con 0x7f4e50159b20 2023-10-16T13:51:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.133+0000 7f4e4effd640 1 --2- 172.21.15.60:0/3502556353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159b20 0x7f4e5006bbe0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f4e5015d770 tx=0x7f4e4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72dcd6183568bb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.133+0000 7f4e4e7fc640 1 -- 172.21.15.60:0/3502556353 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40031830 con 0x7f4e50159b20 2023-10-16T13:51:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.133+0000 7f4e4e7fc640 1 -- 172.21.15.60:0/3502556353 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e40031990 con 0x7f4e50159b20 2023-10-16T13:51:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.133+0000 7f4e4e7fc640 1 -- 172.21.15.60:0/3502556353 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40031d00 con 0x7f4e50159b20 2023-10-16T13:51:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 -- 172.21.15.60:0/3502556353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159b20 msgr2=0x7f4e5006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 --2- 172.21.15.60:0/3502556353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159b20 0x7f4e5006bbe0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f4e5015d770 tx=0x7f4e4002b8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 -- 172.21.15.60:0/3502556353 shutdown_connections 2023-10-16T13:51:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 --2- 172.21.15.60:0/3502556353 >> v2:172.21.15.144:3300/0 conn(0x7f4e50159f00 0x7f4e500b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 --2- 172.21.15.60:0/3502556353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159b20 0x7f4e5006bbe0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 -- 172.21.15.60:0/3502556353 >> 172.21.15.60:0/3502556353 conn(0x7f4e500cb410 msgr2=0x7f4e500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 -- 172.21.15.60:0/3502556353 shutdown_connections 2023-10-16T13:51:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.134+0000 7f4e5554e640 1 -- 172.21.15.60:0/3502556353 wait complete. 2023-10-16T13:51:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.136+0000 7f4e5554e640 1 Processor -- start 2023-10-16T13:51:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.136+0000 7f4e5554e640 1 -- start start 2023-10-16T13:51:19.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.137+0000 7f4e5554e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e5006c120 0x7f4e500c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:19.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.137+0000 7f4e5554e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e50159b20 0x7f4e50150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:19.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.137+0000 7f4e5554e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 0x7f4e50151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:19.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.137+0000 7f4e5554e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5015bbc0 con 0x7f4e50159f00 2023-10-16T13:51:19.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.137+0000 7f4e5554e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500c3220 con 0x7f4e50159b20 2023-10-16T13:51:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.137+0000 7f4e5554e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500c5a60 con 0x7f4e5006c120 2023-10-16T13:51:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 0x7f4e50151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 0x7f4e50151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46254/0 (socket says 172.21.15.60:46254) 2023-10-16T13:51:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 -- 172.21.15.60:0/1127730380 learned_addr learned my addr 172.21.15.60:0/1127730380 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4effd640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e5006c120 0x7f4e500c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 -- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e5006c120 msgr2=0x7f4e500c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e5006c120 0x7f4e500c49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 -- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e50159b20 msgr2=0x7f4e50150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e47fff640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e50159b20 0x7f4e50150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e50159b20 0x7f4e50150e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.138+0000 7f4e4f7fe640 1 -- 172.21.15.60:0/1127730380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e40009400 con 0x7f4e50159f00 2023-10-16T13:51:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.139+0000 7f4e4f7fe640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 0x7f4e50151340 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800cb00 tx=0x7f4e4800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.139+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4800aca0 con 0x7f4e50159f00 2023-10-16T13:51:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.139+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4800ae00 con 0x7f4e50159f00 2023-10-16T13:51:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.139+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48012620 con 0x7f4e50159f00 2023-10-16T13:51:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.139+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5015afb0 con 0x7f4e50159f00 2023-10-16T13:51:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.139+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5015b570 con 0x7f4e50159f00 2023-10-16T13:51:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.140+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5015d1b0 con 0x7f4e50159f00 2023-10-16T13:51:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.141+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e48007070 con 0x7f4e50159f00 2023-10-16T13:51:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.142+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4e3006af90 0x7f4e3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.142+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f4e48089630 con 0x7f4e50159f00 2023-10-16T13:51:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.143+0000 7f4e4effd640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4e3006af90 0x7f4e3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.144+0000 7f4e4effd640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4e3006af90 0x7f4e3006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e40036000 tx=0x7f4e40008030 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.147+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e480546b0 con 0x7f4e50159f00 2023-10-16T13:51:19.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.422+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f4e5015bd70 con 0x7f4e50159f00 2023-10-16T13:51:19.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.955+0000 7f4e477fe640 1 -- 172.21.15.60:0/1127730380 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2558) v1 ==== 168+0+0 (secure 0 0 0) 0x7f4e5015bd70 con 0x7f4e50159f00 2023-10-16T13:51:19.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.959+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4e3006af90 msgr2=0x7f4e3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:19.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.959+0000 7f4e5554e640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4e3006af90 0x7f4e3006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4e40036000 tx=0x7f4e40008030 comp rx=0 tx=0).stop 2023-10-16T13:51:19.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.959+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 msgr2=0x7f4e50151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:19.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.959+0000 7f4e5554e640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 0x7f4e50151340 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800cb00 tx=0x7f4e4800a060 comp rx=0 tx=0).stop 2023-10-16T13:51:19.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.959+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 shutdown_connections 2023-10-16T13:51:19.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4e3006af90 0x7f4e3006d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e50159f00 0x7f4e50151340 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e50159b20 0x7f4e50150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 --2- 172.21.15.60:0/1127730380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e5006c120 0x7f4e500c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:19.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 >> 172.21.15.60:0/1127730380 conn(0x7f4e500cb410 msgr2=0x7f4e500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:19.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 shutdown_connections 2023-10-16T13:51:19.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:19.960+0000 7f4e5554e640 1 -- 172.21.15.60:0/1127730380 wait complete. 2023-10-16T13:51:19.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:51:20.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.216+0000 7f2cb1b4e640 1 Processor -- start 2023-10-16T13:51:20.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.216+0000 7f2cb1b4e640 1 -- start start 2023-10-16T13:51:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.217+0000 7f2cb1b4e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac15cd70 0x7f2cac0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.217+0000 7f2cb1b4e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2cac0b8480 0x7f2cac0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2cb1b4e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cac15d150 con 0x7f2cac0d0be0 2023-10-16T13:51:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2cb1b4e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cac0d1180 con 0x7f2cac0b8480 2023-10-16T13:51:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2cb1b4e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac0d1460 con 0x7f2cac15cd70 2023-10-16T13:51:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2ca3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2cac0b8480 0x7f2cac0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2ca3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2cac0b8480 0x7f2cac0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46270/0 (socket says 172.21.15.60:46270) 2023-10-16T13:51:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2ca3fff640 1 -- 172.21.15.60:0/1472465524 learned_addr learned my addr 172.21.15.60:0/1472465524 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:20.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.218+0000 7f2cb0b4c640 1 --2- 172.21.15.60:0/1472465524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac15cd70 0x7f2cac0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.219+0000 7f2ca3fff640 1 -- 172.21.15.60:0/1472465524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac15cd70 msgr2=0x7f2cac0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.219+0000 7f2ca3fff640 1 --2- 172.21.15.60:0/1472465524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac15cd70 0x7f2cac0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.219+0000 7f2ca3fff640 1 -- 172.21.15.60:0/1472465524 >> v1:172.21.15.144:6789/0 conn(0x7f2cac0d0be0 legacy=0x7f2cac0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.219+0000 7f2ca3fff640 1 -- 172.21.15.60:0/1472465524 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cac0d1740 con 0x7f2cac0b8480 2023-10-16T13:51:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.220+0000 7f2ca3fff640 1 --2- 172.21.15.60:0/1472465524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac0b8480 0x7f2cac0cd7c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009870 tx=0x7f2c9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1041830239595b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:20.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.220+0000 7f2ca37fe640 1 -- 172.21.15.60:0/1472465524 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c031640 con 0x7f2cac0b8480 2023-10-16T13:51:20.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.220+0000 7f2ca37fe640 1 -- 172.21.15.60:0/1472465524 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9c0317a0 con 0x7f2cac0b8480 2023-10-16T13:51:20.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.220+0000 7f2ca37fe640 1 -- 172.21.15.60:0/1472465524 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c031b30 con 0x7f2cac0b8480 2023-10-16T13:51:20.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/1472465524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac0b8480 msgr2=0x7f2cac0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:20.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/1472465524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac0b8480 0x7f2cac0cd7c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009870 tx=0x7f2c9c02b8f0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/1472465524 shutdown_connections 2023-10-16T13:51:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/1472465524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac0b8480 0x7f2cac0cd7c0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/1472465524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac15cd70 0x7f2cac0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/1472465524 >> 172.21.15.60:0/1472465524 conn(0x7f2cac0cf570 msgr2=0x7f2cac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/1472465524 shutdown_connections 2023-10-16T13:51:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.221+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/1472465524 wait complete. 2023-10-16T13:51:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.223+0000 7f2cb1b4e640 1 Processor -- start 2023-10-16T13:51:20.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.223+0000 7f2cb1b4e640 1 -- start start 2023-10-16T13:51:20.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb1b4e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac0b8480 0x7f2cac1f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:20.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb1b4e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cac0d0be0 0x7f2cac1f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:20.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb1b4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 0x7f2cac0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:20.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb1b4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac15fe70 con 0x7f2cac15cd70 2023-10-16T13:51:20.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb1b4e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac15dd30 con 0x7f2cac0d0be0 2023-10-16T13:51:20.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb1b4e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac0cca90 con 0x7f2cac0b8480 2023-10-16T13:51:20.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb134d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 0x7f2cac0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:20.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb134d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 0x7f2cac0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46286/0 (socket says 172.21.15.60:46286) 2023-10-16T13:51:20.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.224+0000 7f2cb134d640 1 -- 172.21.15.60:0/2034527795 learned_addr learned my addr 172.21.15.60:0/2034527795 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:20.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2ca3fff640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cac0d0be0 0x7f2cac1f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:20.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb0b4c640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac0b8480 0x7f2cac1f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb134d640 1 -- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac0b8480 msgr2=0x7f2cac1f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb134d640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac0b8480 0x7f2cac1f1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb134d640 1 -- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cac0d0be0 msgr2=0x7f2cac1f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb134d640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cac0d0be0 0x7f2cac1f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb134d640 1 -- 172.21.15.60:0/2034527795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0094d0 con 0x7f2cac15cd70 2023-10-16T13:51:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2ca3fff640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cac0d0be0 0x7f2cac1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.225+0000 7f2cb134d640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 0x7f2cac0c74a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c7b0 tx=0x7f2ca400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.226+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4004b10 con 0x7f2cac15cd70 2023-10-16T13:51:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.226+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca4004c70 con 0x7f2cac15cd70 2023-10-16T13:51:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.226+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400ce60 con 0x7f2cac15cd70 2023-10-16T13:51:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.226+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cac15d400 con 0x7f2cac15cd70 2023-10-16T13:51:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.226+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cac15d9c0 con 0x7f2cac15cd70 2023-10-16T13:51:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.227+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cac04e9b0 con 0x7f2cac15cd70 2023-10-16T13:51:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.228+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ca400a850 con 0x7f2cac15cd70 2023-10-16T13:51:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.229+0000 7f2ca17fa640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2c9806af90 0x7f2c9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.229+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f2ca4089820 con 0x7f2cac15cd70 2023-10-16T13:51:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.230+0000 7f2cb0b4c640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2c9806af90 0x7f2c9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:20.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.231+0000 7f2cb0b4c640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2c9806af90 0x7f2c9806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006fd0 tx=0x7f2ca8008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:20.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.233+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ca4054800 con 0x7f2cac15cd70 2023-10-16T13:51:20.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.434+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2cac06c050 con 0x7f2cac15cd70 2023-10-16T13:51:20.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.441+0000 7f2ca17fa640 1 -- 172.21.15.60:0/2034527795 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ca4057e20 con 0x7f2cac15cd70 2023-10-16T13:51:20.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2c9806af90 msgr2=0x7f2c9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:20.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2c9806af90 0x7f2c9806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006fd0 tx=0x7f2ca8008040 comp rx=0 tx=0).stop 2023-10-16T13:51:20.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 msgr2=0x7f2cac0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:20.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 0x7f2cac0c74a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c7b0 tx=0x7f2ca400cc70 comp rx=0 tx=0).stop 2023-10-16T13:51:20.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 shutdown_connections 2023-10-16T13:51:20.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f2c9806af90 0x7f2c9806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cac15cd70 0x7f2cac0c74a0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cac0d0be0 0x7f2cac1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 --2- 172.21.15.60:0/2034527795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cac0b8480 0x7f2cac1f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:20.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 >> 172.21.15.60:0/2034527795 conn(0x7f2cac0cf570 msgr2=0x7f2cac0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:20.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 shutdown_connections 2023-10-16T13:51:20.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:20.444+0000 7f2cb1b4e640 1 -- 172.21.15.60:0/2034527795 wait complete. 2023-10-16T13:51:20.467 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:51:20.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:51:20.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:51:20.522 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:51:20.585 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:51:20.684 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:51:20.685 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:51:20.685 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:20.750 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T13:51:20.751 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:51:20.751 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:20.809 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:51:20.809 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:51:20.809 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:20.873 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T13:51:20.874 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T13:51:20.874 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:20.936 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T13:51:20.937 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:51:20.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:21.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.167+0000 7ff22e46e640 1 Processor -- start 2023-10-16T13:51:21.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.167+0000 7ff22e46e640 1 -- start start 2023-10-16T13:51:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22e46e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9060 0x7ff22806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22e46e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2280c9440 0x7ff22806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22e46e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280ca560 con 0x7ff22806c9e0 2023-10-16T13:51:21.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22e46e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2280c9950 con 0x7ff2280c9440 2023-10-16T13:51:21.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22e46e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280c9c30 con 0x7ff2280c9060 2023-10-16T13:51:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22cc6b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff22806c9e0 0x7ff228164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47954/0 (socket says 172.21.15.60:47954) 2023-10-16T13:51:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22cc6b640 1 -- 172.21.15.60:0/3676899697 learned_addr learned my addr 172.21.15.60:0/3676899697 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.169+0000 7ff22dc6d640 1 --2- 172.21.15.60:0/3676899697 >> v2:172.21.15.60:3300/0 conn(0x7ff2280c9440 0x7ff22806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.170+0000 7ff22d46c640 1 --2- 172.21.15.60:0/3676899697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9060 0x7ff22806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.170+0000 7ff21ffff640 1 -- 172.21.15.60:0/3676899697 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825386904 0 0) 0x7ff2280ca560 con 0x7ff22806c9e0 2023-10-16T13:51:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.170+0000 7ff22dc6d640 1 -- 172.21.15.60:0/3676899697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9060 msgr2=0x7ff22806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.170+0000 7ff22dc6d640 1 --2- 172.21.15.60:0/3676899697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9060 0x7ff22806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.170+0000 7ff22dc6d640 1 -- 172.21.15.60:0/3676899697 >> v1:172.21.15.60:6789/0 conn(0x7ff22806c9e0 legacy=0x7ff228164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.170+0000 7ff22dc6d640 1 -- 172.21.15.60:0/3676899697 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280c9f10 con 0x7ff2280c9440 2023-10-16T13:51:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.172+0000 7ff22dc6d640 1 --2- 172.21.15.60:0/3676899697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2280c9440 0x7ff22806f980 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff2200097a0 tx=0x7ff22002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0b92bc3b705777 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.172+0000 7ff21ffff640 1 -- 172.21.15.60:0/3676899697 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22002d040 con 0x7ff2280c9440 2023-10-16T13:51:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.172+0000 7ff21ffff640 1 -- 172.21.15.60:0/3676899697 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff220031810 con 0x7ff2280c9440 2023-10-16T13:51:21.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.172+0000 7ff21ffff640 1 -- 172.21.15.60:0/3676899697 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220031ba0 con 0x7ff2280c9440 2023-10-16T13:51:21.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.172+0000 7ff22e46e640 1 -- 172.21.15.60:0/3676899697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2280c9440 msgr2=0x7ff22806f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.172+0000 7ff22e46e640 1 --2- 172.21.15.60:0/3676899697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2280c9440 0x7ff22806f980 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff2200097a0 tx=0x7ff22002b8b0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.173+0000 7ff22e46e640 1 -- 172.21.15.60:0/3676899697 shutdown_connections 2023-10-16T13:51:21.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.173+0000 7ff22e46e640 1 --2- 172.21.15.60:0/3676899697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2280c9440 0x7ff22806f980 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.173+0000 7ff22e46e640 1 --2- 172.21.15.60:0/3676899697 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9060 0x7ff22806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.173+0000 7ff22e46e640 1 -- 172.21.15.60:0/3676899697 >> 172.21.15.60:0/3676899697 conn(0x7ff2280ce6a0 msgr2=0x7ff2280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.173+0000 7ff22e46e640 1 -- 172.21.15.60:0/3676899697 shutdown_connections 2023-10-16T13:51:21.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.173+0000 7ff22e46e640 1 -- 172.21.15.60:0/3676899697 wait complete. 2023-10-16T13:51:21.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.175+0000 7ff22e46e640 1 Processor -- start 2023-10-16T13:51:21.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.175+0000 7ff22e46e640 1 -- start start 2023-10-16T13:51:21.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22e46e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 0x7ff2281f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22e46e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2280c9060 0x7ff228164120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22e46e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9440 0x7ff2281f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22e46e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2281f77f0 con 0x7ff22806c9e0 2023-10-16T13:51:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22e46e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228164660 con 0x7ff2280c9060 2023-10-16T13:51:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22e46e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2281648b0 con 0x7ff2280c9440 2023-10-16T13:51:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 0x7ff2281f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 0x7ff2281f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46310/0 (socket says 172.21.15.60:46310) 2023-10-16T13:51:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 -- 172.21.15.60:0/4168840850 learned_addr learned my addr 172.21.15.60:0/4168840850 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 -- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9440 msgr2=0x7ff2281f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9440 0x7ff2281f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 -- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2280c9060 msgr2=0x7ff228164120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2280c9060 0x7ff228164120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.176+0000 7ff22d46c640 1 -- 172.21.15.60:0/4168840850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff220009400 con 0x7ff22806c9e0 2023-10-16T13:51:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.177+0000 7ff22d46c640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 0x7ff2281f72b0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7ff224007f20 tx=0x7ff22400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.177+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224014620 con 0x7ff22806c9e0 2023-10-16T13:51:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.177+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff224014780 con 0x7ff22806c9e0 2023-10-16T13:51:21.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.177+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224014b30 con 0x7ff22806c9e0 2023-10-16T13:51:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.177+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2280c21a0 con 0x7ff22806c9e0 2023-10-16T13:51:21.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.177+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2280c2760 con 0x7ff22806c9e0 2023-10-16T13:51:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.179+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22804e9b0 con 0x7ff22806c9e0 2023-10-16T13:51:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.180+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff224014c90 con 0x7ff22806c9e0 2023-10-16T13:51:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.181+0000 7ff21dffb640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff20c06af90 0x7ff20c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.181+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ff22408a170 con 0x7ff22806c9e0 2023-10-16T13:51:21.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.181+0000 7ff22cc6b640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff20c06af90 0x7ff20c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.183+0000 7ff22cc6b640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff20c06af90 0x7ff20c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff218007fd0 tx=0x7ff218007920 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.185+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff224055150 con 0x7ff22806c9e0 2023-10-16T13:51:21.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.425+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2280b4620 con 0x7ff22806c9e0 2023-10-16T13:51:21.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.426+0000 7ff21dffb640 1 -- 172.21.15.60:0/4168840850 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3705 (secure 0 0 0) 0x7ff224058770 con 0x7ff22806c9e0 2023-10-16T13:51:21.429 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:21.429 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7735,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/3194771921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3194771921},{"type":"v1","addr":"172.21.15.144:6836","nonce":3194771921}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":7738,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1972258445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1972258445},{"type":"v1","addr":"172.21.15.144:6837","nonce":1972258445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3922764927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":3922764927},{"type":"v1","addr":"172.21.15.60:6835","nonce":3922764927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":9882,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1831644650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1831644650},{"type":"v1","addr":"172.21.15.60:6834","nonce":1831644650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[]} 2023-10-16T13:51:21.430 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 89 2023-10-16T13:51:21.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff20c06af90 msgr2=0x7ff20c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff20c06af90 0x7ff20c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff218007fd0 tx=0x7ff218007920 comp rx=0 tx=0).stop 2023-10-16T13:51:21.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 msgr2=0x7ff2281f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 0x7ff2281f72b0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7ff224007f20 tx=0x7ff22400e570 comp rx=0 tx=0).stop 2023-10-16T13:51:21.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 shutdown_connections 2023-10-16T13:51:21.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff20c06af90 0x7ff20c06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff2280c9440 0x7ff2281f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2280c9060 0x7ff228164120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 --2- 172.21.15.60:0/4168840850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22806c9e0 0x7ff2281f72b0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 >> 172.21.15.60:0/4168840850 conn(0x7ff2280ce6a0 msgr2=0x7ff2280c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:21.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 shutdown_connections 2023-10-16T13:51:21.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.429+0000 7ff22e46e640 1 -- 172.21.15.60:0/4168840850 wait complete. 2023-10-16T13:51:21.453 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:51:21.453 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:51:21.454 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:51:21.454 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:51:21.454 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:51:21.454 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:51:21.455 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:51:21.455 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:51:21.455 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:51:21.455 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:51:21.456 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:51:21.456 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:51:21.456 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:51:21.457 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:51:21.457 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:51:21.457 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:51:21.470 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:51:21.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:51:21.473 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:51:21.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:51:21.482 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:51:21.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:51:21.486 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:51:21.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:51:21.522 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:51:21.523 DEBUG:teuthology.parallel:result is None 2023-10-16T13:51:21.530 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:51:21.531 DEBUG:teuthology.parallel:result is None 2023-10-16T13:51:21.534 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:51:21.535 DEBUG:teuthology.parallel:result is None 2023-10-16T13:51:21.542 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:51:21.542 DEBUG:teuthology.parallel:result is None 2023-10-16T13:51:21.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:51:21.689 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T13:51:21.692 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T13:51:21.696 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T13:51:21.707 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T13:51:21.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.742+0000 7fe4f1451640 1 Processor -- start 2023-10-16T13:51:21.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.742+0000 7fe4f1451640 1 -- start start 2023-10-16T13:51:21.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.743+0000 7fe4f1451640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe4ec070830 0x7fe4ec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.743+0000 7fe4f1451640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec161740 con 0x7fe4ec0cf5f0 2023-10-16T13:51:21.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4f1451640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec06b0f0 con 0x7fe4ec070c10 2023-10-16T13:51:21.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4f1451640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec1619b0 con 0x7fe4ec070830 2023-10-16T13:51:21.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4eaffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe4ec070830 0x7fe4ec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4eaffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe4ec070830 0x7fe4ec06d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46340/0 (socket says 172.21.15.60:46340) 2023-10-16T13:51:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4eaffd640 1 -- 172.21.15.60:0/3521062564 learned_addr learned my addr 172.21.15.60:0/3521062564 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4eaffd640 1 -- 172.21.15.60:0/3521062564 >> v1:172.21.15.144:6789/0 conn(0x7fe4ec070c10 legacy=0x7fe4ec0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4eaffd640 1 -- 172.21.15.60:0/3521062564 >> v1:172.21.15.60:6789/0 conn(0x7fe4ec0cf5f0 legacy=0x7fe4ec06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.744+0000 7fe4eaffd640 1 -- 172.21.15.60:0/3521062564 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec06b3d0 con 0x7fe4ec070830 2023-10-16T13:51:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.745+0000 7fe4eaffd640 1 --2- 172.21.15.60:0/3521062564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec06d3c0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009870 tx=0x7fe4dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc37e141a07b10e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:21.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4ea7fc640 1 -- 172.21.15.60:0/3521062564 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc031870 con 0x7fe4ec070830 2023-10-16T13:51:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4ea7fc640 1 -- 172.21.15.60:0/3521062564 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc0319d0 con 0x7fe4ec070830 2023-10-16T13:51:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4ea7fc640 1 -- 172.21.15.60:0/3521062564 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc031d60 con 0x7fe4ec070830 2023-10-16T13:51:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4f1451640 1 -- 172.21.15.60:0/3521062564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 msgr2=0x7fe4ec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4f1451640 1 --2- 172.21.15.60:0/3521062564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec06d3c0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009870 tx=0x7fe4dc02b910 comp rx=0 tx=0).stop 2023-10-16T13:51:21.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4f1451640 1 -- 172.21.15.60:0/3521062564 shutdown_connections 2023-10-16T13:51:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4f1451640 1 --2- 172.21.15.60:0/3521062564 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec06d3c0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.746+0000 7fe4f1451640 1 -- 172.21.15.60:0/3521062564 >> 172.21.15.60:0/3521062564 conn(0x7fe4ec0cfac0 msgr2=0x7fe4ec0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.747+0000 7fe4f1451640 1 -- 172.21.15.60:0/3521062564 shutdown_connections 2023-10-16T13:51:21.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.747+0000 7fe4f1451640 1 -- 172.21.15.60:0/3521062564 wait complete. 2023-10-16T13:51:21.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.748+0000 7fe4f1451640 1 Processor -- start 2023-10-16T13:51:21.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.748+0000 7fe4f1451640 1 -- start start 2023-10-16T13:51:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.749+0000 7fe4f1451640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.749+0000 7fe4f1451640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4ec070c10 0x7fe4ec158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.749+0000 7fe4f1451640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4ec0cf5f0 0x7fe4ec0ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.749+0000 7fe4f1451640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec159f90 con 0x7fe4ec070830 2023-10-16T13:51:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.749+0000 7fe4f1451640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec1f25f0 con 0x7fe4ec0cf5f0 2023-10-16T13:51:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.749+0000 7fe4f1451640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec1587d0 con 0x7fe4ec070c10 2023-10-16T13:51:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46344/0 (socket says 172.21.15.60:46344) 2023-10-16T13:51:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 -- 172.21.15.60:0/1730893245 learned_addr learned my addr 172.21.15.60:0/1730893245 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4e3fff640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4ec070c10 0x7fe4ec158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eb7fe640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4ec0cf5f0 0x7fe4ec0ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 -- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4ec070c10 msgr2=0x7fe4ec158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4ec070c10 0x7fe4ec158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 -- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4ec0cf5f0 msgr2=0x7fe4ec0ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:21.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4ec0cf5f0 0x7fe4ec0ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:21.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eaffd640 1 -- 172.21.15.60:0/1730893245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc0094d0 con 0x7fe4ec070830 2023-10-16T13:51:21.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.750+0000 7fe4eb7fe640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4ec0cf5f0 0x7fe4ec0ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:21.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.751+0000 7fe4eaffd640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec158290 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009870 tx=0x7fe4dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:21.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.751+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc031820 con 0x7fe4ec070830 2023-10-16T13:51:21.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.751+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc0443f0 con 0x7fe4ec070830 2023-10-16T13:51:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.751+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0447a0 con 0x7fe4ec070830 2023-10-16T13:51:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.751+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec159410 con 0x7fe4ec070830 2023-10-16T13:51:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.751+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec1599d0 con 0x7fe4ec070830 2023-10-16T13:51:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.752+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec1f2760 con 0x7fe4ec070830 2023-10-16T13:51:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.753+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe4dc04d450 con 0x7fe4ec070830 2023-10-16T13:51:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.755+0000 7fe4e37fe640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fe4cc06af90 0x7fe4cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.755+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe4dc02d040 con 0x7fe4ec070830 2023-10-16T13:51:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.755+0000 7fe4e3fff640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fe4cc06af90 0x7fe4cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.756+0000 7fe4e3fff640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fe4cc06af90 0x7fe4cc06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0c69e0 tx=0x7fe4d4009040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.760+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4ec1f2760 con 0x7fe4ec070830 2023-10-16T13:51:21.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:21.972+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe4ec1f2760 con 0x7fe4ec070830 2023-10-16T13:51:23.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.357+0000 7fe4e37fe640 1 -- 172.21.15.60:0/1730893245 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v193) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe4ec1f2760 con 0x7fe4ec070830 2023-10-16T13:51:23.361 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T13:51:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fe4cc06af90 msgr2=0x7fe4cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fe4cc06af90 0x7fe4cc06d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0c69e0 tx=0x7fe4d4009040 comp rx=0 tx=0).stop 2023-10-16T13:51:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 msgr2=0x7fe4ec158290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec158290 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009870 tx=0x7fe4dc0046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 shutdown_connections 2023-10-16T13:51:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fe4cc06af90 0x7fe4cc06d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4ec070c10 0x7fe4ec158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4ec0cf5f0 0x7fe4ec0ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 --2- 172.21.15.60:0/1730893245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4ec070830 0x7fe4ec158290 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 >> 172.21.15.60:0/1730893245 conn(0x7fe4ec0cfac0 msgr2=0x7fe4ec160100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.363+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 shutdown_connections 2023-10-16T13:51:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.364+0000 7fe4f1451640 1 -- 172.21.15.60:0/1730893245 wait complete. 2023-10-16T13:51:23.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:51:23.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.613+0000 7f340b459640 1 Processor -- start 2023-10-16T13:51:23.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.613+0000 7f340b459640 1 -- start start 2023-10-16T13:51:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.615+0000 7f340b459640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f340406fb50 0x7f34040cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.615+0000 7f340b459640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040ca560 con 0x7f34040c9060 2023-10-16T13:51:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.615+0000 7f340b459640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040c9950 con 0x7f340406c9e0 2023-10-16T13:51:23.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.615+0000 7f340b459640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34040c9c30 con 0x7f340406fb50 2023-10-16T13:51:23.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.615+0000 7f340a457640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f34040c9060 0x7f340406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47988/0 (socket says 172.21.15.60:47988) 2023-10-16T13:51:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.615+0000 7f340a457640 1 -- 172.21.15.60:0/774918941 learned_addr learned my addr 172.21.15.60:0/774918941 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409c56640 1 --2- 172.21.15.60:0/774918941 >> v2:172.21.15.144:3300/0 conn(0x7f340406fb50 0x7f34040cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203679392 0 0) 0x7f34040ca560 con 0x7f34040c9060 2023-10-16T13:51:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0003520 con 0x7f34040c9060 2023-10-16T13:51:23.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952305192 0 0) 0x7f33f0003520 con 0x7f34040c9060 2023-10-16T13:51:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 >> v2:172.21.15.144:3300/0 conn(0x7f340406fb50 msgr2=0x7f34040cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 --2- 172.21.15.60:0/774918941 >> v2:172.21.15.144:3300/0 conn(0x7f340406fb50 0x7f34040cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 >> v1:172.21.15.144:6789/0 conn(0x7f340406c9e0 legacy=0x7f34040ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34040c9f10 con 0x7f34040c9060 2023-10-16T13:51:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f3400001670 con 0x7f34040c9060 2023-10-16T13:51:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 >> v1:172.21.15.60:6789/0 conn(0x7f34040c9060 legacy=0x7f340406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003970 0x7f33f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33f00047c0 0x7f33f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0005180 0x7f33f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34040c9c30 con 0x7f33f0003970 2023-10-16T13:51:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f000c020 con 0x7f33f00047c0 2023-10-16T13:51:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.616+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f000c1e0 con 0x7f33f0005180 2023-10-16T13:51:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340a457640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003970 0x7f33f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340a457640 1 -- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0005180 msgr2=0x7f33f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340a457640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0005180 0x7f33f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340a457640 1 -- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33f00047c0 msgr2=0x7f33f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f3409c56640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33f00047c0 0x7f33f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340a457640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33f00047c0 0x7f33f0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340ac58640 1 -- 172.21.15.60:0/774918941 reap_dead start 2023-10-16T13:51:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.617+0000 7f340a457640 1 -- 172.21.15.60:0/774918941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0003520 con 0x7f33f0003970 2023-10-16T13:51:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.618+0000 7f340a457640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003970 0x7f33f0003d90 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f340000c240 tx=0x7f33f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc5d348b64f896c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400005ad0 con 0x7f33f0003970 2023-10-16T13:51:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3400005c30 con 0x7f33f0003970 2023-10-16T13:51:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f3409455640 1 -- 172.21.15.60:0/774918941 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340002c8b0 con 0x7f33f0003970 2023-10-16T13:51:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 -- 172.21.15.60:0/774918941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003970 msgr2=0x7f33f0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003970 0x7f33f0003d90 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f340000c240 tx=0x7f33f00057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 -- 172.21.15.60:0/774918941 shutdown_connections 2023-10-16T13:51:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 --2- 172.21.15.60:0/774918941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003970 0x7f33f0003d90 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 -- 172.21.15.60:0/774918941 >> 172.21.15.60:0/774918941 conn(0x7f34040ce6a0 msgr2=0x7f34040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 -- 172.21.15.60:0/774918941 shutdown_connections 2023-10-16T13:51:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.619+0000 7f340b459640 1 -- 172.21.15.60:0/774918941 wait complete. 2023-10-16T13:51:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.621+0000 7f340b459640 1 Processor -- start 2023-10-16T13:51:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.621+0000 7f340b459640 1 -- start start 2023-10-16T13:51:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.622+0000 7f340b459640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0003970 0x7f3404163af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.622+0000 7f340b459640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 0x7f34040c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.622+0000 7f340b459640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f340415b510 0x7f340415b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.622+0000 7f340b459640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340415bed0 con 0x7f34040c3560 2023-10-16T13:51:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.622+0000 7f340b459640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34041faba0 con 0x7f340415b510 2023-10-16T13:51:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.622+0000 7f340b459640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34040c4000 con 0x7f33f0003970 2023-10-16T13:51:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 0x7f34040c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 0x7f34040c39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46360/0 (socket says 172.21.15.60:46360) 2023-10-16T13:51:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 -- 172.21.15.60:0/3340855381 learned_addr learned my addr 172.21.15.60:0/3340855381 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f340a457640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0003970 0x7f3404163af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f340ac58640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f340415b510 0x7f340415b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 -- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0003970 msgr2=0x7f3404163af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0003970 0x7f3404163af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 -- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f340415b510 msgr2=0x7f340415b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f340415b510 0x7f340415b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 -- 172.21.15.60:0/3340855381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34041fb030 con 0x7f34040c3560 2023-10-16T13:51:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.623+0000 7f3409c56640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 0x7f34040c39c0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f33f4010790 tx=0x7f33f4010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.624+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4009630 con 0x7f34040c3560 2023-10-16T13:51:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.624+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f400eb00 con 0x7f34040c3560 2023-10-16T13:51:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.624+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f400ee90 con 0x7f34040c3560 2023-10-16T13:51:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.624+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34041fa270 con 0x7f34040c3560 2023-10-16T13:51:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.624+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34041fa830 con 0x7f34040c3560 2023-10-16T13:51:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.625+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340404e9b0 con 0x7f34040c3560 2023-10-16T13:51:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.626+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33f401c400 con 0x7f34040c3560 2023-10-16T13:51:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.627+0000 7f33faffd640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f33e806afc0 0x7f33e806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.627+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33f4089830 con 0x7f34040c3560 2023-10-16T13:51:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.628+0000 7f340a457640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f33e806afc0 0x7f33e806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.629+0000 7f340a457640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f33e806afc0 0x7f33e806d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3400034000 tx=0x7f3400009030 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.632+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33f4054a80 con 0x7f34040c3560 2023-10-16T13:51:23.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.906+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f34041face0 con 0x7f34040c3560 2023-10-16T13:51:23.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.907+0000 7f33faffd640 1 -- 172.21.15.60:0/3340855381 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v47) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f33f4010ea0 con 0x7f34040c3560 2023-10-16T13:51:23.911 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:23.911 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T13:51:23.911 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T13:51:23.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.912 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T13:51:23.912 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T13:51:23.912 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.912 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.912 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:51:23.913 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:51:23.913 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.913 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.913 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T13:51:23.914 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T13:51:23.914 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.914 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.914 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:51:23.914 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:51:23.915 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.915 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T13:51:23.916 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T13:51:23.916 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.916 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.916 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:51:23.916 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:51:23.916 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.917 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T13:51:23.917 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T13:51:23.917 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.918 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.918 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:51:23.918 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:51:23.918 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.919 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T13:51:23.919 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T13:51:23.919 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.919 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.920 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.920 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.921 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T13:51:23.921 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T13:51:23.921 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.921 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.921 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.922 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.922 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.922 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T13:51:23.922 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T13:51:23.923 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.923 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.923 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.923 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.924 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T13:51:23.924 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T13:51:23.924 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.925 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.925 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.925 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.925 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.926 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T13:51:23.926 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T13:51:23.926 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.926 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.926 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.927 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.927 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T13:51:23.927 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T13:51:23.928 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.928 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.928 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.928 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.928 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.929 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T13:51:23.929 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T13:51:23.929 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.929 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.930 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.930 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.930 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.931 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T13:51:23.931 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T13:51:23.931 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.931 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T13:51:23.931 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T13:51:23.931 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.932 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.932 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T13:51:23.932 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T13:51:23.932 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.933 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.933 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T13:51:23.933 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:51:23.933 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.934 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T13:51:23.934 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T13:51:23.934 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.934 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.934 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T13:51:23.934 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T13:51:23.935 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T13:51:23.935 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.936 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T13:51:23.936 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T13:51:23.936 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.936 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.936 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T13:51:23.937 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T13:51:23.937 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T13:51:23.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.938 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T13:51:23.938 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T13:51:23.938 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.938 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T13:51:23.938 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T13:51:23.939 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T13:51:23.939 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T13:51:23.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.940 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T13:51:23.940 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T13:51:23.940 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.940 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T13:51:23.940 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T13:51:23.940 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T13:51:23.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.941 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.941 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T13:51:23.942 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T13:51:23.942 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T13:51:23.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T13:51:23.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.943 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T13:51:23.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:23.943 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T13:51:23.943 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T13:51:23.944 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T13:51:23.944 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T13:51:23.944 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T13:51:23.944 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T13:51:23.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:23.945 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T13:51:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T13:51:23.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f33e806afc0 msgr2=0x7f33e806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f33e806afc0 0x7f33e806d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3400034000 tx=0x7f3400009030 comp rx=0 tx=0).stop 2023-10-16T13:51:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 msgr2=0x7f34040c39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 0x7f34040c39c0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f33f4010790 tx=0x7f33f4010c50 comp rx=0 tx=0).stop 2023-10-16T13:51:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 shutdown_connections 2023-10-16T13:51:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f33e806afc0 0x7f33e806d480 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f340415b510 0x7f340415b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34040c3560 0x7f34040c39c0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 --2- 172.21.15.60:0/3340855381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33f0003970 0x7f3404163af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.910+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 >> 172.21.15.60:0/3340855381 conn(0x7f34040ce6a0 msgr2=0x7f3404069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.911+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 shutdown_connections 2023-10-16T13:51:23.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:23.911+0000 7f340b459640 1 -- 172.21.15.60:0/3340855381 wait complete. 2023-10-16T13:51:23.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.177+0000 7f6d7cb65640 1 Processor -- start 2023-10-16T13:51:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.177+0000 7f6d7cb65640 1 -- start start 2023-10-16T13:51:24.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d7cb65640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6d78065ab0 0x7f6d780c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d7cb65640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6d780ce280 0x7f6d780ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d7cb65640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d780acd70 con 0x7f6d7806d220 2023-10-16T13:51:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d7cb65640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d780ac4a0 con 0x7f6d78065ab0 2023-10-16T13:51:24.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d7cb65640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d78065f40 con 0x7f6d780ce280 2023-10-16T13:51:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d777fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6d7806d220 0x7f6d780cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47996/0 (socket says 172.21.15.60:47996) 2023-10-16T13:51:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d777fe640 1 -- 172.21.15.60:0/766620363 learned_addr learned my addr 172.21.15.60:0/766620363 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.179+0000 7f6d6ffff640 1 --2- 172.21.15.60:0/766620363 >> v2:172.21.15.60:3300/0 conn(0x7f6d78065ab0 0x7f6d780c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d76ffd640 1 -- 172.21.15.60:0/766620363 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600876623 0 0) 0x7f6d780acd70 con 0x7f6d7806d220 2023-10-16T13:51:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d77fff640 1 --2- 172.21.15.60:0/766620363 >> v2:172.21.15.144:3300/0 conn(0x7f6d780ce280 0x7f6d780ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d76ffd640 1 -- 172.21.15.60:0/766620363 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d64003520 con 0x7f6d7806d220 2023-10-16T13:51:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d6ffff640 1 -- 172.21.15.60:0/766620363 >> v2:172.21.15.144:3300/0 conn(0x7f6d780ce280 msgr2=0x7f6d780ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d6ffff640 1 --2- 172.21.15.60:0/766620363 >> v2:172.21.15.144:3300/0 conn(0x7f6d780ce280 0x7f6d780ad200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d6ffff640 1 -- 172.21.15.60:0/766620363 >> v1:172.21.15.60:6789/0 conn(0x7f6d7806d220 legacy=0x7f6d780cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.180+0000 7f6d6ffff640 1 -- 172.21.15.60:0/766620363 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d780ac720 con 0x7f6d78065ab0 2023-10-16T13:51:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.181+0000 7f6d6ffff640 1 --2- 172.21.15.60:0/766620363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d78065ab0 0x7f6d780c95e0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f6d78071fd0 tx=0x7f6d6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b46188460d09a15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.182+0000 7f6d76ffd640 1 -- 172.21.15.60:0/766620363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6002e070 con 0x7f6d78065ab0 2023-10-16T13:51:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.182+0000 7f6d76ffd640 1 -- 172.21.15.60:0/766620363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d600315a0 con 0x7f6d78065ab0 2023-10-16T13:51:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.182+0000 7f6d76ffd640 1 -- 172.21.15.60:0/766620363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60031930 con 0x7f6d78065ab0 2023-10-16T13:51:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.182+0000 7f6d7cb65640 1 -- 172.21.15.60:0/766620363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d78065ab0 msgr2=0x7f6d780c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.182+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/766620363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d78065ab0 0x7f6d780c95e0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f6d78071fd0 tx=0x7f6d6002b840 comp rx=0 tx=0).stop 2023-10-16T13:51:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.183+0000 7f6d7cb65640 1 -- 172.21.15.60:0/766620363 shutdown_connections 2023-10-16T13:51:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.183+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/766620363 >> v2:172.21.15.144:3300/0 conn(0x7f6d780ce280 0x7f6d780ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.183+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/766620363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d78065ab0 0x7f6d780c95e0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.183+0000 7f6d7cb65640 1 -- 172.21.15.60:0/766620363 >> 172.21.15.60:0/766620363 conn(0x7f6d780b0a80 msgr2=0x7f6d780b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.183+0000 7f6d7cb65640 1 -- 172.21.15.60:0/766620363 shutdown_connections 2023-10-16T13:51:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.183+0000 7f6d7cb65640 1 -- 172.21.15.60:0/766620363 wait complete. 2023-10-16T13:51:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.185+0000 7f6d7cb65640 1 Processor -- start 2023-10-16T13:51:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.185+0000 7f6d7cb65640 1 -- start start 2023-10-16T13:51:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d7cb65640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6d7806d220 0x7f6d7809c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d7cb65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 0x7f6d7809c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d7cb65640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d78099a10 0x7f6d781f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d7cb65640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7809aae0 con 0x7f6d780ce280 2023-10-16T13:51:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d7cb65640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78099ed0 con 0x7f6d78099a10 2023-10-16T13:51:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d7cb65640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7809a1b0 con 0x7f6d7806d220 2023-10-16T13:51:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d6ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 0x7f6d7809c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d6ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 0x7f6d7809c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46372/0 (socket says 172.21.15.60:46372) 2023-10-16T13:51:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.186+0000 7f6d6ffff640 1 -- 172.21.15.60:0/3881391634 learned_addr learned my addr 172.21.15.60:0/3881391634 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d777fe640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6d7806d220 0x7f6d7809c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d77fff640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d78099a10 0x7f6d781f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d6ffff640 1 -- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6d7806d220 msgr2=0x7f6d7809c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d6ffff640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6d7806d220 0x7f6d7809c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d6ffff640 1 -- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d78099a10 msgr2=0x7f6d781f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d6ffff640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d78099a10 0x7f6d781f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d6ffff640 1 -- 172.21.15.60:0/3881391634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60009400 con 0x7f6d780ce280 2023-10-16T13:51:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d777fe640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6d7806d220 0x7f6d7809c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d77fff640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d78099a10 0x7f6d781f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d6ffff640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 0x7f6d7809c5c0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f6d60000c00 tx=0x7f6d600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60032840 con 0x7f6d780ce280 2023-10-16T13:51:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d600329c0 con 0x7f6d780ce280 2023-10-16T13:51:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7809a490 con 0x7f6d780ce280 2023-10-16T13:51:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.187+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60032d70 con 0x7f6d780ce280 2023-10-16T13:51:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.188+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d781f2900 con 0x7f6d780ce280 2023-10-16T13:51:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.189+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d780cdb70 con 0x7f6d780ce280 2023-10-16T13:51:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.190+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6d6004d3f0 con 0x7f6d780ce280 2023-10-16T13:51:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.191+0000 7f6d74ff9640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6d5806afc0 0x7f6d5806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.191+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6d600ab070 con 0x7f6d780ce280 2023-10-16T13:51:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.191+0000 7f6d777fe640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6d5806afc0 0x7f6d5806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.193+0000 7f6d777fe640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6d5806afc0 0x7f6d5806d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6d680011e0 tx=0x7f6d68009040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.196+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6d600ddfe0 con 0x7f6d780ce280 2023-10-16T13:51:24.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.393+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d7804e9b0 con 0x7f6d780ce280 2023-10-16T13:51:24.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.395+0000 7f6d74ff9640 1 -- 172.21.15.60:0/3881391634 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+12855 (secure 0 0 0) 0x7f6d600761b0 con 0x7f6d780ce280 2023-10-16T13:51:24.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:24.399 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":193,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:23.354730+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:24.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6d5806afc0 msgr2=0x7f6d5806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6d5806afc0 0x7f6d5806d480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6d680011e0 tx=0x7f6d68009040 comp rx=0 tx=0).stop 2023-10-16T13:51:24.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 msgr2=0x7f6d7809c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 0x7f6d7809c5c0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f6d60000c00 tx=0x7f6d600046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 shutdown_connections 2023-10-16T13:51:24.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6d5806afc0 0x7f6d5806d480 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d78099a10 0x7f6d781f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d780ce280 0x7f6d7809c5c0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 --2- 172.21.15.60:0/3881391634 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6d7806d220 0x7f6d7809c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 >> 172.21.15.60:0/3881391634 conn(0x7f6d780b0a80 msgr2=0x7f6d780c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:24.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 shutdown_connections 2023-10-16T13:51:24.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.398+0000 7f6d7cb65640 1 -- 172.21.15.60:0/3881391634 wait complete. 2023-10-16T13:51:24.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:51:24.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:51:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.658+0000 7f0995321640 1 Processor -- start 2023-10-16T13:51:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.658+0000 7f0995321640 1 -- start start 2023-10-16T13:51:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0995321640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f099015acd0 0x7f09900b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0995321640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09900afcf0 0x7f09900cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0995321640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09900cdc50 con 0x7f09900cd7e0 2023-10-16T13:51:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0995321640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099015b0b0 con 0x7f099015acd0 2023-10-16T13:51:24.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0995321640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09900b6730 con 0x7f09900afcf0 2023-10-16T13:51:24.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f098ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f099015acd0 0x7f09900b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0994b20640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f09900cd7e0 0x7f09900714c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47998/0 (socket says 172.21.15.60:47998) 2023-10-16T13:51:24.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f0994b20640 1 -- 172.21.15.60:0/3369085184 learned_addr learned my addr 172.21.15.60:0/3369085184 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:24.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.660+0000 7f098f7fe640 1 --2- 172.21.15.60:0/3369085184 >> v2:172.21.15.144:3300/0 conn(0x7f09900afcf0 0x7f09900cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.661+0000 7f098ffff640 1 -- 172.21.15.60:0/3369085184 >> v2:172.21.15.144:3300/0 conn(0x7f09900afcf0 msgr2=0x7f09900cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:24.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.661+0000 7f098ffff640 1 --2- 172.21.15.60:0/3369085184 >> v2:172.21.15.144:3300/0 conn(0x7f09900afcf0 0x7f09900cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.661+0000 7f098ffff640 1 -- 172.21.15.60:0/3369085184 >> v1:172.21.15.60:6789/0 conn(0x7f09900cd7e0 legacy=0x7f09900714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.661+0000 7f098ffff640 1 -- 172.21.15.60:0/3369085184 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09900cc6f0 con 0x7f099015acd0 2023-10-16T13:51:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.661+0000 7f098effd640 1 -- 172.21.15.60:0/3369085184 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269614052 0 0) 0x7f09900cdc50 con 0x7f09900cd7e0 2023-10-16T13:51:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.662+0000 7f098ffff640 1 --2- 172.21.15.60:0/3369085184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f099015acd0 0x7f09900b60a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f09780097a0 tx=0x7f097802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1353e7d985e8bab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.662+0000 7f098effd640 1 -- 172.21.15.60:0/3369085184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978031830 con 0x7f099015acd0 2023-10-16T13:51:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.662+0000 7f098effd640 1 -- 172.21.15.60:0/3369085184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0978031990 con 0x7f099015acd0 2023-10-16T13:51:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.662+0000 7f098effd640 1 -- 172.21.15.60:0/3369085184 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978031d20 con 0x7f099015acd0 2023-10-16T13:51:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 -- 172.21.15.60:0/3369085184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f099015acd0 msgr2=0x7f09900b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 --2- 172.21.15.60:0/3369085184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f099015acd0 0x7f09900b60a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f09780097a0 tx=0x7f097802b8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 -- 172.21.15.60:0/3369085184 shutdown_connections 2023-10-16T13:51:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 --2- 172.21.15.60:0/3369085184 >> v2:172.21.15.144:3300/0 conn(0x7f09900afcf0 0x7f09900cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 --2- 172.21.15.60:0/3369085184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f099015acd0 0x7f09900b60a0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 -- 172.21.15.60:0/3369085184 >> 172.21.15.60:0/3369085184 conn(0x7f09900b23b0 msgr2=0x7f09900b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 -- 172.21.15.60:0/3369085184 shutdown_connections 2023-10-16T13:51:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.663+0000 7f0995321640 1 -- 172.21.15.60:0/3369085184 wait complete. 2023-10-16T13:51:24.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.665+0000 7f0995321640 1 Processor -- start 2023-10-16T13:51:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.665+0000 7f0995321640 1 -- start start 2023-10-16T13:51:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f0995321640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 0x7f09901f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f0995321640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09900cd7e0 0x7f09900ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f0995321640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f099015acd0 0x7f09900ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f0995321640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099006b460 con 0x7f09900afcf0 2023-10-16T13:51:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f0995321640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099006b700 con 0x7f09900cd7e0 2023-10-16T13:51:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f0995321640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099006b9e0 con 0x7f099015acd0 2023-10-16T13:51:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 0x7f09901f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 0x7f09901f3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46394/0 (socket says 172.21.15.60:46394) 2023-10-16T13:51:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 -- 172.21.15.60:0/3585429154 learned_addr learned my addr 172.21.15.60:0/3585429154 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 -- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f099015acd0 msgr2=0x7f09900ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f099015acd0 0x7f09900ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 -- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09900cd7e0 msgr2=0x7f09900ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09900cd7e0 0x7f09900ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.666+0000 7f098ffff640 1 -- 172.21.15.60:0/3585429154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0978009400 con 0x7f09900afcf0 2023-10-16T13:51:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.667+0000 7f098ffff640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 0x7f09901f3370 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f097802bee0 tx=0x7f09780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.667+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097803a6f0 con 0x7f09900afcf0 2023-10-16T13:51:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.667+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097803a850 con 0x7f09900afcf0 2023-10-16T13:51:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.667+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099006bcc0 con 0x7f09900afcf0 2023-10-16T13:51:24.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.667+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097803ac00 con 0x7f09900afcf0 2023-10-16T13:51:24.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.667+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09901f38b0 con 0x7f09900afcf0 2023-10-16T13:51:24.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.668+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099006c1e0 con 0x7f09900afcf0 2023-10-16T13:51:24.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.670+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0978030380 con 0x7f09900afcf0 2023-10-16T13:51:24.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.670+0000 7f098cff9640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f097006af90 0x7f097006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:24.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.671+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09780aaff0 con 0x7f09900afcf0 2023-10-16T13:51:24.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.671+0000 7f098f7fe640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f097006af90 0x7f097006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:24.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.672+0000 7f098f7fe640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f097006af90 0x7f097006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09800094c0 tx=0x7f0980009570 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:24.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.675+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09780ddfe0 con 0x7f09900afcf0 2023-10-16T13:51:24.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.872+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f099004e9b0 con 0x7f09900afcf0 2023-10-16T13:51:24.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.873+0000 7f098cff9640 1 -- 172.21.15.60:0/3585429154 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09780350a0 con 0x7f09900afcf0 2023-10-16T13:51:24.877 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T13:51:24.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.876+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f097006af90 msgr2=0x7f097006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.876+0000 7f0995321640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f097006af90 0x7f097006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09800094c0 tx=0x7f0980009570 comp rx=0 tx=0).stop 2023-10-16T13:51:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.876+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 msgr2=0x7f09901f3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.876+0000 7f0995321640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 0x7f09901f3370 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f097802bee0 tx=0x7f09780046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 shutdown_connections 2023-10-16T13:51:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f097006af90 0x7f097006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f099015acd0 0x7f09900ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09900cd7e0 0x7f09900ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 --2- 172.21.15.60:0/3585429154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09900afcf0 0x7f09901f3370 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:24.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 >> 172.21.15.60:0/3585429154 conn(0x7f09900b23b0 msgr2=0x7f0990159600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:24.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 shutdown_connections 2023-10-16T13:51:24.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:24.877+0000 7f0995321640 1 -- 172.21.15.60:0/3585429154 wait complete. 2023-10-16T13:51:24.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:51:25.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.123+0000 7f04546a2640 1 Processor -- start 2023-10-16T13:51:25.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.123+0000 7f04546a2640 1 -- start start 2023-10-16T13:51:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.125+0000 7f04546a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f044c15aaa0 0x7f044c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.125+0000 7f04546a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 0x7f044c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.125+0000 7f04546a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044c15ed60 con 0x7f044c15ae80 2023-10-16T13:51:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.125+0000 7f04546a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c06b810 con 0x7f044c15aaa0 2023-10-16T13:51:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.125+0000 7f04546a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0707e0 con 0x7f044c0ce390 2023-10-16T13:51:25.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.126+0000 7f0452c3f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 0x7f044c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.126+0000 7f045243e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f044c15aaa0 0x7f044c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.126+0000 7f0452c3f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 0x7f044c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:42548/0 (socket says 172.21.15.60:42548) 2023-10-16T13:51:25.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.126+0000 7f0452c3f640 1 -- 172.21.15.60:0/639385139 learned_addr learned my addr 172.21.15.60:0/639385139 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:25.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.127+0000 7f0452c3f640 1 -- 172.21.15.60:0/639385139 >> v2:172.21.15.144:3300/0 conn(0x7f044c15aaa0 msgr2=0x7f044c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:25.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.127+0000 7f0452c3f640 1 --2- 172.21.15.60:0/639385139 >> v2:172.21.15.144:3300/0 conn(0x7f044c15aaa0 0x7f044c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.127+0000 7f0452c3f640 1 -- 172.21.15.60:0/639385139 >> v1:172.21.15.144:6789/0 conn(0x7f044c15ae80 legacy=0x7f044c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.127+0000 7f0452c3f640 1 -- 172.21.15.60:0/639385139 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c15ef90 con 0x7f044c0ce390 2023-10-16T13:51:25.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.127+0000 7f045143c640 1 -- 172.21.15.60:0/639385139 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975198958 0 0) 0x7f044c15ed60 con 0x7f044c15ae80 2023-10-16T13:51:25.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.127+0000 7f045243e640 1 --2- 172.21.15.60:0/639385139 >> v2:172.21.15.144:3300/0 conn(0x7f044c15aaa0 0x7f044c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:51:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.128+0000 7f0452c3f640 1 --2- 172.21.15.60:0/639385139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 0x7f044c070f50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0444009e50 tx=0x7f044402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c66f2e57c0dd87d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f045143c640 1 -- 172.21.15.60:0/639385139 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044402cb00 con 0x7f044c0ce390 2023-10-16T13:51:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f045143c640 1 -- 172.21.15.60:0/639385139 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044402cc60 con 0x7f044c0ce390 2023-10-16T13:51:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f045143c640 1 -- 172.21.15.60:0/639385139 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04440343f0 con 0x7f044c0ce390 2023-10-16T13:51:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f04546a2640 1 -- 172.21.15.60:0/639385139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 msgr2=0x7f044c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f04546a2640 1 --2- 172.21.15.60:0/639385139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 0x7f044c070f50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0444009e50 tx=0x7f044402bf60 comp rx=0 tx=0).stop 2023-10-16T13:51:25.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f04546a2640 1 -- 172.21.15.60:0/639385139 shutdown_connections 2023-10-16T13:51:25.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f04546a2640 1 --2- 172.21.15.60:0/639385139 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c0ce390 0x7f044c070f50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f04546a2640 1 --2- 172.21.15.60:0/639385139 >> v2:172.21.15.144:3300/0 conn(0x7f044c15aaa0 0x7f044c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.129+0000 7f04546a2640 1 -- 172.21.15.60:0/639385139 >> 172.21.15.60:0/639385139 conn(0x7f044c0cb410 msgr2=0x7f044c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:25.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.130+0000 7f04546a2640 1 -- 172.21.15.60:0/639385139 shutdown_connections 2023-10-16T13:51:25.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.130+0000 7f04546a2640 1 -- 172.21.15.60:0/639385139 wait complete. 2023-10-16T13:51:25.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.131+0000 7f04546a2640 1 Processor -- start 2023-10-16T13:51:25.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.132+0000 7f04546a2640 1 -- start start 2023-10-16T13:51:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f04546a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f044c0ce390 0x7f044c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f04546a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c15aaa0 0x7f044c157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f04546a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 0x7f044c15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f04546a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0c9520 con 0x7f044c15ae80 2023-10-16T13:51:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f04546a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0c6d80 con 0x7f044c0ce390 2023-10-16T13:51:25.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f04546a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0c7060 con 0x7f044c15aaa0 2023-10-16T13:51:25.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f0452c3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 0x7f044c15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f0452c3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 0x7f044c15cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46408/0 (socket says 172.21.15.60:46408) 2023-10-16T13:51:25.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f0452c3f640 1 -- 172.21.15.60:0/869210950 learned_addr learned my addr 172.21.15.60:0/869210950 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:25.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f0452c3f640 1 -- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c15aaa0 msgr2=0x7f044c157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f045243e640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f044c0ce390 0x7f044c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.133+0000 7f0451c3d640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c15aaa0 0x7f044c157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0452c3f640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c15aaa0 0x7f044c157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0452c3f640 1 -- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f044c0ce390 msgr2=0x7f044c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0452c3f640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f044c0ce390 0x7f044c0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0452c3f640 1 -- 172.21.15.60:0/869210950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448009400 con 0x7f044c15ae80 2023-10-16T13:51:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f045243e640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f044c0ce390 0x7f044c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0451c3d640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c15aaa0 0x7f044c157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0452c3f640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 0x7f044c15cd80 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f0444036040 tx=0x7f0444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:25.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444009510 con 0x7f044c15ae80 2023-10-16T13:51:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044403e450 con 0x7f044c15ae80 2023-10-16T13:51:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044403e7e0 con 0x7f044c15ae80 2023-10-16T13:51:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0444009b30 con 0x7f044c15ae80 2023-10-16T13:51:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.134+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c0c7620 con 0x7f044c15ae80 2023-10-16T13:51:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.136+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c04e160 con 0x7f044c15ae80 2023-10-16T13:51:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.137+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f044403e940 con 0x7f044c15ae80 2023-10-16T13:51:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.137+0000 7f0442ffd640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f043006af90 0x7f043006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.138+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f04440ac1a0 con 0x7f044c15ae80 2023-10-16T13:51:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.138+0000 7f045243e640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f043006af90 0x7f043006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.139+0000 7f045243e640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f043006af90 0x7f043006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0448009c50 tx=0x7f0448008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:25.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.142+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0444077480 con 0x7f044c15ae80 2023-10-16T13:51:25.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.340+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f044c0cdc60 con 0x7f044c15ae80 2023-10-16T13:51:25.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.341+0000 7f0442ffd640 1 -- 172.21.15.60:0/869210950 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v193) v1 ==== 84+0+65 (secure 0 0 0) 0x7f044407aaa0 con 0x7f044c15ae80 2023-10-16T13:51:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:25.345 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T13:51:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T13:51:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T13:51:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:25.346 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T13:51:25.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f043006af90 msgr2=0x7f043006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f043006af90 0x7f043006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0448009c50 tx=0x7f0448008040 comp rx=0 tx=0).stop 2023-10-16T13:51:25.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 msgr2=0x7f044c15cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 0x7f044c15cd80 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f0444036040 tx=0x7f0444004a40 comp rx=0 tx=0).stop 2023-10-16T13:51:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 shutdown_connections 2023-10-16T13:51:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f043006af90 0x7f043006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f044c15aaa0 0x7f044c157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f044c15ae80 0x7f044c15cd80 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 --2- 172.21.15.60:0/869210950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f044c0ce390 0x7f044c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 >> 172.21.15.60:0/869210950 conn(0x7f044c0cb410 msgr2=0x7f044c15a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 shutdown_connections 2023-10-16T13:51:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.344+0000 7f04546a2640 1 -- 172.21.15.60:0/869210950 wait complete. 2023-10-16T13:51:25.367 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:51:25.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:51:25.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.587+0000 7fed35cec640 1 Processor -- start 2023-10-16T13:51:25.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.588+0000 7fed35cec640 1 -- start start 2023-10-16T13:51:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.589+0000 7fed35cec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed3015cf60 0x7fed3006f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.589+0000 7fed35cec640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed300c9620 con 0x7fed300b0360 2023-10-16T13:51:25.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.589+0000 7fed35cec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed300ad4a0 con 0x7fed3006f9d0 2023-10-16T13:51:25.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.589+0000 7fed35cec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3015d340 con 0x7fed3015cf60 2023-10-16T13:51:25.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.590+0000 7fed2ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fed300b0360 0x7fed300ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48006/0 (socket says 172.21.15.60:48006) 2023-10-16T13:51:25.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.590+0000 7fed2ffff640 1 -- 172.21.15.60:0/3749784420 learned_addr learned my addr 172.21.15.60:0/3749784420 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:25.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.590+0000 7fed2f7fe640 1 --2- 172.21.15.60:0/3749784420 >> v2:172.21.15.144:3300/0 conn(0x7fed3015cf60 0x7fed3006f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.591+0000 7fed2effd640 1 -- 172.21.15.60:0/3749784420 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93616769 0 0) 0x7fed300c9620 con 0x7fed300b0360 2023-10-16T13:51:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.591+0000 7fed2effd640 1 -- 172.21.15.60:0/3749784420 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c003520 con 0x7fed300b0360 2023-10-16T13:51:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.591+0000 7fed2f7fe640 1 -- 172.21.15.60:0/3749784420 >> v1:172.21.15.144:6789/0 conn(0x7fed3006f9d0 legacy=0x7fed300c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.591+0000 7fed2f7fe640 1 -- 172.21.15.60:0/3749784420 >> v1:172.21.15.60:6789/0 conn(0x7fed300b0360 legacy=0x7fed300ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.591+0000 7fed2f7fe640 1 -- 172.21.15.60:0/3749784420 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300c8fd0 con 0x7fed3015cf60 2023-10-16T13:51:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.592+0000 7fed2f7fe640 1 --2- 172.21.15.60:0/3749784420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3015cf60 0x7fed3006f490 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed200097d0 tx=0x7fed2002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaf441a768366d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.593+0000 7fed2effd640 1 -- 172.21.15.60:0/3749784420 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20030800 con 0x7fed3015cf60 2023-10-16T13:51:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.593+0000 7fed2effd640 1 -- 172.21.15.60:0/3749784420 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed20030960 con 0x7fed3015cf60 2023-10-16T13:51:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.593+0000 7fed2effd640 1 -- 172.21.15.60:0/3749784420 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20030cf0 con 0x7fed3015cf60 2023-10-16T13:51:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 -- 172.21.15.60:0/3749784420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3015cf60 msgr2=0x7fed3006f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 --2- 172.21.15.60:0/3749784420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3015cf60 0x7fed3006f490 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed200097d0 tx=0x7fed2002b940 comp rx=0 tx=0).stop 2023-10-16T13:51:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 -- 172.21.15.60:0/3749784420 shutdown_connections 2023-10-16T13:51:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 --2- 172.21.15.60:0/3749784420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3015cf60 0x7fed3006f490 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 -- 172.21.15.60:0/3749784420 >> 172.21.15.60:0/3749784420 conn(0x7fed300cb360 msgr2=0x7fed300cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 -- 172.21.15.60:0/3749784420 shutdown_connections 2023-10-16T13:51:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.594+0000 7fed35cec640 1 -- 172.21.15.60:0/3749784420 wait complete. 2023-10-16T13:51:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.596+0000 7fed35cec640 1 Processor -- start 2023-10-16T13:51:25.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.597+0000 7fed35cec640 1 -- start start 2023-10-16T13:51:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed35cec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3006f9d0 0x7fed3014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed35cec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed300b0360 0x7fed300c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed35cec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 0x7fed30160850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed35cec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3015d870 con 0x7fed3015cf60 2023-10-16T13:51:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed35cec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3015db50 con 0x7fed3006f9d0 2023-10-16T13:51:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed35cec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3015de30 con 0x7fed300b0360 2023-10-16T13:51:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 0x7fed30160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 0x7fed30160850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46418/0 (socket says 172.21.15.60:46418) 2023-10-16T13:51:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.598+0000 7fed2ffff640 1 -- 172.21.15.60:0/1222347341 learned_addr learned my addr 172.21.15.60:0/1222347341 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2f7fe640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3006f9d0 0x7fed3014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed27fff640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed300b0360 0x7fed300c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2ffff640 1 -- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed300b0360 msgr2=0x7fed300c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2ffff640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed300b0360 0x7fed300c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2ffff640 1 -- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3006f9d0 msgr2=0x7fed3014de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2ffff640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3006f9d0 0x7fed3014de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2ffff640 1 -- 172.21.15.60:0/1222347341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20009400 con 0x7fed3015cf60 2023-10-16T13:51:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.599+0000 7fed2ffff640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 0x7fed30160850 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fed2800cb40 tx=0x7fed2800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.600+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2800aca0 con 0x7fed3015cf60 2023-10-16T13:51:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.600+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2800ae00 con 0x7fed3015cf60 2023-10-16T13:51:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.600+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed28012620 con 0x7fed3015cf60 2023-10-16T13:51:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.600+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3015e110 con 0x7fed3015cf60 2023-10-16T13:51:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.600+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3015d490 con 0x7fed3015cf60 2023-10-16T13:51:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.601+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3004e160 con 0x7fed3015cf60 2023-10-16T13:51:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.602+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed28007070 con 0x7fed3015cf60 2023-10-16T13:51:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.603+0000 7fed2cff9640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fed1006af90 0x7fed1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.603+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fed28089520 con 0x7fed3015cf60 2023-10-16T13:51:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.604+0000 7fed2f7fe640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fed1006af90 0x7fed1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.605+0000 7fed2f7fe640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fed1006af90 0x7fed1006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fed20036000 tx=0x7fed2003b040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:25.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.607+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed28054770 con 0x7fed3015cf60 2023-10-16T13:51:25.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:25.805+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fed300cb0d0 con 0x7fed3015cf60 2023-10-16T13:51:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.381+0000 7fed2cff9640 1 -- 172.21.15.60:0/1222347341 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v194) v1 ==== 138+0+0 (secure 0 0 0) 0x7fed280170b0 con 0x7fed3015cf60 2023-10-16T13:51:26.385 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T13:51:26.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fed1006af90 msgr2=0x7fed1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:26.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fed1006af90 0x7fed1006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fed20036000 tx=0x7fed2003b040 comp rx=0 tx=0).stop 2023-10-16T13:51:26.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 msgr2=0x7fed30160850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:26.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 0x7fed30160850 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fed2800cb40 tx=0x7fed2800a060 comp rx=0 tx=0).stop 2023-10-16T13:51:26.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 shutdown_connections 2023-10-16T13:51:26.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fed1006af90 0x7fed1006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fed300b0360 0x7fed300c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fed3015cf60 0x7fed30160850 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 --2- 172.21.15.60:0/1222347341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed3006f9d0 0x7fed3014de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 >> 172.21.15.60:0/1222347341 conn(0x7fed300cb360 msgr2=0x7fed3006fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:26.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.386+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 shutdown_connections 2023-10-16T13:51:26.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.387+0000 7fed35cec640 1 -- 172.21.15.60:0/1222347341 wait complete. 2023-10-16T13:51:26.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:51:26.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.640+0000 7f19d2013640 1 Processor -- start 2023-10-16T13:51:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.640+0000 7f19d2013640 1 -- start start 2023-10-16T13:51:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d2013640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19cc070bf0 0x7f19cc071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d2013640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19cc0ac890 0x7f19cc0accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d2013640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc070760 con 0x7f19cc15dc90 2023-10-16T13:51:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d2013640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0715c0 con 0x7f19cc0ac890 2023-10-16T13:51:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d2013640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0ad1f0 con 0x7f19cc070bf0 2023-10-16T13:51:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d1011640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f19cc15dc90 0x7f19cc0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48008/0 (socket says 172.21.15.60:48008) 2023-10-16T13:51:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d1011640 1 -- 172.21.15.60:0/2779928585 learned_addr learned my addr 172.21.15.60:0/2779928585 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d1812640 1 --2- 172.21.15.60:0/2779928585 >> v2:172.21.15.60:3300/0 conn(0x7f19cc0ac890 0x7f19cc0accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.642+0000 7f19d0810640 1 --2- 172.21.15.60:0/2779928585 >> v2:172.21.15.144:3300/0 conn(0x7f19cc070bf0 0x7f19cc071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.643+0000 7f19cbfff640 1 -- 172.21.15.60:0/2779928585 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609188526 0 0) 0x7f19cc070760 con 0x7f19cc15dc90 2023-10-16T13:51:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.643+0000 7f19cbfff640 1 -- 172.21.15.60:0/2779928585 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4003520 con 0x7f19cc15dc90 2023-10-16T13:51:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.643+0000 7f19d1812640 1 -- 172.21.15.60:0/2779928585 >> v2:172.21.15.144:3300/0 conn(0x7f19cc070bf0 msgr2=0x7f19cc071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.643+0000 7f19d1812640 1 --2- 172.21.15.60:0/2779928585 >> v2:172.21.15.144:3300/0 conn(0x7f19cc070bf0 0x7f19cc071080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.643+0000 7f19d1812640 1 -- 172.21.15.60:0/2779928585 >> v1:172.21.15.60:6789/0 conn(0x7f19cc15dc90 legacy=0x7f19cc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.643+0000 7f19d1812640 1 -- 172.21.15.60:0/2779928585 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0cdac0 con 0x7f19cc0ac890 2023-10-16T13:51:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.644+0000 7f19d1812640 1 --2- 172.21.15.60:0/2779928585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc0ac890 0x7f19cc0accb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f19cc071a90 tx=0x7f19c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eef40281efe0910 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.644+0000 7f19cbfff640 1 -- 172.21.15.60:0/2779928585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4031830 con 0x7f19cc0ac890 2023-10-16T13:51:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.644+0000 7f19cbfff640 1 -- 172.21.15.60:0/2779928585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4031990 con 0x7f19cc0ac890 2023-10-16T13:51:26.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.644+0000 7f19cbfff640 1 -- 172.21.15.60:0/2779928585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4031d00 con 0x7f19cc0ac890 2023-10-16T13:51:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 -- 172.21.15.60:0/2779928585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc0ac890 msgr2=0x7f19cc0accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 --2- 172.21.15.60:0/2779928585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc0ac890 0x7f19cc0accb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f19cc071a90 tx=0x7f19c402b8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 -- 172.21.15.60:0/2779928585 shutdown_connections 2023-10-16T13:51:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 --2- 172.21.15.60:0/2779928585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc0ac890 0x7f19cc0accb0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 --2- 172.21.15.60:0/2779928585 >> v2:172.21.15.144:3300/0 conn(0x7f19cc070bf0 0x7f19cc071080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 -- 172.21.15.60:0/2779928585 >> 172.21.15.60:0/2779928585 conn(0x7f19cc0ce6a0 msgr2=0x7f19cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 -- 172.21.15.60:0/2779928585 shutdown_connections 2023-10-16T13:51:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.645+0000 7f19d2013640 1 -- 172.21.15.60:0/2779928585 wait complete. 2023-10-16T13:51:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.647+0000 7f19d2013640 1 Processor -- start 2023-10-16T13:51:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.647+0000 7f19d2013640 1 -- start start 2023-10-16T13:51:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d2013640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 0x7f19cc0c2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d2013640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19cc0ac890 0x7f19cc156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d2013640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19cc15dc90 0x7f19cc15e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d2013640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0b1f20 con 0x7f19cc070bf0 2023-10-16T13:51:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d2013640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0c6780 con 0x7f19cc0ac890 2023-10-16T13:51:26.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d2013640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0b12c0 con 0x7f19cc15dc90 2023-10-16T13:51:26.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d1011640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 0x7f19cc0c2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:26.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d1011640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 0x7f19cc0c2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46428/0 (socket says 172.21.15.60:46428) 2023-10-16T13:51:26.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.648+0000 7f19d1011640 1 -- 172.21.15.60:0/3991894136 learned_addr learned my addr 172.21.15.60:0/3991894136 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:26.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d0810640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19cc0ac890 0x7f19cc156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:26.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1812640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19cc15dc90 0x7f19cc15e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1011640 1 -- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19cc15dc90 msgr2=0x7f19cc15e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1011640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19cc15dc90 0x7f19cc15e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1011640 1 -- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19cc0ac890 msgr2=0x7f19cc156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1011640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19cc0ac890 0x7f19cc156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:26.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1011640 1 -- 172.21.15.60:0/3991894136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4009400 con 0x7f19cc070bf0 2023-10-16T13:51:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.649+0000 7f19d1011640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 0x7f19cc0c2070 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00c390 tx=0x7f19bc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.650+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc005ce0 con 0x7f19cc070bf0 2023-10-16T13:51:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.650+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc005e40 con 0x7f19cc070bf0 2023-10-16T13:51:26.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.650+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0072f0 con 0x7f19cc070bf0 2023-10-16T13:51:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.650+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc0b1580 con 0x7f19cc070bf0 2023-10-16T13:51:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.650+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc0b1b40 con 0x7f19cc070bf0 2023-10-16T13:51:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.651+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc04e9b0 con 0x7f19cc070bf0 2023-10-16T13:51:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.652+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f19bc007450 con 0x7f19cc070bf0 2023-10-16T13:51:26.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.653+0000 7f19c9ffb640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f19b806af90 0x7f19b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:26.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.653+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f19bc088fa0 con 0x7f19cc070bf0 2023-10-16T13:51:26.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.654+0000 7f19d0810640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f19b806af90 0x7f19b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:26.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.655+0000 7f19d0810640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f19b806af90 0x7f19b806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0c7e70 tx=0x7f19c0003990 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:26.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.657+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19bc054190 con 0x7f19cc070bf0 2023-10-16T13:51:26.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:26.853+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f19cc0b4620 con 0x7f19cc070bf0 2023-10-16T13:51:27.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.400+0000 7f19c9ffb640 1 -- 172.21.15.60:0/3991894136 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v195) v1 ==== 170+0+0 (secure 0 0 0) 0x7f19bc0170d0 con 0x7f19cc070bf0 2023-10-16T13:51:27.404 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T13:51:27.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f19b806af90 msgr2=0x7f19b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:27.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f19b806af90 0x7f19b806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0c7e70 tx=0x7f19c0003990 comp rx=0 tx=0).stop 2023-10-16T13:51:27.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 msgr2=0x7f19cc0c2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:27.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 0x7f19cc0c2070 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00c390 tx=0x7f19bc00c850 comp rx=0 tx=0).stop 2023-10-16T13:51:27.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 shutdown_connections 2023-10-16T13:51:27.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f19b806af90 0x7f19b806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19cc15dc90 0x7f19cc15e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19cc0ac890 0x7f19cc156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 --2- 172.21.15.60:0/3991894136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19cc070bf0 0x7f19cc0c2070 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 >> 172.21.15.60:0/3991894136 conn(0x7f19cc0ce6a0 msgr2=0x7f19cc160850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:27.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 shutdown_connections 2023-10-16T13:51:27.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.404+0000 7f19d2013640 1 -- 172.21.15.60:0/3991894136 wait complete. 2023-10-16T13:51:27.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:51:27.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.629+0000 7f356cbc9640 1 Processor -- start 2023-10-16T13:51:27.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.630+0000 7f356cbc9640 1 -- start start 2023-10-16T13:51:27.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.631+0000 7f356cbc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35680ad350 0x7f3568069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.631+0000 7f356cbc9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f356806a2e0 0x7f35680c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.631+0000 7f356cbc9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356806a6f0 con 0x7f356815d150 2023-10-16T13:51:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.631+0000 7f356cbc9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356815d530 con 0x7f356806a2e0 2023-10-16T13:51:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.631+0000 7f356cbc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680ac890 con 0x7f35680ad350 2023-10-16T13:51:27.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.631+0000 7f3566d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f356806a2e0 0x7f35680c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3566d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f356806a2e0 0x7f35680c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46438/0 (socket says 172.21.15.60:46438) 2023-10-16T13:51:27.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3566d9d640 1 -- 172.21.15.60:0/3104054333 learned_addr learned my addr 172.21.15.60:0/3104054333 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f355dd9b640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35680ad350 0x7f3568069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3566d9d640 1 -- 172.21.15.60:0/3104054333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35680ad350 msgr2=0x7f3568069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3566d9d640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35680ad350 0x7f3568069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3566d9d640 1 -- 172.21.15.60:0/3104054333 >> v1:172.21.15.60:6789/0 conn(0x7f356815d150 legacy=0x7f3568160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3566d9d640 1 -- 172.21.15.60:0/3104054333 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3568161650 con 0x7f356806a2e0 2023-10-16T13:51:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f355dd9b640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35680ad350 0x7f3568069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.632+0000 7f3565d9b640 1 -- 172.21.15.60:0/3104054333 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157217209 0 0) 0x7f356806a6f0 con 0x7f356815d150 2023-10-16T13:51:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.633+0000 7f3566d9d640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f35680c9000 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f3560009e50 tx=0x7f356002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35d156fdae2f57d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f3565d9b640 1 -- 172.21.15.60:0/3104054333 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356002cb00 con 0x7f356806a2e0 2023-10-16T13:51:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f3565d9b640 1 -- 172.21.15.60:0/3104054333 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356002cc60 con 0x7f356806a2e0 2023-10-16T13:51:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f3565d9b640 1 -- 172.21.15.60:0/3104054333 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35600343f0 con 0x7f356806a2e0 2023-10-16T13:51:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 -- 172.21.15.60:0/3104054333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 msgr2=0x7f35680c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f35680c9000 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f3560009e50 tx=0x7f356002bf60 comp rx=0 tx=0).stop 2023-10-16T13:51:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 -- 172.21.15.60:0/3104054333 shutdown_connections 2023-10-16T13:51:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f35680c9000 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 --2- 172.21.15.60:0/3104054333 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35680ad350 0x7f3568069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 -- 172.21.15.60:0/3104054333 >> 172.21.15.60:0/3104054333 conn(0x7f35680ce9b0 msgr2=0x7f35680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 -- 172.21.15.60:0/3104054333 shutdown_connections 2023-10-16T13:51:27.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.634+0000 7f356cbc9640 1 -- 172.21.15.60:0/3104054333 wait complete. 2023-10-16T13:51:27.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.636+0000 7f356cbc9640 1 Processor -- start 2023-10-16T13:51:27.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.636+0000 7f356cbc9640 1 -- start start 2023-10-16T13:51:27.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.637+0000 7f356cbc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f3568070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:27.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.637+0000 7f356cbc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35680ad350 0x7f35680c3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:27.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.637+0000 7f356cbc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f356815d150 0x7f35681f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:27.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.637+0000 7f356cbc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680c2a40 con 0x7f356806a2e0 2023-10-16T13:51:27.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.637+0000 7f356cbc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680af130 con 0x7f35680ad350 2023-10-16T13:51:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.637+0000 7f356cbc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680c2c90 con 0x7f356815d150 2023-10-16T13:51:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f3568070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f3568070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46452/0 (socket says 172.21.15.60:46452) 2023-10-16T13:51:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 -- 172.21.15.60:0/2952645430 learned_addr learned my addr 172.21.15.60:0/2952645430 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 -- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f356815d150 msgr2=0x7f35681f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f356815d150 0x7f35681f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 -- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35680ad350 msgr2=0x7f35680c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35680ad350 0x7f35680c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 -- 172.21.15.60:0/2952645430 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3560009b30 con 0x7f356806a2e0 2023-10-16T13:51:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.638+0000 7f356659c640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f3568070b80 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f355800c410 tx=0x7f355800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.639+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35580035f0 con 0x7f356806a2e0 2023-10-16T13:51:27.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.639+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558003770 con 0x7f356806a2e0 2023-10-16T13:51:27.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.639+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558003b20 con 0x7f356806a2e0 2023-10-16T13:51:27.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.639+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35680c2f70 con 0x7f356806a2e0 2023-10-16T13:51:27.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.639+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35680710c0 con 0x7f356806a2e0 2023-10-16T13:51:27.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.640+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35680c3490 con 0x7f356806a2e0 2023-10-16T13:51:27.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.641+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3558003c80 con 0x7f356806a2e0 2023-10-16T13:51:27.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.642+0000 7f355f7fe640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f354806af90 0x7f354806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:27.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.643+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3558088df0 con 0x7f356806a2e0 2023-10-16T13:51:27.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.643+0000 7f355dd9b640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f354806af90 0x7f354806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:27.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.644+0000 7f355dd9b640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f354806af90 0x7f354806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3550006fd0 tx=0x7f3550008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:27.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.647+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35680c3490 con 0x7f356806a2e0 2023-10-16T13:51:27.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:27.879+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f35680c3490 con 0x7f356806a2e0 2023-10-16T13:51:28.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.404+0000 7f355f7fe640 1 -- 172.21.15.60:0/2952645430 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v91) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3558017020 con 0x7f356806a2e0 2023-10-16T13:51:28.408 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 22 and data pool 23 2023-10-16T13:51:28.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f354806af90 msgr2=0x7f354806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f354806af90 0x7f354806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3550006fd0 tx=0x7f3550008040 comp rx=0 tx=0).stop 2023-10-16T13:51:28.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 msgr2=0x7f3568070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f3568070b80 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f355800c410 tx=0x7f355800c8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 shutdown_connections 2023-10-16T13:51:28.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f354806af90 0x7f354806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f356815d150 0x7f35681f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35680ad350 0x7f35680c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 --2- 172.21.15.60:0/2952645430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f356806a2e0 0x7f3568070b80 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.409+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 >> 172.21.15.60:0/2952645430 conn(0x7f35680ce9b0 msgr2=0x7f356815b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:28.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.410+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 shutdown_connections 2023-10-16T13:51:28.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.410+0000 7f356cbc9640 1 -- 172.21.15.60:0/2952645430 wait complete. 2023-10-16T13:51:28.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:28.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.639+0000 7f87b32b4640 1 Processor -- start 2023-10-16T13:51:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.639+0000 7f87b32b4640 1 -- start start 2023-10-16T13:51:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b32b4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87ac06c040 0x7f87ac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b32b4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f87ac0cda80 0x7f87ac15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b32b4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ac06c960 con 0x7f87ac0ad970 2023-10-16T13:51:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b32b4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ac15b9d0 con 0x7f87ac06c040 2023-10-16T13:51:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b32b4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ac15bcb0 con 0x7f87ac0cda80 2023-10-16T13:51:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87ac06c040 0x7f87ac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87ac06c040 0x7f87ac06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46466/0 (socket says 172.21.15.60:46466) 2023-10-16T13:51:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 -- 172.21.15.60:0/105385626 learned_addr learned my addr 172.21.15.60:0/105385626 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 -- 172.21.15.60:0/105385626 >> v2:172.21.15.144:3300/0 conn(0x7f87ac0cda80 msgr2=0x7f87ac15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 --2- 172.21.15.60:0/105385626 >> v2:172.21.15.144:3300/0 conn(0x7f87ac0cda80 0x7f87ac15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 -- 172.21.15.60:0/105385626 >> v1:172.21.15.144:6789/0 conn(0x7f87ac0ad970 legacy=0x7f87ac15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:28.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.641+0000 7f87b22b2640 1 -- 172.21.15.60:0/105385626 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac15bf90 con 0x7f87ac06c040 2023-10-16T13:51:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.643+0000 7f87b22b2640 1 --2- 172.21.15.60:0/105385626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac06c040 0x7f87ac06c420 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0cf300 tx=0x7f87a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9e2bbe0c1ea2be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.643+0000 7f87b12b0640 1 -- 172.21.15.60:0/105385626 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8031870 con 0x7f87ac06c040 2023-10-16T13:51:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.643+0000 7f87b12b0640 1 -- 172.21.15.60:0/105385626 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a80319d0 con 0x7f87ac06c040 2023-10-16T13:51:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.643+0000 7f87b12b0640 1 -- 172.21.15.60:0/105385626 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8031d60 con 0x7f87ac06c040 2023-10-16T13:51:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.643+0000 7f87b32b4640 1 -- 172.21.15.60:0/105385626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac06c040 msgr2=0x7f87ac06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.643+0000 7f87b32b4640 1 --2- 172.21.15.60:0/105385626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac06c040 0x7f87ac06c420 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0cf300 tx=0x7f87a802b910 comp rx=0 tx=0).stop 2023-10-16T13:51:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.644+0000 7f87b32b4640 1 -- 172.21.15.60:0/105385626 shutdown_connections 2023-10-16T13:51:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.644+0000 7f87b32b4640 1 --2- 172.21.15.60:0/105385626 >> v2:172.21.15.144:3300/0 conn(0x7f87ac0cda80 0x7f87ac15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.644+0000 7f87b32b4640 1 --2- 172.21.15.60:0/105385626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac06c040 0x7f87ac06c420 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.644+0000 7f87b32b4640 1 -- 172.21.15.60:0/105385626 >> 172.21.15.60:0/105385626 conn(0x7f87ac0ce6a0 msgr2=0x7f87ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:28.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.644+0000 7f87b32b4640 1 -- 172.21.15.60:0/105385626 shutdown_connections 2023-10-16T13:51:28.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.644+0000 7f87b32b4640 1 -- 172.21.15.60:0/105385626 wait complete. 2023-10-16T13:51:28.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.645+0000 7f87b32b4640 1 Processor -- start 2023-10-16T13:51:28.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.646+0000 7f87b32b4640 1 -- start start 2023-10-16T13:51:28.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.646+0000 7f87b32b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87ac06c040 0x7f87ac156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.646+0000 7f87b32b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87ac0ad970 0x7f87ac0c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.646+0000 7f87b32b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 0x7f87ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b32b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0c7cb0 con 0x7f87ac0cda80 2023-10-16T13:51:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b32b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0c2d80 con 0x7f87ac0ad970 2023-10-16T13:51:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b32b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0c3060 con 0x7f87ac06c040 2023-10-16T13:51:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 0x7f87ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 0x7f87ac0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46474/0 (socket says 172.21.15.60:46474) 2023-10-16T13:51:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 -- 172.21.15.60:0/188994400 learned_addr learned my addr 172.21.15.60:0/188994400 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b22b2640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87ac06c040 0x7f87ac156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b1ab1640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87ac0ad970 0x7f87ac0c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 -- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87ac06c040 msgr2=0x7f87ac156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87ac06c040 0x7f87ac156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 -- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87ac0ad970 msgr2=0x7f87ac0c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87ac0ad970 0x7f87ac0c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.647+0000 7f87b2ab3640 1 -- 172.21.15.60:0/188994400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80094d0 con 0x7f87ac0cda80 2023-10-16T13:51:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.648+0000 7f87b2ab3640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 0x7f87ac0ac890 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f87a400b970 tx=0x7f87a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.648+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a400cb40 con 0x7f87ac0cda80 2023-10-16T13:51:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.648+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a400ccc0 con 0x7f87ac0cda80 2023-10-16T13:51:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.648+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ac0c3340 con 0x7f87ac0cda80 2023-10-16T13:51:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.648+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4014680 con 0x7f87ac0cda80 2023-10-16T13:51:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.648+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ac156db0 con 0x7f87ac0cda80 2023-10-16T13:51:28.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.649+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ac06b330 con 0x7f87ac0cda80 2023-10-16T13:51:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.651+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87a401d3b0 con 0x7f87ac0cda80 2023-10-16T13:51:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.652+0000 7f87a2ffd640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f879006af90 0x7f879006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.652+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87a40897b0 con 0x7f87ac0cda80 2023-10-16T13:51:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.653+0000 7f87b22b2640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f879006af90 0x7f879006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.654+0000 7f87b22b2640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f879006af90 0x7f879006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f87a8036000 tx=0x7f87a803c040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:28.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.656+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87a40546b0 con 0x7f87ac0cda80 2023-10-16T13:51:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.847+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87ac0c3860 con 0x7f87ac0cda80 2023-10-16T13:51:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.850+0000 7f87a2ffd640 1 -- 172.21.15.60:0/188994400 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16572 (secure 0 0 0) 0x7f87ac0c3860 con 0x7f87ac0cda80 2023-10-16T13:51:28.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:28.854 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":196,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:27.881156+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f879006af90 msgr2=0x7f879006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f879006af90 0x7f879006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f87a8036000 tx=0x7f87a803c040 comp rx=0 tx=0).stop 2023-10-16T13:51:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 msgr2=0x7f87ac0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 0x7f87ac0ac890 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f87a400b970 tx=0x7f87a400be30 comp rx=0 tx=0).stop 2023-10-16T13:51:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 shutdown_connections 2023-10-16T13:51:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f879006af90 0x7f879006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87ac0cda80 0x7f87ac0ac890 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87ac0ad970 0x7f87ac0c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 --2- 172.21.15.60:0/188994400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87ac06c040 0x7f87ac156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 >> 172.21.15.60:0/188994400 conn(0x7f87ac0ce6a0 msgr2=0x7f87ac06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 shutdown_connections 2023-10-16T13:51:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:28.853+0000 7f87b32b4640 1 -- 172.21.15.60:0/188994400 wait complete. 2023-10-16T13:51:28.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:29.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.093+0000 7f0e71f90640 1 Processor -- start 2023-10-16T13:51:29.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.093+0000 7f0e71f90640 1 -- start start 2023-10-16T13:51:29.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e71f90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c15aaa0 0x7f0e6c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e71f90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15ae80 0x7f0e6c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e71f90640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e6c0ccaa0 0x7f0e6c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e71f90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c06b810 con 0x7f0e6c15ae80 2023-10-16T13:51:29.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e71f90640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c15ed60 con 0x7f0e6c0ccaa0 2023-10-16T13:51:29.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e71f90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0707e0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.095+0000 7f0e6affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15ae80 0x7f0e6c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15ae80 0x7f0e6c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54942/0 (socket says 172.21.15.60:54942) 2023-10-16T13:51:29.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 -- 172.21.15.60:0/2380148837 learned_addr learned my addr 172.21.15.60:0/2380148837 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:29.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 -- 172.21.15.60:0/2380148837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c15aaa0 msgr2=0x7f0e6c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:29.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 --2- 172.21.15.60:0/2380148837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c15aaa0 0x7f0e6c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 -- 172.21.15.60:0/2380148837 >> v2:172.21.15.144:3300/0 conn(0x7f0e6c0ccaa0 msgr2=0x7f0e6c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:29.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 --2- 172.21.15.60:0/2380148837 >> v2:172.21.15.144:3300/0 conn(0x7f0e6c0ccaa0 0x7f0e6c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.096+0000 7f0e6affd640 1 -- 172.21.15.60:0/2380148837 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c15ef90 con 0x7f0e6c15ae80 2023-10-16T13:51:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.097+0000 7f0e6affd640 1 --2- 172.21.15.60:0/2380148837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15ae80 0x7f0e6c0cc560 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0ae3c0 tx=0x7f0e6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc024213c07ac7f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e6a7fc640 1 -- 172.21.15.60:0/2380148837 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60031870 con 0x7f0e6c15ae80 2023-10-16T13:51:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e6a7fc640 1 -- 172.21.15.60:0/2380148837 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e600319d0 con 0x7f0e6c15ae80 2023-10-16T13:51:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e6a7fc640 1 -- 172.21.15.60:0/2380148837 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60031d60 con 0x7f0e6c15ae80 2023-10-16T13:51:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 -- 172.21.15.60:0/2380148837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15ae80 msgr2=0x7f0e6c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 --2- 172.21.15.60:0/2380148837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15ae80 0x7f0e6c0cc560 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0ae3c0 tx=0x7f0e6002b910 comp rx=0 tx=0).stop 2023-10-16T13:51:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 -- 172.21.15.60:0/2380148837 shutdown_connections 2023-10-16T13:51:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 --2- 172.21.15.60:0/2380148837 >> v2:172.21.15.144:3300/0 conn(0x7f0e6c0ccaa0 0x7f0e6c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 --2- 172.21.15.60:0/2380148837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15ae80 0x7f0e6c0cc560 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 --2- 172.21.15.60:0/2380148837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c15aaa0 0x7f0e6c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.098+0000 7f0e71f90640 1 -- 172.21.15.60:0/2380148837 >> 172.21.15.60:0/2380148837 conn(0x7f0e6c0cb410 msgr2=0x7f0e6c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.099+0000 7f0e71f90640 1 -- 172.21.15.60:0/2380148837 shutdown_connections 2023-10-16T13:51:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.099+0000 7f0e71f90640 1 -- 172.21.15.60:0/2380148837 wait complete. 2023-10-16T13:51:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.101+0000 7f0e71f90640 1 Processor -- start 2023-10-16T13:51:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.101+0000 7f0e71f90640 1 -- start start 2023-10-16T13:51:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.102+0000 7f0e71f90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c0ccaa0 0x7f0e6c15deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.102+0000 7f0e71f90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 0x7f0e6c155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.102+0000 7f0e71f90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6c15ae80 0x7f0e6c15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e71f90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c157710 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e71f90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c156980 con 0x7f0e6c15ae80 2023-10-16T13:51:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e71f90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c156c60 con 0x7f0e6c0ccaa0 2023-10-16T13:51:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 0x7f0e6c155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 0x7f0e6c155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54952/0 (socket says 172.21.15.60:54952) 2023-10-16T13:51:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 -- 172.21.15.60:0/735100788 learned_addr learned my addr 172.21.15.60:0/735100788 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6b7fe640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c0ccaa0 0x7f0e6c15deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 -- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c0ccaa0 msgr2=0x7f0e6c15deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c0ccaa0 0x7f0e6c15deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 -- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6c15ae80 msgr2=0x7f0e6c15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6c15ae80 0x7f0e6c15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6affd640 1 -- 172.21.15.60:0/735100788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e600094d0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.103+0000 7f0e6b7fe640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c0ccaa0 0x7f0e6c15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.104+0000 7f0e6affd640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 0x7f0e6c155b50 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f0e6002b950 tx=0x7f0e600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.104+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6003bd40 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.104+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e600453f0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.104+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6003bea0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.104+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c0c3fc0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.104+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c156f40 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.106+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c04e160 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.107+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0e6004e450 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.108+0000 7f0e53fff640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f0e5806af90 0x7f0e5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.108+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e6002d040 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.108+0000 7f0e6b7fe640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f0e5806af90 0x7f0e5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.109+0000 7f0e6b7fe640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f0e5806af90 0x7f0e5806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0b0c40 tx=0x7f0e5c008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:29.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.114+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e60076eb0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.309+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e6c0cdc60 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.313+0000 7f0e53fff640 1 -- 172.21.15.60:0/735100788 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16572 (secure 0 0 0) 0x7f0e6007a4d0 con 0x7f0e6c15aaa0 2023-10-16T13:51:29.316 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:29.317 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":196,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:27.881156+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.315+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f0e5806af90 msgr2=0x7f0e5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.315+0000 7f0e71f90640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f0e5806af90 0x7f0e5806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0b0c40 tx=0x7f0e5c008040 comp rx=0 tx=0).stop 2023-10-16T13:51:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.315+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 msgr2=0x7f0e6c155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.315+0000 7f0e71f90640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 0x7f0e6c155b50 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f0e6002b950 tx=0x7f0e600046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.315+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 shutdown_connections 2023-10-16T13:51:29.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.315+0000 7f0e71f90640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f0e5806af90 0x7f0e5806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.316+0000 7f0e71f90640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6c15ae80 0x7f0e6c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.316+0000 7f0e71f90640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c15aaa0 0x7f0e6c155b50 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.316+0000 7f0e71f90640 1 --2- 172.21.15.60:0/735100788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e6c0ccaa0 0x7f0e6c15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.316+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 >> 172.21.15.60:0/735100788 conn(0x7f0e6c0cb410 msgr2=0x7f0e6c0b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:29.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.316+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 shutdown_connections 2023-10-16T13:51:29.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.316+0000 7f0e71f90640 1 -- 172.21.15.60:0/735100788 wait complete. 2023-10-16T13:51:29.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:51:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.568+0000 7ff71b1ce640 1 Processor -- start 2023-10-16T13:51:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.568+0000 7ff71b1ce640 1 -- start start 2023-10-16T13:51:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.569+0000 7ff71b1ce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff714159e90 0x7ff7140ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.569+0000 7ff71b1ce640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff71406c140 0x7ff71406c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.569+0000 7ff71b1ce640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71406cad0 con 0x7ff714159ab0 2023-10-16T13:51:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.569+0000 7ff71b1ce640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71415d890 con 0x7ff714159e90 2023-10-16T13:51:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.569+0000 7ff71b1ce640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff714070360 con 0x7ff71406c140 2023-10-16T13:51:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff714159e90 0x7ff7140ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff714159e90 0x7ff7140ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54964/0 (socket says 172.21.15.60:54964) 2023-10-16T13:51:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 -- 172.21.15.60:0/2127479518 learned_addr learned my addr 172.21.15.60:0/2127479518 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff718f6a640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff714159ab0 0x7ff71406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53862/0 (socket says 172.21.15.60:53862) 2023-10-16T13:51:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff71976b640 1 --2- 172.21.15.60:0/2127479518 >> v2:172.21.15.144:3300/0 conn(0x7ff71406c140 0x7ff71406c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 -- 172.21.15.60:0/2127479518 >> v2:172.21.15.144:3300/0 conn(0x7ff71406c140 msgr2=0x7ff71406c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 --2- 172.21.15.60:0/2127479518 >> v2:172.21.15.144:3300/0 conn(0x7ff71406c140 0x7ff71406c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 -- 172.21.15.60:0/2127479518 >> v1:172.21.15.60:6789/0 conn(0x7ff714159ab0 legacy=0x7ff71406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff713fff640 1 -- 172.21.15.60:0/2127479518 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140cd1f0 con 0x7ff714159e90 2023-10-16T13:51:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.570+0000 7ff7137fe640 1 -- 172.21.15.60:0/2127479518 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358199771 0 0) 0x7ff708001110 con 0x7ff714159ab0 2023-10-16T13:51:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.572+0000 7ff713fff640 1 --2- 172.21.15.60:0/2127479518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff714159e90 0x7ff7140ce370 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7ff7140adcd0 tx=0x7ff70002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef9da94cf13cf7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.572+0000 7ff7137fe640 1 -- 172.21.15.60:0/2127479518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7000317e0 con 0x7ff714159e90 2023-10-16T13:51:29.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.572+0000 7ff7137fe640 1 -- 172.21.15.60:0/2127479518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff700031940 con 0x7ff714159e90 2023-10-16T13:51:29.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.572+0000 7ff7137fe640 1 -- 172.21.15.60:0/2127479518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700031cd0 con 0x7ff714159e90 2023-10-16T13:51:29.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 -- 172.21.15.60:0/2127479518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff714159e90 msgr2=0x7ff7140ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/2127479518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff714159e90 0x7ff7140ce370 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7ff7140adcd0 tx=0x7ff70002b880 comp rx=0 tx=0).stop 2023-10-16T13:51:29.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 -- 172.21.15.60:0/2127479518 shutdown_connections 2023-10-16T13:51:29.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/2127479518 >> v2:172.21.15.144:3300/0 conn(0x7ff71406c140 0x7ff71406c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/2127479518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff714159e90 0x7ff7140ce370 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 -- 172.21.15.60:0/2127479518 >> 172.21.15.60:0/2127479518 conn(0x7ff7140ce9b0 msgr2=0x7ff7140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:29.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 -- 172.21.15.60:0/2127479518 shutdown_connections 2023-10-16T13:51:29.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.573+0000 7ff71b1ce640 1 -- 172.21.15.60:0/2127479518 wait complete. 2023-10-16T13:51:29.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.575+0000 7ff71b1ce640 1 Processor -- start 2023-10-16T13:51:29.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.575+0000 7ff71b1ce640 1 -- start start 2023-10-16T13:51:29.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71b1ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 0x7ff714167630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71b1ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff714159ab0 0x7ff71414f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71b1ce640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff714159e90 0x7ff7140ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71b1ce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140c5470 con 0x7ff71406c140 2023-10-16T13:51:29.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71b1ce640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140c36b0 con 0x7ff714159e90 2023-10-16T13:51:29.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71b1ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7140c3990 con 0x7ff714159ab0 2023-10-16T13:51:29.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff718f6a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 0x7ff714167630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff718f6a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 0x7ff714167630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54966/0 (socket says 172.21.15.60:54966) 2023-10-16T13:51:29.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff718f6a640 1 -- 172.21.15.60:0/3426081955 learned_addr learned my addr 172.21.15.60:0/3426081955 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:29.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff718f6a640 1 -- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff714159ab0 msgr2=0x7ff71414f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff713fff640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff714159ab0 0x7ff71414f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.576+0000 7ff71976b640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff714159e90 0x7ff7140ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff718f6a640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff714159ab0 0x7ff71414f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff718f6a640 1 -- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff714159e90 msgr2=0x7ff7140ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:29.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff718f6a640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff714159e90 0x7ff7140ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:29.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff718f6a640 1 -- 172.21.15.60:0/3426081955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7000094d0 con 0x7ff71406c140 2023-10-16T13:51:29.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff718f6a640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 0x7ff714167630 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7ff70800c780 tx=0x7ff70800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:29.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7080072d0 con 0x7ff71406c140 2023-10-16T13:51:29.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff708007450 con 0x7ff71406c140 2023-10-16T13:51:29.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7080047e0 con 0x7ff71406c140 2023-10-16T13:51:29.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7140c3c70 con 0x7ff71406c140 2023-10-16T13:51:29.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.577+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7140c4470 con 0x7ff71406c140 2023-10-16T13:51:29.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.579+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7140c4190 con 0x7ff71406c140 2023-10-16T13:51:29.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.580+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff708004960 con 0x7ff71406c140 2023-10-16T13:51:29.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.581+0000 7ff7117fa640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff6f806af90 0x7ff6f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.581+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff708088890 con 0x7ff71406c140 2023-10-16T13:51:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.582+0000 7ff713fff640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff6f806af90 0x7ff6f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:29.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.583+0000 7ff713fff640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff6f806af90 0x7ff6f806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff700036000 tx=0x7ff70003d040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:29.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.586+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff7140c4190 con 0x7ff71406c140 2023-10-16T13:51:29.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:29.821+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff71406cad0 con 0x7ff71406c140 2023-10-16T13:51:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.452+0000 7ff7117fa640 1 -- 172.21.15.60:0/3426081955 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v94) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff71406cad0 con 0x7ff71406c140 2023-10-16T13:51:30.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff6f806af90 msgr2=0x7ff6f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:30.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff6f806af90 0x7ff6f806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff700036000 tx=0x7ff70003d040 comp rx=0 tx=0).stop 2023-10-16T13:51:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 msgr2=0x7ff714167630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 0x7ff714167630 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7ff70800c780 tx=0x7ff70800cc40 comp rx=0 tx=0).stop 2023-10-16T13:51:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 shutdown_connections 2023-10-16T13:51:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7ff6f806af90 0x7ff6f806d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff714159ab0 0x7ff71414f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff714159e90 0x7ff7140ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 --2- 172.21.15.60:0/3426081955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff71406c140 0x7ff714167630 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 >> 172.21.15.60:0/3426081955 conn(0x7ff7140ce9b0 msgr2=0x7ff714152c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.457+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 shutdown_connections 2023-10-16T13:51:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.458+0000 7ff71b1ce640 1 -- 172.21.15.60:0/3426081955 wait complete. 2023-10-16T13:51:30.483 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:51:30.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:51:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.714+0000 7f6ada741640 1 Processor -- start 2023-10-16T13:51:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.714+0000 7f6ada741640 1 -- start start 2023-10-16T13:51:30.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ada741640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ad41428c0 0x7f6ad40c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:30.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ada741640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad40c9690 0x7f6ad40ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:30.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ada741640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad40cc6b0 con 0x7f6ad406b660 2023-10-16T13:51:30.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ada741640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad40c9b20 con 0x7f6ad41428c0 2023-10-16T13:51:30.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ada741640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad41599b0 con 0x7f6ad40c9690 2023-10-16T13:51:30.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ad3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6ad406b660 0x7f6ad406ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53870/0 (socket says 172.21.15.60:53870) 2023-10-16T13:51:30.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ad3fff640 1 -- 172.21.15.60:0/2683125952 learned_addr learned my addr 172.21.15.60:0/2683125952 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:30.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2683125952 >> v2:172.21.15.60:3300/0 conn(0x7f6ad41428c0 0x7f6ad40c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:30.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.716+0000 7f6ad8cde640 1 --2- 172.21.15.60:0/2683125952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad40c9690 0x7f6ad40ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:30.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.717+0000 7f6acb7fe640 1 -- 172.21.15.60:0/2683125952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad40c9690 msgr2=0x7f6ad40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.717+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2683125952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad40c9690 0x7f6ad40ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.717+0000 7f6acb7fe640 1 -- 172.21.15.60:0/2683125952 >> v1:172.21.15.60:6789/0 conn(0x7f6ad406b660 legacy=0x7f6ad406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.717+0000 7f6acb7fe640 1 -- 172.21.15.60:0/2683125952 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad406c290 con 0x7f6ad41428c0 2023-10-16T13:51:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.717+0000 7f6ad37fe640 1 -- 172.21.15.60:0/2683125952 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668218927 0 0) 0x7f6ad40cc6b0 con 0x7f6ad406b660 2023-10-16T13:51:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.718+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2683125952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad41428c0 0x7f6ad40c9150 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0097a0 tx=0x7f6abc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5324d8bde3913cce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.718+0000 7f6ad37fe640 1 -- 172.21.15.60:0/2683125952 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc031830 con 0x7f6ad41428c0 2023-10-16T13:51:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.718+0000 7f6ad37fe640 1 -- 172.21.15.60:0/2683125952 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc031990 con 0x7f6ad41428c0 2023-10-16T13:51:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.718+0000 7f6ad37fe640 1 -- 172.21.15.60:0/2683125952 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc031d20 con 0x7f6ad41428c0 2023-10-16T13:51:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 -- 172.21.15.60:0/2683125952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad41428c0 msgr2=0x7f6ad40c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 --2- 172.21.15.60:0/2683125952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad41428c0 0x7f6ad40c9150 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0097a0 tx=0x7f6abc02b8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 -- 172.21.15.60:0/2683125952 shutdown_connections 2023-10-16T13:51:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 --2- 172.21.15.60:0/2683125952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad40c9690 0x7f6ad40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 --2- 172.21.15.60:0/2683125952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad41428c0 0x7f6ad40c9150 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 -- 172.21.15.60:0/2683125952 >> 172.21.15.60:0/2683125952 conn(0x7f6ad40ce9b0 msgr2=0x7f6ad40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 -- 172.21.15.60:0/2683125952 shutdown_connections 2023-10-16T13:51:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.719+0000 7f6ada741640 1 -- 172.21.15.60:0/2683125952 wait complete. 2023-10-16T13:51:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.721+0000 7f6ada741640 1 Processor -- start 2023-10-16T13:51:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.721+0000 7f6ada741640 1 -- start start 2023-10-16T13:51:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ada741640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 0x7f6ad4150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ada741640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad40c9690 0x7f6ad4152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ada741640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad41428c0 0x7f6ad414c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ada741640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad4153260 con 0x7f6ad406b660 2023-10-16T13:51:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ada741640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40c83c0 con 0x7f6ad40c9690 2023-10-16T13:51:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ada741640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40c8630 con 0x7f6ad41428c0 2023-10-16T13:51:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.722+0000 7f6ad3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 0x7f6ad4150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 0x7f6ad4150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54986/0 (socket says 172.21.15.60:54986) 2023-10-16T13:51:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 -- 172.21.15.60:0/2494469771 learned_addr learned my addr 172.21.15.60:0/2494469771 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad8cde640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad41428c0 0x7f6ad414c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 -- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad41428c0 msgr2=0x7f6ad414c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad41428c0 0x7f6ad414c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 -- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad40c9690 msgr2=0x7f6ad4152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad40c9690 0x7f6ad4152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:30.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad40c9690 0x7f6ad4152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:30.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 -- 172.21.15.60:0/2494469771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc009400 con 0x7f6ad406b660 2023-10-16T13:51:30.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad40c9690 0x7f6ad4152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:30.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.723+0000 7f6ad3fff640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 0x7f6ad4150c50 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400c460 tx=0x7f6ac400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:30.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.724+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac40074e0 con 0x7f6ad406b660 2023-10-16T13:51:30.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.724+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac4007660 con 0x7f6ad406b660 2023-10-16T13:51:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.724+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4004a20 con 0x7f6ad406b660 2023-10-16T13:51:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.724+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad40c8910 con 0x7f6ad406b660 2023-10-16T13:51:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.724+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad4159560 con 0x7f6ad406b660 2023-10-16T13:51:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.725+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad406aa20 con 0x7f6ad406b660 2023-10-16T13:51:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.726+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ac4004b80 con 0x7f6ad406b660 2023-10-16T13:51:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.727+0000 7f6ad17fa640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6ab406af90 0x7f6ab406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.727+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ac4088a30 con 0x7f6ad406b660 2023-10-16T13:51:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.728+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6ab406af90 0x7f6ab406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.729+0000 7f6acb7fe640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6ab406af90 0x7f6ab406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6abc036000 tx=0x7f6abc03c040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.731+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ac4053930 con 0x7f6ad406b660 2023-10-16T13:51:30.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:30.973+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6ad404e160 con 0x7f6ad406b660 2023-10-16T13:51:31.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.455+0000 7f6ad17fa640 1 -- 172.21.15.60:0/2494469771 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v95) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ac4018070 con 0x7f6ad406b660 2023-10-16T13:51:31.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6ab406af90 msgr2=0x7f6ab406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6ab406af90 0x7f6ab406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6abc036000 tx=0x7f6abc03c040 comp rx=0 tx=0).stop 2023-10-16T13:51:31.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 msgr2=0x7f6ad4150c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 0x7f6ad4150c50 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400c460 tx=0x7f6ac400c920 comp rx=0 tx=0).stop 2023-10-16T13:51:31.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 shutdown_connections 2023-10-16T13:51:31.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f6ab406af90 0x7f6ab406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ad41428c0 0x7f6ad414c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ad40c9690 0x7f6ad4152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.459+0000 7f6ada741640 1 --2- 172.21.15.60:0/2494469771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ad406b660 0x7f6ad4150c50 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.460+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 >> 172.21.15.60:0/2494469771 conn(0x7f6ad40ce9b0 msgr2=0x7f6ad406f470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:31.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.460+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 shutdown_connections 2023-10-16T13:51:31.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.460+0000 7f6ada741640 1 -- 172.21.15.60:0/2494469771 wait complete. 2023-10-16T13:51:31.487 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:51:31.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:31.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.709+0000 7f43dd916640 1 Processor -- start 2023-10-16T13:51:31.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.709+0000 7f43dd916640 1 -- start start 2023-10-16T13:51:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43dd916640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43d8159b20 0x7f43d806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43dd916640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f43d80b0b20 0x7f43d806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43dd916640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d806f7f0 con 0x7f43d8159f00 2023-10-16T13:51:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43dd916640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d806c5e0 con 0x7f43d8159b20 2023-10-16T13:51:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43dd916640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d806c8c0 con 0x7f43d80b0b20 2023-10-16T13:51:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43d6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43d8159b20 0x7f43d806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43d6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43d8159b20 0x7f43d806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55000/0 (socket says 172.21.15.60:55000) 2023-10-16T13:51:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43d6ffd640 1 -- 172.21.15.60:0/3063099227 learned_addr learned my addr 172.21.15.60:0/3063099227 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.711+0000 7f43d77fe640 1 --2- 172.21.15.60:0/3063099227 >> v2:172.21.15.144:3300/0 conn(0x7f43d80b0b20 0x7f43d806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.712+0000 7f43d6ffd640 1 -- 172.21.15.60:0/3063099227 >> v2:172.21.15.144:3300/0 conn(0x7f43d80b0b20 msgr2=0x7f43d806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.712+0000 7f43d6ffd640 1 --2- 172.21.15.60:0/3063099227 >> v2:172.21.15.144:3300/0 conn(0x7f43d80b0b20 0x7f43d806fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.712+0000 7f43d6ffd640 1 -- 172.21.15.60:0/3063099227 >> v1:172.21.15.144:6789/0 conn(0x7f43d8159f00 legacy=0x7f43d80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.712+0000 7f43d6ffd640 1 -- 172.21.15.60:0/3063099227 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d806cba0 con 0x7f43d8159b20 2023-10-16T13:51:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.713+0000 7f43d6ffd640 1 --2- 172.21.15.60:0/3063099227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8159b20 0x7f43d806bbe0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f43d80c8bb0 tx=0x7f43c802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d142f23bc2d94f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.713+0000 7f43d67fc640 1 -- 172.21.15.60:0/3063099227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c802ee90 con 0x7f43d8159b20 2023-10-16T13:51:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.713+0000 7f43d67fc640 1 -- 172.21.15.60:0/3063099227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8036440 con 0x7f43d8159b20 2023-10-16T13:51:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.713+0000 7f43d67fc640 1 -- 172.21.15.60:0/3063099227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80367d0 con 0x7f43d8159b20 2023-10-16T13:51:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 -- 172.21.15.60:0/3063099227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8159b20 msgr2=0x7f43d806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 --2- 172.21.15.60:0/3063099227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8159b20 0x7f43d806bbe0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f43d80c8bb0 tx=0x7f43c802e0e0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 -- 172.21.15.60:0/3063099227 shutdown_connections 2023-10-16T13:51:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 --2- 172.21.15.60:0/3063099227 >> v2:172.21.15.144:3300/0 conn(0x7f43d80b0b20 0x7f43d806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 --2- 172.21.15.60:0/3063099227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d8159b20 0x7f43d806bbe0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 -- 172.21.15.60:0/3063099227 >> 172.21.15.60:0/3063099227 conn(0x7f43d80cb410 msgr2=0x7f43d80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 -- 172.21.15.60:0/3063099227 shutdown_connections 2023-10-16T13:51:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.714+0000 7f43dd916640 1 -- 172.21.15.60:0/3063099227 wait complete. 2023-10-16T13:51:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.716+0000 7f43dd916640 1 Processor -- start 2023-10-16T13:51:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.716+0000 7f43dd916640 1 -- start start 2023-10-16T13:51:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43dd916640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 0x7f43d8150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43dd916640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159b20 0x7f43d80c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43dd916640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d8159f00 0x7f43d80c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43dd916640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d8150f40 con 0x7f43d80b0b20 2023-10-16T13:51:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43dd916640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d8151430 con 0x7f43d8159b20 2023-10-16T13:51:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43dd916640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80b3910 con 0x7f43d8159f00 2023-10-16T13:51:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 0x7f43d8150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 0x7f43d8150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55004/0 (socket says 172.21.15.60:55004) 2023-10-16T13:51:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.717+0000 7f43d6ffd640 1 -- 172.21.15.60:0/1002906837 learned_addr learned my addr 172.21.15.60:0/1002906837 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d6ffd640 1 -- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d8159f00 msgr2=0x7f43d80c2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d6ffd640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d8159f00 0x7f43d80c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d6ffd640 1 -- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159b20 msgr2=0x7f43d80c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d6ffd640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159b20 0x7f43d80c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d6ffd640 1 -- 172.21.15.60:0/1002906837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c800b770 con 0x7f43d80b0b20 2023-10-16T13:51:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d6ffd640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 0x7f43d8150a00 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f43c802e120 tx=0x7f43c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:31.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80348d0 con 0x7f43d80b0b20 2023-10-16T13:51:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8034a30 con 0x7f43d80b0b20 2023-10-16T13:51:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d80b3bf0 con 0x7f43d80b0b20 2023-10-16T13:51:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.718+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8034de0 con 0x7f43d80b0b20 2023-10-16T13:51:31.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.719+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d815a690 con 0x7f43d80b0b20 2023-10-16T13:51:31.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.720+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d804e160 con 0x7f43d80b0b20 2023-10-16T13:51:31.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.721+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43c80403e0 con 0x7f43d80b0b20 2023-10-16T13:51:31.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.722+0000 7f43d4ff9640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f43b806af90 0x7f43b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:31.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.722+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43c80abdf0 con 0x7f43d80b0b20 2023-10-16T13:51:31.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.723+0000 7f43cffff640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f43b806af90 0x7f43b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:31.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.724+0000 7f43cffff640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f43b806af90 0x7f43b806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f43d815a2e0 tx=0x7f43c00079b0 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.727+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43c8076cf0 con 0x7f43d80b0b20 2023-10-16T13:51:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.970+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43d80c1d00 con 0x7f43d80b0b20 2023-10-16T13:51:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.971+0000 7f43d4ff9640 1 -- 172.21.15.60:0/1002906837 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f43c803b090 con 0x7f43d80b0b20 2023-10-16T13:51:31.975 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:31.975 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:51:27.881990+0000","modified":"2023-10-16T13:51:30.974922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9966":{"gid":9966,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/3850041246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3850041246},{"type":"v1","addr":"172.21.15.60:6835","nonce":3850041246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:51:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 95 2023-10-16T13:51:31.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f43b806af90 msgr2=0x7f43b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f43b806af90 0x7f43b806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f43d815a2e0 tx=0x7f43c00079b0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 msgr2=0x7f43d8150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 0x7f43d8150a00 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f43c802e120 tx=0x7f43c80046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 shutdown_connections 2023-10-16T13:51:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f43b806af90 0x7f43b806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43d8159f00 0x7f43d80c2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43d8159b20 0x7f43d80c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 --2- 172.21.15.60:0/1002906837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43d80b0b20 0x7f43d8150a00 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 >> 172.21.15.60:0/1002906837 conn(0x7f43d80cb410 msgr2=0x7f43d80b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 shutdown_connections 2023-10-16T13:51:31.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:31.974+0000 7f43dd916640 1 -- 172.21.15.60:0/1002906837 wait complete. 2023-10-16T13:51:31.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:51:32.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.221+0000 7fab4b573640 1 Processor -- start 2023-10-16T13:51:32.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.221+0000 7fab4b573640 1 -- start start 2023-10-16T13:51:32.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab4b573640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fab4406c040 0x7fab4406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab4b573640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4415f4a0 con 0x7fab440ad970 2023-10-16T13:51:32.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab4b573640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4406c960 con 0x7fab440cda80 2023-10-16T13:51:32.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab4b573640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4415b9d0 con 0x7fab4406c040 2023-10-16T13:51:32.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab49d70640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fab440ad970 0x7fab4415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53882/0 (socket says 172.21.15.60:53882) 2023-10-16T13:51:32.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab49d70640 1 -- 172.21.15.60:0/2476868816 learned_addr learned my addr 172.21.15.60:0/2476868816 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:32.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.223+0000 7fab4a571640 1 --2- 172.21.15.60:0/2476868816 >> v2:172.21.15.144:3300/0 conn(0x7fab4406c040 0x7fab4406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381043637 0 0) 0x7fab4415f4a0 con 0x7fab440ad970 2023-10-16T13:51:32.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab30003520 con 0x7fab440ad970 2023-10-16T13:51:32.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 652865212 0 0) 0x7fab30003520 con 0x7fab440ad970 2023-10-16T13:51:32.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 >> v2:172.21.15.144:3300/0 conn(0x7fab4406c040 msgr2=0x7fab4406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:32.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 --2- 172.21.15.60:0/2476868816 >> v2:172.21.15.144:3300/0 conn(0x7fab4406c040 0x7fab4406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 >> v1:172.21.15.144:6789/0 conn(0x7fab440cda80 legacy=0x7fab4415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4415bcb0 con 0x7fab440ad970 2023-10-16T13:51:32.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fab34001670 con 0x7fab440ad970 2023-10-16T13:51:32.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 >> v1:172.21.15.60:6789/0 conn(0x7fab440ad970 legacy=0x7fab4415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30003970 0x7fab30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab300047c0 0x7fab30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab30005180 0x7fab3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4415b9d0 con 0x7fab30005180 2023-10-16T13:51:32.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3000c020 con 0x7fab300047c0 2023-10-16T13:51:32.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.224+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3000c1e0 con 0x7fab30003970 2023-10-16T13:51:32.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4ad72640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab30005180 0x7fab3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab49d70640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab300047c0 0x7fab30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4a571640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30003970 0x7fab30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4ad72640 1 -- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30003970 msgr2=0x7fab30003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:32.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4ad72640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30003970 0x7fab30003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4ad72640 1 -- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab300047c0 msgr2=0x7fab30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:32.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4ad72640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab300047c0 0x7fab30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.225+0000 7fab4ad72640 1 -- 172.21.15.60:0/2476868816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30003520 con 0x7fab30005180 2023-10-16T13:51:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.226+0000 7fab4ad72640 1 -- 172.21.15.60:0/2476868816 reap_dead start 2023-10-16T13:51:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.226+0000 7fab4ad72640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab30005180 0x7fab3000b980 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fab3c001e50 tx=0x7fab300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1646d56c2fb08f5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:32.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c003b70 con 0x7fab30005180 2023-10-16T13:51:32.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c003cf0 con 0x7fab30005180 2023-10-16T13:51:32.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4956f640 1 -- 172.21.15.60:0/2476868816 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c005310 con 0x7fab30005180 2023-10-16T13:51:32.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4b573640 1 -- 172.21.15.60:0/2476868816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab30005180 msgr2=0x7fab3000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4b573640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab30005180 0x7fab3000b980 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fab3c001e50 tx=0x7fab300057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4b573640 1 -- 172.21.15.60:0/2476868816 shutdown_connections 2023-10-16T13:51:32.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4b573640 1 --2- 172.21.15.60:0/2476868816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab30005180 0x7fab3000b980 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4b573640 1 -- 172.21.15.60:0/2476868816 >> 172.21.15.60:0/2476868816 conn(0x7fab440ce6a0 msgr2=0x7fab440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:32.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.227+0000 7fab4b573640 1 -- 172.21.15.60:0/2476868816 shutdown_connections 2023-10-16T13:51:32.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.228+0000 7fab4b573640 1 -- 172.21.15.60:0/2476868816 wait complete. 2023-10-16T13:51:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.230+0000 7fab4b573640 1 Processor -- start 2023-10-16T13:51:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.230+0000 7fab4b573640 1 -- start start 2023-10-16T13:51:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4b573640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30005180 0x7fab44153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4b573640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 0x7fab440c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4b573640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab440c6d60 0x7fab440c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4b573640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440c7720 con 0x7fab441551d0 2023-10-16T13:51:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4b573640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab44156370 con 0x7fab440c6d60 2023-10-16T13:51:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4b573640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab44151fd0 con 0x7fab30005180 2023-10-16T13:51:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab49d70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 0x7fab440c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab49d70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 0x7fab440c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55022/0 (socket says 172.21.15.60:55022) 2023-10-16T13:51:32.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab49d70640 1 -- 172.21.15.60:0/2869182167 learned_addr learned my addr 172.21.15.60:0/2869182167 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4a571640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30005180 0x7fab44153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.231+0000 7fab4ad72640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab440c6d60 0x7fab440c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab49d70640 1 -- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30005180 msgr2=0x7fab44153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab49d70640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30005180 0x7fab44153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab49d70640 1 -- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab440c6d60 msgr2=0x7fab440c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab49d70640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab440c6d60 0x7fab440c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab49d70640 1 -- 172.21.15.60:0/2869182167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab44155760 con 0x7fab441551d0 2023-10-16T13:51:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab49d70640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 0x7fab440c7fd0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fab3400e690 tx=0x7fab3400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3400c6c0 con 0x7fab441551d0 2023-10-16T13:51:32.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3400c820 con 0x7fab441551d0 2023-10-16T13:51:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3400cbd0 con 0x7fab441551d0 2023-10-16T13:51:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.232+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab44155a40 con 0x7fab441551d0 2023-10-16T13:51:32.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.233+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab44156000 con 0x7fab441551d0 2023-10-16T13:51:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.234+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4406b330 con 0x7fab441551d0 2023-10-16T13:51:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.235+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab3400cd30 con 0x7fab441551d0 2023-10-16T13:51:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.236+0000 7fab3affd640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab2806af90 0x7fab2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.236+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab34089780 con 0x7fab441551d0 2023-10-16T13:51:32.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.236+0000 7fab4a571640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab2806af90 0x7fab2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.237+0000 7fab4a571640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab2806af90 0x7fab2806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab40009a10 tx=0x7fab40008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:32.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.240+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab34054680 con 0x7fab441551d0 2023-10-16T13:51:32.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.439+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab44156190 con 0x7fab441551d0 2023-10-16T13:51:32.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.441+0000 7fab3affd640 1 -- 172.21.15.60:0/2869182167 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16559 (secure 0 0 0) 0x7fab34057ca0 con 0x7fab441551d0 2023-10-16T13:51:32.445 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:32.445 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":197,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T13:51:29.423098+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:51:32.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab2806af90 msgr2=0x7fab2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab2806af90 0x7fab2806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab40009a10 tx=0x7fab40008040 comp rx=0 tx=0).stop 2023-10-16T13:51:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 msgr2=0x7fab440c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 0x7fab440c7fd0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fab3400e690 tx=0x7fab3400eb50 comp rx=0 tx=0).stop 2023-10-16T13:51:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 shutdown_connections 2023-10-16T13:51:32.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab2806af90 0x7fab2806d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab440c6d60 0x7fab440c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab441551d0 0x7fab440c7fd0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 --2- 172.21.15.60:0/2869182167 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab30005180 0x7fab44153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 >> 172.21.15.60:0/2869182167 conn(0x7fab440ce6a0 msgr2=0x7fab44069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.444+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 shutdown_connections 2023-10-16T13:51:32.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.445+0000 7fab4b573640 1 -- 172.21.15.60:0/2869182167 wait complete. 2023-10-16T13:51:32.462 INFO:tasks.ceph.mds.c.smithi060.stderr:2023-10-16T13:51:32.459+0000 7f627adaf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:51:32.468 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:51:32.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:51:32.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.691+0000 7f4ff54a2640 1 Processor -- start 2023-10-16T13:51:32.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.691+0000 7f4ff54a2640 1 -- start start 2023-10-16T13:51:32.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4ff54a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4ff00b0b20 0x7f4ff01615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4ff54a2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff015e3e0 con 0x7f4ff00cd0c0 2023-10-16T13:51:32.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4ff54a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00cd4a0 con 0x7f4ff015d3f0 2023-10-16T13:51:32.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4ff54a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff015e100 con 0x7f4ff00b0b20 2023-10-16T13:51:32.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4feeffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ff00cd0c0 0x7f4ff00b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53892/0 (socket says 172.21.15.60:53892) 2023-10-16T13:51:32.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4feeffd640 1 -- 172.21.15.60:0/1663387112 learned_addr learned my addr 172.21.15.60:0/1663387112 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:32.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.693+0000 7f4fee7fc640 1 --2- 172.21.15.60:0/1663387112 >> v2:172.21.15.144:3300/0 conn(0x7f4ff00b0b20 0x7f4ff01615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.694+0000 7f4fedffb640 1 -- 172.21.15.60:0/1663387112 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013466349 0 0) 0x7f4ff015e3e0 con 0x7f4ff00cd0c0 2023-10-16T13:51:32.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.694+0000 7f4fedffb640 1 -- 172.21.15.60:0/1663387112 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8003520 con 0x7f4ff00cd0c0 2023-10-16T13:51:32.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.694+0000 7f4fee7fc640 1 -- 172.21.15.60:0/1663387112 >> v1:172.21.15.144:6789/0 conn(0x7f4ff015d3f0 legacy=0x7f4ff015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.694+0000 7f4fee7fc640 1 -- 172.21.15.60:0/1663387112 >> v1:172.21.15.60:6789/0 conn(0x7f4ff00cd0c0 legacy=0x7f4ff00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.694+0000 7f4fedffb640 1 -- 172.21.15.60:0/1663387112 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4115034814 0 0) 0x7f4fd8003520 con 0x7f4ff00cd0c0 2023-10-16T13:51:32.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.694+0000 7f4fee7fc640 1 -- 172.21.15.60:0/1663387112 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0160b10 con 0x7f4ff00b0b20 2023-10-16T13:51:32.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.696+0000 7f4fee7fc640 1 --2- 172.21.15.60:0/1663387112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00b0b20 0x7f4ff01615f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40097a0 tx=0x7f4fe402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbbed33beb43911d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:32.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4fedffb640 1 -- 172.21.15.60:0/1663387112 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0001670 con 0x7f4ff00b0b20 2023-10-16T13:51:32.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4fedffb640 1 -- 172.21.15.60:0/1663387112 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe402e050 con 0x7f4ff00b0b20 2023-10-16T13:51:32.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4fedffb640 1 -- 172.21.15.60:0/1663387112 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4030730 con 0x7f4ff00b0b20 2023-10-16T13:51:32.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4ff54a2640 1 -- 172.21.15.60:0/1663387112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00b0b20 msgr2=0x7f4ff01615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/1663387112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00b0b20 0x7f4ff01615f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40097a0 tx=0x7f4fe402b790 comp rx=0 tx=0).stop 2023-10-16T13:51:32.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4ff54a2640 1 -- 172.21.15.60:0/1663387112 shutdown_connections 2023-10-16T13:51:32.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/1663387112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00b0b20 0x7f4ff01615f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.697+0000 7f4ff54a2640 1 -- 172.21.15.60:0/1663387112 >> 172.21.15.60:0/1663387112 conn(0x7f4ff00cb410 msgr2=0x7f4ff00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:32.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.698+0000 7f4ff54a2640 1 -- 172.21.15.60:0/1663387112 shutdown_connections 2023-10-16T13:51:32.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.698+0000 7f4ff54a2640 1 -- 172.21.15.60:0/1663387112 wait complete. 2023-10-16T13:51:32.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.700+0000 7f4ff54a2640 1 Processor -- start 2023-10-16T13:51:32.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.700+0000 7f4ff54a2640 1 -- start start 2023-10-16T13:51:32.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.701+0000 7f4ff54a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 0x7f4ff00b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.701+0000 7f4ff54a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00cd0c0 0x7f4ff014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.701+0000 7f4ff54a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff015d3f0 0x7f4ff014f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.701+0000 7f4ff54a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff014ecf0 con 0x7f4ff00b0b20 2023-10-16T13:51:32.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.701+0000 7f4ff54a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff014d220 con 0x7f4ff00cd0c0 2023-10-16T13:51:32.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.701+0000 7f4ff54a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff014de00 con 0x7f4ff015d3f0 2023-10-16T13:51:32.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 0x7f4ff00b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 0x7f4ff00b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55036/0 (socket says 172.21.15.60:55036) 2023-10-16T13:51:32.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 -- 172.21.15.60:0/3828258273 learned_addr learned my addr 172.21.15.60:0/3828258273 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:32.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4fee7fc640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00cd0c0 0x7f4ff014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4fef7fe640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff015d3f0 0x7f4ff014f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 -- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff015d3f0 msgr2=0x7f4ff014f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff015d3f0 0x7f4ff014f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 -- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00cd0c0 msgr2=0x7f4ff014cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00cd0c0 0x7f4ff014cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.702+0000 7f4feeffd640 1 -- 172.21.15.60:0/3828258273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4009400 con 0x7f4ff00b0b20 2023-10-16T13:51:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.703+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 0x7f4ff00b0520 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000c790 tx=0x7f4fe000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.703+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0005ad0 con 0x7f4ff00b0b20 2023-10-16T13:51:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.703+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe0005c30 con 0x7f4ff00b0b20 2023-10-16T13:51:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.703+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0007070 con 0x7f4ff00b0b20 2023-10-16T13:51:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.703+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff014e0e0 con 0x7f4ff00b0b20 2023-10-16T13:51:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.703+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff014e6a0 con 0x7f4ff00b0b20 2023-10-16T13:51:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.705+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff004e160 con 0x7f4ff00b0b20 2023-10-16T13:51:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.706+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4fe00071d0 con 0x7f4ff00b0b20 2023-10-16T13:51:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.707+0000 7f4fd77fe640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4fdc06af90 0x7f4fdc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.707+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fe0089490 con 0x7f4ff00b0b20 2023-10-16T13:51:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.708+0000 7f4fee7fc640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4fdc06af90 0x7f4fdc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:32.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.709+0000 7f4fee7fc640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4fdc06af90 0x7f4fdc06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4034000 tx=0x7f4fe402f000 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.712+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4fe0054420 con 0x7f4ff00b0b20 2023-10-16T13:51:32.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.904+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 --> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ff01610d0 con 0x7f4fdc06af90 2023-10-16T13:51:32.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.933+0000 7f4fd77fe640 1 -- 172.21.15.60:0/3828258273 <== mgr.7918 v2:172.21.15.144:6832/1700115305 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287197 (secure 0 0 0) 0x7f4fe4041070 con 0x7f4fdc06af90 2023-10-16T13:51:32.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:32.941 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T13:51:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.939+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4fdc06af90 msgr2=0x7f4fdc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.939+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4fdc06af90 0x7f4fdc06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4034000 tx=0x7f4fe402f000 comp rx=0 tx=0).stop 2023-10-16T13:51:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.939+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 msgr2=0x7f4ff00b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.939+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 0x7f4ff00b0520 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000c790 tx=0x7f4fe000cc50 comp rx=0 tx=0).stop 2023-10-16T13:51:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 shutdown_connections 2023-10-16T13:51:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f4fdc06af90 0x7f4fdc06d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff015d3f0 0x7f4ff014f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ff00cd0c0 0x7f4ff014cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 --2- 172.21.15.60:0/3828258273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00b0b20 0x7f4ff00b0520 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:32.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 >> 172.21.15.60:0/3828258273 conn(0x7f4ff00cb410 msgr2=0x7f4ff0152ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:32.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 shutdown_connections 2023-10-16T13:51:32.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:32.940+0000 7f4ff54a2640 1 -- 172.21.15.60:0/3828258273 wait complete. 2023-10-16T13:51:32.966 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T13:51:32.856395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":527,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":334,"ondisk_log_size":334,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":398892,"kb_used_data":20340,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754575828,"statfs":{"total":773094113280,"available":772685647872,"internally_reserved":0,"allocated":20828160,"data_stored":10680288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":387501624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002213"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482072+0000","last_change":"2023-10-16T13:51:28.452872+0000","last_active":"2023-10-16T13:51:29.482072+0000","last_peered":"2023-10-16T13:51:29.482072+0000","last_clean":"2023-10-16T13:51:29.482072+0000","last_became_active":"2023-10-16T13:51:28.452098+0000","last_became_peered":"2023-10-16T13:51:28.452098+0000","last_unstale":"2023-10-16T13:51:29.482072+0000","last_undegraded":"2023-10-16T13:51:29.482072+0000","last_fullsized":"2023-10-16T13:51:29.482072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:04.816035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481483+0000","last_change":"2023-10-16T13:51:28.444421+0000","last_active":"2023-10-16T13:51:29.481483+0000","last_peered":"2023-10-16T13:51:29.481483+0000","last_clean":"2023-10-16T13:51:29.481483+0000","last_became_active":"2023-10-16T13:51:28.444148+0000","last_became_peered":"2023-10-16T13:51:28.444148+0000","last_unstale":"2023-10-16T13:51:29.481483+0000","last_undegraded":"2023-10-16T13:51:29.481483+0000","last_fullsized":"2023-10-16T13:51:29.481483+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:17.878183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436160+0000","last_change":"2023-10-16T13:51:28.446670+0000","last_active":"2023-10-16T13:51:29.436160+0000","last_peered":"2023-10-16T13:51:29.436160+0000","last_clean":"2023-10-16T13:51:29.436160+0000","last_became_active":"2023-10-16T13:51:28.446388+0000","last_became_peered":"2023-10-16T13:51:28.446388+0000","last_unstale":"2023-10-16T13:51:29.436160+0000","last_undegraded":"2023-10-16T13:51:29.436160+0000","last_fullsized":"2023-10-16T13:51:29.436160+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.586198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480092+0000","last_change":"2023-10-16T13:51:28.428006+0000","last_active":"2023-10-16T13:51:29.480092+0000","last_peered":"2023-10-16T13:51:29.480092+0000","last_clean":"2023-10-16T13:51:29.480092+0000","last_became_active":"2023-10-16T13:51:28.427720+0000","last_became_peered":"2023-10-16T13:51:28.427720+0000","last_unstale":"2023-10-16T13:51:29.480092+0000","last_undegraded":"2023-10-16T13:51:29.480092+0000","last_fullsized":"2023-10-16T13:51:29.480092+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:08.355314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480632+0000","last_change":"2023-10-16T13:51:28.435341+0000","last_active":"2023-10-16T13:51:29.480632+0000","last_peered":"2023-10-16T13:51:29.480632+0000","last_clean":"2023-10-16T13:51:29.480632+0000","last_became_active":"2023-10-16T13:51:28.434579+0000","last_became_peered":"2023-10-16T13:51:28.434579+0000","last_unstale":"2023-10-16T13:51:29.480632+0000","last_undegraded":"2023-10-16T13:51:29.480632+0000","last_fullsized":"2023-10-16T13:51:29.480632+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.351079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481493+0000","last_change":"2023-10-16T13:51:28.430796+0000","last_active":"2023-10-16T13:51:29.481493+0000","last_peered":"2023-10-16T13:51:29.481493+0000","last_clean":"2023-10-16T13:51:29.481493+0000","last_became_active":"2023-10-16T13:51:28.430426+0000","last_became_peered":"2023-10-16T13:51:28.430426+0000","last_unstale":"2023-10-16T13:51:29.481493+0000","last_undegraded":"2023-10-16T13:51:29.481493+0000","last_fullsized":"2023-10-16T13:51:29.481493+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:59.941460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479964+0000","last_change":"2023-10-16T13:51:28.436398+0000","last_active":"2023-10-16T13:51:29.479964+0000","last_peered":"2023-10-16T13:51:29.479964+0000","last_clean":"2023-10-16T13:51:29.479964+0000","last_became_active":"2023-10-16T13:51:28.436065+0000","last_became_peered":"2023-10-16T13:51:28.436065+0000","last_unstale":"2023-10-16T13:51:29.479964+0000","last_undegraded":"2023-10-16T13:51:29.479964+0000","last_fullsized":"2023-10-16T13:51:29.479964+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:14.831334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479800+0000","last_change":"2023-10-16T13:51:28.439882+0000","last_active":"2023-10-16T13:51:29.479800+0000","last_peered":"2023-10-16T13:51:29.479800+0000","last_clean":"2023-10-16T13:51:29.479800+0000","last_became_active":"2023-10-16T13:51:28.439609+0000","last_became_peered":"2023-10-16T13:51:28.439609+0000","last_unstale":"2023-10-16T13:51:29.479800+0000","last_undegraded":"2023-10-16T13:51:29.479800+0000","last_fullsized":"2023-10-16T13:51:29.479800+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:33.754375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481730+0000","last_change":"2023-10-16T13:51:28.450316+0000","last_active":"2023-10-16T13:51:29.481730+0000","last_peered":"2023-10-16T13:51:29.481730+0000","last_clean":"2023-10-16T13:51:29.481730+0000","last_became_active":"2023-10-16T13:51:28.450014+0000","last_became_peered":"2023-10-16T13:51:28.450014+0000","last_unstale":"2023-10-16T13:51:29.481730+0000","last_undegraded":"2023-10-16T13:51:29.481730+0000","last_fullsized":"2023-10-16T13:51:29.481730+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:42.609197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481614+0000","last_change":"2023-10-16T13:51:28.447959+0000","last_active":"2023-10-16T13:51:29.481614+0000","last_peered":"2023-10-16T13:51:29.481614+0000","last_clean":"2023-10-16T13:51:29.481614+0000","last_became_active":"2023-10-16T13:51:28.447330+0000","last_became_peered":"2023-10-16T13:51:28.447330+0000","last_unstale":"2023-10-16T13:51:29.481614+0000","last_undegraded":"2023-10-16T13:51:29.481614+0000","last_fullsized":"2023-10-16T13:51:29.481614+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:23.126342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481271+0000","last_change":"2023-10-16T13:51:28.447939+0000","last_active":"2023-10-16T13:51:29.481271+0000","last_peered":"2023-10-16T13:51:29.481271+0000","last_clean":"2023-10-16T13:51:29.481271+0000","last_became_active":"2023-10-16T13:51:28.447592+0000","last_became_peered":"2023-10-16T13:51:28.447592+0000","last_unstale":"2023-10-16T13:51:29.481271+0000","last_undegraded":"2023-10-16T13:51:29.481271+0000","last_fullsized":"2023-10-16T13:51:29.481271+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:05.901412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480625+0000","last_change":"2023-10-16T13:51:28.440586+0000","last_active":"2023-10-16T13:51:29.480625+0000","last_peered":"2023-10-16T13:51:29.480625+0000","last_clean":"2023-10-16T13:51:29.480625+0000","last_became_active":"2023-10-16T13:51:28.440306+0000","last_became_peered":"2023-10-16T13:51:28.440306+0000","last_unstale":"2023-10-16T13:51:29.480625+0000","last_undegraded":"2023-10-16T13:51:29.480625+0000","last_fullsized":"2023-10-16T13:51:29.480625+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.029670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436713+0000","last_change":"2023-10-16T13:51:28.433418+0000","last_active":"2023-10-16T13:51:29.436713+0000","last_peered":"2023-10-16T13:51:29.436713+0000","last_clean":"2023-10-16T13:51:29.436713+0000","last_became_active":"2023-10-16T13:51:28.433121+0000","last_became_peered":"2023-10-16T13:51:28.433121+0000","last_unstale":"2023-10-16T13:51:29.436713+0000","last_undegraded":"2023-10-16T13:51:29.436713+0000","last_fullsized":"2023-10-16T13:51:29.436713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:17.955633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.433914+0000","last_change":"2023-10-16T13:51:28.436213+0000","last_active":"2023-10-16T13:51:29.433914+0000","last_peered":"2023-10-16T13:51:29.433914+0000","last_clean":"2023-10-16T13:51:29.433914+0000","last_became_active":"2023-10-16T13:51:28.435912+0000","last_became_peered":"2023-10-16T13:51:28.435912+0000","last_unstale":"2023-10-16T13:51:29.433914+0000","last_undegraded":"2023-10-16T13:51:29.433914+0000","last_fullsized":"2023-10-16T13:51:29.433914+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:42.309272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480402+0000","last_change":"2023-10-16T13:51:28.448474+0000","last_active":"2023-10-16T13:51:29.480402+0000","last_peered":"2023-10-16T13:51:29.480402+0000","last_clean":"2023-10-16T13:51:29.480402+0000","last_became_active":"2023-10-16T13:51:28.445453+0000","last_became_peered":"2023-10-16T13:51:28.445453+0000","last_unstale":"2023-10-16T13:51:29.480402+0000","last_undegraded":"2023-10-16T13:51:29.480402+0000","last_fullsized":"2023-10-16T13:51:29.480402+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:03.330010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479296+0000","last_change":"2023-10-16T13:51:28.436423+0000","last_active":"2023-10-16T13:51:29.479296+0000","last_peered":"2023-10-16T13:51:29.479296+0000","last_clean":"2023-10-16T13:51:29.479296+0000","last_became_active":"2023-10-16T13:51:28.435664+0000","last_became_peered":"2023-10-16T13:51:28.435664+0000","last_unstale":"2023-10-16T13:51:29.479296+0000","last_undegraded":"2023-10-16T13:51:29.479296+0000","last_fullsized":"2023-10-16T13:51:29.479296+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:59.198148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.483156+0000","last_change":"2023-10-16T13:51:28.446567+0000","last_active":"2023-10-16T13:51:29.483156+0000","last_peered":"2023-10-16T13:51:29.483156+0000","last_clean":"2023-10-16T13:51:29.483156+0000","last_became_active":"2023-10-16T13:51:28.446313+0000","last_became_peered":"2023-10-16T13:51:28.446313+0000","last_unstale":"2023-10-16T13:51:29.483156+0000","last_undegraded":"2023-10-16T13:51:29.483156+0000","last_fullsized":"2023-10-16T13:51:29.483156+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:23.802977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432072+0000","last_change":"2023-10-16T13:51:28.432963+0000","last_active":"2023-10-16T13:51:29.432072+0000","last_peered":"2023-10-16T13:51:29.432072+0000","last_clean":"2023-10-16T13:51:29.432072+0000","last_became_active":"2023-10-16T13:51:28.432313+0000","last_became_peered":"2023-10-16T13:51:28.432313+0000","last_unstale":"2023-10-16T13:51:29.432072+0000","last_undegraded":"2023-10-16T13:51:29.432072+0000","last_fullsized":"2023-10-16T13:51:29.432072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:24.394538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482669+0000","last_change":"2023-10-16T13:51:28.442323+0000","last_active":"2023-10-16T13:51:29.482669+0000","last_peered":"2023-10-16T13:51:29.482669+0000","last_clean":"2023-10-16T13:51:29.482669+0000","last_became_active":"2023-10-16T13:51:28.442030+0000","last_became_peered":"2023-10-16T13:51:28.442030+0000","last_unstale":"2023-10-16T13:51:29.482669+0000","last_undegraded":"2023-10-16T13:51:29.482669+0000","last_fullsized":"2023-10-16T13:51:29.482669+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:58.120912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436424+0000","last_change":"2023-10-16T13:51:28.446888+0000","last_active":"2023-10-16T13:51:29.436424+0000","last_peered":"2023-10-16T13:51:29.436424+0000","last_clean":"2023-10-16T13:51:29.436424+0000","last_became_active":"2023-10-16T13:51:28.446063+0000","last_became_peered":"2023-10-16T13:51:28.446063+0000","last_unstale":"2023-10-16T13:51:29.436424+0000","last_undegraded":"2023-10-16T13:51:29.436424+0000","last_fullsized":"2023-10-16T13:51:29.436424+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:05.879940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481080+0000","last_change":"2023-10-16T13:51:28.446345+0000","last_active":"2023-10-16T13:51:29.481080+0000","last_peered":"2023-10-16T13:51:29.481080+0000","last_clean":"2023-10-16T13:51:29.481080+0000","last_became_active":"2023-10-16T13:51:28.446021+0000","last_became_peered":"2023-10-16T13:51:28.446021+0000","last_unstale":"2023-10-16T13:51:29.481080+0000","last_undegraded":"2023-10-16T13:51:29.481080+0000","last_fullsized":"2023-10-16T13:51:29.481080+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:03.200092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481272+0000","last_change":"2023-10-16T13:51:28.428226+0000","last_active":"2023-10-16T13:51:29.481272+0000","last_peered":"2023-10-16T13:51:29.481272+0000","last_clean":"2023-10-16T13:51:29.481272+0000","last_became_active":"2023-10-16T13:51:28.426771+0000","last_became_peered":"2023-10-16T13:51:28.426771+0000","last_unstale":"2023-10-16T13:51:29.481272+0000","last_undegraded":"2023-10-16T13:51:29.481272+0000","last_fullsized":"2023-10-16T13:51:29.481272+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:32.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481894+0000","last_change":"2023-10-16T13:51:28.433877+0000","last_active":"2023-10-16T13:51:29.481894+0000","last_peered":"2023-10-16T13:51:29.481894+0000","last_clean":"2023-10-16T13:51:29.481894+0000","last_became_active":"2023-10-16T13:51:28.433573+0000","last_became_peered":"2023-10-16T13:51:28.433573+0000","last_unstale":"2023-10-16T13:51:29.481894+0000","last_undegraded":"2023-10-16T13:51:29.481894+0000","last_fullsized":"2023-10-16T13:51:29.481894+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:46.469338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479980+0000","last_change":"2023-10-16T13:51:28.439612+0000","last_active":"2023-10-16T13:51:29.479980+0000","last_peered":"2023-10-16T13:51:29.479980+0000","last_clean":"2023-10-16T13:51:29.479980+0000","last_became_active":"2023-10-16T13:51:28.438873+0000","last_became_peered":"2023-10-16T13:51:28.438873+0000","last_unstale":"2023-10-16T13:51:29.479980+0000","last_undegraded":"2023-10-16T13:51:29.479980+0000","last_fullsized":"2023-10-16T13:51:29.479980+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:08.800708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437289+0000","last_change":"2023-10-16T13:51:28.446332+0000","last_active":"2023-10-16T13:51:29.437289+0000","last_peered":"2023-10-16T13:51:29.437289+0000","last_clean":"2023-10-16T13:51:29.437289+0000","last_became_active":"2023-10-16T13:51:28.446000+0000","last_became_peered":"2023-10-16T13:51:28.446000+0000","last_unstale":"2023-10-16T13:51:29.437289+0000","last_undegraded":"2023-10-16T13:51:29.437289+0000","last_fullsized":"2023-10-16T13:51:29.437289+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:25.250592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482731+0000","last_change":"2023-10-16T13:51:28.443852+0000","last_active":"2023-10-16T13:51:29.482731+0000","last_peered":"2023-10-16T13:51:29.482731+0000","last_clean":"2023-10-16T13:51:29.482731+0000","last_became_active":"2023-10-16T13:51:28.443536+0000","last_became_peered":"2023-10-16T13:51:28.443536+0000","last_unstale":"2023-10-16T13:51:29.482731+0000","last_undegraded":"2023-10-16T13:51:29.482731+0000","last_fullsized":"2023-10-16T13:51:29.482731+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:27.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436531+0000","last_change":"2023-10-16T13:51:28.440374+0000","last_active":"2023-10-16T13:51:29.436531+0000","last_peered":"2023-10-16T13:51:29.436531+0000","last_clean":"2023-10-16T13:51:29.436531+0000","last_became_active":"2023-10-16T13:51:28.440093+0000","last_became_peered":"2023-10-16T13:51:28.440093+0000","last_unstale":"2023-10-16T13:51:29.436531+0000","last_undegraded":"2023-10-16T13:51:29.436531+0000","last_fullsized":"2023-10-16T13:51:29.436531+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:15.210378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480440+0000","last_change":"2023-10-16T13:51:28.445966+0000","last_active":"2023-10-16T13:51:29.480440+0000","last_peered":"2023-10-16T13:51:29.480440+0000","last_clean":"2023-10-16T13:51:29.480440+0000","last_became_active":"2023-10-16T13:51:28.445676+0000","last_became_peered":"2023-10-16T13:51:28.445676+0000","last_unstale":"2023-10-16T13:51:29.480440+0000","last_undegraded":"2023-10-16T13:51:29.480440+0000","last_fullsized":"2023-10-16T13:51:29.480440+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:03.049286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436437+0000","last_change":"2023-10-16T13:51:28.432127+0000","last_active":"2023-10-16T13:51:29.436437+0000","last_peered":"2023-10-16T13:51:29.436437+0000","last_clean":"2023-10-16T13:51:29.436437+0000","last_became_active":"2023-10-16T13:51:28.431848+0000","last_became_peered":"2023-10-16T13:51:28.431848+0000","last_unstale":"2023-10-16T13:51:29.436437+0000","last_undegraded":"2023-10-16T13:51:29.436437+0000","last_fullsized":"2023-10-16T13:51:29.436437+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:42.546116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434371+0000","last_change":"2023-10-16T13:51:28.449840+0000","last_active":"2023-10-16T13:51:29.434371+0000","last_peered":"2023-10-16T13:51:29.434371+0000","last_clean":"2023-10-16T13:51:29.434371+0000","last_became_active":"2023-10-16T13:51:28.449501+0000","last_became_peered":"2023-10-16T13:51:28.449501+0000","last_unstale":"2023-10-16T13:51:29.434371+0000","last_undegraded":"2023-10-16T13:51:29.434371+0000","last_fullsized":"2023-10-16T13:51:29.434371+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.413170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434086+0000","last_change":"2023-10-16T13:51:28.447252+0000","last_active":"2023-10-16T13:51:29.434086+0000","last_peered":"2023-10-16T13:51:29.434086+0000","last_clean":"2023-10-16T13:51:29.434086+0000","last_became_active":"2023-10-16T13:51:28.446968+0000","last_became_peered":"2023-10-16T13:51:28.446968+0000","last_unstale":"2023-10-16T13:51:29.434086+0000","last_undegraded":"2023-10-16T13:51:29.434086+0000","last_fullsized":"2023-10-16T13:51:29.434086+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.486179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431212+0000","last_change":"2023-10-16T13:51:28.432775+0000","last_active":"2023-10-16T13:51:29.431212+0000","last_peered":"2023-10-16T13:51:29.431212+0000","last_clean":"2023-10-16T13:51:29.431212+0000","last_became_active":"2023-10-16T13:51:28.427111+0000","last_became_peered":"2023-10-16T13:51:28.427111+0000","last_unstale":"2023-10-16T13:51:29.431212+0000","last_undegraded":"2023-10-16T13:51:29.431212+0000","last_fullsized":"2023-10-16T13:51:29.431212+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:44.722454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482968+0000","last_change":"2023-10-16T13:51:28.439803+0000","last_active":"2023-10-16T13:51:29.482968+0000","last_peered":"2023-10-16T13:51:29.482968+0000","last_clean":"2023-10-16T13:51:29.482968+0000","last_became_active":"2023-10-16T13:51:28.439476+0000","last_became_peered":"2023-10-16T13:51:28.439476+0000","last_unstale":"2023-10-16T13:51:29.482968+0000","last_undegraded":"2023-10-16T13:51:29.482968+0000","last_fullsized":"2023-10-16T13:51:29.482968+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:50.161706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"196'4","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482217+0000","last_change":"2023-10-16T13:51:27.439181+0000","last_active":"2023-10-16T13:51:29.482217+0000","last_peered":"2023-10-16T13:51:29.482217+0000","last_clean":"2023-10-16T13:51:29.482217+0000","last_became_active":"2023-10-16T13:51:27.438891+0000","last_became_peered":"2023-10-16T13:51:27.438891+0000","last_unstale":"2023-10-16T13:51:29.482217+0000","last_undegraded":"2023-10-16T13:51:29.482217+0000","last_fullsized":"2023-10-16T13:51:29.482217+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:44.978387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436349+0000","last_change":"2023-10-16T13:51:28.441903+0000","last_active":"2023-10-16T13:51:29.436349+0000","last_peered":"2023-10-16T13:51:29.436349+0000","last_clean":"2023-10-16T13:51:29.436349+0000","last_became_active":"2023-10-16T13:51:28.438658+0000","last_became_peered":"2023-10-16T13:51:28.438658+0000","last_unstale":"2023-10-16T13:51:29.436349+0000","last_undegraded":"2023-10-16T13:51:29.436349+0000","last_fullsized":"2023-10-16T13:51:29.436349+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:52.516835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.435251+0000","last_change":"2023-10-16T13:51:27.437751+0000","last_active":"2023-10-16T13:51:29.435251+0000","last_peered":"2023-10-16T13:51:29.435251+0000","last_clean":"2023-10-16T13:51:29.435251+0000","last_became_active":"2023-10-16T13:51:27.436383+0000","last_became_peered":"2023-10-16T13:51:27.436383+0000","last_unstale":"2023-10-16T13:51:29.435251+0000","last_undegraded":"2023-10-16T13:51:29.435251+0000","last_fullsized":"2023-10-16T13:51:29.435251+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:58.928816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481884+0000","last_change":"2023-10-16T13:51:28.442759+0000","last_active":"2023-10-16T13:51:29.481884+0000","last_peered":"2023-10-16T13:51:29.481884+0000","last_clean":"2023-10-16T13:51:29.481884+0000","last_became_active":"2023-10-16T13:51:28.440811+0000","last_became_peered":"2023-10-16T13:51:28.440811+0000","last_unstale":"2023-10-16T13:51:29.481884+0000","last_undegraded":"2023-10-16T13:51:29.481884+0000","last_fullsized":"2023-10-16T13:51:29.481884+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:11.349163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480320+0000","last_change":"2023-10-16T13:51:27.437678+0000","last_active":"2023-10-16T13:51:29.480320+0000","last_peered":"2023-10-16T13:51:29.480320+0000","last_clean":"2023-10-16T13:51:29.480320+0000","last_became_active":"2023-10-16T13:51:27.437069+0000","last_became_peered":"2023-10-16T13:51:27.437069+0000","last_unstale":"2023-10-16T13:51:29.480320+0000","last_undegraded":"2023-10-16T13:51:29.480320+0000","last_fullsized":"2023-10-16T13:51:29.480320+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:45.302196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436939+0000","last_change":"2023-10-16T13:51:28.442692+0000","last_active":"2023-10-16T13:51:29.436939+0000","last_peered":"2023-10-16T13:51:29.436939+0000","last_clean":"2023-10-16T13:51:29.436939+0000","last_became_active":"2023-10-16T13:51:28.442335+0000","last_became_peered":"2023-10-16T13:51:28.442335+0000","last_unstale":"2023-10-16T13:51:29.436939+0000","last_undegraded":"2023-10-16T13:51:29.436939+0000","last_fullsized":"2023-10-16T13:51:29.436939+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:31.461167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480687+0000","last_change":"2023-10-16T13:51:27.432100+0000","last_active":"2023-10-16T13:51:29.480687+0000","last_peered":"2023-10-16T13:51:29.480687+0000","last_clean":"2023-10-16T13:51:29.480687+0000","last_became_active":"2023-10-16T13:51:27.431837+0000","last_became_peered":"2023-10-16T13:51:27.431837+0000","last_unstale":"2023-10-16T13:51:29.480687+0000","last_undegraded":"2023-10-16T13:51:29.480687+0000","last_fullsized":"2023-10-16T13:51:29.480687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:23.540177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479799+0000","last_change":"2023-10-16T13:51:28.428287+0000","last_active":"2023-10-16T13:51:29.479799+0000","last_peered":"2023-10-16T13:51:29.479799+0000","last_clean":"2023-10-16T13:51:29.479799+0000","last_became_active":"2023-10-16T13:51:28.427992+0000","last_became_peered":"2023-10-16T13:51:28.427992+0000","last_unstale":"2023-10-16T13:51:29.479799+0000","last_undegraded":"2023-10-16T13:51:29.479799+0000","last_fullsized":"2023-10-16T13:51:29.479799+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:35.985411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436923+0000","last_change":"2023-10-16T13:51:27.428507+0000","last_active":"2023-10-16T13:51:29.436923+0000","last_peered":"2023-10-16T13:51:29.436923+0000","last_clean":"2023-10-16T13:51:29.436923+0000","last_became_active":"2023-10-16T13:51:27.428235+0000","last_became_peered":"2023-10-16T13:51:27.428235+0000","last_unstale":"2023-10-16T13:51:29.436923+0000","last_undegraded":"2023-10-16T13:51:29.436923+0000","last_fullsized":"2023-10-16T13:51:29.436923+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:20.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480697+0000","last_change":"2023-10-16T13:51:28.437640+0000","last_active":"2023-10-16T13:51:29.480697+0000","last_peered":"2023-10-16T13:51:29.480697+0000","last_clean":"2023-10-16T13:51:29.480697+0000","last_became_active":"2023-10-16T13:51:28.437352+0000","last_became_peered":"2023-10-16T13:51:28.437352+0000","last_unstale":"2023-10-16T13:51:29.480697+0000","last_undegraded":"2023-10-16T13:51:29.480697+0000","last_fullsized":"2023-10-16T13:51:29.480697+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:58.357330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481014+0000","last_change":"2023-10-16T13:51:27.428678+0000","last_active":"2023-10-16T13:51:29.481014+0000","last_peered":"2023-10-16T13:51:29.481014+0000","last_clean":"2023-10-16T13:51:29.481014+0000","last_became_active":"2023-10-16T13:51:27.428404+0000","last_became_peered":"2023-10-16T13:51:27.428404+0000","last_unstale":"2023-10-16T13:51:29.481014+0000","last_undegraded":"2023-10-16T13:51:29.481014+0000","last_fullsized":"2023-10-16T13:51:29.481014+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:51.504190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481892+0000","last_change":"2023-10-16T13:51:28.435960+0000","last_active":"2023-10-16T13:51:29.481892+0000","last_peered":"2023-10-16T13:51:29.481892+0000","last_clean":"2023-10-16T13:51:29.481892+0000","last_became_active":"2023-10-16T13:51:28.435670+0000","last_became_peered":"2023-10-16T13:51:28.435670+0000","last_unstale":"2023-10-16T13:51:29.481892+0000","last_undegraded":"2023-10-16T13:51:29.481892+0000","last_fullsized":"2023-10-16T13:51:29.481892+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:27.595646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479941+0000","last_change":"2023-10-16T13:51:27.431152+0000","last_active":"2023-10-16T13:51:29.479941+0000","last_peered":"2023-10-16T13:51:29.479941+0000","last_clean":"2023-10-16T13:51:29.479941+0000","last_became_active":"2023-10-16T13:51:27.430873+0000","last_became_peered":"2023-10-16T13:51:27.430873+0000","last_unstale":"2023-10-16T13:51:29.479941+0000","last_undegraded":"2023-10-16T13:51:29.479941+0000","last_fullsized":"2023-10-16T13:51:29.479941+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:24.148258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482140+0000","last_change":"2023-10-16T13:51:28.432755+0000","last_active":"2023-10-16T13:51:29.482140+0000","last_peered":"2023-10-16T13:51:29.482140+0000","last_clean":"2023-10-16T13:51:29.482140+0000","last_became_active":"2023-10-16T13:51:28.430015+0000","last_became_peered":"2023-10-16T13:51:28.430015+0000","last_unstale":"2023-10-16T13:51:29.482140+0000","last_undegraded":"2023-10-16T13:51:29.482140+0000","last_fullsized":"2023-10-16T13:51:29.482140+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.413170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481255+0000","last_change":"2023-10-16T13:51:27.427823+0000","last_active":"2023-10-16T13:51:29.481255+0000","last_peered":"2023-10-16T13:51:29.481255+0000","last_clean":"2023-10-16T13:51:29.481255+0000","last_became_active":"2023-10-16T13:51:27.427521+0000","last_became_peered":"2023-10-16T13:51:27.427521+0000","last_unstale":"2023-10-16T13:51:29.481255+0000","last_undegraded":"2023-10-16T13:51:29.481255+0000","last_fullsized":"2023-10-16T13:51:29.481255+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:56.092448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432357+0000","last_change":"2023-10-16T13:51:28.434499+0000","last_active":"2023-10-16T13:51:29.432357+0000","last_peered":"2023-10-16T13:51:29.432357+0000","last_clean":"2023-10-16T13:51:29.432357+0000","last_became_active":"2023-10-16T13:51:28.434188+0000","last_became_peered":"2023-10-16T13:51:28.434188+0000","last_unstale":"2023-10-16T13:51:29.432357+0000","last_undegraded":"2023-10-16T13:51:29.432357+0000","last_fullsized":"2023-10-16T13:51:29.432357+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:59.411175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.483083+0000","last_change":"2023-10-16T13:51:27.430504+0000","last_active":"2023-10-16T13:51:29.483083+0000","last_peered":"2023-10-16T13:51:29.483083+0000","last_clean":"2023-10-16T13:51:29.483083+0000","last_became_active":"2023-10-16T13:51:27.430238+0000","last_became_peered":"2023-10-16T13:51:27.430238+0000","last_unstale":"2023-10-16T13:51:29.483083+0000","last_undegraded":"2023-10-16T13:51:29.483083+0000","last_fullsized":"2023-10-16T13:51:29.483083+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:30.448522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432124+0000","last_change":"2023-10-16T13:51:28.433923+0000","last_active":"2023-10-16T13:51:29.432124+0000","last_peered":"2023-10-16T13:51:29.432124+0000","last_clean":"2023-10-16T13:51:29.432124+0000","last_became_active":"2023-10-16T13:51:28.433669+0000","last_became_peered":"2023-10-16T13:51:28.433669+0000","last_unstale":"2023-10-16T13:51:29.432124+0000","last_undegraded":"2023-10-16T13:51:29.432124+0000","last_fullsized":"2023-10-16T13:51:29.432124+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:30.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"196'3","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482610+0000","last_change":"2023-10-16T13:51:27.432160+0000","last_active":"2023-10-16T13:51:29.482610+0000","last_peered":"2023-10-16T13:51:29.482610+0000","last_clean":"2023-10-16T13:51:29.482610+0000","last_became_active":"2023-10-16T13:51:27.431870+0000","last_became_peered":"2023-10-16T13:51:27.431870+0000","last_unstale":"2023-10-16T13:51:29.482610+0000","last_undegraded":"2023-10-16T13:51:29.482610+0000","last_fullsized":"2023-10-16T13:51:29.482610+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:12.790107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437648+0000","last_change":"2023-10-16T13:51:28.441736+0000","last_active":"2023-10-16T13:51:29.437648+0000","last_peered":"2023-10-16T13:51:29.437648+0000","last_clean":"2023-10-16T13:51:29.437648+0000","last_became_active":"2023-10-16T13:51:28.441304+0000","last_became_peered":"2023-10-16T13:51:28.441304+0000","last_unstale":"2023-10-16T13:51:29.437648+0000","last_undegraded":"2023-10-16T13:51:29.437648+0000","last_fullsized":"2023-10-16T13:51:29.437648+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:58.120912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"196'6","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437637+0000","last_change":"2023-10-16T13:51:27.434373+0000","last_active":"2023-10-16T13:51:29.437637+0000","last_peered":"2023-10-16T13:51:29.437637+0000","last_clean":"2023-10-16T13:51:29.437637+0000","last_became_active":"2023-10-16T13:51:27.433377+0000","last_became_peered":"2023-10-16T13:51:27.433377+0000","last_unstale":"2023-10-16T13:51:29.437637+0000","last_undegraded":"2023-10-16T13:51:29.437637+0000","last_fullsized":"2023-10-16T13:51:29.437637+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:10.336518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431510+0000","last_change":"2023-10-16T13:51:28.437700+0000","last_active":"2023-10-16T13:51:29.431510+0000","last_peered":"2023-10-16T13:51:29.431510+0000","last_clean":"2023-10-16T13:51:29.431510+0000","last_became_active":"2023-10-16T13:51:28.437230+0000","last_became_peered":"2023-10-16T13:51:28.437230+0000","last_unstale":"2023-10-16T13:51:29.431510+0000","last_undegraded":"2023-10-16T13:51:29.431510+0000","last_fullsized":"2023-10-16T13:51:29.431510+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:50.438320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480953+0000","last_change":"2023-10-16T13:51:27.424287+0000","last_active":"2023-10-16T13:51:29.480953+0000","last_peered":"2023-10-16T13:51:29.480953+0000","last_clean":"2023-10-16T13:51:29.480953+0000","last_became_active":"2023-10-16T13:51:27.423990+0000","last_became_peered":"2023-10-16T13:51:27.423990+0000","last_unstale":"2023-10-16T13:51:29.480953+0000","last_undegraded":"2023-10-16T13:51:29.480953+0000","last_fullsized":"2023-10-16T13:51:29.480953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:12.034064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482421+0000","last_change":"2023-10-16T13:51:28.444996+0000","last_active":"2023-10-16T13:51:29.482421+0000","last_peered":"2023-10-16T13:51:29.482421+0000","last_clean":"2023-10-16T13:51:29.482421+0000","last_became_active":"2023-10-16T13:51:28.444735+0000","last_became_peered":"2023-10-16T13:51:28.444735+0000","last_unstale":"2023-10-16T13:51:29.482421+0000","last_undegraded":"2023-10-16T13:51:29.482421+0000","last_fullsized":"2023-10-16T13:51:29.482421+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:00.612037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431840+0000","last_change":"2023-10-16T13:51:27.426783+0000","last_active":"2023-10-16T13:51:29.431840+0000","last_peered":"2023-10-16T13:51:29.431840+0000","last_clean":"2023-10-16T13:51:29.431840+0000","last_became_active":"2023-10-16T13:51:27.426359+0000","last_became_peered":"2023-10-16T13:51:27.426359+0000","last_unstale":"2023-10-16T13:51:29.431840+0000","last_undegraded":"2023-10-16T13:51:29.431840+0000","last_fullsized":"2023-10-16T13:51:29.431840+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:38.237707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480508+0000","last_change":"2023-10-16T13:51:28.442605+0000","last_active":"2023-10-16T13:51:29.480508+0000","last_peered":"2023-10-16T13:51:29.480508+0000","last_clean":"2023-10-16T13:51:29.480508+0000","last_became_active":"2023-10-16T13:51:28.442339+0000","last_became_peered":"2023-10-16T13:51:28.442339+0000","last_unstale":"2023-10-16T13:51:29.480508+0000","last_undegraded":"2023-10-16T13:51:29.480508+0000","last_fullsized":"2023-10-16T13:51:29.480508+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:23.517975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479502+0000","last_change":"2023-10-16T13:51:27.436887+0000","last_active":"2023-10-16T13:51:29.479502+0000","last_peered":"2023-10-16T13:51:29.479502+0000","last_clean":"2023-10-16T13:51:29.479502+0000","last_became_active":"2023-10-16T13:51:27.436546+0000","last_became_peered":"2023-10-16T13:51:27.436546+0000","last_unstale":"2023-10-16T13:51:29.479502+0000","last_undegraded":"2023-10-16T13:51:29.479502+0000","last_fullsized":"2023-10-16T13:51:29.479502+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:02.187448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434839+0000","last_change":"2023-10-16T13:51:28.432571+0000","last_active":"2023-10-16T13:51:29.434839+0000","last_peered":"2023-10-16T13:51:29.434839+0000","last_clean":"2023-10-16T13:51:29.434839+0000","last_became_active":"2023-10-16T13:51:28.431454+0000","last_became_peered":"2023-10-16T13:51:28.431454+0000","last_unstale":"2023-10-16T13:51:29.434839+0000","last_undegraded":"2023-10-16T13:51:29.434839+0000","last_fullsized":"2023-10-16T13:51:29.434839+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:35.908243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"196'8","reported_seq":20,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.433910+0000","last_change":"2023-10-16T13:51:27.435137+0000","last_active":"2023-10-16T13:51:29.433910+0000","last_peered":"2023-10-16T13:51:29.433910+0000","last_clean":"2023-10-16T13:51:29.433910+0000","last_became_active":"2023-10-16T13:51:27.434797+0000","last_became_peered":"2023-10-16T13:51:27.434797+0000","last_unstale":"2023-10-16T13:51:29.433910+0000","last_undegraded":"2023-10-16T13:51:29.433910+0000","last_fullsized":"2023-10-16T13:51:29.433910+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:12.521516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480183+0000","last_change":"2023-10-16T13:51:28.438705+0000","last_active":"2023-10-16T13:51:29.480183+0000","last_peered":"2023-10-16T13:51:29.480183+0000","last_clean":"2023-10-16T13:51:29.480183+0000","last_became_active":"2023-10-16T13:51:28.432126+0000","last_became_peered":"2023-10-16T13:51:28.432126+0000","last_unstale":"2023-10-16T13:51:29.480183+0000","last_undegraded":"2023-10-16T13:51:29.480183+0000","last_fullsized":"2023-10-16T13:51:29.480183+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:10.530427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437372+0000","last_change":"2023-10-16T13:51:27.437255+0000","last_active":"2023-10-16T13:51:29.437372+0000","last_peered":"2023-10-16T13:51:29.437372+0000","last_clean":"2023-10-16T13:51:29.437372+0000","last_became_active":"2023-10-16T13:51:27.436838+0000","last_became_peered":"2023-10-16T13:51:27.436838+0000","last_unstale":"2023-10-16T13:51:29.437372+0000","last_undegraded":"2023-10-16T13:51:29.437372+0000","last_fullsized":"2023-10-16T13:51:29.437372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:32.418327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432665+0000","last_change":"2023-10-16T13:51:28.443763+0000","last_active":"2023-10-16T13:51:29.432665+0000","last_peered":"2023-10-16T13:51:29.432665+0000","last_clean":"2023-10-16T13:51:29.432665+0000","last_became_active":"2023-10-16T13:51:28.443519+0000","last_became_peered":"2023-10-16T13:51:28.443519+0000","last_unstale":"2023-10-16T13:51:29.432665+0000","last_undegraded":"2023-10-16T13:51:29.432665+0000","last_fullsized":"2023-10-16T13:51:29.432665+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:27.937378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436120+0000","last_change":"2023-10-16T13:51:27.435416+0000","last_active":"2023-10-16T13:51:29.436120+0000","last_peered":"2023-10-16T13:51:29.436120+0000","last_clean":"2023-10-16T13:51:29.436120+0000","last_became_active":"2023-10-16T13:51:27.435109+0000","last_became_peered":"2023-10-16T13:51:27.435109+0000","last_unstale":"2023-10-16T13:51:29.436120+0000","last_undegraded":"2023-10-16T13:51:29.436120+0000","last_fullsized":"2023-10-16T13:51:29.436120+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:08.005543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480698+0000","last_change":"2023-10-16T13:51:28.428089+0000","last_active":"2023-10-16T13:51:29.480698+0000","last_peered":"2023-10-16T13:51:29.480698+0000","last_clean":"2023-10-16T13:51:29.480698+0000","last_became_active":"2023-10-16T13:51:28.427825+0000","last_became_peered":"2023-10-16T13:51:28.427825+0000","last_unstale":"2023-10-16T13:51:29.480698+0000","last_undegraded":"2023-10-16T13:51:29.480698+0000","last_fullsized":"2023-10-16T13:51:29.480698+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:53.621443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481811+0000","last_change":"2023-10-16T13:51:27.440583+0000","last_active":"2023-10-16T13:51:29.481811+0000","last_peered":"2023-10-16T13:51:29.481811+0000","last_clean":"2023-10-16T13:51:29.481811+0000","last_became_active":"2023-10-16T13:51:27.440217+0000","last_became_peered":"2023-10-16T13:51:27.440217+0000","last_unstale":"2023-10-16T13:51:29.481811+0000","last_undegraded":"2023-10-16T13:51:29.481811+0000","last_fullsized":"2023-10-16T13:51:29.481811+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:40.748567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480026+0000","last_change":"2023-10-16T13:51:28.445885+0000","last_active":"2023-10-16T13:51:29.480026+0000","last_peered":"2023-10-16T13:51:29.480026+0000","last_clean":"2023-10-16T13:51:29.480026+0000","last_became_active":"2023-10-16T13:51:28.445639+0000","last_became_peered":"2023-10-16T13:51:28.445639+0000","last_unstale":"2023-10-16T13:51:29.480026+0000","last_undegraded":"2023-10-16T13:51:29.480026+0000","last_fullsized":"2023-10-16T13:51:29.480026+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:07.608491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481195+0000","last_change":"2023-10-16T13:51:27.436822+0000","last_active":"2023-10-16T13:51:29.481195+0000","last_peered":"2023-10-16T13:51:29.481195+0000","last_clean":"2023-10-16T13:51:29.481195+0000","last_became_active":"2023-10-16T13:51:27.436263+0000","last_became_peered":"2023-10-16T13:51:27.436263+0000","last_unstale":"2023-10-16T13:51:29.481195+0000","last_undegraded":"2023-10-16T13:51:29.481195+0000","last_fullsized":"2023-10-16T13:51:29.481195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:05.986789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434930+0000","last_change":"2023-10-16T13:51:28.438253+0000","last_active":"2023-10-16T13:51:29.434930+0000","last_peered":"2023-10-16T13:51:29.434930+0000","last_clean":"2023-10-16T13:51:29.434930+0000","last_became_active":"2023-10-16T13:51:28.437932+0000","last_became_peered":"2023-10-16T13:51:28.437932+0000","last_unstale":"2023-10-16T13:51:29.434930+0000","last_undegraded":"2023-10-16T13:51:29.434930+0000","last_fullsized":"2023-10-16T13:51:29.434930+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:05.621437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481195+0000","last_change":"2023-10-16T13:51:27.434212+0000","last_active":"2023-10-16T13:51:29.481195+0000","last_peered":"2023-10-16T13:51:29.481195+0000","last_clean":"2023-10-16T13:51:29.481195+0000","last_became_active":"2023-10-16T13:51:27.433935+0000","last_became_peered":"2023-10-16T13:51:27.433935+0000","last_unstale":"2023-10-16T13:51:29.481195+0000","last_undegraded":"2023-10-16T13:51:29.481195+0000","last_fullsized":"2023-10-16T13:51:29.481195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:31.315080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479964+0000","last_change":"2023-10-16T13:51:28.436205+0000","last_active":"2023-10-16T13:51:29.479964+0000","last_peered":"2023-10-16T13:51:29.479964+0000","last_clean":"2023-10-16T13:51:29.479964+0000","last_became_active":"2023-10-16T13:51:28.433734+0000","last_became_peered":"2023-10-16T13:51:28.433734+0000","last_unstale":"2023-10-16T13:51:29.479964+0000","last_undegraded":"2023-10-16T13:51:29.479964+0000","last_fullsized":"2023-10-16T13:51:29.479964+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:27.196301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480871+0000","last_change":"2023-10-16T13:51:27.435396+0000","last_active":"2023-10-16T13:51:29.480871+0000","last_peered":"2023-10-16T13:51:29.480871+0000","last_clean":"2023-10-16T13:51:29.480871+0000","last_became_active":"2023-10-16T13:51:27.434893+0000","last_became_peered":"2023-10-16T13:51:27.434893+0000","last_unstale":"2023-10-16T13:51:29.480871+0000","last_undegraded":"2023-10-16T13:51:29.480871+0000","last_fullsized":"2023-10-16T13:51:29.480871+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:01.583417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480273+0000","last_change":"2023-10-16T13:51:28.443268+0000","last_active":"2023-10-16T13:51:29.480273+0000","last_peered":"2023-10-16T13:51:29.480273+0000","last_clean":"2023-10-16T13:51:29.480273+0000","last_became_active":"2023-10-16T13:51:28.442911+0000","last_became_peered":"2023-10-16T13:51:28.442911+0000","last_unstale":"2023-10-16T13:51:29.480273+0000","last_undegraded":"2023-10-16T13:51:29.480273+0000","last_fullsized":"2023-10-16T13:51:29.480273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:37.097378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479590+0000","last_change":"2023-10-16T13:51:27.423627+0000","last_active":"2023-10-16T13:51:29.479590+0000","last_peered":"2023-10-16T13:51:29.479590+0000","last_clean":"2023-10-16T13:51:29.479590+0000","last_became_active":"2023-10-16T13:51:27.422588+0000","last_became_peered":"2023-10-16T13:51:27.422588+0000","last_unstale":"2023-10-16T13:51:29.479590+0000","last_undegraded":"2023-10-16T13:51:29.479590+0000","last_fullsized":"2023-10-16T13:51:29.479590+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:10.336518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482424+0000","last_change":"2023-10-16T13:51:28.435886+0000","last_active":"2023-10-16T13:51:29.482424+0000","last_peered":"2023-10-16T13:51:29.482424+0000","last_clean":"2023-10-16T13:51:29.482424+0000","last_became_active":"2023-10-16T13:51:28.434528+0000","last_became_peered":"2023-10-16T13:51:28.434528+0000","last_unstale":"2023-10-16T13:51:29.482424+0000","last_undegraded":"2023-10-16T13:51:29.482424+0000","last_fullsized":"2023-10-16T13:51:29.482424+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:52.516835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480749+0000","last_change":"2023-10-16T13:51:27.438230+0000","last_active":"2023-10-16T13:51:29.480749+0000","last_peered":"2023-10-16T13:51:29.480749+0000","last_clean":"2023-10-16T13:51:29.480749+0000","last_became_active":"2023-10-16T13:51:27.437908+0000","last_became_peered":"2023-10-16T13:51:27.437908+0000","last_unstale":"2023-10-16T13:51:29.480749+0000","last_undegraded":"2023-10-16T13:51:29.480749+0000","last_fullsized":"2023-10-16T13:51:29.480749+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:46.259204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432214+0000","last_change":"2023-10-16T13:51:28.441896+0000","last_active":"2023-10-16T13:51:29.432214+0000","last_peered":"2023-10-16T13:51:29.432214+0000","last_clean":"2023-10-16T13:51:29.432214+0000","last_became_active":"2023-10-16T13:51:28.441622+0000","last_became_peered":"2023-10-16T13:51:28.441622+0000","last_unstale":"2023-10-16T13:51:29.432214+0000","last_undegraded":"2023-10-16T13:51:29.432214+0000","last_fullsized":"2023-10-16T13:51:29.432214+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:39.322635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432604+0000","last_change":"2023-10-16T13:51:27.437461+0000","last_active":"2023-10-16T13:51:29.432604+0000","last_peered":"2023-10-16T13:51:29.432604+0000","last_clean":"2023-10-16T13:51:29.432604+0000","last_became_active":"2023-10-16T13:51:27.437038+0000","last_became_peered":"2023-10-16T13:51:27.437038+0000","last_unstale":"2023-10-16T13:51:29.432604+0000","last_undegraded":"2023-10-16T13:51:29.432604+0000","last_fullsized":"2023-10-16T13:51:29.432604+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:11.972312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481476+0000","last_change":"2023-10-16T13:51:28.434053+0000","last_active":"2023-10-16T13:51:29.481476+0000","last_peered":"2023-10-16T13:51:29.481476+0000","last_clean":"2023-10-16T13:51:29.481476+0000","last_became_active":"2023-10-16T13:51:28.433767+0000","last_became_peered":"2023-10-16T13:51:28.433767+0000","last_unstale":"2023-10-16T13:51:29.481476+0000","last_undegraded":"2023-10-16T13:51:29.481476+0000","last_fullsized":"2023-10-16T13:51:29.481476+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:21.255420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436388+0000","last_change":"2023-10-16T13:51:27.425258+0000","last_active":"2023-10-16T13:51:29.436388+0000","last_peered":"2023-10-16T13:51:29.436388+0000","last_clean":"2023-10-16T13:51:29.436388+0000","last_became_active":"2023-10-16T13:51:27.424953+0000","last_became_peered":"2023-10-16T13:51:27.424953+0000","last_unstale":"2023-10-16T13:51:29.436388+0000","last_undegraded":"2023-10-16T13:51:29.436388+0000","last_fullsized":"2023-10-16T13:51:29.436388+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:29.760617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482121+0000","last_change":"2023-10-16T13:51:28.435763+0000","last_active":"2023-10-16T13:51:29.482121+0000","last_peered":"2023-10-16T13:51:29.482121+0000","last_clean":"2023-10-16T13:51:29.482121+0000","last_became_active":"2023-10-16T13:51:28.435009+0000","last_became_peered":"2023-10-16T13:51:28.435009+0000","last_unstale":"2023-10-16T13:51:29.482121+0000","last_undegraded":"2023-10-16T13:51:29.482121+0000","last_fullsized":"2023-10-16T13:51:29.482121+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.486179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481696+0000","last_change":"2023-10-16T13:51:27.436782+0000","last_active":"2023-10-16T13:51:29.481696+0000","last_peered":"2023-10-16T13:51:29.481696+0000","last_clean":"2023-10-16T13:51:29.481696+0000","last_became_active":"2023-10-16T13:51:27.436480+0000","last_became_peered":"2023-10-16T13:51:27.436480+0000","last_unstale":"2023-10-16T13:51:29.481696+0000","last_undegraded":"2023-10-16T13:51:29.481696+0000","last_fullsized":"2023-10-16T13:51:29.481696+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:51.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431668+0000","last_change":"2023-10-16T13:51:28.435367+0000","last_active":"2023-10-16T13:51:29.431668+0000","last_peered":"2023-10-16T13:51:29.431668+0000","last_clean":"2023-10-16T13:51:29.431668+0000","last_became_active":"2023-10-16T13:51:28.435087+0000","last_became_peered":"2023-10-16T13:51:28.435087+0000","last_unstale":"2023-10-16T13:51:29.431668+0000","last_undegraded":"2023-10-16T13:51:29.431668+0000","last_fullsized":"2023-10-16T13:51:29.431668+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:50.781893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432375+0000","last_change":"2023-10-16T13:51:27.431128+0000","last_active":"2023-10-16T13:51:29.432375+0000","last_peered":"2023-10-16T13:51:29.432375+0000","last_clean":"2023-10-16T13:51:29.432375+0000","last_became_active":"2023-10-16T13:51:27.430856+0000","last_became_peered":"2023-10-16T13:51:27.430856+0000","last_unstale":"2023-10-16T13:51:29.432375+0000","last_undegraded":"2023-10-16T13:51:29.432375+0000","last_fullsized":"2023-10-16T13:51:29.432375+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:01.132629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431713+0000","last_change":"2023-10-16T13:51:28.429069+0000","last_active":"2023-10-16T13:51:29.431713+0000","last_peered":"2023-10-16T13:51:29.431713+0000","last_clean":"2023-10-16T13:51:29.431713+0000","last_became_active":"2023-10-16T13:51:28.427120+0000","last_became_peered":"2023-10-16T13:51:28.427120+0000","last_unstale":"2023-10-16T13:51:29.431713+0000","last_undegraded":"2023-10-16T13:51:29.431713+0000","last_fullsized":"2023-10-16T13:51:29.431713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:09.125941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437722+0000","last_change":"2023-10-16T13:51:27.431593+0000","last_active":"2023-10-16T13:51:29.437722+0000","last_peered":"2023-10-16T13:51:29.437722+0000","last_clean":"2023-10-16T13:51:29.437722+0000","last_became_active":"2023-10-16T13:51:27.431204+0000","last_became_peered":"2023-10-16T13:51:27.431204+0000","last_unstale":"2023-10-16T13:51:29.437722+0000","last_undegraded":"2023-10-16T13:51:29.437722+0000","last_fullsized":"2023-10-16T13:51:29.437722+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:15.612658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437672+0000","last_change":"2023-10-16T13:51:28.446659+0000","last_active":"2023-10-16T13:51:29.437672+0000","last_peered":"2023-10-16T13:51:29.437672+0000","last_clean":"2023-10-16T13:51:29.437672+0000","last_became_active":"2023-10-16T13:51:28.446326+0000","last_became_peered":"2023-10-16T13:51:28.446326+0000","last_unstale":"2023-10-16T13:51:29.437672+0000","last_undegraded":"2023-10-16T13:51:29.437672+0000","last_fullsized":"2023-10-16T13:51:29.437672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:23.802977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480299+0000","last_change":"2023-10-16T13:51:27.416141+0000","last_active":"2023-10-16T13:51:29.480299+0000","last_peered":"2023-10-16T13:51:29.480299+0000","last_clean":"2023-10-16T13:51:29.480299+0000","last_became_active":"2023-10-16T13:51:27.415768+0000","last_became_peered":"2023-10-16T13:51:27.415768+0000","last_unstale":"2023-10-16T13:51:29.480299+0000","last_undegraded":"2023-10-16T13:51:29.480299+0000","last_fullsized":"2023-10-16T13:51:29.480299+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:57.108267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481011+0000","last_change":"2023-10-16T13:51:28.436388+0000","last_active":"2023-10-16T13:51:29.481011+0000","last_peered":"2023-10-16T13:51:29.481011+0000","last_clean":"2023-10-16T13:51:29.481011+0000","last_became_active":"2023-10-16T13:51:28.434606+0000","last_became_peered":"2023-10-16T13:51:28.434606+0000","last_unstale":"2023-10-16T13:51:29.481011+0000","last_undegraded":"2023-10-16T13:51:29.481011+0000","last_fullsized":"2023-10-16T13:51:29.481011+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:35.361959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480695+0000","last_change":"2023-10-16T13:51:27.436529+0000","last_active":"2023-10-16T13:51:29.480695+0000","last_peered":"2023-10-16T13:51:29.480695+0000","last_clean":"2023-10-16T13:51:29.480695+0000","last_became_active":"2023-10-16T13:51:27.436162+0000","last_became_peered":"2023-10-16T13:51:27.436162+0000","last_unstale":"2023-10-16T13:51:29.480695+0000","last_undegraded":"2023-10-16T13:51:29.480695+0000","last_fullsized":"2023-10-16T13:51:29.480695+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:02.870854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.435193+0000","last_change":"2023-10-16T13:51:28.431455+0000","last_active":"2023-10-16T13:51:29.435193+0000","last_peered":"2023-10-16T13:51:29.435193+0000","last_clean":"2023-10-16T13:51:29.435193+0000","last_became_active":"2023-10-16T13:51:28.431138+0000","last_became_peered":"2023-10-16T13:51:28.431138+0000","last_unstale":"2023-10-16T13:51:29.435193+0000","last_undegraded":"2023-10-16T13:51:29.435193+0000","last_fullsized":"2023-10-16T13:51:29.435193+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:22.692101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434037+0000","last_change":"2023-10-16T13:51:27.433370+0000","last_active":"2023-10-16T13:51:29.434037+0000","last_peered":"2023-10-16T13:51:29.434037+0000","last_clean":"2023-10-16T13:51:29.434037+0000","last_became_active":"2023-10-16T13:51:27.432932+0000","last_became_peered":"2023-10-16T13:51:27.432932+0000","last_unstale":"2023-10-16T13:51:29.434037+0000","last_undegraded":"2023-10-16T13:51:29.434037+0000","last_fullsized":"2023-10-16T13:51:29.434037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:00.236535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432071+0000","last_change":"2023-10-16T13:51:28.443052+0000","last_active":"2023-10-16T13:51:29.432071+0000","last_peered":"2023-10-16T13:51:29.432071+0000","last_clean":"2023-10-16T13:51:29.432071+0000","last_became_active":"2023-10-16T13:51:28.442190+0000","last_became_peered":"2023-10-16T13:51:28.442190+0000","last_unstale":"2023-10-16T13:51:29.432071+0000","last_undegraded":"2023-10-16T13:51:29.432071+0000","last_fullsized":"2023-10-16T13:51:29.432071+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:33.523038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436607+0000","last_change":"2023-10-16T13:51:27.423465+0000","last_active":"2023-10-16T13:51:29.436607+0000","last_peered":"2023-10-16T13:51:29.436607+0000","last_clean":"2023-10-16T13:51:29.436607+0000","last_became_active":"2023-10-16T13:51:27.423042+0000","last_became_peered":"2023-10-16T13:51:27.423042+0000","last_unstale":"2023-10-16T13:51:29.436607+0000","last_undegraded":"2023-10-16T13:51:29.436607+0000","last_fullsized":"2023-10-16T13:51:29.436607+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:45.456693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"185'296","reported_seq":990,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-16T13:51:28.430329+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T13:51:28.430329+0000","last_peered":"2023-10-16T13:51:28.430329+0000","last_clean":"2023-10-16T13:51:28.430329+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T13:51:28.430329+0000","last_undegraded":"2023-10-16T13:51:28.430329+0000","last_fullsized":"2023-10-16T13:51:28.430329+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543074,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48060,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323780,"statfs":{"total":96636764160,"available":96587550720,"internally_reserved":0,"allocated":3063808,"data_stored":1778776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3320000000000001,"15min":1.327},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.73999999999999999},"max":{"1min":1.7729999999999999,"5min":1.835,"15min":1.976},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.423,"15min":1.407},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.302,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.6899999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1779999999999999,"15min":1.218},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6279999999999999,"5min":1.637,"15min":1.8100000000000001},"last":1.7509999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.321,"15min":1.28},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.67900000000000005},"max":{"1min":1.901,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.1850000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.276,"15min":1.274},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.5529999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.115},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3440000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7450000000000001,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.569}]},{"osd":3,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.399,"15min":1.385},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.8120000000000001,"5min":1.871,"15min":2.1070000000000002},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4710000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.84999999999999998},"max":{"1min":2.093,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.833}]},{"osd":4,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3,"15min":1.2609999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.976,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.397},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.446,"15min":1.399},"min":{"1min":0.96699999999999997,"5min":0.84399999999999997,"15min":0.81599999999999995},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.1829999999999998},"last":1.8680000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.4379999999999999,"15min":1.399},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.1880000000000002,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.272},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":1.0389999999999999,"5min":1.0369999999999999,"15min":0.94199999999999995},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":2.9700000000000002},"last":1.976}]},{"osd":6,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.637,"15min":1.6180000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.2229999999999999,"5min":2.306,"15min":2.306},"last":1.927},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.532,"15min":1.524},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85199999999999998},"max":{"1min":1.8919999999999999,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.944}]}]},{"osd":6,"up_from":14,"seq":60129543075,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41896,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329944,"statfs":{"total":96636764160,"available":96593862656,"internally_reserved":0,"allocated":2449408,"data_stored":1186076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.349,"15min":1.3420000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.69599999999999995},"max":{"1min":1.748,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.2},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.8049999999999999,"5min":1.8169999999999999,"15min":1.9990000000000001},"last":1.3939999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.208,"15min":1.23},"min":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.82099999999999995},"max":{"1min":1.6299999999999999,"5min":1.6319999999999999,"15min":1.8280000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2270000000000001,"15min":1.232},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6890000000000001,"5min":1.776,"15min":1.788},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2090000000000001,"15min":1.254},"min":{"1min":0.89900000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6990000000000001,"5min":1.897,"15min":2.0350000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.325,"15min":1.3240000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.72,"5min":1.77,"15min":1.859},"last":1.006}]},{"osd":3,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.3759999999999999},"min":{"1min":0.93999999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.764,"5min":1.946,"15min":1.9850000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4039999999999999,"15min":1.399},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.8200000000000001,"5min":1.8400000000000001,"15min":1.9159999999999999},"last":1.615}]},{"osd":4,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.395,"15min":1.3779999999999999},"min":{"1min":1.1020000000000001,"5min":0.86799999999999999,"15min":0.79600000000000004},"max":{"1min":1.7529999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.542,"15min":1.5369999999999999},"min":{"1min":1.3440000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.02,"5min":2.02,"15min":2.1219999999999999},"last":1.665}]},{"osd":5,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.359,"15min":1.331},"min":{"1min":0.876,"5min":0.86899999999999999,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":1.903,"15min":2.2669999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.498,"15min":1.472},"min":{"1min":0.91500000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.895,"5min":1.9690000000000001,"15min":2.0510000000000002},"last":1.3500000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.4810000000000001},"min":{"1min":1.2190000000000001,"5min":0.88700000000000001,"15min":0.87},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.173},"last":1.548},{"interface":"front","average":{"1min":1.546,"5min":1.548,"15min":1.512},"min":{"1min":1.1779999999999999,"5min":1.0129999999999999,"15min":0.92500000000000004},"max":{"1min":1.923,"5min":1.9850000000000001,"15min":2.1419999999999999},"last":1.246}]}]},{"osd":5,"up_from":14,"seq":60129543076,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41900,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329940,"statfs":{"total":96636764160,"available":96593858560,"internally_reserved":0,"allocated":2453504,"data_stored":1186658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2290000000000001,"15min":1.2070000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.821},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.256,"15min":1.2729999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.746,"5min":1.839,"15min":1.9610000000000001},"last":1.0800000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.8460000000000001,"5min":1.992,"15min":2.1640000000000001},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.228,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.76000000000000001},"max":{"1min":1.6279999999999999,"5min":1.754,"15min":2.431},"last":0.88600000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.337,"15min":1.3560000000000001},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.879},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.327,"15min":1.3300000000000001},"min":{"1min":0.93500000000000005,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.2669999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.3839999999999999,"15min":1.401},"min":{"1min":1.046,"5min":0.98299999999999998,"15min":0.95499999999999996},"max":{"1min":1.827,"5min":2.0990000000000002,"15min":2.5289999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.4410000000000001,"15min":1.446},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.762,"5min":1.782,"15min":2.242},"last":1.7709999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.329,"15min":1.294},"min":{"1min":0.88100000000000001,"5min":0.82399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7030000000000001,"5min":1.782,"15min":2.3679999999999999},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.4650000000000001,"15min":1.429},"min":{"1min":1.125,"5min":0.81599999999999995,"15min":0.80300000000000005},"max":{"1min":1.778,"5min":2.2530000000000001,"15min":2.4020000000000001},"last":1.6899999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4419999999999999,"15min":1.391},"min":{"1min":1.071,"5min":0.91400000000000003,"15min":0.88},"max":{"1min":1.952,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.4950000000000001,"15min":1.4870000000000001},"min":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":0.97299999999999998},"max":{"1min":1.9119999999999999,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.849}]},{"osd":7,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.5760000000000001,"15min":1.5409999999999999},"min":{"1min":1.238,"5min":1.0269999999999999,"15min":0.93300000000000005},"max":{"1min":2.1600000000000001,"5min":2.371,"15min":2.6320000000000001},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.613,"15min":1.6000000000000001},"min":{"1min":1.325,"5min":1.0640000000000001,"15min":1.0089999999999999},"max":{"1min":1.8049999999999999,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.6459999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543076,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55796,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316044,"statfs":{"total":96636764160,"available":96579629056,"internally_reserved":0,"allocated":2461696,"data_stored":1187736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.2350000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.242},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1699999999999999,"15min":1.212},"min":{"1min":0.83999999999999997,"5min":0.82399999999999995,"15min":0.77900000000000003},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2549999999999999,"15min":1.2589999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.7350000000000003,"5min":4.7350000000000003,"15min":4.7350000000000003},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.3,"5min":1.28,"15min":1.3080000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":3.746,"5min":3.746,"15min":3.746},"last":1.6240000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.3320000000000001,"15min":1.3129999999999999},"min":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.88400000000000001},"max":{"1min":5.2599999999999998,"5min":5.2599999999999998,"15min":5.2599999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.3899999999999999,"15min":1.4099999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":3.9100000000000001,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":1.7629999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.304,"15min":1.3149999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.458,"15min":1.4399999999999999},"min":{"1min":1.1200000000000001,"5min":0.995,"15min":0.90600000000000003},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.427}]},{"osd":5,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.3720000000000001,"15min":1.3320000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.1280000000000001,"5min":4.1280000000000001,"15min":4.1280000000000001},"last":1.861},{"interface":"front","average":{"1min":1.419,"5min":1.3640000000000001,"15min":1.329},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.752},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":1.3320000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.395,"15min":1.399},"min":{"1min":0.875,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.3520000000000003,"5min":5.3520000000000003,"15min":5.3520000000000003},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.5229999999999999,"15min":1.52},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"last":1.9490000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.48,"15min":1.4970000000000001},"min":{"1min":1.0760000000000001,"5min":0.92800000000000005,"15min":0.92600000000000005},"max":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.712,"5min":1.639,"15min":1.6459999999999999},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":5.6200000000000001,"5min":5.6200000000000001,"15min":5.6200000000000001},"last":1.7130000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543076,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51628,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320212,"statfs":{"total":96636764160,"available":96583897088,"internally_reserved":0,"allocated":2449408,"data_stored":1186088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95899999999999996,"15min":0.97299999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3080000000000001,"5min":2.04,"15min":3.1000000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.2090000000000001,"15min":1.2310000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5309999999999999,"5min":1.9179999999999999,"15min":2.714},"last":1.2709999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.1240000000000001,"15min":1.099},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.52,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.125,"15min":1.133},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.649,"5min":1.879,"15min":1.879},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.1319999999999999,"15min":1.1910000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.0609999999999999,"5min":2.234,"15min":2.782},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.264,"5min":1.4219999999999999,"15min":1.4670000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.752,"5min":2.1400000000000001,"15min":2.609},"last":1.113}]},{"osd":4,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.8959999999999999,"5min":1.8979999999999999,"15min":2.2330000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.409,"15min":1.4630000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.8660000000000001,"5min":1.9710000000000001,"15min":2.1000000000000001},"last":1.036}]},{"osd":5,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.429,"15min":1.454},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.6779999999999999,"5min":2.1110000000000002,"15min":2.306},"last":1.165},{"interface":"front","average":{"1min":1.427,"5min":1.546,"15min":1.534},"min":{"1min":1.139,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.7909999999999999,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.3740000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4850000000000001,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8029999999999999,"5min":2.1389999999999998,"15min":2.4430000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.4850000000000001,"15min":1.4990000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.387},"last":1.2470000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6339999999999999,"15min":1.6200000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.9690000000000001,"5min":2.2730000000000001,"15min":2.3340000000000001},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.6759999999999999,"15min":1.679},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":2.0110000000000001,"5min":2.387,"15min":2.387},"last":1.6379999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51620,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320220,"statfs":{"total":96636764160,"available":96583905280,"internally_reserved":0,"allocated":2449408,"data_stored":1186144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3480000000000001,"5min":1.732,"15min":2.1499999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.125,"15min":1.159},"min":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.367,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.141,"15min":1.151},"min":{"1min":0.75,"5min":0.75,"15min":0.72199999999999998},"max":{"1min":1.532,"5min":1.956,"15min":2.4550000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.26,"5min":1.2809999999999999,"15min":1.286},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.754,"5min":1.804,"15min":2.3490000000000002},"last":0.81599999999999995}]},{"osd":3,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1339999999999999,"15min":1.1519999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.806,"5min":1.849,"15min":1.982},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.345,"5min":1.359,"15min":1.379},"min":{"1min":0.98799999999999999,"5min":0.91200000000000003,"15min":0.90700000000000003},"max":{"1min":1.7090000000000001,"5min":1.855,"15min":2.7970000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4950000000000001,"15min":1.5069999999999999},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.054},"max":{"1min":1.9430000000000001,"5min":2.1429999999999998,"15min":2.6589999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.452,"5min":1.492,"15min":1.514},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":1.762,"5min":2.0510000000000002,"15min":2.2629999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.575},"min":{"1min":1.129,"5min":1.129,"15min":1.056},"max":{"1min":1.921,"5min":2.101,"15min":2.5569999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.5009999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.7490000000000001,"5min":2.2650000000000001,"15min":2.899},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5549999999999999},"min":{"1min":1.0760000000000001,"5min":1.032,"15min":1.032},"max":{"1min":1.843,"5min":2.4300000000000002,"15min":3.0339999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6080000000000001,"15min":1.637},"min":{"1min":1.145,"5min":1.145,"15min":1.145},"max":{"1min":1.982,"5min":2.0779999999999998,"15min":3.0790000000000002},"last":0.626}]},{"osd":7,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7050000000000001,"15min":1.6919999999999999},"min":{"1min":1.24,"5min":1.127,"15min":1.127},"max":{"1min":2.1880000000000002,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.651,"5min":1.643,"15min":1.661},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2429999999999999,"5min":2.5529999999999999,"15min":3.1320000000000001},"last":0.73399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47472,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324368,"statfs":{"total":96636764160,"available":96588152832,"internally_reserved":0,"allocated":2461696,"data_stored":1192388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.92900000000000005,"15min":0.94699999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4930000000000001,"5min":1.8,"15min":1.8},"last":1.135}]},{"osd":2,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0289999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.6559999999999999,"5min":1.71,"15min":1.8069999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.2190000000000001,"15min":1.2470000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.373,"5min":1.992,"15min":1.994},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.385,"15min":1.3899999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6220000000000001,"5min":1.883,"15min":1.9610000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.677,"5min":2.1269999999999998,"15min":2.2610000000000001},"last":1.044}]},{"osd":4,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.3520000000000001,"15min":1.339},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.631,"5min":1.867,"15min":2.1539999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.49,"15min":1.506},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.71899999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.3779999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.5,"15min":1.5009999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.0630000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.776},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4710000000000001,"15min":1.4670000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8069999999999999,"5min":1.9770000000000001,"15min":2.028},"last":1.343}]},{"osd":6,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.615,"15min":1.611},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.6120000000000001,"15min":1.605},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.76900000000000002},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":1.706}]},{"osd":7,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.732,"15min":1.704},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1949999999999998,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.611,"15min":1.6040000000000001},"min":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"max":{"1min":1.9550000000000001,"5min":2.1040000000000001,"15min":2.3140000000000001},"last":1.522}]}]},{"osd":0,"up_from":14,"seq":60129543075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60520,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311320,"statfs":{"total":96636764160,"available":96574791680,"internally_reserved":0,"allocated":3039232,"data_stored":1776422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":0.92600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.399,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1479999999999999,"15min":1.1579999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.0870000000000002,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.0269999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1519999999999999,"15min":1.161},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.498,"5min":1.7010000000000001,"15min":1.9830000000000001},"last":1.385},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2250000000000001,"15min":1.248},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.155}]},{"osd":3,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.665,"5min":2.3340000000000001,"15min":3.0350000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3460000000000001,"15min":1.3640000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.613,"5min":2.2450000000000001,"15min":2.9670000000000001},"last":1.278}]},{"osd":4,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.365,"15min":1.393},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.6379999999999999,"5min":1.76,"15min":2.8740000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.462,"15min":1.5029999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9119999999999999,"5min":2.0150000000000001,"15min":4.6159999999999997},"last":1.47}]},{"osd":5,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.528,"15min":1.5269999999999999},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":1.615},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.4419999999999999,"15min":1.446},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.7969999999999999,"5min":2.0880000000000001,"15min":3.1589999999999998},"last":1.7390000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.613,"15min":1.6240000000000001},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.117,"5min":2.2610000000000001,"15min":4.0270000000000001},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5880000000000001,"15min":1.605},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.992,"5min":2.992,"15min":2.992},"last":1.6919999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.704,"15min":1.7110000000000001},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6200000000000001,"15min":1.6180000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.081,"5min":2.286,"15min":3.3010000000000002},"last":1.8520000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:51:32.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:51:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.238+0000 7f7fbc51d640 1 Processor -- start 2023-10-16T13:51:33.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.238+0000 7f7fbc51d640 1 -- start start 2023-10-16T13:51:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.239+0000 7f7fbc51d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 0x7f7fb415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fbc51d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb415ed60 con 0x7f7fb415ae80 2023-10-16T13:51:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fbc51d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb40707e0 con 0x7f7fb40ce390 2023-10-16T13:51:33.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fbc51d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb406b810 con 0x7f7fb415aaa0 2023-10-16T13:51:33.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fb9ab8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7fb415ae80 0x7f7fb40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53906/0 (socket says 172.21.15.60:53906) 2023-10-16T13:51:33.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fb9ab8640 1 -- 172.21.15.60:0/2348486928 learned_addr learned my addr 172.21.15.60:0/2348486928 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:33.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fba2b9640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 0x7f7fb415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273966761 0 0) 0x7f7fb415ed60 con 0x7f7fb415ae80 2023-10-16T13:51:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.240+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa0003520 con 0x7f7fb415ae80 2023-10-16T13:51:33.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2768359099 0 0) 0x7f7fa0003520 con 0x7f7fb415ae80 2023-10-16T13:51:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 msgr2=0x7f7fb415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 0x7f7fb415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 >> v1:172.21.15.144:6789/0 conn(0x7f7fb40ce390 legacy=0x7f7fb4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb415ef90 con 0x7f7fb415ae80 2023-10-16T13:51:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7fa4001670 con 0x7f7fb415ae80 2023-10-16T13:51:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 >> v1:172.21.15.60:6789/0 conn(0x7f7fb415ae80 legacy=0x7f7fb40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 0x7f7fa00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fa0003f70 0x7f7fa00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fa0004d00 0x7f7fa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb406b810 con 0x7f7fa0004d00 2023-10-16T13:51:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa000bac0 con 0x7f7fa0003f70 2023-10-16T13:51:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa000bc80 con 0x7f7fb415aaa0 2023-10-16T13:51:33.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fbaaba640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fa0004d00 0x7f7fa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.241+0000 7f7fb9ab8640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fa0003f70 0x7f7fa00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.242+0000 7f7fba2b9640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 0x7f7fa00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.242+0000 7f7fbaaba640 1 -- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 msgr2=0x7f7fa00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.242+0000 7f7fbaaba640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fb415aaa0 0x7f7fa00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.242+0000 7f7fbaaba640 1 -- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fa0003f70 msgr2=0x7f7fa00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:33.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.242+0000 7f7fbaaba640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fa0003f70 0x7f7fa00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.242+0000 7f7fbaaba640 1 -- 172.21.15.60:0/2348486928 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa0003520 con 0x7f7fa0004d00 2023-10-16T13:51:33.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.243+0000 7f7fbaaba640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fa0004d00 0x7f7fa0005180 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f7fac001e50 tx=0x7f7fa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5305443c3171c919 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:33.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.243+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac003b70 con 0x7f7fa0004d00 2023-10-16T13:51:33.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.243+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac003cd0 con 0x7f7fa0004d00 2023-10-16T13:51:33.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.243+0000 7f7fb92b7640 1 -- 172.21.15.60:0/2348486928 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac005310 con 0x7f7fa0004d00 2023-10-16T13:51:33.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2348486928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fa0004d00 msgr2=0x7f7fa0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2348486928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fa0004d00 0x7f7fa0005180 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f7fac001e50 tx=0x7f7fa00057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbaaba640 1 -- 172.21.15.60:0/2348486928 reap_dead start 2023-10-16T13:51:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2348486928 shutdown_connections 2023-10-16T13:51:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2348486928 >> 172.21.15.60:0/2348486928 conn(0x7f7fb40cb410 msgr2=0x7f7fb40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2348486928 shutdown_connections 2023-10-16T13:51:33.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.244+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2348486928 wait complete. 2023-10-16T13:51:33.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.246+0000 7f7fbc51d640 1 Processor -- start 2023-10-16T13:51:33.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.246+0000 7f7fbc51d640 1 -- start start 2023-10-16T13:51:33.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fbc51d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fa0004d00 0x7f7fb41f1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fbc51d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 0x7f7fb40c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fbc51d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40c4e10 0x7f7fb41575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fbc51d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40c52c0 con 0x7f7fb415cd30 2023-10-16T13:51:33.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fbc51d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb415d5a0 con 0x7f7fb40c4e10 2023-10-16T13:51:33.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fbc51d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb415d880 con 0x7f7fa0004d00 2023-10-16T13:51:33.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fb9ab8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 0x7f7fb40c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fb9ab8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 0x7f7fb40c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55044/0 (socket says 172.21.15.60:55044) 2023-10-16T13:51:33.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.247+0000 7f7fb9ab8640 1 -- 172.21.15.60:0/2842815854 learned_addr learned my addr 172.21.15.60:0/2842815854 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:33.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fba2b9640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fa0004d00 0x7f7fb41f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fbaaba640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40c4e10 0x7f7fb41575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fb9ab8640 1 -- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fa0004d00 msgr2=0x7f7fb41f1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fb9ab8640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fa0004d00 0x7f7fb41f1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fb9ab8640 1 -- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40c4e10 msgr2=0x7f7fb41575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fb9ab8640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40c4e10 0x7f7fb41575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fb9ab8640 1 -- 172.21.15.60:0/2842815854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb415db60 con 0x7f7fb415cd30 2023-10-16T13:51:33.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fba2b9640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fa0004d00 0x7f7fb41f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:33.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7fb9ab8640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 0x7f7fb40c48d0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400e7e0 tx=0x7f7fa400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:33.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400c8d0 con 0x7f7fb415cd30 2023-10-16T13:51:33.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.248+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa400ca30 con 0x7f7fb415cd30 2023-10-16T13:51:33.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.249+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb415de40 con 0x7f7fb415cd30 2023-10-16T13:51:33.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.249+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400cde0 con 0x7f7fb415cd30 2023-10-16T13:51:33.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.249+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb415d1b0 con 0x7f7fb415cd30 2023-10-16T13:51:33.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.250+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb415d370 con 0x7f7fb415cd30 2023-10-16T13:51:33.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.251+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7fa401d460 con 0x7f7fb415cd30 2023-10-16T13:51:33.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.252+0000 7f7faaffd640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7f9806af90 0x7f7f9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.252+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fa4089850 con 0x7f7fb415cd30 2023-10-16T13:51:33.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.252+0000 7f7fba2b9640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7f9806af90 0x7f7f9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.254+0000 7f7fba2b9640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7f9806af90 0x7f7f9806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40c7a60 tx=0x7f7fb0008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:33.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.256+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7fa40547e0 con 0x7f7fb415cd30 2023-10-16T13:51:33.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.447+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 --> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fb415f550 con 0x7f7f9806af90 2023-10-16T13:51:33.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.471+0000 7f7faaffd640 1 -- 172.21.15.60:0/2842815854 <== mgr.7918 v2:172.21.15.144:6832/1700115305 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287197 (secure 0 0 0) 0x7f7fb0009690 con 0x7f7f9806af90 2023-10-16T13:51:33.477 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:33.479 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T13:51:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7f9806af90 msgr2=0x7f7f9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7f9806af90 0x7f7f9806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40c7a60 tx=0x7f7fb0008040 comp rx=0 tx=0).stop 2023-10-16T13:51:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 msgr2=0x7f7fb40c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 0x7f7fb40c48d0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400e7e0 tx=0x7f7fa400eca0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 shutdown_connections 2023-10-16T13:51:33.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f7f9806af90 0x7f7f9806d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7fb40c4e10 0x7f7fb41575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fb415cd30 0x7f7fb40c48d0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 --2- 172.21.15.60:0/2842815854 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7fa0004d00 0x7f7fb41f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.476+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 >> 172.21.15.60:0/2842815854 conn(0x7f7fb40cb410 msgr2=0x7f7fb415a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:33.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.477+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 shutdown_connections 2023-10-16T13:51:33.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.477+0000 7f7fbc51d640 1 -- 172.21.15.60:0/2842815854 wait complete. 2023-10-16T13:51:33.498 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T13:51:32.856395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":527,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":334,"ondisk_log_size":334,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":398892,"kb_used_data":20340,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754575828,"statfs":{"total":773094113280,"available":772685647872,"internally_reserved":0,"allocated":20828160,"data_stored":10680288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":387501624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002213"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482072+0000","last_change":"2023-10-16T13:51:28.452872+0000","last_active":"2023-10-16T13:51:29.482072+0000","last_peered":"2023-10-16T13:51:29.482072+0000","last_clean":"2023-10-16T13:51:29.482072+0000","last_became_active":"2023-10-16T13:51:28.452098+0000","last_became_peered":"2023-10-16T13:51:28.452098+0000","last_unstale":"2023-10-16T13:51:29.482072+0000","last_undegraded":"2023-10-16T13:51:29.482072+0000","last_fullsized":"2023-10-16T13:51:29.482072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:36:04.816035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481483+0000","last_change":"2023-10-16T13:51:28.444421+0000","last_active":"2023-10-16T13:51:29.481483+0000","last_peered":"2023-10-16T13:51:29.481483+0000","last_clean":"2023-10-16T13:51:29.481483+0000","last_became_active":"2023-10-16T13:51:28.444148+0000","last_became_peered":"2023-10-16T13:51:28.444148+0000","last_unstale":"2023-10-16T13:51:29.481483+0000","last_undegraded":"2023-10-16T13:51:29.481483+0000","last_fullsized":"2023-10-16T13:51:29.481483+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:17.878183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436160+0000","last_change":"2023-10-16T13:51:28.446670+0000","last_active":"2023-10-16T13:51:29.436160+0000","last_peered":"2023-10-16T13:51:29.436160+0000","last_clean":"2023-10-16T13:51:29.436160+0000","last_became_active":"2023-10-16T13:51:28.446388+0000","last_became_peered":"2023-10-16T13:51:28.446388+0000","last_unstale":"2023-10-16T13:51:29.436160+0000","last_undegraded":"2023-10-16T13:51:29.436160+0000","last_fullsized":"2023-10-16T13:51:29.436160+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.586198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480092+0000","last_change":"2023-10-16T13:51:28.428006+0000","last_active":"2023-10-16T13:51:29.480092+0000","last_peered":"2023-10-16T13:51:29.480092+0000","last_clean":"2023-10-16T13:51:29.480092+0000","last_became_active":"2023-10-16T13:51:28.427720+0000","last_became_peered":"2023-10-16T13:51:28.427720+0000","last_unstale":"2023-10-16T13:51:29.480092+0000","last_undegraded":"2023-10-16T13:51:29.480092+0000","last_fullsized":"2023-10-16T13:51:29.480092+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:08.355314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480632+0000","last_change":"2023-10-16T13:51:28.435341+0000","last_active":"2023-10-16T13:51:29.480632+0000","last_peered":"2023-10-16T13:51:29.480632+0000","last_clean":"2023-10-16T13:51:29.480632+0000","last_became_active":"2023-10-16T13:51:28.434579+0000","last_became_peered":"2023-10-16T13:51:28.434579+0000","last_unstale":"2023-10-16T13:51:29.480632+0000","last_undegraded":"2023-10-16T13:51:29.480632+0000","last_fullsized":"2023-10-16T13:51:29.480632+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:09:05.351079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481493+0000","last_change":"2023-10-16T13:51:28.430796+0000","last_active":"2023-10-16T13:51:29.481493+0000","last_peered":"2023-10-16T13:51:29.481493+0000","last_clean":"2023-10-16T13:51:29.481493+0000","last_became_active":"2023-10-16T13:51:28.430426+0000","last_became_peered":"2023-10-16T13:51:28.430426+0000","last_unstale":"2023-10-16T13:51:29.481493+0000","last_undegraded":"2023-10-16T13:51:29.481493+0000","last_fullsized":"2023-10-16T13:51:29.481493+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:59.941460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479964+0000","last_change":"2023-10-16T13:51:28.436398+0000","last_active":"2023-10-16T13:51:29.479964+0000","last_peered":"2023-10-16T13:51:29.479964+0000","last_clean":"2023-10-16T13:51:29.479964+0000","last_became_active":"2023-10-16T13:51:28.436065+0000","last_became_peered":"2023-10-16T13:51:28.436065+0000","last_unstale":"2023-10-16T13:51:29.479964+0000","last_undegraded":"2023-10-16T13:51:29.479964+0000","last_fullsized":"2023-10-16T13:51:29.479964+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:14.831334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479800+0000","last_change":"2023-10-16T13:51:28.439882+0000","last_active":"2023-10-16T13:51:29.479800+0000","last_peered":"2023-10-16T13:51:29.479800+0000","last_clean":"2023-10-16T13:51:29.479800+0000","last_became_active":"2023-10-16T13:51:28.439609+0000","last_became_peered":"2023-10-16T13:51:28.439609+0000","last_unstale":"2023-10-16T13:51:29.479800+0000","last_undegraded":"2023-10-16T13:51:29.479800+0000","last_fullsized":"2023-10-16T13:51:29.479800+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:33.754375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481730+0000","last_change":"2023-10-16T13:51:28.450316+0000","last_active":"2023-10-16T13:51:29.481730+0000","last_peered":"2023-10-16T13:51:29.481730+0000","last_clean":"2023-10-16T13:51:29.481730+0000","last_became_active":"2023-10-16T13:51:28.450014+0000","last_became_peered":"2023-10-16T13:51:28.450014+0000","last_unstale":"2023-10-16T13:51:29.481730+0000","last_undegraded":"2023-10-16T13:51:29.481730+0000","last_fullsized":"2023-10-16T13:51:29.481730+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:42.609197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481614+0000","last_change":"2023-10-16T13:51:28.447959+0000","last_active":"2023-10-16T13:51:29.481614+0000","last_peered":"2023-10-16T13:51:29.481614+0000","last_clean":"2023-10-16T13:51:29.481614+0000","last_became_active":"2023-10-16T13:51:28.447330+0000","last_became_peered":"2023-10-16T13:51:28.447330+0000","last_unstale":"2023-10-16T13:51:29.481614+0000","last_undegraded":"2023-10-16T13:51:29.481614+0000","last_fullsized":"2023-10-16T13:51:29.481614+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:23.126342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481271+0000","last_change":"2023-10-16T13:51:28.447939+0000","last_active":"2023-10-16T13:51:29.481271+0000","last_peered":"2023-10-16T13:51:29.481271+0000","last_clean":"2023-10-16T13:51:29.481271+0000","last_became_active":"2023-10-16T13:51:28.447592+0000","last_became_peered":"2023-10-16T13:51:28.447592+0000","last_unstale":"2023-10-16T13:51:29.481271+0000","last_undegraded":"2023-10-16T13:51:29.481271+0000","last_fullsized":"2023-10-16T13:51:29.481271+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:05.901412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480625+0000","last_change":"2023-10-16T13:51:28.440586+0000","last_active":"2023-10-16T13:51:29.480625+0000","last_peered":"2023-10-16T13:51:29.480625+0000","last_clean":"2023-10-16T13:51:29.480625+0000","last_became_active":"2023-10-16T13:51:28.440306+0000","last_became_peered":"2023-10-16T13:51:28.440306+0000","last_unstale":"2023-10-16T13:51:29.480625+0000","last_undegraded":"2023-10-16T13:51:29.480625+0000","last_fullsized":"2023-10-16T13:51:29.480625+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.029670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436713+0000","last_change":"2023-10-16T13:51:28.433418+0000","last_active":"2023-10-16T13:51:29.436713+0000","last_peered":"2023-10-16T13:51:29.436713+0000","last_clean":"2023-10-16T13:51:29.436713+0000","last_became_active":"2023-10-16T13:51:28.433121+0000","last_became_peered":"2023-10-16T13:51:28.433121+0000","last_unstale":"2023-10-16T13:51:29.436713+0000","last_undegraded":"2023-10-16T13:51:29.436713+0000","last_fullsized":"2023-10-16T13:51:29.436713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:17.955633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.433914+0000","last_change":"2023-10-16T13:51:28.436213+0000","last_active":"2023-10-16T13:51:29.433914+0000","last_peered":"2023-10-16T13:51:29.433914+0000","last_clean":"2023-10-16T13:51:29.433914+0000","last_became_active":"2023-10-16T13:51:28.435912+0000","last_became_peered":"2023-10-16T13:51:28.435912+0000","last_unstale":"2023-10-16T13:51:29.433914+0000","last_undegraded":"2023-10-16T13:51:29.433914+0000","last_fullsized":"2023-10-16T13:51:29.433914+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:42.309272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480402+0000","last_change":"2023-10-16T13:51:28.448474+0000","last_active":"2023-10-16T13:51:29.480402+0000","last_peered":"2023-10-16T13:51:29.480402+0000","last_clean":"2023-10-16T13:51:29.480402+0000","last_became_active":"2023-10-16T13:51:28.445453+0000","last_became_peered":"2023-10-16T13:51:28.445453+0000","last_unstale":"2023-10-16T13:51:29.480402+0000","last_undegraded":"2023-10-16T13:51:29.480402+0000","last_fullsized":"2023-10-16T13:51:29.480402+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:03.330010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479296+0000","last_change":"2023-10-16T13:51:28.436423+0000","last_active":"2023-10-16T13:51:29.479296+0000","last_peered":"2023-10-16T13:51:29.479296+0000","last_clean":"2023-10-16T13:51:29.479296+0000","last_became_active":"2023-10-16T13:51:28.435664+0000","last_became_peered":"2023-10-16T13:51:28.435664+0000","last_unstale":"2023-10-16T13:51:29.479296+0000","last_undegraded":"2023-10-16T13:51:29.479296+0000","last_fullsized":"2023-10-16T13:51:29.479296+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:59.198148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.483156+0000","last_change":"2023-10-16T13:51:28.446567+0000","last_active":"2023-10-16T13:51:29.483156+0000","last_peered":"2023-10-16T13:51:29.483156+0000","last_clean":"2023-10-16T13:51:29.483156+0000","last_became_active":"2023-10-16T13:51:28.446313+0000","last_became_peered":"2023-10-16T13:51:28.446313+0000","last_unstale":"2023-10-16T13:51:29.483156+0000","last_undegraded":"2023-10-16T13:51:29.483156+0000","last_fullsized":"2023-10-16T13:51:29.483156+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:23.802977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432072+0000","last_change":"2023-10-16T13:51:28.432963+0000","last_active":"2023-10-16T13:51:29.432072+0000","last_peered":"2023-10-16T13:51:29.432072+0000","last_clean":"2023-10-16T13:51:29.432072+0000","last_became_active":"2023-10-16T13:51:28.432313+0000","last_became_peered":"2023-10-16T13:51:28.432313+0000","last_unstale":"2023-10-16T13:51:29.432072+0000","last_undegraded":"2023-10-16T13:51:29.432072+0000","last_fullsized":"2023-10-16T13:51:29.432072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:24.394538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482669+0000","last_change":"2023-10-16T13:51:28.442323+0000","last_active":"2023-10-16T13:51:29.482669+0000","last_peered":"2023-10-16T13:51:29.482669+0000","last_clean":"2023-10-16T13:51:29.482669+0000","last_became_active":"2023-10-16T13:51:28.442030+0000","last_became_peered":"2023-10-16T13:51:28.442030+0000","last_unstale":"2023-10-16T13:51:29.482669+0000","last_undegraded":"2023-10-16T13:51:29.482669+0000","last_fullsized":"2023-10-16T13:51:29.482669+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:58.120912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436424+0000","last_change":"2023-10-16T13:51:28.446888+0000","last_active":"2023-10-16T13:51:29.436424+0000","last_peered":"2023-10-16T13:51:29.436424+0000","last_clean":"2023-10-16T13:51:29.436424+0000","last_became_active":"2023-10-16T13:51:28.446063+0000","last_became_peered":"2023-10-16T13:51:28.446063+0000","last_unstale":"2023-10-16T13:51:29.436424+0000","last_undegraded":"2023-10-16T13:51:29.436424+0000","last_fullsized":"2023-10-16T13:51:29.436424+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:05.879940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481080+0000","last_change":"2023-10-16T13:51:28.446345+0000","last_active":"2023-10-16T13:51:29.481080+0000","last_peered":"2023-10-16T13:51:29.481080+0000","last_clean":"2023-10-16T13:51:29.481080+0000","last_became_active":"2023-10-16T13:51:28.446021+0000","last_became_peered":"2023-10-16T13:51:28.446021+0000","last_unstale":"2023-10-16T13:51:29.481080+0000","last_undegraded":"2023-10-16T13:51:29.481080+0000","last_fullsized":"2023-10-16T13:51:29.481080+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:03.200092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481272+0000","last_change":"2023-10-16T13:51:28.428226+0000","last_active":"2023-10-16T13:51:29.481272+0000","last_peered":"2023-10-16T13:51:29.481272+0000","last_clean":"2023-10-16T13:51:29.481272+0000","last_became_active":"2023-10-16T13:51:28.426771+0000","last_became_peered":"2023-10-16T13:51:28.426771+0000","last_unstale":"2023-10-16T13:51:29.481272+0000","last_undegraded":"2023-10-16T13:51:29.481272+0000","last_fullsized":"2023-10-16T13:51:29.481272+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:32.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481894+0000","last_change":"2023-10-16T13:51:28.433877+0000","last_active":"2023-10-16T13:51:29.481894+0000","last_peered":"2023-10-16T13:51:29.481894+0000","last_clean":"2023-10-16T13:51:29.481894+0000","last_became_active":"2023-10-16T13:51:28.433573+0000","last_became_peered":"2023-10-16T13:51:28.433573+0000","last_unstale":"2023-10-16T13:51:29.481894+0000","last_undegraded":"2023-10-16T13:51:29.481894+0000","last_fullsized":"2023-10-16T13:51:29.481894+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:46.469338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479980+0000","last_change":"2023-10-16T13:51:28.439612+0000","last_active":"2023-10-16T13:51:29.479980+0000","last_peered":"2023-10-16T13:51:29.479980+0000","last_clean":"2023-10-16T13:51:29.479980+0000","last_became_active":"2023-10-16T13:51:28.438873+0000","last_became_peered":"2023-10-16T13:51:28.438873+0000","last_unstale":"2023-10-16T13:51:29.479980+0000","last_undegraded":"2023-10-16T13:51:29.479980+0000","last_fullsized":"2023-10-16T13:51:29.479980+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:08.800708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437289+0000","last_change":"2023-10-16T13:51:28.446332+0000","last_active":"2023-10-16T13:51:29.437289+0000","last_peered":"2023-10-16T13:51:29.437289+0000","last_clean":"2023-10-16T13:51:29.437289+0000","last_became_active":"2023-10-16T13:51:28.446000+0000","last_became_peered":"2023-10-16T13:51:28.446000+0000","last_unstale":"2023-10-16T13:51:29.437289+0000","last_undegraded":"2023-10-16T13:51:29.437289+0000","last_fullsized":"2023-10-16T13:51:29.437289+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:25.250592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482731+0000","last_change":"2023-10-16T13:51:28.443852+0000","last_active":"2023-10-16T13:51:29.482731+0000","last_peered":"2023-10-16T13:51:29.482731+0000","last_clean":"2023-10-16T13:51:29.482731+0000","last_became_active":"2023-10-16T13:51:28.443536+0000","last_became_peered":"2023-10-16T13:51:28.443536+0000","last_unstale":"2023-10-16T13:51:29.482731+0000","last_undegraded":"2023-10-16T13:51:29.482731+0000","last_fullsized":"2023-10-16T13:51:29.482731+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:27.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436531+0000","last_change":"2023-10-16T13:51:28.440374+0000","last_active":"2023-10-16T13:51:29.436531+0000","last_peered":"2023-10-16T13:51:29.436531+0000","last_clean":"2023-10-16T13:51:29.436531+0000","last_became_active":"2023-10-16T13:51:28.440093+0000","last_became_peered":"2023-10-16T13:51:28.440093+0000","last_unstale":"2023-10-16T13:51:29.436531+0000","last_undegraded":"2023-10-16T13:51:29.436531+0000","last_fullsized":"2023-10-16T13:51:29.436531+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:04:15.210378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480440+0000","last_change":"2023-10-16T13:51:28.445966+0000","last_active":"2023-10-16T13:51:29.480440+0000","last_peered":"2023-10-16T13:51:29.480440+0000","last_clean":"2023-10-16T13:51:29.480440+0000","last_became_active":"2023-10-16T13:51:28.445676+0000","last_became_peered":"2023-10-16T13:51:28.445676+0000","last_unstale":"2023-10-16T13:51:29.480440+0000","last_undegraded":"2023-10-16T13:51:29.480440+0000","last_fullsized":"2023-10-16T13:51:29.480440+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:03.049286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436437+0000","last_change":"2023-10-16T13:51:28.432127+0000","last_active":"2023-10-16T13:51:29.436437+0000","last_peered":"2023-10-16T13:51:29.436437+0000","last_clean":"2023-10-16T13:51:29.436437+0000","last_became_active":"2023-10-16T13:51:28.431848+0000","last_became_peered":"2023-10-16T13:51:28.431848+0000","last_unstale":"2023-10-16T13:51:29.436437+0000","last_undegraded":"2023-10-16T13:51:29.436437+0000","last_fullsized":"2023-10-16T13:51:29.436437+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:42.546116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434371+0000","last_change":"2023-10-16T13:51:28.449840+0000","last_active":"2023-10-16T13:51:29.434371+0000","last_peered":"2023-10-16T13:51:29.434371+0000","last_clean":"2023-10-16T13:51:29.434371+0000","last_became_active":"2023-10-16T13:51:28.449501+0000","last_became_peered":"2023-10-16T13:51:28.449501+0000","last_unstale":"2023-10-16T13:51:29.434371+0000","last_undegraded":"2023-10-16T13:51:29.434371+0000","last_fullsized":"2023-10-16T13:51:29.434371+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.413170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434086+0000","last_change":"2023-10-16T13:51:28.447252+0000","last_active":"2023-10-16T13:51:29.434086+0000","last_peered":"2023-10-16T13:51:29.434086+0000","last_clean":"2023-10-16T13:51:29.434086+0000","last_became_active":"2023-10-16T13:51:28.446968+0000","last_became_peered":"2023-10-16T13:51:28.446968+0000","last_unstale":"2023-10-16T13:51:29.434086+0000","last_undegraded":"2023-10-16T13:51:29.434086+0000","last_fullsized":"2023-10-16T13:51:29.434086+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.486179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431212+0000","last_change":"2023-10-16T13:51:28.432775+0000","last_active":"2023-10-16T13:51:29.431212+0000","last_peered":"2023-10-16T13:51:29.431212+0000","last_clean":"2023-10-16T13:51:29.431212+0000","last_became_active":"2023-10-16T13:51:28.427111+0000","last_became_peered":"2023-10-16T13:51:28.427111+0000","last_unstale":"2023-10-16T13:51:29.431212+0000","last_undegraded":"2023-10-16T13:51:29.431212+0000","last_fullsized":"2023-10-16T13:51:29.431212+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:44.722454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482968+0000","last_change":"2023-10-16T13:51:28.439803+0000","last_active":"2023-10-16T13:51:29.482968+0000","last_peered":"2023-10-16T13:51:29.482968+0000","last_clean":"2023-10-16T13:51:29.482968+0000","last_became_active":"2023-10-16T13:51:28.439476+0000","last_became_peered":"2023-10-16T13:51:28.439476+0000","last_unstale":"2023-10-16T13:51:29.482968+0000","last_undegraded":"2023-10-16T13:51:29.482968+0000","last_fullsized":"2023-10-16T13:51:29.482968+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:50.161706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"196'4","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482217+0000","last_change":"2023-10-16T13:51:27.439181+0000","last_active":"2023-10-16T13:51:29.482217+0000","last_peered":"2023-10-16T13:51:29.482217+0000","last_clean":"2023-10-16T13:51:29.482217+0000","last_became_active":"2023-10-16T13:51:27.438891+0000","last_became_peered":"2023-10-16T13:51:27.438891+0000","last_unstale":"2023-10-16T13:51:29.482217+0000","last_undegraded":"2023-10-16T13:51:29.482217+0000","last_fullsized":"2023-10-16T13:51:29.482217+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:44.978387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436349+0000","last_change":"2023-10-16T13:51:28.441903+0000","last_active":"2023-10-16T13:51:29.436349+0000","last_peered":"2023-10-16T13:51:29.436349+0000","last_clean":"2023-10-16T13:51:29.436349+0000","last_became_active":"2023-10-16T13:51:28.438658+0000","last_became_peered":"2023-10-16T13:51:28.438658+0000","last_unstale":"2023-10-16T13:51:29.436349+0000","last_undegraded":"2023-10-16T13:51:29.436349+0000","last_fullsized":"2023-10-16T13:51:29.436349+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:52.516835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.435251+0000","last_change":"2023-10-16T13:51:27.437751+0000","last_active":"2023-10-16T13:51:29.435251+0000","last_peered":"2023-10-16T13:51:29.435251+0000","last_clean":"2023-10-16T13:51:29.435251+0000","last_became_active":"2023-10-16T13:51:27.436383+0000","last_became_peered":"2023-10-16T13:51:27.436383+0000","last_unstale":"2023-10-16T13:51:29.435251+0000","last_undegraded":"2023-10-16T13:51:29.435251+0000","last_fullsized":"2023-10-16T13:51:29.435251+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:58.928816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481884+0000","last_change":"2023-10-16T13:51:28.442759+0000","last_active":"2023-10-16T13:51:29.481884+0000","last_peered":"2023-10-16T13:51:29.481884+0000","last_clean":"2023-10-16T13:51:29.481884+0000","last_became_active":"2023-10-16T13:51:28.440811+0000","last_became_peered":"2023-10-16T13:51:28.440811+0000","last_unstale":"2023-10-16T13:51:29.481884+0000","last_undegraded":"2023-10-16T13:51:29.481884+0000","last_fullsized":"2023-10-16T13:51:29.481884+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:11.349163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480320+0000","last_change":"2023-10-16T13:51:27.437678+0000","last_active":"2023-10-16T13:51:29.480320+0000","last_peered":"2023-10-16T13:51:29.480320+0000","last_clean":"2023-10-16T13:51:29.480320+0000","last_became_active":"2023-10-16T13:51:27.437069+0000","last_became_peered":"2023-10-16T13:51:27.437069+0000","last_unstale":"2023-10-16T13:51:29.480320+0000","last_undegraded":"2023-10-16T13:51:29.480320+0000","last_fullsized":"2023-10-16T13:51:29.480320+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:43:45.302196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436939+0000","last_change":"2023-10-16T13:51:28.442692+0000","last_active":"2023-10-16T13:51:29.436939+0000","last_peered":"2023-10-16T13:51:29.436939+0000","last_clean":"2023-10-16T13:51:29.436939+0000","last_became_active":"2023-10-16T13:51:28.442335+0000","last_became_peered":"2023-10-16T13:51:28.442335+0000","last_unstale":"2023-10-16T13:51:29.436939+0000","last_undegraded":"2023-10-16T13:51:29.436939+0000","last_fullsized":"2023-10-16T13:51:29.436939+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:31.461167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480687+0000","last_change":"2023-10-16T13:51:27.432100+0000","last_active":"2023-10-16T13:51:29.480687+0000","last_peered":"2023-10-16T13:51:29.480687+0000","last_clean":"2023-10-16T13:51:29.480687+0000","last_became_active":"2023-10-16T13:51:27.431837+0000","last_became_peered":"2023-10-16T13:51:27.431837+0000","last_unstale":"2023-10-16T13:51:29.480687+0000","last_undegraded":"2023-10-16T13:51:29.480687+0000","last_fullsized":"2023-10-16T13:51:29.480687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:23.540177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479799+0000","last_change":"2023-10-16T13:51:28.428287+0000","last_active":"2023-10-16T13:51:29.479799+0000","last_peered":"2023-10-16T13:51:29.479799+0000","last_clean":"2023-10-16T13:51:29.479799+0000","last_became_active":"2023-10-16T13:51:28.427992+0000","last_became_peered":"2023-10-16T13:51:28.427992+0000","last_unstale":"2023-10-16T13:51:29.479799+0000","last_undegraded":"2023-10-16T13:51:29.479799+0000","last_fullsized":"2023-10-16T13:51:29.479799+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:35.985411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436923+0000","last_change":"2023-10-16T13:51:27.428507+0000","last_active":"2023-10-16T13:51:29.436923+0000","last_peered":"2023-10-16T13:51:29.436923+0000","last_clean":"2023-10-16T13:51:29.436923+0000","last_became_active":"2023-10-16T13:51:27.428235+0000","last_became_peered":"2023-10-16T13:51:27.428235+0000","last_unstale":"2023-10-16T13:51:29.436923+0000","last_undegraded":"2023-10-16T13:51:29.436923+0000","last_fullsized":"2023-10-16T13:51:29.436923+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:20.168836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480697+0000","last_change":"2023-10-16T13:51:28.437640+0000","last_active":"2023-10-16T13:51:29.480697+0000","last_peered":"2023-10-16T13:51:29.480697+0000","last_clean":"2023-10-16T13:51:29.480697+0000","last_became_active":"2023-10-16T13:51:28.437352+0000","last_became_peered":"2023-10-16T13:51:28.437352+0000","last_unstale":"2023-10-16T13:51:29.480697+0000","last_undegraded":"2023-10-16T13:51:29.480697+0000","last_fullsized":"2023-10-16T13:51:29.480697+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:58.357330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481014+0000","last_change":"2023-10-16T13:51:27.428678+0000","last_active":"2023-10-16T13:51:29.481014+0000","last_peered":"2023-10-16T13:51:29.481014+0000","last_clean":"2023-10-16T13:51:29.481014+0000","last_became_active":"2023-10-16T13:51:27.428404+0000","last_became_peered":"2023-10-16T13:51:27.428404+0000","last_unstale":"2023-10-16T13:51:29.481014+0000","last_undegraded":"2023-10-16T13:51:29.481014+0000","last_fullsized":"2023-10-16T13:51:29.481014+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:51.504190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481892+0000","last_change":"2023-10-16T13:51:28.435960+0000","last_active":"2023-10-16T13:51:29.481892+0000","last_peered":"2023-10-16T13:51:29.481892+0000","last_clean":"2023-10-16T13:51:29.481892+0000","last_became_active":"2023-10-16T13:51:28.435670+0000","last_became_peered":"2023-10-16T13:51:28.435670+0000","last_unstale":"2023-10-16T13:51:29.481892+0000","last_undegraded":"2023-10-16T13:51:29.481892+0000","last_fullsized":"2023-10-16T13:51:29.481892+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:27.595646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479941+0000","last_change":"2023-10-16T13:51:27.431152+0000","last_active":"2023-10-16T13:51:29.479941+0000","last_peered":"2023-10-16T13:51:29.479941+0000","last_clean":"2023-10-16T13:51:29.479941+0000","last_became_active":"2023-10-16T13:51:27.430873+0000","last_became_peered":"2023-10-16T13:51:27.430873+0000","last_unstale":"2023-10-16T13:51:29.479941+0000","last_undegraded":"2023-10-16T13:51:29.479941+0000","last_fullsized":"2023-10-16T13:51:29.479941+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:24.148258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482140+0000","last_change":"2023-10-16T13:51:28.432755+0000","last_active":"2023-10-16T13:51:29.482140+0000","last_peered":"2023-10-16T13:51:29.482140+0000","last_clean":"2023-10-16T13:51:29.482140+0000","last_became_active":"2023-10-16T13:51:28.430015+0000","last_became_peered":"2023-10-16T13:51:28.430015+0000","last_unstale":"2023-10-16T13:51:29.482140+0000","last_undegraded":"2023-10-16T13:51:29.482140+0000","last_fullsized":"2023-10-16T13:51:29.482140+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.413170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481255+0000","last_change":"2023-10-16T13:51:27.427823+0000","last_active":"2023-10-16T13:51:29.481255+0000","last_peered":"2023-10-16T13:51:29.481255+0000","last_clean":"2023-10-16T13:51:29.481255+0000","last_became_active":"2023-10-16T13:51:27.427521+0000","last_became_peered":"2023-10-16T13:51:27.427521+0000","last_unstale":"2023-10-16T13:51:29.481255+0000","last_undegraded":"2023-10-16T13:51:29.481255+0000","last_fullsized":"2023-10-16T13:51:29.481255+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:56.092448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432357+0000","last_change":"2023-10-16T13:51:28.434499+0000","last_active":"2023-10-16T13:51:29.432357+0000","last_peered":"2023-10-16T13:51:29.432357+0000","last_clean":"2023-10-16T13:51:29.432357+0000","last_became_active":"2023-10-16T13:51:28.434188+0000","last_became_peered":"2023-10-16T13:51:28.434188+0000","last_unstale":"2023-10-16T13:51:29.432357+0000","last_undegraded":"2023-10-16T13:51:29.432357+0000","last_fullsized":"2023-10-16T13:51:29.432357+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:59.411175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.483083+0000","last_change":"2023-10-16T13:51:27.430504+0000","last_active":"2023-10-16T13:51:29.483083+0000","last_peered":"2023-10-16T13:51:29.483083+0000","last_clean":"2023-10-16T13:51:29.483083+0000","last_became_active":"2023-10-16T13:51:27.430238+0000","last_became_peered":"2023-10-16T13:51:27.430238+0000","last_unstale":"2023-10-16T13:51:29.483083+0000","last_undegraded":"2023-10-16T13:51:29.483083+0000","last_fullsized":"2023-10-16T13:51:29.483083+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:36:30.448522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432124+0000","last_change":"2023-10-16T13:51:28.433923+0000","last_active":"2023-10-16T13:51:29.432124+0000","last_peered":"2023-10-16T13:51:29.432124+0000","last_clean":"2023-10-16T13:51:29.432124+0000","last_became_active":"2023-10-16T13:51:28.433669+0000","last_became_peered":"2023-10-16T13:51:28.433669+0000","last_unstale":"2023-10-16T13:51:29.432124+0000","last_undegraded":"2023-10-16T13:51:29.432124+0000","last_fullsized":"2023-10-16T13:51:29.432124+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:30.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"196'3","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482610+0000","last_change":"2023-10-16T13:51:27.432160+0000","last_active":"2023-10-16T13:51:29.482610+0000","last_peered":"2023-10-16T13:51:29.482610+0000","last_clean":"2023-10-16T13:51:29.482610+0000","last_became_active":"2023-10-16T13:51:27.431870+0000","last_became_peered":"2023-10-16T13:51:27.431870+0000","last_unstale":"2023-10-16T13:51:29.482610+0000","last_undegraded":"2023-10-16T13:51:29.482610+0000","last_fullsized":"2023-10-16T13:51:29.482610+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:03:12.790107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437648+0000","last_change":"2023-10-16T13:51:28.441736+0000","last_active":"2023-10-16T13:51:29.437648+0000","last_peered":"2023-10-16T13:51:29.437648+0000","last_clean":"2023-10-16T13:51:29.437648+0000","last_became_active":"2023-10-16T13:51:28.441304+0000","last_became_peered":"2023-10-16T13:51:28.441304+0000","last_unstale":"2023-10-16T13:51:29.437648+0000","last_undegraded":"2023-10-16T13:51:29.437648+0000","last_fullsized":"2023-10-16T13:51:29.437648+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:58.120912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"196'6","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437637+0000","last_change":"2023-10-16T13:51:27.434373+0000","last_active":"2023-10-16T13:51:29.437637+0000","last_peered":"2023-10-16T13:51:29.437637+0000","last_clean":"2023-10-16T13:51:29.437637+0000","last_became_active":"2023-10-16T13:51:27.433377+0000","last_became_peered":"2023-10-16T13:51:27.433377+0000","last_unstale":"2023-10-16T13:51:29.437637+0000","last_undegraded":"2023-10-16T13:51:29.437637+0000","last_fullsized":"2023-10-16T13:51:29.437637+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:10.336518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431510+0000","last_change":"2023-10-16T13:51:28.437700+0000","last_active":"2023-10-16T13:51:29.431510+0000","last_peered":"2023-10-16T13:51:29.431510+0000","last_clean":"2023-10-16T13:51:29.431510+0000","last_became_active":"2023-10-16T13:51:28.437230+0000","last_became_peered":"2023-10-16T13:51:28.437230+0000","last_unstale":"2023-10-16T13:51:29.431510+0000","last_undegraded":"2023-10-16T13:51:29.431510+0000","last_fullsized":"2023-10-16T13:51:29.431510+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:50.438320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480953+0000","last_change":"2023-10-16T13:51:27.424287+0000","last_active":"2023-10-16T13:51:29.480953+0000","last_peered":"2023-10-16T13:51:29.480953+0000","last_clean":"2023-10-16T13:51:29.480953+0000","last_became_active":"2023-10-16T13:51:27.423990+0000","last_became_peered":"2023-10-16T13:51:27.423990+0000","last_unstale":"2023-10-16T13:51:29.480953+0000","last_undegraded":"2023-10-16T13:51:29.480953+0000","last_fullsized":"2023-10-16T13:51:29.480953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:12.034064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482421+0000","last_change":"2023-10-16T13:51:28.444996+0000","last_active":"2023-10-16T13:51:29.482421+0000","last_peered":"2023-10-16T13:51:29.482421+0000","last_clean":"2023-10-16T13:51:29.482421+0000","last_became_active":"2023-10-16T13:51:28.444735+0000","last_became_peered":"2023-10-16T13:51:28.444735+0000","last_unstale":"2023-10-16T13:51:29.482421+0000","last_undegraded":"2023-10-16T13:51:29.482421+0000","last_fullsized":"2023-10-16T13:51:29.482421+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:00.612037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431840+0000","last_change":"2023-10-16T13:51:27.426783+0000","last_active":"2023-10-16T13:51:29.431840+0000","last_peered":"2023-10-16T13:51:29.431840+0000","last_clean":"2023-10-16T13:51:29.431840+0000","last_became_active":"2023-10-16T13:51:27.426359+0000","last_became_peered":"2023-10-16T13:51:27.426359+0000","last_unstale":"2023-10-16T13:51:29.431840+0000","last_undegraded":"2023-10-16T13:51:29.431840+0000","last_fullsized":"2023-10-16T13:51:29.431840+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:38.237707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480508+0000","last_change":"2023-10-16T13:51:28.442605+0000","last_active":"2023-10-16T13:51:29.480508+0000","last_peered":"2023-10-16T13:51:29.480508+0000","last_clean":"2023-10-16T13:51:29.480508+0000","last_became_active":"2023-10-16T13:51:28.442339+0000","last_became_peered":"2023-10-16T13:51:28.442339+0000","last_unstale":"2023-10-16T13:51:29.480508+0000","last_undegraded":"2023-10-16T13:51:29.480508+0000","last_fullsized":"2023-10-16T13:51:29.480508+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:23.517975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479502+0000","last_change":"2023-10-16T13:51:27.436887+0000","last_active":"2023-10-16T13:51:29.479502+0000","last_peered":"2023-10-16T13:51:29.479502+0000","last_clean":"2023-10-16T13:51:29.479502+0000","last_became_active":"2023-10-16T13:51:27.436546+0000","last_became_peered":"2023-10-16T13:51:27.436546+0000","last_unstale":"2023-10-16T13:51:29.479502+0000","last_undegraded":"2023-10-16T13:51:29.479502+0000","last_fullsized":"2023-10-16T13:51:29.479502+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:02.187448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434839+0000","last_change":"2023-10-16T13:51:28.432571+0000","last_active":"2023-10-16T13:51:29.434839+0000","last_peered":"2023-10-16T13:51:29.434839+0000","last_clean":"2023-10-16T13:51:29.434839+0000","last_became_active":"2023-10-16T13:51:28.431454+0000","last_became_peered":"2023-10-16T13:51:28.431454+0000","last_unstale":"2023-10-16T13:51:29.434839+0000","last_undegraded":"2023-10-16T13:51:29.434839+0000","last_fullsized":"2023-10-16T13:51:29.434839+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:35.908243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"196'8","reported_seq":20,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.433910+0000","last_change":"2023-10-16T13:51:27.435137+0000","last_active":"2023-10-16T13:51:29.433910+0000","last_peered":"2023-10-16T13:51:29.433910+0000","last_clean":"2023-10-16T13:51:29.433910+0000","last_became_active":"2023-10-16T13:51:27.434797+0000","last_became_peered":"2023-10-16T13:51:27.434797+0000","last_unstale":"2023-10-16T13:51:29.433910+0000","last_undegraded":"2023-10-16T13:51:29.433910+0000","last_fullsized":"2023-10-16T13:51:29.433910+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:12.521516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480183+0000","last_change":"2023-10-16T13:51:28.438705+0000","last_active":"2023-10-16T13:51:29.480183+0000","last_peered":"2023-10-16T13:51:29.480183+0000","last_clean":"2023-10-16T13:51:29.480183+0000","last_became_active":"2023-10-16T13:51:28.432126+0000","last_became_peered":"2023-10-16T13:51:28.432126+0000","last_unstale":"2023-10-16T13:51:29.480183+0000","last_undegraded":"2023-10-16T13:51:29.480183+0000","last_fullsized":"2023-10-16T13:51:29.480183+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:10.530427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437372+0000","last_change":"2023-10-16T13:51:27.437255+0000","last_active":"2023-10-16T13:51:29.437372+0000","last_peered":"2023-10-16T13:51:29.437372+0000","last_clean":"2023-10-16T13:51:29.437372+0000","last_became_active":"2023-10-16T13:51:27.436838+0000","last_became_peered":"2023-10-16T13:51:27.436838+0000","last_unstale":"2023-10-16T13:51:29.437372+0000","last_undegraded":"2023-10-16T13:51:29.437372+0000","last_fullsized":"2023-10-16T13:51:29.437372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:32.418327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432665+0000","last_change":"2023-10-16T13:51:28.443763+0000","last_active":"2023-10-16T13:51:29.432665+0000","last_peered":"2023-10-16T13:51:29.432665+0000","last_clean":"2023-10-16T13:51:29.432665+0000","last_became_active":"2023-10-16T13:51:28.443519+0000","last_became_peered":"2023-10-16T13:51:28.443519+0000","last_unstale":"2023-10-16T13:51:29.432665+0000","last_undegraded":"2023-10-16T13:51:29.432665+0000","last_fullsized":"2023-10-16T13:51:29.432665+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:27.937378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436120+0000","last_change":"2023-10-16T13:51:27.435416+0000","last_active":"2023-10-16T13:51:29.436120+0000","last_peered":"2023-10-16T13:51:29.436120+0000","last_clean":"2023-10-16T13:51:29.436120+0000","last_became_active":"2023-10-16T13:51:27.435109+0000","last_became_peered":"2023-10-16T13:51:27.435109+0000","last_unstale":"2023-10-16T13:51:29.436120+0000","last_undegraded":"2023-10-16T13:51:29.436120+0000","last_fullsized":"2023-10-16T13:51:29.436120+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:08.005543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480698+0000","last_change":"2023-10-16T13:51:28.428089+0000","last_active":"2023-10-16T13:51:29.480698+0000","last_peered":"2023-10-16T13:51:29.480698+0000","last_clean":"2023-10-16T13:51:29.480698+0000","last_became_active":"2023-10-16T13:51:28.427825+0000","last_became_peered":"2023-10-16T13:51:28.427825+0000","last_unstale":"2023-10-16T13:51:29.480698+0000","last_undegraded":"2023-10-16T13:51:29.480698+0000","last_fullsized":"2023-10-16T13:51:29.480698+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:53.621443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"196'1","reported_seq":13,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481811+0000","last_change":"2023-10-16T13:51:27.440583+0000","last_active":"2023-10-16T13:51:29.481811+0000","last_peered":"2023-10-16T13:51:29.481811+0000","last_clean":"2023-10-16T13:51:29.481811+0000","last_became_active":"2023-10-16T13:51:27.440217+0000","last_became_peered":"2023-10-16T13:51:27.440217+0000","last_unstale":"2023-10-16T13:51:29.481811+0000","last_undegraded":"2023-10-16T13:51:29.481811+0000","last_fullsized":"2023-10-16T13:51:29.481811+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:40.748567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480026+0000","last_change":"2023-10-16T13:51:28.445885+0000","last_active":"2023-10-16T13:51:29.480026+0000","last_peered":"2023-10-16T13:51:29.480026+0000","last_clean":"2023-10-16T13:51:29.480026+0000","last_became_active":"2023-10-16T13:51:28.445639+0000","last_became_peered":"2023-10-16T13:51:28.445639+0000","last_unstale":"2023-10-16T13:51:29.480026+0000","last_undegraded":"2023-10-16T13:51:29.480026+0000","last_fullsized":"2023-10-16T13:51:29.480026+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:07.608491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481195+0000","last_change":"2023-10-16T13:51:27.436822+0000","last_active":"2023-10-16T13:51:29.481195+0000","last_peered":"2023-10-16T13:51:29.481195+0000","last_clean":"2023-10-16T13:51:29.481195+0000","last_became_active":"2023-10-16T13:51:27.436263+0000","last_became_peered":"2023-10-16T13:51:27.436263+0000","last_unstale":"2023-10-16T13:51:29.481195+0000","last_undegraded":"2023-10-16T13:51:29.481195+0000","last_fullsized":"2023-10-16T13:51:29.481195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:05.986789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434930+0000","last_change":"2023-10-16T13:51:28.438253+0000","last_active":"2023-10-16T13:51:29.434930+0000","last_peered":"2023-10-16T13:51:29.434930+0000","last_clean":"2023-10-16T13:51:29.434930+0000","last_became_active":"2023-10-16T13:51:28.437932+0000","last_became_peered":"2023-10-16T13:51:28.437932+0000","last_unstale":"2023-10-16T13:51:29.434930+0000","last_undegraded":"2023-10-16T13:51:29.434930+0000","last_fullsized":"2023-10-16T13:51:29.434930+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:05.621437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481195+0000","last_change":"2023-10-16T13:51:27.434212+0000","last_active":"2023-10-16T13:51:29.481195+0000","last_peered":"2023-10-16T13:51:29.481195+0000","last_clean":"2023-10-16T13:51:29.481195+0000","last_became_active":"2023-10-16T13:51:27.433935+0000","last_became_peered":"2023-10-16T13:51:27.433935+0000","last_unstale":"2023-10-16T13:51:29.481195+0000","last_undegraded":"2023-10-16T13:51:29.481195+0000","last_fullsized":"2023-10-16T13:51:29.481195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:31.315080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479964+0000","last_change":"2023-10-16T13:51:28.436205+0000","last_active":"2023-10-16T13:51:29.479964+0000","last_peered":"2023-10-16T13:51:29.479964+0000","last_clean":"2023-10-16T13:51:29.479964+0000","last_became_active":"2023-10-16T13:51:28.433734+0000","last_became_peered":"2023-10-16T13:51:28.433734+0000","last_unstale":"2023-10-16T13:51:29.479964+0000","last_undegraded":"2023-10-16T13:51:29.479964+0000","last_fullsized":"2023-10-16T13:51:29.479964+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:27.196301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480871+0000","last_change":"2023-10-16T13:51:27.435396+0000","last_active":"2023-10-16T13:51:29.480871+0000","last_peered":"2023-10-16T13:51:29.480871+0000","last_clean":"2023-10-16T13:51:29.480871+0000","last_became_active":"2023-10-16T13:51:27.434893+0000","last_became_peered":"2023-10-16T13:51:27.434893+0000","last_unstale":"2023-10-16T13:51:29.480871+0000","last_undegraded":"2023-10-16T13:51:29.480871+0000","last_fullsized":"2023-10-16T13:51:29.480871+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:01.583417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480273+0000","last_change":"2023-10-16T13:51:28.443268+0000","last_active":"2023-10-16T13:51:29.480273+0000","last_peered":"2023-10-16T13:51:29.480273+0000","last_clean":"2023-10-16T13:51:29.480273+0000","last_became_active":"2023-10-16T13:51:28.442911+0000","last_became_peered":"2023-10-16T13:51:28.442911+0000","last_unstale":"2023-10-16T13:51:29.480273+0000","last_undegraded":"2023-10-16T13:51:29.480273+0000","last_fullsized":"2023-10-16T13:51:29.480273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:37.097378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.479590+0000","last_change":"2023-10-16T13:51:27.423627+0000","last_active":"2023-10-16T13:51:29.479590+0000","last_peered":"2023-10-16T13:51:29.479590+0000","last_clean":"2023-10-16T13:51:29.479590+0000","last_became_active":"2023-10-16T13:51:27.422588+0000","last_became_peered":"2023-10-16T13:51:27.422588+0000","last_unstale":"2023-10-16T13:51:29.479590+0000","last_undegraded":"2023-10-16T13:51:29.479590+0000","last_fullsized":"2023-10-16T13:51:29.479590+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:10.336518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482424+0000","last_change":"2023-10-16T13:51:28.435886+0000","last_active":"2023-10-16T13:51:29.482424+0000","last_peered":"2023-10-16T13:51:29.482424+0000","last_clean":"2023-10-16T13:51:29.482424+0000","last_became_active":"2023-10-16T13:51:28.434528+0000","last_became_peered":"2023-10-16T13:51:28.434528+0000","last_unstale":"2023-10-16T13:51:29.482424+0000","last_undegraded":"2023-10-16T13:51:29.482424+0000","last_fullsized":"2023-10-16T13:51:29.482424+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:02:52.516835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480749+0000","last_change":"2023-10-16T13:51:27.438230+0000","last_active":"2023-10-16T13:51:29.480749+0000","last_peered":"2023-10-16T13:51:29.480749+0000","last_clean":"2023-10-16T13:51:29.480749+0000","last_became_active":"2023-10-16T13:51:27.437908+0000","last_became_peered":"2023-10-16T13:51:27.437908+0000","last_unstale":"2023-10-16T13:51:29.480749+0000","last_undegraded":"2023-10-16T13:51:29.480749+0000","last_fullsized":"2023-10-16T13:51:29.480749+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:46.259204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432214+0000","last_change":"2023-10-16T13:51:28.441896+0000","last_active":"2023-10-16T13:51:29.432214+0000","last_peered":"2023-10-16T13:51:29.432214+0000","last_clean":"2023-10-16T13:51:29.432214+0000","last_became_active":"2023-10-16T13:51:28.441622+0000","last_became_peered":"2023-10-16T13:51:28.441622+0000","last_unstale":"2023-10-16T13:51:29.432214+0000","last_undegraded":"2023-10-16T13:51:29.432214+0000","last_fullsized":"2023-10-16T13:51:29.432214+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:39.322635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"196'2","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432604+0000","last_change":"2023-10-16T13:51:27.437461+0000","last_active":"2023-10-16T13:51:29.432604+0000","last_peered":"2023-10-16T13:51:29.432604+0000","last_clean":"2023-10-16T13:51:29.432604+0000","last_became_active":"2023-10-16T13:51:27.437038+0000","last_became_peered":"2023-10-16T13:51:27.437038+0000","last_unstale":"2023-10-16T13:51:29.432604+0000","last_undegraded":"2023-10-16T13:51:29.432604+0000","last_fullsized":"2023-10-16T13:51:29.432604+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:11.972312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481476+0000","last_change":"2023-10-16T13:51:28.434053+0000","last_active":"2023-10-16T13:51:29.481476+0000","last_peered":"2023-10-16T13:51:29.481476+0000","last_clean":"2023-10-16T13:51:29.481476+0000","last_became_active":"2023-10-16T13:51:28.433767+0000","last_became_peered":"2023-10-16T13:51:28.433767+0000","last_unstale":"2023-10-16T13:51:29.481476+0000","last_undegraded":"2023-10-16T13:51:29.481476+0000","last_fullsized":"2023-10-16T13:51:29.481476+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:21.255420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436388+0000","last_change":"2023-10-16T13:51:27.425258+0000","last_active":"2023-10-16T13:51:29.436388+0000","last_peered":"2023-10-16T13:51:29.436388+0000","last_clean":"2023-10-16T13:51:29.436388+0000","last_became_active":"2023-10-16T13:51:27.424953+0000","last_became_peered":"2023-10-16T13:51:27.424953+0000","last_unstale":"2023-10-16T13:51:29.436388+0000","last_undegraded":"2023-10-16T13:51:29.436388+0000","last_fullsized":"2023-10-16T13:51:29.436388+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:39:29.760617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.482121+0000","last_change":"2023-10-16T13:51:28.435763+0000","last_active":"2023-10-16T13:51:29.482121+0000","last_peered":"2023-10-16T13:51:29.482121+0000","last_clean":"2023-10-16T13:51:29.482121+0000","last_became_active":"2023-10-16T13:51:28.435009+0000","last_became_peered":"2023-10-16T13:51:28.435009+0000","last_unstale":"2023-10-16T13:51:29.482121+0000","last_undegraded":"2023-10-16T13:51:29.482121+0000","last_fullsized":"2023-10-16T13:51:29.482121+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.486179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481696+0000","last_change":"2023-10-16T13:51:27.436782+0000","last_active":"2023-10-16T13:51:29.481696+0000","last_peered":"2023-10-16T13:51:29.481696+0000","last_clean":"2023-10-16T13:51:29.481696+0000","last_became_active":"2023-10-16T13:51:27.436480+0000","last_became_peered":"2023-10-16T13:51:27.436480+0000","last_unstale":"2023-10-16T13:51:29.481696+0000","last_undegraded":"2023-10-16T13:51:29.481696+0000","last_fullsized":"2023-10-16T13:51:29.481696+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:51.087164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431668+0000","last_change":"2023-10-16T13:51:28.435367+0000","last_active":"2023-10-16T13:51:29.431668+0000","last_peered":"2023-10-16T13:51:29.431668+0000","last_clean":"2023-10-16T13:51:29.431668+0000","last_became_active":"2023-10-16T13:51:28.435087+0000","last_became_peered":"2023-10-16T13:51:28.435087+0000","last_unstale":"2023-10-16T13:51:29.431668+0000","last_undegraded":"2023-10-16T13:51:29.431668+0000","last_fullsized":"2023-10-16T13:51:29.431668+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:50.781893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432375+0000","last_change":"2023-10-16T13:51:27.431128+0000","last_active":"2023-10-16T13:51:29.432375+0000","last_peered":"2023-10-16T13:51:29.432375+0000","last_clean":"2023-10-16T13:51:29.432375+0000","last_became_active":"2023-10-16T13:51:27.430856+0000","last_became_peered":"2023-10-16T13:51:27.430856+0000","last_unstale":"2023-10-16T13:51:29.432375+0000","last_undegraded":"2023-10-16T13:51:29.432375+0000","last_fullsized":"2023-10-16T13:51:29.432375+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:01.132629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.431713+0000","last_change":"2023-10-16T13:51:28.429069+0000","last_active":"2023-10-16T13:51:29.431713+0000","last_peered":"2023-10-16T13:51:29.431713+0000","last_clean":"2023-10-16T13:51:29.431713+0000","last_became_active":"2023-10-16T13:51:28.427120+0000","last_became_peered":"2023-10-16T13:51:28.427120+0000","last_unstale":"2023-10-16T13:51:29.431713+0000","last_undegraded":"2023-10-16T13:51:29.431713+0000","last_fullsized":"2023-10-16T13:51:29.431713+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:09.125941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437722+0000","last_change":"2023-10-16T13:51:27.431593+0000","last_active":"2023-10-16T13:51:29.437722+0000","last_peered":"2023-10-16T13:51:29.437722+0000","last_clean":"2023-10-16T13:51:29.437722+0000","last_became_active":"2023-10-16T13:51:27.431204+0000","last_became_peered":"2023-10-16T13:51:27.431204+0000","last_unstale":"2023-10-16T13:51:29.437722+0000","last_undegraded":"2023-10-16T13:51:29.437722+0000","last_fullsized":"2023-10-16T13:51:29.437722+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:15.612658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.437672+0000","last_change":"2023-10-16T13:51:28.446659+0000","last_active":"2023-10-16T13:51:29.437672+0000","last_peered":"2023-10-16T13:51:29.437672+0000","last_clean":"2023-10-16T13:51:29.437672+0000","last_became_active":"2023-10-16T13:51:28.446326+0000","last_became_peered":"2023-10-16T13:51:28.446326+0000","last_unstale":"2023-10-16T13:51:29.437672+0000","last_undegraded":"2023-10-16T13:51:29.437672+0000","last_fullsized":"2023-10-16T13:51:29.437672+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:23.802977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480299+0000","last_change":"2023-10-16T13:51:27.416141+0000","last_active":"2023-10-16T13:51:29.480299+0000","last_peered":"2023-10-16T13:51:29.480299+0000","last_clean":"2023-10-16T13:51:29.480299+0000","last_became_active":"2023-10-16T13:51:27.415768+0000","last_became_peered":"2023-10-16T13:51:27.415768+0000","last_unstale":"2023-10-16T13:51:29.480299+0000","last_undegraded":"2023-10-16T13:51:29.480299+0000","last_fullsized":"2023-10-16T13:51:29.480299+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:57.108267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.481011+0000","last_change":"2023-10-16T13:51:28.436388+0000","last_active":"2023-10-16T13:51:29.481011+0000","last_peered":"2023-10-16T13:51:29.481011+0000","last_clean":"2023-10-16T13:51:29.481011+0000","last_became_active":"2023-10-16T13:51:28.434606+0000","last_became_peered":"2023-10-16T13:51:28.434606+0000","last_unstale":"2023-10-16T13:51:29.481011+0000","last_undegraded":"2023-10-16T13:51:29.481011+0000","last_fullsized":"2023-10-16T13:51:29.481011+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:35.361959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.480695+0000","last_change":"2023-10-16T13:51:27.436529+0000","last_active":"2023-10-16T13:51:29.480695+0000","last_peered":"2023-10-16T13:51:29.480695+0000","last_clean":"2023-10-16T13:51:29.480695+0000","last_became_active":"2023-10-16T13:51:27.436162+0000","last_became_peered":"2023-10-16T13:51:27.436162+0000","last_unstale":"2023-10-16T13:51:29.480695+0000","last_undegraded":"2023-10-16T13:51:29.480695+0000","last_fullsized":"2023-10-16T13:51:29.480695+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:02.870854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.435193+0000","last_change":"2023-10-16T13:51:28.431455+0000","last_active":"2023-10-16T13:51:29.435193+0000","last_peered":"2023-10-16T13:51:29.435193+0000","last_clean":"2023-10-16T13:51:29.435193+0000","last_became_active":"2023-10-16T13:51:28.431138+0000","last_became_peered":"2023-10-16T13:51:28.431138+0000","last_unstale":"2023-10-16T13:51:29.435193+0000","last_undegraded":"2023-10-16T13:51:29.435193+0000","last_fullsized":"2023-10-16T13:51:29.435193+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:32:22.692101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.434037+0000","last_change":"2023-10-16T13:51:27.433370+0000","last_active":"2023-10-16T13:51:29.434037+0000","last_peered":"2023-10-16T13:51:29.434037+0000","last_clean":"2023-10-16T13:51:29.434037+0000","last_became_active":"2023-10-16T13:51:27.432932+0000","last_became_peered":"2023-10-16T13:51:27.432932+0000","last_unstale":"2023-10-16T13:51:29.434037+0000","last_undegraded":"2023-10-16T13:51:29.434037+0000","last_fullsized":"2023-10-16T13:51:29.434037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:00.236535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.432071+0000","last_change":"2023-10-16T13:51:28.443052+0000","last_active":"2023-10-16T13:51:29.432071+0000","last_peered":"2023-10-16T13:51:29.432071+0000","last_clean":"2023-10-16T13:51:29.432071+0000","last_became_active":"2023-10-16T13:51:28.442190+0000","last_became_peered":"2023-10-16T13:51:28.442190+0000","last_unstale":"2023-10-16T13:51:29.432071+0000","last_undegraded":"2023-10-16T13:51:29.432071+0000","last_fullsized":"2023-10-16T13:51:29.432071+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:27.383475+0000","last_clean_scrub_stamp":"2023-10-16T13:51:27.383475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:33.523038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-10-16T13:51:29.436607+0000","last_change":"2023-10-16T13:51:27.423465+0000","last_active":"2023-10-16T13:51:29.436607+0000","last_peered":"2023-10-16T13:51:29.436607+0000","last_clean":"2023-10-16T13:51:29.436607+0000","last_became_active":"2023-10-16T13:51:27.423042+0000","last_became_peered":"2023-10-16T13:51:27.423042+0000","last_unstale":"2023-10-16T13:51:29.436607+0000","last_undegraded":"2023-10-16T13:51:29.436607+0000","last_fullsized":"2023-10-16T13:51:29.436607+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:51:26.370831+0000","last_clean_scrub_stamp":"2023-10-16T13:51:26.370831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:45.456693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"185'296","reported_seq":990,"reported_epoch":196,"state":"active+clean","last_fresh":"2023-10-16T13:51:28.430329+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T13:51:28.430329+0000","last_peered":"2023-10-16T13:51:28.430329+0000","last_clean":"2023-10-16T13:51:28.430329+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T13:51:28.430329+0000","last_undegraded":"2023-10-16T13:51:28.430329+0000","last_fullsized":"2023-10-16T13:51:28.430329+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1675,"num_read_kb":3478,"num_write":483,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543074,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48060,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323780,"statfs":{"total":96636764160,"available":96587550720,"internally_reserved":0,"allocated":3063808,"data_stored":1778776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3320000000000001,"15min":1.327},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.73999999999999999},"max":{"1min":1.7729999999999999,"5min":1.835,"15min":1.976},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.423,"15min":1.407},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.302,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.6899999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1779999999999999,"15min":1.218},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6279999999999999,"5min":1.637,"15min":1.8100000000000001},"last":1.7509999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.321,"15min":1.28},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.67900000000000005},"max":{"1min":1.901,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.1850000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.276,"15min":1.274},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.5529999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.115},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3440000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7450000000000001,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.569}]},{"osd":3,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.399,"15min":1.385},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.8120000000000001,"5min":1.871,"15min":2.1070000000000002},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4710000000000001},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.84999999999999998},"max":{"1min":2.093,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.833}]},{"osd":4,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3,"15min":1.2609999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.976,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.397},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.446,"15min":1.399},"min":{"1min":0.96699999999999997,"5min":0.84399999999999997,"15min":0.81599999999999995},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":2.1829999999999998},"last":1.8680000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.4379999999999999,"15min":1.399},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.1880000000000002,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.272},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":1.0389999999999999,"5min":1.0369999999999999,"15min":0.94199999999999995},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":2.9700000000000002},"last":1.976}]},{"osd":6,"last update":"Mon Oct 16 13:50:46 2023","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.637,"15min":1.6180000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.2229999999999999,"5min":2.306,"15min":2.306},"last":1.927},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.532,"15min":1.524},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85199999999999998},"max":{"1min":1.8919999999999999,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.944}]}]},{"osd":6,"up_from":14,"seq":60129543075,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41896,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329944,"statfs":{"total":96636764160,"available":96593862656,"internally_reserved":0,"allocated":2449408,"data_stored":1186076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.349,"15min":1.3420000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.69599999999999995},"max":{"1min":1.748,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.2},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.8049999999999999,"5min":1.8169999999999999,"15min":1.9990000000000001},"last":1.3939999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.208,"15min":1.23},"min":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.82099999999999995},"max":{"1min":1.6299999999999999,"5min":1.6319999999999999,"15min":1.8280000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2270000000000001,"15min":1.232},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6890000000000001,"5min":1.776,"15min":1.788},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2090000000000001,"15min":1.254},"min":{"1min":0.89900000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.6990000000000001,"5min":1.897,"15min":2.0350000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.325,"15min":1.3240000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.72,"5min":1.77,"15min":1.859},"last":1.006}]},{"osd":3,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.3759999999999999},"min":{"1min":0.93999999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.764,"5min":1.946,"15min":1.9850000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4039999999999999,"15min":1.399},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":1.8200000000000001,"5min":1.8400000000000001,"15min":1.9159999999999999},"last":1.615}]},{"osd":4,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.395,"15min":1.3779999999999999},"min":{"1min":1.1020000000000001,"5min":0.86799999999999999,"15min":0.79600000000000004},"max":{"1min":1.7529999999999999,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.542,"15min":1.5369999999999999},"min":{"1min":1.3440000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.02,"5min":2.02,"15min":2.1219999999999999},"last":1.665}]},{"osd":5,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.359,"15min":1.331},"min":{"1min":0.876,"5min":0.86899999999999999,"15min":0.82699999999999996},"max":{"1min":1.839,"5min":1.903,"15min":2.2669999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.498,"15min":1.472},"min":{"1min":0.91500000000000004,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.895,"5min":1.9690000000000001,"15min":2.0510000000000002},"last":1.3500000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:50:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.4810000000000001},"min":{"1min":1.2190000000000001,"5min":0.88700000000000001,"15min":0.87},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.173},"last":1.548},{"interface":"front","average":{"1min":1.546,"5min":1.548,"15min":1.512},"min":{"1min":1.1779999999999999,"5min":1.0129999999999999,"15min":0.92500000000000004},"max":{"1min":1.923,"5min":1.9850000000000001,"15min":2.1419999999999999},"last":1.246}]}]},{"osd":5,"up_from":14,"seq":60129543076,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41900,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329940,"statfs":{"total":96636764160,"available":96593858560,"internally_reserved":0,"allocated":2453504,"data_stored":1186658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2290000000000001,"15min":1.2070000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.821},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.256,"15min":1.2729999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.746,"5min":1.839,"15min":1.9610000000000001},"last":1.0800000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.8460000000000001,"5min":1.992,"15min":2.1640000000000001},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.228,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.76000000000000001},"max":{"1min":1.6279999999999999,"5min":1.754,"15min":2.431},"last":0.88600000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.337,"15min":1.3560000000000001},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":1.879},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.327,"15min":1.3300000000000001},"min":{"1min":0.93500000000000005,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.2669999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.3839999999999999,"15min":1.401},"min":{"1min":1.046,"5min":0.98299999999999998,"15min":0.95499999999999996},"max":{"1min":1.827,"5min":2.0990000000000002,"15min":2.5289999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.4410000000000001,"15min":1.446},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.762,"5min":1.782,"15min":2.242},"last":1.7709999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.329,"15min":1.294},"min":{"1min":0.88100000000000001,"5min":0.82399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7030000000000001,"5min":1.782,"15min":2.3679999999999999},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.4650000000000001,"15min":1.429},"min":{"1min":1.125,"5min":0.81599999999999995,"15min":0.80300000000000005},"max":{"1min":1.778,"5min":2.2530000000000001,"15min":2.4020000000000001},"last":1.6899999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4419999999999999,"15min":1.391},"min":{"1min":1.071,"5min":0.91400000000000003,"15min":0.88},"max":{"1min":1.952,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.4950000000000001,"15min":1.4870000000000001},"min":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":0.97299999999999998},"max":{"1min":1.9119999999999999,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.849}]},{"osd":7,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.5760000000000001,"15min":1.5409999999999999},"min":{"1min":1.238,"5min":1.0269999999999999,"15min":0.93300000000000005},"max":{"1min":2.1600000000000001,"5min":2.371,"15min":2.6320000000000001},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.613,"15min":1.6000000000000001},"min":{"1min":1.325,"5min":1.0640000000000001,"15min":1.0089999999999999},"max":{"1min":1.8049999999999999,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.6459999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543076,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55796,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316044,"statfs":{"total":96636764160,"available":96579629056,"internally_reserved":0,"allocated":2461696,"data_stored":1187736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2470000000000001,"15min":1.2350000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.242},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1699999999999999,"15min":1.212},"min":{"1min":0.83999999999999997,"5min":0.82399999999999995,"15min":0.77900000000000003},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2549999999999999,"15min":1.2589999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.7350000000000003,"5min":4.7350000000000003,"15min":4.7350000000000003},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.3,"5min":1.28,"15min":1.3080000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":3.746,"5min":3.746,"15min":3.746},"last":1.6240000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.3320000000000001,"15min":1.3129999999999999},"min":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.88400000000000001},"max":{"1min":5.2599999999999998,"5min":5.2599999999999998,"15min":5.2599999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.3899999999999999,"15min":1.4099999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":3.9100000000000001,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":1.7629999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.304,"15min":1.3149999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.458,"15min":1.4399999999999999},"min":{"1min":1.1200000000000001,"5min":0.995,"15min":0.90600000000000003},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":5.3949999999999996},"last":1.427}]},{"osd":5,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.3720000000000001,"15min":1.3320000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.1280000000000001,"5min":4.1280000000000001,"15min":4.1280000000000001},"last":1.861},{"interface":"front","average":{"1min":1.419,"5min":1.3640000000000001,"15min":1.329},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.752},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":1.3320000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.395,"15min":1.399},"min":{"1min":0.875,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":5.3520000000000003,"5min":5.3520000000000003,"15min":5.3520000000000003},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.5229999999999999,"15min":1.52},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":5.0460000000000003,"5min":5.0460000000000003,"15min":5.0460000000000003},"last":1.9490000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:50:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.48,"15min":1.4970000000000001},"min":{"1min":1.0760000000000001,"5min":0.92800000000000005,"15min":0.92600000000000005},"max":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.712,"5min":1.639,"15min":1.6459999999999999},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":5.6200000000000001,"5min":5.6200000000000001,"15min":5.6200000000000001},"last":1.7130000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543076,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51628,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320212,"statfs":{"total":96636764160,"available":96583897088,"internally_reserved":0,"allocated":2449408,"data_stored":1186088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95899999999999996,"15min":0.97299999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3080000000000001,"5min":2.04,"15min":3.1000000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.2090000000000001,"15min":1.2310000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5309999999999999,"5min":1.9179999999999999,"15min":2.714},"last":1.2709999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.1240000000000001,"15min":1.099},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.52,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.125,"15min":1.133},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.649,"5min":1.879,"15min":1.879},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.1319999999999999,"15min":1.1910000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.0609999999999999,"5min":2.234,"15min":2.782},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.264,"5min":1.4219999999999999,"15min":1.4670000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.752,"5min":2.1400000000000001,"15min":2.609},"last":1.113}]},{"osd":4,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.486,"15min":1.5209999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.8959999999999999,"5min":1.8979999999999999,"15min":2.2330000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.409,"15min":1.4630000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.8660000000000001,"5min":1.9710000000000001,"15min":2.1000000000000001},"last":1.036}]},{"osd":5,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.429,"15min":1.454},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.6779999999999999,"5min":2.1110000000000002,"15min":2.306},"last":1.165},{"interface":"front","average":{"1min":1.427,"5min":1.546,"15min":1.534},"min":{"1min":1.139,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":1.7909999999999999,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.3740000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4850000000000001,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8029999999999999,"5min":2.1389999999999998,"15min":2.4430000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.4850000000000001,"15min":1.4990000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.387},"last":1.2470000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:51:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.6339999999999999,"15min":1.6200000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.9690000000000001,"5min":2.2730000000000001,"15min":2.3340000000000001},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.6759999999999999,"15min":1.679},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":2.0110000000000001,"5min":2.387,"15min":2.387},"last":1.6379999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543074,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51620,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320220,"statfs":{"total":96636764160,"available":96583905280,"internally_reserved":0,"allocated":2449408,"data_stored":1186144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3480000000000001,"5min":1.732,"15min":2.1499999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.125,"15min":1.159},"min":{"1min":0.76200000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.367,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.141,"15min":1.151},"min":{"1min":0.75,"5min":0.75,"15min":0.72199999999999998},"max":{"1min":1.532,"5min":1.956,"15min":2.4550000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.26,"5min":1.2809999999999999,"15min":1.286},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.754,"5min":1.804,"15min":2.3490000000000002},"last":0.81599999999999995}]},{"osd":3,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1339999999999999,"15min":1.1519999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.806,"5min":1.849,"15min":1.982},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.345,"5min":1.359,"15min":1.379},"min":{"1min":0.98799999999999999,"5min":0.91200000000000003,"15min":0.90700000000000003},"max":{"1min":1.7090000000000001,"5min":1.855,"15min":2.7970000000000002},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4950000000000001,"15min":1.5069999999999999},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.054},"max":{"1min":1.9430000000000001,"5min":2.1429999999999998,"15min":2.6589999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.452,"5min":1.492,"15min":1.514},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":1.762,"5min":2.0510000000000002,"15min":2.2629999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.575},"min":{"1min":1.129,"5min":1.129,"15min":1.056},"max":{"1min":1.921,"5min":2.101,"15min":2.5569999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.5009999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.7490000000000001,"5min":2.2650000000000001,"15min":2.899},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5549999999999999},"min":{"1min":1.0760000000000001,"5min":1.032,"15min":1.032},"max":{"1min":1.843,"5min":2.4300000000000002,"15min":3.0339999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.6080000000000001,"15min":1.637},"min":{"1min":1.145,"5min":1.145,"15min":1.145},"max":{"1min":1.982,"5min":2.0779999999999998,"15min":3.0790000000000002},"last":0.626}]},{"osd":7,"last update":"Mon Oct 16 13:50:40 2023","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.7050000000000001,"15min":1.6919999999999999},"min":{"1min":1.24,"5min":1.127,"15min":1.127},"max":{"1min":2.1880000000000002,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.651,"5min":1.643,"15min":1.661},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2429999999999999,"5min":2.5529999999999999,"15min":3.1320000000000001},"last":0.73399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47472,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324368,"statfs":{"total":96636764160,"available":96588152832,"internally_reserved":0,"allocated":2461696,"data_stored":1192388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.92900000000000005,"15min":0.94699999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1619999999999999,"15min":1.1759999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4930000000000001,"5min":1.8,"15min":1.8},"last":1.135}]},{"osd":2,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.0249999999999999,"15min":1.0289999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.6559999999999999,"5min":1.71,"15min":1.8069999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.2190000000000001,"15min":1.2470000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.373,"5min":1.992,"15min":1.994},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.385,"15min":1.3899999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.6220000000000001,"5min":1.883,"15min":1.9610000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.677,"5min":2.1269999999999998,"15min":2.2610000000000001},"last":1.044}]},{"osd":4,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.3520000000000001,"15min":1.339},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.631,"5min":1.867,"15min":2.1539999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.49,"15min":1.506},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.71899999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.3779999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.5,"15min":1.5009999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.0630000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.776},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4710000000000001,"15min":1.4670000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8069999999999999,"5min":1.9770000000000001,"15min":2.028},"last":1.343}]},{"osd":6,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.615,"15min":1.611},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.6120000000000001,"15min":1.605},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.76900000000000002},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":1.706}]},{"osd":7,"last update":"Mon Oct 16 13:50:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.732,"15min":1.704},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1949999999999998,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.611,"15min":1.6040000000000001},"min":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"max":{"1min":1.9550000000000001,"5min":2.1040000000000001,"15min":2.3140000000000001},"last":1.522}]}]},{"osd":0,"up_from":14,"seq":60129543075,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60520,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311320,"statfs":{"total":96636764160,"available":96574791680,"internally_reserved":0,"allocated":3039232,"data_stored":1776422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":0.92600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.399,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1479999999999999,"15min":1.1579999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":3.0870000000000002,"5min":3.0870000000000002,"15min":3.0870000000000002},"last":1.0269999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.1519999999999999,"15min":1.161},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.498,"5min":1.7010000000000001,"15min":1.9830000000000001},"last":1.385},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2250000000000001,"15min":1.248},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.155}]},{"osd":3,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.665,"5min":2.3340000000000001,"15min":3.0350000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3460000000000001,"15min":1.3640000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.613,"5min":2.2450000000000001,"15min":2.9670000000000001},"last":1.278}]},{"osd":4,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.365,"15min":1.393},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.6379999999999999,"5min":1.76,"15min":2.8740000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.462,"15min":1.5029999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9119999999999999,"5min":2.0150000000000001,"15min":4.6159999999999997},"last":1.47}]},{"osd":5,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.528,"15min":1.5269999999999999},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":1.615},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.4419999999999999,"15min":1.446},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.7969999999999999,"5min":2.0880000000000001,"15min":3.1589999999999998},"last":1.7390000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.613,"15min":1.6240000000000001},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":2.117,"5min":2.2610000000000001,"15min":4.0270000000000001},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5880000000000001,"15min":1.605},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.992,"5min":2.992,"15min":2.992},"last":1.6919999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:50:58 2023","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.704,"15min":1.7110000000000001},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.6200000000000001,"15min":1.6180000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.081,"5min":2.286,"15min":3.3010000000000002},"last":1.8520000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:51:33.504 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:51:33.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:51:33.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.785+0000 7fca6a5ce640 1 Processor -- start 2023-10-16T13:51:33.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.785+0000 7fca6a5ce640 1 -- start start 2023-10-16T13:51:33.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.786+0000 7fca6a5ce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca640b1840 0x7fca640b0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.786+0000 7fca6a5ce640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b1c50 con 0x7fca640cdb00 2023-10-16T13:51:33.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.786+0000 7fca6a5ce640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b07b0 con 0x7fca6415f4c0 2023-10-16T13:51:33.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.786+0000 7fca6a5ce640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca640ccb50 con 0x7fca640b1840 2023-10-16T13:51:33.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca68b6b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca640b1840 0x7fca640b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca637fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fca640cdb00 0x7fca640ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53918/0 (socket says 172.21.15.60:53918) 2023-10-16T13:51:33.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca637fe640 1 -- 172.21.15.60:0/1369449621 learned_addr learned my addr 172.21.15.60:0/1369449621 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:33.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca68b6b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca640b1840 0x7fca640b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55048/0 (socket says 172.21.15.60:55048) 2023-10-16T13:51:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca68b6b640 1 -- 172.21.15.60:0/1369449621 >> v1:172.21.15.144:6789/0 conn(0x7fca6415f4c0 legacy=0x7fca640cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:33.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca68b6b640 1 -- 172.21.15.60:0/1369449621 >> v1:172.21.15.60:6789/0 conn(0x7fca640cdb00 legacy=0x7fca640ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca68b6b640 1 -- 172.21.15.60:0/1369449621 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640b15f0 con 0x7fca640b1840 2023-10-16T13:51:33.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.787+0000 7fca62ffd640 1 -- 172.21.15.60:0/1369449621 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975096627 0 0) 0x7fca640b1c50 con 0x7fca640cdb00 2023-10-16T13:51:33.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.788+0000 7fca68b6b640 1 --2- 172.21.15.60:0/1369449621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640b0270 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0097a0 tx=0x7fca5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e84c4ade3069750 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:33.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca62ffd640 1 -- 172.21.15.60:0/1369449621 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c031830 con 0x7fca640b1840 2023-10-16T13:51:33.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca62ffd640 1 -- 172.21.15.60:0/1369449621 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c031990 con 0x7fca640b1840 2023-10-16T13:51:33.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca62ffd640 1 -- 172.21.15.60:0/1369449621 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c031d20 con 0x7fca640b1840 2023-10-16T13:51:33.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca6a5ce640 1 -- 172.21.15.60:0/1369449621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 msgr2=0x7fca640b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/1369449621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640b0270 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0097a0 tx=0x7fca5c02b8d0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca6a5ce640 1 -- 172.21.15.60:0/1369449621 shutdown_connections 2023-10-16T13:51:33.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.789+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/1369449621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640b0270 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.790+0000 7fca6a5ce640 1 -- 172.21.15.60:0/1369449621 >> 172.21.15.60:0/1369449621 conn(0x7fca64099ea0 msgr2=0x7fca640d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:33.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.790+0000 7fca6a5ce640 1 -- 172.21.15.60:0/1369449621 shutdown_connections 2023-10-16T13:51:33.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.790+0000 7fca6a5ce640 1 -- 172.21.15.60:0/1369449621 wait complete. 2023-10-16T13:51:33.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.791+0000 7fca6a5ce640 1 Processor -- start 2023-10-16T13:51:33.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.792+0000 7fca6a5ce640 1 -- start start 2023-10-16T13:51:33.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca6a5ce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca6a5ce640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca640cdb00 0x7fca6415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca6a5ce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca6415f4c0 0x7fca640b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca6a5ce640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca640ca590 con 0x7fca640b1840 2023-10-16T13:51:33.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca6a5ce640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca640c39a0 con 0x7fca640cdb00 2023-10-16T13:51:33.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca6a5ce640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca640c8fb0 con 0x7fca6415f4c0 2023-10-16T13:51:33.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640ca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55050/0 (socket says 172.21.15.60:55050) 2023-10-16T13:51:33.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 -- 172.21.15.60:0/3948696684 learned_addr learned my addr 172.21.15.60:0/3948696684 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:33.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 -- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca6415f4c0 msgr2=0x7fca640b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca6415f4c0 0x7fca640b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 -- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca640cdb00 msgr2=0x7fca6415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:33.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca637fe640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca640cdb00 0x7fca6415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca640cdb00 0x7fca6415afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:33.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.793+0000 7fca63fff640 1 -- 172.21.15.60:0/3948696684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c009400 con 0x7fca640b1840 2023-10-16T13:51:33.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.794+0000 7fca63fff640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640ca050 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fca5400ba50 tx=0x7fca5400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:33.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.794+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5400cbe0 con 0x7fca640b1840 2023-10-16T13:51:33.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.794+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5400cd80 con 0x7fca640b1840 2023-10-16T13:51:33.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.794+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54014680 con 0x7fca640b1840 2023-10-16T13:51:33.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.794+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca640c9290 con 0x7fca640b1840 2023-10-16T13:51:33.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.794+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca640c9850 con 0x7fca640b1840 2023-10-16T13:51:33.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.795+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6404e160 con 0x7fca640b1840 2023-10-16T13:51:33.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.796+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fca540147e0 con 0x7fca640b1840 2023-10-16T13:51:33.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.797+0000 7fca60ff9640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fca4406af90 0x7fca4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:33.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.797+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca54089410 con 0x7fca640b1840 2023-10-16T13:51:33.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.798+0000 7fca637fe640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fca4406af90 0x7fca4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:33.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.799+0000 7fca637fe640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fca4406af90 0x7fca4406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007fd0 tx=0x7fca4c009040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:33.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:33.802+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca540544f0 con 0x7fca640b1840 2023-10-16T13:51:34.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.074+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fca640d1a30 con 0x7fca640b1840 2023-10-16T13:51:34.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.078+0000 7fca60ff9640 1 -- 172.21.15.60:0/3948696684 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v48) v1 ==== 190+0+147 (secure 0 0 0) 0x7fca54057b10 con 0x7fca640b1840 2023-10-16T13:51:34.082 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T13:51:34.082 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T13:51:34.082 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T13:51:34.083 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T13:51:34.083 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:51:34.083 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T13:51:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fca4406af90 msgr2=0x7fca4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fca4406af90 0x7fca4406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007fd0 tx=0x7fca4c009040 comp rx=0 tx=0).stop 2023-10-16T13:51:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 msgr2=0x7fca640ca050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640ca050 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fca5400ba50 tx=0x7fca5400bf10 comp rx=0 tx=0).stop 2023-10-16T13:51:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 shutdown_connections 2023-10-16T13:51:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fca4406af90 0x7fca4406d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca6415f4c0 0x7fca640b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca640cdb00 0x7fca6415afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 --2- 172.21.15.60:0/3948696684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca640b1840 0x7fca640ca050 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 >> 172.21.15.60:0/3948696684 conn(0x7fca64099ea0 msgr2=0x7fca6406e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 shutdown_connections 2023-10-16T13:51:34.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.081+0000 7fca6a5ce640 1 -- 172.21.15.60:0/3948696684 wait complete. 2023-10-16T13:51:34.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:51:34.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.329+0000 7f3e32bf4640 1 Processor -- start 2023-10-16T13:51:34.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.329+0000 7f3e32bf4640 1 -- start start 2023-10-16T13:51:34.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.331+0000 7f3e32bf4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3e2c0ad970 0x7f3e2c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.331+0000 7f3e32bf4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c15b9d0 con 0x7f3e2c0ccfb0 2023-10-16T13:51:34.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.331+0000 7f3e32bf4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c15bcb0 con 0x7f3e2c06c040 2023-10-16T13:51:34.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.331+0000 7f3e32bf4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c15bf90 con 0x7f3e2c0ad970 2023-10-16T13:51:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.331+0000 7f3e323f3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3e2c0ccfb0 0x7f3e2c0cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53920/0 (socket says 172.21.15.60:53920) 2023-10-16T13:51:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.331+0000 7f3e323f3640 1 -- 172.21.15.60:0/1154013382 learned_addr learned my addr 172.21.15.60:0/1154013382 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e313f1640 1 --2- 172.21.15.60:0/1154013382 >> v2:172.21.15.144:3300/0 conn(0x7f3e2c0ad970 0x7f3e2c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187008035 0 0) 0x7f3e2c15b9d0 con 0x7f3e2c0ccfb0 2023-10-16T13:51:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e18003520 con 0x7f3e2c0ccfb0 2023-10-16T13:51:34.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878452345 0 0) 0x7f3e18003520 con 0x7f3e2c0ccfb0 2023-10-16T13:51:34.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 >> v2:172.21.15.144:3300/0 conn(0x7f3e2c0ad970 msgr2=0x7f3e2c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:34.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 --2- 172.21.15.60:0/1154013382 >> v2:172.21.15.144:3300/0 conn(0x7f3e2c0ad970 0x7f3e2c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 >> v1:172.21.15.144:6789/0 conn(0x7f3e2c06c040 legacy=0x7f3e2c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0cc270 con 0x7f3e2c0ccfb0 2023-10-16T13:51:34.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f3e240019f0 con 0x7f3e2c0ccfb0 2023-10-16T13:51:34.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 >> v1:172.21.15.60:6789/0 conn(0x7f3e2c0ccfb0 legacy=0x7f3e2c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e180039c0 0x7f3e18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e180047c0 0x7f3e18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e18005180 0x7f3e1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c15bf90 con 0x7f3e18005180 2023-10-16T13:51:34.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1800c080 con 0x7f3e180039c0 2023-10-16T13:51:34.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.332+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1800c240 con 0x7f3e180047c0 2023-10-16T13:51:34.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e323f3640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e18005180 0x7f3e1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e31bf2640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e180039c0 0x7f3e18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e323f3640 1 -- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e180047c0 msgr2=0x7f3e18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:34.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e323f3640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e180047c0 0x7f3e18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e323f3640 1 -- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e180039c0 msgr2=0x7f3e18003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:34.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e323f3640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e180039c0 0x7f3e18003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e323f3640 1 -- 172.21.15.60:0/1154013382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18003520 con 0x7f3e18005180 2023-10-16T13:51:34.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.333+0000 7f3e31bf2640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e180039c0 0x7f3e18003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:34.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.334+0000 7f3e323f3640 1 -- 172.21.15.60:0/1154013382 reap_dead start 2023-10-16T13:51:34.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.334+0000 7f3e323f3640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e18005180 0x7f3e1800b980 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400a760 tx=0x7f3e180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a8ba47241af613 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:34.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24034630 con 0x7f3e18005180 2023-10-16T13:51:34.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2402f070 con 0x7f3e18005180 2023-10-16T13:51:34.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e30bf0640 1 -- 172.21.15.60:0/1154013382 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24034a10 con 0x7f3e18005180 2023-10-16T13:51:34.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1154013382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e18005180 msgr2=0x7f3e1800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e18005180 0x7f3e1800b980 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400a760 tx=0x7f3e180057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1154013382 shutdown_connections 2023-10-16T13:51:34.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1154013382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e18005180 0x7f3e1800b980 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1154013382 >> 172.21.15.60:0/1154013382 conn(0x7f3e2c0ce6a0 msgr2=0x7f3e2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:34.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1154013382 shutdown_connections 2023-10-16T13:51:34.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.335+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1154013382 wait complete. 2023-10-16T13:51:34.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.337+0000 7f3e32bf4640 1 Processor -- start 2023-10-16T13:51:34.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.337+0000 7f3e32bf4640 1 -- start start 2023-10-16T13:51:34.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.338+0000 7f3e32bf4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e18005180 0x7f3e2c1579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.338+0000 7f3e32bf4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 0x7f3e2c15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e32bf4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e2c15c690 0x7f3e2c15cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e32bf4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c153ff0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e32bf4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0ae7c0 con 0x7f3e18005180 2023-10-16T13:51:34.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e32bf4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c15ac80 con 0x7f3e2c15c690 2023-10-16T13:51:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 0x7f3e2c15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 0x7f3e2c15a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55076/0 (socket says 172.21.15.60:55076) 2023-10-16T13:51:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 -- 172.21.15.60:0/1744356546 learned_addr learned my addr 172.21.15.60:0/1744356546 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e323f3640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e2c15c690 0x7f3e2c15cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e31bf2640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e18005180 0x7f3e2c1579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 -- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e2c15c690 msgr2=0x7f3e2c15cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e2c15c690 0x7f3e2c15cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 -- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e18005180 msgr2=0x7f3e2c1579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e18005180 0x7f3e2c1579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e313f1640 1 -- 172.21.15.60:0/1744356546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c15e0f0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e323f3640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e2c15c690 0x7f3e2c15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:34.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.339+0000 7f3e31bf2640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e18005180 0x7f3e2c1579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:34.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.340+0000 7f3e313f1640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 0x7f3e2c15a740 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00e7c0 tx=0x7f3e1c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:34.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.340+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00eed0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.340+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c00c8c0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.340+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00cc50 con 0x7f3e2c15dc90 2023-10-16T13:51:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.340+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c0ae200 con 0x7f3e2c15dc90 2023-10-16T13:51:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.340+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c159f40 con 0x7f3e2c15dc90 2023-10-16T13:51:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.341+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c06b330 con 0x7f3e2c15dc90 2023-10-16T13:51:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.342+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e1c013470 con 0x7f3e2c15dc90 2023-10-16T13:51:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.343+0000 7f3e227fc640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3e1006afc0 0x7f3e1006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.343+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e1c089c40 con 0x7f3e2c15dc90 2023-10-16T13:51:34.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.344+0000 7f3e31bf2640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3e1006afc0 0x7f3e1006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:34.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.345+0000 7f3e31bf2640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3e1006afc0 0x7f3e1006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e280016f0 tx=0x7f3e28009040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:34.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.348+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e1c0585d0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.593+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e2c159da0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.594+0000 7f3e227fc640 1 -- 172.21.15.60:0/1744356546 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f3e2c159da0 con 0x7f3e2c15dc90 2023-10-16T13:51:34.597 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T13:51:34.598 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:51:27.881990+0000","modified":"2023-10-16T13:51:30.974922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9966":{"gid":9966,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/3850041246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3850041246},{"type":"v1","addr":"172.21.15.60:6835","nonce":3850041246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:51:34.598 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 95 2023-10-16T13:51:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3e1006afc0 msgr2=0x7f3e1006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3e1006afc0 0x7f3e1006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e280016f0 tx=0x7f3e28009040 comp rx=0 tx=0).stop 2023-10-16T13:51:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 msgr2=0x7f3e2c15a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 0x7f3e2c15a740 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00e7c0 tx=0x7f3e1c00ec80 comp rx=0 tx=0).stop 2023-10-16T13:51:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 shutdown_connections 2023-10-16T13:51:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3e1006afc0 0x7f3e1006d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e2c15c690 0x7f3e2c15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e2c15dc90 0x7f3e2c15a740 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 --2- 172.21.15.60:0/1744356546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e18005180 0x7f3e2c1579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.597+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 >> 172.21.15.60:0/1744356546 conn(0x7f3e2c0ce6a0 msgr2=0x7f3e2c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:34.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.598+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 shutdown_connections 2023-10-16T13:51:34.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:34.598+0000 7f3e32bf4640 1 -- 172.21.15.60:0/1744356546 wait complete. 2023-10-16T13:51:35.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:51:27.881990+0000', 'modified': '2023-10-16T13:51:30.974922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9966}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9966': {'gid': 9966, 'name': 'c', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/3850041246', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 3850041246}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 3850041246}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:51:35.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:51:35.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T13:51:35.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.261+0000 7fab414d1640 1 Processor -- start 2023-10-16T13:51:35.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.261+0000 7fab414d1640 1 -- start start 2023-10-16T13:51:35.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.262+0000 7fab414d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fab3c0b8480 0x7fab3c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.262+0000 7fab414d1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c15d150 con 0x7fab3c15cd70 2023-10-16T13:51:35.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.262+0000 7fab414d1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c0d1180 con 0x7fab3c0d0be0 2023-10-16T13:51:35.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.262+0000 7fab414d1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3c0d1460 con 0x7fab3c0b8480 2023-10-16T13:51:35.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fab3c15cd70 0x7fab3c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53936/0 (socket says 172.21.15.60:53936) 2023-10-16T13:51:35.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3bfff640 1 -- 172.21.15.60:0/767073856 learned_addr learned my addr 172.21.15.60:0/767073856 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:35.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/767073856 >> v2:172.21.15.144:3300/0 conn(0x7fab3c0b8480 0x7fab3c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261492458 0 0) 0x7fab3c15d150 con 0x7fab3c15cd70 2023-10-16T13:51:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab28003520 con 0x7fab3c15cd70 2023-10-16T13:51:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815382604 0 0) 0x7fab28003520 con 0x7fab3c15cd70 2023-10-16T13:51:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 >> v2:172.21.15.144:3300/0 conn(0x7fab3c0b8480 msgr2=0x7fab3c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3affd640 1 --2- 172.21.15.60:0/767073856 >> v2:172.21.15.144:3300/0 conn(0x7fab3c0b8480 0x7fab3c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.263+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 >> v1:172.21.15.144:6789/0 conn(0x7fab3c0d0be0 legacy=0x7fab3c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c0d1740 con 0x7fab3c15cd70 2023-10-16T13:51:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fab2c0019f0 con 0x7fab3c15cd70 2023-10-16T13:51:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 >> v1:172.21.15.60:6789/0 conn(0x7fab3c15cd70 legacy=0x7fab3c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab280039c0 0x7fab28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab280047c0 0x7fab28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280051d0 0x7fab2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0d1460 con 0x7fab280039c0 2023-10-16T13:51:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2800c100 con 0x7fab280047c0 2023-10-16T13:51:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2800c2c0 con 0x7fab280051d0 2023-10-16T13:51:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3bfff640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab280039c0 0x7fab28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab280047c0 0x7fab28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3bfff640 1 -- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280051d0 msgr2=0x7fab2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab40cd0640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280051d0 0x7fab2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3bfff640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280051d0 0x7fab2800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3bfff640 1 -- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab280047c0 msgr2=0x7fab28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3bfff640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab280047c0 0x7fab28004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.264+0000 7fab3bfff640 1 -- 172.21.15.60:0/767073856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab28003520 con 0x7fab280039c0 2023-10-16T13:51:35.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.265+0000 7fab40cd0640 1 -- 172.21.15.60:0/767073856 reap_dead start 2023-10-16T13:51:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.266+0000 7fab3bfff640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab280039c0 0x7fab28003dc0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00a810 tx=0x7fab280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d2452d6030b92f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.266+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c034630 con 0x7fab280039c0 2023-10-16T13:51:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.266+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c02f070 con 0x7fab280039c0 2023-10-16T13:51:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.266+0000 7fab3affd640 1 -- 172.21.15.60:0/767073856 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c034a10 con 0x7fab280039c0 2023-10-16T13:51:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.266+0000 7fab414d1640 1 -- 172.21.15.60:0/767073856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab280039c0 msgr2=0x7fab28003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.266+0000 7fab414d1640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab280039c0 0x7fab28003dc0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00a810 tx=0x7fab280057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.267+0000 7fab414d1640 1 -- 172.21.15.60:0/767073856 shutdown_connections 2023-10-16T13:51:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.267+0000 7fab414d1640 1 --2- 172.21.15.60:0/767073856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab280039c0 0x7fab28003dc0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.267+0000 7fab414d1640 1 -- 172.21.15.60:0/767073856 >> 172.21.15.60:0/767073856 conn(0x7fab3c0cf570 msgr2=0x7fab3c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.267+0000 7fab414d1640 1 -- 172.21.15.60:0/767073856 shutdown_connections 2023-10-16T13:51:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.267+0000 7fab414d1640 1 -- 172.21.15.60:0/767073856 wait complete. 2023-10-16T13:51:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.268+0000 7fab414d1640 1 Processor -- start 2023-10-16T13:51:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.269+0000 7fab414d1640 1 -- start start 2023-10-16T13:51:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab414d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280039c0 0x7fab3c159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab414d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 0x7fab3c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab414d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab3c0ca100 0x7fab3c1f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab414d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0ca580 con 0x7fab3c15e770 2023-10-16T13:51:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab414d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0cb190 con 0x7fab3c0ca100 2023-10-16T13:51:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab414d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c15a490 con 0x7fab280039c0 2023-10-16T13:51:35.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 0x7fab3c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 0x7fab3c0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55096/0 (socket says 172.21.15.60:55096) 2023-10-16T13:51:35.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 -- 172.21.15.60:0/2909112783 learned_addr learned my addr 172.21.15.60:0/2909112783 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3bfff640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280039c0 0x7fab3c159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab40cd0640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab3c0ca100 0x7fab3c1f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 -- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280039c0 msgr2=0x7fab3c159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280039c0 0x7fab3c159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 -- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab3c0ca100 msgr2=0x7fab3c1f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.270+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab3c0ca100 0x7fab3c1f2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab3b7fe640 1 -- 172.21.15.60:0/2909112783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c15d6d0 con 0x7fab3c15e770 2023-10-16T13:51:35.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab40cd0640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab3c0ca100 0x7fab3c1f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:35.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 0x7fab3c0c9bc0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fab24010910 tx=0x7fab24010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab38ff9640 1 -- 172.21.15.60:0/2909112783 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24009690 con 0x7fab3c15e770 2023-10-16T13:51:35.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab38ff9640 1 -- 172.21.15.60:0/2909112783 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2400ec20 con 0x7fab3c15e770 2023-10-16T13:51:35.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab38ff9640 1 -- 172.21.15.60:0/2909112783 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2401c400 con 0x7fab3c15e770 2023-10-16T13:51:35.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c15d9b0 con 0x7fab3c15e770 2023-10-16T13:51:35.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.271+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c15df70 con 0x7fab3c15e770 2023-10-16T13:51:35.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.274+0000 7fab38ff9640 1 -- 172.21.15.60:0/2909112783 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab2401c560 con 0x7fab3c15e770 2023-10-16T13:51:35.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.274+0000 7fab38ff9640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab1c06afc0 0x7fab1c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.275+0000 7fab38ff9640 1 -- 172.21.15.60:0/2909112783 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab24089d40 con 0x7fab3c15e770 2023-10-16T13:51:35.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.275+0000 7fab4539d740 1 Processor -- start 2023-10-16T13:51:35.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.275+0000 7fab3bfff640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab1c06afc0 0x7fab1c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.275+0000 7fab4539d740 1 -- start start 2023-10-16T13:51:35.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.276+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82170 0x56285ab55070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.276+0000 7fab4539d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285ab84f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.276+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab555b0 0x56285ab85440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.276+0000 7fab4539d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285aa757a0 con 0x56285ab82550 2023-10-16T13:51:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.276+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285aac7740 con 0x56285ab82170 2023-10-16T13:51:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.276+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x56285aac4e30 con 0x56285ab555b0 2023-10-16T13:51:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3bfff640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab1c06afc0 0x7fab1c06d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab2c034000 tx=0x7fab2c02f000 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285ab84f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285ab84f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55100/0 (socket says 172.21.15.60:55100) 2023-10-16T13:51:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 -- 172.21.15.60:0/1137215374 learned_addr learned my addr 172.21.15.60:0/1137215374 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab40cd0640 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab555b0 0x56285ab85440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3bfff640 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82170 0x56285ab55070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 -- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab555b0 msgr2=0x56285ab85440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab555b0 0x56285ab85440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 -- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82170 msgr2=0x56285ab55070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82170 0x56285ab55070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.277+0000 7fab3b7fe640 1 -- 172.21.15.60:0/1137215374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56285aa84910 con 0x56285ab82550 2023-10-16T13:51:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.278+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285ab84f00 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fab24054eb0 tx=0x7fab24010110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b8a15bcdf432ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.278+0000 7fab31ffb640 1 -- 172.21.15.60:0/1137215374 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24060c00 con 0x56285ab82550 2023-10-16T13:51:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.278+0000 7fab31ffb640 1 -- 172.21.15.60:0/1137215374 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab240881c0 con 0x56285ab82550 2023-10-16T13:51:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.278+0000 7fab31ffb640 1 -- 172.21.15.60:0/1137215374 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24088330 con 0x56285ab82550 2023-10-16T13:51:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.278+0000 7fab4539d740 1 -- 172.21.15.60:0/1137215374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 msgr2=0x56285ab84f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.278+0000 7fab4539d740 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285ab84f00 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fab24054eb0 tx=0x7fab24010110 comp rx=0 tx=0).stop 2023-10-16T13:51:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 -- 172.21.15.60:0/1137215374 shutdown_connections 2023-10-16T13:51:35.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab555b0 0x56285ab85440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285ab84f00 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 --2- 172.21.15.60:0/1137215374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82170 0x56285ab55070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 -- 172.21.15.60:0/1137215374 >> 172.21.15.60:0/1137215374 conn(0x56285ab57380 msgr2=0x56285ab9ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 -- 172.21.15.60:0/1137215374 shutdown_connections 2023-10-16T13:51:35.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 -- 172.21.15.60:0/1137215374 wait complete. 2023-10-16T13:51:35.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.279+0000 7fab4539d740 1 -- start start 2023-10-16T13:51:35.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.280+0000 7fab4539d740 1 Processor -- start 2023-10-16T13:51:35.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285ab9a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285ab9aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 0x56285ab87af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285aac0720 con 0x56285ab87670 2023-10-16T13:51:35.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285aa5e970 con 0x56285ab82550 2023-10-16T13:51:35.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x56285aa5e610 con 0x56285ab82170 2023-10-16T13:51:35.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab4539d740 10 client.0 _ll_register_callbacks cb 0x56285aa809d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:51:35.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab40cd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 0x56285ab87af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab40cd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 0x56285ab87af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55108/0 (socket says 172.21.15.60:55108) 2023-10-16T13:51:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.281+0000 7fab40cd0640 1 -- 172.21.15.60:0/3470032023 learned_addr learned my addr 172.21.15.60:0/3470032023 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab3bfff640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285ab9a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285ab9aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab40cd0640 1 -- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 msgr2=0x56285ab9a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab40cd0640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285ab9a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab40cd0640 1 -- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 msgr2=0x56285ab9aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab40cd0640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285ab9aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab40cd0640 1 -- 172.21.15.60:0/3470032023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c15df70 con 0x56285ab87670 2023-10-16T13:51:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab3bfff640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285ab9a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285ab9aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab40cd0640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 0x56285ab87af0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0b2520 tx=0x7fab34004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5636bd60eef193b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab1affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T13:51:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab1affd640 1 -- 172.21.15.60:0/3470032023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab340054f0 con 0x56285ab87670 2023-10-16T13:51:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab1affd640 1 -- 172.21.15.60:0/3470032023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab3400c030 con 0x56285ab87670 2023-10-16T13:51:35.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56285aae7160 con 0x56285ab87670 2023-10-16T13:51:35.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab1affd640 1 -- 172.21.15.60:0/3470032023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab34007020 con 0x56285ab87670 2023-10-16T13:51:35.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.282+0000 7fab4539d740 10 client.10116 fetch_fsmap learned FSMap version 95 2023-10-16T13:51:35.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x56285aabeda0 con 0x56285ab87670 2023-10-16T13:51:35.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab1affd640 1 -- 172.21.15.60:0/3470032023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56285aae7160 con 0x56285ab87670 2023-10-16T13:51:35.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab1affd640 1 -- 172.21.15.60:0/3470032023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 95) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fab34007180 con 0x56285ab87670 2023-10-16T13:51:35.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 10 client.10116 fetch_fsmap finished waiting for FSMap version 95 2023-10-16T13:51:35.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 10 client.10116 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T13:51:35.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 20 client.10116 populate_metadata read hostname 'smithi060' 2023-10-16T13:51:35.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285ab71ed0 0x56285ab70d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 4 client.10116 mds_command: new command op to 9966 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:51:35.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 --> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56285aa92cc0 con 0x56285ab71ed0 2023-10-16T13:51:35.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.283+0000 7fab3bfff640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285ab71ed0 0x56285ab70d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.284+0000 7fab3bfff640 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285ab71ed0 0x56285ab70d80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b35d715e2a08a458 server_cookie=e1ce0eb33719927f in_seq=0 out_seq=0 2023-10-16T13:51:35.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.284+0000 7fab1affd640 10 client.10116 ms_handle_connect on v2:172.21.15.60:6834/3850041246 2023-10-16T13:51:35.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab1affd640 1 -- 172.21.15.60:0/3470032023 <== mds.0 v2:172.21.15.60:6834/3850041246 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56285aa92cc0 con 0x56285ab71ed0 2023-10-16T13:51:35.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab1affd640 10 client.10116 handle_command_reply: tid=0 2023-10-16T13:51:35.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 client.10116 shutdown 2023-10-16T13:51:35.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 msgr2=0x56285ab87af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 0x56285ab87af0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0b2520 tx=0x7fab34004c40 comp rx=0 tx=0).stop 2023-10-16T13:51:35.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 shutdown_connections 2023-10-16T13:51:35.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285ab71ed0 0x56285ab70d80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab87670 0x56285ab87af0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285ab9aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab1affd640 0 client.10116 ms_handle_reset on v2:172.21.15.60:6834/3850041246 2023-10-16T13:51:35.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 --2- 172.21.15.60:0/3470032023 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285ab9a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 >> 172.21.15.60:0/3470032023 conn(0x56285ab555b0 msgr2=0x56285ab59130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 shutdown_connections 2023-10-16T13:51:35.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.285+0000 7fab4539d740 1 -- 172.21.15.60:0/3470032023 wait complete. 2023-10-16T13:51:35.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.286+0000 7fab4539d740 20 client.10116 trim_cache size 0 max 16384 2023-10-16T13:51:35.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.302+0000 7fab4539d740 1 Processor -- start 2023-10-16T13:51:35.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 -- start start 2023-10-16T13:51:35.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285aba26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285aba2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab87670 0x56285aba1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285a824f20 con 0x56285ab82550 2023-10-16T13:51:35.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285ab5c250 con 0x56285ab87670 2023-10-16T13:51:35.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.303+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x56285aa665c0 con 0x56285ab82170 2023-10-16T13:51:35.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285aba2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285aba2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55122/0 (socket says 172.21.15.60:55122) 2023-10-16T13:51:35.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 -- 172.21.15.60:0/450852872 learned_addr learned my addr 172.21.15.60:0/450852872 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 -- 172.21.15.60:0/450852872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 msgr2=0x56285aba26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285aba26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 -- 172.21.15.60:0/450852872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab87670 msgr2=0x56285aba1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab87670 0x56285aba1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 -- 172.21.15.60:0/450852872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56285aa5db80 con 0x56285ab82550 2023-10-16T13:51:35.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.304+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285aba2bf0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fab24058940 tx=0x7fab2405b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cd42f5aa99f9a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab327fc640 1 -- 172.21.15.60:0/450852872 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24088330 con 0x56285ab82550 2023-10-16T13:51:35.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab327fc640 1 -- 172.21.15.60:0/450852872 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab24060c00 con 0x56285ab82550 2023-10-16T13:51:35.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab327fc640 1 -- 172.21.15.60:0/450852872 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240550c0 con 0x56285ab82550 2023-10-16T13:51:35.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 -- 172.21.15.60:0/450852872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 msgr2=0x56285aba2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285aba2bf0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fab24058940 tx=0x7fab2405b000 comp rx=0 tx=0).stop 2023-10-16T13:51:35.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 -- 172.21.15.60:0/450852872 shutdown_connections 2023-10-16T13:51:35.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab87670 0x56285aba1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82550 0x56285aba2bf0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 --2- 172.21.15.60:0/450852872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab82170 0x56285aba26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 -- 172.21.15.60:0/450852872 >> 172.21.15.60:0/450852872 conn(0x56285ab555b0 msgr2=0x56285ab9b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 -- 172.21.15.60:0/450852872 shutdown_connections 2023-10-16T13:51:35.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.305+0000 7fab4539d740 1 -- 172.21.15.60:0/450852872 wait complete. 2023-10-16T13:51:35.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.306+0000 7fab4539d740 1 -- start start 2023-10-16T13:51:35.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.306+0000 7fab4539d740 1 Processor -- start 2023-10-16T13:51:35.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 0x56285ac08a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285abedc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab87670 0x56285ac0b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285ac0c2b0 con 0x56285ab82170 2023-10-16T13:51:35.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285ab711a0 con 0x56285ab82550 2023-10-16T13:51:35.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x56285ab55c50 con 0x56285ab87670 2023-10-16T13:51:35.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.307+0000 7fab4539d740 10 client.0 _ll_register_callbacks cb 0x7fab3400f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:51:35.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 0x56285ac08a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 0x56285ac08a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55124/0 (socket says 172.21.15.60:55124) 2023-10-16T13:51:35.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 -- 172.21.15.60:0/2359881448 learned_addr learned my addr 172.21.15.60:0/2359881448 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 -- 172.21.15.60:0/2359881448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab87670 msgr2=0x56285ac0b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab87670 0x56285ac0b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 -- 172.21.15.60:0/2359881448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 msgr2=0x56285abedc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285abedc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 -- 172.21.15.60:0/2359881448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56285aa84910 con 0x56285ab82170 2023-10-16T13:51:35.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab3bfff640 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 0x56285ac08a70 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fab2c034640 tx=0x7fab2c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814f579ffac8846f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.308+0000 7fab1affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T13:51:35.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab1affd640 1 -- 172.21.15.60:0/2359881448 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c052070 con 0x56285ab82170 2023-10-16T13:51:35.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab1affd640 1 -- 172.21.15.60:0/2359881448 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab2c04c440 con 0x56285ab82170 2023-10-16T13:51:35.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab240881c0 con 0x56285ab82170 2023-10-16T13:51:35.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab1affd640 1 -- 172.21.15.60:0/2359881448 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab2c034430 con 0x56285ab82170 2023-10-16T13:51:35.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 10 client.10122 fetch_fsmap learned FSMap version 95 2023-10-16T13:51:35.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x56285aa83170 con 0x56285ab82170 2023-10-16T13:51:35.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab1affd640 1 -- 172.21.15.60:0/2359881448 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab240881c0 con 0x56285ab82170 2023-10-16T13:51:35.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab1affd640 1 -- 172.21.15.60:0/2359881448 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 95) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fab2c04c600 con 0x56285ab82170 2023-10-16T13:51:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 10 client.10122 fetch_fsmap finished waiting for FSMap version 95 2023-10-16T13:51:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 10 client.10122 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T13:51:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 20 client.10122 populate_metadata read hostname 'smithi060' 2023-10-16T13:51:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285aba2790 0x56285aba2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 4 client.10122 mds_command: new command op to 9966 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:51:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.309+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 --> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56285aab9700 con 0x56285aba2790 2023-10-16T13:51:35.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.310+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285aba2790 0x56285aba2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.310+0000 7fab3b7fe640 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285aba2790 0x56285aba2b70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc1c4a8e1a57eaf6 server_cookie=42c7099d2775697a in_seq=0 out_seq=0 2023-10-16T13:51:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.310+0000 7fab1affd640 10 client.10122 ms_handle_connect on v2:172.21.15.60:6834/3850041246 2023-10-16T13:51:35.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab1affd640 1 -- 172.21.15.60:0/2359881448 <== mds.0 v2:172.21.15.60:6834/3850041246 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x56285aab9700 con 0x56285aba2790 2023-10-16T13:51:35.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab1affd640 10 client.10122 handle_command_reply: tid=0 2023-10-16T13:51:35.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 client.10122 shutdown 2023-10-16T13:51:35.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 msgr2=0x56285ac08a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 0x56285ac08a70 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fab2c034640 tx=0x7fab2c001cb0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 shutdown_connections 2023-10-16T13:51:35.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:6834/3850041246,v1:172.21.15.60:6835/3850041246] conn(0x56285aba2790 0x56285aba2b70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x56285ab87670 0x56285ac0b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56285ab82550 0x56285abedc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 --2- 172.21.15.60:0/2359881448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56285ab82170 0x56285ac08a70 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab1affd640 0 client.10122 ms_handle_reset on v2:172.21.15.60:6834/3850041246 2023-10-16T13:51:35.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 >> 172.21.15.60:0/2359881448 conn(0x56285ab555b0 msgr2=0x56285ab50810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 shutdown_connections 2023-10-16T13:51:35.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 1 -- 172.21.15.60:0/2359881448 wait complete. 2023-10-16T13:51:35.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.311+0000 7fab4539d740 20 client.10122 trim_cache size 0 max 16384 2023-10-16T13:51:35.327 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T13:51:35.327 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T13:51:35.327 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T13:51:35.327 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 9966, 2023-10-16T13:51:35.328 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T13:51:35.328 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T13:51:35.328 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T13:51:35.328 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.9046708079999997, 2023-10-16T13:51:35.328 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 95, 2023-10-16T13:51:35.329 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 197, 2023-10-16T13:51:35.329 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 197, 2023-10-16T13:51:35.329 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.61667044 2023-10-16T13:51:35.329 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T13:51:35.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab1c06afc0 msgr2=0x7fab1c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab1c06afc0 0x7fab1c06d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab2c034000 tx=0x7fab2c02f000 comp rx=0 tx=0).stop 2023-10-16T13:51:35.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 msgr2=0x7fab3c0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 0x7fab3c0c9bc0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fab24010910 tx=0x7fab24010dd0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 shutdown_connections 2023-10-16T13:51:35.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fab1c06afc0 0x7fab1c06d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab3c0ca100 0x7fab3c1f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab3c15e770 0x7fab3c0c9bc0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 --2- 172.21.15.60:0/2909112783 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab280039c0 0x7fab3c159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 >> 172.21.15.60:0/2909112783 conn(0x7fab3c0cf570 msgr2=0x7fab3c0ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 shutdown_connections 2023-10-16T13:51:35.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.312+0000 7fab414d1640 1 -- 172.21.15.60:0/2909112783 wait complete. 2023-10-16T13:51:35.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:51:35.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.555+0000 7f3f0518e640 1 Processor -- start 2023-10-16T13:51:35.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.555+0000 7f3f0518e640 1 -- start start 2023-10-16T13:51:35.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3f0518e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3f000cd0c0 0x7f3f001615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3f0518e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f000b0410 0x7f3f000b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3f0518e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0015e3e0 con 0x7f3f0015d3f0 2023-10-16T13:51:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3f0518e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000b0de0 con 0x7f3f000b0410 2023-10-16T13:51:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3f0518e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000cd4a0 con 0x7f3f000cd0c0 2023-10-16T13:51:35.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3ef7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f000b0410 0x7f3f000b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3ef7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f000b0410 0x7f3f000b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55128/0 (socket says 172.21.15.60:55128) 2023-10-16T13:51:35.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.557+0000 7f3ef7fff640 1 -- 172.21.15.60:0/2440121724 learned_addr learned my addr 172.21.15.60:0/2440121724 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.558+0000 7f3efed9d640 1 --2- 172.21.15.60:0/2440121724 >> v2:172.21.15.144:3300/0 conn(0x7f3f000cd0c0 0x7f3f001615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.558+0000 7f3ef7fff640 1 -- 172.21.15.60:0/2440121724 >> v2:172.21.15.144:3300/0 conn(0x7f3f000cd0c0 msgr2=0x7f3f001615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.558+0000 7f3ef7fff640 1 --2- 172.21.15.60:0/2440121724 >> v2:172.21.15.144:3300/0 conn(0x7f3f000cd0c0 0x7f3f001615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.558+0000 7f3ef7fff640 1 -- 172.21.15.60:0/2440121724 >> v1:172.21.15.144:6789/0 conn(0x7f3f0015d3f0 legacy=0x7f3f0015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.558+0000 7f3ef7fff640 1 -- 172.21.15.60:0/2440121724 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0015e100 con 0x7f3f000b0410 2023-10-16T13:51:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.559+0000 7f3ef7fff640 1 --2- 172.21.15.60:0/2440121724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f000b0410 0x7f3f000b08a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f3f000cc4e0 tx=0x7f3ee802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da554f62444bca6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.559+0000 7f3efe59c640 1 -- 172.21.15.60:0/2440121724 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80317e0 con 0x7f3f000b0410 2023-10-16T13:51:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.559+0000 7f3efe59c640 1 -- 172.21.15.60:0/2440121724 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee8031940 con 0x7f3f000b0410 2023-10-16T13:51:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.559+0000 7f3efe59c640 1 -- 172.21.15.60:0/2440121724 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8031cd0 con 0x7f3f000b0410 2023-10-16T13:51:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 -- 172.21.15.60:0/2440121724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f000b0410 msgr2=0x7f3f000b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 --2- 172.21.15.60:0/2440121724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f000b0410 0x7f3f000b08a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f3f000cc4e0 tx=0x7f3ee802b880 comp rx=0 tx=0).stop 2023-10-16T13:51:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 -- 172.21.15.60:0/2440121724 shutdown_connections 2023-10-16T13:51:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 --2- 172.21.15.60:0/2440121724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f000b0410 0x7f3f000b08a0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 --2- 172.21.15.60:0/2440121724 >> v2:172.21.15.144:3300/0 conn(0x7f3f000cd0c0 0x7f3f001615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 -- 172.21.15.60:0/2440121724 >> 172.21.15.60:0/2440121724 conn(0x7f3f000cb410 msgr2=0x7f3f000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 -- 172.21.15.60:0/2440121724 shutdown_connections 2023-10-16T13:51:35.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.560+0000 7f3f0518e640 1 -- 172.21.15.60:0/2440121724 wait complete. 2023-10-16T13:51:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.562+0000 7f3f0518e640 1 Processor -- start 2023-10-16T13:51:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.562+0000 7f3f0518e640 1 -- start start 2023-10-16T13:51:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3f0518e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f000b0410 0x7f3f00152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3f0518e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f000cd0c0 0x7f3f0015f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3f0518e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 0x7f3f000ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3f0518e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f001f6540 con 0x7f3f0015d3f0 2023-10-16T13:51:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3f0518e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f001f6790 con 0x7f3f000cd0c0 2023-10-16T13:51:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3f0518e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f001f6a70 con 0x7f3f000b0410 2023-10-16T13:51:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.563+0000 7f3eff59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 0x7f3f000ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 0x7f3f000ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55140/0 (socket says 172.21.15.60:55140) 2023-10-16T13:51:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 -- 172.21.15.60:0/1937173545 learned_addr learned my addr 172.21.15.60:0/1937173545 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 -- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f000b0410 msgr2=0x7f3f00152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f000b0410 0x7f3f00152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 -- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f000cd0c0 msgr2=0x7f3f0015f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f000cd0c0 0x7f3f0015f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 -- 172.21.15.60:0/1937173545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80094d0 con 0x7f3f0015d3f0 2023-10-16T13:51:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3eff59e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 0x7f3f000ae3f0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c780 tx=0x7f3ef800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.564+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8003d80 con 0x7f3f0015d3f0 2023-10-16T13:51:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.565+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef800ce90 con 0x7f3f0015d3f0 2023-10-16T13:51:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.565+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80052b0 con 0x7f3f0015d3f0 2023-10-16T13:51:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.565+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f001f6d50 con 0x7f3f0015d3f0 2023-10-16T13:51:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.565+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f001f7310 con 0x7f3f0015d3f0 2023-10-16T13:51:35.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.566+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0004e160 con 0x7f3f0015d3f0 2023-10-16T13:51:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.567+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3ef8005410 con 0x7f3f0015d3f0 2023-10-16T13:51:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.568+0000 7f3ef77fe640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3ee006af90 0x7f3ee006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.568+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ef8088cb0 con 0x7f3f0015d3f0 2023-10-16T13:51:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.569+0000 7f3efed9d640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3ee006af90 0x7f3ee006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.570+0000 7f3efed9d640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3ee006af90 0x7f3ee006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0006fd0 tx=0x7f3ef0008040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.572+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ef8053bb0 con 0x7f3f0015d3f0 2023-10-16T13:51:35.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.811+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f000ccc00 con 0x7f3f0015d3f0 2023-10-16T13:51:35.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.812+0000 7f3ef77fe640 1 -- 172.21.15.60:0/1937173545 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3ef80571d0 con 0x7f3f0015d3f0 2023-10-16T13:51:35.816 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:51:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3ee006af90 msgr2=0x7f3ee006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3ee006af90 0x7f3ee006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0006fd0 tx=0x7f3ef0008040 comp rx=0 tx=0).stop 2023-10-16T13:51:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 msgr2=0x7f3f000ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 0x7f3f000ae3f0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c780 tx=0x7f3ef800cc40 comp rx=0 tx=0).stop 2023-10-16T13:51:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 shutdown_connections 2023-10-16T13:51:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f3ee006af90 0x7f3ee006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f0015d3f0 0x7f3f000ae3f0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f000cd0c0 0x7f3f0015f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 --2- 172.21.15.60:0/1937173545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f000b0410 0x7f3f00152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 >> 172.21.15.60:0/1937173545 conn(0x7f3f000cb410 msgr2=0x7f3f0015c790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:35.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 shutdown_connections 2023-10-16T13:51:35.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:35.815+0000 7f3f0518e640 1 -- 172.21.15.60:0/1937173545 wait complete. 2023-10-16T13:51:35.837 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:51:35.837 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:51:35.838 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T13:51:35.838 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:51:35.838 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:51:35.839 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:51:35.839 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:51:35.839 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:51:35.840 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:51:35.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:35.841 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T13:51:35.892 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:51:35.893 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:51:35.893 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:51:35.893 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.893 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T13:51:35.893 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.894 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:51:35.894 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T13:51:35.894 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T13:51:35.894 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:51:35.894 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T13:51:35.894 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T13:51:35.895 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:51:35.895 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:51:35.895 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T13:51:35.895 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:51:35.895 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 37691sec preferred_lft 37691sec 2023-10-16T13:51:35.895 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T13:51:35.896 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.896 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:51:35.896 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:51:35.896 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T13:51:35.896 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:51:35.897 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T13:51:35.897 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:51:35.897 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.897 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T13:51:35.897 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.897 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:51:35.898 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:51:35.898 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T13:51:35.898 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.898 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:51:35.898 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:51:35.899 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T13:51:35.899 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:51:35.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:35.900 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T13:51:35.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:35.992 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T13:51:36.050 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T13:51:36.050 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:51:36.050 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:51:36.050 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:51:36.051 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:51:36.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:36.052 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T13:51:36.052 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T13:51:36.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:51:36.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:51:36.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:51:36.054 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T13:51:36.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:36.254 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T13:51:36.311 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:51:36.312 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:51:36.313 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:51:36.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:36.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T13:51:36.416 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:51:36.416 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:51:36.474 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:51:36.474 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:51:36.475 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:51:36.522 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T13:51:36.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:36.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T13:51:36.719 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T13:51:36.720 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T13:51:36.753 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T13:51:36.753 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T13:51:36.754 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T13:51:36.754 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T13:51:36.754 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T13:51:36.755 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T13:51:36.756 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T13:51:36.756 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T13:51:36.756 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T13:51:36.756 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T13:51:36.759 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:51:36.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:36.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:51:36.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:36.838 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:51:36.903 INFO:teuthology.orchestra.run.smithi060.stdout:client10131 2023-10-16T13:51:36.903 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T13:51:36.903 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:51:36.952 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:51:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.227+0000 7fb56ff8a640 1 Processor -- start 2023-10-16T13:51:37.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.227+0000 7fb56ff8a640 1 -- start start 2023-10-16T13:51:37.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.229+0000 7fb56ff8a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680caac0 0x7fb5680ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.229+0000 7fb56ff8a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680cb030 con 0x7fb56806a4b0 2023-10-16T13:51:37.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.229+0000 7fb56ff8a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680c91c0 con 0x7fb56815ad70 2023-10-16T13:51:37.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.229+0000 7fb56ff8a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680c94a0 con 0x7fb5680caac0 2023-10-16T13:51:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.230+0000 7fb56d525640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb56806a4b0 0x7fb5680ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53948/0 (socket says 172.21.15.60:53948) 2023-10-16T13:51:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.230+0000 7fb56d525640 1 -- 172.21.15.60:0/1294979356 learned_addr learned my addr 172.21.15.60:0/1294979356 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.230+0000 7fb56e527640 1 --2- 172.21.15.60:0/1294979356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680caac0 0x7fb5680ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.230+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366916511 0 0) 0x7fb5680cb030 con 0x7fb56806a4b0 2023-10-16T13:51:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.230+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb554003520 con 0x7fb56806a4b0 2023-10-16T13:51:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.230+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214136181 0 0) 0x7fb5680c91c0 con 0x7fb56815ad70 2023-10-16T13:51:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.231+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5680cb030 con 0x7fb56815ad70 2023-10-16T13:51:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.231+0000 7fb56e527640 1 -- 172.21.15.60:0/1294979356 >> v1:172.21.15.144:6789/0 conn(0x7fb56815ad70 legacy=0x7fb568069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.231+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 308740224 0 0) 0x7fb554003520 con 0x7fb56806a4b0 2023-10-16T13:51:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.231+0000 7fb56e527640 1 -- 172.21.15.60:0/1294979356 >> v1:172.21.15.60:6789/0 conn(0x7fb56806a4b0 legacy=0x7fb5680ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.231+0000 7fb56e527640 1 -- 172.21.15.60:0/1294979356 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5680c9780 con 0x7fb5680caac0 2023-10-16T13:51:37.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.232+0000 7fb56e527640 1 --2- 172.21.15.60:0/1294979356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680caac0 0x7fb5680ad4d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb560009dc0 tx=0x7fb56002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60005c56ef5c9f38 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:37.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.232+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558001670 con 0x7fb5680caac0 2023-10-16T13:51:37.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.232+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56002c910 con 0x7fb5680caac0 2023-10-16T13:51:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.232+0000 7fb56cd24640 1 -- 172.21.15.60:0/1294979356 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56002cca0 con 0x7fb5680caac0 2023-10-16T13:51:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 -- 172.21.15.60:0/1294979356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680caac0 msgr2=0x7fb5680ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/1294979356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680caac0 0x7fb5680ad4d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb560009dc0 tx=0x7fb56002bde0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 -- 172.21.15.60:0/1294979356 shutdown_connections 2023-10-16T13:51:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/1294979356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5680caac0 0x7fb5680ad4d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 -- 172.21.15.60:0/1294979356 >> 172.21.15.60:0/1294979356 conn(0x7fb5680ce9b0 msgr2=0x7fb5680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:37.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 -- 172.21.15.60:0/1294979356 shutdown_connections 2023-10-16T13:51:37.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.233+0000 7fb56ff8a640 1 -- 172.21.15.60:0/1294979356 wait complete. 2023-10-16T13:51:37.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.235+0000 7fb56ff8a640 1 Processor -- start 2023-10-16T13:51:37.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.235+0000 7fb56ff8a640 1 -- start start 2023-10-16T13:51:37.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56ff8a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 0x7fb5680ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56ff8a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680caac0 0x7fb56815be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56ff8a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb56815ad70 0x7fb568154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56ff8a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56815c3a0 con 0x7fb56806a4b0 2023-10-16T13:51:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56ff8a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb568155360 con 0x7fb5680caac0 2023-10-16T13:51:37.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56ff8a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56815ce50 con 0x7fb56815ad70 2023-10-16T13:51:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56dd26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 0x7fb5680ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56dd26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 0x7fb5680ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55144/0 (socket says 172.21.15.60:55144) 2023-10-16T13:51:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.236+0000 7fb56dd26640 1 -- 172.21.15.60:0/2083581874 learned_addr learned my addr 172.21.15.60:0/2083581874 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56e527640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb56815ad70 0x7fb568154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56dd26640 1 -- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb56815ad70 msgr2=0x7fb568154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56d525640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680caac0 0x7fb56815be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56dd26640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb56815ad70 0x7fb568154e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56dd26640 1 -- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680caac0 msgr2=0x7fb56815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56dd26640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680caac0 0x7fb56815be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56dd26640 1 -- 172.21.15.60:0/2083581874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560009a70 con 0x7fb56806a4b0 2023-10-16T13:51:37.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb56dd26640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 0x7fb5680ad310 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fb56400c430 tx=0x7fb56400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:37.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5640052d0 con 0x7fb56806a4b0 2023-10-16T13:51:37.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb564005450 con 0x7fb56806a4b0 2023-10-16T13:51:37.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.237+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564003830 con 0x7fb56806a4b0 2023-10-16T13:51:37.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.238+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56815c5b0 con 0x7fb56806a4b0 2023-10-16T13:51:37.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.238+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56815cb70 con 0x7fb56806a4b0 2023-10-16T13:51:37.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.239+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56804e160 con 0x7fb56806a4b0 2023-10-16T13:51:37.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.240+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb564003990 con 0x7fb56806a4b0 2023-10-16T13:51:37.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.241+0000 7fb55e7fc640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb54c06af90 0x7fb54c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.241+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb564088d10 con 0x7fb56806a4b0 2023-10-16T13:51:37.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.241+0000 7fb56d525640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb54c06af90 0x7fb54c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.243+0000 7fb56d525640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb54c06af90 0x7fb54c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5580011e0 tx=0x7fb558009040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:37.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.246+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb564053d40 con 0x7fb56806a4b0 2023-10-16T13:51:37.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.465+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 --> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5680c9ca0 con 0x7fb54c06af90 2023-10-16T13:51:37.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.469+0000 7fb55e7fc640 1 -- 172.21.15.60:0/2083581874 <== mgr.7918 v2:172.21.15.144:6832/1700115305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb558001ba0 con 0x7fb54c06af90 2023-10-16T13:51:37.473 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T13:51:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T13:51:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T13:51:37.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T13:51:37.474 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T13:51:37.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb54c06af90 msgr2=0x7fb54c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb54c06af90 0x7fb54c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5580011e0 tx=0x7fb558009040 comp rx=0 tx=0).stop 2023-10-16T13:51:37.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 msgr2=0x7fb5680ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 0x7fb5680ad310 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fb56400c430 tx=0x7fb56400c8f0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 shutdown_connections 2023-10-16T13:51:37.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fb54c06af90 0x7fb54c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb56815ad70 0x7fb568154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5680caac0 0x7fb56815be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 --2- 172.21.15.60:0/2083581874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb56806a4b0 0x7fb5680ad310 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 >> 172.21.15.60:0/2083581874 conn(0x7fb5680ce9b0 msgr2=0x7fb5680b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:37.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 shutdown_connections 2023-10-16T13:51:37.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.472+0000 7fb56ff8a640 1 -- 172.21.15.60:0/2083581874 wait complete. 2023-10-16T13:51:37.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:51:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.706+0000 7f5e64b90640 1 Processor -- start 2023-10-16T13:51:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.707+0000 7f5e64b90640 1 -- start start 2023-10-16T13:51:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.708+0000 7f5e64b90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e6015ad70 0x7f5e60069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.708+0000 7f5e64b90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e6006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.708+0000 7f5e64b90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600cb030 con 0x7f5e600c9ca0 2023-10-16T13:51:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.708+0000 7f5e64b90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e600c91c0 con 0x7f5e6015ad70 2023-10-16T13:51:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.708+0000 7f5e64b90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c94a0 con 0x7f5e600caac0 2023-10-16T13:51:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5dd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e600c9ca0 0x7f5e600ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53954/0 (socket says 172.21.15.60:53954) 2023-10-16T13:51:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5dd9b640 1 -- 172.21.15.60:0/1718051860 learned_addr learned my addr 172.21.15.60:0/1718051860 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5e59c640 1 --2- 172.21.15.60:0/1718051860 >> v2:172.21.15.60:3300/0 conn(0x7f5e6015ad70 0x7f5e60069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5d59a640 1 -- 172.21.15.60:0/1718051860 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560048438 0 0) 0x7f5e600cb030 con 0x7f5e600c9ca0 2023-10-16T13:51:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5d59a640 1 -- 172.21.15.60:0/1718051860 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e48003520 con 0x7f5e600c9ca0 2023-10-16T13:51:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5ed9d640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e6006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5e59c640 1 -- 172.21.15.60:0/1718051860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 msgr2=0x7f5e6006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5e59c640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e6006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5e59c640 1 -- 172.21.15.60:0/1718051860 >> v1:172.21.15.60:6789/0 conn(0x7f5e600c9ca0 legacy=0x7f5e600ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5e59c640 1 -- 172.21.15.60:0/1718051860 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600c9780 con 0x7f5e6015ad70 2023-10-16T13:51:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.709+0000 7f5e5ed9d640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e6006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.710+0000 7f5e5e59c640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e60069da0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097a0 tx=0x7f5e5002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2423859c1fa41b76 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e5d59a640 1 -- 172.21.15.60:0/1718051860 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5002d040 con 0x7f5e6015ad70 2023-10-16T13:51:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e5d59a640 1 -- 172.21.15.60:0/1718051860 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50031810 con 0x7f5e6015ad70 2023-10-16T13:51:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e5d59a640 1 -- 172.21.15.60:0/1718051860 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50031b80 con 0x7f5e6015ad70 2023-10-16T13:51:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e64b90640 1 -- 172.21.15.60:0/1718051860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 msgr2=0x7f5e60069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e64b90640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e60069da0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097a0 tx=0x7f5e5002b8b0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e64b90640 1 -- 172.21.15.60:0/1718051860 shutdown_connections 2023-10-16T13:51:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e64b90640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e6006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e64b90640 1 --2- 172.21.15.60:0/1718051860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e60069da0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.711+0000 7f5e64b90640 1 -- 172.21.15.60:0/1718051860 >> 172.21.15.60:0/1718051860 conn(0x7f5e600ce9b0 msgr2=0x7f5e600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:37.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.712+0000 7f5e64b90640 1 -- 172.21.15.60:0/1718051860 shutdown_connections 2023-10-16T13:51:37.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.712+0000 7f5e64b90640 1 -- 172.21.15.60:0/1718051860 wait complete. 2023-10-16T13:51:37.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.713+0000 7f5e64b90640 1 Processor -- start 2023-10-16T13:51:37.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.714+0000 7f5e64b90640 1 -- start start 2023-10-16T13:51:37.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.714+0000 7f5e64b90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e600c9ca0 0x7f5e601f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e64b90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e600cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e64b90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e6015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e64b90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6015dba0 con 0x7f5e6015ad70 2023-10-16T13:51:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e64b90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6015e370 con 0x7f5e600c9ca0 2023-10-16T13:51:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e64b90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6015ddb0 con 0x7f5e600caac0 2023-10-16T13:51:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e6015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e6015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55164/0 (socket says 172.21.15.60:55164) 2023-10-16T13:51:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 -- 172.21.15.60:0/4201036690 learned_addr learned my addr 172.21.15.60:0/4201036690 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5dd9b640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e600cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5e59c640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e600c9ca0 0x7f5e601f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 -- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 msgr2=0x7f5e600cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e600cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 -- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e600c9ca0 msgr2=0x7f5e601f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e600c9ca0 0x7f5e601f2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:37.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.715+0000 7f5e5ed9d640 1 -- 172.21.15.60:0/4201036690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50009400 con 0x7f5e6015ad70 2023-10-16T13:51:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.716+0000 7f5e5ed9d640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e6015cdd0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800cef0 tx=0x7f5e5800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.716+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800d840 con 0x7f5e6015ad70 2023-10-16T13:51:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.716+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5800d9c0 con 0x7f5e6015ad70 2023-10-16T13:51:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.716+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800dd70 con 0x7f5e6015ad70 2023-10-16T13:51:37.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.716+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6015e090 con 0x7f5e6015ad70 2023-10-16T13:51:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.716+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6015c8b0 con 0x7f5e6015ad70 2023-10-16T13:51:37.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.718+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6004e160 con 0x7f5e6015ad70 2023-10-16T13:51:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.718+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 97) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5e5801e3b0 con 0x7f5e6015ad70 2023-10-16T13:51:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.719+0000 7f5e4effd640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5e4006afe0 0x7f5e4006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.719+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e5808a0e0 con 0x7f5e6015ad70 2023-10-16T13:51:37.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.720+0000 7f5e5e59c640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5e4006afe0 0x7f5e4006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.721+0000 7f5e5e59c640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5e4006afe0 0x7f5e4006d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e50036000 tx=0x7f5e5003c040 comp rx=0 tx=0).ready entity=mgr.7918 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:37.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.724+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e580552b0 con 0x7f5e6015ad70 2023-10-16T13:51:37.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:37.967+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5e60158780 con 0x7f5e6015ad70 2023-10-16T13:51:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.500+0000 7f5e4effd640 1 -- 172.21.15.60:0/4201036690 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v98) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5e580588d0 con 0x7f5e6015ad70 2023-10-16T13:51:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.505+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5e4006afe0 msgr2=0x7f5e4006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.505+0000 7f5e64b90640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5e4006afe0 0x7f5e4006d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e50036000 tx=0x7f5e5003c040 comp rx=0 tx=0).stop 2023-10-16T13:51:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.505+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 msgr2=0x7f5e6015cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.505+0000 7f5e64b90640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e6015cdd0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800cef0 tx=0x7f5e5800a710 comp rx=0 tx=0).stop 2023-10-16T13:51:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 shutdown_connections 2023-10-16T13:51:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f5e4006afe0 0x7f5e4006d4a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5e600caac0 0x7f5e600cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e6015ad70 0x7f5e6015cdd0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 --2- 172.21.15.60:0/4201036690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e600c9ca0 0x7f5e601f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 >> 172.21.15.60:0/4201036690 conn(0x7f5e600ce9b0 msgr2=0x7f5e600b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.507+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 shutdown_connections 2023-10-16T13:51:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.508+0000 7f5e64b90640 1 -- 172.21.15.60:0/4201036690 wait complete. 2023-10-16T13:51:38.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:51:38.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.744+0000 7fbfb0cc9640 1 Processor -- start 2023-10-16T13:51:38.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.744+0000 7fbfb0cc9640 1 -- start start 2023-10-16T13:51:38.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.745+0000 7fbfb0cc9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbfac0caac0 0x7fbfac0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.745+0000 7fbfb0cc9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0cb030 con 0x7fbfac06a4b0 2023-10-16T13:51:38.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.745+0000 7fbfb0cc9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0c91c0 con 0x7fbfac15ad70 2023-10-16T13:51:38.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.745+0000 7fbfb0cc9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfac0c94a0 con 0x7fbfac0caac0 2023-10-16T13:51:38.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.746+0000 7fbfa1d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbfac06a4b0 0x7fbfac0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53972/0 (socket says 172.21.15.60:53972) 2023-10-16T13:51:38.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.746+0000 7fbfa1d9b640 1 -- 172.21.15.60:0/235041814 learned_addr learned my addr 172.21.15.60:0/235041814 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:38.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.746+0000 7fbfaad9d640 1 --2- 172.21.15.60:0/235041814 >> v2:172.21.15.144:3300/0 conn(0x7fbfac0caac0 0x7fbfac0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.746+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941868312 0 0) 0x7fbfac0cb030 con 0x7fbfac06a4b0 2023-10-16T13:51:38.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.746+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf98003520 con 0x7fbfac06a4b0 2023-10-16T13:51:38.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4207872121 0 0) 0x7fbf98003520 con 0x7fbfac06a4b0 2023-10-16T13:51:38.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 >> v2:172.21.15.144:3300/0 conn(0x7fbfac0caac0 msgr2=0x7fbfac0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:38.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 --2- 172.21.15.60:0/235041814 >> v2:172.21.15.144:3300/0 conn(0x7fbfac0caac0 0x7fbfac0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 >> v1:172.21.15.144:6789/0 conn(0x7fbfac15ad70 legacy=0x7fbfac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0c9780 con 0x7fbfac06a4b0 2023-10-16T13:51:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fbf94001670 con 0x7fbfac06a4b0 2023-10-16T13:51:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 >> v1:172.21.15.60:6789/0 conn(0x7fbfac06a4b0 legacy=0x7fbfac0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf98003970 0x7fbf98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf980047c0 0x7fbf98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf98005180 0x7fbf9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0c94a0 con 0x7fbf98003970 2023-10-16T13:51:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9800c020 con 0x7fbf980047c0 2023-10-16T13:51:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9800c1e0 con 0x7fbf98005180 2023-10-16T13:51:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf98003970 0x7fbf98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfa1d9b640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf980047c0 0x7fbf98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.747+0000 7fbfaad9d640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf98005180 0x7fbf9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.748+0000 7fbfaa59c640 1 -- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf98005180 msgr2=0x7fbf9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.748+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf98005180 0x7fbf9800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.748+0000 7fbfaa59c640 1 -- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf980047c0 msgr2=0x7fbf98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:38.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.748+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf980047c0 0x7fbf98004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.748+0000 7fbfaa59c640 1 -- 172.21.15.60:0/235041814 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf98003520 con 0x7fbf98003970 2023-10-16T13:51:38.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.748+0000 7fbfaad9d640 1 -- 172.21.15.60:0/235041814 reap_dead start 2023-10-16T13:51:38.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.749+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf98003970 0x7fbf98003d90 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c001750 tx=0x7fbf980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c26d8ec64443cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:38.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.749+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0052d0 con 0x7fbf98003970 2023-10-16T13:51:38.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.749+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c02e050 con 0x7fbf98003970 2023-10-16T13:51:38.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.749+0000 7fbfa9d9b640 1 -- 172.21.15.60:0/235041814 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c003830 con 0x7fbf98003970 2023-10-16T13:51:38.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/235041814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf98003970 msgr2=0x7fbf98003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf98003970 0x7fbf98003d90 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c001750 tx=0x7fbf980057f0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/235041814 shutdown_connections 2023-10-16T13:51:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/235041814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf98003970 0x7fbf98003d90 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/235041814 >> 172.21.15.60:0/235041814 conn(0x7fbfac0ce9b0 msgr2=0x7fbfac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/235041814 shutdown_connections 2023-10-16T13:51:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.750+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/235041814 wait complete. 2023-10-16T13:51:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.752+0000 7fbfb0cc9640 1 Processor -- start 2023-10-16T13:51:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.752+0000 7fbfb0cc9640 1 -- start start 2023-10-16T13:51:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfb0cc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf98003970 0x7fbfac1f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfb0cc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 0x7fbfac0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfb0cc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfac0cdd30 0x7fbfac1f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfb0cc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0ce1e0 con 0x7fbfac0c6430 2023-10-16T13:51:38.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfb0cc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0c7310 con 0x7fbf98003970 2023-10-16T13:51:38.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfb0cc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac1f2930 con 0x7fbfac0cdd30 2023-10-16T13:51:38.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfa1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 0x7fbfac0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfa1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 0x7fbfac0cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55200/0 (socket says 172.21.15.60:55200) 2023-10-16T13:51:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.753+0000 7fbfa1d9b640 1 -- 172.21.15.60:0/2981475422 learned_addr learned my addr 172.21.15.60:0/2981475422 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf98003970 0x7fbfac1f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfaad9d640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfac0cdd30 0x7fbfac1f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa1d9b640 1 -- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfac0cdd30 msgr2=0x7fbfac1f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa1d9b640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfac0cdd30 0x7fbfac1f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa1d9b640 1 -- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf98003970 msgr2=0x7fbfac1f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa1d9b640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf98003970 0x7fbfac1f3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa1d9b640 1 -- 172.21.15.60:0/2981475422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac1f2c10 con 0x7fbfac0c6430 2023-10-16T13:51:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa1d9b640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 0x7fbfac0cd7f0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e780 tx=0x7fbf9400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400c660 con 0x7fbfac0c6430 2023-10-16T13:51:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400c800 con 0x7fbfac0c6430 2023-10-16T13:51:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400cbb0 con 0x7fbfac0c6430 2023-10-16T13:51:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.754+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac1f2ef0 con 0x7fbfac0c6430 2023-10-16T13:51:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.755+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac1f3a90 con 0x7fbfac0c6430 2023-10-16T13:51:38.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.756+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac0c7480 con 0x7fbfac0c6430 2023-10-16T13:51:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.757+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 98) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fbf9400cd10 con 0x7fbfac0c6430 2023-10-16T13:51:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.758+0000 7fbfa37fe640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 0x7fbf8c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.758+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf94089780 con 0x7fbfac0c6430 2023-10-16T13:51:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.758+0000 7fbfaa59c640 1 -- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 msgr2=0x7fbf8c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.758+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 0x7fbf8c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:38.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.762+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf94054680 con 0x7fbfac0c6430 2023-10-16T13:51:38.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.959+0000 7fbfaa59c640 1 -- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 msgr2=0x7fbf8c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:38.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:38.959+0000 7fbfaa59c640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 0x7fbf8c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:39.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.014+0000 7f50fba18040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:51:39.070 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.067+0000 7f50fba18040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:51:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.070+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fbfac04e160 con 0x7fbfac0c6430 2023-10-16T13:51:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.077+0000 7fbfa37fe640 1 -- 172.21.15.60:0/2981475422 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbf94057ca0 con 0x7fbfac0c6430 2023-10-16T13:51:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 msgr2=0x7fbf8c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 0x7fbf8c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 msgr2=0x7fbfac0cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 0x7fbfac0cd7f0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e780 tx=0x7fbf9400ec40 comp rx=0 tx=0).stop 2023-10-16T13:51:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 shutdown_connections 2023-10-16T13:51:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fbf8c06afe0 0x7fbf8c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfac0cdd30 0x7fbfac1f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfac0c6430 0x7fbfac0cd7f0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 --2- 172.21.15.60:0/2981475422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf98003970 0x7fbfac1f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 >> 172.21.15.60:0/2981475422 conn(0x7fbfac0ce9b0 msgr2=0x7fbfac15c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 shutdown_connections 2023-10-16T13:51:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.081+0000 7fbfb0cc9640 1 -- 172.21.15.60:0/2981475422 wait complete. 2023-10-16T13:51:39.110 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:51:39.172 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.169+0000 7f969e812040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:51:39.231 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.228+0000 7f969e812040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:51:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.311+0000 7f10879f5640 1 Processor -- start 2023-10-16T13:51:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.311+0000 7f10879f5640 1 -- start start 2023-10-16T13:51:39.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.312+0000 7f10879f5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10800cd970 0x7f10800d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.312+0000 7f10879f5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10800b5d90 con 0x7f108015d410 2023-10-16T13:51:39.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.312+0000 7f10879f5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1080160fc0 con 0x7f108015d030 2023-10-16T13:51:39.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.312+0000 7f10879f5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800d2b50 con 0x7f10800cd970 2023-10-16T13:51:39.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1084f90640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f108015d410 0x7f10800cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45676/0 (socket says 172.21.15.60:45676) 2023-10-16T13:51:39.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1084f90640 1 -- 172.21.15.60:0/3269460388 learned_addr learned my addr 172.21.15.60:0/3269460388 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T13:51:39.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1085f92640 1 --2- 172.21.15.60:0/3269460388 >> v2:172.21.15.60:3300/0 conn(0x7f10800cd970 0x7f10800d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1085f92640 1 -- 172.21.15.60:0/3269460388 >> v1:172.21.15.144:6789/0 conn(0x7f108015d030 legacy=0x7f1080161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1085f92640 1 -- 172.21.15.60:0/3269460388 >> v1:172.21.15.60:6789/0 conn(0x7f108015d410 legacy=0x7f10800cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1077fff640 1 -- 172.21.15.60:0/3269460388 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210282665 0 0) 0x7f10800b5d90 con 0x7f108015d410 2023-10-16T13:51:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.313+0000 7f1085f92640 1 -- 172.21.15.60:0/3269460388 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10800d25d0 con 0x7f10800cd970 2023-10-16T13:51:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.315+0000 7f1085f92640 1 --2- 172.21.15.60:0/3269460388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800cd970 0x7f10800d1630 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f1078009d90 tx=0x7f107802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04c4b7e00b3150b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.315+0000 7f1077fff640 1 -- 172.21.15.60:0/3269460388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107802cd00 con 0x7f10800cd970 2023-10-16T13:51:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.315+0000 7f1077fff640 1 -- 172.21.15.60:0/3269460388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107802ce60 con 0x7f10800cd970 2023-10-16T13:51:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.315+0000 7f1077fff640 1 -- 172.21.15.60:0/3269460388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078034660 con 0x7f10800cd970 2023-10-16T13:51:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.315+0000 7f10879f5640 1 -- 172.21.15.60:0/3269460388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800cd970 msgr2=0x7f10800d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.315+0000 7f10879f5640 1 --2- 172.21.15.60:0/3269460388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800cd970 0x7f10800d1630 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f1078009d90 tx=0x7f107802bf50 comp rx=0 tx=0).stop 2023-10-16T13:51:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.316+0000 7f10879f5640 1 -- 172.21.15.60:0/3269460388 shutdown_connections 2023-10-16T13:51:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.316+0000 7f10879f5640 1 --2- 172.21.15.60:0/3269460388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800cd970 0x7f10800d1630 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.316+0000 7f10879f5640 1 -- 172.21.15.60:0/3269460388 >> 172.21.15.60:0/3269460388 conn(0x7f10800cf720 msgr2=0x7f10800cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.316+0000 7f10879f5640 1 -- 172.21.15.60:0/3269460388 shutdown_connections 2023-10-16T13:51:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.316+0000 7f10879f5640 1 -- 172.21.15.60:0/3269460388 wait complete. 2023-10-16T13:51:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.317+0000 7f10879f5640 1 Processor -- start 2023-10-16T13:51:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.318+0000 7f10879f5640 1 -- start start 2023-10-16T13:51:39.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.318+0000 7f10879f5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108015d030 0x7f1080159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.318+0000 7f10879f5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108015d410 0x7f10800c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.318+0000 7f10879f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 0x7f10800c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f10879f5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800c8210 con 0x7f10800c8630 2023-10-16T13:51:39.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f10879f5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800cbab0 con 0x7f108015d410 2023-10-16T13:51:39.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f10879f5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10800c75b0 con 0x7f108015d030 2023-10-16T13:51:39.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 0x7f10800c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 0x7f10800c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53504/0 (socket says 172.21.15.60:53504) 2023-10-16T13:51:39.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 -- 172.21.15.60:0/1265729207 learned_addr learned my addr 172.21.15.60:0/1265729207 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:39.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1084f90640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108015d410 0x7f10800c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085791640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108015d030 0x7f1080159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 -- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108015d030 msgr2=0x7f1080159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108015d030 0x7f1080159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 -- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108015d410 msgr2=0x7f10800c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108015d410 0x7f10800c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.319+0000 7f1085f92640 1 -- 172.21.15.60:0/1265729207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078009a70 con 0x7f10800c8630 2023-10-16T13:51:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.320+0000 7f1085f92640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 0x7f10800c9270 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f1078009d90 tx=0x7f1078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.320+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107803e670 con 0x7f10800c8630 2023-10-16T13:51:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.320+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107803e7f0 con 0x7f10800c8630 2023-10-16T13:51:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.320+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107803eba0 con 0x7f10800c8630 2023-10-16T13:51:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.320+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10800c7870 con 0x7f10800c8630 2023-10-16T13:51:39.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.320+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10800c7e30 con 0x7f10800c8630 2023-10-16T13:51:39.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.321+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10800c8350 con 0x7f10800c8630 2023-10-16T13:51:39.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.322+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 98) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f107803b040 con 0x7f10800c8630 2023-10-16T13:51:39.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.323+0000 7f1075ffb640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 0x7f106406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.323+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10780ac4a0 con 0x7f10800c8630 2023-10-16T13:51:39.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.324+0000 7f1085791640 1 -- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 msgr2=0x7f106406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:39.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.324+0000 7f1085791640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 0x7f106406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:39.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.327+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10800c8350 con 0x7f10800c8630 2023-10-16T13:51:39.515 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.511+0000 7f50fba18040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:51:39.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.522+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f10800c8350 con 0x7f10800c8630 2023-10-16T13:51:39.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.525+0000 7f1085791640 1 -- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 msgr2=0x7f106406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:39.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.525+0000 7f1085791640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 0x7f106406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:39.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.528+0000 7f1075ffb640 1 -- 172.21.15.60:0/1265729207 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f10800c8350 con 0x7f10800c8630 2023-10-16T13:51:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 msgr2=0x7f106406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 0x7f106406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 msgr2=0x7f10800c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 0x7f10800c9270 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f1078009d90 tx=0x7f1078004a40 comp rx=0 tx=0).stop 2023-10-16T13:51:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 shutdown_connections 2023-10-16T13:51:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7f106406afe0 0x7f106406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10800c8630 0x7f10800c9270 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f108015d410 0x7f10800c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 --2- 172.21.15.60:0/1265729207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f108015d030 0x7f1080159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 >> 172.21.15.60:0/1265729207 conn(0x7f10800cf720 msgr2=0x7f10800b7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.531+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 shutdown_connections 2023-10-16T13:51:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.532+0000 7f10879f5640 1 -- 172.21.15.60:0/1265729207 wait complete. 2023-10-16T13:51:39.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:39.552 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T13:51:39.568 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.565+0000 7f50fba18040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:51:39.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:51:39.611 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-16T13:51:39.619 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.616+0000 7f50fba18040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:51:39.661 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-10-16T13:51:39.679 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.675+0000 7f50fba18040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:51:39.708 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.705+0000 7f969e812040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:51:39.727 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.724+0000 7f50fba18040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:51:39.764 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.761+0000 7f969e812040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:51:39.818 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.815+0000 7f969e812040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:51:39.836 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.833+0000 7f50fba18040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:51:39.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.859+0000 7fa157fff640 1 Processor -- start 2023-10-16T13:51:39.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.859+0000 7fa157fff640 1 -- start start 2023-10-16T13:51:39.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa157fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa15815aa60 0x7fa1580af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa157fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa1580afd60 0x7fa1580cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa157fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1580b3730 con 0x7fa1580cc820 2023-10-16T13:51:39.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa157fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580b3980 con 0x7fa15815aa60 2023-10-16T13:51:39.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa157fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580b3c60 con 0x7fa1580afd60 2023-10-16T13:51:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa156ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa15815aa60 0x7fa1580af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa156ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa15815aa60 0x7fa1580af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53514/0 (socket says 172.21.15.60:53514) 2023-10-16T13:51:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa156ffd640 1 -- 172.21.15.60:0/2053467891 learned_addr learned my addr 172.21.15.60:0/2053467891 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.861+0000 7fa14ffff640 1 --2- 172.21.15.60:0/2053467891 >> v2:172.21.15.144:3300/0 conn(0x7fa1580afd60 0x7fa1580cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.862+0000 7fa156ffd640 1 -- 172.21.15.60:0/2053467891 >> v2:172.21.15.144:3300/0 conn(0x7fa1580afd60 msgr2=0x7fa1580cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:39.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.862+0000 7fa156ffd640 1 --2- 172.21.15.60:0/2053467891 >> v2:172.21.15.144:3300/0 conn(0x7fa1580afd60 0x7fa1580cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.862+0000 7fa156ffd640 1 -- 172.21.15.60:0/2053467891 >> v1:172.21.15.60:6789/0 conn(0x7fa1580cc820 legacy=0x7fa1580b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.862+0000 7fa156ffd640 1 -- 172.21.15.60:0/2053467891 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580b3f40 con 0x7fa15815aa60 2023-10-16T13:51:39.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.863+0000 7fa156ffd640 1 --2- 172.21.15.60:0/2053467891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580af820 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fa1580d1610 tx=0x7fa14802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9718df01677925f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:39.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa1567fc640 1 -- 172.21.15.60:0/2053467891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148030870 con 0x7fa15815aa60 2023-10-16T13:51:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa1567fc640 1 -- 172.21.15.60:0/2053467891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1480309d0 con 0x7fa15815aa60 2023-10-16T13:51:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa1567fc640 1 -- 172.21.15.60:0/2053467891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148030d60 con 0x7fa15815aa60 2023-10-16T13:51:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa157fff640 1 -- 172.21.15.60:0/2053467891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 msgr2=0x7fa1580af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa157fff640 1 --2- 172.21.15.60:0/2053467891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580af820 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fa1580d1610 tx=0x7fa14802b9b0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa157fff640 1 -- 172.21.15.60:0/2053467891 shutdown_connections 2023-10-16T13:51:39.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa157fff640 1 --2- 172.21.15.60:0/2053467891 >> v2:172.21.15.144:3300/0 conn(0x7fa1580afd60 0x7fa1580cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa157fff640 1 --2- 172.21.15.60:0/2053467891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580af820 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.864+0000 7fa157fff640 1 -- 172.21.15.60:0/2053467891 >> 172.21.15.60:0/2053467891 conn(0x7fa158065ed0 msgr2=0x7fa1580662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:39.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.865+0000 7fa157fff640 1 -- 172.21.15.60:0/2053467891 shutdown_connections 2023-10-16T13:51:39.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.865+0000 7fa157fff640 1 -- 172.21.15.60:0/2053467891 wait complete. 2023-10-16T13:51:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.866+0000 7fa157fff640 1 Processor -- start 2023-10-16T13:51:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.866+0000 7fa157fff640 1 -- start start 2023-10-16T13:51:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.867+0000 7fa157fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1580afd60 0x7fa1580af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.867+0000 7fa157fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580cc820 0x7fa1580addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.867+0000 7fa157fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.867+0000 7fa157fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580af780 con 0x7fa15815aa60 2023-10-16T13:51:39.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.867+0000 7fa157fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15814b470 con 0x7fa1580cc820 2023-10-16T13:51:39.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.867+0000 7fa157fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580acd20 con 0x7fa1580afd60 2023-10-16T13:51:39.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580b1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53526/0 (socket says 172.21.15.60:53526) 2023-10-16T13:51:39.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 -- 172.21.15.60:0/277872382 learned_addr learned my addr 172.21.15.60:0/277872382 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:39.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa14ffff640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580cc820 0x7fa1580addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1580afd60 0x7fa1580af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:39.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1580afd60 msgr2=0x7fa1580af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1580afd60 0x7fa1580af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580cc820 msgr2=0x7fa1580addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580cc820 0x7fa1580addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa1577fe640 1 -- 172.21.15.60:0/277872382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1480094d0 con 0x7fa15815aa60 2023-10-16T13:51:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.868+0000 7fa14ffff640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1580cc820 0x7fa1580addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:51:39.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.869+0000 7fa1577fe640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580b1e30 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fa15000c7a0 tx=0x7fa15000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:39.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.869+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150003b70 con 0x7fa15815aa60 2023-10-16T13:51:39.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.869+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa150003cf0 con 0x7fa15815aa60 2023-10-16T13:51:39.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.869+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1580ad000 con 0x7fa15815aa60 2023-10-16T13:51:39.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.869+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa150003ea0 con 0x7fa15815aa60 2023-10-16T13:51:39.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.869+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1580ad5c0 con 0x7fa15815aa60 2023-10-16T13:51:39.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.870+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15804e9b0 con 0x7fa15815aa60 2023-10-16T13:51:39.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.871+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 98) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fa15000aa90 con 0x7fa15815aa60 2023-10-16T13:51:39.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.873+0000 7fa14f7fe640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 0x7fa13806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:39.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.873+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1500896f0 con 0x7fa15815aa60 2023-10-16T13:51:39.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.873+0000 7fa156ffd640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 msgr2=0x7fa13806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:39.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.873+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 0x7fa13806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:51:39.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:39.877+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1500545f0 con 0x7fa15815aa60 2023-10-16T13:51:39.885 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.882+0000 7f969e812040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:51:39.899 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.896+0000 7f50fba18040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:39.937 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:39.934+0000 7f969e812040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:51:39.948 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:39.944+0000 7f50fba18040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:51:40.005 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.001+0000 7f50fba18040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:51:40.055 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.051+0000 7f969e812040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:51:40.055 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.051+0000 7f50fba18040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:51:40.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:40.074+0000 7fa156ffd640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 msgr2=0x7fa13806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:40.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:40.074+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 0x7fa13806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:51:40.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:40.097+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 --> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1580ccc90 con 0x7fa13806afe0 2023-10-16T13:51:40.121 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.118+0000 7f969e812040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:40.173 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.170+0000 7f969e812040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:51:40.232 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.229+0000 7f969e812040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:51:40.284 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.281+0000 7f969e812040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:51:40.434 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.431+0000 7f50fba18040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:51:40.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:40.475+0000 7fa156ffd640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 msgr2=0x7fa13806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:40.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:40.475+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 0x7fa13806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:51:40.595 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.592+0000 7f50fba18040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:51:40.684 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.681+0000 7f969e812040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:51:40.702 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.698+0000 7f50fba18040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:51:40.752 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.749+0000 7f50fba18040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:51:40.852 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.848+0000 7f969e812040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:51:40.858 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.855+0000 7f50fba18040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:40.964 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:40.961+0000 7f50fba18040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:51:40.965 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:40.961+0000 7f969e812040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:51:41.017 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:41.014+0000 7f969e812040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:51:41.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:41.116+0000 7f50fba18040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:41.129 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:41.126+0000 7f969e812040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:41.169 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:41.165+0000 7f50fba18040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:41.241 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:41.237+0000 7f969e812040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:51:41.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:41.276+0000 7fa156ffd640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 msgr2=0x7fa13806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1700115305 2023-10-16T13:51:41.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:41.276+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 0x7fa13806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:51:41.287 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:41.283+0000 7f50fba18040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:51:41.406 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:41.402+0000 7f969e812040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:51:41.457 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:41.454+0000 7f969e812040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:51:41.582 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:41.578+0000 7f969e812040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:51:42.079 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.076+0000 7f50fba18040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:51:42.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.369+0000 7f50fba18040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:51:42.430 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.426+0000 7f50fba18040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:51:42.475 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:42.472+0000 7f969e812040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:51:42.542 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.539+0000 7f50fba18040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:51:42.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:42.559+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 99) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fa15000ad40 con 0x7fa15815aa60 2023-10-16T13:51:42.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:42.560+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 msgr2=0x7fa13806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:51:42.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:42.560+0000 7fa14f7fe640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/1700115305,v1:172.21.15.144:6833/1700115305] conn(0x7fa13806afe0 0x7fa13806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:42.761 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.757+0000 7f507b079640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.757+0000 7f507b079640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.757+0000 7f507b079640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.757+0000 7f507b079640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.757+0000 7f507b079640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.758+0000 7f5078033640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.763 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.758+0000 7f5078033640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.763 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.758+0000 7f5078033640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.763 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.758+0000 7f5078033640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.763 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:51:42.758+0000 7f5078033640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:51:42.796 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:42.793+0000 7f969e812040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:51:42.856 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:42.852+0000 7f969e812040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:51:42.975 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T13:51:42.972+0000 7f969e812040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:51:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.570+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 100) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa1500549c0 con 0x7fa15815aa60 2023-10-16T13:51:43.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.572+0000 7fa14f7fe640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fa138035880 0x7fa138074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:43.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.572+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 --> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa138035c60 con 0x7fa138035880 2023-10-16T13:51:43.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.582+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fa138035880 0x7fa138074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:43.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.591+0000 7fa156ffd640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fa138035880 0x7fa138074ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa148036000 tx=0x7fa14803b040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.751+0000 7fa14f7fe640 1 -- 172.21.15.60:0/277872382 <== mgr.7223 v2:172.21.15.144:6832/861800364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa138035c60 con 0x7fa138035880 2023-10-16T13:51:43.755 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-16T13:51:43.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.754+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fa138035880 msgr2=0x7fa138074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:43.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.754+0000 7fa157fff640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fa138035880 0x7fa138074ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa148036000 tx=0x7fa14803b040 comp rx=0 tx=0).stop 2023-10-16T13:51:43.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.754+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 msgr2=0x7fa1580b1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.754+0000 7fa157fff640 1 --2- 172.21.15.60:0/277872382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa15815aa60 0x7fa1580b1e30 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fa15000c7a0 tx=0x7fa15000cc60 comp rx=0 tx=0).stop 2023-10-16T13:51:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.755+0000 7fa1577fe640 1 -- 172.21.15.60:0/277872382 reap_dead start 2023-10-16T13:51:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.755+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 shutdown_connections 2023-10-16T13:51:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.755+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 >> 172.21.15.60:0/277872382 conn(0x7fa158065ed0 msgr2=0x7fa1580cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:43.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.755+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 shutdown_connections 2023-10-16T13:51:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:43.755+0000 7fa157fff640 1 -- 172.21.15.60:0/277872382 wait complete. 2023-10-16T13:51:43.780 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-16T13:51:43.780 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-10-16T13:51:44.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.057+0000 7fd79261a640 1 Processor -- start 2023-10-16T13:51:44.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.057+0000 7fd79261a640 1 -- start start 2023-10-16T13:51:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.058+0000 7fd79261a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd78c06c040 0x7fd78c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.058+0000 7fd79261a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd78c0ad970 0x7fd78c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.058+0000 7fd79261a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c15ebc0 0x7fd78c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.058+0000 7fd79261a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78c06c960 con 0x7fd78c0ad970 2023-10-16T13:51:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.058+0000 7fd79261a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78c15b9d0 con 0x7fd78c06c040 2023-10-16T13:51:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.058+0000 7fd79261a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c15bcb0 con 0x7fd78c15ebc0 2023-10-16T13:51:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd78c0ad970 0x7fd78c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd78c0ad970 0x7fd78c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53538/0 (socket says 172.21.15.60:53538) 2023-10-16T13:51:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 -- 172.21.15.60:0/940039745 learned_addr learned my addr 172.21.15.60:0/940039745 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd791618640 1 --2- 172.21.15.60:0/940039745 >> v2:172.21.15.144:3300/0 conn(0x7fd78c06c040 0x7fd78c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd791e19640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c15ebc0 0x7fd78c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 -- 172.21.15.60:0/940039745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c15ebc0 msgr2=0x7fd78c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:44.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c15ebc0 0x7fd78c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 -- 172.21.15.60:0/940039745 >> v2:172.21.15.144:3300/0 conn(0x7fd78c06c040 msgr2=0x7fd78c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:51:44.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 --2- 172.21.15.60:0/940039745 >> v2:172.21.15.144:3300/0 conn(0x7fd78c06c040 0x7fd78c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd790e17640 1 -- 172.21.15.60:0/940039745 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c15bf90 con 0x7fd78c0ad970 2023-10-16T13:51:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.059+0000 7fd791e19640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c15ebc0 0x7fd78c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:51:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.060+0000 7fd790e17640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c15e680 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0cf350 tx=0x7fd77c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1509955a7c32d2e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.061+0000 7fd783fff640 1 -- 172.21.15.60:0/940039745 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c031870 con 0x7fd78c0ad970 2023-10-16T13:51:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.061+0000 7fd783fff640 1 -- 172.21.15.60:0/940039745 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c0319d0 con 0x7fd78c0ad970 2023-10-16T13:51:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.061+0000 7fd783fff640 1 -- 172.21.15.60:0/940039745 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c031d60 con 0x7fd78c0ad970 2023-10-16T13:51:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.061+0000 7fd79261a640 1 -- 172.21.15.60:0/940039745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 msgr2=0x7fd78c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.061+0000 7fd79261a640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c15e680 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0cf350 tx=0x7fd77c02b910 comp rx=0 tx=0).stop 2023-10-16T13:51:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 -- 172.21.15.60:0/940039745 shutdown_connections 2023-10-16T13:51:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c15ebc0 0x7fd78c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 --2- 172.21.15.60:0/940039745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c15e680 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 --2- 172.21.15.60:0/940039745 >> v2:172.21.15.144:3300/0 conn(0x7fd78c06c040 0x7fd78c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 -- 172.21.15.60:0/940039745 >> 172.21.15.60:0/940039745 conn(0x7fd78c0ce6a0 msgr2=0x7fd78c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 -- 172.21.15.60:0/940039745 shutdown_connections 2023-10-16T13:51:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.062+0000 7fd79261a640 1 -- 172.21.15.60:0/940039745 wait complete. 2023-10-16T13:51:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.063+0000 7fd79261a640 1 Processor -- start 2023-10-16T13:51:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.064+0000 7fd79261a640 1 -- start start 2023-10-16T13:51:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd79261a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c06c040 0x7fd78c14d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd79261a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c0c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd79261a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c15ebc0 0x7fd78c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd79261a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0cadb0 con 0x7fd78c0ad970 2023-10-16T13:51:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd79261a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0ca920 con 0x7fd78c15ebc0 2023-10-16T13:51:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd79261a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c14deb0 con 0x7fd78c06c040 2023-10-16T13:51:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c0c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c0c72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53552/0 (socket says 172.21.15.60:53552) 2023-10-16T13:51:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 -- 172.21.15.60:0/1232639915 learned_addr learned my addr 172.21.15.60:0/1232639915 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T13:51:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd791618640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c06c040 0x7fd78c14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 -- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c06c040 msgr2=0x7fd78c14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c06c040 0x7fd78c14d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 -- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c15ebc0 msgr2=0x7fd78c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c15ebc0 0x7fd78c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.065+0000 7fd790e17640 1 -- 172.21.15.60:0/1232639915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c0094d0 con 0x7fd78c0ad970 2023-10-16T13:51:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.066+0000 7fd790e17640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c0c72d0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0097c0 tx=0x7fd77c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.066+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c03ac60 con 0x7fd78c0ad970 2023-10-16T13:51:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.066+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c03adc0 con 0x7fd78c0ad970 2023-10-16T13:51:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.066+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c0c9d10 con 0x7fd78c0ad970 2023-10-16T13:51:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.066+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c044610 con 0x7fd78c0ad970 2023-10-16T13:51:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.066+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c0ca2d0 con 0x7fd78c0ad970 2023-10-16T13:51:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.067+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c06b330 con 0x7fd78c0ad970 2023-10-16T13:51:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.068+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 100) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd77c04d450 con 0x7fd78c0ad970 2023-10-16T13:51:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.070+0000 7fd781ffb640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fd77006ad10 0x7fd77006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:51:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.070+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd77c02d040 con 0x7fd78c0ad970 2023-10-16T13:51:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.070+0000 7fd791618640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fd77006ad10 0x7fd77006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:51:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.071+0000 7fd791618640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fd77006ad10 0x7fd77006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd788006fd0 tx=0x7fd788008040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:51:44.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.074+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd77c076c60 con 0x7fd78c0ad970 2023-10-16T13:51:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.284+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 --> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd78c0cc550 con 0x7fd77006ad10 2023-10-16T13:51:44.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.418+0000 7fd781ffb640 1 -- 172.21.15.60:0/1232639915 <== mgr.7223 v2:172.21.15.144:6832/861800364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fd78800e510 con 0x7fd77006ad10 2023-10-16T13:51:44.422 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T13:51:44.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fd77006ad10 msgr2=0x7fd77006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:44.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fd77006ad10 0x7fd77006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd788006fd0 tx=0x7fd788008040 comp rx=0 tx=0).stop 2023-10-16T13:51:44.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 msgr2=0x7fd78c0c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:51:44.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c0c72d0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0097c0 tx=0x7fd77c0046c0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 shutdown_connections 2023-10-16T13:51:44.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7fd77006ad10 0x7fd77006d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c15ebc0 0x7fd78c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd78c0ad970 0x7fd78c0c72d0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 --2- 172.21.15.60:0/1232639915 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd78c06c040 0x7fd78c14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:51:44.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 >> 172.21.15.60:0/1232639915 conn(0x7fd78c0ce6a0 msgr2=0x7fd78c0cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:51:44.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 shutdown_connections 2023-10-16T13:51:44.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T13:51:44.421+0000 7fd79261a640 1 -- 172.21.15.60:0/1232639915 wait complete. 2023-10-16T13:51:44.446 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T13:51:50.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:51:50.676 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:52:00.444 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:52:00.442+0000 7f506a017640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:52:00.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:52:00.442+0000 7f506a017640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:52:00.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:52:00.442+0000 7f506a017640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:52:00.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:52:00.442+0000 7f506a017640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:52:00.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T13:52:00.442+0000 7f506a017640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:52:20.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:52:20.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:52:50.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:52:50.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:53:21.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:53:21.142 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:53:51.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:53:51.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:54:21.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:54:21.443 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:54:51.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:54:51.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:55:21.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:55:21.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:55:52.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:55:52.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:56:22.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:56:22.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:56:52.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:56:52.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:57:22.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:57:22.602 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:57:52.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:57:52.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:58:23.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:58:23.048 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:58:53.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:58:53.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:59:23.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:59:23.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:59:53.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:59:53.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:00:23.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:00:23.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:00:54.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:00:54.095 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:01:24.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:01:24.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:01:54.386 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:01:54.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:02:16.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:02:16.448 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T14:02:16.517 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_52_00_UTC 2023-10-16T14:02:16.517 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_53_00_UTC 2023-10-16T14:02:16.518 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_54_00_UTC 2023-10-16T14:02:16.518 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_55_00_UTC 2023-10-16T14:02:16.518 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_56_00_UTC 2023-10-16T14:02:16.518 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_57_00_UTC 2023-10-16T14:02:16.518 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_58_00_UTC 2023-10-16T14:02:16.519 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_59_00_UTC 2023-10-16T14:02:16.519 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_00_00_UTC 2023-10-16T14:02:16.519 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_01_00_UTC 2023-10-16T14:02:16.519 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_02_00_UTC 2023-10-16T14:02:16.520 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-13_52_00_UTC', 'scheduled-2023-10-16-13_53_00_UTC', 'scheduled-2023-10-16-13_54_00_UTC', 'scheduled-2023-10-16-13_55_00_UTC', 'scheduled-2023-10-16-13_56_00_UTC', 'scheduled-2023-10-16-13_57_00_UTC', 'scheduled-2023-10-16-13_58_00_UTC', 'scheduled-2023-10-16-13_59_00_UTC', 'scheduled-2023-10-16-14_00_00_UTC', 'scheduled-2023-10-16-14_01_00_UTC', 'scheduled-2023-10-16-14_02_00_UTC'] 2023-10-16T14:02:16.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T14:02:16.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.747+0000 7f27ce565640 1 Processor -- start 2023-10-16T14:02:16.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.747+0000 7f27ce565640 1 -- start start 2023-10-16T14:02:16.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.749+0000 7f27ce565640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8070b60 0x7f27c815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:16.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.749+0000 7f27ce565640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27c806c100 0x7f27c806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:16.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.749+0000 7f27ce565640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c815aef0 con 0x7f27c8159950 2023-10-16T14:02:16.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.749+0000 7f27ce565640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c815e490 con 0x7f27c806c100 2023-10-16T14:02:16.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.749+0000 7f27ce565640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c815d910 con 0x7f27c8070b60 2023-10-16T14:02:16.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cd563640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f27c8159950 0x7f27c8070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39728/0 (socket says 172.21.15.60:39728) 2023-10-16T14:02:16.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cd563640 1 -- 172.21.15.60:0/1742801485 learned_addr learned my addr 172.21.15.60:0/1742801485 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:02:16.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cdd64640 1 --2- 172.21.15.60:0/1742801485 >> v2:172.21.15.60:3300/0 conn(0x7f27c806c100 0x7f27c806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:16.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27ccd62640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8070b60 0x7f27c815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:16.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cdd64640 1 -- 172.21.15.60:0/1742801485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8070b60 msgr2=0x7f27c815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:02:16.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cdd64640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8070b60 0x7f27c815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:16.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cdd64640 1 -- 172.21.15.60:0/1742801485 >> v1:172.21.15.60:6789/0 conn(0x7f27c8159950 legacy=0x7f27c8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:16.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27cdd64640 1 -- 172.21.15.60:0/1742801485 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c815dbf0 con 0x7f27c806c100 2023-10-16T14:02:16.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.750+0000 7f27ccd62640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8070b60 0x7f27c815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:02:16.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.752+0000 7f27cdd64640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c806c550 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009dc0 tx=0x7f27c002bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf772f8f55d67c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:02:16.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.752+0000 7f27bffff640 1 -- 172.21.15.60:0/1742801485 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c002cba0 con 0x7f27c806c100 2023-10-16T14:02:16.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.752+0000 7f27bffff640 1 -- 172.21.15.60:0/1742801485 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c002cd00 con 0x7f27c806c100 2023-10-16T14:02:16.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.752+0000 7f27bffff640 1 -- 172.21.15.60:0/1742801485 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c0034660 con 0x7f27c806c100 2023-10-16T14:02:16.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 -- 172.21.15.60:0/1742801485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 msgr2=0x7f27c806c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:16.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c806c550 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009dc0 tx=0x7f27c002bdf0 comp rx=0 tx=0).stop 2023-10-16T14:02:16.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 -- 172.21.15.60:0/1742801485 shutdown_connections 2023-10-16T14:02:16.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c806c550 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:16.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 --2- 172.21.15.60:0/1742801485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8070b60 0x7f27c815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:16.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 -- 172.21.15.60:0/1742801485 >> 172.21.15.60:0/1742801485 conn(0x7f27c80ce6a0 msgr2=0x7f27c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:02:16.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 -- 172.21.15.60:0/1742801485 shutdown_connections 2023-10-16T14:02:16.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.753+0000 7f27ce565640 1 -- 172.21.15.60:0/1742801485 wait complete. 2023-10-16T14:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.756+0000 7f27ce565640 1 Processor -- start 2023-10-16T14:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.756+0000 7f27ce565640 1 -- start start 2023-10-16T14:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ce565640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c81658f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ce565640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27c8070b60 0x7f27c814eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ce565640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8159950 0x7f27c80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ce565640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c814f080 con 0x7f27c806c100 2023-10-16T14:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ce565640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c8165ed0 con 0x7f27c8070b60 2023-10-16T14:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ce565640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c81661b0 con 0x7f27c8159950 2023-10-16T14:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c81658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c81658f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51628/0 (socket says 172.21.15.60:51628) 2023-10-16T14:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 -- 172.21.15.60:0/84512398 learned_addr learned my addr 172.21.15.60:0/84512398 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27ccd62640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27c8070b60 0x7f27c814eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cdd64640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8159950 0x7f27c80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:16.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 -- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8159950 msgr2=0x7f27c80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:16.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8159950 0x7f27c80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:16.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 -- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27c8070b60 msgr2=0x7f27c814eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:16.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27c8070b60 0x7f27c814eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:16.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.757+0000 7f27cd563640 1 -- 172.21.15.60:0/84512398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c0009a70 con 0x7f27c806c100 2023-10-16T14:02:16.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27cdd64640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8159950 0x7f27c80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:02:16.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27cd563640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c81658f0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c480 tx=0x7f27c400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:02:16.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4005ce0 con 0x7f27c806c100 2023-10-16T14:02:16.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4005e40 con 0x7f27c806c100 2023-10-16T14:02:16.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40072f0 con 0x7f27c806c100 2023-10-16T14:02:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c8166490 con 0x7f27c806c100 2023-10-16T14:02:16.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.758+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c8166a50 con 0x7f27c806c100 2023-10-16T14:02:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.760+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c8071430 con 0x7f27c806c100 2023-10-16T14:02:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.761+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 104) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f27c4007510 con 0x7f27c806c100 2023-10-16T14:02:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.761+0000 7f27bdffb640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f27ac06afe0 0x7f27ac06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.762+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f27c4088cd0 con 0x7f27c806c100 2023-10-16T14:02:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.762+0000 7f27ccd62640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f27ac06afe0 0x7f27ac06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.764+0000 7f27ccd62640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f27ac06afe0 0x7f27ac06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f27c81f6130 tx=0x7f27b8008040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:02:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.767+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27c4053bb0 con 0x7f27c806c100 2023-10-16T14:02:16.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:16.986+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 --> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27c815e1b0 con 0x7f27ac06afe0 2023-10-16T14:02:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.005+0000 7f27bdffb640 1 -- 172.21.15.60:0/84512398 <== mgr.7223 v2:172.21.15.144:6832/861800364 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f27b800a660 con 0x7f27ac06afe0 2023-10-16T14:02:17.008 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:02:17.008 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:51:43", "first": "2023-10-16T13:52:00", "last": "2023-10-16T14:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T14:02:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f27ac06afe0 msgr2=0x7f27ac06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f27ac06afe0 0x7f27ac06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f27c81f6130 tx=0x7f27b8008040 comp rx=0 tx=0).stop 2023-10-16T14:02:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 msgr2=0x7f27c81658f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c81658f0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c480 tx=0x7f27c400c940 comp rx=0 tx=0).stop 2023-10-16T14:02:17.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 shutdown_connections 2023-10-16T14:02:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f27ac06afe0 0x7f27ac06d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27c8159950 0x7f27c80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27c8070b60 0x7f27c814eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 --2- 172.21.15.60:0/84512398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27c806c100 0x7f27c81658f0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 >> 172.21.15.60:0/84512398 conn(0x7f27c80ce6a0 msgr2=0x7f27c8153100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:02:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 shutdown_connections 2023-10-16T14:02:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.008+0000 7f27ce565640 1 -- 172.21.15.60:0/84512398 wait complete. 2023-10-16T14:02:17.032 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:51:43", "first": "2023-10-16T13:52:00", "last": "2023-10-16T14:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T14:02:17.032 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T13:51:43', 'first': '2023-10-16T13:52:00', 'last': '2023-10-16T14:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-16T14:02:17.032 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T14:02:17.033 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T14:02:17.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T14:02:17.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.262+0000 7f55a1fcc640 1 Processor -- start 2023-10-16T14:02:17.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.263+0000 7f55a1fcc640 1 -- start start 2023-10-16T14:02:17.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.264+0000 7f55a1fcc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f559c15c850 0x7f559c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.264+0000 7f55a1fcc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c06ffe0 con 0x7f559c06b4c0 2023-10-16T14:02:17.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.264+0000 7f55a1fcc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0cb9e0 con 0x7f559c0cdc40 2023-10-16T14:02:17.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.264+0000 7f55a1fcc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559c0ccf40 con 0x7f559c15c850 2023-10-16T14:02:17.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.264+0000 7f559bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f559c06b4c0 0x7f559c0c9890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39738/0 (socket says 172.21.15.60:39738) 2023-10-16T14:02:17.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.264+0000 7f559bfff640 1 -- 172.21.15.60:0/2337602521 learned_addr learned my addr 172.21.15.60:0/2337602521 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:02:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559b7fe640 1 --2- 172.21.15.60:0/2337602521 >> v2:172.21.15.144:3300/0 conn(0x7f559c15c850 0x7f559c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143618529 0 0) 0x7f559c06ffe0 con 0x7f559c06b4c0 2023-10-16T14:02:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5588003520 con 0x7f559c06b4c0 2023-10-16T14:02:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 983399604 0 0) 0x7f5588003520 con 0x7f559c06b4c0 2023-10-16T14:02:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 >> v2:172.21.15.144:3300/0 conn(0x7f559c15c850 msgr2=0x7f559c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:02:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559a7fc640 1 --2- 172.21.15.60:0/2337602521 >> v2:172.21.15.144:3300/0 conn(0x7f559c15c850 0x7f559c0cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.265+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 >> v1:172.21.15.144:6789/0 conn(0x7f559c0cdc40 legacy=0x7f559c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:02:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0cb420 con 0x7f559c06b4c0 2023-10-16T14:02:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5594001670 con 0x7f559c06b4c0 2023-10-16T14:02:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 >> v1:172.21.15.60:6789/0 conn(0x7f559c06b4c0 legacy=0x7f559c0c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5588003970 0x7f5588003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55880047c0 0x7f5588004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880052a0 0x7f558800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c0ccf40 con 0x7f55880047c0 2023-10-16T14:02:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558800c170 con 0x7f55880052a0 2023-10-16T14:02:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558800c330 con 0x7f5588003970 2023-10-16T14:02:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559affd640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55880047c0 0x7f5588004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559bfff640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880052a0 0x7f558800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.266+0000 7f559b7fe640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5588003970 0x7f5588003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.267+0000 7f559affd640 1 -- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5588003970 msgr2=0x7f5588003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:02:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.267+0000 7f559affd640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5588003970 0x7f5588003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.267+0000 7f559affd640 1 -- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880052a0 msgr2=0x7f558800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:02:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.267+0000 7f559affd640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880052a0 0x7f558800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.267+0000 7f559affd640 1 -- 172.21.15.60:0/2337602521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588003520 con 0x7f55880047c0 2023-10-16T14:02:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.267+0000 7f559bfff640 1 -- 172.21.15.60:0/2337602521 reap_dead start 2023-10-16T14:02:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.268+0000 7f559affd640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55880047c0 0x7f5588004c40 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f5584001720 tx=0x7f55880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1046b5fe6392296a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:02:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.268+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55840052d0 con 0x7f55880047c0 2023-10-16T14:02:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.268+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558402e050 con 0x7f55880047c0 2023-10-16T14:02:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.268+0000 7f559a7fc640 1 -- 172.21.15.60:0/2337602521 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584003830 con 0x7f55880047c0 2023-10-16T14:02:17.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 -- 172.21.15.60:0/2337602521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55880047c0 msgr2=0x7f5588004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55880047c0 0x7f5588004c40 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f5584001720 tx=0x7f55880057f0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 -- 172.21.15.60:0/2337602521 shutdown_connections 2023-10-16T14:02:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/2337602521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55880047c0 0x7f5588004c40 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 -- 172.21.15.60:0/2337602521 >> 172.21.15.60:0/2337602521 conn(0x7f559c0b0d30 msgr2=0x7f559c0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:02:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 -- 172.21.15.60:0/2337602521 shutdown_connections 2023-10-16T14:02:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.269+0000 7f55a1fcc640 1 -- 172.21.15.60:0/2337602521 wait complete. 2023-10-16T14:02:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.271+0000 7f55a1fcc640 1 Processor -- start 2023-10-16T14:02:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.271+0000 7f55a1fcc640 1 -- start start 2023-10-16T14:02:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f55a1fcc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880047c0 0x7f559c14bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f55a1fcc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 0x7f559c0b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f55a1fcc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f559c0b2070 0x7f559c0b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f55a1fcc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c09b330 con 0x7f559c0c5150 2023-10-16T14:02:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f55a1fcc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c09aa60 con 0x7f55880047c0 2023-10-16T14:02:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f55a1fcc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c0b24f0 con 0x7f559c0b2070 2023-10-16T14:02:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 0x7f559c0b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 0x7f559c0b1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51638/0 (socket says 172.21.15.60:51638) 2023-10-16T14:02:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 -- 172.21.15.60:0/3768596884 learned_addr learned my addr 172.21.15.60:0/3768596884 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:02:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559bfff640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f559c0b2070 0x7f559c0b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559b7fe640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880047c0 0x7f559c14bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 -- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f559c0b2070 msgr2=0x7f559c0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f559c0b2070 0x7f559c0b3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 -- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880047c0 msgr2=0x7f559c14bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880047c0 0x7f559c14bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.272+0000 7f559affd640 1 -- 172.21.15.60:0/3768596884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0b4d30 con 0x7f559c0c5150 2023-10-16T14:02:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.273+0000 7f559affd640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 0x7f559c0b1b30 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f558402bbc0 tx=0x7f5584001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:02:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.273+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584007640 con 0x7f559c0c5150 2023-10-16T14:02:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.273+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55840069e0 con 0x7f559c0c5150 2023-10-16T14:02:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.273+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c09ac70 con 0x7f559c0c5150 2023-10-16T14:02:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.273+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584006d70 con 0x7f559c0c5150 2023-10-16T14:02:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.273+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c0c55d0 con 0x7f559c0c5150 2023-10-16T14:02:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.274+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c04e160 con 0x7f559c0c5150 2023-10-16T14:02:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.275+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 104) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f55840035f0 con 0x7f559c0c5150 2023-10-16T14:02:17.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.276+0000 7f5593fff640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f557c06afe0 0x7f557c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:02:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.276+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f55840ab7e0 con 0x7f559c0c5150 2023-10-16T14:02:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.277+0000 7f559b7fe640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f557c06afe0 0x7f557c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:02:17.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.278+0000 7f559b7fe640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f557c06afe0 0x7f557c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f559c069740 tx=0x7f558c00b040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:02:17.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.281+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5584076500 con 0x7f559c0c5150 2023-10-16T14:02:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.524+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f559c0ce470 con 0x7f559c0c5150 2023-10-16T14:02:17.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.533+0000 7f5593fff640 1 -- 172.21.15.60:0/3768596884 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v104) v1 ==== 74+0+171244 (secure 0 0 0) 0x7f5584079b20 con 0x7f559c0c5150 2023-10-16T14:02:17.539 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:02:17.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f557c06afe0 msgr2=0x7f557c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f557c06afe0 0x7f557c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f559c069740 tx=0x7f558c00b040 comp rx=0 tx=0).stop 2023-10-16T14:02:17.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 msgr2=0x7f559c0b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:02:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 0x7f559c0b1b30 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f558402bbc0 tx=0x7f5584001120 comp rx=0 tx=0).stop 2023-10-16T14:02:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 shutdown_connections 2023-10-16T14:02:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:6832/861800364,v1:172.21.15.144:6833/861800364] conn(0x7f557c06afe0 0x7f557c06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f559c0b2070 0x7f559c0b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f559c0c5150 0x7f559c0b1b30 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 --2- 172.21.15.60:0/3768596884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55880047c0 0x7f559c14bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:02:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.537+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 >> 172.21.15.60:0/3768596884 conn(0x7f559c0b0d30 msgr2=0x7f559c09f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:02:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.538+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 shutdown_connections 2023-10-16T14:02:17.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:02:17.538+0000 7f55a1fcc640 1 -- 172.21.15.60:0/3768596884 wait complete. 2023-10-16T14:02:17.561 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"active_gid":7223,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":861800364},{"type":"v1","addr":"172.21.15.144:6833","nonce":861800364}]},"active_addr":"172.21.15.144:6833/861800364","active_change":"2023-10-16T13:51:42.545288+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":8197,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":198,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1022250268}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1287992351}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1895637100}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1247236577}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3061731632}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":1129229162}]}]} 2023-10-16T14:02:17.567 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-10-16T14:02:17.567 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T14:02:17.568 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-10-16T14:02:17.568 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T14:02:17.569 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:02:17.654 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T14:02:17.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T14:02:17.664 INFO:tasks.ceph.mgr.x:Started 2023-10-16T14:02:18.286 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.284+0000 7faae0e3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:02:18.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.337+0000 7faae0e3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:02:18.780 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.777+0000 7faae0e3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:02:18.832 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.829+0000 7faae0e3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:02:18.882 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.879+0000 7faae0e3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:02:18.941 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.938+0000 7faae0e3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:02:18.990 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:18.987+0000 7faae0e3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:02:19.100 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.098+0000 7faae0e3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:02:19.163 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.160+0000 7faae0e3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:02:19.211 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.208+0000 7faae0e3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:02:19.267 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.264+0000 7faae0e3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:02:19.316 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.313+0000 7faae0e3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:02:19.691 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.688+0000 7faae0e3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:02:19.848 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.845+0000 7faae0e3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:02:19.954 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:19.951+0000 7faae0e3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:02:20.004 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:20.001+0000 7faae0e3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:02:20.111 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:20.109+0000 7faae0e3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:02:20.217 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:20.215+0000 7faae0e3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:02:20.373 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:20.370+0000 7faae0e3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:02:20.421 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:20.418+0000 7faae0e3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:02:20.539 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:20.536+0000 7faae0e3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:02:21.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:21.368+0000 7faae0e3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:02:21.667 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:21.664+0000 7faae0e3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:02:21.723 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:21.720+0000 7faae0e3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:02:21.837 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:21.834+0000 7faae0e3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:02:22.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.127+0000 7faa5f92c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.128+0000 7faa5f92c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.128+0000 7faa5f92c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.128+0000 7faa5f92c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.128+0000 7faa5f92c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.128+0000 7faa5d127640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.129+0000 7faa5d127640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.133 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.129+0000 7faa5d127640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.133 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.129+0000 7faa5d127640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:22.133 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:02:22.129+0000 7faa5d127640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:02:24.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:02:24.548 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:02:54.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:02:54.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:03:00.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:03:00.129+0000 7faa6817d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:03:00.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:03:00.129+0000 7faa6817d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:03:00.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:03:00.129+0000 7faa6817d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:03:00.132 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:03:00.129+0000 7faa6817d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:03:00.133 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:03:00.129+0000 7faa6817d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:03:25.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:03:25.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:03:55.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:03:55.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:04:25.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:04:25.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:04:55.633 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:04:55.644 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:05:25.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:05:25.807 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:05:55.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:05:55.960 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:06:26.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:06:26.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:06:56.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:06:56.327 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:07:17.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-10-16T14:07:17.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.905+0000 7f1433834640 1 Processor -- start 2023-10-16T14:07:17.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.905+0000 7f1433834640 1 -- start start 2023-10-16T14:07:17.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.906+0000 7f1433834640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f142c0caa80 0x7f142c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:17.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.906+0000 7f1433834640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142c0caef0 con 0x7f142c0addb0 2023-10-16T14:07:17.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.906+0000 7f1433834640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142c06a6f0 con 0x7f142c06a2e0 2023-10-16T14:07:17.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.906+0000 7f1433834640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c161a50 con 0x7f142c0caa80 2023-10-16T14:07:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f14315d0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f142c0addb0 0x7f142c0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50942/0 (socket says 172.21.15.60:50942) 2023-10-16T14:07:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f14315d0640 1 -- 172.21.15.60:0/2182134131 learned_addr learned my addr 172.21.15.60:0/2182134131 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f1430dcf640 1 --2- 172.21.15.60:0/2182134131 >> v2:172.21.15.60:3300/0 conn(0x7f142c0caa80 0x7f142c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f1423fff640 1 -- 172.21.15.60:0/2182134131 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077871574 0 0) 0x7f142c0caef0 con 0x7f142c0addb0 2023-10-16T14:07:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f1423fff640 1 -- 172.21.15.60:0/2182134131 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1418003520 con 0x7f142c0addb0 2023-10-16T14:07:17.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f1430dcf640 1 -- 172.21.15.60:0/2182134131 >> v1:172.21.15.144:6789/0 conn(0x7f142c06a2e0 legacy=0x7f142c1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:17.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f1430dcf640 1 -- 172.21.15.60:0/2182134131 >> v1:172.21.15.60:6789/0 conn(0x7f142c0addb0 legacy=0x7f142c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:17.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.907+0000 7f1430dcf640 1 -- 172.21.15.60:0/2182134131 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c0cb140 con 0x7f142c0caa80 2023-10-16T14:07:17.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.909+0000 7f1430dcf640 1 --2- 172.21.15.60:0/2182134131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c069da0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f141c0097a0 tx=0x7f141c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a06c86e1e86e834 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:17.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.909+0000 7f1423fff640 1 -- 172.21.15.60:0/2182134131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c031830 con 0x7f142c0caa80 2023-10-16T14:07:17.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.909+0000 7f1423fff640 1 -- 172.21.15.60:0/2182134131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c031990 con 0x7f142c0caa80 2023-10-16T14:07:17.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.909+0000 7f1423fff640 1 -- 172.21.15.60:0/2182134131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c031d00 con 0x7f142c0caa80 2023-10-16T14:07:17.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.909+0000 7f1433834640 1 -- 172.21.15.60:0/2182134131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 msgr2=0x7f142c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:17.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.909+0000 7f1433834640 1 --2- 172.21.15.60:0/2182134131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c069da0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f141c0097a0 tx=0x7f141c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:07:17.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.910+0000 7f1433834640 1 -- 172.21.15.60:0/2182134131 shutdown_connections 2023-10-16T14:07:17.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.910+0000 7f1433834640 1 --2- 172.21.15.60:0/2182134131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c069da0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:17.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.910+0000 7f1433834640 1 -- 172.21.15.60:0/2182134131 >> 172.21.15.60:0/2182134131 conn(0x7f142c0ce9b0 msgr2=0x7f142c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:17.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.910+0000 7f1433834640 1 -- 172.21.15.60:0/2182134131 shutdown_connections 2023-10-16T14:07:17.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.910+0000 7f1433834640 1 -- 172.21.15.60:0/2182134131 wait complete. 2023-10-16T14:07:17.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.912+0000 7f1433834640 1 Processor -- start 2023-10-16T14:07:17.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.912+0000 7f1433834640 1 -- start start 2023-10-16T14:07:17.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.913+0000 7f1433834640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f142c06a2e0 0x7f142c0b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:17.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.913+0000 7f1433834640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f142c0addb0 0x7f142c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:17.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.913+0000 7f1433834640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:17.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1433834640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0c40b0 con 0x7f142c0caa80 2023-10-16T14:07:17.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1433834640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c15e410 con 0x7f142c0addb0 2023-10-16T14:07:17.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1433834640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0b4960 con 0x7f142c06a2e0 2023-10-16T14:07:17.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:17.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55600/0 (socket says 172.21.15.60:55600) 2023-10-16T14:07:17.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 -- 172.21.15.60:0/3898488784 learned_addr learned my addr 172.21.15.60:0/3898488784 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:17.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f14315d0640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f142c06a2e0 0x7f142c0b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:17.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1430dcf640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f142c0addb0 0x7f142c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:17.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 -- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f142c06a2e0 msgr2=0x7f142c0b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:17.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f142c06a2e0 0x7f142c0b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:17.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 -- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f142c0addb0 msgr2=0x7f142c15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:17.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f142c0addb0 0x7f142c15ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:17.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1431dd1640 1 -- 172.21.15.60:0/3898488784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c009400 con 0x7f142c0caa80 2023-10-16T14:07:17.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.914+0000 7f1430dcf640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f142c0addb0 0x7f142c15ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:17.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.915+0000 7f1431dd1640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c0b0440 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f142400c7b0 tx=0x7f142400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:17.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.915+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424004b60 con 0x7f142c0caa80 2023-10-16T14:07:17.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.915+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1424004cc0 con 0x7f142c0caa80 2023-10-16T14:07:17.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.915+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c0c3750 con 0x7f142c0caa80 2023-10-16T14:07:17.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.915+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400aa90 con 0x7f142c0caa80 2023-10-16T14:07:17.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.915+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c0c3cd0 con 0x7f142c0caa80 2023-10-16T14:07:17.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.916+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c0c41f0 con 0x7f142c0caa80 2023-10-16T14:07:17.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.917+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 110) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f142400abf0 con 0x7f142c0caa80 2023-10-16T14:07:17.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.919+0000 7f1421ffb640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f141006afe0 0x7f141006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:17.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.919+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1424089c30 con 0x7f142c0caa80 2023-10-16T14:07:17.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.920+0000 7f14315d0640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f141006afe0 0x7f141006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:17.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.921+0000 7f14315d0640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f141006afe0 0x7f141006d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1428005880 tx=0x7f1428009040 comp rx=0 tx=0).ready entity=mgr.7286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:17.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:17.924+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f142c0c41f0 con 0x7f142c0caa80 2023-10-16T14:07:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.148+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 --> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f142c06b310 con 0x7f141006afe0 2023-10-16T14:07:18.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.321+0000 7f1421ffb640 1 -- 172.21.15.60:0/3898488784 <== mgr.7286 v2:172.21.15.144:6832/345630502 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1428001d00 con 0x7f141006afe0 2023-10-16T14:07:18.325 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T14:07:18.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f141006afe0 msgr2=0x7f141006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f141006afe0 0x7f141006d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1428005880 tx=0x7f1428009040 comp rx=0 tx=0).stop 2023-10-16T14:07:18.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 msgr2=0x7f142c0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c0b0440 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f142400c7b0 tx=0x7f142400cc70 comp rx=0 tx=0).stop 2023-10-16T14:07:18.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 shutdown_connections 2023-10-16T14:07:18.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f141006afe0 0x7f141006d4a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f142c0caa80 0x7f142c0b0440 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f142c0addb0 0x7f142c15ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 --2- 172.21.15.60:0/3898488784 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f142c06a2e0 0x7f142c0b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 >> 172.21.15.60:0/3898488784 conn(0x7f142c0ce9b0 msgr2=0x7f142c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:18.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.323+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 shutdown_connections 2023-10-16T14:07:18.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.324+0000 7f1433834640 1 -- 172.21.15.60:0/3898488784 wait complete. 2023-10-16T14:07:18.349 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T14:07:18.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:18.351 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T14:07:18.412 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_52_00_UTC 2023-10-16T14:07:18.412 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_53_00_UTC 2023-10-16T14:07:18.413 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_54_00_UTC 2023-10-16T14:07:18.413 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_55_00_UTC 2023-10-16T14:07:18.413 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_56_00_UTC 2023-10-16T14:07:18.413 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_57_00_UTC 2023-10-16T14:07:18.414 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_58_00_UTC 2023-10-16T14:07:18.414 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_59_00_UTC 2023-10-16T14:07:18.415 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_00_00_UTC 2023-10-16T14:07:18.415 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_01_00_UTC 2023-10-16T14:07:18.415 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_02_00_UTC 2023-10-16T14:07:18.416 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_03_00_UTC 2023-10-16T14:07:18.416 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_04_00_UTC 2023-10-16T14:07:18.417 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_05_00_UTC 2023-10-16T14:07:18.417 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_06_00_UTC 2023-10-16T14:07:18.417 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_07_00_UTC 2023-10-16T14:07:18.419 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-13_52_00_UTC', 'scheduled-2023-10-16-13_53_00_UTC', 'scheduled-2023-10-16-13_54_00_UTC', 'scheduled-2023-10-16-13_55_00_UTC', 'scheduled-2023-10-16-13_56_00_UTC', 'scheduled-2023-10-16-13_57_00_UTC', 'scheduled-2023-10-16-13_58_00_UTC', 'scheduled-2023-10-16-13_59_00_UTC', 'scheduled-2023-10-16-14_00_00_UTC', 'scheduled-2023-10-16-14_01_00_UTC', 'scheduled-2023-10-16-14_02_00_UTC', 'scheduled-2023-10-16-14_03_00_UTC', 'scheduled-2023-10-16-14_04_00_UTC', 'scheduled-2023-10-16-14_05_00_UTC', 'scheduled-2023-10-16-14_06_00_UTC', 'scheduled-2023-10-16-14_07_00_UTC'] 2023-10-16T14:07:18.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T14:07:18.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.648+0000 7fc240b35640 1 Processor -- start 2023-10-16T14:07:18.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.649+0000 7fc240b35640 1 -- start start 2023-10-16T14:07:18.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.650+0000 7fc240b35640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc23c06c040 0x7fc23c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.650+0000 7fc240b35640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc23c0ad970 0x7fc23c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.650+0000 7fc240b35640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.650+0000 7fc240b35640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c06c960 con 0x7fc23c06c040 2023-10-16T14:07:18.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.650+0000 7fc240b35640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c15b9d0 con 0x7fc23c0ad970 2023-10-16T14:07:18.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.650+0000 7fc240b35640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c15bcb0 con 0x7fc23c15ebc0 2023-10-16T14:07:18.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc23c06c040 0x7fc23c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc23c06c040 0x7fc23c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55604/0 (socket says 172.21.15.60:55604) 2023-10-16T14:07:18.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 -- 172.21.15.60:0/3661914676 learned_addr learned my addr 172.21.15.60:0/3661914676 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:18.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23bfff640 1 --2- 172.21.15.60:0/3661914676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23affd640 1 --2- 172.21.15.60:0/3661914676 >> v2:172.21.15.144:3300/0 conn(0x7fc23c0ad970 0x7fc23c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 -- 172.21.15.60:0/3661914676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 msgr2=0x7fc23c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:18.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 --2- 172.21.15.60:0/3661914676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 -- 172.21.15.60:0/3661914676 >> v2:172.21.15.144:3300/0 conn(0x7fc23c0ad970 msgr2=0x7fc23c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:18.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 --2- 172.21.15.60:0/3661914676 >> v2:172.21.15.144:3300/0 conn(0x7fc23c0ad970 0x7fc23c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23b7fe640 1 -- 172.21.15.60:0/3661914676 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c15bf90 con 0x7fc23c06c040 2023-10-16T14:07:18.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.651+0000 7fc23affd640 1 --2- 172.21.15.60:0/3661914676 >> v2:172.21.15.144:3300/0 conn(0x7fc23c0ad970 0x7fc23c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:18.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.652+0000 7fc23b7fe640 1 --2- 172.21.15.60:0/3661914676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c06c420 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0cb060 tx=0x7fc22c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c94be0d805d421 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.653+0000 7fc23a7fc640 1 -- 172.21.15.60:0/3661914676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c02ca70 con 0x7fc23c06c040 2023-10-16T14:07:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.653+0000 7fc23a7fc640 1 -- 172.21.15.60:0/3661914676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c02cbd0 con 0x7fc23c06c040 2023-10-16T14:07:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.653+0000 7fc23a7fc640 1 -- 172.21.15.60:0/3661914676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0343f0 con 0x7fc23c06c040 2023-10-16T14:07:18.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.653+0000 7fc240b35640 1 -- 172.21.15.60:0/3661914676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 msgr2=0x7fc23c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.653+0000 7fc240b35640 1 --2- 172.21.15.60:0/3661914676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c06c420 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0cb060 tx=0x7fc22c02bed0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 -- 172.21.15.60:0/3661914676 shutdown_connections 2023-10-16T14:07:18.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 --2- 172.21.15.60:0/3661914676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 --2- 172.21.15.60:0/3661914676 >> v2:172.21.15.144:3300/0 conn(0x7fc23c0ad970 0x7fc23c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 --2- 172.21.15.60:0/3661914676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c06c420 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 -- 172.21.15.60:0/3661914676 >> 172.21.15.60:0/3661914676 conn(0x7fc23c0ce6a0 msgr2=0x7fc23c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 -- 172.21.15.60:0/3661914676 shutdown_connections 2023-10-16T14:07:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.654+0000 7fc240b35640 1 -- 172.21.15.60:0/3661914676 wait complete. 2023-10-16T14:07:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.655+0000 7fc240b35640 1 Processor -- start 2023-10-16T14:07:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.656+0000 7fc240b35640 1 -- start start 2023-10-16T14:07:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.656+0000 7fc240b35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c0aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc240b35640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc23c0ad970 0x7fc23c0b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc240b35640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c1f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc240b35640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0c5770 con 0x7fc23c06c040 2023-10-16T14:07:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc240b35640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0af150 con 0x7fc23c0ad970 2023-10-16T14:07:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc240b35640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0c5a10 con 0x7fc23c15ebc0 2023-10-16T14:07:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c0aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c0aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55616/0 (socket says 172.21.15.60:55616) 2023-10-16T14:07:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 -- 172.21.15.60:0/2214541404 learned_addr learned my addr 172.21.15.60:0/2214541404 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23affd640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc23c0ad970 0x7fc23c0b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23bfff640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c1f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 -- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 msgr2=0x7fc23c1f1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c1f1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 -- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc23c0ad970 msgr2=0x7fc23c0b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc23c0ad970 0x7fc23c0b38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23b7fe640 1 -- 172.21.15.60:0/2214541404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c009b30 con 0x7fc23c06c040 2023-10-16T14:07:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23bfff640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c1f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.657+0000 7fc23affd640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc23c0ad970 0x7fc23c0b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.658+0000 7fc23b7fe640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c0aec10 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009e20 tx=0x7fc22c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.658+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c009510 con 0x7fc23c06c040 2023-10-16T14:07:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.658+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c03e450 con 0x7fc23c06c040 2023-10-16T14:07:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.658+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c03e800 con 0x7fc23c06c040 2023-10-16T14:07:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.658+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c0c5cf0 con 0x7fc23c06c040 2023-10-16T14:07:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.658+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c0c62b0 con 0x7fc23c06c040 2023-10-16T14:07:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.660+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c06b330 con 0x7fc23c06c040 2023-10-16T14:07:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.660+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 110) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fc22c03e960 con 0x7fc23c06c040 2023-10-16T14:07:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.661+0000 7fc233fff640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc21c06afe0 0x7fc21c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.662+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc22c0ac860 con 0x7fc23c06c040 2023-10-16T14:07:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.662+0000 7fc23affd640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc21c06afe0 0x7fc21c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.663+0000 7fc23affd640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc21c06afe0 0x7fc21c06d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc224006fd0 tx=0x7fc224008040 comp rx=0 tx=0).ready entity=mgr.7286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.666+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc22c077540 con 0x7fc23c06c040 2023-10-16T14:07:18.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.883+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 --> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc23c0cc550 con 0x7fc21c06afe0 2023-10-16T14:07:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.902+0000 7fc233fff640 1 -- 172.21.15.60:0/2214541404 <== mgr.7286 v2:172.21.15.144:6832/345630502 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7fc22400c040 con 0x7fc21c06afe0 2023-10-16T14:07:18.905 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:18.905 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:51:43", "first": "2023-10-16T13:52:00", "last": "2023-10-16T14:07:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T14:07:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc21c06afe0 msgr2=0x7fc21c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc21c06afe0 0x7fc21c06d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc224006fd0 tx=0x7fc224008040 comp rx=0 tx=0).stop 2023-10-16T14:07:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 msgr2=0x7fc23c0aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c0aec10 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009e20 tx=0x7fc22c004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 shutdown_connections 2023-10-16T14:07:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc21c06afe0 0x7fc21c06d4a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc23c15ebc0 0x7fc23c1f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc23c0ad970 0x7fc23c0b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 --2- 172.21.15.60:0/2214541404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c06c040 0x7fc23c0aec10 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.903+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 >> 172.21.15.60:0/2214541404 conn(0x7fc23c0ce6a0 msgr2=0x7fc23c0705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.904+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 shutdown_connections 2023-10-16T14:07:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:18.904+0000 7fc240b35640 1 -- 172.21.15.60:0/2214541404 wait complete. 2023-10-16T14:07:18.926 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:51:43", "first": "2023-10-16T13:52:00", "last": "2023-10-16T14:07:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T14:07:18.927 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T13:51:43', 'first': '2023-10-16T13:52:00', 'last': '2023-10-16T14:07:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-16T14:07:18.927 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-16T14:07:18.927 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-16T14:07:18.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-10-16T14:07:19.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.153+0000 7f8ccaa61640 1 Processor -- start 2023-10-16T14:07:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.153+0000 7f8ccaa61640 1 -- start start 2023-10-16T14:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8ccaa61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 0x7f8cc41681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8ccaa61640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc4161900 con 0x7f8cc4160ad0 2023-10-16T14:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8ccaa61640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40ce8a0 con 0x7f8cc40ae260 2023-10-16T14:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8ccaa61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc406b790 con 0x7f8cc4165df0 2023-10-16T14:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8cc3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8cc4160ad0 0x7f8cc40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49356/0 (socket says 172.21.15.60:49356) 2023-10-16T14:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8cc3fff640 1 -- 172.21.15.60:0/1773099717 learned_addr learned my addr 172.21.15.60:0/1773099717 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:19.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916478223 0 0) 0x7f8cc4161900 con 0x7f8cc4160ad0 2023-10-16T14:07:19.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac003520 con 0x7f8cc4160ad0 2023-10-16T14:07:19.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.155+0000 7f8cc8ffe640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 0x7f8cc41681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89817721 0 0) 0x7f8cac003520 con 0x7f8cc4160ad0 2023-10-16T14:07:19.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 msgr2=0x7f8cc41681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:19.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 0x7f8cc41681e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 >> v1:172.21.15.144:6789/0 conn(0x7f8cc40ae260 legacy=0x7f8cc4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40ceb80 con 0x7f8cc4160ad0 2023-10-16T14:07:19.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f8cb4001670 con 0x7f8cc4160ad0 2023-10-16T14:07:19.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 >> v1:172.21.15.60:6789/0 conn(0x7f8cc4160ad0 legacy=0x7f8cc40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cac0039a0 0x7f8cac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 0x7f8cac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cac004d00 0x7f8cac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac00bb60 con 0x7f8cac0039a0 2023-10-16T14:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac00bcd0 con 0x7f8cac004d00 2023-10-16T14:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac00be90 con 0x7f8cc4165df0 2023-10-16T14:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.156+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cac0039a0 0x7f8cac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.157+0000 7f8cc3fff640 1 -- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 msgr2=0x7f8cac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.157+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cc4165df0 0x7f8cac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.157+0000 7f8cc3fff640 1 -- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cac004d00 msgr2=0x7f8cac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.157+0000 7f8cc8ffe640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cac004d00 0x7f8cac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.157+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cac004d00 0x7f8cac005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.157+0000 7f8cc3fff640 1 -- 172.21.15.60:0/1773099717 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac003520 con 0x7f8cac0039a0 2023-10-16T14:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.158+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cac0039a0 0x7f8cac003dc0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c0e0 tx=0x7f8cac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5869152f5438cf84 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.158+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4005ad0 con 0x7f8cac0039a0 2023-10-16T14:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.158+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb4005c30 con 0x7f8cac0039a0 2023-10-16T14:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.158+0000 7f8cc2ffd640 1 -- 172.21.15.60:0/1773099717 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb402c810 con 0x7f8cac0039a0 2023-10-16T14:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1773099717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cac0039a0 msgr2=0x7f8cac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1773099717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cac0039a0 0x7f8cac003dc0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c0e0 tx=0x7f8cac0057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8cc8ffe640 1 -- 172.21.15.60:0/1773099717 reap_dead start 2023-10-16T14:07:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1773099717 shutdown_connections 2023-10-16T14:07:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1773099717 >> 172.21.15.60:0/1773099717 conn(0x7f8cc406bc10 msgr2=0x7f8cc406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1773099717 shutdown_connections 2023-10-16T14:07:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.159+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1773099717 wait complete. 2023-10-16T14:07:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.161+0000 7f8ccaa61640 1 Processor -- start 2023-10-16T14:07:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.161+0000 7f8ccaa61640 1 -- start start 2023-10-16T14:07:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8ccaa61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cac0039a0 0x7f8cc40b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8ccaa61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 0x7f8cc41454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8ccaa61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cc40ccd60 0x7f8cc40cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8ccaa61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40cd700 con 0x7f8cc415d870 2023-10-16T14:07:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8ccaa61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b4470 con 0x7f8cc40ccd60 2023-10-16T14:07:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8ccaa61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b3860 con 0x7f8cac0039a0 2023-10-16T14:07:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8cc37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 0x7f8cc41454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8cc37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 0x7f8cc41454b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48512/0 (socket says 172.21.15.60:48512) 2023-10-16T14:07:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8cc37fe640 1 -- 172.21.15.60:0/1462202864 learned_addr learned my addr 172.21.15.60:0/1462202864 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cac0039a0 0x7f8cc40b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.162+0000 7f8cc8ffe640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cc40ccd60 0x7f8cc40cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc37fe640 1 -- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cac0039a0 msgr2=0x7f8cc40b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc37fe640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cac0039a0 0x7f8cc40b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc37fe640 1 -- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cc40ccd60 msgr2=0x7f8cc40cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc37fe640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cc40ccd60 0x7f8cc40cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc37fe640 1 -- 172.21.15.60:0/1462202864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40b3b40 con 0x7f8cc415d870 2023-10-16T14:07:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc8ffe640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cc40ccd60 0x7f8cc40cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cac0039a0 0x7f8cc40b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc37fe640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 0x7f8cc41454b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c800 tx=0x7f8cb800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:19.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8003b70 con 0x7f8cc415d870 2023-10-16T14:07:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8003cf0 con 0x7f8cc415d870 2023-10-16T14:07:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb800aa90 con 0x7f8cc415d870 2023-10-16T14:07:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40b3e20 con 0x7f8cc415d870 2023-10-16T14:07:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.163+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc415dcd0 con 0x7f8cc415d870 2023-10-16T14:07:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.165+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc40b4600 con 0x7f8cc415d870 2023-10-16T14:07:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.165+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 110) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f8cb800abf0 con 0x7f8cc415d870 2023-10-16T14:07:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.167+0000 7f8cc0ff9640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f8cb006b010 0x7f8cb006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.167+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8cb8089950 con 0x7f8cc415d870 2023-10-16T14:07:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.168+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f8cb006b010 0x7f8cb006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.169+0000 7f8cc3fff640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f8cb006b010 0x7f8cb006d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4034000 tx=0x7f8cb4009030 comp rx=0 tx=0).ready entity=mgr.7286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:19.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.171+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8cb8054630 con 0x7f8cc415d870 2023-10-16T14:07:19.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.391+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 --> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc40ae820 con 0x7f8cb006b010 2023-10-16T14:07:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.588+0000 7f8cc0ff9640 1 -- 172.21.15.60:0/1462202864 <== mgr.7286 v2:172.21.15.144:6832/345630502 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f8cb4005a00 con 0x7f8cb006b010 2023-10-16T14:07:19.592 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T14:07:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f8cb006b010 msgr2=0x7f8cb006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f8cb006b010 0x7f8cb006d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4034000 tx=0x7f8cb4009030 comp rx=0 tx=0).stop 2023-10-16T14:07:19.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 msgr2=0x7f8cc41454b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 0x7f8cc41454b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c800 tx=0x7f8cb800ccc0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 shutdown_connections 2023-10-16T14:07:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f8cb006b010 0x7f8cb006d4d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cc40ccd60 0x7f8cc40cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cc415d870 0x7f8cc41454b0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 --2- 172.21.15.60:0/1462202864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8cac0039a0 0x7f8cc40b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 >> 172.21.15.60:0/1462202864 conn(0x7f8cc406bc10 msgr2=0x7f8cc415c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 shutdown_connections 2023-10-16T14:07:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:19.591+0000 7f8ccaa61640 1 -- 172.21.15.60:0/1462202864 wait complete. 2023-10-16T14:07:19.617 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T14:07:19.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:19.618 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T14:07:19.673 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_52_00_UTC 2023-10-16T14:07:19.673 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_53_00_UTC 2023-10-16T14:07:19.673 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_54_00_UTC 2023-10-16T14:07:19.674 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_55_00_UTC 2023-10-16T14:07:19.674 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_56_00_UTC 2023-10-16T14:07:19.674 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_57_00_UTC 2023-10-16T14:07:19.674 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_58_00_UTC 2023-10-16T14:07:19.675 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-13_59_00_UTC 2023-10-16T14:07:19.675 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_00_00_UTC 2023-10-16T14:07:19.675 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_01_00_UTC 2023-10-16T14:07:19.675 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_02_00_UTC 2023-10-16T14:07:19.675 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_03_00_UTC 2023-10-16T14:07:19.676 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_04_00_UTC 2023-10-16T14:07:19.676 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_05_00_UTC 2023-10-16T14:07:19.676 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_06_00_UTC 2023-10-16T14:07:19.676 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_07_00_UTC 2023-10-16T14:07:19.677 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_52_00_UTC 2023-10-16T14:07:19.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:19.678 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_52_00_UTC) 2023-10-16T14:07:19.745 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_53_00_UTC 2023-10-16T14:07:19.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:19.746 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_53_00_UTC) 2023-10-16T14:07:19.822 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_54_00_UTC 2023-10-16T14:07:19.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:19.823 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_54_00_UTC) 2023-10-16T14:07:19.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_55_00_UTC 2023-10-16T14:07:19.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:19.897 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_55_00_UTC) 2023-10-16T14:07:19.974 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_56_00_UTC 2023-10-16T14:07:19.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:19.975 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_56_00_UTC) 2023-10-16T14:07:20.053 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_57_00_UTC 2023-10-16T14:07:20.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.054 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_57_00_UTC) 2023-10-16T14:07:20.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_58_00_UTC 2023-10-16T14:07:20.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.128 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_58_00_UTC) 2023-10-16T14:07:20.199 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_59_00_UTC 2023-10-16T14:07:20.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.200 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_59_00_UTC) 2023-10-16T14:07:20.271 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_00_00_UTC 2023-10-16T14:07:20.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.272 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_00_00_UTC) 2023-10-16T14:07:20.345 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_01_00_UTC 2023-10-16T14:07:20.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.346 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_01_00_UTC) 2023-10-16T14:07:20.420 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_02_00_UTC 2023-10-16T14:07:20.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.421 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_02_00_UTC) 2023-10-16T14:07:20.499 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_03_00_UTC 2023-10-16T14:07:20.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.500 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_03_00_UTC) 2023-10-16T14:07:20.571 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_04_00_UTC 2023-10-16T14:07:20.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.572 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_04_00_UTC) 2023-10-16T14:07:20.640 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_05_00_UTC 2023-10-16T14:07:20.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.641 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_05_00_UTC) 2023-10-16T14:07:20.709 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_06_00_UTC 2023-10-16T14:07:20.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.710 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_06_00_UTC) 2023-10-16T14:07:20.786 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_07_00_UTC 2023-10-16T14:07:20.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.787 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-14_07_00_UTC) 2023-10-16T14:07:20.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:20.863 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-16T14:07:20.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:07:21.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.144+0000 7fc6dd5c9640 1 Processor -- start 2023-10-16T14:07:21.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.144+0000 7fc6dd5c9640 1 -- start start 2023-10-16T14:07:21.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6dd5c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc6d81573e0 0x7fc6d80ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6dd5c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d80ca3a0 con 0x7fc6d8070950 2023-10-16T14:07:21.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6dd5c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d80cad70 con 0x7fc6d8159e70 2023-10-16T14:07:21.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6dd5c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d80c9d70 con 0x7fc6d81573e0 2023-10-16T14:07:21.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6dcdc8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc6d8070950 0x7fc6d806bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49370/0 (socket says 172.21.15.60:49370) 2023-10-16T14:07:21.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6dcdc8640 1 -- 172.21.15.60:0/4258109031 learned_addr learned my addr 172.21.15.60:0/4258109031 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.146+0000 7fc6d7fff640 1 --2- 172.21.15.60:0/4258109031 >> v2:172.21.15.144:3300/0 conn(0x7fc6d81573e0 0x7fc6d80ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972961179 0 0) 0x7fc6d80ca3a0 con 0x7fc6d8070950 2023-10-16T14:07:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c4003520 con 0x7fc6d8070950 2023-10-16T14:07:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3714955014 0 0) 0x7fc6c4003520 con 0x7fc6d8070950 2023-10-16T14:07:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 >> v2:172.21.15.144:3300/0 conn(0x7fc6d81573e0 msgr2=0x7fc6d80ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 --2- 172.21.15.60:0/4258109031 >> v2:172.21.15.144:3300/0 conn(0x7fc6d81573e0 0x7fc6d80ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 >> v1:172.21.15.144:6789/0 conn(0x7fc6d8159e70 legacy=0x7fc6d8070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d81577c0 con 0x7fc6d8070950 2023-10-16T14:07:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc6d0001670 con 0x7fc6d8070950 2023-10-16T14:07:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 >> v1:172.21.15.60:6789/0 conn(0x7fc6d8070950 legacy=0x7fc6d806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6c40039c0 0x7fc6c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c40047c0 0x7fc6c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c4005180 0x7fc6c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80c9d70 con 0x7fc6c40047c0 2023-10-16T14:07:21.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c400c080 con 0x7fc6c40039c0 2023-10-16T14:07:21.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.147+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c400c240 con 0x7fc6c4005180 2023-10-16T14:07:21.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6cf7fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c40047c0 0x7fc6c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6d7fff640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6c40039c0 0x7fc6c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6dcdc8640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c4005180 0x7fc6c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6cf7fe640 1 -- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c4005180 msgr2=0x7fc6c400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:21.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6cf7fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c4005180 0x7fc6c400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6cf7fe640 1 -- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6c40039c0 msgr2=0x7fc6c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:21.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6cf7fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6c40039c0 0x7fc6c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6cf7fe640 1 -- 172.21.15.60:0/4258109031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4003520 con 0x7fc6c40047c0 2023-10-16T14:07:21.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.148+0000 7fc6dcdc8640 1 -- 172.21.15.60:0/4258109031 reap_dead start 2023-10-16T14:07:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.149+0000 7fc6cf7fe640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c40047c0 0x7fc6c4004c40 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0001750 tx=0x7fc6c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32faf344a70716e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c00052d0 con 0x7fc6c40047c0 2023-10-16T14:07:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c002e050 con 0x7fc6c40047c0 2023-10-16T14:07:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6d77fe640 1 -- 172.21.15.60:0/4258109031 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0003830 con 0x7fc6c40047c0 2023-10-16T14:07:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4258109031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c40047c0 msgr2=0x7fc6c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c40047c0 0x7fc6c4004c40 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0001750 tx=0x7fc6c40057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4258109031 shutdown_connections 2023-10-16T14:07:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4258109031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c40047c0 0x7fc6c4004c40 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4258109031 >> 172.21.15.60:0/4258109031 conn(0x7fc6d80cc230 msgr2=0x7fc6d80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4258109031 shutdown_connections 2023-10-16T14:07:21.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.150+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4258109031 wait complete. 2023-10-16T14:07:21.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.152+0000 7fc6dd5c9640 1 Processor -- start 2023-10-16T14:07:21.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.152+0000 7fc6dd5c9640 1 -- start start 2023-10-16T14:07:21.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.153+0000 7fc6dd5c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c40047c0 0x7fc6d8155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.153+0000 7fc6dd5c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d8153830 0x7fc6d80c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.153+0000 7fc6dd5c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 0x7fc6d8152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.153+0000 7fc6dd5c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80c72a0 con 0x7fc6d815c180 2023-10-16T14:07:21.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.153+0000 7fc6dd5c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d8155db0 con 0x7fc6d8153830 2023-10-16T14:07:21.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.153+0000 7fc6dd5c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80c7540 con 0x7fc6c40047c0 2023-10-16T14:07:21.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 0x7fc6d8152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 0x7fc6d8152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48524/0 (socket says 172.21.15.60:48524) 2023-10-16T14:07:21.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 -- 172.21.15.60:0/4292126580 learned_addr learned my addr 172.21.15.60:0/4292126580 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:21.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6d7fff640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c40047c0 0x7fc6d8155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6cf7fe640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d8153830 0x7fc6d80c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 -- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c40047c0 msgr2=0x7fc6d8155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c40047c0 0x7fc6d8155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 -- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d8153830 msgr2=0x7fc6d80c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d8153830 0x7fc6d80c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.154+0000 7fc6dcdc8640 1 -- 172.21.15.60:0/4292126580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80c7820 con 0x7fc6d815c180 2023-10-16T14:07:21.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.155+0000 7fc6dcdc8640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 0x7fc6d8152ca0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80ca310 tx=0x7fc6d000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:21.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.155+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0010510 con 0x7fc6d815c180 2023-10-16T14:07:21.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.155+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d0010670 con 0x7fc6d815c180 2023-10-16T14:07:21.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.155+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0010a20 con 0x7fc6d815c180 2023-10-16T14:07:21.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.155+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d80c7b00 con 0x7fc6d815c180 2023-10-16T14:07:21.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.155+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d80c8840 con 0x7fc6d815c180 2023-10-16T14:07:21.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.157+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d80c8020 con 0x7fc6d815c180 2023-10-16T14:07:21.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.158+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 110) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fc6d0010b80 con 0x7fc6d815c180 2023-10-16T14:07:21.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.159+0000 7fc6d57fa640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc6b806afe0 0x7fc6b806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.159+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc6d00892b0 con 0x7fc6d815c180 2023-10-16T14:07:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.159+0000 7fc6d7fff640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc6b806afe0 0x7fc6b806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.161+0000 7fc6d7fff640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc6b806afe0 0x7fc6b806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80d07d0 tx=0x7fc6c800b040 comp rx=0 tx=0).ready entity=mgr.7286 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.164+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6d80c8020 con 0x7fc6d815c180 2023-10-16T14:07:21.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.410+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc6d80c8020 con 0x7fc6d815c180 2023-10-16T14:07:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.841+0000 7fc6d57fa640 1 -- 172.21.15.60:0/4292126580 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v111) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc6d80c8020 con 0x7fc6d815c180 2023-10-16T14:07:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.846+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc6b806afe0 msgr2=0x7fc6b806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.846+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc6b806afe0 0x7fc6b806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80d07d0 tx=0x7fc6c800b040 comp rx=0 tx=0).stop 2023-10-16T14:07:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.846+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 msgr2=0x7fc6d8152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.846+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 0x7fc6d8152ca0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80ca310 tx=0x7fc6d000e910 comp rx=0 tx=0).stop 2023-10-16T14:07:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.847+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 shutdown_connections 2023-10-16T14:07:21.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.847+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fc6b806afe0 0x7fc6b806d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.847+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d815c180 0x7fc6d8152ca0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.847+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d8153830 0x7fc6d80c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.847+0000 7fc6dd5c9640 1 --2- 172.21.15.60:0/4292126580 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6c40047c0 0x7fc6d8155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:21.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.847+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 >> 172.21.15.60:0/4292126580 conn(0x7fc6d80cc230 msgr2=0x7fc6d8070240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:21.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.851+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 shutdown_connections 2023-10-16T14:07:21.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:21.852+0000 7fc6dd5c9640 1 -- 172.21.15.60:0/4292126580 wait complete. 2023-10-16T14:07:21.891 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:07:21.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:07:21.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:07:22.088 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:07:22.089 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:07:22.089 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:22.143 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:07:22.143 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:22.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:22.434 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:22.430+0000 7f20c783a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:07:22.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.461+0000 7feddc94e640 1 Processor -- start 2023-10-16T14:07:22.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.462+0000 7feddc94e640 1 -- start start 2023-10-16T14:07:22.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.464+0000 7feddc94e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd80adb50 0x7fedd806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.464+0000 7feddc94e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fedd80b0b20 0x7fedd8164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.464+0000 7feddc94e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd8071450 con 0x7fedd8070fd0 2023-10-16T14:07:22.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.464+0000 7feddc94e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80cc600 con 0x7fedd80b0b20 2023-10-16T14:07:22.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.464+0000 7feddc94e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80cc8e0 con 0x7fedd80adb50 2023-10-16T14:07:22.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.465+0000 7fedd6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fedd80b0b20 0x7fedd8164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.465+0000 7fedd5d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fedd8070fd0 0x7fedd80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49386/0 (socket says 172.21.15.60:49386) 2023-10-16T14:07:22.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.465+0000 7fedd5d9b640 1 -- 172.21.15.60:0/3922364662 learned_addr learned my addr 172.21.15.60:0/3922364662 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:22.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.465+0000 7fedd659c640 1 --2- 172.21.15.60:0/3922364662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd80adb50 0x7fedd806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.466+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3922364662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd80adb50 msgr2=0x7fedd806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:22.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.466+0000 7fedd6d9d640 1 --2- 172.21.15.60:0/3922364662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd80adb50 0x7fedd806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.466+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3922364662 >> v1:172.21.15.60:6789/0 conn(0x7fedd8070fd0 legacy=0x7fedd80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.466+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3922364662 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80ccbc0 con 0x7fedd80b0b20 2023-10-16T14:07:22.471 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:22.466+0000 7fecc8c12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:07:22.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.467+0000 7fedd6d9d640 1 --2- 172.21.15.60:0/3922364662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd8164f60 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009d90 tx=0x7fedd002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2904d320536f149f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:22.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.468+0000 7fedd559a640 1 -- 172.21.15.60:0/3922364662 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd002cc50 con 0x7fedd80b0b20 2023-10-16T14:07:22.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.468+0000 7fedd559a640 1 -- 172.21.15.60:0/3922364662 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd002cdb0 con 0x7fedd80b0b20 2023-10-16T14:07:22.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.468+0000 7fedd559a640 1 -- 172.21.15.60:0/3922364662 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0034660 con 0x7fedd80b0b20 2023-10-16T14:07:22.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.469+0000 7feddc94e640 1 -- 172.21.15.60:0/3922364662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 msgr2=0x7fedd8164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.469+0000 7feddc94e640 1 --2- 172.21.15.60:0/3922364662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd8164f60 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009d90 tx=0x7fedd002bea0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.469+0000 7feddc94e640 1 -- 172.21.15.60:0/3922364662 shutdown_connections 2023-10-16T14:07:22.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.469+0000 7feddc94e640 1 --2- 172.21.15.60:0/3922364662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd8164f60 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.469+0000 7feddc94e640 1 --2- 172.21.15.60:0/3922364662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd80adb50 0x7fedd806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.469+0000 7feddc94e640 1 -- 172.21.15.60:0/3922364662 >> 172.21.15.60:0/3922364662 conn(0x7fedd80cb410 msgr2=0x7fedd80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:22.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.470+0000 7feddc94e640 1 -- 172.21.15.60:0/3922364662 shutdown_connections 2023-10-16T14:07:22.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.470+0000 7feddc94e640 1 -- 172.21.15.60:0/3922364662 wait complete. 2023-10-16T14:07:22.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.472+0000 7feddc94e640 1 Processor -- start 2023-10-16T14:07:22.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.473+0000 7feddc94e640 1 -- start start 2023-10-16T14:07:22.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.474+0000 7feddc94e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd8070fd0 0x7fedd81f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.474+0000 7feddc94e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd80adb50 0x7fedd80bf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.474+0000 7feddc94e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd80bf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.474+0000 7feddc94e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80b1590 con 0x7fedd80b0b20 2023-10-16T14:07:22.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.474+0000 7feddc94e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80be3a0 con 0x7fedd80adb50 2023-10-16T14:07:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.474+0000 7feddc94e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80be680 con 0x7fedd8070fd0 2023-10-16T14:07:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd80bf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd80bf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48576/0 (socket says 172.21.15.60:48576) 2023-10-16T14:07:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3377092170 learned_addr learned my addr 172.21.15.60:0/3377092170 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd5d9b640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd80adb50 0x7fedd80bf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd659c640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd8070fd0 0x7fedd81f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd8070fd0 msgr2=0x7fedd81f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd8070fd0 0x7fedd81f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd80adb50 msgr2=0x7fedd80bf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd80adb50 0x7fedd80bf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.475+0000 7fedd6d9d640 1 -- 172.21.15.60:0/3377092170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0009a70 con 0x7fedd80b0b20 2023-10-16T14:07:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.476+0000 7fedd6d9d640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd80bf970 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009eb0 tx=0x7fedd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.476+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd003e610 con 0x7fedd80b0b20 2023-10-16T14:07:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.477+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd003e770 con 0x7fedd80b0b20 2023-10-16T14:07:22.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.477+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd003eb20 con 0x7fedd80b0b20 2023-10-16T14:07:22.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.477+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd80b17e0 con 0x7fedd80b0b20 2023-10-16T14:07:22.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.477+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd81f6960 con 0x7fedd80b0b20 2023-10-16T14:07:22.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.478+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd80b1d00 con 0x7fedd80b0b20 2023-10-16T14:07:22.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.482+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fedd003b050 con 0x7fedd80b0b20 2023-10-16T14:07:22.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.484+0000 7fedc6ffd640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 0x7fedb806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.485+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6485+0+0 (secure 0 0 0) 0x7fedd00ac4b0 con 0x7fedd80b0b20 2023-10-16T14:07:22.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.485+0000 7fedd659c640 1 -- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 msgr2=0x7fedb806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:22.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.485+0000 7fedd659c640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 0x7fedb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:22.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.489+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fedd00772b0 con 0x7fedd80b0b20 2023-10-16T14:07:22.506 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:22.502+0000 7f20c783a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:07:22.543 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:22.539+0000 7fecc8c12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:07:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.686+0000 7fedd659c640 1 -- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 msgr2=0x7fedb806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:22.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.686+0000 7fedd659c640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 0x7fedb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:22.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.716+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd80b1d00 con 0x7fedd80b0b20 2023-10-16T14:07:22.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.717+0000 7fedc6ffd640 1 -- 172.21.15.60:0/3377092170 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) v1 ==== 74+0+17739 (secure 0 0 0) 0x7fedd80b1d00 con 0x7fedd80b0b20 2023-10-16T14:07:22.721 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:22.722 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":200,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:21.828563+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"200","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":32,"snap_epoch":200,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1247236577":"2023-10-17T14:02:21.839066+0000","172.21.15.144:6833/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1895637100":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1129229162":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3061731632":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1022250268":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/103168750":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6833/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/2474164444":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1144561621":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3710907701":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1287992351":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3658895152":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1768527142":"2023-10-17T13:51:42.544813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":23,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:22.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 msgr2=0x7fedb806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 0x7fedb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 msgr2=0x7fedd80bf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd80bf970 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009eb0 tx=0x7fedd0004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 shutdown_connections 2023-10-16T14:07:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fedb806afe0 0x7fedb806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fedd80b0b20 0x7fedd80bf970 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd80adb50 0x7fedd80bf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 --2- 172.21.15.60:0/3377092170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fedd8070fd0 0x7fedd81f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 >> 172.21.15.60:0/3377092170 conn(0x7fedd80cb410 msgr2=0x7fedd80af640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 shutdown_connections 2023-10-16T14:07:22.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.720+0000 7feddc94e640 1 -- 172.21.15.60:0/3377092170 wait complete. 2023-10-16T14:07:22.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:51:25.863711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:51:26.910746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '200', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 200, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:07:22.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:07:22.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.953+0000 7f44bffff640 1 Processor -- start 2023-10-16T14:07:22.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.953+0000 7f44bffff640 1 -- start start 2023-10-16T14:07:22.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.954+0000 7f44bffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44c00cb000 0x7f44c00cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.954+0000 7f44bffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c0070810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.954+0000 7f44bffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c015d0e0 con 0x7f44c015cd00 2023-10-16T14:07:22.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.954+0000 7f44bffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c0070d50 con 0x7f44c00cb000 2023-10-16T14:07:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.954+0000 7f44bffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00cbd80 con 0x7f44c0070380 2023-10-16T14:07:22.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c0070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c0070810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:50422/0 (socket says 172.21.15.60:50422) 2023-10-16T14:07:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 -- 172.21.15.60:0/2952393824 learned_addr learned my addr 172.21.15.60:0/2952393824 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44be7fc640 1 --2- 172.21.15.60:0/2952393824 >> v2:172.21.15.144:3300/0 conn(0x7f44c00cb000 0x7f44c00cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 -- 172.21.15.60:0/2952393824 >> v2:172.21.15.144:3300/0 conn(0x7f44c00cb000 msgr2=0x7f44c00cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 --2- 172.21.15.60:0/2952393824 >> v2:172.21.15.144:3300/0 conn(0x7f44c00cb000 0x7f44c00cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 -- 172.21.15.60:0/2952393824 >> v1:172.21.15.144:6789/0 conn(0x7f44c015cd00 legacy=0x7f44c00cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:22.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.955+0000 7f44bf7fe640 1 -- 172.21.15.60:0/2952393824 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c00cbaa0 con 0x7f44c0070380 2023-10-16T14:07:22.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.956+0000 7f44bf7fe640 1 --2- 172.21.15.60:0/2952393824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c0070810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009e50 tx=0x7f44b802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=157dbee20f072b06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.957+0000 7f44bdffb640 1 -- 172.21.15.60:0/2952393824 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b802ca70 con 0x7f44c0070380 2023-10-16T14:07:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.957+0000 7f44bdffb640 1 -- 172.21.15.60:0/2952393824 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b802cbd0 con 0x7f44c0070380 2023-10-16T14:07:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.957+0000 7f44bdffb640 1 -- 172.21.15.60:0/2952393824 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80343f0 con 0x7f44c0070380 2023-10-16T14:07:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.957+0000 7f44bffff640 1 -- 172.21.15.60:0/2952393824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 msgr2=0x7f44c0070810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.957+0000 7f44bffff640 1 --2- 172.21.15.60:0/2952393824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c0070810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009e50 tx=0x7f44b802bed0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.958+0000 7f44bffff640 1 -- 172.21.15.60:0/2952393824 shutdown_connections 2023-10-16T14:07:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.958+0000 7f44bffff640 1 --2- 172.21.15.60:0/2952393824 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c0070810 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.958+0000 7f44bffff640 1 --2- 172.21.15.60:0/2952393824 >> v2:172.21.15.144:3300/0 conn(0x7f44c00cb000 0x7f44c00cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.958+0000 7f44bffff640 1 -- 172.21.15.60:0/2952393824 >> 172.21.15.60:0/2952393824 conn(0x7f44c00cf790 msgr2=0x7f44c00cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:22.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.958+0000 7f44bffff640 1 -- 172.21.15.60:0/2952393824 shutdown_connections 2023-10-16T14:07:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.958+0000 7f44bffff640 1 -- 172.21.15.60:0/2952393824 wait complete. 2023-10-16T14:07:22.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.959+0000 7f44bffff640 1 Processor -- start 2023-10-16T14:07:22.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 -- start start 2023-10-16T14:07:22.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c00ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c00cb000 0x7f44c01f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 0x7f44c014b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c01f61b0 con 0x7f44c015cd00 2023-10-16T14:07:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00ca860 con 0x7f44c00cb000 2023-10-16T14:07:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.960+0000 7f44bffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c01f5bc0 con 0x7f44c0070380 2023-10-16T14:07:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 0x7f44c014b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 0x7f44c014b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48592/0 (socket says 172.21.15.60:48592) 2023-10-16T14:07:22.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 -- 172.21.15.60:0/3512981514 learned_addr learned my addr 172.21.15.60:0/3512981514 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44beffd640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c00ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44be7fc640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c00cb000 0x7f44c01f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 -- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 msgr2=0x7f44c00ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c00ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 -- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c00cb000 msgr2=0x7f44c01f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c00cb000 0x7f44c01f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.961+0000 7f44bf7fe640 1 -- 172.21.15.60:0/3512981514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8009b30 con 0x7f44c015cd00 2023-10-16T14:07:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.962+0000 7f44bf7fe640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 0x7f44c014b870 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009e50 tx=0x7f44b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:22.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.962+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8009510 con 0x7f44c015cd00 2023-10-16T14:07:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.962+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b803e450 con 0x7f44c015cd00 2023-10-16T14:07:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.962+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c014c330 con 0x7f44c015cd00 2023-10-16T14:07:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.962+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b803e7e0 con 0x7f44c015cd00 2023-10-16T14:07:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.962+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c01f5dd0 con 0x7f44c015cd00 2023-10-16T14:07:22.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.963+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c01f62f0 con 0x7f44c015cd00 2023-10-16T14:07:22.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.964+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f44b803e940 con 0x7f44c015cd00 2023-10-16T14:07:22.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.965+0000 7f44a77fe640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f44ac06af90 0x7f44ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:22.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.966+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f44b80ac740 con 0x7f44c015cd00 2023-10-16T14:07:22.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.966+0000 7f44beffd640 1 -- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f44ac06af90 msgr2=0x7f44ac06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:22.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.966+0000 7f44beffd640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f44ac06af90 0x7f44ac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:22.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:22.970+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44c01f62f0 con 0x7f44c015cd00 2023-10-16T14:07:22.994 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:22.990+0000 7f20c783a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:07:23.047 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.042+0000 7f20c783a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:07:23.052 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.048+0000 7fecc8c12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:07:23.099 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.095+0000 7f20c783a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:07:23.106 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.102+0000 7fecc8c12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:07:23.158 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.155+0000 7fecc8c12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:07:23.159 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.155+0000 7f20c783a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:07:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.158+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44c01f62f0 con 0x7f44c015cd00 2023-10-16T14:07:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.159+0000 7f44a77fe640 1 -- 172.21.15.60:0/3512981514 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44c01f62f0 con 0x7f44c015cd00 2023-10-16T14:07:23.162 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:07:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f44ac06af90 msgr2=0x7f44ac06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f44ac06af90 0x7f44ac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 msgr2=0x7f44c014b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 0x7f44c014b870 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009e50 tx=0x7f44b8004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 shutdown_connections 2023-10-16T14:07:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f44ac06af90 0x7f44ac06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44c015cd00 0x7f44c014b870 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44c00cb000 0x7f44c01f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.161+0000 7f44bffff640 1 --2- 172.21.15.60:0/3512981514 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44c0070380 0x7f44c00ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.162+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 >> 172.21.15.60:0/3512981514 conn(0x7f44c00cf790 msgr2=0x7f44c00cfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.162+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 shutdown_connections 2023-10-16T14:07:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.162+0000 7f44bffff640 1 -- 172.21.15.60:0/3512981514 wait complete. 2023-10-16T14:07:23.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:07:23.207 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.203+0000 7f20c783a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:07:23.226 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.222+0000 7fecc8c12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:07:23.281 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.278+0000 7fecc8c12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:07:23.321 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.317+0000 7f20c783a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:07:23.384 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.381+0000 7f20c783a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:23.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.392+0000 7f7947dd5640 1 Processor -- start 2023-10-16T14:07:23.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.392+0000 7f7947dd5640 1 -- start start 2023-10-16T14:07:23.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7947dd5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7940159ab0 0x7f79400ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7947dd5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940070360 con 0x7f794006c140 2023-10-16T14:07:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7947dd5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79400cd1f0 con 0x7f7940159e90 2023-10-16T14:07:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7947dd5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794015d890 con 0x7f7940159ab0 2023-10-16T14:07:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7945b71640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7940159ab0 0x7f79400ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7946372640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f794006c140 0x7f794015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49392/0 (socket says 172.21.15.60:49392) 2023-10-16T14:07:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.394+0000 7f7946372640 1 -- 172.21.15.60:0/3021770341 learned_addr learned my addr 172.21.15.60:0/3021770341 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:23.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.395+0000 7f7945b71640 1 -- 172.21.15.60:0/3021770341 >> v1:172.21.15.144:6789/0 conn(0x7f7940159e90 legacy=0x7f794006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:23.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.395+0000 7f7945b71640 1 -- 172.21.15.60:0/3021770341 >> v1:172.21.15.60:6789/0 conn(0x7f794006c140 legacy=0x7f794015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.395+0000 7f7945b71640 1 -- 172.21.15.60:0/3021770341 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79400cd4d0 con 0x7f7940159ab0 2023-10-16T14:07:23.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.395+0000 7f7944b6f640 1 -- 172.21.15.60:0/3021770341 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795327745 0 0) 0x7f7940070360 con 0x7f794006c140 2023-10-16T14:07:23.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.396+0000 7f7945b71640 1 --2- 172.21.15.60:0/3021770341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79400ce370 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f793c0097a0 tx=0x7f793c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46f0febfd43247d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:23.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.396+0000 7f7944b6f640 1 -- 172.21.15.60:0/3021770341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c02f050 con 0x7f7940159ab0 2023-10-16T14:07:23.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7944b6f640 1 -- 172.21.15.60:0/3021770341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c031700 con 0x7f7940159ab0 2023-10-16T14:07:23.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7944b6f640 1 -- 172.21.15.60:0/3021770341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c031a90 con 0x7f7940159ab0 2023-10-16T14:07:23.402 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.397+0000 7fecc8c12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:07:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7947dd5640 1 -- 172.21.15.60:0/3021770341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 msgr2=0x7f79400ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7947dd5640 1 --2- 172.21.15.60:0/3021770341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79400ce370 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f793c0097a0 tx=0x7f793c02b7a0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7947dd5640 1 -- 172.21.15.60:0/3021770341 shutdown_connections 2023-10-16T14:07:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7947dd5640 1 --2- 172.21.15.60:0/3021770341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79400ce370 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7947dd5640 1 -- 172.21.15.60:0/3021770341 >> 172.21.15.60:0/3021770341 conn(0x7f79400ce9b0 msgr2=0x7f79400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.397+0000 7f7947dd5640 1 -- 172.21.15.60:0/3021770341 shutdown_connections 2023-10-16T14:07:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.398+0000 7f7947dd5640 1 -- 172.21.15.60:0/3021770341 wait complete. 2023-10-16T14:07:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.399+0000 7f7947dd5640 1 Processor -- start 2023-10-16T14:07:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.399+0000 7f7947dd5640 1 -- start start 2023-10-16T14:07:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.400+0000 7f7947dd5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f794006c140 0x7f794014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.400+0000 7f7947dd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79401564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.400+0000 7f7947dd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7940159e90 0x7f7940156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.400+0000 7f7947dd5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940154af0 con 0x7f7940159ab0 2023-10-16T14:07:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.400+0000 7f7947dd5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940154dd0 con 0x7f794006c140 2023-10-16T14:07:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.400+0000 7f7947dd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79401550b0 con 0x7f7940159e90 2023-10-16T14:07:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79401564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79401564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48604/0 (socket says 172.21.15.60:48604) 2023-10-16T14:07:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 -- 172.21.15.60:0/2192613598 learned_addr learned my addr 172.21.15.60:0/2192613598 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945b71640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f794006c140 0x7f794014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7946372640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7940159e90 0x7f7940156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 -- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7940159e90 msgr2=0x7f7940156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7940159e90 0x7f7940156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 -- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f794006c140 msgr2=0x7f794014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f794006c140 0x7f794014e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.401+0000 7f7945370640 1 -- 172.21.15.60:0/2192613598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c009400 con 0x7f7940159ab0 2023-10-16T14:07:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.402+0000 7f7945370640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79401564d0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f793000c430 tx=0x7f793000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.402+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930011790 con 0x7f7940159ab0 2023-10-16T14:07:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.402+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79300118f0 con 0x7f7940159ab0 2023-10-16T14:07:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.402+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930011ce0 con 0x7f7940159ab0 2023-10-16T14:07:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.402+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940155390 con 0x7f7940159ab0 2023-10-16T14:07:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.402+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794014e5a0 con 0x7f7940159ab0 2023-10-16T14:07:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.403+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794004e160 con 0x7f7940159ab0 2023-10-16T14:07:23.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.404+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f793001c460 con 0x7f7940159ab0 2023-10-16T14:07:23.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.405+0000 7f79367fc640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 0x7f792406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.405+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f7930089a10 con 0x7f7940159ab0 2023-10-16T14:07:23.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.405+0000 7f7945b71640 1 -- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 msgr2=0x7f792406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:23.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.405+0000 7f7945b71640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 0x7f792406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:23.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.409+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79300546a0 con 0x7f7940159ab0 2023-10-16T14:07:23.434 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.430+0000 7f20c783a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:07:23.469 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.465+0000 7fecc8c12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:23.490 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.486+0000 7f20c783a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:07:23.521 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.518+0000 7fecc8c12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:07:23.540 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.536+0000 7f20c783a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:07:23.581 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.578+0000 7fecc8c12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.604+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f79400adf40 con 0x7f7940159ab0 2023-10-16T14:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.605+0000 7f79367fc640 1 -- 172.21.15.60:0/2192613598 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v201) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7930057cc0 con 0x7f7940159ab0 2023-10-16T14:07:23.609 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:07:23.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.606+0000 7f7945b71640 1 -- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 msgr2=0x7f792406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:23.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.606+0000 7f7945b71640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 0x7f792406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:23.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 msgr2=0x7f792406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:23.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 0x7f792406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 msgr2=0x7f79401564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79401564d0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f793000c430 tx=0x7f793000c8f0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 shutdown_connections 2023-10-16T14:07:23.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f792406af90 0x7f792406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7940159e90 0x7f7940156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7940159ab0 0x7f79401564d0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 --2- 172.21.15.60:0/2192613598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f794006c140 0x7f794014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 >> 172.21.15.60:0/2192613598 conn(0x7f79400ce9b0 msgr2=0x7f79400ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:23.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 shutdown_connections 2023-10-16T14:07:23.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.608+0000 7f7947dd5640 1 -- 172.21.15.60:0/2192613598 wait complete. 2023-10-16T14:07:23.630 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:07:23.636 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:23.633+0000 7fecc8c12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:07:23.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.833+0000 7f35ba489640 1 Processor -- start 2023-10-16T14:07:23.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.833+0000 7f35ba489640 1 -- start start 2023-10-16T14:07:23.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.834+0000 7f35ba489640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.834+0000 7f35ba489640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35b40cd260 0x7f35b40cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.834+0000 7f35ba489640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b40b5d90 con 0x7f35b415d410 2023-10-16T14:07:23.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.834+0000 7f35ba489640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b40d2b50 con 0x7f35b40cd260 2023-10-16T14:07:23.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.834+0000 7f35ba489640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b4160fc0 con 0x7f35b415d030 2023-10-16T14:07:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35b40cd260 0x7f35b40cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35b40cd260 0x7f35b40cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48618/0 (socket says 172.21.15.60:48618) 2023-10-16T14:07:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 -- 172.21.15.60:0/3332312052 learned_addr learned my addr 172.21.15.60:0/3332312052 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b3fff640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 -- 172.21.15.60:0/3332312052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 msgr2=0x7f35b40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:23.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b40d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 -- 172.21.15.60:0/3332312052 >> v1:172.21.15.60:6789/0 conn(0x7f35b415d410 legacy=0x7f35b4161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b8a26640 1 -- 172.21.15.60:0/3332312052 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40d25d0 con 0x7f35b40cd260 2023-10-16T14:07:23.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.835+0000 7f35b3fff640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:07:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.836+0000 7f35b8a26640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40cd260 0x7f35b40cd6f0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009e50 tx=0x7f35ac02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2d9ac179617ab8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35b2ffd640 1 -- 172.21.15.60:0/3332312052 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac02cb90 con 0x7f35b40cd260 2023-10-16T14:07:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35b2ffd640 1 -- 172.21.15.60:0/3332312052 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac02ccf0 con 0x7f35b40cd260 2023-10-16T14:07:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35b2ffd640 1 -- 172.21.15.60:0/3332312052 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac034610 con 0x7f35b40cd260 2023-10-16T14:07:23.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35ba489640 1 -- 172.21.15.60:0/3332312052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40cd260 msgr2=0x7f35b40cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35ba489640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40cd260 0x7f35b40cd6f0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009e50 tx=0x7f35ac02bff0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35ba489640 1 -- 172.21.15.60:0/3332312052 shutdown_connections 2023-10-16T14:07:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35ba489640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40cd260 0x7f35b40cd6f0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35ba489640 1 --2- 172.21.15.60:0/3332312052 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.837+0000 7f35ba489640 1 -- 172.21.15.60:0/3332312052 >> 172.21.15.60:0/3332312052 conn(0x7f35b40cf720 msgr2=0x7f35b40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:23.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.838+0000 7f35ba489640 1 -- 172.21.15.60:0/3332312052 shutdown_connections 2023-10-16T14:07:23.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.838+0000 7f35ba489640 1 -- 172.21.15.60:0/3332312052 wait complete. 2023-10-16T14:07:23.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.839+0000 7f35ba489640 1 Processor -- start 2023-10-16T14:07:23.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.839+0000 7f35ba489640 1 -- start start 2023-10-16T14:07:23.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.840+0000 7f35ba489640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35b40cd260 0x7f35b41f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.840+0000 7f35ba489640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b415d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.840+0000 7f35ba489640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 0x7f35b415ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.840+0000 7f35ba489640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40c7210 con 0x7f35b415d410 2023-10-16T14:07:23.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.840+0000 7f35ba489640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b415a190 con 0x7f35b40cd260 2023-10-16T14:07:23.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.840+0000 7f35ba489640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40c6870 con 0x7f35b415d030 2023-10-16T14:07:23.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 0x7f35b415ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 0x7f35b415ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48630/0 (socket says 172.21.15.60:48630) 2023-10-16T14:07:23.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 -- 172.21.15.60:0/2945858984 learned_addr learned my addr 172.21.15.60:0/2945858984 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:23.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b37fe640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b3fff640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35b40cd260 0x7f35b41f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:23.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 -- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 msgr2=0x7f35b415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b415d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 -- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35b40cd260 msgr2=0x7f35b41f2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:23.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35b40cd260 0x7f35b41f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:23.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 -- 172.21.15.60:0/2945858984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac009b30 con 0x7f35b415d410 2023-10-16T14:07:23.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.841+0000 7f35b8a26640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 0x7f35b415ddd0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009e50 tx=0x7f35ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:23.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.842+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac009510 con 0x7f35b415d410 2023-10-16T14:07:23.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.842+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac03e450 con 0x7f35b415d410 2023-10-16T14:07:23.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.842+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b40c6b50 con 0x7f35b415d410 2023-10-16T14:07:23.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.842+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac03e800 con 0x7f35b415d410 2023-10-16T14:07:23.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.842+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b4159410 con 0x7f35b415d410 2023-10-16T14:07:23.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.843+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b40c7350 con 0x7f35b415d410 2023-10-16T14:07:23.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.845+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f35ac03e960 con 0x7f35b415d410 2023-10-16T14:07:23.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.846+0000 7f35b0ff9640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 0x7f359406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:23.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.846+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f35ac0ac540 con 0x7f35b415d410 2023-10-16T14:07:23.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.846+0000 7f35b3fff640 1 -- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 msgr2=0x7f359406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:23.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.846+0000 7f35b3fff640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 0x7f359406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:23.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:23.851+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35b40c7350 con 0x7f35b415d410 2023-10-16T14:07:23.922 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:23.919+0000 7f20c783a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:07:24.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.047+0000 7f35b3fff640 1 -- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 msgr2=0x7f359406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:24.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.047+0000 7f35b3fff640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 0x7f359406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:24.051 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.047+0000 7fecc8c12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:07:24.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.053+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35b40c7350 con 0x7f35b415d410 2023-10-16T14:07:24.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.054+0000 7f35b0ff9640 1 -- 172.21.15.60:0/2945858984 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v201) v1 ==== 100+0+11 (secure 0 0 0) 0x7f35b40c7350 con 0x7f35b415d410 2023-10-16T14:07:24.058 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:07:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 msgr2=0x7f359406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 0x7f359406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 msgr2=0x7f35b415ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 0x7f35b415ddd0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009e50 tx=0x7f35ac004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 shutdown_connections 2023-10-16T14:07:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f359406afe0 0x7f359406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35b415d030 0x7f35b415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b415d410 0x7f35b415ddd0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 --2- 172.21.15.60:0/2945858984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f35b40cd260 0x7f35b41f2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.057+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 >> 172.21.15.60:0/2945858984 conn(0x7f35b40cf720 msgr2=0x7f35b415c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.058+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 shutdown_connections 2023-10-16T14:07:24.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.058+0000 7f35ba489640 1 -- 172.21.15.60:0/2945858984 wait complete. 2023-10-16T14:07:24.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:07:24.079 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.076+0000 7f20c783a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:07:24.186 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.182+0000 7f20c783a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:07:24.223 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.220+0000 7fecc8c12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:07:24.235 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.231+0000 7f20c783a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:07:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.281+0000 7f334cca2640 1 Processor -- start 2023-10-16T14:07:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.281+0000 7f334cca2640 1 -- start start 2023-10-16T14:07:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f334cca2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f33480cf710 0x7f33480b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f334cca2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f33480b6cd0 0x7f33480b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f334cca2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33480d3980 con 0x7f33480cfaf0 2023-10-16T14:07:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f334cca2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33480d3c60 con 0x7f33480b6cd0 2023-10-16T14:07:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f334cca2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33480ce630 con 0x7f33480cf710 2023-10-16T14:07:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f3346d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f33480b6cd0 0x7f33480b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f3346d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f33480b6cd0 0x7f33480b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48640/0 (socket says 172.21.15.60:48640) 2023-10-16T14:07:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f3346d9d640 1 -- 172.21.15.60:0/840786702 learned_addr learned my addr 172.21.15.60:0/840786702 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.283+0000 7f334659c640 1 --2- 172.21.15.60:0/840786702 >> v2:172.21.15.144:3300/0 conn(0x7f33480cf710 0x7f33480b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.284+0000 7f3346d9d640 1 -- 172.21.15.60:0/840786702 >> v2:172.21.15.144:3300/0 conn(0x7f33480cf710 msgr2=0x7f33480b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.284+0000 7f3346d9d640 1 --2- 172.21.15.60:0/840786702 >> v2:172.21.15.144:3300/0 conn(0x7f33480cf710 0x7f33480b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.284+0000 7f3346d9d640 1 -- 172.21.15.60:0/840786702 >> v1:172.21.15.144:6789/0 conn(0x7f33480cfaf0 legacy=0x7f33480d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.284+0000 7f3346d9d640 1 -- 172.21.15.60:0/840786702 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480ce910 con 0x7f33480b6cd0 2023-10-16T14:07:24.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.285+0000 7f3346d9d640 1 --2- 172.21.15.60:0/840786702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b3250 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f33480d2c20 tx=0x7f334002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91184e69615bcdd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.285+0000 7f334559a640 1 -- 172.21.15.60:0/840786702 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334002c9f0 con 0x7f33480b6cd0 2023-10-16T14:07:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.285+0000 7f334559a640 1 -- 172.21.15.60:0/840786702 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334002cb50 con 0x7f33480b6cd0 2023-10-16T14:07:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.285+0000 7f334559a640 1 -- 172.21.15.60:0/840786702 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33400343f0 con 0x7f33480b6cd0 2023-10-16T14:07:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 -- 172.21.15.60:0/840786702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 msgr2=0x7f33480b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 --2- 172.21.15.60:0/840786702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b3250 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f33480d2c20 tx=0x7f334002be50 comp rx=0 tx=0).stop 2023-10-16T14:07:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 -- 172.21.15.60:0/840786702 shutdown_connections 2023-10-16T14:07:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 --2- 172.21.15.60:0/840786702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b3250 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 --2- 172.21.15.60:0/840786702 >> v2:172.21.15.144:3300/0 conn(0x7f33480cf710 0x7f33480b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 -- 172.21.15.60:0/840786702 >> 172.21.15.60:0/840786702 conn(0x7f33480d2cc0 msgr2=0x7f33480d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 -- 172.21.15.60:0/840786702 shutdown_connections 2023-10-16T14:07:24.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.286+0000 7f334cca2640 1 -- 172.21.15.60:0/840786702 wait complete. 2023-10-16T14:07:24.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.288+0000 7f334cca2640 1 Processor -- start 2023-10-16T14:07:24.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.289+0000 7f334cca2640 1 -- start start 2023-10-16T14:07:24.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.290+0000 7f334cca2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:24.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.290+0000 7f334cca2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33480cf710 0x7f334809c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:24.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.290+0000 7f334cca2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33480cfaf0 0x7f3348152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:24.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.290+0000 7f334cca2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33480b2690 con 0x7f33480b6cd0 2023-10-16T14:07:24.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.290+0000 7f334cca2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334809de00 con 0x7f33480cf710 2023-10-16T14:07:24.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.290+0000 7f334cca2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334809d1f0 con 0x7f33480cfaf0 2023-10-16T14:07:24.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:24.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48646/0 (socket says 172.21.15.60:48646) 2023-10-16T14:07:24.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 -- 172.21.15.60:0/608886526 learned_addr learned my addr 172.21.15.60:0/608886526 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:24.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f3346d9d640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33480cfaf0 0x7f3348152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f3345d9b640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33480cf710 0x7f334809c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 -- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33480cfaf0 msgr2=0x7f3348152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33480cfaf0 0x7f3348152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 -- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33480cf710 msgr2=0x7f334809c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33480cf710 0x7f334809c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 -- 172.21.15.60:0/608886526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340009b30 con 0x7f33480b6cd0 2023-10-16T14:07:24.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f3346d9d640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33480cfaf0 0x7f3348152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:07:24.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.291+0000 7f334659c640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b2150 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f333000d7d0 tx=0x7f333000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:24.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.292+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33300096c0 con 0x7f33480b6cd0 2023-10-16T14:07:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.292+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3330009820 con 0x7f33480b6cd0 2023-10-16T14:07:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.292+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330009bd0 con 0x7f33480b6cd0 2023-10-16T14:07:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.292+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334809d4d0 con 0x7f33480b6cd0 2023-10-16T14:07:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.292+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334809da90 con 0x7f33480b6cd0 2023-10-16T14:07:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.293+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33480ce050 con 0x7f33480b6cd0 2023-10-16T14:07:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.295+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f3330009d30 con 0x7f33480b6cd0 2023-10-16T14:07:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.296+0000 7f333effd640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 0x7f332806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.296+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f333008a2b0 con 0x7f33480b6cd0 2023-10-16T14:07:24.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.296+0000 7f3345d9b640 1 -- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 msgr2=0x7f332806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:24.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.296+0000 7f3345d9b640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 0x7f332806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.300+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3330054f40 con 0x7f33480b6cd0 2023-10-16T14:07:24.341 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.338+0000 7fecc8c12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:07:24.343 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.340+0000 7f20c783a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:24.394 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.390+0000 7fecc8c12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:07:24.449 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.446+0000 7f20c783a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:07:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.497+0000 7f3345d9b640 1 -- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 msgr2=0x7f332806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.497+0000 7f3345d9b640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 0x7f332806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:24.507 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.504+0000 7fecc8c12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:24.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.513+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f334804e160 con 0x7f33480b6cd0 2023-10-16T14:07:24.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.514+0000 7f333effd640 1 -- 172.21.15.60:0/608886526 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v201) v1 ==== 81+0+953 (secure 0 0 0) 0x7f3330058560 con 0x7f33480b6cd0 2023-10-16T14:07:24.518 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1247236577 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.518 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/861800364 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.518 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1895637100 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.519 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1129229162 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.519 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3061731632 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.519 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1022250268 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.519 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/103168750 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.520 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/1700115305 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.520 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2474164444 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.520 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/1700115305 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.520 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1144561621 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.520 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3710907701 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.521 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/861800364 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.521 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1287992351 2023-10-17T14:02:21.839066+0000 2023-10-16T14:07:24.521 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3658895152 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.522 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1768527142 2023-10-17T13:51:42.544813+0000 2023-10-16T14:07:24.523 INFO:teuthology.orchestra.run.smithi060.stderr:listed 16 entries 2023-10-16T14:07:24.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 msgr2=0x7f332806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:24.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 0x7f332806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 msgr2=0x7f33480b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:24.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b2150 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f333000d7d0 tx=0x7f333000dc90 comp rx=0 tx=0).stop 2023-10-16T14:07:24.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 shutdown_connections 2023-10-16T14:07:24.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f332806af90 0x7f332806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33480cfaf0 0x7f3348152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33480cf710 0x7f334809c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 --2- 172.21.15.60:0/608886526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33480b6cd0 0x7f33480b2150 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 >> 172.21.15.60:0/608886526 conn(0x7f33480d2cc0 msgr2=0x7f33480b7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 shutdown_connections 2023-10-16T14:07:24.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:24.517+0000 7f334cca2640 1 -- 172.21.15.60:0/608886526 wait complete. 2023-10-16T14:07:24.539 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:07:24.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:24.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:07:24.604 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.600+0000 7f20c783a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:24.624 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.621+0000 7fecc8c12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:07:24.644 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:07:24.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:24.645 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:07:24.654 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.651+0000 7f20c783a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:24.690 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:07:24.691 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:24.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:07:24.747 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:07:24.748 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:24.777 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:24.773+0000 7f20c783a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:07:24.791 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.787+0000 7fecc8c12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:24.804 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:07:24.804 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:07:24.805 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:24.842 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.839+0000 7fecc8c12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:24.862 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:07:24.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:24.970 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:24.966+0000 7fecc8c12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:07:25.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.069+0000 7fe37caa9640 1 Processor -- start 2023-10-16T14:07:25.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.069+0000 7fe37caa9640 1 -- start start 2023-10-16T14:07:25.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.071+0000 7fe37caa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 0x7fe3780cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.071+0000 7fe37caa9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3781614f0 con 0x7fe3780ae220 2023-10-16T14:07:25.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.071+0000 7fe37caa9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3781617d0 con 0x7fe37806a4b0 2023-10-16T14:07:25.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.071+0000 7fe37caa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780cf760 con 0x7fe378160f30 2023-10-16T14:07:25.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.071+0000 7fe37659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3780ae220 0x7fe378069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49406/0 (socket says 172.21.15.60:49406) 2023-10-16T14:07:25.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.071+0000 7fe37659c640 1 -- 172.21.15.60:0/2892638948 learned_addr learned my addr 172.21.15.60:0/2892638948 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:25.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 0x7fe3780cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580318423 0 0) 0x7fe3781614f0 con 0x7fe3780ae220 2023-10-16T14:07:25.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe360003520 con 0x7fe3780ae220 2023-10-16T14:07:25.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903698866 0 0) 0x7fe360003520 con 0x7fe3780ae220 2023-10-16T14:07:25.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 msgr2=0x7fe3780cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 0x7fe3780cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 >> v1:172.21.15.144:6789/0 conn(0x7fe37806a4b0 legacy=0x7fe3780cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378161ab0 con 0x7fe3780ae220 2023-10-16T14:07:25.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe368001690 con 0x7fe3780ae220 2023-10-16T14:07:25.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 >> v1:172.21.15.60:6789/0 conn(0x7fe3780ae220 legacy=0x7fe378069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3600039a0 0x7fe360003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 0x7fe3600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe360004d00 0x7fe360005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780cf760 con 0x7fe360004d00 2023-10-16T14:07:25.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36000ba30 con 0x7fe3600039a0 2023-10-16T14:07:25.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.072+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36000bbf0 con 0x7fe378160f30 2023-10-16T14:07:25.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe360004d00 0x7fe360005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe37659c640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3600039a0 0x7fe360003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe375d9b640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 0x7fe3600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe376d9d640 1 -- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 msgr2=0x7fe3600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe378160f30 0x7fe3600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe376d9d640 1 -- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3600039a0 msgr2=0x7fe360003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3600039a0 0x7fe360003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.073+0000 7fe376d9d640 1 -- 172.21.15.60:0/2892638948 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe360003520 con 0x7fe360004d00 2023-10-16T14:07:25.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.074+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe360004d00 0x7fe360005180 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fe370009f90 tx=0x7fe3600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865648ae7c24a348 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:25.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.074+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37002cbc0 con 0x7fe360004d00 2023-10-16T14:07:25.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.074+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37000f040 con 0x7fe360004d00 2023-10-16T14:07:25.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.074+0000 7fe37559a640 1 -- 172.21.15.60:0/2892638948 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370034440 con 0x7fe360004d00 2023-10-16T14:07:25.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe37caa9640 1 -- 172.21.15.60:0/2892638948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe360004d00 msgr2=0x7fe360005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2892638948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe360004d00 0x7fe360005180 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fe370009f90 tx=0x7fe3600057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe376d9d640 1 -- 172.21.15.60:0/2892638948 reap_dead start 2023-10-16T14:07:25.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe37caa9640 1 -- 172.21.15.60:0/2892638948 shutdown_connections 2023-10-16T14:07:25.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe37caa9640 1 -- 172.21.15.60:0/2892638948 >> 172.21.15.60:0/2892638948 conn(0x7fe3780cfb70 msgr2=0x7fe3780cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:25.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe37caa9640 1 -- 172.21.15.60:0/2892638948 shutdown_connections 2023-10-16T14:07:25.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.075+0000 7fe37caa9640 1 -- 172.21.15.60:0/2892638948 wait complete. 2023-10-16T14:07:25.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.077+0000 7fe37caa9640 1 Processor -- start 2023-10-16T14:07:25.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.077+0000 7fe37caa9640 1 -- start start 2023-10-16T14:07:25.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe37caa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe360004d00 0x7fe37814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe37caa9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3780b4610 0x7fe3780b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe37caa9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 0x7fe3780b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe37caa9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780af560 con 0x7fe3780b2700 2023-10-16T14:07:25.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe37caa9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780b30c0 con 0x7fe3780b4610 2023-10-16T14:07:25.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe37caa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3780af800 con 0x7fe360004d00 2023-10-16T14:07:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe376d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 0x7fe3780b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe376d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 0x7fe3780b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48658/0 (socket says 172.21.15.60:48658) 2023-10-16T14:07:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.078+0000 7fe376d9d640 1 -- 172.21.15.60:0/2689936703 learned_addr learned my addr 172.21.15.60:0/2689936703 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe37659c640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe360004d00 0x7fe37814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe375d9b640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3780b4610 0x7fe3780b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe376d9d640 1 -- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe360004d00 msgr2=0x7fe37814ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe360004d00 0x7fe37814ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe376d9d640 1 -- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3780b4610 msgr2=0x7fe3780b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3780b4610 0x7fe3780b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe376d9d640 1 -- 172.21.15.60:0/2689936703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37814b340 con 0x7fe3780b2700 2023-10-16T14:07:25.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe376d9d640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 0x7fe3780b2b80 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fe370037040 tx=0x7fe370009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:25.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370003b90 con 0x7fe3780b2700 2023-10-16T14:07:25.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe370003d10 con 0x7fe3780b2700 2023-10-16T14:07:25.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3780b5160 con 0x7fe3780b2700 2023-10-16T14:07:25.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370034b60 con 0x7fe3780b2700 2023-10-16T14:07:25.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.079+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3780b5720 con 0x7fe3780b2700 2023-10-16T14:07:25.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.080+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3780b5940 con 0x7fe3780b2700 2023-10-16T14:07:25.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.082+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe370034cc0 con 0x7fe3780b2700 2023-10-16T14:07:25.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.082+0000 7fe366ffd640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 0x7fe35806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.083+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7fe3700ab270 con 0x7fe3780b2700 2023-10-16T14:07:25.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.083+0000 7fe37659c640 1 -- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 msgr2=0x7fe35806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:25.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.083+0000 7fe37659c640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 0x7fe35806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:25.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.087+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe370075f20 con 0x7fe3780b2700 2023-10-16T14:07:25.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.284+0000 7fe37659c640 1 -- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 msgr2=0x7fe35806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:25.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.284+0000 7fe37659c640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 0x7fe35806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:25.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.317+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe37806aa30 con 0x7fe3780b2700 2023-10-16T14:07:25.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.318+0000 7fe366ffd640 1 -- 172.21.15.60:0/2689936703 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fe37007ae60 con 0x7fe3780b2700 2023-10-16T14:07:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:25.323 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:51:27.881990+0000","modified":"2023-10-16T13:51:30.974922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9966":{"gid":9966,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/3850041246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3850041246},{"type":"v1","addr":"172.21.15.60:6835","nonce":3850041246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T14:07:25.323 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 95 2023-10-16T14:07:25.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 msgr2=0x7fe35806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:25.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 0x7fe35806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 msgr2=0x7fe3780b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 0x7fe3780b2b80 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fe370037040 tx=0x7fe370009750 comp rx=0 tx=0).stop 2023-10-16T14:07:25.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 shutdown_connections 2023-10-16T14:07:25.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7fe35806af90 0x7fe35806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3780b2700 0x7fe3780b2b80 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3780b4610 0x7fe3780b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 --2- 172.21.15.60:0/2689936703 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe360004d00 0x7fe37814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 >> 172.21.15.60:0/2689936703 conn(0x7fe3780cfb70 msgr2=0x7fe37815fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:25.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 shutdown_connections 2023-10-16T14:07:25.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.322+0000 7fe37caa9640 1 -- 172.21.15.60:0/2689936703 wait complete. 2023-10-16T14:07:25.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:25.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.539+0000 7f525f494640 1 Processor -- start 2023-10-16T14:07:25.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.539+0000 7f525f494640 1 -- start start 2023-10-16T14:07:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.540+0000 7f525f494640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f525815aaa0 0x7f525815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525f494640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5258070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525f494640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525815ed60 con 0x7f525815ae80 2023-10-16T14:07:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525f494640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525806b810 con 0x7f525815aaa0 2023-10-16T14:07:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525f494640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52580707e0 con 0x7f52580ce390 2023-10-16T14:07:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525ca2f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f525815ae80 0x7f52580cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49408/0 (socket says 172.21.15.60:49408) 2023-10-16T14:07:25.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525ca2f640 1 -- 172.21.15.60:0/1594306565 learned_addr learned my addr 172.21.15.60:0/1594306565 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525da31640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5258070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f525d230640 1 --2- 172.21.15.60:0/1594306565 >> v2:172.21.15.144:3300/0 conn(0x7f525815aaa0 0x7f525815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380162549 0 0) 0x7f525815ed60 con 0x7f525815ae80 2023-10-16T14:07:25.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.541+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5244003520 con 0x7f525815ae80 2023-10-16T14:07:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1445810707 0 0) 0x7f5244003520 con 0x7f525815ae80 2023-10-16T14:07:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 msgr2=0x7f5258070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5258070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 >> v2:172.21.15.144:3300/0 conn(0x7f525815aaa0 msgr2=0x7f525815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 --2- 172.21.15.60:0/1594306565 >> v2:172.21.15.144:3300/0 conn(0x7f525815aaa0 0x7f525815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525815ef90 con 0x7f525815ae80 2023-10-16T14:07:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5248001690 con 0x7f525815ae80 2023-10-16T14:07:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 >> v1:172.21.15.60:6789/0 conn(0x7f525815ae80 legacy=0x7f52580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5244003970 0x7f5244003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440047c0 0x7f5244004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5244005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525806b810 con 0x7f52440047c0 2023-10-16T14:07:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52580707e0 con 0x7f5244003970 2023-10-16T14:07:25.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524400b980 con 0x7f52580ce390 2023-10-16T14:07:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.542+0000 7f525ca2f640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440047c0 0x7f5244004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525d230640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5244003970 0x7f5244003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525da31640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5244005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525ca2f640 1 -- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 msgr2=0x7f5244005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525ca2f640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5244005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525ca2f640 1 -- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5244003970 msgr2=0x7f5244003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525ca2f640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5244003970 0x7f5244003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525ca2f640 1 -- 172.21.15.60:0/1594306565 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5244003520 con 0x7f52440047c0 2023-10-16T14:07:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.543+0000 7f525da31640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52580ce390 0x7f5244005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.544+0000 7f525ca2f640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440047c0 0x7f5244004c40 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f525815b2f0 tx=0x7f52440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9754017dbb43e7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.544+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248005ad0 con 0x7f52440047c0 2023-10-16T14:07:25.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.544+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5248005c30 con 0x7f52440047c0 2023-10-16T14:07:25.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.544+0000 7f524ffff640 1 -- 172.21.15.60:0/1594306565 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524802c7b0 con 0x7f52440047c0 2023-10-16T14:07:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525f494640 1 -- 172.21.15.60:0/1594306565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440047c0 msgr2=0x7f5244004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525f494640 1 --2- 172.21.15.60:0/1594306565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440047c0 0x7f5244004c40 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f525815b2f0 tx=0x7f52440057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525da31640 1 -- 172.21.15.60:0/1594306565 reap_dead start 2023-10-16T14:07:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525f494640 1 -- 172.21.15.60:0/1594306565 shutdown_connections 2023-10-16T14:07:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525f494640 1 -- 172.21.15.60:0/1594306565 >> 172.21.15.60:0/1594306565 conn(0x7f52580cb410 msgr2=0x7f52580cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525f494640 1 -- 172.21.15.60:0/1594306565 shutdown_connections 2023-10-16T14:07:25.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.545+0000 7f525f494640 1 -- 172.21.15.60:0/1594306565 wait complete. 2023-10-16T14:07:25.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.547+0000 7f525f494640 1 Processor -- start 2023-10-16T14:07:25.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.547+0000 7f525f494640 1 -- start start 2023-10-16T14:07:25.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525f494640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52440047c0 0x7f52581efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525f494640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52580c6b50 0x7f52580c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525f494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 0x7f525815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525f494640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52580c34b0 con 0x7f525815c8d0 2023-10-16T14:07:25.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525f494640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52580c2830 con 0x7f52580c6b50 2023-10-16T14:07:25.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525f494640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52580c2b10 con 0x7f52440047c0 2023-10-16T14:07:25.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525da31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 0x7f525815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525da31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 0x7f525815cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48670/0 (socket says 172.21.15.60:48670) 2023-10-16T14:07:25.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.548+0000 7f525da31640 1 -- 172.21.15.60:0/3034014602 learned_addr learned my addr 172.21.15.60:0/3034014602 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:25.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525d230640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52440047c0 0x7f52581efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525ca2f640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52580c6b50 0x7f52580c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:25.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525da31640 1 -- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52440047c0 msgr2=0x7f52581efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525da31640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52440047c0 0x7f52581efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525da31640 1 -- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52580c6b50 msgr2=0x7f52580c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525da31640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52580c6b50 0x7f52580c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525da31640 1 -- 172.21.15.60:0/3034014602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52580c2df0 con 0x7f525815c8d0 2023-10-16T14:07:25.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.549+0000 7f525da31640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 0x7f525815cd50 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f52580c5990 tx=0x7f525000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:25.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.550+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250017400 con 0x7f525815c8d0 2023-10-16T14:07:25.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.550+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250011040 con 0x7f525815c8d0 2023-10-16T14:07:25.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.550+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250017780 con 0x7f525815c8d0 2023-10-16T14:07:25.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.550+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52580c30d0 con 0x7f525815c8d0 2023-10-16T14:07:25.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.550+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525815d290 con 0x7f525815c8d0 2023-10-16T14:07:25.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.551+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525804e160 con 0x7f525815c8d0 2023-10-16T14:07:25.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.553+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f52500203b0 con 0x7f525815c8d0 2023-10-16T14:07:25.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.554+0000 7f524dffb640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 0x7f523c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.554+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f5250089e60 con 0x7f525815c8d0 2023-10-16T14:07:25.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.554+0000 7f525d230640 1 -- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 msgr2=0x7f523c06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:25.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.554+0000 7f525d230640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 0x7f523c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:25.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.558+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52500549e0 con 0x7f525815c8d0 2023-10-16T14:07:25.609 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:25.605+0000 7f20c783a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:07:25.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.749+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52580cdc60 con 0x7f525815c8d0 2023-10-16T14:07:25.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.752+0000 7f524dffb640 1 -- 172.21.15.60:0/3034014602 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+17713 (secure 0 0 0) 0x7f5250058000 con 0x7f525815c8d0 2023-10-16T14:07:25.756 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:25.757 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":201,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:22.840535+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1247236577":"2023-10-17T14:02:21.839066+0000","172.21.15.144:6833/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1895637100":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1129229162":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3061731632":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1022250268":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/103168750":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6833/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/2474164444":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1144561621":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3710907701":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1287992351":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3658895152":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1768527142":"2023-10-17T13:51:42.544813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:25.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525d230640 1 -- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 msgr2=0x7f523c06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:25.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 msgr2=0x7f523c06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:25.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525d230640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 0x7f523c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:25.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 0x7f523c06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 msgr2=0x7f525815cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:25.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 0x7f525815cd50 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f52580c5990 tx=0x7f525000f4a0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 shutdown_connections 2023-10-16T14:07:25.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f523c06afc0 0x7f523c06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f525815c8d0 0x7f525815cd50 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52580c6b50 0x7f52580c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 --2- 172.21.15.60:0/3034014602 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52440047c0 0x7f52581efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:25.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 >> 172.21.15.60:0/3034014602 conn(0x7f52580cb410 msgr2=0x7f52580b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:25.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 shutdown_connections 2023-10-16T14:07:25.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.755+0000 7f525f494640 1 -- 172.21.15.60:0/3034014602 wait complete. 2023-10-16T14:07:25.777 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:51:25.863711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:51:26.910746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 201, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:07:25.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:07:25.814 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:25.811+0000 7fecc8c12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:07:25.905 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:25.901+0000 7f20c783a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:07:25.961 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:25.958+0000 7f20c783a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:07:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.993+0000 7f1984cb4640 1 Processor -- start 2023-10-16T14:07:25.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.994+0000 7f1984cb4640 1 -- start start 2023-10-16T14:07:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f1984cb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198015ce00 0x7f1980160980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f1984cb4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f198006a4b0 0x7f19800ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f1984cb4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19801618b0 con 0x7f198006ea40 2023-10-16T14:07:25.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f1984cb4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800cdc60 con 0x7f198006a4b0 2023-10-16T14:07:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f1984cb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1980161010 con 0x7f198015ce00 2023-10-16T14:07:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f197ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f198006a4b0 0x7f19800ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f197ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f198006a4b0 0x7f19800ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48684/0 (socket says 172.21.15.60:48684) 2023-10-16T14:07:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f197ed9d640 1 -- 172.21.15.60:0/2119310435 learned_addr learned my addr 172.21.15.60:0/2119310435 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.995+0000 7f197e59c640 1 --2- 172.21.15.60:0/2119310435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198015ce00 0x7f1980160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.996+0000 7f197ed9d640 1 -- 172.21.15.60:0/2119310435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198015ce00 msgr2=0x7f1980160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:26.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.996+0000 7f197ed9d640 1 --2- 172.21.15.60:0/2119310435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198015ce00 0x7f1980160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.996+0000 7f197ed9d640 1 -- 172.21.15.60:0/2119310435 >> v1:172.21.15.144:6789/0 conn(0x7f198006ea40 legacy=0x7f1980069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:26.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.996+0000 7f197ed9d640 1 -- 172.21.15.60:0/2119310435 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19801612f0 con 0x7f198006a4b0 2023-10-16T14:07:26.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.997+0000 7f197ed9d640 1 --2- 172.21.15.60:0/2119310435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198006a4b0 0x7f19800ce080 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f1978009e50 tx=0x7f197802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a5a4327ce8fa81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:26.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.997+0000 7f197dd9b640 1 -- 172.21.15.60:0/2119310435 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197802c9e0 con 0x7f198006a4b0 2023-10-16T14:07:26.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.997+0000 7f197dd9b640 1 -- 172.21.15.60:0/2119310435 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197802cb40 con 0x7f198006a4b0 2023-10-16T14:07:26.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.997+0000 7f197dd9b640 1 -- 172.21.15.60:0/2119310435 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19780343f0 con 0x7f198006a4b0 2023-10-16T14:07:26.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 -- 172.21.15.60:0/2119310435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198006a4b0 msgr2=0x7f19800ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 --2- 172.21.15.60:0/2119310435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198006a4b0 0x7f19800ce080 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f1978009e50 tx=0x7f197802be40 comp rx=0 tx=0).stop 2023-10-16T14:07:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 -- 172.21.15.60:0/2119310435 shutdown_connections 2023-10-16T14:07:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 --2- 172.21.15.60:0/2119310435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198006a4b0 0x7f19800ce080 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 --2- 172.21.15.60:0/2119310435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198015ce00 0x7f1980160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 -- 172.21.15.60:0/2119310435 >> 172.21.15.60:0/2119310435 conn(0x7f19800cc460 msgr2=0x7f19800cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:26.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 -- 172.21.15.60:0/2119310435 shutdown_connections 2023-10-16T14:07:26.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:25.998+0000 7f1984cb4640 1 -- 172.21.15.60:0/2119310435 wait complete. 2023-10-16T14:07:26.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.000+0000 7f1984cb4640 1 Processor -- start 2023-10-16T14:07:26.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.000+0000 7f1984cb4640 1 -- start start 2023-10-16T14:07:26.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f1984cb4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f198006a4b0 0x7f1980156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f1984cb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198006ea40 0x7f1980146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f1984cb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 0x7f19801f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f1984cb4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19801577f0 con 0x7f198015ce00 2023-10-16T14:07:26.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f1984cb4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19801465b0 con 0x7f198006a4b0 2023-10-16T14:07:26.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f1984cb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1980156b70 con 0x7f198006ea40 2023-10-16T14:07:26.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 0x7f19801f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f197ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 0x7f19801f5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48696/0 (socket says 172.21.15.60:48696) 2023-10-16T14:07:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.001+0000 7f197ed9d640 1 -- 172.21.15.60:0/3201890641 learned_addr learned my addr 172.21.15.60:0/3201890641 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197e59c640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f198006a4b0 0x7f1980156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f1977fff640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198006ea40 0x7f1980146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197ed9d640 1 -- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198006ea40 msgr2=0x7f1980146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197ed9d640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198006ea40 0x7f1980146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197ed9d640 1 -- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f198006a4b0 msgr2=0x7f1980156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197ed9d640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f198006a4b0 0x7f1980156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197ed9d640 1 -- 172.21.15.60:0/3201890641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1978009b30 con 0x7f198015ce00 2023-10-16T14:07:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f197ed9d640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 0x7f19801f5290 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f1978036040 tx=0x7f1978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1978009510 con 0x7f198015ce00 2023-10-16T14:07:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197803e450 con 0x7f198015ce00 2023-10-16T14:07:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197803e7e0 con 0x7f198015ce00 2023-10-16T14:07:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.002+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1980156e50 con 0x7f198015ce00 2023-10-16T14:07:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.003+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1980157410 con 0x7f198015ce00 2023-10-16T14:07:26.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.004+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1980157930 con 0x7f198015ce00 2023-10-16T14:07:26.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.005+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 111) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f197803e940 con 0x7f198015ce00 2023-10-16T14:07:26.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.006+0000 7f1976ffd640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f196006af90 0x7f196006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.006+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f19780ac8f0 con 0x7f198015ce00 2023-10-16T14:07:26.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.006+0000 7f197e59c640 1 -- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f196006af90 msgr2=0x7f196006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/345630502 2023-10-16T14:07:26.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.006+0000 7f197e59c640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f196006af90 0x7f196006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:26.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.011+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1978077620 con 0x7f198015ce00 2023-10-16T14:07:26.076 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.072+0000 7f20c783a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:07:26.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.091+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 112) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f19780779f0 con 0x7f198015ce00 2023-10-16T14:07:26.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.091+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f196006af90 msgr2=0x7f196006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:26.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.091+0000 7f1976ffd640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f196006af90 0x7f196006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.138 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:26.135+0000 7fecc8c12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:07:26.173 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f204c391640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.173 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f204c391640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.173 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f204c391640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.173 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f204c391640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f204c391640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f2048389640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f2048389640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f2048389640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f2048389640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.175 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:26.169+0000 7f2048389640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:26.200 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:26.196+0000 7fecc8c12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:07:26.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.216+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f198004e160 con 0x7f198015ce00 2023-10-16T14:07:26.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.217+0000 7f1976ffd640 1 -- 172.21.15.60:0/3201890641 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v202) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1978081020 con 0x7f198015ce00 2023-10-16T14:07:26.220 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:07:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 msgr2=0x7f19801f5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 0x7f19801f5290 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f1978036040 tx=0x7f1978004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 shutdown_connections 2023-10-16T14:07:26.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:6832/345630502,v1:172.21.15.144:6833/345630502] conn(0x7f196006af90 0x7f196006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f198006ea40 0x7f1980146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f198015ce00 0x7f19801f5290 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 --2- 172.21.15.60:0/3201890641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f198006a4b0 0x7f1980156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 >> 172.21.15.60:0/3201890641 conn(0x7f19800cc460 msgr2=0x7f1980070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:26.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 shutdown_connections 2023-10-16T14:07:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.219+0000 7f1984cb4640 1 -- 172.21.15.60:0/3201890641 wait complete. 2023-10-16T14:07:26.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:07:26.320 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:26.317+0000 7fecc8c12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:07:26.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:07:26.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:07:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.497+0000 7fb2230c9640 1 Processor -- start 2023-10-16T14:07:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.498+0000 7fb2230c9640 1 -- start start 2023-10-16T14:07:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.499+0000 7fb2230c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 0x7fb21c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.499+0000 7fb2230c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c06a6f0 con 0x7fb21c15d150 2023-10-16T14:07:26.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.499+0000 7fb2230c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c0ac890 con 0x7fb21c06a2e0 2023-10-16T14:07:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.499+0000 7fb2230c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c15d530 con 0x7fb21c0ad350 2023-10-16T14:07:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.500+0000 7fb220e65640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb21c15d150 0x7fb21c160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49414/0 (socket says 172.21.15.60:49414) 2023-10-16T14:07:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.500+0000 7fb220e65640 1 -- 172.21.15.60:0/3141313794 learned_addr learned my addr 172.21.15.60:0/3141313794 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.500+0000 7fb21bfff640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 0x7fb21c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.500+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486342654 0 0) 0x7fb21c06a6f0 con 0x7fb21c15d150 2023-10-16T14:07:26.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.500+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb20c003520 con 0x7fb21c15d150 2023-10-16T14:07:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1718112053 0 0) 0x7fb20c003520 con 0x7fb21c15d150 2023-10-16T14:07:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 msgr2=0x7fb21c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 0x7fb21c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 >> v1:172.21.15.144:6789/0 conn(0x7fb21c06a2e0 legacy=0x7fb21c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c161650 con 0x7fb21c15d150 2023-10-16T14:07:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb210001670 con 0x7fb21c15d150 2023-10-16T14:07:26.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 >> v1:172.21.15.60:6789/0 conn(0x7fb21c15d150 legacy=0x7fb21c160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb20c0039a0 0x7fb20c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c004390 0x7fb20c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 0x7fb20c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c15d530 con 0x7fb20c0039a0 2023-10-16T14:07:26.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c00b9b0 con 0x7fb20c004390 2023-10-16T14:07:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c00bb70 con 0x7fb21c0ad350 2023-10-16T14:07:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.501+0000 7fb220e65640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb20c0039a0 0x7fb20c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb21bfff640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c004390 0x7fb20c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb221666640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 0x7fb20c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb220e65640 1 -- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 msgr2=0x7fb20c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb220e65640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0ad350 0x7fb20c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb220e65640 1 -- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c004390 msgr2=0x7fb20c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb220e65640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c004390 0x7fb20c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.502+0000 7fb220e65640 1 -- 172.21.15.60:0/3141313794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c161650 con 0x7fb20c0039a0 2023-10-16T14:07:26.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.503+0000 7fb220e65640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb20c0039a0 0x7fb20c003dc0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fb21000c340 tx=0x7fb20c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14dcb3d5f7d2d4e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.503+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21002cbb0 con 0x7fb20c0039a0 2023-10-16T14:07:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.503+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21002cd10 con 0x7fb20c0039a0 2023-10-16T14:07:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.503+0000 7fb21b7fe640 1 -- 172.21.15.60:0/3141313794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210005b10 con 0x7fb20c0039a0 2023-10-16T14:07:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb2230c9640 1 -- 172.21.15.60:0/3141313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb20c0039a0 msgr2=0x7fb20c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb2230c9640 1 --2- 172.21.15.60:0/3141313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb20c0039a0 0x7fb20c003dc0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fb21000c340 tx=0x7fb20c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb221666640 1 -- 172.21.15.60:0/3141313794 reap_dead start 2023-10-16T14:07:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb2230c9640 1 -- 172.21.15.60:0/3141313794 shutdown_connections 2023-10-16T14:07:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb2230c9640 1 -- 172.21.15.60:0/3141313794 >> 172.21.15.60:0/3141313794 conn(0x7fb21c0ce9b0 msgr2=0x7fb21c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb2230c9640 1 -- 172.21.15.60:0/3141313794 shutdown_connections 2023-10-16T14:07:26.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.504+0000 7fb2230c9640 1 -- 172.21.15.60:0/3141313794 wait complete. 2023-10-16T14:07:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.506+0000 7fb2230c9640 1 Processor -- start 2023-10-16T14:07:26.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.506+0000 7fb2230c9640 1 -- start start 2023-10-16T14:07:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb2230c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c0039a0 0x7fb21c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb2230c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 0x7fb21c1f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb2230c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0c4200 0x7fb21c070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb2230c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c1f6ee0 con 0x7fb21c0c4c40 2023-10-16T14:07:26.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb2230c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c1f38b0 con 0x7fb20c0039a0 2023-10-16T14:07:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb2230c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c1f2ca0 con 0x7fb21c0c4200 2023-10-16T14:07:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 0x7fb21c1f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 0x7fb21c1f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48722/0 (socket says 172.21.15.60:48722) 2023-10-16T14:07:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.507+0000 7fb21bfff640 1 -- 172.21.15.60:0/829421887 learned_addr learned my addr 172.21.15.60:0/829421887 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb220e65640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c0039a0 0x7fb21c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb21bfff640 1 -- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0c4200 msgr2=0x7fb21c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb21bfff640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0c4200 0x7fb21c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb21bfff640 1 -- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c0039a0 msgr2=0x7fb21c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb21bfff640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c0039a0 0x7fb21c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb21bfff640 1 -- 172.21.15.60:0/829421887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c1f2f80 con 0x7fb21c0c4c40 2023-10-16T14:07:26.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb21bfff640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 0x7fb21c1f69a0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fb208011600 tx=0x7fb208011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:26.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.508+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20800f920 con 0x7fb21c0c4c40 2023-10-16T14:07:26.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.509+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20800fa80 con 0x7fb21c0c4c40 2023-10-16T14:07:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.509+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20800fdf0 con 0x7fb21c0c4c40 2023-10-16T14:07:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.509+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c1f3260 con 0x7fb21c0c4c40 2023-10-16T14:07:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.509+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c1f29a0 con 0x7fb21c0c4c40 2023-10-16T14:07:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.510+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c04e160 con 0x7fb21c0c4c40 2023-10-16T14:07:26.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.511+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 112) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb208015460 con 0x7fb21c0c4c40 2023-10-16T14:07:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.512+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7fb208089a20 con 0x7fb21c0c4c40 2023-10-16T14:07:26.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.516+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb208054a00 con 0x7fb21c0c4c40 2023-10-16T14:07:26.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.715+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb21c0c8d70 con 0x7fb21c0c4c40 2023-10-16T14:07:26.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.716+0000 7fb2197fa640 1 -- 172.21.15.60:0/829421887 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v202) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb208058020 con 0x7fb21c0c4c40 2023-10-16T14:07:26.720 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:07:26.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 msgr2=0x7fb21c1f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 0x7fb21c1f69a0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fb208011600 tx=0x7fb208011ac0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 shutdown_connections 2023-10-16T14:07:26.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb21c0c4200 0x7fb21c070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb21c0c4c40 0x7fb21c1f69a0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 --2- 172.21.15.60:0/829421887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb20c0039a0 0x7fb21c0c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.718+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 >> 172.21.15.60:0/829421887 conn(0x7fb21c0ce9b0 msgr2=0x7fb21c158bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:26.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.719+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 shutdown_connections 2023-10-16T14:07:26.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.719+0000 7fb2230c9640 1 -- 172.21.15.60:0/829421887 wait complete. 2023-10-16T14:07:26.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:07:26.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.964+0000 7fd39f085640 1 Processor -- start 2023-10-16T14:07:26.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.964+0000 7fd39f085640 1 -- start start 2023-10-16T14:07:26.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39f085640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd39815cd70 0x7fd3980d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39f085640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3980cded0 0x7fd3980b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39f085640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39815d150 con 0x7fd3980b8480 2023-10-16T14:07:26.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39f085640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3980d1180 con 0x7fd3980cded0 2023-10-16T14:07:26.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39f085640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3980d1460 con 0x7fd39815cd70 2023-10-16T14:07:26.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39d882640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd3980b8480 0x7fd3980cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49426/0 (socket says 172.21.15.60:49426) 2023-10-16T14:07:26.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39d882640 1 -- 172.21.15.60:0/3599366052 learned_addr learned my addr 172.21.15.60:0/3599366052 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.966+0000 7fd39e884640 1 --2- 172.21.15.60:0/3599366052 >> v2:172.21.15.60:3300/0 conn(0x7fd3980cded0 0x7fd3980b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.967+0000 7fd39e083640 1 --2- 172.21.15.60:0/3599366052 >> v2:172.21.15.144:3300/0 conn(0x7fd39815cd70 0x7fd3980d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.967+0000 7fd39e884640 1 -- 172.21.15.60:0/3599366052 >> v2:172.21.15.144:3300/0 conn(0x7fd39815cd70 msgr2=0x7fd3980d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.967+0000 7fd39e884640 1 --2- 172.21.15.60:0/3599366052 >> v2:172.21.15.144:3300/0 conn(0x7fd39815cd70 0x7fd3980d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.967+0000 7fd39e884640 1 -- 172.21.15.60:0/3599366052 >> v1:172.21.15.60:6789/0 conn(0x7fd3980b8480 legacy=0x7fd3980cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.967+0000 7fd39d081640 1 -- 172.21.15.60:0/3599366052 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384495393 0 0) 0x7fd39815d150 con 0x7fd3980b8480 2023-10-16T14:07:26.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.967+0000 7fd39e884640 1 -- 172.21.15.60:0/3599366052 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980d1740 con 0x7fd3980cded0 2023-10-16T14:07:26.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.968+0000 7fd39e884640 1 --2- 172.21.15.60:0/3599366052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980b5c90 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fd3980b56a0 tx=0x7fd39002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387eb808acce8ada server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:26.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.968+0000 7fd39d081640 1 -- 172.21.15.60:0/3599366052 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39002cc80 con 0x7fd3980cded0 2023-10-16T14:07:26.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.968+0000 7fd39d081640 1 -- 172.21.15.60:0/3599366052 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39002cde0 con 0x7fd3980cded0 2023-10-16T14:07:26.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.968+0000 7fd39d081640 1 -- 172.21.15.60:0/3599366052 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390034660 con 0x7fd3980cded0 2023-10-16T14:07:26.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 -- 172.21.15.60:0/3599366052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 msgr2=0x7fd3980b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 --2- 172.21.15.60:0/3599366052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980b5c90 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fd3980b56a0 tx=0x7fd39002bed0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 -- 172.21.15.60:0/3599366052 shutdown_connections 2023-10-16T14:07:26.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 --2- 172.21.15.60:0/3599366052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980b5c90 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 --2- 172.21.15.60:0/3599366052 >> v2:172.21.15.144:3300/0 conn(0x7fd39815cd70 0x7fd3980d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 -- 172.21.15.60:0/3599366052 >> 172.21.15.60:0/3599366052 conn(0x7fd3980cf570 msgr2=0x7fd3980cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 -- 172.21.15.60:0/3599366052 shutdown_connections 2023-10-16T14:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.969+0000 7fd39f085640 1 -- 172.21.15.60:0/3599366052 wait complete. 2023-10-16T14:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.971+0000 7fd39f085640 1 Processor -- start 2023-10-16T14:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.971+0000 7fd39f085640 1 -- start start 2023-10-16T14:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.972+0000 7fd39f085640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3980b8480 0x7fd39815a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.972+0000 7fd39f085640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.972+0000 7fd39f085640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd39815cd70 0x7fd3980cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.972+0000 7fd39f085640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd398151400 con 0x7fd3980cded0 2023-10-16T14:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.972+0000 7fd39f085640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39814fb50 con 0x7fd3980b8480 2023-10-16T14:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.972+0000 7fd39f085640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd398150110 con 0x7fd39815cd70 2023-10-16T14:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48740/0 (socket says 172.21.15.60:48740) 2023-10-16T14:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 -- 172.21.15.60:0/2903528581 learned_addr learned my addr 172.21.15.60:0/2903528581 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 -- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd39815cd70 msgr2=0x7fd3980cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd39815cd70 0x7fd3980cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 -- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3980b8480 msgr2=0x7fd39815a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39e083640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3980b8480 0x7fd39815a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3980b8480 0x7fd39815a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 -- 172.21.15.60:0/2903528581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390009a70 con 0x7fd3980cded0 2023-10-16T14:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.973+0000 7fd39d882640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980cbda0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c500 tx=0x7fd38800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.974+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388007560 con 0x7fd3980cded0 2023-10-16T14:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.974+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3880047e0 con 0x7fd3980cded0 2023-10-16T14:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.974+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388004b90 con 0x7fd3980cded0 2023-10-16T14:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.974+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39815e1c0 con 0x7fd3980cded0 2023-10-16T14:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.974+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39814fda0 con 0x7fd3980cded0 2023-10-16T14:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.975+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3981502a0 con 0x7fd3980cded0 2023-10-16T14:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.976+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 112) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd388004cf0 con 0x7fd3980cded0 2023-10-16T14:07:26.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.977+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7fd388088780 con 0x7fd3980cded0 2023-10-16T14:07:26.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:26.981+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd388053760 con 0x7fd3980cded0 2023-10-16T14:07:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.099+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 113) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd388053210 con 0x7fd3980cded0 2023-10-16T14:07:27.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.099+0000 7fd38effd640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd37c070870 0x7fd37c072c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.101+0000 7fd39e083640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd37c070870 0x7fd37c072c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.110+0000 7fd39e083640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd37c070870 0x7fd37c072c60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd394006fd0 tx=0x7fd394008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.185+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd39804e9b0 con 0x7fd3980cded0 2023-10-16T14:07:27.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.186+0000 7fd38effd640 1 -- 172.21.15.60:0/2903528581 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v202) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3880180a0 con 0x7fd3980cded0 2023-10-16T14:07:27.190 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:07:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd37c070870 msgr2=0x7fd37c072c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd37c070870 0x7fd37c072c60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd394006fd0 tx=0x7fd394008040 comp rx=0 tx=0).stop 2023-10-16T14:07:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 msgr2=0x7fd3980cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980cbda0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c500 tx=0x7fd38800c9c0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 shutdown_connections 2023-10-16T14:07:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd39815cd70 0x7fd3980cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd37c070870 0x7fd37c072c60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980cded0 0x7fd3980cbda0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 --2- 172.21.15.60:0/2903528581 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3980b8480 0x7fd39815a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 >> 172.21.15.60:0/2903528581 conn(0x7fd3980cf570 msgr2=0x7fd3980d3310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 shutdown_connections 2023-10-16T14:07:27.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.189+0000 7fd39f085640 1 -- 172.21.15.60:0/2903528581 wait complete. 2023-10-16T14:07:27.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:27.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.437+0000 7f4cb1fa7640 1 Processor -- start 2023-10-16T14:07:27.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.437+0000 7f4cb1fa7640 1 -- start start 2023-10-16T14:07:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.438+0000 7f4cb1fa7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4cac0ad970 0x7f4cac0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cb1fa7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac15b9d0 con 0x7f4cac06c040 2023-10-16T14:07:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cb1fa7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac15bcb0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cb1fa7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac15bf90 con 0x7f4cac0ad970 2023-10-16T14:07:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cb0fa5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4cac06c040 0x7f4cac06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49440/0 (socket says 172.21.15.60:49440) 2023-10-16T14:07:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cb0fa5640 1 -- 172.21.15.60:0/1879389566 learned_addr learned my addr 172.21.15.60:0/1879389566 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:27.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cabfff640 1 --2- 172.21.15.60:0/1879389566 >> v2:172.21.15.60:3300/0 conn(0x7f4cac0ad970 0x7f4cac0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cab7fe640 1 -- 172.21.15.60:0/1879389566 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321880587 0 0) 0x7f4cac15b9d0 con 0x7f4cac06c040 2023-10-16T14:07:27.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cabfff640 1 -- 172.21.15.60:0/1879389566 >> v1:172.21.15.144:6789/0 conn(0x7f4cac0ccfb0 legacy=0x7f4cac0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:27.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cabfff640 1 -- 172.21.15.60:0/1879389566 >> v1:172.21.15.60:6789/0 conn(0x7f4cac06c040 legacy=0x7f4cac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.439+0000 7f4cabfff640 1 -- 172.21.15.60:0/1879389566 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0cc270 con 0x7f4cac0ad970 2023-10-16T14:07:27.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.441+0000 7f4cabfff640 1 --2- 172.21.15.60:0/1879389566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ad970 0x7f4cac0cca70 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f4c940097d0 tx=0x7f4c9402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870b2b42405f9d24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.441+0000 7f4cab7fe640 1 -- 172.21.15.60:0/1879389566 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9402d040 con 0x7f4cac0ad970 2023-10-16T14:07:27.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.441+0000 7f4cab7fe640 1 -- 172.21.15.60:0/1879389566 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c94031810 con 0x7f4cac0ad970 2023-10-16T14:07:27.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.441+0000 7f4cab7fe640 1 -- 172.21.15.60:0/1879389566 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94031ba0 con 0x7f4cac0ad970 2023-10-16T14:07:27.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.441+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/1879389566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ad970 msgr2=0x7f4cac0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.441+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/1879389566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ad970 0x7f4cac0cca70 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f4c940097d0 tx=0x7f4c9402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.442+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/1879389566 shutdown_connections 2023-10-16T14:07:27.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.442+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/1879389566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ad970 0x7f4cac0cca70 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.442+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/1879389566 >> 172.21.15.60:0/1879389566 conn(0x7f4cac0ce6a0 msgr2=0x7f4cac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:27.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.442+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/1879389566 shutdown_connections 2023-10-16T14:07:27.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.442+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/1879389566 wait complete. 2023-10-16T14:07:27.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.443+0000 7f4cb1fa7640 1 Processor -- start 2023-10-16T14:07:27.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.444+0000 7f4cb1fa7640 1 -- start start 2023-10-16T14:07:27.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.444+0000 7f4cb1fa7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cac06c040 0x7f4cac0ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb1fa7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cac0ad970 0x7f4cac1f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb1fa7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 0x7f4cac1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb1fa7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0af680 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb1fa7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0aea00 con 0x7f4cac06c040 2023-10-16T14:07:27.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb1fa7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0aece0 con 0x7f4cac0ad970 2023-10-16T14:07:27.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 0x7f4cac1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 0x7f4cac1f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48748/0 (socket says 172.21.15.60:48748) 2023-10-16T14:07:27.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 -- 172.21.15.60:0/2378482271 learned_addr learned my addr 172.21.15.60:0/2378482271 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:27.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb0fa5640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cac06c040 0x7f4cac0ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cabfff640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cac0ad970 0x7f4cac1f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 -- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cac0ad970 msgr2=0x7f4cac1f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cac0ad970 0x7f4cac1f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 -- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cac06c040 msgr2=0x7f4cac0ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cac06c040 0x7f4cac0ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.445+0000 7f4cb17a6640 1 -- 172.21.15.60:0/2378482271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c94009400 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.446+0000 7f4cb17a6640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 0x7f4cac1f3d00 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c410 tx=0x7f4ca400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.446+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4011790 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.446+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca40118f0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.446+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4011ce0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.446+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac0aefc0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.446+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac0ae5f0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.448+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac06b330 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.448+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 113) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4ca401c460 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.449+0000 7f4ca97fa640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4c8c06acc0 0x7f4c8c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.450+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7f4ca4089330 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.450+0000 7f4cb0fa5640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4c8c06acc0 0x7f4c8c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.451+0000 7f4cb0fa5640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4c8c06acc0 0x7f4c8c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c007ed0 tx=0x7f4c9c007e60 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.454+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ca40542f0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.693+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cac0af7c0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.695+0000 7f4ca97fa640 1 -- 172.21.15.60:0/2378482271 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f4cac0af7c0 con 0x7f4cac0ccfb0 2023-10-16T14:07:27.699 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:27.699 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:51:27.881990+0000","modified":"2023-10-16T13:51:30.974922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9966":{"gid":9966,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/3850041246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3850041246},{"type":"v1","addr":"172.21.15.60:6835","nonce":3850041246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T14:07:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 95 2023-10-16T14:07:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4c8c06acc0 msgr2=0x7f4c8c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4c8c06acc0 0x7f4c8c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c007ed0 tx=0x7f4c9c007e60 comp rx=0 tx=0).stop 2023-10-16T14:07:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 msgr2=0x7f4cac1f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 0x7f4cac1f3d00 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c410 tx=0x7f4ca400c8d0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 shutdown_connections 2023-10-16T14:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cac0ad970 0x7f4cac1f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4c8c06acc0 0x7f4c8c06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cac0ccfb0 0x7f4cac1f3d00 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 --2- 172.21.15.60:0/2378482271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cac06c040 0x7f4cac0ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 >> 172.21.15.60:0/2378482271 conn(0x7f4cac0ce6a0 msgr2=0x7f4cac06a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 shutdown_connections 2023-10-16T14:07:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.698+0000 7f4cb1fa7640 1 -- 172.21.15.60:0/2378482271 wait complete. 2023-10-16T14:07:27.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.938+0000 7f38ddd21640 1 Processor -- start 2023-10-16T14:07:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.938+0000 7f38ddd21640 1 -- start start 2023-10-16T14:07:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38ddd21640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f38d815dde0 0x7f38d80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38ddd21640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38d8070f20 0x7f38d80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38ddd21640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8160a90 con 0x7f38d815c9f0 2023-10-16T14:07:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38ddd21640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d8071350 con 0x7f38d8070f20 2023-10-16T14:07:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38ddd21640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d80cdec0 con 0x7f38d815dde0 2023-10-16T14:07:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38dd520640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38d8070f20 0x7f38d80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38dcd1f640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f38d815c9f0 0x7f38d80c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49446/0 (socket says 172.21.15.60:49446) 2023-10-16T14:07:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.940+0000 7f38dcd1f640 1 -- 172.21.15.60:0/180421746 learned_addr learned my addr 172.21.15.60:0/180421746 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.941+0000 7f38d7fff640 1 --2- 172.21.15.60:0/180421746 >> v2:172.21.15.144:3300/0 conn(0x7f38d815dde0 0x7f38d80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.941+0000 7f38dd520640 1 -- 172.21.15.60:0/180421746 >> v2:172.21.15.144:3300/0 conn(0x7f38d815dde0 msgr2=0x7f38d80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:27.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.941+0000 7f38dd520640 1 --2- 172.21.15.60:0/180421746 >> v2:172.21.15.144:3300/0 conn(0x7f38d815dde0 0x7f38d80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.941+0000 7f38dd520640 1 -- 172.21.15.60:0/180421746 >> v1:172.21.15.60:6789/0 conn(0x7f38d815c9f0 legacy=0x7f38d80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.941+0000 7f38dd520640 1 -- 172.21.15.60:0/180421746 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d80cd900 con 0x7f38d8070f20 2023-10-16T14:07:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.942+0000 7f38dd520640 1 --2- 172.21.15.60:0/180421746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d8070f20 0x7f38d80ac890 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f38d80b1260 tx=0x7f38d002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3876960301461af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.942+0000 7f38d77fe640 1 -- 172.21.15.60:0/180421746 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d002cc70 con 0x7f38d8070f20 2023-10-16T14:07:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38d77fe640 1 -- 172.21.15.60:0/180421746 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d002cdd0 con 0x7f38d8070f20 2023-10-16T14:07:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38d77fe640 1 -- 172.21.15.60:0/180421746 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0034660 con 0x7f38d8070f20 2023-10-16T14:07:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 -- 172.21.15.60:0/180421746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d8070f20 msgr2=0x7f38d80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 --2- 172.21.15.60:0/180421746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d8070f20 0x7f38d80ac890 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f38d80b1260 tx=0x7f38d002bec0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 -- 172.21.15.60:0/180421746 shutdown_connections 2023-10-16T14:07:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 --2- 172.21.15.60:0/180421746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d8070f20 0x7f38d80ac890 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 --2- 172.21.15.60:0/180421746 >> v2:172.21.15.144:3300/0 conn(0x7f38d815dde0 0x7f38d80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 -- 172.21.15.60:0/180421746 >> 172.21.15.60:0/180421746 conn(0x7f38d80ce6a0 msgr2=0x7f38d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 -- 172.21.15.60:0/180421746 shutdown_connections 2023-10-16T14:07:27.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.943+0000 7f38ddd21640 1 -- 172.21.15.60:0/180421746 wait complete. 2023-10-16T14:07:27.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.945+0000 7f38ddd21640 1 Processor -- start 2023-10-16T14:07:27.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.945+0000 7f38ddd21640 1 -- start start 2023-10-16T14:07:27.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38ddd21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8070f20 0x7f38d8151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38ddd21640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 0x7f38d806b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38ddd21640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d815dde0 0x7f38d815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38ddd21640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d80c3570 con 0x7f38d815c9f0 2023-10-16T14:07:27.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38ddd21640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d80c3e10 con 0x7f38d815dde0 2023-10-16T14:07:27.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38ddd21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d80c2fb0 con 0x7f38d8070f20 2023-10-16T14:07:27.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 0x7f38d806b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 0x7f38d806b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48768/0 (socket says 172.21.15.60:48768) 2023-10-16T14:07:27.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.946+0000 7f38d7fff640 1 -- 172.21.15.60:0/3686955730 learned_addr learned my addr 172.21.15.60:0/3686955730 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:27.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d7fff640 1 -- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8070f20 msgr2=0x7f38d8151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d7fff640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8070f20 0x7f38d8151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d7fff640 1 -- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d815dde0 msgr2=0x7f38d815f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:27.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d7fff640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d815dde0 0x7f38d815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:27.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d7fff640 1 -- 172.21.15.60:0/3686955730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d0009a70 con 0x7f38d815c9f0 2023-10-16T14:07:27.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d7fff640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 0x7f38d806b110 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f38c400d620 tx=0x7f38c400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.947+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c400b620 con 0x7f38d815c9f0 2023-10-16T14:07:27.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.948+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c400b780 con 0x7f38d815c9f0 2023-10-16T14:07:27.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.948+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c400bb30 con 0x7f38d815c9f0 2023-10-16T14:07:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.948+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d80c37c0 con 0x7f38d815c9f0 2023-10-16T14:07:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.948+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d806b650 con 0x7f38d815c9f0 2023-10-16T14:07:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.949+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d8160d70 con 0x7f38d815c9f0 2023-10-16T14:07:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.950+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 113) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f38c400bc90 con 0x7f38d815c9f0 2023-10-16T14:07:27.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.951+0000 7f38d57fa640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f38bc06acc0 0x7f38bc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:27.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.951+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7f38c4089b10 con 0x7f38d815c9f0 2023-10-16T14:07:27.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.951+0000 7f38dcd1f640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f38bc06acc0 0x7f38bc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:27.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.952+0000 7f38dcd1f640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f38bc06acc0 0x7f38bc06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f38d80cae10 tx=0x7f38cc007a40 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:27.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:27.955+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38c4054ad0 con 0x7f38d815c9f0 2023-10-16T14:07:28.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.117+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 114) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f38c4054ea0 con 0x7f38d815c9f0 2023-10-16T14:07:28.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.156+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38d80c3f50 con 0x7f38d815c9f0 2023-10-16T14:07:28.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.160+0000 7f38d57fa640 1 -- 172.21.15.60:0/3686955730 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+18150 (secure 0 0 0) 0x7f38d80c3f50 con 0x7f38d815c9f0 2023-10-16T14:07:28.164 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:28.164 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":202,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:26.077964+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3534900853":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/933999404":"2023-10-17T14:07:26.077866+0000","172.21.15.144:6833/345630502":"2023-10-17T14:07:26.077866+0000","172.21.15.144:6832/345630502":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1247236577":"2023-10-17T14:02:21.839066+0000","172.21.15.144:6833/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/483866973":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1895637100":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1129229162":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3061731632":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1482302803":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1876059409":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1022250268":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/103168750":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6833/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3241678538":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/2474164444":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1144561621":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3710907701":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1287992351":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3658895152":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1768527142":"2023-10-17T13:51:42.544813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.162+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f38bc06acc0 msgr2=0x7f38bc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.162+0000 7f38ddd21640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f38bc06acc0 0x7f38bc06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f38d80cae10 tx=0x7f38cc007a40 comp rx=0 tx=0).stop 2023-10-16T14:07:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.162+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 msgr2=0x7f38d806b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.162+0000 7f38ddd21640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 0x7f38d806b110 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f38c400d620 tx=0x7f38c400dae0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 shutdown_connections 2023-10-16T14:07:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f38bc06acc0 0x7f38bc06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38d815dde0 0x7f38d815f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38d815c9f0 0x7f38d806b110 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 --2- 172.21.15.60:0/3686955730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f38d8070f20 0x7f38d8151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 >> 172.21.15.60:0/3686955730 conn(0x7f38d80ce6a0 msgr2=0x7f38d80af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 shutdown_connections 2023-10-16T14:07:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.163+0000 7f38ddd21640 1 -- 172.21.15.60:0/3686955730 wait complete. 2023-10-16T14:07:28.186 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:07:28.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:28.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.404+0000 7fca99bc4640 1 Processor -- start 2023-10-16T14:07:28.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.405+0000 7fca99bc4640 1 -- start start 2023-10-16T14:07:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.406+0000 7fca99bc4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fca9415dde0 0x7fca940709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.406+0000 7fca99bc4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca94160a90 con 0x7fca94070f20 2023-10-16T14:07:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.406+0000 7fca99bc4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca940cd900 con 0x7fca9415c9f0 2023-10-16T14:07:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.406+0000 7fca99bc4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca94071350 con 0x7fca9415dde0 2023-10-16T14:07:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.407+0000 7fca993c3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fca94070f20 0x7fca940ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49452/0 (socket says 172.21.15.60:49452) 2023-10-16T14:07:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.407+0000 7fca993c3640 1 -- 172.21.15.60:0/2193954600 learned_addr learned my addr 172.21.15.60:0/2193954600 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.407+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334399695 0 0) 0x7fca94160a90 con 0x7fca94070f20 2023-10-16T14:07:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.407+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca84003520 con 0x7fca94070f20 2023-10-16T14:07:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.407+0000 7fca93fff640 1 --2- 172.21.15.60:0/2193954600 >> v2:172.21.15.144:3300/0 conn(0x7fca9415dde0 0x7fca940709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.407+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4069261955 0 0) 0x7fca84003520 con 0x7fca94070f20 2023-10-16T14:07:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 >> v2:172.21.15.144:3300/0 conn(0x7fca9415dde0 msgr2=0x7fca940709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 --2- 172.21.15.60:0/2193954600 >> v2:172.21.15.144:3300/0 conn(0x7fca9415dde0 0x7fca940709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 >> v1:172.21.15.144:6789/0 conn(0x7fca9415c9f0 legacy=0x7fca940c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940cdbe0 con 0x7fca94070f20 2023-10-16T14:07:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fca8c0019f0 con 0x7fca94070f20 2023-10-16T14:07:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 >> v1:172.21.15.60:6789/0 conn(0x7fca94070f20 legacy=0x7fca940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84003970 0x7fca84003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca94071350 con 0x7fca84005180 2023-10-16T14:07:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8400c0b0 con 0x7fca840047c0 2023-10-16T14:07:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8400c270 con 0x7fca84003970 2023-10-16T14:07:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca993c3640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca98bc2640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84003970 0x7fca84003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca93fff640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca993c3640 1 -- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84003970 msgr2=0x7fca84003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca993c3640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84003970 0x7fca84003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.408+0000 7fca993c3640 1 -- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca840047c0 msgr2=0x7fca84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.409+0000 7fca993c3640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.409+0000 7fca993c3640 1 -- 172.21.15.60:0/2193954600 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84003520 con 0x7fca84005180 2023-10-16T14:07:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.409+0000 7fca98bc2640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca84003970 0x7fca84003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.409+0000 7fca93fff640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.409+0000 7fca993c3640 1 -- 172.21.15.60:0/2193954600 reap_dead start 2023-10-16T14:07:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.410+0000 7fca993c3640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca8400b980 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fca8c002230 tx=0x7fca840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cd2f0cb463ff59 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.410+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c034630 con 0x7fca84005180 2023-10-16T14:07:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.410+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c02f070 con 0x7fca84005180 2023-10-16T14:07:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.410+0000 7fca937fe640 1 -- 172.21.15.60:0/2193954600 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0349f0 con 0x7fca84005180 2023-10-16T14:07:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 -- 172.21.15.60:0/2193954600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 msgr2=0x7fca8400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca8400b980 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fca8c002230 tx=0x7fca840057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 -- 172.21.15.60:0/2193954600 shutdown_connections 2023-10-16T14:07:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 --2- 172.21.15.60:0/2193954600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca8400b980 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 -- 172.21.15.60:0/2193954600 >> 172.21.15.60:0/2193954600 conn(0x7fca940ce6a0 msgr2=0x7fca940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:28.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 -- 172.21.15.60:0/2193954600 shutdown_connections 2023-10-16T14:07:28.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.411+0000 7fca99bc4640 1 -- 172.21.15.60:0/2193954600 wait complete. 2023-10-16T14:07:28.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.413+0000 7fca99bc4640 1 Processor -- start 2023-10-16T14:07:28.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.413+0000 7fca99bc4640 1 -- start start 2023-10-16T14:07:28.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca99bc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca94069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca99bc4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca94157d70 0x7fca9414a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca99bc4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca941471e0 0x7fca940c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca99bc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9406a2e0 con 0x7fca84005180 2023-10-16T14:07:28.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca99bc4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9414abf0 con 0x7fca941471e0 2023-10-16T14:07:28.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca99bc4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca94147db0 con 0x7fca94157d70 2023-10-16T14:07:28.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca98bc2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca94069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca98bc2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca94069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48784/0 (socket says 172.21.15.60:48784) 2023-10-16T14:07:28.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.415+0000 7fca98bc2640 1 -- 172.21.15.60:0/1843017777 learned_addr learned my addr 172.21.15.60:0/1843017777 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:28.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca98bc2640 1 -- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca94157d70 msgr2=0x7fca9414a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca98bc2640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca94157d70 0x7fca9414a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca98bc2640 1 -- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca941471e0 msgr2=0x7fca940c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca98bc2640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca941471e0 0x7fca940c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca98bc2640 1 -- 172.21.15.60:0/1843017777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94148090 con 0x7fca84005180 2023-10-16T14:07:28.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca98bc2640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca94069da0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fca8800d5c0 tx=0x7fca8800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:28.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88007220 con 0x7fca84005180 2023-10-16T14:07:28.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca880073a0 con 0x7fca84005180 2023-10-16T14:07:28.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca880067d0 con 0x7fca84005180 2023-10-16T14:07:28.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca94148370 con 0x7fca84005180 2023-10-16T14:07:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.416+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca94148930 con 0x7fca84005180 2023-10-16T14:07:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.417+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca94148b50 con 0x7fca84005180 2023-10-16T14:07:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.419+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 114) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fca88006930 con 0x7fca84005180 2023-10-16T14:07:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.420+0000 7fca917fa640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fca7806af80 0x7fca7806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.420+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7fca88089870 con 0x7fca84005180 2023-10-16T14:07:28.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.420+0000 7fca93fff640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fca7806af80 0x7fca7806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.421+0000 7fca93fff640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fca7806af80 0x7fca7806d440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca9415b040 tx=0x7fca80009350 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:28.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.425+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca88054370 con 0x7fca84005180 2023-10-16T14:07:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.664+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca9404e9b0 con 0x7fca84005180 2023-10-16T14:07:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.665+0000 7fca917fa640 1 -- 172.21.15.60:0/1843017777 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fca88057990 con 0x7fca84005180 2023-10-16T14:07:28.669 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:28.670 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:51:27.881990+0000","modified":"2023-10-16T13:51:30.974922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9966":{"gid":9966,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/3850041246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3850041246},{"type":"v1","addr":"172.21.15.60:6835","nonce":3850041246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T14:07:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 95 2023-10-16T14:07:28.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.668+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fca7806af80 msgr2=0x7fca7806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.668+0000 7fca99bc4640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fca7806af80 0x7fca7806d440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca9415b040 tx=0x7fca80009350 comp rx=0 tx=0).stop 2023-10-16T14:07:28.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.668+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 msgr2=0x7fca94069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.668+0000 7fca99bc4640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca94069da0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fca8800d5c0 tx=0x7fca8800da80 comp rx=0 tx=0).stop 2023-10-16T14:07:28.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 shutdown_connections 2023-10-16T14:07:28.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fca94157d70 0x7fca9414a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fca7806af80 0x7fca7806d440 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fca941471e0 0x7fca940c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 --2- 172.21.15.60:0/1843017777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca84005180 0x7fca94069da0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 >> 172.21.15.60:0/1843017777 conn(0x7fca940ce6a0 msgr2=0x7fca9415b870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:28.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 shutdown_connections 2023-10-16T14:07:28.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.669+0000 7fca99bc4640 1 -- 172.21.15.60:0/1843017777 wait complete. 2023-10-16T14:07:28.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.910+0000 7fc9aefef640 1 Processor -- start 2023-10-16T14:07:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.911+0000 7fc9aefef640 1 -- start start 2023-10-16T14:07:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.912+0000 7fc9aefef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a815aa60 0x7fc9a80af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.912+0000 7fc9aefef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc9a80cc9f0 0x7fc9a80b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.912+0000 7fc9aefef640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80b3730 con 0x7fc9a80afd60 2023-10-16T14:07:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.912+0000 7fc9aefef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a80b3980 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.912+0000 7fc9aefef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80b3c60 con 0x7fc9a815aa60 2023-10-16T14:07:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.913+0000 7fc9ad7ec640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc9a80afd60 0x7fc9a80cc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56084/0 (socket says 172.21.15.60:56084) 2023-10-16T14:07:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.913+0000 7fc9ad7ec640 1 -- 172.21.15.60:0/3223029435 learned_addr learned my addr 172.21.15.60:0/3223029435 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.913+0000 7fc9adfed640 1 --2- 172.21.15.60:0/3223029435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a815aa60 0x7fc9a80af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.913+0000 7fc9ae7ee640 1 --2- 172.21.15.60:0/3223029435 >> v2:172.21.15.144:3300/0 conn(0x7fc9a80cc9f0 0x7fc9a80b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.913+0000 7fc9acfeb640 1 -- 172.21.15.60:0/3223029435 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499295026 0 0) 0x7fc9a80b3730 con 0x7fc9a80afd60 2023-10-16T14:07:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.913+0000 7fc9acfeb640 1 -- 172.21.15.60:0/3223029435 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc994003520 con 0x7fc9a80afd60 2023-10-16T14:07:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.914+0000 7fc9adfed640 1 -- 172.21.15.60:0/3223029435 >> v2:172.21.15.144:3300/0 conn(0x7fc9a80cc9f0 msgr2=0x7fc9a80b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.914+0000 7fc9adfed640 1 --2- 172.21.15.60:0/3223029435 >> v2:172.21.15.144:3300/0 conn(0x7fc9a80cc9f0 0x7fc9a80b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.914+0000 7fc9adfed640 1 -- 172.21.15.60:0/3223029435 >> v1:172.21.15.60:6789/0 conn(0x7fc9a80afd60 legacy=0x7fc9a80cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.914+0000 7fc9adfed640 1 -- 172.21.15.60:0/3223029435 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80b3f40 con 0x7fc9a815aa60 2023-10-16T14:07:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.915+0000 7fc9adfed640 1 --2- 172.21.15.60:0/3223029435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a815aa60 0x7fc9a80af820 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40097a0 tx=0x7fc9a402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72083a792c78df3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.916+0000 7fc9acfeb640 1 -- 172.21.15.60:0/3223029435 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4031780 con 0x7fc9a815aa60 2023-10-16T14:07:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.916+0000 7fc9acfeb640 1 -- 172.21.15.60:0/3223029435 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a40318e0 con 0x7fc9a815aa60 2023-10-16T14:07:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.916+0000 7fc9acfeb640 1 -- 172.21.15.60:0/3223029435 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4031c70 con 0x7fc9a815aa60 2023-10-16T14:07:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 -- 172.21.15.60:0/3223029435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a815aa60 msgr2=0x7fc9a80af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 --2- 172.21.15.60:0/3223029435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a815aa60 0x7fc9a80af820 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40097a0 tx=0x7fc9a402b820 comp rx=0 tx=0).stop 2023-10-16T14:07:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 -- 172.21.15.60:0/3223029435 shutdown_connections 2023-10-16T14:07:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 --2- 172.21.15.60:0/3223029435 >> v2:172.21.15.144:3300/0 conn(0x7fc9a80cc9f0 0x7fc9a80b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 --2- 172.21.15.60:0/3223029435 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a815aa60 0x7fc9a80af820 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 -- 172.21.15.60:0/3223029435 >> 172.21.15.60:0/3223029435 conn(0x7fc9a8065ed0 msgr2=0x7fc9a80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:28.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 -- 172.21.15.60:0/3223029435 shutdown_connections 2023-10-16T14:07:28.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.917+0000 7fc9aefef640 1 -- 172.21.15.60:0/3223029435 wait complete. 2023-10-16T14:07:28.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.919+0000 7fc9aefef640 1 Processor -- start 2023-10-16T14:07:28.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.919+0000 7fc9aefef640 1 -- start start 2023-10-16T14:07:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9aefef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a80afd60 0x7fc9a8145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9aefef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 0x7fc9a81f1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9aefef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9a815aa60 0x7fc9a80aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9aefef640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8157690 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9aefef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8156520 con 0x7fc9a815aa60 2023-10-16T14:07:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9aefef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8156800 con 0x7fc9a80afd60 2023-10-16T14:07:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9ad7ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 0x7fc9a81f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9ad7ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 0x7fc9a81f1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35550/0 (socket says 172.21.15.60:35550) 2023-10-16T14:07:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9ad7ec640 1 -- 172.21.15.60:0/2884437916 learned_addr learned my addr 172.21.15.60:0/2884437916 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9adfed640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a80afd60 0x7fc9a8145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.921+0000 7fc9ae7ee640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9a815aa60 0x7fc9a80aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9ad7ec640 1 -- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a80afd60 msgr2=0x7fc9a8145430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9ad7ec640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a80afd60 0x7fc9a8145430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9ad7ec640 1 -- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9a815aa60 msgr2=0x7fc9a80aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:28.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9ad7ec640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9a815aa60 0x7fc9a80aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:28.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9ad7ec640 1 -- 172.21.15.60:0/2884437916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4009400 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9ad7ec640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 0x7fc9a81f1940 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fc99800c410 tx=0x7fc99800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:28.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998005ce0 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc998005e40 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a8156ae0 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.922+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9980072f0 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.923+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81f1e80 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.924+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a804e9b0 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.925+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 114) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc998007450 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.926+0000 7fc99e7fc640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc98c06af30 0x7fc98c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.926+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7fc998089190 con 0x7fc9a80cc9f0 2023-10-16T14:07:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.927+0000 7fc9adfed640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc98c06af30 0x7fc98c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.928+0000 7fc9adfed640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc98c06af30 0x7fc98c06d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4036000 tx=0x7fc9a403d040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:28.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:28.930+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc998057880 con 0x7fc9a80cc9f0 2023-10-16T14:07:29.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.170+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9a80cf6b0 con 0x7fc9a80cc9f0 2023-10-16T14:07:29.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.171+0000 7fc99e7fc640 1 -- 172.21.15.60:0/2884437916 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fc998057410 con 0x7fc9a80cc9f0 2023-10-16T14:07:29.175 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:29.175 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:51:27.881990+0000","modified":"2023-10-16T13:51:30.974922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9966},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9966":{"gid":9966,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/3850041246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3850041246},{"type":"v1","addr":"172.21.15.60:6835","nonce":3850041246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T14:07:29.175 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 95 2023-10-16T14:07:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc98c06af30 msgr2=0x7fc98c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc98c06af30 0x7fc98c06d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4036000 tx=0x7fc9a403d040 comp rx=0 tx=0).stop 2023-10-16T14:07:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 msgr2=0x7fc9a81f1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 0x7fc9a81f1940 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fc99800c410 tx=0x7fc99800c8d0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 shutdown_connections 2023-10-16T14:07:29.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc98c06af30 0x7fc98c06d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9a815aa60 0x7fc9a80aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9a80cc9f0 0x7fc9a81f1940 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 --2- 172.21.15.60:0/2884437916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9a80afd60 0x7fc9a8145430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 >> 172.21.15.60:0/2884437916 conn(0x7fc9a8065ed0 msgr2=0x7fc9a80cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:29.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 shutdown_connections 2023-10-16T14:07:29.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.174+0000 7fc9aefef640 1 -- 172.21.15.60:0/2884437916 wait complete. 2023-10-16T14:07:29.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:29.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.417+0000 7f65d2968640 1 Processor -- start 2023-10-16T14:07:29.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.417+0000 7f65d2968640 1 -- start start 2023-10-16T14:07:29.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.418+0000 7f65d2968640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65cc15aaa0 0x7f65cc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.418+0000 7f65d2968640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 0x7f65cc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.418+0000 7f65d2968640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc15ed60 con 0x7f65cc0ccaa0 2023-10-16T14:07:29.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.418+0000 7f65d2968640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc06b810 con 0x7f65cc15aaa0 2023-10-16T14:07:29.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.418+0000 7f65d2968640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0707e0 con 0x7f65cc15ae80 2023-10-16T14:07:29.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.419+0000 7f65d0f05640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f65cc0ccaa0 0x7f65cc070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56094/0 (socket says 172.21.15.60:56094) 2023-10-16T14:07:29.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.419+0000 7f65d0f05640 1 -- 172.21.15.60:0/3228510802 learned_addr learned my addr 172.21.15.60:0/3228510802 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:29.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.419+0000 7f65cbfff640 1 --2- 172.21.15.60:0/3228510802 >> v2:172.21.15.144:3300/0 conn(0x7f65cc15aaa0 0x7f65cc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.419+0000 7f65cb7fe640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 0x7f65cc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.419+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717461440 0 0) 0x7f65cc15ed60 con 0x7f65cc0ccaa0 2023-10-16T14:07:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.419+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b8003520 con 0x7f65cc0ccaa0 2023-10-16T14:07:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435942946 0 0) 0x7f65b8003520 con 0x7f65cc0ccaa0 2023-10-16T14:07:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 msgr2=0x7f65cc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:29.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 0x7f65cc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 >> v2:172.21.15.144:3300/0 conn(0x7f65cc15aaa0 msgr2=0x7f65cc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 --2- 172.21.15.60:0/3228510802 >> v2:172.21.15.144:3300/0 conn(0x7f65cc15aaa0 0x7f65cc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc15ef90 con 0x7f65cc0ccaa0 2023-10-16T14:07:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f65c4001670 con 0x7f65cc0ccaa0 2023-10-16T14:07:29.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 >> v1:172.21.15.60:6789/0 conn(0x7f65cc0ccaa0 legacy=0x7f65cc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8003970 0x7f65b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b80047c0 0x7f65b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 0x7f65b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc06b810 con 0x7f65b8003970 2023-10-16T14:07:29.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0707e0 con 0x7f65b80047c0 2023-10-16T14:07:29.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b800b980 con 0x7f65cc15ae80 2023-10-16T14:07:29.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.420+0000 7f65cbfff640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8003970 0x7f65b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65cb7fe640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b80047c0 0x7f65b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65d0f05640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 0x7f65b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65cbfff640 1 -- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 msgr2=0x7f65b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:29.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65cbfff640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15ae80 0x7f65b8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65cbfff640 1 -- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b80047c0 msgr2=0x7f65b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:29.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65cbfff640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b80047c0 0x7f65b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.421+0000 7f65cbfff640 1 -- 172.21.15.60:0/3228510802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b8003520 con 0x7f65b8003970 2023-10-16T14:07:29.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.422+0000 7f65cbfff640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8003970 0x7f65b8003d90 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009fc0 tx=0x7f65b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c407c946a81bbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:29.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.422+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc02cc30 con 0x7f65b8003970 2023-10-16T14:07:29.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.422+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc00f040 con 0x7f65b8003970 2023-10-16T14:07:29.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.422+0000 7f65caffd640 1 -- 172.21.15.60:0/3228510802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc034440 con 0x7f65b8003970 2023-10-16T14:07:29.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d2968640 1 -- 172.21.15.60:0/3228510802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8003970 msgr2=0x7f65b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d2968640 1 --2- 172.21.15.60:0/3228510802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8003970 0x7f65b8003d90 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009fc0 tx=0x7f65b80057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d0f05640 1 -- 172.21.15.60:0/3228510802 reap_dead start 2023-10-16T14:07:29.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d2968640 1 -- 172.21.15.60:0/3228510802 shutdown_connections 2023-10-16T14:07:29.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d2968640 1 -- 172.21.15.60:0/3228510802 >> 172.21.15.60:0/3228510802 conn(0x7f65cc0cb410 msgr2=0x7f65cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:29.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d2968640 1 -- 172.21.15.60:0/3228510802 shutdown_connections 2023-10-16T14:07:29.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.423+0000 7f65d2968640 1 -- 172.21.15.60:0/3228510802 wait complete. 2023-10-16T14:07:29.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.425+0000 7f65d2968640 1 Processor -- start 2023-10-16T14:07:29.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.425+0000 7f65d2968640 1 -- start start 2023-10-16T14:07:29.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d2968640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b8003970 0x7f65cc157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d2968640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15bf20 0x7f65cc15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d2968640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 0x7f65cc0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d2968640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0c8200 con 0x7f65cc0c7840 2023-10-16T14:07:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d2968640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0c24a0 con 0x7f65b8003970 2023-10-16T14:07:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d2968640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0c5a60 con 0x7f65cc15bf20 2023-10-16T14:07:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d0f05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 0x7f65cc0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d0f05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 0x7f65cc0c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35564/0 (socket says 172.21.15.60:35564) 2023-10-16T14:07:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65d0f05640 1 -- 172.21.15.60:0/1477664730 learned_addr learned my addr 172.21.15.60:0/1477664730 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65cb7fe640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15bf20 0x7f65cc15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.426+0000 7f65cbfff640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b8003970 0x7f65cc157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d0f05640 1 -- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15bf20 msgr2=0x7f65cc15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d0f05640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15bf20 0x7f65cc15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d0f05640 1 -- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b8003970 msgr2=0x7f65cc157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d0f05640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b8003970 0x7f65cc157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d0f05640 1 -- 172.21.15.60:0/1477664730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc15ca70 con 0x7f65cc0c7840 2023-10-16T14:07:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d0f05640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 0x7f65cc0c7cc0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ce2e0 tx=0x7f65c400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4007440 con 0x7f65cc0c7840 2023-10-16T14:07:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c40075c0 con 0x7f65cc0c7840 2023-10-16T14:07:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4004a20 con 0x7f65cc0c7840 2023-10-16T14:07:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.427+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc15cd50 con 0x7f65cc0c7840 2023-10-16T14:07:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.428+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc15d310 con 0x7f65cc0c7840 2023-10-16T14:07:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.429+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc0c5c80 con 0x7f65cc0c7840 2023-10-16T14:07:29.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.430+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 114) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f65c4004b80 con 0x7f65cc0c7840 2023-10-16T14:07:29.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.430+0000 7f65c8ff9640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f65ac06af00 0x7f65ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.431+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7f65c4089810 con 0x7f65cc0c7840 2023-10-16T14:07:29.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.431+0000 7f65cbfff640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f65ac06af00 0x7f65ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.432+0000 7f65cbfff640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f65ac06af00 0x7f65ac06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f65bc036000 tx=0x7f65bc02e000 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:29.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.435+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65cc0c5c80 con 0x7f65cc0c7840 2023-10-16T14:07:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.634+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65cc0c70c0 con 0x7f65cc0c7840 2023-10-16T14:07:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.636+0000 7f65c8ff9640 1 -- 172.21.15.60:0/1477664730 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+18150 (secure 0 0 0) 0x7f65c4054390 con 0x7f65cc0c7840 2023-10-16T14:07:29.640 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:29.640 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":202,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:26.077964+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:51:25.863711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-16T13:51:26.910746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3534900853":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/933999404":"2023-10-17T14:07:26.077866+0000","172.21.15.144:6833/345630502":"2023-10-17T14:07:26.077866+0000","172.21.15.144:6832/345630502":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1247236577":"2023-10-17T14:02:21.839066+0000","172.21.15.144:6833/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/483866973":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1895637100":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1129229162":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3061731632":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1482302803":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1876059409":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1022250268":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/103168750":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6833/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3241678538":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/2474164444":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1144561621":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3710907701":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1287992351":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3658895152":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1768527142":"2023-10-17T13:51:42.544813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:29.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f65ac06af00 msgr2=0x7f65ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f65ac06af00 0x7f65ac06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f65bc036000 tx=0x7f65bc02e000 comp rx=0 tx=0).stop 2023-10-16T14:07:29.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 msgr2=0x7f65cc0c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 0x7f65cc0c7cc0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ce2e0 tx=0x7f65c400d950 comp rx=0 tx=0).stop 2023-10-16T14:07:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 shutdown_connections 2023-10-16T14:07:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65cc15bf20 0x7f65cc15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f65ac06af00 0x7f65ac06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65cc0c7840 0x7f65cc0c7cc0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 --2- 172.21.15.60:0/1477664730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b8003970 0x7f65cc157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 >> 172.21.15.60:0/1477664730 conn(0x7f65cc0cb410 msgr2=0x7f65cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.639+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 shutdown_connections 2023-10-16T14:07:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.640+0000 7f65d2968640 1 -- 172.21.15.60:0/1477664730 wait complete. 2023-10-16T14:07:29.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:07:29.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.885+0000 7efd97c15640 1 Processor -- start 2023-10-16T14:07:29.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.885+0000 7efd97c15640 1 -- start start 2023-10-16T14:07:29.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.887+0000 7efd97c15640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efd9006a4b0 0x7efd900ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.887+0000 7efd97c15640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 0x7efd900ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.887+0000 7efd97c15640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd900cb030 con 0x7efd9015ad70 2023-10-16T14:07:29.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.887+0000 7efd97c15640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd900cae30 con 0x7efd9006a4b0 2023-10-16T14:07:29.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.887+0000 7efd97c15640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd900c91c0 con 0x7efd900c9ca0 2023-10-16T14:07:29.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 0x7efd900ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd951b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efd9006a4b0 0x7efd900ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 0x7efd900ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:43956/0 (socket says 172.21.15.60:43956) 2023-10-16T14:07:29.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 -- 172.21.15.60:0/932949507 learned_addr learned my addr 172.21.15.60:0/932949507 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:29.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 -- 172.21.15.60:0/932949507 >> v2:172.21.15.144:3300/0 conn(0x7efd9006a4b0 msgr2=0x7efd900ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:29.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 --2- 172.21.15.60:0/932949507 >> v2:172.21.15.144:3300/0 conn(0x7efd9006a4b0 0x7efd900ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 -- 172.21.15.60:0/932949507 >> v1:172.21.15.144:6789/0 conn(0x7efd9015ad70 legacy=0x7efd90069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.888+0000 7efd961b2640 1 -- 172.21.15.60:0/932949507 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900c94a0 con 0x7efd900c9ca0 2023-10-16T14:07:29.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.890+0000 7efd961b2640 1 --2- 172.21.15.60:0/932949507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 0x7efd900ca8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efd88009870 tx=0x7efd8802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76099eee8b1dbdc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:29.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.890+0000 7efd949af640 1 -- 172.21.15.60:0/932949507 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88030840 con 0x7efd900c9ca0 2023-10-16T14:07:29.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.890+0000 7efd949af640 1 -- 172.21.15.60:0/932949507 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd880309a0 con 0x7efd900c9ca0 2023-10-16T14:07:29.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.890+0000 7efd949af640 1 -- 172.21.15.60:0/932949507 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88030d30 con 0x7efd900c9ca0 2023-10-16T14:07:29.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 -- 172.21.15.60:0/932949507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 msgr2=0x7efd900ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 --2- 172.21.15.60:0/932949507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 0x7efd900ca8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efd88009870 tx=0x7efd8802b980 comp rx=0 tx=0).stop 2023-10-16T14:07:29.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 -- 172.21.15.60:0/932949507 shutdown_connections 2023-10-16T14:07:29.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 --2- 172.21.15.60:0/932949507 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd900c9ca0 0x7efd900ca8f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 --2- 172.21.15.60:0/932949507 >> v2:172.21.15.144:3300/0 conn(0x7efd9006a4b0 0x7efd900ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 -- 172.21.15.60:0/932949507 >> 172.21.15.60:0/932949507 conn(0x7efd900ce9b0 msgr2=0x7efd900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:29.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 -- 172.21.15.60:0/932949507 shutdown_connections 2023-10-16T14:07:29.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.891+0000 7efd97c15640 1 -- 172.21.15.60:0/932949507 wait complete. 2023-10-16T14:07:29.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.892+0000 7efd97c15640 1 Processor -- start 2023-10-16T14:07:29.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.893+0000 7efd97c15640 1 -- start start 2023-10-16T14:07:29.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd97c15640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9006a4b0 0x7efd900c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd97c15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 0x7efd90158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd97c15640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd9015ad70 0x7efd901f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd97c15640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900cc580 con 0x7efd900c9ca0 2023-10-16T14:07:29.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd97c15640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900ce410 con 0x7efd9006a4b0 2023-10-16T14:07:29.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd97c15640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd900cc7d0 con 0x7efd9015ad70 2023-10-16T14:07:29.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 0x7efd90158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 0x7efd90158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35576/0 (socket says 172.21.15.60:35576) 2023-10-16T14:07:29.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 -- 172.21.15.60:0/1109763427 learned_addr learned my addr 172.21.15.60:0/1109763427 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:29.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd959b1640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9006a4b0 0x7efd900c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd961b2640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd9015ad70 0x7efd901f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 -- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd9015ad70 msgr2=0x7efd901f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd9015ad70 0x7efd901f29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 -- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9006a4b0 msgr2=0x7efd900c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:29.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9006a4b0 0x7efd900c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:29.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.894+0000 7efd951b0640 1 -- 172.21.15.60:0/1109763427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880094d0 con 0x7efd900c9ca0 2023-10-16T14:07:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.895+0000 7efd951b0640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 0x7efd90158630 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7efd8000e9b0 tx=0x7efd8000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.895+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80009670 con 0x7efd900c9ca0 2023-10-16T14:07:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.895+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8000c9a0 con 0x7efd900c9ca0 2023-10-16T14:07:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.895+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd900ccab0 con 0x7efd900c9ca0 2023-10-16T14:07:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.895+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8000cd50 con 0x7efd900c9ca0 2023-10-16T14:07:29.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.895+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd90158b70 con 0x7efd900c9ca0 2023-10-16T14:07:29.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.897+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9004e160 con 0x7efd900c9ca0 2023-10-16T14:07:29.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.898+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 114) v1 ==== 89275+0+0 (secure 0 0 0) 0x7efd80013470 con 0x7efd900c9ca0 2023-10-16T14:07:29.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.899+0000 7efd867fc640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7efd7406af00 0x7efd7406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:29.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.900+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6769+0+0 (secure 0 0 0) 0x7efd80089d00 con 0x7efd900c9ca0 2023-10-16T14:07:29.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.900+0000 7efd959b1640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7efd7406af00 0x7efd7406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:29.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.901+0000 7efd959b1640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7efd7406af00 0x7efd7406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd9015c180 tx=0x7efd8c009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:29.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:29.903+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd8008d030 con 0x7efd900c9ca0 2023-10-16T14:07:30.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.137+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd80054d60 con 0x7efd900c9ca0 2023-10-16T14:07:30.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.143+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efd9015e310 con 0x7efd900c9ca0 2023-10-16T14:07:30.148 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:07:30.144+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:07:30.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.148+0000 7efd867fc640 1 -- 172.21.15.60:0/1109763427 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7efd80021020 con 0x7efd900c9ca0 2023-10-16T14:07:30.152 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:07:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7efd7406af00 msgr2=0x7efd7406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7efd7406af00 0x7efd7406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd9015c180 tx=0x7efd8c009040 comp rx=0 tx=0).stop 2023-10-16T14:07:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 msgr2=0x7efd90158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 0x7efd90158630 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7efd8000e9b0 tx=0x7efd8000ee70 comp rx=0 tx=0).stop 2023-10-16T14:07:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 shutdown_connections 2023-10-16T14:07:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd9015ad70 0x7efd901f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7efd7406af00 0x7efd7406d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd900c9ca0 0x7efd90158630 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 --2- 172.21.15.60:0/1109763427 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd9006a4b0 0x7efd900c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.151+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 >> 172.21.15.60:0/1109763427 conn(0x7efd900ce9b0 msgr2=0x7efd900ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.153+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 shutdown_connections 2023-10-16T14:07:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.153+0000 7efd97c15640 1 -- 172.21.15.60:0/1109763427 wait complete. 2023-10-16T14:07:30.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:07:30.281 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:07:30.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.372+0000 7f732caaf640 1 Processor -- start 2023-10-16T14:07:30.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.372+0000 7f732caaf640 1 -- start start 2023-10-16T14:07:30.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.373+0000 7f732caaf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73280addb0 0x7f73280caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:30.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.373+0000 7f732caaf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f732806a4b0 0x7f73281650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:30.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.373+0000 7f732caaf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73280cb140 con 0x7f73280c9000 2023-10-16T14:07:30.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.373+0000 7f732caaf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7328161a50 con 0x7f732806a4b0 2023-10-16T14:07:30.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.373+0000 7f732caaf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732806b030 con 0x7f73280addb0 2023-10-16T14:07:30.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f732806a4b0 0x7f73281650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:30.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f732806a4b0 0x7f73281650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35616/0 (socket says 172.21.15.60:35616) 2023-10-16T14:07:30.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 -- 172.21.15.60:0/1139580339 learned_addr learned my addr 172.21.15.60:0/1139580339 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:30.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f732659c640 1 --2- 172.21.15.60:0/1139580339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73280addb0 0x7f73280caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 -- 172.21.15.60:0/1139580339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73280addb0 msgr2=0x7f73280caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 --2- 172.21.15.60:0/1139580339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73280addb0 0x7f73280caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 -- 172.21.15.60:0/1139580339 >> v1:172.21.15.144:6789/0 conn(0x7f73280c9000 legacy=0x7f7328069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:30.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.374+0000 7f7326d9d640 1 -- 172.21.15.60:0/1139580339 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732806b310 con 0x7f732806a4b0 2023-10-16T14:07:30.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.375+0000 7f7326d9d640 1 --2- 172.21.15.60:0/1139580339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f732806a4b0 0x7f73281650d0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f7320009e50 tx=0x7f732002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11960614da7ad4c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f7325d9b640 1 -- 172.21.15.60:0/1139580339 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732002ca80 con 0x7f732806a4b0 2023-10-16T14:07:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f7325d9b640 1 -- 172.21.15.60:0/1139580339 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732002cbe0 con 0x7f732806a4b0 2023-10-16T14:07:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f7325d9b640 1 -- 172.21.15.60:0/1139580339 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73200343f0 con 0x7f732806a4b0 2023-10-16T14:07:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f732caaf640 1 -- 172.21.15.60:0/1139580339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f732806a4b0 msgr2=0x7f73281650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f732caaf640 1 --2- 172.21.15.60:0/1139580339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f732806a4b0 0x7f73281650d0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f7320009e50 tx=0x7f732002bee0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f732caaf640 1 -- 172.21.15.60:0/1139580339 shutdown_connections 2023-10-16T14:07:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f732caaf640 1 --2- 172.21.15.60:0/1139580339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f732806a4b0 0x7f73281650d0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f732caaf640 1 --2- 172.21.15.60:0/1139580339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73280addb0 0x7f73280caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.376+0000 7f732caaf640 1 -- 172.21.15.60:0/1139580339 >> 172.21.15.60:0/1139580339 conn(0x7f73280ce9b0 msgr2=0x7f73280cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.377+0000 7f732caaf640 1 -- 172.21.15.60:0/1139580339 shutdown_connections 2023-10-16T14:07:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.377+0000 7f732caaf640 1 -- 172.21.15.60:0/1139580339 wait complete. 2023-10-16T14:07:30.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.378+0000 7f732caaf640 1 Processor -- start 2023-10-16T14:07:30.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.378+0000 7f732caaf640 1 -- start start 2023-10-16T14:07:30.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.379+0000 7f732caaf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f732806a4b0 0x7f7328155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:30.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.379+0000 7f732caaf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 0x7f73280b3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:30.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.379+0000 7f732caaf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73280c9000 0x7f73280b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:30.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.379+0000 7f732caaf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73280c2fe0 con 0x7f73280addb0 2023-10-16T14:07:30.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.379+0000 7f732caaf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73280b2520 con 0x7f73280c9000 2023-10-16T14:07:30.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.379+0000 7f732caaf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73280b2800 con 0x7f732806a4b0 2023-10-16T14:07:30.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 0x7f73280b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:30.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 0x7f73280b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35630/0 (socket says 172.21.15.60:35630) 2023-10-16T14:07:30.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 -- 172.21.15.60:0/1419845251 learned_addr learned my addr 172.21.15.60:0/1419845251 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:30.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f732659c640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f732806a4b0 0x7f7328155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:30.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f7326d9d640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73280c9000 0x7f73280b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:30.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 -- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f732806a4b0 msgr2=0x7f7328155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:30.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f732806a4b0 0x7f7328155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 -- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73280c9000 msgr2=0x7f73280b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:30.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73280c9000 0x7f73280b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:30.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 -- 172.21.15.60:0/1419845251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320009b30 con 0x7f73280addb0 2023-10-16T14:07:30.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.380+0000 7f731ffff640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 0x7f73280b3820 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f731000b3f0 tx=0x7f731000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:30.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.381+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310011830 con 0x7f73280addb0 2023-10-16T14:07:30.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.381+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310011990 con 0x7f73280addb0 2023-10-16T14:07:30.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.381+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310011d40 con 0x7f73280addb0 2023-10-16T14:07:30.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.381+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73280b2ae0 con 0x7f73280addb0 2023-10-16T14:07:30.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.381+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7328156500 con 0x7f73280addb0 2023-10-16T14:07:30.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.382+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732804e160 con 0x7f73280addb0 2023-10-16T14:07:30.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.383+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f731001a400 con 0x7f73280addb0 2023-10-16T14:07:30.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.385+0000 7f731effd640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f730806af90 0x7f730806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:30.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.385+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f7310089b60 con 0x7f73280addb0 2023-10-16T14:07:30.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.385+0000 7f732659c640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f730806af90 0x7f730806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:30.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.387+0000 7f732659c640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f730806af90 0x7f730806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7318006fd0 tx=0x7f7318008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:30.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.389+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73100545e0 con 0x7f73280addb0 2023-10-16T14:07:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:30.625+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f73280aef10 con 0x7f73280addb0 2023-10-16T14:07:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.151+0000 7f731effd640 1 -- 172.21.15.60:0/1419845251 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7310057c00 con 0x7f73280addb0 2023-10-16T14:07:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.155+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f730806af90 msgr2=0x7f730806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.155+0000 7f732caaf640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f730806af90 0x7f730806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7318006fd0 tx=0x7f7318008040 comp rx=0 tx=0).stop 2023-10-16T14:07:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.155+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 msgr2=0x7f73280b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.155+0000 7f732caaf640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 0x7f73280b3820 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f731000b3f0 tx=0x7f731000b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 shutdown_connections 2023-10-16T14:07:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f730806af90 0x7f730806d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73280c9000 0x7f73280b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73280addb0 0x7f73280b3820 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 --2- 172.21.15.60:0/1419845251 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f732806a4b0 0x7f7328155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 >> 172.21.15.60:0/1419845251 conn(0x7f73280ce9b0 msgr2=0x7f732806bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:31.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 shutdown_connections 2023-10-16T14:07:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.156+0000 7f732caaf640 1 -- 172.21.15.60:0/1419845251 wait complete. 2023-10-16T14:07:31.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:07:31.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.403+0000 7f1832052640 1 Processor -- start 2023-10-16T14:07:31.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.403+0000 7f1832052640 1 -- start start 2023-10-16T14:07:31.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.404+0000 7f1832052640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c159950 0x7f182c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1832052640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f182c070450 0x7f182c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1832052640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182c070df0 con 0x7f182c06c100 2023-10-16T14:07:31.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1832052640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182c15e490 con 0x7f182c070450 2023-10-16T14:07:31.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1832052640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c15aef0 con 0x7f182c159950 2023-10-16T14:07:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1831851640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f182c06c100 0x7f182c06c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56108/0 (socket says 172.21.15.60:56108) 2023-10-16T14:07:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1831851640 1 -- 172.21.15.60:0/1094944797 learned_addr learned my addr 172.21.15.60:0/1094944797 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f183084f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f182c070450 0x7f182c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.405+0000 7f1831050640 1 --2- 172.21.15.60:0/1094944797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c159950 0x7f182c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.406+0000 7f183084f640 1 -- 172.21.15.60:0/1094944797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c159950 msgr2=0x7f182c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.406+0000 7f183084f640 1 --2- 172.21.15.60:0/1094944797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c159950 0x7f182c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.406+0000 7f183084f640 1 -- 172.21.15.60:0/1094944797 >> v1:172.21.15.60:6789/0 conn(0x7f182c06c100 legacy=0x7f182c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.406+0000 7f183084f640 1 -- 172.21.15.60:0/1094944797 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c06cc80 con 0x7f182c070450 2023-10-16T14:07:31.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.406+0000 7f1823fff640 1 -- 172.21.15.60:0/1094944797 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432611261 0 0) 0x7f182c070df0 con 0x7f182c06c100 2023-10-16T14:07:31.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.407+0000 7f183084f640 1 --2- 172.21.15.60:0/1094944797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0708b0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f181c0097a0 tx=0x7f181c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b282ebb9987ad46b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:31.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.407+0000 7f1823fff640 1 -- 172.21.15.60:0/1094944797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c02e070 con 0x7f182c070450 2023-10-16T14:07:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.407+0000 7f1823fff640 1 -- 172.21.15.60:0/1094944797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c0315a0 con 0x7f182c070450 2023-10-16T14:07:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.407+0000 7f1823fff640 1 -- 172.21.15.60:0/1094944797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c031930 con 0x7f182c070450 2023-10-16T14:07:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 -- 172.21.15.60:0/1094944797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 msgr2=0x7f182c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 --2- 172.21.15.60:0/1094944797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0708b0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f181c0097a0 tx=0x7f181c02b840 comp rx=0 tx=0).stop 2023-10-16T14:07:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 -- 172.21.15.60:0/1094944797 shutdown_connections 2023-10-16T14:07:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 --2- 172.21.15.60:0/1094944797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0708b0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 --2- 172.21.15.60:0/1094944797 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c159950 0x7f182c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 -- 172.21.15.60:0/1094944797 >> 172.21.15.60:0/1094944797 conn(0x7f182c0ce6a0 msgr2=0x7f182c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 -- 172.21.15.60:0/1094944797 shutdown_connections 2023-10-16T14:07:31.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.408+0000 7f1832052640 1 -- 172.21.15.60:0/1094944797 wait complete. 2023-10-16T14:07:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.410+0000 7f1832052640 1 Processor -- start 2023-10-16T14:07:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.411+0000 7f1832052640 1 -- start start 2023-10-16T14:07:31.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f1832052640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c06c100 0x7f182c15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f1832052640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f1832052640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f182c159950 0x7f182c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f1832052640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c15c3a0 con 0x7f182c070450 2023-10-16T14:07:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f1832052640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c15b790 con 0x7f182c159950 2023-10-16T14:07:31.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f1832052640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c15ba70 con 0x7f182c06c100 2023-10-16T14:07:31.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f183084f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f183084f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35644/0 (socket says 172.21.15.60:35644) 2023-10-16T14:07:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.412+0000 7f183084f640 1 -- 172.21.15.60:0/1313722280 learned_addr learned my addr 172.21.15.60:0/1313722280 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f1831050640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c06c100 0x7f182c15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f1831851640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f182c159950 0x7f182c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f183084f640 1 -- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c06c100 msgr2=0x7f182c15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f183084f640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c06c100 0x7f182c15b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f183084f640 1 -- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f182c159950 msgr2=0x7f182c150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f183084f640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f182c159950 0x7f182c150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f183084f640 1 -- 172.21.15.60:0/1313722280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c009400 con 0x7f182c070450 2023-10-16T14:07:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.413+0000 7f183084f640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0b3c30 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f181c034040 tx=0x7f181c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.414+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c032840 con 0x7f182c070450 2023-10-16T14:07:31.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.414+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c0329a0 con 0x7f182c070450 2023-10-16T14:07:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.414+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c15bd50 con 0x7f182c070450 2023-10-16T14:07:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.414+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c032d50 con 0x7f182c070450 2023-10-16T14:07:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.414+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c150fa0 con 0x7f182c070450 2023-10-16T14:07:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.415+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c15c550 con 0x7f182c070450 2023-10-16T14:07:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.416+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f181c04e430 con 0x7f182c070450 2023-10-16T14:07:31.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.417+0000 7f1821ffb640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f181006af90 0x7f181006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.417+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f181c0abdc0 con 0x7f182c070450 2023-10-16T14:07:31.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.418+0000 7f1831050640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f181006af90 0x7f181006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.419+0000 7f1831050640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f181006af90 0x7f181006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1828006fd0 tx=0x7f1828008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.421+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f182c15c550 con 0x7f182c070450 2023-10-16T14:07:31.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.619+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f182c04e9b0 con 0x7f182c070450 2023-10-16T14:07:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.732+0000 7f1821ffb640 1 -- 172.21.15.60:0/1313722280 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v204) v1 ==== 181+0+0 (secure 0 0 0) 0x7f181c076730 con 0x7f182c070450 2023-10-16T14:07:31.736 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:07:31.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.735+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f181006af90 msgr2=0x7f181006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.735+0000 7f1832052640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f181006af90 0x7f181006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1828006fd0 tx=0x7f1828008040 comp rx=0 tx=0).stop 2023-10-16T14:07:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.735+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 msgr2=0x7f182c0b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.735+0000 7f1832052640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0b3c30 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f181c034040 tx=0x7f181c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 shutdown_connections 2023-10-16T14:07:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f181006af90 0x7f181006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f182c159950 0x7f182c150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f182c070450 0x7f182c0b3c30 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 --2- 172.21.15.60:0/1313722280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f182c06c100 0x7f182c15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 >> 172.21.15.60:0/1313722280 conn(0x7f182c0ce6a0 msgr2=0x7f182c0c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:31.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.736+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 shutdown_connections 2023-10-16T14:07:31.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.737+0000 7f1832052640 1 -- 172.21.15.60:0/1313722280 wait complete. 2023-10-16T14:07:31.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:07:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.970+0000 7feef4fd5640 1 Processor -- start 2023-10-16T14:07:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.970+0000 7feef4fd5640 1 -- start start 2023-10-16T14:07:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.971+0000 7feef4fd5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feef00cca70 0x7feef00cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.971+0000 7feef4fd5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feef00cfa00 0x7feef015c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.971+0000 7feef4fd5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef006bbc0 con 0x7feef006cc60 2023-10-16T14:07:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.971+0000 7feef4fd5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef006bea0 con 0x7feef00cfa00 2023-10-16T14:07:31.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.971+0000 7feef4fd5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef006c180 con 0x7feef00cca70 2023-10-16T14:07:31.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feef00cfa00 0x7feef015c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feef00cfa00 0x7feef015c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35650/0 (socket says 172.21.15.60:35650) 2023-10-16T14:07:31.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 -- 172.21.15.60:0/182299727 learned_addr learned my addr 172.21.15.60:0/182299727 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:31.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeee59c640 1 --2- 172.21.15.60:0/182299727 >> v2:172.21.15.144:3300/0 conn(0x7feef00cca70 0x7feef00cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 -- 172.21.15.60:0/182299727 >> v2:172.21.15.144:3300/0 conn(0x7feef00cca70 msgr2=0x7feef00cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 --2- 172.21.15.60:0/182299727 >> v2:172.21.15.144:3300/0 conn(0x7feef00cca70 0x7feef00cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 -- 172.21.15.60:0/182299727 >> v1:172.21.15.144:6789/0 conn(0x7feef006cc60 legacy=0x7feef00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:31.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.972+0000 7feeeed9d640 1 -- 172.21.15.60:0/182299727 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef006c460 con 0x7feef00cfa00 2023-10-16T14:07:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.973+0000 7feeeed9d640 1 --2- 172.21.15.60:0/182299727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cfa00 0x7feef015c730 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7feef00b3670 tx=0x7feee802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5a76737cc5c3cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:31.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.974+0000 7feeed59a640 1 -- 172.21.15.60:0/182299727 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8031870 con 0x7feef00cfa00 2023-10-16T14:07:31.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.974+0000 7feeed59a640 1 -- 172.21.15.60:0/182299727 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee80319d0 con 0x7feef00cfa00 2023-10-16T14:07:31.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.974+0000 7feeed59a640 1 -- 172.21.15.60:0/182299727 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8031d60 con 0x7feef00cfa00 2023-10-16T14:07:31.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.974+0000 7feef4fd5640 1 -- 172.21.15.60:0/182299727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cfa00 msgr2=0x7feef015c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.974+0000 7feef4fd5640 1 --2- 172.21.15.60:0/182299727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cfa00 0x7feef015c730 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7feef00b3670 tx=0x7feee802b910 comp rx=0 tx=0).stop 2023-10-16T14:07:31.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.975+0000 7feef4fd5640 1 -- 172.21.15.60:0/182299727 shutdown_connections 2023-10-16T14:07:31.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.975+0000 7feef4fd5640 1 --2- 172.21.15.60:0/182299727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cfa00 0x7feef015c730 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.975+0000 7feef4fd5640 1 --2- 172.21.15.60:0/182299727 >> v2:172.21.15.144:3300/0 conn(0x7feef00cca70 0x7feef00cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.975+0000 7feef4fd5640 1 -- 172.21.15.60:0/182299727 >> 172.21.15.60:0/182299727 conn(0x7feef0099e30 msgr2=0x7feef00d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.975+0000 7feef4fd5640 1 -- 172.21.15.60:0/182299727 shutdown_connections 2023-10-16T14:07:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.975+0000 7feef4fd5640 1 -- 172.21.15.60:0/182299727 wait complete. 2023-10-16T14:07:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.976+0000 7feef4fd5640 1 Processor -- start 2023-10-16T14:07:31.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.977+0000 7feef4fd5640 1 -- start start 2023-10-16T14:07:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.977+0000 7feef4fd5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef006cc60 0x7feef014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.977+0000 7feef4fd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 0x7feef00c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feef4fd5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cfa00 0x7feef00c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feef4fd5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef00c9df0 con 0x7feef00cca70 2023-10-16T14:07:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feef4fd5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef00ca0d0 con 0x7feef00cfa00 2023-10-16T14:07:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feef4fd5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef00ca3b0 con 0x7feef006cc60 2023-10-16T14:07:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 0x7feef00c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 0x7feef00c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35654/0 (socket says 172.21.15.60:35654) 2023-10-16T14:07:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 -- 172.21.15.60:0/430132979 learned_addr learned my addr 172.21.15.60:0/430132979 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeeed9d640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cfa00 0x7feef00c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeee59c640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef006cc60 0x7feef014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 -- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef006cc60 msgr2=0x7feef014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef006cc60 0x7feef014ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 -- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cfa00 msgr2=0x7feef00c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cfa00 0x7feef00c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeedd9b640 1 -- 172.21.15.60:0/430132979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee80094d0 con 0x7feef00cca70 2023-10-16T14:07:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.978+0000 7feeeed9d640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cfa00 0x7feef00c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.979+0000 7feeedd9b640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 0x7feef00c9330 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7feed800b3f0 tx=0x7feed800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.979+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed8011830 con 0x7feef00cca70 2023-10-16T14:07:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.979+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed8011990 con 0x7feef00cca70 2023-10-16T14:07:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.979+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed8011d40 con 0x7feef00cca70 2023-10-16T14:07:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.979+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef00ca690 con 0x7feef00cca70 2023-10-16T14:07:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.979+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef00b66a0 con 0x7feef00cca70 2023-10-16T14:07:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.980+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef00b2420 con 0x7feef00cca70 2023-10-16T14:07:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.981+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feed801a400 con 0x7feef00cca70 2023-10-16T14:07:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.982+0000 7feee6ffd640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7feed006af90 0x7feed006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.982+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6417+0+0 (secure 0 0 0) 0x7feed8089880 con 0x7feef00cca70 2023-10-16T14:07:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.983+0000 7feeee59c640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7feed006af90 0x7feed006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.984+0000 7feeee59c640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7feed006af90 0x7feed006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feee0009e90 tx=0x7feee000b040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:31.987+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feed80545e0 con 0x7feef00cca70 2023-10-16T14:07:32.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:32.187+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feef00b35c0 con 0x7feef00cca70 2023-10-16T14:07:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.150+0000 7feee6ffd640 1 -- 172.21.15.60:0/430132979 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v205) v1 ==== 169+0+0 (secure 0 0 0) 0x7feef00b35c0 con 0x7feef00cca70 2023-10-16T14:07:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T14:07:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.154+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7feed006af90 msgr2=0x7feed006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.154+0000 7feef4fd5640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7feed006af90 0x7feed006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feee0009e90 tx=0x7feee000b040 comp rx=0 tx=0).stop 2023-10-16T14:07:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.155+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 msgr2=0x7feef00c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.155+0000 7feef4fd5640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 0x7feef00c9330 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7feed800b3f0 tx=0x7feed800b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.156+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 shutdown_connections 2023-10-16T14:07:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.156+0000 7feef4fd5640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7feed006af90 0x7feed006d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.156+0000 7feef4fd5640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feef00cfa00 0x7feef00c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.157+0000 7feef4fd5640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feef00cca70 0x7feef00c9330 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.157+0000 7feef4fd5640 1 --2- 172.21.15.60:0/430132979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feef006cc60 0x7feef014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.157+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 >> 172.21.15.60:0/430132979 conn(0x7feef0099e30 msgr2=0x7feef00ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.157+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 shutdown_connections 2023-10-16T14:07:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.158+0000 7feef4fd5640 1 -- 172.21.15.60:0/430132979 wait complete. 2023-10-16T14:07:33.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:07:33.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.400+0000 7fe0d3d90640 1 Processor -- start 2023-10-16T14:07:33.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.400+0000 7fe0d3d90640 1 -- start start 2023-10-16T14:07:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d3d90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc0adbe0 0x7fe0cc0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d3d90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0cc070fd0 0x7fe0cc06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d3d90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc0cc600 con 0x7fe0cc0adfc0 2023-10-16T14:07:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d3d90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0cc071470 con 0x7fe0cc070fd0 2023-10-16T14:07:33.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d3d90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0cc870 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d132b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0cc070fd0 0x7fe0cc06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d232d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe0cc0adfc0 0x7fe0cc0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56118/0 (socket says 172.21.15.60:56118) 2023-10-16T14:07:33.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.402+0000 7fe0d232d640 1 -- 172.21.15.60:0/2909700301 learned_addr learned my addr 172.21.15.60:0/2909700301 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:33.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d1b2c640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc0adbe0 0x7fe0cc0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d132b640 1 -- 172.21.15.60:0/2909700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc0adbe0 msgr2=0x7fe0cc0cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:33.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d132b640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc0adbe0 0x7fe0cc0cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d132b640 1 -- 172.21.15.60:0/2909700301 >> v1:172.21.15.60:6789/0 conn(0x7fe0cc0adfc0 legacy=0x7fe0cc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d132b640 1 -- 172.21.15.60:0/2909700301 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0ccb50 con 0x7fe0cc070fd0 2023-10-16T14:07:33.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d0b2a640 1 -- 172.21.15.60:0/2909700301 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266004274 0 0) 0x7fe0cc0cc600 con 0x7fe0cc0adfc0 2023-10-16T14:07:33.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.403+0000 7fe0d1b2c640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc0adbe0 0x7fe0cc0cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:33.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.404+0000 7fe0d132b640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc070fd0 0x7fe0cc06f550 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00097a0 tx=0x7fe0c002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38cd398f700412f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:33.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.404+0000 7fe0d0b2a640 1 -- 172.21.15.60:0/2909700301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c002d040 con 0x7fe0cc070fd0 2023-10-16T14:07:33.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.404+0000 7fe0d0b2a640 1 -- 172.21.15.60:0/2909700301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c0031810 con 0x7fe0cc070fd0 2023-10-16T14:07:33.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.404+0000 7fe0d0b2a640 1 -- 172.21.15.60:0/2909700301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c0031ba0 con 0x7fe0cc070fd0 2023-10-16T14:07:33.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2909700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc070fd0 msgr2=0x7fe0cc06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc070fd0 0x7fe0cc06f550 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00097a0 tx=0x7fe0c002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2909700301 shutdown_connections 2023-10-16T14:07:33.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc070fd0 0x7fe0cc06f550 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2909700301 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc0adbe0 0x7fe0cc0cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2909700301 >> 172.21.15.60:0/2909700301 conn(0x7fe0cc0cb410 msgr2=0x7fe0cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:33.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2909700301 shutdown_connections 2023-10-16T14:07:33.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.405+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2909700301 wait complete. 2023-10-16T14:07:33.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.407+0000 7fe0d3d90640 1 Processor -- start 2023-10-16T14:07:33.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.407+0000 7fe0d3d90640 1 -- start start 2023-10-16T14:07:33.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d3d90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc070fd0 0x7fe0cc0c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d3d90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 0x7fe0cc0c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d3d90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0cc0adfc0 0x7fe0cc0c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d3d90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0c1520 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d3d90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0c55c0 con 0x7fe0cc0adfc0 2023-10-16T14:07:33.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d3d90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0c41d0 con 0x7fe0cc070fd0 2023-10-16T14:07:33.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d132b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 0x7fe0cc0c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d132b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 0x7fe0cc0c0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35682/0 (socket says 172.21.15.60:35682) 2023-10-16T14:07:33.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.408+0000 7fe0d132b640 1 -- 172.21.15.60:0/2349369293 learned_addr learned my addr 172.21.15.60:0/2349369293 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:33.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d1b2c640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc070fd0 0x7fe0cc0c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d232d640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0cc0adfc0 0x7fe0cc0c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d132b640 1 -- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc070fd0 msgr2=0x7fe0cc0c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d132b640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc070fd0 0x7fe0cc0c3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d132b640 1 -- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0cc0adfc0 msgr2=0x7fe0cc0c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d132b640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0cc0adfc0 0x7fe0cc0c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d132b640 1 -- 172.21.15.60:0/2349369293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c0009400 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0d132b640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 0x7fe0cc0c0fe0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0000c00 tx=0x7fe0c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:33.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.409+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c003ad60 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.410+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c00443f0 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.410+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c00447a0 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.410+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc0c44b0 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.410+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc1630a0 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.411+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc0c49d0 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.412+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe0c004d450 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.413+0000 7fe0ca7fc640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe0b806af90 0x7fe0b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.413+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7fe0c00abd60 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.413+0000 7fe0d1b2c640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe0b806af90 0x7fe0b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.415+0000 7fe0d1b2c640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe0b806af90 0x7fe0b806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0c4b90 tx=0x7fe0bc0084f0 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:33.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.417+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0cc0c49d0 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.635+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe0cc162f00 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.640+0000 7fe0ca7fc640 1 -- 172.21.15.60:0/2349369293 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe0cc162f00 con 0x7fe0cc0adbe0 2023-10-16T14:07:33.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe0b806af90 msgr2=0x7fe0b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe0b806af90 0x7fe0b806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0c4b90 tx=0x7fe0bc0084f0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 msgr2=0x7fe0cc0c0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 0x7fe0cc0c0fe0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0000c00 tx=0x7fe0c00046c0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 shutdown_connections 2023-10-16T14:07:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe0b806af90 0x7fe0b806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0cc0adfc0 0x7fe0cc0c5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0cc0adbe0 0x7fe0cc0c0fe0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 --2- 172.21.15.60:0/2349369293 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0cc070fd0 0x7fe0cc0c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 >> 172.21.15.60:0/2349369293 conn(0x7fe0cc0cb410 msgr2=0x7fe0cc0b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 shutdown_connections 2023-10-16T14:07:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.643+0000 7fe0d3d90640 1 -- 172.21.15.60:0/2349369293 wait complete. 2023-10-16T14:07:33.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:07:33.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.880+0000 7f33479e2640 1 Processor -- start 2023-10-16T14:07:33.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.880+0000 7f33479e2640 1 -- start start 2023-10-16T14:07:33.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f33479e2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f334006a4b0 0x7f33400c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f33479e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33400ac890 con 0x7f33400ad350 2023-10-16T14:07:33.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f33479e2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33400ad7a0 con 0x7f334015d150 2023-10-16T14:07:33.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f33479e2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334015d530 con 0x7f334006a4b0 2023-10-16T14:07:33.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3344f7d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f33400ad350 0x7f3340069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56134/0 (socket says 172.21.15.60:56134) 2023-10-16T14:07:33.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3344f7d640 1 -- 172.21.15.60:0/2359681228 learned_addr learned my addr 172.21.15.60:0/2359681228 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:33.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3345f7f640 1 --2- 172.21.15.60:0/2359681228 >> v2:172.21.15.144:3300/0 conn(0x7f334006a4b0 0x7f33400c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003265330 0 0) 0x7f33400ac890 con 0x7f33400ad350 2023-10-16T14:07:33.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c003520 con 0x7f33400ad350 2023-10-16T14:07:33.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2857078048 0 0) 0x7f332c003520 con 0x7f33400ad350 2023-10-16T14:07:33.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 >> v2:172.21.15.144:3300/0 conn(0x7f334006a4b0 msgr2=0x7f33400c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:33.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 --2- 172.21.15.60:0/2359681228 >> v2:172.21.15.144:3300/0 conn(0x7f334006a4b0 0x7f33400c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 >> v1:172.21.15.144:6789/0 conn(0x7f334015d150 legacy=0x7f3340160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340161650 con 0x7f33400ad350 2023-10-16T14:07:33.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f3330001670 con 0x7f33400ad350 2023-10-16T14:07:33.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 >> v1:172.21.15.60:6789/0 conn(0x7f33400ad350 legacy=0x7f3340069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0039c0 0x7f332c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f332c0047c0 0x7f332c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.883+0000 7f3337fff640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332c005180 0x7f332c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334015d530 con 0x7f332c0039c0 2023-10-16T14:07:33.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c080 con 0x7f332c005180 2023-10-16T14:07:33.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00c240 con 0x7f332c0047c0 2023-10-16T14:07:33.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f334577e640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0039c0 0x7f332c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f3344f7d640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f332c0047c0 0x7f332c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f3345f7f640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332c005180 0x7f332c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f334577e640 1 -- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f332c0047c0 msgr2=0x7f332c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:33.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f334577e640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f332c0047c0 0x7f332c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f334577e640 1 -- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332c005180 msgr2=0x7f332c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:33.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f334577e640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f332c005180 0x7f332c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f334577e640 1 -- 172.21.15.60:0/2359681228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c003520 con 0x7f332c0039c0 2023-10-16T14:07:33.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.884+0000 7f3345f7f640 1 -- 172.21.15.60:0/2359681228 reap_dead start 2023-10-16T14:07:33.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f334577e640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0039c0 0x7f332c003de0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f333c001750 tx=0x7f332c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32af14123de1fc23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0052d0 con 0x7f332c0039c0 2023-10-16T14:07:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c02e050 con 0x7f332c0039c0 2023-10-16T14:07:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f3337fff640 1 -- 172.21.15.60:0/2359681228 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c003830 con 0x7f332c0039c0 2023-10-16T14:07:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f33479e2640 1 -- 172.21.15.60:0/2359681228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0039c0 msgr2=0x7f332c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f33479e2640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0039c0 0x7f332c003de0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f333c001750 tx=0x7f332c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f33479e2640 1 -- 172.21.15.60:0/2359681228 shutdown_connections 2023-10-16T14:07:33.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f33479e2640 1 --2- 172.21.15.60:0/2359681228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0039c0 0x7f332c003de0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.886+0000 7f33479e2640 1 -- 172.21.15.60:0/2359681228 >> 172.21.15.60:0/2359681228 conn(0x7f33400ce9b0 msgr2=0x7f33400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:33.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.887+0000 7f33479e2640 1 -- 172.21.15.60:0/2359681228 shutdown_connections 2023-10-16T14:07:33.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.887+0000 7f33479e2640 1 -- 172.21.15.60:0/2359681228 wait complete. 2023-10-16T14:07:33.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.888+0000 7f33479e2640 1 Processor -- start 2023-10-16T14:07:33.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.889+0000 7f33479e2640 1 -- start start 2023-10-16T14:07:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f33479e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 0x7f33401f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f33479e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33400b2d20 0x7f33400b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f33479e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33400c4af0 0x7f33400c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f33479e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33400b1c80 con 0x7f33401f31a0 2023-10-16T14:07:33.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f33479e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33400b1f60 con 0x7f33400c4af0 2023-10-16T14:07:33.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f33479e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33400b2240 con 0x7f33400b2d20 2023-10-16T14:07:33.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f334577e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 0x7f33401f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f334577e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 0x7f33401f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35700/0 (socket says 172.21.15.60:35700) 2023-10-16T14:07:33.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f334577e640 1 -- 172.21.15.60:0/630706710 learned_addr learned my addr 172.21.15.60:0/630706710 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:33.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f3344f7d640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33400b2d20 0x7f33400b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f3345f7f640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33400c4af0 0x7f33400c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f334577e640 1 -- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33400b2d20 msgr2=0x7f33400b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.890+0000 7f334577e640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33400b2d20 0x7f33400b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f334577e640 1 -- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33400c4af0 msgr2=0x7f33400c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:33.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f334577e640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33400c4af0 0x7f33400c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f334577e640 1 -- 172.21.15.60:0/630706710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33400b2520 con 0x7f33401f31a0 2023-10-16T14:07:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f334577e640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 0x7f33401f3580 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f333c000c00 tx=0x7f333c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0076c0 con 0x7f33401f31a0 2023-10-16T14:07:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c007a30 con 0x7f33401f31a0 2023-10-16T14:07:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33400b2800 con 0x7f33401f31a0 2023-10-16T14:07:33.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c006c20 con 0x7f33401f31a0 2023-10-16T14:07:33.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.891+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33400c2bf0 con 0x7f33401f31a0 2023-10-16T14:07:33.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.892+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334004e160 con 0x7f33401f31a0 2023-10-16T14:07:33.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.893+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f333c0035f0 con 0x7f33401f31a0 2023-10-16T14:07:33.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.894+0000 7f3335ffb640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f332406af90 0x7f332406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:33.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.895+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f333c0ab3e0 con 0x7f33401f31a0 2023-10-16T14:07:33.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.895+0000 7f3344f7d640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f332406af90 0x7f332406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:33.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.896+0000 7f3344f7d640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f332406af90 0x7f332406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f33400c2700 tx=0x7f3330009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:33.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:33.900+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f333c0760c0 con 0x7f33401f31a0 2023-10-16T14:07:34.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.096+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f33400b1220 con 0x7f33401f31a0 2023-10-16T14:07:34.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.103+0000 7f3335ffb640 1 -- 172.21.15.60:0/630706710 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f333c0796e0 con 0x7f33401f31a0 2023-10-16T14:07:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f332406af90 msgr2=0x7f332406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f332406af90 0x7f332406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f33400c2700 tx=0x7f3330009040 comp rx=0 tx=0).stop 2023-10-16T14:07:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 msgr2=0x7f33401f3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 0x7f33401f3580 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f333c000c00 tx=0x7f333c001120 comp rx=0 tx=0).stop 2023-10-16T14:07:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 shutdown_connections 2023-10-16T14:07:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f33400b2d20 0x7f33400b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f332406af90 0x7f332406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33400c4af0 0x7f33400c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 --2- 172.21.15.60:0/630706710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33401f31a0 0x7f33401f3580 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 >> 172.21.15.60:0/630706710 conn(0x7f33400ce9b0 msgr2=0x7f33400b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 shutdown_connections 2023-10-16T14:07:34.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.106+0000 7f33479e2640 1 -- 172.21.15.60:0/630706710 wait complete. 2023-10-16T14:07:34.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:07:34.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.346+0000 7f309a5ff640 1 Processor -- start 2023-10-16T14:07:34.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.347+0000 7f309a5ff640 1 -- start start 2023-10-16T14:07:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f309a5ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f30940b8480 0x7f30940cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f309a5ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940d0be0 0x7f30940cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f309a5ff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309415d150 con 0x7f309415cd70 2023-10-16T14:07:34.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f309a5ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30940d1180 con 0x7f30940b8480 2023-10-16T14:07:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f309a5ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30940d1460 con 0x7f30940d0be0 2023-10-16T14:07:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f3098dfc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f30940b8480 0x7f30940cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f3098dfc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f30940b8480 0x7f30940cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35710/0 (socket says 172.21.15.60:35710) 2023-10-16T14:07:34.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.348+0000 7f3098dfc640 1 -- 172.21.15.60:0/1573307072 learned_addr learned my addr 172.21.15.60:0/1573307072 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:34.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.349+0000 7f3099dfe640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940d0be0 0x7f30940cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.349+0000 7f3098dfc640 1 -- 172.21.15.60:0/1573307072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940d0be0 msgr2=0x7f30940cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:34.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.349+0000 7f3098dfc640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940d0be0 0x7f30940cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.349+0000 7f3098dfc640 1 -- 172.21.15.60:0/1573307072 >> v1:172.21.15.144:6789/0 conn(0x7f309415cd70 legacy=0x7f30940d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.349+0000 7f3098dfc640 1 -- 172.21.15.60:0/1573307072 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940d1740 con 0x7f30940b8480 2023-10-16T14:07:34.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.349+0000 7f3099dfe640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940d0be0 0x7f30940cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:34.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.350+0000 7f3098dfc640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30940b8480 0x7f30940cd7c0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f3088009870 tx=0x7f308802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948d1e00a7275190 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:34.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f3093fff640 1 -- 172.21.15.60:0/1573307072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088030840 con 0x7f30940b8480 2023-10-16T14:07:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f3093fff640 1 -- 172.21.15.60:0/1573307072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30880309a0 con 0x7f30940b8480 2023-10-16T14:07:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f3093fff640 1 -- 172.21.15.60:0/1573307072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088030d30 con 0x7f30940b8480 2023-10-16T14:07:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 -- 172.21.15.60:0/1573307072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30940b8480 msgr2=0x7f30940cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30940b8480 0x7f30940cd7c0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f3088009870 tx=0x7f308802b980 comp rx=0 tx=0).stop 2023-10-16T14:07:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 -- 172.21.15.60:0/1573307072 shutdown_connections 2023-10-16T14:07:34.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940d0be0 0x7f30940cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 --2- 172.21.15.60:0/1573307072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30940b8480 0x7f30940cd7c0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 -- 172.21.15.60:0/1573307072 >> 172.21.15.60:0/1573307072 conn(0x7f30940cf570 msgr2=0x7f30940cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:34.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 -- 172.21.15.60:0/1573307072 shutdown_connections 2023-10-16T14:07:34.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.351+0000 7f309a5ff640 1 -- 172.21.15.60:0/1573307072 wait complete. 2023-10-16T14:07:34.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.353+0000 7f309a5ff640 1 Processor -- start 2023-10-16T14:07:34.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.353+0000 7f309a5ff640 1 -- start start 2023-10-16T14:07:34.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.354+0000 7f309a5ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940b8480 0x7f309415fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.354+0000 7f309a5ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30940d0be0 0x7f3094155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.354+0000 7f309a5ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 0x7f30941f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.354+0000 7f309a5ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309415f390 con 0x7f309415cd70 2023-10-16T14:07:34.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.354+0000 7f309a5ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3094160300 con 0x7f30940d0be0 2023-10-16T14:07:34.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.354+0000 7f309a5ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30941f2de0 con 0x7f30940b8480 2023-10-16T14:07:34.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 0x7f30941f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 0x7f30941f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35722/0 (socket says 172.21.15.60:35722) 2023-10-16T14:07:34.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 -- 172.21.15.60:0/3160687543 learned_addr learned my addr 172.21.15.60:0/3160687543 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:34.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f30995fd640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940b8480 0x7f309415fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3098dfc640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30940d0be0 0x7f3094155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 -- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940b8480 msgr2=0x7f309415fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940b8480 0x7f309415fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 -- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30940d0be0 msgr2=0x7f3094155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30940d0be0 0x7f3094155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.355+0000 7f3099dfe640 1 -- 172.21.15.60:0/3160687543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30880094d0 con 0x7f309415cd70 2023-10-16T14:07:34.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.356+0000 7f3099dfe640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 0x7f30941f22b0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c5e0 tx=0x7f308c00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:34.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.356+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c011800 con 0x7f309415cd70 2023-10-16T14:07:34.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.356+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c0119a0 con 0x7f309415cd70 2023-10-16T14:07:34.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.356+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30941f30c0 con 0x7f309415cd70 2023-10-16T14:07:34.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.356+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c011d50 con 0x7f309415cd70 2023-10-16T14:07:34.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.356+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309415f8a0 con 0x7f309415cd70 2023-10-16T14:07:34.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.357+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30940b82a0 con 0x7f309415cd70 2023-10-16T14:07:34.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.358+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f308c022350 con 0x7f309415cd70 2023-10-16T14:07:34.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.359+0000 7f3091ffb640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f308006af90 0x7f308006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.359+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f308c089b00 con 0x7f309415cd70 2023-10-16T14:07:34.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.360+0000 7f30995fd640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f308006af90 0x7f308006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.361+0000 7f30995fd640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f308006af90 0x7f308006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30840012e0 tx=0x7f3084009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:34.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.364+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f308c08be60 con 0x7f309415cd70 2023-10-16T14:07:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.561+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f309404e9b0 con 0x7f309415cd70 2023-10-16T14:07:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.567+0000 7f3091ffb640 1 -- 172.21.15.60:0/3160687543 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7f308c0547e0 con 0x7f309415cd70 2023-10-16T14:07:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f308006af90 msgr2=0x7f308006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f308006af90 0x7f308006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30840012e0 tx=0x7f3084009040 comp rx=0 tx=0).stop 2023-10-16T14:07:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 msgr2=0x7f30941f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 0x7f30941f22b0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c5e0 tx=0x7f308c00caa0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 shutdown_connections 2023-10-16T14:07:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f308006af90 0x7f308006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309415cd70 0x7f30941f22b0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30940d0be0 0x7f3094155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 --2- 172.21.15.60:0/3160687543 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f30940b8480 0x7f309415fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 >> 172.21.15.60:0/3160687543 conn(0x7f30940cf570 msgr2=0x7f30940b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:34.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.570+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 shutdown_connections 2023-10-16T14:07:34.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.571+0000 7f309a5ff640 1 -- 172.21.15.60:0/3160687543 wait complete. 2023-10-16T14:07:34.593 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T14:07:34.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.818+0000 7f16bdc52640 1 Processor -- start 2023-10-16T14:07:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.819+0000 7f16bdc52640 1 -- start start 2023-10-16T14:07:34.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.819+0000 7f16bdc52640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.819+0000 7f16bdc52640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16b80cda80 0x7f16b815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.819+0000 7f16bdc52640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806c960 con 0x7f16b80ad970 2023-10-16T14:07:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.819+0000 7f16bdc52640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b815b9d0 con 0x7f16b80cda80 2023-10-16T14:07:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.819+0000 7f16bdc52640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b815bcb0 con 0x7f16b806c040 2023-10-16T14:07:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.820+0000 7f16bcc50640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.820+0000 7f16bd451640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16b80cda80 0x7f16b815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.820+0000 7f16bd451640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16b80cda80 0x7f16b815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:50408/0 (socket says 172.21.15.60:50408) 2023-10-16T14:07:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.820+0000 7f16bd451640 1 -- 172.21.15.60:0/1739528195 learned_addr learned my addr 172.21.15.60:0/1739528195 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.821+0000 7f16bd451640 1 -- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b806c040 msgr2=0x7f16b806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.821+0000 7f16bd451640 1 --2- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.821+0000 7f16bd451640 1 -- 172.21.15.60:0/1739528195 >> v1:172.21.15.144:6789/0 conn(0x7f16b80ad970 legacy=0x7f16b815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.821+0000 7f16bd451640 1 -- 172.21.15.60:0/1739528195 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b815bf90 con 0x7f16b80cda80 2023-10-16T14:07:34.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.822+0000 7f16bd451640 1 --2- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80cda80 0x7f16b815ed90 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009e50 tx=0x7f16b002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=242a19807f38c90a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:34.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.823+0000 7f16b77fe640 1 -- 172.21.15.60:0/1739528195 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b002cb00 con 0x7f16b80cda80 2023-10-16T14:07:34.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.823+0000 7f16b77fe640 1 -- 172.21.15.60:0/1739528195 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b002cc60 con 0x7f16b80cda80 2023-10-16T14:07:34.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.823+0000 7f16b77fe640 1 -- 172.21.15.60:0/1739528195 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00343f0 con 0x7f16b80cda80 2023-10-16T14:07:34.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.823+0000 7f16bdc52640 1 -- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80cda80 msgr2=0x7f16b815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.823+0000 7f16bdc52640 1 --2- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80cda80 0x7f16b815ed90 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009e50 tx=0x7f16b002bf60 comp rx=0 tx=0).stop 2023-10-16T14:07:34.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.824+0000 7f16bdc52640 1 -- 172.21.15.60:0/1739528195 shutdown_connections 2023-10-16T14:07:34.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.824+0000 7f16bdc52640 1 --2- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80cda80 0x7f16b815ed90 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.824+0000 7f16bdc52640 1 --2- 172.21.15.60:0/1739528195 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b806c040 0x7f16b806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.824+0000 7f16bdc52640 1 -- 172.21.15.60:0/1739528195 >> 172.21.15.60:0/1739528195 conn(0x7f16b80ce6a0 msgr2=0x7f16b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:34.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.824+0000 7f16bdc52640 1 -- 172.21.15.60:0/1739528195 shutdown_connections 2023-10-16T14:07:34.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.824+0000 7f16bdc52640 1 -- 172.21.15.60:0/1739528195 wait complete. 2023-10-16T14:07:34.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.826+0000 7f16bdc52640 1 Processor -- start 2023-10-16T14:07:34.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.826+0000 7f16bdc52640 1 -- start start 2023-10-16T14:07:34.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bdc52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b814fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bdc52640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80ad970 0x7f16b8150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bdc52640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80cda80 0x7f16b8159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bdc52640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8151b50 con 0x7f16b806c040 2023-10-16T14:07:34.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bdc52640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c7c40 con 0x7f16b80ad970 2023-10-16T14:07:34.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bdc52640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c7e50 con 0x7f16b80cda80 2023-10-16T14:07:34.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bcc50640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b814fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bcc50640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b814fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35730/0 (socket says 172.21.15.60:35730) 2023-10-16T14:07:34.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bcc50640 1 -- 172.21.15.60:0/3686097226 learned_addr learned my addr 172.21.15.60:0/3686097226 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:34.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16bd451640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80cda80 0x7f16b8159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.827+0000 7f16b7fff640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80ad970 0x7f16b8150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bcc50640 1 -- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80cda80 msgr2=0x7f16b8159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bcc50640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80cda80 0x7f16b8159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bcc50640 1 -- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80ad970 msgr2=0x7f16b8150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:34.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bcc50640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80ad970 0x7f16b8150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:34.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bcc50640 1 -- 172.21.15.60:0/3686097226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0009b30 con 0x7f16b806c040 2023-10-16T14:07:34.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bcc50640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b814fac0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00e870 tx=0x7f16ac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:34.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0094f0 con 0x7f16b806c040 2023-10-16T14:07:34.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac00c940 con 0x7f16b806c040 2023-10-16T14:07:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00ccf0 con 0x7f16b806c040 2023-10-16T14:07:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b80c8130 con 0x7f16b806c040 2023-10-16T14:07:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.828+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80c62f0 con 0x7f16b806c040 2023-10-16T14:07:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.830+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b806b330 con 0x7f16b806c040 2023-10-16T14:07:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.831+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f16ac013470 con 0x7f16b806c040 2023-10-16T14:07:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.831+0000 7f16b57fa640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f169c06af90 0x7f169c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.832+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f16ac089960 con 0x7f16b806c040 2023-10-16T14:07:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.832+0000 7f16b7fff640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f169c06af90 0x7f169c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.833+0000 7f16b7fff640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f169c06af90 0x7f169c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16b8159ac0 tx=0x7f16a4009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:34.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:34.836+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16ac0547e0 con 0x7f16b806c040 2023-10-16T14:07:35.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.101+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f16b80c8370 con 0x7f16b806c040 2023-10-16T14:07:35.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.156+0000 7f16b57fa640 1 -- 172.21.15.60:0/3686097226 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3062) v1 ==== 165+0+0 (secure 0 0 0) 0x7f16b80c8370 con 0x7f16b806c040 2023-10-16T14:07:35.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f169c06af90 msgr2=0x7f169c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f169c06af90 0x7f169c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16b8159ac0 tx=0x7f16a4009040 comp rx=0 tx=0).stop 2023-10-16T14:07:35.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 msgr2=0x7f16b814fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b814fac0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00e870 tx=0x7f16ac00ed30 comp rx=0 tx=0).stop 2023-10-16T14:07:35.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 shutdown_connections 2023-10-16T14:07:35.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16b80cda80 0x7f16b8159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f169c06af90 0x7f169c06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16b80ad970 0x7f16b8150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.160+0000 7f16bdc52640 1 --2- 172.21.15.60:0/3686097226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16b806c040 0x7f16b814fac0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.161+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 >> 172.21.15.60:0/3686097226 conn(0x7f16b80ce6a0 msgr2=0x7f16b8069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.161+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 shutdown_connections 2023-10-16T14:07:35.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.161+0000 7f16bdc52640 1 -- 172.21.15.60:0/3686097226 wait complete. 2023-10-16T14:07:35.186 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-16T14:07:35.186 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-16T14:07:35.186 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T14:07:35.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:35.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.400+0000 7fd8af59e640 1 Processor -- start 2023-10-16T14:07:35.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.401+0000 7fd8af59e640 1 -- start start 2023-10-16T14:07:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8af59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 0x7fd8b00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8af59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b015d150 con 0x7fd8b015cd70 2023-10-16T14:07:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8af59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00d1180 con 0x7fd8b00b8480 2023-10-16T14:07:35.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8af59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00d1460 con 0x7fd8b00cded0 2023-10-16T14:07:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8ae59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd8b015cd70 0x7fd8b00d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56146/0 (socket says 172.21.15.60:56146) 2023-10-16T14:07:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8ae59c640 1 -- 172.21.15.60:0/3336495061 learned_addr learned my addr 172.21.15.60:0/3336495061 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8aed9d640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 0x7fd8b00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371533851 0 0) 0x7fd8b015d150 con 0x7fd8b015cd70 2023-10-16T14:07:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89c003520 con 0x7fd8b015cd70 2023-10-16T14:07:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474835068 0 0) 0x7fd89c003520 con 0x7fd8b015cd70 2023-10-16T14:07:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 msgr2=0x7fd8b00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:35.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 0x7fd8b00b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 >> v1:172.21.15.144:6789/0 conn(0x7fd8b00b8480 legacy=0x7fd8b00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00d1740 con 0x7fd8b015cd70 2023-10-16T14:07:35.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd8a0001690 con 0x7fd8b015cd70 2023-10-16T14:07:35.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 >> v1:172.21.15.60:6789/0 conn(0x7fd8b015cd70 legacy=0x7fd8b00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd89c0039a0 0x7fd89c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.402+0000 7fd8add9b640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89c0047c0 0x7fd89c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8add9b640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 0x7fd89c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00d1460 con 0x7fd89c0039a0 2023-10-16T14:07:35.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c00bab0 con 0x7fd89c0047c0 2023-10-16T14:07:35.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c00bc70 con 0x7fd8b00cded0 2023-10-16T14:07:35.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd89c0039a0 0x7fd89c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8aed9d640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 0x7fd89c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8a7fff640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89c0047c0 0x7fd89c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8ae59c640 1 -- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 msgr2=0x7fd89c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:35.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8b00cded0 0x7fd89c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8ae59c640 1 -- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89c0047c0 msgr2=0x7fd89c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:35.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89c0047c0 0x7fd89c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8ae59c640 1 -- 172.21.15.60:0/3336495061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c003520 con 0x7fd89c0039a0 2023-10-16T14:07:35.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.403+0000 7fd8a7fff640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd89c0047c0 0x7fd89c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:35.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.404+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd89c0039a0 0x7fd89c003dc0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c160 tx=0x7fd89c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ac3b055bfd996b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:35.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0005ad0 con 0x7fd89c0039a0 2023-10-16T14:07:35.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0005c30 con 0x7fd89c0039a0 2023-10-16T14:07:35.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8add9b640 1 -- 172.21.15.60:0/3336495061 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a002c7c0 con 0x7fd89c0039a0 2023-10-16T14:07:35.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8af59e640 1 -- 172.21.15.60:0/3336495061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd89c0039a0 msgr2=0x7fd89c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3336495061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd89c0039a0 0x7fd89c003dc0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c160 tx=0x7fd89c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8aed9d640 1 -- 172.21.15.60:0/3336495061 reap_dead start 2023-10-16T14:07:35.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8af59e640 1 -- 172.21.15.60:0/3336495061 shutdown_connections 2023-10-16T14:07:35.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8af59e640 1 -- 172.21.15.60:0/3336495061 >> 172.21.15.60:0/3336495061 conn(0x7fd8b00cf570 msgr2=0x7fd8b00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:35.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.405+0000 7fd8af59e640 1 -- 172.21.15.60:0/3336495061 shutdown_connections 2023-10-16T14:07:35.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.406+0000 7fd8af59e640 1 -- 172.21.15.60:0/3336495061 wait complete. 2023-10-16T14:07:35.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.407+0000 7fd8af59e640 1 Processor -- start 2023-10-16T14:07:35.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.407+0000 7fd8af59e640 1 -- start start 2023-10-16T14:07:35.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.408+0000 7fd8af59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89c0039a0 0x7fd8b01f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.408+0000 7fd8af59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 0x7fd8b0156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.408+0000 7fd8af59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b0156940 0x7fd8b00b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8af59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0158510 con 0x7fd8b0158070 2023-10-16T14:07:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8af59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01587f0 con 0x7fd8b0156940 2023-10-16T14:07:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8af59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0158ad0 con 0x7fd89c0039a0 2023-10-16T14:07:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 0x7fd8b0156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 0x7fd8b0156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35754/0 (socket says 172.21.15.60:35754) 2023-10-16T14:07:35.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 -- 172.21.15.60:0/3273193434 learned_addr learned my addr 172.21.15.60:0/3273193434 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:35.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89c0039a0 0x7fd8b01f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8aed9d640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b0156940 0x7fd8b00b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 -- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89c0039a0 msgr2=0x7fd8b01f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89c0039a0 0x7fd8b01f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 -- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b0156940 msgr2=0x7fd8b00b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b0156940 0x7fd8b00b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.409+0000 7fd8a7fff640 1 -- 172.21.15.60:0/3273193434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b0158db0 con 0x7fd8b0158070 2023-10-16T14:07:35.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.410+0000 7fd8a7fff640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 0x7fd8b0156400 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fd89800d7a0 tx=0x7fd89800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:35.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.410+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898017070 con 0x7fd8b0158070 2023-10-16T14:07:35.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.410+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898005070 con 0x7fd8b0158070 2023-10-16T14:07:35.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.410+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898005420 con 0x7fd8b0158070 2023-10-16T14:07:35.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.410+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b0159090 con 0x7fd8b0158070 2023-10-16T14:07:35.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.410+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b01595b0 con 0x7fd8b0158070 2023-10-16T14:07:35.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.412+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b004e9b0 con 0x7fd8b0158070 2023-10-16T14:07:35.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.414+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd898007030 con 0x7fd8b0158070 2023-10-16T14:07:35.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.415+0000 7fd8a6ffd640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd89006afc0 0x7fd89006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.415+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7fd8980891e0 con 0x7fd8b0158070 2023-10-16T14:07:35.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.416+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd89006afc0 0x7fd89006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.417+0000 7fd8ae59c640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd89006afc0 0x7fd89006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0034000 tx=0x7fd8a0037040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:35.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.419+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd898053f60 con 0x7fd8b0158070 2023-10-16T14:07:35.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.616+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8b00cd4a0 con 0x7fd8b0158070 2023-10-16T14:07:35.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.619+0000 7fd8a6ffd640 1 -- 172.21.15.60:0/3273193434 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+14507 (secure 0 0 0) 0x7fd898057580 con 0x7fd8b0158070 2023-10-16T14:07:35.622 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:35.622 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":205,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:33.140593+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/3850041246":"2023-10-17T14:07:30.145266+0000","172.21.15.144:0/3534900853":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/933999404":"2023-10-17T14:07:26.077866+0000","172.21.15.144:6833/345630502":"2023-10-17T14:07:26.077866+0000","172.21.15.60:6834/3850041246":"2023-10-17T14:07:30.145266+0000","172.21.15.144:6832/345630502":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1247236577":"2023-10-17T14:02:21.839066+0000","172.21.15.144:6833/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/483866973":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1895637100":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1129229162":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3061731632":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1482302803":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1876059409":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/1022250268":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/103168750":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6833/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3241678538":"2023-10-17T14:07:26.077866+0000","172.21.15.144:0/2474164444":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/1700115305":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1144561621":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/3710907701":"2023-10-17T13:51:42.544813+0000","172.21.15.144:6832/861800364":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/1287992351":"2023-10-17T14:02:21.839066+0000","172.21.15.144:0/3658895152":"2023-10-17T13:51:42.544813+0000","172.21.15.144:0/1768527142":"2023-10-17T13:51:42.544813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:35.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd89006afc0 msgr2=0x7fd89006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd89006afc0 0x7fd89006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0034000 tx=0x7fd8a0037040 comp rx=0 tx=0).stop 2023-10-16T14:07:35.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 msgr2=0x7fd8b0156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 0x7fd8b0156400 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fd89800d7a0 tx=0x7fd89800dc60 comp rx=0 tx=0).stop 2023-10-16T14:07:35.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 shutdown_connections 2023-10-16T14:07:35.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fd89006afc0 0x7fd89006d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8b0156940 0x7fd8b00b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0158070 0x7fd8b0156400 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 --2- 172.21.15.60:0/3273193434 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd89c0039a0 0x7fd8b01f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 >> 172.21.15.60:0/3273193434 conn(0x7fd8b00cf570 msgr2=0x7fd8b00b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:35.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 shutdown_connections 2023-10-16T14:07:35.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.622+0000 7fd8af59e640 1 -- 172.21.15.60:0/3273193434 wait complete. 2023-10-16T14:07:35.645 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:07:35.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:07:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.874+0000 7fc41de7e640 1 Processor -- start 2023-10-16T14:07:35.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.874+0000 7fc41de7e640 1 -- start start 2023-10-16T14:07:35.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.876+0000 7fc41de7e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4180b2080 0x7fc4180d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.876+0000 7fc41de7e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180d0970 con 0x7fc4180b1ca0 2023-10-16T14:07:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.876+0000 7fc41de7e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180d0ea0 con 0x7fc4180cfc40 2023-10-16T14:07:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.876+0000 7fc41de7e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180d0bc0 con 0x7fc4180b2080 2023-10-16T14:07:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.876+0000 7fc41ce7c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc4180b1ca0 0x7fc4180ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56162/0 (socket says 172.21.15.60:56162) 2023-10-16T14:07:35.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.876+0000 7fc41ce7c640 1 -- 172.21.15.60:0/765042984 learned_addr learned my addr 172.21.15.60:0/765042984 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:35.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40ffff640 1 --2- 172.21.15.60:0/765042984 >> v2:172.21.15.144:3300/0 conn(0x7fc4180b2080 0x7fc4180d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496615857 0 0) 0x7fc4180d0970 con 0x7fc4180b1ca0 2023-10-16T14:07:35.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc400003520 con 0x7fc4180b1ca0 2023-10-16T14:07:35.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1232106577 0 0) 0x7fc400003520 con 0x7fc4180b1ca0 2023-10-16T14:07:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 >> v2:172.21.15.144:3300/0 conn(0x7fc4180b2080 msgr2=0x7fc4180d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 --2- 172.21.15.60:0/765042984 >> v2:172.21.15.144:3300/0 conn(0x7fc4180b2080 0x7fc4180d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 >> v1:172.21.15.144:6789/0 conn(0x7fc4180cfc40 legacy=0x7fc4180b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180cd940 con 0x7fc4180b1ca0 2023-10-16T14:07:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc4140019f0 con 0x7fc4180b1ca0 2023-10-16T14:07:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 >> v1:172.21.15.60:6789/0 conn(0x7fc4180b1ca0 legacy=0x7fc4180ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000039c0 0x7fc400003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc400005180 0x7fc40000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180d0bc0 con 0x7fc4000047c0 2023-10-16T14:07:35.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40000c060 con 0x7fc4000039c0 2023-10-16T14:07:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.877+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40000c220 con 0x7fc400005180 2023-10-16T14:07:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc40ffff640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc41ce7c640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000039c0 0x7fc400003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc41d67d640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc400005180 0x7fc40000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc40ffff640 1 -- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc400005180 msgr2=0x7fc40000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc40ffff640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc400005180 0x7fc40000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc40ffff640 1 -- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000039c0 msgr2=0x7fc400003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:35.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc40ffff640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000039c0 0x7fc400003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc40ffff640 1 -- 172.21.15.60:0/765042984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400003520 con 0x7fc4000047c0 2023-10-16T14:07:35.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.878+0000 7fc41d67d640 1 -- 172.21.15.60:0/765042984 reap_dead start 2023-10-16T14:07:35.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.879+0000 7fc40ffff640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fc408009ac0 tx=0x7fc4000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85e8c6a4d32b60 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40802c6b0 con 0x7fc4000047c0 2023-10-16T14:07:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40802c810 con 0x7fc4000047c0 2023-10-16T14:07:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc40f7fe640 1 -- 172.21.15.60:0/765042984 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40802cba0 con 0x7fc4000047c0 2023-10-16T14:07:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc41de7e640 1 -- 172.21.15.60:0/765042984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4000047c0 msgr2=0x7fc400004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc41de7e640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fc408009ac0 tx=0x7fc4000057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc41de7e640 1 -- 172.21.15.60:0/765042984 shutdown_connections 2023-10-16T14:07:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc41de7e640 1 --2- 172.21.15.60:0/765042984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4000047c0 0x7fc400004c40 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.880+0000 7fc41de7e640 1 -- 172.21.15.60:0/765042984 >> 172.21.15.60:0/765042984 conn(0x7fc4180d2b90 msgr2=0x7fc4180d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.881+0000 7fc41de7e640 1 -- 172.21.15.60:0/765042984 shutdown_connections 2023-10-16T14:07:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.881+0000 7fc41de7e640 1 -- 172.21.15.60:0/765042984 wait complete. 2023-10-16T14:07:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.882+0000 7fc41de7e640 1 Processor -- start 2023-10-16T14:07:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.883+0000 7fc41de7e640 1 -- start start 2023-10-16T14:07:35.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.883+0000 7fc41de7e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000047c0 0x7fc418154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41de7e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4180cb0c0 0x7fc4180cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41de7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 0x7fc41815fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41de7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc418160130 con 0x7fc41815f760 2023-10-16T14:07:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41de7e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180cbb80 con 0x7fc4000047c0 2023-10-16T14:07:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41de7e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180cbe60 con 0x7fc4180cb0c0 2023-10-16T14:07:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 0x7fc41815fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 0x7fc41815fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35762/0 (socket says 172.21.15.60:35762) 2023-10-16T14:07:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 -- 172.21.15.60:0/46617373 learned_addr learned my addr 172.21.15.60:0/46617373 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41ce7c640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000047c0 0x7fc418154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc40ffff640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4180cb0c0 0x7fc4180cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 -- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4180cb0c0 msgr2=0x7fc4180cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4180cb0c0 0x7fc4180cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 -- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000047c0 msgr2=0x7fc418154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000047c0 0x7fc418154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.884+0000 7fc41d67d640 1 -- 172.21.15.60:0/46617373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180cc140 con 0x7fc41815f760 2023-10-16T14:07:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.885+0000 7fc41d67d640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 0x7fc41815fbf0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fc4180d0020 tx=0x7fc41000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.885+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4100052e0 con 0x7fc41815f760 2023-10-16T14:07:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.885+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc410005440 con 0x7fc41815f760 2023-10-16T14:07:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.885+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc410003830 con 0x7fc41815f760 2023-10-16T14:07:35.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.885+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4180cc420 con 0x7fc41815f760 2023-10-16T14:07:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.885+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4181e6fe0 con 0x7fc41815f760 2023-10-16T14:07:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.886+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4180cc940 con 0x7fc41815f760 2023-10-16T14:07:35.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.888+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc410003990 con 0x7fc41815f760 2023-10-16T14:07:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.889+0000 7fc40d7fa640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc40406af90 0x7fc40406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.889+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7fc410088e90 con 0x7fc41815f760 2023-10-16T14:07:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.890+0000 7fc41ce7c640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc40406af90 0x7fc40406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:35.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.891+0000 7fc41ce7c640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc40406af90 0x7fc40406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc414001330 tx=0x7fc414009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:35.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:35.893+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4180cc940 con 0x7fc41815f760 2023-10-16T14:07:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.087+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4180cc940 con 0x7fc41815f760 2023-10-16T14:07:36.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.088+0000 7fc40d7fa640 1 -- 172.21.15.60:0/46617373 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4180cc940 con 0x7fc41815f760 2023-10-16T14:07:36.091 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:07:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.090+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc40406af90 msgr2=0x7fc40406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.090+0000 7fc41de7e640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc40406af90 0x7fc40406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc414001330 tx=0x7fc414009040 comp rx=0 tx=0).stop 2023-10-16T14:07:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.090+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 msgr2=0x7fc41815fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:36.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.090+0000 7fc41de7e640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 0x7fc41815fbf0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fc4180d0020 tx=0x7fc41000e8a0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 shutdown_connections 2023-10-16T14:07:36.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4180cb0c0 0x7fc4180cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc40406af90 0x7fc40406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc41815f760 0x7fc41815fbf0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 --2- 172.21.15.60:0/46617373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4000047c0 0x7fc418154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 >> 172.21.15.60:0/46617373 conn(0x7fc4180d2b90 msgr2=0x7fc418155ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:36.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 shutdown_connections 2023-10-16T14:07:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.091+0000 7fc41de7e640 1 -- 172.21.15.60:0/46617373 wait complete. 2023-10-16T14:07:36.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T14:07:36.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.341+0000 7f2085c40640 1 Processor -- start 2023-10-16T14:07:36.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.342+0000 7f2085c40640 1 -- start start 2023-10-16T14:07:36.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.343+0000 7f2085c40640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f208006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.343+0000 7f2085c40640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20800ad970 0x7f208015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.343+0000 7f2085c40640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f208015ebc0 0x7f208015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.343+0000 7f2085c40640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208006c960 con 0x7f208015ebc0 2023-10-16T14:07:36.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.343+0000 7f2085c40640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208015b9d0 con 0x7f20800ad970 2023-10-16T14:07:36.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.343+0000 7f2085c40640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208015bcb0 con 0x7f208006c040 2023-10-16T14:07:36.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f208015ebc0 0x7f208015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f208015ebc0 0x7f208015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35778/0 (socket says 172.21.15.60:35778) 2023-10-16T14:07:36.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 -- 172.21.15.60:0/4107807147 learned_addr learned my addr 172.21.15.60:0/4107807147 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:36.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f2077fff640 1 --2- 172.21.15.60:0/4107807147 >> v2:172.21.15.144:3300/0 conn(0x7f20800ad970 0x7f208015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f2084c3e640 1 --2- 172.21.15.60:0/4107807147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f208006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 -- 172.21.15.60:0/4107807147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 msgr2=0x7f208006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:36.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 --2- 172.21.15.60:0/4107807147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f208006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 -- 172.21.15.60:0/4107807147 >> v2:172.21.15.144:3300/0 conn(0x7f20800ad970 msgr2=0x7f208015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:36.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 --2- 172.21.15.60:0/4107807147 >> v2:172.21.15.144:3300/0 conn(0x7f20800ad970 0x7f208015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.344+0000 7f208543f640 1 -- 172.21.15.60:0/4107807147 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208015bf90 con 0x7f208015ebc0 2023-10-16T14:07:36.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.346+0000 7f208543f640 1 --2- 172.21.15.60:0/4107807147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f208015f050 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f2078009e50 tx=0x7f207802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd472eed63aecee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:36.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.346+0000 7f20777fe640 1 -- 172.21.15.60:0/4107807147 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207802ca70 con 0x7f208015ebc0 2023-10-16T14:07:36.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.346+0000 7f20777fe640 1 -- 172.21.15.60:0/4107807147 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207802cbd0 con 0x7f208015ebc0 2023-10-16T14:07:36.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.346+0000 7f20777fe640 1 -- 172.21.15.60:0/4107807147 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20780343f0 con 0x7f208015ebc0 2023-10-16T14:07:36.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 -- 172.21.15.60:0/4107807147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 msgr2=0x7f208015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:36.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 --2- 172.21.15.60:0/4107807147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f208015f050 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f2078009e50 tx=0x7f207802bed0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 -- 172.21.15.60:0/4107807147 shutdown_connections 2023-10-16T14:07:36.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 --2- 172.21.15.60:0/4107807147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f208015f050 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 --2- 172.21.15.60:0/4107807147 >> v2:172.21.15.144:3300/0 conn(0x7f20800ad970 0x7f208015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 --2- 172.21.15.60:0/4107807147 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f208006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 -- 172.21.15.60:0/4107807147 >> 172.21.15.60:0/4107807147 conn(0x7f20800ce6a0 msgr2=0x7f20800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:36.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 -- 172.21.15.60:0/4107807147 shutdown_connections 2023-10-16T14:07:36.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.347+0000 7f2085c40640 1 -- 172.21.15.60:0/4107807147 wait complete. 2023-10-16T14:07:36.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.349+0000 7f2085c40640 1 Processor -- start 2023-10-16T14:07:36.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.349+0000 7f2085c40640 1 -- start start 2023-10-16T14:07:36.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2085c40640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f20801482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2085c40640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20800ad970 0x7f208014ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2085c40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f2080149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2085c40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20801494e0 con 0x7f208015ebc0 2023-10-16T14:07:36.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2085c40640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208014b2d0 con 0x7f20800ad970 2023-10-16T14:07:36.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2085c40640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20801488d0 con 0x7f208006c040 2023-10-16T14:07:36.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f208543f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f2080149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f208543f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f2080149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35786/0 (socket says 172.21.15.60:35786) 2023-10-16T14:07:36.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f208543f640 1 -- 172.21.15.60:0/3247718288 learned_addr learned my addr 172.21.15.60:0/3247718288 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:36.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2077fff640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20800ad970 0x7f208014ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.351+0000 7f2084c3e640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f20801482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f208543f640 1 -- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 msgr2=0x7f20801482f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:36.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f208543f640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f20801482f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f208543f640 1 -- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20800ad970 msgr2=0x7f208014ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:36.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f208543f640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20800ad970 0x7f208014ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:36.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f208543f640 1 -- 172.21.15.60:0/3247718288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078009b30 con 0x7f208015ebc0 2023-10-16T14:07:36.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f2084c3e640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f20801482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:07:36.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f208543f640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f2080149970 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f2078009e50 tx=0x7f2078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:36.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078040670 con 0x7f208015ebc0 2023-10-16T14:07:36.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20780407d0 con 0x7f208015ebc0 2023-10-16T14:07:36.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2078040b80 con 0x7f208015ebc0 2023-10-16T14:07:36.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.352+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2080148bb0 con 0x7f208015ebc0 2023-10-16T14:07:36.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.353+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2080149170 con 0x7f208015ebc0 2023-10-16T14:07:36.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.354+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208006b330 con 0x7f208015ebc0 2023-10-16T14:07:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.355+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2078040ce0 con 0x7f208015ebc0 2023-10-16T14:07:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.356+0000 7f20757fa640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f206c06af90 0x7f206c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.356+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f20780ac900 con 0x7f208015ebc0 2023-10-16T14:07:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.357+0000 7f2084c3e640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f206c06af90 0x7f206c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.358+0000 7f2084c3e640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f206c06af90 0x7f206c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f207c00ba50 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:36.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.362+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2078077810 con 0x7f208015ebc0 2023-10-16T14:07:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:36.637+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f208004e9b0 con 0x7f208015ebc0 2023-10-16T14:07:37.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.171+0000 7f20757fa640 1 -- 172.21.15.60:0/3247718288 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3064) v1 ==== 166+0+0 (secure 0 0 0) 0x7f2078009510 con 0x7f208015ebc0 2023-10-16T14:07:37.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.174+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f206c06af90 msgr2=0x7f206c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.174+0000 7f2085c40640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f206c06af90 0x7f206c06d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f207c00ba50 tx=0x7f207c008040 comp rx=0 tx=0).stop 2023-10-16T14:07:37.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.174+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 msgr2=0x7f2080149970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.174+0000 7f2085c40640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f2080149970 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f2078009e50 tx=0x7f2078004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:37.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 shutdown_connections 2023-10-16T14:07:37.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f206c06af90 0x7f206c06d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f208015ebc0 0x7f2080149970 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20800ad970 0x7f208014ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 --2- 172.21.15.60:0/3247718288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f208006c040 0x7f20801482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 >> 172.21.15.60:0/3247718288 conn(0x7f20800ce6a0 msgr2=0x7f20800cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:37.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 shutdown_connections 2023-10-16T14:07:37.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.175+0000 7f2085c40640 1 -- 172.21.15.60:0/3247718288 wait complete. 2023-10-16T14:07:37.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:07:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.429+0000 7f08b7fa9640 1 Processor -- start 2023-10-16T14:07:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.429+0000 7f08b7fa9640 1 -- start start 2023-10-16T14:07:37.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.431+0000 7f08b7fa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b00caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:37.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.431+0000 7f08b7fa9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f08b006a4b0 0x7f08b01650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:37.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.431+0000 7f08b7fa9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00cb140 con 0x7f08b00c9000 2023-10-16T14:07:37.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.431+0000 7f08b7fa9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b0161a50 con 0x7f08b006a4b0 2023-10-16T14:07:37.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.431+0000 7f08b7fa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b006b030 con 0x7f08b00addb0 2023-10-16T14:07:37.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.432+0000 7f08b5544640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f08b00c9000 0x7f08b0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56170/0 (socket says 172.21.15.60:56170) 2023-10-16T14:07:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.432+0000 7f08b5544640 1 -- 172.21.15.60:0/3092358664 learned_addr learned my addr 172.21.15.60:0/3092358664 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.432+0000 7f08b5d45640 1 --2- 172.21.15.60:0/3092358664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b00caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:37.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.432+0000 7f08b6546640 1 --2- 172.21.15.60:0/3092358664 >> v2:172.21.15.144:3300/0 conn(0x7f08b006a4b0 0x7f08b01650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.432+0000 7f08b4d43640 1 -- 172.21.15.60:0/3092358664 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112838114 0 0) 0x7f08b00cb140 con 0x7f08b00c9000 2023-10-16T14:07:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.432+0000 7f08b4d43640 1 -- 172.21.15.60:0/3092358664 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f089c003520 con 0x7f08b00c9000 2023-10-16T14:07:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.433+0000 7f08b5d45640 1 -- 172.21.15.60:0/3092358664 >> v2:172.21.15.144:3300/0 conn(0x7f08b006a4b0 msgr2=0x7f08b01650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.433+0000 7f08b5d45640 1 --2- 172.21.15.60:0/3092358664 >> v2:172.21.15.144:3300/0 conn(0x7f08b006a4b0 0x7f08b01650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.433+0000 7f08b5d45640 1 -- 172.21.15.60:0/3092358664 >> v1:172.21.15.60:6789/0 conn(0x7f08b00c9000 legacy=0x7f08b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.433+0000 7f08b5d45640 1 -- 172.21.15.60:0/3092358664 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b006b310 con 0x7f08b00addb0 2023-10-16T14:07:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.433+0000 7f08b6546640 1 --2- 172.21.15.60:0/3092358664 >> v2:172.21.15.144:3300/0 conn(0x7f08b006a4b0 0x7f08b01650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:07:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.434+0000 7f08b5d45640 1 --2- 172.21.15.60:0/3092358664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b00caa80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0097a0 tx=0x7f08ac02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f1172ce68cd49e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.435+0000 7f08b4d43640 1 -- 172.21.15.60:0/3092358664 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac02d040 con 0x7f08b00addb0 2023-10-16T14:07:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.435+0000 7f08b4d43640 1 -- 172.21.15.60:0/3092358664 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac031810 con 0x7f08b00addb0 2023-10-16T14:07:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.435+0000 7f08b4d43640 1 -- 172.21.15.60:0/3092358664 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac031ba0 con 0x7f08b00addb0 2023-10-16T14:07:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 -- 172.21.15.60:0/3092358664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 msgr2=0x7f08b00caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/3092358664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b00caa80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0097a0 tx=0x7f08ac02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 -- 172.21.15.60:0/3092358664 shutdown_connections 2023-10-16T14:07:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/3092358664 >> v2:172.21.15.144:3300/0 conn(0x7f08b006a4b0 0x7f08b01650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/3092358664 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b00caa80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 -- 172.21.15.60:0/3092358664 >> 172.21.15.60:0/3092358664 conn(0x7f08b00ce9b0 msgr2=0x7f08b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 -- 172.21.15.60:0/3092358664 shutdown_connections 2023-10-16T14:07:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.436+0000 7f08b7fa9640 1 -- 172.21.15.60:0/3092358664 wait complete. 2023-10-16T14:07:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.438+0000 7f08b7fa9640 1 Processor -- start 2023-10-16T14:07:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.438+0000 7f08b7fa9640 1 -- start start 2023-10-16T14:07:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.439+0000 7f08b7fa9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 0x7f08b00711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.439+0000 7f08b7fa9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b015eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.439+0000 7f08b7fa9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08b00c9000 0x7f08b00c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.439+0000 7f08b7fa9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c4160 con 0x7f08b006a4b0 2023-10-16T14:07:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.439+0000 7f08b7fa9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c34e0 con 0x7f08b00c9000 2023-10-16T14:07:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.439+0000 7f08b7fa9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c37c0 con 0x7f08b00addb0 2023-10-16T14:07:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 0x7f08b00711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 0x7f08b00711a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35798/0 (socket says 172.21.15.60:35798) 2023-10-16T14:07:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 -- 172.21.15.60:0/1212549471 learned_addr learned my addr 172.21.15.60:0/1212549471 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b6546640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08b00c9000 0x7f08b00c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5544640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b015eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 -- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 msgr2=0x7f08b015eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b015eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 -- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08b00c9000 msgr2=0x7f08b00c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08b00c9000 0x7f08b00c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.440+0000 7f08b5d45640 1 -- 172.21.15.60:0/1212549471 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8009a70 con 0x7f08b006a4b0 2023-10-16T14:07:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.441+0000 7f08b5d45640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 0x7f08b00711a0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098c0 tx=0x7f08ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.441+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac03ad60 con 0x7f08b006a4b0 2023-10-16T14:07:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.441+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac044400 con 0x7f08b006a4b0 2023-10-16T14:07:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.441+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0445b0 con 0x7f08b006a4b0 2023-10-16T14:07:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.441+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac009400 con 0x7f08b006a4b0 2023-10-16T14:07:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.441+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b00c3d80 con 0x7f08b006a4b0 2023-10-16T14:07:37.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.442+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b004e160 con 0x7f08b006a4b0 2023-10-16T14:07:37.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.443+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08ac04d440 con 0x7f08b006a4b0 2023-10-16T14:07:37.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.444+0000 7f08a67fc640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f089406afc0 0x7f089406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:37.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.444+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f08ac0abc10 con 0x7f08b006a4b0 2023-10-16T14:07:37.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.445+0000 7f08b5544640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f089406afc0 0x7f089406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:37.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.446+0000 7f08b5544640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f089406afc0 0x7f089406d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f08b00c8a80 tx=0x7f08a0009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:37.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.449+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08ac076ab0 con 0x7f08b006a4b0 2023-10-16T14:07:37.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.648+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f08b006c5b0 con 0x7f08b006a4b0 2023-10-16T14:07:37.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.653+0000 7f08a67fc640 1 -- 172.21.15.60:0/1212549471 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f08ac0350c0 con 0x7f08b006a4b0 2023-10-16T14:07:37.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f089406afc0 msgr2=0x7f089406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f089406afc0 0x7f089406d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f08b00c8a80 tx=0x7f08a0009040 comp rx=0 tx=0).stop 2023-10-16T14:07:37.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 msgr2=0x7f08b00711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:37.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 0x7f08b00711a0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0098c0 tx=0x7f08ac0046c0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 shutdown_connections 2023-10-16T14:07:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f08b00addb0 0x7f08b015eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f089406afc0 0x7f089406d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08b00c9000 0x7f08b00c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 --2- 172.21.15.60:0/1212549471 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08b006a4b0 0x7f08b00711a0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:37.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 >> 172.21.15.60:0/1212549471 conn(0x7f08b00ce9b0 msgr2=0x7f08b00afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:37.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 shutdown_connections 2023-10-16T14:07:37.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:37.656+0000 7f08b7fa9640 1 -- 172.21.15.60:0/1212549471 wait complete. 2023-10-16T14:07:37.679 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:07:37.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:07:37.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:07:37.943 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:07:37.943 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:07:37.944 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:38.007 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:07:38.008 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:07:38.008 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:38.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:07:38.069 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:07:38.070 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:38.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:07:38.134 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:07:38.135 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:38.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:07:38.197 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:07:38.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:38.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.435+0000 7f6960a85640 1 Processor -- start 2023-10-16T14:07:38.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.435+0000 7f6960a85640 1 -- start start 2023-10-16T14:07:38.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.436+0000 7f6960a85640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f695c06c040 0x7f695c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.436+0000 7f6960a85640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c0ad970 0x7f695c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.436+0000 7f6960a85640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c06c960 con 0x7f695c15ebc0 2023-10-16T14:07:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.436+0000 7f6960a85640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c15b9d0 con 0x7f695c06c040 2023-10-16T14:07:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.436+0000 7f6960a85640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c15bcb0 con 0x7f695c0ad970 2023-10-16T14:07:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.437+0000 7f695affd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c0ad970 0x7f695c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.437+0000 7f695b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f695c06c040 0x7f695c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.437+0000 7f695affd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c0ad970 0x7f695c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:44194/0 (socket says 172.21.15.60:44194) 2023-10-16T14:07:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.437+0000 7f695affd640 1 -- 172.21.15.60:0/3521511598 learned_addr learned my addr 172.21.15.60:0/3521511598 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.438+0000 7f695b7fe640 1 -- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c0ad970 msgr2=0x7f695c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.438+0000 7f695b7fe640 1 --2- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c0ad970 0x7f695c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.438+0000 7f695b7fe640 1 -- 172.21.15.60:0/3521511598 >> v1:172.21.15.144:6789/0 conn(0x7f695c15ebc0 legacy=0x7f695c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:38.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.438+0000 7f695b7fe640 1 -- 172.21.15.60:0/3521511598 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c15bf90 con 0x7f695c06c040 2023-10-16T14:07:38.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.439+0000 7f695b7fe640 1 --2- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c06c420 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f694c0097d0 tx=0x7f694c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ca7e61293682e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:38.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f695a7fc640 1 -- 172.21.15.60:0/3521511598 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c02d040 con 0x7f695c06c040 2023-10-16T14:07:38.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f695a7fc640 1 -- 172.21.15.60:0/3521511598 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c031810 con 0x7f695c06c040 2023-10-16T14:07:38.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f695a7fc640 1 -- 172.21.15.60:0/3521511598 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c031ba0 con 0x7f695c06c040 2023-10-16T14:07:38.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 -- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 msgr2=0x7f695c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:38.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 --2- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c06c420 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f694c0097d0 tx=0x7f694c02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 -- 172.21.15.60:0/3521511598 shutdown_connections 2023-10-16T14:07:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 --2- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c0ad970 0x7f695c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 --2- 172.21.15.60:0/3521511598 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c06c420 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 -- 172.21.15.60:0/3521511598 >> 172.21.15.60:0/3521511598 conn(0x7f695c0ce6a0 msgr2=0x7f695c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 -- 172.21.15.60:0/3521511598 shutdown_connections 2023-10-16T14:07:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.440+0000 7f6960a85640 1 -- 172.21.15.60:0/3521511598 wait complete. 2023-10-16T14:07:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.442+0000 7f6960a85640 1 Processor -- start 2023-10-16T14:07:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.442+0000 7f6960a85640 1 -- start start 2023-10-16T14:07:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.443+0000 7f6960a85640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.443+0000 7f6960a85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 0x7f695c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.443+0000 7f6960a85640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c15ebc0 0x7f695c1f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.443+0000 7f6960a85640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c8cf0 con 0x7f695c0ad970 2023-10-16T14:07:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.443+0000 7f6960a85640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c14e3e0 con 0x7f695c06c040 2023-10-16T14:07:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.443+0000 7f6960a85640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c9550 con 0x7f695c15ebc0 2023-10-16T14:07:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 0x7f695c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 0x7f695c0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35802/0 (socket says 172.21.15.60:35802) 2023-10-16T14:07:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 -- 172.21.15.60:0/614731154 learned_addr learned my addr 172.21.15.60:0/614731154 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695bfff640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c15ebc0 0x7f695c1f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695b7fe640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 -- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c15ebc0 msgr2=0x7f695c1f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c15ebc0 0x7f695c1f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 -- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 msgr2=0x7f695c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695affd640 1 -- 172.21.15.60:0/614731154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c009400 con 0x7f695c0ad970 2023-10-16T14:07:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695bfff640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c15ebc0 0x7f695c1f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.444+0000 7f695b7fe640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.445+0000 7f695affd640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 0x7f695c0acdd0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f694400efc0 tx=0x7f694400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.445+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944009b60 con 0x7f695c0ad970 2023-10-16T14:07:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.445+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694400f040 con 0x7f695c0ad970 2023-10-16T14:07:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.445+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69440092e0 con 0x7f695c0ad970 2023-10-16T14:07:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.445+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c1f67a0 con 0x7f695c0ad970 2023-10-16T14:07:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.445+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c0c91e0 con 0x7f695c0ad970 2023-10-16T14:07:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.446+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c06b330 con 0x7f695c0ad970 2023-10-16T14:07:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.447+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6944014480 con 0x7f695c0ad970 2023-10-16T14:07:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.449+0000 7f6953fff640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f693c06afe0 0x7f693c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.449+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f694408a090 con 0x7f695c0ad970 2023-10-16T14:07:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.449+0000 7f695b7fe640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f693c06afe0 0x7f693c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.451+0000 7f695b7fe640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f693c06afe0 0x7f693c06d4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f694c036000 tx=0x7f694c03c040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.453+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6944054e80 con 0x7f695c0ad970 2023-10-16T14:07:38.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.690+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f695c04e9b0 con 0x7f695c0ad970 2023-10-16T14:07:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.691+0000 7f6953fff640 1 -- 172.21.15.60:0/614731154 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3703 (secure 0 0 0) 0x7f69440584a0 con 0x7f695c0ad970 2023-10-16T14:07:38.694 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:38.695 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":97,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/2560471112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":2560471112},{"type":"v1","addr":"172.21.15.144:6837","nonce":2560471112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2114242797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2114242797},{"type":"v1","addr":"172.21.15.60:6833","nonce":2114242797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/955049435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":955049435},{"type":"v1","addr":"172.21.15.144:6835","nonce":955049435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":10407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4283367164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":4283367164},{"type":"v1","addr":"172.21.15.60:6835","nonce":4283367164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[]} 2023-10-16T14:07:38.695 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 97 2023-10-16T14:07:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f693c06afe0 msgr2=0x7f693c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f693c06afe0 0x7f693c06d4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f694c036000 tx=0x7f694c03c040 comp rx=0 tx=0).stop 2023-10-16T14:07:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 msgr2=0x7f695c0acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 0x7f695c0acdd0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f694400efc0 tx=0x7f694400c490 comp rx=0 tx=0).stop 2023-10-16T14:07:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 shutdown_connections 2023-10-16T14:07:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f695c15ebc0 0x7f695c1f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f693c06afe0 0x7f693c06d4a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f695c0ad970 0x7f695c0acdd0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 --2- 172.21.15.60:0/614731154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c06c040 0x7f695c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:38.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 >> 172.21.15.60:0/614731154 conn(0x7f695c0ce6a0 msgr2=0x7f695c15d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:38.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.694+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 shutdown_connections 2023-10-16T14:07:38.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.695+0000 7f6960a85640 1 -- 172.21.15.60:0/614731154 wait complete. 2023-10-16T14:07:38.716 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:07:38.716 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:07:38.717 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:07:38.717 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:07:38.717 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:07:38.718 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:07:38.718 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:07:38.719 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:07:38.719 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:07:38.719 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:07:38.719 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:07:38.720 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:07:38.720 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:07:38.720 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:07:38.721 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:07:38.721 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:07:38.740 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:07:38.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:07:38.742 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:07:38.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:07:38.745 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:07:38.745 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:07:38.749 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:07:38.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:07:38.789 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:07:38.790 DEBUG:teuthology.parallel:result is None 2023-10-16T14:07:38.796 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:07:38.796 DEBUG:teuthology.parallel:result is None 2023-10-16T14:07:38.797 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:07:38.798 DEBUG:teuthology.parallel:result is None 2023-10-16T14:07:38.805 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:07:38.805 DEBUG:teuthology.parallel:result is None 2023-10-16T14:07:38.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:07:38.955 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:07:38.956 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:07:38.959 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:07:38.969 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T14:07:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.998+0000 7fae00f47640 1 Processor -- start 2023-10-16T14:07:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.998+0000 7fae00f47640 1 -- start start 2023-10-16T14:07:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fae00f47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fadfc0b3420 0x7fadfc0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fae00f47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fadfc0cfaf0 0x7fadfc0b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fae00f47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc0d3980 con 0x7fadfc0cf710 2023-10-16T14:07:39.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fae00f47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc0d3c60 con 0x7fadfc0cfaf0 2023-10-16T14:07:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fae00f47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc0ce630 con 0x7fadfc0b3420 2023-10-16T14:07:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fadfa59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fadfc0cf710 0x7fadfc0b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39654/0 (socket says 172.21.15.60:39654) 2023-10-16T14:07:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fadfa59c640 1 -- 172.21.15.60:0/1700664089 learned_addr learned my addr 172.21.15.60:0/1700664089 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fadfad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fadfc0cfaf0 0x7fadfc0b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:38.999+0000 7fadf3fff640 1 --2- 172.21.15.60:0/1700664089 >> v2:172.21.15.144:3300/0 conn(0x7fadfc0b3420 0x7fadfc0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.000+0000 7fadfad9d640 1 -- 172.21.15.60:0/1700664089 >> v2:172.21.15.144:3300/0 conn(0x7fadfc0b3420 msgr2=0x7fadfc0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.000+0000 7fadfad9d640 1 --2- 172.21.15.60:0/1700664089 >> v2:172.21.15.144:3300/0 conn(0x7fadfc0b3420 0x7fadfc0b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.000+0000 7fadfad9d640 1 -- 172.21.15.60:0/1700664089 >> v1:172.21.15.60:6789/0 conn(0x7fadfc0cf710 legacy=0x7fadfc0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.000+0000 7fadfad9d640 1 -- 172.21.15.60:0/1700664089 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc0ce910 con 0x7fadfc0cfaf0 2023-10-16T14:07:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.000+0000 7fadf9d9b640 1 -- 172.21.15.60:0/1700664089 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905750667 0 0) 0x7fadfc0d3980 con 0x7fadfc0cf710 2023-10-16T14:07:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.001+0000 7fadfad9d640 1 --2- 172.21.15.60:0/1700664089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cfaf0 0x7fadfc0b6200 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fadfc08d210 tx=0x7fadf402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd547590b296a26f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.001+0000 7fadf9d9b640 1 -- 172.21.15.60:0/1700664089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf402cbf0 con 0x7fadfc0cfaf0 2023-10-16T14:07:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.001+0000 7fadf9d9b640 1 -- 172.21.15.60:0/1700664089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf402cd50 con 0x7fadfc0cfaf0 2023-10-16T14:07:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.001+0000 7fadf9d9b640 1 -- 172.21.15.60:0/1700664089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4034660 con 0x7fadfc0cfaf0 2023-10-16T14:07:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 -- 172.21.15.60:0/1700664089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cfaf0 msgr2=0x7fadfc0b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 --2- 172.21.15.60:0/1700664089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cfaf0 0x7fadfc0b6200 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fadfc08d210 tx=0x7fadf402be40 comp rx=0 tx=0).stop 2023-10-16T14:07:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 -- 172.21.15.60:0/1700664089 shutdown_connections 2023-10-16T14:07:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 --2- 172.21.15.60:0/1700664089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cfaf0 0x7fadfc0b6200 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 --2- 172.21.15.60:0/1700664089 >> v2:172.21.15.144:3300/0 conn(0x7fadfc0b3420 0x7fadfc0b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 -- 172.21.15.60:0/1700664089 >> 172.21.15.60:0/1700664089 conn(0x7fadfc0d2cc0 msgr2=0x7fadfc0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 -- 172.21.15.60:0/1700664089 shutdown_connections 2023-10-16T14:07:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.002+0000 7fae00f47640 1 -- 172.21.15.60:0/1700664089 wait complete. 2023-10-16T14:07:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.004+0000 7fae00f47640 1 Processor -- start 2023-10-16T14:07:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.004+0000 7fae00f47640 1 -- start start 2023-10-16T14:07:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fae00f47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadfc0b3420 0x7fadfc15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fae00f47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 0x7fadfc0cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fae00f47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fadfc0cfaf0 0x7fadfc159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fae00f47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc15bf60 con 0x7fadfc0cf710 2023-10-16T14:07:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fae00f47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc15b350 con 0x7fadfc0b3420 2023-10-16T14:07:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fae00f47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc15b630 con 0x7fadfc0cfaf0 2023-10-16T14:07:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fadf3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 0x7fadfc0cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fadf3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 0x7fadfc0cb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48836/0 (socket says 172.21.15.60:48836) 2023-10-16T14:07:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fadf3fff640 1 -- 172.21.15.60:0/2868808702 learned_addr learned my addr 172.21.15.60:0/2868808702 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fadfa59c640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadfc0b3420 0x7fadfc15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.005+0000 7fadfad9d640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fadfc0cfaf0 0x7fadfc159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf3fff640 1 -- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fadfc0cfaf0 msgr2=0x7fadfc159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf3fff640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fadfc0cfaf0 0x7fadfc159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf3fff640 1 -- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadfc0b3420 msgr2=0x7fadfc15ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf3fff640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadfc0b3420 0x7fadfc15ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf3fff640 1 -- 172.21.15.60:0/2868808702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4009a70 con 0x7fadfc0cf710 2023-10-16T14:07:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadfad9d640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fadfc0cfaf0 0x7fadfc159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:07:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf3fff640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 0x7fadfc0cb8b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7fade400e870 tx=0x7fade400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade40094f0 con 0x7fadfc0cf710 2023-10-16T14:07:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade400c940 con 0x7fadfc0cf710 2023-10-16T14:07:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc15b910 con 0x7fadfc0cf710 2023-10-16T14:07:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.006+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade400ccf0 con 0x7fadfc0cf710 2023-10-16T14:07:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.007+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc0cbdf0 con 0x7fadfc0cf710 2023-10-16T14:07:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.008+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc0b8340 con 0x7fadfc0cf710 2023-10-16T14:07:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.009+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fade4013470 con 0x7fadfc0cf710 2023-10-16T14:07:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.010+0000 7fadf2ffd640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7faddc06afc0 0x7faddc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.010+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5975+0+0 (secure 0 0 0) 0x7fade40898e0 con 0x7fadfc0cf710 2023-10-16T14:07:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.010+0000 7fadfa59c640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7faddc06afc0 0x7faddc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.011+0000 7fadfa59c640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7faddc06afc0 0x7faddc06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fadec0017b0 tx=0x7fadec009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.014+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fade4054810 con 0x7fadfc0cf710 2023-10-16T14:07:39.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.221+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fadfc0b4160 con 0x7fadfc0cf710 2023-10-16T14:07:39.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.685+0000 7fadf2ffd640 1 -- 172.21.15.60:0/2868808702 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v207) v1 ==== 97+0+0 (secure 0 0 0) 0x7fade4021080 con 0x7fadfc0cf710 2023-10-16T14:07:39.689 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T14:07:39.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7faddc06afc0 msgr2=0x7faddc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7faddc06afc0 0x7faddc06d480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fadec0017b0 tx=0x7fadec009040 comp rx=0 tx=0).stop 2023-10-16T14:07:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 msgr2=0x7fadfc0cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 0x7fadfc0cb8b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7fade400e870 tx=0x7fade400ed30 comp rx=0 tx=0).stop 2023-10-16T14:07:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 shutdown_connections 2023-10-16T14:07:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fadfc0cfaf0 0x7fadfc159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7faddc06afc0 0x7faddc06d480 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0cf710 0x7fadfc0cb8b0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 --2- 172.21.15.60:0/2868808702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fadfc0b3420 0x7fadfc15ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 >> 172.21.15.60:0/2868808702 conn(0x7fadfc0d2cc0 msgr2=0x7fadfc0cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:39.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 shutdown_connections 2023-10-16T14:07:39.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.689+0000 7fae00f47640 1 -- 172.21.15.60:0/2868808702 wait complete. 2023-10-16T14:07:39.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:07:39.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.938+0000 7f351ed97640 1 Processor -- start 2023-10-16T14:07:39.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.938+0000 7f351ed97640 1 -- start start 2023-10-16T14:07:39.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351ed97640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f351815ae80 0x7f351815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351ed97640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35180ce390 0x7f3518070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351ed97640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351815ed60 con 0x7f351815aaa0 2023-10-16T14:07:39.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351ed97640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351806b810 con 0x7f35180ce390 2023-10-16T14:07:39.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351ed97640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180707e0 con 0x7f351815ae80 2023-10-16T14:07:39.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351d334640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35180ce390 0x7f3518070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351cb33640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f351815aaa0 0x7f35180cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39658/0 (socket says 172.21.15.60:39658) 2023-10-16T14:07:39.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351d334640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35180ce390 0x7f3518070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48850/0 (socket says 172.21.15.60:48850) 2023-10-16T14:07:39.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f351cb33640 1 -- 172.21.15.60:0/3288709678 learned_addr learned my addr 172.21.15.60:0/3288709678 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:39.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.940+0000 7f3517fff640 1 --2- 172.21.15.60:0/3288709678 >> v2:172.21.15.144:3300/0 conn(0x7f351815ae80 0x7f351815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.941+0000 7f351d334640 1 -- 172.21.15.60:0/3288709678 >> v2:172.21.15.144:3300/0 conn(0x7f351815ae80 msgr2=0x7f351815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:39.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.941+0000 7f351d334640 1 --2- 172.21.15.60:0/3288709678 >> v2:172.21.15.144:3300/0 conn(0x7f351815ae80 0x7f351815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.941+0000 7f351d334640 1 -- 172.21.15.60:0/3288709678 >> v1:172.21.15.60:6789/0 conn(0x7f351815aaa0 legacy=0x7f35180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.941+0000 7f351d334640 1 -- 172.21.15.60:0/3288709678 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351815ef90 con 0x7f35180ce390 2023-10-16T14:07:39.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.941+0000 7f3517fff640 1 --2- 172.21.15.60:0/3288709678 >> v2:172.21.15.144:3300/0 conn(0x7f351815ae80 0x7f351815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:39.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.941+0000 7f35177fe640 1 -- 172.21.15.60:0/3288709678 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256796259 0 0) 0x7f351815ed60 con 0x7f351815aaa0 2023-10-16T14:07:39.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.942+0000 7f351d334640 1 --2- 172.21.15.60:0/3288709678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35180ce390 0x7f3518070f50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f35180c59c0 tx=0x7f351002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3268ed3246ab63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:39.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.942+0000 7f35177fe640 1 -- 172.21.15.60:0/3288709678 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351002cc80 con 0x7f35180ce390 2023-10-16T14:07:39.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.942+0000 7f35177fe640 1 -- 172.21.15.60:0/3288709678 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351002cde0 con 0x7f35180ce390 2023-10-16T14:07:39.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.942+0000 7f35177fe640 1 -- 172.21.15.60:0/3288709678 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510034660 con 0x7f35180ce390 2023-10-16T14:07:39.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 -- 172.21.15.60:0/3288709678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35180ce390 msgr2=0x7f3518070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 --2- 172.21.15.60:0/3288709678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35180ce390 0x7f3518070f50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f35180c59c0 tx=0x7f351002bed0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 -- 172.21.15.60:0/3288709678 shutdown_connections 2023-10-16T14:07:39.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 --2- 172.21.15.60:0/3288709678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35180ce390 0x7f3518070f50 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 --2- 172.21.15.60:0/3288709678 >> v2:172.21.15.144:3300/0 conn(0x7f351815ae80 0x7f351815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 -- 172.21.15.60:0/3288709678 >> 172.21.15.60:0/3288709678 conn(0x7f35180cb410 msgr2=0x7f35180cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:39.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 -- 172.21.15.60:0/3288709678 shutdown_connections 2023-10-16T14:07:39.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.943+0000 7f351ed97640 1 -- 172.21.15.60:0/3288709678 wait complete. 2023-10-16T14:07:39.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.945+0000 7f351ed97640 1 Processor -- start 2023-10-16T14:07:39.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.945+0000 7f351ed97640 1 -- start start 2023-10-16T14:07:39.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351ed97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35180ce390 0x7f35181f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351ed97640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351815aaa0 0x7f35180c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351ed97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 0x7f35180c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351ed97640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180c46a0 con 0x7f351815ae80 2023-10-16T14:07:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351ed97640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518149710 con 0x7f351815aaa0 2023-10-16T14:07:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351ed97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35180c3a90 con 0x7f35180ce390 2023-10-16T14:07:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351d334640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 0x7f35180c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351d334640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 0x7f35180c4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48864/0 (socket says 172.21.15.60:48864) 2023-10-16T14:07:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351d334640 1 -- 172.21.15.60:0/3782639603 learned_addr learned my addr 172.21.15.60:0/3782639603 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f351cb33640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35180ce390 0x7f35181f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.946+0000 7f3517fff640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351815aaa0 0x7f35180c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351d334640 1 -- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35180ce390 msgr2=0x7f35181f5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351d334640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35180ce390 0x7f35181f5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351d334640 1 -- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351815aaa0 msgr2=0x7f35180c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351d334640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351815aaa0 0x7f35180c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351d334640 1 -- 172.21.15.60:0/3782639603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510009a70 con 0x7f351815ae80 2023-10-16T14:07:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351cb33640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35180ce390 0x7f35181f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351d334640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 0x7f35180c4b30 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f3510037040 tx=0x7f3510004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510040650 con 0x7f351815ae80 2023-10-16T14:07:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35100407b0 con 0x7f351815ae80 2023-10-16T14:07:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510040b60 con 0x7f351815ae80 2023-10-16T14:07:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.947+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35180c3d70 con 0x7f351815ae80 2023-10-16T14:07:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.948+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35180c4330 con 0x7f351815ae80 2023-10-16T14:07:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.949+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35180c4850 con 0x7f351815ae80 2023-10-16T14:07:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.950+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3510040cc0 con 0x7f351815ae80 2023-10-16T14:07:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.951+0000 7f35157fa640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f34fc06afc0 0x7f34fc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.951+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f35100abfe0 con 0x7f351815ae80 2023-10-16T14:07:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.951+0000 7f351cb33640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f34fc06afc0 0x7f34fc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.953+0000 7f351cb33640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f34fc06afc0 0x7f34fc06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f350c000f60 tx=0x7f350c009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:39.955+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35180c4850 con 0x7f351815ae80 2023-10-16T14:07:40.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.220+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f351804e160 con 0x7f351815ae80 2023-10-16T14:07:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.222+0000 7f35157fa640 1 -- 172.21.15.60:0/3782639603 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3510077390 con 0x7f351815ae80 2023-10-16T14:07:40.225 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:40.225 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:07:40.225 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T14:07:40.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.226 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T14:07:40.226 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T14:07:40.226 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:07:40.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:07:40.227 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.227 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.229 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T14:07:40.229 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T14:07:40.230 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.230 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.231 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:07:40.231 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:07:40.232 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.233 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T14:07:40.233 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T14:07:40.234 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.234 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.234 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:07:40.235 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:07:40.235 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.236 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T14:07:40.236 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T14:07:40.236 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.236 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.236 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:07:40.237 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:07:40.237 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.238 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T14:07:40.238 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T14:07:40.238 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.239 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.239 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.239 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.239 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.240 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T14:07:40.240 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T14:07:40.240 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.240 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.241 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.241 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.241 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T14:07:40.242 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T14:07:40.242 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.242 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.242 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.242 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T14:07:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T14:07:40.244 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.244 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.244 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.245 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.246 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T14:07:40.246 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T14:07:40.246 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.247 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.247 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.250 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T14:07:40.250 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T14:07:40.250 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.251 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.251 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.251 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T14:07:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T14:07:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T14:07:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T14:07:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:07:40.255 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:07:40.255 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.256 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T14:07:40.256 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T14:07:40.256 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.256 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.257 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T14:07:40.257 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:07:40.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.257 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T14:07:40.258 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T14:07:40.258 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:07:40.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:07:40.259 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:07:40.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.259 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T14:07:40.260 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T14:07:40.260 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:07:40.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:07:40.261 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:07:40.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.261 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T14:07:40.262 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T14:07:40.262 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.262 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:07:40.262 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:07:40.262 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:07:40.263 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:07:40.263 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.263 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T14:07:40.264 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T14:07:40.264 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.264 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:07:40.264 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:07:40.264 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:07:40.265 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.265 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T14:07:40.266 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T14:07:40.266 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.266 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:07:40.267 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:07:40.267 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:07:40.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:40.268 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T14:07:40.268 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T14:07:40.268 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:07:40.268 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:07:40.268 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:07:40.269 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:07:40.269 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.269 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:40.269 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T14:07:40.269 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:07:40.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f34fc06afc0 msgr2=0x7f34fc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f34fc06afc0 0x7f34fc06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f350c000f60 tx=0x7f350c009040 comp rx=0 tx=0).stop 2023-10-16T14:07:40.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 msgr2=0x7f35180c4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 0x7f35180c4b30 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f3510037040 tx=0x7f3510004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:40.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 shutdown_connections 2023-10-16T14:07:40.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f34fc06afc0 0x7f34fc06d480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351815ae80 0x7f35180c4b30 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351815aaa0 0x7f35180c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 --2- 172.21.15.60:0/3782639603 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f35180ce390 0x7f35181f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.224+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 >> 172.21.15.60:0/3782639603 conn(0x7f35180cb410 msgr2=0x7f351815a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:40.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.225+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 shutdown_connections 2023-10-16T14:07:40.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.225+0000 7f351ed97640 1 -- 172.21.15.60:0/3782639603 wait complete. 2023-10-16T14:07:40.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.505+0000 7f985718a640 1 Processor -- start 2023-10-16T14:07:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.505+0000 7f985718a640 1 -- start start 2023-10-16T14:07:40.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f985718a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98500b0b20 0x7f9850164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f985718a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9850071450 con 0x7f98500adb50 2023-10-16T14:07:40.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f985718a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98500cc600 con 0x7f9850070fd0 2023-10-16T14:07:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f985718a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98500cc8e0 con 0x7f98500b0b20 2023-10-16T14:07:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f9854f26640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f98500adb50 0x7f985006f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39660/0 (socket says 172.21.15.60:39660) 2023-10-16T14:07:40.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f9854f26640 1 -- 172.21.15.60:0/1653180532 learned_addr learned my addr 172.21.15.60:0/1653180532 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.507+0000 7f9855727640 1 --2- 172.21.15.60:0/1653180532 >> v2:172.21.15.60:3300/0 conn(0x7f98500b0b20 0x7f9850164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.508+0000 7f984f7fe640 1 -- 172.21.15.60:0/1653180532 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462874120 0 0) 0x7f9850071450 con 0x7f98500adb50 2023-10-16T14:07:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.508+0000 7f984f7fe640 1 -- 172.21.15.60:0/1653180532 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9840003520 con 0x7f98500adb50 2023-10-16T14:07:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.508+0000 7f9855727640 1 -- 172.21.15.60:0/1653180532 >> v1:172.21.15.144:6789/0 conn(0x7f9850070fd0 legacy=0x7f98500b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.508+0000 7f9855727640 1 -- 172.21.15.60:0/1653180532 >> v1:172.21.15.60:6789/0 conn(0x7f98500adb50 legacy=0x7f985006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.508+0000 7f9855727640 1 -- 172.21.15.60:0/1653180532 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500ccbc0 con 0x7f98500b0b20 2023-10-16T14:07:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.509+0000 7f9855727640 1 --2- 172.21.15.60:0/1653180532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500b0b20 0x7f9850164f60 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f9848009d90 tx=0x7f984802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7c08c0d4570250 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.509+0000 7f984f7fe640 1 -- 172.21.15.60:0/1653180532 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984802cd00 con 0x7f98500b0b20 2023-10-16T14:07:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.509+0000 7f984f7fe640 1 -- 172.21.15.60:0/1653180532 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984802ce60 con 0x7f98500b0b20 2023-10-16T14:07:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.509+0000 7f984f7fe640 1 -- 172.21.15.60:0/1653180532 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848034660 con 0x7f98500b0b20 2023-10-16T14:07:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 -- 172.21.15.60:0/1653180532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500b0b20 msgr2=0x7f9850164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 --2- 172.21.15.60:0/1653180532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500b0b20 0x7f9850164f60 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f9848009d90 tx=0x7f984802bf50 comp rx=0 tx=0).stop 2023-10-16T14:07:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 -- 172.21.15.60:0/1653180532 shutdown_connections 2023-10-16T14:07:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 --2- 172.21.15.60:0/1653180532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500b0b20 0x7f9850164f60 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 -- 172.21.15.60:0/1653180532 >> 172.21.15.60:0/1653180532 conn(0x7f98500cb410 msgr2=0x7f98500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 -- 172.21.15.60:0/1653180532 shutdown_connections 2023-10-16T14:07:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.510+0000 7f985718a640 1 -- 172.21.15.60:0/1653180532 wait complete. 2023-10-16T14:07:40.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.512+0000 7f985718a640 1 Processor -- start 2023-10-16T14:07:40.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.512+0000 7f985718a640 1 -- start start 2023-10-16T14:07:40.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.513+0000 7f985718a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9850070fd0 0x7f98501f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.513+0000 7f985718a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 0x7f98500aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.513+0000 7f985718a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98500b0b20 0x7f9850162400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.513+0000 7f985718a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9850155970 con 0x7f98500adb50 2023-10-16T14:07:40.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.513+0000 7f985718a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500af110 con 0x7f9850070fd0 2023-10-16T14:07:40.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.513+0000 7f985718a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9850162940 con 0x7f98500b0b20 2023-10-16T14:07:40.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 0x7f98500aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 0x7f98500aebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48876/0 (socket says 172.21.15.60:48876) 2023-10-16T14:07:40.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 -- 172.21.15.60:0/465484039 learned_addr learned my addr 172.21.15.60:0/465484039 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:40.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f9854f26640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9850070fd0 0x7f98501f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 -- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98500b0b20 msgr2=0x7f9850162400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f9855727640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98500b0b20 0x7f9850162400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98500b0b20 0x7f9850162400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 -- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9850070fd0 msgr2=0x7f98501f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9850070fd0 0x7f98501f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 -- 172.21.15.60:0/465484039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9848009a70 con 0x7f98500adb50 2023-10-16T14:07:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.514+0000 7f984ffff640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 0x7f98500aebd0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f983c00c400 tx=0x7f983c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:40.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.515+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c0052d0 con 0x7f98500adb50 2023-10-16T14:07:40.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.515+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c005430 con 0x7f98500adb50 2023-10-16T14:07:40.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.515+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9850162b70 con 0x7f98500adb50 2023-10-16T14:07:40.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.515+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c003830 con 0x7f98500adb50 2023-10-16T14:07:40.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.515+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98500b2960 con 0x7f98500adb50 2023-10-16T14:07:40.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.516+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985004e160 con 0x7f98500adb50 2023-10-16T14:07:40.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.517+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f983c003990 con 0x7f98500adb50 2023-10-16T14:07:40.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.518+0000 7f984d7fa640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f983406af90 0x7f983406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.518+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f983c0889f0 con 0x7f98500adb50 2023-10-16T14:07:40.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.519+0000 7f9854f26640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f983406af90 0x7f983406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.520+0000 7f9854f26640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f983406af90 0x7f983406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98440011e0 tx=0x7f9844009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:40.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.523+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f983c053c40 con 0x7f98500adb50 2023-10-16T14:07:40.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.725+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98500b4910 con 0x7f98500adb50 2023-10-16T14:07:40.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.727+0000 7f984d7fa640 1 -- 172.21.15.60:0/465484039 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+12855 (secure 0 0 0) 0x7f983c017090 con 0x7f98500adb50 2023-10-16T14:07:40.730 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:40.730 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":209,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:39.699545+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:40.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f983406af90 msgr2=0x7f983406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f983406af90 0x7f983406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98440011e0 tx=0x7f9844009040 comp rx=0 tx=0).stop 2023-10-16T14:07:40.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 msgr2=0x7f98500aebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 0x7f98500aebd0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f983c00c400 tx=0x7f983c00c8c0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 shutdown_connections 2023-10-16T14:07:40.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98500b0b20 0x7f9850162400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f983406af90 0x7f983406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98500adb50 0x7f98500aebd0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 --2- 172.21.15.60:0/465484039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9850070fd0 0x7f98501f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.730+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 >> 172.21.15.60:0/465484039 conn(0x7f98500cb410 msgr2=0x7f985006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:40.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.731+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 shutdown_connections 2023-10-16T14:07:40.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.731+0000 7f985718a640 1 -- 172.21.15.60:0/465484039 wait complete. 2023-10-16T14:07:40.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:07:40.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:07:40.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.984+0000 7fec48a43640 1 Processor -- start 2023-10-16T14:07:40.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.985+0000 7fec48a43640 1 -- start start 2023-10-16T14:07:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.986+0000 7fec48a43640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec440cd260 0x7fec440ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.986+0000 7fec48a43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec440c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.986+0000 7fec48a43640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440ce480 con 0x7fec4415cc00 2023-10-16T14:07:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.986+0000 7fec48a43640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec440cd7b0 con 0x7fec440cd260 2023-10-16T14:07:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.986+0000 7fec48a43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec440cda90 con 0x7fec440ad670 2023-10-16T14:07:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.986+0000 7fec4259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec440cd260 0x7fec440ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec440cd260 0x7fec440ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48892/0 (socket says 172.21.15.60:48892) 2023-10-16T14:07:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4259c640 1 -- 172.21.15.60:0/1186857013 learned_addr learned my addr 172.21.15.60:0/1186857013 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4359e640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec440c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4259c640 1 -- 172.21.15.60:0/1186857013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 msgr2=0x7fec440c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4259c640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec440c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4259c640 1 -- 172.21.15.60:0/1186857013 >> v1:172.21.15.144:6789/0 conn(0x7fec4415cc00 legacy=0x7fec440ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4259c640 1 -- 172.21.15.60:0/1186857013 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440cdd70 con 0x7fec440cd260 2023-10-16T14:07:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.987+0000 7fec4359e640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec440c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.988+0000 7fec4259c640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec440cd260 0x7fec440ad130 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fec380098a0 tx=0x7fec3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6137e41fb7e884 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.988+0000 7fec41d9b640 1 -- 172.21.15.60:0/1186857013 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38031640 con 0x7fec440cd260 2023-10-16T14:07:40.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec41d9b640 1 -- 172.21.15.60:0/1186857013 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec380317a0 con 0x7fec440cd260 2023-10-16T14:07:40.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec41d9b640 1 -- 172.21.15.60:0/1186857013 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38031b30 con 0x7fec440cd260 2023-10-16T14:07:40.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 -- 172.21.15.60:0/1186857013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec440cd260 msgr2=0x7fec440ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:40.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec440cd260 0x7fec440ad130 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fec380098a0 tx=0x7fec3802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 -- 172.21.15.60:0/1186857013 shutdown_connections 2023-10-16T14:07:40.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec440c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 --2- 172.21.15.60:0/1186857013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec440cd260 0x7fec440ad130 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:40.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 -- 172.21.15.60:0/1186857013 >> 172.21.15.60:0/1186857013 conn(0x7fec440b0170 msgr2=0x7fec440b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:40.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 -- 172.21.15.60:0/1186857013 shutdown_connections 2023-10-16T14:07:40.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.989+0000 7fec48a43640 1 -- 172.21.15.60:0/1186857013 wait complete. 2023-10-16T14:07:40.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.991+0000 7fec48a43640 1 Processor -- start 2023-10-16T14:07:40.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.991+0000 7fec48a43640 1 -- start start 2023-10-16T14:07:40.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.992+0000 7fec48a43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec441f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.992+0000 7fec48a43640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec440cd260 0x7fec440c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.992+0000 7fec48a43640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 0x7fec4415ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:40.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.992+0000 7fec48a43640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440b13b0 con 0x7fec4415cc00 2023-10-16T14:07:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.992+0000 7fec48a43640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4406b730 con 0x7fec440cd260 2023-10-16T14:07:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.992+0000 7fec48a43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4406ba10 con 0x7fec440ad670 2023-10-16T14:07:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 0x7fec4415ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 0x7fec4415ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48900/0 (socket says 172.21.15.60:48900) 2023-10-16T14:07:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 -- 172.21.15.60:0/2520432432 learned_addr learned my addr 172.21.15.60:0/2520432432 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:41.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec42d9d640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec441f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4259c640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec440cd260 0x7fec440c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 -- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 msgr2=0x7fec441f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec441f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 -- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec440cd260 msgr2=0x7fec440c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec440cd260 0x7fec440c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.993+0000 7fec4359e640 1 -- 172.21.15.60:0/2520432432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380094d0 con 0x7fec4415cc00 2023-10-16T14:07:41.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.994+0000 7fec4359e640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 0x7fec4415ab00 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00cef0 tx=0x7fec3c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.994+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c00d840 con 0x7fec4415cc00 2023-10-16T14:07:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.994+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c00d9c0 con 0x7fec4415cc00 2023-10-16T14:07:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.994+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3c00dd70 con 0x7fec4415cc00 2023-10-16T14:07:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.994+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4406bcf0 con 0x7fec4415cc00 2023-10-16T14:07:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.994+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec440c44c0 con 0x7fec4415cc00 2023-10-16T14:07:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.995+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4406bf30 con 0x7fec4415cc00 2023-10-16T14:07:41.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.996+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec3c01e3b0 con 0x7fec4415cc00 2023-10-16T14:07:41.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.997+0000 7fec2b7fe640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fec3006af90 0x7fec3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.997+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec3c089ec0 con 0x7fec4415cc00 2023-10-16T14:07:41.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.998+0000 7fec42d9d640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fec3006af90 0x7fec3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:40.999+0000 7fec42d9d640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fec3006af90 0x7fec3006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec34006fd0 tx=0x7fec34008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.002+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec4406bf30 con 0x7fec4415cc00 2023-10-16T14:07:41.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.198+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fec440b0ea0 con 0x7fec4415cc00 2023-10-16T14:07:41.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.199+0000 7fec2b7fe640 1 -- 172.21.15.60:0/2520432432 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) v1 ==== 93+0+10 (secure 0 0 0) 0x7fec440b0ea0 con 0x7fec4415cc00 2023-10-16T14:07:41.202 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:07:41.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fec3006af90 msgr2=0x7fec3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fec3006af90 0x7fec3006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec34006fd0 tx=0x7fec34008040 comp rx=0 tx=0).stop 2023-10-16T14:07:41.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 msgr2=0x7fec4415ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 0x7fec4415ab00 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00cef0 tx=0x7fec3c00a710 comp rx=0 tx=0).stop 2023-10-16T14:07:41.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 shutdown_connections 2023-10-16T14:07:41.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fec3006af90 0x7fec3006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4415cc00 0x7fec4415ab00 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec440cd260 0x7fec440c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 --2- 172.21.15.60:0/2520432432 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fec440ad670 0x7fec441f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.201+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 >> 172.21.15.60:0/2520432432 conn(0x7fec440b0170 msgr2=0x7fec44161140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:41.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.202+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 shutdown_connections 2023-10-16T14:07:41.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.202+0000 7fec48a43640 1 -- 172.21.15.60:0/2520432432 wait complete. 2023-10-16T14:07:41.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:07:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.450+0000 7fdc34239640 1 Processor -- start 2023-10-16T14:07:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.450+0000 7fdc34239640 1 -- start start 2023-10-16T14:07:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc34239640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdc2c15ae80 0x7fdc2c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc34239640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc2c0ce390 0x7fdc2c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc34239640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c15ed60 con 0x7fdc2c15aaa0 2023-10-16T14:07:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc34239640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c06b810 con 0x7fdc2c0ce390 2023-10-16T14:07:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc34239640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c0707e0 con 0x7fdc2c15ae80 2023-10-16T14:07:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc31fd5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdc2c15aaa0 0x7fdc2c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39670/0 (socket says 172.21.15.60:39670) 2023-10-16T14:07:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc31fd5640 1 -- 172.21.15.60:0/3879608117 learned_addr learned my addr 172.21.15.60:0/3879608117 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.452+0000 7fdc327d6640 1 --2- 172.21.15.60:0/3879608117 >> v2:172.21.15.60:3300/0 conn(0x7fdc2c0ce390 0x7fdc2c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc317d4640 1 --2- 172.21.15.60:0/3879608117 >> v2:172.21.15.144:3300/0 conn(0x7fdc2c15ae80 0x7fdc2c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc327d6640 1 -- 172.21.15.60:0/3879608117 >> v2:172.21.15.144:3300/0 conn(0x7fdc2c15ae80 msgr2=0x7fdc2c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc327d6640 1 --2- 172.21.15.60:0/3879608117 >> v2:172.21.15.144:3300/0 conn(0x7fdc2c15ae80 0x7fdc2c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc327d6640 1 -- 172.21.15.60:0/3879608117 >> v1:172.21.15.60:6789/0 conn(0x7fdc2c15aaa0 legacy=0x7fdc2c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc327d6640 1 -- 172.21.15.60:0/3879608117 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c15ef90 con 0x7fdc2c0ce390 2023-10-16T14:07:41.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc30fd3640 1 -- 172.21.15.60:0/3879608117 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742933326 0 0) 0x7fdc2c15ed60 con 0x7fdc2c15aaa0 2023-10-16T14:07:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.453+0000 7fdc317d4640 1 --2- 172.21.15.60:0/3879608117 >> v2:172.21.15.144:3300/0 conn(0x7fdc2c15ae80 0x7fdc2c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:41.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.454+0000 7fdc327d6640 1 --2- 172.21.15.60:0/3879608117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c0ce390 0x7fdc2c070f50 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0c59c0 tx=0x7fdc2402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f280bc0ff978a66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.454+0000 7fdc30fd3640 1 -- 172.21.15.60:0/3879608117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2402cc70 con 0x7fdc2c0ce390 2023-10-16T14:07:41.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc30fd3640 1 -- 172.21.15.60:0/3879608117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2402cdd0 con 0x7fdc2c0ce390 2023-10-16T14:07:41.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc30fd3640 1 -- 172.21.15.60:0/3879608117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24034660 con 0x7fdc2c0ce390 2023-10-16T14:07:41.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 -- 172.21.15.60:0/3879608117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c0ce390 msgr2=0x7fdc2c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 --2- 172.21.15.60:0/3879608117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c0ce390 0x7fdc2c070f50 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0c59c0 tx=0x7fdc2402bec0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 -- 172.21.15.60:0/3879608117 shutdown_connections 2023-10-16T14:07:41.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 --2- 172.21.15.60:0/3879608117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c0ce390 0x7fdc2c070f50 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 --2- 172.21.15.60:0/3879608117 >> v2:172.21.15.144:3300/0 conn(0x7fdc2c15ae80 0x7fdc2c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 -- 172.21.15.60:0/3879608117 >> 172.21.15.60:0/3879608117 conn(0x7fdc2c0cb410 msgr2=0x7fdc2c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:41.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 -- 172.21.15.60:0/3879608117 shutdown_connections 2023-10-16T14:07:41.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.455+0000 7fdc34239640 1 -- 172.21.15.60:0/3879608117 wait complete. 2023-10-16T14:07:41.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.457+0000 7fdc34239640 1 Processor -- start 2023-10-16T14:07:41.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.457+0000 7fdc34239640 1 -- start start 2023-10-16T14:07:41.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc34239640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c0ce390 0x7fdc2c0ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc34239640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c15aaa0 0x7fdc2c0c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc34239640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 0x7fdc2c1f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc34239640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0c35a0 con 0x7fdc2c15ae80 2023-10-16T14:07:41.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc34239640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0ca9b0 con 0x7fdc2c15aaa0 2023-10-16T14:07:41.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc34239640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0c2990 con 0x7fdc2c0ce390 2023-10-16T14:07:41.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc327d6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 0x7fdc2c1f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc327d6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 0x7fdc2c1f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48912/0 (socket says 172.21.15.60:48912) 2023-10-16T14:07:41.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.458+0000 7fdc327d6640 1 -- 172.21.15.60:0/4181180548 learned_addr learned my addr 172.21.15.60:0/4181180548 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:41.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc317d4640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c15aaa0 0x7fdc2c0c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc31fd5640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c0ce390 0x7fdc2c0ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc327d6640 1 -- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c0ce390 msgr2=0x7fdc2c0ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc327d6640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c0ce390 0x7fdc2c0ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc327d6640 1 -- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c15aaa0 msgr2=0x7fdc2c0c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc327d6640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c15aaa0 0x7fdc2c0c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc327d6640 1 -- 172.21.15.60:0/4181180548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc24009a70 con 0x7fdc2c15ae80 2023-10-16T14:07:41.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc31fd5640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c0ce390 0x7fdc2c0ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:41.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.459+0000 7fdc327d6640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 0x7fdc2c1f2590 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fdc2402c4d0 tx=0x7fdc24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.460+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2403e610 con 0x7fdc2c15ae80 2023-10-16T14:07:41.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.460+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2403e770 con 0x7fdc2c15ae80 2023-10-16T14:07:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.460+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2403eb20 con 0x7fdc2c15ae80 2023-10-16T14:07:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.460+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c0c2c70 con 0x7fdc2c15ae80 2023-10-16T14:07:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.460+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c0c3230 con 0x7fdc2c15ae80 2023-10-16T14:07:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.461+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c0c3750 con 0x7fdc2c15ae80 2023-10-16T14:07:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.462+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdc2403b050 con 0x7fdc2c15ae80 2023-10-16T14:07:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.463+0000 7fdc227fc640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fdc1006afc0 0x7fdc1006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.463+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdc240ab850 con 0x7fdc2c15ae80 2023-10-16T14:07:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.464+0000 7fdc31fd5640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fdc1006afc0 0x7fdc1006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.465+0000 7fdc31fd5640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fdc1006afc0 0x7fdc1006d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc28000f60 tx=0x7fdc28009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.467+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc2c0c3750 con 0x7fdc2c15ae80 2023-10-16T14:07:41.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.665+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdc2c04e160 con 0x7fdc2c15ae80 2023-10-16T14:07:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.665+0000 7fdc227fc640 1 -- 172.21.15.60:0/4181180548 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v209) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdc2404d060 con 0x7fdc2c15ae80 2023-10-16T14:07:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:41.669 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:07:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:07:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:07:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:41.670 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:07:41.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fdc1006afc0 msgr2=0x7fdc1006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fdc1006afc0 0x7fdc1006d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc28000f60 tx=0x7fdc28009040 comp rx=0 tx=0).stop 2023-10-16T14:07:41.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 msgr2=0x7fdc2c1f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 0x7fdc2c1f2590 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fdc2402c4d0 tx=0x7fdc24004a40 comp rx=0 tx=0).stop 2023-10-16T14:07:41.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 shutdown_connections 2023-10-16T14:07:41.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fdc1006afc0 0x7fdc1006d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc2c15ae80 0x7fdc2c1f2590 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc2c15aaa0 0x7fdc2c0c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 --2- 172.21.15.60:0/4181180548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc2c0ce390 0x7fdc2c0ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 >> 172.21.15.60:0/4181180548 conn(0x7fdc2c0cb410 msgr2=0x7fdc2c0b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:41.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 shutdown_connections 2023-10-16T14:07:41.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.668+0000 7fdc34239640 1 -- 172.21.15.60:0/4181180548 wait complete. 2023-10-16T14:07:41.690 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:07:41.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:07:41.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.921+0000 7f4dddd89640 1 Processor -- start 2023-10-16T14:07:41.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.922+0000 7f4dddd89640 1 -- start start 2023-10-16T14:07:41.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.923+0000 7f4dddd89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4dd815ac10 0x7f4dd806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.923+0000 7f4dddd89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd80b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.923+0000 7f4dddd89640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd815b310 con 0x7f4dd80b0360 2023-10-16T14:07:41.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.923+0000 7f4dddd89640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd815f800 con 0x7f4dd815ac10 2023-10-16T14:07:41.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.923+0000 7f4dddd89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd80b0d30 con 0x7f4dd80cac20 2023-10-16T14:07:41.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.923+0000 7f4dd77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4dd815ac10 0x7f4dd806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4dd815ac10 0x7f4dd806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48922/0 (socket says 172.21.15.60:48922) 2023-10-16T14:07:41.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd77fe640 1 -- 172.21.15.60:0/2941580709 learned_addr learned my addr 172.21.15.60:0/2941580709 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:41.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd7fff640 1 --2- 172.21.15.60:0/2941580709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd80b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd77fe640 1 -- 172.21.15.60:0/2941580709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 msgr2=0x7f4dd80b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:41.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd77fe640 1 --2- 172.21.15.60:0/2941580709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd80b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd77fe640 1 -- 172.21.15.60:0/2941580709 >> v1:172.21.15.60:6789/0 conn(0x7f4dd80b0360 legacy=0x7f4dd80ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd77fe640 1 -- 172.21.15.60:0/2941580709 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd806f640 con 0x7f4dd815ac10 2023-10-16T14:07:41.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.924+0000 7f4dd6ffd640 1 -- 172.21.15.60:0/2941580709 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499056609 0 0) 0x7f4dd815b310 con 0x7f4dd80b0360 2023-10-16T14:07:41.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.925+0000 7f4dd77fe640 1 --2- 172.21.15.60:0/2941580709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806c730 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009e80 tx=0x7f4dc802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e8d5b370ab6725 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dd6ffd640 1 -- 172.21.15.60:0/2941580709 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc802ca50 con 0x7f4dd815ac10 2023-10-16T14:07:41.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dd6ffd640 1 -- 172.21.15.60:0/2941580709 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc802cbb0 con 0x7f4dd815ac10 2023-10-16T14:07:41.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dd6ffd640 1 -- 172.21.15.60:0/2941580709 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80343f0 con 0x7f4dd815ac10 2023-10-16T14:07:41.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 -- 172.21.15.60:0/2941580709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 msgr2=0x7f4dd806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2941580709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806c730 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009e80 tx=0x7f4dc802beb0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 -- 172.21.15.60:0/2941580709 shutdown_connections 2023-10-16T14:07:41.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2941580709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd80b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2941580709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806c730 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 -- 172.21.15.60:0/2941580709 >> 172.21.15.60:0/2941580709 conn(0x7f4dd80cb360 msgr2=0x7f4dd80cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 -- 172.21.15.60:0/2941580709 shutdown_connections 2023-10-16T14:07:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.926+0000 7f4dddd89640 1 -- 172.21.15.60:0/2941580709 wait complete. 2023-10-16T14:07:41.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.928+0000 7f4dddd89640 1 Processor -- start 2023-10-16T14:07:41.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.928+0000 7f4dddd89640 1 -- start start 2023-10-16T14:07:41.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.929+0000 7f4dddd89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd80b0360 0x7f4dd80c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.929+0000 7f4dddd89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd806b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.929+0000 7f4dddd89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.929+0000 7f4dddd89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd81f3230 con 0x7f4dd815ac10 2023-10-16T14:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.929+0000 7f4dddd89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd81f25d0 con 0x7f4dd80b0360 2023-10-16T14:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.929+0000 7f4dddd89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd81f2890 con 0x7f4dd80cac20 2023-10-16T14:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48928/0 (socket says 172.21.15.60:48928) 2023-10-16T14:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 -- 172.21.15.60:0/2356652902 learned_addr learned my addr 172.21.15.60:0/2356652902 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dceffd640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd806b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd77fe640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd80b0360 0x7f4dd80c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 -- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 msgr2=0x7f4dd806b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd806b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 -- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd80b0360 msgr2=0x7f4dd80c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd80b0360 0x7f4dd80c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 -- 172.21.15.60:0/2356652902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8009b30 con 0x7f4dd815ac10 2023-10-16T14:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.930+0000 7f4dd7fff640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806bb10 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000d800 tx=0x7f4dd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.931+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000b6c0 con 0x7f4dd815ac10 2023-10-16T14:07:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.931+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd000b840 con 0x7f4dd815ac10 2023-10-16T14:07:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.931+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000bbf0 con 0x7f4dd815ac10 2023-10-16T14:07:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.931+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd81f2b70 con 0x7f4dd815ac10 2023-10-16T14:07:41.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.931+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd80c4550 con 0x7f4dd815ac10 2023-10-16T14:07:41.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.932+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd81f3370 con 0x7f4dd815ac10 2023-10-16T14:07:41.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.933+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4dd000bd50 con 0x7f4dd815ac10 2023-10-16T14:07:41.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.934+0000 7f4dd4ff9640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4db806af90 0x7f4db806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:41.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.934+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4dd0089a70 con 0x7f4dd815ac10 2023-10-16T14:07:41.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.935+0000 7f4dd77fe640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4db806af90 0x7f4db806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:41.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.936+0000 7f4dd77fe640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4db806af90 0x7f4db806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8035000 tx=0x7f4dc8030000 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:41.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:41.939+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4dd81f3370 con 0x7f4dd815ac10 2023-10-16T14:07:42.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.139+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4dd81f3370 con 0x7f4dd815ac10 2023-10-16T14:07:42.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.753+0000 7f4dd4ff9640 1 -- 172.21.15.60:0/2356652902 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v210) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4dd81f3370 con 0x7f4dd815ac10 2023-10-16T14:07:42.756 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T14:07:42.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4db806af90 msgr2=0x7f4db806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:42.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4db806af90 0x7f4db806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8035000 tx=0x7f4dc8030000 comp rx=0 tx=0).stop 2023-10-16T14:07:42.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 msgr2=0x7f4dd806bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:42.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806bb10 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000d800 tx=0x7f4dd000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:07:42.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 shutdown_connections 2023-10-16T14:07:42.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4dd80cac20 0x7f4dd806b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:42.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f4db806af90 0x7f4db806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:42.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4dd815ac10 0x7f4dd806bb10 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:42.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 --2- 172.21.15.60:0/2356652902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4dd80b0360 0x7f4dd80c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:42.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 >> 172.21.15.60:0/2356652902 conn(0x7f4dd80cb360 msgr2=0x7f4dd80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:42.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 shutdown_connections 2023-10-16T14:07:42.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:42.757+0000 7f4dddd89640 1 -- 172.21.15.60:0/2356652902 wait complete. 2023-10-16T14:07:42.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:07:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.008+0000 7fc2e6b68640 1 Processor -- start 2023-10-16T14:07:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.008+0000 7fc2e6b68640 1 -- start start 2023-10-16T14:07:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.009+0000 7fc2e6b68640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2e015a970 0x7fc2e006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2e6b68640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e0070660 con 0x7fc2e015d430 2023-10-16T14:07:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2e6b68640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e0071530 con 0x7fc2e00cd880 2023-10-16T14:07:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2e6b68640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e015dc60 con 0x7fc2e015a970 2023-10-16T14:07:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2d7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc2e015d430 0x7fc2e0070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39676/0 (socket says 172.21.15.60:39676) 2023-10-16T14:07:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2d7fff640 1 -- 172.21.15.60:0/4063802284 learned_addr learned my addr 172.21.15.60:0/4063802284 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2e4904640 1 --2- 172.21.15.60:0/4063802284 >> v2:172.21.15.144:3300/0 conn(0x7fc2e015a970 0x7fc2e006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213029856 0 0) 0x7fc2e0070660 con 0x7fc2e015d430 2023-10-16T14:07:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.010+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c8003520 con 0x7fc2e015d430 2023-10-16T14:07:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109435725 0 0) 0x7fc2c8003520 con 0x7fc2e015d430 2023-10-16T14:07:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 >> v2:172.21.15.144:3300/0 conn(0x7fc2e015a970 msgr2=0x7fc2e006ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 --2- 172.21.15.60:0/4063802284 >> v2:172.21.15.144:3300/0 conn(0x7fc2e015a970 0x7fc2e006ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 >> v1:172.21.15.144:6789/0 conn(0x7fc2e00cd880 legacy=0x7fc2e0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e015df40 con 0x7fc2e015d430 2023-10-16T14:07:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc2d0001670 con 0x7fc2e015d430 2023-10-16T14:07:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 >> v1:172.21.15.60:6789/0 conn(0x7fc2e015d430 legacy=0x7fc2e0070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c8003970 0x7fc2c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c8005180 0x7fc2c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e015dc60 con 0x7fc2c80047c0 2023-10-16T14:07:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800c020 con 0x7fc2c8003970 2023-10-16T14:07:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c800c1e0 con 0x7fc2c8005180 2023-10-16T14:07:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.011+0000 7fc2d7fff640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2e4904640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c8003970 0x7fc2c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2d7fff640 1 -- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c8005180 msgr2=0x7fc2c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2d7fff640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2c8005180 0x7fc2c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2d7fff640 1 -- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c8003970 msgr2=0x7fc2c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2d7fff640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2c8003970 0x7fc2c8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2d7fff640 1 -- 172.21.15.60:0/4063802284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c8003520 con 0x7fc2c80047c0 2023-10-16T14:07:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.012+0000 7fc2e5105640 1 -- 172.21.15.60:0/4063802284 reap_dead start 2023-10-16T14:07:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.013+0000 7fc2d7fff640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2c8004c40 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c240 tx=0x7fc2c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e010dbc568e07e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.013+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0005ad0 con 0x7fc2c80047c0 2023-10-16T14:07:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.013+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d0005c30 con 0x7fc2c80047c0 2023-10-16T14:07:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.013+0000 7fc2d77fe640 1 -- 172.21.15.60:0/4063802284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d002caf0 con 0x7fc2c80047c0 2023-10-16T14:07:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 -- 172.21.15.60:0/4063802284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 msgr2=0x7fc2c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2c8004c40 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c240 tx=0x7fc2c80057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 -- 172.21.15.60:0/4063802284 shutdown_connections 2023-10-16T14:07:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/4063802284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2c8004c40 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 -- 172.21.15.60:0/4063802284 >> 172.21.15.60:0/4063802284 conn(0x7fc2e00cfaa0 msgr2=0x7fc2e00cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 -- 172.21.15.60:0/4063802284 shutdown_connections 2023-10-16T14:07:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.014+0000 7fc2e6b68640 1 -- 172.21.15.60:0/4063802284 wait complete. 2023-10-16T14:07:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.016+0000 7fc2e6b68640 1 Processor -- start 2023-10-16T14:07:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.016+0000 7fc2e6b68640 1 -- start start 2023-10-16T14:07:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e6b68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2e00cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e6b68640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e01e8830 0x7fc2e00d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e6b68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e0158670 0x7fc2e0158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e6b68640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e00c6b70 con 0x7fc2c80047c0 2023-10-16T14:07:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e6b68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e00c6dc0 con 0x7fc2e0158670 2023-10-16T14:07:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e6b68640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e00c70a0 con 0x7fc2e01e8830 2023-10-16T14:07:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.017+0000 7fc2e4904640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2e00cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2e00cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48944/0 (socket says 172.21.15.60:48944) 2023-10-16T14:07:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 -- 172.21.15.60:0/153476483 learned_addr learned my addr 172.21.15.60:0/153476483 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e5105640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e0158670 0x7fc2e0158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 -- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e01e8830 msgr2=0x7fc2e00d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e01e8830 0x7fc2e00d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 -- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e0158670 msgr2=0x7fc2e0158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e0158670 0x7fc2e0158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 -- 172.21.15.60:0/153476483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e00c7380 con 0x7fc2c80047c0 2023-10-16T14:07:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.018+0000 7fc2e4904640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2e00cfe80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc011530 tx=0x7fc2dc0119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.019+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc015870 con 0x7fc2c80047c0 2023-10-16T14:07:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.019+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc0159f0 con 0x7fc2c80047c0 2023-10-16T14:07:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.019+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc015da0 con 0x7fc2c80047c0 2023-10-16T14:07:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.019+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e00c7660 con 0x7fc2c80047c0 2023-10-16T14:07:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.019+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e0159040 con 0x7fc2c80047c0 2023-10-16T14:07:43.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.020+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e0159230 con 0x7fc2c80047c0 2023-10-16T14:07:43.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.021+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc2dc027420 con 0x7fc2c80047c0 2023-10-16T14:07:43.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.022+0000 7fc2d57fa640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc2cc06af90 0x7fc2cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.022+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc2dc089cd0 con 0x7fc2c80047c0 2023-10-16T14:07:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.023+0000 7fc2d7fff640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc2cc06af90 0x7fc2cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.024+0000 7fc2d7fff640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc2cc06af90 0x7fc2cc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0034000 tx=0x7fc2d0039040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.026+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc2dc054e30 con 0x7fc2c80047c0 2023-10-16T14:07:43.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.223+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc2e0159230 con 0x7fc2c80047c0 2023-10-16T14:07:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.760+0000 7fc2d57fa640 1 -- 172.21.15.60:0/153476483 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v211) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc2e0159230 con 0x7fc2c80047c0 2023-10-16T14:07:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T14:07:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.763+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc2cc06af90 msgr2=0x7fc2cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.763+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc2cc06af90 0x7fc2cc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0034000 tx=0x7fc2d0039040 comp rx=0 tx=0).stop 2023-10-16T14:07:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.763+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 msgr2=0x7fc2e00cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.763+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2e00cfe80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc011530 tx=0x7fc2dc0119f0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 shutdown_connections 2023-10-16T14:07:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc2e01e8830 0x7fc2e00d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fc2cc06af90 0x7fc2cc06d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e0158670 0x7fc2e0158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 --2- 172.21.15.60:0/153476483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c80047c0 0x7fc2e00cfe80 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:43.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 >> 172.21.15.60:0/153476483 conn(0x7fc2e00cfaa0 msgr2=0x7fc2e00c9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 shutdown_connections 2023-10-16T14:07:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.764+0000 7fc2e6b68640 1 -- 172.21.15.60:0/153476483 wait complete. 2023-10-16T14:07:43.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:07:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.991+0000 7fe30a664640 1 Processor -- start 2023-10-16T14:07:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.992+0000 7fe30a664640 1 -- start start 2023-10-16T14:07:44.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.993+0000 7fe30a664640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe3040706c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.993+0000 7fe30a664640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30406d7e0 con 0x7fe3040c9e10 2023-10-16T14:07:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.993+0000 7fe30a664640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30406fb00 con 0x7fe304159c60 2023-10-16T14:07:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.993+0000 7fe30a664640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040c8e90 con 0x7fe3040ad8b0 2023-10-16T14:07:44.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.994+0000 7fe3037fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3040c9e10 0x7fe30406ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39678/0 (socket says 172.21.15.60:39678) 2023-10-16T14:07:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.994+0000 7fe3037fe640 1 -- 172.21.15.60:0/3074523680 learned_addr learned my addr 172.21.15.60:0/3074523680 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.994+0000 7fe308c01640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe3040706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.994+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069679539 0 0) 0x7fe30406d7e0 con 0x7fe3040c9e10 2023-10-16T14:07:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.994+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec003520 con 0x7fe3040c9e10 2023-10-16T14:07:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641269902 0 0) 0x7fe2ec003520 con 0x7fe3040c9e10 2023-10-16T14:07:44.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 msgr2=0x7fe3040706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe3040706c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 >> v1:172.21.15.144:6789/0 conn(0x7fe304159c60 legacy=0x7fe3040c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30406b8c0 con 0x7fe3040c9e10 2023-10-16T14:07:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fe2f8001690 con 0x7fe3040c9e10 2023-10-16T14:07:44.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 >> v1:172.21.15.60:6789/0 conn(0x7fe3040c9e10 legacy=0x7fe30406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0039a0 0x7fe2ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0047c0 0x7fe2ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe2ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040c8e90 con 0x7fe2ec0039a0 2023-10-16T14:07:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec00baf0 con 0x7fe2ec0047c0 2023-10-16T14:07:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec00dec0 con 0x7fe3040ad8b0 2023-10-16T14:07:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe303fff640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0039a0 0x7fe2ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe3037fe640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0047c0 0x7fe2ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.995+0000 7fe308c01640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe2ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.996+0000 7fe303fff640 1 -- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 msgr2=0x7fe2ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.996+0000 7fe303fff640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe2ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.996+0000 7fe303fff640 1 -- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0047c0 msgr2=0x7fe2ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.996+0000 7fe303fff640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0047c0 0x7fe2ec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.996+0000 7fe303fff640 1 -- 172.21.15.60:0/3074523680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec003520 con 0x7fe2ec0039a0 2023-10-16T14:07:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.996+0000 7fe308c01640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040ad8b0 0x7fe2ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.997+0000 7fe303fff640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0039a0 0x7fe2ec003dc0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4001e20 tx=0x7fe2ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16f89b07db6756d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.997+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4003b70 con 0x7fe2ec0039a0 2023-10-16T14:07:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.997+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4003cd0 con 0x7fe2ec0039a0 2023-10-16T14:07:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.997+0000 7fe302ffd640 1 -- 172.21.15.60:0/3074523680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4005310 con 0x7fe2ec0039a0 2023-10-16T14:07:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe30a664640 1 -- 172.21.15.60:0/3074523680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0039a0 msgr2=0x7fe2ec003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe30a664640 1 --2- 172.21.15.60:0/3074523680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2ec0039a0 0x7fe2ec003dc0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4001e20 tx=0x7fe2ec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe308c01640 1 -- 172.21.15.60:0/3074523680 reap_dead start 2023-10-16T14:07:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe30a664640 1 -- 172.21.15.60:0/3074523680 shutdown_connections 2023-10-16T14:07:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe30a664640 1 -- 172.21.15.60:0/3074523680 >> 172.21.15.60:0/3074523680 conn(0x7fe3040ce840 msgr2=0x7fe3040cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:44.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe30a664640 1 -- 172.21.15.60:0/3074523680 shutdown_connections 2023-10-16T14:07:44.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:43.998+0000 7fe30a664640 1 -- 172.21.15.60:0/3074523680 wait complete. 2023-10-16T14:07:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.000+0000 7fe30a664640 1 Processor -- start 2023-10-16T14:07:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.000+0000 7fe30a664640 1 -- start start 2023-10-16T14:07:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe30a664640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0039a0 0x7fe3040b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe30a664640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 0x7fe304154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe30a664640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040c7800 0x7fe3040c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe30a664640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040c4970 con 0x7fe3041540a0 2023-10-16T14:07:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe30a664640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040b3ae0 con 0x7fe2ec0039a0 2023-10-16T14:07:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe30a664640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe304154a40 con 0x7fe3040c7800 2023-10-16T14:07:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe3037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 0x7fe304154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe3037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 0x7fe304154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48952/0 (socket says 172.21.15.60:48952) 2023-10-16T14:07:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe3037fe640 1 -- 172.21.15.60:0/1049216287 learned_addr learned my addr 172.21.15.60:0/1049216287 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe303fff640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0039a0 0x7fe3040b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.001+0000 7fe308c01640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040c7800 0x7fe3040c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe3037fe640 1 -- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040c7800 msgr2=0x7fe3040c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe3037fe640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040c7800 0x7fe3040c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe3037fe640 1 -- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0039a0 msgr2=0x7fe3040b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe3037fe640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0039a0 0x7fe3040b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe3037fe640 1 -- 172.21.15.60:0/1049216287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3040c4080 con 0x7fe3041540a0 2023-10-16T14:07:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe308c01640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040c7800 0x7fe3040c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe3037fe640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 0x7fe304154500 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800e790 tx=0x7fe2f800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800eea0 con 0x7fe3041540a0 2023-10-16T14:07:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f800c8d0 con 0x7fe3041540a0 2023-10-16T14:07:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800cc80 con 0x7fe3041540a0 2023-10-16T14:07:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3040c4320 con 0x7fe3041540a0 2023-10-16T14:07:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.002+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3040c4e00 con 0x7fe3041540a0 2023-10-16T14:07:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.003+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3040caf20 con 0x7fe3041540a0 2023-10-16T14:07:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.005+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2f801d460 con 0x7fe3041540a0 2023-10-16T14:07:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.007+0000 7fe300ff9640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe2f006af90 0x7fe2f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.007+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe2f808a2a0 con 0x7fe3041540a0 2023-10-16T14:07:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.008+0000 7fe303fff640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe2f006af90 0x7fe2f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.009+0000 7fe303fff640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe2f006af90 0x7fe2f006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4034000 tx=0x7fe2f4009030 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:44.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.011+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2f8055310 con 0x7fe3041540a0 2023-10-16T14:07:44.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.250+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe30404e160 con 0x7fe3041540a0 2023-10-16T14:07:44.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.810+0000 7fe300ff9640 1 -- 172.21.15.60:0/1049216287 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v100) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe2f805de60 con 0x7fe3041540a0 2023-10-16T14:07:44.814 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 24 and data pool 25 2023-10-16T14:07:44.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe2f006af90 msgr2=0x7fe2f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe2f006af90 0x7fe2f006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4034000 tx=0x7fe2f4009030 comp rx=0 tx=0).stop 2023-10-16T14:07:44.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 msgr2=0x7fe304154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:44.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 0x7fe304154500 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800e790 tx=0x7fe2f800ec50 comp rx=0 tx=0).stop 2023-10-16T14:07:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 shutdown_connections 2023-10-16T14:07:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe3040c7800 0x7fe3040c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fe2f006af90 0x7fe2f006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041540a0 0x7fe304154500 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 --2- 172.21.15.60:0/1049216287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2ec0039a0 0x7fe3040b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.822+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 >> 172.21.15.60:0/1049216287 conn(0x7fe3040ce840 msgr2=0x7fe30406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.823+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 shutdown_connections 2023-10-16T14:07:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:44.823+0000 7fe30a664640 1 -- 172.21.15.60:0/1049216287 wait complete. 2023-10-16T14:07:44.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.038+0000 7f2b6b59e640 1 Processor -- start 2023-10-16T14:07:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.038+0000 7f2b6b59e640 1 -- start start 2023-10-16T14:07:45.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.040+0000 7f2b6b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2b6c06c040 0x7f2b6c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.040+0000 7f2b6b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b6c0cda80 0x7f2b6c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.040+0000 7f2b6b59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6c06c960 con 0x7f2b6c0ad970 2023-10-16T14:07:45.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.040+0000 7f2b6b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6c15b9d0 con 0x7f2b6c0cda80 2023-10-16T14:07:45.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.040+0000 7f2b6b59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6c15bcb0 con 0x7f2b6c06c040 2023-10-16T14:07:45.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2b6c0cda80 0x7f2b6c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6a59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2b6c06c040 0x7f2b6c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6a59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2b6c06c040 0x7f2b6c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:49876/0 (socket says 172.21.15.60:49876) 2023-10-16T14:07:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6a59c640 1 -- 172.21.15.60:0/1022525358 learned_addr learned my addr 172.21.15.60:0/1022525358 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:45.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6ad9d640 1 -- 172.21.15.60:0/1022525358 >> v2:172.21.15.144:3300/0 conn(0x7f2b6c06c040 msgr2=0x7f2b6c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6ad9d640 1 --2- 172.21.15.60:0/1022525358 >> v2:172.21.15.144:3300/0 conn(0x7f2b6c06c040 0x7f2b6c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6ad9d640 1 -- 172.21.15.60:0/1022525358 >> v1:172.21.15.144:6789/0 conn(0x7f2b6c0ad970 legacy=0x7f2b6c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.041+0000 7f2b6ad9d640 1 -- 172.21.15.60:0/1022525358 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c15bf90 con 0x7f2b6c0cda80 2023-10-16T14:07:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.042+0000 7f2b6ad9d640 1 --2- 172.21.15.60:0/1022525358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c0cda80 0x7f2b6c15ed90 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0cf350 tx=0x7f2b6402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9420b98f930fec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.042+0000 7f2b69d9b640 1 -- 172.21.15.60:0/1022525358 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6402cbf0 con 0x7f2b6c0cda80 2023-10-16T14:07:45.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b69d9b640 1 -- 172.21.15.60:0/1022525358 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6402cd50 con 0x7f2b6c0cda80 2023-10-16T14:07:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b69d9b640 1 -- 172.21.15.60:0/1022525358 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64034660 con 0x7f2b6c0cda80 2023-10-16T14:07:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b6b59e640 1 -- 172.21.15.60:0/1022525358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c0cda80 msgr2=0x7f2b6c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/1022525358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c0cda80 0x7f2b6c15ed90 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0cf350 tx=0x7f2b6402be40 comp rx=0 tx=0).stop 2023-10-16T14:07:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b6b59e640 1 -- 172.21.15.60:0/1022525358 shutdown_connections 2023-10-16T14:07:45.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/1022525358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c0cda80 0x7f2b6c15ed90 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/1022525358 >> v2:172.21.15.144:3300/0 conn(0x7f2b6c06c040 0x7f2b6c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.043+0000 7f2b6b59e640 1 -- 172.21.15.60:0/1022525358 >> 172.21.15.60:0/1022525358 conn(0x7f2b6c0ce6a0 msgr2=0x7f2b6c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:45.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.044+0000 7f2b6b59e640 1 -- 172.21.15.60:0/1022525358 shutdown_connections 2023-10-16T14:07:45.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.044+0000 7f2b6b59e640 1 -- 172.21.15.60:0/1022525358 wait complete. 2023-10-16T14:07:45.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.045+0000 7f2b6b59e640 1 Processor -- start 2023-10-16T14:07:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.046+0000 7f2b6b59e640 1 -- start start 2023-10-16T14:07:45.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 0x7f2b6c154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b6c0ad970 0x7f2b6c0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b6c0cda80 0x7f2b6c0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c1efc90 con 0x7f2b6c06c040 2023-10-16T14:07:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c1ef0a0 con 0x7f2b6c0ad970 2023-10-16T14:07:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c1ef2f0 con 0x7f2b6c0cda80 2023-10-16T14:07:45.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 0x7f2b6c154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 0x7f2b6c154b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48970/0 (socket says 172.21.15.60:48970) 2023-10-16T14:07:45.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6a59c640 1 -- 172.21.15.60:0/2362263911 learned_addr learned my addr 172.21.15.60:0/2362263911 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6a59c640 1 -- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b6c0cda80 msgr2=0x7f2b6c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6ad9d640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b6c0cda80 0x7f2b6c0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b63fff640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b6c0ad970 0x7f2b6c0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.047+0000 7f2b6a59c640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b6c0cda80 0x7f2b6c0b2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b6a59c640 1 -- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b6c0ad970 msgr2=0x7f2b6c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b6a59c640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b6c0ad970 0x7f2b6c0b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b6a59c640 1 -- 172.21.15.60:0/2362263911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009a70 con 0x7f2b6c06c040 2023-10-16T14:07:45.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b63fff640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b6c0ad970 0x7f2b6c0b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b6a59c640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 0x7f2b6c154b10 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00dfd0 tx=0x7f2b5c010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c00dac0 con 0x7f2b6c06c040 2023-10-16T14:07:45.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c013040 con 0x7f2b6c06c040 2023-10-16T14:07:45.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c010de0 con 0x7f2b6c06c040 2023-10-16T14:07:45.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c1ef5d0 con 0x7f2b6c06c040 2023-10-16T14:07:45.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.048+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c1586a0 con 0x7f2b6c06c040 2023-10-16T14:07:45.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.049+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c06b330 con 0x7f2b6c06c040 2023-10-16T14:07:45.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.051+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b5c018480 con 0x7f2b6c06c040 2023-10-16T14:07:45.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.051+0000 7f2b62ffd640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2b4c06af90 0x7f2b4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.052+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b5c08d920 con 0x7f2b6c06c040 2023-10-16T14:07:45.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.052+0000 7f2b63fff640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2b4c06af90 0x7f2b4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.053+0000 7f2b63fff640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2b4c06af90 0x7f2b4c06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0c9a80 tx=0x7f2b54009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.056+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b5c058840 con 0x7f2b6c06c040 2023-10-16T14:07:45.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.249+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b6c04e9b0 con 0x7f2b6c06c040 2023-10-16T14:07:45.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.251+0000 7f2b62ffd640 1 -- 172.21.15.60:0/2362263911 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f2b5c00d2d0 con 0x7f2b6c06c040 2023-10-16T14:07:45.254 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:45.254 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":212,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:44.251924+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:45.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2b4c06af90 msgr2=0x7f2b4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2b4c06af90 0x7f2b4c06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c0c9a80 tx=0x7f2b54009040 comp rx=0 tx=0).stop 2023-10-16T14:07:45.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 msgr2=0x7f2b6c154b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 0x7f2b6c154b10 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00dfd0 tx=0x7f2b5c010490 comp rx=0 tx=0).stop 2023-10-16T14:07:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 shutdown_connections 2023-10-16T14:07:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2b6c0cda80 0x7f2b6c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2b4c06af90 0x7f2b4c06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b6c0ad970 0x7f2b6c0b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 --2- 172.21.15.60:0/2362263911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2b6c06c040 0x7f2b6c154b10 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 >> 172.21.15.60:0/2362263911 conn(0x7f2b6c0ce6a0 msgr2=0x7f2b6c0c9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 shutdown_connections 2023-10-16T14:07:45.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.254+0000 7f2b6b59e640 1 -- 172.21.15.60:0/2362263911 wait complete. 2023-10-16T14:07:45.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:45.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.488+0000 7f438e4df640 1 Processor -- start 2023-10-16T14:07:45.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.489+0000 7f438e4df640 1 -- start start 2023-10-16T14:07:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438e4df640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4388153740 0x7f438815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438e4df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 0x7f43880ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438e4df640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43880cd4a0 con 0x7f43880ad4d0 2023-10-16T14:07:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438e4df640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438815ac60 con 0x7f4388153740 2023-10-16T14:07:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438e4df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4388070030 con 0x7f43880ccb00 2023-10-16T14:07:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438dcde640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f43880ad4d0 0x7f43880ad960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39684/0 (socket says 172.21.15.60:39684) 2023-10-16T14:07:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.490+0000 7f438dcde640 1 -- 172.21.15.60:0/3191997232 learned_addr learned my addr 172.21.15.60:0/3191997232 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f438ccdc640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 0x7f43880ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f438d4dd640 1 --2- 172.21.15.60:0/3191997232 >> v2:172.21.15.144:3300/0 conn(0x7f4388153740 0x7f438815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302440513 0 0) 0x7f43880cd4a0 con 0x7f43880ad4d0 2023-10-16T14:07:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4374003520 con 0x7f43880ad4d0 2023-10-16T14:07:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106636743 0 0) 0x7f4374003520 con 0x7f43880ad4d0 2023-10-16T14:07:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 msgr2=0x7f43880ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 0x7f43880ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 >> v2:172.21.15.144:3300/0 conn(0x7f4388153740 msgr2=0x7f438815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 --2- 172.21.15.60:0/3191997232 >> v2:172.21.15.144:3300/0 conn(0x7f4388153740 0x7f438815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4388070310 con 0x7f43880ad4d0 2023-10-16T14:07:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f43800019f0 con 0x7f43880ad4d0 2023-10-16T14:07:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 >> v1:172.21.15.60:6789/0 conn(0x7f43880ad4d0 legacy=0x7f43880ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 0x7f4374003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4374004d00 0x7f4374005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438815ac60 con 0x7f4374003ef0 2023-10-16T14:07:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388070030 con 0x7f4374004d00 2023-10-16T14:07:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.491+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437400b9d0 con 0x7f43880ccb00 2023-10-16T14:07:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438ccdc640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438dcde640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4374004d00 0x7f4374005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438d4dd640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 0x7f4374003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438ccdc640 1 -- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 msgr2=0x7f4374003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438ccdc640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f43880ccb00 0x7f4374003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438ccdc640 1 -- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4374004d00 msgr2=0x7f4374005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438ccdc640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4374004d00 0x7f4374005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438ccdc640 1 -- 172.21.15.60:0/3191997232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4374003520 con 0x7f4374003ef0 2023-10-16T14:07:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.492+0000 7f438dcde640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4374004d00 0x7f4374005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.493+0000 7f438ccdc640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43740047c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f437800aa70 tx=0x7f43740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed869b948a92f79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.493+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437802cbb0 con 0x7f4374003ef0 2023-10-16T14:07:45.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.493+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437802cd10 con 0x7f4374003ef0 2023-10-16T14:07:45.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.493+0000 7f437ffff640 1 -- 172.21.15.60:0/3191997232 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378034660 con 0x7f4374003ef0 2023-10-16T14:07:45.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438e4df640 1 -- 172.21.15.60:0/3191997232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 msgr2=0x7f43740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438e4df640 1 --2- 172.21.15.60:0/3191997232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43740047c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f437800aa70 tx=0x7f43740057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438dcde640 1 -- 172.21.15.60:0/3191997232 reap_dead start 2023-10-16T14:07:45.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438e4df640 1 -- 172.21.15.60:0/3191997232 shutdown_connections 2023-10-16T14:07:45.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438e4df640 1 -- 172.21.15.60:0/3191997232 >> 172.21.15.60:0/3191997232 conn(0x7f43880ce6a0 msgr2=0x7f43880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:45.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438e4df640 1 -- 172.21.15.60:0/3191997232 shutdown_connections 2023-10-16T14:07:45.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.494+0000 7f438e4df640 1 -- 172.21.15.60:0/3191997232 wait complete. 2023-10-16T14:07:45.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.496+0000 7f438e4df640 1 Processor -- start 2023-10-16T14:07:45.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.496+0000 7f438e4df640 1 -- start start 2023-10-16T14:07:45.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438e4df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43880c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438e4df640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4388148640 0x7f43880ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438e4df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4388148ac0 0x7f43880acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438e4df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388149020 con 0x7f4374003ef0 2023-10-16T14:07:45.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438e4df640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388149300 con 0x7f4388148640 2023-10-16T14:07:45.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438e4df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43881495e0 con 0x7f4388148ac0 2023-10-16T14:07:45.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43880c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43880c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48988/0 (socket says 172.21.15.60:48988) 2023-10-16T14:07:45.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 -- 172.21.15.60:0/1903039133 learned_addr learned my addr 172.21.15.60:0/1903039133 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:45.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 -- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4388148ac0 msgr2=0x7f43880acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4388148ac0 0x7f43880acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438ccdc640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4388148640 0x7f43880ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 -- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4388148640 msgr2=0x7f43880ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.497+0000 7f438d4dd640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4388148640 0x7f43880ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f438d4dd640 1 -- 172.21.15.60:0/1903039133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43881498c0 con 0x7f4374003ef0 2023-10-16T14:07:45.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f438d4dd640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43880c49a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f4384010890 tx=0x7f4384010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384009630 con 0x7f4374003ef0 2023-10-16T14:07:45.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438400eb60 con 0x7f4374003ef0 2023-10-16T14:07:45.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438401c400 con 0x7f4374003ef0 2023-10-16T14:07:45.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4388149ba0 con 0x7f4374003ef0 2023-10-16T14:07:45.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.498+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438806c710 con 0x7f4374003ef0 2023-10-16T14:07:45.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.499+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438806fc30 con 0x7f4374003ef0 2023-10-16T14:07:45.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.500+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43840143f0 con 0x7f4374003ef0 2023-10-16T14:07:45.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.502+0000 7f437dffb640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f436c06af90 0x7f436c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.502+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4384089a90 con 0x7f4374003ef0 2023-10-16T14:07:45.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.502+0000 7f438ccdc640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f436c06af90 0x7f436c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.503+0000 7f438ccdc640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f436c06af90 0x7f436c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4378036000 tx=0x7f4378030000 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.506+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4384054a20 con 0x7f4374003ef0 2023-10-16T14:07:45.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.695+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43880705c0 con 0x7f4374003ef0 2023-10-16T14:07:45.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.697+0000 7f437dffb640 1 -- 172.21.15.60:0/1903039133 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f4384058040 con 0x7f4374003ef0 2023-10-16T14:07:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:45.701 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":212,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:44.251924+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:45.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f436c06af90 msgr2=0x7f436c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f436c06af90 0x7f436c06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4378036000 tx=0x7f4378030000 comp rx=0 tx=0).stop 2023-10-16T14:07:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 msgr2=0x7f43880c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43880c49a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f4384010890 tx=0x7f4384010d50 comp rx=0 tx=0).stop 2023-10-16T14:07:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 shutdown_connections 2023-10-16T14:07:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4388148ac0 0x7f43880acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f436c06af90 0x7f436c06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4388148640 0x7f43880ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 --2- 172.21.15.60:0/1903039133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4374003ef0 0x7f43880c49a0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 >> 172.21.15.60:0/1903039133 conn(0x7f43880ce6a0 msgr2=0x7f43880ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 shutdown_connections 2023-10-16T14:07:45.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.700+0000 7f438e4df640 1 -- 172.21.15.60:0/1903039133 wait complete. 2023-10-16T14:07:45.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:07:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.940+0000 7f65bdd5a640 1 Processor -- start 2023-10-16T14:07:45.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.940+0000 7f65bdd5a640 1 -- start start 2023-10-16T14:07:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.941+0000 7f65bdd5a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65b80cf710 0x7f65b80b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.941+0000 7f65bdd5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 0x7f65b80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.941+0000 7f65bdd5a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b80d3980 con 0x7f65b80cfaf0 2023-10-16T14:07:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.941+0000 7f65bdd5a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b80d3c60 con 0x7f65b80cf710 2023-10-16T14:07:45.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.941+0000 7f65bdd5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b80ce630 con 0x7f65b80b6cd0 2023-10-16T14:07:45.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.942+0000 7f65b6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f65b80cfaf0 0x7f65b80d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39698/0 (socket says 172.21.15.60:39698) 2023-10-16T14:07:45.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.942+0000 7f65b6ffd640 1 -- 172.21.15.60:0/4102673984 learned_addr learned my addr 172.21.15.60:0/4102673984 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:45.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.942+0000 7f65b77fe640 1 --2- 172.21.15.60:0/4102673984 >> v2:172.21.15.144:3300/0 conn(0x7f65b80cf710 0x7f65b80b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.942+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 0x7f65b80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.942+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664478849 0 0) 0x7f65b80d3980 con 0x7f65b80cfaf0 2023-10-16T14:07:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.942+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a4003520 con 0x7f65b80cfaf0 2023-10-16T14:07:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3324255813 0 0) 0x7f65a4003520 con 0x7f65b80cfaf0 2023-10-16T14:07:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 msgr2=0x7f65b80b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 0x7f65b80b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 >> v2:172.21.15.144:3300/0 conn(0x7f65b80cf710 msgr2=0x7f65b80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 --2- 172.21.15.60:0/4102673984 >> v2:172.21.15.144:3300/0 conn(0x7f65b80cf710 0x7f65b80b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b80ce910 con 0x7f65b80cfaf0 2023-10-16T14:07:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f65a0001670 con 0x7f65b80cfaf0 2023-10-16T14:07:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 >> v1:172.21.15.60:6789/0 conn(0x7f65b80cfaf0 legacy=0x7f65b80d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65a4003970 0x7f65a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 0x7f65a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65a4004d00 0x7f65a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b80d3c60 con 0x7f65a4004d00 2023-10-16T14:07:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b80ce630 con 0x7f65a4003970 2023-10-16T14:07:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a400ba00 con 0x7f65b80b6cd0 2023-10-16T14:07:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65a4004d00 0x7f65a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.943+0000 7f65b77fe640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65a4003970 0x7f65a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.944+0000 7f65b6ffd640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 0x7f65a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.944+0000 7f65b7fff640 1 -- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 msgr2=0x7f65a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.944+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65b80b6cd0 0x7f65a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.944+0000 7f65b7fff640 1 -- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65a4003970 msgr2=0x7f65a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.944+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65a4003970 0x7f65a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.944+0000 7f65b7fff640 1 -- 172.21.15.60:0/4102673984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4003520 con 0x7f65a4004d00 2023-10-16T14:07:45.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.945+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65a4004d00 0x7f65a4005180 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009a90 tx=0x7f65a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e753116a60111ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.945+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b002c680 con 0x7f65a4004d00 2023-10-16T14:07:45.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.945+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b002c7e0 con 0x7f65a4004d00 2023-10-16T14:07:45.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.945+0000 7f65b67fc640 1 -- 172.21.15.60:0/4102673984 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b002cb90 con 0x7f65a4004d00 2023-10-16T14:07:45.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4102673984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65a4004d00 msgr2=0x7f65a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4102673984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65a4004d00 0x7f65a4005180 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009a90 tx=0x7f65a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65b7fff640 1 -- 172.21.15.60:0/4102673984 reap_dead start 2023-10-16T14:07:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4102673984 shutdown_connections 2023-10-16T14:07:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4102673984 >> 172.21.15.60:0/4102673984 conn(0x7f65b80d2cc0 msgr2=0x7f65b80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4102673984 shutdown_connections 2023-10-16T14:07:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.946+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4102673984 wait complete. 2023-10-16T14:07:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.948+0000 7f65bdd5a640 1 Processor -- start 2023-10-16T14:07:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.948+0000 7f65bdd5a640 1 -- start start 2023-10-16T14:07:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65bdd5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65a4004d00 0x7f65b8156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65bdd5a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b815fd70 0x7f65b80b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65bdd5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 0x7f65b81e69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65bdd5a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b815f950 con 0x7f65b8160380 2023-10-16T14:07:45.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65bdd5a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b815ecd0 con 0x7f65b815fd70 2023-10-16T14:07:45.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65bdd5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b815efb0 con 0x7f65a4004d00 2023-10-16T14:07:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 0x7f65b81e69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 0x7f65b81e69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49010/0 (socket says 172.21.15.60:49010) 2023-10-16T14:07:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.949+0000 7f65b7fff640 1 -- 172.21.15.60:0/4294359190 learned_addr learned my addr 172.21.15.60:0/4294359190 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b77fe640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65a4004d00 0x7f65b8156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b6ffd640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b815fd70 0x7f65b80b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b7fff640 1 -- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65a4004d00 msgr2=0x7f65b8156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65a4004d00 0x7f65b8156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b7fff640 1 -- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b815fd70 msgr2=0x7f65b80b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:45.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b815fd70 0x7f65b80b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:45.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b7fff640 1 -- 172.21.15.60:0/4294359190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b815f290 con 0x7f65b8160380 2023-10-16T14:07:45.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b77fe640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65a4004d00 0x7f65b8156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:45.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65b7fff640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 0x7f65b81e69a0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f65b0037040 tx=0x7f65b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b003e850 con 0x7f65b8160380 2023-10-16T14:07:45.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.950+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b003e9d0 con 0x7f65b8160380 2023-10-16T14:07:45.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.951+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b003ed80 con 0x7f65b8160380 2023-10-16T14:07:45.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.951+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b815f570 con 0x7f65b8160380 2023-10-16T14:07:45.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.951+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b81e6ee0 con 0x7f65b8160380 2023-10-16T14:07:45.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.952+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b815fa90 con 0x7f65b8160380 2023-10-16T14:07:45.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.953+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65b0034440 con 0x7f65b8160380 2023-10-16T14:07:45.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.954+0000 7f65affff640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f659806af90 0x7f659806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:45.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.954+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65b00ab020 con 0x7f65b8160380 2023-10-16T14:07:45.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.955+0000 7f65b77fe640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f659806af90 0x7f659806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:45.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.956+0000 7f65b77fe640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f659806af90 0x7f659806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007520 tx=0x7f65a800b040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:45.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:45.958+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65b815fa90 con 0x7f65b8160380 2023-10-16T14:07:46.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.193+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f65b815fa90 con 0x7f65b8160380 2023-10-16T14:07:46.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.822+0000 7f65affff640 1 -- 172.21.15.60:0/4294359190 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v103) v1 ==== 118+0+0 (secure 0 0 0) 0x7f65b815fa90 con 0x7f65b8160380 2023-10-16T14:07:46.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f659806af90 msgr2=0x7f659806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:46.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f659806af90 0x7f659806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007520 tx=0x7f65a800b040 comp rx=0 tx=0).stop 2023-10-16T14:07:46.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 msgr2=0x7f65b81e69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:46.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 0x7f65b81e69a0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f65b0037040 tx=0x7f65b0004730 comp rx=0 tx=0).stop 2023-10-16T14:07:46.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 shutdown_connections 2023-10-16T14:07:46.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f659806af90 0x7f659806d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:46.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65b8160380 0x7f65b81e69a0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:46.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65b815fd70 0x7f65b80b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:46.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.825+0000 7f65bdd5a640 1 --2- 172.21.15.60:0/4294359190 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65a4004d00 0x7f65b8156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:46.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.826+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 >> 172.21.15.60:0/4294359190 conn(0x7f65b80d2cc0 msgr2=0x7f65b806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:46.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.826+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 shutdown_connections 2023-10-16T14:07:46.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:46.826+0000 7f65bdd5a640 1 -- 172.21.15.60:0/4294359190 wait complete. 2023-10-16T14:07:46.851 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:07:46.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:07:47.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.085+0000 7fb4fa3b0640 1 Processor -- start 2023-10-16T14:07:47.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.086+0000 7fb4fa3b0640 1 -- start start 2023-10-16T14:07:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.087+0000 7fb4fa3b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4f406a4b0 0x7fb4f40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.087+0000 7fb4fa3b0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40cb030 con 0x7fb4f40c9ca0 2023-10-16T14:07:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.087+0000 7fb4fa3b0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40c91c0 con 0x7fb4f415ad70 2023-10-16T14:07:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.087+0000 7fb4fa3b0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f40c94a0 con 0x7fb4f406a4b0 2023-10-16T14:07:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.088+0000 7fb4f894d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb4f40c9ca0 0x7fb4f40ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39714/0 (socket says 172.21.15.60:39714) 2023-10-16T14:07:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.088+0000 7fb4f894d640 1 -- 172.21.15.60:0/1164204893 learned_addr learned my addr 172.21.15.60:0/1164204893 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:47.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.088+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1164204893 >> v2:172.21.15.144:3300/0 conn(0x7fb4f406a4b0 0x7fb4f40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:47.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.088+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366114824 0 0) 0x7fb4f40cb030 con 0x7fb4f40c9ca0 2023-10-16T14:07:47.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.088+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e0003520 con 0x7fb4f40c9ca0 2023-10-16T14:07:47.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2804991391 0 0) 0x7fb4e0003520 con 0x7fb4f40c9ca0 2023-10-16T14:07:47.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 >> v2:172.21.15.144:3300/0 conn(0x7fb4f406a4b0 msgr2=0x7fb4f40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:47.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 --2- 172.21.15.60:0/1164204893 >> v2:172.21.15.144:3300/0 conn(0x7fb4f406a4b0 0x7fb4f40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 >> v1:172.21.15.144:6789/0 conn(0x7fb4f415ad70 legacy=0x7fb4f4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40c9780 con 0x7fb4f40c9ca0 2023-10-16T14:07:47.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb4ec0019f0 con 0x7fb4f40c9ca0 2023-10-16T14:07:47.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 >> v1:172.21.15.60:6789/0 conn(0x7fb4f40c9ca0 legacy=0x7fb4f40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4e0003970 0x7fb4e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e00047c0 0x7fb4e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e0005180 0x7fb4e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c94a0 con 0x7fb4e00047c0 2023-10-16T14:07:47.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e000c020 con 0x7fb4e0005180 2023-10-16T14:07:47.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e000c1e0 con 0x7fb4e0003970 2023-10-16T14:07:47.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e00047c0 0x7fb4e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:47.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f894d640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e0005180 0x7fb4e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:47.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4e0003970 msgr2=0x7fb4e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:47.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4e0003970 0x7fb4e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e0005180 msgr2=0x7fb4e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:47.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e0005180 0x7fb4e000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1164204893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0003520 con 0x7fb4e00047c0 2023-10-16T14:07:47.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f894d640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e0005180 0x7fb4e000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:47.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.089+0000 7fb4f894d640 1 -- 172.21.15.60:0/1164204893 reap_dead start 2023-10-16T14:07:47.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.091+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e00047c0 0x7fb4e0004c40 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009ac0 tx=0x7fb4e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533f3cd26c68a856 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:47.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.091+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc02c6b0 con 0x7fb4e00047c0 2023-10-16T14:07:47.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.091+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc02c810 con 0x7fb4e00047c0 2023-10-16T14:07:47.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.091+0000 7fb4f37fe640 1 -- 172.21.15.60:0/1164204893 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc02cba0 con 0x7fb4e00047c0 2023-10-16T14:07:47.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1164204893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e00047c0 msgr2=0x7fb4e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e00047c0 0x7fb4e0004c40 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009ac0 tx=0x7fb4e00057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1164204893 shutdown_connections 2023-10-16T14:07:47.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1164204893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4e00047c0 0x7fb4e0004c40 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1164204893 >> 172.21.15.60:0/1164204893 conn(0x7fb4f40ce9b0 msgr2=0x7fb4f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:47.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1164204893 shutdown_connections 2023-10-16T14:07:47.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.092+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1164204893 wait complete. 2023-10-16T14:07:47.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.094+0000 7fb4fa3b0640 1 Processor -- start 2023-10-16T14:07:47.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.094+0000 7fb4fa3b0640 1 -- start start 2023-10-16T14:07:47.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4fa3b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e00047c0 0x7fb4f414ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4fa3b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 0x7fb4f415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4fa3b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4f40ca150 0x7fb4f40ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4fa3b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f415cc10 con 0x7fb4f414b860 2023-10-16T14:07:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4fa3b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40cab10 con 0x7fb4e00047c0 2023-10-16T14:07:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4fa3b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f415c000 con 0x7fb4f40ca150 2023-10-16T14:07:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 0x7fb4f415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 0x7fb4f415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49022/0 (socket says 172.21.15.60:49022) 2023-10-16T14:07:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.095+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1498567076 learned_addr learned my addr 172.21.15.60:0/1498567076 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4f40ca150 msgr2=0x7fb4f40ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4f40ca150 0x7fb4f40ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e00047c0 msgr2=0x7fb4f414ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e00047c0 0x7fb4f414ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4ebfff640 1 -- 172.21.15.60:0/1498567076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f415c2e0 con 0x7fb4f414b860 2023-10-16T14:07:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4ebfff640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 0x7fb4f415b450 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc02c250 tx=0x7fb4dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc003810 con 0x7fb4f414b860 2023-10-16T14:07:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc003970 con 0x7fb4f414b860 2023-10-16T14:07:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc003d00 con 0x7fb4f414b860 2023-10-16T14:07:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.096+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f415c5c0 con 0x7fb4f414b860 2023-10-16T14:07:47.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.097+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f414b2c0 con 0x7fb4f414b860 2023-10-16T14:07:47.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.098+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f404e160 con 0x7fb4f414b860 2023-10-16T14:07:47.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.099+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb4dc034440 con 0x7fb4f414b860 2023-10-16T14:07:47.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.099+0000 7fb4f17fa640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fb4d406afe0 0x7fb4d406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:47.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.100+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4dc0ab9f0 con 0x7fb4f414b860 2023-10-16T14:07:47.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.100+0000 7fb4f3fff640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fb4d406afe0 0x7fb4d406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:47.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.101+0000 7fb4f3fff640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fb4d406afe0 0x7fb4d406d4a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4069740 tx=0x7fb4e4007990 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:47.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.104+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4dc0ade60 con 0x7fb4f414b860 2023-10-16T14:07:47.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.342+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb4f40736e0 con 0x7fb4f414b860 2023-10-16T14:07:47.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.833+0000 7fb4f17fa640 1 -- 172.21.15.60:0/1498567076 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v104) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb4dc04c0c0 con 0x7fb4f414b860 2023-10-16T14:07:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fb4d406afe0 msgr2=0x7fb4d406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fb4d406afe0 0x7fb4d406d4a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4069740 tx=0x7fb4e4007990 comp rx=0 tx=0).stop 2023-10-16T14:07:47.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 msgr2=0x7fb4f415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 0x7fb4f415b450 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc02c250 tx=0x7fb4dc004730 comp rx=0 tx=0).stop 2023-10-16T14:07:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 shutdown_connections 2023-10-16T14:07:47.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb4f40ca150 0x7fb4f40ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fb4d406afe0 0x7fb4d406d4a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f414b860 0x7fb4f415b450 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 --2- 172.21.15.60:0/1498567076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4e00047c0 0x7fb4f414ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 >> 172.21.15.60:0/1498567076 conn(0x7fb4f40ce9b0 msgr2=0x7fb4f406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:47.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 shutdown_connections 2023-10-16T14:07:47.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:47.837+0000 7fb4fa3b0640 1 -- 172.21.15.60:0/1498567076 wait complete. 2023-10-16T14:07:47.860 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:07:47.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:48.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.088+0000 7f2af4f1b640 1 Processor -- start 2023-10-16T14:07:48.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.088+0000 7f2af4f1b640 1 -- start start 2023-10-16T14:07:48.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2af4f1b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2af4f1b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2af00ad970 0x7f2af015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2af4f1b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2af015ebc0 0x7f2af015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2af4f1b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af006c960 con 0x7f2af00ad970 2023-10-16T14:07:48.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2af4f1b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af015b9d0 con 0x7f2af015ebc0 2023-10-16T14:07:48.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2af4f1b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af015bcb0 con 0x7f2af006c040 2023-10-16T14:07:48.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.090+0000 7f2aeeffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2af00ad970 0x7f2af015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2af00ad970 0x7f2af015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49024/0 (socket says 172.21.15.60:49024) 2023-10-16T14:07:48.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1532733091 learned_addr learned my addr 172.21.15.60:0/1532733091 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:48.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeffff640 1 --2- 172.21.15.60:0/1532733091 >> v2:172.21.15.144:3300/0 conn(0x7f2af015ebc0 0x7f2af015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aef7fe640 1 --2- 172.21.15.60:0/1532733091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1532733091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 msgr2=0x7f2af006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:48.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 --2- 172.21.15.60:0/1532733091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1532733091 >> v2:172.21.15.144:3300/0 conn(0x7f2af015ebc0 msgr2=0x7f2af015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:48.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 --2- 172.21.15.60:0/1532733091 >> v2:172.21.15.144:3300/0 conn(0x7f2af015ebc0 0x7f2af015f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1532733091 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af015bf90 con 0x7f2af00ad970 2023-10-16T14:07:48.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.091+0000 7f2aeffff640 1 --2- 172.21.15.60:0/1532733091 >> v2:172.21.15.144:3300/0 conn(0x7f2af015ebc0 0x7f2af015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:48.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.092+0000 7f2aeeffd640 1 --2- 172.21.15.60:0/1532733091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af015e680 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009870 tx=0x7f2ae402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20580e00e89ef21e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:48.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.093+0000 7f2aee7fc640 1 -- 172.21.15.60:0/1532733091 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4031640 con 0x7f2af00ad970 2023-10-16T14:07:48.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.093+0000 7f2aee7fc640 1 -- 172.21.15.60:0/1532733091 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae40317a0 con 0x7f2af00ad970 2023-10-16T14:07:48.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.093+0000 7f2aee7fc640 1 -- 172.21.15.60:0/1532733091 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4031b30 con 0x7f2af00ad970 2023-10-16T14:07:48.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1532733091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 msgr2=0x7f2af015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1532733091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af015e680 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009870 tx=0x7f2ae402b8f0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1532733091 shutdown_connections 2023-10-16T14:07:48.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1532733091 >> v2:172.21.15.144:3300/0 conn(0x7f2af015ebc0 0x7f2af015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1532733091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af015e680 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1532733091 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1532733091 >> 172.21.15.60:0/1532733091 conn(0x7f2af00ce6a0 msgr2=0x7f2af00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:48.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1532733091 shutdown_connections 2023-10-16T14:07:48.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.094+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1532733091 wait complete. 2023-10-16T14:07:48.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.096+0000 7f2af4f1b640 1 Processor -- start 2023-10-16T14:07:48.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.096+0000 7f2af4f1b640 1 -- start start 2023-10-16T14:07:48.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2af4f1b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af0158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2af4f1b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af00b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2af4f1b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af015ebc0 0x7f2af00b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2af4f1b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af0159310 con 0x7f2af00ad970 2023-10-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2af4f1b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af00c5770 con 0x7f2af015ebc0 2023-10-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2af4f1b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af00c5a50 con 0x7f2af006c040 2023-10-16T14:07:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af00b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af00b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49034/0 (socket says 172.21.15.60:49034) 2023-10-16T14:07:48.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1090902284 learned_addr learned my addr 172.21.15.60:0/1090902284 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:48.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeffff640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af015ebc0 0x7f2af00b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aef7fe640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af0158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 msgr2=0x7f2af0158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af0158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af015ebc0 msgr2=0x7f2af00b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af015ebc0 0x7f2af00b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.097+0000 7f2aeeffd640 1 -- 172.21.15.60:0/1090902284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae40094d0 con 0x7f2af00ad970 2023-10-16T14:07:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2aeffff640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af015ebc0 0x7f2af00b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2aeeffd640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af00b0710 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4000c00 tx=0x7f2ae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae403ac60 con 0x7f2af00ad970 2023-10-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae403adc0 con 0x7f2af00ad970 2023-10-16T14:07:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af00c5d30 con 0x7f2af00ad970 2023-10-16T14:07:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4044630 con 0x7f2af00ad970 2023-10-16T14:07:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.098+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af00c62f0 con 0x7f2af00ad970 2023-10-16T14:07:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.099+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af006b330 con 0x7f2af00ad970 2023-10-16T14:07:48.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.100+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ae402f050 con 0x7f2af00ad970 2023-10-16T14:07:48.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.101+0000 7f2ad7fff640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2adc06af90 0x7f2adc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.101+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ae40ab740 con 0x7f2af00ad970 2023-10-16T14:07:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.102+0000 7f2aef7fe640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2adc06af90 0x7f2adc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.103+0000 7f2aef7fe640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2adc06af90 0x7f2adc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0006fd0 tx=0x7f2ae0008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:48.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.106+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ae40765d0 con 0x7f2af00ad970 2023-10-16T14:07:48.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.343+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2af004e9b0 con 0x7f2af00ad970 2023-10-16T14:07:48.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.344+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1090902284 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f2ae4031e60 con 0x7f2af00ad970 2023-10-16T14:07:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:07:44.252784+0000","modified":"2023-10-16T14:07:47.343844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8539":{"gid":8539,"name":"b","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6835/3999834969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3999834969},{"type":"v1","addr":"172.21.15.144:6835","nonce":3999834969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T14:07:48.347 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 104 2023-10-16T14:07:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2adc06af90 msgr2=0x7f2adc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2adc06af90 0x7f2adc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0006fd0 tx=0x7f2ae0008040 comp rx=0 tx=0).stop 2023-10-16T14:07:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 msgr2=0x7f2af00b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af00b0710 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4000c00 tx=0x7f2ae40046c0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 shutdown_connections 2023-10-16T14:07:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2adc06af90 0x7f2adc06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2af015ebc0 0x7f2af00b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2af00ad970 0x7f2af00b0710 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 --2- 172.21.15.60:0/1090902284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2af006c040 0x7f2af0158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.346+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 >> 172.21.15.60:0/1090902284 conn(0x7f2af00ce6a0 msgr2=0x7f2af00cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.347+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 shutdown_connections 2023-10-16T14:07:48.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.347+0000 7f2af4f1b640 1 -- 172.21.15.60:0/1090902284 wait complete. 2023-10-16T14:07:48.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:07:48.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.595+0000 7f8ca6b35640 1 Processor -- start 2023-10-16T14:07:48.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.596+0000 7f8ca6b35640 1 -- start start 2023-10-16T14:07:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.597+0000 7f8ca6b35640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca00ad970 0x7f8ca00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.597+0000 7f8ca6b35640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8ca00ccfb0 0x7f8ca00cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.597+0000 7f8ca6b35640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca015b9d0 con 0x7f8ca006c040 2023-10-16T14:07:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.597+0000 7f8ca6b35640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca015bcb0 con 0x7f8ca00ad970 2023-10-16T14:07:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.597+0000 7f8ca6b35640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca015bf90 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5b33640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8ca006c040 0x7f8ca006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39718/0 (socket says 172.21.15.60:39718) 2023-10-16T14:07:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5b33640 1 -- 172.21.15.60:0/12189955 learned_addr learned my addr 172.21.15.60:0/12189955 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:48.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5332640 1 --2- 172.21.15.60:0/12189955 >> v2:172.21.15.60:3300/0 conn(0x7f8ca00ad970 0x7f8ca00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca6334640 1 --2- 172.21.15.60:0/12189955 >> v2:172.21.15.144:3300/0 conn(0x7f8ca00ccfb0 0x7f8ca00cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca4b31640 1 -- 172.21.15.60:0/12189955 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190865781 0 0) 0x7f8ca015b9d0 con 0x7f8ca006c040 2023-10-16T14:07:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca4b31640 1 -- 172.21.15.60:0/12189955 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c003520 con 0x7f8ca006c040 2023-10-16T14:07:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5332640 1 -- 172.21.15.60:0/12189955 >> v2:172.21.15.144:3300/0 conn(0x7f8ca00ccfb0 msgr2=0x7f8ca00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5332640 1 --2- 172.21.15.60:0/12189955 >> v2:172.21.15.144:3300/0 conn(0x7f8ca00ccfb0 0x7f8ca00cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5332640 1 -- 172.21.15.60:0/12189955 >> v1:172.21.15.60:6789/0 conn(0x7f8ca006c040 legacy=0x7f8ca006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.598+0000 7f8ca5332640 1 -- 172.21.15.60:0/12189955 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca00cc270 con 0x7f8ca00ad970 2023-10-16T14:07:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.599+0000 7f8ca5332640 1 --2- 172.21.15.60:0/12189955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ad970 0x7f8ca00cca70 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f8c900097a0 tx=0x7f8c9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d8ea6606abad3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca4b31640 1 -- 172.21.15.60:0/12189955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90031830 con 0x7f8ca00ad970 2023-10-16T14:07:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca4b31640 1 -- 172.21.15.60:0/12189955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c90031990 con 0x7f8ca00ad970 2023-10-16T14:07:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca4b31640 1 -- 172.21.15.60:0/12189955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90031d00 con 0x7f8ca00ad970 2023-10-16T14:07:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca6b35640 1 -- 172.21.15.60:0/12189955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ad970 msgr2=0x7f8ca00cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/12189955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ad970 0x7f8ca00cca70 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f8c900097a0 tx=0x7f8c9002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca6b35640 1 -- 172.21.15.60:0/12189955 shutdown_connections 2023-10-16T14:07:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/12189955 >> v2:172.21.15.144:3300/0 conn(0x7f8ca00ccfb0 0x7f8ca00cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/12189955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ad970 0x7f8ca00cca70 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.600+0000 7f8ca6b35640 1 -- 172.21.15.60:0/12189955 >> 172.21.15.60:0/12189955 conn(0x7f8ca00ce6a0 msgr2=0x7f8ca00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.601+0000 7f8ca6b35640 1 -- 172.21.15.60:0/12189955 shutdown_connections 2023-10-16T14:07:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.601+0000 7f8ca6b35640 1 -- 172.21.15.60:0/12189955 wait complete. 2023-10-16T14:07:48.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.602+0000 7f8ca6b35640 1 Processor -- start 2023-10-16T14:07:48.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.602+0000 7f8ca6b35640 1 -- start start 2023-10-16T14:07:48.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.603+0000 7f8ca6b35640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca006c040 0x7f8ca00aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.603+0000 7f8ca6b35640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca00ad970 0x7f8ca0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.603+0000 7f8ca6b35640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 0x7f8ca015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.603+0000 7f8ca6b35640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00b22e0 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.603+0000 7f8ca6b35640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca015aa30 con 0x7f8ca006c040 2023-10-16T14:07:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.603+0000 7f8ca6b35640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00af090 con 0x7f8ca00ad970 2023-10-16T14:07:48.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 0x7f8ca015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 0x7f8ca015a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49048/0 (socket says 172.21.15.60:49048) 2023-10-16T14:07:48.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 -- 172.21.15.60:0/4264242123 learned_addr learned my addr 172.21.15.60:0/4264242123 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:48.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca5b33640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca006c040 0x7f8ca00aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca5332640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca00ad970 0x7f8ca0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 -- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca00ad970 msgr2=0x7f8ca0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca00ad970 0x7f8ca0159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 -- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca006c040 msgr2=0x7f8ca00aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca006c040 0x7f8ca00aeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.604+0000 7f8ca6334640 1 -- 172.21.15.60:0/4264242123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c90009400 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.605+0000 7f8ca6334640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 0x7f8ca015a4f0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800d5d0 tx=0x7f8c9800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:48.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.605+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c980125f0 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.605+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c98012770 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.605+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98012b20 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.605+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca0158330 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.605+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca01588f0 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.606+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca006b330 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.607+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8c98012c80 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.608+0000 7f8c967fc640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8c8406af90 0x7f8c8406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.608+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c98089d50 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.609+0000 7f8ca5b33640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8c8406af90 0x7f8c8406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.610+0000 7f8ca5b33640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8c8406af90 0x7f8c8406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c005c00 tx=0x7f8c9c009040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:48.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.613+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c98054c60 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.814+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ca004e9b0 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.815+0000 7f8c967fc640 1 -- 172.21.15.60:0/4264242123 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16567 (secure 0 0 0) 0x7f8c98058280 con 0x7f8ca00ccfb0 2023-10-16T14:07:48.819 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:48.819 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":213,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:07:45.808069+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:07:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8c8406af90 msgr2=0x7f8c8406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8c8406af90 0x7f8c8406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c005c00 tx=0x7f8c9c009040 comp rx=0 tx=0).stop 2023-10-16T14:07:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 msgr2=0x7f8ca015a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 0x7f8ca015a4f0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800d5d0 tx=0x7f8c9800da90 comp rx=0 tx=0).stop 2023-10-16T14:07:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 shutdown_connections 2023-10-16T14:07:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca00ad970 0x7f8ca0159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8c8406af90 0x7f8c8406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca00ccfb0 0x7f8ca015a4f0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 --2- 172.21.15.60:0/4264242123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca006c040 0x7f8ca00aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.818+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 >> 172.21.15.60:0/4264242123 conn(0x7f8ca00ce6a0 msgr2=0x7f8ca015d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.819+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 shutdown_connections 2023-10-16T14:07:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:48.819+0000 7f8ca6b35640 1 -- 172.21.15.60:0/4264242123 wait complete. 2023-10-16T14:07:48.839 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:07:48.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:07:49.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.061+0000 7f2098d43640 1 Processor -- start 2023-10-16T14:07:49.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.061+0000 7f2098d43640 1 -- start start 2023-10-16T14:07:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f2098d43640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2094159950 0x7f209415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f2098d43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f20940708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f2098d43640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f209406c100 0x7f209406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f2098d43640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209415e490 con 0x7f2094159950 2023-10-16T14:07:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f2098d43640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2094070df0 con 0x7f209406c100 2023-10-16T14:07:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f2098d43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209415aef0 con 0x7f2094070450 2023-10-16T14:07:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.062+0000 7f20937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2094159950 0x7f209415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2094159950 0x7f209415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59852/0 (socket says 172.21.15.60:59852) 2023-10-16T14:07:49.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 -- 172.21.15.60:0/3102276652 learned_addr learned my addr 172.21.15.60:0/3102276652 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:49.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f2092ffd640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f20940708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f2093fff640 1 --2- 172.21.15.60:0/3102276652 >> v2:172.21.15.144:3300/0 conn(0x7f209406c100 0x7f209406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 -- 172.21.15.60:0/3102276652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 msgr2=0x7f20940708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f20940708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 -- 172.21.15.60:0/3102276652 >> v2:172.21.15.144:3300/0 conn(0x7f209406c100 msgr2=0x7f209406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:49.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 --2- 172.21.15.60:0/3102276652 >> v2:172.21.15.144:3300/0 conn(0x7f209406c100 0x7f209406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f20937fe640 1 -- 172.21.15.60:0/3102276652 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209415d910 con 0x7f2094159950 2023-10-16T14:07:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f2093fff640 1 --2- 172.21.15.60:0/3102276652 >> v2:172.21.15.144:3300/0 conn(0x7f209406c100 0x7f209406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.063+0000 7f2092ffd640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f20940708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.065+0000 7f20937fe640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159950 0x7f209415a9b0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f20940b0120 tx=0x7f208402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172e3543589d4971 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.065+0000 7f20927fc640 1 -- 172.21.15.60:0/3102276652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208402edf0 con 0x7f2094159950 2023-10-16T14:07:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.065+0000 7f20927fc640 1 -- 172.21.15.60:0/3102276652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084036440 con 0x7f2094159950 2023-10-16T14:07:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.065+0000 7f20927fc640 1 -- 172.21.15.60:0/3102276652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20840367d0 con 0x7f2094159950 2023-10-16T14:07:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 -- 172.21.15.60:0/3102276652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159950 msgr2=0x7f209415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159950 0x7f209415a9b0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f20940b0120 tx=0x7f208402e040 comp rx=0 tx=0).stop 2023-10-16T14:07:49.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 -- 172.21.15.60:0/3102276652 shutdown_connections 2023-10-16T14:07:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 --2- 172.21.15.60:0/3102276652 >> v2:172.21.15.144:3300/0 conn(0x7f209406c100 0x7f209406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f20940708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 --2- 172.21.15.60:0/3102276652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2094159950 0x7f209415a9b0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 -- 172.21.15.60:0/3102276652 >> 172.21.15.60:0/3102276652 conn(0x7f20940ce6a0 msgr2=0x7f20940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 -- 172.21.15.60:0/3102276652 shutdown_connections 2023-10-16T14:07:49.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.066+0000 7f2098d43640 1 -- 172.21.15.60:0/3102276652 wait complete. 2023-10-16T14:07:49.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.068+0000 7f2098d43640 1 Processor -- start 2023-10-16T14:07:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.068+0000 7f2098d43640 1 -- start start 2023-10-16T14:07:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2098d43640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 0x7f2094154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2098d43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f209406b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2098d43640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2094159950 0x7f20940c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2098d43640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941557f0 con 0x7f209406c100 2023-10-16T14:07:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2098d43640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20941edeb0 con 0x7f2094159950 2023-10-16T14:07:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2098d43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20940c21b0 con 0x7f2094070450 2023-10-16T14:07:49.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f20937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 0x7f2094154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f20937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 0x7f2094154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59860/0 (socket says 172.21.15.60:59860) 2023-10-16T14:07:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f20937fe640 1 -- 172.21.15.60:0/3948168317 learned_addr learned my addr 172.21.15.60:0/3948168317 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.069+0000 7f2092ffd640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f209406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f2093fff640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2094159950 0x7f20940c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f20937fe640 1 -- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 msgr2=0x7f209406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f20937fe640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f209406b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f20937fe640 1 -- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2094159950 msgr2=0x7f20940c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f20937fe640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2094159950 0x7f20940c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f20937fe640 1 -- 172.21.15.60:0/3948168317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208400b770 con 0x7f209406c100 2023-10-16T14:07:49.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f20937fe640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 0x7f2094154750 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f208402e080 tx=0x7f20840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084034b80 con 0x7f209406c100 2023-10-16T14:07:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084034ce0 con 0x7f209406c100 2023-10-16T14:07:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084041680 con 0x7f209406c100 2023-10-16T14:07:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.070+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20940c2490 con 0x7f209406c100 2023-10-16T14:07:49.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.071+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2094155ce0 con 0x7f209406c100 2023-10-16T14:07:49.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.072+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209404e9b0 con 0x7f209406c100 2023-10-16T14:07:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.073+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f208403d040 con 0x7f209406c100 2023-10-16T14:07:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.073+0000 7f208bfff640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f207406af90 0x7f207406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.074+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20840ae6f0 con 0x7f209406c100 2023-10-16T14:07:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.074+0000 7f2092ffd640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f207406af90 0x7f207406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.076+0000 7f2092ffd640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f207406af90 0x7f207406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20941f1490 tx=0x7f207c007990 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:49.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.078+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2084079690 con 0x7f209406c100 2023-10-16T14:07:49.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.271+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 --> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f209415ded0 con 0x7f207406af90 2023-10-16T14:07:49.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.300+0000 7f208bfff640 1 -- 172.21.15.60:0/3948168317 <== mgr.8350 v2:172.21.15.144:6832/1274087552 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286992 (secure 0 0 0) 0x7f207c0183f0 con 0x7f207406af90 2023-10-16T14:07:49.306 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:07:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.306+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f207406af90 msgr2=0x7f207406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.306+0000 7f2098d43640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f207406af90 0x7f207406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20941f1490 tx=0x7f207c007990 comp rx=0 tx=0).stop 2023-10-16T14:07:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.306+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 msgr2=0x7f2094154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.306+0000 7f2098d43640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 0x7f2094154750 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f208402e080 tx=0x7f20840046c0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.306+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 shutdown_connections 2023-10-16T14:07:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2094070450 0x7f209406b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f207406af90 0x7f207406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2094159950 0x7f20940c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 --2- 172.21.15.60:0/3948168317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f209406c100 0x7f2094154750 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 >> 172.21.15.60:0/3948168317 conn(0x7f20940ce6a0 msgr2=0x7f2094156570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 shutdown_connections 2023-10-16T14:07:49.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.307+0000 7f2098d43640 1 -- 172.21.15.60:0/3948168317 wait complete. 2023-10-16T14:07:49.330 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T14:07:48.125224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":587,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392208,"kb_used_data":21988,"kb_used_omap":12,"kb_used_meta":370099,"kb_avail":754582512,"statfs":{"total":773094113280,"available":772692492288,"internally_reserved":0,"allocated":22515712,"data_stored":11532104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":378981944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.369860"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212493+0000","last_change":"2023-10-16T14:07:44.832980+0000","last_active":"2023-10-16T14:07:46.212493+0000","last_peered":"2023-10-16T14:07:46.212493+0000","last_clean":"2023-10-16T14:07:46.212493+0000","last_became_active":"2023-10-16T14:07:44.832687+0000","last_became_peered":"2023-10-16T14:07:44.832687+0000","last_unstale":"2023-10-16T14:07:46.212493+0000","last_undegraded":"2023-10-16T14:07:46.212493+0000","last_fullsized":"2023-10-16T14:07:46.212493+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:17.284692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829340+0000","last_change":"2023-10-16T14:07:44.820860+0000","last_active":"2023-10-16T14:07:45.829340+0000","last_peered":"2023-10-16T14:07:45.829340+0000","last_clean":"2023-10-16T14:07:45.829340+0000","last_became_active":"2023-10-16T14:07:44.820368+0000","last_became_peered":"2023-10-16T14:07:44.820368+0000","last_unstale":"2023-10-16T14:07:45.829340+0000","last_undegraded":"2023-10-16T14:07:45.829340+0000","last_fullsized":"2023-10-16T14:07:45.829340+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:54.438384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713109+0000","last_change":"2023-10-16T14:07:44.810207+0000","last_active":"2023-10-16T14:07:46.713109+0000","last_peered":"2023-10-16T14:07:46.713109+0000","last_clean":"2023-10-16T14:07:46.713109+0000","last_became_active":"2023-10-16T14:07:44.809861+0000","last_became_peered":"2023-10-16T14:07:44.809861+0000","last_unstale":"2023-10-16T14:07:46.713109+0000","last_undegraded":"2023-10-16T14:07:46.713109+0000","last_fullsized":"2023-10-16T14:07:46.713109+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:35.506431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830184+0000","last_change":"2023-10-16T14:07:44.820903+0000","last_active":"2023-10-16T14:07:45.830184+0000","last_peered":"2023-10-16T14:07:45.830184+0000","last_clean":"2023-10-16T14:07:45.830184+0000","last_became_active":"2023-10-16T14:07:44.820563+0000","last_became_peered":"2023-10-16T14:07:44.820563+0000","last_unstale":"2023-10-16T14:07:45.830184+0000","last_undegraded":"2023-10-16T14:07:45.830184+0000","last_fullsized":"2023-10-16T14:07:45.830184+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:58.685010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968913+0000","last_change":"2023-10-16T14:07:44.823286+0000","last_active":"2023-10-16T14:07:45.968913+0000","last_peered":"2023-10-16T14:07:45.968913+0000","last_clean":"2023-10-16T14:07:45.968913+0000","last_became_active":"2023-10-16T14:07:44.822801+0000","last_became_peered":"2023-10-16T14:07:44.822801+0000","last_unstale":"2023-10-16T14:07:45.968913+0000","last_undegraded":"2023-10-16T14:07:45.968913+0000","last_fullsized":"2023-10-16T14:07:45.968913+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:15.477649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712642+0000","last_change":"2023-10-16T14:07:44.825170+0000","last_active":"2023-10-16T14:07:46.712642+0000","last_peered":"2023-10-16T14:07:46.712642+0000","last_clean":"2023-10-16T14:07:46.712642+0000","last_became_active":"2023-10-16T14:07:44.824860+0000","last_became_peered":"2023-10-16T14:07:44.824860+0000","last_unstale":"2023-10-16T14:07:46.712642+0000","last_undegraded":"2023-10-16T14:07:46.712642+0000","last_fullsized":"2023-10-16T14:07:46.712642+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:22.722924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713871+0000","last_change":"2023-10-16T14:07:44.813889+0000","last_active":"2023-10-16T14:07:46.713871+0000","last_peered":"2023-10-16T14:07:46.713871+0000","last_clean":"2023-10-16T14:07:46.713871+0000","last_became_active":"2023-10-16T14:07:44.813627+0000","last_became_peered":"2023-10-16T14:07:44.813627+0000","last_unstale":"2023-10-16T14:07:46.713871+0000","last_undegraded":"2023-10-16T14:07:46.713871+0000","last_fullsized":"2023-10-16T14:07:46.713871+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:23.659882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434318+0000","last_change":"2023-10-16T14:07:44.810629+0000","last_active":"2023-10-16T14:07:46.434318+0000","last_peered":"2023-10-16T14:07:46.434318+0000","last_clean":"2023-10-16T14:07:46.434318+0000","last_became_active":"2023-10-16T14:07:44.810349+0000","last_became_peered":"2023-10-16T14:07:44.810349+0000","last_unstale":"2023-10-16T14:07:46.434318+0000","last_undegraded":"2023-10-16T14:07:46.434318+0000","last_fullsized":"2023-10-16T14:07:46.434318+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:39.001087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830374+0000","last_change":"2023-10-16T14:07:44.817045+0000","last_active":"2023-10-16T14:07:45.830374+0000","last_peered":"2023-10-16T14:07:45.830374+0000","last_clean":"2023-10-16T14:07:45.830374+0000","last_became_active":"2023-10-16T14:07:44.816040+0000","last_became_peered":"2023-10-16T14:07:44.816040+0000","last_unstale":"2023-10-16T14:07:45.830374+0000","last_undegraded":"2023-10-16T14:07:45.830374+0000","last_fullsized":"2023-10-16T14:07:45.830374+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:59.541962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824142+0000","last_change":"2023-10-16T14:07:44.817360+0000","last_active":"2023-10-16T14:07:45.824142+0000","last_peered":"2023-10-16T14:07:45.824142+0000","last_clean":"2023-10-16T14:07:45.824142+0000","last_became_active":"2023-10-16T14:07:44.814906+0000","last_became_peered":"2023-10-16T14:07:44.814906+0000","last_unstale":"2023-10-16T14:07:45.824142+0000","last_undegraded":"2023-10-16T14:07:45.824142+0000","last_fullsized":"2023-10-16T14:07:45.824142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:48.796679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713611+0000","last_change":"2023-10-16T14:07:44.810568+0000","last_active":"2023-10-16T14:07:46.713611+0000","last_peered":"2023-10-16T14:07:46.713611+0000","last_clean":"2023-10-16T14:07:46.713611+0000","last_became_active":"2023-10-16T14:07:44.809635+0000","last_became_peered":"2023-10-16T14:07:44.809635+0000","last_unstale":"2023-10-16T14:07:46.713611+0000","last_undegraded":"2023-10-16T14:07:46.713611+0000","last_fullsized":"2023-10-16T14:07:46.713611+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:59.541962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830520+0000","last_change":"2023-10-16T14:07:44.830503+0000","last_active":"2023-10-16T14:07:45.830520+0000","last_peered":"2023-10-16T14:07:45.830520+0000","last_clean":"2023-10-16T14:07:45.830520+0000","last_became_active":"2023-10-16T14:07:44.830229+0000","last_became_peered":"2023-10-16T14:07:44.830229+0000","last_unstale":"2023-10-16T14:07:45.830520+0000","last_undegraded":"2023-10-16T14:07:45.830520+0000","last_fullsized":"2023-10-16T14:07:45.830520+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:04.984637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829253+0000","last_change":"2023-10-16T14:07:44.811857+0000","last_active":"2023-10-16T14:07:45.829253+0000","last_peered":"2023-10-16T14:07:45.829253+0000","last_clean":"2023-10-16T14:07:45.829253+0000","last_became_active":"2023-10-16T14:07:44.811393+0000","last_became_peered":"2023-10-16T14:07:44.811393+0000","last_unstale":"2023-10-16T14:07:45.829253+0000","last_undegraded":"2023-10-16T14:07:45.829253+0000","last_fullsized":"2023-10-16T14:07:45.829253+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:07.048482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830138+0000","last_change":"2023-10-16T14:07:44.815217+0000","last_active":"2023-10-16T14:07:45.830138+0000","last_peered":"2023-10-16T14:07:45.830138+0000","last_clean":"2023-10-16T14:07:45.830138+0000","last_became_active":"2023-10-16T14:07:44.814910+0000","last_became_peered":"2023-10-16T14:07:44.814910+0000","last_unstale":"2023-10-16T14:07:45.830138+0000","last_undegraded":"2023-10-16T14:07:45.830138+0000","last_fullsized":"2023-10-16T14:07:45.830138+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:04.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829840+0000","last_change":"2023-10-16T14:07:44.829603+0000","last_active":"2023-10-16T14:07:45.829840+0000","last_peered":"2023-10-16T14:07:45.829840+0000","last_clean":"2023-10-16T14:07:45.829840+0000","last_became_active":"2023-10-16T14:07:44.828751+0000","last_became_peered":"2023-10-16T14:07:44.828751+0000","last_unstale":"2023-10-16T14:07:45.829840+0000","last_undegraded":"2023-10-16T14:07:45.829840+0000","last_fullsized":"2023-10-16T14:07:45.829840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:42.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434310+0000","last_change":"2023-10-16T14:07:44.815993+0000","last_active":"2023-10-16T14:07:46.434310+0000","last_peered":"2023-10-16T14:07:46.434310+0000","last_clean":"2023-10-16T14:07:46.434310+0000","last_became_active":"2023-10-16T14:07:44.815677+0000","last_became_peered":"2023-10-16T14:07:44.815677+0000","last_unstale":"2023-10-16T14:07:46.434310+0000","last_undegraded":"2023-10-16T14:07:46.434310+0000","last_fullsized":"2023-10-16T14:07:46.434310+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:31.505442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969468+0000","last_change":"2023-10-16T14:07:44.818440+0000","last_active":"2023-10-16T14:07:45.969468+0000","last_peered":"2023-10-16T14:07:45.969468+0000","last_clean":"2023-10-16T14:07:45.969468+0000","last_became_active":"2023-10-16T14:07:44.818078+0000","last_became_peered":"2023-10-16T14:07:44.818078+0000","last_unstale":"2023-10-16T14:07:45.969468+0000","last_undegraded":"2023-10-16T14:07:45.969468+0000","last_fullsized":"2023-10-16T14:07:45.969468+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:04.200924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212568+0000","last_change":"2023-10-16T14:07:44.825752+0000","last_active":"2023-10-16T14:07:46.212568+0000","last_peered":"2023-10-16T14:07:46.212568+0000","last_clean":"2023-10-16T14:07:46.212568+0000","last_became_active":"2023-10-16T14:07:44.824616+0000","last_became_peered":"2023-10-16T14:07:44.824616+0000","last_unstale":"2023-10-16T14:07:46.212568+0000","last_undegraded":"2023-10-16T14:07:46.212568+0000","last_fullsized":"2023-10-16T14:07:46.212568+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:20.297567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824174+0000","last_change":"2023-10-16T14:07:44.819207+0000","last_active":"2023-10-16T14:07:45.824174+0000","last_peered":"2023-10-16T14:07:45.824174+0000","last_clean":"2023-10-16T14:07:45.824174+0000","last_became_active":"2023-10-16T14:07:44.818928+0000","last_became_peered":"2023-10-16T14:07:44.818928+0000","last_unstale":"2023-10-16T14:07:45.824174+0000","last_undegraded":"2023-10-16T14:07:45.824174+0000","last_fullsized":"2023-10-16T14:07:45.824174+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:36.435342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969201+0000","last_change":"2023-10-16T14:07:44.820978+0000","last_active":"2023-10-16T14:07:45.969201+0000","last_peered":"2023-10-16T14:07:45.969201+0000","last_clean":"2023-10-16T14:07:45.969201+0000","last_became_active":"2023-10-16T14:07:44.820707+0000","last_became_peered":"2023-10-16T14:07:44.820707+0000","last_unstale":"2023-10-16T14:07:45.969201+0000","last_undegraded":"2023-10-16T14:07:45.969201+0000","last_fullsized":"2023-10-16T14:07:45.969201+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:25.578215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969019+0000","last_change":"2023-10-16T14:07:44.823084+0000","last_active":"2023-10-16T14:07:45.969019+0000","last_peered":"2023-10-16T14:07:45.969019+0000","last_clean":"2023-10-16T14:07:45.969019+0000","last_became_active":"2023-10-16T14:07:44.822410+0000","last_became_peered":"2023-10-16T14:07:44.822410+0000","last_unstale":"2023-10-16T14:07:45.969019+0000","last_undegraded":"2023-10-16T14:07:45.969019+0000","last_fullsized":"2023-10-16T14:07:45.969019+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:58.772982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823364+0000","last_change":"2023-10-16T14:07:44.823572+0000","last_active":"2023-10-16T14:07:45.823364+0000","last_peered":"2023-10-16T14:07:45.823364+0000","last_clean":"2023-10-16T14:07:45.823364+0000","last_became_active":"2023-10-16T14:07:44.822342+0000","last_became_peered":"2023-10-16T14:07:44.822342+0000","last_unstale":"2023-10-16T14:07:45.823364+0000","last_undegraded":"2023-10-16T14:07:45.823364+0000","last_fullsized":"2023-10-16T14:07:45.823364+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:36.970078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712496+0000","last_change":"2023-10-16T14:07:44.815479+0000","last_active":"2023-10-16T14:07:46.712496+0000","last_peered":"2023-10-16T14:07:46.712496+0000","last_clean":"2023-10-16T14:07:46.712496+0000","last_became_active":"2023-10-16T14:07:44.814716+0000","last_became_peered":"2023-10-16T14:07:44.814716+0000","last_unstale":"2023-10-16T14:07:46.712496+0000","last_undegraded":"2023-10-16T14:07:46.712496+0000","last_fullsized":"2023-10-16T14:07:46.712496+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:35.069827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434993+0000","last_change":"2023-10-16T14:07:44.813649+0000","last_active":"2023-10-16T14:07:46.434993+0000","last_peered":"2023-10-16T14:07:46.434993+0000","last_clean":"2023-10-16T14:07:46.434993+0000","last_became_active":"2023-10-16T14:07:44.813324+0000","last_became_peered":"2023-10-16T14:07:44.813324+0000","last_unstale":"2023-10-16T14:07:46.434993+0000","last_undegraded":"2023-10-16T14:07:46.434993+0000","last_fullsized":"2023-10-16T14:07:46.434993+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:33.069562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.831011+0000","last_change":"2023-10-16T14:07:44.817683+0000","last_active":"2023-10-16T14:07:45.831011+0000","last_peered":"2023-10-16T14:07:45.831011+0000","last_clean":"2023-10-16T14:07:45.831011+0000","last_became_active":"2023-10-16T14:07:44.816829+0000","last_became_peered":"2023-10-16T14:07:44.816829+0000","last_unstale":"2023-10-16T14:07:45.831011+0000","last_undegraded":"2023-10-16T14:07:45.831011+0000","last_fullsized":"2023-10-16T14:07:45.831011+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:06.616493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713889+0000","last_change":"2023-10-16T14:07:44.818842+0000","last_active":"2023-10-16T14:07:46.713889+0000","last_peered":"2023-10-16T14:07:46.713889+0000","last_clean":"2023-10-16T14:07:46.713889+0000","last_became_active":"2023-10-16T14:07:44.814496+0000","last_became_peered":"2023-10-16T14:07:44.814496+0000","last_unstale":"2023-10-16T14:07:46.713889+0000","last_undegraded":"2023-10-16T14:07:46.713889+0000","last_fullsized":"2023-10-16T14:07:46.713889+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:20.813774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713109+0000","last_change":"2023-10-16T14:07:44.808447+0000","last_active":"2023-10-16T14:07:46.713109+0000","last_peered":"2023-10-16T14:07:46.713109+0000","last_clean":"2023-10-16T14:07:46.713109+0000","last_became_active":"2023-10-16T14:07:44.808133+0000","last_became_peered":"2023-10-16T14:07:44.808133+0000","last_unstale":"2023-10-16T14:07:46.713109+0000","last_undegraded":"2023-10-16T14:07:46.713109+0000","last_fullsized":"2023-10-16T14:07:46.713109+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:51.717001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969296+0000","last_change":"2023-10-16T14:07:44.815202+0000","last_active":"2023-10-16T14:07:45.969296+0000","last_peered":"2023-10-16T14:07:45.969296+0000","last_clean":"2023-10-16T14:07:45.969296+0000","last_became_active":"2023-10-16T14:07:44.814565+0000","last_became_peered":"2023-10-16T14:07:44.814565+0000","last_unstale":"2023-10-16T14:07:45.969296+0000","last_undegraded":"2023-10-16T14:07:45.969296+0000","last_fullsized":"2023-10-16T14:07:45.969296+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:22.722924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829395+0000","last_change":"2023-10-16T14:07:44.824502+0000","last_active":"2023-10-16T14:07:45.829395+0000","last_peered":"2023-10-16T14:07:45.829395+0000","last_clean":"2023-10-16T14:07:45.829395+0000","last_became_active":"2023-10-16T14:07:44.824186+0000","last_became_peered":"2023-10-16T14:07:44.824186+0000","last_unstale":"2023-10-16T14:07:45.829395+0000","last_undegraded":"2023-10-16T14:07:45.829395+0000","last_fullsized":"2023-10-16T14:07:45.829395+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:58.751855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830595+0000","last_change":"2023-10-16T14:07:44.822176+0000","last_active":"2023-10-16T14:07:45.830595+0000","last_peered":"2023-10-16T14:07:45.830595+0000","last_clean":"2023-10-16T14:07:45.830595+0000","last_became_active":"2023-10-16T14:07:44.821871+0000","last_became_peered":"2023-10-16T14:07:44.821871+0000","last_unstale":"2023-10-16T14:07:45.830595+0000","last_undegraded":"2023-10-16T14:07:45.830595+0000","last_fullsized":"2023-10-16T14:07:45.830595+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:20.813774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712735+0000","last_change":"2023-10-16T14:07:44.816572+0000","last_active":"2023-10-16T14:07:46.712735+0000","last_peered":"2023-10-16T14:07:46.712735+0000","last_clean":"2023-10-16T14:07:46.712735+0000","last_became_active":"2023-10-16T14:07:44.816303+0000","last_became_peered":"2023-10-16T14:07:44.816303+0000","last_unstale":"2023-10-16T14:07:46.712735+0000","last_undegraded":"2023-10-16T14:07:46.712735+0000","last_fullsized":"2023-10-16T14:07:46.712735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:04.200924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968715+0000","last_change":"2023-10-16T14:07:44.812574+0000","last_active":"2023-10-16T14:07:45.968715+0000","last_peered":"2023-10-16T14:07:45.968715+0000","last_clean":"2023-10-16T14:07:45.968715+0000","last_became_active":"2023-10-16T14:07:44.812284+0000","last_became_peered":"2023-10-16T14:07:44.812284+0000","last_unstale":"2023-10-16T14:07:45.968715+0000","last_undegraded":"2023-10-16T14:07:45.968715+0000","last_fullsized":"2023-10-16T14:07:45.968715+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:33.302528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435353+0000","last_change":"2023-10-16T14:07:44.816661+0000","last_active":"2023-10-16T14:07:46.435353+0000","last_peered":"2023-10-16T14:07:46.435353+0000","last_clean":"2023-10-16T14:07:46.435353+0000","last_became_active":"2023-10-16T14:07:44.816394+0000","last_became_peered":"2023-10-16T14:07:44.816394+0000","last_unstale":"2023-10-16T14:07:46.435353+0000","last_undegraded":"2023-10-16T14:07:46.435353+0000","last_fullsized":"2023-10-16T14:07:46.435353+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:16.459394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"212'4","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969502+0000","last_change":"2023-10-16T14:07:43.800016+0000","last_active":"2023-10-16T14:07:45.969502+0000","last_peered":"2023-10-16T14:07:45.969502+0000","last_clean":"2023-10-16T14:07:45.969502+0000","last_became_active":"2023-10-16T14:07:43.799731+0000","last_became_peered":"2023-10-16T14:07:43.799731+0000","last_unstale":"2023-10-16T14:07:45.969502+0000","last_undegraded":"2023-10-16T14:07:45.969502+0000","last_fullsized":"2023-10-16T14:07:45.969502+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:18.118851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824997+0000","last_change":"2023-10-16T14:07:44.822017+0000","last_active":"2023-10-16T14:07:45.824997+0000","last_peered":"2023-10-16T14:07:45.824997+0000","last_clean":"2023-10-16T14:07:45.824997+0000","last_became_active":"2023-10-16T14:07:44.821735+0000","last_became_peered":"2023-10-16T14:07:44.821735+0000","last_unstale":"2023-10-16T14:07:45.824997+0000","last_undegraded":"2023-10-16T14:07:45.824997+0000","last_fullsized":"2023-10-16T14:07:45.824997+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:58.319245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.831288+0000","last_change":"2023-10-16T14:07:43.790976+0000","last_active":"2023-10-16T14:07:45.831288+0000","last_peered":"2023-10-16T14:07:45.831288+0000","last_clean":"2023-10-16T14:07:45.831288+0000","last_became_active":"2023-10-16T14:07:43.790570+0000","last_became_peered":"2023-10-16T14:07:43.790570+0000","last_unstale":"2023-10-16T14:07:45.831288+0000","last_undegraded":"2023-10-16T14:07:45.831288+0000","last_fullsized":"2023-10-16T14:07:45.831288+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:21.360587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713695+0000","last_change":"2023-10-16T14:07:44.813549+0000","last_active":"2023-10-16T14:07:46.713695+0000","last_peered":"2023-10-16T14:07:46.713695+0000","last_clean":"2023-10-16T14:07:46.713695+0000","last_became_active":"2023-10-16T14:07:44.813285+0000","last_became_peered":"2023-10-16T14:07:44.813285+0000","last_unstale":"2023-10-16T14:07:46.713695+0000","last_undegraded":"2023-10-16T14:07:46.713695+0000","last_fullsized":"2023-10-16T14:07:46.713695+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:01.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713442+0000","last_change":"2023-10-16T14:07:43.794019+0000","last_active":"2023-10-16T14:07:46.713442+0000","last_peered":"2023-10-16T14:07:46.713442+0000","last_clean":"2023-10-16T14:07:46.713442+0000","last_became_active":"2023-10-16T14:07:43.793101+0000","last_became_peered":"2023-10-16T14:07:43.793101+0000","last_unstale":"2023-10-16T14:07:46.713442+0000","last_undegraded":"2023-10-16T14:07:46.713442+0000","last_fullsized":"2023-10-16T14:07:46.713442+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:24.563590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830517+0000","last_change":"2023-10-16T14:07:44.811185+0000","last_active":"2023-10-16T14:07:45.830517+0000","last_peered":"2023-10-16T14:07:45.830517+0000","last_clean":"2023-10-16T14:07:45.830517+0000","last_became_active":"2023-10-16T14:07:44.810803+0000","last_became_peered":"2023-10-16T14:07:44.810803+0000","last_unstale":"2023-10-16T14:07:45.830517+0000","last_undegraded":"2023-10-16T14:07:45.830517+0000","last_fullsized":"2023-10-16T14:07:45.830517+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:23.659882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830813+0000","last_change":"2023-10-16T14:07:43.799786+0000","last_active":"2023-10-16T14:07:45.830813+0000","last_peered":"2023-10-16T14:07:45.830813+0000","last_clean":"2023-10-16T14:07:45.830813+0000","last_became_active":"2023-10-16T14:07:43.798587+0000","last_became_peered":"2023-10-16T14:07:43.798587+0000","last_unstale":"2023-10-16T14:07:45.830813+0000","last_undegraded":"2023-10-16T14:07:45.830813+0000","last_fullsized":"2023-10-16T14:07:45.830813+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:36.490113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823366+0000","last_change":"2023-10-16T14:07:44.821312+0000","last_active":"2023-10-16T14:07:45.823366+0000","last_peered":"2023-10-16T14:07:45.823366+0000","last_clean":"2023-10-16T14:07:45.823366+0000","last_became_active":"2023-10-16T14:07:44.820574+0000","last_became_peered":"2023-10-16T14:07:44.820574+0000","last_unstale":"2023-10-16T14:07:45.823366+0000","last_undegraded":"2023-10-16T14:07:45.823366+0000","last_fullsized":"2023-10-16T14:07:45.823366+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:21.169581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969007+0000","last_change":"2023-10-16T14:07:43.796733+0000","last_active":"2023-10-16T14:07:45.969007+0000","last_peered":"2023-10-16T14:07:45.969007+0000","last_clean":"2023-10-16T14:07:45.969007+0000","last_became_active":"2023-10-16T14:07:43.796245+0000","last_became_peered":"2023-10-16T14:07:43.796245+0000","last_unstale":"2023-10-16T14:07:45.969007+0000","last_undegraded":"2023-10-16T14:07:45.969007+0000","last_fullsized":"2023-10-16T14:07:45.969007+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:58.913804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435318+0000","last_change":"2023-10-16T14:07:44.811959+0000","last_active":"2023-10-16T14:07:46.435318+0000","last_peered":"2023-10-16T14:07:46.435318+0000","last_clean":"2023-10-16T14:07:46.435318+0000","last_became_active":"2023-10-16T14:07:44.811698+0000","last_became_peered":"2023-10-16T14:07:44.811698+0000","last_unstale":"2023-10-16T14:07:46.435318+0000","last_undegraded":"2023-10-16T14:07:46.435318+0000","last_fullsized":"2023-10-16T14:07:46.435318+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:05:06.296499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434660+0000","last_change":"2023-10-16T14:07:43.795912+0000","last_active":"2023-10-16T14:07:46.434660+0000","last_peered":"2023-10-16T14:07:46.434660+0000","last_clean":"2023-10-16T14:07:46.434660+0000","last_became_active":"2023-10-16T14:07:43.795629+0000","last_became_peered":"2023-10-16T14:07:43.795629+0000","last_unstale":"2023-10-16T14:07:46.434660+0000","last_undegraded":"2023-10-16T14:07:46.434660+0000","last_fullsized":"2023-10-16T14:07:46.434660+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:38.077728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713933+0000","last_change":"2023-10-16T14:07:44.827360+0000","last_active":"2023-10-16T14:07:46.713933+0000","last_peered":"2023-10-16T14:07:46.713933+0000","last_clean":"2023-10-16T14:07:46.713933+0000","last_became_active":"2023-10-16T14:07:44.827104+0000","last_became_peered":"2023-10-16T14:07:44.827104+0000","last_unstale":"2023-10-16T14:07:46.713933+0000","last_undegraded":"2023-10-16T14:07:46.713933+0000","last_fullsized":"2023-10-16T14:07:46.713933+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:19.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211349+0000","last_change":"2023-10-16T14:07:43.799859+0000","last_active":"2023-10-16T14:07:46.211349+0000","last_peered":"2023-10-16T14:07:46.211349+0000","last_clean":"2023-10-16T14:07:46.211349+0000","last_became_active":"2023-10-16T14:07:43.799267+0000","last_became_peered":"2023-10-16T14:07:43.799267+0000","last_unstale":"2023-10-16T14:07:46.211349+0000","last_undegraded":"2023-10-16T14:07:46.211349+0000","last_fullsized":"2023-10-16T14:07:46.211349+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:56:50.764862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713337+0000","last_change":"2023-10-16T14:07:44.812645+0000","last_active":"2023-10-16T14:07:46.713337+0000","last_peered":"2023-10-16T14:07:46.713337+0000","last_clean":"2023-10-16T14:07:46.713337+0000","last_became_active":"2023-10-16T14:07:44.812302+0000","last_became_peered":"2023-10-16T14:07:44.812302+0000","last_unstale":"2023-10-16T14:07:46.713337+0000","last_undegraded":"2023-10-16T14:07:46.713337+0000","last_fullsized":"2023-10-16T14:07:46.713337+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:43.265721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713870+0000","last_change":"2023-10-16T14:07:43.782583+0000","last_active":"2023-10-16T14:07:46.713870+0000","last_peered":"2023-10-16T14:07:46.713870+0000","last_clean":"2023-10-16T14:07:46.713870+0000","last_became_active":"2023-10-16T14:07:43.780706+0000","last_became_peered":"2023-10-16T14:07:43.780706+0000","last_unstale":"2023-10-16T14:07:46.713870+0000","last_undegraded":"2023-10-16T14:07:46.713870+0000","last_fullsized":"2023-10-16T14:07:46.713870+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:14.043426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435144+0000","last_change":"2023-10-16T14:07:44.812473+0000","last_active":"2023-10-16T14:07:46.435144+0000","last_peered":"2023-10-16T14:07:46.435144+0000","last_clean":"2023-10-16T14:07:46.435144+0000","last_became_active":"2023-10-16T14:07:44.811563+0000","last_became_peered":"2023-10-16T14:07:44.811563+0000","last_unstale":"2023-10-16T14:07:46.435144+0000","last_undegraded":"2023-10-16T14:07:46.435144+0000","last_fullsized":"2023-10-16T14:07:46.435144+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:05.455592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969340+0000","last_change":"2023-10-16T14:07:43.797313+0000","last_active":"2023-10-16T14:07:45.969340+0000","last_peered":"2023-10-16T14:07:45.969340+0000","last_clean":"2023-10-16T14:07:45.969340+0000","last_became_active":"2023-10-16T14:07:43.796993+0000","last_became_peered":"2023-10-16T14:07:43.796993+0000","last_unstale":"2023-10-16T14:07:45.969340+0000","last_undegraded":"2023-10-16T14:07:45.969340+0000","last_fullsized":"2023-10-16T14:07:45.969340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:36.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713944+0000","last_change":"2023-10-16T14:07:44.811612+0000","last_active":"2023-10-16T14:07:46.713944+0000","last_peered":"2023-10-16T14:07:46.713944+0000","last_clean":"2023-10-16T14:07:46.713944+0000","last_became_active":"2023-10-16T14:07:44.811320+0000","last_became_peered":"2023-10-16T14:07:44.811320+0000","last_unstale":"2023-10-16T14:07:46.713944+0000","last_undegraded":"2023-10-16T14:07:46.713944+0000","last_fullsized":"2023-10-16T14:07:46.713944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:00.517821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"212'3","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969324+0000","last_change":"2023-10-16T14:07:43.799330+0000","last_active":"2023-10-16T14:07:45.969324+0000","last_peered":"2023-10-16T14:07:45.969324+0000","last_clean":"2023-10-16T14:07:45.969324+0000","last_became_active":"2023-10-16T14:07:43.799043+0000","last_became_peered":"2023-10-16T14:07:43.799043+0000","last_unstale":"2023-10-16T14:07:45.969324+0000","last_undegraded":"2023-10-16T14:07:45.969324+0000","last_fullsized":"2023-10-16T14:07:45.969324+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:18.726291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824025+0000","last_change":"2023-10-16T14:07:44.815955+0000","last_active":"2023-10-16T14:07:45.824025+0000","last_peered":"2023-10-16T14:07:45.824025+0000","last_clean":"2023-10-16T14:07:45.824025+0000","last_became_active":"2023-10-16T14:07:44.815663+0000","last_became_peered":"2023-10-16T14:07:44.815663+0000","last_unstale":"2023-10-16T14:07:45.824025+0000","last_undegraded":"2023-10-16T14:07:45.824025+0000","last_fullsized":"2023-10-16T14:07:45.824025+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:58.529404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"212'6","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712831+0000","last_change":"2023-10-16T14:07:43.790980+0000","last_active":"2023-10-16T14:07:46.712831+0000","last_peered":"2023-10-16T14:07:46.712831+0000","last_clean":"2023-10-16T14:07:46.712831+0000","last_became_active":"2023-10-16T14:07:43.790665+0000","last_became_peered":"2023-10-16T14:07:43.790665+0000","last_unstale":"2023-10-16T14:07:46.712831+0000","last_undegraded":"2023-10-16T14:07:46.712831+0000","last_fullsized":"2023-10-16T14:07:46.712831+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:24.507670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713033+0000","last_change":"2023-10-16T14:07:44.812385+0000","last_active":"2023-10-16T14:07:46.713033+0000","last_peered":"2023-10-16T14:07:46.713033+0000","last_clean":"2023-10-16T14:07:46.713033+0000","last_became_active":"2023-10-16T14:07:44.812030+0000","last_became_peered":"2023-10-16T14:07:44.812030+0000","last_unstale":"2023-10-16T14:07:46.713033+0000","last_undegraded":"2023-10-16T14:07:46.713033+0000","last_fullsized":"2023-10-16T14:07:46.713033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:09.438847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212226+0000","last_change":"2023-10-16T14:07:43.797128+0000","last_active":"2023-10-16T14:07:46.212226+0000","last_peered":"2023-10-16T14:07:46.212226+0000","last_clean":"2023-10-16T14:07:46.212226+0000","last_became_active":"2023-10-16T14:07:43.795724+0000","last_became_peered":"2023-10-16T14:07:43.795724+0000","last_unstale":"2023-10-16T14:07:46.212226+0000","last_undegraded":"2023-10-16T14:07:46.212226+0000","last_fullsized":"2023-10-16T14:07:46.212226+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:43.257103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713493+0000","last_change":"2023-10-16T14:07:44.813345+0000","last_active":"2023-10-16T14:07:46.713493+0000","last_peered":"2023-10-16T14:07:46.713493+0000","last_clean":"2023-10-16T14:07:46.713493+0000","last_became_active":"2023-10-16T14:07:44.813084+0000","last_became_peered":"2023-10-16T14:07:44.813084+0000","last_unstale":"2023-10-16T14:07:46.713493+0000","last_undegraded":"2023-10-16T14:07:46.713493+0000","last_fullsized":"2023-10-16T14:07:46.713493+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:04.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823839+0000","last_change":"2023-10-16T14:07:43.795728+0000","last_active":"2023-10-16T14:07:45.823839+0000","last_peered":"2023-10-16T14:07:45.823839+0000","last_clean":"2023-10-16T14:07:45.823839+0000","last_became_active":"2023-10-16T14:07:43.795411+0000","last_became_peered":"2023-10-16T14:07:43.795411+0000","last_unstale":"2023-10-16T14:07:45.823839+0000","last_undegraded":"2023-10-16T14:07:45.823839+0000","last_fullsized":"2023-10-16T14:07:45.823839+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:53.532327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829965+0000","last_change":"2023-10-16T14:07:44.818755+0000","last_active":"2023-10-16T14:07:45.829965+0000","last_peered":"2023-10-16T14:07:45.829965+0000","last_clean":"2023-10-16T14:07:45.829965+0000","last_became_active":"2023-10-16T14:07:44.815413+0000","last_became_peered":"2023-10-16T14:07:44.815413+0000","last_unstale":"2023-10-16T14:07:45.829965+0000","last_undegraded":"2023-10-16T14:07:45.829965+0000","last_fullsized":"2023-10-16T14:07:45.829965+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:19.201828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434214+0000","last_change":"2023-10-16T14:07:43.798522+0000","last_active":"2023-10-16T14:07:46.434214+0000","last_peered":"2023-10-16T14:07:46.434214+0000","last_clean":"2023-10-16T14:07:46.434214+0000","last_became_active":"2023-10-16T14:07:43.798243+0000","last_became_peered":"2023-10-16T14:07:43.798243+0000","last_unstale":"2023-10-16T14:07:46.434214+0000","last_undegraded":"2023-10-16T14:07:46.434214+0000","last_fullsized":"2023-10-16T14:07:46.434214+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:56:50.764862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823656+0000","last_change":"2023-10-16T14:07:44.825912+0000","last_active":"2023-10-16T14:07:45.823656+0000","last_peered":"2023-10-16T14:07:45.823656+0000","last_clean":"2023-10-16T14:07:45.823656+0000","last_became_active":"2023-10-16T14:07:44.825570+0000","last_became_peered":"2023-10-16T14:07:44.825570+0000","last_unstale":"2023-10-16T14:07:45.823656+0000","last_undegraded":"2023-10-16T14:07:45.823656+0000","last_fullsized":"2023-10-16T14:07:45.823656+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:22.319291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"212'8","reported_seq":20,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211946+0000","last_change":"2023-10-16T14:07:43.799575+0000","last_active":"2023-10-16T14:07:46.211946+0000","last_peered":"2023-10-16T14:07:46.211946+0000","last_clean":"2023-10-16T14:07:46.211946+0000","last_became_active":"2023-10-16T14:07:43.798872+0000","last_became_peered":"2023-10-16T14:07:43.798872+0000","last_unstale":"2023-10-16T14:07:46.211946+0000","last_undegraded":"2023-10-16T14:07:46.211946+0000","last_fullsized":"2023-10-16T14:07:46.211946+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:10.696490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824215+0000","last_change":"2023-10-16T14:07:44.826736+0000","last_active":"2023-10-16T14:07:45.824215+0000","last_peered":"2023-10-16T14:07:45.824215+0000","last_clean":"2023-10-16T14:07:45.824215+0000","last_became_active":"2023-10-16T14:07:44.826363+0000","last_became_peered":"2023-10-16T14:07:44.826363+0000","last_unstale":"2023-10-16T14:07:45.824215+0000","last_undegraded":"2023-10-16T14:07:45.824215+0000","last_fullsized":"2023-10-16T14:07:45.824215+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:43.210356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969571+0000","last_change":"2023-10-16T14:07:43.799382+0000","last_active":"2023-10-16T14:07:45.969571+0000","last_peered":"2023-10-16T14:07:45.969571+0000","last_clean":"2023-10-16T14:07:45.969571+0000","last_became_active":"2023-10-16T14:07:43.798998+0000","last_became_peered":"2023-10-16T14:07:43.798998+0000","last_unstale":"2023-10-16T14:07:45.969571+0000","last_undegraded":"2023-10-16T14:07:45.969571+0000","last_fullsized":"2023-10-16T14:07:45.969571+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:59.503196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830612+0000","last_change":"2023-10-16T14:07:44.818549+0000","last_active":"2023-10-16T14:07:45.830612+0000","last_peered":"2023-10-16T14:07:45.830612+0000","last_clean":"2023-10-16T14:07:45.830612+0000","last_became_active":"2023-10-16T14:07:44.815471+0000","last_became_peered":"2023-10-16T14:07:44.815471+0000","last_unstale":"2023-10-16T14:07:45.830612+0000","last_undegraded":"2023-10-16T14:07:45.830612+0000","last_fullsized":"2023-10-16T14:07:45.830612+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:19.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435375+0000","last_change":"2023-10-16T14:07:43.789962+0000","last_active":"2023-10-16T14:07:46.435375+0000","last_peered":"2023-10-16T14:07:46.435375+0000","last_clean":"2023-10-16T14:07:46.435375+0000","last_became_active":"2023-10-16T14:07:43.789699+0000","last_became_peered":"2023-10-16T14:07:43.789699+0000","last_unstale":"2023-10-16T14:07:46.435375+0000","last_undegraded":"2023-10-16T14:07:46.435375+0000","last_fullsized":"2023-10-16T14:07:46.435375+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:43.257103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830430+0000","last_change":"2023-10-16T14:07:44.808623+0000","last_active":"2023-10-16T14:07:45.830430+0000","last_peered":"2023-10-16T14:07:45.830430+0000","last_clean":"2023-10-16T14:07:45.830430+0000","last_became_active":"2023-10-16T14:07:44.808282+0000","last_became_peered":"2023-10-16T14:07:44.808282+0000","last_unstale":"2023-10-16T14:07:45.830430+0000","last_undegraded":"2023-10-16T14:07:45.830430+0000","last_fullsized":"2023-10-16T14:07:45.830430+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:01.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212227+0000","last_change":"2023-10-16T14:07:43.799777+0000","last_active":"2023-10-16T14:07:46.212227+0000","last_peered":"2023-10-16T14:07:46.212227+0000","last_clean":"2023-10-16T14:07:46.212227+0000","last_became_active":"2023-10-16T14:07:43.799481+0000","last_became_peered":"2023-10-16T14:07:43.799481+0000","last_unstale":"2023-10-16T14:07:46.212227+0000","last_undegraded":"2023-10-16T14:07:46.212227+0000","last_fullsized":"2023-10-16T14:07:46.212227+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:30.490818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829359+0000","last_change":"2023-10-16T14:07:44.820436+0000","last_active":"2023-10-16T14:07:45.829359+0000","last_peered":"2023-10-16T14:07:45.829359+0000","last_clean":"2023-10-16T14:07:45.829359+0000","last_became_active":"2023-10-16T14:07:44.820164+0000","last_became_peered":"2023-10-16T14:07:44.820164+0000","last_unstale":"2023-10-16T14:07:45.829359+0000","last_undegraded":"2023-10-16T14:07:45.829359+0000","last_fullsized":"2023-10-16T14:07:45.829359+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:55.293783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713920+0000","last_change":"2023-10-16T14:07:43.786727+0000","last_active":"2023-10-16T14:07:46.713920+0000","last_peered":"2023-10-16T14:07:46.713920+0000","last_clean":"2023-10-16T14:07:46.713920+0000","last_became_active":"2023-10-16T14:07:43.786272+0000","last_became_peered":"2023-10-16T14:07:43.786272+0000","last_unstale":"2023-10-16T14:07:46.713920+0000","last_undegraded":"2023-10-16T14:07:46.713920+0000","last_fullsized":"2023-10-16T14:07:46.713920+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:18.187203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823548+0000","last_change":"2023-10-16T14:07:44.812592+0000","last_active":"2023-10-16T14:07:45.823548+0000","last_peered":"2023-10-16T14:07:45.823548+0000","last_clean":"2023-10-16T14:07:45.823548+0000","last_became_active":"2023-10-16T14:07:44.812247+0000","last_became_peered":"2023-10-16T14:07:44.812247+0000","last_unstale":"2023-10-16T14:07:45.823548+0000","last_undegraded":"2023-10-16T14:07:45.823548+0000","last_fullsized":"2023-10-16T14:07:45.823548+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:40.141847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712595+0000","last_change":"2023-10-16T14:07:43.798513+0000","last_active":"2023-10-16T14:07:46.712595+0000","last_peered":"2023-10-16T14:07:46.712595+0000","last_clean":"2023-10-16T14:07:46.712595+0000","last_became_active":"2023-10-16T14:07:43.797262+0000","last_became_peered":"2023-10-16T14:07:43.797262+0000","last_unstale":"2023-10-16T14:07:46.712595+0000","last_undegraded":"2023-10-16T14:07:46.712595+0000","last_fullsized":"2023-10-16T14:07:46.712595+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:36.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.714062+0000","last_change":"2023-10-16T14:07:44.824444+0000","last_active":"2023-10-16T14:07:46.714062+0000","last_peered":"2023-10-16T14:07:46.714062+0000","last_clean":"2023-10-16T14:07:46.714062+0000","last_became_active":"2023-10-16T14:07:44.824159+0000","last_became_peered":"2023-10-16T14:07:44.824159+0000","last_unstale":"2023-10-16T14:07:46.714062+0000","last_undegraded":"2023-10-16T14:07:46.714062+0000","last_fullsized":"2023-10-16T14:07:46.714062+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:14.036727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968936+0000","last_change":"2023-10-16T14:07:43.797313+0000","last_active":"2023-10-16T14:07:45.968936+0000","last_peered":"2023-10-16T14:07:45.968936+0000","last_clean":"2023-10-16T14:07:45.968936+0000","last_became_active":"2023-10-16T14:07:43.797004+0000","last_became_peered":"2023-10-16T14:07:43.797004+0000","last_unstale":"2023-10-16T14:07:45.968936+0000","last_undegraded":"2023-10-16T14:07:45.968936+0000","last_fullsized":"2023-10-16T14:07:45.968936+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:13.022102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.828886+0000","last_change":"2023-10-16T14:07:44.825441+0000","last_active":"2023-10-16T14:07:45.828886+0000","last_peered":"2023-10-16T14:07:45.828886+0000","last_clean":"2023-10-16T14:07:45.828886+0000","last_became_active":"2023-10-16T14:07:44.825154+0000","last_became_peered":"2023-10-16T14:07:44.825154+0000","last_unstale":"2023-10-16T14:07:45.828886+0000","last_undegraded":"2023-10-16T14:07:45.828886+0000","last_fullsized":"2023-10-16T14:07:45.828886+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:44.027350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712975+0000","last_change":"2023-10-16T14:07:43.797330+0000","last_active":"2023-10-16T14:07:46.712975+0000","last_peered":"2023-10-16T14:07:46.712975+0000","last_clean":"2023-10-16T14:07:46.712975+0000","last_became_active":"2023-10-16T14:07:43.796947+0000","last_became_peered":"2023-10-16T14:07:43.796947+0000","last_unstale":"2023-10-16T14:07:46.712975+0000","last_undegraded":"2023-10-16T14:07:46.712975+0000","last_fullsized":"2023-10-16T14:07:46.712975+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:32.287903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968763+0000","last_change":"2023-10-16T14:07:44.814498+0000","last_active":"2023-10-16T14:07:45.968763+0000","last_peered":"2023-10-16T14:07:45.968763+0000","last_clean":"2023-10-16T14:07:45.968763+0000","last_became_active":"2023-10-16T14:07:44.814195+0000","last_became_peered":"2023-10-16T14:07:44.814195+0000","last_unstale":"2023-10-16T14:07:45.968763+0000","last_undegraded":"2023-10-16T14:07:45.968763+0000","last_fullsized":"2023-10-16T14:07:45.968763+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:31.212385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829801+0000","last_change":"2023-10-16T14:07:43.796252+0000","last_active":"2023-10-16T14:07:45.829801+0000","last_peered":"2023-10-16T14:07:45.829801+0000","last_clean":"2023-10-16T14:07:45.829801+0000","last_became_active":"2023-10-16T14:07:43.795441+0000","last_became_peered":"2023-10-16T14:07:43.795441+0000","last_unstale":"2023-10-16T14:07:45.829801+0000","last_undegraded":"2023-10-16T14:07:45.829801+0000","last_fullsized":"2023-10-16T14:07:45.829801+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:14.043426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969277+0000","last_change":"2023-10-16T14:07:44.819106+0000","last_active":"2023-10-16T14:07:45.969277+0000","last_peered":"2023-10-16T14:07:45.969277+0000","last_clean":"2023-10-16T14:07:45.969277+0000","last_became_active":"2023-10-16T14:07:44.818816+0000","last_became_peered":"2023-10-16T14:07:44.818816+0000","last_unstale":"2023-10-16T14:07:45.969277+0000","last_undegraded":"2023-10-16T14:07:45.969277+0000","last_fullsized":"2023-10-16T14:07:45.969277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:35.069827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212378+0000","last_change":"2023-10-16T14:07:43.801180+0000","last_active":"2023-10-16T14:07:46.212378+0000","last_peered":"2023-10-16T14:07:46.212378+0000","last_clean":"2023-10-16T14:07:46.212378+0000","last_became_active":"2023-10-16T14:07:43.800743+0000","last_became_peered":"2023-10-16T14:07:43.800743+0000","last_unstale":"2023-10-16T14:07:46.212378+0000","last_undegraded":"2023-10-16T14:07:46.212378+0000","last_fullsized":"2023-10-16T14:07:46.212378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:05:05.281874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.825125+0000","last_change":"2023-10-16T14:07:44.832039+0000","last_active":"2023-10-16T14:07:45.825125+0000","last_peered":"2023-10-16T14:07:45.825125+0000","last_clean":"2023-10-16T14:07:45.825125+0000","last_became_active":"2023-10-16T14:07:44.831730+0000","last_became_peered":"2023-10-16T14:07:44.831730+0000","last_unstale":"2023-10-16T14:07:45.825125+0000","last_undegraded":"2023-10-16T14:07:45.825125+0000","last_fullsized":"2023-10-16T14:07:45.825125+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:01.519902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824257+0000","last_change":"2023-10-16T14:07:43.793950+0000","last_active":"2023-10-16T14:07:45.824257+0000","last_peered":"2023-10-16T14:07:45.824257+0000","last_clean":"2023-10-16T14:07:45.824257+0000","last_became_active":"2023-10-16T14:07:43.793645+0000","last_became_peered":"2023-10-16T14:07:43.793645+0000","last_unstale":"2023-10-16T14:07:45.824257+0000","last_undegraded":"2023-10-16T14:07:45.824257+0000","last_fullsized":"2023-10-16T14:07:45.824257+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:42.416005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435384+0000","last_change":"2023-10-16T14:07:44.813393+0000","last_active":"2023-10-16T14:07:46.435384+0000","last_peered":"2023-10-16T14:07:46.435384+0000","last_clean":"2023-10-16T14:07:46.435384+0000","last_became_active":"2023-10-16T14:07:44.813109+0000","last_became_peered":"2023-10-16T14:07:44.813109+0000","last_unstale":"2023-10-16T14:07:46.435384+0000","last_undegraded":"2023-10-16T14:07:46.435384+0000","last_fullsized":"2023-10-16T14:07:46.435384+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:23.227382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824135+0000","last_change":"2023-10-16T14:07:43.789095+0000","last_active":"2023-10-16T14:07:45.824135+0000","last_peered":"2023-10-16T14:07:45.824135+0000","last_clean":"2023-10-16T14:07:45.824135+0000","last_became_active":"2023-10-16T14:07:43.788816+0000","last_became_peered":"2023-10-16T14:07:43.788816+0000","last_unstale":"2023-10-16T14:07:45.824135+0000","last_undegraded":"2023-10-16T14:07:45.824135+0000","last_fullsized":"2023-10-16T14:07:45.824135+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:55.334715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435304+0000","last_change":"2023-10-16T14:07:44.816045+0000","last_active":"2023-10-16T14:07:46.435304+0000","last_peered":"2023-10-16T14:07:46.435304+0000","last_clean":"2023-10-16T14:07:46.435304+0000","last_became_active":"2023-10-16T14:07:44.815730+0000","last_became_peered":"2023-10-16T14:07:44.815730+0000","last_unstale":"2023-10-16T14:07:46.435304+0000","last_undegraded":"2023-10-16T14:07:46.435304+0000","last_fullsized":"2023-10-16T14:07:46.435304+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:22.305405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435568+0000","last_change":"2023-10-16T14:07:43.787074+0000","last_active":"2023-10-16T14:07:46.435568+0000","last_peered":"2023-10-16T14:07:46.435568+0000","last_clean":"2023-10-16T14:07:46.435568+0000","last_became_active":"2023-10-16T14:07:43.786733+0000","last_became_peered":"2023-10-16T14:07:43.786733+0000","last_unstale":"2023-10-16T14:07:46.435568+0000","last_undegraded":"2023-10-16T14:07:46.435568+0000","last_fullsized":"2023-10-16T14:07:46.435568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:10.696490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211916+0000","last_change":"2023-10-16T14:07:44.828126+0000","last_active":"2023-10-16T14:07:46.211916+0000","last_peered":"2023-10-16T14:07:46.211916+0000","last_clean":"2023-10-16T14:07:46.211916+0000","last_became_active":"2023-10-16T14:07:44.827813+0000","last_became_peered":"2023-10-16T14:07:44.827813+0000","last_unstale":"2023-10-16T14:07:46.211916+0000","last_undegraded":"2023-10-16T14:07:46.211916+0000","last_fullsized":"2023-10-16T14:07:46.211916+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:16.459394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830979+0000","last_change":"2023-10-16T14:07:43.778246+0000","last_active":"2023-10-16T14:07:45.830979+0000","last_peered":"2023-10-16T14:07:45.830979+0000","last_clean":"2023-10-16T14:07:45.830979+0000","last_became_active":"2023-10-16T14:07:43.776838+0000","last_became_peered":"2023-10-16T14:07:43.776838+0000","last_unstale":"2023-10-16T14:07:45.830979+0000","last_undegraded":"2023-10-16T14:07:45.830979+0000","last_fullsized":"2023-10-16T14:07:45.830979+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:31.810785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211979+0000","last_change":"2023-10-16T14:07:44.820067+0000","last_active":"2023-10-16T14:07:46.211979+0000","last_peered":"2023-10-16T14:07:46.211979+0000","last_clean":"2023-10-16T14:07:46.211979+0000","last_became_active":"2023-10-16T14:07:44.819780+0000","last_became_peered":"2023-10-16T14:07:44.819780+0000","last_unstale":"2023-10-16T14:07:46.211979+0000","last_undegraded":"2023-10-16T14:07:46.211979+0000","last_fullsized":"2023-10-16T14:07:46.211979+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:23.227382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.831520+0000","last_change":"2023-10-16T14:07:43.794159+0000","last_active":"2023-10-16T14:07:45.831520+0000","last_peered":"2023-10-16T14:07:45.831520+0000","last_clean":"2023-10-16T14:07:45.831520+0000","last_became_active":"2023-10-16T14:07:43.793850+0000","last_became_peered":"2023-10-16T14:07:43.793850+0000","last_unstale":"2023-10-16T14:07:45.831520+0000","last_undegraded":"2023-10-16T14:07:45.831520+0000","last_fullsized":"2023-10-16T14:07:45.831520+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:17.106938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829096+0000","last_change":"2023-10-16T14:07:44.814739+0000","last_active":"2023-10-16T14:07:45.829096+0000","last_peered":"2023-10-16T14:07:45.829096+0000","last_clean":"2023-10-16T14:07:45.829096+0000","last_became_active":"2023-10-16T14:07:44.812428+0000","last_became_peered":"2023-10-16T14:07:44.812428+0000","last_unstale":"2023-10-16T14:07:45.829096+0000","last_undegraded":"2023-10-16T14:07:45.829096+0000","last_fullsized":"2023-10-16T14:07:45.829096+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:50.817563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830063+0000","last_change":"2023-10-16T14:07:43.798547+0000","last_active":"2023-10-16T14:07:45.830063+0000","last_peered":"2023-10-16T14:07:45.830063+0000","last_clean":"2023-10-16T14:07:45.830063+0000","last_became_active":"2023-10-16T14:07:43.797762+0000","last_became_peered":"2023-10-16T14:07:43.797762+0000","last_unstale":"2023-10-16T14:07:45.830063+0000","last_undegraded":"2023-10-16T14:07:45.830063+0000","last_fullsized":"2023-10-16T14:07:45.830063+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:36.893741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968811+0000","last_change":"2023-10-16T14:07:44.817973+0000","last_active":"2023-10-16T14:07:45.968811+0000","last_peered":"2023-10-16T14:07:45.968811+0000","last_clean":"2023-10-16T14:07:45.968811+0000","last_became_active":"2023-10-16T14:07:44.817652+0000","last_became_peered":"2023-10-16T14:07:44.817652+0000","last_unstale":"2023-10-16T14:07:45.968811+0000","last_undegraded":"2023-10-16T14:07:45.968811+0000","last_fullsized":"2023-10-16T14:07:45.968811+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:25.522295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712610+0000","last_change":"2023-10-16T14:07:43.799949+0000","last_active":"2023-10-16T14:07:46.712610+0000","last_peered":"2023-10-16T14:07:46.712610+0000","last_clean":"2023-10-16T14:07:46.712610+0000","last_became_active":"2023-10-16T14:07:43.799257+0000","last_became_peered":"2023-10-16T14:07:43.799257+0000","last_unstale":"2023-10-16T14:07:46.712610+0000","last_undegraded":"2023-10-16T14:07:46.712610+0000","last_fullsized":"2023-10-16T14:07:46.712610+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:20.648407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823157+0000","last_change":"2023-10-16T14:07:44.827336+0000","last_active":"2023-10-16T14:07:45.823157+0000","last_peered":"2023-10-16T14:07:45.823157+0000","last_clean":"2023-10-16T14:07:45.823157+0000","last_became_active":"2023-10-16T14:07:44.825558+0000","last_became_peered":"2023-10-16T14:07:44.825558+0000","last_unstale":"2023-10-16T14:07:45.823157+0000","last_undegraded":"2023-10-16T14:07:45.823157+0000","last_fullsized":"2023-10-16T14:07:45.823157+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:40.029184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211814+0000","last_change":"2023-10-16T14:07:43.797242+0000","last_active":"2023-10-16T14:07:46.211814+0000","last_peered":"2023-10-16T14:07:46.211814+0000","last_clean":"2023-10-16T14:07:46.211814+0000","last_became_active":"2023-10-16T14:07:43.796965+0000","last_became_peered":"2023-10-16T14:07:43.796965+0000","last_unstale":"2023-10-16T14:07:46.211814+0000","last_undegraded":"2023-10-16T14:07:46.211814+0000","last_fullsized":"2023-10-16T14:07:46.211814+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:32.054938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'329","reported_seq":1128,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-10-16T14:07:44.803382+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:07:44.803382+0000","last_peered":"2023-10-16T14:07:44.803382+0000","last_clean":"2023-10-16T14:07:44.803382+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:07:44.803382+0000","last_undegraded":"2023-10-16T14:07:44.803382+0000","last_fullsized":"2023-10-16T14:07:44.803382+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":329,"ondisk_log_size":329,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52416,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319424,"statfs":{"total":96636764160,"available":96583090176,"internally_reserved":0,"allocated":3264512,"data_stored":1886153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.381,"15min":1.365},"min":{"1min":0.77800000000000002,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7050000000000001,"5min":1.792,"15min":2.0659999999999998},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4379999999999999,"15min":1.4259999999999999},"min":{"1min":1.048,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.7190000000000001,"5min":1.829,"15min":2.0179999999999998},"last":1.395}]},{"osd":1,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.222,"15min":1.248},"min":{"1min":0.86499999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5369999999999999,"5min":1.776,"15min":1.8999999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.2769999999999999,"15min":1.2709999999999999},"min":{"1min":0.85399999999999998,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.0620000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.292,"15min":1.278},"min":{"1min":0.96699999999999997,"5min":0.89400000000000002,"15min":0.86199999999999999},"max":{"1min":1.714,"5min":1.8109999999999999,"15min":1.8700000000000001},"last":1.347},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.395,"15min":1.3740000000000001},"min":{"1min":0.97299999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.6879999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4430000000000001,"15min":1.431},"min":{"1min":0.998,"5min":0.96699999999999997,"15min":0.91300000000000003},"max":{"1min":1.651,"5min":1.9119999999999999,"15min":2.097},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.4610000000000001,"15min":1.4550000000000001},"min":{"1min":1.1539999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8740000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.5620000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.2210000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76600000000000001},"max":{"1min":1.581,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.373,"15min":1.411},"min":{"1min":0.97799999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.675,"5min":1.833,"15min":2.0590000000000002},"last":1.6279999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3759999999999999,"15min":1.3879999999999999},"min":{"1min":1.018,"5min":0.88700000000000001,"15min":0.879},"max":{"1min":1.8169999999999999,"5min":2.0590000000000002,"15min":2.3730000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.5389999999999999,"15min":1.5580000000000001},"min":{"1min":1.0780000000000001,"5min":1,"15min":0.91500000000000004},"max":{"1min":1.869,"5min":1.9770000000000001,"15min":2.2229999999999999},"last":2.0609999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6259999999999999,"15min":1.647},"min":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":1.806,"5min":2.056,"15min":2.4169999999999998},"last":1.968},{"interface":"front","average":{"1min":1.52,"5min":1.4750000000000001,"15min":1.5},"min":{"1min":0.98899999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.145,"5min":2.145,"15min":2.423},"last":2.0110000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543268,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42100,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329740,"statfs":{"total":96636764160,"available":96593653760,"internally_reserved":0,"allocated":2662400,"data_stored":1295681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.357,"15min":1.3380000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.76800000000000002},"max":{"1min":1.587,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.3640000000000001,"15min":1.3759999999999999},"min":{"1min":1.0549999999999999,"5min":0.99299999999999999,"15min":0.91800000000000004},"max":{"1min":1.508,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.349}]},{"osd":1,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.228,"15min":1.2330000000000001},"min":{"1min":0.93600000000000005,"5min":0.83499999999999996,"15min":0.71799999999999997},"max":{"1min":1.5940000000000001,"5min":2.008,"15min":2.008},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.224,"15min":1.2509999999999999},"min":{"1min":0.87,"5min":0.82399999999999995,"15min":0.76800000000000002},"max":{"1min":1.748,"5min":1.77,"15min":1.9419999999999999},"last":0.94299999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.246,"15min":1.26},"min":{"1min":0.93799999999999994,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.6679999999999999,"5min":1.776,"15min":1.78},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":1.0680000000000001,"5min":0.96899999999999997,"15min":0.82199999999999995},"max":{"1min":1.647,"5min":1.8200000000000001,"15min":1.9390000000000001},"last":0.88500000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.401,"15min":1.363},"min":{"1min":1.147,"5min":0.96499999999999997,"15min":0.91600000000000004},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.4119999999999999,"15min":1.4159999999999999},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.92600000000000005},"max":{"1min":1.8440000000000001,"5min":1.8799999999999999,"15min":1.9259999999999999},"last":1.234}]},{"osd":4,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.431,"15min":1.4119999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.0529999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.575,"15min":1.536},"min":{"1min":1.2490000000000001,"5min":0.90500000000000003,"15min":0.77000000000000002},"max":{"1min":1.948,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.137}]},{"osd":5,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3819999999999999,"15min":1.3640000000000001},"min":{"1min":1.0209999999999999,"5min":0.85299999999999998,"15min":0.80800000000000005},"max":{"1min":1.716,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":1.177},{"interface":"front","average":{"1min":1.506,"5min":1.5209999999999999,"15min":1.5029999999999999},"min":{"1min":1.0329999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1909999999999998,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.3680000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5229999999999999,"15min":1.5069999999999999},"min":{"1min":1.109,"5min":0.94599999999999995,"15min":0.93200000000000005},"max":{"1min":2.004,"5min":2.2730000000000001,"15min":2.339},"last":1.397},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5780000000000001,"15min":1.5449999999999999},"min":{"1min":1.264,"5min":1.0649999999999999,"15min":0.95299999999999996},"max":{"1min":1.8080000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.0780000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543269,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29620,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342220,"statfs":{"total":96636764160,"available":96606433280,"internally_reserved":0,"allocated":2662400,"data_stored":1295681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.2310000000000001},"min":{"1min":0.81699999999999995,"5min":0.77500000000000002,"15min":0.76000000000000001},"max":{"1min":1.492,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.226,"15min":1.2529999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.712,"5min":1.7729999999999999,"15min":1.7769999999999999},"last":1.411}]},{"osd":1,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.93100000000000005,"5min":0.91200000000000003,"15min":0.85099999999999998},"max":{"1min":1.5449999999999999,"5min":1.823,"15min":1.823},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.88,"5min":0.754,"15min":0.754},"max":{"1min":1.591,"5min":1.873,"15min":1.917},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3759999999999999,"15min":1.3480000000000001},"min":{"1min":0.98699999999999999,"5min":0.89900000000000002,"15min":0.877},"max":{"1min":1.7490000000000001,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.333,"15min":1.3340000000000001},"min":{"1min":1.0169999999999999,"5min":0.90400000000000003,"15min":0.89100000000000001},"max":{"1min":1.764,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.375}]},{"osd":3,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.393,"15min":1.4199999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93100000000000005},"max":{"1min":1.7310000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.4390000000000001,"15min":1.4470000000000001},"min":{"1min":1.0629999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":1.9990000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.583}]},{"osd":4,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.296,"15min":1.284},"min":{"1min":0.92600000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6699999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.431,"15min":1.4319999999999999},"min":{"1min":1.135,"5min":0.87,"15min":0.84699999999999998},"max":{"1min":1.907,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.2989999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":0.91600000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.671,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4710000000000001,"15min":1.482},"min":{"1min":1.111,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8520000000000001,"5min":2.391,"15min":2.391},"last":1.8009999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.5269999999999999,"15min":1.5529999999999999},"min":{"1min":1.073,"5min":1.0269999999999999,"15min":0.97999999999999998},"max":{"1min":2.177,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.653},{"interface":"front","average":{"1min":1.597,"5min":1.589,"15min":1.6040000000000001},"min":{"1min":1.2509999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.1560000000000001,"5min":2.681,"15min":2.681},"last":1.839}]}]},{"osd":4,"up_from":14,"seq":60129543269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64320,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307520,"statfs":{"total":96636764160,"available":96570900480,"internally_reserved":0,"allocated":2666496,"data_stored":1291575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.226,"15min":1.218},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.478,"5min":1.8420000000000001,"15min":1.9039999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.196,"15min":1.2050000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8160000000000001,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.071}]},{"osd":1,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.288,"15min":1.2629999999999999},"min":{"1min":0.89900000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5640000000000001,"5min":1.7090000000000001,"15min":1.78},"last":1.004},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3200000000000001,"15min":1.3029999999999999},"min":{"1min":1.046,"5min":0.875,"15min":0.875},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.1539999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.339,"15min":1.321},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.653,"5min":1.865,"15min":1.865},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.383,"15min":1.389},"min":{"1min":1.0960000000000001,"5min":1.0129999999999999,"15min":0.84499999999999997},"max":{"1min":1.6120000000000001,"5min":1.7210000000000001,"15min":2.0110000000000001},"last":1.482}]},{"osd":3,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.427,"15min":1.425},"min":{"1min":0.93200000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":1.726,"5min":1.875,"15min":1.9319999999999999},"last":1.611}]},{"osd":5,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.325,"15min":1.3440000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.9930000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.3160000000000001,"15min":1.337},"min":{"1min":0.98999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.9510000000000001},"last":1.542}]},{"osd":6,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.369,"15min":1.401},"min":{"1min":0.95899999999999996,"5min":0.92500000000000004,"15min":0.86699999999999999},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.1179999999999999},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.4990000000000001,"15min":1.532},"min":{"1min":1.2450000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.859,"5min":1.859,"15min":2.6429999999999998},"last":1.587}]},{"osd":7,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4650000000000001,"15min":1.502},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.931,"5min":2.0049999999999999,"15min":2.1949999999999998},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.633,"15min":1.6459999999999999},"min":{"1min":1.1719999999999999,"5min":1.008,"15min":1.008},"max":{"1min":2.02,"5min":2.1349999999999998,"15min":2.4529999999999998},"last":1.8180000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51828,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320012,"statfs":{"total":96636764160,"available":96583692288,"internally_reserved":0,"allocated":2662400,"data_stored":1289575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.92200000000000004,"15min":0.95599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.173,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.214,"15min":1.2350000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.554,"5min":1.798,"15min":2.0579999999999998},"last":1.1279999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.0649999999999999,"15min":1.0940000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.296,"5min":1.855,"15min":1.9750000000000001},"last":1.194},{"interface":"front","average":{"1min":1.004,"5min":1.0920000000000001,"15min":1.1299999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6040000000000001,"5min":2.077,"15min":2.077},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.181,"15min":1.2150000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.8700000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.8420000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.4079999999999999,"15min":1.4610000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.879,"5min":1.879,"15min":2.1230000000000002},"last":1.9319999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.502,"15min":1.516},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.8009999999999999,"5min":1.9550000000000001,"15min":2.0489999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.448,"15min":1.4610000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":1.8580000000000001,"15min":2.2109999999999999},"last":1.732}]},{"osd":5,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.5149999999999999,"15min":1.4890000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":2.1019999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.5489999999999999,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":1.6639999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.512,"15min":1.516},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8260000000000001,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.48,"15min":1.5},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0249999999999999,"5min":2.407,"15min":2.407},"last":2.4180000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.548,"15min":1.6160000000000001},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":2.0449999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6619999999999999,"15min":1.6739999999999999},"min":{"1min":0.90600000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0419999999999998,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.7929999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543267,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51844,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319996,"statfs":{"total":96636764160,"available":96583675904,"internally_reserved":0,"allocated":2674688,"data_stored":1296375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.97299999999999998,"15min":0.97099999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5569999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.1830000000000001,"15min":1.171},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.88800000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1759999999999999,"15min":1.1839999999999999},"min":{"1min":0.875,"5min":0.80900000000000005,"15min":0.78100000000000003},"max":{"1min":1.6779999999999999,"5min":1.827,"15min":1.859},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.343,"5min":1.3009999999999999,"15min":1.2709999999999999},"min":{"1min":0.89900000000000002,"5min":0.87,"15min":0.70199999999999996},"max":{"1min":1.768,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.58099999999999996}]},{"osd":3,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.149,"15min":1.127},"min":{"1min":0.872,"5min":0.80500000000000005,"15min":0.74199999999999999},"max":{"1min":1.75,"5min":1.75,"15min":2.2719999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4179999999999999,"15min":1.4019999999999999},"min":{"1min":1.0580000000000001,"5min":0.96299999999999997,"15min":0.89100000000000001},"max":{"1min":1.8919999999999999,"5min":1.9279999999999999,"15min":2.0760000000000001},"last":0.625}]},{"osd":4,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.518,"15min":1.5249999999999999},"min":{"1min":1.1899999999999999,"5min":1.089,"15min":1.089},"max":{"1min":2.0590000000000002,"5min":2.0710000000000002,"15min":2.488},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.524,"15min":1.502},"min":{"1min":1.0700000000000001,"5min":0.999,"15min":0.999},"max":{"1min":2.0659999999999998,"5min":2.2530000000000001,"15min":2.4180000000000001},"last":1.169}]},{"osd":5,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.5940000000000001,"15min":1.5780000000000001},"min":{"1min":1.353,"5min":1.01,"15min":1.01},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4970000000000001,"15min":1.476},"min":{"1min":1.1160000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.921,"5min":2.23,"15min":2.3260000000000001},"last":1.6399999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.556,"15min":1.5269999999999999},"min":{"1min":1.48,"5min":1.056,"15min":1.0349999999999999},"max":{"1min":2.0379999999999998,"5min":2.323,"15min":2.6400000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.6499999999999999,"15min":1.6439999999999999},"min":{"1min":1.2889999999999999,"5min":1.25,"15min":1.2},"max":{"1min":2.1800000000000002,"5min":2.2999999999999998,"15min":2.7749999999999999},"last":1.1379999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.7070000000000001,"15min":1.673},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":0.96699999999999997},"max":{"1min":2.25,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.6619999999999999,"15min":1.653},"min":{"1min":1.2310000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"max":{"1min":2.149,"5min":2.3319999999999999,"15min":2.9159999999999999},"last":1.4890000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543267,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51828,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320012,"statfs":{"total":96636764160,"available":96583692288,"internally_reserved":0,"allocated":2662400,"data_stored":1290993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.96699999999999997,"15min":0.94499999999999995},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.624},"max":{"1min":1.387,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1739999999999999,"15min":1.1619999999999999},"min":{"1min":0.77200000000000002,"5min":0.74099999999999999,"15min":0.72399999999999998},"max":{"1min":1.579,"5min":1.851,"15min":1.851},"last":1.8720000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.073,"15min":1.05},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.65000000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.234,"5min":1.2629999999999999,"15min":1.274},"min":{"1min":0.88600000000000001,"5min":0.84599999999999997,"15min":0.80900000000000005},"max":{"1min":1.677,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.5509999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.4330000000000001,"15min":1.413},"min":{"1min":1.153,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.3009999999999999,"15min":1.306},"min":{"1min":0.89600000000000002,"5min":0.872,"15min":0.81499999999999995},"max":{"1min":1.7150000000000001,"5min":1.835,"15min":1.9299999999999999},"last":1.0229999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3999999999999999,"15min":1.375},"min":{"1min":1.0780000000000001,"5min":0.97299999999999998,"15min":0.92200000000000004},"max":{"1min":1.776,"5min":1.901,"15min":2.1259999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.5669999999999999,"15min":1.5389999999999999},"min":{"1min":1.264,"5min":1.151,"15min":1.042},"max":{"1min":2.133,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.681}]},{"osd":5,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.0589999999999999},"max":{"1min":1.8240000000000001,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.502,"15min":1.504},"min":{"1min":1.05,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.849,"5min":1.9359999999999999,"15min":2.1440000000000001},"last":1.4059999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.671,"15min":1.6479999999999999},"min":{"1min":1.3049999999999999,"5min":1.238,"15min":1.161},"max":{"1min":2.2309999999999999,"5min":2.4239999999999999,"15min":2.7200000000000002},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.631,"15min":1.6439999999999999},"min":{"1min":1.135,"5min":1.091,"15min":1.091},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.6379999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.734},"min":{"1min":1.262,"5min":1.262,"15min":1.234},"max":{"1min":2.528,"5min":2.528,"15min":2.528},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.665,"15min":1.671},"min":{"1min":1.2889999999999999,"5min":1.1859999999999999,"15min":1.1419999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.5169999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543268,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48252,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323588,"statfs":{"total":96636764160,"available":96587354112,"internally_reserved":0,"allocated":3260416,"data_stored":1886071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.96199999999999997,"15min":0.93000000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.292,"5min":1.6699999999999999,"15min":1.891},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.2,"5min":1.1950000000000001,"15min":1.1899999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.48699999999999999},"max":{"1min":1.659,"5min":1.851,"15min":1.9299999999999999},"last":0.88500000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.22,"15min":1.173},"min":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.76000000000000001},"max":{"1min":1.641,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2509999999999999,"15min":1.252},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.734,"5min":1.734,"15min":1.887},"last":0.75}]},{"osd":3,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.1970000000000001,"15min":1.1759999999999999},"min":{"1min":0.81200000000000006,"5min":0.79200000000000004,"15min":0.72599999999999998},"max":{"1min":1.609,"5min":1.8520000000000001,"15min":1.8600000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.446,"15min":1.4039999999999999},"min":{"1min":1.1120000000000001,"5min":0.96399999999999997,"15min":0.73799999999999999},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.383,"15min":1.4019999999999999},"min":{"1min":1.1120000000000001,"5min":0.98599999999999999,"15min":0.91600000000000004},"max":{"1min":1.819,"5min":2.0070000000000001,"15min":2.048},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.5780000000000001,"15min":1.526},"min":{"1min":1.133,"5min":1.133,"15min":1.0169999999999999},"max":{"1min":2.601,"5min":2.601,"15min":2.601},"last":1.2809999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.52,"15min":1.5209999999999999},"min":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0740000000000001},"max":{"1min":2.2250000000000001,"5min":2.3279999999999998,"15min":2.4300000000000002},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5009999999999999,"15min":1.5089999999999999},"min":{"1min":1.22,"5min":1.111,"15min":0.92500000000000004},"max":{"1min":1.915,"5min":2.2829999999999999,"15min":2.2869999999999999},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6579999999999999,"15min":1.629},"min":{"1min":1.266,"5min":1.2130000000000001,"15min":1.1319999999999999},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.2},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.627,"15min":1.623},"min":{"1min":1.1499999999999999,"5min":1.1140000000000001,"15min":0.97199999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.3769999999999998},"last":1.139}]},{"osd":7,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.7410000000000001,"15min":1.73},"min":{"1min":1.26,"5min":1.2529999999999999,"15min":1.095},"max":{"1min":2.484,"5min":2.484,"15min":2.5270000000000001},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.671,"15min":1.6659999999999999},"min":{"1min":1.3859999999999999,"5min":1.1850000000000001,"15min":1.1579999999999999},"max":{"1min":2.2949999999999999,"5min":2.464,"15min":2.464},"last":1.3280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:07:49.337 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:07:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.601+0000 7f21abfff640 1 Processor -- start 2023-10-16T14:07:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.601+0000 7f21abfff640 1 -- start start 2023-10-16T14:07:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21abfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f21ac06c040 0x7f21ac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21abfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f21ac15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21abfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac06c960 con 0x7f21ac0ad970 2023-10-16T14:07:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21abfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac15b9d0 con 0x7f21ac06c040 2023-10-16T14:07:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21abfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac15bcb0 con 0x7f21ac0cda80 2023-10-16T14:07:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21a3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f21ac0ad970 0x7f21ac15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34602/0 (socket says 172.21.15.60:34602) 2023-10-16T14:07:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.603+0000 7f21a3fff640 1 -- 172.21.15.60:0/2202489661 learned_addr learned my addr 172.21.15.60:0/2202489661 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236723078 0 0) 0x7f21ac06c960 con 0x7f21ac0ad970 2023-10-16T14:07:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198003520 con 0x7f21ac0ad970 2023-10-16T14:07:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aaffd640 1 --2- 172.21.15.60:0/2202489661 >> v2:172.21.15.144:3300/0 conn(0x7f21ac06c040 0x7f21ac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21ab7fe640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f21ac15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265346826 0 0) 0x7f2198003520 con 0x7f21ac0ad970 2023-10-16T14:07:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 msgr2=0x7f21ac15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f21ac15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 >> v2:172.21.15.144:3300/0 conn(0x7f21ac06c040 msgr2=0x7f21ac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 --2- 172.21.15.60:0/2202489661 >> v2:172.21.15.144:3300/0 conn(0x7f21ac06c040 0x7f21ac06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac15bf90 con 0x7f21ac0ad970 2023-10-16T14:07:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2194001670 con 0x7f21ac0ad970 2023-10-16T14:07:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 >> v1:172.21.15.60:6789/0 conn(0x7f21ac0ad970 legacy=0x7f21ac15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.604+0000 7f21aa7fc640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f2198003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aa7fc640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aa7fc640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2198004d00 0x7f2198005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac15b9d0 con 0x7f2198003ef0 2023-10-16T14:07:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac15bcb0 con 0x7f2198004d00 2023-10-16T14:07:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219800b9d0 con 0x7f21ac0cda80 2023-10-16T14:07:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21a3fff640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aaffd640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f2198003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21a3fff640 1 -- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 msgr2=0x7f2198003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21a3fff640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f2198003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21a3fff640 1 -- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2198004d00 msgr2=0x7f2198005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21a3fff640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2198004d00 0x7f2198005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21a3fff640 1 -- 172.21.15.60:0/2202489661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198003520 con 0x7f2198003ef0 2023-10-16T14:07:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.605+0000 7f21aaffd640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac0cda80 0x7f2198003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.607+0000 7f21a3fff640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21980047c0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f219400c210 tx=0x7f21980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8187c0cfe960b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.607+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194005ad0 con 0x7f2198003ef0 2023-10-16T14:07:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.607+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2194005c30 con 0x7f2198003ef0 2023-10-16T14:07:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.607+0000 7f21aa7fc640 1 -- 172.21.15.60:0/2202489661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219402caf0 con 0x7f2198003ef0 2023-10-16T14:07:49.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21abfff640 1 -- 172.21.15.60:0/2202489661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 msgr2=0x7f21980047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21abfff640 1 --2- 172.21.15.60:0/2202489661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21980047c0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f219400c210 tx=0x7f21980057f0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21ab7fe640 1 -- 172.21.15.60:0/2202489661 reap_dead start 2023-10-16T14:07:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21abfff640 1 -- 172.21.15.60:0/2202489661 shutdown_connections 2023-10-16T14:07:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21abfff640 1 -- 172.21.15.60:0/2202489661 >> 172.21.15.60:0/2202489661 conn(0x7f21ac0ce6a0 msgr2=0x7f21ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21abfff640 1 -- 172.21.15.60:0/2202489661 shutdown_connections 2023-10-16T14:07:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.608+0000 7f21abfff640 1 -- 172.21.15.60:0/2202489661 wait complete. 2023-10-16T14:07:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.610+0000 7f21abfff640 1 Processor -- start 2023-10-16T14:07:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.610+0000 7f21abfff640 1 -- start start 2023-10-16T14:07:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21ac14fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21abfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac15d490 0x7f21ac15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21abfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21ac0aeaa0 0x7f21ac0aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21abfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac15de30 con 0x7f2198003ef0 2023-10-16T14:07:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21abfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0af7c0 con 0x7f21ac0aeaa0 2023-10-16T14:07:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21abfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0afaa0 con 0x7f21ac15d490 2023-10-16T14:07:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21aaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21ac14fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21aaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21ac14fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59888/0 (socket says 172.21.15.60:59888) 2023-10-16T14:07:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21aaffd640 1 -- 172.21.15.60:0/1426613207 learned_addr learned my addr 172.21.15.60:0/1426613207 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.611+0000 7f21ab7fe640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21ac0aeaa0 0x7f21ac0aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21aaffd640 1 -- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac15d490 msgr2=0x7f21ac15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21aaffd640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac15d490 0x7f21ac15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21aaffd640 1 -- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21ac0aeaa0 msgr2=0x7f21ac0aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21aaffd640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21ac0aeaa0 0x7f21ac0aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21aaffd640 1 -- 172.21.15.60:0/1426613207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac0afd80 con 0x7f2198003ef0 2023-10-16T14:07:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21ab7fe640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21ac0aeaa0 0x7f21ac0aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21aaffd640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21ac14fae0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f219c0109d0 tx=0x7f219c010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c00ec20 con 0x7f2198003ef0 2023-10-16T14:07:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c00ed80 con 0x7f2198003ef0 2023-10-16T14:07:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c01c620 con 0x7f2198003ef0 2023-10-16T14:07:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac0b0060 con 0x7f2198003ef0 2023-10-16T14:07:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.612+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac0af490 con 0x7f2198003ef0 2023-10-16T14:07:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.614+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac04e9b0 con 0x7f2198003ef0 2023-10-16T14:07:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.615+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f219c01c780 con 0x7f2198003ef0 2023-10-16T14:07:49.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.616+0000 7f21a37fe640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f218c06afc0 0x7f218c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.616+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f219c089f60 con 0x7f2198003ef0 2023-10-16T14:07:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.617+0000 7f21a3fff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f218c06afc0 0x7f218c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.618+0000 7f21a3fff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f218c06afc0 0x7f218c06d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2194034000 tx=0x7f219402e000 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.621+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f219c054e80 con 0x7f2198003ef0 2023-10-16T14:07:49.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.808+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 --> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21ac0cc550 con 0x7f218c06afc0 2023-10-16T14:07:49.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.830+0000 7f21a37fe640 1 -- 172.21.15.60:0/1426613207 <== mgr.8350 v2:172.21.15.144:6832/1274087552 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286992 (secure 0 0 0) 0x7f2194001c50 con 0x7f218c06afc0 2023-10-16T14:07:49.837 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:49.839 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:07:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.837+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f218c06afc0 msgr2=0x7f218c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.837+0000 7f21abfff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f218c06afc0 0x7f218c06d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2194034000 tx=0x7f219402e000 comp rx=0 tx=0).stop 2023-10-16T14:07:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.837+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 msgr2=0x7f21ac14fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.837+0000 7f21abfff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21ac14fae0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f219c0109d0 tx=0x7f219c010e90 comp rx=0 tx=0).stop 2023-10-16T14:07:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.837+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 shutdown_connections 2023-10-16T14:07:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.837+0000 7f21abfff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f21ac15d490 0x7f21ac15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.838+0000 7f21abfff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f218c06afc0 0x7f218c06d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.838+0000 7f21abfff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21ac0aeaa0 0x7f21ac0aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.838+0000 7f21abfff640 1 --2- 172.21.15.60:0/1426613207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198003ef0 0x7f21ac14fae0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.838+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 >> 172.21.15.60:0/1426613207 conn(0x7f21ac0ce6a0 msgr2=0x7f21ac06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.838+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 shutdown_connections 2023-10-16T14:07:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:49.838+0000 7f21abfff640 1 -- 172.21.15.60:0/1426613207 wait complete. 2023-10-16T14:07:49.858 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-16T14:07:48.125224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":587,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392208,"kb_used_data":21988,"kb_used_omap":12,"kb_used_meta":370099,"kb_avail":754582512,"statfs":{"total":773094113280,"available":772692492288,"internally_reserved":0,"allocated":22515712,"data_stored":11532104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":378981944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.369860"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212493+0000","last_change":"2023-10-16T14:07:44.832980+0000","last_active":"2023-10-16T14:07:46.212493+0000","last_peered":"2023-10-16T14:07:46.212493+0000","last_clean":"2023-10-16T14:07:46.212493+0000","last_became_active":"2023-10-16T14:07:44.832687+0000","last_became_peered":"2023-10-16T14:07:44.832687+0000","last_unstale":"2023-10-16T14:07:46.212493+0000","last_undegraded":"2023-10-16T14:07:46.212493+0000","last_fullsized":"2023-10-16T14:07:46.212493+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:17.284692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829340+0000","last_change":"2023-10-16T14:07:44.820860+0000","last_active":"2023-10-16T14:07:45.829340+0000","last_peered":"2023-10-16T14:07:45.829340+0000","last_clean":"2023-10-16T14:07:45.829340+0000","last_became_active":"2023-10-16T14:07:44.820368+0000","last_became_peered":"2023-10-16T14:07:44.820368+0000","last_unstale":"2023-10-16T14:07:45.829340+0000","last_undegraded":"2023-10-16T14:07:45.829340+0000","last_fullsized":"2023-10-16T14:07:45.829340+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:54.438384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713109+0000","last_change":"2023-10-16T14:07:44.810207+0000","last_active":"2023-10-16T14:07:46.713109+0000","last_peered":"2023-10-16T14:07:46.713109+0000","last_clean":"2023-10-16T14:07:46.713109+0000","last_became_active":"2023-10-16T14:07:44.809861+0000","last_became_peered":"2023-10-16T14:07:44.809861+0000","last_unstale":"2023-10-16T14:07:46.713109+0000","last_undegraded":"2023-10-16T14:07:46.713109+0000","last_fullsized":"2023-10-16T14:07:46.713109+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:35.506431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830184+0000","last_change":"2023-10-16T14:07:44.820903+0000","last_active":"2023-10-16T14:07:45.830184+0000","last_peered":"2023-10-16T14:07:45.830184+0000","last_clean":"2023-10-16T14:07:45.830184+0000","last_became_active":"2023-10-16T14:07:44.820563+0000","last_became_peered":"2023-10-16T14:07:44.820563+0000","last_unstale":"2023-10-16T14:07:45.830184+0000","last_undegraded":"2023-10-16T14:07:45.830184+0000","last_fullsized":"2023-10-16T14:07:45.830184+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:58.685010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968913+0000","last_change":"2023-10-16T14:07:44.823286+0000","last_active":"2023-10-16T14:07:45.968913+0000","last_peered":"2023-10-16T14:07:45.968913+0000","last_clean":"2023-10-16T14:07:45.968913+0000","last_became_active":"2023-10-16T14:07:44.822801+0000","last_became_peered":"2023-10-16T14:07:44.822801+0000","last_unstale":"2023-10-16T14:07:45.968913+0000","last_undegraded":"2023-10-16T14:07:45.968913+0000","last_fullsized":"2023-10-16T14:07:45.968913+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:15.477649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712642+0000","last_change":"2023-10-16T14:07:44.825170+0000","last_active":"2023-10-16T14:07:46.712642+0000","last_peered":"2023-10-16T14:07:46.712642+0000","last_clean":"2023-10-16T14:07:46.712642+0000","last_became_active":"2023-10-16T14:07:44.824860+0000","last_became_peered":"2023-10-16T14:07:44.824860+0000","last_unstale":"2023-10-16T14:07:46.712642+0000","last_undegraded":"2023-10-16T14:07:46.712642+0000","last_fullsized":"2023-10-16T14:07:46.712642+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:22.722924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713871+0000","last_change":"2023-10-16T14:07:44.813889+0000","last_active":"2023-10-16T14:07:46.713871+0000","last_peered":"2023-10-16T14:07:46.713871+0000","last_clean":"2023-10-16T14:07:46.713871+0000","last_became_active":"2023-10-16T14:07:44.813627+0000","last_became_peered":"2023-10-16T14:07:44.813627+0000","last_unstale":"2023-10-16T14:07:46.713871+0000","last_undegraded":"2023-10-16T14:07:46.713871+0000","last_fullsized":"2023-10-16T14:07:46.713871+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:23.659882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434318+0000","last_change":"2023-10-16T14:07:44.810629+0000","last_active":"2023-10-16T14:07:46.434318+0000","last_peered":"2023-10-16T14:07:46.434318+0000","last_clean":"2023-10-16T14:07:46.434318+0000","last_became_active":"2023-10-16T14:07:44.810349+0000","last_became_peered":"2023-10-16T14:07:44.810349+0000","last_unstale":"2023-10-16T14:07:46.434318+0000","last_undegraded":"2023-10-16T14:07:46.434318+0000","last_fullsized":"2023-10-16T14:07:46.434318+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:39.001087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830374+0000","last_change":"2023-10-16T14:07:44.817045+0000","last_active":"2023-10-16T14:07:45.830374+0000","last_peered":"2023-10-16T14:07:45.830374+0000","last_clean":"2023-10-16T14:07:45.830374+0000","last_became_active":"2023-10-16T14:07:44.816040+0000","last_became_peered":"2023-10-16T14:07:44.816040+0000","last_unstale":"2023-10-16T14:07:45.830374+0000","last_undegraded":"2023-10-16T14:07:45.830374+0000","last_fullsized":"2023-10-16T14:07:45.830374+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:59.541962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824142+0000","last_change":"2023-10-16T14:07:44.817360+0000","last_active":"2023-10-16T14:07:45.824142+0000","last_peered":"2023-10-16T14:07:45.824142+0000","last_clean":"2023-10-16T14:07:45.824142+0000","last_became_active":"2023-10-16T14:07:44.814906+0000","last_became_peered":"2023-10-16T14:07:44.814906+0000","last_unstale":"2023-10-16T14:07:45.824142+0000","last_undegraded":"2023-10-16T14:07:45.824142+0000","last_fullsized":"2023-10-16T14:07:45.824142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:48.796679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713611+0000","last_change":"2023-10-16T14:07:44.810568+0000","last_active":"2023-10-16T14:07:46.713611+0000","last_peered":"2023-10-16T14:07:46.713611+0000","last_clean":"2023-10-16T14:07:46.713611+0000","last_became_active":"2023-10-16T14:07:44.809635+0000","last_became_peered":"2023-10-16T14:07:44.809635+0000","last_unstale":"2023-10-16T14:07:46.713611+0000","last_undegraded":"2023-10-16T14:07:46.713611+0000","last_fullsized":"2023-10-16T14:07:46.713611+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:59.541962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830520+0000","last_change":"2023-10-16T14:07:44.830503+0000","last_active":"2023-10-16T14:07:45.830520+0000","last_peered":"2023-10-16T14:07:45.830520+0000","last_clean":"2023-10-16T14:07:45.830520+0000","last_became_active":"2023-10-16T14:07:44.830229+0000","last_became_peered":"2023-10-16T14:07:44.830229+0000","last_unstale":"2023-10-16T14:07:45.830520+0000","last_undegraded":"2023-10-16T14:07:45.830520+0000","last_fullsized":"2023-10-16T14:07:45.830520+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:04.984637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829253+0000","last_change":"2023-10-16T14:07:44.811857+0000","last_active":"2023-10-16T14:07:45.829253+0000","last_peered":"2023-10-16T14:07:45.829253+0000","last_clean":"2023-10-16T14:07:45.829253+0000","last_became_active":"2023-10-16T14:07:44.811393+0000","last_became_peered":"2023-10-16T14:07:44.811393+0000","last_unstale":"2023-10-16T14:07:45.829253+0000","last_undegraded":"2023-10-16T14:07:45.829253+0000","last_fullsized":"2023-10-16T14:07:45.829253+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:07.048482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830138+0000","last_change":"2023-10-16T14:07:44.815217+0000","last_active":"2023-10-16T14:07:45.830138+0000","last_peered":"2023-10-16T14:07:45.830138+0000","last_clean":"2023-10-16T14:07:45.830138+0000","last_became_active":"2023-10-16T14:07:44.814910+0000","last_became_peered":"2023-10-16T14:07:44.814910+0000","last_unstale":"2023-10-16T14:07:45.830138+0000","last_undegraded":"2023-10-16T14:07:45.830138+0000","last_fullsized":"2023-10-16T14:07:45.830138+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:04.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829840+0000","last_change":"2023-10-16T14:07:44.829603+0000","last_active":"2023-10-16T14:07:45.829840+0000","last_peered":"2023-10-16T14:07:45.829840+0000","last_clean":"2023-10-16T14:07:45.829840+0000","last_became_active":"2023-10-16T14:07:44.828751+0000","last_became_peered":"2023-10-16T14:07:44.828751+0000","last_unstale":"2023-10-16T14:07:45.829840+0000","last_undegraded":"2023-10-16T14:07:45.829840+0000","last_fullsized":"2023-10-16T14:07:45.829840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:42.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434310+0000","last_change":"2023-10-16T14:07:44.815993+0000","last_active":"2023-10-16T14:07:46.434310+0000","last_peered":"2023-10-16T14:07:46.434310+0000","last_clean":"2023-10-16T14:07:46.434310+0000","last_became_active":"2023-10-16T14:07:44.815677+0000","last_became_peered":"2023-10-16T14:07:44.815677+0000","last_unstale":"2023-10-16T14:07:46.434310+0000","last_undegraded":"2023-10-16T14:07:46.434310+0000","last_fullsized":"2023-10-16T14:07:46.434310+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:31.505442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969468+0000","last_change":"2023-10-16T14:07:44.818440+0000","last_active":"2023-10-16T14:07:45.969468+0000","last_peered":"2023-10-16T14:07:45.969468+0000","last_clean":"2023-10-16T14:07:45.969468+0000","last_became_active":"2023-10-16T14:07:44.818078+0000","last_became_peered":"2023-10-16T14:07:44.818078+0000","last_unstale":"2023-10-16T14:07:45.969468+0000","last_undegraded":"2023-10-16T14:07:45.969468+0000","last_fullsized":"2023-10-16T14:07:45.969468+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:04.200924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212568+0000","last_change":"2023-10-16T14:07:44.825752+0000","last_active":"2023-10-16T14:07:46.212568+0000","last_peered":"2023-10-16T14:07:46.212568+0000","last_clean":"2023-10-16T14:07:46.212568+0000","last_became_active":"2023-10-16T14:07:44.824616+0000","last_became_peered":"2023-10-16T14:07:44.824616+0000","last_unstale":"2023-10-16T14:07:46.212568+0000","last_undegraded":"2023-10-16T14:07:46.212568+0000","last_fullsized":"2023-10-16T14:07:46.212568+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:20.297567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824174+0000","last_change":"2023-10-16T14:07:44.819207+0000","last_active":"2023-10-16T14:07:45.824174+0000","last_peered":"2023-10-16T14:07:45.824174+0000","last_clean":"2023-10-16T14:07:45.824174+0000","last_became_active":"2023-10-16T14:07:44.818928+0000","last_became_peered":"2023-10-16T14:07:44.818928+0000","last_unstale":"2023-10-16T14:07:45.824174+0000","last_undegraded":"2023-10-16T14:07:45.824174+0000","last_fullsized":"2023-10-16T14:07:45.824174+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:36.435342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969201+0000","last_change":"2023-10-16T14:07:44.820978+0000","last_active":"2023-10-16T14:07:45.969201+0000","last_peered":"2023-10-16T14:07:45.969201+0000","last_clean":"2023-10-16T14:07:45.969201+0000","last_became_active":"2023-10-16T14:07:44.820707+0000","last_became_peered":"2023-10-16T14:07:44.820707+0000","last_unstale":"2023-10-16T14:07:45.969201+0000","last_undegraded":"2023-10-16T14:07:45.969201+0000","last_fullsized":"2023-10-16T14:07:45.969201+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:25.578215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969019+0000","last_change":"2023-10-16T14:07:44.823084+0000","last_active":"2023-10-16T14:07:45.969019+0000","last_peered":"2023-10-16T14:07:45.969019+0000","last_clean":"2023-10-16T14:07:45.969019+0000","last_became_active":"2023-10-16T14:07:44.822410+0000","last_became_peered":"2023-10-16T14:07:44.822410+0000","last_unstale":"2023-10-16T14:07:45.969019+0000","last_undegraded":"2023-10-16T14:07:45.969019+0000","last_fullsized":"2023-10-16T14:07:45.969019+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:58.772982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823364+0000","last_change":"2023-10-16T14:07:44.823572+0000","last_active":"2023-10-16T14:07:45.823364+0000","last_peered":"2023-10-16T14:07:45.823364+0000","last_clean":"2023-10-16T14:07:45.823364+0000","last_became_active":"2023-10-16T14:07:44.822342+0000","last_became_peered":"2023-10-16T14:07:44.822342+0000","last_unstale":"2023-10-16T14:07:45.823364+0000","last_undegraded":"2023-10-16T14:07:45.823364+0000","last_fullsized":"2023-10-16T14:07:45.823364+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:36.970078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712496+0000","last_change":"2023-10-16T14:07:44.815479+0000","last_active":"2023-10-16T14:07:46.712496+0000","last_peered":"2023-10-16T14:07:46.712496+0000","last_clean":"2023-10-16T14:07:46.712496+0000","last_became_active":"2023-10-16T14:07:44.814716+0000","last_became_peered":"2023-10-16T14:07:44.814716+0000","last_unstale":"2023-10-16T14:07:46.712496+0000","last_undegraded":"2023-10-16T14:07:46.712496+0000","last_fullsized":"2023-10-16T14:07:46.712496+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:35.069827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434993+0000","last_change":"2023-10-16T14:07:44.813649+0000","last_active":"2023-10-16T14:07:46.434993+0000","last_peered":"2023-10-16T14:07:46.434993+0000","last_clean":"2023-10-16T14:07:46.434993+0000","last_became_active":"2023-10-16T14:07:44.813324+0000","last_became_peered":"2023-10-16T14:07:44.813324+0000","last_unstale":"2023-10-16T14:07:46.434993+0000","last_undegraded":"2023-10-16T14:07:46.434993+0000","last_fullsized":"2023-10-16T14:07:46.434993+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:33.069562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.831011+0000","last_change":"2023-10-16T14:07:44.817683+0000","last_active":"2023-10-16T14:07:45.831011+0000","last_peered":"2023-10-16T14:07:45.831011+0000","last_clean":"2023-10-16T14:07:45.831011+0000","last_became_active":"2023-10-16T14:07:44.816829+0000","last_became_peered":"2023-10-16T14:07:44.816829+0000","last_unstale":"2023-10-16T14:07:45.831011+0000","last_undegraded":"2023-10-16T14:07:45.831011+0000","last_fullsized":"2023-10-16T14:07:45.831011+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:29:06.616493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713889+0000","last_change":"2023-10-16T14:07:44.818842+0000","last_active":"2023-10-16T14:07:46.713889+0000","last_peered":"2023-10-16T14:07:46.713889+0000","last_clean":"2023-10-16T14:07:46.713889+0000","last_became_active":"2023-10-16T14:07:44.814496+0000","last_became_peered":"2023-10-16T14:07:44.814496+0000","last_unstale":"2023-10-16T14:07:46.713889+0000","last_undegraded":"2023-10-16T14:07:46.713889+0000","last_fullsized":"2023-10-16T14:07:46.713889+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:20.813774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713109+0000","last_change":"2023-10-16T14:07:44.808447+0000","last_active":"2023-10-16T14:07:46.713109+0000","last_peered":"2023-10-16T14:07:46.713109+0000","last_clean":"2023-10-16T14:07:46.713109+0000","last_became_active":"2023-10-16T14:07:44.808133+0000","last_became_peered":"2023-10-16T14:07:44.808133+0000","last_unstale":"2023-10-16T14:07:46.713109+0000","last_undegraded":"2023-10-16T14:07:46.713109+0000","last_fullsized":"2023-10-16T14:07:46.713109+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:51.717001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969296+0000","last_change":"2023-10-16T14:07:44.815202+0000","last_active":"2023-10-16T14:07:45.969296+0000","last_peered":"2023-10-16T14:07:45.969296+0000","last_clean":"2023-10-16T14:07:45.969296+0000","last_became_active":"2023-10-16T14:07:44.814565+0000","last_became_peered":"2023-10-16T14:07:44.814565+0000","last_unstale":"2023-10-16T14:07:45.969296+0000","last_undegraded":"2023-10-16T14:07:45.969296+0000","last_fullsized":"2023-10-16T14:07:45.969296+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:22.722924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829395+0000","last_change":"2023-10-16T14:07:44.824502+0000","last_active":"2023-10-16T14:07:45.829395+0000","last_peered":"2023-10-16T14:07:45.829395+0000","last_clean":"2023-10-16T14:07:45.829395+0000","last_became_active":"2023-10-16T14:07:44.824186+0000","last_became_peered":"2023-10-16T14:07:44.824186+0000","last_unstale":"2023-10-16T14:07:45.829395+0000","last_undegraded":"2023-10-16T14:07:45.829395+0000","last_fullsized":"2023-10-16T14:07:45.829395+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:58.751855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830595+0000","last_change":"2023-10-16T14:07:44.822176+0000","last_active":"2023-10-16T14:07:45.830595+0000","last_peered":"2023-10-16T14:07:45.830595+0000","last_clean":"2023-10-16T14:07:45.830595+0000","last_became_active":"2023-10-16T14:07:44.821871+0000","last_became_peered":"2023-10-16T14:07:44.821871+0000","last_unstale":"2023-10-16T14:07:45.830595+0000","last_undegraded":"2023-10-16T14:07:45.830595+0000","last_fullsized":"2023-10-16T14:07:45.830595+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:20.813774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712735+0000","last_change":"2023-10-16T14:07:44.816572+0000","last_active":"2023-10-16T14:07:46.712735+0000","last_peered":"2023-10-16T14:07:46.712735+0000","last_clean":"2023-10-16T14:07:46.712735+0000","last_became_active":"2023-10-16T14:07:44.816303+0000","last_became_peered":"2023-10-16T14:07:44.816303+0000","last_unstale":"2023-10-16T14:07:46.712735+0000","last_undegraded":"2023-10-16T14:07:46.712735+0000","last_fullsized":"2023-10-16T14:07:46.712735+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:04.200924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968715+0000","last_change":"2023-10-16T14:07:44.812574+0000","last_active":"2023-10-16T14:07:45.968715+0000","last_peered":"2023-10-16T14:07:45.968715+0000","last_clean":"2023-10-16T14:07:45.968715+0000","last_became_active":"2023-10-16T14:07:44.812284+0000","last_became_peered":"2023-10-16T14:07:44.812284+0000","last_unstale":"2023-10-16T14:07:45.968715+0000","last_undegraded":"2023-10-16T14:07:45.968715+0000","last_fullsized":"2023-10-16T14:07:45.968715+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:33.302528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435353+0000","last_change":"2023-10-16T14:07:44.816661+0000","last_active":"2023-10-16T14:07:46.435353+0000","last_peered":"2023-10-16T14:07:46.435353+0000","last_clean":"2023-10-16T14:07:46.435353+0000","last_became_active":"2023-10-16T14:07:44.816394+0000","last_became_peered":"2023-10-16T14:07:44.816394+0000","last_unstale":"2023-10-16T14:07:46.435353+0000","last_undegraded":"2023-10-16T14:07:46.435353+0000","last_fullsized":"2023-10-16T14:07:46.435353+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:16.459394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"212'4","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969502+0000","last_change":"2023-10-16T14:07:43.800016+0000","last_active":"2023-10-16T14:07:45.969502+0000","last_peered":"2023-10-16T14:07:45.969502+0000","last_clean":"2023-10-16T14:07:45.969502+0000","last_became_active":"2023-10-16T14:07:43.799731+0000","last_became_peered":"2023-10-16T14:07:43.799731+0000","last_unstale":"2023-10-16T14:07:45.969502+0000","last_undegraded":"2023-10-16T14:07:45.969502+0000","last_fullsized":"2023-10-16T14:07:45.969502+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:18.118851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824997+0000","last_change":"2023-10-16T14:07:44.822017+0000","last_active":"2023-10-16T14:07:45.824997+0000","last_peered":"2023-10-16T14:07:45.824997+0000","last_clean":"2023-10-16T14:07:45.824997+0000","last_became_active":"2023-10-16T14:07:44.821735+0000","last_became_peered":"2023-10-16T14:07:44.821735+0000","last_unstale":"2023-10-16T14:07:45.824997+0000","last_undegraded":"2023-10-16T14:07:45.824997+0000","last_fullsized":"2023-10-16T14:07:45.824997+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:58.319245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.831288+0000","last_change":"2023-10-16T14:07:43.790976+0000","last_active":"2023-10-16T14:07:45.831288+0000","last_peered":"2023-10-16T14:07:45.831288+0000","last_clean":"2023-10-16T14:07:45.831288+0000","last_became_active":"2023-10-16T14:07:43.790570+0000","last_became_peered":"2023-10-16T14:07:43.790570+0000","last_unstale":"2023-10-16T14:07:45.831288+0000","last_undegraded":"2023-10-16T14:07:45.831288+0000","last_fullsized":"2023-10-16T14:07:45.831288+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:21.360587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713695+0000","last_change":"2023-10-16T14:07:44.813549+0000","last_active":"2023-10-16T14:07:46.713695+0000","last_peered":"2023-10-16T14:07:46.713695+0000","last_clean":"2023-10-16T14:07:46.713695+0000","last_became_active":"2023-10-16T14:07:44.813285+0000","last_became_peered":"2023-10-16T14:07:44.813285+0000","last_unstale":"2023-10-16T14:07:46.713695+0000","last_undegraded":"2023-10-16T14:07:46.713695+0000","last_fullsized":"2023-10-16T14:07:46.713695+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:01.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713442+0000","last_change":"2023-10-16T14:07:43.794019+0000","last_active":"2023-10-16T14:07:46.713442+0000","last_peered":"2023-10-16T14:07:46.713442+0000","last_clean":"2023-10-16T14:07:46.713442+0000","last_became_active":"2023-10-16T14:07:43.793101+0000","last_became_peered":"2023-10-16T14:07:43.793101+0000","last_unstale":"2023-10-16T14:07:46.713442+0000","last_undegraded":"2023-10-16T14:07:46.713442+0000","last_fullsized":"2023-10-16T14:07:46.713442+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:24.563590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830517+0000","last_change":"2023-10-16T14:07:44.811185+0000","last_active":"2023-10-16T14:07:45.830517+0000","last_peered":"2023-10-16T14:07:45.830517+0000","last_clean":"2023-10-16T14:07:45.830517+0000","last_became_active":"2023-10-16T14:07:44.810803+0000","last_became_peered":"2023-10-16T14:07:44.810803+0000","last_unstale":"2023-10-16T14:07:45.830517+0000","last_undegraded":"2023-10-16T14:07:45.830517+0000","last_fullsized":"2023-10-16T14:07:45.830517+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:23.659882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830813+0000","last_change":"2023-10-16T14:07:43.799786+0000","last_active":"2023-10-16T14:07:45.830813+0000","last_peered":"2023-10-16T14:07:45.830813+0000","last_clean":"2023-10-16T14:07:45.830813+0000","last_became_active":"2023-10-16T14:07:43.798587+0000","last_became_peered":"2023-10-16T14:07:43.798587+0000","last_unstale":"2023-10-16T14:07:45.830813+0000","last_undegraded":"2023-10-16T14:07:45.830813+0000","last_fullsized":"2023-10-16T14:07:45.830813+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:36.490113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823366+0000","last_change":"2023-10-16T14:07:44.821312+0000","last_active":"2023-10-16T14:07:45.823366+0000","last_peered":"2023-10-16T14:07:45.823366+0000","last_clean":"2023-10-16T14:07:45.823366+0000","last_became_active":"2023-10-16T14:07:44.820574+0000","last_became_peered":"2023-10-16T14:07:44.820574+0000","last_unstale":"2023-10-16T14:07:45.823366+0000","last_undegraded":"2023-10-16T14:07:45.823366+0000","last_fullsized":"2023-10-16T14:07:45.823366+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:21.169581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969007+0000","last_change":"2023-10-16T14:07:43.796733+0000","last_active":"2023-10-16T14:07:45.969007+0000","last_peered":"2023-10-16T14:07:45.969007+0000","last_clean":"2023-10-16T14:07:45.969007+0000","last_became_active":"2023-10-16T14:07:43.796245+0000","last_became_peered":"2023-10-16T14:07:43.796245+0000","last_unstale":"2023-10-16T14:07:45.969007+0000","last_undegraded":"2023-10-16T14:07:45.969007+0000","last_fullsized":"2023-10-16T14:07:45.969007+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:12:58.913804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435318+0000","last_change":"2023-10-16T14:07:44.811959+0000","last_active":"2023-10-16T14:07:46.435318+0000","last_peered":"2023-10-16T14:07:46.435318+0000","last_clean":"2023-10-16T14:07:46.435318+0000","last_became_active":"2023-10-16T14:07:44.811698+0000","last_became_peered":"2023-10-16T14:07:44.811698+0000","last_unstale":"2023-10-16T14:07:46.435318+0000","last_undegraded":"2023-10-16T14:07:46.435318+0000","last_fullsized":"2023-10-16T14:07:46.435318+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:05:06.296499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434660+0000","last_change":"2023-10-16T14:07:43.795912+0000","last_active":"2023-10-16T14:07:46.434660+0000","last_peered":"2023-10-16T14:07:46.434660+0000","last_clean":"2023-10-16T14:07:46.434660+0000","last_became_active":"2023-10-16T14:07:43.795629+0000","last_became_peered":"2023-10-16T14:07:43.795629+0000","last_unstale":"2023-10-16T14:07:46.434660+0000","last_undegraded":"2023-10-16T14:07:46.434660+0000","last_fullsized":"2023-10-16T14:07:46.434660+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:38.077728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713933+0000","last_change":"2023-10-16T14:07:44.827360+0000","last_active":"2023-10-16T14:07:46.713933+0000","last_peered":"2023-10-16T14:07:46.713933+0000","last_clean":"2023-10-16T14:07:46.713933+0000","last_became_active":"2023-10-16T14:07:44.827104+0000","last_became_peered":"2023-10-16T14:07:44.827104+0000","last_unstale":"2023-10-16T14:07:46.713933+0000","last_undegraded":"2023-10-16T14:07:46.713933+0000","last_fullsized":"2023-10-16T14:07:46.713933+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:19.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211349+0000","last_change":"2023-10-16T14:07:43.799859+0000","last_active":"2023-10-16T14:07:46.211349+0000","last_peered":"2023-10-16T14:07:46.211349+0000","last_clean":"2023-10-16T14:07:46.211349+0000","last_became_active":"2023-10-16T14:07:43.799267+0000","last_became_peered":"2023-10-16T14:07:43.799267+0000","last_unstale":"2023-10-16T14:07:46.211349+0000","last_undegraded":"2023-10-16T14:07:46.211349+0000","last_fullsized":"2023-10-16T14:07:46.211349+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:56:50.764862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713337+0000","last_change":"2023-10-16T14:07:44.812645+0000","last_active":"2023-10-16T14:07:46.713337+0000","last_peered":"2023-10-16T14:07:46.713337+0000","last_clean":"2023-10-16T14:07:46.713337+0000","last_became_active":"2023-10-16T14:07:44.812302+0000","last_became_peered":"2023-10-16T14:07:44.812302+0000","last_unstale":"2023-10-16T14:07:46.713337+0000","last_undegraded":"2023-10-16T14:07:46.713337+0000","last_fullsized":"2023-10-16T14:07:46.713337+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:43.265721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713870+0000","last_change":"2023-10-16T14:07:43.782583+0000","last_active":"2023-10-16T14:07:46.713870+0000","last_peered":"2023-10-16T14:07:46.713870+0000","last_clean":"2023-10-16T14:07:46.713870+0000","last_became_active":"2023-10-16T14:07:43.780706+0000","last_became_peered":"2023-10-16T14:07:43.780706+0000","last_unstale":"2023-10-16T14:07:46.713870+0000","last_undegraded":"2023-10-16T14:07:46.713870+0000","last_fullsized":"2023-10-16T14:07:46.713870+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:14.043426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435144+0000","last_change":"2023-10-16T14:07:44.812473+0000","last_active":"2023-10-16T14:07:46.435144+0000","last_peered":"2023-10-16T14:07:46.435144+0000","last_clean":"2023-10-16T14:07:46.435144+0000","last_became_active":"2023-10-16T14:07:44.811563+0000","last_became_peered":"2023-10-16T14:07:44.811563+0000","last_unstale":"2023-10-16T14:07:46.435144+0000","last_undegraded":"2023-10-16T14:07:46.435144+0000","last_fullsized":"2023-10-16T14:07:46.435144+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:05.455592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969340+0000","last_change":"2023-10-16T14:07:43.797313+0000","last_active":"2023-10-16T14:07:45.969340+0000","last_peered":"2023-10-16T14:07:45.969340+0000","last_clean":"2023-10-16T14:07:45.969340+0000","last_became_active":"2023-10-16T14:07:43.796993+0000","last_became_peered":"2023-10-16T14:07:43.796993+0000","last_unstale":"2023-10-16T14:07:45.969340+0000","last_undegraded":"2023-10-16T14:07:45.969340+0000","last_fullsized":"2023-10-16T14:07:45.969340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:36.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713944+0000","last_change":"2023-10-16T14:07:44.811612+0000","last_active":"2023-10-16T14:07:46.713944+0000","last_peered":"2023-10-16T14:07:46.713944+0000","last_clean":"2023-10-16T14:07:46.713944+0000","last_became_active":"2023-10-16T14:07:44.811320+0000","last_became_peered":"2023-10-16T14:07:44.811320+0000","last_unstale":"2023-10-16T14:07:46.713944+0000","last_undegraded":"2023-10-16T14:07:46.713944+0000","last_fullsized":"2023-10-16T14:07:46.713944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:00.517821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"212'3","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969324+0000","last_change":"2023-10-16T14:07:43.799330+0000","last_active":"2023-10-16T14:07:45.969324+0000","last_peered":"2023-10-16T14:07:45.969324+0000","last_clean":"2023-10-16T14:07:45.969324+0000","last_became_active":"2023-10-16T14:07:43.799043+0000","last_became_peered":"2023-10-16T14:07:43.799043+0000","last_unstale":"2023-10-16T14:07:45.969324+0000","last_undegraded":"2023-10-16T14:07:45.969324+0000","last_fullsized":"2023-10-16T14:07:45.969324+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:18.726291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824025+0000","last_change":"2023-10-16T14:07:44.815955+0000","last_active":"2023-10-16T14:07:45.824025+0000","last_peered":"2023-10-16T14:07:45.824025+0000","last_clean":"2023-10-16T14:07:45.824025+0000","last_became_active":"2023-10-16T14:07:44.815663+0000","last_became_peered":"2023-10-16T14:07:44.815663+0000","last_unstale":"2023-10-16T14:07:45.824025+0000","last_undegraded":"2023-10-16T14:07:45.824025+0000","last_fullsized":"2023-10-16T14:07:45.824025+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:58.529404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"212'6","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712831+0000","last_change":"2023-10-16T14:07:43.790980+0000","last_active":"2023-10-16T14:07:46.712831+0000","last_peered":"2023-10-16T14:07:46.712831+0000","last_clean":"2023-10-16T14:07:46.712831+0000","last_became_active":"2023-10-16T14:07:43.790665+0000","last_became_peered":"2023-10-16T14:07:43.790665+0000","last_unstale":"2023-10-16T14:07:46.712831+0000","last_undegraded":"2023-10-16T14:07:46.712831+0000","last_fullsized":"2023-10-16T14:07:46.712831+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:24.507670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713033+0000","last_change":"2023-10-16T14:07:44.812385+0000","last_active":"2023-10-16T14:07:46.713033+0000","last_peered":"2023-10-16T14:07:46.713033+0000","last_clean":"2023-10-16T14:07:46.713033+0000","last_became_active":"2023-10-16T14:07:44.812030+0000","last_became_peered":"2023-10-16T14:07:44.812030+0000","last_unstale":"2023-10-16T14:07:46.713033+0000","last_undegraded":"2023-10-16T14:07:46.713033+0000","last_fullsized":"2023-10-16T14:07:46.713033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:09.438847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212226+0000","last_change":"2023-10-16T14:07:43.797128+0000","last_active":"2023-10-16T14:07:46.212226+0000","last_peered":"2023-10-16T14:07:46.212226+0000","last_clean":"2023-10-16T14:07:46.212226+0000","last_became_active":"2023-10-16T14:07:43.795724+0000","last_became_peered":"2023-10-16T14:07:43.795724+0000","last_unstale":"2023-10-16T14:07:46.212226+0000","last_undegraded":"2023-10-16T14:07:46.212226+0000","last_fullsized":"2023-10-16T14:07:46.212226+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:43.257103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713493+0000","last_change":"2023-10-16T14:07:44.813345+0000","last_active":"2023-10-16T14:07:46.713493+0000","last_peered":"2023-10-16T14:07:46.713493+0000","last_clean":"2023-10-16T14:07:46.713493+0000","last_became_active":"2023-10-16T14:07:44.813084+0000","last_became_peered":"2023-10-16T14:07:44.813084+0000","last_unstale":"2023-10-16T14:07:46.713493+0000","last_undegraded":"2023-10-16T14:07:46.713493+0000","last_fullsized":"2023-10-16T14:07:46.713493+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:04:04.654122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823839+0000","last_change":"2023-10-16T14:07:43.795728+0000","last_active":"2023-10-16T14:07:45.823839+0000","last_peered":"2023-10-16T14:07:45.823839+0000","last_clean":"2023-10-16T14:07:45.823839+0000","last_became_active":"2023-10-16T14:07:43.795411+0000","last_became_peered":"2023-10-16T14:07:43.795411+0000","last_unstale":"2023-10-16T14:07:45.823839+0000","last_undegraded":"2023-10-16T14:07:45.823839+0000","last_fullsized":"2023-10-16T14:07:45.823839+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:53.532327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829965+0000","last_change":"2023-10-16T14:07:44.818755+0000","last_active":"2023-10-16T14:07:45.829965+0000","last_peered":"2023-10-16T14:07:45.829965+0000","last_clean":"2023-10-16T14:07:45.829965+0000","last_became_active":"2023-10-16T14:07:44.815413+0000","last_became_peered":"2023-10-16T14:07:44.815413+0000","last_unstale":"2023-10-16T14:07:45.829965+0000","last_undegraded":"2023-10-16T14:07:45.829965+0000","last_fullsized":"2023-10-16T14:07:45.829965+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:19.201828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.434214+0000","last_change":"2023-10-16T14:07:43.798522+0000","last_active":"2023-10-16T14:07:46.434214+0000","last_peered":"2023-10-16T14:07:46.434214+0000","last_clean":"2023-10-16T14:07:46.434214+0000","last_became_active":"2023-10-16T14:07:43.798243+0000","last_became_peered":"2023-10-16T14:07:43.798243+0000","last_unstale":"2023-10-16T14:07:46.434214+0000","last_undegraded":"2023-10-16T14:07:46.434214+0000","last_fullsized":"2023-10-16T14:07:46.434214+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:56:50.764862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823656+0000","last_change":"2023-10-16T14:07:44.825912+0000","last_active":"2023-10-16T14:07:45.823656+0000","last_peered":"2023-10-16T14:07:45.823656+0000","last_clean":"2023-10-16T14:07:45.823656+0000","last_became_active":"2023-10-16T14:07:44.825570+0000","last_became_peered":"2023-10-16T14:07:44.825570+0000","last_unstale":"2023-10-16T14:07:45.823656+0000","last_undegraded":"2023-10-16T14:07:45.823656+0000","last_fullsized":"2023-10-16T14:07:45.823656+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:22.319291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"212'8","reported_seq":20,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211946+0000","last_change":"2023-10-16T14:07:43.799575+0000","last_active":"2023-10-16T14:07:46.211946+0000","last_peered":"2023-10-16T14:07:46.211946+0000","last_clean":"2023-10-16T14:07:46.211946+0000","last_became_active":"2023-10-16T14:07:43.798872+0000","last_became_peered":"2023-10-16T14:07:43.798872+0000","last_unstale":"2023-10-16T14:07:46.211946+0000","last_undegraded":"2023-10-16T14:07:46.211946+0000","last_fullsized":"2023-10-16T14:07:46.211946+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:10.696490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824215+0000","last_change":"2023-10-16T14:07:44.826736+0000","last_active":"2023-10-16T14:07:45.824215+0000","last_peered":"2023-10-16T14:07:45.824215+0000","last_clean":"2023-10-16T14:07:45.824215+0000","last_became_active":"2023-10-16T14:07:44.826363+0000","last_became_peered":"2023-10-16T14:07:44.826363+0000","last_unstale":"2023-10-16T14:07:45.824215+0000","last_undegraded":"2023-10-16T14:07:45.824215+0000","last_fullsized":"2023-10-16T14:07:45.824215+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:43.210356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969571+0000","last_change":"2023-10-16T14:07:43.799382+0000","last_active":"2023-10-16T14:07:45.969571+0000","last_peered":"2023-10-16T14:07:45.969571+0000","last_clean":"2023-10-16T14:07:45.969571+0000","last_became_active":"2023-10-16T14:07:43.798998+0000","last_became_peered":"2023-10-16T14:07:43.798998+0000","last_unstale":"2023-10-16T14:07:45.969571+0000","last_undegraded":"2023-10-16T14:07:45.969571+0000","last_fullsized":"2023-10-16T14:07:45.969571+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:59.503196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830612+0000","last_change":"2023-10-16T14:07:44.818549+0000","last_active":"2023-10-16T14:07:45.830612+0000","last_peered":"2023-10-16T14:07:45.830612+0000","last_clean":"2023-10-16T14:07:45.830612+0000","last_became_active":"2023-10-16T14:07:44.815471+0000","last_became_peered":"2023-10-16T14:07:44.815471+0000","last_unstale":"2023-10-16T14:07:45.830612+0000","last_undegraded":"2023-10-16T14:07:45.830612+0000","last_fullsized":"2023-10-16T14:07:45.830612+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:19.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435375+0000","last_change":"2023-10-16T14:07:43.789962+0000","last_active":"2023-10-16T14:07:46.435375+0000","last_peered":"2023-10-16T14:07:46.435375+0000","last_clean":"2023-10-16T14:07:46.435375+0000","last_became_active":"2023-10-16T14:07:43.789699+0000","last_became_peered":"2023-10-16T14:07:43.789699+0000","last_unstale":"2023-10-16T14:07:46.435375+0000","last_undegraded":"2023-10-16T14:07:46.435375+0000","last_fullsized":"2023-10-16T14:07:46.435375+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:43.257103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830430+0000","last_change":"2023-10-16T14:07:44.808623+0000","last_active":"2023-10-16T14:07:45.830430+0000","last_peered":"2023-10-16T14:07:45.830430+0000","last_clean":"2023-10-16T14:07:45.830430+0000","last_became_active":"2023-10-16T14:07:44.808282+0000","last_became_peered":"2023-10-16T14:07:44.808282+0000","last_unstale":"2023-10-16T14:07:45.830430+0000","last_undegraded":"2023-10-16T14:07:45.830430+0000","last_fullsized":"2023-10-16T14:07:45.830430+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:01.593860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"212'1","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212227+0000","last_change":"2023-10-16T14:07:43.799777+0000","last_active":"2023-10-16T14:07:46.212227+0000","last_peered":"2023-10-16T14:07:46.212227+0000","last_clean":"2023-10-16T14:07:46.212227+0000","last_became_active":"2023-10-16T14:07:43.799481+0000","last_became_peered":"2023-10-16T14:07:43.799481+0000","last_unstale":"2023-10-16T14:07:46.212227+0000","last_undegraded":"2023-10-16T14:07:46.212227+0000","last_fullsized":"2023-10-16T14:07:46.212227+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:30.490818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829359+0000","last_change":"2023-10-16T14:07:44.820436+0000","last_active":"2023-10-16T14:07:45.829359+0000","last_peered":"2023-10-16T14:07:45.829359+0000","last_clean":"2023-10-16T14:07:45.829359+0000","last_became_active":"2023-10-16T14:07:44.820164+0000","last_became_peered":"2023-10-16T14:07:44.820164+0000","last_unstale":"2023-10-16T14:07:45.829359+0000","last_undegraded":"2023-10-16T14:07:45.829359+0000","last_fullsized":"2023-10-16T14:07:45.829359+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:55.293783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.713920+0000","last_change":"2023-10-16T14:07:43.786727+0000","last_active":"2023-10-16T14:07:46.713920+0000","last_peered":"2023-10-16T14:07:46.713920+0000","last_clean":"2023-10-16T14:07:46.713920+0000","last_became_active":"2023-10-16T14:07:43.786272+0000","last_became_peered":"2023-10-16T14:07:43.786272+0000","last_unstale":"2023-10-16T14:07:46.713920+0000","last_undegraded":"2023-10-16T14:07:46.713920+0000","last_fullsized":"2023-10-16T14:07:46.713920+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:18.187203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823548+0000","last_change":"2023-10-16T14:07:44.812592+0000","last_active":"2023-10-16T14:07:45.823548+0000","last_peered":"2023-10-16T14:07:45.823548+0000","last_clean":"2023-10-16T14:07:45.823548+0000","last_became_active":"2023-10-16T14:07:44.812247+0000","last_became_peered":"2023-10-16T14:07:44.812247+0000","last_unstale":"2023-10-16T14:07:45.823548+0000","last_undegraded":"2023-10-16T14:07:45.823548+0000","last_fullsized":"2023-10-16T14:07:45.823548+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:40.141847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712595+0000","last_change":"2023-10-16T14:07:43.798513+0000","last_active":"2023-10-16T14:07:46.712595+0000","last_peered":"2023-10-16T14:07:46.712595+0000","last_clean":"2023-10-16T14:07:46.712595+0000","last_became_active":"2023-10-16T14:07:43.797262+0000","last_became_peered":"2023-10-16T14:07:43.797262+0000","last_unstale":"2023-10-16T14:07:46.712595+0000","last_undegraded":"2023-10-16T14:07:46.712595+0000","last_fullsized":"2023-10-16T14:07:46.712595+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:36.128910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.714062+0000","last_change":"2023-10-16T14:07:44.824444+0000","last_active":"2023-10-16T14:07:46.714062+0000","last_peered":"2023-10-16T14:07:46.714062+0000","last_clean":"2023-10-16T14:07:46.714062+0000","last_became_active":"2023-10-16T14:07:44.824159+0000","last_became_peered":"2023-10-16T14:07:44.824159+0000","last_unstale":"2023-10-16T14:07:46.714062+0000","last_undegraded":"2023-10-16T14:07:46.714062+0000","last_fullsized":"2023-10-16T14:07:46.714062+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:14.036727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968936+0000","last_change":"2023-10-16T14:07:43.797313+0000","last_active":"2023-10-16T14:07:45.968936+0000","last_peered":"2023-10-16T14:07:45.968936+0000","last_clean":"2023-10-16T14:07:45.968936+0000","last_became_active":"2023-10-16T14:07:43.797004+0000","last_became_peered":"2023-10-16T14:07:43.797004+0000","last_unstale":"2023-10-16T14:07:45.968936+0000","last_undegraded":"2023-10-16T14:07:45.968936+0000","last_fullsized":"2023-10-16T14:07:45.968936+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:13.022102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.828886+0000","last_change":"2023-10-16T14:07:44.825441+0000","last_active":"2023-10-16T14:07:45.828886+0000","last_peered":"2023-10-16T14:07:45.828886+0000","last_clean":"2023-10-16T14:07:45.828886+0000","last_became_active":"2023-10-16T14:07:44.825154+0000","last_became_peered":"2023-10-16T14:07:44.825154+0000","last_unstale":"2023-10-16T14:07:45.828886+0000","last_undegraded":"2023-10-16T14:07:45.828886+0000","last_fullsized":"2023-10-16T14:07:45.828886+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:44.027350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712975+0000","last_change":"2023-10-16T14:07:43.797330+0000","last_active":"2023-10-16T14:07:46.712975+0000","last_peered":"2023-10-16T14:07:46.712975+0000","last_clean":"2023-10-16T14:07:46.712975+0000","last_became_active":"2023-10-16T14:07:43.796947+0000","last_became_peered":"2023-10-16T14:07:43.796947+0000","last_unstale":"2023-10-16T14:07:46.712975+0000","last_undegraded":"2023-10-16T14:07:46.712975+0000","last_fullsized":"2023-10-16T14:07:46.712975+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:32.287903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968763+0000","last_change":"2023-10-16T14:07:44.814498+0000","last_active":"2023-10-16T14:07:45.968763+0000","last_peered":"2023-10-16T14:07:45.968763+0000","last_clean":"2023-10-16T14:07:45.968763+0000","last_became_active":"2023-10-16T14:07:44.814195+0000","last_became_peered":"2023-10-16T14:07:44.814195+0000","last_unstale":"2023-10-16T14:07:45.968763+0000","last_undegraded":"2023-10-16T14:07:45.968763+0000","last_fullsized":"2023-10-16T14:07:45.968763+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:57:31.212385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829801+0000","last_change":"2023-10-16T14:07:43.796252+0000","last_active":"2023-10-16T14:07:45.829801+0000","last_peered":"2023-10-16T14:07:45.829801+0000","last_clean":"2023-10-16T14:07:45.829801+0000","last_became_active":"2023-10-16T14:07:43.795441+0000","last_became_peered":"2023-10-16T14:07:43.795441+0000","last_unstale":"2023-10-16T14:07:45.829801+0000","last_undegraded":"2023-10-16T14:07:45.829801+0000","last_fullsized":"2023-10-16T14:07:45.829801+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:14.043426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.969277+0000","last_change":"2023-10-16T14:07:44.819106+0000","last_active":"2023-10-16T14:07:45.969277+0000","last_peered":"2023-10-16T14:07:45.969277+0000","last_clean":"2023-10-16T14:07:45.969277+0000","last_became_active":"2023-10-16T14:07:44.818816+0000","last_became_peered":"2023-10-16T14:07:44.818816+0000","last_unstale":"2023-10-16T14:07:45.969277+0000","last_undegraded":"2023-10-16T14:07:45.969277+0000","last_fullsized":"2023-10-16T14:07:45.969277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:35.069827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"212'2","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.212378+0000","last_change":"2023-10-16T14:07:43.801180+0000","last_active":"2023-10-16T14:07:46.212378+0000","last_peered":"2023-10-16T14:07:46.212378+0000","last_clean":"2023-10-16T14:07:46.212378+0000","last_became_active":"2023-10-16T14:07:43.800743+0000","last_became_peered":"2023-10-16T14:07:43.800743+0000","last_unstale":"2023-10-16T14:07:46.212378+0000","last_undegraded":"2023-10-16T14:07:46.212378+0000","last_fullsized":"2023-10-16T14:07:46.212378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:05:05.281874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.825125+0000","last_change":"2023-10-16T14:07:44.832039+0000","last_active":"2023-10-16T14:07:45.825125+0000","last_peered":"2023-10-16T14:07:45.825125+0000","last_clean":"2023-10-16T14:07:45.825125+0000","last_became_active":"2023-10-16T14:07:44.831730+0000","last_became_peered":"2023-10-16T14:07:44.831730+0000","last_unstale":"2023-10-16T14:07:45.825125+0000","last_undegraded":"2023-10-16T14:07:45.825125+0000","last_fullsized":"2023-10-16T14:07:45.825125+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:01.519902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824257+0000","last_change":"2023-10-16T14:07:43.793950+0000","last_active":"2023-10-16T14:07:45.824257+0000","last_peered":"2023-10-16T14:07:45.824257+0000","last_clean":"2023-10-16T14:07:45.824257+0000","last_became_active":"2023-10-16T14:07:43.793645+0000","last_became_peered":"2023-10-16T14:07:43.793645+0000","last_unstale":"2023-10-16T14:07:45.824257+0000","last_undegraded":"2023-10-16T14:07:45.824257+0000","last_fullsized":"2023-10-16T14:07:45.824257+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:42.416005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435384+0000","last_change":"2023-10-16T14:07:44.813393+0000","last_active":"2023-10-16T14:07:46.435384+0000","last_peered":"2023-10-16T14:07:46.435384+0000","last_clean":"2023-10-16T14:07:46.435384+0000","last_became_active":"2023-10-16T14:07:44.813109+0000","last_became_peered":"2023-10-16T14:07:44.813109+0000","last_unstale":"2023-10-16T14:07:46.435384+0000","last_undegraded":"2023-10-16T14:07:46.435384+0000","last_fullsized":"2023-10-16T14:07:46.435384+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:23.227382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.824135+0000","last_change":"2023-10-16T14:07:43.789095+0000","last_active":"2023-10-16T14:07:45.824135+0000","last_peered":"2023-10-16T14:07:45.824135+0000","last_clean":"2023-10-16T14:07:45.824135+0000","last_became_active":"2023-10-16T14:07:43.788816+0000","last_became_peered":"2023-10-16T14:07:43.788816+0000","last_unstale":"2023-10-16T14:07:45.824135+0000","last_undegraded":"2023-10-16T14:07:45.824135+0000","last_fullsized":"2023-10-16T14:07:45.824135+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:55.334715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435304+0000","last_change":"2023-10-16T14:07:44.816045+0000","last_active":"2023-10-16T14:07:46.435304+0000","last_peered":"2023-10-16T14:07:46.435304+0000","last_clean":"2023-10-16T14:07:46.435304+0000","last_became_active":"2023-10-16T14:07:44.815730+0000","last_became_peered":"2023-10-16T14:07:44.815730+0000","last_unstale":"2023-10-16T14:07:46.435304+0000","last_undegraded":"2023-10-16T14:07:46.435304+0000","last_fullsized":"2023-10-16T14:07:46.435304+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:22.305405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.435568+0000","last_change":"2023-10-16T14:07:43.787074+0000","last_active":"2023-10-16T14:07:46.435568+0000","last_peered":"2023-10-16T14:07:46.435568+0000","last_clean":"2023-10-16T14:07:46.435568+0000","last_became_active":"2023-10-16T14:07:43.786733+0000","last_became_peered":"2023-10-16T14:07:43.786733+0000","last_unstale":"2023-10-16T14:07:46.435568+0000","last_undegraded":"2023-10-16T14:07:46.435568+0000","last_fullsized":"2023-10-16T14:07:46.435568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:10.696490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211916+0000","last_change":"2023-10-16T14:07:44.828126+0000","last_active":"2023-10-16T14:07:46.211916+0000","last_peered":"2023-10-16T14:07:46.211916+0000","last_clean":"2023-10-16T14:07:46.211916+0000","last_became_active":"2023-10-16T14:07:44.827813+0000","last_became_peered":"2023-10-16T14:07:44.827813+0000","last_unstale":"2023-10-16T14:07:46.211916+0000","last_undegraded":"2023-10-16T14:07:46.211916+0000","last_fullsized":"2023-10-16T14:07:46.211916+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:16.459394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830979+0000","last_change":"2023-10-16T14:07:43.778246+0000","last_active":"2023-10-16T14:07:45.830979+0000","last_peered":"2023-10-16T14:07:45.830979+0000","last_clean":"2023-10-16T14:07:45.830979+0000","last_became_active":"2023-10-16T14:07:43.776838+0000","last_became_peered":"2023-10-16T14:07:43.776838+0000","last_unstale":"2023-10-16T14:07:45.830979+0000","last_undegraded":"2023-10-16T14:07:45.830979+0000","last_fullsized":"2023-10-16T14:07:45.830979+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:31.810785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211979+0000","last_change":"2023-10-16T14:07:44.820067+0000","last_active":"2023-10-16T14:07:46.211979+0000","last_peered":"2023-10-16T14:07:46.211979+0000","last_clean":"2023-10-16T14:07:46.211979+0000","last_became_active":"2023-10-16T14:07:44.819780+0000","last_became_peered":"2023-10-16T14:07:44.819780+0000","last_unstale":"2023-10-16T14:07:46.211979+0000","last_undegraded":"2023-10-16T14:07:46.211979+0000","last_fullsized":"2023-10-16T14:07:46.211979+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:23.227382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.831520+0000","last_change":"2023-10-16T14:07:43.794159+0000","last_active":"2023-10-16T14:07:45.831520+0000","last_peered":"2023-10-16T14:07:45.831520+0000","last_clean":"2023-10-16T14:07:45.831520+0000","last_became_active":"2023-10-16T14:07:43.793850+0000","last_became_peered":"2023-10-16T14:07:43.793850+0000","last_unstale":"2023-10-16T14:07:45.831520+0000","last_undegraded":"2023-10-16T14:07:45.831520+0000","last_fullsized":"2023-10-16T14:07:45.831520+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:17.106938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.829096+0000","last_change":"2023-10-16T14:07:44.814739+0000","last_active":"2023-10-16T14:07:45.829096+0000","last_peered":"2023-10-16T14:07:45.829096+0000","last_clean":"2023-10-16T14:07:45.829096+0000","last_became_active":"2023-10-16T14:07:44.812428+0000","last_became_peered":"2023-10-16T14:07:44.812428+0000","last_unstale":"2023-10-16T14:07:45.829096+0000","last_undegraded":"2023-10-16T14:07:45.829096+0000","last_fullsized":"2023-10-16T14:07:45.829096+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:30:50.817563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.830063+0000","last_change":"2023-10-16T14:07:43.798547+0000","last_active":"2023-10-16T14:07:45.830063+0000","last_peered":"2023-10-16T14:07:45.830063+0000","last_clean":"2023-10-16T14:07:45.830063+0000","last_became_active":"2023-10-16T14:07:43.797762+0000","last_became_peered":"2023-10-16T14:07:43.797762+0000","last_unstale":"2023-10-16T14:07:45.830063+0000","last_undegraded":"2023-10-16T14:07:45.830063+0000","last_fullsized":"2023-10-16T14:07:45.830063+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:36.893741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.968811+0000","last_change":"2023-10-16T14:07:44.817973+0000","last_active":"2023-10-16T14:07:45.968811+0000","last_peered":"2023-10-16T14:07:45.968811+0000","last_clean":"2023-10-16T14:07:45.968811+0000","last_became_active":"2023-10-16T14:07:44.817652+0000","last_became_peered":"2023-10-16T14:07:44.817652+0000","last_unstale":"2023-10-16T14:07:45.968811+0000","last_undegraded":"2023-10-16T14:07:45.968811+0000","last_fullsized":"2023-10-16T14:07:45.968811+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:25.522295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.712610+0000","last_change":"2023-10-16T14:07:43.799949+0000","last_active":"2023-10-16T14:07:46.712610+0000","last_peered":"2023-10-16T14:07:46.712610+0000","last_clean":"2023-10-16T14:07:46.712610+0000","last_became_active":"2023-10-16T14:07:43.799257+0000","last_became_peered":"2023-10-16T14:07:43.799257+0000","last_unstale":"2023-10-16T14:07:46.712610+0000","last_undegraded":"2023-10-16T14:07:46.712610+0000","last_fullsized":"2023-10-16T14:07:46.712610+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:20.648407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:45.823157+0000","last_change":"2023-10-16T14:07:44.827336+0000","last_active":"2023-10-16T14:07:45.823157+0000","last_peered":"2023-10-16T14:07:45.823157+0000","last_clean":"2023-10-16T14:07:45.823157+0000","last_became_active":"2023-10-16T14:07:44.825558+0000","last_became_peered":"2023-10-16T14:07:44.825558+0000","last_unstale":"2023-10-16T14:07:45.823157+0000","last_undegraded":"2023-10-16T14:07:45.823157+0000","last_fullsized":"2023-10-16T14:07:45.823157+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:43.752523+0000","last_clean_scrub_stamp":"2023-10-16T14:07:43.752523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:40.029184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-10-16T14:07:46.211814+0000","last_change":"2023-10-16T14:07:43.797242+0000","last_active":"2023-10-16T14:07:46.211814+0000","last_peered":"2023-10-16T14:07:46.211814+0000","last_clean":"2023-10-16T14:07:46.211814+0000","last_became_active":"2023-10-16T14:07:43.796965+0000","last_became_peered":"2023-10-16T14:07:43.796965+0000","last_unstale":"2023-10-16T14:07:46.211814+0000","last_undegraded":"2023-10-16T14:07:46.211814+0000","last_fullsized":"2023-10-16T14:07:46.211814+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:07:42.737898+0000","last_clean_scrub_stamp":"2023-10-16T14:07:42.737898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:32.054938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'329","reported_seq":1128,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-10-16T14:07:44.803382+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:07:44.803382+0000","last_peered":"2023-10-16T14:07:44.803382+0000","last_clean":"2023-10-16T14:07:44.803382+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:07:44.803382+0000","last_undegraded":"2023-10-16T14:07:44.803382+0000","last_fullsized":"2023-10-16T14:07:44.803382+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1963,"num_read_kb":4096,"num_write":543,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":329,"ondisk_log_size":329,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52416,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319424,"statfs":{"total":96636764160,"available":96583090176,"internally_reserved":0,"allocated":3264512,"data_stored":1886153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.381,"15min":1.365},"min":{"1min":0.77800000000000002,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7050000000000001,"5min":1.792,"15min":2.0659999999999998},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4379999999999999,"15min":1.4259999999999999},"min":{"1min":1.048,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.7190000000000001,"5min":1.829,"15min":2.0179999999999998},"last":1.395}]},{"osd":1,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.222,"15min":1.248},"min":{"1min":0.86499999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.5369999999999999,"5min":1.776,"15min":1.8999999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.2769999999999999,"15min":1.2709999999999999},"min":{"1min":0.85399999999999998,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.0620000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.292,"15min":1.278},"min":{"1min":0.96699999999999997,"5min":0.89400000000000002,"15min":0.86199999999999999},"max":{"1min":1.714,"5min":1.8109999999999999,"15min":1.8700000000000001},"last":1.347},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.395,"15min":1.3740000000000001},"min":{"1min":0.97299999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.6879999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4430000000000001,"15min":1.431},"min":{"1min":0.998,"5min":0.96699999999999997,"15min":0.91300000000000003},"max":{"1min":1.651,"5min":1.9119999999999999,"15min":2.097},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.4610000000000001,"15min":1.4550000000000001},"min":{"1min":1.1539999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8740000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.5620000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.2210000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.76600000000000001},"max":{"1min":1.581,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.373,"15min":1.411},"min":{"1min":0.97799999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.675,"5min":1.833,"15min":2.0590000000000002},"last":1.6279999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3759999999999999,"15min":1.3879999999999999},"min":{"1min":1.018,"5min":0.88700000000000001,"15min":0.879},"max":{"1min":1.8169999999999999,"5min":2.0590000000000002,"15min":2.3730000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.5389999999999999,"15min":1.5580000000000001},"min":{"1min":1.0780000000000001,"5min":1,"15min":0.91500000000000004},"max":{"1min":1.869,"5min":1.9770000000000001,"15min":2.2229999999999999},"last":2.0609999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6259999999999999,"15min":1.647},"min":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":1.806,"5min":2.056,"15min":2.4169999999999998},"last":1.968},{"interface":"front","average":{"1min":1.52,"5min":1.4750000000000001,"15min":1.5},"min":{"1min":0.98899999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.145,"5min":2.145,"15min":2.423},"last":2.0110000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543268,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42100,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329740,"statfs":{"total":96636764160,"available":96593653760,"internally_reserved":0,"allocated":2662400,"data_stored":1295681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.357,"15min":1.3380000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.76800000000000002},"max":{"1min":1.587,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.3640000000000001,"15min":1.3759999999999999},"min":{"1min":1.0549999999999999,"5min":0.99299999999999999,"15min":0.91800000000000004},"max":{"1min":1.508,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.349}]},{"osd":1,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.228,"15min":1.2330000000000001},"min":{"1min":0.93600000000000005,"5min":0.83499999999999996,"15min":0.71799999999999997},"max":{"1min":1.5940000000000001,"5min":2.008,"15min":2.008},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.224,"15min":1.2509999999999999},"min":{"1min":0.87,"5min":0.82399999999999995,"15min":0.76800000000000002},"max":{"1min":1.748,"5min":1.77,"15min":1.9419999999999999},"last":0.94299999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.246,"15min":1.26},"min":{"1min":0.93799999999999994,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.6679999999999999,"5min":1.776,"15min":1.78},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":1.0680000000000001,"5min":0.96899999999999997,"15min":0.82199999999999995},"max":{"1min":1.647,"5min":1.8200000000000001,"15min":1.9390000000000001},"last":0.88500000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.401,"15min":1.363},"min":{"1min":1.147,"5min":0.96499999999999997,"15min":0.91600000000000004},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.4119999999999999,"15min":1.4159999999999999},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.92600000000000005},"max":{"1min":1.8440000000000001,"5min":1.8799999999999999,"15min":1.9259999999999999},"last":1.234}]},{"osd":4,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.431,"15min":1.4119999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.0529999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.575,"15min":1.536},"min":{"1min":1.2490000000000001,"5min":0.90500000000000003,"15min":0.77000000000000002},"max":{"1min":1.948,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.137}]},{"osd":5,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3819999999999999,"15min":1.3640000000000001},"min":{"1min":1.0209999999999999,"5min":0.85299999999999998,"15min":0.80800000000000005},"max":{"1min":1.716,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":1.177},{"interface":"front","average":{"1min":1.506,"5min":1.5209999999999999,"15min":1.5029999999999999},"min":{"1min":1.0329999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1909999999999998,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.3680000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:07:20 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5229999999999999,"15min":1.5069999999999999},"min":{"1min":1.109,"5min":0.94599999999999995,"15min":0.93200000000000005},"max":{"1min":2.004,"5min":2.2730000000000001,"15min":2.339},"last":1.397},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.5780000000000001,"15min":1.5449999999999999},"min":{"1min":1.264,"5min":1.0649999999999999,"15min":0.95299999999999996},"max":{"1min":1.8080000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.0780000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543269,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29620,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342220,"statfs":{"total":96636764160,"available":96606433280,"internally_reserved":0,"allocated":2662400,"data_stored":1295681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.2310000000000001},"min":{"1min":0.81699999999999995,"5min":0.77500000000000002,"15min":0.76000000000000001},"max":{"1min":1.492,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.226,"15min":1.2529999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.712,"5min":1.7729999999999999,"15min":1.7769999999999999},"last":1.411}]},{"osd":1,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.302,"15min":1.3140000000000001},"min":{"1min":0.93100000000000005,"5min":0.91200000000000003,"15min":0.85099999999999998},"max":{"1min":1.5449999999999999,"5min":1.823,"15min":1.823},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.88,"5min":0.754,"15min":0.754},"max":{"1min":1.591,"5min":1.873,"15min":1.917},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3759999999999999,"15min":1.3480000000000001},"min":{"1min":0.98699999999999999,"5min":0.89900000000000002,"15min":0.877},"max":{"1min":1.7490000000000001,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.333,"15min":1.3340000000000001},"min":{"1min":1.0169999999999999,"5min":0.90400000000000003,"15min":0.89100000000000001},"max":{"1min":1.764,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.375}]},{"osd":3,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.393,"15min":1.4199999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93100000000000005},"max":{"1min":1.7310000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.4390000000000001,"15min":1.4470000000000001},"min":{"1min":1.0629999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":1.9990000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.583}]},{"osd":4,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.296,"15min":1.284},"min":{"1min":0.92600000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6699999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.431,"15min":1.4319999999999999},"min":{"1min":1.135,"5min":0.87,"15min":0.84699999999999998},"max":{"1min":1.907,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.2989999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":0.91600000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.671,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4710000000000001,"15min":1.482},"min":{"1min":1.111,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8520000000000001,"5min":2.391,"15min":2.391},"last":1.8009999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:07:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.5269999999999999,"15min":1.5529999999999999},"min":{"1min":1.073,"5min":1.0269999999999999,"15min":0.97999999999999998},"max":{"1min":2.177,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.653},{"interface":"front","average":{"1min":1.597,"5min":1.589,"15min":1.6040000000000001},"min":{"1min":1.2509999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.1560000000000001,"5min":2.681,"15min":2.681},"last":1.839}]}]},{"osd":4,"up_from":14,"seq":60129543269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64320,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307520,"statfs":{"total":96636764160,"available":96570900480,"internally_reserved":0,"allocated":2666496,"data_stored":1291575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.226,"15min":1.218},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.478,"5min":1.8420000000000001,"15min":1.9039999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.196,"15min":1.2050000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8160000000000001,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.071}]},{"osd":1,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.288,"15min":1.2629999999999999},"min":{"1min":0.89900000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5640000000000001,"5min":1.7090000000000001,"15min":1.78},"last":1.004},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3200000000000001,"15min":1.3029999999999999},"min":{"1min":1.046,"5min":0.875,"15min":0.875},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.1539999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.339,"15min":1.321},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.653,"5min":1.865,"15min":1.865},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.383,"15min":1.389},"min":{"1min":1.0960000000000001,"5min":1.0129999999999999,"15min":0.84499999999999997},"max":{"1min":1.6120000000000001,"5min":1.7210000000000001,"15min":2.0110000000000001},"last":1.482}]},{"osd":3,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.427,"15min":1.425},"min":{"1min":0.93200000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":1.726,"5min":1.875,"15min":1.9319999999999999},"last":1.611}]},{"osd":5,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.325,"15min":1.3440000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.9930000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.3160000000000001,"15min":1.337},"min":{"1min":0.98999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.9510000000000001},"last":1.542}]},{"osd":6,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.369,"15min":1.401},"min":{"1min":0.95899999999999996,"5min":0.92500000000000004,"15min":0.86699999999999999},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.1179999999999999},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.4990000000000001,"15min":1.532},"min":{"1min":1.2450000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.859,"5min":1.859,"15min":2.6429999999999998},"last":1.587}]},{"osd":7,"last update":"Mon Oct 16 14:07:25 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4650000000000001,"15min":1.502},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.931,"5min":2.0049999999999999,"15min":2.1949999999999998},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.633,"15min":1.6459999999999999},"min":{"1min":1.1719999999999999,"5min":1.008,"15min":1.008},"max":{"1min":2.02,"5min":2.1349999999999998,"15min":2.4529999999999998},"last":1.8180000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51828,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320012,"statfs":{"total":96636764160,"available":96583692288,"internally_reserved":0,"allocated":2662400,"data_stored":1289575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.92200000000000004,"15min":0.95599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.173,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.214,"15min":1.2350000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.554,"5min":1.798,"15min":2.0579999999999998},"last":1.1279999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.0649999999999999,"15min":1.0940000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.296,"5min":1.855,"15min":1.9750000000000001},"last":1.194},{"interface":"front","average":{"1min":1.004,"5min":1.0920000000000001,"15min":1.1299999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6040000000000001,"5min":2.077,"15min":2.077},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.181,"15min":1.2150000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.8700000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.8420000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.4079999999999999,"15min":1.4610000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.879,"5min":1.879,"15min":2.1230000000000002},"last":1.9319999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.502,"15min":1.516},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.8009999999999999,"5min":1.9550000000000001,"15min":2.0489999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.448,"15min":1.4610000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.8089999999999999,"5min":1.8580000000000001,"15min":2.2109999999999999},"last":1.732}]},{"osd":5,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.5149999999999999,"15min":1.4890000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":2.1019999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.5489999999999999,"15min":1.5609999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":1.6639999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.512,"15min":1.516},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8260000000000001,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.48,"15min":1.5},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0249999999999999,"5min":2.407,"15min":2.407},"last":2.4180000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:07:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.548,"15min":1.6160000000000001},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":2.0449999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6619999999999999,"15min":1.6739999999999999},"min":{"1min":0.90600000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0419999999999998,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.7929999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543267,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51844,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319996,"statfs":{"total":96636764160,"available":96583675904,"internally_reserved":0,"allocated":2674688,"data_stored":1296375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.97299999999999998,"15min":0.97099999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5569999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.1830000000000001,"15min":1.171},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.88800000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1759999999999999,"15min":1.1839999999999999},"min":{"1min":0.875,"5min":0.80900000000000005,"15min":0.78100000000000003},"max":{"1min":1.6779999999999999,"5min":1.827,"15min":1.859},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.343,"5min":1.3009999999999999,"15min":1.2709999999999999},"min":{"1min":0.89900000000000002,"5min":0.87,"15min":0.70199999999999996},"max":{"1min":1.768,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.58099999999999996}]},{"osd":3,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.149,"15min":1.127},"min":{"1min":0.872,"5min":0.80500000000000005,"15min":0.74199999999999999},"max":{"1min":1.75,"5min":1.75,"15min":2.2719999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4179999999999999,"15min":1.4019999999999999},"min":{"1min":1.0580000000000001,"5min":0.96299999999999997,"15min":0.89100000000000001},"max":{"1min":1.8919999999999999,"5min":1.9279999999999999,"15min":2.0760000000000001},"last":0.625}]},{"osd":4,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.518,"15min":1.5249999999999999},"min":{"1min":1.1899999999999999,"5min":1.089,"15min":1.089},"max":{"1min":2.0590000000000002,"5min":2.0710000000000002,"15min":2.488},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.524,"15min":1.502},"min":{"1min":1.0700000000000001,"5min":0.999,"15min":0.999},"max":{"1min":2.0659999999999998,"5min":2.2530000000000001,"15min":2.4180000000000001},"last":1.169}]},{"osd":5,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.5940000000000001,"15min":1.5780000000000001},"min":{"1min":1.353,"5min":1.01,"15min":1.01},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4970000000000001,"15min":1.476},"min":{"1min":1.1160000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.921,"5min":2.23,"15min":2.3260000000000001},"last":1.6399999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.556,"15min":1.5269999999999999},"min":{"1min":1.48,"5min":1.056,"15min":1.0349999999999999},"max":{"1min":2.0379999999999998,"5min":2.323,"15min":2.6400000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.6499999999999999,"15min":1.6439999999999999},"min":{"1min":1.2889999999999999,"5min":1.25,"15min":1.2},"max":{"1min":2.1800000000000002,"5min":2.2999999999999998,"15min":2.7749999999999999},"last":1.1379999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.7070000000000001,"15min":1.673},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":0.96699999999999997},"max":{"1min":2.25,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.6619999999999999,"15min":1.653},"min":{"1min":1.2310000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"max":{"1min":2.149,"5min":2.3319999999999999,"15min":2.9159999999999999},"last":1.4890000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543267,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51828,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320012,"statfs":{"total":96636764160,"available":96583692288,"internally_reserved":0,"allocated":2662400,"data_stored":1290993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.96699999999999997,"15min":0.94499999999999995},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.624},"max":{"1min":1.387,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.1739999999999999,"15min":1.1619999999999999},"min":{"1min":0.77200000000000002,"5min":0.74099999999999999,"15min":0.72399999999999998},"max":{"1min":1.579,"5min":1.851,"15min":1.851},"last":1.8720000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.073,"15min":1.05},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.65000000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.234,"5min":1.2629999999999999,"15min":1.274},"min":{"1min":0.88600000000000001,"5min":0.84599999999999997,"15min":0.80900000000000005},"max":{"1min":1.677,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.5509999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.4330000000000001,"15min":1.413},"min":{"1min":1.153,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.3009999999999999,"15min":1.306},"min":{"1min":0.89600000000000002,"5min":0.872,"15min":0.81499999999999995},"max":{"1min":1.7150000000000001,"5min":1.835,"15min":1.9299999999999999},"last":1.0229999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3999999999999999,"15min":1.375},"min":{"1min":1.0780000000000001,"5min":0.97299999999999998,"15min":0.92200000000000004},"max":{"1min":1.776,"5min":1.901,"15min":2.1259999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.5669999999999999,"15min":1.5389999999999999},"min":{"1min":1.264,"5min":1.151,"15min":1.042},"max":{"1min":2.133,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":1.681}]},{"osd":5,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5009999999999999,"15min":1.5009999999999999},"min":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.0589999999999999},"max":{"1min":1.8240000000000001,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.502,"15min":1.504},"min":{"1min":1.05,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.849,"5min":1.9359999999999999,"15min":2.1440000000000001},"last":1.4059999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.671,"15min":1.6479999999999999},"min":{"1min":1.3049999999999999,"5min":1.238,"15min":1.161},"max":{"1min":2.2309999999999999,"5min":2.4239999999999999,"15min":2.7200000000000002},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.631,"15min":1.6439999999999999},"min":{"1min":1.135,"5min":1.091,"15min":1.091},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.6379999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:07:16 2023","interfaces":[{"interface":"back","average":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.734},"min":{"1min":1.262,"5min":1.262,"15min":1.234},"max":{"1min":2.528,"5min":2.528,"15min":2.528},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.665,"15min":1.671},"min":{"1min":1.2889999999999999,"5min":1.1859999999999999,"15min":1.1419999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.5169999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543268,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48252,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323588,"statfs":{"total":96636764160,"available":96587354112,"internally_reserved":0,"allocated":3260416,"data_stored":1886071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.96199999999999997,"15min":0.93000000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.292,"5min":1.6699999999999999,"15min":1.891},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.2,"5min":1.1950000000000001,"15min":1.1899999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.48699999999999999},"max":{"1min":1.659,"5min":1.851,"15min":1.9299999999999999},"last":0.88500000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.22,"15min":1.173},"min":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.76000000000000001},"max":{"1min":1.641,"5min":2.0569999999999999,"15min":2.0569999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2509999999999999,"15min":1.252},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.734,"5min":1.734,"15min":1.887},"last":0.75}]},{"osd":3,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.1970000000000001,"15min":1.1759999999999999},"min":{"1min":0.81200000000000006,"5min":0.79200000000000004,"15min":0.72599999999999998},"max":{"1min":1.609,"5min":1.8520000000000001,"15min":1.8600000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.446,"15min":1.4039999999999999},"min":{"1min":1.1120000000000001,"5min":0.96399999999999997,"15min":0.73799999999999999},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.383,"15min":1.4019999999999999},"min":{"1min":1.1120000000000001,"5min":0.98599999999999999,"15min":0.91600000000000004},"max":{"1min":1.819,"5min":2.0070000000000001,"15min":2.048},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.5780000000000001,"15min":1.526},"min":{"1min":1.133,"5min":1.133,"15min":1.0169999999999999},"max":{"1min":2.601,"5min":2.601,"15min":2.601},"last":1.2809999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.52,"15min":1.5209999999999999},"min":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0740000000000001},"max":{"1min":2.2250000000000001,"5min":2.3279999999999998,"15min":2.4300000000000002},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5009999999999999,"15min":1.5089999999999999},"min":{"1min":1.22,"5min":1.111,"15min":0.92500000000000004},"max":{"1min":1.915,"5min":2.2829999999999999,"15min":2.2869999999999999},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6579999999999999,"15min":1.629},"min":{"1min":1.266,"5min":1.2130000000000001,"15min":1.1319999999999999},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.2},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.627,"15min":1.623},"min":{"1min":1.1499999999999999,"5min":1.1140000000000001,"15min":0.97199999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.3769999999999998},"last":1.139}]},{"osd":7,"last update":"Mon Oct 16 14:07:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.7410000000000001,"15min":1.73},"min":{"1min":1.26,"5min":1.2529999999999999,"15min":1.095},"max":{"1min":2.484,"5min":2.484,"15min":2.5270000000000001},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.671,"15min":1.6659999999999999},"min":{"1min":1.3859999999999999,"5min":1.1850000000000001,"15min":1.1579999999999999},"max":{"1min":2.2949999999999999,"5min":2.464,"15min":2.464},"last":1.3280000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:07:49.864 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:07:49.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:07:50.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.134+0000 7f8a74ebb640 1 Processor -- start 2023-10-16T14:07:50.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.134+0000 7f8a74ebb640 1 -- start start 2023-10-16T14:07:50.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a74ebb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a700ca850 0x7f8a7015a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a74ebb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a70070820 0x7f8a700ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a74ebb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a7006f230 0x7f8a700cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a74ebb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a700cac30 con 0x7f8a70070820 2023-10-16T14:07:50.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a74ebb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7015a660 con 0x7f8a700ca850 2023-10-16T14:07:50.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a74ebb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700ce050 con 0x7f8a7006f230 2023-10-16T14:07:50.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.136+0000 7f8a67fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a70070820 0x7f8a700ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8a70070820 0x7f8a700ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59904/0 (socket says 172.21.15.60:59904) 2023-10-16T14:07:50.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 -- 172.21.15.60:0/3380721892 learned_addr learned my addr 172.21.15.60:0/3380721892 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:50.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a6f7fe640 1 --2- 172.21.15.60:0/3380721892 >> v2:172.21.15.144:3300/0 conn(0x7f8a700ca850 0x7f8a7015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 -- 172.21.15.60:0/3380721892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a7006f230 msgr2=0x7f8a700cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 --2- 172.21.15.60:0/3380721892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a7006f230 0x7f8a700cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 -- 172.21.15.60:0/3380721892 >> v2:172.21.15.144:3300/0 conn(0x7f8a700ca850 msgr2=0x7f8a7015a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 --2- 172.21.15.60:0/3380721892 >> v2:172.21.15.144:3300/0 conn(0x7f8a700ca850 0x7f8a7015a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.137+0000 7f8a67fff640 1 -- 172.21.15.60:0/3380721892 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7006f780 con 0x7f8a70070820 2023-10-16T14:07:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.138+0000 7f8a67fff640 1 --2- 172.21.15.60:0/3380721892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a70070820 0x7f8a700ccb00 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009870 tx=0x7f8a5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a083fbf7f17b53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:50.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.138+0000 7f8a6effd640 1 -- 172.21.15.60:0/3380721892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58031640 con 0x7f8a70070820 2023-10-16T14:07:50.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.138+0000 7f8a6effd640 1 -- 172.21.15.60:0/3380721892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a580317a0 con 0x7f8a70070820 2023-10-16T14:07:50.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.138+0000 7f8a6effd640 1 -- 172.21.15.60:0/3380721892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58031b30 con 0x7f8a70070820 2023-10-16T14:07:50.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 -- 172.21.15.60:0/3380721892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a70070820 msgr2=0x7f8a700ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/3380721892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a70070820 0x7f8a700ccb00 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009870 tx=0x7f8a5802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 -- 172.21.15.60:0/3380721892 shutdown_connections 2023-10-16T14:07:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/3380721892 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a7006f230 0x7f8a700cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/3380721892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a70070820 0x7f8a700ccb00 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/3380721892 >> v2:172.21.15.144:3300/0 conn(0x7f8a700ca850 0x7f8a7015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 -- 172.21.15.60:0/3380721892 >> 172.21.15.60:0/3380721892 conn(0x7f8a700ce6a0 msgr2=0x7f8a700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 -- 172.21.15.60:0/3380721892 shutdown_connections 2023-10-16T14:07:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.139+0000 7f8a74ebb640 1 -- 172.21.15.60:0/3380721892 wait complete. 2023-10-16T14:07:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.142+0000 7f8a74ebb640 1 Processor -- start 2023-10-16T14:07:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.142+0000 7f8a74ebb640 1 -- start start 2023-10-16T14:07:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.143+0000 7f8a74ebb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a7006f230 0x7f8a701586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.143+0000 7f8a74ebb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a70070820 0x7f8a701488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.143+0000 7f8a74ebb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 0x7f8a701561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.143+0000 7f8a74ebb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70157410 con 0x7f8a700ca850 2023-10-16T14:07:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.143+0000 7f8a74ebb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a701e2ab0 con 0x7f8a7006f230 2023-10-16T14:07:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.143+0000 7f8a74ebb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a701e2d30 con 0x7f8a70070820 2023-10-16T14:07:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 0x7f8a701561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 0x7f8a701561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59906/0 (socket says 172.21.15.60:59906) 2023-10-16T14:07:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 -- 172.21.15.60:0/2145856513 learned_addr learned my addr 172.21.15.60:0/2145856513 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6f7fe640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a7006f230 0x7f8a701586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a67fff640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a70070820 0x7f8a701488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 -- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a70070820 msgr2=0x7f8a701488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a70070820 0x7f8a701488b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 -- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a7006f230 msgr2=0x7f8a701586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a7006f230 0x7f8a701586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.144+0000 7f8a6ffff640 1 -- 172.21.15.60:0/2145856513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a580094d0 con 0x7f8a700ca850 2023-10-16T14:07:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.145+0000 7f8a6ffff640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 0x7f8a701561f0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800b9a0 tx=0x7f8a6800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.145+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800cba0 con 0x7f8a700ca850 2023-10-16T14:07:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.145+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6800cd20 con 0x7f8a700ca850 2023-10-16T14:07:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.145+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68014680 con 0x7f8a700ca850 2023-10-16T14:07:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.145+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a70156dc0 con 0x7f8a700ca850 2023-10-16T14:07:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.145+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a701e3010 con 0x7f8a700ca850 2023-10-16T14:07:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.146+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a701575c0 con 0x7f8a700ca850 2023-10-16T14:07:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.147+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a6801d3b0 con 0x7f8a700ca850 2023-10-16T14:07:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.148+0000 7f8a6cff9640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8a5006af90 0x7f8a5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.148+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a68089730 con 0x7f8a700ca850 2023-10-16T14:07:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.149+0000 7f8a6f7fe640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8a5006af90 0x7f8a5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.150+0000 7f8a6f7fe640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8a5006af90 0x7f8a5006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a60006fd0 tx=0x7f8a60008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.152+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a701575c0 con 0x7f8a700ca850 2023-10-16T14:07:50.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.419+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8a701575c0 con 0x7f8a700ca850 2023-10-16T14:07:50.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.424+0000 7f8a6cff9640 1 -- 172.21.15.60:0/2145856513 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8a701575c0 con 0x7f8a700ca850 2023-10-16T14:07:50.427 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T14:07:50.427 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T14:07:50.428 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T14:07:50.428 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T14:07:50.428 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:07:50.428 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T14:07:50.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8a5006af90 msgr2=0x7f8a5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8a5006af90 0x7f8a5006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a60006fd0 tx=0x7f8a60008040 comp rx=0 tx=0).stop 2023-10-16T14:07:50.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 msgr2=0x7f8a701561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 0x7f8a701561f0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800b9a0 tx=0x7f8a6800be60 comp rx=0 tx=0).stop 2023-10-16T14:07:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 shutdown_connections 2023-10-16T14:07:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8a70070820 0x7f8a701488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f8a5006af90 0x7f8a5006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a700ca850 0x7f8a701561f0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 --2- 172.21.15.60:0/2145856513 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a7006f230 0x7f8a701586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 >> 172.21.15.60:0/2145856513 conn(0x7f8a700ce6a0 msgr2=0x7f8a7006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:50.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 shutdown_connections 2023-10-16T14:07:50.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.427+0000 7f8a74ebb640 1 -- 172.21.15.60:0/2145856513 wait complete. 2023-10-16T14:07:50.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:07:50.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.667+0000 7f3e35747640 1 Processor -- start 2023-10-16T14:07:50.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.667+0000 7f3e35747640 1 -- start start 2023-10-16T14:07:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.668+0000 7f3e35747640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e30070b60 0x7f3e3015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.668+0000 7f3e35747640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3015aef0 con 0x7f3e30159950 2023-10-16T14:07:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.668+0000 7f3e35747640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3015d910 con 0x7f3e3006c100 2023-10-16T14:07:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.668+0000 7f3e35747640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3015e490 con 0x7f3e30070b60 2023-10-16T14:07:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e2ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3e30159950 0x7f3e30070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34606/0 (socket says 172.21.15.60:34606) 2023-10-16T14:07:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e2ffff640 1 -- 172.21.15.60:0/334454077 learned_addr learned my addr 172.21.15.60:0/334454077 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e277fe640 1 --2- 172.21.15.60:0/334454077 >> v2:172.21.15.60:3300/0 conn(0x7f3e30070b60 0x7f3e3015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e2f7fe640 1 -- 172.21.15.60:0/334454077 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281184655 0 0) 0x7f3e20001490 con 0x7f3e30159950 2023-10-16T14:07:50.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e2f7fe640 1 -- 172.21.15.60:0/334454077 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c005730 con 0x7f3e30159950 2023-10-16T14:07:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e277fe640 1 -- 172.21.15.60:0/334454077 >> v1:172.21.15.144:6789/0 conn(0x7f3e3006c100 legacy=0x7f3e3006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e277fe640 1 -- 172.21.15.60:0/334454077 >> v1:172.21.15.60:6789/0 conn(0x7f3e30159950 legacy=0x7f3e30070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.669+0000 7f3e277fe640 1 -- 172.21.15.60:0/334454077 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3015dbf0 con 0x7f3e30070b60 2023-10-16T14:07:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.670+0000 7f3e277fe640 1 --2- 172.21.15.60:0/334454077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e30070b60 0x7f3e3015a9b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f3e180097a0 tx=0x7f3e1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9332c3c486dc86d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e2f7fe640 1 -- 172.21.15.60:0/334454077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1802e070 con 0x7f3e30070b60 2023-10-16T14:07:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e2f7fe640 1 -- 172.21.15.60:0/334454077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e180315a0 con 0x7f3e30070b60 2023-10-16T14:07:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e2f7fe640 1 -- 172.21.15.60:0/334454077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18031930 con 0x7f3e30070b60 2023-10-16T14:07:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e35747640 1 -- 172.21.15.60:0/334454077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e30070b60 msgr2=0x7f3e3015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e35747640 1 --2- 172.21.15.60:0/334454077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e30070b60 0x7f3e3015a9b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f3e180097a0 tx=0x7f3e1802b840 comp rx=0 tx=0).stop 2023-10-16T14:07:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e35747640 1 -- 172.21.15.60:0/334454077 shutdown_connections 2023-10-16T14:07:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e35747640 1 --2- 172.21.15.60:0/334454077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e30070b60 0x7f3e3015a9b0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e35747640 1 -- 172.21.15.60:0/334454077 >> 172.21.15.60:0/334454077 conn(0x7f3e300ce6a0 msgr2=0x7f3e300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.671+0000 7f3e35747640 1 -- 172.21.15.60:0/334454077 shutdown_connections 2023-10-16T14:07:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.672+0000 7f3e35747640 1 -- 172.21.15.60:0/334454077 wait complete. 2023-10-16T14:07:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.673+0000 7f3e35747640 1 Processor -- start 2023-10-16T14:07:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.673+0000 7f3e35747640 1 -- start start 2023-10-16T14:07:50.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.674+0000 7f3e35747640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 0x7f3e300c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.674+0000 7f3e35747640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e30070b60 0x7f3e300c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.674+0000 7f3e35747640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e30159950 0x7f3e3015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.674+0000 7f3e35747640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300c6f40 con 0x7f3e3006c100 2023-10-16T14:07:50.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.674+0000 7f3e35747640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300c6980 con 0x7f3e30159950 2023-10-16T14:07:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.674+0000 7f3e35747640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300c6c60 con 0x7f3e30070b60 2023-10-16T14:07:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 0x7f3e300c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 0x7f3e300c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59928/0 (socket says 172.21.15.60:59928) 2023-10-16T14:07:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 -- 172.21.15.60:0/3888189991 learned_addr learned my addr 172.21.15.60:0/3888189991 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 -- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e30070b60 msgr2=0x7f3e300c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e30070b60 0x7f3e300c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 -- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e30159950 msgr2=0x7f3e3015ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e30159950 0x7f3e3015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 -- 172.21.15.60:0/3888189991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18009400 con 0x7f3e3006c100 2023-10-16T14:07:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.675+0000 7f3e2ffff640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 0x7f3e300c60a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000cbb0 tx=0x7f3e2000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.676+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000ad40 con 0x7f3e3006c100 2023-10-16T14:07:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.676+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e20012400 con 0x7f3e3006c100 2023-10-16T14:07:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.676+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000aea0 con 0x7f3e3006c100 2023-10-16T14:07:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.676+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3015c240 con 0x7f3e3006c100 2023-10-16T14:07:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.676+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e300c65e0 con 0x7f3e3006c100 2023-10-16T14:07:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.677+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3015c760 con 0x7f3e3006c100 2023-10-16T14:07:50.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.678+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e200127d0 con 0x7f3e3006c100 2023-10-16T14:07:50.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.679+0000 7f3e2d7fa640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f3e1006af90 0x7f3e1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:50.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.679+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e20089700 con 0x7f3e3006c100 2023-10-16T14:07:50.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.680+0000 7f3e277fe640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f3e1006af90 0x7f3e1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:50.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.681+0000 7f3e277fe640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f3e1006af90 0x7f3e1006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e18036000 tx=0x7f3e18030000 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:50.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.683+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e3015c760 con 0x7f3e3006c100 2023-10-16T14:07:50.702 INFO:tasks.ceph.mds.b.smithi144.stderr:2023-10-16T14:07:50.699+0000 7ff6f5864640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:07:50.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.916+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e3015dea0 con 0x7f3e3006c100 2023-10-16T14:07:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.917+0000 7f3e2d7fa640 1 -- 172.21.15.60:0/3888189991 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f3e20058090 con 0x7f3e3006c100 2023-10-16T14:07:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:07:50.920 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:07:44.252784+0000","modified":"2023-10-16T14:07:47.343844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8539":{"gid":8539,"name":"b","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6835/3999834969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3999834969},{"type":"v1","addr":"172.21.15.144:6835","nonce":3999834969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T14:07:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 104 2023-10-16T14:07:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.919+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f3e1006af90 msgr2=0x7f3e1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.919+0000 7f3e35747640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f3e1006af90 0x7f3e1006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e18036000 tx=0x7f3e18030000 comp rx=0 tx=0).stop 2023-10-16T14:07:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.919+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 msgr2=0x7f3e300c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.919+0000 7f3e35747640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 0x7f3e300c60a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000cbb0 tx=0x7f3e2000a430 comp rx=0 tx=0).stop 2023-10-16T14:07:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 shutdown_connections 2023-10-16T14:07:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3e30070b60 0x7f3e300c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f3e1006af90 0x7f3e1006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e30159950 0x7f3e3015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 --2- 172.21.15.60:0/3888189991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e3006c100 0x7f3e300c60a0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 >> 172.21.15.60:0/3888189991 conn(0x7f3e300ce6a0 msgr2=0x7f3e3006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 shutdown_connections 2023-10-16T14:07:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:50.920+0000 7f3e35747640 1 -- 172.21.15.60:0/3888189991 wait complete. 2023-10-16T14:07:51.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:07:44.252784+0000', 'modified': '2023-10-16T14:07:47.343844+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8539}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8539': {'gid': 8539, 'name': 'b', 'rank': 0, 'incarnation': 100, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.144:6835/3999834969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6834', 'nonce': 3999834969}, {'type': 'v1', 'addr': '172.21.15.144:6835', 'nonce': 3999834969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:07:51.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:07:51.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T14:07:51.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.575+0000 7f029575a640 1 Processor -- start 2023-10-16T14:07:51.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.575+0000 7f029575a640 1 -- start start 2023-10-16T14:07:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f029575a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 0x7f02900ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f029575a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02900c9ca0 0x7f02900ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f029575a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02900cb030 con 0x7f029015ad70 2023-10-16T14:07:51.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f029575a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02900cae30 con 0x7f02900c9ca0 2023-10-16T14:07:51.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f029575a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02900c91c0 con 0x7f029006a4b0 2023-10-16T14:07:51.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 0x7f02900ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.577+0000 7f028f7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02900c9ca0 0x7f02900ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.578+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 0x7f02900ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:46878/0 (socket says 172.21.15.60:46878) 2023-10-16T14:07:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.578+0000 7f028e7fc640 1 -- 172.21.15.60:0/2351517288 learned_addr learned my addr 172.21.15.60:0/2351517288 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.578+0000 7f028e7fc640 1 -- 172.21.15.60:0/2351517288 >> v2:172.21.15.144:3300/0 conn(0x7f02900c9ca0 msgr2=0x7f02900ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.578+0000 7f028e7fc640 1 --2- 172.21.15.60:0/2351517288 >> v2:172.21.15.144:3300/0 conn(0x7f02900c9ca0 0x7f02900ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.578+0000 7f028e7fc640 1 -- 172.21.15.60:0/2351517288 >> v1:172.21.15.144:6789/0 conn(0x7f029015ad70 legacy=0x7f0290069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.578+0000 7f028e7fc640 1 -- 172.21.15.60:0/2351517288 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02900c94a0 con 0x7f029006a4b0 2023-10-16T14:07:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.580+0000 7f028e7fc640 1 --2- 172.21.15.60:0/2351517288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 0x7f02900ca3b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0278009870 tx=0x7f027802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=455b15c5a879a954 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.580+0000 7f028dffb640 1 -- 172.21.15.60:0/2351517288 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278030840 con 0x7f029006a4b0 2023-10-16T14:07:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.580+0000 7f028dffb640 1 -- 172.21.15.60:0/2351517288 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02780309a0 con 0x7f029006a4b0 2023-10-16T14:07:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.580+0000 7f028dffb640 1 -- 172.21.15.60:0/2351517288 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278030d30 con 0x7f029006a4b0 2023-10-16T14:07:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 -- 172.21.15.60:0/2351517288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 msgr2=0x7f02900ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 --2- 172.21.15.60:0/2351517288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 0x7f02900ca3b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0278009870 tx=0x7f027802b980 comp rx=0 tx=0).stop 2023-10-16T14:07:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 -- 172.21.15.60:0/2351517288 shutdown_connections 2023-10-16T14:07:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 --2- 172.21.15.60:0/2351517288 >> v2:172.21.15.144:3300/0 conn(0x7f02900c9ca0 0x7f02900ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 --2- 172.21.15.60:0/2351517288 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029006a4b0 0x7f02900ca3b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 -- 172.21.15.60:0/2351517288 >> 172.21.15.60:0/2351517288 conn(0x7f02900ce9b0 msgr2=0x7f02900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 -- 172.21.15.60:0/2351517288 shutdown_connections 2023-10-16T14:07:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.581+0000 7f029575a640 1 -- 172.21.15.60:0/2351517288 wait complete. 2023-10-16T14:07:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.583+0000 7f029575a640 1 Processor -- start 2023-10-16T14:07:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.583+0000 7f029575a640 1 -- start start 2023-10-16T14:07:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f029575a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 0x7f02900c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f029575a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02900c9ca0 0x7f02900aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f029575a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029015ad70 0x7f02900af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f029575a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02900ae0f0 con 0x7f029006a4b0 2023-10-16T14:07:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f029575a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02900ad4e0 con 0x7f02900c9ca0 2023-10-16T14:07:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f029575a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02900ad7c0 con 0x7f029015ad70 2023-10-16T14:07:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.584+0000 7f028effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 0x7f02900c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 0x7f02900c7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59938/0 (socket says 172.21.15.60:59938) 2023-10-16T14:07:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 -- 172.21.15.60:0/3296860057 learned_addr learned my addr 172.21.15.60:0/3296860057 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028f7fe640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029015ad70 0x7f02900af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028e7fc640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02900c9ca0 0x7f02900aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 -- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029015ad70 msgr2=0x7f02900af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029015ad70 0x7f02900af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 -- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02900c9ca0 msgr2=0x7f02900aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02900c9ca0 0x7f02900aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 -- 172.21.15.60:0/3296860057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02780094d0 con 0x7f029006a4b0 2023-10-16T14:07:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.585+0000 7f028effd640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 0x7f02900c7d00 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f028000c400 tx=0x7f028000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.586+0000 7f02877fe640 1 -- 172.21.15.60:0/3296860057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02800052d0 con 0x7f029006a4b0 2023-10-16T14:07:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.586+0000 7f02877fe640 1 -- 172.21.15.60:0/3296860057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0280005430 con 0x7f029006a4b0 2023-10-16T14:07:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.586+0000 7f02877fe640 1 -- 172.21.15.60:0/3296860057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280003830 con 0x7f029006a4b0 2023-10-16T14:07:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.586+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02900adaa0 con 0x7f029006a4b0 2023-10-16T14:07:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.586+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02900c8240 con 0x7f029006a4b0 2023-10-16T14:07:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.589+0000 7f02877fe640 1 -- 172.21.15.60:0/3296860057 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0280003990 con 0x7f029006a4b0 2023-10-16T14:07:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.590+0000 7f02877fe640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f027406af90 0x7f027406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.590+0000 7f02877fe640 1 -- 172.21.15.60:0/3296860057 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0280088db0 con 0x7f029006a4b0 2023-10-16T14:07:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.590+0000 7f02983cb740 1 Processor -- start 2023-10-16T14:07:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.590+0000 7f02983cb740 1 -- start start 2023-10-16T14:07:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f028e7fc640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f027406af90 0x7f027406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f02983cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d62701ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 0x55d62704cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62701d3c0 0x55d62704d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f02983cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f3d7a0 con 0x55d627049f80 2023-10-16T14:07:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f8f740 con 0x55d62701d3c0 2023-10-16T14:07:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.591+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d626f8ce30 con 0x55d62704a360 2023-10-16T14:07:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d62701ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d62701ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59948/0 (socket says 172.21.15.60:59948) 2023-10-16T14:07:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 -- 172.21.15.60:0/3528477463 learned_addr learned my addr 172.21.15.60:0/3528477463 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028e7fc640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f027406af90 0x7f027406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0278036000 tx=0x7f0278008030 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028f7fe640 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62701d3c0 0x55d62704d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 -- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 msgr2=0x55d62704cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 0x55d62704cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 -- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62701d3c0 msgr2=0x55d62704d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62701d3c0 0x55d62704d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.592+0000 7f028effd640 1 -- 172.21.15.60:0/3528477463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d626f4c910 con 0x55d627049f80 2023-10-16T14:07:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.593+0000 7f028effd640 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d62701ce80 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0280000f30 tx=0x7f0280058130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6560cec2d7cafbf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.593+0000 7f0284ff9640 1 -- 172.21.15.60:0/3528477463 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028005fc70 con 0x55d627049f80 2023-10-16T14:07:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.593+0000 7f0284ff9640 1 -- 172.21.15.60:0/3528477463 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028006f170 con 0x55d627049f80 2023-10-16T14:07:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.593+0000 7f0284ff9640 1 -- 172.21.15.60:0/3528477463 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280054080 con 0x55d627049f80 2023-10-16T14:07:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.593+0000 7f02983cb740 1 -- 172.21.15.60:0/3528477463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 msgr2=0x55d62701ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.593+0000 7f02983cb740 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d62701ce80 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0280000f30 tx=0x7f0280058130 comp rx=0 tx=0).stop 2023-10-16T14:07:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 -- 172.21.15.60:0/3528477463 shutdown_connections 2023-10-16T14:07:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 0x55d62704cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62701d3c0 0x55d62704d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 --2- 172.21.15.60:0/3528477463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d62701ce80 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 -- 172.21.15.60:0/3528477463 >> 172.21.15.60:0/3528477463 conn(0x55d62701f190 msgr2=0x55d627067d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 -- 172.21.15.60:0/3528477463 shutdown_connections 2023-10-16T14:07:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 -- 172.21.15.60:0/3528477463 wait complete. 2023-10-16T14:07:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.594+0000 7f02983cb740 1 -- start start 2023-10-16T14:07:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.595+0000 7f02983cb740 1 Processor -- start 2023-10-16T14:07:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627048d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 0x55d627044490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f88720 con 0x55d62704a360 2023-10-16T14:07:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f26970 con 0x55d627049f80 2023-10-16T14:07:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d626f26610 con 0x55d627045fe0 2023-10-16T14:07:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f02983cb740 10 client.0 _ll_register_callbacks cb 0x55d626f489d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:07:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 0x55d627044490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 0x55d627044490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59956/0 (socket says 172.21.15.60:59956) 2023-10-16T14:07:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f028e7fc640 1 -- 172.21.15.60:0/1485933586 learned_addr learned my addr 172.21.15.60:0/1485933586 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f028effd640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627048d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.596+0000 7f028f7fe640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f028e7fc640 1 -- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 msgr2=0x55d6270430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f028e7fc640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f028e7fc640 1 -- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 msgr2=0x55d627048d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f028e7fc640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627048d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f028e7fc640 1 -- 172.21.15.60:0/1485933586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02900c8240 con 0x55d62704a360 2023-10-16T14:07:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f028e7fc640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 0x55d627044490 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f0278009990 tx=0x7f02780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3918a6b41e978e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f027dffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:07:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f027dffb640 1 -- 172.21.15.60:0/1485933586 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278049070 con 0x55d62704a360 2023-10-16T14:07:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f027dffb640 1 -- 172.21.15.60:0/1485933586 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f027803e040 con 0x55d62704a360 2023-10-16T14:07:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.597+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d626faf160 con 0x55d62704a360 2023-10-16T14:07:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f027dffb640 1 -- 172.21.15.60:0/1485933586 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0278039720 con 0x55d62704a360 2023-10-16T14:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 10 client.10641 fetch_fsmap learned FSMap version 104 2023-10-16T14:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x55d626f86da0 con 0x55d62704a360 2023-10-16T14:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f027dffb640 1 -- 172.21.15.60:0/1485933586 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d626faf160 con 0x55d62704a360 2023-10-16T14:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f027dffb640 1 -- 172.21.15.60:0/1485933586 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 104) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0278042050 con 0x55d62704a360 2023-10-16T14:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 10 client.10641 fetch_fsmap finished waiting for FSMap version 104 2023-10-16T14:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 10 client.10641 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 20 client.10641 populate_metadata read hostname 'smithi060' 2023-10-16T14:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270424d0 0x55d62707b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 4 client.10641 mds_command: new command op to 8539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.598+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 --> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d626f5acc0 con 0x55d6270424d0 2023-10-16T14:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.599+0000 7f028effd640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270424d0 0x55d62707b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.600+0000 7f028effd640 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270424d0 0x55d62707b900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7568b55b8513ffa0 server_cookie=2b9d55f09e75990e in_seq=0 out_seq=0 2023-10-16T14:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.600+0000 7f027dffb640 10 client.10641 ms_handle_connect on v2:172.21.15.144:6834/3999834969 2023-10-16T14:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f027dffb640 1 -- 172.21.15.60:0/1485933586 <== mds.0 v2:172.21.15.144:6834/3999834969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d626f5acc0 con 0x55d6270424d0 2023-10-16T14:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f027dffb640 10 client.10641 handle_command_reply: tid=0 2023-10-16T14:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 client.10641 shutdown 2023-10-16T14:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 msgr2=0x55d627044490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 0x55d627044490 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f0278009990 tx=0x7f02780046c0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 shutdown_connections 2023-10-16T14:07:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270424d0 0x55d62707b900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d62704a360 0x55d627044490 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f027dffb640 0 client.10641 ms_handle_reset on v2:172.21.15.144:6834/3999834969 2023-10-16T14:07:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 --2- 172.21.15.60:0/1485933586 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627048d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.602+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 >> 172.21.15.60:0/1485933586 conn(0x55d62701d3c0 msgr2=0x55d627020f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.603+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 shutdown_connections 2023-10-16T14:07:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.603+0000 7f02983cb740 1 -- 172.21.15.60:0/1485933586 wait complete. 2023-10-16T14:07:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.603+0000 7f02983cb740 20 client.10641 trim_cache size 0 max 16384 2023-10-16T14:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.619+0000 7f02983cb740 1 Processor -- start 2023-10-16T14:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 -- start start 2023-10-16T14:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 0x55d6270412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627050640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 0x55d627067e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626cecf20 con 0x55d627045fe0 2023-10-16T14:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d627041810 con 0x55d627049f80 2023-10-16T14:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.620+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d627067920 con 0x55d62704a360 2023-10-16T14:07:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 0x55d6270412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 0x55d6270412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59966/0 (socket says 172.21.15.60:59966) 2023-10-16T14:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 -- 172.21.15.60:0/3598906187 learned_addr learned my addr 172.21.15.60:0/3598906187 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 -- 172.21.15.60:0/3598906187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 msgr2=0x55d627067e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 0x55d627067e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 -- 172.21.15.60:0/3598906187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 msgr2=0x55d627050640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627050640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 -- 172.21.15.60:0/3598906187 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d626f25b80 con 0x55d627045fe0 2023-10-16T14:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.621+0000 7f028effd640 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 0x55d6270412d0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f0280055cc0 tx=0x7f028001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8628eeaeb9fa9f19 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02857fa640 1 -- 172.21.15.60:0/3598906187 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028005fc70 con 0x55d627045fe0 2023-10-16T14:07:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02857fa640 1 -- 172.21.15.60:0/3598906187 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0280084070 con 0x55d627045fe0 2023-10-16T14:07:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02857fa640 1 -- 172.21.15.60:0/3598906187 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280087230 con 0x55d627045fe0 2023-10-16T14:07:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 -- 172.21.15.60:0/3598906187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 msgr2=0x55d6270412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 0x55d6270412d0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f0280055cc0 tx=0x7f028001e000 comp rx=0 tx=0).stop 2023-10-16T14:07:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 -- 172.21.15.60:0/3598906187 shutdown_connections 2023-10-16T14:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d62704a360 0x55d627067e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d627049f80 0x55d627050640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 --2- 172.21.15.60:0/3598906187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627045fe0 0x55d6270412d0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 -- 172.21.15.60:0/3598906187 >> 172.21.15.60:0/3598906187 conn(0x55d62701d3c0 msgr2=0x55d627020a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.622+0000 7f02983cb740 1 -- 172.21.15.60:0/3598906187 shutdown_connections 2023-10-16T14:07:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.623+0000 7f02983cb740 1 -- 172.21.15.60:0/3598906187 wait complete. 2023-10-16T14:07:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.623+0000 7f02983cb740 1 -- start start 2023-10-16T14:07:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.623+0000 7f02983cb740 1 Processor -- start 2023-10-16T14:07:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.624+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.624+0000 7f02983cb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d627044310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.624+0000 7f02983cb740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62704a360 0x55d627044850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.624+0000 7f02983cb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6270b9f50 con 0x55d627049f80 2023-10-16T14:07:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.624+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6270d1b20 con 0x55d62704a360 2023-10-16T14:07:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.624+0000 7f02983cb740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d627040f50 con 0x55d627045fe0 2023-10-16T14:07:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f02983cb740 10 client.0 _ll_register_callbacks cb 0x7f02880095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:07:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d627044310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d627044310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59968/0 (socket says 172.21.15.60:59968) 2023-10-16T14:07:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 -- 172.21.15.60:0/1382450480 learned_addr learned my addr 172.21.15.60:0/1382450480 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028f7fe640 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62704a360 0x55d627044850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 -- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 msgr2=0x55d6270b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 -- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62704a360 msgr2=0x55d627044850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62704a360 0x55d627044850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.625+0000 7f028e7fc640 1 -- 172.21.15.60:0/1382450480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d626f4c910 con 0x55d627049f80 2023-10-16T14:07:51.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f028e7fc640 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d627044310 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f0278006f40 tx=0x7f027803b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ef4608f00449a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f027dffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:07:51.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f027dffb640 1 -- 172.21.15.60:0/1382450480 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278030ea0 con 0x55d627049f80 2023-10-16T14:07:51.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f027dffb640 1 -- 172.21.15.60:0/1382450480 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f027804d840 con 0x55d627049f80 2023-10-16T14:07:51.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f028006f170 con 0x55d627049f80 2023-10-16T14:07:51.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f027dffb640 1 -- 172.21.15.60:0/1382450480 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0278039720 con 0x55d627049f80 2023-10-16T14:07:51.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f02983cb740 10 client.10647 fetch_fsmap learned FSMap version 104 2023-10-16T14:07:51.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.626+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x55d626f4b170 con 0x55d627049f80 2023-10-16T14:07:51.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f027dffb640 1 -- 172.21.15.60:0/1382450480 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028006f170 con 0x55d627049f80 2023-10-16T14:07:51.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f027dffb640 1 -- 172.21.15.60:0/1382450480 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 104) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f027804da00 con 0x55d627049f80 2023-10-16T14:07:51.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f02983cb740 10 client.10647 fetch_fsmap finished waiting for FSMap version 104 2023-10-16T14:07:51.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f02983cb740 10 client.10647 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:07:51.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f02983cb740 20 client.10647 populate_metadata read hostname 'smithi060' 2023-10-16T14:07:51.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f02983cb740 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270d0c60 0x55d6270d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f02983cb740 4 client.10647 mds_command: new command op to 8539 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:07:51.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 --> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d626f81700 con 0x55d6270d0c60 2023-10-16T14:07:51.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.627+0000 7f028effd640 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270d0c60 0x55d6270d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.628+0000 7f028effd640 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270d0c60 0x55d6270d1040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7faf239407d711a5 server_cookie=ceb331fa7263066c in_seq=0 out_seq=0 2023-10-16T14:07:51.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.628+0000 7f027dffb640 10 client.10647 ms_handle_connect on v2:172.21.15.144:6834/3999834969 2023-10-16T14:07:51.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f027dffb640 1 -- 172.21.15.60:0/1382450480 <== mds.0 v2:172.21.15.144:6834/3999834969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55d626f81700 con 0x55d6270d0c60 2023-10-16T14:07:51.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f027dffb640 10 client.10647 handle_command_reply: tid=0 2023-10-16T14:07:51.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 client.10647 shutdown 2023-10-16T14:07:51.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 msgr2=0x55d627044310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d627044310 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f0278006f40 tx=0x7f027803b040 comp rx=0 tx=0).stop 2023-10-16T14:07:51.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 shutdown_connections 2023-10-16T14:07:51.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:6834/3999834969,v1:172.21.15.144:6835/3999834969] conn(0x55d6270d0c60 0x55d6270d1040 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55d62704a360 0x55d627044850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d627049f80 0x55d627044310 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 --2- 172.21.15.60:0/1382450480 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55d627045fe0 0x55d6270b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f027dffb640 0 client.10647 ms_handle_reset on v2:172.21.15.144:6834/3999834969 2023-10-16T14:07:51.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.629+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 >> 172.21.15.60:0/1382450480 conn(0x55d62701d3c0 msgr2=0x55d62701be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 shutdown_connections 2023-10-16T14:07:51.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f02983cb740 1 -- 172.21.15.60:0/1382450480 wait complete. 2023-10-16T14:07:51.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f02983cb740 20 client.10647 trim_cache size 0 max 16384 2023-10-16T14:07:51.644 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:07:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:07:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:07:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 8539, 2023-10-16T14:07:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:07:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:07:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T14:07:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8155188420000004, 2023-10-16T14:07:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 104, 2023-10-16T14:07:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 213, 2023-10-16T14:07:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 213, 2023-10-16T14:07:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.667661329 2023-10-16T14:07:51.647 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:07:51.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f027406af90 msgr2=0x7f027406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f029575a640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f027406af90 0x7f027406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0278036000 tx=0x7f0278008030 comp rx=0 tx=0).stop 2023-10-16T14:07:51.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 msgr2=0x7f02900c7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.630+0000 7f029575a640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 0x7f02900c7d00 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f028000c400 tx=0x7f028000c8c0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 shutdown_connections 2023-10-16T14:07:51.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f029015ad70 0x7f02900af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f027406af90 0x7f027406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02900c9ca0 0x7f02900aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 --2- 172.21.15.60:0/3296860057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f029006a4b0 0x7f02900c7d00 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 >> 172.21.15.60:0/3296860057 conn(0x7f02900ce9b0 msgr2=0x7f02900cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 shutdown_connections 2023-10-16T14:07:51.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.631+0000 7f029575a640 1 -- 172.21.15.60:0/3296860057 wait complete. 2023-10-16T14:07:51.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:07:51.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.875+0000 7f0a69e1a640 1 Processor -- start 2023-10-16T14:07:51.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.875+0000 7f0a69e1a640 1 -- start start 2023-10-16T14:07:51.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.876+0000 7f0a69e1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.876+0000 7f0a69e1a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0a640c9a20 0x7f0a640ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.876+0000 7f0a69e1a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0a640acdd0 0x7f0a6406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.876+0000 7f0a69e1a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6415d190 con 0x7f0a640acdd0 2023-10-16T14:07:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.876+0000 7f0a69e1a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640ad240 con 0x7f0a640c9a20 2023-10-16T14:07:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.876+0000 7f0a69e1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640c8cf0 con 0x7f0a6415cdb0 2023-10-16T14:07:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0a640acdd0 0x7f0a6406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0a640acdd0 0x7f0a6406fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59976/0 (socket says 172.21.15.60:59976) 2023-10-16T14:07:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 -- 172.21.15.60:0/4250643932 learned_addr learned my addr 172.21.15.60:0/4250643932 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a63fff640 1 --2- 172.21.15.60:0/4250643932 >> v2:172.21.15.144:3300/0 conn(0x7f0a640c9a20 0x7f0a640ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a68e18640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 -- 172.21.15.60:0/4250643932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 msgr2=0x7f0a640cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 -- 172.21.15.60:0/4250643932 >> v2:172.21.15.144:3300/0 conn(0x7f0a640c9a20 msgr2=0x7f0a640ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 --2- 172.21.15.60:0/4250643932 >> v2:172.21.15.144:3300/0 conn(0x7f0a640c9a20 0x7f0a640ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a69619640 1 -- 172.21.15.60:0/4250643932 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640ccca0 con 0x7f0a640acdd0 2023-10-16T14:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.877+0000 7f0a68e18640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.878+0000 7f0a69619640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640acdd0 0x7f0a6406fea0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f0a6406d4d0 tx=0x7f0a5c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576c2f9e21a4fdf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.879+0000 7f0a637fe640 1 -- 172.21.15.60:0/4250643932 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c02c9f0 con 0x7f0a640acdd0 2023-10-16T14:07:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.879+0000 7f0a637fe640 1 -- 172.21.15.60:0/4250643932 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c02cb50 con 0x7f0a640acdd0 2023-10-16T14:07:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.879+0000 7f0a637fe640 1 -- 172.21.15.60:0/4250643932 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0343f0 con 0x7f0a640acdd0 2023-10-16T14:07:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.879+0000 7f0a69e1a640 1 -- 172.21.15.60:0/4250643932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640acdd0 msgr2=0x7f0a6406fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.879+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640acdd0 0x7f0a6406fea0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f0a6406d4d0 tx=0x7f0a5c02be50 comp rx=0 tx=0).stop 2023-10-16T14:07:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 -- 172.21.15.60:0/4250643932 shutdown_connections 2023-10-16T14:07:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640acdd0 0x7f0a6406fea0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/4250643932 >> v2:172.21.15.144:3300/0 conn(0x7f0a640c9a20 0x7f0a640ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/4250643932 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 -- 172.21.15.60:0/4250643932 >> 172.21.15.60:0/4250643932 conn(0x7f0a640b0a90 msgr2=0x7f0a640ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 -- 172.21.15.60:0/4250643932 shutdown_connections 2023-10-16T14:07:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.880+0000 7f0a69e1a640 1 -- 172.21.15.60:0/4250643932 wait complete. 2023-10-16T14:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.881+0000 7f0a69e1a640 1 Processor -- start 2023-10-16T14:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.882+0000 7f0a69e1a640 1 -- start start 2023-10-16T14:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a69e1a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a640acdd0 0x7f0a641549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a69e1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 0x7f0a640aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a69e1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a69e1a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415f8c0 con 0x7f0a640c9a20 2023-10-16T14:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a69e1a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415d310 con 0x7f0a640acdd0 2023-10-16T14:07:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a69e1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415de90 con 0x7f0a6415cdb0 2023-10-16T14:07:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 0x7f0a640aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 0x7f0a640aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59992/0 (socket says 172.21.15.60:59992) 2023-10-16T14:07:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 -- 172.21.15.60:0/1520330474 learned_addr learned my addr 172.21.15.60:0/1520330474 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 -- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 msgr2=0x7f0a640af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 -- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a640acdd0 msgr2=0x7f0a641549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a640acdd0 0x7f0a641549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.883+0000 7f0a63fff640 1 -- 172.21.15.60:0/1520330474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c009b30 con 0x7f0a640c9a20 2023-10-16T14:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.884+0000 7f0a63fff640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 0x7f0a640aee40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000d830 tx=0x7f0a5000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.884+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a500096a0 con 0x7f0a640c9a20 2023-10-16T14:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.884+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a50009800 con 0x7f0a640c9a20 2023-10-16T14:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.884+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50009b90 con 0x7f0a640c9a20 2023-10-16T14:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.884+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a64154f00 con 0x7f0a640c9a20 2023-10-16T14:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.884+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6415d840 con 0x7f0a640c9a20 2023-10-16T14:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.885+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a640c2020 con 0x7f0a640c9a20 2023-10-16T14:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.886+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0a50014410 con 0x7f0a640c9a20 2023-10-16T14:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.888+0000 7f0a617fa640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0a4806af90 0x7f0a4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.888+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a5008a0d0 con 0x7f0a640c9a20 2023-10-16T14:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.888+0000 7f0a68e18640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0a4806af90 0x7f0a4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.889+0000 7f0a68e18640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0a4806af90 0x7f0a4806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0a58006fd0 tx=0x7f0a58008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:51.893+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a50055060 con 0x7f0a640c9a20 2023-10-16T14:07:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.117+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0a6415dfd0 con 0x7f0a640c9a20 2023-10-16T14:07:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.118+0000 7f0a617fa640 1 -- 172.21.15.60:0/1520330474 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v104) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0a6415dfd0 con 0x7f0a640c9a20 2023-10-16T14:07:52.121 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:07:52.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0a4806af90 msgr2=0x7f0a4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:52.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0a4806af90 0x7f0a4806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0a58006fd0 tx=0x7f0a58008040 comp rx=0 tx=0).stop 2023-10-16T14:07:52.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 msgr2=0x7f0a640aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:52.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 0x7f0a640aee40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000d830 tx=0x7f0a5000dcf0 comp rx=0 tx=0).stop 2023-10-16T14:07:52.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 shutdown_connections 2023-10-16T14:07:52.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0a6415cdb0 0x7f0a640af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:52.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0a4806af90 0x7f0a4806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:52.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a640c9a20 0x7f0a640aee40 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:52.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 --2- 172.21.15.60:0/1520330474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a640acdd0 0x7f0a641549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:52.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 >> 172.21.15.60:0/1520330474 conn(0x7f0a640b0a90 msgr2=0x7f0a640cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:52.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 shutdown_connections 2023-10-16T14:07:52.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:52.121+0000 7f0a69e1a640 1 -- 172.21.15.60:0/1520330474 wait complete. 2023-10-16T14:07:52.143 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:07:52.144 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:07:52.144 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T14:07:52.144 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:07:52.145 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:07:52.145 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:07:52.145 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:07:52.146 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:07:52.146 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:07:52.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.147 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T14:07:52.201 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:07:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:07:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:07:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T14:07:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.204 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:07:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T14:07:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T14:07:52.205 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:07:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T14:07:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T14:07:52.205 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:07:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:07:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T14:07:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:07:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 36715sec preferred_lft 36715sec 2023-10-16T14:07:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T14:07:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.207 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:07:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:07:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T14:07:52.207 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:07:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T14:07:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:07:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T14:07:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.208 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:07:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:07:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T14:07:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.209 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:07:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:07:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T14:07:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:07:52.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.211 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:07:52.311 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T14:07:52.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.316 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T14:07:52.372 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T14:07:52.372 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:07:52.372 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:07:52.373 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:07:52.373 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:07:52.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.374 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T14:07:52.374 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T14:07:52.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:07:52.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:07:52.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:07:52.375 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T14:07:52.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.579 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:07:52.637 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:07:52.637 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:07:52.638 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:07:52.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T14:07:52.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:07:52.733 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:07:52.791 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:07:52.792 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:07:52.792 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:07:52.845 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T14:07:52.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:52.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:07:52.981 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:07:52.981 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:07:53.015 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:07:53.015 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:07:53.015 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T14:07:53.016 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T14:07:53.016 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T14:07:53.018 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:07:53.019 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:07:53.019 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:07:53.019 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:07:53.019 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:07:53.023 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:07:53.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:53.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:07:53.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:53.116 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:07:53.197 INFO:teuthology.orchestra.run.smithi060.stdout:client8620 2023-10-16T14:07:53.198 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:07:53.198 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:07:53.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:07:53.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.488+0000 7f7b7f402640 1 Processor -- start 2023-10-16T14:07:53.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.488+0000 7f7b7f402640 1 -- start start 2023-10-16T14:07:53.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7f402640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815aee0 0x7f7b780b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7f402640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b780cf730 0x7f7b780af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7f402640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7815b2c0 con 0x7f7b780ae0d0 2023-10-16T14:07:53.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7f402640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b780ae5d0 con 0x7f7b780cf730 2023-10-16T14:07:53.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7f402640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780ae8b0 con 0x7f7b7815aee0 2023-10-16T14:07:53.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7d99f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b780cf730 0x7f7b780af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7d99f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b780cf730 0x7f7b780af670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59996/0 (socket says 172.21.15.60:59996) 2023-10-16T14:07:53.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7d99f640 1 -- 172.21.15.60:0/4215955916 learned_addr learned my addr 172.21.15.60:0/4215955916 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.490+0000 7f7b7d19e640 1 --2- 172.21.15.60:0/4215955916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815aee0 0x7f7b780b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.491+0000 7f7b7d99f640 1 -- 172.21.15.60:0/4215955916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815aee0 msgr2=0x7f7b780b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.491+0000 7f7b7d99f640 1 --2- 172.21.15.60:0/4215955916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815aee0 0x7f7b780b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.491+0000 7f7b7d99f640 1 -- 172.21.15.60:0/4215955916 >> v1:172.21.15.144:6789/0 conn(0x7f7b780ae0d0 legacy=0x7f7b780cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.491+0000 7f7b7d99f640 1 -- 172.21.15.60:0/4215955916 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780aeb90 con 0x7f7b780cf730 2023-10-16T14:07:53.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.492+0000 7f7b7d99f640 1 --2- 172.21.15.60:0/4215955916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780af670 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009e50 tx=0x7f7b7002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bf540bd351caf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:53.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.492+0000 7f7b6ffff640 1 -- 172.21.15.60:0/4215955916 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7002cb90 con 0x7f7b780cf730 2023-10-16T14:07:53.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.492+0000 7f7b6ffff640 1 -- 172.21.15.60:0/4215955916 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7002ccf0 con 0x7f7b780cf730 2023-10-16T14:07:53.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.492+0000 7f7b6ffff640 1 -- 172.21.15.60:0/4215955916 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70034610 con 0x7f7b780cf730 2023-10-16T14:07:53.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 -- 172.21.15.60:0/4215955916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 msgr2=0x7f7b780af670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:53.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4215955916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780af670 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009e50 tx=0x7f7b7002bff0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 -- 172.21.15.60:0/4215955916 shutdown_connections 2023-10-16T14:07:53.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4215955916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780af670 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4215955916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815aee0 0x7f7b780b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 -- 172.21.15.60:0/4215955916 >> 172.21.15.60:0/4215955916 conn(0x7f7b780d0ad0 msgr2=0x7f7b780d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:53.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 -- 172.21.15.60:0/4215955916 shutdown_connections 2023-10-16T14:07:53.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.493+0000 7f7b7f402640 1 -- 172.21.15.60:0/4215955916 wait complete. 2023-10-16T14:07:53.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.495+0000 7f7b7f402640 1 Processor -- start 2023-10-16T14:07:53.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.495+0000 7f7b7f402640 1 -- start start 2023-10-16T14:07:53.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.496+0000 7f7b7f402640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b780ae0d0 0x7f7b781f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.496+0000 7f7b7f402640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.496+0000 7f7b7f402640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7815aee0 0x7f7b780b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.496+0000 7f7b7f402640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780c9240 con 0x7f7b780cf730 2023-10-16T14:07:53.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.496+0000 7f7b7f402640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780b0540 con 0x7f7b7815aee0 2023-10-16T14:07:53.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.496+0000 7f7b7f402640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b781f2890 con 0x7f7b780ae0d0 2023-10-16T14:07:53.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60004/0 (socket says 172.21.15.60:60004) 2023-10-16T14:07:53.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 -- 172.21.15.60:0/4203866535 learned_addr learned my addr 172.21.15.60:0/4203866535 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:53.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7d99f640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7815aee0 0x7f7b780b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7d19e640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b780ae0d0 0x7f7b781f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 -- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b780ae0d0 msgr2=0x7f7b781f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:53.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b780ae0d0 0x7f7b781f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 -- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7815aee0 msgr2=0x7f7b780b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7815aee0 0x7f7b780b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 -- 172.21.15.60:0/4203866535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70009b30 con 0x7f7b780cf730 2023-10-16T14:07:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7d19e640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b780ae0d0 0x7f7b781f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7d99f640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7815aee0 0x7f7b780b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.497+0000 7f7b7c99d640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780c8d00 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800c430 tx=0x7f7b6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:53.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.498+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680052d0 con 0x7f7b780cf730 2023-10-16T14:07:53.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.498+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68005430 con 0x7f7b780cf730 2023-10-16T14:07:53.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.498+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68003830 con 0x7f7b780cf730 2023-10-16T14:07:53.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.498+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b781f2b70 con 0x7f7b780cf730 2023-10-16T14:07:53.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.498+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b781f3130 con 0x7f7b780cf730 2023-10-16T14:07:53.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.499+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7804e160 con 0x7f7b780cf730 2023-10-16T14:07:53.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.500+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b68003990 con 0x7f7b780cf730 2023-10-16T14:07:53.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.501+0000 7f7b6dffb640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f7b5c06afe0 0x7f7b5c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.501+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b68088cf0 con 0x7f7b780cf730 2023-10-16T14:07:53.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.502+0000 7f7b7d19e640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f7b5c06afe0 0x7f7b5c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.503+0000 7f7b7d19e640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f7b5c06afe0 0x7f7b5c06d4a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b7815d720 tx=0x7f7b74008040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.505+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b68053d00 con 0x7f7b780cf730 2023-10-16T14:07:53.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.726+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 --> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b780af150 con 0x7f7b5c06afe0 2023-10-16T14:07:53.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.730+0000 7f7b6dffb640 1 -- 172.21.15.60:0/4203866535 <== mgr.8350 v2:172.21.15.144:6832/1274087552 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7b74009df0 con 0x7f7b5c06afe0 2023-10-16T14:07:53.732 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:07:53.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:07:53.733 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T14:07:53.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:07:53.733 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:07:53.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f7b5c06afe0 msgr2=0x7f7b5c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:53.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f7b5c06afe0 0x7f7b5c06d4a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b7815d720 tx=0x7f7b74008040 comp rx=0 tx=0).stop 2023-10-16T14:07:53.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 msgr2=0x7f7b780c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780c8d00 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800c430 tx=0x7f7b6800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 shutdown_connections 2023-10-16T14:07:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f7b5c06afe0 0x7f7b5c06d4a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7815aee0 0x7f7b780b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b780cf730 0x7f7b780c8d00 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 --2- 172.21.15.60:0/4203866535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b780ae0d0 0x7f7b781f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 >> 172.21.15.60:0/4203866535 conn(0x7f7b780d0ad0 msgr2=0x7f7b780cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:53.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.732+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 shutdown_connections 2023-10-16T14:07:53.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.733+0000 7f7b7f402640 1 -- 172.21.15.60:0/4203866535 wait complete. 2023-10-16T14:07:53.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:07:53.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.986+0000 7f480551d640 1 Processor -- start 2023-10-16T14:07:53.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.986+0000 7f480551d640 1 -- start start 2023-10-16T14:07:53.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f480551d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48000addb0 0x7f48000caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f480551d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f480006a4b0 0x7f48001650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f480551d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000cb140 con 0x7f48000c9000 2023-10-16T14:07:53.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f480551d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4800161a50 con 0x7f48000addb0 2023-10-16T14:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f480551d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480006b030 con 0x7f480006a4b0 2023-10-16T14:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f47feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48000addb0 0x7f48000caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f47feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48000addb0 0x7f48000caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60010/0 (socket says 172.21.15.60:60010) 2023-10-16T14:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.988+0000 7f47feffd640 1 -- 172.21.15.60:0/1467996845 learned_addr learned my addr 172.21.15.60:0/1467996845 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.989+0000 7f47feffd640 1 -- 172.21.15.60:0/1467996845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f480006a4b0 msgr2=0x7f48001650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:53.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.989+0000 7f47feffd640 1 --2- 172.21.15.60:0/1467996845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f480006a4b0 0x7f48001650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.989+0000 7f47feffd640 1 -- 172.21.15.60:0/1467996845 >> v1:172.21.15.144:6789/0 conn(0x7f48000c9000 legacy=0x7f4800069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:53.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.989+0000 7f47feffd640 1 -- 172.21.15.60:0/1467996845 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480006b310 con 0x7f48000addb0 2023-10-16T14:07:53.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.990+0000 7f47feffd640 1 --2- 172.21.15.60:0/1467996845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000caa80 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009870 tx=0x7f47f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70132349b0988af3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:53.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.990+0000 7f47fdffb640 1 -- 172.21.15.60:0/1467996845 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0031640 con 0x7f48000addb0 2023-10-16T14:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.990+0000 7f47fdffb640 1 -- 172.21.15.60:0/1467996845 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f00317a0 con 0x7f48000addb0 2023-10-16T14:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.990+0000 7f47fdffb640 1 -- 172.21.15.60:0/1467996845 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0031b30 con 0x7f48000addb0 2023-10-16T14:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 -- 172.21.15.60:0/1467996845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 msgr2=0x7f48000caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 --2- 172.21.15.60:0/1467996845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000caa80 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009870 tx=0x7f47f002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 -- 172.21.15.60:0/1467996845 shutdown_connections 2023-10-16T14:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 --2- 172.21.15.60:0/1467996845 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f480006a4b0 0x7f48001650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 --2- 172.21.15.60:0/1467996845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000caa80 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 -- 172.21.15.60:0/1467996845 >> 172.21.15.60:0/1467996845 conn(0x7f48000ce9b0 msgr2=0x7f48000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 -- 172.21.15.60:0/1467996845 shutdown_connections 2023-10-16T14:07:53.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.991+0000 7f480551d640 1 -- 172.21.15.60:0/1467996845 wait complete. 2023-10-16T14:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.993+0000 7f480551d640 1 Processor -- start 2023-10-16T14:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.993+0000 7f480551d640 1 -- start start 2023-10-16T14:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.994+0000 7f480551d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480006a4b0 0x7f48000c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.994+0000 7f480551d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.994+0000 7f480551d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48000c9000 0x7f4800155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.994+0000 7f480551d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000c3390 con 0x7f48000addb0 2023-10-16T14:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.994+0000 7f480551d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000c24c0 con 0x7f480006a4b0 2023-10-16T14:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.994+0000 7f480551d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000c2710 con 0x7f48000c9000 2023-10-16T14:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:54.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60018/0 (socket says 172.21.15.60:60018) 2023-10-16T14:07:54.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 -- 172.21.15.60:0/3679578850 learned_addr learned my addr 172.21.15.60:0/3679578850 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:54.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47feffd640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480006a4b0 0x7f48000c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:54.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47ff7fe640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48000c9000 0x7f4800155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:54.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 -- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48000c9000 msgr2=0x7f4800155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:54.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48000c9000 0x7f4800155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 -- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480006a4b0 msgr2=0x7f48000c37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:54.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480006a4b0 0x7f48000c37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 -- 172.21.15.60:0/3679578850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00094d0 con 0x7f48000addb0 2023-10-16T14:07:54.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.995+0000 7f47fe7fc640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000c8330 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f47f400b3e0 tx=0x7f47f400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.996+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4011830 con 0x7f48000addb0 2023-10-16T14:07:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.996+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f40119b0 con 0x7f48000addb0 2023-10-16T14:07:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.996+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4011d60 con 0x7f48000addb0 2023-10-16T14:07:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.996+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48000c29f0 con 0x7f48000addb0 2023-10-16T14:07:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.996+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48000c2fb0 con 0x7f48000addb0 2023-10-16T14:07:54.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.997+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48000c34d0 con 0x7f48000addb0 2023-10-16T14:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.998+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 115) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f47f401a400 con 0x7f48000addb0 2023-10-16T14:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.999+0000 7f47e77fe640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f47ec06afe0 0x7f47ec06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:53.999+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47f40898b0 con 0x7f48000addb0 2023-10-16T14:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.000+0000 7f47feffd640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f47ec06afe0 0x7f47ec06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.001+0000 7f47feffd640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f47ec06afe0 0x7f47ec06d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f47f0036000 tx=0x7f47f003c040 comp rx=0 tx=0).ready entity=mgr.8350 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:54.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.003+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48000c34d0 con 0x7f48000addb0 2023-10-16T14:07:54.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.248+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f48000c34d0 con 0x7f48000addb0 2023-10-16T14:07:54.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.893+0000 7f47e77fe640 1 -- 172.21.15.60:0/3679578850 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v116) v1 ==== 92+0+0 (secure 0 0 0) 0x7f48000c34d0 con 0x7f48000addb0 2023-10-16T14:07:54.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f47ec06afe0 msgr2=0x7f47ec06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f47ec06afe0 0x7f47ec06d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f47f0036000 tx=0x7f47f003c040 comp rx=0 tx=0).stop 2023-10-16T14:07:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 msgr2=0x7f48000c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000c8330 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f47f400b3e0 tx=0x7f47f400b8a0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 shutdown_connections 2023-10-16T14:07:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f48000c9000 0x7f4800155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f47ec06afe0 0x7f47ec06d4a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48000addb0 0x7f48000c8330 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 --2- 172.21.15.60:0/3679578850 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480006a4b0 0x7f48000c37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:54.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.897+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 >> 172.21.15.60:0/3679578850 conn(0x7f48000ce9b0 msgr2=0x7f48000cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:54.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.898+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 shutdown_connections 2023-10-16T14:07:54.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:54.898+0000 7f480551d640 1 -- 172.21.15.60:0/3679578850 wait complete. 2023-10-16T14:07:54.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:07:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.133+0000 7f2fecd48640 1 Processor -- start 2023-10-16T14:07:55.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.133+0000 7f2fecd48640 1 -- start start 2023-10-16T14:07:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fecd48640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fe815ab70 0x7f2fe8070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fecd48640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe80ccf80 0x7f2fe80ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fecd48640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe806a770 con 0x7f2fe806c660 2023-10-16T14:07:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fecd48640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80c6930 con 0x7f2fe815ab70 2023-10-16T14:07:55.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fecd48640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80af0a0 con 0x7f2fe80ccf80 2023-10-16T14:07:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fe659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fe815ab70 0x7f2fe8070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fe6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2fe806c660 0x7f2fe815b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34630/0 (socket says 172.21.15.60:34630) 2023-10-16T14:07:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.135+0000 7f2fe6d9d640 1 -- 172.21.15.60:0/2312296490 learned_addr learned my addr 172.21.15.60:0/2312296490 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.136+0000 7f2fe5d9b640 1 --2- 172.21.15.60:0/2312296490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe80ccf80 0x7f2fe80ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.136+0000 7f2fe659c640 1 -- 172.21.15.60:0/2312296490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe80ccf80 msgr2=0x7f2fe80ae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.136+0000 7f2fe659c640 1 --2- 172.21.15.60:0/2312296490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe80ccf80 0x7f2fe80ae9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.136+0000 7f2fe659c640 1 -- 172.21.15.60:0/2312296490 >> v1:172.21.15.60:6789/0 conn(0x7f2fe806c660 legacy=0x7f2fe815b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.136+0000 7f2fe659c640 1 -- 172.21.15.60:0/2312296490 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8069e40 con 0x7f2fe815ab70 2023-10-16T14:07:55.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.137+0000 7f2fe659c640 1 --2- 172.21.15.60:0/2312296490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe8070660 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80097a0 tx=0x7f2fd802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d55650dad87997 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:55.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.137+0000 7f2fe559a640 1 -- 172.21.15.60:0/2312296490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8030710 con 0x7f2fe815ab70 2023-10-16T14:07:55.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.137+0000 7f2fe559a640 1 -- 172.21.15.60:0/2312296490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd802e050 con 0x7f2fe815ab70 2023-10-16T14:07:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.137+0000 7f2fe559a640 1 -- 172.21.15.60:0/2312296490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8030a90 con 0x7f2fe815ab70 2023-10-16T14:07:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 -- 172.21.15.60:0/2312296490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 msgr2=0x7f2fe8070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 --2- 172.21.15.60:0/2312296490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe8070660 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80097a0 tx=0x7f2fd802b710 comp rx=0 tx=0).stop 2023-10-16T14:07:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 -- 172.21.15.60:0/2312296490 shutdown_connections 2023-10-16T14:07:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 --2- 172.21.15.60:0/2312296490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe80ccf80 0x7f2fe80ae9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 --2- 172.21.15.60:0/2312296490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe8070660 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 -- 172.21.15.60:0/2312296490 >> 172.21.15.60:0/2312296490 conn(0x7f2fe80cc460 msgr2=0x7f2fe80cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:55.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 -- 172.21.15.60:0/2312296490 shutdown_connections 2023-10-16T14:07:55.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.138+0000 7f2fecd48640 1 -- 172.21.15.60:0/2312296490 wait complete. 2023-10-16T14:07:55.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.140+0000 7f2fecd48640 1 Processor -- start 2023-10-16T14:07:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.140+0000 7f2fecd48640 1 -- start start 2023-10-16T14:07:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.141+0000 7f2fecd48640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe806c660 0x7f2fe815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.141+0000 7f2fecd48640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe80ccf80 0x7f2fe81f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.141+0000 7f2fecd48640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe80af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.141+0000 7f2fecd48640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80b3220 con 0x7f2fe815ab70 2023-10-16T14:07:55.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.141+0000 7f2fecd48640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81f2e10 con 0x7f2fe80ccf80 2023-10-16T14:07:55.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.141+0000 7f2fecd48640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81f30f0 con 0x7f2fe806c660 2023-10-16T14:07:55.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe80af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe80af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60062/0 (socket says 172.21.15.60:60062) 2023-10-16T14:07:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 -- 172.21.15.60:0/1724993080 learned_addr learned my addr 172.21.15.60:0/1724993080 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe659c640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe806c660 0x7f2fe815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe5d9b640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe80ccf80 0x7f2fe81f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 -- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe806c660 msgr2=0x7f2fe815a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe806c660 0x7f2fe815a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 -- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe80ccf80 msgr2=0x7f2fe81f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe80ccf80 0x7f2fe81f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 -- 172.21.15.60:0/1724993080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8009400 con 0x7f2fe815ab70 2023-10-16T14:07:55.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe5d9b640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe80ccf80 0x7f2fe81f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:55.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.142+0000 7f2fe6d9d640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe80af1e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000caf0 tx=0x7f2fe00020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:55.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.143+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000a760 con 0x7f2fe815ab70 2023-10-16T14:07:55.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.143+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe000a8e0 con 0x7f2fe815ab70 2023-10-16T14:07:55.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.143+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000acd0 con 0x7f2fe815ab70 2023-10-16T14:07:55.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.143+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe81f33d0 con 0x7f2fe815ab70 2023-10-16T14:07:55.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.143+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe80af720 con 0x7f2fe815ab70 2023-10-16T14:07:55.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.145+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe806dc40 con 0x7f2fe815ab70 2023-10-16T14:07:55.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.145+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 116) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2fe00070d0 con 0x7f2fe815ab70 2023-10-16T14:07:55.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.146+0000 7f2fd6ffd640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 0x7f2fc806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.146+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fe0088fc0 con 0x7f2fe815ab70 2023-10-16T14:07:55.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.146+0000 7f2fe659c640 1 -- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 msgr2=0x7f2fc806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:55.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.146+0000 7f2fe659c640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 0x7f2fc806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:55.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.150+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fe0053f50 con 0x7f2fe815ab70 2023-10-16T14:07:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.346+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2fe804e160 con 0x7f2fe815ab70 2023-10-16T14:07:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.347+0000 7f2fe659c640 1 -- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 msgr2=0x7f2fc806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.347+0000 7f2fe659c640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 0x7f2fc806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.350+0000 7f2fd6ffd640 1 -- 172.21.15.60:0/1724993080 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2fe0018070 con 0x7f2fe815ab70 2023-10-16T14:07:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 msgr2=0x7f2fc806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 0x7f2fc806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 msgr2=0x7f2fe80af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe80af1e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000caf0 tx=0x7f2fe00020f0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 shutdown_connections 2023-10-16T14:07:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f2fc806afe0 0x7f2fc806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fe815ab70 0x7f2fe80af1e0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe80ccf80 0x7f2fe81f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 --2- 172.21.15.60:0/1724993080 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fe806c660 0x7f2fe815a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.354+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 >> 172.21.15.60:0/1724993080 conn(0x7f2fe80cc460 msgr2=0x7f2fe806e530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.355+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 shutdown_connections 2023-10-16T14:07:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.355+0000 7f2fecd48640 1 -- 172.21.15.60:0/1724993080 wait complete. 2023-10-16T14:07:55.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:07:55.426 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:55.423+0000 7f96b5c3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:07:55.467 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:55.464+0000 7fbecd0c9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:07:55.479 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:55.476+0000 7f96b5c3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:07:55.525 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:55.522+0000 7fbecd0c9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:07:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.594+0000 7fa465f78640 1 Processor -- start 2023-10-16T14:07:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.594+0000 7fa465f78640 1 -- start start 2023-10-16T14:07:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa465f78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600cf710 0x7fa4600b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa465f78640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600d18c0 con 0x7fa4600b6cd0 2023-10-16T14:07:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa465f78640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600d3980 con 0x7fa4600cfaf0 2023-10-16T14:07:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa465f78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600d3c60 con 0x7fa4600cf710 2023-10-16T14:07:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa45ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa4600b6cd0 0x7fa4600d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34640/0 (socket says 172.21.15.60:34640) 2023-10-16T14:07:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa45ffff640 1 -- 172.21.15.60:0/2238293050 learned_addr learned my addr 172.21.15.60:0/2238293050 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:07:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.596+0000 7fa45f7fe640 1 --2- 172.21.15.60:0/2238293050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600cf710 0x7fa4600b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.597+0000 7fa45effd640 1 -- 172.21.15.60:0/2238293050 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581481132 0 0) 0x7fa4600d18c0 con 0x7fa4600b6cd0 2023-10-16T14:07:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.597+0000 7fa45effd640 1 -- 172.21.15.60:0/2238293050 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c003520 con 0x7fa4600b6cd0 2023-10-16T14:07:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.597+0000 7fa45f7fe640 1 -- 172.21.15.60:0/2238293050 >> v1:172.21.15.144:6789/0 conn(0x7fa4600cfaf0 legacy=0x7fa4600d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.597+0000 7fa45f7fe640 1 -- 172.21.15.60:0/2238293050 >> v1:172.21.15.60:6789/0 conn(0x7fa4600b6cd0 legacy=0x7fa4600d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.597+0000 7fa45f7fe640 1 -- 172.21.15.60:0/2238293050 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4600ce630 con 0x7fa4600cf710 2023-10-16T14:07:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.598+0000 7fa45f7fe640 1 --2- 172.21.15.60:0/2238293050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600cf710 0x7fa4600b2d10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa4500097c0 tx=0x7fa45002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20f09260207214d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa45effd640 1 -- 172.21.15.60:0/2238293050 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450031800 con 0x7fa4600cf710 2023-10-16T14:07:55.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa45effd640 1 -- 172.21.15.60:0/2238293050 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa450031960 con 0x7fa4600cf710 2023-10-16T14:07:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa45effd640 1 -- 172.21.15.60:0/2238293050 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450031cd0 con 0x7fa4600cf710 2023-10-16T14:07:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa465f78640 1 -- 172.21.15.60:0/2238293050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600cf710 msgr2=0x7fa4600b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa465f78640 1 --2- 172.21.15.60:0/2238293050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600cf710 0x7fa4600b2d10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa4500097c0 tx=0x7fa45002b8a0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa465f78640 1 -- 172.21.15.60:0/2238293050 shutdown_connections 2023-10-16T14:07:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa465f78640 1 --2- 172.21.15.60:0/2238293050 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600cf710 0x7fa4600b2d10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.599+0000 7fa465f78640 1 -- 172.21.15.60:0/2238293050 >> 172.21.15.60:0/2238293050 conn(0x7fa4600d2cc0 msgr2=0x7fa4600d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.600+0000 7fa465f78640 1 -- 172.21.15.60:0/2238293050 shutdown_connections 2023-10-16T14:07:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.600+0000 7fa465f78640 1 -- 172.21.15.60:0/2238293050 wait complete. 2023-10-16T14:07:55.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.601+0000 7fa465f78640 1 Processor -- start 2023-10-16T14:07:55.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.601+0000 7fa465f78640 1 -- start start 2023-10-16T14:07:55.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.602+0000 7fa465f78640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600b6cd0 0x7fa4601567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.602+0000 7fa465f78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600cf710 0x7fa4601530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.602+0000 7fa465f78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 0x7fa46006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa465f78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4601563b0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa465f78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460155a10 con 0x7fa4600cf710 2023-10-16T14:07:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa465f78640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa460155cf0 con 0x7fa4600b6cd0 2023-10-16T14:07:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 0x7fa46006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 0x7fa46006afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60064/0 (socket says 172.21.15.60:60064) 2023-10-16T14:07:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 -- 172.21.15.60:0/3604597882 learned_addr learned my addr 172.21.15.60:0/3604597882 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:55.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45f7fe640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600b6cd0 0x7fa4601567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa457fff640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600cf710 0x7fa4601530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:55.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 -- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600b6cd0 msgr2=0x7fa4601567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600b6cd0 0x7fa4601567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 -- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600cf710 msgr2=0x7fa4601530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600cf710 0x7fa4601530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.603+0000 7fa45ffff640 1 -- 172.21.15.60:0/3604597882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500093a0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.604+0000 7fa45ffff640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 0x7fa46006afc0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa45800cbb0 tx=0x7fa45800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:55.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.604+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45800ad40 con 0x7fa4600cfaf0 2023-10-16T14:07:55.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.604+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa458012400 con 0x7fa4600cfaf0 2023-10-16T14:07:55.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.604+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45800aea0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.604+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa460155fd0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.604+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa460153610 con 0x7fa4600cfaf0 2023-10-16T14:07:55.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.606+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4601564f0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.606+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 116) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fa458007070 con 0x7fa4600cfaf0 2023-10-16T14:07:55.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.608+0000 7fa45cff9640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 0x7fa44006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:55.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.608+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa458089770 con 0x7fa4600cfaf0 2023-10-16T14:07:55.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.608+0000 7fa45f7fe640 1 -- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 msgr2=0x7fa44006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:55.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.608+0000 7fa45f7fe640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 0x7fa44006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:55.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.612+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4601564f0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.808+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa4601564f0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.809+0000 7fa45f7fe640 1 -- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 msgr2=0x7fa44006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:55.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.809+0000 7fa45f7fe640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 0x7fa44006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:55.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.813+0000 7fa45cff9640 1 -- 172.21.15.60:0/3604597882 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa4601564f0 con 0x7fa4600cfaf0 2023-10-16T14:07:55.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 msgr2=0x7fa44006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:55.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 0x7fa44006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 msgr2=0x7fa46006afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:55.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 0x7fa46006afc0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa45800cbb0 tx=0x7fa45800a430 comp rx=0 tx=0).stop 2023-10-16T14:07:55.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 shutdown_connections 2023-10-16T14:07:55.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7fa44006afe0 0x7fa44006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4600cfaf0 0x7fa46006afc0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600cf710 0x7fa4601530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 --2- 172.21.15.60:0/3604597882 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa4600b6cd0 0x7fa4601567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:55.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 >> 172.21.15.60:0/3604597882 conn(0x7fa4600d2cc0 msgr2=0x7fa4600c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:55.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 shutdown_connections 2023-10-16T14:07:55.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:55.816+0000 7fa465f78640 1 -- 172.21.15.60:0/3604597882 wait complete. 2023-10-16T14:07:55.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:07:55.837 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:07:55.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T14:07:55.932 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:55.929+0000 7f96b5c3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:07:55.984 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:55.981+0000 7f96b5c3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:07:56.008 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.006+0000 7fbecd0c9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:07:56.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.031+0000 7f96b5c3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:07:56.064 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.061+0000 7fbecd0c9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:07:56.094 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.092+0000 7f96b5c3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:07:56.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.105+0000 7f0be7fff640 1 Processor -- start 2023-10-16T14:07:56.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.106+0000 7f0be7fff640 1 -- start start 2023-10-16T14:07:56.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.107+0000 7f0be7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0be80b8480 0x7f0be80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:56.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.107+0000 7f0be7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:56.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.107+0000 7f0be7fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be815d150 con 0x7f0be815cd70 2023-10-16T14:07:56.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.107+0000 7f0be7fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be80d1180 con 0x7f0be80b8480 2023-10-16T14:07:56.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.107+0000 7f0be7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80d1460 con 0x7f0be80d0be0 2023-10-16T14:07:56.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.108+0000 7f0be77fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:56.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.108+0000 7f0be67fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0be80b8480 0x7f0be80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:56.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.108+0000 7f0be77fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:47000/0 (socket says 172.21.15.60:47000) 2023-10-16T14:07:56.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.108+0000 7f0be67fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0be80b8480 0x7f0be80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:51934/0 (socket says 172.21.15.60:51934) 2023-10-16T14:07:56.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.108+0000 7f0be77fe640 1 -- 172.21.15.60:0/3578211956 learned_addr learned my addr 172.21.15.60:0/3578211956 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:56.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.109+0000 7f0be67fc640 1 -- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 msgr2=0x7f0be80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:07:56.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.109+0000 7f0be67fc640 1 --2- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:56.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.109+0000 7f0be67fc640 1 -- 172.21.15.60:0/3578211956 >> v1:172.21.15.144:6789/0 conn(0x7f0be815cd70 legacy=0x7f0be80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:56.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.109+0000 7f0be67fc640 1 -- 172.21.15.60:0/3578211956 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80d1740 con 0x7f0be80b8480 2023-10-16T14:07:56.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.111+0000 7f0be67fc640 1 --2- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be80b8480 0x7f0be80cd7c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80097a0 tx=0x7f0bd802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a36c200f11aba628 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:56.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.111+0000 7f0be5ffb640 1 -- 172.21.15.60:0/3578211956 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd802d040 con 0x7f0be80b8480 2023-10-16T14:07:56.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.111+0000 7f0be5ffb640 1 -- 172.21.15.60:0/3578211956 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd8031810 con 0x7f0be80b8480 2023-10-16T14:07:56.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.111+0000 7f0be5ffb640 1 -- 172.21.15.60:0/3578211956 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8031ba0 con 0x7f0be80b8480 2023-10-16T14:07:56.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 -- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be80b8480 msgr2=0x7f0be80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:56.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 --2- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be80b8480 0x7f0be80cd7c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80097a0 tx=0x7f0bd802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:07:56.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 -- 172.21.15.60:0/3578211956 shutdown_connections 2023-10-16T14:07:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 --2- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 --2- 172.21.15.60:0/3578211956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be80b8480 0x7f0be80cd7c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 -- 172.21.15.60:0/3578211956 >> 172.21.15.60:0/3578211956 conn(0x7f0be80cf570 msgr2=0x7f0be80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:07:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 -- 172.21.15.60:0/3578211956 shutdown_connections 2023-10-16T14:07:56.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.112+0000 7f0be7fff640 1 -- 172.21.15.60:0/3578211956 wait complete. 2023-10-16T14:07:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.114+0000 7f0be7fff640 1 Processor -- start 2023-10-16T14:07:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.114+0000 7f0be7fff640 1 -- start start 2023-10-16T14:07:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be80b8480 0x7f0be81f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be815cd70 0x7f0be80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be7fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be81f3030 con 0x7f0be80b8480 2023-10-16T14:07:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be7fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be81f23f0 con 0x7f0be815cd70 2023-10-16T14:07:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be81f2690 con 0x7f0be80d0be0 2023-10-16T14:07:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.115+0000 7f0be6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be80b8480 0x7f0be81f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be80b8480 0x7f0be81f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60066/0 (socket says 172.21.15.60:60066) 2023-10-16T14:07:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 -- 172.21.15.60:0/1826773037 learned_addr learned my addr 172.21.15.60:0/1826773037 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:07:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be77fe640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be815cd70 0x7f0be80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:07:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 msgr2=0x7f0be80cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be815cd70 msgr2=0x7f0be80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:07:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be815cd70 0x7f0be80cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 -- 172.21.15.60:0/1826773037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8009400 con 0x7f0be80b8480 2023-10-16T14:07:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be77fe640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0be815cd70 0x7f0be80cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0be80d0be0 0x7f0be80cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:07:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.116+0000 7f0be6ffd640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be80b8480 0x7f0be81f1af0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c450 tx=0x7f0bd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:07:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.117+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00052d0 con 0x7f0be80b8480 2023-10-16T14:07:56.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.117+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd0005430 con 0x7f0be80b8480 2023-10-16T14:07:56.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.117+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0012870 con 0x7f0be80b8480 2023-10-16T14:07:56.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.117+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be81f2970 con 0x7f0be80b8480 2023-10-16T14:07:56.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.117+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be81f2030 con 0x7f0be80b8480 2023-10-16T14:07:56.126 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.117+0000 7fbecd0c9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:07:56.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.118+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be81f21c0 con 0x7f0be80b8480 2023-10-16T14:07:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.120+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 116) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0bd00129d0 con 0x7f0be80b8480 2023-10-16T14:07:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.121+0000 7f0bdf7fe640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 0x7f0bc806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:07:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.121+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bd0089380 con 0x7f0be80b8480 2023-10-16T14:07:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.121+0000 7f0be67fc640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 msgr2=0x7f0bc806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.121+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 0x7f0bc806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:07:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.126+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bd00542a0 con 0x7f0be80b8480 2023-10-16T14:07:56.144 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.142+0000 7f96b5c3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:07:56.186 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.183+0000 7fbecd0c9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:07:56.238 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.236+0000 7fbecd0c9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:07:56.254 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.251+0000 7f96b5c3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:07:56.316 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.313+0000 7f96b5c3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.322+0000 7f0be67fc640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 msgr2=0x7f0bc806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.322+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 0x7f0bc806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:07:56.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.341+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 --> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0be80d1d00 con 0x7f0bc806b010 2023-10-16T14:07:56.356 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.354+0000 7fbecd0c9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:07:56.366 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.362+0000 7f96b5c3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:07:56.422 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.419+0000 7f96b5c3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:07:56.422 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.420+0000 7fbecd0c9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:56.471 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.469+0000 7f96b5c3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:07:56.474 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.471+0000 7fbecd0c9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:07:56.534 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.531+0000 7fbecd0c9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:07:56.587 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.584+0000 7fbecd0c9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:07:56.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:07:56.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:07:56.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.723+0000 7f0be67fc640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 msgr2=0x7f0bc806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:56.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:56.723+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 0x7f0bc806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:07:56.858 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:56.855+0000 7f96b5c3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:07:56.993 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:56.990+0000 7fbecd0c9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:07:57.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.015+0000 7f96b5c3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:07:57.127 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.124+0000 7f96b5c3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:07:57.162 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.158+0000 7fbecd0c9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:07:57.176 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.174+0000 7f96b5c3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:07:57.275 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.271+0000 7fbecd0c9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:07:57.283 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.280+0000 7f96b5c3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:57.327 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.323+0000 7fbecd0c9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:07:57.389 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.387+0000 7f96b5c3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:07:57.440 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.437+0000 7fbecd0c9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:57.523+0000 7f0be67fc640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 msgr2=0x7f0bc806d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1274087552 2023-10-16T14:07:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:57.523+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 0x7f0bc806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:07:57.545 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.542+0000 7f96b5c3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:57.553 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.549+0000 7fbecd0c9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:07:57.594 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.591+0000 7f96b5c3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:57.715 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:57.711+0000 7f96b5c3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:07:57.717 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.714+0000 7fbecd0c9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:07:57.769 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.765+0000 7fbecd0c9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:07:57.895 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:57.891+0000 7fbecd0c9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:07:58.532 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:58.529+0000 7f96b5c3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:07:58.782 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:58.778+0000 7fbecd0c9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:07:58.825 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:58.822+0000 7f96b5c3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:07:58.881 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:58.879+0000 7f96b5c3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:07:58.994 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:58.992+0000 7f96b5c3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:07:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:59.013+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 117) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f0bd00878a0 con 0x7f0be80b8480 2023-10-16T14:07:59.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:59.014+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 msgr2=0x7f0bc806d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:07:59.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:07:59.014+0000 7f0bdf7fe640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/1274087552,v1:172.21.15.144:6833/1274087552] conn(0x7f0bc806b010 0x7f0bc806d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:07:59.105 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:59.101+0000 7fbecd0c9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:07:59.177 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:59.173+0000 7fbecd0c9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:07:59.321 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:07:59.318+0000 7fbecd0c9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:07:59.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f963576e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f963576e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.340 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f963576e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.340 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f963576e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.340 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f963576e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f9632728640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.341 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f9632728640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f9632728640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f9632728640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:07:59.342 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:07:59.335+0000 7f9632728640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:08:00.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.021+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 118) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f0bd0054670 con 0x7f0be80b8480 2023-10-16T14:08:00.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.023+0000 7f0bdf7fe640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f0bc80358b0 0x7f0bc8074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.023+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 --> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bc8035c90 con 0x7f0bc80358b0 2023-10-16T14:08:00.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.032+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f0bc80358b0 0x7f0bc8074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.036+0000 7f0be67fc640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f0bc80358b0 0x7f0bc8074ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8036000 tx=0x7f0bd8008030 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:00.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.233+0000 7f0bdf7fe640 1 -- 172.21.15.60:0/1826773037 <== mgr.7577 v2:172.21.15.144:6832/166679164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0bc8035c90 con 0x7f0bc80358b0 2023-10-16T14:08:00.238 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1 2023-10-16T14:08:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.236+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f0bc80358b0 msgr2=0x7f0bc8074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.236+0000 7f0be7fff640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f0bc80358b0 0x7f0bc8074ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8036000 tx=0x7f0bd8008030 comp rx=0 tx=0).stop 2023-10-16T14:08:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be80b8480 msgr2=0x7f0be81f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be7fff640 1 --2- 172.21.15.60:0/1826773037 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be80b8480 0x7f0be81f1af0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c450 tx=0x7f0bd000c910 comp rx=0 tx=0).stop 2023-10-16T14:08:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be77fe640 1 -- 172.21.15.60:0/1826773037 reap_dead start 2023-10-16T14:08:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 shutdown_connections 2023-10-16T14:08:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 >> 172.21.15.60:0/1826773037 conn(0x7f0be80cf570 msgr2=0x7f0be80cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:08:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 shutdown_connections 2023-10-16T14:08:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.237+0000 7f0be7fff640 1 -- 172.21.15.60:0/1826773037 wait complete. 2023-10-16T14:08:00.263 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T14:08:00.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T14:08:00.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.492+0000 7f101da61640 1 Processor -- start 2023-10-16T14:08:00.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.492+0000 7f101da61640 1 -- start start 2023-10-16T14:08:00.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f101da61640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f101815ac10 0x7f101806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f101da61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f101da61640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101815b310 con 0x7f10180b0360 2023-10-16T14:08:00.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f101da61640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101815f800 con 0x7f101815ac10 2023-10-16T14:08:00.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f101da61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10180b0d30 con 0x7f10180cac20 2023-10-16T14:08:00.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f1016ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f101815ac10 0x7f101806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f1016ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f101815ac10 0x7f101806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59156/0 (socket says 172.21.15.60:59156) 2023-10-16T14:08:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.494+0000 7f1016ffd640 1 -- 172.21.15.60:0/350406002 learned_addr learned my addr 172.21.15.60:0/350406002 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:08:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.495+0000 7f10177fe640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.495+0000 7f1016ffd640 1 -- 172.21.15.60:0/350406002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 msgr2=0x7f10180b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:08:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.495+0000 7f1016ffd640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.495+0000 7f1016ffd640 1 -- 172.21.15.60:0/350406002 >> v1:172.21.15.144:6789/0 conn(0x7f10180b0360 legacy=0x7f10180ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:08:00.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.495+0000 7f1016ffd640 1 -- 172.21.15.60:0/350406002 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101806f640 con 0x7f101815ac10 2023-10-16T14:08:00.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.495+0000 7f10177fe640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:08:00.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.496+0000 7f1016ffd640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f101815ac10 0x7f101806c730 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f10080098a0 tx=0x7f100802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d359c7580bceca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:00.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.496+0000 7f1015ffb640 1 -- 172.21.15.60:0/350406002 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080317c0 con 0x7f101815ac10 2023-10-16T14:08:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.496+0000 7f1015ffb640 1 -- 172.21.15.60:0/350406002 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1008031920 con 0x7f101815ac10 2023-10-16T14:08:00.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.496+0000 7f1015ffb640 1 -- 172.21.15.60:0/350406002 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008031cb0 con 0x7f101815ac10 2023-10-16T14:08:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 -- 172.21.15.60:0/350406002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f101815ac10 msgr2=0x7f101806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f101815ac10 0x7f101806c730 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f10080098a0 tx=0x7f100802b860 comp rx=0 tx=0).stop 2023-10-16T14:08:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 -- 172.21.15.60:0/350406002 shutdown_connections 2023-10-16T14:08:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 --2- 172.21.15.60:0/350406002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f101815ac10 0x7f101806c730 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 -- 172.21.15.60:0/350406002 >> 172.21.15.60:0/350406002 conn(0x7f10180cb360 msgr2=0x7f10180cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:08:00.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 -- 172.21.15.60:0/350406002 shutdown_connections 2023-10-16T14:08:00.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.497+0000 7f101da61640 1 -- 172.21.15.60:0/350406002 wait complete. 2023-10-16T14:08:00.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.499+0000 7f101da61640 1 Processor -- start 2023-10-16T14:08:00.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.499+0000 7f101da61640 1 -- start start 2023-10-16T14:08:00.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f101da61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 0x7f1018154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f101da61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f101da61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f101815ac10 0x7f10180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f101da61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10180b34e0 con 0x7f10180b0360 2023-10-16T14:08:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f101da61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10180c6430 con 0x7f101815ac10 2023-10-16T14:08:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f101da61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10180b2960 con 0x7f10180cac20 2023-10-16T14:08:00.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f1016ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 0x7f1018154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f1016ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 0x7f1018154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59162/0 (socket says 172.21.15.60:59162) 2023-10-16T14:08:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f1016ffd640 1 -- 172.21.15.60:0/1853875193 learned_addr learned my addr 172.21.15.60:0/1853875193 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:08:00.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f10177fe640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f101815ac10 0x7f10180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.500+0000 7f10167fc640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f1016ffd640 1 -- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 msgr2=0x7f10180b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f1016ffd640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f1016ffd640 1 -- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f101815ac10 msgr2=0x7f10180ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f1016ffd640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f101815ac10 0x7f10180ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f1016ffd640 1 -- 172.21.15.60:0/1853875193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080094d0 con 0x7f10180b0360 2023-10-16T14:08:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f1016ffd640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 0x7f1018154060 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f10080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080329a0 con 0x7f10180b0360 2023-10-16T14:08:00.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1008032b20 con 0x7f10180b0360 2023-10-16T14:08:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.501+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080453f0 con 0x7f10180b0360 2023-10-16T14:08:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.502+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10180b2c40 con 0x7f10180b0360 2023-10-16T14:08:00.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.502+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10180b3200 con 0x7f10180b0360 2023-10-16T14:08:00.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.503+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101804e160 con 0x7f10180b0360 2023-10-16T14:08:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.504+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 118) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f100804e450 con 0x7f10180b0360 2023-10-16T14:08:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.505+0000 7f0ffffff640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f100406ad10 0x7f100406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:00.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.505+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f10080ab860 con 0x7f10180b0360 2023-10-16T14:08:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.505+0000 7f10167fc640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f100406ad10 0x7f100406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.506+0000 7f10167fc640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f100406ad10 0x7f100406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f100c006fd0 tx=0x7f100c008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:00.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.510+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1008076b40 con 0x7f10180b0360 2023-10-16T14:08:00.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.734+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 --> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f101815b890 con 0x7f100406ad10 2023-10-16T14:08:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.915+0000 7f0ffffff640 1 -- 172.21.15.60:0/1853875193 <== mgr.7577 v2:172.21.15.144:6832/166679164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f100c00be70 con 0x7f100406ad10 2023-10-16T14:08:00.919 INFO:teuthology.orchestra.run.smithi060.stdout:Retention added to path snap_test_dir1 2023-10-16T14:08:00.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f100406ad10 msgr2=0x7f100406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f100406ad10 0x7f100406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f100c006fd0 tx=0x7f100c008040 comp rx=0 tx=0).stop 2023-10-16T14:08:00.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 msgr2=0x7f1018154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:00.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 0x7f1018154060 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f10080046c0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 shutdown_connections 2023-10-16T14:08:00.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f100406ad10 0x7f100406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f10180cac20 0x7f10180b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f101815ac10 0x7f10180ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 --2- 172.21.15.60:0/1853875193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10180b0360 0x7f1018154060 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:00.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 >> 172.21.15.60:0/1853875193 conn(0x7f10180cb360 msgr2=0x7f10180cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:08:00.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 shutdown_connections 2023-10-16T14:08:00.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:00.919+0000 7f101da61640 1 -- 172.21.15.60:0/1853875193 wait complete. 2023-10-16T14:08:00.946 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T14:08:00.946 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-14_09_00 in ~60s... 2023-10-16T14:08:00.946 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T14:08:00.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T14:08:01.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.177+0000 7fdd31c32640 1 Processor -- start 2023-10-16T14:08:01.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.177+0000 7fdd31c32640 1 -- start start 2023-10-16T14:08:01.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd31c32640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd2c06c040 0x7fdd2c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:01.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd31c32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c0cda80 0x7fdd2c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:01.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd31c32640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c06c960 con 0x7fdd2c0ad970 2023-10-16T14:08:01.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd31c32640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c15b9d0 con 0x7fdd2c06c040 2023-10-16T14:08:01.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd31c32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c15bcb0 con 0x7fdd2c0cda80 2023-10-16T14:08:01.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd30c30640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd2c06c040 0x7fdd2c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:01.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd30c30640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd2c06c040 0x7fdd2c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59164/0 (socket says 172.21.15.60:59164) 2023-10-16T14:08:01.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd30c30640 1 -- 172.21.15.60:0/1095533626 learned_addr learned my addr 172.21.15.60:0/1095533626 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:08:01.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd31431640 1 --2- 172.21.15.60:0/1095533626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c0cda80 0x7fdd2c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:01.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd30c30640 1 -- 172.21.15.60:0/1095533626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c0cda80 msgr2=0x7fdd2c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:08:01.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd30c30640 1 --2- 172.21.15.60:0/1095533626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c0cda80 0x7fdd2c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.179+0000 7fdd30c30640 1 -- 172.21.15.60:0/1095533626 >> v1:172.21.15.144:6789/0 conn(0x7fdd2c0ad970 legacy=0x7fdd2c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:01.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.180+0000 7fdd30c30640 1 -- 172.21.15.60:0/1095533626 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c15bf90 con 0x7fdd2c06c040 2023-10-16T14:08:01.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.181+0000 7fdd30c30640 1 --2- 172.21.15.60:0/1095533626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c06c040 0x7fdd2c06c420 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009870 tx=0x7fdd2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70150838294ac366 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:01.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.181+0000 7fdd2b7fe640 1 -- 172.21.15.60:0/1095533626 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20030840 con 0x7fdd2c06c040 2023-10-16T14:08:01.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.181+0000 7fdd2b7fe640 1 -- 172.21.15.60:0/1095533626 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd200309a0 con 0x7fdd2c06c040 2023-10-16T14:08:01.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.181+0000 7fdd2b7fe640 1 -- 172.21.15.60:0/1095533626 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20030d30 con 0x7fdd2c06c040 2023-10-16T14:08:01.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 -- 172.21.15.60:0/1095533626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c06c040 msgr2=0x7fdd2c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:01.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1095533626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c06c040 0x7fdd2c06c420 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009870 tx=0x7fdd2002b980 comp rx=0 tx=0).stop 2023-10-16T14:08:01.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 -- 172.21.15.60:0/1095533626 shutdown_connections 2023-10-16T14:08:01.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1095533626 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c0cda80 0x7fdd2c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1095533626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c06c040 0x7fdd2c06c420 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 -- 172.21.15.60:0/1095533626 >> 172.21.15.60:0/1095533626 conn(0x7fdd2c0ce6a0 msgr2=0x7fdd2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:08:01.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 -- 172.21.15.60:0/1095533626 shutdown_connections 2023-10-16T14:08:01.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.182+0000 7fdd31c32640 1 -- 172.21.15.60:0/1095533626 wait complete. 2023-10-16T14:08:01.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.184+0000 7fdd31c32640 1 Processor -- start 2023-10-16T14:08:01.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.184+0000 7fdd31c32640 1 -- start start 2023-10-16T14:08:01.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31c32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c06c040 0x7fdd2c0c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:01.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31c32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 0x7fdd2c0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:01.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31c32640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd2c0cda80 0x7fdd2c1532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:01.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31c32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b2cb0 con 0x7fdd2c0ad970 2023-10-16T14:08:01.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31c32640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b2f00 con 0x7fdd2c0cda80 2023-10-16T14:08:01.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31c32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0b31e0 con 0x7fdd2c06c040 2023-10-16T14:08:01.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 0x7fdd2c0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:01.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 0x7fdd2c0acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59172/0 (socket says 172.21.15.60:59172) 2023-10-16T14:08:01.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd2bfff640 1 -- 172.21.15.60:0/1966216916 learned_addr learned my addr 172.21.15.60:0/1966216916 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:08:01.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd30c30640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c06c040 0x7fdd2c0c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:01.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.185+0000 7fdd31431640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd2c0cda80 0x7fdd2c1532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:01.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd2bfff640 1 -- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c06c040 msgr2=0x7fdd2c0c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:01.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd2bfff640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c06c040 0x7fdd2c0c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd2bfff640 1 -- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd2c0cda80 msgr2=0x7fdd2c1532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:01.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd2bfff640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd2c0cda80 0x7fdd2c1532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd2bfff640 1 -- 172.21.15.60:0/1966216916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200094d0 con 0x7fdd2c0ad970 2023-10-16T14:08:01.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd2bfff640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 0x7fdd2c0acb00 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800c3c0 tx=0x7fdd1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:01.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd180052d0 con 0x7fdd2c0ad970 2023-10-16T14:08:01.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18005450 con 0x7fdd2c0ad970 2023-10-16T14:08:01.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18003830 con 0x7fdd2c0ad970 2023-10-16T14:08:01.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.186+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c0b34c0 con 0x7fdd2c0ad970 2023-10-16T14:08:01.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.187+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c0c6d90 con 0x7fdd2c0ad970 2023-10-16T14:08:01.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.188+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c06b330 con 0x7fdd2c0ad970 2023-10-16T14:08:01.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.189+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 119) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fdd18003990 con 0x7fdd2c0ad970 2023-10-16T14:08:01.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.190+0000 7fdd297fa640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fdd1006afe0 0x7fdd1006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:08:01.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.190+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fdd18088e80 con 0x7fdd2c0ad970 2023-10-16T14:08:01.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.190+0000 7fdd30c30640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fdd1006afe0 0x7fdd1006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:08:01.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.191+0000 7fdd30c30640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fdd1006afe0 0x7fdd1006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdd20036000 tx=0x7fdd20008030 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:08:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.194+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd18053cd0 con 0x7fdd2c0ad970 2023-10-16T14:08:01.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.413+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 --> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd2c0b3070 con 0x7fdd1006afe0 2023-10-16T14:08:01.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.429+0000 7fdd297fa640 1 -- 172.21.15.60:0/1966216916 <== mgr.7577 v2:172.21.15.144:6832/166679164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fdd2c0b3070 con 0x7fdd1006afe0 2023-10-16T14:08:01.433 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:08:01.433 INFO:teuthology.orchestra.run.smithi060.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T14:08:01.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fdd1006afe0 msgr2=0x7fdd1006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fdd1006afe0 0x7fdd1006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdd20036000 tx=0x7fdd20008030 comp rx=0 tx=0).stop 2023-10-16T14:08:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 msgr2=0x7fdd2c0acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:08:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 0x7fdd2c0acb00 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800c3c0 tx=0x7fdd1800c880 comp rx=0 tx=0).stop 2023-10-16T14:08:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 shutdown_connections 2023-10-16T14:08:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fdd1006afe0 0x7fdd1006d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd2c0cda80 0x7fdd2c1532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd2c0ad970 0x7fdd2c0acb00 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 --2- 172.21.15.60:0/1966216916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdd2c06c040 0x7fdd2c0c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:08:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 >> 172.21.15.60:0/1966216916 conn(0x7fdd2c0ce6a0 msgr2=0x7fdd2c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:08:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 shutdown_connections 2023-10-16T14:08:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:08:01.432+0000 7fdd31c32640 1 -- 172.21.15.60:0/1966216916 wait complete. 2023-10-16T14:08:01.455 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T14:08:01.456 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T14:08:01.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:01.457 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:01.517 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:01.517 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:01.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:02.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:02.520 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:02.586 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:02.587 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:02.587 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:03.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:03.589 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:03.656 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:03.657 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:03.657 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:04.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:04.659 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:04.729 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:04.730 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:04.730 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:05.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:05.732 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:05.799 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:05.800 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:05.800 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:06.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:06.801 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:06.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:06.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:06.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:07.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:07.871 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:07.939 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:07.939 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:07.940 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:08.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:08.941 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:09.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:09.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:09.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:10.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:10.013 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:10.082 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:10.083 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:10.083 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:11.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:11.085 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:11.153 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:11.154 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:11.154 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:12.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:12.156 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:12.223 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:12.223 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:12.223 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:13.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:13.225 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:13.292 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:13.293 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:13.293 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:14.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:14.295 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:14.363 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:14.363 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:14.363 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:15.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:15.366 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:15.434 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:15.434 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:15.435 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:16.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:16.437 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:16.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:16.508 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:16.508 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:17.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:17.509 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:17.577 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:17.577 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:17.578 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:18.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:18.579 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:18.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:18.646 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:18.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:19.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:19.682 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:19.749 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:19.750 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:19.750 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:20.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:20.752 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:20.819 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:20.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:20.819 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:21.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:21.822 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:21.889 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:21.889 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:21.890 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:22.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:22.892 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:22.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:22.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:22.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:23.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:23.962 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:24.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:24.031 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:24.031 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:25.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:25.032 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:25.100 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:25.100 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:25.101 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:26.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:26.103 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:26.171 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:26.171 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:26.172 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:27.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:08:27.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:08:27.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:27.174 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:27.240 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:27.240 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:27.241 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:28.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:28.242 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:28.312 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:28.312 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:28.312 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:29.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:29.314 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:29.381 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:29.381 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:29.382 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:30.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:30.383 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:30.450 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:30.451 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:30.451 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:31.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:31.453 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:31.518 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:31.519 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:31.519 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:32.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:32.520 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:32.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:32.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:32.590 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:33.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:33.592 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:33.657 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:33.658 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:33.658 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:34.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:34.660 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:34.730 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:34.730 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:34.730 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:35.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:35.732 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:35.800 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:35.800 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:35.801 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:36.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:36.803 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:36.870 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:36.871 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:36.871 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:37.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:37.872 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:37.939 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:37.940 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:37.940 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:38.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:38.942 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:39.010 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:39.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:39.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:40.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:40.012 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:40.076 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:40.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:40.077 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:41.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:41.079 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:41.146 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:41.147 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:41.147 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:42.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:42.149 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:42.219 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:42.219 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:42.220 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:43.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:43.222 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:43.290 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:43.290 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:43.290 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:44.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:44.292 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:44.359 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:44.360 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:44.360 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:45.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:45.362 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:45.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:45.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:45.432 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:46.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:46.433 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:46.496 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:46.497 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:46.497 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:47.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:47.499 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:47.567 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:47.567 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:47.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:48.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:48.569 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:48.638 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:48.638 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:48.639 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:49.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:49.641 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:49.706 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:49.707 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:49.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:50.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:50.709 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:50.777 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:50.778 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:50.778 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:51.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:51.780 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:51.848 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:51.849 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:51.849 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:52.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:52.850 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:52.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:52.918 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:52.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:53.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:53.920 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:53.988 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:53.989 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:53.989 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:54.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:54.990 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:55.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:55.058 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:55.059 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:56.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:56.060 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:56.128 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:56.128 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:56.128 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:57.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:57.130 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:57.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:08:57.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:08:57.201 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:57.202 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:57.202 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:58.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:58.204 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:58.271 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:58.272 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:58.272 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:08:59.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:08:59.274 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:08:59.343 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:08:59.343 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:08:59.344 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:00.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:00.346 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:00.413 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T14:09:00.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:00.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:00.950 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:09:00.947+0000 7f962470c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:09:00.950 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:09:00.947+0000 7f962470c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:09:00.950 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:09:00.947+0000 7f962470c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:09:00.951 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:09:00.947+0000 7f962470c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:09:00.951 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:09:00.947+0000 7f962470c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:09:01.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:01.416 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:01.477 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:01.479 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:01.479 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:01.479 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:01.479 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-14_09_00_UTC'] 2023-10-16T14:09:01.480 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697465280.9466076, now = 1697465341.4803078, timo = 60 2023-10-16T14:09:02.481 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-16-14_09_00 in ~120s... 2023-10-16T14:09:02.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:02.482 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:02.546 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:02.548 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:02.548 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:02.548 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:03.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:03.551 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:03.615 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:03.617 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:03.617 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:03.617 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:04.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:04.619 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:04.686 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:04.688 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:04.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:04.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:05.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:05.690 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:05.755 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:05.756 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:05.757 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:05.757 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:06.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:06.759 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:06.825 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:06.826 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:06.827 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:06.827 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:07.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:07.829 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:07.895 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:07.897 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:07.897 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:07.897 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:08.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:08.899 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:08.965 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:08.966 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:08.967 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:08.967 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:09.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:09.969 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:10.035 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:10.037 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:10.038 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:10.038 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:11.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:11.040 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:11.105 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:11.106 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:11.107 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:11.107 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:12.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:12.108 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:12.174 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:12.175 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:12.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:12.176 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:13.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:13.178 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:13.243 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:13.244 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:13.244 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:13.245 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:14.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:14.246 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:14.313 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:14.314 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:14.315 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:14.315 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:15.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:15.316 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:15.383 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:15.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:15.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:15.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:16.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:16.387 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:16.455 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:16.457 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:16.457 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:16.458 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:17.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:17.459 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:17.524 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:17.525 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:17.526 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:17.526 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:18.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:18.528 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:18.593 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:18.594 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:18.594 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:18.595 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:19.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:19.597 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:19.665 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:19.666 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:19.667 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:19.667 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:20.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:20.669 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:20.736 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:20.737 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:20.738 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:20.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:21.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:21.740 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:21.803 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:21.805 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:21.805 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:21.805 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:22.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:22.807 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:22.873 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:22.875 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:22.875 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:22.875 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:23.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:23.877 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:23.941 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:23.943 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:23.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:23.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:24.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:24.945 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:25.010 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:25.012 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:25.012 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:25.012 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:26.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:26.014 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:26.082 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:26.083 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:26.083 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:26.084 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:27.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:27.086 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:27.152 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:27.153 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:27.153 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:27.154 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:27.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:09:27.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:09:28.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:28.156 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:28.221 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:28.222 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:28.223 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:28.223 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:29.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:29.225 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:29.292 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:29.293 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:29.293 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:29.293 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:30.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:30.295 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:30.363 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:30.365 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:30.365 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:30.365 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:31.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:31.367 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:31.431 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:31.432 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:31.433 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:31.433 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:32.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:32.434 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:32.501 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:32.502 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:32.503 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:32.503 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:33.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:33.505 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:33.571 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:33.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:33.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:33.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:34.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:34.575 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:34.640 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:34.641 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:34.643 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:34.643 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:35.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:35.644 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:35.709 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:35.710 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:35.711 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:35.711 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:36.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:36.713 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:36.779 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:36.780 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:36.781 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:36.781 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:37.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:37.783 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:37.850 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:37.851 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:37.851 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:37.852 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:38.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:38.853 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:38.916 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:38.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:38.918 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:38.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:39.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:39.920 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:39.987 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:39.987 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:39.988 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:40.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:40.990 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:41.055 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:41.056 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:41.056 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:41.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:42.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:42.058 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:42.124 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:42.125 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:42.126 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:42.126 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:43.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:43.128 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:43.193 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:43.194 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:43.194 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:43.195 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:44.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:44.197 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:44.265 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:44.267 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:44.267 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:44.267 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:45.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:45.269 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:45.335 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:45.336 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:45.337 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:45.337 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:46.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:46.338 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:46.405 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:46.406 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:46.407 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:46.407 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:47.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:47.409 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:47.472 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:47.474 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:47.474 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:47.474 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:48.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:48.477 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:48.543 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:48.545 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:48.545 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:48.546 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:49.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:49.547 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:49.614 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:49.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:49.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:49.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:50.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:50.618 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:50.685 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:50.686 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:50.686 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:50.687 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:51.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:51.690 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:51.755 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:51.756 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:51.757 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:51.757 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:52.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:52.760 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:52.829 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:52.830 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:52.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:52.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:53.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:53.832 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:53.899 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:53.901 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:53.901 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:53.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:54.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:54.903 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:54.966 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:54.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:54.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:54.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:55.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:55.969 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:56.036 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:56.038 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:56.038 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:56.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:57.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:57.041 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:57.105 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:57.106 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:57.107 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:57.107 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:57.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:09:57.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:09:58.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:58.109 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:58.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:58.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:58.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:09:59.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:09:59.179 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:09:59.246 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:09:59.247 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:09:59.247 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:09:59.248 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:10:00.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:00.250 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:10:00.311 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_09_00_UTC 2023-10-16T14:10:00.311 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_10_00_UTC 2023-10-16T14:10:00.312 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_10_00_UTC', 'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:10:00.313 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-14_10_00_UTC'} 2023-10-16T14:10:00.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T14:10:01.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:01.315 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:10:01.381 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_10_00_UTC 2023-10-16T14:10:01.382 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-14_10_00_UTC'} 2023-10-16T14:10:01.383 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T14:10:01.383 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-16-14_09_00_UTC'} 2023-10-16T14:10:01.383 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-16-14_09_00_UTC'] 2023-10-16T14:10:01.384 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697465280.9466076, now = 1697465401.3839874, timo = 120 2023-10-16T14:10:02.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T14:10:02.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.622+0000 7fe49bf5a640 1 Processor -- start 2023-10-16T14:10:02.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.622+0000 7fe49bf5a640 1 -- start start 2023-10-16T14:10:02.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe49bf5a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe494159ab0 0x7fe4940ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:02.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe49bf5a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe49406c140 0x7fe49406c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:02.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe49bf5a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49406cad0 con 0x7fe494159e90 2023-10-16T14:10:02.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe49bf5a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49415d890 con 0x7fe49406c140 2023-10-16T14:10:02.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe49bf5a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe494070360 con 0x7fe494159ab0 2023-10-16T14:10:02.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe4994f5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe494159e90 0x7fe49406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49564/0 (socket says 172.21.15.60:49564) 2023-10-16T14:10:02.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe4994f5640 1 -- 172.21.15.60:0/2122336265 learned_addr learned my addr 172.21.15.60:0/2122336265 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:02.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.624+0000 7fe49a4f7640 1 --2- 172.21.15.60:0/2122336265 >> v2:172.21.15.60:3300/0 conn(0x7fe49406c140 0x7fe49406c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:02.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.625+0000 7fe499cf6640 1 --2- 172.21.15.60:0/2122336265 >> v2:172.21.15.144:3300/0 conn(0x7fe494159ab0 0x7fe4940ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:02.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.625+0000 7fe49a4f7640 1 -- 172.21.15.60:0/2122336265 >> v2:172.21.15.144:3300/0 conn(0x7fe494159ab0 msgr2=0x7fe4940ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:02.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.625+0000 7fe49a4f7640 1 --2- 172.21.15.60:0/2122336265 >> v2:172.21.15.144:3300/0 conn(0x7fe494159ab0 0x7fe4940ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.625+0000 7fe49a4f7640 1 -- 172.21.15.60:0/2122336265 >> v1:172.21.15.60:6789/0 conn(0x7fe494159e90 legacy=0x7fe49406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.625+0000 7fe49a4f7640 1 -- 172.21.15.60:0/2122336265 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4940cd1f0 con 0x7fe49406c140 2023-10-16T14:10:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.626+0000 7fe49a4f7640 1 --2- 172.21.15.60:0/2122336265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe49406c140 0x7fe49406c590 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fe4940cfa00 tx=0x7fe48c02dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936b1cbfecaef242 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.626+0000 7fe498cf4640 1 -- 172.21.15.60:0/2122336265 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c02ed70 con 0x7fe49406c140 2023-10-16T14:10:02.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.626+0000 7fe498cf4640 1 -- 172.21.15.60:0/2122336265 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c036440 con 0x7fe49406c140 2023-10-16T14:10:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe498cf4640 1 -- 172.21.15.60:0/2122336265 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0367d0 con 0x7fe49406c140 2023-10-16T14:10:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 -- 172.21.15.60:0/2122336265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe49406c140 msgr2=0x7fe49406c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/2122336265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe49406c140 0x7fe49406c590 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fe4940cfa00 tx=0x7fe48c02dfc0 comp rx=0 tx=0).stop 2023-10-16T14:10:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 -- 172.21.15.60:0/2122336265 shutdown_connections 2023-10-16T14:10:02.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/2122336265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe49406c140 0x7fe49406c590 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:02.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/2122336265 >> v2:172.21.15.144:3300/0 conn(0x7fe494159ab0 0x7fe4940ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:02.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 -- 172.21.15.60:0/2122336265 >> 172.21.15.60:0/2122336265 conn(0x7fe4940ce9b0 msgr2=0x7fe4940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:02.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 -- 172.21.15.60:0/2122336265 shutdown_connections 2023-10-16T14:10:02.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.627+0000 7fe49bf5a640 1 -- 172.21.15.60:0/2122336265 wait complete. 2023-10-16T14:10:02.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.629+0000 7fe49bf5a640 1 Processor -- start 2023-10-16T14:10:02.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.629+0000 7fe49bf5a640 1 -- start start 2023-10-16T14:10:02.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.630+0000 7fe49bf5a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe49406c140 0x7fe49414e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:02.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.630+0000 7fe49bf5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe494159ab0 0x7fe4941edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:02.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.630+0000 7fe49bf5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 0x7fe4940af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:02.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.630+0000 7fe49bf5a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4940c1a20 con 0x7fe494159e90 2023-10-16T14:10:02.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.630+0000 7fe49bf5a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49415d670 con 0x7fe49406c140 2023-10-16T14:10:02.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.630+0000 7fe49bf5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4941511f0 con 0x7fe494159ab0 2023-10-16T14:10:02.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 0x7fe4940af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 0x7fe4940af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55384/0 (socket says 172.21.15.60:55384) 2023-10-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 -- 172.21.15.60:0/654918453 learned_addr learned my addr 172.21.15.60:0/654918453 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe499cf6640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe49406c140 0x7fe49414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:02.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe4994f5640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe494159ab0 0x7fe4941edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 -- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe494159ab0 msgr2=0x7fe4941edeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe494159ab0 0x7fe4941edeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 -- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe49406c140 msgr2=0x7fe49414e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe49406c140 0x7fe49414e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:02.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 -- 172.21.15.60:0/654918453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c009a70 con 0x7fe494159e90 2023-10-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe4994f5640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe494159ab0 0x7fe4941edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.631+0000 7fe49a4f7640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 0x7fe4940af120 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fe48c02e5d0 tx=0x7fe48c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.632+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c042450 con 0x7fe494159e90 2023-10-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.632+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c0425d0 con 0x7fe494159e90 2023-10-16T14:10:02.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.632+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c042980 con 0x7fe494159e90 2023-10-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.632+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4941514d0 con 0x7fe494159e90 2023-10-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.632+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe494151a90 con 0x7fe494159e90 2023-10-16T14:10:02.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.633+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49404e160 con 0x7fe494159e90 2023-10-16T14:10:02.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.634+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 122) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fe48c042ae0 con 0x7fe494159e90 2023-10-16T14:10:02.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.635+0000 7fe48a7fc640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fe47806afe0 0x7fe47806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:02.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.635+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fe48c0ae910 con 0x7fe494159e90 2023-10-16T14:10:02.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.636+0000 7fe499cf6640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fe47806afe0 0x7fe47806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:02.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.637+0000 7fe499cf6640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fe47806afe0 0x7fe47806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe490006fd0 tx=0x7fe490008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:02.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.640+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe48c0795f0 con 0x7fe494159e90 2023-10-16T14:10:02.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:02.858+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 --> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4940cd7b0 con 0x7fe47806afe0 2023-10-16T14:10:03.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.049+0000 7fe48a7fc640 1 -- 172.21.15.60:0/654918453 <== mgr.7577 v2:172.21.15.144:6832/166679164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe49000b200 con 0x7fe47806afe0 2023-10-16T14:10:03.053 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path snap_test_dir1 2023-10-16T14:10:03.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fe47806afe0 msgr2=0x7fe47806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fe47806afe0 0x7fe47806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe490006fd0 tx=0x7fe490008040 comp rx=0 tx=0).stop 2023-10-16T14:10:03.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 msgr2=0x7fe4940af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 0x7fe4940af120 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fe48c02e5d0 tx=0x7fe48c004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:03.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 shutdown_connections 2023-10-16T14:10:03.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fe47806afe0 0x7fe47806d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe494159ab0 0x7fe4941edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe494159e90 0x7fe4940af120 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 --2- 172.21.15.60:0/654918453 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe49406c140 0x7fe49414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 >> 172.21.15.60:0/654918453 conn(0x7fe4940ce9b0 msgr2=0x7fe4941580c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:03.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 shutdown_connections 2023-10-16T14:10:03.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.053+0000 7fe49bf5a640 1 -- 172.21.15.60:0/654918453 wait complete. 2023-10-16T14:10:03.078 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T14:10:03.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:03.079 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:10:03.134 INFO:teuthology.orchestra.run.smithi060.stdout:scheduled-2023-10-16-14_10_00_UTC 2023-10-16T14:10:03.135 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-14_10_00_UTC 2023-10-16T14:10:03.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:03.136 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-14_10_00_UTC) 2023-10-16T14:10:03.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:03.204 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T14:10:03.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:10:03.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.486+0000 7fa5b5c75640 1 Processor -- start 2023-10-16T14:10:03.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.486+0000 7fa5b5c75640 1 -- start start 2023-10-16T14:10:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.488+0000 7fa5b5c75640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5b0159950 0x7fa5b015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.488+0000 7fa5b5c75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 0x7fa5b00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.488+0000 7fa5b5c75640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b0070df0 con 0x7fa5b006c100 2023-10-16T14:10:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.488+0000 7fa5b5c75640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b015e490 con 0x7fa5b0159950 2023-10-16T14:10:03.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.488+0000 7fa5b5c75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b015aef0 con 0x7fa5b0070450 2023-10-16T14:10:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.488+0000 7fa5a7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 0x7fa5b00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5a7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 0x7fa5b00708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:54430/0 (socket says 172.21.15.60:54430) 2023-10-16T14:10:03.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5a7fff640 1 -- 172.21.15.60:0/123495487 learned_addr learned my addr 172.21.15.60:0/123495487 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5b4c73640 1 --2- 172.21.15.60:0/123495487 >> v2:172.21.15.144:3300/0 conn(0x7fa5b0159950 0x7fa5b015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5a7fff640 1 -- 172.21.15.60:0/123495487 >> v2:172.21.15.144:3300/0 conn(0x7fa5b0159950 msgr2=0x7fa5b015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5a7fff640 1 --2- 172.21.15.60:0/123495487 >> v2:172.21.15.144:3300/0 conn(0x7fa5b0159950 0x7fa5b015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5a7fff640 1 -- 172.21.15.60:0/123495487 >> v1:172.21.15.144:6789/0 conn(0x7fa5b006c100 legacy=0x7fa5b006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.489+0000 7fa5a7fff640 1 -- 172.21.15.60:0/123495487 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b006cc80 con 0x7fa5b0070450 2023-10-16T14:10:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.491+0000 7fa5a7fff640 1 --2- 172.21.15.60:0/123495487 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 0x7fa5b00708b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009870 tx=0x7fa5a002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f533a7decfd9711d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:03.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.491+0000 7fa5a77fe640 1 -- 172.21.15.60:0/123495487 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0031640 con 0x7fa5b0070450 2023-10-16T14:10:03.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.491+0000 7fa5a77fe640 1 -- 172.21.15.60:0/123495487 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a00317a0 con 0x7fa5b0070450 2023-10-16T14:10:03.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.491+0000 7fa5a77fe640 1 -- 172.21.15.60:0/123495487 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0031b30 con 0x7fa5b0070450 2023-10-16T14:10:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.492+0000 7fa5b5c75640 1 -- 172.21.15.60:0/123495487 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 msgr2=0x7fa5b00708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.492+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/123495487 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 0x7fa5b00708b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009870 tx=0x7fa5a002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.492+0000 7fa5b5c75640 1 -- 172.21.15.60:0/123495487 shutdown_connections 2023-10-16T14:10:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.492+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/123495487 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b0070450 0x7fa5b00708b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.492+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/123495487 >> v2:172.21.15.144:3300/0 conn(0x7fa5b0159950 0x7fa5b015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.492+0000 7fa5b5c75640 1 -- 172.21.15.60:0/123495487 >> 172.21.15.60:0/123495487 conn(0x7fa5b00ce6a0 msgr2=0x7fa5b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.493+0000 7fa5b5c75640 1 -- 172.21.15.60:0/123495487 shutdown_connections 2023-10-16T14:10:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.493+0000 7fa5b5c75640 1 -- 172.21.15.60:0/123495487 wait complete. 2023-10-16T14:10:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.495+0000 7fa5b5c75640 1 Processor -- start 2023-10-16T14:10:03.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.495+0000 7fa5b5c75640 1 -- start start 2023-10-16T14:10:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.496+0000 7fa5b5c75640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b006c100 0x7fa5b0148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.496+0000 7fa5b5c75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 0x7fa5b00b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:03.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.496+0000 7fa5b5c75640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b0159950 0x7fa5b006b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.496+0000 7fa5b5c75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b0149090 con 0x7fa5b0070450 2023-10-16T14:10:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.496+0000 7fa5b5c75640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00b2600 con 0x7fa5b0159950 2023-10-16T14:10:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.496+0000 7fa5b5c75640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00b28e0 con 0x7fa5b006c100 2023-10-16T14:10:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 0x7fa5b00b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 0x7fa5b00b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55390/0 (socket says 172.21.15.60:55390) 2023-10-16T14:10:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 -- 172.21.15.60:0/1477928989 learned_addr learned my addr 172.21.15.60:0/1477928989 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5b4c73640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b006c100 0x7fa5b0148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 -- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b006c100 msgr2=0x7fa5b0148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b006c100 0x7fa5b0148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 -- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b0159950 msgr2=0x7fa5b006b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5b5474640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b0159950 0x7fa5b006b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b0159950 0x7fa5b006b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5a7fff640 1 -- 172.21.15.60:0/1477928989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00094d0 con 0x7fa5b0070450 2023-10-16T14:10:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.497+0000 7fa5b5474640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b0159950 0x7fa5b006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.498+0000 7fa5a7fff640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 0x7fa5b00b1cd0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.498+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a003a930 con 0x7fa5b0070450 2023-10-16T14:10:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.498+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a003aa90 con 0x7fa5b0070450 2023-10-16T14:10:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.498+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a003ae40 con 0x7fa5b0070450 2023-10-16T14:10:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.498+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b00b2bc0 con 0x7fa5b0070450 2023-10-16T14:10:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.498+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b00b3180 con 0x7fa5b0070450 2023-10-16T14:10:03.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.500+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b004e9b0 con 0x7fa5b0070450 2023-10-16T14:10:03.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.501+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 122) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fa5a002f050 con 0x7fa5b0070450 2023-10-16T14:10:03.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.502+0000 7fa5a57fa640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fa59c06afe0 0x7fa59c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:03.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.502+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fa5a00ab990 con 0x7fa5b0070450 2023-10-16T14:10:03.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.502+0000 7fa5b4c73640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fa59c06afe0 0x7fa59c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:03.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.503+0000 7fa5b4c73640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fa59c06afe0 0x7fa59c06d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0074a0 tx=0x7fa5ac00b040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:03.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.508+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5a00765f0 con 0x7fa5b0070450 2023-10-16T14:10:03.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.757+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fa5b00c8610 con 0x7fa5b0070450 2023-10-16T14:10:03.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.965+0000 7fa5a57fa640 1 -- 172.21.15.60:0/1477928989 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v123) v1 ==== 93+0+0 (secure 0 0 0) 0x7fa5a00350a0 con 0x7fa5b0070450 2023-10-16T14:10:03.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.970+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fa59c06afe0 msgr2=0x7fa59c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.970+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fa59c06afe0 0x7fa59c06d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0074a0 tx=0x7fa5ac00b040 comp rx=0 tx=0).stop 2023-10-16T14:10:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.970+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 msgr2=0x7fa5b00b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.970+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 0x7fa5b00b1cd0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a00046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.971+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 shutdown_connections 2023-10-16T14:10:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.971+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fa59c06afe0 0x7fa59c06d4a0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0074a0 tx=0x7fa5ac00b040 comp rx=0 tx=0).stop 2023-10-16T14:10:03.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.971+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b0159950 0x7fa5b006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.971+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5b0070450 0x7fa5b00b1cd0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.971+0000 7fa5b5c75640 1 --2- 172.21.15.60:0/1477928989 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5b006c100 0x7fa5b0148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:03.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.971+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 >> 172.21.15.60:0/1477928989 conn(0x7fa5b00ce6a0 msgr2=0x7fa5b006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:03.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.972+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 shutdown_connections 2023-10-16T14:10:03.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:03.973+0000 7fa5b5c75640 1 -- 172.21.15.60:0/1477928989 wait complete. 2023-10-16T14:10:04.007 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:10:04.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:10:04.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:10:04.215 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:10:04.215 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:10:04.216 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:04.269 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:10:04.270 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:04.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:04.518 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:04.516+0000 7f6cc8630040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:10:04.535 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:04.531+0000 7fe0c1c2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:10:04.572 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:04.569+0000 7f6cc8630040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:10:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.583+0000 7f7724741640 1 Processor -- start 2023-10-16T14:10:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.583+0000 7f7724741640 1 -- start start 2023-10-16T14:10:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f7724741640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 0x7f771c06ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f7724741640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f771c15d430 0x7f771c070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f7724741640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c070660 con 0x7f771c071360 2023-10-16T14:10:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f7724741640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c15dc60 con 0x7f771c15d430 2023-10-16T14:10:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f7724741640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c15df40 con 0x7f771c15a970 2023-10-16T14:10:04.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f77224dd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 0x7f771c06ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f77224dd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 0x7f771c06ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:54462/0 (socket says 172.21.15.60:54462) 2023-10-16T14:10:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.585+0000 7f77224dd640 1 -- 172.21.15.60:0/3673402960 learned_addr learned my addr 172.21.15.60:0/3673402960 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.586+0000 7f7721cdc640 1 --2- 172.21.15.60:0/3673402960 >> v2:172.21.15.144:3300/0 conn(0x7f771c15d430 0x7f771c070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.586+0000 7f77224dd640 1 -- 172.21.15.60:0/3673402960 >> v2:172.21.15.144:3300/0 conn(0x7f771c15d430 msgr2=0x7f771c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.586+0000 7f77224dd640 1 --2- 172.21.15.60:0/3673402960 >> v2:172.21.15.144:3300/0 conn(0x7f771c15d430 0x7f771c070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.586+0000 7f77224dd640 1 -- 172.21.15.60:0/3673402960 >> v1:172.21.15.144:6789/0 conn(0x7f771c071360 legacy=0x7f771c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.586+0000 7f77224dd640 1 -- 172.21.15.60:0/3673402960 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c15e220 con 0x7f771c15a970 2023-10-16T14:10:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.588+0000 7f77224dd640 1 --2- 172.21.15.60:0/3673402960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 0x7f771c06ecd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7718009870 tx=0x7f771802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d669be7018d1c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.588+0000 7f77214db640 1 -- 172.21.15.60:0/3673402960 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718030840 con 0x7f771c15a970 2023-10-16T14:10:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.588+0000 7f77214db640 1 -- 172.21.15.60:0/3673402960 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77180309a0 con 0x7f771c15a970 2023-10-16T14:10:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.588+0000 7f77214db640 1 -- 172.21.15.60:0/3673402960 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718030d30 con 0x7f771c15a970 2023-10-16T14:10:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 -- 172.21.15.60:0/3673402960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 msgr2=0x7f771c06ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 --2- 172.21.15.60:0/3673402960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 0x7f771c06ecd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7718009870 tx=0x7f771802b980 comp rx=0 tx=0).stop 2023-10-16T14:10:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 -- 172.21.15.60:0/3673402960 shutdown_connections 2023-10-16T14:10:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 --2- 172.21.15.60:0/3673402960 >> v2:172.21.15.144:3300/0 conn(0x7f771c15d430 0x7f771c070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 --2- 172.21.15.60:0/3673402960 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15a970 0x7f771c06ecd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 -- 172.21.15.60:0/3673402960 >> 172.21.15.60:0/3673402960 conn(0x7f771c0cfaa0 msgr2=0x7f771c0cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 -- 172.21.15.60:0/3673402960 shutdown_connections 2023-10-16T14:10:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.589+0000 7f7724741640 1 -- 172.21.15.60:0/3673402960 wait complete. 2023-10-16T14:10:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.591+0000 7f7724741640 1 Processor -- start 2023-10-16T14:10:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.591+0000 7f7724741640 1 -- start start 2023-10-16T14:10:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7724741640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c15a970 0x7f771c0b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7724741640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15d430 0x7f771c153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7724741640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 0x7f771c0aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7724741640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0b3b60 con 0x7f771c1f5ad0 2023-10-16T14:10:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7724741640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0c6ca0 con 0x7f771c15a970 2023-10-16T14:10:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7724741640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c154070 con 0x7f771c15d430 2023-10-16T14:10:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 0x7f771c0aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 0x7f771c0aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55418/0 (socket says 172.21.15.60:55418) 2023-10-16T14:10:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 -- 172.21.15.60:0/86856144 learned_addr learned my addr 172.21.15.60:0/86856144 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7721cdc640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15d430 0x7f771c153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:04.600 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:04.592+0000 7fe0c1c2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:10:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f77224dd640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c15a970 0x7f771c0b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 -- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15d430 msgr2=0x7f771c153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15d430 0x7f771c153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 -- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c15a970 msgr2=0x7f771c0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c15a970 0x7f771c0b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.592+0000 7f7722cde640 1 -- 172.21.15.60:0/86856144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180094d0 con 0x7f771c1f5ad0 2023-10-16T14:10:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.593+0000 7f7722cde640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 0x7f771c0aec10 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f771400c680 tx=0x7f771400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.593+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714003b70 con 0x7f771c1f5ad0 2023-10-16T14:10:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.593+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7714003cd0 con 0x7f771c1f5ad0 2023-10-16T14:10:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.593+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714005310 con 0x7f771c1f5ad0 2023-10-16T14:10:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.593+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1f6090 con 0x7f771c1f5ad0 2023-10-16T14:10:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.593+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1f6650 con 0x7f771c1f5ad0 2023-10-16T14:10:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.594+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c0c6e50 con 0x7f771c1f5ad0 2023-10-16T14:10:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.596+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7714005470 con 0x7f771c1f5ad0 2023-10-16T14:10:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.596+0000 7f7712ffd640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 0x7f770006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.597+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f7714089760 con 0x7f771c1f5ad0 2023-10-16T14:10:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.597+0000 7f77224dd640 1 -- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 msgr2=0x7f770006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.597+0000 7f77224dd640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 0x7f770006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.601+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f771c0c6e50 con 0x7f771c1f5ad0 2023-10-16T14:10:04.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.795+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771c0c6e50 con 0x7f771c1f5ad0 2023-10-16T14:10:04.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.798+0000 7f77224dd640 1 -- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 msgr2=0x7f770006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:04.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.798+0000 7f77224dd640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 0x7f770006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:04.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.798+0000 7f7712ffd640 1 -- 172.21.15.60:0/86856144 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+17115 (secure 0 0 0) 0x7f771c0c6e50 con 0x7f771c1f5ad0 2023-10-16T14:10:04.803 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:04.803 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":216,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:09:00.934343+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":1}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/351085041":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/992081872":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/4042889662":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/166629007":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6832/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2940276091":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6833/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2969783877":"2023-10-17T14:07:58.997006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 msgr2=0x7f770006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:04.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 0x7f770006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 msgr2=0x7f771c0aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:04.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 0x7f771c0aec10 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f771400c680 tx=0x7f771400cb40 comp rx=0 tx=0).stop 2023-10-16T14:10:04.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 shutdown_connections 2023-10-16T14:10:04.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f770006af90 0x7f770006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f771c15d430 0x7f771c153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f771c1f5ad0 0x7f771c0aec10 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 --2- 172.21.15.60:0/86856144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f771c15a970 0x7f771c0b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:04.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 >> 172.21.15.60:0/86856144 conn(0x7f771c0cfaa0 msgr2=0x7f771c0c9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:04.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 shutdown_connections 2023-10-16T14:10:04.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:04.801+0000 7f7724741640 1 -- 172.21.15.60:0/86856144 wait complete. 2023-10-16T14:10:04.824 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:07:42.198213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:07:43.287022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:10:04.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:10:05.024 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.021+0000 7f6cc8630040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:10:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.050+0000 7fb97ce3f640 1 Processor -- start 2023-10-16T14:10:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.050+0000 7fb97ce3f640 1 -- start start 2023-10-16T14:10:05.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb97ce3f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb97815ad70 0x7fb978069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb97ce3f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9780caac0 0x7fb97806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb97ce3f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9780cb030 con 0x7fb9780c9ca0 2023-10-16T14:10:05.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb97ce3f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9780c91c0 con 0x7fb9780caac0 2023-10-16T14:10:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb97ce3f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9780c94a0 con 0x7fb97815ad70 2023-10-16T14:10:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb976d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9780caac0 0x7fb97806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb976d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9780caac0 0x7fb97806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55422/0 (socket says 172.21.15.60:55422) 2023-10-16T14:10:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.051+0000 7fb976d9d640 1 -- 172.21.15.60:0/3236400118 learned_addr learned my addr 172.21.15.60:0/3236400118 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:05.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.052+0000 7fb97659c640 1 --2- 172.21.15.60:0/3236400118 >> v2:172.21.15.144:3300/0 conn(0x7fb97815ad70 0x7fb978069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.052+0000 7fb976d9d640 1 -- 172.21.15.60:0/3236400118 >> v2:172.21.15.144:3300/0 conn(0x7fb97815ad70 msgr2=0x7fb978069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:05.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.052+0000 7fb976d9d640 1 --2- 172.21.15.60:0/3236400118 >> v2:172.21.15.144:3300/0 conn(0x7fb97815ad70 0x7fb978069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.052+0000 7fb976d9d640 1 -- 172.21.15.60:0/3236400118 >> v1:172.21.15.144:6789/0 conn(0x7fb9780c9ca0 legacy=0x7fb9780ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.052+0000 7fb976d9d640 1 -- 172.21.15.60:0/3236400118 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9780c9780 con 0x7fb9780caac0 2023-10-16T14:10:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.053+0000 7fb976d9d640 1 --2- 172.21.15.60:0/3236400118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97806a2e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fb9780cf780 tx=0x7fb97002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff21cc930b7874fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.053+0000 7fb975d9b640 1 -- 172.21.15.60:0/3236400118 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97002ca80 con 0x7fb9780caac0 2023-10-16T14:10:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb975d9b640 1 -- 172.21.15.60:0/3236400118 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97002cbe0 con 0x7fb9780caac0 2023-10-16T14:10:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb975d9b640 1 -- 172.21.15.60:0/3236400118 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700343f0 con 0x7fb9780caac0 2023-10-16T14:10:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3236400118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 msgr2=0x7fb97806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3236400118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97806a2e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fb9780cf780 tx=0x7fb97002bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3236400118 shutdown_connections 2023-10-16T14:10:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3236400118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97806a2e0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3236400118 >> v2:172.21.15.144:3300/0 conn(0x7fb97815ad70 0x7fb978069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3236400118 >> 172.21.15.60:0/3236400118 conn(0x7fb9780ce9b0 msgr2=0x7fb9780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3236400118 shutdown_connections 2023-10-16T14:10:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.054+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3236400118 wait complete. 2023-10-16T14:10:05.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.056+0000 7fb97ce3f640 1 Processor -- start 2023-10-16T14:10:05.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.056+0000 7fb97ce3f640 1 -- start start 2023-10-16T14:10:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb97ce3f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9780c9ca0 0x7fb978157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb97ce3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb97ce3f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb97815ad70 0x7fb97815b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb97ce3f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9781575a0 con 0x7fb9780caac0 2023-10-16T14:10:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb97ce3f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9781f0400 con 0x7fb97815ad70 2023-10-16T14:10:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb97ce3f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9780c79c0 con 0x7fb9780c9ca0 2023-10-16T14:10:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.057+0000 7fb96ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55432/0 (socket says 172.21.15.60:55432) 2023-10-16T14:10:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 -- 172.21.15.60:0/3391636300 learned_addr learned my addr 172.21.15.60:0/3391636300 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb976d9d640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb97815ad70 0x7fb97815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb97659c640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9780c9ca0 0x7fb978157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 -- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9780c9ca0 msgr2=0x7fb978157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9780c9ca0 0x7fb978157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 -- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb97815ad70 msgr2=0x7fb97815b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb97815ad70 0x7fb97815b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb96ffff640 1 -- 172.21.15.60:0/3391636300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970009b30 con 0x7fb9780caac0 2023-10-16T14:10:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.058+0000 7fb976d9d640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb97815ad70 0x7fb97815b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:10:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.059+0000 7fb96ffff640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97815b450 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fb96000c460 tx=0x7fb96000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.059+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960012830 con 0x7fb9780caac0 2023-10-16T14:10:05.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.059+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9600129d0 con 0x7fb9780caac0 2023-10-16T14:10:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.059+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960012d80 con 0x7fb9780caac0 2023-10-16T14:10:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.059+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9780c7ca0 con 0x7fb9780caac0 2023-10-16T14:10:05.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.059+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9780c8260 con 0x7fb9780caac0 2023-10-16T14:10:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.060+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97804e160 con 0x7fb9780caac0 2023-10-16T14:10:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.061+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb96001b400 con 0x7fb9780caac0 2023-10-16T14:10:05.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.063+0000 7fb96effd640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 0x7fb95806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.063+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fb960089b90 con 0x7fb9780caac0 2023-10-16T14:10:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.063+0000 7fb97659c640 1 -- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 msgr2=0x7fb95806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.063+0000 7fb97659c640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 0x7fb95806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.067+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb960054930 con 0x7fb9780caac0 2023-10-16T14:10:05.078 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.074+0000 7f6cc8630040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:10:05.081 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.077+0000 7fe0c1c2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:10:05.127 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.124+0000 7f6cc8630040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:10:05.136 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.132+0000 7fe0c1c2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:10:05.187 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.184+0000 7f6cc8630040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:10:05.189 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.185+0000 7fe0c1c2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:10:05.235 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.232+0000 7f6cc8630040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:10:05.252 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.248+0000 7fe0c1c2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:10:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.264+0000 7fb97659c640 1 -- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 msgr2=0x7fb95806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.264+0000 7fb97659c640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 0x7fb95806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:05.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.266+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb97806ce40 con 0x7fb9780caac0 2023-10-16T14:10:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.267+0000 7fb96effd640 1 -- 172.21.15.60:0/3391636300 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb960057f50 con 0x7fb9780caac0 2023-10-16T14:10:05.271 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:10:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.269+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 msgr2=0x7fb95806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.269+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 0x7fb95806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 msgr2=0x7fb97815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97815b450 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fb96000c460 tx=0x7fb96000c920 comp rx=0 tx=0).stop 2023-10-16T14:10:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 shutdown_connections 2023-10-16T14:10:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fb95806af90 0x7fb95806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb97815ad70 0x7fb97815b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9780caac0 0x7fb97815b450 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 --2- 172.21.15.60:0/3391636300 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9780c9ca0 0x7fb978157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 >> 172.21.15.60:0/3391636300 conn(0x7fb9780ce9b0 msgr2=0x7fb9780ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 shutdown_connections 2023-10-16T14:10:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.270+0000 7fb97ce3f640 1 -- 172.21.15.60:0/3391636300 wait complete. 2023-10-16T14:10:05.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:10:05.305 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.301+0000 7fe0c1c2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:10:05.347 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.344+0000 7f6cc8630040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:10:05.408 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.406+0000 7f6cc8630040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:05.424 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.421+0000 7fe0c1c2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:10:05.457 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.454+0000 7f6cc8630040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:10:05.492 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.488+0000 7fe0c1c2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:05.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.499+0000 7fd166ef7640 1 Processor -- start 2023-10-16T14:10:05.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.499+0000 7fd166ef7640 1 -- start start 2023-10-16T14:10:05.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.501+0000 7fd166ef7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd16015af80 0x7fd1600cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.502+0000 7fd166ef7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16015eff0 con 0x7fd160065ab0 2023-10-16T14:10:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.502+0000 7fd166ef7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16015ecc0 con 0x7fd16006faf0 2023-10-16T14:10:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.502+0000 7fd166ef7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600d0930 con 0x7fd16015af80 2023-10-16T14:10:05.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.502+0000 7fd165494640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd160065ab0 0x7fd1600ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49592/0 (socket says 172.21.15.60:49592) 2023-10-16T14:10:05.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.502+0000 7fd165494640 1 -- 172.21.15.60:0/2665376589 learned_addr learned my addr 172.21.15.60:0/2665376589 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:05.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.504+0000 7fd164c93640 1 --2- 172.21.15.60:0/2665376589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd16015af80 0x7fd1600cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.504+0000 7fd15f7fe640 1 -- 172.21.15.60:0/2665376589 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804281323 0 0) 0x7fd16015eff0 con 0x7fd160065ab0 2023-10-16T14:10:05.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.504+0000 7fd15f7fe640 1 -- 172.21.15.60:0/2665376589 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd148003520 con 0x7fd160065ab0 2023-10-16T14:10:05.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.505+0000 7fd164c93640 1 -- 172.21.15.60:0/2665376589 >> v1:172.21.15.144:6789/0 conn(0x7fd16006faf0 legacy=0x7fd16015f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.505+0000 7fd164c93640 1 -- 172.21.15.60:0/2665376589 >> v1:172.21.15.60:6789/0 conn(0x7fd160065ab0 legacy=0x7fd1600ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.505+0000 7fd164c93640 1 -- 172.21.15.60:0/2665376589 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16015b360 con 0x7fd16015af80 2023-10-16T14:10:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.508+0000 7fd164c93640 1 --2- 172.21.15.60:0/2665376589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd16015af80 0x7fd1600cb1c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd1500097d0 tx=0x7fd15002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e072bd0197f1eb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.508+0000 7fd15f7fe640 1 -- 172.21.15.60:0/2665376589 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15002d040 con 0x7fd16015af80 2023-10-16T14:10:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.508+0000 7fd15f7fe640 1 -- 172.21.15.60:0/2665376589 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd150031810 con 0x7fd16015af80 2023-10-16T14:10:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.508+0000 7fd15f7fe640 1 -- 172.21.15.60:0/2665376589 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150031b80 con 0x7fd16015af80 2023-10-16T14:10:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 -- 172.21.15.60:0/2665376589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd16015af80 msgr2=0x7fd1600cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 --2- 172.21.15.60:0/2665376589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd16015af80 0x7fd1600cb1c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd1500097d0 tx=0x7fd15002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 -- 172.21.15.60:0/2665376589 shutdown_connections 2023-10-16T14:10:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 --2- 172.21.15.60:0/2665376589 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd16015af80 0x7fd1600cb1c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 -- 172.21.15.60:0/2665376589 >> 172.21.15.60:0/2665376589 conn(0x7fd1600d0ab0 msgr2=0x7fd1600d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 -- 172.21.15.60:0/2665376589 shutdown_connections 2023-10-16T14:10:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.509+0000 7fd166ef7640 1 -- 172.21.15.60:0/2665376589 wait complete. 2023-10-16T14:10:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.511+0000 7fd166ef7640 1 Processor -- start 2023-10-16T14:10:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.511+0000 7fd166ef7640 1 -- start start 2023-10-16T14:10:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd166ef7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd160065ab0 0x7fd160152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd166ef7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd16006faf0 0x7fd16015b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd166ef7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 0x7fd16015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd166ef7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600b1b60 con 0x7fd16015af80 2023-10-16T14:10:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd166ef7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600b15a0 con 0x7fd16006faf0 2023-10-16T14:10:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd166ef7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600b1880 con 0x7fd160065ab0 2023-10-16T14:10:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.512+0000 7fd165494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 0x7fd16015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 0x7fd16015bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55436/0 (socket says 172.21.15.60:55436) 2023-10-16T14:10:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 -- 172.21.15.60:0/3798494252 learned_addr learned my addr 172.21.15.60:0/3798494252 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd164c93640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd160065ab0 0x7fd160152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd15ffff640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd16006faf0 0x7fd16015b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 -- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd160065ab0 msgr2=0x7fd160152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd160065ab0 0x7fd160152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 -- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd16006faf0 msgr2=0x7fd16015b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd16006faf0 0x7fd16015b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 -- 172.21.15.60:0/3798494252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150009400 con 0x7fd16015af80 2023-10-16T14:10:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd164c93640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd160065ab0 0x7fd160152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd165494640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 0x7fd16015bb50 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd15800caa0 tx=0x7fd15800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.513+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15800ac00 con 0x7fd16015af80 2023-10-16T14:10:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.514+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15800ad60 con 0x7fd16015af80 2023-10-16T14:10:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.514+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158012620 con 0x7fd16015af80 2023-10-16T14:10:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.514+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1600c45f0 con 0x7fd16015af80 2023-10-16T14:10:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.514+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd160153220 con 0x7fd16015af80 2023-10-16T14:10:05.522 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.517+0000 7f6cc8630040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:10:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.515+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1600d0610 con 0x7fd16015af80 2023-10-16T14:10:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.517+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd158007070 con 0x7fd16015af80 2023-10-16T14:10:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.518+0000 7fd15d7fa640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 0x7fd14c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.518+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6199+0+0 (secure 0 0 0) 0x7fd158089980 con 0x7fd16015af80 2023-10-16T14:10:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.518+0000 7fd164c93640 1 -- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 msgr2=0x7fd14c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.518+0000 7fd164c93640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 0x7fd14c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:05.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.522+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd158054720 con 0x7fd16015af80 2023-10-16T14:10:05.552 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.548+0000 7fe0c1c2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:10:05.570 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.567+0000 7f6cc8630040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:10:05.612 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.608+0000 7fe0c1c2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:10:05.667 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:05.661+0000 7fe0c1c2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:10:05.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.715+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1600c4810 con 0x7fd16015af80 2023-10-16T14:10:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.716+0000 7fd15d7fa640 1 -- 172.21.15.60:0/3798494252 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v217) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1580170a0 con 0x7fd16015af80 2023-10-16T14:10:05.720 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:10:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd164c93640 1 -- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 msgr2=0x7fd14c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 msgr2=0x7fd14c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:10:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd164c93640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 0x7fd14c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 0x7fd14c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 msgr2=0x7fd16015bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 0x7fd16015bb50 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd15800caa0 tx=0x7fd15800cf60 comp rx=0 tx=0).stop 2023-10-16T14:10:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 shutdown_connections 2023-10-16T14:10:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd14c06af90 0x7fd14c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd16015af80 0x7fd16015bb50 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd16006faf0 0x7fd16015b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 --2- 172.21.15.60:0/3798494252 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd160065ab0 0x7fd160152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 >> 172.21.15.60:0/3798494252 conn(0x7fd1600d0ab0 msgr2=0x7fd16015d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 shutdown_connections 2023-10-16T14:10:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:05.719+0000 7fd166ef7640 1 -- 172.21.15.60:0/3798494252 wait complete. 2023-10-16T14:10:05.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:10:05.947 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:05.944+0000 7f6cc8630040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:10:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.008+0000 7f5ed160e640 1 Processor -- start 2023-10-16T14:10:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.008+0000 7f5ed160e640 1 -- start start 2023-10-16T14:10:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ed160e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ecc0caf80 0x7f5ecc0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ed160e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc065ab0 0x7f5ecc0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ed160e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0cb8a0 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ed160e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc15b1e0 con 0x7f5ecc0caf80 2023-10-16T14:10:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ed160e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0c9930 con 0x7f5ecc065ab0 2023-10-16T14:10:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ecaffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ecc0caf80 0x7f5ecc0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ecaffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ecc0caf80 0x7f5ecc0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55444/0 (socket says 172.21.15.60:55444) 2023-10-16T14:10:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5ecaffd640 1 -- 172.21.15.60:0/4029568397 learned_addr learned my addr 172.21.15.60:0/4029568397 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.010+0000 7f5eca7fc640 1 --2- 172.21.15.60:0/4029568397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc065ab0 0x7f5ecc0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.011+0000 7f5ecaffd640 1 -- 172.21.15.60:0/4029568397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc065ab0 msgr2=0x7f5ecc0cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.011+0000 7f5ecaffd640 1 --2- 172.21.15.60:0/4029568397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc065ab0 0x7f5ecc0cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.011+0000 7f5ecaffd640 1 -- 172.21.15.60:0/4029568397 >> v1:172.21.15.144:6789/0 conn(0x7f5ecc0b4e50 legacy=0x7f5ecc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.011+0000 7f5ecaffd640 1 -- 172.21.15.60:0/4029568397 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0c9c10 con 0x7f5ecc0caf80 2023-10-16T14:10:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.012+0000 7f5ecaffd640 1 --2- 172.21.15.60:0/4029568397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0caf80 0x7f5ecc0ca9d0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098a0 tx=0x7f5ebc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a068019b16e57988 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.012+0000 7f5ec9ffb640 1 -- 172.21.15.60:0/4029568397 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0317c0 con 0x7f5ecc0caf80 2023-10-16T14:10:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.012+0000 7f5ec9ffb640 1 -- 172.21.15.60:0/4029568397 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc031920 con 0x7f5ecc0caf80 2023-10-16T14:10:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.012+0000 7f5ec9ffb640 1 -- 172.21.15.60:0/4029568397 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc031cb0 con 0x7f5ecc0caf80 2023-10-16T14:10:06.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 -- 172.21.15.60:0/4029568397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0caf80 msgr2=0x7f5ecc0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4029568397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0caf80 0x7f5ecc0ca9d0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098a0 tx=0x7f5ebc02b860 comp rx=0 tx=0).stop 2023-10-16T14:10:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 -- 172.21.15.60:0/4029568397 shutdown_connections 2023-10-16T14:10:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4029568397 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc065ab0 0x7f5ecc0cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4029568397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0caf80 0x7f5ecc0ca9d0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 -- 172.21.15.60:0/4029568397 >> 172.21.15.60:0/4029568397 conn(0x7f5ecc06bc10 msgr2=0x7f5ecc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 -- 172.21.15.60:0/4029568397 shutdown_connections 2023-10-16T14:10:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.013+0000 7f5ed160e640 1 -- 172.21.15.60:0/4029568397 wait complete. 2023-10-16T14:10:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.015+0000 7f5ed160e640 1 Processor -- start 2023-10-16T14:10:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.015+0000 7f5ed160e640 1 -- start start 2023-10-16T14:10:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.016+0000 7f5ed160e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ecc065ab0 0x7f5ecc145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.016+0000 7f5ed160e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 0x7f5ecc152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.016+0000 7f5ed160e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc0caf80 0x7f5ecc1f1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.016+0000 7f5ed160e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0c2d40 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.016+0000 7f5ed160e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc145b60 con 0x7f5ecc065ab0 2023-10-16T14:10:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.016+0000 7f5ed160e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc1529d0 con 0x7f5ecc0caf80 2023-10-16T14:10:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 0x7f5ecc152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 0x7f5ecc152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55448/0 (socket says 172.21.15.60:55448) 2023-10-16T14:10:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 -- 172.21.15.60:0/4031398015 learned_addr learned my addr 172.21.15.60:0/4031398015 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5ecaffd640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ecc065ab0 0x7f5ecc145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5ecb7fe640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc0caf80 0x7f5ecc1f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 -- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc0caf80 msgr2=0x7f5ecc1f1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc0caf80 0x7f5ecc1f1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 -- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ecc065ab0 msgr2=0x7f5ecc145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ecc065ab0 0x7f5ecc145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.017+0000 7f5eca7fc640 1 -- 172.21.15.60:0/4031398015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0094d0 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.018+0000 7f5eca7fc640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 0x7f5ecc152180 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000d5f0 tx=0x7f5ec000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.018+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00125f0 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.018+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0012750 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.018+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0012b00 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.018+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc152cb0 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.018+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc0c3260 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.019+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc0c3420 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.020+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5ec0012c60 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.021+0000 7f5eb37fe640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f5eb806af90 0x7f5eb806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.021+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f5ec0089ee0 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.021+0000 7f5ecaffd640 1 -- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f5eb806af90 msgr2=0x7f5eb806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:06.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.021+0000 7f5ecaffd640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f5eb806af90 0x7f5eb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:06.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.025+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ec0054c00 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.075 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.071+0000 7fe0c1c2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:10:06.115 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.112+0000 7f6cc8630040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:10:06.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.218+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ecc04e160 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.222 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.219+0000 7f6cc8630040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:10:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.218+0000 7f5eb37fe640 1 -- 172.21.15.60:0/4031398015 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v218) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ec0058220 con 0x7f5ecc0b4e50 2023-10-16T14:10:06.223 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:10:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f5eb806af90 msgr2=0x7f5eb806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f5eb806af90 0x7f5eb806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 msgr2=0x7f5ecc152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 0x7f5ecc152180 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000d5f0 tx=0x7f5ec000dab0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 shutdown_connections 2023-10-16T14:10:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f5eb806af90 0x7f5eb806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ecc0caf80 0x7f5ecc1f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ecc0b4e50 0x7f5ecc152180 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 --2- 172.21.15.60:0/4031398015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ecc065ab0 0x7f5ecc145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 >> 172.21.15.60:0/4031398015 conn(0x7f5ecc06bc10 msgr2=0x7f5ecc0b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 shutdown_connections 2023-10-16T14:10:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.221+0000 7f5ed160e640 1 -- 172.21.15.60:0/4031398015 wait complete. 2023-10-16T14:10:06.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:10:06.248 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.241+0000 7fe0c1c2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:10:06.271 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.268+0000 7f6cc8630040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:10:06.361 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.357+0000 7fe0c1c2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:10:06.385 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.382+0000 7f6cc8630040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:06.415 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.411+0000 7fe0c1c2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:10:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.445+0000 7fd8f8621640 1 Processor -- start 2023-10-16T14:10:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.446+0000 7fd8f8621640 1 -- start start 2023-10-16T14:10:06.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f8621640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd8f00cf5f0 0x7fd8f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f8621640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd8f0070c10 0x7fd8f00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f8621640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f0161740 con 0x7fd8f0070830 2023-10-16T14:10:06.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f8621640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f00cdbd0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f8621640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f006b0f0 con 0x7fd8f0070c10 2023-10-16T14:10:06.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f5bbc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd8f00cf5f0 0x7fd8f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f5bbc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd8f00cf5f0 0x7fd8f006d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55450/0 (socket says 172.21.15.60:55450) 2023-10-16T14:10:06.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.447+0000 7fd8f5bbc640 1 -- 172.21.15.60:0/3678643802 learned_addr learned my addr 172.21.15.60:0/3678643802 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:06.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.448+0000 7fd8f6bbe640 1 --2- 172.21.15.60:0/3678643802 >> v2:172.21.15.144:3300/0 conn(0x7fd8f0070c10 0x7fd8f00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.448+0000 7fd8f5bbc640 1 -- 172.21.15.60:0/3678643802 >> v2:172.21.15.144:3300/0 conn(0x7fd8f0070c10 msgr2=0x7fd8f00cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:06.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.448+0000 7fd8f5bbc640 1 --2- 172.21.15.60:0/3678643802 >> v2:172.21.15.144:3300/0 conn(0x7fd8f0070c10 0x7fd8f00cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.448+0000 7fd8f5bbc640 1 -- 172.21.15.60:0/3678643802 >> v1:172.21.15.144:6789/0 conn(0x7fd8f0070830 legacy=0x7fd8f006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:06.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.448+0000 7fd8f5bbc640 1 -- 172.21.15.60:0/3678643802 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f01619b0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.449+0000 7fd8f5bbc640 1 --2- 172.21.15.60:0/3678643802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f006d3c0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00d0b90 tx=0x7fd8e002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cb6814fb0f8b72 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:06.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.449+0000 7fd8f53bb640 1 -- 172.21.15.60:0/3678643802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031750 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.449+0000 7fd8f53bb640 1 -- 172.21.15.60:0/3678643802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e00318b0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.449+0000 7fd8f53bb640 1 -- 172.21.15.60:0/3678643802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031c40 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 -- 172.21.15.60:0/3678643802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 msgr2=0x7fd8f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 --2- 172.21.15.60:0/3678643802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f006d3c0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00d0b90 tx=0x7fd8e002b7f0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 -- 172.21.15.60:0/3678643802 shutdown_connections 2023-10-16T14:10:06.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 --2- 172.21.15.60:0/3678643802 >> v2:172.21.15.144:3300/0 conn(0x7fd8f0070c10 0x7fd8f00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 --2- 172.21.15.60:0/3678643802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f006d3c0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 -- 172.21.15.60:0/3678643802 >> 172.21.15.60:0/3678643802 conn(0x7fd8f00cfac0 msgr2=0x7fd8f00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:06.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 -- 172.21.15.60:0/3678643802 shutdown_connections 2023-10-16T14:10:06.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.450+0000 7fd8f8621640 1 -- 172.21.15.60:0/3678643802 wait complete. 2023-10-16T14:10:06.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.452+0000 7fd8f8621640 1 Processor -- start 2023-10-16T14:10:06.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.452+0000 7fd8f8621640 1 -- start start 2023-10-16T14:10:06.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.453+0000 7fd8f8621640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8f0070830 0x7fd8f0158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.453+0000 7fd8f8621640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8f0070c10 0x7fd8f00c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.453+0000 7fd8f8621640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f015eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.453+0000 7fd8f8621640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00c4e90 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.453+0000 7fd8f8621640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00c5410 con 0x7fd8f0070c10 2023-10-16T14:10:06.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.453+0000 7fd8f8621640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00c33e0 con 0x7fd8f0070830 2023-10-16T14:10:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f015eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f015eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55464/0 (socket says 172.21.15.60:55464) 2023-10-16T14:10:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 -- 172.21.15.60:0/2065309686 learned_addr learned my addr 172.21.15.60:0/2065309686 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f63bd640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8f0070830 0x7fd8f0158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f5bbc640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8f0070c10 0x7fd8f00c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 -- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8f0070830 msgr2=0x7fd8f0158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8f0070830 0x7fd8f0158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 -- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8f0070c10 msgr2=0x7fd8f00c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8f0070c10 0x7fd8f00c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 -- 172.21.15.60:0/2065309686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00094d0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.454+0000 7fd8f6bbe640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f015eb90 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8007e40 tx=0x7fd8e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.455+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8014620 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.455+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e800ee90 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.455+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80149e0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.455+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f00c36c0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.455+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f00c39a0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.456+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f00cd9f0 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.457+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd8e8014b40 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.458+0000 7fd8e6ffd640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 0x7fd8d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.458+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6183+0+0 (secure 0 0 0) 0x7fd8e808a390 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.458+0000 7fd8f63bd640 1 -- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 msgr2=0x7fd8d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.458+0000 7fd8f63bd640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 0x7fd8d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.462+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8e8055150 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.492 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.489+0000 7f6cc8630040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:10:06.534 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.530+0000 7fe0c1c2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:06.653 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.649+0000 7f6cc8630040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:06.653 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.650+0000 7fe0c1c2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:10:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.659+0000 7fd8f63bd640 1 -- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 msgr2=0x7fd8d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.659+0000 7fd8f63bd640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 0x7fd8d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:06.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.680+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd8f006ba30 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.681+0000 7fd8e6ffd640 1 -- 172.21.15.60:0/2065309686 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v218) v1 ==== 80+0+475 (secure 0 0 0) 0x7fd8e8058770 con 0x7fd8f00cf5f0 2023-10-16T14:10:06.685 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/351085041 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.685 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/992081872 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.685 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/4042889662 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.686 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/166629007 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.686 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/1274087552 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.686 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2940276091 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.686 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/1274087552 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.687 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2969783877 2023-10-17T14:07:58.997006+0000 2023-10-16T14:10:06.687 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T14:10:06.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 msgr2=0x7fd8d406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:06.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 0x7fd8d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 msgr2=0x7fd8f015eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:06.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f015eb90 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8007e40 tx=0x7fd8e800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 shutdown_connections 2023-10-16T14:10:06.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fd8d406af90 0x7fd8d406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8f00cf5f0 0x7fd8f015eb90 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd8f0070c10 0x7fd8f00c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 --2- 172.21.15.60:0/2065309686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd8f0070830 0x7fd8f0158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:06.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 >> 172.21.15.60:0/2065309686 conn(0x7fd8f00cfac0 msgr2=0x7fd8f006c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:06.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 shutdown_connections 2023-10-16T14:10:06.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:06.684+0000 7fd8f8621640 1 -- 172.21.15.60:0/2065309686 wait complete. 2023-10-16T14:10:06.700 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.697+0000 7f6cc8630040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:06.706 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:10:06.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:06.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:10:06.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:10:06.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:06.817 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:10:06.822 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:06.818+0000 7f6cc8630040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:10:06.825 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.821+0000 7fe0c1c2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:06.865 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:10:06.866 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:06.879 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:06.875+0000 7fe0c1c2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:06.921 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:10:06.921 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:10:06.922 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:06.977 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:10:06.978 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:10:06.978 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:07.006 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:07.003+0000 7fe0c1c2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:10:07.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:10:07.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.246+0000 7fafe2c00640 1 Processor -- start 2023-10-16T14:10:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.246+0000 7fafe2c00640 1 -- start start 2023-10-16T14:10:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafe2c00640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafe2c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fafdc06fa90 0x7fafdc0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafe2c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc15c1b0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafe2c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafdc0ce2d0 con 0x7fafdc06fa90 2023-10-16T14:10:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafe2c00640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0b63d0 con 0x7fafdc15ace0 2023-10-16T14:10:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafdbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fafdc15b0c0 0x7fafdc15baa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49604/0 (socket says 172.21.15.60:49604) 2023-10-16T14:10:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.248+0000 7fafdbfff640 1 -- 172.21.15.60:0/2835087887 learned_addr learned my addr 172.21.15.60:0/2835087887 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafe119d640 1 --2- 172.21.15.60:0/2835087887 >> v2:172.21.15.60:3300/0 conn(0x7fafdc06fa90 0x7fafdc0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafe099c640 1 --2- 172.21.15.60:0/2835087887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafdb7fe640 1 -- 172.21.15.60:0/2835087887 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340230477 0 0) 0x7fafdc15c1b0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafdb7fe640 1 -- 172.21.15.60:0/2835087887 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc003520 con 0x7fafdc15b0c0 2023-10-16T14:10:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafe119d640 1 -- 172.21.15.60:0/2835087887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 msgr2=0x7fafdc0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafe119d640 1 --2- 172.21.15.60:0/2835087887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc0b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafe119d640 1 -- 172.21.15.60:0/2835087887 >> v1:172.21.15.60:6789/0 conn(0x7fafdc15b0c0 legacy=0x7fafdc15baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.249+0000 7fafe119d640 1 -- 172.21.15.60:0/2835087887 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0b66b0 con 0x7fafdc06fa90 2023-10-16T14:10:07.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.251+0000 7fafe119d640 1 --2- 172.21.15.60:0/2835087887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc06fa90 0x7fafdc0cfea0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009d90 tx=0x7fafd402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41247ea7015d5ab1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.251+0000 7fafdb7fe640 1 -- 172.21.15.60:0/2835087887 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd402cbc0 con 0x7fafdc06fa90 2023-10-16T14:10:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.251+0000 7fafdb7fe640 1 -- 172.21.15.60:0/2835087887 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd402cd20 con 0x7fafdc06fa90 2023-10-16T14:10:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.251+0000 7fafdb7fe640 1 -- 172.21.15.60:0/2835087887 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4034660 con 0x7fafdc06fa90 2023-10-16T14:10:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 -- 172.21.15.60:0/2835087887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc06fa90 msgr2=0x7fafdc0cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 --2- 172.21.15.60:0/2835087887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc06fa90 0x7fafdc0cfea0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009d90 tx=0x7fafd402be10 comp rx=0 tx=0).stop 2023-10-16T14:10:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 -- 172.21.15.60:0/2835087887 shutdown_connections 2023-10-16T14:10:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 --2- 172.21.15.60:0/2835087887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc06fa90 0x7fafdc0cfea0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 --2- 172.21.15.60:0/2835087887 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 -- 172.21.15.60:0/2835087887 >> 172.21.15.60:0/2835087887 conn(0x7fafdc06d700 msgr2=0x7fafdc06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 -- 172.21.15.60:0/2835087887 shutdown_connections 2023-10-16T14:10:07.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.252+0000 7fafe2c00640 1 -- 172.21.15.60:0/2835087887 wait complete. 2023-10-16T14:10:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.254+0000 7fafe2c00640 1 Processor -- start 2023-10-16T14:10:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.254+0000 7fafe2c00640 1 -- start start 2023-10-16T14:10:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe2c00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafdc06fa90 0x7fafdc0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe2c00640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc1f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe2c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 0x7fafdc15cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe2c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc158660 con 0x7fafdc15b0c0 2023-10-16T14:10:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe2c00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0c8cd0 con 0x7fafdc06fa90 2023-10-16T14:10:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe2c00640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0c8f50 con 0x7fafdc15ace0 2023-10-16T14:10:07.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe119d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 0x7fafdc15cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe119d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 0x7fafdc15cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55482/0 (socket says 172.21.15.60:55482) 2023-10-16T14:10:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe119d640 1 -- 172.21.15.60:0/4120919896 learned_addr learned my addr 172.21.15.60:0/4120919896 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafdbfff640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc1f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.255+0000 7fafe099c640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafdc06fa90 0x7fafdc0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe119d640 1 -- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 msgr2=0x7fafdc1f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe119d640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc1f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe119d640 1 -- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafdc06fa90 msgr2=0x7fafdc0c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe119d640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafdc06fa90 0x7fafdc0c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe119d640 1 -- 172.21.15.60:0/4120919896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd4009a70 con 0x7fafdc15b0c0 2023-10-16T14:10:07.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe119d640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 0x7fafdc15cce0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009d90 tx=0x7fafd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4040650 con 0x7fafdc15b0c0 2023-10-16T14:10:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd402ce90 con 0x7fafdc15b0c0 2023-10-16T14:10:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40409f0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.256+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc0c9230 con 0x7fafdc15b0c0 2023-10-16T14:10:07.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.257+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc0c97f0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.258+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafdc0d03c0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.259+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fafd4040b50 con 0x7fafdc15b0c0 2023-10-16T14:10:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.260+0000 7fafd97fa640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 0x7fafc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.261+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6183+0+0 (secure 0 0 0) 0x7fafd40ac530 con 0x7fafdc15b0c0 2023-10-16T14:10:07.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.261+0000 7fafe099c640 1 -- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 msgr2=0x7fafc006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:07.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.261+0000 7fafe099c640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 0x7fafc006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.265+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafdc0d03c0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.462+0000 7fafe099c640 1 -- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 msgr2=0x7fafc006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:07.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.462+0000 7fafe099c640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 0x7fafc006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.496+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafdc0d03c0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.497+0000 7fafd97fa640 1 -- 172.21.15.60:0/4120919896 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4923 (secure 0 0 0) 0x7fafdc0d03c0 con 0x7fafdc15b0c0 2023-10-16T14:10:07.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:07.501 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:07:44.252784+0000","modified":"2023-10-16T14:07:47.343844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8539":{"gid":8539,"name":"b","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6835/3999834969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3999834969},{"type":"v1","addr":"172.21.15.144:6835","nonce":3999834969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T14:10:07.501 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 104 2023-10-16T14:10:07.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 msgr2=0x7fafc006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:07.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 0x7fafc006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 msgr2=0x7fafdc15cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 0x7fafdc15cce0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009d90 tx=0x7fafd4004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 shutdown_connections 2023-10-16T14:10:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7fafc006af90 0x7fafc006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fafdc15ace0 0x7fafdc1f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fafdc15b0c0 0x7fafdc15cce0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 --2- 172.21.15.60:0/4120919896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafdc06fa90 0x7fafdc0c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.499+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 >> 172.21.15.60:0/4120919896 conn(0x7fafdc06d700 msgr2=0x7fafdc06dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.500+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 shutdown_connections 2023-10-16T14:10:07.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.500+0000 7fafe2c00640 1 -- 172.21.15.60:0/4120919896 wait complete. 2023-10-16T14:10:07.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:07.661 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:07.658+0000 7f6cc8630040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:10:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.708+0000 7f1a74548640 1 Processor -- start 2023-10-16T14:10:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.708+0000 7f1a74548640 1 -- start start 2023-10-16T14:10:07.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a74548640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159f00 0x7f1a6c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a74548640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a6c15d880 0x7f1a6c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a74548640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c06f7f0 con 0x7f1a6c159b20 2023-10-16T14:10:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a74548640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c06c5e0 con 0x7f1a6c15d880 2023-10-16T14:10:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a74548640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c06c8c0 con 0x7f1a6c159f00 2023-10-16T14:10:07.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a6c15d880 0x7f1a6c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a6c15d880 0x7f1a6c0b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55486/0 (socket says 172.21.15.60:55486) 2023-10-16T14:10:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 -- 172.21.15.60:0/1307634761 learned_addr learned my addr 172.21.15.60:0/1307634761 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a71ae3640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159f00 0x7f1a6c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 -- 172.21.15.60:0/1307634761 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159f00 msgr2=0x7f1a6c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159f00 0x7f1a6c0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 -- 172.21.15.60:0/1307634761 >> v1:172.21.15.144:6789/0 conn(0x7f1a6c159b20 legacy=0x7f1a6c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a72ae5640 1 -- 172.21.15.60:0/1307634761 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c06cba0 con 0x7f1a6c15d880 2023-10-16T14:10:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.710+0000 7f1a71ae3640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159f00 0x7f1a6c0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.712+0000 7f1a72ae5640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0b0950 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009e50 tx=0x7f1a6402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b5551f1afd4728 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.712+0000 7f1a712e2640 1 -- 172.21.15.60:0/1307634761 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6402ca80 con 0x7f1a6c15d880 2023-10-16T14:10:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.712+0000 7f1a712e2640 1 -- 172.21.15.60:0/1307634761 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6402cbe0 con 0x7f1a6c15d880 2023-10-16T14:10:07.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.712+0000 7f1a712e2640 1 -- 172.21.15.60:0/1307634761 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a640343f0 con 0x7f1a6c15d880 2023-10-16T14:10:07.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.712+0000 7f1a74548640 1 -- 172.21.15.60:0/1307634761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c15d880 msgr2=0x7f1a6c0b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.712+0000 7f1a74548640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0b0950 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009e50 tx=0x7f1a6402bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.713+0000 7f1a74548640 1 -- 172.21.15.60:0/1307634761 shutdown_connections 2023-10-16T14:10:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.713+0000 7f1a74548640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0b0950 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.713+0000 7f1a74548640 1 --2- 172.21.15.60:0/1307634761 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159f00 0x7f1a6c0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.713+0000 7f1a74548640 1 -- 172.21.15.60:0/1307634761 >> 172.21.15.60:0/1307634761 conn(0x7f1a6c0cb410 msgr2=0x7f1a6c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.713+0000 7f1a74548640 1 -- 172.21.15.60:0/1307634761 shutdown_connections 2023-10-16T14:10:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.713+0000 7f1a74548640 1 -- 172.21.15.60:0/1307634761 wait complete. 2023-10-16T14:10:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.714+0000 7f1a74548640 1 Processor -- start 2023-10-16T14:10:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.714+0000 7f1a74548640 1 -- start start 2023-10-16T14:10:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.715+0000 7f1a74548640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c1550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.715+0000 7f1a74548640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 0x7f1a6c0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.715+0000 7f1a74548640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.715+0000 7f1a74548640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15d090 con 0x7f1a6c159f00 2023-10-16T14:10:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.715+0000 7f1a74548640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1555e0 con 0x7f1a6c15d880 2023-10-16T14:10:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.715+0000 7f1a74548640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15c480 con 0x7f1a6c159b20 2023-10-16T14:10:07.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 0x7f1a6c0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 0x7f1a6c0c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55498/0 (socket says 172.21.15.60:55498) 2023-10-16T14:10:07.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 -- 172.21.15.60:0/1510434423 learned_addr learned my addr 172.21.15.60:0/1510434423 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:07.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a72ae5640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a722e4640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c1550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 -- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 msgr2=0x7f1a6c1550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c1550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 -- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c15d880 msgr2=0x7f1a6c0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a71ae3640 1 -- 172.21.15.60:0/1510434423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a64009b30 con 0x7f1a6c159f00 2023-10-16T14:10:07.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a722e4640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:07.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.716+0000 7f1a72ae5640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:07.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.717+0000 7f1a71ae3640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 0x7f1a6c0c6860 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c7f0 tx=0x7f1a5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:07.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.717+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c00cea0 con 0x7f1a6c159f00 2023-10-16T14:10:07.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.717+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c007a30 con 0x7f1a6c159f00 2023-10-16T14:10:07.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.717+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c007dc0 con 0x7f1a6c159f00 2023-10-16T14:10:07.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.717+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c15c760 con 0x7f1a6c159f00 2023-10-16T14:10:07.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.717+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c15cd20 con 0x7f1a6c159f00 2023-10-16T14:10:07.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.718+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c15d240 con 0x7f1a6c159f00 2023-10-16T14:10:07.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.720+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 123) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1a5c0113f0 con 0x7f1a6c159f00 2023-10-16T14:10:07.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.721+0000 7f1a62ffd640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 0x7f1a5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:07.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.721+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6183+0+0 (secure 0 0 0) 0x7f1a5c089bc0 con 0x7f1a6c159f00 2023-10-16T14:10:07.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.721+0000 7f1a722e4640 1 -- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 msgr2=0x7f1a5006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:07.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.721+0000 7f1a722e4640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 0x7f1a5006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:07.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.726+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a5c054860 con 0x7f1a6c159f00 2023-10-16T14:10:07.864 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:07.860+0000 7fe0c1c2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:10:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.922+0000 7f1a722e4640 1 -- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 msgr2=0x7f1a5006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/166679164 2023-10-16T14:10:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.922+0000 7f1a722e4640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 0x7f1a5006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:07.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.926+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a6c04e160 con 0x7f1a6c159f00 2023-10-16T14:10:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.928+0000 7f1a62ffd640 1 -- 172.21.15.60:0/1510434423 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+17249 (secure 0 0 0) 0x7f1a5c01f090 con 0x7f1a6c159f00 2023-10-16T14:10:07.933 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:07.933 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":218,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:05.974841+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":218,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":1}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/351085041":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/992081872":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/4042889662":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/166629007":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6832/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2940276091":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6833/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2969783877":"2023-10-17T14:07:58.997006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 msgr2=0x7f1a5006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 0x7f1a5006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 msgr2=0x7f1a6c0c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 0x7f1a6c0c6860 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c7f0 tx=0x7f1a5c00ccb0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 shutdown_connections 2023-10-16T14:10:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:6832/166679164,v1:172.21.15.144:6833/166679164] conn(0x7f1a5006af90 0x7f1a5006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a6c15d880 0x7f1a6c0ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a6c159f00 0x7f1a6c0c6860 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 --2- 172.21.15.60:0/1510434423 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1a6c159b20 0x7f1a6c1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:07.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 >> 172.21.15.60:0/1510434423 conn(0x7f1a6c0cb410 msgr2=0x7f1a6c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:07.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 shutdown_connections 2023-10-16T14:10:07.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:07.932+0000 7f1a74548640 1 -- 172.21.15.60:0/1510434423 wait complete. 2023-10-16T14:10:07.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:07:42.198213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:07:43.287022+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 218, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:10:07.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:10:07.958 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:07.956+0000 7f6cc8630040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:10:08.018 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.015+0000 7f6cc8630040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:10:08.131 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.128+0000 7f6cc8630040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:10:08.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.166+0000 7f1d2cc90640 1 Processor -- start 2023-10-16T14:10:08.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.166+0000 7f1d2cc90640 1 -- start start 2023-10-16T14:10:08.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.167+0000 7f1d2cc90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1d2815d030 0x7f1d280d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.167+0000 7f1d2cc90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d28161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.167+0000 7f1d2cc90640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1d28161990 0x7f1d280cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.167+0000 7f1d2cc90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d280d2b50 con 0x7f1d2815d030 2023-10-16T14:10:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.167+0000 7f1d2cc90640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d280b5d90 con 0x7f1d28161990 2023-10-16T14:10:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.167+0000 7f1d2cc90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28160fc0 con 0x7f1d2815d410 2023-10-16T14:10:08.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1d2815d030 0x7f1d280d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1d2815d030 0x7f1d280d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55504/0 (socket says 172.21.15.60:55504) 2023-10-16T14:10:08.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d26d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1d28161990 0x7f1d280cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 -- 172.21.15.60:0/3973238582 learned_addr learned my addr 172.21.15.60:0/3973238582 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d25d9b640 1 --2- 172.21.15.60:0/3973238582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d28161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 -- 172.21.15.60:0/3973238582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 msgr2=0x7f1d28161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 --2- 172.21.15.60:0/3973238582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d28161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 -- 172.21.15.60:0/3973238582 >> v2:172.21.15.144:3300/0 conn(0x7f1d28161990 msgr2=0x7f1d280cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:08.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 --2- 172.21.15.60:0/3973238582 >> v2:172.21.15.144:3300/0 conn(0x7f1d28161990 0x7f1d280cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.168+0000 7f1d2659c640 1 -- 172.21.15.60:0/3973238582 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d280d25d0 con 0x7f1d2815d030 2023-10-16T14:10:08.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.169+0000 7f1d2659c640 1 --2- 172.21.15.60:0/3973238582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d2815d030 0x7f1d280d1630 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f1d28073850 tx=0x7f1d1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719b4de8cfc47d21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:08.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.172+0000 7f1d2559a640 1 -- 172.21.15.60:0/3973238582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18030860 con 0x7f1d2815d030 2023-10-16T14:10:08.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.172+0000 7f1d2559a640 1 -- 172.21.15.60:0/3973238582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d180309c0 con 0x7f1d2815d030 2023-10-16T14:10:08.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.172+0000 7f1d2559a640 1 -- 172.21.15.60:0/3973238582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18030d50 con 0x7f1d2815d030 2023-10-16T14:10:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 -- 172.21.15.60:0/3973238582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d2815d030 msgr2=0x7f1d280d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/3973238582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d2815d030 0x7f1d280d1630 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f1d28073850 tx=0x7f1d1802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 -- 172.21.15.60:0/3973238582 shutdown_connections 2023-10-16T14:10:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/3973238582 >> v2:172.21.15.144:3300/0 conn(0x7f1d28161990 0x7f1d280cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/3973238582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d28161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/3973238582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d2815d030 0x7f1d280d1630 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 -- 172.21.15.60:0/3973238582 >> 172.21.15.60:0/3973238582 conn(0x7f1d280cf720 msgr2=0x7f1d280cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 -- 172.21.15.60:0/3973238582 shutdown_connections 2023-10-16T14:10:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.173+0000 7f1d2cc90640 1 -- 172.21.15.60:0/3973238582 wait complete. 2023-10-16T14:10:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.175+0000 7f1d2cc90640 1 Processor -- start 2023-10-16T14:10:08.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.175+0000 7f1d2cc90640 1 -- start start 2023-10-16T14:10:08.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.176+0000 7f1d2cc90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2815d030 0x7f1d281f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.176+0000 7f1d2cc90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d280c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.176+0000 7f1d2cc90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d28161990 0x7f1d281577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.176+0000 7f1d2cc90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281f2830 con 0x7f1d28161990 2023-10-16T14:10:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.176+0000 7f1d2cc90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281f2a90 con 0x7f1d2815d030 2023-10-16T14:10:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.176+0000 7f1d2cc90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281f64c0 con 0x7f1d2815d410 2023-10-16T14:10:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d2659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2815d030 0x7f1d281f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d28161990 0x7f1d281577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d2659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2815d030 0x7f1d281f22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:58030/0 (socket says 172.21.15.60:58030) 2023-10-16T14:10:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d25d9b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d280c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d2659c640 1 -- 172.21.15.60:0/1332749178 learned_addr learned my addr 172.21.15.60:0/1332749178 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 -- 172.21.15.60:0/1332749178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 msgr2=0x7f1d280c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d280c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 -- 172.21.15.60:0/1332749178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2815d030 msgr2=0x7f1d281f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2815d030 0x7f1d281f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 -- 172.21.15.60:0/1332749178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180094d0 con 0x7f1d28161990 2023-10-16T14:10:08.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.177+0000 7f1d26d9d640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d28161990 0x7f1d281577d0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000efc0 tx=0x7f1d2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:08.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.178+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d20009c00 con 0x7f1d28161990 2023-10-16T14:10:08.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.178+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2000f040 con 0x7f1d28161990 2023-10-16T14:10:08.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.178+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200146c0 con 0x7f1d28161990 2023-10-16T14:10:08.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.178+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d281f67a0 con 0x7f1d28161990 2023-10-16T14:10:08.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.178+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d281f6d60 con 0x7f1d28161990 2023-10-16T14:10:08.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.179+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d281f2c60 con 0x7f1d28161990 2023-10-16T14:10:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.181+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 124) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1d20014820 con 0x7f1d28161990 2023-10-16T14:10:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.182+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f1d2008a200 con 0x7f1d28161990 2023-10-16T14:10:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.186+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d281f2c60 con 0x7f1d28161990 2023-10-16T14:10:08.191 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:08.187+0000 7fe0c1c2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:10:08.226 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4d1b7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.226 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4d1b7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.227 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4d1b7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.227 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4d1b7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.227 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4d1b7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.227 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4a1b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.228 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4a1b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.228 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4a1b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.228 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4a1b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.228 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:08.223+0000 7f6c4a1b1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:08.255 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:08.251+0000 7fe0c1c2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:10:08.374 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:08.371+0000 7fe0c1c2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:10:08.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.381+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d281f2c00 con 0x7f1d28161990 2023-10-16T14:10:08.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.382+0000 7f1d16ffd640 1 -- 172.21.15.60:0/1332749178 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d200553b0 con 0x7f1d28161990 2023-10-16T14:10:08.385 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:10:08.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d28161990 msgr2=0x7f1d281577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d28161990 0x7f1d281577d0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000efc0 tx=0x7f1d2000c490 comp rx=0 tx=0).stop 2023-10-16T14:10:08.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 shutdown_connections 2023-10-16T14:10:08.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1d2815d410 0x7f1d280c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d28161990 0x7f1d281577d0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 --2- 172.21.15.60:0/1332749178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2815d030 0x7f1d281f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.384+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 >> 172.21.15.60:0/1332749178 conn(0x7f1d280cf720 msgr2=0x7f1d2815b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:08.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.385+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 shutdown_connections 2023-10-16T14:10:08.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.385+0000 7f1d2cc90640 1 -- 172.21.15.60:0/1332749178 wait complete. 2023-10-16T14:10:08.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:10:08.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.626+0000 7f9fc1741640 1 Processor -- start 2023-10-16T14:10:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.626+0000 7f9fc1741640 1 -- start start 2023-10-16T14:10:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.627+0000 7f9fc1741640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9fbc15ad70 0x7f9fbc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.627+0000 7f9fc1741640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fbc0c9ca0 0x7f9fbc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.627+0000 7f9fc1741640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0cb030 con 0x7f9fbc0ca3b0 2023-10-16T14:10:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.627+0000 7f9fc1741640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0c91c0 con 0x7f9fbc0c9ca0 2023-10-16T14:10:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.627+0000 7f9fc1741640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0c94a0 con 0x7f9fbc15ad70 2023-10-16T14:10:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fba7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fbc0c9ca0 0x7f9fbc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fbb7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9fbc0ca3b0 0x7f9fbc0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49610/0 (socket says 172.21.15.60:49610) 2023-10-16T14:10:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fbb7fe640 1 -- 172.21.15.60:0/522002329 learned_addr learned my addr 172.21.15.60:0/522002329 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fba7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fbc0c9ca0 0x7f9fbc06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55514/0 (socket says 172.21.15.60:55514) 2023-10-16T14:10:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fbaffd640 1 --2- 172.21.15.60:0/522002329 >> v2:172.21.15.144:3300/0 conn(0x7f9fbc15ad70 0x7f9fbc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fba7fc640 1 -- 172.21.15.60:0/522002329 >> v2:172.21.15.144:3300/0 conn(0x7f9fbc15ad70 msgr2=0x7f9fbc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fba7fc640 1 --2- 172.21.15.60:0/522002329 >> v2:172.21.15.144:3300/0 conn(0x7f9fbc15ad70 0x7f9fbc069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fba7fc640 1 -- 172.21.15.60:0/522002329 >> v1:172.21.15.60:6789/0 conn(0x7f9fbc0ca3b0 legacy=0x7f9fbc0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fba7fc640 1 -- 172.21.15.60:0/522002329 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0c9780 con 0x7f9fbc0c9ca0 2023-10-16T14:10:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fbaffd640 1 --2- 172.21.15.60:0/522002329 >> v2:172.21.15.144:3300/0 conn(0x7f9fbc15ad70 0x7f9fbc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:10:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.628+0000 7f9fb9ffb640 1 -- 172.21.15.60:0/522002329 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218963407 0 0) 0x7f9fbc0cb030 con 0x7f9fbc0ca3b0 2023-10-16T14:10:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.629+0000 7f9fba7fc640 1 --2- 172.21.15.60:0/522002329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc06a2e0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0cf780 tx=0x7f9fa402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705dba55717358b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fb9ffb640 1 -- 172.21.15.60:0/522002329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa402cc70 con 0x7f9fbc0c9ca0 2023-10-16T14:10:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fb9ffb640 1 -- 172.21.15.60:0/522002329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa402cdd0 con 0x7f9fbc0c9ca0 2023-10-16T14:10:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fb9ffb640 1 -- 172.21.15.60:0/522002329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4034660 con 0x7f9fbc0c9ca0 2023-10-16T14:10:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fc1741640 1 -- 172.21.15.60:0/522002329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc0c9ca0 msgr2=0x7f9fbc06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fc1741640 1 --2- 172.21.15.60:0/522002329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc06a2e0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0cf780 tx=0x7f9fa402bec0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fc1741640 1 -- 172.21.15.60:0/522002329 shutdown_connections 2023-10-16T14:10:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fc1741640 1 --2- 172.21.15.60:0/522002329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc06a2e0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fc1741640 1 --2- 172.21.15.60:0/522002329 >> v2:172.21.15.144:3300/0 conn(0x7f9fbc15ad70 0x7f9fbc069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.630+0000 7f9fc1741640 1 -- 172.21.15.60:0/522002329 >> 172.21.15.60:0/522002329 conn(0x7f9fbc0ce9b0 msgr2=0x7f9fbc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.631+0000 7f9fc1741640 1 -- 172.21.15.60:0/522002329 shutdown_connections 2023-10-16T14:10:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.631+0000 7f9fc1741640 1 -- 172.21.15.60:0/522002329 wait complete. 2023-10-16T14:10:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.632+0000 7f9fc1741640 1 Processor -- start 2023-10-16T14:10:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.632+0000 7f9fc1741640 1 -- start start 2023-10-16T14:10:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.633+0000 7f9fc1741640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.633+0000 7f9fc1741640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fbc0ca3b0 0x7f9fbc15ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.633+0000 7f9fc1741640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 0x7f9fbc1e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.633+0000 7f9fc1741640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc158270 con 0x7f9fbc15ad70 2023-10-16T14:10:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.633+0000 7f9fc1741640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc15b450 con 0x7f9fbc0c9ca0 2023-10-16T14:10:08.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.633+0000 7f9fc1741640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc15b730 con 0x7f9fbc0ca3b0 2023-10-16T14:10:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 0x7f9fbc1e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 0x7f9fbc1e4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55516/0 (socket says 172.21.15.60:55516) 2023-10-16T14:10:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 -- 172.21.15.60:0/456503755 learned_addr learned my addr 172.21.15.60:0/456503755 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbaffd640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:08.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 -- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fbc0ca3b0 msgr2=0x7f9fbc15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fbc0ca3b0 0x7f9fbc15ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 -- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fbc0c9ca0 msgr2=0x7f9fbc1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc1575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 -- 172.21.15.60:0/456503755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4009a70 con 0x7f9fbc15ad70 2023-10-16T14:10:08.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.634+0000 7f9fbb7fe640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 0x7f9fbc1e4c10 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f9fb400c450 tx=0x7f9fb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.635+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40074e0 con 0x7f9fbc15ad70 2023-10-16T14:10:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.635+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb4007660 con 0x7f9fbc15ad70 2023-10-16T14:10:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.635+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc15ba10 con 0x7f9fbc15ad70 2023-10-16T14:10:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.635+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4004a20 con 0x7f9fbc15ad70 2023-10-16T14:10:08.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.635+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc15bfd0 con 0x7f9fbc15ad70 2023-10-16T14:10:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.636+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc04e160 con 0x7f9fbc15ad70 2023-10-16T14:10:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.637+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 124) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9fb4004b80 con 0x7f9fbc15ad70 2023-10-16T14:10:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.638+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f9fb4088970 con 0x7f9fbc15ad70 2023-10-16T14:10:08.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.642+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9fb4053b30 con 0x7f9fbc15ad70 2023-10-16T14:10:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.845+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9fbc0b2640 con 0x7f9fbc15ad70 2023-10-16T14:10:08.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.846+0000 7f9fb37fe640 1 -- 172.21.15.60:0/456503755 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v219) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9fb4018020 con 0x7f9fbc15ad70 2023-10-16T14:10:08.850 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:10:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 msgr2=0x7f9fbc1e4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 0x7f9fbc1e4c10 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f9fb400c450 tx=0x7f9fb400c910 comp rx=0 tx=0).stop 2023-10-16T14:10:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 shutdown_connections 2023-10-16T14:10:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fbc0ca3b0 0x7f9fbc15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fbc15ad70 0x7f9fbc1e4c10 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 --2- 172.21.15.60:0/456503755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fbc0c9ca0 0x7f9fbc1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 >> 172.21.15.60:0/456503755 conn(0x7f9fbc0ce9b0 msgr2=0x7f9fbc0b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.849+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 shutdown_connections 2023-10-16T14:10:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:08.850+0000 7f9fc1741640 1 -- 172.21.15.60:0/456503755 wait complete. 2023-10-16T14:10:08.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:10:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.080+0000 7f29d49bf640 1 Processor -- start 2023-10-16T14:10:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.081+0000 7f29d49bf640 1 -- start start 2023-10-16T14:10:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29d49bf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29d006c040 0x7f29d006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29d49bf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29d49bf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d006c960 con 0x7f29d00ad970 2023-10-16T14:10:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29d49bf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d015b9d0 con 0x7f29d006c040 2023-10-16T14:10:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29d49bf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d015bcb0 con 0x7f29d00cda80 2023-10-16T14:10:09.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29ced9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29d006c040 0x7f29d006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29ced9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29d006c040 0x7f29d006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43376/0 (socket says 172.21.15.60:43376) 2023-10-16T14:10:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.082+0000 7f29ced9d640 1 -- 172.21.15.60:0/1785931877 learned_addr learned my addr 172.21.15.60:0/1785931877 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.083+0000 7f29ced9d640 1 -- 172.21.15.60:0/1785931877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 msgr2=0x7f29d015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.083+0000 7f29ced9d640 1 --2- 172.21.15.60:0/1785931877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.083+0000 7f29ced9d640 1 -- 172.21.15.60:0/1785931877 >> v1:172.21.15.144:6789/0 conn(0x7f29d00ad970 legacy=0x7f29d015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.083+0000 7f29ced9d640 1 -- 172.21.15.60:0/1785931877 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d015bf90 con 0x7f29d006c040 2023-10-16T14:10:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.084+0000 7f29ced9d640 1 --2- 172.21.15.60:0/1785931877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d006c040 0x7f29d006c420 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009e50 tx=0x7f29c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51167633ef39ac7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.084+0000 7f29cdd9b640 1 -- 172.21.15.60:0/1785931877 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c002ca80 con 0x7f29d006c040 2023-10-16T14:10:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.084+0000 7f29cdd9b640 1 -- 172.21.15.60:0/1785931877 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c002cbe0 con 0x7f29d006c040 2023-10-16T14:10:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.084+0000 7f29cdd9b640 1 -- 172.21.15.60:0/1785931877 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c00343f0 con 0x7f29d006c040 2023-10-16T14:10:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 -- 172.21.15.60:0/1785931877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d006c040 msgr2=0x7f29d006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 --2- 172.21.15.60:0/1785931877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d006c040 0x7f29d006c420 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009e50 tx=0x7f29c002bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 -- 172.21.15.60:0/1785931877 shutdown_connections 2023-10-16T14:10:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 --2- 172.21.15.60:0/1785931877 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 --2- 172.21.15.60:0/1785931877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d006c040 0x7f29d006c420 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 -- 172.21.15.60:0/1785931877 >> 172.21.15.60:0/1785931877 conn(0x7f29d00ce6a0 msgr2=0x7f29d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 -- 172.21.15.60:0/1785931877 shutdown_connections 2023-10-16T14:10:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.085+0000 7f29d49bf640 1 -- 172.21.15.60:0/1785931877 wait complete. 2023-10-16T14:10:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.087+0000 7f29d49bf640 1 Processor -- start 2023-10-16T14:10:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.087+0000 7f29d49bf640 1 -- start start 2023-10-16T14:10:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29d49bf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d006c040 0x7f29d014fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29d49bf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 0x7f29d00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29d49bf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d00c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29d49bf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d00c6280 con 0x7f29d00ad970 2023-10-16T14:10:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29d49bf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d00c63f0 con 0x7f29d006c040 2023-10-16T14:10:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29d49bf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d00ad500 con 0x7f29d00cda80 2023-10-16T14:10:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29ce59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 0x7f29d00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29ce59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 0x7f29d00c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43382/0 (socket says 172.21.15.60:43382) 2023-10-16T14:10:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29ce59c640 1 -- 172.21.15.60:0/537054579 learned_addr learned my addr 172.21.15.60:0/537054579 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29ced9d640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d006c040 0x7f29d014fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.088+0000 7f29cf59e640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d00c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29ce59c640 1 -- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 msgr2=0x7f29d00c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29ce59c640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d00c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29ce59c640 1 -- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d006c040 msgr2=0x7f29d014fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29ce59c640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d006c040 0x7f29d014fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29ce59c640 1 -- 172.21.15.60:0/537054579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c0009b30 con 0x7f29d00ad970 2023-10-16T14:10:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29ce59c640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 0x7f29d00c8cf0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c400 tx=0x7f29c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c40035f0 con 0x7f29d00ad970 2023-10-16T14:10:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4003750 con 0x7f29d00ad970 2023-10-16T14:10:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4003b00 con 0x7f29d00ad970 2023-10-16T14:10:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.089+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d00c98b0 con 0x7f29d00ad970 2023-10-16T14:10:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.090+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d00c9e70 con 0x7f29d00ad970 2023-10-16T14:10:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.091+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d006b330 con 0x7f29d00ad970 2023-10-16T14:10:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.092+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 124) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f29c4003c60 con 0x7f29d00ad970 2023-10-16T14:10:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.093+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f29c4088540 con 0x7f29d00ad970 2023-10-16T14:10:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.097+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29c40537e0 con 0x7f29d00ad970 2023-10-16T14:10:09.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.155+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 125) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f29c4053290 con 0x7f29d00ad970 2023-10-16T14:10:09.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.156+0000 7f29b77fe640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f29bc0706b0 0x7f29bc072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.159+0000 7f29ced9d640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f29bc0706b0 0x7f29bc072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.170+0000 7f29ced9d640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f29bc0706b0 0x7f29bc072aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f29c0035000 tx=0x7f29c0030000 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.301+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29d00c1b40 con 0x7f29d00ad970 2023-10-16T14:10:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.302+0000 7f29b77fe640 1 -- 172.21.15.60:0/537054579 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v219) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29c4072dc0 con 0x7f29d00ad970 2023-10-16T14:10:09.306 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:10:09.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f29bc0706b0 msgr2=0x7f29bc072aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f29bc0706b0 0x7f29bc072aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f29c0035000 tx=0x7f29c0030000 comp rx=0 tx=0).stop 2023-10-16T14:10:09.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 msgr2=0x7f29d00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 0x7f29d00c8cf0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c400 tx=0x7f29c400c8c0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 shutdown_connections 2023-10-16T14:10:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f29bc0706b0 0x7f29bc072aa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29d00cda80 0x7f29d00c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29d00ad970 0x7f29d00c8cf0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 --2- 172.21.15.60:0/537054579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29d006c040 0x7f29d014fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 >> 172.21.15.60:0/537054579 conn(0x7f29d00ce6a0 msgr2=0x7f29d015da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:09.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 shutdown_connections 2023-10-16T14:10:09.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.305+0000 7f29d49bf640 1 -- 172.21.15.60:0/537054579 wait complete. 2023-10-16T14:10:09.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.540+0000 7f632abcb640 1 Processor -- start 2023-10-16T14:10:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.540+0000 7f632abcb640 1 -- start start 2023-10-16T14:10:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.541+0000 7f632abcb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6324159950 0x7f632415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.541+0000 7f632abcb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f632406c810 0x7f63240708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.541+0000 7f632abcb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6324070df0 con 0x7f6324070450 2023-10-16T14:10:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.541+0000 7f632abcb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632415e490 con 0x7f632406c810 2023-10-16T14:10:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.541+0000 7f632abcb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632415aef0 con 0x7f6324159950 2023-10-16T14:10:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.542+0000 7f6329bc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6324159950 0x7f632415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.542+0000 7f63293c8640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f6324070450 0x7f632406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:45120/0 (socket says 172.21.15.60:45120) 2023-10-16T14:10:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.542+0000 7f63293c8640 1 -- 172.21.15.60:0/2004351253 learned_addr learned my addr 172.21.15.60:0/2004351253 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.542+0000 7f632a3ca640 1 --2- 172.21.15.60:0/2004351253 >> v2:172.21.15.144:3300/0 conn(0x7f632406c810 0x7f63240708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.543+0000 7f632a3ca640 1 -- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6324159950 msgr2=0x7f632415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.543+0000 7f632a3ca640 1 --2- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6324159950 0x7f632415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.543+0000 7f632a3ca640 1 -- 172.21.15.60:0/2004351253 >> v1:172.21.15.144:6789/0 conn(0x7f6324070450 legacy=0x7f632406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.543+0000 7f632a3ca640 1 -- 172.21.15.60:0/2004351253 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632406cc80 con 0x7f632406c810 2023-10-16T14:10:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.543+0000 7f6329bc9640 1 --2- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6324159950 0x7f632415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:10:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.544+0000 7f632a3ca640 1 --2- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f632406c810 0x7f63240708b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f631c0097a0 tx=0x7f631c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b9198ba193f0a59 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.545+0000 7f6328bc7640 1 -- 172.21.15.60:0/2004351253 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c031780 con 0x7f632406c810 2023-10-16T14:10:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.545+0000 7f6328bc7640 1 -- 172.21.15.60:0/2004351253 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c0318e0 con 0x7f632406c810 2023-10-16T14:10:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.545+0000 7f6328bc7640 1 -- 172.21.15.60:0/2004351253 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c031c70 con 0x7f632406c810 2023-10-16T14:10:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.545+0000 7f632abcb640 1 -- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f632406c810 msgr2=0x7f63240708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.545+0000 7f632abcb640 1 --2- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f632406c810 0x7f63240708b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f631c0097a0 tx=0x7f631c02b820 comp rx=0 tx=0).stop 2023-10-16T14:10:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.546+0000 7f632abcb640 1 -- 172.21.15.60:0/2004351253 shutdown_connections 2023-10-16T14:10:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.546+0000 7f632abcb640 1 --2- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f632406c810 0x7f63240708b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.546+0000 7f632abcb640 1 --2- 172.21.15.60:0/2004351253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6324159950 0x7f632415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.546+0000 7f632abcb640 1 -- 172.21.15.60:0/2004351253 >> 172.21.15.60:0/2004351253 conn(0x7f63240ce6a0 msgr2=0x7f63240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.546+0000 7f632abcb640 1 -- 172.21.15.60:0/2004351253 shutdown_connections 2023-10-16T14:10:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.546+0000 7f632abcb640 1 -- 172.21.15.60:0/2004351253 wait complete. 2023-10-16T14:10:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.547+0000 7f632abcb640 1 Processor -- start 2023-10-16T14:10:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.548+0000 7f632abcb640 1 -- start start 2023-10-16T14:10:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632abcb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f632406c810 0x7f632415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632abcb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 0x7f6324146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632abcb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6324159950 0x7f63240c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632abcb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632414a0a0 con 0x7f6324070450 2023-10-16T14:10:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632abcb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63241413a0 con 0x7f6324159950 2023-10-16T14:10:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632abcb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6324152860 con 0x7f632406c810 2023-10-16T14:10:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 0x7f6324146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 0x7f6324146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43390/0 (socket says 172.21.15.60:43390) 2023-10-16T14:10:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 -- 172.21.15.60:0/55504525 learned_addr learned my addr 172.21.15.60:0/55504525 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f6329bc9640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f632406c810 0x7f632415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f632a3ca640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6324159950 0x7f63240c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 -- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f632406c810 msgr2=0x7f632415ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f632406c810 0x7f632415ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 -- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6324159950 msgr2=0x7f63240c0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.549+0000 7f63293c8640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6324159950 0x7f63240c0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f63293c8640 1 -- 172.21.15.60:0/55504525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6320009a70 con 0x7f6324070450 2023-10-16T14:10:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f632a3ca640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6324159950 0x7f63240c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f63293c8640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 0x7f6324146f70 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f631400c4b0 tx=0x7f631400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314005ce0 con 0x7f6324070450 2023-10-16T14:10:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314005e40 con 0x7f6324070450 2023-10-16T14:10:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314007270 con 0x7f6324070450 2023-10-16T14:10:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c009400 con 0x7f6324070450 2023-10-16T14:10:09.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.550+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6324151aa0 con 0x7f6324070450 2023-10-16T14:10:09.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.551+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632404e9b0 con 0x7f6324070450 2023-10-16T14:10:09.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.552+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 125) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f63140073d0 con 0x7f6324070450 2023-10-16T14:10:09.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.553+0000 7f631a7fc640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f630806acc0 0x7f630806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:09.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.553+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f6314088a90 con 0x7f6324070450 2023-10-16T14:10:09.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.554+0000 7f6329bc9640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f630806acc0 0x7f630806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:09.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.555+0000 7f6329bc9640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f630806acc0 0x7f630806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6320007600 tx=0x7f632000b040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:09.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.558+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6314053b50 con 0x7f6324070450 2023-10-16T14:10:09.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.785+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63240b4620 con 0x7f6324070450 2023-10-16T14:10:09.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.786+0000 7f631a7fc640 1 -- 172.21.15.60:0/55504525 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f6314017090 con 0x7f6324070450 2023-10-16T14:10:09.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:09.790 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:07:44.252784+0000","modified":"2023-10-16T14:07:47.343844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8539":{"gid":8539,"name":"b","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6835/3999834969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3999834969},{"type":"v1","addr":"172.21.15.144:6835","nonce":3999834969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T14:10:09.791 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 104 2023-10-16T14:10:09.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f630806acc0 msgr2=0x7f630806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f630806acc0 0x7f630806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6320007600 tx=0x7f632000b040 comp rx=0 tx=0).stop 2023-10-16T14:10:09.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 msgr2=0x7f6324146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:09.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 0x7f6324146f70 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f631400c4b0 tx=0x7f631400c970 comp rx=0 tx=0).stop 2023-10-16T14:10:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 shutdown_connections 2023-10-16T14:10:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f630806acc0 0x7f630806d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6324159950 0x7f63240c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6324070450 0x7f6324146f70 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 --2- 172.21.15.60:0/55504525 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f632406c810 0x7f632415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 >> 172.21.15.60:0/55504525 conn(0x7f63240ce6a0 msgr2=0x7f6324152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 shutdown_connections 2023-10-16T14:10:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:09.789+0000 7f632abcb640 1 -- 172.21.15.60:0/55504525 wait complete. 2023-10-16T14:10:09.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:10.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.025+0000 7f26e64bb640 1 Processor -- start 2023-10-16T14:10:10.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.026+0000 7f26e64bb640 1 -- start start 2023-10-16T14:10:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e64bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 0x7f26e00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e64bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e0160a90 con 0x7f26e015dde0 2023-10-16T14:10:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e64bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00cd900 con 0x7f26e00c9230 2023-10-16T14:10:10.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e64bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00cdec0 con 0x7f26e015c9f0 2023-10-16T14:10:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e5cba640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f26e015dde0 0x7f26e00ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52882/0 (socket says 172.21.15.60:52882) 2023-10-16T14:10:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e5cba640 1 -- 172.21.15.60:0/3204060587 learned_addr learned my addr 172.21.15.60:0/3204060587 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.027+0000 7f26e54b9640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 0x7f26e00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106635773 0 0) 0x7f26e0160a90 con 0x7f26e015dde0 2023-10-16T14:10:10.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26cc003520 con 0x7f26e015dde0 2023-10-16T14:10:10.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5859516 0 0) 0x7f26cc003520 con 0x7f26e015dde0 2023-10-16T14:10:10.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 msgr2=0x7f26e00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 0x7f26e00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 >> v1:172.21.15.144:6789/0 conn(0x7f26e00c9230 legacy=0x7f26e00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00cdbe0 con 0x7f26e015dde0 2023-10-16T14:10:10.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f26d80019f0 con 0x7f26e015dde0 2023-10-16T14:10:10.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 >> v1:172.21.15.60:6789/0 conn(0x7f26e015dde0 legacy=0x7f26e00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 0x7f26cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26cc004d00 0x7f26cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc00ba30 con 0x7f26cc0039a0 2023-10-16T14:10:10.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc00bba0 con 0x7f26cc004d00 2023-10-16T14:10:10.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.028+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc00bd60 con 0x7f26e015c9f0 2023-10-16T14:10:10.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.029+0000 7f26e54b9640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.029+0000 7f26e54b9640 1 -- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 msgr2=0x7f26cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.029+0000 7f26e54b9640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e015c9f0 0x7f26cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.029+0000 7f26e54b9640 1 -- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26cc004d00 msgr2=0x7f26cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.029+0000 7f26e54b9640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26cc004d00 0x7f26cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.029+0000 7f26e54b9640 1 -- 172.21.15.60:0/3204060587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc003520 con 0x7f26cc0039a0 2023-10-16T14:10:10.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.030+0000 7f26e54b9640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26cc003dc0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009a90 tx=0x7f26cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe3da0690706674 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.030+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc02c6b0 con 0x7f26cc0039a0 2023-10-16T14:10:10.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.030+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc02c810 con 0x7f26cc0039a0 2023-10-16T14:10:10.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.030+0000 7f26d3fff640 1 -- 172.21.15.60:0/3204060587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc02cba0 con 0x7f26cc0039a0 2023-10-16T14:10:10.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e64bb640 1 -- 172.21.15.60:0/3204060587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 msgr2=0x7f26cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e64bb640 1 --2- 172.21.15.60:0/3204060587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26cc003dc0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009a90 tx=0x7f26cc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e5cba640 1 -- 172.21.15.60:0/3204060587 reap_dead start 2023-10-16T14:10:10.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e64bb640 1 -- 172.21.15.60:0/3204060587 shutdown_connections 2023-10-16T14:10:10.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e64bb640 1 -- 172.21.15.60:0/3204060587 >> 172.21.15.60:0/3204060587 conn(0x7f26e00ce6a0 msgr2=0x7f26e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:10.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e64bb640 1 -- 172.21.15.60:0/3204060587 shutdown_connections 2023-10-16T14:10:10.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.031+0000 7f26e64bb640 1 -- 172.21.15.60:0/3204060587 wait complete. 2023-10-16T14:10:10.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.033+0000 7f26e64bb640 1 Processor -- start 2023-10-16T14:10:10.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.033+0000 7f26e64bb640 1 -- start start 2023-10-16T14:10:10.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e64bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26e006c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e64bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00c5770 0x7f26e00c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e64bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e014b650 0x7f26e014bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e64bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e014c010 con 0x7f26cc0039a0 2023-10-16T14:10:10.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e64bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00c3fa0 con 0x7f26e014b650 2023-10-16T14:10:10.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e64bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00c4280 con 0x7f26e00c5770 2023-10-16T14:10:10.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26e006c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26e006c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43408/0 (socket says 172.21.15.60:43408) 2023-10-16T14:10:10.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 -- 172.21.15.60:0/2945636317 learned_addr learned my addr 172.21.15.60:0/2945636317 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:10.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e4cb8640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00c5770 0x7f26e00c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 -- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00c5770 msgr2=0x7f26e00c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00c5770 0x7f26e00c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 -- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e014b650 msgr2=0x7f26e014bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e014b650 0x7f26e014bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.034+0000 7f26e54b9640 1 -- 172.21.15.60:0/2945636317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00c4560 con 0x7f26cc0039a0 2023-10-16T14:10:10.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.035+0000 7f26e54b9640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26e006c6c0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f26dc02c250 tx=0x7f26dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.035+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc003810 con 0x7f26cc0039a0 2023-10-16T14:10:10.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.035+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc003970 con 0x7f26cc0039a0 2023-10-16T14:10:10.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.035+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e00c4840 con 0x7f26cc0039a0 2023-10-16T14:10:10.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.035+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc003d00 con 0x7f26cc0039a0 2023-10-16T14:10:10.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.035+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e006cc00 con 0x7f26cc0039a0 2023-10-16T14:10:10.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.036+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e004e9b0 con 0x7f26cc0039a0 2023-10-16T14:10:10.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.037+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 125) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f26dc034440 con 0x7f26cc0039a0 2023-10-16T14:10:10.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.038+0000 7f26d1ffb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f26c406acf0 0x7f26c406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.038+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f26dc0ab870 con 0x7f26cc0039a0 2023-10-16T14:10:10.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.039+0000 7f26e4cb8640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f26c406acf0 0x7f26c406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.040+0000 7f26e4cb8640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f26c406acf0 0x7f26c406d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f26e00af550 tx=0x7f26d4009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.042+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26dc0769c0 con 0x7f26cc0039a0 2023-10-16T14:10:10.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.171+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 126) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f26dc0adcf0 con 0x7f26cc0039a0 2023-10-16T14:10:10.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.238+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e00adac0 con 0x7f26cc0039a0 2023-10-16T14:10:10.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.241+0000 7f26d1ffb640 1 -- 172.21.15.60:0/2945636317 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+17687 (secure 0 0 0) 0x7f26dc07d6e0 con 0x7f26cc0039a0 2023-10-16T14:10:10.245 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:10.245 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":219,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:08.132864+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":218,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":1}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/652243704":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/2452241572":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/1049005214":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/4233146004":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/732610814":"2023-10-17T14:10:08.132792+0000","172.21.15.144:6833/166679164":"2023-10-17T14:10:08.132792+0000","172.21.15.144:6832/166679164":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/351085041":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/992081872":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/4042889662":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/301721702":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/166629007":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6832/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2940276091":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6833/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2969783877":"2023-10-17T14:07:58.997006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f26c406acf0 msgr2=0x7f26c406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f26c406acf0 0x7f26c406d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f26e00af550 tx=0x7f26d4009040 comp rx=0 tx=0).stop 2023-10-16T14:10:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 msgr2=0x7f26e006c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26e006c6c0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f26dc02c250 tx=0x7f26dc004730 comp rx=0 tx=0).stop 2023-10-16T14:10:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 shutdown_connections 2023-10-16T14:10:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f26c406acf0 0x7f26c406d1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f26e00c5770 0x7f26e00c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e014b650 0x7f26e014bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 --2- 172.21.15.60:0/2945636317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26cc0039a0 0x7f26e006c6c0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 >> 172.21.15.60:0/2945636317 conn(0x7f26e00ce6a0 msgr2=0x7f26e00c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:10.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 shutdown_connections 2023-10-16T14:10:10.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.244+0000 7f26e64bb640 1 -- 172.21.15.60:0/2945636317 wait complete. 2023-10-16T14:10:10.268 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:10:10.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:10.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.483+0000 7f5f73fff640 1 Processor -- start 2023-10-16T14:10:10.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.483+0000 7f5f73fff640 1 -- start start 2023-10-16T14:10:10.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f73fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 0x7f5f7415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f73fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f74070450 0x7f5f740708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f73fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f74070df0 con 0x7f5f7406c100 2023-10-16T14:10:10.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f73fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7415e490 con 0x7f5f74070450 2023-10-16T14:10:10.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f73fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7415aef0 con 0x7f5f74159950 2023-10-16T14:10:10.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f737fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5f7406c100 0x7f5f7406c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52890/0 (socket says 172.21.15.60:52890) 2023-10-16T14:10:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.485+0000 7f5f737fe640 1 -- 172.21.15.60:0/3750912742 learned_addr learned my addr 172.21.15.60:0/3750912742 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f72ffd640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 0x7f5f7415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f6bfff640 1 --2- 172.21.15.60:0/3750912742 >> v2:172.21.15.144:3300/0 conn(0x7f5f74070450 0x7f5f740708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169487621 0 0) 0x7f5f74070df0 con 0x7f5f7406c100 2023-10-16T14:10:10.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60003520 con 0x7f5f7406c100 2023-10-16T14:10:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3325011606 0 0) 0x7f5f60003520 con 0x7f5f7406c100 2023-10-16T14:10:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 msgr2=0x7f5f7415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 0x7f5f7415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 >> v2:172.21.15.144:3300/0 conn(0x7f5f74070450 msgr2=0x7f5f740708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 --2- 172.21.15.60:0/3750912742 >> v2:172.21.15.144:3300/0 conn(0x7f5f74070450 0x7f5f740708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7406cc80 con 0x7f5f7406c100 2023-10-16T14:10:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f5f6c0019f0 con 0x7f5f7406c100 2023-10-16T14:10:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 >> v1:172.21.15.60:6789/0 conn(0x7f5f7406c100 legacy=0x7f5f7406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f60003970 0x7f5f60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.486+0000 7f5f727fc640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f600047c0 0x7f5f60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f727fc640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 0x7f5f60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7415e490 con 0x7f5f600047c0 2023-10-16T14:10:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7415aef0 con 0x7f5f60003970 2023-10-16T14:10:10.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6000dbe0 con 0x7f5f74159950 2023-10-16T14:10:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f6bfff640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f600047c0 0x7f5f60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f72ffd640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f60003970 0x7f5f60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f6bfff640 1 -- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 msgr2=0x7f5f60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f6bfff640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f74159950 0x7f5f60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f6bfff640 1 -- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f60003970 msgr2=0x7f5f60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f6bfff640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f60003970 0x7f5f60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f6bfff640 1 -- 172.21.15.60:0/3750912742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f60003520 con 0x7f5f600047c0 2023-10-16T14:10:10.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.487+0000 7f5f72ffd640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f60003970 0x7f5f60003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:10.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f6bfff640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f600047c0 0x7f5f60004c40 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009af0 tx=0x7f5f600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b17e988b16c110d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c02c710 con 0x7f5f600047c0 2023-10-16T14:10:10.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c02c870 con 0x7f5f600047c0 2023-10-16T14:10:10.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f727fc640 1 -- 172.21.15.60:0/3750912742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c02cc00 con 0x7f5f600047c0 2023-10-16T14:10:10.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f73fff640 1 -- 172.21.15.60:0/3750912742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f600047c0 msgr2=0x7f5f60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3750912742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f600047c0 0x7f5f60004c40 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009af0 tx=0x7f5f600057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.489+0000 7f5f737fe640 1 -- 172.21.15.60:0/3750912742 reap_dead start 2023-10-16T14:10:10.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.490+0000 7f5f73fff640 1 -- 172.21.15.60:0/3750912742 shutdown_connections 2023-10-16T14:10:10.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.490+0000 7f5f73fff640 1 -- 172.21.15.60:0/3750912742 >> 172.21.15.60:0/3750912742 conn(0x7f5f740ce6a0 msgr2=0x7f5f740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:10.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.490+0000 7f5f73fff640 1 -- 172.21.15.60:0/3750912742 shutdown_connections 2023-10-16T14:10:10.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.490+0000 7f5f73fff640 1 -- 172.21.15.60:0/3750912742 wait complete. 2023-10-16T14:10:10.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.491+0000 7f5f73fff640 1 Processor -- start 2023-10-16T14:10:10.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.492+0000 7f5f73fff640 1 -- start start 2023-10-16T14:10:10.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.492+0000 7f5f73fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f600047c0 0x7f5f740b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.492+0000 7f5f73fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f74157e90 0x7f5f741f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f73fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 0x7f5f740cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f73fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740ce060 con 0x7f5f740cd6a0 2023-10-16T14:10:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f73fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f74158310 con 0x7f5f74157e90 2023-10-16T14:10:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f73fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740b19c0 con 0x7f5f600047c0 2023-10-16T14:10:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 0x7f5f740cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 0x7f5f740cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43426/0 (socket says 172.21.15.60:43426) 2023-10-16T14:10:10.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 -- 172.21.15.60:0/3602510289 learned_addr learned my addr 172.21.15.60:0/3602510289 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:10.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f72ffd640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f600047c0 0x7f5f740b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f6bfff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f74157e90 0x7f5f741f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 -- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f600047c0 msgr2=0x7f5f740b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f600047c0 0x7f5f740b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 -- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f74157e90 msgr2=0x7f5f741f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f74157e90 0x7f5f741f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.493+0000 7f5f737fe640 1 -- 172.21.15.60:0/3602510289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740b1ca0 con 0x7f5f740cd6a0 2023-10-16T14:10:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.494+0000 7f5f737fe640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 0x7f5f740cdb20 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f5f740ca950 tx=0x7f5f6c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.494+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00ceb0 con 0x7f5f740cd6a0 2023-10-16T14:10:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.494+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00a610 con 0x7f5f740cd6a0 2023-10-16T14:10:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.494+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00a9a0 con 0x7f5f740cd6a0 2023-10-16T14:10:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.494+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f740b1f80 con 0x7f5f740cd6a0 2023-10-16T14:10:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.494+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f74144a00 con 0x7f5f740cd6a0 2023-10-16T14:10:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.495+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7404e9b0 con 0x7f5f740cd6a0 2023-10-16T14:10:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.497+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 126) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5f6c00ab00 con 0x7f5f740cd6a0 2023-10-16T14:10:10.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.498+0000 7f5f6b7fe640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5f5406af00 0x7f5f5406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.498+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f5f6c089280 con 0x7f5f740cd6a0 2023-10-16T14:10:10.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.498+0000 7f5f72ffd640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5f5406af00 0x7f5f5406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.500+0000 7f5f72ffd640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5f5406af00 0x7f5f5406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5f640099c0 tx=0x7f5f64008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.502+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f6c0579c0 con 0x7f5f740cd6a0 2023-10-16T14:10:10.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.736+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f740cc250 con 0x7f5f740cd6a0 2023-10-16T14:10:10.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.738+0000 7f5f6b7fe640 1 -- 172.21.15.60:0/3602510289 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f5f6c019090 con 0x7f5f740cd6a0 2023-10-16T14:10:10.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:10.742 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:07:44.252784+0000","modified":"2023-10-16T14:07:47.343844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8539":{"gid":8539,"name":"b","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6835/3999834969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3999834969},{"type":"v1","addr":"172.21.15.144:6835","nonce":3999834969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T14:10:10.743 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 104 2023-10-16T14:10:10.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5f5406af00 msgr2=0x7f5f5406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5f5406af00 0x7f5f5406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5f640099c0 tx=0x7f5f64008040 comp rx=0 tx=0).stop 2023-10-16T14:10:10.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 msgr2=0x7f5f740cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 0x7f5f740cdb20 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f5f740ca950 tx=0x7f5f6c00cc00 comp rx=0 tx=0).stop 2023-10-16T14:10:10.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 shutdown_connections 2023-10-16T14:10:10.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5f5406af00 0x7f5f5406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f740cd6a0 0x7f5f740cdb20 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f74157e90 0x7f5f741f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 --2- 172.21.15.60:0/3602510289 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f600047c0 0x7f5f740b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 >> 172.21.15.60:0/3602510289 conn(0x7f5f740ce6a0 msgr2=0x7f5f740c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:10.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 shutdown_connections 2023-10-16T14:10:10.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.741+0000 7f5f73fff640 1 -- 172.21.15.60:0/3602510289 wait complete. 2023-10-16T14:10:10.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:10.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.975+0000 7f81bb485640 1 Processor -- start 2023-10-16T14:10:10.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.975+0000 7f81bb485640 1 -- start start 2023-10-16T14:10:10.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.976+0000 7f81bb485640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f81b415cd70 0x7f81b40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.976+0000 7f81bb485640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f81b40b8480 0x7f81b40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.976+0000 7f81bb485640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b415d150 con 0x7f81b40d0be0 2023-10-16T14:10:10.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.976+0000 7f81bb485640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40d1180 con 0x7f81b40b8480 2023-10-16T14:10:10.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.976+0000 7f81bb485640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40d1460 con 0x7f81b415cd70 2023-10-16T14:10:10.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81bac84640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f81b40d0be0 0x7f81b40b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52906/0 (socket says 172.21.15.60:52906) 2023-10-16T14:10:10.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81bac84640 1 -- 172.21.15.60:0/2921702097 learned_addr learned my addr 172.21.15.60:0/2921702097 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:10.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81b9c82640 1 --2- 172.21.15.60:0/2921702097 >> v2:172.21.15.60:3300/0 conn(0x7f81b40b8480 0x7f81b40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81ba483640 1 --2- 172.21.15.60:0/2921702097 >> v2:172.21.15.144:3300/0 conn(0x7f81b415cd70 0x7f81b40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81b9c82640 1 -- 172.21.15.60:0/2921702097 >> v2:172.21.15.144:3300/0 conn(0x7f81b415cd70 msgr2=0x7f81b40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:10.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81b9c82640 1 --2- 172.21.15.60:0/2921702097 >> v2:172.21.15.144:3300/0 conn(0x7f81b415cd70 0x7f81b40d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81b9c82640 1 -- 172.21.15.60:0/2921702097 >> v1:172.21.15.60:6789/0 conn(0x7f81b40d0be0 legacy=0x7f81b40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.977+0000 7f81b9c82640 1 -- 172.21.15.60:0/2921702097 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40d1740 con 0x7f81b40b8480 2023-10-16T14:10:10.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.978+0000 7f81b9c82640 1 --2- 172.21.15.60:0/2921702097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b40b8480 0x7f81b40cd7c0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f81b40b56a0 tx=0x7f81a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92569aa79e2ecc91 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81b9481640 1 -- 172.21.15.60:0/2921702097 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a402e070 con 0x7f81b40b8480 2023-10-16T14:10:10.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81b9481640 1 -- 172.21.15.60:0/2921702097 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a40315a0 con 0x7f81b40b8480 2023-10-16T14:10:10.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81b9481640 1 -- 172.21.15.60:0/2921702097 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4031930 con 0x7f81b40b8480 2023-10-16T14:10:10.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 -- 172.21.15.60:0/2921702097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b40b8480 msgr2=0x7f81b40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 --2- 172.21.15.60:0/2921702097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b40b8480 0x7f81b40cd7c0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f81b40b56a0 tx=0x7f81a402b840 comp rx=0 tx=0).stop 2023-10-16T14:10:10.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 -- 172.21.15.60:0/2921702097 shutdown_connections 2023-10-16T14:10:10.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 --2- 172.21.15.60:0/2921702097 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b40b8480 0x7f81b40cd7c0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 --2- 172.21.15.60:0/2921702097 >> v2:172.21.15.144:3300/0 conn(0x7f81b415cd70 0x7f81b40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 -- 172.21.15.60:0/2921702097 >> 172.21.15.60:0/2921702097 conn(0x7f81b40cf570 msgr2=0x7f81b40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:10.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 -- 172.21.15.60:0/2921702097 shutdown_connections 2023-10-16T14:10:10.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.979+0000 7f81bb485640 1 -- 172.21.15.60:0/2921702097 wait complete. 2023-10-16T14:10:10.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.981+0000 7f81bb485640 1 Processor -- start 2023-10-16T14:10:10.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.981+0000 7f81bb485640 1 -- start start 2023-10-16T14:10:10.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bb485640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b40b8480 0x7f81b41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bb485640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b40d0be0 0x7f81b40cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bb485640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 0x7f81b4148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bb485640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b41551f0 con 0x7f81b415cd70 2023-10-16T14:10:10.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bb485640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4152f80 con 0x7f81b40b8480 2023-10-16T14:10:10.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bb485640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4149280 con 0x7f81b40d0be0 2023-10-16T14:10:10.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.982+0000 7f81bac84640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 0x7f81b4148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 0x7f81b4148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43456/0 (socket says 172.21.15.60:43456) 2023-10-16T14:10:10.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 -- 172.21.15.60:0/632058904 learned_addr learned my addr 172.21.15.60:0/632058904 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:10.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81ba483640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b40b8480 0x7f81b41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 -- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b40d0be0 msgr2=0x7f81b40cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81b9c82640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b40d0be0 0x7f81b40cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b40d0be0 0x7f81b40cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 -- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b40b8480 msgr2=0x7f81b41529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:10.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b40b8480 0x7f81b41529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:10.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 -- 172.21.15.60:0/632058904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4009400 con 0x7f81b415cd70 2023-10-16T14:10:10.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81ba483640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b40b8480 0x7f81b41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:10.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81b9c82640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b40d0be0 0x7f81b40cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:10.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.983+0000 7f81bac84640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 0x7f81b4148d40 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00ca70 tx=0x7f81ac00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.984+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00abd0 con 0x7f81b415cd70 2023-10-16T14:10:10.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.984+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac00ad30 con 0x7f81b415cd70 2023-10-16T14:10:10.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.984+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b4155490 con 0x7f81b415cd70 2023-10-16T14:10:10.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.984+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac012600 con 0x7f81b415cd70 2023-10-16T14:10:10.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.984+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b4155a50 con 0x7f81b415cd70 2023-10-16T14:10:10.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.985+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b4155c40 con 0x7f81b415cd70 2023-10-16T14:10:10.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.986+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 126) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f81ac007070 con 0x7f81b415cd70 2023-10-16T14:10:10.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.987+0000 7f81aaffd640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f819806af00 0x7f819806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:10.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.987+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f81ac0897d0 con 0x7f81b415cd70 2023-10-16T14:10:10.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.988+0000 7f81ba483640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f819806af00 0x7f819806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:10.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.989+0000 7f81ba483640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f819806af00 0x7f819806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009ac0 tx=0x7f81b0008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:10.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:10.991+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81ac054530 con 0x7f81b415cd70 2023-10-16T14:10:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.229+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b404e9b0 con 0x7f81b415cd70 2023-10-16T14:10:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.230+0000 7f81aaffd640 1 -- 172.21.15.60:0/632058904 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f81ac057b50 con 0x7f81b415cd70 2023-10-16T14:10:11.234 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:11.234 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:07:44.252784+0000","modified":"2023-10-16T14:07:47.343844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8539":{"gid":8539,"name":"b","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6835/3999834969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":3999834969},{"type":"v1","addr":"172.21.15.144:6835","nonce":3999834969}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T14:10:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 104 2023-10-16T14:10:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f819806af00 msgr2=0x7f819806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f819806af00 0x7f819806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009ac0 tx=0x7f81b0008040 comp rx=0 tx=0).stop 2023-10-16T14:10:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 msgr2=0x7f81b4148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 0x7f81b4148d40 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00ca70 tx=0x7f81ac00cf30 comp rx=0 tx=0).stop 2023-10-16T14:10:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 shutdown_connections 2023-10-16T14:10:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f819806af00 0x7f819806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81b40d0be0 0x7f81b40cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b415cd70 0x7f81b4148d40 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 --2- 172.21.15.60:0/632058904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b40b8480 0x7f81b41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 >> 172.21.15.60:0/632058904 conn(0x7f81b40cf570 msgr2=0x7f81b415bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 shutdown_connections 2023-10-16T14:10:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.233+0000 7f81bb485640 1 -- 172.21.15.60:0/632058904 wait complete. 2023-10-16T14:10:11.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:11.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.464+0000 7f25c1328640 1 Processor -- start 2023-10-16T14:10:11.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.464+0000 7f25c1328640 1 -- start start 2023-10-16T14:10:11.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.465+0000 7f25c1328640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 0x7f25bc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.465+0000 7f25c1328640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f25bc0ad970 0x7f25bc15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25c1328640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc06c960 con 0x7f25bc15ebc0 2023-10-16T14:10:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25c1328640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc15b9d0 con 0x7f25bc0ad970 2023-10-16T14:10:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25c1328640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc15bcb0 con 0x7f25bc06c040 2023-10-16T14:10:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25c0b27640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f25bc15ebc0 0x7f25bc0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52914/0 (socket says 172.21.15.60:52914) 2023-10-16T14:10:11.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25c0b27640 1 -- 172.21.15.60:0/1289324683 learned_addr learned my addr 172.21.15.60:0/1289324683 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/1289324683 >> v2:172.21.15.144:3300/0 conn(0x7f25bc0ad970 0x7f25bc15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.466+0000 7f25bbfff640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 0x7f25bc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213853630 0 0) 0x7f25bc06c960 con 0x7f25bc15ebc0 2023-10-16T14:10:11.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a4003520 con 0x7f25bc15ebc0 2023-10-16T14:10:11.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 619325997 0 0) 0x7f25a4003520 con 0x7f25bc15ebc0 2023-10-16T14:10:11.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 msgr2=0x7f25bc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:11.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 0x7f25bc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 >> v2:172.21.15.144:3300/0 conn(0x7f25bc0ad970 msgr2=0x7f25bc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 --2- 172.21.15.60:0/1289324683 >> v2:172.21.15.144:3300/0 conn(0x7f25bc0ad970 0x7f25bc15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc15bf90 con 0x7f25bc15ebc0 2023-10-16T14:10:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f25b40019f0 con 0x7f25bc15ebc0 2023-10-16T14:10:11.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 >> v1:172.21.15.60:6789/0 conn(0x7f25bc15ebc0 legacy=0x7f25bc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 0x7f25a4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25a4003ef0 0x7f25a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25a4004d00 0x7f25a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc15b9d0 con 0x7f25a4003ef0 2023-10-16T14:10:11.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc15bcb0 con 0x7f25a4004d00 2023-10-16T14:10:11.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a400b980 con 0x7f25bc06c040 2023-10-16T14:10:11.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.467+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/1289324683 >> v2:172.21.15.144:3300/0 conn(0x7f25bc0ad970 0x7f25bc15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:10:11.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25a4003ef0 0x7f25a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25c0b27640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25a4004d00 0x7f25a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bbfff640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 0x7f25a4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bb7fe640 1 -- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 msgr2=0x7f25a4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:11.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25bc06c040 0x7f25a4003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bb7fe640 1 -- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25a4004d00 msgr2=0x7f25a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:11.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25a4004d00 0x7f25a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25bb7fe640 1 -- 172.21.15.60:0/1289324683 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4003520 con 0x7f25a4003ef0 2023-10-16T14:10:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.468+0000 7f25c0b27640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25a4004d00 0x7f25a4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.469+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25a4003ef0 0x7f25a40047c0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009c70 tx=0x7f25a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed7e1c662f5593 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.469+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0004020 con 0x7f25a4003ef0 2023-10-16T14:10:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0004180 con 0x7f25a4003ef0 2023-10-16T14:10:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25baffd640 1 -- 172.21.15.60:0/1289324683 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b002c950 con 0x7f25a4003ef0 2023-10-16T14:10:11.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c1328640 1 -- 172.21.15.60:0/1289324683 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25a4003ef0 msgr2=0x7f25a40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c1328640 1 --2- 172.21.15.60:0/1289324683 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25a4003ef0 0x7f25a40047c0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009c70 tx=0x7f25a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c0b27640 1 -- 172.21.15.60:0/1289324683 reap_dead start 2023-10-16T14:10:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c1328640 1 -- 172.21.15.60:0/1289324683 shutdown_connections 2023-10-16T14:10:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c1328640 1 -- 172.21.15.60:0/1289324683 >> 172.21.15.60:0/1289324683 conn(0x7f25bc0ce6a0 msgr2=0x7f25bc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c1328640 1 -- 172.21.15.60:0/1289324683 shutdown_connections 2023-10-16T14:10:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.470+0000 7f25c1328640 1 -- 172.21.15.60:0/1289324683 wait complete. 2023-10-16T14:10:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.472+0000 7f25c1328640 1 Processor -- start 2023-10-16T14:10:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.472+0000 7f25c1328640 1 -- start start 2023-10-16T14:10:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.473+0000 7f25c1328640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25a4003ef0 0x7f25bc0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.473+0000 7f25c1328640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 0x7f25bc14ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.473+0000 7f25c1328640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25bc0ac890 0x7f25bc0acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.473+0000 7f25c1328640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc0c8620 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.473+0000 7f25c1328640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc0ad260 con 0x7f25bc0ac890 2023-10-16T14:10:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.473+0000 7f25c1328640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc0c7c90 con 0x7f25a4003ef0 2023-10-16T14:10:11.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 0x7f25bc14ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 0x7f25bc14ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43462/0 (socket says 172.21.15.60:43462) 2023-10-16T14:10:11.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 -- 172.21.15.60:0/2101510639 learned_addr learned my addr 172.21.15.60:0/2101510639 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:11.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bbfff640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25a4003ef0 0x7f25bc0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25c0b27640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25bc0ac890 0x7f25bc0acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 -- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25a4003ef0 msgr2=0x7f25bc0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25a4003ef0 0x7f25bc0c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 -- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25bc0ac890 msgr2=0x7f25bc0acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25bc0ac890 0x7f25bc0acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 -- 172.21.15.60:0/2101510639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0ae7a0 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.474+0000 7f25bb7fe640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 0x7f25bc14ea40 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009b90 tx=0x7f25b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.475+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b002cb60 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.475+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b002ccc0 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.475+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0006260 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.475+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc0c9180 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.475+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc0c9a20 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.476+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25bc06b330 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.477+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 126) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f25b00063c0 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.478+0000 7f25b8ff9640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f25a806af00 0x7f25a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.478+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6483+0+0 (secure 0 0 0) 0x7f25b00abec0 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.479+0000 7f25bbfff640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f25a806af00 0x7f25a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.480+0000 7f25bbfff640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f25a806af00 0x7f25a806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009930 tx=0x7f25ac008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:11.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.482+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f25b0076d90 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.679+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25bc04e9b0 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.683+0000 7f25b8ff9640 1 -- 172.21.15.60:0/2101510639 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+17687 (secure 0 0 0) 0x7f25b0043090 con 0x7f25bc0c8cf0 2023-10-16T14:10:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:11.687 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":219,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:08.132864+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:07:42.198213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-16T14:07:43.287022+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":218,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.11","mappings":[{"from":0,"to":1}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/652243704":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/2452241572":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/1049005214":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/4233146004":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/732610814":"2023-10-17T14:10:08.132792+0000","172.21.15.144:6833/166679164":"2023-10-17T14:10:08.132792+0000","172.21.15.144:6832/166679164":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/351085041":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/992081872":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/4042889662":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/301721702":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/166629007":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6832/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2940276091":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6833/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2969783877":"2023-10-17T14:07:58.997006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f25a806af00 msgr2=0x7f25a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f25a806af00 0x7f25a806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009930 tx=0x7f25ac008040 comp rx=0 tx=0).stop 2023-10-16T14:10:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 msgr2=0x7f25bc14ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 0x7f25bc14ea40 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009b90 tx=0x7f25b0004730 comp rx=0 tx=0).stop 2023-10-16T14:10:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 shutdown_connections 2023-10-16T14:10:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f25a806af00 0x7f25a806d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25bc0ac890 0x7f25bc0acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25bc0c8cf0 0x7f25bc14ea40 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 --2- 172.21.15.60:0/2101510639 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f25a4003ef0 0x7f25bc0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.685+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 >> 172.21.15.60:0/2101510639 conn(0x7f25bc0ce6a0 msgr2=0x7f25bc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.686+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 shutdown_connections 2023-10-16T14:10:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.686+0000 7f25c1328640 1 -- 172.21.15.60:0/2101510639 wait complete. 2023-10-16T14:10:11.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:10:11.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.931+0000 7fd48671d640 1 Processor -- start 2023-10-16T14:10:11.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.931+0000 7fd48671d640 1 -- start start 2023-10-16T14:10:11.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd48671d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd4800cf5f0 0x7fd48006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd48671d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd480070c10 0x7fd4800cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd48671d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd480161740 con 0x7fd480070830 2023-10-16T14:10:11.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd48671d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4800cdbd0 con 0x7fd480070c10 2023-10-16T14:10:11.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd48671d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48006b0f0 con 0x7fd4800cf5f0 2023-10-16T14:10:11.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd47ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd480070830 0x7fd48006c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52930/0 (socket says 172.21.15.60:52930) 2023-10-16T14:10:11.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd47ffff640 1 -- 172.21.15.60:0/2496613934 learned_addr learned my addr 172.21.15.60:0/2496613934 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:11.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd484cba640 1 --2- 172.21.15.60:0/2496613934 >> v2:172.21.15.60:3300/0 conn(0x7fd480070c10 0x7fd4800cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.933+0000 7fd4777fe640 1 --2- 172.21.15.60:0/2496613934 >> v2:172.21.15.144:3300/0 conn(0x7fd4800cf5f0 0x7fd48006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.934+0000 7fd47f7fe640 1 -- 172.21.15.60:0/2496613934 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000836742 0 0) 0x7fd480161740 con 0x7fd480070830 2023-10-16T14:10:11.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.934+0000 7fd47f7fe640 1 -- 172.21.15.60:0/2496613934 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c003520 con 0x7fd480070830 2023-10-16T14:10:11.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.934+0000 7fd484cba640 1 -- 172.21.15.60:0/2496613934 >> v2:172.21.15.144:3300/0 conn(0x7fd4800cf5f0 msgr2=0x7fd48006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:11.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.934+0000 7fd484cba640 1 --2- 172.21.15.60:0/2496613934 >> v2:172.21.15.144:3300/0 conn(0x7fd4800cf5f0 0x7fd48006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.934+0000 7fd484cba640 1 -- 172.21.15.60:0/2496613934 >> v1:172.21.15.60:6789/0 conn(0x7fd480070830 legacy=0x7fd48006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.934+0000 7fd484cba640 1 -- 172.21.15.60:0/2496613934 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4801619b0 con 0x7fd480070c10 2023-10-16T14:10:11.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.935+0000 7fd484cba640 1 --2- 172.21.15.60:0/2496613934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070c10 0x7fd4800cd690 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fd4800cbee0 tx=0x7fd47802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94a9359f8f91564 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:11.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.935+0000 7fd47f7fe640 1 -- 172.21.15.60:0/2496613934 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47802cb60 con 0x7fd480070c10 2023-10-16T14:10:11.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.935+0000 7fd47f7fe640 1 -- 172.21.15.60:0/2496613934 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47802ccc0 con 0x7fd480070c10 2023-10-16T14:10:11.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.935+0000 7fd47f7fe640 1 -- 172.21.15.60:0/2496613934 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478034640 con 0x7fd480070c10 2023-10-16T14:10:11.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 -- 172.21.15.60:0/2496613934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070c10 msgr2=0x7fd4800cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 --2- 172.21.15.60:0/2496613934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070c10 0x7fd4800cd690 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fd4800cbee0 tx=0x7fd47802bdb0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 -- 172.21.15.60:0/2496613934 shutdown_connections 2023-10-16T14:10:11.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 --2- 172.21.15.60:0/2496613934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070c10 0x7fd4800cd690 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 --2- 172.21.15.60:0/2496613934 >> v2:172.21.15.144:3300/0 conn(0x7fd4800cf5f0 0x7fd48006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 -- 172.21.15.60:0/2496613934 >> 172.21.15.60:0/2496613934 conn(0x7fd4800cfac0 msgr2=0x7fd4800cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:11.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 -- 172.21.15.60:0/2496613934 shutdown_connections 2023-10-16T14:10:11.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.936+0000 7fd48671d640 1 -- 172.21.15.60:0/2496613934 wait complete. 2023-10-16T14:10:11.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.938+0000 7fd48671d640 1 Processor -- start 2023-10-16T14:10:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.938+0000 7fd48671d640 1 -- start start 2023-10-16T14:10:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd48671d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 0x7fd4800af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd48671d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd480070c10 0x7fd4800b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd48671d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800cf5f0 0x7fd4800b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd48671d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800b3850 con 0x7fd480070830 2023-10-16T14:10:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd48671d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800aff70 con 0x7fd480070c10 2023-10-16T14:10:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd48671d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800b0250 con 0x7fd4800cf5f0 2023-10-16T14:10:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd47ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 0x7fd4800af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd47ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 0x7fd4800af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43478/0 (socket says 172.21.15.60:43478) 2023-10-16T14:10:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.939+0000 7fd47ffff640 1 -- 172.21.15.60:0/4095239011 learned_addr learned my addr 172.21.15.60:0/4095239011 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:11.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd4777fe640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd480070c10 0x7fd4800b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd484cba640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800cf5f0 0x7fd4800b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47ffff640 1 -- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800cf5f0 msgr2=0x7fd4800b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47ffff640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800cf5f0 0x7fd4800b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47ffff640 1 -- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd480070c10 msgr2=0x7fd4800b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47ffff640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd480070c10 0x7fd4800b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47ffff640 1 -- 172.21.15.60:0/4095239011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478009a70 con 0x7fd480070830 2023-10-16T14:10:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd4777fe640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd480070c10 0x7fd4800b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47ffff640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 0x7fd4800af520 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c430 tx=0x7fd47000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470005ce0 con 0x7fd480070830 2023-10-16T14:10:11.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd470005e40 con 0x7fd480070830 2023-10-16T14:10:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4700072f0 con 0x7fd480070830 2023-10-16T14:10:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4800b0530 con 0x7fd480070830 2023-10-16T14:10:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.940+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4801f2d90 con 0x7fd480070830 2023-10-16T14:10:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.942+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4800b3340 con 0x7fd480070830 2023-10-16T14:10:11.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.942+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 126) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd470007450 con 0x7fd480070830 2023-10-16T14:10:11.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.943+0000 7fd47d7fa640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd46006af00 0x7fd46006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:11.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.943+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 6483+0+0 (secure 0 0 0) 0x7fd470088f40 con 0x7fd480070830 2023-10-16T14:10:11.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.944+0000 7fd4777fe640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd46006af00 0x7fd46006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:11.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.945+0000 7fd4777fe640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd46006af00 0x7fd46006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4800b5550 tx=0x7fd468008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:11.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:11.948+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4700bbfe0 con 0x7fd480070830 2023-10-16T14:10:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.184+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd48004e160 con 0x7fd480070830 2023-10-16T14:10:12.188 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:10:12.185+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:10:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.188+0000 7fd47d7fa640 1 -- 172.21.15.60:0/4095239011 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v105) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd470017020 con 0x7fd480070830 2023-10-16T14:10:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:10:12.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd46006af00 msgr2=0x7fd46006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:12.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd46006af00 0x7fd46006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4800b5550 tx=0x7fd468008040 comp rx=0 tx=0).stop 2023-10-16T14:10:12.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 msgr2=0x7fd4800af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:12.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 0x7fd4800af520 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c430 tx=0x7fd47000c8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 shutdown_connections 2023-10-16T14:10:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd46006af00 0x7fd46006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd4800cf5f0 0x7fd4800b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd480070c10 0x7fd4800b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 --2- 172.21.15.60:0/4095239011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd480070830 0x7fd4800af520 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.191+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 >> 172.21.15.60:0/4095239011 conn(0x7fd4800cfac0 msgr2=0x7fd4800ae200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.192+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 shutdown_connections 2023-10-16T14:10:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.192+0000 7fd48671d640 1 -- 172.21.15.60:0/4095239011 wait complete. 2023-10-16T14:10:12.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:10:12.310 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:10:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.429+0000 7fd1c2768640 1 Processor -- start 2023-10-16T14:10:12.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.429+0000 7fd1c2768640 1 -- start start 2023-10-16T14:10:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.432+0000 7fd1c2768640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1bc0cfc70 0x7fd1bc0c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.432+0000 7fd1c2768640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0b3e80 con 0x7fd1bc0cf890 2023-10-16T14:10:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.432+0000 7fd1c2768640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0ce630 con 0x7fd1bc0c5cf0 2023-10-16T14:10:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.432+0000 7fd1c2768640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0b4630 con 0x7fd1bc0cfc70 2023-10-16T14:10:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.432+0000 7fd1bbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd1bc0cf890 0x7fd1bc0b5b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52944/0 (socket says 172.21.15.60:52944) 2023-10-16T14:10:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.432+0000 7fd1bbfff640 1 -- 172.21.15.60:0/4141635706 learned_addr learned my addr 172.21.15.60:0/4141635706 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.433+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580481238 0 0) 0x7fd1bc0b3e80 con 0x7fd1bc0cf890 2023-10-16T14:10:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.433+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a4003520 con 0x7fd1bc0cf890 2023-10-16T14:10:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632823545 0 0) 0x7fd1a4003520 con 0x7fd1bc0cf890 2023-10-16T14:10:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1bc0cfc70 msgr2=0x7fd1bc0c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1bc0cfc70 0x7fd1bc0c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 >> v1:172.21.15.144:6789/0 conn(0x7fd1bc0c5cf0 legacy=0x7fd1bc1653a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0d3c20 con 0x7fd1bc0cf890 2023-10-16T14:10:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd1ac0050b0 con 0x7fd1bc0cf890 2023-10-16T14:10:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 >> v1:172.21.15.60:6789/0 conn(0x7fd1bc0cf890 legacy=0x7fd1bc0b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a40039a0 0x7fd1a4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a40047c0 0x7fd1a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1bc0cfc70 0x7fd1a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0b4630 con 0x7fd1a40047c0 2023-10-16T14:10:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40055f0 con 0x7fd1a40039a0 2023-10-16T14:10:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a400bec0 con 0x7fd1bc0cfc70 2023-10-16T14:10:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.434+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a40047c0 0x7fd1a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.435+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1bc0cfc70 msgr2=0x7fd1a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.435+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1bc0cfc70 0x7fd1a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.435+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a40039a0 msgr2=0x7fd1a4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.435+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1a40039a0 0x7fd1a4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.435+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/4141635706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a4003520 con 0x7fd1a40047c0 2023-10-16T14:10:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.436+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a40047c0 0x7fd1a4004c40 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0003db0 tx=0x7fd1a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e46027053a8a17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.436+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0032600 con 0x7fd1a40047c0 2023-10-16T14:10:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.436+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b002e050 con 0x7fd1a40047c0 2023-10-16T14:10:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1baffd640 1 -- 172.21.15.60:0/4141635706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00329a0 con 0x7fd1a40047c0 2023-10-16T14:10:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1c2768640 1 -- 172.21.15.60:0/4141635706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a40047c0 msgr2=0x7fd1a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1c2768640 1 --2- 172.21.15.60:0/4141635706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1a40047c0 0x7fd1a4004c40 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0003db0 tx=0x7fd1a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1c0d05640 1 -- 172.21.15.60:0/4141635706 reap_dead start 2023-10-16T14:10:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1c2768640 1 -- 172.21.15.60:0/4141635706 shutdown_connections 2023-10-16T14:10:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1c2768640 1 -- 172.21.15.60:0/4141635706 >> 172.21.15.60:0/4141635706 conn(0x7fd1bc0d2cc0 msgr2=0x7fd1bc0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.437+0000 7fd1c2768640 1 -- 172.21.15.60:0/4141635706 shutdown_connections 2023-10-16T14:10:12.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.438+0000 7fd1c2768640 1 -- 172.21.15.60:0/4141635706 wait complete. 2023-10-16T14:10:12.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.439+0000 7fd1c2768640 1 Processor -- start 2023-10-16T14:10:12.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.440+0000 7fd1c2768640 1 -- start start 2023-10-16T14:10:12.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.440+0000 7fd1c2768640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a40047c0 0x7fd1bc155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.440+0000 7fd1c2768640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 0x7fd1bc0cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.440+0000 7fd1c2768640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1bc15ed80 0x7fd1bc1ea860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1c2768640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc15e960 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1c2768640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc155850 con 0x7fd1bc15ed80 2023-10-16T14:10:12.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1c2768640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0cb780 con 0x7fd1a40047c0 2023-10-16T14:10:12.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 0x7fd1bc0cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:12.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 0x7fd1bc0cb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43504/0 (socket says 172.21.15.60:43504) 2023-10-16T14:10:12.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/1626389557 learned_addr learned my addr 172.21.15.60:0/1626389557 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:12.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bbfff640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a40047c0 0x7fd1bc155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:12.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1c0d05640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1bc15ed80 0x7fd1bc1ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:12.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a40047c0 msgr2=0x7fd1bc155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:12.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a40047c0 0x7fd1bc155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1bc15ed80 msgr2=0x7fd1bc1ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:12.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1bc15ed80 0x7fd1bc1ea860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:12.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1bb7fe640 1 -- 172.21.15.60:0/1626389557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc1606d0 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.441+0000 7fd1c0d05640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1bc15ed80 0x7fd1bc1ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:12.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.442+0000 7fd1bb7fe640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 0x7fd1bc0cb240 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0003ed0 tx=0x7fd1b00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:12.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.442+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b003d830 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.442+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b003d990 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.442+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b003dd00 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.442+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc15fb10 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.442+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc1600d0 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.443+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc15eaa0 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.444+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 126) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd1b0032400 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.445+0000 7fd1b8ff9640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd1a806af30 0x7fd1a806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.445+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fd1b00ab880 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.446+0000 7fd1bbfff640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd1a806af30 0x7fd1a806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:12.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.447+0000 7fd1bbfff640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd1a806af30 0x7fd1a806d3f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac004870 tx=0x7fd1ac004f90 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:12.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.449+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1bc15eaa0 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.685+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd1bc15eaa0 con 0x7fd1bc0c8e50 2023-10-16T14:10:12.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:12.832+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd1b0076990 con 0x7fd1bc0c8e50 2023-10-16T14:10:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.192+0000 7fd1b8ff9640 1 -- 172.21.15.60:0/1626389557 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v106) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd1b007df50 con 0x7fd1bc0c8e50 2023-10-16T14:10:13.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.196+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd1a806af30 msgr2=0x7fd1a806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:13.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.196+0000 7fd1c2768640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd1a806af30 0x7fd1a806d3f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac004870 tx=0x7fd1ac004f90 comp rx=0 tx=0).stop 2023-10-16T14:10:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.196+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 msgr2=0x7fd1bc0cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.196+0000 7fd1c2768640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 0x7fd1bc0cb240 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0003ed0 tx=0x7fd1b00036f0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.196+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 shutdown_connections 2023-10-16T14:10:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.196+0000 7fd1c2768640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd1a806af30 0x7fd1a806d3f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.197+0000 7fd1c2768640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd1bc15ed80 0x7fd1bc1ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.197+0000 7fd1c2768640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1bc0c8e50 0x7fd1bc0cb240 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.197+0000 7fd1c2768640 1 --2- 172.21.15.60:0/1626389557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1a40047c0 0x7fd1bc155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.197+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 >> 172.21.15.60:0/1626389557 conn(0x7fd1bc0d2cc0 msgr2=0x7fd1bc0d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:13.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.197+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 shutdown_connections 2023-10-16T14:10:13.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.197+0000 7fd1c2768640 1 -- 172.21.15.60:0/1626389557 wait complete. 2023-10-16T14:10:13.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:10:13.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.446+0000 7fee5d021640 1 Processor -- start 2023-10-16T14:10:13.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.447+0000 7fee5d021640 1 -- start start 2023-10-16T14:10:13.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.448+0000 7fee5d021640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fee581599c0 0x7fee580c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:13.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.448+0000 7fee5d021640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee58159da0 0x7fee5815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:13.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.448+0000 7fee5d021640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5815dd10 con 0x7fee580cce20 2023-10-16T14:10:13.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.448+0000 7fee5d021640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5815dfb0 con 0x7fee58159da0 2023-10-16T14:10:13.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.448+0000 7fee5d021640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5815e290 con 0x7fee581599c0 2023-10-16T14:10:13.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee58159da0 0x7fee5815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:13.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee58159da0 0x7fee5815d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43508/0 (socket says 172.21.15.60:43508) 2023-10-16T14:10:13.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 -- 172.21.15.60:0/3964944892 learned_addr learned my addr 172.21.15.60:0/3964944892 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:13.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee56d9d640 1 --2- 172.21.15.60:0/3964944892 >> v2:172.21.15.144:3300/0 conn(0x7fee581599c0 0x7fee580c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:13.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 -- 172.21.15.60:0/3964944892 >> v2:172.21.15.144:3300/0 conn(0x7fee581599c0 msgr2=0x7fee580c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:13.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 --2- 172.21.15.60:0/3964944892 >> v2:172.21.15.144:3300/0 conn(0x7fee581599c0 0x7fee580c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 -- 172.21.15.60:0/3964944892 >> v1:172.21.15.60:6789/0 conn(0x7fee580cce20 legacy=0x7fee5806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:13.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.449+0000 7fee5659c640 1 -- 172.21.15.60:0/3964944892 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5815e570 con 0x7fee58159da0 2023-10-16T14:10:13.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.451+0000 7fee5659c640 1 --2- 172.21.15.60:0/3964944892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58159da0 0x7fee5815d7d0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fee5806d730 tx=0x7fee4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3949c8b051071295 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:13.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.451+0000 7fee55d9b640 1 -- 172.21.15.60:0/3964944892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c030860 con 0x7fee58159da0 2023-10-16T14:10:13.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.451+0000 7fee55d9b640 1 -- 172.21.15.60:0/3964944892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c0309c0 con 0x7fee58159da0 2023-10-16T14:10:13.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.451+0000 7fee55d9b640 1 -- 172.21.15.60:0/3964944892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c030d50 con 0x7fee58159da0 2023-10-16T14:10:13.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 -- 172.21.15.60:0/3964944892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58159da0 msgr2=0x7fee5815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:13.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 --2- 172.21.15.60:0/3964944892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58159da0 0x7fee5815d7d0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fee5806d730 tx=0x7fee4c02b9a0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 -- 172.21.15.60:0/3964944892 shutdown_connections 2023-10-16T14:10:13.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 --2- 172.21.15.60:0/3964944892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee58159da0 0x7fee5815d7d0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 --2- 172.21.15.60:0/3964944892 >> v2:172.21.15.144:3300/0 conn(0x7fee581599c0 0x7fee580c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 -- 172.21.15.60:0/3964944892 >> 172.21.15.60:0/3964944892 conn(0x7fee580cb360 msgr2=0x7fee580cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:13.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 -- 172.21.15.60:0/3964944892 shutdown_connections 2023-10-16T14:10:13.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.452+0000 7fee5d021640 1 -- 172.21.15.60:0/3964944892 wait complete. 2023-10-16T14:10:13.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.454+0000 7fee5d021640 1 Processor -- start 2023-10-16T14:10:13.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.454+0000 7fee5d021640 1 -- start start 2023-10-16T14:10:13.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5d021640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee580cce20 0x7fee580afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:13.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5d021640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 0x7fee580aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:13.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5d021640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee58159da0 0x7fee580ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:13.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5d021640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580b0300 con 0x7fee581599c0 2023-10-16T14:10:13.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5d021640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580b05e0 con 0x7fee580cce20 2023-10-16T14:10:13.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5d021640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580b08c0 con 0x7fee58159da0 2023-10-16T14:10:13.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 0x7fee580aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:13.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 0x7fee580aca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43518/0 (socket says 172.21.15.60:43518) 2023-10-16T14:10:13.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5659c640 1 -- 172.21.15.60:0/1593201021 learned_addr learned my addr 172.21.15.60:0/1593201021 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:13.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee56d9d640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee580cce20 0x7fee580afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:13.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5659c640 1 -- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee58159da0 msgr2=0x7fee580ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:13.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.455+0000 7fee5759e640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee58159da0 0x7fee580ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5659c640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee58159da0 0x7fee580ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5659c640 1 -- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee580cce20 msgr2=0x7fee580afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5659c640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee580cce20 0x7fee580afab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5659c640 1 -- 172.21.15.60:0/1593201021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0094d0 con 0x7fee581599c0 2023-10-16T14:10:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee56d9d640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee580cce20 0x7fee580afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5759e640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee58159da0 0x7fee580ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5659c640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 0x7fee580aca70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fee4c02b9e0 tx=0x7fee4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c030850 con 0x7fee581599c0 2023-10-16T14:10:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c043400 con 0x7fee581599c0 2023-10-16T14:10:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0437b0 con 0x7fee581599c0 2023-10-16T14:10:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee580b0ba0 con 0x7fee581599c0 2023-10-16T14:10:13.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.456+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee580afff0 con 0x7fee581599c0 2023-10-16T14:10:13.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.457+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5804e160 con 0x7fee581599c0 2023-10-16T14:10:13.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.458+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee4c043910 con 0x7fee581599c0 2023-10-16T14:10:13.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.460+0000 7fee3f7fe640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fee4406af90 0x7fee4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:13.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.460+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fee4c03e070 con 0x7fee581599c0 2023-10-16T14:10:13.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.461+0000 7fee56d9d640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fee4406af90 0x7fee4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.462+0000 7fee56d9d640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fee4406af90 0x7fee4406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b820 tx=0x7fee48008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.465+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee4c077b90 con 0x7fee581599c0 2023-10-16T14:10:13.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:13.663+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fee580b4860 con 0x7fee581599c0 2023-10-16T14:10:14.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.202+0000 7fee3f7fe640 1 -- 172.21.15.60:0/1593201021 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v222) v1 ==== 181+0+0 (secure 0 0 0) 0x7fee4c07b1b0 con 0x7fee581599c0 2023-10-16T14:10:14.207 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:10:14.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.206+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fee4406af90 msgr2=0x7fee4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.206+0000 7fee5d021640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fee4406af90 0x7fee4406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b820 tx=0x7fee48008040 comp rx=0 tx=0).stop 2023-10-16T14:10:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.206+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 msgr2=0x7fee580aca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.206+0000 7fee5d021640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 0x7fee580aca70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fee4c02b9e0 tx=0x7fee4c004730 comp rx=0 tx=0).stop 2023-10-16T14:10:14.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.209+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 shutdown_connections 2023-10-16T14:10:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.209+0000 7fee5d021640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fee4406af90 0x7fee4406d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.209+0000 7fee5d021640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fee58159da0 0x7fee580ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.209+0000 7fee5d021640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee581599c0 0x7fee580aca70 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.209+0000 7fee5d021640 1 --2- 172.21.15.60:0/1593201021 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fee580cce20 0x7fee580afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.209+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 >> 172.21.15.60:0/1593201021 conn(0x7fee580cb360 msgr2=0x7fee58070160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.210+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 shutdown_connections 2023-10-16T14:10:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.210+0000 7fee5d021640 1 -- 172.21.15.60:0/1593201021 wait complete. 2023-10-16T14:10:14.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:10:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.457+0000 7fd60f59e640 1 Processor -- start 2023-10-16T14:10:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.458+0000 7fd60f59e640 1 -- start start 2023-10-16T14:10:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.459+0000 7fd60f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd61006d220 0x7fd6100c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:14.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.459+0000 7fd60f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd610065ab0 0x7fd6100cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.459+0000 7fd60f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6100acd70 con 0x7fd61006d600 2023-10-16T14:10:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.459+0000 7fd60f59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6100ac4a0 con 0x7fd61006d220 2023-10-16T14:10:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.459+0000 7fd60f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd610065f40 con 0x7fd610065ab0 2023-10-16T14:10:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd61006d220 0x7fd6100c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:14.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd61006d220 0x7fd6100c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43526/0 (socket says 172.21.15.60:43526) 2023-10-16T14:10:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 -- 172.21.15.60:0/680511378 learned_addr learned my addr 172.21.15.60:0/680511378 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd607fff640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd610065ab0 0x7fd6100cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:14.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 -- 172.21.15.60:0/680511378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd610065ab0 msgr2=0x7fd6100cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd610065ab0 0x7fd6100cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 -- 172.21.15.60:0/680511378 >> v1:172.21.15.144:6789/0 conn(0x7fd61006d600 legacy=0x7fd6100ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd60e59c640 1 -- 172.21.15.60:0/680511378 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100ac720 con 0x7fd61006d220 2023-10-16T14:10:14.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.460+0000 7fd607fff640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd610065ab0 0x7fd6100cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.461+0000 7fd60e59c640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd6100c95e0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fd600009870 tx=0x7fd60002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00ad89f8fff9215 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.462+0000 7fd60dd9b640 1 -- 172.21.15.60:0/680511378 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600031640 con 0x7fd61006d220 2023-10-16T14:10:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.462+0000 7fd60dd9b640 1 -- 172.21.15.60:0/680511378 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6000317a0 con 0x7fd61006d220 2023-10-16T14:10:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.462+0000 7fd60dd9b640 1 -- 172.21.15.60:0/680511378 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600031b30 con 0x7fd61006d220 2023-10-16T14:10:14.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.462+0000 7fd60f59e640 1 -- 172.21.15.60:0/680511378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 msgr2=0x7fd6100c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.462+0000 7fd60f59e640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd6100c95e0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fd600009870 tx=0x7fd60002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.463+0000 7fd60f59e640 1 -- 172.21.15.60:0/680511378 shutdown_connections 2023-10-16T14:10:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.463+0000 7fd60f59e640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd610065ab0 0x7fd6100cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.463+0000 7fd60f59e640 1 --2- 172.21.15.60:0/680511378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd6100c95e0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.463+0000 7fd60f59e640 1 -- 172.21.15.60:0/680511378 >> 172.21.15.60:0/680511378 conn(0x7fd6100b0a80 msgr2=0x7fd6100b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.463+0000 7fd60f59e640 1 -- 172.21.15.60:0/680511378 shutdown_connections 2023-10-16T14:10:14.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.463+0000 7fd60f59e640 1 -- 172.21.15.60:0/680511378 wait complete. 2023-10-16T14:10:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.464+0000 7fd60f59e640 1 Processor -- start 2023-10-16T14:10:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.465+0000 7fd60f59e640 1 -- start start 2023-10-16T14:10:14.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd610159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61006d600 0x7fd6100c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6100c8220 0x7fd6100b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100c8db0 con 0x7fd61006d220 2023-10-16T14:10:14.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd610159f90 con 0x7fd61006d600 2023-10-16T14:10:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100c87f0 con 0x7fd6100c8220 2023-10-16T14:10:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd610159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd610159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43530/0 (socket says 172.21.15.60:43530) 2023-10-16T14:10:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 -- 172.21.15.60:0/1593086679 learned_addr learned my addr 172.21.15.60:0/1593086679 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:14.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd607fff640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61006d600 0x7fd6100c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60ed9d640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6100c8220 0x7fd6100b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 -- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6100c8220 msgr2=0x7fd6100b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6100c8220 0x7fd6100b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 -- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61006d600 msgr2=0x7fd6100c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61006d600 0x7fd6100c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.466+0000 7fd60e59c640 1 -- 172.21.15.60:0/1593086679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6000094d0 con 0x7fd61006d220 2023-10-16T14:10:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.467+0000 7fd60e59c640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd610159a50 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fd600009870 tx=0x7fd6000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.467+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60003ac60 con 0x7fd61006d220 2023-10-16T14:10:14.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.467+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60003ade0 con 0x7fd61006d220 2023-10-16T14:10:14.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.467+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600044630 con 0x7fd61006d220 2023-10-16T14:10:14.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.467+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6100c8ad0 con 0x7fd61006d220 2023-10-16T14:10:14.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.467+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61009adc0 con 0x7fd61006d220 2023-10-16T14:10:14.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.468+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61009a940 con 0x7fd61006d220 2023-10-16T14:10:14.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.469+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd60002f050 con 0x7fd61006d220 2023-10-16T14:10:14.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.471+0000 7fd606ffd640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd5f006afe0 0x7fd5f006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.471+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6103+0+0 (secure 0 0 0) 0x7fd6000abb80 con 0x7fd61006d220 2023-10-16T14:10:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.472+0000 7fd607fff640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd5f006afe0 0x7fd5f006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:14.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.473+0000 7fd607fff640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd5f006afe0 0x7fd5f006d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd61015a950 tx=0x7fd5f8008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:14.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.477+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd61009a940 con 0x7fd61006d220 2023-10-16T14:10:14.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:14.680+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd61009a940 con 0x7fd61006d220 2023-10-16T14:10:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.206+0000 7fd606ffd640 1 -- 172.21.15.60:0/1593086679 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v223) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd61009a940 con 0x7fd61006d220 2023-10-16T14:10:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T14:10:15.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.211+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd5f006afe0 msgr2=0x7fd5f006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.211+0000 7fd60f59e640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd5f006afe0 0x7fd5f006d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd61015a950 tx=0x7fd5f8008040 comp rx=0 tx=0).stop 2023-10-16T14:10:15.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.211+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 msgr2=0x7fd610159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.211+0000 7fd60f59e640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd610159a50 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fd600009870 tx=0x7fd6000046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.212+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 shutdown_connections 2023-10-16T14:10:15.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.212+0000 7fd60f59e640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fd5f006afe0 0x7fd5f006d4a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.212+0000 7fd60f59e640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd6100c8220 0x7fd6100b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.212+0000 7fd60f59e640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd61006d600 0x7fd6100c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.212+0000 7fd60f59e640 1 --2- 172.21.15.60:0/1593086679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd61006d220 0x7fd610159a50 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.212+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 >> 172.21.15.60:0/1593086679 conn(0x7fd6100b0a80 msgr2=0x7fd6100ce1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:15.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.213+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 shutdown_connections 2023-10-16T14:10:15.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.213+0000 7fd60f59e640 1 -- 172.21.15.60:0/1593086679 wait complete. 2023-10-16T14:10:15.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:10:15.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.447+0000 7fdf46261640 1 Processor -- start 2023-10-16T14:10:15.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.447+0000 7fdf46261640 1 -- start start 2023-10-16T14:10:15.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.449+0000 7fdf46261640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdf40070fd0 0x7fdf400cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.449+0000 7fdf46261640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf400adfc0 0x7fdf400b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.449+0000 7fdf46261640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf400cc600 con 0x7fdf400adbe0 2023-10-16T14:10:15.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.449+0000 7fdf46261640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf40071470 con 0x7fdf400adfc0 2023-10-16T14:10:15.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.449+0000 7fdf46261640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf400cc870 con 0x7fdf40070fd0 2023-10-16T14:10:15.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.449+0000 7fdf3ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf400adfc0 0x7fdf400b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf400adfc0 0x7fdf400b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43534/0 (socket says 172.21.15.60:43534) 2023-10-16T14:10:15.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3ffff640 1 -- 172.21.15.60:0/1177347140 learned_addr learned my addr 172.21.15.60:0/1177347140 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3effd640 1 --2- 172.21.15.60:0/1177347140 >> v2:172.21.15.144:3300/0 conn(0x7fdf40070fd0 0x7fdf400cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3ffff640 1 -- 172.21.15.60:0/1177347140 >> v2:172.21.15.144:3300/0 conn(0x7fdf40070fd0 msgr2=0x7fdf400cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3ffff640 1 --2- 172.21.15.60:0/1177347140 >> v2:172.21.15.144:3300/0 conn(0x7fdf40070fd0 0x7fdf400cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3ffff640 1 -- 172.21.15.60:0/1177347140 >> v1:172.21.15.144:6789/0 conn(0x7fdf400adbe0 legacy=0x7fdf4006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.450+0000 7fdf3ffff640 1 -- 172.21.15.60:0/1177347140 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf400ccb50 con 0x7fdf400adfc0 2023-10-16T14:10:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.451+0000 7fdf3ffff640 1 --2- 172.21.15.60:0/1177347140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf400adfc0 0x7fdf400b0410 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fdf400cc2b0 tx=0x7fdf3802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a87b106fb0d15d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf3e7fc640 1 -- 172.21.15.60:0/1177347140 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3802ca80 con 0x7fdf400adfc0 2023-10-16T14:10:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf3e7fc640 1 -- 172.21.15.60:0/1177347140 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3802cbe0 con 0x7fdf400adfc0 2023-10-16T14:10:15.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf3e7fc640 1 -- 172.21.15.60:0/1177347140 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf380343f0 con 0x7fdf400adfc0 2023-10-16T14:10:15.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 -- 172.21.15.60:0/1177347140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf400adfc0 msgr2=0x7fdf400b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 --2- 172.21.15.60:0/1177347140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf400adfc0 0x7fdf400b0410 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fdf400cc2b0 tx=0x7fdf3802bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 -- 172.21.15.60:0/1177347140 shutdown_connections 2023-10-16T14:10:15.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 --2- 172.21.15.60:0/1177347140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf400adfc0 0x7fdf400b0410 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 --2- 172.21.15.60:0/1177347140 >> v2:172.21.15.144:3300/0 conn(0x7fdf40070fd0 0x7fdf400cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 -- 172.21.15.60:0/1177347140 >> 172.21.15.60:0/1177347140 conn(0x7fdf400cb410 msgr2=0x7fdf400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:15.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 -- 172.21.15.60:0/1177347140 shutdown_connections 2023-10-16T14:10:15.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.452+0000 7fdf46261640 1 -- 172.21.15.60:0/1177347140 wait complete. 2023-10-16T14:10:15.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.454+0000 7fdf46261640 1 Processor -- start 2023-10-16T14:10:15.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.454+0000 7fdf46261640 1 -- start start 2023-10-16T14:10:15.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.455+0000 7fdf46261640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 0x7fdf4015cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.455+0000 7fdf46261640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf400adbe0 0x7fdf400c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.455+0000 7fdf46261640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf400adfc0 0x7fdf401f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.455+0000 7fdf46261640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4015d260 con 0x7fdf40070fd0 2023-10-16T14:10:15.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.455+0000 7fdf46261640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400c3b90 con 0x7fdf400adbe0 2023-10-16T14:10:15.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.455+0000 7fdf46261640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400c3e70 con 0x7fdf400adfc0 2023-10-16T14:10:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 0x7fdf4015cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 0x7fdf4015cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43540/0 (socket says 172.21.15.60:43540) 2023-10-16T14:10:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 -- 172.21.15.60:0/1692828994 learned_addr learned my addr 172.21.15.60:0/1692828994 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3effd640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf400adbe0 0x7fdf400c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3ffff640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf400adfc0 0x7fdf401f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 -- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf400adfc0 msgr2=0x7fdf401f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf400adfc0 0x7fdf401f5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 -- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf400adbe0 msgr2=0x7fdf400c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf400adbe0 0x7fdf400c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.456+0000 7fdf3f7fe640 1 -- 172.21.15.60:0/1692828994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf38009b30 con 0x7fdf40070fd0 2023-10-16T14:10:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.457+0000 7fdf3f7fe640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 0x7fdf4015cbc0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c410 tx=0x7fdf3000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.457+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf300035f0 con 0x7fdf40070fd0 2023-10-16T14:10:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.457+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf30003750 con 0x7fdf40070fd0 2023-10-16T14:10:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.457+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf400c4150 con 0x7fdf40070fd0 2023-10-16T14:10:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.457+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30003b40 con 0x7fdf40070fd0 2023-10-16T14:10:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.457+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf400c37f0 con 0x7fdf40070fd0 2023-10-16T14:10:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.458+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4004e160 con 0x7fdf40070fd0 2023-10-16T14:10:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.459+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf30003ca0 con 0x7fdf40070fd0 2023-10-16T14:10:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.460+0000 7fdf27fff640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf2c06b010 0x7fdf2c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.460+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fdf30088d60 con 0x7fdf40070fd0 2023-10-16T14:10:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.461+0000 7fdf3effd640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf2c06b010 0x7fdf2c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.462+0000 7fdf3effd640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf2c06b010 0x7fdf2c06d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf34006fd0 tx=0x7fdf34008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.464+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf30053e30 con 0x7fdf40070fd0 2023-10-16T14:10:15.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.660+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fdf4015f6a0 con 0x7fdf40070fd0 2023-10-16T14:10:15.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.667+0000 7fdf27fff640 1 -- 172.21.15.60:0/1692828994 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7fdf30017020 con 0x7fdf40070fd0 2023-10-16T14:10:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf2c06b010 msgr2=0x7fdf2c06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf2c06b010 0x7fdf2c06d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdf34006fd0 tx=0x7fdf34008040 comp rx=0 tx=0).stop 2023-10-16T14:10:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 msgr2=0x7fdf4015cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 0x7fdf4015cbc0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c410 tx=0x7fdf3000c8d0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 shutdown_connections 2023-10-16T14:10:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf2c06b010 0x7fdf2c06d4d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf400adfc0 0x7fdf401f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf400adbe0 0x7fdf400c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 --2- 172.21.15.60:0/1692828994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf40070fd0 0x7fdf4015cbc0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 >> 172.21.15.60:0/1692828994 conn(0x7fdf400cb410 msgr2=0x7fdf400af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 shutdown_connections 2023-10-16T14:10:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.670+0000 7fdf46261640 1 -- 172.21.15.60:0/1692828994 wait complete. 2023-10-16T14:10:15.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:10:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.926+0000 7ff18132e640 1 Processor -- start 2023-10-16T14:10:15.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.927+0000 7ff18132e640 1 -- start start 2023-10-16T14:10:15.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.928+0000 7ff18132e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 0x7ff17c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.928+0000 7ff18132e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c0ca560 con 0x7ff17c0c9060 2023-10-16T14:10:15.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff18132e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c0c9950 con 0x7ff17c06c9e0 2023-10-16T14:10:15.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff18132e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c0c9c30 con 0x7ff17c06fb50 2023-10-16T14:10:15.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff17bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff17c0c9060 0x7ff17c06f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52970/0 (socket says 172.21.15.60:52970) 2023-10-16T14:10:15.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff17bfff640 1 -- 172.21.15.60:0/3537988545 learned_addr learned my addr 172.21.15.60:0/3537988545 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:15.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff173fff640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 0x7ff17c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887540966 0 0) 0x7ff17c0ca560 con 0x7ff17c0c9060 2023-10-16T14:10:15.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.929+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff168003520 con 0x7ff17c0c9060 2023-10-16T14:10:15.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991389221 0 0) 0x7ff168003520 con 0x7ff17c0c9060 2023-10-16T14:10:15.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 msgr2=0x7ff17c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:15.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 0x7ff17c0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 >> v1:172.21.15.144:6789/0 conn(0x7ff17c06c9e0 legacy=0x7ff17c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c0c9f10 con 0x7ff17c0c9060 2023-10-16T14:10:15.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff16c001670 con 0x7ff17c0c9060 2023-10-16T14:10:15.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 >> v1:172.21.15.60:6789/0 conn(0x7ff17c0c9060 legacy=0x7ff17c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1680039a0 0x7ff168003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 0x7ff1680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff168004d00 0x7ff168005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c0c9c30 con 0x7ff168004d00 2023-10-16T14:10:15.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16800ba60 con 0x7ff1680039a0 2023-10-16T14:10:15.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16800bc20 con 0x7ff17c06fb50 2023-10-16T14:10:15.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.930+0000 7ff180b2d640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff168004d00 0x7ff168005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff17bfff640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1680039a0 0x7ff168003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff173fff640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 0x7ff1680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff180b2d640 1 -- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 msgr2=0x7ff1680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:15.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff180b2d640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c06fb50 0x7ff1680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff180b2d640 1 -- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1680039a0 msgr2=0x7ff168003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:15.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff180b2d640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1680039a0 0x7ff168003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.931+0000 7ff180b2d640 1 -- 172.21.15.60:0/3537988545 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff168003520 con 0x7ff168004d00 2023-10-16T14:10:15.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.932+0000 7ff180b2d640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff168004d00 0x7ff168005180 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7ff17c06cde0 tx=0x7ff1680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da086b6356d03c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:15.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff174003b70 con 0x7ff168004d00 2023-10-16T14:10:15.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff174003cd0 con 0x7ff168004d00 2023-10-16T14:10:15.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff17b7fe640 1 -- 172.21.15.60:0/3537988545 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff174005310 con 0x7ff168004d00 2023-10-16T14:10:15.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff18132e640 1 -- 172.21.15.60:0/3537988545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff168004d00 msgr2=0x7ff168005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff18132e640 1 --2- 172.21.15.60:0/3537988545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff168004d00 0x7ff168005180 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7ff17c06cde0 tx=0x7ff1680057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff180b2d640 1 -- 172.21.15.60:0/3537988545 reap_dead start 2023-10-16T14:10:15.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff18132e640 1 -- 172.21.15.60:0/3537988545 shutdown_connections 2023-10-16T14:10:15.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff18132e640 1 -- 172.21.15.60:0/3537988545 >> 172.21.15.60:0/3537988545 conn(0x7ff17c0ce6a0 msgr2=0x7ff17c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff18132e640 1 -- 172.21.15.60:0/3537988545 shutdown_connections 2023-10-16T14:10:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.933+0000 7ff18132e640 1 -- 172.21.15.60:0/3537988545 wait complete. 2023-10-16T14:10:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.935+0000 7ff18132e640 1 Processor -- start 2023-10-16T14:10:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.935+0000 7ff18132e640 1 -- start start 2023-10-16T14:10:15.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff18132e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168004d00 0x7ff17c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff18132e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 0x7ff17c0c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff18132e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c15f320 0x7ff17c0c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff18132e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c162120 con 0x7ff17c161c90 2023-10-16T14:10:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff18132e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c1631d0 con 0x7ff168004d00 2023-10-16T14:10:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff18132e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c1634b0 con 0x7ff17c15f320 2023-10-16T14:10:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff173fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 0x7ff17c0c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.936+0000 7ff173fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 0x7ff17c0c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43560/0 (socket says 172.21.15.60:43560) 2023-10-16T14:10:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 -- 172.21.15.60:0/2957893864 learned_addr learned my addr 172.21.15.60:0/2957893864 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 -- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c15f320 msgr2=0x7ff17c0c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c15f320 0x7ff17c0c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 -- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168004d00 msgr2=0x7ff17c0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168004d00 0x7ff17c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 -- 172.21.15.60:0/2957893864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c15fea0 con 0x7ff17c161c90 2023-10-16T14:10:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff173fff640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 0x7ff17c0c5cb0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7ff164011600 tx=0x7ff164011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.937+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff164015870 con 0x7ff17c161c90 2023-10-16T14:10:15.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.938+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1640159d0 con 0x7ff17c161c90 2023-10-16T14:10:15.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.938+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff164015d40 con 0x7ff17c161c90 2023-10-16T14:10:15.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.938+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17c160090 con 0x7ff17c161c90 2023-10-16T14:10:15.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.938+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17c15f9e0 con 0x7ff17c161c90 2023-10-16T14:10:15.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.939+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17c163670 con 0x7ff17c161c90 2023-10-16T14:10:15.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.940+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff16400f490 con 0x7ff17c161c90 2023-10-16T14:10:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.941+0000 7ff1797fa640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff15c06af90 0x7ff15c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.941+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7ff164089b40 con 0x7ff17c161c90 2023-10-16T14:10:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.942+0000 7ff17bfff640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff15c06af90 0x7ff15c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.943+0000 7ff17bfff640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff15c06af90 0x7ff15c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff16c005990 tx=0x7ff16c000ef0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:15.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:15.945+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff164054bc0 con 0x7ff17c161c90 2023-10-16T14:10:16.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.141+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff17c163030 con 0x7ff17c161c90 2023-10-16T14:10:16.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.147+0000 7ff1797fa640 1 -- 172.21.15.60:0/2957893864 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff17c163030 con 0x7ff17c161c90 2023-10-16T14:10:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.149+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff15c06af90 msgr2=0x7ff15c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.149+0000 7ff18132e640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff15c06af90 0x7ff15c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff16c005990 tx=0x7ff16c000ef0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.149+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 msgr2=0x7ff17c0c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.149+0000 7ff18132e640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 0x7ff17c0c5cb0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7ff164011600 tx=0x7ff164011ac0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 shutdown_connections 2023-10-16T14:10:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff15c06af90 0x7ff15c06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff17c15f320 0x7ff17c0c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff17c161c90 0x7ff17c0c5cb0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 --2- 172.21.15.60:0/2957893864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168004d00 0x7ff17c0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 >> 172.21.15.60:0/2957893864 conn(0x7ff17c0ce6a0 msgr2=0x7ff17c0af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 shutdown_connections 2023-10-16T14:10:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.150+0000 7ff18132e640 1 -- 172.21.15.60:0/2957893864 wait complete. 2023-10-16T14:10:16.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:10:16.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.386+0000 7f75f8dfb640 1 Processor -- start 2023-10-16T14:10:16.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.386+0000 7f75f8dfb640 1 -- start start 2023-10-16T14:10:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f8dfb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75f40b3420 0x7f75f40b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f8dfb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f75f40cfaf0 0x7f75f40b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f8dfb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40d3980 con 0x7f75f40cf710 2023-10-16T14:10:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f8dfb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40d3c60 con 0x7f75f40b3420 2023-10-16T14:10:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f8dfb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40ce630 con 0x7f75f40cfaf0 2023-10-16T14:10:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75f40cf710 0x7f75f40b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52974/0 (socket says 172.21.15.60:52974) 2023-10-16T14:10:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.388+0000 7f75f259c640 1 -- 172.21.15.60:0/2109515528 learned_addr learned my addr 172.21.15.60:0/2109515528 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75e9d9b640 1 --2- 172.21.15.60:0/2109515528 >> v2:172.21.15.60:3300/0 conn(0x7f75f40b3420 0x7f75f40b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75f1d9b640 1 -- 172.21.15.60:0/2109515528 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021146320 0 0) 0x7f75f40d3980 con 0x7f75f40cf710 2023-10-16T14:10:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75f2d9d640 1 --2- 172.21.15.60:0/2109515528 >> v2:172.21.15.144:3300/0 conn(0x7f75f40cfaf0 0x7f75f40b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75f1d9b640 1 -- 172.21.15.60:0/2109515528 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e0003520 con 0x7f75f40cf710 2023-10-16T14:10:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75e9d9b640 1 -- 172.21.15.60:0/2109515528 >> v2:172.21.15.144:3300/0 conn(0x7f75f40cfaf0 msgr2=0x7f75f40b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75e9d9b640 1 --2- 172.21.15.60:0/2109515528 >> v2:172.21.15.144:3300/0 conn(0x7f75f40cfaf0 0x7f75f40b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75e9d9b640 1 -- 172.21.15.60:0/2109515528 >> v1:172.21.15.60:6789/0 conn(0x7f75f40cf710 legacy=0x7f75f40b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.389+0000 7f75e9d9b640 1 -- 172.21.15.60:0/2109515528 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40ce910 con 0x7f75f40b3420 2023-10-16T14:10:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.390+0000 7f75e9d9b640 1 --2- 172.21.15.60:0/2109515528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b3420 0x7f75f40b5cc0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f75f40d07a0 tx=0x7f75dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9b6c4583584e57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f1d9b640 1 -- 172.21.15.60:0/2109515528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc02e070 con 0x7f75f40b3420 2023-10-16T14:10:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f1d9b640 1 -- 172.21.15.60:0/2109515528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc0315a0 con 0x7f75f40b3420 2023-10-16T14:10:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f1d9b640 1 -- 172.21.15.60:0/2109515528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc031930 con 0x7f75f40b3420 2023-10-16T14:10:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f8dfb640 1 -- 172.21.15.60:0/2109515528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b3420 msgr2=0x7f75f40b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/2109515528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b3420 0x7f75f40b5cc0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f75f40d07a0 tx=0x7f75dc02b840 comp rx=0 tx=0).stop 2023-10-16T14:10:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f8dfb640 1 -- 172.21.15.60:0/2109515528 shutdown_connections 2023-10-16T14:10:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/2109515528 >> v2:172.21.15.144:3300/0 conn(0x7f75f40cfaf0 0x7f75f40b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/2109515528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b3420 0x7f75f40b5cc0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.391+0000 7f75f8dfb640 1 -- 172.21.15.60:0/2109515528 >> 172.21.15.60:0/2109515528 conn(0x7f75f40d2cc0 msgr2=0x7f75f40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.392+0000 7f75f8dfb640 1 -- 172.21.15.60:0/2109515528 shutdown_connections 2023-10-16T14:10:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.392+0000 7f75f8dfb640 1 -- 172.21.15.60:0/2109515528 wait complete. 2023-10-16T14:10:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.393+0000 7f75f8dfb640 1 Processor -- start 2023-10-16T14:10:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 -- start start 2023-10-16T14:10:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f40b3420 0x7f75f40cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 0x7f75f40ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f40cfaf0 0x7f75f40b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40c92c0 con 0x7f75f40cf710 2023-10-16T14:10:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40ccc00 con 0x7f75f40b3420 2023-10-16T14:10:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.394+0000 7f75f8dfb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40cc920 con 0x7f75f40cfaf0 2023-10-16T14:10:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 0x7f75f40ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 0x7f75f40ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43576/0 (socket says 172.21.15.60:43576) 2023-10-16T14:10:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 -- 172.21.15.60:0/4076368706 learned_addr learned my addr 172.21.15.60:0/4076368706 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75f2d9d640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f40cfaf0 0x7f75f40b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75f259c640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f40b3420 0x7f75f40cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 -- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f40cfaf0 msgr2=0x7f75f40b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f40cfaf0 0x7f75f40b7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 -- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f40b3420 msgr2=0x7f75f40cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f40b3420 0x7f75f40cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.395+0000 7f75e9d9b640 1 -- 172.21.15.60:0/4076368706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc009400 con 0x7f75f40cf710 2023-10-16T14:10:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.396+0000 7f75e9d9b640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 0x7f75f40ca9a0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f75dc000c00 tx=0x7f75dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.396+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc032840 con 0x7f75f40cf710 2023-10-16T14:10:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.396+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc0329a0 con 0x7f75f40cf710 2023-10-16T14:10:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.396+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc032d30 con 0x7f75f40cf710 2023-10-16T14:10:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.396+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f40c9ec0 con 0x7f75f40cf710 2023-10-16T14:10:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.396+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f40ca480 con 0x7f75f40cf710 2023-10-16T14:10:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.397+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f40ca670 con 0x7f75f40cf710 2023-10-16T14:10:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.398+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75dc04d3f0 con 0x7f75f40cf710 2023-10-16T14:10:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.399+0000 7f75eb7fe640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f75d406af90 0x7f75d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.399+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f75dc0ab3d0 con 0x7f75f40cf710 2023-10-16T14:10:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.400+0000 7f75f259c640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f75d406af90 0x7f75d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.401+0000 7f75f259c640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f75d406af90 0x7f75d406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75e40010e0 tx=0x7f75e4009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.404+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75dc0762a0 con 0x7f75f40cf710 2023-10-16T14:10:16.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.600+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f75f415e810 con 0x7f75f40cf710 2023-10-16T14:10:16.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.608+0000 7f75eb7fe640 1 -- 172.21.15.60:0/4076368706 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f75dc0798c0 con 0x7f75f40cf710 2023-10-16T14:10:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f75d406af90 msgr2=0x7f75d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f75d406af90 0x7f75d406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75e40010e0 tx=0x7f75e4009040 comp rx=0 tx=0).stop 2023-10-16T14:10:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 msgr2=0x7f75f40ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 0x7f75f40ca9a0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f75dc000c00 tx=0x7f75dc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 shutdown_connections 2023-10-16T14:10:16.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f75d406af90 0x7f75d406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f75f40cfaf0 0x7f75f40b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40cf710 0x7f75f40ca9a0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 --2- 172.21.15.60:0/4076368706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75f40b3420 0x7f75f40cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 >> 172.21.15.60:0/4076368706 conn(0x7f75f40d2cc0 msgr2=0x7f75f4160140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:16.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.611+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 shutdown_connections 2023-10-16T14:10:16.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.612+0000 7f75f8dfb640 1 -- 172.21.15.60:0/4076368706 wait complete. 2023-10-16T14:10:16.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T14:10:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.857+0000 7f707afc4640 1 Processor -- start 2023-10-16T14:10:16.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.857+0000 7f707afc4640 1 -- start start 2023-10-16T14:10:16.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f707afc4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7074159950 0x7f707415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f707afc4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707415aef0 con 0x7f7074070450 2023-10-16T14:10:16.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f707afc4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707415d910 con 0x7f707406c810 2023-10-16T14:10:16.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f707afc4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707415e490 con 0x7f7074159950 2023-10-16T14:10:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f7079fc2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7074159950 0x7f707415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f7079fc2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7074159950 0x7f707415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43588/0 (socket says 172.21.15.60:43588) 2023-10-16T14:10:16.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.859+0000 7f7079fc2640 1 -- 172.21.15.60:0/1315110862 learned_addr learned my addr 172.21.15.60:0/1315110862 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:16.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.860+0000 7f7079fc2640 1 -- 172.21.15.60:0/1315110862 >> v1:172.21.15.144:6789/0 conn(0x7f707406c810 legacy=0x7f70740c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.860+0000 7f7079fc2640 1 -- 172.21.15.60:0/1315110862 >> v1:172.21.15.60:6789/0 conn(0x7f7074070450 legacy=0x7f707406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.860+0000 7f7079fc2640 1 -- 172.21.15.60:0/1315110862 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707415dbf0 con 0x7f7074159950 2023-10-16T14:10:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.861+0000 7f7079fc2640 1 --2- 172.21.15.60:0/1315110862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074159950 0x7f707415a9b0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f7064009e50 tx=0x7f706402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1b153dc7b5fc13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:16.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.861+0000 7f7078fc0640 1 -- 172.21.15.60:0/1315110862 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706402ec90 con 0x7f7074159950 2023-10-16T14:10:16.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.861+0000 7f7078fc0640 1 -- 172.21.15.60:0/1315110862 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706402edf0 con 0x7f7074159950 2023-10-16T14:10:16.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.861+0000 7f7078fc0640 1 -- 172.21.15.60:0/1315110862 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064036610 con 0x7f7074159950 2023-10-16T14:10:16.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 -- 172.21.15.60:0/1315110862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074159950 msgr2=0x7f707415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 --2- 172.21.15.60:0/1315110862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074159950 0x7f707415a9b0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f7064009e50 tx=0x7f706402e0f0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 -- 172.21.15.60:0/1315110862 shutdown_connections 2023-10-16T14:10:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 --2- 172.21.15.60:0/1315110862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074159950 0x7f707415a9b0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 -- 172.21.15.60:0/1315110862 >> 172.21.15.60:0/1315110862 conn(0x7f70740ce6a0 msgr2=0x7f70740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 -- 172.21.15.60:0/1315110862 shutdown_connections 2023-10-16T14:10:16.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.862+0000 7f707afc4640 1 -- 172.21.15.60:0/1315110862 wait complete. 2023-10-16T14:10:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.864+0000 7f707afc4640 1 Processor -- start 2023-10-16T14:10:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.864+0000 7f707afc4640 1 -- start start 2023-10-16T14:10:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f707afc4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f707406c810 0x7f70740c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f707afc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 0x7f70740c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f707afc4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7074159950 0x7f70741582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f707afc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70740c7190 con 0x7f7074070450 2023-10-16T14:10:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f707afc4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70740c7cd0 con 0x7f7074159950 2023-10-16T14:10:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f707afc4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7074158f00 con 0x7f707406c810 2023-10-16T14:10:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f70797c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 0x7f70740c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f70797c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 0x7f70740c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43592/0 (socket says 172.21.15.60:43592) 2023-10-16T14:10:16.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.865+0000 7f70797c1640 1 -- 172.21.15.60:0/150465485 learned_addr learned my addr 172.21.15.60:0/150465485 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f7079fc2640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f707406c810 0x7f70740c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70797c1640 1 -- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f707406c810 msgr2=0x7f70740c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f707a7c3640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7074159950 0x7f70741582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70797c1640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f707406c810 0x7f70740c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70797c1640 1 -- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7074159950 msgr2=0x7f70741582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70797c1640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7074159950 0x7f70741582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70797c1640 1 -- 172.21.15.60:0/150465485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064009b30 con 0x7f7074070450 2023-10-16T14:10:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f707a7c3640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7074159950 0x7f70741582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70797c1640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 0x7f70740c2e50 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f706800c450 tx=0x7f706800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680052d0 con 0x7f7074070450 2023-10-16T14:10:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7068005430 con 0x7f7074070450 2023-10-16T14:10:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.866+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068012870 con 0x7f7074070450 2023-10-16T14:10:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.867+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70740c7370 con 0x7f7074070450 2023-10-16T14:10:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.867+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70740c7610 con 0x7f7074070450 2023-10-16T14:10:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.868+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70740c7e10 con 0x7f7074070450 2023-10-16T14:10:16.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.869+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70680129d0 con 0x7f7074070450 2023-10-16T14:10:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.871+0000 7f70727fc640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f706006af90 0x7f706006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.871+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7068089100 con 0x7f7074070450 2023-10-16T14:10:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.871+0000 7f7079fc2640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f706006af90 0x7f706006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:16.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.873+0000 7f7079fc2640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f706006af90 0x7f706006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7064037000 tx=0x7f7064032000 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:16.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:16.876+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70740c7e10 con 0x7f7074070450 2023-10-16T14:10:17.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.145+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f707415aef0 con 0x7f7074070450 2023-10-16T14:10:17.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.226+0000 7f70727fc640 1 -- 172.21.15.60:0/150465485 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3163) v1 ==== 163+0+0 (secure 0 0 0) 0x7f707415aef0 con 0x7f7074070450 2023-10-16T14:10:17.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f706006af90 msgr2=0x7f706006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f706006af90 0x7f706006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7064037000 tx=0x7f7064032000 comp rx=0 tx=0).stop 2023-10-16T14:10:17.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 msgr2=0x7f70740c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 0x7f70740c2e50 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f706800c450 tx=0x7f706800c910 comp rx=0 tx=0).stop 2023-10-16T14:10:17.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 shutdown_connections 2023-10-16T14:10:17.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f706006af90 0x7f706006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7074159950 0x7f70741582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7074070450 0x7f70740c2e50 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.229+0000 7f707afc4640 1 --2- 172.21.15.60:0/150465485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f707406c810 0x7f70740c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.230+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 >> 172.21.15.60:0/150465485 conn(0x7f70740ce6a0 msgr2=0x7f70740ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.230+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 shutdown_connections 2023-10-16T14:10:17.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.230+0000 7f707afc4640 1 -- 172.21.15.60:0/150465485 wait complete. 2023-10-16T14:10:17.254 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-16T14:10:17.255 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2023-10-16T14:10:17.255 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T14:10:17.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:17.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.487+0000 7f3993fff640 1 Processor -- start 2023-10-16T14:10:17.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.487+0000 7f3993fff640 1 -- start start 2023-10-16T14:10:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.488+0000 7f3993fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3994070240 0x7f3994070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.488+0000 7f3993fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940b35b0 con 0x7f399415abb0 2023-10-16T14:10:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.488+0000 7f3993fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3994070b90 con 0x7f39940cbad0 2023-10-16T14:10:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f3993fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f399415c040 con 0x7f3994070240 2023-10-16T14:10:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f39937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3994070240 0x7f3994070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f39937fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3994070240 0x7f3994070650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43606/0 (socket says 172.21.15.60:43606) 2023-10-16T14:10:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f39937fe640 1 -- 172.21.15.60:0/2748852057 learned_addr learned my addr 172.21.15.60:0/2748852057 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f39937fe640 1 -- 172.21.15.60:0/2748852057 >> v1:172.21.15.144:6789/0 conn(0x7f39940cbad0 legacy=0x7f399415af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f39937fe640 1 -- 172.21.15.60:0/2748852057 >> v1:172.21.15.60:6789/0 conn(0x7f399415abb0 legacy=0x7f39940b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.489+0000 7f39937fe640 1 -- 172.21.15.60:0/2748852057 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399415ba80 con 0x7f3994070240 2023-10-16T14:10:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.490+0000 7f39937fe640 1 --2- 172.21.15.60:0/2748852057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3994070240 0x7f3994070650 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f398c009870 tx=0x7f398c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb236ed5da0be6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.491+0000 7f3991ffb640 1 -- 172.21.15.60:0/2748852057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c031870 con 0x7f3994070240 2023-10-16T14:10:17.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.491+0000 7f3991ffb640 1 -- 172.21.15.60:0/2748852057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c0319d0 con 0x7f3994070240 2023-10-16T14:10:17.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.491+0000 7f3991ffb640 1 -- 172.21.15.60:0/2748852057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c031d60 con 0x7f3994070240 2023-10-16T14:10:17.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.491+0000 7f3993fff640 1 -- 172.21.15.60:0/2748852057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3994070240 msgr2=0x7f3994070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.491+0000 7f3993fff640 1 --2- 172.21.15.60:0/2748852057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3994070240 0x7f3994070650 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f398c009870 tx=0x7f398c02b910 comp rx=0 tx=0).stop 2023-10-16T14:10:17.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.492+0000 7f3993fff640 1 -- 172.21.15.60:0/2748852057 shutdown_connections 2023-10-16T14:10:17.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.492+0000 7f3993fff640 1 --2- 172.21.15.60:0/2748852057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3994070240 0x7f3994070650 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.492+0000 7f3993fff640 1 -- 172.21.15.60:0/2748852057 >> 172.21.15.60:0/2748852057 conn(0x7f39940cc230 msgr2=0x7f39940cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:17.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.492+0000 7f3993fff640 1 -- 172.21.15.60:0/2748852057 shutdown_connections 2023-10-16T14:10:17.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.492+0000 7f3993fff640 1 -- 172.21.15.60:0/2748852057 wait complete. 2023-10-16T14:10:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.493+0000 7f3993fff640 1 Processor -- start 2023-10-16T14:10:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 -- start start 2023-10-16T14:10:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3994070240 0x7f39940c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 0x7f39940c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f399415abb0 0x7f39940c3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39941f2660 con 0x7f39940cbad0 2023-10-16T14:10:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39940c7f00 con 0x7f399415abb0 2023-10-16T14:10:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.494+0000 7f3993fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39940c81e0 con 0x7f3994070240 2023-10-16T14:10:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 0x7f39940c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 0x7f39940c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43614/0 (socket says 172.21.15.60:43614) 2023-10-16T14:10:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 -- 172.21.15.60:0/3998367621 learned_addr learned my addr 172.21.15.60:0/3998367621 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39937fe640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f399415abb0 0x7f39940c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 -- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3994070240 msgr2=0x7f39940c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3994070240 0x7f39940c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 -- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f399415abb0 msgr2=0x7f39940c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f399415abb0 0x7f39940c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39927fc640 1 -- 172.21.15.60:0/3998367621 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c0094d0 con 0x7f39940cbad0 2023-10-16T14:10:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.495+0000 7f39937fe640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f399415abb0 0x7f39940c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.496+0000 7f39927fc640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 0x7f39940c3230 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f397c00ba50 tx=0x7f397c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.496+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c00cbe0 con 0x7f39940cbad0 2023-10-16T14:10:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.496+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c00cd80 con 0x7f39940cbad0 2023-10-16T14:10:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.496+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c014680 con 0x7f39940cbad0 2023-10-16T14:10:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.496+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39940c84c0 con 0x7f39940cbad0 2023-10-16T14:10:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.496+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39940c8a80 con 0x7f39940cbad0 2023-10-16T14:10:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.497+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399404e6d0 con 0x7f39940cbad0 2023-10-16T14:10:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.498+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f397c0147e0 con 0x7f39940cbad0 2023-10-16T14:10:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.499+0000 7f398b7fe640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f397406af90 0x7f397406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.499+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f397c0895a0 con 0x7f39940cbad0 2023-10-16T14:10:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.500+0000 7f3992ffd640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f397406af90 0x7f397406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.501+0000 7f3992ffd640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f397406af90 0x7f397406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3984000f60 tx=0x7f39840079b0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.503+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f397c054640 con 0x7f39940cbad0 2023-10-16T14:10:17.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.704+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39940c8ca0 con 0x7f39940cbad0 2023-10-16T14:10:17.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.706+0000 7f398b7fe640 1 -- 172.21.15.60:0/3998367621 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+13998 (secure 0 0 0) 0x7f397c057c60 con 0x7f39940cbad0 2023-10-16T14:10:17.710 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:17.710 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":223,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:15.203475+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6835/3999834969":"2023-10-17T14:10:12.186378+0000","172.21.15.144:6834/3999834969":"2023-10-17T14:10:12.186378+0000","172.21.15.144:0/652243704":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/2452241572":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/1049005214":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/4233146004":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/732610814":"2023-10-17T14:10:08.132792+0000","172.21.15.144:6833/166679164":"2023-10-17T14:10:08.132792+0000","172.21.15.144:6832/166679164":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/351085041":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/992081872":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/4042889662":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/301721702":"2023-10-17T14:10:08.132792+0000","172.21.15.144:0/166629007":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6832/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2940276091":"2023-10-17T14:07:58.997006+0000","172.21.15.144:6833/1274087552":"2023-10-17T14:07:58.997006+0000","172.21.15.144:0/2969783877":"2023-10-17T14:07:58.997006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:17.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f397406af90 msgr2=0x7f397406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f397406af90 0x7f397406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3984000f60 tx=0x7f39840079b0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 msgr2=0x7f39940c3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 0x7f39940c3230 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f397c00ba50 tx=0x7f397c00bf10 comp rx=0 tx=0).stop 2023-10-16T14:10:17.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 shutdown_connections 2023-10-16T14:10:17.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f397406af90 0x7f397406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f399415abb0 0x7f39940c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39940cbad0 0x7f39940c3230 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 --2- 172.21.15.60:0/3998367621 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3994070240 0x7f39940c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.708+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 >> 172.21.15.60:0/3998367621 conn(0x7f39940cc230 msgr2=0x7f399415e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:17.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.709+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 shutdown_connections 2023-10-16T14:10:17.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.709+0000 7f3993fff640 1 -- 172.21.15.60:0/3998367621 wait complete. 2023-10-16T14:10:17.732 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:10:17.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:10:17.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.961+0000 7f17e6ffb640 1 Processor -- start 2023-10-16T14:10:17.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.961+0000 7f17e6ffb640 1 -- start start 2023-10-16T14:10:17.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.963+0000 7f17e6ffb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 0x7f17e00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.963+0000 7f17e6ffb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e0070360 con 0x7f17e0159ab0 2023-10-16T14:10:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.963+0000 7f17e6ffb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e00cd1f0 con 0x7f17e006c140 2023-10-16T14:10:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.963+0000 7f17e6ffb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e015d890 con 0x7f17e0159e90 2023-10-16T14:10:17.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.963+0000 7f17e4d97640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f17e0159ab0 0x7f17e006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53000/0 (socket says 172.21.15.60:53000) 2023-10-16T14:10:17.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.963+0000 7f17e4d97640 1 -- 172.21.15.60:0/407059309 learned_addr learned my addr 172.21.15.60:0/407059309 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d7fff640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 0x7f17e00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005962105 0 0) 0x7f17e0070360 con 0x7f17e0159ab0 2023-10-16T14:10:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c8003520 con 0x7f17e0159ab0 2023-10-16T14:10:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1318145713 0 0) 0x7f17c8003520 con 0x7f17e0159ab0 2023-10-16T14:10:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 msgr2=0x7f17e00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 0x7f17e00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 >> v1:172.21.15.144:6789/0 conn(0x7f17e006c140 legacy=0x7f17e015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e00cd4d0 con 0x7f17e0159ab0 2023-10-16T14:10:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.964+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f17dc0019f0 con 0x7f17e0159ab0 2023-10-16T14:10:17.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 >> v1:172.21.15.60:6789/0 conn(0x7f17e0159ab0 legacy=0x7f17e006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 0x7f17c80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c8003f70 0x7f17c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c8004d00 0x7f17c8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e015d890 con 0x7f17c8003f70 2023-10-16T14:10:17.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c800bc10 con 0x7f17c8004d00 2023-10-16T14:10:17.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c800dfe0 con 0x7f17e0159e90 2023-10-16T14:10:17.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17d7fff640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c8003f70 0x7f17c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17e4d97640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 0x7f17c80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.965+0000 7f17e5598640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c8004d00 0x7f17c8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.966+0000 7f17d7fff640 1 -- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 msgr2=0x7f17c80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:17.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.966+0000 7f17d7fff640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17e0159e90 0x7f17c80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.966+0000 7f17d7fff640 1 -- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c8004d00 msgr2=0x7f17c8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:17.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.966+0000 7f17d7fff640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c8004d00 0x7f17c8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.966+0000 7f17d7fff640 1 -- 172.21.15.60:0/407059309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8003520 con 0x7f17c8003f70 2023-10-16T14:10:17.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.966+0000 7f17e5598640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c8004d00 0x7f17c8005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:17.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.967+0000 7f17d7fff640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c8003f70 0x7f17c80047c0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009ac0 tx=0x7f17c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3985e2c023bccb73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:17.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.967+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002c6b0 con 0x7f17c8003f70 2023-10-16T14:10:17.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.967+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d002c810 con 0x7f17c8003f70 2023-10-16T14:10:17.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.967+0000 7f17d77fe640 1 -- 172.21.15.60:0/407059309 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002cba0 con 0x7f17c8003f70 2023-10-16T14:10:17.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.967+0000 7f17e6ffb640 1 -- 172.21.15.60:0/407059309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c8003f70 msgr2=0x7f17c80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.967+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/407059309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17c8003f70 0x7f17c80047c0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009ac0 tx=0x7f17c80057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.968+0000 7f17e5598640 1 -- 172.21.15.60:0/407059309 reap_dead start 2023-10-16T14:10:17.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.968+0000 7f17e6ffb640 1 -- 172.21.15.60:0/407059309 shutdown_connections 2023-10-16T14:10:17.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.968+0000 7f17e6ffb640 1 -- 172.21.15.60:0/407059309 >> 172.21.15.60:0/407059309 conn(0x7f17e00ce9b0 msgr2=0x7f17e00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:17.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.968+0000 7f17e6ffb640 1 -- 172.21.15.60:0/407059309 shutdown_connections 2023-10-16T14:10:17.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.968+0000 7f17e6ffb640 1 -- 172.21.15.60:0/407059309 wait complete. 2023-10-16T14:10:17.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.970+0000 7f17e6ffb640 1 Processor -- start 2023-10-16T14:10:17.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.970+0000 7f17e6ffb640 1 -- start start 2023-10-16T14:10:17.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17e6ffb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c8003f70 0x7f17e013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17e6ffb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 0x7f17e015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17e6ffb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17e015d140 0x7f17e015a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17e6ffb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e015bd00 con 0x7f17e015ccc0 2023-10-16T14:10:17.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17e6ffb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e006b330 con 0x7f17e015d140 2023-10-16T14:10:17.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17e6ffb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e00c3030 con 0x7f17c8003f70 2023-10-16T14:10:17.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 0x7f17e015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 0x7f17e015a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43628/0 (socket says 172.21.15.60:43628) 2023-10-16T14:10:17.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.971+0000 7f17d7fff640 1 -- 172.21.15.60:0/4265051376 learned_addr learned my addr 172.21.15.60:0/4265051376 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:17.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17e4d97640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c8003f70 0x7f17e013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17e5598640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17e015d140 0x7f17e015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d7fff640 1 -- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c8003f70 msgr2=0x7f17e013e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d7fff640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c8003f70 0x7f17e013e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d7fff640 1 -- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17e015d140 msgr2=0x7f17e015a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:17.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d7fff640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17e015d140 0x7f17e015a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:17.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d7fff640 1 -- 172.21.15.60:0/4265051376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e00c3310 con 0x7f17e015ccc0 2023-10-16T14:10:17.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17e4d97640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c8003f70 0x7f17e013e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:17.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d7fff640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 0x7f17e015a330 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f17d002c250 tx=0x7f17d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:17.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0003810 con 0x7f17e015ccc0 2023-10-16T14:10:17.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0003970 con 0x7f17e015ccc0 2023-10-16T14:10:17.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0003ce0 con 0x7f17e015ccc0 2023-10-16T14:10:17.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.972+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e00c35f0 con 0x7f17e015ccc0 2023-10-16T14:10:17.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.973+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e00c3bb0 con 0x7f17e015ccc0 2023-10-16T14:10:17.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.974+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e00c3d40 con 0x7f17e015ccc0 2023-10-16T14:10:17.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.975+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f17d0034440 con 0x7f17e015ccc0 2023-10-16T14:10:17.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.976+0000 7f17d57fa640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17cc06af90 0x7f17cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:17.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.976+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f17d00abae0 con 0x7f17e015ccc0 2023-10-16T14:10:17.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.976+0000 7f17e4d97640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17cc06af90 0x7f17cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:17.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.977+0000 7f17e4d97640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17cc06af90 0x7f17cc06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f17dc002230 tx=0x7f17dc0012d0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:17.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:17.980+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17d0076ae0 con 0x7f17e015ccc0 2023-10-16T14:10:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.175+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17e00cda60 con 0x7f17e015ccc0 2023-10-16T14:10:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.176+0000 7f17d57fa640 1 -- 172.21.15.60:0/4265051376 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17d004c0a0 con 0x7f17e015ccc0 2023-10-16T14:10:18.180 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:10:18.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.179+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17cc06af90 msgr2=0x7f17cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:18.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.179+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17cc06af90 0x7f17cc06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f17dc002230 tx=0x7f17dc0012d0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.179+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 msgr2=0x7f17e015a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:18.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.179+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 0x7f17e015a330 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f17d002c250 tx=0x7f17d0004730 comp rx=0 tx=0).stop 2023-10-16T14:10:18.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 shutdown_connections 2023-10-16T14:10:18.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17cc06af90 0x7f17cc06d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17e015d140 0x7f17e015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17e015ccc0 0x7f17e015a330 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 --2- 172.21.15.60:0/4265051376 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f17c8003f70 0x7f17e013e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 >> 172.21.15.60:0/4265051376 conn(0x7f17e00ce9b0 msgr2=0x7f17e014db30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:18.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 shutdown_connections 2023-10-16T14:10:18.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.180+0000 7f17e6ffb640 1 -- 172.21.15.60:0/4265051376 wait complete. 2023-10-16T14:10:18.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T14:10:18.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.427+0000 7f713759e640 1 Processor -- start 2023-10-16T14:10:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.428+0000 7f713759e640 1 -- start start 2023-10-16T14:10:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f71380ad970 0x7f71380cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:18.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71380ccfb0 0x7f71380cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713815b9d0 con 0x7f713806c040 2023-10-16T14:10:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713815bcb0 con 0x7f71380ccfb0 2023-10-16T14:10:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713815bf90 con 0x7f71380ad970 2023-10-16T14:10:18.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f713806c040 0x7f713806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53006/0 (socket says 172.21.15.60:53006) 2023-10-16T14:10:18.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f713659c640 1 -- 172.21.15.60:0/2447645090 learned_addr learned my addr 172.21.15.60:0/2447645090 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.429+0000 7f7136d9d640 1 --2- 172.21.15.60:0/2447645090 >> v2:172.21.15.60:3300/0 conn(0x7f71380ccfb0 0x7f71380cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.430+0000 7f7135d9b640 1 --2- 172.21.15.60:0/2447645090 >> v2:172.21.15.144:3300/0 conn(0x7f71380ad970 0x7f71380cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.430+0000 7f7136d9d640 1 -- 172.21.15.60:0/2447645090 >> v2:172.21.15.144:3300/0 conn(0x7f71380ad970 msgr2=0x7f71380cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.430+0000 7f7136d9d640 1 --2- 172.21.15.60:0/2447645090 >> v2:172.21.15.144:3300/0 conn(0x7f71380ad970 0x7f71380cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.430+0000 7f7136d9d640 1 -- 172.21.15.60:0/2447645090 >> v1:172.21.15.60:6789/0 conn(0x7f713806c040 legacy=0x7f713806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:18.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.430+0000 7f7136d9d640 1 -- 172.21.15.60:0/2447645090 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71380cc270 con 0x7f71380ccfb0 2023-10-16T14:10:18.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.430+0000 7f713559a640 1 -- 172.21.15.60:0/2447645090 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627113539 0 0) 0x7f713815b9d0 con 0x7f713806c040 2023-10-16T14:10:18.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.431+0000 7f7136d9d640 1 --2- 172.21.15.60:0/2447645090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ccfb0 0x7f71380cd460 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f7130009d90 tx=0x7f713002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2490177c4dabe6a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:18.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.431+0000 7f713559a640 1 -- 172.21.15.60:0/2447645090 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713002cc60 con 0x7f71380ccfb0 2023-10-16T14:10:18.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.431+0000 7f713559a640 1 -- 172.21.15.60:0/2447645090 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713002cdc0 con 0x7f71380ccfb0 2023-10-16T14:10:18.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.431+0000 7f713559a640 1 -- 172.21.15.60:0/2447645090 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130034660 con 0x7f71380ccfb0 2023-10-16T14:10:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 -- 172.21.15.60:0/2447645090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ccfb0 msgr2=0x7f71380cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 --2- 172.21.15.60:0/2447645090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ccfb0 0x7f71380cd460 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f7130009d90 tx=0x7f713002beb0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 -- 172.21.15.60:0/2447645090 shutdown_connections 2023-10-16T14:10:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 --2- 172.21.15.60:0/2447645090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ccfb0 0x7f71380cd460 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 --2- 172.21.15.60:0/2447645090 >> v2:172.21.15.144:3300/0 conn(0x7f71380ad970 0x7f71380cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 -- 172.21.15.60:0/2447645090 >> 172.21.15.60:0/2447645090 conn(0x7f71380ce6a0 msgr2=0x7f71380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:18.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.432+0000 7f713759e640 1 -- 172.21.15.60:0/2447645090 shutdown_connections 2023-10-16T14:10:18.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.433+0000 7f713759e640 1 -- 172.21.15.60:0/2447645090 wait complete. 2023-10-16T14:10:18.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.434+0000 7f713759e640 1 Processor -- start 2023-10-16T14:10:18.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.435+0000 7f713759e640 1 -- start start 2023-10-16T14:10:18.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f713806c040 0x7f71380acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:18.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 0x7f71380c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:18.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71380ccfb0 0x7f71381f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:18.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71380b4320 con 0x7f71380ad970 2023-10-16T14:10:18.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71381f2dc0 con 0x7f713806c040 2023-10-16T14:10:18.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71381f30a0 con 0x7f71380ccfb0 2023-10-16T14:10:18.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 0x7f71380c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:18.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 0x7f71380c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43646/0 (socket says 172.21.15.60:43646) 2023-10-16T14:10:18.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 -- 172.21.15.60:0/3396728862 learned_addr learned my addr 172.21.15.60:0/3396728862 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:18.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713659c640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f713806c040 0x7f71380acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:18.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7136d9d640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71380ccfb0 0x7f71381f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:18.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 -- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71380ccfb0 msgr2=0x7f71381f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:18.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71380ccfb0 0x7f71381f21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 -- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f713806c040 msgr2=0x7f71380acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:18.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f713806c040 0x7f71380acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:18.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f7135d9b640 1 -- 172.21.15.60:0/3396728862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009a70 con 0x7f71380ad970 2023-10-16T14:10:18.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.436+0000 7f713659c640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f713806c040 0x7f71380acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:18.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.437+0000 7f7135d9b640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 0x7f71380c8cf0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f712000d7d0 tx=0x7f712000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:18.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.437+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120018070 con 0x7f71380ad970 2023-10-16T14:10:18.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.437+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120004020 con 0x7f71380ad970 2023-10-16T14:10:18.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.437+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71200041b0 con 0x7f71380ad970 2023-10-16T14:10:18.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.437+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71380ade50 con 0x7f71380ad970 2023-10-16T14:10:18.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.437+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71380ae410 con 0x7f71380ad970 2023-10-16T14:10:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.438+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713806b330 con 0x7f71380ad970 2023-10-16T14:10:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.439+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7120004310 con 0x7f71380ad970 2023-10-16T14:10:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.441+0000 7f712effd640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f711806af90 0x7f711806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.441+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f7120089660 con 0x7f71380ad970 2023-10-16T14:10:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.442+0000 7f713659c640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f711806af90 0x7f711806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:18.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.443+0000 7f713659c640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f711806af90 0x7f711806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7128000f60 tx=0x7f7128009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:18.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.446+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7120054640 con 0x7f71380ad970 2023-10-16T14:10:18.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:18.718+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f71380ae5a0 con 0x7f71380ad970 2023-10-16T14:10:19.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.239+0000 7f712effd640 1 -- 172.21.15.60:0/3396728862 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3165) v1 ==== 189+0+0 (secure 0 0 0) 0x7f7120057c60 con 0x7f71380ad970 2023-10-16T14:10:19.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.243+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f711806af90 msgr2=0x7f711806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.243+0000 7f713759e640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f711806af90 0x7f711806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7128000f60 tx=0x7f7128009040 comp rx=0 tx=0).stop 2023-10-16T14:10:19.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 msgr2=0x7f71380c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 0x7f71380c8cf0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f712000d7d0 tx=0x7f712000dc90 comp rx=0 tx=0).stop 2023-10-16T14:10:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 shutdown_connections 2023-10-16T14:10:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f711806af90 0x7f711806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71380ccfb0 0x7f71381f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71380ad970 0x7f71380c8cf0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 --2- 172.21.15.60:0/3396728862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f713806c040 0x7f71380acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 >> 172.21.15.60:0/3396728862 conn(0x7f71380ce6a0 msgr2=0x7f71380c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 shutdown_connections 2023-10-16T14:10:19.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.244+0000 7f713759e640 1 -- 172.21.15.60:0/3396728862 wait complete. 2023-10-16T14:10:19.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:10:19.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.506+0000 7f4e71890640 1 Processor -- start 2023-10-16T14:10:19.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.506+0000 7f4e71890640 1 -- start start 2023-10-16T14:10:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e71890640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e6c15ad70 0x7f4e6c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e71890640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4e6c0caac0 0x7f4e6c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e71890640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c0cb030 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e71890640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0c91c0 con 0x7f4e6c15ad70 2023-10-16T14:10:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e71890640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0c94a0 con 0x7f4e6c0caac0 2023-10-16T14:10:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e6affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e6c15ad70 0x7f4e6c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e6affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e6c15ad70 0x7f4e6c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35316/0 (socket says 172.21.15.60:35316) 2023-10-16T14:10:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.508+0000 7f4e6affd640 1 -- 172.21.15.60:0/3585241331 learned_addr learned my addr 172.21.15.60:0/3585241331 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:19.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.509+0000 7f4e6b7fe640 1 --2- 172.21.15.60:0/3585241331 >> v2:172.21.15.144:3300/0 conn(0x7f4e6c0caac0 0x7f4e6c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.509+0000 7f4e6affd640 1 -- 172.21.15.60:0/3585241331 >> v2:172.21.15.144:3300/0 conn(0x7f4e6c0caac0 msgr2=0x7f4e6c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.509+0000 7f4e6affd640 1 --2- 172.21.15.60:0/3585241331 >> v2:172.21.15.144:3300/0 conn(0x7f4e6c0caac0 0x7f4e6c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.509+0000 7f4e6affd640 1 -- 172.21.15.60:0/3585241331 >> v1:172.21.15.60:6789/0 conn(0x7f4e6c0c9ca0 legacy=0x7f4e6c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.509+0000 7f4e6affd640 1 -- 172.21.15.60:0/3585241331 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0c9780 con 0x7f4e6c15ad70 2023-10-16T14:10:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.510+0000 7f4e6affd640 1 --2- 172.21.15.60:0/3585241331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c15ad70 0x7f4e6c069da0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c06d090 tx=0x7f4e5c02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee85a8f05819add2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.510+0000 7f4e69ffb640 1 -- 172.21.15.60:0/3585241331 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c030870 con 0x7f4e6c15ad70 2023-10-16T14:10:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.510+0000 7f4e69ffb640 1 -- 172.21.15.60:0/3585241331 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c0309d0 con 0x7f4e6c15ad70 2023-10-16T14:10:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.510+0000 7f4e69ffb640 1 -- 172.21.15.60:0/3585241331 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c030d60 con 0x7f4e6c15ad70 2023-10-16T14:10:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 -- 172.21.15.60:0/3585241331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c15ad70 msgr2=0x7f4e6c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 --2- 172.21.15.60:0/3585241331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c15ad70 0x7f4e6c069da0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c06d090 tx=0x7f4e5c02b9b0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 -- 172.21.15.60:0/3585241331 shutdown_connections 2023-10-16T14:10:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 --2- 172.21.15.60:0/3585241331 >> v2:172.21.15.144:3300/0 conn(0x7f4e6c0caac0 0x7f4e6c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 --2- 172.21.15.60:0/3585241331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c15ad70 0x7f4e6c069da0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 -- 172.21.15.60:0/3585241331 >> 172.21.15.60:0/3585241331 conn(0x7f4e6c0ce9b0 msgr2=0x7f4e6c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 -- 172.21.15.60:0/3585241331 shutdown_connections 2023-10-16T14:10:19.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.511+0000 7f4e71890640 1 -- 172.21.15.60:0/3585241331 wait complete. 2023-10-16T14:10:19.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.513+0000 7f4e71890640 1 Processor -- start 2023-10-16T14:10:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.513+0000 7f4e71890640 1 -- start start 2023-10-16T14:10:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e71890640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 0x7f4e6c15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e71890640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e6c0caac0 0x7f4e6c150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e71890640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e6c15ad70 0x7f4e6c0ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e71890640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0cd600 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e71890640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0cdcb0 con 0x7f4e6c0caac0 2023-10-16T14:10:19.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e71890640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c150960 con 0x7f4e6c15ad70 2023-10-16T14:10:19.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 0x7f4e6c15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:19.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 0x7f4e6c15c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35322/0 (socket says 172.21.15.60:35322) 2023-10-16T14:10:19.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.514+0000 7f4e6affd640 1 -- 172.21.15.60:0/3223698407 learned_addr learned my addr 172.21.15.60:0/3223698407 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:19.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6a7fc640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e6c0caac0 0x7f4e6c150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:19.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6affd640 1 -- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e6c15ad70 msgr2=0x7f4e6c0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6affd640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e6c15ad70 0x7f4e6c0ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6affd640 1 -- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e6c0caac0 msgr2=0x7f4e6c150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6affd640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e6c0caac0 0x7f4e6c150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6affd640 1 -- 172.21.15.60:0/3223698407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0094d0 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e6affd640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 0x7f4e6c15c040 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c000c00 tx=0x7f4e5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:19.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c030840 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c039d90 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.515+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c0ce560 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.516+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c043630 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.516+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c0ccd60 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.517+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c04e160 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.518+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e5c02f3c0 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.519+0000 7f4e63fff640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4e4c06af90 0x7f4e4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.519+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f4e5c0abf80 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.519+0000 7f4e6a7fc640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4e4c06af90 0x7f4e4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.521+0000 7f4e6a7fc640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4e4c06af90 0x7f4e4c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e54007fd0 tx=0x7f4e54009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.523+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e5c076e90 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.719+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4e6c0ca810 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.727+0000 7f4e63fff640 1 -- 172.21.15.60:0/3223698407 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4e5c0340c0 con 0x7f4e6c0c9ca0 2023-10-16T14:10:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4e4c06af90 msgr2=0x7f4e4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4e4c06af90 0x7f4e4c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e54007fd0 tx=0x7f4e54009040 comp rx=0 tx=0).stop 2023-10-16T14:10:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 msgr2=0x7f4e6c15c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 0x7f4e6c15c040 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c000c00 tx=0x7f4e5c004730 comp rx=0 tx=0).stop 2023-10-16T14:10:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 shutdown_connections 2023-10-16T14:10:19.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4e4c06af90 0x7f4e4c06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4e6c15ad70 0x7f4e6c0ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e6c0caac0 0x7f4e6c150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 --2- 172.21.15.60:0/3223698407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e6c0c9ca0 0x7f4e6c15c040 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:19.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 >> 172.21.15.60:0/3223698407 conn(0x7f4e6c0ce9b0 msgr2=0x7f4e6c0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:19.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 shutdown_connections 2023-10-16T14:10:19.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:19.730+0000 7f4e71890640 1 -- 172.21.15.60:0/3223698407 wait complete. 2023-10-16T14:10:19.754 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:10:19.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:10:19.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:10:20.003 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:10:20.003 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:10:20.004 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:20.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:10:20.069 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:10:20.069 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:20.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:10:20.130 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:10:20.131 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:20.194 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:10:20.195 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:10:20.195 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:20.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:10:20.257 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:10:20.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:20.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.491+0000 7f1b4a0bb640 1 Processor -- start 2023-10-16T14:10:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.492+0000 7f1b4a0bb640 1 -- start start 2023-10-16T14:10:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.493+0000 7f1b4a0bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 0x7f1b4406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.493+0000 7f1b4a0bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4415f4a0 con 0x7f1b440cda80 2023-10-16T14:10:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.493+0000 7f1b4a0bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4406c960 con 0x7f1b440ad970 2023-10-16T14:10:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.494+0000 7f1b4a0bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415b9d0 con 0x7f1b4406c040 2023-10-16T14:10:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.494+0000 7f1b498ba640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1b440cda80 0x7f1b4415ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52802/0 (socket says 172.21.15.60:52802) 2023-10-16T14:10:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.494+0000 7f1b498ba640 1 -- 172.21.15.60:0/3976577072 learned_addr learned my addr 172.21.15.60:0/3976577072 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:20.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.494+0000 7f1b490b9640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 0x7f1b4406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.494+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087107667 0 0) 0x7f1b4415f4a0 con 0x7f1b440cda80 2023-10-16T14:10:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.494+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30003520 con 0x7f1b440cda80 2023-10-16T14:10:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326653328 0 0) 0x7f1b30003520 con 0x7f1b440cda80 2023-10-16T14:10:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 msgr2=0x7f1b4406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 0x7f1b4406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 >> v1:172.21.15.144:6789/0 conn(0x7f1b440ad970 legacy=0x7f1b4415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4415bcb0 con 0x7f1b440cda80 2023-10-16T14:10:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1b3c0019f0 con 0x7f1b440cda80 2023-10-16T14:10:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 >> v1:172.21.15.60:6789/0 conn(0x7f1b440cda80 legacy=0x7f1b4415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b300039a0 0x7f1b30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b300047c0 0x7f1b30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 0x7f1b30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415b9d0 con 0x7f1b300047c0 2023-10-16T14:10:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3000ba60 con 0x7f1b300039a0 2023-10-16T14:10:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3000bc20 con 0x7f1b4406c040 2023-10-16T14:10:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.495+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b300047c0 0x7f1b30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b498ba640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 0x7f1b30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b490b9640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b300039a0 0x7f1b30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b488b8640 1 -- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 msgr2=0x7f1b30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b4406c040 0x7f1b30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b488b8640 1 -- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b300039a0 msgr2=0x7f1b30003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b300039a0 0x7f1b30003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b488b8640 1 -- 172.21.15.60:0/3976577072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30003520 con 0x7f1b300047c0 2023-10-16T14:10:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.496+0000 7f1b490b9640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b300039a0 0x7f1b30003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.497+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b300047c0 0x7f1b30004c40 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f1b34001720 tx=0x7f1b300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74801032b550877d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.497+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b340052d0 con 0x7f1b300047c0 2023-10-16T14:10:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.497+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3402e050 con 0x7f1b300047c0 2023-10-16T14:10:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.497+0000 7f1b3bfff640 1 -- 172.21.15.60:0/3976577072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b340035f0 con 0x7f1b300047c0 2023-10-16T14:10:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3976577072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b300047c0 msgr2=0x7f1b30004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3976577072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b300047c0 0x7f1b30004c40 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f1b34001720 tx=0x7f1b300057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b498ba640 1 -- 172.21.15.60:0/3976577072 reap_dead start 2023-10-16T14:10:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3976577072 shutdown_connections 2023-10-16T14:10:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3976577072 >> 172.21.15.60:0/3976577072 conn(0x7f1b440ce6a0 msgr2=0x7f1b440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3976577072 shutdown_connections 2023-10-16T14:10:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.498+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3976577072 wait complete. 2023-10-16T14:10:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.500+0000 7f1b4a0bb640 1 Processor -- start 2023-10-16T14:10:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.500+0000 7f1b4a0bb640 1 -- start start 2023-10-16T14:10:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b4a0bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b300047c0 0x7f1b441f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b4a0bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 0x7f1b440c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b4a0bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b440c7bb0 0x7f1b440c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b4a0bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440c85a0 con 0x7f1b441484d0 2023-10-16T14:10:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b4a0bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b441f2180 con 0x7f1b440c7bb0 2023-10-16T14:10:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b4a0bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b441f2460 con 0x7f1b300047c0 2023-10-16T14:10:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b488b8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 0x7f1b440c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b488b8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 0x7f1b440c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35350/0 (socket says 172.21.15.60:35350) 2023-10-16T14:10:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.501+0000 7f1b488b8640 1 -- 172.21.15.60:0/3322740064 learned_addr learned my addr 172.21.15.60:0/3322740064 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b498ba640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b440c7bb0 0x7f1b440c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b488b8640 1 -- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b300047c0 msgr2=0x7f1b441f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b490b9640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b300047c0 0x7f1b441f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b300047c0 0x7f1b441f1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b488b8640 1 -- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b440c7bb0 msgr2=0x7f1b440c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b440c7bb0 0x7f1b440c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b488b8640 1 -- 172.21.15.60:0/3322740064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b441f2740 con 0x7f1b441484d0 2023-10-16T14:10:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b488b8640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 0x7f1b440c59e0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f1b34008fd0 tx=0x7f1b34001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.502+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b340076c0 con 0x7f1b441484d0 2023-10-16T14:10:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.503+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b34007a30 con 0x7f1b441484d0 2023-10-16T14:10:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.503+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b34006be0 con 0x7f1b441484d0 2023-10-16T14:10:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.503+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b44155840 con 0x7f1b441484d0 2023-10-16T14:10:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.503+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b441458b0 con 0x7f1b441484d0 2023-10-16T14:10:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.504+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4406b330 con 0x7f1b441484d0 2023-10-16T14:10:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.505+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1b34006d40 con 0x7f1b441484d0 2023-10-16T14:10:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.506+0000 7f1b39ffb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1b2806afc0 0x7f1b2806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.506+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7f1b340aba30 con 0x7f1b441484d0 2023-10-16T14:10:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.506+0000 7f1b490b9640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1b2806afc0 0x7f1b2806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.508+0000 7f1b490b9640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1b2806afc0 0x7f1b2806d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b400099c0 tx=0x7f1b40008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.510+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b34076870 con 0x7f1b441484d0 2023-10-16T14:10:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.745+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b441f25a0 con 0x7f1b441484d0 2023-10-16T14:10:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.745+0000 7f1b39ffb640 1 -- 172.21.15.60:0/3322740064 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3701 (secure 0 0 0) 0x7f1b441f25a0 con 0x7f1b441484d0 2023-10-16T14:10:20.749 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:20.750 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":106,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2507854648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2507854648},{"type":"v1","addr":"172.21.15.60:6835","nonce":2507854648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/450864295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":450864295},{"type":"v1","addr":"172.21.15.144:6837","nonce":450864295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":7763,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/184100955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":184100955},{"type":"v1","addr":"172.21.15.144:6835","nonce":184100955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3199177005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3199177005},{"type":"v1","addr":"172.21.15.60:6834","nonce":3199177005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[]} 2023-10-16T14:10:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 106 2023-10-16T14:10:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1b2806afc0 msgr2=0x7f1b2806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1b2806afc0 0x7f1b2806d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b400099c0 tx=0x7f1b40008040 comp rx=0 tx=0).stop 2023-10-16T14:10:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 msgr2=0x7f1b440c59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 0x7f1b440c59e0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f1b34008fd0 tx=0x7f1b34001120 comp rx=0 tx=0).stop 2023-10-16T14:10:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 shutdown_connections 2023-10-16T14:10:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1b2806afc0 0x7f1b2806d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1b440c7bb0 0x7f1b440c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b441484d0 0x7f1b440c59e0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 --2- 172.21.15.60:0/3322740064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1b300047c0 0x7f1b441f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 >> 172.21.15.60:0/3322740064 conn(0x7f1b440ce6a0 msgr2=0x7f1b44158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 shutdown_connections 2023-10-16T14:10:20.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:20.748+0000 7f1b4a0bb640 1 -- 172.21.15.60:0/3322740064 wait complete. 2023-10-16T14:10:20.772 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:10:20.772 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:10:20.772 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:10:20.773 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:10:20.773 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:10:20.773 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:10:20.773 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:10:20.774 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:10:20.774 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:10:20.774 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:10:20.775 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:10:20.775 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:10:20.775 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:10:20.775 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:10:20.776 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:10:20.776 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:10:20.794 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:10:20.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:10:20.795 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:10:20.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:10:20.797 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:10:20.797 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:10:20.798 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:10:20.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:10:20.845 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:10:20.846 DEBUG:teuthology.parallel:result is None 2023-10-16T14:10:20.847 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:10:20.847 DEBUG:teuthology.parallel:result is None 2023-10-16T14:10:20.852 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:10:20.852 DEBUG:teuthology.parallel:result is None 2023-10-16T14:10:20.855 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:10:20.856 DEBUG:teuthology.parallel:result is None 2023-10-16T14:10:20.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:10:21.011 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:10:21.014 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:10:21.021 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T14:10:21.022 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:10:21.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.053+0000 7fe47dcf2640 1 Processor -- start 2023-10-16T14:10:21.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.053+0000 7fe47dcf2640 1 -- start start 2023-10-16T14:10:21.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.054+0000 7fe47dcf2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe4780cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:21.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe47dcf2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe47806b5c0 0x7fe47806ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:21.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe47dcf2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780c3610 con 0x7fe4780ca780 2023-10-16T14:10:21.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe47dcf2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47815a220 con 0x7fe47806b5c0 2023-10-16T14:10:21.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe47dcf2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780cafd0 con 0x7fe478159e40 2023-10-16T14:10:21.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe46ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe4780ca780 0x7fe47806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52810/0 (socket says 172.21.15.60:52810) 2023-10-16T14:10:21.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe46ffff640 1 -- 172.21.15.60:0/4245687782 learned_addr learned my addr 172.21.15.60:0/4245687782 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:21.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe4777fe640 1 --2- 172.21.15.60:0/4245687782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe4780cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:21.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe477fff640 1 --2- 172.21.15.60:0/4245687782 >> v2:172.21.15.60:3300/0 conn(0x7fe47806b5c0 0x7fe47806ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:21.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe476ffd640 1 -- 172.21.15.60:0/4245687782 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711192004 0 0) 0x7fe4780c3610 con 0x7fe4780ca780 2023-10-16T14:10:21.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.055+0000 7fe476ffd640 1 -- 172.21.15.60:0/4245687782 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe464003520 con 0x7fe4780ca780 2023-10-16T14:10:21.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.056+0000 7fe477fff640 1 -- 172.21.15.60:0/4245687782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 msgr2=0x7fe4780cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:21.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.056+0000 7fe477fff640 1 --2- 172.21.15.60:0/4245687782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe4780cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.056+0000 7fe477fff640 1 -- 172.21.15.60:0/4245687782 >> v1:172.21.15.60:6789/0 conn(0x7fe4780ca780 legacy=0x7fe47806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:21.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.056+0000 7fe477fff640 1 -- 172.21.15.60:0/4245687782 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780cb2b0 con 0x7fe47806b5c0 2023-10-16T14:10:21.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe477fff640 1 --2- 172.21.15.60:0/4245687782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe47806b5c0 0x7fe47806ccc0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fe47000bfa0 tx=0x7fe47002e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5069f80b78e75b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:21.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe476ffd640 1 -- 172.21.15.60:0/4245687782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47002ea40 con 0x7fe47806b5c0 2023-10-16T14:10:21.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe476ffd640 1 -- 172.21.15.60:0/4245687782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47002eba0 con 0x7fe47806b5c0 2023-10-16T14:10:21.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe476ffd640 1 -- 172.21.15.60:0/4245687782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470009540 con 0x7fe47806b5c0 2023-10-16T14:10:21.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe47dcf2640 1 -- 172.21.15.60:0/4245687782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe47806b5c0 msgr2=0x7fe47806ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/4245687782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe47806b5c0 0x7fe47806ccc0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fe47000bfa0 tx=0x7fe47002e0a0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe47dcf2640 1 -- 172.21.15.60:0/4245687782 shutdown_connections 2023-10-16T14:10:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/4245687782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe47806b5c0 0x7fe47806ccc0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/4245687782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe4780cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.057+0000 7fe47dcf2640 1 -- 172.21.15.60:0/4245687782 >> 172.21.15.60:0/4245687782 conn(0x7fe4780cfac0 msgr2=0x7fe4780cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.058+0000 7fe47dcf2640 1 -- 172.21.15.60:0/4245687782 shutdown_connections 2023-10-16T14:10:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.058+0000 7fe47dcf2640 1 -- 172.21.15.60:0/4245687782 wait complete. 2023-10-16T14:10:21.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.059+0000 7fe47dcf2640 1 Processor -- start 2023-10-16T14:10:21.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.059+0000 7fe47dcf2640 1 -- start start 2023-10-16T14:10:21.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.060+0000 7fe47dcf2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe47806b5c0 0x7fe4780c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:21.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.060+0000 7fe47dcf2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 0x7fe4781e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:21.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.060+0000 7fe47dcf2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe47814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:21.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.060+0000 7fe47dcf2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c3e20 con 0x7fe4780ca780 2023-10-16T14:10:21.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.060+0000 7fe47dcf2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781e5d00 con 0x7fe47806b5c0 2023-10-16T14:10:21.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.060+0000 7fe47dcf2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781e5fe0 con 0x7fe478159e40 2023-10-16T14:10:21.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 0x7fe4781e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 0x7fe4781e6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35406/0 (socket says 172.21.15.60:35406) 2023-10-16T14:10:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 -- 172.21.15.60:0/3259899461 learned_addr learned my addr 172.21.15.60:0/3259899461 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe477fff640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe47814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe4777fe640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe47806b5c0 0x7fe4780c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 -- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 msgr2=0x7fe47814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe47814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 -- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe47806b5c0 msgr2=0x7fe4780c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe47806b5c0 0x7fe4780c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 -- 172.21.15.60:0/3259899461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47000bc80 con 0x7fe4780ca780 2023-10-16T14:10:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.061+0000 7fe46ffff640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 0x7fe4781e6d80 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c410 tx=0x7fe46000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.062+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460005ce0 con 0x7fe4780ca780 2023-10-16T14:10:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.062+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460005e40 con 0x7fe4780ca780 2023-10-16T14:10:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.062+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600072f0 con 0x7fe4780ca780 2023-10-16T14:10:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.062+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4781e62c0 con 0x7fe4780ca780 2023-10-16T14:10:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.062+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4781e6880 con 0x7fe4780ca780 2023-10-16T14:10:21.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.063+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4781e6a10 con 0x7fe4780ca780 2023-10-16T14:10:21.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.064+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe460007510 con 0x7fe4780ca780 2023-10-16T14:10:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.065+0000 7fe474ff9640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe45806af90 0x7fe45806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.065+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5631+0+0 (secure 0 0 0) 0x7fe460088e00 con 0x7fe4780ca780 2023-10-16T14:10:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.065+0000 7fe4777fe640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe45806af90 0x7fe45806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.066+0000 7fe4777fe640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe45806af90 0x7fe45806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe468009b40 tx=0x7fe468009190 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:21.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.070+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe460053ec0 con 0x7fe4780ca780 2023-10-16T14:10:21.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.278+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe47804e160 con 0x7fe4780ca780 2023-10-16T14:10:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.752+0000 7fe474ff9640 1 -- 172.21.15.60:0/3259899461 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v225) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe460017080 con 0x7fe4780ca780 2023-10-16T14:10:21.756 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T14:10:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe45806af90 msgr2=0x7fe45806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe45806af90 0x7fe45806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe468009b40 tx=0x7fe468009190 comp rx=0 tx=0).stop 2023-10-16T14:10:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 msgr2=0x7fe4781e6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 0x7fe4781e6d80 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c410 tx=0x7fe46000c8d0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 shutdown_connections 2023-10-16T14:10:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe45806af90 0x7fe45806d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe478159e40 0x7fe47814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4780ca780 0x7fe4781e6d80 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 --2- 172.21.15.60:0/3259899461 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe47806b5c0 0x7fe4780c38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 >> 172.21.15.60:0/3259899461 conn(0x7fe4780cfac0 msgr2=0x7fe478069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.756+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 shutdown_connections 2023-10-16T14:10:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.757+0000 7fe47dcf2640 1 -- 172.21.15.60:0/3259899461 wait complete. 2023-10-16T14:10:21.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:10:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:21.999+0000 7f6fc19cc640 1 Processor -- start 2023-10-16T14:10:22.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.000+0000 7f6fc19cc640 1 -- start start 2023-10-16T14:10:22.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fc19cc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6fbc15aaa0 0x7f6fbc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fc19cc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6fbc0ce390 0x7f6fbc070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fc19cc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc15ed60 con 0x7f6fbc15ae80 2023-10-16T14:10:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fc19cc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc06b810 con 0x7f6fbc0ce390 2023-10-16T14:10:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fc19cc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc0707e0 con 0x7f6fbc15aaa0 2023-10-16T14:10:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fbb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6fbc0ce390 0x7f6fbc070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.001+0000 7f6fbb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6fbc0ce390 0x7f6fbc070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35420/0 (socket says 172.21.15.60:35420) 2023-10-16T14:10:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.002+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2751160682 learned_addr learned my addr 172.21.15.60:0/2751160682 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.002+0000 7f6fbaffd640 1 --2- 172.21.15.60:0/2751160682 >> v2:172.21.15.144:3300/0 conn(0x7f6fbc15aaa0 0x7f6fbc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.002+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2751160682 >> v2:172.21.15.144:3300/0 conn(0x7f6fbc15aaa0 msgr2=0x7f6fbc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.002+0000 7f6fbb7fe640 1 --2- 172.21.15.60:0/2751160682 >> v2:172.21.15.144:3300/0 conn(0x7f6fbc15aaa0 0x7f6fbc15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.002+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2751160682 >> v1:172.21.15.60:6789/0 conn(0x7f6fbc15ae80 legacy=0x7f6fbc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.002+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2751160682 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc15ef90 con 0x7f6fbc0ce390 2023-10-16T14:10:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.003+0000 7f6fbb7fe640 1 --2- 172.21.15.60:0/2751160682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc0ce390 0x7f6fbc070f50 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0cd990 tx=0x7f6fb402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1cf86769809822 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.003+0000 7f6fb9ffb640 1 -- 172.21.15.60:0/2751160682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4030870 con 0x7f6fbc0ce390 2023-10-16T14:10:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.003+0000 7f6fb9ffb640 1 -- 172.21.15.60:0/2751160682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb40309d0 con 0x7f6fbc0ce390 2023-10-16T14:10:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.003+0000 7f6fb9ffb640 1 -- 172.21.15.60:0/2751160682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4030d60 con 0x7f6fbc0ce390 2023-10-16T14:10:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2751160682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc0ce390 msgr2=0x7f6fbc070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2751160682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc0ce390 0x7f6fbc070f50 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0cd990 tx=0x7f6fb402b9b0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2751160682 shutdown_connections 2023-10-16T14:10:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2751160682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc0ce390 0x7f6fbc070f50 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2751160682 >> v2:172.21.15.144:3300/0 conn(0x7f6fbc15aaa0 0x7f6fbc15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2751160682 >> 172.21.15.60:0/2751160682 conn(0x7f6fbc0cb410 msgr2=0x7f6fbc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2751160682 shutdown_connections 2023-10-16T14:10:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.004+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2751160682 wait complete. 2023-10-16T14:10:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.006+0000 7f6fc19cc640 1 Processor -- start 2023-10-16T14:10:22.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.006+0000 7f6fc19cc640 1 -- start start 2023-10-16T14:10:22.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fc19cc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fbc0ce390 0x7f6fbc15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fc19cc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fbc15aaa0 0x7f6fbc14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fc19cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 0x7f6fbc0afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fc19cc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc15c060 con 0x7f6fbc15ae80 2023-10-16T14:10:22.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fc19cc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc15b450 con 0x7f6fbc15aaa0 2023-10-16T14:10:22.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fc19cc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc15b730 con 0x7f6fbc0ce390 2023-10-16T14:10:22.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fbb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 0x7f6fbc0afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fbb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 0x7f6fbc0afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35428/0 (socket says 172.21.15.60:35428) 2023-10-16T14:10:22.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.007+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2670782680 learned_addr learned my addr 172.21.15.60:0/2670782680 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbaffd640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fbc0ce390 0x7f6fbc15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fbc0ce390 msgr2=0x7f6fbc15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fba7fc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fbc15aaa0 0x7f6fbc14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbb7fe640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fbc0ce390 0x7f6fbc15c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fbc15aaa0 msgr2=0x7f6fbc14eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbb7fe640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fbc15aaa0 0x7f6fbc14eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbb7fe640 1 -- 172.21.15.60:0/2670782680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb40094d0 con 0x7f6fbc15ae80 2023-10-16T14:10:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fbb7fe640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 0x7f6fbc0afdc0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f6fb402b9f0 tx=0x7f6fb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4030800 con 0x7f6fbc15ae80 2023-10-16T14:10:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb4043400 con 0x7f6fbc15ae80 2023-10-16T14:10:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40437b0 con 0x7f6fbc15ae80 2023-10-16T14:10:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.008+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc15ba10 con 0x7f6fbc15ae80 2023-10-16T14:10:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.009+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc14f040 con 0x7f6fbc15ae80 2023-10-16T14:10:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.010+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc15c210 con 0x7f6fbc15ae80 2023-10-16T14:10:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.011+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6fb4043910 con 0x7f6fbc15ae80 2023-10-16T14:10:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.011+0000 7f6fb3fff640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6f9c06afc0 0x7f6f9c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.012+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6fb403e070 con 0x7f6fbc15ae80 2023-10-16T14:10:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.012+0000 7f6fbaffd640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6f9c06afc0 0x7f6f9c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.013+0000 7f6fbaffd640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6f9c06afc0 0x7f6f9c06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6fac008ef0 tx=0x7f6fac008440 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.016+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fbc15c210 con 0x7f6fbc15ae80 2023-10-16T14:10:22.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.272+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6fbc04e160 con 0x7f6fbc15ae80 2023-10-16T14:10:22.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.273+0000 7f6fb3fff640 1 -- 172.21.15.60:0/2670782680 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v55) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6fb4077a70 con 0x7f6fbc15ae80 2023-10-16T14:10:22.277 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:22.277 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:10:22.277 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T14:10:22.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.278 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T14:10:22.278 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T14:10:22.278 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.278 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.278 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:10:22.279 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:10:22.279 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.279 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T14:10:22.280 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T14:10:22.280 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.280 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.280 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:10:22.280 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:10:22.281 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.281 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.282 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T14:10:22.282 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T14:10:22.282 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:10:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:10:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.284 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T14:10:22.286 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T14:10:22.286 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.287 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.287 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:10:22.287 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:10:22.287 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.288 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T14:10:22.288 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T14:10:22.288 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.289 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.289 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.289 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.290 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T14:10:22.290 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T14:10:22.290 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.290 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.291 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.291 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.291 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.292 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T14:10:22.292 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T14:10:22.292 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.292 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.292 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.292 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T14:10:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T14:10:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T14:10:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T14:10:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.297 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T14:10:22.297 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T14:10:22.297 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.297 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.297 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.298 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.298 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T14:10:22.299 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T14:10:22.299 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.299 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.299 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.299 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.300 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T14:10:22.300 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T14:10:22.300 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.301 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:10:22.301 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:10:22.301 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.302 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T14:10:22.302 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T14:10:22.302 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.302 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.303 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T14:10:22.303 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:10:22.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.304 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T14:10:22.304 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T14:10:22.304 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.304 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.304 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:10:22.304 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:10:22.305 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:10:22.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.305 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T14:10:22.306 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T14:10:22.306 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:10:22.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:10:22.307 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:10:22.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.307 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T14:10:22.307 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T14:10:22.308 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.308 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:10:22.308 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:10:22.308 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:10:22.308 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:10:22.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.309 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T14:10:22.309 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T14:10:22.309 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.310 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:10:22.310 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:10:22.310 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:10:22.310 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.311 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T14:10:22.311 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T14:10:22.311 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.312 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:10:22.312 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:10:22.312 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:22.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T14:10:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T14:10:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:10:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:10:22.313 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:10:22.314 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:10:22.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:22.314 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T14:10:22.314 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:10:22.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.275+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6f9c06afc0 msgr2=0x7f6f9c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.275+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6f9c06afc0 0x7f6f9c06d480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6fac008ef0 tx=0x7f6fac008440 comp rx=0 tx=0).stop 2023-10-16T14:10:22.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.275+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 msgr2=0x7f6fbc0afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 0x7f6fbc0afdc0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f6fb402b9f0 tx=0x7f6fb4004730 comp rx=0 tx=0).stop 2023-10-16T14:10:22.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 shutdown_connections 2023-10-16T14:10:22.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6f9c06afc0 0x7f6f9c06d480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fbc15ae80 0x7f6fbc0afdc0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6fbc15aaa0 0x7f6fbc14eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 --2- 172.21.15.60:0/2670782680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6fbc0ce390 0x7f6fbc15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 >> 172.21.15.60:0/2670782680 conn(0x7f6fbc0cb410 msgr2=0x7f6fbc0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:22.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 shutdown_connections 2023-10-16T14:10:22.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.276+0000 7f6fc19cc640 1 -- 172.21.15.60:0/2670782680 wait complete. 2023-10-16T14:10:22.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:22.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.540+0000 7f824a380640 1 Processor -- start 2023-10-16T14:10:22.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.541+0000 7f824a380640 1 -- start start 2023-10-16T14:10:22.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.542+0000 7f824a380640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f824406c040 0x7f824406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.542+0000 7f824a380640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82440cda80 0x7f824415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.542+0000 7f824a380640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824406c960 con 0x7f82440ad970 2023-10-16T14:10:22.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.542+0000 7f824a380640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824415b9d0 con 0x7f82440cda80 2023-10-16T14:10:22.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.542+0000 7f824a380640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824415bcb0 con 0x7f824406c040 2023-10-16T14:10:22.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8249b7f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82440cda80 0x7f824415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8248b7d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82440ad970 0x7f824415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52826/0 (socket says 172.21.15.60:52826) 2023-10-16T14:10:22.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8248b7d640 1 -- 172.21.15.60:0/2194974691 learned_addr learned my addr 172.21.15.60:0/2194974691 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:22.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f824937e640 1 --2- 172.21.15.60:0/2194974691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f824406c040 0x7f824406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8249b7f640 1 -- 172.21.15.60:0/2194974691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f824406c040 msgr2=0x7f824406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:22.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8249b7f640 1 --2- 172.21.15.60:0/2194974691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f824406c040 0x7f824406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8249b7f640 1 -- 172.21.15.60:0/2194974691 >> v1:172.21.15.60:6789/0 conn(0x7f82440ad970 legacy=0x7f824415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f8249b7f640 1 -- 172.21.15.60:0/2194974691 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824415bf90 con 0x7f82440cda80 2023-10-16T14:10:22.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.543+0000 7f823bfff640 1 -- 172.21.15.60:0/2194974691 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249885970 0 0) 0x7f824406c960 con 0x7f82440ad970 2023-10-16T14:10:22.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.544+0000 7f8249b7f640 1 --2- 172.21.15.60:0/2194974691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824415ed90 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f823c009d90 tx=0x7f823c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65113cb709447a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:22.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f823bfff640 1 -- 172.21.15.60:0/2194974691 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c02cc60 con 0x7f82440cda80 2023-10-16T14:10:22.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f823bfff640 1 -- 172.21.15.60:0/2194974691 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c02cdc0 con 0x7f82440cda80 2023-10-16T14:10:22.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f823bfff640 1 -- 172.21.15.60:0/2194974691 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c034660 con 0x7f82440cda80 2023-10-16T14:10:22.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 -- 172.21.15.60:0/2194974691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 msgr2=0x7f824415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 --2- 172.21.15.60:0/2194974691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824415ed90 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f823c009d90 tx=0x7f823c02beb0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 -- 172.21.15.60:0/2194974691 shutdown_connections 2023-10-16T14:10:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 --2- 172.21.15.60:0/2194974691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824415ed90 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 --2- 172.21.15.60:0/2194974691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f824406c040 0x7f824406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 -- 172.21.15.60:0/2194974691 >> 172.21.15.60:0/2194974691 conn(0x7f82440ce6a0 msgr2=0x7f82440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 -- 172.21.15.60:0/2194974691 shutdown_connections 2023-10-16T14:10:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.545+0000 7f824a380640 1 -- 172.21.15.60:0/2194974691 wait complete. 2023-10-16T14:10:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.547+0000 7f824a380640 1 Processor -- start 2023-10-16T14:10:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.547+0000 7f824a380640 1 -- start start 2023-10-16T14:10:22.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f824a380640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f824406c040 0x7f8244153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f824a380640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82440ad970 0x7f8244150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f824a380640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824414f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f824a380640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244157e10 con 0x7f82440cda80 2023-10-16T14:10:22.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f824a380640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244151520 con 0x7f824406c040 2023-10-16T14:10:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f824a380640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8244151800 con 0x7f82440ad970 2023-10-16T14:10:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f8249b7f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824414f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f8249b7f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824414f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35448/0 (socket says 172.21.15.60:35448) 2023-10-16T14:10:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.548+0000 7f8249b7f640 1 -- 172.21.15.60:0/691900373 learned_addr learned my addr 172.21.15.60:0/691900373 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f824937e640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f824406c040 0x7f8244153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8249b7f640 1 -- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82440ad970 msgr2=0x7f8244150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8249b7f640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82440ad970 0x7f8244150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8249b7f640 1 -- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f824406c040 msgr2=0x7f8244153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8249b7f640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f824406c040 0x7f8244153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8249b7f640 1 -- 172.21.15.60:0/691900373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c009a70 con 0x7f82440cda80 2023-10-16T14:10:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8249b7f640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824414f9e0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f823c009d90 tx=0x7f823c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c03e610 con 0x7f82440cda80 2023-10-16T14:10:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c03e790 con 0x7f82440cda80 2023-10-16T14:10:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c03eb40 con 0x7f82440cda80 2023-10-16T14:10:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.549+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8244151ae0 con 0x7f82440cda80 2023-10-16T14:10:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.550+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82441539a0 con 0x7f82440cda80 2023-10-16T14:10:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.551+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824406b330 con 0x7f82440cda80 2023-10-16T14:10:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.551+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f823c03b050 con 0x7f82440cda80 2023-10-16T14:10:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.552+0000 7f8239ffb640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f822806afc0 0x7f822806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.553+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f823c0abba0 con 0x7f82440cda80 2023-10-16T14:10:22.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.553+0000 7f824937e640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f822806afc0 0x7f822806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.554+0000 7f824937e640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f822806afc0 0x7f822806d480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8240006fd0 tx=0x7f8240008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:22.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.557+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f823c076df0 con 0x7f82440cda80 2023-10-16T14:10:22.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.759+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8244153b30 con 0x7f82440cda80 2023-10-16T14:10:22.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.760+0000 7f8239ffb640 1 -- 172.21.15.60:0/691900373 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+12855 (secure 0 0 0) 0x7f823c039090 con 0x7f82440cda80 2023-10-16T14:10:22.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:22.765 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":227,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:21.765176+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.763+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f822806afc0 msgr2=0x7f822806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.763+0000 7f824a380640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f822806afc0 0x7f822806d480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8240006fd0 tx=0x7f8240008040 comp rx=0 tx=0).stop 2023-10-16T14:10:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.763+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 msgr2=0x7f824414f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.763+0000 7f824a380640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824414f9e0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f823c009d90 tx=0x7f823c004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 shutdown_connections 2023-10-16T14:10:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f822806afc0 0x7f822806d480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82440ad970 0x7f8244150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82440cda80 0x7f824414f9e0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 --2- 172.21.15.60:0/691900373 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f824406c040 0x7f8244153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 >> 172.21.15.60:0/691900373 conn(0x7f82440ce6a0 msgr2=0x7f824415c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 shutdown_connections 2023-10-16T14:10:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:22.764+0000 7f824a380640 1 -- 172.21.15.60:0/691900373 wait complete. 2023-10-16T14:10:22.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:10:22.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:10:23.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.004+0000 7f4cac215640 1 Processor -- start 2023-10-16T14:10:23.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.004+0000 7f4cac215640 1 -- start start 2023-10-16T14:10:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4cac215640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4ca40caac0 0x7f4ca40ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4cac215640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca40cb030 con 0x7f4ca406a4b0 2023-10-16T14:10:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4cac215640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca40c91c0 con 0x7f4ca415ad70 2023-10-16T14:10:23.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4cac215640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca40c94a0 con 0x7f4ca40caac0 2023-10-16T14:10:23.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4ca97b0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ca406a4b0 0x7f4ca40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52830/0 (socket says 172.21.15.60:52830) 2023-10-16T14:10:23.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4ca97b0640 1 -- 172.21.15.60:0/2792379327 learned_addr learned my addr 172.21.15.60:0/2792379327 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.006+0000 7f4caa7b2640 1 --2- 172.21.15.60:0/2792379327 >> v2:172.21.15.144:3300/0 conn(0x7f4ca40caac0 0x7f4ca40ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398992589 0 0) 0x7f4ca40cb030 con 0x7f4ca406a4b0 2023-10-16T14:10:23.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c003520 con 0x7f4ca406a4b0 2023-10-16T14:10:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3891342732 0 0) 0x7f4c8c003520 con 0x7f4ca406a4b0 2023-10-16T14:10:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 >> v2:172.21.15.144:3300/0 conn(0x7f4ca40caac0 msgr2=0x7f4ca40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 --2- 172.21.15.60:0/2792379327 >> v2:172.21.15.144:3300/0 conn(0x7f4ca40caac0 0x7f4ca40ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 >> v1:172.21.15.144:6789/0 conn(0x7f4ca415ad70 legacy=0x7f4ca4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca40c9780 con 0x7f4ca406a4b0 2023-10-16T14:10:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4c98001670 con 0x7f4ca406a4b0 2023-10-16T14:10:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 >> v1:172.21.15.60:6789/0 conn(0x7f4ca406a4b0 legacy=0x7f4ca40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c8c0039c0 0x7f4c8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0047c0 0x7f4c8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c005180 0x7f4c8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40c94a0 con 0x7f4c8c0039c0 2023-10-16T14:10:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c00c080 con 0x7f4c8c005180 2023-10-16T14:10:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.007+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c00c240 con 0x7f4c8c0047c0 2023-10-16T14:10:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c8c0039c0 0x7f4c8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4caa7b2640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c005180 0x7f4c8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca97b0640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0047c0 0x7f4c8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca9fb1640 1 -- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0047c0 msgr2=0x7f4c8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4c8c0047c0 0x7f4c8c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca9fb1640 1 -- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c005180 msgr2=0x7f4c8c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c005180 0x7f4c8c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4ca9fb1640 1 -- 172.21.15.60:0/2792379327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c003520 con 0x7f4c8c0039c0 2023-10-16T14:10:23.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.008+0000 7f4caa7b2640 1 -- 172.21.15.60:0/2792379327 reap_dead start 2023-10-16T14:10:23.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.009+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c8c0039c0 0x7f4c8c003de0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f4c94001e50 tx=0x7f4c8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff78cffaab5d67be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94003b70 con 0x7f4c8c0039c0 2023-10-16T14:10:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c94003cd0 con 0x7f4c8c0039c0 2023-10-16T14:10:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4ca8faf640 1 -- 172.21.15.60:0/2792379327 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94005310 con 0x7f4c8c0039c0 2023-10-16T14:10:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 -- 172.21.15.60:0/2792379327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c8c0039c0 msgr2=0x7f4c8c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c8c0039c0 0x7f4c8c003de0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f4c94001e50 tx=0x7f4c8c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 -- 172.21.15.60:0/2792379327 shutdown_connections 2023-10-16T14:10:23.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 --2- 172.21.15.60:0/2792379327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c8c0039c0 0x7f4c8c003de0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 -- 172.21.15.60:0/2792379327 >> 172.21.15.60:0/2792379327 conn(0x7f4ca40ce9b0 msgr2=0x7f4ca40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 -- 172.21.15.60:0/2792379327 shutdown_connections 2023-10-16T14:10:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.010+0000 7f4cac215640 1 -- 172.21.15.60:0/2792379327 wait complete. 2023-10-16T14:10:23.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.012+0000 7f4cac215640 1 Processor -- start 2023-10-16T14:10:23.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.012+0000 7f4cac215640 1 -- start start 2023-10-16T14:10:23.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.013+0000 7f4cac215640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0039c0 0x7f4ca414e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.013+0000 7f4cac215640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 0x7f4ca414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.013+0000 7f4cac215640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ca41487a0 0x7f4ca4148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.013+0000 7f4cac215640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca414b180 con 0x7f4ca414a7c0 2023-10-16T14:10:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.013+0000 7f4cac215640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca414a5b0 con 0x7f4c8c0039c0 2023-10-16T14:10:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.013+0000 7f4cac215640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca414ec90 con 0x7f4ca41487a0 2023-10-16T14:10:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 0x7f4ca414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 0x7f4ca414ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35468/0 (socket says 172.21.15.60:35468) 2023-10-16T14:10:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 -- 172.21.15.60:0/3344436062 learned_addr learned my addr 172.21.15.60:0/3344436062 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0039c0 0x7f4ca414e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 -- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ca41487a0 msgr2=0x7f4ca4148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ca41487a0 0x7f4ca4148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 -- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0039c0 msgr2=0x7f4ca414e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0039c0 0x7f4ca414e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 -- 172.21.15.60:0/3344436062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4149510 con 0x7f4ca414a7c0 2023-10-16T14:10:23.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.014+0000 7f4ca97b0640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 0x7f4ca414ac40 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800d7d0 tx=0x7f4c9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.015+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c980072d0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.015+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c98007430 con 0x7f4ca414a7c0 2023-10-16T14:10:23.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.015+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca41497f0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.015+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c980047e0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.015+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca4149db0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.016+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca414cda0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.017+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4c98004940 con 0x7f4ca414a7c0 2023-10-16T14:10:23.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.019+0000 7f4ca27fc640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4c9006af90 0x7f4c9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.019+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4c98088be0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.019+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4c9006af90 0x7f4c9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.021+0000 7f4ca9fb1640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4c9006af90 0x7f4c9006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c94034000 tx=0x7f4c94009030 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.023+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c98053dd0 con 0x7f4ca414a7c0 2023-10-16T14:10:23.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.217+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ca404e160 con 0x7f4ca414a7c0 2023-10-16T14:10:23.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.217+0000 7f4ca27fc640 1 -- 172.21.15.60:0/3344436062 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v227) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c98019020 con 0x7f4ca414a7c0 2023-10-16T14:10:23.221 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:10:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4c9006af90 msgr2=0x7f4c9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4c9006af90 0x7f4c9006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c94034000 tx=0x7f4c94009030 comp rx=0 tx=0).stop 2023-10-16T14:10:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 msgr2=0x7f4ca414ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 0x7f4ca414ac40 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800d7d0 tx=0x7f4c9800dc90 comp rx=0 tx=0).stop 2023-10-16T14:10:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 shutdown_connections 2023-10-16T14:10:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f4c9006af90 0x7f4c9006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ca41487a0 0x7f4ca4148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ca414a7c0 0x7f4ca414ac40 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 --2- 172.21.15.60:0/3344436062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c8c0039c0 0x7f4ca414e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 >> 172.21.15.60:0/3344436062 conn(0x7f4ca40ce9b0 msgr2=0x7f4ca4159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 shutdown_connections 2023-10-16T14:10:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.220+0000 7f4cac215640 1 -- 172.21.15.60:0/3344436062 wait complete. 2023-10-16T14:10:23.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:10:23.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.472+0000 7f9809c2e640 1 Processor -- start 2023-10-16T14:10:23.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.472+0000 7f9809c2e640 1 -- start start 2023-10-16T14:10:23.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9809c2e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f980406c040 0x7f980406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9809c2e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980415f4a0 con 0x7f98040ad970 2023-10-16T14:10:23.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9809c2e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980406c960 con 0x7f98040cda80 2023-10-16T14:10:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9809c2e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980415b9d0 con 0x7f980406c040 2023-10-16T14:10:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9803fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f98040ad970 0x7f980415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52836/0 (socket says 172.21.15.60:52836) 2023-10-16T14:10:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9803fff640 1 -- 172.21.15.60:0/833361586 learned_addr learned my addr 172.21.15.60:0/833361586 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.474+0000 7f9808c2c640 1 --2- 172.21.15.60:0/833361586 >> v2:172.21.15.60:3300/0 conn(0x7f980406c040 0x7f980406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.475+0000 7f98037fe640 1 -- 172.21.15.60:0/833361586 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719281847 0 0) 0x7f980415f4a0 con 0x7f98040ad970 2023-10-16T14:10:23.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.475+0000 7f98037fe640 1 -- 172.21.15.60:0/833361586 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f4003520 con 0x7f98040ad970 2023-10-16T14:10:23.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.475+0000 7f9808c2c640 1 -- 172.21.15.60:0/833361586 >> v1:172.21.15.144:6789/0 conn(0x7f98040cda80 legacy=0x7f980415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.475+0000 7f9808c2c640 1 -- 172.21.15.60:0/833361586 >> v1:172.21.15.60:6789/0 conn(0x7f98040ad970 legacy=0x7f980415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.475+0000 7f9808c2c640 1 -- 172.21.15.60:0/833361586 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980415bcb0 con 0x7f980406c040 2023-10-16T14:10:23.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.477+0000 7f9808c2c640 1 --2- 172.21.15.60:0/833361586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980406c040 0x7f980406c420 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f97f80097a0 tx=0x7f97f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7653db371da0d68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.477+0000 7f98037fe640 1 -- 172.21.15.60:0/833361586 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8030610 con 0x7f980406c040 2023-10-16T14:10:23.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.477+0000 7f98037fe640 1 -- 172.21.15.60:0/833361586 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f8030770 con 0x7f980406c040 2023-10-16T14:10:23.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.477+0000 7f98037fe640 1 -- 172.21.15.60:0/833361586 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8030b00 con 0x7f980406c040 2023-10-16T14:10:23.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 -- 172.21.15.60:0/833361586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980406c040 msgr2=0x7f980406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 --2- 172.21.15.60:0/833361586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980406c040 0x7f980406c420 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f97f80097a0 tx=0x7f97f802b960 comp rx=0 tx=0).stop 2023-10-16T14:10:23.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 -- 172.21.15.60:0/833361586 shutdown_connections 2023-10-16T14:10:23.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 --2- 172.21.15.60:0/833361586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980406c040 0x7f980406c420 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 -- 172.21.15.60:0/833361586 >> 172.21.15.60:0/833361586 conn(0x7f98040ce6a0 msgr2=0x7f98040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:23.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 -- 172.21.15.60:0/833361586 shutdown_connections 2023-10-16T14:10:23.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.478+0000 7f9809c2e640 1 -- 172.21.15.60:0/833361586 wait complete. 2023-10-16T14:10:23.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.480+0000 7f9809c2e640 1 Processor -- start 2023-10-16T14:10:23.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.480+0000 7f9809c2e640 1 -- start start 2023-10-16T14:10:23.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9809c2e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980406c040 0x7f98040c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9809c2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 0x7f980414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9809c2e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98040cda80 0x7f98040c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9809c2e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9804155f90 con 0x7f98040ad970 2023-10-16T14:10:23.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9809c2e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040b0c20 con 0x7f980406c040 2023-10-16T14:10:23.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9809c2e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98040b0940 con 0x7f98040cda80 2023-10-16T14:10:23.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9803fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 0x7f980414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9803fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 0x7f980414da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35494/0 (socket says 172.21.15.60:35494) 2023-10-16T14:10:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.481+0000 7f9803fff640 1 -- 172.21.15.60:0/3017765652 learned_addr learned my addr 172.21.15.60:0/3017765652 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9808c2c640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980406c040 0x7f98040c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9803fff640 1 -- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98040cda80 msgr2=0x7f98040c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9803fff640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98040cda80 0x7f98040c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9803fff640 1 -- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980406c040 msgr2=0x7f98040c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9803fff640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980406c040 0x7f98040c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9803fff640 1 -- 172.21.15.60:0/3017765652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8009400 con 0x7f98040ad970 2023-10-16T14:10:23.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9808c2c640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980406c040 0x7f98040c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:23.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f9803fff640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 0x7f980414da00 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c4d0 tx=0x7f97f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.482+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0005ce0 con 0x7f98040ad970 2023-10-16T14:10:23.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.483+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f0005e40 con 0x7f98040ad970 2023-10-16T14:10:23.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.483+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0007270 con 0x7f98040ad970 2023-10-16T14:10:23.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.483+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040b3390 con 0x7f98040ad970 2023-10-16T14:10:23.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.483+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98040b3950 con 0x7f98040ad970 2023-10-16T14:10:23.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.484+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980406b330 con 0x7f98040ad970 2023-10-16T14:10:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.485+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97f0007490 con 0x7f98040ad970 2023-10-16T14:10:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.486+0000 7f98017fa640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f97e806af90 0x7f97e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.486+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f97f0088870 con 0x7f98040ad970 2023-10-16T14:10:23.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.487+0000 7f9808c2c640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f97e806af90 0x7f97e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.488+0000 7f9808c2c640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f97e806af90 0x7f97e806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97f8036000 tx=0x7f97f803b040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.490+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97f0053c10 con 0x7f98040ad970 2023-10-16T14:10:23.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.688+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f98040b3b70 con 0x7f98040ad970 2023-10-16T14:10:23.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.689+0000 7f98017fa640 1 -- 172.21.15.60:0/3017765652 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v227) v1 ==== 84+0+65 (secure 0 0 0) 0x7f97f0017020 con 0x7f98040ad970 2023-10-16T14:10:23.693 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:23.694 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:10:23.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:23.694 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:10:23.694 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:10:23.695 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:23.695 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:10:23.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.692+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f97e806af90 msgr2=0x7f97e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.692+0000 7f9809c2e640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f97e806af90 0x7f97e806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97f8036000 tx=0x7f97f803b040 comp rx=0 tx=0).stop 2023-10-16T14:10:23.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.692+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 msgr2=0x7f980414da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.692+0000 7f9809c2e640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 0x7f980414da00 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c4d0 tx=0x7f97f000c990 comp rx=0 tx=0).stop 2023-10-16T14:10:23.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 shutdown_connections 2023-10-16T14:10:23.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f97e806af90 0x7f97e806d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98040cda80 0x7f98040c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98040ad970 0x7f980414da00 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 --2- 172.21.15.60:0/3017765652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980406c040 0x7f98040c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 >> 172.21.15.60:0/3017765652 conn(0x7f98040ce6a0 msgr2=0x7f98040ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:23.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 shutdown_connections 2023-10-16T14:10:23.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.693+0000 7f9809c2e640 1 -- 172.21.15.60:0/3017765652 wait complete. 2023-10-16T14:10:23.717 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:10:23.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:10:23.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.943+0000 7fa2dd10f640 1 Processor -- start 2023-10-16T14:10:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.943+0000 7fa2dd10f640 1 -- start start 2023-10-16T14:10:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2dd10f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2d80cefa0 0x7fa2d80b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2dd10f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80b3880 0x7fa2d80ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2dd10f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d80b19a0 con 0x7fa2d815aad0 2023-10-16T14:10:23.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2dd10f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d80b3d10 con 0x7fa2d80cefa0 2023-10-16T14:10:23.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2dd10f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80b1c40 con 0x7fa2d80b3880 2023-10-16T14:10:23.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2d659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2d80cefa0 0x7fa2d80b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2d759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80b3880 0x7fa2d80ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2d659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2d80cefa0 0x7fa2d80b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55716/0 (socket says 172.21.15.60:55716) 2023-10-16T14:10:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.945+0000 7fa2d659c640 1 -- 172.21.15.60:0/1093966852 learned_addr learned my addr 172.21.15.60:0/1093966852 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:23.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.946+0000 7fa2d659c640 1 -- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80b3880 msgr2=0x7fa2d80ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.946+0000 7fa2d659c640 1 --2- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80b3880 0x7fa2d80ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.946+0000 7fa2d659c640 1 -- 172.21.15.60:0/1093966852 >> v1:172.21.15.144:6789/0 conn(0x7fa2d815aad0 legacy=0x7fa2d80adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.946+0000 7fa2d659c640 1 -- 172.21.15.60:0/1093966852 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d80b1f20 con 0x7fa2d80cefa0 2023-10-16T14:10:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.948+0000 7fa2d659c640 1 --2- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d80cefa0 0x7fa2d80b3340 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009870 tx=0x7fa2cc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4024e1d2ea3946bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.948+0000 7fa2d5d9b640 1 -- 172.21.15.60:0/1093966852 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc031640 con 0x7fa2d80cefa0 2023-10-16T14:10:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.948+0000 7fa2d5d9b640 1 -- 172.21.15.60:0/1093966852 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc0317a0 con 0x7fa2d80cefa0 2023-10-16T14:10:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.948+0000 7fa2d5d9b640 1 -- 172.21.15.60:0/1093966852 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc031b30 con 0x7fa2d80cefa0 2023-10-16T14:10:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 -- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d80cefa0 msgr2=0x7fa2d80b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d80cefa0 0x7fa2d80b3340 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009870 tx=0x7fa2cc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 -- 172.21.15.60:0/1093966852 shutdown_connections 2023-10-16T14:10:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80b3880 0x7fa2d80ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/1093966852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d80cefa0 0x7fa2d80b3340 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 -- 172.21.15.60:0/1093966852 >> 172.21.15.60:0/1093966852 conn(0x7fa2d8069da0 msgr2=0x7fa2d806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 -- 172.21.15.60:0/1093966852 shutdown_connections 2023-10-16T14:10:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.949+0000 7fa2dd10f640 1 -- 172.21.15.60:0/1093966852 wait complete. 2023-10-16T14:10:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.951+0000 7fa2dd10f640 1 Processor -- start 2023-10-16T14:10:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.951+0000 7fa2dd10f640 1 -- start start 2023-10-16T14:10:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2dd10f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 0x7fa2d81464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2dd10f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80cefa0 0x7fa2d815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2dd10f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d815aad0 0x7fa2d815db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2dd10f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d815bb60 con 0x7fa2d80b3880 2023-10-16T14:10:23.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2dd10f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d815c9c0 con 0x7fa2d815aad0 2023-10-16T14:10:23.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2dd10f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d815bdb0 con 0x7fa2d80cefa0 2023-10-16T14:10:23.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.952+0000 7fa2d6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 0x7fa2d81464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 0x7fa2d81464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35510/0 (socket says 172.21.15.60:35510) 2023-10-16T14:10:23.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 -- 172.21.15.60:0/354196485 learned_addr learned my addr 172.21.15.60:0/354196485 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:23.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d759e640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d815aad0 0x7fa2d815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 -- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80cefa0 msgr2=0x7fa2d815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80cefa0 0x7fa2d815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 -- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d815aad0 msgr2=0x7fa2d815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:23.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d815aad0 0x7fa2d815db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:23.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 -- 172.21.15.60:0/354196485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc0094d0 con 0x7fa2d80b3880 2023-10-16T14:10:23.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d759e640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d815aad0 0x7fa2d815db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.953+0000 7fa2d6d9d640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 0x7fa2d81464f0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c430 tx=0x7fa2c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.954+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c80052d0 con 0x7fa2d80b3880 2023-10-16T14:10:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.954+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c8005430 con 0x7fa2d80b3880 2023-10-16T14:10:23.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.954+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8003830 con 0x7fa2d80b3880 2023-10-16T14:10:23.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.954+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d815c090 con 0x7fa2d80b3880 2023-10-16T14:10:23.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.954+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d815c650 con 0x7fa2d80b3880 2023-10-16T14:10:23.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.955+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d804e160 con 0x7fa2d80b3880 2023-10-16T14:10:23.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.956+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa2c8003990 con 0x7fa2d80b3880 2023-10-16T14:10:23.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.957+0000 7fa2bf7fe640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:23.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.957+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2c8088a20 con 0x7fa2d80b3880 2023-10-16T14:10:23.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.958+0000 7fa2d659c640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:23.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.959+0000 7fa2d659c640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa2c406af90 0x7fa2c406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc036000 tx=0x7fa2cc030000 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:23.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:23.961+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa2c8053d10 con 0x7fa2d80b3880 2023-10-16T14:10:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.159+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa2d8158300 con 0x7fa2d80b3880 2023-10-16T14:10:24.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.772+0000 7fa2bf7fe640 1 -- 172.21.15.60:0/354196485 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v228) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa2c80170b0 con 0x7fa2d80b3880 2023-10-16T14:10:24.777 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T14:10:24.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa2c406af90 msgr2=0x7fa2c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:24.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa2c406af90 0x7fa2c406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc036000 tx=0x7fa2cc030000 comp rx=0 tx=0).stop 2023-10-16T14:10:24.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 msgr2=0x7fa2d81464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:24.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 0x7fa2d81464f0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c430 tx=0x7fa2c800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:24.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 shutdown_connections 2023-10-16T14:10:24.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa2c406af90 0x7fa2c406d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:24.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2d80cefa0 0x7fa2d815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:24.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2d815aad0 0x7fa2d815db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:24.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 --2- 172.21.15.60:0/354196485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2d80b3880 0x7fa2d81464f0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:24.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 >> 172.21.15.60:0/354196485 conn(0x7fa2d8069da0 msgr2=0x7fa2d80ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:24.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 shutdown_connections 2023-10-16T14:10:24.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:24.777+0000 7fa2dd10f640 1 -- 172.21.15.60:0/354196485 wait complete. 2023-10-16T14:10:24.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:10:25.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.040+0000 7f010f7a5640 1 Processor -- start 2023-10-16T14:10:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.041+0000 7f010f7a5640 1 -- start start 2023-10-16T14:10:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.042+0000 7f010f7a5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0108160420 0x7f01080ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.042+0000 7f010f7a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010806b4c0 0x7f01080cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.042+0000 7f010f7a5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010806e170 con 0x7f0108160800 2023-10-16T14:10:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.042+0000 7f010f7a5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01080cde30 con 0x7f0108160420 2023-10-16T14:10:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.042+0000 7f010f7a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01080cd460 con 0x7f010806b4c0 2023-10-16T14:10:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0108160420 0x7f01080ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0108160420 0x7f01080ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35518/0 (socket says 172.21.15.60:35518) 2023-10-16T14:10:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 -- 172.21.15.60:0/882383399 learned_addr learned my addr 172.21.15.60:0/882383399 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010dd42640 1 --2- 172.21.15.60:0/882383399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010806b4c0 0x7f01080cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 -- 172.21.15.60:0/882383399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010806b4c0 msgr2=0x7f01080cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 --2- 172.21.15.60:0/882383399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010806b4c0 0x7f01080cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 -- 172.21.15.60:0/882383399 >> v1:172.21.15.144:6789/0 conn(0x7f0108160800 legacy=0x7f010806adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.043+0000 7f010d541640 1 -- 172.21.15.60:0/882383399 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010806e3c0 con 0x7f0108160420 2023-10-16T14:10:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.045+0000 7f010d541640 1 --2- 172.21.15.60:0/882383399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080ae170 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f0104009870 tx=0x7f010402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470c92fcf8db439c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.045+0000 7f00fffff640 1 -- 172.21.15.60:0/882383399 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104030840 con 0x7f0108160420 2023-10-16T14:10:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.045+0000 7f00fffff640 1 -- 172.21.15.60:0/882383399 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01040309a0 con 0x7f0108160420 2023-10-16T14:10:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.045+0000 7f00fffff640 1 -- 172.21.15.60:0/882383399 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104030d30 con 0x7f0108160420 2023-10-16T14:10:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.045+0000 7f010f7a5640 1 -- 172.21.15.60:0/882383399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 msgr2=0x7f01080ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.045+0000 7f010f7a5640 1 --2- 172.21.15.60:0/882383399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080ae170 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f0104009870 tx=0x7f010402b980 comp rx=0 tx=0).stop 2023-10-16T14:10:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.046+0000 7f010f7a5640 1 -- 172.21.15.60:0/882383399 shutdown_connections 2023-10-16T14:10:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.046+0000 7f010f7a5640 1 --2- 172.21.15.60:0/882383399 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f010806b4c0 0x7f01080cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.046+0000 7f010f7a5640 1 --2- 172.21.15.60:0/882383399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080ae170 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.046+0000 7f010f7a5640 1 -- 172.21.15.60:0/882383399 >> 172.21.15.60:0/882383399 conn(0x7f01080b0d30 msgr2=0x7f01080b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.046+0000 7f010f7a5640 1 -- 172.21.15.60:0/882383399 shutdown_connections 2023-10-16T14:10:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.046+0000 7f010f7a5640 1 -- 172.21.15.60:0/882383399 wait complete. 2023-10-16T14:10:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.047+0000 7f010f7a5640 1 Processor -- start 2023-10-16T14:10:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.048+0000 7f010f7a5640 1 -- start start 2023-10-16T14:10:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010f7a5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010806b4c0 0x7f01080c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010f7a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010f7a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0108160800 0x7f01080c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010f7a5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010809e2a0 con 0x7f0108160420 2023-10-16T14:10:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010f7a5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010809d3b0 con 0x7f010806b4c0 2023-10-16T14:10:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010f7a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010809d690 con 0x7f0108160800 2023-10-16T14:10:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35520/0 (socket says 172.21.15.60:35520) 2023-10-16T14:10:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 -- 172.21.15.60:0/4176620200 learned_addr learned my addr 172.21.15.60:0/4176620200 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010d541640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010806b4c0 0x7f01080c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010dd42640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0108160800 0x7f01080c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 -- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0108160800 msgr2=0x7f01080c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0108160800 0x7f01080c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 -- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010806b4c0 msgr2=0x7f01080c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010806b4c0 0x7f01080c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010cd40640 1 -- 172.21.15.60:0/4176620200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01040094d0 con 0x7f0108160420 2023-10-16T14:10:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010d541640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010806b4c0 0x7f01080c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.049+0000 7f010dd42640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0108160800 0x7f01080c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.050+0000 7f010cd40640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080b4290 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f00f800c400 tx=0x7f00f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.050+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f80052d0 con 0x7f0108160420 2023-10-16T14:10:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.050+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f8005430 con 0x7f0108160420 2023-10-16T14:10:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.050+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8003830 con 0x7f0108160420 2023-10-16T14:10:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.050+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010809d970 con 0x7f0108160420 2023-10-16T14:10:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.050+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010809df30 con 0x7f0108160420 2023-10-16T14:10:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.051+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010806dfe0 con 0x7f0108160420 2023-10-16T14:10:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.054+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f00f8003990 con 0x7f0108160420 2023-10-16T14:10:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.055+0000 7f00fdffb640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f00ec06afc0 0x7f00ec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.055+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f00f8088f20 con 0x7f0108160420 2023-10-16T14:10:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.055+0000 7f010d541640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f00ec06afc0 0x7f00ec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.056+0000 7f010d541640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f00ec06afc0 0x7f00ec06d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0104036000 tx=0x7f010403b040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.058+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00f8054110 con 0x7f0108160420 2023-10-16T14:10:25.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.252+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f010804e160 con 0x7f0108160420 2023-10-16T14:10:25.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.781+0000 7f00fdffb640 1 -- 172.21.15.60:0/4176620200 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v229) v1 ==== 170+0+0 (secure 0 0 0) 0x7f00f8057730 con 0x7f0108160420 2023-10-16T14:10:25.785 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T14:10:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f00ec06afc0 msgr2=0x7f00ec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f00ec06afc0 0x7f00ec06d480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0104036000 tx=0x7f010403b040 comp rx=0 tx=0).stop 2023-10-16T14:10:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 msgr2=0x7f01080b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080b4290 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f00f800c400 tx=0x7f00f800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 shutdown_connections 2023-10-16T14:10:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f00ec06afc0 0x7f00ec06d480 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0108160800 0x7f01080c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0108160420 0x7f01080b4290 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 --2- 172.21.15.60:0/4176620200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f010806b4c0 0x7f01080c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 >> 172.21.15.60:0/4176620200 conn(0x7f01080b0d30 msgr2=0x7f0108070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 shutdown_connections 2023-10-16T14:10:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:25.785+0000 7f010f7a5640 1 -- 172.21.15.60:0/4176620200 wait complete. 2023-10-16T14:10:25.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:10:26.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.014+0000 7f2d603fb640 1 Processor -- start 2023-10-16T14:10:26.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.014+0000 7f2d603fb640 1 -- start start 2023-10-16T14:10:26.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.015+0000 7f2d603fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d580ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.015+0000 7f2d603fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d58070360 con 0x7f2d58159ab0 2023-10-16T14:10:26.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.015+0000 7f2d603fb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580cd1f0 con 0x7f2d5806c140 2023-10-16T14:10:26.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.015+0000 7f2d603fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5815d890 con 0x7f2d58159e90 2023-10-16T14:10:26.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.016+0000 7f2d5e197640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2d58159ab0 0x7f2d5806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52844/0 (socket says 172.21.15.60:52844) 2023-10-16T14:10:26.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.016+0000 7f2d5e197640 1 -- 172.21.15.60:0/2021765846 learned_addr learned my addr 172.21.15.60:0/2021765846 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:26.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.016+0000 7f2d5d996640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d580ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.016+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544712986 0 0) 0x7f2d58070360 con 0x7f2d58159ab0 2023-10-16T14:10:26.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.016+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d44003520 con 0x7f2d58159ab0 2023-10-16T14:10:26.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721893069 0 0) 0x7f2d44003520 con 0x7f2d58159ab0 2023-10-16T14:10:26.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 msgr2=0x7f2d580ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:26.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d580ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 >> v1:172.21.15.144:6789/0 conn(0x7f2d5806c140 legacy=0x7f2d5815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580cd4d0 con 0x7f2d58159ab0 2023-10-16T14:10:26.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2d54001670 con 0x7f2d58159ab0 2023-10-16T14:10:26.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 >> v1:172.21.15.60:6789/0 conn(0x7f2d58159ab0 legacy=0x7f2d5806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d440039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d44003f20 0x7f2d440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44004d00 0x7f2d44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5815d890 con 0x7f2d44003f20 2023-10-16T14:10:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4400bbc0 con 0x7f2d44004d00 2023-10-16T14:10:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4400df90 con 0x7f2d58159e90 2023-10-16T14:10:26.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d996640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d44003f20 0x7f2d440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5e197640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d440039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5e998640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44004d00 0x7f2d44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.017+0000 7f2d5d996640 1 -- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 msgr2=0x7f2d440039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:26.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.018+0000 7f2d5d996640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d440039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.018+0000 7f2d5d996640 1 -- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44004d00 msgr2=0x7f2d44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:26.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.018+0000 7f2d5d996640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44004d00 0x7f2d44005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.018+0000 7f2d5d996640 1 -- 172.21.15.60:0/2021765846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44003520 con 0x7f2d44003f20 2023-10-16T14:10:26.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.018+0000 7f2d5e197640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d58159e90 0x7f2d440039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:26.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.018+0000 7f2d5e998640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44004d00 0x7f2d44005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:26.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.019+0000 7f2d5d996640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d44003f20 0x7f2d440047c0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009ac0 tx=0x7f2d440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c909ea74afdac1e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:26.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.019+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4802c6b0 con 0x7f2d44003f20 2023-10-16T14:10:26.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.019+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4802c810 con 0x7f2d44003f20 2023-10-16T14:10:26.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.019+0000 7f2d5d195640 1 -- 172.21.15.60:0/2021765846 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4802cba0 con 0x7f2d44003f20 2023-10-16T14:10:26.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d603fb640 1 -- 172.21.15.60:0/2021765846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d44003f20 msgr2=0x7f2d440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d603fb640 1 --2- 172.21.15.60:0/2021765846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d44003f20 0x7f2d440047c0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009ac0 tx=0x7f2d440057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d5e998640 1 -- 172.21.15.60:0/2021765846 reap_dead start 2023-10-16T14:10:26.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d603fb640 1 -- 172.21.15.60:0/2021765846 shutdown_connections 2023-10-16T14:10:26.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d603fb640 1 -- 172.21.15.60:0/2021765846 >> 172.21.15.60:0/2021765846 conn(0x7f2d580ce9b0 msgr2=0x7f2d580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:26.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d603fb640 1 -- 172.21.15.60:0/2021765846 shutdown_connections 2023-10-16T14:10:26.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.020+0000 7f2d603fb640 1 -- 172.21.15.60:0/2021765846 wait complete. 2023-10-16T14:10:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.022+0000 7f2d603fb640 1 Processor -- start 2023-10-16T14:10:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.022+0000 7f2d603fb640 1 -- start start 2023-10-16T14:10:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d603fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44003f20 0x7f2d580c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d603fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 0x7f2d580c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d603fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d580b0680 0x7f2d580b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d603fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580c7330 con 0x7f2d580c86f0 2023-10-16T14:10:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d603fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580b1040 con 0x7f2d44003f20 2023-10-16T14:10:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d603fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5814b4d0 con 0x7f2d580b0680 2023-10-16T14:10:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d5d996640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 0x7f2d580c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d5d996640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 0x7f2d580c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35544/0 (socket says 172.21.15.60:35544) 2023-10-16T14:10:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d5d996640 1 -- 172.21.15.60:0/4157768855 learned_addr learned my addr 172.21.15.60:0/4157768855 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d5e197640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44003f20 0x7f2d580c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.023+0000 7f2d5e998640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d580b0680 0x7f2d580b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d5d996640 1 -- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d580b0680 msgr2=0x7f2d580b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d5d996640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d580b0680 0x7f2d580b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d5d996640 1 -- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44003f20 msgr2=0x7f2d580c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d5d996640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44003f20 0x7f2d580c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d5d996640 1 -- 172.21.15.60:0/4157768855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580c5fc0 con 0x7f2d580c86f0 2023-10-16T14:10:26.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d5d996640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 0x7f2d580c5a80 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f2d4802c250 tx=0x7f2d48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:26.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003810 con 0x7f2d580c86f0 2023-10-16T14:10:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d48003970 con 0x7f2d580c86f0 2023-10-16T14:10:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003ce0 con 0x7f2d580c86f0 2023-10-16T14:10:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.024+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d580c8c80 con 0x7f2d580c86f0 2023-10-16T14:10:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.025+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d580c9240 con 0x7f2d580c86f0 2023-10-16T14:10:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.026+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5804e160 con 0x7f2d580c86f0 2023-10-16T14:10:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.027+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d48034440 con 0x7f2d580c86f0 2023-10-16T14:10:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.028+0000 7f2d4effd640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2d3c06af90 0x7f2d3c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.028+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2d480abbb0 con 0x7f2d580c86f0 2023-10-16T14:10:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.029+0000 7f2d5e197640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2d3c06af90 0x7f2d3c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.030+0000 7f2d5e197640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2d3c06af90 0x7f2d3c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d54000f60 tx=0x7f2d54009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.032+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d48076dd0 con 0x7f2d580c86f0 2023-10-16T14:10:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.267+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2d580cda60 con 0x7f2d580c86f0 2023-10-16T14:10:26.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.831+0000 7f2d4effd640 1 -- 172.21.15.60:0/4157768855 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v109) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2d480390c0 con 0x7f2d580c86f0 2023-10-16T14:10:26.841 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 26 and data pool 27 2023-10-16T14:10:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.842+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2d3c06af90 msgr2=0x7f2d3c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.842+0000 7f2d603fb640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2d3c06af90 0x7f2d3c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d54000f60 tx=0x7f2d54009040 comp rx=0 tx=0).stop 2023-10-16T14:10:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.842+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 msgr2=0x7f2d580c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.842+0000 7f2d603fb640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 0x7f2d580c5a80 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f2d4802c250 tx=0x7f2d48004730 comp rx=0 tx=0).stop 2023-10-16T14:10:26.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.843+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 shutdown_connections 2023-10-16T14:10:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.843+0000 7f2d603fb640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2d3c06af90 0x7f2d3c06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.843+0000 7f2d603fb640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d580b0680 0x7f2d580b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.843+0000 7f2d603fb640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d580c86f0 0x7f2d580c5a80 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.843+0000 7f2d603fb640 1 --2- 172.21.15.60:0/4157768855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d44003f20 0x7f2d580c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.843+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 >> 172.21.15.60:0/4157768855 conn(0x7f2d580ce9b0 msgr2=0x7f2d58156c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:26.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.846+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 shutdown_connections 2023-10-16T14:10:26.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:26.846+0000 7f2d603fb640 1 -- 172.21.15.60:0/4157768855 wait complete. 2023-10-16T14:10:26.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:27.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.055+0000 7f3c8e84b640 1 Processor -- start 2023-10-16T14:10:27.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.056+0000 7f3c8e84b640 1 -- start start 2023-10-16T14:10:27.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.057+0000 7f3c8e84b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3c880cd0c0 0x7f3c881615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.057+0000 7f3c8e84b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3c880b0410 0x7f3c880b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.057+0000 7f3c8e84b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8815d3f0 0x7f3c8815d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.057+0000 7f3c8e84b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c880b0de0 con 0x7f3c880b0410 2023-10-16T14:10:27.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.057+0000 7f3c8e84b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8815e3e0 con 0x7f3c880cd0c0 2023-10-16T14:10:27.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.057+0000 7f3c8e84b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880cd4a0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3c880b0410 0x7f3c880b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3c880b0410 0x7f3c880b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35550/0 (socket says 172.21.15.60:35550) 2023-10-16T14:10:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 -- 172.21.15.60:0/823286786 learned_addr learned my addr 172.21.15.60:0/823286786 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c87fff640 1 --2- 172.21.15.60:0/823286786 >> v2:172.21.15.144:3300/0 conn(0x7f3c880cd0c0 0x7f3c881615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c8cde8640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8815d3f0 0x7f3c8815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 -- 172.21.15.60:0/823286786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8815d3f0 msgr2=0x7f3c8815d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8815d3f0 0x7f3c8815d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 -- 172.21.15.60:0/823286786 >> v2:172.21.15.144:3300/0 conn(0x7f3c880cd0c0 msgr2=0x7f3c881615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 --2- 172.21.15.60:0/823286786 >> v2:172.21.15.144:3300/0 conn(0x7f3c880cd0c0 0x7f3c881615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c7ffff640 1 -- 172.21.15.60:0/823286786 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8815e100 con 0x7f3c880b0410 2023-10-16T14:10:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c87fff640 1 --2- 172.21.15.60:0/823286786 >> v2:172.21.15.144:3300/0 conn(0x7f3c880cd0c0 0x7f3c881615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.058+0000 7f3c8cde8640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8815d3f0 0x7f3c8815d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.059+0000 7f3c7ffff640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c880b0410 0x7f3c880b08a0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f3c880cc4e0 tx=0x7f3c7002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f42554d3afbd4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.059+0000 7f3c877fe640 1 -- 172.21.15.60:0/823286786 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700317d0 con 0x7f3c880b0410 2023-10-16T14:10:27.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c877fe640 1 -- 172.21.15.60:0/823286786 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70031930 con 0x7f3c880b0410 2023-10-16T14:10:27.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c877fe640 1 -- 172.21.15.60:0/823286786 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70031cc0 con 0x7f3c880b0410 2023-10-16T14:10:27.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 -- 172.21.15.60:0/823286786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c880b0410 msgr2=0x7f3c880b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c880b0410 0x7f3c880b08a0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f3c880cc4e0 tx=0x7f3c7002b870 comp rx=0 tx=0).stop 2023-10-16T14:10:27.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 -- 172.21.15.60:0/823286786 shutdown_connections 2023-10-16T14:10:27.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8815d3f0 0x7f3c8815d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/823286786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c880b0410 0x7f3c880b08a0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/823286786 >> v2:172.21.15.144:3300/0 conn(0x7f3c880cd0c0 0x7f3c881615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 -- 172.21.15.60:0/823286786 >> 172.21.15.60:0/823286786 conn(0x7f3c880cb410 msgr2=0x7f3c880cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:27.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 -- 172.21.15.60:0/823286786 shutdown_connections 2023-10-16T14:10:27.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.060+0000 7f3c8e84b640 1 -- 172.21.15.60:0/823286786 wait complete. 2023-10-16T14:10:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.062+0000 7f3c8e84b640 1 Processor -- start 2023-10-16T14:10:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.062+0000 7f3c8e84b640 1 -- start start 2023-10-16T14:10:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8e84b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c880b0410 0x7f3c8806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8e84b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c880cd0c0 0x7f3c880c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8e84b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 0x7f3c88153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8e84b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880b34e0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8e84b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880c5fa0 con 0x7f3c880b0410 2023-10-16T14:10:27.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8e84b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880c88d0 con 0x7f3c880cd0c0 2023-10-16T14:10:27.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.063+0000 7f3c8cde8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 0x7f3c88153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 0x7f3c88153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35564/0 (socket says 172.21.15.60:35564) 2023-10-16T14:10:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 -- 172.21.15.60:0/4245278925 learned_addr learned my addr 172.21.15.60:0/4245278925 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 -- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c880cd0c0 msgr2=0x7f3c880c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c880cd0c0 0x7f3c880c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 -- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c880b0410 msgr2=0x7f3c8806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c880b0410 0x7f3c8806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 -- 172.21.15.60:0/4245278925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700094d0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8cde8640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 0x7f3c88153800 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000cef0 tx=0x7f3c8000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8000d840 con 0x7f3c8815d3f0 2023-10-16T14:10:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8000d9c0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.064+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8806bf90 con 0x7f3c8815d3f0 2023-10-16T14:10:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.065+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8000dd70 con 0x7f3c8815d3f0 2023-10-16T14:10:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.065+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8806c550 con 0x7f3c8815d3f0 2023-10-16T14:10:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.066+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8806c740 con 0x7f3c8815d3f0 2023-10-16T14:10:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.067+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3c8001e3b0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.068+0000 7f3c857fa640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f3c6806af90 0x7f3c6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.068+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c8008a310 con 0x7f3c8815d3f0 2023-10-16T14:10:27.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.068+0000 7f3c87fff640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f3c6806af90 0x7f3c6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.069+0000 7f3c87fff640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f3c6806af90 0x7f3c6806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c78006b10 tx=0x7f3c78008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.072+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c800552d0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.267+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c8804e160 con 0x7f3c8815d3f0 2023-10-16T14:10:27.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.270+0000 7f3c857fa640 1 -- 172.21.15.60:0/4245278925 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f3c800588f0 con 0x7f3c8815d3f0 2023-10-16T14:10:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:27.275 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":230,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:26.269846+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:27.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f3c6806af90 msgr2=0x7f3c6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f3c6806af90 0x7f3c6806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c78006b10 tx=0x7f3c78008040 comp rx=0 tx=0).stop 2023-10-16T14:10:27.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 msgr2=0x7f3c88153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 0x7f3c88153800 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000cef0 tx=0x7f3c8000a710 comp rx=0 tx=0).stop 2023-10-16T14:10:27.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 shutdown_connections 2023-10-16T14:10:27.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f3c6806af90 0x7f3c6806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c880cd0c0 0x7f3c880c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c8815d3f0 0x7f3c88153800 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 --2- 172.21.15.60:0/4245278925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c880b0410 0x7f3c8806ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.272+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 >> 172.21.15.60:0/4245278925 conn(0x7f3c880cb410 msgr2=0x7f3c8815cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:27.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.273+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 shutdown_connections 2023-10-16T14:10:27.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.273+0000 7f3c8e84b640 1 -- 172.21.15.60:0/4245278925 wait complete. 2023-10-16T14:10:27.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:27.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.498+0000 7fa8b930a640 1 Processor -- start 2023-10-16T14:10:27.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.498+0000 7fa8b930a640 1 -- start start 2023-10-16T14:10:27.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.499+0000 7fa8b930a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa8b406c040 0x7fa8b406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b930a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b40cda80 0x7fa8b415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b930a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b406c960 con 0x7fa8b40ad970 2023-10-16T14:10:27.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b930a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b415b9d0 con 0x7fa8b406c040 2023-10-16T14:10:27.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b930a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b415bcb0 con 0x7fa8b40cda80 2023-10-16T14:10:27.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa8b406c040 0x7fa8b406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8abfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa8b40ad970 0x7fa8b415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52854/0 (socket says 172.21.15.60:52854) 2023-10-16T14:10:27.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8abfff640 1 -- 172.21.15.60:0/983090339 learned_addr learned my addr 172.21.15.60:0/983090339 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:27.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b8b09640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b40cda80 0x7fa8b415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b3fff640 1 -- 172.21.15.60:0/983090339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b40cda80 msgr2=0x7fa8b415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:27.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b3fff640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b40cda80 0x7fa8b415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b3fff640 1 -- 172.21.15.60:0/983090339 >> v1:172.21.15.60:6789/0 conn(0x7fa8b40ad970 legacy=0x7fa8b415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b3fff640 1 -- 172.21.15.60:0/983090339 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b415bf90 con 0x7fa8b406c040 2023-10-16T14:10:27.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b8b09640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b40cda80 0x7fa8b415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:27.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.500+0000 7fa8b37fe640 1 -- 172.21.15.60:0/983090339 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631087170 0 0) 0x7fa8b406c960 con 0x7fa8b40ad970 2023-10-16T14:10:27.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.502+0000 7fa8b3fff640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b406c040 0x7fa8b406c420 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40097a0 tx=0x7fa8a402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327957782ed667c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.502+0000 7fa8b37fe640 1 -- 172.21.15.60:0/983090339 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a402d040 con 0x7fa8b406c040 2023-10-16T14:10:27.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.502+0000 7fa8b37fe640 1 -- 172.21.15.60:0/983090339 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a4031810 con 0x7fa8b406c040 2023-10-16T14:10:27.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.502+0000 7fa8b37fe640 1 -- 172.21.15.60:0/983090339 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4031ba0 con 0x7fa8b406c040 2023-10-16T14:10:27.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.502+0000 7fa8b930a640 1 -- 172.21.15.60:0/983090339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b406c040 msgr2=0x7fa8b406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.502+0000 7fa8b930a640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b406c040 0x7fa8b406c420 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40097a0 tx=0x7fa8a402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.503+0000 7fa8b930a640 1 -- 172.21.15.60:0/983090339 shutdown_connections 2023-10-16T14:10:27.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.503+0000 7fa8b930a640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b40cda80 0x7fa8b415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.503+0000 7fa8b930a640 1 --2- 172.21.15.60:0/983090339 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b406c040 0x7fa8b406c420 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.503+0000 7fa8b930a640 1 -- 172.21.15.60:0/983090339 >> 172.21.15.60:0/983090339 conn(0x7fa8b40ce6a0 msgr2=0x7fa8b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:27.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.503+0000 7fa8b930a640 1 -- 172.21.15.60:0/983090339 shutdown_connections 2023-10-16T14:10:27.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.503+0000 7fa8b930a640 1 -- 172.21.15.60:0/983090339 wait complete. 2023-10-16T14:10:27.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.504+0000 7fa8b930a640 1 Processor -- start 2023-10-16T14:10:27.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.504+0000 7fa8b930a640 1 -- start start 2023-10-16T14:10:27.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.505+0000 7fa8b930a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b406c040 0x7fa8b40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.505+0000 7fa8b930a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 0x7fa8b4156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.505+0000 7fa8b930a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b40cda80 0x7fa8b415cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.505+0000 7fa8b930a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40c24f0 con 0x7fa8b40ad970 2023-10-16T14:10:27.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.505+0000 7fa8b930a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40c9cd0 con 0x7fa8b40cda80 2023-10-16T14:10:27.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.505+0000 7fa8b930a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40c27b0 con 0x7fa8b406c040 2023-10-16T14:10:27.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 0x7fa8b4156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 0x7fa8b4156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35574/0 (socket says 172.21.15.60:35574) 2023-10-16T14:10:27.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 -- 172.21.15.60:0/2271355200 learned_addr learned my addr 172.21.15.60:0/2271355200 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:27.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8b3fff640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b406c040 0x7fa8b40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 -- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b406c040 msgr2=0x7fa8b40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8b8b09640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b40cda80 0x7fa8b415cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b406c040 0x7fa8b40c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 -- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b40cda80 msgr2=0x7fa8b415cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b40cda80 0x7fa8b415cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 -- 172.21.15.60:0/2271355200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4009400 con 0x7fa8b40ad970 2023-10-16T14:10:27.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.506+0000 7fa8abfff640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 0x7fa8b4156dc0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fa89c00c4b0 tx=0x7fa89c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.507+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c0074e0 con 0x7fa8b40ad970 2023-10-16T14:10:27.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.507+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c007660 con 0x7fa8b40ad970 2023-10-16T14:10:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.507+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c004a20 con 0x7fa8b40ad970 2023-10-16T14:10:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.507+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b40c9780 con 0x7fa8b40ad970 2023-10-16T14:10:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.507+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b4157300 con 0x7fa8b40ad970 2023-10-16T14:10:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.508+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b406b330 con 0x7fa8b40ad970 2023-10-16T14:10:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.509+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa89c004b80 con 0x7fa8b40ad970 2023-10-16T14:10:27.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.510+0000 7fa8b17fa640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa89406afe0 0x7fa89406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.510+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa89c088e00 con 0x7fa8b40ad970 2023-10-16T14:10:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.511+0000 7fa8b3fff640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa89406afe0 0x7fa89406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.512+0000 7fa8b3fff640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa89406afe0 0x7fa89406d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4036000 tx=0x7fa8a403c040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.515+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa89c053e40 con 0x7fa8b40ad970 2023-10-16T14:10:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.712+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8b40c2920 con 0x7fa8b40ad970 2023-10-16T14:10:27.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.715+0000 7fa8b17fa640 1 -- 172.21.15.60:0/2271355200 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+16584 (secure 0 0 0) 0x7fa89c018070 con 0x7fa8b40ad970 2023-10-16T14:10:27.719 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:27.720 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":230,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:26.269846+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa89406afe0 msgr2=0x7fa89406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa89406afe0 0x7fa89406d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4036000 tx=0x7fa8a403c040 comp rx=0 tx=0).stop 2023-10-16T14:10:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 msgr2=0x7fa8b4156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 0x7fa8b4156dc0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fa89c00c4b0 tx=0x7fa89c00c970 comp rx=0 tx=0).stop 2023-10-16T14:10:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 shutdown_connections 2023-10-16T14:10:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa89406afe0 0x7fa89406d4a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b40cda80 0x7fa8b415cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8b40ad970 0x7fa8b4156dc0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 --2- 172.21.15.60:0/2271355200 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8b406c040 0x7fa8b40c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 >> 172.21.15.60:0/2271355200 conn(0x7fa8b40ce6a0 msgr2=0x7fa8b415e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:27.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 shutdown_connections 2023-10-16T14:10:27.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.718+0000 7fa8b930a640 1 -- 172.21.15.60:0/2271355200 wait complete. 2023-10-16T14:10:27.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:10:27.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:10:27.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:10:27.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.963+0000 7f1996d3c640 1 Processor -- start 2023-10-16T14:10:27.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.963+0000 7f1996d3c640 1 -- start start 2023-10-16T14:10:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1996d3c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1990159950 0x7f199015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1996d3c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1990070450 0x7f19900708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1996d3c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f199006c100 0x7f199006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1996d3c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1990070e20 con 0x7f1990159950 2023-10-16T14:10:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1996d3c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199015e490 con 0x7f1990070450 2023-10-16T14:10:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1996d3c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199015aef0 con 0x7f199006c100 2023-10-16T14:10:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.965+0000 7f1995d3a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1990159950 0x7f199015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1990159950 0x7f199015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35582/0 (socket says 172.21.15.60:35582) 2023-10-16T14:10:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 -- 172.21.15.60:0/4181912284 learned_addr learned my addr 172.21.15.60:0/4181912284 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f199653b640 1 --2- 172.21.15.60:0/4181912284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f199006c100 0x7f199006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995539640 1 --2- 172.21.15.60:0/4181912284 >> v2:172.21.15.144:3300/0 conn(0x7f1990070450 0x7f19900708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 -- 172.21.15.60:0/4181912284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f199006c100 msgr2=0x7f199006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 --2- 172.21.15.60:0/4181912284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f199006c100 0x7f199006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 -- 172.21.15.60:0/4181912284 >> v2:172.21.15.144:3300/0 conn(0x7f1990070450 msgr2=0x7f19900708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 --2- 172.21.15.60:0/4181912284 >> v2:172.21.15.144:3300/0 conn(0x7f1990070450 0x7f19900708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.966+0000 7f1995d3a640 1 -- 172.21.15.60:0/4181912284 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199015d910 con 0x7f1990159950 2023-10-16T14:10:27.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.967+0000 7f1995d3a640 1 --2- 172.21.15.60:0/4181912284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1990159950 0x7f199015a9b0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f19900cab70 tx=0x7f198c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528040707a0d8d3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1994d38640 1 -- 172.21.15.60:0/4181912284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0317c0 con 0x7f1990159950 2023-10-16T14:10:27.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1994d38640 1 -- 172.21.15.60:0/4181912284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c031920 con 0x7f1990159950 2023-10-16T14:10:27.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1994d38640 1 -- 172.21.15.60:0/4181912284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c031cb0 con 0x7f1990159950 2023-10-16T14:10:27.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 -- 172.21.15.60:0/4181912284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1990159950 msgr2=0x7f199015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 --2- 172.21.15.60:0/4181912284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1990159950 0x7f199015a9b0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f19900cab70 tx=0x7f198c02b860 comp rx=0 tx=0).stop 2023-10-16T14:10:27.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 -- 172.21.15.60:0/4181912284 shutdown_connections 2023-10-16T14:10:27.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 --2- 172.21.15.60:0/4181912284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f199006c100 0x7f199006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 --2- 172.21.15.60:0/4181912284 >> v2:172.21.15.144:3300/0 conn(0x7f1990070450 0x7f19900708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 --2- 172.21.15.60:0/4181912284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1990159950 0x7f199015a9b0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.968+0000 7f1996d3c640 1 -- 172.21.15.60:0/4181912284 >> 172.21.15.60:0/4181912284 conn(0x7f19900ce6a0 msgr2=0x7f19900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:27.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.969+0000 7f1996d3c640 1 -- 172.21.15.60:0/4181912284 shutdown_connections 2023-10-16T14:10:27.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.969+0000 7f1996d3c640 1 -- 172.21.15.60:0/4181912284 wait complete. 2023-10-16T14:10:27.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.970+0000 7f1996d3c640 1 Processor -- start 2023-10-16T14:10:27.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.970+0000 7f1996d3c640 1 -- start start 2023-10-16T14:10:27.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.971+0000 7f1996d3c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 0x7f1990154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.971+0000 7f1996d3c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1990070450 0x7f19901f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.971+0000 7f1996d3c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1990159950 0x7f19900b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.971+0000 7f1996d3c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1990155250 con 0x7f199006c100 2023-10-16T14:10:27.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.971+0000 7f1996d3c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1990156b10 con 0x7f1990159950 2023-10-16T14:10:27.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.971+0000 7f1996d3c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1990156df0 con 0x7f1990070450 2023-10-16T14:10:27.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 0x7f1990154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 0x7f1990154d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35598/0 (socket says 172.21.15.60:35598) 2023-10-16T14:10:27.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 -- 172.21.15.60:0/1403385168 learned_addr learned my addr 172.21.15.60:0/1403385168 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:27.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f199653b640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1990159950 0x7f19900b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 -- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1990070450 msgr2=0x7f19901f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1990070450 0x7f19901f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 -- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1990159950 msgr2=0x7f19900b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1990159950 0x7f19900b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f1995d3a640 1 -- 172.21.15.60:0/1403385168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094d0 con 0x7f199006c100 2023-10-16T14:10:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.972+0000 7f199653b640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1990159950 0x7f19900b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.973+0000 7f1995d3a640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 0x7f1990154d10 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f198c02b8a0 tx=0x7f198c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.973+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c032860 con 0x7f199006c100 2023-10-16T14:10:27.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.973+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c0329e0 con 0x7f199006c100 2023-10-16T14:10:27.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.973+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c032d90 con 0x7f199006c100 2023-10-16T14:10:27.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.973+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19901570d0 con 0x7f199006c100 2023-10-16T14:10:27.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.973+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1990157690 con 0x7f199006c100 2023-10-16T14:10:27.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.974+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19901578b0 con 0x7f199006c100 2023-10-16T14:10:27.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.975+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f198c04e450 con 0x7f199006c100 2023-10-16T14:10:27.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.976+0000 7f19867fc640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f197406af90 0x7f197406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:27.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.976+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f198c0ab9b0 con 0x7f199006c100 2023-10-16T14:10:27.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.977+0000 7f1995539640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f197406af90 0x7f197406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:27.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.978+0000 7f1995539640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f197406af90 0x7f197406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19901f40c0 tx=0x7f1980008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:27.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:27.981+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f198c0769d0 con 0x7f199006c100 2023-10-16T14:10:28.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.218+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f199004e9b0 con 0x7f199006c100 2023-10-16T14:10:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.851+0000 7f19867fc640 1 -- 172.21.15.60:0/1403385168 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v112) v1 ==== 118+0+0 (secure 0 0 0) 0x7f198c079ff0 con 0x7f199006c100 2023-10-16T14:10:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.855+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f197406af90 msgr2=0x7f197406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:28.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.855+0000 7f1996d3c640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f197406af90 0x7f197406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19901f40c0 tx=0x7f1980008040 comp rx=0 tx=0).stop 2023-10-16T14:10:28.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.855+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 msgr2=0x7f1990154d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:28.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.855+0000 7f1996d3c640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 0x7f1990154d10 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f198c02b8a0 tx=0x7f198c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:28.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 shutdown_connections 2023-10-16T14:10:28.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f197406af90 0x7f197406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:28.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1990070450 0x7f19901f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:28.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1990159950 0x7f19900b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:28.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 --2- 172.21.15.60:0/1403385168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f199006c100 0x7f1990154d10 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:28.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 >> 172.21.15.60:0/1403385168 conn(0x7f19900ce6a0 msgr2=0x7f199006ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:28.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 shutdown_connections 2023-10-16T14:10:28.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:28.856+0000 7f1996d3c640 1 -- 172.21.15.60:0/1403385168 wait complete. 2023-10-16T14:10:28.879 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:10:28.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:10:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.152+0000 7fbfcdae5640 1 Processor -- start 2023-10-16T14:10:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.152+0000 7fbfcdae5640 1 -- start start 2023-10-16T14:10:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.153+0000 7fbfcdae5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfc80cded0 0x7fbfc80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.153+0000 7fbfcdae5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc815d150 con 0x7fbfc80b8480 2023-10-16T14:10:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.153+0000 7fbfcdae5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc80d1180 con 0x7fbfc815cd70 2023-10-16T14:10:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.153+0000 7fbfcdae5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc80d1460 con 0x7fbfc80cded0 2023-10-16T14:10:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfc7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbfc80b8480 0x7fbfc80cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45952/0 (socket says 172.21.15.60:45952) 2023-10-16T14:10:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfc7fff640 1 -- 172.21.15.60:0/349987644 learned_addr learned my addr 172.21.15.60:0/349987644 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfcd2e4640 1 --2- 172.21.15.60:0/349987644 >> v2:172.21.15.60:3300/0 conn(0x7fbfc80cded0 0x7fbfc80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfc77fe640 1 -- 172.21.15.60:0/349987644 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216837250 0 0) 0x7fbfc815d150 con 0x7fbfc80b8480 2023-10-16T14:10:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfc77fe640 1 -- 172.21.15.60:0/349987644 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb4003520 con 0x7fbfc80b8480 2023-10-16T14:10:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfcd2e4640 1 -- 172.21.15.60:0/349987644 >> v1:172.21.15.144:6789/0 conn(0x7fbfc815cd70 legacy=0x7fbfc80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfcd2e4640 1 -- 172.21.15.60:0/349987644 >> v1:172.21.15.60:6789/0 conn(0x7fbfc80b8480 legacy=0x7fbfc80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.154+0000 7fbfcd2e4640 1 -- 172.21.15.60:0/349987644 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc80d1740 con 0x7fbfc80cded0 2023-10-16T14:10:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.156+0000 7fbfcd2e4640 1 --2- 172.21.15.60:0/349987644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc80b5c90 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00097a0 tx=0x7fbfc002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561c33b1529c24bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.156+0000 7fbfc77fe640 1 -- 172.21.15.60:0/349987644 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0030610 con 0x7fbfc80cded0 2023-10-16T14:10:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.156+0000 7fbfc77fe640 1 -- 172.21.15.60:0/349987644 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0030770 con 0x7fbfc80cded0 2023-10-16T14:10:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.156+0000 7fbfc77fe640 1 -- 172.21.15.60:0/349987644 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0030b00 con 0x7fbfc80cded0 2023-10-16T14:10:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 -- 172.21.15.60:0/349987644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 msgr2=0x7fbfc80b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/349987644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc80b5c90 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00097a0 tx=0x7fbfc002b960 comp rx=0 tx=0).stop 2023-10-16T14:10:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 -- 172.21.15.60:0/349987644 shutdown_connections 2023-10-16T14:10:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/349987644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc80b5c90 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 -- 172.21.15.60:0/349987644 >> 172.21.15.60:0/349987644 conn(0x7fbfc80cf570 msgr2=0x7fbfc80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 -- 172.21.15.60:0/349987644 shutdown_connections 2023-10-16T14:10:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.157+0000 7fbfcdae5640 1 -- 172.21.15.60:0/349987644 wait complete. 2023-10-16T14:10:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.159+0000 7fbfcdae5640 1 Processor -- start 2023-10-16T14:10:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.159+0000 7fbfcdae5640 1 -- start start 2023-10-16T14:10:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfcdae5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfc80b8480 0x7fbfc81f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfcdae5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc815a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfcdae5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc815cd70 0x7fbfc80aa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfcdae5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80aaf40 con 0x7fbfc80cded0 2023-10-16T14:10:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfcdae5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc815ab00 con 0x7fbfc815cd70 2023-10-16T14:10:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfcdae5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80aac60 con 0x7fbfc80b8480 2023-10-16T14:10:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc815a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc815a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45004/0 (socket says 172.21.15.60:45004) 2023-10-16T14:10:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.160+0000 7fbfc7fff640 1 -- 172.21.15.60:0/2046224143 learned_addr learned my addr 172.21.15.60:0/2046224143 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfcd2e4640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc815cd70 0x7fbfc80aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfccae3640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfc80b8480 0x7fbfc81f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc7fff640 1 -- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfc80b8480 msgr2=0x7fbfc81f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc7fff640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfc80b8480 0x7fbfc81f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc7fff640 1 -- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc815cd70 msgr2=0x7fbfc80aa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc7fff640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc815cd70 0x7fbfc80aa380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc7fff640 1 -- 172.21.15.60:0/2046224143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0009400 con 0x7fbfc80cded0 2023-10-16T14:10:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfcd2e4640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc815cd70 0x7fbfc80aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfccae3640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfc80b8480 0x7fbfc81f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc7fff640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc815a5c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000c530 tx=0x7fbfb000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:29.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0005ce0 con 0x7fbfc80cded0 2023-10-16T14:10:29.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0005e40 con 0x7fbfc80cded0 2023-10-16T14:10:29.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc80cb540 con 0x7fbfc80cded0 2023-10-16T14:10:29.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00072b0 con 0x7fbfc80cded0 2023-10-16T14:10:29.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.161+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc80cbb00 con 0x7fbfc80cded0 2023-10-16T14:10:29.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.163+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc80cb3a0 con 0x7fbfc80cded0 2023-10-16T14:10:29.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.164+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbfb0007410 con 0x7fbfc80cded0 2023-10-16T14:10:29.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.165+0000 7fbfc57fa640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fbfa806af90 0x7fbfa806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:29.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.165+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfb0088d90 con 0x7fbfc80cded0 2023-10-16T14:10:29.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.165+0000 7fbfccae3640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fbfa806af90 0x7fbfa806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:29.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.167+0000 7fbfccae3640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fbfa806af90 0x7fbfa806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80a9d70 tx=0x7fbfb8009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:29.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.169+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbfc80cb3a0 con 0x7fbfc80cded0 2023-10-16T14:10:29.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.412+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbfc80cb3a0 con 0x7fbfc80cded0 2023-10-16T14:10:29.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.854+0000 7fbfc57fa640 1 -- 172.21.15.60:0/2046224143 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v113) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbfc80cb3a0 con 0x7fbfc80cded0 2023-10-16T14:10:29.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fbfa806af90 msgr2=0x7fbfa806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fbfa806af90 0x7fbfa806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80a9d70 tx=0x7fbfb8009040 comp rx=0 tx=0).stop 2023-10-16T14:10:29.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 msgr2=0x7fbfc815a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:29.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc815a5c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000c530 tx=0x7fbfb000c9f0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 shutdown_connections 2023-10-16T14:10:29.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fbfa806af90 0x7fbfa806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfc815cd70 0x7fbfc80aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfc80cded0 0x7fbfc815a5c0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 --2- 172.21.15.60:0/2046224143 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbfc80b8480 0x7fbfc81f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:29.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.858+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 >> 172.21.15.60:0/2046224143 conn(0x7fbfc80cf570 msgr2=0x7fbfc80b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:29.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.859+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 shutdown_connections 2023-10-16T14:10:29.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:29.859+0000 7fbfcdae5640 1 -- 172.21.15.60:0/2046224143 wait complete. 2023-10-16T14:10:29.885 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:10:29.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.109+0000 7fa81b59e640 1 Processor -- start 2023-10-16T14:10:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.109+0000 7fa81b59e640 1 -- start start 2023-10-16T14:10:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa81b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa81c0ad970 0x7fa81c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa81b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c15b9d0 con 0x7fa81c06c040 2023-10-16T14:10:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa81b59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c15bcb0 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa81b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c15bf90 con 0x7fa81c0ad970 2023-10-16T14:10:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa81a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa81c06c040 0x7fa81c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45960/0 (socket says 172.21.15.60:45960) 2023-10-16T14:10:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa813fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa81c0ad970 0x7fa81c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.111+0000 7fa81a59c640 1 -- 172.21.15.60:0/4269309422 learned_addr learned my addr 172.21.15.60:0/4269309422 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.112+0000 7fa813fff640 1 -- 172.21.15.60:0/4269309422 >> v1:172.21.15.144:6789/0 conn(0x7fa81c0ccfb0 legacy=0x7fa81c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.112+0000 7fa813fff640 1 -- 172.21.15.60:0/4269309422 >> v1:172.21.15.60:6789/0 conn(0x7fa81c06c040 legacy=0x7fa81c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.112+0000 7fa813fff640 1 -- 172.21.15.60:0/4269309422 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0cc270 con 0x7fa81c0ad970 2023-10-16T14:10:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.113+0000 7fa813fff640 1 --2- 172.21.15.60:0/4269309422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ad970 0x7fa81c0cca70 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fa8040097d0 tx=0x7fa80402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291c1ed1c5e6ca86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.113+0000 7fa819d9b640 1 -- 172.21.15.60:0/4269309422 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80402d040 con 0x7fa81c0ad970 2023-10-16T14:10:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.113+0000 7fa819d9b640 1 -- 172.21.15.60:0/4269309422 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa804031810 con 0x7fa81c0ad970 2023-10-16T14:10:30.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.113+0000 7fa819d9b640 1 -- 172.21.15.60:0/4269309422 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804031ba0 con 0x7fa81c0ad970 2023-10-16T14:10:30.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 -- 172.21.15.60:0/4269309422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ad970 msgr2=0x7fa81c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 --2- 172.21.15.60:0/4269309422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ad970 0x7fa81c0cca70 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fa8040097d0 tx=0x7fa80402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 -- 172.21.15.60:0/4269309422 shutdown_connections 2023-10-16T14:10:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 --2- 172.21.15.60:0/4269309422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ad970 0x7fa81c0cca70 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 -- 172.21.15.60:0/4269309422 >> 172.21.15.60:0/4269309422 conn(0x7fa81c0ce6a0 msgr2=0x7fa81c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:30.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 -- 172.21.15.60:0/4269309422 shutdown_connections 2023-10-16T14:10:30.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.114+0000 7fa81b59e640 1 -- 172.21.15.60:0/4269309422 wait complete. 2023-10-16T14:10:30.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.116+0000 7fa81b59e640 1 Processor -- start 2023-10-16T14:10:30.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.116+0000 7fa81b59e640 1 -- start start 2023-10-16T14:10:30.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa81c06c040 0x7fa81c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa81c0ad970 0x7fa81c0b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 0x7fa81c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1e47d0 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0b2cf0 con 0x7fa81c0ad970 2023-10-16T14:10:30.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0ae100 con 0x7fa81c06c040 2023-10-16T14:10:30.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 0x7fa81c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 0x7fa81c0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45024/0 (socket says 172.21.15.60:45024) 2023-10-16T14:10:30.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81ad9d640 1 -- 172.21.15.60:0/56953951 learned_addr learned my addr 172.21.15.60:0/56953951 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:30.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.117+0000 7fa81a59c640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa81c06c040 0x7fa81c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa813fff640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa81c0ad970 0x7fa81c0b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81ad9d640 1 -- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa81c06c040 msgr2=0x7fa81c15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81ad9d640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa81c06c040 0x7fa81c15ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81ad9d640 1 -- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa81c0ad970 msgr2=0x7fa81c0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81ad9d640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa81c0ad970 0x7fa81c0b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81ad9d640 1 -- 172.21.15.60:0/56953951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa804009400 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81a59c640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa81c06c040 0x7fa81c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:30.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81ad9d640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 0x7fa81c0b0b90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c780 tx=0x7fa81400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:30.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814003d80 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81400ce90 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8140052b0 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c0ae3e0 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.118+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c0b3220 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.120+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c06b330 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.121+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa814005410 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.121+0000 7fa812ffd640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa7fc06af90 0x7fa7fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.122+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa814088c60 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.122+0000 7fa81a59c640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa7fc06af90 0x7fa7fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.123+0000 7fa81a59c640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa7fc06af90 0x7fa7fc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa80c007fd0 tx=0x7fa80c009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:30.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.126+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa814053c00 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.359+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa81c04e9b0 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.360+0000 7fa812ffd640 1 -- 172.21.15.60:0/56953951 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4928 (secure 0 0 0) 0x7fa814017020 con 0x7fa81c0ccfb0 2023-10-16T14:10:30.364 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:30.364 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T14:10:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 113 2023-10-16T14:10:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.362+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa7fc06af90 msgr2=0x7fa7fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.362+0000 7fa81b59e640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa7fc06af90 0x7fa7fc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa80c007fd0 tx=0x7fa80c009040 comp rx=0 tx=0).stop 2023-10-16T14:10:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.362+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 msgr2=0x7fa81c0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.362+0000 7fa81b59e640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 0x7fa81c0b0b90 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c780 tx=0x7fa81400cc40 comp rx=0 tx=0).stop 2023-10-16T14:10:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 shutdown_connections 2023-10-16T14:10:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa7fc06af90 0x7fa7fc06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa81c0ccfb0 0x7fa81c0b0b90 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa81c0ad970 0x7fa81c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 --2- 172.21.15.60:0/56953951 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa81c06c040 0x7fa81c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 >> 172.21.15.60:0/56953951 conn(0x7fa81c0ce6a0 msgr2=0x7fa81c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 shutdown_connections 2023-10-16T14:10:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.363+0000 7fa81b59e640 1 -- 172.21.15.60:0/56953951 wait complete. 2023-10-16T14:10:30.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:30.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.613+0000 7fb291015640 1 Processor -- start 2023-10-16T14:10:30.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.614+0000 7fb291015640 1 -- start start 2023-10-16T14:10:30.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.615+0000 7fb291015640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c15ae80 0x7fb28c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.615+0000 7fb291015640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb28c0ce390 0x7fb28c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.615+0000 7fb291015640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28c15ed60 con 0x7fb28c15aaa0 2023-10-16T14:10:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.615+0000 7fb291015640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28c06b810 con 0x7fb28c0ce390 2023-10-16T14:10:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.615+0000 7fb291015640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c0707e0 con 0x7fb28c15ae80 2023-10-16T14:10:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb28c0ce390 0x7fb28c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb28c0ce390 0x7fb28c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45032/0 (socket says 172.21.15.60:45032) 2023-10-16T14:10:30.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 -- 172.21.15.60:0/405621898 learned_addr learned my addr 172.21.15.60:0/405621898 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:30.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28259c640 1 --2- 172.21.15.60:0/405621898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c15ae80 0x7fb28c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 -- 172.21.15.60:0/405621898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c15ae80 msgr2=0x7fb28c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:30.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 --2- 172.21.15.60:0/405621898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c15ae80 0x7fb28c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 -- 172.21.15.60:0/405621898 >> v1:172.21.15.144:6789/0 conn(0x7fb28c15aaa0 legacy=0x7fb28c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.616+0000 7fb28b59e640 1 -- 172.21.15.60:0/405621898 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c15ef90 con 0x7fb28c0ce390 2023-10-16T14:10:30.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.618+0000 7fb28b59e640 1 --2- 172.21.15.60:0/405621898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c0ce390 0x7fb28c070f50 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fb284009e50 tx=0x7fb28402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff3db77a5ce58b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:30.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.618+0000 7fb28a59c640 1 -- 172.21.15.60:0/405621898 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28402cb00 con 0x7fb28c0ce390 2023-10-16T14:10:30.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.618+0000 7fb28a59c640 1 -- 172.21.15.60:0/405621898 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28402cc60 con 0x7fb28c0ce390 2023-10-16T14:10:30.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.618+0000 7fb28a59c640 1 -- 172.21.15.60:0/405621898 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2840343f0 con 0x7fb28c0ce390 2023-10-16T14:10:30.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 -- 172.21.15.60:0/405621898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c0ce390 msgr2=0x7fb28c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 --2- 172.21.15.60:0/405621898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c0ce390 0x7fb28c070f50 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fb284009e50 tx=0x7fb28402bf60 comp rx=0 tx=0).stop 2023-10-16T14:10:30.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 -- 172.21.15.60:0/405621898 shutdown_connections 2023-10-16T14:10:30.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 --2- 172.21.15.60:0/405621898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c0ce390 0x7fb28c070f50 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 --2- 172.21.15.60:0/405621898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c15ae80 0x7fb28c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 -- 172.21.15.60:0/405621898 >> 172.21.15.60:0/405621898 conn(0x7fb28c0cb410 msgr2=0x7fb28c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:30.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 -- 172.21.15.60:0/405621898 shutdown_connections 2023-10-16T14:10:30.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.619+0000 7fb291015640 1 -- 172.21.15.60:0/405621898 wait complete. 2023-10-16T14:10:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.621+0000 7fb291015640 1 Processor -- start 2023-10-16T14:10:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.621+0000 7fb291015640 1 -- start start 2023-10-16T14:10:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb291015640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c0ce390 0x7fb28c14e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb291015640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 0x7fb28c0c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb291015640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb28c15ae80 0x7fb28c0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb291015640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1f6f20 con 0x7fb28c15aaa0 2023-10-16T14:10:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb291015640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c14ef30 con 0x7fb28c15ae80 2023-10-16T14:10:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb291015640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1f6c40 con 0x7fb28c0ce390 2023-10-16T14:10:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb28259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 0x7fb28c0c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb28259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 0x7fb28c0c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45042/0 (socket says 172.21.15.60:45042) 2023-10-16T14:10:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.622+0000 7fb28259c640 1 -- 172.21.15.60:0/2468799375 learned_addr learned my addr 172.21.15.60:0/2468799375 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28b59e640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb28c15ae80 0x7fb28c0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28ad9d640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c0ce390 0x7fb28c14e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28259c640 1 -- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c0ce390 msgr2=0x7fb28c14e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28259c640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c0ce390 0x7fb28c14e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28259c640 1 -- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb28c15ae80 msgr2=0x7fb28c0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28259c640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb28c15ae80 0x7fb28c0c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28259c640 1 -- 172.21.15.60:0/2468799375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb284009b30 con 0x7fb28c15aaa0 2023-10-16T14:10:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb28259c640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 0x7fb28c0c24a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fb27400c7f0 tx=0x7fb27400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274004220 con 0x7fb28c15aaa0 2023-10-16T14:10:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2740043a0 con 0x7fb28c15aaa0 2023-10-16T14:10:30.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274007860 con 0x7fb28c15aaa0 2023-10-16T14:10:30.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.623+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28c157a70 con 0x7fb28c15aaa0 2023-10-16T14:10:30.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.624+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28c158030 con 0x7fb28c15aaa0 2023-10-16T14:10:30.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.625+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c158250 con 0x7fb28c15aaa0 2023-10-16T14:10:30.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.626+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb274007bd0 con 0x7fb28c15aaa0 2023-10-16T14:10:30.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.627+0000 7fb283fff640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb26c06af90 0x7fb26c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:30.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.627+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb274089800 con 0x7fb28c15aaa0 2023-10-16T14:10:30.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.628+0000 7fb28ad9d640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb26c06af90 0x7fb26c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:30.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.629+0000 7fb28ad9d640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb26c06af90 0x7fb26c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb27c000f60 tx=0x7fb27c009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:30.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.632+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb274054680 con 0x7fb28c15aaa0 2023-10-16T14:10:30.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.829+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28c04e160 con 0x7fb28c15aaa0 2023-10-16T14:10:30.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.831+0000 7fb283fff640 1 -- 172.21.15.60:0/2468799375 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fb27401e090 con 0x7fb28c15aaa0 2023-10-16T14:10:30.836 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:30.836 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":231,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:27.831029+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:30.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb26c06af90 msgr2=0x7fb26c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb26c06af90 0x7fb26c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb27c000f60 tx=0x7fb27c009040 comp rx=0 tx=0).stop 2023-10-16T14:10:30.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 msgr2=0x7fb28c0c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:30.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 0x7fb28c0c24a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fb27400c7f0 tx=0x7fb27400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 shutdown_connections 2023-10-16T14:10:30.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb26c06af90 0x7fb26c06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb28c15ae80 0x7fb28c0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb28c15aaa0 0x7fb28c0c24a0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 --2- 172.21.15.60:0/2468799375 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb28c0ce390 0x7fb28c14e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:30.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 >> 172.21.15.60:0/2468799375 conn(0x7fb28c0cb410 msgr2=0x7fb28c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:30.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.834+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 shutdown_connections 2023-10-16T14:10:30.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:30.835+0000 7fb291015640 1 -- 172.21.15.60:0/2468799375 wait complete. 2023-10-16T14:10:30.859 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:10:30.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:31.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.081+0000 7f6a26032640 1 Processor -- start 2023-10-16T14:10:31.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.081+0000 7f6a26032640 1 -- start start 2023-10-16T14:10:31.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a26032640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a200ad970 0x7f6a200cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a26032640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a200ccfb0 0x7f6a200cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a26032640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2015b9d0 con 0x7f6a2006c040 2023-10-16T14:10:31.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a26032640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2015bcb0 con 0x7f6a200ccfb0 2023-10-16T14:10:31.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a26032640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2015bf90 con 0x7f6a200ad970 2023-10-16T14:10:31.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a25030640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6a2006c040 0x7f6a2006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45974/0 (socket says 172.21.15.60:45974) 2023-10-16T14:10:31.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a25030640 1 -- 172.21.15.60:0/3553851192 learned_addr learned my addr 172.21.15.60:0/3553851192 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:31.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.083+0000 7f6a25831640 1 --2- 172.21.15.60:0/3553851192 >> v2:172.21.15.60:3300/0 conn(0x7f6a200ccfb0 0x7f6a200cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a2482f640 1 --2- 172.21.15.60:0/3553851192 >> v2:172.21.15.144:3300/0 conn(0x7f6a200ad970 0x7f6a200cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a17fff640 1 -- 172.21.15.60:0/3553851192 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547735577 0 0) 0x7f6a2015b9d0 con 0x7f6a2006c040 2023-10-16T14:10:31.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a17fff640 1 -- 172.21.15.60:0/3553851192 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c003520 con 0x7f6a2006c040 2023-10-16T14:10:31.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a25831640 1 -- 172.21.15.60:0/3553851192 >> v2:172.21.15.144:3300/0 conn(0x7f6a200ad970 msgr2=0x7f6a200cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:31.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a25831640 1 --2- 172.21.15.60:0/3553851192 >> v2:172.21.15.144:3300/0 conn(0x7f6a200ad970 0x7f6a200cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a25831640 1 -- 172.21.15.60:0/3553851192 >> v1:172.21.15.60:6789/0 conn(0x7f6a2006c040 legacy=0x7f6a2006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.084+0000 7f6a25831640 1 -- 172.21.15.60:0/3553851192 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200cc270 con 0x7f6a200ccfb0 2023-10-16T14:10:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.085+0000 7f6a25831640 1 --2- 172.21.15.60:0/3553851192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200cd460 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009d90 tx=0x7f6a1802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca69373a59c7194 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a17fff640 1 -- 172.21.15.60:0/3553851192 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1802cc60 con 0x7f6a200ccfb0 2023-10-16T14:10:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a17fff640 1 -- 172.21.15.60:0/3553851192 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1802cdc0 con 0x7f6a200ccfb0 2023-10-16T14:10:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a17fff640 1 -- 172.21.15.60:0/3553851192 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18034660 con 0x7f6a200ccfb0 2023-10-16T14:10:31.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a26032640 1 -- 172.21.15.60:0/3553851192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 msgr2=0x7f6a200cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a26032640 1 --2- 172.21.15.60:0/3553851192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200cd460 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009d90 tx=0x7f6a1802beb0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a26032640 1 -- 172.21.15.60:0/3553851192 shutdown_connections 2023-10-16T14:10:31.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a26032640 1 --2- 172.21.15.60:0/3553851192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200cd460 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.086+0000 7f6a26032640 1 --2- 172.21.15.60:0/3553851192 >> v2:172.21.15.144:3300/0 conn(0x7f6a200ad970 0x7f6a200cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.087+0000 7f6a26032640 1 -- 172.21.15.60:0/3553851192 >> 172.21.15.60:0/3553851192 conn(0x7f6a200ce6a0 msgr2=0x7f6a200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.087+0000 7f6a26032640 1 -- 172.21.15.60:0/3553851192 shutdown_connections 2023-10-16T14:10:31.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.087+0000 7f6a26032640 1 -- 172.21.15.60:0/3553851192 wait complete. 2023-10-16T14:10:31.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.088+0000 7f6a26032640 1 Processor -- start 2023-10-16T14:10:31.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.089+0000 7f6a26032640 1 -- start start 2023-10-16T14:10:31.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a26032640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a2006c040 0x7f6a201f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a26032640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a200ad970 0x7f6a200b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a26032640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a26032640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200b2760 con 0x7f6a200ccfb0 2023-10-16T14:10:31.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a26032640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200af160 con 0x7f6a200ad970 2023-10-16T14:10:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a26032640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200afba0 con 0x7f6a2006c040 2023-10-16T14:10:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200aec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45070/0 (socket says 172.21.15.60:45070) 2023-10-16T14:10:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 -- 172.21.15.60:0/1353522575 learned_addr learned my addr 172.21.15.60:0/1353522575 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:31.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25030640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a2006c040 0x7f6a201f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a2482f640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a200ad970 0x7f6a200b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 -- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a2006c040 msgr2=0x7f6a201f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a2006c040 0x7f6a201f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 -- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a200ad970 msgr2=0x7f6a200b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a200ad970 0x7f6a200b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.090+0000 7f6a25831640 1 -- 172.21.15.60:0/1353522575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009a70 con 0x7f6a200ccfb0 2023-10-16T14:10:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.091+0000 7f6a25831640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200aec20 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009d90 tx=0x7f6a18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.091+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18040650 con 0x7f6a200ccfb0 2023-10-16T14:10:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.091+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a180407b0 con 0x7f6a200ccfb0 2023-10-16T14:10:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.091+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18040b60 con 0x7f6a200ccfb0 2023-10-16T14:10:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.091+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a200af590 con 0x7f6a200ccfb0 2023-10-16T14:10:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.091+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a200af830 con 0x7f6a200ccfb0 2023-10-16T14:10:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.093+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2006b330 con 0x7f6a200ccfb0 2023-10-16T14:10:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.094+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a18040cc0 con 0x7f6a200ccfb0 2023-10-16T14:10:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.094+0000 7f6a15ffb640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a0406af90 0x7f6a0406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.095+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a180ac570 con 0x7f6a200ccfb0 2023-10-16T14:10:31.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.095+0000 7f6a25030640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a0406af90 0x7f6a0406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.096+0000 7f6a25030640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a0406af90 0x7f6a0406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0011e0 tx=0x7f6a1c009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.099+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a18077510 con 0x7f6a200ccfb0 2023-10-16T14:10:31.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.292+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a200cc550 con 0x7f6a0406af90 2023-10-16T14:10:31.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.322+0000 7f6a15ffb640 1 -- 172.21.15.60:0/1353522575 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286783 (secure 0 0 0) 0x7f6a1c001ce0 con 0x7f6a0406af90 2023-10-16T14:10:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:31.334 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:31.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.329+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a0406af90 msgr2=0x7f6a0406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.329+0000 7f6a26032640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a0406af90 0x7f6a0406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0011e0 tx=0x7f6a1c009040 comp rx=0 tx=0).stop 2023-10-16T14:10:31.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.329+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 msgr2=0x7f6a200aec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.329+0000 7f6a26032640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200aec20 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009d90 tx=0x7f6a18004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:31.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 shutdown_connections 2023-10-16T14:10:31.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a0406af90 0x7f6a0406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a200ccfb0 0x7f6a200aec20 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a200ad970 0x7f6a200b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 --2- 172.21.15.60:0/1353522575 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a2006c040 0x7f6a201f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 >> 172.21.15.60:0/1353522575 conn(0x7f6a200ce6a0 msgr2=0x7f6a200ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:31.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 shutdown_connections 2023-10-16T14:10:31.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.330+0000 7f6a26032640 1 -- 172.21.15.60:0/1353522575 wait complete. 2023-10-16T14:10:31.356 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T14:10:30.177283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":627,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377384,"kb_used_data":23780,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754597336,"statfs":{"total":773094113280,"available":772707672064,"internally_reserved":0,"allocated":24350720,"data_stored":12442048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.401852"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761024+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:28.761024+0000","last_peered":"2023-10-16T14:10:28.761024+0000","last_clean":"2023-10-16T14:10:28.761024+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:28.761024+0000","last_undegraded":"2023-10-16T14:10:28.761024+0000","last_fullsized":"2023-10-16T14:10:28.761024+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759603+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:28.759603+0000","last_peered":"2023-10-16T14:10:28.759603+0000","last_clean":"2023-10-16T14:10:28.759603+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:28.759603+0000","last_undegraded":"2023-10-16T14:10:28.759603+0000","last_fullsized":"2023-10-16T14:10:28.759603+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845460+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:27.845460+0000","last_peered":"2023-10-16T14:10:27.845460+0000","last_clean":"2023-10-16T14:10:27.845460+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:27.845460+0000","last_undegraded":"2023-10-16T14:10:27.845460+0000","last_fullsized":"2023-10-16T14:10:27.845460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851231+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:27.851231+0000","last_peered":"2023-10-16T14:10:27.851231+0000","last_clean":"2023-10-16T14:10:27.851231+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:27.851231+0000","last_undegraded":"2023-10-16T14:10:27.851231+0000","last_fullsized":"2023-10-16T14:10:27.851231+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759117+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:28.759117+0000","last_peered":"2023-10-16T14:10:28.759117+0000","last_clean":"2023-10-16T14:10:28.759117+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:28.759117+0000","last_undegraded":"2023-10-16T14:10:28.759117+0000","last_fullsized":"2023-10-16T14:10:28.759117+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759037+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:28.759037+0000","last_peered":"2023-10-16T14:10:28.759037+0000","last_clean":"2023-10-16T14:10:28.759037+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:28.759037+0000","last_undegraded":"2023-10-16T14:10:28.759037+0000","last_fullsized":"2023-10-16T14:10:28.759037+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760164+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:28.760164+0000","last_peered":"2023-10-16T14:10:28.760164+0000","last_clean":"2023-10-16T14:10:28.760164+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:28.760164+0000","last_undegraded":"2023-10-16T14:10:28.760164+0000","last_fullsized":"2023-10-16T14:10:28.760164+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758409+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:28.758409+0000","last_peered":"2023-10-16T14:10:28.758409+0000","last_clean":"2023-10-16T14:10:28.758409+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:28.758409+0000","last_undegraded":"2023-10-16T14:10:28.758409+0000","last_fullsized":"2023-10-16T14:10:28.758409+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759995+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:28.759995+0000","last_peered":"2023-10-16T14:10:28.759995+0000","last_clean":"2023-10-16T14:10:28.759995+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:28.759995+0000","last_undegraded":"2023-10-16T14:10:28.759995+0000","last_fullsized":"2023-10-16T14:10:28.759995+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380768+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:28.380768+0000","last_peered":"2023-10-16T14:10:28.380768+0000","last_clean":"2023-10-16T14:10:28.380768+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:28.380768+0000","last_undegraded":"2023-10-16T14:10:28.380768+0000","last_fullsized":"2023-10-16T14:10:28.380768+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845678+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:27.845678+0000","last_peered":"2023-10-16T14:10:27.845678+0000","last_clean":"2023-10-16T14:10:27.845678+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:27.845678+0000","last_undegraded":"2023-10-16T14:10:27.845678+0000","last_fullsized":"2023-10-16T14:10:27.845678+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849448+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:27.849448+0000","last_peered":"2023-10-16T14:10:27.849448+0000","last_clean":"2023-10-16T14:10:27.849448+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:27.849448+0000","last_undegraded":"2023-10-16T14:10:27.849448+0000","last_fullsized":"2023-10-16T14:10:27.849448+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850229+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:27.850229+0000","last_peered":"2023-10-16T14:10:27.850229+0000","last_clean":"2023-10-16T14:10:27.850229+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:27.850229+0000","last_undegraded":"2023-10-16T14:10:27.850229+0000","last_fullsized":"2023-10-16T14:10:27.850229+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760487+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:28.760487+0000","last_peered":"2023-10-16T14:10:28.760487+0000","last_clean":"2023-10-16T14:10:28.760487+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:28.760487+0000","last_undegraded":"2023-10-16T14:10:28.760487+0000","last_fullsized":"2023-10-16T14:10:28.760487+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760283+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:28.760283+0000","last_peered":"2023-10-16T14:10:28.760283+0000","last_clean":"2023-10-16T14:10:28.760283+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:28.760283+0000","last_undegraded":"2023-10-16T14:10:28.760283+0000","last_fullsized":"2023-10-16T14:10:28.760283+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380111+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:28.380111+0000","last_peered":"2023-10-16T14:10:28.380111+0000","last_clean":"2023-10-16T14:10:28.380111+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:28.380111+0000","last_undegraded":"2023-10-16T14:10:28.380111+0000","last_fullsized":"2023-10-16T14:10:28.380111+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761479+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:28.761479+0000","last_peered":"2023-10-16T14:10:28.761479+0000","last_clean":"2023-10-16T14:10:28.761479+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:28.761479+0000","last_undegraded":"2023-10-16T14:10:28.761479+0000","last_fullsized":"2023-10-16T14:10:28.761479+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381002+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:28.381002+0000","last_peered":"2023-10-16T14:10:28.381002+0000","last_clean":"2023-10-16T14:10:28.381002+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:28.381002+0000","last_undegraded":"2023-10-16T14:10:28.381002+0000","last_fullsized":"2023-10-16T14:10:28.381002+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381139+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:28.381139+0000","last_peered":"2023-10-16T14:10:28.381139+0000","last_clean":"2023-10-16T14:10:28.381139+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:28.381139+0000","last_undegraded":"2023-10-16T14:10:28.381139+0000","last_fullsized":"2023-10-16T14:10:28.381139+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759538+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:28.759538+0000","last_peered":"2023-10-16T14:10:28.759538+0000","last_clean":"2023-10-16T14:10:28.759538+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:28.759538+0000","last_undegraded":"2023-10-16T14:10:28.759538+0000","last_fullsized":"2023-10-16T14:10:28.759538+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850789+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:27.850789+0000","last_peered":"2023-10-16T14:10:27.850789+0000","last_clean":"2023-10-16T14:10:27.850789+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:27.850789+0000","last_undegraded":"2023-10-16T14:10:27.850789+0000","last_fullsized":"2023-10-16T14:10:27.850789+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637693+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:28.637693+0000","last_peered":"2023-10-16T14:10:28.637693+0000","last_clean":"2023-10-16T14:10:28.637693+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:28.637693+0000","last_undegraded":"2023-10-16T14:10:28.637693+0000","last_fullsized":"2023-10-16T14:10:28.637693+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637416+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:28.637416+0000","last_peered":"2023-10-16T14:10:28.637416+0000","last_clean":"2023-10-16T14:10:28.637416+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:28.637416+0000","last_undegraded":"2023-10-16T14:10:28.637416+0000","last_fullsized":"2023-10-16T14:10:28.637416+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.848305+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:27.848305+0000","last_peered":"2023-10-16T14:10:27.848305+0000","last_clean":"2023-10-16T14:10:27.848305+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:27.848305+0000","last_undegraded":"2023-10-16T14:10:27.848305+0000","last_fullsized":"2023-10-16T14:10:27.848305+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759688+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:28.759688+0000","last_peered":"2023-10-16T14:10:28.759688+0000","last_clean":"2023-10-16T14:10:28.759688+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:28.759688+0000","last_undegraded":"2023-10-16T14:10:28.759688+0000","last_fullsized":"2023-10-16T14:10:28.759688+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761149+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:28.761149+0000","last_peered":"2023-10-16T14:10:28.761149+0000","last_clean":"2023-10-16T14:10:28.761149+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:28.761149+0000","last_undegraded":"2023-10-16T14:10:28.761149+0000","last_fullsized":"2023-10-16T14:10:28.761149+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849296+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:27.849296+0000","last_peered":"2023-10-16T14:10:27.849296+0000","last_clean":"2023-10-16T14:10:27.849296+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:27.849296+0000","last_undegraded":"2023-10-16T14:10:27.849296+0000","last_fullsized":"2023-10-16T14:10:27.849296+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845100+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:27.845100+0000","last_peered":"2023-10-16T14:10:27.845100+0000","last_clean":"2023-10-16T14:10:27.845100+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:27.845100+0000","last_undegraded":"2023-10-16T14:10:27.845100+0000","last_fullsized":"2023-10-16T14:10:27.845100+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849906+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:27.849906+0000","last_peered":"2023-10-16T14:10:27.849906+0000","last_clean":"2023-10-16T14:10:27.849906+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:27.849906+0000","last_undegraded":"2023-10-16T14:10:27.849906+0000","last_fullsized":"2023-10-16T14:10:27.849906+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759089+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:28.759089+0000","last_peered":"2023-10-16T14:10:28.759089+0000","last_clean":"2023-10-16T14:10:28.759089+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:28.759089+0000","last_undegraded":"2023-10-16T14:10:28.759089+0000","last_fullsized":"2023-10-16T14:10:28.759089+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380100+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:28.380100+0000","last_peered":"2023-10-16T14:10:28.380100+0000","last_clean":"2023-10-16T14:10:28.380100+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:28.380100+0000","last_undegraded":"2023-10-16T14:10:28.380100+0000","last_fullsized":"2023-10-16T14:10:28.380100+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759401+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:28.759401+0000","last_peered":"2023-10-16T14:10:28.759401+0000","last_clean":"2023-10-16T14:10:28.759401+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:28.759401+0000","last_undegraded":"2023-10-16T14:10:28.759401+0000","last_fullsized":"2023-10-16T14:10:28.759401+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759519+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:28.759519+0000","last_peered":"2023-10-16T14:10:28.759519+0000","last_clean":"2023-10-16T14:10:28.759519+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:28.759519+0000","last_undegraded":"2023-10-16T14:10:28.759519+0000","last_fullsized":"2023-10-16T14:10:28.759519+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":16,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849655+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:27.849655+0000","last_peered":"2023-10-16T14:10:27.849655+0000","last_clean":"2023-10-16T14:10:27.849655+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:27.849655+0000","last_undegraded":"2023-10-16T14:10:27.849655+0000","last_fullsized":"2023-10-16T14:10:27.849655+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637967+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:28.637967+0000","last_peered":"2023-10-16T14:10:28.637967+0000","last_clean":"2023-10-16T14:10:28.637967+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:28.637967+0000","last_undegraded":"2023-10-16T14:10:28.637967+0000","last_fullsized":"2023-10-16T14:10:28.637967+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759844+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:28.759844+0000","last_peered":"2023-10-16T14:10:28.759844+0000","last_clean":"2023-10-16T14:10:28.759844+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:28.759844+0000","last_undegraded":"2023-10-16T14:10:28.759844+0000","last_fullsized":"2023-10-16T14:10:28.759844+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761520+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:28.761520+0000","last_peered":"2023-10-16T14:10:28.761520+0000","last_clean":"2023-10-16T14:10:28.761520+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:28.761520+0000","last_undegraded":"2023-10-16T14:10:28.761520+0000","last_fullsized":"2023-10-16T14:10:28.761520+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845645+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:27.845645+0000","last_peered":"2023-10-16T14:10:27.845645+0000","last_clean":"2023-10-16T14:10:27.845645+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:27.845645+0000","last_undegraded":"2023-10-16T14:10:27.845645+0000","last_fullsized":"2023-10-16T14:10:27.845645+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381105+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:28.381105+0000","last_peered":"2023-10-16T14:10:28.381105+0000","last_clean":"2023-10-16T14:10:28.381105+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:28.381105+0000","last_undegraded":"2023-10-16T14:10:28.381105+0000","last_fullsized":"2023-10-16T14:10:28.381105+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851635+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:27.851635+0000","last_peered":"2023-10-16T14:10:27.851635+0000","last_clean":"2023-10-16T14:10:27.851635+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:27.851635+0000","last_undegraded":"2023-10-16T14:10:27.851635+0000","last_fullsized":"2023-10-16T14:10:27.851635+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851136+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:27.851136+0000","last_peered":"2023-10-16T14:10:27.851136+0000","last_clean":"2023-10-16T14:10:27.851136+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:27.851136+0000","last_undegraded":"2023-10-16T14:10:27.851136+0000","last_fullsized":"2023-10-16T14:10:27.851136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759683+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:28.759683+0000","last_peered":"2023-10-16T14:10:28.759683+0000","last_clean":"2023-10-16T14:10:28.759683+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:28.759683+0000","last_undegraded":"2023-10-16T14:10:28.759683+0000","last_fullsized":"2023-10-16T14:10:28.759683+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758605+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:28.758605+0000","last_peered":"2023-10-16T14:10:28.758605+0000","last_clean":"2023-10-16T14:10:28.758605+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:28.758605+0000","last_undegraded":"2023-10-16T14:10:28.758605+0000","last_fullsized":"2023-10-16T14:10:28.758605+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758200+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:28.758200+0000","last_peered":"2023-10-16T14:10:28.758200+0000","last_clean":"2023-10-16T14:10:28.758200+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:28.758200+0000","last_undegraded":"2023-10-16T14:10:28.758200+0000","last_fullsized":"2023-10-16T14:10:28.758200+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849860+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:27.849860+0000","last_peered":"2023-10-16T14:10:27.849860+0000","last_clean":"2023-10-16T14:10:27.849860+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:27.849860+0000","last_undegraded":"2023-10-16T14:10:27.849860+0000","last_fullsized":"2023-10-16T14:10:27.849860+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637029+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:28.637029+0000","last_peered":"2023-10-16T14:10:28.637029+0000","last_clean":"2023-10-16T14:10:28.637029+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:28.637029+0000","last_undegraded":"2023-10-16T14:10:28.637029+0000","last_fullsized":"2023-10-16T14:10:28.637029+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.852429+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:27.852429+0000","last_peered":"2023-10-16T14:10:27.852429+0000","last_clean":"2023-10-16T14:10:27.852429+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:27.852429+0000","last_undegraded":"2023-10-16T14:10:27.852429+0000","last_fullsized":"2023-10-16T14:10:27.852429+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759375+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:28.759375+0000","last_peered":"2023-10-16T14:10:28.759375+0000","last_clean":"2023-10-16T14:10:28.759375+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:28.759375+0000","last_undegraded":"2023-10-16T14:10:28.759375+0000","last_fullsized":"2023-10-16T14:10:28.759375+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845837+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:27.845837+0000","last_peered":"2023-10-16T14:10:27.845837+0000","last_clean":"2023-10-16T14:10:27.845837+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:27.845837+0000","last_undegraded":"2023-10-16T14:10:27.845837+0000","last_fullsized":"2023-10-16T14:10:27.845837+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.638017+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:28.638017+0000","last_peered":"2023-10-16T14:10:28.638017+0000","last_clean":"2023-10-16T14:10:28.638017+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:28.638017+0000","last_undegraded":"2023-10-16T14:10:28.638017+0000","last_fullsized":"2023-10-16T14:10:28.638017+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851609+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:27.851609+0000","last_peered":"2023-10-16T14:10:27.851609+0000","last_clean":"2023-10-16T14:10:27.851609+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:27.851609+0000","last_undegraded":"2023-10-16T14:10:27.851609+0000","last_fullsized":"2023-10-16T14:10:27.851609+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381292+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:28.381292+0000","last_peered":"2023-10-16T14:10:28.381292+0000","last_clean":"2023-10-16T14:10:28.381292+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:28.381292+0000","last_undegraded":"2023-10-16T14:10:28.381292+0000","last_fullsized":"2023-10-16T14:10:28.381292+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761145+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:28.761145+0000","last_peered":"2023-10-16T14:10:28.761145+0000","last_clean":"2023-10-16T14:10:28.761145+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:28.761145+0000","last_undegraded":"2023-10-16T14:10:28.761145+0000","last_fullsized":"2023-10-16T14:10:28.761145+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"230'6","reported_seq":18,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851860+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:27.851860+0000","last_peered":"2023-10-16T14:10:27.851860+0000","last_clean":"2023-10-16T14:10:27.851860+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:27.851860+0000","last_undegraded":"2023-10-16T14:10:27.851860+0000","last_fullsized":"2023-10-16T14:10:27.851860+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759578+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:28.759578+0000","last_peered":"2023-10-16T14:10:28.759578+0000","last_clean":"2023-10-16T14:10:28.759578+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:28.759578+0000","last_undegraded":"2023-10-16T14:10:28.759578+0000","last_fullsized":"2023-10-16T14:10:28.759578+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637467+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:28.637467+0000","last_peered":"2023-10-16T14:10:28.637467+0000","last_clean":"2023-10-16T14:10:28.637467+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:28.637467+0000","last_undegraded":"2023-10-16T14:10:28.637467+0000","last_fullsized":"2023-10-16T14:10:28.637467+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380848+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:28.380848+0000","last_peered":"2023-10-16T14:10:28.380848+0000","last_clean":"2023-10-16T14:10:28.380848+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:28.380848+0000","last_undegraded":"2023-10-16T14:10:28.380848+0000","last_fullsized":"2023-10-16T14:10:28.380848+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637980+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:28.637980+0000","last_peered":"2023-10-16T14:10:28.637980+0000","last_clean":"2023-10-16T14:10:28.637980+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:28.637980+0000","last_undegraded":"2023-10-16T14:10:28.637980+0000","last_fullsized":"2023-10-16T14:10:28.637980+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845073+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:27.845073+0000","last_peered":"2023-10-16T14:10:27.845073+0000","last_clean":"2023-10-16T14:10:27.845073+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:27.845073+0000","last_undegraded":"2023-10-16T14:10:27.845073+0000","last_fullsized":"2023-10-16T14:10:27.845073+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380567+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:28.380567+0000","last_peered":"2023-10-16T14:10:28.380567+0000","last_clean":"2023-10-16T14:10:28.380567+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:28.380567+0000","last_undegraded":"2023-10-16T14:10:28.380567+0000","last_fullsized":"2023-10-16T14:10:28.380567+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760622+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:28.760622+0000","last_peered":"2023-10-16T14:10:28.760622+0000","last_clean":"2023-10-16T14:10:28.760622+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:28.760622+0000","last_undegraded":"2023-10-16T14:10:28.760622+0000","last_fullsized":"2023-10-16T14:10:28.760622+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":20,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845181+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:27.845181+0000","last_peered":"2023-10-16T14:10:27.845181+0000","last_clean":"2023-10-16T14:10:27.845181+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:27.845181+0000","last_undegraded":"2023-10-16T14:10:27.845181+0000","last_fullsized":"2023-10-16T14:10:27.845181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759751+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:28.759751+0000","last_peered":"2023-10-16T14:10:28.759751+0000","last_clean":"2023-10-16T14:10:28.759751+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:28.759751+0000","last_undegraded":"2023-10-16T14:10:28.759751+0000","last_fullsized":"2023-10-16T14:10:28.759751+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759630+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:28.759630+0000","last_peered":"2023-10-16T14:10:28.759630+0000","last_clean":"2023-10-16T14:10:28.759630+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:28.759630+0000","last_undegraded":"2023-10-16T14:10:28.759630+0000","last_fullsized":"2023-10-16T14:10:28.759630+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759678+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:28.759678+0000","last_peered":"2023-10-16T14:10:28.759678+0000","last_clean":"2023-10-16T14:10:28.759678+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:28.759678+0000","last_undegraded":"2023-10-16T14:10:28.759678+0000","last_fullsized":"2023-10-16T14:10:28.759678+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759892+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:28.759892+0000","last_peered":"2023-10-16T14:10:28.759892+0000","last_clean":"2023-10-16T14:10:28.759892+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:28.759892+0000","last_undegraded":"2023-10-16T14:10:28.759892+0000","last_fullsized":"2023-10-16T14:10:28.759892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845385+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:27.845385+0000","last_peered":"2023-10-16T14:10:27.845385+0000","last_clean":"2023-10-16T14:10:27.845385+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:27.845385+0000","last_undegraded":"2023-10-16T14:10:27.845385+0000","last_fullsized":"2023-10-16T14:10:27.845385+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381352+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:28.381352+0000","last_peered":"2023-10-16T14:10:28.381352+0000","last_clean":"2023-10-16T14:10:28.381352+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:28.381352+0000","last_undegraded":"2023-10-16T14:10:28.381352+0000","last_fullsized":"2023-10-16T14:10:28.381352+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380945+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:28.380945+0000","last_peered":"2023-10-16T14:10:28.380945+0000","last_clean":"2023-10-16T14:10:28.380945+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:28.380945+0000","last_undegraded":"2023-10-16T14:10:28.380945+0000","last_fullsized":"2023-10-16T14:10:28.380945+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845364+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:27.845364+0000","last_peered":"2023-10-16T14:10:27.845364+0000","last_clean":"2023-10-16T14:10:27.845364+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:27.845364+0000","last_undegraded":"2023-10-16T14:10:27.845364+0000","last_fullsized":"2023-10-16T14:10:27.845364+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380483+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:28.380483+0000","last_peered":"2023-10-16T14:10:28.380483+0000","last_clean":"2023-10-16T14:10:28.380483+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:28.380483+0000","last_undegraded":"2023-10-16T14:10:28.380483+0000","last_fullsized":"2023-10-16T14:10:28.380483+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380860+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:28.380860+0000","last_peered":"2023-10-16T14:10:28.380860+0000","last_clean":"2023-10-16T14:10:28.380860+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:28.380860+0000","last_undegraded":"2023-10-16T14:10:28.380860+0000","last_fullsized":"2023-10-16T14:10:28.380860+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845100+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:27.845100+0000","last_peered":"2023-10-16T14:10:27.845100+0000","last_clean":"2023-10-16T14:10:27.845100+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:27.845100+0000","last_undegraded":"2023-10-16T14:10:27.845100+0000","last_fullsized":"2023-10-16T14:10:27.845100+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761155+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:28.761155+0000","last_peered":"2023-10-16T14:10:28.761155+0000","last_clean":"2023-10-16T14:10:28.761155+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:28.761155+0000","last_undegraded":"2023-10-16T14:10:28.761155+0000","last_fullsized":"2023-10-16T14:10:28.761155+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758860+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:28.758860+0000","last_peered":"2023-10-16T14:10:28.758860+0000","last_clean":"2023-10-16T14:10:28.758860+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:28.758860+0000","last_undegraded":"2023-10-16T14:10:28.758860+0000","last_fullsized":"2023-10-16T14:10:28.758860+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760527+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:28.760527+0000","last_peered":"2023-10-16T14:10:28.760527+0000","last_clean":"2023-10-16T14:10:28.760527+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:28.760527+0000","last_undegraded":"2023-10-16T14:10:28.760527+0000","last_fullsized":"2023-10-16T14:10:28.760527+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637480+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:28.637480+0000","last_peered":"2023-10-16T14:10:28.637480+0000","last_clean":"2023-10-16T14:10:28.637480+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:28.637480+0000","last_undegraded":"2023-10-16T14:10:28.637480+0000","last_fullsized":"2023-10-16T14:10:28.637480+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758656+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:28.758656+0000","last_peered":"2023-10-16T14:10:28.758656+0000","last_clean":"2023-10-16T14:10:28.758656+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:28.758656+0000","last_undegraded":"2023-10-16T14:10:28.758656+0000","last_fullsized":"2023-10-16T14:10:28.758656+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849333+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:27.849333+0000","last_peered":"2023-10-16T14:10:27.849333+0000","last_clean":"2023-10-16T14:10:27.849333+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:27.849333+0000","last_undegraded":"2023-10-16T14:10:27.849333+0000","last_fullsized":"2023-10-16T14:10:27.849333+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845797+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:27.845797+0000","last_peered":"2023-10-16T14:10:27.845797+0000","last_clean":"2023-10-16T14:10:27.845797+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:27.845797+0000","last_undegraded":"2023-10-16T14:10:27.845797+0000","last_fullsized":"2023-10-16T14:10:27.845797+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761448+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:28.761448+0000","last_peered":"2023-10-16T14:10:28.761448+0000","last_clean":"2023-10-16T14:10:28.761448+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:28.761448+0000","last_undegraded":"2023-10-16T14:10:28.761448+0000","last_fullsized":"2023-10-16T14:10:28.761448+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.852382+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:27.852382+0000","last_peered":"2023-10-16T14:10:27.852382+0000","last_clean":"2023-10-16T14:10:27.852382+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:27.852382+0000","last_undegraded":"2023-10-16T14:10:27.852382+0000","last_fullsized":"2023-10-16T14:10:27.852382+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759300+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:28.759300+0000","last_peered":"2023-10-16T14:10:28.759300+0000","last_clean":"2023-10-16T14:10:28.759300+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:28.759300+0000","last_undegraded":"2023-10-16T14:10:28.759300+0000","last_fullsized":"2023-10-16T14:10:28.759300+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759379+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:28.759379+0000","last_peered":"2023-10-16T14:10:28.759379+0000","last_clean":"2023-10-16T14:10:28.759379+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:28.759379+0000","last_undegraded":"2023-10-16T14:10:28.759379+0000","last_fullsized":"2023-10-16T14:10:28.759379+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637873+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:28.637873+0000","last_peered":"2023-10-16T14:10:28.637873+0000","last_clean":"2023-10-16T14:10:28.637873+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:28.637873+0000","last_undegraded":"2023-10-16T14:10:28.637873+0000","last_fullsized":"2023-10-16T14:10:28.637873+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845491+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:27.845491+0000","last_peered":"2023-10-16T14:10:27.845491+0000","last_clean":"2023-10-16T14:10:27.845491+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:27.845491+0000","last_undegraded":"2023-10-16T14:10:27.845491+0000","last_fullsized":"2023-10-16T14:10:27.845491+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760859+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:28.760859+0000","last_peered":"2023-10-16T14:10:28.760859+0000","last_clean":"2023-10-16T14:10:28.760859+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:28.760859+0000","last_undegraded":"2023-10-16T14:10:28.760859+0000","last_fullsized":"2023-10-16T14:10:28.760859+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851349+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:27.851349+0000","last_peered":"2023-10-16T14:10:27.851349+0000","last_clean":"2023-10-16T14:10:27.851349+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:27.851349+0000","last_undegraded":"2023-10-16T14:10:27.851349+0000","last_fullsized":"2023-10-16T14:10:27.851349+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850071+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:27.850071+0000","last_peered":"2023-10-16T14:10:27.850071+0000","last_clean":"2023-10-16T14:10:27.850071+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:27.850071+0000","last_undegraded":"2023-10-16T14:10:27.850071+0000","last_fullsized":"2023-10-16T14:10:27.850071+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759811+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:28.759811+0000","last_peered":"2023-10-16T14:10:28.759811+0000","last_clean":"2023-10-16T14:10:28.759811+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:28.759811+0000","last_undegraded":"2023-10-16T14:10:28.759811+0000","last_fullsized":"2023-10-16T14:10:28.759811+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845021+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:27.845021+0000","last_peered":"2023-10-16T14:10:27.845021+0000","last_clean":"2023-10-16T14:10:27.845021+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:27.845021+0000","last_undegraded":"2023-10-16T14:10:27.845021+0000","last_fullsized":"2023-10-16T14:10:27.845021+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637746+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:28.637746+0000","last_peered":"2023-10-16T14:10:28.637746+0000","last_clean":"2023-10-16T14:10:28.637746+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:28.637746+0000","last_undegraded":"2023-10-16T14:10:28.637746+0000","last_fullsized":"2023-10-16T14:10:28.637746+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758964+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:28.758964+0000","last_peered":"2023-10-16T14:10:28.758964+0000","last_clean":"2023-10-16T14:10:28.758964+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:28.758964+0000","last_undegraded":"2023-10-16T14:10:28.758964+0000","last_fullsized":"2023-10-16T14:10:28.758964+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849034+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:27.849034+0000","last_peered":"2023-10-16T14:10:27.849034+0000","last_clean":"2023-10-16T14:10:27.849034+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:27.849034+0000","last_undegraded":"2023-10-16T14:10:27.849034+0000","last_fullsized":"2023-10-16T14:10:27.849034+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760640+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:28.760640+0000","last_peered":"2023-10-16T14:10:28.760640+0000","last_clean":"2023-10-16T14:10:28.760640+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:28.760640+0000","last_undegraded":"2023-10-16T14:10:28.760640+0000","last_fullsized":"2023-10-16T14:10:28.760640+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850344+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:27.850344+0000","last_peered":"2023-10-16T14:10:27.850344+0000","last_clean":"2023-10-16T14:10:27.850344+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:27.850344+0000","last_undegraded":"2023-10-16T14:10:27.850344+0000","last_fullsized":"2023-10-16T14:10:27.850344+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1222,"reported_epoch":230,"state":"active+clean","last_fresh":"2023-10-16T14:10:26.817840+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:26.817840+0000","last_peered":"2023-10-16T14:10:26.817840+0000","last_clean":"2023-10-16T14:10:26.817840+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:26.817840+0000","last_undegraded":"2023-10-16T14:10:26.817840+0000","last_fullsized":"2023-10-16T14:10:26.817840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":3497984,"data_stored":2001404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":1.575}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.98699999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.482},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.4239999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.7969999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.133},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.534}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.7709999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.944},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.8939999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543299,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42336,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329504,"statfs":{"total":96636764160,"available":96593412096,"internally_reserved":0,"allocated":2895872,"data_stored":1410006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":0.998},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":0.77200000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.099}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.0580000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.246},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.3959999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.609}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.4590000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.1990000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543300,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1403706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0900000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.79400000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.02}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.1379999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.3220000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.631}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.482}]}]},{"osd":4,"up_from":14,"seq":60129543300,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47904,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323936,"statfs":{"total":96636764160,"available":96587710464,"internally_reserved":0,"allocated":2899968,"data_stored":1404840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.21,"15min":1.2210000000000001},"min":{"1min":0.81299999999999994,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.679,"5min":1.7829999999999999,"15min":1.9039999999999999},"last":1.762},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.179,"15min":1.2},"min":{"1min":0.92900000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.75,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.26,"15min":1.254},"min":{"1min":0.89700000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.516,"5min":1.7090000000000001,"15min":1.78},"last":1.282},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.93000000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.585,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.077}]},{"osd":2,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.327,"15min":1.321},"min":{"1min":0.94299999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.6850000000000001,"5min":1.865,"15min":1.865},"last":1.714},{"interface":"front","average":{"1min":1.367,"5min":1.363,"15min":1.383},"min":{"1min":0.91800000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.796,"5min":1.796,"15min":2.0110000000000001},"last":1.4550000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.294,"15min":1.3109999999999999},"min":{"1min":1.0549999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.772,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.25},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4379999999999999,"15min":1.4179999999999999},"min":{"1min":0.92800000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.7030000000000001,"5min":1.8759999999999999,"15min":1.919},"last":1.381}]},{"osd":5,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.337},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":1.7170000000000001,"5min":1.827,"15min":1.889},"last":1.329},{"interface":"front","average":{"1min":1.425,"5min":1.359,"15min":1.339},"min":{"1min":0.90900000000000003,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.921,"5min":1.921,"15min":1.9510000000000001},"last":1.0389999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.391,"15min":1.3899999999999999},"min":{"1min":1.02,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.865,"5min":1.9159999999999999,"15min":1.956},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.494,"15min":1.522},"min":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":0.94199999999999995},"max":{"1min":1.869,"5min":1.869,"15min":2.6429999999999998},"last":1.871}]},{"osd":7,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.516,"15min":1.5009999999999999},"min":{"1min":1.2270000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.873,"5min":2.0049999999999999,"15min":2.1949999999999998},"last":1.137},{"interface":"front","average":{"1min":1.651,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.2170000000000001,"5min":1.008,"15min":1.008},"max":{"1min":2.1280000000000001,"5min":2.1349999999999998,"15min":2.3660000000000001},"last":1.6200000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543300,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2904064,"data_stored":1409606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":1.1599999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":1.456}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":1.089}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.77},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":1.6879999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.954},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":1.732}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.8200000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":1.395},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.9299999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52056,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319784,"statfs":{"total":96636764160,"available":96583458816,"internally_reserved":0,"allocated":2891776,"data_stored":1409424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.89000000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":1.5860000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.5},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.6299999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":1.726}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.389}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":1.524}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.849},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.6890000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543298,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52056,"kb_used_data":2820,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319784,"statfs":{"total":96636764160,"available":96583458816,"internally_reserved":0,"allocated":2887680,"data_stored":1403180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":0.83799999999999997}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":0.73899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543299,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1999882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.96399999999999997,"15min":0.94499999999999995},"min":{"1min":0.68400000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.704,"5min":1.704,"15min":1.891},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.119,"15min":1.1779999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.48699999999999999},"max":{"1min":1.6539999999999999,"5min":1.659,"15min":1.9299999999999999},"last":0.84799999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.177,"15min":1.1759999999999999},"min":{"1min":0.90600000000000003,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.617,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.234,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.60199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.78,"5min":1.893,"15min":1.893},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.198,"15min":1.1699999999999999},"min":{"1min":0.63400000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7869999999999999,"5min":1.8520000000000001,"15min":1.8600000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3859999999999999,"15min":1.391},"min":{"1min":0.79300000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9630000000000001,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.88100000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.365,"15min":1.389},"min":{"1min":1.028,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":1.6870000000000001,"5min":1.819,"15min":2.048},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5629999999999999,"15min":1.53},"min":{"1min":1.254,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.1070000000000002,"5min":2.601,"15min":2.601},"last":0.99099999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5089999999999999,"15min":1.518},"min":{"1min":0.95999999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.867,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.48,"15min":1.4970000000000001},"min":{"1min":1.145,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.796,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.329}]},{"osd":6,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.641,"15min":1.6259999999999999},"min":{"1min":1.1060000000000001,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.2549999999999999,"5min":2.669,"15min":2.669},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.579,"15min":1.6100000000000001},"min":{"1min":1.0649999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.032,"5min":2.355,"15min":2.3769999999999998},"last":1.367}]},{"osd":7,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.694,"15min":1.722},"min":{"1min":1.181,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.008,"5min":2.484,"15min":2.5270000000000001},"last":1.419},{"interface":"front","average":{"1min":1.643,"5min":1.6539999999999999,"15min":1.657},"min":{"1min":1.1950000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.9350000000000001,"5min":2.2949999999999999,"15min":2.464},"last":1.1439999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:31.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:31.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.627+0000 7efe26787640 1 Processor -- start 2023-10-16T14:10:31.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.628+0000 7efe26787640 1 -- start start 2023-10-16T14:10:31.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.629+0000 7efe26787640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 0x7efe2015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.629+0000 7efe26787640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe2006c100 0x7efe2006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.629+0000 7efe26787640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2015aef0 con 0x7efe20159950 2023-10-16T14:10:31.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.629+0000 7efe26787640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2015e490 con 0x7efe2006c100 2023-10-16T14:10:31.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.629+0000 7efe26787640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2015d910 con 0x7efe20070b60 2023-10-16T14:10:31.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe25785640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efe20159950 0x7efe20070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45980/0 (socket says 172.21.15.60:45980) 2023-10-16T14:10:31.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe25785640 1 -- 172.21.15.60:0/1096441657 learned_addr learned my addr 172.21.15.60:0/1096441657 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:31.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe24f84640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 0x7efe2015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe25f86640 1 --2- 172.21.15.60:0/1096441657 >> v2:172.21.15.144:3300/0 conn(0x7efe2006c100 0x7efe2006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247236865 0 0) 0x7efe1c001110 con 0x7efe20159950 2023-10-16T14:10:31.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003520 con 0x7efe20159950 2023-10-16T14:10:31.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.630+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2543031279 0 0) 0x7efe0c003520 con 0x7efe20159950 2023-10-16T14:10:31.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 msgr2=0x7efe2015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:31.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 0x7efe2015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 >> v2:172.21.15.144:3300/0 conn(0x7efe2006c100 msgr2=0x7efe2006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:31.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 --2- 172.21.15.60:0/1096441657 >> v2:172.21.15.144:3300/0 conn(0x7efe2006c100 0x7efe2006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2015dbf0 con 0x7efe20159950 2023-10-16T14:10:31.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efe1c001800 con 0x7efe20159950 2023-10-16T14:10:31.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 >> v1:172.21.15.60:6789/0 conn(0x7efe20159950 legacy=0x7efe20070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 0x7efe0c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003f40 0x7efe0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c004d00 0x7efe0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2015e490 con 0x7efe0c004d00 2023-10-16T14:10:31.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2015d910 con 0x7efe0c003f40 2023-10-16T14:10:31.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00ba30 con 0x7efe20070b60 2023-10-16T14:10:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe25f86640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c004d00 0x7efe0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe25785640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 0x7efe0c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe24f84640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003f40 0x7efe0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe25f86640 1 -- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 msgr2=0x7efe0c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:31.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.631+0000 7efe25f86640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe20070b60 0x7efe0c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.632+0000 7efe25f86640 1 -- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003f40 msgr2=0x7efe0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:31.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.632+0000 7efe25f86640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003f40 0x7efe0c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.632+0000 7efe25f86640 1 -- 172.21.15.60:0/1096441657 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c003520 con 0x7efe0c004d00 2023-10-16T14:10:31.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.632+0000 7efe24f84640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003f40 0x7efe0c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:31.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe25f86640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c004d00 0x7efe0c005180 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7efe18009fc0 tx=0x7efe0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19f44f878582ec8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:31.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1802cc60 con 0x7efe0c004d00 2023-10-16T14:10:31.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1800f040 con 0x7efe0c004d00 2023-10-16T14:10:31.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe17fff640 1 -- 172.21.15.60:0/1096441657 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18034440 con 0x7efe0c004d00 2023-10-16T14:10:31.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe26787640 1 -- 172.21.15.60:0/1096441657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c004d00 msgr2=0x7efe0c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe26787640 1 --2- 172.21.15.60:0/1096441657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c004d00 0x7efe0c005180 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7efe18009fc0 tx=0x7efe0c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.633+0000 7efe25f86640 1 -- 172.21.15.60:0/1096441657 reap_dead start 2023-10-16T14:10:31.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.634+0000 7efe26787640 1 -- 172.21.15.60:0/1096441657 shutdown_connections 2023-10-16T14:10:31.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.634+0000 7efe26787640 1 -- 172.21.15.60:0/1096441657 >> 172.21.15.60:0/1096441657 conn(0x7efe200ce6a0 msgr2=0x7efe200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:31.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.634+0000 7efe26787640 1 -- 172.21.15.60:0/1096441657 shutdown_connections 2023-10-16T14:10:31.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.634+0000 7efe26787640 1 -- 172.21.15.60:0/1096441657 wait complete. 2023-10-16T14:10:31.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.636+0000 7efe26787640 1 Processor -- start 2023-10-16T14:10:31.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.636+0000 7efe26787640 1 -- start start 2023-10-16T14:10:31.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe26787640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe0c004d00 0x7efe20157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe26787640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe20158150 0x7efe2014f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe26787640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 0x7efe200c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe26787640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe20159050 con 0x7efe2014ff10 2023-10-16T14:10:31.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe26787640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe20150390 con 0x7efe20158150 2023-10-16T14:10:31.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe26787640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe20158720 con 0x7efe0c004d00 2023-10-16T14:10:31.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe25f86640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 0x7efe200c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe25f86640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 0x7efe200c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45090/0 (socket says 172.21.15.60:45090) 2023-10-16T14:10:31.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe25f86640 1 -- 172.21.15.60:0/1466026796 learned_addr learned my addr 172.21.15.60:0/1466026796 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:31.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe25785640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe0c004d00 0x7efe20157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.637+0000 7efe24f84640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe20158150 0x7efe2014f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe25f86640 1 -- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe0c004d00 msgr2=0x7efe20157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe25f86640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe0c004d00 0x7efe20157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe25f86640 1 -- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe20158150 msgr2=0x7efe2014f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe25f86640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe20158150 0x7efe2014f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe25f86640 1 -- 172.21.15.60:0/1466026796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20158a00 con 0x7efe2014ff10 2023-10-16T14:10:31.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe24f84640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe20158150 0x7efe2014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:31.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe25f86640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 0x7efe200c6220 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7efe18037040 tx=0x7efe18009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:31.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18034d50 con 0x7efe2014ff10 2023-10-16T14:10:31.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe18003990 con 0x7efe2014ff10 2023-10-16T14:10:31.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18003d40 con 0x7efe2014ff10 2023-10-16T14:10:31.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe20158ce0 con 0x7efe2014ff10 2023-10-16T14:10:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.638+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe200c6760 con 0x7efe2014ff10 2023-10-16T14:10:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.640+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe20159190 con 0x7efe2014ff10 2023-10-16T14:10:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.641+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe18004030 con 0x7efe2014ff10 2023-10-16T14:10:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.642+0000 7efe15ffb640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efe0406af90 0x7efe0406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.642+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe180aae60 con 0x7efe2014ff10 2023-10-16T14:10:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.642+0000 7efe25785640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efe0406af90 0x7efe0406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.643+0000 7efe25785640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efe0406af90 0x7efe0406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efe1c001930 tx=0x7efe1c009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.646+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe18075eb0 con 0x7efe2014ff10 2023-10-16T14:10:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.840+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe2015e1b0 con 0x7efe0406af90 2023-10-16T14:10:31.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.864+0000 7efe15ffb640 1 -- 172.21.15.60:0/1466026796 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286783 (secure 0 0 0) 0x7efe1c00d050 con 0x7efe0406af90 2023-10-16T14:10:31.869 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:31.873 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:31.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efe0406af90 msgr2=0x7efe0406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efe0406af90 0x7efe0406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efe1c001930 tx=0x7efe1c009040 comp rx=0 tx=0).stop 2023-10-16T14:10:31.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 msgr2=0x7efe200c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:31.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 0x7efe200c6220 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7efe18037040 tx=0x7efe18009750 comp rx=0 tx=0).stop 2023-10-16T14:10:31.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 shutdown_connections 2023-10-16T14:10:31.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efe0406af90 0x7efe0406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe2014ff10 0x7efe200c6220 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe20158150 0x7efe2014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 --2- 172.21.15.60:0/1466026796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe0c004d00 0x7efe20157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:31.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 >> 172.21.15.60:0/1466026796 conn(0x7efe200ce6a0 msgr2=0x7efe20070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:31.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 shutdown_connections 2023-10-16T14:10:31.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:31.872+0000 7efe26787640 1 -- 172.21.15.60:0/1466026796 wait complete. 2023-10-16T14:10:31.898 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T14:10:30.177283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":627,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377384,"kb_used_data":23780,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754597336,"statfs":{"total":773094113280,"available":772707672064,"internally_reserved":0,"allocated":24350720,"data_stored":12442048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.401852"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761024+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:28.761024+0000","last_peered":"2023-10-16T14:10:28.761024+0000","last_clean":"2023-10-16T14:10:28.761024+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:28.761024+0000","last_undegraded":"2023-10-16T14:10:28.761024+0000","last_fullsized":"2023-10-16T14:10:28.761024+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759603+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:28.759603+0000","last_peered":"2023-10-16T14:10:28.759603+0000","last_clean":"2023-10-16T14:10:28.759603+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:28.759603+0000","last_undegraded":"2023-10-16T14:10:28.759603+0000","last_fullsized":"2023-10-16T14:10:28.759603+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845460+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:27.845460+0000","last_peered":"2023-10-16T14:10:27.845460+0000","last_clean":"2023-10-16T14:10:27.845460+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:27.845460+0000","last_undegraded":"2023-10-16T14:10:27.845460+0000","last_fullsized":"2023-10-16T14:10:27.845460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851231+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:27.851231+0000","last_peered":"2023-10-16T14:10:27.851231+0000","last_clean":"2023-10-16T14:10:27.851231+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:27.851231+0000","last_undegraded":"2023-10-16T14:10:27.851231+0000","last_fullsized":"2023-10-16T14:10:27.851231+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759117+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:28.759117+0000","last_peered":"2023-10-16T14:10:28.759117+0000","last_clean":"2023-10-16T14:10:28.759117+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:28.759117+0000","last_undegraded":"2023-10-16T14:10:28.759117+0000","last_fullsized":"2023-10-16T14:10:28.759117+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759037+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:28.759037+0000","last_peered":"2023-10-16T14:10:28.759037+0000","last_clean":"2023-10-16T14:10:28.759037+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:28.759037+0000","last_undegraded":"2023-10-16T14:10:28.759037+0000","last_fullsized":"2023-10-16T14:10:28.759037+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760164+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:28.760164+0000","last_peered":"2023-10-16T14:10:28.760164+0000","last_clean":"2023-10-16T14:10:28.760164+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:28.760164+0000","last_undegraded":"2023-10-16T14:10:28.760164+0000","last_fullsized":"2023-10-16T14:10:28.760164+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758409+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:28.758409+0000","last_peered":"2023-10-16T14:10:28.758409+0000","last_clean":"2023-10-16T14:10:28.758409+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:28.758409+0000","last_undegraded":"2023-10-16T14:10:28.758409+0000","last_fullsized":"2023-10-16T14:10:28.758409+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759995+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:28.759995+0000","last_peered":"2023-10-16T14:10:28.759995+0000","last_clean":"2023-10-16T14:10:28.759995+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:28.759995+0000","last_undegraded":"2023-10-16T14:10:28.759995+0000","last_fullsized":"2023-10-16T14:10:28.759995+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380768+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:28.380768+0000","last_peered":"2023-10-16T14:10:28.380768+0000","last_clean":"2023-10-16T14:10:28.380768+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:28.380768+0000","last_undegraded":"2023-10-16T14:10:28.380768+0000","last_fullsized":"2023-10-16T14:10:28.380768+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845678+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:27.845678+0000","last_peered":"2023-10-16T14:10:27.845678+0000","last_clean":"2023-10-16T14:10:27.845678+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:27.845678+0000","last_undegraded":"2023-10-16T14:10:27.845678+0000","last_fullsized":"2023-10-16T14:10:27.845678+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849448+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:27.849448+0000","last_peered":"2023-10-16T14:10:27.849448+0000","last_clean":"2023-10-16T14:10:27.849448+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:27.849448+0000","last_undegraded":"2023-10-16T14:10:27.849448+0000","last_fullsized":"2023-10-16T14:10:27.849448+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850229+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:27.850229+0000","last_peered":"2023-10-16T14:10:27.850229+0000","last_clean":"2023-10-16T14:10:27.850229+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:27.850229+0000","last_undegraded":"2023-10-16T14:10:27.850229+0000","last_fullsized":"2023-10-16T14:10:27.850229+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760487+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:28.760487+0000","last_peered":"2023-10-16T14:10:28.760487+0000","last_clean":"2023-10-16T14:10:28.760487+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:28.760487+0000","last_undegraded":"2023-10-16T14:10:28.760487+0000","last_fullsized":"2023-10-16T14:10:28.760487+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760283+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:28.760283+0000","last_peered":"2023-10-16T14:10:28.760283+0000","last_clean":"2023-10-16T14:10:28.760283+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:28.760283+0000","last_undegraded":"2023-10-16T14:10:28.760283+0000","last_fullsized":"2023-10-16T14:10:28.760283+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380111+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:28.380111+0000","last_peered":"2023-10-16T14:10:28.380111+0000","last_clean":"2023-10-16T14:10:28.380111+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:28.380111+0000","last_undegraded":"2023-10-16T14:10:28.380111+0000","last_fullsized":"2023-10-16T14:10:28.380111+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761479+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:28.761479+0000","last_peered":"2023-10-16T14:10:28.761479+0000","last_clean":"2023-10-16T14:10:28.761479+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:28.761479+0000","last_undegraded":"2023-10-16T14:10:28.761479+0000","last_fullsized":"2023-10-16T14:10:28.761479+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381002+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:28.381002+0000","last_peered":"2023-10-16T14:10:28.381002+0000","last_clean":"2023-10-16T14:10:28.381002+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:28.381002+0000","last_undegraded":"2023-10-16T14:10:28.381002+0000","last_fullsized":"2023-10-16T14:10:28.381002+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381139+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:28.381139+0000","last_peered":"2023-10-16T14:10:28.381139+0000","last_clean":"2023-10-16T14:10:28.381139+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:28.381139+0000","last_undegraded":"2023-10-16T14:10:28.381139+0000","last_fullsized":"2023-10-16T14:10:28.381139+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759538+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:28.759538+0000","last_peered":"2023-10-16T14:10:28.759538+0000","last_clean":"2023-10-16T14:10:28.759538+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:28.759538+0000","last_undegraded":"2023-10-16T14:10:28.759538+0000","last_fullsized":"2023-10-16T14:10:28.759538+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850789+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:27.850789+0000","last_peered":"2023-10-16T14:10:27.850789+0000","last_clean":"2023-10-16T14:10:27.850789+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:27.850789+0000","last_undegraded":"2023-10-16T14:10:27.850789+0000","last_fullsized":"2023-10-16T14:10:27.850789+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637693+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:28.637693+0000","last_peered":"2023-10-16T14:10:28.637693+0000","last_clean":"2023-10-16T14:10:28.637693+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:28.637693+0000","last_undegraded":"2023-10-16T14:10:28.637693+0000","last_fullsized":"2023-10-16T14:10:28.637693+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637416+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:28.637416+0000","last_peered":"2023-10-16T14:10:28.637416+0000","last_clean":"2023-10-16T14:10:28.637416+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:28.637416+0000","last_undegraded":"2023-10-16T14:10:28.637416+0000","last_fullsized":"2023-10-16T14:10:28.637416+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.848305+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:27.848305+0000","last_peered":"2023-10-16T14:10:27.848305+0000","last_clean":"2023-10-16T14:10:27.848305+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:27.848305+0000","last_undegraded":"2023-10-16T14:10:27.848305+0000","last_fullsized":"2023-10-16T14:10:27.848305+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759688+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:28.759688+0000","last_peered":"2023-10-16T14:10:28.759688+0000","last_clean":"2023-10-16T14:10:28.759688+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:28.759688+0000","last_undegraded":"2023-10-16T14:10:28.759688+0000","last_fullsized":"2023-10-16T14:10:28.759688+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761149+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:28.761149+0000","last_peered":"2023-10-16T14:10:28.761149+0000","last_clean":"2023-10-16T14:10:28.761149+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:28.761149+0000","last_undegraded":"2023-10-16T14:10:28.761149+0000","last_fullsized":"2023-10-16T14:10:28.761149+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849296+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:27.849296+0000","last_peered":"2023-10-16T14:10:27.849296+0000","last_clean":"2023-10-16T14:10:27.849296+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:27.849296+0000","last_undegraded":"2023-10-16T14:10:27.849296+0000","last_fullsized":"2023-10-16T14:10:27.849296+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845100+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:27.845100+0000","last_peered":"2023-10-16T14:10:27.845100+0000","last_clean":"2023-10-16T14:10:27.845100+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:27.845100+0000","last_undegraded":"2023-10-16T14:10:27.845100+0000","last_fullsized":"2023-10-16T14:10:27.845100+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849906+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:27.849906+0000","last_peered":"2023-10-16T14:10:27.849906+0000","last_clean":"2023-10-16T14:10:27.849906+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:27.849906+0000","last_undegraded":"2023-10-16T14:10:27.849906+0000","last_fullsized":"2023-10-16T14:10:27.849906+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759089+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:28.759089+0000","last_peered":"2023-10-16T14:10:28.759089+0000","last_clean":"2023-10-16T14:10:28.759089+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:28.759089+0000","last_undegraded":"2023-10-16T14:10:28.759089+0000","last_fullsized":"2023-10-16T14:10:28.759089+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380100+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:28.380100+0000","last_peered":"2023-10-16T14:10:28.380100+0000","last_clean":"2023-10-16T14:10:28.380100+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:28.380100+0000","last_undegraded":"2023-10-16T14:10:28.380100+0000","last_fullsized":"2023-10-16T14:10:28.380100+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759401+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:28.759401+0000","last_peered":"2023-10-16T14:10:28.759401+0000","last_clean":"2023-10-16T14:10:28.759401+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:28.759401+0000","last_undegraded":"2023-10-16T14:10:28.759401+0000","last_fullsized":"2023-10-16T14:10:28.759401+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759519+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:28.759519+0000","last_peered":"2023-10-16T14:10:28.759519+0000","last_clean":"2023-10-16T14:10:28.759519+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:28.759519+0000","last_undegraded":"2023-10-16T14:10:28.759519+0000","last_fullsized":"2023-10-16T14:10:28.759519+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":16,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849655+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:27.849655+0000","last_peered":"2023-10-16T14:10:27.849655+0000","last_clean":"2023-10-16T14:10:27.849655+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:27.849655+0000","last_undegraded":"2023-10-16T14:10:27.849655+0000","last_fullsized":"2023-10-16T14:10:27.849655+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637967+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:28.637967+0000","last_peered":"2023-10-16T14:10:28.637967+0000","last_clean":"2023-10-16T14:10:28.637967+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:28.637967+0000","last_undegraded":"2023-10-16T14:10:28.637967+0000","last_fullsized":"2023-10-16T14:10:28.637967+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759844+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:28.759844+0000","last_peered":"2023-10-16T14:10:28.759844+0000","last_clean":"2023-10-16T14:10:28.759844+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:28.759844+0000","last_undegraded":"2023-10-16T14:10:28.759844+0000","last_fullsized":"2023-10-16T14:10:28.759844+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761520+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:28.761520+0000","last_peered":"2023-10-16T14:10:28.761520+0000","last_clean":"2023-10-16T14:10:28.761520+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:28.761520+0000","last_undegraded":"2023-10-16T14:10:28.761520+0000","last_fullsized":"2023-10-16T14:10:28.761520+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845645+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:27.845645+0000","last_peered":"2023-10-16T14:10:27.845645+0000","last_clean":"2023-10-16T14:10:27.845645+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:27.845645+0000","last_undegraded":"2023-10-16T14:10:27.845645+0000","last_fullsized":"2023-10-16T14:10:27.845645+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381105+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:28.381105+0000","last_peered":"2023-10-16T14:10:28.381105+0000","last_clean":"2023-10-16T14:10:28.381105+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:28.381105+0000","last_undegraded":"2023-10-16T14:10:28.381105+0000","last_fullsized":"2023-10-16T14:10:28.381105+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851635+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:27.851635+0000","last_peered":"2023-10-16T14:10:27.851635+0000","last_clean":"2023-10-16T14:10:27.851635+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:27.851635+0000","last_undegraded":"2023-10-16T14:10:27.851635+0000","last_fullsized":"2023-10-16T14:10:27.851635+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851136+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:27.851136+0000","last_peered":"2023-10-16T14:10:27.851136+0000","last_clean":"2023-10-16T14:10:27.851136+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:27.851136+0000","last_undegraded":"2023-10-16T14:10:27.851136+0000","last_fullsized":"2023-10-16T14:10:27.851136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759683+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:28.759683+0000","last_peered":"2023-10-16T14:10:28.759683+0000","last_clean":"2023-10-16T14:10:28.759683+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:28.759683+0000","last_undegraded":"2023-10-16T14:10:28.759683+0000","last_fullsized":"2023-10-16T14:10:28.759683+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758605+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:28.758605+0000","last_peered":"2023-10-16T14:10:28.758605+0000","last_clean":"2023-10-16T14:10:28.758605+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:28.758605+0000","last_undegraded":"2023-10-16T14:10:28.758605+0000","last_fullsized":"2023-10-16T14:10:28.758605+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758200+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:28.758200+0000","last_peered":"2023-10-16T14:10:28.758200+0000","last_clean":"2023-10-16T14:10:28.758200+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:28.758200+0000","last_undegraded":"2023-10-16T14:10:28.758200+0000","last_fullsized":"2023-10-16T14:10:28.758200+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849860+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:27.849860+0000","last_peered":"2023-10-16T14:10:27.849860+0000","last_clean":"2023-10-16T14:10:27.849860+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:27.849860+0000","last_undegraded":"2023-10-16T14:10:27.849860+0000","last_fullsized":"2023-10-16T14:10:27.849860+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637029+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:28.637029+0000","last_peered":"2023-10-16T14:10:28.637029+0000","last_clean":"2023-10-16T14:10:28.637029+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:28.637029+0000","last_undegraded":"2023-10-16T14:10:28.637029+0000","last_fullsized":"2023-10-16T14:10:28.637029+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.852429+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:27.852429+0000","last_peered":"2023-10-16T14:10:27.852429+0000","last_clean":"2023-10-16T14:10:27.852429+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:27.852429+0000","last_undegraded":"2023-10-16T14:10:27.852429+0000","last_fullsized":"2023-10-16T14:10:27.852429+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759375+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:28.759375+0000","last_peered":"2023-10-16T14:10:28.759375+0000","last_clean":"2023-10-16T14:10:28.759375+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:28.759375+0000","last_undegraded":"2023-10-16T14:10:28.759375+0000","last_fullsized":"2023-10-16T14:10:28.759375+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845837+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:27.845837+0000","last_peered":"2023-10-16T14:10:27.845837+0000","last_clean":"2023-10-16T14:10:27.845837+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:27.845837+0000","last_undegraded":"2023-10-16T14:10:27.845837+0000","last_fullsized":"2023-10-16T14:10:27.845837+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.638017+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:28.638017+0000","last_peered":"2023-10-16T14:10:28.638017+0000","last_clean":"2023-10-16T14:10:28.638017+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:28.638017+0000","last_undegraded":"2023-10-16T14:10:28.638017+0000","last_fullsized":"2023-10-16T14:10:28.638017+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851609+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:27.851609+0000","last_peered":"2023-10-16T14:10:27.851609+0000","last_clean":"2023-10-16T14:10:27.851609+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:27.851609+0000","last_undegraded":"2023-10-16T14:10:27.851609+0000","last_fullsized":"2023-10-16T14:10:27.851609+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381292+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:28.381292+0000","last_peered":"2023-10-16T14:10:28.381292+0000","last_clean":"2023-10-16T14:10:28.381292+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:28.381292+0000","last_undegraded":"2023-10-16T14:10:28.381292+0000","last_fullsized":"2023-10-16T14:10:28.381292+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761145+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:28.761145+0000","last_peered":"2023-10-16T14:10:28.761145+0000","last_clean":"2023-10-16T14:10:28.761145+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:28.761145+0000","last_undegraded":"2023-10-16T14:10:28.761145+0000","last_fullsized":"2023-10-16T14:10:28.761145+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"230'6","reported_seq":18,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851860+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:27.851860+0000","last_peered":"2023-10-16T14:10:27.851860+0000","last_clean":"2023-10-16T14:10:27.851860+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:27.851860+0000","last_undegraded":"2023-10-16T14:10:27.851860+0000","last_fullsized":"2023-10-16T14:10:27.851860+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759578+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:28.759578+0000","last_peered":"2023-10-16T14:10:28.759578+0000","last_clean":"2023-10-16T14:10:28.759578+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:28.759578+0000","last_undegraded":"2023-10-16T14:10:28.759578+0000","last_fullsized":"2023-10-16T14:10:28.759578+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637467+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:28.637467+0000","last_peered":"2023-10-16T14:10:28.637467+0000","last_clean":"2023-10-16T14:10:28.637467+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:28.637467+0000","last_undegraded":"2023-10-16T14:10:28.637467+0000","last_fullsized":"2023-10-16T14:10:28.637467+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380848+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:28.380848+0000","last_peered":"2023-10-16T14:10:28.380848+0000","last_clean":"2023-10-16T14:10:28.380848+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:28.380848+0000","last_undegraded":"2023-10-16T14:10:28.380848+0000","last_fullsized":"2023-10-16T14:10:28.380848+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637980+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:28.637980+0000","last_peered":"2023-10-16T14:10:28.637980+0000","last_clean":"2023-10-16T14:10:28.637980+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:28.637980+0000","last_undegraded":"2023-10-16T14:10:28.637980+0000","last_fullsized":"2023-10-16T14:10:28.637980+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845073+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:27.845073+0000","last_peered":"2023-10-16T14:10:27.845073+0000","last_clean":"2023-10-16T14:10:27.845073+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:27.845073+0000","last_undegraded":"2023-10-16T14:10:27.845073+0000","last_fullsized":"2023-10-16T14:10:27.845073+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380567+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:28.380567+0000","last_peered":"2023-10-16T14:10:28.380567+0000","last_clean":"2023-10-16T14:10:28.380567+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:28.380567+0000","last_undegraded":"2023-10-16T14:10:28.380567+0000","last_fullsized":"2023-10-16T14:10:28.380567+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760622+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:28.760622+0000","last_peered":"2023-10-16T14:10:28.760622+0000","last_clean":"2023-10-16T14:10:28.760622+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:28.760622+0000","last_undegraded":"2023-10-16T14:10:28.760622+0000","last_fullsized":"2023-10-16T14:10:28.760622+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":20,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845181+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:27.845181+0000","last_peered":"2023-10-16T14:10:27.845181+0000","last_clean":"2023-10-16T14:10:27.845181+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:27.845181+0000","last_undegraded":"2023-10-16T14:10:27.845181+0000","last_fullsized":"2023-10-16T14:10:27.845181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759751+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:28.759751+0000","last_peered":"2023-10-16T14:10:28.759751+0000","last_clean":"2023-10-16T14:10:28.759751+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:28.759751+0000","last_undegraded":"2023-10-16T14:10:28.759751+0000","last_fullsized":"2023-10-16T14:10:28.759751+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759630+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:28.759630+0000","last_peered":"2023-10-16T14:10:28.759630+0000","last_clean":"2023-10-16T14:10:28.759630+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:28.759630+0000","last_undegraded":"2023-10-16T14:10:28.759630+0000","last_fullsized":"2023-10-16T14:10:28.759630+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759678+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:28.759678+0000","last_peered":"2023-10-16T14:10:28.759678+0000","last_clean":"2023-10-16T14:10:28.759678+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:28.759678+0000","last_undegraded":"2023-10-16T14:10:28.759678+0000","last_fullsized":"2023-10-16T14:10:28.759678+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759892+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:28.759892+0000","last_peered":"2023-10-16T14:10:28.759892+0000","last_clean":"2023-10-16T14:10:28.759892+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:28.759892+0000","last_undegraded":"2023-10-16T14:10:28.759892+0000","last_fullsized":"2023-10-16T14:10:28.759892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845385+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:27.845385+0000","last_peered":"2023-10-16T14:10:27.845385+0000","last_clean":"2023-10-16T14:10:27.845385+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:27.845385+0000","last_undegraded":"2023-10-16T14:10:27.845385+0000","last_fullsized":"2023-10-16T14:10:27.845385+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.381352+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:28.381352+0000","last_peered":"2023-10-16T14:10:28.381352+0000","last_clean":"2023-10-16T14:10:28.381352+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:28.381352+0000","last_undegraded":"2023-10-16T14:10:28.381352+0000","last_fullsized":"2023-10-16T14:10:28.381352+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380945+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:28.380945+0000","last_peered":"2023-10-16T14:10:28.380945+0000","last_clean":"2023-10-16T14:10:28.380945+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:28.380945+0000","last_undegraded":"2023-10-16T14:10:28.380945+0000","last_fullsized":"2023-10-16T14:10:28.380945+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845364+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:27.845364+0000","last_peered":"2023-10-16T14:10:27.845364+0000","last_clean":"2023-10-16T14:10:27.845364+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:27.845364+0000","last_undegraded":"2023-10-16T14:10:27.845364+0000","last_fullsized":"2023-10-16T14:10:27.845364+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380483+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:28.380483+0000","last_peered":"2023-10-16T14:10:28.380483+0000","last_clean":"2023-10-16T14:10:28.380483+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:28.380483+0000","last_undegraded":"2023-10-16T14:10:28.380483+0000","last_fullsized":"2023-10-16T14:10:28.380483+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.380860+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:28.380860+0000","last_peered":"2023-10-16T14:10:28.380860+0000","last_clean":"2023-10-16T14:10:28.380860+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:28.380860+0000","last_undegraded":"2023-10-16T14:10:28.380860+0000","last_fullsized":"2023-10-16T14:10:28.380860+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845100+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:27.845100+0000","last_peered":"2023-10-16T14:10:27.845100+0000","last_clean":"2023-10-16T14:10:27.845100+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:27.845100+0000","last_undegraded":"2023-10-16T14:10:27.845100+0000","last_fullsized":"2023-10-16T14:10:27.845100+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761155+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:28.761155+0000","last_peered":"2023-10-16T14:10:28.761155+0000","last_clean":"2023-10-16T14:10:28.761155+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:28.761155+0000","last_undegraded":"2023-10-16T14:10:28.761155+0000","last_fullsized":"2023-10-16T14:10:28.761155+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758860+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:28.758860+0000","last_peered":"2023-10-16T14:10:28.758860+0000","last_clean":"2023-10-16T14:10:28.758860+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:28.758860+0000","last_undegraded":"2023-10-16T14:10:28.758860+0000","last_fullsized":"2023-10-16T14:10:28.758860+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760527+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:28.760527+0000","last_peered":"2023-10-16T14:10:28.760527+0000","last_clean":"2023-10-16T14:10:28.760527+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:28.760527+0000","last_undegraded":"2023-10-16T14:10:28.760527+0000","last_fullsized":"2023-10-16T14:10:28.760527+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637480+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:28.637480+0000","last_peered":"2023-10-16T14:10:28.637480+0000","last_clean":"2023-10-16T14:10:28.637480+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:28.637480+0000","last_undegraded":"2023-10-16T14:10:28.637480+0000","last_fullsized":"2023-10-16T14:10:28.637480+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758656+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:28.758656+0000","last_peered":"2023-10-16T14:10:28.758656+0000","last_clean":"2023-10-16T14:10:28.758656+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:28.758656+0000","last_undegraded":"2023-10-16T14:10:28.758656+0000","last_fullsized":"2023-10-16T14:10:28.758656+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849333+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:27.849333+0000","last_peered":"2023-10-16T14:10:27.849333+0000","last_clean":"2023-10-16T14:10:27.849333+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:27.849333+0000","last_undegraded":"2023-10-16T14:10:27.849333+0000","last_fullsized":"2023-10-16T14:10:27.849333+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845797+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:27.845797+0000","last_peered":"2023-10-16T14:10:27.845797+0000","last_clean":"2023-10-16T14:10:27.845797+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:27.845797+0000","last_undegraded":"2023-10-16T14:10:27.845797+0000","last_fullsized":"2023-10-16T14:10:27.845797+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.761448+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:28.761448+0000","last_peered":"2023-10-16T14:10:28.761448+0000","last_clean":"2023-10-16T14:10:28.761448+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:28.761448+0000","last_undegraded":"2023-10-16T14:10:28.761448+0000","last_fullsized":"2023-10-16T14:10:28.761448+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.852382+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:27.852382+0000","last_peered":"2023-10-16T14:10:27.852382+0000","last_clean":"2023-10-16T14:10:27.852382+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:27.852382+0000","last_undegraded":"2023-10-16T14:10:27.852382+0000","last_fullsized":"2023-10-16T14:10:27.852382+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759300+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:28.759300+0000","last_peered":"2023-10-16T14:10:28.759300+0000","last_clean":"2023-10-16T14:10:28.759300+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:28.759300+0000","last_undegraded":"2023-10-16T14:10:28.759300+0000","last_fullsized":"2023-10-16T14:10:28.759300+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759379+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:28.759379+0000","last_peered":"2023-10-16T14:10:28.759379+0000","last_clean":"2023-10-16T14:10:28.759379+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:28.759379+0000","last_undegraded":"2023-10-16T14:10:28.759379+0000","last_fullsized":"2023-10-16T14:10:28.759379+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637873+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:28.637873+0000","last_peered":"2023-10-16T14:10:28.637873+0000","last_clean":"2023-10-16T14:10:28.637873+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:28.637873+0000","last_undegraded":"2023-10-16T14:10:28.637873+0000","last_fullsized":"2023-10-16T14:10:28.637873+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845491+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:27.845491+0000","last_peered":"2023-10-16T14:10:27.845491+0000","last_clean":"2023-10-16T14:10:27.845491+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:27.845491+0000","last_undegraded":"2023-10-16T14:10:27.845491+0000","last_fullsized":"2023-10-16T14:10:27.845491+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760859+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:28.760859+0000","last_peered":"2023-10-16T14:10:28.760859+0000","last_clean":"2023-10-16T14:10:28.760859+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:28.760859+0000","last_undegraded":"2023-10-16T14:10:28.760859+0000","last_fullsized":"2023-10-16T14:10:28.760859+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.851349+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:27.851349+0000","last_peered":"2023-10-16T14:10:27.851349+0000","last_clean":"2023-10-16T14:10:27.851349+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:27.851349+0000","last_undegraded":"2023-10-16T14:10:27.851349+0000","last_fullsized":"2023-10-16T14:10:27.851349+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850071+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:27.850071+0000","last_peered":"2023-10-16T14:10:27.850071+0000","last_clean":"2023-10-16T14:10:27.850071+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:27.850071+0000","last_undegraded":"2023-10-16T14:10:27.850071+0000","last_fullsized":"2023-10-16T14:10:27.850071+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.759811+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:28.759811+0000","last_peered":"2023-10-16T14:10:28.759811+0000","last_clean":"2023-10-16T14:10:28.759811+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:28.759811+0000","last_undegraded":"2023-10-16T14:10:28.759811+0000","last_fullsized":"2023-10-16T14:10:28.759811+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.845021+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:27.845021+0000","last_peered":"2023-10-16T14:10:27.845021+0000","last_clean":"2023-10-16T14:10:27.845021+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:27.845021+0000","last_undegraded":"2023-10-16T14:10:27.845021+0000","last_fullsized":"2023-10-16T14:10:27.845021+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.637746+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:28.637746+0000","last_peered":"2023-10-16T14:10:28.637746+0000","last_clean":"2023-10-16T14:10:28.637746+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:28.637746+0000","last_undegraded":"2023-10-16T14:10:28.637746+0000","last_fullsized":"2023-10-16T14:10:28.637746+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.758964+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:28.758964+0000","last_peered":"2023-10-16T14:10:28.758964+0000","last_clean":"2023-10-16T14:10:28.758964+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:28.758964+0000","last_undegraded":"2023-10-16T14:10:28.758964+0000","last_fullsized":"2023-10-16T14:10:28.758964+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.849034+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:27.849034+0000","last_peered":"2023-10-16T14:10:27.849034+0000","last_clean":"2023-10-16T14:10:27.849034+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:27.849034+0000","last_undegraded":"2023-10-16T14:10:27.849034+0000","last_fullsized":"2023-10-16T14:10:27.849034+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:28.760640+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:28.760640+0000","last_peered":"2023-10-16T14:10:28.760640+0000","last_clean":"2023-10-16T14:10:28.760640+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:28.760640+0000","last_undegraded":"2023-10-16T14:10:28.760640+0000","last_fullsized":"2023-10-16T14:10:28.760640+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:27.850344+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:27.850344+0000","last_peered":"2023-10-16T14:10:27.850344+0000","last_clean":"2023-10-16T14:10:27.850344+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:27.850344+0000","last_undegraded":"2023-10-16T14:10:27.850344+0000","last_fullsized":"2023-10-16T14:10:27.850344+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1222,"reported_epoch":230,"state":"active+clean","last_fresh":"2023-10-16T14:10:26.817840+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:26.817840+0000","last_peered":"2023-10-16T14:10:26.817840+0000","last_clean":"2023-10-16T14:10:26.817840+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:26.817840+0000","last_undegraded":"2023-10-16T14:10:26.817840+0000","last_fullsized":"2023-10-16T14:10:26.817840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":3497984,"data_stored":2001404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":1.575}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.98699999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.482},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.4239999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.7969999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.133},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.534}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.7709999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.944},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.8939999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543299,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42336,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329504,"statfs":{"total":96636764160,"available":96593412096,"internally_reserved":0,"allocated":2895872,"data_stored":1410006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":0.998},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":0.77200000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.099}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.0580000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.246},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.3959999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.609}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.4590000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.1990000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543300,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2883584,"data_stored":1403706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0900000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.79400000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.02}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.1379999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.3220000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.631}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.482}]}]},{"osd":4,"up_from":14,"seq":60129543300,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47904,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323936,"statfs":{"total":96636764160,"available":96587710464,"internally_reserved":0,"allocated":2899968,"data_stored":1404840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.21,"15min":1.2210000000000001},"min":{"1min":0.81299999999999994,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.679,"5min":1.7829999999999999,"15min":1.9039999999999999},"last":1.762},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.179,"15min":1.2},"min":{"1min":0.92900000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.75,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.26,"15min":1.254},"min":{"1min":0.89700000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.516,"5min":1.7090000000000001,"15min":1.78},"last":1.282},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.93000000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.585,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.077}]},{"osd":2,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.327,"15min":1.321},"min":{"1min":0.94299999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.6850000000000001,"5min":1.865,"15min":1.865},"last":1.714},{"interface":"front","average":{"1min":1.367,"5min":1.363,"15min":1.383},"min":{"1min":0.91800000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.796,"5min":1.796,"15min":2.0110000000000001},"last":1.4550000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.294,"15min":1.3109999999999999},"min":{"1min":1.0549999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.772,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.25},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4379999999999999,"15min":1.4179999999999999},"min":{"1min":0.92800000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.7030000000000001,"5min":1.8759999999999999,"15min":1.919},"last":1.381}]},{"osd":5,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.337},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":1.7170000000000001,"5min":1.827,"15min":1.889},"last":1.329},{"interface":"front","average":{"1min":1.425,"5min":1.359,"15min":1.339},"min":{"1min":0.90900000000000003,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.921,"5min":1.921,"15min":1.9510000000000001},"last":1.0389999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.391,"15min":1.3899999999999999},"min":{"1min":1.02,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.865,"5min":1.9159999999999999,"15min":1.956},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.494,"15min":1.522},"min":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":0.94199999999999995},"max":{"1min":1.869,"5min":1.869,"15min":2.6429999999999998},"last":1.871}]},{"osd":7,"last update":"Mon Oct 16 14:09:31 2023","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.516,"15min":1.5009999999999999},"min":{"1min":1.2270000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.873,"5min":2.0049999999999999,"15min":2.1949999999999998},"last":1.137},{"interface":"front","average":{"1min":1.651,"5min":1.6459999999999999,"15min":1.6359999999999999},"min":{"1min":1.2170000000000001,"5min":1.008,"15min":1.008},"max":{"1min":2.1280000000000001,"5min":2.1349999999999998,"15min":2.3660000000000001},"last":1.6200000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543300,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2904064,"data_stored":1409606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":1.1599999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":1.456}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":1.089}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.77},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":1.6879999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.954},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":1.732}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.8200000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":1.395},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.9299999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52056,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319784,"statfs":{"total":96636764160,"available":96583458816,"internally_reserved":0,"allocated":2891776,"data_stored":1409424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.89000000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":1.5860000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.5},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.6299999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":1.726}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.389}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":1.524}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.849},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.6890000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543298,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52056,"kb_used_data":2820,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319784,"statfs":{"total":96636764160,"available":96583458816,"internally_reserved":0,"allocated":2887680,"data_stored":1403180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":0.83799999999999997}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":0.73899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543299,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1999882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.96399999999999997,"15min":0.94499999999999995},"min":{"1min":0.68400000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.704,"5min":1.704,"15min":1.891},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.119,"15min":1.1779999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.48699999999999999},"max":{"1min":1.6539999999999999,"5min":1.659,"15min":1.9299999999999999},"last":0.84799999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.177,"15min":1.1759999999999999},"min":{"1min":0.90600000000000003,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.617,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.234,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.60199999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.78,"5min":1.893,"15min":1.893},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.198,"15min":1.1699999999999999},"min":{"1min":0.63400000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7869999999999999,"5min":1.8520000000000001,"15min":1.8600000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3859999999999999,"15min":1.391},"min":{"1min":0.79300000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9630000000000001,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.88100000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.365,"15min":1.389},"min":{"1min":1.028,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":1.6870000000000001,"5min":1.819,"15min":2.048},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5629999999999999,"15min":1.53},"min":{"1min":1.254,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.1070000000000002,"5min":2.601,"15min":2.601},"last":0.99099999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5089999999999999,"15min":1.518},"min":{"1min":0.95999999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.867,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.48,"15min":1.4970000000000001},"min":{"1min":1.145,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.796,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.329}]},{"osd":6,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.641,"15min":1.6259999999999999},"min":{"1min":1.1060000000000001,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":2.2549999999999999,"5min":2.669,"15min":2.669},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.579,"15min":1.6100000000000001},"min":{"1min":1.0649999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.032,"5min":2.355,"15min":2.3769999999999998},"last":1.367}]},{"osd":7,"last update":"Mon Oct 16 14:09:29 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.694,"15min":1.722},"min":{"1min":1.181,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.008,"5min":2.484,"15min":2.5270000000000001},"last":1.419},{"interface":"front","average":{"1min":1.643,"5min":1.6539999999999999,"15min":1.657},"min":{"1min":1.1950000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.9350000000000001,"5min":2.2949999999999999,"15min":2.464},"last":1.1439999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:31.905 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:10:31.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:10:32.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.168+0000 7f0d8f59e640 1 Processor -- start 2023-10-16T14:10:32.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.168+0000 7f0d8f59e640 1 -- start start 2023-10-16T14:10:32.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.170+0000 7f0d8f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d900b4160 0x7f0d900cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.170+0000 7f0d8f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d900cc850 con 0x7f0d9015ac70 2023-10-16T14:10:32.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.170+0000 7f0d8f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d900ccb30 con 0x7f0d9006a4b0 2023-10-16T14:10:32.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.170+0000 7f0d8f59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d900cce10 con 0x7f0d900b4160 2023-10-16T14:10:32.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8e59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0d9015ac70 0x7f0d90069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45990/0 (socket says 172.21.15.60:45990) 2023-10-16T14:10:32.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8e59c640 1 -- 172.21.15.60:0/700415162 learned_addr learned my addr 172.21.15.60:0/700415162 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:32.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8ed9d640 1 --2- 172.21.15.60:0/700415162 >> v2:172.21.15.60:3300/0 conn(0x7f0d900b4160 0x7f0d900cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8dd9b640 1 -- 172.21.15.60:0/700415162 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334622840 0 0) 0x7f0d900cc850 con 0x7f0d9015ac70 2023-10-16T14:10:32.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8ed9d640 1 -- 172.21.15.60:0/700415162 >> v1:172.21.15.144:6789/0 conn(0x7f0d9006a4b0 legacy=0x7f0d9006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8ed9d640 1 -- 172.21.15.60:0/700415162 >> v1:172.21.15.60:6789/0 conn(0x7f0d9015ac70 legacy=0x7f0d90069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.171+0000 7f0d8ed9d640 1 -- 172.21.15.60:0/700415162 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900cd0f0 con 0x7f0d900b4160 2023-10-16T14:10:32.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.172+0000 7f0d8ed9d640 1 --2- 172.21.15.60:0/700415162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d900b4160 0x7f0d900cc270 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009d90 tx=0x7f0d8802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7c28b7d8ba4e1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:32.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8dd9b640 1 -- 172.21.15.60:0/700415162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8802cd00 con 0x7f0d900b4160 2023-10-16T14:10:32.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8dd9b640 1 -- 172.21.15.60:0/700415162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8802ce60 con 0x7f0d900b4160 2023-10-16T14:10:32.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8dd9b640 1 -- 172.21.15.60:0/700415162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88034660 con 0x7f0d900b4160 2023-10-16T14:10:32.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8f59e640 1 -- 172.21.15.60:0/700415162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d900b4160 msgr2=0x7f0d900cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/700415162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d900b4160 0x7f0d900cc270 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009d90 tx=0x7f0d8802bf50 comp rx=0 tx=0).stop 2023-10-16T14:10:32.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8f59e640 1 -- 172.21.15.60:0/700415162 shutdown_connections 2023-10-16T14:10:32.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/700415162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d900b4160 0x7f0d900cc270 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.173+0000 7f0d8f59e640 1 -- 172.21.15.60:0/700415162 >> 172.21.15.60:0/700415162 conn(0x7f0d900ce6a0 msgr2=0x7f0d900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:32.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.174+0000 7f0d8f59e640 1 -- 172.21.15.60:0/700415162 shutdown_connections 2023-10-16T14:10:32.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.174+0000 7f0d8f59e640 1 -- 172.21.15.60:0/700415162 wait complete. 2023-10-16T14:10:32.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.175+0000 7f0d8f59e640 1 Processor -- start 2023-10-16T14:10:32.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.175+0000 7f0d8f59e640 1 -- start start 2023-10-16T14:10:32.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.176+0000 7f0d8f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 0x7f0d901f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.176+0000 7f0d8f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d900b4160 0x7f0d900c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.176+0000 7f0d8f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d9015ac70 0x7f0d900c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.176+0000 7f0d8f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d901f9850 con 0x7f0d9006a4b0 2023-10-16T14:10:32.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.176+0000 7f0d8f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900c9740 con 0x7f0d9015ac70 2023-10-16T14:10:32.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.176+0000 7f0d8f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900c9a20 con 0x7f0d900b4160 2023-10-16T14:10:32.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 0x7f0d901f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 0x7f0d901f9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45106/0 (socket says 172.21.15.60:45106) 2023-10-16T14:10:32.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 -- 172.21.15.60:0/805885731 learned_addr learned my addr 172.21.15.60:0/805885731 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 -- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d900b4160 msgr2=0x7f0d900c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d900b4160 0x7f0d900c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 -- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d9015ac70 msgr2=0x7f0d900c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d9015ac70 0x7f0d900c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 -- 172.21.15.60:0/805885731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88009a70 con 0x7f0d9006a4b0 2023-10-16T14:10:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d8e59c640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 0x7f0d901f9310 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000c3f0 tx=0x7f0d8000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:32.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.177+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d800074e0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.178+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80007640 con 0x7f0d9006a4b0 2023-10-16T14:10:32.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.178+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80004a00 con 0x7f0d9006a4b0 2023-10-16T14:10:32.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.178+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d900c9d00 con 0x7f0d9006a4b0 2023-10-16T14:10:32.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.178+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d900ca2c0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.179+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d900ca4b0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.180+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0d80004b60 con 0x7f0d9006a4b0 2023-10-16T14:10:32.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.180+0000 7f0d86ffd640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f0d7006af90 0x7f0d7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.181+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d800896b0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.181+0000 7f0d87fff640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f0d7006af90 0x7f0d7006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.183+0000 7f0d87fff640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f0d7006af90 0x7f0d7006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d78000f60 tx=0x7f0d780079b0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:32.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.185+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d800545b0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.450+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0d9004e9b0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.454+0000 7f0d86ffd640 1 -- 172.21.15.60:0/805885731 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v56) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0d80057bd0 con 0x7f0d9006a4b0 2023-10-16T14:10:32.458 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T14:10:32.459 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T14:10:32.459 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T14:10:32.459 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T14:10:32.459 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:10:32.460 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T14:10:32.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.457+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f0d7006af90 msgr2=0x7f0d7006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.457+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f0d7006af90 0x7f0d7006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d78000f60 tx=0x7f0d780079b0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.457+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 msgr2=0x7f0d901f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.457+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 0x7f0d901f9310 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000c3f0 tx=0x7f0d8000c8b0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 shutdown_connections 2023-10-16T14:10:32.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f0d7006af90 0x7f0d7006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0d900b4160 0x7f0d900c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d9015ac70 0x7f0d900c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 --2- 172.21.15.60:0/805885731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d9006a4b0 0x7f0d901f9310 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 >> 172.21.15.60:0/805885731 conn(0x7f0d900ce6a0 msgr2=0x7f0d900b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 shutdown_connections 2023-10-16T14:10:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.458+0000 7f0d8f59e640 1 -- 172.21.15.60:0/805885731 wait complete. 2023-10-16T14:10:32.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:32.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.702+0000 7f8ea4cbb640 1 Processor -- start 2023-10-16T14:10:32.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.703+0000 7f8ea4cbb640 1 -- start start 2023-10-16T14:10:32.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.704+0000 7f8ea4cbb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea015aaa0 0x7f8ea015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.704+0000 7f8ea4cbb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8ea015ae80 0x7f8ea00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.704+0000 7f8ea4cbb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ea00ccaa0 0x7f8ea0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.704+0000 7f8ea4cbb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea006b810 con 0x7f8ea00ccaa0 2023-10-16T14:10:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.704+0000 7f8ea4cbb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea015ed60 con 0x7f8ea015ae80 2023-10-16T14:10:32.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.704+0000 7f8ea4cbb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00707e0 con 0x7f8ea015aaa0 2023-10-16T14:10:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ea00ccaa0 0x7f8ea0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ea00ccaa0 0x7f8ea0070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45122/0 (socket says 172.21.15.60:45122) 2023-10-16T14:10:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/640782800 learned_addr learned my addr 172.21.15.60:0/640782800 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9dd9b640 1 --2- 172.21.15.60:0/640782800 >> v2:172.21.15.144:3300/0 conn(0x7f8ea015ae80 0x7f8ea00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9e59c640 1 --2- 172.21.15.60:0/640782800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea015aaa0 0x7f8ea015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/640782800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea015aaa0 msgr2=0x7f8ea015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:32.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 --2- 172.21.15.60:0/640782800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea015aaa0 0x7f8ea015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/640782800 >> v2:172.21.15.144:3300/0 conn(0x7f8ea015ae80 msgr2=0x7f8ea00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 --2- 172.21.15.60:0/640782800 >> v2:172.21.15.144:3300/0 conn(0x7f8ea015ae80 0x7f8ea00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.705+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/640782800 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea015ef90 con 0x7f8ea00ccaa0 2023-10-16T14:10:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.707+0000 7f8e9ed9d640 1 --2- 172.21.15.60:0/640782800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea00ccaa0 0x7f8ea0070f50 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f8ea00cb270 tx=0x7f8e9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4257fede350dcd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.707+0000 7f8e9d59a640 1 -- 172.21.15.60:0/640782800 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98031870 con 0x7f8ea00ccaa0 2023-10-16T14:10:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.707+0000 7f8e9d59a640 1 -- 172.21.15.60:0/640782800 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e980319d0 con 0x7f8ea00ccaa0 2023-10-16T14:10:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.707+0000 7f8e9d59a640 1 -- 172.21.15.60:0/640782800 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98031d60 con 0x7f8ea00ccaa0 2023-10-16T14:10:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/640782800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea00ccaa0 msgr2=0x7f8ea0070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/640782800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea00ccaa0 0x7f8ea0070f50 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f8ea00cb270 tx=0x7f8e9802b910 comp rx=0 tx=0).stop 2023-10-16T14:10:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/640782800 shutdown_connections 2023-10-16T14:10:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/640782800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea00ccaa0 0x7f8ea0070f50 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/640782800 >> v2:172.21.15.144:3300/0 conn(0x7f8ea015ae80 0x7f8ea00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/640782800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea015aaa0 0x7f8ea015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/640782800 >> 172.21.15.60:0/640782800 conn(0x7f8ea00cb410 msgr2=0x7f8ea00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/640782800 shutdown_connections 2023-10-16T14:10:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.708+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/640782800 wait complete. 2023-10-16T14:10:32.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.710+0000 7f8ea4cbb640 1 Processor -- start 2023-10-16T14:10:32.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.710+0000 7f8ea4cbb640 1 -- start start 2023-10-16T14:10:32.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8ea4cbb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea00ccaa0 0x7f8ea01589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8ea4cbb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ea015aaa0 0x7f8ea0158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8ea4cbb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 0x7f8ea014d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8ea4cbb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00c3bd0 con 0x7f8ea015ae80 2023-10-16T14:10:32.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8ea4cbb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0159440 con 0x7f8ea015aaa0 2023-10-16T14:10:32.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8ea4cbb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0159710 con 0x7f8ea00ccaa0 2023-10-16T14:10:32.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8e9ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 0x7f8ea014d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8e9ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 0x7f8ea014d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45134/0 (socket says 172.21.15.60:45134) 2023-10-16T14:10:32.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/1016615247 learned_addr learned my addr 172.21.15.60:0/1016615247 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8e9dd9b640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ea015aaa0 0x7f8ea0158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.711+0000 7f8e9e59c640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea00ccaa0 0x7f8ea01589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea00ccaa0 msgr2=0x7f8ea01589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9ed9d640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea00ccaa0 0x7f8ea01589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ea015aaa0 msgr2=0x7f8ea0158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9ed9d640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ea015aaa0 0x7f8ea0158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9ed9d640 1 -- 172.21.15.60:0/1016615247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e980094d0 con 0x7f8ea015ae80 2023-10-16T14:10:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9e59c640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea00ccaa0 0x7f8ea01589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9dd9b640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ea015aaa0 0x7f8ea0158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:10:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e9ed9d640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 0x7f8ea014d0a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f8e980090b0 tx=0x7f8e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9803ac60 con 0x7f8ea015ae80 2023-10-16T14:10:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9803ade0 con 0x7f8ea015ae80 2023-10-16T14:10:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98044630 con 0x7f8ea015ae80 2023-10-16T14:10:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.712+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea01599f0 con 0x7f8ea015ae80 2023-10-16T14:10:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.713+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea00c37f0 con 0x7f8ea015ae80 2023-10-16T14:10:32.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.714+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea00c3d10 con 0x7f8ea015ae80 2023-10-16T14:10:32.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.715+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8e9804d450 con 0x7f8ea015ae80 2023-10-16T14:10:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.716+0000 7f8e96ffd640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.716+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e9802d040 con 0x7f8ea015ae80 2023-10-16T14:10:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.716+0000 7f8e9e59c640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.717+0000 7f8e9e59c640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f8e8006af90 0x7f8e8006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009ed0 tx=0x7f8e9000b040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.720+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8ea00c3d10 con 0x7f8ea015ae80 2023-10-16T14:10:32.772 INFO:tasks.ceph.mds.b.smithi144.stderr:2023-10-16T14:10:32.769+0000 7f181bba1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:10:32.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.957+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ea00c3d10 con 0x7f8ea015ae80 2023-10-16T14:10:32.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.958+0000 7f8e96ffd640 1 -- 172.21.15.60:0/1016615247 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4928 (secure 0 0 0) 0x7f8ea00c3d10 con 0x7f8ea015ae80 2023-10-16T14:10:32.970 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:32.971 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":10998,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T14:10:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 113 2023-10-16T14:10:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f8e8006af90 msgr2=0x7f8e8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f8e8006af90 0x7f8e8006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009ed0 tx=0x7f8e9000b040 comp rx=0 tx=0).stop 2023-10-16T14:10:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 msgr2=0x7f8ea014d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 0x7f8ea014d0a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f8e980090b0 tx=0x7f8e980046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 shutdown_connections 2023-10-16T14:10:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ea015ae80 0x7f8ea014d0a0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ea015aaa0 0x7f8ea0158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 --2- 172.21.15.60:0/1016615247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ea00ccaa0 0x7f8ea01589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 >> 172.21.15.60:0/1016615247 conn(0x7f8ea00cb410 msgr2=0x7f8ea00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 shutdown_connections 2023-10-16T14:10:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:32.961+0000 7f8ea4cbb640 1 -- 172.21.15.60:0/1016615247 wait complete. 2023-10-16T14:10:33.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:10:26.270730+0000', 'modified': '2023-10-16T14:10:29.414557+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11004}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11004': {'gid': 11004, 'name': 'b', 'rank': 0, 'incarnation': 109, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.144:6837/2009359724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6835', 'nonce': 2009359724}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 2009359724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:10:33.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:10:33.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T14:10:33.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.621+0000 7ff129d0f640 1 Processor -- start 2023-10-16T14:10:33.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.621+0000 7ff129d0f640 1 -- start start 2023-10-16T14:10:33.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.622+0000 7ff129d0f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff1240cf710 0x7ff1240b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.622+0000 7ff129d0f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 0x7ff1240b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.622+0000 7ff129d0f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1240d3980 con 0x7ff1240b38e0 2023-10-16T14:10:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.622+0000 7ff129d0f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1240d3c60 con 0x7ff1240cf710 2023-10-16T14:10:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.622+0000 7ff129d0f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240ce630 con 0x7ff1240cfaf0 2023-10-16T14:10:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.623+0000 7ff123fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff1240b38e0 0x7ff1240d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45996/0 (socket says 172.21.15.60:45996) 2023-10-16T14:10:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.623+0000 7ff123fff640 1 -- 172.21.15.60:0/518781100 learned_addr learned my addr 172.21.15.60:0/518781100 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.623+0000 7ff1237fe640 1 --2- 172.21.15.60:0/518781100 >> v2:172.21.15.144:3300/0 conn(0x7ff1240cf710 0x7ff1240b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.623+0000 7ff122ffd640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 0x7ff1240b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.623+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528570760 0 0) 0x7ff1240d3980 con 0x7ff1240b38e0 2023-10-16T14:10:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.623+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff110003520 con 0x7ff1240b38e0 2023-10-16T14:10:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949926378 0 0) 0x7ff110003520 con 0x7ff1240b38e0 2023-10-16T14:10:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 msgr2=0x7ff1240b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 0x7ff1240b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 >> v2:172.21.15.144:3300/0 conn(0x7ff1240cf710 msgr2=0x7ff1240b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 --2- 172.21.15.60:0/518781100 >> v2:172.21.15.144:3300/0 conn(0x7ff1240cf710 0x7ff1240b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240ce910 con 0x7ff1240b38e0 2023-10-16T14:10:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff11c0019f0 con 0x7ff1240b38e0 2023-10-16T14:10:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 >> v1:172.21.15.60:6789/0 conn(0x7ff1240b38e0 legacy=0x7ff1240d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff110003970 0x7ff110003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1100047c0 0x7ff110004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 0x7ff110005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240d3c60 con 0x7ff110003970 2023-10-16T14:10:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240ce630 con 0x7ff1100047c0 2023-10-16T14:10:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11000b9d0 con 0x7ff1240cfaf0 2023-10-16T14:10:33.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1237fe640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff110003970 0x7ff110003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1237fe640 1 -- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 msgr2=0x7ff110005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1237fe640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240cfaf0 0x7ff110005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1237fe640 1 -- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1100047c0 msgr2=0x7ff110004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1237fe640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff1100047c0 0x7ff110004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.624+0000 7ff1237fe640 1 -- 172.21.15.60:0/518781100 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110003520 con 0x7ff110003970 2023-10-16T14:10:33.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.625+0000 7ff1237fe640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff110003970 0x7ff110003d90 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7ff114009ae0 tx=0x7ff1100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89be82c5cdff366c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114005990 con 0x7ff110003970 2023-10-16T14:10:33.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114005af0 con 0x7ff110003970 2023-10-16T14:10:33.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff1227fc640 1 -- 172.21.15.60:0/518781100 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114005ea0 con 0x7ff110003970 2023-10-16T14:10:33.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff129d0f640 1 -- 172.21.15.60:0/518781100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff110003970 msgr2=0x7ff110003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff129d0f640 1 --2- 172.21.15.60:0/518781100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff110003970 0x7ff110003d90 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7ff114009ae0 tx=0x7ff1100057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff123fff640 1 -- 172.21.15.60:0/518781100 reap_dead start 2023-10-16T14:10:33.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff129d0f640 1 -- 172.21.15.60:0/518781100 shutdown_connections 2023-10-16T14:10:33.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff129d0f640 1 -- 172.21.15.60:0/518781100 >> 172.21.15.60:0/518781100 conn(0x7ff1240d2cc0 msgr2=0x7ff1240d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff129d0f640 1 -- 172.21.15.60:0/518781100 shutdown_connections 2023-10-16T14:10:33.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.626+0000 7ff129d0f640 1 -- 172.21.15.60:0/518781100 wait complete. 2023-10-16T14:10:33.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.628+0000 7ff129d0f640 1 Processor -- start 2023-10-16T14:10:33.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.628+0000 7ff129d0f640 1 -- start start 2023-10-16T14:10:33.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.629+0000 7ff129d0f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff110003970 0x7ff1240cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.629+0000 7ff129d0f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 0x7ff1240cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.629+0000 7ff129d0f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240af890 0x7ff1240c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.629+0000 7ff129d0f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240b1ba0 con 0x7ff12415a7a0 2023-10-16T14:10:33.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.629+0000 7ff129d0f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240b0b00 con 0x7ff110003970 2023-10-16T14:10:33.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.629+0000 7ff129d0f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240b0de0 con 0x7ff1240af890 2023-10-16T14:10:33.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 0x7ff1240cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 0x7ff1240cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45150/0 (socket says 172.21.15.60:45150) 2023-10-16T14:10:33.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 -- 172.21.15.60:0/1165379717 learned_addr learned my addr 172.21.15.60:0/1165379717 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff1237fe640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff110003970 0x7ff1240cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff123fff640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240af890 0x7ff1240c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 -- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240af890 msgr2=0x7ff1240c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240af890 0x7ff1240c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 -- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff110003970 msgr2=0x7ff1240cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff110003970 0x7ff1240cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 -- 172.21.15.60:0/1165379717 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240b10c0 con 0x7ff12415a7a0 2023-10-16T14:10:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff1237fe640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff110003970 0x7ff1240cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.630+0000 7ff122ffd640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 0x7ff1240cb350 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00e7a0 tx=0x7ff10c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.631+0000 7ff11bfff640 1 -- 172.21.15.60:0/1165379717 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00eeb0 con 0x7ff12415a7a0 2023-10-16T14:10:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.631+0000 7ff11bfff640 1 -- 172.21.15.60:0/1165379717 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c00c940 con 0x7ff12415a7a0 2023-10-16T14:10:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.631+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1240b13a0 con 0x7ff12415a7a0 2023-10-16T14:10:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.631+0000 7ff11bfff640 1 -- 172.21.15.60:0/1165379717 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00cd10 con 0x7ff12415a7a0 2023-10-16T14:10:33.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.631+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1240b07b0 con 0x7ff12415a7a0 2023-10-16T14:10:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.634+0000 7ff11bfff640 1 -- 172.21.15.60:0/1165379717 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff10c013470 con 0x7ff12415a7a0 2023-10-16T14:10:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.635+0000 7ff11bfff640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff10406afc0 0x7ff10406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.635+0000 7ff11bfff640 1 -- 172.21.15.60:0/1165379717 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff10c089840 con 0x7ff12415a7a0 2023-10-16T14:10:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.635+0000 7ff12c97d740 1 Processor -- start 2023-10-16T14:10:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.636+0000 7ff12c97d740 1 -- start start 2023-10-16T14:10:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.636+0000 7ff1237fe640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff10406afc0 0x7ff10406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.636+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 0x55e73f244e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.636+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f272360 0x55e73f274d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f275250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff12c97d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f1657a0 con 0x55e73f271f80 2023-10-16T14:10:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f1b7740 con 0x55e73f272360 2023-10-16T14:10:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e73f1b4e30 con 0x55e73f2453c0 2023-10-16T14:10:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 0x55e73f244e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 0x55e73f244e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45162/0 (socket says 172.21.15.60:45162) 2023-10-16T14:10:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 -- 172.21.15.60:0/2281903168 learned_addr learned my addr 172.21.15.60:0/2281903168 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff123fff640 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f275250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff122ffd640 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f272360 0x55e73f274d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 -- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 msgr2=0x55e73f275250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f275250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 -- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f272360 msgr2=0x55e73f274d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f272360 0x55e73f274d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff1237fe640 1 -- 172.21.15.60:0/2281903168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e73f174910 con 0x55e73f271f80 2023-10-16T14:10:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.637+0000 7ff122ffd640 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f272360 0x55e73f274d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:33.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.638+0000 7ff1237fe640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff10406afc0 0x7ff10406d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff1140098d0 tx=0x7ff11402e070 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.638+0000 7ff1237fe640 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 0x55e73f244e80 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7ff114000c00 tx=0x7ff114046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431463ffd311f1c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.638+0000 7ff1197fa640 1 -- 172.21.15.60:0/2281903168 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114006230 con 0x55e73f271f80 2023-10-16T14:10:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.638+0000 7ff1197fa640 1 -- 172.21.15.60:0/2281903168 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114006390 con 0x55e73f271f80 2023-10-16T14:10:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.638+0000 7ff1197fa640 1 -- 172.21.15.60:0/2281903168 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114006740 con 0x55e73f271f80 2023-10-16T14:10:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 -- 172.21.15.60:0/2281903168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 msgr2=0x55e73f244e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 0x55e73f244e80 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7ff114000c00 tx=0x7ff114046000 comp rx=0 tx=0).stop 2023-10-16T14:10:33.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 -- 172.21.15.60:0/2281903168 shutdown_connections 2023-10-16T14:10:33.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f275250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f272360 0x55e73f274d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 --2- 172.21.15.60:0/2281903168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f271f80 0x55e73f244e80 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 -- 172.21.15.60:0/2281903168 >> 172.21.15.60:0/2281903168 conn(0x55e73f247190 msgr2=0x55e73f28fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 -- 172.21.15.60:0/2281903168 shutdown_connections 2023-10-16T14:10:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.639+0000 7ff12c97d740 1 -- 172.21.15.60:0/2281903168 wait complete. 2023-10-16T14:10:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.640+0000 7ff12c97d740 1 -- start start 2023-10-16T14:10:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.640+0000 7ff12c97d740 1 Processor -- start 2023-10-16T14:10:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 0x55e73f25f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 0x55e73f274220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f26d750 0x55e73f276480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f1b0720 con 0x55e73f272360 2023-10-16T14:10:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f14e970 con 0x55e73f26d750 2023-10-16T14:10:33.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e73f14e610 con 0x55e73f271f80 2023-10-16T14:10:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff12c97d740 10 client.0 _ll_register_callbacks cb 0x55e73f1709d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:10:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 0x55e73f274220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 0x55e73f274220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45172/0 (socket says 172.21.15.60:45172) 2023-10-16T14:10:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff122ffd640 1 -- 172.21.15.60:0/3003676241 learned_addr learned my addr 172.21.15.60:0/3003676241 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.641+0000 7ff123fff640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f26d750 0x55e73f276480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff122ffd640 1 -- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 msgr2=0x55e73f25f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff122ffd640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 0x55e73f25f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff122ffd640 1 -- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f26d750 msgr2=0x55e73f276480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff122ffd640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f26d750 0x55e73f276480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff122ffd640 1 -- 172.21.15.60:0/3003676241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff114049070 con 0x55e73f272360 2023-10-16T14:10:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff123fff640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f26d750 0x55e73f276480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff122ffd640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 0x55e73f274220 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff10c087c40 tx=0x7ff10c0585d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b7e8c04b4694d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff1027fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:10:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff1027fc640 1 -- 172.21.15.60:0/3003676241 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c089040 con 0x55e73f272360 2023-10-16T14:10:33.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff1027fc640 1 -- 172.21.15.60:0/3003676241 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff10c05c370 con 0x55e73f272360 2023-10-16T14:10:33.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e73f1d7160 con 0x55e73f272360 2023-10-16T14:10:33.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff1027fc640 1 -- 172.21.15.60:0/3003676241 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff10c060300 con 0x55e73f272360 2023-10-16T14:10:33.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff12c97d740 10 client.11148 fetch_fsmap learned FSMap version 113 2023-10-16T14:10:33.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.642+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x55e73f1aeda0 con 0x55e73f272360 2023-10-16T14:10:33.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff1027fc640 1 -- 172.21.15.60:0/3003676241 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c073020 con 0x55e73f272360 2023-10-16T14:10:33.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff1027fc640 1 -- 172.21.15.60:0/3003676241 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 113) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff10c011070 con 0x55e73f272360 2023-10-16T14:10:33.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff12c97d740 10 client.11148 fetch_fsmap finished waiting for FSMap version 113 2023-10-16T14:10:33.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff12c97d740 10 client.11148 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:10:33.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff12c97d740 20 client.11148 populate_metadata read hostname 'smithi060' 2023-10-16T14:10:33.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff12c97d740 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f26b270 0x55e73f2a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff12c97d740 4 client.11148 mds_command: new command op to 11004 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:10:33.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.643+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 --> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e73f182cc0 con 0x55e73f26b270 2023-10-16T14:10:33.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.644+0000 7ff1237fe640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f26b270 0x55e73f2a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.645+0000 7ff1237fe640 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f26b270 0x55e73f2a31f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91de3e58e427f1e5 server_cookie=7ebe2d2b74181b53 in_seq=0 out_seq=0 2023-10-16T14:10:33.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.645+0000 7ff1027fc640 10 client.11148 ms_handle_connect on v2:172.21.15.144:6835/2009359724 2023-10-16T14:10:33.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.646+0000 7ff1027fc640 1 -- 172.21.15.60:0/3003676241 <== mds.0 v2:172.21.15.144:6835/2009359724 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e73f182cc0 con 0x55e73f26b270 2023-10-16T14:10:33.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.646+0000 7ff1027fc640 10 client.11148 handle_command_reply: tid=0 2023-10-16T14:10:33.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.646+0000 7ff12c97d740 1 client.11148 shutdown 2023-10-16T14:10:33.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 msgr2=0x55e73f274220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 0x55e73f274220 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff10c087c40 tx=0x7ff10c0585d0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 shutdown_connections 2023-10-16T14:10:33.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f26d750 0x55e73f276480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f272360 0x55e73f274220 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f26b270 0x55e73f2a31f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 --2- 172.21.15.60:0/3003676241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 0x55e73f25f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 >> 172.21.15.60:0/3003676241 conn(0x55e73f2453c0 msgr2=0x55e73f248e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff1027fc640 0 client.11148 ms_handle_reset on v2:172.21.15.144:6835/2009359724 2023-10-16T14:10:33.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 shutdown_connections 2023-10-16T14:10:33.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 1 -- 172.21.15.60:0/3003676241 wait complete. 2023-10-16T14:10:33.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.647+0000 7ff12c97d740 20 client.11148 trim_cache size 0 max 16384 2023-10-16T14:10:33.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.664+0000 7ff12c97d740 1 Processor -- start 2023-10-16T14:10:33.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.664+0000 7ff12c97d740 1 -- start start 2023-10-16T14:10:33.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f26b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2762f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f271f80 0x55e73f278320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff12c97d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73ef14f20 con 0x55e73f26d750 2023-10-16T14:10:33.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f25e5c0 con 0x55e73f271f80 2023-10-16T14:10:33.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e73f2446f0 con 0x55e73f2453c0 2023-10-16T14:10:33.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2762f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45176/0 (socket says 172.21.15.60:45176) 2023-10-16T14:10:33.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.665+0000 7ff122ffd640 1 -- 172.21.15.60:0/1205026128 learned_addr learned my addr 172.21.15.60:0/1205026128 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff122ffd640 1 -- 172.21.15.60:0/1205026128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 msgr2=0x55e73f26b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff122ffd640 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f26b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff122ffd640 1 -- 172.21.15.60:0/1205026128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f271f80 msgr2=0x55e73f278320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff122ffd640 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f271f80 0x55e73f278320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff122ffd640 1 -- 172.21.15.60:0/1205026128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e73f14db80 con 0x55e73f26d750 2023-10-16T14:10:33.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff122ffd640 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2762f0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7ff10c058190 tx=0x7ff10c05b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4823d5c530ed896 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff119ffb640 1 -- 172.21.15.60:0/1205026128 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c060300 con 0x55e73f26d750 2023-10-16T14:10:33.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff119ffb640 1 -- 172.21.15.60:0/1205026128 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c073070 con 0x55e73f26d750 2023-10-16T14:10:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.666+0000 7ff119ffb640 1 -- 172.21.15.60:0/1205026128 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c05c370 con 0x55e73f26d750 2023-10-16T14:10:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 -- 172.21.15.60:0/1205026128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 msgr2=0x55e73f2762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2762f0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7ff10c058190 tx=0x7ff10c05b000 comp rx=0 tx=0).stop 2023-10-16T14:10:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 -- 172.21.15.60:0/1205026128 shutdown_connections 2023-10-16T14:10:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f271f80 0x55e73f278320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2762f0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 --2- 172.21.15.60:0/1205026128 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f2453c0 0x55e73f26b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 -- 172.21.15.60:0/1205026128 >> 172.21.15.60:0/1205026128 conn(0x55e73f2743e0 msgr2=0x55e73f243e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 -- 172.21.15.60:0/1205026128 shutdown_connections 2023-10-16T14:10:33.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.667+0000 7ff12c97d740 1 -- 172.21.15.60:0/1205026128 wait complete. 2023-10-16T14:10:33.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.668+0000 7ff12c97d740 1 -- start start 2023-10-16T14:10:33.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.668+0000 7ff12c97d740 1 Processor -- start 2023-10-16T14:10:33.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f2453c0 0x55e73f2f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 0x55e73f291600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f278ed0 con 0x55e73f26d750 2023-10-16T14:10:33.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73f269160 con 0x55e73f2453c0 2023-10-16T14:10:33.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e73f2695c0 con 0x55e73f271f80 2023-10-16T14:10:33.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff12c97d740 10 client.0 _ll_register_callbacks cb 0x7ff10c057fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:10:33.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff122ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2910c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45182/0 (socket says 172.21.15.60:45182) 2023-10-16T14:10:33.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.669+0000 7ff122ffd640 1 -- 172.21.15.60:0/407297567 learned_addr learned my addr 172.21.15.60:0/407297567 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff122ffd640 1 -- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 msgr2=0x55e73f291600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff122ffd640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 0x55e73f291600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff1237fe640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f2453c0 0x55e73f2f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff122ffd640 1 -- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f2453c0 msgr2=0x55e73f2f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff122ffd640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f2453c0 0x55e73f2f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff122ffd640 1 -- 172.21.15.60:0/407297567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e73f1aeda0 con 0x55e73f26d750 2023-10-16T14:10:33.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff1237fe640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f2453c0 0x55e73f2f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:33.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff122ffd640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2910c0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff10c058d40 tx=0x7ff10c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cd4873456a61da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff1027fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:10:33.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff1027fc640 1 -- 172.21.15.60:0/407297567 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0547d0 con 0x55e73f26d750 2023-10-16T14:10:33.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff1027fc640 1 -- 172.21.15.60:0/407297567 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff10c074080 con 0x55e73f26d750 2023-10-16T14:10:33.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.670+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff114006390 con 0x55e73f26d750 2023-10-16T14:10:33.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff1027fc640 1 -- 172.21.15.60:0/407297567 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff10c05c380 con 0x55e73f26d750 2023-10-16T14:10:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 10 client.11154 fetch_fsmap learned FSMap version 113 2023-10-16T14:10:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x55e73f173170 con 0x55e73f26d750 2023-10-16T14:10:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff1027fc640 1 -- 172.21.15.60:0/407297567 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114006390 con 0x55e73f26d750 2023-10-16T14:10:33.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff1027fc640 1 -- 172.21.15.60:0/407297567 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 113) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff10c011040 con 0x55e73f26d750 2023-10-16T14:10:33.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 10 client.11154 fetch_fsmap finished waiting for FSMap version 113 2023-10-16T14:10:33.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 10 client.11154 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:10:33.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 20 client.11154 populate_metadata read hostname 'smithi060' 2023-10-16T14:10:33.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f272360 0x55e73f270710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 4 client.11154 mds_command: new command op to 11004 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:10:33.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.671+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 --> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e73f1a9700 con 0x55e73f272360 2023-10-16T14:10:33.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.672+0000 7ff1237fe640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f272360 0x55e73f270710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.673+0000 7ff1237fe640 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f272360 0x55e73f270710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a081deaade2798 server_cookie=a135b4e7151b3253 in_seq=0 out_seq=0 2023-10-16T14:10:33.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.673+0000 7ff1027fc640 10 client.11154 ms_handle_connect on v2:172.21.15.144:6835/2009359724 2023-10-16T14:10:33.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff1027fc640 1 -- 172.21.15.60:0/407297567 <== mds.0 v2:172.21.15.144:6835/2009359724 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55e73f1a9700 con 0x55e73f272360 2023-10-16T14:10:33.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff1027fc640 10 client.11154 handle_command_reply: tid=0 2023-10-16T14:10:33.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 client.11154 shutdown 2023-10-16T14:10:33.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 msgr2=0x55e73f2910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2910c0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff10c058d40 tx=0x7ff10c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 shutdown_connections 2023-10-16T14:10:33.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:6835/2009359724,v1:172.21.15.144:6837/2009359724] conn(0x55e73f272360 0x55e73f270710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55e73f271f80 0x55e73f291600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e73f26d750 0x55e73f2910c0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 --2- 172.21.15.60:0/407297567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e73f2453c0 0x55e73f2f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff1027fc640 0 client.11154 ms_handle_reset on v2:172.21.15.144:6835/2009359724 2023-10-16T14:10:33.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 >> 172.21.15.60:0/407297567 conn(0x55e73f2743e0 msgr2=0x55e73f274c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 shutdown_connections 2023-10-16T14:10:33.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 1 -- 172.21.15.60:0/407297567 wait complete. 2023-10-16T14:10:33.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.674+0000 7ff12c97d740 20 client.11154 trim_cache size 0 max 16384 2023-10-16T14:10:33.689 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:10:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:10:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:10:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 11004, 2023-10-16T14:10:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:10:33.690 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:10:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T14:10:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8412040000000003, 2023-10-16T14:10:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 113, 2023-10-16T14:10:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 231, 2023-10-16T14:10:33.691 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 231, 2023-10-16T14:10:33.692 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.650487527999999 2023-10-16T14:10:33.692 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:10:33.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff10406afc0 msgr2=0x7ff10406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff10406afc0 0x7ff10406d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff1140098d0 tx=0x7ff11402e070 comp rx=0 tx=0).stop 2023-10-16T14:10:33.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 msgr2=0x7ff1240cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 0x7ff1240cb350 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00e7a0 tx=0x7ff10c00ec60 comp rx=0 tx=0).stop 2023-10-16T14:10:33.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 shutdown_connections 2023-10-16T14:10:33.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7ff10406afc0 0x7ff10406d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff1240af890 0x7ff1240c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12415a7a0 0x7ff1240cb350 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 --2- 172.21.15.60:0/1165379717 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff110003970 0x7ff1240cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 >> 172.21.15.60:0/1165379717 conn(0x7ff1240d2cc0 msgr2=0x7ff1240b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 shutdown_connections 2023-10-16T14:10:33.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.675+0000 7ff129d0f640 1 -- 172.21.15.60:0/1165379717 wait complete. 2023-10-16T14:10:33.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:10:33.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.905+0000 7f1ee5c7e640 1 Processor -- start 2023-10-16T14:10:33.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.905+0000 7f1ee5c7e640 1 -- start start 2023-10-16T14:10:33.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ee5c7e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ee006f480 0x7f1ee006f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ee5c7e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee00cce80 con 0x7f1ee006c4d0 2023-10-16T14:10:33.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ee5c7e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee00cd160 con 0x7f1ee00cc270 2023-10-16T14:10:33.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ee5c7e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee00adf50 con 0x7f1ee006f480 2023-10-16T14:10:33.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ed7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ee006f480 0x7f1ee006f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ed7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ee006f480 0x7f1ee006f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45184/0 (socket says 172.21.15.60:45184) 2023-10-16T14:10:33.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.907+0000 7f1ed7fff640 1 -- 172.21.15.60:0/228377883 learned_addr learned my addr 172.21.15.60:0/228377883 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.908+0000 7f1ed7fff640 1 -- 172.21.15.60:0/228377883 >> v1:172.21.15.144:6789/0 conn(0x7f1ee00cc270 legacy=0x7f1ee0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.908+0000 7f1ed7fff640 1 -- 172.21.15.60:0/228377883 >> v1:172.21.15.60:6789/0 conn(0x7f1ee006c4d0 legacy=0x7f1ee00cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.908+0000 7f1ed7fff640 1 -- 172.21.15.60:0/228377883 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00cd440 con 0x7f1ee006f480 2023-10-16T14:10:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.909+0000 7f1ed7fff640 1 --2- 172.21.15.60:0/228377883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee006f8f0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fadb17332037e2d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.909+0000 7f1ed77fe640 1 -- 172.21.15.60:0/228377883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0030860 con 0x7f1ee006f480 2023-10-16T14:10:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.909+0000 7f1ed77fe640 1 -- 172.21.15.60:0/228377883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed00309c0 con 0x7f1ee006f480 2023-10-16T14:10:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.909+0000 7f1ed77fe640 1 -- 172.21.15.60:0/228377883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0030d50 con 0x7f1ee006f480 2023-10-16T14:10:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/228377883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 msgr2=0x7f1ee006f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/228377883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee006f8f0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed002b9a0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/228377883 shutdown_connections 2023-10-16T14:10:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/228377883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee006f8f0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/228377883 >> 172.21.15.60:0/228377883 conn(0x7f1ee00b0a90 msgr2=0x7f1ee00ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/228377883 shutdown_connections 2023-10-16T14:10:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.910+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/228377883 wait complete. 2023-10-16T14:10:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.911+0000 7f1ee5c7e640 1 Processor -- start 2023-10-16T14:10:33.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.912+0000 7f1ee5c7e640 1 -- start start 2023-10-16T14:10:33.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee5c7e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ee006c4d0 0x7f1ee014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee5c7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee01643d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee5c7e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ee00cc270 0x7f1ee0161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee5c7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee014fc90 con 0x7f1ee006f480 2023-10-16T14:10:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee5c7e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee014f010 con 0x7f1ee00cc270 2023-10-16T14:10:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee5c7e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee014f2f0 con 0x7f1ee006c4d0 2023-10-16T14:10:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee01643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee01643d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45198/0 (socket says 172.21.15.60:45198) 2023-10-16T14:10:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 -- 172.21.15.60:0/2814022587 learned_addr learned my addr 172.21.15.60:0/2814022587 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee4c7c640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ee006c4d0 0x7f1ee014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee547d640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ee00cc270 0x7f1ee0161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 -- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ee006c4d0 msgr2=0x7f1ee014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ee006c4d0 0x7f1ee014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 -- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ee00cc270 msgr2=0x7f1ee0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ee00cc270 0x7f1ee0161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ed7fff640 1 -- 172.21.15.60:0/2814022587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed00094d0 con 0x7f1ee006f480 2023-10-16T14:10:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.913+0000 7f1ee547d640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ee00cc270 0x7f1ee0161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.914+0000 7f1ed7fff640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee01643d0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.914+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0030810 con 0x7f1ee006f480 2023-10-16T14:10:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.914+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0043400 con 0x7f1ee006f480 2023-10-16T14:10:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.914+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00437b0 con 0x7f1ee006f480 2023-10-16T14:10:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.914+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee014f5d0 con 0x7f1ee006f480 2023-10-16T14:10:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.914+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee014e790 con 0x7f1ee006f480 2023-10-16T14:10:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.915+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee006fdb0 con 0x7f1ee006f480 2023-10-16T14:10:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.916+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1ed0043910 con 0x7f1ee006f480 2023-10-16T14:10:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.918+0000 7f1ed57fa640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1ecc06afe0 0x7f1ecc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.918+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ed003e070 con 0x7f1ee006f480 2023-10-16T14:10:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.919+0000 7f1ee4c7c640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1ecc06afe0 0x7f1ecc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.920+0000 7f1ee4c7c640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1ecc06afe0 0x7f1ecc06d4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0012e0 tx=0x7f1edc009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:33.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:33.923+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ed0077b90 con 0x7f1ee006f480 2023-10-16T14:10:34.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.157+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1ee004e9b0 con 0x7f1ee006f480 2023-10-16T14:10:34.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.158+0000 7f1ed57fa640 1 -- 172.21.15.60:0/2814022587 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v113) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1ed0034070 con 0x7f1ee006f480 2023-10-16T14:10:34.161 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:10:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1ecc06afe0 msgr2=0x7f1ecc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1ecc06afe0 0x7f1ecc06d4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0012e0 tx=0x7f1edc009040 comp rx=0 tx=0).stop 2023-10-16T14:10:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 msgr2=0x7f1ee01643d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee01643d0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed0004730 comp rx=0 tx=0).stop 2023-10-16T14:10:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 shutdown_connections 2023-10-16T14:10:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1ecc06afe0 0x7f1ecc06d4a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ee00cc270 0x7f1ee0161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ee006f480 0x7f1ee01643d0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 --2- 172.21.15.60:0/2814022587 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1ee006c4d0 0x7f1ee014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.160+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 >> 172.21.15.60:0/2814022587 conn(0x7f1ee00b0a90 msgr2=0x7f1ee006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.161+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 shutdown_connections 2023-10-16T14:10:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:34.161+0000 7f1ee5c7e640 1 -- 172.21.15.60:0/2814022587 wait complete. 2023-10-16T14:10:34.184 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:10:34.184 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:10:34.185 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T14:10:34.185 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:10:34.185 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:10:34.186 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:10:34.186 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:10:34.186 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:10:34.187 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:10:34.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.188 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T14:10:34.245 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:10:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:10:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:10:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T14:10:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.246 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:10:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T14:10:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T14:10:34.247 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:10:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T14:10:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T14:10:34.247 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:10:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:10:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T14:10:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:10:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 36553sec preferred_lft 36553sec 2023-10-16T14:10:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T14:10:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.249 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:10:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:10:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T14:10:34.249 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:10:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T14:10:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:10:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T14:10:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.251 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:10:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:10:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T14:10:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.252 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:10:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:10:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T14:10:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:10:34.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.254 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:10:34.350 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T14:10:34.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.355 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T14:10:34.411 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T14:10:34.411 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:10:34.411 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:10:34.411 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:10:34.412 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:10:34.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.412 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T14:10:34.413 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T14:10:34.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:10:34.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:10:34.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:10:34.414 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T14:10:34.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.618 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:10:34.675 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:10:34.676 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:10:34.677 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:10:34.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T14:10:34.782 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:10:34.782 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:10:34.846 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:10:34.847 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:10:34.848 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:10:34.901 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T14:10:34.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:34.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:10:35.047 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:10:35.047 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:10:35.083 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:10:35.084 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:10:35.084 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T14:10:35.084 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T14:10:35.084 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T14:10:35.088 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:10:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:10:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:10:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:10:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:10:35.093 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:10:35.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:35.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:10:35.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:35.193 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:10:35.276 INFO:teuthology.orchestra.run.smithi060.stdout:client11166 2023-10-16T14:10:35.277 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:10:35.277 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:10:35.339 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T14:10:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.565+0000 7fdf3e613640 1 Processor -- start 2023-10-16T14:10:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.565+0000 7fdf3e613640 1 -- start start 2023-10-16T14:10:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3e613640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf38070f40 0x7fdf380c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3e613640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf380ce2b0 0x7fdf380ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3e613640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf380cde20 con 0x7fdf38069da0 2023-10-16T14:10:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3e613640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf380ca3c0 con 0x7fdf380ce2b0 2023-10-16T14:10:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3e613640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380713d0 con 0x7fdf38070f40 2023-10-16T14:10:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3de12640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf380ce2b0 0x7fdf380ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3de12640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf380ce2b0 0x7fdf380ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45212/0 (socket says 172.21.15.60:45212) 2023-10-16T14:10:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3de12640 1 -- 172.21.15.60:0/1970010065 learned_addr learned my addr 172.21.15.60:0/1970010065 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.567+0000 7fdf3ce10640 1 --2- 172.21.15.60:0/1970010065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf38070f40 0x7fdf380c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.568+0000 7fdf3de12640 1 -- 172.21.15.60:0/1970010065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf38070f40 msgr2=0x7fdf380c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.568+0000 7fdf3de12640 1 --2- 172.21.15.60:0/1970010065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf38070f40 0x7fdf380c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.568+0000 7fdf3de12640 1 -- 172.21.15.60:0/1970010065 >> v1:172.21.15.60:6789/0 conn(0x7fdf38069da0 legacy=0x7fdf3806a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.568+0000 7fdf2ffff640 1 -- 172.21.15.60:0/1970010065 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719830863 0 0) 0x7fdf380cde20 con 0x7fdf38069da0 2023-10-16T14:10:35.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.568+0000 7fdf3de12640 1 -- 172.21.15.60:0/1970010065 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380cab70 con 0x7fdf380ce2b0 2023-10-16T14:10:35.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.569+0000 7fdf3de12640 1 --2- 172.21.15.60:0/1970010065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf380ce2b0 0x7fdf380ce740 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009e50 tx=0x7fdf3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c644dd508d3e3f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.569+0000 7fdf2ffff640 1 -- 172.21.15.60:0/1970010065 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3002ca70 con 0x7fdf380ce2b0 2023-10-16T14:10:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.569+0000 7fdf2ffff640 1 -- 172.21.15.60:0/1970010065 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3002cbd0 con 0x7fdf380ce2b0 2023-10-16T14:10:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.569+0000 7fdf2ffff640 1 -- 172.21.15.60:0/1970010065 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf300343f0 con 0x7fdf380ce2b0 2023-10-16T14:10:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 -- 172.21.15.60:0/1970010065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf380ce2b0 msgr2=0x7fdf380ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 --2- 172.21.15.60:0/1970010065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf380ce2b0 0x7fdf380ce740 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009e50 tx=0x7fdf3002bed0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 -- 172.21.15.60:0/1970010065 shutdown_connections 2023-10-16T14:10:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 --2- 172.21.15.60:0/1970010065 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf380ce2b0 0x7fdf380ce740 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 --2- 172.21.15.60:0/1970010065 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf38070f40 0x7fdf380c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 -- 172.21.15.60:0/1970010065 >> 172.21.15.60:0/1970010065 conn(0x7fdf380cf830 msgr2=0x7fdf380cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 -- 172.21.15.60:0/1970010065 shutdown_connections 2023-10-16T14:10:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.570+0000 7fdf3e613640 1 -- 172.21.15.60:0/1970010065 wait complete. 2023-10-16T14:10:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.572+0000 7fdf3e613640 1 Processor -- start 2023-10-16T14:10:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.572+0000 7fdf3e613640 1 -- start start 2023-10-16T14:10:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3e613640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 0x7fdf380b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3e613640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf38070f40 0x7fdf380b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3e613640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf380ce2b0 0x7fdf381f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3e613640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380b28c0 con 0x7fdf38069da0 2023-10-16T14:10:35.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3e613640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380b2ba0 con 0x7fdf38070f40 2023-10-16T14:10:35.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3e613640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380b2e80 con 0x7fdf380ce2b0 2023-10-16T14:10:35.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3d611640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 0x7fdf380b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:35.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3d611640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 0x7fdf380b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45226/0 (socket says 172.21.15.60:45226) 2023-10-16T14:10:35.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.573+0000 7fdf3d611640 1 -- 172.21.15.60:0/2916607970 learned_addr learned my addr 172.21.15.60:0/2916607970 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:35.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf3d611640 1 -- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf380ce2b0 msgr2=0x7fdf381f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:35.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf3d611640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf380ce2b0 0x7fdf381f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf3d611640 1 -- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf38070f40 msgr2=0x7fdf380b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:35.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf3d611640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf38070f40 0x7fdf380b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf3d611640 1 -- 172.21.15.60:0/2916607970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf30009b30 con 0x7fdf38069da0 2023-10-16T14:10:35.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf3d611640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 0x7fdf380b4440 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400c410 tx=0x7fdf3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:35.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf340035f0 con 0x7fdf38069da0 2023-10-16T14:10:35.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.574+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34003770 con 0x7fdf38069da0 2023-10-16T14:10:35.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.575+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34003b20 con 0x7fdf38069da0 2023-10-16T14:10:35.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.575+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf380b3160 con 0x7fdf38069da0 2023-10-16T14:10:35.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.575+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf380b3960 con 0x7fdf38069da0 2023-10-16T14:10:35.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.576+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf380b3680 con 0x7fdf38069da0 2023-10-16T14:10:35.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.577+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf34003c80 con 0x7fdf38069da0 2023-10-16T14:10:35.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.578+0000 7fdf2dffb640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf1c06af90 0x7fdf1c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:35.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.578+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf34088ed0 con 0x7fdf38069da0 2023-10-16T14:10:35.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.579+0000 7fdf3ce10640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf1c06af90 0x7fdf1c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:35.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.580+0000 7fdf3ce10640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf1c06af90 0x7fdf1c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf28006fd0 tx=0x7fdf28008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:35.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.582+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf380b3680 con 0x7fdf38069da0 2023-10-16T14:10:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.818+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdf380b3680 con 0x7fdf38069da0 2023-10-16T14:10:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.891+0000 7fdf2dffb640 1 -- 172.21.15.60:0/2916607970 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 136+0+0 (secure 0 0 0) 0x7fdf380b3680 con 0x7fdf38069da0 2023-10-16T14:10:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.895+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf1c06af90 msgr2=0x7fdf1c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:35.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.895+0000 7fdf3e613640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf1c06af90 0x7fdf1c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf28006fd0 tx=0x7fdf28008040 comp rx=0 tx=0).stop 2023-10-16T14:10:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 msgr2=0x7fdf380b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 0x7fdf380b4440 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400c410 tx=0x7fdf3400c8d0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 shutdown_connections 2023-10-16T14:10:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fdf1c06af90 0x7fdf1c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf380ce2b0 0x7fdf381f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf38070f40 0x7fdf380b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 --2- 172.21.15.60:0/2916607970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf38069da0 0x7fdf380b4440 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 >> 172.21.15.60:0/2916607970 conn(0x7fdf380cf830 msgr2=0x7fdf380aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:35.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 shutdown_connections 2023-10-16T14:10:35.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:35.896+0000 7fdf3e613640 1 -- 172.21.15.60:0/2916607970 wait complete. 2023-10-16T14:10:35.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.136+0000 7fc8809e9640 1 Processor -- start 2023-10-16T14:10:36.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.137+0000 7fc8809e9640 1 -- start start 2023-10-16T14:10:36.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.138+0000 7fc8809e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 0x7fc87c165690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.138+0000 7fc8809e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c0b3730 con 0x7fc87c0cc2e0 2023-10-16T14:10:36.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.138+0000 7fc8809e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c0b3980 con 0x7fc87c15aa60 2023-10-16T14:10:36.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.138+0000 7fc8809e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0b3c60 con 0x7fc87c0b5750 2023-10-16T14:10:36.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.139+0000 7fc872ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc87c0cc2e0 0x7fc87c0cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46034/0 (socket says 172.21.15.60:46034) 2023-10-16T14:10:36.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.139+0000 7fc872ffd640 1 -- 172.21.15.60:0/537002241 learned_addr learned my addr 172.21.15.60:0/537002241 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:36.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.139+0000 7fc87bfff640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 0x7fc87c165690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.139+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822231037 0 0) 0x7fc87c0b3730 con 0x7fc87c0cc2e0 2023-10-16T14:10:36.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.139+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc868003520 con 0x7fc87c0cc2e0 2023-10-16T14:10:36.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706855999 0 0) 0x7fc868003520 con 0x7fc87c0cc2e0 2023-10-16T14:10:36.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 msgr2=0x7fc87c165690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:36.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 0x7fc87c165690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 >> v1:172.21.15.144:6789/0 conn(0x7fc87c15aa60 legacy=0x7fc87c0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0b3f40 con 0x7fc87c0cc2e0 2023-10-16T14:10:36.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc864001670 con 0x7fc87c0cc2e0 2023-10-16T14:10:36.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 >> v1:172.21.15.60:6789/0 conn(0x7fc87c0cc2e0 legacy=0x7fc87c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8680039a0 0x7fc868003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 0x7fc8680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc868005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0b3c60 con 0x7fc868004d00 2023-10-16T14:10:36.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86800ba30 con 0x7fc8680039a0 2023-10-16T14:10:36.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86800bbf0 con 0x7fc87c0b5750 2023-10-16T14:10:36.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87bfff640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc868005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.140+0000 7fc87b7fe640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8680039a0 0x7fc868003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.141+0000 7fc87bfff640 1 -- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 msgr2=0x7fc8680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:36.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.141+0000 7fc87bfff640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0b5750 0x7fc8680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.141+0000 7fc87bfff640 1 -- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8680039a0 msgr2=0x7fc868003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:36.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.141+0000 7fc87bfff640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8680039a0 0x7fc868003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.141+0000 7fc87bfff640 1 -- 172.21.15.60:0/537002241 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc868003520 con 0x7fc868004d00 2023-10-16T14:10:36.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc87bfff640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc868005180 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0b5be0 tx=0x7fc8680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbd8063dd7b9d32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:36.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87402c680 con 0x7fc868004d00 2023-10-16T14:10:36.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87402c7e0 con 0x7fc868004d00 2023-10-16T14:10:36.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc87affd640 1 -- 172.21.15.60:0/537002241 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87402cb90 con 0x7fc868004d00 2023-10-16T14:10:36.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc8809e9640 1 -- 172.21.15.60:0/537002241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 msgr2=0x7fc868005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc8809e9640 1 --2- 172.21.15.60:0/537002241 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc868005180 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0b5be0 tx=0x7fc8680057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.142+0000 7fc87bfff640 1 -- 172.21.15.60:0/537002241 reap_dead start 2023-10-16T14:10:36.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.143+0000 7fc8809e9640 1 -- 172.21.15.60:0/537002241 shutdown_connections 2023-10-16T14:10:36.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.143+0000 7fc8809e9640 1 -- 172.21.15.60:0/537002241 >> 172.21.15.60:0/537002241 conn(0x7fc87c065ed0 msgr2=0x7fc87c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:36.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.143+0000 7fc8809e9640 1 -- 172.21.15.60:0/537002241 shutdown_connections 2023-10-16T14:10:36.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.143+0000 7fc8809e9640 1 -- 172.21.15.60:0/537002241 wait complete. 2023-10-16T14:10:36.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.144+0000 7fc8809e9640 1 Processor -- start 2023-10-16T14:10:36.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.145+0000 7fc8809e9640 1 -- start start 2023-10-16T14:10:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc8809e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc87c1f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc8809e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0cf750 0x7fc87c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc8809e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc87c0b1290 0x7fc87c154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc8809e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0b2950 con 0x7fc868004d00 2023-10-16T14:10:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc8809e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0b3190 con 0x7fc87c0b1290 2023-10-16T14:10:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc8809e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0b2bd0 con 0x7fc87c0cf750 2023-10-16T14:10:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc87b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc87c1f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc87b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc87c1f0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45252/0 (socket says 172.21.15.60:45252) 2023-10-16T14:10:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.146+0000 7fc87b7fe640 1 -- 172.21.15.60:0/1614446806 learned_addr learned my addr 172.21.15.60:0/1614446806 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87bfff640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc87c0b1290 0x7fc87c154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87b7fe640 1 -- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0cf750 msgr2=0x7fc87c0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87b7fe640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0cf750 0x7fc87c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87b7fe640 1 -- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc87c0b1290 msgr2=0x7fc87c154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87b7fe640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc87c0b1290 0x7fc87c154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87b7fe640 1 -- 172.21.15.60:0/1614446806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0b2eb0 con 0x7fc868004d00 2023-10-16T14:10:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc87b7fe640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc87c1f0530 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00e8c0 tx=0x7fc86c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c003b80 con 0x7fc868004d00 2023-10-16T14:10:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c003ce0 con 0x7fc868004d00 2023-10-16T14:10:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c1efd30 con 0x7fc868004d00 2023-10-16T14:10:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c0052f0 con 0x7fc868004d00 2023-10-16T14:10:36.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.147+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c0b05b0 con 0x7fc868004d00 2023-10-16T14:10:36.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.149+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c04e9b0 con 0x7fc868004d00 2023-10-16T14:10:36.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.150+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc86c005450 con 0x7fc868004d00 2023-10-16T14:10:36.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.151+0000 7fc878ff9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc85c06afe0 0x7fc85c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.151+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc86c089920 con 0x7fc868004d00 2023-10-16T14:10:36.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.152+0000 7fc872ffd640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc85c06afe0 0x7fc85c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.153+0000 7fc872ffd640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc85c06afe0 0x7fc85c06d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc864000f60 tx=0x7fc864001570 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:36.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.156+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc86c0548a0 con 0x7fc868004d00 2023-10-16T14:10:36.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.351+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc87c1573e0 con 0x7fc868004d00 2023-10-16T14:10:36.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.354+0000 7fc878ff9640 1 -- 172.21.15.60:0/1614446806 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fc86c01a310 con 0x7fc868004d00 2023-10-16T14:10:36.359 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:36.359 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":231,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:27.831029+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":229,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:36.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.357+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc85c06afe0 msgr2=0x7fc85c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.357+0000 7fc8809e9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc85c06afe0 0x7fc85c06d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc864000f60 tx=0x7fc864001570 comp rx=0 tx=0).stop 2023-10-16T14:10:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.357+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 msgr2=0x7fc87c1f0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.357+0000 7fc8809e9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc87c1f0530 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00e8c0 tx=0x7fc86c00ed80 comp rx=0 tx=0).stop 2023-10-16T14:10:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 shutdown_connections 2023-10-16T14:10:36.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc85c06afe0 0x7fc85c06d4a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc87c0cf750 0x7fc87c0b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc87c0b1290 0x7fc87c154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 --2- 172.21.15.60:0/1614446806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc868004d00 0x7fc87c1f0530 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 >> 172.21.15.60:0/1614446806 conn(0x7fc87c065ed0 msgr2=0x7fc87c0c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:36.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 shutdown_connections 2023-10-16T14:10:36.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.358+0000 7fc8809e9640 1 -- 172.21.15.60:0/1614446806 wait complete. 2023-10-16T14:10:36.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:10:24.220406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:10:25.316288+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:10:36.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:10:36.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.612+0000 7f50bee13640 1 Processor -- start 2023-10-16T14:10:36.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.612+0000 7f50bee13640 1 -- start start 2023-10-16T14:10:36.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.613+0000 7f50bee13640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50b8070f30 0x7f50b806aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.613+0000 7f50bee13640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50b806b3e0 0x7f50b80c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.613+0000 7f50bee13640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b80cd260 con 0x7f50b8159ad0 2023-10-16T14:10:36.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bee13640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b815de30 con 0x7f50b8070f30 2023-10-16T14:10:36.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bee13640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b815e110 con 0x7f50b806b3e0 2023-10-16T14:10:36.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bde11640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f50b8159ad0 0x7f50b815d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46038/0 (socket says 172.21.15.60:46038) 2023-10-16T14:10:36.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bd610640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50b8070f30 0x7f50b806aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bde11640 1 -- 172.21.15.60:0/3791558064 learned_addr learned my addr 172.21.15.60:0/3791558064 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:36.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50be612640 1 --2- 172.21.15.60:0/3791558064 >> v2:172.21.15.144:3300/0 conn(0x7f50b806b3e0 0x7f50b80c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bd610640 1 -- 172.21.15.60:0/3791558064 >> v2:172.21.15.144:3300/0 conn(0x7f50b806b3e0 msgr2=0x7f50b80c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:36.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bd610640 1 --2- 172.21.15.60:0/3791558064 >> v2:172.21.15.144:3300/0 conn(0x7f50b806b3e0 0x7f50b80c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bd610640 1 -- 172.21.15.60:0/3791558064 >> v1:172.21.15.60:6789/0 conn(0x7f50b8159ad0 legacy=0x7f50b815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bd610640 1 -- 172.21.15.60:0/3791558064 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b815e3f0 con 0x7f50b8070f30 2023-10-16T14:10:36.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.614+0000 7f50bce0f640 1 -- 172.21.15.60:0/3791558064 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956227770 0 0) 0x7f50b80cd260 con 0x7f50b8159ad0 2023-10-16T14:10:36.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.616+0000 7f50bd610640 1 --2- 172.21.15.60:0/3791558064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b8070f30 0x7f50b806aea0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f50b806d720 tx=0x7f50ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f40bc8f293451c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:36.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.616+0000 7f50bce0f640 1 -- 172.21.15.60:0/3791558064 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac031830 con 0x7f50b8070f30 2023-10-16T14:10:36.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.616+0000 7f50bce0f640 1 -- 172.21.15.60:0/3791558064 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac031990 con 0x7f50b8070f30 2023-10-16T14:10:36.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.616+0000 7f50bce0f640 1 -- 172.21.15.60:0/3791558064 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac031d20 con 0x7f50b8070f30 2023-10-16T14:10:36.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 -- 172.21.15.60:0/3791558064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b8070f30 msgr2=0x7f50b806aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 --2- 172.21.15.60:0/3791558064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b8070f30 0x7f50b806aea0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f50b806d720 tx=0x7f50ac02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 -- 172.21.15.60:0/3791558064 shutdown_connections 2023-10-16T14:10:36.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 --2- 172.21.15.60:0/3791558064 >> v2:172.21.15.144:3300/0 conn(0x7f50b806b3e0 0x7f50b80c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 --2- 172.21.15.60:0/3791558064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b8070f30 0x7f50b806aea0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 -- 172.21.15.60:0/3791558064 >> 172.21.15.60:0/3791558064 conn(0x7f50b80b0170 msgr2=0x7f50b80b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:36.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 -- 172.21.15.60:0/3791558064 shutdown_connections 2023-10-16T14:10:36.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.617+0000 7f50bee13640 1 -- 172.21.15.60:0/3791558064 wait complete. 2023-10-16T14:10:36.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.618+0000 7f50bee13640 1 Processor -- start 2023-10-16T14:10:36.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.619+0000 7f50bee13640 1 -- start start 2023-10-16T14:10:36.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bee13640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 0x7f50b80b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bee13640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b8070f30 0x7f50b814dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bee13640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b8159ad0 0x7f50b80b1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bee13640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80b44c0 con 0x7f50b806b3e0 2023-10-16T14:10:36.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bee13640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b815a600 con 0x7f50b8159ad0 2023-10-16T14:10:36.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bee13640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b815ab70 con 0x7f50b8070f30 2023-10-16T14:10:36.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 0x7f50b80b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 0x7f50b80b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45272/0 (socket says 172.21.15.60:45272) 2023-10-16T14:10:36.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 -- 172.21.15.60:0/3434820847 learned_addr learned my addr 172.21.15.60:0/3434820847 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:36.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50be612640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b8159ad0 0x7f50b80b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bd610640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b8070f30 0x7f50b814dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 -- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b8070f30 msgr2=0x7f50b814dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b8070f30 0x7f50b814dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 -- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b8159ad0 msgr2=0x7f50b80b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b8159ad0 0x7f50b80b1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.620+0000 7f50bde11640 1 -- 172.21.15.60:0/3434820847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009400 con 0x7f50b806b3e0 2023-10-16T14:10:36.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.621+0000 7f50bde11640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 0x7f50b80b38b0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f50a800c7c0 tx=0x7f50a800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:36.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.621+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800a640 con 0x7f50b806b3e0 2023-10-16T14:10:36.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.621+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a800a7c0 con 0x7f50b806b3e0 2023-10-16T14:10:36.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.621+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800ab70 con 0x7f50b806b3e0 2023-10-16T14:10:36.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.621+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b80b3f00 con 0x7f50b806b3e0 2023-10-16T14:10:36.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.621+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b815a800 con 0x7f50b806b3e0 2023-10-16T14:10:36.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.622+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b804e9b0 con 0x7f50b806b3e0 2023-10-16T14:10:36.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.623+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f50a800acd0 con 0x7f50b806b3e0 2023-10-16T14:10:36.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.624+0000 7f50b67fc640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f50a406afe0 0x7f50a406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:36.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.625+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50a8089130 con 0x7f50b806b3e0 2023-10-16T14:10:36.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.625+0000 7f50bd610640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f50a406afe0 0x7f50a406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:36.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.626+0000 7f50bd610640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f50a406afe0 0x7f50a406d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f50ac036000 tx=0x7f50ac008030 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:36.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.628+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50a8054140 con 0x7f50b806b3e0 2023-10-16T14:10:36.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.826+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50b806d530 con 0x7f50b806b3e0 2023-10-16T14:10:36.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.827+0000 7f50b67fc640 1 -- 172.21.15.60:0/3434820847 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v231) v1 ==== 93+0+10 (secure 0 0 0) 0x7f50a8018020 con 0x7f50b806b3e0 2023-10-16T14:10:36.831 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:10:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f50a406afe0 msgr2=0x7f50a406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f50a406afe0 0x7f50a406d4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f50ac036000 tx=0x7f50ac008030 comp rx=0 tx=0).stop 2023-10-16T14:10:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 msgr2=0x7f50b80b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 0x7f50b80b38b0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f50a800c7c0 tx=0x7f50a800cc80 comp rx=0 tx=0).stop 2023-10-16T14:10:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 shutdown_connections 2023-10-16T14:10:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f50a406afe0 0x7f50a406d4a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50b8070f30 0x7f50b814dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50b8159ad0 0x7f50b80b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 --2- 172.21.15.60:0/3434820847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b806b3e0 0x7f50b80b38b0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 >> 172.21.15.60:0/3434820847 conn(0x7f50b80b0170 msgr2=0x7f50b81575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 shutdown_connections 2023-10-16T14:10:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:36.830+0000 7f50bee13640 1 -- 172.21.15.60:0/3434820847 wait complete. 2023-10-16T14:10:36.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:10:37.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.080+0000 7f6e76cf8640 1 Processor -- start 2023-10-16T14:10:37.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.081+0000 7f6e76cf8640 1 -- start start 2023-10-16T14:10:37.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.082+0000 7f6e76cf8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6e7015dde0 0x7f6e700709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.082+0000 7f6e76cf8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e70070f20 0x7f6e700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.082+0000 7f6e76cf8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70160a90 con 0x7f6e7015c9f0 2023-10-16T14:10:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.082+0000 7f6e76cf8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e70071350 con 0x7f6e70070f20 2023-10-16T14:10:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.082+0000 7f6e76cf8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e700cdec0 con 0x7f6e7015dde0 2023-10-16T14:10:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e70070f20 0x7f6e700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e70070f20 0x7f6e700ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45288/0 (socket says 172.21.15.60:45288) 2023-10-16T14:10:37.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 -- 172.21.15.60:0/2445999234 learned_addr learned my addr 172.21.15.60:0/2445999234 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:37.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e754f5640 1 --2- 172.21.15.60:0/2445999234 >> v2:172.21.15.144:3300/0 conn(0x7f6e7015dde0 0x7f6e700709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 -- 172.21.15.60:0/2445999234 >> v2:172.21.15.144:3300/0 conn(0x7f6e7015dde0 msgr2=0x7f6e700709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 --2- 172.21.15.60:0/2445999234 >> v2:172.21.15.144:3300/0 conn(0x7f6e7015dde0 0x7f6e700709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 -- 172.21.15.60:0/2445999234 >> v1:172.21.15.144:6789/0 conn(0x7f6e7015c9f0 legacy=0x7f6e700c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e764f7640 1 -- 172.21.15.60:0/2445999234 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700cd900 con 0x7f6e70070f20 2023-10-16T14:10:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.083+0000 7f6e754f5640 1 --2- 172.21.15.60:0/2445999234 >> v2:172.21.15.144:3300/0 conn(0x7f6e7015dde0 0x7f6e700709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.084+0000 7f6e764f7640 1 --2- 172.21.15.60:0/2445999234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e70070f20 0x7f6e700ac890 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f6e700b1260 tx=0x7f6e6802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb8ca4050ebd616 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:37.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e74cf4640 1 -- 172.21.15.60:0/2445999234 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6802ca80 con 0x7f6e70070f20 2023-10-16T14:10:37.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e74cf4640 1 -- 172.21.15.60:0/2445999234 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6802cbe0 con 0x7f6e70070f20 2023-10-16T14:10:37.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e74cf4640 1 -- 172.21.15.60:0/2445999234 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680343f0 con 0x7f6e70070f20 2023-10-16T14:10:37.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2445999234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e70070f20 msgr2=0x7f6e700ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2445999234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e70070f20 0x7f6e700ac890 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f6e700b1260 tx=0x7f6e6802bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2445999234 shutdown_connections 2023-10-16T14:10:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2445999234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e70070f20 0x7f6e700ac890 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2445999234 >> v2:172.21.15.144:3300/0 conn(0x7f6e7015dde0 0x7f6e700709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.085+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2445999234 >> 172.21.15.60:0/2445999234 conn(0x7f6e700ce6a0 msgr2=0x7f6e700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:37.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.086+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2445999234 shutdown_connections 2023-10-16T14:10:37.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.086+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2445999234 wait complete. 2023-10-16T14:10:37.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.087+0000 7f6e76cf8640 1 Processor -- start 2023-10-16T14:10:37.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.087+0000 7f6e76cf8640 1 -- start start 2023-10-16T14:10:37.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.088+0000 7f6e76cf8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e70070f20 0x7f6e700c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.088+0000 7f6e76cf8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 0x7f6e70158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.088+0000 7f6e76cf8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e7015dde0 0x7f6e701594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.088+0000 7f6e76cf8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700c4450 con 0x7f6e7015c9f0 2023-10-16T14:10:37.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.088+0000 7f6e76cf8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700c4cb0 con 0x7f6e7015dde0 2023-10-16T14:10:37.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.088+0000 7f6e76cf8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700c4170 con 0x7f6e70070f20 2023-10-16T14:10:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 0x7f6e70158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 0x7f6e70158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45304/0 (socket says 172.21.15.60:45304) 2023-10-16T14:10:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 -- 172.21.15.60:0/2160663231 learned_addr learned my addr 172.21.15.60:0/2160663231 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e764f7640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e7015dde0 0x7f6e701594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e75cf6640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e70070f20 0x7f6e700c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 -- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e70070f20 msgr2=0x7f6e700c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e70070f20 0x7f6e700c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 -- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e7015dde0 msgr2=0x7f6e701594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e7015dde0 0x7f6e701594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 -- 172.21.15.60:0/2160663231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e68009b30 con 0x7f6e7015c9f0 2023-10-16T14:10:37.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e75cf6640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e70070f20 0x7f6e700c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:37.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.089+0000 7f6e754f5640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 0x7f6e70158f60 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000c820 tx=0x7f6e6000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:37.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.090+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60007a30 con 0x7f6e7015c9f0 2023-10-16T14:10:37.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.090+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60007b90 con 0x7f6e7015c9f0 2023-10-16T14:10:37.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.090+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70069f40 con 0x7f6e7015c9f0 2023-10-16T14:10:37.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.090+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000ce70 con 0x7f6e7015c9f0 2023-10-16T14:10:37.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.090+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7006a500 con 0x7f6e7015c9f0 2023-10-16T14:10:37.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.091+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e70069da0 con 0x7f6e7015c9f0 2023-10-16T14:10:37.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.093+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e600113f0 con 0x7f6e7015c9f0 2023-10-16T14:10:37.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.093+0000 7f6e667fc640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6e5406afe0 0x7f6e5406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.094+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e600899c0 con 0x7f6e7015c9f0 2023-10-16T14:10:37.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.094+0000 7f6e75cf6640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6e5406afe0 0x7f6e5406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.095+0000 7f6e75cf6640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6e5406afe0 0x7f6e5406d4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c007fd0 tx=0x7f6e6c009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:37.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.099+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e70069da0 con 0x7f6e7015c9f0 2023-10-16T14:10:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.288+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6e70069da0 con 0x7f6e7015c9f0 2023-10-16T14:10:37.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.289+0000 7f6e667fc640 1 -- 172.21.15.60:0/2160663231 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v231) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6e70069da0 con 0x7f6e7015c9f0 2023-10-16T14:10:37.293 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:10:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6e5406afe0 msgr2=0x7f6e5406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6e5406afe0 0x7f6e5406d4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c007fd0 tx=0x7f6e6c009040 comp rx=0 tx=0).stop 2023-10-16T14:10:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 msgr2=0x7f6e70158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 0x7f6e70158f60 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000c820 tx=0x7f6e6000cc20 comp rx=0 tx=0).stop 2023-10-16T14:10:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 shutdown_connections 2023-10-16T14:10:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6e5406afe0 0x7f6e5406d4a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e7015dde0 0x7f6e701594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e7015c9f0 0x7f6e70158f60 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 --2- 172.21.15.60:0/2160663231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e70070f20 0x7f6e700c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 >> 172.21.15.60:0/2160663231 conn(0x7f6e700ce6a0 msgr2=0x7f6e700aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 shutdown_connections 2023-10-16T14:10:37.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.292+0000 7f6e76cf8640 1 -- 172.21.15.60:0/2160663231 wait complete. 2023-10-16T14:10:37.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:10:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.533+0000 7f439c161640 1 Processor -- start 2023-10-16T14:10:37.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.533+0000 7f439c161640 1 -- start start 2023-10-16T14:10:37.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439c161640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4394159ab0 0x7f43940ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439c161640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f439406c140 0x7f439406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439c161640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394070360 con 0x7f4394159e90 2023-10-16T14:10:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439c161640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439406cb10 con 0x7f439406c140 2023-10-16T14:10:37.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439c161640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439415d890 con 0x7f4394159ab0 2023-10-16T14:10:37.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439a6fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f439406c140 0x7f439406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439a6fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f439406c140 0x7f439406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45318/0 (socket says 172.21.15.60:45318) 2023-10-16T14:10:37.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.535+0000 7f439a6fe640 1 -- 172.21.15.60:0/3730836856 learned_addr learned my addr 172.21.15.60:0/3730836856 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:37.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.536+0000 7f4399efd640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4394159ab0 0x7f43940ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.536+0000 7f439a6fe640 1 -- 172.21.15.60:0/3730836856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4394159ab0 msgr2=0x7f43940ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:37.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.536+0000 7f439a6fe640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4394159ab0 0x7f43940ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.536+0000 7f439a6fe640 1 -- 172.21.15.60:0/3730836856 >> v1:172.21.15.60:6789/0 conn(0x7f4394159e90 legacy=0x7f439406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.536+0000 7f439a6fe640 1 -- 172.21.15.60:0/3730836856 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43940cd1f0 con 0x7f439406c140 2023-10-16T14:10:37.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.536+0000 7f4399efd640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4394159ab0 0x7f43940ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:37.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.537+0000 7f439a6fe640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439406c140 0x7f439406c5d0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f438c0098a0 tx=0x7f438c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9772ad6df5e7d2b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:37.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.537+0000 7f4398efb640 1 -- 172.21.15.60:0/3730836856 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c0317c0 con 0x7f439406c140 2023-10-16T14:10:37.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.537+0000 7f4398efb640 1 -- 172.21.15.60:0/3730836856 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c031920 con 0x7f439406c140 2023-10-16T14:10:37.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.537+0000 7f4398efb640 1 -- 172.21.15.60:0/3730836856 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c031cb0 con 0x7f439406c140 2023-10-16T14:10:37.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 -- 172.21.15.60:0/3730836856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439406c140 msgr2=0x7f439406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439406c140 0x7f439406c5d0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f438c0098a0 tx=0x7f438c02b860 comp rx=0 tx=0).stop 2023-10-16T14:10:37.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 -- 172.21.15.60:0/3730836856 shutdown_connections 2023-10-16T14:10:37.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f439406c140 0x7f439406c5d0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 --2- 172.21.15.60:0/3730836856 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4394159ab0 0x7f43940ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 -- 172.21.15.60:0/3730836856 >> 172.21.15.60:0/3730836856 conn(0x7f43940ce9b0 msgr2=0x7f43940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 -- 172.21.15.60:0/3730836856 shutdown_connections 2023-10-16T14:10:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.538+0000 7f439c161640 1 -- 172.21.15.60:0/3730836856 wait complete. 2023-10-16T14:10:37.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.540+0000 7f439c161640 1 Processor -- start 2023-10-16T14:10:37.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.540+0000 7f439c161640 1 -- start start 2023-10-16T14:10:37.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f439c161640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439406c140 0x7f4394153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f439c161640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 0x7f43940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f439c161640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4394159e90 0x7f43940acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f439c161640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43940c50d0 con 0x7f4394159ab0 2023-10-16T14:10:37.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f439c161640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43940c24e0 con 0x7f4394159e90 2023-10-16T14:10:37.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f439c161640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43940c3060 con 0x7f439406c140 2023-10-16T14:10:37.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f43996fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 0x7f43940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f43996fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 0x7f43940ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45330/0 (socket says 172.21.15.60:45330) 2023-10-16T14:10:37.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.541+0000 7f43996fc640 1 -- 172.21.15.60:0/1219171389 learned_addr learned my addr 172.21.15.60:0/1219171389 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:37.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f43996fc640 1 -- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439406c140 msgr2=0x7f4394153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f4399efd640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439406c140 0x7f4394153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f43996fc640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439406c140 0x7f4394153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f43996fc640 1 -- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4394159e90 msgr2=0x7f43940acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f439a6fe640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4394159e90 0x7f43940acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f43996fc640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4394159e90 0x7f43940acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f43996fc640 1 -- 172.21.15.60:0/1219171389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c0094d0 con 0x7f4394159ab0 2023-10-16T14:10:37.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f4399efd640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439406c140 0x7f4394153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:37.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f43996fc640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 0x7f43940ac890 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f438400c450 tx=0x7f438400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:37.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.542+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43840052d0 con 0x7f4394159ab0 2023-10-16T14:10:37.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.543+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4384005430 con 0x7f4394159ab0 2023-10-16T14:10:37.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.543+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43940c34f0 con 0x7f4394159ab0 2023-10-16T14:10:37.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.543+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384012870 con 0x7f4394159ab0 2023-10-16T14:10:37.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.543+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43940c2a10 con 0x7f4394159ab0 2023-10-16T14:10:37.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.544+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439404e160 con 0x7f4394159ab0 2023-10-16T14:10:37.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.545+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43840129d0 con 0x7f4394159ab0 2023-10-16T14:10:37.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.546+0000 7f438a7fc640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f437806af90 0x7f437806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:37.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.546+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4384089200 con 0x7f4394159ab0 2023-10-16T14:10:37.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.547+0000 7f4399efd640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f437806af90 0x7f437806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:37.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.548+0000 7f4399efd640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f437806af90 0x7f437806d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43940c9b70 tx=0x7f43900074b0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:37.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.550+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4384054230 con 0x7f4394159ab0 2023-10-16T14:10:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.745+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43940ccb60 con 0x7f4394159ab0 2023-10-16T14:10:37.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.746+0000 7f438a7fc640 1 -- 172.21.15.60:0/1219171389 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v231) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4384057850 con 0x7f4394159ab0 2023-10-16T14:10:37.750 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:10:37.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.748+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f437806af90 msgr2=0x7f437806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.748+0000 7f439c161640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f437806af90 0x7f437806d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43940c9b70 tx=0x7f43900074b0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 msgr2=0x7f43940ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 0x7f43940ac890 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f438400c450 tx=0x7f438400c910 comp rx=0 tx=0).stop 2023-10-16T14:10:37.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 shutdown_connections 2023-10-16T14:10:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f437806af90 0x7f437806d450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4394159e90 0x7f43940acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4394159ab0 0x7f43940ac890 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 --2- 172.21.15.60:0/1219171389 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f439406c140 0x7f4394153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 >> 172.21.15.60:0/1219171389 conn(0x7f43940ce9b0 msgr2=0x7f43940ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 shutdown_connections 2023-10-16T14:10:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:37.749+0000 7f439c161640 1 -- 172.21.15.60:0/1219171389 wait complete. 2023-10-16T14:10:37.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:10:38.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.002+0000 7f7de775a640 1 Processor -- start 2023-10-16T14:10:38.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.003+0000 7f7de775a640 1 -- start start 2023-10-16T14:10:38.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.004+0000 7f7de775a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7de015aaa0 0x7f7de015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.004+0000 7f7de775a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de015ed60 con 0x7f7de00ce390 2023-10-16T14:10:38.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.004+0000 7f7de775a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00707e0 con 0x7f7de015ae80 2023-10-16T14:10:38.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.004+0000 7f7de775a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de006b810 con 0x7f7de015aaa0 2023-10-16T14:10:38.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7de54f6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7de015aaa0 0x7f7de015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7de54f6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7de015aaa0 0x7f7de015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45344/0 (socket says 172.21.15.60:45344) 2023-10-16T14:10:38.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7de54f6640 1 -- 172.21.15.60:0/2935906955 learned_addr learned my addr 172.21.15.60:0/2935906955 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:38.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7de54f6640 1 -- 172.21.15.60:0/2935906955 >> v1:172.21.15.144:6789/0 conn(0x7f7de015ae80 legacy=0x7f7de00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:38.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7de54f6640 1 -- 172.21.15.60:0/2935906955 >> v1:172.21.15.60:6789/0 conn(0x7f7de00ce390 legacy=0x7f7de0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7de54f6640 1 -- 172.21.15.60:0/2935906955 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de015ef90 con 0x7f7de015aaa0 2023-10-16T14:10:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.005+0000 7f7dd7fff640 1 -- 172.21.15.60:0/2935906955 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160757503 0 0) 0x7f7de015ed60 con 0x7f7de00ce390 2023-10-16T14:10:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.006+0000 7f7de54f6640 1 --2- 172.21.15.60:0/2935906955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015aaa0 0x7f7de015e820 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009870 tx=0x7f7ddc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff674859ae112d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7dd7fff640 1 -- 172.21.15.60:0/2935906955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc030860 con 0x7f7de015aaa0 2023-10-16T14:10:38.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7dd7fff640 1 -- 172.21.15.60:0/2935906955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc0309c0 con 0x7f7de015aaa0 2023-10-16T14:10:38.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7dd7fff640 1 -- 172.21.15.60:0/2935906955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc030d50 con 0x7f7de015aaa0 2023-10-16T14:10:38.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7de775a640 1 -- 172.21.15.60:0/2935906955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015aaa0 msgr2=0x7f7de015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7de775a640 1 --2- 172.21.15.60:0/2935906955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015aaa0 0x7f7de015e820 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009870 tx=0x7f7ddc02b9a0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7de775a640 1 -- 172.21.15.60:0/2935906955 shutdown_connections 2023-10-16T14:10:38.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7de775a640 1 --2- 172.21.15.60:0/2935906955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015aaa0 0x7f7de015e820 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7de775a640 1 -- 172.21.15.60:0/2935906955 >> 172.21.15.60:0/2935906955 conn(0x7f7de00cb410 msgr2=0x7f7de00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.007+0000 7f7de775a640 1 -- 172.21.15.60:0/2935906955 shutdown_connections 2023-10-16T14:10:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.008+0000 7f7de775a640 1 -- 172.21.15.60:0/2935906955 wait complete. 2023-10-16T14:10:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.009+0000 7f7de775a640 1 Processor -- start 2023-10-16T14:10:38.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.009+0000 7f7de775a640 1 -- start start 2023-10-16T14:10:38.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.010+0000 7f7de775a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7de00ce390 0x7f7de015d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.010+0000 7f7de775a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7de015aaa0 0x7f7de0158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.010+0000 7f7de775a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 0x7f7de015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.010+0000 7f7de775a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de015c360 con 0x7f7de015ae80 2023-10-16T14:10:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.010+0000 7f7de775a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de015c640 con 0x7f7de00ce390 2023-10-16T14:10:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.010+0000 7f7de775a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de015c920 con 0x7f7de015aaa0 2023-10-16T14:10:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 0x7f7de015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 0x7f7de015dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45348/0 (socket says 172.21.15.60:45348) 2023-10-16T14:10:38.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 -- 172.21.15.60:0/1288812437 learned_addr learned my addr 172.21.15.60:0/1288812437 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:38.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de54f6640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7de00ce390 0x7f7de015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de4cf5640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7de015aaa0 0x7f7de0158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 -- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7de015aaa0 msgr2=0x7f7de0158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7de015aaa0 0x7f7de0158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 -- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7de00ce390 msgr2=0x7f7de015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7de00ce390 0x7f7de015d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.011+0000 7f7de5cf7640 1 -- 172.21.15.60:0/1288812437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc0094d0 con 0x7f7de015ae80 2023-10-16T14:10:38.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.012+0000 7f7de5cf7640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 0x7f7de015dce0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8001560 tx=0x7f7dd800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:38.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.012+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8003da0 con 0x7f7de015ae80 2023-10-16T14:10:38.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.012+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd8005070 con 0x7f7de015ae80 2023-10-16T14:10:38.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.012+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8005420 con 0x7f7de015ae80 2023-10-16T14:10:38.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.012+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de015cc00 con 0x7f7de015ae80 2023-10-16T14:10:38.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.012+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de015e220 con 0x7f7de015ae80 2023-10-16T14:10:38.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.013+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de004e160 con 0x7f7de015ae80 2023-10-16T14:10:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.014+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7dd8005580 con 0x7f7de015ae80 2023-10-16T14:10:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.015+0000 7f7dd5ffb640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7dc406afe0 0x7f7dc406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.015+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dd8088fe0 con 0x7f7de015ae80 2023-10-16T14:10:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.016+0000 7f7de54f6640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7dc406afe0 0x7f7dc406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.017+0000 7f7de54f6640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7dc406afe0 0x7f7dc406d4a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc036000 tx=0x7f7ddc008030 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:38.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.019+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7dd80bbfe0 con 0x7f7de015ae80 2023-10-16T14:10:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.218+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7de006f8a0 con 0x7f7de015ae80 2023-10-16T14:10:38.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.219+0000 7f7dd5ffb640 1 -- 172.21.15.60:0/1288812437 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v231) v1 ==== 84+0+209 (secure 0 0 0) 0x7f7dd8017020 con 0x7f7de015ae80 2023-10-16T14:10:38.223 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:38.224 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:10:38.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:38.224 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:10:38.224 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:10:38.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:38.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:38.225 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 26, 2023-10-16T14:10:38.225 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "cephfs_metadata" 2023-10-16T14:10:38.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:38.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:38.225 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 27, 2023-10-16T14:10:38.226 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "cephfs_data" 2023-10-16T14:10:38.226 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:38.226 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:10:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7dc406afe0 msgr2=0x7f7dc406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7dc406afe0 0x7f7dc406d4a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc036000 tx=0x7f7ddc008030 comp rx=0 tx=0).stop 2023-10-16T14:10:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 msgr2=0x7f7de015dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 0x7f7de015dce0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8001560 tx=0x7f7dd800cc70 comp rx=0 tx=0).stop 2023-10-16T14:10:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 shutdown_connections 2023-10-16T14:10:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7dc406afe0 0x7f7dc406d4a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7de015aaa0 0x7f7de0158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7de015ae80 0x7f7de015dce0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 --2- 172.21.15.60:0/1288812437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7de00ce390 0x7f7de015d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 >> 172.21.15.60:0/1288812437 conn(0x7f7de00cb410 msgr2=0x7f7de00cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 shutdown_connections 2023-10-16T14:10:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.222+0000 7f7de775a640 1 -- 172.21.15.60:0/1288812437 wait complete. 2023-10-16T14:10:38.244 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T14:10:38.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T14:10:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.465+0000 7f710af80640 1 Processor -- start 2023-10-16T14:10:38.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.466+0000 7f710af80640 1 -- start start 2023-10-16T14:10:38.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.467+0000 7f710af80640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040b8480 0x7f71040cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.467+0000 7f710af80640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71040d0be0 0x7f71040cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.467+0000 7f710af80640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710415d150 con 0x7f710415cd70 2023-10-16T14:10:38.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.467+0000 7f710af80640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71040d1180 con 0x7f71040d0be0 2023-10-16T14:10:38.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.467+0000 7f710af80640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71040d1460 con 0x7f71040b8480 2023-10-16T14:10:38.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71040d0be0 0x7f71040cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71040d0be0 0x7f71040cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45358/0 (socket says 172.21.15.60:45358) 2023-10-16T14:10:38.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 -- 172.21.15.60:0/3244511447 learned_addr learned my addr 172.21.15.60:0/3244511447 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:38.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710977d640 1 --2- 172.21.15.60:0/3244511447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040b8480 0x7f71040cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 -- 172.21.15.60:0/3244511447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040b8480 msgr2=0x7f71040cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:38.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 --2- 172.21.15.60:0/3244511447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040b8480 0x7f71040cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 -- 172.21.15.60:0/3244511447 >> v1:172.21.15.144:6789/0 conn(0x7f710415cd70 legacy=0x7f71040d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:38.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.468+0000 7f710a77f640 1 -- 172.21.15.60:0/3244511447 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040d1740 con 0x7f71040d0be0 2023-10-16T14:10:38.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.469+0000 7f710a77f640 1 --2- 172.21.15.60:0/3244511447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71040d0be0 0x7f71040cdd00 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009870 tx=0x7f70fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dddd6abc707968 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:38.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.470+0000 7f7108f7c640 1 -- 172.21.15.60:0/3244511447 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc031640 con 0x7f71040d0be0 2023-10-16T14:10:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.470+0000 7f7108f7c640 1 -- 172.21.15.60:0/3244511447 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc0317a0 con 0x7f71040d0be0 2023-10-16T14:10:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.470+0000 7f7108f7c640 1 -- 172.21.15.60:0/3244511447 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc031b30 con 0x7f71040d0be0 2023-10-16T14:10:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.470+0000 7f710af80640 1 -- 172.21.15.60:0/3244511447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71040d0be0 msgr2=0x7f71040cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.470+0000 7f710af80640 1 --2- 172.21.15.60:0/3244511447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71040d0be0 0x7f71040cdd00 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009870 tx=0x7f70fc02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.471+0000 7f710af80640 1 -- 172.21.15.60:0/3244511447 shutdown_connections 2023-10-16T14:10:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.471+0000 7f710af80640 1 --2- 172.21.15.60:0/3244511447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71040d0be0 0x7f71040cdd00 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.471+0000 7f710af80640 1 --2- 172.21.15.60:0/3244511447 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040b8480 0x7f71040cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.471+0000 7f710af80640 1 -- 172.21.15.60:0/3244511447 >> 172.21.15.60:0/3244511447 conn(0x7f71040cf570 msgr2=0x7f71040cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.471+0000 7f710af80640 1 -- 172.21.15.60:0/3244511447 shutdown_connections 2023-10-16T14:10:38.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.471+0000 7f710af80640 1 -- 172.21.15.60:0/3244511447 wait complete. 2023-10-16T14:10:38.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.473+0000 7f710af80640 1 Processor -- start 2023-10-16T14:10:38.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.473+0000 7f710af80640 1 -- start start 2023-10-16T14:10:38.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710af80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71040b8480 0x7f71040c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710af80640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040d0be0 0x7f71040c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710af80640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 0x7f710415f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710af80640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71040c9400 con 0x7f710415cd70 2023-10-16T14:10:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710af80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104151ce0 con 0x7f71040b8480 2023-10-16T14:10:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710af80640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71040c74d0 con 0x7f71040d0be0 2023-10-16T14:10:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710a77f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 0x7f710415f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710a77f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 0x7f710415f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45368/0 (socket says 172.21.15.60:45368) 2023-10-16T14:10:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.474+0000 7f710a77f640 1 -- 172.21.15.60:0/2118860305 learned_addr learned my addr 172.21.15.60:0/2118860305 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f7109f7e640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71040b8480 0x7f71040c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710977d640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040d0be0 0x7f71040c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710a77f640 1 -- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040d0be0 msgr2=0x7f71040c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710a77f640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040d0be0 0x7f71040c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710a77f640 1 -- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71040b8480 msgr2=0x7f71040c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710a77f640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71040b8480 0x7f71040c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710a77f640 1 -- 172.21.15.60:0/2118860305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0094d0 con 0x7f710415cd70 2023-10-16T14:10:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f7109f7e640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71040b8480 0x7f71040c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.475+0000 7f710a77f640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 0x7f710415f340 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f70fc02b930 tx=0x7f70fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.476+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc03a9f0 con 0x7f710415cd70 2023-10-16T14:10:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.476+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc03ab70 con 0x7f710415cd70 2023-10-16T14:10:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.476+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71040c77b0 con 0x7f710415cd70 2023-10-16T14:10:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.476+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0443f0 con 0x7f710415cd70 2023-10-16T14:10:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.476+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710415ee20 con 0x7f710415cd70 2023-10-16T14:10:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.477+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710415f010 con 0x7f710415cd70 2023-10-16T14:10:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.478+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70fc02f050 con 0x7f710415cd70 2023-10-16T14:10:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.479+0000 7f71027fc640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f70f006af90 0x7f70f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.479+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70fc0ab7f0 con 0x7f710415cd70 2023-10-16T14:10:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.480+0000 7f7109f7e640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f70f006af90 0x7f70f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.481+0000 7f7109f7e640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f70f006af90 0x7f70f006d450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004520 tx=0x7f70f4009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.483+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70fc076670 con 0x7f710415cd70 2023-10-16T14:10:38.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.677+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f710404e9b0 con 0x7f710415cd70 2023-10-16T14:10:38.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.929+0000 7f71027fc640 1 -- 172.21.15.60:0/2118860305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v232) v1 ==== 144+0+0 (secure 0 0 0) 0x7f70fc044550 con 0x7f710415cd70 2023-10-16T14:10:38.933 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_metadata' created 2023-10-16T14:10:38.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f70f006af90 msgr2=0x7f70f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f70f006af90 0x7f70f006d450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004520 tx=0x7f70f4009040 comp rx=0 tx=0).stop 2023-10-16T14:10:38.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 msgr2=0x7f710415f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:38.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 0x7f710415f340 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f70fc02b930 tx=0x7f70fc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 shutdown_connections 2023-10-16T14:10:38.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f70f006af90 0x7f70f006d450 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f71040d0be0 0x7f71040c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f710415cd70 0x7f710415f340 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 --2- 172.21.15.60:0/2118860305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71040b8480 0x7f71040c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.933+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 >> 172.21.15.60:0/2118860305 conn(0x7f71040cf570 msgr2=0x7f71040d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.934+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 shutdown_connections 2023-10-16T14:10:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:38.934+0000 7f710af80640 1 -- 172.21.15.60:0/2118860305 wait complete. 2023-10-16T14:10:38.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:10:39.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.181+0000 7f1804a3f640 1 Processor -- start 2023-10-16T14:10:39.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.182+0000 7f1804a3f640 1 -- start start 2023-10-16T14:10:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.183+0000 7f1804a3f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f180015e260 0x7f18000ca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.183+0000 7f1804a3f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000cb0f0 0x7f180006d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.183+0000 7f1804a3f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180015d810 con 0x7f18000ceec0 2023-10-16T14:10:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.184+0000 7f1804a3f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180015d660 con 0x7f180015e260 2023-10-16T14:10:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.184+0000 7f1804a3f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18000cab50 con 0x7f18000cb0f0 2023-10-16T14:10:39.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.184+0000 7f17f7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f180015e260 0x7f18000ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:39.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.184+0000 7f17f7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f180015e260 0x7f18000ca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:40036/0 (socket says 172.21.15.60:40036) 2023-10-16T14:10:39.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.184+0000 7f17f7fff640 1 -- 172.21.15.60:0/733579643 learned_addr learned my addr 172.21.15.60:0/733579643 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:39.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.184+0000 7f17fed9d640 1 --2- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000cb0f0 0x7f180006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:39.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.185+0000 7f17f7fff640 1 -- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000cb0f0 msgr2=0x7f180006d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:39.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.185+0000 7f17f7fff640 1 --2- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000cb0f0 0x7f180006d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.185+0000 7f17f7fff640 1 -- 172.21.15.60:0/733579643 >> v1:172.21.15.144:6789/0 conn(0x7f18000ceec0 legacy=0x7f18000cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:39.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.185+0000 7f17f7fff640 1 -- 172.21.15.60:0/733579643 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180015e6c0 con 0x7f180015e260 2023-10-16T14:10:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.187+0000 7f17f7fff640 1 --2- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f18000ca610 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009870 tx=0x7f17e802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a455cd83693e5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.187+0000 7f17fdd9b640 1 -- 172.21.15.60:0/733579643 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8031640 con 0x7f180015e260 2023-10-16T14:10:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.187+0000 7f17fdd9b640 1 -- 172.21.15.60:0/733579643 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e80317a0 con 0x7f180015e260 2023-10-16T14:10:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.187+0000 7f17fdd9b640 1 -- 172.21.15.60:0/733579643 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8031b30 con 0x7f180015e260 2023-10-16T14:10:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 -- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 msgr2=0x7f18000ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 --2- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f18000ca610 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009870 tx=0x7f17e802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 -- 172.21.15.60:0/733579643 shutdown_connections 2023-10-16T14:10:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 --2- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000cb0f0 0x7f180006d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 --2- 172.21.15.60:0/733579643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f18000ca610 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 -- 172.21.15.60:0/733579643 >> 172.21.15.60:0/733579643 conn(0x7f18000cfaa0 msgr2=0x7f18000cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 -- 172.21.15.60:0/733579643 shutdown_connections 2023-10-16T14:10:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.188+0000 7f1804a3f640 1 -- 172.21.15.60:0/733579643 wait complete. 2023-10-16T14:10:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.190+0000 7f1804a3f640 1 Processor -- start 2023-10-16T14:10:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.190+0000 7f1804a3f640 1 -- start start 2023-10-16T14:10:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.191+0000 7f1804a3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 0x7f1800149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.191+0000 7f1804a3f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000ceec0 0x7f18000c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.191+0000 7f1804a3f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f180014a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.191+0000 7f1804a3f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180014a0d0 con 0x7f18000cb0f0 2023-10-16T14:10:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.191+0000 7f1804a3f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18000c70b0 con 0x7f180015e260 2023-10-16T14:10:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.191+0000 7f1804a3f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18000c7350 con 0x7f18000ceec0 2023-10-16T14:10:39.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 0x7f1800149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:39.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 0x7f1800149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37938/0 (socket says 172.21.15.60:37938) 2023-10-16T14:10:39.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 -- 172.21.15.60:0/1560588922 learned_addr learned my addr 172.21.15.60:0/1560588922 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:39.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17f7fff640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000ceec0 0x7f18000c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:39.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fed9d640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f180014a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:39.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 -- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000ceec0 msgr2=0x7f18000c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:39.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000ceec0 0x7f18000c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 -- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 msgr2=0x7f180014a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:39.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f180014a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.192+0000 7f17fe59c640 1 -- 172.21.15.60:0/1560588922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80094d0 con 0x7f18000cb0f0 2023-10-16T14:10:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.193+0000 7f17fe59c640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 0x7f1800149b90 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f17f000c400 tx=0x7f17f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.193+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f00052d0 con 0x7f18000cb0f0 2023-10-16T14:10:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.193+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f0005450 con 0x7f18000cb0f0 2023-10-16T14:10:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.193+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0007260 con 0x7f18000cb0f0 2023-10-16T14:10:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.193+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18000c7630 con 0x7f18000cb0f0 2023-10-16T14:10:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.193+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18000c7bf0 con 0x7f18000cb0f0 2023-10-16T14:10:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.194+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180004e160 con 0x7f18000cb0f0 2023-10-16T14:10:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.197+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f17f00073c0 con 0x7f18000cb0f0 2023-10-16T14:10:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.198+0000 7f17f6ffd640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17e006af90 0x7f17e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.198+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f17f0089610 con 0x7f18000cb0f0 2023-10-16T14:10:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.198+0000 7f17f7fff640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17e006af90 0x7f17e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:39.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.200+0000 7f17f7fff640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17e006af90 0x7f17e006d450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f17e8036000 tx=0x7f17e803c040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:39.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.202+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17f0054430 con 0x7f18000cb0f0 2023-10-16T14:10:39.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.400+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f18000716e0 con 0x7f18000cb0f0 2023-10-16T14:10:39.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.949+0000 7f17f6ffd640 1 -- 172.21.15.60:0/1560588922 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v233) v1 ==== 176+0+0 (secure 0 0 0) 0x7f17f00200d0 con 0x7f18000cb0f0 2023-10-16T14:10:39.953 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_data' created 2023-10-16T14:10:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17e006af90 msgr2=0x7f17e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:39.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17e006af90 0x7f17e006d450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f17e8036000 tx=0x7f17e803c040 comp rx=0 tx=0).stop 2023-10-16T14:10:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 msgr2=0x7f1800149b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:39.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 0x7f1800149b90 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f17f000c400 tx=0x7f17f000c8c0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 shutdown_connections 2023-10-16T14:10:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f17e006af90 0x7f17e006d450 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f18000ceec0 0x7f18000c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.953+0000 7f1804a3f640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f180015e260 0x7f180014a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.954+0000 7f1804a3f640 1 --2- 172.21.15.60:0/1560588922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18000cb0f0 0x7f1800149b90 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.954+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 >> 172.21.15.60:0/1560588922 conn(0x7f18000cfaa0 msgr2=0x7f180015ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.954+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 shutdown_connections 2023-10-16T14:10:39.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:39.954+0000 7f1804a3f640 1 -- 172.21.15.60:0/1560588922 wait complete. 2023-10-16T14:10:39.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T14:10:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.185+0000 7f6ab5ffc640 1 Processor -- start 2023-10-16T14:10:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.186+0000 7f6ab5ffc640 1 -- start start 2023-10-16T14:10:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.187+0000 7f6ab5ffc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159ab0 0x7f6ab00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.187+0000 7f6ab5ffc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ab006c140 0x7f6ab006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.187+0000 7f6ab5ffc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab0070360 con 0x7f6ab0159e90 2023-10-16T14:10:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.187+0000 7f6ab5ffc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab006cb10 con 0x7f6ab006c140 2023-10-16T14:10:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.187+0000 7f6ab5ffc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab015d890 con 0x7f6ab0159ab0 2023-10-16T14:10:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.188+0000 7f6aa7fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f6ab0159e90 0x7f6ab006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:48944/0 (socket says 172.21.15.60:48944) 2023-10-16T14:10:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.188+0000 7f6aa7fff640 1 -- 172.21.15.60:0/129640740 learned_addr learned my addr 172.21.15.60:0/129640740 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.188+0000 7f6aaf7fe640 1 --2- 172.21.15.60:0/129640740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159ab0 0x7f6ab00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.188+0000 7f6aaffff640 1 --2- 172.21.15.60:0/129640740 >> v2:172.21.15.144:3300/0 conn(0x7f6ab006c140 0x7f6ab006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.189+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/129640740 >> v2:172.21.15.144:3300/0 conn(0x7f6ab006c140 msgr2=0x7f6ab006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.189+0000 7f6aaf7fe640 1 --2- 172.21.15.60:0/129640740 >> v2:172.21.15.144:3300/0 conn(0x7f6ab006c140 0x7f6ab006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.189+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/129640740 >> v1:172.21.15.144:6789/0 conn(0x7f6ab0159e90 legacy=0x7f6ab006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.189+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/129640740 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab00cd1f0 con 0x7f6ab0159ab0 2023-10-16T14:10:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.189+0000 7f6aaffff640 1 --2- 172.21.15.60:0/129640740 >> v2:172.21.15.144:3300/0 conn(0x7f6ab006c140 0x7f6ab006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:10:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.191+0000 7f6aaf7fe640 1 --2- 172.21.15.60:0/129640740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159ab0 0x7f6ab00ce370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00097d0 tx=0x7f6aa002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9ee6bb5800c72ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:40.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.191+0000 7f6aaeffd640 1 -- 172.21.15.60:0/129640740 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00316f0 con 0x7f6ab0159ab0 2023-10-16T14:10:40.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.191+0000 7f6aaeffd640 1 -- 172.21.15.60:0/129640740 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0031850 con 0x7f6ab0159ab0 2023-10-16T14:10:40.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.191+0000 7f6aaeffd640 1 -- 172.21.15.60:0/129640740 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0031be0 con 0x7f6ab0159ab0 2023-10-16T14:10:40.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/129640740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159ab0 msgr2=0x7f6ab00ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:40.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/129640740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159ab0 0x7f6ab00ce370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00097d0 tx=0x7f6aa002b790 comp rx=0 tx=0).stop 2023-10-16T14:10:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/129640740 shutdown_connections 2023-10-16T14:10:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/129640740 >> v2:172.21.15.144:3300/0 conn(0x7f6ab006c140 0x7f6ab006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/129640740 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159ab0 0x7f6ab00ce370 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/129640740 >> 172.21.15.60:0/129640740 conn(0x7f6ab00ce9b0 msgr2=0x7f6ab00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/129640740 shutdown_connections 2023-10-16T14:10:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.192+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/129640740 wait complete. 2023-10-16T14:10:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.194+0000 7f6ab5ffc640 1 Processor -- start 2023-10-16T14:10:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.194+0000 7f6ab5ffc640 1 -- start start 2023-10-16T14:10:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.195+0000 7f6ab5ffc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 0x7f6ab014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.195+0000 7f6ab5ffc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab0159ab0 0x7f6ab014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.195+0000 7f6ab5ffc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159e90 0x7f6ab00b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.195+0000 7f6ab5ffc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0166ad0 con 0x7f6ab006c140 2023-10-16T14:10:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.195+0000 7f6ab5ffc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab01678e0 con 0x7f6ab0159ab0 2023-10-16T14:10:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.195+0000 7f6ab5ffc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0166fb0 con 0x7f6ab0159e90 2023-10-16T14:10:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 0x7f6ab014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 0x7f6ab014d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37946/0 (socket says 172.21.15.60:37946) 2023-10-16T14:10:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/1804605304 learned_addr learned my addr 172.21.15.60:0/1804605304 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aa7fff640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab0159ab0 0x7f6ab014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaffff640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159e90 0x7f6ab00b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159e90 msgr2=0x7f6ab00b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159e90 0x7f6ab00b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab0159ab0 msgr2=0x7f6ab014abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab0159ab0 0x7f6ab014abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aaf7fe640 1 -- 172.21.15.60:0/1804605304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8009a70 con 0x7f6ab006c140 2023-10-16T14:10:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.196+0000 7f6aa7fff640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab0159ab0 0x7f6ab014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:10:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.197+0000 7f6aaf7fe640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 0x7f6ab014d8f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f6aa002bda0 tx=0x7f6aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.197+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0034680 con 0x7f6ab006c140 2023-10-16T14:10:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.197+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0034800 con 0x7f6ab006c140 2023-10-16T14:10:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.197+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa0009400 con 0x7f6ab006c140 2023-10-16T14:10:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.197+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0034bb0 con 0x7f6ab006c140 2023-10-16T14:10:40.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.197+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0167570 con 0x7f6ab006c140 2023-10-16T14:10:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.199+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab004e160 con 0x7f6ab006c140 2023-10-16T14:10:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.200+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6aa003b040 con 0x7f6ab006c140 2023-10-16T14:10:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.201+0000 7f6aacff9640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a9006af90 0x7f6a9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.201+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f6aa00ab8b0 con 0x7f6ab006c140 2023-10-16T14:10:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.202+0000 7f6aa7fff640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a9006af90 0x7f6a9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.203+0000 7f6aa7fff640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a9006af90 0x7f6a9006d450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00c2370 tx=0x7f6a98009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:40.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.205+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6aa0076420 con 0x7f6ab006c140 2023-10-16T14:10:40.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.440+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f6ab00ccb60 con 0x7f6ab006c140 2023-10-16T14:10:40.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.977+0000 7f6aacff9640 1 -- 172.21.15.60:0/1804605304 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v115) v1 ==== 183+0+0 (secure 0 0 0) 0x7f6aa0079a40 con 0x7f6ab006c140 2023-10-16T14:10:40.988 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 28 and data pool 29 2023-10-16T14:10:40.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a9006af90 msgr2=0x7f6a9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:40.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a9006af90 0x7f6a9006d450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00c2370 tx=0x7f6a98009040 comp rx=0 tx=0).stop 2023-10-16T14:10:40.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 msgr2=0x7f6ab014d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:40.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 0x7f6ab014d8f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f6aa002bda0 tx=0x7f6aa00046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 shutdown_connections 2023-10-16T14:10:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f6a9006af90 0x7f6a9006d450 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00c2370 tx=0x7f6a98009040 comp rx=0 tx=0).stop 2023-10-16T14:10:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6ab0159e90 0x7f6ab00b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ab0159ab0 0x7f6ab014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 --2- 172.21.15.60:0/1804605304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab006c140 0x7f6ab014d8f0 secure :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f6aa002bda0 tx=0x7f6aa00046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.987+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 >> 172.21.15.60:0/1804605304 conn(0x7f6ab00ce9b0 msgr2=0x7f6ab00b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.989+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 shutdown_connections 2023-10-16T14:10:40.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:40.990+0000 7f6ab5ffc640 1 -- 172.21.15.60:0/1804605304 wait complete. 2023-10-16T14:10:41.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:41.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.208+0000 7feb173fb640 1 Processor -- start 2023-10-16T14:10:41.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.209+0000 7feb173fb640 1 -- start start 2023-10-16T14:10:41.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb173fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb1006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb173fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb100b0b20 0x7feb10164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb173fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10071450 con 0x7feb10070fd0 2023-10-16T14:10:41.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb173fb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb100cc600 con 0x7feb100b0b20 2023-10-16T14:10:41.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb173fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb100cc8e0 con 0x7feb100adb50 2023-10-16T14:10:41.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb15998640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb100b0b20 0x7feb10164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb14996640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feb10070fd0 0x7feb100b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37838/0 (socket says 172.21.15.60:37838) 2023-10-16T14:10:41.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.210+0000 7feb14996640 1 -- 172.21.15.60:0/1760506193 learned_addr learned my addr 172.21.15.60:0/1760506193 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:41.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.211+0000 7feb15197640 1 --2- 172.21.15.60:0/1760506193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb1006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.211+0000 7feb15998640 1 -- 172.21.15.60:0/1760506193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 msgr2=0x7feb1006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:41.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.211+0000 7feb15998640 1 --2- 172.21.15.60:0/1760506193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb1006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.211+0000 7feb15998640 1 -- 172.21.15.60:0/1760506193 >> v1:172.21.15.60:6789/0 conn(0x7feb10070fd0 legacy=0x7feb100b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.211+0000 7feb15998640 1 -- 172.21.15.60:0/1760506193 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100ccbc0 con 0x7feb100b0b20 2023-10-16T14:10:41.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.212+0000 7feb15998640 1 --2- 172.21.15.60:0/1760506193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb100b0b20 0x7feb10164f60 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7feb08009d90 tx=0x7feb0802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a1420f3d6c71b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.212+0000 7feb03fff640 1 -- 172.21.15.60:0/1760506193 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0802cc60 con 0x7feb100b0b20 2023-10-16T14:10:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.212+0000 7feb03fff640 1 -- 172.21.15.60:0/1760506193 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0802cdc0 con 0x7feb100b0b20 2023-10-16T14:10:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.212+0000 7feb03fff640 1 -- 172.21.15.60:0/1760506193 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08034660 con 0x7feb100b0b20 2023-10-16T14:10:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 -- 172.21.15.60:0/1760506193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb100b0b20 msgr2=0x7feb10164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 --2- 172.21.15.60:0/1760506193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb100b0b20 0x7feb10164f60 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7feb08009d90 tx=0x7feb0802beb0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 -- 172.21.15.60:0/1760506193 shutdown_connections 2023-10-16T14:10:41.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 --2- 172.21.15.60:0/1760506193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb100b0b20 0x7feb10164f60 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 --2- 172.21.15.60:0/1760506193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb1006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 -- 172.21.15.60:0/1760506193 >> 172.21.15.60:0/1760506193 conn(0x7feb100cb410 msgr2=0x7feb100cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:41.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 -- 172.21.15.60:0/1760506193 shutdown_connections 2023-10-16T14:10:41.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.213+0000 7feb173fb640 1 -- 172.21.15.60:0/1760506193 wait complete. 2023-10-16T14:10:41.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.215+0000 7feb173fb640 1 Processor -- start 2023-10-16T14:10:41.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.215+0000 7feb173fb640 1 -- start start 2023-10-16T14:10:41.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb173fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 0x7feb1015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb173fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb100b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb173fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb100b0b20 0x7feb100bd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb173fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1015c990 con 0x7feb10070fd0 2023-10-16T14:10:41.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb173fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb100bf7e0 con 0x7feb100b0b20 2023-10-16T14:10:41.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb173fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb100b3870 con 0x7feb100adb50 2023-10-16T14:10:41.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb15197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 0x7feb1015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb15197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 0x7feb1015c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37972/0 (socket says 172.21.15.60:37972) 2023-10-16T14:10:41.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.216+0000 7feb15197640 1 -- 172.21.15.60:0/3552978237 learned_addr learned my addr 172.21.15.60:0/3552978237 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:41.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb15197640 1 -- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 msgr2=0x7feb100b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb14996640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb100b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb15197640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb100b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb15197640 1 -- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb100b0b20 msgr2=0x7feb100bd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb15197640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb100b0b20 0x7feb100bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb15197640 1 -- 172.21.15.60:0/3552978237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb08009a70 con 0x7feb10070fd0 2023-10-16T14:10:41.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb14996640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb100b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:41.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.217+0000 7feb15197640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 0x7feb1015c450 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00d830 tx=0x7feb0c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:41.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.218+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00b6c0 con 0x7feb10070fd0 2023-10-16T14:10:41.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.218+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c00b820 con 0x7feb10070fd0 2023-10-16T14:10:41.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.218+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00bbb0 con 0x7feb10070fd0 2023-10-16T14:10:41.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.218+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb100b2fe0 con 0x7feb10070fd0 2023-10-16T14:10:41.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.218+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb100b3590 con 0x7feb10070fd0 2023-10-16T14:10:41.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.219+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb100c20d0 con 0x7feb10070fd0 2023-10-16T14:10:41.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.220+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb0c01e460 con 0x7feb10070fd0 2023-10-16T14:10:41.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.221+0000 7feb01ffb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7feaf406af90 0x7feaf406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.221+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6573+0+0 (secure 0 0 0) 0x7feb0c08a2b0 con 0x7feb10070fd0 2023-10-16T14:10:41.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.222+0000 7feb14996640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7feaf406af90 0x7feaf406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.223+0000 7feb14996640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7feaf406af90 0x7feaf406d450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb100b2020 tx=0x7feb04009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:41.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.226+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb0c054ee0 con 0x7feb10070fd0 2023-10-16T14:10:41.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.426+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb1004e160 con 0x7feb10070fd0 2023-10-16T14:10:41.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.430+0000 7feb01ffb640 1 -- 172.21.15.60:0/3552978237 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+20308 (secure 0 0 0) 0x7feb0c058500 con 0x7feb10070fd0 2023-10-16T14:10:41.434 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:41.435 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":234,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:40.442863+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:41.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7feaf406af90 msgr2=0x7feaf406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7feaf406af90 0x7feaf406d450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb100b2020 tx=0x7feb04009040 comp rx=0 tx=0).stop 2023-10-16T14:10:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 msgr2=0x7feb1015c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 0x7feb1015c450 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00d830 tx=0x7feb0c00dcf0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 shutdown_connections 2023-10-16T14:10:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7feaf406af90 0x7feaf406d450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feb100adb50 0x7feb100b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb100b0b20 0x7feb100bd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 --2- 172.21.15.60:0/3552978237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb10070fd0 0x7feb1015c450 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.432+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 >> 172.21.15.60:0/3552978237 conn(0x7feb100cb410 msgr2=0x7feb1006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.433+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 shutdown_connections 2023-10-16T14:10:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.433+0000 7feb173fb640 1 -- 172.21.15.60:0/3552978237 wait complete. 2023-10-16T14:10:41.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:41.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.670+0000 7f02ff59e640 1 Processor -- start 2023-10-16T14:10:41.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.670+0000 7f02ff59e640 1 -- start start 2023-10-16T14:10:41.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.671+0000 7f02ff59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03000ca0a0 0x7f030006c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.671+0000 7f02ff59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030015d910 con 0x7f0300070b60 2023-10-16T14:10:41.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.671+0000 7f02ff59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030015dbf0 con 0x7f0300159950 2023-10-16T14:10:41.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.671+0000 7f02ff59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030015e490 con 0x7f03000ca0a0 2023-10-16T14:10:41.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.672+0000 7f02fed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03000ca0a0 0x7f030006c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.672+0000 7f02fdd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0300070b60 0x7f030006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37846/0 (socket says 172.21.15.60:37846) 2023-10-16T14:10:41.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.672+0000 7f02fdd9b640 1 -- 172.21.15.60:0/2156806399 learned_addr learned my addr 172.21.15.60:0/2156806399 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:41.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.672+0000 7f02fed9d640 1 -- 172.21.15.60:0/2156806399 >> v1:172.21.15.144:6789/0 conn(0x7f0300159950 legacy=0x7f0300070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:41.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.672+0000 7f02fed9d640 1 -- 172.21.15.60:0/2156806399 >> v1:172.21.15.60:6789/0 conn(0x7f0300070b60 legacy=0x7f030006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.672+0000 7f02fed9d640 1 -- 172.21.15.60:0/2156806399 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030015ded0 con 0x7f03000ca0a0 2023-10-16T14:10:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.673+0000 7f02fed9d640 1 --2- 172.21.15.60:0/2156806399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03000ca0a0 0x7f030006c810 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f02f80097a0 tx=0x7f02f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1891a355a0ae93b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02fd59a640 1 -- 172.21.15.60:0/2156806399 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f802e070 con 0x7f03000ca0a0 2023-10-16T14:10:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02fd59a640 1 -- 172.21.15.60:0/2156806399 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f80315a0 con 0x7f03000ca0a0 2023-10-16T14:10:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02fd59a640 1 -- 172.21.15.60:0/2156806399 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8031930 con 0x7f03000ca0a0 2023-10-16T14:10:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 -- 172.21.15.60:0/2156806399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03000ca0a0 msgr2=0x7f030006c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 --2- 172.21.15.60:0/2156806399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03000ca0a0 0x7f030006c810 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f02f80097a0 tx=0x7f02f802b840 comp rx=0 tx=0).stop 2023-10-16T14:10:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 -- 172.21.15.60:0/2156806399 shutdown_connections 2023-10-16T14:10:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 --2- 172.21.15.60:0/2156806399 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03000ca0a0 0x7f030006c810 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 -- 172.21.15.60:0/2156806399 >> 172.21.15.60:0/2156806399 conn(0x7f03000ce6a0 msgr2=0x7f03000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:41.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 -- 172.21.15.60:0/2156806399 shutdown_connections 2023-10-16T14:10:41.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.674+0000 7f02ff59e640 1 -- 172.21.15.60:0/2156806399 wait complete. 2023-10-16T14:10:41.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.676+0000 7f02ff59e640 1 Processor -- start 2023-10-16T14:10:41.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.676+0000 7f02ff59e640 1 -- start start 2023-10-16T14:10:41.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.677+0000 7f02ff59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0300070b60 0x7f0300158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.677+0000 7f02ff59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03000ca0a0 0x7f03000b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.677+0000 7f02ff59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 0x7f0300143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.677+0000 7f02ff59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03000b2560 con 0x7f0300159950 2023-10-16T14:10:41.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.677+0000 7f02ff59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001512d0 con 0x7f0300070b60 2023-10-16T14:10:41.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.677+0000 7f02ff59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0300143fd0 con 0x7f03000ca0a0 2023-10-16T14:10:41.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 0x7f0300143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 0x7f0300143a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37994/0 (socket says 172.21.15.60:37994) 2023-10-16T14:10:41.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 -- 172.21.15.60:0/3114344962 learned_addr learned my addr 172.21.15.60:0/3114344962 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:41.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fdd9b640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03000ca0a0 0x7f03000b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fe59c640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0300070b60 0x7f0300158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 -- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03000ca0a0 msgr2=0x7f03000b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03000ca0a0 0x7f03000b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 -- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0300070b60 msgr2=0x7f0300158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0300070b60 0x7f0300158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.678+0000 7f02fed9d640 1 -- 172.21.15.60:0/3114344962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f8009400 con 0x7f0300159950 2023-10-16T14:10:41.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.679+0000 7f02fed9d640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 0x7f0300143a90 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f02f80097a0 tx=0x7f02f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:41.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.679+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8032840 con 0x7f0300159950 2023-10-16T14:10:41.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.679+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f80329c0 con 0x7f0300159950 2023-10-16T14:10:41.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.679+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8032d70 con 0x7f0300159950 2023-10-16T14:10:41.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.679+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03000b1950 con 0x7f0300159950 2023-10-16T14:10:41.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.679+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03000b1f10 con 0x7f0300159950 2023-10-16T14:10:41.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.680+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030006cc80 con 0x7f0300159950 2023-10-16T14:10:41.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.681+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f02f803b3f0 con 0x7f0300159950 2023-10-16T14:10:41.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.682+0000 7f02f6ffd640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f02e006af90 0x7f02e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:41.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.682+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f02f80abd40 con 0x7f0300159950 2023-10-16T14:10:41.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.683+0000 7f02fe59c640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f02e006af90 0x7f02e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:41.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.684+0000 7f02fe59c640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f02e006af90 0x7f02e006d450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009f20 tx=0x7f02f00094d0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:41.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.687+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02f80767d0 con 0x7f0300159950 2023-10-16T14:10:41.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.890+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f030004e9b0 con 0x7f0300159950 2023-10-16T14:10:41.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.892+0000 7f02f6ffd640 1 -- 172.21.15.60:0/3114344962 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+20308 (secure 0 0 0) 0x7f02f8079df0 con 0x7f0300159950 2023-10-16T14:10:41.897 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:41.897 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":234,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:40.442863+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:41.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.895+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f02e006af90 msgr2=0x7f02e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.895+0000 7f02ff59e640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f02e006af90 0x7f02e006d450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009f20 tx=0x7f02f00094d0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.895+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 msgr2=0x7f0300143a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:41.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.895+0000 7f02ff59e640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 0x7f0300143a90 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f02f80097a0 tx=0x7f02f80046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 shutdown_connections 2023-10-16T14:10:41.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f02e006af90 0x7f02e006d450 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f03000ca0a0 0x7f03000b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0300159950 0x7f0300143a90 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 --2- 172.21.15.60:0/3114344962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0300070b60 0x7f0300158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:41.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 >> 172.21.15.60:0/3114344962 conn(0x7f03000ce6a0 msgr2=0x7f0300070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:41.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 shutdown_connections 2023-10-16T14:10:41.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:41.896+0000 7f02ff59e640 1 -- 172.21.15.60:0/3114344962 wait complete. 2023-10-16T14:10:41.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T14:10:42.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.145+0000 7f276f283640 1 Processor -- start 2023-10-16T14:10:42.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.145+0000 7f276f283640 1 -- start start 2023-10-16T14:10:42.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276f283640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f276806a4b0 0x7f27680ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276f283640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27680cb030 con 0x7f27680c9ca0 2023-10-16T14:10:42.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276f283640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27680c91c0 con 0x7f276815ad70 2023-10-16T14:10:42.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276f283640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27680c94a0 con 0x7f276806a4b0 2023-10-16T14:10:42.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276d820640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f27680c9ca0 0x7f27680ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37850/0 (socket says 172.21.15.60:37850) 2023-10-16T14:10:42.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276d820640 1 -- 172.21.15.60:0/1415805402 learned_addr learned my addr 172.21.15.60:0/1415805402 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:42.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.147+0000 7f276c81e640 1 --2- 172.21.15.60:0/1415805402 >> v2:172.21.15.144:3300/0 conn(0x7f276806a4b0 0x7f27680ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:42.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251352287 0 0) 0x7f27680cb030 con 0x7f27680c9ca0 2023-10-16T14:10:42.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2754003520 con 0x7f27680c9ca0 2023-10-16T14:10:42.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1764176370 0 0) 0x7f2754003520 con 0x7f27680c9ca0 2023-10-16T14:10:42.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 >> v2:172.21.15.144:3300/0 conn(0x7f276806a4b0 msgr2=0x7f27680ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:42.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 --2- 172.21.15.60:0/1415805402 >> v2:172.21.15.144:3300/0 conn(0x7f276806a4b0 0x7f27680ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 >> v1:172.21.15.144:6789/0 conn(0x7f276815ad70 legacy=0x7f2768069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:42.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27680c9780 con 0x7f27680c9ca0 2023-10-16T14:10:42.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f27600019f0 con 0x7f27680c9ca0 2023-10-16T14:10:42.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.148+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 >> v1:172.21.15.60:6789/0 conn(0x7f27680c9ca0 legacy=0x7f27680ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:42.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f275ffff640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27540039c0 0x7f2754003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f275ffff640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27540047c0 0x7f2754004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f275ffff640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2754005180 0x7f275400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27680c94a0 con 0x7f27540047c0 2023-10-16T14:10:42.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275400c080 con 0x7f27540039c0 2023-10-16T14:10:42.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275400c240 con 0x7f2754005180 2023-10-16T14:10:42.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276c81e640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27540047c0 0x7f2754004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:42.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276d01f640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27540039c0 0x7f2754003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:42.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276d820640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2754005180 0x7f275400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:42.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276c81e640 1 -- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2754005180 msgr2=0x7f275400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:42.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276c81e640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2754005180 0x7f275400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276c81e640 1 -- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27540039c0 msgr2=0x7f2754003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:42.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276c81e640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27540039c0 0x7f2754003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.149+0000 7f276c81e640 1 -- 172.21.15.60:0/1415805402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2754003520 con 0x7f27540047c0 2023-10-16T14:10:42.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.150+0000 7f276d820640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2754005180 0x7f275400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:42.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.150+0000 7f276d01f640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27540039c0 0x7f2754003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:42.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.150+0000 7f276d820640 1 -- 172.21.15.60:0/1415805402 reap_dead start 2023-10-16T14:10:42.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.151+0000 7f276c81e640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27540047c0 0x7f2754004c40 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f2758009ac0 tx=0x7f27540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940c76e8ac75bc1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:42.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.151+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275802c6d0 con 0x7f27540047c0 2023-10-16T14:10:42.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.151+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275802c830 con 0x7f27540047c0 2023-10-16T14:10:42.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.151+0000 7f275ffff640 1 -- 172.21.15.60:0/1415805402 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275802cbc0 con 0x7f27540047c0 2023-10-16T14:10:42.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.151+0000 7f276f283640 1 -- 172.21.15.60:0/1415805402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27540047c0 msgr2=0x7f2754004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:42.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.151+0000 7f276f283640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27540047c0 0x7f2754004c40 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f2758009ac0 tx=0x7f27540057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.152+0000 7f276f283640 1 -- 172.21.15.60:0/1415805402 shutdown_connections 2023-10-16T14:10:42.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.152+0000 7f276f283640 1 --2- 172.21.15.60:0/1415805402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27540047c0 0x7f2754004c40 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.152+0000 7f276f283640 1 -- 172.21.15.60:0/1415805402 >> 172.21.15.60:0/1415805402 conn(0x7f27680ce9b0 msgr2=0x7f27680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:42.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.152+0000 7f276f283640 1 -- 172.21.15.60:0/1415805402 shutdown_connections 2023-10-16T14:10:42.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.152+0000 7f276f283640 1 -- 172.21.15.60:0/1415805402 wait complete. 2023-10-16T14:10:42.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.153+0000 7f276f283640 1 Processor -- start 2023-10-16T14:10:42.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.154+0000 7f276f283640 1 -- start start 2023-10-16T14:10:42.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.154+0000 7f276f283640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27540047c0 0x7f27680c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276f283640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 0x7f276814df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276f283640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f276815c460 0x7f276815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276f283640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27680ade30 con 0x7f27680ac890 2023-10-16T14:10:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276f283640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276815ce20 con 0x7f276815c460 2023-10-16T14:10:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276f283640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27680ce590 con 0x7f27540047c0 2023-10-16T14:10:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 0x7f276814df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 0x7f276814df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38002/0 (socket says 172.21.15.60:38002) 2023-10-16T14:10:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 -- 172.21.15.60:0/3307473270 learned_addr learned my addr 172.21.15.60:0/3307473270 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 -- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27540047c0 msgr2=0x7f27680c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27540047c0 0x7f27680c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 -- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f276815c460 msgr2=0x7f276815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f276815c460 0x7f276815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.155+0000 7f276c81e640 1 -- 172.21.15.60:0/3307473270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276814e480 con 0x7f27680ac890 2023-10-16T14:10:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.156+0000 7f276c81e640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 0x7f276814df40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f275802c270 tx=0x7f2758004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.156+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758003810 con 0x7f27680ac890 2023-10-16T14:10:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.156+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2758003970 con 0x7f27680ac890 2023-10-16T14:10:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.156+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758003d00 con 0x7f27680ac890 2023-10-16T14:10:42.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.156+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27680ae250 con 0x7f27680ac890 2023-10-16T14:10:42.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.156+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27680acd10 con 0x7f27680ac890 2023-10-16T14:10:42.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.157+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276804e160 con 0x7f27680ac890 2023-10-16T14:10:42.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.158+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2758034440 con 0x7f27680ac890 2023-10-16T14:10:42.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.159+0000 7f275dffb640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f274c06af90 0x7f274c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:42.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.159+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f27580abd80 con 0x7f27680ac890 2023-10-16T14:10:42.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.160+0000 7f276d01f640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f274c06af90 0x7f274c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:42.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.161+0000 7f276d01f640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f274c06af90 0x7f274c06d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27640016d0 tx=0x7f2764009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:42.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.164+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2758076b90 con 0x7f27680ac890 2023-10-16T14:10:42.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.405+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f27680acfe0 con 0x7f27680ac890 2023-10-16T14:10:42.771 INFO:tasks.ceph.mds.c.smithi060.stderr:2023-10-16T14:10:42.767+0000 7f2d1d243640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:10:43.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:42.997+0000 7f275dffb640 1 -- 172.21.15.60:0/3307473270 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v117) v1 ==== 121+0+0 (secure 0 0 0) 0x7f275804c0c0 con 0x7f27680ac890 2023-10-16T14:10:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f274c06af90 msgr2=0x7f274c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f274c06af90 0x7f274c06d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27640016d0 tx=0x7f2764009040 comp rx=0 tx=0).stop 2023-10-16T14:10:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 msgr2=0x7f276814df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 0x7f276814df40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f275802c270 tx=0x7f2758004730 comp rx=0 tx=0).stop 2023-10-16T14:10:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 shutdown_connections 2023-10-16T14:10:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f274c06af90 0x7f274c06d450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f276815c460 0x7f276815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27680ac890 0x7f276814df40 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 --2- 172.21.15.60:0/3307473270 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f27540047c0 0x7f27680c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 >> 172.21.15.60:0/3307473270 conn(0x7f27680ce9b0 msgr2=0x7f2768159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 shutdown_connections 2023-10-16T14:10:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.002+0000 7f276f283640 1 -- 172.21.15.60:0/3307473270 wait complete. 2023-10-16T14:10:43.029 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:10:43.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T14:10:43.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.261+0000 7f148e7a9640 1 Processor -- start 2023-10-16T14:10:43.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.262+0000 7f148e7a9640 1 -- start start 2023-10-16T14:10:43.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.263+0000 7f148e7a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148815aee0 0x7f14880b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:43.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.263+0000 7f148e7a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14880ae0d0 0x7f14880cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:43.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.263+0000 7f148e7a9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148815b2c0 con 0x7f14880ce820 2023-10-16T14:10:43.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.263+0000 7f148e7a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14880ae5d0 con 0x7f14880ae0d0 2023-10-16T14:10:43.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.263+0000 7f148e7a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880ae8b0 con 0x7f148815aee0 2023-10-16T14:10:43.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14880ae0d0 0x7f14880cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:43.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f1487fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148815aee0 0x7f14880b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:43.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14880ae0d0 0x7f14880cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:40114/0 (socket says 172.21.15.60:40114) 2023-10-16T14:10:43.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 -- 172.21.15.60:0/2870174566 learned_addr learned my addr 172.21.15.60:0/2870174566 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:43.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 -- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148815aee0 msgr2=0x7f14880b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:43.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 --2- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148815aee0 0x7f14880b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 -- 172.21.15.60:0/2870174566 >> v1:172.21.15.144:6789/0 conn(0x7f14880ce820 legacy=0x7f14880af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:43.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.264+0000 7f147ffff640 1 -- 172.21.15.60:0/2870174566 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880aeb90 con 0x7f14880ae0d0 2023-10-16T14:10:43.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.266+0000 7f147ffff640 1 --2- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ae0d0 0x7f14880cf020 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1470009870 tx=0x7f147002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bea96d604ff4d028 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:43.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.266+0000 7f14877fe640 1 -- 172.21.15.60:0/2870174566 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147002f830 con 0x7f14880ae0d0 2023-10-16T14:10:43.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f14877fe640 1 -- 172.21.15.60:0/2870174566 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147002f990 con 0x7f14880ae0d0 2023-10-16T14:10:43.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f14877fe640 1 -- 172.21.15.60:0/2870174566 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147002fd20 con 0x7f14880ae0d0 2023-10-16T14:10:43.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 -- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ae0d0 msgr2=0x7f14880cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:43.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 --2- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ae0d0 0x7f14880cf020 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1470009870 tx=0x7f147002ba10 comp rx=0 tx=0).stop 2023-10-16T14:10:43.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 -- 172.21.15.60:0/2870174566 shutdown_connections 2023-10-16T14:10:43.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 --2- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ae0d0 0x7f14880cf020 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 --2- 172.21.15.60:0/2870174566 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f148815aee0 0x7f14880b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 -- 172.21.15.60:0/2870174566 >> 172.21.15.60:0/2870174566 conn(0x7f14880d0ad0 msgr2=0x7f14880d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:43.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 -- 172.21.15.60:0/2870174566 shutdown_connections 2023-10-16T14:10:43.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.267+0000 7f148e7a9640 1 -- 172.21.15.60:0/2870174566 wait complete. 2023-10-16T14:10:43.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.269+0000 7f148e7a9640 1 Processor -- start 2023-10-16T14:10:43.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.269+0000 7f148e7a9640 1 -- start start 2023-10-16T14:10:43.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.270+0000 7f148e7a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880ae0d0 0x7f1488150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:43.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.270+0000 7f148e7a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ce820 0x7f1488151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:43.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.270+0000 7f148e7a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 0x7f1488151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:43.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.270+0000 7f148e7a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cac50 con 0x7f148815aee0 2023-10-16T14:10:43.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.270+0000 7f148e7a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880c9fd0 con 0x7f14880ce820 2023-10-16T14:10:43.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.270+0000 7f148e7a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880ca2b0 con 0x7f14880ae0d0 2023-10-16T14:10:43.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.271+0000 7f148cd46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 0x7f1488151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:43.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.271+0000 7f148cd46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 0x7f1488151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38008/0 (socket says 172.21.15.60:38008) 2023-10-16T14:10:43.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.271+0000 7f148cd46640 1 -- 172.21.15.60:0/3454731044 learned_addr learned my addr 172.21.15.60:0/3454731044 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:43.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.271+0000 7f1487fff640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880ae0d0 0x7f1488150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:43.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.271+0000 7f147ffff640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ce820 0x7f1488151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:43.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.271+0000 7f148cd46640 1 -- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880ae0d0 msgr2=0x7f1488150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:43.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148cd46640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880ae0d0 0x7f1488150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148cd46640 1 -- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ce820 msgr2=0x7f1488151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:43.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148cd46640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ce820 0x7f1488151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:43.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148cd46640 1 -- 172.21.15.60:0/3454731044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700094d0 con 0x7f148815aee0 2023-10-16T14:10:43.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f147ffff640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ce820 0x7f1488151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:10:43.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148cd46640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 0x7f1488151750 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f148000c730 tx=0x7f148000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:43.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480003b70 con 0x7f148815aee0 2023-10-16T14:10:43.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480003cd0 con 0x7f148815aee0 2023-10-16T14:10:43.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000aa90 con 0x7f148815aee0 2023-10-16T14:10:43.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14880ca590 con 0x7f148815aee0 2023-10-16T14:10:43.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.272+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14880c7dd0 con 0x7f148815aee0 2023-10-16T14:10:43.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.274+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14880cad90 con 0x7f148815aee0 2023-10-16T14:10:43.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.275+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f148000abf0 con 0x7f148815aee0 2023-10-16T14:10:43.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.275+0000 7f14857fa640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f146806af90 0x7f146806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:43.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.276+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f1480089930 con 0x7f148815aee0 2023-10-16T14:10:43.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.276+0000 7f1487fff640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f146806af90 0x7f146806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:43.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.277+0000 7f1487fff640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f146806af90 0x7f146806d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1478009a90 tx=0x7f1478009110 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.280+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14880cad90 con 0x7f148815aee0 2023-10-16T14:10:43.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:43.518+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f14880cad90 con 0x7f148815aee0 2023-10-16T14:10:44.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.001+0000 7f14857fa640 1 -- 172.21.15.60:0/3454731044 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v118) v1 ==== 125+0+0 (secure 0 0 0) 0x7f14880cad90 con 0x7f148815aee0 2023-10-16T14:10:44.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f146806af90 msgr2=0x7f146806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f146806af90 0x7f146806d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1478009a90 tx=0x7f1478009110 comp rx=0 tx=0).stop 2023-10-16T14:10:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 msgr2=0x7f1488151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 0x7f1488151750 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f148000c730 tx=0x7f148000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 shutdown_connections 2023-10-16T14:10:44.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f146806af90 0x7f146806d450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f148815aee0 0x7f1488151750 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14880ce820 0x7f1488151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 --2- 172.21.15.60:0/3454731044 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14880ae0d0 0x7f1488150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.005+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 >> 172.21.15.60:0/3454731044 conn(0x7f14880d0ad0 msgr2=0x7f14880af570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:44.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.006+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 shutdown_connections 2023-10-16T14:10:44.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.006+0000 7f148e7a9640 1 -- 172.21.15.60:0/3454731044 wait complete. 2023-10-16T14:10:44.030 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T14:10:44.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:44.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.254+0000 7fc474850640 1 Processor -- start 2023-10-16T14:10:44.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.254+0000 7fc474850640 1 -- start start 2023-10-16T14:10:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc474850640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc47015dde0 0x7fc4700709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc474850640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc470070f20 0x7fc4700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc474850640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc470160a90 con 0x7fc47015c9f0 2023-10-16T14:10:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc474850640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc470071350 con 0x7fc470070f20 2023-10-16T14:10:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc474850640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4700cdec0 con 0x7fc47015dde0 2023-10-16T14:10:44.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc46f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc470070f20 0x7fc4700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc46f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc470070f20 0x7fc4700ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38020/0 (socket says 172.21.15.60:38020) 2023-10-16T14:10:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.256+0000 7fc46f59e640 1 -- 172.21.15.60:0/482069449 learned_addr learned my addr 172.21.15.60:0/482069449 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.257+0000 7fc46e59c640 1 --2- 172.21.15.60:0/482069449 >> v2:172.21.15.144:3300/0 conn(0x7fc47015dde0 0x7fc4700709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.257+0000 7fc46f59e640 1 -- 172.21.15.60:0/482069449 >> v2:172.21.15.144:3300/0 conn(0x7fc47015dde0 msgr2=0x7fc4700709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:44.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.257+0000 7fc46f59e640 1 --2- 172.21.15.60:0/482069449 >> v2:172.21.15.144:3300/0 conn(0x7fc47015dde0 0x7fc4700709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.257+0000 7fc46f59e640 1 -- 172.21.15.60:0/482069449 >> v1:172.21.15.144:6789/0 conn(0x7fc47015c9f0 legacy=0x7fc4700c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:44.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.257+0000 7fc46f59e640 1 -- 172.21.15.60:0/482069449 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700cd900 con 0x7fc470070f20 2023-10-16T14:10:44.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.257+0000 7fc46e59c640 1 --2- 172.21.15.60:0/482069449 >> v2:172.21.15.144:3300/0 conn(0x7fc47015dde0 0x7fc4700709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:44.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.258+0000 7fc46f59e640 1 --2- 172.21.15.60:0/482069449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc470070f20 0x7fc4700ac890 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fc4700b1260 tx=0x7fc46802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab2f5482b4118a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:44.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc46dd9b640 1 -- 172.21.15.60:0/482069449 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46802ca80 con 0x7fc470070f20 2023-10-16T14:10:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc46dd9b640 1 -- 172.21.15.60:0/482069449 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46802cbe0 con 0x7fc470070f20 2023-10-16T14:10:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc46dd9b640 1 -- 172.21.15.60:0/482069449 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4680343f0 con 0x7fc470070f20 2023-10-16T14:10:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 -- 172.21.15.60:0/482069449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc470070f20 msgr2=0x7fc4700ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 --2- 172.21.15.60:0/482069449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc470070f20 0x7fc4700ac890 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fc4700b1260 tx=0x7fc46802bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 -- 172.21.15.60:0/482069449 shutdown_connections 2023-10-16T14:10:44.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 --2- 172.21.15.60:0/482069449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc470070f20 0x7fc4700ac890 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 --2- 172.21.15.60:0/482069449 >> v2:172.21.15.144:3300/0 conn(0x7fc47015dde0 0x7fc4700709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 -- 172.21.15.60:0/482069449 >> 172.21.15.60:0/482069449 conn(0x7fc4700ce6a0 msgr2=0x7fc4700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 -- 172.21.15.60:0/482069449 shutdown_connections 2023-10-16T14:10:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.259+0000 7fc474850640 1 -- 172.21.15.60:0/482069449 wait complete. 2023-10-16T14:10:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.261+0000 7fc474850640 1 Processor -- start 2023-10-16T14:10:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.261+0000 7fc474850640 1 -- start start 2023-10-16T14:10:44.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.262+0000 7fc474850640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc470070f20 0x7fc4700c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.262+0000 7fc474850640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc47015c9f0 0x7fc4700c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.262+0000 7fc474850640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 0x7fc470158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.262+0000 7fc474850640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700c7150 con 0x7fc47015dde0 2023-10-16T14:10:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.262+0000 7fc474850640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47006aad0 con 0x7fc47015c9f0 2023-10-16T14:10:44.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.262+0000 7fc474850640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47006adb0 con 0x7fc470070f20 2023-10-16T14:10:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 0x7fc470158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 0x7fc470158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38036/0 (socket says 172.21.15.60:38036) 2023-10-16T14:10:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 -- 172.21.15.60:0/3385340078 learned_addr learned my addr 172.21.15.60:0/3385340078 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46e59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc47015c9f0 0x7fc4700c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46ed9d640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc470070f20 0x7fc4700c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 -- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc470070f20 msgr2=0x7fc4700c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc470070f20 0x7fc4700c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 -- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc47015c9f0 msgr2=0x7fc4700c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc47015c9f0 0x7fc4700c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.263+0000 7fc46f59e640 1 -- 172.21.15.60:0/3385340078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468009b30 con 0x7fc47015dde0 2023-10-16T14:10:44.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.264+0000 7fc46f59e640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 0x7fc470158b40 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fc468036040 tx=0x7fc468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:44.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.264+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46803e450 con 0x7fc47015dde0 2023-10-16T14:10:44.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.264+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46803e5d0 con 0x7fc47015dde0 2023-10-16T14:10:44.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.264+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46803e980 con 0x7fc47015dde0 2023-10-16T14:10:44.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.264+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47006b7e0 con 0x7fc47015dde0 2023-10-16T14:10:44.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.264+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47006b480 con 0x7fc47015dde0 2023-10-16T14:10:44.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.265+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47006b610 con 0x7fc47015dde0 2023-10-16T14:10:44.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.267+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc46802cd50 con 0x7fc47015dde0 2023-10-16T14:10:44.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.267+0000 7fc4577fe640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc45c06af90 0x7fc45c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.268+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fc4680ac650 con 0x7fc47015dde0 2023-10-16T14:10:44.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.268+0000 7fc46ed9d640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc45c06af90 0x7fc45c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.270+0000 7fc46ed9d640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc45c06af90 0x7fc45c06d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc460007fd0 tx=0x7fc460009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:44.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.272+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4680773a0 con 0x7fc47015dde0 2023-10-16T14:10:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.508+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc47006bca0 con 0x7fc47015dde0 2023-10-16T14:10:44.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.510+0000 7fc4577fe640 1 -- 172.21.15.60:0/3385340078 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6153 (secure 0 0 0) 0x7fc46807a9c0 con 0x7fc47015dde0 2023-10-16T14:10:44.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:10:44.515 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 118 2023-10-16T14:10:44.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.512+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc45c06af90 msgr2=0x7fc45c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.512+0000 7fc474850640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc45c06af90 0x7fc45c06d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc460007fd0 tx=0x7fc460009040 comp rx=0 tx=0).stop 2023-10-16T14:10:44.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 msgr2=0x7fc470158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 0x7fc470158b40 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fc468036040 tx=0x7fc468004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:44.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 shutdown_connections 2023-10-16T14:10:44.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fc45c06af90 0x7fc45c06d450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc47015dde0 0x7fc470158b40 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc47015c9f0 0x7fc4700c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 --2- 172.21.15.60:0/3385340078 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc470070f20 0x7fc4700c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 >> 172.21.15.60:0/3385340078 conn(0x7fc4700ce6a0 msgr2=0x7fc4700cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 shutdown_connections 2023-10-16T14:10:44.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.513+0000 7fc474850640 1 -- 172.21.15.60:0/3385340078 wait complete. 2023-10-16T14:10:44.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:10:44.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.757+0000 7f7a2bfff640 1 Processor -- start 2023-10-16T14:10:44.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.757+0000 7f7a2bfff640 1 -- start start 2023-10-16T14:10:44.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.759+0000 7f7a2bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a2c06c040 0x7f7a2c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.759+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c0ad970 0x7f7a2c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.759+0000 7f7a2bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c06c960 con 0x7f7a2c15ebc0 2023-10-16T14:10:44.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.759+0000 7f7a2bfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c15b9d0 con 0x7f7a2c06c040 2023-10-16T14:10:44.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.759+0000 7f7a2bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c15bcb0 con 0x7f7a2c0ad970 2023-10-16T14:10:44.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a2c06c040 0x7f7a2c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a2c06c040 0x7f7a2c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38042/0 (socket says 172.21.15.60:38042) 2023-10-16T14:10:44.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 -- 172.21.15.60:0/1700237948 learned_addr learned my addr 172.21.15.60:0/1700237948 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:44.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a23fff640 1 --2- 172.21.15.60:0/1700237948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c0ad970 0x7f7a2c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 -- 172.21.15.60:0/1700237948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c0ad970 msgr2=0x7f7a2c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 --2- 172.21.15.60:0/1700237948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c0ad970 0x7f7a2c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 -- 172.21.15.60:0/1700237948 >> v1:172.21.15.144:6789/0 conn(0x7f7a2c15ebc0 legacy=0x7f7a2c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.760+0000 7f7a2affd640 1 -- 172.21.15.60:0/1700237948 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c15bf90 con 0x7f7a2c06c040 2023-10-16T14:10:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.761+0000 7f7a2affd640 1 --2- 172.21.15.60:0/1700237948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c06c040 0x7f7a2c06c420 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0098a0 tx=0x7f7a1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa64a1abdffac1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2a7fc640 1 -- 172.21.15.60:0/1700237948 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c031640 con 0x7f7a2c06c040 2023-10-16T14:10:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2a7fc640 1 -- 172.21.15.60:0/1700237948 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c0317a0 con 0x7f7a2c06c040 2023-10-16T14:10:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2a7fc640 1 -- 172.21.15.60:0/1700237948 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c031b30 con 0x7f7a2c06c040 2023-10-16T14:10:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2bfff640 1 -- 172.21.15.60:0/1700237948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c06c040 msgr2=0x7f7a2c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/1700237948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c06c040 0x7f7a2c06c420 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0098a0 tx=0x7f7a1c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2bfff640 1 -- 172.21.15.60:0/1700237948 shutdown_connections 2023-10-16T14:10:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/1700237948 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c0ad970 0x7f7a2c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/1700237948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c06c040 0x7f7a2c06c420 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.762+0000 7f7a2bfff640 1 -- 172.21.15.60:0/1700237948 >> 172.21.15.60:0/1700237948 conn(0x7f7a2c0ce6a0 msgr2=0x7f7a2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.763+0000 7f7a2bfff640 1 -- 172.21.15.60:0/1700237948 shutdown_connections 2023-10-16T14:10:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.763+0000 7f7a2bfff640 1 -- 172.21.15.60:0/1700237948 wait complete. 2023-10-16T14:10:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.764+0000 7f7a2bfff640 1 Processor -- start 2023-10-16T14:10:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 -- start start 2023-10-16T14:10:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2c06c040 0x7f7a2c1e4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 0x7f7a2c0afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c15ebc0 0x7f7a2c0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0b0170 con 0x7f7a2c0ad970 2023-10-16T14:10:44.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0cd0b0 con 0x7f7a2c06c040 2023-10-16T14:10:44.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.765+0000 7f7a2bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0cd630 con 0x7f7a2c15ebc0 2023-10-16T14:10:44.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 0x7f7a2c0afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 0x7f7a2c0afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38058/0 (socket says 172.21.15.60:38058) 2023-10-16T14:10:44.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 -- 172.21.15.60:0/2234572929 learned_addr learned my addr 172.21.15.60:0/2234572929 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a2affd640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2c06c040 0x7f7a2c1e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 -- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c15ebc0 msgr2=0x7f7a2c0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a2b7fe640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c15ebc0 0x7f7a2c0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c15ebc0 0x7f7a2c0ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 -- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2c06c040 msgr2=0x7f7a2c1e4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2c06c040 0x7f7a2c1e4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a23fff640 1 -- 172.21.15.60:0/2234572929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c0094d0 con 0x7f7a2c0ad970 2023-10-16T14:10:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.766+0000 7f7a2affd640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2c06c040 0x7f7a2c1e4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.767+0000 7f7a23fff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 0x7f7a2c0afc30 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400c7f0 tx=0x7f7a1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:44.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.767+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14004220 con 0x7f7a2c0ad970 2023-10-16T14:10:44.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.767+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a14004380 con 0x7f7a2c0ad970 2023-10-16T14:10:44.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.767+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14007840 con 0x7f7a2c0ad970 2023-10-16T14:10:44.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.767+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c0cd350 con 0x7f7a2c0ad970 2023-10-16T14:10:44.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.767+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c0af430 con 0x7f7a2c0ad970 2023-10-16T14:10:44.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.768+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c06b330 con 0x7f7a2c0ad970 2023-10-16T14:10:44.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.769+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a14007bb0 con 0x7f7a2c0ad970 2023-10-16T14:10:44.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.770+0000 7f7a237fe640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7a0c06af90 0x7f7a0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:44.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.770+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f7a14089be0 con 0x7f7a2c0ad970 2023-10-16T14:10:44.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.771+0000 7f7a2affd640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7a0c06af90 0x7f7a0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:44.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.772+0000 7f7a2affd640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7a0c06af90 0x7f7a0c06d450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c036000 tx=0x7f7a1c03c040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:44.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.775+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a14054670 con 0x7f7a2c0ad970 2023-10-16T14:10:44.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.976+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a2c04e9b0 con 0x7f7a2c0ad970 2023-10-16T14:10:44.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.981+0000 7f7a237fe640 1 -- 172.21.15.60:0/2234572929 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+20295 (secure 0 0 0) 0x7f7a1401e020 con 0x7f7a2c0ad970 2023-10-16T14:10:44.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:44.985 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":235,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:41.975126+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:10:44.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7a0c06af90 msgr2=0x7f7a0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7a0c06af90 0x7f7a0c06d450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c036000 tx=0x7f7a1c03c040 comp rx=0 tx=0).stop 2023-10-16T14:10:44.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 msgr2=0x7f7a2c0afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:44.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 0x7f7a2c0afc30 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400c7f0 tx=0x7f7a1400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 shutdown_connections 2023-10-16T14:10:44.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f7a0c06af90 0x7f7a0c06d450 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a2c15ebc0 0x7f7a2c0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a2c0ad970 0x7f7a2c0afc30 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.983+0000 7f7a2bfff640 1 --2- 172.21.15.60:0/2234572929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a2c06c040 0x7f7a2c1e4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.984+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 >> 172.21.15.60:0/2234572929 conn(0x7f7a2c0ce6a0 msgr2=0x7f7a2c15f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.984+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 shutdown_connections 2023-10-16T14:10:44.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:44.984+0000 7f7a2bfff640 1 -- 172.21.15.60:0/2234572929 wait complete. 2023-10-16T14:10:45.008 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:10:45.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:45.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.224+0000 7f1561ea2640 1 Processor -- start 2023-10-16T14:10:45.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.224+0000 7f1561ea2640 1 -- start start 2023-10-16T14:10:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.225+0000 7f1561ea2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 0x7f155c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.225+0000 7f1561ea2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f155c0b0b20 0x7f155c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.225+0000 7f1561ea2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c06f7f0 con 0x7f155c159f00 2023-10-16T14:10:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.225+0000 7f1561ea2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c06c5e0 con 0x7f155c0b0b20 2023-10-16T14:10:45.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.225+0000 7f1561ea2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c06c8c0 con 0x7f155c159b20 2023-10-16T14:10:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.226+0000 7f155affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f155c159f00 0x7f155c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37858/0 (socket says 172.21.15.60:37858) 2023-10-16T14:10:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.226+0000 7f155affd640 1 -- 172.21.15.60:0/3020178131 learned_addr learned my addr 172.21.15.60:0/3020178131 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.226+0000 7f155b7fe640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 0x7f155c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.226+0000 7f155bfff640 1 --2- 172.21.15.60:0/3020178131 >> v2:172.21.15.144:3300/0 conn(0x7f155c0b0b20 0x7f155c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.226+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155088188 0 0) 0x7f155c06f7f0 con 0x7f155c159f00 2023-10-16T14:10:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.226+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1548003520 con 0x7f155c159f00 2023-10-16T14:10:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554361419 0 0) 0x7f1548003520 con 0x7f155c159f00 2023-10-16T14:10:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 msgr2=0x7f155c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 0x7f155c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 >> v2:172.21.15.144:3300/0 conn(0x7f155c0b0b20 msgr2=0x7f155c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 --2- 172.21.15.60:0/3020178131 >> v2:172.21.15.144:3300/0 conn(0x7f155c0b0b20 0x7f155c06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c06cba0 con 0x7f155c159f00 2023-10-16T14:10:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f15440019f0 con 0x7f155c159f00 2023-10-16T14:10:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 >> v1:172.21.15.60:6789/0 conn(0x7f155c159f00 legacy=0x7f155c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 0x7f1548003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1548003ef0 0x7f15480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1548004d00 0x7f1548005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c06c5e0 con 0x7f1548003ef0 2023-10-16T14:10:45.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c06c8c0 con 0x7f1548004d00 2023-10-16T14:10:45.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154800db90 con 0x7f155c159b20 2023-10-16T14:10:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.227+0000 7f155affd640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1548003ef0 0x7f15480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.228+0000 7f155affd640 1 -- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 msgr2=0x7f1548003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.228+0000 7f155affd640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f155c159b20 0x7f1548003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.228+0000 7f155affd640 1 -- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1548004d00 msgr2=0x7f1548005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.228+0000 7f155affd640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1548004d00 0x7f1548005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.228+0000 7f155affd640 1 -- 172.21.15.60:0/3020178131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548003520 con 0x7f1548003ef0 2023-10-16T14:10:45.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.229+0000 7f155affd640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1548003ef0 0x7f15480047c0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f154400a830 tx=0x7f15480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d31e336f7b1327f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:45.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.229+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154402cea0 con 0x7f1548003ef0 2023-10-16T14:10:45.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.229+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154402f070 con 0x7f1548003ef0 2023-10-16T14:10:45.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.229+0000 7f155a7fc640 1 -- 172.21.15.60:0/3020178131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15440348d0 con 0x7f1548003ef0 2023-10-16T14:10:45.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f1561ea2640 1 -- 172.21.15.60:0/3020178131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1548003ef0 msgr2=0x7f15480047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f1561ea2640 1 --2- 172.21.15.60:0/3020178131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1548003ef0 0x7f15480047c0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f154400a830 tx=0x7f15480057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f155bfff640 1 -- 172.21.15.60:0/3020178131 reap_dead start 2023-10-16T14:10:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f1561ea2640 1 -- 172.21.15.60:0/3020178131 shutdown_connections 2023-10-16T14:10:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f1561ea2640 1 -- 172.21.15.60:0/3020178131 >> 172.21.15.60:0/3020178131 conn(0x7f155c0cb410 msgr2=0x7f155c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f1561ea2640 1 -- 172.21.15.60:0/3020178131 shutdown_connections 2023-10-16T14:10:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.230+0000 7f1561ea2640 1 -- 172.21.15.60:0/3020178131 wait complete. 2023-10-16T14:10:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.232+0000 7f1561ea2640 1 Processor -- start 2023-10-16T14:10:45.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.232+0000 7f1561ea2640 1 -- start start 2023-10-16T14:10:45.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f1561ea2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1548003ef0 0x7f155c14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f1561ea2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155c0cd760 0x7f155c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f1561ea2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 0x7f155c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f1561ea2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c15caf0 con 0x7f155c0cdbe0 2023-10-16T14:10:45.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f1561ea2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0ce1b0 con 0x7f155c0cd760 2023-10-16T14:10:45.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f1561ea2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c15c4a0 con 0x7f1548003ef0 2023-10-16T14:10:45.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f155bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 0x7f155c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f155bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 0x7f155c0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38066/0 (socket says 172.21.15.60:38066) 2023-10-16T14:10:45.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.233+0000 7f155bfff640 1 -- 172.21.15.60:0/964198096 learned_addr learned my addr 172.21.15.60:0/964198096 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:45.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155affd640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155c0cd760 0x7f155c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155b7fe640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1548003ef0 0x7f155c14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155bfff640 1 -- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1548003ef0 msgr2=0x7f155c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155bfff640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1548003ef0 0x7f155c14ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155bfff640 1 -- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155c0cd760 msgr2=0x7f155c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155bfff640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155c0cd760 0x7f155c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155bfff640 1 -- 172.21.15.60:0/964198096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c15c780 con 0x7f155c0cdbe0 2023-10-16T14:10:45.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155b7fe640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1548003ef0 0x7f155c14ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:45.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f155bfff640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 0x7f155c0acdd0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f155c06d360 tx=0x7f155400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:45.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155400c680 con 0x7f155c0cdbe0 2023-10-16T14:10:45.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.235+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c14f490 con 0x7f155c0cdbe0 2023-10-16T14:10:45.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.234+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155400c7e0 con 0x7f155c0cdbe0 2023-10-16T14:10:45.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.235+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155400cb90 con 0x7f155c0cdbe0 2023-10-16T14:10:45.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.235+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c156220 con 0x7f155c0cdbe0 2023-10-16T14:10:45.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.236+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c06ce60 con 0x7f155c0cdbe0 2023-10-16T14:10:45.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.237+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f155400ccf0 con 0x7f155c0cdbe0 2023-10-16T14:10:45.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.238+0000 7f1553fff640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f153c06af90 0x7f153c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.238+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f15540896f0 con 0x7f155c0cdbe0 2023-10-16T14:10:45.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.238+0000 7f155b7fe640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f153c06af90 0x7f153c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.240+0000 7f155b7fe640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f153c06af90 0x7f153c06d450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f154c006ce0 tx=0x7f154c008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:45.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.242+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f155c06ce60 con 0x7f155c0cdbe0 2023-10-16T14:10:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.432+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f155c15c320 con 0x7f153c06af90 2023-10-16T14:10:45.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.476+0000 7f1553fff640 1 -- 172.21.15.60:0/964198096 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+522429 (secure 0 0 0) 0x7f154c0095c0 con 0x7f153c06af90 2023-10-16T14:10:45.487 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:45.491 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f153c06af90 msgr2=0x7f153c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f153c06af90 0x7f153c06d450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f154c006ce0 tx=0x7f154c008040 comp rx=0 tx=0).stop 2023-10-16T14:10:45.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 msgr2=0x7f155c0acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 0x7f155c0acdd0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f155c06d360 tx=0x7f155400ec00 comp rx=0 tx=0).stop 2023-10-16T14:10:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 shutdown_connections 2023-10-16T14:10:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f153c06af90 0x7f153c06d450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f155c0cdbe0 0x7f155c0acdd0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f155c0cd760 0x7f155c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 --2- 172.21.15.60:0/964198096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1548003ef0 0x7f155c14ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 >> 172.21.15.60:0/964198096 conn(0x7f155c0cb410 msgr2=0x7f155c157ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 shutdown_connections 2023-10-16T14:10:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.485+0000 7f1561ea2640 1 -- 172.21.15.60:0/964198096 wait complete. 2023-10-16T14:10:45.512 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-16T14:10:44.182486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596411,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":657,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":346,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377744,"kb_used_data":24144,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754596976,"statfs":{"total":773094113280,"available":772707303424,"internally_reserved":0,"allocated":24723456,"data_stored":12632191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3633,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.250307"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244206+0000","last_change":"2023-10-16T14:10:40.992171+0000","last_active":"2023-10-16T14:10:43.244206+0000","last_peered":"2023-10-16T14:10:43.244206+0000","last_clean":"2023-10-16T14:10:43.244206+0000","last_became_active":"2023-10-16T14:10:40.991865+0000","last_became_peered":"2023-10-16T14:10:40.991865+0000","last_unstale":"2023-10-16T14:10:43.244206+0000","last_undegraded":"2023-10-16T14:10:43.244206+0000","last_fullsized":"2023-10-16T14:10:43.244206+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:36.935225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1226,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974179+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:40.974179+0000","last_peered":"2023-10-16T14:10:40.974179+0000","last_clean":"2023-10-16T14:10:40.974179+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:40.974179+0000","last_undegraded":"2023-10-16T14:10:40.974179+0000","last_fullsized":"2023-10-16T14:10:40.974179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179564+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:29.179564+0000","last_peered":"2023-10-16T14:10:29.179564+0000","last_clean":"2023-10-16T14:10:29.179564+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:29.179564+0000","last_undegraded":"2023-10-16T14:10:29.179564+0000","last_fullsized":"2023-10-16T14:10:29.179564+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980445+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:40.980445+0000","last_peered":"2023-10-16T14:10:40.980445+0000","last_clean":"2023-10-16T14:10:40.980445+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:40.980445+0000","last_undegraded":"2023-10-16T14:10:40.980445+0000","last_fullsized":"2023-10-16T14:10:40.980445+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008653+0000","last_change":"2023-10-16T14:10:39.986399+0000","last_active":"2023-10-16T14:10:41.008653+0000","last_peered":"2023-10-16T14:10:41.008653+0000","last_clean":"2023-10-16T14:10:41.008653+0000","last_became_active":"2023-10-16T14:10:39.986098+0000","last_became_peered":"2023-10-16T14:10:39.986098+0000","last_unstale":"2023-10-16T14:10:41.008653+0000","last_undegraded":"2023-10-16T14:10:41.008653+0000","last_fullsized":"2023-10-16T14:10:41.008653+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:15.423316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179310+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:29.179310+0000","last_peered":"2023-10-16T14:10:29.179310+0000","last_clean":"2023-10-16T14:10:29.179310+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:29.179310+0000","last_undegraded":"2023-10-16T14:10:29.179310+0000","last_fullsized":"2023-10-16T14:10:29.179310+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970555+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:40.970555+0000","last_peered":"2023-10-16T14:10:40.970555+0000","last_clean":"2023-10-16T14:10:40.970555+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:40.970555+0000","last_undegraded":"2023-10-16T14:10:40.970555+0000","last_fullsized":"2023-10-16T14:10:40.970555+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979944+0000","last_change":"2023-10-16T14:10:39.988566+0000","last_active":"2023-10-16T14:10:40.979944+0000","last_peered":"2023-10-16T14:10:40.979944+0000","last_clean":"2023-10-16T14:10:40.979944+0000","last_became_active":"2023-10-16T14:10:39.988037+0000","last_became_peered":"2023-10-16T14:10:39.988037+0000","last_unstale":"2023-10-16T14:10:40.979944+0000","last_undegraded":"2023-10-16T14:10:40.979944+0000","last_fullsized":"2023-10-16T14:10:40.979944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.694200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125491+0000","last_change":"2023-10-16T14:10:40.987050+0000","last_active":"2023-10-16T14:10:42.125491+0000","last_peered":"2023-10-16T14:10:42.125491+0000","last_clean":"2023-10-16T14:10:42.125491+0000","last_became_active":"2023-10-16T14:10:40.986733+0000","last_became_peered":"2023-10-16T14:10:40.986733+0000","last_unstale":"2023-10-16T14:10:42.125491+0000","last_undegraded":"2023-10-16T14:10:42.125491+0000","last_fullsized":"2023-10-16T14:10:42.125491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:53.222315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179774+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:29.179774+0000","last_peered":"2023-10-16T14:10:29.179774+0000","last_clean":"2023-10-16T14:10:29.179774+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:29.179774+0000","last_undegraded":"2023-10-16T14:10:29.179774+0000","last_fullsized":"2023-10-16T14:10:29.179774+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975719+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:40.975719+0000","last_peered":"2023-10-16T14:10:40.975719+0000","last_clean":"2023-10-16T14:10:40.975719+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:40.975719+0000","last_undegraded":"2023-10-16T14:10:40.975719+0000","last_fullsized":"2023-10-16T14:10:40.975719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008672+0000","last_change":"2023-10-16T14:10:39.985490+0000","last_active":"2023-10-16T14:10:41.008672+0000","last_peered":"2023-10-16T14:10:41.008672+0000","last_clean":"2023-10-16T14:10:41.008672+0000","last_became_active":"2023-10-16T14:10:39.985187+0000","last_became_peered":"2023-10-16T14:10:39.985187+0000","last_unstale":"2023-10-16T14:10:41.008672+0000","last_undegraded":"2023-10-16T14:10:41.008672+0000","last_fullsized":"2023-10-16T14:10:41.008672+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:32.423740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125498+0000","last_change":"2023-10-16T14:10:40.989384+0000","last_active":"2023-10-16T14:10:42.125498+0000","last_peered":"2023-10-16T14:10:42.125498+0000","last_clean":"2023-10-16T14:10:42.125498+0000","last_became_active":"2023-10-16T14:10:40.989064+0000","last_became_peered":"2023-10-16T14:10:40.989064+0000","last_unstale":"2023-10-16T14:10:42.125498+0000","last_undegraded":"2023-10-16T14:10:42.125498+0000","last_fullsized":"2023-10-16T14:10:42.125498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:50.164078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974785+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:40.974785+0000","last_peered":"2023-10-16T14:10:40.974785+0000","last_clean":"2023-10-16T14:10:40.974785+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:40.974785+0000","last_undegraded":"2023-10-16T14:10:40.974785+0000","last_fullsized":"2023-10-16T14:10:40.974785+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179447+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:29.179447+0000","last_peered":"2023-10-16T14:10:29.179447+0000","last_clean":"2023-10-16T14:10:29.179447+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:29.179447+0000","last_undegraded":"2023-10-16T14:10:29.179447+0000","last_fullsized":"2023-10-16T14:10:29.179447+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.952450+0000","last_change":"2023-10-16T14:10:38.952450+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.952450+0000","last_undegraded":"2023-10-16T14:10:38.952450+0000","last_fullsized":"2023-10-16T14:10:38.952450+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977551+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:40.977551+0000","last_peered":"2023-10-16T14:10:40.977551+0000","last_clean":"2023-10-16T14:10:40.977551+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:40.977551+0000","last_undegraded":"2023-10-16T14:10:40.977551+0000","last_fullsized":"2023-10-16T14:10:40.977551+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981376+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:40.981376+0000","last_peered":"2023-10-16T14:10:40.981376+0000","last_clean":"2023-10-16T14:10:40.981376+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:40.981376+0000","last_undegraded":"2023-10-16T14:10:40.981376+0000","last_fullsized":"2023-10-16T14:10:40.981376+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985469+0000","last_change":"2023-10-16T14:10:39.989343+0000","last_active":"2023-10-16T14:10:40.985469+0000","last_peered":"2023-10-16T14:10:40.985469+0000","last_clean":"2023-10-16T14:10:40.985469+0000","last_became_active":"2023-10-16T14:10:39.989043+0000","last_became_peered":"2023-10-16T14:10:39.989043+0000","last_unstale":"2023-10-16T14:10:40.985469+0000","last_undegraded":"2023-10-16T14:10:40.985469+0000","last_fullsized":"2023-10-16T14:10:40.985469+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:57.519740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979800+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:40.979800+0000","last_peered":"2023-10-16T14:10:40.979800+0000","last_clean":"2023-10-16T14:10:40.979800+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:40.979800+0000","last_undegraded":"2023-10-16T14:10:40.979800+0000","last_fullsized":"2023-10-16T14:10:40.979800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179954+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:29.179954+0000","last_peered":"2023-10-16T14:10:29.179954+0000","last_clean":"2023-10-16T14:10:29.179954+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:29.179954+0000","last_undegraded":"2023-10-16T14:10:29.179954+0000","last_fullsized":"2023-10-16T14:10:29.179954+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980253+0000","last_change":"2023-10-16T14:10:39.979128+0000","last_active":"2023-10-16T14:10:40.980253+0000","last_peered":"2023-10-16T14:10:40.980253+0000","last_clean":"2023-10-16T14:10:40.980253+0000","last_became_active":"2023-10-16T14:10:39.978782+0000","last_became_peered":"2023-10-16T14:10:39.978782+0000","last_unstale":"2023-10-16T14:10:40.980253+0000","last_undegraded":"2023-10-16T14:10:40.980253+0000","last_fullsized":"2023-10-16T14:10:40.980253+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:28.231837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971953+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:40.971953+0000","last_peered":"2023-10-16T14:10:40.971953+0000","last_clean":"2023-10-16T14:10:40.971953+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:40.971953+0000","last_undegraded":"2023-10-16T14:10:40.971953+0000","last_fullsized":"2023-10-16T14:10:40.971953+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972632+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:40.972632+0000","last_peered":"2023-10-16T14:10:40.972632+0000","last_clean":"2023-10-16T14:10:40.972632+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:40.972632+0000","last_undegraded":"2023-10-16T14:10:40.972632+0000","last_fullsized":"2023-10-16T14:10:40.972632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969118+0000","last_change":"2023-10-16T14:10:39.982849+0000","last_active":"2023-10-16T14:10:40.969118+0000","last_peered":"2023-10-16T14:10:40.969118+0000","last_clean":"2023-10-16T14:10:40.969118+0000","last_became_active":"2023-10-16T14:10:39.982560+0000","last_became_peered":"2023-10-16T14:10:39.982560+0000","last_unstale":"2023-10-16T14:10:40.969118+0000","last_undegraded":"2023-10-16T14:10:40.969118+0000","last_fullsized":"2023-10-16T14:10:40.969118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:51.220017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242743+0000","last_change":"2023-10-16T14:10:40.991901+0000","last_active":"2023-10-16T14:10:43.242743+0000","last_peered":"2023-10-16T14:10:43.242743+0000","last_clean":"2023-10-16T14:10:43.242743+0000","last_became_active":"2023-10-16T14:10:40.991610+0000","last_became_peered":"2023-10-16T14:10:40.991610+0000","last_unstale":"2023-10-16T14:10:43.242743+0000","last_undegraded":"2023-10-16T14:10:43.242743+0000","last_fullsized":"2023-10-16T14:10:43.242743+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"231'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979815+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:40.979815+0000","last_peered":"2023-10-16T14:10:40.979815+0000","last_clean":"2023-10-16T14:10:40.979815+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:40.979815+0000","last_undegraded":"2023-10-16T14:10:40.979815+0000","last_fullsized":"2023-10-16T14:10:40.979815+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978358+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:40.978358+0000","last_peered":"2023-10-16T14:10:40.978358+0000","last_clean":"2023-10-16T14:10:40.978358+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:40.978358+0000","last_undegraded":"2023-10-16T14:10:40.978358+0000","last_fullsized":"2023-10-16T14:10:40.978358+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.951042+0000","last_change":"2023-10-16T14:10:38.951042+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.951042+0000","last_undegraded":"2023-10-16T14:10:38.951042+0000","last_fullsized":"2023-10-16T14:10:38.951042+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980892+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:40.980892+0000","last_peered":"2023-10-16T14:10:40.980892+0000","last_clean":"2023-10-16T14:10:40.980892+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:40.980892+0000","last_undegraded":"2023-10-16T14:10:40.980892+0000","last_fullsized":"2023-10-16T14:10:40.980892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179694+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:29.179694+0000","last_peered":"2023-10-16T14:10:29.179694+0000","last_clean":"2023-10-16T14:10:29.179694+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:29.179694+0000","last_undegraded":"2023-10-16T14:10:29.179694+0000","last_fullsized":"2023-10-16T14:10:29.179694+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976136+0000","last_change":"2023-10-16T14:10:39.987697+0000","last_active":"2023-10-16T14:10:40.976136+0000","last_peered":"2023-10-16T14:10:40.976136+0000","last_clean":"2023-10-16T14:10:40.976136+0000","last_became_active":"2023-10-16T14:10:39.987388+0000","last_became_peered":"2023-10-16T14:10:39.987388+0000","last_unstale":"2023-10-16T14:10:40.976136+0000","last_undegraded":"2023-10-16T14:10:40.976136+0000","last_fullsized":"2023-10-16T14:10:40.976136+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:46.559129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244071+0000","last_change":"2023-10-16T14:10:40.983968+0000","last_active":"2023-10-16T14:10:43.244071+0000","last_peered":"2023-10-16T14:10:43.244071+0000","last_clean":"2023-10-16T14:10:43.244071+0000","last_became_active":"2023-10-16T14:10:40.983650+0000","last_became_peered":"2023-10-16T14:10:40.983650+0000","last_unstale":"2023-10-16T14:10:43.244071+0000","last_undegraded":"2023-10-16T14:10:43.244071+0000","last_fullsized":"2023-10-16T14:10:43.244071+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969088+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:40.969088+0000","last_peered":"2023-10-16T14:10:40.969088+0000","last_clean":"2023-10-16T14:10:40.969088+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:40.969088+0000","last_undegraded":"2023-10-16T14:10:40.969088+0000","last_fullsized":"2023-10-16T14:10:40.969088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179731+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:29.179731+0000","last_peered":"2023-10-16T14:10:29.179731+0000","last_clean":"2023-10-16T14:10:29.179731+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:29.179731+0000","last_undegraded":"2023-10-16T14:10:29.179731+0000","last_fullsized":"2023-10-16T14:10:29.179731+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009061+0000","last_change":"2023-10-16T14:10:39.982498+0000","last_active":"2023-10-16T14:10:41.009061+0000","last_peered":"2023-10-16T14:10:41.009061+0000","last_clean":"2023-10-16T14:10:41.009061+0000","last_became_active":"2023-10-16T14:10:39.982162+0000","last_became_peered":"2023-10-16T14:10:39.982162+0000","last_unstale":"2023-10-16T14:10:41.009061+0000","last_undegraded":"2023-10-16T14:10:41.009061+0000","last_fullsized":"2023-10-16T14:10:41.009061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:11.621670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.187140+0000","last_change":"2023-10-16T14:10:40.990434+0000","last_active":"2023-10-16T14:10:43.187140+0000","last_peered":"2023-10-16T14:10:43.187140+0000","last_clean":"2023-10-16T14:10:43.187140+0000","last_became_active":"2023-10-16T14:10:40.990134+0000","last_became_peered":"2023-10-16T14:10:40.990134+0000","last_unstale":"2023-10-16T14:10:43.187140+0000","last_undegraded":"2023-10-16T14:10:43.187140+0000","last_fullsized":"2023-10-16T14:10:43.187140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:35.937011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978483+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:40.978483+0000","last_peered":"2023-10-16T14:10:40.978483+0000","last_clean":"2023-10-16T14:10:40.978483+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:40.978483+0000","last_undegraded":"2023-10-16T14:10:40.978483+0000","last_fullsized":"2023-10-16T14:10:40.978483+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967437+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:40.967437+0000","last_peered":"2023-10-16T14:10:40.967437+0000","last_clean":"2023-10-16T14:10:40.967437+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:40.967437+0000","last_undegraded":"2023-10-16T14:10:40.967437+0000","last_fullsized":"2023-10-16T14:10:40.967437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.950829+0000","last_change":"2023-10-16T14:10:38.950829+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.950829+0000","last_undegraded":"2023-10-16T14:10:38.950829+0000","last_fullsized":"2023-10-16T14:10:38.950829+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981490+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:40.981490+0000","last_peered":"2023-10-16T14:10:40.981490+0000","last_clean":"2023-10-16T14:10:40.981490+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:40.981490+0000","last_undegraded":"2023-10-16T14:10:40.981490+0000","last_fullsized":"2023-10-16T14:10:40.981490+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981982+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:40.981982+0000","last_peered":"2023-10-16T14:10:40.981982+0000","last_clean":"2023-10-16T14:10:40.981982+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:40.981982+0000","last_undegraded":"2023-10-16T14:10:40.981982+0000","last_fullsized":"2023-10-16T14:10:40.981982+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961192+0000","last_change":"2023-10-16T14:10:39.981457+0000","last_active":"2023-10-16T14:10:40.961192+0000","last_peered":"2023-10-16T14:10:40.961192+0000","last_clean":"2023-10-16T14:10:40.961192+0000","last_became_active":"2023-10-16T14:10:39.981145+0000","last_became_peered":"2023-10-16T14:10:39.981145+0000","last_unstale":"2023-10-16T14:10:40.961192+0000","last_undegraded":"2023-10-16T14:10:40.961192+0000","last_fullsized":"2023-10-16T14:10:40.961192+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:33.143537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960165+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:40.960165+0000","last_peered":"2023-10-16T14:10:40.960165+0000","last_clean":"2023-10-16T14:10:40.960165+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:40.960165+0000","last_undegraded":"2023-10-16T14:10:40.960165+0000","last_fullsized":"2023-10-16T14:10:40.960165+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960903+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:40.960903+0000","last_peered":"2023-10-16T14:10:40.960903+0000","last_clean":"2023-10-16T14:10:40.960903+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:40.960903+0000","last_undegraded":"2023-10-16T14:10:40.960903+0000","last_fullsized":"2023-10-16T14:10:40.960903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010705+0000","last_change":"2023-10-16T14:10:39.984267+0000","last_active":"2023-10-16T14:10:41.010705+0000","last_peered":"2023-10-16T14:10:41.010705+0000","last_clean":"2023-10-16T14:10:41.010705+0000","last_became_active":"2023-10-16T14:10:39.983894+0000","last_became_peered":"2023-10-16T14:10:39.983894+0000","last_unstale":"2023-10-16T14:10:41.010705+0000","last_undegraded":"2023-10-16T14:10:41.010705+0000","last_fullsized":"2023-10-16T14:10:41.010705+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:02.427201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979684+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:40.979684+0000","last_peered":"2023-10-16T14:10:40.979684+0000","last_clean":"2023-10-16T14:10:40.979684+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:40.979684+0000","last_undegraded":"2023-10-16T14:10:40.979684+0000","last_fullsized":"2023-10-16T14:10:40.979684+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961306+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:40.961306+0000","last_peered":"2023-10-16T14:10:40.961306+0000","last_clean":"2023-10-16T14:10:40.961306+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:40.961306+0000","last_undegraded":"2023-10-16T14:10:40.961306+0000","last_fullsized":"2023-10-16T14:10:40.961306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971464+0000","last_change":"2023-10-16T14:10:39.974033+0000","last_active":"2023-10-16T14:10:40.971464+0000","last_peered":"2023-10-16T14:10:40.971464+0000","last_clean":"2023-10-16T14:10:40.971464+0000","last_became_active":"2023-10-16T14:10:39.973723+0000","last_became_peered":"2023-10-16T14:10:39.973723+0000","last_unstale":"2023-10-16T14:10:40.971464+0000","last_undegraded":"2023-10-16T14:10:40.971464+0000","last_fullsized":"2023-10-16T14:10:40.971464+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:08.572098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124517+0000","last_change":"2023-10-16T14:10:40.994623+0000","last_active":"2023-10-16T14:10:42.124517+0000","last_peered":"2023-10-16T14:10:42.124517+0000","last_clean":"2023-10-16T14:10:42.124517+0000","last_became_active":"2023-10-16T14:10:40.994347+0000","last_became_peered":"2023-10-16T14:10:40.994347+0000","last_unstale":"2023-10-16T14:10:42.124517+0000","last_undegraded":"2023-10-16T14:10:42.124517+0000","last_fullsized":"2023-10-16T14:10:42.124517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:18.643655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961579+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:40.961579+0000","last_peered":"2023-10-16T14:10:40.961579+0000","last_clean":"2023-10-16T14:10:40.961579+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:40.961579+0000","last_undegraded":"2023-10-16T14:10:40.961579+0000","last_fullsized":"2023-10-16T14:10:40.961579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979554+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:40.979554+0000","last_peered":"2023-10-16T14:10:40.979554+0000","last_clean":"2023-10-16T14:10:40.979554+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:40.979554+0000","last_undegraded":"2023-10-16T14:10:40.979554+0000","last_fullsized":"2023-10-16T14:10:40.979554+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.949894+0000","last_change":"2023-10-16T14:10:38.949894+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.949894+0000","last_undegraded":"2023-10-16T14:10:38.949894+0000","last_fullsized":"2023-10-16T14:10:38.949894+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124542+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:42.124542+0000","last_peered":"2023-10-16T14:10:42.124542+0000","last_clean":"2023-10-16T14:10:42.124542+0000","last_became_active":"2023-10-16T14:10:40.993130+0000","last_became_peered":"2023-10-16T14:10:40.993130+0000","last_unstale":"2023-10-16T14:10:42.124542+0000","last_undegraded":"2023-10-16T14:10:42.124542+0000","last_fullsized":"2023-10-16T14:10:42.124542+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:25.881584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973127+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:40.973127+0000","last_peered":"2023-10-16T14:10:40.973127+0000","last_clean":"2023-10-16T14:10:40.973127+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:40.973127+0000","last_undegraded":"2023-10-16T14:10:40.973127+0000","last_fullsized":"2023-10-16T14:10:40.973127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974510+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:40.974510+0000","last_peered":"2023-10-16T14:10:40.974510+0000","last_clean":"2023-10-16T14:10:40.974510+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:40.974510+0000","last_undegraded":"2023-10-16T14:10:40.974510+0000","last_fullsized":"2023-10-16T14:10:40.974510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.953322+0000","last_change":"2023-10-16T14:10:38.953322+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.953322+0000","last_undegraded":"2023-10-16T14:10:38.953322+0000","last_fullsized":"2023-10-16T14:10:38.953322+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185918+0000","last_change":"2023-10-16T14:10:40.993254+0000","last_active":"2023-10-16T14:10:43.185918+0000","last_peered":"2023-10-16T14:10:43.185918+0000","last_clean":"2023-10-16T14:10:43.185918+0000","last_became_active":"2023-10-16T14:10:40.992918+0000","last_became_peered":"2023-10-16T14:10:40.992918+0000","last_unstale":"2023-10-16T14:10:43.185918+0000","last_undegraded":"2023-10-16T14:10:43.185918+0000","last_fullsized":"2023-10-16T14:10:43.185918+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:07.376549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976318+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:40.976318+0000","last_peered":"2023-10-16T14:10:40.976318+0000","last_clean":"2023-10-16T14:10:40.976318+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:40.976318+0000","last_undegraded":"2023-10-16T14:10:40.976318+0000","last_fullsized":"2023-10-16T14:10:40.976318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972611+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:40.972611+0000","last_peered":"2023-10-16T14:10:40.972611+0000","last_clean":"2023-10-16T14:10:40.972611+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:40.972611+0000","last_undegraded":"2023-10-16T14:10:40.972611+0000","last_fullsized":"2023-10-16T14:10:40.972611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970330+0000","last_change":"2023-10-16T14:10:39.989051+0000","last_active":"2023-10-16T14:10:40.970330+0000","last_peered":"2023-10-16T14:10:40.970330+0000","last_clean":"2023-10-16T14:10:40.970330+0000","last_became_active":"2023-10-16T14:10:39.987944+0000","last_became_peered":"2023-10-16T14:10:39.987944+0000","last_unstale":"2023-10-16T14:10:40.970330+0000","last_undegraded":"2023-10-16T14:10:40.970330+0000","last_fullsized":"2023-10-16T14:10:40.970330+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244927+0000","last_change":"2023-10-16T14:10:40.996616+0000","last_active":"2023-10-16T14:10:43.244927+0000","last_peered":"2023-10-16T14:10:43.244927+0000","last_clean":"2023-10-16T14:10:43.244927+0000","last_became_active":"2023-10-16T14:10:40.996057+0000","last_became_peered":"2023-10-16T14:10:40.996057+0000","last_unstale":"2023-10-16T14:10:43.244927+0000","last_undegraded":"2023-10-16T14:10:43.244927+0000","last_fullsized":"2023-10-16T14:10:43.244927+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:32.328604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981955+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:40.981955+0000","last_peered":"2023-10-16T14:10:40.981955+0000","last_clean":"2023-10-16T14:10:40.981955+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:40.981955+0000","last_undegraded":"2023-10-16T14:10:40.981955+0000","last_fullsized":"2023-10-16T14:10:40.981955+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978301+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:40.978301+0000","last_peered":"2023-10-16T14:10:40.978301+0000","last_clean":"2023-10-16T14:10:40.978301+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:40.978301+0000","last_undegraded":"2023-10-16T14:10:40.978301+0000","last_fullsized":"2023-10-16T14:10:40.978301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980655+0000","last_change":"2023-10-16T14:10:39.972329+0000","last_active":"2023-10-16T14:10:40.980655+0000","last_peered":"2023-10-16T14:10:40.980655+0000","last_clean":"2023-10-16T14:10:40.980655+0000","last_became_active":"2023-10-16T14:10:39.972015+0000","last_became_peered":"2023-10-16T14:10:39.972015+0000","last_unstale":"2023-10-16T14:10:40.980655+0000","last_undegraded":"2023-10-16T14:10:40.980655+0000","last_fullsized":"2023-10-16T14:10:40.980655+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:17.467680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959024+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:40.959024+0000","last_peered":"2023-10-16T14:10:40.959024+0000","last_clean":"2023-10-16T14:10:40.959024+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:40.959024+0000","last_undegraded":"2023-10-16T14:10:40.959024+0000","last_fullsized":"2023-10-16T14:10:40.959024+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975612+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:40.975612+0000","last_peered":"2023-10-16T14:10:40.975612+0000","last_clean":"2023-10-16T14:10:40.975612+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:40.975612+0000","last_undegraded":"2023-10-16T14:10:40.975612+0000","last_fullsized":"2023-10-16T14:10:40.975612+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985242+0000","last_change":"2023-10-16T14:10:39.990904+0000","last_active":"2023-10-16T14:10:40.985242+0000","last_peered":"2023-10-16T14:10:40.985242+0000","last_clean":"2023-10-16T14:10:40.985242+0000","last_became_active":"2023-10-16T14:10:39.990155+0000","last_became_peered":"2023-10-16T14:10:39.990155+0000","last_unstale":"2023-10-16T14:10:40.985242+0000","last_undegraded":"2023-10-16T14:10:40.985242+0000","last_fullsized":"2023-10-16T14:10:40.985242+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:56.282178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243767+0000","last_change":"2023-10-16T14:10:40.993640+0000","last_active":"2023-10-16T14:10:43.243767+0000","last_peered":"2023-10-16T14:10:43.243767+0000","last_clean":"2023-10-16T14:10:43.243767+0000","last_became_active":"2023-10-16T14:10:40.993084+0000","last_became_peered":"2023-10-16T14:10:40.993084+0000","last_unstale":"2023-10-16T14:10:43.243767+0000","last_undegraded":"2023-10-16T14:10:43.243767+0000","last_fullsized":"2023-10-16T14:10:43.243767+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:12.061147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179859+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:29.179859+0000","last_peered":"2023-10-16T14:10:29.179859+0000","last_clean":"2023-10-16T14:10:29.179859+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:29.179859+0000","last_undegraded":"2023-10-16T14:10:29.179859+0000","last_fullsized":"2023-10-16T14:10:29.179859+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959827+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:40.959827+0000","last_peered":"2023-10-16T14:10:40.959827+0000","last_clean":"2023-10-16T14:10:40.959827+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:40.959827+0000","last_undegraded":"2023-10-16T14:10:40.959827+0000","last_fullsized":"2023-10-16T14:10:40.959827+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"234'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013944+0000","last_change":"2023-10-16T14:10:39.971085+0000","last_active":"2023-10-16T14:10:41.013944+0000","last_peered":"2023-10-16T14:10:41.013944+0000","last_clean":"2023-10-16T14:10:41.013944+0000","last_became_active":"2023-10-16T14:10:39.970777+0000","last_became_peered":"2023-10-16T14:10:39.970777+0000","last_unstale":"2023-10-16T14:10:41.013944+0000","last_undegraded":"2023-10-16T14:10:41.013944+0000","last_fullsized":"2023-10-16T14:10:41.013944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:03.090258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.998098+0000","last_change":"2023-10-16T14:10:40.985478+0000","last_active":"2023-10-16T14:10:41.998098+0000","last_peered":"2023-10-16T14:10:41.998098+0000","last_clean":"2023-10-16T14:10:41.998098+0000","last_became_active":"2023-10-16T14:10:40.985218+0000","last_became_peered":"2023-10-16T14:10:40.985218+0000","last_unstale":"2023-10-16T14:10:41.998098+0000","last_undegraded":"2023-10-16T14:10:41.998098+0000","last_fullsized":"2023-10-16T14:10:41.998098+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:22.942748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179906+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:29.179906+0000","last_peered":"2023-10-16T14:10:29.179906+0000","last_clean":"2023-10-16T14:10:29.179906+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:29.179906+0000","last_undegraded":"2023-10-16T14:10:29.179906+0000","last_fullsized":"2023-10-16T14:10:29.179906+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974315+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:40.974315+0000","last_peered":"2023-10-16T14:10:40.974315+0000","last_clean":"2023-10-16T14:10:40.974315+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:40.974315+0000","last_undegraded":"2023-10-16T14:10:40.974315+0000","last_fullsized":"2023-10-16T14:10:40.974315+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"234'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008061+0000","last_change":"2023-10-16T14:10:39.973954+0000","last_active":"2023-10-16T14:10:41.008061+0000","last_peered":"2023-10-16T14:10:41.008061+0000","last_clean":"2023-10-16T14:10:41.008061+0000","last_became_active":"2023-10-16T14:10:39.973602+0000","last_became_peered":"2023-10-16T14:10:39.973602+0000","last_unstale":"2023-10-16T14:10:41.008061+0000","last_undegraded":"2023-10-16T14:10:41.008061+0000","last_fullsized":"2023-10-16T14:10:41.008061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:18.496303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.14","version":"231'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.983108+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:40.983108+0000","last_peered":"2023-10-16T14:10:40.983108+0000","last_clean":"2023-10-16T14:10:40.983108+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:40.983108+0000","last_undegraded":"2023-10-16T14:10:40.983108+0000","last_fullsized":"2023-10-16T14:10:40.983108+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977730+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:40.977730+0000","last_peered":"2023-10-16T14:10:40.977730+0000","last_clean":"2023-10-16T14:10:40.977730+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:40.977730+0000","last_undegraded":"2023-10-16T14:10:40.977730+0000","last_fullsized":"2023-10-16T14:10:40.977730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980420+0000","last_change":"2023-10-16T14:10:39.976858+0000","last_active":"2023-10-16T14:10:40.980420+0000","last_peered":"2023-10-16T14:10:40.980420+0000","last_clean":"2023-10-16T14:10:40.980420+0000","last_became_active":"2023-10-16T14:10:39.976566+0000","last_became_peered":"2023-10-16T14:10:39.976566+0000","last_unstale":"2023-10-16T14:10:40.980420+0000","last_undegraded":"2023-10-16T14:10:40.980420+0000","last_fullsized":"2023-10-16T14:10:40.980420+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:00.617867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.990318+0000","last_change":"2023-10-16T14:10:40.985099+0000","last_active":"2023-10-16T14:10:41.990318+0000","last_peered":"2023-10-16T14:10:41.990318+0000","last_clean":"2023-10-16T14:10:41.990318+0000","last_became_active":"2023-10-16T14:10:40.984799+0000","last_became_peered":"2023-10-16T14:10:40.984799+0000","last_unstale":"2023-10-16T14:10:41.990318+0000","last_undegraded":"2023-10-16T14:10:41.990318+0000","last_fullsized":"2023-10-16T14:10:41.990318+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:02.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":19,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961436+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:40.961436+0000","last_peered":"2023-10-16T14:10:40.961436+0000","last_clean":"2023-10-16T14:10:40.961436+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:40.961436+0000","last_undegraded":"2023-10-16T14:10:40.961436+0000","last_fullsized":"2023-10-16T14:10:40.961436+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982473+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:40.982473+0000","last_peered":"2023-10-16T14:10:40.982473+0000","last_clean":"2023-10-16T14:10:40.982473+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:40.982473+0000","last_undegraded":"2023-10-16T14:10:40.982473+0000","last_fullsized":"2023-10-16T14:10:40.982473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011710+0000","last_change":"2023-10-16T14:10:39.986334+0000","last_active":"2023-10-16T14:10:41.011710+0000","last_peered":"2023-10-16T14:10:41.011710+0000","last_clean":"2023-10-16T14:10:41.011710+0000","last_became_active":"2023-10-16T14:10:39.986002+0000","last_became_peered":"2023-10-16T14:10:39.986002+0000","last_unstale":"2023-10-16T14:10:41.011710+0000","last_undegraded":"2023-10-16T14:10:41.011710+0000","last_fullsized":"2023-10-16T14:10:41.011710+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994013+0000","last_change":"2023-10-16T14:10:40.976822+0000","last_active":"2023-10-16T14:10:41.994013+0000","last_peered":"2023-10-16T14:10:41.994013+0000","last_clean":"2023-10-16T14:10:41.994013+0000","last_became_active":"2023-10-16T14:10:40.976523+0000","last_became_peered":"2023-10-16T14:10:40.976523+0000","last_unstale":"2023-10-16T14:10:41.994013+0000","last_undegraded":"2023-10-16T14:10:41.994013+0000","last_fullsized":"2023-10-16T14:10:41.994013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179997+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:29.179997+0000","last_peered":"2023-10-16T14:10:29.179997+0000","last_clean":"2023-10-16T14:10:29.179997+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:29.179997+0000","last_undegraded":"2023-10-16T14:10:29.179997+0000","last_fullsized":"2023-10-16T14:10:29.179997+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980779+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:40.980779+0000","last_peered":"2023-10-16T14:10:40.980779+0000","last_clean":"2023-10-16T14:10:40.980779+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:40.980779+0000","last_undegraded":"2023-10-16T14:10:40.980779+0000","last_fullsized":"2023-10-16T14:10:40.980779+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.937645+0000","last_change":"2023-10-16T14:10:38.937645+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.937645+0000","last_undegraded":"2023-10-16T14:10:38.937645+0000","last_fullsized":"2023-10-16T14:10:38.937645+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976181+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:40.976181+0000","last_peered":"2023-10-16T14:10:40.976181+0000","last_clean":"2023-10-16T14:10:40.976181+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:40.976181+0000","last_undegraded":"2023-10-16T14:10:40.976181+0000","last_fullsized":"2023-10-16T14:10:40.976181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979705+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:40.979705+0000","last_peered":"2023-10-16T14:10:40.979705+0000","last_clean":"2023-10-16T14:10:40.979705+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:40.979705+0000","last_undegraded":"2023-10-16T14:10:40.979705+0000","last_fullsized":"2023-10-16T14:10:40.979705+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975789+0000","last_change":"2023-10-16T14:10:39.968896+0000","last_active":"2023-10-16T14:10:40.975789+0000","last_peered":"2023-10-16T14:10:40.975789+0000","last_clean":"2023-10-16T14:10:40.975789+0000","last_became_active":"2023-10-16T14:10:39.968633+0000","last_became_peered":"2023-10-16T14:10:39.968633+0000","last_unstale":"2023-10-16T14:10:40.975789+0000","last_undegraded":"2023-10-16T14:10:40.975789+0000","last_fullsized":"2023-10-16T14:10:40.975789+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:26.024819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186867+0000","last_change":"2023-10-16T14:10:40.999802+0000","last_active":"2023-10-16T14:10:43.186867+0000","last_peered":"2023-10-16T14:10:43.186867+0000","last_clean":"2023-10-16T14:10:43.186867+0000","last_became_active":"2023-10-16T14:10:40.999483+0000","last_became_peered":"2023-10-16T14:10:40.999483+0000","last_unstale":"2023-10-16T14:10:43.186867+0000","last_undegraded":"2023-10-16T14:10:43.186867+0000","last_fullsized":"2023-10-16T14:10:43.186867+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:19.512934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.940148+0000","last_change":"2023-10-16T14:10:38.940148+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.940148+0000","last_undegraded":"2023-10-16T14:10:38.940148+0000","last_fullsized":"2023-10-16T14:10:38.940148+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179626+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:29.179626+0000","last_peered":"2023-10-16T14:10:29.179626+0000","last_clean":"2023-10-16T14:10:29.179626+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:29.179626+0000","last_undegraded":"2023-10-16T14:10:29.179626+0000","last_fullsized":"2023-10-16T14:10:29.179626+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976062+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:40.976062+0000","last_peered":"2023-10-16T14:10:40.976062+0000","last_clean":"2023-10-16T14:10:40.976062+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:40.976062+0000","last_undegraded":"2023-10-16T14:10:40.976062+0000","last_fullsized":"2023-10-16T14:10:40.976062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1f","version":"234'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009270+0000","last_change":"2023-10-16T14:10:39.973971+0000","last_active":"2023-10-16T14:10:41.009270+0000","last_peered":"2023-10-16T14:10:41.009270+0000","last_clean":"2023-10-16T14:10:41.009270+0000","last_became_active":"2023-10-16T14:10:39.973675+0000","last_became_peered":"2023-10-16T14:10:39.973675+0000","last_unstale":"2023-10-16T14:10:41.009270+0000","last_undegraded":"2023-10-16T14:10:41.009270+0000","last_fullsized":"2023-10-16T14:10:41.009270+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:01.458774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976322+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:40.976322+0000","last_peered":"2023-10-16T14:10:40.976322+0000","last_clean":"2023-10-16T14:10:40.976322+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:40.976322+0000","last_undegraded":"2023-10-16T14:10:40.976322+0000","last_fullsized":"2023-10-16T14:10:40.976322+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968823+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:40.968823+0000","last_peered":"2023-10-16T14:10:40.968823+0000","last_clean":"2023-10-16T14:10:40.968823+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:40.968823+0000","last_undegraded":"2023-10-16T14:10:40.968823+0000","last_fullsized":"2023-10-16T14:10:40.968823+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995327+0000","last_change":"2023-10-16T14:10:40.984926+0000","last_active":"2023-10-16T14:10:41.995327+0000","last_peered":"2023-10-16T14:10:41.995327+0000","last_clean":"2023-10-16T14:10:41.995327+0000","last_became_active":"2023-10-16T14:10:40.984364+0000","last_became_peered":"2023-10-16T14:10:40.984364+0000","last_unstale":"2023-10-16T14:10:41.995327+0000","last_undegraded":"2023-10-16T14:10:41.995327+0000","last_fullsized":"2023-10-16T14:10:41.995327+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974558+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:40.974558+0000","last_peered":"2023-10-16T14:10:40.974558+0000","last_clean":"2023-10-16T14:10:40.974558+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:40.974558+0000","last_undegraded":"2023-10-16T14:10:40.974558+0000","last_fullsized":"2023-10-16T14:10:40.974558+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977408+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:40.977408+0000","last_peered":"2023-10-16T14:10:40.977408+0000","last_clean":"2023-10-16T14:10:40.977408+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:40.977408+0000","last_undegraded":"2023-10-16T14:10:40.977408+0000","last_fullsized":"2023-10-16T14:10:40.977408+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.939295+0000","last_change":"2023-10-16T14:10:38.939295+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.939295+0000","last_undegraded":"2023-10-16T14:10:38.939295+0000","last_fullsized":"2023-10-16T14:10:38.939295+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125751+0000","last_change":"2023-10-16T14:10:40.997566+0000","last_active":"2023-10-16T14:10:42.125751+0000","last_peered":"2023-10-16T14:10:42.125751+0000","last_clean":"2023-10-16T14:10:42.125751+0000","last_became_active":"2023-10-16T14:10:40.997273+0000","last_became_peered":"2023-10-16T14:10:40.997273+0000","last_unstale":"2023-10-16T14:10:42.125751+0000","last_undegraded":"2023-10-16T14:10:42.125751+0000","last_fullsized":"2023-10-16T14:10:42.125751+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:38.615513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982594+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:40.982594+0000","last_peered":"2023-10-16T14:10:40.982594+0000","last_clean":"2023-10-16T14:10:40.982594+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:40.982594+0000","last_undegraded":"2023-10-16T14:10:40.982594+0000","last_fullsized":"2023-10-16T14:10:40.982594+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960137+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:40.960137+0000","last_peered":"2023-10-16T14:10:40.960137+0000","last_clean":"2023-10-16T14:10:40.960137+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:40.960137+0000","last_undegraded":"2023-10-16T14:10:40.960137+0000","last_fullsized":"2023-10-16T14:10:40.960137+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.952123+0000","last_change":"2023-10-16T14:10:38.952123+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.952123+0000","last_undegraded":"2023-10-16T14:10:38.952123+0000","last_fullsized":"2023-10-16T14:10:38.952123+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243037+0000","last_change":"2023-10-16T14:10:40.993412+0000","last_active":"2023-10-16T14:10:43.243037+0000","last_peered":"2023-10-16T14:10:43.243037+0000","last_clean":"2023-10-16T14:10:43.243037+0000","last_became_active":"2023-10-16T14:10:40.993119+0000","last_became_peered":"2023-10-16T14:10:40.993119+0000","last_unstale":"2023-10-16T14:10:43.243037+0000","last_undegraded":"2023-10-16T14:10:43.243037+0000","last_fullsized":"2023-10-16T14:10:43.243037+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:58.022778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981800+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:40.981800+0000","last_peered":"2023-10-16T14:10:40.981800+0000","last_clean":"2023-10-16T14:10:40.981800+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:40.981800+0000","last_undegraded":"2023-10-16T14:10:40.981800+0000","last_fullsized":"2023-10-16T14:10:40.981800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980136+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:40.980136+0000","last_peered":"2023-10-16T14:10:40.980136+0000","last_clean":"2023-10-16T14:10:40.980136+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:40.980136+0000","last_undegraded":"2023-10-16T14:10:40.980136+0000","last_fullsized":"2023-10-16T14:10:40.980136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011809+0000","last_change":"2023-10-16T14:10:39.969078+0000","last_active":"2023-10-16T14:10:41.011809+0000","last_peered":"2023-10-16T14:10:41.011809+0000","last_clean":"2023-10-16T14:10:41.011809+0000","last_became_active":"2023-10-16T14:10:39.968736+0000","last_became_peered":"2023-10-16T14:10:39.968736+0000","last_unstale":"2023-10-16T14:10:41.011809+0000","last_undegraded":"2023-10-16T14:10:41.011809+0000","last_fullsized":"2023-10-16T14:10:41.011809+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:59.354986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972873+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:40.972873+0000","last_peered":"2023-10-16T14:10:40.972873+0000","last_clean":"2023-10-16T14:10:40.972873+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:40.972873+0000","last_undegraded":"2023-10-16T14:10:40.972873+0000","last_fullsized":"2023-10-16T14:10:40.972873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.180040+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:29.180040+0000","last_peered":"2023-10-16T14:10:29.180040+0000","last_clean":"2023-10-16T14:10:29.180040+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:29.180040+0000","last_undegraded":"2023-10-16T14:10:29.180040+0000","last_fullsized":"2023-10-16T14:10:29.180040+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959567+0000","last_change":"2023-10-16T14:10:39.973296+0000","last_active":"2023-10-16T14:10:40.959567+0000","last_peered":"2023-10-16T14:10:40.959567+0000","last_clean":"2023-10-16T14:10:40.959567+0000","last_became_active":"2023-10-16T14:10:39.972592+0000","last_became_peered":"2023-10-16T14:10:39.972592+0000","last_unstale":"2023-10-16T14:10:40.959567+0000","last_undegraded":"2023-10-16T14:10:40.959567+0000","last_fullsized":"2023-10-16T14:10:40.959567+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:04.406174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242344+0000","last_change":"2023-10-16T14:10:40.987363+0000","last_active":"2023-10-16T14:10:43.242344+0000","last_peered":"2023-10-16T14:10:43.242344+0000","last_clean":"2023-10-16T14:10:43.242344+0000","last_became_active":"2023-10-16T14:10:40.985914+0000","last_became_peered":"2023-10-16T14:10:40.985914+0000","last_unstale":"2023-10-16T14:10:43.242344+0000","last_undegraded":"2023-10-16T14:10:43.242344+0000","last_fullsized":"2023-10-16T14:10:43.242344+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:10.423467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":17,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179737+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:29.179737+0000","last_peered":"2023-10-16T14:10:29.179737+0000","last_clean":"2023-10-16T14:10:29.179737+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:29.179737+0000","last_undegraded":"2023-10-16T14:10:29.179737+0000","last_fullsized":"2023-10-16T14:10:29.179737+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972050+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:40.972050+0000","last_peered":"2023-10-16T14:10:40.972050+0000","last_clean":"2023-10-16T14:10:40.972050+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:40.972050+0000","last_undegraded":"2023-10-16T14:10:40.972050+0000","last_fullsized":"2023-10-16T14:10:40.972050+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970323+0000","last_change":"2023-10-16T14:10:39.972648+0000","last_active":"2023-10-16T14:10:40.970323+0000","last_peered":"2023-10-16T14:10:40.970323+0000","last_clean":"2023-10-16T14:10:40.970323+0000","last_became_active":"2023-10-16T14:10:39.972328+0000","last_became_peered":"2023-10-16T14:10:39.972328+0000","last_unstale":"2023-10-16T14:10:40.970323+0000","last_undegraded":"2023-10-16T14:10:40.970323+0000","last_fullsized":"2023-10-16T14:10:40.970323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186986+0000","last_change":"2023-10-16T14:10:40.995949+0000","last_active":"2023-10-16T14:10:43.186986+0000","last_peered":"2023-10-16T14:10:43.186986+0000","last_clean":"2023-10-16T14:10:43.186986+0000","last_became_active":"2023-10-16T14:10:40.995689+0000","last_became_peered":"2023-10-16T14:10:40.995689+0000","last_unstale":"2023-10-16T14:10:43.186986+0000","last_undegraded":"2023-10-16T14:10:43.186986+0000","last_fullsized":"2023-10-16T14:10:43.186986+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:12.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977098+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:40.977098+0000","last_peered":"2023-10-16T14:10:40.977098+0000","last_clean":"2023-10-16T14:10:40.977098+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:40.977098+0000","last_undegraded":"2023-10-16T14:10:40.977098+0000","last_fullsized":"2023-10-16T14:10:40.977098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244569+0000","last_change":"2023-10-16T14:10:41.001647+0000","last_active":"2023-10-16T14:10:43.244569+0000","last_peered":"2023-10-16T14:10:43.244569+0000","last_clean":"2023-10-16T14:10:43.244569+0000","last_became_active":"2023-10-16T14:10:41.000840+0000","last_became_peered":"2023-10-16T14:10:41.000840+0000","last_unstale":"2023-10-16T14:10:43.244569+0000","last_undegraded":"2023-10-16T14:10:43.244569+0000","last_fullsized":"2023-10-16T14:10:43.244569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:55.904685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959155+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:40.959155+0000","last_peered":"2023-10-16T14:10:40.959155+0000","last_clean":"2023-10-16T14:10:40.959155+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:40.959155+0000","last_undegraded":"2023-10-16T14:10:40.959155+0000","last_fullsized":"2023-10-16T14:10:40.959155+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243482+0000","last_change":"2023-10-16T14:10:40.991703+0000","last_active":"2023-10-16T14:10:43.243482+0000","last_peered":"2023-10-16T14:10:43.243482+0000","last_clean":"2023-10-16T14:10:43.243482+0000","last_became_active":"2023-10-16T14:10:40.991364+0000","last_became_peered":"2023-10-16T14:10:40.991364+0000","last_unstale":"2023-10-16T14:10:43.243482+0000","last_undegraded":"2023-10-16T14:10:43.243482+0000","last_fullsized":"2023-10-16T14:10:43.243482+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:30.238400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971730+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:40.971730+0000","last_peered":"2023-10-16T14:10:40.971730+0000","last_clean":"2023-10-16T14:10:40.971730+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:40.971730+0000","last_undegraded":"2023-10-16T14:10:40.971730+0000","last_fullsized":"2023-10-16T14:10:40.971730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244638+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:43.244638+0000","last_peered":"2023-10-16T14:10:43.244638+0000","last_clean":"2023-10-16T14:10:43.244638+0000","last_became_active":"2023-10-16T14:10:40.992264+0000","last_became_peered":"2023-10-16T14:10:40.992264+0000","last_unstale":"2023-10-16T14:10:43.244638+0000","last_undegraded":"2023-10-16T14:10:43.244638+0000","last_fullsized":"2023-10-16T14:10:43.244638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:32.614992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179374+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:29.179374+0000","last_peered":"2023-10-16T14:10:29.179374+0000","last_clean":"2023-10-16T14:10:29.179374+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:29.179374+0000","last_undegraded":"2023-10-16T14:10:29.179374+0000","last_fullsized":"2023-10-16T14:10:29.179374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242825+0000","last_change":"2023-10-16T14:10:40.989459+0000","last_active":"2023-10-16T14:10:43.242825+0000","last_peered":"2023-10-16T14:10:43.242825+0000","last_clean":"2023-10-16T14:10:43.242825+0000","last_became_active":"2023-10-16T14:10:40.989197+0000","last_became_peered":"2023-10-16T14:10:40.989197+0000","last_unstale":"2023-10-16T14:10:43.242825+0000","last_undegraded":"2023-10-16T14:10:43.242825+0000","last_fullsized":"2023-10-16T14:10:43.242825+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979422+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:40.979422+0000","last_peered":"2023-10-16T14:10:40.979422+0000","last_clean":"2023-10-16T14:10:40.979422+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:40.979422+0000","last_undegraded":"2023-10-16T14:10:40.979422+0000","last_fullsized":"2023-10-16T14:10:40.979422+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994059+0000","last_change":"2023-10-16T14:10:40.986689+0000","last_active":"2023-10-16T14:10:41.994059+0000","last_peered":"2023-10-16T14:10:41.994059+0000","last_clean":"2023-10-16T14:10:41.994059+0000","last_became_active":"2023-10-16T14:10:40.986408+0000","last_became_peered":"2023-10-16T14:10:40.986408+0000","last_unstale":"2023-10-16T14:10:41.994059+0000","last_undegraded":"2023-10-16T14:10:41.994059+0000","last_fullsized":"2023-10-16T14:10:41.994059+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179653+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:29.179653+0000","last_peered":"2023-10-16T14:10:29.179653+0000","last_clean":"2023-10-16T14:10:29.179653+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:29.179653+0000","last_undegraded":"2023-10-16T14:10:29.179653+0000","last_fullsized":"2023-10-16T14:10:29.179653+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989882+0000","last_change":"2023-10-16T14:10:40.987586+0000","last_active":"2023-10-16T14:10:41.989882+0000","last_peered":"2023-10-16T14:10:41.989882+0000","last_clean":"2023-10-16T14:10:41.989882+0000","last_became_active":"2023-10-16T14:10:40.987254+0000","last_became_peered":"2023-10-16T14:10:40.987254+0000","last_unstale":"2023-10-16T14:10:41.989882+0000","last_undegraded":"2023-10-16T14:10:41.989882+0000","last_fullsized":"2023-10-16T14:10:41.989882+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.463597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980939+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:40.980939+0000","last_peered":"2023-10-16T14:10:40.980939+0000","last_clean":"2023-10-16T14:10:40.980939+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:40.980939+0000","last_undegraded":"2023-10-16T14:10:40.980939+0000","last_fullsized":"2023-10-16T14:10:40.980939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124331+0000","last_change":"2023-10-16T14:10:40.990967+0000","last_active":"2023-10-16T14:10:42.124331+0000","last_peered":"2023-10-16T14:10:42.124331+0000","last_clean":"2023-10-16T14:10:42.124331+0000","last_became_active":"2023-10-16T14:10:40.989709+0000","last_became_peered":"2023-10-16T14:10:40.989709+0000","last_unstale":"2023-10-16T14:10:42.124331+0000","last_undegraded":"2023-10-16T14:10:42.124331+0000","last_fullsized":"2023-10-16T14:10:42.124331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.888217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974374+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:40.974374+0000","last_peered":"2023-10-16T14:10:40.974374+0000","last_clean":"2023-10-16T14:10:40.974374+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:40.974374+0000","last_undegraded":"2023-10-16T14:10:40.974374+0000","last_fullsized":"2023-10-16T14:10:40.974374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244140+0000","last_change":"2023-10-16T14:10:40.985097+0000","last_active":"2023-10-16T14:10:43.244140+0000","last_peered":"2023-10-16T14:10:43.244140+0000","last_clean":"2023-10-16T14:10:43.244140+0000","last_became_active":"2023-10-16T14:10:40.984409+0000","last_became_peered":"2023-10-16T14:10:40.984409+0000","last_unstale":"2023-10-16T14:10:43.244140+0000","last_undegraded":"2023-10-16T14:10:43.244140+0000","last_fullsized":"2023-10-16T14:10:43.244140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972580+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:40.972580+0000","last_peered":"2023-10-16T14:10:40.972580+0000","last_clean":"2023-10-16T14:10:40.972580+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:40.972580+0000","last_undegraded":"2023-10-16T14:10:40.972580+0000","last_fullsized":"2023-10-16T14:10:40.972580+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996436+0000","last_change":"2023-10-16T14:10:40.988348+0000","last_active":"2023-10-16T14:10:41.996436+0000","last_peered":"2023-10-16T14:10:41.996436+0000","last_clean":"2023-10-16T14:10:41.996436+0000","last_became_active":"2023-10-16T14:10:40.988060+0000","last_became_peered":"2023-10-16T14:10:40.988060+0000","last_unstale":"2023-10-16T14:10:41.996436+0000","last_undegraded":"2023-10-16T14:10:41.996436+0000","last_fullsized":"2023-10-16T14:10:41.996436+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179686+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:29.179686+0000","last_peered":"2023-10-16T14:10:29.179686+0000","last_clean":"2023-10-16T14:10:29.179686+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:29.179686+0000","last_undegraded":"2023-10-16T14:10:29.179686+0000","last_fullsized":"2023-10-16T14:10:29.179686+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243319+0000","last_change":"2023-10-16T14:10:40.987085+0000","last_active":"2023-10-16T14:10:43.243319+0000","last_peered":"2023-10-16T14:10:43.243319+0000","last_clean":"2023-10-16T14:10:43.243319+0000","last_became_active":"2023-10-16T14:10:40.986831+0000","last_became_peered":"2023-10-16T14:10:40.986831+0000","last_unstale":"2023-10-16T14:10:43.243319+0000","last_undegraded":"2023-10-16T14:10:43.243319+0000","last_fullsized":"2023-10-16T14:10:43.243319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:58.536370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179817+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:29.179817+0000","last_peered":"2023-10-16T14:10:29.179817+0000","last_clean":"2023-10-16T14:10:29.179817+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:29.179817+0000","last_undegraded":"2023-10-16T14:10:29.179817+0000","last_fullsized":"2023-10-16T14:10:29.179817+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125634+0000","last_change":"2023-10-16T14:10:40.998279+0000","last_active":"2023-10-16T14:10:42.125634+0000","last_peered":"2023-10-16T14:10:42.125634+0000","last_clean":"2023-10-16T14:10:42.125634+0000","last_became_active":"2023-10-16T14:10:40.997899+0000","last_became_peered":"2023-10-16T14:10:40.997899+0000","last_unstale":"2023-10-16T14:10:42.125634+0000","last_undegraded":"2023-10-16T14:10:42.125634+0000","last_fullsized":"2023-10-16T14:10:42.125634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:02.766931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977306+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:40.977306+0000","last_peered":"2023-10-16T14:10:40.977306+0000","last_clean":"2023-10-16T14:10:40.977306+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:40.977306+0000","last_undegraded":"2023-10-16T14:10:40.977306+0000","last_fullsized":"2023-10-16T14:10:40.977306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.991518+0000","last_change":"2023-10-16T14:10:40.992264+0000","last_active":"2023-10-16T14:10:41.991518+0000","last_peered":"2023-10-16T14:10:41.991518+0000","last_clean":"2023-10-16T14:10:41.991518+0000","last_became_active":"2023-10-16T14:10:40.991698+0000","last_became_peered":"2023-10-16T14:10:40.991698+0000","last_unstale":"2023-10-16T14:10:41.991518+0000","last_undegraded":"2023-10-16T14:10:41.991518+0000","last_fullsized":"2023-10-16T14:10:41.991518+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:16.407382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978952+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:40.978952+0000","last_peered":"2023-10-16T14:10:40.978952+0000","last_clean":"2023-10-16T14:10:40.978952+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:40.978952+0000","last_undegraded":"2023-10-16T14:10:40.978952+0000","last_fullsized":"2023-10-16T14:10:40.978952+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125182+0000","last_change":"2023-10-16T14:10:40.991982+0000","last_active":"2023-10-16T14:10:42.125182+0000","last_peered":"2023-10-16T14:10:42.125182+0000","last_clean":"2023-10-16T14:10:42.125182+0000","last_became_active":"2023-10-16T14:10:40.991521+0000","last_became_peered":"2023-10-16T14:10:40.991521+0000","last_unstale":"2023-10-16T14:10:42.125182+0000","last_undegraded":"2023-10-16T14:10:42.125182+0000","last_fullsized":"2023-10-16T14:10:42.125182+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:01.654954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961133+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:40.961133+0000","last_peered":"2023-10-16T14:10:40.961133+0000","last_clean":"2023-10-16T14:10:40.961133+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:40.961133+0000","last_undegraded":"2023-10-16T14:10:40.961133+0000","last_fullsized":"2023-10-16T14:10:40.961133+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124085+0000","last_change":"2023-10-16T14:10:40.992568+0000","last_active":"2023-10-16T14:10:42.124085+0000","last_peered":"2023-10-16T14:10:42.124085+0000","last_clean":"2023-10-16T14:10:42.124085+0000","last_became_active":"2023-10-16T14:10:40.992252+0000","last_became_peered":"2023-10-16T14:10:40.992252+0000","last_unstale":"2023-10-16T14:10:42.124085+0000","last_undegraded":"2023-10-16T14:10:42.124085+0000","last_fullsized":"2023-10-16T14:10:42.124085+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:25.262205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965644+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:40.965644+0000","last_peered":"2023-10-16T14:10:40.965644+0000","last_clean":"2023-10-16T14:10:40.965644+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:40.965644+0000","last_undegraded":"2023-10-16T14:10:40.965644+0000","last_fullsized":"2023-10-16T14:10:40.965644+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242786+0000","last_change":"2023-10-16T14:10:40.991785+0000","last_active":"2023-10-16T14:10:43.242786+0000","last_peered":"2023-10-16T14:10:43.242786+0000","last_clean":"2023-10-16T14:10:43.242786+0000","last_became_active":"2023-10-16T14:10:40.991292+0000","last_became_peered":"2023-10-16T14:10:40.991292+0000","last_unstale":"2023-10-16T14:10:43.242786+0000","last_undegraded":"2023-10-16T14:10:43.242786+0000","last_fullsized":"2023-10-16T14:10:43.242786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978297+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:40.978297+0000","last_peered":"2023-10-16T14:10:40.978297+0000","last_clean":"2023-10-16T14:10:40.978297+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:40.978297+0000","last_undegraded":"2023-10-16T14:10:40.978297+0000","last_fullsized":"2023-10-16T14:10:40.978297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243833+0000","last_change":"2023-10-16T14:10:40.985314+0000","last_active":"2023-10-16T14:10:43.243833+0000","last_peered":"2023-10-16T14:10:43.243833+0000","last_clean":"2023-10-16T14:10:43.243833+0000","last_became_active":"2023-10-16T14:10:40.984808+0000","last_became_peered":"2023-10-16T14:10:40.984808+0000","last_unstale":"2023-10-16T14:10:43.243833+0000","last_undegraded":"2023-10-16T14:10:43.243833+0000","last_fullsized":"2023-10-16T14:10:43.243833+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959800+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:40.959800+0000","last_peered":"2023-10-16T14:10:40.959800+0000","last_clean":"2023-10-16T14:10:40.959800+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:40.959800+0000","last_undegraded":"2023-10-16T14:10:40.959800+0000","last_fullsized":"2023-10-16T14:10:40.959800+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978097+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:40.978097+0000","last_peered":"2023-10-16T14:10:40.978097+0000","last_clean":"2023-10-16T14:10:40.978097+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:40.978097+0000","last_undegraded":"2023-10-16T14:10:40.978097+0000","last_fullsized":"2023-10-16T14:10:40.978097+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980317+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:40.980317+0000","last_peered":"2023-10-16T14:10:40.980317+0000","last_clean":"2023-10-16T14:10:40.980317+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:40.980317+0000","last_undegraded":"2023-10-16T14:10:40.980317+0000","last_fullsized":"2023-10-16T14:10:40.980317+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243931+0000","last_change":"2023-10-16T14:10:40.994995+0000","last_active":"2023-10-16T14:10:43.243931+0000","last_peered":"2023-10-16T14:10:43.243931+0000","last_clean":"2023-10-16T14:10:43.243931+0000","last_became_active":"2023-10-16T14:10:40.994421+0000","last_became_peered":"2023-10-16T14:10:40.994421+0000","last_unstale":"2023-10-16T14:10:43.243931+0000","last_undegraded":"2023-10-16T14:10:43.243931+0000","last_fullsized":"2023-10-16T14:10:43.243931+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:39.108794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179499+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:29.179499+0000","last_peered":"2023-10-16T14:10:29.179499+0000","last_clean":"2023-10-16T14:10:29.179499+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:29.179499+0000","last_undegraded":"2023-10-16T14:10:29.179499+0000","last_fullsized":"2023-10-16T14:10:29.179499+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995533+0000","last_change":"2023-10-16T14:10:40.988474+0000","last_active":"2023-10-16T14:10:41.995533+0000","last_peered":"2023-10-16T14:10:41.995533+0000","last_clean":"2023-10-16T14:10:41.995533+0000","last_became_active":"2023-10-16T14:10:40.988193+0000","last_became_peered":"2023-10-16T14:10:40.988193+0000","last_unstale":"2023-10-16T14:10:41.995533+0000","last_undegraded":"2023-10-16T14:10:41.995533+0000","last_fullsized":"2023-10-16T14:10:41.995533+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179611+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:29.179611+0000","last_peered":"2023-10-16T14:10:29.179611+0000","last_clean":"2023-10-16T14:10:29.179611+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:29.179611+0000","last_undegraded":"2023-10-16T14:10:29.179611+0000","last_fullsized":"2023-10-16T14:10:29.179611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981697+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:40.981697+0000","last_peered":"2023-10-16T14:10:40.981697+0000","last_clean":"2023-10-16T14:10:40.981697+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:40.981697+0000","last_undegraded":"2023-10-16T14:10:40.981697+0000","last_fullsized":"2023-10-16T14:10:40.981697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965582+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:40.965582+0000","last_peered":"2023-10-16T14:10:40.965582+0000","last_clean":"2023-10-16T14:10:40.965582+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:40.965582+0000","last_undegraded":"2023-10-16T14:10:40.965582+0000","last_fullsized":"2023-10-16T14:10:40.965582+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989881+0000","last_change":"2023-10-16T14:10:40.987206+0000","last_active":"2023-10-16T14:10:41.989881+0000","last_peered":"2023-10-16T14:10:41.989881+0000","last_clean":"2023-10-16T14:10:41.989881+0000","last_became_active":"2023-10-16T14:10:40.986211+0000","last_became_peered":"2023-10-16T14:10:40.986211+0000","last_unstale":"2023-10-16T14:10:41.989881+0000","last_undegraded":"2023-10-16T14:10:41.989881+0000","last_fullsized":"2023-10-16T14:10:41.989881+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974724+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:40.974724+0000","last_peered":"2023-10-16T14:10:40.974724+0000","last_clean":"2023-10-16T14:10:40.974724+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:40.974724+0000","last_undegraded":"2023-10-16T14:10:40.974724+0000","last_fullsized":"2023-10-16T14:10:40.974724+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968589+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:40.968589+0000","last_peered":"2023-10-16T14:10:40.968589+0000","last_clean":"2023-10-16T14:10:40.968589+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:40.968589+0000","last_undegraded":"2023-10-16T14:10:40.968589+0000","last_fullsized":"2023-10-16T14:10:40.968589+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244429+0000","last_change":"2023-10-16T14:10:41.001901+0000","last_active":"2023-10-16T14:10:43.244429+0000","last_peered":"2023-10-16T14:10:43.244429+0000","last_clean":"2023-10-16T14:10:43.244429+0000","last_became_active":"2023-10-16T14:10:41.001390+0000","last_became_peered":"2023-10-16T14:10:41.001390+0000","last_unstale":"2023-10-16T14:10:43.244429+0000","last_undegraded":"2023-10-16T14:10:43.244429+0000","last_fullsized":"2023-10-16T14:10:43.244429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:38.275860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977891+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:40.977891+0000","last_peered":"2023-10-16T14:10:40.977891+0000","last_clean":"2023-10-16T14:10:40.977891+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:40.977891+0000","last_undegraded":"2023-10-16T14:10:40.977891+0000","last_fullsized":"2023-10-16T14:10:40.977891+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124855+0000","last_change":"2023-10-16T14:10:40.990936+0000","last_active":"2023-10-16T14:10:42.124855+0000","last_peered":"2023-10-16T14:10:42.124855+0000","last_clean":"2023-10-16T14:10:42.124855+0000","last_became_active":"2023-10-16T14:10:40.990264+0000","last_became_peered":"2023-10-16T14:10:40.990264+0000","last_unstale":"2023-10-16T14:10:42.124855+0000","last_undegraded":"2023-10-16T14:10:42.124855+0000","last_fullsized":"2023-10-16T14:10:42.124855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:39.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973916+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:40.973916+0000","last_peered":"2023-10-16T14:10:40.973916+0000","last_clean":"2023-10-16T14:10:40.973916+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:40.973916+0000","last_undegraded":"2023-10-16T14:10:40.973916+0000","last_fullsized":"2023-10-16T14:10:40.973916+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973460+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:40.973460+0000","last_peered":"2023-10-16T14:10:40.973460+0000","last_clean":"2023-10-16T14:10:40.973460+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:40.973460+0000","last_undegraded":"2023-10-16T14:10:40.973460+0000","last_fullsized":"2023-10-16T14:10:40.973460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.241703+0000","last_change":"2023-10-16T14:10:40.982851+0000","last_active":"2023-10-16T14:10:43.241703+0000","last_peered":"2023-10-16T14:10:43.241703+0000","last_clean":"2023-10-16T14:10:43.241703+0000","last_became_active":"2023-10-16T14:10:40.981261+0000","last_became_peered":"2023-10-16T14:10:40.981261+0000","last_unstale":"2023-10-16T14:10:43.241703+0000","last_undegraded":"2023-10-16T14:10:43.241703+0000","last_fullsized":"2023-10-16T14:10:43.241703+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:03.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982369+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:40.982369+0000","last_peered":"2023-10-16T14:10:40.982369+0000","last_clean":"2023-10-16T14:10:40.982369+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:40.982369+0000","last_undegraded":"2023-10-16T14:10:40.982369+0000","last_fullsized":"2023-10-16T14:10:40.982369+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125338+0000","last_change":"2023-10-16T14:10:40.991807+0000","last_active":"2023-10-16T14:10:42.125338+0000","last_peered":"2023-10-16T14:10:42.125338+0000","last_clean":"2023-10-16T14:10:42.125338+0000","last_became_active":"2023-10-16T14:10:40.991143+0000","last_became_peered":"2023-10-16T14:10:40.991143+0000","last_unstale":"2023-10-16T14:10:42.125338+0000","last_undegraded":"2023-10-16T14:10:42.125338+0000","last_fullsized":"2023-10-16T14:10:42.125338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.405248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981437+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:40.981437+0000","last_peered":"2023-10-16T14:10:40.981437+0000","last_clean":"2023-10-16T14:10:40.981437+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:40.981437+0000","last_undegraded":"2023-10-16T14:10:40.981437+0000","last_fullsized":"2023-10-16T14:10:40.981437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185397+0000","last_change":"2023-10-16T14:10:40.997929+0000","last_active":"2023-10-16T14:10:43.185397+0000","last_peered":"2023-10-16T14:10:43.185397+0000","last_clean":"2023-10-16T14:10:43.185397+0000","last_became_active":"2023-10-16T14:10:40.997578+0000","last_became_peered":"2023-10-16T14:10:40.997578+0000","last_unstale":"2023-10-16T14:10:43.185397+0000","last_undegraded":"2023-10-16T14:10:43.185397+0000","last_fullsized":"2023-10-16T14:10:43.185397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:31.595742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968740+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:40.968740+0000","last_peered":"2023-10-16T14:10:40.968740+0000","last_clean":"2023-10-16T14:10:40.968740+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:40.968740+0000","last_undegraded":"2023-10-16T14:10:40.968740+0000","last_fullsized":"2023-10-16T14:10:40.968740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978187+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:40.978187+0000","last_peered":"2023-10-16T14:10:40.978187+0000","last_clean":"2023-10-16T14:10:40.978187+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:40.978187+0000","last_undegraded":"2023-10-16T14:10:40.978187+0000","last_fullsized":"2023-10-16T14:10:40.978187+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":88,"acting":88,"num_store_stats":6},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2388,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543301,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3551232,"data_stored":2031342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":1.4139999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.589}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.4730000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.327}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.087},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.3779999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.228},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.5329999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543301,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42336,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329504,"statfs":{"total":96636764160,"available":96593412096,"internally_reserved":0,"allocated":2895872,"data_stored":1410006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":1.8460000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.6279999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":1.405},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.5069999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.034},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.776}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.679}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":1.339},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.278}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.548},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.5920000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543303,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":2945024,"data_stored":1438629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.02}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.1799999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.254}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.286},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.8080000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.9590000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.4590000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543302,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47908,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323932,"statfs":{"total":96636764160,"available":96587706368,"internally_reserved":0,"allocated":2912256,"data_stored":1412385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.8420000000000001},"last":1.075},{"interface":"front","average":{"1min":1.121,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6699999999999999,"5min":1.8160000000000001,"15min":1.9019999999999999},"last":1.329}]},{"osd":1,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.78},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2549999999999999,"15min":1.294},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.754,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.7250000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.286,"15min":1.3169999999999999},"min":{"1min":0.80200000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.714,"5min":1.714,"15min":1.865},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.367,"15min":1.387},"min":{"1min":1.026,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.276}]},{"osd":3,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.296,"15min":1.3120000000000001},"min":{"1min":0.94599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.615,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.627},{"interface":"front","average":{"1min":1.607,"5min":1.4590000000000001,"15min":1.4299999999999999},"min":{"1min":0.96499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":5.2350000000000003,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.1930000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.329,"15min":1.3400000000000001},"min":{"1min":0.99199999999999999,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3580000000000001,"15min":1.3360000000000001},"min":{"1min":1.0389999999999999,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":1.921,"15min":1.921},"last":1.7629999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3919999999999999,"15min":1.3879999999999999},"min":{"1min":1.004,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.7689999999999999,"5min":1.9159999999999999,"15min":1.956},"last":1.581},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.502,"15min":1.5229999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.8069999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5349999999999999,"15min":1.5049999999999999},"min":{"1min":1.137,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.1949999999999998},"last":1.857},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6379999999999999,"15min":1.6319999999999999},"min":{"1min":1.391,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":1.889,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.6799999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543303,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52128,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319712,"statfs":{"total":96636764160,"available":96583385088,"internally_reserved":0,"allocated":2969600,"data_stored":1439911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":1.05}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":4.1299999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":3.8340000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":0.89300000000000002}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.01}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":2.2799999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":2.569}]}]},{"osd":2,"up_from":14,"seq":60129543301,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":2949120,"data_stored":1432548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":1.337},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.115},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":1.4159999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.268},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.0469999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.103},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":1.397}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.1899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543301,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2953216,"data_stored":1438193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.0740000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":0.752},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.1870000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":1.532}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":1.454},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":2.0979999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.867},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.7969999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543302,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":3547136,"data_stored":2029177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.93300000000000005,"15min":0.93999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.218,"5min":1.704,"15min":1.891},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.1579999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48699999999999999},"max":{"1min":1.627,"5min":1.659,"15min":1.9299999999999999},"last":0.90700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.1499999999999999,"15min":1.171},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5349999999999999,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.1899999999999999,"15min":1.226},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.6539999999999999,"5min":1.893,"15min":1.893},"last":1.0760000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1639999999999999,"15min":1.163},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7390000000000001,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3380000000000001,"15min":1.3759999999999999},"min":{"1min":0.64700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.679,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.2969999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3340000000000001,"15min":1.3799999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6140000000000001,"5min":1.819,"15min":2.048},"last":1.002},{"interface":"front","average":{"1min":1.379,"5min":1.518,"15min":1.524},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.0009999999999999,"5min":2.601,"15min":2.601},"last":1.4850000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.48,"15min":1.5069999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.821,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.452,"15min":1.492},"min":{"1min":0.98199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0169999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.593,"15min":1.621},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9490000000000001,"5min":2.669,"15min":2.669},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.55,"15min":1.5980000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2589999999999999,"5min":2.355,"15min":2.3769999999999998},"last":1.778}]},{"osd":7,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.655,"15min":1.712},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1549999999999998,"5min":2.484,"15min":2.5270000000000001},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.625,"15min":1.649},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.464},"last":1.056}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:45.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:45.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.788+0000 7fb381e13640 1 Processor -- start 2023-10-16T14:10:45.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.788+0000 7fb381e13640 1 -- start start 2023-10-16T14:10:45.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381e13640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb37c15c9f0 0x7fb37c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381e13640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb37c15dde0 0x7fb37c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381e13640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37c160a90 con 0x7fb37c0c9230 2023-10-16T14:10:45.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381e13640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37c0cdec0 con 0x7fb37c15dde0 2023-10-16T14:10:45.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381e13640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37c0cd900 con 0x7fb37c15c9f0 2023-10-16T14:10:45.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381612640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb37c15dde0 0x7fb37c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb37bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb37c0c9230 0x7fb37c0709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37870/0 (socket says 172.21.15.60:37870) 2023-10-16T14:10:45.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb381612640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb37c15dde0 0x7fb37c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38078/0 (socket says 172.21.15.60:38078) 2023-10-16T14:10:45.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.790+0000 7fb37bfff640 1 -- 172.21.15.60:0/1162659653 learned_addr learned my addr 172.21.15.60:0/1162659653 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:45.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.791+0000 7fb381612640 1 -- 172.21.15.60:0/1162659653 >> v2:172.21.15.144:3300/0 conn(0x7fb37c15c9f0 msgr2=0x7fb37c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.791+0000 7fb380e11640 1 --2- 172.21.15.60:0/1162659653 >> v2:172.21.15.144:3300/0 conn(0x7fb37c15c9f0 0x7fb37c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.791+0000 7fb381612640 1 --2- 172.21.15.60:0/1162659653 >> v2:172.21.15.144:3300/0 conn(0x7fb37c15c9f0 0x7fb37c0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.791+0000 7fb381612640 1 -- 172.21.15.60:0/1162659653 >> v1:172.21.15.60:6789/0 conn(0x7fb37c0c9230 legacy=0x7fb37c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.791+0000 7fb381612640 1 -- 172.21.15.60:0/1162659653 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c0cdbe0 con 0x7fb37c15dde0 2023-10-16T14:10:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.791+0000 7fb37b7fe640 1 -- 172.21.15.60:0/1162659653 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565616239 0 0) 0x7fb37c160a90 con 0x7fb37c0c9230 2023-10-16T14:10:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.792+0000 7fb381612640 1 --2- 172.21.15.60:0/1162659653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0ac890 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0ad4e0 tx=0x7fb37402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbed0faf21912bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb37b7fe640 1 -- 172.21.15.60:0/1162659653 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37402cc80 con 0x7fb37c15dde0 2023-10-16T14:10:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb37b7fe640 1 -- 172.21.15.60:0/1162659653 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37402cde0 con 0x7fb37c15dde0 2023-10-16T14:10:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb37b7fe640 1 -- 172.21.15.60:0/1162659653 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374034660 con 0x7fb37c15dde0 2023-10-16T14:10:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb381e13640 1 -- 172.21.15.60:0/1162659653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 msgr2=0x7fb37c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb381e13640 1 --2- 172.21.15.60:0/1162659653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0ac890 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0ad4e0 tx=0x7fb37402bed0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb381e13640 1 -- 172.21.15.60:0/1162659653 shutdown_connections 2023-10-16T14:10:45.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb381e13640 1 --2- 172.21.15.60:0/1162659653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0ac890 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb381e13640 1 --2- 172.21.15.60:0/1162659653 >> v2:172.21.15.144:3300/0 conn(0x7fb37c15c9f0 0x7fb37c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.793+0000 7fb381e13640 1 -- 172.21.15.60:0/1162659653 >> 172.21.15.60:0/1162659653 conn(0x7fb37c0ce6a0 msgr2=0x7fb37c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:45.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.794+0000 7fb381e13640 1 -- 172.21.15.60:0/1162659653 shutdown_connections 2023-10-16T14:10:45.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.794+0000 7fb381e13640 1 -- 172.21.15.60:0/1162659653 wait complete. 2023-10-16T14:10:45.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.795+0000 7fb381e13640 1 Processor -- start 2023-10-16T14:10:45.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.796+0000 7fb381e13640 1 -- start start 2023-10-16T14:10:45.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381e13640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb37c0c9230 0x7fb37c06af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381e13640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb37c15c9f0 0x7fb37c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381e13640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381e13640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c0ae100 con 0x7fb37c15dde0 2023-10-16T14:10:45.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381e13640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c1596e0 con 0x7fb37c15c9f0 2023-10-16T14:10:45.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381e13640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c06c130 con 0x7fb37c0c9230 2023-10-16T14:10:45.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38092/0 (socket says 172.21.15.60:38092) 2023-10-16T14:10:45.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 -- 172.21.15.60:0/803482518 learned_addr learned my addr 172.21.15.60:0/803482518 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:45.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb380e11640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb37c0c9230 0x7fb37c06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb37bfff640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb37c15c9f0 0x7fb37c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 -- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb37c0c9230 msgr2=0x7fb37c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb37c0c9230 0x7fb37c06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 -- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb37c15c9f0 msgr2=0x7fb37c1591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:45.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.797+0000 7fb381612640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb37c15c9f0 0x7fb37c1591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:45.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb381612640 1 -- 172.21.15.60:0/803482518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374009a70 con 0x7fb37c15dde0 2023-10-16T14:10:45.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb381612640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0cd0a0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fb374009e80 tx=0x7fb374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:45.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374040650 con 0x7fb37c15dde0 2023-10-16T14:10:45.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3740407b0 con 0x7fb37c15dde0 2023-10-16T14:10:45.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374040b60 con 0x7fb37c15dde0 2023-10-16T14:10:45.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c0cd5e0 con 0x7fb37c15dde0 2023-10-16T14:10:45.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.798+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c06b890 con 0x7fb37c15dde0 2023-10-16T14:10:45.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.800+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c0ae270 con 0x7fb37c15dde0 2023-10-16T14:10:45.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.801+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb374040cc0 con 0x7fb37c15dde0 2023-10-16T14:10:45.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.801+0000 7fb3797fa640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb36006af90 0x7fb36006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:45.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.802+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fb3740ac8d0 con 0x7fb37c15dde0 2023-10-16T14:10:45.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.802+0000 7fb380e11640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb36006af90 0x7fb36006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:45.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.803+0000 7fb380e11640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb36006af90 0x7fb36006d450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb37000a970 tx=0x7fb370008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:45.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.806+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb37c0ae270 con 0x7fb37c15dde0 2023-10-16T14:10:46.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:45.997+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb37c0704c0 con 0x7fb36006af90 2023-10-16T14:10:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.040+0000 7fb3797fa640 1 -- 172.21.15.60:0/803482518 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+522429 (secure 0 0 0) 0x7fb37000c050 con 0x7fb36006af90 2023-10-16T14:10:46.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:46.055 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:46.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.047+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb36006af90 msgr2=0x7fb36006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.047+0000 7fb381e13640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb36006af90 0x7fb36006d450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb37000a970 tx=0x7fb370008040 comp rx=0 tx=0).stop 2023-10-16T14:10:46.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 msgr2=0x7fb37c0cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0cd0a0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fb374009e80 tx=0x7fb374004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:46.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 shutdown_connections 2023-10-16T14:10:46.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb36006af90 0x7fb36006d450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb37c15dde0 0x7fb37c0cd0a0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb37c15c9f0 0x7fb37c1591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 --2- 172.21.15.60:0/803482518 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb37c0c9230 0x7fb37c06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 >> 172.21.15.60:0/803482518 conn(0x7fb37c0ce6a0 msgr2=0x7fb37c0708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:46.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 shutdown_connections 2023-10-16T14:10:46.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.048+0000 7fb381e13640 1 -- 172.21.15.60:0/803482518 wait complete. 2023-10-16T14:10:46.075 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-16T14:10:44.182486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596411,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":657,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":346,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377744,"kb_used_data":24144,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754596976,"statfs":{"total":773094113280,"available":772707303424,"internally_reserved":0,"allocated":24723456,"data_stored":12632191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3633,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.250307"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244206+0000","last_change":"2023-10-16T14:10:40.992171+0000","last_active":"2023-10-16T14:10:43.244206+0000","last_peered":"2023-10-16T14:10:43.244206+0000","last_clean":"2023-10-16T14:10:43.244206+0000","last_became_active":"2023-10-16T14:10:40.991865+0000","last_became_peered":"2023-10-16T14:10:40.991865+0000","last_unstale":"2023-10-16T14:10:43.244206+0000","last_undegraded":"2023-10-16T14:10:43.244206+0000","last_fullsized":"2023-10-16T14:10:43.244206+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:36.935225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1226,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974179+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:40.974179+0000","last_peered":"2023-10-16T14:10:40.974179+0000","last_clean":"2023-10-16T14:10:40.974179+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:40.974179+0000","last_undegraded":"2023-10-16T14:10:40.974179+0000","last_fullsized":"2023-10-16T14:10:40.974179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":14,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179564+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:29.179564+0000","last_peered":"2023-10-16T14:10:29.179564+0000","last_clean":"2023-10-16T14:10:29.179564+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:29.179564+0000","last_undegraded":"2023-10-16T14:10:29.179564+0000","last_fullsized":"2023-10-16T14:10:29.179564+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980445+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:40.980445+0000","last_peered":"2023-10-16T14:10:40.980445+0000","last_clean":"2023-10-16T14:10:40.980445+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:40.980445+0000","last_undegraded":"2023-10-16T14:10:40.980445+0000","last_fullsized":"2023-10-16T14:10:40.980445+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008653+0000","last_change":"2023-10-16T14:10:39.986399+0000","last_active":"2023-10-16T14:10:41.008653+0000","last_peered":"2023-10-16T14:10:41.008653+0000","last_clean":"2023-10-16T14:10:41.008653+0000","last_became_active":"2023-10-16T14:10:39.986098+0000","last_became_peered":"2023-10-16T14:10:39.986098+0000","last_unstale":"2023-10-16T14:10:41.008653+0000","last_undegraded":"2023-10-16T14:10:41.008653+0000","last_fullsized":"2023-10-16T14:10:41.008653+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:15.423316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179310+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:29.179310+0000","last_peered":"2023-10-16T14:10:29.179310+0000","last_clean":"2023-10-16T14:10:29.179310+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:29.179310+0000","last_undegraded":"2023-10-16T14:10:29.179310+0000","last_fullsized":"2023-10-16T14:10:29.179310+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970555+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:40.970555+0000","last_peered":"2023-10-16T14:10:40.970555+0000","last_clean":"2023-10-16T14:10:40.970555+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:40.970555+0000","last_undegraded":"2023-10-16T14:10:40.970555+0000","last_fullsized":"2023-10-16T14:10:40.970555+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979944+0000","last_change":"2023-10-16T14:10:39.988566+0000","last_active":"2023-10-16T14:10:40.979944+0000","last_peered":"2023-10-16T14:10:40.979944+0000","last_clean":"2023-10-16T14:10:40.979944+0000","last_became_active":"2023-10-16T14:10:39.988037+0000","last_became_peered":"2023-10-16T14:10:39.988037+0000","last_unstale":"2023-10-16T14:10:40.979944+0000","last_undegraded":"2023-10-16T14:10:40.979944+0000","last_fullsized":"2023-10-16T14:10:40.979944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.694200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125491+0000","last_change":"2023-10-16T14:10:40.987050+0000","last_active":"2023-10-16T14:10:42.125491+0000","last_peered":"2023-10-16T14:10:42.125491+0000","last_clean":"2023-10-16T14:10:42.125491+0000","last_became_active":"2023-10-16T14:10:40.986733+0000","last_became_peered":"2023-10-16T14:10:40.986733+0000","last_unstale":"2023-10-16T14:10:42.125491+0000","last_undegraded":"2023-10-16T14:10:42.125491+0000","last_fullsized":"2023-10-16T14:10:42.125491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:53.222315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179774+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:29.179774+0000","last_peered":"2023-10-16T14:10:29.179774+0000","last_clean":"2023-10-16T14:10:29.179774+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:29.179774+0000","last_undegraded":"2023-10-16T14:10:29.179774+0000","last_fullsized":"2023-10-16T14:10:29.179774+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975719+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:40.975719+0000","last_peered":"2023-10-16T14:10:40.975719+0000","last_clean":"2023-10-16T14:10:40.975719+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:40.975719+0000","last_undegraded":"2023-10-16T14:10:40.975719+0000","last_fullsized":"2023-10-16T14:10:40.975719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008672+0000","last_change":"2023-10-16T14:10:39.985490+0000","last_active":"2023-10-16T14:10:41.008672+0000","last_peered":"2023-10-16T14:10:41.008672+0000","last_clean":"2023-10-16T14:10:41.008672+0000","last_became_active":"2023-10-16T14:10:39.985187+0000","last_became_peered":"2023-10-16T14:10:39.985187+0000","last_unstale":"2023-10-16T14:10:41.008672+0000","last_undegraded":"2023-10-16T14:10:41.008672+0000","last_fullsized":"2023-10-16T14:10:41.008672+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:32.423740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125498+0000","last_change":"2023-10-16T14:10:40.989384+0000","last_active":"2023-10-16T14:10:42.125498+0000","last_peered":"2023-10-16T14:10:42.125498+0000","last_clean":"2023-10-16T14:10:42.125498+0000","last_became_active":"2023-10-16T14:10:40.989064+0000","last_became_peered":"2023-10-16T14:10:40.989064+0000","last_unstale":"2023-10-16T14:10:42.125498+0000","last_undegraded":"2023-10-16T14:10:42.125498+0000","last_fullsized":"2023-10-16T14:10:42.125498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:50.164078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974785+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:40.974785+0000","last_peered":"2023-10-16T14:10:40.974785+0000","last_clean":"2023-10-16T14:10:40.974785+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:40.974785+0000","last_undegraded":"2023-10-16T14:10:40.974785+0000","last_fullsized":"2023-10-16T14:10:40.974785+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179447+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:29.179447+0000","last_peered":"2023-10-16T14:10:29.179447+0000","last_clean":"2023-10-16T14:10:29.179447+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:29.179447+0000","last_undegraded":"2023-10-16T14:10:29.179447+0000","last_fullsized":"2023-10-16T14:10:29.179447+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.952450+0000","last_change":"2023-10-16T14:10:38.952450+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.952450+0000","last_undegraded":"2023-10-16T14:10:38.952450+0000","last_fullsized":"2023-10-16T14:10:38.952450+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977551+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:40.977551+0000","last_peered":"2023-10-16T14:10:40.977551+0000","last_clean":"2023-10-16T14:10:40.977551+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:40.977551+0000","last_undegraded":"2023-10-16T14:10:40.977551+0000","last_fullsized":"2023-10-16T14:10:40.977551+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981376+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:40.981376+0000","last_peered":"2023-10-16T14:10:40.981376+0000","last_clean":"2023-10-16T14:10:40.981376+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:40.981376+0000","last_undegraded":"2023-10-16T14:10:40.981376+0000","last_fullsized":"2023-10-16T14:10:40.981376+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985469+0000","last_change":"2023-10-16T14:10:39.989343+0000","last_active":"2023-10-16T14:10:40.985469+0000","last_peered":"2023-10-16T14:10:40.985469+0000","last_clean":"2023-10-16T14:10:40.985469+0000","last_became_active":"2023-10-16T14:10:39.989043+0000","last_became_peered":"2023-10-16T14:10:39.989043+0000","last_unstale":"2023-10-16T14:10:40.985469+0000","last_undegraded":"2023-10-16T14:10:40.985469+0000","last_fullsized":"2023-10-16T14:10:40.985469+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:57.519740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979800+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:40.979800+0000","last_peered":"2023-10-16T14:10:40.979800+0000","last_clean":"2023-10-16T14:10:40.979800+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:40.979800+0000","last_undegraded":"2023-10-16T14:10:40.979800+0000","last_fullsized":"2023-10-16T14:10:40.979800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179954+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:29.179954+0000","last_peered":"2023-10-16T14:10:29.179954+0000","last_clean":"2023-10-16T14:10:29.179954+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:29.179954+0000","last_undegraded":"2023-10-16T14:10:29.179954+0000","last_fullsized":"2023-10-16T14:10:29.179954+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980253+0000","last_change":"2023-10-16T14:10:39.979128+0000","last_active":"2023-10-16T14:10:40.980253+0000","last_peered":"2023-10-16T14:10:40.980253+0000","last_clean":"2023-10-16T14:10:40.980253+0000","last_became_active":"2023-10-16T14:10:39.978782+0000","last_became_peered":"2023-10-16T14:10:39.978782+0000","last_unstale":"2023-10-16T14:10:40.980253+0000","last_undegraded":"2023-10-16T14:10:40.980253+0000","last_fullsized":"2023-10-16T14:10:40.980253+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:28.231837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971953+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:40.971953+0000","last_peered":"2023-10-16T14:10:40.971953+0000","last_clean":"2023-10-16T14:10:40.971953+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:40.971953+0000","last_undegraded":"2023-10-16T14:10:40.971953+0000","last_fullsized":"2023-10-16T14:10:40.971953+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972632+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:40.972632+0000","last_peered":"2023-10-16T14:10:40.972632+0000","last_clean":"2023-10-16T14:10:40.972632+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:40.972632+0000","last_undegraded":"2023-10-16T14:10:40.972632+0000","last_fullsized":"2023-10-16T14:10:40.972632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969118+0000","last_change":"2023-10-16T14:10:39.982849+0000","last_active":"2023-10-16T14:10:40.969118+0000","last_peered":"2023-10-16T14:10:40.969118+0000","last_clean":"2023-10-16T14:10:40.969118+0000","last_became_active":"2023-10-16T14:10:39.982560+0000","last_became_peered":"2023-10-16T14:10:39.982560+0000","last_unstale":"2023-10-16T14:10:40.969118+0000","last_undegraded":"2023-10-16T14:10:40.969118+0000","last_fullsized":"2023-10-16T14:10:40.969118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:51.220017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242743+0000","last_change":"2023-10-16T14:10:40.991901+0000","last_active":"2023-10-16T14:10:43.242743+0000","last_peered":"2023-10-16T14:10:43.242743+0000","last_clean":"2023-10-16T14:10:43.242743+0000","last_became_active":"2023-10-16T14:10:40.991610+0000","last_became_peered":"2023-10-16T14:10:40.991610+0000","last_unstale":"2023-10-16T14:10:43.242743+0000","last_undegraded":"2023-10-16T14:10:43.242743+0000","last_fullsized":"2023-10-16T14:10:43.242743+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"231'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979815+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:40.979815+0000","last_peered":"2023-10-16T14:10:40.979815+0000","last_clean":"2023-10-16T14:10:40.979815+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:40.979815+0000","last_undegraded":"2023-10-16T14:10:40.979815+0000","last_fullsized":"2023-10-16T14:10:40.979815+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978358+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:40.978358+0000","last_peered":"2023-10-16T14:10:40.978358+0000","last_clean":"2023-10-16T14:10:40.978358+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:40.978358+0000","last_undegraded":"2023-10-16T14:10:40.978358+0000","last_fullsized":"2023-10-16T14:10:40.978358+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.951042+0000","last_change":"2023-10-16T14:10:38.951042+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.951042+0000","last_undegraded":"2023-10-16T14:10:38.951042+0000","last_fullsized":"2023-10-16T14:10:38.951042+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980892+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:40.980892+0000","last_peered":"2023-10-16T14:10:40.980892+0000","last_clean":"2023-10-16T14:10:40.980892+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:40.980892+0000","last_undegraded":"2023-10-16T14:10:40.980892+0000","last_fullsized":"2023-10-16T14:10:40.980892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179694+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:29.179694+0000","last_peered":"2023-10-16T14:10:29.179694+0000","last_clean":"2023-10-16T14:10:29.179694+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:29.179694+0000","last_undegraded":"2023-10-16T14:10:29.179694+0000","last_fullsized":"2023-10-16T14:10:29.179694+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976136+0000","last_change":"2023-10-16T14:10:39.987697+0000","last_active":"2023-10-16T14:10:40.976136+0000","last_peered":"2023-10-16T14:10:40.976136+0000","last_clean":"2023-10-16T14:10:40.976136+0000","last_became_active":"2023-10-16T14:10:39.987388+0000","last_became_peered":"2023-10-16T14:10:39.987388+0000","last_unstale":"2023-10-16T14:10:40.976136+0000","last_undegraded":"2023-10-16T14:10:40.976136+0000","last_fullsized":"2023-10-16T14:10:40.976136+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:46.559129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244071+0000","last_change":"2023-10-16T14:10:40.983968+0000","last_active":"2023-10-16T14:10:43.244071+0000","last_peered":"2023-10-16T14:10:43.244071+0000","last_clean":"2023-10-16T14:10:43.244071+0000","last_became_active":"2023-10-16T14:10:40.983650+0000","last_became_peered":"2023-10-16T14:10:40.983650+0000","last_unstale":"2023-10-16T14:10:43.244071+0000","last_undegraded":"2023-10-16T14:10:43.244071+0000","last_fullsized":"2023-10-16T14:10:43.244071+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969088+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:40.969088+0000","last_peered":"2023-10-16T14:10:40.969088+0000","last_clean":"2023-10-16T14:10:40.969088+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:40.969088+0000","last_undegraded":"2023-10-16T14:10:40.969088+0000","last_fullsized":"2023-10-16T14:10:40.969088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179731+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:29.179731+0000","last_peered":"2023-10-16T14:10:29.179731+0000","last_clean":"2023-10-16T14:10:29.179731+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:29.179731+0000","last_undegraded":"2023-10-16T14:10:29.179731+0000","last_fullsized":"2023-10-16T14:10:29.179731+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009061+0000","last_change":"2023-10-16T14:10:39.982498+0000","last_active":"2023-10-16T14:10:41.009061+0000","last_peered":"2023-10-16T14:10:41.009061+0000","last_clean":"2023-10-16T14:10:41.009061+0000","last_became_active":"2023-10-16T14:10:39.982162+0000","last_became_peered":"2023-10-16T14:10:39.982162+0000","last_unstale":"2023-10-16T14:10:41.009061+0000","last_undegraded":"2023-10-16T14:10:41.009061+0000","last_fullsized":"2023-10-16T14:10:41.009061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:11.621670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.187140+0000","last_change":"2023-10-16T14:10:40.990434+0000","last_active":"2023-10-16T14:10:43.187140+0000","last_peered":"2023-10-16T14:10:43.187140+0000","last_clean":"2023-10-16T14:10:43.187140+0000","last_became_active":"2023-10-16T14:10:40.990134+0000","last_became_peered":"2023-10-16T14:10:40.990134+0000","last_unstale":"2023-10-16T14:10:43.187140+0000","last_undegraded":"2023-10-16T14:10:43.187140+0000","last_fullsized":"2023-10-16T14:10:43.187140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:35.937011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978483+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:40.978483+0000","last_peered":"2023-10-16T14:10:40.978483+0000","last_clean":"2023-10-16T14:10:40.978483+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:40.978483+0000","last_undegraded":"2023-10-16T14:10:40.978483+0000","last_fullsized":"2023-10-16T14:10:40.978483+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967437+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:40.967437+0000","last_peered":"2023-10-16T14:10:40.967437+0000","last_clean":"2023-10-16T14:10:40.967437+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:40.967437+0000","last_undegraded":"2023-10-16T14:10:40.967437+0000","last_fullsized":"2023-10-16T14:10:40.967437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.950829+0000","last_change":"2023-10-16T14:10:38.950829+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.950829+0000","last_undegraded":"2023-10-16T14:10:38.950829+0000","last_fullsized":"2023-10-16T14:10:38.950829+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981490+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:40.981490+0000","last_peered":"2023-10-16T14:10:40.981490+0000","last_clean":"2023-10-16T14:10:40.981490+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:40.981490+0000","last_undegraded":"2023-10-16T14:10:40.981490+0000","last_fullsized":"2023-10-16T14:10:40.981490+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981982+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:40.981982+0000","last_peered":"2023-10-16T14:10:40.981982+0000","last_clean":"2023-10-16T14:10:40.981982+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:40.981982+0000","last_undegraded":"2023-10-16T14:10:40.981982+0000","last_fullsized":"2023-10-16T14:10:40.981982+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961192+0000","last_change":"2023-10-16T14:10:39.981457+0000","last_active":"2023-10-16T14:10:40.961192+0000","last_peered":"2023-10-16T14:10:40.961192+0000","last_clean":"2023-10-16T14:10:40.961192+0000","last_became_active":"2023-10-16T14:10:39.981145+0000","last_became_peered":"2023-10-16T14:10:39.981145+0000","last_unstale":"2023-10-16T14:10:40.961192+0000","last_undegraded":"2023-10-16T14:10:40.961192+0000","last_fullsized":"2023-10-16T14:10:40.961192+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:33.143537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960165+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:40.960165+0000","last_peered":"2023-10-16T14:10:40.960165+0000","last_clean":"2023-10-16T14:10:40.960165+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:40.960165+0000","last_undegraded":"2023-10-16T14:10:40.960165+0000","last_fullsized":"2023-10-16T14:10:40.960165+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960903+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:40.960903+0000","last_peered":"2023-10-16T14:10:40.960903+0000","last_clean":"2023-10-16T14:10:40.960903+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:40.960903+0000","last_undegraded":"2023-10-16T14:10:40.960903+0000","last_fullsized":"2023-10-16T14:10:40.960903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010705+0000","last_change":"2023-10-16T14:10:39.984267+0000","last_active":"2023-10-16T14:10:41.010705+0000","last_peered":"2023-10-16T14:10:41.010705+0000","last_clean":"2023-10-16T14:10:41.010705+0000","last_became_active":"2023-10-16T14:10:39.983894+0000","last_became_peered":"2023-10-16T14:10:39.983894+0000","last_unstale":"2023-10-16T14:10:41.010705+0000","last_undegraded":"2023-10-16T14:10:41.010705+0000","last_fullsized":"2023-10-16T14:10:41.010705+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:02.427201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979684+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:40.979684+0000","last_peered":"2023-10-16T14:10:40.979684+0000","last_clean":"2023-10-16T14:10:40.979684+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:40.979684+0000","last_undegraded":"2023-10-16T14:10:40.979684+0000","last_fullsized":"2023-10-16T14:10:40.979684+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961306+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:40.961306+0000","last_peered":"2023-10-16T14:10:40.961306+0000","last_clean":"2023-10-16T14:10:40.961306+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:40.961306+0000","last_undegraded":"2023-10-16T14:10:40.961306+0000","last_fullsized":"2023-10-16T14:10:40.961306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971464+0000","last_change":"2023-10-16T14:10:39.974033+0000","last_active":"2023-10-16T14:10:40.971464+0000","last_peered":"2023-10-16T14:10:40.971464+0000","last_clean":"2023-10-16T14:10:40.971464+0000","last_became_active":"2023-10-16T14:10:39.973723+0000","last_became_peered":"2023-10-16T14:10:39.973723+0000","last_unstale":"2023-10-16T14:10:40.971464+0000","last_undegraded":"2023-10-16T14:10:40.971464+0000","last_fullsized":"2023-10-16T14:10:40.971464+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:08.572098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124517+0000","last_change":"2023-10-16T14:10:40.994623+0000","last_active":"2023-10-16T14:10:42.124517+0000","last_peered":"2023-10-16T14:10:42.124517+0000","last_clean":"2023-10-16T14:10:42.124517+0000","last_became_active":"2023-10-16T14:10:40.994347+0000","last_became_peered":"2023-10-16T14:10:40.994347+0000","last_unstale":"2023-10-16T14:10:42.124517+0000","last_undegraded":"2023-10-16T14:10:42.124517+0000","last_fullsized":"2023-10-16T14:10:42.124517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:18.643655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961579+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:40.961579+0000","last_peered":"2023-10-16T14:10:40.961579+0000","last_clean":"2023-10-16T14:10:40.961579+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:40.961579+0000","last_undegraded":"2023-10-16T14:10:40.961579+0000","last_fullsized":"2023-10-16T14:10:40.961579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979554+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:40.979554+0000","last_peered":"2023-10-16T14:10:40.979554+0000","last_clean":"2023-10-16T14:10:40.979554+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:40.979554+0000","last_undegraded":"2023-10-16T14:10:40.979554+0000","last_fullsized":"2023-10-16T14:10:40.979554+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.949894+0000","last_change":"2023-10-16T14:10:38.949894+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.949894+0000","last_undegraded":"2023-10-16T14:10:38.949894+0000","last_fullsized":"2023-10-16T14:10:38.949894+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124542+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:42.124542+0000","last_peered":"2023-10-16T14:10:42.124542+0000","last_clean":"2023-10-16T14:10:42.124542+0000","last_became_active":"2023-10-16T14:10:40.993130+0000","last_became_peered":"2023-10-16T14:10:40.993130+0000","last_unstale":"2023-10-16T14:10:42.124542+0000","last_undegraded":"2023-10-16T14:10:42.124542+0000","last_fullsized":"2023-10-16T14:10:42.124542+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:25.881584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973127+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:40.973127+0000","last_peered":"2023-10-16T14:10:40.973127+0000","last_clean":"2023-10-16T14:10:40.973127+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:40.973127+0000","last_undegraded":"2023-10-16T14:10:40.973127+0000","last_fullsized":"2023-10-16T14:10:40.973127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974510+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:40.974510+0000","last_peered":"2023-10-16T14:10:40.974510+0000","last_clean":"2023-10-16T14:10:40.974510+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:40.974510+0000","last_undegraded":"2023-10-16T14:10:40.974510+0000","last_fullsized":"2023-10-16T14:10:40.974510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.953322+0000","last_change":"2023-10-16T14:10:38.953322+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.953322+0000","last_undegraded":"2023-10-16T14:10:38.953322+0000","last_fullsized":"2023-10-16T14:10:38.953322+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185918+0000","last_change":"2023-10-16T14:10:40.993254+0000","last_active":"2023-10-16T14:10:43.185918+0000","last_peered":"2023-10-16T14:10:43.185918+0000","last_clean":"2023-10-16T14:10:43.185918+0000","last_became_active":"2023-10-16T14:10:40.992918+0000","last_became_peered":"2023-10-16T14:10:40.992918+0000","last_unstale":"2023-10-16T14:10:43.185918+0000","last_undegraded":"2023-10-16T14:10:43.185918+0000","last_fullsized":"2023-10-16T14:10:43.185918+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:07.376549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976318+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:40.976318+0000","last_peered":"2023-10-16T14:10:40.976318+0000","last_clean":"2023-10-16T14:10:40.976318+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:40.976318+0000","last_undegraded":"2023-10-16T14:10:40.976318+0000","last_fullsized":"2023-10-16T14:10:40.976318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972611+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:40.972611+0000","last_peered":"2023-10-16T14:10:40.972611+0000","last_clean":"2023-10-16T14:10:40.972611+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:40.972611+0000","last_undegraded":"2023-10-16T14:10:40.972611+0000","last_fullsized":"2023-10-16T14:10:40.972611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970330+0000","last_change":"2023-10-16T14:10:39.989051+0000","last_active":"2023-10-16T14:10:40.970330+0000","last_peered":"2023-10-16T14:10:40.970330+0000","last_clean":"2023-10-16T14:10:40.970330+0000","last_became_active":"2023-10-16T14:10:39.987944+0000","last_became_peered":"2023-10-16T14:10:39.987944+0000","last_unstale":"2023-10-16T14:10:40.970330+0000","last_undegraded":"2023-10-16T14:10:40.970330+0000","last_fullsized":"2023-10-16T14:10:40.970330+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244927+0000","last_change":"2023-10-16T14:10:40.996616+0000","last_active":"2023-10-16T14:10:43.244927+0000","last_peered":"2023-10-16T14:10:43.244927+0000","last_clean":"2023-10-16T14:10:43.244927+0000","last_became_active":"2023-10-16T14:10:40.996057+0000","last_became_peered":"2023-10-16T14:10:40.996057+0000","last_unstale":"2023-10-16T14:10:43.244927+0000","last_undegraded":"2023-10-16T14:10:43.244927+0000","last_fullsized":"2023-10-16T14:10:43.244927+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:32.328604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981955+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:40.981955+0000","last_peered":"2023-10-16T14:10:40.981955+0000","last_clean":"2023-10-16T14:10:40.981955+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:40.981955+0000","last_undegraded":"2023-10-16T14:10:40.981955+0000","last_fullsized":"2023-10-16T14:10:40.981955+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978301+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:40.978301+0000","last_peered":"2023-10-16T14:10:40.978301+0000","last_clean":"2023-10-16T14:10:40.978301+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:40.978301+0000","last_undegraded":"2023-10-16T14:10:40.978301+0000","last_fullsized":"2023-10-16T14:10:40.978301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980655+0000","last_change":"2023-10-16T14:10:39.972329+0000","last_active":"2023-10-16T14:10:40.980655+0000","last_peered":"2023-10-16T14:10:40.980655+0000","last_clean":"2023-10-16T14:10:40.980655+0000","last_became_active":"2023-10-16T14:10:39.972015+0000","last_became_peered":"2023-10-16T14:10:39.972015+0000","last_unstale":"2023-10-16T14:10:40.980655+0000","last_undegraded":"2023-10-16T14:10:40.980655+0000","last_fullsized":"2023-10-16T14:10:40.980655+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:17.467680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959024+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:40.959024+0000","last_peered":"2023-10-16T14:10:40.959024+0000","last_clean":"2023-10-16T14:10:40.959024+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:40.959024+0000","last_undegraded":"2023-10-16T14:10:40.959024+0000","last_fullsized":"2023-10-16T14:10:40.959024+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975612+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:40.975612+0000","last_peered":"2023-10-16T14:10:40.975612+0000","last_clean":"2023-10-16T14:10:40.975612+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:40.975612+0000","last_undegraded":"2023-10-16T14:10:40.975612+0000","last_fullsized":"2023-10-16T14:10:40.975612+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985242+0000","last_change":"2023-10-16T14:10:39.990904+0000","last_active":"2023-10-16T14:10:40.985242+0000","last_peered":"2023-10-16T14:10:40.985242+0000","last_clean":"2023-10-16T14:10:40.985242+0000","last_became_active":"2023-10-16T14:10:39.990155+0000","last_became_peered":"2023-10-16T14:10:39.990155+0000","last_unstale":"2023-10-16T14:10:40.985242+0000","last_undegraded":"2023-10-16T14:10:40.985242+0000","last_fullsized":"2023-10-16T14:10:40.985242+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:56.282178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243767+0000","last_change":"2023-10-16T14:10:40.993640+0000","last_active":"2023-10-16T14:10:43.243767+0000","last_peered":"2023-10-16T14:10:43.243767+0000","last_clean":"2023-10-16T14:10:43.243767+0000","last_became_active":"2023-10-16T14:10:40.993084+0000","last_became_peered":"2023-10-16T14:10:40.993084+0000","last_unstale":"2023-10-16T14:10:43.243767+0000","last_undegraded":"2023-10-16T14:10:43.243767+0000","last_fullsized":"2023-10-16T14:10:43.243767+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:12.061147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179859+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:29.179859+0000","last_peered":"2023-10-16T14:10:29.179859+0000","last_clean":"2023-10-16T14:10:29.179859+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:29.179859+0000","last_undegraded":"2023-10-16T14:10:29.179859+0000","last_fullsized":"2023-10-16T14:10:29.179859+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959827+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:40.959827+0000","last_peered":"2023-10-16T14:10:40.959827+0000","last_clean":"2023-10-16T14:10:40.959827+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:40.959827+0000","last_undegraded":"2023-10-16T14:10:40.959827+0000","last_fullsized":"2023-10-16T14:10:40.959827+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"234'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013944+0000","last_change":"2023-10-16T14:10:39.971085+0000","last_active":"2023-10-16T14:10:41.013944+0000","last_peered":"2023-10-16T14:10:41.013944+0000","last_clean":"2023-10-16T14:10:41.013944+0000","last_became_active":"2023-10-16T14:10:39.970777+0000","last_became_peered":"2023-10-16T14:10:39.970777+0000","last_unstale":"2023-10-16T14:10:41.013944+0000","last_undegraded":"2023-10-16T14:10:41.013944+0000","last_fullsized":"2023-10-16T14:10:41.013944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:03.090258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.998098+0000","last_change":"2023-10-16T14:10:40.985478+0000","last_active":"2023-10-16T14:10:41.998098+0000","last_peered":"2023-10-16T14:10:41.998098+0000","last_clean":"2023-10-16T14:10:41.998098+0000","last_became_active":"2023-10-16T14:10:40.985218+0000","last_became_peered":"2023-10-16T14:10:40.985218+0000","last_unstale":"2023-10-16T14:10:41.998098+0000","last_undegraded":"2023-10-16T14:10:41.998098+0000","last_fullsized":"2023-10-16T14:10:41.998098+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:22.942748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":15,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179906+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:29.179906+0000","last_peered":"2023-10-16T14:10:29.179906+0000","last_clean":"2023-10-16T14:10:29.179906+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:29.179906+0000","last_undegraded":"2023-10-16T14:10:29.179906+0000","last_fullsized":"2023-10-16T14:10:29.179906+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974315+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:40.974315+0000","last_peered":"2023-10-16T14:10:40.974315+0000","last_clean":"2023-10-16T14:10:40.974315+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:40.974315+0000","last_undegraded":"2023-10-16T14:10:40.974315+0000","last_fullsized":"2023-10-16T14:10:40.974315+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"234'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008061+0000","last_change":"2023-10-16T14:10:39.973954+0000","last_active":"2023-10-16T14:10:41.008061+0000","last_peered":"2023-10-16T14:10:41.008061+0000","last_clean":"2023-10-16T14:10:41.008061+0000","last_became_active":"2023-10-16T14:10:39.973602+0000","last_became_peered":"2023-10-16T14:10:39.973602+0000","last_unstale":"2023-10-16T14:10:41.008061+0000","last_undegraded":"2023-10-16T14:10:41.008061+0000","last_fullsized":"2023-10-16T14:10:41.008061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:18.496303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.14","version":"231'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.983108+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:40.983108+0000","last_peered":"2023-10-16T14:10:40.983108+0000","last_clean":"2023-10-16T14:10:40.983108+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:40.983108+0000","last_undegraded":"2023-10-16T14:10:40.983108+0000","last_fullsized":"2023-10-16T14:10:40.983108+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977730+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:40.977730+0000","last_peered":"2023-10-16T14:10:40.977730+0000","last_clean":"2023-10-16T14:10:40.977730+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:40.977730+0000","last_undegraded":"2023-10-16T14:10:40.977730+0000","last_fullsized":"2023-10-16T14:10:40.977730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980420+0000","last_change":"2023-10-16T14:10:39.976858+0000","last_active":"2023-10-16T14:10:40.980420+0000","last_peered":"2023-10-16T14:10:40.980420+0000","last_clean":"2023-10-16T14:10:40.980420+0000","last_became_active":"2023-10-16T14:10:39.976566+0000","last_became_peered":"2023-10-16T14:10:39.976566+0000","last_unstale":"2023-10-16T14:10:40.980420+0000","last_undegraded":"2023-10-16T14:10:40.980420+0000","last_fullsized":"2023-10-16T14:10:40.980420+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:00.617867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.990318+0000","last_change":"2023-10-16T14:10:40.985099+0000","last_active":"2023-10-16T14:10:41.990318+0000","last_peered":"2023-10-16T14:10:41.990318+0000","last_clean":"2023-10-16T14:10:41.990318+0000","last_became_active":"2023-10-16T14:10:40.984799+0000","last_became_peered":"2023-10-16T14:10:40.984799+0000","last_unstale":"2023-10-16T14:10:41.990318+0000","last_undegraded":"2023-10-16T14:10:41.990318+0000","last_fullsized":"2023-10-16T14:10:41.990318+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:02.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":19,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961436+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:40.961436+0000","last_peered":"2023-10-16T14:10:40.961436+0000","last_clean":"2023-10-16T14:10:40.961436+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:40.961436+0000","last_undegraded":"2023-10-16T14:10:40.961436+0000","last_fullsized":"2023-10-16T14:10:40.961436+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982473+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:40.982473+0000","last_peered":"2023-10-16T14:10:40.982473+0000","last_clean":"2023-10-16T14:10:40.982473+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:40.982473+0000","last_undegraded":"2023-10-16T14:10:40.982473+0000","last_fullsized":"2023-10-16T14:10:40.982473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011710+0000","last_change":"2023-10-16T14:10:39.986334+0000","last_active":"2023-10-16T14:10:41.011710+0000","last_peered":"2023-10-16T14:10:41.011710+0000","last_clean":"2023-10-16T14:10:41.011710+0000","last_became_active":"2023-10-16T14:10:39.986002+0000","last_became_peered":"2023-10-16T14:10:39.986002+0000","last_unstale":"2023-10-16T14:10:41.011710+0000","last_undegraded":"2023-10-16T14:10:41.011710+0000","last_fullsized":"2023-10-16T14:10:41.011710+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994013+0000","last_change":"2023-10-16T14:10:40.976822+0000","last_active":"2023-10-16T14:10:41.994013+0000","last_peered":"2023-10-16T14:10:41.994013+0000","last_clean":"2023-10-16T14:10:41.994013+0000","last_became_active":"2023-10-16T14:10:40.976523+0000","last_became_peered":"2023-10-16T14:10:40.976523+0000","last_unstale":"2023-10-16T14:10:41.994013+0000","last_undegraded":"2023-10-16T14:10:41.994013+0000","last_fullsized":"2023-10-16T14:10:41.994013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179997+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:29.179997+0000","last_peered":"2023-10-16T14:10:29.179997+0000","last_clean":"2023-10-16T14:10:29.179997+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:29.179997+0000","last_undegraded":"2023-10-16T14:10:29.179997+0000","last_fullsized":"2023-10-16T14:10:29.179997+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980779+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:40.980779+0000","last_peered":"2023-10-16T14:10:40.980779+0000","last_clean":"2023-10-16T14:10:40.980779+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:40.980779+0000","last_undegraded":"2023-10-16T14:10:40.980779+0000","last_fullsized":"2023-10-16T14:10:40.980779+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.937645+0000","last_change":"2023-10-16T14:10:38.937645+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.937645+0000","last_undegraded":"2023-10-16T14:10:38.937645+0000","last_fullsized":"2023-10-16T14:10:38.937645+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976181+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:40.976181+0000","last_peered":"2023-10-16T14:10:40.976181+0000","last_clean":"2023-10-16T14:10:40.976181+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:40.976181+0000","last_undegraded":"2023-10-16T14:10:40.976181+0000","last_fullsized":"2023-10-16T14:10:40.976181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979705+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:40.979705+0000","last_peered":"2023-10-16T14:10:40.979705+0000","last_clean":"2023-10-16T14:10:40.979705+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:40.979705+0000","last_undegraded":"2023-10-16T14:10:40.979705+0000","last_fullsized":"2023-10-16T14:10:40.979705+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975789+0000","last_change":"2023-10-16T14:10:39.968896+0000","last_active":"2023-10-16T14:10:40.975789+0000","last_peered":"2023-10-16T14:10:40.975789+0000","last_clean":"2023-10-16T14:10:40.975789+0000","last_became_active":"2023-10-16T14:10:39.968633+0000","last_became_peered":"2023-10-16T14:10:39.968633+0000","last_unstale":"2023-10-16T14:10:40.975789+0000","last_undegraded":"2023-10-16T14:10:40.975789+0000","last_fullsized":"2023-10-16T14:10:40.975789+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:26.024819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186867+0000","last_change":"2023-10-16T14:10:40.999802+0000","last_active":"2023-10-16T14:10:43.186867+0000","last_peered":"2023-10-16T14:10:43.186867+0000","last_clean":"2023-10-16T14:10:43.186867+0000","last_became_active":"2023-10-16T14:10:40.999483+0000","last_became_peered":"2023-10-16T14:10:40.999483+0000","last_unstale":"2023-10-16T14:10:43.186867+0000","last_undegraded":"2023-10-16T14:10:43.186867+0000","last_fullsized":"2023-10-16T14:10:43.186867+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:19.512934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.940148+0000","last_change":"2023-10-16T14:10:38.940148+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.940148+0000","last_undegraded":"2023-10-16T14:10:38.940148+0000","last_fullsized":"2023-10-16T14:10:38.940148+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":13,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179626+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:29.179626+0000","last_peered":"2023-10-16T14:10:29.179626+0000","last_clean":"2023-10-16T14:10:29.179626+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:29.179626+0000","last_undegraded":"2023-10-16T14:10:29.179626+0000","last_fullsized":"2023-10-16T14:10:29.179626+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976062+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:40.976062+0000","last_peered":"2023-10-16T14:10:40.976062+0000","last_clean":"2023-10-16T14:10:40.976062+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:40.976062+0000","last_undegraded":"2023-10-16T14:10:40.976062+0000","last_fullsized":"2023-10-16T14:10:40.976062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1f","version":"234'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009270+0000","last_change":"2023-10-16T14:10:39.973971+0000","last_active":"2023-10-16T14:10:41.009270+0000","last_peered":"2023-10-16T14:10:41.009270+0000","last_clean":"2023-10-16T14:10:41.009270+0000","last_became_active":"2023-10-16T14:10:39.973675+0000","last_became_peered":"2023-10-16T14:10:39.973675+0000","last_unstale":"2023-10-16T14:10:41.009270+0000","last_undegraded":"2023-10-16T14:10:41.009270+0000","last_fullsized":"2023-10-16T14:10:41.009270+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:01.458774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976322+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:40.976322+0000","last_peered":"2023-10-16T14:10:40.976322+0000","last_clean":"2023-10-16T14:10:40.976322+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:40.976322+0000","last_undegraded":"2023-10-16T14:10:40.976322+0000","last_fullsized":"2023-10-16T14:10:40.976322+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968823+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:40.968823+0000","last_peered":"2023-10-16T14:10:40.968823+0000","last_clean":"2023-10-16T14:10:40.968823+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:40.968823+0000","last_undegraded":"2023-10-16T14:10:40.968823+0000","last_fullsized":"2023-10-16T14:10:40.968823+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995327+0000","last_change":"2023-10-16T14:10:40.984926+0000","last_active":"2023-10-16T14:10:41.995327+0000","last_peered":"2023-10-16T14:10:41.995327+0000","last_clean":"2023-10-16T14:10:41.995327+0000","last_became_active":"2023-10-16T14:10:40.984364+0000","last_became_peered":"2023-10-16T14:10:40.984364+0000","last_unstale":"2023-10-16T14:10:41.995327+0000","last_undegraded":"2023-10-16T14:10:41.995327+0000","last_fullsized":"2023-10-16T14:10:41.995327+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974558+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:40.974558+0000","last_peered":"2023-10-16T14:10:40.974558+0000","last_clean":"2023-10-16T14:10:40.974558+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:40.974558+0000","last_undegraded":"2023-10-16T14:10:40.974558+0000","last_fullsized":"2023-10-16T14:10:40.974558+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977408+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:40.977408+0000","last_peered":"2023-10-16T14:10:40.977408+0000","last_clean":"2023-10-16T14:10:40.977408+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:40.977408+0000","last_undegraded":"2023-10-16T14:10:40.977408+0000","last_fullsized":"2023-10-16T14:10:40.977408+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.939295+0000","last_change":"2023-10-16T14:10:38.939295+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.939295+0000","last_undegraded":"2023-10-16T14:10:38.939295+0000","last_fullsized":"2023-10-16T14:10:38.939295+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125751+0000","last_change":"2023-10-16T14:10:40.997566+0000","last_active":"2023-10-16T14:10:42.125751+0000","last_peered":"2023-10-16T14:10:42.125751+0000","last_clean":"2023-10-16T14:10:42.125751+0000","last_became_active":"2023-10-16T14:10:40.997273+0000","last_became_peered":"2023-10-16T14:10:40.997273+0000","last_unstale":"2023-10-16T14:10:42.125751+0000","last_undegraded":"2023-10-16T14:10:42.125751+0000","last_fullsized":"2023-10-16T14:10:42.125751+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:38.615513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982594+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:40.982594+0000","last_peered":"2023-10-16T14:10:40.982594+0000","last_clean":"2023-10-16T14:10:40.982594+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:40.982594+0000","last_undegraded":"2023-10-16T14:10:40.982594+0000","last_fullsized":"2023-10-16T14:10:40.982594+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960137+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:40.960137+0000","last_peered":"2023-10-16T14:10:40.960137+0000","last_clean":"2023-10-16T14:10:40.960137+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:40.960137+0000","last_undegraded":"2023-10-16T14:10:40.960137+0000","last_fullsized":"2023-10-16T14:10:40.960137+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":1,"reported_epoch":232,"state":"creating+peering","last_fresh":"2023-10-16T14:10:38.952123+0000","last_change":"2023-10-16T14:10:38.952123+0000","last_active":"2023-10-16T14:10:38.914798+0000","last_peered":"2023-10-16T14:10:38.914798+0000","last_clean":"2023-10-16T14:10:38.914798+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:38.952123+0000","last_undegraded":"2023-10-16T14:10:38.952123+0000","last_fullsized":"2023-10-16T14:10:38.952123+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243037+0000","last_change":"2023-10-16T14:10:40.993412+0000","last_active":"2023-10-16T14:10:43.243037+0000","last_peered":"2023-10-16T14:10:43.243037+0000","last_clean":"2023-10-16T14:10:43.243037+0000","last_became_active":"2023-10-16T14:10:40.993119+0000","last_became_peered":"2023-10-16T14:10:40.993119+0000","last_unstale":"2023-10-16T14:10:43.243037+0000","last_undegraded":"2023-10-16T14:10:43.243037+0000","last_fullsized":"2023-10-16T14:10:43.243037+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:58.022778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981800+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:40.981800+0000","last_peered":"2023-10-16T14:10:40.981800+0000","last_clean":"2023-10-16T14:10:40.981800+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:40.981800+0000","last_undegraded":"2023-10-16T14:10:40.981800+0000","last_fullsized":"2023-10-16T14:10:40.981800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980136+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:40.980136+0000","last_peered":"2023-10-16T14:10:40.980136+0000","last_clean":"2023-10-16T14:10:40.980136+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:40.980136+0000","last_undegraded":"2023-10-16T14:10:40.980136+0000","last_fullsized":"2023-10-16T14:10:40.980136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011809+0000","last_change":"2023-10-16T14:10:39.969078+0000","last_active":"2023-10-16T14:10:41.011809+0000","last_peered":"2023-10-16T14:10:41.011809+0000","last_clean":"2023-10-16T14:10:41.011809+0000","last_became_active":"2023-10-16T14:10:39.968736+0000","last_became_peered":"2023-10-16T14:10:39.968736+0000","last_unstale":"2023-10-16T14:10:41.011809+0000","last_undegraded":"2023-10-16T14:10:41.011809+0000","last_fullsized":"2023-10-16T14:10:41.011809+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:59.354986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972873+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:40.972873+0000","last_peered":"2023-10-16T14:10:40.972873+0000","last_clean":"2023-10-16T14:10:40.972873+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:40.972873+0000","last_undegraded":"2023-10-16T14:10:40.972873+0000","last_fullsized":"2023-10-16T14:10:40.972873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.180040+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:29.180040+0000","last_peered":"2023-10-16T14:10:29.180040+0000","last_clean":"2023-10-16T14:10:29.180040+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:29.180040+0000","last_undegraded":"2023-10-16T14:10:29.180040+0000","last_fullsized":"2023-10-16T14:10:29.180040+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959567+0000","last_change":"2023-10-16T14:10:39.973296+0000","last_active":"2023-10-16T14:10:40.959567+0000","last_peered":"2023-10-16T14:10:40.959567+0000","last_clean":"2023-10-16T14:10:40.959567+0000","last_became_active":"2023-10-16T14:10:39.972592+0000","last_became_peered":"2023-10-16T14:10:39.972592+0000","last_unstale":"2023-10-16T14:10:40.959567+0000","last_undegraded":"2023-10-16T14:10:40.959567+0000","last_fullsized":"2023-10-16T14:10:40.959567+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:04.406174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242344+0000","last_change":"2023-10-16T14:10:40.987363+0000","last_active":"2023-10-16T14:10:43.242344+0000","last_peered":"2023-10-16T14:10:43.242344+0000","last_clean":"2023-10-16T14:10:43.242344+0000","last_became_active":"2023-10-16T14:10:40.985914+0000","last_became_peered":"2023-10-16T14:10:40.985914+0000","last_unstale":"2023-10-16T14:10:43.242344+0000","last_undegraded":"2023-10-16T14:10:43.242344+0000","last_fullsized":"2023-10-16T14:10:43.242344+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:10.423467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":17,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179737+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:29.179737+0000","last_peered":"2023-10-16T14:10:29.179737+0000","last_clean":"2023-10-16T14:10:29.179737+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:29.179737+0000","last_undegraded":"2023-10-16T14:10:29.179737+0000","last_fullsized":"2023-10-16T14:10:29.179737+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972050+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:40.972050+0000","last_peered":"2023-10-16T14:10:40.972050+0000","last_clean":"2023-10-16T14:10:40.972050+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:40.972050+0000","last_undegraded":"2023-10-16T14:10:40.972050+0000","last_fullsized":"2023-10-16T14:10:40.972050+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970323+0000","last_change":"2023-10-16T14:10:39.972648+0000","last_active":"2023-10-16T14:10:40.970323+0000","last_peered":"2023-10-16T14:10:40.970323+0000","last_clean":"2023-10-16T14:10:40.970323+0000","last_became_active":"2023-10-16T14:10:39.972328+0000","last_became_peered":"2023-10-16T14:10:39.972328+0000","last_unstale":"2023-10-16T14:10:40.970323+0000","last_undegraded":"2023-10-16T14:10:40.970323+0000","last_fullsized":"2023-10-16T14:10:40.970323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186986+0000","last_change":"2023-10-16T14:10:40.995949+0000","last_active":"2023-10-16T14:10:43.186986+0000","last_peered":"2023-10-16T14:10:43.186986+0000","last_clean":"2023-10-16T14:10:43.186986+0000","last_became_active":"2023-10-16T14:10:40.995689+0000","last_became_peered":"2023-10-16T14:10:40.995689+0000","last_unstale":"2023-10-16T14:10:43.186986+0000","last_undegraded":"2023-10-16T14:10:43.186986+0000","last_fullsized":"2023-10-16T14:10:43.186986+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:12.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977098+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:40.977098+0000","last_peered":"2023-10-16T14:10:40.977098+0000","last_clean":"2023-10-16T14:10:40.977098+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:40.977098+0000","last_undegraded":"2023-10-16T14:10:40.977098+0000","last_fullsized":"2023-10-16T14:10:40.977098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244569+0000","last_change":"2023-10-16T14:10:41.001647+0000","last_active":"2023-10-16T14:10:43.244569+0000","last_peered":"2023-10-16T14:10:43.244569+0000","last_clean":"2023-10-16T14:10:43.244569+0000","last_became_active":"2023-10-16T14:10:41.000840+0000","last_became_peered":"2023-10-16T14:10:41.000840+0000","last_unstale":"2023-10-16T14:10:43.244569+0000","last_undegraded":"2023-10-16T14:10:43.244569+0000","last_fullsized":"2023-10-16T14:10:43.244569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:55.904685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959155+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:40.959155+0000","last_peered":"2023-10-16T14:10:40.959155+0000","last_clean":"2023-10-16T14:10:40.959155+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:40.959155+0000","last_undegraded":"2023-10-16T14:10:40.959155+0000","last_fullsized":"2023-10-16T14:10:40.959155+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243482+0000","last_change":"2023-10-16T14:10:40.991703+0000","last_active":"2023-10-16T14:10:43.243482+0000","last_peered":"2023-10-16T14:10:43.243482+0000","last_clean":"2023-10-16T14:10:43.243482+0000","last_became_active":"2023-10-16T14:10:40.991364+0000","last_became_peered":"2023-10-16T14:10:40.991364+0000","last_unstale":"2023-10-16T14:10:43.243482+0000","last_undegraded":"2023-10-16T14:10:43.243482+0000","last_fullsized":"2023-10-16T14:10:43.243482+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:30.238400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971730+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:40.971730+0000","last_peered":"2023-10-16T14:10:40.971730+0000","last_clean":"2023-10-16T14:10:40.971730+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:40.971730+0000","last_undegraded":"2023-10-16T14:10:40.971730+0000","last_fullsized":"2023-10-16T14:10:40.971730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244638+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:43.244638+0000","last_peered":"2023-10-16T14:10:43.244638+0000","last_clean":"2023-10-16T14:10:43.244638+0000","last_became_active":"2023-10-16T14:10:40.992264+0000","last_became_peered":"2023-10-16T14:10:40.992264+0000","last_unstale":"2023-10-16T14:10:43.244638+0000","last_undegraded":"2023-10-16T14:10:43.244638+0000","last_fullsized":"2023-10-16T14:10:43.244638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:32.614992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179374+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:29.179374+0000","last_peered":"2023-10-16T14:10:29.179374+0000","last_clean":"2023-10-16T14:10:29.179374+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:29.179374+0000","last_undegraded":"2023-10-16T14:10:29.179374+0000","last_fullsized":"2023-10-16T14:10:29.179374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242825+0000","last_change":"2023-10-16T14:10:40.989459+0000","last_active":"2023-10-16T14:10:43.242825+0000","last_peered":"2023-10-16T14:10:43.242825+0000","last_clean":"2023-10-16T14:10:43.242825+0000","last_became_active":"2023-10-16T14:10:40.989197+0000","last_became_peered":"2023-10-16T14:10:40.989197+0000","last_unstale":"2023-10-16T14:10:43.242825+0000","last_undegraded":"2023-10-16T14:10:43.242825+0000","last_fullsized":"2023-10-16T14:10:43.242825+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979422+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:40.979422+0000","last_peered":"2023-10-16T14:10:40.979422+0000","last_clean":"2023-10-16T14:10:40.979422+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:40.979422+0000","last_undegraded":"2023-10-16T14:10:40.979422+0000","last_fullsized":"2023-10-16T14:10:40.979422+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994059+0000","last_change":"2023-10-16T14:10:40.986689+0000","last_active":"2023-10-16T14:10:41.994059+0000","last_peered":"2023-10-16T14:10:41.994059+0000","last_clean":"2023-10-16T14:10:41.994059+0000","last_became_active":"2023-10-16T14:10:40.986408+0000","last_became_peered":"2023-10-16T14:10:40.986408+0000","last_unstale":"2023-10-16T14:10:41.994059+0000","last_undegraded":"2023-10-16T14:10:41.994059+0000","last_fullsized":"2023-10-16T14:10:41.994059+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179653+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:29.179653+0000","last_peered":"2023-10-16T14:10:29.179653+0000","last_clean":"2023-10-16T14:10:29.179653+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:29.179653+0000","last_undegraded":"2023-10-16T14:10:29.179653+0000","last_fullsized":"2023-10-16T14:10:29.179653+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989882+0000","last_change":"2023-10-16T14:10:40.987586+0000","last_active":"2023-10-16T14:10:41.989882+0000","last_peered":"2023-10-16T14:10:41.989882+0000","last_clean":"2023-10-16T14:10:41.989882+0000","last_became_active":"2023-10-16T14:10:40.987254+0000","last_became_peered":"2023-10-16T14:10:40.987254+0000","last_unstale":"2023-10-16T14:10:41.989882+0000","last_undegraded":"2023-10-16T14:10:41.989882+0000","last_fullsized":"2023-10-16T14:10:41.989882+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.463597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980939+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:40.980939+0000","last_peered":"2023-10-16T14:10:40.980939+0000","last_clean":"2023-10-16T14:10:40.980939+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:40.980939+0000","last_undegraded":"2023-10-16T14:10:40.980939+0000","last_fullsized":"2023-10-16T14:10:40.980939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124331+0000","last_change":"2023-10-16T14:10:40.990967+0000","last_active":"2023-10-16T14:10:42.124331+0000","last_peered":"2023-10-16T14:10:42.124331+0000","last_clean":"2023-10-16T14:10:42.124331+0000","last_became_active":"2023-10-16T14:10:40.989709+0000","last_became_peered":"2023-10-16T14:10:40.989709+0000","last_unstale":"2023-10-16T14:10:42.124331+0000","last_undegraded":"2023-10-16T14:10:42.124331+0000","last_fullsized":"2023-10-16T14:10:42.124331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.888217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974374+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:40.974374+0000","last_peered":"2023-10-16T14:10:40.974374+0000","last_clean":"2023-10-16T14:10:40.974374+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:40.974374+0000","last_undegraded":"2023-10-16T14:10:40.974374+0000","last_fullsized":"2023-10-16T14:10:40.974374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244140+0000","last_change":"2023-10-16T14:10:40.985097+0000","last_active":"2023-10-16T14:10:43.244140+0000","last_peered":"2023-10-16T14:10:43.244140+0000","last_clean":"2023-10-16T14:10:43.244140+0000","last_became_active":"2023-10-16T14:10:40.984409+0000","last_became_peered":"2023-10-16T14:10:40.984409+0000","last_unstale":"2023-10-16T14:10:43.244140+0000","last_undegraded":"2023-10-16T14:10:43.244140+0000","last_fullsized":"2023-10-16T14:10:43.244140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972580+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:40.972580+0000","last_peered":"2023-10-16T14:10:40.972580+0000","last_clean":"2023-10-16T14:10:40.972580+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:40.972580+0000","last_undegraded":"2023-10-16T14:10:40.972580+0000","last_fullsized":"2023-10-16T14:10:40.972580+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996436+0000","last_change":"2023-10-16T14:10:40.988348+0000","last_active":"2023-10-16T14:10:41.996436+0000","last_peered":"2023-10-16T14:10:41.996436+0000","last_clean":"2023-10-16T14:10:41.996436+0000","last_became_active":"2023-10-16T14:10:40.988060+0000","last_became_peered":"2023-10-16T14:10:40.988060+0000","last_unstale":"2023-10-16T14:10:41.996436+0000","last_undegraded":"2023-10-16T14:10:41.996436+0000","last_fullsized":"2023-10-16T14:10:41.996436+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179686+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:29.179686+0000","last_peered":"2023-10-16T14:10:29.179686+0000","last_clean":"2023-10-16T14:10:29.179686+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:29.179686+0000","last_undegraded":"2023-10-16T14:10:29.179686+0000","last_fullsized":"2023-10-16T14:10:29.179686+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243319+0000","last_change":"2023-10-16T14:10:40.987085+0000","last_active":"2023-10-16T14:10:43.243319+0000","last_peered":"2023-10-16T14:10:43.243319+0000","last_clean":"2023-10-16T14:10:43.243319+0000","last_became_active":"2023-10-16T14:10:40.986831+0000","last_became_peered":"2023-10-16T14:10:40.986831+0000","last_unstale":"2023-10-16T14:10:43.243319+0000","last_undegraded":"2023-10-16T14:10:43.243319+0000","last_fullsized":"2023-10-16T14:10:43.243319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:58.536370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179817+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:29.179817+0000","last_peered":"2023-10-16T14:10:29.179817+0000","last_clean":"2023-10-16T14:10:29.179817+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:29.179817+0000","last_undegraded":"2023-10-16T14:10:29.179817+0000","last_fullsized":"2023-10-16T14:10:29.179817+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125634+0000","last_change":"2023-10-16T14:10:40.998279+0000","last_active":"2023-10-16T14:10:42.125634+0000","last_peered":"2023-10-16T14:10:42.125634+0000","last_clean":"2023-10-16T14:10:42.125634+0000","last_became_active":"2023-10-16T14:10:40.997899+0000","last_became_peered":"2023-10-16T14:10:40.997899+0000","last_unstale":"2023-10-16T14:10:42.125634+0000","last_undegraded":"2023-10-16T14:10:42.125634+0000","last_fullsized":"2023-10-16T14:10:42.125634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:02.766931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977306+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:40.977306+0000","last_peered":"2023-10-16T14:10:40.977306+0000","last_clean":"2023-10-16T14:10:40.977306+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:40.977306+0000","last_undegraded":"2023-10-16T14:10:40.977306+0000","last_fullsized":"2023-10-16T14:10:40.977306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.991518+0000","last_change":"2023-10-16T14:10:40.992264+0000","last_active":"2023-10-16T14:10:41.991518+0000","last_peered":"2023-10-16T14:10:41.991518+0000","last_clean":"2023-10-16T14:10:41.991518+0000","last_became_active":"2023-10-16T14:10:40.991698+0000","last_became_peered":"2023-10-16T14:10:40.991698+0000","last_unstale":"2023-10-16T14:10:41.991518+0000","last_undegraded":"2023-10-16T14:10:41.991518+0000","last_fullsized":"2023-10-16T14:10:41.991518+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:16.407382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978952+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:40.978952+0000","last_peered":"2023-10-16T14:10:40.978952+0000","last_clean":"2023-10-16T14:10:40.978952+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:40.978952+0000","last_undegraded":"2023-10-16T14:10:40.978952+0000","last_fullsized":"2023-10-16T14:10:40.978952+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125182+0000","last_change":"2023-10-16T14:10:40.991982+0000","last_active":"2023-10-16T14:10:42.125182+0000","last_peered":"2023-10-16T14:10:42.125182+0000","last_clean":"2023-10-16T14:10:42.125182+0000","last_became_active":"2023-10-16T14:10:40.991521+0000","last_became_peered":"2023-10-16T14:10:40.991521+0000","last_unstale":"2023-10-16T14:10:42.125182+0000","last_undegraded":"2023-10-16T14:10:42.125182+0000","last_fullsized":"2023-10-16T14:10:42.125182+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:01.654954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961133+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:40.961133+0000","last_peered":"2023-10-16T14:10:40.961133+0000","last_clean":"2023-10-16T14:10:40.961133+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:40.961133+0000","last_undegraded":"2023-10-16T14:10:40.961133+0000","last_fullsized":"2023-10-16T14:10:40.961133+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124085+0000","last_change":"2023-10-16T14:10:40.992568+0000","last_active":"2023-10-16T14:10:42.124085+0000","last_peered":"2023-10-16T14:10:42.124085+0000","last_clean":"2023-10-16T14:10:42.124085+0000","last_became_active":"2023-10-16T14:10:40.992252+0000","last_became_peered":"2023-10-16T14:10:40.992252+0000","last_unstale":"2023-10-16T14:10:42.124085+0000","last_undegraded":"2023-10-16T14:10:42.124085+0000","last_fullsized":"2023-10-16T14:10:42.124085+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:25.262205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965644+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:40.965644+0000","last_peered":"2023-10-16T14:10:40.965644+0000","last_clean":"2023-10-16T14:10:40.965644+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:40.965644+0000","last_undegraded":"2023-10-16T14:10:40.965644+0000","last_fullsized":"2023-10-16T14:10:40.965644+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242786+0000","last_change":"2023-10-16T14:10:40.991785+0000","last_active":"2023-10-16T14:10:43.242786+0000","last_peered":"2023-10-16T14:10:43.242786+0000","last_clean":"2023-10-16T14:10:43.242786+0000","last_became_active":"2023-10-16T14:10:40.991292+0000","last_became_peered":"2023-10-16T14:10:40.991292+0000","last_unstale":"2023-10-16T14:10:43.242786+0000","last_undegraded":"2023-10-16T14:10:43.242786+0000","last_fullsized":"2023-10-16T14:10:43.242786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978297+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:40.978297+0000","last_peered":"2023-10-16T14:10:40.978297+0000","last_clean":"2023-10-16T14:10:40.978297+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:40.978297+0000","last_undegraded":"2023-10-16T14:10:40.978297+0000","last_fullsized":"2023-10-16T14:10:40.978297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243833+0000","last_change":"2023-10-16T14:10:40.985314+0000","last_active":"2023-10-16T14:10:43.243833+0000","last_peered":"2023-10-16T14:10:43.243833+0000","last_clean":"2023-10-16T14:10:43.243833+0000","last_became_active":"2023-10-16T14:10:40.984808+0000","last_became_peered":"2023-10-16T14:10:40.984808+0000","last_unstale":"2023-10-16T14:10:43.243833+0000","last_undegraded":"2023-10-16T14:10:43.243833+0000","last_fullsized":"2023-10-16T14:10:43.243833+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959800+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:40.959800+0000","last_peered":"2023-10-16T14:10:40.959800+0000","last_clean":"2023-10-16T14:10:40.959800+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:40.959800+0000","last_undegraded":"2023-10-16T14:10:40.959800+0000","last_fullsized":"2023-10-16T14:10:40.959800+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978097+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:40.978097+0000","last_peered":"2023-10-16T14:10:40.978097+0000","last_clean":"2023-10-16T14:10:40.978097+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:40.978097+0000","last_undegraded":"2023-10-16T14:10:40.978097+0000","last_fullsized":"2023-10-16T14:10:40.978097+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980317+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:40.980317+0000","last_peered":"2023-10-16T14:10:40.980317+0000","last_clean":"2023-10-16T14:10:40.980317+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:40.980317+0000","last_undegraded":"2023-10-16T14:10:40.980317+0000","last_fullsized":"2023-10-16T14:10:40.980317+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243931+0000","last_change":"2023-10-16T14:10:40.994995+0000","last_active":"2023-10-16T14:10:43.243931+0000","last_peered":"2023-10-16T14:10:43.243931+0000","last_clean":"2023-10-16T14:10:43.243931+0000","last_became_active":"2023-10-16T14:10:40.994421+0000","last_became_peered":"2023-10-16T14:10:40.994421+0000","last_unstale":"2023-10-16T14:10:43.243931+0000","last_undegraded":"2023-10-16T14:10:43.243931+0000","last_fullsized":"2023-10-16T14:10:43.243931+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:39.108794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179499+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:29.179499+0000","last_peered":"2023-10-16T14:10:29.179499+0000","last_clean":"2023-10-16T14:10:29.179499+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:29.179499+0000","last_undegraded":"2023-10-16T14:10:29.179499+0000","last_fullsized":"2023-10-16T14:10:29.179499+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995533+0000","last_change":"2023-10-16T14:10:40.988474+0000","last_active":"2023-10-16T14:10:41.995533+0000","last_peered":"2023-10-16T14:10:41.995533+0000","last_clean":"2023-10-16T14:10:41.995533+0000","last_became_active":"2023-10-16T14:10:40.988193+0000","last_became_peered":"2023-10-16T14:10:40.988193+0000","last_unstale":"2023-10-16T14:10:41.995533+0000","last_undegraded":"2023-10-16T14:10:41.995533+0000","last_fullsized":"2023-10-16T14:10:41.995533+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":12,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-16T14:10:29.179611+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:29.179611+0000","last_peered":"2023-10-16T14:10:29.179611+0000","last_clean":"2023-10-16T14:10:29.179611+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:29.179611+0000","last_undegraded":"2023-10-16T14:10:29.179611+0000","last_fullsized":"2023-10-16T14:10:29.179611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981697+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:40.981697+0000","last_peered":"2023-10-16T14:10:40.981697+0000","last_clean":"2023-10-16T14:10:40.981697+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:40.981697+0000","last_undegraded":"2023-10-16T14:10:40.981697+0000","last_fullsized":"2023-10-16T14:10:40.981697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965582+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:40.965582+0000","last_peered":"2023-10-16T14:10:40.965582+0000","last_clean":"2023-10-16T14:10:40.965582+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:40.965582+0000","last_undegraded":"2023-10-16T14:10:40.965582+0000","last_fullsized":"2023-10-16T14:10:40.965582+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989881+0000","last_change":"2023-10-16T14:10:40.987206+0000","last_active":"2023-10-16T14:10:41.989881+0000","last_peered":"2023-10-16T14:10:41.989881+0000","last_clean":"2023-10-16T14:10:41.989881+0000","last_became_active":"2023-10-16T14:10:40.986211+0000","last_became_peered":"2023-10-16T14:10:40.986211+0000","last_unstale":"2023-10-16T14:10:41.989881+0000","last_undegraded":"2023-10-16T14:10:41.989881+0000","last_fullsized":"2023-10-16T14:10:41.989881+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974724+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:40.974724+0000","last_peered":"2023-10-16T14:10:40.974724+0000","last_clean":"2023-10-16T14:10:40.974724+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:40.974724+0000","last_undegraded":"2023-10-16T14:10:40.974724+0000","last_fullsized":"2023-10-16T14:10:40.974724+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968589+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:40.968589+0000","last_peered":"2023-10-16T14:10:40.968589+0000","last_clean":"2023-10-16T14:10:40.968589+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:40.968589+0000","last_undegraded":"2023-10-16T14:10:40.968589+0000","last_fullsized":"2023-10-16T14:10:40.968589+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244429+0000","last_change":"2023-10-16T14:10:41.001901+0000","last_active":"2023-10-16T14:10:43.244429+0000","last_peered":"2023-10-16T14:10:43.244429+0000","last_clean":"2023-10-16T14:10:43.244429+0000","last_became_active":"2023-10-16T14:10:41.001390+0000","last_became_peered":"2023-10-16T14:10:41.001390+0000","last_unstale":"2023-10-16T14:10:43.244429+0000","last_undegraded":"2023-10-16T14:10:43.244429+0000","last_fullsized":"2023-10-16T14:10:43.244429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:38.275860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977891+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:40.977891+0000","last_peered":"2023-10-16T14:10:40.977891+0000","last_clean":"2023-10-16T14:10:40.977891+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:40.977891+0000","last_undegraded":"2023-10-16T14:10:40.977891+0000","last_fullsized":"2023-10-16T14:10:40.977891+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124855+0000","last_change":"2023-10-16T14:10:40.990936+0000","last_active":"2023-10-16T14:10:42.124855+0000","last_peered":"2023-10-16T14:10:42.124855+0000","last_clean":"2023-10-16T14:10:42.124855+0000","last_became_active":"2023-10-16T14:10:40.990264+0000","last_became_peered":"2023-10-16T14:10:40.990264+0000","last_unstale":"2023-10-16T14:10:42.124855+0000","last_undegraded":"2023-10-16T14:10:42.124855+0000","last_fullsized":"2023-10-16T14:10:42.124855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:39.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973916+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:40.973916+0000","last_peered":"2023-10-16T14:10:40.973916+0000","last_clean":"2023-10-16T14:10:40.973916+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:40.973916+0000","last_undegraded":"2023-10-16T14:10:40.973916+0000","last_fullsized":"2023-10-16T14:10:40.973916+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973460+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:40.973460+0000","last_peered":"2023-10-16T14:10:40.973460+0000","last_clean":"2023-10-16T14:10:40.973460+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:40.973460+0000","last_undegraded":"2023-10-16T14:10:40.973460+0000","last_fullsized":"2023-10-16T14:10:40.973460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.241703+0000","last_change":"2023-10-16T14:10:40.982851+0000","last_active":"2023-10-16T14:10:43.241703+0000","last_peered":"2023-10-16T14:10:43.241703+0000","last_clean":"2023-10-16T14:10:43.241703+0000","last_became_active":"2023-10-16T14:10:40.981261+0000","last_became_peered":"2023-10-16T14:10:40.981261+0000","last_unstale":"2023-10-16T14:10:43.241703+0000","last_undegraded":"2023-10-16T14:10:43.241703+0000","last_fullsized":"2023-10-16T14:10:43.241703+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:03.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982369+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:40.982369+0000","last_peered":"2023-10-16T14:10:40.982369+0000","last_clean":"2023-10-16T14:10:40.982369+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:40.982369+0000","last_undegraded":"2023-10-16T14:10:40.982369+0000","last_fullsized":"2023-10-16T14:10:40.982369+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125338+0000","last_change":"2023-10-16T14:10:40.991807+0000","last_active":"2023-10-16T14:10:42.125338+0000","last_peered":"2023-10-16T14:10:42.125338+0000","last_clean":"2023-10-16T14:10:42.125338+0000","last_became_active":"2023-10-16T14:10:40.991143+0000","last_became_peered":"2023-10-16T14:10:40.991143+0000","last_unstale":"2023-10-16T14:10:42.125338+0000","last_undegraded":"2023-10-16T14:10:42.125338+0000","last_fullsized":"2023-10-16T14:10:42.125338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.405248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981437+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:40.981437+0000","last_peered":"2023-10-16T14:10:40.981437+0000","last_clean":"2023-10-16T14:10:40.981437+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:40.981437+0000","last_undegraded":"2023-10-16T14:10:40.981437+0000","last_fullsized":"2023-10-16T14:10:40.981437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185397+0000","last_change":"2023-10-16T14:10:40.997929+0000","last_active":"2023-10-16T14:10:43.185397+0000","last_peered":"2023-10-16T14:10:43.185397+0000","last_clean":"2023-10-16T14:10:43.185397+0000","last_became_active":"2023-10-16T14:10:40.997578+0000","last_became_peered":"2023-10-16T14:10:40.997578+0000","last_unstale":"2023-10-16T14:10:43.185397+0000","last_undegraded":"2023-10-16T14:10:43.185397+0000","last_fullsized":"2023-10-16T14:10:43.185397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:31.595742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968740+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:40.968740+0000","last_peered":"2023-10-16T14:10:40.968740+0000","last_clean":"2023-10-16T14:10:40.968740+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:40.968740+0000","last_undegraded":"2023-10-16T14:10:40.968740+0000","last_fullsized":"2023-10-16T14:10:40.968740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T14:10:39.931429+0000","last_change":"2023-10-16T14:10:39.931429+0000","last_active":"2023-10-16T14:10:39.931429+0000","last_peered":"2023-10-16T14:10:39.931429+0000","last_clean":"2023-10-16T14:10:39.931429+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T14:10:39.931429+0000","last_undegraded":"2023-10-16T14:10:39.931429+0000","last_fullsized":"2023-10-16T14:10:39.931429+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978187+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:40.978187+0000","last_peered":"2023-10-16T14:10:40.978187+0000","last_clean":"2023-10-16T14:10:40.978187+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:40.978187+0000","last_undegraded":"2023-10-16T14:10:40.978187+0000","last_fullsized":"2023-10-16T14:10:40.978187+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":88,"acting":88,"num_store_stats":6},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2388,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543301,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3551232,"data_stored":2031342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":1.4139999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.589}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.4730000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.327}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.087},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.3779999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.228},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.5329999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543301,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42336,"kb_used_data":2828,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329504,"statfs":{"total":96636764160,"available":96593412096,"internally_reserved":0,"allocated":2895872,"data_stored":1410006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":1.8460000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.6279999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":1.405},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.5069999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.034},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.776}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.679}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":1.339},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.278}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.548},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.5920000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543303,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":2945024,"data_stored":1438629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.02}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.1799999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.254}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.286},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.8080000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.9590000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.4590000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543302,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47908,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323932,"statfs":{"total":96636764160,"available":96587706368,"internally_reserved":0,"allocated":2912256,"data_stored":1412385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.8420000000000001},"last":1.075},{"interface":"front","average":{"1min":1.121,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6699999999999999,"5min":1.8160000000000001,"15min":1.9019999999999999},"last":1.329}]},{"osd":1,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.78},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2549999999999999,"15min":1.294},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.754,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.7250000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.286,"15min":1.3169999999999999},"min":{"1min":0.80200000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.714,"5min":1.714,"15min":1.865},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.367,"15min":1.387},"min":{"1min":1.026,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.276}]},{"osd":3,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.296,"15min":1.3120000000000001},"min":{"1min":0.94599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.615,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.627},{"interface":"front","average":{"1min":1.607,"5min":1.4590000000000001,"15min":1.4299999999999999},"min":{"1min":0.96499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":5.2350000000000003,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.1930000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.329,"15min":1.3400000000000001},"min":{"1min":0.99199999999999999,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3580000000000001,"15min":1.3360000000000001},"min":{"1min":1.0389999999999999,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":1.921,"15min":1.921},"last":1.7629999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3919999999999999,"15min":1.3879999999999999},"min":{"1min":1.004,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.7689999999999999,"5min":1.9159999999999999,"15min":1.956},"last":1.581},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.502,"15min":1.5229999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.8069999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5349999999999999,"15min":1.5049999999999999},"min":{"1min":1.137,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.1949999999999998},"last":1.857},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6379999999999999,"15min":1.6319999999999999},"min":{"1min":1.391,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":1.889,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.6799999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543303,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52128,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319712,"statfs":{"total":96636764160,"available":96583385088,"internally_reserved":0,"allocated":2969600,"data_stored":1439911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":1.05}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":4.1299999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":3.8340000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":0.89300000000000002}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.01}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":2.2799999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":2.569}]}]},{"osd":2,"up_from":14,"seq":60129543301,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":2949120,"data_stored":1432548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":1.337},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.115},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":1.4159999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.268},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.0469999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.103},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":1.397}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.1899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543301,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2953216,"data_stored":1438193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.0740000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":0.752},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.1870000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":1.532}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":1.454},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":2.0979999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.867},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.7969999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543302,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":3547136,"data_stored":2029177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.93300000000000005,"15min":0.93999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.218,"5min":1.704,"15min":1.891},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.1579999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48699999999999999},"max":{"1min":1.627,"5min":1.659,"15min":1.9299999999999999},"last":0.90700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.1499999999999999,"15min":1.171},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5349999999999999,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.1899999999999999,"15min":1.226},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.6539999999999999,"5min":1.893,"15min":1.893},"last":1.0760000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1639999999999999,"15min":1.163},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7390000000000001,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3380000000000001,"15min":1.3759999999999999},"min":{"1min":0.64700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.679,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.2969999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3340000000000001,"15min":1.3799999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6140000000000001,"5min":1.819,"15min":2.048},"last":1.002},{"interface":"front","average":{"1min":1.379,"5min":1.518,"15min":1.524},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.0009999999999999,"5min":2.601,"15min":2.601},"last":1.4850000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.48,"15min":1.5069999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.821,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.452,"15min":1.492},"min":{"1min":0.98199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0169999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.593,"15min":1.621},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9490000000000001,"5min":2.669,"15min":2.669},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.55,"15min":1.5980000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2589999999999999,"5min":2.355,"15min":2.3769999999999998},"last":1.778}]},{"osd":7,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.655,"15min":1.712},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1549999999999998,"5min":2.484,"15min":2.5270000000000001},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.625,"15min":1.649},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.464},"last":1.056}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:46.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.348+0000 7f2ccda46640 1 Processor -- start 2023-10-16T14:10:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.348+0000 7f2ccda46640 1 -- start start 2023-10-16T14:10:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2ccda46640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2cc8159f00 0x7f2cc80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2ccda46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2cc815d880 0x7f2cc80b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2ccda46640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc806f7f0 con 0x7f2cc8159b20 2023-10-16T14:10:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2ccda46640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc806c5e0 con 0x7f2cc815d880 2023-10-16T14:10:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2ccda46640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc806c8c0 con 0x7f2cc8159f00 2023-10-16T14:10:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2cc77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2cc815d880 0x7f2cc80b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2cc77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2cc815d880 0x7f2cc80b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38094/0 (socket says 172.21.15.60:38094) 2023-10-16T14:10:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2cc77fe640 1 -- 172.21.15.60:0/771334322 learned_addr learned my addr 172.21.15.60:0/771334322 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.350+0000 7f2cbffff640 1 --2- 172.21.15.60:0/771334322 >> v2:172.21.15.144:3300/0 conn(0x7f2cc8159f00 0x7f2cc80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.351+0000 7f2cc77fe640 1 -- 172.21.15.60:0/771334322 >> v2:172.21.15.144:3300/0 conn(0x7f2cc8159f00 msgr2=0x7f2cc80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.351+0000 7f2cc77fe640 1 --2- 172.21.15.60:0/771334322 >> v2:172.21.15.144:3300/0 conn(0x7f2cc8159f00 0x7f2cc80b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.351+0000 7f2cc77fe640 1 -- 172.21.15.60:0/771334322 >> v1:172.21.15.144:6789/0 conn(0x7f2cc8159b20 legacy=0x7f2cc806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.351+0000 7f2cc77fe640 1 -- 172.21.15.60:0/771334322 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc806cba0 con 0x7f2cc815d880 2023-10-16T14:10:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.352+0000 7f2cc77fe640 1 --2- 172.21.15.60:0/771334322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80b0950 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f2cc815d770 tx=0x7f2cc002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfad136638fad3f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:46.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.352+0000 7f2cc67fc640 1 -- 172.21.15.60:0/771334322 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc002ca80 con 0x7f2cc815d880 2023-10-16T14:10:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.352+0000 7f2cc67fc640 1 -- 172.21.15.60:0/771334322 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc002cbe0 con 0x7f2cc815d880 2023-10-16T14:10:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.352+0000 7f2cc67fc640 1 -- 172.21.15.60:0/771334322 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc00343f0 con 0x7f2cc815d880 2023-10-16T14:10:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 -- 172.21.15.60:0/771334322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 msgr2=0x7f2cc80b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 --2- 172.21.15.60:0/771334322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80b0950 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f2cc815d770 tx=0x7f2cc002bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 -- 172.21.15.60:0/771334322 shutdown_connections 2023-10-16T14:10:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 --2- 172.21.15.60:0/771334322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80b0950 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 --2- 172.21.15.60:0/771334322 >> v2:172.21.15.144:3300/0 conn(0x7f2cc8159f00 0x7f2cc80b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 -- 172.21.15.60:0/771334322 >> 172.21.15.60:0/771334322 conn(0x7f2cc80cb410 msgr2=0x7f2cc80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 -- 172.21.15.60:0/771334322 shutdown_connections 2023-10-16T14:10:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.353+0000 7f2ccda46640 1 -- 172.21.15.60:0/771334322 wait complete. 2023-10-16T14:10:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.355+0000 7f2ccda46640 1 Processor -- start 2023-10-16T14:10:46.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.355+0000 7f2ccda46640 1 -- start start 2023-10-16T14:10:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2ccda46640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cc8159b20 0x7f2cc80c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2ccda46640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cc8159f00 0x7f2cc80cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2ccda46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2ccda46640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81507e0 con 0x7f2cc815d880 2023-10-16T14:10:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2ccda46640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc80c2d20 con 0x7f2cc8159f00 2023-10-16T14:10:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2ccda46640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc814d8a0 con 0x7f2cc8159b20 2023-10-16T14:10:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2cc77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2cc77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38104/0 (socket says 172.21.15.60:38104) 2023-10-16T14:10:46.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.356+0000 7f2cc77fe640 1 -- 172.21.15.60:0/2923295572 learned_addr learned my addr 172.21.15.60:0/2923295572 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cbffff640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cc8159f00 0x7f2cc80cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc77fe640 1 -- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cc8159b20 msgr2=0x7f2cc80c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc77fe640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cc8159b20 0x7f2cc80c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc77fe640 1 -- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cc8159f00 msgr2=0x7f2cc80cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc77fe640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cc8159f00 0x7f2cc80cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc77fe640 1 -- 172.21.15.60:0/2923295572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0009b30 con 0x7f2cc815d880 2023-10-16T14:10:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc77fe640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80c5a60 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0036040 tx=0x7f2cc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0009510 con 0x7f2cc815d880 2023-10-16T14:10:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc003e450 con 0x7f2cc815d880 2023-10-16T14:10:46.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.357+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc8150500 con 0x7f2cc815d880 2023-10-16T14:10:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.358+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc003e7e0 con 0x7f2cc815d880 2023-10-16T14:10:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.358+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc80c2710 con 0x7f2cc815d880 2023-10-16T14:10:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.359+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc80c24c0 con 0x7f2cc815d880 2023-10-16T14:10:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.360+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2cc002cd50 con 0x7f2cc815d880 2023-10-16T14:10:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.361+0000 7f2cc4ff9640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2ca806af90 0x7f2ca806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.361+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f2cc00ac6d0 con 0x7f2cc815d880 2023-10-16T14:10:46.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.362+0000 7f2cc6ffd640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2ca806af90 0x7f2ca806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.363+0000 7f2cc6ffd640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2ca806af90 0x7f2ca806d450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8006fd0 tx=0x7f2cb8006bc0 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:46.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.365+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2cc0077270 con 0x7f2cc815d880 2023-10-16T14:10:46.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.557+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cc806ce80 con 0x7f2ca806af90 2023-10-16T14:10:46.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.599+0000 7f2cc4ff9640 1 -- 172.21.15.60:0/2923295572 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526047 (secure 0 0 0) 0x7f2cb8013430 con 0x7f2ca806af90 2023-10-16T14:10:46.607 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:46.613 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:46.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2ca806af90 msgr2=0x7f2ca806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2ca806af90 0x7f2ca806d450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8006fd0 tx=0x7f2cb8006bc0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 msgr2=0x7f2cc80c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80c5a60 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0036040 tx=0x7f2cc0004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:46.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 shutdown_connections 2023-10-16T14:10:46.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f2ca806af90 0x7f2ca806d450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2cc815d880 0x7f2cc80c5a60 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cc8159f00 0x7f2cc80cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 --2- 172.21.15.60:0/2923295572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2cc8159b20 0x7f2cc80c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.610+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 >> 172.21.15.60:0/2923295572 conn(0x7f2cc80cb410 msgr2=0x7f2cc80ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:46.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.611+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 shutdown_connections 2023-10-16T14:10:46.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.611+0000 7f2ccda46640 1 -- 172.21.15.60:0/2923295572 wait complete. 2023-10-16T14:10:46.634 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-16T14:10:46.184131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":674,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":356,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377856,"kb_used_data":24252,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754596864,"statfs":{"total":773094113280,"available":772707188736,"internally_reserved":0,"allocated":24834048,"data_stored":12690075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.232069"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244206+0000","last_change":"2023-10-16T14:10:40.992171+0000","last_active":"2023-10-16T14:10:43.244206+0000","last_peered":"2023-10-16T14:10:43.244206+0000","last_clean":"2023-10-16T14:10:43.244206+0000","last_became_active":"2023-10-16T14:10:40.991865+0000","last_became_peered":"2023-10-16T14:10:40.991865+0000","last_unstale":"2023-10-16T14:10:43.244206+0000","last_undegraded":"2023-10-16T14:10:43.244206+0000","last_fullsized":"2023-10-16T14:10:43.244206+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:36.935225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1226,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974179+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:40.974179+0000","last_peered":"2023-10-16T14:10:40.974179+0000","last_clean":"2023-10-16T14:10:40.974179+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:40.974179+0000","last_undegraded":"2023-10-16T14:10:40.974179+0000","last_fullsized":"2023-10-16T14:10:40.974179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968913+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:40.968913+0000","last_peered":"2023-10-16T14:10:40.968913+0000","last_clean":"2023-10-16T14:10:40.968913+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:40.968913+0000","last_undegraded":"2023-10-16T14:10:40.968913+0000","last_fullsized":"2023-10-16T14:10:40.968913+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980445+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:40.980445+0000","last_peered":"2023-10-16T14:10:40.980445+0000","last_clean":"2023-10-16T14:10:40.980445+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:40.980445+0000","last_undegraded":"2023-10-16T14:10:40.980445+0000","last_fullsized":"2023-10-16T14:10:40.980445+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008653+0000","last_change":"2023-10-16T14:10:39.986399+0000","last_active":"2023-10-16T14:10:41.008653+0000","last_peered":"2023-10-16T14:10:41.008653+0000","last_clean":"2023-10-16T14:10:41.008653+0000","last_became_active":"2023-10-16T14:10:39.986098+0000","last_became_peered":"2023-10-16T14:10:39.986098+0000","last_unstale":"2023-10-16T14:10:41.008653+0000","last_undegraded":"2023-10-16T14:10:41.008653+0000","last_fullsized":"2023-10-16T14:10:41.008653+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:15.423316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144154+0000","last_change":"2023-10-16T14:10:40.991925+0000","last_active":"2023-10-16T14:10:43.144154+0000","last_peered":"2023-10-16T14:10:43.144154+0000","last_clean":"2023-10-16T14:10:43.144154+0000","last_became_active":"2023-10-16T14:10:40.991667+0000","last_became_peered":"2023-10-16T14:10:40.991667+0000","last_unstale":"2023-10-16T14:10:43.144154+0000","last_undegraded":"2023-10-16T14:10:43.144154+0000","last_fullsized":"2023-10-16T14:10:43.144154+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:25.199441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977915+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:40.977915+0000","last_peered":"2023-10-16T14:10:40.977915+0000","last_clean":"2023-10-16T14:10:40.977915+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:40.977915+0000","last_undegraded":"2023-10-16T14:10:40.977915+0000","last_fullsized":"2023-10-16T14:10:40.977915+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970555+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:40.970555+0000","last_peered":"2023-10-16T14:10:40.970555+0000","last_clean":"2023-10-16T14:10:40.970555+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:40.970555+0000","last_undegraded":"2023-10-16T14:10:40.970555+0000","last_fullsized":"2023-10-16T14:10:40.970555+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979944+0000","last_change":"2023-10-16T14:10:39.988566+0000","last_active":"2023-10-16T14:10:40.979944+0000","last_peered":"2023-10-16T14:10:40.979944+0000","last_clean":"2023-10-16T14:10:40.979944+0000","last_became_active":"2023-10-16T14:10:39.988037+0000","last_became_peered":"2023-10-16T14:10:39.988037+0000","last_unstale":"2023-10-16T14:10:40.979944+0000","last_undegraded":"2023-10-16T14:10:40.979944+0000","last_fullsized":"2023-10-16T14:10:40.979944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.694200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125491+0000","last_change":"2023-10-16T14:10:40.987050+0000","last_active":"2023-10-16T14:10:42.125491+0000","last_peered":"2023-10-16T14:10:42.125491+0000","last_clean":"2023-10-16T14:10:42.125491+0000","last_became_active":"2023-10-16T14:10:40.986733+0000","last_became_peered":"2023-10-16T14:10:40.986733+0000","last_unstale":"2023-10-16T14:10:42.125491+0000","last_undegraded":"2023-10-16T14:10:42.125491+0000","last_fullsized":"2023-10-16T14:10:42.125491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:53.222315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.989242+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:40.989242+0000","last_peered":"2023-10-16T14:10:40.989242+0000","last_clean":"2023-10-16T14:10:40.989242+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:40.989242+0000","last_undegraded":"2023-10-16T14:10:40.989242+0000","last_fullsized":"2023-10-16T14:10:40.989242+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975719+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:40.975719+0000","last_peered":"2023-10-16T14:10:40.975719+0000","last_clean":"2023-10-16T14:10:40.975719+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:40.975719+0000","last_undegraded":"2023-10-16T14:10:40.975719+0000","last_fullsized":"2023-10-16T14:10:40.975719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008672+0000","last_change":"2023-10-16T14:10:39.985490+0000","last_active":"2023-10-16T14:10:41.008672+0000","last_peered":"2023-10-16T14:10:41.008672+0000","last_clean":"2023-10-16T14:10:41.008672+0000","last_became_active":"2023-10-16T14:10:39.985187+0000","last_became_peered":"2023-10-16T14:10:39.985187+0000","last_unstale":"2023-10-16T14:10:41.008672+0000","last_undegraded":"2023-10-16T14:10:41.008672+0000","last_fullsized":"2023-10-16T14:10:41.008672+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:32.423740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125498+0000","last_change":"2023-10-16T14:10:40.989384+0000","last_active":"2023-10-16T14:10:42.125498+0000","last_peered":"2023-10-16T14:10:42.125498+0000","last_clean":"2023-10-16T14:10:42.125498+0000","last_became_active":"2023-10-16T14:10:40.989064+0000","last_became_peered":"2023-10-16T14:10:40.989064+0000","last_unstale":"2023-10-16T14:10:42.125498+0000","last_undegraded":"2023-10-16T14:10:42.125498+0000","last_fullsized":"2023-10-16T14:10:42.125498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:50.164078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974785+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:40.974785+0000","last_peered":"2023-10-16T14:10:40.974785+0000","last_clean":"2023-10-16T14:10:40.974785+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:40.974785+0000","last_undegraded":"2023-10-16T14:10:40.974785+0000","last_fullsized":"2023-10-16T14:10:40.974785+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967632+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:40.967632+0000","last_peered":"2023-10-16T14:10:40.967632+0000","last_clean":"2023-10-16T14:10:40.967632+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:40.967632+0000","last_undegraded":"2023-10-16T14:10:40.967632+0000","last_fullsized":"2023-10-16T14:10:40.967632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971923+0000","last_change":"2023-10-16T14:10:39.986732+0000","last_active":"2023-10-16T14:10:40.971923+0000","last_peered":"2023-10-16T14:10:40.971923+0000","last_clean":"2023-10-16T14:10:40.971923+0000","last_became_active":"2023-10-16T14:10:39.986276+0000","last_became_peered":"2023-10-16T14:10:39.986276+0000","last_unstale":"2023-10-16T14:10:40.971923+0000","last_undegraded":"2023-10-16T14:10:40.971923+0000","last_fullsized":"2023-10-16T14:10:40.971923+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:52.213467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143312+0000","last_change":"2023-10-16T14:10:40.994651+0000","last_active":"2023-10-16T14:10:43.143312+0000","last_peered":"2023-10-16T14:10:43.143312+0000","last_clean":"2023-10-16T14:10:43.143312+0000","last_became_active":"2023-10-16T14:10:40.994387+0000","last_became_peered":"2023-10-16T14:10:40.994387+0000","last_unstale":"2023-10-16T14:10:43.143312+0000","last_undegraded":"2023-10-16T14:10:43.143312+0000","last_fullsized":"2023-10-16T14:10:43.143312+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:54.855496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977551+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:40.977551+0000","last_peered":"2023-10-16T14:10:40.977551+0000","last_clean":"2023-10-16T14:10:40.977551+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:40.977551+0000","last_undegraded":"2023-10-16T14:10:40.977551+0000","last_fullsized":"2023-10-16T14:10:40.977551+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981376+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:40.981376+0000","last_peered":"2023-10-16T14:10:40.981376+0000","last_clean":"2023-10-16T14:10:40.981376+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:40.981376+0000","last_undegraded":"2023-10-16T14:10:40.981376+0000","last_fullsized":"2023-10-16T14:10:40.981376+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985469+0000","last_change":"2023-10-16T14:10:39.989343+0000","last_active":"2023-10-16T14:10:40.985469+0000","last_peered":"2023-10-16T14:10:40.985469+0000","last_clean":"2023-10-16T14:10:40.985469+0000","last_became_active":"2023-10-16T14:10:39.989043+0000","last_became_peered":"2023-10-16T14:10:39.989043+0000","last_unstale":"2023-10-16T14:10:40.985469+0000","last_undegraded":"2023-10-16T14:10:40.985469+0000","last_fullsized":"2023-10-16T14:10:40.985469+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:57.519740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144062+0000","last_change":"2023-10-16T14:10:40.993217+0000","last_active":"2023-10-16T14:10:43.144062+0000","last_peered":"2023-10-16T14:10:43.144062+0000","last_clean":"2023-10-16T14:10:43.144062+0000","last_became_active":"2023-10-16T14:10:40.992887+0000","last_became_peered":"2023-10-16T14:10:40.992887+0000","last_unstale":"2023-10-16T14:10:43.144062+0000","last_undegraded":"2023-10-16T14:10:43.144062+0000","last_fullsized":"2023-10-16T14:10:43.144062+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.772938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979800+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:40.979800+0000","last_peered":"2023-10-16T14:10:40.979800+0000","last_clean":"2023-10-16T14:10:40.979800+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:40.979800+0000","last_undegraded":"2023-10-16T14:10:40.979800+0000","last_fullsized":"2023-10-16T14:10:40.979800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987734+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:40.987734+0000","last_peered":"2023-10-16T14:10:40.987734+0000","last_clean":"2023-10-16T14:10:40.987734+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:40.987734+0000","last_undegraded":"2023-10-16T14:10:40.987734+0000","last_fullsized":"2023-10-16T14:10:40.987734+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980253+0000","last_change":"2023-10-16T14:10:39.979128+0000","last_active":"2023-10-16T14:10:40.980253+0000","last_peered":"2023-10-16T14:10:40.980253+0000","last_clean":"2023-10-16T14:10:40.980253+0000","last_became_active":"2023-10-16T14:10:39.978782+0000","last_became_peered":"2023-10-16T14:10:39.978782+0000","last_unstale":"2023-10-16T14:10:40.980253+0000","last_undegraded":"2023-10-16T14:10:40.980253+0000","last_fullsized":"2023-10-16T14:10:40.980253+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:28.231837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997313+0000","last_change":"2023-10-16T14:10:40.989722+0000","last_active":"2023-10-16T14:10:41.997313+0000","last_peered":"2023-10-16T14:10:41.997313+0000","last_clean":"2023-10-16T14:10:41.997313+0000","last_became_active":"2023-10-16T14:10:40.989477+0000","last_became_peered":"2023-10-16T14:10:40.989477+0000","last_unstale":"2023-10-16T14:10:41.997313+0000","last_undegraded":"2023-10-16T14:10:41.997313+0000","last_fullsized":"2023-10-16T14:10:41.997313+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971953+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:40.971953+0000","last_peered":"2023-10-16T14:10:40.971953+0000","last_clean":"2023-10-16T14:10:40.971953+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:40.971953+0000","last_undegraded":"2023-10-16T14:10:40.971953+0000","last_fullsized":"2023-10-16T14:10:40.971953+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972632+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:40.972632+0000","last_peered":"2023-10-16T14:10:40.972632+0000","last_clean":"2023-10-16T14:10:40.972632+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:40.972632+0000","last_undegraded":"2023-10-16T14:10:40.972632+0000","last_fullsized":"2023-10-16T14:10:40.972632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969118+0000","last_change":"2023-10-16T14:10:39.982849+0000","last_active":"2023-10-16T14:10:40.969118+0000","last_peered":"2023-10-16T14:10:40.969118+0000","last_clean":"2023-10-16T14:10:40.969118+0000","last_became_active":"2023-10-16T14:10:39.982560+0000","last_became_peered":"2023-10-16T14:10:39.982560+0000","last_unstale":"2023-10-16T14:10:40.969118+0000","last_undegraded":"2023-10-16T14:10:40.969118+0000","last_fullsized":"2023-10-16T14:10:40.969118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:51.220017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242743+0000","last_change":"2023-10-16T14:10:40.991901+0000","last_active":"2023-10-16T14:10:43.242743+0000","last_peered":"2023-10-16T14:10:43.242743+0000","last_clean":"2023-10-16T14:10:43.242743+0000","last_became_active":"2023-10-16T14:10:40.991610+0000","last_became_peered":"2023-10-16T14:10:40.991610+0000","last_unstale":"2023-10-16T14:10:43.242743+0000","last_undegraded":"2023-10-16T14:10:43.242743+0000","last_fullsized":"2023-10-16T14:10:43.242743+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"231'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979815+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:40.979815+0000","last_peered":"2023-10-16T14:10:40.979815+0000","last_clean":"2023-10-16T14:10:40.979815+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:40.979815+0000","last_undegraded":"2023-10-16T14:10:40.979815+0000","last_fullsized":"2023-10-16T14:10:40.979815+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978358+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:40.978358+0000","last_peered":"2023-10-16T14:10:40.978358+0000","last_clean":"2023-10-16T14:10:40.978358+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:40.978358+0000","last_undegraded":"2023-10-16T14:10:40.978358+0000","last_fullsized":"2023-10-16T14:10:40.978358+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967454+0000","last_change":"2023-10-16T14:10:39.986619+0000","last_active":"2023-10-16T14:10:40.967454+0000","last_peered":"2023-10-16T14:10:40.967454+0000","last_clean":"2023-10-16T14:10:40.967454+0000","last_became_active":"2023-10-16T14:10:39.985261+0000","last_became_peered":"2023-10-16T14:10:39.985261+0000","last_unstale":"2023-10-16T14:10:40.967454+0000","last_undegraded":"2023-10-16T14:10:40.967454+0000","last_fullsized":"2023-10-16T14:10:40.967454+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:00.760156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.141717+0000","last_change":"2023-10-16T14:10:40.990315+0000","last_active":"2023-10-16T14:10:43.141717+0000","last_peered":"2023-10-16T14:10:43.141717+0000","last_clean":"2023-10-16T14:10:43.141717+0000","last_became_active":"2023-10-16T14:10:40.990060+0000","last_became_peered":"2023-10-16T14:10:40.990060+0000","last_unstale":"2023-10-16T14:10:43.141717+0000","last_undegraded":"2023-10-16T14:10:43.141717+0000","last_fullsized":"2023-10-16T14:10:43.141717+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.954735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980892+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:40.980892+0000","last_peered":"2023-10-16T14:10:40.980892+0000","last_clean":"2023-10-16T14:10:40.980892+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:40.980892+0000","last_undegraded":"2023-10-16T14:10:40.980892+0000","last_fullsized":"2023-10-16T14:10:40.980892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969751+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:40.969751+0000","last_peered":"2023-10-16T14:10:40.969751+0000","last_clean":"2023-10-16T14:10:40.969751+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:40.969751+0000","last_undegraded":"2023-10-16T14:10:40.969751+0000","last_fullsized":"2023-10-16T14:10:40.969751+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976136+0000","last_change":"2023-10-16T14:10:39.987697+0000","last_active":"2023-10-16T14:10:40.976136+0000","last_peered":"2023-10-16T14:10:40.976136+0000","last_clean":"2023-10-16T14:10:40.976136+0000","last_became_active":"2023-10-16T14:10:39.987388+0000","last_became_peered":"2023-10-16T14:10:39.987388+0000","last_unstale":"2023-10-16T14:10:40.976136+0000","last_undegraded":"2023-10-16T14:10:40.976136+0000","last_fullsized":"2023-10-16T14:10:40.976136+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:46.559129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244071+0000","last_change":"2023-10-16T14:10:40.983968+0000","last_active":"2023-10-16T14:10:43.244071+0000","last_peered":"2023-10-16T14:10:43.244071+0000","last_clean":"2023-10-16T14:10:43.244071+0000","last_became_active":"2023-10-16T14:10:40.983650+0000","last_became_peered":"2023-10-16T14:10:40.983650+0000","last_unstale":"2023-10-16T14:10:43.244071+0000","last_undegraded":"2023-10-16T14:10:43.244071+0000","last_fullsized":"2023-10-16T14:10:43.244071+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969088+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:40.969088+0000","last_peered":"2023-10-16T14:10:40.969088+0000","last_clean":"2023-10-16T14:10:40.969088+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:40.969088+0000","last_undegraded":"2023-10-16T14:10:40.969088+0000","last_fullsized":"2023-10-16T14:10:40.969088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.989095+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:40.989095+0000","last_peered":"2023-10-16T14:10:40.989095+0000","last_clean":"2023-10-16T14:10:40.989095+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:40.989095+0000","last_undegraded":"2023-10-16T14:10:40.989095+0000","last_fullsized":"2023-10-16T14:10:40.989095+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009061+0000","last_change":"2023-10-16T14:10:39.982498+0000","last_active":"2023-10-16T14:10:41.009061+0000","last_peered":"2023-10-16T14:10:41.009061+0000","last_clean":"2023-10-16T14:10:41.009061+0000","last_became_active":"2023-10-16T14:10:39.982162+0000","last_became_peered":"2023-10-16T14:10:39.982162+0000","last_unstale":"2023-10-16T14:10:41.009061+0000","last_undegraded":"2023-10-16T14:10:41.009061+0000","last_fullsized":"2023-10-16T14:10:41.009061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:11.621670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.187140+0000","last_change":"2023-10-16T14:10:40.990434+0000","last_active":"2023-10-16T14:10:43.187140+0000","last_peered":"2023-10-16T14:10:43.187140+0000","last_clean":"2023-10-16T14:10:43.187140+0000","last_became_active":"2023-10-16T14:10:40.990134+0000","last_became_peered":"2023-10-16T14:10:40.990134+0000","last_unstale":"2023-10-16T14:10:43.187140+0000","last_undegraded":"2023-10-16T14:10:43.187140+0000","last_fullsized":"2023-10-16T14:10:43.187140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:35.937011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978483+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:40.978483+0000","last_peered":"2023-10-16T14:10:40.978483+0000","last_clean":"2023-10-16T14:10:40.978483+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:40.978483+0000","last_undegraded":"2023-10-16T14:10:40.978483+0000","last_fullsized":"2023-10-16T14:10:40.978483+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967437+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:40.967437+0000","last_peered":"2023-10-16T14:10:40.967437+0000","last_clean":"2023-10-16T14:10:40.967437+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:40.967437+0000","last_undegraded":"2023-10-16T14:10:40.967437+0000","last_fullsized":"2023-10-16T14:10:40.967437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010987+0000","last_change":"2023-10-16T14:10:39.984550+0000","last_active":"2023-10-16T14:10:41.010987+0000","last_peered":"2023-10-16T14:10:41.010987+0000","last_clean":"2023-10-16T14:10:41.010987+0000","last_became_active":"2023-10-16T14:10:39.984260+0000","last_became_peered":"2023-10-16T14:10:39.984260+0000","last_unstale":"2023-10-16T14:10:41.010987+0000","last_undegraded":"2023-10-16T14:10:41.010987+0000","last_fullsized":"2023-10-16T14:10:41.010987+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:14.438775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143524+0000","last_change":"2023-10-16T14:10:40.983245+0000","last_active":"2023-10-16T14:10:43.143524+0000","last_peered":"2023-10-16T14:10:43.143524+0000","last_clean":"2023-10-16T14:10:43.143524+0000","last_became_active":"2023-10-16T14:10:40.982951+0000","last_became_peered":"2023-10-16T14:10:40.982951+0000","last_unstale":"2023-10-16T14:10:43.143524+0000","last_undegraded":"2023-10-16T14:10:43.143524+0000","last_fullsized":"2023-10-16T14:10:43.143524+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:14.127769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981490+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:40.981490+0000","last_peered":"2023-10-16T14:10:40.981490+0000","last_clean":"2023-10-16T14:10:40.981490+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:40.981490+0000","last_undegraded":"2023-10-16T14:10:40.981490+0000","last_fullsized":"2023-10-16T14:10:40.981490+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981982+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:40.981982+0000","last_peered":"2023-10-16T14:10:40.981982+0000","last_clean":"2023-10-16T14:10:40.981982+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:40.981982+0000","last_undegraded":"2023-10-16T14:10:40.981982+0000","last_fullsized":"2023-10-16T14:10:40.981982+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961192+0000","last_change":"2023-10-16T14:10:39.981457+0000","last_active":"2023-10-16T14:10:40.961192+0000","last_peered":"2023-10-16T14:10:40.961192+0000","last_clean":"2023-10-16T14:10:40.961192+0000","last_became_active":"2023-10-16T14:10:39.981145+0000","last_became_peered":"2023-10-16T14:10:39.981145+0000","last_unstale":"2023-10-16T14:10:40.961192+0000","last_undegraded":"2023-10-16T14:10:40.961192+0000","last_fullsized":"2023-10-16T14:10:40.961192+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:33.143537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143133+0000","last_change":"2023-10-16T14:10:40.982170+0000","last_active":"2023-10-16T14:10:43.143133+0000","last_peered":"2023-10-16T14:10:43.143133+0000","last_clean":"2023-10-16T14:10:43.143133+0000","last_became_active":"2023-10-16T14:10:40.981890+0000","last_became_peered":"2023-10-16T14:10:40.981890+0000","last_unstale":"2023-10-16T14:10:43.143133+0000","last_undegraded":"2023-10-16T14:10:43.143133+0000","last_fullsized":"2023-10-16T14:10:43.143133+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:09.758522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960165+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:40.960165+0000","last_peered":"2023-10-16T14:10:40.960165+0000","last_clean":"2023-10-16T14:10:40.960165+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:40.960165+0000","last_undegraded":"2023-10-16T14:10:40.960165+0000","last_fullsized":"2023-10-16T14:10:40.960165+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960903+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:40.960903+0000","last_peered":"2023-10-16T14:10:40.960903+0000","last_clean":"2023-10-16T14:10:40.960903+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:40.960903+0000","last_undegraded":"2023-10-16T14:10:40.960903+0000","last_fullsized":"2023-10-16T14:10:40.960903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010705+0000","last_change":"2023-10-16T14:10:39.984267+0000","last_active":"2023-10-16T14:10:41.010705+0000","last_peered":"2023-10-16T14:10:41.010705+0000","last_clean":"2023-10-16T14:10:41.010705+0000","last_became_active":"2023-10-16T14:10:39.983894+0000","last_became_peered":"2023-10-16T14:10:39.983894+0000","last_unstale":"2023-10-16T14:10:41.010705+0000","last_undegraded":"2023-10-16T14:10:41.010705+0000","last_fullsized":"2023-10-16T14:10:41.010705+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:02.427201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.142906+0000","last_change":"2023-10-16T14:10:40.994626+0000","last_active":"2023-10-16T14:10:43.142906+0000","last_peered":"2023-10-16T14:10:43.142906+0000","last_clean":"2023-10-16T14:10:43.142906+0000","last_became_active":"2023-10-16T14:10:40.994283+0000","last_became_peered":"2023-10-16T14:10:40.994283+0000","last_unstale":"2023-10-16T14:10:43.142906+0000","last_undegraded":"2023-10-16T14:10:43.142906+0000","last_fullsized":"2023-10-16T14:10:43.142906+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:16.777756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979684+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:40.979684+0000","last_peered":"2023-10-16T14:10:40.979684+0000","last_clean":"2023-10-16T14:10:40.979684+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:40.979684+0000","last_undegraded":"2023-10-16T14:10:40.979684+0000","last_fullsized":"2023-10-16T14:10:40.979684+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961306+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:40.961306+0000","last_peered":"2023-10-16T14:10:40.961306+0000","last_clean":"2023-10-16T14:10:40.961306+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:40.961306+0000","last_undegraded":"2023-10-16T14:10:40.961306+0000","last_fullsized":"2023-10-16T14:10:40.961306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971464+0000","last_change":"2023-10-16T14:10:39.974033+0000","last_active":"2023-10-16T14:10:40.971464+0000","last_peered":"2023-10-16T14:10:40.971464+0000","last_clean":"2023-10-16T14:10:40.971464+0000","last_became_active":"2023-10-16T14:10:39.973723+0000","last_became_peered":"2023-10-16T14:10:39.973723+0000","last_unstale":"2023-10-16T14:10:40.971464+0000","last_undegraded":"2023-10-16T14:10:40.971464+0000","last_fullsized":"2023-10-16T14:10:40.971464+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:08.572098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124517+0000","last_change":"2023-10-16T14:10:40.994623+0000","last_active":"2023-10-16T14:10:42.124517+0000","last_peered":"2023-10-16T14:10:42.124517+0000","last_clean":"2023-10-16T14:10:42.124517+0000","last_became_active":"2023-10-16T14:10:40.994347+0000","last_became_peered":"2023-10-16T14:10:40.994347+0000","last_unstale":"2023-10-16T14:10:42.124517+0000","last_undegraded":"2023-10-16T14:10:42.124517+0000","last_fullsized":"2023-10-16T14:10:42.124517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:18.643655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961579+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:40.961579+0000","last_peered":"2023-10-16T14:10:40.961579+0000","last_clean":"2023-10-16T14:10:40.961579+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:40.961579+0000","last_undegraded":"2023-10-16T14:10:40.961579+0000","last_fullsized":"2023-10-16T14:10:40.961579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979554+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:40.979554+0000","last_peered":"2023-10-16T14:10:40.979554+0000","last_clean":"2023-10-16T14:10:40.979554+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:40.979554+0000","last_undegraded":"2023-10-16T14:10:40.979554+0000","last_fullsized":"2023-10-16T14:10:40.979554+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009801+0000","last_change":"2023-10-16T14:10:39.986465+0000","last_active":"2023-10-16T14:10:41.009801+0000","last_peered":"2023-10-16T14:10:41.009801+0000","last_clean":"2023-10-16T14:10:41.009801+0000","last_became_active":"2023-10-16T14:10:39.986165+0000","last_became_peered":"2023-10-16T14:10:39.986165+0000","last_unstale":"2023-10-16T14:10:41.009801+0000","last_undegraded":"2023-10-16T14:10:41.009801+0000","last_fullsized":"2023-10-16T14:10:41.009801+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:08.068911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124542+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:42.124542+0000","last_peered":"2023-10-16T14:10:42.124542+0000","last_clean":"2023-10-16T14:10:42.124542+0000","last_became_active":"2023-10-16T14:10:40.993130+0000","last_became_peered":"2023-10-16T14:10:40.993130+0000","last_unstale":"2023-10-16T14:10:42.124542+0000","last_undegraded":"2023-10-16T14:10:42.124542+0000","last_fullsized":"2023-10-16T14:10:42.124542+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:25.881584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973127+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:40.973127+0000","last_peered":"2023-10-16T14:10:40.973127+0000","last_clean":"2023-10-16T14:10:40.973127+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:40.973127+0000","last_undegraded":"2023-10-16T14:10:40.973127+0000","last_fullsized":"2023-10-16T14:10:40.973127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974510+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:40.974510+0000","last_peered":"2023-10-16T14:10:40.974510+0000","last_clean":"2023-10-16T14:10:40.974510+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:40.974510+0000","last_undegraded":"2023-10-16T14:10:40.974510+0000","last_fullsized":"2023-10-16T14:10:40.974510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967349+0000","last_change":"2023-10-16T14:10:39.986684+0000","last_active":"2023-10-16T14:10:40.967349+0000","last_peered":"2023-10-16T14:10:40.967349+0000","last_clean":"2023-10-16T14:10:40.967349+0000","last_became_active":"2023-10-16T14:10:39.986367+0000","last_became_peered":"2023-10-16T14:10:39.986367+0000","last_unstale":"2023-10-16T14:10:40.967349+0000","last_undegraded":"2023-10-16T14:10:40.967349+0000","last_fullsized":"2023-10-16T14:10:40.967349+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:33:11.665429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185918+0000","last_change":"2023-10-16T14:10:40.993254+0000","last_active":"2023-10-16T14:10:43.185918+0000","last_peered":"2023-10-16T14:10:43.185918+0000","last_clean":"2023-10-16T14:10:43.185918+0000","last_became_active":"2023-10-16T14:10:40.992918+0000","last_became_peered":"2023-10-16T14:10:40.992918+0000","last_unstale":"2023-10-16T14:10:43.185918+0000","last_undegraded":"2023-10-16T14:10:43.185918+0000","last_fullsized":"2023-10-16T14:10:43.185918+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:07.376549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976318+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:40.976318+0000","last_peered":"2023-10-16T14:10:40.976318+0000","last_clean":"2023-10-16T14:10:40.976318+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:40.976318+0000","last_undegraded":"2023-10-16T14:10:40.976318+0000","last_fullsized":"2023-10-16T14:10:40.976318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972611+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:40.972611+0000","last_peered":"2023-10-16T14:10:40.972611+0000","last_clean":"2023-10-16T14:10:40.972611+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:40.972611+0000","last_undegraded":"2023-10-16T14:10:40.972611+0000","last_fullsized":"2023-10-16T14:10:40.972611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970330+0000","last_change":"2023-10-16T14:10:39.989051+0000","last_active":"2023-10-16T14:10:40.970330+0000","last_peered":"2023-10-16T14:10:40.970330+0000","last_clean":"2023-10-16T14:10:40.970330+0000","last_became_active":"2023-10-16T14:10:39.987944+0000","last_became_peered":"2023-10-16T14:10:39.987944+0000","last_unstale":"2023-10-16T14:10:40.970330+0000","last_undegraded":"2023-10-16T14:10:40.970330+0000","last_fullsized":"2023-10-16T14:10:40.970330+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244927+0000","last_change":"2023-10-16T14:10:40.996616+0000","last_active":"2023-10-16T14:10:43.244927+0000","last_peered":"2023-10-16T14:10:43.244927+0000","last_clean":"2023-10-16T14:10:43.244927+0000","last_became_active":"2023-10-16T14:10:40.996057+0000","last_became_peered":"2023-10-16T14:10:40.996057+0000","last_unstale":"2023-10-16T14:10:43.244927+0000","last_undegraded":"2023-10-16T14:10:43.244927+0000","last_fullsized":"2023-10-16T14:10:43.244927+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:32.328604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981955+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:40.981955+0000","last_peered":"2023-10-16T14:10:40.981955+0000","last_clean":"2023-10-16T14:10:40.981955+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:40.981955+0000","last_undegraded":"2023-10-16T14:10:40.981955+0000","last_fullsized":"2023-10-16T14:10:40.981955+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978301+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:40.978301+0000","last_peered":"2023-10-16T14:10:40.978301+0000","last_clean":"2023-10-16T14:10:40.978301+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:40.978301+0000","last_undegraded":"2023-10-16T14:10:40.978301+0000","last_fullsized":"2023-10-16T14:10:40.978301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980655+0000","last_change":"2023-10-16T14:10:39.972329+0000","last_active":"2023-10-16T14:10:40.980655+0000","last_peered":"2023-10-16T14:10:40.980655+0000","last_clean":"2023-10-16T14:10:40.980655+0000","last_became_active":"2023-10-16T14:10:39.972015+0000","last_became_peered":"2023-10-16T14:10:39.972015+0000","last_unstale":"2023-10-16T14:10:40.980655+0000","last_undegraded":"2023-10-16T14:10:40.980655+0000","last_fullsized":"2023-10-16T14:10:40.980655+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:17.467680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996971+0000","last_change":"2023-10-16T14:10:41.001961+0000","last_active":"2023-10-16T14:10:41.996971+0000","last_peered":"2023-10-16T14:10:41.996971+0000","last_clean":"2023-10-16T14:10:41.996971+0000","last_became_active":"2023-10-16T14:10:41.001699+0000","last_became_peered":"2023-10-16T14:10:41.001699+0000","last_unstale":"2023-10-16T14:10:41.996971+0000","last_undegraded":"2023-10-16T14:10:41.996971+0000","last_fullsized":"2023-10-16T14:10:41.996971+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959024+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:40.959024+0000","last_peered":"2023-10-16T14:10:40.959024+0000","last_clean":"2023-10-16T14:10:40.959024+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:40.959024+0000","last_undegraded":"2023-10-16T14:10:40.959024+0000","last_fullsized":"2023-10-16T14:10:40.959024+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975612+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:40.975612+0000","last_peered":"2023-10-16T14:10:40.975612+0000","last_clean":"2023-10-16T14:10:40.975612+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:40.975612+0000","last_undegraded":"2023-10-16T14:10:40.975612+0000","last_fullsized":"2023-10-16T14:10:40.975612+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985242+0000","last_change":"2023-10-16T14:10:39.990904+0000","last_active":"2023-10-16T14:10:40.985242+0000","last_peered":"2023-10-16T14:10:40.985242+0000","last_clean":"2023-10-16T14:10:40.985242+0000","last_became_active":"2023-10-16T14:10:39.990155+0000","last_became_peered":"2023-10-16T14:10:39.990155+0000","last_unstale":"2023-10-16T14:10:40.985242+0000","last_undegraded":"2023-10-16T14:10:40.985242+0000","last_fullsized":"2023-10-16T14:10:40.985242+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:56.282178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243767+0000","last_change":"2023-10-16T14:10:40.993640+0000","last_active":"2023-10-16T14:10:43.243767+0000","last_peered":"2023-10-16T14:10:43.243767+0000","last_clean":"2023-10-16T14:10:43.243767+0000","last_became_active":"2023-10-16T14:10:40.993084+0000","last_became_peered":"2023-10-16T14:10:40.993084+0000","last_unstale":"2023-10-16T14:10:43.243767+0000","last_undegraded":"2023-10-16T14:10:43.243767+0000","last_fullsized":"2023-10-16T14:10:43.243767+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:12.061147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987125+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:40.987125+0000","last_peered":"2023-10-16T14:10:40.987125+0000","last_clean":"2023-10-16T14:10:40.987125+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:40.987125+0000","last_undegraded":"2023-10-16T14:10:40.987125+0000","last_fullsized":"2023-10-16T14:10:40.987125+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959827+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:40.959827+0000","last_peered":"2023-10-16T14:10:40.959827+0000","last_clean":"2023-10-16T14:10:40.959827+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:40.959827+0000","last_undegraded":"2023-10-16T14:10:40.959827+0000","last_fullsized":"2023-10-16T14:10:40.959827+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"234'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013944+0000","last_change":"2023-10-16T14:10:39.971085+0000","last_active":"2023-10-16T14:10:41.013944+0000","last_peered":"2023-10-16T14:10:41.013944+0000","last_clean":"2023-10-16T14:10:41.013944+0000","last_became_active":"2023-10-16T14:10:39.970777+0000","last_became_peered":"2023-10-16T14:10:39.970777+0000","last_unstale":"2023-10-16T14:10:41.013944+0000","last_undegraded":"2023-10-16T14:10:41.013944+0000","last_fullsized":"2023-10-16T14:10:41.013944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:03.090258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.998098+0000","last_change":"2023-10-16T14:10:40.985478+0000","last_active":"2023-10-16T14:10:41.998098+0000","last_peered":"2023-10-16T14:10:41.998098+0000","last_clean":"2023-10-16T14:10:41.998098+0000","last_became_active":"2023-10-16T14:10:40.985218+0000","last_became_peered":"2023-10-16T14:10:40.985218+0000","last_unstale":"2023-10-16T14:10:41.998098+0000","last_undegraded":"2023-10-16T14:10:41.998098+0000","last_fullsized":"2023-10-16T14:10:41.998098+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:22.942748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.986478+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:40.986478+0000","last_peered":"2023-10-16T14:10:40.986478+0000","last_clean":"2023-10-16T14:10:40.986478+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:40.986478+0000","last_undegraded":"2023-10-16T14:10:40.986478+0000","last_fullsized":"2023-10-16T14:10:40.986478+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974315+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:40.974315+0000","last_peered":"2023-10-16T14:10:40.974315+0000","last_clean":"2023-10-16T14:10:40.974315+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:40.974315+0000","last_undegraded":"2023-10-16T14:10:40.974315+0000","last_fullsized":"2023-10-16T14:10:40.974315+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"234'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008061+0000","last_change":"2023-10-16T14:10:39.973954+0000","last_active":"2023-10-16T14:10:41.008061+0000","last_peered":"2023-10-16T14:10:41.008061+0000","last_clean":"2023-10-16T14:10:41.008061+0000","last_became_active":"2023-10-16T14:10:39.973602+0000","last_became_peered":"2023-10-16T14:10:39.973602+0000","last_unstale":"2023-10-16T14:10:41.008061+0000","last_undegraded":"2023-10-16T14:10:41.008061+0000","last_fullsized":"2023-10-16T14:10:41.008061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:18.496303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143885+0000","last_change":"2023-10-16T14:10:40.991452+0000","last_active":"2023-10-16T14:10:43.143885+0000","last_peered":"2023-10-16T14:10:43.143885+0000","last_clean":"2023-10-16T14:10:43.143885+0000","last_became_active":"2023-10-16T14:10:40.985777+0000","last_became_peered":"2023-10-16T14:10:40.985777+0000","last_unstale":"2023-10-16T14:10:43.143885+0000","last_undegraded":"2023-10-16T14:10:43.143885+0000","last_fullsized":"2023-10-16T14:10:43.143885+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.881711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"231'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.983108+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:40.983108+0000","last_peered":"2023-10-16T14:10:40.983108+0000","last_clean":"2023-10-16T14:10:40.983108+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:40.983108+0000","last_undegraded":"2023-10-16T14:10:40.983108+0000","last_fullsized":"2023-10-16T14:10:40.983108+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977730+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:40.977730+0000","last_peered":"2023-10-16T14:10:40.977730+0000","last_clean":"2023-10-16T14:10:40.977730+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:40.977730+0000","last_undegraded":"2023-10-16T14:10:40.977730+0000","last_fullsized":"2023-10-16T14:10:40.977730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980420+0000","last_change":"2023-10-16T14:10:39.976858+0000","last_active":"2023-10-16T14:10:40.980420+0000","last_peered":"2023-10-16T14:10:40.980420+0000","last_clean":"2023-10-16T14:10:40.980420+0000","last_became_active":"2023-10-16T14:10:39.976566+0000","last_became_peered":"2023-10-16T14:10:39.976566+0000","last_unstale":"2023-10-16T14:10:40.980420+0000","last_undegraded":"2023-10-16T14:10:40.980420+0000","last_fullsized":"2023-10-16T14:10:40.980420+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:00.617867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.990318+0000","last_change":"2023-10-16T14:10:40.985099+0000","last_active":"2023-10-16T14:10:41.990318+0000","last_peered":"2023-10-16T14:10:41.990318+0000","last_clean":"2023-10-16T14:10:41.990318+0000","last_became_active":"2023-10-16T14:10:40.984799+0000","last_became_peered":"2023-10-16T14:10:40.984799+0000","last_unstale":"2023-10-16T14:10:41.990318+0000","last_undegraded":"2023-10-16T14:10:41.990318+0000","last_fullsized":"2023-10-16T14:10:41.990318+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:02.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":19,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961436+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:40.961436+0000","last_peered":"2023-10-16T14:10:40.961436+0000","last_clean":"2023-10-16T14:10:40.961436+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:40.961436+0000","last_undegraded":"2023-10-16T14:10:40.961436+0000","last_fullsized":"2023-10-16T14:10:40.961436+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982473+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:40.982473+0000","last_peered":"2023-10-16T14:10:40.982473+0000","last_clean":"2023-10-16T14:10:40.982473+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:40.982473+0000","last_undegraded":"2023-10-16T14:10:40.982473+0000","last_fullsized":"2023-10-16T14:10:40.982473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011710+0000","last_change":"2023-10-16T14:10:39.986334+0000","last_active":"2023-10-16T14:10:41.011710+0000","last_peered":"2023-10-16T14:10:41.011710+0000","last_clean":"2023-10-16T14:10:41.011710+0000","last_became_active":"2023-10-16T14:10:39.986002+0000","last_became_peered":"2023-10-16T14:10:39.986002+0000","last_unstale":"2023-10-16T14:10:41.011710+0000","last_undegraded":"2023-10-16T14:10:41.011710+0000","last_fullsized":"2023-10-16T14:10:41.011710+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994013+0000","last_change":"2023-10-16T14:10:40.976822+0000","last_active":"2023-10-16T14:10:41.994013+0000","last_peered":"2023-10-16T14:10:41.994013+0000","last_clean":"2023-10-16T14:10:41.994013+0000","last_became_active":"2023-10-16T14:10:40.976523+0000","last_became_peered":"2023-10-16T14:10:40.976523+0000","last_unstale":"2023-10-16T14:10:41.994013+0000","last_undegraded":"2023-10-16T14:10:41.994013+0000","last_fullsized":"2023-10-16T14:10:41.994013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987031+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:40.987031+0000","last_peered":"2023-10-16T14:10:40.987031+0000","last_clean":"2023-10-16T14:10:40.987031+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:40.987031+0000","last_undegraded":"2023-10-16T14:10:40.987031+0000","last_fullsized":"2023-10-16T14:10:40.987031+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980779+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:40.980779+0000","last_peered":"2023-10-16T14:10:40.980779+0000","last_clean":"2023-10-16T14:10:40.980779+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:40.980779+0000","last_undegraded":"2023-10-16T14:10:40.980779+0000","last_fullsized":"2023-10-16T14:10:40.980779+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"234'8","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013387+0000","last_change":"2023-10-16T14:10:39.979255+0000","last_active":"2023-10-16T14:10:41.013387+0000","last_peered":"2023-10-16T14:10:41.013387+0000","last_clean":"2023-10-16T14:10:41.013387+0000","last_became_active":"2023-10-16T14:10:39.977217+0000","last_became_peered":"2023-10-16T14:10:39.977217+0000","last_unstale":"2023-10-16T14:10:41.013387+0000","last_undegraded":"2023-10-16T14:10:41.013387+0000","last_fullsized":"2023-10-16T14:10:41.013387+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:30.019236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.142508+0000","last_change":"2023-10-16T14:10:40.989802+0000","last_active":"2023-10-16T14:10:43.142508+0000","last_peered":"2023-10-16T14:10:43.142508+0000","last_clean":"2023-10-16T14:10:43.142508+0000","last_became_active":"2023-10-16T14:10:40.988409+0000","last_became_peered":"2023-10-16T14:10:40.988409+0000","last_unstale":"2023-10-16T14:10:43.142508+0000","last_undegraded":"2023-10-16T14:10:43.142508+0000","last_fullsized":"2023-10-16T14:10:43.142508+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:16.220155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976181+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:40.976181+0000","last_peered":"2023-10-16T14:10:40.976181+0000","last_clean":"2023-10-16T14:10:40.976181+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:40.976181+0000","last_undegraded":"2023-10-16T14:10:40.976181+0000","last_fullsized":"2023-10-16T14:10:40.976181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979705+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:40.979705+0000","last_peered":"2023-10-16T14:10:40.979705+0000","last_clean":"2023-10-16T14:10:40.979705+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:40.979705+0000","last_undegraded":"2023-10-16T14:10:40.979705+0000","last_fullsized":"2023-10-16T14:10:40.979705+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975789+0000","last_change":"2023-10-16T14:10:39.968896+0000","last_active":"2023-10-16T14:10:40.975789+0000","last_peered":"2023-10-16T14:10:40.975789+0000","last_clean":"2023-10-16T14:10:40.975789+0000","last_became_active":"2023-10-16T14:10:39.968633+0000","last_became_peered":"2023-10-16T14:10:39.968633+0000","last_unstale":"2023-10-16T14:10:40.975789+0000","last_undegraded":"2023-10-16T14:10:40.975789+0000","last_fullsized":"2023-10-16T14:10:40.975789+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:26.024819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186867+0000","last_change":"2023-10-16T14:10:40.999802+0000","last_active":"2023-10-16T14:10:43.186867+0000","last_peered":"2023-10-16T14:10:43.186867+0000","last_clean":"2023-10-16T14:10:43.186867+0000","last_became_active":"2023-10-16T14:10:40.999483+0000","last_became_peered":"2023-10-16T14:10:40.999483+0000","last_unstale":"2023-10-16T14:10:43.186867+0000","last_undegraded":"2023-10-16T14:10:43.186867+0000","last_fullsized":"2023-10-16T14:10:43.186867+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:19.512934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011187+0000","last_change":"2023-10-16T14:10:39.979452+0000","last_active":"2023-10-16T14:10:41.011187+0000","last_peered":"2023-10-16T14:10:41.011187+0000","last_clean":"2023-10-16T14:10:41.011187+0000","last_became_active":"2023-10-16T14:10:39.978243+0000","last_became_peered":"2023-10-16T14:10:39.978243+0000","last_unstale":"2023-10-16T14:10:41.011187+0000","last_undegraded":"2023-10-16T14:10:41.011187+0000","last_fullsized":"2023-10-16T14:10:41.011187+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.984244+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:40.984244+0000","last_peered":"2023-10-16T14:10:40.984244+0000","last_clean":"2023-10-16T14:10:40.984244+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:40.984244+0000","last_undegraded":"2023-10-16T14:10:40.984244+0000","last_fullsized":"2023-10-16T14:10:40.984244+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976062+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:40.976062+0000","last_peered":"2023-10-16T14:10:40.976062+0000","last_clean":"2023-10-16T14:10:40.976062+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:40.976062+0000","last_undegraded":"2023-10-16T14:10:40.976062+0000","last_fullsized":"2023-10-16T14:10:40.976062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144120+0000","last_change":"2023-10-16T14:10:40.996750+0000","last_active":"2023-10-16T14:10:43.144120+0000","last_peered":"2023-10-16T14:10:43.144120+0000","last_clean":"2023-10-16T14:10:43.144120+0000","last_became_active":"2023-10-16T14:10:40.996369+0000","last_became_peered":"2023-10-16T14:10:40.996369+0000","last_unstale":"2023-10-16T14:10:43.144120+0000","last_undegraded":"2023-10-16T14:10:43.144120+0000","last_fullsized":"2023-10-16T14:10:43.144120+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:29.608507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"234'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009270+0000","last_change":"2023-10-16T14:10:39.973971+0000","last_active":"2023-10-16T14:10:41.009270+0000","last_peered":"2023-10-16T14:10:41.009270+0000","last_clean":"2023-10-16T14:10:41.009270+0000","last_became_active":"2023-10-16T14:10:39.973675+0000","last_became_peered":"2023-10-16T14:10:39.973675+0000","last_unstale":"2023-10-16T14:10:41.009270+0000","last_undegraded":"2023-10-16T14:10:41.009270+0000","last_fullsized":"2023-10-16T14:10:41.009270+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:01.458774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976322+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:40.976322+0000","last_peered":"2023-10-16T14:10:40.976322+0000","last_clean":"2023-10-16T14:10:40.976322+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:40.976322+0000","last_undegraded":"2023-10-16T14:10:40.976322+0000","last_fullsized":"2023-10-16T14:10:40.976322+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968823+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:40.968823+0000","last_peered":"2023-10-16T14:10:40.968823+0000","last_clean":"2023-10-16T14:10:40.968823+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:40.968823+0000","last_undegraded":"2023-10-16T14:10:40.968823+0000","last_fullsized":"2023-10-16T14:10:40.968823+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995327+0000","last_change":"2023-10-16T14:10:40.984926+0000","last_active":"2023-10-16T14:10:41.995327+0000","last_peered":"2023-10-16T14:10:41.995327+0000","last_clean":"2023-10-16T14:10:41.995327+0000","last_became_active":"2023-10-16T14:10:40.984364+0000","last_became_peered":"2023-10-16T14:10:40.984364+0000","last_unstale":"2023-10-16T14:10:41.995327+0000","last_undegraded":"2023-10-16T14:10:41.995327+0000","last_fullsized":"2023-10-16T14:10:41.995327+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974558+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:40.974558+0000","last_peered":"2023-10-16T14:10:40.974558+0000","last_clean":"2023-10-16T14:10:40.974558+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:40.974558+0000","last_undegraded":"2023-10-16T14:10:40.974558+0000","last_fullsized":"2023-10-16T14:10:40.974558+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977408+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:40.977408+0000","last_peered":"2023-10-16T14:10:40.977408+0000","last_clean":"2023-10-16T14:10:40.977408+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:40.977408+0000","last_undegraded":"2023-10-16T14:10:40.977408+0000","last_fullsized":"2023-10-16T14:10:40.977408+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011689+0000","last_change":"2023-10-16T14:10:39.979320+0000","last_active":"2023-10-16T14:10:41.011689+0000","last_peered":"2023-10-16T14:10:41.011689+0000","last_clean":"2023-10-16T14:10:41.011689+0000","last_became_active":"2023-10-16T14:10:39.978947+0000","last_became_peered":"2023-10-16T14:10:39.978947+0000","last_unstale":"2023-10-16T14:10:41.011689+0000","last_undegraded":"2023-10-16T14:10:41.011689+0000","last_fullsized":"2023-10-16T14:10:41.011689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125751+0000","last_change":"2023-10-16T14:10:40.997566+0000","last_active":"2023-10-16T14:10:42.125751+0000","last_peered":"2023-10-16T14:10:42.125751+0000","last_clean":"2023-10-16T14:10:42.125751+0000","last_became_active":"2023-10-16T14:10:40.997273+0000","last_became_peered":"2023-10-16T14:10:40.997273+0000","last_unstale":"2023-10-16T14:10:42.125751+0000","last_undegraded":"2023-10-16T14:10:42.125751+0000","last_fullsized":"2023-10-16T14:10:42.125751+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:38.615513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982594+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:40.982594+0000","last_peered":"2023-10-16T14:10:40.982594+0000","last_clean":"2023-10-16T14:10:40.982594+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:40.982594+0000","last_undegraded":"2023-10-16T14:10:40.982594+0000","last_fullsized":"2023-10-16T14:10:40.982594+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960137+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:40.960137+0000","last_peered":"2023-10-16T14:10:40.960137+0000","last_clean":"2023-10-16T14:10:40.960137+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:40.960137+0000","last_undegraded":"2023-10-16T14:10:40.960137+0000","last_fullsized":"2023-10-16T14:10:40.960137+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009413+0000","last_change":"2023-10-16T14:10:39.983838+0000","last_active":"2023-10-16T14:10:41.009413+0000","last_peered":"2023-10-16T14:10:41.009413+0000","last_clean":"2023-10-16T14:10:41.009413+0000","last_became_active":"2023-10-16T14:10:39.983415+0000","last_became_peered":"2023-10-16T14:10:39.983415+0000","last_unstale":"2023-10-16T14:10:41.009413+0000","last_undegraded":"2023-10-16T14:10:41.009413+0000","last_fullsized":"2023-10-16T14:10:41.009413+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:58.890985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243037+0000","last_change":"2023-10-16T14:10:40.993412+0000","last_active":"2023-10-16T14:10:43.243037+0000","last_peered":"2023-10-16T14:10:43.243037+0000","last_clean":"2023-10-16T14:10:43.243037+0000","last_became_active":"2023-10-16T14:10:40.993119+0000","last_became_peered":"2023-10-16T14:10:40.993119+0000","last_unstale":"2023-10-16T14:10:43.243037+0000","last_undegraded":"2023-10-16T14:10:43.243037+0000","last_fullsized":"2023-10-16T14:10:43.243037+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:58.022778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981800+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:40.981800+0000","last_peered":"2023-10-16T14:10:40.981800+0000","last_clean":"2023-10-16T14:10:40.981800+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:40.981800+0000","last_undegraded":"2023-10-16T14:10:40.981800+0000","last_fullsized":"2023-10-16T14:10:40.981800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980136+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:40.980136+0000","last_peered":"2023-10-16T14:10:40.980136+0000","last_clean":"2023-10-16T14:10:40.980136+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:40.980136+0000","last_undegraded":"2023-10-16T14:10:40.980136+0000","last_fullsized":"2023-10-16T14:10:40.980136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011809+0000","last_change":"2023-10-16T14:10:39.969078+0000","last_active":"2023-10-16T14:10:41.011809+0000","last_peered":"2023-10-16T14:10:41.011809+0000","last_clean":"2023-10-16T14:10:41.011809+0000","last_became_active":"2023-10-16T14:10:39.968736+0000","last_became_peered":"2023-10-16T14:10:39.968736+0000","last_unstale":"2023-10-16T14:10:41.011809+0000","last_undegraded":"2023-10-16T14:10:41.011809+0000","last_fullsized":"2023-10-16T14:10:41.011809+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:59.354986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997370+0000","last_change":"2023-10-16T14:10:40.997896+0000","last_active":"2023-10-16T14:10:41.997370+0000","last_peered":"2023-10-16T14:10:41.997370+0000","last_clean":"2023-10-16T14:10:41.997370+0000","last_became_active":"2023-10-16T14:10:40.997623+0000","last_became_peered":"2023-10-16T14:10:40.997623+0000","last_unstale":"2023-10-16T14:10:41.997370+0000","last_undegraded":"2023-10-16T14:10:41.997370+0000","last_fullsized":"2023-10-16T14:10:41.997370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972873+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:40.972873+0000","last_peered":"2023-10-16T14:10:40.972873+0000","last_clean":"2023-10-16T14:10:40.972873+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:40.972873+0000","last_undegraded":"2023-10-16T14:10:40.972873+0000","last_fullsized":"2023-10-16T14:10:40.972873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.988803+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:40.988803+0000","last_peered":"2023-10-16T14:10:40.988803+0000","last_clean":"2023-10-16T14:10:40.988803+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:40.988803+0000","last_undegraded":"2023-10-16T14:10:40.988803+0000","last_fullsized":"2023-10-16T14:10:40.988803+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959567+0000","last_change":"2023-10-16T14:10:39.973296+0000","last_active":"2023-10-16T14:10:40.959567+0000","last_peered":"2023-10-16T14:10:40.959567+0000","last_clean":"2023-10-16T14:10:40.959567+0000","last_became_active":"2023-10-16T14:10:39.972592+0000","last_became_peered":"2023-10-16T14:10:39.972592+0000","last_unstale":"2023-10-16T14:10:40.959567+0000","last_undegraded":"2023-10-16T14:10:40.959567+0000","last_fullsized":"2023-10-16T14:10:40.959567+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:04.406174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242344+0000","last_change":"2023-10-16T14:10:40.987363+0000","last_active":"2023-10-16T14:10:43.242344+0000","last_peered":"2023-10-16T14:10:43.242344+0000","last_clean":"2023-10-16T14:10:43.242344+0000","last_became_active":"2023-10-16T14:10:40.985914+0000","last_became_peered":"2023-10-16T14:10:40.985914+0000","last_unstale":"2023-10-16T14:10:43.242344+0000","last_undegraded":"2023-10-16T14:10:43.242344+0000","last_fullsized":"2023-10-16T14:10:43.242344+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:10.423467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975348+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:40.975348+0000","last_peered":"2023-10-16T14:10:40.975348+0000","last_clean":"2023-10-16T14:10:40.975348+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:40.975348+0000","last_undegraded":"2023-10-16T14:10:40.975348+0000","last_fullsized":"2023-10-16T14:10:40.975348+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972050+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:40.972050+0000","last_peered":"2023-10-16T14:10:40.972050+0000","last_clean":"2023-10-16T14:10:40.972050+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:40.972050+0000","last_undegraded":"2023-10-16T14:10:40.972050+0000","last_fullsized":"2023-10-16T14:10:40.972050+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970323+0000","last_change":"2023-10-16T14:10:39.972648+0000","last_active":"2023-10-16T14:10:40.970323+0000","last_peered":"2023-10-16T14:10:40.970323+0000","last_clean":"2023-10-16T14:10:40.970323+0000","last_became_active":"2023-10-16T14:10:39.972328+0000","last_became_peered":"2023-10-16T14:10:39.972328+0000","last_unstale":"2023-10-16T14:10:40.970323+0000","last_undegraded":"2023-10-16T14:10:40.970323+0000","last_fullsized":"2023-10-16T14:10:40.970323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186986+0000","last_change":"2023-10-16T14:10:40.995949+0000","last_active":"2023-10-16T14:10:43.186986+0000","last_peered":"2023-10-16T14:10:43.186986+0000","last_clean":"2023-10-16T14:10:43.186986+0000","last_became_active":"2023-10-16T14:10:40.995689+0000","last_became_peered":"2023-10-16T14:10:40.995689+0000","last_unstale":"2023-10-16T14:10:43.186986+0000","last_undegraded":"2023-10-16T14:10:43.186986+0000","last_fullsized":"2023-10-16T14:10:43.186986+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:12.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977098+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:40.977098+0000","last_peered":"2023-10-16T14:10:40.977098+0000","last_clean":"2023-10-16T14:10:40.977098+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:40.977098+0000","last_undegraded":"2023-10-16T14:10:40.977098+0000","last_fullsized":"2023-10-16T14:10:40.977098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244569+0000","last_change":"2023-10-16T14:10:41.001647+0000","last_active":"2023-10-16T14:10:43.244569+0000","last_peered":"2023-10-16T14:10:43.244569+0000","last_clean":"2023-10-16T14:10:43.244569+0000","last_became_active":"2023-10-16T14:10:41.000840+0000","last_became_peered":"2023-10-16T14:10:41.000840+0000","last_unstale":"2023-10-16T14:10:43.244569+0000","last_undegraded":"2023-10-16T14:10:43.244569+0000","last_fullsized":"2023-10-16T14:10:43.244569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:55.904685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959155+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:40.959155+0000","last_peered":"2023-10-16T14:10:40.959155+0000","last_clean":"2023-10-16T14:10:40.959155+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:40.959155+0000","last_undegraded":"2023-10-16T14:10:40.959155+0000","last_fullsized":"2023-10-16T14:10:40.959155+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243482+0000","last_change":"2023-10-16T14:10:40.991703+0000","last_active":"2023-10-16T14:10:43.243482+0000","last_peered":"2023-10-16T14:10:43.243482+0000","last_clean":"2023-10-16T14:10:43.243482+0000","last_became_active":"2023-10-16T14:10:40.991364+0000","last_became_peered":"2023-10-16T14:10:40.991364+0000","last_unstale":"2023-10-16T14:10:43.243482+0000","last_undegraded":"2023-10-16T14:10:43.243482+0000","last_fullsized":"2023-10-16T14:10:43.243482+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:30.238400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971730+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:40.971730+0000","last_peered":"2023-10-16T14:10:40.971730+0000","last_clean":"2023-10-16T14:10:40.971730+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:40.971730+0000","last_undegraded":"2023-10-16T14:10:40.971730+0000","last_fullsized":"2023-10-16T14:10:40.971730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244638+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:43.244638+0000","last_peered":"2023-10-16T14:10:43.244638+0000","last_clean":"2023-10-16T14:10:43.244638+0000","last_became_active":"2023-10-16T14:10:40.992264+0000","last_became_peered":"2023-10-16T14:10:40.992264+0000","last_unstale":"2023-10-16T14:10:43.244638+0000","last_undegraded":"2023-10-16T14:10:43.244638+0000","last_fullsized":"2023-10-16T14:10:43.244638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:32.614992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967372+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:40.967372+0000","last_peered":"2023-10-16T14:10:40.967372+0000","last_clean":"2023-10-16T14:10:40.967372+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:40.967372+0000","last_undegraded":"2023-10-16T14:10:40.967372+0000","last_fullsized":"2023-10-16T14:10:40.967372+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242825+0000","last_change":"2023-10-16T14:10:40.989459+0000","last_active":"2023-10-16T14:10:43.242825+0000","last_peered":"2023-10-16T14:10:43.242825+0000","last_clean":"2023-10-16T14:10:43.242825+0000","last_became_active":"2023-10-16T14:10:40.989197+0000","last_became_peered":"2023-10-16T14:10:40.989197+0000","last_unstale":"2023-10-16T14:10:43.242825+0000","last_undegraded":"2023-10-16T14:10:43.242825+0000","last_fullsized":"2023-10-16T14:10:43.242825+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979422+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:40.979422+0000","last_peered":"2023-10-16T14:10:40.979422+0000","last_clean":"2023-10-16T14:10:40.979422+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:40.979422+0000","last_undegraded":"2023-10-16T14:10:40.979422+0000","last_fullsized":"2023-10-16T14:10:40.979422+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994059+0000","last_change":"2023-10-16T14:10:40.986689+0000","last_active":"2023-10-16T14:10:41.994059+0000","last_peered":"2023-10-16T14:10:41.994059+0000","last_clean":"2023-10-16T14:10:41.994059+0000","last_became_active":"2023-10-16T14:10:40.986408+0000","last_became_peered":"2023-10-16T14:10:40.986408+0000","last_unstale":"2023-10-16T14:10:41.994059+0000","last_undegraded":"2023-10-16T14:10:41.994059+0000","last_fullsized":"2023-10-16T14:10:41.994059+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975431+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:40.975431+0000","last_peered":"2023-10-16T14:10:40.975431+0000","last_clean":"2023-10-16T14:10:40.975431+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:40.975431+0000","last_undegraded":"2023-10-16T14:10:40.975431+0000","last_fullsized":"2023-10-16T14:10:40.975431+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989882+0000","last_change":"2023-10-16T14:10:40.987586+0000","last_active":"2023-10-16T14:10:41.989882+0000","last_peered":"2023-10-16T14:10:41.989882+0000","last_clean":"2023-10-16T14:10:41.989882+0000","last_became_active":"2023-10-16T14:10:40.987254+0000","last_became_peered":"2023-10-16T14:10:40.987254+0000","last_unstale":"2023-10-16T14:10:41.989882+0000","last_undegraded":"2023-10-16T14:10:41.989882+0000","last_fullsized":"2023-10-16T14:10:41.989882+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.463597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980939+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:40.980939+0000","last_peered":"2023-10-16T14:10:40.980939+0000","last_clean":"2023-10-16T14:10:40.980939+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:40.980939+0000","last_undegraded":"2023-10-16T14:10:40.980939+0000","last_fullsized":"2023-10-16T14:10:40.980939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124331+0000","last_change":"2023-10-16T14:10:40.990967+0000","last_active":"2023-10-16T14:10:42.124331+0000","last_peered":"2023-10-16T14:10:42.124331+0000","last_clean":"2023-10-16T14:10:42.124331+0000","last_became_active":"2023-10-16T14:10:40.989709+0000","last_became_peered":"2023-10-16T14:10:40.989709+0000","last_unstale":"2023-10-16T14:10:42.124331+0000","last_undegraded":"2023-10-16T14:10:42.124331+0000","last_fullsized":"2023-10-16T14:10:42.124331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.888217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974374+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:40.974374+0000","last_peered":"2023-10-16T14:10:40.974374+0000","last_clean":"2023-10-16T14:10:40.974374+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:40.974374+0000","last_undegraded":"2023-10-16T14:10:40.974374+0000","last_fullsized":"2023-10-16T14:10:40.974374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244140+0000","last_change":"2023-10-16T14:10:40.985097+0000","last_active":"2023-10-16T14:10:43.244140+0000","last_peered":"2023-10-16T14:10:43.244140+0000","last_clean":"2023-10-16T14:10:43.244140+0000","last_became_active":"2023-10-16T14:10:40.984409+0000","last_became_peered":"2023-10-16T14:10:40.984409+0000","last_unstale":"2023-10-16T14:10:43.244140+0000","last_undegraded":"2023-10-16T14:10:43.244140+0000","last_fullsized":"2023-10-16T14:10:43.244140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972580+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:40.972580+0000","last_peered":"2023-10-16T14:10:40.972580+0000","last_clean":"2023-10-16T14:10:40.972580+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:40.972580+0000","last_undegraded":"2023-10-16T14:10:40.972580+0000","last_fullsized":"2023-10-16T14:10:40.972580+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996436+0000","last_change":"2023-10-16T14:10:40.988348+0000","last_active":"2023-10-16T14:10:41.996436+0000","last_peered":"2023-10-16T14:10:41.996436+0000","last_clean":"2023-10-16T14:10:41.996436+0000","last_became_active":"2023-10-16T14:10:40.988060+0000","last_became_peered":"2023-10-16T14:10:40.988060+0000","last_unstale":"2023-10-16T14:10:41.996436+0000","last_undegraded":"2023-10-16T14:10:41.996436+0000","last_fullsized":"2023-10-16T14:10:41.996436+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.988939+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:40.988939+0000","last_peered":"2023-10-16T14:10:40.988939+0000","last_clean":"2023-10-16T14:10:40.988939+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:40.988939+0000","last_undegraded":"2023-10-16T14:10:40.988939+0000","last_fullsized":"2023-10-16T14:10:40.988939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243319+0000","last_change":"2023-10-16T14:10:40.987085+0000","last_active":"2023-10-16T14:10:43.243319+0000","last_peered":"2023-10-16T14:10:43.243319+0000","last_clean":"2023-10-16T14:10:43.243319+0000","last_became_active":"2023-10-16T14:10:40.986831+0000","last_became_peered":"2023-10-16T14:10:40.986831+0000","last_unstale":"2023-10-16T14:10:43.243319+0000","last_undegraded":"2023-10-16T14:10:43.243319+0000","last_fullsized":"2023-10-16T14:10:43.243319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:58.536370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987030+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:40.987030+0000","last_peered":"2023-10-16T14:10:40.987030+0000","last_clean":"2023-10-16T14:10:40.987030+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:40.987030+0000","last_undegraded":"2023-10-16T14:10:40.987030+0000","last_fullsized":"2023-10-16T14:10:40.987030+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125634+0000","last_change":"2023-10-16T14:10:40.998279+0000","last_active":"2023-10-16T14:10:42.125634+0000","last_peered":"2023-10-16T14:10:42.125634+0000","last_clean":"2023-10-16T14:10:42.125634+0000","last_became_active":"2023-10-16T14:10:40.997899+0000","last_became_peered":"2023-10-16T14:10:40.997899+0000","last_unstale":"2023-10-16T14:10:42.125634+0000","last_undegraded":"2023-10-16T14:10:42.125634+0000","last_fullsized":"2023-10-16T14:10:42.125634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:02.766931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977306+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:40.977306+0000","last_peered":"2023-10-16T14:10:40.977306+0000","last_clean":"2023-10-16T14:10:40.977306+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:40.977306+0000","last_undegraded":"2023-10-16T14:10:40.977306+0000","last_fullsized":"2023-10-16T14:10:40.977306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.991518+0000","last_change":"2023-10-16T14:10:40.992264+0000","last_active":"2023-10-16T14:10:41.991518+0000","last_peered":"2023-10-16T14:10:41.991518+0000","last_clean":"2023-10-16T14:10:41.991518+0000","last_became_active":"2023-10-16T14:10:40.991698+0000","last_became_peered":"2023-10-16T14:10:40.991698+0000","last_unstale":"2023-10-16T14:10:41.991518+0000","last_undegraded":"2023-10-16T14:10:41.991518+0000","last_fullsized":"2023-10-16T14:10:41.991518+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:16.407382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978952+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:40.978952+0000","last_peered":"2023-10-16T14:10:40.978952+0000","last_clean":"2023-10-16T14:10:40.978952+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:40.978952+0000","last_undegraded":"2023-10-16T14:10:40.978952+0000","last_fullsized":"2023-10-16T14:10:40.978952+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125182+0000","last_change":"2023-10-16T14:10:40.991982+0000","last_active":"2023-10-16T14:10:42.125182+0000","last_peered":"2023-10-16T14:10:42.125182+0000","last_clean":"2023-10-16T14:10:42.125182+0000","last_became_active":"2023-10-16T14:10:40.991521+0000","last_became_peered":"2023-10-16T14:10:40.991521+0000","last_unstale":"2023-10-16T14:10:42.125182+0000","last_undegraded":"2023-10-16T14:10:42.125182+0000","last_fullsized":"2023-10-16T14:10:42.125182+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:01.654954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961133+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:40.961133+0000","last_peered":"2023-10-16T14:10:40.961133+0000","last_clean":"2023-10-16T14:10:40.961133+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:40.961133+0000","last_undegraded":"2023-10-16T14:10:40.961133+0000","last_fullsized":"2023-10-16T14:10:40.961133+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124085+0000","last_change":"2023-10-16T14:10:40.992568+0000","last_active":"2023-10-16T14:10:42.124085+0000","last_peered":"2023-10-16T14:10:42.124085+0000","last_clean":"2023-10-16T14:10:42.124085+0000","last_became_active":"2023-10-16T14:10:40.992252+0000","last_became_peered":"2023-10-16T14:10:40.992252+0000","last_unstale":"2023-10-16T14:10:42.124085+0000","last_undegraded":"2023-10-16T14:10:42.124085+0000","last_fullsized":"2023-10-16T14:10:42.124085+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:25.262205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965644+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:40.965644+0000","last_peered":"2023-10-16T14:10:40.965644+0000","last_clean":"2023-10-16T14:10:40.965644+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:40.965644+0000","last_undegraded":"2023-10-16T14:10:40.965644+0000","last_fullsized":"2023-10-16T14:10:40.965644+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242786+0000","last_change":"2023-10-16T14:10:40.991785+0000","last_active":"2023-10-16T14:10:43.242786+0000","last_peered":"2023-10-16T14:10:43.242786+0000","last_clean":"2023-10-16T14:10:43.242786+0000","last_became_active":"2023-10-16T14:10:40.991292+0000","last_became_peered":"2023-10-16T14:10:40.991292+0000","last_unstale":"2023-10-16T14:10:43.242786+0000","last_undegraded":"2023-10-16T14:10:43.242786+0000","last_fullsized":"2023-10-16T14:10:43.242786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978297+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:40.978297+0000","last_peered":"2023-10-16T14:10:40.978297+0000","last_clean":"2023-10-16T14:10:40.978297+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:40.978297+0000","last_undegraded":"2023-10-16T14:10:40.978297+0000","last_fullsized":"2023-10-16T14:10:40.978297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243833+0000","last_change":"2023-10-16T14:10:40.985314+0000","last_active":"2023-10-16T14:10:43.243833+0000","last_peered":"2023-10-16T14:10:43.243833+0000","last_clean":"2023-10-16T14:10:43.243833+0000","last_became_active":"2023-10-16T14:10:40.984808+0000","last_became_peered":"2023-10-16T14:10:40.984808+0000","last_unstale":"2023-10-16T14:10:43.243833+0000","last_undegraded":"2023-10-16T14:10:43.243833+0000","last_fullsized":"2023-10-16T14:10:43.243833+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959800+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:40.959800+0000","last_peered":"2023-10-16T14:10:40.959800+0000","last_clean":"2023-10-16T14:10:40.959800+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:40.959800+0000","last_undegraded":"2023-10-16T14:10:40.959800+0000","last_fullsized":"2023-10-16T14:10:40.959800+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144022+0000","last_change":"2023-10-16T14:10:40.995986+0000","last_active":"2023-10-16T14:10:43.144022+0000","last_peered":"2023-10-16T14:10:43.144022+0000","last_clean":"2023-10-16T14:10:43.144022+0000","last_became_active":"2023-10-16T14:10:40.994817+0000","last_became_peered":"2023-10-16T14:10:40.994817+0000","last_unstale":"2023-10-16T14:10:43.144022+0000","last_undegraded":"2023-10-16T14:10:43.144022+0000","last_fullsized":"2023-10-16T14:10:43.144022+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:17.570998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978097+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:40.978097+0000","last_peered":"2023-10-16T14:10:40.978097+0000","last_clean":"2023-10-16T14:10:40.978097+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:40.978097+0000","last_undegraded":"2023-10-16T14:10:40.978097+0000","last_fullsized":"2023-10-16T14:10:40.978097+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997726+0000","last_change":"2023-10-16T14:10:41.001565+0000","last_active":"2023-10-16T14:10:41.997726+0000","last_peered":"2023-10-16T14:10:41.997726+0000","last_clean":"2023-10-16T14:10:41.997726+0000","last_became_active":"2023-10-16T14:10:40.999604+0000","last_became_peered":"2023-10-16T14:10:40.999604+0000","last_unstale":"2023-10-16T14:10:41.997726+0000","last_undegraded":"2023-10-16T14:10:41.997726+0000","last_fullsized":"2023-10-16T14:10:41.997726+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980317+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:40.980317+0000","last_peered":"2023-10-16T14:10:40.980317+0000","last_clean":"2023-10-16T14:10:40.980317+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:40.980317+0000","last_undegraded":"2023-10-16T14:10:40.980317+0000","last_fullsized":"2023-10-16T14:10:40.980317+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243931+0000","last_change":"2023-10-16T14:10:40.994995+0000","last_active":"2023-10-16T14:10:43.243931+0000","last_peered":"2023-10-16T14:10:43.243931+0000","last_clean":"2023-10-16T14:10:43.243931+0000","last_became_active":"2023-10-16T14:10:40.994421+0000","last_became_peered":"2023-10-16T14:10:40.994421+0000","last_unstale":"2023-10-16T14:10:43.243931+0000","last_undegraded":"2023-10-16T14:10:43.243931+0000","last_fullsized":"2023-10-16T14:10:43.243931+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:39.108794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968652+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:40.968652+0000","last_peered":"2023-10-16T14:10:40.968652+0000","last_clean":"2023-10-16T14:10:40.968652+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:40.968652+0000","last_undegraded":"2023-10-16T14:10:40.968652+0000","last_fullsized":"2023-10-16T14:10:40.968652+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995533+0000","last_change":"2023-10-16T14:10:40.988474+0000","last_active":"2023-10-16T14:10:41.995533+0000","last_peered":"2023-10-16T14:10:41.995533+0000","last_clean":"2023-10-16T14:10:41.995533+0000","last_became_active":"2023-10-16T14:10:40.988193+0000","last_became_peered":"2023-10-16T14:10:40.988193+0000","last_unstale":"2023-10-16T14:10:41.995533+0000","last_undegraded":"2023-10-16T14:10:41.995533+0000","last_fullsized":"2023-10-16T14:10:41.995533+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972255+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:40.972255+0000","last_peered":"2023-10-16T14:10:40.972255+0000","last_clean":"2023-10-16T14:10:40.972255+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:40.972255+0000","last_undegraded":"2023-10-16T14:10:40.972255+0000","last_fullsized":"2023-10-16T14:10:40.972255+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994820+0000","last_change":"2023-10-16T14:10:40.994079+0000","last_active":"2023-10-16T14:10:41.994820+0000","last_peered":"2023-10-16T14:10:41.994820+0000","last_clean":"2023-10-16T14:10:41.994820+0000","last_became_active":"2023-10-16T14:10:40.991884+0000","last_became_peered":"2023-10-16T14:10:40.991884+0000","last_unstale":"2023-10-16T14:10:41.994820+0000","last_undegraded":"2023-10-16T14:10:41.994820+0000","last_fullsized":"2023-10-16T14:10:41.994820+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:14.260538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981697+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:40.981697+0000","last_peered":"2023-10-16T14:10:40.981697+0000","last_clean":"2023-10-16T14:10:40.981697+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:40.981697+0000","last_undegraded":"2023-10-16T14:10:40.981697+0000","last_fullsized":"2023-10-16T14:10:40.981697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.993382+0000","last_change":"2023-10-16T14:10:41.002135+0000","last_active":"2023-10-16T14:10:41.993382+0000","last_peered":"2023-10-16T14:10:41.993382+0000","last_clean":"2023-10-16T14:10:41.993382+0000","last_became_active":"2023-10-16T14:10:41.000234+0000","last_became_peered":"2023-10-16T14:10:41.000234+0000","last_unstale":"2023-10-16T14:10:41.993382+0000","last_undegraded":"2023-10-16T14:10:41.993382+0000","last_fullsized":"2023-10-16T14:10:41.993382+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:09.588728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965582+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:40.965582+0000","last_peered":"2023-10-16T14:10:40.965582+0000","last_clean":"2023-10-16T14:10:40.965582+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:40.965582+0000","last_undegraded":"2023-10-16T14:10:40.965582+0000","last_fullsized":"2023-10-16T14:10:40.965582+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989881+0000","last_change":"2023-10-16T14:10:40.987206+0000","last_active":"2023-10-16T14:10:41.989881+0000","last_peered":"2023-10-16T14:10:41.989881+0000","last_clean":"2023-10-16T14:10:41.989881+0000","last_became_active":"2023-10-16T14:10:40.986211+0000","last_became_peered":"2023-10-16T14:10:40.986211+0000","last_unstale":"2023-10-16T14:10:41.989881+0000","last_undegraded":"2023-10-16T14:10:41.989881+0000","last_fullsized":"2023-10-16T14:10:41.989881+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974724+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:40.974724+0000","last_peered":"2023-10-16T14:10:40.974724+0000","last_clean":"2023-10-16T14:10:40.974724+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:40.974724+0000","last_undegraded":"2023-10-16T14:10:40.974724+0000","last_fullsized":"2023-10-16T14:10:40.974724+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996483+0000","last_change":"2023-10-16T14:10:40.998615+0000","last_active":"2023-10-16T14:10:41.996483+0000","last_peered":"2023-10-16T14:10:41.996483+0000","last_clean":"2023-10-16T14:10:41.996483+0000","last_became_active":"2023-10-16T14:10:40.997677+0000","last_became_peered":"2023-10-16T14:10:40.997677+0000","last_unstale":"2023-10-16T14:10:41.996483+0000","last_undegraded":"2023-10-16T14:10:41.996483+0000","last_fullsized":"2023-10-16T14:10:41.996483+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968589+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:40.968589+0000","last_peered":"2023-10-16T14:10:40.968589+0000","last_clean":"2023-10-16T14:10:40.968589+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:40.968589+0000","last_undegraded":"2023-10-16T14:10:40.968589+0000","last_fullsized":"2023-10-16T14:10:40.968589+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244429+0000","last_change":"2023-10-16T14:10:41.001901+0000","last_active":"2023-10-16T14:10:43.244429+0000","last_peered":"2023-10-16T14:10:43.244429+0000","last_clean":"2023-10-16T14:10:43.244429+0000","last_became_active":"2023-10-16T14:10:41.001390+0000","last_became_peered":"2023-10-16T14:10:41.001390+0000","last_unstale":"2023-10-16T14:10:43.244429+0000","last_undegraded":"2023-10-16T14:10:43.244429+0000","last_fullsized":"2023-10-16T14:10:43.244429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:38.275860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977891+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:40.977891+0000","last_peered":"2023-10-16T14:10:40.977891+0000","last_clean":"2023-10-16T14:10:40.977891+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:40.977891+0000","last_undegraded":"2023-10-16T14:10:40.977891+0000","last_fullsized":"2023-10-16T14:10:40.977891+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124855+0000","last_change":"2023-10-16T14:10:40.990936+0000","last_active":"2023-10-16T14:10:42.124855+0000","last_peered":"2023-10-16T14:10:42.124855+0000","last_clean":"2023-10-16T14:10:42.124855+0000","last_became_active":"2023-10-16T14:10:40.990264+0000","last_became_peered":"2023-10-16T14:10:40.990264+0000","last_unstale":"2023-10-16T14:10:42.124855+0000","last_undegraded":"2023-10-16T14:10:42.124855+0000","last_fullsized":"2023-10-16T14:10:42.124855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:39.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973916+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:40.973916+0000","last_peered":"2023-10-16T14:10:40.973916+0000","last_clean":"2023-10-16T14:10:40.973916+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:40.973916+0000","last_undegraded":"2023-10-16T14:10:40.973916+0000","last_fullsized":"2023-10-16T14:10:40.973916+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143331+0000","last_change":"2023-10-16T14:10:40.998298+0000","last_active":"2023-10-16T14:10:43.143331+0000","last_peered":"2023-10-16T14:10:43.143331+0000","last_clean":"2023-10-16T14:10:43.143331+0000","last_became_active":"2023-10-16T14:10:40.997963+0000","last_became_peered":"2023-10-16T14:10:40.997963+0000","last_unstale":"2023-10-16T14:10:43.143331+0000","last_undegraded":"2023-10-16T14:10:43.143331+0000","last_fullsized":"2023-10-16T14:10:43.143331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:12.160466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973460+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:40.973460+0000","last_peered":"2023-10-16T14:10:40.973460+0000","last_clean":"2023-10-16T14:10:40.973460+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:40.973460+0000","last_undegraded":"2023-10-16T14:10:40.973460+0000","last_fullsized":"2023-10-16T14:10:40.973460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.241703+0000","last_change":"2023-10-16T14:10:40.982851+0000","last_active":"2023-10-16T14:10:43.241703+0000","last_peered":"2023-10-16T14:10:43.241703+0000","last_clean":"2023-10-16T14:10:43.241703+0000","last_became_active":"2023-10-16T14:10:40.981261+0000","last_became_peered":"2023-10-16T14:10:40.981261+0000","last_unstale":"2023-10-16T14:10:43.241703+0000","last_undegraded":"2023-10-16T14:10:43.241703+0000","last_fullsized":"2023-10-16T14:10:43.241703+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:03.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982369+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:40.982369+0000","last_peered":"2023-10-16T14:10:40.982369+0000","last_clean":"2023-10-16T14:10:40.982369+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:40.982369+0000","last_undegraded":"2023-10-16T14:10:40.982369+0000","last_fullsized":"2023-10-16T14:10:40.982369+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125338+0000","last_change":"2023-10-16T14:10:40.991807+0000","last_active":"2023-10-16T14:10:42.125338+0000","last_peered":"2023-10-16T14:10:42.125338+0000","last_clean":"2023-10-16T14:10:42.125338+0000","last_became_active":"2023-10-16T14:10:40.991143+0000","last_became_peered":"2023-10-16T14:10:40.991143+0000","last_unstale":"2023-10-16T14:10:42.125338+0000","last_undegraded":"2023-10-16T14:10:42.125338+0000","last_fullsized":"2023-10-16T14:10:42.125338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.405248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981437+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:40.981437+0000","last_peered":"2023-10-16T14:10:40.981437+0000","last_clean":"2023-10-16T14:10:40.981437+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:40.981437+0000","last_undegraded":"2023-10-16T14:10:40.981437+0000","last_fullsized":"2023-10-16T14:10:40.981437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185397+0000","last_change":"2023-10-16T14:10:40.997929+0000","last_active":"2023-10-16T14:10:43.185397+0000","last_peered":"2023-10-16T14:10:43.185397+0000","last_clean":"2023-10-16T14:10:43.185397+0000","last_became_active":"2023-10-16T14:10:40.997578+0000","last_became_peered":"2023-10-16T14:10:40.997578+0000","last_unstale":"2023-10-16T14:10:43.185397+0000","last_undegraded":"2023-10-16T14:10:43.185397+0000","last_fullsized":"2023-10-16T14:10:43.185397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:31.595742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968740+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:40.968740+0000","last_peered":"2023-10-16T14:10:40.968740+0000","last_clean":"2023-10-16T14:10:40.968740+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:40.968740+0000","last_undegraded":"2023-10-16T14:10:40.968740+0000","last_fullsized":"2023-10-16T14:10:40.968740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.141865+0000","last_change":"2023-10-16T14:10:40.982299+0000","last_active":"2023-10-16T14:10:43.141865+0000","last_peered":"2023-10-16T14:10:43.141865+0000","last_clean":"2023-10-16T14:10:43.141865+0000","last_became_active":"2023-10-16T14:10:40.982047+0000","last_became_peered":"2023-10-16T14:10:40.982047+0000","last_unstale":"2023-10-16T14:10:43.141865+0000","last_undegraded":"2023-10-16T14:10:43.141865+0000","last_fullsized":"2023-10-16T14:10:43.141865+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:09.508655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978187+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:40.978187+0000","last_peered":"2023-10-16T14:10:40.978187+0000","last_clean":"2023-10-16T14:10:40.978187+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:40.978187+0000","last_undegraded":"2023-10-16T14:10:40.978187+0000","last_fullsized":"2023-10-16T14:10:40.978187+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543301,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3551232,"data_stored":2031342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":1.4139999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.589}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.4730000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.327}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.087},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.3779999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.228},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.5329999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543302,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42388,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329452,"statfs":{"total":96636764160,"available":96593358848,"internally_reserved":0,"allocated":2953216,"data_stored":1438685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":1.788},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":0.86299999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.042},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.6779999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.3879999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.089}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":2.2749999999999999},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":2.0990000000000002}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.45}]}]},{"osd":5,"up_from":14,"seq":60129543303,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":2945024,"data_stored":1438629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.02}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.1799999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.254}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.286},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.8080000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.9590000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.4590000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47968,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323872,"statfs":{"total":96636764160,"available":96587644928,"internally_reserved":0,"allocated":2965504,"data_stored":1441590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.8420000000000001},"last":1.306},{"interface":"front","average":{"1min":1.121,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6699999999999999,"5min":1.8160000000000001,"15min":1.9019999999999999},"last":1.3700000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.78},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2549999999999999,"15min":1.294},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.754,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.5489999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.286,"15min":1.3169999999999999},"min":{"1min":0.80200000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.714,"5min":1.714,"15min":1.865},"last":1.026},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.367,"15min":1.387},"min":{"1min":1.026,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.476}]},{"osd":3,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.296,"15min":1.3120000000000001},"min":{"1min":0.94599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.615,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.607,"5min":1.4590000000000001,"15min":1.4299999999999999},"min":{"1min":0.96499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":5.2350000000000003,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.427}]},{"osd":5,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.329,"15min":1.3400000000000001},"min":{"1min":0.99199999999999999,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.615},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3580000000000001,"15min":1.3360000000000001},"min":{"1min":1.0389999999999999,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":1.921,"15min":1.921},"last":1.0740000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3919999999999999,"15min":1.3879999999999999},"min":{"1min":1.004,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.7689999999999999,"5min":1.9159999999999999,"15min":1.956},"last":1.149},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.502,"15min":1.5229999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.843}]},{"osd":7,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5349999999999999,"15min":1.5049999999999999},"min":{"1min":1.137,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.1949999999999998},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6379999999999999,"15min":1.6319999999999999},"min":{"1min":1.391,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":1.889,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.575}]}]},{"osd":3,"up_from":14,"seq":60129543303,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52128,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319712,"statfs":{"total":96636764160,"available":96583385088,"internally_reserved":0,"allocated":2969600,"data_stored":1439911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":1.05}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":4.1299999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":3.8340000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":0.89300000000000002}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.01}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":2.2799999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":2.569}]}]},{"osd":2,"up_from":14,"seq":60129543301,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":2949120,"data_stored":1432548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":1.337},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.115},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":1.4159999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.268},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.0469999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.103},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":1.397}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.1899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543301,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2953216,"data_stored":1438193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.0740000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":0.752},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.1870000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":1.532}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":1.454},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":2.0979999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.867},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.7969999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543302,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":3547136,"data_stored":2029177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.93300000000000005,"15min":0.93999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.218,"5min":1.704,"15min":1.891},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.1579999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48699999999999999},"max":{"1min":1.627,"5min":1.659,"15min":1.9299999999999999},"last":0.90700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.1499999999999999,"15min":1.171},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5349999999999999,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.1899999999999999,"15min":1.226},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.6539999999999999,"5min":1.893,"15min":1.893},"last":1.0760000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1639999999999999,"15min":1.163},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7390000000000001,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3380000000000001,"15min":1.3759999999999999},"min":{"1min":0.64700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.679,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.2969999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3340000000000001,"15min":1.3799999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6140000000000001,"5min":1.819,"15min":2.048},"last":1.002},{"interface":"front","average":{"1min":1.379,"5min":1.518,"15min":1.524},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.0009999999999999,"5min":2.601,"15min":2.601},"last":1.4850000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.48,"15min":1.5069999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.821,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.452,"15min":1.492},"min":{"1min":0.98199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0169999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.593,"15min":1.621},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9490000000000001,"5min":2.669,"15min":2.669},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.55,"15min":1.5980000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2589999999999999,"5min":2.355,"15min":2.3769999999999998},"last":1.778}]},{"osd":7,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.655,"15min":1.712},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1549999999999998,"5min":2.484,"15min":2.5270000000000001},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.625,"15min":1.649},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.464},"last":1.056}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:46.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-16T14:10:46.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.915+0000 7fa9d29d2640 1 Processor -- start 2023-10-16T14:10:46.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.915+0000 7fa9d29d2640 1 -- start start 2023-10-16T14:10:46.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d29d2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9cc159a40 0x7fa9cc15e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d29d2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa9cc06c980 0x7fa9cc0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d29d2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06ce10 0x7fa9cc0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d29d2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9cc06c4f0 con 0x7fa9cc159a40 2023-10-16T14:10:46.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d29d2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9cc15e5a0 con 0x7fa9cc06c980 2023-10-16T14:10:46.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d29d2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc06b8e0 con 0x7fa9cc06ce10 2023-10-16T14:10:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d19d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9cc159a40 0x7fa9cc15e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d19d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9cc159a40 0x7fa9cc15e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38108/0 (socket says 172.21.15.60:38108) 2023-10-16T14:10:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.917+0000 7fa9d19d0640 1 -- 172.21.15.60:0/1595269069 learned_addr learned my addr 172.21.15.60:0/1595269069 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:46.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d11cf640 1 --2- 172.21.15.60:0/1595269069 >> v2:172.21.15.144:3300/0 conn(0x7fa9cc06c980 0x7fa9cc0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d21d1640 1 --2- 172.21.15.60:0/1595269069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06ce10 0x7fa9cc0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d19d0640 1 -- 172.21.15.60:0/1595269069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06ce10 msgr2=0x7fa9cc0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d19d0640 1 --2- 172.21.15.60:0/1595269069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06ce10 0x7fa9cc0cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d19d0640 1 -- 172.21.15.60:0/1595269069 >> v2:172.21.15.144:3300/0 conn(0x7fa9cc06c980 msgr2=0x7fa9cc0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:46.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d19d0640 1 --2- 172.21.15.60:0/1595269069 >> v2:172.21.15.144:3300/0 conn(0x7fa9cc06c980 0x7fa9cc0cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.918+0000 7fa9d19d0640 1 -- 172.21.15.60:0/1595269069 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc06bbc0 con 0x7fa9cc159a40 2023-10-16T14:10:46.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.919+0000 7fa9d19d0640 1 --2- 172.21.15.60:0/1595269069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc159a40 0x7fa9cc15e060 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0cc060 tx=0x7fa9c802db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb186854b274c6bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:46.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.919+0000 7fa9d09ce640 1 -- 172.21.15.60:0/1595269069 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c802f8e0 con 0x7fa9cc159a40 2023-10-16T14:10:46.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.919+0000 7fa9d09ce640 1 -- 172.21.15.60:0/1595269069 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c802fa40 con 0x7fa9cc159a40 2023-10-16T14:10:46.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.919+0000 7fa9d09ce640 1 -- 172.21.15.60:0/1595269069 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c802fdd0 con 0x7fa9cc159a40 2023-10-16T14:10:46.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1595269069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc159a40 msgr2=0x7fa9cc15e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1595269069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc159a40 0x7fa9cc15e060 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0cc060 tx=0x7fa9c802db20 comp rx=0 tx=0).stop 2023-10-16T14:10:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1595269069 shutdown_connections 2023-10-16T14:10:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1595269069 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06ce10 0x7fa9cc0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1595269069 >> v2:172.21.15.144:3300/0 conn(0x7fa9cc06c980 0x7fa9cc0cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1595269069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc159a40 0x7fa9cc15e060 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1595269069 >> 172.21.15.60:0/1595269069 conn(0x7fa9cc0cb1a0 msgr2=0x7fa9cc0cb580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1595269069 shutdown_connections 2023-10-16T14:10:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.920+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1595269069 wait complete. 2023-10-16T14:10:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.922+0000 7fa9d29d2640 1 Processor -- start 2023-10-16T14:10:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.922+0000 7fa9d29d2640 1 -- start start 2023-10-16T14:10:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d29d2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06c980 0x7fa9cc157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d29d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 0x7fa9cc0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d29d2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9cc159a40 0x7fa9cc0c2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d29d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc158400 con 0x7fa9cc06ce10 2023-10-16T14:10:46.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d29d2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc0c37d0 con 0x7fa9cc159a40 2023-10-16T14:10:46.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d29d2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc0c2bc0 con 0x7fa9cc06c980 2023-10-16T14:10:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d11cf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 0x7fa9cc0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.923+0000 7fa9d11cf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 0x7fa9cc0b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38114/0 (socket says 172.21.15.60:38114) 2023-10-16T14:10:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 -- 172.21.15.60:0/1149839764 learned_addr learned my addr 172.21.15.60:0/1149839764 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d19d0640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06c980 0x7fa9cc157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d21d1640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9cc159a40 0x7fa9cc0c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 -- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06c980 msgr2=0x7fa9cc157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06c980 0x7fa9cc157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 -- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9cc159a40 msgr2=0x7fa9cc0c2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9cc159a40 0x7fa9cc0c2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 -- 172.21.15.60:0/1149839764 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c80094d0 con 0x7fa9cc06ce10 2023-10-16T14:10:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.924+0000 7fa9d11cf640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 0x7fa9cc0b1570 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00d800 tx=0x7fa9bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.925+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc00b6c0 con 0x7fa9cc06ce10 2023-10-16T14:10:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.925+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc00b820 con 0x7fa9cc06ce10 2023-10-16T14:10:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.925+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc00bbb0 con 0x7fa9cc06ce10 2023-10-16T14:10:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.925+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc0c2ea0 con 0x7fa9cc06ce10 2023-10-16T14:10:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.925+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc0c3460 con 0x7fa9cc06ce10 2023-10-16T14:10:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.926+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9cc04e160 con 0x7fa9cc06ce10 2023-10-16T14:10:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.927+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa9bc00bd10 con 0x7fa9cc06ce10 2023-10-16T14:10:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.928+0000 7fa9c27fc640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa9b006af90 0x7fa9b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.928+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fa9bc08a060 con 0x7fa9cc06ce10 2023-10-16T14:10:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.929+0000 7fa9d19d0640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa9b006af90 0x7fa9b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:46.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.930+0000 7fa9d19d0640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa9b006af90 0x7fa9b006d450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8038000 tx=0x7fa9c8008030 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:46.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:46.933+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa9bc0bcfe0 con 0x7fa9cc06ce10 2023-10-16T14:10:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.204+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa9cc0c2010 con 0x7fa9cc06ce10 2023-10-16T14:10:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.205+0000 7fa9c27fc640 1 -- 172.21.15.60:0/1149839764 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1125 (secure 0 0 0) 0x7fa9bc054c90 con 0x7fa9cc06ce10 2023-10-16T14:10:47.209 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:47.209 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5862,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":235,"num_osds":8,"num_up_osds":8,"osd_up_since":1697459596,"num_in_osds":8,"osd_in_since":1697459588,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":45,"data_bytes":596433,"bytes_used":386924544,"bytes_avail":772707188736,"bytes_total":773094113280,"write_bytes_sec":2300,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":118,"id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"c","status":"up:active","gid":10998},{"filesystem_id":13,"rank":0,"name":"b","status":"up:active","gid":11004}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2023-10-16T14:09:01.058067+0000","services":{}},"progress_events":{}} 2023-10-16T14:10:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa9b006af90 msgr2=0x7fa9b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa9b006af90 0x7fa9b006d450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8038000 tx=0x7fa9c8008030 comp rx=0 tx=0).stop 2023-10-16T14:10:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 msgr2=0x7fa9cc0b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 0x7fa9cc0b1570 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00d800 tx=0x7fa9bc00dcc0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 shutdown_connections 2023-10-16T14:10:47.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fa9b006af90 0x7fa9b006d450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9cc159a40 0x7fa9cc0c2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9cc06ce10 0x7fa9cc0b1570 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 --2- 172.21.15.60:0/1149839764 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9cc06c980 0x7fa9cc157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 >> 172.21.15.60:0/1149839764 conn(0x7fa9cc0cb1a0 msgr2=0x7fa9cc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 shutdown_connections 2023-10-16T14:10:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.208+0000 7fa9d29d2640 1 -- 172.21.15.60:0/1149839764 wait complete. 2023-10-16T14:10:47.232 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-16T14:10:47.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:47.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.444+0000 7fef394d1640 1 Processor -- start 2023-10-16T14:10:47.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.444+0000 7fef394d1640 1 -- start start 2023-10-16T14:10:47.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.445+0000 7fef394d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3406c040 0x7fef3406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.445+0000 7fef394d1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef340ad970 0x7fef3415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.445+0000 7fef394d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef3415ebc0 0x7fef3415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.445+0000 7fef394d1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3406c960 con 0x7fef340ad970 2023-10-16T14:10:47.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef394d1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3415b9d0 con 0x7fef3415ebc0 2023-10-16T14:10:47.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef394d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3415bcb0 con 0x7fef3406c040 2023-10-16T14:10:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef2bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef340ad970 0x7fef3415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef2bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef340ad970 0x7fef3415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38124/0 (socket says 172.21.15.60:38124) 2023-10-16T14:10:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef2bfff640 1 -- 172.21.15.60:0/1141071822 learned_addr learned my addr 172.21.15.60:0/1141071822 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef33fff640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3406c040 0x7fef3406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.446+0000 7fef38cd0640 1 --2- 172.21.15.60:0/1141071822 >> v2:172.21.15.144:3300/0 conn(0x7fef3415ebc0 0x7fef3415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef2bfff640 1 -- 172.21.15.60:0/1141071822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3406c040 msgr2=0x7fef3406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:47.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef2bfff640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3406c040 0x7fef3406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef2bfff640 1 -- 172.21.15.60:0/1141071822 >> v2:172.21.15.144:3300/0 conn(0x7fef3415ebc0 msgr2=0x7fef3415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:47.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef2bfff640 1 --2- 172.21.15.60:0/1141071822 >> v2:172.21.15.144:3300/0 conn(0x7fef3415ebc0 0x7fef3415f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef2bfff640 1 -- 172.21.15.60:0/1141071822 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3415bf90 con 0x7fef340ad970 2023-10-16T14:10:47.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef38cd0640 1 --2- 172.21.15.60:0/1141071822 >> v2:172.21.15.144:3300/0 conn(0x7fef3415ebc0 0x7fef3415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:47.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.447+0000 7fef33fff640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3406c040 0x7fef3406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:10:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.448+0000 7fef2bfff640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef3415e680 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009870 tx=0x7fef1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab648be7fececce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.448+0000 7fef337fe640 1 -- 172.21.15.60:0/1141071822 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c031640 con 0x7fef340ad970 2023-10-16T14:10:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.448+0000 7fef337fe640 1 -- 172.21.15.60:0/1141071822 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c0317a0 con 0x7fef340ad970 2023-10-16T14:10:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.448+0000 7fef337fe640 1 -- 172.21.15.60:0/1141071822 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c031b30 con 0x7fef340ad970 2023-10-16T14:10:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 -- 172.21.15.60:0/1141071822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 msgr2=0x7fef3415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef3415e680 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009870 tx=0x7fef1c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 -- 172.21.15.60:0/1141071822 shutdown_connections 2023-10-16T14:10:47.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 --2- 172.21.15.60:0/1141071822 >> v2:172.21.15.144:3300/0 conn(0x7fef3415ebc0 0x7fef3415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef3415e680 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 --2- 172.21.15.60:0/1141071822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3406c040 0x7fef3406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 -- 172.21.15.60:0/1141071822 >> 172.21.15.60:0/1141071822 conn(0x7fef340ce6a0 msgr2=0x7fef340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:47.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 -- 172.21.15.60:0/1141071822 shutdown_connections 2023-10-16T14:10:47.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.449+0000 7fef394d1640 1 -- 172.21.15.60:0/1141071822 wait complete. 2023-10-16T14:10:47.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.452+0000 7fef394d1640 1 Processor -- start 2023-10-16T14:10:47.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.452+0000 7fef394d1640 1 -- start start 2023-10-16T14:10:47.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef394d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef3406c040 0x7fef340c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef394d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef340ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef394d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3415ebc0 0x7fef340acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef394d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341f2a90 con 0x7fef340ad970 2023-10-16T14:10:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef394d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef34156bf0 con 0x7fef3406c040 2023-10-16T14:10:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef394d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef341f2d10 con 0x7fef3415ebc0 2023-10-16T14:10:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.453+0000 7fef2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef340ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef340ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38130/0 (socket says 172.21.15.60:38130) 2023-10-16T14:10:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 -- 172.21.15.60:0/1953213054 learned_addr learned my addr 172.21.15.60:0/1953213054 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef33fff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef3406c040 0x7fef340c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef38cd0640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3415ebc0 0x7fef340acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 -- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3415ebc0 msgr2=0x7fef340acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3415ebc0 0x7fef340acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 -- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef3406c040 msgr2=0x7fef340c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef3406c040 0x7fef340c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 -- 172.21.15.60:0/1953213054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0094d0 con 0x7fef340ad970 2023-10-16T14:10:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef33fff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef3406c040 0x7fef340c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.454+0000 7fef2bfff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef340ac890 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000c00 tx=0x7fef1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.455+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c03ac60 con 0x7fef340ad970 2023-10-16T14:10:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.455+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c03adc0 con 0x7fef340ad970 2023-10-16T14:10:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.455+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c044630 con 0x7fef340ad970 2023-10-16T14:10:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.455+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef341f2ff0 con 0x7fef340ad970 2023-10-16T14:10:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.455+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef34159a40 con 0x7fef340ad970 2023-10-16T14:10:47.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.456+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3406b330 con 0x7fef340ad970 2023-10-16T14:10:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.457+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef1c02f050 con 0x7fef340ad970 2023-10-16T14:10:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.458+0000 7fef317fa640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fef1406af90 0x7fef1406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.458+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fef1c0abac0 con 0x7fef340ad970 2023-10-16T14:10:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.458+0000 7fef33fff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fef1406af90 0x7fef1406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.460+0000 7fef33fff640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fef1406af90 0x7fef1406d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fef24006fd0 tx=0x7fef24008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.462+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef1c0765e0 con 0x7fef340ad970 2023-10-16T14:10:47.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.649+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef340cc550 con 0x7fef1406af90 2023-10-16T14:10:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.688+0000 7fef317fa640 1 -- 172.21.15.60:0/1953213054 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526047 (secure 0 0 0) 0x7fef2400e510 con 0x7fef1406af90 2023-10-16T14:10:47.695 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:47.703 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fef1406af90 msgr2=0x7fef1406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fef1406af90 0x7fef1406d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fef24006fd0 tx=0x7fef24008040 comp rx=0 tx=0).stop 2023-10-16T14:10:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 msgr2=0x7fef340ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef340ac890 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000c00 tx=0x7fef1c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 shutdown_connections 2023-10-16T14:10:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fef1406af90 0x7fef1406d450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fef3415ebc0 0x7fef340acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef340ad970 0x7fef340ac890 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 --2- 172.21.15.60:0/1953213054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef3406c040 0x7fef340c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:47.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 >> 172.21.15.60:0/1953213054 conn(0x7fef340ce6a0 msgr2=0x7fef34159da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 shutdown_connections 2023-10-16T14:10:47.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:47.699+0000 7fef394d1640 1 -- 172.21.15.60:0/1953213054 wait complete. 2023-10-16T14:10:47.727 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-16T14:10:46.184131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":674,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":356,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377856,"kb_used_data":24252,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754596864,"statfs":{"total":773094113280,"available":772707188736,"internally_reserved":0,"allocated":24834048,"data_stored":12690075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.232069"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244206+0000","last_change":"2023-10-16T14:10:40.992171+0000","last_active":"2023-10-16T14:10:43.244206+0000","last_peered":"2023-10-16T14:10:43.244206+0000","last_clean":"2023-10-16T14:10:43.244206+0000","last_became_active":"2023-10-16T14:10:40.991865+0000","last_became_peered":"2023-10-16T14:10:40.991865+0000","last_unstale":"2023-10-16T14:10:43.244206+0000","last_undegraded":"2023-10-16T14:10:43.244206+0000","last_fullsized":"2023-10-16T14:10:43.244206+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:36.935225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1226,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974179+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:40.974179+0000","last_peered":"2023-10-16T14:10:40.974179+0000","last_clean":"2023-10-16T14:10:40.974179+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:40.974179+0000","last_undegraded":"2023-10-16T14:10:40.974179+0000","last_fullsized":"2023-10-16T14:10:40.974179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968913+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:40.968913+0000","last_peered":"2023-10-16T14:10:40.968913+0000","last_clean":"2023-10-16T14:10:40.968913+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:40.968913+0000","last_undegraded":"2023-10-16T14:10:40.968913+0000","last_fullsized":"2023-10-16T14:10:40.968913+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980445+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:40.980445+0000","last_peered":"2023-10-16T14:10:40.980445+0000","last_clean":"2023-10-16T14:10:40.980445+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:40.980445+0000","last_undegraded":"2023-10-16T14:10:40.980445+0000","last_fullsized":"2023-10-16T14:10:40.980445+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008653+0000","last_change":"2023-10-16T14:10:39.986399+0000","last_active":"2023-10-16T14:10:41.008653+0000","last_peered":"2023-10-16T14:10:41.008653+0000","last_clean":"2023-10-16T14:10:41.008653+0000","last_became_active":"2023-10-16T14:10:39.986098+0000","last_became_peered":"2023-10-16T14:10:39.986098+0000","last_unstale":"2023-10-16T14:10:41.008653+0000","last_undegraded":"2023-10-16T14:10:41.008653+0000","last_fullsized":"2023-10-16T14:10:41.008653+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:15.423316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144154+0000","last_change":"2023-10-16T14:10:40.991925+0000","last_active":"2023-10-16T14:10:43.144154+0000","last_peered":"2023-10-16T14:10:43.144154+0000","last_clean":"2023-10-16T14:10:43.144154+0000","last_became_active":"2023-10-16T14:10:40.991667+0000","last_became_peered":"2023-10-16T14:10:40.991667+0000","last_unstale":"2023-10-16T14:10:43.144154+0000","last_undegraded":"2023-10-16T14:10:43.144154+0000","last_fullsized":"2023-10-16T14:10:43.144154+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:25.199441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977915+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:40.977915+0000","last_peered":"2023-10-16T14:10:40.977915+0000","last_clean":"2023-10-16T14:10:40.977915+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:40.977915+0000","last_undegraded":"2023-10-16T14:10:40.977915+0000","last_fullsized":"2023-10-16T14:10:40.977915+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970555+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:40.970555+0000","last_peered":"2023-10-16T14:10:40.970555+0000","last_clean":"2023-10-16T14:10:40.970555+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:40.970555+0000","last_undegraded":"2023-10-16T14:10:40.970555+0000","last_fullsized":"2023-10-16T14:10:40.970555+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979944+0000","last_change":"2023-10-16T14:10:39.988566+0000","last_active":"2023-10-16T14:10:40.979944+0000","last_peered":"2023-10-16T14:10:40.979944+0000","last_clean":"2023-10-16T14:10:40.979944+0000","last_became_active":"2023-10-16T14:10:39.988037+0000","last_became_peered":"2023-10-16T14:10:39.988037+0000","last_unstale":"2023-10-16T14:10:40.979944+0000","last_undegraded":"2023-10-16T14:10:40.979944+0000","last_fullsized":"2023-10-16T14:10:40.979944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.694200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125491+0000","last_change":"2023-10-16T14:10:40.987050+0000","last_active":"2023-10-16T14:10:42.125491+0000","last_peered":"2023-10-16T14:10:42.125491+0000","last_clean":"2023-10-16T14:10:42.125491+0000","last_became_active":"2023-10-16T14:10:40.986733+0000","last_became_peered":"2023-10-16T14:10:40.986733+0000","last_unstale":"2023-10-16T14:10:42.125491+0000","last_undegraded":"2023-10-16T14:10:42.125491+0000","last_fullsized":"2023-10-16T14:10:42.125491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:53.222315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.989242+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:40.989242+0000","last_peered":"2023-10-16T14:10:40.989242+0000","last_clean":"2023-10-16T14:10:40.989242+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:40.989242+0000","last_undegraded":"2023-10-16T14:10:40.989242+0000","last_fullsized":"2023-10-16T14:10:40.989242+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975719+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:40.975719+0000","last_peered":"2023-10-16T14:10:40.975719+0000","last_clean":"2023-10-16T14:10:40.975719+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:40.975719+0000","last_undegraded":"2023-10-16T14:10:40.975719+0000","last_fullsized":"2023-10-16T14:10:40.975719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008672+0000","last_change":"2023-10-16T14:10:39.985490+0000","last_active":"2023-10-16T14:10:41.008672+0000","last_peered":"2023-10-16T14:10:41.008672+0000","last_clean":"2023-10-16T14:10:41.008672+0000","last_became_active":"2023-10-16T14:10:39.985187+0000","last_became_peered":"2023-10-16T14:10:39.985187+0000","last_unstale":"2023-10-16T14:10:41.008672+0000","last_undegraded":"2023-10-16T14:10:41.008672+0000","last_fullsized":"2023-10-16T14:10:41.008672+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:32.423740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125498+0000","last_change":"2023-10-16T14:10:40.989384+0000","last_active":"2023-10-16T14:10:42.125498+0000","last_peered":"2023-10-16T14:10:42.125498+0000","last_clean":"2023-10-16T14:10:42.125498+0000","last_became_active":"2023-10-16T14:10:40.989064+0000","last_became_peered":"2023-10-16T14:10:40.989064+0000","last_unstale":"2023-10-16T14:10:42.125498+0000","last_undegraded":"2023-10-16T14:10:42.125498+0000","last_fullsized":"2023-10-16T14:10:42.125498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:50.164078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974785+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:40.974785+0000","last_peered":"2023-10-16T14:10:40.974785+0000","last_clean":"2023-10-16T14:10:40.974785+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:40.974785+0000","last_undegraded":"2023-10-16T14:10:40.974785+0000","last_fullsized":"2023-10-16T14:10:40.974785+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967632+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:40.967632+0000","last_peered":"2023-10-16T14:10:40.967632+0000","last_clean":"2023-10-16T14:10:40.967632+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:40.967632+0000","last_undegraded":"2023-10-16T14:10:40.967632+0000","last_fullsized":"2023-10-16T14:10:40.967632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971923+0000","last_change":"2023-10-16T14:10:39.986732+0000","last_active":"2023-10-16T14:10:40.971923+0000","last_peered":"2023-10-16T14:10:40.971923+0000","last_clean":"2023-10-16T14:10:40.971923+0000","last_became_active":"2023-10-16T14:10:39.986276+0000","last_became_peered":"2023-10-16T14:10:39.986276+0000","last_unstale":"2023-10-16T14:10:40.971923+0000","last_undegraded":"2023-10-16T14:10:40.971923+0000","last_fullsized":"2023-10-16T14:10:40.971923+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:52.213467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143312+0000","last_change":"2023-10-16T14:10:40.994651+0000","last_active":"2023-10-16T14:10:43.143312+0000","last_peered":"2023-10-16T14:10:43.143312+0000","last_clean":"2023-10-16T14:10:43.143312+0000","last_became_active":"2023-10-16T14:10:40.994387+0000","last_became_peered":"2023-10-16T14:10:40.994387+0000","last_unstale":"2023-10-16T14:10:43.143312+0000","last_undegraded":"2023-10-16T14:10:43.143312+0000","last_fullsized":"2023-10-16T14:10:43.143312+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:54.855496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977551+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:40.977551+0000","last_peered":"2023-10-16T14:10:40.977551+0000","last_clean":"2023-10-16T14:10:40.977551+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:40.977551+0000","last_undegraded":"2023-10-16T14:10:40.977551+0000","last_fullsized":"2023-10-16T14:10:40.977551+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981376+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:40.981376+0000","last_peered":"2023-10-16T14:10:40.981376+0000","last_clean":"2023-10-16T14:10:40.981376+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:40.981376+0000","last_undegraded":"2023-10-16T14:10:40.981376+0000","last_fullsized":"2023-10-16T14:10:40.981376+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985469+0000","last_change":"2023-10-16T14:10:39.989343+0000","last_active":"2023-10-16T14:10:40.985469+0000","last_peered":"2023-10-16T14:10:40.985469+0000","last_clean":"2023-10-16T14:10:40.985469+0000","last_became_active":"2023-10-16T14:10:39.989043+0000","last_became_peered":"2023-10-16T14:10:39.989043+0000","last_unstale":"2023-10-16T14:10:40.985469+0000","last_undegraded":"2023-10-16T14:10:40.985469+0000","last_fullsized":"2023-10-16T14:10:40.985469+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:57.519740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144062+0000","last_change":"2023-10-16T14:10:40.993217+0000","last_active":"2023-10-16T14:10:43.144062+0000","last_peered":"2023-10-16T14:10:43.144062+0000","last_clean":"2023-10-16T14:10:43.144062+0000","last_became_active":"2023-10-16T14:10:40.992887+0000","last_became_peered":"2023-10-16T14:10:40.992887+0000","last_unstale":"2023-10-16T14:10:43.144062+0000","last_undegraded":"2023-10-16T14:10:43.144062+0000","last_fullsized":"2023-10-16T14:10:43.144062+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.772938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979800+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:40.979800+0000","last_peered":"2023-10-16T14:10:40.979800+0000","last_clean":"2023-10-16T14:10:40.979800+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:40.979800+0000","last_undegraded":"2023-10-16T14:10:40.979800+0000","last_fullsized":"2023-10-16T14:10:40.979800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987734+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:40.987734+0000","last_peered":"2023-10-16T14:10:40.987734+0000","last_clean":"2023-10-16T14:10:40.987734+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:40.987734+0000","last_undegraded":"2023-10-16T14:10:40.987734+0000","last_fullsized":"2023-10-16T14:10:40.987734+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980253+0000","last_change":"2023-10-16T14:10:39.979128+0000","last_active":"2023-10-16T14:10:40.980253+0000","last_peered":"2023-10-16T14:10:40.980253+0000","last_clean":"2023-10-16T14:10:40.980253+0000","last_became_active":"2023-10-16T14:10:39.978782+0000","last_became_peered":"2023-10-16T14:10:39.978782+0000","last_unstale":"2023-10-16T14:10:40.980253+0000","last_undegraded":"2023-10-16T14:10:40.980253+0000","last_fullsized":"2023-10-16T14:10:40.980253+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:28.231837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997313+0000","last_change":"2023-10-16T14:10:40.989722+0000","last_active":"2023-10-16T14:10:41.997313+0000","last_peered":"2023-10-16T14:10:41.997313+0000","last_clean":"2023-10-16T14:10:41.997313+0000","last_became_active":"2023-10-16T14:10:40.989477+0000","last_became_peered":"2023-10-16T14:10:40.989477+0000","last_unstale":"2023-10-16T14:10:41.997313+0000","last_undegraded":"2023-10-16T14:10:41.997313+0000","last_fullsized":"2023-10-16T14:10:41.997313+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971953+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:40.971953+0000","last_peered":"2023-10-16T14:10:40.971953+0000","last_clean":"2023-10-16T14:10:40.971953+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:40.971953+0000","last_undegraded":"2023-10-16T14:10:40.971953+0000","last_fullsized":"2023-10-16T14:10:40.971953+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972632+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:40.972632+0000","last_peered":"2023-10-16T14:10:40.972632+0000","last_clean":"2023-10-16T14:10:40.972632+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:40.972632+0000","last_undegraded":"2023-10-16T14:10:40.972632+0000","last_fullsized":"2023-10-16T14:10:40.972632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969118+0000","last_change":"2023-10-16T14:10:39.982849+0000","last_active":"2023-10-16T14:10:40.969118+0000","last_peered":"2023-10-16T14:10:40.969118+0000","last_clean":"2023-10-16T14:10:40.969118+0000","last_became_active":"2023-10-16T14:10:39.982560+0000","last_became_peered":"2023-10-16T14:10:39.982560+0000","last_unstale":"2023-10-16T14:10:40.969118+0000","last_undegraded":"2023-10-16T14:10:40.969118+0000","last_fullsized":"2023-10-16T14:10:40.969118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:51.220017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242743+0000","last_change":"2023-10-16T14:10:40.991901+0000","last_active":"2023-10-16T14:10:43.242743+0000","last_peered":"2023-10-16T14:10:43.242743+0000","last_clean":"2023-10-16T14:10:43.242743+0000","last_became_active":"2023-10-16T14:10:40.991610+0000","last_became_peered":"2023-10-16T14:10:40.991610+0000","last_unstale":"2023-10-16T14:10:43.242743+0000","last_undegraded":"2023-10-16T14:10:43.242743+0000","last_fullsized":"2023-10-16T14:10:43.242743+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"231'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979815+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:40.979815+0000","last_peered":"2023-10-16T14:10:40.979815+0000","last_clean":"2023-10-16T14:10:40.979815+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:40.979815+0000","last_undegraded":"2023-10-16T14:10:40.979815+0000","last_fullsized":"2023-10-16T14:10:40.979815+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978358+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:40.978358+0000","last_peered":"2023-10-16T14:10:40.978358+0000","last_clean":"2023-10-16T14:10:40.978358+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:40.978358+0000","last_undegraded":"2023-10-16T14:10:40.978358+0000","last_fullsized":"2023-10-16T14:10:40.978358+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967454+0000","last_change":"2023-10-16T14:10:39.986619+0000","last_active":"2023-10-16T14:10:40.967454+0000","last_peered":"2023-10-16T14:10:40.967454+0000","last_clean":"2023-10-16T14:10:40.967454+0000","last_became_active":"2023-10-16T14:10:39.985261+0000","last_became_peered":"2023-10-16T14:10:39.985261+0000","last_unstale":"2023-10-16T14:10:40.967454+0000","last_undegraded":"2023-10-16T14:10:40.967454+0000","last_fullsized":"2023-10-16T14:10:40.967454+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:00.760156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.141717+0000","last_change":"2023-10-16T14:10:40.990315+0000","last_active":"2023-10-16T14:10:43.141717+0000","last_peered":"2023-10-16T14:10:43.141717+0000","last_clean":"2023-10-16T14:10:43.141717+0000","last_became_active":"2023-10-16T14:10:40.990060+0000","last_became_peered":"2023-10-16T14:10:40.990060+0000","last_unstale":"2023-10-16T14:10:43.141717+0000","last_undegraded":"2023-10-16T14:10:43.141717+0000","last_fullsized":"2023-10-16T14:10:43.141717+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.954735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980892+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:40.980892+0000","last_peered":"2023-10-16T14:10:40.980892+0000","last_clean":"2023-10-16T14:10:40.980892+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:40.980892+0000","last_undegraded":"2023-10-16T14:10:40.980892+0000","last_fullsized":"2023-10-16T14:10:40.980892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969751+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:40.969751+0000","last_peered":"2023-10-16T14:10:40.969751+0000","last_clean":"2023-10-16T14:10:40.969751+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:40.969751+0000","last_undegraded":"2023-10-16T14:10:40.969751+0000","last_fullsized":"2023-10-16T14:10:40.969751+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976136+0000","last_change":"2023-10-16T14:10:39.987697+0000","last_active":"2023-10-16T14:10:40.976136+0000","last_peered":"2023-10-16T14:10:40.976136+0000","last_clean":"2023-10-16T14:10:40.976136+0000","last_became_active":"2023-10-16T14:10:39.987388+0000","last_became_peered":"2023-10-16T14:10:39.987388+0000","last_unstale":"2023-10-16T14:10:40.976136+0000","last_undegraded":"2023-10-16T14:10:40.976136+0000","last_fullsized":"2023-10-16T14:10:40.976136+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:46.559129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244071+0000","last_change":"2023-10-16T14:10:40.983968+0000","last_active":"2023-10-16T14:10:43.244071+0000","last_peered":"2023-10-16T14:10:43.244071+0000","last_clean":"2023-10-16T14:10:43.244071+0000","last_became_active":"2023-10-16T14:10:40.983650+0000","last_became_peered":"2023-10-16T14:10:40.983650+0000","last_unstale":"2023-10-16T14:10:43.244071+0000","last_undegraded":"2023-10-16T14:10:43.244071+0000","last_fullsized":"2023-10-16T14:10:43.244071+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969088+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:40.969088+0000","last_peered":"2023-10-16T14:10:40.969088+0000","last_clean":"2023-10-16T14:10:40.969088+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:40.969088+0000","last_undegraded":"2023-10-16T14:10:40.969088+0000","last_fullsized":"2023-10-16T14:10:40.969088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.989095+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:40.989095+0000","last_peered":"2023-10-16T14:10:40.989095+0000","last_clean":"2023-10-16T14:10:40.989095+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:40.989095+0000","last_undegraded":"2023-10-16T14:10:40.989095+0000","last_fullsized":"2023-10-16T14:10:40.989095+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009061+0000","last_change":"2023-10-16T14:10:39.982498+0000","last_active":"2023-10-16T14:10:41.009061+0000","last_peered":"2023-10-16T14:10:41.009061+0000","last_clean":"2023-10-16T14:10:41.009061+0000","last_became_active":"2023-10-16T14:10:39.982162+0000","last_became_peered":"2023-10-16T14:10:39.982162+0000","last_unstale":"2023-10-16T14:10:41.009061+0000","last_undegraded":"2023-10-16T14:10:41.009061+0000","last_fullsized":"2023-10-16T14:10:41.009061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:11.621670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.187140+0000","last_change":"2023-10-16T14:10:40.990434+0000","last_active":"2023-10-16T14:10:43.187140+0000","last_peered":"2023-10-16T14:10:43.187140+0000","last_clean":"2023-10-16T14:10:43.187140+0000","last_became_active":"2023-10-16T14:10:40.990134+0000","last_became_peered":"2023-10-16T14:10:40.990134+0000","last_unstale":"2023-10-16T14:10:43.187140+0000","last_undegraded":"2023-10-16T14:10:43.187140+0000","last_fullsized":"2023-10-16T14:10:43.187140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:35.937011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978483+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:40.978483+0000","last_peered":"2023-10-16T14:10:40.978483+0000","last_clean":"2023-10-16T14:10:40.978483+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:40.978483+0000","last_undegraded":"2023-10-16T14:10:40.978483+0000","last_fullsized":"2023-10-16T14:10:40.978483+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967437+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:40.967437+0000","last_peered":"2023-10-16T14:10:40.967437+0000","last_clean":"2023-10-16T14:10:40.967437+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:40.967437+0000","last_undegraded":"2023-10-16T14:10:40.967437+0000","last_fullsized":"2023-10-16T14:10:40.967437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010987+0000","last_change":"2023-10-16T14:10:39.984550+0000","last_active":"2023-10-16T14:10:41.010987+0000","last_peered":"2023-10-16T14:10:41.010987+0000","last_clean":"2023-10-16T14:10:41.010987+0000","last_became_active":"2023-10-16T14:10:39.984260+0000","last_became_peered":"2023-10-16T14:10:39.984260+0000","last_unstale":"2023-10-16T14:10:41.010987+0000","last_undegraded":"2023-10-16T14:10:41.010987+0000","last_fullsized":"2023-10-16T14:10:41.010987+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:14.438775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143524+0000","last_change":"2023-10-16T14:10:40.983245+0000","last_active":"2023-10-16T14:10:43.143524+0000","last_peered":"2023-10-16T14:10:43.143524+0000","last_clean":"2023-10-16T14:10:43.143524+0000","last_became_active":"2023-10-16T14:10:40.982951+0000","last_became_peered":"2023-10-16T14:10:40.982951+0000","last_unstale":"2023-10-16T14:10:43.143524+0000","last_undegraded":"2023-10-16T14:10:43.143524+0000","last_fullsized":"2023-10-16T14:10:43.143524+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:14.127769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981490+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:40.981490+0000","last_peered":"2023-10-16T14:10:40.981490+0000","last_clean":"2023-10-16T14:10:40.981490+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:40.981490+0000","last_undegraded":"2023-10-16T14:10:40.981490+0000","last_fullsized":"2023-10-16T14:10:40.981490+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981982+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:40.981982+0000","last_peered":"2023-10-16T14:10:40.981982+0000","last_clean":"2023-10-16T14:10:40.981982+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:40.981982+0000","last_undegraded":"2023-10-16T14:10:40.981982+0000","last_fullsized":"2023-10-16T14:10:40.981982+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961192+0000","last_change":"2023-10-16T14:10:39.981457+0000","last_active":"2023-10-16T14:10:40.961192+0000","last_peered":"2023-10-16T14:10:40.961192+0000","last_clean":"2023-10-16T14:10:40.961192+0000","last_became_active":"2023-10-16T14:10:39.981145+0000","last_became_peered":"2023-10-16T14:10:39.981145+0000","last_unstale":"2023-10-16T14:10:40.961192+0000","last_undegraded":"2023-10-16T14:10:40.961192+0000","last_fullsized":"2023-10-16T14:10:40.961192+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:33.143537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143133+0000","last_change":"2023-10-16T14:10:40.982170+0000","last_active":"2023-10-16T14:10:43.143133+0000","last_peered":"2023-10-16T14:10:43.143133+0000","last_clean":"2023-10-16T14:10:43.143133+0000","last_became_active":"2023-10-16T14:10:40.981890+0000","last_became_peered":"2023-10-16T14:10:40.981890+0000","last_unstale":"2023-10-16T14:10:43.143133+0000","last_undegraded":"2023-10-16T14:10:43.143133+0000","last_fullsized":"2023-10-16T14:10:43.143133+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:09.758522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960165+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:40.960165+0000","last_peered":"2023-10-16T14:10:40.960165+0000","last_clean":"2023-10-16T14:10:40.960165+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:40.960165+0000","last_undegraded":"2023-10-16T14:10:40.960165+0000","last_fullsized":"2023-10-16T14:10:40.960165+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960903+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:40.960903+0000","last_peered":"2023-10-16T14:10:40.960903+0000","last_clean":"2023-10-16T14:10:40.960903+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:40.960903+0000","last_undegraded":"2023-10-16T14:10:40.960903+0000","last_fullsized":"2023-10-16T14:10:40.960903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010705+0000","last_change":"2023-10-16T14:10:39.984267+0000","last_active":"2023-10-16T14:10:41.010705+0000","last_peered":"2023-10-16T14:10:41.010705+0000","last_clean":"2023-10-16T14:10:41.010705+0000","last_became_active":"2023-10-16T14:10:39.983894+0000","last_became_peered":"2023-10-16T14:10:39.983894+0000","last_unstale":"2023-10-16T14:10:41.010705+0000","last_undegraded":"2023-10-16T14:10:41.010705+0000","last_fullsized":"2023-10-16T14:10:41.010705+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:02.427201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.142906+0000","last_change":"2023-10-16T14:10:40.994626+0000","last_active":"2023-10-16T14:10:43.142906+0000","last_peered":"2023-10-16T14:10:43.142906+0000","last_clean":"2023-10-16T14:10:43.142906+0000","last_became_active":"2023-10-16T14:10:40.994283+0000","last_became_peered":"2023-10-16T14:10:40.994283+0000","last_unstale":"2023-10-16T14:10:43.142906+0000","last_undegraded":"2023-10-16T14:10:43.142906+0000","last_fullsized":"2023-10-16T14:10:43.142906+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:16.777756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979684+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:40.979684+0000","last_peered":"2023-10-16T14:10:40.979684+0000","last_clean":"2023-10-16T14:10:40.979684+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:40.979684+0000","last_undegraded":"2023-10-16T14:10:40.979684+0000","last_fullsized":"2023-10-16T14:10:40.979684+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961306+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:40.961306+0000","last_peered":"2023-10-16T14:10:40.961306+0000","last_clean":"2023-10-16T14:10:40.961306+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:40.961306+0000","last_undegraded":"2023-10-16T14:10:40.961306+0000","last_fullsized":"2023-10-16T14:10:40.961306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971464+0000","last_change":"2023-10-16T14:10:39.974033+0000","last_active":"2023-10-16T14:10:40.971464+0000","last_peered":"2023-10-16T14:10:40.971464+0000","last_clean":"2023-10-16T14:10:40.971464+0000","last_became_active":"2023-10-16T14:10:39.973723+0000","last_became_peered":"2023-10-16T14:10:39.973723+0000","last_unstale":"2023-10-16T14:10:40.971464+0000","last_undegraded":"2023-10-16T14:10:40.971464+0000","last_fullsized":"2023-10-16T14:10:40.971464+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:08.572098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124517+0000","last_change":"2023-10-16T14:10:40.994623+0000","last_active":"2023-10-16T14:10:42.124517+0000","last_peered":"2023-10-16T14:10:42.124517+0000","last_clean":"2023-10-16T14:10:42.124517+0000","last_became_active":"2023-10-16T14:10:40.994347+0000","last_became_peered":"2023-10-16T14:10:40.994347+0000","last_unstale":"2023-10-16T14:10:42.124517+0000","last_undegraded":"2023-10-16T14:10:42.124517+0000","last_fullsized":"2023-10-16T14:10:42.124517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:18.643655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961579+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:40.961579+0000","last_peered":"2023-10-16T14:10:40.961579+0000","last_clean":"2023-10-16T14:10:40.961579+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:40.961579+0000","last_undegraded":"2023-10-16T14:10:40.961579+0000","last_fullsized":"2023-10-16T14:10:40.961579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979554+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:40.979554+0000","last_peered":"2023-10-16T14:10:40.979554+0000","last_clean":"2023-10-16T14:10:40.979554+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:40.979554+0000","last_undegraded":"2023-10-16T14:10:40.979554+0000","last_fullsized":"2023-10-16T14:10:40.979554+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009801+0000","last_change":"2023-10-16T14:10:39.986465+0000","last_active":"2023-10-16T14:10:41.009801+0000","last_peered":"2023-10-16T14:10:41.009801+0000","last_clean":"2023-10-16T14:10:41.009801+0000","last_became_active":"2023-10-16T14:10:39.986165+0000","last_became_peered":"2023-10-16T14:10:39.986165+0000","last_unstale":"2023-10-16T14:10:41.009801+0000","last_undegraded":"2023-10-16T14:10:41.009801+0000","last_fullsized":"2023-10-16T14:10:41.009801+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:08.068911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124542+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:42.124542+0000","last_peered":"2023-10-16T14:10:42.124542+0000","last_clean":"2023-10-16T14:10:42.124542+0000","last_became_active":"2023-10-16T14:10:40.993130+0000","last_became_peered":"2023-10-16T14:10:40.993130+0000","last_unstale":"2023-10-16T14:10:42.124542+0000","last_undegraded":"2023-10-16T14:10:42.124542+0000","last_fullsized":"2023-10-16T14:10:42.124542+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:25.881584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973127+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:40.973127+0000","last_peered":"2023-10-16T14:10:40.973127+0000","last_clean":"2023-10-16T14:10:40.973127+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:40.973127+0000","last_undegraded":"2023-10-16T14:10:40.973127+0000","last_fullsized":"2023-10-16T14:10:40.973127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974510+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:40.974510+0000","last_peered":"2023-10-16T14:10:40.974510+0000","last_clean":"2023-10-16T14:10:40.974510+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:40.974510+0000","last_undegraded":"2023-10-16T14:10:40.974510+0000","last_fullsized":"2023-10-16T14:10:40.974510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967349+0000","last_change":"2023-10-16T14:10:39.986684+0000","last_active":"2023-10-16T14:10:40.967349+0000","last_peered":"2023-10-16T14:10:40.967349+0000","last_clean":"2023-10-16T14:10:40.967349+0000","last_became_active":"2023-10-16T14:10:39.986367+0000","last_became_peered":"2023-10-16T14:10:39.986367+0000","last_unstale":"2023-10-16T14:10:40.967349+0000","last_undegraded":"2023-10-16T14:10:40.967349+0000","last_fullsized":"2023-10-16T14:10:40.967349+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:33:11.665429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185918+0000","last_change":"2023-10-16T14:10:40.993254+0000","last_active":"2023-10-16T14:10:43.185918+0000","last_peered":"2023-10-16T14:10:43.185918+0000","last_clean":"2023-10-16T14:10:43.185918+0000","last_became_active":"2023-10-16T14:10:40.992918+0000","last_became_peered":"2023-10-16T14:10:40.992918+0000","last_unstale":"2023-10-16T14:10:43.185918+0000","last_undegraded":"2023-10-16T14:10:43.185918+0000","last_fullsized":"2023-10-16T14:10:43.185918+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:07.376549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976318+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:40.976318+0000","last_peered":"2023-10-16T14:10:40.976318+0000","last_clean":"2023-10-16T14:10:40.976318+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:40.976318+0000","last_undegraded":"2023-10-16T14:10:40.976318+0000","last_fullsized":"2023-10-16T14:10:40.976318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972611+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:40.972611+0000","last_peered":"2023-10-16T14:10:40.972611+0000","last_clean":"2023-10-16T14:10:40.972611+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:40.972611+0000","last_undegraded":"2023-10-16T14:10:40.972611+0000","last_fullsized":"2023-10-16T14:10:40.972611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970330+0000","last_change":"2023-10-16T14:10:39.989051+0000","last_active":"2023-10-16T14:10:40.970330+0000","last_peered":"2023-10-16T14:10:40.970330+0000","last_clean":"2023-10-16T14:10:40.970330+0000","last_became_active":"2023-10-16T14:10:39.987944+0000","last_became_peered":"2023-10-16T14:10:39.987944+0000","last_unstale":"2023-10-16T14:10:40.970330+0000","last_undegraded":"2023-10-16T14:10:40.970330+0000","last_fullsized":"2023-10-16T14:10:40.970330+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244927+0000","last_change":"2023-10-16T14:10:40.996616+0000","last_active":"2023-10-16T14:10:43.244927+0000","last_peered":"2023-10-16T14:10:43.244927+0000","last_clean":"2023-10-16T14:10:43.244927+0000","last_became_active":"2023-10-16T14:10:40.996057+0000","last_became_peered":"2023-10-16T14:10:40.996057+0000","last_unstale":"2023-10-16T14:10:43.244927+0000","last_undegraded":"2023-10-16T14:10:43.244927+0000","last_fullsized":"2023-10-16T14:10:43.244927+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:32.328604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981955+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:40.981955+0000","last_peered":"2023-10-16T14:10:40.981955+0000","last_clean":"2023-10-16T14:10:40.981955+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:40.981955+0000","last_undegraded":"2023-10-16T14:10:40.981955+0000","last_fullsized":"2023-10-16T14:10:40.981955+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978301+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:40.978301+0000","last_peered":"2023-10-16T14:10:40.978301+0000","last_clean":"2023-10-16T14:10:40.978301+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:40.978301+0000","last_undegraded":"2023-10-16T14:10:40.978301+0000","last_fullsized":"2023-10-16T14:10:40.978301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980655+0000","last_change":"2023-10-16T14:10:39.972329+0000","last_active":"2023-10-16T14:10:40.980655+0000","last_peered":"2023-10-16T14:10:40.980655+0000","last_clean":"2023-10-16T14:10:40.980655+0000","last_became_active":"2023-10-16T14:10:39.972015+0000","last_became_peered":"2023-10-16T14:10:39.972015+0000","last_unstale":"2023-10-16T14:10:40.980655+0000","last_undegraded":"2023-10-16T14:10:40.980655+0000","last_fullsized":"2023-10-16T14:10:40.980655+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:17.467680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996971+0000","last_change":"2023-10-16T14:10:41.001961+0000","last_active":"2023-10-16T14:10:41.996971+0000","last_peered":"2023-10-16T14:10:41.996971+0000","last_clean":"2023-10-16T14:10:41.996971+0000","last_became_active":"2023-10-16T14:10:41.001699+0000","last_became_peered":"2023-10-16T14:10:41.001699+0000","last_unstale":"2023-10-16T14:10:41.996971+0000","last_undegraded":"2023-10-16T14:10:41.996971+0000","last_fullsized":"2023-10-16T14:10:41.996971+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959024+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:40.959024+0000","last_peered":"2023-10-16T14:10:40.959024+0000","last_clean":"2023-10-16T14:10:40.959024+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:40.959024+0000","last_undegraded":"2023-10-16T14:10:40.959024+0000","last_fullsized":"2023-10-16T14:10:40.959024+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975612+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:40.975612+0000","last_peered":"2023-10-16T14:10:40.975612+0000","last_clean":"2023-10-16T14:10:40.975612+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:40.975612+0000","last_undegraded":"2023-10-16T14:10:40.975612+0000","last_fullsized":"2023-10-16T14:10:40.975612+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985242+0000","last_change":"2023-10-16T14:10:39.990904+0000","last_active":"2023-10-16T14:10:40.985242+0000","last_peered":"2023-10-16T14:10:40.985242+0000","last_clean":"2023-10-16T14:10:40.985242+0000","last_became_active":"2023-10-16T14:10:39.990155+0000","last_became_peered":"2023-10-16T14:10:39.990155+0000","last_unstale":"2023-10-16T14:10:40.985242+0000","last_undegraded":"2023-10-16T14:10:40.985242+0000","last_fullsized":"2023-10-16T14:10:40.985242+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:56.282178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243767+0000","last_change":"2023-10-16T14:10:40.993640+0000","last_active":"2023-10-16T14:10:43.243767+0000","last_peered":"2023-10-16T14:10:43.243767+0000","last_clean":"2023-10-16T14:10:43.243767+0000","last_became_active":"2023-10-16T14:10:40.993084+0000","last_became_peered":"2023-10-16T14:10:40.993084+0000","last_unstale":"2023-10-16T14:10:43.243767+0000","last_undegraded":"2023-10-16T14:10:43.243767+0000","last_fullsized":"2023-10-16T14:10:43.243767+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:12.061147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987125+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:40.987125+0000","last_peered":"2023-10-16T14:10:40.987125+0000","last_clean":"2023-10-16T14:10:40.987125+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:40.987125+0000","last_undegraded":"2023-10-16T14:10:40.987125+0000","last_fullsized":"2023-10-16T14:10:40.987125+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959827+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:40.959827+0000","last_peered":"2023-10-16T14:10:40.959827+0000","last_clean":"2023-10-16T14:10:40.959827+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:40.959827+0000","last_undegraded":"2023-10-16T14:10:40.959827+0000","last_fullsized":"2023-10-16T14:10:40.959827+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"234'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013944+0000","last_change":"2023-10-16T14:10:39.971085+0000","last_active":"2023-10-16T14:10:41.013944+0000","last_peered":"2023-10-16T14:10:41.013944+0000","last_clean":"2023-10-16T14:10:41.013944+0000","last_became_active":"2023-10-16T14:10:39.970777+0000","last_became_peered":"2023-10-16T14:10:39.970777+0000","last_unstale":"2023-10-16T14:10:41.013944+0000","last_undegraded":"2023-10-16T14:10:41.013944+0000","last_fullsized":"2023-10-16T14:10:41.013944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:03.090258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.998098+0000","last_change":"2023-10-16T14:10:40.985478+0000","last_active":"2023-10-16T14:10:41.998098+0000","last_peered":"2023-10-16T14:10:41.998098+0000","last_clean":"2023-10-16T14:10:41.998098+0000","last_became_active":"2023-10-16T14:10:40.985218+0000","last_became_peered":"2023-10-16T14:10:40.985218+0000","last_unstale":"2023-10-16T14:10:41.998098+0000","last_undegraded":"2023-10-16T14:10:41.998098+0000","last_fullsized":"2023-10-16T14:10:41.998098+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:22.942748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.986478+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:40.986478+0000","last_peered":"2023-10-16T14:10:40.986478+0000","last_clean":"2023-10-16T14:10:40.986478+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:40.986478+0000","last_undegraded":"2023-10-16T14:10:40.986478+0000","last_fullsized":"2023-10-16T14:10:40.986478+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974315+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:40.974315+0000","last_peered":"2023-10-16T14:10:40.974315+0000","last_clean":"2023-10-16T14:10:40.974315+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:40.974315+0000","last_undegraded":"2023-10-16T14:10:40.974315+0000","last_fullsized":"2023-10-16T14:10:40.974315+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"234'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008061+0000","last_change":"2023-10-16T14:10:39.973954+0000","last_active":"2023-10-16T14:10:41.008061+0000","last_peered":"2023-10-16T14:10:41.008061+0000","last_clean":"2023-10-16T14:10:41.008061+0000","last_became_active":"2023-10-16T14:10:39.973602+0000","last_became_peered":"2023-10-16T14:10:39.973602+0000","last_unstale":"2023-10-16T14:10:41.008061+0000","last_undegraded":"2023-10-16T14:10:41.008061+0000","last_fullsized":"2023-10-16T14:10:41.008061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:18.496303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143885+0000","last_change":"2023-10-16T14:10:40.991452+0000","last_active":"2023-10-16T14:10:43.143885+0000","last_peered":"2023-10-16T14:10:43.143885+0000","last_clean":"2023-10-16T14:10:43.143885+0000","last_became_active":"2023-10-16T14:10:40.985777+0000","last_became_peered":"2023-10-16T14:10:40.985777+0000","last_unstale":"2023-10-16T14:10:43.143885+0000","last_undegraded":"2023-10-16T14:10:43.143885+0000","last_fullsized":"2023-10-16T14:10:43.143885+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.881711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"231'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.983108+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:40.983108+0000","last_peered":"2023-10-16T14:10:40.983108+0000","last_clean":"2023-10-16T14:10:40.983108+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:40.983108+0000","last_undegraded":"2023-10-16T14:10:40.983108+0000","last_fullsized":"2023-10-16T14:10:40.983108+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977730+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:40.977730+0000","last_peered":"2023-10-16T14:10:40.977730+0000","last_clean":"2023-10-16T14:10:40.977730+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:40.977730+0000","last_undegraded":"2023-10-16T14:10:40.977730+0000","last_fullsized":"2023-10-16T14:10:40.977730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980420+0000","last_change":"2023-10-16T14:10:39.976858+0000","last_active":"2023-10-16T14:10:40.980420+0000","last_peered":"2023-10-16T14:10:40.980420+0000","last_clean":"2023-10-16T14:10:40.980420+0000","last_became_active":"2023-10-16T14:10:39.976566+0000","last_became_peered":"2023-10-16T14:10:39.976566+0000","last_unstale":"2023-10-16T14:10:40.980420+0000","last_undegraded":"2023-10-16T14:10:40.980420+0000","last_fullsized":"2023-10-16T14:10:40.980420+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:00.617867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.990318+0000","last_change":"2023-10-16T14:10:40.985099+0000","last_active":"2023-10-16T14:10:41.990318+0000","last_peered":"2023-10-16T14:10:41.990318+0000","last_clean":"2023-10-16T14:10:41.990318+0000","last_became_active":"2023-10-16T14:10:40.984799+0000","last_became_peered":"2023-10-16T14:10:40.984799+0000","last_unstale":"2023-10-16T14:10:41.990318+0000","last_undegraded":"2023-10-16T14:10:41.990318+0000","last_fullsized":"2023-10-16T14:10:41.990318+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:02.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":19,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961436+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:40.961436+0000","last_peered":"2023-10-16T14:10:40.961436+0000","last_clean":"2023-10-16T14:10:40.961436+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:40.961436+0000","last_undegraded":"2023-10-16T14:10:40.961436+0000","last_fullsized":"2023-10-16T14:10:40.961436+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982473+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:40.982473+0000","last_peered":"2023-10-16T14:10:40.982473+0000","last_clean":"2023-10-16T14:10:40.982473+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:40.982473+0000","last_undegraded":"2023-10-16T14:10:40.982473+0000","last_fullsized":"2023-10-16T14:10:40.982473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011710+0000","last_change":"2023-10-16T14:10:39.986334+0000","last_active":"2023-10-16T14:10:41.011710+0000","last_peered":"2023-10-16T14:10:41.011710+0000","last_clean":"2023-10-16T14:10:41.011710+0000","last_became_active":"2023-10-16T14:10:39.986002+0000","last_became_peered":"2023-10-16T14:10:39.986002+0000","last_unstale":"2023-10-16T14:10:41.011710+0000","last_undegraded":"2023-10-16T14:10:41.011710+0000","last_fullsized":"2023-10-16T14:10:41.011710+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994013+0000","last_change":"2023-10-16T14:10:40.976822+0000","last_active":"2023-10-16T14:10:41.994013+0000","last_peered":"2023-10-16T14:10:41.994013+0000","last_clean":"2023-10-16T14:10:41.994013+0000","last_became_active":"2023-10-16T14:10:40.976523+0000","last_became_peered":"2023-10-16T14:10:40.976523+0000","last_unstale":"2023-10-16T14:10:41.994013+0000","last_undegraded":"2023-10-16T14:10:41.994013+0000","last_fullsized":"2023-10-16T14:10:41.994013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987031+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:40.987031+0000","last_peered":"2023-10-16T14:10:40.987031+0000","last_clean":"2023-10-16T14:10:40.987031+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:40.987031+0000","last_undegraded":"2023-10-16T14:10:40.987031+0000","last_fullsized":"2023-10-16T14:10:40.987031+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980779+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:40.980779+0000","last_peered":"2023-10-16T14:10:40.980779+0000","last_clean":"2023-10-16T14:10:40.980779+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:40.980779+0000","last_undegraded":"2023-10-16T14:10:40.980779+0000","last_fullsized":"2023-10-16T14:10:40.980779+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"234'8","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013387+0000","last_change":"2023-10-16T14:10:39.979255+0000","last_active":"2023-10-16T14:10:41.013387+0000","last_peered":"2023-10-16T14:10:41.013387+0000","last_clean":"2023-10-16T14:10:41.013387+0000","last_became_active":"2023-10-16T14:10:39.977217+0000","last_became_peered":"2023-10-16T14:10:39.977217+0000","last_unstale":"2023-10-16T14:10:41.013387+0000","last_undegraded":"2023-10-16T14:10:41.013387+0000","last_fullsized":"2023-10-16T14:10:41.013387+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:30.019236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.142508+0000","last_change":"2023-10-16T14:10:40.989802+0000","last_active":"2023-10-16T14:10:43.142508+0000","last_peered":"2023-10-16T14:10:43.142508+0000","last_clean":"2023-10-16T14:10:43.142508+0000","last_became_active":"2023-10-16T14:10:40.988409+0000","last_became_peered":"2023-10-16T14:10:40.988409+0000","last_unstale":"2023-10-16T14:10:43.142508+0000","last_undegraded":"2023-10-16T14:10:43.142508+0000","last_fullsized":"2023-10-16T14:10:43.142508+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:16.220155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976181+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:40.976181+0000","last_peered":"2023-10-16T14:10:40.976181+0000","last_clean":"2023-10-16T14:10:40.976181+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:40.976181+0000","last_undegraded":"2023-10-16T14:10:40.976181+0000","last_fullsized":"2023-10-16T14:10:40.976181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979705+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:40.979705+0000","last_peered":"2023-10-16T14:10:40.979705+0000","last_clean":"2023-10-16T14:10:40.979705+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:40.979705+0000","last_undegraded":"2023-10-16T14:10:40.979705+0000","last_fullsized":"2023-10-16T14:10:40.979705+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975789+0000","last_change":"2023-10-16T14:10:39.968896+0000","last_active":"2023-10-16T14:10:40.975789+0000","last_peered":"2023-10-16T14:10:40.975789+0000","last_clean":"2023-10-16T14:10:40.975789+0000","last_became_active":"2023-10-16T14:10:39.968633+0000","last_became_peered":"2023-10-16T14:10:39.968633+0000","last_unstale":"2023-10-16T14:10:40.975789+0000","last_undegraded":"2023-10-16T14:10:40.975789+0000","last_fullsized":"2023-10-16T14:10:40.975789+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:26.024819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186867+0000","last_change":"2023-10-16T14:10:40.999802+0000","last_active":"2023-10-16T14:10:43.186867+0000","last_peered":"2023-10-16T14:10:43.186867+0000","last_clean":"2023-10-16T14:10:43.186867+0000","last_became_active":"2023-10-16T14:10:40.999483+0000","last_became_peered":"2023-10-16T14:10:40.999483+0000","last_unstale":"2023-10-16T14:10:43.186867+0000","last_undegraded":"2023-10-16T14:10:43.186867+0000","last_fullsized":"2023-10-16T14:10:43.186867+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:19.512934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011187+0000","last_change":"2023-10-16T14:10:39.979452+0000","last_active":"2023-10-16T14:10:41.011187+0000","last_peered":"2023-10-16T14:10:41.011187+0000","last_clean":"2023-10-16T14:10:41.011187+0000","last_became_active":"2023-10-16T14:10:39.978243+0000","last_became_peered":"2023-10-16T14:10:39.978243+0000","last_unstale":"2023-10-16T14:10:41.011187+0000","last_undegraded":"2023-10-16T14:10:41.011187+0000","last_fullsized":"2023-10-16T14:10:41.011187+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.984244+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:40.984244+0000","last_peered":"2023-10-16T14:10:40.984244+0000","last_clean":"2023-10-16T14:10:40.984244+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:40.984244+0000","last_undegraded":"2023-10-16T14:10:40.984244+0000","last_fullsized":"2023-10-16T14:10:40.984244+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976062+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:40.976062+0000","last_peered":"2023-10-16T14:10:40.976062+0000","last_clean":"2023-10-16T14:10:40.976062+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:40.976062+0000","last_undegraded":"2023-10-16T14:10:40.976062+0000","last_fullsized":"2023-10-16T14:10:40.976062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144120+0000","last_change":"2023-10-16T14:10:40.996750+0000","last_active":"2023-10-16T14:10:43.144120+0000","last_peered":"2023-10-16T14:10:43.144120+0000","last_clean":"2023-10-16T14:10:43.144120+0000","last_became_active":"2023-10-16T14:10:40.996369+0000","last_became_peered":"2023-10-16T14:10:40.996369+0000","last_unstale":"2023-10-16T14:10:43.144120+0000","last_undegraded":"2023-10-16T14:10:43.144120+0000","last_fullsized":"2023-10-16T14:10:43.144120+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:29.608507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"234'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009270+0000","last_change":"2023-10-16T14:10:39.973971+0000","last_active":"2023-10-16T14:10:41.009270+0000","last_peered":"2023-10-16T14:10:41.009270+0000","last_clean":"2023-10-16T14:10:41.009270+0000","last_became_active":"2023-10-16T14:10:39.973675+0000","last_became_peered":"2023-10-16T14:10:39.973675+0000","last_unstale":"2023-10-16T14:10:41.009270+0000","last_undegraded":"2023-10-16T14:10:41.009270+0000","last_fullsized":"2023-10-16T14:10:41.009270+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:01.458774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976322+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:40.976322+0000","last_peered":"2023-10-16T14:10:40.976322+0000","last_clean":"2023-10-16T14:10:40.976322+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:40.976322+0000","last_undegraded":"2023-10-16T14:10:40.976322+0000","last_fullsized":"2023-10-16T14:10:40.976322+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968823+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:40.968823+0000","last_peered":"2023-10-16T14:10:40.968823+0000","last_clean":"2023-10-16T14:10:40.968823+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:40.968823+0000","last_undegraded":"2023-10-16T14:10:40.968823+0000","last_fullsized":"2023-10-16T14:10:40.968823+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995327+0000","last_change":"2023-10-16T14:10:40.984926+0000","last_active":"2023-10-16T14:10:41.995327+0000","last_peered":"2023-10-16T14:10:41.995327+0000","last_clean":"2023-10-16T14:10:41.995327+0000","last_became_active":"2023-10-16T14:10:40.984364+0000","last_became_peered":"2023-10-16T14:10:40.984364+0000","last_unstale":"2023-10-16T14:10:41.995327+0000","last_undegraded":"2023-10-16T14:10:41.995327+0000","last_fullsized":"2023-10-16T14:10:41.995327+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974558+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:40.974558+0000","last_peered":"2023-10-16T14:10:40.974558+0000","last_clean":"2023-10-16T14:10:40.974558+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:40.974558+0000","last_undegraded":"2023-10-16T14:10:40.974558+0000","last_fullsized":"2023-10-16T14:10:40.974558+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977408+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:40.977408+0000","last_peered":"2023-10-16T14:10:40.977408+0000","last_clean":"2023-10-16T14:10:40.977408+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:40.977408+0000","last_undegraded":"2023-10-16T14:10:40.977408+0000","last_fullsized":"2023-10-16T14:10:40.977408+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011689+0000","last_change":"2023-10-16T14:10:39.979320+0000","last_active":"2023-10-16T14:10:41.011689+0000","last_peered":"2023-10-16T14:10:41.011689+0000","last_clean":"2023-10-16T14:10:41.011689+0000","last_became_active":"2023-10-16T14:10:39.978947+0000","last_became_peered":"2023-10-16T14:10:39.978947+0000","last_unstale":"2023-10-16T14:10:41.011689+0000","last_undegraded":"2023-10-16T14:10:41.011689+0000","last_fullsized":"2023-10-16T14:10:41.011689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125751+0000","last_change":"2023-10-16T14:10:40.997566+0000","last_active":"2023-10-16T14:10:42.125751+0000","last_peered":"2023-10-16T14:10:42.125751+0000","last_clean":"2023-10-16T14:10:42.125751+0000","last_became_active":"2023-10-16T14:10:40.997273+0000","last_became_peered":"2023-10-16T14:10:40.997273+0000","last_unstale":"2023-10-16T14:10:42.125751+0000","last_undegraded":"2023-10-16T14:10:42.125751+0000","last_fullsized":"2023-10-16T14:10:42.125751+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:38.615513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982594+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:40.982594+0000","last_peered":"2023-10-16T14:10:40.982594+0000","last_clean":"2023-10-16T14:10:40.982594+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:40.982594+0000","last_undegraded":"2023-10-16T14:10:40.982594+0000","last_fullsized":"2023-10-16T14:10:40.982594+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960137+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:40.960137+0000","last_peered":"2023-10-16T14:10:40.960137+0000","last_clean":"2023-10-16T14:10:40.960137+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:40.960137+0000","last_undegraded":"2023-10-16T14:10:40.960137+0000","last_fullsized":"2023-10-16T14:10:40.960137+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009413+0000","last_change":"2023-10-16T14:10:39.983838+0000","last_active":"2023-10-16T14:10:41.009413+0000","last_peered":"2023-10-16T14:10:41.009413+0000","last_clean":"2023-10-16T14:10:41.009413+0000","last_became_active":"2023-10-16T14:10:39.983415+0000","last_became_peered":"2023-10-16T14:10:39.983415+0000","last_unstale":"2023-10-16T14:10:41.009413+0000","last_undegraded":"2023-10-16T14:10:41.009413+0000","last_fullsized":"2023-10-16T14:10:41.009413+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:58.890985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243037+0000","last_change":"2023-10-16T14:10:40.993412+0000","last_active":"2023-10-16T14:10:43.243037+0000","last_peered":"2023-10-16T14:10:43.243037+0000","last_clean":"2023-10-16T14:10:43.243037+0000","last_became_active":"2023-10-16T14:10:40.993119+0000","last_became_peered":"2023-10-16T14:10:40.993119+0000","last_unstale":"2023-10-16T14:10:43.243037+0000","last_undegraded":"2023-10-16T14:10:43.243037+0000","last_fullsized":"2023-10-16T14:10:43.243037+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:58.022778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981800+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:40.981800+0000","last_peered":"2023-10-16T14:10:40.981800+0000","last_clean":"2023-10-16T14:10:40.981800+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:40.981800+0000","last_undegraded":"2023-10-16T14:10:40.981800+0000","last_fullsized":"2023-10-16T14:10:40.981800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980136+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:40.980136+0000","last_peered":"2023-10-16T14:10:40.980136+0000","last_clean":"2023-10-16T14:10:40.980136+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:40.980136+0000","last_undegraded":"2023-10-16T14:10:40.980136+0000","last_fullsized":"2023-10-16T14:10:40.980136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011809+0000","last_change":"2023-10-16T14:10:39.969078+0000","last_active":"2023-10-16T14:10:41.011809+0000","last_peered":"2023-10-16T14:10:41.011809+0000","last_clean":"2023-10-16T14:10:41.011809+0000","last_became_active":"2023-10-16T14:10:39.968736+0000","last_became_peered":"2023-10-16T14:10:39.968736+0000","last_unstale":"2023-10-16T14:10:41.011809+0000","last_undegraded":"2023-10-16T14:10:41.011809+0000","last_fullsized":"2023-10-16T14:10:41.011809+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:59.354986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997370+0000","last_change":"2023-10-16T14:10:40.997896+0000","last_active":"2023-10-16T14:10:41.997370+0000","last_peered":"2023-10-16T14:10:41.997370+0000","last_clean":"2023-10-16T14:10:41.997370+0000","last_became_active":"2023-10-16T14:10:40.997623+0000","last_became_peered":"2023-10-16T14:10:40.997623+0000","last_unstale":"2023-10-16T14:10:41.997370+0000","last_undegraded":"2023-10-16T14:10:41.997370+0000","last_fullsized":"2023-10-16T14:10:41.997370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972873+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:40.972873+0000","last_peered":"2023-10-16T14:10:40.972873+0000","last_clean":"2023-10-16T14:10:40.972873+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:40.972873+0000","last_undegraded":"2023-10-16T14:10:40.972873+0000","last_fullsized":"2023-10-16T14:10:40.972873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.988803+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:40.988803+0000","last_peered":"2023-10-16T14:10:40.988803+0000","last_clean":"2023-10-16T14:10:40.988803+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:40.988803+0000","last_undegraded":"2023-10-16T14:10:40.988803+0000","last_fullsized":"2023-10-16T14:10:40.988803+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959567+0000","last_change":"2023-10-16T14:10:39.973296+0000","last_active":"2023-10-16T14:10:40.959567+0000","last_peered":"2023-10-16T14:10:40.959567+0000","last_clean":"2023-10-16T14:10:40.959567+0000","last_became_active":"2023-10-16T14:10:39.972592+0000","last_became_peered":"2023-10-16T14:10:39.972592+0000","last_unstale":"2023-10-16T14:10:40.959567+0000","last_undegraded":"2023-10-16T14:10:40.959567+0000","last_fullsized":"2023-10-16T14:10:40.959567+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:04.406174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242344+0000","last_change":"2023-10-16T14:10:40.987363+0000","last_active":"2023-10-16T14:10:43.242344+0000","last_peered":"2023-10-16T14:10:43.242344+0000","last_clean":"2023-10-16T14:10:43.242344+0000","last_became_active":"2023-10-16T14:10:40.985914+0000","last_became_peered":"2023-10-16T14:10:40.985914+0000","last_unstale":"2023-10-16T14:10:43.242344+0000","last_undegraded":"2023-10-16T14:10:43.242344+0000","last_fullsized":"2023-10-16T14:10:43.242344+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:10.423467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975348+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:40.975348+0000","last_peered":"2023-10-16T14:10:40.975348+0000","last_clean":"2023-10-16T14:10:40.975348+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:40.975348+0000","last_undegraded":"2023-10-16T14:10:40.975348+0000","last_fullsized":"2023-10-16T14:10:40.975348+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972050+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:40.972050+0000","last_peered":"2023-10-16T14:10:40.972050+0000","last_clean":"2023-10-16T14:10:40.972050+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:40.972050+0000","last_undegraded":"2023-10-16T14:10:40.972050+0000","last_fullsized":"2023-10-16T14:10:40.972050+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970323+0000","last_change":"2023-10-16T14:10:39.972648+0000","last_active":"2023-10-16T14:10:40.970323+0000","last_peered":"2023-10-16T14:10:40.970323+0000","last_clean":"2023-10-16T14:10:40.970323+0000","last_became_active":"2023-10-16T14:10:39.972328+0000","last_became_peered":"2023-10-16T14:10:39.972328+0000","last_unstale":"2023-10-16T14:10:40.970323+0000","last_undegraded":"2023-10-16T14:10:40.970323+0000","last_fullsized":"2023-10-16T14:10:40.970323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186986+0000","last_change":"2023-10-16T14:10:40.995949+0000","last_active":"2023-10-16T14:10:43.186986+0000","last_peered":"2023-10-16T14:10:43.186986+0000","last_clean":"2023-10-16T14:10:43.186986+0000","last_became_active":"2023-10-16T14:10:40.995689+0000","last_became_peered":"2023-10-16T14:10:40.995689+0000","last_unstale":"2023-10-16T14:10:43.186986+0000","last_undegraded":"2023-10-16T14:10:43.186986+0000","last_fullsized":"2023-10-16T14:10:43.186986+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:12.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977098+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:40.977098+0000","last_peered":"2023-10-16T14:10:40.977098+0000","last_clean":"2023-10-16T14:10:40.977098+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:40.977098+0000","last_undegraded":"2023-10-16T14:10:40.977098+0000","last_fullsized":"2023-10-16T14:10:40.977098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244569+0000","last_change":"2023-10-16T14:10:41.001647+0000","last_active":"2023-10-16T14:10:43.244569+0000","last_peered":"2023-10-16T14:10:43.244569+0000","last_clean":"2023-10-16T14:10:43.244569+0000","last_became_active":"2023-10-16T14:10:41.000840+0000","last_became_peered":"2023-10-16T14:10:41.000840+0000","last_unstale":"2023-10-16T14:10:43.244569+0000","last_undegraded":"2023-10-16T14:10:43.244569+0000","last_fullsized":"2023-10-16T14:10:43.244569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:55.904685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959155+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:40.959155+0000","last_peered":"2023-10-16T14:10:40.959155+0000","last_clean":"2023-10-16T14:10:40.959155+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:40.959155+0000","last_undegraded":"2023-10-16T14:10:40.959155+0000","last_fullsized":"2023-10-16T14:10:40.959155+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243482+0000","last_change":"2023-10-16T14:10:40.991703+0000","last_active":"2023-10-16T14:10:43.243482+0000","last_peered":"2023-10-16T14:10:43.243482+0000","last_clean":"2023-10-16T14:10:43.243482+0000","last_became_active":"2023-10-16T14:10:40.991364+0000","last_became_peered":"2023-10-16T14:10:40.991364+0000","last_unstale":"2023-10-16T14:10:43.243482+0000","last_undegraded":"2023-10-16T14:10:43.243482+0000","last_fullsized":"2023-10-16T14:10:43.243482+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:30.238400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971730+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:40.971730+0000","last_peered":"2023-10-16T14:10:40.971730+0000","last_clean":"2023-10-16T14:10:40.971730+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:40.971730+0000","last_undegraded":"2023-10-16T14:10:40.971730+0000","last_fullsized":"2023-10-16T14:10:40.971730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244638+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:43.244638+0000","last_peered":"2023-10-16T14:10:43.244638+0000","last_clean":"2023-10-16T14:10:43.244638+0000","last_became_active":"2023-10-16T14:10:40.992264+0000","last_became_peered":"2023-10-16T14:10:40.992264+0000","last_unstale":"2023-10-16T14:10:43.244638+0000","last_undegraded":"2023-10-16T14:10:43.244638+0000","last_fullsized":"2023-10-16T14:10:43.244638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:32.614992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967372+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:40.967372+0000","last_peered":"2023-10-16T14:10:40.967372+0000","last_clean":"2023-10-16T14:10:40.967372+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:40.967372+0000","last_undegraded":"2023-10-16T14:10:40.967372+0000","last_fullsized":"2023-10-16T14:10:40.967372+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242825+0000","last_change":"2023-10-16T14:10:40.989459+0000","last_active":"2023-10-16T14:10:43.242825+0000","last_peered":"2023-10-16T14:10:43.242825+0000","last_clean":"2023-10-16T14:10:43.242825+0000","last_became_active":"2023-10-16T14:10:40.989197+0000","last_became_peered":"2023-10-16T14:10:40.989197+0000","last_unstale":"2023-10-16T14:10:43.242825+0000","last_undegraded":"2023-10-16T14:10:43.242825+0000","last_fullsized":"2023-10-16T14:10:43.242825+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979422+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:40.979422+0000","last_peered":"2023-10-16T14:10:40.979422+0000","last_clean":"2023-10-16T14:10:40.979422+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:40.979422+0000","last_undegraded":"2023-10-16T14:10:40.979422+0000","last_fullsized":"2023-10-16T14:10:40.979422+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994059+0000","last_change":"2023-10-16T14:10:40.986689+0000","last_active":"2023-10-16T14:10:41.994059+0000","last_peered":"2023-10-16T14:10:41.994059+0000","last_clean":"2023-10-16T14:10:41.994059+0000","last_became_active":"2023-10-16T14:10:40.986408+0000","last_became_peered":"2023-10-16T14:10:40.986408+0000","last_unstale":"2023-10-16T14:10:41.994059+0000","last_undegraded":"2023-10-16T14:10:41.994059+0000","last_fullsized":"2023-10-16T14:10:41.994059+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975431+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:40.975431+0000","last_peered":"2023-10-16T14:10:40.975431+0000","last_clean":"2023-10-16T14:10:40.975431+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:40.975431+0000","last_undegraded":"2023-10-16T14:10:40.975431+0000","last_fullsized":"2023-10-16T14:10:40.975431+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989882+0000","last_change":"2023-10-16T14:10:40.987586+0000","last_active":"2023-10-16T14:10:41.989882+0000","last_peered":"2023-10-16T14:10:41.989882+0000","last_clean":"2023-10-16T14:10:41.989882+0000","last_became_active":"2023-10-16T14:10:40.987254+0000","last_became_peered":"2023-10-16T14:10:40.987254+0000","last_unstale":"2023-10-16T14:10:41.989882+0000","last_undegraded":"2023-10-16T14:10:41.989882+0000","last_fullsized":"2023-10-16T14:10:41.989882+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.463597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980939+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:40.980939+0000","last_peered":"2023-10-16T14:10:40.980939+0000","last_clean":"2023-10-16T14:10:40.980939+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:40.980939+0000","last_undegraded":"2023-10-16T14:10:40.980939+0000","last_fullsized":"2023-10-16T14:10:40.980939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124331+0000","last_change":"2023-10-16T14:10:40.990967+0000","last_active":"2023-10-16T14:10:42.124331+0000","last_peered":"2023-10-16T14:10:42.124331+0000","last_clean":"2023-10-16T14:10:42.124331+0000","last_became_active":"2023-10-16T14:10:40.989709+0000","last_became_peered":"2023-10-16T14:10:40.989709+0000","last_unstale":"2023-10-16T14:10:42.124331+0000","last_undegraded":"2023-10-16T14:10:42.124331+0000","last_fullsized":"2023-10-16T14:10:42.124331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.888217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974374+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:40.974374+0000","last_peered":"2023-10-16T14:10:40.974374+0000","last_clean":"2023-10-16T14:10:40.974374+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:40.974374+0000","last_undegraded":"2023-10-16T14:10:40.974374+0000","last_fullsized":"2023-10-16T14:10:40.974374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244140+0000","last_change":"2023-10-16T14:10:40.985097+0000","last_active":"2023-10-16T14:10:43.244140+0000","last_peered":"2023-10-16T14:10:43.244140+0000","last_clean":"2023-10-16T14:10:43.244140+0000","last_became_active":"2023-10-16T14:10:40.984409+0000","last_became_peered":"2023-10-16T14:10:40.984409+0000","last_unstale":"2023-10-16T14:10:43.244140+0000","last_undegraded":"2023-10-16T14:10:43.244140+0000","last_fullsized":"2023-10-16T14:10:43.244140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972580+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:40.972580+0000","last_peered":"2023-10-16T14:10:40.972580+0000","last_clean":"2023-10-16T14:10:40.972580+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:40.972580+0000","last_undegraded":"2023-10-16T14:10:40.972580+0000","last_fullsized":"2023-10-16T14:10:40.972580+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996436+0000","last_change":"2023-10-16T14:10:40.988348+0000","last_active":"2023-10-16T14:10:41.996436+0000","last_peered":"2023-10-16T14:10:41.996436+0000","last_clean":"2023-10-16T14:10:41.996436+0000","last_became_active":"2023-10-16T14:10:40.988060+0000","last_became_peered":"2023-10-16T14:10:40.988060+0000","last_unstale":"2023-10-16T14:10:41.996436+0000","last_undegraded":"2023-10-16T14:10:41.996436+0000","last_fullsized":"2023-10-16T14:10:41.996436+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.988939+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:40.988939+0000","last_peered":"2023-10-16T14:10:40.988939+0000","last_clean":"2023-10-16T14:10:40.988939+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:40.988939+0000","last_undegraded":"2023-10-16T14:10:40.988939+0000","last_fullsized":"2023-10-16T14:10:40.988939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243319+0000","last_change":"2023-10-16T14:10:40.987085+0000","last_active":"2023-10-16T14:10:43.243319+0000","last_peered":"2023-10-16T14:10:43.243319+0000","last_clean":"2023-10-16T14:10:43.243319+0000","last_became_active":"2023-10-16T14:10:40.986831+0000","last_became_peered":"2023-10-16T14:10:40.986831+0000","last_unstale":"2023-10-16T14:10:43.243319+0000","last_undegraded":"2023-10-16T14:10:43.243319+0000","last_fullsized":"2023-10-16T14:10:43.243319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:58.536370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987030+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:40.987030+0000","last_peered":"2023-10-16T14:10:40.987030+0000","last_clean":"2023-10-16T14:10:40.987030+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:40.987030+0000","last_undegraded":"2023-10-16T14:10:40.987030+0000","last_fullsized":"2023-10-16T14:10:40.987030+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125634+0000","last_change":"2023-10-16T14:10:40.998279+0000","last_active":"2023-10-16T14:10:42.125634+0000","last_peered":"2023-10-16T14:10:42.125634+0000","last_clean":"2023-10-16T14:10:42.125634+0000","last_became_active":"2023-10-16T14:10:40.997899+0000","last_became_peered":"2023-10-16T14:10:40.997899+0000","last_unstale":"2023-10-16T14:10:42.125634+0000","last_undegraded":"2023-10-16T14:10:42.125634+0000","last_fullsized":"2023-10-16T14:10:42.125634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:02.766931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977306+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:40.977306+0000","last_peered":"2023-10-16T14:10:40.977306+0000","last_clean":"2023-10-16T14:10:40.977306+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:40.977306+0000","last_undegraded":"2023-10-16T14:10:40.977306+0000","last_fullsized":"2023-10-16T14:10:40.977306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.991518+0000","last_change":"2023-10-16T14:10:40.992264+0000","last_active":"2023-10-16T14:10:41.991518+0000","last_peered":"2023-10-16T14:10:41.991518+0000","last_clean":"2023-10-16T14:10:41.991518+0000","last_became_active":"2023-10-16T14:10:40.991698+0000","last_became_peered":"2023-10-16T14:10:40.991698+0000","last_unstale":"2023-10-16T14:10:41.991518+0000","last_undegraded":"2023-10-16T14:10:41.991518+0000","last_fullsized":"2023-10-16T14:10:41.991518+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:16.407382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978952+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:40.978952+0000","last_peered":"2023-10-16T14:10:40.978952+0000","last_clean":"2023-10-16T14:10:40.978952+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:40.978952+0000","last_undegraded":"2023-10-16T14:10:40.978952+0000","last_fullsized":"2023-10-16T14:10:40.978952+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125182+0000","last_change":"2023-10-16T14:10:40.991982+0000","last_active":"2023-10-16T14:10:42.125182+0000","last_peered":"2023-10-16T14:10:42.125182+0000","last_clean":"2023-10-16T14:10:42.125182+0000","last_became_active":"2023-10-16T14:10:40.991521+0000","last_became_peered":"2023-10-16T14:10:40.991521+0000","last_unstale":"2023-10-16T14:10:42.125182+0000","last_undegraded":"2023-10-16T14:10:42.125182+0000","last_fullsized":"2023-10-16T14:10:42.125182+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:01.654954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961133+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:40.961133+0000","last_peered":"2023-10-16T14:10:40.961133+0000","last_clean":"2023-10-16T14:10:40.961133+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:40.961133+0000","last_undegraded":"2023-10-16T14:10:40.961133+0000","last_fullsized":"2023-10-16T14:10:40.961133+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124085+0000","last_change":"2023-10-16T14:10:40.992568+0000","last_active":"2023-10-16T14:10:42.124085+0000","last_peered":"2023-10-16T14:10:42.124085+0000","last_clean":"2023-10-16T14:10:42.124085+0000","last_became_active":"2023-10-16T14:10:40.992252+0000","last_became_peered":"2023-10-16T14:10:40.992252+0000","last_unstale":"2023-10-16T14:10:42.124085+0000","last_undegraded":"2023-10-16T14:10:42.124085+0000","last_fullsized":"2023-10-16T14:10:42.124085+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:25.262205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965644+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:40.965644+0000","last_peered":"2023-10-16T14:10:40.965644+0000","last_clean":"2023-10-16T14:10:40.965644+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:40.965644+0000","last_undegraded":"2023-10-16T14:10:40.965644+0000","last_fullsized":"2023-10-16T14:10:40.965644+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242786+0000","last_change":"2023-10-16T14:10:40.991785+0000","last_active":"2023-10-16T14:10:43.242786+0000","last_peered":"2023-10-16T14:10:43.242786+0000","last_clean":"2023-10-16T14:10:43.242786+0000","last_became_active":"2023-10-16T14:10:40.991292+0000","last_became_peered":"2023-10-16T14:10:40.991292+0000","last_unstale":"2023-10-16T14:10:43.242786+0000","last_undegraded":"2023-10-16T14:10:43.242786+0000","last_fullsized":"2023-10-16T14:10:43.242786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978297+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:40.978297+0000","last_peered":"2023-10-16T14:10:40.978297+0000","last_clean":"2023-10-16T14:10:40.978297+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:40.978297+0000","last_undegraded":"2023-10-16T14:10:40.978297+0000","last_fullsized":"2023-10-16T14:10:40.978297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243833+0000","last_change":"2023-10-16T14:10:40.985314+0000","last_active":"2023-10-16T14:10:43.243833+0000","last_peered":"2023-10-16T14:10:43.243833+0000","last_clean":"2023-10-16T14:10:43.243833+0000","last_became_active":"2023-10-16T14:10:40.984808+0000","last_became_peered":"2023-10-16T14:10:40.984808+0000","last_unstale":"2023-10-16T14:10:43.243833+0000","last_undegraded":"2023-10-16T14:10:43.243833+0000","last_fullsized":"2023-10-16T14:10:43.243833+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959800+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:40.959800+0000","last_peered":"2023-10-16T14:10:40.959800+0000","last_clean":"2023-10-16T14:10:40.959800+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:40.959800+0000","last_undegraded":"2023-10-16T14:10:40.959800+0000","last_fullsized":"2023-10-16T14:10:40.959800+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144022+0000","last_change":"2023-10-16T14:10:40.995986+0000","last_active":"2023-10-16T14:10:43.144022+0000","last_peered":"2023-10-16T14:10:43.144022+0000","last_clean":"2023-10-16T14:10:43.144022+0000","last_became_active":"2023-10-16T14:10:40.994817+0000","last_became_peered":"2023-10-16T14:10:40.994817+0000","last_unstale":"2023-10-16T14:10:43.144022+0000","last_undegraded":"2023-10-16T14:10:43.144022+0000","last_fullsized":"2023-10-16T14:10:43.144022+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:17.570998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978097+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:40.978097+0000","last_peered":"2023-10-16T14:10:40.978097+0000","last_clean":"2023-10-16T14:10:40.978097+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:40.978097+0000","last_undegraded":"2023-10-16T14:10:40.978097+0000","last_fullsized":"2023-10-16T14:10:40.978097+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997726+0000","last_change":"2023-10-16T14:10:41.001565+0000","last_active":"2023-10-16T14:10:41.997726+0000","last_peered":"2023-10-16T14:10:41.997726+0000","last_clean":"2023-10-16T14:10:41.997726+0000","last_became_active":"2023-10-16T14:10:40.999604+0000","last_became_peered":"2023-10-16T14:10:40.999604+0000","last_unstale":"2023-10-16T14:10:41.997726+0000","last_undegraded":"2023-10-16T14:10:41.997726+0000","last_fullsized":"2023-10-16T14:10:41.997726+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980317+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:40.980317+0000","last_peered":"2023-10-16T14:10:40.980317+0000","last_clean":"2023-10-16T14:10:40.980317+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:40.980317+0000","last_undegraded":"2023-10-16T14:10:40.980317+0000","last_fullsized":"2023-10-16T14:10:40.980317+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243931+0000","last_change":"2023-10-16T14:10:40.994995+0000","last_active":"2023-10-16T14:10:43.243931+0000","last_peered":"2023-10-16T14:10:43.243931+0000","last_clean":"2023-10-16T14:10:43.243931+0000","last_became_active":"2023-10-16T14:10:40.994421+0000","last_became_peered":"2023-10-16T14:10:40.994421+0000","last_unstale":"2023-10-16T14:10:43.243931+0000","last_undegraded":"2023-10-16T14:10:43.243931+0000","last_fullsized":"2023-10-16T14:10:43.243931+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:39.108794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968652+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:40.968652+0000","last_peered":"2023-10-16T14:10:40.968652+0000","last_clean":"2023-10-16T14:10:40.968652+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:40.968652+0000","last_undegraded":"2023-10-16T14:10:40.968652+0000","last_fullsized":"2023-10-16T14:10:40.968652+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995533+0000","last_change":"2023-10-16T14:10:40.988474+0000","last_active":"2023-10-16T14:10:41.995533+0000","last_peered":"2023-10-16T14:10:41.995533+0000","last_clean":"2023-10-16T14:10:41.995533+0000","last_became_active":"2023-10-16T14:10:40.988193+0000","last_became_peered":"2023-10-16T14:10:40.988193+0000","last_unstale":"2023-10-16T14:10:41.995533+0000","last_undegraded":"2023-10-16T14:10:41.995533+0000","last_fullsized":"2023-10-16T14:10:41.995533+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972255+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:40.972255+0000","last_peered":"2023-10-16T14:10:40.972255+0000","last_clean":"2023-10-16T14:10:40.972255+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:40.972255+0000","last_undegraded":"2023-10-16T14:10:40.972255+0000","last_fullsized":"2023-10-16T14:10:40.972255+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994820+0000","last_change":"2023-10-16T14:10:40.994079+0000","last_active":"2023-10-16T14:10:41.994820+0000","last_peered":"2023-10-16T14:10:41.994820+0000","last_clean":"2023-10-16T14:10:41.994820+0000","last_became_active":"2023-10-16T14:10:40.991884+0000","last_became_peered":"2023-10-16T14:10:40.991884+0000","last_unstale":"2023-10-16T14:10:41.994820+0000","last_undegraded":"2023-10-16T14:10:41.994820+0000","last_fullsized":"2023-10-16T14:10:41.994820+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:14.260538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981697+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:40.981697+0000","last_peered":"2023-10-16T14:10:40.981697+0000","last_clean":"2023-10-16T14:10:40.981697+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:40.981697+0000","last_undegraded":"2023-10-16T14:10:40.981697+0000","last_fullsized":"2023-10-16T14:10:40.981697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.993382+0000","last_change":"2023-10-16T14:10:41.002135+0000","last_active":"2023-10-16T14:10:41.993382+0000","last_peered":"2023-10-16T14:10:41.993382+0000","last_clean":"2023-10-16T14:10:41.993382+0000","last_became_active":"2023-10-16T14:10:41.000234+0000","last_became_peered":"2023-10-16T14:10:41.000234+0000","last_unstale":"2023-10-16T14:10:41.993382+0000","last_undegraded":"2023-10-16T14:10:41.993382+0000","last_fullsized":"2023-10-16T14:10:41.993382+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:09.588728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965582+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:40.965582+0000","last_peered":"2023-10-16T14:10:40.965582+0000","last_clean":"2023-10-16T14:10:40.965582+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:40.965582+0000","last_undegraded":"2023-10-16T14:10:40.965582+0000","last_fullsized":"2023-10-16T14:10:40.965582+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989881+0000","last_change":"2023-10-16T14:10:40.987206+0000","last_active":"2023-10-16T14:10:41.989881+0000","last_peered":"2023-10-16T14:10:41.989881+0000","last_clean":"2023-10-16T14:10:41.989881+0000","last_became_active":"2023-10-16T14:10:40.986211+0000","last_became_peered":"2023-10-16T14:10:40.986211+0000","last_unstale":"2023-10-16T14:10:41.989881+0000","last_undegraded":"2023-10-16T14:10:41.989881+0000","last_fullsized":"2023-10-16T14:10:41.989881+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974724+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:40.974724+0000","last_peered":"2023-10-16T14:10:40.974724+0000","last_clean":"2023-10-16T14:10:40.974724+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:40.974724+0000","last_undegraded":"2023-10-16T14:10:40.974724+0000","last_fullsized":"2023-10-16T14:10:40.974724+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996483+0000","last_change":"2023-10-16T14:10:40.998615+0000","last_active":"2023-10-16T14:10:41.996483+0000","last_peered":"2023-10-16T14:10:41.996483+0000","last_clean":"2023-10-16T14:10:41.996483+0000","last_became_active":"2023-10-16T14:10:40.997677+0000","last_became_peered":"2023-10-16T14:10:40.997677+0000","last_unstale":"2023-10-16T14:10:41.996483+0000","last_undegraded":"2023-10-16T14:10:41.996483+0000","last_fullsized":"2023-10-16T14:10:41.996483+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968589+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:40.968589+0000","last_peered":"2023-10-16T14:10:40.968589+0000","last_clean":"2023-10-16T14:10:40.968589+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:40.968589+0000","last_undegraded":"2023-10-16T14:10:40.968589+0000","last_fullsized":"2023-10-16T14:10:40.968589+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244429+0000","last_change":"2023-10-16T14:10:41.001901+0000","last_active":"2023-10-16T14:10:43.244429+0000","last_peered":"2023-10-16T14:10:43.244429+0000","last_clean":"2023-10-16T14:10:43.244429+0000","last_became_active":"2023-10-16T14:10:41.001390+0000","last_became_peered":"2023-10-16T14:10:41.001390+0000","last_unstale":"2023-10-16T14:10:43.244429+0000","last_undegraded":"2023-10-16T14:10:43.244429+0000","last_fullsized":"2023-10-16T14:10:43.244429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:38.275860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977891+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:40.977891+0000","last_peered":"2023-10-16T14:10:40.977891+0000","last_clean":"2023-10-16T14:10:40.977891+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:40.977891+0000","last_undegraded":"2023-10-16T14:10:40.977891+0000","last_fullsized":"2023-10-16T14:10:40.977891+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124855+0000","last_change":"2023-10-16T14:10:40.990936+0000","last_active":"2023-10-16T14:10:42.124855+0000","last_peered":"2023-10-16T14:10:42.124855+0000","last_clean":"2023-10-16T14:10:42.124855+0000","last_became_active":"2023-10-16T14:10:40.990264+0000","last_became_peered":"2023-10-16T14:10:40.990264+0000","last_unstale":"2023-10-16T14:10:42.124855+0000","last_undegraded":"2023-10-16T14:10:42.124855+0000","last_fullsized":"2023-10-16T14:10:42.124855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:39.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973916+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:40.973916+0000","last_peered":"2023-10-16T14:10:40.973916+0000","last_clean":"2023-10-16T14:10:40.973916+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:40.973916+0000","last_undegraded":"2023-10-16T14:10:40.973916+0000","last_fullsized":"2023-10-16T14:10:40.973916+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143331+0000","last_change":"2023-10-16T14:10:40.998298+0000","last_active":"2023-10-16T14:10:43.143331+0000","last_peered":"2023-10-16T14:10:43.143331+0000","last_clean":"2023-10-16T14:10:43.143331+0000","last_became_active":"2023-10-16T14:10:40.997963+0000","last_became_peered":"2023-10-16T14:10:40.997963+0000","last_unstale":"2023-10-16T14:10:43.143331+0000","last_undegraded":"2023-10-16T14:10:43.143331+0000","last_fullsized":"2023-10-16T14:10:43.143331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:12.160466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973460+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:40.973460+0000","last_peered":"2023-10-16T14:10:40.973460+0000","last_clean":"2023-10-16T14:10:40.973460+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:40.973460+0000","last_undegraded":"2023-10-16T14:10:40.973460+0000","last_fullsized":"2023-10-16T14:10:40.973460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.241703+0000","last_change":"2023-10-16T14:10:40.982851+0000","last_active":"2023-10-16T14:10:43.241703+0000","last_peered":"2023-10-16T14:10:43.241703+0000","last_clean":"2023-10-16T14:10:43.241703+0000","last_became_active":"2023-10-16T14:10:40.981261+0000","last_became_peered":"2023-10-16T14:10:40.981261+0000","last_unstale":"2023-10-16T14:10:43.241703+0000","last_undegraded":"2023-10-16T14:10:43.241703+0000","last_fullsized":"2023-10-16T14:10:43.241703+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:03.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982369+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:40.982369+0000","last_peered":"2023-10-16T14:10:40.982369+0000","last_clean":"2023-10-16T14:10:40.982369+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:40.982369+0000","last_undegraded":"2023-10-16T14:10:40.982369+0000","last_fullsized":"2023-10-16T14:10:40.982369+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125338+0000","last_change":"2023-10-16T14:10:40.991807+0000","last_active":"2023-10-16T14:10:42.125338+0000","last_peered":"2023-10-16T14:10:42.125338+0000","last_clean":"2023-10-16T14:10:42.125338+0000","last_became_active":"2023-10-16T14:10:40.991143+0000","last_became_peered":"2023-10-16T14:10:40.991143+0000","last_unstale":"2023-10-16T14:10:42.125338+0000","last_undegraded":"2023-10-16T14:10:42.125338+0000","last_fullsized":"2023-10-16T14:10:42.125338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.405248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981437+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:40.981437+0000","last_peered":"2023-10-16T14:10:40.981437+0000","last_clean":"2023-10-16T14:10:40.981437+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:40.981437+0000","last_undegraded":"2023-10-16T14:10:40.981437+0000","last_fullsized":"2023-10-16T14:10:40.981437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185397+0000","last_change":"2023-10-16T14:10:40.997929+0000","last_active":"2023-10-16T14:10:43.185397+0000","last_peered":"2023-10-16T14:10:43.185397+0000","last_clean":"2023-10-16T14:10:43.185397+0000","last_became_active":"2023-10-16T14:10:40.997578+0000","last_became_peered":"2023-10-16T14:10:40.997578+0000","last_unstale":"2023-10-16T14:10:43.185397+0000","last_undegraded":"2023-10-16T14:10:43.185397+0000","last_fullsized":"2023-10-16T14:10:43.185397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:31.595742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968740+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:40.968740+0000","last_peered":"2023-10-16T14:10:40.968740+0000","last_clean":"2023-10-16T14:10:40.968740+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:40.968740+0000","last_undegraded":"2023-10-16T14:10:40.968740+0000","last_fullsized":"2023-10-16T14:10:40.968740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.141865+0000","last_change":"2023-10-16T14:10:40.982299+0000","last_active":"2023-10-16T14:10:43.141865+0000","last_peered":"2023-10-16T14:10:43.141865+0000","last_clean":"2023-10-16T14:10:43.141865+0000","last_became_active":"2023-10-16T14:10:40.982047+0000","last_became_peered":"2023-10-16T14:10:40.982047+0000","last_unstale":"2023-10-16T14:10:43.141865+0000","last_undegraded":"2023-10-16T14:10:43.141865+0000","last_fullsized":"2023-10-16T14:10:43.141865+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:09.508655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978187+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:40.978187+0000","last_peered":"2023-10-16T14:10:40.978187+0000","last_clean":"2023-10-16T14:10:40.978187+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:40.978187+0000","last_undegraded":"2023-10-16T14:10:40.978187+0000","last_fullsized":"2023-10-16T14:10:40.978187+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543301,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3551232,"data_stored":2031342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":1.4139999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.589}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.4730000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.327}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.087},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.3779999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.228},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.5329999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543302,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42388,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329452,"statfs":{"total":96636764160,"available":96593358848,"internally_reserved":0,"allocated":2953216,"data_stored":1438685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":1.788},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":0.86299999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.042},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.6779999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.3879999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.089}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":2.2749999999999999},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":2.0990000000000002}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.45}]}]},{"osd":5,"up_from":14,"seq":60129543303,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":2945024,"data_stored":1438629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.02}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.1799999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.254}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.286},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.8080000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.9590000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.4590000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47968,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323872,"statfs":{"total":96636764160,"available":96587644928,"internally_reserved":0,"allocated":2965504,"data_stored":1441590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.8420000000000001},"last":1.306},{"interface":"front","average":{"1min":1.121,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6699999999999999,"5min":1.8160000000000001,"15min":1.9019999999999999},"last":1.3700000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.78},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2549999999999999,"15min":1.294},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.754,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.5489999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.286,"15min":1.3169999999999999},"min":{"1min":0.80200000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.714,"5min":1.714,"15min":1.865},"last":1.026},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.367,"15min":1.387},"min":{"1min":1.026,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.476}]},{"osd":3,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.296,"15min":1.3120000000000001},"min":{"1min":0.94599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.615,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.607,"5min":1.4590000000000001,"15min":1.4299999999999999},"min":{"1min":0.96499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":5.2350000000000003,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.427}]},{"osd":5,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.329,"15min":1.3400000000000001},"min":{"1min":0.99199999999999999,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.615},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3580000000000001,"15min":1.3360000000000001},"min":{"1min":1.0389999999999999,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":1.921,"15min":1.921},"last":1.0740000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3919999999999999,"15min":1.3879999999999999},"min":{"1min":1.004,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.7689999999999999,"5min":1.9159999999999999,"15min":1.956},"last":1.149},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.502,"15min":1.5229999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.843}]},{"osd":7,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5349999999999999,"15min":1.5049999999999999},"min":{"1min":1.137,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.1949999999999998},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6379999999999999,"15min":1.6319999999999999},"min":{"1min":1.391,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":1.889,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.575}]}]},{"osd":3,"up_from":14,"seq":60129543303,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52128,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319712,"statfs":{"total":96636764160,"available":96583385088,"internally_reserved":0,"allocated":2969600,"data_stored":1439911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":1.05}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":4.1299999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":3.8340000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":0.89300000000000002}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.01}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":2.2799999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":2.569}]}]},{"osd":2,"up_from":14,"seq":60129543301,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":2949120,"data_stored":1432548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":1.337},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.115},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.2290000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":1.4159999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.268},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.0469999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.103},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":1.397}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.1899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543301,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2953216,"data_stored":1438193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.0740000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":0.752},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.1870000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":1.532}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":1.454},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":2.0979999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.867},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.7969999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543302,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":3547136,"data_stored":2029177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.93300000000000005,"15min":0.93999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.218,"5min":1.704,"15min":1.891},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.1579999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48699999999999999},"max":{"1min":1.627,"5min":1.659,"15min":1.9299999999999999},"last":0.90700000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.1499999999999999,"15min":1.171},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5349999999999999,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.1899999999999999,"15min":1.226},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.6539999999999999,"5min":1.893,"15min":1.893},"last":1.0760000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1639999999999999,"15min":1.163},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7390000000000001,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3380000000000001,"15min":1.3759999999999999},"min":{"1min":0.64700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.679,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.2969999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3340000000000001,"15min":1.3799999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6140000000000001,"5min":1.819,"15min":2.048},"last":1.002},{"interface":"front","average":{"1min":1.379,"5min":1.518,"15min":1.524},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.0009999999999999,"5min":2.601,"15min":2.601},"last":1.4850000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.48,"15min":1.5069999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.821,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.452,"15min":1.492},"min":{"1min":0.98199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0169999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.593,"15min":1.621},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9490000000000001,"5min":2.669,"15min":2.669},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.55,"15min":1.5980000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2589999999999999,"5min":2.355,"15min":2.3769999999999998},"last":1.778}]},{"osd":7,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.655,"15min":1.712},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1549999999999998,"5min":2.484,"15min":2.5270000000000001},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.625,"15min":1.649},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.464},"last":1.056}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:50.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:10:50.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.978+0000 7efd54d89640 1 Processor -- start 2023-10-16T14:10:50.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.978+0000 7efd54d89640 1 -- start start 2023-10-16T14:10:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.979+0000 7efd54d89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 0x7efd5015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd54d89640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5015ed60 con 0x7efd500ce390 2023-10-16T14:10:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd54d89640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd500707e0 con 0x7efd5015aaa0 2023-10-16T14:10:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd54d89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5006b810 con 0x7efd5015ae80 2023-10-16T14:10:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd4ed9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd500ce390 0x7efd50070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52594/0 (socket says 172.21.15.60:52594) 2023-10-16T14:10:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd4ed9d640 1 -- 172.21.15.60:0/3568479122 learned_addr learned my addr 172.21.15.60:0/3568479122 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd4dd9b640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 0x7efd5015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942594456 0 0) 0x7efd5015ed60 con 0x7efd500ce390 2023-10-16T14:10:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.980+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c003520 con 0x7efd500ce390 2023-10-16T14:10:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1894887212 0 0) 0x7efd3c003520 con 0x7efd500ce390 2023-10-16T14:10:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 msgr2=0x7efd5015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 0x7efd5015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 >> v1:172.21.15.144:6789/0 conn(0x7efd5015aaa0 legacy=0x7efd500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5015ef90 con 0x7efd500ce390 2023-10-16T14:10:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efd48001670 con 0x7efd500ce390 2023-10-16T14:10:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 >> v1:172.21.15.60:6789/0 conn(0x7efd500ce390 legacy=0x7efd50070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd3c0039a0 0x7efd3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd3c0047c0 0x7efd3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 0x7efd3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5006b810 con 0x7efd3c0039a0 2023-10-16T14:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c00ba60 con 0x7efd3c0047c0 2023-10-16T14:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c00bc20 con 0x7efd5015ae80 2023-10-16T14:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.981+0000 7efd4e59c640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd3c0039a0 0x7efd3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:50.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.982+0000 7efd4e59c640 1 -- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 msgr2=0x7efd3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.982+0000 7efd4e59c640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5015ae80 0x7efd3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.982+0000 7efd4e59c640 1 -- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd3c0047c0 msgr2=0x7efd3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.982+0000 7efd4e59c640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd3c0047c0 0x7efd3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.982+0000 7efd4e59c640 1 -- 172.21.15.60:0/3568479122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c003520 con 0x7efd3c0039a0 2023-10-16T14:10:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.983+0000 7efd4e59c640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd3c0039a0 0x7efd3c003dc0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7efd40001e20 tx=0x7efd3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb73c7ae4ec36c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.983+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40003b70 con 0x7efd3c0039a0 2023-10-16T14:10:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.983+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd40003cd0 con 0x7efd3c0039a0 2023-10-16T14:10:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.983+0000 7efd4d59a640 1 -- 172.21.15.60:0/3568479122 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40005310 con 0x7efd3c0039a0 2023-10-16T14:10:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd54d89640 1 -- 172.21.15.60:0/3568479122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd3c0039a0 msgr2=0x7efd3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd54d89640 1 --2- 172.21.15.60:0/3568479122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd3c0039a0 0x7efd3c003dc0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7efd40001e20 tx=0x7efd3c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:10:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd4ed9d640 1 -- 172.21.15.60:0/3568479122 reap_dead start 2023-10-16T14:10:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd54d89640 1 -- 172.21.15.60:0/3568479122 shutdown_connections 2023-10-16T14:10:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd54d89640 1 -- 172.21.15.60:0/3568479122 >> 172.21.15.60:0/3568479122 conn(0x7efd500cb410 msgr2=0x7efd500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd54d89640 1 -- 172.21.15.60:0/3568479122 shutdown_connections 2023-10-16T14:10:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.984+0000 7efd54d89640 1 -- 172.21.15.60:0/3568479122 wait complete. 2023-10-16T14:10:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.986+0000 7efd54d89640 1 Processor -- start 2023-10-16T14:10:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.986+0000 7efd54d89640 1 -- start start 2023-10-16T14:10:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.987+0000 7efd54d89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd3c0039a0 0x7efd5014fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.987+0000 7efd54d89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5015cf60 0x7efd500b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.987+0000 7efd54d89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 0x7efd5015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.987+0000 7efd54d89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500b2a20 con 0x7efd500b25a0 2023-10-16T14:10:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.987+0000 7efd54d89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd50157ff0 con 0x7efd5015cf60 2023-10-16T14:10:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.987+0000 7efd54d89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5015bec0 con 0x7efd3c0039a0 2023-10-16T14:10:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 0x7efd5015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 0x7efd5015b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57720/0 (socket says 172.21.15.60:57720) 2023-10-16T14:10:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 -- 172.21.15.60:0/3511939208 learned_addr learned my addr 172.21.15.60:0/3511939208 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4dd9b640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5015cf60 0x7efd500b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 -- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd3c0039a0 msgr2=0x7efd5014fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd3c0039a0 0x7efd5014fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 -- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5015cf60 msgr2=0x7efd500b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5015cf60 0x7efd500b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 -- 172.21.15.60:0/3511939208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5015c1a0 con 0x7efd500b25a0 2023-10-16T14:10:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4dd9b640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5015cf60 0x7efd500b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.988+0000 7efd4ed9d640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 0x7efd5015b2e0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7efd500ce2e0 tx=0x7efd4800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.989+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48007440 con 0x7efd500b25a0 2023-10-16T14:10:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.989+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd480075c0 con 0x7efd500b25a0 2023-10-16T14:10:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.989+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48004a20 con 0x7efd500b25a0 2023-10-16T14:10:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.989+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5015c480 con 0x7efd500b25a0 2023-10-16T14:10:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.989+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5015ca40 con 0x7efd500b25a0 2023-10-16T14:10:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.990+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5015cc60 con 0x7efd500b25a0 2023-10-16T14:10:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.991+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd48004b80 con 0x7efd500b25a0 2023-10-16T14:10:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.992+0000 7efd46ffd640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efd3006af90 0x7efd3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.992+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7efd48089080 con 0x7efd500b25a0 2023-10-16T14:10:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.993+0000 7efd4e59c640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efd3006af90 0x7efd3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.994+0000 7efd4e59c640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efd3006af90 0x7efd3006d450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd40034000 tx=0x7efd40030000 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:50.996+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd48053c30 con 0x7efd500b25a0 2023-10-16T14:10:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.190+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd5015f550 con 0x7efd3006af90 2023-10-16T14:10:51.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.228+0000 7efd46ffd640 1 -- 172.21.15.60:0/3511939208 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525695 (secure 0 0 0) 0x7efd40003a50 con 0x7efd3006af90 2023-10-16T14:10:51.241 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:10:51.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.239+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efd3006af90 msgr2=0x7efd3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.239+0000 7efd54d89640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efd3006af90 0x7efd3006d450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd40034000 tx=0x7efd40030000 comp rx=0 tx=0).stop 2023-10-16T14:10:51.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.239+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 msgr2=0x7efd5015b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.239+0000 7efd54d89640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 0x7efd5015b2e0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7efd500ce2e0 tx=0x7efd4800c930 comp rx=0 tx=0).stop 2023-10-16T14:10:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 shutdown_connections 2023-10-16T14:10:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7efd3006af90 0x7efd3006d450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500b25a0 0x7efd5015b2e0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5015cf60 0x7efd500b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 --2- 172.21.15.60:0/3511939208 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd3c0039a0 0x7efd5014fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 >> 172.21.15.60:0/3511939208 conn(0x7efd500cb410 msgr2=0x7efd500b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 shutdown_connections 2023-10-16T14:10:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.240+0000 7efd54d89640 1 -- 172.21.15.60:0/3511939208 wait complete. 2023-10-16T14:10:51.270 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2023-10-16T14:10:50.185681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":675,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":431,"ondisk_log_size":431,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":377860,"kb_used_data":24264,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754596860,"statfs":{"total":773094113280,"available":772707184640,"internally_reserved":0,"allocated":24846336,"data_stored":12697242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":361942584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.229825"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244206+0000","last_change":"2023-10-16T14:10:40.992171+0000","last_active":"2023-10-16T14:10:43.244206+0000","last_peered":"2023-10-16T14:10:43.244206+0000","last_clean":"2023-10-16T14:10:43.244206+0000","last_became_active":"2023-10-16T14:10:40.991865+0000","last_became_peered":"2023-10-16T14:10:40.991865+0000","last_unstale":"2023-10-16T14:10:43.244206+0000","last_undegraded":"2023-10-16T14:10:43.244206+0000","last_fullsized":"2023-10-16T14:10:43.244206+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:36.935225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"219'351","reported_seq":1226,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974179+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:10:40.974179+0000","last_peered":"2023-10-16T14:10:40.974179+0000","last_clean":"2023-10-16T14:10:40.974179+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:10:40.974179+0000","last_undegraded":"2023-10-16T14:10:40.974179+0000","last_fullsized":"2023-10-16T14:10:40.974179+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968913+0000","last_change":"2023-10-16T14:10:25.816125+0000","last_active":"2023-10-16T14:10:40.968913+0000","last_peered":"2023-10-16T14:10:40.968913+0000","last_clean":"2023-10-16T14:10:40.968913+0000","last_became_active":"2023-10-16T14:10:25.815475+0000","last_became_peered":"2023-10-16T14:10:25.815475+0000","last_unstale":"2023-10-16T14:10:40.968913+0000","last_undegraded":"2023-10-16T14:10:40.968913+0000","last_fullsized":"2023-10-16T14:10:40.968913+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:07.545588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980445+0000","last_change":"2023-10-16T14:10:26.837956+0000","last_active":"2023-10-16T14:10:40.980445+0000","last_peered":"2023-10-16T14:10:40.980445+0000","last_clean":"2023-10-16T14:10:40.980445+0000","last_became_active":"2023-10-16T14:10:26.837665+0000","last_became_peered":"2023-10-16T14:10:26.837665+0000","last_unstale":"2023-10-16T14:10:40.980445+0000","last_undegraded":"2023-10-16T14:10:40.980445+0000","last_fullsized":"2023-10-16T14:10:40.980445+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:08.744930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008653+0000","last_change":"2023-10-16T14:10:39.986399+0000","last_active":"2023-10-16T14:10:41.008653+0000","last_peered":"2023-10-16T14:10:41.008653+0000","last_clean":"2023-10-16T14:10:41.008653+0000","last_became_active":"2023-10-16T14:10:39.986098+0000","last_became_peered":"2023-10-16T14:10:39.986098+0000","last_unstale":"2023-10-16T14:10:41.008653+0000","last_undegraded":"2023-10-16T14:10:41.008653+0000","last_fullsized":"2023-10-16T14:10:41.008653+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:15.423316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144154+0000","last_change":"2023-10-16T14:10:40.991925+0000","last_active":"2023-10-16T14:10:43.144154+0000","last_peered":"2023-10-16T14:10:43.144154+0000","last_clean":"2023-10-16T14:10:43.144154+0000","last_became_active":"2023-10-16T14:10:40.991667+0000","last_became_peered":"2023-10-16T14:10:40.991667+0000","last_unstale":"2023-10-16T14:10:43.144154+0000","last_undegraded":"2023-10-16T14:10:43.144154+0000","last_fullsized":"2023-10-16T14:10:43.144154+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:25.199441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977915+0000","last_change":"2023-10-16T14:10:25.819745+0000","last_active":"2023-10-16T14:10:40.977915+0000","last_peered":"2023-10-16T14:10:40.977915+0000","last_clean":"2023-10-16T14:10:40.977915+0000","last_became_active":"2023-10-16T14:10:25.819457+0000","last_became_peered":"2023-10-16T14:10:25.819457+0000","last_unstale":"2023-10-16T14:10:40.977915+0000","last_undegraded":"2023-10-16T14:10:40.977915+0000","last_fullsized":"2023-10-16T14:10:40.977915+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:49.530642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970555+0000","last_change":"2023-10-16T14:10:26.848302+0000","last_active":"2023-10-16T14:10:40.970555+0000","last_peered":"2023-10-16T14:10:40.970555+0000","last_clean":"2023-10-16T14:10:40.970555+0000","last_became_active":"2023-10-16T14:10:26.848005+0000","last_became_peered":"2023-10-16T14:10:26.848005+0000","last_unstale":"2023-10-16T14:10:40.970555+0000","last_undegraded":"2023-10-16T14:10:40.970555+0000","last_fullsized":"2023-10-16T14:10:40.970555+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:02.318662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"235'1","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244227+0000","last_change":"2023-10-16T14:10:39.988566+0000","last_active":"2023-10-16T14:10:43.244227+0000","last_peered":"2023-10-16T14:10:43.244227+0000","last_clean":"2023-10-16T14:10:43.244227+0000","last_became_active":"2023-10-16T14:10:39.988037+0000","last_became_peered":"2023-10-16T14:10:39.988037+0000","last_unstale":"2023-10-16T14:10:43.244227+0000","last_undegraded":"2023-10-16T14:10:43.244227+0000","last_fullsized":"2023-10-16T14:10:43.244227+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:46.694200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125491+0000","last_change":"2023-10-16T14:10:40.987050+0000","last_active":"2023-10-16T14:10:42.125491+0000","last_peered":"2023-10-16T14:10:42.125491+0000","last_clean":"2023-10-16T14:10:42.125491+0000","last_became_active":"2023-10-16T14:10:40.986733+0000","last_became_peered":"2023-10-16T14:10:40.986733+0000","last_unstale":"2023-10-16T14:10:42.125491+0000","last_undegraded":"2023-10-16T14:10:42.125491+0000","last_fullsized":"2023-10-16T14:10:42.125491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:53.222315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.989242+0000","last_change":"2023-10-16T14:10:25.815834+0000","last_active":"2023-10-16T14:10:40.989242+0000","last_peered":"2023-10-16T14:10:40.989242+0000","last_clean":"2023-10-16T14:10:40.989242+0000","last_became_active":"2023-10-16T14:10:25.814487+0000","last_became_peered":"2023-10-16T14:10:25.814487+0000","last_unstale":"2023-10-16T14:10:40.989242+0000","last_undegraded":"2023-10-16T14:10:40.989242+0000","last_fullsized":"2023-10-16T14:10:40.989242+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:30.663946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975719+0000","last_change":"2023-10-16T14:10:26.847814+0000","last_active":"2023-10-16T14:10:40.975719+0000","last_peered":"2023-10-16T14:10:40.975719+0000","last_clean":"2023-10-16T14:10:40.975719+0000","last_became_active":"2023-10-16T14:10:26.847113+0000","last_became_peered":"2023-10-16T14:10:26.847113+0000","last_unstale":"2023-10-16T14:10:40.975719+0000","last_undegraded":"2023-10-16T14:10:40.975719+0000","last_fullsized":"2023-10-16T14:10:40.975719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:02.282135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008672+0000","last_change":"2023-10-16T14:10:39.985490+0000","last_active":"2023-10-16T14:10:41.008672+0000","last_peered":"2023-10-16T14:10:41.008672+0000","last_clean":"2023-10-16T14:10:41.008672+0000","last_became_active":"2023-10-16T14:10:39.985187+0000","last_became_peered":"2023-10-16T14:10:39.985187+0000","last_unstale":"2023-10-16T14:10:41.008672+0000","last_undegraded":"2023-10-16T14:10:41.008672+0000","last_fullsized":"2023-10-16T14:10:41.008672+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:32.423740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125498+0000","last_change":"2023-10-16T14:10:40.989384+0000","last_active":"2023-10-16T14:10:42.125498+0000","last_peered":"2023-10-16T14:10:42.125498+0000","last_clean":"2023-10-16T14:10:42.125498+0000","last_became_active":"2023-10-16T14:10:40.989064+0000","last_became_peered":"2023-10-16T14:10:40.989064+0000","last_unstale":"2023-10-16T14:10:42.125498+0000","last_undegraded":"2023-10-16T14:10:42.125498+0000","last_fullsized":"2023-10-16T14:10:42.125498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:50.164078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974785+0000","last_change":"2023-10-16T14:10:25.813460+0000","last_active":"2023-10-16T14:10:40.974785+0000","last_peered":"2023-10-16T14:10:40.974785+0000","last_clean":"2023-10-16T14:10:40.974785+0000","last_became_active":"2023-10-16T14:10:25.810761+0000","last_became_peered":"2023-10-16T14:10:25.810761+0000","last_unstale":"2023-10-16T14:10:40.974785+0000","last_undegraded":"2023-10-16T14:10:40.974785+0000","last_fullsized":"2023-10-16T14:10:40.974785+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:26.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967632+0000","last_change":"2023-10-16T14:10:26.843389+0000","last_active":"2023-10-16T14:10:40.967632+0000","last_peered":"2023-10-16T14:10:40.967632+0000","last_clean":"2023-10-16T14:10:40.967632+0000","last_became_active":"2023-10-16T14:10:26.843095+0000","last_became_peered":"2023-10-16T14:10:26.843095+0000","last_unstale":"2023-10-16T14:10:40.967632+0000","last_undegraded":"2023-10-16T14:10:40.967632+0000","last_fullsized":"2023-10-16T14:10:40.967632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:02.379979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971923+0000","last_change":"2023-10-16T14:10:39.986732+0000","last_active":"2023-10-16T14:10:40.971923+0000","last_peered":"2023-10-16T14:10:40.971923+0000","last_clean":"2023-10-16T14:10:40.971923+0000","last_became_active":"2023-10-16T14:10:39.986276+0000","last_became_peered":"2023-10-16T14:10:39.986276+0000","last_unstale":"2023-10-16T14:10:40.971923+0000","last_undegraded":"2023-10-16T14:10:40.971923+0000","last_fullsized":"2023-10-16T14:10:40.971923+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:55:52.213467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143312+0000","last_change":"2023-10-16T14:10:40.994651+0000","last_active":"2023-10-16T14:10:43.143312+0000","last_peered":"2023-10-16T14:10:43.143312+0000","last_clean":"2023-10-16T14:10:43.143312+0000","last_became_active":"2023-10-16T14:10:40.994387+0000","last_became_peered":"2023-10-16T14:10:40.994387+0000","last_unstale":"2023-10-16T14:10:43.143312+0000","last_undegraded":"2023-10-16T14:10:43.143312+0000","last_fullsized":"2023-10-16T14:10:43.143312+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:54.855496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977551+0000","last_change":"2023-10-16T14:10:25.821070+0000","last_active":"2023-10-16T14:10:40.977551+0000","last_peered":"2023-10-16T14:10:40.977551+0000","last_clean":"2023-10-16T14:10:40.977551+0000","last_became_active":"2023-10-16T14:10:25.820661+0000","last_became_peered":"2023-10-16T14:10:25.820661+0000","last_unstale":"2023-10-16T14:10:40.977551+0000","last_undegraded":"2023-10-16T14:10:40.977551+0000","last_fullsized":"2023-10-16T14:10:40.977551+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:52.532561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981376+0000","last_change":"2023-10-16T14:10:26.853756+0000","last_active":"2023-10-16T14:10:40.981376+0000","last_peered":"2023-10-16T14:10:40.981376+0000","last_clean":"2023-10-16T14:10:40.981376+0000","last_became_active":"2023-10-16T14:10:26.852090+0000","last_became_peered":"2023-10-16T14:10:26.852090+0000","last_unstale":"2023-10-16T14:10:40.981376+0000","last_undegraded":"2023-10-16T14:10:40.981376+0000","last_fullsized":"2023-10-16T14:10:40.981376+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985469+0000","last_change":"2023-10-16T14:10:39.989343+0000","last_active":"2023-10-16T14:10:40.985469+0000","last_peered":"2023-10-16T14:10:40.985469+0000","last_clean":"2023-10-16T14:10:40.985469+0000","last_became_active":"2023-10-16T14:10:39.989043+0000","last_became_peered":"2023-10-16T14:10:39.989043+0000","last_unstale":"2023-10-16T14:10:40.985469+0000","last_undegraded":"2023-10-16T14:10:40.985469+0000","last_fullsized":"2023-10-16T14:10:40.985469+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:57.519740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144062+0000","last_change":"2023-10-16T14:10:40.993217+0000","last_active":"2023-10-16T14:10:43.144062+0000","last_peered":"2023-10-16T14:10:43.144062+0000","last_clean":"2023-10-16T14:10:43.144062+0000","last_became_active":"2023-10-16T14:10:40.992887+0000","last_became_peered":"2023-10-16T14:10:40.992887+0000","last_unstale":"2023-10-16T14:10:43.144062+0000","last_undegraded":"2023-10-16T14:10:43.144062+0000","last_fullsized":"2023-10-16T14:10:43.144062+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:43:49.772938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979800+0000","last_change":"2023-10-16T14:10:25.808171+0000","last_active":"2023-10-16T14:10:40.979800+0000","last_peered":"2023-10-16T14:10:40.979800+0000","last_clean":"2023-10-16T14:10:40.979800+0000","last_became_active":"2023-10-16T14:10:25.807906+0000","last_became_peered":"2023-10-16T14:10:25.807906+0000","last_unstale":"2023-10-16T14:10:40.979800+0000","last_undegraded":"2023-10-16T14:10:40.979800+0000","last_fullsized":"2023-10-16T14:10:40.979800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987734+0000","last_change":"2023-10-16T14:10:26.853933+0000","last_active":"2023-10-16T14:10:40.987734+0000","last_peered":"2023-10-16T14:10:40.987734+0000","last_clean":"2023-10-16T14:10:40.987734+0000","last_became_active":"2023-10-16T14:10:26.853258+0000","last_became_peered":"2023-10-16T14:10:26.853258+0000","last_unstale":"2023-10-16T14:10:40.987734+0000","last_undegraded":"2023-10-16T14:10:40.987734+0000","last_fullsized":"2023-10-16T14:10:40.987734+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:07.364769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980253+0000","last_change":"2023-10-16T14:10:39.979128+0000","last_active":"2023-10-16T14:10:40.980253+0000","last_peered":"2023-10-16T14:10:40.980253+0000","last_clean":"2023-10-16T14:10:40.980253+0000","last_became_active":"2023-10-16T14:10:39.978782+0000","last_became_peered":"2023-10-16T14:10:39.978782+0000","last_unstale":"2023-10-16T14:10:40.980253+0000","last_undegraded":"2023-10-16T14:10:40.980253+0000","last_fullsized":"2023-10-16T14:10:40.980253+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:28.231837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997313+0000","last_change":"2023-10-16T14:10:40.989722+0000","last_active":"2023-10-16T14:10:41.997313+0000","last_peered":"2023-10-16T14:10:41.997313+0000","last_clean":"2023-10-16T14:10:41.997313+0000","last_became_active":"2023-10-16T14:10:40.989477+0000","last_became_peered":"2023-10-16T14:10:40.989477+0000","last_unstale":"2023-10-16T14:10:41.997313+0000","last_undegraded":"2023-10-16T14:10:41.997313+0000","last_fullsized":"2023-10-16T14:10:41.997313+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971953+0000","last_change":"2023-10-16T14:10:25.816450+0000","last_active":"2023-10-16T14:10:40.971953+0000","last_peered":"2023-10-16T14:10:40.971953+0000","last_clean":"2023-10-16T14:10:40.971953+0000","last_became_active":"2023-10-16T14:10:25.816155+0000","last_became_peered":"2023-10-16T14:10:25.816155+0000","last_unstale":"2023-10-16T14:10:40.971953+0000","last_undegraded":"2023-10-16T14:10:40.971953+0000","last_fullsized":"2023-10-16T14:10:40.971953+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:06.019614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972632+0000","last_change":"2023-10-16T14:10:26.851970+0000","last_active":"2023-10-16T14:10:40.972632+0000","last_peered":"2023-10-16T14:10:40.972632+0000","last_clean":"2023-10-16T14:10:40.972632+0000","last_became_active":"2023-10-16T14:10:26.851693+0000","last_became_peered":"2023-10-16T14:10:26.851693+0000","last_unstale":"2023-10-16T14:10:40.972632+0000","last_undegraded":"2023-10-16T14:10:40.972632+0000","last_fullsized":"2023-10-16T14:10:40.972632+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:02.249572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969118+0000","last_change":"2023-10-16T14:10:39.982849+0000","last_active":"2023-10-16T14:10:40.969118+0000","last_peered":"2023-10-16T14:10:40.969118+0000","last_clean":"2023-10-16T14:10:40.969118+0000","last_became_active":"2023-10-16T14:10:39.982560+0000","last_became_peered":"2023-10-16T14:10:39.982560+0000","last_unstale":"2023-10-16T14:10:40.969118+0000","last_undegraded":"2023-10-16T14:10:40.969118+0000","last_fullsized":"2023-10-16T14:10:40.969118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:51.220017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242743+0000","last_change":"2023-10-16T14:10:40.991901+0000","last_active":"2023-10-16T14:10:43.242743+0000","last_peered":"2023-10-16T14:10:43.242743+0000","last_clean":"2023-10-16T14:10:43.242743+0000","last_became_active":"2023-10-16T14:10:40.991610+0000","last_became_peered":"2023-10-16T14:10:40.991610+0000","last_unstale":"2023-10-16T14:10:43.242743+0000","last_undegraded":"2023-10-16T14:10:43.242743+0000","last_fullsized":"2023-10-16T14:10:43.242743+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"231'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979815+0000","last_change":"2023-10-16T14:10:25.819055+0000","last_active":"2023-10-16T14:10:40.979815+0000","last_peered":"2023-10-16T14:10:40.979815+0000","last_clean":"2023-10-16T14:10:40.979815+0000","last_became_active":"2023-10-16T14:10:25.818738+0000","last_became_peered":"2023-10-16T14:10:25.818738+0000","last_unstale":"2023-10-16T14:10:40.979815+0000","last_undegraded":"2023-10-16T14:10:40.979815+0000","last_fullsized":"2023-10-16T14:10:40.979815+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:59.889955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978358+0000","last_change":"2023-10-16T14:10:26.852481+0000","last_active":"2023-10-16T14:10:40.978358+0000","last_peered":"2023-10-16T14:10:40.978358+0000","last_clean":"2023-10-16T14:10:40.978358+0000","last_became_active":"2023-10-16T14:10:26.850006+0000","last_became_peered":"2023-10-16T14:10:26.850006+0000","last_unstale":"2023-10-16T14:10:40.978358+0000","last_undegraded":"2023-10-16T14:10:40.978358+0000","last_fullsized":"2023-10-16T14:10:40.978358+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:58.808728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967454+0000","last_change":"2023-10-16T14:10:39.986619+0000","last_active":"2023-10-16T14:10:40.967454+0000","last_peered":"2023-10-16T14:10:40.967454+0000","last_clean":"2023-10-16T14:10:40.967454+0000","last_became_active":"2023-10-16T14:10:39.985261+0000","last_became_peered":"2023-10-16T14:10:39.985261+0000","last_unstale":"2023-10-16T14:10:40.967454+0000","last_undegraded":"2023-10-16T14:10:40.967454+0000","last_fullsized":"2023-10-16T14:10:40.967454+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:00.760156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.141717+0000","last_change":"2023-10-16T14:10:40.990315+0000","last_active":"2023-10-16T14:10:43.141717+0000","last_peered":"2023-10-16T14:10:43.141717+0000","last_clean":"2023-10-16T14:10:43.141717+0000","last_became_active":"2023-10-16T14:10:40.990060+0000","last_became_peered":"2023-10-16T14:10:40.990060+0000","last_unstale":"2023-10-16T14:10:43.141717+0000","last_undegraded":"2023-10-16T14:10:43.141717+0000","last_fullsized":"2023-10-16T14:10:43.141717+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:15:15.954735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980892+0000","last_change":"2023-10-16T14:10:25.812213+0000","last_active":"2023-10-16T14:10:40.980892+0000","last_peered":"2023-10-16T14:10:40.980892+0000","last_clean":"2023-10-16T14:10:40.980892+0000","last_became_active":"2023-10-16T14:10:25.811854+0000","last_became_peered":"2023-10-16T14:10:25.811854+0000","last_unstale":"2023-10-16T14:10:40.980892+0000","last_undegraded":"2023-10-16T14:10:40.980892+0000","last_fullsized":"2023-10-16T14:10:40.980892+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:37.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969751+0000","last_change":"2023-10-16T14:10:26.846848+0000","last_active":"2023-10-16T14:10:40.969751+0000","last_peered":"2023-10-16T14:10:40.969751+0000","last_clean":"2023-10-16T14:10:40.969751+0000","last_became_active":"2023-10-16T14:10:26.845086+0000","last_became_peered":"2023-10-16T14:10:26.845086+0000","last_unstale":"2023-10-16T14:10:40.969751+0000","last_undegraded":"2023-10-16T14:10:40.969751+0000","last_fullsized":"2023-10-16T14:10:40.969751+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:23.535748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976136+0000","last_change":"2023-10-16T14:10:39.987697+0000","last_active":"2023-10-16T14:10:40.976136+0000","last_peered":"2023-10-16T14:10:40.976136+0000","last_clean":"2023-10-16T14:10:40.976136+0000","last_became_active":"2023-10-16T14:10:39.987388+0000","last_became_peered":"2023-10-16T14:10:39.987388+0000","last_unstale":"2023-10-16T14:10:40.976136+0000","last_undegraded":"2023-10-16T14:10:40.976136+0000","last_fullsized":"2023-10-16T14:10:40.976136+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:51:46.559129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244071+0000","last_change":"2023-10-16T14:10:40.983968+0000","last_active":"2023-10-16T14:10:43.244071+0000","last_peered":"2023-10-16T14:10:43.244071+0000","last_clean":"2023-10-16T14:10:43.244071+0000","last_became_active":"2023-10-16T14:10:40.983650+0000","last_became_peered":"2023-10-16T14:10:40.983650+0000","last_unstale":"2023-10-16T14:10:43.244071+0000","last_undegraded":"2023-10-16T14:10:43.244071+0000","last_fullsized":"2023-10-16T14:10:43.244071+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.969088+0000","last_change":"2023-10-16T14:10:25.805878+0000","last_active":"2023-10-16T14:10:40.969088+0000","last_peered":"2023-10-16T14:10:40.969088+0000","last_clean":"2023-10-16T14:10:40.969088+0000","last_became_active":"2023-10-16T14:10:25.805589+0000","last_became_peered":"2023-10-16T14:10:25.805589+0000","last_unstale":"2023-10-16T14:10:40.969088+0000","last_undegraded":"2023-10-16T14:10:40.969088+0000","last_fullsized":"2023-10-16T14:10:40.969088+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:59.088791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.989095+0000","last_change":"2023-10-16T14:10:26.846623+0000","last_active":"2023-10-16T14:10:40.989095+0000","last_peered":"2023-10-16T14:10:40.989095+0000","last_clean":"2023-10-16T14:10:40.989095+0000","last_became_active":"2023-10-16T14:10:26.846319+0000","last_became_peered":"2023-10-16T14:10:26.846319+0000","last_unstale":"2023-10-16T14:10:40.989095+0000","last_undegraded":"2023-10-16T14:10:40.989095+0000","last_fullsized":"2023-10-16T14:10:40.989095+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009061+0000","last_change":"2023-10-16T14:10:39.982498+0000","last_active":"2023-10-16T14:10:41.009061+0000","last_peered":"2023-10-16T14:10:41.009061+0000","last_clean":"2023-10-16T14:10:41.009061+0000","last_became_active":"2023-10-16T14:10:39.982162+0000","last_became_peered":"2023-10-16T14:10:39.982162+0000","last_unstale":"2023-10-16T14:10:41.009061+0000","last_undegraded":"2023-10-16T14:10:41.009061+0000","last_fullsized":"2023-10-16T14:10:41.009061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:11.621670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.187140+0000","last_change":"2023-10-16T14:10:40.990434+0000","last_active":"2023-10-16T14:10:43.187140+0000","last_peered":"2023-10-16T14:10:43.187140+0000","last_clean":"2023-10-16T14:10:43.187140+0000","last_became_active":"2023-10-16T14:10:40.990134+0000","last_became_peered":"2023-10-16T14:10:40.990134+0000","last_unstale":"2023-10-16T14:10:43.187140+0000","last_undegraded":"2023-10-16T14:10:43.187140+0000","last_fullsized":"2023-10-16T14:10:43.187140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:35.937011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978483+0000","last_change":"2023-10-16T14:10:25.813339+0000","last_active":"2023-10-16T14:10:40.978483+0000","last_peered":"2023-10-16T14:10:40.978483+0000","last_clean":"2023-10-16T14:10:40.978483+0000","last_became_active":"2023-10-16T14:10:25.813017+0000","last_became_peered":"2023-10-16T14:10:25.813017+0000","last_unstale":"2023-10-16T14:10:40.978483+0000","last_undegraded":"2023-10-16T14:10:40.978483+0000","last_fullsized":"2023-10-16T14:10:40.978483+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:08.149493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967437+0000","last_change":"2023-10-16T14:10:26.837742+0000","last_active":"2023-10-16T14:10:40.967437+0000","last_peered":"2023-10-16T14:10:40.967437+0000","last_clean":"2023-10-16T14:10:40.967437+0000","last_became_active":"2023-10-16T14:10:26.837444+0000","last_became_peered":"2023-10-16T14:10:26.837444+0000","last_unstale":"2023-10-16T14:10:40.967437+0000","last_undegraded":"2023-10-16T14:10:40.967437+0000","last_fullsized":"2023-10-16T14:10:40.967437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:46:59.305787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010987+0000","last_change":"2023-10-16T14:10:39.984550+0000","last_active":"2023-10-16T14:10:41.010987+0000","last_peered":"2023-10-16T14:10:41.010987+0000","last_clean":"2023-10-16T14:10:41.010987+0000","last_became_active":"2023-10-16T14:10:39.984260+0000","last_became_peered":"2023-10-16T14:10:39.984260+0000","last_unstale":"2023-10-16T14:10:41.010987+0000","last_undegraded":"2023-10-16T14:10:41.010987+0000","last_fullsized":"2023-10-16T14:10:41.010987+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:14.438775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143524+0000","last_change":"2023-10-16T14:10:40.983245+0000","last_active":"2023-10-16T14:10:43.143524+0000","last_peered":"2023-10-16T14:10:43.143524+0000","last_clean":"2023-10-16T14:10:43.143524+0000","last_became_active":"2023-10-16T14:10:40.982951+0000","last_became_peered":"2023-10-16T14:10:40.982951+0000","last_unstale":"2023-10-16T14:10:43.143524+0000","last_undegraded":"2023-10-16T14:10:43.143524+0000","last_fullsized":"2023-10-16T14:10:43.143524+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:14.127769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981490+0000","last_change":"2023-10-16T14:10:25.809572+0000","last_active":"2023-10-16T14:10:40.981490+0000","last_peered":"2023-10-16T14:10:40.981490+0000","last_clean":"2023-10-16T14:10:40.981490+0000","last_became_active":"2023-10-16T14:10:25.809136+0000","last_became_peered":"2023-10-16T14:10:25.809136+0000","last_unstale":"2023-10-16T14:10:40.981490+0000","last_undegraded":"2023-10-16T14:10:40.981490+0000","last_fullsized":"2023-10-16T14:10:40.981490+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:29.939076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981982+0000","last_change":"2023-10-16T14:10:26.839603+0000","last_active":"2023-10-16T14:10:40.981982+0000","last_peered":"2023-10-16T14:10:40.981982+0000","last_clean":"2023-10-16T14:10:40.981982+0000","last_became_active":"2023-10-16T14:10:26.836763+0000","last_became_peered":"2023-10-16T14:10:26.836763+0000","last_unstale":"2023-10-16T14:10:40.981982+0000","last_undegraded":"2023-10-16T14:10:40.981982+0000","last_fullsized":"2023-10-16T14:10:40.981982+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:23.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961192+0000","last_change":"2023-10-16T14:10:39.981457+0000","last_active":"2023-10-16T14:10:40.961192+0000","last_peered":"2023-10-16T14:10:40.961192+0000","last_clean":"2023-10-16T14:10:40.961192+0000","last_became_active":"2023-10-16T14:10:39.981145+0000","last_became_peered":"2023-10-16T14:10:39.981145+0000","last_unstale":"2023-10-16T14:10:40.961192+0000","last_undegraded":"2023-10-16T14:10:40.961192+0000","last_fullsized":"2023-10-16T14:10:40.961192+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:45:33.143537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143133+0000","last_change":"2023-10-16T14:10:40.982170+0000","last_active":"2023-10-16T14:10:43.143133+0000","last_peered":"2023-10-16T14:10:43.143133+0000","last_clean":"2023-10-16T14:10:43.143133+0000","last_became_active":"2023-10-16T14:10:40.981890+0000","last_became_peered":"2023-10-16T14:10:40.981890+0000","last_unstale":"2023-10-16T14:10:43.143133+0000","last_undegraded":"2023-10-16T14:10:43.143133+0000","last_fullsized":"2023-10-16T14:10:43.143133+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:09.758522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960165+0000","last_change":"2023-10-16T14:10:25.801908+0000","last_active":"2023-10-16T14:10:40.960165+0000","last_peered":"2023-10-16T14:10:40.960165+0000","last_clean":"2023-10-16T14:10:40.960165+0000","last_became_active":"2023-10-16T14:10:25.801605+0000","last_became_peered":"2023-10-16T14:10:25.801605+0000","last_unstale":"2023-10-16T14:10:40.960165+0000","last_undegraded":"2023-10-16T14:10:40.960165+0000","last_fullsized":"2023-10-16T14:10:40.960165+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:59:38.050567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960903+0000","last_change":"2023-10-16T14:10:26.839645+0000","last_active":"2023-10-16T14:10:40.960903+0000","last_peered":"2023-10-16T14:10:40.960903+0000","last_clean":"2023-10-16T14:10:40.960903+0000","last_became_active":"2023-10-16T14:10:26.836237+0000","last_became_peered":"2023-10-16T14:10:26.836237+0000","last_unstale":"2023-10-16T14:10:40.960903+0000","last_undegraded":"2023-10-16T14:10:40.960903+0000","last_fullsized":"2023-10-16T14:10:40.960903+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.010705+0000","last_change":"2023-10-16T14:10:39.984267+0000","last_active":"2023-10-16T14:10:41.010705+0000","last_peered":"2023-10-16T14:10:41.010705+0000","last_clean":"2023-10-16T14:10:41.010705+0000","last_became_active":"2023-10-16T14:10:39.983894+0000","last_became_peered":"2023-10-16T14:10:39.983894+0000","last_unstale":"2023-10-16T14:10:41.010705+0000","last_undegraded":"2023-10-16T14:10:41.010705+0000","last_fullsized":"2023-10-16T14:10:41.010705+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:02.427201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.142906+0000","last_change":"2023-10-16T14:10:40.994626+0000","last_active":"2023-10-16T14:10:43.142906+0000","last_peered":"2023-10-16T14:10:43.142906+0000","last_clean":"2023-10-16T14:10:43.142906+0000","last_became_active":"2023-10-16T14:10:40.994283+0000","last_became_peered":"2023-10-16T14:10:40.994283+0000","last_unstale":"2023-10-16T14:10:43.142906+0000","last_undegraded":"2023-10-16T14:10:43.142906+0000","last_fullsized":"2023-10-16T14:10:43.142906+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:16.777756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979684+0000","last_change":"2023-10-16T14:10:25.809562+0000","last_active":"2023-10-16T14:10:40.979684+0000","last_peered":"2023-10-16T14:10:40.979684+0000","last_clean":"2023-10-16T14:10:40.979684+0000","last_became_active":"2023-10-16T14:10:25.809284+0000","last_became_peered":"2023-10-16T14:10:25.809284+0000","last_unstale":"2023-10-16T14:10:40.979684+0000","last_undegraded":"2023-10-16T14:10:40.979684+0000","last_fullsized":"2023-10-16T14:10:40.979684+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961306+0000","last_change":"2023-10-16T14:10:26.829549+0000","last_active":"2023-10-16T14:10:40.961306+0000","last_peered":"2023-10-16T14:10:40.961306+0000","last_clean":"2023-10-16T14:10:40.961306+0000","last_became_active":"2023-10-16T14:10:26.829159+0000","last_became_peered":"2023-10-16T14:10:26.829159+0000","last_unstale":"2023-10-16T14:10:40.961306+0000","last_undegraded":"2023-10-16T14:10:40.961306+0000","last_fullsized":"2023-10-16T14:10:40.961306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:04.686065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971464+0000","last_change":"2023-10-16T14:10:39.974033+0000","last_active":"2023-10-16T14:10:40.971464+0000","last_peered":"2023-10-16T14:10:40.971464+0000","last_clean":"2023-10-16T14:10:40.971464+0000","last_became_active":"2023-10-16T14:10:39.973723+0000","last_became_peered":"2023-10-16T14:10:39.973723+0000","last_unstale":"2023-10-16T14:10:40.971464+0000","last_undegraded":"2023-10-16T14:10:40.971464+0000","last_fullsized":"2023-10-16T14:10:40.971464+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:08.572098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124517+0000","last_change":"2023-10-16T14:10:40.994623+0000","last_active":"2023-10-16T14:10:42.124517+0000","last_peered":"2023-10-16T14:10:42.124517+0000","last_clean":"2023-10-16T14:10:42.124517+0000","last_became_active":"2023-10-16T14:10:40.994347+0000","last_became_peered":"2023-10-16T14:10:40.994347+0000","last_unstale":"2023-10-16T14:10:42.124517+0000","last_undegraded":"2023-10-16T14:10:42.124517+0000","last_fullsized":"2023-10-16T14:10:42.124517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:18.643655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961579+0000","last_change":"2023-10-16T14:10:25.808392+0000","last_active":"2023-10-16T14:10:40.961579+0000","last_peered":"2023-10-16T14:10:40.961579+0000","last_clean":"2023-10-16T14:10:40.961579+0000","last_became_active":"2023-10-16T14:10:25.808070+0000","last_became_peered":"2023-10-16T14:10:25.808070+0000","last_unstale":"2023-10-16T14:10:40.961579+0000","last_undegraded":"2023-10-16T14:10:40.961579+0000","last_fullsized":"2023-10-16T14:10:40.961579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:51.554249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979554+0000","last_change":"2023-10-16T14:10:26.840216+0000","last_active":"2023-10-16T14:10:40.979554+0000","last_peered":"2023-10-16T14:10:40.979554+0000","last_clean":"2023-10-16T14:10:40.979554+0000","last_became_active":"2023-10-16T14:10:26.839793+0000","last_became_peered":"2023-10-16T14:10:26.839793+0000","last_unstale":"2023-10-16T14:10:40.979554+0000","last_undegraded":"2023-10-16T14:10:40.979554+0000","last_fullsized":"2023-10-16T14:10:40.979554+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:55.430917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009801+0000","last_change":"2023-10-16T14:10:39.986465+0000","last_active":"2023-10-16T14:10:41.009801+0000","last_peered":"2023-10-16T14:10:41.009801+0000","last_clean":"2023-10-16T14:10:41.009801+0000","last_became_active":"2023-10-16T14:10:39.986165+0000","last_became_peered":"2023-10-16T14:10:39.986165+0000","last_unstale":"2023-10-16T14:10:41.009801+0000","last_undegraded":"2023-10-16T14:10:41.009801+0000","last_fullsized":"2023-10-16T14:10:41.009801+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:08.068911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124542+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:42.124542+0000","last_peered":"2023-10-16T14:10:42.124542+0000","last_clean":"2023-10-16T14:10:42.124542+0000","last_became_active":"2023-10-16T14:10:40.993130+0000","last_became_peered":"2023-10-16T14:10:40.993130+0000","last_unstale":"2023-10-16T14:10:42.124542+0000","last_undegraded":"2023-10-16T14:10:42.124542+0000","last_fullsized":"2023-10-16T14:10:42.124542+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:25.881584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973127+0000","last_change":"2023-10-16T14:10:25.811819+0000","last_active":"2023-10-16T14:10:40.973127+0000","last_peered":"2023-10-16T14:10:40.973127+0000","last_clean":"2023-10-16T14:10:40.973127+0000","last_became_active":"2023-10-16T14:10:25.811526+0000","last_became_peered":"2023-10-16T14:10:25.811526+0000","last_unstale":"2023-10-16T14:10:40.973127+0000","last_undegraded":"2023-10-16T14:10:40.973127+0000","last_fullsized":"2023-10-16T14:10:40.973127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:57.466552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974510+0000","last_change":"2023-10-16T14:10:26.843616+0000","last_active":"2023-10-16T14:10:40.974510+0000","last_peered":"2023-10-16T14:10:40.974510+0000","last_clean":"2023-10-16T14:10:40.974510+0000","last_became_active":"2023-10-16T14:10:26.842563+0000","last_became_peered":"2023-10-16T14:10:26.842563+0000","last_unstale":"2023-10-16T14:10:40.974510+0000","last_undegraded":"2023-10-16T14:10:40.974510+0000","last_fullsized":"2023-10-16T14:10:40.974510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:56.718654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967349+0000","last_change":"2023-10-16T14:10:39.986684+0000","last_active":"2023-10-16T14:10:40.967349+0000","last_peered":"2023-10-16T14:10:40.967349+0000","last_clean":"2023-10-16T14:10:40.967349+0000","last_became_active":"2023-10-16T14:10:39.986367+0000","last_became_peered":"2023-10-16T14:10:39.986367+0000","last_unstale":"2023-10-16T14:10:40.967349+0000","last_undegraded":"2023-10-16T14:10:40.967349+0000","last_fullsized":"2023-10-16T14:10:40.967349+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:33:11.665429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185918+0000","last_change":"2023-10-16T14:10:40.993254+0000","last_active":"2023-10-16T14:10:43.185918+0000","last_peered":"2023-10-16T14:10:43.185918+0000","last_clean":"2023-10-16T14:10:43.185918+0000","last_became_active":"2023-10-16T14:10:40.992918+0000","last_became_peered":"2023-10-16T14:10:40.992918+0000","last_unstale":"2023-10-16T14:10:43.185918+0000","last_undegraded":"2023-10-16T14:10:43.185918+0000","last_fullsized":"2023-10-16T14:10:43.185918+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:07.376549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976318+0000","last_change":"2023-10-16T14:10:25.811278+0000","last_active":"2023-10-16T14:10:40.976318+0000","last_peered":"2023-10-16T14:10:40.976318+0000","last_clean":"2023-10-16T14:10:40.976318+0000","last_became_active":"2023-10-16T14:10:25.806140+0000","last_became_peered":"2023-10-16T14:10:25.806140+0000","last_unstale":"2023-10-16T14:10:40.976318+0000","last_undegraded":"2023-10-16T14:10:40.976318+0000","last_fullsized":"2023-10-16T14:10:40.976318+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972611+0000","last_change":"2023-10-16T14:10:26.841758+0000","last_active":"2023-10-16T14:10:40.972611+0000","last_peered":"2023-10-16T14:10:40.972611+0000","last_clean":"2023-10-16T14:10:40.972611+0000","last_became_active":"2023-10-16T14:10:26.841460+0000","last_became_peered":"2023-10-16T14:10:26.841460+0000","last_unstale":"2023-10-16T14:10:40.972611+0000","last_undegraded":"2023-10-16T14:10:40.972611+0000","last_fullsized":"2023-10-16T14:10:40.972611+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:49.611678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970330+0000","last_change":"2023-10-16T14:10:39.989051+0000","last_active":"2023-10-16T14:10:40.970330+0000","last_peered":"2023-10-16T14:10:40.970330+0000","last_clean":"2023-10-16T14:10:40.970330+0000","last_became_active":"2023-10-16T14:10:39.987944+0000","last_became_peered":"2023-10-16T14:10:39.987944+0000","last_unstale":"2023-10-16T14:10:40.970330+0000","last_undegraded":"2023-10-16T14:10:40.970330+0000","last_fullsized":"2023-10-16T14:10:40.970330+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244927+0000","last_change":"2023-10-16T14:10:40.996616+0000","last_active":"2023-10-16T14:10:43.244927+0000","last_peered":"2023-10-16T14:10:43.244927+0000","last_clean":"2023-10-16T14:10:43.244927+0000","last_became_active":"2023-10-16T14:10:40.996057+0000","last_became_peered":"2023-10-16T14:10:40.996057+0000","last_unstale":"2023-10-16T14:10:43.244927+0000","last_undegraded":"2023-10-16T14:10:43.244927+0000","last_fullsized":"2023-10-16T14:10:43.244927+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:32.328604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"230'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981955+0000","last_change":"2023-10-16T14:10:25.815813+0000","last_active":"2023-10-16T14:10:40.981955+0000","last_peered":"2023-10-16T14:10:40.981955+0000","last_clean":"2023-10-16T14:10:40.981955+0000","last_became_active":"2023-10-16T14:10:25.815445+0000","last_became_peered":"2023-10-16T14:10:25.815445+0000","last_unstale":"2023-10-16T14:10:40.981955+0000","last_undegraded":"2023-10-16T14:10:40.981955+0000","last_fullsized":"2023-10-16T14:10:40.981955+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:31.764614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978301+0000","last_change":"2023-10-16T14:10:26.847748+0000","last_active":"2023-10-16T14:10:40.978301+0000","last_peered":"2023-10-16T14:10:40.978301+0000","last_clean":"2023-10-16T14:10:40.978301+0000","last_became_active":"2023-10-16T14:10:26.847264+0000","last_became_peered":"2023-10-16T14:10:26.847264+0000","last_unstale":"2023-10-16T14:10:40.978301+0000","last_undegraded":"2023-10-16T14:10:40.978301+0000","last_fullsized":"2023-10-16T14:10:40.978301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:21.365719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980655+0000","last_change":"2023-10-16T14:10:39.972329+0000","last_active":"2023-10-16T14:10:40.980655+0000","last_peered":"2023-10-16T14:10:40.980655+0000","last_clean":"2023-10-16T14:10:40.980655+0000","last_became_active":"2023-10-16T14:10:39.972015+0000","last_became_peered":"2023-10-16T14:10:39.972015+0000","last_unstale":"2023-10-16T14:10:40.980655+0000","last_undegraded":"2023-10-16T14:10:40.980655+0000","last_fullsized":"2023-10-16T14:10:40.980655+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:17.467680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996971+0000","last_change":"2023-10-16T14:10:41.001961+0000","last_active":"2023-10-16T14:10:41.996971+0000","last_peered":"2023-10-16T14:10:41.996971+0000","last_clean":"2023-10-16T14:10:41.996971+0000","last_became_active":"2023-10-16T14:10:41.001699+0000","last_became_peered":"2023-10-16T14:10:41.001699+0000","last_unstale":"2023-10-16T14:10:41.996971+0000","last_undegraded":"2023-10-16T14:10:41.996971+0000","last_fullsized":"2023-10-16T14:10:41.996971+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959024+0000","last_change":"2023-10-16T14:10:25.815387+0000","last_active":"2023-10-16T14:10:40.959024+0000","last_peered":"2023-10-16T14:10:40.959024+0000","last_clean":"2023-10-16T14:10:40.959024+0000","last_became_active":"2023-10-16T14:10:25.814845+0000","last_became_peered":"2023-10-16T14:10:25.814845+0000","last_unstale":"2023-10-16T14:10:40.959024+0000","last_undegraded":"2023-10-16T14:10:40.959024+0000","last_fullsized":"2023-10-16T14:10:40.959024+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:34.992330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975612+0000","last_change":"2023-10-16T14:10:26.848002+0000","last_active":"2023-10-16T14:10:40.975612+0000","last_peered":"2023-10-16T14:10:40.975612+0000","last_clean":"2023-10-16T14:10:40.975612+0000","last_became_active":"2023-10-16T14:10:26.847510+0000","last_became_peered":"2023-10-16T14:10:26.847510+0000","last_unstale":"2023-10-16T14:10:40.975612+0000","last_undegraded":"2023-10-16T14:10:40.975612+0000","last_fullsized":"2023-10-16T14:10:40.975612+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:16.878056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.985242+0000","last_change":"2023-10-16T14:10:39.990904+0000","last_active":"2023-10-16T14:10:40.985242+0000","last_peered":"2023-10-16T14:10:40.985242+0000","last_clean":"2023-10-16T14:10:40.985242+0000","last_became_active":"2023-10-16T14:10:39.990155+0000","last_became_peered":"2023-10-16T14:10:39.990155+0000","last_unstale":"2023-10-16T14:10:40.985242+0000","last_undegraded":"2023-10-16T14:10:40.985242+0000","last_fullsized":"2023-10-16T14:10:40.985242+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:56.282178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243767+0000","last_change":"2023-10-16T14:10:40.993640+0000","last_active":"2023-10-16T14:10:43.243767+0000","last_peered":"2023-10-16T14:10:43.243767+0000","last_clean":"2023-10-16T14:10:43.243767+0000","last_became_active":"2023-10-16T14:10:40.993084+0000","last_became_peered":"2023-10-16T14:10:40.993084+0000","last_unstale":"2023-10-16T14:10:43.243767+0000","last_undegraded":"2023-10-16T14:10:43.243767+0000","last_fullsized":"2023-10-16T14:10:43.243767+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:50:12.061147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987125+0000","last_change":"2023-10-16T14:10:25.807294+0000","last_active":"2023-10-16T14:10:40.987125+0000","last_peered":"2023-10-16T14:10:40.987125+0000","last_clean":"2023-10-16T14:10:40.987125+0000","last_became_active":"2023-10-16T14:10:25.806808+0000","last_became_peered":"2023-10-16T14:10:25.806808+0000","last_unstale":"2023-10-16T14:10:40.987125+0000","last_undegraded":"2023-10-16T14:10:40.987125+0000","last_fullsized":"2023-10-16T14:10:40.987125+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:04.234540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959827+0000","last_change":"2023-10-16T14:10:26.838564+0000","last_active":"2023-10-16T14:10:40.959827+0000","last_peered":"2023-10-16T14:10:40.959827+0000","last_clean":"2023-10-16T14:10:40.959827+0000","last_became_active":"2023-10-16T14:10:26.838299+0000","last_became_peered":"2023-10-16T14:10:26.838299+0000","last_unstale":"2023-10-16T14:10:40.959827+0000","last_undegraded":"2023-10-16T14:10:40.959827+0000","last_fullsized":"2023-10-16T14:10:40.959827+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:34.925197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"234'6","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013944+0000","last_change":"2023-10-16T14:10:39.971085+0000","last_active":"2023-10-16T14:10:41.013944+0000","last_peered":"2023-10-16T14:10:41.013944+0000","last_clean":"2023-10-16T14:10:41.013944+0000","last_became_active":"2023-10-16T14:10:39.970777+0000","last_became_peered":"2023-10-16T14:10:39.970777+0000","last_unstale":"2023-10-16T14:10:41.013944+0000","last_undegraded":"2023-10-16T14:10:41.013944+0000","last_fullsized":"2023-10-16T14:10:41.013944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:03.090258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.998098+0000","last_change":"2023-10-16T14:10:40.985478+0000","last_active":"2023-10-16T14:10:41.998098+0000","last_peered":"2023-10-16T14:10:41.998098+0000","last_clean":"2023-10-16T14:10:41.998098+0000","last_became_active":"2023-10-16T14:10:40.985218+0000","last_became_peered":"2023-10-16T14:10:40.985218+0000","last_unstale":"2023-10-16T14:10:41.998098+0000","last_undegraded":"2023-10-16T14:10:41.998098+0000","last_fullsized":"2023-10-16T14:10:41.998098+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:22.942748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.986478+0000","last_change":"2023-10-16T14:10:25.805122+0000","last_active":"2023-10-16T14:10:40.986478+0000","last_peered":"2023-10-16T14:10:40.986478+0000","last_clean":"2023-10-16T14:10:40.986478+0000","last_became_active":"2023-10-16T14:10:25.804713+0000","last_became_peered":"2023-10-16T14:10:25.804713+0000","last_unstale":"2023-10-16T14:10:40.986478+0000","last_undegraded":"2023-10-16T14:10:40.986478+0000","last_fullsized":"2023-10-16T14:10:40.986478+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:47.692654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974315+0000","last_change":"2023-10-16T14:10:26.845159+0000","last_active":"2023-10-16T14:10:40.974315+0000","last_peered":"2023-10-16T14:10:40.974315+0000","last_clean":"2023-10-16T14:10:40.974315+0000","last_became_active":"2023-10-16T14:10:26.844673+0000","last_became_peered":"2023-10-16T14:10:26.844673+0000","last_unstale":"2023-10-16T14:10:40.974315+0000","last_undegraded":"2023-10-16T14:10:40.974315+0000","last_fullsized":"2023-10-16T14:10:40.974315+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:36.384056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"234'3","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.008061+0000","last_change":"2023-10-16T14:10:39.973954+0000","last_active":"2023-10-16T14:10:41.008061+0000","last_peered":"2023-10-16T14:10:41.008061+0000","last_clean":"2023-10-16T14:10:41.008061+0000","last_became_active":"2023-10-16T14:10:39.973602+0000","last_became_peered":"2023-10-16T14:10:39.973602+0000","last_unstale":"2023-10-16T14:10:41.008061+0000","last_undegraded":"2023-10-16T14:10:41.008061+0000","last_fullsized":"2023-10-16T14:10:41.008061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:18.496303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143885+0000","last_change":"2023-10-16T14:10:40.991452+0000","last_active":"2023-10-16T14:10:43.143885+0000","last_peered":"2023-10-16T14:10:43.143885+0000","last_clean":"2023-10-16T14:10:43.143885+0000","last_became_active":"2023-10-16T14:10:40.985777+0000","last_became_peered":"2023-10-16T14:10:40.985777+0000","last_unstale":"2023-10-16T14:10:43.143885+0000","last_undegraded":"2023-10-16T14:10:43.143885+0000","last_fullsized":"2023-10-16T14:10:43.143885+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.881711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"231'8","reported_seq":24,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.983108+0000","last_change":"2023-10-16T14:10:25.818271+0000","last_active":"2023-10-16T14:10:40.983108+0000","last_peered":"2023-10-16T14:10:40.983108+0000","last_clean":"2023-10-16T14:10:40.983108+0000","last_became_active":"2023-10-16T14:10:25.816998+0000","last_became_peered":"2023-10-16T14:10:25.816998+0000","last_unstale":"2023-10-16T14:10:40.983108+0000","last_undegraded":"2023-10-16T14:10:40.983108+0000","last_fullsized":"2023-10-16T14:10:40.983108+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:04:32.671236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977730+0000","last_change":"2023-10-16T14:10:26.846069+0000","last_active":"2023-10-16T14:10:40.977730+0000","last_peered":"2023-10-16T14:10:40.977730+0000","last_clean":"2023-10-16T14:10:40.977730+0000","last_became_active":"2023-10-16T14:10:26.845674+0000","last_became_peered":"2023-10-16T14:10:26.845674+0000","last_unstale":"2023-10-16T14:10:40.977730+0000","last_undegraded":"2023-10-16T14:10:40.977730+0000","last_fullsized":"2023-10-16T14:10:40.977730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:43.707413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980420+0000","last_change":"2023-10-16T14:10:39.976858+0000","last_active":"2023-10-16T14:10:40.980420+0000","last_peered":"2023-10-16T14:10:40.980420+0000","last_clean":"2023-10-16T14:10:40.980420+0000","last_became_active":"2023-10-16T14:10:39.976566+0000","last_became_peered":"2023-10-16T14:10:39.976566+0000","last_unstale":"2023-10-16T14:10:40.980420+0000","last_undegraded":"2023-10-16T14:10:40.980420+0000","last_fullsized":"2023-10-16T14:10:40.980420+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:02:00.617867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.990318+0000","last_change":"2023-10-16T14:10:40.985099+0000","last_active":"2023-10-16T14:10:41.990318+0000","last_peered":"2023-10-16T14:10:41.990318+0000","last_clean":"2023-10-16T14:10:41.990318+0000","last_became_active":"2023-10-16T14:10:40.984799+0000","last_became_peered":"2023-10-16T14:10:40.984799+0000","last_unstale":"2023-10-16T14:10:41.990318+0000","last_undegraded":"2023-10-16T14:10:41.990318+0000","last_fullsized":"2023-10-16T14:10:41.990318+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:02.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"230'3","reported_seq":19,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961436+0000","last_change":"2023-10-16T14:10:25.804269+0000","last_active":"2023-10-16T14:10:40.961436+0000","last_peered":"2023-10-16T14:10:40.961436+0000","last_clean":"2023-10-16T14:10:40.961436+0000","last_became_active":"2023-10-16T14:10:25.803750+0000","last_became_peered":"2023-10-16T14:10:25.803750+0000","last_unstale":"2023-10-16T14:10:40.961436+0000","last_undegraded":"2023-10-16T14:10:40.961436+0000","last_fullsized":"2023-10-16T14:10:40.961436+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:25.997242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982473+0000","last_change":"2023-10-16T14:10:26.847990+0000","last_active":"2023-10-16T14:10:40.982473+0000","last_peered":"2023-10-16T14:10:40.982473+0000","last_clean":"2023-10-16T14:10:40.982473+0000","last_became_active":"2023-10-16T14:10:26.847703+0000","last_became_peered":"2023-10-16T14:10:26.847703+0000","last_unstale":"2023-10-16T14:10:40.982473+0000","last_undegraded":"2023-10-16T14:10:40.982473+0000","last_fullsized":"2023-10-16T14:10:40.982473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.576065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011710+0000","last_change":"2023-10-16T14:10:39.986334+0000","last_active":"2023-10-16T14:10:41.011710+0000","last_peered":"2023-10-16T14:10:41.011710+0000","last_clean":"2023-10-16T14:10:41.011710+0000","last_became_active":"2023-10-16T14:10:39.986002+0000","last_became_peered":"2023-10-16T14:10:39.986002+0000","last_unstale":"2023-10-16T14:10:41.011710+0000","last_undegraded":"2023-10-16T14:10:41.011710+0000","last_fullsized":"2023-10-16T14:10:41.011710+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:53.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994013+0000","last_change":"2023-10-16T14:10:40.976822+0000","last_active":"2023-10-16T14:10:41.994013+0000","last_peered":"2023-10-16T14:10:41.994013+0000","last_clean":"2023-10-16T14:10:41.994013+0000","last_became_active":"2023-10-16T14:10:40.976523+0000","last_became_peered":"2023-10-16T14:10:40.976523+0000","last_unstale":"2023-10-16T14:10:41.994013+0000","last_undegraded":"2023-10-16T14:10:41.994013+0000","last_fullsized":"2023-10-16T14:10:41.994013+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987031+0000","last_change":"2023-10-16T14:10:25.807512+0000","last_active":"2023-10-16T14:10:40.987031+0000","last_peered":"2023-10-16T14:10:40.987031+0000","last_clean":"2023-10-16T14:10:40.987031+0000","last_became_active":"2023-10-16T14:10:25.807079+0000","last_became_peered":"2023-10-16T14:10:25.807079+0000","last_unstale":"2023-10-16T14:10:40.987031+0000","last_undegraded":"2023-10-16T14:10:40.987031+0000","last_fullsized":"2023-10-16T14:10:40.987031+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:01.304725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980779+0000","last_change":"2023-10-16T14:10:26.846262+0000","last_active":"2023-10-16T14:10:40.980779+0000","last_peered":"2023-10-16T14:10:40.980779+0000","last_clean":"2023-10-16T14:10:40.980779+0000","last_became_active":"2023-10-16T14:10:26.845946+0000","last_became_peered":"2023-10-16T14:10:26.845946+0000","last_unstale":"2023-10-16T14:10:40.980779+0000","last_undegraded":"2023-10-16T14:10:40.980779+0000","last_fullsized":"2023-10-16T14:10:40.980779+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:28.963086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"234'8","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.013387+0000","last_change":"2023-10-16T14:10:39.979255+0000","last_active":"2023-10-16T14:10:41.013387+0000","last_peered":"2023-10-16T14:10:41.013387+0000","last_clean":"2023-10-16T14:10:41.013387+0000","last_became_active":"2023-10-16T14:10:39.977217+0000","last_became_peered":"2023-10-16T14:10:39.977217+0000","last_unstale":"2023-10-16T14:10:41.013387+0000","last_undegraded":"2023-10-16T14:10:41.013387+0000","last_fullsized":"2023-10-16T14:10:41.013387+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:30.019236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.142508+0000","last_change":"2023-10-16T14:10:40.989802+0000","last_active":"2023-10-16T14:10:43.142508+0000","last_peered":"2023-10-16T14:10:43.142508+0000","last_clean":"2023-10-16T14:10:43.142508+0000","last_became_active":"2023-10-16T14:10:40.988409+0000","last_became_peered":"2023-10-16T14:10:40.988409+0000","last_unstale":"2023-10-16T14:10:43.142508+0000","last_undegraded":"2023-10-16T14:10:43.142508+0000","last_fullsized":"2023-10-16T14:10:43.142508+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:16.220155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976181+0000","last_change":"2023-10-16T14:10:25.816505+0000","last_active":"2023-10-16T14:10:40.976181+0000","last_peered":"2023-10-16T14:10:40.976181+0000","last_clean":"2023-10-16T14:10:40.976181+0000","last_became_active":"2023-10-16T14:10:25.816207+0000","last_became_peered":"2023-10-16T14:10:25.816207+0000","last_unstale":"2023-10-16T14:10:40.976181+0000","last_undegraded":"2023-10-16T14:10:40.976181+0000","last_fullsized":"2023-10-16T14:10:40.976181+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.500430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979705+0000","last_change":"2023-10-16T14:10:26.839991+0000","last_active":"2023-10-16T14:10:40.979705+0000","last_peered":"2023-10-16T14:10:40.979705+0000","last_clean":"2023-10-16T14:10:40.979705+0000","last_became_active":"2023-10-16T14:10:26.839716+0000","last_became_peered":"2023-10-16T14:10:26.839716+0000","last_unstale":"2023-10-16T14:10:40.979705+0000","last_undegraded":"2023-10-16T14:10:40.979705+0000","last_fullsized":"2023-10-16T14:10:40.979705+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975789+0000","last_change":"2023-10-16T14:10:39.968896+0000","last_active":"2023-10-16T14:10:40.975789+0000","last_peered":"2023-10-16T14:10:40.975789+0000","last_clean":"2023-10-16T14:10:40.975789+0000","last_became_active":"2023-10-16T14:10:39.968633+0000","last_became_peered":"2023-10-16T14:10:39.968633+0000","last_unstale":"2023-10-16T14:10:40.975789+0000","last_undegraded":"2023-10-16T14:10:40.975789+0000","last_fullsized":"2023-10-16T14:10:40.975789+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:26.024819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186867+0000","last_change":"2023-10-16T14:10:40.999802+0000","last_active":"2023-10-16T14:10:43.186867+0000","last_peered":"2023-10-16T14:10:43.186867+0000","last_clean":"2023-10-16T14:10:43.186867+0000","last_became_active":"2023-10-16T14:10:40.999483+0000","last_became_peered":"2023-10-16T14:10:40.999483+0000","last_unstale":"2023-10-16T14:10:43.186867+0000","last_undegraded":"2023-10-16T14:10:43.186867+0000","last_fullsized":"2023-10-16T14:10:43.186867+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:19.512934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011187+0000","last_change":"2023-10-16T14:10:39.979452+0000","last_active":"2023-10-16T14:10:41.011187+0000","last_peered":"2023-10-16T14:10:41.011187+0000","last_clean":"2023-10-16T14:10:41.011187+0000","last_became_active":"2023-10-16T14:10:39.978243+0000","last_became_peered":"2023-10-16T14:10:39.978243+0000","last_unstale":"2023-10-16T14:10:41.011187+0000","last_undegraded":"2023-10-16T14:10:41.011187+0000","last_fullsized":"2023-10-16T14:10:41.011187+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.984244+0000","last_change":"2023-10-16T14:10:25.812944+0000","last_active":"2023-10-16T14:10:40.984244+0000","last_peered":"2023-10-16T14:10:40.984244+0000","last_clean":"2023-10-16T14:10:40.984244+0000","last_became_active":"2023-10-16T14:10:25.812623+0000","last_became_peered":"2023-10-16T14:10:25.812623+0000","last_unstale":"2023-10-16T14:10:40.984244+0000","last_undegraded":"2023-10-16T14:10:40.984244+0000","last_fullsized":"2023-10-16T14:10:40.984244+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:46.462750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976062+0000","last_change":"2023-10-16T14:10:26.833883+0000","last_active":"2023-10-16T14:10:40.976062+0000","last_peered":"2023-10-16T14:10:40.976062+0000","last_clean":"2023-10-16T14:10:40.976062+0000","last_became_active":"2023-10-16T14:10:26.833512+0000","last_became_peered":"2023-10-16T14:10:26.833512+0000","last_unstale":"2023-10-16T14:10:40.976062+0000","last_undegraded":"2023-10-16T14:10:40.976062+0000","last_fullsized":"2023-10-16T14:10:40.976062+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:52.568186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144120+0000","last_change":"2023-10-16T14:10:40.996750+0000","last_active":"2023-10-16T14:10:43.144120+0000","last_peered":"2023-10-16T14:10:43.144120+0000","last_clean":"2023-10-16T14:10:43.144120+0000","last_became_active":"2023-10-16T14:10:40.996369+0000","last_became_peered":"2023-10-16T14:10:40.996369+0000","last_unstale":"2023-10-16T14:10:43.144120+0000","last_undegraded":"2023-10-16T14:10:43.144120+0000","last_fullsized":"2023-10-16T14:10:43.144120+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:29.608507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"234'4","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009270+0000","last_change":"2023-10-16T14:10:39.973971+0000","last_active":"2023-10-16T14:10:41.009270+0000","last_peered":"2023-10-16T14:10:41.009270+0000","last_clean":"2023-10-16T14:10:41.009270+0000","last_became_active":"2023-10-16T14:10:39.973675+0000","last_became_peered":"2023-10-16T14:10:39.973675+0000","last_unstale":"2023-10-16T14:10:41.009270+0000","last_undegraded":"2023-10-16T14:10:41.009270+0000","last_fullsized":"2023-10-16T14:10:41.009270+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:01.458774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.976322+0000","last_change":"2023-10-16T14:10:25.818154+0000","last_active":"2023-10-16T14:10:40.976322+0000","last_peered":"2023-10-16T14:10:40.976322+0000","last_clean":"2023-10-16T14:10:40.976322+0000","last_became_active":"2023-10-16T14:10:25.817861+0000","last_became_peered":"2023-10-16T14:10:25.817861+0000","last_unstale":"2023-10-16T14:10:40.976322+0000","last_undegraded":"2023-10-16T14:10:40.976322+0000","last_fullsized":"2023-10-16T14:10:40.976322+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:02.446106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968823+0000","last_change":"2023-10-16T14:10:26.839294+0000","last_active":"2023-10-16T14:10:40.968823+0000","last_peered":"2023-10-16T14:10:40.968823+0000","last_clean":"2023-10-16T14:10:40.968823+0000","last_became_active":"2023-10-16T14:10:26.838962+0000","last_became_peered":"2023-10-16T14:10:26.838962+0000","last_unstale":"2023-10-16T14:10:40.968823+0000","last_undegraded":"2023-10-16T14:10:40.968823+0000","last_fullsized":"2023-10-16T14:10:40.968823+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:26.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995327+0000","last_change":"2023-10-16T14:10:40.984926+0000","last_active":"2023-10-16T14:10:41.995327+0000","last_peered":"2023-10-16T14:10:41.995327+0000","last_clean":"2023-10-16T14:10:41.995327+0000","last_became_active":"2023-10-16T14:10:40.984364+0000","last_became_peered":"2023-10-16T14:10:40.984364+0000","last_unstale":"2023-10-16T14:10:41.995327+0000","last_undegraded":"2023-10-16T14:10:41.995327+0000","last_fullsized":"2023-10-16T14:10:41.995327+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:50.805583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974558+0000","last_change":"2023-10-16T14:10:25.815368+0000","last_active":"2023-10-16T14:10:40.974558+0000","last_peered":"2023-10-16T14:10:40.974558+0000","last_clean":"2023-10-16T14:10:40.974558+0000","last_became_active":"2023-10-16T14:10:25.814172+0000","last_became_peered":"2023-10-16T14:10:25.814172+0000","last_unstale":"2023-10-16T14:10:40.974558+0000","last_undegraded":"2023-10-16T14:10:40.974558+0000","last_fullsized":"2023-10-16T14:10:40.974558+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:15.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977408+0000","last_change":"2023-10-16T14:10:26.830396+0000","last_active":"2023-10-16T14:10:40.977408+0000","last_peered":"2023-10-16T14:10:40.977408+0000","last_clean":"2023-10-16T14:10:40.977408+0000","last_became_active":"2023-10-16T14:10:26.829520+0000","last_became_peered":"2023-10-16T14:10:26.829520+0000","last_unstale":"2023-10-16T14:10:40.977408+0000","last_undegraded":"2023-10-16T14:10:40.977408+0000","last_fullsized":"2023-10-16T14:10:40.977408+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"234'2","reported_seq":14,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011689+0000","last_change":"2023-10-16T14:10:39.979320+0000","last_active":"2023-10-16T14:10:41.011689+0000","last_peered":"2023-10-16T14:10:41.011689+0000","last_clean":"2023-10-16T14:10:41.011689+0000","last_became_active":"2023-10-16T14:10:39.978947+0000","last_became_peered":"2023-10-16T14:10:39.978947+0000","last_unstale":"2023-10-16T14:10:41.011689+0000","last_undegraded":"2023-10-16T14:10:41.011689+0000","last_fullsized":"2023-10-16T14:10:41.011689+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:37.126385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125751+0000","last_change":"2023-10-16T14:10:40.997566+0000","last_active":"2023-10-16T14:10:42.125751+0000","last_peered":"2023-10-16T14:10:42.125751+0000","last_clean":"2023-10-16T14:10:42.125751+0000","last_became_active":"2023-10-16T14:10:40.997273+0000","last_became_peered":"2023-10-16T14:10:40.997273+0000","last_unstale":"2023-10-16T14:10:42.125751+0000","last_undegraded":"2023-10-16T14:10:42.125751+0000","last_fullsized":"2023-10-16T14:10:42.125751+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:38.615513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982594+0000","last_change":"2023-10-16T14:10:25.816704+0000","last_active":"2023-10-16T14:10:40.982594+0000","last_peered":"2023-10-16T14:10:40.982594+0000","last_clean":"2023-10-16T14:10:40.982594+0000","last_became_active":"2023-10-16T14:10:25.816384+0000","last_became_peered":"2023-10-16T14:10:25.816384+0000","last_unstale":"2023-10-16T14:10:40.982594+0000","last_undegraded":"2023-10-16T14:10:40.982594+0000","last_fullsized":"2023-10-16T14:10:40.982594+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:55:48.966370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.960137+0000","last_change":"2023-10-16T14:10:26.841933+0000","last_active":"2023-10-16T14:10:40.960137+0000","last_peered":"2023-10-16T14:10:40.960137+0000","last_clean":"2023-10-16T14:10:40.960137+0000","last_became_active":"2023-10-16T14:10:26.840417+0000","last_became_peered":"2023-10-16T14:10:26.840417+0000","last_unstale":"2023-10-16T14:10:40.960137+0000","last_undegraded":"2023-10-16T14:10:40.960137+0000","last_fullsized":"2023-10-16T14:10:40.960137+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:42.569529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.009413+0000","last_change":"2023-10-16T14:10:39.983838+0000","last_active":"2023-10-16T14:10:41.009413+0000","last_peered":"2023-10-16T14:10:41.009413+0000","last_clean":"2023-10-16T14:10:41.009413+0000","last_became_active":"2023-10-16T14:10:39.983415+0000","last_became_peered":"2023-10-16T14:10:39.983415+0000","last_unstale":"2023-10-16T14:10:41.009413+0000","last_undegraded":"2023-10-16T14:10:41.009413+0000","last_fullsized":"2023-10-16T14:10:41.009413+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:58.890985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243037+0000","last_change":"2023-10-16T14:10:40.993412+0000","last_active":"2023-10-16T14:10:43.243037+0000","last_peered":"2023-10-16T14:10:43.243037+0000","last_clean":"2023-10-16T14:10:43.243037+0000","last_became_active":"2023-10-16T14:10:40.993119+0000","last_became_peered":"2023-10-16T14:10:40.993119+0000","last_unstale":"2023-10-16T14:10:43.243037+0000","last_undegraded":"2023-10-16T14:10:43.243037+0000","last_fullsized":"2023-10-16T14:10:43.243037+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:58.022778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"230'2","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981800+0000","last_change":"2023-10-16T14:10:25.816435+0000","last_active":"2023-10-16T14:10:40.981800+0000","last_peered":"2023-10-16T14:10:40.981800+0000","last_clean":"2023-10-16T14:10:40.981800+0000","last_became_active":"2023-10-16T14:10:25.816129+0000","last_became_peered":"2023-10-16T14:10:25.816129+0000","last_unstale":"2023-10-16T14:10:40.981800+0000","last_undegraded":"2023-10-16T14:10:40.981800+0000","last_fullsized":"2023-10-16T14:10:40.981800+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:53.320143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980136+0000","last_change":"2023-10-16T14:10:26.847776+0000","last_active":"2023-10-16T14:10:40.980136+0000","last_peered":"2023-10-16T14:10:40.980136+0000","last_clean":"2023-10-16T14:10:40.980136+0000","last_became_active":"2023-10-16T14:10:26.847427+0000","last_became_peered":"2023-10-16T14:10:26.847427+0000","last_unstale":"2023-10-16T14:10:40.980136+0000","last_undegraded":"2023-10-16T14:10:40.980136+0000","last_fullsized":"2023-10-16T14:10:40.980136+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.003813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"234'1","reported_seq":13,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.011809+0000","last_change":"2023-10-16T14:10:39.969078+0000","last_active":"2023-10-16T14:10:41.011809+0000","last_peered":"2023-10-16T14:10:41.011809+0000","last_clean":"2023-10-16T14:10:41.011809+0000","last_became_active":"2023-10-16T14:10:39.968736+0000","last_became_peered":"2023-10-16T14:10:39.968736+0000","last_unstale":"2023-10-16T14:10:41.011809+0000","last_undegraded":"2023-10-16T14:10:41.011809+0000","last_fullsized":"2023-10-16T14:10:41.011809+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:54:59.354986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997370+0000","last_change":"2023-10-16T14:10:40.997896+0000","last_active":"2023-10-16T14:10:41.997370+0000","last_peered":"2023-10-16T14:10:41.997370+0000","last_clean":"2023-10-16T14:10:41.997370+0000","last_became_active":"2023-10-16T14:10:40.997623+0000","last_became_peered":"2023-10-16T14:10:40.997623+0000","last_unstale":"2023-10-16T14:10:41.997370+0000","last_undegraded":"2023-10-16T14:10:41.997370+0000","last_fullsized":"2023-10-16T14:10:41.997370+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"230'1","reported_seq":17,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972873+0000","last_change":"2023-10-16T14:10:25.816805+0000","last_active":"2023-10-16T14:10:40.972873+0000","last_peered":"2023-10-16T14:10:40.972873+0000","last_clean":"2023-10-16T14:10:40.972873+0000","last_became_active":"2023-10-16T14:10:25.816525+0000","last_became_peered":"2023-10-16T14:10:25.816525+0000","last_unstale":"2023-10-16T14:10:40.972873+0000","last_undegraded":"2023-10-16T14:10:40.972873+0000","last_fullsized":"2023-10-16T14:10:40.972873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.312563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.988803+0000","last_change":"2023-10-16T14:10:26.846786+0000","last_active":"2023-10-16T14:10:40.988803+0000","last_peered":"2023-10-16T14:10:40.988803+0000","last_clean":"2023-10-16T14:10:40.988803+0000","last_became_active":"2023-10-16T14:10:26.846373+0000","last_became_peered":"2023-10-16T14:10:26.846373+0000","last_unstale":"2023-10-16T14:10:40.988803+0000","last_undegraded":"2023-10-16T14:10:40.988803+0000","last_fullsized":"2023-10-16T14:10:40.988803+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:00.349292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959567+0000","last_change":"2023-10-16T14:10:39.973296+0000","last_active":"2023-10-16T14:10:40.959567+0000","last_peered":"2023-10-16T14:10:40.959567+0000","last_clean":"2023-10-16T14:10:40.959567+0000","last_became_active":"2023-10-16T14:10:39.972592+0000","last_became_peered":"2023-10-16T14:10:39.972592+0000","last_unstale":"2023-10-16T14:10:40.959567+0000","last_undegraded":"2023-10-16T14:10:40.959567+0000","last_fullsized":"2023-10-16T14:10:40.959567+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:04.406174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242344+0000","last_change":"2023-10-16T14:10:40.987363+0000","last_active":"2023-10-16T14:10:43.242344+0000","last_peered":"2023-10-16T14:10:43.242344+0000","last_clean":"2023-10-16T14:10:43.242344+0000","last_became_active":"2023-10-16T14:10:40.985914+0000","last_became_peered":"2023-10-16T14:10:40.985914+0000","last_unstale":"2023-10-16T14:10:43.242344+0000","last_undegraded":"2023-10-16T14:10:43.242344+0000","last_fullsized":"2023-10-16T14:10:43.242344+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:10.423467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"230'4","reported_seq":20,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975348+0000","last_change":"2023-10-16T14:10:25.818461+0000","last_active":"2023-10-16T14:10:40.975348+0000","last_peered":"2023-10-16T14:10:40.975348+0000","last_clean":"2023-10-16T14:10:40.975348+0000","last_became_active":"2023-10-16T14:10:25.818156+0000","last_became_peered":"2023-10-16T14:10:25.818156+0000","last_unstale":"2023-10-16T14:10:40.975348+0000","last_undegraded":"2023-10-16T14:10:40.975348+0000","last_fullsized":"2023-10-16T14:10:40.975348+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:24.759681+0000","last_clean_scrub_stamp":"2023-10-16T14:10:24.759681+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:26:59.025872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972050+0000","last_change":"2023-10-16T14:10:26.845915+0000","last_active":"2023-10-16T14:10:40.972050+0000","last_peered":"2023-10-16T14:10:40.972050+0000","last_clean":"2023-10-16T14:10:40.972050+0000","last_became_active":"2023-10-16T14:10:26.844219+0000","last_became_peered":"2023-10-16T14:10:26.844219+0000","last_unstale":"2023-10-16T14:10:40.972050+0000","last_undegraded":"2023-10-16T14:10:40.972050+0000","last_fullsized":"2023-10-16T14:10:40.972050+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:57.910783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.970323+0000","last_change":"2023-10-16T14:10:39.972648+0000","last_active":"2023-10-16T14:10:40.970323+0000","last_peered":"2023-10-16T14:10:40.970323+0000","last_clean":"2023-10-16T14:10:40.970323+0000","last_became_active":"2023-10-16T14:10:39.972328+0000","last_became_peered":"2023-10-16T14:10:39.972328+0000","last_unstale":"2023-10-16T14:10:40.970323+0000","last_undegraded":"2023-10-16T14:10:40.970323+0000","last_fullsized":"2023-10-16T14:10:40.970323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:38.914798+0000","last_clean_scrub_stamp":"2023-10-16T14:10:38.914798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:40.537056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.186986+0000","last_change":"2023-10-16T14:10:40.995949+0000","last_active":"2023-10-16T14:10:43.186986+0000","last_peered":"2023-10-16T14:10:43.186986+0000","last_clean":"2023-10-16T14:10:43.186986+0000","last_became_active":"2023-10-16T14:10:40.995689+0000","last_became_peered":"2023-10-16T14:10:40.995689+0000","last_unstale":"2023-10-16T14:10:43.186986+0000","last_undegraded":"2023-10-16T14:10:43.186986+0000","last_fullsized":"2023-10-16T14:10:43.186986+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:12.855676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977098+0000","last_change":"2023-10-16T14:10:26.840800+0000","last_active":"2023-10-16T14:10:40.977098+0000","last_peered":"2023-10-16T14:10:40.977098+0000","last_clean":"2023-10-16T14:10:40.977098+0000","last_became_active":"2023-10-16T14:10:26.840526+0000","last_became_peered":"2023-10-16T14:10:26.840526+0000","last_unstale":"2023-10-16T14:10:40.977098+0000","last_undegraded":"2023-10-16T14:10:40.977098+0000","last_fullsized":"2023-10-16T14:10:40.977098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:28:20.752925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244569+0000","last_change":"2023-10-16T14:10:41.001647+0000","last_active":"2023-10-16T14:10:43.244569+0000","last_peered":"2023-10-16T14:10:43.244569+0000","last_clean":"2023-10-16T14:10:43.244569+0000","last_became_active":"2023-10-16T14:10:41.000840+0000","last_became_peered":"2023-10-16T14:10:41.000840+0000","last_unstale":"2023-10-16T14:10:43.244569+0000","last_undegraded":"2023-10-16T14:10:43.244569+0000","last_fullsized":"2023-10-16T14:10:43.244569+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:55.904685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959155+0000","last_change":"2023-10-16T14:10:26.831202+0000","last_active":"2023-10-16T14:10:40.959155+0000","last_peered":"2023-10-16T14:10:40.959155+0000","last_clean":"2023-10-16T14:10:40.959155+0000","last_became_active":"2023-10-16T14:10:26.830804+0000","last_became_peered":"2023-10-16T14:10:26.830804+0000","last_unstale":"2023-10-16T14:10:40.959155+0000","last_undegraded":"2023-10-16T14:10:40.959155+0000","last_fullsized":"2023-10-16T14:10:40.959155+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:06.592664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243482+0000","last_change":"2023-10-16T14:10:40.991703+0000","last_active":"2023-10-16T14:10:43.243482+0000","last_peered":"2023-10-16T14:10:43.243482+0000","last_clean":"2023-10-16T14:10:43.243482+0000","last_became_active":"2023-10-16T14:10:40.991364+0000","last_became_peered":"2023-10-16T14:10:40.991364+0000","last_unstale":"2023-10-16T14:10:43.243482+0000","last_undegraded":"2023-10-16T14:10:43.243482+0000","last_fullsized":"2023-10-16T14:10:43.243482+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:30.238400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.971730+0000","last_change":"2023-10-16T14:10:26.837636+0000","last_active":"2023-10-16T14:10:40.971730+0000","last_peered":"2023-10-16T14:10:40.971730+0000","last_clean":"2023-10-16T14:10:40.971730+0000","last_became_active":"2023-10-16T14:10:26.837329+0000","last_became_peered":"2023-10-16T14:10:26.837329+0000","last_unstale":"2023-10-16T14:10:40.971730+0000","last_undegraded":"2023-10-16T14:10:40.971730+0000","last_fullsized":"2023-10-16T14:10:40.971730+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:48.706591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244638+0000","last_change":"2023-10-16T14:10:40.993410+0000","last_active":"2023-10-16T14:10:43.244638+0000","last_peered":"2023-10-16T14:10:43.244638+0000","last_clean":"2023-10-16T14:10:43.244638+0000","last_became_active":"2023-10-16T14:10:40.992264+0000","last_became_peered":"2023-10-16T14:10:40.992264+0000","last_unstale":"2023-10-16T14:10:43.244638+0000","last_undegraded":"2023-10-16T14:10:43.244638+0000","last_fullsized":"2023-10-16T14:10:43.244638+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:32.614992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.967372+0000","last_change":"2023-10-16T14:10:26.846001+0000","last_active":"2023-10-16T14:10:40.967372+0000","last_peered":"2023-10-16T14:10:40.967372+0000","last_clean":"2023-10-16T14:10:40.967372+0000","last_became_active":"2023-10-16T14:10:26.845229+0000","last_became_peered":"2023-10-16T14:10:26.845229+0000","last_unstale":"2023-10-16T14:10:40.967372+0000","last_undegraded":"2023-10-16T14:10:40.967372+0000","last_fullsized":"2023-10-16T14:10:40.967372+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:15.730062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242825+0000","last_change":"2023-10-16T14:10:40.989459+0000","last_active":"2023-10-16T14:10:43.242825+0000","last_peered":"2023-10-16T14:10:43.242825+0000","last_clean":"2023-10-16T14:10:43.242825+0000","last_became_active":"2023-10-16T14:10:40.989197+0000","last_became_peered":"2023-10-16T14:10:40.989197+0000","last_unstale":"2023-10-16T14:10:43.242825+0000","last_undegraded":"2023-10-16T14:10:43.242825+0000","last_fullsized":"2023-10-16T14:10:43.242825+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.979422+0000","last_change":"2023-10-16T14:10:26.840057+0000","last_active":"2023-10-16T14:10:40.979422+0000","last_peered":"2023-10-16T14:10:40.979422+0000","last_clean":"2023-10-16T14:10:40.979422+0000","last_became_active":"2023-10-16T14:10:26.839413+0000","last_became_peered":"2023-10-16T14:10:26.839413+0000","last_unstale":"2023-10-16T14:10:40.979422+0000","last_undegraded":"2023-10-16T14:10:40.979422+0000","last_fullsized":"2023-10-16T14:10:40.979422+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:27.395876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994059+0000","last_change":"2023-10-16T14:10:40.986689+0000","last_active":"2023-10-16T14:10:41.994059+0000","last_peered":"2023-10-16T14:10:41.994059+0000","last_clean":"2023-10-16T14:10:41.994059+0000","last_became_active":"2023-10-16T14:10:40.986408+0000","last_became_peered":"2023-10-16T14:10:40.986408+0000","last_unstale":"2023-10-16T14:10:41.994059+0000","last_undegraded":"2023-10-16T14:10:41.994059+0000","last_fullsized":"2023-10-16T14:10:41.994059+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:43.120897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.975431+0000","last_change":"2023-10-16T14:10:26.845912+0000","last_active":"2023-10-16T14:10:40.975431+0000","last_peered":"2023-10-16T14:10:40.975431+0000","last_clean":"2023-10-16T14:10:40.975431+0000","last_became_active":"2023-10-16T14:10:26.844770+0000","last_became_peered":"2023-10-16T14:10:26.844770+0000","last_unstale":"2023-10-16T14:10:40.975431+0000","last_undegraded":"2023-10-16T14:10:40.975431+0000","last_fullsized":"2023-10-16T14:10:40.975431+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:43.937633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989882+0000","last_change":"2023-10-16T14:10:40.987586+0000","last_active":"2023-10-16T14:10:41.989882+0000","last_peered":"2023-10-16T14:10:41.989882+0000","last_clean":"2023-10-16T14:10:41.989882+0000","last_became_active":"2023-10-16T14:10:40.987254+0000","last_became_peered":"2023-10-16T14:10:40.987254+0000","last_unstale":"2023-10-16T14:10:41.989882+0000","last_undegraded":"2023-10-16T14:10:41.989882+0000","last_fullsized":"2023-10-16T14:10:41.989882+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.463597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980939+0000","last_change":"2023-10-16T14:10:26.855136+0000","last_active":"2023-10-16T14:10:40.980939+0000","last_peered":"2023-10-16T14:10:40.980939+0000","last_clean":"2023-10-16T14:10:40.980939+0000","last_became_active":"2023-10-16T14:10:26.853662+0000","last_became_peered":"2023-10-16T14:10:26.853662+0000","last_unstale":"2023-10-16T14:10:40.980939+0000","last_undegraded":"2023-10-16T14:10:40.980939+0000","last_fullsized":"2023-10-16T14:10:40.980939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:32.669577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124331+0000","last_change":"2023-10-16T14:10:40.990967+0000","last_active":"2023-10-16T14:10:42.124331+0000","last_peered":"2023-10-16T14:10:42.124331+0000","last_clean":"2023-10-16T14:10:42.124331+0000","last_became_active":"2023-10-16T14:10:40.989709+0000","last_became_peered":"2023-10-16T14:10:40.989709+0000","last_unstale":"2023-10-16T14:10:42.124331+0000","last_undegraded":"2023-10-16T14:10:42.124331+0000","last_fullsized":"2023-10-16T14:10:42.124331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:59.888217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974374+0000","last_change":"2023-10-16T14:10:26.848409+0000","last_active":"2023-10-16T14:10:40.974374+0000","last_peered":"2023-10-16T14:10:40.974374+0000","last_clean":"2023-10-16T14:10:40.974374+0000","last_became_active":"2023-10-16T14:10:26.848149+0000","last_became_peered":"2023-10-16T14:10:26.848149+0000","last_unstale":"2023-10-16T14:10:40.974374+0000","last_undegraded":"2023-10-16T14:10:40.974374+0000","last_fullsized":"2023-10-16T14:10:40.974374+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244140+0000","last_change":"2023-10-16T14:10:40.985097+0000","last_active":"2023-10-16T14:10:43.244140+0000","last_peered":"2023-10-16T14:10:43.244140+0000","last_clean":"2023-10-16T14:10:43.244140+0000","last_became_active":"2023-10-16T14:10:40.984409+0000","last_became_peered":"2023-10-16T14:10:40.984409+0000","last_unstale":"2023-10-16T14:10:43.244140+0000","last_undegraded":"2023-10-16T14:10:43.244140+0000","last_fullsized":"2023-10-16T14:10:43.244140+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:29.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972580+0000","last_change":"2023-10-16T14:10:26.840940+0000","last_active":"2023-10-16T14:10:40.972580+0000","last_peered":"2023-10-16T14:10:40.972580+0000","last_clean":"2023-10-16T14:10:40.972580+0000","last_became_active":"2023-10-16T14:10:26.840513+0000","last_became_peered":"2023-10-16T14:10:26.840513+0000","last_unstale":"2023-10-16T14:10:40.972580+0000","last_undegraded":"2023-10-16T14:10:40.972580+0000","last_fullsized":"2023-10-16T14:10:40.972580+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:27.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996436+0000","last_change":"2023-10-16T14:10:40.988348+0000","last_active":"2023-10-16T14:10:41.996436+0000","last_peered":"2023-10-16T14:10:41.996436+0000","last_clean":"2023-10-16T14:10:41.996436+0000","last_became_active":"2023-10-16T14:10:40.988060+0000","last_became_peered":"2023-10-16T14:10:40.988060+0000","last_unstale":"2023-10-16T14:10:41.996436+0000","last_undegraded":"2023-10-16T14:10:41.996436+0000","last_fullsized":"2023-10-16T14:10:41.996436+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.988939+0000","last_change":"2023-10-16T14:10:26.854195+0000","last_active":"2023-10-16T14:10:40.988939+0000","last_peered":"2023-10-16T14:10:40.988939+0000","last_clean":"2023-10-16T14:10:40.988939+0000","last_became_active":"2023-10-16T14:10:26.853411+0000","last_became_peered":"2023-10-16T14:10:26.853411+0000","last_unstale":"2023-10-16T14:10:40.988939+0000","last_undegraded":"2023-10-16T14:10:40.988939+0000","last_fullsized":"2023-10-16T14:10:40.988939+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:34.329659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243319+0000","last_change":"2023-10-16T14:10:40.987085+0000","last_active":"2023-10-16T14:10:43.243319+0000","last_peered":"2023-10-16T14:10:43.243319+0000","last_clean":"2023-10-16T14:10:43.243319+0000","last_became_active":"2023-10-16T14:10:40.986831+0000","last_became_peered":"2023-10-16T14:10:40.986831+0000","last_unstale":"2023-10-16T14:10:43.243319+0000","last_undegraded":"2023-10-16T14:10:43.243319+0000","last_fullsized":"2023-10-16T14:10:43.243319+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:58.536370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.987030+0000","last_change":"2023-10-16T14:10:26.831572+0000","last_active":"2023-10-16T14:10:40.987030+0000","last_peered":"2023-10-16T14:10:40.987030+0000","last_clean":"2023-10-16T14:10:40.987030+0000","last_became_active":"2023-10-16T14:10:26.831044+0000","last_became_peered":"2023-10-16T14:10:26.831044+0000","last_unstale":"2023-10-16T14:10:40.987030+0000","last_undegraded":"2023-10-16T14:10:40.987030+0000","last_fullsized":"2023-10-16T14:10:40.987030+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125634+0000","last_change":"2023-10-16T14:10:40.998279+0000","last_active":"2023-10-16T14:10:42.125634+0000","last_peered":"2023-10-16T14:10:42.125634+0000","last_clean":"2023-10-16T14:10:42.125634+0000","last_became_active":"2023-10-16T14:10:40.997899+0000","last_became_peered":"2023-10-16T14:10:40.997899+0000","last_unstale":"2023-10-16T14:10:42.125634+0000","last_undegraded":"2023-10-16T14:10:42.125634+0000","last_fullsized":"2023-10-16T14:10:42.125634+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:02.766931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977306+0000","last_change":"2023-10-16T14:10:26.844363+0000","last_active":"2023-10-16T14:10:40.977306+0000","last_peered":"2023-10-16T14:10:40.977306+0000","last_clean":"2023-10-16T14:10:40.977306+0000","last_became_active":"2023-10-16T14:10:26.842635+0000","last_became_peered":"2023-10-16T14:10:26.842635+0000","last_unstale":"2023-10-16T14:10:40.977306+0000","last_undegraded":"2023-10-16T14:10:40.977306+0000","last_fullsized":"2023-10-16T14:10:40.977306+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:21.570482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.991518+0000","last_change":"2023-10-16T14:10:40.992264+0000","last_active":"2023-10-16T14:10:41.991518+0000","last_peered":"2023-10-16T14:10:41.991518+0000","last_clean":"2023-10-16T14:10:41.991518+0000","last_became_active":"2023-10-16T14:10:40.991698+0000","last_became_peered":"2023-10-16T14:10:40.991698+0000","last_unstale":"2023-10-16T14:10:41.991518+0000","last_undegraded":"2023-10-16T14:10:41.991518+0000","last_fullsized":"2023-10-16T14:10:41.991518+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:16.407382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978952+0000","last_change":"2023-10-16T14:10:26.852615+0000","last_active":"2023-10-16T14:10:40.978952+0000","last_peered":"2023-10-16T14:10:40.978952+0000","last_clean":"2023-10-16T14:10:40.978952+0000","last_became_active":"2023-10-16T14:10:26.852338+0000","last_became_peered":"2023-10-16T14:10:26.852338+0000","last_unstale":"2023-10-16T14:10:40.978952+0000","last_undegraded":"2023-10-16T14:10:40.978952+0000","last_fullsized":"2023-10-16T14:10:40.978952+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:32.297261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125182+0000","last_change":"2023-10-16T14:10:40.991982+0000","last_active":"2023-10-16T14:10:42.125182+0000","last_peered":"2023-10-16T14:10:42.125182+0000","last_clean":"2023-10-16T14:10:42.125182+0000","last_became_active":"2023-10-16T14:10:40.991521+0000","last_became_peered":"2023-10-16T14:10:40.991521+0000","last_unstale":"2023-10-16T14:10:42.125182+0000","last_undegraded":"2023-10-16T14:10:42.125182+0000","last_fullsized":"2023-10-16T14:10:42.125182+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:06:01.654954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.961133+0000","last_change":"2023-10-16T14:10:26.849025+0000","last_active":"2023-10-16T14:10:40.961133+0000","last_peered":"2023-10-16T14:10:40.961133+0000","last_clean":"2023-10-16T14:10:40.961133+0000","last_became_active":"2023-10-16T14:10:26.848746+0000","last_became_peered":"2023-10-16T14:10:26.848746+0000","last_unstale":"2023-10-16T14:10:40.961133+0000","last_undegraded":"2023-10-16T14:10:40.961133+0000","last_fullsized":"2023-10-16T14:10:40.961133+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:21.453385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124085+0000","last_change":"2023-10-16T14:10:40.992568+0000","last_active":"2023-10-16T14:10:42.124085+0000","last_peered":"2023-10-16T14:10:42.124085+0000","last_clean":"2023-10-16T14:10:42.124085+0000","last_became_active":"2023-10-16T14:10:40.992252+0000","last_became_peered":"2023-10-16T14:10:40.992252+0000","last_unstale":"2023-10-16T14:10:42.124085+0000","last_undegraded":"2023-10-16T14:10:42.124085+0000","last_fullsized":"2023-10-16T14:10:42.124085+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:25.262205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965644+0000","last_change":"2023-10-16T14:10:26.840986+0000","last_active":"2023-10-16T14:10:40.965644+0000","last_peered":"2023-10-16T14:10:40.965644+0000","last_clean":"2023-10-16T14:10:40.965644+0000","last_became_active":"2023-10-16T14:10:26.840708+0000","last_became_peered":"2023-10-16T14:10:26.840708+0000","last_unstale":"2023-10-16T14:10:40.965644+0000","last_undegraded":"2023-10-16T14:10:40.965644+0000","last_fullsized":"2023-10-16T14:10:40.965644+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:51.988310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.242786+0000","last_change":"2023-10-16T14:10:40.991785+0000","last_active":"2023-10-16T14:10:43.242786+0000","last_peered":"2023-10-16T14:10:43.242786+0000","last_clean":"2023-10-16T14:10:43.242786+0000","last_became_active":"2023-10-16T14:10:40.991292+0000","last_became_peered":"2023-10-16T14:10:40.991292+0000","last_unstale":"2023-10-16T14:10:43.242786+0000","last_undegraded":"2023-10-16T14:10:43.242786+0000","last_fullsized":"2023-10-16T14:10:43.242786+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:39.355417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978297+0000","last_change":"2023-10-16T14:10:26.854511+0000","last_active":"2023-10-16T14:10:40.978297+0000","last_peered":"2023-10-16T14:10:40.978297+0000","last_clean":"2023-10-16T14:10:40.978297+0000","last_became_active":"2023-10-16T14:10:26.854106+0000","last_became_peered":"2023-10-16T14:10:26.854106+0000","last_unstale":"2023-10-16T14:10:40.978297+0000","last_undegraded":"2023-10-16T14:10:40.978297+0000","last_fullsized":"2023-10-16T14:10:40.978297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:36.511252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243833+0000","last_change":"2023-10-16T14:10:40.985314+0000","last_active":"2023-10-16T14:10:43.243833+0000","last_peered":"2023-10-16T14:10:43.243833+0000","last_clean":"2023-10-16T14:10:43.243833+0000","last_became_active":"2023-10-16T14:10:40.984808+0000","last_became_peered":"2023-10-16T14:10:40.984808+0000","last_unstale":"2023-10-16T14:10:43.243833+0000","last_undegraded":"2023-10-16T14:10:43.243833+0000","last_fullsized":"2023-10-16T14:10:43.243833+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:03:06.220351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.959800+0000","last_change":"2023-10-16T14:10:26.849677+0000","last_active":"2023-10-16T14:10:40.959800+0000","last_peered":"2023-10-16T14:10:40.959800+0000","last_clean":"2023-10-16T14:10:40.959800+0000","last_became_active":"2023-10-16T14:10:26.849406+0000","last_became_peered":"2023-10-16T14:10:26.849406+0000","last_unstale":"2023-10-16T14:10:40.959800+0000","last_undegraded":"2023-10-16T14:10:40.959800+0000","last_fullsized":"2023-10-16T14:10:40.959800+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.144022+0000","last_change":"2023-10-16T14:10:40.995986+0000","last_active":"2023-10-16T14:10:43.144022+0000","last_peered":"2023-10-16T14:10:43.144022+0000","last_clean":"2023-10-16T14:10:43.144022+0000","last_became_active":"2023-10-16T14:10:40.994817+0000","last_became_peered":"2023-10-16T14:10:40.994817+0000","last_unstale":"2023-10-16T14:10:43.144022+0000","last_undegraded":"2023-10-16T14:10:43.144022+0000","last_fullsized":"2023-10-16T14:10:43.144022+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:17.570998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978097+0000","last_change":"2023-10-16T14:10:26.842774+0000","last_active":"2023-10-16T14:10:40.978097+0000","last_peered":"2023-10-16T14:10:40.978097+0000","last_clean":"2023-10-16T14:10:40.978097+0000","last_became_active":"2023-10-16T14:10:26.842422+0000","last_became_peered":"2023-10-16T14:10:26.842422+0000","last_unstale":"2023-10-16T14:10:40.978097+0000","last_undegraded":"2023-10-16T14:10:40.978097+0000","last_fullsized":"2023-10-16T14:10:40.978097+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:09.231732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.997726+0000","last_change":"2023-10-16T14:10:41.001565+0000","last_active":"2023-10-16T14:10:41.997726+0000","last_peered":"2023-10-16T14:10:41.997726+0000","last_clean":"2023-10-16T14:10:41.997726+0000","last_became_active":"2023-10-16T14:10:40.999604+0000","last_became_peered":"2023-10-16T14:10:40.999604+0000","last_unstale":"2023-10-16T14:10:41.997726+0000","last_undegraded":"2023-10-16T14:10:41.997726+0000","last_fullsized":"2023-10-16T14:10:41.997726+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:19.596851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.980317+0000","last_change":"2023-10-16T14:10:26.853398+0000","last_active":"2023-10-16T14:10:40.980317+0000","last_peered":"2023-10-16T14:10:40.980317+0000","last_clean":"2023-10-16T14:10:40.980317+0000","last_became_active":"2023-10-16T14:10:26.852825+0000","last_became_peered":"2023-10-16T14:10:26.852825+0000","last_unstale":"2023-10-16T14:10:40.980317+0000","last_undegraded":"2023-10-16T14:10:40.980317+0000","last_fullsized":"2023-10-16T14:10:40.980317+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:28.717423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.243931+0000","last_change":"2023-10-16T14:10:40.994995+0000","last_active":"2023-10-16T14:10:43.243931+0000","last_peered":"2023-10-16T14:10:43.243931+0000","last_clean":"2023-10-16T14:10:43.243931+0000","last_became_active":"2023-10-16T14:10:40.994421+0000","last_became_peered":"2023-10-16T14:10:40.994421+0000","last_unstale":"2023-10-16T14:10:43.243931+0000","last_undegraded":"2023-10-16T14:10:43.243931+0000","last_fullsized":"2023-10-16T14:10:43.243931+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:39.108794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968652+0000","last_change":"2023-10-16T14:10:26.852808+0000","last_active":"2023-10-16T14:10:40.968652+0000","last_peered":"2023-10-16T14:10:40.968652+0000","last_clean":"2023-10-16T14:10:40.968652+0000","last_became_active":"2023-10-16T14:10:26.852409+0000","last_became_peered":"2023-10-16T14:10:26.852409+0000","last_unstale":"2023-10-16T14:10:40.968652+0000","last_undegraded":"2023-10-16T14:10:40.968652+0000","last_fullsized":"2023-10-16T14:10:40.968652+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:26.272968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.995533+0000","last_change":"2023-10-16T14:10:40.988474+0000","last_active":"2023-10-16T14:10:41.995533+0000","last_peered":"2023-10-16T14:10:41.995533+0000","last_clean":"2023-10-16T14:10:41.995533+0000","last_became_active":"2023-10-16T14:10:40.988193+0000","last_became_peered":"2023-10-16T14:10:40.988193+0000","last_unstale":"2023-10-16T14:10:41.995533+0000","last_undegraded":"2023-10-16T14:10:41.995533+0000","last_fullsized":"2023-10-16T14:10:41.995533+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:56:40.550827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.972255+0000","last_change":"2023-10-16T14:10:26.844758+0000","last_active":"2023-10-16T14:10:40.972255+0000","last_peered":"2023-10-16T14:10:40.972255+0000","last_clean":"2023-10-16T14:10:40.972255+0000","last_became_active":"2023-10-16T14:10:26.844101+0000","last_became_peered":"2023-10-16T14:10:26.844101+0000","last_unstale":"2023-10-16T14:10:40.972255+0000","last_undegraded":"2023-10-16T14:10:40.972255+0000","last_fullsized":"2023-10-16T14:10:40.972255+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:05.186025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.994820+0000","last_change":"2023-10-16T14:10:40.994079+0000","last_active":"2023-10-16T14:10:41.994820+0000","last_peered":"2023-10-16T14:10:41.994820+0000","last_clean":"2023-10-16T14:10:41.994820+0000","last_became_active":"2023-10-16T14:10:40.991884+0000","last_became_peered":"2023-10-16T14:10:40.991884+0000","last_unstale":"2023-10-16T14:10:41.994820+0000","last_undegraded":"2023-10-16T14:10:41.994820+0000","last_fullsized":"2023-10-16T14:10:41.994820+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:14.260538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981697+0000","last_change":"2023-10-16T14:10:26.845441+0000","last_active":"2023-10-16T14:10:40.981697+0000","last_peered":"2023-10-16T14:10:40.981697+0000","last_clean":"2023-10-16T14:10:40.981697+0000","last_became_active":"2023-10-16T14:10:26.845116+0000","last_became_peered":"2023-10-16T14:10:26.845116+0000","last_unstale":"2023-10-16T14:10:40.981697+0000","last_undegraded":"2023-10-16T14:10:40.981697+0000","last_fullsized":"2023-10-16T14:10:40.981697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:03.460043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.993382+0000","last_change":"2023-10-16T14:10:41.002135+0000","last_active":"2023-10-16T14:10:41.993382+0000","last_peered":"2023-10-16T14:10:41.993382+0000","last_clean":"2023-10-16T14:10:41.993382+0000","last_became_active":"2023-10-16T14:10:41.000234+0000","last_became_peered":"2023-10-16T14:10:41.000234+0000","last_unstale":"2023-10-16T14:10:41.993382+0000","last_undegraded":"2023-10-16T14:10:41.993382+0000","last_fullsized":"2023-10-16T14:10:41.993382+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:09.588728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.965582+0000","last_change":"2023-10-16T14:10:26.846184+0000","last_active":"2023-10-16T14:10:40.965582+0000","last_peered":"2023-10-16T14:10:40.965582+0000","last_clean":"2023-10-16T14:10:40.965582+0000","last_became_active":"2023-10-16T14:10:26.845893+0000","last_became_peered":"2023-10-16T14:10:26.845893+0000","last_unstale":"2023-10-16T14:10:40.965582+0000","last_undegraded":"2023-10-16T14:10:40.965582+0000","last_fullsized":"2023-10-16T14:10:40.965582+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:58.479368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.989881+0000","last_change":"2023-10-16T14:10:40.987206+0000","last_active":"2023-10-16T14:10:41.989881+0000","last_peered":"2023-10-16T14:10:41.989881+0000","last_clean":"2023-10-16T14:10:41.989881+0000","last_became_active":"2023-10-16T14:10:40.986211+0000","last_became_peered":"2023-10-16T14:10:40.986211+0000","last_unstale":"2023-10-16T14:10:41.989881+0000","last_undegraded":"2023-10-16T14:10:41.989881+0000","last_fullsized":"2023-10-16T14:10:41.989881+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:16.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.974724+0000","last_change":"2023-10-16T14:10:26.847070+0000","last_active":"2023-10-16T14:10:40.974724+0000","last_peered":"2023-10-16T14:10:40.974724+0000","last_clean":"2023-10-16T14:10:40.974724+0000","last_became_active":"2023-10-16T14:10:26.845515+0000","last_became_peered":"2023-10-16T14:10:26.845515+0000","last_unstale":"2023-10-16T14:10:40.974724+0000","last_undegraded":"2023-10-16T14:10:40.974724+0000","last_fullsized":"2023-10-16T14:10:40.974724+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.614594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:41.996483+0000","last_change":"2023-10-16T14:10:40.998615+0000","last_active":"2023-10-16T14:10:41.996483+0000","last_peered":"2023-10-16T14:10:41.996483+0000","last_clean":"2023-10-16T14:10:41.996483+0000","last_became_active":"2023-10-16T14:10:40.997677+0000","last_became_peered":"2023-10-16T14:10:40.997677+0000","last_unstale":"2023-10-16T14:10:41.996483+0000","last_undegraded":"2023-10-16T14:10:41.996483+0000","last_fullsized":"2023-10-16T14:10:41.996483+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:17.617854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968589+0000","last_change":"2023-10-16T14:10:26.851340+0000","last_active":"2023-10-16T14:10:40.968589+0000","last_peered":"2023-10-16T14:10:40.968589+0000","last_clean":"2023-10-16T14:10:40.968589+0000","last_became_active":"2023-10-16T14:10:26.851056+0000","last_became_peered":"2023-10-16T14:10:26.851056+0000","last_unstale":"2023-10-16T14:10:40.968589+0000","last_undegraded":"2023-10-16T14:10:40.968589+0000","last_fullsized":"2023-10-16T14:10:40.968589+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:36.647772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.244429+0000","last_change":"2023-10-16T14:10:41.001901+0000","last_active":"2023-10-16T14:10:43.244429+0000","last_peered":"2023-10-16T14:10:43.244429+0000","last_clean":"2023-10-16T14:10:43.244429+0000","last_became_active":"2023-10-16T14:10:41.001390+0000","last_became_peered":"2023-10-16T14:10:41.001390+0000","last_unstale":"2023-10-16T14:10:43.244429+0000","last_undegraded":"2023-10-16T14:10:43.244429+0000","last_fullsized":"2023-10-16T14:10:43.244429+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:38.275860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.977891+0000","last_change":"2023-10-16T14:10:26.847022+0000","last_active":"2023-10-16T14:10:40.977891+0000","last_peered":"2023-10-16T14:10:40.977891+0000","last_clean":"2023-10-16T14:10:40.977891+0000","last_became_active":"2023-10-16T14:10:26.845879+0000","last_became_peered":"2023-10-16T14:10:26.845879+0000","last_unstale":"2023-10-16T14:10:40.977891+0000","last_undegraded":"2023-10-16T14:10:40.977891+0000","last_fullsized":"2023-10-16T14:10:40.977891+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:29.611811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.124855+0000","last_change":"2023-10-16T14:10:40.990936+0000","last_active":"2023-10-16T14:10:42.124855+0000","last_peered":"2023-10-16T14:10:42.124855+0000","last_clean":"2023-10-16T14:10:42.124855+0000","last_became_active":"2023-10-16T14:10:40.990264+0000","last_became_peered":"2023-10-16T14:10:40.990264+0000","last_unstale":"2023-10-16T14:10:42.124855+0000","last_undegraded":"2023-10-16T14:10:42.124855+0000","last_fullsized":"2023-10-16T14:10:42.124855+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:39.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973916+0000","last_change":"2023-10-16T14:10:26.841296+0000","last_active":"2023-10-16T14:10:40.973916+0000","last_peered":"2023-10-16T14:10:40.973916+0000","last_clean":"2023-10-16T14:10:40.973916+0000","last_became_active":"2023-10-16T14:10:26.841021+0000","last_became_peered":"2023-10-16T14:10:26.841021+0000","last_unstale":"2023-10-16T14:10:40.973916+0000","last_undegraded":"2023-10-16T14:10:40.973916+0000","last_fullsized":"2023-10-16T14:10:40.973916+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:20:05.439040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.143331+0000","last_change":"2023-10-16T14:10:40.998298+0000","last_active":"2023-10-16T14:10:43.143331+0000","last_peered":"2023-10-16T14:10:43.143331+0000","last_clean":"2023-10-16T14:10:43.143331+0000","last_became_active":"2023-10-16T14:10:40.997963+0000","last_became_peered":"2023-10-16T14:10:40.997963+0000","last_unstale":"2023-10-16T14:10:43.143331+0000","last_undegraded":"2023-10-16T14:10:43.143331+0000","last_fullsized":"2023-10-16T14:10:43.143331+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:12.160466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.973460+0000","last_change":"2023-10-16T14:10:26.841873+0000","last_active":"2023-10-16T14:10:40.973460+0000","last_peered":"2023-10-16T14:10:40.973460+0000","last_clean":"2023-10-16T14:10:40.973460+0000","last_became_active":"2023-10-16T14:10:26.841602+0000","last_became_peered":"2023-10-16T14:10:26.841602+0000","last_unstale":"2023-10-16T14:10:40.973460+0000","last_undegraded":"2023-10-16T14:10:40.973460+0000","last_fullsized":"2023-10-16T14:10:40.973460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:16:08.784937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.241703+0000","last_change":"2023-10-16T14:10:40.982851+0000","last_active":"2023-10-16T14:10:43.241703+0000","last_peered":"2023-10-16T14:10:43.241703+0000","last_clean":"2023-10-16T14:10:43.241703+0000","last_became_active":"2023-10-16T14:10:40.981261+0000","last_became_peered":"2023-10-16T14:10:40.981261+0000","last_unstale":"2023-10-16T14:10:43.241703+0000","last_undegraded":"2023-10-16T14:10:43.241703+0000","last_fullsized":"2023-10-16T14:10:43.241703+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:03.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.982369+0000","last_change":"2023-10-16T14:10:26.829330+0000","last_active":"2023-10-16T14:10:40.982369+0000","last_peered":"2023-10-16T14:10:40.982369+0000","last_clean":"2023-10-16T14:10:40.982369+0000","last_became_active":"2023-10-16T14:10:26.828786+0000","last_became_peered":"2023-10-16T14:10:26.828786+0000","last_unstale":"2023-10-16T14:10:40.982369+0000","last_undegraded":"2023-10-16T14:10:40.982369+0000","last_fullsized":"2023-10-16T14:10:40.982369+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:25.492433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:42.125338+0000","last_change":"2023-10-16T14:10:40.991807+0000","last_active":"2023-10-16T14:10:42.125338+0000","last_peered":"2023-10-16T14:10:42.125338+0000","last_clean":"2023-10-16T14:10:42.125338+0000","last_became_active":"2023-10-16T14:10:40.991143+0000","last_became_peered":"2023-10-16T14:10:40.991143+0000","last_unstale":"2023-10-16T14:10:42.125338+0000","last_undegraded":"2023-10-16T14:10:42.125338+0000","last_fullsized":"2023-10-16T14:10:42.125338+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:32.405248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.981437+0000","last_change":"2023-10-16T14:10:26.842085+0000","last_active":"2023-10-16T14:10:40.981437+0000","last_peered":"2023-10-16T14:10:40.981437+0000","last_clean":"2023-10-16T14:10:40.981437+0000","last_became_active":"2023-10-16T14:10:26.841273+0000","last_became_peered":"2023-10-16T14:10:26.841273+0000","last_unstale":"2023-10-16T14:10:40.981437+0000","last_undegraded":"2023-10-16T14:10:40.981437+0000","last_fullsized":"2023-10-16T14:10:40.981437+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:00.102728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.185397+0000","last_change":"2023-10-16T14:10:40.997929+0000","last_active":"2023-10-16T14:10:43.185397+0000","last_peered":"2023-10-16T14:10:43.185397+0000","last_clean":"2023-10-16T14:10:43.185397+0000","last_became_active":"2023-10-16T14:10:40.997578+0000","last_became_peered":"2023-10-16T14:10:40.997578+0000","last_unstale":"2023-10-16T14:10:43.185397+0000","last_undegraded":"2023-10-16T14:10:43.185397+0000","last_fullsized":"2023-10-16T14:10:43.185397+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:31.595742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.968740+0000","last_change":"2023-10-16T14:10:26.837973+0000","last_active":"2023-10-16T14:10:40.968740+0000","last_peered":"2023-10-16T14:10:40.968740+0000","last_clean":"2023-10-16T14:10:40.968740+0000","last_became_active":"2023-10-16T14:10:26.837715+0000","last_became_peered":"2023-10-16T14:10:26.837715+0000","last_unstale":"2023-10-16T14:10:40.968740+0000","last_undegraded":"2023-10-16T14:10:40.968740+0000","last_fullsized":"2023-10-16T14:10:40.968740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:31.677883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-10-16T14:10:43.141865+0000","last_change":"2023-10-16T14:10:40.982299+0000","last_active":"2023-10-16T14:10:43.141865+0000","last_peered":"2023-10-16T14:10:43.141865+0000","last_clean":"2023-10-16T14:10:43.141865+0000","last_became_active":"2023-10-16T14:10:40.982047+0000","last_became_peered":"2023-10-16T14:10:40.982047+0000","last_unstale":"2023-10-16T14:10:43.141865+0000","last_undegraded":"2023-10-16T14:10:43.141865+0000","last_fullsized":"2023-10-16T14:10:43.141865+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:39.931429+0000","last_clean_scrub_stamp":"2023-10-16T14:10:39.931429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:02:09.508655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-10-16T14:10:40.978187+0000","last_change":"2023-10-16T14:10:26.855883+0000","last_active":"2023-10-16T14:10:40.978187+0000","last_peered":"2023-10-16T14:10:40.978187+0000","last_clean":"2023-10-16T14:10:40.978187+0000","last_became_active":"2023-10-16T14:10:26.855535+0000","last_became_peered":"2023-10-16T14:10:26.855535+0000","last_unstale":"2023-10-16T14:10:40.978187+0000","last_undegraded":"2023-10-16T14:10:40.978187+0000","last_fullsized":"2023-10-16T14:10:40.978187+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:10:25.773618+0000","last_clean_scrub_stamp":"2023-10-16T14:10:25.773618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:02.192494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2155,"num_read_kb":4508,"num_write":583,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543302,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3551232,"data_stored":2031342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.373,"15min":1.369},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.7370000000000001,"5min":1.8180000000000001,"15min":2.0659999999999998},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.3959999999999999,"15min":1.409},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":2.0179999999999998},"last":0.98499999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2450000000000001,"15min":1.2490000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.69599999999999995},"max":{"1min":1.8049999999999999,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":1.117},{"interface":"front","average":{"1min":1.196,"5min":1.252,"15min":1.264},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.64400000000000002},"max":{"1min":1.893,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.84199999999999997}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.268,"15min":1.272},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.3879999999999999,"15min":1.369},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.75900000000000001},"max":{"1min":1.8400000000000001,"5min":2.028,"15min":2.028},"last":1.391}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4079999999999999,"15min":1.4159999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.7529999999999999,"5min":1.921,"15min":2.097},"last":1.242},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.94699999999999995,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7370000000000001,"5min":2.0169999999999999,"15min":2.1019999999999999},"last":1.577}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.244,"15min":1.248},"min":{"1min":0.81399999999999995,"5min":0.79100000000000004,"15min":0.76600000000000001},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.79300000000000004},"max":{"1min":1.982,"5min":1.982,"15min":2.0590000000000002},"last":1.6080000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.391,"15min":1.3879999999999999},"min":{"1min":0.97099999999999997,"5min":0.90800000000000003,"15min":0.879},"max":{"1min":1.7769999999999999,"5min":1.905,"15min":2.3730000000000002},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.536,"15min":1.5409999999999999},"min":{"1min":1.2410000000000001,"5min":0.98599999999999999,"15min":0.91500000000000004},"max":{"1min":1.9370000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.5369999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.6479999999999999,"15min":1.6419999999999999},"min":{"1min":1.3640000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":2.052,"5min":2.056,"15min":2.4169999999999998},"last":1.667},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5249999999999999,"15min":1.4910000000000001},"min":{"1min":1.0069999999999999,"5min":0.95499999999999996,"15min":0.81499999999999995},"max":{"1min":1.984,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.5069999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543303,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42388,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329452,"statfs":{"total":96636764160,"available":96593358848,"internally_reserved":0,"allocated":2953216,"data_stored":1438685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.67300000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.748,"5min":1.887,"15min":1.887},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.361,"15min":1.367},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.9390000000000001},"last":1.393}]},{"osd":1,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.214,"15min":1.2270000000000001},"min":{"1min":0.77900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2130000000000001,"15min":1.2330000000000001},"min":{"1min":0.82399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.587,"5min":1.768,"15min":1.77},"last":1.1140000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.736,"5min":1.823,"15min":1.823},"last":1.262},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.2969999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.333,"15min":1.347},"min":{"1min":0.89000000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.754,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.427},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4099999999999999,"15min":1.4119999999999999},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.3440000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.3819999999999999,"15min":1.407},"min":{"1min":1.131,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.9019999999999999,"5min":2.0529999999999999,"15min":2.0750000000000002},"last":1.488},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.5800000000000001,"15min":1.546},"min":{"1min":1.151,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":0.96699999999999997}]},{"osd":5,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4079999999999999,"15min":1.371},"min":{"1min":0.98099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1549999999999998},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.536,"5min":1.4790000000000001,"15min":1.4950000000000001},"min":{"1min":0.95799999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9770000000000001,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.2170000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5169999999999999,"15min":1.5049999999999999},"min":{"1min":1.0269999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9670000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.452},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.548,"15min":1.5429999999999999},"min":{"1min":1.3160000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.6120000000000001},"last":1.369}]}]},{"osd":5,"up_from":14,"seq":60129543304,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":2945024,"data_stored":1438629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.151,"15min":1.218},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4319999999999999,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.242},{"interface":"front","average":{"1min":1.246,"5min":1.234,"15min":1.242},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.4419999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.109},{"interface":"front","average":{"1min":1.109,"5min":1.202,"15min":1.266},"min":{"1min":0.755,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.621,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.159}]},{"osd":2,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.341,"15min":1.347},"min":{"1min":0.65400000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.827,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.034},{"interface":"front","average":{"1min":1.202,"5min":1.2769999999999999,"15min":1.3220000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.4790000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.399}]},{"osd":3,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.411,"15min":1.415},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.302},{"interface":"front","average":{"1min":1.391,"5min":1.421,"15min":1.4390000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.744,"5min":2.75,"15min":2.75},"last":1.542}]},{"osd":4,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.294,"15min":1.2889999999999999},"min":{"1min":0.746,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.77,"5min":2.774,"15min":2.774},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.4279999999999999},"min":{"1min":0.98699999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.3330000000000002},"last":1.212}]},{"osd":6,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.427},"min":{"1min":0.93999999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9199999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.486,"15min":1.49},"min":{"1min":1.0309999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.881,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.5920000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.581,"15min":1.554},"min":{"1min":1.0529999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1469999999999998,"5min":2.177,"15min":2.4609999999999999},"last":1.514},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.613,"15min":1.605},"min":{"1min":0.92900000000000005,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.851,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.655}]}]},{"osd":4,"up_from":14,"seq":60129543304,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47968,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323872,"statfs":{"total":96636764160,"available":96587644928,"internally_reserved":0,"allocated":2965504,"data_stored":1441590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.8420000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6699999999999999,"5min":1.8160000000000001,"15min":1.9019999999999999},"last":0.97899999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.71,"5min":1.71,"15min":1.78},"last":1.087},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2549999999999999,"15min":1.294},"min":{"1min":0.90700000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.754,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":0.88500000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.286,"15min":1.3169999999999999},"min":{"1min":0.80200000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.714,"5min":1.714,"15min":1.865},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.367,"15min":1.387},"min":{"1min":1.026,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.4450000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.296,"15min":1.3120000000000001},"min":{"1min":0.94599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.615,"5min":1.8740000000000001,"15min":1.9650000000000001},"last":1.139},{"interface":"front","average":{"1min":1.607,"5min":1.4590000000000001,"15min":1.4299999999999999},"min":{"1min":0.96499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":5.2350000000000003,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.486}]},{"osd":5,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.329,"15min":1.3400000000000001},"min":{"1min":0.99199999999999999,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3580000000000001,"15min":1.3360000000000001},"min":{"1min":1.0389999999999999,"5min":0.875,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":1.921,"15min":1.921},"last":1.782}]},{"osd":6,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3919999999999999,"15min":1.3879999999999999},"min":{"1min":1.004,"5min":0.90400000000000003,"15min":0.86699999999999999},"max":{"1min":1.7689999999999999,"5min":1.9159999999999999,"15min":1.956},"last":1.609},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.502,"15min":1.5229999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.546}]},{"osd":7,"last update":"Mon Oct 16 14:10:31 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5349999999999999,"15min":1.5049999999999999},"min":{"1min":1.137,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":2.1949999999999998},"last":1.897},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6379999999999999,"15min":1.6319999999999999},"min":{"1min":1.391,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":1.889,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.966}]}]},{"osd":3,"up_from":14,"seq":60129543304,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52128,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319712,"statfs":{"total":96636764160,"available":96583385088,"internally_reserved":0,"allocated":2969600,"data_stored":1439911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.91800000000000004,"15min":0.95299999999999996},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.52,"5min":1.52,"15min":1.5580000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.81200000000000006,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.798,"15min":2.0579999999999998},"last":0.93000000000000005}]},{"osd":1,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0660000000000001,"15min":1.0900000000000001},"min":{"1min":0.67100000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.474,"5min":1.855,"15min":1.9750000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.133,"15min":1.133},"min":{"1min":0.71899999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.944,"5min":2.077,"15min":2.077},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.1879999999999999,"15min":1.214},"min":{"1min":0.56399999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.546,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4359999999999999,"15min":1.464},"min":{"1min":0.84099999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.72,"5min":1.9319999999999999,"15min":2.1230000000000002},"last":1.427}]},{"osd":4,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5069999999999999,"15min":1.5129999999999999},"min":{"1min":0.97299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.48},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.448,"15min":1.464},"min":{"1min":1.0029999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2109999999999999},"last":1.337}]},{"osd":5,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4970000000000001,"15min":1.496},"min":{"1min":1.0920000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.048,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.611,"5min":1.5740000000000001,"15min":1.5649999999999999},"min":{"1min":1.1120000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9890000000000001,"5min":2.0249999999999999,"15min":2.3420000000000001},"last":1.179}]},{"osd":6,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.5600000000000001,"15min":1.536},"min":{"1min":1.018,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.1309999999999998,"5min":2.254,"15min":2.4329999999999998},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.51,"15min":1.506},"min":{"1min":1.153,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0059999999999998,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:09:46 2023","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.6140000000000001,"15min":1.619},"min":{"1min":1.304,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.968,"5min":2.1829999999999998,"15min":2.2320000000000002},"last":1.538},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6859999999999999,"15min":1.6839999999999999},"min":{"1min":1.349,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.218,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.0940000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543302,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2961408,"data_stored":1439715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.95899999999999996},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.597,"5min":1.597,"15min":2.0089999999999999},"last":0.87},{"interface":"front","average":{"1min":1.103,"5min":1.1220000000000001,"15min":1.1519999999999999},"min":{"1min":0.73599999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8100000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.012}]},{"osd":1,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.1819999999999999},"min":{"1min":0.68600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.651,"5min":1.847,"15min":1.847},"last":1.385},{"interface":"front","average":{"1min":1.244,"5min":1.258,"15min":1.254},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7210000000000001,"5min":1.8380000000000001,"15min":1.9530000000000001},"last":1.486}]},{"osd":3,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1299999999999999,"15min":1.1279999999999999},"min":{"1min":0.56799999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.415,"5min":2.1299999999999999,"15min":2.2719999999999998},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3620000000000001,"15min":1.385},"min":{"1min":0.76300000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.845,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.5269999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.49,"15min":1.512},"min":{"1min":0.91500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8540000000000001,"5min":2.0710000000000002,"15min":2.488},"last":1.806},{"interface":"front","average":{"1min":1.502,"5min":1.482,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.0019999999999998,"5min":2.0659999999999998,"15min":2.4180000000000001},"last":2.1789999999999998}]},{"osd":5,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.548,"15min":1.569},"min":{"1min":1.0349999999999999,"5min":0.995,"15min":0.995},"max":{"1min":2.1259999999999999,"5min":2.1819999999999999,"15min":2.5840000000000001},"last":1.855},{"interface":"front","average":{"1min":1.538,"5min":1.494,"15min":1.4810000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.1829999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.575}]},{"osd":6,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.542,"15min":1.5329999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.871,"5min":2.117,"15min":2.6400000000000001},"last":1.77},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.595,"15min":1.6319999999999999},"min":{"1min":1.1100000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.226,"5min":2.226,"15min":2.7749999999999999},"last":2.0960000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:10:13 2023","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.661,"15min":1.667},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.105,"5min":2.3540000000000001,"15min":2.7890000000000001},"last":2.0379999999999998},{"interface":"front","average":{"1min":1.665,"5min":1.6619999999999999,"15min":1.6559999999999999},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.0990000000000002,"5min":2.2799999999999998,"15min":2.9159999999999999},"last":1.4379999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543302,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2953216,"data_stored":1438193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95699999999999996,"15min":0.94899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1339999999999999,"15min":1.143},"min":{"1min":0.76600000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.847,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.1639999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.2270000000000001,"15min":1.266},"min":{"1min":0.82899999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":1.23}]},{"osd":3,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.391,"15min":1.403},"min":{"1min":0.78000000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.819,"5min":2.0659999999999998,"15min":2.4369999999999998},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.2569999999999999,"15min":1.29},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.399}]},{"osd":4,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3109999999999999,"15min":1.349},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4610000000000001,"5min":1.776,"15min":2.1259999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.524,"15min":1.5369999999999999},"min":{"1min":0.88,"5min":0.748,"15min":0.748},"max":{"1min":1.8759999999999999,"5min":2.133,"15min":2.3410000000000002},"last":1.45}]},{"osd":5,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.478,"15min":1.4970000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.9140000000000001,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.524},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4059999999999999,"15min":1.4810000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.734,"5min":1.8620000000000001,"15min":2.1440000000000001},"last":1.2789999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6220000000000001,"15min":1.6419999999999999},"min":{"1min":0.93799999999999994,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.99,"5min":2.2309999999999999,"15min":2.7200000000000002},"last":1.341},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.571,"15min":1.6140000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.651}]},{"osd":7,"last update":"Mon Oct 16 14:10:18 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6850000000000001,"15min":1.7110000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.585,"15min":1.633},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0539999999999998,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.4890000000000001}]}]},{"osd":0,"up_from":14,"seq":60129543303,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":3547136,"data_stored":2029177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.93300000000000005,"15min":0.93999999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.218,"5min":1.704,"15min":1.891},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0840000000000001,"15min":1.1579999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48699999999999999},"max":{"1min":1.627,"5min":1.659,"15min":1.9299999999999999},"last":1.206}]},{"osd":2,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.1499999999999999,"15min":1.171},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5349999999999999,"5min":1.7789999999999999,"15min":2.0569999999999999},"last":1.234},{"interface":"front","average":{"1min":1.087,"5min":1.1899999999999999,"15min":1.226},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.6539999999999999,"5min":1.893,"15min":1.893},"last":1.1259999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1639999999999999,"15min":1.163},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7390000000000001,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.077},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3380000000000001,"15min":1.3759999999999999},"min":{"1min":0.64700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.679,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.573}]},{"osd":4,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3340000000000001,"15min":1.3799999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6140000000000001,"5min":1.819,"15min":2.048},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.518,"15min":1.524},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.0009999999999999,"5min":2.601,"15min":2.601},"last":1.323}]},{"osd":5,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.48,"15min":1.5069999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.821,"5min":2.2250000000000001,"15min":2.4300000000000002},"last":1.843},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.452,"15min":1.492},"min":{"1min":0.98199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0169999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.5169999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.593,"15min":1.621},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9490000000000001,"5min":2.669,"15min":2.669},"last":1.698},{"interface":"front","average":{"1min":1.476,"5min":1.55,"15min":1.5980000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2589999999999999,"5min":2.355,"15min":2.3769999999999998},"last":1.925}]},{"osd":7,"last update":"Mon Oct 16 14:10:34 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.655,"15min":1.712},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1549999999999998,"5min":2.484,"15min":2.5270000000000001},"last":1.9930000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.625,"15min":1.649},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.464},"last":1.6519999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:10:51.281 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:10:51.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:10:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.539+0000 7f62b1825640 1 Processor -- start 2023-10-16T14:10:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.539+0000 7f62b1825640 1 -- start start 2023-10-16T14:10:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1825640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1825640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62ac0cded0 0x7f62ac0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1825640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ac15d150 con 0x7f62ac0b8480 2023-10-16T14:10:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1825640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ac0d1180 con 0x7f62ac0cded0 2023-10-16T14:10:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1825640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0d1460 con 0x7f62ac15cd70 2023-10-16T14:10:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1024640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62ac0cded0 0x7f62ac0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1024640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62ac0cded0 0x7f62ac0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57730/0 (socket says 172.21.15.60:57730) 2023-10-16T14:10:51.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b1024640 1 -- 172.21.15.60:0/607926707 learned_addr learned my addr 172.21.15.60:0/607926707 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:51.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.541+0000 7f62b0823640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.542+0000 7f62b1024640 1 -- 172.21.15.60:0/607926707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 msgr2=0x7f62ac0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:51.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.542+0000 7f62b1024640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.542+0000 7f62b1024640 1 -- 172.21.15.60:0/607926707 >> v1:172.21.15.144:6789/0 conn(0x7f62ac0b8480 legacy=0x7f62ac0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:51.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.542+0000 7f62b1024640 1 -- 172.21.15.60:0/607926707 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0d1740 con 0x7f62ac0cded0 2023-10-16T14:10:51.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.542+0000 7f62b0823640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:51.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.543+0000 7f62b1024640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0b5c90 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009e50 tx=0x7f62a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682eb66604d89e98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:51.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.543+0000 7f62ab7fe640 1 -- 172.21.15.60:0/607926707 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a402ca80 con 0x7f62ac0cded0 2023-10-16T14:10:51.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.543+0000 7f62ab7fe640 1 -- 172.21.15.60:0/607926707 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a402cbe0 con 0x7f62ac0cded0 2023-10-16T14:10:51.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.543+0000 7f62ab7fe640 1 -- 172.21.15.60:0/607926707 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40343f0 con 0x7f62ac0cded0 2023-10-16T14:10:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 -- 172.21.15.60:0/607926707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 msgr2=0x7f62ac0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0b5c90 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009e50 tx=0x7f62a402bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 -- 172.21.15.60:0/607926707 shutdown_connections 2023-10-16T14:10:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0b5c90 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 --2- 172.21.15.60:0/607926707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 -- 172.21.15.60:0/607926707 >> 172.21.15.60:0/607926707 conn(0x7f62ac0cf570 msgr2=0x7f62ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 -- 172.21.15.60:0/607926707 shutdown_connections 2023-10-16T14:10:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.544+0000 7f62b1825640 1 -- 172.21.15.60:0/607926707 wait complete. 2023-10-16T14:10:51.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.546+0000 7f62b1825640 1 Processor -- start 2023-10-16T14:10:51.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.546+0000 7f62b1825640 1 -- start start 2023-10-16T14:10:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1825640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac0b8480 0x7f62ac15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1825640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1825640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1825640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0c8650 con 0x7f62ac0cded0 2023-10-16T14:10:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1825640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0c7710 con 0x7f62ac0b8480 2023-10-16T14:10:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1825640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0c7a80 con 0x7f62ac15cd70 2023-10-16T14:10:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57736/0 (socket says 172.21.15.60:57736) 2023-10-16T14:10:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62abfff640 1 -- 172.21.15.60:0/1578885924 learned_addr learned my addr 172.21.15.60:0/1578885924 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b0823640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac0b8480 0x7f62ac15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.547+0000 7f62b1024640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62abfff640 1 -- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 msgr2=0x7f62ac0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62abfff640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62abfff640 1 -- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac0b8480 msgr2=0x7f62ac15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62abfff640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac0b8480 0x7f62ac15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62abfff640 1 -- 172.21.15.60:0/1578885924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4009b30 con 0x7f62ac0cded0 2023-10-16T14:10:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62abfff640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0c9bc0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f629800c430 tx=0x7f629800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298011790 con 0x7f62ac0cded0 2023-10-16T14:10:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62980118f0 con 0x7f62ac0cded0 2023-10-16T14:10:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298011ce0 con 0x7f62ac0cded0 2023-10-16T14:10:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac0c7d20 con 0x7f62ac0cded0 2023-10-16T14:10:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.548+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac0c82e0 con 0x7f62ac0cded0 2023-10-16T14:10:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.549+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac0c8800 con 0x7f62ac0cded0 2023-10-16T14:10:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.551+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f629801c460 con 0x7f62ac0cded0 2023-10-16T14:10:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.552+0000 7f62a97fa640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f629006af90 0x7f629006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.552+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f629808a360 con 0x7f62ac0cded0 2023-10-16T14:10:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.553+0000 7f62b0823640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f629006af90 0x7f629006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.554+0000 7f62b0823640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f629006af90 0x7f629006d450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f62a000a950 tx=0x7f62a0008040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.555+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6298055010 con 0x7f62ac0cded0 2023-10-16T14:10:51.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.786+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62ac0c8470 con 0x7f62ac0cded0 2023-10-16T14:10:51.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.787+0000 7f62a97fa640 1 -- 172.21.15.60:0/1578885924 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6153 (secure 0 0 0) 0x7f6298058630 con 0x7f62ac0cded0 2023-10-16T14:10:51.791 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:10:51.792 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:10:51.792 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 118 2023-10-16T14:10:51.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f629006af90 msgr2=0x7f629006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f629006af90 0x7f629006d450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f62a000a950 tx=0x7f62a0008040 comp rx=0 tx=0).stop 2023-10-16T14:10:51.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 msgr2=0x7f62ac0c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:51.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0c9bc0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f629800c430 tx=0x7f629800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 shutdown_connections 2023-10-16T14:10:51.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f629006af90 0x7f629006d450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac15cd70 0x7f62ac0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac0cded0 0x7f62ac0c9bc0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 --2- 172.21.15.60:0/1578885924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac0b8480 0x7f62ac15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:51.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 >> 172.21.15.60:0/1578885924 conn(0x7f62ac0cf570 msgr2=0x7f62ac0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:51.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 shutdown_connections 2023-10-16T14:10:51.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:51.790+0000 7f62b1825640 1 -- 172.21.15.60:0/1578885924 wait complete. 2023-10-16T14:10:52.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 118, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:10:40.444083+0000', 'modified': '2023-10-16T14:10:43.520528+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10998}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10998': {'gid': 10998, 'name': 'c', 'rank': 0, 'incarnation': 115, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.60:6835/462373098', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 462373098}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 462373098}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:10:52.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:10:52.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T14:10:52.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.457+0000 7f5a8e727640 1 Processor -- start 2023-10-16T14:10:52.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.458+0000 7f5a8e727640 1 -- start start 2023-10-16T14:10:52.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.459+0000 7f5a8e727640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a8806c040 0x7f5a8806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.459+0000 7f5a8e727640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8815f4a0 con 0x7f5a880ad970 2023-10-16T14:10:52.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.459+0000 7f5a8e727640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8806c960 con 0x7f5a880cda80 2023-10-16T14:10:52.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.459+0000 7f5a8e727640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8815b9d0 con 0x7f5a8806c040 2023-10-16T14:10:52.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8d725640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a8806c040 0x7f5a8806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8cf24640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5a880ad970 0x7f5a8815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52604/0 (socket says 172.21.15.60:52604) 2023-10-16T14:10:52.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8cf24640 1 -- 172.21.15.60:0/2967676673 learned_addr learned my addr 172.21.15.60:0/2967676673 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:52.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8d725640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a8806c040 0x7f5a8806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57750/0 (socket says 172.21.15.60:57750) 2023-10-16T14:10:52.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8d725640 1 -- 172.21.15.60:0/2967676673 >> v1:172.21.15.144:6789/0 conn(0x7f5a880cda80 legacy=0x7f5a8815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8d725640 1 -- 172.21.15.60:0/2967676673 >> v1:172.21.15.60:6789/0 conn(0x7f5a880ad970 legacy=0x7f5a8815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.460+0000 7f5a8d725640 1 -- 172.21.15.60:0/2967676673 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8815bcb0 con 0x7f5a8806c040 2023-10-16T14:10:52.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.461+0000 7f5a8d725640 1 --2- 172.21.15.60:0/2967676673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8806c420 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f5a840097a0 tx=0x7f5a8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a095218cbefddc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a7ffff640 1 -- 172.21.15.60:0/2967676673 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84031830 con 0x7f5a8806c040 2023-10-16T14:10:52.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a7ffff640 1 -- 172.21.15.60:0/2967676673 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a84031990 con 0x7f5a8806c040 2023-10-16T14:10:52.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a7ffff640 1 -- 172.21.15.60:0/2967676673 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84031d20 con 0x7f5a8806c040 2023-10-16T14:10:52.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a8e727640 1 -- 172.21.15.60:0/2967676673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 msgr2=0x7f5a8806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a8e727640 1 --2- 172.21.15.60:0/2967676673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8806c420 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f5a840097a0 tx=0x7f5a8402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a8e727640 1 -- 172.21.15.60:0/2967676673 shutdown_connections 2023-10-16T14:10:52.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a8e727640 1 --2- 172.21.15.60:0/2967676673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8806c420 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.462+0000 7f5a8e727640 1 -- 172.21.15.60:0/2967676673 >> 172.21.15.60:0/2967676673 conn(0x7f5a880ce6a0 msgr2=0x7f5a880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.463+0000 7f5a8e727640 1 -- 172.21.15.60:0/2967676673 shutdown_connections 2023-10-16T14:10:52.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.463+0000 7f5a8e727640 1 -- 172.21.15.60:0/2967676673 wait complete. 2023-10-16T14:10:52.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.464+0000 7f5a8e727640 1 Processor -- start 2023-10-16T14:10:52.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 -- start start 2023-10-16T14:10:52.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8814b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a880ad970 0x7f5a880ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a880cda80 0x7f5a880acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880c6520 con 0x7f5a8806c040 2023-10-16T14:10:52.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880c5910 con 0x7f5a880ad970 2023-10-16T14:10:52.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.465+0000 7f5a8e727640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880c5bf0 con 0x7f5a880cda80 2023-10-16T14:10:52.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8814b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8814b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57758/0 (socket says 172.21.15.60:57758) 2023-10-16T14:10:52.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 -- 172.21.15.60:0/250817593 learned_addr learned my addr 172.21.15.60:0/250817593 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8cf24640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a880ad970 0x7f5a880ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8df26640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a880cda80 0x7f5a880acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 -- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a880cda80 msgr2=0x7f5a880acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a880cda80 0x7f5a880acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 -- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a880ad970 msgr2=0x7f5a880ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a880ad970 0x7f5a880ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8d725640 1 -- 172.21.15.60:0/250817593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a84009400 con 0x7f5a8806c040 2023-10-16T14:10:52.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8df26640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a880cda80 0x7f5a880acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:52.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.466+0000 7f5a8cf24640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a880ad970 0x7f5a880ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:52.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.467+0000 7f5a8d725640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8814b9c0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f5a84000c00 tx=0x7f5a840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.467+0000 7f5a7dffb640 1 -- 172.21.15.60:0/250817593 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8403aa60 con 0x7f5a8806c040 2023-10-16T14:10:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.467+0000 7f5a7dffb640 1 -- 172.21.15.60:0/250817593 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8403abc0 con 0x7f5a8806c040 2023-10-16T14:10:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.467+0000 7f5a7dffb640 1 -- 172.21.15.60:0/250817593 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a840443f0 con 0x7f5a8806c040 2023-10-16T14:10:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.467+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a880c5ed0 con 0x7f5a8806c040 2023-10-16T14:10:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.467+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8814bf00 con 0x7f5a8806c040 2023-10-16T14:10:52.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.469+0000 7f5a7dffb640 1 -- 172.21.15.60:0/250817593 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a8404d450 con 0x7f5a8806c040 2023-10-16T14:10:52.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.470+0000 7f5a7dffb640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5a6c06af90 0x7f5a6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.470+0000 7f5a7dffb640 1 -- 172.21.15.60:0/250817593 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f5a840abea0 con 0x7f5a8806c040 2023-10-16T14:10:52.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.471+0000 7f5a925f7740 1 Processor -- start 2023-10-16T14:10:52.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.471+0000 7f5a8cf24640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5a6c06af90 0x7f5a6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.471+0000 7f5a925f7740 1 -- start start 2023-10-16T14:10:52.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa784e3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 0x55fa78512f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 0x55fa78513440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a925f7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa784037a0 con 0x55fa784e35b0 2023-10-16T14:10:52.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa78455740 con 0x55fa78510170 2023-10-16T14:10:52.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa78452e30 con 0x55fa78510550 2023-10-16T14:10:52.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a8cf24640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5a6c06af90 0x7f5a6c06d450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5a880b4570 tx=0x7f5a78009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 0x55fa78513440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 0x55fa78513440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57772/0 (socket says 172.21.15.60:57772) 2023-10-16T14:10:52.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.472+0000 7f5a8df26640 1 -- 172.21.15.60:0/4286057084 learned_addr learned my addr 172.21.15.60:0/4286057084 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a8df26640 1 -- 172.21.15.60:0/4286057084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 msgr2=0x55fa78512f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a8df26640 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 0x55fa78512f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a8df26640 1 -- 172.21.15.60:0/4286057084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 msgr2=0x55fa784e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a8df26640 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa784e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a8df26640 1 -- 172.21.15.60:0/4286057084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa78412910 con 0x55fa784e35b0 2023-10-16T14:10:52.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a8df26640 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 0x55fa78513440 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f5a80004570 tx=0x7f5a8000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da656425039a8d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a6affd640 1 -- 172.21.15.60:0/4286057084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8000de30 con 0x55fa784e35b0 2023-10-16T14:10:52.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a6affd640 1 -- 172.21.15.60:0/4286057084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80006b50 con 0x55fa784e35b0 2023-10-16T14:10:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.473+0000 7f5a6affd640 1 -- 172.21.15.60:0/4286057084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80007030 con 0x55fa784e35b0 2023-10-16T14:10:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 -- 172.21.15.60:0/4286057084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 msgr2=0x55fa78513440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 0x55fa78513440 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f5a80004570 tx=0x7f5a8000d520 comp rx=0 tx=0).stop 2023-10-16T14:10:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 -- 172.21.15.60:0/4286057084 shutdown_connections 2023-10-16T14:10:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 0x55fa78512f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa784e35b0 0x55fa78513440 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 --2- 172.21.15.60:0/4286057084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa784e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 -- 172.21.15.60:0/4286057084 >> 172.21.15.60:0/4286057084 conn(0x55fa784e5380 msgr2=0x55fa7852df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 -- 172.21.15.60:0/4286057084 shutdown_connections 2023-10-16T14:10:52.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.474+0000 7f5a925f7740 1 -- 172.21.15.60:0/4286057084 wait complete. 2023-10-16T14:10:52.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.475+0000 7f5a925f7740 1 -- start start 2023-10-16T14:10:52.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.475+0000 7f5a925f7740 1 Processor -- start 2023-10-16T14:10:52.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7853eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 0x55fa78531410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 0x55fa78531950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7844e720 con 0x55fa78501630 2023-10-16T14:10:52.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa783ec970 con 0x55fa78510170 2023-10-16T14:10:52.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa783ec610 con 0x55fa78510550 2023-10-16T14:10:52.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.476+0000 7f5a925f7740 10 client.0 _ll_register_callbacks cb 0x55fa7840e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:10:52.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 0x55fa78531950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 0x55fa78531950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57774/0 (socket says 172.21.15.60:57774) 2023-10-16T14:10:52.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 -- 172.21.15.60:0/3511798539 learned_addr learned my addr 172.21.15.60:0/3511798539 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8d725640 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7853eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 -- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 msgr2=0x55fa78531410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 0x55fa78531410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 -- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 msgr2=0x55fa7853eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7853eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 -- 172.21.15.60:0/3511798539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a80004170 con 0x55fa78501630 2023-10-16T14:10:52.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a8df26640 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 0x55fa78531950 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f5a80007000 tx=0x7f5a80003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fcb46919603134 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.477+0000 7f5a63fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:10:52.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a63fff640 1 -- 172.21.15.60:0/3511798539 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80007560 con 0x55fa78501630 2023-10-16T14:10:52.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a63fff640 1 -- 172.21.15.60:0/3511798539 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f5a80027e40 con 0x55fa78501630 2023-10-16T14:10:52.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa78475160 con 0x55fa78501630 2023-10-16T14:10:52.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a63fff640 1 -- 172.21.15.60:0/3511798539 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a800076c0 con 0x55fa78501630 2023-10-16T14:10:52.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a925f7740 10 client.11325 fetch_fsmap learned FSMap version 118 2023-10-16T14:10:52.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) v3 -- 0x55fa7844cda0 con 0x55fa78501630 2023-10-16T14:10:52.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a63fff640 1 -- 172.21.15.60:0/3511798539 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fa78475160 con 0x55fa78501630 2023-10-16T14:10:52.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.478+0000 7f5a63fff640 1 -- 172.21.15.60:0/3511798539 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 118) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f5a80019070 con 0x55fa78501630 2023-10-16T14:10:52.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a925f7740 10 client.11325 fetch_fsmap finished waiting for FSMap version 118 2023-10-16T14:10:52.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a925f7740 10 client.11325 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T14:10:52.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a925f7740 20 client.11325 populate_metadata read hostname 'smithi060' 2023-10-16T14:10:52.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a925f7740 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78540f40 0x55fa78514f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a925f7740 4 client.11325 mds_command: new command op to 10998 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:10:52.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 --> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa78420cc0 con 0x55fa78540f40 2023-10-16T14:10:52.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a8d725640 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78540f40 0x55fa78514f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a8d725640 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78540f40 0x55fa78514f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a9291b7aef506c server_cookie=18b7dc8fbb03cca2 in_seq=0 out_seq=0 2023-10-16T14:10:52.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.479+0000 7f5a63fff640 10 client.11325 ms_handle_connect on v2:172.21.15.60:6834/462373098 2023-10-16T14:10:52.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.480+0000 7f5a63fff640 1 -- 172.21.15.60:0/3511798539 <== mds.0 v2:172.21.15.60:6834/462373098 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fa78420cc0 con 0x55fa78540f40 2023-10-16T14:10:52.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.480+0000 7f5a63fff640 10 client.11325 handle_command_reply: tid=0 2023-10-16T14:10:52.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.480+0000 7f5a925f7740 1 client.11325 shutdown 2023-10-16T14:10:52.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.480+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 msgr2=0x55fa78531950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.480+0000 7f5a925f7740 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 0x55fa78531950 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f5a80007000 tx=0x7f5a80003a70 comp rx=0 tx=0).stop 2023-10-16T14:10:52.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 shutdown_connections 2023-10-16T14:10:52.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78540f40 0x55fa78514f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510550 0x55fa78531410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78501630 0x55fa78531950 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 --2- 172.21.15.60:0/3511798539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7853eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a63fff640 0 client.11325 ms_handle_reset on v2:172.21.15.60:6834/462373098 2023-10-16T14:10:52.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 >> 172.21.15.60:0/3511798539 conn(0x55fa784e35b0 msgr2=0x55fa78512ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 shutdown_connections 2023-10-16T14:10:52.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 1 -- 172.21.15.60:0/3511798539 wait complete. 2023-10-16T14:10:52.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.481+0000 7f5a925f7740 20 client.11325 trim_cache size 0 max 16384 2023-10-16T14:10:52.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.498+0000 7f5a925f7740 1 Processor -- start 2023-10-16T14:10:52.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.498+0000 7f5a925f7740 1 -- start start 2023-10-16T14:10:52.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78501630 0x55fa78510930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510170 0x55fa7853ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7852e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a925f7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa781b2f20 con 0x55fa78510550 2023-10-16T14:10:52.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa78517090 con 0x55fa78501630 2023-10-16T14:10:52.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa78515f00 con 0x55fa78510170 2023-10-16T14:10:52.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7852e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7852e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57778/0 (socket says 172.21.15.60:57778) 2023-10-16T14:10:52.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.499+0000 7f5a8df26640 1 -- 172.21.15.60:0/24717601 learned_addr learned my addr 172.21.15.60:0/24717601 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a8df26640 1 -- 172.21.15.60:0/24717601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510170 msgr2=0x55fa7853ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a8df26640 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510170 0x55fa7853ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a8df26640 1 -- 172.21.15.60:0/24717601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78501630 msgr2=0x55fa78510930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a8df26640 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78501630 0x55fa78510930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a8df26640 1 -- 172.21.15.60:0/24717601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa783ebb80 con 0x55fa78510550 2023-10-16T14:10:52.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a8df26640 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7852e390 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f5a80003be0 tx=0x7f5a8001f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bda898b018e9376 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a6b7fe640 1 -- 172.21.15.60:0/24717601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80007240 con 0x55fa78510550 2023-10-16T14:10:52.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a6b7fe640 1 -- 172.21.15.60:0/24717601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80021070 con 0x55fa78510550 2023-10-16T14:10:52.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.500+0000 7f5a6b7fe640 1 -- 172.21.15.60:0/24717601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80007600 con 0x55fa78510550 2023-10-16T14:10:52.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 -- 172.21.15.60:0/24717601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 msgr2=0x55fa7852e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7852e390 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f5a80003be0 tx=0x7f5a8001f070 comp rx=0 tx=0).stop 2023-10-16T14:10:52.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 -- 172.21.15.60:0/24717601 shutdown_connections 2023-10-16T14:10:52.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78510170 0x55fa7853ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7852e390 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 --2- 172.21.15.60:0/24717601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78501630 0x55fa78510930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 -- 172.21.15.60:0/24717601 >> 172.21.15.60:0/24717601 conn(0x55fa784e35b0 msgr2=0x55fa784ea1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 -- 172.21.15.60:0/24717601 shutdown_connections 2023-10-16T14:10:52.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.501+0000 7f5a925f7740 1 -- 172.21.15.60:0/24717601 wait complete. 2023-10-16T14:10:52.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.502+0000 7f5a925f7740 1 -- start start 2023-10-16T14:10:52.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.502+0000 7f5a925f7740 1 Processor -- start 2023-10-16T14:10:52.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78501630 0x55fa78513420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7854d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7854ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa784e2a20 con 0x55fa78510550 2023-10-16T14:10:52.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa78527920 con 0x55fa78510170 2023-10-16T14:10:52.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa7854d820 con 0x55fa78501630 2023-10-16T14:10:52.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a925f7740 10 client.0 _ll_register_callbacks cb 0x7f5a80004390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:10:52.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7854ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a8df26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7854ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57788/0 (socket says 172.21.15.60:57788) 2023-10-16T14:10:52.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.503+0000 7f5a8df26640 1 -- 172.21.15.60:0/716961778 learned_addr learned my addr 172.21.15.60:0/716961778 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8cf24640 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7854d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8df26640 1 -- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78501630 msgr2=0x55fa78513420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8df26640 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78501630 0x55fa78513420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8df26640 1 -- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 msgr2=0x55fa7854d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8df26640 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7854d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8df26640 1 -- 172.21.15.60:0/716961778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa7844cda0 con 0x55fa78510550 2023-10-16T14:10:52.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a8df26640 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7854ecf0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f5a80003be0 tx=0x7f5a80009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebed4a6a9fc6578 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a63fff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:10:52.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a63fff640 1 -- 172.21.15.60:0/716961778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8003d070 con 0x55fa78510550 2023-10-16T14:10:52.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a63fff640 1 -- 172.21.15.60:0/716961778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f5a80035070 con 0x55fa78510550 2023-10-16T14:10:52.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.504+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a80006b50 con 0x55fa78510550 2023-10-16T14:10:52.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a63fff640 1 -- 172.21.15.60:0/716961778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a80007cd0 con 0x55fa78510550 2023-10-16T14:10:52.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 10 client.11331 fetch_fsmap learned FSMap version 118 2023-10-16T14:10:52.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) v3 -- 0x55fa78411170 con 0x55fa78510550 2023-10-16T14:10:52.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a63fff640 1 -- 172.21.15.60:0/716961778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80006b50 con 0x55fa78510550 2023-10-16T14:10:52.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a63fff640 1 -- 172.21.15.60:0/716961778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 118) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f5a80019040 con 0x55fa78510550 2023-10-16T14:10:52.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 10 client.11331 fetch_fsmap finished waiting for FSMap version 118 2023-10-16T14:10:52.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 10 client.11331 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T14:10:52.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 20 client.11331 populate_metadata read hostname 'smithi060' 2023-10-16T14:10:52.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78598e40 0x55fa7854f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 4 client.11331 mds_command: new command op to 10998 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:10:52.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 --> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fa78447700 con 0x55fa78598e40 2023-10-16T14:10:52.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.505+0000 7f5a8d725640 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78598e40 0x55fa7854f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a63fff640 10 client.11331 ms_handle_connect on v2:172.21.15.60:6834/462373098 2023-10-16T14:10:52.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a8d725640 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78598e40 0x55fa7854f230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e6a5a0bb6d49982 server_cookie=6a03ea2425f3cf65 in_seq=0 out_seq=0 2023-10-16T14:10:52.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a63fff640 1 -- 172.21.15.60:0/716961778 <== mds.0 v2:172.21.15.60:6834/462373098 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55fa78447700 con 0x55fa78598e40 2023-10-16T14:10:52.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a63fff640 10 client.11331 handle_command_reply: tid=0 2023-10-16T14:10:52.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 client.11331 shutdown 2023-10-16T14:10:52.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 msgr2=0x55fa7854ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7854ecf0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f5a80003be0 tx=0x7f5a80009040 comp rx=0 tx=0).stop 2023-10-16T14:10:52.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 shutdown_connections 2023-10-16T14:10:52.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:6834/462373098,v1:172.21.15.60:6835/462373098] conn(0x55fa78598e40 0x55fa7854f230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fa78510550 0x55fa7854ecf0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fa78510170 0x55fa7854d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a63fff640 0 client.11331 ms_handle_reset on v2:172.21.15.60:6834/462373098 2023-10-16T14:10:52.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 --2- 172.21.15.60:0/716961778 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55fa78501630 0x55fa78513420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.506+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 >> 172.21.15.60:0/716961778 conn(0x55fa784e35b0 msgr2=0x55fa784e20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 shutdown_connections 2023-10-16T14:10:52.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a925f7740 1 -- 172.21.15.60:0/716961778 wait complete. 2023-10-16T14:10:52.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a925f7740 20 client.11331 trim_cache size 0 max 16384 2023-10-16T14:10:52.523 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:10:52.523 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:10:52.523 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:10:52.524 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 10998, 2023-10-16T14:10:52.524 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:10:52.524 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:10:52.524 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "backup_fs", 2023-10-16T14:10:52.524 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 11.527830178, 2023-10-16T14:10:52.525 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 118, 2023-10-16T14:10:52.525 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 235, 2023-10-16T14:10:52.525 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 235, 2023-10-16T14:10:52.525 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 31.489913555000001 2023-10-16T14:10:52.525 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:10:52.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5a6c06af90 msgr2=0x7f5a6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a8e727640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5a6c06af90 0x7f5a6c06d450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5a880b4570 tx=0x7f5a78009040 comp rx=0 tx=0).stop 2023-10-16T14:10:52.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 msgr2=0x7f5a8814b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.507+0000 7f5a8e727640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8814b9c0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f5a84000c00 tx=0x7f5a840046c0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 shutdown_connections 2023-10-16T14:10:52.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f5a6c06af90 0x7f5a6c06d450 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5a880cda80 0x7f5a880acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a880ad970 0x7f5a880ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 --2- 172.21.15.60:0/250817593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a8806c040 0x7f5a8814b9c0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 >> 172.21.15.60:0/250817593 conn(0x7f5a880ce6a0 msgr2=0x7f5a8815d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 shutdown_connections 2023-10-16T14:10:52.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.508+0000 7f5a8e727640 1 -- 172.21.15.60:0/250817593 wait complete. 2023-10-16T14:10:52.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:10:52.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.731+0000 7fe936239640 1 Processor -- start 2023-10-16T14:10:52.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.731+0000 7fe936239640 1 -- start start 2023-10-16T14:10:52.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe936239640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe93015dde0 0x7fe9300709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe936239640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe930070f20 0x7fe9300ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe936239640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe930160a90 con 0x7fe93015c9f0 2023-10-16T14:10:52.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe936239640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe930071350 con 0x7fe93015dde0 2023-10-16T14:10:52.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe936239640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9300cdec0 con 0x7fe930070f20 2023-10-16T14:10:52.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe934a36640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe93015dde0 0x7fe9300709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe934a36640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe93015dde0 0x7fe9300709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57796/0 (socket says 172.21.15.60:57796) 2023-10-16T14:10:52.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe934a36640 1 -- 172.21.15.60:0/164146965 learned_addr learned my addr 172.21.15.60:0/164146965 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.732+0000 7fe935a38640 1 --2- 172.21.15.60:0/164146965 >> v2:172.21.15.144:3300/0 conn(0x7fe930070f20 0x7fe9300ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.733+0000 7fe934a36640 1 -- 172.21.15.60:0/164146965 >> v2:172.21.15.144:3300/0 conn(0x7fe930070f20 msgr2=0x7fe9300ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:52.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.733+0000 7fe934a36640 1 --2- 172.21.15.60:0/164146965 >> v2:172.21.15.144:3300/0 conn(0x7fe930070f20 0x7fe9300ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.733+0000 7fe934a36640 1 -- 172.21.15.60:0/164146965 >> v1:172.21.15.60:6789/0 conn(0x7fe93015c9f0 legacy=0x7fe9300c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.733+0000 7fe934a36640 1 -- 172.21.15.60:0/164146965 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300cd900 con 0x7fe93015dde0 2023-10-16T14:10:52.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.734+0000 7fe934a36640 1 --2- 172.21.15.60:0/164146965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe93015dde0 0x7fe9300709e0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fe93006fcc0 tx=0x7fe92002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731fc5d3096be653 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.734+0000 7fe927fff640 1 -- 172.21.15.60:0/164146965 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920030880 con 0x7fe93015dde0 2023-10-16T14:10:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.734+0000 7fe927fff640 1 -- 172.21.15.60:0/164146965 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9200309e0 con 0x7fe93015dde0 2023-10-16T14:10:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.734+0000 7fe927fff640 1 -- 172.21.15.60:0/164146965 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920030d70 con 0x7fe93015dde0 2023-10-16T14:10:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 -- 172.21.15.60:0/164146965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe93015dde0 msgr2=0x7fe9300709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 --2- 172.21.15.60:0/164146965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe93015dde0 0x7fe9300709e0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fe93006fcc0 tx=0x7fe92002b9c0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 -- 172.21.15.60:0/164146965 shutdown_connections 2023-10-16T14:10:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 --2- 172.21.15.60:0/164146965 >> v2:172.21.15.144:3300/0 conn(0x7fe930070f20 0x7fe9300ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 --2- 172.21.15.60:0/164146965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe93015dde0 0x7fe9300709e0 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 -- 172.21.15.60:0/164146965 >> 172.21.15.60:0/164146965 conn(0x7fe9300ce6a0 msgr2=0x7fe9300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 -- 172.21.15.60:0/164146965 shutdown_connections 2023-10-16T14:10:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.735+0000 7fe936239640 1 -- 172.21.15.60:0/164146965 wait complete. 2023-10-16T14:10:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.737+0000 7fe936239640 1 Processor -- start 2023-10-16T14:10:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.737+0000 7fe936239640 1 -- start start 2023-10-16T14:10:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe936239640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 0x7fe9300aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe936239640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93015c9f0 0x7fe930150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe936239640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe93015dde0 0x7fe930156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe936239640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930157080 con 0x7fe930070f20 2023-10-16T14:10:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe936239640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930157360 con 0x7fe93015c9f0 2023-10-16T14:10:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe936239640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe930157640 con 0x7fe93015dde0 2023-10-16T14:10:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe935237640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 0x7fe9300aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe935237640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 0x7fe9300aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57812/0 (socket says 172.21.15.60:57812) 2023-10-16T14:10:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.738+0000 7fe935237640 1 -- 172.21.15.60:0/15263364 learned_addr learned my addr 172.21.15.60:0/15263364 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935a38640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe93015dde0 0x7fe930156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe934a36640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93015c9f0 0x7fe930150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935237640 1 -- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe93015dde0 msgr2=0x7fe930156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935237640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe93015dde0 0x7fe930156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935237640 1 -- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93015c9f0 msgr2=0x7fe930150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935237640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93015c9f0 0x7fe930150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935237640 1 -- 172.21.15.60:0/15263364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200094d0 con 0x7fe930070f20 2023-10-16T14:10:52.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe935237640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 0x7fe9300aece0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00c400 tx=0x7fe92c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c0052d0 con 0x7fe930070f20 2023-10-16T14:10:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c005450 con 0x7fe930070f20 2023-10-16T14:10:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c007260 con 0x7fe930070f20 2023-10-16T14:10:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.739+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe930157920 con 0x7fe930070f20 2023-10-16T14:10:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.740+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe930150aa0 con 0x7fe930070f20 2023-10-16T14:10:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.741+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9301406e0 con 0x7fe930070f20 2023-10-16T14:10:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.743+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe92c0073c0 con 0x7fe930070f20 2023-10-16T14:10:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.743+0000 7fe925ffb640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe91406af90 0x7fe91406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.743+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fe92c088f90 con 0x7fe930070f20 2023-10-16T14:10:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.744+0000 7fe934a36640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe91406af90 0x7fe91406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.745+0000 7fe934a36640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe91406af90 0x7fe91406d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe92003b040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.748+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9301406e0 con 0x7fe930070f20 2023-10-16T14:10:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.968+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9301574d0 con 0x7fe91406af90 2023-10-16T14:10:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.972+0000 7fe925ffb640 1 -- 172.21.15.60:0/15263364 <== mgr.7664 v2:172.21.15.144:6832/1724812344 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fe9301574d0 con 0x7fe91406af90 2023-10-16T14:10:52.975 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:10:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T14:10:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:10:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:10:52.976 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "backup_fs" 2023-10-16T14:10:52.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:10:52.977 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:10:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe91406af90 msgr2=0x7fe91406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe91406af90 0x7fe91406d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe92003b040 comp rx=0 tx=0).stop 2023-10-16T14:10:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 msgr2=0x7fe9300aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 0x7fe9300aece0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00c400 tx=0x7fe92c00c8c0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 shutdown_connections 2023-10-16T14:10:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe91406af90 0x7fe91406d450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe93015dde0 0x7fe930156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93015c9f0 0x7fe930150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 --2- 172.21.15.60:0/15263364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe930070f20 0x7fe9300aece0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 >> 172.21.15.60:0/15263364 conn(0x7fe9300ce6a0 msgr2=0x7fe9300cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 shutdown_connections 2023-10-16T14:10:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:52.974+0000 7fe936239640 1 -- 172.21.15.60:0/15263364 wait complete. 2023-10-16T14:10:52.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:10:53.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.225+0000 7f9313fff640 1 Processor -- start 2023-10-16T14:10:53.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.225+0000 7f9313fff640 1 -- start start 2023-10-16T14:10:53.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f9313fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f93140ad970 0x7f93140cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:53.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f9313fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93140ccfb0 0x7f93140cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:53.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f9313fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931415b9d0 con 0x7f931406c040 2023-10-16T14:10:53.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f9313fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931415bcb0 con 0x7f93140ccfb0 2023-10-16T14:10:53.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f9313fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931415bf90 con 0x7f93140ad970 2023-10-16T14:10:53.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93140ccfb0 0x7f93140cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:53.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93140ccfb0 0x7f93140cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57824/0 (socket says 172.21.15.60:57824) 2023-10-16T14:10:53.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 -- 172.21.15.60:0/1270981560 learned_addr learned my addr 172.21.15.60:0/1270981560 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:53.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f930bfff640 1 --2- 172.21.15.60:0/1270981560 >> v2:172.21.15.144:3300/0 conn(0x7f93140ad970 0x7f93140cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:53.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 -- 172.21.15.60:0/1270981560 >> v2:172.21.15.144:3300/0 conn(0x7f93140ad970 msgr2=0x7f93140cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:53.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 --2- 172.21.15.60:0/1270981560 >> v2:172.21.15.144:3300/0 conn(0x7f93140ad970 0x7f93140cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:53.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 -- 172.21.15.60:0/1270981560 >> v1:172.21.15.144:6789/0 conn(0x7f931406c040 legacy=0x7f931406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:53.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.228+0000 7f93137fe640 1 -- 172.21.15.60:0/1270981560 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93140cc270 con 0x7f93140ccfb0 2023-10-16T14:10:53.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.229+0000 7f93137fe640 1 --2- 172.21.15.60:0/1270981560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140cd460 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f930c009e50 tx=0x7f930c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12de3b112c771aeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:53.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.229+0000 7f93127fc640 1 -- 172.21.15.60:0/1270981560 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c02ca80 con 0x7f93140ccfb0 2023-10-16T14:10:53.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.229+0000 7f93127fc640 1 -- 172.21.15.60:0/1270981560 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c02cbe0 con 0x7f93140ccfb0 2023-10-16T14:10:53.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.229+0000 7f93127fc640 1 -- 172.21.15.60:0/1270981560 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c0343f0 con 0x7f93140ccfb0 2023-10-16T14:10:53.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 -- 172.21.15.60:0/1270981560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 msgr2=0x7f93140cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 --2- 172.21.15.60:0/1270981560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140cd460 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f930c009e50 tx=0x7f930c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 -- 172.21.15.60:0/1270981560 shutdown_connections 2023-10-16T14:10:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 --2- 172.21.15.60:0/1270981560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140cd460 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 --2- 172.21.15.60:0/1270981560 >> v2:172.21.15.144:3300/0 conn(0x7f93140ad970 0x7f93140cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 -- 172.21.15.60:0/1270981560 >> 172.21.15.60:0/1270981560 conn(0x7f93140ce6a0 msgr2=0x7f93140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:53.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 -- 172.21.15.60:0/1270981560 shutdown_connections 2023-10-16T14:10:53.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.230+0000 7f9313fff640 1 -- 172.21.15.60:0/1270981560 wait complete. 2023-10-16T14:10:53.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.232+0000 7f9313fff640 1 Processor -- start 2023-10-16T14:10:53.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.232+0000 7f9313fff640 1 -- start start 2023-10-16T14:10:53.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9313fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f931406c040 0x7f93141555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:53.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9313fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93140ad970 0x7f93140b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:53.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9313fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:53.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9313fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9314155b10 con 0x7f93140ccfb0 2023-10-16T14:10:53.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9313fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93140c7740 con 0x7f931406c040 2023-10-16T14:10:53.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9313fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93140c7a20 con 0x7f93140ad970 2023-10-16T14:10:53.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:53.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57832/0 (socket says 172.21.15.60:57832) 2023-10-16T14:10:53.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 -- 172.21.15.60:0/471649 learned_addr learned my addr 172.21.15.60:0/471649 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:53.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f9312ffd640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f931406c040 0x7f93141555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:53.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f930bfff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93140ad970 0x7f93140b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:53.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 -- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93140ad970 msgr2=0x7f93140b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:53.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93140ad970 0x7f93140b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:53.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 -- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f931406c040 msgr2=0x7f93141555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:53.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f931406c040 0x7f93141555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:53.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 -- 172.21.15.60:0/471649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c009b30 con 0x7f93140ccfb0 2023-10-16T14:10:53.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f930bfff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93140ad970 0x7f93140b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:53.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.234+0000 7f93137fe640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140ae890 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f930c009e50 tx=0x7f930c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:53.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.235+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c040670 con 0x7f93140ccfb0 2023-10-16T14:10:53.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.235+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c0407f0 con 0x7f93140ccfb0 2023-10-16T14:10:53.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.235+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c040ba0 con 0x7f93140ccfb0 2023-10-16T14:10:53.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.235+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93140c7d00 con 0x7f93140ccfb0 2023-10-16T14:10:53.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.235+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93140b0f50 con 0x7f93140ccfb0 2023-10-16T14:10:53.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.236+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931406b330 con 0x7f93140ccfb0 2023-10-16T14:10:53.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.237+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 127) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f930c02cd50 con 0x7f93140ccfb0 2023-10-16T14:10:53.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.238+0000 7f930b7fe640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f92f406afc0 0x7f92f406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:53.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.238+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f930c0ac640 con 0x7f93140ccfb0 2023-10-16T14:10:53.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.240+0000 7f9312ffd640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f92f406afc0 0x7f92f406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:53.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.240+0000 7f9312ffd640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f92f406afc0 0x7f92f406d480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93140af4e0 tx=0x7f9304009040 comp rx=0 tx=0).ready entity=mgr.7664 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:53.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.243+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f930c077280 con 0x7f93140ccfb0 2023-10-16T14:10:53.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:53.501+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f931406a0e0 con 0x7f93140ccfb0 2023-10-16T14:10:54.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.070+0000 7f930b7fe640 1 -- 172.21.15.60:0/471649 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v128) v1 ==== 92+0+0 (secure 0 0 0) 0x7f930c07a8a0 con 0x7f93140ccfb0 2023-10-16T14:10:54.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f92f406afc0 msgr2=0x7f92f406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f92f406afc0 0x7f92f406d480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93140af4e0 tx=0x7f9304009040 comp rx=0 tx=0).stop 2023-10-16T14:10:54.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 msgr2=0x7f93140ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140ae890 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f930c009e50 tx=0x7f930c004a40 comp rx=0 tx=0).stop 2023-10-16T14:10:54.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 shutdown_connections 2023-10-16T14:10:54.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f92f406afc0 0x7f92f406d480 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93140ad970 0x7f93140b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93140ccfb0 0x7f93140ae890 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 --2- 172.21.15.60:0/471649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f931406c040 0x7f93141555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.073+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 >> 172.21.15.60:0/471649 conn(0x7f93140ce6a0 msgr2=0x7f93140ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:54.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.074+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 shutdown_connections 2023-10-16T14:10:54.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.074+0000 7f9313fff640 1 -- 172.21.15.60:0/471649 wait complete. 2023-10-16T14:10:54.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:10:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.298+0000 7fe2cd378640 1 Processor -- start 2023-10-16T14:10:54.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.299+0000 7fe2cd378640 1 -- start start 2023-10-16T14:10:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2cd378640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c80addb0 0x7fe2c80caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2cd378640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2c80c9000 0x7fe2c8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2cd378640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80cb140 con 0x7fe2c806a2e0 2023-10-16T14:10:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2cd378640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c8161a50 con 0x7fe2c80c9000 2023-10-16T14:10:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2cd378640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c806a750 con 0x7fe2c80addb0 2023-10-16T14:10:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2c67fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2c80c9000 0x7fe2c8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2c77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2c806a2e0 0x7fe2c80cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52638/0 (socket says 172.21.15.60:52638) 2023-10-16T14:10:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.300+0000 7fe2c77fe640 1 -- 172.21.15.60:0/3473174213 learned_addr learned my addr 172.21.15.60:0/3473174213 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:10:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c6ffd640 1 --2- 172.21.15.60:0/3473174213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c80addb0 0x7fe2c80caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c5ffb640 1 -- 172.21.15.60:0/3473174213 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617250249 0 0) 0x7fe2c80cb140 con 0x7fe2c806a2e0 2023-10-16T14:10:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c5ffb640 1 -- 172.21.15.60:0/3473174213 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b4003520 con 0x7fe2c806a2e0 2023-10-16T14:10:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c67fc640 1 -- 172.21.15.60:0/3473174213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c80addb0 msgr2=0x7fe2c80caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c67fc640 1 --2- 172.21.15.60:0/3473174213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c80addb0 0x7fe2c80caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c67fc640 1 -- 172.21.15.60:0/3473174213 >> v1:172.21.15.60:6789/0 conn(0x7fe2c806a2e0 legacy=0x7fe2c80cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.301+0000 7fe2c67fc640 1 -- 172.21.15.60:0/3473174213 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c806b030 con 0x7fe2c80c9000 2023-10-16T14:10:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.302+0000 7fe2c67fc640 1 --2- 172.21.15.60:0/3473174213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c8069da0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00097d0 tx=0x7fe2b002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a3936fb3bbfdd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.302+0000 7fe2c5ffb640 1 -- 172.21.15.60:0/3473174213 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0031800 con 0x7fe2c80c9000 2023-10-16T14:10:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2c5ffb640 1 -- 172.21.15.60:0/3473174213 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0031960 con 0x7fe2c80c9000 2023-10-16T14:10:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2c5ffb640 1 -- 172.21.15.60:0/3473174213 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0031cf0 con 0x7fe2c80c9000 2023-10-16T14:10:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 -- 172.21.15.60:0/3473174213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 msgr2=0x7fe2c8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 --2- 172.21.15.60:0/3473174213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c8069da0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00097d0 tx=0x7fe2b002b8a0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 -- 172.21.15.60:0/3473174213 shutdown_connections 2023-10-16T14:10:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 --2- 172.21.15.60:0/3473174213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c8069da0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 --2- 172.21.15.60:0/3473174213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c80addb0 0x7fe2c80caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 -- 172.21.15.60:0/3473174213 >> 172.21.15.60:0/3473174213 conn(0x7fe2c80ce9b0 msgr2=0x7fe2c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.303+0000 7fe2cd378640 1 -- 172.21.15.60:0/3473174213 shutdown_connections 2023-10-16T14:10:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.304+0000 7fe2cd378640 1 -- 172.21.15.60:0/3473174213 wait complete. 2023-10-16T14:10:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.305+0000 7fe2cd378640 1 Processor -- start 2023-10-16T14:10:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.305+0000 7fe2cd378640 1 -- start start 2023-10-16T14:10:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.306+0000 7fe2cd378640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c806a2e0 0x7fe2c81f9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.306+0000 7fe2cd378640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c80addb0 0x7fe2c80c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.306+0000 7fe2cd378640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c80c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.306+0000 7fe2cd378640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c815f800 con 0x7fe2c80c9000 2023-10-16T14:10:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.306+0000 7fe2cd378640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80c74e0 con 0x7fe2c80addb0 2023-10-16T14:10:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.306+0000 7fe2cd378640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80b20d0 con 0x7fe2c806a2e0 2023-10-16T14:10:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c80c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c80c6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57878/0 (socket says 172.21.15.60:57878) 2023-10-16T14:10:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 -- 172.21.15.60:0/816314468 learned_addr learned my addr 172.21.15.60:0/816314468 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c6ffd640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c806a2e0 0x7fe2c81f9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c67fc640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c80addb0 0x7fe2c80c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 -- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c806a2e0 msgr2=0x7fe2c81f9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c806a2e0 0x7fe2c81f9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 -- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c80addb0 msgr2=0x7fe2c80c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c80addb0 0x7fe2c80c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 -- 172.21.15.60:0/816314468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b0009400 con 0x7fe2c80c9000 2023-10-16T14:10:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c67fc640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c80addb0 0x7fe2c80c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.307+0000 7fe2c77fe640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c80c6fa0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c430 tx=0x7fe2c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.308+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0005ce0 con 0x7fe2c80c9000 2023-10-16T14:10:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.308+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0005e40 con 0x7fe2c80c9000 2023-10-16T14:10:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.308+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007310 con 0x7fe2c80c9000 2023-10-16T14:10:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.308+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80b23b0 con 0x7fe2c80c9000 2023-10-16T14:10:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.308+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80b2970 con 0x7fe2c80c9000 2023-10-16T14:10:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.309+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c80b2b00 con 0x7fe2c80c9000 2023-10-16T14:10:54.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.310+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 128) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe2c0007470 con 0x7fe2c80c9000 2023-10-16T14:10:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.311+0000 7fe2bf7fe640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 0x7fe2a806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.311+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fe2c00891d0 con 0x7fe2c80c9000 2023-10-16T14:10:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.312+0000 7fe2c6ffd640 1 -- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 msgr2=0x7fe2a806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.312+0000 7fe2c6ffd640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 0x7fe2a806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.320+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2c0053e20 con 0x7fe2c80c9000 2023-10-16T14:10:54.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.513+0000 7fe2c6ffd640 1 -- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 msgr2=0x7fe2a806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:54.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.513+0000 7fe2c6ffd640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 0x7fe2a806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:54.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.519+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe2c804e160 con 0x7fe2c80c9000 2023-10-16T14:10:54.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.523+0000 7fe2bf7fe640 1 -- 172.21.15.60:0/816314468 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe2c0057440 con 0x7fe2c80c9000 2023-10-16T14:10:54.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 msgr2=0x7fe2a806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:54.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 0x7fe2a806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 msgr2=0x7fe2c80c6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c80c6fa0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c430 tx=0x7fe2c000c8f0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 shutdown_connections 2023-10-16T14:10:54.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fe2a806afe0 0x7fe2a806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2c80c9000 0x7fe2c80c6fa0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c80addb0 0x7fe2c80c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 --2- 172.21.15.60:0/816314468 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe2c806a2e0 0x7fe2c81f9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 >> 172.21.15.60:0/816314468 conn(0x7fe2c80ce9b0 msgr2=0x7fe2c80ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:54.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 shutdown_connections 2023-10-16T14:10:54.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.526+0000 7fe2cd378640 1 -- 172.21.15.60:0/816314468 wait complete. 2023-10-16T14:10:54.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:10:54.591 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:54.588+0000 7f078400d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:10:54.632 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:54.628+0000 7fc9b0e37040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:10:54.645 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:54.642+0000 7f078400d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:10:54.690 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:54.686+0000 7fc9b0e37040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:10:54.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.751+0000 7f1de8061640 1 Processor -- start 2023-10-16T14:10:54.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.752+0000 7f1de8061640 1 -- start start 2023-10-16T14:10:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.753+0000 7f1de8061640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1de015d030 0x7f1de00d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.753+0000 7f1de8061640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1de00cd260 0x7f1de00cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.753+0000 7f1de8061640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de00b5d90 con 0x7f1de015d410 2023-10-16T14:10:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.753+0000 7f1de8061640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00d2b50 con 0x7f1de00cd260 2023-10-16T14:10:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.753+0000 7f1de8061640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00cdbf0 con 0x7f1de015d030 2023-10-16T14:10:54.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1de00cd260 0x7f1de00cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1de00cd260 0x7f1de00cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57892/0 (socket says 172.21.15.60:57892) 2023-10-16T14:10:54.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 -- 172.21.15.60:0/168651205 learned_addr learned my addr 172.21.15.60:0/168651205 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:54.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de5dfd640 1 --2- 172.21.15.60:0/168651205 >> v2:172.21.15.144:3300/0 conn(0x7f1de015d030 0x7f1de00d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 -- 172.21.15.60:0/168651205 >> v2:172.21.15.144:3300/0 conn(0x7f1de015d030 msgr2=0x7f1de00d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 --2- 172.21.15.60:0/168651205 >> v2:172.21.15.144:3300/0 conn(0x7f1de015d030 0x7f1de00d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 -- 172.21.15.60:0/168651205 >> v1:172.21.15.144:6789/0 conn(0x7f1de015d410 legacy=0x7f1de0161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.754+0000 7f1de65fe640 1 -- 172.21.15.60:0/168651205 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de0160fc0 con 0x7f1de00cd260 2023-10-16T14:10:54.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.755+0000 7f1de65fe640 1 --2- 172.21.15.60:0/168651205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de00cd260 0x7f1de00cd6b0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f1de006cd10 tx=0x7f1dd802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181a19ded2c8f1d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:54.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de4dfb640 1 -- 172.21.15.60:0/168651205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd802cb20 con 0x7f1de00cd260 2023-10-16T14:10:54.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de4dfb640 1 -- 172.21.15.60:0/168651205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd802cc80 con 0x7f1de00cd260 2023-10-16T14:10:54.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de4dfb640 1 -- 172.21.15.60:0/168651205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80343f0 con 0x7f1de00cd260 2023-10-16T14:10:54.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 -- 172.21.15.60:0/168651205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de00cd260 msgr2=0x7f1de00cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 --2- 172.21.15.60:0/168651205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de00cd260 0x7f1de00cd6b0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f1de006cd10 tx=0x7f1dd802bf80 comp rx=0 tx=0).stop 2023-10-16T14:10:54.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 -- 172.21.15.60:0/168651205 shutdown_connections 2023-10-16T14:10:54.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 --2- 172.21.15.60:0/168651205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de00cd260 0x7f1de00cd6b0 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 --2- 172.21.15.60:0/168651205 >> v2:172.21.15.144:3300/0 conn(0x7f1de015d030 0x7f1de00d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 -- 172.21.15.60:0/168651205 >> 172.21.15.60:0/168651205 conn(0x7f1de00cf720 msgr2=0x7f1de00cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:54.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 -- 172.21.15.60:0/168651205 shutdown_connections 2023-10-16T14:10:54.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.756+0000 7f1de8061640 1 -- 172.21.15.60:0/168651205 wait complete. 2023-10-16T14:10:54.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.758+0000 7f1de8061640 1 Processor -- start 2023-10-16T14:10:54.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.758+0000 7f1de8061640 1 -- start start 2023-10-16T14:10:54.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de8061640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00cd260 0x7f1de015e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de8061640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 0x7f1de00c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de8061640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de015d410 0x7f1de00c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de8061640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de01f5d80 con 0x7f1de015d030 2023-10-16T14:10:54.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de8061640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015efc0 con 0x7f1de00cd260 2023-10-16T14:10:54.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de8061640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015f2a0 con 0x7f1de015d410 2023-10-16T14:10:54.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.759+0000 7f1de55fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 0x7f1de00c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 0x7f1de00c7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57900/0 (socket says 172.21.15.60:57900) 2023-10-16T14:10:54.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 -- 172.21.15.60:0/2726738184 learned_addr learned my addr 172.21.15.60:0/2726738184 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:54.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de5dfd640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00cd260 0x7f1de015e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de65fe640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de015d410 0x7f1de00c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:54.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 -- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de015d410 msgr2=0x7f1de00c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de015d410 0x7f1de00c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 -- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00cd260 msgr2=0x7f1de015e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00cd260 0x7f1de015e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 -- 172.21.15.60:0/2726738184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8009b30 con 0x7f1de015d030 2023-10-16T14:10:54.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.760+0000 7f1de55fc640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 0x7f1de00c7bb0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000b3c0 tx=0x7f1dd000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:54.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.761+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011830 con 0x7f1de015d030 2023-10-16T14:10:54.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.761+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0011990 con 0x7f1de015d030 2023-10-16T14:10:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.761+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011d40 con 0x7f1de015d030 2023-10-16T14:10:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.761+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de015f580 con 0x7f1de015d030 2023-10-16T14:10:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.761+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de015fb40 con 0x7f1de015d030 2023-10-16T14:10:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.762+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de004e160 con 0x7f1de015d030 2023-10-16T14:10:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.763+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 128) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1dd001b460 con 0x7f1de015d030 2023-10-16T14:10:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.764+0000 7f1dd67fc640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 0x7f1dc406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.764+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f1dd00895f0 con 0x7f1de015d030 2023-10-16T14:10:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.764+0000 7f1de5dfd640 1 -- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 msgr2=0x7f1dc406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.764+0000 7f1de5dfd640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 0x7f1dc406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:54.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.772+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1dd0054220 con 0x7f1de015d030 2023-10-16T14:10:54.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.965+0000 7f1de5dfd640 1 -- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 msgr2=0x7f1dc406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:54.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.965+0000 7f1de5dfd640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 0x7f1dc406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:54.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.966+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f1de00c9aa0 con 0x7f1de015d030 2023-10-16T14:10:54.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.971+0000 7f1dd67fc640 1 -- 172.21.15.60:0/2726738184 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1dd0057840 con 0x7f1de015d030 2023-10-16T14:10:54.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 msgr2=0x7f1dc406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 0x7f1dc406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 msgr2=0x7f1de00c7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 0x7f1de00c7bb0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000b3c0 tx=0x7f1dd000b880 comp rx=0 tx=0).stop 2023-10-16T14:10:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 shutdown_connections 2023-10-16T14:10:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7f1dc406b010 0x7f1dc406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1de015d410 0x7f1de00c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1de015d030 0x7f1de00c7bb0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 --2- 172.21.15.60:0/2726738184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1de00cd260 0x7f1de015e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:54.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 >> 172.21.15.60:0/2726738184 conn(0x7f1de00cf720 msgr2=0x7f1de00b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:54.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 shutdown_connections 2023-10-16T14:10:54.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:54.974+0000 7f1de8061640 1 -- 172.21.15.60:0/2726738184 wait complete. 2023-10-16T14:10:54.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:10:54.996 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T14:10:55.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T14:10:55.099 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.097+0000 7f078400d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:10:55.151 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.148+0000 7f078400d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:10:55.172 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.170+0000 7fc9b0e37040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:10:55.201 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.198+0000 7f078400d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:10:55.227 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.225+0000 7fc9b0e37040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:10:55.261 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.258+0000 7f078400d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:10:55.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.262+0000 7fb431a1a640 1 Processor -- start 2023-10-16T14:10:55.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.262+0000 7fb431a1a640 1 -- start start 2023-10-16T14:10:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.263+0000 7fb431a1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 0x7fb42c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.263+0000 7fb431a1a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb42c0ce280 0x7fb42c0ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.263+0000 7fb431a1a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c0acd70 con 0x7fb42c06d220 2023-10-16T14:10:55.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.264+0000 7fb431a1a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c0ac4a0 con 0x7fb42c0ce280 2023-10-16T14:10:55.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.264+0000 7fb431a1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c065f40 con 0x7fb42c065ab0 2023-10-16T14:10:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.264+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 0x7fb42c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.264+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 0x7fb42c0c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:54726/0 (socket says 172.21.15.60:54726) 2023-10-16T14:10:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.264+0000 7fb42bfff640 1 -- 172.21.15.60:0/1280885187 learned_addr learned my addr 172.21.15.60:0/1280885187 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.264+0000 7fb431219640 1 --2- 172.21.15.60:0/1280885187 >> v2:172.21.15.144:3300/0 conn(0x7fb42c0ce280 0x7fb42c0ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.265+0000 7fb42bfff640 1 -- 172.21.15.60:0/1280885187 >> v2:172.21.15.144:3300/0 conn(0x7fb42c0ce280 msgr2=0x7fb42c0ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.265+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1280885187 >> v2:172.21.15.144:3300/0 conn(0x7fb42c0ce280 0x7fb42c0ad200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.265+0000 7fb42bfff640 1 -- 172.21.15.60:0/1280885187 >> v1:172.21.15.144:6789/0 conn(0x7fb42c06d220 legacy=0x7fb42c0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.265+0000 7fb42bfff640 1 -- 172.21.15.60:0/1280885187 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0ac720 con 0x7fb42c065ab0 2023-10-16T14:10:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.266+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1280885187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 0x7fb42c0c95e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009870 tx=0x7fb41c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=decd9b261dd701a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.266+0000 7fb42b7fe640 1 -- 172.21.15.60:0/1280885187 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c030840 con 0x7fb42c065ab0 2023-10-16T14:10:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.266+0000 7fb42b7fe640 1 -- 172.21.15.60:0/1280885187 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c0309a0 con 0x7fb42c065ab0 2023-10-16T14:10:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.266+0000 7fb42b7fe640 1 -- 172.21.15.60:0/1280885187 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c030d30 con 0x7fb42c065ab0 2023-10-16T14:10:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 -- 172.21.15.60:0/1280885187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 msgr2=0x7fb42c0c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 --2- 172.21.15.60:0/1280885187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 0x7fb42c0c95e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009870 tx=0x7fb41c02b980 comp rx=0 tx=0).stop 2023-10-16T14:10:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 -- 172.21.15.60:0/1280885187 shutdown_connections 2023-10-16T14:10:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 --2- 172.21.15.60:0/1280885187 >> v2:172.21.15.144:3300/0 conn(0x7fb42c0ce280 0x7fb42c0ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 --2- 172.21.15.60:0/1280885187 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c065ab0 0x7fb42c0c95e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 -- 172.21.15.60:0/1280885187 >> 172.21.15.60:0/1280885187 conn(0x7fb42c0b0a80 msgr2=0x7fb42c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 -- 172.21.15.60:0/1280885187 shutdown_connections 2023-10-16T14:10:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.267+0000 7fb431a1a640 1 -- 172.21.15.60:0/1280885187 wait complete. 2023-10-16T14:10:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.270+0000 7fb431a1a640 1 Processor -- start 2023-10-16T14:10:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.270+0000 7fb431a1a640 1 -- start start 2023-10-16T14:10:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.271+0000 7fb431a1a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c06d220 0x7fb42c09c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.271+0000 7fb431a1a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb42c0ce280 0x7fb42c157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.271+0000 7fb431a1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb42c15fa10 0x7fb42c09b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.271+0000 7fb431a1a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c09b210 con 0x7fb42c15fa10 2023-10-16T14:10:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.271+0000 7fb431a1a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c157eb0 con 0x7fb42c0ce280 2023-10-16T14:10:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.271+0000 7fb431a1a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c0b2880 con 0x7fb42c06d220 2023-10-16T14:10:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb431219640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb42c15fa10 0x7fb42c09b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb430a18640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c06d220 0x7fb42c09c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb431219640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb42c15fa10 0x7fb42c09b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57902/0 (socket says 172.21.15.60:57902) 2023-10-16T14:10:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb431219640 1 -- 172.21.15.60:0/1802825148 learned_addr learned my addr 172.21.15.60:0/1802825148 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb42c0ce280 0x7fb42c157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb430a18640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb42c0ce280 msgr2=0x7fb42c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb430a18640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb42c0ce280 0x7fb42c157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb430a18640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb42c15fa10 msgr2=0x7fb42c09b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb430a18640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb42c15fa10 0x7fb42c09b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb430a18640 1 -- 172.21.15.60:0/1802825148 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0094d0 con 0x7fb42c06d220 2023-10-16T14:10:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.272+0000 7fb431219640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb42c15fa10 0x7fb42c09b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:10:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.273+0000 7fb430a18640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c06d220 0x7fb42c09c520 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb42000c390 tx=0x7fb42000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.273+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200052d0 con 0x7fb42c06d220 2023-10-16T14:10:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.273+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb420005430 con 0x7fb42c06d220 2023-10-16T14:10:55.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.273+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420003830 con 0x7fb42c06d220 2023-10-16T14:10:55.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.273+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c0ce070 con 0x7fb42c06d220 2023-10-16T14:10:55.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.274+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c09ab50 con 0x7fb42c06d220 2023-10-16T14:10:55.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.275+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c0b4ac0 con 0x7fb42c06d220 2023-10-16T14:10:55.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.276+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 128) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fb420003990 con 0x7fb42c06d220 2023-10-16T14:10:55.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.278+0000 7fb4297fa640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 0x7fb41806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:55.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.278+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fb420089120 con 0x7fb42c06d220 2023-10-16T14:10:55.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.278+0000 7fb42bfff640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 msgr2=0x7fb41806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:55.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.278+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 0x7fb41806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:10:55.283 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.280+0000 7fc9b0e37040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:10:55.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.292+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb420053cf0 con 0x7fb42c06d220 2023-10-16T14:10:55.313 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.310+0000 7f078400d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:10:55.384 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.381+0000 7fc9b0e37040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:10:55.425 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.422+0000 7f078400d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:10:55.471 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.468+0000 7fc9b0e37040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:10:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.479+0000 7fb42bfff640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 msgr2=0x7fb41806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:55.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.479+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 0x7fb41806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:10:55.487 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.485+0000 7f078400d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:55.537 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.534+0000 7f078400d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:10:55.595 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.592+0000 7f078400d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:10:55.623 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.621+0000 7fc9b0e37040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:10:55.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.638+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 --> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb42c0aca00 con 0x7fb41806afe0 2023-10-16T14:10:55.644 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:55.642+0000 7f078400d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:10:55.691 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.688+0000 7fc9b0e37040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:55.746 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.743+0000 7fc9b0e37040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:10:55.808 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.806+0000 7fc9b0e37040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:10:55.862 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:55.859+0000 7fc9b0e37040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:10:55.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.880+0000 7fb42bfff640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 msgr2=0x7fb41806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:55.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:55.880+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 0x7fb41806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:10:56.029 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.026+0000 7f078400d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:10:56.187 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.185+0000 7f078400d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:10:56.263 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.260+0000 7fc9b0e37040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:10:56.295 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.292+0000 7f078400d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:10:56.344 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.341+0000 7f078400d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:10:56.430 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.428+0000 7fc9b0e37040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:10:56.451 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.448+0000 7f078400d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:56.545 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.542+0000 7fc9b0e37040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:10:56.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.555+0000 7f078400d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:10:56.597 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.594+0000 7fc9b0e37040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:10:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:56.681+0000 7fb42bfff640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 msgr2=0x7fb41806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1724812344 2023-10-16T14:10:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:56.681+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 0x7fb41806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:10:56.709 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.707+0000 7fc9b0e37040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:56.714 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.711+0000 7f078400d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:56.762 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.760+0000 7f078400d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:56.821 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.818+0000 7fc9b0e37040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:10:56.883 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:56.881+0000 7f078400d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:10:56.986 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:56.983+0000 7fc9b0e37040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:10:57.042 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:57.039+0000 7fc9b0e37040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:10:57.168 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:57.165+0000 7fc9b0e37040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:10:57.707 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:57.704+0000 7f078400d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:10:57.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:10:57.985 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:10:58.006 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.004+0000 7f078400d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:10:58.047 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:58.044+0000 7fc9b0e37040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:10:58.066 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.064+0000 7f078400d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:10:58.180 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.178+0000 7f078400d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:10:58.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:58.208+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mgrmap(e 129) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fb4200540c0 con 0x7fb42c06d220 2023-10-16T14:10:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:58.208+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 msgr2=0x7fb41806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:10:58.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:58.208+0000 7fb4297fa640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1724812344,v1:172.21.15.144:6833/1724812344] conn(0x7fb41806afe0 0x7fb41806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:58.396 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:58.393+0000 7fc9b0e37040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:10:58.458 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:58.456+0000 7fc9b0e37040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:10:58.586 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:10:58.583+0000 7fc9b0e37040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:10:58.716 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.716 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.717 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.718 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.718 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.713+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.731 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.728+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.731 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.728+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.732 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.728+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.732 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.728+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.732 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.728+0000 7f0702e7d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.732 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.729+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.733 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.729+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.733 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.729+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.733 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.729+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:58.733 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:10:58.729+0000 7f06fee75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:10:59.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.216+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mon.2 v2:172.21.15.144:3301/0 8 ==== mgrmap(e 130) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fb4200765e0 con 0x7fb42c06d220 2023-10-16T14:10:59.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.217+0000 7fb4297fa640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fb4180358e0 0x7fb418077450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.217+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb418035cc0 con 0x7fb4180358e0 2023-10-16T14:10:59.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.222+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fb4180358e0 0x7fb418077450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.229+0000 7fb42bfff640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fb4180358e0 0x7fb418077450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb41c036000 tx=0x7fb41c008030 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:59.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.418+0000 7fb4297fa640 1 -- 172.21.15.60:0/1802825148 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7fb418035cc0 con 0x7fb4180358e0 2023-10-16T14:10:59.421 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T14:10:59.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.421+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fb4180358e0 msgr2=0x7fb418077450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.421+0000 7fb431a1a640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fb4180358e0 0x7fb418077450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb41c036000 tx=0x7fb41c008030 comp rx=0 tx=0).stop 2023-10-16T14:10:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.421+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c06d220 msgr2=0x7fb42c09c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.421+0000 7fb431a1a640 1 --2- 172.21.15.60:0/1802825148 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb42c06d220 0x7fb42c09c520 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb42000c390 tx=0x7fb42000c850 comp rx=0 tx=0).stop 2023-10-16T14:10:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.422+0000 7fb431219640 1 -- 172.21.15.60:0/1802825148 reap_dead start 2023-10-16T14:10:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.422+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 shutdown_connections 2023-10-16T14:10:59.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.422+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 >> 172.21.15.60:0/1802825148 conn(0x7fb42c0b0a80 msgr2=0x7fb42c0af040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:59.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.422+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 shutdown_connections 2023-10-16T14:10:59.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.422+0000 7fb431a1a640 1 -- 172.21.15.60:0/1802825148 wait complete. 2023-10-16T14:10:59.447 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T14:10:59.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T14:10:59.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.678+0000 7f73554aa640 1 Processor -- start 2023-10-16T14:10:59.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.678+0000 7f73554aa640 1 -- start start 2023-10-16T14:10:59.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.679+0000 7f73554aa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f735006c040 0x7f735006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.679+0000 7f73554aa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f73500cda80 0x7f735015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.679+0000 7f73554aa640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735006c960 con 0x7f73500ad970 2023-10-16T14:10:59.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.679+0000 7f73554aa640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735015b9d0 con 0x7f735006c040 2023-10-16T14:10:59.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.679+0000 7f73554aa640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735015bcb0 con 0x7f73500cda80 2023-10-16T14:10:59.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f735006c040 0x7f735006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f735006c040 0x7f735006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55334/0 (socket says 172.21.15.60:55334) 2023-10-16T14:10:59.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 -- 172.21.15.60:0/3193771293 learned_addr learned my addr 172.21.15.60:0/3193771293 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:59.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f7354ca9640 1 --2- 172.21.15.60:0/3193771293 >> v2:172.21.15.144:3300/0 conn(0x7f73500cda80 0x7f735015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 -- 172.21.15.60:0/3193771293 >> v2:172.21.15.144:3300/0 conn(0x7f73500cda80 msgr2=0x7f735015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:59.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 --2- 172.21.15.60:0/3193771293 >> v2:172.21.15.144:3300/0 conn(0x7f73500cda80 0x7f735015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:59.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 -- 172.21.15.60:0/3193771293 >> v1:172.21.15.144:6789/0 conn(0x7f73500ad970 legacy=0x7f735015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:10:59.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f734ffff640 1 -- 172.21.15.60:0/3193771293 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735015bf90 con 0x7f735006c040 2023-10-16T14:10:59.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.680+0000 7f7354ca9640 1 --2- 172.21.15.60:0/3193771293 >> v2:172.21.15.144:3300/0 conn(0x7f73500cda80 0x7f735015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:10:59.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.681+0000 7f734ffff640 1 --2- 172.21.15.60:0/3193771293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f735006c040 0x7f735006c420 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f73500cf300 tx=0x7f734002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1264dc986e0a221a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:59.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f734effd640 1 -- 172.21.15.60:0/3193771293 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734002ca80 con 0x7f735006c040 2023-10-16T14:10:59.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f734effd640 1 -- 172.21.15.60:0/3193771293 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734002cbe0 con 0x7f735006c040 2023-10-16T14:10:59.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f734effd640 1 -- 172.21.15.60:0/3193771293 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73400343f0 con 0x7f735006c040 2023-10-16T14:10:59.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f73554aa640 1 -- 172.21.15.60:0/3193771293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f735006c040 msgr2=0x7f735006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:59.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f73554aa640 1 --2- 172.21.15.60:0/3193771293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f735006c040 0x7f735006c420 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f73500cf300 tx=0x7f734002bee0 comp rx=0 tx=0).stop 2023-10-16T14:10:59.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f73554aa640 1 -- 172.21.15.60:0/3193771293 shutdown_connections 2023-10-16T14:10:59.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f73554aa640 1 --2- 172.21.15.60:0/3193771293 >> v2:172.21.15.144:3300/0 conn(0x7f73500cda80 0x7f735015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:59.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f73554aa640 1 --2- 172.21.15.60:0/3193771293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f735006c040 0x7f735006c420 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:59.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.682+0000 7f73554aa640 1 -- 172.21.15.60:0/3193771293 >> 172.21.15.60:0/3193771293 conn(0x7f73500ce6a0 msgr2=0x7f73500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:10:59.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.683+0000 7f73554aa640 1 -- 172.21.15.60:0/3193771293 shutdown_connections 2023-10-16T14:10:59.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.683+0000 7f73554aa640 1 -- 172.21.15.60:0/3193771293 wait complete. 2023-10-16T14:10:59.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.684+0000 7f73554aa640 1 Processor -- start 2023-10-16T14:10:59.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.684+0000 7f73554aa640 1 -- start start 2023-10-16T14:10:59.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.685+0000 7f73554aa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f735006c040 0x7f73500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.685+0000 7f73554aa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73500ad970 0x7f73500acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.685+0000 7f73554aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 0x7f73500c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.685+0000 7f73554aa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73500c7aa0 con 0x7f73500cda80 2023-10-16T14:10:59.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.685+0000 7f73554aa640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73500ca7c0 con 0x7f735006c040 2023-10-16T14:10:59.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.685+0000 7f73554aa640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73500c6f20 con 0x7f73500ad970 2023-10-16T14:10:59.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 0x7f73500c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 0x7f73500c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55340/0 (socket says 172.21.15.60:55340) 2023-10-16T14:10:59.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 -- 172.21.15.60:0/2809895699 learned_addr learned my addr 172.21.15.60:0/2809895699 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:10:59.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f734ffff640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f735006c040 0x7f73500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f734f7fe640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73500ad970 0x7f73500acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 -- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73500ad970 msgr2=0x7f73500acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:59.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73500ad970 0x7f73500acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:59.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 -- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f735006c040 msgr2=0x7f73500ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:10:59.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f735006c040 0x7f73500ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:10:59.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.686+0000 7f7354ca9640 1 -- 172.21.15.60:0/2809895699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7340009b30 con 0x7f73500cda80 2023-10-16T14:10:59.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.687+0000 7f7354ca9640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 0x7f73500c62b0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f734800c7f0 tx=0x7f734800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:59.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.687+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348007820 con 0x7f73500cda80 2023-10-16T14:10:59.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.687+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7348007980 con 0x7f73500cda80 2023-10-16T14:10:59.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.687+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348007d70 con 0x7f73500cda80 2023-10-16T14:10:59.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.687+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73500c7200 con 0x7f73500cda80 2023-10-16T14:10:59.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.687+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73500c77c0 con 0x7f73500cda80 2023-10-16T14:10:59.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.688+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735006b330 con 0x7f73500cda80 2023-10-16T14:10:59.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.689+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 130) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f73480113f0 con 0x7f73500cda80 2023-10-16T14:10:59.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.690+0000 7f734cff9640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f733c06ad10 0x7f733c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:10:59.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.690+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f7348089c00 con 0x7f73500cda80 2023-10-16T14:10:59.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.691+0000 7f734ffff640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f733c06ad10 0x7f733c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:10:59.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.692+0000 7f734ffff640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f733c06ad10 0x7f733c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7340035000 tx=0x7f7340030000 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:10:59.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.695+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7348054a60 con 0x7f73500cda80 2023-10-16T14:10:59.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:10:59.919+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f73500cc550 con 0x7f733c06ad10 2023-10-16T14:11:00.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.060+0000 7f734cff9640 1 -- 172.21.15.60:0/2809895699 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7340034470 con 0x7f733c06ad10 2023-10-16T14:11:00.063 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T14:11:00.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f733c06ad10 msgr2=0x7f733c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f733c06ad10 0x7f733c06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7340035000 tx=0x7f7340030000 comp rx=0 tx=0).stop 2023-10-16T14:11:00.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 msgr2=0x7f73500c62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 0x7f73500c62b0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f734800c7f0 tx=0x7f734800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 shutdown_connections 2023-10-16T14:11:00.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f733c06ad10 0x7f733c06d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f73500ad970 0x7f73500acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73500cda80 0x7f73500c62b0 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 --2- 172.21.15.60:0/2809895699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f735006c040 0x7f73500ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.063+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 >> 172.21.15.60:0/2809895699 conn(0x7f73500ce6a0 msgr2=0x7f735015e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:00.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.064+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 shutdown_connections 2023-10-16T14:11:00.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.064+0000 7f73554aa640 1 -- 172.21.15.60:0/2809895699 wait complete. 2023-10-16T14:11:00.089 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T14:11:00.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T14:11:00.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.327+0000 7f8906490640 1 Processor -- start 2023-10-16T14:11:00.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.327+0000 7f8906490640 1 -- start start 2023-10-16T14:11:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f8906490640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f8900160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f8906490640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89000ae260 0x7f890006a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f8906490640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000ce800 con 0x7f8900160ad0 2023-10-16T14:11:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f8906490640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89000cdec0 con 0x7f89000ae260 2023-10-16T14:11:00.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f8906490640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89001613f0 con 0x7f89000ce350 2023-10-16T14:11:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f8904a2d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89000ae260 0x7f890006a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f88fffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f8900160ad0 0x7f89000ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:50024/0 (socket says 172.21.15.60:50024) 2023-10-16T14:11:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f88fffff640 1 -- 172.21.15.60:0/4164029831 learned_addr learned my addr 172.21.15.60:0/4164029831 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:00.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.329+0000 7f88ff7fe640 1 --2- 172.21.15.60:0/4164029831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f8900160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.330+0000 7f8904a2d640 1 -- 172.21.15.60:0/4164029831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 msgr2=0x7f8900160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.330+0000 7f8904a2d640 1 --2- 172.21.15.60:0/4164029831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f8900160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.330+0000 7f8904a2d640 1 -- 172.21.15.60:0/4164029831 >> v1:172.21.15.144:6789/0 conn(0x7f8900160ad0 legacy=0x7f89000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.330+0000 7f8904a2d640 1 -- 172.21.15.60:0/4164029831 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000ae740 con 0x7f89000ae260 2023-10-16T14:11:00.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.330+0000 7f88feffd640 1 -- 172.21.15.60:0/4164029831 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547473910 0 0) 0x7f89000ce800 con 0x7f8900160ad0 2023-10-16T14:11:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.331+0000 7f8904a2d640 1 --2- 172.21.15.60:0/4164029831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890006a780 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f88f80097a0 tx=0x7f88f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e285df73fc0eae06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.331+0000 7f88feffd640 1 -- 172.21.15.60:0/4164029831 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8031830 con 0x7f89000ae260 2023-10-16T14:11:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.331+0000 7f88feffd640 1 -- 172.21.15.60:0/4164029831 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f8031990 con 0x7f89000ae260 2023-10-16T14:11:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.331+0000 7f88feffd640 1 -- 172.21.15.60:0/4164029831 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8031d20 con 0x7f89000ae260 2023-10-16T14:11:00.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 -- 172.21.15.60:0/4164029831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 msgr2=0x7f890006a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 --2- 172.21.15.60:0/4164029831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890006a780 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f88f80097a0 tx=0x7f88f802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 -- 172.21.15.60:0/4164029831 shutdown_connections 2023-10-16T14:11:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 --2- 172.21.15.60:0/4164029831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890006a780 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 --2- 172.21.15.60:0/4164029831 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f8900160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 -- 172.21.15.60:0/4164029831 >> 172.21.15.60:0/4164029831 conn(0x7f890006bc10 msgr2=0x7f890006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:00.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 -- 172.21.15.60:0/4164029831 shutdown_connections 2023-10-16T14:11:00.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.332+0000 7f8906490640 1 -- 172.21.15.60:0/4164029831 wait complete. 2023-10-16T14:11:00.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.334+0000 7f8906490640 1 Processor -- start 2023-10-16T14:11:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.334+0000 7f8906490640 1 -- start start 2023-10-16T14:11:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.335+0000 7f8906490640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.335+0000 7f8906490640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f89000c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.335+0000 7f8906490640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8900160ad0 0x7f89001f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:00.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.335+0000 7f8906490640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c2ce0 con 0x7f89000ae260 2023-10-16T14:11:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.335+0000 7f8906490640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890014ebf0 con 0x7f8900160ad0 2023-10-16T14:11:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.335+0000 7f8906490640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c3c30 con 0x7f89000ce350 2023-10-16T14:11:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890014e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55356/0 (socket says 172.21.15.60:55356) 2023-10-16T14:11:00.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 -- 172.21.15.60:0/3850969691 learned_addr learned my addr 172.21.15.60:0/3850969691 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88ff7fe640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f89000c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f8904a2d640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8900160ad0 0x7f89001f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 -- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 msgr2=0x7f89000c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f89000c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 -- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8900160ad0 msgr2=0x7f89001f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8900160ad0 0x7f89001f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 -- 172.21.15.60:0/3850969691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8009400 con 0x7f89000ae260 2023-10-16T14:11:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.336+0000 7f88fffff640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890014e6b0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f88f000c890 tx=0x7f88f000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.337+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f000a5e0 con 0x7f89000ae260 2023-10-16T14:11:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.337+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f000a760 con 0x7f89000ae260 2023-10-16T14:11:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.337+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f000ab10 con 0x7f89000ae260 2023-10-16T14:11:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.337+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89000c3260 con 0x7f89000ae260 2023-10-16T14:11:00.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.337+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89001f6d00 con 0x7f89000ae260 2023-10-16T14:11:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.338+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890004e160 con 0x7f89000ae260 2023-10-16T14:11:00.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.339+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 131) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f88f000ac70 con 0x7f89000ae260 2023-10-16T14:11:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.340+0000 7f88fcff9640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f88ec06b100 0x7f88ec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.341+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f88f0089350 con 0x7f89000ae260 2023-10-16T14:11:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.341+0000 7f88ff7fe640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f88ec06b100 0x7f88ec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:00.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.342+0000 7f88ff7fe640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f88ec06b100 0x7f88ec06d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f89000b4970 tx=0x7f88f4009190 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.345+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88f0053c90 con 0x7f89000ae260 2023-10-16T14:11:00.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.569+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f89000c5ca0 con 0x7f88ec06b100 2023-10-16T14:11:00.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.740+0000 7f88fcff9640 1 -- 172.21.15.60:0/3850969691 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f89000c5ca0 con 0x7f88ec06b100 2023-10-16T14:11:00.743 INFO:teuthology.orchestra.run.smithi060.stdout:Retention added to path mandatory_fs_argument_test_dir 2023-10-16T14:11:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.743+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f88ec06b100 msgr2=0x7f88ec06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.743+0000 7f8906490640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f88ec06b100 0x7f88ec06d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f89000b4970 tx=0x7f88f4009190 comp rx=0 tx=0).stop 2023-10-16T14:11:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.743+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 msgr2=0x7f890014e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:00.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.743+0000 7f8906490640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890014e6b0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f88f000c890 tx=0x7f88f000cd50 comp rx=0 tx=0).stop 2023-10-16T14:11:00.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 shutdown_connections 2023-10-16T14:11:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f88ec06b100 0x7f88ec06d5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f89000ce350 0x7f89000c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8900160ad0 0x7f89001f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 --2- 172.21.15.60:0/3850969691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89000ae260 0x7f890014e6b0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 >> 172.21.15.60:0/3850969691 conn(0x7f890006bc10 msgr2=0x7f8900069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:00.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 shutdown_connections 2023-10-16T14:11:00.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:00.744+0000 7f8906490640 1 -- 172.21.15.60:0/3850969691 wait complete. 2023-10-16T14:11:00.770 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2023-10-16T14:11:00.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T14:11:01.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.006+0000 7f2ec6911640 1 Processor -- start 2023-10-16T14:11:01.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.006+0000 7f2ec6911640 1 -- start start 2023-10-16T14:11:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.007+0000 7f2ec6911640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.007+0000 7f2ec6911640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ec0070950 0x7f2ec015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.007+0000 7f2ec6911640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec00ca3a0 con 0x7f2ec0159e70 2023-10-16T14:11:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.007+0000 7f2ec6911640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec00cad70 con 0x7f2ec0070950 2023-10-16T14:11:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.007+0000 7f2ec6911640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00c9d70 con 0x7f2ec01573e0 2023-10-16T14:11:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.008+0000 7f2ec6110640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ec0070950 0x7f2ec015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.008+0000 7f2ec590f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.008+0000 7f2ec6110640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ec0070950 0x7f2ec015a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:51702/0 (socket says 172.21.15.60:51702) 2023-10-16T14:11:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.008+0000 7f2ec6110640 1 -- 172.21.15.60:0/1364295687 learned_addr learned my addr 172.21.15.60:0/1364295687 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.009+0000 7f2ec6110640 1 -- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 msgr2=0x7f2ec00ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.009+0000 7f2ec6110640 1 --2- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.009+0000 7f2ec6110640 1 -- 172.21.15.60:0/1364295687 >> v1:172.21.15.144:6789/0 conn(0x7f2ec0159e70 legacy=0x7f2ec0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.009+0000 7f2ec6110640 1 -- 172.21.15.60:0/1364295687 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec01577c0 con 0x7f2ec0070950 2023-10-16T14:11:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.010+0000 7f2ec6110640 1 --2- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec015a2f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800bb10 tx=0x7f2eb802e150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3c0c26d795d5b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.010+0000 7f2ec490d640 1 -- 172.21.15.60:0/1364295687 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8036440 con 0x7f2ec0070950 2023-10-16T14:11:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec490d640 1 -- 172.21.15.60:0/1364295687 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb80365a0 con 0x7f2ec0070950 2023-10-16T14:11:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec490d640 1 -- 172.21.15.60:0/1364295687 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8036930 con 0x7f2ec0070950 2023-10-16T14:11:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 -- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 msgr2=0x7f2ec015a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec015a2f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800bb10 tx=0x7f2eb802e150 comp rx=0 tx=0).stop 2023-10-16T14:11:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 -- 172.21.15.60:0/1364295687 shutdown_connections 2023-10-16T14:11:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec015a2f0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1364295687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 -- 172.21.15.60:0/1364295687 >> 172.21.15.60:0/1364295687 conn(0x7f2ec00cc230 msgr2=0x7f2ec00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 -- 172.21.15.60:0/1364295687 shutdown_connections 2023-10-16T14:11:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.011+0000 7f2ec6911640 1 -- 172.21.15.60:0/1364295687 wait complete. 2023-10-16T14:11:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.013+0000 7f2ec6911640 1 Processor -- start 2023-10-16T14:11:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.013+0000 7f2ec6911640 1 -- start start 2023-10-16T14:11:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.014+0000 7f2ec6911640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec01f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.014+0000 7f2ec6911640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.014+0000 7f2ec6911640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 0x7f2ec00c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.014+0000 7f2ec6911640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec01e9050 con 0x7f2ec0159e70 2023-10-16T14:11:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.014+0000 7f2ec6911640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00c67f0 con 0x7f2ec0070950 2023-10-16T14:11:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.014+0000 7f2ec6911640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00c6ad0 con 0x7f2ec01573e0 2023-10-16T14:11:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.015+0000 7f2ec6110640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 0x7f2ec00c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.015+0000 7f2ec6110640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 0x7f2ec00c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55370/0 (socket says 172.21.15.60:55370) 2023-10-16T14:11:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.015+0000 7f2ec6110640 1 -- 172.21.15.60:0/1741285780 learned_addr learned my addr 172.21.15.60:0/1741285780 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.015+0000 7f2ec590f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec01f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.015+0000 7f2ec510e640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6110640 1 -- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 msgr2=0x7f2ec00c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6110640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6110640 1 -- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 msgr2=0x7f2ec01f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6110640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec01f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6110640 1 -- 172.21.15.60:0/1741285780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb800b770 con 0x7f2ec0159e70 2023-10-16T14:11:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6110640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 0x7f2ec00c80b0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f2eb802e190 tx=0x7f2eb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8036440 con 0x7f2ec0159e70 2023-10-16T14:11:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb8034980 con 0x7f2ec0159e70 2023-10-16T14:11:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8034d10 con 0x7f2ec0159e70 2023-10-16T14:11:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec00c6db0 con 0x7f2ec0159e70 2023-10-16T14:11:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.016+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec00c7370 con 0x7f2ec0159e70 2023-10-16T14:11:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.017+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec004e9b0 con 0x7f2ec0159e70 2023-10-16T14:11:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.018+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 131) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f2eb8041480 con 0x7f2ec0159e70 2023-10-16T14:11:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.019+0000 7f2eb67fc640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f2ea406b100 0x7f2ea406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.020+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f2eb80ad010 con 0x7f2ec0159e70 2023-10-16T14:11:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.020+0000 7f2ec590f640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f2ea406b100 0x7f2ea406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.022+0000 7f2ec590f640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f2ea406b100 0x7f2ea406d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0113e0 tx=0x7f2ebc00b040 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.024+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2eb8077820 con 0x7f2ec0159e70 2023-10-16T14:11:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.247+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ec015ac80 con 0x7f2ea406b100 2023-10-16T14:11:01.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.262+0000 7f2eb67fc640 1 -- 172.21.15.60:0/1741285780 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f2ebc00f040 con 0x7f2ea406b100 2023-10-16T14:11:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:01.266 INFO:teuthology.orchestra.run.smithi060.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T14:11:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.265+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f2ea406b100 msgr2=0x7f2ea406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.265+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f2ea406b100 0x7f2ea406d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0113e0 tx=0x7f2ebc00b040 comp rx=0 tx=0).stop 2023-10-16T14:11:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.265+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 msgr2=0x7f2ec00c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.265+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 0x7f2ec00c80b0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f2eb802e190 tx=0x7f2eb8004730 comp rx=0 tx=0).stop 2023-10-16T14:11:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 shutdown_connections 2023-10-16T14:11:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f2ea406b100 0x7f2ea406d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ec01573e0 0x7f2ec00c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ec0159e70 0x7f2ec00c80b0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 --2- 172.21.15.60:0/1741285780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ec0070950 0x7f2ec01f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 >> 172.21.15.60:0/1741285780 conn(0x7f2ec00cc230 msgr2=0x7f2ec0157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:01.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 shutdown_connections 2023-10-16T14:11:01.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.266+0000 7f2ec6911640 1 -- 172.21.15.60:0/1741285780 wait complete. 2023-10-16T14:11:01.289 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T14:11:01.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T14:11:01.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.519+0000 7fa93fc2d640 1 Processor -- start 2023-10-16T14:11:01.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.520+0000 7fa93fc2d640 1 -- start start 2023-10-16T14:11:01.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.521+0000 7fa93fc2d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 0x7fa93806f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.521+0000 7fa93fc2d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa93806f9d0 0x7fa9380c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93fc2d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380c9620 con 0x7fa9380c9ff0 2023-10-16T14:11:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93fc2d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93815d340 con 0x7fa93806f9d0 2023-10-16T14:11:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93fc2d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380cc860 con 0x7fa93815cf60 2023-10-16T14:11:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93e1ca640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa9380c9ff0 0x7fa9380b0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56978/0 (socket says 172.21.15.60:56978) 2023-10-16T14:11:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93e1ca640 1 -- 172.21.15.60:0/1082300035 learned_addr learned my addr 172.21.15.60:0/1082300035 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:01.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93d9c9640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 0x7fa93806f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/1082300035 >> v2:172.21.15.144:3300/0 conn(0x7fa93806f9d0 0x7fa9380c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.522+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603827039 0 0) 0x7fa9380c9620 con 0x7fa9380c9ff0 2023-10-16T14:11:01.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa924003520 con 0x7fa9380c9ff0 2023-10-16T14:11:01.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533331955 0 0) 0x7fa924003520 con 0x7fa9380c9ff0 2023-10-16T14:11:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 msgr2=0x7fa93806f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 0x7fa93806f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 >> v2:172.21.15.144:3300/0 conn(0x7fa93806f9d0 msgr2=0x7fa9380c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 --2- 172.21.15.60:0/1082300035 >> v2:172.21.15.144:3300/0 conn(0x7fa93806f9d0 0x7fa9380c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9380ad4a0 con 0x7fa9380c9ff0 2023-10-16T14:11:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa930001670 con 0x7fa9380c9ff0 2023-10-16T14:11:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 >> v1:172.21.15.60:6789/0 conn(0x7fa9380c9ff0 legacy=0x7fa9380b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 0x7fa9240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa924003fb0 0x7fa9240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004d00 0x7fa924005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380cc860 con 0x7fa924003fb0 2023-10-16T14:11:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92400b980 con 0x7fa924004d00 2023-10-16T14:11:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.523+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92400bb40 con 0x7fa93815cf60 2023-10-16T14:11:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa924003fb0 0x7fa9240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d9c9640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 0x7fa9240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93e1ca640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004d00 0x7fa924005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d1c8640 1 -- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 msgr2=0x7fa9240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa93815cf60 0x7fa9240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d1c8640 1 -- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004d00 msgr2=0x7fa924005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:01.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924004d00 0x7fa924005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.524+0000 7fa93d1c8640 1 -- 172.21.15.60:0/1082300035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924003520 con 0x7fa924003fb0 2023-10-16T14:11:01.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.525+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa924003fb0 0x7fa9240047c0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fa928009ac0 tx=0x7fa9240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7c4851b4f1fe4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92802c710 con 0x7fa924003fb0 2023-10-16T14:11:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92802c870 con 0x7fa924003fb0 2023-10-16T14:11:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93c9c7640 1 -- 172.21.15.60:0/1082300035 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92802cc00 con 0x7fa924003fb0 2023-10-16T14:11:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93fc2d640 1 -- 172.21.15.60:0/1082300035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa924003fb0 msgr2=0x7fa9240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/1082300035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa924003fb0 0x7fa9240047c0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fa928009ac0 tx=0x7fa9240057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93e1ca640 1 -- 172.21.15.60:0/1082300035 reap_dead start 2023-10-16T14:11:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93fc2d640 1 -- 172.21.15.60:0/1082300035 shutdown_connections 2023-10-16T14:11:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93fc2d640 1 -- 172.21.15.60:0/1082300035 >> 172.21.15.60:0/1082300035 conn(0x7fa9380cb360 msgr2=0x7fa9380cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:01.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93fc2d640 1 -- 172.21.15.60:0/1082300035 shutdown_connections 2023-10-16T14:11:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.526+0000 7fa93fc2d640 1 -- 172.21.15.60:0/1082300035 wait complete. 2023-10-16T14:11:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.528+0000 7fa93fc2d640 1 Processor -- start 2023-10-16T14:11:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.528+0000 7fa93fc2d640 1 -- start start 2023-10-16T14:11:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.529+0000 7fa93fc2d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003fb0 0x7fa93815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.529+0000 7fa93fc2d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 0x7fa9380b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.529+0000 7fa93fc2d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9380b41b0 0x7fa93806c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.529+0000 7fa93fc2d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b4650 con 0x7fa9381e6850 2023-10-16T14:11:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.529+0000 7fa93fc2d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93815d470 con 0x7fa924003fb0 2023-10-16T14:11:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.529+0000 7fa93fc2d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93815d750 con 0x7fa9380b41b0 2023-10-16T14:11:01.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 0x7fa9380b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 0x7fa9380b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55388/0 (socket says 172.21.15.60:55388) 2023-10-16T14:11:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 -- 172.21.15.60:0/2462433804 learned_addr learned my addr 172.21.15.60:0/2462433804 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d9c9640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003fb0 0x7fa93815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93e1ca640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9380b41b0 0x7fa93806c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 -- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9380b41b0 msgr2=0x7fa93806c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9380b41b0 0x7fa93806c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 -- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003fb0 msgr2=0x7fa93815aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003fb0 0x7fa93815aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 -- 172.21.15.60:0/2462433804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93815da30 con 0x7fa9381e6850 2023-10-16T14:11:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d9c9640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003fb0 0x7fa93815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.530+0000 7fa93d1c8640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 0x7fa9380b3c70 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fa92802c2b0 tx=0x7fa928004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.531+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928003810 con 0x7fa9381e6850 2023-10-16T14:11:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.531+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92802de90 con 0x7fa9381e6850 2023-10-16T14:11:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.531+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928003b70 con 0x7fa9381e6850 2023-10-16T14:11:01.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.531+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93815dd10 con 0x7fa9381e6850 2023-10-16T14:11:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.531+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9381608a0 con 0x7fa9381e6850 2023-10-16T14:11:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.532+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa938160a90 con 0x7fa9381e6850 2023-10-16T14:11:01.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.533+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 131) v1 ==== 89451+0+0 (secure 0 0 0) 0x7fa928003cd0 con 0x7fa9381e6850 2023-10-16T14:11:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.534+0000 7fa92e7fc640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fa91c06b130 0x7fa91c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.534+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fa9280abe20 con 0x7fa9381e6850 2023-10-16T14:11:01.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.535+0000 7fa93d9c9640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fa91c06b130 0x7fa91c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.536+0000 7fa93d9c9640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fa91c06b130 0x7fa91c06d5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa934012430 tx=0x7fa93400b040 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.539+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa938160a90 con 0x7fa9381e6850 2023-10-16T14:11:01.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.758+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa93815d5e0 con 0x7fa91c06b130 2023-10-16T14:11:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.774+0000 7fa92e7fc640 1 -- 172.21.15.60:0/2462433804 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7fa93815d5e0 con 0x7fa91c06b130 2023-10-16T14:11:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:01.777 INFO:teuthology.orchestra.run.smithi060.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:10:59", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T14:11:01.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fa91c06b130 msgr2=0x7fa91c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fa91c06b130 0x7fa91c06d5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa934012430 tx=0x7fa93400b040 comp rx=0 tx=0).stop 2023-10-16T14:11:01.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 msgr2=0x7fa9380b3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:01.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 0x7fa9380b3c70 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fa92802c2b0 tx=0x7fa928004730 comp rx=0 tx=0).stop 2023-10-16T14:11:01.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 shutdown_connections 2023-10-16T14:11:01.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fa91c06b130 0x7fa91c06d5f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa9380b41b0 0x7fa93806c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9381e6850 0x7fa9380b3c70 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 --2- 172.21.15.60:0/2462433804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa924003fb0 0x7fa93815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:01.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 >> 172.21.15.60:0/2462433804 conn(0x7fa9380cb360 msgr2=0x7fa9380cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:01.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 shutdown_connections 2023-10-16T14:11:01.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:01.777+0000 7fa93fc2d640 1 -- 172.21.15.60:0/2462433804 wait complete. 2023-10-16T14:11:01.800 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T14:10:59", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T14:11:01.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T14:11:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.024+0000 7efd61727640 1 Processor -- start 2023-10-16T14:11:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.024+0000 7efd61727640 1 -- start start 2023-10-16T14:11:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.026+0000 7efd61727640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efd5c15cc00 0x7efd5c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.026+0000 7efd61727640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 0x7efd5c0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.026+0000 7efd61727640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c0ce480 con 0x7efd5c0ad670 2023-10-16T14:11:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.026+0000 7efd61727640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c0ca950 con 0x7efd5c15cc00 2023-10-16T14:11:02.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.026+0000 7efd61727640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0cd7b0 con 0x7efd5c0cd260 2023-10-16T14:11:02.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.027+0000 7efd60f26640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd5c0ad670 0x7efd5c0c8cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56994/0 (socket says 172.21.15.60:56994) 2023-10-16T14:11:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.027+0000 7efd60f26640 1 -- 172.21.15.60:0/654860638 learned_addr learned my addr 172.21.15.60:0/654860638 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.027+0000 7efd5bfff640 1 --2- 172.21.15.60:0/654860638 >> v2:172.21.15.144:3300/0 conn(0x7efd5c15cc00 0x7efd5c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.027+0000 7efd5b7fe640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 0x7efd5c0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.027+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466247955 0 0) 0x7efd5c0ce480 con 0x7efd5c0ad670 2023-10-16T14:11:02.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.027+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd44003520 con 0x7efd5c0ad670 2023-10-16T14:11:02.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895682166 0 0) 0x7efd44003520 con 0x7efd5c0ad670 2023-10-16T14:11:02.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 msgr2=0x7efd5c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:02.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 0x7efd5c0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 >> v2:172.21.15.144:3300/0 conn(0x7efd5c15cc00 msgr2=0x7efd5c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:02.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 --2- 172.21.15.60:0/654860638 >> v2:172.21.15.144:3300/0 conn(0x7efd5c15cc00 0x7efd5c0ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0cda90 con 0x7efd5c0ad670 2023-10-16T14:11:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efd540019f0 con 0x7efd5c0ad670 2023-10-16T14:11:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 >> v1:172.21.15.60:6789/0 conn(0x7efd5c0ad670 legacy=0x7efd5c0c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd44003970 0x7efd44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd440047c0 0x7efd44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 0x7efd44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0ca950 con 0x7efd44003970 2023-10-16T14:11:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0cd7b0 con 0x7efd440047c0 2023-10-16T14:11:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4400b980 con 0x7efd5c0cd260 2023-10-16T14:11:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5bfff640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd44003970 0x7efd44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5b7fe640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd440047c0 0x7efd44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5bfff640 1 -- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 msgr2=0x7efd44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5bfff640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd5c0cd260 0x7efd44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5bfff640 1 -- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd440047c0 msgr2=0x7efd44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5bfff640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd440047c0 0x7efd44004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5bfff640 1 -- 172.21.15.60:0/654860638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd44003520 con 0x7efd44003970 2023-10-16T14:11:02.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.028+0000 7efd5b7fe640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd440047c0 0x7efd44004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd5bfff640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd44003970 0x7efd44003d90 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009ac0 tx=0x7efd440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d7d46e961abd35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c02c710 con 0x7efd44003970 2023-10-16T14:11:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c02c870 con 0x7efd44003970 2023-10-16T14:11:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd5affd640 1 -- 172.21.15.60:0/654860638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c02cc00 con 0x7efd44003970 2023-10-16T14:11:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd61727640 1 -- 172.21.15.60:0/654860638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd44003970 msgr2=0x7efd44003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd61727640 1 --2- 172.21.15.60:0/654860638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd44003970 0x7efd44003d90 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009ac0 tx=0x7efd440057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd60f26640 1 -- 172.21.15.60:0/654860638 reap_dead start 2023-10-16T14:11:02.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd61727640 1 -- 172.21.15.60:0/654860638 shutdown_connections 2023-10-16T14:11:02.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.030+0000 7efd61727640 1 -- 172.21.15.60:0/654860638 >> 172.21.15.60:0/654860638 conn(0x7efd5c0b0170 msgr2=0x7efd5c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:02.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.031+0000 7efd61727640 1 -- 172.21.15.60:0/654860638 shutdown_connections 2023-10-16T14:11:02.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.031+0000 7efd61727640 1 -- 172.21.15.60:0/654860638 wait complete. 2023-10-16T14:11:02.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.032+0000 7efd61727640 1 Processor -- start 2023-10-16T14:11:02.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.033+0000 7efd61727640 1 -- start start 2023-10-16T14:11:02.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.033+0000 7efd61727640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd44003970 0x7efd5c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.033+0000 7efd61727640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 0x7efd5c1f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd61727640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5c0b1b40 0x7efd5c1f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd61727640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0c7170 con 0x7efd5c150700 2023-10-16T14:11:02.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd61727640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0c7450 con 0x7efd5c0b1b40 2023-10-16T14:11:02.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd61727640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0af660 con 0x7efd44003970 2023-10-16T14:11:02.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 0x7efd5c1f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 0x7efd5c1f23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55410/0 (socket says 172.21.15.60:55410) 2023-10-16T14:11:02.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 -- 172.21.15.60:0/1971899745 learned_addr learned my addr 172.21.15.60:0/1971899745 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:02.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5bfff640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd44003970 0x7efd5c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd60f26640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5c0b1b40 0x7efd5c1f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 -- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd44003970 msgr2=0x7efd5c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd44003970 0x7efd5c0b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 -- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5c0b1b40 msgr2=0x7efd5c1f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5c0b1b40 0x7efd5c1f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd5b7fe640 1 -- 172.21.15.60:0/1971899745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0af940 con 0x7efd5c150700 2023-10-16T14:11:02.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.034+0000 7efd60f26640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5c0b1b40 0x7efd5c1f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:02.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.035+0000 7efd5b7fe640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 0x7efd5c1f23a0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7efd500108e0 tx=0x7efd50010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:02.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.035+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5000ec20 con 0x7efd5c150700 2023-10-16T14:11:02.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.035+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5000ed80 con 0x7efd5c150700 2023-10-16T14:11:02.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.035+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5001c600 con 0x7efd5c150700 2023-10-16T14:11:02.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.035+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c150ee0 con 0x7efd5c150700 2023-10-16T14:11:02.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.035+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c0b21a0 con 0x7efd5c150700 2023-10-16T14:11:02.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.036+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c0674f0 con 0x7efd5c150700 2023-10-16T14:11:02.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.037+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 131) v1 ==== 89451+0+0 (secure 0 0 0) 0x7efd5001c760 con 0x7efd5c150700 2023-10-16T14:11:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.038+0000 7efd58ff9640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efd4806b130 0x7efd4806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.038+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7efd5008a5f0 con 0x7efd5c150700 2023-10-16T14:11:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.039+0000 7efd5bfff640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efd4806b130 0x7efd4806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.040+0000 7efd5bfff640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efd4806b130 0x7efd4806d5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd4c036000 tx=0x7efd4c030000 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.043+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd50054f60 con 0x7efd5c150700 2023-10-16T14:11:02.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.259+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7efd5c0ac930 con 0x7efd4806b130 2023-10-16T14:11:02.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.415+0000 7efd58ff9640 1 -- 172.21.15.60:0/1971899745 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7efd4c02ca60 con 0x7efd4806b130 2023-10-16T14:11:02.418 INFO:teuthology.orchestra.run.smithi060.stdout:Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T14:11:02.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efd4806b130 msgr2=0x7efd4806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efd4806b130 0x7efd4806d5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd4c036000 tx=0x7efd4c030000 comp rx=0 tx=0).stop 2023-10-16T14:11:02.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 msgr2=0x7efd5c1f23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 0x7efd5c1f23a0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7efd500108e0 tx=0x7efd50010da0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 shutdown_connections 2023-10-16T14:11:02.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efd4806b130 0x7efd4806d5f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd5c0b1b40 0x7efd5c1f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5c150700 0x7efd5c1f23a0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 --2- 172.21.15.60:0/1971899745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efd44003970 0x7efd5c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 >> 172.21.15.60:0/1971899745 conn(0x7efd5c0b0170 msgr2=0x7efd5c0c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:02.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 shutdown_connections 2023-10-16T14:11:02.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.418+0000 7efd61727640 1 -- 172.21.15.60:0/1971899745 wait complete. 2023-10-16T14:11:02.440 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T14:11:02.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T14:11:02.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.665+0000 7ffbd241a640 1 Processor -- start 2023-10-16T14:11:02.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.666+0000 7ffbd241a640 1 -- start start 2023-10-16T14:11:02.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.667+0000 7ffbd241a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffbcc15c9f0 0x7ffbcc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.667+0000 7ffbd241a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffbcc0c9230 0x7ffbcc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.667+0000 7ffbd241a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc160a90 con 0x7ffbcc070f20 2023-10-16T14:11:02.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.667+0000 7ffbd241a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc0cdec0 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.667+0000 7ffbd241a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc0cd900 con 0x7ffbcc0c9230 2023-10-16T14:11:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffbcc15c9f0 0x7ffbcc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffbcc15c9f0 0x7ffbcc0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55414/0 (socket says 172.21.15.60:55414) 2023-10-16T14:11:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 -- 172.21.15.60:0/320597386 learned_addr learned my addr 172.21.15.60:0/320597386 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd0c17640 1 --2- 172.21.15.60:0/320597386 >> v2:172.21.15.144:3300/0 conn(0x7ffbcc0c9230 0x7ffbcc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 -- 172.21.15.60:0/320597386 >> v2:172.21.15.144:3300/0 conn(0x7ffbcc0c9230 msgr2=0x7ffbcc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:02.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 --2- 172.21.15.60:0/320597386 >> v2:172.21.15.144:3300/0 conn(0x7ffbcc0c9230 0x7ffbcc0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 -- 172.21.15.60:0/320597386 >> v1:172.21.15.144:6789/0 conn(0x7ffbcc070f20 legacy=0x7ffbcc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.668+0000 7ffbd1418640 1 -- 172.21.15.60:0/320597386 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc0cdbe0 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.670+0000 7ffbd1418640 1 --2- 172.21.15.60:0/320597386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0c8cf0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0b1260 tx=0x7ffbc802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea2f4197e077333 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:02.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbc3fff640 1 -- 172.21.15.60:0/320597386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc802cb20 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbc3fff640 1 -- 172.21.15.60:0/320597386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc802cc80 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbc3fff640 1 -- 172.21.15.60:0/320597386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80343f0 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 -- 172.21.15.60:0/320597386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 msgr2=0x7ffbcc0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 --2- 172.21.15.60:0/320597386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0c8cf0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0b1260 tx=0x7ffbc802bf80 comp rx=0 tx=0).stop 2023-10-16T14:11:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 -- 172.21.15.60:0/320597386 shutdown_connections 2023-10-16T14:11:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 --2- 172.21.15.60:0/320597386 >> v2:172.21.15.144:3300/0 conn(0x7ffbcc0c9230 0x7ffbcc0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 --2- 172.21.15.60:0/320597386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0c8cf0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 -- 172.21.15.60:0/320597386 >> 172.21.15.60:0/320597386 conn(0x7ffbcc0ce6a0 msgr2=0x7ffbcc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:02.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.671+0000 7ffbd241a640 1 -- 172.21.15.60:0/320597386 shutdown_connections 2023-10-16T14:11:02.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.672+0000 7ffbd241a640 1 -- 172.21.15.60:0/320597386 wait complete. 2023-10-16T14:11:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.673+0000 7ffbd241a640 1 Processor -- start 2023-10-16T14:11:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.673+0000 7ffbd241a640 1 -- start start 2023-10-16T14:11:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.674+0000 7ffbd241a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffbcc070f20 0x7ffbcc146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.674+0000 7ffbd241a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbcc0c9230 0x7ffbcc0ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.674+0000 7ffbd241a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.674+0000 7ffbd241a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc147120 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.674+0000 7ffbd241a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc15dbe0 con 0x7ffbcc0c9230 2023-10-16T14:11:02.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.674+0000 7ffbd241a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc147370 con 0x7ffbcc070f20 2023-10-16T14:11:02.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0ca890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55430/0 (socket says 172.21.15.60:55430) 2023-10-16T14:11:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 -- 172.21.15.60:0/1882483029 learned_addr learned my addr 172.21.15.60:0/1882483029 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd0c17640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbcc0c9230 0x7ffbcc0ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1418640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffbcc070f20 0x7ffbcc146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 -- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffbcc070f20 msgr2=0x7ffbcc146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffbcc070f20 0x7ffbcc146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 -- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbcc0c9230 msgr2=0x7ffbcc0ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:02.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbcc0c9230 0x7ffbcc0ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:02.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.675+0000 7ffbd1c19640 1 -- 172.21.15.60:0/1882483029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc8009b30 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.676+0000 7ffbd1c19640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0ca890 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4001eb0 tx=0x7ffbc400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:02.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.676+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc40052d0 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.676+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4005430 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.676+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc147ef0 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.676+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4003810 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.676+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc1478a0 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.677+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc148080 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.678+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 131) v1 ==== 89451+0+0 (secure 0 0 0) 0x7ffbc4003970 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.680+0000 7ffbc1ffb640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ffbb006b100 0x7ffbb006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.680+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7ffbc4089410 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.680+0000 7ffbd1418640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ffbb006b100 0x7ffbb006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.681+0000 7ffbd1418640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ffbb006b100 0x7ffbb006d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8035000 tx=0x7ffbc8030000 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:02.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.684+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffbc4053d70 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.846+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 132) v1 ==== 89507+0+0 (secure 0 0 0) 0x7ffbc4054140 con 0x7ffbcc15c9f0 2023-10-16T14:11:02.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:02.899+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbcc0704c0 con 0x7ffbb006b100 2023-10-16T14:11:03.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.061+0000 7ffbc1ffb640 1 -- 172.21.15.60:0/1882483029 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7ffbc80440e0 con 0x7ffbb006b100 2023-10-16T14:11:03.064 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T14:11:03.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.063+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ffbb006b100 msgr2=0x7ffbb006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.063+0000 7ffbd241a640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ffbb006b100 0x7ffbb006d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8035000 tx=0x7ffbc8030000 comp rx=0 tx=0).stop 2023-10-16T14:11:03.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.063+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 msgr2=0x7ffbcc0ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.063+0000 7ffbd241a640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0ca890 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4001eb0 tx=0x7ffbc400c890 comp rx=0 tx=0).stop 2023-10-16T14:11:03.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 shutdown_connections 2023-10-16T14:11:03.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ffbb006b100 0x7ffbb006d5c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbcc15c9f0 0x7ffbcc0ca890 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffbcc0c9230 0x7ffbcc0ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 --2- 172.21.15.60:0/1882483029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffbcc070f20 0x7ffbcc146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 >> 172.21.15.60:0/1882483029 conn(0x7ffbcc0ce6a0 msgr2=0x7ffbcc15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:03.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 shutdown_connections 2023-10-16T14:11:03.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.064+0000 7ffbd241a640 1 -- 172.21.15.60:0/1882483029 wait complete. 2023-10-16T14:11:03.086 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T14:11:03.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T14:11:03.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.311+0000 7efe5eb93640 1 Processor -- start 2023-10-16T14:11:03.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.311+0000 7efe5eb93640 1 -- start start 2023-10-16T14:11:03.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5eb93640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe5815c9f0 0x7efe580c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:03.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5eb93640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe5815dde0 0x7efe580ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:03.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5eb93640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe58160a90 con 0x7efe580c9230 2023-10-16T14:11:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5eb93640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe580cdec0 con 0x7efe5815dde0 2023-10-16T14:11:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5eb93640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe580cd900 con 0x7efe5815c9f0 2023-10-16T14:11:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5e392640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe5815dde0 0x7efe580ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5e392640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe5815dde0 0x7efe580ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55446/0 (socket says 172.21.15.60:55446) 2023-10-16T14:11:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5e392640 1 -- 172.21.15.60:0/1536434002 learned_addr learned my addr 172.21.15.60:0/1536434002 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.313+0000 7efe5db91640 1 --2- 172.21.15.60:0/1536434002 >> v2:172.21.15.144:3300/0 conn(0x7efe5815c9f0 0x7efe580c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:03.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.314+0000 7efe5e392640 1 -- 172.21.15.60:0/1536434002 >> v2:172.21.15.144:3300/0 conn(0x7efe5815c9f0 msgr2=0x7efe580c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:03.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.314+0000 7efe5e392640 1 --2- 172.21.15.60:0/1536434002 >> v2:172.21.15.144:3300/0 conn(0x7efe5815c9f0 0x7efe580c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.314+0000 7efe5e392640 1 -- 172.21.15.60:0/1536434002 >> v1:172.21.15.144:6789/0 conn(0x7efe580c9230 legacy=0x7efe580709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:03.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.314+0000 7efe5e392640 1 -- 172.21.15.60:0/1536434002 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580cdbe0 con 0x7efe5815dde0 2023-10-16T14:11:03.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.315+0000 7efe5e392640 1 --2- 172.21.15.60:0/1536434002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815dde0 0x7efe580ac890 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7efe580ad4e0 tx=0x7efe5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343a73880204b62f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:03.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.315+0000 7efe5cb8f640 1 -- 172.21.15.60:0/1536434002 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5002ca80 con 0x7efe5815dde0 2023-10-16T14:11:03.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.315+0000 7efe5cb8f640 1 -- 172.21.15.60:0/1536434002 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5002cbe0 con 0x7efe5815dde0 2023-10-16T14:11:03.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.315+0000 7efe5cb8f640 1 -- 172.21.15.60:0/1536434002 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe500343f0 con 0x7efe5815dde0 2023-10-16T14:11:03.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 -- 172.21.15.60:0/1536434002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815dde0 msgr2=0x7efe580ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 --2- 172.21.15.60:0/1536434002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815dde0 0x7efe580ac890 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7efe580ad4e0 tx=0x7efe5002bee0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 -- 172.21.15.60:0/1536434002 shutdown_connections 2023-10-16T14:11:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 --2- 172.21.15.60:0/1536434002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815dde0 0x7efe580ac890 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 --2- 172.21.15.60:0/1536434002 >> v2:172.21.15.144:3300/0 conn(0x7efe5815c9f0 0x7efe580c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 -- 172.21.15.60:0/1536434002 >> 172.21.15.60:0/1536434002 conn(0x7efe580ce6a0 msgr2=0x7efe580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 -- 172.21.15.60:0/1536434002 shutdown_connections 2023-10-16T14:11:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.316+0000 7efe5eb93640 1 -- 172.21.15.60:0/1536434002 wait complete. 2023-10-16T14:11:03.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.318+0000 7efe5eb93640 1 Processor -- start 2023-10-16T14:11:03.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.318+0000 7efe5eb93640 1 -- start start 2023-10-16T14:11:03.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5eb93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 0x7efe5806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:03.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5eb93640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815c9f0 0x7efe580c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:03.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5eb93640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5815dde0 0x7efe580ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:03.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5eb93640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580c81d0 con 0x7efe580c9230 2023-10-16T14:11:03.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5eb93640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580c7130 con 0x7efe5815c9f0 2023-10-16T14:11:03.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5eb93640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe580c7410 con 0x7efe5815dde0 2023-10-16T14:11:03.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5db91640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 0x7efe5806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:03.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5db91640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 0x7efe5806c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55460/0 (socket says 172.21.15.60:55460) 2023-10-16T14:11:03.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.319+0000 7efe5db91640 1 -- 172.21.15.60:0/3390838873 learned_addr learned my addr 172.21.15.60:0/3390838873 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:03.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5db91640 1 -- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5815dde0 msgr2=0x7efe580ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5db91640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5815dde0 0x7efe580ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5db91640 1 -- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815c9f0 msgr2=0x7efe580c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5d390640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815c9f0 0x7efe580c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:03.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5db91640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815c9f0 0x7efe580c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5db91640 1 -- 172.21.15.60:0/3390838873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe50009b30 con 0x7efe580c9230 2023-10-16T14:11:03.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.320+0000 7efe5db91640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 0x7efe5806c5d0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7efe5400d800 tx=0x7efe5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:03.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.321+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5400b6c0 con 0x7efe580c9230 2023-10-16T14:11:03.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.321+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5400b820 con 0x7efe580c9230 2023-10-16T14:11:03.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.321+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5400bbb0 con 0x7efe580c9230 2023-10-16T14:11:03.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.321+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe580c76f0 con 0x7efe580c9230 2023-10-16T14:11:03.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.321+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe580c7cb0 con 0x7efe580c9230 2023-10-16T14:11:03.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.322+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5804e9b0 con 0x7efe580c9230 2023-10-16T14:11:03.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.323+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 132) v1 ==== 89507+0+0 (secure 0 0 0) 0x7efe5400bd10 con 0x7efe580c9230 2023-10-16T14:11:03.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.324+0000 7efe4e7fc640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efe3c06b190 0x7efe3c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:03.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.324+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7efe5408a510 con 0x7efe580c9230 2023-10-16T14:11:03.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.324+0000 7efe5d390640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efe3c06b190 0x7efe3c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:03.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.326+0000 7efe5d390640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efe3c06b190 0x7efe3c06d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efe580c55b0 tx=0x7efe48009040 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:03.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.328+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe54054d70 con 0x7efe580c9230 2023-10-16T14:11:03.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.550+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 --> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7efe580c7580 con 0x7efe3c06b190 2023-10-16T14:11:03.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.724+0000 7efe4e7fc640 1 -- 172.21.15.60:0/3390838873 <== mgr.7919 v2:172.21.15.144:6832/1394204497 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7efe580c7580 con 0x7efe3c06b190 2023-10-16T14:11:03.728 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T14:11:03.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.728+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efe3c06b190 msgr2=0x7efe3c06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.728+0000 7efe5eb93640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efe3c06b190 0x7efe3c06d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efe580c55b0 tx=0x7efe48009040 comp rx=0 tx=0).stop 2023-10-16T14:11:03.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.728+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 msgr2=0x7efe5806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:03.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.728+0000 7efe5eb93640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 0x7efe5806c5d0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7efe5400d800 tx=0x7efe5400dcc0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 shutdown_connections 2023-10-16T14:11:03.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7efe3c06b190 0x7efe3c06d650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe5815dde0 0x7efe580ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815c9f0 0x7efe580c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 --2- 172.21.15.60:0/3390838873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe580c9230 0x7efe5806c5d0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:03.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 >> 172.21.15.60:0/3390838873 conn(0x7efe580ce6a0 msgr2=0x7efe580ca120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:03.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 shutdown_connections 2023-10-16T14:11:03.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:03.729+0000 7efe5eb93640 1 -- 172.21.15.60:0/3390838873 wait complete. 2023-10-16T14:11:03.754 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T14:11:03.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:03.755 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T14:11:03.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:11:04.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.050+0000 7f1c4ed1d640 1 Processor -- start 2023-10-16T14:11:04.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.051+0000 7f1c4ed1d640 1 -- start start 2023-10-16T14:11:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.052+0000 7f1c4ed1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1c4815ad70 0x7f1c48069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.052+0000 7f1c4ed1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480cb030 con 0x7f1c480caac0 2023-10-16T14:11:04.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.052+0000 7f1c4ed1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480c91c0 con 0x7f1c480c9ca0 2023-10-16T14:11:04.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.052+0000 7f1c4ed1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480c94a0 con 0x7f1c4815ad70 2023-10-16T14:11:04.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c4d2ba640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1c480caac0 0x7f1c480ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57008/0 (socket says 172.21.15.60:57008) 2023-10-16T14:11:04.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c4d2ba640 1 -- 172.21.15.60:0/1091345561 learned_addr learned my addr 172.21.15.60:0/1091345561 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:04.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/1091345561 >> v2:172.21.15.144:3300/0 conn(0x7f1c4815ad70 0x7f1c48069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821510628 0 0) 0x7f1c480cb030 con 0x7f1c480caac0 2023-10-16T14:11:04.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30003520 con 0x7f1c480caac0 2023-10-16T14:11:04.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915044955 0 0) 0x7f1c30003520 con 0x7f1c480caac0 2023-10-16T14:11:04.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 >> v2:172.21.15.144:3300/0 conn(0x7f1c4815ad70 msgr2=0x7f1c48069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:04.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.053+0000 7f1c477fe640 1 --2- 172.21.15.60:0/1091345561 >> v2:172.21.15.144:3300/0 conn(0x7f1c4815ad70 0x7f1c48069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 >> v1:172.21.15.144:6789/0 conn(0x7f1c480c9ca0 legacy=0x7f1c480ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480c9780 con 0x7f1c480caac0 2023-10-16T14:11:04.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f1c40001670 con 0x7f1c480caac0 2023-10-16T14:11:04.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 >> v1:172.21.15.60:6789/0 conn(0x7f1c480caac0 legacy=0x7f1c480ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c30003970 0x7f1c30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c300047c0 0x7f1c30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30005180 0x7f1c3000fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480c94a0 con 0x7f1c30003970 2023-10-16T14:11:04.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c30010430 con 0x7f1c30005180 2023-10-16T14:11:04.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c300105f0 con 0x7f1c300047c0 2023-10-16T14:11:04.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c30003970 0x7f1c30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c47fff640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c300047c0 0x7f1c30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c4d2ba640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30005180 0x7f1c3000fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c4cab9640 1 -- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c300047c0 msgr2=0x7f1c30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:04.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.054+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c300047c0 0x7f1c30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.055+0000 7f1c4cab9640 1 -- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30005180 msgr2=0x7f1c3000fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:04.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.055+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30005180 0x7f1c3000fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.055+0000 7f1c4cab9640 1 -- 172.21.15.60:0/1091345561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30003520 con 0x7f1c30003970 2023-10-16T14:11:04.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.055+0000 7f1c47fff640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c300047c0 0x7f1c30004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:04.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.055+0000 7f1c4d2ba640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30005180 0x7f1c3000fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:04.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.055+0000 7f1c4d2ba640 1 -- 172.21.15.60:0/1091345561 reap_dead start 2023-10-16T14:11:04.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.056+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c30003970 0x7f1c30003d90 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009fc0 tx=0x7f1c300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dde3e9cdc48d0ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:04.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.056+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802cc20 con 0x7f1c30003970 2023-10-16T14:11:04.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.056+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3800f040 con 0x7f1c30003970 2023-10-16T14:11:04.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.056+0000 7f1c477fe640 1 -- 172.21.15.60:0/1091345561 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38034440 con 0x7f1c30003970 2023-10-16T14:11:04.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.056+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/1091345561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c30003970 msgr2=0x7f1c30003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.056+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c30003970 0x7f1c30003d90 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009fc0 tx=0x7f1c300057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.057+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/1091345561 shutdown_connections 2023-10-16T14:11:04.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.057+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/1091345561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c30003970 0x7f1c30003d90 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.057+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/1091345561 >> 172.21.15.60:0/1091345561 conn(0x7f1c480ce9b0 msgr2=0x7f1c480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:04.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.057+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/1091345561 shutdown_connections 2023-10-16T14:11:04.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.057+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/1091345561 wait complete. 2023-10-16T14:11:04.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.059+0000 7f1c4ed1d640 1 Processor -- start 2023-10-16T14:11:04.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.059+0000 7f1c4ed1d640 1 -- start start 2023-10-16T14:11:04.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4ed1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30003970 0x7f1c4815d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4ed1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c481565e0 0x7f1c48156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4ed1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 0x7f1c480c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4ed1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48159770 con 0x7f1c480c5a80 2023-10-16T14:11:04.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4ed1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4815ceb0 con 0x7f1c30003970 2023-10-16T14:11:04.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4ed1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48156f80 con 0x7f1c481565e0 2023-10-16T14:11:04.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4d2ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 0x7f1c480c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4d2ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 0x7f1c480c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55472/0 (socket says 172.21.15.60:55472) 2023-10-16T14:11:04.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4d2ba640 1 -- 172.21.15.60:0/2001815158 learned_addr learned my addr 172.21.15.60:0/2001815158 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:04.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c47fff640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c481565e0 0x7f1c48156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.060+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30003970 0x7f1c4815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4d2ba640 1 -- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c481565e0 msgr2=0x7f1c48156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4d2ba640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c481565e0 0x7f1c48156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4d2ba640 1 -- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30003970 msgr2=0x7f1c4815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4d2ba640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30003970 0x7f1c4815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4d2ba640 1 -- 172.21.15.60:0/2001815158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4815de40 con 0x7f1c480c5a80 2023-10-16T14:11:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4d2ba640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 0x7f1c480c5ee0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f1c480c5a30 tx=0x7f1c4000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40007440 con 0x7f1c480c5a80 2023-10-16T14:11:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c400075c0 con 0x7f1c480c5a80 2023-10-16T14:11:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40004a20 con 0x7f1c480c5a80 2023-10-16T14:11:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.061+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c480c6970 con 0x7f1c480c5a80 2023-10-16T14:11:04.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.062+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4815d100 con 0x7f1c480c5a80 2023-10-16T14:11:04.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.063+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4804e160 con 0x7f1c480c5a80 2023-10-16T14:11:04.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.064+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 132) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f1c40004b80 con 0x7f1c480c5a80 2023-10-16T14:11:04.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.065+0000 7f1c457fa640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1c3406b1c0 0x7f1c3406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:04.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.065+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f1c40089ac0 con 0x7f1c480c5a80 2023-10-16T14:11:04.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.066+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1c3406b1c0 0x7f1c3406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:04.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.067+0000 7f1c4cab9640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1c3406b1c0 0x7f1c3406d680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c38036000 tx=0x7f1c3802e000 comp rx=0 tx=0).ready entity=mgr.7919 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:04.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.070+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c40054180 con 0x7f1c480c5a80 2023-10-16T14:11:04.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.320+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f1c480736e0 con 0x7f1c480c5a80 2023-10-16T14:11:04.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.853+0000 7f1c457fa640 1 -- 172.21.15.60:0/2001815158 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v133) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1c4001a070 con 0x7f1c480c5a80 2023-10-16T14:11:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.857+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1c3406b1c0 msgr2=0x7f1c3406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.857+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1c3406b1c0 0x7f1c3406d680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c38036000 tx=0x7f1c3802e000 comp rx=0 tx=0).stop 2023-10-16T14:11:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.857+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 msgr2=0x7f1c480c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.857+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 0x7f1c480c5ee0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f1c480c5a30 tx=0x7f1c4000d890 comp rx=0 tx=0).stop 2023-10-16T14:11:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 shutdown_connections 2023-10-16T14:11:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1c3406b1c0 0x7f1c3406d680 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c38036000 tx=0x7f1c3802e000 comp rx=0 tx=0).stop 2023-10-16T14:11:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c481565e0 0x7f1c48156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c480c5a80 0x7f1c480c5ee0 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 --2- 172.21.15.60:0/2001815158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c30003970 0x7f1c4815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 >> 172.21.15.60:0/2001815158 conn(0x7f1c480ce9b0 msgr2=0x7f1c480b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.858+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 shutdown_connections 2023-10-16T14:11:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:04.859+0000 7f1c4ed1d640 1 -- 172.21.15.60:0/2001815158 wait complete. 2023-10-16T14:11:04.885 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:11:04.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:11:04.985 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:11:05.076 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:11:05.077 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:11:05.077 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:05.134 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:11:05.135 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:05.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:05.396 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:05.393+0000 7f48c3a13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:11:05.420 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:05.417+0000 7f0611e11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:11:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.434+0000 7ff206f89640 1 Processor -- start 2023-10-16T14:11:05.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.434+0000 7ff206f89640 1 -- start start 2023-10-16T14:11:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff206f89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff20015ae80 0x7ff20015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff206f89640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff2000ce390 0x7ff200070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff206f89640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20015ed60 con 0x7ff20015aaa0 2023-10-16T14:11:05.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff206f89640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20006b810 con 0x7ff20015ae80 2023-10-16T14:11:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff206f89640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2000707e0 con 0x7ff2000ce390 2023-10-16T14:11:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff204d25640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff20015aaa0 0x7ff2000cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57020/0 (socket says 172.21.15.60:57020) 2023-10-16T14:11:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff1fffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff20015ae80 0x7ff20015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.436+0000 7ff204d25640 1 -- 172.21.15.60:0/2848959225 learned_addr learned my addr 172.21.15.60:0/2848959225 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.437+0000 7ff205526640 1 --2- 172.21.15.60:0/2848959225 >> v2:172.21.15.144:3300/0 conn(0x7ff2000ce390 0x7ff200070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.437+0000 7ff1ff7fe640 1 -- 172.21.15.60:0/2848959225 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994434954 0 0) 0x7ff20015ed60 con 0x7ff20015aaa0 2023-10-16T14:11:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.437+0000 7ff1fffff640 1 -- 172.21.15.60:0/2848959225 >> v2:172.21.15.144:3300/0 conn(0x7ff2000ce390 msgr2=0x7ff200070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.437+0000 7ff1fffff640 1 --2- 172.21.15.60:0/2848959225 >> v2:172.21.15.144:3300/0 conn(0x7ff2000ce390 0x7ff200070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.437+0000 7ff1fffff640 1 -- 172.21.15.60:0/2848959225 >> v1:172.21.15.60:6789/0 conn(0x7ff20015aaa0 legacy=0x7ff2000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.437+0000 7ff1fffff640 1 -- 172.21.15.60:0/2848959225 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20015ef90 con 0x7ff20015ae80 2023-10-16T14:11:05.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.439+0000 7ff1fffff640 1 --2- 172.21.15.60:0/2848959225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20015ae80 0x7ff20015e820 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7ff2000c2490 tx=0x7ff1ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86c52229d5a1f6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.439+0000 7ff1ff7fe640 1 -- 172.21.15.60:0/2848959225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec031830 con 0x7ff20015ae80 2023-10-16T14:11:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.439+0000 7ff1ff7fe640 1 -- 172.21.15.60:0/2848959225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ec031990 con 0x7ff20015ae80 2023-10-16T14:11:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.439+0000 7ff1ff7fe640 1 -- 172.21.15.60:0/2848959225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec031d20 con 0x7ff20015ae80 2023-10-16T14:11:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 -- 172.21.15.60:0/2848959225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20015ae80 msgr2=0x7ff20015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 --2- 172.21.15.60:0/2848959225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20015ae80 0x7ff20015e820 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7ff2000c2490 tx=0x7ff1ec02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 -- 172.21.15.60:0/2848959225 shutdown_connections 2023-10-16T14:11:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 --2- 172.21.15.60:0/2848959225 >> v2:172.21.15.144:3300/0 conn(0x7ff2000ce390 0x7ff200070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 --2- 172.21.15.60:0/2848959225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff20015ae80 0x7ff20015e820 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 -- 172.21.15.60:0/2848959225 >> 172.21.15.60:0/2848959225 conn(0x7ff2000cb410 msgr2=0x7ff2000cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 -- 172.21.15.60:0/2848959225 shutdown_connections 2023-10-16T14:11:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.440+0000 7ff206f89640 1 -- 172.21.15.60:0/2848959225 wait complete. 2023-10-16T14:11:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.442+0000 7ff206f89640 1 Processor -- start 2023-10-16T14:11:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.442+0000 7ff206f89640 1 -- start start 2023-10-16T14:11:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff206f89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 0x7ff20015de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff206f89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff20015aaa0 0x7ff2000c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff206f89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20015ae80 0x7ff200158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff206f89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20015e370 con 0x7ff2000ce390 2023-10-16T14:11:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff206f89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200147d00 con 0x7ff20015aaa0 2023-10-16T14:11:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff206f89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20015b440 con 0x7ff20015ae80 2023-10-16T14:11:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff204d25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 0x7ff20015de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff204d25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 0x7ff20015de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55488/0 (socket says 172.21.15.60:55488) 2023-10-16T14:11:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.443+0000 7ff204d25640 1 -- 172.21.15.60:0/1321618388 learned_addr learned my addr 172.21.15.60:0/1321618388 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff1fffff640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff20015aaa0 0x7ff2000c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff204d25640 1 -- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20015ae80 msgr2=0x7ff200158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff204d25640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20015ae80 0x7ff200158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff204d25640 1 -- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff20015aaa0 msgr2=0x7ff2000c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff204d25640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff20015aaa0 0x7ff2000c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff204d25640 1 -- 172.21.15.60:0/1321618388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec009400 con 0x7ff2000ce390 2023-10-16T14:11:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff204d25640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 0x7ff20015de30 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400c480 tx=0x7ff1f400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40074e0 con 0x7ff2000ce390 2023-10-16T14:11:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4007660 con 0x7ff2000ce390 2023-10-16T14:11:05.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4004a20 con 0x7ff2000ce390 2023-10-16T14:11:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2000c87b0 con 0x7ff2000ce390 2023-10-16T14:11:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.444+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff200147f50 con 0x7ff2000ce390 2023-10-16T14:11:05.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.446+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20004e160 con 0x7ff2000ce390 2023-10-16T14:11:05.454 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:05.449+0000 7f48c3a13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:11:05.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.446+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7ff1f4004b80 con 0x7ff2000ce390 2023-10-16T14:11:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.447+0000 7ff1fd7fa640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 0x7ff1e406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.448+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7ff1f4089110 con 0x7ff2000ce390 2023-10-16T14:11:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.448+0000 7ff1fffff640 1 -- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 msgr2=0x7ff1e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.448+0000 7ff1fffff640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 0x7ff1e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.456+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1f4053960 con 0x7ff2000ce390 2023-10-16T14:11:05.483 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:05.481+0000 7f0611e11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:11:05.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.649+0000 7ff1fffff640 1 -- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 msgr2=0x7ff1e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:05.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.649+0000 7ff1fffff640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 0x7ff1e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:11:05.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.652+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2000cdc60 con 0x7ff2000ce390 2023-10-16T14:11:05.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.655+0000 7ff1fd7fa640 1 -- 172.21.15.60:0/1321618388 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+20802 (secure 0 0 0) 0x7ff1f4056f80 con 0x7ff2000ce390 2023-10-16T14:11:05.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:05.659 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":236,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:10:58.183379+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:05.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.657+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 msgr2=0x7ff1e406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.657+0000 7ff206f89640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 0x7ff1e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.657+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 msgr2=0x7ff20015de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 0x7ff20015de30 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400c480 tx=0x7ff1f400c940 comp rx=0 tx=0).stop 2023-10-16T14:11:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 shutdown_connections 2023-10-16T14:11:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7ff1e406b140 0x7ff1e406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff20015ae80 0x7ff200158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff20015aaa0 0x7ff2000c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 --2- 172.21.15.60:0/1321618388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2000ce390 0x7ff20015de30 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 >> 172.21.15.60:0/1321618388 conn(0x7ff2000cb410 msgr2=0x7ff2000700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 shutdown_connections 2023-10-16T14:11:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.658+0000 7ff206f89640 1 -- 172.21.15.60:0/1321618388 wait complete. 2023-10-16T14:11:05.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:10:24.220406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:10:25.316288+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T14:10:38.740891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T14:10:39.464305+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:11:05.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:11:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.896+0000 7fc9ef59e640 1 Processor -- start 2023-10-16T14:11:05.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.897+0000 7fc9ef59e640 1 -- start start 2023-10-16T14:11:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.898+0000 7fc9ef59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b1ca0 0x7fc9f00d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.898+0000 7fc9ef59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc9f00cfc40 0x7fc9f00b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.898+0000 7fc9ef59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f00d0970 con 0x7fc9f00b2080 2023-10-16T14:11:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.898+0000 7fc9ef59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f00d0ea0 con 0x7fc9f00cfc40 2023-10-16T14:11:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.898+0000 7fc9ef59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00d0bc0 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9ee59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b1ca0 0x7fc9f00d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9eed9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc9f00cfc40 0x7fc9f00b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9ee59c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b1ca0 0x7fc9f00d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:60252/0 (socket says 172.21.15.60:60252) 2023-10-16T14:11:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9ee59c640 1 -- 172.21.15.60:0/3921102253 learned_addr learned my addr 172.21.15.60:0/3921102253 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:05.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9eed9d640 1 -- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b1ca0 msgr2=0x7fc9f00d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9eed9d640 1 --2- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b1ca0 0x7fc9f00d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9eed9d640 1 -- 172.21.15.60:0/3921102253 >> v1:172.21.15.144:6789/0 conn(0x7fc9f00b2080 legacy=0x7fc9f00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.899+0000 7fc9eed9d640 1 -- 172.21.15.60:0/3921102253 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f00cd940 con 0x7fc9f00cfc40 2023-10-16T14:11:05.905 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:05.901+0000 7f48c3a13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:11:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.901+0000 7fc9eed9d640 1 --2- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f00b24c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009d90 tx=0x7fc9e802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7c5bb142de70fa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.902+0000 7fc9edd9b640 1 -- 172.21.15.60:0/3921102253 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e802cce0 con 0x7fc9f00cfc40 2023-10-16T14:11:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.902+0000 7fc9edd9b640 1 -- 172.21.15.60:0/3921102253 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e802ce40 con 0x7fc9f00cfc40 2023-10-16T14:11:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.902+0000 7fc9edd9b640 1 -- 172.21.15.60:0/3921102253 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8034660 con 0x7fc9f00cfc40 2023-10-16T14:11:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.902+0000 7fc9ef59e640 1 -- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 msgr2=0x7fc9f00b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.902+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f00b24c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009d90 tx=0x7fc9e802bf30 comp rx=0 tx=0).stop 2023-10-16T14:11:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.903+0000 7fc9ef59e640 1 -- 172.21.15.60:0/3921102253 shutdown_connections 2023-10-16T14:11:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.903+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f00b24c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.903+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/3921102253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b1ca0 0x7fc9f00d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.903+0000 7fc9ef59e640 1 -- 172.21.15.60:0/3921102253 >> 172.21.15.60:0/3921102253 conn(0x7fc9f00d2b90 msgr2=0x7fc9f00d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.903+0000 7fc9ef59e640 1 -- 172.21.15.60:0/3921102253 shutdown_connections 2023-10-16T14:11:05.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.903+0000 7fc9ef59e640 1 -- 172.21.15.60:0/3921102253 wait complete. 2023-10-16T14:11:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.905+0000 7fc9ef59e640 1 Processor -- start 2023-10-16T14:11:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.905+0000 7fc9ef59e640 1 -- start start 2023-10-16T14:11:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.906+0000 7fc9ef59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 0x7fc9f00b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.906+0000 7fc9ef59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b2080 0x7fc9f0151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.906+0000 7fc9ef59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f01f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.906+0000 7fc9ef59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00c6940 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.906+0000 7fc9ef59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00c9680 con 0x7fc9f00cfc40 2023-10-16T14:11:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.906+0000 7fc9ef59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00c6b50 con 0x7fc9f00b2080 2023-10-16T14:11:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 0x7fc9f00b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 0x7fc9f00b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55502/0 (socket says 172.21.15.60:55502) 2023-10-16T14:11:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 -- 172.21.15.60:0/2513569110 learned_addr learned my addr 172.21.15.60:0/2513569110 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9e7fff640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b2080 0x7fc9f0151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9eed9d640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f01f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 -- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b2080 msgr2=0x7fc9f0151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b2080 0x7fc9f0151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 -- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 msgr2=0x7fc9f01f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f01f5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.907+0000 7fc9ee59c640 1 -- 172.21.15.60:0/2513569110 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8009a70 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.908+0000 7fc9ee59c640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 0x7fc9f00b43f0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000e840 tx=0x7fc9e000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.908+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000c8e0 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.908+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e000ca40 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.908+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000cdf0 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.908+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f00c6e30 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.908+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f0152410 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.909+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f004e9b0 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.910+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc9e0013470 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.911+0000 7fc9e6ffd640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 0x7fc9d006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.911+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fc9e008a070 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.912+0000 7fc9e7fff640 1 -- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 msgr2=0x7fc9d006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:05.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.912+0000 7fc9e7fff640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 0x7fc9d006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:05.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:05.916+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9e0054890 con 0x7fc9f00b1ca0 2023-10-16T14:11:05.958 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:05.954+0000 7f48c3a13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:11:05.970 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:05.967+0000 7f0611e11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:11:06.007 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.003+0000 7f48c3a13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:11:06.026 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.023+0000 7f0611e11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:11:06.068 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.064+0000 7f48c3a13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:11:06.079 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.076+0000 7f0611e11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:11:06.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.110+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9f0073a40 con 0x7fc9f00b1ca0 2023-10-16T14:11:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.111+0000 7fc9e6ffd640 1 -- 172.21.15.60:0/2513569110 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v236) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9e0057eb0 con 0x7fc9f00b1ca0 2023-10-16T14:11:06.114 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:11:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.113+0000 7fc9e7fff640 1 -- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 msgr2=0x7fc9d006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.113+0000 7fc9e7fff640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 0x7fc9d006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:11:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 msgr2=0x7fc9d006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 0x7fc9d006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 msgr2=0x7fc9f00b43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 0x7fc9f00b43f0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000e840 tx=0x7fc9e000ed00 comp rx=0 tx=0).stop 2023-10-16T14:11:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 shutdown_connections 2023-10-16T14:11:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fc9d006b140 0x7fc9d006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc9f00b2080 0x7fc9f0151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9f00cfc40 0x7fc9f01f5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 --2- 172.21.15.60:0/2513569110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc9f00b1ca0 0x7fc9f00b43f0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 >> 172.21.15.60:0/2513569110 conn(0x7fc9f00d2b90 msgr2=0x7fc9f00d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 shutdown_connections 2023-10-16T14:11:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.114+0000 7fc9ef59e640 1 -- 172.21.15.60:0/2513569110 wait complete. 2023-10-16T14:11:06.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.113+0000 7f48c3a13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:11:06.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:11:06.144 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.142+0000 7f0611e11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:11:06.197 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.195+0000 7f0611e11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:11:06.228 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.224+0000 7f48c3a13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:11:06.289 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.285+0000 7f48c3a13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:11:06.316 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.313+0000 7f0611e11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:11:06.339 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.335+0000 7f48c3a13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:11:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.352+0000 7f6f92475640 1 Processor -- start 2023-10-16T14:11:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.352+0000 7f6f92475640 1 -- start start 2023-10-16T14:11:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.353+0000 7f6f92475640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f8c0cf5f0 0x7f6f8c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.353+0000 7f6f92475640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c070c10 0x7f6f8c0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.353+0000 7f6f92475640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c161740 con 0x7f6f8c070830 2023-10-16T14:11:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f92475640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8c0cdbd0 con 0x7f6f8c0cf5f0 2023-10-16T14:11:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f92475640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c06b0f0 con 0x7f6f8c070c10 2023-10-16T14:11:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f8c0cf5f0 0x7f6f8c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f8c0cf5f0 0x7f6f8c06d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55518/0 (socket says 172.21.15.60:55518) 2023-10-16T14:11:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 -- 172.21.15.60:0/2396863853 learned_addr learned my addr 172.21.15.60:0/2396863853 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f90a12640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c070c10 0x7f6f8c0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 -- 172.21.15.60:0/2396863853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c070c10 msgr2=0x7f6f8c0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c070c10 0x7f6f8c0cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 -- 172.21.15.60:0/2396863853 >> v1:172.21.15.144:6789/0 conn(0x7f6f8c070830 legacy=0x7f6f8c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.354+0000 7f6f83fff640 1 -- 172.21.15.60:0/2396863853 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c1619b0 con 0x7f6f8c0cf5f0 2023-10-16T14:11:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.355+0000 7f6f90a12640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c070c10 0x7f6f8c0cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.356+0000 7f6f83fff640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c0cf5f0 0x7f6f8c06d3c0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009870 tx=0x7f6f7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a744949a720775e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f8b7fe640 1 -- 172.21.15.60:0/2396863853 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f740317c0 con 0x7f6f8c0cf5f0 2023-10-16T14:11:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f8b7fe640 1 -- 172.21.15.60:0/2396863853 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f74031920 con 0x7f6f8c0cf5f0 2023-10-16T14:11:06.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f8b7fe640 1 -- 172.21.15.60:0/2396863853 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74031cb0 con 0x7f6f8c0cf5f0 2023-10-16T14:11:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f92475640 1 -- 172.21.15.60:0/2396863853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c0cf5f0 msgr2=0x7f6f8c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f92475640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c0cf5f0 0x7f6f8c06d3c0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009870 tx=0x7f6f7402b860 comp rx=0 tx=0).stop 2023-10-16T14:11:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f92475640 1 -- 172.21.15.60:0/2396863853 shutdown_connections 2023-10-16T14:11:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f92475640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c070c10 0x7f6f8c0cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f92475640 1 --2- 172.21.15.60:0/2396863853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c0cf5f0 0x7f6f8c06d3c0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.357+0000 7f6f92475640 1 -- 172.21.15.60:0/2396863853 >> 172.21.15.60:0/2396863853 conn(0x7f6f8c0cfac0 msgr2=0x7f6f8c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.358+0000 7f6f92475640 1 -- 172.21.15.60:0/2396863853 shutdown_connections 2023-10-16T14:11:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.358+0000 7f6f92475640 1 -- 172.21.15.60:0/2396863853 wait complete. 2023-10-16T14:11:06.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.359+0000 7f6f92475640 1 Processor -- start 2023-10-16T14:11:06.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 -- start start 2023-10-16T14:11:06.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f8c070830 0x7f6f8c0c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 0x7f6f8c0c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c0cf5f0 0x7f6f8c0b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0af620 con 0x7f6f8c070c10 2023-10-16T14:11:06.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0b12b0 con 0x7f6f8c070830 2023-10-16T14:11:06.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.360+0000 7f6f92475640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0c4940 con 0x7f6f8c0cf5f0 2023-10-16T14:11:06.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 0x7f6f8c0c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 0x7f6f8c0c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55530/0 (socket says 172.21.15.60:55530) 2023-10-16T14:11:06.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 -- 172.21.15.60:0/1319535303 learned_addr learned my addr 172.21.15.60:0/1319535303 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:06.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f90a12640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c0cf5f0 0x7f6f8c0b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f8bfff640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f8c070830 0x7f6f8c0c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 -- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c0cf5f0 msgr2=0x7f6f8c0b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c0cf5f0 0x7f6f8c0b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 -- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f8c070830 msgr2=0x7f6f8c0c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f8c070830 0x7f6f8c0c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f83fff640 1 -- 172.21.15.60:0/1319535303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740094d0 con 0x7f6f8c070c10 2023-10-16T14:11:06.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.361+0000 7f6f90a12640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c0cf5f0 0x7f6f8c0b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:06.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.362+0000 7f6f83fff640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 0x7f6f8c0c42b0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f6f7402b8a0 tx=0x7f6f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:06.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.362+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74032860 con 0x7f6f8c070c10 2023-10-16T14:11:06.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.362+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f740329e0 con 0x7f6f8c070c10 2023-10-16T14:11:06.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.362+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74032d90 con 0x7f6f8c070c10 2023-10-16T14:11:06.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.362+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c0c4c20 con 0x7f6f8c070c10 2023-10-16T14:11:06.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.362+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c0c51e0 con 0x7f6f8c070c10 2023-10-16T14:11:06.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.363+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c0af760 con 0x7f6f8c070c10 2023-10-16T14:11:06.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.364+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f6f7404e450 con 0x7f6f8c070c10 2023-10-16T14:11:06.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.365+0000 7f6f897fa640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 0x7f6f6c06d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.366+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f6f740ac0a0 con 0x7f6f8c070c10 2023-10-16T14:11:06.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.366+0000 7f6f8bfff640 1 -- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 msgr2=0x7f6f6c06d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:06.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.366+0000 7f6f8bfff640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 0x7f6f6c06d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:06.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.370+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f8c0af760 con 0x7f6f8c070c10 2023-10-16T14:11:06.386 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.383+0000 7f0611e11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:11:06.396 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.393+0000 7f48c3a13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:11:06.439 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.436+0000 7f0611e11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:11:06.446 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.442+0000 7f48c3a13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:11:06.499 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.497+0000 7f0611e11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:11:06.553 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.550+0000 7f0611e11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:11:06.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.564+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f8c0af760 con 0x7f6f8c070c10 2023-10-16T14:11:06.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.565+0000 7f6f897fa640 1 -- 172.21.15.60:0/1319535303 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v236) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f8c0af760 con 0x7f6f8c070c10 2023-10-16T14:11:06.568 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:11:06.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.567+0000 7f6f8bfff640 1 -- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 msgr2=0x7f6f6c06d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:06.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.567+0000 7f6f8bfff640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 0x7f6f6c06d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:11:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 msgr2=0x7f6f6c06d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 0x7f6f6c06d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 msgr2=0x7f6f8c0c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 0x7f6f8c0c42b0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f6f7402b8a0 tx=0x7f6f740046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 shutdown_connections 2023-10-16T14:11:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f6f6c06b170 0x7f6f6c06d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f8c0cf5f0 0x7f6f8c0b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8c070c10 0x7f6f8c0c42b0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 --2- 172.21.15.60:0/1319535303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f8c070830 0x7f6f8c0c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 >> 172.21.15.60:0/1319535303 conn(0x7f6f8c0cfac0 msgr2=0x7f6f8c15f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 shutdown_connections 2023-10-16T14:11:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.568+0000 7f6f92475640 1 -- 172.21.15.60:0/1319535303 wait complete. 2023-10-16T14:11:06.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:11:06.829 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.825+0000 7f48c3a13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:11:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.857+0000 7f01c5426640 1 Processor -- start 2023-10-16T14:11:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.857+0000 7f01c5426640 1 -- start start 2023-10-16T14:11:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.858+0000 7f01c5426640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f01c015ae10 0x7f01c00ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.858+0000 7f01c5426640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f01c00cf330 0x7f01c00caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01c5426640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c015b1f0 con 0x7f01c006d570 2023-10-16T14:11:06.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01c5426640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c006f650 con 0x7f01c015ae10 2023-10-16T14:11:06.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01c5426640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c006bb70 con 0x7f01c00cf330 2023-10-16T14:11:06.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f01c015ae10 0x7f01c00ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f01c015ae10 0x7f01c00ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55544/0 (socket says 172.21.15.60:55544) 2023-10-16T14:11:06.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 -- 172.21.15.60:0/3936011997 learned_addr learned my addr 172.21.15.60:0/3936011997 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:06.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01bf7fe640 1 --2- 172.21.15.60:0/3936011997 >> v2:172.21.15.144:3300/0 conn(0x7f01c00cf330 0x7f01c00caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 -- 172.21.15.60:0/3936011997 >> v2:172.21.15.144:3300/0 conn(0x7f01c00cf330 msgr2=0x7f01c00caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 --2- 172.21.15.60:0/3936011997 >> v2:172.21.15.144:3300/0 conn(0x7f01c00cf330 0x7f01c00caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 -- 172.21.15.60:0/3936011997 >> v1:172.21.15.144:6789/0 conn(0x7f01c006d570 legacy=0x7f01c006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.859+0000 7f01beffd640 1 -- 172.21.15.60:0/3936011997 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00b4190 con 0x7f01c015ae10 2023-10-16T14:11:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.861+0000 7f01beffd640 1 --2- 172.21.15.60:0/3936011997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c015ae10 0x7f01c00ca590 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f01c00ca210 tx=0x7f01b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d23627d07a2dbdc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.861+0000 7f01bdffb640 1 -- 172.21.15.60:0/3936011997 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0031870 con 0x7f01c015ae10 2023-10-16T14:11:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.861+0000 7f01bdffb640 1 -- 172.21.15.60:0/3936011997 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b00319d0 con 0x7f01c015ae10 2023-10-16T14:11:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.861+0000 7f01bdffb640 1 -- 172.21.15.60:0/3936011997 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0031d60 con 0x7f01c015ae10 2023-10-16T14:11:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 -- 172.21.15.60:0/3936011997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c015ae10 msgr2=0x7f01c00ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 --2- 172.21.15.60:0/3936011997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c015ae10 0x7f01c00ca590 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f01c00ca210 tx=0x7f01b002b910 comp rx=0 tx=0).stop 2023-10-16T14:11:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 -- 172.21.15.60:0/3936011997 shutdown_connections 2023-10-16T14:11:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 --2- 172.21.15.60:0/3936011997 >> v2:172.21.15.144:3300/0 conn(0x7f01c00cf330 0x7f01c00caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 --2- 172.21.15.60:0/3936011997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c015ae10 0x7f01c00ca590 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 -- 172.21.15.60:0/3936011997 >> 172.21.15.60:0/3936011997 conn(0x7f01c00ce840 msgr2=0x7f01c00cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:06.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 -- 172.21.15.60:0/3936011997 shutdown_connections 2023-10-16T14:11:06.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.862+0000 7f01c5426640 1 -- 172.21.15.60:0/3936011997 wait complete. 2023-10-16T14:11:06.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.864+0000 7f01c5426640 1 Processor -- start 2023-10-16T14:11:06.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.864+0000 7f01c5426640 1 -- start start 2023-10-16T14:11:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01c5426640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01c006d570 0x7f01c01f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01c5426640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 0x7f01c00c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01c5426640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01c015ae10 0x7f01c00ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01c5426640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c5a40 con 0x7f01c00cf330 2023-10-16T14:11:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01c5426640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c5d20 con 0x7f01c006d570 2023-10-16T14:11:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01c5426640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c6000 con 0x7f01c015ae10 2023-10-16T14:11:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01be7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 0x7f01c00c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01be7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 0x7f01c00c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55558/0 (socket says 172.21.15.60:55558) 2023-10-16T14:11:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01be7fc640 1 -- 172.21.15.60:0/288363820 learned_addr learned my addr 172.21.15.60:0/288363820 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.865+0000 7f01beffd640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01c006d570 0x7f01c01f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01bf7fe640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01c015ae10 0x7f01c00ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01be7fc640 1 -- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01c015ae10 msgr2=0x7f01c00ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01be7fc640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01c015ae10 0x7f01c00ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01be7fc640 1 -- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01c006d570 msgr2=0x7f01c01f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01be7fc640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01c006d570 0x7f01c01f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01be7fc640 1 -- 172.21.15.60:0/288363820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b00094d0 con 0x7f01c00cf330 2023-10-16T14:11:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01be7fc640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 0x7f01c00c7fa0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f01b400b3f0 tx=0x7f01b400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4011830 con 0x7f01c00cf330 2023-10-16T14:11:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b4011990 con 0x7f01c00cf330 2023-10-16T14:11:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c00c62e0 con 0x7f01c00cf330 2023-10-16T14:11:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4011d40 con 0x7f01c00cf330 2023-10-16T14:11:06.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.866+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c00c6ae0 con 0x7f01c00cf330 2023-10-16T14:11:06.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.868+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c00c6800 con 0x7f01c00cf330 2023-10-16T14:11:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.869+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f01b401a400 con 0x7f01c00cf330 2023-10-16T14:11:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.869+0000 7f01a77fe640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 0x7f01ac06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.870+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f01b4089ce0 con 0x7f01c00cf330 2023-10-16T14:11:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.870+0000 7f01beffd640 1 -- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 msgr2=0x7f01ac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:06.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.870+0000 7f01beffd640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 0x7f01ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:06.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:06.874+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f01c00c6800 con 0x7f01c00cf330 2023-10-16T14:11:06.974 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:06.972+0000 7f0611e11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:11:06.995 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:06.992+0000 7f48c3a13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:11:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.069+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f01c00c6800 con 0x7f01c00cf330 2023-10-16T14:11:07.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.070+0000 7f01a77fe640 1 -- 172.21.15.60:0/288363820 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v236) v1 ==== 100+0+11 (secure 0 0 0) 0x7f01c00c6800 con 0x7f01c00cf330 2023-10-16T14:11:07.073 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:11:07.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.071+0000 7f01beffd640 1 -- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 msgr2=0x7f01ac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:07.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.071+0000 7f01beffd640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 0x7f01ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:11:07.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 msgr2=0x7f01ac06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:07.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 0x7f01ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 msgr2=0x7f01c00c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 0x7f01c00c7fa0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f01b400b3f0 tx=0x7f01b400b8b0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 shutdown_connections 2023-10-16T14:11:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f01ac06b140 0x7f01ac06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f01c015ae10 0x7f01c00ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01c00cf330 0x7f01c00c7fa0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 --2- 172.21.15.60:0/288363820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01c006d570 0x7f01c01f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 >> 172.21.15.60:0/288363820 conn(0x7f01c00ce840 msgr2=0x7f01c015d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 shutdown_connections 2023-10-16T14:11:07.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.073+0000 7f01c5426640 1 -- 172.21.15.60:0/288363820 wait complete. 2023-10-16T14:11:07.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T14:11:07.104 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.100+0000 7f48c3a13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:11:07.149 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.147+0000 7f0611e11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:11:07.154 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.150+0000 7f48c3a13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:11:07.262 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.258+0000 7f48c3a13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:11:07.266 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.263+0000 7f0611e11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:11:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.301+0000 7f358b59e640 1 Processor -- start 2023-10-16T14:11:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.301+0000 7f358b59e640 1 -- start start 2023-10-16T14:11:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f358b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f358c0cded0 0x7f358c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f358b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c15d150 con 0x7f358c0b8480 2023-10-16T14:11:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f358b59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c0d1180 con 0x7f358c15cd70 2023-10-16T14:11:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f358b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358c0d1460 con 0x7f358c0cded0 2023-10-16T14:11:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f358ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f358c0cded0 0x7f358c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f3583fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f358c0b8480 0x7f358c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57030/0 (socket says 172.21.15.60:57030) 2023-10-16T14:11:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.303+0000 7f3583fff640 1 -- 172.21.15.60:0/1917977738 learned_addr learned my addr 172.21.15.60:0/1917977738 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.304+0000 7f358ad9d640 1 -- 172.21.15.60:0/1917977738 >> v1:172.21.15.144:6789/0 conn(0x7f358c15cd70 legacy=0x7f358c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.304+0000 7f358ad9d640 1 -- 172.21.15.60:0/1917977738 >> v1:172.21.15.60:6789/0 conn(0x7f358c0b8480 legacy=0x7f358c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.304+0000 7f358ad9d640 1 -- 172.21.15.60:0/1917977738 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0d1740 con 0x7f358c0cded0 2023-10-16T14:11:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.305+0000 7f358ad9d640 1 --2- 172.21.15.60:0/1917977738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c0cded0 0x7f358c0b5c90 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f3584009d90 tx=0x7f358402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1e7dfe854d1a5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f3589d9b640 1 -- 172.21.15.60:0/1917977738 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358402ccf0 con 0x7f358c0cded0 2023-10-16T14:11:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f3589d9b640 1 -- 172.21.15.60:0/1917977738 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358402ce50 con 0x7f358c0cded0 2023-10-16T14:11:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f3589d9b640 1 -- 172.21.15.60:0/1917977738 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584034660 con 0x7f358c0cded0 2023-10-16T14:11:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f358b59e640 1 -- 172.21.15.60:0/1917977738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c0cded0 msgr2=0x7f358c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f358b59e640 1 --2- 172.21.15.60:0/1917977738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c0cded0 0x7f358c0b5c90 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f3584009d90 tx=0x7f358402bf40 comp rx=0 tx=0).stop 2023-10-16T14:11:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f358b59e640 1 -- 172.21.15.60:0/1917977738 shutdown_connections 2023-10-16T14:11:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f358b59e640 1 --2- 172.21.15.60:0/1917977738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c0cded0 0x7f358c0b5c90 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f358b59e640 1 -- 172.21.15.60:0/1917977738 >> 172.21.15.60:0/1917977738 conn(0x7f358c0cf570 msgr2=0x7f358c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.306+0000 7f358b59e640 1 -- 172.21.15.60:0/1917977738 shutdown_connections 2023-10-16T14:11:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.307+0000 7f358b59e640 1 -- 172.21.15.60:0/1917977738 wait complete. 2023-10-16T14:11:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.308+0000 7f358b59e640 1 Processor -- start 2023-10-16T14:11:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.308+0000 7f358b59e640 1 -- start start 2023-10-16T14:11:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.309+0000 7f358b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f358c0b8480 0x7f358c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.309+0000 7f358b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f358c0cded0 0x7f358c1f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.309+0000 7f358b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 0x7f358c0aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.309+0000 7f358b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c15f3e0 con 0x7f358c15cd70 2023-10-16T14:11:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.309+0000 7f358b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c15f6c0 con 0x7f358c0b8480 2023-10-16T14:11:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.309+0000 7f358b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c15f9a0 con 0x7f358c0cded0 2023-10-16T14:11:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 0x7f358c0aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 0x7f358c0aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55564/0 (socket says 172.21.15.60:55564) 2023-10-16T14:11:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 -- 172.21.15.60:0/323752687 learned_addr learned my addr 172.21.15.60:0/323752687 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:07.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358a59c640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f358c0b8480 0x7f358c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f3583fff640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f358c0cded0 0x7f358c1f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 -- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f358c0cded0 msgr2=0x7f358c1f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f358c0cded0 0x7f358c1f2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 -- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f358c0b8480 msgr2=0x7f358c15ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f358c0b8480 0x7f358c15ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.310+0000 7f358ad9d640 1 -- 172.21.15.60:0/323752687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584009a70 con 0x7f358c15cd70 2023-10-16T14:11:07.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.311+0000 7f358ad9d640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 0x7f358c0aa0e0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f3584037040 tx=0x7f3584004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:07.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.311+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358403d600 con 0x7f358c15cd70 2023-10-16T14:11:07.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.311+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358403d780 con 0x7f358c15cd70 2023-10-16T14:11:07.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.311+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c15fc80 con 0x7f358c15cd70 2023-10-16T14:11:07.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.311+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358403db30 con 0x7f358c15cd70 2023-10-16T14:11:07.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.311+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358c1f3390 con 0x7f358c15cd70 2023-10-16T14:11:07.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.312+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358c1601a0 con 0x7f358c15cd70 2023-10-16T14:11:07.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.313+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f358403b040 con 0x7f358c15cd70 2023-10-16T14:11:07.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.314+0000 7f3582ffd640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f356c06b140 0x7f356c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.314+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f35840ac520 con 0x7f358c15cd70 2023-10-16T14:11:07.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.314+0000 7f358a59c640 1 -- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f356c06b140 msgr2=0x7f356c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:07.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.314+0000 7f358a59c640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f356c06b140 0x7f356c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:07.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.318+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f358c1601a0 con 0x7f358c15cd70 2023-10-16T14:11:07.322 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.319+0000 7f0611e11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:11:07.369 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.365+0000 7f48c3a13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:11:07.437 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.434+0000 7f0611e11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:11:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.512+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f358c1601a0 con 0x7f358c15cd70 2023-10-16T14:11:07.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.512+0000 7f3582ffd640 1 -- 172.21.15.60:0/323752687 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v236) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3584039020 con 0x7f358c15cd70 2023-10-16T14:11:07.515 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:11:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f356c06b140 msgr2=0x7f356c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T14:11:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f356c06b140 0x7f356c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 msgr2=0x7f358c0aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 0x7f358c0aa0e0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f3584037040 tx=0x7f3584004a40 comp rx=0 tx=0).stop 2023-10-16T14:11:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 shutdown_connections 2023-10-16T14:11:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f356c06b140 0x7f356c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f358c0cded0 0x7f358c1f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f358c15cd70 0x7f358c0aa0e0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 --2- 172.21.15.60:0/323752687 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f358c0b8480 0x7f358c15ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 >> 172.21.15.60:0/323752687 conn(0x7f358c0cf570 msgr2=0x7f358c0b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 shutdown_connections 2023-10-16T14:11:07.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.515+0000 7f358b59e640 1 -- 172.21.15.60:0/323752687 wait complete. 2023-10-16T14:11:07.526 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.522+0000 7f48c3a13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:11:07.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T14:11:07.553 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.551+0000 7f0611e11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:11:07.574 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.571+0000 7f48c3a13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:11:07.693 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:07.688+0000 7f48c3a13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:11:07.723 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.720+0000 7f0611e11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:11:07.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.742+0000 7f1f655c4640 1 Processor -- start 2023-10-16T14:11:07.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.742+0000 7f1f655c4640 1 -- start start 2023-10-16T14:11:07.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f655c4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f6015cd70 0x7f1f600d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f655c4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1f600b8480 0x7f1f600cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f655c4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f600cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f655c4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6015d150 con 0x7f1f6015cd70 2023-10-16T14:11:07.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f655c4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f600d1180 con 0x7f1f600b8480 2023-10-16T14:11:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f655c4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600d1460 con 0x7f1f600d0be0 2023-10-16T14:11:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f5ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f6015cd70 0x7f1f600d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f5ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f6015cd70 0x7f1f600d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55570/0 (socket says 172.21.15.60:55570) 2023-10-16T14:11:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f5ffff640 1 -- 172.21.15.60:0/2335652207 learned_addr learned my addr 172.21.15.60:0/2335652207 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:07.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.744+0000 7f1f5f7fe640 1 --2- 172.21.15.60:0/2335652207 >> v2:172.21.15.144:3300/0 conn(0x7f1f600b8480 0x7f1f600cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f64dc3640 1 --2- 172.21.15.60:0/2335652207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f600cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f5ffff640 1 -- 172.21.15.60:0/2335652207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 msgr2=0x7f1f600cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f5ffff640 1 --2- 172.21.15.60:0/2335652207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f600cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f5ffff640 1 -- 172.21.15.60:0/2335652207 >> v2:172.21.15.144:3300/0 conn(0x7f1f600b8480 msgr2=0x7f1f600cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:07.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f5ffff640 1 --2- 172.21.15.60:0/2335652207 >> v2:172.21.15.144:3300/0 conn(0x7f1f600b8480 0x7f1f600cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f5ffff640 1 -- 172.21.15.60:0/2335652207 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600d1740 con 0x7f1f6015cd70 2023-10-16T14:11:07.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.745+0000 7f1f5f7fe640 1 --2- 172.21.15.60:0/2335652207 >> v2:172.21.15.144:3300/0 conn(0x7f1f600b8480 0x7f1f600cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:07.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.746+0000 7f1f5ffff640 1 --2- 172.21.15.60:0/2335652207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f6015cd70 0x7f1f600d2220 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f1f60066a00 tx=0x7f1f5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f13978d02385de1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:07.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.747+0000 7f1f5effd640 1 -- 172.21.15.60:0/2335652207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50031870 con 0x7f1f6015cd70 2023-10-16T14:11:07.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.747+0000 7f1f5effd640 1 -- 172.21.15.60:0/2335652207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f500319d0 con 0x7f1f6015cd70 2023-10-16T14:11:07.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.747+0000 7f1f5effd640 1 -- 172.21.15.60:0/2335652207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50031d60 con 0x7f1f6015cd70 2023-10-16T14:11:07.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 -- 172.21.15.60:0/2335652207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f6015cd70 msgr2=0x7f1f600d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 --2- 172.21.15.60:0/2335652207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f6015cd70 0x7f1f600d2220 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f1f60066a00 tx=0x7f1f5002b910 comp rx=0 tx=0).stop 2023-10-16T14:11:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 -- 172.21.15.60:0/2335652207 shutdown_connections 2023-10-16T14:11:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 --2- 172.21.15.60:0/2335652207 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f600cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 --2- 172.21.15.60:0/2335652207 >> v2:172.21.15.144:3300/0 conn(0x7f1f600b8480 0x7f1f600cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 --2- 172.21.15.60:0/2335652207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f6015cd70 0x7f1f600d2220 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 -- 172.21.15.60:0/2335652207 >> 172.21.15.60:0/2335652207 conn(0x7f1f600cf570 msgr2=0x7f1f600cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:07.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 -- 172.21.15.60:0/2335652207 shutdown_connections 2023-10-16T14:11:07.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.748+0000 7f1f655c4640 1 -- 172.21.15.60:0/2335652207 wait complete. 2023-10-16T14:11:07.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.750+0000 7f1f655c4640 1 Processor -- start 2023-10-16T14:11:07.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.750+0000 7f1f655c4640 1 -- start start 2023-10-16T14:11:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f655c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 0x7f1f60150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f655c4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f6015fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f655c4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f6015cd70 0x7f1f6015dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f655c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6015e7b0 con 0x7f1f600b8480 2023-10-16T14:11:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f655c4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6015ea90 con 0x7f1f6015cd70 2023-10-16T14:11:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f655c4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6015ed70 con 0x7f1f600d0be0 2023-10-16T14:11:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f5ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 0x7f1f60150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f5ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 0x7f1f60150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55586/0 (socket says 172.21.15.60:55586) 2023-10-16T14:11:07.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f5ffff640 1 -- 172.21.15.60:0/902554490 learned_addr learned my addr 172.21.15.60:0/902554490 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f64dc3640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f6015cd70 0x7f1f6015dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.751+0000 7f1f5f7fe640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f6015fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5ffff640 1 -- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 msgr2=0x7f1f6015fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5ffff640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f6015fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5ffff640 1 -- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f6015cd70 msgr2=0x7f1f6015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5ffff640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f6015cd70 0x7f1f6015dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5ffff640 1 -- 172.21.15.60:0/902554490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f500094d0 con 0x7f1f600b8480 2023-10-16T14:11:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f64dc3640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f6015cd70 0x7f1f6015dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5ffff640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 0x7f1f60150210 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f1f5002b950 tx=0x7f1f500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:07.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5003ad30 con 0x7f1f600b8480 2023-10-16T14:11:07.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f500443f0 con 0x7f1f600b8480 2023-10-16T14:11:07.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f500447a0 con 0x7f1f600b8480 2023-10-16T14:11:07.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.752+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6015f050 con 0x7f1f600b8480 2023-10-16T14:11:07.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.753+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6015f610 con 0x7f1f600b8480 2023-10-16T14:11:07.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.754+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6015f800 con 0x7f1f600b8480 2023-10-16T14:11:07.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.756+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f1f5004d450 con 0x7f1f600b8480 2023-10-16T14:11:07.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.757+0000 7f1f5cff9640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 0x7f1f4c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:07.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.757+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f1f5002d040 con 0x7f1f600b8480 2023-10-16T14:11:07.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.757+0000 7f1f5f7fe640 1 -- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 msgr2=0x7f1f4c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:07.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.757+0000 7f1f5f7fe640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 0x7f1f4c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:07.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.761+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f50076de0 con 0x7f1f600b8480 2023-10-16T14:11:07.779 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.776+0000 7f0611e11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:11:07.908 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:07.905+0000 7f0611e11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:11:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.958+0000 7f1f5f7fe640 1 -- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 msgr2=0x7f1f4c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:07.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.958+0000 7f1f5f7fe640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 0x7f1f4c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:11:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.964+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f1f6004e9b0 con 0x7f1f600b8480 2023-10-16T14:11:07.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.964+0000 7f1f5cff9640 1 -- 172.21.15.60:0/902554490 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v236) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1f500350a0 con 0x7f1f600b8480 2023-10-16T14:11:07.967 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:11:07.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 msgr2=0x7f1f4c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:07.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 0x7f1f4c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 msgr2=0x7f1f60150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:07.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 0x7f1f60150210 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f1f5002b950 tx=0x7f1f500046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 shutdown_connections 2023-10-16T14:11:07.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f1f4c06b140 0x7f1f4c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f600d0be0 0x7f1f6015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f6015cd70 0x7f1f6015dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 --2- 172.21.15.60:0/902554490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f600b8480 0x7f1f60150210 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:07.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 >> 172.21.15.60:0/902554490 conn(0x7f1f600cf570 msgr2=0x7f1f600ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 shutdown_connections 2023-10-16T14:11:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:07.967+0000 7f1f655c4640 1 -- 172.21.15.60:0/902554490 wait complete. 2023-10-16T14:11:07.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:11:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.231+0000 7fbd6e95a640 1 Processor -- start 2023-10-16T14:11:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.231+0000 7fbd6e95a640 1 -- start start 2023-10-16T14:11:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd6e95a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6806a4b0 0x7fbd680c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd6e95a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd680ac890 con 0x7fbd680ad350 2023-10-16T14:11:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd6e95a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd680ad7a0 con 0x7fbd6815d150 2023-10-16T14:11:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd6e95a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6815d530 con 0x7fbd6806a4b0 2023-10-16T14:11:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd5ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbd680ad350 0x7fbd68069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57042/0 (socket says 172.21.15.60:57042) 2023-10-16T14:11:08.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd5ffff640 1 -- 172.21.15.60:0/4158548189 learned_addr learned my addr 172.21.15.60:0/4158548189 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:08.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.233+0000 7fbd6cef7640 1 --2- 172.21.15.60:0/4158548189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6806a4b0 0x7fbd680c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.234+0000 7fbd677fe640 1 -- 172.21.15.60:0/4158548189 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276943185 0 0) 0x7fbd680ac890 con 0x7fbd680ad350 2023-10-16T14:11:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.234+0000 7fbd677fe640 1 -- 172.21.15.60:0/4158548189 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd54003520 con 0x7fbd680ad350 2023-10-16T14:11:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.234+0000 7fbd6cef7640 1 -- 172.21.15.60:0/4158548189 >> v1:172.21.15.144:6789/0 conn(0x7fbd6815d150 legacy=0x7fbd68160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.234+0000 7fbd677fe640 1 -- 172.21.15.60:0/4158548189 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2427614269 0 0) 0x7fbd54003520 con 0x7fbd680ad350 2023-10-16T14:11:08.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.234+0000 7fbd6cef7640 1 -- 172.21.15.60:0/4158548189 >> v1:172.21.15.60:6789/0 conn(0x7fbd680ad350 legacy=0x7fbd68069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.234+0000 7fbd6cef7640 1 -- 172.21.15.60:0/4158548189 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd68161650 con 0x7fbd6806a4b0 2023-10-16T14:11:08.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.235+0000 7fbd6cef7640 1 --2- 172.21.15.60:0/4158548189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6806a4b0 0x7fbd680c9000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009dc0 tx=0x7fbd6002bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44625c7319bedefc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd677fe640 1 -- 172.21.15.60:0/4158548189 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50001670 con 0x7fbd6806a4b0 2023-10-16T14:11:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd677fe640 1 -- 172.21.15.60:0/4158548189 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6002c770 con 0x7fbd6806a4b0 2023-10-16T14:11:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd677fe640 1 -- 172.21.15.60:0/4158548189 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6002cb00 con 0x7fbd6806a4b0 2023-10-16T14:11:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd6e95a640 1 -- 172.21.15.60:0/4158548189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6806a4b0 msgr2=0x7fbd680c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/4158548189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6806a4b0 0x7fbd680c9000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009dc0 tx=0x7fbd6002bc40 comp rx=0 tx=0).stop 2023-10-16T14:11:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd6e95a640 1 -- 172.21.15.60:0/4158548189 shutdown_connections 2023-10-16T14:11:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/4158548189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6806a4b0 0x7fbd680c9000 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd6e95a640 1 -- 172.21.15.60:0/4158548189 >> 172.21.15.60:0/4158548189 conn(0x7fbd680ce9b0 msgr2=0x7fbd680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.236+0000 7fbd6e95a640 1 -- 172.21.15.60:0/4158548189 shutdown_connections 2023-10-16T14:11:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.237+0000 7fbd6e95a640 1 -- 172.21.15.60:0/4158548189 wait complete. 2023-10-16T14:11:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.238+0000 7fbd6e95a640 1 Processor -- start 2023-10-16T14:11:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.238+0000 7fbd6e95a640 1 -- start start 2023-10-16T14:11:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.239+0000 7fbd6e95a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd6806a4b0 0x7fbd681558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.239+0000 7fbd6e95a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 0x7fbd68158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.239+0000 7fbd6e95a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6815d150 0x7fbd681551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.239+0000 7fbd6e95a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd681e9380 con 0x7fbd680ad350 2023-10-16T14:11:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.239+0000 7fbd6e95a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd681586a0 con 0x7fbd6806a4b0 2023-10-16T14:11:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.239+0000 7fbd6e95a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68155710 con 0x7fbd6815d150 2023-10-16T14:11:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 0x7fbd68158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 0x7fbd68158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55600/0 (socket says 172.21.15.60:55600) 2023-10-16T14:11:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 -- 172.21.15.60:0/1140266041 learned_addr learned my addr 172.21.15.60:0/1140266041 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd67fff640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd6806a4b0 0x7fbd681558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd6cef7640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6815d150 0x7fbd681551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 -- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6815d150 msgr2=0x7fbd681551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6815d150 0x7fbd681551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 -- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd6806a4b0 msgr2=0x7fbd681558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd6806a4b0 0x7fbd681558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd5ffff640 1 -- 172.21.15.60:0/1140266041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd60009a70 con 0x7fbd680ad350 2023-10-16T14:11:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.240+0000 7fbd6cef7640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6815d150 0x7fbd681551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.241+0000 7fbd5ffff640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 0x7fbd68158160 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c560 tx=0x7fbd5000ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.241+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd500074e0 con 0x7fbd680ad350 2023-10-16T14:11:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.241+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50007660 con 0x7fbd680ad350 2023-10-16T14:11:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.241+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50004a20 con 0x7fbd680ad350 2023-10-16T14:11:08.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.241+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd681e8dc0 con 0x7fbd680ad350 2023-10-16T14:11:08.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.241+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd68157960 con 0x7fbd680ad350 2023-10-16T14:11:08.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.242+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd681577c0 con 0x7fbd680ad350 2023-10-16T14:11:08.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.243+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fbd50004b80 con 0x7fbd680ad350 2023-10-16T14:11:08.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.244+0000 7fbd657fa640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 0x7fbd4806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.244+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fbd50089200 con 0x7fbd680ad350 2023-10-16T14:11:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.244+0000 7fbd67fff640 1 -- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 msgr2=0x7fbd4806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.244+0000 7fbd67fff640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 0x7fbd4806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.248+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbd681577c0 con 0x7fbd680ad350 2023-10-16T14:11:08.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.445+0000 7fbd67fff640 1 -- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 msgr2=0x7fbd4806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:08.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.445+0000 7fbd67fff640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 0x7fbd4806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:11:08.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.454+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd681577c0 con 0x7fbd680ad350 2023-10-16T14:11:08.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.455+0000 7fbd657fa640 1 -- 172.21.15.60:0/1140266041 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v236) v1 ==== 80+0+476 (secure 0 0 0) 0x7fbd681577c0 con 0x7fbd680ad350 2023-10-16T14:11:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1462669726 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1709631677 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/136559516 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2366541172 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3137196642 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.459 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/483659352 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.459 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/1724812344 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.459 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/1724812344 2023-10-17T14:10:58.183289+0000 2023-10-16T14:11:08.459 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T14:11:08.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 msgr2=0x7fbd4806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 0x7fbd4806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 msgr2=0x7fbd68158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 0x7fbd68158160 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c560 tx=0x7fbd5000ca20 comp rx=0 tx=0).stop 2023-10-16T14:11:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 shutdown_connections 2023-10-16T14:11:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7fbd4806b140 0x7fbd4806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbd6815d150 0x7fbd681551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd680ad350 0x7fbd68158160 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 --2- 172.21.15.60:0/1140266041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd6806a4b0 0x7fbd681558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.457+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 >> 172.21.15.60:0/1140266041 conn(0x7fbd680ce9b0 msgr2=0x7fbd68070170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:08.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.458+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 shutdown_connections 2023-10-16T14:11:08.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.458+0000 7fbd6e95a640 1 -- 172.21.15.60:0/1140266041 wait complete. 2023-10-16T14:11:08.477 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:11:08.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:08.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:11:08.538 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:08.534+0000 7f48c3a13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:11:08.562 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:11:08.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:08.563 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:11:08.604 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:11:08.605 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:08.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:11:08.659 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:11:08.659 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:08.717 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:11:08.717 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:11:08.718 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:08.771 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:11:08.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:08.781 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:08.776+0000 7f0611e11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:11:08.843 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:08.839+0000 7f48c3a13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:11:08.901 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:08.898+0000 7f48c3a13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:11:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.979+0000 7f15416af640 1 Processor -- start 2023-10-16T14:11:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.980+0000 7f15416af640 1 -- start start 2023-10-16T14:11:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.981+0000 7f15416af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c160420 0x7f153c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.981+0000 7f15416af640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f153c160800 0x7f153c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.981+0000 7f15416af640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f153c0adc40 0x7f153c06adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.981+0000 7f15416af640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0cde30 con 0x7f153c160800 2023-10-16T14:11:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.981+0000 7f15416af640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c06e170 con 0x7f153c0adc40 2023-10-16T14:11:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.981+0000 7f15416af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0cd460 con 0x7f153c160420 2023-10-16T14:11:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f153c160800 0x7f153c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f153c160800 0x7f153c0cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59158/0 (socket says 172.21.15.60:59158) 2023-10-16T14:11:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 -- 172.21.15.60:0/3917705984 learned_addr learned my addr 172.21.15.60:0/3917705984 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f153affd640 1 --2- 172.21.15.60:0/3917705984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c160420 0x7f153c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f153b7fe640 1 --2- 172.21.15.60:0/3917705984 >> v2:172.21.15.144:3300/0 conn(0x7f153c0adc40 0x7f153c06adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 -- 172.21.15.60:0/3917705984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c160420 msgr2=0x7f153c0ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 --2- 172.21.15.60:0/3917705984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c160420 0x7f153c0ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 -- 172.21.15.60:0/3917705984 >> v2:172.21.15.144:3300/0 conn(0x7f153c0adc40 msgr2=0x7f153c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:08.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 --2- 172.21.15.60:0/3917705984 >> v2:172.21.15.144:3300/0 conn(0x7f153c0adc40 0x7f153c06adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.982+0000 7f1533fff640 1 -- 172.21.15.60:0/3917705984 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c06e3c0 con 0x7f153c160800 2023-10-16T14:11:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.983+0000 7f1533fff640 1 --2- 172.21.15.60:0/3917705984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c160800 0x7f153c0cd8f0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f153c0b1ae0 tx=0x7f152402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e994f9c6c0fd26e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.984+0000 7f153a7fc640 1 -- 172.21.15.60:0/3917705984 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524031860 con 0x7f153c160800 2023-10-16T14:11:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.984+0000 7f153a7fc640 1 -- 172.21.15.60:0/3917705984 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15240319c0 con 0x7f153c160800 2023-10-16T14:11:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.984+0000 7f153a7fc640 1 -- 172.21.15.60:0/3917705984 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524031d50 con 0x7f153c160800 2023-10-16T14:11:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 -- 172.21.15.60:0/3917705984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c160800 msgr2=0x7f153c0cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 --2- 172.21.15.60:0/3917705984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c160800 0x7f153c0cd8f0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f153c0b1ae0 tx=0x7f152402b900 comp rx=0 tx=0).stop 2023-10-16T14:11:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 -- 172.21.15.60:0/3917705984 shutdown_connections 2023-10-16T14:11:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 --2- 172.21.15.60:0/3917705984 >> v2:172.21.15.144:3300/0 conn(0x7f153c0adc40 0x7f153c06adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 --2- 172.21.15.60:0/3917705984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c160800 0x7f153c0cd8f0 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 --2- 172.21.15.60:0/3917705984 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c160420 0x7f153c0ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 -- 172.21.15.60:0/3917705984 >> 172.21.15.60:0/3917705984 conn(0x7f153c0b0d30 msgr2=0x7f153c0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 -- 172.21.15.60:0/3917705984 shutdown_connections 2023-10-16T14:11:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.985+0000 7f15416af640 1 -- 172.21.15.60:0/3917705984 wait complete. 2023-10-16T14:11:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.987+0000 7f15416af640 1 Processor -- start 2023-10-16T14:11:08.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.987+0000 7f15416af640 1 -- start start 2023-10-16T14:11:08.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f15416af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 0x7f153c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f15416af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f153c160420 0x7f153c09f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f15416af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c09c7f0 0x7f153c09fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:08.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f15416af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c09bda0 con 0x7f153c0adc40 2023-10-16T14:11:08.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f15416af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0a0340 con 0x7f153c160420 2023-10-16T14:11:08.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f15416af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f3590 con 0x7f153c09c7f0 2023-10-16T14:11:08.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f153affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 0x7f153c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f153affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 0x7f153c15b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59166/0 (socket says 172.21.15.60:59166) 2023-10-16T14:11:08.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f153affd640 1 -- 172.21.15.60:0/83566585 learned_addr learned my addr 172.21.15.60:0/83566585 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:08.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.988+0000 7f1533fff640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f153c160420 0x7f153c09f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153b7fe640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c09c7f0 0x7f153c09fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:08.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153affd640 1 -- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c09c7f0 msgr2=0x7f153c09fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153affd640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c09c7f0 0x7f153c09fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153affd640 1 -- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f153c160420 msgr2=0x7f153c09f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153affd640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f153c160420 0x7f153c09f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153affd640 1 -- 172.21.15.60:0/83566585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240094d0 con 0x7f153c0adc40 2023-10-16T14:11:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153b7fe640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c09c7f0 0x7f153c09fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f1533fff640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f153c160420 0x7f153c09f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f153affd640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 0x7f153c15b220 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f152c00d5f0 tx=0x7f152c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:08.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c0125f0 con 0x7f153c0adc40 2023-10-16T14:11:08.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c012750 con 0x7f153c0adc40 2023-10-16T14:11:08.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c012b00 con 0x7f153c0adc40 2023-10-16T14:11:08.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.989+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1f3870 con 0x7f153c0adc40 2023-10-16T14:11:08.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.990+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c09c2d0 con 0x7f153c0adc40 2023-10-16T14:11:08.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.991+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c09c490 con 0x7f153c0adc40 2023-10-16T14:11:08.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.992+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f152c012c60 con 0x7f153c0adc40 2023-10-16T14:11:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.993+0000 7f15337fe640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f151c06b140 0x7f151c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.993+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f152c08a3f0 con 0x7f153c0adc40 2023-10-16T14:11:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.993+0000 7f1533fff640 1 -- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f151c06b140 msgr2=0x7f151c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1394204497 2023-10-16T14:11:09.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:08.993+0000 7f1533fff640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f151c06b140 0x7f151c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:11:09.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.001+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f152c054bd0 con 0x7f153c0adc40 2023-10-16T14:11:09.017 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.013+0000 7f48c3a13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:11:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.032+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f152c054fa0 con 0x7f153c0adc40 2023-10-16T14:11:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.033+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f151c06b140 msgr2=0x7f151c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:11:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.033+0000 7f15337fe640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f151c06b140 0x7f151c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.107 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.107 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.107 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.108 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.108 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.108 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.108 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.108 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.108 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.109 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.103+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.117 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:09.114+0000 7f0611e11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:11:09.119 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.116+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.119 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.116+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.116+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.116+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.116+0000 7f4845a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.116+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.117+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.117+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.117+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.122 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:11:09.117+0000 7f4848a08640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:11:09.179 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:09.177+0000 7f0611e11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:11:09.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.250+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153c1f3a60 con 0x7f153c0adc40 2023-10-16T14:11:09.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.251+0000 7f15337fe640 1 -- 172.21.15.60:0/83566585 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6153 (secure 0 0 0) 0x7f152c073d80 con 0x7f153c0adc40 2023-10-16T14:11:09.254 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:09.254 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:09.255 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 118 2023-10-16T14:11:09.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.253+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 msgr2=0x7f153c15b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.253+0000 7f15416af640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 0x7f153c15b220 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f152c00d5f0 tx=0x7f152c00dab0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 shutdown_connections 2023-10-16T14:11:09.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:6832/1394204497,v1:172.21.15.144:6833/1394204497] conn(0x7f151c06b140 0x7f151c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f153c09c7f0 0x7f153c09fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f153c160420 0x7f153c09f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 --2- 172.21.15.60:0/83566585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f153c0adc40 0x7f153c15b220 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 >> 172.21.15.60:0/83566585 conn(0x7f153c0b0d30 msgr2=0x7f153c070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 shutdown_connections 2023-10-16T14:11:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.254+0000 7f15416af640 1 -- 172.21.15.60:0/83566585 wait complete. 2023-10-16T14:11:09.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:09.302 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:11:09.300+0000 7f0611e11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:11:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.469+0000 7f9c12b81640 1 Processor -- start 2023-10-16T14:11:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.469+0000 7f9c12b81640 1 -- start start 2023-10-16T14:11:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12b81640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c0c06d520 0x7f9c0c0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12b81640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c06fa20 0x7f9c0c0ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12b81640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c06d900 con 0x7f9c0c065cc0 2023-10-16T14:11:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12b81640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c15b1d0 con 0x7f9c0c06d520 2023-10-16T14:11:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12b81640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c06feb0 con 0x7f9c0c06fa20 2023-10-16T14:11:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12380640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9c0c065cc0 0x7f9c0c070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40954/0 (socket says 172.21.15.60:40954) 2023-10-16T14:11:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.471+0000 7f9c12380640 1 -- 172.21.15.60:0/4090540221 learned_addr learned my addr 172.21.15.60:0/4090540221 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c1137e640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c06fa20 0x7f9c0c0ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c11b7f640 1 --2- 172.21.15.60:0/4090540221 >> v2:172.21.15.144:3300/0 conn(0x7f9c0c06d520 0x7f9c0c0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191696139 0 0) 0x7f9c0c06d900 con 0x7f9c0c065cc0 2023-10-16T14:11:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf8003520 con 0x7f9c0c065cc0 2023-10-16T14:11:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3048852452 0 0) 0x7f9bf8003520 con 0x7f9c0c065cc0 2023-10-16T14:11:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c06fa20 msgr2=0x7f9c0c0ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c06fa20 0x7f9c0c0ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 >> v2:172.21.15.144:3300/0 conn(0x7f9c0c06d520 msgr2=0x7f9c0c0b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 --2- 172.21.15.60:0/4090540221 >> v2:172.21.15.144:3300/0 conn(0x7f9c0c06d520 0x7f9c0c0b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c070190 con 0x7f9c0c065cc0 2023-10-16T14:11:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9c040019f0 con 0x7f9c0c065cc0 2023-10-16T14:11:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 >> v1:172.21.15.60:6789/0 conn(0x7f9c0c065cc0 legacy=0x7f9c0c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bf8003970 0x7f9bf8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80042d0 0x7f9bf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.472+0000 7f9c10b7d640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf8004eb0 0x7f9bf800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c15b1d0 con 0x7f9bf80042d0 2023-10-16T14:11:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c06feb0 con 0x7f9bf8004eb0 2023-10-16T14:11:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80102d0 con 0x7f9bf8003970 2023-10-16T14:11:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c1137e640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80042d0 0x7f9bf80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c11b7f640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bf8003970 0x7f9bf8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c1137e640 1 -- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bf8003970 msgr2=0x7f9bf8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c1137e640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9bf8003970 0x7f9bf8003d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c1137e640 1 -- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf8004eb0 msgr2=0x7f9bf800fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c1137e640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf8004eb0 0x7f9bf800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.473+0000 7f9c1137e640 1 -- 172.21.15.60:0/4090540221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8003520 con 0x7f9bf80042d0 2023-10-16T14:11:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.474+0000 7f9c1137e640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80042d0 0x7f9bf80047c0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009ac0 tx=0x7f9bf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0611698aabbb05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc02c6b0 con 0x7f9bf80042d0 2023-10-16T14:11:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc02c810 con 0x7f9bf80042d0 2023-10-16T14:11:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c10b7d640 1 -- 172.21.15.60:0/4090540221 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc02cba0 con 0x7f9bf80042d0 2023-10-16T14:11:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12b81640 1 -- 172.21.15.60:0/4090540221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80042d0 msgr2=0x7f9bf80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12b81640 1 --2- 172.21.15.60:0/4090540221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80042d0 0x7f9bf80047c0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009ac0 tx=0x7f9bf80057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12380640 1 -- 172.21.15.60:0/4090540221 reap_dead start 2023-10-16T14:11:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12b81640 1 -- 172.21.15.60:0/4090540221 shutdown_connections 2023-10-16T14:11:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12b81640 1 -- 172.21.15.60:0/4090540221 >> 172.21.15.60:0/4090540221 conn(0x7f9c0c0b0a80 msgr2=0x7f9c0c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12b81640 1 -- 172.21.15.60:0/4090540221 shutdown_connections 2023-10-16T14:11:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.475+0000 7f9c12b81640 1 -- 172.21.15.60:0/4090540221 wait complete. 2023-10-16T14:11:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.477+0000 7f9c12b81640 1 Processor -- start 2023-10-16T14:11:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.477+0000 7f9c12b81640 1 -- start start 2023-10-16T14:11:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.478+0000 7f9c12b81640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf80042d0 0x7f9c0c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.478+0000 7f9c12b81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 0x7f9c0c1f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.478+0000 7f9c12b81640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15dcf0 0x7f9c0c1f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.478+0000 7f9c12b81640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c158270 con 0x7f9c0c15d870 2023-10-16T14:11:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.478+0000 7f9c12b81640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c15c080 con 0x7f9bf80042d0 2023-10-16T14:11:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.478+0000 7f9c12b81640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c1584a0 con 0x7f9c0c15dcf0 2023-10-16T14:11:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 0x7f9c0c1f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 0x7f9c0c1f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59170/0 (socket says 172.21.15.60:59170) 2023-10-16T14:11:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 -- 172.21.15.60:0/3378385975 learned_addr learned my addr 172.21.15.60:0/3378385975 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c11b7f640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf80042d0 0x7f9c0c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c12380640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15dcf0 0x7f9c0c1f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 -- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15dcf0 msgr2=0x7f9c0c1f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15dcf0 0x7f9c0c1f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 -- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf80042d0 msgr2=0x7f9c0c157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf80042d0 0x7f9c0c157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 -- 172.21.15.60:0/3378385975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c158780 con 0x7f9c0c15d870 2023-10-16T14:11:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c12380640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15dcf0 0x7f9c0c1f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c11b7f640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf80042d0 0x7f9c0c157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c1137e640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 0x7f9c0c1f1930 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc02c250 tx=0x7f9bfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.479+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc003810 con 0x7f9c0c15d870 2023-10-16T14:11:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.480+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc003970 con 0x7f9c0c15d870 2023-10-16T14:11:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.480+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc003ce0 con 0x7f9c0c15d870 2023-10-16T14:11:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.480+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c158a60 con 0x7f9c0c15d870 2023-10-16T14:11:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.480+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c159020 con 0x7f9c0c15d870 2023-10-16T14:11:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.481+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c1591e0 con 0x7f9c0c15d870 2023-10-16T14:11:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.482+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9bfc034440 con 0x7f9c0c15d870 2023-10-16T14:11:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.483+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7f9bfc0ab9d0 con 0x7f9c0c15d870 2023-10-16T14:11:09.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.487+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9bfc076800 con 0x7f9c0c15d870 2023-10-16T14:11:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.674+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c0c158270 con 0x7f9c0c15d870 2023-10-16T14:11:09.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.677+0000 7f9c027fc640 1 -- 172.21.15.60:0/3378385975 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+21499 (secure 0 0 0) 0x7f9bfc0044e0 con 0x7f9c0c15d870 2023-10-16T14:11:09.681 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:09.681 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":237,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:09.018780+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:09.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 msgr2=0x7f9c0c1f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 0x7f9c0c1f1930 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc02c250 tx=0x7f9bfc004730 comp rx=0 tx=0).stop 2023-10-16T14:11:09.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 shutdown_connections 2023-10-16T14:11:09.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15dcf0 0x7f9c0c1f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c15d870 0x7f9c0c1f1930 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 --2- 172.21.15.60:0/3378385975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9bf80042d0 0x7f9c0c157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 >> 172.21.15.60:0/3378385975 conn(0x7f9c0c0b0a80 msgr2=0x7f9c0c15a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:09.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 shutdown_connections 2023-10-16T14:11:09.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.680+0000 7f9c12b81640 1 -- 172.21.15.60:0/3378385975 wait complete. 2023-10-16T14:11:09.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:10:24.220406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:10:25.316288+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T14:10:38.740891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T14:10:39.464305+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:11:09.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:11:09.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.970+0000 7fc6de667640 1 Processor -- start 2023-10-16T14:11:09.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.971+0000 7fc6de667640 1 -- start start 2023-10-16T14:11:09.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.972+0000 7fc6de667640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b1ca0 0x7fc6d80d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.972+0000 7fc6de667640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6d80cfc40 0x7fc6d80b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.972+0000 7fc6de667640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d80d0970 con 0x7fc6d80b2080 2023-10-16T14:11:09.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.972+0000 7fc6de667640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d80d0ea0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.972+0000 7fc6de667640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80d0bc0 con 0x7fc6d80b1ca0 2023-10-16T14:11:09.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.972+0000 7fc6dde66640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6d80cfc40 0x7fc6d80b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dde66640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6d80cfc40 0x7fc6d80b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59186/0 (socket says 172.21.15.60:59186) 2023-10-16T14:11:09.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dde66640 1 -- 172.21.15.60:0/2506144499 learned_addr learned my addr 172.21.15.60:0/2506144499 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:09.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dce64640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc6d80b2080 0x7fc6d80ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40962/0 (socket says 172.21.15.60:40962) 2023-10-16T14:11:09.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dd665640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b1ca0 0x7fc6d80d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dde66640 1 -- 172.21.15.60:0/2506144499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b1ca0 msgr2=0x7fc6d80d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:09.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dde66640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b1ca0 0x7fc6d80d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dde66640 1 -- 172.21.15.60:0/2506144499 >> v1:172.21.15.60:6789/0 conn(0x7fc6d80b2080 legacy=0x7fc6d80ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6d7fff640 1 -- 172.21.15.60:0/2506144499 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959295591 0 0) 0x7fc6d80d0970 con 0x7fc6d80b2080 2023-10-16T14:11:09.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dde66640 1 -- 172.21.15.60:0/2506144499 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80cd940 con 0x7fc6d80cfc40 2023-10-16T14:11:09.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.973+0000 7fc6dd665640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b1ca0 0x7fc6d80d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:09.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.974+0000 7fc6dde66640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d80b24c0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009870 tx=0x7fc6d002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b78a8326ef5c6f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:09.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6d7fff640 1 -- 172.21.15.60:0/2506144499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d002f830 con 0x7fc6d80cfc40 2023-10-16T14:11:09.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6d7fff640 1 -- 172.21.15.60:0/2506144499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d002f990 con 0x7fc6d80cfc40 2023-10-16T14:11:09.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6d7fff640 1 -- 172.21.15.60:0/2506144499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d002fd20 con 0x7fc6d80cfc40 2023-10-16T14:11:09.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 -- 172.21.15.60:0/2506144499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 msgr2=0x7fc6d80b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d80b24c0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009870 tx=0x7fc6d002ba10 comp rx=0 tx=0).stop 2023-10-16T14:11:09.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 -- 172.21.15.60:0/2506144499 shutdown_connections 2023-10-16T14:11:09.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d80b24c0 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 --2- 172.21.15.60:0/2506144499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b1ca0 0x7fc6d80d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 -- 172.21.15.60:0/2506144499 >> 172.21.15.60:0/2506144499 conn(0x7fc6d80d2b90 msgr2=0x7fc6d80d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:09.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 -- 172.21.15.60:0/2506144499 shutdown_connections 2023-10-16T14:11:09.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.975+0000 7fc6de667640 1 -- 172.21.15.60:0/2506144499 wait complete. 2023-10-16T14:11:09.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.977+0000 7fc6de667640 1 Processor -- start 2023-10-16T14:11:09.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.977+0000 7fc6de667640 1 -- start start 2023-10-16T14:11:09.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.978+0000 7fc6de667640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d80b1ca0 0x7fc6d80c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.978+0000 7fc6de667640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b2080 0x7fc6d809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.978+0000 7fc6de667640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d809ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.978+0000 7fc6de667640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80a02d0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.978+0000 7fc6de667640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d809f230 con 0x7fc6d80b1ca0 2023-10-16T14:11:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.978+0000 7fc6de667640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d809f510 con 0x7fc6d80b2080 2023-10-16T14:11:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d809ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d809ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59192/0 (socket says 172.21.15.60:59192) 2023-10-16T14:11:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 -- 172.21.15.60:0/196132833 learned_addr learned my addr 172.21.15.60:0/196132833 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dce64640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b2080 0x7fc6d809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dd665640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d80b1ca0 0x7fc6d80c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 -- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b2080 msgr2=0x7fc6d809e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b2080 0x7fc6d809e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 -- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d80b1ca0 msgr2=0x7fc6d80c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d80b1ca0 0x7fc6d80c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dde66640 1 -- 172.21.15.60:0/196132833 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00094d0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dce64640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b2080 0x7fc6d809e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.979+0000 7fc6dd665640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d80b1ca0 0x7fc6d80c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.980+0000 7fc6dde66640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d809ebf0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00090b0 tx=0x7fc6d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.980+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d003b040 con 0x7fc6d80cfc40 2023-10-16T14:11:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.980+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d0037cd0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.980+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0004270 con 0x7fc6d80cfc40 2023-10-16T14:11:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.980+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d809f7f0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.980+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d809fdb0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.981+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d804e9b0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.982+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc6d00043d0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.983+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7fc6d00ac7d0 con 0x7fc6d80cfc40 2023-10-16T14:11:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:09.987+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6d0077550 con 0x7fc6d80cfc40 2023-10-16T14:11:10.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.041+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc6d0077000 con 0x7fc6d80cfc40 2023-10-16T14:11:10.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.041+0000 7fc6d5ffb640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc6c4072d30 0x7fc6c4075120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.050+0000 7fc6dd665640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc6c4072d30 0x7fc6c4075120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.062+0000 7fc6dd665640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc6c4072d30 0x7fc6c4075120 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8007f40 tx=0x7fc6c8007490 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.193+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6d8073a40 con 0x7fc6d80cfc40 2023-10-16T14:11:10.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.194+0000 7fc6d5ffb640 1 -- 172.21.15.60:0/196132833 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v237) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6d007e360 con 0x7fc6d80cfc40 2023-10-16T14:11:10.197 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:11:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.197+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc6c4072d30 msgr2=0x7fc6c4075120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.197+0000 7fc6de667640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc6c4072d30 0x7fc6c4075120 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8007f40 tx=0x7fc6c8007490 comp rx=0 tx=0).stop 2023-10-16T14:11:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.197+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 msgr2=0x7fc6d809ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.197+0000 7fc6de667640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d809ebf0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00090b0 tx=0x7fc6d00046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 shutdown_connections 2023-10-16T14:11:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc6c4072d30 0x7fc6c4075120 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc6d80b2080 0x7fc6d809e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6d80cfc40 0x7fc6d809ebf0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 --2- 172.21.15.60:0/196132833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6d80b1ca0 0x7fc6d80c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 >> 172.21.15.60:0/196132833 conn(0x7fc6d80d2b90 msgr2=0x7fc6d8160120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 shutdown_connections 2023-10-16T14:11:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.198+0000 7fc6de667640 1 -- 172.21.15.60:0/196132833 wait complete. 2023-10-16T14:11:10.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:11:10.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.429+0000 7f49b3207640 1 Processor -- start 2023-10-16T14:11:10.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.429+0000 7f49b3207640 1 -- start start 2023-10-16T14:11:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.430+0000 7f49b3207640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49ac15d150 0x7f49ac0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.430+0000 7f49b3207640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f49ac160b90 0x7f49ac160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b3207640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0ac890 con 0x7f49ac069da0 2023-10-16T14:11:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b3207640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac15d530 con 0x7f49ac15d150 2023-10-16T14:11:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b3207640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac161650 con 0x7f49ac160b90 2023-10-16T14:11:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49ac15d150 0x7f49ac0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49ac15d150 0x7f49ac0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59194/0 (socket says 172.21.15.60:59194) 2023-10-16T14:11:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 -- 172.21.15.60:0/1380877753 learned_addr learned my addr 172.21.15.60:0/1380877753 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 -- 172.21.15.60:0/1380877753 >> v2:172.21.15.144:3300/0 conn(0x7f49ac160b90 msgr2=0x7f49ac160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 --2- 172.21.15.60:0/1380877753 >> v2:172.21.15.144:3300/0 conn(0x7f49ac160b90 0x7f49ac160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 -- 172.21.15.60:0/1380877753 >> v1:172.21.15.144:6789/0 conn(0x7f49ac069da0 legacy=0x7f49ac0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.431+0000 7f49b0fa3640 1 -- 172.21.15.60:0/1380877753 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac161930 con 0x7f49ac15d150 2023-10-16T14:11:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.432+0000 7f49b0fa3640 1 --2- 172.21.15.60:0/1380877753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac15d150 0x7f49ac0ad350 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0cf990 tx=0x7f499c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e2ebc5283e4d05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.433+0000 7f49ab7fe640 1 -- 172.21.15.60:0/1380877753 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c0317e0 con 0x7f49ac15d150 2023-10-16T14:11:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.433+0000 7f49ab7fe640 1 -- 172.21.15.60:0/1380877753 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c031940 con 0x7f49ac15d150 2023-10-16T14:11:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.433+0000 7f49ab7fe640 1 -- 172.21.15.60:0/1380877753 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c031cd0 con 0x7f49ac15d150 2023-10-16T14:11:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 -- 172.21.15.60:0/1380877753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac15d150 msgr2=0x7f49ac0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 --2- 172.21.15.60:0/1380877753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac15d150 0x7f49ac0ad350 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0cf990 tx=0x7f499c02b880 comp rx=0 tx=0).stop 2023-10-16T14:11:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 -- 172.21.15.60:0/1380877753 shutdown_connections 2023-10-16T14:11:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 --2- 172.21.15.60:0/1380877753 >> v2:172.21.15.144:3300/0 conn(0x7f49ac160b90 0x7f49ac160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 --2- 172.21.15.60:0/1380877753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac15d150 0x7f49ac0ad350 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 -- 172.21.15.60:0/1380877753 >> 172.21.15.60:0/1380877753 conn(0x7f49ac0ce9b0 msgr2=0x7f49ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:10.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 -- 172.21.15.60:0/1380877753 shutdown_connections 2023-10-16T14:11:10.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.434+0000 7f49b3207640 1 -- 172.21.15.60:0/1380877753 wait complete. 2023-10-16T14:11:10.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.436+0000 7f49b3207640 1 Processor -- start 2023-10-16T14:11:10.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.437+0000 7f49b3207640 1 -- start start 2023-10-16T14:11:10.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b3207640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49ac069da0 0x7f49ac15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b3207640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49ac15d150 0x7f49ac1e8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b3207640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 0x7f49ac0c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b3207640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c7240 con 0x7f49ac160b90 2023-10-16T14:11:10.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b3207640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac15bd40 con 0x7f49ac15d150 2023-10-16T14:11:10.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b3207640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c6630 con 0x7f49ac069da0 2023-10-16T14:11:10.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b17a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 0x7f49ac0c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b17a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 0x7f49ac0c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59200/0 (socket says 172.21.15.60:59200) 2023-10-16T14:11:10.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.438+0000 7f49b17a4640 1 -- 172.21.15.60:0/2882054908 learned_addr learned my addr 172.21.15.60:0/2882054908 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:10.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49abfff640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49ac15d150 0x7f49ac1e8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b0fa3640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49ac069da0 0x7f49ac15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b17a4640 1 -- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49ac069da0 msgr2=0x7f49ac15aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b17a4640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49ac069da0 0x7f49ac15aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b17a4640 1 -- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49ac15d150 msgr2=0x7f49ac1e8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b17a4640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49ac15d150 0x7f49ac1e8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b17a4640 1 -- 172.21.15.60:0/2882054908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c0094d0 con 0x7f49ac160b90 2023-10-16T14:11:10.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b0fa3640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49ac069da0 0x7f49ac15aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:10.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.439+0000 7f49b17a4640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 0x7f49ac0c46b0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f49a400b990 tx=0x7f49a400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.440+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a400cba0 con 0x7f49ac160b90 2023-10-16T14:11:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.440+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a400cd00 con 0x7f49ac160b90 2023-10-16T14:11:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.440+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4014680 con 0x7f49ac160b90 2023-10-16T14:11:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.440+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac0c6910 con 0x7f49ac160b90 2023-10-16T14:11:10.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.440+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0c6ed0 con 0x7f49ac160b90 2023-10-16T14:11:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.441+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac0c73f0 con 0x7f49ac160b90 2023-10-16T14:11:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.442+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f49a401d3b0 con 0x7f49ac160b90 2023-10-16T14:11:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.443+0000 7f49a97fa640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f499806acc0 0x7f499806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.443+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7f49a4089900 con 0x7f49ac160b90 2023-10-16T14:11:10.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.444+0000 7f49b0fa3640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f499806acc0 0x7f499806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.445+0000 7f49b0fa3640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f499806acc0 0x7f499806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f499c036000 tx=0x7f499c03d040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.449+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49ac0c73f0 con 0x7f49ac160b90 2023-10-16T14:11:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.642+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f49ac04e160 con 0x7f49ac160b90 2023-10-16T14:11:10.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.643+0000 7f49a97fa640 1 -- 172.21.15.60:0/2882054908 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 104+0+11 (secure 0 0 0) 0x7f49a4019020 con 0x7f49ac160b90 2023-10-16T14:11:10.646 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:11:10.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f499806acc0 msgr2=0x7f499806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f499806acc0 0x7f499806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f499c036000 tx=0x7f499c03d040 comp rx=0 tx=0).stop 2023-10-16T14:11:10.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 msgr2=0x7f49ac0c46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 0x7f49ac0c46b0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f49a400b990 tx=0x7f49a400be50 comp rx=0 tx=0).stop 2023-10-16T14:11:10.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 shutdown_connections 2023-10-16T14:11:10.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f499806acc0 0x7f499806d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49ac160b90 0x7f49ac0c46b0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49ac15d150 0x7f49ac1e8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 --2- 172.21.15.60:0/2882054908 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49ac069da0 0x7f49ac15aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 >> 172.21.15.60:0/2882054908 conn(0x7f49ac0ce9b0 msgr2=0x7f49ac0ad350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 shutdown_connections 2023-10-16T14:11:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.646+0000 7f49b3207640 1 -- 172.21.15.60:0/2882054908 wait complete. 2023-10-16T14:11:10.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:11:10.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.888+0000 7f00c3fff640 1 Processor -- start 2023-10-16T14:11:10.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.888+0000 7f00c3fff640 1 -- start start 2023-10-16T14:11:10.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 0x7f00c40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c3fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c415b9d0 con 0x7f00c406c040 2023-10-16T14:11:10.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c3fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c415bcb0 con 0x7f00c40ccfb0 2023-10-16T14:11:10.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c415bf90 con 0x7f00c40ad970 2023-10-16T14:11:10.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f00c406c040 0x7f00c406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40970/0 (socket says 172.21.15.60:40970) 2023-10-16T14:11:10.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c2ffd640 1 -- 172.21.15.60:0/3472305345 learned_addr learned my addr 172.21.15.60:0/3472305345 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:10.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069714117 0 0) 0x7f00c415b9d0 con 0x7f00c406c040 2023-10-16T14:11:10.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.890+0000 7f00c27fc640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 0x7f00c40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac003520 con 0x7f00c406c040 2023-10-16T14:11:10.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011269797 0 0) 0x7f00ac003520 con 0x7f00c406c040 2023-10-16T14:11:10.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 msgr2=0x7f00c40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:10.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 0x7f00c40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 >> v1:172.21.15.144:6789/0 conn(0x7f00c40ccfb0 legacy=0x7f00c40cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c40cc270 con 0x7f00c406c040 2023-10-16T14:11:10.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f00b4001670 con 0x7f00c406c040 2023-10-16T14:11:10.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 >> v1:172.21.15.60:6789/0 conn(0x7f00c406c040 legacy=0x7f00c406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 0x7f00ac0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac003f70 0x7f00ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ac004d00 0x7f00ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c415bf90 con 0x7f00ac003f70 2023-10-16T14:11:10.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac00bac0 con 0x7f00ac004d00 2023-10-16T14:11:10.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac00bc80 con 0x7f00c40ad970 2023-10-16T14:11:10.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.891+0000 7f00c27fc640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac003f70 0x7f00ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c37fe640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ac004d00 0x7f00ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c27fc640 1 -- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 msgr2=0x7f00ac0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:10.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c27fc640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00c40ad970 0x7f00ac0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c27fc640 1 -- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ac004d00 msgr2=0x7f00ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:10.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c27fc640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ac004d00 0x7f00ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c27fc640 1 -- 172.21.15.60:0/3472305345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac003520 con 0x7f00ac003f70 2023-10-16T14:11:10.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.892+0000 7f00c37fe640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00ac004d00 0x7f00ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:10.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.893+0000 7f00c27fc640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac003f70 0x7f00ac0047c0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009ac0 tx=0x7f00ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddba33de60846fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.893+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b802c630 con 0x7f00ac003f70 2023-10-16T14:11:10.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.893+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b802c790 con 0x7f00ac003f70 2023-10-16T14:11:10.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.893+0000 7f00c1ffb640 1 -- 172.21.15.60:0/3472305345 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b802cb20 con 0x7f00ac003f70 2023-10-16T14:11:10.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c3fff640 1 -- 172.21.15.60:0/3472305345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac003f70 msgr2=0x7f00ac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c3fff640 1 --2- 172.21.15.60:0/3472305345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac003f70 0x7f00ac0047c0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009ac0 tx=0x7f00ac0057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c37fe640 1 -- 172.21.15.60:0/3472305345 reap_dead start 2023-10-16T14:11:10.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c3fff640 1 -- 172.21.15.60:0/3472305345 shutdown_connections 2023-10-16T14:11:10.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c3fff640 1 -- 172.21.15.60:0/3472305345 >> 172.21.15.60:0/3472305345 conn(0x7f00c40ce6a0 msgr2=0x7f00c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:10.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c3fff640 1 -- 172.21.15.60:0/3472305345 shutdown_connections 2023-10-16T14:11:10.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.894+0000 7f00c3fff640 1 -- 172.21.15.60:0/3472305345 wait complete. 2023-10-16T14:11:10.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.896+0000 7f00c3fff640 1 Processor -- start 2023-10-16T14:11:10.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.896+0000 7f00c3fff640 1 -- start start 2023-10-16T14:11:10.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00ac003f70 0x7f00c415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 0x7f00c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00c40ae100 0x7f00c40ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c3fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40af9d0 con 0x7f00c406f700 2023-10-16T14:11:10.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c3fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40c81b0 con 0x7f00c40ae100 2023-10-16T14:11:10.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40700c0 con 0x7f00ac003f70 2023-10-16T14:11:10.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 0x7f00c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 0x7f00c406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59230/0 (socket says 172.21.15.60:59230) 2023-10-16T14:11:10.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.897+0000 7f00c27fc640 1 -- 172.21.15.60:0/1983025988 learned_addr learned my addr 172.21.15.60:0/1983025988 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:10.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c27fc640 1 -- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00ac003f70 msgr2=0x7f00c415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c27fc640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00ac003f70 0x7f00c415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c37fe640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00c40ae100 0x7f00c40ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c27fc640 1 -- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00c40ae100 msgr2=0x7f00c40ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:10.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c27fc640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00c40ae100 0x7f00c40ae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:10.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c27fc640 1 -- 172.21.15.60:0/1983025988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c40cdcd0 con 0x7f00c406f700 2023-10-16T14:11:10.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00c27fc640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 0x7f00c406fb80 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f00b802c1d0 tx=0x7f00b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8003810 con 0x7f00c406f700 2023-10-16T14:11:10.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b8003970 con 0x7f00c406f700 2023-10-16T14:11:10.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.898+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b8003d00 con 0x7f00c406f700 2023-10-16T14:11:10.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.899+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c40aeac0 con 0x7f00c406f700 2023-10-16T14:11:10.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.899+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c40af030 con 0x7f00c406f700 2023-10-16T14:11:10.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.900+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c40afc40 con 0x7f00c406f700 2023-10-16T14:11:10.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.901+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f00b8034440 con 0x7f00c406f700 2023-10-16T14:11:10.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.902+0000 7f00ab7fe640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f00b006acc0 0x7f00b006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:10.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.902+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7f00b80ab8f0 con 0x7f00c406f700 2023-10-16T14:11:10.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.902+0000 7f00c2ffd640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f00b006acc0 0x7f00b006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:10.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.903+0000 7f00c2ffd640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f00b006acc0 0x7f00b006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f00b40079b0 tx=0x7f00b4001540 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:10.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:10.906+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00b80766a0 con 0x7f00c406f700 2023-10-16T14:11:11.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.064+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f00b8076a70 con 0x7f00c406f700 2023-10-16T14:11:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.113+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f00c404e9b0 con 0x7f00c406f700 2023-10-16T14:11:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.114+0000 7f00ab7fe640 1 -- 172.21.15.60:0/1983025988 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v237) v1 ==== 100+0+11 (secure 0 0 0) 0x7f00b808e7e0 con 0x7f00c406f700 2023-10-16T14:11:11.117 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:11:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f00b006acc0 msgr2=0x7f00b006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f00b006acc0 0x7f00b006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f00b40079b0 tx=0x7f00b4001540 comp rx=0 tx=0).stop 2023-10-16T14:11:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 msgr2=0x7f00c406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 0x7f00c406fb80 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f00b802c1d0 tx=0x7f00b8004730 comp rx=0 tx=0).stop 2023-10-16T14:11:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 shutdown_connections 2023-10-16T14:11:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f00b006acc0 0x7f00b006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00c40ae100 0x7f00c40ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c406f700 0x7f00c406fb80 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 --2- 172.21.15.60:0/1983025988 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f00ac003f70 0x7f00c415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.117+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 >> 172.21.15.60:0/1983025988 conn(0x7f00c40ce6a0 msgr2=0x7f00c40ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.118+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 shutdown_connections 2023-10-16T14:11:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.118+0000 7f00c3fff640 1 -- 172.21.15.60:0/1983025988 wait complete. 2023-10-16T14:11:11.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T14:11:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.359+0000 7fdc864b4640 1 Processor -- start 2023-10-16T14:11:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.359+0000 7fdc864b4640 1 -- start start 2023-10-16T14:11:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc864b4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc8006c040 0x7fdc8006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc864b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800cda80 0x7fdc8015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc864b4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8006c960 con 0x7fdc800ad970 2023-10-16T14:11:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc864b4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8015b9d0 con 0x7fdc8006c040 2023-10-16T14:11:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc864b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8015bcb0 con 0x7fdc800cda80 2023-10-16T14:11:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc854b2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc8006c040 0x7fdc8006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc854b2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc8006c040 0x7fdc8006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59246/0 (socket says 172.21.15.60:59246) 2023-10-16T14:11:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.361+0000 7fdc854b2640 1 -- 172.21.15.60:0/1431692888 learned_addr learned my addr 172.21.15.60:0/1431692888 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:11.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.362+0000 7fdc854b2640 1 -- 172.21.15.60:0/1431692888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800cda80 msgr2=0x7fdc8015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.362+0000 7fdc854b2640 1 --2- 172.21.15.60:0/1431692888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800cda80 0x7fdc8015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.362+0000 7fdc854b2640 1 -- 172.21.15.60:0/1431692888 >> v1:172.21.15.60:6789/0 conn(0x7fdc800ad970 legacy=0x7fdc8015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.362+0000 7fdc854b2640 1 -- 172.21.15.60:0/1431692888 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8015bf90 con 0x7fdc8006c040 2023-10-16T14:11:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.363+0000 7fdc854b2640 1 --2- 172.21.15.60:0/1431692888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc8006c040 0x7fdc8006c420 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009e50 tx=0x7fdc7c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a4eee069ea5395 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.363+0000 7fdc77fff640 1 -- 172.21.15.60:0/1431692888 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c02cb00 con 0x7fdc8006c040 2023-10-16T14:11:11.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.363+0000 7fdc77fff640 1 -- 172.21.15.60:0/1431692888 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c02cc60 con 0x7fdc8006c040 2023-10-16T14:11:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.363+0000 7fdc77fff640 1 -- 172.21.15.60:0/1431692888 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0343f0 con 0x7fdc8006c040 2023-10-16T14:11:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 -- 172.21.15.60:0/1431692888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc8006c040 msgr2=0x7fdc8006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1431692888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc8006c040 0x7fdc8006c420 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009e50 tx=0x7fdc7c02bf60 comp rx=0 tx=0).stop 2023-10-16T14:11:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 -- 172.21.15.60:0/1431692888 shutdown_connections 2023-10-16T14:11:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1431692888 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800cda80 0x7fdc8015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1431692888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc8006c040 0x7fdc8006c420 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 -- 172.21.15.60:0/1431692888 >> 172.21.15.60:0/1431692888 conn(0x7fdc800ce6a0 msgr2=0x7fdc800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 -- 172.21.15.60:0/1431692888 shutdown_connections 2023-10-16T14:11:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.364+0000 7fdc864b4640 1 -- 172.21.15.60:0/1431692888 wait complete. 2023-10-16T14:11:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.366+0000 7fdc864b4640 1 Processor -- start 2023-10-16T14:11:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.366+0000 7fdc864b4640 1 -- start start 2023-10-16T14:11:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc864b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc8006c040 0x7fdc80154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc864b4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800ad970 0x7fdc800c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc864b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 0x7fdc800c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc864b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc800c5020 con 0x7fdc800cda80 2023-10-16T14:11:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc864b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc800c80b0 con 0x7fdc8006c040 2023-10-16T14:11:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc864b4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc800c8390 con 0x7fdc800ad970 2023-10-16T14:11:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc85cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 0x7fdc800c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc85cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 0x7fdc800c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59260/0 (socket says 172.21.15.60:59260) 2023-10-16T14:11:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.367+0000 7fdc85cb3640 1 -- 172.21.15.60:0/1406089959 learned_addr learned my addr 172.21.15.60:0/1406089959 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc85cb3640 1 -- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800ad970 msgr2=0x7fdc800c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc84cb1640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800ad970 0x7fdc800c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc85cb3640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800ad970 0x7fdc800c4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc85cb3640 1 -- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc8006c040 msgr2=0x7fdc80154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc85cb3640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc8006c040 0x7fdc80154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc85cb3640 1 -- 172.21.15.60:0/1406089959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c009b30 con 0x7fdc800cda80 2023-10-16T14:11:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc85cb3640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 0x7fdc800c7760 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fdc78009fd0 tx=0x7fdc7800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78011830 con 0x7fdc800cda80 2023-10-16T14:11:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc78011990 con 0x7fdc800cda80 2023-10-16T14:11:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78011d20 con 0x7fdc800cda80 2023-10-16T14:11:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc800b1460 con 0x7fdc800cda80 2023-10-16T14:11:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.368+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc800c7ca0 con 0x7fdc800cda80 2023-10-16T14:11:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.370+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8006b330 con 0x7fdc800cda80 2023-10-16T14:11:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.371+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fdc7801b460 con 0x7fdc800cda80 2023-10-16T14:11:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.372+0000 7fdc75ffb640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdc6406b070 0x7fdc6406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.372+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7fdc78089da0 con 0x7fdc800cda80 2023-10-16T14:11:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.372+0000 7fdc854b2640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdc6406b070 0x7fdc6406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.373+0000 7fdc854b2640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdc6406b070 0x7fdc6406d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c035000 tx=0x7fdc7c02e000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.376+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc780545b0 con 0x7fdc800cda80 2023-10-16T14:11:11.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.573+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdc800b12c0 con 0x7fdc800cda80 2023-10-16T14:11:11.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.574+0000 7fdc75ffb640 1 -- 172.21.15.60:0/1406089959 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 107+0+11 (secure 0 0 0) 0x7fdc800b12c0 con 0x7fdc800cda80 2023-10-16T14:11:11.577 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:11:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdc6406b070 msgr2=0x7fdc6406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdc6406b070 0x7fdc6406d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c035000 tx=0x7fdc7c02e000 comp rx=0 tx=0).stop 2023-10-16T14:11:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 msgr2=0x7fdc800c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 0x7fdc800c7760 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fdc78009fd0 tx=0x7fdc7800b8e0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 shutdown_connections 2023-10-16T14:11:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdc6406b070 0x7fdc6406d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdc800ad970 0x7fdc800c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc800cda80 0x7fdc800c7760 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 --2- 172.21.15.60:0/1406089959 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc8006c040 0x7fdc80154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.577+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 >> 172.21.15.60:0/1406089959 conn(0x7fdc800ce6a0 msgr2=0x7fdc800c8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.578+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 shutdown_connections 2023-10-16T14:11:11.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.578+0000 7fdc864b4640 1 -- 172.21.15.60:0/1406089959 wait complete. 2023-10-16T14:11:11.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T14:11:11.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.819+0000 7fc07f167640 1 Processor -- start 2023-10-16T14:11:11.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.819+0000 7fc07f167640 1 -- start start 2023-10-16T14:11:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07f167640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc078070b60 0x7fc07815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07f167640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07815aef0 con 0x7fc07806c100 2023-10-16T14:11:11.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07f167640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07815d910 con 0x7fc078159950 2023-10-16T14:11:11.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07f167640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07815e490 con 0x7fc078070b60 2023-10-16T14:11:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07d964640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc078070b60 0x7fc07815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07d964640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc078070b60 0x7fc07815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59268/0 (socket says 172.21.15.60:59268) 2023-10-16T14:11:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07d964640 1 -- 172.21.15.60:0/2497984511 learned_addr learned my addr 172.21.15.60:0/2497984511 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.821+0000 7fc07e966640 1 --1- 172.21.15.60:0/2497984511 >> v1:172.21.15.60:6789/0 conn(0x7fc07806c100 0x7fc07806c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40992/0 (socket says 172.21.15.60:40992) 2023-10-16T14:11:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.822+0000 7fc07d964640 1 -- 172.21.15.60:0/2497984511 >> v1:172.21.15.144:6789/0 conn(0x7fc078159950 legacy=0x7fc078070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.822+0000 7fc07d964640 1 -- 172.21.15.60:0/2497984511 >> v1:172.21.15.60:6789/0 conn(0x7fc07806c100 legacy=0x7fc07806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.822+0000 7fc07d964640 1 -- 172.21.15.60:0/2497984511 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07815dbf0 con 0x7fc078070b60 2023-10-16T14:11:11.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.823+0000 7fc07d964640 1 --2- 172.21.15.60:0/2497984511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc07815a9b0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fc068009870 tx=0x7fc06802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2df9c4b61bca98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.825+0000 7fc07d163640 1 -- 172.21.15.60:0/2497984511 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068030850 con 0x7fc078070b60 2023-10-16T14:11:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.825+0000 7fc07d163640 1 -- 172.21.15.60:0/2497984511 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0680309b0 con 0x7fc078070b60 2023-10-16T14:11:11.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.825+0000 7fc07d163640 1 -- 172.21.15.60:0/2497984511 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068030d40 con 0x7fc078070b60 2023-10-16T14:11:11.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.825+0000 7fc07f167640 1 -- 172.21.15.60:0/2497984511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 msgr2=0x7fc07815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.825+0000 7fc07f167640 1 --2- 172.21.15.60:0/2497984511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc07815a9b0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fc068009870 tx=0x7fc06802b990 comp rx=0 tx=0).stop 2023-10-16T14:11:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.826+0000 7fc07f167640 1 -- 172.21.15.60:0/2497984511 shutdown_connections 2023-10-16T14:11:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.826+0000 7fc07f167640 1 --2- 172.21.15.60:0/2497984511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc07815a9b0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.826+0000 7fc07f167640 1 -- 172.21.15.60:0/2497984511 >> 172.21.15.60:0/2497984511 conn(0x7fc0780ce6a0 msgr2=0x7fc0780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:11.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.826+0000 7fc07f167640 1 -- 172.21.15.60:0/2497984511 shutdown_connections 2023-10-16T14:11:11.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.826+0000 7fc07f167640 1 -- 172.21.15.60:0/2497984511 wait complete. 2023-10-16T14:11:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.828+0000 7fc07f167640 1 Processor -- start 2023-10-16T14:11:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.829+0000 7fc07f167640 1 -- start start 2023-10-16T14:11:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07f167640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc07806c100 0x7fc07814f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07f167640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc0780ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07f167640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc078159950 0x7fc0780cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07f167640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078150240 con 0x7fc078070b60 2023-10-16T14:11:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07f167640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07815c930 con 0x7fc078159950 2023-10-16T14:11:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07f167640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc078157450 con 0x7fc07806c100 2023-10-16T14:11:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07d964640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc0780ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07d964640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc0780ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59278/0 (socket says 172.21.15.60:59278) 2023-10-16T14:11:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.830+0000 7fc07d964640 1 -- 172.21.15.60:0/3637586072 learned_addr learned my addr 172.21.15.60:0/3637586072 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:11.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07e966640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc078159950 0x7fc0780cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07e165640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc07806c100 0x7fc07814f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07d964640 1 -- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc07806c100 msgr2=0x7fc07814f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07d964640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc07806c100 0x7fc07814f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07d964640 1 -- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc078159950 msgr2=0x7fc0780cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:11.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07d964640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc078159950 0x7fc0780cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:11.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07d964640 1 -- 172.21.15.60:0/3637586072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680094d0 con 0x7fc078070b60 2023-10-16T14:11:11.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07e966640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc078159950 0x7fc0780cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:11.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc07d964640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc0780ccb00 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fc068000c00 tx=0x7fc068004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:11.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.831+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068030820 con 0x7fc078070b60 2023-10-16T14:11:11.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.832+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc068043400 con 0x7fc078070b60 2023-10-16T14:11:11.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.832+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0680437d0 con 0x7fc078070b60 2023-10-16T14:11:11.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.832+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc078156690 con 0x7fc078070b60 2023-10-16T14:11:11.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.832+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc078156c50 con 0x7fc078070b60 2023-10-16T14:11:11.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.833+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07804e9b0 con 0x7fc078070b60 2023-10-16T14:11:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.834+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fc06802e070 con 0x7fc078070b60 2023-10-16T14:11:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.835+0000 7fc06effd640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc05c06b020 0x7fc05c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:11.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.836+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7fc06803e070 con 0x7fc078070b60 2023-10-16T14:11:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.836+0000 7fc07e165640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc05c06b020 0x7fc05c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:11.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.837+0000 7fc07e165640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc05c06b020 0x7fc05c06d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc074000f60 tx=0x7fc074009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:11.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:11.840+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc068077780 con 0x7fc078070b60 2023-10-16T14:11:12.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.040+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fc0780b4620 con 0x7fc078070b60 2023-10-16T14:11:12.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.041+0000 7fc06effd640 1 -- 172.21.15.60:0/3637586072 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v237) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc0680340a0 con 0x7fc078070b60 2023-10-16T14:11:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:11:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc05c06b020 msgr2=0x7fc05c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc05c06b020 0x7fc05c06d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc074000f60 tx=0x7fc074009040 comp rx=0 tx=0).stop 2023-10-16T14:11:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 msgr2=0x7fc0780ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc0780ccb00 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fc068000c00 tx=0x7fc068004730 comp rx=0 tx=0).stop 2023-10-16T14:11:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 shutdown_connections 2023-10-16T14:11:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc05c06b020 0x7fc05c06d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc078159950 0x7fc0780cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc078070b60 0x7fc0780ccb00 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 --2- 172.21.15.60:0/3637586072 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc07806c100 0x7fc07814f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.043+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 >> 172.21.15.60:0/3637586072 conn(0x7fc0780ce6a0 msgr2=0x7fc078070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.044+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 shutdown_connections 2023-10-16T14:11:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.044+0000 7fc07f167640 1 -- 172.21.15.60:0/3637586072 wait complete. 2023-10-16T14:11:12.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:12.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.293+0000 7ff50728a640 1 Processor -- start 2023-10-16T14:11:12.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.294+0000 7ff50728a640 1 -- start start 2023-10-16T14:11:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.295+0000 7ff50728a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff50015aa60 0x7ff5000af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.295+0000 7ff50728a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5000afd60 0x7ff5000cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.295+0000 7ff50728a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 0x7ff5000b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.295+0000 7ff50728a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5000b3730 con 0x7ff5000afd60 2023-10-16T14:11:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.295+0000 7ff50728a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5000b3980 con 0x7ff50015aa60 2023-10-16T14:11:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.295+0000 7ff50728a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000b3c60 con 0x7ff5000cc820 2023-10-16T14:11:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5000afd60 0x7ff5000cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5000afd60 0x7ff5000cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59288/0 (socket says 172.21.15.60:59288) 2023-10-16T14:11:12.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 -- 172.21.15.60:0/1326222189 learned_addr learned my addr 172.21.15.60:0/1326222189 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:12.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 -- 172.21.15.60:0/1326222189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 msgr2=0x7ff5000b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 --2- 172.21.15.60:0/1326222189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 0x7ff5000b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 -- 172.21.15.60:0/1326222189 >> v2:172.21.15.144:3300/0 conn(0x7ff50015aa60 msgr2=0x7ff5000af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 --2- 172.21.15.60:0/1326222189 >> v2:172.21.15.144:3300/0 conn(0x7ff50015aa60 0x7ff5000af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.296+0000 7ff505a87640 1 -- 172.21.15.60:0/1326222189 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5000b3f40 con 0x7ff5000afd60 2023-10-16T14:11:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.297+0000 7ff505a87640 1 --2- 172.21.15.60:0/1326222189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff5000cc2e0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7ff5000ce210 tx=0x7ff4f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd81416cebdb86ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.298+0000 7ff505286640 1 -- 172.21.15.60:0/1326222189 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0031870 con 0x7ff5000afd60 2023-10-16T14:11:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.298+0000 7ff505286640 1 -- 172.21.15.60:0/1326222189 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f00319d0 con 0x7ff5000afd60 2023-10-16T14:11:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.298+0000 7ff505286640 1 -- 172.21.15.60:0/1326222189 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0031d60 con 0x7ff5000afd60 2023-10-16T14:11:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.298+0000 7ff50728a640 1 -- 172.21.15.60:0/1326222189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 msgr2=0x7ff5000cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.298+0000 7ff50728a640 1 --2- 172.21.15.60:0/1326222189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff5000cc2e0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7ff5000ce210 tx=0x7ff4f002b910 comp rx=0 tx=0).stop 2023-10-16T14:11:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 -- 172.21.15.60:0/1326222189 shutdown_connections 2023-10-16T14:11:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 --2- 172.21.15.60:0/1326222189 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 0x7ff5000b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 --2- 172.21.15.60:0/1326222189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff5000cc2e0 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 --2- 172.21.15.60:0/1326222189 >> v2:172.21.15.144:3300/0 conn(0x7ff50015aa60 0x7ff5000af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 -- 172.21.15.60:0/1326222189 >> 172.21.15.60:0/1326222189 conn(0x7ff500065ed0 msgr2=0x7ff5000662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 -- 172.21.15.60:0/1326222189 shutdown_connections 2023-10-16T14:11:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.299+0000 7ff50728a640 1 -- 172.21.15.60:0/1326222189 wait complete. 2023-10-16T14:11:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.301+0000 7ff50728a640 1 Processor -- start 2023-10-16T14:11:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.302+0000 7ff50728a640 1 -- start start 2023-10-16T14:11:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff50728a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff50015bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff50728a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 0x7ff5000b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff50728a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff50015aa60 0x7ff500157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff50728a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50015c430 con 0x7ff5000afd60 2023-10-16T14:11:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff50728a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000b0820 con 0x7ff50015aa60 2023-10-16T14:11:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff50728a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5001575d0 con 0x7ff5000cc820 2023-10-16T14:11:12.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff506288640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff50015bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff506288640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff50015bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59302/0 (socket says 172.21.15.60:59302) 2023-10-16T14:11:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.303+0000 7ff506288640 1 -- 172.21.15.60:0/1285651356 learned_addr learned my addr 172.21.15.60:0/1285651356 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506288640 1 -- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 msgr2=0x7ff5000b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506288640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 0x7ff5000b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506288640 1 -- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff50015aa60 msgr2=0x7ff500157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506a89640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff50015aa60 0x7ff500157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506288640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff50015aa60 0x7ff500157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506288640 1 -- 172.21.15.60:0/1285651356 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f00094d0 con 0x7ff5000afd60 2023-10-16T14:11:12.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff506288640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff50015bef0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b500 tx=0x7ff4fc00b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:12.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.304+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc010870 con 0x7ff5000afd60 2023-10-16T14:11:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.305+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0109d0 con 0x7ff5000afd60 2023-10-16T14:11:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.305+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc010d80 con 0x7ff5000afd60 2023-10-16T14:11:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.305+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5000b2900 con 0x7ff5000afd60 2023-10-16T14:11:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.305+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5000c5be0 con 0x7ff5000afd60 2023-10-16T14:11:12.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.306+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50004e9b0 con 0x7ff5000afd60 2023-10-16T14:11:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.307+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7ff4fc00f3c0 con 0x7ff5000afd60 2023-10-16T14:11:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.308+0000 7ff4f6ffd640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff4e406b0f0 0x7ff4e406d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.308+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7ff4fc089fa0 con 0x7ff5000afd60 2023-10-16T14:11:12.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.309+0000 7ff505a87640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff4e406b0f0 0x7ff4e406d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.311+0000 7ff505a87640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff4e406b0f0 0x7ff4e406d5b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0036000 tx=0x7ff4f0030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:12.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.312+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4fc054730 con 0x7ff5000afd60 2023-10-16T14:11:12.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.545+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5000d0810 con 0x7ff5000afd60 2023-10-16T14:11:12.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.546+0000 7ff4f6ffd640 1 -- 172.21.15.60:0/1285651356 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6153 (secure 0 0 0) 0x7ff4fc014090 con 0x7ff5000afd60 2023-10-16T14:11:12.549 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:12.550 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 118 2023-10-16T14:11:12.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff4e406b0f0 msgr2=0x7ff4e406d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff4e406b0f0 0x7ff4e406d5b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0036000 tx=0x7ff4f0030000 comp rx=0 tx=0).stop 2023-10-16T14:11:12.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 msgr2=0x7ff50015bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff50015bef0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b500 tx=0x7ff4fc00b9c0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 shutdown_connections 2023-10-16T14:11:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff4e406b0f0 0x7ff4e406d5b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5000cc820 0x7ff5000b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff50015aa60 0x7ff500157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 --2- 172.21.15.60:0/1285651356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5000afd60 0x7ff50015bef0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 >> 172.21.15.60:0/1285651356 conn(0x7ff500065ed0 msgr2=0x7ff5000c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 shutdown_connections 2023-10-16T14:11:12.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.549+0000 7ff50728a640 1 -- 172.21.15.60:0/1285651356 wait complete. 2023-10-16T14:11:12.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:12.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.796+0000 7f9dcafa5640 1 Processor -- start 2023-10-16T14:11:12.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.796+0000 7f9dcafa5640 1 -- start start 2023-10-16T14:11:12.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.798+0000 7f9dcafa5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.798+0000 7f9dcafa5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dc415ae80 0x7f9dc40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.798+0000 7f9dcafa5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc415ed60 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.798+0000 7f9dcafa5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc406b810 con 0x7f9dc415ae80 2023-10-16T14:11:12.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.798+0000 7f9dcafa5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40707e0 con 0x7f9dc415aaa0 2023-10-16T14:11:12.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.798+0000 7f9dc3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dc415ae80 0x7f9dc40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dc415ae80 0x7f9dc40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59314/0 (socket says 172.21.15.60:59314) 2023-10-16T14:11:12.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc3fff640 1 -- 172.21.15.60:0/3736393957 learned_addr learned my addr 172.21.15.60:0/3736393957 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:12.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc8d41640 1 --2- 172.21.15.60:0/3736393957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc3fff640 1 -- 172.21.15.60:0/3736393957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 msgr2=0x7f9dc415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:12.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc3fff640 1 --2- 172.21.15.60:0/3736393957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc3fff640 1 -- 172.21.15.60:0/3736393957 >> v1:172.21.15.60:6789/0 conn(0x7f9dc40ccaa0 legacy=0x7f9dc4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.799+0000 7f9dc3fff640 1 -- 172.21.15.60:0/3736393957 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc415ef90 con 0x7f9dc415ae80 2023-10-16T14:11:12.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.800+0000 7f9dc3fff640 1 --2- 172.21.15.60:0/3736393957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc415ae80 0x7f9dc40cc560 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009870 tx=0x7f9db002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd98aca72e924e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:12.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dc37fe640 1 -- 172.21.15.60:0/3736393957 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0030840 con 0x7f9dc415ae80 2023-10-16T14:11:12.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dc37fe640 1 -- 172.21.15.60:0/3736393957 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db00309a0 con 0x7f9dc415ae80 2023-10-16T14:11:12.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dc37fe640 1 -- 172.21.15.60:0/3736393957 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0030d30 con 0x7f9dc415ae80 2023-10-16T14:11:12.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dcafa5640 1 -- 172.21.15.60:0/3736393957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc415ae80 msgr2=0x7f9dc40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/3736393957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc415ae80 0x7f9dc40cc560 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009870 tx=0x7f9db002b980 comp rx=0 tx=0).stop 2023-10-16T14:11:12.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dcafa5640 1 -- 172.21.15.60:0/3736393957 shutdown_connections 2023-10-16T14:11:12.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/3736393957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc415ae80 0x7f9dc40cc560 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/3736393957 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.801+0000 7f9dcafa5640 1 -- 172.21.15.60:0/3736393957 >> 172.21.15.60:0/3736393957 conn(0x7f9dc40cb410 msgr2=0x7f9dc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:12.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.802+0000 7f9dcafa5640 1 -- 172.21.15.60:0/3736393957 shutdown_connections 2023-10-16T14:11:12.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.802+0000 7f9dcafa5640 1 -- 172.21.15.60:0/3736393957 wait complete. 2023-10-16T14:11:12.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.803+0000 7f9dcafa5640 1 Processor -- start 2023-10-16T14:11:12.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.803+0000 7f9dcafa5640 1 -- start start 2023-10-16T14:11:12.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.804+0000 7f9dcafa5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 0x7f9dc40c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.804+0000 7f9dcafa5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc41f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.804+0000 7f9dcafa5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dc415ae80 0x7f9dc41f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.804+0000 7f9dcafa5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4156d90 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.804+0000 7f9dcafa5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4158830 con 0x7f9dc415ae80 2023-10-16T14:11:12.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.804+0000 7f9dcafa5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4156810 con 0x7f9dc415aaa0 2023-10-16T14:11:12.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 0x7f9dc40c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 0x7f9dc40c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59324/0 (socket says 172.21.15.60:59324) 2023-10-16T14:11:12.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 -- 172.21.15.60:0/198160129 learned_addr learned my addr 172.21.15.60:0/198160129 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:12.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc3fff640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc41f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 -- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 msgr2=0x7f9dc41f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc41f3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 -- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dc415ae80 msgr2=0x7f9dc41f5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:12.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dc415ae80 0x7f9dc41f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:12.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 -- 172.21.15.60:0/198160129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db00094d0 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.805+0000 7f9dc8d41640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 0x7f9dc40c34f0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f9db800d800 tx=0x7f9db800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:12.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.806+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800b6c0 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.806+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db800b820 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.806+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800bbb0 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.806+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4158220 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.806+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc41584c0 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.807+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc404e160 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.808+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f9db800bd10 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.809+0000 7f9dc17fa640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9da806b020 0x7f9da806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:12.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.809+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7f9db808a570 con 0x7f9dc40ccaa0 2023-10-16T14:11:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.810+0000 7f9dc3fff640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9da806b020 0x7f9da806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.811+0000 7f9dc3fff640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9da806b020 0x7f9da806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9db0036000 tx=0x7f9db003b040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:12.813+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9db8054d00 con 0x7f9dc40ccaa0 2023-10-16T14:11:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.021+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dc40cdc60 con 0x7f9dc40ccaa0 2023-10-16T14:11:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.024+0000 7f9dc17fa640 1 -- 172.21.15.60:0/198160129 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+21499 (secure 0 0 0) 0x7f9db8058320 con 0x7f9dc40ccaa0 2023-10-16T14:11:13.027 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:13.028 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":237,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:09.018780+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9da806b020 msgr2=0x7f9da806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9da806b020 0x7f9da806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9db0036000 tx=0x7f9db003b040 comp rx=0 tx=0).stop 2023-10-16T14:11:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 msgr2=0x7f9dc40c34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 0x7f9dc40c34f0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f9db800d800 tx=0x7f9db800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 shutdown_connections 2023-10-16T14:11:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9da806b020 0x7f9da806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dc415aaa0 0x7f9dc41f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dc415ae80 0x7f9dc41f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 --2- 172.21.15.60:0/198160129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc40ccaa0 0x7f9dc40c34f0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 >> 172.21.15.60:0/198160129 conn(0x7f9dc40cb410 msgr2=0x7f9dc40b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 shutdown_connections 2023-10-16T14:11:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.028+0000 7f9dcafa5640 1 -- 172.21.15.60:0/198160129 wait complete. 2023-10-16T14:11:13.051 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:11:13.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:13.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.274+0000 7f7dda60e640 1 Processor -- start 2023-10-16T14:11:13.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.274+0000 7f7dda60e640 1 -- start start 2023-10-16T14:11:13.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.276+0000 7f7dda60e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7dd4159f00 0x7f7dd40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.276+0000 7f7dda60e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd40b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.276+0000 7f7dda60e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd406f7f0 con 0x7f7dd4159b20 2023-10-16T14:11:13.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.276+0000 7f7dda60e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd406c5e0 con 0x7f7dd4159f00 2023-10-16T14:11:13.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.276+0000 7f7dda60e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd406c8c0 con 0x7f7dd415d880 2023-10-16T14:11:13.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.276+0000 7f7dcb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7dd4159f00 0x7f7dd40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dcb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7dd4159f00 0x7f7dd40b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59336/0 (socket says 172.21.15.60:59336) 2023-10-16T14:11:13.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2186721006 learned_addr learned my addr 172.21.15.60:0/2186721006 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:13.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dd8bab640 1 --2- 172.21.15.60:0/2186721006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd40b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2186721006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 msgr2=0x7f7dd40b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:13.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dcb7fe640 1 --2- 172.21.15.60:0/2186721006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd40b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2186721006 >> v1:172.21.15.144:6789/0 conn(0x7f7dd4159b20 legacy=0x7f7dd406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.277+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2186721006 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd406cba0 con 0x7f7dd4159f00 2023-10-16T14:11:13.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.278+0000 7f7dcb7fe640 1 --2- 172.21.15.60:0/2186721006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd40b0410 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009870 tx=0x7f7dbc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecca410a71c2f18f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:13.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.279+0000 7f7dd37fe640 1 -- 172.21.15.60:0/2186721006 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc030840 con 0x7f7dd4159f00 2023-10-16T14:11:13.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.279+0000 7f7dd37fe640 1 -- 172.21.15.60:0/2186721006 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc0309a0 con 0x7f7dd4159f00 2023-10-16T14:11:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.279+0000 7f7dd37fe640 1 -- 172.21.15.60:0/2186721006 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc030d30 con 0x7f7dd4159f00 2023-10-16T14:11:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.279+0000 7f7dda60e640 1 -- 172.21.15.60:0/2186721006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 msgr2=0x7f7dd40b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.279+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2186721006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd40b0410 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009870 tx=0x7f7dbc02b980 comp rx=0 tx=0).stop 2023-10-16T14:11:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.279+0000 7f7dda60e640 1 -- 172.21.15.60:0/2186721006 shutdown_connections 2023-10-16T14:11:13.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.280+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2186721006 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd40b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.280+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2186721006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd40b0410 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.280+0000 7f7dda60e640 1 -- 172.21.15.60:0/2186721006 >> 172.21.15.60:0/2186721006 conn(0x7f7dd40cb410 msgr2=0x7f7dd40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:13.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.280+0000 7f7dda60e640 1 -- 172.21.15.60:0/2186721006 shutdown_connections 2023-10-16T14:11:13.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.280+0000 7f7dda60e640 1 -- 172.21.15.60:0/2186721006 wait complete. 2023-10-16T14:11:13.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.282+0000 7f7dda60e640 1 Processor -- start 2023-10-16T14:11:13.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.282+0000 7f7dda60e640 1 -- start start 2023-10-16T14:11:13.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.283+0000 7f7dda60e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd4159b20 0x7f7dd406fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.283+0000 7f7dda60e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd4150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.283+0000 7f7dda60e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd4151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.283+0000 7f7dda60e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4151d30 con 0x7f7dd4159f00 2023-10-16T14:11:13.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.283+0000 7f7dda60e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4152010 con 0x7f7dd4159b20 2023-10-16T14:11:13.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.283+0000 7f7dda60e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd41522f0 con 0x7f7dd415d880 2023-10-16T14:11:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd4150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd4150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59352/0 (socket says 172.21.15.60:59352) 2023-10-16T14:11:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2147569414 learned_addr learned my addr 172.21.15.60:0/2147569414 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dd3fff640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd4159b20 0x7f7dd406fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 msgr2=0x7f7dd4151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dd8bab640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd4151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd4151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd4159b20 msgr2=0x7f7dd406fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd4159b20 0x7f7dd406fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.284+0000 7f7dcb7fe640 1 -- 172.21.15.60:0/2147569414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0094d0 con 0x7f7dd4159f00 2023-10-16T14:11:13.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.285+0000 7f7dcb7fe640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd4150fc0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc000c00 tx=0x7f7dbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:13.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.285+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc039c90 con 0x7f7dd4159f00 2023-10-16T14:11:13.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.285+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc039df0 con 0x7f7dd4159f00 2023-10-16T14:11:13.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.285+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd41525d0 con 0x7f7dd4159f00 2023-10-16T14:11:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.285+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc043640 con 0x7f7dd4159f00 2023-10-16T14:11:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.285+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd4151a40 con 0x7f7dd4159f00 2023-10-16T14:11:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.287+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd406c270 con 0x7f7dd4159f00 2023-10-16T14:11:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.287+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7dbc0437a0 con 0x7f7dd4159f00 2023-10-16T14:11:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.289+0000 7f7dd17fa640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0b0 0x7f7db806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.289+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7f7dbc03e070 con 0x7f7dd4159f00 2023-10-16T14:11:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.290+0000 7f7dd3fff640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0b0 0x7f7db806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.291+0000 7f7dd3fff640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0b0 0x7f7db806d570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4000f60 tx=0x7f7dc4009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.293+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7dbc077880 con 0x7f7dd4159f00 2023-10-16T14:11:13.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.526+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dd404e160 con 0x7f7dd4159f00 2023-10-16T14:11:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.527+0000 7f7dd17fa640 1 -- 172.21.15.60:0/2147569414 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6153 (secure 0 0 0) 0x7f7dbc07aea0 con 0x7f7dd4159f00 2023-10-16T14:11:13.530 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:13.531 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 118 2023-10-16T14:11:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0b0 msgr2=0x7f7db806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0b0 0x7f7db806d570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4000f60 tx=0x7f7dc4009040 comp rx=0 tx=0).stop 2023-10-16T14:11:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 msgr2=0x7f7dd4150fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd4150fc0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc000c00 tx=0x7f7dbc004730 comp rx=0 tx=0).stop 2023-10-16T14:11:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 shutdown_connections 2023-10-16T14:11:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0b0 0x7f7db806d570 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd415d880 0x7f7dd4151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4159f00 0x7f7dd4150fc0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 --2- 172.21.15.60:0/2147569414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd4159b20 0x7f7dd406fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 >> 172.21.15.60:0/2147569414 conn(0x7f7dd40cb410 msgr2=0x7f7dd40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 shutdown_connections 2023-10-16T14:11:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.530+0000 7f7dda60e640 1 -- 172.21.15.60:0/2147569414 wait complete. 2023-10-16T14:11:13.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:13.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.781+0000 7f8157c90640 1 Processor -- start 2023-10-16T14:11:13.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.781+0000 7f8157c90640 1 -- start start 2023-10-16T14:11:13.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.782+0000 7f8157c90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f815015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.782+0000 7f8157c90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815015ed60 con 0x7f815015aaa0 2023-10-16T14:11:13.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.782+0000 7f8157c90640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81500707e0 con 0x7f81500ce390 2023-10-16T14:11:13.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.782+0000 7f8157c90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815006b810 con 0x7f815015ae80 2023-10-16T14:11:13.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.783+0000 7f8155a2c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f815015aaa0 0x7f81500cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41008/0 (socket says 172.21.15.60:41008) 2023-10-16T14:11:13.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.783+0000 7f8155a2c640 1 -- 172.21.15.60:0/132513064 learned_addr learned my addr 172.21.15.60:0/132513064 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:13.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.783+0000 7f815522b640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f815015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.783+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954701912 0 0) 0x7f815015ed60 con 0x7f815015aaa0 2023-10-16T14:11:13.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.783+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f813c003520 con 0x7f815015aaa0 2023-10-16T14:11:13.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913322138 0 0) 0x7f813c003520 con 0x7f815015aaa0 2023-10-16T14:11:13.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 msgr2=0x7f815015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:13.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f815015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 >> v1:172.21.15.144:6789/0 conn(0x7f81500ce390 legacy=0x7f8150070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815015ef90 con 0x7f815015aaa0 2023-10-16T14:11:13.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f814c0019f0 con 0x7f815015aaa0 2023-10-16T14:11:13.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 >> v1:172.21.15.60:6789/0 conn(0x7f815015aaa0 legacy=0x7f81500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0039a0 0x7f813c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f813c0047c0 0x7f813c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f813c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815006b810 con 0x7f813c0047c0 2023-10-16T14:11:13.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c00ba60 con 0x7f813c0039a0 2023-10-16T14:11:13.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c00bc20 con 0x7f815015ae80 2023-10-16T14:11:13.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.784+0000 7f815522b640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f813c0047c0 0x7f813c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f8155a2c640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0039a0 0x7f813c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815622d640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f813c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815522b640 1 -- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 msgr2=0x7f813c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:13.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815522b640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f813c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815522b640 1 -- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0039a0 msgr2=0x7f813c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:13.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815522b640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0039a0 0x7f813c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815522b640 1 -- 172.21.15.60:0/132513064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c003520 con 0x7f813c0047c0 2023-10-16T14:11:13.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f8155a2c640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0039a0 0x7f813c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:13.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.785+0000 7f815622d640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f815015ae80 0x7f813c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:13.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.786+0000 7f815522b640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f813c0047c0 0x7f813c004c40 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f8140009a90 tx=0x7f813c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bff7724b1d1729a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:13.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.786+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814002c660 con 0x7f813c0047c0 2023-10-16T14:11:13.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.786+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814002c7c0 con 0x7f813c0047c0 2023-10-16T14:11:13.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.786+0000 7f8154a2a640 1 -- 172.21.15.60:0/132513064 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814002cb50 con 0x7f813c0047c0 2023-10-16T14:11:13.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f8157c90640 1 -- 172.21.15.60:0/132513064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f813c0047c0 msgr2=0x7f813c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f8157c90640 1 --2- 172.21.15.60:0/132513064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f813c0047c0 0x7f813c004c40 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f8140009a90 tx=0x7f813c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f815622d640 1 -- 172.21.15.60:0/132513064 reap_dead start 2023-10-16T14:11:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f8157c90640 1 -- 172.21.15.60:0/132513064 shutdown_connections 2023-10-16T14:11:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f8157c90640 1 -- 172.21.15.60:0/132513064 >> 172.21.15.60:0/132513064 conn(0x7f81500cb410 msgr2=0x7f81500cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f8157c90640 1 -- 172.21.15.60:0/132513064 shutdown_connections 2023-10-16T14:11:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.787+0000 7f8157c90640 1 -- 172.21.15.60:0/132513064 wait complete. 2023-10-16T14:11:13.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.789+0000 7f8157c90640 1 Processor -- start 2023-10-16T14:11:13.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.789+0000 7f8157c90640 1 -- start start 2023-10-16T14:11:13.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.790+0000 7f8157c90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0047c0 0x7f815015dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.790+0000 7f8157c90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81500c5a60 0x7f81501f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.790+0000 7f8157c90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 0x7f81501f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.790+0000 7f8157c90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815015e190 con 0x7f81501f5a90 2023-10-16T14:11:13.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.790+0000 7f8157c90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81501f6450 con 0x7f813c0047c0 2023-10-16T14:11:13.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.790+0000 7f8157c90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815015b320 con 0x7f81500c5a60 2023-10-16T14:11:13.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 0x7f81501f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 0x7f81501f5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59372/0 (socket says 172.21.15.60:59372) 2023-10-16T14:11:13.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 -- 172.21.15.60:0/1504590619 learned_addr learned my addr 172.21.15.60:0/1504590619 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:13.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f8155a2c640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0047c0 0x7f815015dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815522b640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81500c5a60 0x7f81501f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 -- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81500c5a60 msgr2=0x7f81501f1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81500c5a60 0x7f81501f1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 -- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0047c0 msgr2=0x7f815015dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:13.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0047c0 0x7f815015dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:13.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 -- 172.21.15.60:0/1504590619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815015b600 con 0x7f81501f5a90 2023-10-16T14:11:13.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.791+0000 7f815622d640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 0x7f81501f5f10 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f81500ce2e0 tx=0x7f814800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:13.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.792+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814800b070 con 0x7f81501f5a90 2023-10-16T14:11:13.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.792+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81480054a0 con 0x7f81501f5a90 2023-10-16T14:11:13.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.792+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815015b8e0 con 0x7f81501f5a90 2023-10-16T14:11:13.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.792+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148007270 con 0x7f81501f5a90 2023-10-16T14:11:13.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.792+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815015bea0 con 0x7f81501f5a90 2023-10-16T14:11:13.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.793+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815004e160 con 0x7f81501f5a90 2023-10-16T14:11:13.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.794+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f81480073d0 con 0x7f81501f5a90 2023-10-16T14:11:13.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.795+0000 7f81467fc640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f813406b0e0 0x7f813406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:13.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.795+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7f8148089c60 con 0x7f81501f5a90 2023-10-16T14:11:13.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.796+0000 7f8155a2c640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f813406b0e0 0x7f813406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:13.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.797+0000 7f8155a2c640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f813406b0e0 0x7f813406d5a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f814c002230 tx=0x7f814c009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:13.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:13.799+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8148054170 con 0x7f81501f5a90 2023-10-16T14:11:14.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.037+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f815015f520 con 0x7f81501f5a90 2023-10-16T14:11:14.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.038+0000 7f81467fc640 1 -- 172.21.15.60:0/1504590619 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6153 (secure 0 0 0) 0x7f8148057790 con 0x7f81501f5a90 2023-10-16T14:11:14.041 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:14.041 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":118,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:26.270730+0000","modified":"2023-10-16T14:10:29.414557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11004},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11004":{"gid":11004,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/2009359724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":2009359724},{"type":"v1","addr":"172.21.15.144:6837","nonce":2009359724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 118 2023-10-16T14:11:14.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f813406b0e0 msgr2=0x7f813406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f813406b0e0 0x7f813406d5a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f814c002230 tx=0x7f814c009040 comp rx=0 tx=0).stop 2023-10-16T14:11:14.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 msgr2=0x7f81501f5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 0x7f81501f5f10 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f81500ce2e0 tx=0x7f814800d890 comp rx=0 tx=0).stop 2023-10-16T14:11:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 shutdown_connections 2023-10-16T14:11:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f813406b0e0 0x7f813406d5a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f81500c5a60 0x7f81501f1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81501f5a90 0x7f81501f5f10 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 --2- 172.21.15.60:0/1504590619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f813c0047c0 0x7f815015dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 >> 172.21.15.60:0/1504590619 conn(0x7f81500cb410 msgr2=0x7f81500b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:14.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 shutdown_connections 2023-10-16T14:11:14.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.041+0000 7f8157c90640 1 -- 172.21.15.60:0/1504590619 wait complete. 2023-10-16T14:11:14.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.278+0000 7ff784b16640 1 Processor -- start 2023-10-16T14:11:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.278+0000 7ff784b16640 1 -- start start 2023-10-16T14:11:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff784b16640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff780159950 0x7ff78015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff784b16640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78015aef0 con 0x7ff78006c810 2023-10-16T14:11:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff784b16640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78015d910 con 0x7ff780070450 2023-10-16T14:11:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff784b16640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78015e490 con 0x7ff780159950 2023-10-16T14:11:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff77f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff780159950 0x7ff78015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff77f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff780159950 0x7ff78015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59376/0 (socket says 172.21.15.60:59376) 2023-10-16T14:11:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.280+0000 7ff77f7fe640 1 -- 172.21.15.60:0/1097950170 learned_addr learned my addr 172.21.15.60:0/1097950170 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:14.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.281+0000 7ff77f7fe640 1 -- 172.21.15.60:0/1097950170 >> v1:172.21.15.144:6789/0 conn(0x7ff780070450 legacy=0x7ff78006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.281+0000 7ff77f7fe640 1 -- 172.21.15.60:0/1097950170 >> v1:172.21.15.60:6789/0 conn(0x7ff78006c810 legacy=0x7ff7800c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.281+0000 7ff77f7fe640 1 -- 172.21.15.60:0/1097950170 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78015dbf0 con 0x7ff780159950 2023-10-16T14:11:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.282+0000 7ff77f7fe640 1 --2- 172.21.15.60:0/1097950170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff78015a9b0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7ff770009870 tx=0x7ff77002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72dcd2303367e22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.282+0000 7ff77effd640 1 -- 172.21.15.60:0/1097950170 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff770031860 con 0x7ff780159950 2023-10-16T14:11:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.282+0000 7ff77effd640 1 -- 172.21.15.60:0/1097950170 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7700319c0 con 0x7ff780159950 2023-10-16T14:11:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.282+0000 7ff77effd640 1 -- 172.21.15.60:0/1097950170 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff770031d50 con 0x7ff780159950 2023-10-16T14:11:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 -- 172.21.15.60:0/1097950170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 msgr2=0x7ff78015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 --2- 172.21.15.60:0/1097950170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff78015a9b0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7ff770009870 tx=0x7ff77002b900 comp rx=0 tx=0).stop 2023-10-16T14:11:14.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 -- 172.21.15.60:0/1097950170 shutdown_connections 2023-10-16T14:11:14.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 --2- 172.21.15.60:0/1097950170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff78015a9b0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 -- 172.21.15.60:0/1097950170 >> 172.21.15.60:0/1097950170 conn(0x7ff7800ce6a0 msgr2=0x7ff7800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:14.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 -- 172.21.15.60:0/1097950170 shutdown_connections 2023-10-16T14:11:14.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.283+0000 7ff784b16640 1 -- 172.21.15.60:0/1097950170 wait complete. 2023-10-16T14:11:14.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.285+0000 7ff784b16640 1 Processor -- start 2023-10-16T14:11:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.285+0000 7ff784b16640 1 -- start start 2023-10-16T14:11:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff784b16640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff78006c810 0x7ff7801f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff784b16640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff780070450 0x7ff7800aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff784b16640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff7800a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff784b16640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7800c3f20 con 0x7ff780159950 2023-10-16T14:11:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff784b16640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7801f28d0 con 0x7ff780070450 2023-10-16T14:11:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff784b16640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff780158d70 con 0x7ff78006c810 2023-10-16T14:11:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.286+0000 7ff77ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff7800a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff7800a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59378/0 (socket says 172.21.15.60:59378) 2023-10-16T14:11:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 -- 172.21.15.60:0/2531700459 learned_addr learned my addr 172.21.15.60:0/2531700459 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77f7fe640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff78006c810 0x7ff7801f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 -- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff78006c810 msgr2=0x7ff7801f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff78006c810 0x7ff7801f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 -- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff780070450 msgr2=0x7ff7800aa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff780070450 0x7ff7800aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 -- 172.21.15.60:0/2531700459 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7700094d0 con 0x7ff780159950 2023-10-16T14:11:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77ffff640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff7800a8480 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c7a0 tx=0x7ff77800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.287+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778003b70 con 0x7ff780159950 2023-10-16T14:11:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.288+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778003cf0 con 0x7ff780159950 2023-10-16T14:11:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.288+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77800aa90 con 0x7ff780159950 2023-10-16T14:11:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.288+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7800aa900 con 0x7ff780159950 2023-10-16T14:11:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.288+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7800c3580 con 0x7ff780159950 2023-10-16T14:11:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.289+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7800c4060 con 0x7ff780159950 2023-10-16T14:11:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.290+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff77800abf0 con 0x7ff780159950 2023-10-16T14:11:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.291+0000 7ff77cff9640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff76006b0b0 0x7ff76006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.291+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7ff778089d80 con 0x7ff780159950 2023-10-16T14:11:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.292+0000 7ff77f7fe640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff76006b0b0 0x7ff76006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.293+0000 7ff77f7fe640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff76006b0b0 0x7ff76006d570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff770036000 tx=0x7ff77003c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.296+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff7800c4060 con 0x7ff780159950 2023-10-16T14:11:14.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.492+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff78015aef0 con 0x7ff780159950 2023-10-16T14:11:14.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.496+0000 7ff77cff9640 1 -- 172.21.15.60:0/2531700459 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+21499 (secure 0 0 0) 0x7ff78015aef0 con 0x7ff780159950 2023-10-16T14:11:14.499 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:14.499 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":237,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:09.018780+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:10:24.220406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-16T14:10:25.316288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:14.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff76006b0b0 msgr2=0x7ff76006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff76006b0b0 0x7ff76006d570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff770036000 tx=0x7ff77003c040 comp rx=0 tx=0).stop 2023-10-16T14:11:14.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 msgr2=0x7ff7800a8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff7800a8480 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7ff77800c7a0 tx=0x7ff77800cc60 comp rx=0 tx=0).stop 2023-10-16T14:11:14.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 shutdown_connections 2023-10-16T14:11:14.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff76006b0b0 0x7ff76006d570 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff780159950 0x7ff7800a8480 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff780070450 0x7ff7800aa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 --2- 172.21.15.60:0/2531700459 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff78006c810 0x7ff7801f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 >> 172.21.15.60:0/2531700459 conn(0x7ff7800ce6a0 msgr2=0x7ff78015d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:14.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 shutdown_connections 2023-10-16T14:11:14.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.499+0000 7ff784b16640 1 -- 172.21.15.60:0/2531700459 wait complete. 2023-10-16T14:11:14.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:11:14.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.742+0000 7fd4133d2640 1 Processor -- start 2023-10-16T14:11:14.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.742+0000 7fd4133d2640 1 -- start start 2023-10-16T14:11:14.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd4133d2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd40c06c040 0x7fd40c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd4133d2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd40c0cda80 0x7fd40c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd4133d2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c06c960 con 0x7fd40c0ad970 2023-10-16T14:11:14.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd4133d2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c15b9d0 con 0x7fd40c0cda80 2023-10-16T14:11:14.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd4133d2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c15bcb0 con 0x7fd40c06c040 2023-10-16T14:11:14.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd412bd1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd40c0cda80 0x7fd40c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd412bd1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd40c0cda80 0x7fd40c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59384/0 (socket says 172.21.15.60:59384) 2023-10-16T14:11:14.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd412bd1640 1 -- 172.21.15.60:0/261668725 learned_addr learned my addr 172.21.15.60:0/261668725 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:14.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.744+0000 7fd4123d0640 1 --2- 172.21.15.60:0/261668725 >> v2:172.21.15.144:3300/0 conn(0x7fd40c06c040 0x7fd40c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.745+0000 7fd412bd1640 1 -- 172.21.15.60:0/261668725 >> v2:172.21.15.144:3300/0 conn(0x7fd40c06c040 msgr2=0x7fd40c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:14.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.745+0000 7fd412bd1640 1 --2- 172.21.15.60:0/261668725 >> v2:172.21.15.144:3300/0 conn(0x7fd40c06c040 0x7fd40c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.745+0000 7fd412bd1640 1 -- 172.21.15.60:0/261668725 >> v1:172.21.15.144:6789/0 conn(0x7fd40c0ad970 legacy=0x7fd40c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:14.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.745+0000 7fd412bd1640 1 -- 172.21.15.60:0/261668725 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c15bf90 con 0x7fd40c0cda80 2023-10-16T14:11:14.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.746+0000 7fd412bd1640 1 --2- 172.21.15.60:0/261668725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0cda80 0x7fd40c15ed90 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0cf350 tx=0x7fd40402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b838bbca49a9e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:14.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.746+0000 7fd4113ce640 1 -- 172.21.15.60:0/261668725 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404031870 con 0x7fd40c0cda80 2023-10-16T14:11:14.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.746+0000 7fd4113ce640 1 -- 172.21.15.60:0/261668725 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4040319d0 con 0x7fd40c0cda80 2023-10-16T14:11:14.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.746+0000 7fd4113ce640 1 -- 172.21.15.60:0/261668725 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404031d60 con 0x7fd40c0cda80 2023-10-16T14:11:14.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 -- 172.21.15.60:0/261668725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0cda80 msgr2=0x7fd40c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 --2- 172.21.15.60:0/261668725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0cda80 0x7fd40c15ed90 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0cf350 tx=0x7fd40402b910 comp rx=0 tx=0).stop 2023-10-16T14:11:14.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 -- 172.21.15.60:0/261668725 shutdown_connections 2023-10-16T14:11:14.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 --2- 172.21.15.60:0/261668725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0cda80 0x7fd40c15ed90 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 --2- 172.21.15.60:0/261668725 >> v2:172.21.15.144:3300/0 conn(0x7fd40c06c040 0x7fd40c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 -- 172.21.15.60:0/261668725 >> 172.21.15.60:0/261668725 conn(0x7fd40c0ce6a0 msgr2=0x7fd40c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:14.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 -- 172.21.15.60:0/261668725 shutdown_connections 2023-10-16T14:11:14.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.747+0000 7fd4133d2640 1 -- 172.21.15.60:0/261668725 wait complete. 2023-10-16T14:11:14.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.749+0000 7fd4133d2640 1 Processor -- start 2023-10-16T14:11:14.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.749+0000 7fd4133d2640 1 -- start start 2023-10-16T14:11:14.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd4133d2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd40c06c040 0x7fd40c14dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd4133d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 0x7fd40c154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd4133d2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd40c0cda80 0x7fd40c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd4133d2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c155eb0 con 0x7fd40c0ad970 2023-10-16T14:11:14.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd4133d2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c15cbe0 con 0x7fd40c0cda80 2023-10-16T14:11:14.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd4133d2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c1558f0 con 0x7fd40c06c040 2023-10-16T14:11:14.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd411bcf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 0x7fd40c154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd411bcf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 0x7fd40c154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59386/0 (socket says 172.21.15.60:59386) 2023-10-16T14:11:14.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.750+0000 7fd411bcf640 1 -- 172.21.15.60:0/3615318864 learned_addr learned my addr 172.21.15.60:0/3615318864 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:14.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd4123d0640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd40c06c040 0x7fd40c14dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd412bd1640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd40c0cda80 0x7fd40c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd411bcf640 1 -- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd40c06c040 msgr2=0x7fd40c14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd411bcf640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd40c06c040 0x7fd40c14dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd411bcf640 1 -- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd40c0cda80 msgr2=0x7fd40c1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd411bcf640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd40c0cda80 0x7fd40c1552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd411bcf640 1 -- 172.21.15.60:0/3615318864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4040094d0 con 0x7fd40c0ad970 2023-10-16T14:11:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd411bcf640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 0x7fd40c154da0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c400 tx=0x7fd3fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.751+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc011790 con 0x7fd40c0ad970 2023-10-16T14:11:14.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.752+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc011930 con 0x7fd40c0ad970 2023-10-16T14:11:14.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.752+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc011ce0 con 0x7fd40c0ad970 2023-10-16T14:11:14.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.752+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c155bd0 con 0x7fd40c0ad970 2023-10-16T14:11:14.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.752+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c1f1760 con 0x7fd40c0ad970 2023-10-16T14:11:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.753+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c06b330 con 0x7fd40c0ad970 2023-10-16T14:11:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.754+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd3fc01c460 con 0x7fd40c0ad970 2023-10-16T14:11:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.755+0000 7fd402ffd640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd3f006b0b0 0x7fd3f006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.755+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7390+0+0 (secure 0 0 0) 0x7fd3fc089fe0 con 0x7fd40c0ad970 2023-10-16T14:11:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.756+0000 7fd4123d0640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd3f006b0b0 0x7fd3f006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:14.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.757+0000 7fd4123d0640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd3f006b0b0 0x7fd3f006d570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd40c157520 tx=0x7fd4080039b0 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:14.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.759+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3fc054710 con 0x7fd40c0ad970 2023-10-16T14:11:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.994+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd40c1f12c0 con 0x7fd40c0ad970 2023-10-16T14:11:14.998 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:11:14.996+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:11:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:14.998+0000 7fd402ffd640 1 -- 172.21.15.60:0/3615318864 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v119) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd3fc057d30 con 0x7fd40c0ad970 2023-10-16T14:11:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:11:15.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd3f006b0b0 msgr2=0x7fd3f006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd3f006b0b0 0x7fd3f006d570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd40c157520 tx=0x7fd4080039b0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 msgr2=0x7fd40c154da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 0x7fd40c154da0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c400 tx=0x7fd3fc00c8c0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 shutdown_connections 2023-10-16T14:11:15.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd3f006b0b0 0x7fd3f006d570 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd40c0cda80 0x7fd40c1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c0ad970 0x7fd40c154da0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 --2- 172.21.15.60:0/3615318864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd40c06c040 0x7fd40c14dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.002+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 >> 172.21.15.60:0/3615318864 conn(0x7fd40c0ce6a0 msgr2=0x7fd40c15e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:15.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.003+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 shutdown_connections 2023-10-16T14:11:15.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.003+0000 7fd4133d2640 1 -- 172.21.15.60:0/3615318864 wait complete. 2023-10-16T14:11:15.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:11:15.036 INFO:tasks.ceph.mds.b.smithi144.stderr: -4792> 2023-10-16T14:10:32.769+0000 7f181bba1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:11:15.128 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:11:15.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.241+0000 7f5962090640 1 Processor -- start 2023-10-16T14:11:15.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.241+0000 7f5962090640 1 -- start start 2023-10-16T14:11:15.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.243+0000 7f5962090640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f595c160530 0x7f595c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:15.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.243+0000 7f5962090640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595c0ac890 con 0x7f595c069fa0 2023-10-16T14:11:15.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.243+0000 7f5962090640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595c0ae800 con 0x7f595c065ab0 2023-10-16T14:11:15.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.243+0000 7f5962090640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c0ae1f0 con 0x7f595c160530 2023-10-16T14:11:15.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.244+0000 7f595b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f595c160530 0x7f595c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:15.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.244+0000 7f595bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f595c069fa0 0x7f595c06a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41022/0 (socket says 172.21.15.60:41022) 2023-10-16T14:11:15.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.244+0000 7f595bfff640 1 -- 172.21.15.60:0/1753583143 learned_addr learned my addr 172.21.15.60:0/1753583143 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:15.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.244+0000 7f595b7fe640 1 -- 172.21.15.60:0/1753583143 >> v1:172.21.15.144:6789/0 conn(0x7f595c065ab0 legacy=0x7f595c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.244+0000 7f595b7fe640 1 -- 172.21.15.60:0/1753583143 >> v1:172.21.15.60:6789/0 conn(0x7f595c069fa0 legacy=0x7f595c06a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.244+0000 7f595b7fe640 1 -- 172.21.15.60:0/1753583143 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c0acae0 con 0x7f595c160530 2023-10-16T14:11:15.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.245+0000 7f595b7fe640 1 --2- 172.21.15.60:0/1753583143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c1609f0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f594c0097a0 tx=0x7f594c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28eb28b2a9cdc85e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:15.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f595affd640 1 -- 172.21.15.60:0/1753583143 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c031830 con 0x7f595c160530 2023-10-16T14:11:15.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f595affd640 1 -- 172.21.15.60:0/1753583143 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c031990 con 0x7f595c160530 2023-10-16T14:11:15.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f595affd640 1 -- 172.21.15.60:0/1753583143 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c031d20 con 0x7f595c160530 2023-10-16T14:11:15.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f5962090640 1 -- 172.21.15.60:0/1753583143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 msgr2=0x7f595c1609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f5962090640 1 --2- 172.21.15.60:0/1753583143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c1609f0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f594c0097a0 tx=0x7f594c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f5962090640 1 -- 172.21.15.60:0/1753583143 shutdown_connections 2023-10-16T14:11:15.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f5962090640 1 --2- 172.21.15.60:0/1753583143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c1609f0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.246+0000 7f5962090640 1 -- 172.21.15.60:0/1753583143 >> 172.21.15.60:0/1753583143 conn(0x7f595c06bc10 msgr2=0x7f595c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:15.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.247+0000 7f5962090640 1 -- 172.21.15.60:0/1753583143 shutdown_connections 2023-10-16T14:11:15.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.247+0000 7f5962090640 1 -- 172.21.15.60:0/1753583143 wait complete. 2023-10-16T14:11:15.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.248+0000 7f5962090640 1 Processor -- start 2023-10-16T14:11:15.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.249+0000 7f5962090640 1 -- start start 2023-10-16T14:11:15.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.249+0000 7f5962090640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f595c065ab0 0x7f595c14f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:15.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.249+0000 7f5962090640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f595c069fa0 0x7f595c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:15.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.249+0000 7f5962090640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:15.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.249+0000 7f5962090640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c14fc00 con 0x7f595c160530 2023-10-16T14:11:15.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.249+0000 7f5962090640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c15c270 con 0x7f595c065ab0 2023-10-16T14:11:15.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f5962090640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c15d0d0 con 0x7f595c069fa0 2023-10-16T14:11:15.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c0b2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59402/0 (socket says 172.21.15.60:59402) 2023-10-16T14:11:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 -- 172.21.15.60:0/3067066638 learned_addr learned my addr 172.21.15.60:0/3067066638 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595b7fe640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f595c065ab0 0x7f595c14f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f5953fff640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f595c069fa0 0x7f595c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 -- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f595c069fa0 msgr2=0x7f595c0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f595c069fa0 0x7f595c0b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 -- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f595c065ab0 msgr2=0x7f595c14f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:15.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f595c065ab0 0x7f595c14f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:15.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.250+0000 7f595bfff640 1 -- 172.21.15.60:0/3067066638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c009400 con 0x7f595c160530 2023-10-16T14:11:15.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.251+0000 7f595bfff640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c0b2090 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f595400cbf0 tx=0x7f595400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.251+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595400ad40 con 0x7f595c160530 2023-10-16T14:11:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.251+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954012400 con 0x7f595c160530 2023-10-16T14:11:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.251+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595400aea0 con 0x7f595c160530 2023-10-16T14:11:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.251+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c159bd0 con 0x7f595c160530 2023-10-16T14:11:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.251+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c15c7a0 con 0x7f595c160530 2023-10-16T14:11:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.252+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c04e160 con 0x7f595c160530 2023-10-16T14:11:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.253+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5954007070 con 0x7f595c160530 2023-10-16T14:11:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.254+0000 7f5958ff9640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.254+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7476+0+0 (secure 0 0 0) 0x7f595408a010 con 0x7f595c160530 2023-10-16T14:11:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.255+0000 7f595b7fe640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.256+0000 7f595b7fe640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f593c06b0b0 0x7f593c06d570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f594c036000 tx=0x7f594c03c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:15.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.258+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59540546a0 con 0x7f595c160530 2023-10-16T14:11:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:15.494+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f595c15d210 con 0x7f595c160530 2023-10-16T14:11:16.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.087+0000 7f5958ff9640 1 -- 172.21.15.60:0/3067066638 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v120) v1 ==== 104+0+0 (secure 0 0 0) 0x7f59540170a0 con 0x7f595c160530 2023-10-16T14:11:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f593c06b0b0 msgr2=0x7f593c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f593c06b0b0 0x7f593c06d570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f594c036000 tx=0x7f594c03c040 comp rx=0 tx=0).stop 2023-10-16T14:11:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 msgr2=0x7f595c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c0b2090 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f595400cbf0 tx=0x7f595400a430 comp rx=0 tx=0).stop 2023-10-16T14:11:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 shutdown_connections 2023-10-16T14:11:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f593c06b0b0 0x7f593c06d570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f595c069fa0 0x7f595c0b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f595c160530 0x7f595c0b2090 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 --2- 172.21.15.60:0/3067066638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f595c065ab0 0x7f595c14f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 >> 172.21.15.60:0/3067066638 conn(0x7f595c06bc10 msgr2=0x7f595c06d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 shutdown_connections 2023-10-16T14:11:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.091+0000 7f5962090640 1 -- 172.21.15.60:0/3067066638 wait complete. 2023-10-16T14:11:16.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:11:16.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.345+0000 7f67a636c640 1 Processor -- start 2023-10-16T14:11:16.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.345+0000 7f67a636c640 1 -- start start 2023-10-16T14:11:16.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.347+0000 7f67a636c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67a00b0360 0x7f67a00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.347+0000 7f67a636c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a015b310 con 0x7f67a00ca510 2023-10-16T14:11:16.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.347+0000 7f67a636c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a006f640 con 0x7f67a015ac10 2023-10-16T14:11:16.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.347+0000 7f67a636c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a015f800 con 0x7f67a00b0360 2023-10-16T14:11:16.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f67a4909640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f67a00ca510 0x7f67a00ca920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41032/0 (socket says 172.21.15.60:41032) 2023-10-16T14:11:16.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f67a4909640 1 -- 172.21.15.60:0/1508835679 learned_addr learned my addr 172.21.15.60:0/1508835679 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:16.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f6797fff640 1 --2- 172.21.15.60:0/1508835679 >> v2:172.21.15.144:3300/0 conn(0x7f67a00b0360 0x7f67a00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353510170 0 0) 0x7f67a015b310 con 0x7f67a00ca510 2023-10-16T14:11:16.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678c003520 con 0x7f67a00ca510 2023-10-16T14:11:16.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3586024389 0 0) 0x7f678c003520 con 0x7f67a00ca510 2023-10-16T14:11:16.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 >> v2:172.21.15.144:3300/0 conn(0x7f67a00b0360 msgr2=0x7f67a00b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:16.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f679f7fe640 1 --2- 172.21.15.60:0/1508835679 >> v2:172.21.15.144:3300/0 conn(0x7f67a00b0360 0x7f67a00b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.348+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 >> v1:172.21.15.144:6789/0 conn(0x7f67a015ac10 legacy=0x7f67a006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a015b5b0 con 0x7f67a00ca510 2023-10-16T14:11:16.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f67980019f0 con 0x7f67a00ca510 2023-10-16T14:11:16.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 >> v1:172.21.15.60:6789/0 conn(0x7f67a00ca510 legacy=0x7f67a00ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c0039e0 0x7f678c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0047c0 0x7f678c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0051f0 0x7f678c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a015f800 con 0x7f678c0039e0 2023-10-16T14:11:16.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c00c150 con 0x7f678c0051f0 2023-10-16T14:11:16.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c00c310 con 0x7f678c0047c0 2023-10-16T14:11:16.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f679ffff640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c0039e0 0x7f678c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f6797fff640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0047c0 0x7f678c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.349+0000 7f67a4909640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0051f0 0x7f678c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.350+0000 7f679ffff640 1 -- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0047c0 msgr2=0x7f678c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:16.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.350+0000 7f679ffff640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f678c0047c0 0x7f678c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.350+0000 7f679ffff640 1 -- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0051f0 msgr2=0x7f678c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:16.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.350+0000 7f679ffff640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0051f0 0x7f678c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.350+0000 7f679ffff640 1 -- 172.21.15.60:0/1508835679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c003520 con 0x7f678c0039e0 2023-10-16T14:11:16.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.350+0000 7f67a4909640 1 -- 172.21.15.60:0/1508835679 reap_dead start 2023-10-16T14:11:16.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.351+0000 7f679ffff640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c0039e0 0x7f678c003e00 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f6790001750 tx=0x7f678c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990a63398dc983a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:16.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.351+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67900052d0 con 0x7f678c0039e0 2023-10-16T14:11:16.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.351+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679002e050 con 0x7f678c0039e0 2023-10-16T14:11:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.351+0000 7f679f7fe640 1 -- 172.21.15.60:0/1508835679 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790003830 con 0x7f678c0039e0 2023-10-16T14:11:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 -- 172.21.15.60:0/1508835679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c0039e0 msgr2=0x7f678c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c0039e0 0x7f678c003e00 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f6790001750 tx=0x7f678c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 -- 172.21.15.60:0/1508835679 shutdown_connections 2023-10-16T14:11:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 --2- 172.21.15.60:0/1508835679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f678c0039e0 0x7f678c003e00 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 -- 172.21.15.60:0/1508835679 >> 172.21.15.60:0/1508835679 conn(0x7f67a00cb360 msgr2=0x7f67a00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 -- 172.21.15.60:0/1508835679 shutdown_connections 2023-10-16T14:11:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.352+0000 7f67a636c640 1 -- 172.21.15.60:0/1508835679 wait complete. 2023-10-16T14:11:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.354+0000 7f67a636c640 1 Processor -- start 2023-10-16T14:11:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.354+0000 7f67a636c640 1 -- start start 2023-10-16T14:11:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a636c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0039e0 0x7f67a01510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a636c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a01f0440 0x7f67a00c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a636c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 0x7f67a00c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a636c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00ca2f0 con 0x7f67a00c7990 2023-10-16T14:11:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a636c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00c8380 con 0x7f678c0039e0 2023-10-16T14:11:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a636c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00c77a0 con 0x7f67a01f0440 2023-10-16T14:11:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a4909640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 0x7f67a00c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a4909640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 0x7f67a00c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59410/0 (socket says 172.21.15.60:59410) 2023-10-16T14:11:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f67a4909640 1 -- 172.21.15.60:0/4189747528 learned_addr learned my addr 172.21.15.60:0/4189747528 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.355+0000 7f679ffff640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0039e0 0x7f67a01510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f6797fff640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a01f0440 0x7f67a00c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a4909640 1 -- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a01f0440 msgr2=0x7f67a00c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a4909640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a01f0440 0x7f67a00c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a4909640 1 -- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0039e0 msgr2=0x7f67a01510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a4909640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0039e0 0x7f67a01510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:16.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a4909640 1 -- 172.21.15.60:0/4189747528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00c9a50 con 0x7f67a00c7990 2023-10-16T14:11:16.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f679ffff640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0039e0 0x7f67a01510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:16.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f6797fff640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a01f0440 0x7f67a00c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:16.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a4909640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 0x7f67a00c7e40 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f67a00b4160 tx=0x7f6798007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:16.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800ea80 con 0x7f67a00c7990 2023-10-16T14:11:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679800ec00 con 0x7f67a00c7990 2023-10-16T14:11:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a00c9d30 con 0x7f67a00c7990 2023-10-16T14:11:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798013420 con 0x7f67a00c7990 2023-10-16T14:11:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.356+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a00b16a0 con 0x7f67a00c7990 2023-10-16T14:11:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.358+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a00b1500 con 0x7f67a00c7990 2023-10-16T14:11:16.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.359+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6798013580 con 0x7f67a00c7990 2023-10-16T14:11:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.360+0000 7f679d7fa640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f678006b0b0 0x7f678006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.360+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7476+0+0 (secure 0 0 0) 0x7f679808a490 con 0x7f67a00c7990 2023-10-16T14:11:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.360+0000 7f679ffff640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f678006b0b0 0x7f678006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.362+0000 7f679ffff640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f678006b0b0 0x7f678006d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6790033000 tx=0x7f6790009030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.364+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67a00b1500 con 0x7f67a00c7990 2023-10-16T14:11:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:16.560+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f67a00b1500 con 0x7f67a00c7990 2023-10-16T14:11:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.093+0000 7f679d7fa640 1 -- 172.21.15.60:0/4189747528 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v239) v1 ==== 181+0+0 (secure 0 0 0) 0x7f67a00b1500 con 0x7f67a00c7990 2023-10-16T14:11:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:11:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.097+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f678006b0b0 msgr2=0x7f678006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.097+0000 7f67a636c640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f678006b0b0 0x7f678006d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6790033000 tx=0x7f6790009030 comp rx=0 tx=0).stop 2023-10-16T14:11:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.098+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 msgr2=0x7f67a00c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.098+0000 7f67a636c640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 0x7f67a00c7e40 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f67a00b4160 tx=0x7f6798007bc0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.099+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 shutdown_connections 2023-10-16T14:11:17.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.099+0000 7f67a636c640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f678006b0b0 0x7f678006d570 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.099+0000 7f67a636c640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67a01f0440 0x7f67a00c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.099+0000 7f67a636c640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67a00c7990 0x7f67a00c7e40 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.099+0000 7f67a636c640 1 --2- 172.21.15.60:0/4189747528 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f678c0039e0 0x7f67a01510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.099+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 >> 172.21.15.60:0/4189747528 conn(0x7f67a00cb360 msgr2=0x7f67a015e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:17.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.100+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 shutdown_connections 2023-10-16T14:11:17.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.100+0000 7f67a636c640 1 -- 172.21.15.60:0/4189747528 wait complete. 2023-10-16T14:11:17.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:11:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.351+0000 7ff70e347640 1 Processor -- start 2023-10-16T14:11:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.351+0000 7ff70e347640 1 -- start start 2023-10-16T14:11:17.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70e347640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff7080cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70e347640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70815b9d0 con 0x7ff7080ccfb0 2023-10-16T14:11:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70e347640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70815bcb0 con 0x7ff70806c040 2023-10-16T14:11:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70e347640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70815bf90 con 0x7ff7080ad970 2023-10-16T14:11:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70db46640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff7080ccfb0 0x7ff7080cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41034/0 (socket says 172.21.15.60:41034) 2023-10-16T14:11:17.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70db46640 1 -- 172.21.15.60:0/588318910 learned_addr learned my addr 172.21.15.60:0/588318910 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:17.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.353+0000 7ff70cb44640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff7080cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790821723 0 0) 0x7ff70815b9d0 con 0x7ff7080ccfb0 2023-10-16T14:11:17.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f4003520 con 0x7ff7080ccfb0 2023-10-16T14:11:17.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3796733284 0 0) 0x7ff6f4003520 con 0x7ff7080ccfb0 2023-10-16T14:11:17.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 msgr2=0x7ff7080cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:17.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff7080cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 >> v1:172.21.15.144:6789/0 conn(0x7ff70806c040 legacy=0x7ff70806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080cc270 con 0x7ff7080ccfb0 2023-10-16T14:11:17.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff7000019f0 con 0x7ff7080ccfb0 2023-10-16T14:11:17.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff70cb44640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff7080cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:17.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 >> v1:172.21.15.60:6789/0 conn(0x7ff7080ccfb0 legacy=0x7ff7080cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff6f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f4003fe0 0x7ff6f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff6f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70815bf90 con 0x7ff6f4004d00 2023-10-16T14:11:17.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f400bb40 con 0x7ff6f4003fe0 2023-10-16T14:11:17.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.354+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f400bd00 con 0x7ff7080ad970 2023-10-16T14:11:17.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70db46640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff6f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70cb44640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f4003fe0 0x7ff6f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70d345640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff6f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70db46640 1 -- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 msgr2=0x7ff6f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:17.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70db46640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080ad970 0x7ff6f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70db46640 1 -- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f4003fe0 msgr2=0x7ff6f40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:17.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70db46640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6f4003fe0 0x7ff6f40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.355+0000 7ff70db46640 1 -- 172.21.15.60:0/588318910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4003520 con 0x7ff6f4004d00 2023-10-16T14:11:17.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.356+0000 7ff70db46640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff6f4005180 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7ff70000a760 tx=0x7ff6f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3dbe6df64d0f1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:17.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.356+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700034630 con 0x7ff6f4004d00 2023-10-16T14:11:17.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.356+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70002f070 con 0x7ff6f4004d00 2023-10-16T14:11:17.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.356+0000 7ff6fffff640 1 -- 172.21.15.60:0/588318910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700034a50 con 0x7ff6f4004d00 2023-10-16T14:11:17.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70e347640 1 -- 172.21.15.60:0/588318910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 msgr2=0x7ff6f4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70e347640 1 --2- 172.21.15.60:0/588318910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff6f4005180 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7ff70000a760 tx=0x7ff6f40057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70db46640 1 -- 172.21.15.60:0/588318910 reap_dead start 2023-10-16T14:11:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70e347640 1 -- 172.21.15.60:0/588318910 shutdown_connections 2023-10-16T14:11:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70e347640 1 -- 172.21.15.60:0/588318910 >> 172.21.15.60:0/588318910 conn(0x7ff7080ce6a0 msgr2=0x7ff7080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70e347640 1 -- 172.21.15.60:0/588318910 shutdown_connections 2023-10-16T14:11:17.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.357+0000 7ff70e347640 1 -- 172.21.15.60:0/588318910 wait complete. 2023-10-16T14:11:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.359+0000 7ff70e347640 1 Processor -- start 2023-10-16T14:11:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.359+0000 7ff70e347640 1 -- start start 2023-10-16T14:11:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.360+0000 7ff70e347640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff70814f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.360+0000 7ff70e347640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70814c330 0x7ff7080c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.360+0000 7ff70e347640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080c2710 0x7ff7080ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.360+0000 7ff70e347640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080ca840 con 0x7ff6f4004d00 2023-10-16T14:11:17.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.360+0000 7ff70e347640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080c2bc0 con 0x7ff70814c330 2023-10-16T14:11:17.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.360+0000 7ff70e347640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080c2de0 con 0x7ff7080c2710 2023-10-16T14:11:17.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff70814f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff70814f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59440/0 (socket says 172.21.15.60:59440) 2023-10-16T14:11:17.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 -- 172.21.15.60:0/3594995386 learned_addr learned my addr 172.21.15.60:0/3594995386 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:17.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70cb44640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70814c330 0x7ff7080c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70db46640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080c2710 0x7ff7080ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 -- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080c2710 msgr2=0x7ff7080ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080c2710 0x7ff7080ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 -- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70814c330 msgr2=0x7ff7080c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:17.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70814c330 0x7ff7080c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:17.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.361+0000 7ff70d345640 1 -- 172.21.15.60:0/3594995386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080cdcd0 con 0x7ff6f4004d00 2023-10-16T14:11:17.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.362+0000 7ff70d345640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff70814f000 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e580 tx=0x7ff70400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:17.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.362+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7040052e0 con 0x7ff6f4004d00 2023-10-16T14:11:17.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.362+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff704005440 con 0x7ff6f4004d00 2023-10-16T14:11:17.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.362+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704003810 con 0x7ff6f4004d00 2023-10-16T14:11:17.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.362+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7080c30c0 con 0x7ff6f4004d00 2023-10-16T14:11:17.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.362+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7080c8e70 con 0x7ff6f4004d00 2023-10-16T14:11:17.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.363+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7080c9030 con 0x7ff6f4004d00 2023-10-16T14:11:17.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.364+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff704003970 con 0x7ff6f4004d00 2023-10-16T14:11:17.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.365+0000 7ff6fdffb640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff6ec06b0b0 0x7ff6ec06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:17.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.365+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 7038+0+0 (secure 0 0 0) 0x7ff704089d80 con 0x7ff6f4004d00 2023-10-16T14:11:17.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.366+0000 7ff70cb44640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff6ec06b0b0 0x7ff6ec06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:17.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.367+0000 7ff70cb44640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff6ec06b0b0 0x7ff6ec06d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff70814c750 tx=0x7ff6f8008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:17.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.370+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff704054510 con 0x7ff6f4004d00 2023-10-16T14:11:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:17.575+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff70806b330 con 0x7ff6f4004d00 2023-10-16T14:11:18.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.106+0000 7ff6fdffb640 1 -- 172.21.15.60:0/3594995386 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v240) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff704057b30 con 0x7ff6f4004d00 2023-10-16T14:11:18.110 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T14:11:18.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff6ec06b0b0 msgr2=0x7ff6ec06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff6ec06b0b0 0x7ff6ec06d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff70814c750 tx=0x7ff6f8008040 comp rx=0 tx=0).stop 2023-10-16T14:11:18.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 msgr2=0x7ff70814f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff70814f000 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7ff70400e580 tx=0x7ff70400ea40 comp rx=0 tx=0).stop 2023-10-16T14:11:18.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 shutdown_connections 2023-10-16T14:11:18.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff6ec06b0b0 0x7ff6ec06d570 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7080c2710 0x7ff7080ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff70814c330 0x7ff7080c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 --2- 172.21.15.60:0/3594995386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f4004d00 0x7ff70814f000 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 >> 172.21.15.60:0/3594995386 conn(0x7ff7080ce6a0 msgr2=0x7ff70806a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:18.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.110+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 shutdown_connections 2023-10-16T14:11:18.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.111+0000 7ff70e347640 1 -- 172.21.15.60:0/3594995386 wait complete. 2023-10-16T14:11:18.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:18.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.345+0000 7f52ce56c640 1 Processor -- start 2023-10-16T14:11:18.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.345+0000 7f52ce56c640 1 -- start start 2023-10-16T14:11:18.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.346+0000 7f52ce56c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52c80ad970 0x7f52c80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.346+0000 7f52ce56c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f52c80ccfb0 0x7f52c80cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.346+0000 7f52ce56c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c815b9d0 con 0x7f52c806c040 2023-10-16T14:11:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.346+0000 7f52ce56c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c815bcb0 con 0x7f52c80ad970 2023-10-16T14:11:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.346+0000 7f52ce56c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c815bf90 con 0x7f52c80ccfb0 2023-10-16T14:11:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.347+0000 7f52cd56a640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f52c806c040 0x7f52c806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41044/0 (socket says 172.21.15.60:41044) 2023-10-16T14:11:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.347+0000 7f52cd56a640 1 -- 172.21.15.60:0/2290690213 learned_addr learned my addr 172.21.15.60:0/2290690213 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.347+0000 7f52ccd69640 1 --2- 172.21.15.60:0/2290690213 >> v2:172.21.15.60:3300/0 conn(0x7f52c80ad970 0x7f52c80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.347+0000 7f52cdd6b640 1 --2- 172.21.15.60:0/2290690213 >> v2:172.21.15.144:3300/0 conn(0x7f52c80ccfb0 0x7f52c80cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.347+0000 7f52bffff640 1 -- 172.21.15.60:0/2290690213 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749766298 0 0) 0x7f52c815b9d0 con 0x7f52c806c040 2023-10-16T14:11:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.348+0000 7f52bffff640 1 -- 172.21.15.60:0/2290690213 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b4003520 con 0x7f52c806c040 2023-10-16T14:11:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.348+0000 7f52ccd69640 1 -- 172.21.15.60:0/2290690213 >> v2:172.21.15.144:3300/0 conn(0x7f52c80ccfb0 msgr2=0x7f52c80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.348+0000 7f52ccd69640 1 --2- 172.21.15.60:0/2290690213 >> v2:172.21.15.144:3300/0 conn(0x7f52c80ccfb0 0x7f52c80cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.348+0000 7f52ccd69640 1 -- 172.21.15.60:0/2290690213 >> v1:172.21.15.60:6789/0 conn(0x7f52c806c040 legacy=0x7f52c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.348+0000 7f52ccd69640 1 -- 172.21.15.60:0/2290690213 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80cc270 con 0x7f52c80ad970 2023-10-16T14:11:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.349+0000 7f52ccd69640 1 --2- 172.21.15.60:0/2290690213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80cca70 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f52b80097a0 tx=0x7f52b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e3b55b8fa2a337 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.349+0000 7f52bffff640 1 -- 172.21.15.60:0/2290690213 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8031830 con 0x7f52c80ad970 2023-10-16T14:11:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.349+0000 7f52bffff640 1 -- 172.21.15.60:0/2290690213 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b8031990 con 0x7f52c80ad970 2023-10-16T14:11:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.349+0000 7f52bffff640 1 -- 172.21.15.60:0/2290690213 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8031d00 con 0x7f52c80ad970 2023-10-16T14:11:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 -- 172.21.15.60:0/2290690213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 msgr2=0x7f52c80cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 --2- 172.21.15.60:0/2290690213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80cca70 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f52b80097a0 tx=0x7f52b802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 -- 172.21.15.60:0/2290690213 shutdown_connections 2023-10-16T14:11:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 --2- 172.21.15.60:0/2290690213 >> v2:172.21.15.144:3300/0 conn(0x7f52c80ccfb0 0x7f52c80cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 --2- 172.21.15.60:0/2290690213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80cca70 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 -- 172.21.15.60:0/2290690213 >> 172.21.15.60:0/2290690213 conn(0x7f52c80ce6a0 msgr2=0x7f52c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 -- 172.21.15.60:0/2290690213 shutdown_connections 2023-10-16T14:11:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.350+0000 7f52ce56c640 1 -- 172.21.15.60:0/2290690213 wait complete. 2023-10-16T14:11:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.352+0000 7f52ce56c640 1 Processor -- start 2023-10-16T14:11:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.352+0000 7f52ce56c640 1 -- start start 2023-10-16T14:11:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ce56c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c806c040 0x7f52c81554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ce56c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ce56c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c80ccfb0 0x7f52c80c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ce56c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80c7290 con 0x7f52c80ad970 2023-10-16T14:11:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ce56c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80c7570 con 0x7f52c80ccfb0 2023-10-16T14:11:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ce56c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80c7850 con 0x7f52c806c040 2023-10-16T14:11:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80c6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59454/0 (socket says 172.21.15.60:59454) 2023-10-16T14:11:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 -- 172.21.15.60:0/4003768042 learned_addr learned my addr 172.21.15.60:0/4003768042 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52cdd6b640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c80ccfb0 0x7f52c80c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52cd56a640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c806c040 0x7f52c81554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 -- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c806c040 msgr2=0x7f52c81554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c806c040 0x7f52c81554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 -- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c80ccfb0 msgr2=0x7f52c80c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.353+0000 7f52ccd69640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c80ccfb0 0x7f52c80c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52ccd69640 1 -- 172.21.15.60:0/4003768042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b8009400 con 0x7f52c80ad970 2023-10-16T14:11:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52cdd6b640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c80ccfb0 0x7f52c80c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52ccd69640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80c6990 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f52b80098c0 tx=0x7f52b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b803a6f0 con 0x7f52c80ad970 2023-10-16T14:11:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b803a850 con 0x7f52c80ad970 2023-10-16T14:11:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b803ac00 con 0x7f52c80ad970 2023-10-16T14:11:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c80c7b30 con 0x7f52c80ad970 2023-10-16T14:11:18.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.354+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c8155a30 con 0x7f52c80ad970 2023-10-16T14:11:18.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.356+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c806b330 con 0x7f52c80ad970 2023-10-16T14:11:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.356+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f52b8030380 con 0x7f52c80ad970 2023-10-16T14:11:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.357+0000 7f52bdffb640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f52ac06b0b0 0x7f52ac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.358+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f52b80ab840 con 0x7f52c80ad970 2023-10-16T14:11:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.358+0000 7f52cd56a640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f52ac06b0b0 0x7f52ac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.359+0000 7f52cd56a640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f52ac06b0b0 0x7f52ac06d570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52c4007fd0 tx=0x7f52c4009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:18.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.362+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52b8076100 con 0x7f52c80ad970 2023-10-16T14:11:18.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.560+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52c80c8050 con 0x7f52c80ad970 2023-10-16T14:11:18.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.563+0000 7f52bdffb640 1 -- 172.21.15.60:0/4003768042 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+17915 (secure 0 0 0) 0x7f52c80c8050 con 0x7f52c80ad970 2023-10-16T14:11:18.568 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:18.568 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":240,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:18.095169+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6835/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f52ac06b0b0 msgr2=0x7f52ac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f52ac06b0b0 0x7f52ac06d570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52c4007fd0 tx=0x7f52c4009040 comp rx=0 tx=0).stop 2023-10-16T14:11:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 msgr2=0x7f52c80c6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80c6990 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f52b80098c0 tx=0x7f52b80046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 shutdown_connections 2023-10-16T14:11:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f52ac06b0b0 0x7f52ac06d570 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52c80ccfb0 0x7f52c80c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52c80ad970 0x7f52c80c6990 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 --2- 172.21.15.60:0/4003768042 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52c806c040 0x7f52c81554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 >> 172.21.15.60:0/4003768042 conn(0x7f52c80ce6a0 msgr2=0x7f52c80cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 shutdown_connections 2023-10-16T14:11:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.566+0000 7f52ce56c640 1 -- 172.21.15.60:0/4003768042 wait complete. 2023-10-16T14:11:18.589 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T14:10:38.740891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T14:10:39.464305+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:11:18.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:11:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.819+0000 7f9c3d1a5640 1 Processor -- start 2023-10-16T14:11:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.820+0000 7f9c3d1a5640 1 -- start start 2023-10-16T14:11:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.821+0000 7f9c3d1a5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c3815cc00 0x7f9c380ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.821+0000 7f9c3d1a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380cd260 0x7f9c380ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.821+0000 7f9c3d1a5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c380ad670 0x7f9c380c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.821+0000 7f9c3d1a5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c380ce480 con 0x7f9c3815cc00 2023-10-16T14:11:18.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.821+0000 7f9c3d1a5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c380ca950 con 0x7f9c380ad670 2023-10-16T14:11:18.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.821+0000 7f9c3d1a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380cd7b0 con 0x7f9c380cd260 2023-10-16T14:11:18.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c3815cc00 0x7f9c380ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c3815cc00 0x7f9c380ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59464/0 (socket says 172.21.15.60:59464) 2023-10-16T14:11:18.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 -- 172.21.15.60:0/3736168096 learned_addr learned my addr 172.21.15.60:0/3736168096 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:18.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c377fe640 1 --2- 172.21.15.60:0/3736168096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380cd260 0x7f9c380ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c3c9a4640 1 --2- 172.21.15.60:0/3736168096 >> v2:172.21.15.144:3300/0 conn(0x7f9c380ad670 0x7f9c380c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 -- 172.21.15.60:0/3736168096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380cd260 msgr2=0x7f9c380ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:18.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 --2- 172.21.15.60:0/3736168096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380cd260 0x7f9c380ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 -- 172.21.15.60:0/3736168096 >> v2:172.21.15.144:3300/0 conn(0x7f9c380ad670 msgr2=0x7f9c380c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:18.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.822+0000 7f9c37fff640 1 --2- 172.21.15.60:0/3736168096 >> v2:172.21.15.144:3300/0 conn(0x7f9c380ad670 0x7f9c380c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.823+0000 7f9c37fff640 1 -- 172.21.15.60:0/3736168096 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380cda90 con 0x7f9c3815cc00 2023-10-16T14:11:18.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.824+0000 7f9c37fff640 1 --2- 172.21.15.60:0/3736168096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3815cc00 0x7f9c380ca410 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f9c38073cb0 tx=0x7f9c2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4660d030e2864e4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:18.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.824+0000 7f9c36ffd640 1 -- 172.21.15.60:0/3736168096 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28031870 con 0x7f9c3815cc00 2023-10-16T14:11:18.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.824+0000 7f9c36ffd640 1 -- 172.21.15.60:0/3736168096 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c280319d0 con 0x7f9c3815cc00 2023-10-16T14:11:18.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.824+0000 7f9c36ffd640 1 -- 172.21.15.60:0/3736168096 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28031d60 con 0x7f9c3815cc00 2023-10-16T14:11:18.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/3736168096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3815cc00 msgr2=0x7f9c380ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/3736168096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3815cc00 0x7f9c380ca410 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f9c38073cb0 tx=0x7f9c2802b910 comp rx=0 tx=0).stop 2023-10-16T14:11:18.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/3736168096 shutdown_connections 2023-10-16T14:11:18.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/3736168096 >> v2:172.21.15.144:3300/0 conn(0x7f9c380ad670 0x7f9c380c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/3736168096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380cd260 0x7f9c380ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/3736168096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c3815cc00 0x7f9c380ca410 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/3736168096 >> 172.21.15.60:0/3736168096 conn(0x7f9c380b0170 msgr2=0x7f9c380b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.825+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/3736168096 shutdown_connections 2023-10-16T14:11:18.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.826+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/3736168096 wait complete. 2023-10-16T14:11:18.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.827+0000 7f9c3d1a5640 1 Processor -- start 2023-10-16T14:11:18.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.827+0000 7f9c3d1a5640 1 -- start start 2023-10-16T14:11:18.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.828+0000 7f9c3d1a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380ad670 0x7f9c380ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.828+0000 7f9c3d1a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 0x7f9c381577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.828+0000 7f9c3d1a5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c3815cc00 0x7f9c38157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.828+0000 7f9c3d1a5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38159170 con 0x7f9c380cd260 2023-10-16T14:11:18.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.828+0000 7f9c3d1a5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38158560 con 0x7f9c3815cc00 2023-10-16T14:11:18.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.828+0000 7f9c3d1a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38158840 con 0x7f9c380ad670 2023-10-16T14:11:18.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 0x7f9c381577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 0x7f9c381577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59468/0 (socket says 172.21.15.60:59468) 2023-10-16T14:11:18.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 -- 172.21.15.60:0/609957474 learned_addr learned my addr 172.21.15.60:0/609957474 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:18.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c37fff640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380ad670 0x7f9c380ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 -- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380ad670 msgr2=0x7f9c380ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380ad670 0x7f9c380ae180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 -- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c3815cc00 msgr2=0x7f9c38157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:18.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c3c9a4640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c3815cc00 0x7f9c38157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c3815cc00 0x7f9c38157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:18.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.829+0000 7f9c377fe640 1 -- 172.21.15.60:0/609957474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c280094d0 con 0x7f9c380cd260 2023-10-16T14:11:18.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.830+0000 7f9c377fe640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 0x7f9c381577c0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00d5f0 tx=0x7f9c2c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:18.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.830+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0125f0 con 0x7f9c380cd260 2023-10-16T14:11:18.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.830+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c012750 con 0x7f9c380cd260 2023-10-16T14:11:18.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.830+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c012b00 con 0x7f9c380cd260 2023-10-16T14:11:18.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.830+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38158b20 con 0x7f9c380cd260 2023-10-16T14:11:18.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.830+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c38158240 con 0x7f9c380cd260 2023-10-16T14:11:18.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.831+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c380b0ea0 con 0x7f9c380cd260 2023-10-16T14:11:18.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.832+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9c2c012c60 con 0x7f9c380cd260 2023-10-16T14:11:18.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.834+0000 7f9c34ff9640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9c2406b0b0 0x7f9c2406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:18.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.834+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f9c2c08a200 con 0x7f9c380cd260 2023-10-16T14:11:18.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.834+0000 7f9c37fff640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9c2406b0b0 0x7f9c2406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:18.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.836+0000 7f9c37fff640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9c2406b0b0 0x7f9c2406d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c28036000 tx=0x7f9c2803c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:18.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:18.838+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c380b0ea0 con 0x7f9c380cd260 2023-10-16T14:11:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.035+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c380b0ea0 con 0x7f9c380cd260 2023-10-16T14:11:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.036+0000 7f9c34ff9640 1 -- 172.21.15.60:0/609957474 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v240) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c380b0ea0 con 0x7f9c380cd260 2023-10-16T14:11:19.040 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:11:19.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9c2406b0b0 msgr2=0x7f9c2406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9c2406b0b0 0x7f9c2406d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c28036000 tx=0x7f9c2803c040 comp rx=0 tx=0).stop 2023-10-16T14:11:19.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 msgr2=0x7f9c381577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 0x7f9c381577c0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00d5f0 tx=0x7f9c2c00dab0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 shutdown_connections 2023-10-16T14:11:19.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9c2406b0b0 0x7f9c2406d570 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c3815cc00 0x7f9c38157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c380cd260 0x7f9c381577c0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 --2- 172.21.15.60:0/609957474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c380ad670 0x7f9c380ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.039+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 >> 172.21.15.60:0/609957474 conn(0x7f9c380b0170 msgr2=0x7f9c380af2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:19.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.040+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 shutdown_connections 2023-10-16T14:11:19.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.040+0000 7f9c3d1a5640 1 -- 172.21.15.60:0/609957474 wait complete. 2023-10-16T14:11:19.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T14:11:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.303+0000 7f6ac3a90640 1 Processor -- start 2023-10-16T14:11:19.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.303+0000 7f6ac3a90640 1 -- start start 2023-10-16T14:11:19.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.304+0000 7f6ac3a90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6abc159e90 0x7f6abc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.304+0000 7f6ac3a90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc070360 con 0x7f6abc06c140 2023-10-16T14:11:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.304+0000 7f6ac3a90640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0cd1f0 con 0x7f6abc159ab0 2023-10-16T14:11:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.304+0000 7f6ac3a90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc15d890 con 0x7f6abc159e90 2023-10-16T14:11:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.305+0000 7f6ac102b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6abc159e90 0x7f6abc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.305+0000 7f6ac102b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6abc159e90 0x7f6abc0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46954/0 (socket says 172.21.15.60:46954) 2023-10-16T14:11:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.305+0000 7f6ac102b640 1 -- 172.21.15.60:0/4093340679 learned_addr learned my addr 172.21.15.60:0/4093340679 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.305+0000 7f6ac102b640 1 -- 172.21.15.60:0/4093340679 >> v1:172.21.15.144:6789/0 conn(0x7f6abc159ab0 legacy=0x7f6abc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.305+0000 7f6ac102b640 1 -- 172.21.15.60:0/4093340679 >> v1:172.21.15.60:6789/0 conn(0x7f6abc06c140 legacy=0x7f6abc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.305+0000 7f6ac102b640 1 -- 172.21.15.60:0/4093340679 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0cd4d0 con 0x7f6abc159e90 2023-10-16T14:11:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.306+0000 7f6ac102b640 1 --2- 172.21.15.60:0/4093340679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159e90 0x7f6abc0ce370 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009870 tx=0x7f6ab802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57058408fa2cb60 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac082a640 1 -- 172.21.15.60:0/4093340679 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80317e0 con 0x7f6abc159e90 2023-10-16T14:11:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac082a640 1 -- 172.21.15.60:0/4093340679 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8031940 con 0x7f6abc159e90 2023-10-16T14:11:19.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac082a640 1 -- 172.21.15.60:0/4093340679 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8031cd0 con 0x7f6abc159e90 2023-10-16T14:11:19.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac3a90640 1 -- 172.21.15.60:0/4093340679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159e90 msgr2=0x7f6abc0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/4093340679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159e90 0x7f6abc0ce370 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009870 tx=0x7f6ab802b880 comp rx=0 tx=0).stop 2023-10-16T14:11:19.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac3a90640 1 -- 172.21.15.60:0/4093340679 shutdown_connections 2023-10-16T14:11:19.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/4093340679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159e90 0x7f6abc0ce370 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac3a90640 1 -- 172.21.15.60:0/4093340679 >> 172.21.15.60:0/4093340679 conn(0x7f6abc0ce9b0 msgr2=0x7f6abc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:19.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.307+0000 7f6ac3a90640 1 -- 172.21.15.60:0/4093340679 shutdown_connections 2023-10-16T14:11:19.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.308+0000 7f6ac3a90640 1 -- 172.21.15.60:0/4093340679 wait complete. 2023-10-16T14:11:19.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.309+0000 7f6ac3a90640 1 Processor -- start 2023-10-16T14:11:19.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.309+0000 7f6ac3a90640 1 -- start start 2023-10-16T14:11:19.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.310+0000 7f6ac3a90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6abc06c140 0x7f6abc0c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.310+0000 7f6ac3a90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 0x7f6abc15caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.310+0000 7f6ac3a90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6abc159e90 0x7f6abc0af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.310+0000 7f6ac3a90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1f1690 con 0x7f6abc159ab0 2023-10-16T14:11:19.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.310+0000 7f6ac3a90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1f1970 con 0x7f6abc06c140 2023-10-16T14:11:19.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.310+0000 7f6ac3a90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1f1c50 con 0x7f6abc159e90 2023-10-16T14:11:19.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 0x7f6abc15caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 0x7f6abc15caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46970/0 (socket says 172.21.15.60:46970) 2023-10-16T14:11:19.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 -- 172.21.15.60:0/2739654949 learned_addr learned my addr 172.21.15.60:0/2739654949 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:19.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 -- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6abc159e90 msgr2=0x7f6abc0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6abc159e90 0x7f6abc0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 -- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6abc06c140 msgr2=0x7f6abc0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6abc06c140 0x7f6abc0c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 -- 172.21.15.60:0/2739654949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80094d0 con 0x7f6abc159ab0 2023-10-16T14:11:19.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.311+0000 7f6ac102b640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 0x7f6abc15caf0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f6ab802b8c0 tx=0x7f6ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:19.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.312+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8032980 con 0x7f6abc159ab0 2023-10-16T14:11:19.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.312+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8032ae0 con 0x7f6abc159ab0 2023-10-16T14:11:19.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.312+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8032e90 con 0x7f6abc159ab0 2023-10-16T14:11:19.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.312+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc1f1f30 con 0x7f6abc159ab0 2023-10-16T14:11:19.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.312+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc1f24f0 con 0x7f6abc159ab0 2023-10-16T14:11:19.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.313+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc04e160 con 0x7f6abc159ab0 2023-10-16T14:11:19.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.314+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6ab804e450 con 0x7f6abc159ab0 2023-10-16T14:11:19.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.315+0000 7f6ab27fc640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6aa006f4c0 0x7f6aa0071980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.316+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f6ab80ac370 con 0x7f6abc159ab0 2023-10-16T14:11:19.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.316+0000 7f6ac182c640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6aa006f4c0 0x7f6aa0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.317+0000 7f6ac182c640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6aa006f4c0 0x7f6aa0071980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6aac003cc0 tx=0x7f6aac0079b0 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:19.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.320+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ab8076bb0 con 0x7f6abc159ab0 2023-10-16T14:11:19.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.516+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6abc0ccb60 con 0x7f6abc159ab0 2023-10-16T14:11:19.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.517+0000 7f6ab27fc640 1 -- 172.21.15.60:0/2739654949 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v240) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6ab807a1d0 con 0x7f6abc159ab0 2023-10-16T14:11:19.521 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:11:19.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6aa006f4c0 msgr2=0x7f6aa0071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6aa006f4c0 0x7f6aa0071980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6aac003cc0 tx=0x7f6aac0079b0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 msgr2=0x7f6abc15caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 0x7f6abc15caf0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f6ab802b8c0 tx=0x7f6ab80046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 shutdown_connections 2023-10-16T14:11:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6aa006f4c0 0x7f6aa0071980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6abc159e90 0x7f6abc0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6abc159ab0 0x7f6abc15caf0 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 --2- 172.21.15.60:0/2739654949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6abc06c140 0x7f6abc0c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.520+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 >> 172.21.15.60:0/2739654949 conn(0x7f6abc0ce9b0 msgr2=0x7f6abc0ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.521+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 shutdown_connections 2023-10-16T14:11:19.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.521+0000 7f6ac3a90640 1 -- 172.21.15.60:0/2739654949 wait complete. 2023-10-16T14:11:19.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T14:11:19.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.765+0000 7f75508d1640 1 Processor -- start 2023-10-16T14:11:19.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.765+0000 7f75508d1640 1 -- start start 2023-10-16T14:11:19.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f75508d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f754c160c80 0x7f754c161060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f75508d1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f754c0ad7a0 0x7f754c070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f75508d1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0ae1a0 con 0x7f754c0adc30 2023-10-16T14:11:19.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f75508d1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754c0ae480 con 0x7f754c0ad7a0 2023-10-16T14:11:19.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f75508d1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754c0ae760 con 0x7f754c160c80 2023-10-16T14:11:19.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f754affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f754c0ad7a0 0x7f754c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f754affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f754c0ad7a0 0x7f754c070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46984/0 (socket says 172.21.15.60:46984) 2023-10-16T14:11:19.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f754affd640 1 -- 172.21.15.60:0/603932089 learned_addr learned my addr 172.21.15.60:0/603932089 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:19.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.767+0000 7f754b7fe640 1 --2- 172.21.15.60:0/603932089 >> v2:172.21.15.144:3300/0 conn(0x7f754c160c80 0x7f754c161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.768+0000 7f754affd640 1 -- 172.21.15.60:0/603932089 >> v2:172.21.15.144:3300/0 conn(0x7f754c160c80 msgr2=0x7f754c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:19.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.768+0000 7f754affd640 1 --2- 172.21.15.60:0/603932089 >> v2:172.21.15.144:3300/0 conn(0x7f754c160c80 0x7f754c161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.768+0000 7f754affd640 1 -- 172.21.15.60:0/603932089 >> v1:172.21.15.144:6789/0 conn(0x7f754c0adc30 legacy=0x7f754c142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.768+0000 7f754affd640 1 -- 172.21.15.60:0/603932089 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c0aea40 con 0x7f754c0ad7a0 2023-10-16T14:11:19.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.769+0000 7f754affd640 1 --2- 172.21.15.60:0/603932089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0ad7a0 0x7f754c070240 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f754c073aa0 tx=0x7f754002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74070a41247d847 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:19.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.769+0000 7f754a7fc640 1 -- 172.21.15.60:0/603932089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540031870 con 0x7f754c0ad7a0 2023-10-16T14:11:19.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.769+0000 7f754a7fc640 1 -- 172.21.15.60:0/603932089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75400319d0 con 0x7f754c0ad7a0 2023-10-16T14:11:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.769+0000 7f754a7fc640 1 -- 172.21.15.60:0/603932089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540031d60 con 0x7f754c0ad7a0 2023-10-16T14:11:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 -- 172.21.15.60:0/603932089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0ad7a0 msgr2=0x7f754c070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 --2- 172.21.15.60:0/603932089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0ad7a0 0x7f754c070240 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f754c073aa0 tx=0x7f754002b910 comp rx=0 tx=0).stop 2023-10-16T14:11:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 -- 172.21.15.60:0/603932089 shutdown_connections 2023-10-16T14:11:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 --2- 172.21.15.60:0/603932089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0ad7a0 0x7f754c070240 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 --2- 172.21.15.60:0/603932089 >> v2:172.21.15.144:3300/0 conn(0x7f754c160c80 0x7f754c161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 -- 172.21.15.60:0/603932089 >> 172.21.15.60:0/603932089 conn(0x7f754c0cc230 msgr2=0x7f754c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 -- 172.21.15.60:0/603932089 shutdown_connections 2023-10-16T14:11:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.770+0000 7f75508d1640 1 -- 172.21.15.60:0/603932089 wait complete. 2023-10-16T14:11:19.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.772+0000 7f75508d1640 1 Processor -- start 2023-10-16T14:11:19.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.772+0000 7f75508d1640 1 -- start start 2023-10-16T14:11:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f75508d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f754c0ad7a0 0x7f754c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f75508d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 0x7f754c15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f75508d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f754c160c80 0x7f754c0c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f75508d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c15c830 con 0x7f754c0adc30 2023-10-16T14:11:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f75508d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c06c7a0 con 0x7f754c0ad7a0 2023-10-16T14:11:19.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f75508d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0c96c0 con 0x7f754c160c80 2023-10-16T14:11:19.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f754affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 0x7f754c15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f754affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 0x7f754c15c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47000/0 (socket says 172.21.15.60:47000) 2023-10-16T14:11:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.773+0000 7f754affd640 1 -- 172.21.15.60:0/3062201123 learned_addr learned my addr 172.21.15.60:0/3062201123 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f754affd640 1 -- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f754c160c80 msgr2=0x7f754c0c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f754affd640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f754c160c80 0x7f754c0c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f754affd640 1 -- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f754c0ad7a0 msgr2=0x7f754c06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f754affd640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f754c0ad7a0 0x7f754c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f754affd640 1 -- 172.21.15.60:0/3062201123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75400094d0 con 0x7f754c0adc30 2023-10-16T14:11:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f754affd640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 0x7f754c15c2f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f7540000c00 tx=0x7f75400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754003abe0 con 0x7f754c0adc30 2023-10-16T14:11:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754003ad40 con 0x7f754c0adc30 2023-10-16T14:11:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c1f61d0 con 0x7f754c0adc30 2023-10-16T14:11:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.774+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540044610 con 0x7f754c0adc30 2023-10-16T14:11:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.775+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c1f6790 con 0x7f754c0adc30 2023-10-16T14:11:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.776+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c04e9b0 con 0x7f754c0adc30 2023-10-16T14:11:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.777+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f754004d450 con 0x7f754c0adc30 2023-10-16T14:11:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.778+0000 7f7533fff640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f753806b0b0 0x7f753806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.778+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f754002d040 con 0x7f754c0adc30 2023-10-16T14:11:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.778+0000 7f754b7fe640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f753806b0b0 0x7f753806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.779+0000 7f754b7fe640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f753806b0b0 0x7f753806d570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f753c006fd0 tx=0x7f753c008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.782+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7540076de0 con 0x7f754c0adc30 2023-10-16T14:11:19.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.982+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f754c0b5730 con 0x7f754c0adc30 2023-10-16T14:11:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.983+0000 7f7533fff640 1 -- 172.21.15.60:0/3062201123 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v240) v1 ==== 103+0+11 (secure 0 0 0) 0x7f754007a400 con 0x7f754c0adc30 2023-10-16T14:11:19.987 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:11:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.986+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f753806b0b0 msgr2=0x7f753806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.986+0000 7f75508d1640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f753806b0b0 0x7f753806d570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f753c006fd0 tx=0x7f753c008040 comp rx=0 tx=0).stop 2023-10-16T14:11:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.986+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 msgr2=0x7f754c15c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.986+0000 7f75508d1640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 0x7f754c15c2f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f7540000c00 tx=0x7f75400046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 shutdown_connections 2023-10-16T14:11:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f753806b0b0 0x7f753806d570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f754c160c80 0x7f754c0c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f754c0adc30 0x7f754c15c2f0 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 --2- 172.21.15.60:0/3062201123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f754c0ad7a0 0x7f754c06c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 >> 172.21.15.60:0/3062201123 conn(0x7f754c0cc230 msgr2=0x7f754c0c32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 shutdown_connections 2023-10-16T14:11:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:19.987+0000 7f75508d1640 1 -- 172.21.15.60:0/3062201123 wait complete. 2023-10-16T14:11:20.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:20.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.233+0000 7f44d6306640 1 Processor -- start 2023-10-16T14:11:20.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.233+0000 7f44d6306640 1 -- start start 2023-10-16T14:11:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.235+0000 7f44d6306640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44d015b320 0x7f44d00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.235+0000 7f44d6306640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d015b9d0 con 0x7f44d00ad970 2023-10-16T14:11:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.235+0000 7f44d6306640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d015bcb0 con 0x7f44d006c040 2023-10-16T14:11:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.235+0000 7f44d6306640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d015b7d0 con 0x7f44d015b320 2023-10-16T14:11:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.235+0000 7f44d4b03640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f44d00ad970 0x7f44d00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35842/0 (socket says 172.21.15.60:35842) 2023-10-16T14:11:20.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.235+0000 7f44d4b03640 1 -- 172.21.15.60:0/4204147590 learned_addr learned my addr 172.21.15.60:0/4204147590 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:20.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44d5b05640 1 --2- 172.21.15.60:0/4204147590 >> v2:172.21.15.144:3300/0 conn(0x7f44d015b320 0x7f44d00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623018654 0 0) 0x7f44d015b9d0 con 0x7f44d00ad970 2023-10-16T14:11:20.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc003520 con 0x7f44d00ad970 2023-10-16T14:11:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441110589 0 0) 0x7f44bc003520 con 0x7f44d00ad970 2023-10-16T14:11:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 >> v2:172.21.15.144:3300/0 conn(0x7f44d015b320 msgr2=0x7f44d00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 --2- 172.21.15.60:0/4204147590 >> v2:172.21.15.144:3300/0 conn(0x7f44d015b320 0x7f44d00cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 >> v1:172.21.15.144:6789/0 conn(0x7f44d006c040 legacy=0x7f44d006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d015bf90 con 0x7f44d00ad970 2023-10-16T14:11:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f44c0001670 con 0x7f44d00ad970 2023-10-16T14:11:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 >> v1:172.21.15.60:6789/0 conn(0x7f44d00ad970 legacy=0x7f44d00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44bc0039f0 0x7f44bc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.236+0000 7f44c7fff640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44c7fff640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44bc005180 0x7f44bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d015b7d0 con 0x7f44bc0047c0 2023-10-16T14:11:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc00c0b0 con 0x7f44bc0039f0 2023-10-16T14:11:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc00c270 con 0x7f44bc005180 2023-10-16T14:11:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d4b03640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d5304640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44bc0039f0 0x7f44bc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d5b05640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44bc005180 0x7f44bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d4b03640 1 -- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44bc005180 msgr2=0x7f44bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:20.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d4b03640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44bc005180 0x7f44bc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d4b03640 1 -- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44bc0039f0 msgr2=0x7f44bc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:20.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d4b03640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44bc0039f0 0x7f44bc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d4b03640 1 -- 172.21.15.60:0/4204147590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc003520 con 0x7f44bc0047c0 2023-10-16T14:11:20.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d5b05640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44bc005180 0x7f44bc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:20.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.237+0000 7f44d5b05640 1 -- 172.21.15.60:0/4204147590 reap_dead start 2023-10-16T14:11:20.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.238+0000 7f44d4b03640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44bc004c40 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f44d00ade20 tx=0x7f44bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d4ae8ccccdfe80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:20.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0005ad0 con 0x7f44bc0047c0 2023-10-16T14:11:20.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c0005c30 con 0x7f44bc0047c0 2023-10-16T14:11:20.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44c7fff640 1 -- 172.21.15.60:0/4204147590 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0007070 con 0x7f44bc0047c0 2023-10-16T14:11:20.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44d6306640 1 -- 172.21.15.60:0/4204147590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 msgr2=0x7f44bc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44d6306640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44bc004c40 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f44d00ade20 tx=0x7f44bc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44d6306640 1 -- 172.21.15.60:0/4204147590 shutdown_connections 2023-10-16T14:11:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44d6306640 1 --2- 172.21.15.60:0/4204147590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44bc004c40 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.239+0000 7f44d6306640 1 -- 172.21.15.60:0/4204147590 >> 172.21.15.60:0/4204147590 conn(0x7f44d00ce6a0 msgr2=0x7f44d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.240+0000 7f44d6306640 1 -- 172.21.15.60:0/4204147590 shutdown_connections 2023-10-16T14:11:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.240+0000 7f44d6306640 1 -- 172.21.15.60:0/4204147590 wait complete. 2023-10-16T14:11:20.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.241+0000 7f44d6306640 1 Processor -- start 2023-10-16T14:11:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.242+0000 7f44d6306640 1 -- start start 2023-10-16T14:11:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.242+0000 7f44d6306640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44d01f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.242+0000 7f44d6306640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d00af400 0x7f44d014ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d6306640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d00addc0 0x7f44d00afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d6306640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01f36d0 con 0x7f44bc0047c0 2023-10-16T14:11:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d6306640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00ae360 con 0x7f44d00af400 2023-10-16T14:11:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d6306640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00ae640 con 0x7f44d00addc0 2023-10-16T14:11:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44d01f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44d01f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47010/0 (socket says 172.21.15.60:47010) 2023-10-16T14:11:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 -- 172.21.15.60:0/3592854547 learned_addr learned my addr 172.21.15.60:0/3592854547 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d4b03640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d00af400 0x7f44d014ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5b05640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d00addc0 0x7f44d00afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 -- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d00addc0 msgr2=0x7f44d00afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d00addc0 0x7f44d00afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 -- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d00af400 msgr2=0x7f44d014ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d00af400 0x7f44d014ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5304640 1 -- 172.21.15.60:0/3592854547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00ae920 con 0x7f44bc0047c0 2023-10-16T14:11:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d4b03640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d00af400 0x7f44d014ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.243+0000 7f44d5b05640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d00addc0 0x7f44d00afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.244+0000 7f44d5304640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44d01f3190 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00e440 tx=0x7f44cc00e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.244+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc0052e0 con 0x7f44bc0047c0 2023-10-16T14:11:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.244+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44cc005440 con 0x7f44bc0047c0 2023-10-16T14:11:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.244+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc003810 con 0x7f44bc0047c0 2023-10-16T14:11:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.244+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d00aec00 con 0x7f44bc0047c0 2023-10-16T14:11:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.244+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d014f010 con 0x7f44bc0047c0 2023-10-16T14:11:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.245+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d006b330 con 0x7f44bc0047c0 2023-10-16T14:11:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.246+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f44cc003970 con 0x7f44bc0047c0 2023-10-16T14:11:20.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.247+0000 7f44c5ffb640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f44b406b0b0 0x7f44b406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.247+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f44cc089b50 con 0x7f44bc0047c0 2023-10-16T14:11:20.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.248+0000 7f44d4b03640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f44b406b0b0 0x7f44b406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.249+0000 7f44d4b03640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f44b406b0b0 0x7f44b406d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44c0034000 tx=0x7f44c0009030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:20.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.252+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44cc054490 con 0x7f44bc0047c0 2023-10-16T14:11:20.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.484+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44d004e9b0 con 0x7f44bc0047c0 2023-10-16T14:11:20.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.486+0000 7f44c5ffb640 1 -- 172.21.15.60:0/3592854547 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4927 (secure 0 0 0) 0x7f44cc057ab0 con 0x7f44bc0047c0 2023-10-16T14:11:20.490 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:20.490 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":120,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":8123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/499924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":499924806},{"type":"v1","addr":"172.21.15.144:6837","nonce":499924806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:20.490 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 120 2023-10-16T14:11:20.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f44b406b0b0 msgr2=0x7f44b406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f44b406b0b0 0x7f44b406d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44c0034000 tx=0x7f44c0009030 comp rx=0 tx=0).stop 2023-10-16T14:11:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 msgr2=0x7f44d01f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44d01f3190 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00e440 tx=0x7f44cc00e900 comp rx=0 tx=0).stop 2023-10-16T14:11:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 shutdown_connections 2023-10-16T14:11:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f44b406b0b0 0x7f44b406d570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44d00addc0 0x7f44d00afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44d00af400 0x7f44d014ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 --2- 172.21.15.60:0/3592854547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44bc0047c0 0x7f44d01f3190 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.488+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 >> 172.21.15.60:0/3592854547 conn(0x7f44d00ce6a0 msgr2=0x7f44d00c2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.489+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 shutdown_connections 2023-10-16T14:11:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.489+0000 7f44d6306640 1 -- 172.21.15.60:0/3592854547 wait complete. 2023-10-16T14:11:20.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:20.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.728+0000 7f1924cf2640 1 Processor -- start 2023-10-16T14:11:20.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.728+0000 7f1924cf2640 1 -- start start 2023-10-16T14:11:20.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.729+0000 7f1924cf2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015aaa0 0x7f192015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.729+0000 7f1924cf2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f192015ae80 0x7f19200cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.729+0000 7f1924cf2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19200ccaa0 0x7f1920070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.729+0000 7f1924cf2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192006b810 con 0x7f192015ae80 2023-10-16T14:11:20.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.729+0000 7f1924cf2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192015ed60 con 0x7f19200ccaa0 2023-10-16T14:11:20.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.729+0000 7f1924cf2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19200707e0 con 0x7f192015aaa0 2023-10-16T14:11:20.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f192015ae80 0x7f19200cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f192015ae80 0x7f19200cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47016/0 (socket says 172.21.15.60:47016) 2023-10-16T14:11:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 -- 172.21.15.60:0/1197320928 learned_addr learned my addr 172.21.15.60:0/1197320928 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 -- 172.21.15.60:0/1197320928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015aaa0 msgr2=0x7f192015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 --2- 172.21.15.60:0/1197320928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015aaa0 0x7f192015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 -- 172.21.15.60:0/1197320928 >> v2:172.21.15.144:3300/0 conn(0x7f19200ccaa0 msgr2=0x7f1920070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:20.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 --2- 172.21.15.60:0/1197320928 >> v2:172.21.15.144:3300/0 conn(0x7f19200ccaa0 0x7f1920070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.730+0000 7f191dd9b640 1 -- 172.21.15.60:0/1197320928 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192015ef90 con 0x7f192015ae80 2023-10-16T14:11:20.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.731+0000 7f191dd9b640 1 --2- 172.21.15.60:0/1197320928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200cc560 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f19200ae3c0 tx=0x7f190802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa81a6c6c1ded75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:20.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f191d59a640 1 -- 172.21.15.60:0/1197320928 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908031880 con 0x7f192015ae80 2023-10-16T14:11:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f191d59a640 1 -- 172.21.15.60:0/1197320928 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19080319e0 con 0x7f192015ae80 2023-10-16T14:11:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f191d59a640 1 -- 172.21.15.60:0/1197320928 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908031d70 con 0x7f192015ae80 2023-10-16T14:11:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 -- 172.21.15.60:0/1197320928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 msgr2=0x7f19200cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 --2- 172.21.15.60:0/1197320928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200cc560 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f19200ae3c0 tx=0x7f190802b920 comp rx=0 tx=0).stop 2023-10-16T14:11:20.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 -- 172.21.15.60:0/1197320928 shutdown_connections 2023-10-16T14:11:20.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 --2- 172.21.15.60:0/1197320928 >> v2:172.21.15.144:3300/0 conn(0x7f19200ccaa0 0x7f1920070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 --2- 172.21.15.60:0/1197320928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200cc560 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 --2- 172.21.15.60:0/1197320928 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192015aaa0 0x7f192015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.732+0000 7f1924cf2640 1 -- 172.21.15.60:0/1197320928 >> 172.21.15.60:0/1197320928 conn(0x7f19200cb410 msgr2=0x7f19200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.733+0000 7f1924cf2640 1 -- 172.21.15.60:0/1197320928 shutdown_connections 2023-10-16T14:11:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.733+0000 7f1924cf2640 1 -- 172.21.15.60:0/1197320928 wait complete. 2023-10-16T14:11:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.734+0000 7f1924cf2640 1 Processor -- start 2023-10-16T14:11:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.734+0000 7f1924cf2640 1 -- start start 2023-10-16T14:11:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.735+0000 7f1924cf2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19200ccaa0 0x7f192014dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.735+0000 7f1924cf2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015aaa0 0x7f19201523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.735+0000 7f1924cf2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.735+0000 7f1924cf2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1920152d60 con 0x7f192015ae80 2023-10-16T14:11:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.735+0000 7f1924cf2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1920153040 con 0x7f192015aaa0 2023-10-16T14:11:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.735+0000 7f1924cf2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1920153320 con 0x7f19200ccaa0 2023-10-16T14:11:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200c2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47032/0 (socket says 172.21.15.60:47032) 2023-10-16T14:11:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 -- 172.21.15.60:0/2803497421 learned_addr learned my addr 172.21.15.60:0/2803497421 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191dd9b640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015aaa0 0x7f19201523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191e59c640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19200ccaa0 0x7f192014dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 -- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19200ccaa0 msgr2=0x7f192014dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19200ccaa0 0x7f192014dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 -- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015aaa0 msgr2=0x7f19201523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015aaa0 0x7f19201523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 -- 172.21.15.60:0/2803497421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19080094d0 con 0x7f192015ae80 2023-10-16T14:11:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191e59c640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19200ccaa0 0x7f192014dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.736+0000 7f191ed9d640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200c2670 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f191800b410 tx=0x7f191800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.737+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918011830 con 0x7f192015ae80 2023-10-16T14:11:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.737+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19180119b0 con 0x7f192015ae80 2023-10-16T14:11:20.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.737+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918011d60 con 0x7f192015ae80 2023-10-16T14:11:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.737+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1920153600 con 0x7f192015ae80 2023-10-16T14:11:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.737+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192014e120 con 0x7f192015ae80 2023-10-16T14:11:20.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.738+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192015f530 con 0x7f192015ae80 2023-10-16T14:11:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.739+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f191801a400 con 0x7f192015ae80 2023-10-16T14:11:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.740+0000 7f1916ffd640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f190006b0b0 0x7f190006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.740+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f1918089ff0 con 0x7f192015ae80 2023-10-16T14:11:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.741+0000 7f191e59c640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f190006b0b0 0x7f190006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.742+0000 7f191e59c640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f190006b0b0 0x7f190006d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1910007eb0 tx=0x7f19100073c0 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:20.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.744+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19180bcfe0 con 0x7f192015ae80 2023-10-16T14:11:20.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.946+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1920153b20 con 0x7f192015ae80 2023-10-16T14:11:20.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.948+0000 7f1916ffd640 1 -- 172.21.15.60:0/2803497421 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+17915 (secure 0 0 0) 0x7f1920153b20 con 0x7f192015ae80 2023-10-16T14:11:20.953 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:20.953 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":240,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:18.095169+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6835/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f190006b0b0 msgr2=0x7f190006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f190006b0b0 0x7f190006d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1910007eb0 tx=0x7f19100073c0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 msgr2=0x7f19200c2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200c2670 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f191800b410 tx=0x7f191800b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 shutdown_connections 2023-10-16T14:11:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f190006b0b0 0x7f190006d570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f192015ae80 0x7f19200c2670 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f192015aaa0 0x7f19201523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 --2- 172.21.15.60:0/2803497421 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19200ccaa0 0x7f192014dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 >> 172.21.15.60:0/2803497421 conn(0x7f19200cb410 msgr2=0x7f192006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.951+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 shutdown_connections 2023-10-16T14:11:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:20.952+0000 7f1924cf2640 1 -- 172.21.15.60:0/2803497421 wait complete. 2023-10-16T14:11:20.975 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T14:11:20.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:21.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.188+0000 7fb2d7fff640 1 Processor -- start 2023-10-16T14:11:21.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.189+0000 7fb2d7fff640 1 -- start start 2023-10-16T14:11:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.190+0000 7fb2d7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2d8159950 0x7fb2d815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8070450 0x7fb2d80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d7fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d8070df0 con 0x7fb2d806c100 2023-10-16T14:11:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d7fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d815e490 con 0x7fb2d8159950 2023-10-16T14:11:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d815aef0 con 0x7fb2d8070450 2023-10-16T14:11:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb2d806c100 0x7fb2d806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35854/0 (socket says 172.21.15.60:35854) 2023-10-16T14:11:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d6ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2d8159950 0x7fb2d815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2d77fe640 1 -- 172.21.15.60:0/395302053 learned_addr learned my addr 172.21.15.60:0/395302053 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:21.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.191+0000 7fb2cffff640 1 --2- 172.21.15.60:0/395302053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8070450 0x7fb2d80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.192+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/395302053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8070450 msgr2=0x7fb2d80708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.192+0000 7fb2d6ffd640 1 --2- 172.21.15.60:0/395302053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8070450 0x7fb2d80708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.192+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/395302053 >> v1:172.21.15.60:6789/0 conn(0x7fb2d806c100 legacy=0x7fb2d806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.192+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/395302053 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d806cc80 con 0x7fb2d8159950 2023-10-16T14:11:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.193+0000 7fb2d6ffd640 1 --2- 172.21.15.60:0/395302053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d8159950 0x7fb2d815a9b0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80097a0 tx=0x7fb2c802da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628cd6200e49f295 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.193+0000 7fb2d67fc640 1 -- 172.21.15.60:0/395302053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8031850 con 0x7fb2d8159950 2023-10-16T14:11:21.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.193+0000 7fb2d67fc640 1 -- 172.21.15.60:0/395302053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c80319b0 con 0x7fb2d8159950 2023-10-16T14:11:21.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.193+0000 7fb2d67fc640 1 -- 172.21.15.60:0/395302053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8031d40 con 0x7fb2d8159950 2023-10-16T14:11:21.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 -- 172.21.15.60:0/395302053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d8159950 msgr2=0x7fb2d815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/395302053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d8159950 0x7fb2d815a9b0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80097a0 tx=0x7fb2c802da30 comp rx=0 tx=0).stop 2023-10-16T14:11:21.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 -- 172.21.15.60:0/395302053 shutdown_connections 2023-10-16T14:11:21.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/395302053 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8070450 0x7fb2d80708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/395302053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d8159950 0x7fb2d815a9b0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 -- 172.21.15.60:0/395302053 >> 172.21.15.60:0/395302053 conn(0x7fb2d80ce6a0 msgr2=0x7fb2d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:21.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 -- 172.21.15.60:0/395302053 shutdown_connections 2023-10-16T14:11:21.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.194+0000 7fb2d7fff640 1 -- 172.21.15.60:0/395302053 wait complete. 2023-10-16T14:11:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.196+0000 7fb2d7fff640 1 Processor -- start 2023-10-16T14:11:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.197+0000 7fb2d7fff640 1 -- start start 2023-10-16T14:11:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 0x7fb2d80c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2d8070450 0x7fb2d8152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8159950 0x7fb2d8156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d7fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80b3a90 con 0x7fb2d806c100 2023-10-16T14:11:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d7fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80c2800 con 0x7fb2d8070450 2023-10-16T14:11:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d7fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d81532c0 con 0x7fb2d8159950 2023-10-16T14:11:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 0x7fb2d80c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 0x7fb2d80c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47058/0 (socket says 172.21.15.60:47058) 2023-10-16T14:11:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.198+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/3313456873 learned_addr learned my addr 172.21.15.60:0/3313456873 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8159950 msgr2=0x7fb2d8156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2d6ffd640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8159950 0x7fb2d8156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2d8070450 msgr2=0x7fb2d8152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2d6ffd640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2d8070450 0x7fb2d8152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2d6ffd640 1 -- 172.21.15.60:0/3313456873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8009400 con 0x7fb2d806c100 2023-10-16T14:11:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2d6ffd640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 0x7fb2d80c5170 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7fb2c802e040 tx=0x7fb2c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c803ac50 con 0x7fb2d806c100 2023-10-16T14:11:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.199+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c803add0 con 0x7fb2d806c100 2023-10-16T14:11:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.200+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8044630 con 0x7fb2d806c100 2023-10-16T14:11:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.200+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d80c1ed0 con 0x7fb2d806c100 2023-10-16T14:11:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.200+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d80c2490 con 0x7fb2d806c100 2023-10-16T14:11:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.201+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d804e9b0 con 0x7fb2d806c100 2023-10-16T14:11:21.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.202+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb2c804d450 con 0x7fb2d806c100 2023-10-16T14:11:21.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.203+0000 7fb2cf7fe640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb2b806b0b0 0x7fb2b806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.203+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7fb2c80ac3f0 con 0x7fb2d806c100 2023-10-16T14:11:21.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.204+0000 7fb2cffff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb2b806b0b0 0x7fb2b806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.205+0000 7fb2cffff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb2b806b0b0 0x7fb2b806d570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb2d81f5a80 tx=0x7fb2c0008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:21.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.207+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2c8076d80 con 0x7fb2d806c100 2023-10-16T14:11:21.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.437+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d80c8af0 con 0x7fb2d806c100 2023-10-16T14:11:21.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.439+0000 7fb2cf7fe640 1 -- 172.21.15.60:0/3313456873 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4927 (secure 0 0 0) 0x7fb2c807a3a0 con 0x7fb2d806c100 2023-10-16T14:11:21.443 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:21.443 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":120,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":8123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/499924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":499924806},{"type":"v1","addr":"172.21.15.144:6837","nonce":499924806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:21.444 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 120 2023-10-16T14:11:21.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb2b806b0b0 msgr2=0x7fb2b806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb2b806b0b0 0x7fb2b806d570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb2d81f5a80 tx=0x7fb2c0008040 comp rx=0 tx=0).stop 2023-10-16T14:11:21.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 msgr2=0x7fb2d80c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 0x7fb2d80c5170 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7fb2c802e040 tx=0x7fb2c80046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 shutdown_connections 2023-10-16T14:11:21.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb2b806b0b0 0x7fb2b806d570 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb2d8159950 0x7fb2d8156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2d8070450 0x7fb2d8152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 --2- 172.21.15.60:0/3313456873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2d806c100 0x7fb2d80c5170 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.441+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 >> 172.21.15.60:0/3313456873 conn(0x7fb2d80ce6a0 msgr2=0x7fb2d80c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:21.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.442+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 shutdown_connections 2023-10-16T14:11:21.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.442+0000 7fb2d7fff640 1 -- 172.21.15.60:0/3313456873 wait complete. 2023-10-16T14:11:21.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.681+0000 7f58f642e640 1 Processor -- start 2023-10-16T14:11:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.681+0000 7f58f642e640 1 -- start start 2023-10-16T14:11:21.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f642e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f58f0065ab0 0x7f58f00c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f642e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f00acd70 con 0x7f58f00ce280 2023-10-16T14:11:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f642e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f00ac4a0 con 0x7f58f006d220 2023-10-16T14:11:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f642e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f0065f40 con 0x7f58f0065ab0 2023-10-16T14:11:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f5c2d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f58f00ce280 0x7f58f00ad200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35856/0 (socket says 172.21.15.60:35856) 2023-10-16T14:11:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f4c2b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f58f0065ab0 0x7f58f00c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.683+0000 7f58f5c2d640 1 -- 172.21.15.60:0/4257450601 learned_addr learned my addr 172.21.15.60:0/4257450601 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.684+0000 7f58f4c2b640 1 -- 172.21.15.60:0/4257450601 >> v1:172.21.15.144:6789/0 conn(0x7f58f006d220 legacy=0x7f58f00cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.684+0000 7f58f4c2b640 1 -- 172.21.15.60:0/4257450601 >> v1:172.21.15.60:6789/0 conn(0x7f58f00ce280 legacy=0x7f58f00ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.684+0000 7f58f4c2b640 1 -- 172.21.15.60:0/4257450601 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00ac720 con 0x7f58f0065ab0 2023-10-16T14:11:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.685+0000 7f58f4c2b640 1 --2- 172.21.15.60:0/4257450601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0065ab0 0x7f58f00c95e0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f58e00097a0 tx=0x7f58e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89baf5597b9a1131 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.685+0000 7f58e7fff640 1 -- 172.21.15.60:0/4257450601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e002e070 con 0x7f58f0065ab0 2023-10-16T14:11:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.685+0000 7f58e7fff640 1 -- 172.21.15.60:0/4257450601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e00315a0 con 0x7f58f0065ab0 2023-10-16T14:11:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.685+0000 7f58e7fff640 1 -- 172.21.15.60:0/4257450601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0031930 con 0x7f58f0065ab0 2023-10-16T14:11:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 -- 172.21.15.60:0/4257450601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0065ab0 msgr2=0x7f58f00c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 --2- 172.21.15.60:0/4257450601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0065ab0 0x7f58f00c95e0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f58e00097a0 tx=0x7f58e002b840 comp rx=0 tx=0).stop 2023-10-16T14:11:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 -- 172.21.15.60:0/4257450601 shutdown_connections 2023-10-16T14:11:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 --2- 172.21.15.60:0/4257450601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0065ab0 0x7f58f00c95e0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 -- 172.21.15.60:0/4257450601 >> 172.21.15.60:0/4257450601 conn(0x7f58f00b0a80 msgr2=0x7f58f00b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 -- 172.21.15.60:0/4257450601 shutdown_connections 2023-10-16T14:11:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.686+0000 7f58f642e640 1 -- 172.21.15.60:0/4257450601 wait complete. 2023-10-16T14:11:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.688+0000 7f58f642e640 1 Processor -- start 2023-10-16T14:11:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.688+0000 7f58f642e640 1 -- start start 2023-10-16T14:11:21.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.689+0000 7f58f642e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 0x7f58f014a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.689+0000 7f58f642e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00ce280 0x7f58f00b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.689+0000 7f58f642e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f009c630 0x7f58f00b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.689+0000 7f58f642e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f006c7f0 con 0x7f58f006d220 2023-10-16T14:11:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.689+0000 7f58f642e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f014aa10 con 0x7f58f009c630 2023-10-16T14:11:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.689+0000 7f58f642e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f0099ac0 con 0x7f58f00ce280 2023-10-16T14:11:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 0x7f58f014a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 0x7f58f014a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47076/0 (socket says 172.21.15.60:47076) 2023-10-16T14:11:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 -- 172.21.15.60:0/1768376426 learned_addr learned my addr 172.21.15.60:0/1768376426 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 -- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00ce280 msgr2=0x7f58f00b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00ce280 0x7f58f00b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f5c2d640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f009c630 0x7f58f00b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 -- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f009c630 msgr2=0x7f58f00b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f009c630 0x7f58f00b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 -- 172.21.15.60:0/1768376426 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e0009400 con 0x7f58f006d220 2023-10-16T14:11:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f5c2d640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f009c630 0x7f58f00b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.690+0000 7f58f542c640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 0x7f58f014a4d0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00c430 tx=0x7f58ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.691+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec011790 con 0x7f58f006d220 2023-10-16T14:11:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.691+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec0118f0 con 0x7f58f006d220 2023-10-16T14:11:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.691+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f00b20f0 con 0x7f58f006d220 2023-10-16T14:11:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.691+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec011ce0 con 0x7f58f006d220 2023-10-16T14:11:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.691+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f00b45f0 con 0x7f58f006d220 2023-10-16T14:11:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.692+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f004e9b0 con 0x7f58f006d220 2023-10-16T14:11:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.693+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f58ec01c460 con 0x7f58f006d220 2023-10-16T14:11:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.694+0000 7f58e5ffb640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f58d406b0b0 0x7f58d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.694+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7f58ec089cd0 con 0x7f58f006d220 2023-10-16T14:11:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.695+0000 7f58f4c2b640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f58d406b0b0 0x7f58d406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.696+0000 7f58f4c2b640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f58d406b0b0 0x7f58d406d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58e0036000 tx=0x7f58e0030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.699+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58ec0546e0 con 0x7f58f006d220 2023-10-16T14:11:21.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.938+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58f00b0f00 con 0x7f58f006d220 2023-10-16T14:11:21.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.939+0000 7f58e5ffb640 1 -- 172.21.15.60:0/1768376426 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4927 (secure 0 0 0) 0x7f58ec057d00 con 0x7f58f006d220 2023-10-16T14:11:21.943 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:21.944 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":120,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":8123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/499924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":499924806},{"type":"v1","addr":"172.21.15.144:6837","nonce":499924806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:10:40.444083+0000","modified":"2023-10-16T14:10:43.520528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10998},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10998":{"gid":10998,"name":"c","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.60:6835/462373098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":462373098},{"type":"v1","addr":"172.21.15.60:6835","nonce":462373098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T14:11:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 120 2023-10-16T14:11:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f58d406b0b0 msgr2=0x7f58d406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f58d406b0b0 0x7f58d406d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58e0036000 tx=0x7f58e0030000 comp rx=0 tx=0).stop 2023-10-16T14:11:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 msgr2=0x7f58f014a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:21.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 0x7f58f014a4d0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00c430 tx=0x7f58ec00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 shutdown_connections 2023-10-16T14:11:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f58d406b0b0 0x7f58d406d570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f58f00ce280 0x7f58f00b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58f009c630 0x7f58f00b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 --2- 172.21.15.60:0/1768376426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f006d220 0x7f58f014a4d0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 >> 172.21.15.60:0/1768376426 conn(0x7f58f00b0a80 msgr2=0x7f58f0070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 shutdown_connections 2023-10-16T14:11:21.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:21.942+0000 7f58f642e640 1 -- 172.21.15.60:0/1768376426 wait complete. 2023-10-16T14:11:21.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:22.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.188+0000 7fc81f8a2640 1 Processor -- start 2023-10-16T14:11:22.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.188+0000 7fc81f8a2640 1 -- start start 2023-10-16T14:11:22.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81f8a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc81815ae80 0x7fc81815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81f8a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8180ce390 0x7fc818070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81f8a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81815ed60 con 0x7fc81815aaa0 2023-10-16T14:11:22.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81f8a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81806b810 con 0x7fc81815ae80 2023-10-16T14:11:22.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81f8a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8180707e0 con 0x7fc8180ce390 2023-10-16T14:11:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81ce3d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc81815ae80 0x7fc81815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81de3f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8180ce390 0x7fc818070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81ce3d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc81815ae80 0x7fc81815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:44702/0 (socket says 172.21.15.60:44702) 2023-10-16T14:11:22.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.190+0000 7fc81ce3d640 1 -- 172.21.15.60:0/4097769098 learned_addr learned my addr 172.21.15.60:0/4097769098 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:22.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.191+0000 7fc81ce3d640 1 -- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8180ce390 msgr2=0x7fc818070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:22.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.191+0000 7fc81ce3d640 1 --2- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8180ce390 0x7fc818070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.191+0000 7fc81ce3d640 1 -- 172.21.15.60:0/4097769098 >> v1:172.21.15.144:6789/0 conn(0x7fc81815aaa0 legacy=0x7fc8180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.191+0000 7fc80ffff640 1 -- 172.21.15.60:0/4097769098 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856087445 0 0) 0x7fc81815ed60 con 0x7fc81815aaa0 2023-10-16T14:11:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.191+0000 7fc81ce3d640 1 -- 172.21.15.60:0/4097769098 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81815ef90 con 0x7fc81815ae80 2023-10-16T14:11:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.191+0000 7fc81de3f640 1 --2- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8180ce390 0x7fc818070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:11:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.193+0000 7fc81ce3d640 1 --2- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc81815ae80 0x7fc81815e820 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc808009870 tx=0x7fc80802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2ef6c521da7009e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.193+0000 7fc80ffff640 1 -- 172.21.15.60:0/4097769098 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808030840 con 0x7fc81815ae80 2023-10-16T14:11:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.193+0000 7fc80ffff640 1 -- 172.21.15.60:0/4097769098 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8080309a0 con 0x7fc81815ae80 2023-10-16T14:11:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.193+0000 7fc80ffff640 1 -- 172.21.15.60:0/4097769098 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808030d30 con 0x7fc81815ae80 2023-10-16T14:11:22.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 -- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc81815ae80 msgr2=0x7fc81815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc81815ae80 0x7fc81815e820 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc808009870 tx=0x7fc80802b980 comp rx=0 tx=0).stop 2023-10-16T14:11:22.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 -- 172.21.15.60:0/4097769098 shutdown_connections 2023-10-16T14:11:22.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc8180ce390 0x7fc818070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/4097769098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc81815ae80 0x7fc81815e820 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 -- 172.21.15.60:0/4097769098 >> 172.21.15.60:0/4097769098 conn(0x7fc8180cb410 msgr2=0x7fc8180cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 -- 172.21.15.60:0/4097769098 shutdown_connections 2023-10-16T14:11:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.194+0000 7fc81f8a2640 1 -- 172.21.15.60:0/4097769098 wait complete. 2023-10-16T14:11:22.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.196+0000 7fc81f8a2640 1 Processor -- start 2023-10-16T14:11:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.196+0000 7fc81f8a2640 1 -- start start 2023-10-16T14:11:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81f8a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8180ce390 0x7fc8180c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81f8a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc81815aaa0 0x7fc818155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81f8a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 0x7fc8180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81f8a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8181f3950 con 0x7fc8181f28f0 2023-10-16T14:11:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81f8a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8180c9330 con 0x7fc8180ce390 2023-10-16T14:11:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81f8a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc818156450 con 0x7fc81815aaa0 2023-10-16T14:11:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81de3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 0x7fc8180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81de3f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 0x7fc8180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47080/0 (socket says 172.21.15.60:47080) 2023-10-16T14:11:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.197+0000 7fc81de3f640 1 -- 172.21.15.60:0/565823605 learned_addr learned my addr 172.21.15.60:0/565823605 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81ce3d640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc81815aaa0 0x7fc818155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81d63e640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8180ce390 0x7fc8180c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81de3f640 1 -- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc81815aaa0 msgr2=0x7fc818155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81de3f640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc81815aaa0 0x7fc818155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81de3f640 1 -- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8180ce390 msgr2=0x7fc8180c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81de3f640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8180ce390 0x7fc8180c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81de3f640 1 -- 172.21.15.60:0/565823605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810009a70 con 0x7fc8181f28f0 2023-10-16T14:11:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81ce3d640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc81815aaa0 0x7fc818155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc81de3f640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 0x7fc8180ac890 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fc81000efc0 tx=0x7fc810009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810009d70 con 0x7fc8181f28f0 2023-10-16T14:11:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81000f040 con 0x7fc8181f28f0 2023-10-16T14:11:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.198+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810003990 con 0x7fc8181f28f0 2023-10-16T14:11:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.199+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8080094d0 con 0x7fc8181f28f0 2023-10-16T14:11:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.199+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8181f3390 con 0x7fc8181f28f0 2023-10-16T14:11:22.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.200+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81804e160 con 0x7fc8181f28f0 2023-10-16T14:11:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.201+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc8100144a0 con 0x7fc8181f28f0 2023-10-16T14:11:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.202+0000 7fc80dffb640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc7fc06b0b0 0x7fc7fc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.202+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7fc81008aca0 con 0x7fc8181f28f0 2023-10-16T14:11:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.202+0000 7fc81d63e640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc7fc06b0b0 0x7fc7fc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.204+0000 7fc81d63e640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc7fc06b0b0 0x7fc7fc06d570 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc814007fd0 tx=0x7fc814009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:22.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.207+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc810055450 con 0x7fc8181f28f0 2023-10-16T14:11:22.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.405+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8180cdc60 con 0x7fc8181f28f0 2023-10-16T14:11:22.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.408+0000 7fc80dffb640 1 -- 172.21.15.60:0/565823605 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v240) v1 ==== 74+0+17915 (secure 0 0 0) 0x7fc810058a70 con 0x7fc8181f28f0 2023-10-16T14:11:22.412 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:22.413 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":240,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:18.095169+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:10:38.740891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:10:39.464305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6835/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:22.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc7fc06b0b0 msgr2=0x7fc7fc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc7fc06b0b0 0x7fc7fc06d570 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc814007fd0 tx=0x7fc814009040 comp rx=0 tx=0).stop 2023-10-16T14:11:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 msgr2=0x7fc8180ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 0x7fc8180ac890 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fc81000efc0 tx=0x7fc810009f90 comp rx=0 tx=0).stop 2023-10-16T14:11:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 shutdown_connections 2023-10-16T14:11:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fc7fc06b0b0 0x7fc7fc06d570 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc81815aaa0 0x7fc818155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8181f28f0 0x7fc8180ac890 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 --2- 172.21.15.60:0/565823605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8180ce390 0x7fc8180c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 >> 172.21.15.60:0/565823605 conn(0x7fc8180cb410 msgr2=0x7fc8180b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 shutdown_connections 2023-10-16T14:11:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.411+0000 7fc81f8a2640 1 -- 172.21.15.60:0/565823605 wait complete. 2023-10-16T14:11:22.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T14:11:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.664+0000 7fb3f8a90640 1 Processor -- start 2023-10-16T14:11:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.665+0000 7fb3f8a90640 1 -- start start 2023-10-16T14:11:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.666+0000 7fb3f8a90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 0x7fb3f40ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.666+0000 7fb3f8a90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40cb030 con 0x7fb3f406a4b0 2023-10-16T14:11:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.666+0000 7fb3f8a90640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40c91c0 con 0x7fb3f415ad70 2023-10-16T14:11:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.666+0000 7fb3f8a90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40c94a0 con 0x7fb3f40caac0 2023-10-16T14:11:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.667+0000 7fb3ebfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3f406a4b0 0x7fb3f40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35860/0 (socket says 172.21.15.60:35860) 2023-10-16T14:11:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.667+0000 7fb3ebfff640 1 -- 172.21.15.60:0/456309332 learned_addr learned my addr 172.21.15.60:0/456309332 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:22.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.667+0000 7fb3f2d9d640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 0x7fb3f40ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.667+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34969841 0 0) 0x7fb3f40cb030 con 0x7fb3f406a4b0 2023-10-16T14:11:22.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.667+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e0003520 con 0x7fb3f406a4b0 2023-10-16T14:11:22.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789035425 0 0) 0x7fb3e0003520 con 0x7fb3f406a4b0 2023-10-16T14:11:22.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 msgr2=0x7fb3f40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:22.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 0x7fb3f40ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 >> v1:172.21.15.144:6789/0 conn(0x7fb3f415ad70 legacy=0x7fb3f4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f40c9780 con 0x7fb3f406a4b0 2023-10-16T14:11:22.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb3dc001670 con 0x7fb3f406a4b0 2023-10-16T14:11:22.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 >> v1:172.21.15.60:6789/0 conn(0x7fb3f406a4b0 legacy=0x7fb3f40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3e00039a0 0x7fb3e0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3e0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 0x7fb3e0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40c94a0 con 0x7fb3e0004350 2023-10-16T14:11:22.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e000b980 con 0x7fb3e00039a0 2023-10-16T14:11:22.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e000bb40 con 0x7fb3f40caac0 2023-10-16T14:11:22.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.668+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3e0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3f259c640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3e00039a0 0x7fb3e0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3f2d9d640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 0x7fb3e0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3ebfff640 1 -- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 msgr2=0x7fb3e0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40caac0 0x7fb3e0005080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3ebfff640 1 -- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3e00039a0 msgr2=0x7fb3e0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3e00039a0 0x7fb3e0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.669+0000 7fb3ebfff640 1 -- 172.21.15.60:0/456309332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e0003520 con 0x7fb3e0004350 2023-10-16T14:11:22.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.670+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3e0004820 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c190 tx=0x7fb3e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792f6a34cb0f96df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.670+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc005ad0 con 0x7fb3e0004350 2023-10-16T14:11:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.670+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc005c50 con 0x7fb3e0004350 2023-10-16T14:11:22.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.670+0000 7fb3f1d9b640 1 -- 172.21.15.60:0/456309332 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc02c830 con 0x7fb3e0004350 2023-10-16T14:11:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f8a90640 1 -- 172.21.15.60:0/456309332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 msgr2=0x7fb3e0004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/456309332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3e0004820 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c190 tx=0x7fb3e00057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f2d9d640 1 -- 172.21.15.60:0/456309332 reap_dead start 2023-10-16T14:11:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f8a90640 1 -- 172.21.15.60:0/456309332 shutdown_connections 2023-10-16T14:11:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f8a90640 1 -- 172.21.15.60:0/456309332 >> 172.21.15.60:0/456309332 conn(0x7fb3f40ce9b0 msgr2=0x7fb3f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:22.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f8a90640 1 -- 172.21.15.60:0/456309332 shutdown_connections 2023-10-16T14:11:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.671+0000 7fb3f8a90640 1 -- 172.21.15.60:0/456309332 wait complete. 2023-10-16T14:11:22.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.673+0000 7fb3f8a90640 1 Processor -- start 2023-10-16T14:11:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.673+0000 7fb3f8a90640 1 -- start start 2023-10-16T14:11:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f8a90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3f40c3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f8a90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40c3ea0 0x7fb3f40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f8a90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f40acdd0 0x7fb3f40ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f8a90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40c5210 con 0x7fb3e0004350 2023-10-16T14:11:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f8a90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40c4690 con 0x7fb3f40acdd0 2023-10-16T14:11:22.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f8a90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40c4970 con 0x7fb3f40c3ea0 2023-10-16T14:11:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3f40c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.674+0000 7fb3f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3f40c3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47108/0 (socket says 172.21.15.60:47108) 2023-10-16T14:11:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 -- 172.21.15.60:0/152160104 learned_addr learned my addr 172.21.15.60:0/152160104 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:22.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f2d9d640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f40acdd0 0x7fb3f40ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40c3ea0 0x7fb3f40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 -- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40c3ea0 msgr2=0x7fb3f40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40c3ea0 0x7fb3f40ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 -- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f40acdd0 msgr2=0x7fb3f40ad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f40acdd0 0x7fb3f40ad250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 -- 172.21.15.60:0/152160104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f40c4c50 con 0x7fb3e0004350 2023-10-16T14:11:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f2d9d640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f40acdd0 0x7fb3f40ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.675+0000 7fb3f259c640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3f40c3960 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400e5c0 tx=0x7fb3e400ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.676+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40052e0 con 0x7fb3e0004350 2023-10-16T14:11:22.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.676+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4005440 con 0x7fb3e0004350 2023-10-16T14:11:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.676+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f40c4f30 con 0x7fb3e0004350 2023-10-16T14:11:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.676+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40037f0 con 0x7fb3e0004350 2023-10-16T14:11:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.676+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f40ae070 con 0x7fb3e0004350 2023-10-16T14:11:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.677+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f404e160 con 0x7fb3e0004350 2023-10-16T14:11:22.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.678+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb3e4003950 con 0x7fb3e0004350 2023-10-16T14:11:22.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.679+0000 7fb3eaffd640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb3d406b0b0 0x7fb3d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.679+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6624+0+0 (secure 0 0 0) 0x7fb3e4089c40 con 0x7fb3e0004350 2023-10-16T14:11:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.680+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb3d406b0b0 0x7fb3d406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.681+0000 7fb3ebfff640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb3d406b0b0 0x7fb3d406d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c190 tx=0x7fb3dc009030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:22.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.683+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3e4054610 con 0x7fb3e0004350 2023-10-16T14:11:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.924+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fb3f40ca820 con 0x7fb3e0004350 2023-10-16T14:11:22.928 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:11:22.925+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:11:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.929+0000 7fb3eaffd640 1 -- 172.21.15.60:0/152160104 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v121) v1 ==== 167+0+0 (secure 0 0 0) 0x7fb3e4057c30 con 0x7fb3e0004350 2023-10-16T14:11:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:11:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb3d406b0b0 msgr2=0x7fb3d406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb3d406b0b0 0x7fb3d406d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c190 tx=0x7fb3dc009030 comp rx=0 tx=0).stop 2023-10-16T14:11:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 msgr2=0x7fb3f40c3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3f40c3960 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400e5c0 tx=0x7fb3e400ea80 comp rx=0 tx=0).stop 2023-10-16T14:11:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 shutdown_connections 2023-10-16T14:11:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fb3d406b0b0 0x7fb3d406d570 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3f40c3ea0 0x7fb3f40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3f40acdd0 0x7fb3f40ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 --2- 172.21.15.60:0/152160104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3e0004350 0x7fb3f40c3960 secure :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400e5c0 tx=0x7fb3e400ea80 comp rx=0 tx=0).stop 2023-10-16T14:11:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.932+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 >> 172.21.15.60:0/152160104 conn(0x7fb3f40ce9b0 msgr2=0x7fb3f40afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.933+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 shutdown_connections 2023-10-16T14:11:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:22.933+0000 7fb3f8a90640 1 -- 172.21.15.60:0/152160104 wait complete. 2023-10-16T14:11:22.947 INFO:tasks.ceph.mds.c.smithi060.stderr: -2191> 2023-10-16T14:10:42.767+0000 7f2d1d243640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:11:22.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T14:11:23.008 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:11:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.155+0000 7f4cb7b1d640 1 Processor -- start 2023-10-16T14:11:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.155+0000 7f4cb7b1d640 1 -- start start 2023-10-16T14:11:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.156+0000 7f4cb7b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4cb00cf710 0x7f4cb00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.156+0000 7f4cb7b1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 0x7f4cb00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.156+0000 7f4cb7b1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00d3980 con 0x7f4cb00cfaf0 2023-10-16T14:11:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.156+0000 7f4cb7b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb00d3c60 con 0x7f4cb00cf710 2023-10-16T14:11:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.156+0000 7f4cb7b1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00ce630 con 0x7f4cb00b6cd0 2023-10-16T14:11:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb50b8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4cb00cfaf0 0x7f4cb00d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35880/0 (socket says 172.21.15.60:35880) 2023-10-16T14:11:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb50b8640 1 -- 172.21.15.60:0/1642190871 learned_addr learned my addr 172.21.15.60:0/1642190871 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb60ba640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 0x7f4cb00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb58b9640 1 --2- 172.21.15.60:0/1642190871 >> v2:172.21.15.144:3300/0 conn(0x7f4cb00cf710 0x7f4cb00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327617968 0 0) 0x7f4cb00d3980 con 0x7f4cb00cfaf0 2023-10-16T14:11:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c98003520 con 0x7f4cb00cfaf0 2023-10-16T14:11:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849234834 0 0) 0x7f4c98003520 con 0x7f4cb00cfaf0 2023-10-16T14:11:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 msgr2=0x7f4cb00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 0x7f4cb00b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 >> v2:172.21.15.144:3300/0 conn(0x7f4cb00cf710 msgr2=0x7f4cb00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 --2- 172.21.15.60:0/1642190871 >> v2:172.21.15.144:3300/0 conn(0x7f4cb00cf710 0x7f4cb00b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00ce910 con 0x7f4cb00cfaf0 2023-10-16T14:11:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.157+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4ca4001670 con 0x7f4cb00cfaf0 2023-10-16T14:11:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 >> v1:172.21.15.60:6789/0 conn(0x7f4cb00cfaf0 legacy=0x7f4cb00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c98003970 0x7f4c98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4c98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 0x7f4c98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00d3c60 con 0x7f4c980047c0 2023-10-16T14:11:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00ce630 con 0x7f4c98003970 2023-10-16T14:11:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9800ba60 con 0x7f4cb00b6cd0 2023-10-16T14:11:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4c98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb58b9640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c98003970 0x7f4c98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb60ba640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 0x7f4c98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb50b8640 1 -- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 msgr2=0x7f4c98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b6cd0 0x7f4c98005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb50b8640 1 -- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c98003970 msgr2=0x7f4c98003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c98003970 0x7f4c98003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.158+0000 7f4cb50b8640 1 -- 172.21.15.60:0/1642190871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98003520 con 0x7f4c980047c0 2023-10-16T14:11:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4c98004c40 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4001eb0 tx=0x7f4c980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e6413ebff7af1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4005ad0 con 0x7f4c980047c0 2023-10-16T14:11:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4005c30 con 0x7f4c980047c0 2023-10-16T14:11:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb48b7640 1 -- 172.21.15.60:0/1642190871 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca402c7d0 con 0x7f4c980047c0 2023-10-16T14:11:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1642190871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 msgr2=0x7f4c98004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1642190871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4c98004c40 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4001eb0 tx=0x7f4c980057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb60ba640 1 -- 172.21.15.60:0/1642190871 reap_dead start 2023-10-16T14:11:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1642190871 shutdown_connections 2023-10-16T14:11:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.160+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1642190871 >> 172.21.15.60:0/1642190871 conn(0x7f4cb00d2cc0 msgr2=0x7f4cb00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.161+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1642190871 shutdown_connections 2023-10-16T14:11:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.161+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1642190871 wait complete. 2023-10-16T14:11:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.162+0000 7f4cb7b1d640 1 Processor -- start 2023-10-16T14:11:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.162+0000 7f4cb7b1d640 1 -- start start 2023-10-16T14:11:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.163+0000 7f4cb7b1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4cb00cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.163+0000 7f4cb7b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb00c9d90 0x7f4cb00b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.163+0000 7f4cb7b1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b2ae0 0x7f4cb00b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.163+0000 7f4cb7b1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00b34b0 con 0x7f4c980047c0 2023-10-16T14:11:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.163+0000 7f4cb7b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00cd0f0 con 0x7f4cb00c9d90 2023-10-16T14:11:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.163+0000 7f4cb7b1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00ca1f0 con 0x7f4cb00b2ae0 2023-10-16T14:11:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4cb00cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4cb00cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47134/0 (socket says 172.21.15.60:47134) 2023-10-16T14:11:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 -- 172.21.15.60:0/1356445230 learned_addr learned my addr 172.21.15.60:0/1356445230 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb00c9d90 0x7f4cb00b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb60ba640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b2ae0 0x7f4cb00b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 -- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b2ae0 msgr2=0x7f4cb00b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b2ae0 0x7f4cb00b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 -- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb00c9d90 msgr2=0x7f4cb00b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb00c9d90 0x7f4cb00b1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb58b9640 1 -- 172.21.15.60:0/1356445230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00cc850 con 0x7f4c980047c0 2023-10-16T14:11:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb00c9d90 0x7f4cb00b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.164+0000 7f4cb60ba640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b2ae0 0x7f4cb00b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.165+0000 7f4cb58b9640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4cb00cc220 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0007fb0 tx=0x7f4ca0011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.165+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca000e030 con 0x7f4c980047c0 2023-10-16T14:11:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.165+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca0011e90 con 0x7f4c980047c0 2023-10-16T14:11:23.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.165+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00168c0 con 0x7f4c980047c0 2023-10-16T14:11:23.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.165+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb00ccb30 con 0x7f4c980047c0 2023-10-16T14:11:23.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.165+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb00afb30 con 0x7f4c980047c0 2023-10-16T14:11:23.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.166+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb00afcf0 con 0x7f4c980047c0 2023-10-16T14:11:23.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.167+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4ca0016a20 con 0x7f4c980047c0 2023-10-16T14:11:23.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.168+0000 7f4cae7fc640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f4c9c06b0b0 0x7f4c9c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:23.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.169+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6710+0+0 (secure 0 0 0) 0x7f4ca000f070 con 0x7f4c980047c0 2023-10-16T14:11:23.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.169+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f4c9c06b0b0 0x7f4c9c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:23.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.170+0000 7f4cb50b8640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f4c9c06b0b0 0x7f4c9c06d570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4034000 tx=0x7f4ca4007000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.173+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ca00558b0 con 0x7f4c980047c0 2023-10-16T14:11:23.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.408+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4cb004e160 con 0x7f4c980047c0 2023-10-16T14:11:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.932+0000 7f4cae7fc640 1 -- 172.21.15.60:0/1356445230 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v122) v1 ==== 107+0+0 (secure 0 0 0) 0x7f4ca0058ed0 con 0x7f4c980047c0 2023-10-16T14:11:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f4c9c06b0b0 msgr2=0x7f4c9c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f4c9c06b0b0 0x7f4c9c06d570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4034000 tx=0x7f4ca4007000 comp rx=0 tx=0).stop 2023-10-16T14:11:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 msgr2=0x7f4cb00cc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4cb00cc220 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0007fb0 tx=0x7f4ca0011660 comp rx=0 tx=0).stop 2023-10-16T14:11:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 shutdown_connections 2023-10-16T14:11:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f4c9c06b0b0 0x7f4c9c06d570 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4cb00b2ae0 0x7f4cb00b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb00c9d90 0x7f4cb00b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 --2- 172.21.15.60:0/1356445230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c980047c0 0x7f4cb00cc220 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 >> 172.21.15.60:0/1356445230 conn(0x7f4cb00d2cc0 msgr2=0x7f4cb015d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 shutdown_connections 2023-10-16T14:11:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:23.936+0000 7f4cb7b1d640 1 -- 172.21.15.60:0/1356445230 wait complete. 2023-10-16T14:11:23.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T14:11:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.180+0000 7ff3ee640640 1 Processor -- start 2023-10-16T14:11:24.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.181+0000 7ff3ee640640 1 -- start start 2023-10-16T14:11:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.182+0000 7ff3ee640640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3e815cc80 0x7ff3e80cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:24.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.182+0000 7ff3ee640640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3e80cdd60 0x7ff3e8071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.182+0000 7ff3ee640640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80d12f0 con 0x7ff3e8071fa0 2023-10-16T14:11:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.182+0000 7ff3ee640640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e80d0500 con 0x7ff3e80cdd60 2023-10-16T14:11:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.182+0000 7ff3ee640640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e80ce1f0 con 0x7ff3e815cc80 2023-10-16T14:11:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3e80cdd60 0x7ff3e8071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:24.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3e80cdd60 0x7ff3e8071a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47138/0 (socket says 172.21.15.60:47138) 2023-10-16T14:11:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 -- 172.21.15.60:0/1428669150 learned_addr learned my addr 172.21.15.60:0/1428669150 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ed63e640 1 --2- 172.21.15.60:0/1428669150 >> v2:172.21.15.144:3300/0 conn(0x7ff3e815cc80 0x7ff3e80cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 -- 172.21.15.60:0/1428669150 >> v2:172.21.15.144:3300/0 conn(0x7ff3e815cc80 msgr2=0x7ff3e80cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 --2- 172.21.15.60:0/1428669150 >> v2:172.21.15.144:3300/0 conn(0x7ff3e815cc80 0x7ff3e80cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 -- 172.21.15.60:0/1428669150 >> v1:172.21.15.144:6789/0 conn(0x7ff3e8071fa0 legacy=0x7ff3e81607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.183+0000 7ff3ece3d640 1 -- 172.21.15.60:0/1428669150 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e80723d0 con 0x7ff3e80cdd60 2023-10-16T14:11:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.184+0000 7ff3ece3d640 1 --2- 172.21.15.60:0/1428669150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e80cdd60 0x7ff3e8071a60 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80ce370 tx=0x7ff3d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6eccfbd6ec5c449 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.185+0000 7ff3dffff640 1 -- 172.21.15.60:0/1428669150 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8031870 con 0x7ff3e80cdd60 2023-10-16T14:11:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.185+0000 7ff3dffff640 1 -- 172.21.15.60:0/1428669150 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d80319d0 con 0x7ff3e80cdd60 2023-10-16T14:11:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.185+0000 7ff3dffff640 1 -- 172.21.15.60:0/1428669150 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8031d60 con 0x7ff3e80cdd60 2023-10-16T14:11:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 -- 172.21.15.60:0/1428669150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e80cdd60 msgr2=0x7ff3e8071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 --2- 172.21.15.60:0/1428669150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e80cdd60 0x7ff3e8071a60 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80ce370 tx=0x7ff3d802b910 comp rx=0 tx=0).stop 2023-10-16T14:11:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 -- 172.21.15.60:0/1428669150 shutdown_connections 2023-10-16T14:11:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 --2- 172.21.15.60:0/1428669150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e80cdd60 0x7ff3e8071a60 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 --2- 172.21.15.60:0/1428669150 >> v2:172.21.15.144:3300/0 conn(0x7ff3e815cc80 0x7ff3e80cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 -- 172.21.15.60:0/1428669150 >> 172.21.15.60:0/1428669150 conn(0x7ff3e8065ed0 msgr2=0x7ff3e80662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 -- 172.21.15.60:0/1428669150 shutdown_connections 2023-10-16T14:11:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.186+0000 7ff3ee640640 1 -- 172.21.15.60:0/1428669150 wait complete. 2023-10-16T14:11:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.188+0000 7ff3ee640640 1 Processor -- start 2023-10-16T14:11:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.188+0000 7ff3ee640640 1 -- start start 2023-10-16T14:11:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ee640640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 0x7ff3e80b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ee640640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3e80cdd60 0x7ff3e81534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ee640640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3e815cc80 0x7ff3e8153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ee640640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80b0340 con 0x7ff3e8071fa0 2023-10-16T14:11:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ee640640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815a240 con 0x7ff3e80cdd60 2023-10-16T14:11:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ee640640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815a4b0 con 0x7ff3e815cc80 2023-10-16T14:11:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 0x7ff3e80b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 0x7ff3e80b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47154/0 (socket says 172.21.15.60:47154) 2023-10-16T14:11:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 -- 172.21.15.60:0/4047241346 learned_addr learned my addr 172.21.15.60:0/4047241346 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 -- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3e815cc80 msgr2=0x7ff3e8153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3e815cc80 0x7ff3e8153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 -- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3e80cdd60 msgr2=0x7ff3e81534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3e80cdd60 0x7ff3e81534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.189+0000 7ff3ed63e640 1 -- 172.21.15.60:0/4047241346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80094d0 con 0x7ff3e8071fa0 2023-10-16T14:11:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.190+0000 7ff3ed63e640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 0x7ff3e80b3cf0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4007f20 tx=0x7ff3e400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.190+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4014620 con 0x7ff3e8071fa0 2023-10-16T14:11:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.190+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e4014780 con 0x7ff3e8071fa0 2023-10-16T14:11:24.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.190+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4014b30 con 0x7ff3e8071fa0 2023-10-16T14:11:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.190+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e815a790 con 0x7ff3e8071fa0 2023-10-16T14:11:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.190+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e81f3160 con 0x7ff3e8071fa0 2023-10-16T14:11:24.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.191+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e81f3350 con 0x7ff3e8071fa0 2023-10-16T14:11:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.192+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff3e4014c90 con 0x7ff3e8071fa0 2023-10-16T14:11:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.193+0000 7ff3ddffb640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff3cc06b0e0 0x7ff3cc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:24.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.193+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6710+0+0 (secure 0 0 0) 0x7ff3e408a770 con 0x7ff3e8071fa0 2023-10-16T14:11:24.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.194+0000 7ff3ece3d640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff3cc06b0e0 0x7ff3cc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:24.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.195+0000 7ff3ece3d640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff3cc06b0e0 0x7ff3cc06d5a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8036000 tx=0x7ff3d8030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:24.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.198+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3e4055190 con 0x7ff3e8071fa0 2023-10-16T14:11:24.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:24.393+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3e804e9b0 con 0x7ff3e8071fa0 2023-10-16T14:11:25.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.150+0000 7ff3ddffb640 1 -- 172.21.15.60:0/4047241346 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v242) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff3e40587b0 con 0x7ff3e8071fa0 2023-10-16T14:11:25.154 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_metadata' removed 2023-10-16T14:11:25.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff3cc06b0e0 msgr2=0x7ff3cc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:25.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff3cc06b0e0 0x7ff3cc06d5a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8036000 tx=0x7ff3d8030000 comp rx=0 tx=0).stop 2023-10-16T14:11:25.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 msgr2=0x7ff3e80b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:25.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 0x7ff3e80b3cf0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4007f20 tx=0x7ff3e400e570 comp rx=0 tx=0).stop 2023-10-16T14:11:25.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 shutdown_connections 2023-10-16T14:11:25.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7ff3cc06b0e0 0x7ff3cc06d5a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3e815cc80 0x7ff3e8153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3e80cdd60 0x7ff3e81534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 --2- 172.21.15.60:0/4047241346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3e8071fa0 0x7ff3e80b3cf0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.154+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 >> 172.21.15.60:0/4047241346 conn(0x7ff3e8065ed0 msgr2=0x7ff3e80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:25.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.157+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 shutdown_connections 2023-10-16T14:11:25.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.157+0000 7ff3ee640640 1 -- 172.21.15.60:0/4047241346 wait complete. 2023-10-16T14:11:25.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T14:11:25.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.413+0000 7f3ad7894640 1 Processor -- start 2023-10-16T14:11:25.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.414+0000 7f3ad7894640 1 -- start start 2023-10-16T14:11:25.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad7894640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad00c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:25.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad7894640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ad0159da0 0x7f3ad015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:25.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad7894640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad015dd10 con 0x7f3ad00cce20 2023-10-16T14:11:25.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad7894640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad015dfb0 con 0x7f3ad0159da0 2023-10-16T14:11:25.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad7894640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad015e290 con 0x7f3ad01599c0 2023-10-16T14:11:25.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad4e2f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ad0159da0 0x7f3ad015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:25.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad4e2f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3ad0159da0 0x7f3ad015d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47168/0 (socket says 172.21.15.60:47168) 2023-10-16T14:11:25.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.415+0000 7f3ad4e2f640 1 -- 172.21.15.60:0/1892598827 learned_addr learned my addr 172.21.15.60:0/1892598827 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:25.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.416+0000 7f3ad4e2f640 1 -- 172.21.15.60:0/1892598827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 msgr2=0x7f3ad00c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:25.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.416+0000 7f3ad4e2f640 1 --2- 172.21.15.60:0/1892598827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad00c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.416+0000 7f3ad4e2f640 1 -- 172.21.15.60:0/1892598827 >> v1:172.21.15.60:6789/0 conn(0x7f3ad00cce20 legacy=0x7f3ad006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:25.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.416+0000 7f3ad4e2f640 1 -- 172.21.15.60:0/1892598827 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad015e570 con 0x7f3ad0159da0 2023-10-16T14:11:25.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.417+0000 7f3ad4e2f640 1 --2- 172.21.15.60:0/1892598827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad015d7d0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009870 tx=0x7f3ac002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91abc238000b2d27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:25.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.418+0000 7f3ac7fff640 1 -- 172.21.15.60:0/1892598827 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0030840 con 0x7f3ad0159da0 2023-10-16T14:11:25.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.418+0000 7f3ac7fff640 1 -- 172.21.15.60:0/1892598827 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac00309a0 con 0x7f3ad0159da0 2023-10-16T14:11:25.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.418+0000 7f3ac7fff640 1 -- 172.21.15.60:0/1892598827 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0030d30 con 0x7f3ad0159da0 2023-10-16T14:11:25.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.419+0000 7f3ad7894640 1 -- 172.21.15.60:0/1892598827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 msgr2=0x7f3ad015d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:25.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.419+0000 7f3ad7894640 1 --2- 172.21.15.60:0/1892598827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad015d7d0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009870 tx=0x7f3ac002b980 comp rx=0 tx=0).stop 2023-10-16T14:11:25.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.421+0000 7f3ad7894640 1 -- 172.21.15.60:0/1892598827 shutdown_connections 2023-10-16T14:11:25.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.421+0000 7f3ad7894640 1 --2- 172.21.15.60:0/1892598827 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad015d7d0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.421+0000 7f3ad7894640 1 --2- 172.21.15.60:0/1892598827 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad00c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.421+0000 7f3ad7894640 1 -- 172.21.15.60:0/1892598827 >> 172.21.15.60:0/1892598827 conn(0x7f3ad00cb360 msgr2=0x7f3ad00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:25.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.422+0000 7f3ad7894640 1 -- 172.21.15.60:0/1892598827 shutdown_connections 2023-10-16T14:11:25.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.422+0000 7f3ad7894640 1 -- 172.21.15.60:0/1892598827 wait complete. 2023-10-16T14:11:25.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.425+0000 7f3ad7894640 1 Processor -- start 2023-10-16T14:11:25.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.425+0000 7f3ad7894640 1 -- start start 2023-10-16T14:11:25.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.427+0000 7f3ad7894640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad00cce20 0x7f3ad0155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:25.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.427+0000 7f3ad7894640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad015a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:25.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.427+0000 7f3ad7894640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad014e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:25.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.427+0000 7f3ad7894640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad015ac60 con 0x7f3ad0159da0 2023-10-16T14:11:25.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.427+0000 7f3ad7894640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad015af40 con 0x7f3ad00cce20 2023-10-16T14:11:25.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.427+0000 7f3ad7894640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad015b220 con 0x7f3ad01599c0 2023-10-16T14:11:25.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.428+0000 7f3ad5e31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad014e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:25.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.428+0000 7f3ad5e31640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad014e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47172/0 (socket says 172.21.15.60:47172) 2023-10-16T14:11:25.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.428+0000 7f3ad5e31640 1 -- 172.21.15.60:0/4091436805 learned_addr learned my addr 172.21.15.60:0/4091436805 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:25.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.428+0000 7f3ad4e2f640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad015a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:25.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad5e31640 1 -- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 msgr2=0x7f3ad015a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:25.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad5e31640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad015a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad5e31640 1 -- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad00cce20 msgr2=0x7f3ad0155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:25.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad5e31640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad00cce20 0x7f3ad0155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:25.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad5e31640 1 -- 172.21.15.60:0/4091436805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00094d0 con 0x7f3ad0159da0 2023-10-16T14:11:25.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad5e31640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad014e6c0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c7a0 tx=0x7f3ac800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:25.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8003b70 con 0x7f3ad0159da0 2023-10-16T14:11:25.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8003cf0 con 0x7f3ad0159da0 2023-10-16T14:11:25.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad015b500 con 0x7f3ad0159da0 2023-10-16T14:11:25.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8003ea0 con 0x7f3ad0159da0 2023-10-16T14:11:25.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.429+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad00c3ca0 con 0x7f3ad0159da0 2023-10-16T14:11:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.431+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad004e160 con 0x7f3ad0159da0 2023-10-16T14:11:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.432+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3ac800aa90 con 0x7f3ad0159da0 2023-10-16T14:11:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.433+0000 7f3ac5ffb640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f3ab406b0b0 0x7f3ab406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.433+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6266+0+0 (secure 0 0 0) 0x7f3ac8089b70 con 0x7f3ad0159da0 2023-10-16T14:11:25.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.433+0000 7f3ad5630640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f3ab406b0b0 0x7f3ab406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:25.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.435+0000 7f3ad5630640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f3ab406b0b0 0x7f3ab406d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009710 tx=0x7f3acc006c90 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:25.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.437+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ac80545f0 con 0x7f3ad0159da0 2023-10-16T14:11:25.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:25.631+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ad00b4860 con 0x7f3ad0159da0 2023-10-16T14:11:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.153+0000 7f3ac5ffb640 1 -- 172.21.15.60:0/4091436805 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v243) v1 ==== 178+0+0 (secure 0 0 0) 0x7f3ac8057c10 con 0x7f3ad0159da0 2023-10-16T14:11:26.158 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_data' removed 2023-10-16T14:11:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.158+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f3ab406b0b0 msgr2=0x7f3ab406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.158+0000 7f3ad7894640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f3ab406b0b0 0x7f3ab406d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009710 tx=0x7f3acc006c90 comp rx=0 tx=0).stop 2023-10-16T14:11:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.158+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 msgr2=0x7f3ad014e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.158+0000 7f3ad7894640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad014e6c0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c7a0 tx=0x7f3ac800cc60 comp rx=0 tx=0).stop 2023-10-16T14:11:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.160+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 shutdown_connections 2023-10-16T14:11:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.160+0000 7f3ad7894640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f3ab406b0b0 0x7f3ab406d570 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.160+0000 7f3ad7894640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3ad01599c0 0x7f3ad015a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.160+0000 7f3ad7894640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ad0159da0 0x7f3ad014e6c0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.160+0000 7f3ad7894640 1 --2- 172.21.15.60:0/4091436805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad00cce20 0x7f3ad0155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.160+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 >> 172.21.15.60:0/4091436805 conn(0x7f3ad00cb360 msgr2=0x7f3ad00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.161+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 shutdown_connections 2023-10-16T14:11:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.161+0000 7f3ad7894640 1 -- 172.21.15.60:0/4091436805 wait complete. 2023-10-16T14:11:26.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:11:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.406+0000 7f351bfff640 1 Processor -- start 2023-10-16T14:11:26.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.406+0000 7f351bfff640 1 -- start start 2023-10-16T14:11:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f351bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f351c0b1ca0 0x7f351c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f351bfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c0d0970 con 0x7f351c0b2080 2023-10-16T14:11:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f351bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c0d0ea0 con 0x7f351c0cfc40 2023-10-16T14:11:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f351bfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351c0d0bc0 con 0x7f351c0b1ca0 2023-10-16T14:11:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f351affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f351c0b1ca0 0x7f351c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f3513fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f351c0b2080 0x7f351c0ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35902/0 (socket says 172.21.15.60:35902) 2023-10-16T14:11:26.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f3513fff640 1 -- 172.21.15.60:0/1987743563 learned_addr learned my addr 172.21.15.60:0/1987743563 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:26.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.408+0000 7f351affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f351c0b1ca0 0x7f351c0d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47180/0 (socket says 172.21.15.60:47180) 2023-10-16T14:11:26.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.409+0000 7f351affd640 1 -- 172.21.15.60:0/1987743563 >> v1:172.21.15.144:6789/0 conn(0x7f351c0cfc40 legacy=0x7f351c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:26.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.409+0000 7f351affd640 1 -- 172.21.15.60:0/1987743563 >> v1:172.21.15.60:6789/0 conn(0x7f351c0b2080 legacy=0x7f351c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.409+0000 7f351affd640 1 -- 172.21.15.60:0/1987743563 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0cd940 con 0x7f351c0b1ca0 2023-10-16T14:11:26.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.409+0000 7f351a7fc640 1 -- 172.21.15.60:0/1987743563 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325169173 0 0) 0x7f351c0d0970 con 0x7f351c0b2080 2023-10-16T14:11:26.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.410+0000 7f351affd640 1 --2- 172.21.15.60:0/1987743563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0d36b0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f350c0097a0 tx=0x7f350c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ba13050d876e71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:26.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.410+0000 7f351a7fc640 1 -- 172.21.15.60:0/1987743563 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c030610 con 0x7f351c0b1ca0 2023-10-16T14:11:26.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.410+0000 7f351a7fc640 1 -- 172.21.15.60:0/1987743563 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c030770 con 0x7f351c0b1ca0 2023-10-16T14:11:26.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.410+0000 7f351a7fc640 1 -- 172.21.15.60:0/1987743563 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c030b00 con 0x7f351c0b1ca0 2023-10-16T14:11:26.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 -- 172.21.15.60:0/1987743563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 msgr2=0x7f351c0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 --2- 172.21.15.60:0/1987743563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0d36b0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f350c0097a0 tx=0x7f350c02b960 comp rx=0 tx=0).stop 2023-10-16T14:11:26.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 -- 172.21.15.60:0/1987743563 shutdown_connections 2023-10-16T14:11:26.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 --2- 172.21.15.60:0/1987743563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0d36b0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 -- 172.21.15.60:0/1987743563 >> 172.21.15.60:0/1987743563 conn(0x7f351c0d2b90 msgr2=0x7f351c0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:26.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 -- 172.21.15.60:0/1987743563 shutdown_connections 2023-10-16T14:11:26.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.411+0000 7f351bfff640 1 -- 172.21.15.60:0/1987743563 wait complete. 2023-10-16T14:11:26.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.413+0000 7f351bfff640 1 Processor -- start 2023-10-16T14:11:26.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.413+0000 7f351bfff640 1 -- start start 2023-10-16T14:11:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b2080 0x7f351c1f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0cfc40 0x7f351c0b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0c7f60 con 0x7f351c0b1ca0 2023-10-16T14:11:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0c8240 con 0x7f351c0b2080 2023-10-16T14:11:26.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0c8520 con 0x7f351c0cfc40 2023-10-16T14:11:26.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0c7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47186/0 (socket says 172.21.15.60:47186) 2023-10-16T14:11:26.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.414+0000 7f351affd640 1 -- 172.21.15.60:0/3877636290 learned_addr learned my addr 172.21.15.60:0/3877636290 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:26.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f351affd640 1 -- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0cfc40 msgr2=0x7f351c0b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f351affd640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0cfc40 0x7f351c0b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f351affd640 1 -- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b2080 msgr2=0x7f351c1f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f351affd640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b2080 0x7f351c1f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f351affd640 1 -- 172.21.15.60:0/3877636290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c009400 con 0x7f351c0b1ca0 2023-10-16T14:11:26.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f351affd640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0c7710 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f350c0098c0 tx=0x7f350c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c03e070 con 0x7f351c0b1ca0 2023-10-16T14:11:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c039a10 con 0x7f351c0b1ca0 2023-10-16T14:11:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.415+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c039da0 con 0x7f351c0b1ca0 2023-10-16T14:11:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.416+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c0c8800 con 0x7f351c0b1ca0 2023-10-16T14:11:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.416+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c0c7c50 con 0x7f351c0b1ca0 2023-10-16T14:11:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.417+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c06b9d0 con 0x7f351c0b1ca0 2023-10-16T14:11:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.418+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f350c04b400 con 0x7f351c0b1ca0 2023-10-16T14:11:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.419+0000 7f35137fe640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34fc06b0b0 0x7f34fc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.419+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f350c0ac0d0 con 0x7f351c0b1ca0 2023-10-16T14:11:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.419+0000 7f3513fff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34fc06b0b0 0x7f34fc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.421+0000 7f3513fff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34fc06b0b0 0x7f34fc06d570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3504007fd0 tx=0x7f3504007960 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.423+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f350c076d10 con 0x7f351c0b1ca0 2023-10-16T14:11:26.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.618+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f351c1606b0 con 0x7f351c0b1ca0 2023-10-16T14:11:26.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.625+0000 7f35137fe640 1 -- 172.21.15.60:0/3877636290 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7f350c07a330 con 0x7f351c0b1ca0 2023-10-16T14:11:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34fc06b0b0 msgr2=0x7f34fc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34fc06b0b0 0x7f34fc06d570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3504007fd0 tx=0x7f3504007960 comp rx=0 tx=0).stop 2023-10-16T14:11:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 msgr2=0x7f351c0c7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0c7710 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f350c0098c0 tx=0x7f350c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 shutdown_connections 2023-10-16T14:11:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34fc06b0b0 0x7f34fc06d570 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f351c0cfc40 0x7f351c0b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f351c0b2080 0x7f351c1f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 --2- 172.21.15.60:0/3877636290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f351c0b1ca0 0x7f351c0c7710 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.628+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 >> 172.21.15.60:0/3877636290 conn(0x7f351c0d2b90 msgr2=0x7f351c0ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.629+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 shutdown_connections 2023-10-16T14:11:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.629+0000 7f351bfff640 1 -- 172.21.15.60:0/3877636290 wait complete. 2023-10-16T14:11:26.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:11:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.874+0000 7fa863190640 1 Processor -- start 2023-10-16T14:11:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.874+0000 7fa863190640 1 -- start start 2023-10-16T14:11:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa863190640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c15d150 0x7fa85c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa863190640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa85c06a4b0 0x7fa85c160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa863190640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c0ac890 con 0x7fa85c160b90 2023-10-16T14:11:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa863190640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85c15d530 con 0x7fa85c06a4b0 2023-10-16T14:11:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa863190640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c161650 con 0x7fa85c15d150 2023-10-16T14:11:26.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa860f2c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c15d150 0x7fa85c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.876+0000 7fa86172d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa85c06a4b0 0x7fa85c160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.877+0000 7fa860f2c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c15d150 0x7fa85c0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:35634/0 (socket says 172.21.15.60:35634) 2023-10-16T14:11:26.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.877+0000 7fa860f2c640 1 -- 172.21.15.60:0/2554035366 learned_addr learned my addr 172.21.15.60:0/2554035366 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:26.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.877+0000 7fa86172d640 1 -- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c15d150 msgr2=0x7fa85c0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:26.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.877+0000 7fa86172d640 1 --2- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c15d150 0x7fa85c0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.877+0000 7fa86172d640 1 -- 172.21.15.60:0/2554035366 >> v1:172.21.15.144:6789/0 conn(0x7fa85c160b90 legacy=0x7fa85c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.877+0000 7fa86172d640 1 -- 172.21.15.60:0/2554035366 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c161930 con 0x7fa85c06a4b0 2023-10-16T14:11:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.879+0000 7fa86172d640 1 --2- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c06a4b0 0x7fa85c160fb0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa854009d90 tx=0x7fa85402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ae22ecba3555f34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.879+0000 7fa85b7fe640 1 -- 172.21.15.60:0/2554035366 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85402cc50 con 0x7fa85c06a4b0 2023-10-16T14:11:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.879+0000 7fa85b7fe640 1 -- 172.21.15.60:0/2554035366 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85402cdb0 con 0x7fa85c06a4b0 2023-10-16T14:11:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.879+0000 7fa85b7fe640 1 -- 172.21.15.60:0/2554035366 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854034660 con 0x7fa85c06a4b0 2023-10-16T14:11:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 -- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c06a4b0 msgr2=0x7fa85c160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 --2- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c06a4b0 0x7fa85c160fb0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa854009d90 tx=0x7fa85402bea0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 -- 172.21.15.60:0/2554035366 shutdown_connections 2023-10-16T14:11:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 --2- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c06a4b0 0x7fa85c160fb0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 --2- 172.21.15.60:0/2554035366 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c15d150 0x7fa85c0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 -- 172.21.15.60:0/2554035366 >> 172.21.15.60:0/2554035366 conn(0x7fa85c0ce9b0 msgr2=0x7fa85c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 -- 172.21.15.60:0/2554035366 shutdown_connections 2023-10-16T14:11:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.880+0000 7fa863190640 1 -- 172.21.15.60:0/2554035366 wait complete. 2023-10-16T14:11:26.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.882+0000 7fa863190640 1 Processor -- start 2023-10-16T14:11:26.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.882+0000 7fa863190640 1 -- start start 2023-10-16T14:11:26.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa863190640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c06a4b0 0x7fa85c15e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa863190640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c15d150 0x7fa85c153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa863190640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 0x7fa85c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa863190640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c15eed0 con 0x7fa85c160b90 2023-10-16T14:11:26.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa863190640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c1541b0 con 0x7fa85c15d150 2023-10-16T14:11:26.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa863190640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c154530 con 0x7fa85c06a4b0 2023-10-16T14:11:26.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa86172d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 0x7fa85c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa86172d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 0x7fa85c0c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47200/0 (socket says 172.21.15.60:47200) 2023-10-16T14:11:26.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.883+0000 7fa86172d640 1 -- 172.21.15.60:0/3732449391 learned_addr learned my addr 172.21.15.60:0/3732449391 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:26.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa860f2c640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c06a4b0 0x7fa85c15e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa85bfff640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c15d150 0x7fa85c153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa86172d640 1 -- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c06a4b0 msgr2=0x7fa85c15e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa86172d640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c06a4b0 0x7fa85c15e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa86172d640 1 -- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c15d150 msgr2=0x7fa85c153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa86172d640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c15d150 0x7fa85c153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa86172d640 1 -- 172.21.15.60:0/3732449391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854009a70 con 0x7fa85c160b90 2023-10-16T14:11:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa860f2c640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c06a4b0 0x7fa85c15e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa85bfff640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c15d150 0x7fa85c153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:26.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa86172d640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 0x7fa85c0c3de0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fa854009eb0 tx=0x7fa854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:26.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85403e610 con 0x7fa85c160b90 2023-10-16T14:11:26.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85403e770 con 0x7fa85c160b90 2023-10-16T14:11:26.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.884+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85403eb20 con 0x7fa85c160b90 2023-10-16T14:11:26.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.885+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c15f100 con 0x7fa85c160b90 2023-10-16T14:11:26.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.885+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c15f6c0 con 0x7fa85c160b90 2023-10-16T14:11:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.886+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c04e160 con 0x7fa85c160b90 2023-10-16T14:11:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.887+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa85403b050 con 0x7fa85c160b90 2023-10-16T14:11:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.888+0000 7fa8597fa640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa84006b0b0 0x7fa84006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.888+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fa8540ac3f0 con 0x7fa85c160b90 2023-10-16T14:11:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.888+0000 7fa860f2c640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa84006b0b0 0x7fa84006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.890+0000 7fa860f2c640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa84006b0b0 0x7fa84006d570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0b08b0 tx=0x7fa850008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:26.892+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa854077130 con 0x7fa85c160b90 2023-10-16T14:11:27.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.091+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa85c0c4980 con 0x7fa85c160b90 2023-10-16T14:11:27.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.098+0000 7fa8597fa640 1 -- 172.21.15.60:0/3732449391 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa85407a750 con 0x7fa85c160b90 2023-10-16T14:11:27.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa84006b0b0 msgr2=0x7fa84006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa84006b0b0 0x7fa84006d570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0b08b0 tx=0x7fa850008040 comp rx=0 tx=0).stop 2023-10-16T14:11:27.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 msgr2=0x7fa85c0c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 0x7fa85c0c3de0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fa854009eb0 tx=0x7fa854004a40 comp rx=0 tx=0).stop 2023-10-16T14:11:27.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 shutdown_connections 2023-10-16T14:11:27.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa84006b0b0 0x7fa84006d570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa85c160b90 0x7fa85c0c3de0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85c15d150 0x7fa85c153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 --2- 172.21.15.60:0/3732449391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa85c06a4b0 0x7fa85c15e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.101+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 >> 172.21.15.60:0/3732449391 conn(0x7fa85c0ce9b0 msgr2=0x7fa85c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:27.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.102+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 shutdown_connections 2023-10-16T14:11:27.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.102+0000 7fa863190640 1 -- 172.21.15.60:0/3732449391 wait complete. 2023-10-16T14:11:27.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:11:27.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.354+0000 7f2c5fc61640 1 Processor -- start 2023-10-16T14:11:27.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.354+0000 7f2c5fc61640 1 -- start start 2023-10-16T14:11:27.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5fc61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c580ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5fc61640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c58160b90 0x7f2c58160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5fc61640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580ac890 con 0x7f2c58069da0 2023-10-16T14:11:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5fc61640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5815d530 con 0x7f2c58160b90 2023-10-16T14:11:27.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5fc61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58161650 con 0x7f2c5815d150 2023-10-16T14:11:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5d1fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c58160b90 0x7f2c58160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5e1fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2c58069da0 0x7f2c5806a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35916/0 (socket says 172.21.15.60:35916) 2023-10-16T14:11:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5e1fe640 1 -- 172.21.15.60:0/1158702061 learned_addr learned my addr 172.21.15.60:0/1158702061 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.356+0000 7f2c5d9fd640 1 --2- 172.21.15.60:0/1158702061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c580ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.357+0000 7f2c5d1fc640 1 -- 172.21.15.60:0/1158702061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 msgr2=0x7f2c580ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.357+0000 7f2c5d1fc640 1 --2- 172.21.15.60:0/1158702061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c580ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.357+0000 7f2c5d1fc640 1 -- 172.21.15.60:0/1158702061 >> v1:172.21.15.60:6789/0 conn(0x7f2c58069da0 legacy=0x7f2c5806a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.357+0000 7f2c5d1fc640 1 -- 172.21.15.60:0/1158702061 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58161930 con 0x7f2c58160b90 2023-10-16T14:11:27.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.357+0000 7f2c5c9fb640 1 -- 172.21.15.60:0/1158702061 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382826107 0 0) 0x7f2c580ac890 con 0x7f2c58069da0 2023-10-16T14:11:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.358+0000 7f2c5d1fc640 1 --2- 172.21.15.60:0/1158702061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58160b90 0x7f2c58160fb0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f2c480097a0 tx=0x7f2c4802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdda61f2a142792d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:27.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.358+0000 7f2c5c9fb640 1 -- 172.21.15.60:0/1158702061 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4802d040 con 0x7f2c58160b90 2023-10-16T14:11:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.358+0000 7f2c5c9fb640 1 -- 172.21.15.60:0/1158702061 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c48031810 con 0x7f2c58160b90 2023-10-16T14:11:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.358+0000 7f2c5c9fb640 1 -- 172.21.15.60:0/1158702061 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48031ba0 con 0x7f2c58160b90 2023-10-16T14:11:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1158702061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58160b90 msgr2=0x7f2c58160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1158702061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58160b90 0x7f2c58160fb0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f2c480097a0 tx=0x7f2c4802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1158702061 shutdown_connections 2023-10-16T14:11:27.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1158702061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58160b90 0x7f2c58160fb0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1158702061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c580ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1158702061 >> 172.21.15.60:0/1158702061 conn(0x7f2c580ce9b0 msgr2=0x7f2c580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:27.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1158702061 shutdown_connections 2023-10-16T14:11:27.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.359+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1158702061 wait complete. 2023-10-16T14:11:27.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.361+0000 7f2c5fc61640 1 Processor -- start 2023-10-16T14:11:27.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.361+0000 7f2c5fc61640 1 -- start start 2023-10-16T14:11:27.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.362+0000 7f2c5fc61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 0x7f2c581521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.362+0000 7f2c5fc61640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c5815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.362+0000 7f2c5fc61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c58160b90 0x7f2c5815c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.362+0000 7f2c5fc61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c581526e0 con 0x7f2c58069da0 2023-10-16T14:11:27.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.362+0000 7f2c5fc61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580c49f0 con 0x7f2c58160b90 2023-10-16T14:11:27.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.362+0000 7f2c5fc61640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580c4cd0 con 0x7f2c5815d150 2023-10-16T14:11:27.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 0x7f2c581521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 0x7f2c581521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47216/0 (socket says 172.21.15.60:47216) 2023-10-16T14:11:27.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 -- 172.21.15.60:0/1853574910 learned_addr learned my addr 172.21.15.60:0/1853574910 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:27.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d1fc640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c5815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5e1fe640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c58160b90 0x7f2c5815c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 -- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 msgr2=0x7f2c5815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c5815bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 -- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c58160b90 msgr2=0x7f2c5815c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c58160b90 0x7f2c5815c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 -- 172.21.15.60:0/1853574910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c48009400 con 0x7f2c58069da0 2023-10-16T14:11:27.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5e1fe640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c58160b90 0x7f2c5815c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:27.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.363+0000 7f2c5d9fd640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 0x7f2c581521a0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400e9b0 tx=0x7f2c5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:27.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.364+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54009670 con 0x7f2c58069da0 2023-10-16T14:11:27.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.364+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5400c9a0 con 0x7f2c58069da0 2023-10-16T14:11:27.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.364+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5400cd50 con 0x7f2c58069da0 2023-10-16T14:11:27.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.364+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c580c4fb0 con 0x7f2c58069da0 2023-10-16T14:11:27.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.364+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c580c3c50 con 0x7f2c58069da0 2023-10-16T14:11:27.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.365+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5804e160 con 0x7f2c58069da0 2023-10-16T14:11:27.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.366+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2c54013470 con 0x7f2c58069da0 2023-10-16T14:11:27.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.367+0000 7f2c4e7fc640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2c3c06b0b0 0x7f2c3c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.367+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f2c54089d90 con 0x7f2c58069da0 2023-10-16T14:11:27.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.368+0000 7f2c5d1fc640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2c3c06b0b0 0x7f2c3c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.369+0000 7f2c5d1fc640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2c3c06b0b0 0x7f2c3c06d570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c48036000 tx=0x7f2c4803c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:27.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.372+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2c54054a40 con 0x7f2c58069da0 2023-10-16T14:11:27.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.567+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2c580c5c20 con 0x7f2c58069da0 2023-10-16T14:11:27.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.574+0000 7f2c4e7fc640 1 -- 172.21.15.60:0/1853574910 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2c54058060 con 0x7f2c58069da0 2023-10-16T14:11:27.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2c3c06b0b0 msgr2=0x7f2c3c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2c3c06b0b0 0x7f2c3c06d570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c48036000 tx=0x7f2c4803c040 comp rx=0 tx=0).stop 2023-10-16T14:11:27.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 msgr2=0x7f2c581521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 0x7f2c581521a0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400e9b0 tx=0x7f2c5400ee70 comp rx=0 tx=0).stop 2023-10-16T14:11:27.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 shutdown_connections 2023-10-16T14:11:27.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2c3c06b0b0 0x7f2c3c06d570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c5815d150 0x7f2c5815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c58160b90 0x7f2c5815c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 --2- 172.21.15.60:0/1853574910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c58069da0 0x7f2c581521a0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.577+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 >> 172.21.15.60:0/1853574910 conn(0x7f2c580ce9b0 msgr2=0x7f2c580c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:27.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.578+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 shutdown_connections 2023-10-16T14:11:27.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.578+0000 7f2c5fc61640 1 -- 172.21.15.60:0/1853574910 wait complete. 2023-10-16T14:11:27.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T14:11:27.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.828+0000 7f8e4526f640 1 Processor -- start 2023-10-16T14:11:27.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.828+0000 7f8e4526f640 1 -- start start 2023-10-16T14:11:27.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.829+0000 7f8e4526f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015ae80 0x7f8e4015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.829+0000 7f8e4526f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e400ce390 0x7f8e40070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.829+0000 7f8e4526f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4015ed60 con 0x7f8e4015aaa0 2023-10-16T14:11:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.829+0000 7f8e4526f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4006b810 con 0x7f8e400ce390 2023-10-16T14:11:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.829+0000 7f8e4526f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e400707e0 con 0x7f8e4015ae80 2023-10-16T14:11:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e400ce390 0x7f8e40070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e400ce390 0x7f8e40070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47220/0 (socket says 172.21.15.60:47220) 2023-10-16T14:11:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 -- 172.21.15.60:0/1494980126 learned_addr learned my addr 172.21.15.60:0/1494980126 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3e7fc640 1 --2- 172.21.15.60:0/1494980126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015ae80 0x7f8e4015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 -- 172.21.15.60:0/1494980126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015ae80 msgr2=0x7f8e4015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 --2- 172.21.15.60:0/1494980126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015ae80 0x7f8e4015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 -- 172.21.15.60:0/1494980126 >> v1:172.21.15.144:6789/0 conn(0x7f8e4015aaa0 legacy=0x7f8e400cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.830+0000 7f8e3f7fe640 1 -- 172.21.15.60:0/1494980126 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4015ef90 con 0x7f8e400ce390 2023-10-16T14:11:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.831+0000 7f8e3f7fe640 1 --2- 172.21.15.60:0/1494980126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40070f50 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009e50 tx=0x7f8e3802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a6f70830b04117 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.832+0000 7f8e3dffb640 1 -- 172.21.15.60:0/1494980126 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3802ca80 con 0x7f8e400ce390 2023-10-16T14:11:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.832+0000 7f8e3dffb640 1 -- 172.21.15.60:0/1494980126 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3802cbe0 con 0x7f8e400ce390 2023-10-16T14:11:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.832+0000 7f8e3dffb640 1 -- 172.21.15.60:0/1494980126 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e380343f0 con 0x7f8e400ce390 2023-10-16T14:11:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.832+0000 7f8e4526f640 1 -- 172.21.15.60:0/1494980126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 msgr2=0x7f8e40070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.832+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1494980126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40070f50 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009e50 tx=0x7f8e3802bee0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.833+0000 7f8e4526f640 1 -- 172.21.15.60:0/1494980126 shutdown_connections 2023-10-16T14:11:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.833+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1494980126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40070f50 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.833+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1494980126 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015ae80 0x7f8e4015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.833+0000 7f8e4526f640 1 -- 172.21.15.60:0/1494980126 >> 172.21.15.60:0/1494980126 conn(0x7f8e400cb410 msgr2=0x7f8e400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.833+0000 7f8e4526f640 1 -- 172.21.15.60:0/1494980126 shutdown_connections 2023-10-16T14:11:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.833+0000 7f8e4526f640 1 -- 172.21.15.60:0/1494980126 wait complete. 2023-10-16T14:11:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.834+0000 7f8e4526f640 1 Processor -- start 2023-10-16T14:11:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.835+0000 7f8e4526f640 1 -- start start 2023-10-16T14:11:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e4526f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e4526f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015aaa0 0x7f8e400ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e4526f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e4015ae80 0x7f8e40157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e4526f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e401578c0 con 0x7f8e400ce390 2023-10-16T14:11:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e4526f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e40156740 con 0x7f8e4015ae80 2023-10-16T14:11:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e4526f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e400ca870 con 0x7f8e4015aaa0 2023-10-16T14:11:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40155ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47230/0 (socket says 172.21.15.60:47230) 2023-10-16T14:11:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 -- 172.21.15.60:0/1105063750 learned_addr learned my addr 172.21.15.60:0/1105063750 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3f7fe640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e4015ae80 0x7f8e40157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3e7fc640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015aaa0 0x7f8e400ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 -- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015aaa0 msgr2=0x7f8e400ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015aaa0 0x7f8e400ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 -- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e4015ae80 msgr2=0x7f8e40157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e4015ae80 0x7f8e40157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.836+0000 7f8e3effd640 1 -- 172.21.15.60:0/1105063750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38009b30 con 0x7f8e400ce390 2023-10-16T14:11:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.837+0000 7f8e3effd640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40155ea0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000b3f0 tx=0x7f8e3000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.837+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30011830 con 0x7f8e400ce390 2023-10-16T14:11:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.837+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e30011990 con 0x7f8e400ce390 2023-10-16T14:11:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.837+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30011d40 con 0x7f8e400ce390 2023-10-16T14:11:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.837+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4015b380 con 0x7f8e400ce390 2023-10-16T14:11:27.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.838+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4015b940 con 0x7f8e400ce390 2023-10-16T14:11:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.839+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4004e160 con 0x7f8e400ce390 2023-10-16T14:11:27.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.840+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8e3001a400 con 0x7f8e400ce390 2023-10-16T14:11:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.841+0000 7f8e277fe640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f8e2c06b080 0x7f8e2c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.841+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f8e30089960 con 0x7f8e400ce390 2023-10-16T14:11:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.841+0000 7f8e3e7fc640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f8e2c06b080 0x7f8e2c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:27.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.843+0000 7f8e3e7fc640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f8e2c06b080 0x7f8e2c06d540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e40159ed0 tx=0x7f8e34008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:27.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:27.846+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e30054590 con 0x7f8e400ce390 2023-10-16T14:11:28.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.113+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f8e4006c7a0 con 0x7f8e400ce390 2023-10-16T14:11:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.172+0000 7f8e277fe640 1 -- 172.21.15.60:0/1105063750 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3229) v1 ==== 186+0+0 (secure 0 0 0) 0x7f8e30015020 con 0x7f8e400ce390 2023-10-16T14:11:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f8e2c06b080 msgr2=0x7f8e2c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f8e2c06b080 0x7f8e2c06d540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e40159ed0 tx=0x7f8e34008040 comp rx=0 tx=0).stop 2023-10-16T14:11:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 msgr2=0x7f8e40155ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40155ea0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000b3f0 tx=0x7f8e3000b8b0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 shutdown_connections 2023-10-16T14:11:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f8e2c06b080 0x7f8e2c06d540 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8e4015aaa0 0x7f8e400ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e4015ae80 0x7f8e40157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 --2- 172.21.15.60:0/1105063750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e400ce390 0x7f8e40155ea0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 >> 172.21.15.60:0/1105063750 conn(0x7f8e400cb410 msgr2=0x7f8e4006f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 shutdown_connections 2023-10-16T14:11:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.175+0000 7f8e4526f640 1 -- 172.21.15.60:0/1105063750 wait complete. 2023-10-16T14:11:28.199 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2023-10-16T14:11:28.199 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2023-10-16T14:11:28.200 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T14:11:28.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:28.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:11:28.305 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:11:28.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.416+0000 7f7bcab5a640 1 Processor -- start 2023-10-16T14:11:28.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.416+0000 7f7bcab5a640 1 -- start start 2023-10-16T14:11:28.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bcab5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bcab5a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7bc40b0410 0x7f7bc40b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bcab5a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bc415d3f0 0x7f7bc415d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bcab5a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc40b0db0 con 0x7f7bc40b0410 2023-10-16T14:11:28.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bcab5a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc415e3e0 con 0x7f7bc415d3f0 2023-10-16T14:11:28.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bcab5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40cd4a0 con 0x7f7bc40cd0c0 2023-10-16T14:11:28.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bc3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7bc40b0410 0x7f7bc40b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bc3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7bc40b0410 0x7f7bc40b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47244/0 (socket says 172.21.15.60:47244) 2023-10-16T14:11:28.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bc3fff640 1 -- 172.21.15.60:0/1783058345 learned_addr learned my addr 172.21.15.60:0/1783058345 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:28.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bc90f7640 1 --2- 172.21.15.60:0/1783058345 >> v2:172.21.15.144:3300/0 conn(0x7f7bc415d3f0 0x7f7bc415d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.418+0000 7f7bc88f6640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc3fff640 1 -- 172.21.15.60:0/1783058345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 msgr2=0x7f7bc41615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:28.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc3fff640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc41615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc3fff640 1 -- 172.21.15.60:0/1783058345 >> v2:172.21.15.144:3300/0 conn(0x7f7bc415d3f0 msgr2=0x7f7bc415d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:28.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc3fff640 1 --2- 172.21.15.60:0/1783058345 >> v2:172.21.15.144:3300/0 conn(0x7f7bc415d3f0 0x7f7bc415d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc3fff640 1 -- 172.21.15.60:0/1783058345 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc415e100 con 0x7f7bc40b0410 2023-10-16T14:11:28.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc90f7640 1 --2- 172.21.15.60:0/1783058345 >> v2:172.21.15.144:3300/0 conn(0x7f7bc415d3f0 0x7f7bc415d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:28.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.419+0000 7f7bc88f6640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:28.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.420+0000 7f7bc3fff640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc40b0410 0x7f7bc40b0870 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f7bc406da30 tx=0x7f7bb402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed522053a077707 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.420+0000 7f7bc37fe640 1 -- 172.21.15.60:0/1783058345 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40317e0 con 0x7f7bc40b0410 2023-10-16T14:11:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.420+0000 7f7bc37fe640 1 -- 172.21.15.60:0/1783058345 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb4031940 con 0x7f7bc40b0410 2023-10-16T14:11:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.420+0000 7f7bc37fe640 1 -- 172.21.15.60:0/1783058345 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4031cd0 con 0x7f7bc40b0410 2023-10-16T14:11:28.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1783058345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc40b0410 msgr2=0x7f7bc40b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc40b0410 0x7f7bc40b0870 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f7bc406da30 tx=0x7f7bb402b880 comp rx=0 tx=0).stop 2023-10-16T14:11:28.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1783058345 shutdown_connections 2023-10-16T14:11:28.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1783058345 >> v2:172.21.15.144:3300/0 conn(0x7f7bc415d3f0 0x7f7bc415d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc40b0410 0x7f7bc40b0870 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1783058345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1783058345 >> 172.21.15.60:0/1783058345 conn(0x7f7bc40cb410 msgr2=0x7f7bc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1783058345 shutdown_connections 2023-10-16T14:11:28.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.421+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1783058345 wait complete. 2023-10-16T14:11:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.423+0000 7f7bcab5a640 1 Processor -- start 2023-10-16T14:11:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.423+0000 7f7bcab5a640 1 -- start start 2023-10-16T14:11:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bcab5a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc40b0410 0x7f7bc40add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bcab5a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc4157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bcab5a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 0x7f7bc4159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bcab5a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc40b1cd0 con 0x7f7bc415d3f0 2023-10-16T14:11:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bcab5a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc415a960 con 0x7f7bc40b0410 2023-10-16T14:11:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bcab5a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc415a3b0 con 0x7f7bc40cd0c0 2023-10-16T14:11:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bc90f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 0x7f7bc4159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bc90f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 0x7f7bc4159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47248/0 (socket says 172.21.15.60:47248) 2023-10-16T14:11:28.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.424+0000 7f7bc90f7640 1 -- 172.21.15.60:0/1621472714 learned_addr learned my addr 172.21.15.60:0/1621472714 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc88f6640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc40b0410 0x7f7bc40add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc3fff640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc4157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc90f7640 1 -- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 msgr2=0x7f7bc4157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc90f7640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc4157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc90f7640 1 -- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc40b0410 msgr2=0x7f7bc40add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc90f7640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc40b0410 0x7f7bc40add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc90f7640 1 -- 172.21.15.60:0/1621472714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40094d0 con 0x7f7bc415d3f0 2023-10-16T14:11:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc88f6640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc40b0410 0x7f7bc40add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc3fff640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc4157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.425+0000 7f7bc90f7640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 0x7f7bc4159e70 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc007f20 tx=0x7f7bbc00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.426+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00dee0 con 0x7f7bc415d3f0 2023-10-16T14:11:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.426+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc00f040 con 0x7f7bc415d3f0 2023-10-16T14:11:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.426+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc003dd0 con 0x7f7bc415d3f0 2023-10-16T14:11:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.426+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc415a680 con 0x7f7bc415d3f0 2023-10-16T14:11:28.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.426+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc40b1960 con 0x7f7bc415d3f0 2023-10-16T14:11:28.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.427+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc40b1e80 con 0x7f7bc415d3f0 2023-10-16T14:11:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.428+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7bbc004030 con 0x7f7bc415d3f0 2023-10-16T14:11:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.429+0000 7f7bc17fa640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7bb006b0e0 0x7f7bb006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.429+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f7bbc089f90 con 0x7f7bc415d3f0 2023-10-16T14:11:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.430+0000 7f7bc88f6640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7bb006b0e0 0x7f7bb006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.431+0000 7f7bc88f6640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7bb006b0e0 0x7f7bb006d5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8006fd0 tx=0x7f7bb8008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:28.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.433+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7bc40b1e80 con 0x7f7bc415d3f0 2023-10-16T14:11:28.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.635+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bc40b1e80 con 0x7f7bc415d3f0 2023-10-16T14:11:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.636+0000 7f7bc17fa640 1 -- 172.21.15.60:0/1621472714 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+14319 (secure 0 0 0) 0x7f7bc40b1e80 con 0x7f7bc415d3f0 2023-10-16T14:11:28.640 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:28.640 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":243,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:26.150910+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/462373098":"2023-10-17T14:11:22.926057+0000","172.21.15.60:6834/462373098":"2023-10-17T14:11:22.926057+0000","172.21.15.144:6837/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:0/2316191718":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1718343640":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/360800697":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/3906119794":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6835/2009359724":"2023-10-17T14:11:14.995834+0000","172.21.15.144:6833/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1394204497":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/561725004":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/2722365176":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1462669726":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2449287245":"2023-10-17T14:11:09.018721+0000","172.21.15.144:0/1709631677":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/136559516":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/2366541172":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/3137196642":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/483659352":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1532733597":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6833/1724812344":"2023-10-17T14:10:58.183289+0000","172.21.15.144:0/1906349341":"2023-10-17T14:11:09.018721+0000","172.21.15.144:6832/1724812344":"2023-10-17T14:10:58.183289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:28.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7bb006b0e0 msgr2=0x7f7bb006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7bb006b0e0 0x7f7bb006d5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8006fd0 tx=0x7f7bb8008040 comp rx=0 tx=0).stop 2023-10-16T14:11:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 msgr2=0x7f7bc4159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 0x7f7bc4159e70 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc007f20 tx=0x7f7bbc00d560 comp rx=0 tx=0).stop 2023-10-16T14:11:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 shutdown_connections 2023-10-16T14:11:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7bb006b0e0 0x7f7bb006d5a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc40cd0c0 0x7f7bc4157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc415d3f0 0x7f7bc4159e70 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.638+0000 7f7bcab5a640 1 --2- 172.21.15.60:0/1621472714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc40b0410 0x7f7bc40add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.639+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 >> 172.21.15.60:0/1621472714 conn(0x7f7bc40cb410 msgr2=0x7f7bc415cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.639+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 shutdown_connections 2023-10-16T14:11:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.639+0000 7f7bcab5a640 1 -- 172.21.15.60:0/1621472714 wait complete. 2023-10-16T14:11:28.663 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:11:28.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:11:28.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.939+0000 7f406a99a640 1 Processor -- start 2023-10-16T14:11:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.939+0000 7f406a99a640 1 -- start start 2023-10-16T14:11:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.940+0000 7f406a99a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 0x7f406415aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.940+0000 7f406a99a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406406be30 con 0x7f40640b0ea0 2023-10-16T14:11:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.940+0000 7f406a99a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40640ada10 con 0x7f40640b3ec0 2023-10-16T14:11:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.940+0000 7f406a99a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406406ca00 con 0x7f40640cd970 2023-10-16T14:11:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.941+0000 7f4069197640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f40640b0ea0 0x7f406415a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35120/0 (socket says 172.21.15.60:35120) 2023-10-16T14:11:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.941+0000 7f4069197640 1 -- 172.21.15.60:0/2206289409 learned_addr learned my addr 172.21.15.60:0/2206289409 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.941+0000 7f406a199640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 0x7f406415aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.941+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462899145 0 0) 0x7f406406be30 con 0x7f40640b0ea0 2023-10-16T14:11:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.941+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4050003520 con 0x7f40640b0ea0 2023-10-16T14:11:28.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357901799 0 0) 0x7f4050003520 con 0x7f40640b0ea0 2023-10-16T14:11:28.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 msgr2=0x7f406415aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:28.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 0x7f406415aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 >> v1:172.21.15.144:6789/0 conn(0x7f40640b3ec0 legacy=0x7f40640cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406406c060 con 0x7f40640b0ea0 2023-10-16T14:11:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4054001670 con 0x7f40640b0ea0 2023-10-16T14:11:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 >> v1:172.21.15.60:6789/0 conn(0x7f40640b0ea0 legacy=0x7f406415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40500039a0 0x7f4050003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 0x7f40500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4050004d00 0x7f4050005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406406ca00 con 0x7f4050004d00 2023-10-16T14:11:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405000bb40 con 0x7f40500039a0 2023-10-16T14:11:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405000bd00 con 0x7f40640cd970 2023-10-16T14:11:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f406a199640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4050004d00 0x7f4050005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4069998640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40500039a0 0x7f4050003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.942+0000 7f4069197640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 0x7f40500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.943+0000 7f406a199640 1 -- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 msgr2=0x7f40500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.943+0000 7f406a199640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f40640cd970 0x7f40500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.943+0000 7f406a199640 1 -- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40500039a0 msgr2=0x7f4050003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.943+0000 7f406a199640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40500039a0 0x7f4050003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.943+0000 7f406a199640 1 -- 172.21.15.60:0/2206289409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4050003520 con 0x7f4050004d00 2023-10-16T14:11:28.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.944+0000 7f406a199640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4050004d00 0x7f4050005180 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f405c009f90 tx=0x7f40500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7edb3732d8d7847 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:28.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.944+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c02cbd0 con 0x7f4050004d00 2023-10-16T14:11:28.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.944+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c00f040 con 0x7f4050004d00 2023-10-16T14:11:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.944+0000 7f4068996640 1 -- 172.21.15.60:0/2206289409 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c034440 con 0x7f4050004d00 2023-10-16T14:11:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a99a640 1 -- 172.21.15.60:0/2206289409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4050004d00 msgr2=0x7f4050005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a99a640 1 --2- 172.21.15.60:0/2206289409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4050004d00 0x7f4050005180 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f405c009f90 tx=0x7f40500057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a199640 1 -- 172.21.15.60:0/2206289409 reap_dead start 2023-10-16T14:11:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a99a640 1 -- 172.21.15.60:0/2206289409 shutdown_connections 2023-10-16T14:11:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a99a640 1 -- 172.21.15.60:0/2206289409 >> 172.21.15.60:0/2206289409 conn(0x7f40640b0170 msgr2=0x7f40640b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a99a640 1 -- 172.21.15.60:0/2206289409 shutdown_connections 2023-10-16T14:11:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.945+0000 7f406a99a640 1 -- 172.21.15.60:0/2206289409 wait complete. 2023-10-16T14:11:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.947+0000 7f406a99a640 1 Processor -- start 2023-10-16T14:11:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.947+0000 7f406a99a640 1 -- start start 2023-10-16T14:11:28.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a99a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4050004d00 0x7f40640c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a99a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 0x7f40640aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a99a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40640c3930 0x7f40640c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a99a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640c3200 con 0x7f40641f2520 2023-10-16T14:11:28.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a99a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40641f2d70 con 0x7f40640c3930 2023-10-16T14:11:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a99a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40641f3050 con 0x7f4050004d00 2023-10-16T14:11:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 0x7f40640aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 0x7f40640aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50794/0 (socket says 172.21.15.60:50794) 2023-10-16T14:11:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 -- 172.21.15.60:0/449473725 learned_addr learned my addr 172.21.15.60:0/449473725 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069998640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4050004d00 0x7f40640c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 -- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4050004d00 msgr2=0x7f40640c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f406a199640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40640c3930 0x7f40640c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4050004d00 0x7f40640c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 -- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40640c3930 msgr2=0x7f40640c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40640c3930 0x7f40640c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.948+0000 7f4069197640 1 -- 172.21.15.60:0/449473725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40641f3330 con 0x7f40641f2520 2023-10-16T14:11:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f4069998640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4050004d00 0x7f40640c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f4069197640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 0x7f40640aed20 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f405400e800 tx=0x7f405400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405400c870 con 0x7f40641f2520 2023-10-16T14:11:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405400c9d0 con 0x7f40641f2520 2023-10-16T14:11:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405400cd80 con 0x7f40641f2520 2023-10-16T14:11:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40640c3410 con 0x7f40641f2520 2023-10-16T14:11:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.949+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40640af260 con 0x7f40641f2520 2023-10-16T14:11:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.950+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40640c3600 con 0x7f40641f2520 2023-10-16T14:11:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.951+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f405401d460 con 0x7f40641f2520 2023-10-16T14:11:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.952+0000 7f405a7fc640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f404806b130 0x7f404806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.952+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f4054089a30 con 0x7f40641f2520 2023-10-16T14:11:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.953+0000 7f4069998640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f404806b130 0x7f404806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.954+0000 7f4069998640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f404806b130 0x7f404806d5f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f4060009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:28.957+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40540547a0 con 0x7f40641f2520 2023-10-16T14:11:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.153+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40641f34c0 con 0x7f40641f2520 2023-10-16T14:11:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.154+0000 7f405a7fc640 1 -- 172.21.15.60:0/449473725 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v243) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4054001560 con 0x7f40641f2520 2023-10-16T14:11:29.158 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:11:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f404806b130 msgr2=0x7f404806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f404806b130 0x7f404806d5f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f4060009040 comp rx=0 tx=0).stop 2023-10-16T14:11:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 msgr2=0x7f40640aed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 0x7f40640aed20 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f405400e800 tx=0x7f405400ecc0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 shutdown_connections 2023-10-16T14:11:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f404806b130 0x7f404806d5f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40640c3930 0x7f40640c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40641f2520 0x7f40640aed20 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 --2- 172.21.15.60:0/449473725 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4050004d00 0x7f40640c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 >> 172.21.15.60:0/449473725 conn(0x7f40640b0170 msgr2=0x7f4064070600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 shutdown_connections 2023-10-16T14:11:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.157+0000 7f406a99a640 1 -- 172.21.15.60:0/449473725 wait complete. 2023-10-16T14:11:29.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T14:11:29.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.393+0000 7f0383f1d640 1 Processor -- start 2023-10-16T14:11:29.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.393+0000 7f0383f1d640 1 -- start start 2023-10-16T14:11:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f0383f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f037c15ae80 0x7f037c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f0383f1d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f037c0ce390 0x7f037c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f0383f1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c15ed60 con 0x7f037c15aaa0 2023-10-16T14:11:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f0383f1d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037c06b810 con 0x7f037c0ce390 2023-10-16T14:11:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f0383f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037c0707e0 con 0x7f037c15ae80 2023-10-16T14:11:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f03824ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f037c0ce390 0x7f037c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f03824ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f037c0ce390 0x7f037c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50798/0 (socket says 172.21.15.60:50798) 2023-10-16T14:11:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.395+0000 7f03824ba640 1 -- 172.21.15.60:0/2453315401 learned_addr learned my addr 172.21.15.60:0/2453315401 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.396+0000 7f03814b8640 1 --2- 172.21.15.60:0/2453315401 >> v2:172.21.15.144:3300/0 conn(0x7f037c15ae80 0x7f037c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.396+0000 7f03824ba640 1 -- 172.21.15.60:0/2453315401 >> v2:172.21.15.144:3300/0 conn(0x7f037c15ae80 msgr2=0x7f037c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.396+0000 7f03824ba640 1 --2- 172.21.15.60:0/2453315401 >> v2:172.21.15.144:3300/0 conn(0x7f037c15ae80 0x7f037c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.396+0000 7f03824ba640 1 -- 172.21.15.60:0/2453315401 >> v1:172.21.15.144:6789/0 conn(0x7f037c15aaa0 legacy=0x7f037c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.396+0000 7f03824ba640 1 -- 172.21.15.60:0/2453315401 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c15ef90 con 0x7f037c0ce390 2023-10-16T14:11:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.396+0000 7f03814b8640 1 --2- 172.21.15.60:0/2453315401 >> v2:172.21.15.144:3300/0 conn(0x7f037c15ae80 0x7f037c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.397+0000 7f03824ba640 1 --2- 172.21.15.60:0/2453315401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c070f50 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f037c0c59c0 tx=0x7f037402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145c19bc83cb3c86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.397+0000 7f0380cb7640 1 -- 172.21.15.60:0/2453315401 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037402ca80 con 0x7f037c0ce390 2023-10-16T14:11:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.397+0000 7f0380cb7640 1 -- 172.21.15.60:0/2453315401 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037402cbe0 con 0x7f037c0ce390 2023-10-16T14:11:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.397+0000 7f0380cb7640 1 -- 172.21.15.60:0/2453315401 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03740343f0 con 0x7f037c0ce390 2023-10-16T14:11:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 -- 172.21.15.60:0/2453315401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 msgr2=0x7f037c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2453315401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c070f50 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f037c0c59c0 tx=0x7f037402bee0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 -- 172.21.15.60:0/2453315401 shutdown_connections 2023-10-16T14:11:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2453315401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c070f50 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2453315401 >> v2:172.21.15.144:3300/0 conn(0x7f037c15ae80 0x7f037c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 -- 172.21.15.60:0/2453315401 >> 172.21.15.60:0/2453315401 conn(0x7f037c0cb410 msgr2=0x7f037c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 -- 172.21.15.60:0/2453315401 shutdown_connections 2023-10-16T14:11:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.398+0000 7f0383f1d640 1 -- 172.21.15.60:0/2453315401 wait complete. 2023-10-16T14:11:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.400+0000 7f0383f1d640 1 Processor -- start 2023-10-16T14:11:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.400+0000 7f0383f1d640 1 -- start start 2023-10-16T14:11:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0383f1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0383f1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f037c15aaa0 0x7f037c0c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0383f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f037c15ae80 0x7f037c0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0383f1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c156380 con 0x7f037c0ce390 2023-10-16T14:11:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0383f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c0b26f0 con 0x7f037c15ae80 2023-10-16T14:11:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0383f1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c1568c0 con 0x7f037c15aaa0 2023-10-16T14:11:29.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0381cb9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:29.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0381cb9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c1549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50812/0 (socket says 172.21.15.60:50812) 2023-10-16T14:11:29.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.401+0000 7f0381cb9640 1 -- 172.21.15.60:0/2043069303 learned_addr learned my addr 172.21.15.60:0/2043069303 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:29.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0381cb9640 1 -- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f037c15aaa0 msgr2=0x7f037c0c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:29.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0381cb9640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f037c15aaa0 0x7f037c0c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0381cb9640 1 -- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f037c15ae80 msgr2=0x7f037c0c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0381cb9640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f037c15ae80 0x7f037c0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0381cb9640 1 -- 172.21.15.60:0/2043069303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374009b30 con 0x7f037c0ce390 2023-10-16T14:11:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0381cb9640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c1549b0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f037800c430 tx=0x7f037800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378011790 con 0x7f037c0ce390 2023-10-16T14:11:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03780118f0 con 0x7f037c0ce390 2023-10-16T14:11:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c155a50 con 0x7f037c0ce390 2023-10-16T14:11:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378011ce0 con 0x7f037c0ce390 2023-10-16T14:11:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.402+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c156010 con 0x7f037c0ce390 2023-10-16T14:11:29.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.404+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c04e160 con 0x7f037c0ce390 2023-10-16T14:11:29.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.405+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f037801c460 con 0x7f037c0ce390 2023-10-16T14:11:29.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.405+0000 7f03727fc640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f036006b0b0 0x7f036006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:29.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.406+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f03780899d0 con 0x7f037c0ce390 2023-10-16T14:11:29.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.406+0000 7f03814b8640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f036006b0b0 0x7f036006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:29.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.408+0000 7f03814b8640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f036006b0b0 0x7f036006d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f036c006fd0 tx=0x7f036c008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:29.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.410+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0378054750 con 0x7f037c0ce390 2023-10-16T14:11:29.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:29.677+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f037c0cdc60 con 0x7f037c0ce390 2023-10-16T14:11:30.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.185+0000 7f03727fc640 1 -- 172.21.15.60:0/2043069303 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3231) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0378057d70 con 0x7f037c0ce390 2023-10-16T14:11:30.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f036006b0b0 msgr2=0x7f036006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f036006b0b0 0x7f036006d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f036c006fd0 tx=0x7f036c008040 comp rx=0 tx=0).stop 2023-10-16T14:11:30.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 msgr2=0x7f037c1549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c1549b0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f037800c430 tx=0x7f037800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 shutdown_connections 2023-10-16T14:11:30.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f036006b0b0 0x7f036006d570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f037c15aaa0 0x7f037c0c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f037c15ae80 0x7f037c0c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.189+0000 7f0383f1d640 1 --2- 172.21.15.60:0/2043069303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f037c0ce390 0x7f037c1549b0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.190+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 >> 172.21.15.60:0/2043069303 conn(0x7f037c0cb410 msgr2=0x7f037c06f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:30.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.190+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 shutdown_connections 2023-10-16T14:11:30.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.190+0000 7f0383f1d640 1 -- 172.21.15.60:0/2043069303 wait complete. 2023-10-16T14:11:30.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:11:30.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.444+0000 7fae59da0640 1 Processor -- start 2023-10-16T14:11:30.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.444+0000 7fae59da0640 1 -- start start 2023-10-16T14:11:30.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.446+0000 7fae59da0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 0x7fae540d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.446+0000 7fae59da0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5415d150 con 0x7fae540b8480 2023-10-16T14:11:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.446+0000 7fae59da0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae540d1180 con 0x7fae540cded0 2023-10-16T14:11:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.446+0000 7fae59da0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae540d1460 con 0x7fae5415cd70 2023-10-16T14:11:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.446+0000 7fae4bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fae540b8480 0x7fae540cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35132/0 (socket says 172.21.15.60:35132) 2023-10-16T14:11:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.446+0000 7fae4bfff640 1 -- 172.21.15.60:0/3589137662 learned_addr learned my addr 172.21.15.60:0/3589137662 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae58d9e640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 0x7fae540d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317249637 0 0) 0x7fae5415d150 con 0x7fae540b8480 2023-10-16T14:11:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c003520 con 0x7fae540b8480 2023-10-16T14:11:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2986717387 0 0) 0x7fae3c003520 con 0x7fae540b8480 2023-10-16T14:11:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 msgr2=0x7fae540d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 0x7fae540d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 >> v1:172.21.15.144:6789/0 conn(0x7fae540cded0 legacy=0x7fae540b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae540d1740 con 0x7fae540b8480 2023-10-16T14:11:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fae44001670 con 0x7fae540b8480 2023-10-16T14:11:30.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.447+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 >> v1:172.21.15.60:6789/0 conn(0x7fae540b8480 legacy=0x7fae540cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4b7fe640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 0x7fae3c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4b7fe640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c003f70 0x7fae3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4b7fe640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae3c004d00 0x7fae3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540d1460 con 0x7fae3c004d00 2023-10-16T14:11:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c00bac0 con 0x7fae3c003f70 2023-10-16T14:11:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c00bc80 con 0x7fae5415cd70 2023-10-16T14:11:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae5959f640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae3c004d00 0x7fae3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae58d9e640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 0x7fae3c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4bfff640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c003f70 0x7fae3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae5959f640 1 -- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 msgr2=0x7fae3c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae5959f640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae5415cd70 0x7fae3c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae5959f640 1 -- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c003f70 msgr2=0x7fae3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae5959f640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c003f70 0x7fae3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae5959f640 1 -- 172.21.15.60:0/3589137662 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c003520 con 0x7fae3c004d00 2023-10-16T14:11:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.448+0000 7fae4bfff640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c003f70 0x7fae3c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae5959f640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae3c004d00 0x7fae3c005180 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fae4c001e50 tx=0x7fae3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68798b2565581cc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c003b70 con 0x7fae3c004d00 2023-10-16T14:11:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c003cd0 con 0x7fae3c004d00 2023-10-16T14:11:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae4b7fe640 1 -- 172.21.15.60:0/3589137662 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c005310 con 0x7fae3c004d00 2023-10-16T14:11:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae59da0640 1 -- 172.21.15.60:0/3589137662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae3c004d00 msgr2=0x7fae3c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae59da0640 1 --2- 172.21.15.60:0/3589137662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae3c004d00 0x7fae3c005180 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fae4c001e50 tx=0x7fae3c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.450+0000 7fae5959f640 1 -- 172.21.15.60:0/3589137662 reap_dead start 2023-10-16T14:11:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.451+0000 7fae59da0640 1 -- 172.21.15.60:0/3589137662 shutdown_connections 2023-10-16T14:11:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.451+0000 7fae59da0640 1 -- 172.21.15.60:0/3589137662 >> 172.21.15.60:0/3589137662 conn(0x7fae540cf570 msgr2=0x7fae540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.451+0000 7fae59da0640 1 -- 172.21.15.60:0/3589137662 shutdown_connections 2023-10-16T14:11:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.451+0000 7fae59da0640 1 -- 172.21.15.60:0/3589137662 wait complete. 2023-10-16T14:11:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.453+0000 7fae59da0640 1 Processor -- start 2023-10-16T14:11:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.453+0000 7fae59da0640 1 -- start start 2023-10-16T14:11:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae59da0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c004d00 0x7fae54157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae59da0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 0x7fae541520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae59da0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae540cbea0 0x7fae540cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae59da0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540c9f10 con 0x7fae541551e0 2023-10-16T14:11:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae59da0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540cc860 con 0x7fae3c004d00 2023-10-16T14:11:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae59da0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae54152610 con 0x7fae540cbea0 2023-10-16T14:11:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 0x7fae541520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 0x7fae541520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50830/0 (socket says 172.21.15.60:50830) 2023-10-16T14:11:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 -- 172.21.15.60:0/2118766688 learned_addr learned my addr 172.21.15.60:0/2118766688 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae58d9e640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c004d00 0x7fae54157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae5959f640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae540cbea0 0x7fae540cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 -- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae540cbea0 msgr2=0x7fae540cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae540cbea0 0x7fae540cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 -- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c004d00 msgr2=0x7fae54157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c004d00 0x7fae54157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.454+0000 7fae4bfff640 1 -- 172.21.15.60:0/2118766688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae54158120 con 0x7fae541551e0 2023-10-16T14:11:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.455+0000 7fae4bfff640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 0x7fae541520d0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fae440017a0 tx=0x7fae4400dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.455+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440074e0 con 0x7fae541551e0 2023-10-16T14:11:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.455+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae44007640 con 0x7fae541551e0 2023-10-16T14:11:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.455+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440049e0 con 0x7fae541551e0 2023-10-16T14:11:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.455+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae54158400 con 0x7fae541551e0 2023-10-16T14:11:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.455+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae54152cd0 con 0x7fae541551e0 2023-10-16T14:11:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.456+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae54158640 con 0x7fae541551e0 2023-10-16T14:11:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.458+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fae44004b40 con 0x7fae541551e0 2023-10-16T14:11:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.458+0000 7fae497fa640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae4006b0b0 0x7fae4006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.459+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fae440896a0 con 0x7fae541551e0 2023-10-16T14:11:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.459+0000 7fae58d9e640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae4006b0b0 0x7fae4006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.460+0000 7fae58d9e640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae4006b0b0 0x7fae4006d570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fae50009a10 tx=0x7fae50008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.463+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae54158640 con 0x7fae541551e0 2023-10-16T14:11:30.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.658+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fae54152850 con 0x7fae541551e0 2023-10-16T14:11:30.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.664+0000 7fae497fa640 1 -- 172.21.15.60:0/2118766688 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7fae54152850 con 0x7fae541551e0 2023-10-16T14:11:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae4006b0b0 msgr2=0x7fae4006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae4006b0b0 0x7fae4006d570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fae50009a10 tx=0x7fae50008040 comp rx=0 tx=0).stop 2023-10-16T14:11:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 msgr2=0x7fae541520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 0x7fae541520d0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fae440017a0 tx=0x7fae4400dd20 comp rx=0 tx=0).stop 2023-10-16T14:11:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 shutdown_connections 2023-10-16T14:11:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae4006b0b0 0x7fae4006d570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae540cbea0 0x7fae540cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae541551e0 0x7fae541520d0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 --2- 172.21.15.60:0/2118766688 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae3c004d00 0x7fae54157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:30.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 >> 172.21.15.60:0/2118766688 conn(0x7fae540cf570 msgr2=0x7fae540b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:30.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 shutdown_connections 2023-10-16T14:11:30.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:30.667+0000 7fae59da0640 1 -- 172.21.15.60:0/2118766688 wait complete. 2023-10-16T14:11:30.691 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:11:30.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:11:30.828 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:11:30.947 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:11:30.947 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:11:30.948 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:31.011 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:11:31.011 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:11:31.012 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:31.072 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:11:31.073 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:11:31.073 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:31.135 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:11:31.136 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:11:31.136 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:31.199 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:11:31.199 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:11:31.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:31.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.431+0000 7fd1613e4640 1 Processor -- start 2023-10-16T14:11:31.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.432+0000 7fd1613e4640 1 -- start start 2023-10-16T14:11:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.433+0000 7fd1613e4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd15c06c040 0x7fd15c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.433+0000 7fd1613e4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c15f4a0 con 0x7fd15c0ad970 2023-10-16T14:11:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.433+0000 7fd1613e4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c06c960 con 0x7fd15c0cda80 2023-10-16T14:11:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.433+0000 7fd1613e4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c15b9d0 con 0x7fd15c06c040 2023-10-16T14:11:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.433+0000 7fd15b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd15c0ad970 0x7fd15c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35138/0 (socket says 172.21.15.60:35138) 2023-10-16T14:11:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.433+0000 7fd15b7fe640 1 -- 172.21.15.60:0/1424327788 learned_addr learned my addr 172.21.15.60:0/1424327788 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1424327788 >> v2:172.21.15.144:3300/0 conn(0x7fd15c06c040 0x7fd15c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794404636 0 0) 0x7fd15c15f4a0 con 0x7fd15c0ad970 2023-10-16T14:11:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd144003520 con 0x7fd15c0ad970 2023-10-16T14:11:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 658490501 0 0) 0x7fd144003520 con 0x7fd15c0ad970 2023-10-16T14:11:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 >> v2:172.21.15.144:3300/0 conn(0x7fd15c06c040 msgr2=0x7fd15c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 --2- 172.21.15.60:0/1424327788 >> v2:172.21.15.144:3300/0 conn(0x7fd15c06c040 0x7fd15c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 >> v1:172.21.15.144:6789/0 conn(0x7fd15c0cda80 legacy=0x7fd15c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c15bcb0 con 0x7fd15c0ad970 2023-10-16T14:11:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd150001670 con 0x7fd15c0ad970 2023-10-16T14:11:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 >> v1:172.21.15.60:6789/0 conn(0x7fd15c0ad970 legacy=0x7fd15c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd144003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1440047c0 0x7fd144004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.434+0000 7fd15affd640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd144005180 0x7fd14400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c15b9d0 con 0x7fd144003970 2023-10-16T14:11:31.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14400c020 con 0x7fd144005180 2023-10-16T14:11:31.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14400c1e0 con 0x7fd1440047c0 2023-10-16T14:11:31.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd144003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15b7fe640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1440047c0 0x7fd144004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd160be3640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd144005180 0x7fd14400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15bfff640 1 -- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1440047c0 msgr2=0x7fd144004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd1440047c0 0x7fd144004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15bfff640 1 -- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd144005180 msgr2=0x7fd14400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd144005180 0x7fd14400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd15bfff640 1 -- 172.21.15.60:0/1424327788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144003520 con 0x7fd144003970 2023-10-16T14:11:31.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.435+0000 7fd160be3640 1 -- 172.21.15.60:0/1424327788 reap_dead start 2023-10-16T14:11:31.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.436+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd144003d90 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ac0 tx=0x7fd1440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa015724d3fd999f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:31.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.437+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c02c6b0 con 0x7fd144003970 2023-10-16T14:11:31.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.437+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c02c810 con 0x7fd144003970 2023-10-16T14:11:31.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.437+0000 7fd15affd640 1 -- 172.21.15.60:0/1424327788 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c02cba0 con 0x7fd144003970 2023-10-16T14:11:31.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.437+0000 7fd1613e4640 1 -- 172.21.15.60:0/1424327788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 msgr2=0x7fd144003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.437+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd144003d90 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ac0 tx=0x7fd1440057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.438+0000 7fd1613e4640 1 -- 172.21.15.60:0/1424327788 shutdown_connections 2023-10-16T14:11:31.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.438+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1424327788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd144003d90 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.438+0000 7fd1613e4640 1 -- 172.21.15.60:0/1424327788 >> 172.21.15.60:0/1424327788 conn(0x7fd15c0ce6a0 msgr2=0x7fd15c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:31.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.438+0000 7fd1613e4640 1 -- 172.21.15.60:0/1424327788 shutdown_connections 2023-10-16T14:11:31.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.438+0000 7fd1613e4640 1 -- 172.21.15.60:0/1424327788 wait complete. 2023-10-16T14:11:31.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.439+0000 7fd1613e4640 1 Processor -- start 2023-10-16T14:11:31.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.440+0000 7fd1613e4640 1 -- start start 2023-10-16T14:11:31.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd1613e4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd15c0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd1613e4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd15c0c8060 0x7fd15c0c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd1613e4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15c14f7b0 0x7fd15c14fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd1613e4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c15c650 con 0x7fd144003970 2023-10-16T14:11:31.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd1613e4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c150170 con 0x7fd15c14f7b0 2023-10-16T14:11:31.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd1613e4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0b36f0 con 0x7fd15c0c8060 2023-10-16T14:11:31.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd15c0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd15c0b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50862/0 (socket says 172.21.15.60:50862) 2023-10-16T14:11:31.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 -- 172.21.15.60:0/1924710174 learned_addr learned my addr 172.21.15.60:0/1924710174 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:31.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15b7fe640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd15c0c8060 0x7fd15c0c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd160be3640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15c14f7b0 0x7fd15c14fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 -- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd15c0c8060 msgr2=0x7fd15c0c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd15c0c8060 0x7fd15c0c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 -- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15c14f7b0 msgr2=0x7fd15c14fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15c14f7b0 0x7fd15c14fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.441+0000 7fd15bfff640 1 -- 172.21.15.60:0/1924710174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0b39d0 con 0x7fd144003970 2023-10-16T14:11:31.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd15b7fe640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd15c0c8060 0x7fd15c0c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:31.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd15bfff640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd15c0b30d0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fd14c02c250 tx=0x7fd14c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c003810 con 0x7fd144003970 2023-10-16T14:11:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c003970 con 0x7fd144003970 2023-10-16T14:11:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c003d00 con 0x7fd144003970 2023-10-16T14:11:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c0b3cb0 con 0x7fd144003970 2023-10-16T14:11:31.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.442+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c0ade50 con 0x7fd144003970 2023-10-16T14:11:31.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.443+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c06b330 con 0x7fd144003970 2023-10-16T14:11:31.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.444+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd14c034440 con 0x7fd144003970 2023-10-16T14:11:31.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.445+0000 7fd158ff9640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd14806b0e0 0x7fd14806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.445+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7fd14c0abd00 con 0x7fd144003970 2023-10-16T14:11:31.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.446+0000 7fd15b7fe640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd14806b0e0 0x7fd14806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.447+0000 7fd15b7fe640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd14806b0e0 0x7fd14806d5a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd150001200 tx=0x7fd150001150 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:31.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.450+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd14c076a60 con 0x7fd144003970 2023-10-16T14:11:31.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.685+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd15c04e9b0 con 0x7fd144003970 2023-10-16T14:11:31.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.686+0000 7fd158ff9640 1 -- 172.21.15.60:0/1924710174 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+3706 (secure 0 0 0) 0x7fd14c039e90 con 0x7fd144003970 2023-10-16T14:11:31.690 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:31.690 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":122,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/191863928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":191863928},{"type":"v1","addr":"172.21.15.144:6836","nonce":191863928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":8123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/499924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":499924806},{"type":"v1","addr":"172.21.15.144:6837","nonce":499924806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":10995,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2963056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2963056009},{"type":"v1","addr":"172.21.15.60:6833","nonce":2963056009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":11709,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2779100925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2779100925},{"type":"v1","addr":"172.21.15.60:6835","nonce":2779100925}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[]} 2023-10-16T14:11:31.690 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 122 2023-10-16T14:11:31.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.688+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd14806b0e0 msgr2=0x7fd14806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.688+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd14806b0e0 0x7fd14806d5a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd150001200 tx=0x7fd150001150 comp rx=0 tx=0).stop 2023-10-16T14:11:31.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.688+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 msgr2=0x7fd15c0b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.688+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd15c0b30d0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fd14c02c250 tx=0x7fd14c004730 comp rx=0 tx=0).stop 2023-10-16T14:11:31.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 shutdown_connections 2023-10-16T14:11:31.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd14806b0e0 0x7fd14806d5a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd15c0c8060 0x7fd15c0c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15c14f7b0 0x7fd15c14fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 --2- 172.21.15.60:0/1924710174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd144003970 0x7fd15c0b30d0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:31.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 >> 172.21.15.60:0/1924710174 conn(0x7fd15c0ce6a0 msgr2=0x7fd15c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:31.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 shutdown_connections 2023-10-16T14:11:31.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.689+0000 7fd1613e4640 1 -- 172.21.15.60:0/1924710174 wait complete. 2023-10-16T14:11:31.712 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:11:31.712 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:11:31.713 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:11:31.713 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:11:31.713 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:11:31.713 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:11:31.714 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:11:31.714 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:11:31.714 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:11:31.715 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:11:31.715 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:11:31.715 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:11:31.716 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:11:31.716 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:11:31.717 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:11:31.717 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:11:31.729 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:11:31.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:11:31.731 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:11:31.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:11:31.737 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:11:31.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:11:31.743 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:11:31.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:11:31.779 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:11:31.780 DEBUG:teuthology.parallel:result is None 2023-10-16T14:11:31.787 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:11:31.788 DEBUG:teuthology.parallel:result is None 2023-10-16T14:11:31.792 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:11:31.792 DEBUG:teuthology.parallel:result is None 2023-10-16T14:11:31.799 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:11:31.800 DEBUG:teuthology.parallel:result is None 2023-10-16T14:11:31.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:11:31.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-10-16T14:11:31.945 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:11:31.946 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:11:31.954 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T14:11:31.955 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:11:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.990+0000 7f34d2cd7640 1 Processor -- start 2023-10-16T14:11:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.990+0000 7f34d2cd7640 1 -- start start 2023-10-16T14:11:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.991+0000 7f34d2cd7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f34cc06a4b0 0x7f34cc0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.991+0000 7f34d2cd7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0cb030 con 0x7f34cc0c9ca0 2023-10-16T14:11:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.991+0000 7f34d2cd7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0c91c0 con 0x7f34cc15ad70 2023-10-16T14:11:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.991+0000 7f34d2cd7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc0c94a0 con 0x7f34cc06a4b0 2023-10-16T14:11:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34d1274640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f34cc0c9ca0 0x7f34cc0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35168/0 (socket says 172.21.15.60:35168) 2023-10-16T14:11:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34d1274640 1 -- 172.21.15.60:0/2648577372 learned_addr learned my addr 172.21.15.60:0/2648577372 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cbfff640 1 --2- 172.21.15.60:0/2648577372 >> v2:172.21.15.144:3300/0 conn(0x7f34cc06a4b0 0x7f34cc0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cb7fe640 1 -- 172.21.15.60:0/2648577372 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115158580 0 0) 0x7f34cc0cb030 con 0x7f34cc0c9ca0 2023-10-16T14:11:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cb7fe640 1 -- 172.21.15.60:0/2648577372 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc003520 con 0x7f34cc0c9ca0 2023-10-16T14:11:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cbfff640 1 -- 172.21.15.60:0/2648577372 >> v1:172.21.15.144:6789/0 conn(0x7f34cc15ad70 legacy=0x7f34cc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cb7fe640 1 -- 172.21.15.60:0/2648577372 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928370474 0 0) 0x7f34cc0c91c0 con 0x7f34cc15ad70 2023-10-16T14:11:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cbfff640 1 -- 172.21.15.60:0/2648577372 >> v1:172.21.15.60:6789/0 conn(0x7f34cc0c9ca0 legacy=0x7f34cc0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.992+0000 7f34cbfff640 1 -- 172.21.15.60:0/2648577372 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0c9780 con 0x7f34cc06a4b0 2023-10-16T14:11:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.993+0000 7f34cbfff640 1 --2- 172.21.15.60:0/2648577372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc06a4b0 0x7f34cc0ca3b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f34b80097a0 tx=0x7f34b802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54486868ea5d4667 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.994+0000 7f34cb7fe640 1 -- 172.21.15.60:0/2648577372 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8030610 con 0x7f34cc06a4b0 2023-10-16T14:11:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.994+0000 7f34cb7fe640 1 -- 172.21.15.60:0/2648577372 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b8030770 con 0x7f34cc06a4b0 2023-10-16T14:11:32.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.994+0000 7f34cb7fe640 1 -- 172.21.15.60:0/2648577372 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8030b00 con 0x7f34cc06a4b0 2023-10-16T14:11:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.994+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2648577372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc06a4b0 msgr2=0x7f34cc0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.994+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2648577372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc06a4b0 0x7f34cc0ca3b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f34b80097a0 tx=0x7f34b802b960 comp rx=0 tx=0).stop 2023-10-16T14:11:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.995+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2648577372 shutdown_connections 2023-10-16T14:11:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.995+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2648577372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc06a4b0 0x7f34cc0ca3b0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.995+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2648577372 >> 172.21.15.60:0/2648577372 conn(0x7f34cc0ce9b0 msgr2=0x7f34cc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.995+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2648577372 shutdown_connections 2023-10-16T14:11:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.995+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2648577372 wait complete. 2023-10-16T14:11:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.996+0000 7f34d2cd7640 1 Processor -- start 2023-10-16T14:11:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.996+0000 7f34d2cd7640 1 -- start start 2023-10-16T14:11:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.997+0000 7f34d2cd7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 0x7f34cc15c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.997+0000 7f34d2cd7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0c9ca0 0x7f34cc159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.997+0000 7f34d2cd7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc15ad70 0x7f34cc0b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.997+0000 7f34d2cd7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc15c860 con 0x7f34cc06a4b0 2023-10-16T14:11:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.997+0000 7f34d2cd7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc15d000 con 0x7f34cc0c9ca0 2023-10-16T14:11:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.997+0000 7f34d2cd7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc15d2e0 con 0x7f34cc15ad70 2023-10-16T14:11:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 0x7f34cc15c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 0x7f34cc15c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50902/0 (socket says 172.21.15.60:50902) 2023-10-16T14:11:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 -- 172.21.15.60:0/2613220961 learned_addr learned my addr 172.21.15.60:0/2613220961 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34cbfff640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0c9ca0 0x7f34cc159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d1274640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc15ad70 0x7f34cc0b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:32.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 -- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc15ad70 msgr2=0x7f34cc0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:32.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc15ad70 0x7f34cc0b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:32.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 -- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0c9ca0 msgr2=0x7f34cc159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:32.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0c9ca0 0x7f34cc159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:32.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d0a73640 1 -- 172.21.15.60:0/2613220961 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b8009400 con 0x7f34cc06a4b0 2023-10-16T14:11:32.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34cbfff640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0c9ca0 0x7f34cc159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:32.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.998+0000 7f34d1274640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc15ad70 0x7f34cc0b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:32.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.999+0000 7f34d0a73640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 0x7f34cc15c320 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f34c000c410 tx=0x7f34c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:32.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.999+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c00035f0 con 0x7f34cc06a4b0 2023-10-16T14:11:32.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.999+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c0003770 con 0x7f34cc06a4b0 2023-10-16T14:11:32.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.999+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0003b20 con 0x7f34cc06a4b0 2023-10-16T14:11:32.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.999+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc15d5c0 con 0x7f34cc06a4b0 2023-10-16T14:11:32.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:31.999+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc15db80 con 0x7f34cc06a4b0 2023-10-16T14:11:32.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.000+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc15dd70 con 0x7f34cc06a4b0 2023-10-16T14:11:32.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.001+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f34c0003c80 con 0x7f34cc06a4b0 2023-10-16T14:11:32.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.002+0000 7f34c97fa640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34b006b0b0 0x7f34b006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.002+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 5846+0+0 (secure 0 0 0) 0x7f34c00891a0 con 0x7f34cc06a4b0 2023-10-16T14:11:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.003+0000 7f34cbfff640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34b006b0b0 0x7f34b006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.003+0000 7f34cbfff640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34b006b0b0 0x7f34b006d570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34b8036000 tx=0x7f34b8008030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.007+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34c0053dc0 con 0x7f34cc06a4b0 2023-10-16T14:11:32.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:32.215+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f34cc04e160 con 0x7f34cc06a4b0 2023-10-16T14:11:33.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.705+0000 7f34c97fa640 1 -- 172.21.15.60:0/2613220961 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v248) v1 ==== 97+0+0 (secure 0 0 0) 0x7f34c00573e0 con 0x7f34cc06a4b0 2023-10-16T14:11:33.710 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T14:11:33.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34b006b0b0 msgr2=0x7f34b006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:33.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34b006b0b0 0x7f34b006d570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34b8036000 tx=0x7f34b8008030 comp rx=0 tx=0).stop 2023-10-16T14:11:33.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 msgr2=0x7f34cc15c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:33.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 0x7f34cc15c320 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f34c000c410 tx=0x7f34c000c8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 shutdown_connections 2023-10-16T14:11:33.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f34b006b0b0 0x7f34b006d570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f34cc15ad70 0x7f34cc0b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f34cc0c9ca0 0x7f34cc159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 --2- 172.21.15.60:0/2613220961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34cc06a4b0 0x7f34cc15c320 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 >> 172.21.15.60:0/2613220961 conn(0x7f34cc0ce9b0 msgr2=0x7f34cc0c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:33.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 shutdown_connections 2023-10-16T14:11:33.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.711+0000 7f34d2cd7640 1 -- 172.21.15.60:0/2613220961 wait complete. 2023-10-16T14:11:33.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:11:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.951+0000 7f7bc7844640 1 Processor -- start 2023-10-16T14:11:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.952+0000 7f7bc7844640 1 -- start start 2023-10-16T14:11:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc7844640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bc0160420 0x7f7bc00ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc7844640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc7844640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7bc00adc40 0x7f7bc006adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc7844640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc00cde30 con 0x7f7bc00adc40 2023-10-16T14:11:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc7844640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc006e170 con 0x7f7bc0160420 2023-10-16T14:11:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc7844640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00cd460 con 0x7f7bc0160800 2023-10-16T14:11:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc5de1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7bc00adc40 0x7f7bc006adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc5de1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7bc00adc40 0x7f7bc006adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50908/0 (socket says 172.21.15.60:50908) 2023-10-16T14:11:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc5de1640 1 -- 172.21.15.60:0/2735122103 learned_addr learned my addr 172.21.15.60:0/2735122103 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.953+0000 7f7bc55e0640 1 --2- 172.21.15.60:0/2735122103 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0160420 0x7f7bc00ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.954+0000 7f7bc4ddf640 1 --2- 172.21.15.60:0/2735122103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.954+0000 7f7bc5de1640 1 -- 172.21.15.60:0/2735122103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 msgr2=0x7f7bc00cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.954+0000 7f7bc5de1640 1 --2- 172.21.15.60:0/2735122103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.954+0000 7f7bc5de1640 1 -- 172.21.15.60:0/2735122103 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0160420 msgr2=0x7f7bc00ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.954+0000 7f7bc5de1640 1 --2- 172.21.15.60:0/2735122103 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0160420 0x7f7bc00ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.954+0000 7f7bc5de1640 1 -- 172.21.15.60:0/2735122103 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc006e3c0 con 0x7f7bc00adc40 2023-10-16T14:11:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.955+0000 7f7bc5de1640 1 --2- 172.21.15.60:0/2735122103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc00adc40 0x7f7bc006adb0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f7bc006f470 tx=0x7f7bb802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41984b2e647f8f2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.955+0000 7f7bb7fff640 1 -- 172.21.15.60:0/2735122103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb802ca70 con 0x7f7bc00adc40 2023-10-16T14:11:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.955+0000 7f7bb7fff640 1 -- 172.21.15.60:0/2735122103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb802cbd0 con 0x7f7bc00adc40 2023-10-16T14:11:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.955+0000 7f7bb7fff640 1 -- 172.21.15.60:0/2735122103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80343f0 con 0x7f7bc00adc40 2023-10-16T14:11:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 -- 172.21.15.60:0/2735122103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc00adc40 msgr2=0x7f7bc006adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 --2- 172.21.15.60:0/2735122103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc00adc40 0x7f7bc006adb0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f7bc006f470 tx=0x7f7bb802bed0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 -- 172.21.15.60:0/2735122103 shutdown_connections 2023-10-16T14:11:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 --2- 172.21.15.60:0/2735122103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc00adc40 0x7f7bc006adb0 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 --2- 172.21.15.60:0/2735122103 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 --2- 172.21.15.60:0/2735122103 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0160420 0x7f7bc00ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 -- 172.21.15.60:0/2735122103 >> 172.21.15.60:0/2735122103 conn(0x7f7bc00b0d30 msgr2=0x7f7bc00b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 -- 172.21.15.60:0/2735122103 shutdown_connections 2023-10-16T14:11:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.956+0000 7f7bc7844640 1 -- 172.21.15.60:0/2735122103 wait complete. 2023-10-16T14:11:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.958+0000 7f7bc7844640 1 Processor -- start 2023-10-16T14:11:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.958+0000 7f7bc7844640 1 -- start start 2023-10-16T14:11:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc7844640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc00adc40 0x7f7bc014ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc7844640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 0x7f7bc015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc7844640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc7844640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc009de70 con 0x7f7bc0160420 2023-10-16T14:11:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc7844640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc014ac30 con 0x7f7bc00adc40 2023-10-16T14:11:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc7844640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc014af10 con 0x7f7bc0160800 2023-10-16T14:11:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc4ddf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 0x7f7bc015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc4ddf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 0x7f7bc015aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50910/0 (socket says 172.21.15.60:50910) 2023-10-16T14:11:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc4ddf640 1 -- 172.21.15.60:0/3319992601 learned_addr learned my addr 172.21.15.60:0/3319992601 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc55e0640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc00adc40 0x7f7bc014ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc4ddf640 1 -- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 msgr2=0x7f7bc00c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc5de1640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.959+0000 7f7bc4ddf640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00c3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bc4ddf640 1 -- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc00adc40 msgr2=0x7f7bc014ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bc4ddf640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc00adc40 0x7f7bc014ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bc4ddf640 1 -- 172.21.15.60:0/3319992601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb8009b30 con 0x7f7bc0160420 2023-10-16T14:11:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bc4ddf640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 0x7f7bc015aa20 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d650 tx=0x7f7bb000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb000b6c0 con 0x7f7bc0160420 2023-10-16T14:11:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb000b820 con 0x7f7bc0160420 2023-10-16T14:11:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb000bbd0 con 0x7f7bc0160420 2023-10-16T14:11:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc014d200 con 0x7f7bc0160420 2023-10-16T14:11:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.960+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc014d7b0 con 0x7f7bc0160420 2023-10-16T14:11:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.961+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc004e160 con 0x7f7bc0160420 2023-10-16T14:11:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.962+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7bb000bd30 con 0x7f7bc0160420 2023-10-16T14:11:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.963+0000 7f7bb5ffb640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7ba406b0b0 0x7f7ba406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.963+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7bb0089b60 con 0x7f7bc0160420 2023-10-16T14:11:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.964+0000 7f7bc55e0640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7ba406b0b0 0x7f7ba406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.965+0000 7f7bc55e0640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7ba406b0b0 0x7f7ba406d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc006fd0 tx=0x7f7bbc008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:33.968+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7bb0054bf0 con 0x7f7bc0160420 2023-10-16T14:11:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.235+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7bc014b050 con 0x7f7bc0160420 2023-10-16T14:11:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.237+0000 7f7bb5ffb640 1 -- 172.21.15.60:0/3319992601 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v56) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7bb0058210 con 0x7f7bc0160420 2023-10-16T14:11:34.240 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:34.241 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:11:34.241 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T14:11:34.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.241 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T14:11:34.242 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T14:11:34.242 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.242 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.242 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:11:34.242 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:11:34.243 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.243 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T14:11:34.244 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T14:11:34.244 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.244 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.244 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:11:34.244 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:11:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.245 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T14:11:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T14:11:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:11:34.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:11:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.247 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T14:11:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T14:11:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:11:34.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:11:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.249 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T14:11:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T14:11:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T14:11:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T14:11:34.251 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.253 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T14:11:34.253 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T14:11:34.253 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.253 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.254 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.257 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T14:11:34.257 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T14:11:34.257 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.258 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.259 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T14:11:34.259 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T14:11:34.259 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.260 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.261 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T14:11:34.261 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T14:11:34.261 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.261 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.262 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.262 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.262 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.263 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T14:11:34.263 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T14:11:34.263 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.263 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.263 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.263 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T14:11:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T14:11:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.265 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:11:34.265 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:11:34.265 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.266 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T14:11:34.266 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T14:11:34.266 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.266 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.267 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T14:11:34.267 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:11:34.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.267 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T14:11:34.268 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T14:11:34.268 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.268 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.268 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:11:34.268 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:11:34.269 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:11:34.269 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.269 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T14:11:34.270 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T14:11:34.270 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.270 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.270 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:11:34.270 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:11:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:11:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T14:11:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T14:11:34.272 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.272 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:11:34.272 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:11:34.272 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:11:34.272 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:11:34.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.273 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T14:11:34.273 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T14:11:34.273 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.274 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:11:34.274 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:11:34.274 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:11:34.274 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.274 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.275 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T14:11:34.275 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T14:11:34.275 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.276 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:11:34.276 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:11:34.276 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.276 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:34.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:34.277 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T14:11:34.277 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T14:11:34.277 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:11:34.278 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:11:34.278 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:11:34.278 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:11:34.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:34.279 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T14:11:34.279 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:11:34.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7ba406b0b0 msgr2=0x7f7ba406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7ba406b0b0 0x7f7ba406d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc006fd0 tx=0x7f7bbc008040 comp rx=0 tx=0).stop 2023-10-16T14:11:34.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 msgr2=0x7f7bc015aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 0x7f7bc015aa20 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000d650 tx=0x7f7bb000db10 comp rx=0 tx=0).stop 2023-10-16T14:11:34.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 shutdown_connections 2023-10-16T14:11:34.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7ba406b0b0 0x7f7ba406d570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7bc0160800 0x7f7bc00c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7bc0160420 0x7f7bc015aa20 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 --2- 172.21.15.60:0/3319992601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc00adc40 0x7f7bc014ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.239+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 >> 172.21.15.60:0/3319992601 conn(0x7f7bc00b0d30 msgr2=0x7f7bc00cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:34.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.240+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 shutdown_connections 2023-10-16T14:11:34.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.240+0000 7f7bc7844640 1 -- 172.21.15.60:0/3319992601 wait complete. 2023-10-16T14:11:34.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:34.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.503+0000 7fe0548ba640 1 Processor -- start 2023-10-16T14:11:34.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.504+0000 7fe0548ba640 1 -- start start 2023-10-16T14:11:34.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.505+0000 7fe0548ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe05006d220 0x7fe0500c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.505+0000 7fe0548ba640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe050065ab0 0x7fe0500cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.505+0000 7fe0548ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0500acd70 con 0x7fe05006d600 2023-10-16T14:11:34.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.505+0000 7fe0548ba640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0500ac4a0 con 0x7fe05006d220 2023-10-16T14:11:34.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.505+0000 7fe0548ba640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe050065f40 con 0x7fe050065ab0 2023-10-16T14:11:34.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe05006d220 0x7fe0500c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe05006d220 0x7fe0500c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50918/0 (socket says 172.21.15.60:50918) 2023-10-16T14:11:34.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 -- 172.21.15.60:0/1039765096 learned_addr learned my addr 172.21.15.60:0/1039765096 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:34.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04659c640 1 --2- 172.21.15.60:0/1039765096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe050065ab0 0x7fe0500cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 -- 172.21.15.60:0/1039765096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe050065ab0 msgr2=0x7fe0500cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:34.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 --2- 172.21.15.60:0/1039765096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe050065ab0 0x7fe0500cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 -- 172.21.15.60:0/1039765096 >> v1:172.21.15.60:6789/0 conn(0x7fe05006d600 legacy=0x7fe0500ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04ed9d640 1 -- 172.21.15.60:0/1039765096 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0500ac720 con 0x7fe05006d220 2023-10-16T14:11:34.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.506+0000 7fe04e59c640 1 -- 172.21.15.60:0/1039765096 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918021968 0 0) 0x7fe0500acd70 con 0x7fe05006d600 2023-10-16T14:11:34.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.507+0000 7fe04ed9d640 1 --2- 172.21.15.60:0/1039765096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d220 0x7fe0500c95e0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fe040009e50 tx=0x7fe04002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a65dc7c61462047 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:34.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe04e59c640 1 -- 172.21.15.60:0/1039765096 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04002cb00 con 0x7fe05006d220 2023-10-16T14:11:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe04e59c640 1 -- 172.21.15.60:0/1039765096 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04002cc60 con 0x7fe05006d220 2023-10-16T14:11:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe04e59c640 1 -- 172.21.15.60:0/1039765096 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400343f0 con 0x7fe05006d220 2023-10-16T14:11:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 -- 172.21.15.60:0/1039765096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d220 msgr2=0x7fe0500c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 --2- 172.21.15.60:0/1039765096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d220 0x7fe0500c95e0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fe040009e50 tx=0x7fe04002bf60 comp rx=0 tx=0).stop 2023-10-16T14:11:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 -- 172.21.15.60:0/1039765096 shutdown_connections 2023-10-16T14:11:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 --2- 172.21.15.60:0/1039765096 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe050065ab0 0x7fe0500cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 --2- 172.21.15.60:0/1039765096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d220 0x7fe0500c95e0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 -- 172.21.15.60:0/1039765096 >> 172.21.15.60:0/1039765096 conn(0x7fe0500b0a80 msgr2=0x7fe0500b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 -- 172.21.15.60:0/1039765096 shutdown_connections 2023-10-16T14:11:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.508+0000 7fe0548ba640 1 -- 172.21.15.60:0/1039765096 wait complete. 2023-10-16T14:11:34.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.510+0000 7fe0548ba640 1 Processor -- start 2023-10-16T14:11:34.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.510+0000 7fe0548ba640 1 -- start start 2023-10-16T14:11:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe0548ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05006d220 0x7fe050147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe0548ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 0x7fe0500afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe0548ba640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0501f5c40 0x7fe0501f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe0548ba640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0500c7ba0 con 0x7fe05006d600 2023-10-16T14:11:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe0548ba640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0501f6700 con 0x7fe05006d220 2023-10-16T14:11:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe0548ba640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0501f69e0 con 0x7fe0501f5c40 2023-10-16T14:11:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe04659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 0x7fe0500afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe04659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 0x7fe0500afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50930/0 (socket says 172.21.15.60:50930) 2023-10-16T14:11:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.511+0000 7fe04659c640 1 -- 172.21.15.60:0/3371080661 learned_addr learned my addr 172.21.15.60:0/3371080661 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04ed9d640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05006d220 0x7fe050147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04659c640 1 -- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0501f5c40 msgr2=0x7fe0501f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04f59e640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0501f5c40 0x7fe0501f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04659c640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0501f5c40 0x7fe0501f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04659c640 1 -- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05006d220 msgr2=0x7fe050147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04659c640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05006d220 0x7fe050147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04659c640 1 -- 172.21.15.60:0/3371080661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040009b30 con 0x7fe05006d600 2023-10-16T14:11:34.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04ed9d640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05006d220 0x7fe050147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:34.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.512+0000 7fe04659c640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 0x7fe0500afc50 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fe03800d5f0 tx=0x7fe03800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.513+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0380125f0 con 0x7fe05006d600 2023-10-16T14:11:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.513+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe038012750 con 0x7fe05006d600 2023-10-16T14:11:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.513+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038012b00 con 0x7fe05006d600 2023-10-16T14:11:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.513+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0500c7dd0 con 0x7fe05006d600 2023-10-16T14:11:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.513+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0500c6260 con 0x7fe05006d600 2023-10-16T14:11:34.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.514+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0501f6ba0 con 0x7fe05006d600 2023-10-16T14:11:34.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.515+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe038012c60 con 0x7fe05006d600 2023-10-16T14:11:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.516+0000 7fe047fff640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe03006b0b0 0x7fe03006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.516+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe038089b30 con 0x7fe05006d600 2023-10-16T14:11:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.517+0000 7fe04ed9d640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe03006b0b0 0x7fe03006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.518+0000 7fe04ed9d640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe03006b0b0 0x7fe03006d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe040035000 tx=0x7fe040030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.521+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe038054c60 con 0x7fe05006d600 2023-10-16T14:11:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.719+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe05004e9b0 con 0x7fe05006d600 2023-10-16T14:11:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.720+0000 7fe047fff640 1 -- 172.21.15.60:0/3371080661 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+12855 (secure 0 0 0) 0x7fe038058280 con 0x7fe05006d600 2023-10-16T14:11:34.724 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:34.724 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":248,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:33.703388+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe03006b0b0 msgr2=0x7fe03006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe03006b0b0 0x7fe03006d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe040035000 tx=0x7fe040030000 comp rx=0 tx=0).stop 2023-10-16T14:11:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 msgr2=0x7fe0500afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 0x7fe0500afc50 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fe03800d5f0 tx=0x7fe03800dab0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 shutdown_connections 2023-10-16T14:11:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe03006b0b0 0x7fe03006d570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe0501f5c40 0x7fe0501f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe05006d600 0x7fe0500afc50 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 --2- 172.21.15.60:0/3371080661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05006d220 0x7fe050147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 >> 172.21.15.60:0/3371080661 conn(0x7fe0500b0a80 msgr2=0x7fe0500cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 shutdown_connections 2023-10-16T14:11:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.723+0000 7fe0548ba640 1 -- 172.21.15.60:0/3371080661 wait complete. 2023-10-16T14:11:34.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:11:34.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:11:34.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.965+0000 7f7900cdf640 1 Processor -- start 2023-10-16T14:11:34.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.965+0000 7f7900cdf640 1 -- start start 2023-10-16T14:11:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f7900cdf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc0adb30 0x7f78fc0cd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f7900cdf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78fc0b0c90 0x7f78fc0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f7900cdf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc06f3a0 con 0x7f78fc06c630 2023-10-16T14:11:34.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f7900cdf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc0c8e70 con 0x7f78fc0b0c90 2023-10-16T14:11:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f7900cdf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0c9150 con 0x7f78fc0adb30 2023-10-16T14:11:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78fbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78fc0b0c90 0x7f78fc0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78fbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78fc0b0c90 0x7f78fc0c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50944/0 (socket says 172.21.15.60:50944) 2023-10-16T14:11:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78fbfff640 1 -- 172.21.15.60:0/2891440810 learned_addr learned my addr 172.21.15.60:0/2891440810 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:34.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78faffd640 1 --2- 172.21.15.60:0/2891440810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc0adb30 0x7f78fc0cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78fbfff640 1 -- 172.21.15.60:0/2891440810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc0adb30 msgr2=0x7f78fc0cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78fbfff640 1 --2- 172.21.15.60:0/2891440810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc0adb30 0x7f78fc0cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.967+0000 7f78fbfff640 1 -- 172.21.15.60:0/2891440810 >> v1:172.21.15.144:6789/0 conn(0x7f78fc06c630 legacy=0x7f78fc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:34.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.968+0000 7f78fbfff640 1 -- 172.21.15.60:0/2891440810 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0c9430 con 0x7f78fc0b0c90 2023-10-16T14:11:34.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.969+0000 7f78fbfff640 1 --2- 172.21.15.60:0/2891440810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc0c9950 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f78fc06f200 tx=0x7f78f402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435439820fc7713d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:34.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.969+0000 7f78fa7fc640 1 -- 172.21.15.60:0/2891440810 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f402c9a0 con 0x7f78fc0b0c90 2023-10-16T14:11:34.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.969+0000 7f78fa7fc640 1 -- 172.21.15.60:0/2891440810 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f402cb00 con 0x7f78fc0b0c90 2023-10-16T14:11:34.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.969+0000 7f78fa7fc640 1 -- 172.21.15.60:0/2891440810 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f402ce90 con 0x7f78fc0b0c90 2023-10-16T14:11:34.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 -- 172.21.15.60:0/2891440810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 msgr2=0x7f78fc0c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 --2- 172.21.15.60:0/2891440810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc0c9950 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f78fc06f200 tx=0x7f78f402be00 comp rx=0 tx=0).stop 2023-10-16T14:11:34.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 -- 172.21.15.60:0/2891440810 shutdown_connections 2023-10-16T14:11:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 --2- 172.21.15.60:0/2891440810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc0c9950 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 --2- 172.21.15.60:0/2891440810 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc0adb30 0x7f78fc0cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 -- 172.21.15.60:0/2891440810 >> 172.21.15.60:0/2891440810 conn(0x7f78fc0b0170 msgr2=0x7f78fc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 -- 172.21.15.60:0/2891440810 shutdown_connections 2023-10-16T14:11:34.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.970+0000 7f7900cdf640 1 -- 172.21.15.60:0/2891440810 wait complete. 2023-10-16T14:11:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.972+0000 7f7900cdf640 1 Processor -- start 2023-10-16T14:11:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.972+0000 7f7900cdf640 1 -- start start 2023-10-16T14:11:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f7900cdf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc06c630 0x7f78fc0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f7900cdf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc0adb30 0x7f78fc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f7900cdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f7900cdf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0c2f50 con 0x7f78fc0b0c90 2023-10-16T14:11:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f7900cdf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0b2b40 con 0x7f78fc0adb30 2023-10-16T14:11:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f7900cdf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0b2210 con 0x7f78fc06c630 2023-10-16T14:11:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc15abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50948/0 (socket says 172.21.15.60:50948) 2023-10-16T14:11:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 -- 172.21.15.60:0/3920224029 learned_addr learned my addr 172.21.15.60:0/3920224029 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fb7fe640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc06c630 0x7f78fc0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78faffd640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc0adb30 0x7f78fc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 -- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc06c630 msgr2=0x7f78fc0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc06c630 0x7f78fc0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 -- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc0adb30 msgr2=0x7f78fc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc0adb30 0x7f78fc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.974+0000 7f78fbfff640 1 -- 172.21.15.60:0/3920224029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4009b30 con 0x7f78fc0b0c90 2023-10-16T14:11:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f78faffd640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc0adb30 0x7f78fc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f78fbfff640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc15abe0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f78f4036040 tx=0x7f78f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4009510 con 0x7f78fc0b0c90 2023-10-16T14:11:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f403e450 con 0x7f78fc0b0c90 2023-10-16T14:11:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f403e7e0 con 0x7f78fc0b0c90 2023-10-16T14:11:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc0b24f0 con 0x7f78fc0b0c90 2023-10-16T14:11:34.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.975+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0b4320 con 0x7f78fc0b0c90 2023-10-16T14:11:34.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.976+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc0b2cf0 con 0x7f78fc0b0c90 2023-10-16T14:11:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.977+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f78f403e940 con 0x7f78fc0b0c90 2023-10-16T14:11:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.978+0000 7f78e3fff640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f78e806b0b0 0x7f78e806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.978+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f78f40ac560 con 0x7f78fc0b0c90 2023-10-16T14:11:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.979+0000 7f78fb7fe640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f78e806b0b0 0x7f78e806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.980+0000 7f78fb7fe640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f78e806b0b0 0x7f78e806d570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78ec004ea0 tx=0x7f78ec009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:34.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:34.982+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78fc0b2cf0 con 0x7f78fc0b0c90 2023-10-16T14:11:35.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.178+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78fc0b2cb0 con 0x7f78fc0b0c90 2023-10-16T14:11:35.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.179+0000 7f78e3fff640 1 -- 172.21.15.60:0/3920224029 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v248) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78f4043020 con 0x7f78fc0b0c90 2023-10-16T14:11:35.183 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:11:35.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f78e806b0b0 msgr2=0x7f78e806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f78e806b0b0 0x7f78e806d570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78ec004ea0 tx=0x7f78ec009040 comp rx=0 tx=0).stop 2023-10-16T14:11:35.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 msgr2=0x7f78fc15abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc15abe0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f78f4036040 tx=0x7f78f4004a40 comp rx=0 tx=0).stop 2023-10-16T14:11:35.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 shutdown_connections 2023-10-16T14:11:35.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f78e806b0b0 0x7f78e806d570 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78fc0b0c90 0x7f78fc15abe0 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc0adb30 0x7f78fc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 --2- 172.21.15.60:0/3920224029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc06c630 0x7f78fc0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 >> 172.21.15.60:0/3920224029 conn(0x7f78fc0b0170 msgr2=0x7f78fc0adfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:35.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 shutdown_connections 2023-10-16T14:11:35.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.182+0000 7f7900cdf640 1 -- 172.21.15.60:0/3920224029 wait complete. 2023-10-16T14:11:35.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:11:35.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.427+0000 7f6507251640 1 Processor -- start 2023-10-16T14:11:35.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.428+0000 7f6507251640 1 -- start start 2023-10-16T14:11:35.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.429+0000 7f6507251640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 0x7f65000ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.429+0000 7f6507251640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f650006b4c0 0x7f65000cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.429+0000 7f6507251640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650006e170 con 0x7f6500160800 2023-10-16T14:11:35.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.429+0000 7f6507251640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65000cde30 con 0x7f650006b4c0 2023-10-16T14:11:35.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.429+0000 7f6507251640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65000cd460 con 0x7f6500160420 2023-10-16T14:11:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.430+0000 7f64fffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6500160800 0x7f650006adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35192/0 (socket says 172.21.15.60:35192) 2023-10-16T14:11:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.430+0000 7f64fffff640 1 -- 172.21.15.60:0/2773547170 learned_addr learned my addr 172.21.15.60:0/2773547170 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.430+0000 7f65057ee640 1 --2- 172.21.15.60:0/2773547170 >> v2:172.21.15.144:3300/0 conn(0x7f650006b4c0 0x7f65000cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.430+0000 7f6504fed640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 0x7f65000ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.430+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630627435 0 0) 0x7f650006e170 con 0x7f6500160800 2023-10-16T14:11:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.430+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f0003520 con 0x7f6500160800 2023-10-16T14:11:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 867305519 0 0) 0x7f64f0003520 con 0x7f6500160800 2023-10-16T14:11:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 msgr2=0x7f65000ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 0x7f65000ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 >> v2:172.21.15.144:3300/0 conn(0x7f650006b4c0 msgr2=0x7f65000cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 --2- 172.21.15.60:0/2773547170 >> v2:172.21.15.144:3300/0 conn(0x7f650006b4c0 0x7f65000cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650006e3c0 con 0x7f6500160800 2023-10-16T14:11:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f64ec001670 con 0x7f6500160800 2023-10-16T14:11:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 >> v1:172.21.15.60:6789/0 conn(0x7f6500160800 legacy=0x7f650006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64f0003970 0x7f64f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f64f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 0x7f64f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65000cde30 con 0x7f64f00047c0 2023-10-16T14:11:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65000cd460 con 0x7f64f0003970 2023-10-16T14:11:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f000b980 con 0x7f6500160420 2023-10-16T14:11:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.431+0000 7f64fffff640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f64f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f6504fed640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64f0003970 0x7f64f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f65057ee640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 0x7f64f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f64fffff640 1 -- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 msgr2=0x7f64f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f64fffff640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6500160420 0x7f64f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f64fffff640 1 -- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64f0003970 msgr2=0x7f64f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:35.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f64fffff640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64f0003970 0x7f64f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.432+0000 7f64fffff640 1 -- 172.21.15.60:0/2773547170 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f0003520 con 0x7f64f00047c0 2023-10-16T14:11:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.433+0000 7f64fffff640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f64f0004c40 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00c110 tx=0x7f64f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62064184095728f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.433+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec005ad0 con 0x7f64f00047c0 2023-10-16T14:11:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.433+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec005c50 con 0x7f64f00047c0 2023-10-16T14:11:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.433+0000 7f64ff7fe640 1 -- 172.21.15.60:0/2773547170 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02c810 con 0x7f64f00047c0 2023-10-16T14:11:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f6507251640 1 -- 172.21.15.60:0/2773547170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 msgr2=0x7f64f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f6507251640 1 --2- 172.21.15.60:0/2773547170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f64f0004c40 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00c110 tx=0x7f64f00057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f65057ee640 1 -- 172.21.15.60:0/2773547170 reap_dead start 2023-10-16T14:11:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f6507251640 1 -- 172.21.15.60:0/2773547170 shutdown_connections 2023-10-16T14:11:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f6507251640 1 -- 172.21.15.60:0/2773547170 >> 172.21.15.60:0/2773547170 conn(0x7f65000b0d30 msgr2=0x7f65000b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f6507251640 1 -- 172.21.15.60:0/2773547170 shutdown_connections 2023-10-16T14:11:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.434+0000 7f6507251640 1 -- 172.21.15.60:0/2773547170 wait complete. 2023-10-16T14:11:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.436+0000 7f6507251640 1 Processor -- start 2023-10-16T14:11:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.436+0000 7f6507251640 1 -- start start 2023-10-16T14:11:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.437+0000 7f6507251640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f650009ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.437+0000 7f6507251640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65001f5f90 0x7f65001f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.437+0000 7f6507251640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65000c7a90 0x7f650006a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.437+0000 7f6507251640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65000c7360 con 0x7f64f00047c0 2023-10-16T14:11:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.437+0000 7f6507251640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65001f6410 con 0x7f65000c7a90 2023-10-16T14:11:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.437+0000 7f6507251640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65001f66e0 con 0x7f65001f5f90 2023-10-16T14:11:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f650009ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f650009ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50960/0 (socket says 172.21.15.60:50960) 2023-10-16T14:11:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 -- 172.21.15.60:0/711036500 learned_addr learned my addr 172.21.15.60:0/711036500 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f65057ee640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65000c7a90 0x7f650006a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f64fffff640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65001f5f90 0x7f65001f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 -- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65001f5f90 msgr2=0x7f65001f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65001f5f90 0x7f65001f21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 -- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65000c7a90 msgr2=0x7f650006a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65000c7a90 0x7f650006a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 -- 172.21.15.60:0/711036500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65001f69c0 con 0x7f64f00047c0 2023-10-16T14:11:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f65057ee640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65000c7a90 0x7f650006a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.438+0000 7f6504fed640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f650009ff40 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f64f4011590 tx=0x7f64f4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.439+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4015870 con 0x7f64f00047c0 2023-10-16T14:11:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.439+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f40159d0 con 0x7f64f00047c0 2023-10-16T14:11:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.439+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4015da0 con 0x7f64f00047c0 2023-10-16T14:11:35.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.439+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65001f6ca0 con 0x7f64f00047c0 2023-10-16T14:11:35.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.439+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65001f2700 con 0x7f64f00047c0 2023-10-16T14:11:35.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.440+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65001f6e90 con 0x7f64f00047c0 2023-10-16T14:11:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.441+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f64f401d400 con 0x7f64f00047c0 2023-10-16T14:11:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.443+0000 7f64fd7fa640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64e406b0b0 0x7f64e406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.443+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64f4089ef0 con 0x7f64f00047c0 2023-10-16T14:11:35.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.444+0000 7f64fffff640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64e406b0b0 0x7f64e406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.445+0000 7f64fffff640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64e406b0b0 0x7f64e406d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64ec034000 tx=0x7f64ec007000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:35.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.447+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64f4054ef0 con 0x7f64f00047c0 2023-10-16T14:11:35.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.646+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f650004e160 con 0x7f64f00047c0 2023-10-16T14:11:35.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.647+0000 7f64fd7fa640 1 -- 172.21.15.60:0/711036500 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v248) v1 ==== 84+0+65 (secure 0 0 0) 0x7f64f4058510 con 0x7f64f00047c0 2023-10-16T14:11:35.650 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:35.651 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:11:35.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:35.651 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:11:35.651 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:11:35.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:35.652 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:11:35.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.649+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64e406b0b0 msgr2=0x7f64e406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.649+0000 7f6507251640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64e406b0b0 0x7f64e406d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64ec034000 tx=0x7f64ec007000 comp rx=0 tx=0).stop 2023-10-16T14:11:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.649+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 msgr2=0x7f650009ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.649+0000 7f6507251640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f650009ff40 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f64f4011590 tx=0x7f64f4011a50 comp rx=0 tx=0).stop 2023-10-16T14:11:35.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 shutdown_connections 2023-10-16T14:11:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64e406b0b0 0x7f64e406d570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f65001f5f90 0x7f65001f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65000c7a90 0x7f650006a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 --2- 172.21.15.60:0/711036500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64f00047c0 0x7f650009ff40 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 >> 172.21.15.60:0/711036500 conn(0x7f65000b0d30 msgr2=0x7f65000cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:35.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 shutdown_connections 2023-10-16T14:11:35.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.650+0000 7f6507251640 1 -- 172.21.15.60:0/711036500 wait complete. 2023-10-16T14:11:35.673 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:11:35.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:11:35.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.905+0000 7f0147fff640 1 Processor -- start 2023-10-16T14:11:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.905+0000 7f0147fff640 1 -- start start 2023-10-16T14:11:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.906+0000 7f0147fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0148069da0 0x7f0148070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.906+0000 7f0147fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f01480c9e80 0x7f014806a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.906+0000 7f0147fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148160b10 0x7f0148160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.906+0000 7f0147fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0148071480 con 0x7f0148069da0 2023-10-16T14:11:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.906+0000 7f0147fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480cde20 con 0x7f01480c9e80 2023-10-16T14:11:35.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.906+0000 7f0147fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01480cab70 con 0x7f0148160b10 2023-10-16T14:11:35.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0148069da0 0x7f0148070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0148069da0 0x7f0148070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50962/0 (socket says 172.21.15.60:50962) 2023-10-16T14:11:35.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 -- 172.21.15.60:0/3936018451 learned_addr learned my addr 172.21.15.60:0/3936018451 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f01477fe640 1 --2- 172.21.15.60:0/3936018451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148160b10 0x7f0148160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f01467fc640 1 --2- 172.21.15.60:0/3936018451 >> v2:172.21.15.144:3300/0 conn(0x7f01480c9e80 0x7f014806a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 -- 172.21.15.60:0/3936018451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148160b10 msgr2=0x7f0148160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 --2- 172.21.15.60:0/3936018451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148160b10 0x7f0148160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 -- 172.21.15.60:0/3936018451 >> v2:172.21.15.144:3300/0 conn(0x7f01480c9e80 msgr2=0x7f014806a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 --2- 172.21.15.60:0/3936018451 >> v2:172.21.15.144:3300/0 conn(0x7f01480c9e80 0x7f014806a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.907+0000 7f0146ffd640 1 -- 172.21.15.60:0/3936018451 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480ca310 con 0x7f0148069da0 2023-10-16T14:11:35.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.909+0000 7f0146ffd640 1 --2- 172.21.15.60:0/3936018451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0148069da0 0x7f0148070f40 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f0148161bb0 tx=0x7f013802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5f42de12c2a594 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.909+0000 7f0145ffb640 1 -- 172.21.15.60:0/3936018451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013802c9f0 con 0x7f0148069da0 2023-10-16T14:11:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.909+0000 7f0145ffb640 1 -- 172.21.15.60:0/3936018451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013802cb50 con 0x7f0148069da0 2023-10-16T14:11:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.909+0000 7f0145ffb640 1 -- 172.21.15.60:0/3936018451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01380343f0 con 0x7f0148069da0 2023-10-16T14:11:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 -- 172.21.15.60:0/3936018451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0148069da0 msgr2=0x7f0148070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 --2- 172.21.15.60:0/3936018451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0148069da0 0x7f0148070f40 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f0148161bb0 tx=0x7f013802be50 comp rx=0 tx=0).stop 2023-10-16T14:11:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 -- 172.21.15.60:0/3936018451 shutdown_connections 2023-10-16T14:11:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 --2- 172.21.15.60:0/3936018451 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148160b10 0x7f0148160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 --2- 172.21.15.60:0/3936018451 >> v2:172.21.15.144:3300/0 conn(0x7f01480c9e80 0x7f014806a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 --2- 172.21.15.60:0/3936018451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0148069da0 0x7f0148070f40 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 -- 172.21.15.60:0/3936018451 >> 172.21.15.60:0/3936018451 conn(0x7f01480cf830 msgr2=0x7f01480cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:35.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 -- 172.21.15.60:0/3936018451 shutdown_connections 2023-10-16T14:11:35.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.910+0000 7f0147fff640 1 -- 172.21.15.60:0/3936018451 wait complete. 2023-10-16T14:11:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.912+0000 7f0147fff640 1 Processor -- start 2023-10-16T14:11:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.912+0000 7f0147fff640 1 -- start start 2023-10-16T14:11:35.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0147fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148069da0 0x7f014815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0147fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 0x7f01480c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0147fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0148160b10 0x7f01481f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0147fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014815dc00 con 0x7f01480c9e80 2023-10-16T14:11:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0147fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481f5ec0 con 0x7f0148160b10 2023-10-16T14:11:35.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0147fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01481f6190 con 0x7f0148069da0 2023-10-16T14:11:35.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f01467fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 0x7f01480c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f01467fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 0x7f01480c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50978/0 (socket says 172.21.15.60:50978) 2023-10-16T14:11:35.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f01467fc640 1 -- 172.21.15.60:0/1072390569 learned_addr learned my addr 172.21.15.60:0/1072390569 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:35.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f0146ffd640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148069da0 0x7f014815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.913+0000 7f01477fe640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0148160b10 0x7f01481f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01467fc640 1 -- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148069da0 msgr2=0x7f014815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01467fc640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148069da0 0x7f014815f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01467fc640 1 -- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0148160b10 msgr2=0x7f01481f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:35.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01467fc640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0148160b10 0x7f01481f5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:35.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01467fc640 1 -- 172.21.15.60:0/1072390569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0138009b30 con 0x7f01480c9e80 2023-10-16T14:11:35.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01477fe640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0148160b10 0x7f01481f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:35.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f01467fc640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 0x7f01480c5610 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f013000d800 tx=0x7f013000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:35.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01300096a0 con 0x7f01480c9e80 2023-10-16T14:11:35.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0130009800 con 0x7f01480c9e80 2023-10-16T14:11:35.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130009b90 con 0x7f01480c9e80 2023-10-16T14:11:35.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.914+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01481f6470 con 0x7f01480c9e80 2023-10-16T14:11:35.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.915+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014815de50 con 0x7f01480c9e80 2023-10-16T14:11:35.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.916+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014804e9b0 con 0x7f01480c9e80 2023-10-16T14:11:35.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.917+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0130009cf0 con 0x7f01480c9e80 2023-10-16T14:11:35.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.918+0000 7f013ffff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f012806b0b0 0x7f012806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:35.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.918+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0130089c50 con 0x7f01480c9e80 2023-10-16T14:11:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.919+0000 7f0146ffd640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f012806b0b0 0x7f012806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.920+0000 7f0146ffd640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f012806b0b0 0x7f012806d570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0138035000 tx=0x7f0138030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:35.923+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0130054c50 con 0x7f01480c9e80 2023-10-16T14:11:36.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.120+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f01480b57b0 con 0x7f01480c9e80 2023-10-16T14:11:36.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.739+0000 7f013ffff640 1 -- 172.21.15.60:0/1072390569 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v249) v1 ==== 138+0+0 (secure 0 0 0) 0x7f013000b440 con 0x7f01480c9e80 2023-10-16T14:11:36.743 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T14:11:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f012806b0b0 msgr2=0x7f012806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f012806b0b0 0x7f012806d570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0138035000 tx=0x7f0138030000 comp rx=0 tx=0).stop 2023-10-16T14:11:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 msgr2=0x7f01480c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 0x7f01480c5610 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f013000d800 tx=0x7f013000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:11:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 shutdown_connections 2023-10-16T14:11:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f012806b0b0 0x7f012806d570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0148160b10 0x7f01481f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01480c9e80 0x7f01480c5610 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 --2- 172.21.15.60:0/1072390569 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0148069da0 0x7f014815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 >> 172.21.15.60:0/1072390569 conn(0x7f01480cf830 msgr2=0x7f014806e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 shutdown_connections 2023-10-16T14:11:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.743+0000 7f0147fff640 1 -- 172.21.15.60:0/1072390569 wait complete. 2023-10-16T14:11:36.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:11:36.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.989+0000 7fdf68fe2640 1 Processor -- start 2023-10-16T14:11:36.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.989+0000 7fdf68fe2640 1 -- start start 2023-10-16T14:11:36.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf68fe2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf640b1690 0x7fdf640b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:36.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf68fe2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdf640b4ab0 0x7fdf640d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:36.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf68fe2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf640b4f00 con 0x7fdf6415cb50 2023-10-16T14:11:36.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf68fe2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf640cdbf0 con 0x7fdf640b1690 2023-10-16T14:11:36.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf68fe2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf640cde90 con 0x7fdf640b4ab0 2023-10-16T14:11:36.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf637fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdf6415cb50 0x7fdf64161180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35194/0 (socket says 172.21.15.60:35194) 2023-10-16T14:11:36.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf637fe640 1 -- 172.21.15.60:0/973256820 learned_addr learned my addr 172.21.15.60:0/973256820 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:36.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.991+0000 7fdf62ffd640 1 --2- 172.21.15.60:0/973256820 >> v2:172.21.15.60:3300/0 conn(0x7fdf640b1690 0x7fdf640b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:36.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf63fff640 1 --2- 172.21.15.60:0/973256820 >> v2:172.21.15.144:3300/0 conn(0x7fdf640b4ab0 0x7fdf640d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:36.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf627fc640 1 -- 172.21.15.60:0/973256820 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739430362 0 0) 0x7fdf54001110 con 0x7fdf6415cb50 2023-10-16T14:11:36.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf627fc640 1 -- 172.21.15.60:0/973256820 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c003520 con 0x7fdf6415cb50 2023-10-16T14:11:36.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf62ffd640 1 -- 172.21.15.60:0/973256820 >> v2:172.21.15.144:3300/0 conn(0x7fdf640b4ab0 msgr2=0x7fdf640d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:36.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf62ffd640 1 --2- 172.21.15.60:0/973256820 >> v2:172.21.15.144:3300/0 conn(0x7fdf640b4ab0 0x7fdf640d1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:36.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf62ffd640 1 -- 172.21.15.60:0/973256820 >> v1:172.21.15.60:6789/0 conn(0x7fdf6415cb50 legacy=0x7fdf64161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:36.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.992+0000 7fdf62ffd640 1 -- 172.21.15.60:0/973256820 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf640ce170 con 0x7fdf640b1690 2023-10-16T14:11:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.993+0000 7fdf62ffd640 1 --2- 172.21.15.60:0/973256820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf640b1690 0x7fdf640b4570 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fdf640b1ae0 tx=0x7fdf5802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fe9426e3672dd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf627fc640 1 -- 172.21.15.60:0/973256820 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58031710 con 0x7fdf640b1690 2023-10-16T14:11:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf627fc640 1 -- 172.21.15.60:0/973256820 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5802f050 con 0x7fdf640b1690 2023-10-16T14:11:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf627fc640 1 -- 172.21.15.60:0/973256820 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58031a90 con 0x7fdf640b1690 2023-10-16T14:11:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf68fe2640 1 -- 172.21.15.60:0/973256820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf640b1690 msgr2=0x7fdf640b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/973256820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf640b1690 0x7fdf640b4570 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fdf640b1ae0 tx=0x7fdf5802b7b0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf68fe2640 1 -- 172.21.15.60:0/973256820 shutdown_connections 2023-10-16T14:11:37.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/973256820 >> v2:172.21.15.144:3300/0 conn(0x7fdf640b4ab0 0x7fdf640d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/973256820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf640b1690 0x7fdf640b4570 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.994+0000 7fdf68fe2640 1 -- 172.21.15.60:0/973256820 >> 172.21.15.60:0/973256820 conn(0x7fdf640cf570 msgr2=0x7fdf640cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:37.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.995+0000 7fdf68fe2640 1 -- 172.21.15.60:0/973256820 shutdown_connections 2023-10-16T14:11:37.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.995+0000 7fdf68fe2640 1 -- 172.21.15.60:0/973256820 wait complete. 2023-10-16T14:11:37.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.996+0000 7fdf68fe2640 1 Processor -- start 2023-10-16T14:11:37.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 -- start start 2023-10-16T14:11:37.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf640b1690 0x7fdf640c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf640b4ab0 0x7fdf6415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 0x7fdf6415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf641f6dd0 con 0x7fdf6415cb50 2023-10-16T14:11:37.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf641f3050 con 0x7fdf640b1690 2023-10-16T14:11:37.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.997+0000 7fdf68fe2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf641f6af0 con 0x7fdf640b4ab0 2023-10-16T14:11:37.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.998+0000 7fdf63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 0x7fdf6415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.998+0000 7fdf63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 0x7fdf6415bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51006/0 (socket says 172.21.15.60:51006) 2023-10-16T14:11:37.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.998+0000 7fdf63fff640 1 -- 172.21.15.60:0/642115085 learned_addr learned my addr 172.21.15.60:0/642115085 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:37.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.998+0000 7fdf637fe640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf640b1690 0x7fdf640c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.998+0000 7fdf62ffd640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf640b4ab0 0x7fdf6415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf63fff640 1 -- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf640b4ab0 msgr2=0x7fdf6415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf63fff640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf640b4ab0 0x7fdf6415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf63fff640 1 -- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf640b1690 msgr2=0x7fdf640c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf63fff640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf640b1690 0x7fdf640c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf63fff640 1 -- 172.21.15.60:0/642115085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58009400 con 0x7fdf6415cb50 2023-10-16T14:11:37.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf63fff640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 0x7fdf6415bcc0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c007e40 tx=0x7fdf5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:37.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c014620 con 0x7fdf6415cb50 2023-10-16T14:11:37.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c00ee90 con 0x7fdf6415cb50 2023-10-16T14:11:37.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0149e0 con 0x7fdf6415cb50 2023-10-16T14:11:37.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:36.999+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf641f2a00 con 0x7fdf6415cb50 2023-10-16T14:11:37.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.000+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf640c6d90 con 0x7fdf6415cb50 2023-10-16T14:11:37.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.000+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf641f2860 con 0x7fdf6415cb50 2023-10-16T14:11:37.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.002+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdf5c014b40 con 0x7fdf6415cb50 2023-10-16T14:11:37.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.004+0000 7fdf4bfff640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdf5006b0e0 0x7fdf5006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.004+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fdf5c08ac30 con 0x7fdf6415cb50 2023-10-16T14:11:37.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.004+0000 7fdf637fe640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdf5006b0e0 0x7fdf5006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.005+0000 7fdf637fe640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdf5006b0e0 0x7fdf5006d5a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf54001270 tx=0x7fdf54007bb0 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:37.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.008+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf5c055dc0 con 0x7fdf6415cb50 2023-10-16T14:11:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.203+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdf641f2860 con 0x7fdf6415cb50 2023-10-16T14:11:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.746+0000 7fdf4bfff640 1 -- 172.21.15.60:0/642115085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v250) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdf641f2860 con 0x7fdf6415cb50 2023-10-16T14:11:37.750 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T14:11:37.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdf5006b0e0 msgr2=0x7fdf5006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdf5006b0e0 0x7fdf5006d5a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf54001270 tx=0x7fdf54007bb0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 msgr2=0x7fdf6415bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 0x7fdf6415bcc0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c007e40 tx=0x7fdf5c00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 shutdown_connections 2023-10-16T14:11:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdf5006b0e0 0x7fdf5006d5a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdf640b4ab0 0x7fdf6415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf6415cb50 0x7fdf6415bcc0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 --2- 172.21.15.60:0/642115085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf640b1690 0x7fdf640c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.751+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 >> 172.21.15.60:0/642115085 conn(0x7fdf640cf570 msgr2=0x7fdf640b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.752+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 shutdown_connections 2023-10-16T14:11:37.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.752+0000 7fdf68fe2640 1 -- 172.21.15.60:0/642115085 wait complete. 2023-10-16T14:11:37.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:11:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.982+0000 7f74a130f640 1 Processor -- start 2023-10-16T14:11:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.982+0000 7f74a130f640 1 -- start start 2023-10-16T14:11:37.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f74a130f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f749c06a4b0 0x7f749c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f74a130f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c0cb030 con 0x7f749c0c9ca0 2023-10-16T14:11:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f74a130f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c0c91c0 con 0x7f749c15ad70 2023-10-16T14:11:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f74a130f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c0c94a0 con 0x7f749c06a4b0 2023-10-16T14:11:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f749a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f749c06a4b0 0x7f749c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f749a7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f749c06a4b0 0x7f749c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51012/0 (socket says 172.21.15.60:51012) 2023-10-16T14:11:37.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f749a7fc640 1 -- 172.21.15.60:0/3485470817 learned_addr learned my addr 172.21.15.60:0/3485470817 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:37.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.984+0000 7f749b7fe640 1 --1- 172.21.15.60:0/3485470817 >> v1:172.21.15.60:6789/0 conn(0x7f749c0c9ca0 0x7f749c0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35204/0 (socket says 172.21.15.60:35204) 2023-10-16T14:11:37.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.985+0000 7f749a7fc640 1 -- 172.21.15.60:0/3485470817 >> v1:172.21.15.144:6789/0 conn(0x7f749c15ad70 legacy=0x7f749c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.985+0000 7f749a7fc640 1 -- 172.21.15.60:0/3485470817 >> v1:172.21.15.60:6789/0 conn(0x7f749c0c9ca0 legacy=0x7f749c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.985+0000 7f749a7fc640 1 -- 172.21.15.60:0/3485470817 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c0c9780 con 0x7f749c06a4b0 2023-10-16T14:11:37.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.985+0000 7f7499ffb640 1 -- 172.21.15.60:0/3485470817 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766055216 0 0) 0x7f749c0cb030 con 0x7f749c0c9ca0 2023-10-16T14:11:37.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.986+0000 7f749a7fc640 1 --2- 172.21.15.60:0/3485470817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0ca3b0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f7490009870 tx=0x7f749002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155b14ecc809ab11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:37.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.986+0000 7f7499ffb640 1 -- 172.21.15.60:0/3485470817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749002f840 con 0x7f749c06a4b0 2023-10-16T14:11:37.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.986+0000 7f7499ffb640 1 -- 172.21.15.60:0/3485470817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749002f9a0 con 0x7f749c06a4b0 2023-10-16T14:11:37.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.986+0000 7f7499ffb640 1 -- 172.21.15.60:0/3485470817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749002fd30 con 0x7f749c06a4b0 2023-10-16T14:11:37.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 -- 172.21.15.60:0/3485470817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 msgr2=0x7f749c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 --2- 172.21.15.60:0/3485470817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0ca3b0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f7490009870 tx=0x7f749002ba20 comp rx=0 tx=0).stop 2023-10-16T14:11:37.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 -- 172.21.15.60:0/3485470817 shutdown_connections 2023-10-16T14:11:37.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 --2- 172.21.15.60:0/3485470817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0ca3b0 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 -- 172.21.15.60:0/3485470817 >> 172.21.15.60:0/3485470817 conn(0x7f749c0ce9b0 msgr2=0x7f749c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:37.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 -- 172.21.15.60:0/3485470817 shutdown_connections 2023-10-16T14:11:37.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.987+0000 7f74a130f640 1 -- 172.21.15.60:0/3485470817 wait complete. 2023-10-16T14:11:37.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.990+0000 7f74a130f640 1 Processor -- start 2023-10-16T14:11:37.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.990+0000 7f74a130f640 1 -- start start 2023-10-16T14:11:37.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f74a130f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f74a130f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749c0c9ca0 0x7f749c0b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f74a130f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f749c15ad70 0x7f749c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:37.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f74a130f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c7ae0 con 0x7f749c06a4b0 2023-10-16T14:11:37.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f74a130f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c6f60 con 0x7f749c0c9ca0 2023-10-16T14:11:37.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f74a130f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c7240 con 0x7f749c15ad70 2023-10-16T14:11:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.991+0000 7f749affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51024/0 (socket says 172.21.15.60:51024) 2023-10-16T14:11:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 -- 172.21.15.60:0/840596913 learned_addr learned my addr 172.21.15.60:0/840596913 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749b7fe640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f749c15ad70 0x7f749c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749a7fc640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749c0c9ca0 0x7f749c0b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 -- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f749c15ad70 msgr2=0x7f749c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f749c15ad70 0x7f749c156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 -- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749c0c9ca0 msgr2=0x7f749c0b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749c0c9ca0 0x7f749c0b3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 -- 172.21.15.60:0/840596913 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900094d0 con 0x7f749c06a4b0 2023-10-16T14:11:38.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749b7fe640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f749c15ad70 0x7f749c156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:38.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.992+0000 7f749affd640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0c8180 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f748c00c400 tx=0x7f748c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:38.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.993+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c0052d0 con 0x7f749c06a4b0 2023-10-16T14:11:38.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.993+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c005430 con 0x7f749c06a4b0 2023-10-16T14:11:38.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.993+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c0c7520 con 0x7f749c06a4b0 2023-10-16T14:11:38.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.993+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c003830 con 0x7f749c06a4b0 2023-10-16T14:11:38.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.993+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c15b450 con 0x7f749c06a4b0 2023-10-16T14:11:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.994+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c04e160 con 0x7f749c06a4b0 2023-10-16T14:11:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.995+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f748c003990 con 0x7f749c06a4b0 2023-10-16T14:11:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.997+0000 7f74837fe640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f748806b0e0 0x7f748806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.997+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f748c088e40 con 0x7f749c06a4b0 2023-10-16T14:11:38.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.998+0000 7f749a7fc640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f748806b0e0 0x7f748806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:38.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:37.999+0000 7f749a7fc640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f748806b0e0 0x7f748806d5a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7490036000 tx=0x7f7490008030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:38.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.003+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f748c053cb0 con 0x7f749c06a4b0 2023-10-16T14:11:38.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.241+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f749c069f40 con 0x7f749c06a4b0 2023-10-16T14:11:38.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-10-16T14:11:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.840+0000 7f74837fe640 1 -- 172.21.15.60:0/840596913 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v124) v1 ==== 174+0+0 (secure 0 0 0) 0x7f748c0572d0 con 0x7f749c06a4b0 2023-10-16T14:11:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 30 and data pool 31 2023-10-16T14:11:38.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f748806b0e0 msgr2=0x7f748806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f748806b0e0 0x7f748806d5a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7490036000 tx=0x7f7490008030 comp rx=0 tx=0).stop 2023-10-16T14:11:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 msgr2=0x7f749c0c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0c8180 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f748c00c400 tx=0x7f748c00c8c0 comp rx=0 tx=0).stop 2023-10-16T14:11:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 shutdown_connections 2023-10-16T14:11:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f748806b0e0 0x7f748806d5a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f749c15ad70 0x7f749c156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f749c0c9ca0 0x7f749c0b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 --2- 172.21.15.60:0/840596913 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f749c06a4b0 0x7f749c0c8180 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 >> 172.21.15.60:0/840596913 conn(0x7f749c0ce9b0 msgr2=0x7f749c0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 shutdown_connections 2023-10-16T14:11:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:38.844+0000 7f74a130f640 1 -- 172.21.15.60:0/840596913 wait complete. 2023-10-16T14:11:38.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:39.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.062+0000 7fae4b9fc640 1 Processor -- start 2023-10-16T14:11:39.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.062+0000 7fae4b9fc640 1 -- start start 2023-10-16T14:11:39.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.064+0000 7fae4b9fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 0x7fae44164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.064+0000 7fae4b9fc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4406f7f0 con 0x7fae44159f00 2023-10-16T14:11:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.064+0000 7fae4b9fc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4406c5e0 con 0x7fae44159b20 2023-10-16T14:11:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.064+0000 7fae4b9fc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4406c8c0 con 0x7fae440b0b20 2023-10-16T14:11:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.064+0000 7fae48f97640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fae44159f00 0x7fae440b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54900/0 (socket says 172.21.15.60:54900) 2023-10-16T14:11:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.064+0000 7fae48f97640 1 -- 172.21.15.60:0/3467199624 learned_addr learned my addr 172.21.15.60:0/3467199624 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae49f99640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 0x7fae44164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603082761 0 0) 0x7fae4406f7f0 con 0x7fae44159f00 2023-10-16T14:11:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae30005730 con 0x7fae44159f00 2023-10-16T14:11:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1854059257 0 0) 0x7fae30005730 con 0x7fae44159f00 2023-10-16T14:11:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 msgr2=0x7fae44164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 0x7fae44164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 >> v1:172.21.15.144:6789/0 conn(0x7fae44159b20 legacy=0x7fae4406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4406cba0 con 0x7fae44159f00 2023-10-16T14:11:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fae34001670 con 0x7fae44159f00 2023-10-16T14:11:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 >> v1:172.21.15.60:6789/0 conn(0x7fae44159f00 legacy=0x7fae440b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 0x7fae30005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae300061e0 0x7fae300069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae30006f10 0x7fae30007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4406c8c0 con 0x7fae300061e0 2023-10-16T14:11:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3000dd50 con 0x7fae30006f10 2023-10-16T14:11:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.065+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3000df10 con 0x7fae440b0b20 2023-10-16T14:11:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae48f97640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae300061e0 0x7fae300069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae49798640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 0x7fae30005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae49f99640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae30006f10 0x7fae30007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae48f97640 1 -- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 msgr2=0x7fae30005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae48f97640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae440b0b20 0x7fae30005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae48f97640 1 -- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae30006f10 msgr2=0x7fae30007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae48f97640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae30006f10 0x7fae30007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.066+0000 7fae48f97640 1 -- 172.21.15.60:0/3467199624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30005730 con 0x7fae300061e0 2023-10-16T14:11:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.067+0000 7fae48f97640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae300061e0 0x7fae300069d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c210 tx=0x7fae30007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b682a24b9e6697 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.067+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34005ad0 con 0x7fae300061e0 2023-10-16T14:11:39.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34005c30 con 0x7fae300061e0 2023-10-16T14:11:39.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae3bfff640 1 -- 172.21.15.60:0/3467199624 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3402cad0 con 0x7fae300061e0 2023-10-16T14:11:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3467199624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae300061e0 msgr2=0x7fae300069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3467199624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae300061e0 0x7fae300069d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c210 tx=0x7fae30007a00 comp rx=0 tx=0).stop 2023-10-16T14:11:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae49f99640 1 -- 172.21.15.60:0/3467199624 reap_dead start 2023-10-16T14:11:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3467199624 shutdown_connections 2023-10-16T14:11:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3467199624 >> 172.21.15.60:0/3467199624 conn(0x7fae440cb410 msgr2=0x7fae440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3467199624 shutdown_connections 2023-10-16T14:11:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.068+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3467199624 wait complete. 2023-10-16T14:11:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.070+0000 7fae4b9fc640 1 Processor -- start 2023-10-16T14:11:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.070+0000 7fae4b9fc640 1 -- start start 2023-10-16T14:11:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.071+0000 7fae4b9fc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae300061e0 0x7fae440b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.071+0000 7fae4b9fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae440b3c90 0x7fae44150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.071+0000 7fae4b9fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 0x7fae440b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.071+0000 7fae4b9fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440b26a0 con 0x7fae440b1ce0 2023-10-16T14:11:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.071+0000 7fae4b9fc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440c5c90 con 0x7fae440b3c90 2023-10-16T14:11:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.071+0000 7fae4b9fc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae441510f0 con 0x7fae300061e0 2023-10-16T14:11:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 0x7fae440b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 0x7fae440b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56294/0 (socket says 172.21.15.60:56294) 2023-10-16T14:11:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 -- 172.21.15.60:0/3592607489 learned_addr learned my addr 172.21.15.60:0/3592607489 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae48f97640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae440b3c90 0x7fae44150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49798640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae300061e0 0x7fae440b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 -- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae300061e0 msgr2=0x7fae440b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae300061e0 0x7fae440b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 -- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae440b3c90 msgr2=0x7fae44150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae440b3c90 0x7fae44150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 -- 172.21.15.60:0/3592607489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae440b4110 con 0x7fae440b1ce0 2023-10-16T14:11:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49798640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae300061e0 0x7fae440b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.072+0000 7fae49f99640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 0x7fae440b2160 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fae440c5650 tx=0x7fae3c00f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.073+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c004440 con 0x7fae440b1ce0 2023-10-16T14:11:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.073+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c003610 con 0x7fae440b1ce0 2023-10-16T14:11:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.073+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c0039c0 con 0x7fae440b1ce0 2023-10-16T14:11:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.073+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae440c6dd0 con 0x7fae440b1ce0 2023-10-16T14:11:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.073+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae440c7390 con 0x7fae440b1ce0 2023-10-16T14:11:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.074+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4404e160 con 0x7fae440b1ce0 2023-10-16T14:11:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.075+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fae3c003b20 con 0x7fae440b1ce0 2023-10-16T14:11:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.076+0000 7fae39ffb640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae2806b080 0x7fae2806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.076+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae3c0897c0 con 0x7fae440b1ce0 2023-10-16T14:11:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.077+0000 7fae49798640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae2806b080 0x7fae2806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.078+0000 7fae49798640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae2806b080 0x7fae2806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae40007fd0 tx=0x7fae40009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.081+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae3c054470 con 0x7fae440b1ce0 2023-10-16T14:11:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.275+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae440c5a60 con 0x7fae440b1ce0 2023-10-16T14:11:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.278+0000 7fae39ffb640 1 -- 172.21.15.60:0/3592607489 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+16520 (secure 0 0 0) 0x7fae3c057a90 con 0x7fae440b1ce0 2023-10-16T14:11:39.282 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:39.282 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":251,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:38.243207+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:39.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae2806b080 msgr2=0x7fae2806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae2806b080 0x7fae2806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae40007fd0 tx=0x7fae40009040 comp rx=0 tx=0).stop 2023-10-16T14:11:39.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 msgr2=0x7fae440b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 0x7fae440b2160 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fae440c5650 tx=0x7fae3c00f920 comp rx=0 tx=0).stop 2023-10-16T14:11:39.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 shutdown_connections 2023-10-16T14:11:39.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fae2806b080 0x7fae2806d540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae440b1ce0 0x7fae440b2160 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae440b3c90 0x7fae44150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 --2- 172.21.15.60:0/3592607489 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fae300061e0 0x7fae440b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.280+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 >> 172.21.15.60:0/3592607489 conn(0x7fae440cb410 msgr2=0x7fae440b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:39.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.281+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 shutdown_connections 2023-10-16T14:11:39.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.281+0000 7fae4b9fc640 1 -- 172.21.15.60:0/3592607489 wait complete. 2023-10-16T14:11:39.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:39.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.508+0000 7f65162d1640 1 Processor -- start 2023-10-16T14:11:39.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.508+0000 7f65162d1640 1 -- start start 2023-10-16T14:11:39.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.509+0000 7f65162d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 0x7f65100709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.509+0000 7f65162d1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6510160a90 con 0x7f6510070f20 2023-10-16T14:11:39.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.510+0000 7f65162d1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65100cd900 con 0x7f651015c9f0 2023-10-16T14:11:39.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.510+0000 7f65162d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65100cdec0 con 0x7f651015dde0 2023-10-16T14:11:39.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.510+0000 7f6515ad0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6510070f20 0x7f65100ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54910/0 (socket says 172.21.15.60:54910) 2023-10-16T14:11:39.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.510+0000 7f6515ad0640 1 -- 172.21.15.60:0/3800256679 learned_addr learned my addr 172.21.15.60:0/3800256679 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:39.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.510+0000 7f6514ace640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 0x7f65100709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.510+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443369892 0 0) 0x7f6510160a90 con 0x7f6510070f20 2023-10-16T14:11:39.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64fc003520 con 0x7f6510070f20 2023-10-16T14:11:39.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1638121503 0 0) 0x7f64fc003520 con 0x7f6510070f20 2023-10-16T14:11:39.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 msgr2=0x7f65100709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 0x7f65100709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 >> v1:172.21.15.144:6789/0 conn(0x7f651015c9f0 legacy=0x7f65100c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65100cdbe0 con 0x7f6510070f20 2023-10-16T14:11:39.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f65080019f0 con 0x7f6510070f20 2023-10-16T14:11:39.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 >> v1:172.21.15.60:6789/0 conn(0x7f6510070f20 legacy=0x7f65100ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0039a0 0x7f64fc003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc004350 0x7f64fc004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 0x7f64fc005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00b980 con 0x7f64fc004350 2023-10-16T14:11:39.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00bac0 con 0x7f64fc0039a0 2023-10-16T14:11:39.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc00bc80 con 0x7f651015dde0 2023-10-16T14:11:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.511+0000 7f6514ace640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc004350 0x7f64fc004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.512+0000 7f6514ace640 1 -- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 msgr2=0x7f64fc005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.512+0000 7f6514ace640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f651015dde0 0x7f64fc005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.512+0000 7f6514ace640 1 -- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0039a0 msgr2=0x7f64fc003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.512+0000 7f6514ace640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64fc0039a0 0x7f64fc003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.512+0000 7f6514ace640 1 -- 172.21.15.60:0/3800256679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc003520 con 0x7f64fc004350 2023-10-16T14:11:39.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.513+0000 7f6514ace640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc004350 0x7f64fc004820 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f6500009a90 tx=0x7f64fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1cbeca1bf0c1065 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.513+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650002c6b0 con 0x7f64fc004350 2023-10-16T14:11:39.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.513+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650002c810 con 0x7f64fc004350 2023-10-16T14:11:39.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.513+0000 7f6507fff640 1 -- 172.21.15.60:0/3800256679 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650002cba0 con 0x7f64fc004350 2023-10-16T14:11:39.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f65162d1640 1 -- 172.21.15.60:0/3800256679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc004350 msgr2=0x7f64fc004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f65162d1640 1 --2- 172.21.15.60:0/3800256679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64fc004350 0x7f64fc004820 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f6500009a90 tx=0x7f64fc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f6515ad0640 1 -- 172.21.15.60:0/3800256679 reap_dead start 2023-10-16T14:11:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f65162d1640 1 -- 172.21.15.60:0/3800256679 shutdown_connections 2023-10-16T14:11:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f65162d1640 1 -- 172.21.15.60:0/3800256679 >> 172.21.15.60:0/3800256679 conn(0x7f65100ce6a0 msgr2=0x7f65100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f65162d1640 1 -- 172.21.15.60:0/3800256679 shutdown_connections 2023-10-16T14:11:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.514+0000 7f65162d1640 1 -- 172.21.15.60:0/3800256679 wait complete. 2023-10-16T14:11:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.516+0000 7f65162d1640 1 Processor -- start 2023-10-16T14:11:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.516+0000 7f65162d1640 1 -- start start 2023-10-16T14:11:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65162d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc004350 0x7f65100c35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65162d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65100c3b30 0x7f651006afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65162d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 0x7f65101497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65162d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651014c760 con 0x7f651006b510 2023-10-16T14:11:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65162d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c4f80 con 0x7f65100c3b30 2023-10-16T14:11:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65162d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c4370 con 0x7f64fc004350 2023-10-16T14:11:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 0x7f65101497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 0x7f65101497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56316/0 (socket says 172.21.15.60:56316) 2023-10-16T14:11:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 -- 172.21.15.60:0/199299372 learned_addr learned my addr 172.21.15.60:0/199299372 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f65152cf640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc004350 0x7f65100c35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 -- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc004350 msgr2=0x7f65100c35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc004350 0x7f65100c35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 -- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65100c3b30 msgr2=0x7f651006afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65100c3b30 0x7f651006afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.517+0000 7f6515ad0640 1 -- 172.21.15.60:0/199299372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65100c4650 con 0x7f651006b510 2023-10-16T14:11:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.518+0000 7f6515ad0640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 0x7f65101497e0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f65100b1260 tx=0x7f650800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.518+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650800cea0 con 0x7f651006b510 2023-10-16T14:11:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.518+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650800a610 con 0x7f651006b510 2023-10-16T14:11:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.518+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65100c4930 con 0x7f651006b510 2023-10-16T14:11:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.518+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650800a9a0 con 0x7f651006b510 2023-10-16T14:11:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.518+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65100c5410 con 0x7f651006b510 2023-10-16T14:11:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.519+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651004e9b0 con 0x7f651006b510 2023-10-16T14:11:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.520+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f650800ab00 con 0x7f651006b510 2023-10-16T14:11:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.521+0000 7f6505ffb640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64f406b0e0 0x7f64f406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.522+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6508089360 con 0x7f651006b510 2023-10-16T14:11:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.522+0000 7f65152cf640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64f406b0e0 0x7f64f406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.523+0000 7f65152cf640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64f406b0e0 0x7f64f406d5a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f651006ce30 tx=0x7f650c009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.525+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6508057aa0 con 0x7f651006b510 2023-10-16T14:11:39.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.725+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65100c50c0 con 0x7f651006b510 2023-10-16T14:11:39.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.727+0000 7f6505ffb640 1 -- 172.21.15.60:0/199299372 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+16520 (secure 0 0 0) 0x7f6508057630 con 0x7f651006b510 2023-10-16T14:11:39.731 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:39.731 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":251,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:38.243207+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:39.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64f406b0e0 msgr2=0x7f64f406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64f406b0e0 0x7f64f406d5a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f651006ce30 tx=0x7f650c009040 comp rx=0 tx=0).stop 2023-10-16T14:11:39.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 msgr2=0x7f65101497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 0x7f65101497e0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f65100b1260 tx=0x7f650800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 shutdown_connections 2023-10-16T14:11:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f64f406b0e0 0x7f64f406d5a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f651006b510 0x7f65101497e0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65100c3b30 0x7f651006afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 --2- 172.21.15.60:0/199299372 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f64fc004350 0x7f65100c35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 >> 172.21.15.60:0/199299372 conn(0x7f65100ce6a0 msgr2=0x7f651015e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:39.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 shutdown_connections 2023-10-16T14:11:39.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.730+0000 7f65162d1640 1 -- 172.21.15.60:0/199299372 wait complete. 2023-10-16T14:11:39.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:11:39.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.966+0000 7f4560c72640 1 Processor -- start 2023-10-16T14:11:39.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.966+0000 7f4560c72640 1 -- start start 2023-10-16T14:11:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.967+0000 7f4560c72640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 0x7f455c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.967+0000 7f4560c72640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455c15d150 con 0x7f455c0cded0 2023-10-16T14:11:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.967+0000 7f4560c72640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455c0d1180 con 0x7f455c0b8480 2023-10-16T14:11:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.967+0000 7f4560c72640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455c0d1460 con 0x7f455c15cd70 2023-10-16T14:11:39.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.968+0000 7f455bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f455c0cded0 0x7f455c0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54922/0 (socket says 172.21.15.60:54922) 2023-10-16T14:11:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.968+0000 7f455bfff640 1 -- 172.21.15.60:0/3255666925 learned_addr learned my addr 172.21.15.60:0/3255666925 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.968+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 0x7f455c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.968+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292533453 0 0) 0x7f455c15d150 con 0x7f455c0cded0 2023-10-16T14:11:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.968+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4548003520 con 0x7f455c0cded0 2023-10-16T14:11:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 250630208 0 0) 0x7f4548003520 con 0x7f455c0cded0 2023-10-16T14:11:39.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 msgr2=0x7f455c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 0x7f455c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 >> v1:172.21.15.144:6789/0 conn(0x7f455c0b8480 legacy=0x7f455c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c0d1740 con 0x7f455c0cded0 2023-10-16T14:11:39.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4554001670 con 0x7f455c0cded0 2023-10-16T14:11:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 >> v1:172.21.15.60:6789/0 conn(0x7f455c0cded0 legacy=0x7f455c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45480039a0 0x7f4548003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480047c0 0x7f4548004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 0x7f4548005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c0d1460 con 0x7f45480039a0 2023-10-16T14:11:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454800ba30 con 0x7f45480047c0 2023-10-16T14:11:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454800bbf0 con 0x7f455c15cd70 2023-10-16T14:11:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.969+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45480039a0 0x7f4548003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455b7fe640 1 -- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 msgr2=0x7f4548005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c15cd70 0x7f4548005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455affd640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480047c0 0x7f4548004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455b7fe640 1 -- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480047c0 msgr2=0x7f4548004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:39.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480047c0 0x7f4548004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455b7fe640 1 -- 172.21.15.60:0/3255666925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548003520 con 0x7f45480039a0 2023-10-16T14:11:39.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.970+0000 7f455affd640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480047c0 0x7f4548004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:39.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.971+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45480039a0 0x7f4548003dc0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f454c009f90 tx=0x7f45480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49d8bf2f42b2769 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.971+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c02cbf0 con 0x7f45480039a0 2023-10-16T14:11:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.971+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454c00f040 con 0x7f45480039a0 2023-10-16T14:11:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.971+0000 7f455a7fc640 1 -- 172.21.15.60:0/3255666925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c034440 con 0x7f45480039a0 2023-10-16T14:11:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f4560c72640 1 -- 172.21.15.60:0/3255666925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45480039a0 msgr2=0x7f4548003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f4560c72640 1 --2- 172.21.15.60:0/3255666925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45480039a0 0x7f4548003dc0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f454c009f90 tx=0x7f45480057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f455bfff640 1 -- 172.21.15.60:0/3255666925 reap_dead start 2023-10-16T14:11:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f4560c72640 1 -- 172.21.15.60:0/3255666925 shutdown_connections 2023-10-16T14:11:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f4560c72640 1 -- 172.21.15.60:0/3255666925 >> 172.21.15.60:0/3255666925 conn(0x7f455c0cf570 msgr2=0x7f455c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f4560c72640 1 -- 172.21.15.60:0/3255666925 shutdown_connections 2023-10-16T14:11:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.972+0000 7f4560c72640 1 -- 172.21.15.60:0/3255666925 wait complete. 2023-10-16T14:11:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.974+0000 7f4560c72640 1 Processor -- start 2023-10-16T14:11:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.974+0000 7f4560c72640 1 -- start start 2023-10-16T14:11:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f4560c72640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480039a0 0x7f455c159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f4560c72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 0x7f455c0d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f4560c72640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c1f2000 0x7f455c0d3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f4560c72640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c158f00 con 0x7f455c1f1b80 2023-10-16T14:11:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f4560c72640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c158630 con 0x7f45480039a0 2023-10-16T14:11:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f4560c72640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455c1493a0 con 0x7f455c1f2000 2023-10-16T14:11:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f455affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 0x7f455c0d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f455affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 0x7f455c0d2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56322/0 (socket says 172.21.15.60:56322) 2023-10-16T14:11:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.975+0000 7f455affd640 1 -- 172.21.15.60:0/3344753274 learned_addr learned my addr 172.21.15.60:0/3344753274 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480039a0 0x7f455c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455affd640 1 -- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c1f2000 msgr2=0x7f455c0d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455affd640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c1f2000 0x7f455c0d3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455affd640 1 -- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480039a0 msgr2=0x7f455c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455affd640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480039a0 0x7f455c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455affd640 1 -- 172.21.15.60:0/3344753274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c1588b0 con 0x7f455c1f1b80 2023-10-16T14:11:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480039a0 0x7f455c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f455affd640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 0x7f455c0d2bf0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f454400d7a0 tx=0x7f454400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544017070 con 0x7f455c1f1b80 2023-10-16T14:11:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4544005070 con 0x7f455c1f1b80 2023-10-16T14:11:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544005420 con 0x7f455c1f1b80 2023-10-16T14:11:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.976+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c158b90 con 0x7f455c1f1b80 2023-10-16T14:11:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.977+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c1465e0 con 0x7f455c1f1b80 2023-10-16T14:11:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.978+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c04e9b0 con 0x7f455c1f1b80 2023-10-16T14:11:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.978+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4544007030 con 0x7f455c1f1b80 2023-10-16T14:11:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.979+0000 7f4553fff640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f453c06b0b0 0x7f453c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.979+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4544089550 con 0x7f455c1f1b80 2023-10-16T14:11:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.980+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f453c06b0b0 0x7f453c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.981+0000 7f455b7fe640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f453c06b0b0 0x7f453c06d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f454c036000 tx=0x7f454c02e000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:39.984+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f45440540f0 con 0x7f455c1f1b80 2023-10-16T14:11:40.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.222+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f455c159040 con 0x7f455c1f1b80 2023-10-16T14:11:40.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.857+0000 7f4553fff640 1 -- 172.21.15.60:0/3344753274 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v127) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4544057710 con 0x7f455c1f1b80 2023-10-16T14:11:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f453c06b0b0 msgr2=0x7f453c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f453c06b0b0 0x7f453c06d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f454c036000 tx=0x7f454c02e000 comp rx=0 tx=0).stop 2023-10-16T14:11:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 msgr2=0x7f455c0d2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 0x7f455c0d2bf0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f454400d7a0 tx=0x7f454400dc60 comp rx=0 tx=0).stop 2023-10-16T14:11:40.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 shutdown_connections 2023-10-16T14:11:40.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f453c06b0b0 0x7f453c06d570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:40.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f455c1f2000 0x7f455c0d3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:40.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f455c1f1b80 0x7f455c0d2bf0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:40.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 --2- 172.21.15.60:0/3344753274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45480039a0 0x7f455c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:40.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 >> 172.21.15.60:0/3344753274 conn(0x7f455c0cf570 msgr2=0x7f455c0cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:40.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 shutdown_connections 2023-10-16T14:11:40.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:40.861+0000 7f4560c72640 1 -- 172.21.15.60:0/3344753274 wait complete. 2023-10-16T14:11:40.888 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:11:40.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:11:41.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.117+0000 7f5216cb4640 1 Processor -- start 2023-10-16T14:11:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.117+0000 7f5216cb4640 1 -- start start 2023-10-16T14:11:41.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.118+0000 7f5216cb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f52100cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f5216cb4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52100d0be0 0x7f52100cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f5216cb4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521015d150 con 0x7f521015cd70 2023-10-16T14:11:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f5216cb4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100d1180 con 0x7f52100d0be0 2023-10-16T14:11:41.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f5216cb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100d1460 con 0x7f52100b8480 2023-10-16T14:11:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f52164b3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52100d0be0 0x7f52100cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f5215cb2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f521015cd70 0x7f52100d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54928/0 (socket says 172.21.15.60:54928) 2023-10-16T14:11:41.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f5215cb2640 1 -- 172.21.15.60:0/572920741 learned_addr learned my addr 172.21.15.60:0/572920741 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.119+0000 7f52154b1640 1 --2- 172.21.15.60:0/572920741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f52100cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.120+0000 7f52164b3640 1 -- 172.21.15.60:0/572920741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 msgr2=0x7f52100cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.120+0000 7f52164b3640 1 --2- 172.21.15.60:0/572920741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f52100cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.120+0000 7f52164b3640 1 -- 172.21.15.60:0/572920741 >> v1:172.21.15.60:6789/0 conn(0x7f521015cd70 legacy=0x7f52100d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:41.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.120+0000 7f52164b3640 1 -- 172.21.15.60:0/572920741 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100d1740 con 0x7f52100d0be0 2023-10-16T14:11:41.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.120+0000 7f5214cb0640 1 -- 172.21.15.60:0/572920741 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545527264 0 0) 0x7f521015d150 con 0x7f521015cd70 2023-10-16T14:11:41.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.121+0000 7f52164b3640 1 --2- 172.21.15.60:0/572920741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100d0be0 0x7f52100cdd00 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f5208009d90 tx=0x7f520802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162145896c36e27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:41.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.121+0000 7f5214cb0640 1 -- 172.21.15.60:0/572920741 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520802cc50 con 0x7f52100d0be0 2023-10-16T14:11:41.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.121+0000 7f5214cb0640 1 -- 172.21.15.60:0/572920741 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520802cdb0 con 0x7f52100d0be0 2023-10-16T14:11:41.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.121+0000 7f5214cb0640 1 -- 172.21.15.60:0/572920741 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208034660 con 0x7f52100d0be0 2023-10-16T14:11:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 -- 172.21.15.60:0/572920741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100d0be0 msgr2=0x7f52100cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 --2- 172.21.15.60:0/572920741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100d0be0 0x7f52100cdd00 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f5208009d90 tx=0x7f520802bea0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 -- 172.21.15.60:0/572920741 shutdown_connections 2023-10-16T14:11:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 --2- 172.21.15.60:0/572920741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52100d0be0 0x7f52100cdd00 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 --2- 172.21.15.60:0/572920741 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f52100cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 -- 172.21.15.60:0/572920741 >> 172.21.15.60:0/572920741 conn(0x7f52100cf570 msgr2=0x7f52100cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 -- 172.21.15.60:0/572920741 shutdown_connections 2023-10-16T14:11:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.122+0000 7f5216cb4640 1 -- 172.21.15.60:0/572920741 wait complete. 2023-10-16T14:11:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.124+0000 7f5216cb4640 1 Processor -- start 2023-10-16T14:11:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.124+0000 7f5216cb4640 1 -- start start 2023-10-16T14:11:41.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.125+0000 7f5216cb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f5210151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.125+0000 7f5216cb4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52100d0be0 0x7f521015e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.125+0000 7f5216cb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 0x7f52100ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.125+0000 7f5216cb4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100ca4b0 con 0x7f521015cd70 2023-10-16T14:11:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.125+0000 7f5216cb4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100c9bc0 con 0x7f52100d0be0 2023-10-16T14:11:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.125+0000 7f5216cb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100ccad0 con 0x7f52100b8480 2023-10-16T14:11:41.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 0x7f52100ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:41.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 0x7f52100ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56332/0 (socket says 172.21.15.60:56332) 2023-10-16T14:11:41.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 -- 172.21.15.60:0/2877449561 learned_addr learned my addr 172.21.15.60:0/2877449561 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:41.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52154b1640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52100d0be0 0x7f521015e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:41.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f5215cb2640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f5210151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:41.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 -- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 msgr2=0x7f5210151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:41.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f5210151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 -- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52100d0be0 msgr2=0x7f521015e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:41.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52100d0be0 0x7f521015e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.126+0000 7f52164b3640 1 -- 172.21.15.60:0/2877449561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208009a70 con 0x7f521015cd70 2023-10-16T14:11:41.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.127+0000 7f52164b3640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 0x7f52100ca940 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f5208009d90 tx=0x7f5208004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:41.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.127+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208040650 con 0x7f521015cd70 2023-10-16T14:11:41.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.127+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52080407b0 con 0x7f521015cd70 2023-10-16T14:11:41.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.127+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52100c9e60 con 0x7f521015cd70 2023-10-16T14:11:41.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.127+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208040b60 con 0x7f521015cd70 2023-10-16T14:11:41.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.127+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52100cc710 con 0x7f521015cd70 2023-10-16T14:11:41.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.128+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52100cc8a0 con 0x7f521015cd70 2023-10-16T14:11:41.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.129+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5208040cc0 con 0x7f521015cd70 2023-10-16T14:11:41.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.130+0000 7f52067fc640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f51f406b100 0x7f51f406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:41.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.131+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52080ac810 con 0x7f521015cd70 2023-10-16T14:11:41.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.131+0000 7f5215cb2640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f51f406b100 0x7f51f406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:41.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.133+0000 7f5215cb2640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f51f406b100 0x7f51f406d5c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f520c000f60 tx=0x7f520c007b50 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:41.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.135+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5208077540 con 0x7f521015cd70 2023-10-16T14:11:41.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.377+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f521004e9b0 con 0x7f521015cd70 2023-10-16T14:11:41.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.871+0000 7f52067fc640 1 -- 172.21.15.60:0/2877449561 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v128) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5208009450 con 0x7f521015cd70 2023-10-16T14:11:41.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.875+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f51f406b100 msgr2=0x7f51f406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:41.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.875+0000 7f5216cb4640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f51f406b100 0x7f51f406d5c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f520c000f60 tx=0x7f520c007b50 comp rx=0 tx=0).stop 2023-10-16T14:11:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.875+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 msgr2=0x7f52100ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.875+0000 7f5216cb4640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 0x7f52100ca940 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f5208009d90 tx=0x7f5208004a40 comp rx=0 tx=0).stop 2023-10-16T14:11:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.875+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 shutdown_connections 2023-10-16T14:11:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f51f406b100 0x7f51f406d5c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f521015cd70 0x7f52100ca940 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52100d0be0 0x7f521015e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 --2- 172.21.15.60:0/2877449561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52100b8480 0x7f5210151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:41.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 >> 172.21.15.60:0/2877449561 conn(0x7f52100cf570 msgr2=0x7f52100cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:41.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 shutdown_connections 2023-10-16T14:11:41.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:41.876+0000 7f5216cb4640 1 -- 172.21.15.60:0/2877449561 wait complete. 2023-10-16T14:11:41.903 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:11:41.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:42.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.115+0000 7f41af76c640 1 Processor -- start 2023-10-16T14:11:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.115+0000 7f41af76c640 1 -- start start 2023-10-16T14:11:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.116+0000 7f41af76c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41a806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.116+0000 7f41af76c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f41a815e540 0x7f41a80b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.116+0000 7f41af76c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80cd630 con 0x7f41a815e0c0 2023-10-16T14:11:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.116+0000 7f41af76c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80ce1b0 con 0x7f41a815e540 2023-10-16T14:11:42.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.116+0000 7f41af76c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a806cd10 con 0x7f41a806c3f0 2023-10-16T14:11:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.117+0000 7f41acd07640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f41a815e0c0 0x7f41a80b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54944/0 (socket says 172.21.15.60:54944) 2023-10-16T14:11:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.117+0000 7f41acd07640 1 -- 172.21.15.60:0/3207879891 learned_addr learned my addr 172.21.15.60:0/3207879891 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.117+0000 7f41add09640 1 --2- 172.21.15.60:0/3207879891 >> v2:172.21.15.144:3300/0 conn(0x7f41a815e540 0x7f41a80b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.117+0000 7f41ad508640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41a806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.117+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099005061 0 0) 0x7f41a80cd630 con 0x7f41a815e0c0 2023-10-16T14:11:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.117+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4194003520 con 0x7f41a815e0c0 2023-10-16T14:11:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133048199 0 0) 0x7f4194003520 con 0x7f41a815e0c0 2023-10-16T14:11:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 msgr2=0x7f41a806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41a806c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 >> v2:172.21.15.144:3300/0 conn(0x7f41a815e540 msgr2=0x7f41a80b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 --2- 172.21.15.60:0/3207879891 >> v2:172.21.15.144:3300/0 conn(0x7f41a815e540 0x7f41a80b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80d2170 con 0x7f41a815e0c0 2023-10-16T14:11:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4198001670 con 0x7f41a815e0c0 2023-10-16T14:11:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 >> v1:172.21.15.60:6789/0 conn(0x7f41a815e0c0 legacy=0x7f41a80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f4194003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4194004d00 0x7f4194005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80ce1b0 con 0x7f4194003970 2023-10-16T14:11:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a806cd10 con 0x7f4194004d00 2023-10-16T14:11:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419400b980 con 0x7f41a806c3f0 2023-10-16T14:11:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f41add09640 1 --2- 172.21.15.60:0/3207879891 >> v2:172.21.15.144:3300/0 conn(0x7f41a815e540 0x7f41a80b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:11:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.118+0000 7f41ad508640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f4194003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41acd07640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41ad508640 1 -- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 msgr2=0x7f41940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41ad508640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41ad508640 1 -- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4194004d00 msgr2=0x7f4194005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:42.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41ad508640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4194004d00 0x7f4194005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41ad508640 1 -- 172.21.15.60:0/3207879891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4194003520 con 0x7f4194003970 2023-10-16T14:11:42.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.119+0000 7f41acd07640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a806c3f0 0x7f41940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.120+0000 7f41ad508640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f4194003d90 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009ac0 tx=0x7f41940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd37de04c8faa10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.120+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a402c6b0 con 0x7f4194003970 2023-10-16T14:11:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.120+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a402c810 con 0x7f4194003970 2023-10-16T14:11:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.120+0000 7f419ffff640 1 -- 172.21.15.60:0/3207879891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a402cba0 con 0x7f4194003970 2023-10-16T14:11:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41af76c640 1 -- 172.21.15.60:0/3207879891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 msgr2=0x7f4194003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41af76c640 1 --2- 172.21.15.60:0/3207879891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f4194003d90 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009ac0 tx=0x7f41940057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41add09640 1 -- 172.21.15.60:0/3207879891 reap_dead start 2023-10-16T14:11:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41af76c640 1 -- 172.21.15.60:0/3207879891 shutdown_connections 2023-10-16T14:11:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41af76c640 1 -- 172.21.15.60:0/3207879891 >> 172.21.15.60:0/3207879891 conn(0x7f41a80d2cb0 msgr2=0x7f41a80d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:42.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41af76c640 1 -- 172.21.15.60:0/3207879891 shutdown_connections 2023-10-16T14:11:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.121+0000 7f41af76c640 1 -- 172.21.15.60:0/3207879891 wait complete. 2023-10-16T14:11:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.123+0000 7f41af76c640 1 Processor -- start 2023-10-16T14:11:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.123+0000 7f41af76c640 1 -- start start 2023-10-16T14:11:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.124+0000 7f41af76c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f41a80cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.124+0000 7f41af76c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a815f4f0 0x7f41a81f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.124+0000 7f41af76c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41a80d0ea0 0x7f41a80d1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.124+0000 7f41af76c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81f2c00 con 0x7f4194003970 2023-10-16T14:11:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.124+0000 7f41af76c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80d1860 con 0x7f41a80d0ea0 2023-10-16T14:11:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.124+0000 7f41af76c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80c9cf0 con 0x7f41a815f4f0 2023-10-16T14:11:42.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f41a80cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f41a80cc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56342/0 (socket says 172.21.15.60:56342) 2023-10-16T14:11:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 -- 172.21.15.60:0/719851786 learned_addr learned my addr 172.21.15.60:0/719851786 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41acd07640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a815f4f0 0x7f41a81f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41add09640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41a80d0ea0 0x7f41a80d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 -- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a815f4f0 msgr2=0x7f41a81f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a815f4f0 0x7f41a81f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 -- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41a80d0ea0 msgr2=0x7f41a80d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41a80d0ea0 0x7f41a80d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 -- 172.21.15.60:0/719851786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80ca590 con 0x7f4194003970 2023-10-16T14:11:42.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.125+0000 7f41ad508640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f41a80cc3f0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f41a402c250 tx=0x7f41a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.126+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a4003810 con 0x7f4194003970 2023-10-16T14:11:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.126+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a4003970 con 0x7f4194003970 2023-10-16T14:11:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.126+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a4003ce0 con 0x7f4194003970 2023-10-16T14:11:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.126+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a80a97b0 con 0x7f4194003970 2023-10-16T14:11:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.126+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a80c9f40 con 0x7f4194003970 2023-10-16T14:11:42.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.127+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a804e160 con 0x7f4194003970 2023-10-16T14:11:42.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.128+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f41a4034440 con 0x7f4194003970 2023-10-16T14:11:42.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.129+0000 7f419dffb640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f418c06b0b0 0x7f418c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.129+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41a40aba40 con 0x7f4194003970 2023-10-16T14:11:42.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.130+0000 7f41acd07640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f418c06b0b0 0x7f418c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.131+0000 7f41acd07640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f418c06b0b0 0x7f418c06d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f41a80cc280 tx=0x7f4198009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:42.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.134+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41a40766f0 con 0x7f4194003970 2023-10-16T14:11:42.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.369+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41a815b240 con 0x7f4194003970 2023-10-16T14:11:42.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.370+0000 7f419dffb640 1 -- 172.21.15.60:0/719851786 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f41a4039090 con 0x7f4194003970 2023-10-16T14:11:42.374 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:42.375 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T14:11:42.375 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 128 2023-10-16T14:11:42.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f418c06b0b0 msgr2=0x7f418c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f418c06b0b0 0x7f418c06d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f41a80cc280 tx=0x7f4198009040 comp rx=0 tx=0).stop 2023-10-16T14:11:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 msgr2=0x7f41a80cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f41a80cc3f0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f41a402c250 tx=0x7f41a4004730 comp rx=0 tx=0).stop 2023-10-16T14:11:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 shutdown_connections 2023-10-16T14:11:42.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f418c06b0b0 0x7f418c06d570 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f41a815f4f0 0x7f41a81f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f41a80d0ea0 0x7f41a80d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 --2- 172.21.15.60:0/719851786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4194003970 0x7f41a80cc3f0 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 >> 172.21.15.60:0/719851786 conn(0x7f41a80d2cb0 msgr2=0x7f41a815d700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:42.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 shutdown_connections 2023-10-16T14:11:42.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.373+0000 7f41af76c640 1 -- 172.21.15.60:0/719851786 wait complete. 2023-10-16T14:11:42.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:42.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.610+0000 7f7a73409640 1 Processor -- start 2023-10-16T14:11:42.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.610+0000 7f7a73409640 1 -- start start 2023-10-16T14:11:42.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.612+0000 7f7a73409640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7a6c0cd0c0 0x7f7a6c1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.612+0000 7f7a73409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c15db00 0x7f7a6c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.612+0000 7f7a73409640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c15e3e0 con 0x7f7a6c0b0410 2023-10-16T14:11:42.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.612+0000 7f7a73409640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6c0b0de0 con 0x7f7a6c0cd0c0 2023-10-16T14:11:42.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.612+0000 7f7a73409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0cd4a0 con 0x7f7a6c15db00 2023-10-16T14:11:42.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.612+0000 7f7a711a5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7a6c0cd0c0 0x7f7a6c1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a719a6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c15db00 0x7f7a6c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a711a5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7a6c0cd0c0 0x7f7a6c1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:42268/0 (socket says 172.21.15.60:42268) 2023-10-16T14:11:42.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a711a5640 1 -- 172.21.15.60:0/2684709274 learned_addr learned my addr 172.21.15.60:0/2684709274 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:42.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a711a5640 1 -- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c15db00 msgr2=0x7f7a6c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:42.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a711a5640 1 --2- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c15db00 0x7f7a6c0b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a711a5640 1 -- 172.21.15.60:0/2684709274 >> v1:172.21.15.144:6789/0 conn(0x7f7a6c0b0410 legacy=0x7f7a6c15d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.613+0000 7f7a711a5640 1 -- 172.21.15.60:0/2684709274 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c15e100 con 0x7f7a6c0cd0c0 2023-10-16T14:11:42.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.615+0000 7f7a711a5640 1 --2- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c0cd0c0 0x7f7a6c1615f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009870 tx=0x7f7a6802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d72380ab5b29682 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:42.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.616+0000 7f7a63fff640 1 -- 172.21.15.60:0/2684709274 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68031640 con 0x7f7a6c0cd0c0 2023-10-16T14:11:42.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.616+0000 7f7a63fff640 1 -- 172.21.15.60:0/2684709274 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a680317a0 con 0x7f7a6c0cd0c0 2023-10-16T14:11:42.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.616+0000 7f7a63fff640 1 -- 172.21.15.60:0/2684709274 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68031b30 con 0x7f7a6c0cd0c0 2023-10-16T14:11:42.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.616+0000 7f7a73409640 1 -- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c0cd0c0 msgr2=0x7f7a6c1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.616+0000 7f7a73409640 1 --2- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c0cd0c0 0x7f7a6c1615f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009870 tx=0x7f7a6802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.617+0000 7f7a73409640 1 -- 172.21.15.60:0/2684709274 shutdown_connections 2023-10-16T14:11:42.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.617+0000 7f7a73409640 1 --2- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c15db00 0x7f7a6c0b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.617+0000 7f7a73409640 1 --2- 172.21.15.60:0/2684709274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c0cd0c0 0x7f7a6c1615f0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.617+0000 7f7a73409640 1 -- 172.21.15.60:0/2684709274 >> 172.21.15.60:0/2684709274 conn(0x7f7a6c0cb410 msgr2=0x7f7a6c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:42.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.617+0000 7f7a73409640 1 -- 172.21.15.60:0/2684709274 shutdown_connections 2023-10-16T14:11:42.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.617+0000 7f7a73409640 1 -- 172.21.15.60:0/2684709274 wait complete. 2023-10-16T14:11:42.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.619+0000 7f7a73409640 1 Processor -- start 2023-10-16T14:11:42.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.619+0000 7f7a73409640 1 -- start start 2023-10-16T14:11:42.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.620+0000 7f7a73409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 0x7f7a6c0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.620+0000 7f7a73409640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c0cd0c0 0x7f7a6c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.620+0000 7f7a73409640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c15db00 0x7f7a6c143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a73409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0c6320 con 0x7f7a6c0b0410 2023-10-16T14:11:42.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a73409640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c15e590 con 0x7f7a6c15db00 2023-10-16T14:11:42.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a73409640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c15f3f0 con 0x7f7a6c0cd0c0 2023-10-16T14:11:42.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 0x7f7a6c0c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 0x7f7a6c0c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56350/0 (socket says 172.21.15.60:56350) 2023-10-16T14:11:42.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 -- 172.21.15.60:0/3011393204 learned_addr learned my addr 172.21.15.60:0/3011393204 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:42.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a719a6640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c15db00 0x7f7a6c143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 -- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c0cd0c0 msgr2=0x7f7a6c0c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a709a4640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c0cd0c0 0x7f7a6c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c0cd0c0 0x7f7a6c0c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 -- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c15db00 msgr2=0x7f7a6c143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c15db00 0x7f7a6c143a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a711a5640 1 -- 172.21.15.60:0/3011393204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680094d0 con 0x7f7a6c0b0410 2023-10-16T14:11:42.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.621+0000 7f7a719a6640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c15db00 0x7f7a6c143a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:42.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.622+0000 7f7a711a5640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 0x7f7a6c0c2960 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7a68000c00 tx=0x7f7a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:42.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.622+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6803a930 con 0x7f7a6c0b0410 2023-10-16T14:11:42.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.622+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6803aa90 con 0x7f7a6c0b0410 2023-10-16T14:11:42.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.622+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6803ae40 con 0x7f7a6c0b0410 2023-10-16T14:11:42.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.622+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c15e7e0 con 0x7f7a6c0b0410 2023-10-16T14:11:42.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.622+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c15eda0 con 0x7f7a6c0b0410 2023-10-16T14:11:42.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.624+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c04e160 con 0x7f7a6c0b0410 2023-10-16T14:11:42.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.625+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7a6802f050 con 0x7f7a6c0b0410 2023-10-16T14:11:42.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.626+0000 7f7a61ffb640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7a5006b0b0 0x7f7a5006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:42.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.626+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a680ab920 con 0x7f7a6c0b0410 2023-10-16T14:11:42.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.627+0000 7f7a709a4640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7a5006b0b0 0x7f7a5006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:42.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.628+0000 7f7a709a4640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7a5006b0b0 0x7f7a5006d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c06ced0 tx=0x7f7a5c009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:42.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.631+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a68076560 con 0x7f7a6c0b0410 2023-10-16T14:11:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.826+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a6c160ae0 con 0x7f7a6c0b0410 2023-10-16T14:11:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.829+0000 7f7a61ffb640 1 -- 172.21.15.60:0/3011393204 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+16507 (secure 0 0 0) 0x7f7a68035090 con 0x7f7a6c0b0410 2023-10-16T14:11:42.833 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:42.833 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":252,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:39.765986+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:42.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7a5006b0b0 msgr2=0x7f7a5006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7a5006b0b0 0x7f7a5006d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c06ced0 tx=0x7f7a5c009040 comp rx=0 tx=0).stop 2023-10-16T14:11:42.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 msgr2=0x7f7a6c0c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:42.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 0x7f7a6c0c2960 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7a68000c00 tx=0x7f7a680046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 shutdown_connections 2023-10-16T14:11:42.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7a5006b0b0 0x7f7a5006d570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a6c0cd0c0 0x7f7a6c0c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a6c15db00 0x7f7a6c143a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 --2- 172.21.15.60:0/3011393204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a6c0b0410 0x7f7a6c0c2960 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:42.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 >> 172.21.15.60:0/3011393204 conn(0x7f7a6c0cb410 msgr2=0x7f7a6c15b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:42.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.832+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 shutdown_connections 2023-10-16T14:11:42.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:42.833+0000 7f7a73409640 1 -- 172.21.15.60:0/3011393204 wait complete. 2023-10-16T14:11:42.856 INFO:tasks.ceph.mds.b.smithi144.stderr:2023-10-16T14:11:42.853+0000 7fb338a3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:11:42.857 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:11:42.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:11:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.088+0000 7f94ce883640 1 Processor -- start 2023-10-16T14:11:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.088+0000 7f94ce883640 1 -- start start 2023-10-16T14:11:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94ce883640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80cd0c0 0x7f94c81615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94ce883640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94c80b0410 0x7f94c80b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94ce883640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c80b0db0 con 0x7f94c815d3f0 2023-10-16T14:11:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94ce883640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c815e3e0 con 0x7f94c80b0410 2023-10-16T14:11:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94ce883640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80cd4a0 con 0x7f94c80cd0c0 2023-10-16T14:11:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94bf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94c80b0410 0x7f94c80b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94cce20640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f94c815d3f0 0x7f94c815d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54958/0 (socket says 172.21.15.60:54958) 2023-10-16T14:11:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.090+0000 7f94cce20640 1 -- 172.21.15.60:0/2661728967 learned_addr learned my addr 172.21.15.60:0/2661728967 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.091+0000 7f94c7fff640 1 --2- 172.21.15.60:0/2661728967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80cd0c0 0x7f94c81615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.091+0000 7f94bf7fe640 1 -- 172.21.15.60:0/2661728967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80cd0c0 msgr2=0x7f94c81615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.091+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/2661728967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80cd0c0 0x7f94c81615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.091+0000 7f94bf7fe640 1 -- 172.21.15.60:0/2661728967 >> v1:172.21.15.60:6789/0 conn(0x7f94c815d3f0 legacy=0x7f94c815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.091+0000 7f94bf7fe640 1 -- 172.21.15.60:0/2661728967 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c815e100 con 0x7f94c80b0410 2023-10-16T14:11:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.092+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/2661728967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80b0410 0x7f94c80b0870 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f94b00097a0 tx=0x7f94b002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89262be82db7534 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.092+0000 7f94c77fe640 1 -- 172.21.15.60:0/2661728967 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0031780 con 0x7f94c80b0410 2023-10-16T14:11:43.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.092+0000 7f94c77fe640 1 -- 172.21.15.60:0/2661728967 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b00318e0 con 0x7f94c80b0410 2023-10-16T14:11:43.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.092+0000 7f94c77fe640 1 -- 172.21.15.60:0/2661728967 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0031c70 con 0x7f94c80b0410 2023-10-16T14:11:43.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 -- 172.21.15.60:0/2661728967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80b0410 msgr2=0x7f94c80b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 --2- 172.21.15.60:0/2661728967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80b0410 0x7f94c80b0870 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f94b00097a0 tx=0x7f94b002b820 comp rx=0 tx=0).stop 2023-10-16T14:11:43.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 -- 172.21.15.60:0/2661728967 shutdown_connections 2023-10-16T14:11:43.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 --2- 172.21.15.60:0/2661728967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80b0410 0x7f94c80b0870 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 --2- 172.21.15.60:0/2661728967 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80cd0c0 0x7f94c81615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 -- 172.21.15.60:0/2661728967 >> 172.21.15.60:0/2661728967 conn(0x7f94c80cb410 msgr2=0x7f94c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:43.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 -- 172.21.15.60:0/2661728967 shutdown_connections 2023-10-16T14:11:43.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.093+0000 7f94ce883640 1 -- 172.21.15.60:0/2661728967 wait complete. 2023-10-16T14:11:43.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.096+0000 7f94ce883640 1 Processor -- start 2023-10-16T14:11:43.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.096+0000 7f94ce883640 1 -- start start 2023-10-16T14:11:43.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.097+0000 7f94ce883640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80b0410 0x7f94c8156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.097+0000 7f94ce883640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 0x7f94c80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.097+0000 7f94ce883640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c815d3f0 0x7f94c80b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94ce883640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80c5290 con 0x7f94c80cd0c0 2023-10-16T14:11:43.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94ce883640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b1570 con 0x7f94c815d3f0 2023-10-16T14:11:43.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94ce883640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b1850 con 0x7f94c80b0410 2023-10-16T14:11:43.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 0x7f94c80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 0x7f94c80b3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56372/0 (socket says 172.21.15.60:56372) 2023-10-16T14:11:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 -- 172.21.15.60:0/3090383596 learned_addr learned my addr 172.21.15.60:0/3090383596 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94cce20640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c815d3f0 0x7f94c80b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94c7fff640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80b0410 0x7f94c8156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 -- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80b0410 msgr2=0x7f94c8156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80b0410 0x7f94c8156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 -- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c815d3f0 msgr2=0x7f94c80b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c815d3f0 0x7f94c80b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94bf7fe640 1 -- 172.21.15.60:0/3090383596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b0009400 con 0x7f94c80cd0c0 2023-10-16T14:11:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.098+0000 7f94c7fff640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80b0410 0x7f94c8156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:43.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.099+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 0x7f94c80b3060 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f94b0034040 tx=0x7f94b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.099+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0032820 con 0x7f94c80cd0c0 2023-10-16T14:11:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.099+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0032980 con 0x7f94c80cd0c0 2023-10-16T14:11:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.099+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0032d30 con 0x7f94c80cd0c0 2023-10-16T14:11:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.099+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c80c4cd0 con 0x7f94c80cd0c0 2023-10-16T14:11:43.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.099+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c80b1b30 con 0x7f94c80cd0c0 2023-10-16T14:11:43.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.101+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c80add80 con 0x7f94c80cd0c0 2023-10-16T14:11:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.101+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f94b004e440 con 0x7f94c80cd0c0 2023-10-16T14:11:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.102+0000 7f94c57fa640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f94a806b0b0 0x7f94a806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.103+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94b00abb70 con 0x7f94c80cd0c0 2023-10-16T14:11:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.103+0000 7f94c7fff640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f94a806b0b0 0x7f94a806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.104+0000 7f94c7fff640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f94a806b0b0 0x7f94a806d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f94b8006fd0 tx=0x7f94b8008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:43.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.107+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94c80add80 con 0x7f94c80cd0c0 2023-10-16T14:11:43.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.300+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 --> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94c8160df0 con 0x7f94a806b0b0 2023-10-16T14:11:43.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.330+0000 7f94c57fa640 1 -- 172.21.15.60:0/3090383596 <== mgr.9169 v2:172.21.15.144:6832/4052124783 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287061 (secure 0 0 0) 0x7f94b80095f0 con 0x7f94a806b0b0 2023-10-16T14:11:43.339 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:43.341 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:11:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f94a806b0b0 msgr2=0x7f94a806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f94a806b0b0 0x7f94a806d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f94b8006fd0 tx=0x7f94b8008040 comp rx=0 tx=0).stop 2023-10-16T14:11:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 msgr2=0x7f94c80b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 0x7f94c80b3060 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f94b0034040 tx=0x7f94b00046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 shutdown_connections 2023-10-16T14:11:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f94a806b0b0 0x7f94a806d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c815d3f0 0x7f94c80b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c80cd0c0 0x7f94c80b3060 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 --2- 172.21.15.60:0/3090383596 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c80b0410 0x7f94c8156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 >> 172.21.15.60:0/3090383596 conn(0x7f94c80cb410 msgr2=0x7f94c80c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 shutdown_connections 2023-10-16T14:11:43.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.336+0000 7f94ce883640 1 -- 172.21.15.60:0/3090383596 wait complete. 2023-10-16T14:11:43.361 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T14:11:43.062286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":667,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":427988,"kb_used_data":25876,"kb_used_omap":12,"kb_used_meta":401971,"kb_avail":754546732,"statfs":{"total":773094113280,"available":772655853568,"internally_reserved":0,"allocated":26497024,"data_stored":13556368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411618872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002264"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810787+0000","last_change":"2023-10-16T14:11:38.800766+0000","last_active":"2023-10-16T14:11:38.810787+0000","last_peered":"2023-10-16T14:11:38.810787+0000","last_clean":"2023-10-16T14:11:38.810787+0000","last_became_active":"2023-10-16T14:11:38.799147+0000","last_became_peered":"2023-10-16T14:11:38.799147+0000","last_unstale":"2023-10-16T14:11:38.810787+0000","last_undegraded":"2023-10-16T14:11:38.810787+0000","last_fullsized":"2023-10-16T14:11:38.810787+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:01.252078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813060+0000","last_change":"2023-10-16T14:11:38.811708+0000","last_active":"2023-10-16T14:11:38.813060+0000","last_peered":"2023-10-16T14:11:38.813060+0000","last_clean":"2023-10-16T14:11:38.813060+0000","last_became_active":"2023-10-16T14:11:38.808060+0000","last_became_peered":"2023-10-16T14:11:38.808060+0000","last_unstale":"2023-10-16T14:11:38.813060+0000","last_undegraded":"2023-10-16T14:11:38.813060+0000","last_fullsized":"2023-10-16T14:11:38.813060+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:45.519076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783555+0000","last_change":"2023-10-16T14:11:38.799851+0000","last_active":"2023-10-16T14:11:39.783555+0000","last_peered":"2023-10-16T14:11:39.783555+0000","last_clean":"2023-10-16T14:11:39.783555+0000","last_became_active":"2023-10-16T14:11:38.799520+0000","last_became_peered":"2023-10-16T14:11:38.799520+0000","last_unstale":"2023-10-16T14:11:39.783555+0000","last_undegraded":"2023-10-16T14:11:39.783555+0000","last_fullsized":"2023-10-16T14:11:39.783555+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:59.480982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813598+0000","last_change":"2023-10-16T14:11:38.811457+0000","last_active":"2023-10-16T14:11:38.813598+0000","last_peered":"2023-10-16T14:11:38.813598+0000","last_clean":"2023-10-16T14:11:38.813598+0000","last_became_active":"2023-10-16T14:11:38.811124+0000","last_became_peered":"2023-10-16T14:11:38.811124+0000","last_unstale":"2023-10-16T14:11:38.813598+0000","last_undegraded":"2023-10-16T14:11:38.813598+0000","last_fullsized":"2023-10-16T14:11:38.813598+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:38.977236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791702+0000","last_change":"2023-10-16T14:11:38.792747+0000","last_active":"2023-10-16T14:11:39.791702+0000","last_peered":"2023-10-16T14:11:39.791702+0000","last_clean":"2023-10-16T14:11:39.791702+0000","last_became_active":"2023-10-16T14:11:38.792334+0000","last_became_peered":"2023-10-16T14:11:38.792334+0000","last_unstale":"2023-10-16T14:11:39.791702+0000","last_undegraded":"2023-10-16T14:11:39.791702+0000","last_fullsized":"2023-10-16T14:11:39.791702+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:23.689830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.814593+0000","last_change":"2023-10-16T14:11:38.814105+0000","last_active":"2023-10-16T14:11:38.814593+0000","last_peered":"2023-10-16T14:11:38.814593+0000","last_clean":"2023-10-16T14:11:38.814593+0000","last_became_active":"2023-10-16T14:11:38.813562+0000","last_became_peered":"2023-10-16T14:11:38.813562+0000","last_unstale":"2023-10-16T14:11:38.814593+0000","last_undegraded":"2023-10-16T14:11:38.814593+0000","last_fullsized":"2023-10-16T14:11:38.814593+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:37.458489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808022+0000","last_change":"2023-10-16T14:11:38.807311+0000","last_active":"2023-10-16T14:11:38.808022+0000","last_peered":"2023-10-16T14:11:38.808022+0000","last_clean":"2023-10-16T14:11:38.808022+0000","last_became_active":"2023-10-16T14:11:38.806946+0000","last_became_peered":"2023-10-16T14:11:38.806946+0000","last_unstale":"2023-10-16T14:11:38.808022+0000","last_undegraded":"2023-10-16T14:11:38.808022+0000","last_fullsized":"2023-10-16T14:11:38.808022+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:39.460099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.788640+0000","last_change":"2023-10-16T14:11:38.796621+0000","last_active":"2023-10-16T14:11:39.788640+0000","last_peered":"2023-10-16T14:11:39.788640+0000","last_clean":"2023-10-16T14:11:39.788640+0000","last_became_active":"2023-10-16T14:11:38.795857+0000","last_became_peered":"2023-10-16T14:11:38.795857+0000","last_unstale":"2023-10-16T14:11:39.788640+0000","last_undegraded":"2023-10-16T14:11:39.788640+0000","last_fullsized":"2023-10-16T14:11:39.788640+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:24.189032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808166+0000","last_change":"2023-10-16T14:11:38.807477+0000","last_active":"2023-10-16T14:11:38.808166+0000","last_peered":"2023-10-16T14:11:38.808166+0000","last_clean":"2023-10-16T14:11:38.808166+0000","last_became_active":"2023-10-16T14:11:38.807149+0000","last_became_peered":"2023-10-16T14:11:38.807149+0000","last_unstale":"2023-10-16T14:11:38.808166+0000","last_undegraded":"2023-10-16T14:11:38.808166+0000","last_fullsized":"2023-10-16T14:11:38.808166+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:43.212625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804329+0000","last_change":"2023-10-16T14:11:38.801109+0000","last_active":"2023-10-16T14:11:38.804329+0000","last_peered":"2023-10-16T14:11:38.804329+0000","last_clean":"2023-10-16T14:11:38.804329+0000","last_became_active":"2023-10-16T14:11:38.800831+0000","last_became_peered":"2023-10-16T14:11:38.800831+0000","last_unstale":"2023-10-16T14:11:38.804329+0000","last_undegraded":"2023-10-16T14:11:38.804329+0000","last_fullsized":"2023-10-16T14:11:38.804329+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:03.392070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811147+0000","last_change":"2023-10-16T14:11:38.806641+0000","last_active":"2023-10-16T14:11:38.811147+0000","last_peered":"2023-10-16T14:11:38.811147+0000","last_clean":"2023-10-16T14:11:38.811147+0000","last_became_active":"2023-10-16T14:11:38.806333+0000","last_became_peered":"2023-10-16T14:11:38.806333+0000","last_unstale":"2023-10-16T14:11:38.811147+0000","last_undegraded":"2023-10-16T14:11:38.811147+0000","last_fullsized":"2023-10-16T14:11:38.811147+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:12.068784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789574+0000","last_change":"2023-10-16T14:11:38.803389+0000","last_active":"2023-10-16T14:11:39.789574+0000","last_peered":"2023-10-16T14:11:39.789574+0000","last_clean":"2023-10-16T14:11:39.789574+0000","last_became_active":"2023-10-16T14:11:38.803101+0000","last_became_peered":"2023-10-16T14:11:38.803101+0000","last_unstale":"2023-10-16T14:11:39.789574+0000","last_undegraded":"2023-10-16T14:11:39.789574+0000","last_fullsized":"2023-10-16T14:11:39.789574+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:45.651230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811853+0000","last_change":"2023-10-16T14:11:38.811074+0000","last_active":"2023-10-16T14:11:38.811853+0000","last_peered":"2023-10-16T14:11:38.811853+0000","last_clean":"2023-10-16T14:11:38.811853+0000","last_became_active":"2023-10-16T14:11:38.810772+0000","last_became_peered":"2023-10-16T14:11:38.810772+0000","last_unstale":"2023-10-16T14:11:38.811853+0000","last_undegraded":"2023-10-16T14:11:38.811853+0000","last_fullsized":"2023-10-16T14:11:38.811853+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:23.246968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813033+0000","last_change":"2023-10-16T14:11:38.806714+0000","last_active":"2023-10-16T14:11:38.813033+0000","last_peered":"2023-10-16T14:11:38.813033+0000","last_clean":"2023-10-16T14:11:38.813033+0000","last_became_active":"2023-10-16T14:11:38.805598+0000","last_became_peered":"2023-10-16T14:11:38.805598+0000","last_unstale":"2023-10-16T14:11:38.813033+0000","last_undegraded":"2023-10-16T14:11:38.813033+0000","last_fullsized":"2023-10-16T14:11:38.813033+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:41.125819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.809098+0000","last_change":"2023-10-16T14:11:38.808575+0000","last_active":"2023-10-16T14:11:38.809098+0000","last_peered":"2023-10-16T14:11:38.809098+0000","last_clean":"2023-10-16T14:11:38.809098+0000","last_became_active":"2023-10-16T14:11:38.808273+0000","last_became_peered":"2023-10-16T14:11:38.808273+0000","last_unstale":"2023-10-16T14:11:38.809098+0000","last_undegraded":"2023-10-16T14:11:38.809098+0000","last_fullsized":"2023-10-16T14:11:38.809098+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:04.882730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782613+0000","last_change":"2023-10-16T14:11:38.804694+0000","last_active":"2023-10-16T14:11:39.782613+0000","last_peered":"2023-10-16T14:11:39.782613+0000","last_clean":"2023-10-16T14:11:39.782613+0000","last_became_active":"2023-10-16T14:11:38.804373+0000","last_became_peered":"2023-10-16T14:11:38.804373+0000","last_unstale":"2023-10-16T14:11:39.782613+0000","last_undegraded":"2023-10-16T14:11:39.782613+0000","last_fullsized":"2023-10-16T14:11:39.782613+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:42.092693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790292+0000","last_change":"2023-10-16T14:11:38.799470+0000","last_active":"2023-10-16T14:11:39.790292+0000","last_peered":"2023-10-16T14:11:39.790292+0000","last_clean":"2023-10-16T14:11:39.790292+0000","last_became_active":"2023-10-16T14:11:38.799170+0000","last_became_peered":"2023-10-16T14:11:38.799170+0000","last_unstale":"2023-10-16T14:11:39.790292+0000","last_undegraded":"2023-10-16T14:11:39.790292+0000","last_fullsized":"2023-10-16T14:11:39.790292+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:23.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807159+0000","last_change":"2023-10-16T14:11:38.801518+0000","last_active":"2023-10-16T14:11:38.807159+0000","last_peered":"2023-10-16T14:11:38.807159+0000","last_clean":"2023-10-16T14:11:38.807159+0000","last_became_active":"2023-10-16T14:11:38.801133+0000","last_became_peered":"2023-10-16T14:11:38.801133+0000","last_unstale":"2023-10-16T14:11:38.807159+0000","last_undegraded":"2023-10-16T14:11:38.807159+0000","last_fullsized":"2023-10-16T14:11:38.807159+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:03.231050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790492+0000","last_change":"2023-10-16T14:11:38.792372+0000","last_active":"2023-10-16T14:11:39.790492+0000","last_peered":"2023-10-16T14:11:39.790492+0000","last_clean":"2023-10-16T14:11:39.790492+0000","last_became_active":"2023-10-16T14:11:38.789719+0000","last_became_peered":"2023-10-16T14:11:38.789719+0000","last_unstale":"2023-10-16T14:11:39.790492+0000","last_undegraded":"2023-10-16T14:11:39.790492+0000","last_fullsized":"2023-10-16T14:11:39.790492+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:30:12.869949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783436+0000","last_change":"2023-10-16T14:11:38.802052+0000","last_active":"2023-10-16T14:11:39.783436+0000","last_peered":"2023-10-16T14:11:39.783436+0000","last_clean":"2023-10-16T14:11:39.783436+0000","last_became_active":"2023-10-16T14:11:38.801625+0000","last_became_peered":"2023-10-16T14:11:38.801625+0000","last_unstale":"2023-10-16T14:11:39.783436+0000","last_undegraded":"2023-10-16T14:11:39.783436+0000","last_fullsized":"2023-10-16T14:11:39.783436+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:27.067528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789528+0000","last_change":"2023-10-16T14:11:38.797163+0000","last_active":"2023-10-16T14:11:39.789528+0000","last_peered":"2023-10-16T14:11:39.789528+0000","last_clean":"2023-10-16T14:11:39.789528+0000","last_became_active":"2023-10-16T14:11:38.796761+0000","last_became_peered":"2023-10-16T14:11:38.796761+0000","last_unstale":"2023-10-16T14:11:39.789528+0000","last_undegraded":"2023-10-16T14:11:39.789528+0000","last_fullsized":"2023-10-16T14:11:39.789528+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:01.946364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.805625+0000","last_change":"2023-10-16T14:11:38.802974+0000","last_active":"2023-10-16T14:11:38.805625+0000","last_peered":"2023-10-16T14:11:38.805625+0000","last_clean":"2023-10-16T14:11:38.805625+0000","last_became_active":"2023-10-16T14:11:38.802711+0000","last_became_peered":"2023-10-16T14:11:38.802711+0000","last_unstale":"2023-10-16T14:11:38.805625+0000","last_undegraded":"2023-10-16T14:11:38.805625+0000","last_fullsized":"2023-10-16T14:11:38.805625+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:33.739529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790726+0000","last_change":"2023-10-16T14:11:38.800092+0000","last_active":"2023-10-16T14:11:39.790726+0000","last_peered":"2023-10-16T14:11:39.790726+0000","last_clean":"2023-10-16T14:11:39.790726+0000","last_became_active":"2023-10-16T14:11:38.799625+0000","last_became_peered":"2023-10-16T14:11:38.799625+0000","last_unstale":"2023-10-16T14:11:39.790726+0000","last_undegraded":"2023-10-16T14:11:39.790726+0000","last_fullsized":"2023-10-16T14:11:39.790726+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:06.091417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.812184+0000","last_change":"2023-10-16T14:11:38.809521+0000","last_active":"2023-10-16T14:11:38.812184+0000","last_peered":"2023-10-16T14:11:38.812184+0000","last_clean":"2023-10-16T14:11:38.812184+0000","last_became_active":"2023-10-16T14:11:38.808121+0000","last_became_peered":"2023-10-16T14:11:38.808121+0000","last_unstale":"2023-10-16T14:11:38.812184+0000","last_undegraded":"2023-10-16T14:11:38.812184+0000","last_fullsized":"2023-10-16T14:11:38.812184+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:18.290336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813613+0000","last_change":"2023-10-16T14:11:38.811608+0000","last_active":"2023-10-16T14:11:38.813613+0000","last_peered":"2023-10-16T14:11:38.813613+0000","last_clean":"2023-10-16T14:11:38.813613+0000","last_became_active":"2023-10-16T14:11:38.811328+0000","last_became_peered":"2023-10-16T14:11:38.811328+0000","last_unstale":"2023-10-16T14:11:38.813613+0000","last_undegraded":"2023-10-16T14:11:38.813613+0000","last_fullsized":"2023-10-16T14:11:38.813613+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:33.536538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811861+0000","last_change":"2023-10-16T14:11:38.811158+0000","last_active":"2023-10-16T14:11:38.811861+0000","last_peered":"2023-10-16T14:11:38.811861+0000","last_clean":"2023-10-16T14:11:38.811861+0000","last_became_active":"2023-10-16T14:11:38.810852+0000","last_became_peered":"2023-10-16T14:11:38.810852+0000","last_unstale":"2023-10-16T14:11:38.811861+0000","last_undegraded":"2023-10-16T14:11:38.811861+0000","last_fullsized":"2023-10-16T14:11:38.811861+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:35:39.262135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810241+0000","last_change":"2023-10-16T14:11:38.804475+0000","last_active":"2023-10-16T14:11:38.810241+0000","last_peered":"2023-10-16T14:11:38.810241+0000","last_clean":"2023-10-16T14:11:38.810241+0000","last_became_active":"2023-10-16T14:11:38.804187+0000","last_became_peered":"2023-10-16T14:11:38.804187+0000","last_unstale":"2023-10-16T14:11:38.810241+0000","last_undegraded":"2023-10-16T14:11:38.810241+0000","last_fullsized":"2023-10-16T14:11:38.810241+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:40.929143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.812844+0000","last_change":"2023-10-16T14:11:38.808466+0000","last_active":"2023-10-16T14:11:38.812844+0000","last_peered":"2023-10-16T14:11:38.812844+0000","last_clean":"2023-10-16T14:11:38.812844+0000","last_became_active":"2023-10-16T14:11:38.808142+0000","last_became_peered":"2023-10-16T14:11:38.808142+0000","last_unstale":"2023-10-16T14:11:38.812844+0000","last_undegraded":"2023-10-16T14:11:38.812844+0000","last_fullsized":"2023-10-16T14:11:38.812844+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:08.231712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.809204+0000","last_change":"2023-10-16T14:11:38.802531+0000","last_active":"2023-10-16T14:11:38.809204+0000","last_peered":"2023-10-16T14:11:38.809204+0000","last_clean":"2023-10-16T14:11:38.809204+0000","last_became_active":"2023-10-16T14:11:38.802127+0000","last_became_peered":"2023-10-16T14:11:38.802127+0000","last_unstale":"2023-10-16T14:11:38.809204+0000","last_undegraded":"2023-10-16T14:11:38.809204+0000","last_fullsized":"2023-10-16T14:11:38.809204+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:07.396974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791820+0000","last_change":"2023-10-16T14:11:38.798475+0000","last_active":"2023-10-16T14:11:39.791820+0000","last_peered":"2023-10-16T14:11:39.791820+0000","last_clean":"2023-10-16T14:11:39.791820+0000","last_became_active":"2023-10-16T14:11:38.797198+0000","last_became_peered":"2023-10-16T14:11:38.797198+0000","last_unstale":"2023-10-16T14:11:39.791820+0000","last_undegraded":"2023-10-16T14:11:39.791820+0000","last_fullsized":"2023-10-16T14:11:39.791820+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:37.361885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808651+0000","last_change":"2023-10-16T14:11:38.803008+0000","last_active":"2023-10-16T14:11:38.808651+0000","last_peered":"2023-10-16T14:11:38.808651+0000","last_clean":"2023-10-16T14:11:38.808651+0000","last_became_active":"2023-10-16T14:11:38.800876+0000","last_became_peered":"2023-10-16T14:11:38.800876+0000","last_unstale":"2023-10-16T14:11:38.808651+0000","last_undegraded":"2023-10-16T14:11:38.808651+0000","last_fullsized":"2023-10-16T14:11:38.808651+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.534242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807455+0000","last_change":"2023-10-16T14:11:38.803699+0000","last_active":"2023-10-16T14:11:38.807455+0000","last_peered":"2023-10-16T14:11:38.807455+0000","last_clean":"2023-10-16T14:11:38.807455+0000","last_became_active":"2023-10-16T14:11:38.801233+0000","last_became_peered":"2023-10-16T14:11:38.801233+0000","last_unstale":"2023-10-16T14:11:38.807455+0000","last_undegraded":"2023-10-16T14:11:38.807455+0000","last_fullsized":"2023-10-16T14:11:38.807455+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:25.816517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.784122+0000","last_change":"2023-10-16T14:11:38.800010+0000","last_active":"2023-10-16T14:11:39.784122+0000","last_peered":"2023-10-16T14:11:39.784122+0000","last_clean":"2023-10-16T14:11:39.784122+0000","last_became_active":"2023-10-16T14:11:38.799422+0000","last_became_peered":"2023-10-16T14:11:38.799422+0000","last_unstale":"2023-10-16T14:11:39.784122+0000","last_undegraded":"2023-10-16T14:11:39.784122+0000","last_fullsized":"2023-10-16T14:11:39.784122+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:01:31.181539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.852015+0000","last_change":"2023-10-16T14:11:37.778697+0000","last_active":"2023-10-16T14:11:38.852015+0000","last_peered":"2023-10-16T14:11:38.852015+0000","last_clean":"2023-10-16T14:11:38.852015+0000","last_became_active":"2023-10-16T14:11:37.778387+0000","last_became_peered":"2023-10-16T14:11:37.778387+0000","last_unstale":"2023-10-16T14:11:38.852015+0000","last_undegraded":"2023-10-16T14:11:38.852015+0000","last_fullsized":"2023-10-16T14:11:38.852015+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:09.302616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810839+0000","last_change":"2023-10-16T14:11:38.805302+0000","last_active":"2023-10-16T14:11:38.810839+0000","last_peered":"2023-10-16T14:11:38.810839+0000","last_clean":"2023-10-16T14:11:38.810839+0000","last_became_active":"2023-10-16T14:11:38.803432+0000","last_became_peered":"2023-10-16T14:11:38.803432+0000","last_unstale":"2023-10-16T14:11:38.810839+0000","last_undegraded":"2023-10-16T14:11:38.810839+0000","last_fullsized":"2023-10-16T14:11:38.810839+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:35.951261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783759+0000","last_change":"2023-10-16T14:11:37.781195+0000","last_active":"2023-10-16T14:11:39.783759+0000","last_peered":"2023-10-16T14:11:39.783759+0000","last_clean":"2023-10-16T14:11:39.783759+0000","last_became_active":"2023-10-16T14:11:37.780551+0000","last_became_peered":"2023-10-16T14:11:37.780551+0000","last_unstale":"2023-10-16T14:11:39.783759+0000","last_undegraded":"2023-10-16T14:11:39.783759+0000","last_fullsized":"2023-10-16T14:11:39.783759+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:33.391410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790836+0000","last_change":"2023-10-16T14:11:38.802734+0000","last_active":"2023-10-16T14:11:39.790836+0000","last_peered":"2023-10-16T14:11:39.790836+0000","last_clean":"2023-10-16T14:11:39.790836+0000","last_became_active":"2023-10-16T14:11:38.802347+0000","last_became_peered":"2023-10-16T14:11:38.802347+0000","last_unstale":"2023-10-16T14:11:39.790836+0000","last_undegraded":"2023-10-16T14:11:39.790836+0000","last_fullsized":"2023-10-16T14:11:39.790836+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:38.977236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783524+0000","last_change":"2023-10-16T14:11:37.783063+0000","last_active":"2023-10-16T14:11:39.783524+0000","last_peered":"2023-10-16T14:11:39.783524+0000","last_clean":"2023-10-16T14:11:39.783524+0000","last_became_active":"2023-10-16T14:11:37.782768+0000","last_became_peered":"2023-10-16T14:11:37.782768+0000","last_unstale":"2023-10-16T14:11:39.783524+0000","last_undegraded":"2023-10-16T14:11:39.783524+0000","last_fullsized":"2023-10-16T14:11:39.783524+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:36.980155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.800168+0000","last_change":"2023-10-16T14:11:38.786427+0000","last_active":"2023-10-16T14:11:38.800168+0000","last_peered":"2023-10-16T14:11:38.800168+0000","last_clean":"2023-10-16T14:11:38.800168+0000","last_became_active":"2023-10-16T14:11:38.785671+0000","last_became_peered":"2023-10-16T14:11:38.785671+0000","last_unstale":"2023-10-16T14:11:38.800168+0000","last_undegraded":"2023-10-16T14:11:38.800168+0000","last_fullsized":"2023-10-16T14:11:38.800168+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:28.844112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.854622+0000","last_change":"2023-10-16T14:11:37.774757+0000","last_active":"2023-10-16T14:11:38.854622+0000","last_peered":"2023-10-16T14:11:38.854622+0000","last_clean":"2023-10-16T14:11:38.854622+0000","last_became_active":"2023-10-16T14:11:37.774442+0000","last_became_peered":"2023-10-16T14:11:37.774442+0000","last_unstale":"2023-10-16T14:11:38.854622+0000","last_undegraded":"2023-10-16T14:11:38.854622+0000","last_fullsized":"2023-10-16T14:11:38.854622+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:45.874083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810591+0000","last_change":"2023-10-16T14:11:38.809976+0000","last_active":"2023-10-16T14:11:38.810591+0000","last_peered":"2023-10-16T14:11:38.810591+0000","last_clean":"2023-10-16T14:11:38.810591+0000","last_became_active":"2023-10-16T14:11:38.809754+0000","last_became_peered":"2023-10-16T14:11:38.809754+0000","last_unstale":"2023-10-16T14:11:38.810591+0000","last_undegraded":"2023-10-16T14:11:38.810591+0000","last_fullsized":"2023-10-16T14:11:38.810591+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:58.363248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.855171+0000","last_change":"2023-10-16T14:11:37.776847+0000","last_active":"2023-10-16T14:11:38.855171+0000","last_peered":"2023-10-16T14:11:38.855171+0000","last_clean":"2023-10-16T14:11:38.855171+0000","last_became_active":"2023-10-16T14:11:37.775028+0000","last_became_peered":"2023-10-16T14:11:37.775028+0000","last_unstale":"2023-10-16T14:11:38.855171+0000","last_undegraded":"2023-10-16T14:11:38.855171+0000","last_fullsized":"2023-10-16T14:11:38.855171+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:54.089883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782646+0000","last_change":"2023-10-16T14:11:38.801326+0000","last_active":"2023-10-16T14:11:39.782646+0000","last_peered":"2023-10-16T14:11:39.782646+0000","last_clean":"2023-10-16T14:11:39.782646+0000","last_became_active":"2023-10-16T14:11:38.800581+0000","last_became_peered":"2023-10-16T14:11:38.800581+0000","last_unstale":"2023-10-16T14:11:39.782646+0000","last_undegraded":"2023-10-16T14:11:39.782646+0000","last_fullsized":"2023-10-16T14:11:39.782646+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:09.367409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.785623+0000","last_change":"2023-10-16T14:11:37.775158+0000","last_active":"2023-10-16T14:11:38.785623+0000","last_peered":"2023-10-16T14:11:38.785623+0000","last_clean":"2023-10-16T14:11:38.785623+0000","last_became_active":"2023-10-16T14:11:37.773264+0000","last_became_peered":"2023-10-16T14:11:37.773264+0000","last_unstale":"2023-10-16T14:11:38.785623+0000","last_undegraded":"2023-10-16T14:11:38.785623+0000","last_fullsized":"2023-10-16T14:11:38.785623+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:27.581413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807413+0000","last_change":"2023-10-16T14:11:38.802528+0000","last_active":"2023-10-16T14:11:38.807413+0000","last_peered":"2023-10-16T14:11:38.807413+0000","last_clean":"2023-10-16T14:11:38.807413+0000","last_became_active":"2023-10-16T14:11:38.802212+0000","last_became_peered":"2023-10-16T14:11:38.802212+0000","last_unstale":"2023-10-16T14:11:38.807413+0000","last_undegraded":"2023-10-16T14:11:38.807413+0000","last_fullsized":"2023-10-16T14:11:38.807413+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:31.968414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782705+0000","last_change":"2023-10-16T14:11:37.780327+0000","last_active":"2023-10-16T14:11:39.782705+0000","last_peered":"2023-10-16T14:11:39.782705+0000","last_clean":"2023-10-16T14:11:39.782705+0000","last_became_active":"2023-10-16T14:11:37.779846+0000","last_became_peered":"2023-10-16T14:11:37.779846+0000","last_unstale":"2023-10-16T14:11:39.782705+0000","last_undegraded":"2023-10-16T14:11:39.782705+0000","last_fullsized":"2023-10-16T14:11:39.782705+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:15.446021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.812563+0000","last_change":"2023-10-16T14:11:38.809523+0000","last_active":"2023-10-16T14:11:38.812563+0000","last_peered":"2023-10-16T14:11:38.812563+0000","last_clean":"2023-10-16T14:11:38.812563+0000","last_became_active":"2023-10-16T14:11:38.807663+0000","last_became_peered":"2023-10-16T14:11:38.807663+0000","last_unstale":"2023-10-16T14:11:38.812563+0000","last_undegraded":"2023-10-16T14:11:38.812563+0000","last_fullsized":"2023-10-16T14:11:38.812563+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:32.228190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790522+0000","last_change":"2023-10-16T14:11:37.784204+0000","last_active":"2023-10-16T14:11:39.790522+0000","last_peered":"2023-10-16T14:11:39.790522+0000","last_clean":"2023-10-16T14:11:39.790522+0000","last_became_active":"2023-10-16T14:11:37.783880+0000","last_became_peered":"2023-10-16T14:11:37.783880+0000","last_unstale":"2023-10-16T14:11:39.790522+0000","last_undegraded":"2023-10-16T14:11:39.790522+0000","last_fullsized":"2023-10-16T14:11:39.790522+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:22.672658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808564+0000","last_change":"2023-10-16T14:11:38.802292+0000","last_active":"2023-10-16T14:11:38.808564+0000","last_peered":"2023-10-16T14:11:38.808564+0000","last_clean":"2023-10-16T14:11:38.808564+0000","last_became_active":"2023-10-16T14:11:38.801904+0000","last_became_peered":"2023-10-16T14:11:38.801904+0000","last_unstale":"2023-10-16T14:11:38.808564+0000","last_undegraded":"2023-10-16T14:11:38.808564+0000","last_fullsized":"2023-10-16T14:11:38.808564+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:17.405096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783757+0000","last_change":"2023-10-16T14:11:37.774897+0000","last_active":"2023-10-16T14:11:39.783757+0000","last_peered":"2023-10-16T14:11:39.783757+0000","last_clean":"2023-10-16T14:11:39.783757+0000","last_became_active":"2023-10-16T14:11:37.774465+0000","last_became_peered":"2023-10-16T14:11:37.774465+0000","last_unstale":"2023-10-16T14:11:39.783757+0000","last_undegraded":"2023-10-16T14:11:39.783757+0000","last_fullsized":"2023-10-16T14:11:39.783757+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783479+0000","last_change":"2023-10-16T14:11:38.809093+0000","last_active":"2023-10-16T14:11:39.783479+0000","last_peered":"2023-10-16T14:11:39.783479+0000","last_clean":"2023-10-16T14:11:39.783479+0000","last_became_active":"2023-10-16T14:11:38.808829+0000","last_became_peered":"2023-10-16T14:11:38.808829+0000","last_unstale":"2023-10-16T14:11:39.783479+0000","last_undegraded":"2023-10-16T14:11:39.783479+0000","last_fullsized":"2023-10-16T14:11:39.783479+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:09.522737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"251'3","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.857819+0000","last_change":"2023-10-16T14:11:37.772650+0000","last_active":"2023-10-16T14:11:38.857819+0000","last_peered":"2023-10-16T14:11:38.857819+0000","last_clean":"2023-10-16T14:11:38.857819+0000","last_became_active":"2023-10-16T14:11:37.772209+0000","last_became_peered":"2023-10-16T14:11:37.772209+0000","last_unstale":"2023-10-16T14:11:38.857819+0000","last_undegraded":"2023-10-16T14:11:38.857819+0000","last_fullsized":"2023-10-16T14:11:38.857819+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:51.981249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804292+0000","last_change":"2023-10-16T14:11:38.800005+0000","last_active":"2023-10-16T14:11:38.804292+0000","last_peered":"2023-10-16T14:11:38.804292+0000","last_clean":"2023-10-16T14:11:38.804292+0000","last_became_active":"2023-10-16T14:11:38.799654+0000","last_became_peered":"2023-10-16T14:11:38.799654+0000","last_unstale":"2023-10-16T14:11:38.804292+0000","last_undegraded":"2023-10-16T14:11:38.804292+0000","last_fullsized":"2023-10-16T14:11:38.804292+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:23.497949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"251'6","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.860833+0000","last_change":"2023-10-16T14:11:37.783855+0000","last_active":"2023-10-16T14:11:38.860833+0000","last_peered":"2023-10-16T14:11:38.860833+0000","last_clean":"2023-10-16T14:11:38.860833+0000","last_became_active":"2023-10-16T14:11:37.783474+0000","last_became_peered":"2023-10-16T14:11:37.783474+0000","last_unstale":"2023-10-16T14:11:38.860833+0000","last_undegraded":"2023-10-16T14:11:38.860833+0000","last_fullsized":"2023-10-16T14:11:38.860833+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.800247+0000","last_change":"2023-10-16T14:11:38.796891+0000","last_active":"2023-10-16T14:11:38.800247+0000","last_peered":"2023-10-16T14:11:38.800247+0000","last_clean":"2023-10-16T14:11:38.800247+0000","last_became_active":"2023-10-16T14:11:38.796574+0000","last_became_peered":"2023-10-16T14:11:38.796574+0000","last_unstale":"2023-10-16T14:11:38.800247+0000","last_undegraded":"2023-10-16T14:11:38.800247+0000","last_fullsized":"2023-10-16T14:11:38.800247+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:46.154550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"251'2","reported_seq":13,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.857824+0000","last_change":"2023-10-16T14:11:37.767366+0000","last_active":"2023-10-16T14:11:38.857824+0000","last_peered":"2023-10-16T14:11:38.857824+0000","last_clean":"2023-10-16T14:11:38.857824+0000","last_became_active":"2023-10-16T14:11:37.766979+0000","last_became_peered":"2023-10-16T14:11:37.766979+0000","last_unstale":"2023-10-16T14:11:38.857824+0000","last_undegraded":"2023-10-16T14:11:38.857824+0000","last_fullsized":"2023-10-16T14:11:38.857824+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:25.104652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.803947+0000","last_change":"2023-10-16T14:11:38.801883+0000","last_active":"2023-10-16T14:11:38.803947+0000","last_peered":"2023-10-16T14:11:38.803947+0000","last_clean":"2023-10-16T14:11:38.803947+0000","last_became_active":"2023-10-16T14:11:38.800123+0000","last_became_peered":"2023-10-16T14:11:38.800123+0000","last_unstale":"2023-10-16T14:11:38.803947+0000","last_undegraded":"2023-10-16T14:11:38.803947+0000","last_fullsized":"2023-10-16T14:11:38.803947+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:34.408581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782904+0000","last_change":"2023-10-16T14:11:37.779309+0000","last_active":"2023-10-16T14:11:39.782904+0000","last_peered":"2023-10-16T14:11:39.782904+0000","last_clean":"2023-10-16T14:11:39.782904+0000","last_became_active":"2023-10-16T14:11:37.778681+0000","last_became_peered":"2023-10-16T14:11:37.778681+0000","last_unstale":"2023-10-16T14:11:39.782904+0000","last_undegraded":"2023-10-16T14:11:39.782904+0000","last_fullsized":"2023-10-16T14:11:39.782904+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:41.372800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.802323+0000","last_change":"2023-10-16T14:11:38.799773+0000","last_active":"2023-10-16T14:11:38.802323+0000","last_peered":"2023-10-16T14:11:38.802323+0000","last_clean":"2023-10-16T14:11:38.802323+0000","last_became_active":"2023-10-16T14:11:38.799446+0000","last_became_peered":"2023-10-16T14:11:38.799446+0000","last_unstale":"2023-10-16T14:11:38.802323+0000","last_undegraded":"2023-10-16T14:11:38.802323+0000","last_fullsized":"2023-10-16T14:11:38.802323+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:06.091417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.785773+0000","last_change":"2023-10-16T14:11:37.777163+0000","last_active":"2023-10-16T14:11:38.785773+0000","last_peered":"2023-10-16T14:11:38.785773+0000","last_clean":"2023-10-16T14:11:38.785773+0000","last_became_active":"2023-10-16T14:11:37.776875+0000","last_became_peered":"2023-10-16T14:11:37.776875+0000","last_unstale":"2023-10-16T14:11:38.785773+0000","last_undegraded":"2023-10-16T14:11:38.785773+0000","last_fullsized":"2023-10-16T14:11:38.785773+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:59.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804361+0000","last_change":"2023-10-16T14:11:38.801468+0000","last_active":"2023-10-16T14:11:38.804361+0000","last_peered":"2023-10-16T14:11:38.804361+0000","last_clean":"2023-10-16T14:11:38.804361+0000","last_became_active":"2023-10-16T14:11:38.801182+0000","last_became_peered":"2023-10-16T14:11:38.801182+0000","last_unstale":"2023-10-16T14:11:38.804361+0000","last_undegraded":"2023-10-16T14:11:38.804361+0000","last_fullsized":"2023-10-16T14:11:38.804361+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:01.648322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"251'8","reported_seq":19,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.859720+0000","last_change":"2023-10-16T14:11:37.784533+0000","last_active":"2023-10-16T14:11:38.859720+0000","last_peered":"2023-10-16T14:11:38.859720+0000","last_clean":"2023-10-16T14:11:38.859720+0000","last_became_active":"2023-10-16T14:11:37.784209+0000","last_became_peered":"2023-10-16T14:11:37.784209+0000","last_unstale":"2023-10-16T14:11:38.859720+0000","last_undegraded":"2023-10-16T14:11:38.859720+0000","last_fullsized":"2023-10-16T14:11:38.859720+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:49.027722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813284+0000","last_change":"2023-10-16T14:11:38.810290+0000","last_active":"2023-10-16T14:11:38.813284+0000","last_peered":"2023-10-16T14:11:38.813284+0000","last_clean":"2023-10-16T14:11:38.813284+0000","last_became_active":"2023-10-16T14:11:38.809830+0000","last_became_peered":"2023-10-16T14:11:38.809830+0000","last_unstale":"2023-10-16T14:11:38.813284+0000","last_undegraded":"2023-10-16T14:11:38.813284+0000","last_fullsized":"2023-10-16T14:11:38.813284+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:18.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.852464+0000","last_change":"2023-10-16T14:11:37.786947+0000","last_active":"2023-10-16T14:11:38.852464+0000","last_peered":"2023-10-16T14:11:38.852464+0000","last_clean":"2023-10-16T14:11:38.852464+0000","last_became_active":"2023-10-16T14:11:37.786340+0000","last_became_peered":"2023-10-16T14:11:37.786340+0000","last_unstale":"2023-10-16T14:11:38.852464+0000","last_undegraded":"2023-10-16T14:11:38.852464+0000","last_fullsized":"2023-10-16T14:11:38.852464+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:01.576946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783898+0000","last_change":"2023-10-16T14:11:38.804059+0000","last_active":"2023-10-16T14:11:39.783898+0000","last_peered":"2023-10-16T14:11:39.783898+0000","last_clean":"2023-10-16T14:11:39.783898+0000","last_became_active":"2023-10-16T14:11:38.803768+0000","last_became_peered":"2023-10-16T14:11:38.803768+0000","last_unstale":"2023-10-16T14:11:39.783898+0000","last_undegraded":"2023-10-16T14:11:39.783898+0000","last_fullsized":"2023-10-16T14:11:39.783898+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:40.313169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783764+0000","last_change":"2023-10-16T14:11:37.780012+0000","last_active":"2023-10-16T14:11:39.783764+0000","last_peered":"2023-10-16T14:11:39.783764+0000","last_clean":"2023-10-16T14:11:39.783764+0000","last_became_active":"2023-10-16T14:11:37.779730+0000","last_became_peered":"2023-10-16T14:11:37.779730+0000","last_unstale":"2023-10-16T14:11:39.783764+0000","last_undegraded":"2023-10-16T14:11:39.783764+0000","last_fullsized":"2023-10-16T14:11:39.783764+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.805206+0000","last_change":"2023-10-16T14:11:38.804003+0000","last_active":"2023-10-16T14:11:38.805206+0000","last_peered":"2023-10-16T14:11:38.805206+0000","last_clean":"2023-10-16T14:11:38.805206+0000","last_became_active":"2023-10-16T14:11:38.803711+0000","last_became_peered":"2023-10-16T14:11:38.803711+0000","last_unstale":"2023-10-16T14:11:38.805206+0000","last_undegraded":"2023-10-16T14:11:38.805206+0000","last_fullsized":"2023-10-16T14:11:38.805206+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:42.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.852774+0000","last_change":"2023-10-16T14:11:37.784775+0000","last_active":"2023-10-16T14:11:38.852774+0000","last_peered":"2023-10-16T14:11:38.852774+0000","last_clean":"2023-10-16T14:11:38.852774+0000","last_became_active":"2023-10-16T14:11:37.784340+0000","last_became_peered":"2023-10-16T14:11:37.784340+0000","last_unstale":"2023-10-16T14:11:38.852774+0000","last_undegraded":"2023-10-16T14:11:38.852774+0000","last_fullsized":"2023-10-16T14:11:38.852774+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791158+0000","last_change":"2023-10-16T14:11:38.797986+0000","last_active":"2023-10-16T14:11:39.791158+0000","last_peered":"2023-10-16T14:11:39.791158+0000","last_clean":"2023-10-16T14:11:39.791158+0000","last_became_active":"2023-10-16T14:11:38.797632+0000","last_became_peered":"2023-10-16T14:11:38.797632+0000","last_unstale":"2023-10-16T14:11:39.791158+0000","last_undegraded":"2023-10-16T14:11:39.791158+0000","last_fullsized":"2023-10-16T14:11:39.791158+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:33.536538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.772569+0000","last_change":"2023-10-16T14:11:37.772519+0000","last_active":"2023-10-16T14:11:38.772569+0000","last_peered":"2023-10-16T14:11:38.772569+0000","last_clean":"2023-10-16T14:11:38.772569+0000","last_became_active":"2023-10-16T14:11:37.772241+0000","last_became_peered":"2023-10-16T14:11:37.772241+0000","last_unstale":"2023-10-16T14:11:38.772569+0000","last_undegraded":"2023-10-16T14:11:38.772569+0000","last_fullsized":"2023-10-16T14:11:38.772569+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:47.596657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789351+0000","last_change":"2023-10-16T14:11:38.801767+0000","last_active":"2023-10-16T14:11:39.789351+0000","last_peered":"2023-10-16T14:11:39.789351+0000","last_clean":"2023-10-16T14:11:39.789351+0000","last_became_active":"2023-10-16T14:11:38.800797+0000","last_became_peered":"2023-10-16T14:11:38.800797+0000","last_unstale":"2023-10-16T14:11:39.789351+0000","last_undegraded":"2023-10-16T14:11:39.789351+0000","last_fullsized":"2023-10-16T14:11:39.789351+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:05.512554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783306+0000","last_change":"2023-10-16T14:11:37.781065+0000","last_active":"2023-10-16T14:11:39.783306+0000","last_peered":"2023-10-16T14:11:39.783306+0000","last_clean":"2023-10-16T14:11:39.783306+0000","last_became_active":"2023-10-16T14:11:37.780556+0000","last_became_peered":"2023-10-16T14:11:37.780556+0000","last_unstale":"2023-10-16T14:11:39.783306+0000","last_undegraded":"2023-10-16T14:11:39.783306+0000","last_fullsized":"2023-10-16T14:11:39.783306+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:03.865559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807195+0000","last_change":"2023-10-16T14:11:38.803116+0000","last_active":"2023-10-16T14:11:38.807195+0000","last_peered":"2023-10-16T14:11:38.807195+0000","last_clean":"2023-10-16T14:11:38.807195+0000","last_became_active":"2023-10-16T14:11:38.802518+0000","last_became_peered":"2023-10-16T14:11:38.802518+0000","last_unstale":"2023-10-16T14:11:38.807195+0000","last_undegraded":"2023-10-16T14:11:38.807195+0000","last_fullsized":"2023-10-16T14:11:38.807195+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:08.898698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.770638+0000","last_change":"2023-10-16T14:11:37.768392+0000","last_active":"2023-10-16T14:11:38.770638+0000","last_peered":"2023-10-16T14:11:38.770638+0000","last_clean":"2023-10-16T14:11:38.770638+0000","last_became_active":"2023-10-16T14:11:37.768081+0000","last_became_peered":"2023-10-16T14:11:37.768081+0000","last_unstale":"2023-10-16T14:11:38.770638+0000","last_undegraded":"2023-10-16T14:11:38.770638+0000","last_fullsized":"2023-10-16T14:11:38.770638+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:13.231020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.806651+0000","last_change":"2023-10-16T14:11:38.796924+0000","last_active":"2023-10-16T14:11:38.806651+0000","last_peered":"2023-10-16T14:11:38.806651+0000","last_clean":"2023-10-16T14:11:38.806651+0000","last_became_active":"2023-10-16T14:11:38.796610+0000","last_became_peered":"2023-10-16T14:11:38.796610+0000","last_unstale":"2023-10-16T14:11:38.806651+0000","last_undegraded":"2023-10-16T14:11:38.806651+0000","last_fullsized":"2023-10-16T14:11:38.806651+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:39.361933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.792228+0000","last_change":"2023-10-16T14:11:37.780387+0000","last_active":"2023-10-16T14:11:38.792228+0000","last_peered":"2023-10-16T14:11:38.792228+0000","last_clean":"2023-10-16T14:11:38.792228+0000","last_became_active":"2023-10-16T14:11:37.779362+0000","last_became_peered":"2023-10-16T14:11:37.779362+0000","last_unstale":"2023-10-16T14:11:38.792228+0000","last_undegraded":"2023-10-16T14:11:38.792228+0000","last_fullsized":"2023-10-16T14:11:38.792228+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791065+0000","last_change":"2023-10-16T14:11:38.799901+0000","last_active":"2023-10-16T14:11:39.791065+0000","last_peered":"2023-10-16T14:11:39.791065+0000","last_clean":"2023-10-16T14:11:39.791065+0000","last_became_active":"2023-10-16T14:11:38.799145+0000","last_became_peered":"2023-10-16T14:11:38.799145+0000","last_unstale":"2023-10-16T14:11:39.791065+0000","last_undegraded":"2023-10-16T14:11:39.791065+0000","last_fullsized":"2023-10-16T14:11:39.791065+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.534242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.785145+0000","last_change":"2023-10-16T14:11:37.779700+0000","last_active":"2023-10-16T14:11:38.785145+0000","last_peered":"2023-10-16T14:11:38.785145+0000","last_clean":"2023-10-16T14:11:38.785145+0000","last_became_active":"2023-10-16T14:11:37.779082+0000","last_became_peered":"2023-10-16T14:11:37.779082+0000","last_unstale":"2023-10-16T14:11:38.785145+0000","last_undegraded":"2023-10-16T14:11:38.785145+0000","last_fullsized":"2023-10-16T14:11:38.785145+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:12.639268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804305+0000","last_change":"2023-10-16T14:11:38.800281+0000","last_active":"2023-10-16T14:11:38.804305+0000","last_peered":"2023-10-16T14:11:38.804305+0000","last_clean":"2023-10-16T14:11:38.804305+0000","last_became_active":"2023-10-16T14:11:38.799813+0000","last_became_peered":"2023-10-16T14:11:38.799813+0000","last_unstale":"2023-10-16T14:11:38.804305+0000","last_undegraded":"2023-10-16T14:11:38.804305+0000","last_fullsized":"2023-10-16T14:11:38.804305+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:52.534482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"251'2","reported_seq":13,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.856884+0000","last_change":"2023-10-16T14:11:37.776293+0000","last_active":"2023-10-16T14:11:38.856884+0000","last_peered":"2023-10-16T14:11:38.856884+0000","last_clean":"2023-10-16T14:11:38.856884+0000","last_became_active":"2023-10-16T14:11:37.775902+0000","last_became_peered":"2023-10-16T14:11:37.775902+0000","last_unstale":"2023-10-16T14:11:38.856884+0000","last_undegraded":"2023-10-16T14:11:38.856884+0000","last_fullsized":"2023-10-16T14:11:38.856884+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:09.428253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811636+0000","last_change":"2023-10-16T14:11:38.806901+0000","last_active":"2023-10-16T14:11:38.811636+0000","last_peered":"2023-10-16T14:11:38.811636+0000","last_clean":"2023-10-16T14:11:38.811636+0000","last_became_active":"2023-10-16T14:11:38.806614+0000","last_became_peered":"2023-10-16T14:11:38.806614+0000","last_unstale":"2023-10-16T14:11:38.811636+0000","last_undegraded":"2023-10-16T14:11:38.811636+0000","last_fullsized":"2023-10-16T14:11:38.811636+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:01.915134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790112+0000","last_change":"2023-10-16T14:11:37.780349+0000","last_active":"2023-10-16T14:11:39.790112+0000","last_peered":"2023-10-16T14:11:39.790112+0000","last_clean":"2023-10-16T14:11:39.790112+0000","last_became_active":"2023-10-16T14:11:37.777254+0000","last_became_peered":"2023-10-16T14:11:37.777254+0000","last_unstale":"2023-10-16T14:11:39.790112+0000","last_undegraded":"2023-10-16T14:11:39.790112+0000","last_fullsized":"2023-10-16T14:11:39.790112+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:22.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811104+0000","last_change":"2023-10-16T14:11:38.805448+0000","last_active":"2023-10-16T14:11:38.811104+0000","last_peered":"2023-10-16T14:11:38.811104+0000","last_clean":"2023-10-16T14:11:38.811104+0000","last_became_active":"2023-10-16T14:11:38.805128+0000","last_became_peered":"2023-10-16T14:11:38.805128+0000","last_unstale":"2023-10-16T14:11:38.811104+0000","last_undegraded":"2023-10-16T14:11:38.811104+0000","last_fullsized":"2023-10-16T14:11:38.811104+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:15.426100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.789362+0000","last_change":"2023-10-16T14:11:37.782304+0000","last_active":"2023-10-16T14:11:38.789362+0000","last_peered":"2023-10-16T14:11:38.789362+0000","last_clean":"2023-10-16T14:11:38.789362+0000","last_became_active":"2023-10-16T14:11:37.781840+0000","last_became_peered":"2023-10-16T14:11:37.781840+0000","last_unstale":"2023-10-16T14:11:38.789362+0000","last_undegraded":"2023-10-16T14:11:38.789362+0000","last_fullsized":"2023-10-16T14:11:38.789362+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.809485+0000","last_change":"2023-10-16T14:11:38.801623+0000","last_active":"2023-10-16T14:11:38.809485+0000","last_peered":"2023-10-16T14:11:38.809485+0000","last_clean":"2023-10-16T14:11:38.809485+0000","last_became_active":"2023-10-16T14:11:38.801313+0000","last_became_peered":"2023-10-16T14:11:38.801313+0000","last_unstale":"2023-10-16T14:11:38.809485+0000","last_undegraded":"2023-10-16T14:11:38.809485+0000","last_fullsized":"2023-10-16T14:11:38.809485+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:57:38.359073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.772893+0000","last_change":"2023-10-16T14:11:37.775930+0000","last_active":"2023-10-16T14:11:38.772893+0000","last_peered":"2023-10-16T14:11:38.772893+0000","last_clean":"2023-10-16T14:11:38.772893+0000","last_became_active":"2023-10-16T14:11:37.775617+0000","last_became_peered":"2023-10-16T14:11:37.775617+0000","last_unstale":"2023-10-16T14:11:38.772893+0000","last_undegraded":"2023-10-16T14:11:38.772893+0000","last_fullsized":"2023-10-16T14:11:38.772893+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:19.733822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.800229+0000","last_change":"2023-10-16T14:11:38.796749+0000","last_active":"2023-10-16T14:11:38.800229+0000","last_peered":"2023-10-16T14:11:38.800229+0000","last_clean":"2023-10-16T14:11:38.800229+0000","last_became_active":"2023-10-16T14:11:38.796468+0000","last_became_peered":"2023-10-16T14:11:38.796468+0000","last_unstale":"2023-10-16T14:11:38.800229+0000","last_undegraded":"2023-10-16T14:11:38.800229+0000","last_fullsized":"2023-10-16T14:11:38.800229+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:18.275683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.789565+0000","last_change":"2023-10-16T14:11:37.785747+0000","last_active":"2023-10-16T14:11:38.789565+0000","last_peered":"2023-10-16T14:11:38.789565+0000","last_clean":"2023-10-16T14:11:38.789565+0000","last_became_active":"2023-10-16T14:11:37.785464+0000","last_became_peered":"2023-10-16T14:11:37.785464+0000","last_unstale":"2023-10-16T14:11:38.789565+0000","last_undegraded":"2023-10-16T14:11:38.789565+0000","last_fullsized":"2023-10-16T14:11:38.789565+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:32.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791444+0000","last_change":"2023-10-16T14:11:38.800958+0000","last_active":"2023-10-16T14:11:39.791444+0000","last_peered":"2023-10-16T14:11:39.791444+0000","last_clean":"2023-10-16T14:11:39.791444+0000","last_became_active":"2023-10-16T14:11:38.800677+0000","last_became_peered":"2023-10-16T14:11:38.800677+0000","last_unstale":"2023-10-16T14:11:39.791444+0000","last_undegraded":"2023-10-16T14:11:39.791444+0000","last_fullsized":"2023-10-16T14:11:39.791444+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:18.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.797327+0000","last_change":"2023-10-16T14:11:37.785497+0000","last_active":"2023-10-16T14:11:38.797327+0000","last_peered":"2023-10-16T14:11:38.797327+0000","last_clean":"2023-10-16T14:11:38.797327+0000","last_became_active":"2023-10-16T14:11:37.785162+0000","last_became_peered":"2023-10-16T14:11:37.785162+0000","last_unstale":"2023-10-16T14:11:38.797327+0000","last_undegraded":"2023-10-16T14:11:38.797327+0000","last_fullsized":"2023-10-16T14:11:38.797327+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:28.524950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.788645+0000","last_change":"2023-10-16T14:11:38.789466+0000","last_active":"2023-10-16T14:11:39.788645+0000","last_peered":"2023-10-16T14:11:39.788645+0000","last_clean":"2023-10-16T14:11:39.788645+0000","last_became_active":"2023-10-16T14:11:38.788914+0000","last_became_peered":"2023-10-16T14:11:38.788914+0000","last_unstale":"2023-10-16T14:11:39.788645+0000","last_undegraded":"2023-10-16T14:11:39.788645+0000","last_fullsized":"2023-10-16T14:11:39.788645+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:36.423758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782705+0000","last_change":"2023-10-16T14:11:37.784108+0000","last_active":"2023-10-16T14:11:39.782705+0000","last_peered":"2023-10-16T14:11:39.782705+0000","last_clean":"2023-10-16T14:11:39.782705+0000","last_became_active":"2023-10-16T14:11:37.783469+0000","last_became_peered":"2023-10-16T14:11:37.783469+0000","last_unstale":"2023-10-16T14:11:39.782705+0000","last_undegraded":"2023-10-16T14:11:39.782705+0000","last_fullsized":"2023-10-16T14:11:39.782705+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.805397+0000","last_change":"2023-10-16T14:11:38.799118+0000","last_active":"2023-10-16T14:11:38.805397+0000","last_peered":"2023-10-16T14:11:38.805397+0000","last_clean":"2023-10-16T14:11:38.805397+0000","last_became_active":"2023-10-16T14:11:38.798708+0000","last_became_peered":"2023-10-16T14:11:38.798708+0000","last_unstale":"2023-10-16T14:11:38.805397+0000","last_undegraded":"2023-10-16T14:11:38.805397+0000","last_fullsized":"2023-10-16T14:11:38.805397+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:39.879989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782779+0000","last_change":"2023-10-16T14:11:37.777958+0000","last_active":"2023-10-16T14:11:39.782779+0000","last_peered":"2023-10-16T14:11:39.782779+0000","last_clean":"2023-10-16T14:11:39.782779+0000","last_became_active":"2023-10-16T14:11:37.777687+0000","last_became_peered":"2023-10-16T14:11:37.777687+0000","last_unstale":"2023-10-16T14:11:39.782779+0000","last_undegraded":"2023-10-16T14:11:39.782779+0000","last_fullsized":"2023-10-16T14:11:39.782779+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:13.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789873+0000","last_change":"2023-10-16T14:11:38.785931+0000","last_active":"2023-10-16T14:11:39.789873+0000","last_peered":"2023-10-16T14:11:39.789873+0000","last_clean":"2023-10-16T14:11:39.789873+0000","last_became_active":"2023-10-16T14:11:38.785345+0000","last_became_peered":"2023-10-16T14:11:38.785345+0000","last_unstale":"2023-10-16T14:11:39.789873+0000","last_undegraded":"2023-10-16T14:11:39.789873+0000","last_fullsized":"2023-10-16T14:11:39.789873+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:16.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"251'4","reported_seq":16,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783948+0000","last_change":"2023-10-16T14:11:37.781168+0000","last_active":"2023-10-16T14:11:39.783948+0000","last_peered":"2023-10-16T14:11:39.783948+0000","last_clean":"2023-10-16T14:11:39.783948+0000","last_became_active":"2023-10-16T14:11:37.780834+0000","last_became_peered":"2023-10-16T14:11:37.780834+0000","last_unstale":"2023-10-16T14:11:39.783948+0000","last_undegraded":"2023-10-16T14:11:39.783948+0000","last_fullsized":"2023-10-16T14:11:39.783948+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:21.622808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"237'373","reported_seq":1318,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T14:11:37.771608+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:11:37.771608+0000","last_peered":"2023-10-16T14:11:37.771608+0000","last_clean":"2023-10-16T14:11:37.771608+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:11:37.771608+0000","last_undegraded":"2023-10-16T14:11:37.771608+0000","last_fullsized":"2023-10-16T14:11:37.771608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543311,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52896,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318944,"statfs":{"total":96636764160,"available":96582598656,"internally_reserved":0,"allocated":3751936,"data_stored":2136002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3460000000000001,"15min":1.3660000000000001},"min":{"1min":0.77900000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.885},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.3480000000000001,"15min":1.3979999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7669999999999999,"5min":1.827,"15min":1.873},"last":1.413}]},{"osd":1,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.2190000000000001,"15min":1.244},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.228,"15min":1.2490000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.72,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.262}]},{"osd":2,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.228,"15min":1.258},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.482,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.224,"5min":1.347,"15min":1.3560000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.6930000000000001,"5min":2.028,"15min":2.028},"last":1.143}]},{"osd":3,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.363,"15min":1.403},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8320000000000001,"5min":1.921,"15min":2.097},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3999999999999999,"15min":1.4259999999999999},"min":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7969999999999999,"5min":1.946,"15min":2.0169999999999999},"last":1.349}]},{"osd":4,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.23,"15min":1.2390000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4390000000000001,"15min":1.4119999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.786,"5min":1.982,"15min":2.0009999999999999},"last":1.2150000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.391,"15min":1.3799999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.0590000000000002},"last":1.665},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.536},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.9550000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.3149999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.6259999999999999,"15min":1.633},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.0009999999999999,"5min":2.052,"15min":2.3250000000000002},"last":1.381},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.5249999999999999,"15min":1.48},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.8939999999999999,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.4650000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543312,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":3162112,"data_stored":1546168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2689999999999999,"15min":1.329},"min":{"1min":0.70299999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.788,"5min":1.887,"15min":1.887},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.3140000000000001,"15min":1.353},"min":{"1min":0.747,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8460000000000001,"5min":1.847,"15min":1.9390000000000001},"last":1.1679999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.206,"15min":1.2250000000000001},"min":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6279999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.21,"15min":1.2270000000000001},"min":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6799999999999999,"5min":1.748,"15min":1.77},"last":0.92600000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.24,"15min":1.2549999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.78,"5min":1.823,"15min":1.823},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.282,"15min":1.3169999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.8759999999999999,"5min":1.962,"15min":1.962},"last":1.0049999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.327,"15min":1.345},"min":{"1min":0.93400000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":1.075,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.776,"5min":1.911,"15min":1.911},"last":1.4430000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.393,"15min":1.409},"min":{"1min":1.1259999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5609999999999999,"15min":1.5509999999999999},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.3740000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.79800000000000004,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4990000000000001,"15min":1.5},"min":{"1min":1.2170000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.0990000000000002,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.413}]},{"osd":7,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.512,"15min":1.5049999999999999},"min":{"1min":0.999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0150000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.546,"15min":1.542},"min":{"1min":1.0269999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.157,"5min":2.157,"15min":2.6120000000000001},"last":1.1000000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543313,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30120,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341720,"statfs":{"total":96636764160,"available":96605921280,"internally_reserved":0,"allocated":3166208,"data_stored":1551864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1299999999999999,"15min":1.202},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.4750000000000001,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2150000000000001,"15min":1.2310000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.85299999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.282,"15min":1.298},"min":{"1min":0.77500000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.774,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1830000000000001,"15min":1.254},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.698,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.2}]},{"osd":2,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.304,"15min":1.335},"min":{"1min":0.90300000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.587,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.266,"15min":1.3120000000000001},"min":{"1min":0.80200000000000005,"5min":0.746,"15min":0.746},"max":{"1min":1.806,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.1240000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3859999999999999,"15min":1.407},"min":{"1min":0.89300000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9019999999999999,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.409,"15min":1.431},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8759999999999999,"5min":2.75,"15min":2.75},"last":1.4850000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3160000000000001,"15min":1.2989999999999999},"min":{"1min":0.77300000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.9299999999999999,"5min":2.774,"15min":2.774},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.454,"15min":1.4319999999999999},"min":{"1min":0.93700000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.661}]},{"osd":6,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4390000000000001,"15min":1.4319999999999999},"min":{"1min":1.1699999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9390000000000001,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.508,"15min":1.4910000000000001},"min":{"1min":1.048,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9590000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.6160000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.593,"15min":1.554},"min":{"1min":1.2170000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1680000000000001,"5min":2.177,"15min":2.4609999999999999},"last":1.532},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6200000000000001,"15min":1.603},"min":{"1min":1.113,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1000000000000001,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.5720000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543313,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66152,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305688,"statfs":{"total":96636764160,"available":96569024512,"internally_reserved":0,"allocated":3174400,"data_stored":1547268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.145,"15min":1.2050000000000001},"min":{"1min":0.61799999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1799999999999999,"15min":1.1990000000000001},"min":{"1min":0.748,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":0.89300000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1990000000000001,"15min":1.2490000000000001},"min":{"1min":0.80700000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6899999999999999,"5min":1.71,"15min":1.78},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.244,"5min":1.2450000000000001,"15min":1.2869999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7250000000000001,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.1220000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2829999999999999,"15min":1.3160000000000001},"min":{"1min":0.73899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.591,"5min":1.714,"15min":1.865},"last":1.357},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3500000000000001,"15min":1.3799999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":1.905,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.244}]},{"osd":3,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2889999999999999,"15min":1.302},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.47},{"interface":"front","average":{"1min":1.359,"5min":1.45,"15min":1.4279999999999999},"min":{"1min":1.05,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.6439999999999999,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.5269999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.3520000000000001,"15min":1.339},"min":{"1min":1.0980000000000001,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":1.994,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.387,"15min":1.3420000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.6419999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.409,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.90400000000000003,"15min":0.872},"max":{"1min":1.931,"5min":1.931,"15min":1.956},"last":1.8859999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.516,"15min":1.518},"min":{"1min":1.0840000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.0630000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.5880000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.5660000000000001,"15min":1.5089999999999999},"min":{"1min":1.0149999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1949999999999998},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6479999999999999,"15min":1.6339999999999999},"min":{"1min":1.3620000000000001,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":2.085,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.7829999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56476,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315364,"statfs":{"total":96636764160,"available":96578932736,"internally_reserved":0,"allocated":3158016,"data_stored":1546146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91900000000000004,"15min":0.94999999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.232,"5min":1.52,"15min":1.5580000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.012,"5min":1.177,"15min":1.2230000000000001},"min":{"1min":0.44900000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.5569999999999999,"5min":1.798,"15min":2.0579999999999998},"last":1.4350000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.05,"15min":1.079},"min":{"1min":0.60199999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.496,"5min":1.855,"15min":1.9750000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.004,"5min":1.1080000000000001,"15min":1.1240000000000001},"min":{"1min":0.56299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.456,"5min":2.077,"15min":2.077},"last":1.0109999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.135,"15min":1.2030000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7330000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.3720000000000001,"15min":1.444},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6659999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.381}]},{"osd":4,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.494,"15min":1.508},"min":{"1min":0.81899999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":4.1299999999999999,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.728},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.446,"15min":1.4590000000000001},"min":{"1min":0.84299999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.5169999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.462,"15min":1.492},"min":{"1min":0.88200000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.554,"15min":1.5589999999999999},"min":{"1min":0.89300000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.3420000000000001},"last":1.5580000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.5309999999999999,"15min":1.528},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.1800000000000002,"5min":2.254,"15min":2.4329999999999998},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4850000000000001,"15min":1.4970000000000001},"min":{"1min":0.77300000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8200000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.3089999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.6040000000000001,"15min":1.609},"min":{"1min":0.95899999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.661,"15min":1.677},"min":{"1min":1.002,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.569,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.591}]}]},{"osd":2,"up_from":14,"seq":60129543311,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56472,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315368,"statfs":{"total":96636764160,"available":96578936832,"internally_reserved":0,"allocated":3158016,"data_stored":1545654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.93100000000000005,"15min":0.95299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.5029999999999999,"5min":1.597,"15min":2.0089999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.137},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.431,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.234}]},{"osd":1,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.131,"15min":1.171},"min":{"1min":0.56399999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.72,"5min":1.847,"15min":1.847},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.2010000000000001,"15min":1.242},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5860000000000001,"5min":1.768,"15min":1.9530000000000001},"last":1.3899999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.107,"15min":1.115},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.3009999999999999,"15min":1.3700000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.0619999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.52}]},{"osd":4,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4710000000000001,"15min":1.5},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.837},{"interface":"front","average":{"1min":1.349,"5min":1.464,"15min":1.4870000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.3140000000000001},"last":1.4870000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.522,"15min":1.5529999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.867,"5min":2.1819999999999999,"15min":2.488},"last":1.722},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4550000000000001,"15min":1.4710000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.0059999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.873}]},{"osd":6,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.524,"15min":1.52},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.4079999999999999},"last":2},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.5640000000000001,"15min":1.617},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.5569999999999999},"last":1.9179999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6319999999999999,"15min":1.655},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.7890000000000001},"last":2.1960000000000002},{"interface":"front","average":{"1min":1.496,"5min":1.619,"15min":1.641},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.2599999999999998,"5min":2.2799999999999998,"15min":2.3319999999999999},"last":2.161}]}]},{"osd":1,"up_from":14,"seq":60129543311,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52336,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319504,"statfs":{"total":96636764160,"available":96583172096,"internally_reserved":0,"allocated":3174400,"data_stored":1547300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.93000000000000005,"15min":0.93999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.448,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.109,"15min":1.133},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0129999999999999,"15min":1.0389999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.204,"15min":1.2549999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.92000000000000004}]},{"osd":3,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.4369999999999998},"last":1.038},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.2789999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.544,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.075}]},{"osd":4,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.258,"15min":1.339},"min":{"1min":0.65700000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.534,"5min":1.776,"15min":2.1259999999999999},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.488,"15min":1.5309999999999999},"min":{"1min":0.79100000000000004,"5min":0.748,"15min":0.748},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.2030000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.446,"15min":1.488},"min":{"1min":0.72899999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.0819999999999999,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.659},{"interface":"front","average":{"1min":1.419,"5min":1.385,"15min":1.4790000000000001},"min":{"1min":0.746,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.833}]},{"osd":6,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5800000000000001,"15min":1.633},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.7200000000000002},"last":1.393},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5409999999999999,"15min":1.603},"min":{"1min":0.85099999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0979999999999999,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.5820000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.639,"15min":1.696},"min":{"1min":0.95299999999999996,"5min":0.628,"15min":0.628},"max":{"1min":2.1280000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.524},{"interface":"front","average":{"1min":1.421,"5min":1.52,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0630000000000002,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.9430000000000001}]}]},{"osd":0,"up_from":14,"seq":60129543312,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48736,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323104,"statfs":{"total":96636764160,"available":96586858496,"internally_reserved":0,"allocated":3751936,"data_stored":2135966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.94299999999999995,"15min":0.94399999999999995},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4299999999999999,"5min":1.704,"15min":1.891},"last":1.097},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.054,"15min":1.145},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3009999999999999,"5min":1.659,"15min":1.9299999999999999},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1299999999999999,"15min":1.169},"min":{"1min":0.73299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":1.379},{"interface":"front","average":{"1min":1.157,"5min":1.171,"15min":1.2210000000000001},"min":{"1min":0.68799999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.52,"5min":1.893,"15min":1.893},"last":1.258}]},{"osd":3,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":1.147},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.623,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.032},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.29,"15min":1.3660000000000001},"min":{"1min":0.63200000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.681,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.492}]},{"osd":4,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.302,"15min":1.3640000000000001},"min":{"1min":0.84999999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.625,"5min":1.819,"15min":2.0070000000000001},"last":1.157},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4970000000000001,"15min":1.5189999999999999},"min":{"1min":0.92400000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.919,"5min":2.601,"15min":2.601},"last":1.653}]},{"osd":5,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4910000000000001,"15min":1.504},"min":{"1min":1.129,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.089,"5min":2.2250000000000001,"15min":2.3279999999999998},"last":1.532},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4259999999999999,"15min":1.4790000000000001},"min":{"1min":0.88700000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.319}]},{"osd":6,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.577,"15min":1.613},"min":{"1min":1.1890000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0019999999999998,"5min":2.669,"15min":2.669},"last":1.843},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.552,"15min":1.5920000000000001},"min":{"1min":0.94999999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.327,"5min":2.355,"15min":2.3769999999999998},"last":1.7729999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.6399999999999999,"15min":1.7010000000000001},"min":{"1min":0.88700000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2170000000000001,"5min":2.484,"15min":2.5270000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.591,"15min":1.6359999999999999},"min":{"1min":1.056,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.9490000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.5840000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:11:43.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:11:43.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.619+0000 7fd5aaf27640 1 Processor -- start 2023-10-16T14:11:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.619+0000 7fd5aaf27640 1 -- start start 2023-10-16T14:11:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.621+0000 7fd5aaf27640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd5a415cfc0 0x7fd5a40b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.621+0000 7fd5aaf27640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd5a40d2640 0x7fd5a40b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.621+0000 7fd5aaf27640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a40d15a0 con 0x7fd5a415cbe0 2023-10-16T14:11:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.621+0000 7fd5aaf27640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a40d1880 con 0x7fd5a40d2640 2023-10-16T14:11:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.621+0000 7fd5aaf27640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a40d1b60 con 0x7fd5a415cfc0 2023-10-16T14:11:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd5a40d2640 0x7fd5a40b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd5a40d2640 0x7fd5a40b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56386/0 (socket says 172.21.15.60:56386) 2023-10-16T14:11:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 -- 172.21.15.60:0/3819760470 learned_addr learned my addr 172.21.15.60:0/3819760470 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5a9f25640 1 --1- 172.21.15.60:0/3819760470 >> v1:172.21.15.60:6789/0 conn(0x7fd5a415cbe0 0x7fd5a40d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54964/0 (socket says 172.21.15.60:54964) 2023-10-16T14:11:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5a9724640 1 --2- 172.21.15.60:0/3819760470 >> v2:172.21.15.144:3300/0 conn(0x7fd5a415cfc0 0x7fd5a40b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 -- 172.21.15.60:0/3819760470 >> v2:172.21.15.144:3300/0 conn(0x7fd5a415cfc0 msgr2=0x7fd5a40b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 --2- 172.21.15.60:0/3819760470 >> v2:172.21.15.144:3300/0 conn(0x7fd5a415cfc0 0x7fd5a40b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 -- 172.21.15.60:0/3819760470 >> v1:172.21.15.60:6789/0 conn(0x7fd5a415cbe0 legacy=0x7fd5a40d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.622+0000 7fd5aa726640 1 -- 172.21.15.60:0/3819760470 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40d1e40 con 0x7fd5a40d2640 2023-10-16T14:11:43.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.623+0000 7fd5aa726640 1 --2- 172.21.15.60:0/3819760470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40d2640 0x7fd5a40b4ab0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40cf4b0 tx=0x7fd59c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b8fe035379029f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:43.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.624+0000 7fd5a8f23640 1 -- 172.21.15.60:0/3819760470 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c02cbb0 con 0x7fd5a40d2640 2023-10-16T14:11:43.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.624+0000 7fd5a8f23640 1 -- 172.21.15.60:0/3819760470 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c02cd10 con 0x7fd5a40d2640 2023-10-16T14:11:43.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.624+0000 7fd5a8f23640 1 -- 172.21.15.60:0/3819760470 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c034610 con 0x7fd5a40d2640 2023-10-16T14:11:43.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 -- 172.21.15.60:0/3819760470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40d2640 msgr2=0x7fd5a40b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/3819760470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40d2640 0x7fd5a40b4ab0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40cf4b0 tx=0x7fd59c02c010 comp rx=0 tx=0).stop 2023-10-16T14:11:43.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 -- 172.21.15.60:0/3819760470 shutdown_connections 2023-10-16T14:11:43.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/3819760470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40d2640 0x7fd5a40b4ab0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/3819760470 >> v2:172.21.15.144:3300/0 conn(0x7fd5a415cfc0 0x7fd5a40b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 -- 172.21.15.60:0/3819760470 >> 172.21.15.60:0/3819760470 conn(0x7fd5a40cf570 msgr2=0x7fd5a40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:43.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 -- 172.21.15.60:0/3819760470 shutdown_connections 2023-10-16T14:11:43.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.625+0000 7fd5aaf27640 1 -- 172.21.15.60:0/3819760470 wait complete. 2023-10-16T14:11:43.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.627+0000 7fd5aaf27640 1 Processor -- start 2023-10-16T14:11:43.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.627+0000 7fd5aaf27640 1 -- start start 2023-10-16T14:11:43.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5aaf27640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5a40d2640 0x7fd5a41f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5aaf27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 0x7fd5a41f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5aaf27640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5a415cfc0 0x7fd5a40c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5aaf27640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40c7c70 con 0x7fd5a415cbe0 2023-10-16T14:11:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5aaf27640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a41f2530 con 0x7fd5a415cfc0 2023-10-16T14:11:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5aaf27640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40c7190 con 0x7fd5a40d2640 2023-10-16T14:11:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5a9724640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 0x7fd5a41f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5a9724640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 0x7fd5a41f5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56398/0 (socket says 172.21.15.60:56398) 2023-10-16T14:11:43.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5a9724640 1 -- 172.21.15.60:0/661686367 learned_addr learned my addr 172.21.15.60:0/661686367 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:43.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.628+0000 7fd5a9f25640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5a40d2640 0x7fd5a41f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5aa726640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5a415cfc0 0x7fd5a40c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9724640 1 -- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5a40d2640 msgr2=0x7fd5a41f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9724640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5a40d2640 0x7fd5a41f33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9724640 1 -- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5a415cfc0 msgr2=0x7fd5a40c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9724640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5a415cfc0 0x7fd5a40c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9724640 1 -- 172.21.15.60:0/661686367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c009b30 con 0x7fd5a415cbe0 2023-10-16T14:11:43.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9f25640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5a40d2640 0x7fd5a41f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:43.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5a9724640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 0x7fd5a41f5100 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd59800d5c0 tx=0x7fd59800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:43.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5980125f0 con 0x7fd5a415cbe0 2023-10-16T14:11:43.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598012750 con 0x7fd5a415cbe0 2023-10-16T14:11:43.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598012b00 con 0x7fd5a415cbe0 2023-10-16T14:11:43.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a40c7470 con 0x7fd5a415cbe0 2023-10-16T14:11:43.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.629+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a40c81a0 con 0x7fd5a415cbe0 2023-10-16T14:11:43.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.631+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a40c8360 con 0x7fd5a415cbe0 2023-10-16T14:11:43.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.632+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd598012c60 con 0x7fd5a415cbe0 2023-10-16T14:11:43.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.633+0000 7fd5967fc640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd58806b0b0 0x7fd58806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:43.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.633+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd598089f90 con 0x7fd5a415cbe0 2023-10-16T14:11:43.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.634+0000 7fd5a9f25640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd58806b0b0 0x7fd58806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:43.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.635+0000 7fd5a9f25640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd58806b0b0 0x7fd58806d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0000f60 tx=0x7fd5a0009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:43.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.637+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd598054c40 con 0x7fd5a415cbe0 2023-10-16T14:11:43.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.829+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 --> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5a40d2120 con 0x7fd58806b0b0 2023-10-16T14:11:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.852+0000 7fd5967fc640 1 -- 172.21.15.60:0/661686367 <== mgr.9169 v2:172.21.15.144:6832/4052124783 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287061 (secure 0 0 0) 0x7fd5a0001cc0 con 0x7fd58806b0b0 2023-10-16T14:11:43.861 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:43.863 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:11:43.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd58806b0b0 msgr2=0x7fd58806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd58806b0b0 0x7fd58806d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0000f60 tx=0x7fd5a0009040 comp rx=0 tx=0).stop 2023-10-16T14:11:43.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 msgr2=0x7fd5a41f5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:43.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 0x7fd5a41f5100 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd59800d5c0 tx=0x7fd59800da80 comp rx=0 tx=0).stop 2023-10-16T14:11:43.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 shutdown_connections 2023-10-16T14:11:43.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fd58806b0b0 0x7fd58806d570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5a415cfc0 0x7fd5a40c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a415cbe0 0x7fd5a41f5100 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.857+0000 7fd5aaf27640 1 --2- 172.21.15.60:0/661686367 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5a40d2640 0x7fd5a41f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.858+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 >> 172.21.15.60:0/661686367 conn(0x7fd5a40cf570 msgr2=0x7fd5a40b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.858+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 shutdown_connections 2023-10-16T14:11:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:43.858+0000 7fd5aaf27640 1 -- 172.21.15.60:0/661686367 wait complete. 2023-10-16T14:11:43.880 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T14:11:43.062286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":667,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":427988,"kb_used_data":25876,"kb_used_omap":12,"kb_used_meta":401971,"kb_avail":754546732,"statfs":{"total":773094113280,"available":772655853568,"internally_reserved":0,"allocated":26497024,"data_stored":13556368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411618872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002264"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810787+0000","last_change":"2023-10-16T14:11:38.800766+0000","last_active":"2023-10-16T14:11:38.810787+0000","last_peered":"2023-10-16T14:11:38.810787+0000","last_clean":"2023-10-16T14:11:38.810787+0000","last_became_active":"2023-10-16T14:11:38.799147+0000","last_became_peered":"2023-10-16T14:11:38.799147+0000","last_unstale":"2023-10-16T14:11:38.810787+0000","last_undegraded":"2023-10-16T14:11:38.810787+0000","last_fullsized":"2023-10-16T14:11:38.810787+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:01.252078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813060+0000","last_change":"2023-10-16T14:11:38.811708+0000","last_active":"2023-10-16T14:11:38.813060+0000","last_peered":"2023-10-16T14:11:38.813060+0000","last_clean":"2023-10-16T14:11:38.813060+0000","last_became_active":"2023-10-16T14:11:38.808060+0000","last_became_peered":"2023-10-16T14:11:38.808060+0000","last_unstale":"2023-10-16T14:11:38.813060+0000","last_undegraded":"2023-10-16T14:11:38.813060+0000","last_fullsized":"2023-10-16T14:11:38.813060+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:45.519076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783555+0000","last_change":"2023-10-16T14:11:38.799851+0000","last_active":"2023-10-16T14:11:39.783555+0000","last_peered":"2023-10-16T14:11:39.783555+0000","last_clean":"2023-10-16T14:11:39.783555+0000","last_became_active":"2023-10-16T14:11:38.799520+0000","last_became_peered":"2023-10-16T14:11:38.799520+0000","last_unstale":"2023-10-16T14:11:39.783555+0000","last_undegraded":"2023-10-16T14:11:39.783555+0000","last_fullsized":"2023-10-16T14:11:39.783555+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:59.480982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813598+0000","last_change":"2023-10-16T14:11:38.811457+0000","last_active":"2023-10-16T14:11:38.813598+0000","last_peered":"2023-10-16T14:11:38.813598+0000","last_clean":"2023-10-16T14:11:38.813598+0000","last_became_active":"2023-10-16T14:11:38.811124+0000","last_became_peered":"2023-10-16T14:11:38.811124+0000","last_unstale":"2023-10-16T14:11:38.813598+0000","last_undegraded":"2023-10-16T14:11:38.813598+0000","last_fullsized":"2023-10-16T14:11:38.813598+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:38.977236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791702+0000","last_change":"2023-10-16T14:11:38.792747+0000","last_active":"2023-10-16T14:11:39.791702+0000","last_peered":"2023-10-16T14:11:39.791702+0000","last_clean":"2023-10-16T14:11:39.791702+0000","last_became_active":"2023-10-16T14:11:38.792334+0000","last_became_peered":"2023-10-16T14:11:38.792334+0000","last_unstale":"2023-10-16T14:11:39.791702+0000","last_undegraded":"2023-10-16T14:11:39.791702+0000","last_fullsized":"2023-10-16T14:11:39.791702+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:23.689830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.814593+0000","last_change":"2023-10-16T14:11:38.814105+0000","last_active":"2023-10-16T14:11:38.814593+0000","last_peered":"2023-10-16T14:11:38.814593+0000","last_clean":"2023-10-16T14:11:38.814593+0000","last_became_active":"2023-10-16T14:11:38.813562+0000","last_became_peered":"2023-10-16T14:11:38.813562+0000","last_unstale":"2023-10-16T14:11:38.814593+0000","last_undegraded":"2023-10-16T14:11:38.814593+0000","last_fullsized":"2023-10-16T14:11:38.814593+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:37.458489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808022+0000","last_change":"2023-10-16T14:11:38.807311+0000","last_active":"2023-10-16T14:11:38.808022+0000","last_peered":"2023-10-16T14:11:38.808022+0000","last_clean":"2023-10-16T14:11:38.808022+0000","last_became_active":"2023-10-16T14:11:38.806946+0000","last_became_peered":"2023-10-16T14:11:38.806946+0000","last_unstale":"2023-10-16T14:11:38.808022+0000","last_undegraded":"2023-10-16T14:11:38.808022+0000","last_fullsized":"2023-10-16T14:11:38.808022+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:39.460099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.788640+0000","last_change":"2023-10-16T14:11:38.796621+0000","last_active":"2023-10-16T14:11:39.788640+0000","last_peered":"2023-10-16T14:11:39.788640+0000","last_clean":"2023-10-16T14:11:39.788640+0000","last_became_active":"2023-10-16T14:11:38.795857+0000","last_became_peered":"2023-10-16T14:11:38.795857+0000","last_unstale":"2023-10-16T14:11:39.788640+0000","last_undegraded":"2023-10-16T14:11:39.788640+0000","last_fullsized":"2023-10-16T14:11:39.788640+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:24.189032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808166+0000","last_change":"2023-10-16T14:11:38.807477+0000","last_active":"2023-10-16T14:11:38.808166+0000","last_peered":"2023-10-16T14:11:38.808166+0000","last_clean":"2023-10-16T14:11:38.808166+0000","last_became_active":"2023-10-16T14:11:38.807149+0000","last_became_peered":"2023-10-16T14:11:38.807149+0000","last_unstale":"2023-10-16T14:11:38.808166+0000","last_undegraded":"2023-10-16T14:11:38.808166+0000","last_fullsized":"2023-10-16T14:11:38.808166+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:43.212625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804329+0000","last_change":"2023-10-16T14:11:38.801109+0000","last_active":"2023-10-16T14:11:38.804329+0000","last_peered":"2023-10-16T14:11:38.804329+0000","last_clean":"2023-10-16T14:11:38.804329+0000","last_became_active":"2023-10-16T14:11:38.800831+0000","last_became_peered":"2023-10-16T14:11:38.800831+0000","last_unstale":"2023-10-16T14:11:38.804329+0000","last_undegraded":"2023-10-16T14:11:38.804329+0000","last_fullsized":"2023-10-16T14:11:38.804329+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:03.392070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811147+0000","last_change":"2023-10-16T14:11:38.806641+0000","last_active":"2023-10-16T14:11:38.811147+0000","last_peered":"2023-10-16T14:11:38.811147+0000","last_clean":"2023-10-16T14:11:38.811147+0000","last_became_active":"2023-10-16T14:11:38.806333+0000","last_became_peered":"2023-10-16T14:11:38.806333+0000","last_unstale":"2023-10-16T14:11:38.811147+0000","last_undegraded":"2023-10-16T14:11:38.811147+0000","last_fullsized":"2023-10-16T14:11:38.811147+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:12.068784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789574+0000","last_change":"2023-10-16T14:11:38.803389+0000","last_active":"2023-10-16T14:11:39.789574+0000","last_peered":"2023-10-16T14:11:39.789574+0000","last_clean":"2023-10-16T14:11:39.789574+0000","last_became_active":"2023-10-16T14:11:38.803101+0000","last_became_peered":"2023-10-16T14:11:38.803101+0000","last_unstale":"2023-10-16T14:11:39.789574+0000","last_undegraded":"2023-10-16T14:11:39.789574+0000","last_fullsized":"2023-10-16T14:11:39.789574+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:45.651230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811853+0000","last_change":"2023-10-16T14:11:38.811074+0000","last_active":"2023-10-16T14:11:38.811853+0000","last_peered":"2023-10-16T14:11:38.811853+0000","last_clean":"2023-10-16T14:11:38.811853+0000","last_became_active":"2023-10-16T14:11:38.810772+0000","last_became_peered":"2023-10-16T14:11:38.810772+0000","last_unstale":"2023-10-16T14:11:38.811853+0000","last_undegraded":"2023-10-16T14:11:38.811853+0000","last_fullsized":"2023-10-16T14:11:38.811853+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:23.246968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813033+0000","last_change":"2023-10-16T14:11:38.806714+0000","last_active":"2023-10-16T14:11:38.813033+0000","last_peered":"2023-10-16T14:11:38.813033+0000","last_clean":"2023-10-16T14:11:38.813033+0000","last_became_active":"2023-10-16T14:11:38.805598+0000","last_became_peered":"2023-10-16T14:11:38.805598+0000","last_unstale":"2023-10-16T14:11:38.813033+0000","last_undegraded":"2023-10-16T14:11:38.813033+0000","last_fullsized":"2023-10-16T14:11:38.813033+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:41.125819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.809098+0000","last_change":"2023-10-16T14:11:38.808575+0000","last_active":"2023-10-16T14:11:38.809098+0000","last_peered":"2023-10-16T14:11:38.809098+0000","last_clean":"2023-10-16T14:11:38.809098+0000","last_became_active":"2023-10-16T14:11:38.808273+0000","last_became_peered":"2023-10-16T14:11:38.808273+0000","last_unstale":"2023-10-16T14:11:38.809098+0000","last_undegraded":"2023-10-16T14:11:38.809098+0000","last_fullsized":"2023-10-16T14:11:38.809098+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:04.882730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782613+0000","last_change":"2023-10-16T14:11:38.804694+0000","last_active":"2023-10-16T14:11:39.782613+0000","last_peered":"2023-10-16T14:11:39.782613+0000","last_clean":"2023-10-16T14:11:39.782613+0000","last_became_active":"2023-10-16T14:11:38.804373+0000","last_became_peered":"2023-10-16T14:11:38.804373+0000","last_unstale":"2023-10-16T14:11:39.782613+0000","last_undegraded":"2023-10-16T14:11:39.782613+0000","last_fullsized":"2023-10-16T14:11:39.782613+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:42.092693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790292+0000","last_change":"2023-10-16T14:11:38.799470+0000","last_active":"2023-10-16T14:11:39.790292+0000","last_peered":"2023-10-16T14:11:39.790292+0000","last_clean":"2023-10-16T14:11:39.790292+0000","last_became_active":"2023-10-16T14:11:38.799170+0000","last_became_peered":"2023-10-16T14:11:38.799170+0000","last_unstale":"2023-10-16T14:11:39.790292+0000","last_undegraded":"2023-10-16T14:11:39.790292+0000","last_fullsized":"2023-10-16T14:11:39.790292+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:23.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807159+0000","last_change":"2023-10-16T14:11:38.801518+0000","last_active":"2023-10-16T14:11:38.807159+0000","last_peered":"2023-10-16T14:11:38.807159+0000","last_clean":"2023-10-16T14:11:38.807159+0000","last_became_active":"2023-10-16T14:11:38.801133+0000","last_became_peered":"2023-10-16T14:11:38.801133+0000","last_unstale":"2023-10-16T14:11:38.807159+0000","last_undegraded":"2023-10-16T14:11:38.807159+0000","last_fullsized":"2023-10-16T14:11:38.807159+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:03.231050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790492+0000","last_change":"2023-10-16T14:11:38.792372+0000","last_active":"2023-10-16T14:11:39.790492+0000","last_peered":"2023-10-16T14:11:39.790492+0000","last_clean":"2023-10-16T14:11:39.790492+0000","last_became_active":"2023-10-16T14:11:38.789719+0000","last_became_peered":"2023-10-16T14:11:38.789719+0000","last_unstale":"2023-10-16T14:11:39.790492+0000","last_undegraded":"2023-10-16T14:11:39.790492+0000","last_fullsized":"2023-10-16T14:11:39.790492+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:30:12.869949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783436+0000","last_change":"2023-10-16T14:11:38.802052+0000","last_active":"2023-10-16T14:11:39.783436+0000","last_peered":"2023-10-16T14:11:39.783436+0000","last_clean":"2023-10-16T14:11:39.783436+0000","last_became_active":"2023-10-16T14:11:38.801625+0000","last_became_peered":"2023-10-16T14:11:38.801625+0000","last_unstale":"2023-10-16T14:11:39.783436+0000","last_undegraded":"2023-10-16T14:11:39.783436+0000","last_fullsized":"2023-10-16T14:11:39.783436+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:27.067528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789528+0000","last_change":"2023-10-16T14:11:38.797163+0000","last_active":"2023-10-16T14:11:39.789528+0000","last_peered":"2023-10-16T14:11:39.789528+0000","last_clean":"2023-10-16T14:11:39.789528+0000","last_became_active":"2023-10-16T14:11:38.796761+0000","last_became_peered":"2023-10-16T14:11:38.796761+0000","last_unstale":"2023-10-16T14:11:39.789528+0000","last_undegraded":"2023-10-16T14:11:39.789528+0000","last_fullsized":"2023-10-16T14:11:39.789528+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:01.946364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.805625+0000","last_change":"2023-10-16T14:11:38.802974+0000","last_active":"2023-10-16T14:11:38.805625+0000","last_peered":"2023-10-16T14:11:38.805625+0000","last_clean":"2023-10-16T14:11:38.805625+0000","last_became_active":"2023-10-16T14:11:38.802711+0000","last_became_peered":"2023-10-16T14:11:38.802711+0000","last_unstale":"2023-10-16T14:11:38.805625+0000","last_undegraded":"2023-10-16T14:11:38.805625+0000","last_fullsized":"2023-10-16T14:11:38.805625+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:33.739529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790726+0000","last_change":"2023-10-16T14:11:38.800092+0000","last_active":"2023-10-16T14:11:39.790726+0000","last_peered":"2023-10-16T14:11:39.790726+0000","last_clean":"2023-10-16T14:11:39.790726+0000","last_became_active":"2023-10-16T14:11:38.799625+0000","last_became_peered":"2023-10-16T14:11:38.799625+0000","last_unstale":"2023-10-16T14:11:39.790726+0000","last_undegraded":"2023-10-16T14:11:39.790726+0000","last_fullsized":"2023-10-16T14:11:39.790726+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:06.091417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.812184+0000","last_change":"2023-10-16T14:11:38.809521+0000","last_active":"2023-10-16T14:11:38.812184+0000","last_peered":"2023-10-16T14:11:38.812184+0000","last_clean":"2023-10-16T14:11:38.812184+0000","last_became_active":"2023-10-16T14:11:38.808121+0000","last_became_peered":"2023-10-16T14:11:38.808121+0000","last_unstale":"2023-10-16T14:11:38.812184+0000","last_undegraded":"2023-10-16T14:11:38.812184+0000","last_fullsized":"2023-10-16T14:11:38.812184+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:18.290336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813613+0000","last_change":"2023-10-16T14:11:38.811608+0000","last_active":"2023-10-16T14:11:38.813613+0000","last_peered":"2023-10-16T14:11:38.813613+0000","last_clean":"2023-10-16T14:11:38.813613+0000","last_became_active":"2023-10-16T14:11:38.811328+0000","last_became_peered":"2023-10-16T14:11:38.811328+0000","last_unstale":"2023-10-16T14:11:38.813613+0000","last_undegraded":"2023-10-16T14:11:38.813613+0000","last_fullsized":"2023-10-16T14:11:38.813613+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:33.536538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811861+0000","last_change":"2023-10-16T14:11:38.811158+0000","last_active":"2023-10-16T14:11:38.811861+0000","last_peered":"2023-10-16T14:11:38.811861+0000","last_clean":"2023-10-16T14:11:38.811861+0000","last_became_active":"2023-10-16T14:11:38.810852+0000","last_became_peered":"2023-10-16T14:11:38.810852+0000","last_unstale":"2023-10-16T14:11:38.811861+0000","last_undegraded":"2023-10-16T14:11:38.811861+0000","last_fullsized":"2023-10-16T14:11:38.811861+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:35:39.262135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810241+0000","last_change":"2023-10-16T14:11:38.804475+0000","last_active":"2023-10-16T14:11:38.810241+0000","last_peered":"2023-10-16T14:11:38.810241+0000","last_clean":"2023-10-16T14:11:38.810241+0000","last_became_active":"2023-10-16T14:11:38.804187+0000","last_became_peered":"2023-10-16T14:11:38.804187+0000","last_unstale":"2023-10-16T14:11:38.810241+0000","last_undegraded":"2023-10-16T14:11:38.810241+0000","last_fullsized":"2023-10-16T14:11:38.810241+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:40.929143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.812844+0000","last_change":"2023-10-16T14:11:38.808466+0000","last_active":"2023-10-16T14:11:38.812844+0000","last_peered":"2023-10-16T14:11:38.812844+0000","last_clean":"2023-10-16T14:11:38.812844+0000","last_became_active":"2023-10-16T14:11:38.808142+0000","last_became_peered":"2023-10-16T14:11:38.808142+0000","last_unstale":"2023-10-16T14:11:38.812844+0000","last_undegraded":"2023-10-16T14:11:38.812844+0000","last_fullsized":"2023-10-16T14:11:38.812844+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:08.231712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.809204+0000","last_change":"2023-10-16T14:11:38.802531+0000","last_active":"2023-10-16T14:11:38.809204+0000","last_peered":"2023-10-16T14:11:38.809204+0000","last_clean":"2023-10-16T14:11:38.809204+0000","last_became_active":"2023-10-16T14:11:38.802127+0000","last_became_peered":"2023-10-16T14:11:38.802127+0000","last_unstale":"2023-10-16T14:11:38.809204+0000","last_undegraded":"2023-10-16T14:11:38.809204+0000","last_fullsized":"2023-10-16T14:11:38.809204+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:07.396974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791820+0000","last_change":"2023-10-16T14:11:38.798475+0000","last_active":"2023-10-16T14:11:39.791820+0000","last_peered":"2023-10-16T14:11:39.791820+0000","last_clean":"2023-10-16T14:11:39.791820+0000","last_became_active":"2023-10-16T14:11:38.797198+0000","last_became_peered":"2023-10-16T14:11:38.797198+0000","last_unstale":"2023-10-16T14:11:39.791820+0000","last_undegraded":"2023-10-16T14:11:39.791820+0000","last_fullsized":"2023-10-16T14:11:39.791820+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:37.361885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808651+0000","last_change":"2023-10-16T14:11:38.803008+0000","last_active":"2023-10-16T14:11:38.808651+0000","last_peered":"2023-10-16T14:11:38.808651+0000","last_clean":"2023-10-16T14:11:38.808651+0000","last_became_active":"2023-10-16T14:11:38.800876+0000","last_became_peered":"2023-10-16T14:11:38.800876+0000","last_unstale":"2023-10-16T14:11:38.808651+0000","last_undegraded":"2023-10-16T14:11:38.808651+0000","last_fullsized":"2023-10-16T14:11:38.808651+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.534242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807455+0000","last_change":"2023-10-16T14:11:38.803699+0000","last_active":"2023-10-16T14:11:38.807455+0000","last_peered":"2023-10-16T14:11:38.807455+0000","last_clean":"2023-10-16T14:11:38.807455+0000","last_became_active":"2023-10-16T14:11:38.801233+0000","last_became_peered":"2023-10-16T14:11:38.801233+0000","last_unstale":"2023-10-16T14:11:38.807455+0000","last_undegraded":"2023-10-16T14:11:38.807455+0000","last_fullsized":"2023-10-16T14:11:38.807455+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:25.816517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.784122+0000","last_change":"2023-10-16T14:11:38.800010+0000","last_active":"2023-10-16T14:11:39.784122+0000","last_peered":"2023-10-16T14:11:39.784122+0000","last_clean":"2023-10-16T14:11:39.784122+0000","last_became_active":"2023-10-16T14:11:38.799422+0000","last_became_peered":"2023-10-16T14:11:38.799422+0000","last_unstale":"2023-10-16T14:11:39.784122+0000","last_undegraded":"2023-10-16T14:11:39.784122+0000","last_fullsized":"2023-10-16T14:11:39.784122+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:01:31.181539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.852015+0000","last_change":"2023-10-16T14:11:37.778697+0000","last_active":"2023-10-16T14:11:38.852015+0000","last_peered":"2023-10-16T14:11:38.852015+0000","last_clean":"2023-10-16T14:11:38.852015+0000","last_became_active":"2023-10-16T14:11:37.778387+0000","last_became_peered":"2023-10-16T14:11:37.778387+0000","last_unstale":"2023-10-16T14:11:38.852015+0000","last_undegraded":"2023-10-16T14:11:38.852015+0000","last_fullsized":"2023-10-16T14:11:38.852015+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:09.302616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810839+0000","last_change":"2023-10-16T14:11:38.805302+0000","last_active":"2023-10-16T14:11:38.810839+0000","last_peered":"2023-10-16T14:11:38.810839+0000","last_clean":"2023-10-16T14:11:38.810839+0000","last_became_active":"2023-10-16T14:11:38.803432+0000","last_became_peered":"2023-10-16T14:11:38.803432+0000","last_unstale":"2023-10-16T14:11:38.810839+0000","last_undegraded":"2023-10-16T14:11:38.810839+0000","last_fullsized":"2023-10-16T14:11:38.810839+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:35.951261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783759+0000","last_change":"2023-10-16T14:11:37.781195+0000","last_active":"2023-10-16T14:11:39.783759+0000","last_peered":"2023-10-16T14:11:39.783759+0000","last_clean":"2023-10-16T14:11:39.783759+0000","last_became_active":"2023-10-16T14:11:37.780551+0000","last_became_peered":"2023-10-16T14:11:37.780551+0000","last_unstale":"2023-10-16T14:11:39.783759+0000","last_undegraded":"2023-10-16T14:11:39.783759+0000","last_fullsized":"2023-10-16T14:11:39.783759+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:33.391410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790836+0000","last_change":"2023-10-16T14:11:38.802734+0000","last_active":"2023-10-16T14:11:39.790836+0000","last_peered":"2023-10-16T14:11:39.790836+0000","last_clean":"2023-10-16T14:11:39.790836+0000","last_became_active":"2023-10-16T14:11:38.802347+0000","last_became_peered":"2023-10-16T14:11:38.802347+0000","last_unstale":"2023-10-16T14:11:39.790836+0000","last_undegraded":"2023-10-16T14:11:39.790836+0000","last_fullsized":"2023-10-16T14:11:39.790836+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:38.977236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783524+0000","last_change":"2023-10-16T14:11:37.783063+0000","last_active":"2023-10-16T14:11:39.783524+0000","last_peered":"2023-10-16T14:11:39.783524+0000","last_clean":"2023-10-16T14:11:39.783524+0000","last_became_active":"2023-10-16T14:11:37.782768+0000","last_became_peered":"2023-10-16T14:11:37.782768+0000","last_unstale":"2023-10-16T14:11:39.783524+0000","last_undegraded":"2023-10-16T14:11:39.783524+0000","last_fullsized":"2023-10-16T14:11:39.783524+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:36.980155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.800168+0000","last_change":"2023-10-16T14:11:38.786427+0000","last_active":"2023-10-16T14:11:38.800168+0000","last_peered":"2023-10-16T14:11:38.800168+0000","last_clean":"2023-10-16T14:11:38.800168+0000","last_became_active":"2023-10-16T14:11:38.785671+0000","last_became_peered":"2023-10-16T14:11:38.785671+0000","last_unstale":"2023-10-16T14:11:38.800168+0000","last_undegraded":"2023-10-16T14:11:38.800168+0000","last_fullsized":"2023-10-16T14:11:38.800168+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:59:28.844112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.854622+0000","last_change":"2023-10-16T14:11:37.774757+0000","last_active":"2023-10-16T14:11:38.854622+0000","last_peered":"2023-10-16T14:11:38.854622+0000","last_clean":"2023-10-16T14:11:38.854622+0000","last_became_active":"2023-10-16T14:11:37.774442+0000","last_became_peered":"2023-10-16T14:11:37.774442+0000","last_unstale":"2023-10-16T14:11:38.854622+0000","last_undegraded":"2023-10-16T14:11:38.854622+0000","last_fullsized":"2023-10-16T14:11:38.854622+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:45.874083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.810591+0000","last_change":"2023-10-16T14:11:38.809976+0000","last_active":"2023-10-16T14:11:38.810591+0000","last_peered":"2023-10-16T14:11:38.810591+0000","last_clean":"2023-10-16T14:11:38.810591+0000","last_became_active":"2023-10-16T14:11:38.809754+0000","last_became_peered":"2023-10-16T14:11:38.809754+0000","last_unstale":"2023-10-16T14:11:38.810591+0000","last_undegraded":"2023-10-16T14:11:38.810591+0000","last_fullsized":"2023-10-16T14:11:38.810591+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:58.363248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.855171+0000","last_change":"2023-10-16T14:11:37.776847+0000","last_active":"2023-10-16T14:11:38.855171+0000","last_peered":"2023-10-16T14:11:38.855171+0000","last_clean":"2023-10-16T14:11:38.855171+0000","last_became_active":"2023-10-16T14:11:37.775028+0000","last_became_peered":"2023-10-16T14:11:37.775028+0000","last_unstale":"2023-10-16T14:11:38.855171+0000","last_undegraded":"2023-10-16T14:11:38.855171+0000","last_fullsized":"2023-10-16T14:11:38.855171+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:54.089883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782646+0000","last_change":"2023-10-16T14:11:38.801326+0000","last_active":"2023-10-16T14:11:39.782646+0000","last_peered":"2023-10-16T14:11:39.782646+0000","last_clean":"2023-10-16T14:11:39.782646+0000","last_became_active":"2023-10-16T14:11:38.800581+0000","last_became_peered":"2023-10-16T14:11:38.800581+0000","last_unstale":"2023-10-16T14:11:39.782646+0000","last_undegraded":"2023-10-16T14:11:39.782646+0000","last_fullsized":"2023-10-16T14:11:39.782646+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:09.367409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.785623+0000","last_change":"2023-10-16T14:11:37.775158+0000","last_active":"2023-10-16T14:11:38.785623+0000","last_peered":"2023-10-16T14:11:38.785623+0000","last_clean":"2023-10-16T14:11:38.785623+0000","last_became_active":"2023-10-16T14:11:37.773264+0000","last_became_peered":"2023-10-16T14:11:37.773264+0000","last_unstale":"2023-10-16T14:11:38.785623+0000","last_undegraded":"2023-10-16T14:11:38.785623+0000","last_fullsized":"2023-10-16T14:11:38.785623+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:27.581413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807413+0000","last_change":"2023-10-16T14:11:38.802528+0000","last_active":"2023-10-16T14:11:38.807413+0000","last_peered":"2023-10-16T14:11:38.807413+0000","last_clean":"2023-10-16T14:11:38.807413+0000","last_became_active":"2023-10-16T14:11:38.802212+0000","last_became_peered":"2023-10-16T14:11:38.802212+0000","last_unstale":"2023-10-16T14:11:38.807413+0000","last_undegraded":"2023-10-16T14:11:38.807413+0000","last_fullsized":"2023-10-16T14:11:38.807413+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:31.968414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782705+0000","last_change":"2023-10-16T14:11:37.780327+0000","last_active":"2023-10-16T14:11:39.782705+0000","last_peered":"2023-10-16T14:11:39.782705+0000","last_clean":"2023-10-16T14:11:39.782705+0000","last_became_active":"2023-10-16T14:11:37.779846+0000","last_became_peered":"2023-10-16T14:11:37.779846+0000","last_unstale":"2023-10-16T14:11:39.782705+0000","last_undegraded":"2023-10-16T14:11:39.782705+0000","last_fullsized":"2023-10-16T14:11:39.782705+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:15.446021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.812563+0000","last_change":"2023-10-16T14:11:38.809523+0000","last_active":"2023-10-16T14:11:38.812563+0000","last_peered":"2023-10-16T14:11:38.812563+0000","last_clean":"2023-10-16T14:11:38.812563+0000","last_became_active":"2023-10-16T14:11:38.807663+0000","last_became_peered":"2023-10-16T14:11:38.807663+0000","last_unstale":"2023-10-16T14:11:38.812563+0000","last_undegraded":"2023-10-16T14:11:38.812563+0000","last_fullsized":"2023-10-16T14:11:38.812563+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:32.228190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790522+0000","last_change":"2023-10-16T14:11:37.784204+0000","last_active":"2023-10-16T14:11:39.790522+0000","last_peered":"2023-10-16T14:11:39.790522+0000","last_clean":"2023-10-16T14:11:39.790522+0000","last_became_active":"2023-10-16T14:11:37.783880+0000","last_became_peered":"2023-10-16T14:11:37.783880+0000","last_unstale":"2023-10-16T14:11:39.790522+0000","last_undegraded":"2023-10-16T14:11:39.790522+0000","last_fullsized":"2023-10-16T14:11:39.790522+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:22.672658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.808564+0000","last_change":"2023-10-16T14:11:38.802292+0000","last_active":"2023-10-16T14:11:38.808564+0000","last_peered":"2023-10-16T14:11:38.808564+0000","last_clean":"2023-10-16T14:11:38.808564+0000","last_became_active":"2023-10-16T14:11:38.801904+0000","last_became_peered":"2023-10-16T14:11:38.801904+0000","last_unstale":"2023-10-16T14:11:38.808564+0000","last_undegraded":"2023-10-16T14:11:38.808564+0000","last_fullsized":"2023-10-16T14:11:38.808564+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:17.405096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783757+0000","last_change":"2023-10-16T14:11:37.774897+0000","last_active":"2023-10-16T14:11:39.783757+0000","last_peered":"2023-10-16T14:11:39.783757+0000","last_clean":"2023-10-16T14:11:39.783757+0000","last_became_active":"2023-10-16T14:11:37.774465+0000","last_became_peered":"2023-10-16T14:11:37.774465+0000","last_unstale":"2023-10-16T14:11:39.783757+0000","last_undegraded":"2023-10-16T14:11:39.783757+0000","last_fullsized":"2023-10-16T14:11:39.783757+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783479+0000","last_change":"2023-10-16T14:11:38.809093+0000","last_active":"2023-10-16T14:11:39.783479+0000","last_peered":"2023-10-16T14:11:39.783479+0000","last_clean":"2023-10-16T14:11:39.783479+0000","last_became_active":"2023-10-16T14:11:38.808829+0000","last_became_peered":"2023-10-16T14:11:38.808829+0000","last_unstale":"2023-10-16T14:11:39.783479+0000","last_undegraded":"2023-10-16T14:11:39.783479+0000","last_fullsized":"2023-10-16T14:11:39.783479+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:09.522737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"251'3","reported_seq":14,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.857819+0000","last_change":"2023-10-16T14:11:37.772650+0000","last_active":"2023-10-16T14:11:38.857819+0000","last_peered":"2023-10-16T14:11:38.857819+0000","last_clean":"2023-10-16T14:11:38.857819+0000","last_became_active":"2023-10-16T14:11:37.772209+0000","last_became_peered":"2023-10-16T14:11:37.772209+0000","last_unstale":"2023-10-16T14:11:38.857819+0000","last_undegraded":"2023-10-16T14:11:38.857819+0000","last_fullsized":"2023-10-16T14:11:38.857819+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:51.981249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804292+0000","last_change":"2023-10-16T14:11:38.800005+0000","last_active":"2023-10-16T14:11:38.804292+0000","last_peered":"2023-10-16T14:11:38.804292+0000","last_clean":"2023-10-16T14:11:38.804292+0000","last_became_active":"2023-10-16T14:11:38.799654+0000","last_became_peered":"2023-10-16T14:11:38.799654+0000","last_unstale":"2023-10-16T14:11:38.804292+0000","last_undegraded":"2023-10-16T14:11:38.804292+0000","last_fullsized":"2023-10-16T14:11:38.804292+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:52:23.497949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"251'6","reported_seq":17,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.860833+0000","last_change":"2023-10-16T14:11:37.783855+0000","last_active":"2023-10-16T14:11:38.860833+0000","last_peered":"2023-10-16T14:11:38.860833+0000","last_clean":"2023-10-16T14:11:38.860833+0000","last_became_active":"2023-10-16T14:11:37.783474+0000","last_became_peered":"2023-10-16T14:11:37.783474+0000","last_unstale":"2023-10-16T14:11:38.860833+0000","last_undegraded":"2023-10-16T14:11:38.860833+0000","last_fullsized":"2023-10-16T14:11:38.860833+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.800247+0000","last_change":"2023-10-16T14:11:38.796891+0000","last_active":"2023-10-16T14:11:38.800247+0000","last_peered":"2023-10-16T14:11:38.800247+0000","last_clean":"2023-10-16T14:11:38.800247+0000","last_became_active":"2023-10-16T14:11:38.796574+0000","last_became_peered":"2023-10-16T14:11:38.796574+0000","last_unstale":"2023-10-16T14:11:38.800247+0000","last_undegraded":"2023-10-16T14:11:38.800247+0000","last_fullsized":"2023-10-16T14:11:38.800247+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:46.154550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"251'2","reported_seq":13,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.857824+0000","last_change":"2023-10-16T14:11:37.767366+0000","last_active":"2023-10-16T14:11:38.857824+0000","last_peered":"2023-10-16T14:11:38.857824+0000","last_clean":"2023-10-16T14:11:38.857824+0000","last_became_active":"2023-10-16T14:11:37.766979+0000","last_became_peered":"2023-10-16T14:11:37.766979+0000","last_unstale":"2023-10-16T14:11:38.857824+0000","last_undegraded":"2023-10-16T14:11:38.857824+0000","last_fullsized":"2023-10-16T14:11:38.857824+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:25.104652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.803947+0000","last_change":"2023-10-16T14:11:38.801883+0000","last_active":"2023-10-16T14:11:38.803947+0000","last_peered":"2023-10-16T14:11:38.803947+0000","last_clean":"2023-10-16T14:11:38.803947+0000","last_became_active":"2023-10-16T14:11:38.800123+0000","last_became_peered":"2023-10-16T14:11:38.800123+0000","last_unstale":"2023-10-16T14:11:38.803947+0000","last_undegraded":"2023-10-16T14:11:38.803947+0000","last_fullsized":"2023-10-16T14:11:38.803947+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:34.408581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782904+0000","last_change":"2023-10-16T14:11:37.779309+0000","last_active":"2023-10-16T14:11:39.782904+0000","last_peered":"2023-10-16T14:11:39.782904+0000","last_clean":"2023-10-16T14:11:39.782904+0000","last_became_active":"2023-10-16T14:11:37.778681+0000","last_became_peered":"2023-10-16T14:11:37.778681+0000","last_unstale":"2023-10-16T14:11:39.782904+0000","last_undegraded":"2023-10-16T14:11:39.782904+0000","last_fullsized":"2023-10-16T14:11:39.782904+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:41.372800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.802323+0000","last_change":"2023-10-16T14:11:38.799773+0000","last_active":"2023-10-16T14:11:38.802323+0000","last_peered":"2023-10-16T14:11:38.802323+0000","last_clean":"2023-10-16T14:11:38.802323+0000","last_became_active":"2023-10-16T14:11:38.799446+0000","last_became_peered":"2023-10-16T14:11:38.799446+0000","last_unstale":"2023-10-16T14:11:38.802323+0000","last_undegraded":"2023-10-16T14:11:38.802323+0000","last_fullsized":"2023-10-16T14:11:38.802323+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:06.091417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.785773+0000","last_change":"2023-10-16T14:11:37.777163+0000","last_active":"2023-10-16T14:11:38.785773+0000","last_peered":"2023-10-16T14:11:38.785773+0000","last_clean":"2023-10-16T14:11:38.785773+0000","last_became_active":"2023-10-16T14:11:37.776875+0000","last_became_peered":"2023-10-16T14:11:37.776875+0000","last_unstale":"2023-10-16T14:11:38.785773+0000","last_undegraded":"2023-10-16T14:11:38.785773+0000","last_fullsized":"2023-10-16T14:11:38.785773+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:59.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804361+0000","last_change":"2023-10-16T14:11:38.801468+0000","last_active":"2023-10-16T14:11:38.804361+0000","last_peered":"2023-10-16T14:11:38.804361+0000","last_clean":"2023-10-16T14:11:38.804361+0000","last_became_active":"2023-10-16T14:11:38.801182+0000","last_became_peered":"2023-10-16T14:11:38.801182+0000","last_unstale":"2023-10-16T14:11:38.804361+0000","last_undegraded":"2023-10-16T14:11:38.804361+0000","last_fullsized":"2023-10-16T14:11:38.804361+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:01.648322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"251'8","reported_seq":19,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.859720+0000","last_change":"2023-10-16T14:11:37.784533+0000","last_active":"2023-10-16T14:11:38.859720+0000","last_peered":"2023-10-16T14:11:38.859720+0000","last_clean":"2023-10-16T14:11:38.859720+0000","last_became_active":"2023-10-16T14:11:37.784209+0000","last_became_peered":"2023-10-16T14:11:37.784209+0000","last_unstale":"2023-10-16T14:11:38.859720+0000","last_undegraded":"2023-10-16T14:11:38.859720+0000","last_fullsized":"2023-10-16T14:11:38.859720+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:49.027722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.813284+0000","last_change":"2023-10-16T14:11:38.810290+0000","last_active":"2023-10-16T14:11:38.813284+0000","last_peered":"2023-10-16T14:11:38.813284+0000","last_clean":"2023-10-16T14:11:38.813284+0000","last_became_active":"2023-10-16T14:11:38.809830+0000","last_became_peered":"2023-10-16T14:11:38.809830+0000","last_unstale":"2023-10-16T14:11:38.813284+0000","last_undegraded":"2023-10-16T14:11:38.813284+0000","last_fullsized":"2023-10-16T14:11:38.813284+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:18.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.852464+0000","last_change":"2023-10-16T14:11:37.786947+0000","last_active":"2023-10-16T14:11:38.852464+0000","last_peered":"2023-10-16T14:11:38.852464+0000","last_clean":"2023-10-16T14:11:38.852464+0000","last_became_active":"2023-10-16T14:11:37.786340+0000","last_became_peered":"2023-10-16T14:11:37.786340+0000","last_unstale":"2023-10-16T14:11:38.852464+0000","last_undegraded":"2023-10-16T14:11:38.852464+0000","last_fullsized":"2023-10-16T14:11:38.852464+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:01.576946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783898+0000","last_change":"2023-10-16T14:11:38.804059+0000","last_active":"2023-10-16T14:11:39.783898+0000","last_peered":"2023-10-16T14:11:39.783898+0000","last_clean":"2023-10-16T14:11:39.783898+0000","last_became_active":"2023-10-16T14:11:38.803768+0000","last_became_peered":"2023-10-16T14:11:38.803768+0000","last_unstale":"2023-10-16T14:11:39.783898+0000","last_undegraded":"2023-10-16T14:11:39.783898+0000","last_fullsized":"2023-10-16T14:11:39.783898+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:40.313169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783764+0000","last_change":"2023-10-16T14:11:37.780012+0000","last_active":"2023-10-16T14:11:39.783764+0000","last_peered":"2023-10-16T14:11:39.783764+0000","last_clean":"2023-10-16T14:11:39.783764+0000","last_became_active":"2023-10-16T14:11:37.779730+0000","last_became_peered":"2023-10-16T14:11:37.779730+0000","last_unstale":"2023-10-16T14:11:39.783764+0000","last_undegraded":"2023-10-16T14:11:39.783764+0000","last_fullsized":"2023-10-16T14:11:39.783764+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.805206+0000","last_change":"2023-10-16T14:11:38.804003+0000","last_active":"2023-10-16T14:11:38.805206+0000","last_peered":"2023-10-16T14:11:38.805206+0000","last_clean":"2023-10-16T14:11:38.805206+0000","last_became_active":"2023-10-16T14:11:38.803711+0000","last_became_peered":"2023-10-16T14:11:38.803711+0000","last_unstale":"2023-10-16T14:11:38.805206+0000","last_undegraded":"2023-10-16T14:11:38.805206+0000","last_fullsized":"2023-10-16T14:11:38.805206+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:42.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"251'1","reported_seq":12,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.852774+0000","last_change":"2023-10-16T14:11:37.784775+0000","last_active":"2023-10-16T14:11:38.852774+0000","last_peered":"2023-10-16T14:11:38.852774+0000","last_clean":"2023-10-16T14:11:38.852774+0000","last_became_active":"2023-10-16T14:11:37.784340+0000","last_became_peered":"2023-10-16T14:11:37.784340+0000","last_unstale":"2023-10-16T14:11:38.852774+0000","last_undegraded":"2023-10-16T14:11:38.852774+0000","last_fullsized":"2023-10-16T14:11:38.852774+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791158+0000","last_change":"2023-10-16T14:11:38.797986+0000","last_active":"2023-10-16T14:11:39.791158+0000","last_peered":"2023-10-16T14:11:39.791158+0000","last_clean":"2023-10-16T14:11:39.791158+0000","last_became_active":"2023-10-16T14:11:38.797632+0000","last_became_peered":"2023-10-16T14:11:38.797632+0000","last_unstale":"2023-10-16T14:11:39.791158+0000","last_undegraded":"2023-10-16T14:11:39.791158+0000","last_fullsized":"2023-10-16T14:11:39.791158+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:33.536538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.772569+0000","last_change":"2023-10-16T14:11:37.772519+0000","last_active":"2023-10-16T14:11:38.772569+0000","last_peered":"2023-10-16T14:11:38.772569+0000","last_clean":"2023-10-16T14:11:38.772569+0000","last_became_active":"2023-10-16T14:11:37.772241+0000","last_became_peered":"2023-10-16T14:11:37.772241+0000","last_unstale":"2023-10-16T14:11:38.772569+0000","last_undegraded":"2023-10-16T14:11:38.772569+0000","last_fullsized":"2023-10-16T14:11:38.772569+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:47.596657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789351+0000","last_change":"2023-10-16T14:11:38.801767+0000","last_active":"2023-10-16T14:11:39.789351+0000","last_peered":"2023-10-16T14:11:39.789351+0000","last_clean":"2023-10-16T14:11:39.789351+0000","last_became_active":"2023-10-16T14:11:38.800797+0000","last_became_peered":"2023-10-16T14:11:38.800797+0000","last_unstale":"2023-10-16T14:11:39.789351+0000","last_undegraded":"2023-10-16T14:11:39.789351+0000","last_fullsized":"2023-10-16T14:11:39.789351+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:05.512554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"251'2","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783306+0000","last_change":"2023-10-16T14:11:37.781065+0000","last_active":"2023-10-16T14:11:39.783306+0000","last_peered":"2023-10-16T14:11:39.783306+0000","last_clean":"2023-10-16T14:11:39.783306+0000","last_became_active":"2023-10-16T14:11:37.780556+0000","last_became_peered":"2023-10-16T14:11:37.780556+0000","last_unstale":"2023-10-16T14:11:39.783306+0000","last_undegraded":"2023-10-16T14:11:39.783306+0000","last_fullsized":"2023-10-16T14:11:39.783306+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:03.865559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.807195+0000","last_change":"2023-10-16T14:11:38.803116+0000","last_active":"2023-10-16T14:11:38.807195+0000","last_peered":"2023-10-16T14:11:38.807195+0000","last_clean":"2023-10-16T14:11:38.807195+0000","last_became_active":"2023-10-16T14:11:38.802518+0000","last_became_peered":"2023-10-16T14:11:38.802518+0000","last_unstale":"2023-10-16T14:11:38.807195+0000","last_undegraded":"2023-10-16T14:11:38.807195+0000","last_fullsized":"2023-10-16T14:11:38.807195+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:08.898698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.770638+0000","last_change":"2023-10-16T14:11:37.768392+0000","last_active":"2023-10-16T14:11:38.770638+0000","last_peered":"2023-10-16T14:11:38.770638+0000","last_clean":"2023-10-16T14:11:38.770638+0000","last_became_active":"2023-10-16T14:11:37.768081+0000","last_became_peered":"2023-10-16T14:11:37.768081+0000","last_unstale":"2023-10-16T14:11:38.770638+0000","last_undegraded":"2023-10-16T14:11:38.770638+0000","last_fullsized":"2023-10-16T14:11:38.770638+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:13.231020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.806651+0000","last_change":"2023-10-16T14:11:38.796924+0000","last_active":"2023-10-16T14:11:38.806651+0000","last_peered":"2023-10-16T14:11:38.806651+0000","last_clean":"2023-10-16T14:11:38.806651+0000","last_became_active":"2023-10-16T14:11:38.796610+0000","last_became_peered":"2023-10-16T14:11:38.796610+0000","last_unstale":"2023-10-16T14:11:38.806651+0000","last_undegraded":"2023-10-16T14:11:38.806651+0000","last_fullsized":"2023-10-16T14:11:38.806651+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:13:39.361933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.792228+0000","last_change":"2023-10-16T14:11:37.780387+0000","last_active":"2023-10-16T14:11:38.792228+0000","last_peered":"2023-10-16T14:11:38.792228+0000","last_clean":"2023-10-16T14:11:38.792228+0000","last_became_active":"2023-10-16T14:11:37.779362+0000","last_became_peered":"2023-10-16T14:11:37.779362+0000","last_unstale":"2023-10-16T14:11:38.792228+0000","last_undegraded":"2023-10-16T14:11:38.792228+0000","last_fullsized":"2023-10-16T14:11:38.792228+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791065+0000","last_change":"2023-10-16T14:11:38.799901+0000","last_active":"2023-10-16T14:11:39.791065+0000","last_peered":"2023-10-16T14:11:39.791065+0000","last_clean":"2023-10-16T14:11:39.791065+0000","last_became_active":"2023-10-16T14:11:38.799145+0000","last_became_peered":"2023-10-16T14:11:38.799145+0000","last_unstale":"2023-10-16T14:11:39.791065+0000","last_undegraded":"2023-10-16T14:11:39.791065+0000","last_fullsized":"2023-10-16T14:11:39.791065+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.534242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.785145+0000","last_change":"2023-10-16T14:11:37.779700+0000","last_active":"2023-10-16T14:11:38.785145+0000","last_peered":"2023-10-16T14:11:38.785145+0000","last_clean":"2023-10-16T14:11:38.785145+0000","last_became_active":"2023-10-16T14:11:37.779082+0000","last_became_peered":"2023-10-16T14:11:37.779082+0000","last_unstale":"2023-10-16T14:11:38.785145+0000","last_undegraded":"2023-10-16T14:11:38.785145+0000","last_fullsized":"2023-10-16T14:11:38.785145+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:12.639268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.804305+0000","last_change":"2023-10-16T14:11:38.800281+0000","last_active":"2023-10-16T14:11:38.804305+0000","last_peered":"2023-10-16T14:11:38.804305+0000","last_clean":"2023-10-16T14:11:38.804305+0000","last_became_active":"2023-10-16T14:11:38.799813+0000","last_became_peered":"2023-10-16T14:11:38.799813+0000","last_unstale":"2023-10-16T14:11:38.804305+0000","last_undegraded":"2023-10-16T14:11:38.804305+0000","last_fullsized":"2023-10-16T14:11:38.804305+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:52.534482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"251'2","reported_seq":13,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.856884+0000","last_change":"2023-10-16T14:11:37.776293+0000","last_active":"2023-10-16T14:11:38.856884+0000","last_peered":"2023-10-16T14:11:38.856884+0000","last_clean":"2023-10-16T14:11:38.856884+0000","last_became_active":"2023-10-16T14:11:37.775902+0000","last_became_peered":"2023-10-16T14:11:37.775902+0000","last_unstale":"2023-10-16T14:11:38.856884+0000","last_undegraded":"2023-10-16T14:11:38.856884+0000","last_fullsized":"2023-10-16T14:11:38.856884+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:09.428253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811636+0000","last_change":"2023-10-16T14:11:38.806901+0000","last_active":"2023-10-16T14:11:38.811636+0000","last_peered":"2023-10-16T14:11:38.811636+0000","last_clean":"2023-10-16T14:11:38.811636+0000","last_became_active":"2023-10-16T14:11:38.806614+0000","last_became_peered":"2023-10-16T14:11:38.806614+0000","last_unstale":"2023-10-16T14:11:38.811636+0000","last_undegraded":"2023-10-16T14:11:38.811636+0000","last_fullsized":"2023-10-16T14:11:38.811636+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:01.915134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.790112+0000","last_change":"2023-10-16T14:11:37.780349+0000","last_active":"2023-10-16T14:11:39.790112+0000","last_peered":"2023-10-16T14:11:39.790112+0000","last_clean":"2023-10-16T14:11:39.790112+0000","last_became_active":"2023-10-16T14:11:37.777254+0000","last_became_peered":"2023-10-16T14:11:37.777254+0000","last_unstale":"2023-10-16T14:11:39.790112+0000","last_undegraded":"2023-10-16T14:11:39.790112+0000","last_fullsized":"2023-10-16T14:11:39.790112+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:22.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.811104+0000","last_change":"2023-10-16T14:11:38.805448+0000","last_active":"2023-10-16T14:11:38.811104+0000","last_peered":"2023-10-16T14:11:38.811104+0000","last_clean":"2023-10-16T14:11:38.811104+0000","last_became_active":"2023-10-16T14:11:38.805128+0000","last_became_peered":"2023-10-16T14:11:38.805128+0000","last_unstale":"2023-10-16T14:11:38.811104+0000","last_undegraded":"2023-10-16T14:11:38.811104+0000","last_fullsized":"2023-10-16T14:11:38.811104+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:15.426100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.789362+0000","last_change":"2023-10-16T14:11:37.782304+0000","last_active":"2023-10-16T14:11:38.789362+0000","last_peered":"2023-10-16T14:11:38.789362+0000","last_clean":"2023-10-16T14:11:38.789362+0000","last_became_active":"2023-10-16T14:11:37.781840+0000","last_became_peered":"2023-10-16T14:11:37.781840+0000","last_unstale":"2023-10-16T14:11:38.789362+0000","last_undegraded":"2023-10-16T14:11:38.789362+0000","last_fullsized":"2023-10-16T14:11:38.789362+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.809485+0000","last_change":"2023-10-16T14:11:38.801623+0000","last_active":"2023-10-16T14:11:38.809485+0000","last_peered":"2023-10-16T14:11:38.809485+0000","last_clean":"2023-10-16T14:11:38.809485+0000","last_became_active":"2023-10-16T14:11:38.801313+0000","last_became_peered":"2023-10-16T14:11:38.801313+0000","last_unstale":"2023-10-16T14:11:38.809485+0000","last_undegraded":"2023-10-16T14:11:38.809485+0000","last_fullsized":"2023-10-16T14:11:38.809485+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:57:38.359073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.772893+0000","last_change":"2023-10-16T14:11:37.775930+0000","last_active":"2023-10-16T14:11:38.772893+0000","last_peered":"2023-10-16T14:11:38.772893+0000","last_clean":"2023-10-16T14:11:38.772893+0000","last_became_active":"2023-10-16T14:11:37.775617+0000","last_became_peered":"2023-10-16T14:11:37.775617+0000","last_unstale":"2023-10-16T14:11:38.772893+0000","last_undegraded":"2023-10-16T14:11:38.772893+0000","last_fullsized":"2023-10-16T14:11:38.772893+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:19.733822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.800229+0000","last_change":"2023-10-16T14:11:38.796749+0000","last_active":"2023-10-16T14:11:38.800229+0000","last_peered":"2023-10-16T14:11:38.800229+0000","last_clean":"2023-10-16T14:11:38.800229+0000","last_became_active":"2023-10-16T14:11:38.796468+0000","last_became_peered":"2023-10-16T14:11:38.796468+0000","last_unstale":"2023-10-16T14:11:38.800229+0000","last_undegraded":"2023-10-16T14:11:38.800229+0000","last_fullsized":"2023-10-16T14:11:38.800229+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:18.275683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.789565+0000","last_change":"2023-10-16T14:11:37.785747+0000","last_active":"2023-10-16T14:11:38.789565+0000","last_peered":"2023-10-16T14:11:38.789565+0000","last_clean":"2023-10-16T14:11:38.789565+0000","last_became_active":"2023-10-16T14:11:37.785464+0000","last_became_peered":"2023-10-16T14:11:37.785464+0000","last_unstale":"2023-10-16T14:11:38.789565+0000","last_undegraded":"2023-10-16T14:11:38.789565+0000","last_fullsized":"2023-10-16T14:11:38.789565+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:32.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.791444+0000","last_change":"2023-10-16T14:11:38.800958+0000","last_active":"2023-10-16T14:11:39.791444+0000","last_peered":"2023-10-16T14:11:39.791444+0000","last_clean":"2023-10-16T14:11:39.791444+0000","last_became_active":"2023-10-16T14:11:38.800677+0000","last_became_peered":"2023-10-16T14:11:38.800677+0000","last_unstale":"2023-10-16T14:11:39.791444+0000","last_undegraded":"2023-10-16T14:11:39.791444+0000","last_fullsized":"2023-10-16T14:11:39.791444+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:18.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.797327+0000","last_change":"2023-10-16T14:11:37.785497+0000","last_active":"2023-10-16T14:11:38.797327+0000","last_peered":"2023-10-16T14:11:38.797327+0000","last_clean":"2023-10-16T14:11:38.797327+0000","last_became_active":"2023-10-16T14:11:37.785162+0000","last_became_peered":"2023-10-16T14:11:37.785162+0000","last_unstale":"2023-10-16T14:11:38.797327+0000","last_undegraded":"2023-10-16T14:11:38.797327+0000","last_fullsized":"2023-10-16T14:11:38.797327+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:28.524950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.788645+0000","last_change":"2023-10-16T14:11:38.789466+0000","last_active":"2023-10-16T14:11:39.788645+0000","last_peered":"2023-10-16T14:11:39.788645+0000","last_clean":"2023-10-16T14:11:39.788645+0000","last_became_active":"2023-10-16T14:11:38.788914+0000","last_became_peered":"2023-10-16T14:11:38.788914+0000","last_unstale":"2023-10-16T14:11:39.788645+0000","last_undegraded":"2023-10-16T14:11:39.788645+0000","last_fullsized":"2023-10-16T14:11:39.788645+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:36.423758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782705+0000","last_change":"2023-10-16T14:11:37.784108+0000","last_active":"2023-10-16T14:11:39.782705+0000","last_peered":"2023-10-16T14:11:39.782705+0000","last_clean":"2023-10-16T14:11:39.782705+0000","last_became_active":"2023-10-16T14:11:37.783469+0000","last_became_peered":"2023-10-16T14:11:37.783469+0000","last_unstale":"2023-10-16T14:11:39.782705+0000","last_undegraded":"2023-10-16T14:11:39.782705+0000","last_fullsized":"2023-10-16T14:11:39.782705+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2023-10-16T14:11:38.805397+0000","last_change":"2023-10-16T14:11:38.799118+0000","last_active":"2023-10-16T14:11:38.805397+0000","last_peered":"2023-10-16T14:11:38.805397+0000","last_clean":"2023-10-16T14:11:38.805397+0000","last_became_active":"2023-10-16T14:11:38.798708+0000","last_became_peered":"2023-10-16T14:11:38.798708+0000","last_unstale":"2023-10-16T14:11:38.805397+0000","last_undegraded":"2023-10-16T14:11:38.805397+0000","last_fullsized":"2023-10-16T14:11:38.805397+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:39.879989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.782779+0000","last_change":"2023-10-16T14:11:37.777958+0000","last_active":"2023-10-16T14:11:39.782779+0000","last_peered":"2023-10-16T14:11:39.782779+0000","last_clean":"2023-10-16T14:11:39.782779+0000","last_became_active":"2023-10-16T14:11:37.777687+0000","last_became_peered":"2023-10-16T14:11:37.777687+0000","last_unstale":"2023-10-16T14:11:39.782779+0000","last_undegraded":"2023-10-16T14:11:39.782779+0000","last_fullsized":"2023-10-16T14:11:39.782779+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:13.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.789873+0000","last_change":"2023-10-16T14:11:38.785931+0000","last_active":"2023-10-16T14:11:39.789873+0000","last_peered":"2023-10-16T14:11:39.789873+0000","last_clean":"2023-10-16T14:11:39.789873+0000","last_became_active":"2023-10-16T14:11:38.785345+0000","last_became_peered":"2023-10-16T14:11:38.785345+0000","last_unstale":"2023-10-16T14:11:39.789873+0000","last_undegraded":"2023-10-16T14:11:39.789873+0000","last_fullsized":"2023-10-16T14:11:39.789873+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:16.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"251'4","reported_seq":16,"reported_epoch":252,"state":"active+clean","last_fresh":"2023-10-16T14:11:39.783948+0000","last_change":"2023-10-16T14:11:37.781168+0000","last_active":"2023-10-16T14:11:39.783948+0000","last_peered":"2023-10-16T14:11:39.783948+0000","last_clean":"2023-10-16T14:11:39.783948+0000","last_became_active":"2023-10-16T14:11:37.780834+0000","last_became_peered":"2023-10-16T14:11:37.780834+0000","last_unstale":"2023-10-16T14:11:39.783948+0000","last_undegraded":"2023-10-16T14:11:39.783948+0000","last_fullsized":"2023-10-16T14:11:39.783948+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:21.622808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"237'373","reported_seq":1318,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-16T14:11:37.771608+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:11:37.771608+0000","last_peered":"2023-10-16T14:11:37.771608+0000","last_clean":"2023-10-16T14:11:37.771608+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:11:37.771608+0000","last_undegraded":"2023-10-16T14:11:37.771608+0000","last_fullsized":"2023-10-16T14:11:37.771608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543311,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52896,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318944,"statfs":{"total":96636764160,"available":96582598656,"internally_reserved":0,"allocated":3751936,"data_stored":2136002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3460000000000001,"15min":1.3660000000000001},"min":{"1min":0.77900000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.885},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.3480000000000001,"15min":1.3979999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7669999999999999,"5min":1.827,"15min":1.873},"last":1.413}]},{"osd":1,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.2190000000000001,"15min":1.244},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.228,"15min":1.2490000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.72,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.262}]},{"osd":2,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.228,"15min":1.258},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.482,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.224,"5min":1.347,"15min":1.3560000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.6930000000000001,"5min":2.028,"15min":2.028},"last":1.143}]},{"osd":3,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.363,"15min":1.403},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8320000000000001,"5min":1.921,"15min":2.097},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3999999999999999,"15min":1.4259999999999999},"min":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7969999999999999,"5min":1.946,"15min":2.0169999999999999},"last":1.349}]},{"osd":4,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.23,"15min":1.2390000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4390000000000001,"15min":1.4119999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.786,"5min":1.982,"15min":2.0009999999999999},"last":1.2150000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.391,"15min":1.3799999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.0590000000000002},"last":1.665},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.536},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.9550000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":1.3149999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.6259999999999999,"15min":1.633},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.0009999999999999,"5min":2.052,"15min":2.3250000000000002},"last":1.381},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.5249999999999999,"15min":1.48},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.8939999999999999,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.4650000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543312,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":3162112,"data_stored":1546168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2689999999999999,"15min":1.329},"min":{"1min":0.70299999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.788,"5min":1.887,"15min":1.887},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.3140000000000001,"15min":1.353},"min":{"1min":0.747,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8460000000000001,"5min":1.847,"15min":1.9390000000000001},"last":1.1679999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.206,"15min":1.2250000000000001},"min":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6279999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.21,"15min":1.2270000000000001},"min":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6799999999999999,"5min":1.748,"15min":1.77},"last":0.92600000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.24,"15min":1.2549999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.78,"5min":1.823,"15min":1.823},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.282,"15min":1.3169999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.8759999999999999,"5min":1.962,"15min":1.962},"last":1.0049999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.327,"15min":1.345},"min":{"1min":0.93400000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":1.075,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.776,"5min":1.911,"15min":1.911},"last":1.4430000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.393,"15min":1.409},"min":{"1min":1.1259999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5609999999999999,"15min":1.5509999999999999},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.3740000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.79800000000000004,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4990000000000001,"15min":1.5},"min":{"1min":1.2170000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.0990000000000002,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.413}]},{"osd":7,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.512,"15min":1.5049999999999999},"min":{"1min":0.999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0150000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.546,"15min":1.542},"min":{"1min":1.0269999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.157,"5min":2.157,"15min":2.6120000000000001},"last":1.1000000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543313,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30120,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341720,"statfs":{"total":96636764160,"available":96605921280,"internally_reserved":0,"allocated":3166208,"data_stored":1551864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1299999999999999,"15min":1.202},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.4750000000000001,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2150000000000001,"15min":1.2310000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.85299999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.282,"15min":1.298},"min":{"1min":0.77500000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.774,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1830000000000001,"15min":1.254},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.698,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.2}]},{"osd":2,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.304,"15min":1.335},"min":{"1min":0.90300000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.587,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.266,"15min":1.3120000000000001},"min":{"1min":0.80200000000000005,"5min":0.746,"15min":0.746},"max":{"1min":1.806,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.1240000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3859999999999999,"15min":1.407},"min":{"1min":0.89300000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9019999999999999,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.409,"15min":1.431},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8759999999999999,"5min":2.75,"15min":2.75},"last":1.4850000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3160000000000001,"15min":1.2989999999999999},"min":{"1min":0.77300000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.9299999999999999,"5min":2.774,"15min":2.774},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.454,"15min":1.4319999999999999},"min":{"1min":0.93700000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.661}]},{"osd":6,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4390000000000001,"15min":1.4319999999999999},"min":{"1min":1.1699999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9390000000000001,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.508,"15min":1.4910000000000001},"min":{"1min":1.048,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9590000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.6160000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.593,"15min":1.554},"min":{"1min":1.2170000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1680000000000001,"5min":2.177,"15min":2.4609999999999999},"last":1.532},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6200000000000001,"15min":1.603},"min":{"1min":1.113,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1000000000000001,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.5720000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543313,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66152,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305688,"statfs":{"total":96636764160,"available":96569024512,"internally_reserved":0,"allocated":3174400,"data_stored":1547268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.145,"15min":1.2050000000000001},"min":{"1min":0.61799999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1799999999999999,"15min":1.1990000000000001},"min":{"1min":0.748,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":0.89300000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1990000000000001,"15min":1.2490000000000001},"min":{"1min":0.80700000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6899999999999999,"5min":1.71,"15min":1.78},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.244,"5min":1.2450000000000001,"15min":1.2869999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7250000000000001,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.1220000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2829999999999999,"15min":1.3160000000000001},"min":{"1min":0.73899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.591,"5min":1.714,"15min":1.865},"last":1.357},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3500000000000001,"15min":1.3799999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":1.905,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.244}]},{"osd":3,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2889999999999999,"15min":1.302},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.47},{"interface":"front","average":{"1min":1.359,"5min":1.45,"15min":1.4279999999999999},"min":{"1min":1.05,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.6439999999999999,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.5269999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.3520000000000001,"15min":1.339},"min":{"1min":1.0980000000000001,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":1.994,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.387,"15min":1.3420000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.6419999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.409,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.90400000000000003,"15min":0.872},"max":{"1min":1.931,"5min":1.931,"15min":1.956},"last":1.8859999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.516,"15min":1.518},"min":{"1min":1.0840000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.0630000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.5880000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.5660000000000001,"15min":1.5089999999999999},"min":{"1min":1.0149999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1949999999999998},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6479999999999999,"15min":1.6339999999999999},"min":{"1min":1.3620000000000001,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":2.085,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.7829999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56476,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315364,"statfs":{"total":96636764160,"available":96578932736,"internally_reserved":0,"allocated":3158016,"data_stored":1546146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91900000000000004,"15min":0.94999999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.232,"5min":1.52,"15min":1.5580000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.012,"5min":1.177,"15min":1.2230000000000001},"min":{"1min":0.44900000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.5569999999999999,"5min":1.798,"15min":2.0579999999999998},"last":1.4350000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.05,"15min":1.079},"min":{"1min":0.60199999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.496,"5min":1.855,"15min":1.9750000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.004,"5min":1.1080000000000001,"15min":1.1240000000000001},"min":{"1min":0.56299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.456,"5min":2.077,"15min":2.077},"last":1.0109999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.135,"15min":1.2030000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.7330000000000001,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.3720000000000001,"15min":1.444},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6659999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.381}]},{"osd":4,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.494,"15min":1.508},"min":{"1min":0.81899999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":4.1299999999999999,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.728},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.446,"15min":1.4590000000000001},"min":{"1min":0.84299999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.5169999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.462,"15min":1.492},"min":{"1min":0.88200000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.954,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.554,"15min":1.5589999999999999},"min":{"1min":0.89300000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.3420000000000001},"last":1.5580000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.5309999999999999,"15min":1.528},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.1800000000000002,"5min":2.254,"15min":2.4329999999999998},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4850000000000001,"15min":1.4970000000000001},"min":{"1min":0.77300000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8200000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.3089999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:10:49 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.6040000000000001,"15min":1.609},"min":{"1min":0.95899999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.661,"15min":1.677},"min":{"1min":1.002,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.569,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.591}]}]},{"osd":2,"up_from":14,"seq":60129543311,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56472,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315368,"statfs":{"total":96636764160,"available":96578936832,"internally_reserved":0,"allocated":3158016,"data_stored":1545654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.93100000000000005,"15min":0.95299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.5029999999999999,"5min":1.597,"15min":2.0089999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.137},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.431,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.234}]},{"osd":1,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.131,"15min":1.171},"min":{"1min":0.56399999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.72,"5min":1.847,"15min":1.847},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.2010000000000001,"15min":1.242},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5860000000000001,"5min":1.768,"15min":1.9530000000000001},"last":1.3899999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.107,"15min":1.115},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.3009999999999999,"15min":1.3700000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.0619999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.52}]},{"osd":4,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4710000000000001,"15min":1.5},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.837},{"interface":"front","average":{"1min":1.349,"5min":1.464,"15min":1.4870000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.3140000000000001},"last":1.4870000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.522,"15min":1.5529999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.867,"5min":2.1819999999999999,"15min":2.488},"last":1.722},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4550000000000001,"15min":1.4710000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.0059999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.873}]},{"osd":6,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.524,"15min":1.52},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.4079999999999999},"last":2},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.5640000000000001,"15min":1.617},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.5569999999999999},"last":1.9179999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6319999999999999,"15min":1.655},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.7890000000000001},"last":2.1960000000000002},{"interface":"front","average":{"1min":1.496,"5min":1.619,"15min":1.641},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.2599999999999998,"5min":2.2799999999999998,"15min":2.3319999999999999},"last":2.161}]}]},{"osd":1,"up_from":14,"seq":60129543311,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52336,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319504,"statfs":{"total":96636764160,"available":96583172096,"internally_reserved":0,"allocated":3174400,"data_stored":1547300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.93000000000000005,"15min":0.93999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.448,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.109,"15min":1.133},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0129999999999999,"15min":1.0389999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.204,"15min":1.2549999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.92000000000000004}]},{"osd":3,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.4369999999999998},"last":1.038},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.2789999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.544,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.075}]},{"osd":4,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.258,"15min":1.339},"min":{"1min":0.65700000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.534,"5min":1.776,"15min":2.1259999999999999},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.488,"15min":1.5309999999999999},"min":{"1min":0.79100000000000004,"5min":0.748,"15min":0.748},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.2030000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.446,"15min":1.488},"min":{"1min":0.72899999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.0819999999999999,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.659},{"interface":"front","average":{"1min":1.419,"5min":1.385,"15min":1.4790000000000001},"min":{"1min":0.746,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.833}]},{"osd":6,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5800000000000001,"15min":1.633},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.7200000000000002},"last":1.393},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5409999999999999,"15min":1.603},"min":{"1min":0.85099999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0979999999999999,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.5820000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.639,"15min":1.696},"min":{"1min":0.95299999999999996,"5min":0.628,"15min":0.628},"max":{"1min":2.1280000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.524},{"interface":"front","average":{"1min":1.421,"5min":1.52,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0630000000000002,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.9430000000000001}]}]},{"osd":0,"up_from":14,"seq":60129543312,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48736,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323104,"statfs":{"total":96636764160,"available":96586858496,"internally_reserved":0,"allocated":3751936,"data_stored":2135966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.94299999999999995,"15min":0.94399999999999995},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4299999999999999,"5min":1.704,"15min":1.891},"last":1.097},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.054,"15min":1.145},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3009999999999999,"5min":1.659,"15min":1.9299999999999999},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1299999999999999,"15min":1.169},"min":{"1min":0.73299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":1.379},{"interface":"front","average":{"1min":1.157,"5min":1.171,"15min":1.2210000000000001},"min":{"1min":0.68799999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.52,"5min":1.893,"15min":1.893},"last":1.258}]},{"osd":3,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":1.147},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.623,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.032},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.29,"15min":1.3660000000000001},"min":{"1min":0.63200000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.681,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.492}]},{"osd":4,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.302,"15min":1.3640000000000001},"min":{"1min":0.84999999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.625,"5min":1.819,"15min":2.0070000000000001},"last":1.157},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4970000000000001,"15min":1.5189999999999999},"min":{"1min":0.92400000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.919,"5min":2.601,"15min":2.601},"last":1.653}]},{"osd":5,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4910000000000001,"15min":1.504},"min":{"1min":1.129,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.089,"5min":2.2250000000000001,"15min":2.3279999999999998},"last":1.532},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4259999999999999,"15min":1.4790000000000001},"min":{"1min":0.88700000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.319}]},{"osd":6,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.577,"15min":1.613},"min":{"1min":1.1890000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0019999999999998,"5min":2.669,"15min":2.669},"last":1.843},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.552,"15min":1.5920000000000001},"min":{"1min":0.94999999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.327,"5min":2.355,"15min":2.3769999999999998},"last":1.7729999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.6399999999999999,"15min":1.7010000000000001},"min":{"1min":0.88700000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2170000000000001,"5min":2.484,"15min":2.5270000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.591,"15min":1.6359999999999999},"min":{"1min":1.056,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.9490000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.5840000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:11:43.887 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:11:43.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:11:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.158+0000 7fa19ffff640 1 Processor -- start 2023-10-16T14:11:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.158+0000 7fa19ffff640 1 -- start start 2023-10-16T14:11:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa19ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa1a015cb50 0x7fa1a0161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa19ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1a00b1690 0x7fa1a00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa19ffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00b4f00 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa19ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a01616c0 con 0x7fa1a00b1690 2023-10-16T14:11:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa19ffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a00cdbf0 con 0x7fa1a015cb50 2023-10-16T14:11:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa1967fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1a00b1690 0x7fa1a00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa1967fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1a00b1690 0x7fa1a00b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56414/0 (socket says 172.21.15.60:56414) 2023-10-16T14:11:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa1967fc640 1 -- 172.21.15.60:0/3525857951 learned_addr learned my addr 172.21.15.60:0/3525857951 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa19effd640 1 --2- 172.21.15.60:0/3525857951 >> v2:172.21.15.144:3300/0 conn(0x7fa1a015cb50 0x7fa1a0161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa1967fc640 1 -- 172.21.15.60:0/3525857951 >> v2:172.21.15.144:3300/0 conn(0x7fa1a015cb50 msgr2=0x7fa1a0161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.160+0000 7fa1967fc640 1 --2- 172.21.15.60:0/3525857951 >> v2:172.21.15.144:3300/0 conn(0x7fa1a015cb50 0x7fa1a0161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.161+0000 7fa1967fc640 1 -- 172.21.15.60:0/3525857951 >> v1:172.21.15.144:6789/0 conn(0x7fa1a00b4ab0 legacy=0x7fa1a006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.161+0000 7fa1967fc640 1 -- 172.21.15.60:0/3525857951 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a00cde90 con 0x7fa1a00b1690 2023-10-16T14:11:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.162+0000 7fa1967fc640 1 --2- 172.21.15.60:0/3525857951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b1690 0x7fa1a00b4570 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00724b0 tx=0x7fa18802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1617b523048b9d6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.162+0000 7fa19e7fc640 1 -- 172.21.15.60:0/3525857951 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188031750 con 0x7fa1a00b1690 2023-10-16T14:11:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.162+0000 7fa19e7fc640 1 -- 172.21.15.60:0/3525857951 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1880318b0 con 0x7fa1a00b1690 2023-10-16T14:11:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.162+0000 7fa19e7fc640 1 -- 172.21.15.60:0/3525857951 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188031c40 con 0x7fa1a00b1690 2023-10-16T14:11:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 -- 172.21.15.60:0/3525857951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b1690 msgr2=0x7fa1a00b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 --2- 172.21.15.60:0/3525857951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b1690 0x7fa1a00b4570 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00724b0 tx=0x7fa18802b7f0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 -- 172.21.15.60:0/3525857951 shutdown_connections 2023-10-16T14:11:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 --2- 172.21.15.60:0/3525857951 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b1690 0x7fa1a00b4570 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 --2- 172.21.15.60:0/3525857951 >> v2:172.21.15.144:3300/0 conn(0x7fa1a015cb50 0x7fa1a0161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 -- 172.21.15.60:0/3525857951 >> 172.21.15.60:0/3525857951 conn(0x7fa1a00cf570 msgr2=0x7fa1a00cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 -- 172.21.15.60:0/3525857951 shutdown_connections 2023-10-16T14:11:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.163+0000 7fa19ffff640 1 -- 172.21.15.60:0/3525857951 wait complete. 2023-10-16T14:11:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.165+0000 7fa19ffff640 1 Processor -- start 2023-10-16T14:11:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.165+0000 7fa19ffff640 1 -- start start 2023-10-16T14:11:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa19ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1a00b1690 0x7fa1a015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa19ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 0x7fa1a00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa19ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1a015cb50 0x7fa1a01f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa19ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00cbc70 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa19ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a00cb940 con 0x7fa1a00b1690 2023-10-16T14:11:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa19ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a015d780 con 0x7fa1a015cb50 2023-10-16T14:11:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa1967fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 0x7fa1a00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa1967fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 0x7fa1a00c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56424/0 (socket says 172.21.15.60:56424) 2023-10-16T14:11:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.166+0000 7fa1967fc640 1 -- 172.21.15.60:0/2580334532 learned_addr learned my addr 172.21.15.60:0/2580334532 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa19effd640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1a00b1690 0x7fa1a015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa19f7fe640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1a015cb50 0x7fa1a01f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa1967fc640 1 -- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1a015cb50 msgr2=0x7fa1a01f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa1967fc640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1a015cb50 0x7fa1a01f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa1967fc640 1 -- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1a00b1690 msgr2=0x7fa1a015cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa1967fc640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1a00b1690 0x7fa1a015cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa1967fc640 1 -- 172.21.15.60:0/2580334532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1880094d0 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa1967fc640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 0x7fa1a00c9bc0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7fa188000c00 tx=0x7fa1880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188033c80 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.167+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa188033de0 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.168+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18803c640 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.168+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a015da60 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.168+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a015d470 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.169+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a00cbdb0 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.170+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa18803a040 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.171+0000 7fa197fff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa18006b100 0x7fa18006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.171+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1880ab3d0 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.171+0000 7fa19effd640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa18006b100 0x7fa18006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.173+0000 7fa19effd640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa18006b100 0x7fa18006d5c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00b4710 tx=0x7fa1900039b0 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.175+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1a00cbdb0 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.440+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa1a00b4f00 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.445+0000 7fa197fff640 1 -- 172.21.15.60:0/2580334532 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v57) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa1a00b4f00 con 0x7fa1a00b4ab0 2023-10-16T14:11:44.449 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T14:11:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T14:11:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T14:11:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T14:11:44.450 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:11:44.451 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T14:11:44.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa18006b100 msgr2=0x7fa18006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa18006b100 0x7fa18006d5c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00b4710 tx=0x7fa1900039b0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 msgr2=0x7fa1a00c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 0x7fa1a00c9bc0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7fa188000c00 tx=0x7fa1880046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 shutdown_connections 2023-10-16T14:11:44.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa18006b100 0x7fa18006d5c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1a015cb50 0x7fa1a01f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1a00b4ab0 0x7fa1a00c9bc0 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 --2- 172.21.15.60:0/2580334532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1a00b1690 0x7fa1a015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.448+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 >> 172.21.15.60:0/2580334532 conn(0x7fa1a00cf570 msgr2=0x7fa1a00d22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:44.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.449+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 shutdown_connections 2023-10-16T14:11:44.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.449+0000 7fa19ffff640 1 -- 172.21.15.60:0/2580334532 wait complete. 2023-10-16T14:11:44.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:44.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.707+0000 7f0af14ca640 1 Processor -- start 2023-10-16T14:11:44.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.707+0000 7f0af14ca640 1 -- start start 2023-10-16T14:11:44.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0af14ca640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0aec06c040 0x7f0aec06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0af14ca640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aec15f4a0 con 0x7f0aec0cda80 2023-10-16T14:11:44.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0af14ca640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aec06c960 con 0x7f0aec0ad970 2023-10-16T14:11:44.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0af14ca640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aec15b9d0 con 0x7f0aec06c040 2023-10-16T14:11:44.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0aebfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0aec06c040 0x7f0aec06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0af0cc9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0aec0cda80 0x7f0aec15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54968/0 (socket says 172.21.15.60:54968) 2023-10-16T14:11:44.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.709+0000 7f0af0cc9640 1 -- 172.21.15.60:0/1187267413 learned_addr learned my addr 172.21.15.60:0/1187267413 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:44.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.710+0000 7f0aebfff640 1 -- 172.21.15.60:0/1187267413 >> v1:172.21.15.144:6789/0 conn(0x7f0aec0ad970 legacy=0x7f0aec15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:44.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.710+0000 7f0aebfff640 1 -- 172.21.15.60:0/1187267413 >> v1:172.21.15.60:6789/0 conn(0x7f0aec0cda80 legacy=0x7f0aec15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.710+0000 7f0aebfff640 1 -- 172.21.15.60:0/1187267413 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec15bcb0 con 0x7f0aec06c040 2023-10-16T14:11:44.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.710+0000 7f0aeb7fe640 1 -- 172.21.15.60:0/1187267413 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142999302 0 0) 0x7f0aec15f4a0 con 0x7f0aec0cda80 2023-10-16T14:11:44.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.711+0000 7f0aebfff640 1 --2- 172.21.15.60:0/1187267413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec06c040 0x7f0aec06c420 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0097a0 tx=0x7f0adc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b635e2b43e98adc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:44.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.711+0000 7f0aeb7fe640 1 -- 172.21.15.60:0/1187267413 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc031830 con 0x7f0aec06c040 2023-10-16T14:11:44.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.711+0000 7f0aeb7fe640 1 -- 172.21.15.60:0/1187267413 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0adc031990 con 0x7f0aec06c040 2023-10-16T14:11:44.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.711+0000 7f0aeb7fe640 1 -- 172.21.15.60:0/1187267413 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc031d20 con 0x7f0aec06c040 2023-10-16T14:11:44.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.712+0000 7f0af14ca640 1 -- 172.21.15.60:0/1187267413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec06c040 msgr2=0x7f0aec06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.712+0000 7f0af14ca640 1 --2- 172.21.15.60:0/1187267413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec06c040 0x7f0aec06c420 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0097a0 tx=0x7f0adc02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.712+0000 7f0af14ca640 1 -- 172.21.15.60:0/1187267413 shutdown_connections 2023-10-16T14:11:44.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.712+0000 7f0af14ca640 1 --2- 172.21.15.60:0/1187267413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec06c040 0x7f0aec06c420 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.712+0000 7f0af14ca640 1 -- 172.21.15.60:0/1187267413 >> 172.21.15.60:0/1187267413 conn(0x7f0aec0ce6a0 msgr2=0x7f0aec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:44.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.713+0000 7f0af14ca640 1 -- 172.21.15.60:0/1187267413 shutdown_connections 2023-10-16T14:11:44.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.713+0000 7f0af14ca640 1 -- 172.21.15.60:0/1187267413 wait complete. 2023-10-16T14:11:44.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.714+0000 7f0af14ca640 1 Processor -- start 2023-10-16T14:11:44.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.715+0000 7f0af14ca640 1 -- start start 2023-10-16T14:11:44.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.715+0000 7f0af14ca640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aec06c040 0x7f0aec0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.715+0000 7f0af14ca640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aec0ad970 0x7f0aec0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af14ca640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 0x7f0aec151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af14ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec0c25d0 con 0x7f0aec0cda80 2023-10-16T14:11:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af14ca640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec0c6980 con 0x7f0aec06c040 2023-10-16T14:11:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af14ca640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec0b36c0 con 0x7f0aec0ad970 2023-10-16T14:11:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 0x7f0aec151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 0x7f0aec151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56450/0 (socket says 172.21.15.60:56450) 2023-10-16T14:11:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 -- 172.21.15.60:0/2954772708 learned_addr learned my addr 172.21.15.60:0/2954772708 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:44.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0aebfff640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aec06c040 0x7f0aec0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0ae37fe640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aec0ad970 0x7f0aec0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 -- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aec0ad970 msgr2=0x7f0aec0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aec0ad970 0x7f0aec0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 -- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aec06c040 msgr2=0x7f0aec0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aec06c040 0x7f0aec0c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0af0cc9640 1 -- 172.21.15.60:0/2954772708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc009400 con 0x7f0aec0cda80 2023-10-16T14:11:44.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.716+0000 7f0ae37fe640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aec0ad970 0x7f0aec0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:44.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.717+0000 7f0af0cc9640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 0x7f0aec151020 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c810 tx=0x7f0ae400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:44.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.717+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400a640 con 0x7f0aec0cda80 2023-10-16T14:11:44.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.717+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae400a7c0 con 0x7f0aec0cda80 2023-10-16T14:11:44.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.717+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400ab70 con 0x7f0aec0cda80 2023-10-16T14:11:44.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.717+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec151560 con 0x7f0aec0cda80 2023-10-16T14:11:44.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.717+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec0c60e0 con 0x7f0aec0cda80 2023-10-16T14:11:44.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.718+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aec06b330 con 0x7f0aec0cda80 2023-10-16T14:11:44.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.720+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0ae400acd0 con 0x7f0aec0cda80 2023-10-16T14:11:44.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.721+0000 7f0ae97fa640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f0acc06b0b0 0x7f0acc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:44.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.721+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ae4088ea0 con 0x7f0aec0cda80 2023-10-16T14:11:44.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.721+0000 7f0aebfff640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f0acc06b0b0 0x7f0acc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:44.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.722+0000 7f0aebfff640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f0acc06b0b0 0x7f0acc06d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0adc036000 tx=0x7f0adc03c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:44.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.725+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ae4053c80 con 0x7f0aec0cda80 2023-10-16T14:11:44.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.959+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aec0c2710 con 0x7f0aec0cda80 2023-10-16T14:11:44.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.960+0000 7f0ae97fa640 1 -- 172.21.15.60:0/2954772708 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f0ae4018020 con 0x7f0aec0cda80 2023-10-16T14:11:44.964 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:44.964 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8186,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T14:11:44.965 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 128 2023-10-16T14:11:44.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f0acc06b0b0 msgr2=0x7f0acc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f0acc06b0b0 0x7f0acc06d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0adc036000 tx=0x7f0adc03c040 comp rx=0 tx=0).stop 2023-10-16T14:11:44.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 msgr2=0x7f0aec151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:44.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 0x7f0aec151020 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c810 tx=0x7f0ae400ccd0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 shutdown_connections 2023-10-16T14:11:44.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f0acc06b0b0 0x7f0acc06d570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0aec0ad970 0x7f0aec0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0aec0cda80 0x7f0aec151020 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 --2- 172.21.15.60:0/2954772708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aec06c040 0x7f0aec0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:44.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 >> 172.21.15.60:0/2954772708 conn(0x7f0aec0ce6a0 msgr2=0x7f0aec15e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:44.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 shutdown_connections 2023-10-16T14:11:44.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:44.963+0000 7f0af14ca640 1 -- 172.21.15.60:0/2954772708 wait complete. 2023-10-16T14:11:45.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:11:38.244116+0000', 'modified': '2023-10-16T14:11:41.379569+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9409': {'gid': 9409, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.144:6837/296787071', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6835', 'nonce': 296787071}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 296787071}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:11:45.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:11:45.391 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T14:11:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.629+0000 7f9163fff640 1 Processor -- start 2023-10-16T14:11:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.629+0000 7f9163fff640 1 -- start start 2023-10-16T14:11:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9163fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f916406c040 0x7f916406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9163fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f916415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9163fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916406c960 con 0x7f916415ebc0 2023-10-16T14:11:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9163fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916415b9d0 con 0x7f916406c040 2023-10-16T14:11:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9163fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916415bcb0 con 0x7f91640ad970 2023-10-16T14:11:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9162ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f916406c040 0x7f916406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9162ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f916406c040 0x7f916406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56456/0 (socket says 172.21.15.60:56456) 2023-10-16T14:11:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.631+0000 7f9162ffd640 1 -- 172.21.15.60:0/1741542886 learned_addr learned my addr 172.21.15.60:0/1741542886 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.632+0000 7f9162ffd640 1 -- 172.21.15.60:0/1741542886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 msgr2=0x7f916415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.632+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1741542886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f916415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.632+0000 7f9162ffd640 1 -- 172.21.15.60:0/1741542886 >> v1:172.21.15.144:6789/0 conn(0x7f916415ebc0 legacy=0x7f91640cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.632+0000 7f9162ffd640 1 -- 172.21.15.60:0/1741542886 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916415bf90 con 0x7f916406c040 2023-10-16T14:11:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.634+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1741542886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f916406c420 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f91540098a0 tx=0x7f915402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c2c6af81faebce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.634+0000 7f9161ffb640 1 -- 172.21.15.60:0/1741542886 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154030840 con 0x7f916406c040 2023-10-16T14:11:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.634+0000 7f9161ffb640 1 -- 172.21.15.60:0/1741542886 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91540309a0 con 0x7f916406c040 2023-10-16T14:11:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.634+0000 7f9161ffb640 1 -- 172.21.15.60:0/1741542886 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154030d30 con 0x7f916406c040 2023-10-16T14:11:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 -- 172.21.15.60:0/1741542886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 msgr2=0x7f916406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 --2- 172.21.15.60:0/1741542886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f916406c420 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f91540098a0 tx=0x7f915402b980 comp rx=0 tx=0).stop 2023-10-16T14:11:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 -- 172.21.15.60:0/1741542886 shutdown_connections 2023-10-16T14:11:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 --2- 172.21.15.60:0/1741542886 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f916415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 --2- 172.21.15.60:0/1741542886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f916406c420 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 -- 172.21.15.60:0/1741542886 >> 172.21.15.60:0/1741542886 conn(0x7f91640ce6a0 msgr2=0x7f91640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 -- 172.21.15.60:0/1741542886 shutdown_connections 2023-10-16T14:11:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.635+0000 7f9163fff640 1 -- 172.21.15.60:0/1741542886 wait complete. 2023-10-16T14:11:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.637+0000 7f9163fff640 1 Processor -- start 2023-10-16T14:11:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.638+0000 7f9163fff640 1 -- start start 2023-10-16T14:11:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9163fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f9164156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9163fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f91641572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9163fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916415ebc0 0x7f91640af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9163fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164158560 con 0x7f916406c040 2023-10-16T14:11:45.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9163fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164153060 con 0x7f916415ebc0 2023-10-16T14:11:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9163fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91641578e0 con 0x7f91640ad970 2023-10-16T14:11:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9162ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f9164156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9162ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f9164156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56458/0 (socket says 172.21.15.60:56458) 2023-10-16T14:11:45.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.639+0000 7f9162ffd640 1 -- 172.21.15.60:0/2455651593 learned_addr learned my addr 172.21.15.60:0/2455651593 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f91627fc640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f91641572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f91637fe640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916415ebc0 0x7f91640af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9162ffd640 1 -- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 msgr2=0x7f91641572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9162ffd640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f91641572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9162ffd640 1 -- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916415ebc0 msgr2=0x7f91640af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9162ffd640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916415ebc0 0x7f91640af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9162ffd640 1 -- 172.21.15.60:0/2455651593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540094d0 con 0x7f916406c040 2023-10-16T14:11:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9162ffd640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f9164156d70 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f91540098a0 tx=0x7f9154004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f914b7fe640 1 -- 172.21.15.60:0/2455651593 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91540307f0 con 0x7f916406c040 2023-10-16T14:11:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f914b7fe640 1 -- 172.21.15.60:0/2455651593 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9154039d90 con 0x7f916406c040 2023-10-16T14:11:45.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.640+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9164157bc0 con 0x7f916406c040 2023-10-16T14:11:45.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.641+0000 7f914b7fe640 1 -- 172.21.15.60:0/2455651593 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154043630 con 0x7f916406c040 2023-10-16T14:11:45.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.641+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9164158180 con 0x7f916406c040 2023-10-16T14:11:45.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.643+0000 7f914b7fe640 1 -- 172.21.15.60:0/2455651593 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f915404c3b0 con 0x7f916406c040 2023-10-16T14:11:45.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.644+0000 7f914b7fe640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f915006b0b0 0x7f915006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.645+0000 7f914b7fe640 1 -- 172.21.15.60:0/2455651593 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91540ac220 con 0x7f916406c040 2023-10-16T14:11:45.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.645+0000 7f91627fc640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f915006b0b0 0x7f915006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.645+0000 7f916bf9c740 1 Processor -- start 2023-10-16T14:11:45.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.645+0000 7f916bf9c740 1 -- start start 2023-10-16T14:11:45.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f91627fc640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f915006b0b0 0x7f915006d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f916415e4f0 tx=0x7f9158008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9550 0x5603bd2ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 0x5603bd2ec440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f916bf9c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd1dc7a0 con 0x5603bd2bc5b0 2023-10-16T14:11:45.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd22e740 con 0x5603bd2e9550 2023-10-16T14:11:45.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.646+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603bd22be30 con 0x5603bd2e9170 2023-10-16T14:11:45.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 0x5603bd2ec440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 0x5603bd2ec440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56470/0 (socket says 172.21.15.60:56470) 2023-10-16T14:11:45.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 -- 172.21.15.60:0/1319909637 learned_addr learned my addr 172.21.15.60:0/1319909637 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91627fc640 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9550 0x5603bd2ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 -- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 msgr2=0x5603bd2bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 -- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9550 msgr2=0x5603bd2ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9550 0x5603bd2ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.647+0000 7f91637fe640 1 -- 172.21.15.60:0/1319909637 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603bd1eb910 con 0x5603bd2bc5b0 2023-10-16T14:11:45.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.648+0000 7f91637fe640 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 0x5603bd2ec440 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f915c001c30 tx=0x7f915c0020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb59b62390596c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.648+0000 7f9148ff9640 1 -- 172.21.15.60:0/1319909637 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c00dc60 con 0x5603bd2bc5b0 2023-10-16T14:11:45.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.648+0000 7f9148ff9640 1 -- 172.21.15.60:0/1319909637 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c00ddc0 con 0x5603bd2bc5b0 2023-10-16T14:11:45.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.648+0000 7f9148ff9640 1 -- 172.21.15.60:0/1319909637 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c012610 con 0x5603bd2bc5b0 2023-10-16T14:11:45.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.648+0000 7f916bf9c740 1 -- 172.21.15.60:0/1319909637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 msgr2=0x5603bd2ec440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.648+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 0x5603bd2ec440 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f915c001c30 tx=0x7f915c0020f0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 -- 172.21.15.60:0/1319909637 shutdown_connections 2023-10-16T14:11:45.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2bc5b0 0x5603bd2ec440 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9550 0x5603bd2ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1319909637 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 -- 172.21.15.60:0/1319909637 >> 172.21.15.60:0/1319909637 conn(0x5603bd2be380 msgr2=0x5603bd306f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 -- 172.21.15.60:0/1319909637 shutdown_connections 2023-10-16T14:11:45.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.649+0000 7f916bf9c740 1 -- 172.21.15.60:0/1319909637 wait complete. 2023-10-16T14:11:45.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.650+0000 7f916bf9c740 1 -- start start 2023-10-16T14:11:45.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.650+0000 7f916bf9c740 1 Processor -- start 2023-10-16T14:11:45.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd307af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9550 0x5603bd2ea8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 0x5603bd2e2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd227720 con 0x5603bd2e1bc0 2023-10-16T14:11:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd1c5970 con 0x5603bd2e9170 2023-10-16T14:11:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603bd1c5610 con 0x5603bd2e9550 2023-10-16T14:11:45.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f916bf9c740 10 client.0 _ll_register_callbacks cb 0x5603bd1e79d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:11:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 0x5603bd2e2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 0x5603bd2e2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56478/0 (socket says 172.21.15.60:56478) 2023-10-16T14:11:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.651+0000 7f91637fe640 1 -- 172.21.15.60:0/1702902820 learned_addr learned my addr 172.21.15.60:0/1702902820 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd307af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91627fc640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9550 0x5603bd2ea8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91637fe640 1 -- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9550 msgr2=0x5603bd2ea8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91637fe640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9550 0x5603bd2ea8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91637fe640 1 -- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 msgr2=0x5603bd307af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91637fe640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd307af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91637fe640 1 -- 172.21.15.60:0/1702902820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916406c960 con 0x5603bd2e1bc0 2023-10-16T14:11:45.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91627fc640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9550 0x5603bd2ea8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:45.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd307af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:45.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f91637fe640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 0x5603bd2e2020 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f915c000f80 tx=0x7f915c001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9091dae66d2f99f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f9141ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:11:45.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f9141ffb640 1 -- 172.21.15.60:0/1702902820 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c007450 con 0x5603bd2e1bc0 2023-10-16T14:11:45.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.652+0000 7f9141ffb640 1 -- 172.21.15.60:0/1702902820 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f915c0075b0 con 0x5603bd2e1bc0 2023-10-16T14:11:45.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5603bd24e160 con 0x5603bd2e1bc0 2023-10-16T14:11:45.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f9141ffb640 1 -- 172.21.15.60:0/1702902820 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7f915c006e80 con 0x5603bd2e1bc0 2023-10-16T14:11:45.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 10 client.11955 fetch_fsmap learned FSMap version 128 2023-10-16T14:11:45.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x5603bd225da0 con 0x5603bd2e1bc0 2023-10-16T14:11:45.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f9141ffb640 1 -- 172.21.15.60:0/1702902820 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5603bd24e160 con 0x5603bd2e1bc0 2023-10-16T14:11:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f9141ffb640 1 -- 172.21.15.60:0/1702902820 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 128) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f915c0077b0 con 0x5603bd2e1bc0 2023-10-16T14:11:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 10 client.11955 fetch_fsmap finished waiting for FSMap version 128 2023-10-16T14:11:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 10 client.11955 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:11:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 20 client.11955 populate_metadata read hostname 'smithi060' 2023-10-16T14:11:45.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd2e46a0 0x5603bd2edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 4 client.11955 mds_command: new command op to 9409 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:11:45.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.653+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 --> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603bd1f9cc0 con 0x5603bd2e46a0 2023-10-16T14:11:45.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.654+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd2e46a0 0x5603bd2edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.655+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd2e46a0 0x5603bd2edfa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c362c22583e8a83f server_cookie=6fa05f0bd5005837 in_seq=0 out_seq=0 2023-10-16T14:11:45.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.655+0000 7f9141ffb640 10 client.11955 ms_handle_connect on v2:172.21.15.144:6835/296787071 2023-10-16T14:11:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f9141ffb640 1 -- 172.21.15.60:0/1702902820 <== mds.0 v2:172.21.15.144:6835/296787071 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5603bd1f9cc0 con 0x5603bd2e46a0 2023-10-16T14:11:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f9141ffb640 10 client.11955 handle_command_reply: tid=0 2023-10-16T14:11:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 client.11955 shutdown 2023-10-16T14:11:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 msgr2=0x5603bd2e2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 0x5603bd2e2020 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f915c000f80 tx=0x7f915c001250 comp rx=0 tx=0).stop 2023-10-16T14:11:45.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 shutdown_connections 2023-10-16T14:11:45.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd2e46a0 0x5603bd2edfa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9550 0x5603bd2ea8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e1bc0 0x5603bd2e2020 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1702902820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd307af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f9141ffb640 0 client.11955 ms_handle_reset on v2:172.21.15.144:6835/296787071 2023-10-16T14:11:45.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 >> 172.21.15.60:0/1702902820 conn(0x5603bd2bc5b0 msgr2=0x5603bd2bff20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.657+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 shutdown_connections 2023-10-16T14:11:45.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.658+0000 7f916bf9c740 1 -- 172.21.15.60:0/1702902820 wait complete. 2023-10-16T14:11:45.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.658+0000 7f916bf9c740 20 client.11955 trim_cache size 0 max 16384 2023-10-16T14:11:45.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.674+0000 7f916bf9c740 1 Processor -- start 2023-10-16T14:11:45.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.675+0000 7f916bf9c740 1 -- start start 2023-10-16T14:11:45.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.675+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e1bc0 0x5603bd2ebec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.675+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.675+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd2c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f916bf9c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bcf8bf20 con 0x5603bd2e9550 2023-10-16T14:11:45.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd30ae50 con 0x5603bd2e1bc0 2023-10-16T14:11:45.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603bd321150 con 0x5603bd2e9170 2023-10-16T14:11:45.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd2c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd2c01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56482/0 (socket says 172.21.15.60:56482) 2023-10-16T14:11:45.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 -- 172.21.15.60:0/1516369791 learned_addr learned my addr 172.21.15.60:0/1516369791 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91627fc640 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 -- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 msgr2=0x5603bd2bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 -- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e1bc0 msgr2=0x5603bd2ebec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f9162ffd640 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e1bc0 0x5603bd2ebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e1bc0 0x5603bd2ebec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.676+0000 7f91637fe640 1 -- 172.21.15.60:0/1516369791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603bd1c4b80 con 0x5603bd2e9550 2023-10-16T14:11:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.677+0000 7f91637fe640 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd2c01b0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f915c027500 tx=0x7f915c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f81f0a259b37aea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.677+0000 7f91497fa640 1 -- 172.21.15.60:0/1516369791 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c006d60 con 0x5603bd2e9550 2023-10-16T14:11:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.677+0000 7f91497fa640 1 -- 172.21.15.60:0/1516369791 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c006ec0 con 0x5603bd2e9550 2023-10-16T14:11:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.677+0000 7f91497fa640 1 -- 172.21.15.60:0/1516369791 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c003bb0 con 0x5603bd2e9550 2023-10-16T14:11:45.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.677+0000 7f916bf9c740 1 -- 172.21.15.60:0/1516369791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 msgr2=0x5603bd2c01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.677+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd2c01b0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f915c027500 tx=0x7f915c01e040 comp rx=0 tx=0).stop 2023-10-16T14:11:45.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 -- 172.21.15.60:0/1516369791 shutdown_connections 2023-10-16T14:11:45.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e9170 0x5603bd2bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd2c01b0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 --2- 172.21.15.60:0/1516369791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e1bc0 0x5603bd2ebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 -- 172.21.15.60:0/1516369791 >> 172.21.15.60:0/1516369791 conn(0x5603bd2bc5b0 msgr2=0x5603bd30a410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 -- 172.21.15.60:0/1516369791 shutdown_connections 2023-10-16T14:11:45.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 -- 172.21.15.60:0/1516369791 wait complete. 2023-10-16T14:11:45.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.678+0000 7f916bf9c740 1 -- start start 2023-10-16T14:11:45.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.679+0000 7f916bf9c740 1 Processor -- start 2023-10-16T14:11:45.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e1bc0 0x5603bd36e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd318ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd319420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd316200 con 0x5603bd2e9550 2023-10-16T14:11:45.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603bd319960 con 0x5603bd2e9170 2023-10-16T14:11:45.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5603bd2e2ed0 con 0x5603bd2e1bc0 2023-10-16T14:11:45.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f916bf9c740 10 client.0 _ll_register_callbacks cb 0x7f915c001aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:11:45.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd319420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd319420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56488/0 (socket says 172.21.15.60:56488) 2023-10-16T14:11:45.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 -- 172.21.15.60:0/3687160203 learned_addr learned my addr 172.21.15.60:0/3687160203 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91627fc640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd318ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f9162ffd640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e1bc0 0x5603bd36e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 -- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e1bc0 msgr2=0x5603bd36e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e1bc0 0x5603bd36e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 -- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 msgr2=0x5603bd318ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd318ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.680+0000 7f91637fe640 1 -- 172.21.15.60:0/3687160203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603bd225da0 con 0x5603bd2e9550 2023-10-16T14:11:45.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f91637fe640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd319420 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f915c000f80 tx=0x7f915c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182c0e0307e7f050 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f9141ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:11:45.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f9141ffb640 1 -- 172.21.15.60:0/3687160203 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c007250 con 0x5603bd2e9550 2023-10-16T14:11:45.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f9141ffb640 1 -- 172.21.15.60:0/3687160203 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f915c03dad0 con 0x5603bd2e9550 2023-10-16T14:11:45.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5603bd220700 con 0x5603bd2e9550 2023-10-16T14:11:45.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f9141ffb640 1 -- 172.21.15.60:0/3687160203 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7f915c03dc90 con 0x5603bd2e9550 2023-10-16T14:11:45.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f916bf9c740 10 client.11961 fetch_fsmap learned FSMap version 128 2023-10-16T14:11:45.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x5603bd1ea170 con 0x5603bd2e9550 2023-10-16T14:11:45.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.681+0000 7f9141ffb640 1 -- 172.21.15.60:0/3687160203 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5603bd220700 con 0x5603bd2e9550 2023-10-16T14:11:45.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f9141ffb640 1 -- 172.21.15.60:0/3687160203 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 128) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f915c0073b0 con 0x5603bd2e9550 2023-10-16T14:11:45.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f916bf9c740 10 client.11961 fetch_fsmap finished waiting for FSMap version 128 2023-10-16T14:11:45.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f916bf9c740 10 client.11961 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T14:11:45.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f916bf9c740 20 client.11961 populate_metadata read hostname 'smithi060' 2023-10-16T14:11:45.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f916bf9c740 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd315420 0x5603bd315800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f916bf9c740 4 client.11961 mds_command: new command op to 9409 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:11:45.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 --> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5603bd220420 con 0x5603bd315420 2023-10-16T14:11:45.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.682+0000 7f9162ffd640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd315420 0x5603bd315800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.683+0000 7f9162ffd640 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd315420 0x5603bd315800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=892f14acbf32979f server_cookie=56844697b25d8f48 in_seq=0 out_seq=0 2023-10-16T14:11:45.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.683+0000 7f9141ffb640 10 client.11961 ms_handle_connect on v2:172.21.15.144:6835/296787071 2023-10-16T14:11:45.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.684+0000 7f9141ffb640 1 -- 172.21.15.60:0/3687160203 <== mds.0 v2:172.21.15.144:6835/296787071 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5603bd220420 con 0x5603bd315420 2023-10-16T14:11:45.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.684+0000 7f9141ffb640 10 client.11961 handle_command_reply: tid=0 2023-10-16T14:11:45.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.684+0000 7f916bf9c740 1 client.11961 shutdown 2023-10-16T14:11:45.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.684+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 msgr2=0x5603bd319420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.684+0000 7f916bf9c740 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd319420 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f915c000f80 tx=0x7f915c009040 comp rx=0 tx=0).stop 2023-10-16T14:11:45.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 shutdown_connections 2023-10-16T14:11:45.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:6835/296787071,v1:172.21.15.144:6837/296787071] conn(0x5603bd315420 0x5603bd315800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5603bd2e9550 0x5603bd319420 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5603bd2e9170 0x5603bd318ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 --2- 172.21.15.60:0/3687160203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5603bd2e1bc0 0x5603bd36e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f9141ffb640 0 client.11961 ms_handle_reset on v2:172.21.15.144:6835/296787071 2023-10-16T14:11:45.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 >> 172.21.15.60:0/3687160203 conn(0x5603bd2bc5b0 msgr2=0x5603bd2bfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 shutdown_connections 2023-10-16T14:11:45.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 1 -- 172.21.15.60:0/3687160203 wait complete. 2023-10-16T14:11:45.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f916bf9c740 20 client.11961 trim_cache size 0 max 16384 2023-10-16T14:11:45.700 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:11:45.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:11:45.700 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:11:45.700 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 9409, 2023-10-16T14:11:45.700 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:11:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:11:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T14:11:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8420916800000002, 2023-10-16T14:11:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 128, 2023-10-16T14:11:45.701 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 252, 2023-10-16T14:11:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 252, 2023-10-16T14:11:45.702 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.727264473 2023-10-16T14:11:45.702 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:11:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f915006b0b0 msgr2=0x7f915006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f9163fff640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f915006b0b0 0x7f915006d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f916415e4f0 tx=0x7f9158008040 comp rx=0 tx=0).stop 2023-10-16T14:11:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 msgr2=0x7f9164156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.685+0000 7f9163fff640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f9164156d70 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f91540098a0 tx=0x7f9154004730 comp rx=0 tx=0).stop 2023-10-16T14:11:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 shutdown_connections 2023-10-16T14:11:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f915006b0b0 0x7f915006d570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f91640ad970 0x7f91641572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916415ebc0 0x7f91640af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 --2- 172.21.15.60:0/2455651593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f916406c040 0x7f9164156d70 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 >> 172.21.15.60:0/2455651593 conn(0x7f91640ce6a0 msgr2=0x7f916415a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 shutdown_connections 2023-10-16T14:11:45.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.686+0000 7f9163fff640 1 -- 172.21.15.60:0/2455651593 wait complete. 2023-10-16T14:11:45.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:11:45.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.932+0000 7f9e3898e640 1 Processor -- start 2023-10-16T14:11:45.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.932+0000 7f9e3898e640 1 -- start start 2023-10-16T14:11:45.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.934+0000 7f9e3898e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 0x7f9e3406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.934+0000 7f9e3898e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340b4620 con 0x7f9e340712c0 2023-10-16T14:11:45.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.934+0000 7f9e3898e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340b4340 con 0x7f9e3415aad0 2023-10-16T14:11:45.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.934+0000 7f9e3898e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3406f520 con 0x7f9e34070d70 2023-10-16T14:11:45.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.934+0000 7f9e33fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9e340712c0 0x7f9e34070000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54972/0 (socket says 172.21.15.60:54972) 2023-10-16T14:11:45.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.934+0000 7f9e33fff640 1 -- 172.21.15.60:0/3728210192 learned_addr learned my addr 172.21.15.60:0/3728210192 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:45.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 0x7f9e3406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445585725 0 0) 0x7f9e340b4620 con 0x7f9e340712c0 2023-10-16T14:11:45.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e20003520 con 0x7f9e340712c0 2023-10-16T14:11:45.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941337281 0 0) 0x7f9e20003520 con 0x7f9e340712c0 2023-10-16T14:11:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 msgr2=0x7f9e3406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 0x7f9e3406bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 >> v1:172.21.15.144:6789/0 conn(0x7f9e3415aad0 legacy=0x7f9e340cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3406f800 con 0x7f9e340712c0 2023-10-16T14:11:45.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f9e2c0019f0 con 0x7f9e340712c0 2023-10-16T14:11:45.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.935+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 >> v1:172.21.15.60:6789/0 conn(0x7f9e340712c0 legacy=0x7f9e34070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e32ffd640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e200039a0 0x7f9e20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e32ffd640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e32ffd640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 0x7f9e20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3406f520 con 0x7f9e200047c0 2023-10-16T14:11:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2000bac0 con 0x7f9e200039a0 2023-10-16T14:11:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2000bc80 con 0x7f9e34070d70 2023-10-16T14:11:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e33fff640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 0x7f9e20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e337fe640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e200039a0 0x7f9e20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e2bfff640 1 -- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 msgr2=0x7f9e20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e34070d70 0x7f9e20005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e2bfff640 1 -- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e200039a0 msgr2=0x7f9e20003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:45.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e200039a0 0x7f9e20003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e2bfff640 1 -- 172.21.15.60:0/3728210192 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20003520 con 0x7f9e200047c0 2023-10-16T14:11:45.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.936+0000 7f9e337fe640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e200039a0 0x7f9e20003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.938+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e20004c40 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009a90 tx=0x7f9e200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8976ace3792ea8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.938+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c02c680 con 0x7f9e200047c0 2023-10-16T14:11:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.938+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c02c7e0 con 0x7f9e200047c0 2023-10-16T14:11:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.938+0000 7f9e32ffd640 1 -- 172.21.15.60:0/3728210192 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c02cb70 con 0x7f9e200047c0 2023-10-16T14:11:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.938+0000 7f9e3898e640 1 -- 172.21.15.60:0/3728210192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 msgr2=0x7f9e20004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.938+0000 7f9e3898e640 1 --2- 172.21.15.60:0/3728210192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e20004c40 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009a90 tx=0x7f9e200057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.939+0000 7f9e33fff640 1 -- 172.21.15.60:0/3728210192 reap_dead start 2023-10-16T14:11:45.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.939+0000 7f9e3898e640 1 -- 172.21.15.60:0/3728210192 shutdown_connections 2023-10-16T14:11:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.939+0000 7f9e3898e640 1 -- 172.21.15.60:0/3728210192 >> 172.21.15.60:0/3728210192 conn(0x7f9e340b0a90 msgr2=0x7f9e340ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.939+0000 7f9e3898e640 1 -- 172.21.15.60:0/3728210192 shutdown_connections 2023-10-16T14:11:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.939+0000 7f9e3898e640 1 -- 172.21.15.60:0/3728210192 wait complete. 2023-10-16T14:11:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.941+0000 7f9e3898e640 1 Processor -- start 2023-10-16T14:11:45.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.941+0000 7f9e3898e640 1 -- start start 2023-10-16T14:11:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e3898e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e3415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e3898e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e3414d970 0x7f9e340cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e3898e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e340cc7b0 0x7f9e34155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e3898e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340ccc40 con 0x7f9e200047c0 2023-10-16T14:11:45.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e3898e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340b2ef0 con 0x7f9e3414d970 2023-10-16T14:11:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e3898e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340b31d0 con 0x7f9e340cc7b0 2023-10-16T14:11:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e3415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e3415b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56504/0 (socket says 172.21.15.60:56504) 2023-10-16T14:11:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 -- 172.21.15.60:0/2678516713 learned_addr learned my addr 172.21.15.60:0/2678516713 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e33fff640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e340cc7b0 0x7f9e34155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 -- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e340cc7b0 msgr2=0x7f9e34155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e3414d970 0x7f9e340cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e340cc7b0 0x7f9e34155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 -- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e3414d970 msgr2=0x7f9e340cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e3414d970 0x7f9e340cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e337fe640 1 -- 172.21.15.60:0/2678516713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340b34b0 con 0x7f9e200047c0 2023-10-16T14:11:45.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.942+0000 7f9e33fff640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e340cc7b0 0x7f9e34155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:45.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.943+0000 7f9e337fe640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e3415b180 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d7a0 tx=0x7f9e2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.943+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24017070 con 0x7f9e200047c0 2023-10-16T14:11:45.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.943+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e24005070 con 0x7f9e200047c0 2023-10-16T14:11:45.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.943+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24005420 con 0x7f9e200047c0 2023-10-16T14:11:45.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.943+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e341553e0 con 0x7f9e200047c0 2023-10-16T14:11:45.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.943+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3415b6c0 con 0x7f9e200047c0 2023-10-16T14:11:45.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.945+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3404e9b0 con 0x7f9e200047c0 2023-10-16T14:11:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.945+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9e24007030 con 0x7f9e200047c0 2023-10-16T14:11:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.947+0000 7f9e30ff9640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9e1406b0b0 0x7f9e1406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:45.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.947+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e24089510 con 0x7f9e200047c0 2023-10-16T14:11:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.948+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9e1406b0b0 0x7f9e1406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.949+0000 7f9e2bfff640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9e1406b0b0 0x7f9e1406d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c036000 tx=0x7f9e1c030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:45.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:45.951+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e240540b0 con 0x7f9e200047c0 2023-10-16T14:11:46.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.187+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9e340cf520 con 0x7f9e200047c0 2023-10-16T14:11:46.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.188+0000 7f9e30ff9640 1 -- 172.21.15.60:0/2678516713 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v128) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9e240576d0 con 0x7f9e200047c0 2023-10-16T14:11:46.192 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:11:46.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9e1406b0b0 msgr2=0x7f9e1406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9e1406b0b0 0x7f9e1406d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c036000 tx=0x7f9e1c030000 comp rx=0 tx=0).stop 2023-10-16T14:11:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 msgr2=0x7f9e3415b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e3415b180 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d7a0 tx=0x7f9e2400dc60 comp rx=0 tx=0).stop 2023-10-16T14:11:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 shutdown_connections 2023-10-16T14:11:46.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9e1406b0b0 0x7f9e1406d570 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:46.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9e340cc7b0 0x7f9e34155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:46.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e3414d970 0x7f9e340cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:46.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 --2- 172.21.15.60:0/2678516713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e200047c0 0x7f9e3415b180 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:46.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 >> 172.21.15.60:0/2678516713 conn(0x7f9e340b0a90 msgr2=0x7f9e340ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:46.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 shutdown_connections 2023-10-16T14:11:46.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:46.191+0000 7f9e3898e640 1 -- 172.21.15.60:0/2678516713 wait complete. 2023-10-16T14:11:46.215 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:11:46.215 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:11:46.215 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T14:11:46.216 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:11:46.216 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:11:46.216 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:11:46.217 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:11:46.217 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:11:46.217 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:11:46.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.218 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T14:11:46.276 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:11:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:11:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:11:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T14:11:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.277 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:11:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T14:11:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T14:11:46.277 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:11:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T14:11:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T14:11:46.278 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:11:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:11:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T14:11:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:11:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 36481sec preferred_lft 36481sec 2023-10-16T14:11:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T14:11:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.279 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:11:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:11:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T14:11:46.280 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:11:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T14:11:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:11:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T14:11:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.281 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:11:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:11:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T14:11:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.282 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:11:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:11:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T14:11:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:11:46.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.284 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:11:46.387 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T14:11:46.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.391 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T14:11:46.449 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T14:11:46.450 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:11:46.450 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:11:46.450 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:11:46.451 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:11:46.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.452 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T14:11:46.452 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T14:11:46.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:11:46.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:11:46.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:11:46.454 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T14:11:46.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.656 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:11:46.713 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:11:46.713 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:11:46.714 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:11:46.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T14:11:46.821 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:11:46.822 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:11:46.880 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:11:46.880 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:11:46.881 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:11:46.933 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T14:11:46.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:46.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:11:47.085 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:11:47.085 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:11:47.121 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:11:47.121 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:11:47.122 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T14:11:47.122 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T14:11:47.122 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T14:11:47.124 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:11:47.124 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:11:47.124 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:11:47.125 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:11:47.125 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:11:47.128 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:11:47.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:47.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:11:47.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:11:47.221 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:11:47.299 INFO:teuthology.orchestra.run.smithi060.stdout:client9487 2023-10-16T14:11:47.300 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:11:47.300 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:11:47.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T14:11:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.589+0000 7fdceed3c640 1 Processor -- start 2023-10-16T14:11:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.589+0000 7fdceed3c640 1 -- start start 2023-10-16T14:11:47.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdceed3c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdce815ad80 0x7fdce806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdceed3c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdce80cae40 0x7fdce806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdceed3c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce80c8e70 con 0x7fdce80ad7f0 2023-10-16T14:11:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdceed3c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce80b4170 con 0x7fdce815ad80 2023-10-16T14:11:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdceed3c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce80adde0 con 0x7fdce80cae40 2023-10-16T14:11:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdcedd3a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdce815ad80 0x7fdce806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdcedd3a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdce815ad80 0x7fdce806cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56518/0 (socket says 172.21.15.60:56518) 2023-10-16T14:11:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.591+0000 7fdcedd3a640 1 -- 172.21.15.60:0/3846723361 learned_addr learned my addr 172.21.15.60:0/3846723361 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.592+0000 7fdced539640 1 --2- 172.21.15.60:0/3846723361 >> v2:172.21.15.144:3300/0 conn(0x7fdce80cae40 0x7fdce806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:47.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.592+0000 7fdcedd3a640 1 -- 172.21.15.60:0/3846723361 >> v2:172.21.15.144:3300/0 conn(0x7fdce80cae40 msgr2=0x7fdce806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.592+0000 7fdcedd3a640 1 --2- 172.21.15.60:0/3846723361 >> v2:172.21.15.144:3300/0 conn(0x7fdce80cae40 0x7fdce806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.592+0000 7fdcedd3a640 1 -- 172.21.15.60:0/3846723361 >> v1:172.21.15.60:6789/0 conn(0x7fdce80ad7f0 legacy=0x7fdce806b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.592+0000 7fdcedd3a640 1 -- 172.21.15.60:0/3846723361 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce80b4410 con 0x7fdce815ad80 2023-10-16T14:11:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.593+0000 7fdcedd3a640 1 --2- 172.21.15.60:0/3846723361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce806cb70 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fdce815e6f0 tx=0x7fdce402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b667ad5bc7c998 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:47.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.593+0000 7fdcecd38640 1 -- 172.21.15.60:0/3846723361 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4030870 con 0x7fdce815ad80 2023-10-16T14:11:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.593+0000 7fdcecd38640 1 -- 172.21.15.60:0/3846723361 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce40309d0 con 0x7fdce815ad80 2023-10-16T14:11:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.593+0000 7fdcecd38640 1 -- 172.21.15.60:0/3846723361 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4030d60 con 0x7fdce815ad80 2023-10-16T14:11:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 -- 172.21.15.60:0/3846723361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 msgr2=0x7fdce806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 --2- 172.21.15.60:0/3846723361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce806cb70 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fdce815e6f0 tx=0x7fdce402b9b0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 -- 172.21.15.60:0/3846723361 shutdown_connections 2023-10-16T14:11:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 --2- 172.21.15.60:0/3846723361 >> v2:172.21.15.144:3300/0 conn(0x7fdce80cae40 0x7fdce806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 --2- 172.21.15.60:0/3846723361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce806cb70 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 -- 172.21.15.60:0/3846723361 >> 172.21.15.60:0/3846723361 conn(0x7fdce80ce820 msgr2=0x7fdce80cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 -- 172.21.15.60:0/3846723361 shutdown_connections 2023-10-16T14:11:47.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.594+0000 7fdceed3c640 1 -- 172.21.15.60:0/3846723361 wait complete. 2023-10-16T14:11:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.596+0000 7fdceed3c640 1 Processor -- start 2023-10-16T14:11:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.596+0000 7fdceed3c640 1 -- start start 2023-10-16T14:11:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdceed3c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdce80ad7f0 0x7fdce814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdceed3c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce80cae40 0x7fdce80b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:47.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdceed3c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdceed3c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce814ec20 con 0x7fdce815ad80 2023-10-16T14:11:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdceed3c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80cd170 con 0x7fdce80cae40 2023-10-16T14:11:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdceed3c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80cd450 con 0x7fdce80ad7f0 2023-10-16T14:11:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdcee53b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:47.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdcee53b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56524/0 (socket says 172.21.15.60:56524) 2023-10-16T14:11:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.597+0000 7fdcee53b640 1 -- 172.21.15.60:0/1205049335 learned_addr learned my addr 172.21.15.60:0/1205049335 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdced539640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce80cae40 0x7fdce80b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcee53b640 1 -- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdce80ad7f0 msgr2=0x7fdce814e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcedd3a640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdce80ad7f0 0x7fdce814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:47.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcee53b640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdce80ad7f0 0x7fdce814e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcee53b640 1 -- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce80cae40 msgr2=0x7fdce80b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcee53b640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce80cae40 0x7fdce80b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcee53b640 1 -- 172.21.15.60:0/1205049335 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce40094d0 con 0x7fdce815ad80 2023-10-16T14:11:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdced539640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce80cae40 0x7fdce80b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcee53b640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce80ac890 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7fdce000c730 tx=0x7fdce000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:47.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0003b70 con 0x7fdce815ad80 2023-10-16T14:11:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0003cd0 con 0x7fdce815ad80 2023-10-16T14:11:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.598+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce000aa90 con 0x7fdce815ad80 2023-10-16T14:11:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.599+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce80cd730 con 0x7fdce815ad80 2023-10-16T14:11:47.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.599+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce80cdcf0 con 0x7fdce815ad80 2023-10-16T14:11:47.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.600+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce80cdf10 con 0x7fdce815ad80 2023-10-16T14:11:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.601+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdce000abf0 con 0x7fdce815ad80 2023-10-16T14:11:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.602+0000 7fdcde7fc640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdccc06b0e0 0x7fdccc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.602+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdce00898f0 con 0x7fdce815ad80 2023-10-16T14:11:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.602+0000 7fdcedd3a640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdccc06b0e0 0x7fdccc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.604+0000 7fdcedd3a640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdccc06b0e0 0x7fdccc06d5a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdce4036000 tx=0x7fdce403b040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.606+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdce00545a0 con 0x7fdce815ad80 2023-10-16T14:11:47.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.844+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdce804e9b0 con 0x7fdce815ad80 2023-10-16T14:11:47.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.903+0000 7fdcde7fc640 1 -- 172.21.15.60:0/1205049335 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v129) v1 ==== 136+0+0 (secure 0 0 0) 0x7fdce0057bc0 con 0x7fdce815ad80 2023-10-16T14:11:47.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdccc06b0e0 msgr2=0x7fdccc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:47.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdccc06b0e0 0x7fdccc06d5a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdce4036000 tx=0x7fdce403b040 comp rx=0 tx=0).stop 2023-10-16T14:11:47.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 msgr2=0x7fdce80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:47.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce80ac890 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7fdce000c730 tx=0x7fdce000cbf0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 shutdown_connections 2023-10-16T14:11:47.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdccc06b0e0 0x7fdccc06d5a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdce815ad80 0x7fdce80ac890 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdce80cae40 0x7fdce80b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 --2- 172.21.15.60:0/1205049335 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdce80ad7f0 0x7fdce814e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:47.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 >> 172.21.15.60:0/1205049335 conn(0x7fdce80ce820 msgr2=0x7fdce80c9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:47.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.907+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 shutdown_connections 2023-10-16T14:11:47.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:47.908+0000 7fdceed3c640 1 -- 172.21.15.60:0/1205049335 wait complete. 2023-10-16T14:11:47.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.153+0000 7f6e6ec32640 1 Processor -- start 2023-10-16T14:11:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.153+0000 7f6e6ec32640 1 -- start start 2023-10-16T14:11:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6ec32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68070b60 0x7f6e6815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6ec32640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e6806c100 0x7f6e6806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6ec32640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6815aef0 con 0x7f6e68159950 2023-10-16T14:11:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6ec32640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6815e490 con 0x7f6e6806c100 2023-10-16T14:11:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6ec32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6815d910 con 0x7f6e68070b60 2023-10-16T14:11:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6e431640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e6806c100 0x7f6e6806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6e431640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e6806c100 0x7f6e6806c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56536/0 (socket says 172.21.15.60:56536) 2023-10-16T14:11:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.155+0000 7f6e6e431640 1 -- 172.21.15.60:0/2710081835 learned_addr learned my addr 172.21.15.60:0/2710081835 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.156+0000 7f6e6d42f640 1 --2- 172.21.15.60:0/2710081835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68070b60 0x7f6e6815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.156+0000 7f6e6e431640 1 -- 172.21.15.60:0/2710081835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68070b60 msgr2=0x7f6e6815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.156+0000 7f6e6e431640 1 --2- 172.21.15.60:0/2710081835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68070b60 0x7f6e6815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.156+0000 7f6e6e431640 1 -- 172.21.15.60:0/2710081835 >> v1:172.21.15.144:6789/0 conn(0x7f6e68159950 legacy=0x7f6e68070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.156+0000 7f6e6e431640 1 -- 172.21.15.60:0/2710081835 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6815dbf0 con 0x7f6e6806c100 2023-10-16T14:11:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.158+0000 7f6e6e431640 1 --2- 172.21.15.60:0/2710081835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6806c100 0x7f6e6806c550 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009e80 tx=0x7f6e6002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e801e11386bbca41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.158+0000 7f6e6cc2e640 1 -- 172.21.15.60:0/2710081835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6002ca50 con 0x7f6e6806c100 2023-10-16T14:11:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.158+0000 7f6e6cc2e640 1 -- 172.21.15.60:0/2710081835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6002cbb0 con 0x7f6e6806c100 2023-10-16T14:11:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.158+0000 7f6e6cc2e640 1 -- 172.21.15.60:0/2710081835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e600343f0 con 0x7f6e6806c100 2023-10-16T14:11:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 -- 172.21.15.60:0/2710081835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6806c100 msgr2=0x7f6e6806c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/2710081835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6806c100 0x7f6e6806c550 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009e80 tx=0x7f6e6002beb0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 -- 172.21.15.60:0/2710081835 shutdown_connections 2023-10-16T14:11:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/2710081835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e6806c100 0x7f6e6806c550 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/2710081835 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68070b60 0x7f6e6815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 -- 172.21.15.60:0/2710081835 >> 172.21.15.60:0/2710081835 conn(0x7f6e680ce6a0 msgr2=0x7f6e680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 -- 172.21.15.60:0/2710081835 shutdown_connections 2023-10-16T14:11:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.159+0000 7f6e6ec32640 1 -- 172.21.15.60:0/2710081835 wait complete. 2023-10-16T14:11:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.161+0000 7f6e6ec32640 1 Processor -- start 2023-10-16T14:11:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.161+0000 7f6e6ec32640 1 -- start start 2023-10-16T14:11:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6ec32640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6806c100 0x7f6e680c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6ec32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 0x7f6e6814fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6ec32640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68159950 0x7f6e680c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6ec32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e680c62b0 con 0x7f6e68070b60 2023-10-16T14:11:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6ec32640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68166ff0 con 0x7f6e6806c100 2023-10-16T14:11:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6ec32640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e680c6500 con 0x7f6e68159950 2023-10-16T14:11:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6d42f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 0x7f6e6814fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6d42f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 0x7f6e6814fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56538/0 (socket says 172.21.15.60:56538) 2023-10-16T14:11:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6d42f640 1 -- 172.21.15.60:0/1708431696 learned_addr learned my addr 172.21.15.60:0/1708431696 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6dc30640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6806c100 0x7f6e680c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.162+0000 7f6e6e431640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68159950 0x7f6e680c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6d42f640 1 -- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68159950 msgr2=0x7f6e680c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6d42f640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68159950 0x7f6e680c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6d42f640 1 -- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6806c100 msgr2=0x7f6e680c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6d42f640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6806c100 0x7f6e680c28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6d42f640 1 -- 172.21.15.60:0/1708431696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60009b30 con 0x7f6e68070b60 2023-10-16T14:11:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6d42f640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 0x7f6e6814fc50 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800d830 tx=0x7f6e5800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:48.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5800b6c0 con 0x7f6e68070b60 2023-10-16T14:11:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5800b820 con 0x7f6e68070b60 2023-10-16T14:11:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e680c67e0 con 0x7f6e68070b60 2023-10-16T14:11:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5800bbb0 con 0x7f6e68070b60 2023-10-16T14:11:48.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.163+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e68166c80 con 0x7f6e68070b60 2023-10-16T14:11:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.164+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e68166850 con 0x7f6e68070b60 2023-10-16T14:11:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.166+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6e5801e460 con 0x7f6e68070b60 2023-10-16T14:11:48.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.167+0000 7f6e5e7fc640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6e4c06b0b0 0x7f6e4c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.167+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e5808a1c0 con 0x7f6e68070b60 2023-10-16T14:11:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.168+0000 7f6e6dc30640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6e4c06b0b0 0x7f6e4c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.169+0000 7f6e6dc30640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6e4c06b0b0 0x7f6e4c06d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6e64000f60 tx=0x7f6e64009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:48.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.172+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e58054f10 con 0x7f6e68070b60 2023-10-16T14:11:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.369+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e680c2e80 con 0x7f6e68070b60 2023-10-16T14:11:48.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.372+0000 7f6e5e7fc640 1 -- 172.21.15.60:0/1708431696 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+16507 (secure 0 0 0) 0x7f6e58058530 con 0x7f6e68070b60 2023-10-16T14:11:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":252,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:39.765986+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":250,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6e4c06b0b0 msgr2=0x7f6e4c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6e4c06b0b0 0x7f6e4c06d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6e64000f60 tx=0x7f6e64009040 comp rx=0 tx=0).stop 2023-10-16T14:11:48.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 msgr2=0x7f6e6814fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 0x7f6e6814fc50 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f6e5800d830 tx=0x7f6e5800dcf0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 shutdown_connections 2023-10-16T14:11:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6e4c06b0b0 0x7f6e4c06d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6e68159950 0x7f6e680c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e68070b60 0x7f6e6814fc50 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 --2- 172.21.15.60:0/1708431696 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6e6806c100 0x7f6e680c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 >> 172.21.15.60:0/1708431696 conn(0x7f6e680ce6a0 msgr2=0x7f6e68070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.375+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 shutdown_connections 2023-10-16T14:11:48.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.376+0000 7f6e6ec32640 1 -- 172.21.15.60:0/1708431696 wait complete. 2023-10-16T14:11:48.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:11:36.185089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:11:37.268518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:11:48.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:11:48.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.617+0000 7fe91455e640 1 Processor -- start 2023-10-16T14:11:48.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.617+0000 7fe91455e640 1 -- start start 2023-10-16T14:11:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.618+0000 7fe91455e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe90c15ad70 0x7fe90c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.618+0000 7fe91455e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe90c0c9ca0 0x7fe90c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe91455e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe91455e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c0cad80 con 0x7fe90c15ad70 2023-10-16T14:11:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe91455e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c0cb030 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe91455e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c91c0 con 0x7fe90c0ca3b0 2023-10-16T14:11:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe9122fa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe90c15ad70 0x7fe90c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe9122fa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe90c15ad70 0x7fe90c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56540/0 (socket says 172.21.15.60:56540) 2023-10-16T14:11:48.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe9122fa640 1 -- 172.21.15.60:0/3045760030 learned_addr learned my addr 172.21.15.60:0/3045760030 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:48.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe911af9640 1 --2- 172.21.15.60:0/3045760030 >> v2:172.21.15.144:3300/0 conn(0x7fe90c0c9ca0 0x7fe90c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe912afb640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.619+0000 7fe9122fa640 1 -- 172.21.15.60:0/3045760030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 msgr2=0x7fe90c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:48.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.620+0000 7fe9122fa640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.620+0000 7fe9122fa640 1 -- 172.21.15.60:0/3045760030 >> v2:172.21.15.144:3300/0 conn(0x7fe90c0c9ca0 msgr2=0x7fe90c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.620+0000 7fe9122fa640 1 --2- 172.21.15.60:0/3045760030 >> v2:172.21.15.144:3300/0 conn(0x7fe90c0c9ca0 0x7fe90c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.620+0000 7fe9122fa640 1 -- 172.21.15.60:0/3045760030 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c0c94a0 con 0x7fe90c15ad70 2023-10-16T14:11:48.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.620+0000 7fe912afb640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.621+0000 7fe9122fa640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c15ad70 0x7fe90c069da0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0ce870 tx=0x7fe90802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81718a7e84ed8c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.621+0000 7fe9112f8640 1 -- 172.21.15.60:0/3045760030 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908031870 con 0x7fe90c15ad70 2023-10-16T14:11:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.621+0000 7fe9112f8640 1 -- 172.21.15.60:0/3045760030 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080319d0 con 0x7fe90c15ad70 2023-10-16T14:11:48.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.621+0000 7fe9112f8640 1 -- 172.21.15.60:0/3045760030 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908031d60 con 0x7fe90c15ad70 2023-10-16T14:11:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 -- 172.21.15.60:0/3045760030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c15ad70 msgr2=0x7fe90c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c15ad70 0x7fe90c069da0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0ce870 tx=0x7fe90802b910 comp rx=0 tx=0).stop 2023-10-16T14:11:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 -- 172.21.15.60:0/3045760030 shutdown_connections 2023-10-16T14:11:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 --2- 172.21.15.60:0/3045760030 >> v2:172.21.15.144:3300/0 conn(0x7fe90c0c9ca0 0x7fe90c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 --2- 172.21.15.60:0/3045760030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c15ad70 0x7fe90c069da0 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 -- 172.21.15.60:0/3045760030 >> 172.21.15.60:0/3045760030 conn(0x7fe90c0ce9b0 msgr2=0x7fe90c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 -- 172.21.15.60:0/3045760030 shutdown_connections 2023-10-16T14:11:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.622+0000 7fe91455e640 1 -- 172.21.15.60:0/3045760030 wait complete. 2023-10-16T14:11:48.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.624+0000 7fe91455e640 1 Processor -- start 2023-10-16T14:11:48.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.624+0000 7fe91455e640 1 -- start start 2023-10-16T14:11:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe91455e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 0x7fe90c0c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe91455e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c1593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe91455e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c15ad70 0x7fe90c0cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe91455e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0cd500 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe91455e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c1598f0 con 0x7fe90c15ad70 2023-10-16T14:11:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe91455e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0cded0 con 0x7fe90c0ca3b0 2023-10-16T14:11:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe9122fa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 0x7fe90c0c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe9122fa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 0x7fe90c0c6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56552/0 (socket says 172.21.15.60:56552) 2023-10-16T14:11:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.625+0000 7fe9122fa640 1 -- 172.21.15.60:0/1042056841 learned_addr learned my addr 172.21.15.60:0/1042056841 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:48.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe9122fa640 1 -- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 msgr2=0x7fe90c1593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe9122fa640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c1593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe9122fa640 1 -- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c15ad70 msgr2=0x7fe90c0cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe912afb640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c15ad70 0x7fe90c0cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe9122fa640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c15ad70 0x7fe90c0cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe9122fa640 1 -- 172.21.15.60:0/1042056841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9080094d0 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe9122fa640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 0x7fe90c0c6f80 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fe90802b950 tx=0x7fe9080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90803ad30 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080443f0 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080447a0 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.626+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0cc8f0 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.627+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c0cceb0 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.628+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c04e160 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.629+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe90804d450 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.629+0000 7fe902ffd640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe8f006b0e0 0x7fe8f006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:48.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.630+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe90802d040 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.630+0000 7fe911af9640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe8f006b0e0 0x7fe8f006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:48.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.631+0000 7fe911af9640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe8f006b0e0 0x7fe8f006d5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0c4440 tx=0x7fe8fc008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:48.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.634+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe908076d20 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.831+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe90c0b1ca0 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.832+0000 7fe902ffd640 1 -- 172.21.15.60:0/1042056841 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v252) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe908035020 con 0x7fe90c0c9ca0 2023-10-16T14:11:48.836 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:11:48.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe8f006b0e0 msgr2=0x7fe8f006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe8f006b0e0 0x7fe8f006d5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0c4440 tx=0x7fe8fc008040 comp rx=0 tx=0).stop 2023-10-16T14:11:48.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 msgr2=0x7fe90c0c6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:48.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 0x7fe90c0c6f80 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fe90802b950 tx=0x7fe9080046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 shutdown_connections 2023-10-16T14:11:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fe8f006b0e0 0x7fe8f006d5a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe90c0ca3b0 0x7fe90c1593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe90c15ad70 0x7fe90c0cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 --2- 172.21.15.60:0/1042056841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe90c0c9ca0 0x7fe90c0c6f80 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 >> 172.21.15.60:0/1042056841 conn(0x7fe90c0ce9b0 msgr2=0x7fe90c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 shutdown_connections 2023-10-16T14:11:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:48.835+0000 7fe91455e640 1 -- 172.21.15.60:0/1042056841 wait complete. 2023-10-16T14:11:48.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:11:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.088+0000 7f873f2d1640 1 Processor -- start 2023-10-16T14:11:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.088+0000 7f873f2d1640 1 -- start start 2023-10-16T14:11:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.089+0000 7f873f2d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f873815cd70 0x7f87380d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.090+0000 7f873f2d1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873815d150 con 0x7f87380cded0 2023-10-16T14:11:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.090+0000 7f873f2d1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380d1180 con 0x7f87380b8480 2023-10-16T14:11:49.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.090+0000 7f873f2d1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380d1460 con 0x7f873815cd70 2023-10-16T14:11:49.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.090+0000 7f873ead0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f87380cded0 0x7f87380b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48952/0 (socket says 172.21.15.60:48952) 2023-10-16T14:11:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.090+0000 7f873ead0640 1 -- 172.21.15.60:0/3437036475 learned_addr learned my addr 172.21.15.60:0/3437036475 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.090+0000 7f873e2cf640 1 --2- 172.21.15.60:0/3437036475 >> v2:172.21.15.144:3300/0 conn(0x7f873815cd70 0x7f87380d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008284246 0 0) 0x7f873815d150 con 0x7f87380cded0 2023-10-16T14:11:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8724003520 con 0x7f87380cded0 2023-10-16T14:11:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809618163 0 0) 0x7f8724003520 con 0x7f87380cded0 2023-10-16T14:11:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 >> v2:172.21.15.144:3300/0 conn(0x7f873815cd70 msgr2=0x7f87380d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 --2- 172.21.15.60:0/3437036475 >> v2:172.21.15.144:3300/0 conn(0x7f873815cd70 0x7f87380d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 >> v1:172.21.15.144:6789/0 conn(0x7f87380b8480 legacy=0x7f87380cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380d1740 con 0x7f87380cded0 2023-10-16T14:11:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f87300019f0 con 0x7f87380cded0 2023-10-16T14:11:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 >> v1:172.21.15.60:6789/0 conn(0x7f87380cded0 legacy=0x7f87380b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8724003970 0x7f8724003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87240047c0 0x7f8724004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8724005180 0x7f872400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380d1460 con 0x7f87240047c0 2023-10-16T14:11:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872400c020 con 0x7f8724003970 2023-10-16T14:11:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.091+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872400c1e0 con 0x7f8724005180 2023-10-16T14:11:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873dace640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87240047c0 0x7f8724004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873e2cf640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8724003970 0x7f8724003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873dace640 1 -- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8724005180 msgr2=0x7f872400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873dace640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8724005180 0x7f872400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873dace640 1 -- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8724003970 msgr2=0x7f8724003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873dace640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8724003970 0x7f8724003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873dace640 1 -- 172.21.15.60:0/3437036475 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724003520 con 0x7f87240047c0 2023-10-16T14:11:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.092+0000 7f873ead0640 1 -- 172.21.15.60:0/3437036475 reap_dead start 2023-10-16T14:11:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.093+0000 7f873dace640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87240047c0 0x7f8724004c40 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f872c001750 tx=0x7f87240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542b3e0f75433bea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.093+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0052d0 con 0x7f87240047c0 2023-10-16T14:11:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.093+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c02e050 con 0x7f87240047c0 2023-10-16T14:11:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.093+0000 7f873d2cd640 1 -- 172.21.15.60:0/3437036475 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c003830 con 0x7f87240047c0 2023-10-16T14:11:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 -- 172.21.15.60:0/3437036475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87240047c0 msgr2=0x7f8724004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87240047c0 0x7f8724004c40 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f872c001750 tx=0x7f87240057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 -- 172.21.15.60:0/3437036475 shutdown_connections 2023-10-16T14:11:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 --2- 172.21.15.60:0/3437036475 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87240047c0 0x7f8724004c40 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 -- 172.21.15.60:0/3437036475 >> 172.21.15.60:0/3437036475 conn(0x7f87380cf570 msgr2=0x7f87380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 -- 172.21.15.60:0/3437036475 shutdown_connections 2023-10-16T14:11:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.094+0000 7f873f2d1640 1 -- 172.21.15.60:0/3437036475 wait complete. 2023-10-16T14:11:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.096+0000 7f873f2d1640 1 Processor -- start 2023-10-16T14:11:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.096+0000 7f873f2d1640 1 -- start start 2023-10-16T14:11:49.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873f2d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87240047c0 0x7f87380aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873f2d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87381f63a0 0x7f873815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873f2d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 0x7f87380abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873f2d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380ab320 con 0x7f87380ab6f0 2023-10-16T14:11:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873f2d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380ac0b0 con 0x7f87381f63a0 2023-10-16T14:11:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873f2d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873815db90 con 0x7f87240047c0 2023-10-16T14:11:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873ead0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 0x7f87380abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873ead0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 0x7f87380abb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36258/0 (socket says 172.21.15.60:36258) 2023-10-16T14:11:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.097+0000 7f873ead0640 1 -- 172.21.15.60:0/1764459113 learned_addr learned my addr 172.21.15.60:0/1764459113 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873dace640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87381f63a0 0x7f873815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873e2cf640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87240047c0 0x7f87380aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873ead0640 1 -- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87240047c0 msgr2=0x7f87380aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873ead0640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87240047c0 0x7f87380aade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873ead0640 1 -- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87381f63a0 msgr2=0x7f873815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873ead0640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87381f63a0 0x7f873815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873ead0640 1 -- 172.21.15.60:0/1764459113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873815de70 con 0x7f87380ab6f0 2023-10-16T14:11:49.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873dace640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87381f63a0 0x7f873815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:49.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.098+0000 7f873ead0640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 0x7f87380abb70 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f87380cd0f0 tx=0x7f873000caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:49.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.099+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730007110 con 0x7f87380ab6f0 2023-10-16T14:11:49.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.099+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730007290 con 0x7f87380ab6f0 2023-10-16T14:11:49.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.099+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730007640 con 0x7f87380ab6f0 2023-10-16T14:11:49.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.099+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873815e150 con 0x7f87380ab6f0 2023-10-16T14:11:49.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.099+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873815e710 con 0x7f87380ab6f0 2023-10-16T14:11:49.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.100+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873815e930 con 0x7f87380ab6f0 2023-10-16T14:11:49.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.101+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f87300077a0 con 0x7f87380ab6f0 2023-10-16T14:11:49.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.102+0000 7f872affd640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f871c06b0b0 0x7f871c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.102+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8730088d60 con 0x7f87380ab6f0 2023-10-16T14:11:49.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.103+0000 7f873e2cf640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f871c06b0b0 0x7f871c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.104+0000 7f873e2cf640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f871c06b0b0 0x7f871c06d570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87380ad510 tx=0x7f8734008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:49.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.106+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8730053ab0 con 0x7f87380ab6f0 2023-10-16T14:11:49.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.299+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f873804e9b0 con 0x7f87380ab6f0 2023-10-16T14:11:49.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.300+0000 7f872affd640 1 -- 172.21.15.60:0/1764459113 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v252) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87300570d0 con 0x7f87380ab6f0 2023-10-16T14:11:49.304 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:11:49.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f871c06b0b0 msgr2=0x7f871c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f871c06b0b0 0x7f871c06d570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87380ad510 tx=0x7f8734008040 comp rx=0 tx=0).stop 2023-10-16T14:11:49.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 msgr2=0x7f87380abb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 0x7f87380abb70 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f87380cd0f0 tx=0x7f873000caa0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 shutdown_connections 2023-10-16T14:11:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f871c06b0b0 0x7f871c06d570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.302+0000 7f873f2d1640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87380ab6f0 0x7f87380abb70 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.303+0000 7f873f2d1640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87381f63a0 0x7f873815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.303+0000 7f873f2d1640 1 --2- 172.21.15.60:0/1764459113 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f87240047c0 0x7f87380aade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.303+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 >> 172.21.15.60:0/1764459113 conn(0x7f87380cf570 msgr2=0x7f87380b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:49.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.303+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 shutdown_connections 2023-10-16T14:11:49.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.303+0000 7f873f2d1640 1 -- 172.21.15.60:0/1764459113 wait complete. 2023-10-16T14:11:49.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:11:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.548+0000 7efe5e4da640 1 Processor -- start 2023-10-16T14:11:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.548+0000 7efe5e4da640 1 -- start start 2023-10-16T14:11:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.550+0000 7efe5e4da640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe5815aee0 0x7efe580aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.550+0000 7efe5e4da640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5806c140 con 0x7efe5815e9b0 2023-10-16T14:11:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.550+0000 7efe5e4da640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5815b2c0 con 0x7efe580cb810 2023-10-16T14:11:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.550+0000 7efe5e4da640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5806b270 con 0x7efe5815aee0 2023-10-16T14:11:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.550+0000 7efe57fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe5815aee0 0x7efe580aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.551+0000 7efe57fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efe5815aee0 0x7efe580aeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36274/0 (socket says 172.21.15.60:36274) 2023-10-16T14:11:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.551+0000 7efe57fff640 1 -- 172.21.15.60:0/2410626637 learned_addr learned my addr 172.21.15.60:0/2410626637 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.551+0000 7efe57fff640 1 -- 172.21.15.60:0/2410626637 >> v1:172.21.15.144:6789/0 conn(0x7efe580cb810 legacy=0x7efe580cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.551+0000 7efe57fff640 1 -- 172.21.15.60:0/2410626637 >> v1:172.21.15.60:6789/0 conn(0x7efe5815e9b0 legacy=0x7efe5815ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.551+0000 7efe57fff640 1 -- 172.21.15.60:0/2410626637 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5806daf0 con 0x7efe5815aee0 2023-10-16T14:11:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.552+0000 7efe57fff640 1 --2- 172.21.15.60:0/2410626637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580aeef0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7efe48009870 tx=0x7efe4802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a910fde410d7ca84 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.553+0000 7efe56ffd640 1 -- 172.21.15.60:0/2410626637 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48030860 con 0x7efe5815aee0 2023-10-16T14:11:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.553+0000 7efe56ffd640 1 -- 172.21.15.60:0/2410626637 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe480309c0 con 0x7efe5815aee0 2023-10-16T14:11:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.553+0000 7efe56ffd640 1 -- 172.21.15.60:0/2410626637 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48030d50 con 0x7efe5815aee0 2023-10-16T14:11:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.553+0000 7efe5e4da640 1 -- 172.21.15.60:0/2410626637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 msgr2=0x7efe580aeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.553+0000 7efe5e4da640 1 --2- 172.21.15.60:0/2410626637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580aeef0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7efe48009870 tx=0x7efe4802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.554+0000 7efe5e4da640 1 -- 172.21.15.60:0/2410626637 shutdown_connections 2023-10-16T14:11:49.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.554+0000 7efe5e4da640 1 --2- 172.21.15.60:0/2410626637 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580aeef0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.554+0000 7efe5e4da640 1 -- 172.21.15.60:0/2410626637 >> 172.21.15.60:0/2410626637 conn(0x7efe580cfac0 msgr2=0x7efe580cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:49.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.554+0000 7efe5e4da640 1 -- 172.21.15.60:0/2410626637 shutdown_connections 2023-10-16T14:11:49.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.554+0000 7efe5e4da640 1 -- 172.21.15.60:0/2410626637 wait complete. 2023-10-16T14:11:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.556+0000 7efe5e4da640 1 Processor -- start 2023-10-16T14:11:49.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.556+0000 7efe5e4da640 1 -- start start 2023-10-16T14:11:49.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe5e4da640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe580cb810 0x7efe581511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe5e4da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe5e4da640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815e9b0 0x7efe580cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe5e4da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58151720 con 0x7efe5815aee0 2023-10-16T14:11:49.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe5e4da640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580b4800 con 0x7efe5815e9b0 2023-10-16T14:11:49.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe5e4da640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe580b3bf0 con 0x7efe580cb810 2023-10-16T14:11:49.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580cec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36290/0 (socket says 172.21.15.60:36290) 2023-10-16T14:11:49.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.557+0000 7efe577fe640 1 -- 172.21.15.60:0/924826437 learned_addr learned my addr 172.21.15.60:0/924826437 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:49.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe57fff640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe580cb810 0x7efe581511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe577fe640 1 -- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe580cb810 msgr2=0x7efe581511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe5ca77640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815e9b0 0x7efe580cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe577fe640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe580cb810 0x7efe581511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe577fe640 1 -- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815e9b0 msgr2=0x7efe580cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe577fe640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815e9b0 0x7efe580cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe577fe640 1 -- 172.21.15.60:0/924826437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe480094d0 con 0x7efe5815aee0 2023-10-16T14:11:49.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.558+0000 7efe577fe640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580cec70 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00c450 tx=0x7efe4c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:49.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.559+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c0052d0 con 0x7efe5815aee0 2023-10-16T14:11:49.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.559+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4c005430 con 0x7efe5815aee0 2023-10-16T14:11:49.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.559+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c012870 con 0x7efe5815aee0 2023-10-16T14:11:49.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.559+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe580b3ed0 con 0x7efe5815aee0 2023-10-16T14:11:49.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.559+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe580b4490 con 0x7efe5815aee0 2023-10-16T14:11:49.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.560+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe580b4620 con 0x7efe5815aee0 2023-10-16T14:11:49.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.561+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efe4c0129d0 con 0x7efe5815aee0 2023-10-16T14:11:49.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.562+0000 7efe54ff9640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efe4406b0b0 0x7efe4406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:49.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.562+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe4c089580 con 0x7efe5815aee0 2023-10-16T14:11:49.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.563+0000 7efe57fff640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efe4406b0b0 0x7efe4406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:49.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.564+0000 7efe57fff640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efe4406b0b0 0x7efe4406d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe48036000 tx=0x7efe48008030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:49.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.566+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe4c0542d0 con 0x7efe5815aee0 2023-10-16T14:11:49.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.760+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe5804e160 con 0x7efe5815aee0 2023-10-16T14:11:49.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.761+0000 7efe54ff9640 1 -- 172.21.15.60:0/924826437 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v252) v1 ==== 100+0+11 (secure 0 0 0) 0x7efe4c0578f0 con 0x7efe5815aee0 2023-10-16T14:11:49.765 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:11:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.763+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efe4406b0b0 msgr2=0x7efe4406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.763+0000 7efe5e4da640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efe4406b0b0 0x7efe4406d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe48036000 tx=0x7efe48008030 comp rx=0 tx=0).stop 2023-10-16T14:11:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 msgr2=0x7efe580cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580cec70 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00c450 tx=0x7efe4c00c910 comp rx=0 tx=0).stop 2023-10-16T14:11:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 shutdown_connections 2023-10-16T14:11:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efe4406b0b0 0x7efe4406d570 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe5815e9b0 0x7efe580cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe5815aee0 0x7efe580cec70 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 --2- 172.21.15.60:0/924826437 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe580cb810 0x7efe581511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 >> 172.21.15.60:0/924826437 conn(0x7efe580cfac0 msgr2=0x7efe58159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:49.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 shutdown_connections 2023-10-16T14:11:49.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:49.764+0000 7efe5e4da640 1 -- 172.21.15.60:0/924826437 wait complete. 2023-10-16T14:11:49.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:11:50.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.014+0000 7f49b0148640 1 Processor -- start 2023-10-16T14:11:50.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.014+0000 7f49b0148640 1 -- start start 2023-10-16T14:11:50.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.015+0000 7f49b0148640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a8159b20 0x7f49a806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.015+0000 7f49b0148640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a806f7f0 con 0x7f49a80b0b20 2023-10-16T14:11:50.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.015+0000 7f49b0148640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a806c5e0 con 0x7f49a8159f00 2023-10-16T14:11:50.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.015+0000 7f49b0148640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a806c8c0 con 0x7f49a8159b20 2023-10-16T14:11:50.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.016+0000 7f49ae6e5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f49a80b0b20 0x7f49a806fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48974/0 (socket says 172.21.15.60:48974) 2023-10-16T14:11:50.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.016+0000 7f49ae6e5640 1 -- 172.21.15.60:0/3973181060 learned_addr learned my addr 172.21.15.60:0/3973181060 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:50.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.016+0000 7f49adee4640 1 --2- 172.21.15.60:0/3973181060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a8159b20 0x7f49a806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.016+0000 7f49acee2640 1 -- 172.21.15.60:0/3973181060 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212335538 0 0) 0x7f49a806f7f0 con 0x7f49a80b0b20 2023-10-16T14:11:50.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.017+0000 7f49acee2640 1 -- 172.21.15.60:0/3973181060 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4994003520 con 0x7f49a80b0b20 2023-10-16T14:11:50.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.017+0000 7f49acee2640 1 -- 172.21.15.60:0/3973181060 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2796895336 0 0) 0x7f4994003520 con 0x7f49a80b0b20 2023-10-16T14:11:50.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.017+0000 7f49adee4640 1 -- 172.21.15.60:0/3973181060 >> v1:172.21.15.144:6789/0 conn(0x7f49a8159f00 legacy=0x7f49a80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.017+0000 7f49adee4640 1 -- 172.21.15.60:0/3973181060 >> v1:172.21.15.60:6789/0 conn(0x7f49a80b0b20 legacy=0x7f49a806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.017+0000 7f49adee4640 1 -- 172.21.15.60:0/3973181060 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a806cba0 con 0x7f49a8159b20 2023-10-16T14:11:50.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.018+0000 7f49adee4640 1 --2- 172.21.15.60:0/3973181060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a8159b20 0x7f49a806bbe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f49a40097d0 tx=0x7f49a402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3977653db552bc2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:50.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49acee2640 1 -- 172.21.15.60:0/3973181060 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a00019f0 con 0x7f49a8159b20 2023-10-16T14:11:50.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49acee2640 1 -- 172.21.15.60:0/3973181060 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a402e050 con 0x7f49a8159b20 2023-10-16T14:11:50.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49acee2640 1 -- 172.21.15.60:0/3973181060 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4030730 con 0x7f49a8159b20 2023-10-16T14:11:50.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49b0148640 1 -- 172.21.15.60:0/3973181060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a8159b20 msgr2=0x7f49a806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49b0148640 1 --2- 172.21.15.60:0/3973181060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a8159b20 0x7f49a806bbe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f49a40097d0 tx=0x7f49a402b800 comp rx=0 tx=0).stop 2023-10-16T14:11:50.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49b0148640 1 -- 172.21.15.60:0/3973181060 shutdown_connections 2023-10-16T14:11:50.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49b0148640 1 --2- 172.21.15.60:0/3973181060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a8159b20 0x7f49a806bbe0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.019+0000 7f49b0148640 1 -- 172.21.15.60:0/3973181060 >> 172.21.15.60:0/3973181060 conn(0x7f49a80cb410 msgr2=0x7f49a80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:50.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.020+0000 7f49b0148640 1 -- 172.21.15.60:0/3973181060 shutdown_connections 2023-10-16T14:11:50.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.020+0000 7f49b0148640 1 -- 172.21.15.60:0/3973181060 wait complete. 2023-10-16T14:11:50.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.021+0000 7f49b0148640 1 Processor -- start 2023-10-16T14:11:50.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.021+0000 7f49b0148640 1 -- start start 2023-10-16T14:11:50.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.022+0000 7f49b0148640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a80b0b20 0x7f49a80ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.022+0000 7f49b0148640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8159b20 0x7f49a806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.022+0000 7f49b0148640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 0x7f49a80b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.022+0000 7f49b0148640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a81f33e0 con 0x7f49a8159f00 2023-10-16T14:11:50.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.022+0000 7f49b0148640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a81f36c0 con 0x7f49a8159b20 2023-10-16T14:11:50.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.022+0000 7f49b0148640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a81f39a0 con 0x7f49a80b0b20 2023-10-16T14:11:50.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 0x7f49a80b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 0x7f49a80b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36300/0 (socket says 172.21.15.60:36300) 2023-10-16T14:11:50.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 -- 172.21.15.60:0/1436333921 learned_addr learned my addr 172.21.15.60:0/1436333921 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:50.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49adee4640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a80b0b20 0x7f49a80ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 -- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a80b0b20 msgr2=0x7f49a80ad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ad6e3640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8159b20 0x7f49a806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a80b0b20 0x7f49a80ad690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 -- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8159b20 msgr2=0x7f49a806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8159b20 0x7f49a806fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 -- 172.21.15.60:0/1436333921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4009400 con 0x7f49a8159f00 2023-10-16T14:11:50.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49adee4640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a80b0b20 0x7f49a80ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:50.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ad6e3640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8159b20 0x7f49a806fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:50.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.023+0000 7f49ae6e5640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 0x7f49a80b30d0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f49a0001830 tx=0x7f49a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:50.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.024+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0012400 con 0x7f49a8159f00 2023-10-16T14:11:50.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.024+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a0012580 con 0x7f49a8159f00 2023-10-16T14:11:50.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.024+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0012930 con 0x7f49a8159f00 2023-10-16T14:11:50.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.024+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a81f3c80 con 0x7f49a8159f00 2023-10-16T14:11:50.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.024+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a8070430 con 0x7f49a8159f00 2023-10-16T14:11:50.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.025+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a804e160 con 0x7f49a8159f00 2023-10-16T14:11:50.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.026+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f49a0012a90 con 0x7f49a8159f00 2023-10-16T14:11:50.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.027+0000 7f499a7fc640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f498c06b100 0x7f498c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.027+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49a008a290 con 0x7f49a8159f00 2023-10-16T14:11:50.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.028+0000 7f49adee4640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f498c06b100 0x7f498c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.029+0000 7f49adee4640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f498c06b100 0x7f498c06d5c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f49a4034000 tx=0x7f49a4008030 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:50.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.032+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49a0054fc0 con 0x7f49a8159f00 2023-10-16T14:11:50.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.230+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f49a80b10b0 con 0x7f49a8159f00 2023-10-16T14:11:50.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.231+0000 7f499a7fc640 1 -- 172.21.15.60:0/1436333921 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v252) v1 ==== 84+0+209 (secure 0 0 0) 0x7f49a00585e0 con 0x7f49a8159f00 2023-10-16T14:11:50.235 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:50.235 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:11:50.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:50.235 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:11:50.236 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:11:50.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:50.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:50.236 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 30, 2023-10-16T14:11:50.237 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "cephfs_metadata" 2023-10-16T14:11:50.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:50.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:50.237 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 31, 2023-10-16T14:11:50.237 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "cephfs_data" 2023-10-16T14:11:50.238 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:50.238 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:11:50.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.233+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f498c06b100 msgr2=0x7f498c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.233+0000 7f49b0148640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f498c06b100 0x7f498c06d5c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f49a4034000 tx=0x7f49a4008030 comp rx=0 tx=0).stop 2023-10-16T14:11:50.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.233+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 msgr2=0x7f49a80b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.233+0000 7f49b0148640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 0x7f49a80b30d0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f49a0001830 tx=0x7f49a000a430 comp rx=0 tx=0).stop 2023-10-16T14:11:50.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.233+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 shutdown_connections 2023-10-16T14:11:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f498c06b100 0x7f498c06d5c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a8159f00 0x7f49a80b30d0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8159b20 0x7f49a806fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 --2- 172.21.15.60:0/1436333921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f49a80b0b20 0x7f49a80ad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 >> 172.21.15.60:0/1436333921 conn(0x7f49a80cb410 msgr2=0x7f49a80b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:50.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 shutdown_connections 2023-10-16T14:11:50.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.234+0000 7f49b0148640 1 -- 172.21.15.60:0/1436333921 wait complete. 2023-10-16T14:11:50.257 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T14:11:50.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T14:11:50.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.485+0000 7f82c13e2640 1 Processor -- start 2023-10-16T14:11:50.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.485+0000 7f82c13e2640 1 -- start start 2023-10-16T14:11:50.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82c13e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82bc15cf60 0x7f82bc06f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82c13e2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f82bc0b0360 0x7f82bc0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82c13e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0c9620 con 0x7f82bc06f9d0 2023-10-16T14:11:50.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82c13e2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc15d340 con 0x7f82bc0b0360 2023-10-16T14:11:50.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82c13e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0cc860 con 0x7f82bc15cf60 2023-10-16T14:11:50.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82ba7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82bc06f9d0 0x7f82bc0c9ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48982/0 (socket says 172.21.15.60:48982) 2023-10-16T14:11:50.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.487+0000 7f82ba7fc640 1 -- 172.21.15.60:0/562546650 learned_addr learned my addr 172.21.15.60:0/562546650 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:50.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82baffd640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82bc15cf60 0x7f82bc06f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82bb7fe640 1 --2- 172.21.15.60:0/562546650 >> v2:172.21.15.144:3300/0 conn(0x7f82bc0b0360 0x7f82bc0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685840878 0 0) 0x7f82bc0c9620 con 0x7f82bc06f9d0 2023-10-16T14:11:50.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a4003520 con 0x7f82bc06f9d0 2023-10-16T14:11:50.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500010713 0 0) 0x7f82a4003520 con 0x7f82bc06f9d0 2023-10-16T14:11:50.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82bc15cf60 msgr2=0x7f82bc06f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:50.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82bc15cf60 0x7f82bc06f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 >> v2:172.21.15.144:3300/0 conn(0x7f82bc0b0360 msgr2=0x7f82bc0b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:50.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 --2- 172.21.15.60:0/562546650 >> v2:172.21.15.144:3300/0 conn(0x7f82bc0b0360 0x7f82bc0b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0ad4a0 con 0x7f82bc06f9d0 2023-10-16T14:11:50.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f82b0001670 con 0x7f82bc06f9d0 2023-10-16T14:11:50.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 >> v1:172.21.15.60:6789/0 conn(0x7f82bc06f9d0 legacy=0x7f82bc0c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82a40039e0 0x7f82a4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a40047c0 0x7f82a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a4005180 0x7f82a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.488+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0cc860 con 0x7f82a40047c0 2023-10-16T14:11:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a400bef0 con 0x7f82a40039e0 2023-10-16T14:11:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a400c0b0 con 0x7f82a4005180 2023-10-16T14:11:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82baffd640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82bc15cf60 0x7f82bc06f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:50.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82ba7fc640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a40047c0 0x7f82a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82bb7fe640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a4005180 0x7f82a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82ba7fc640 1 -- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a4005180 msgr2=0x7f82a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:50.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82ba7fc640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a4005180 0x7f82a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82ba7fc640 1 -- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82a40039e0 msgr2=0x7f82a4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:50.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82ba7fc640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82a40039e0 0x7f82a4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82ba7fc640 1 -- 172.21.15.60:0/562546650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a4003520 con 0x7f82a40047c0 2023-10-16T14:11:50.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.489+0000 7f82bb7fe640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a4005180 0x7f82a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:50.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.490+0000 7f82ba7fc640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a40047c0 0x7f82a4004c40 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f82b000c210 tx=0x7f82a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62536dec1d5f3752 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:50.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b0005ad0 con 0x7f82a40047c0 2023-10-16T14:11:50.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b0005c30 con 0x7f82a40047c0 2023-10-16T14:11:50.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82b9ffb640 1 -- 172.21.15.60:0/562546650 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b002c8b0 con 0x7f82a40047c0 2023-10-16T14:11:50.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82c13e2640 1 -- 172.21.15.60:0/562546650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a40047c0 msgr2=0x7f82a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82c13e2640 1 --2- 172.21.15.60:0/562546650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a40047c0 0x7f82a4004c40 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f82b000c210 tx=0x7f82a40057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82bb7fe640 1 -- 172.21.15.60:0/562546650 reap_dead start 2023-10-16T14:11:50.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82c13e2640 1 -- 172.21.15.60:0/562546650 shutdown_connections 2023-10-16T14:11:50.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82c13e2640 1 -- 172.21.15.60:0/562546650 >> 172.21.15.60:0/562546650 conn(0x7f82bc0cb360 msgr2=0x7f82bc0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:50.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82c13e2640 1 -- 172.21.15.60:0/562546650 shutdown_connections 2023-10-16T14:11:50.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.491+0000 7f82c13e2640 1 -- 172.21.15.60:0/562546650 wait complete. 2023-10-16T14:11:50.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.493+0000 7f82c13e2640 1 Processor -- start 2023-10-16T14:11:50.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.493+0000 7f82c13e2640 1 -- start start 2023-10-16T14:11:50.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.494+0000 7f82c13e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a40047c0 0x7f82bc0c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.494+0000 7f82c13e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82bc0c6850 0x7f82bc14a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.494+0000 7f82c13e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 0x7f82bc160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.494+0000 7f82c13e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0b31b0 con 0x7f82bc14ad50 2023-10-16T14:11:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.494+0000 7f82c13e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0c5370 con 0x7f82bc0c6850 2023-10-16T14:11:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.494+0000 7f82c13e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0b25a0 con 0x7f82a40047c0 2023-10-16T14:11:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 0x7f82bc160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 0x7f82bc160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36310/0 (socket says 172.21.15.60:36310) 2023-10-16T14:11:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 -- 172.21.15.60:0/1981136754 learned_addr learned my addr 172.21.15.60:0/1981136754 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82baffd640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a40047c0 0x7f82bc0c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82ba7fc640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82bc0c6850 0x7f82bc14a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 -- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a40047c0 msgr2=0x7f82bc0c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a40047c0 0x7f82bc0c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 -- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82bc0c6850 msgr2=0x7f82bc14a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82bc0c6850 0x7f82bc14a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 -- 172.21.15.60:0/1981136754 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0b2880 con 0x7f82bc14ad50 2023-10-16T14:11:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82baffd640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a40047c0 0x7f82bc0c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.495+0000 7f82bb7fe640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 0x7f82bc160830 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f82bc0b0740 tx=0x7f82b400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.496+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4016620 con 0x7f82bc14ad50 2023-10-16T14:11:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.496+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b4016780 con 0x7f82bc14ad50 2023-10-16T14:11:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.496+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4016b10 con 0x7f82bc14ad50 2023-10-16T14:11:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.496+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc0b2b60 con 0x7f82bc14ad50 2023-10-16T14:11:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.496+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc0c8bd0 con 0x7f82bc14ad50 2023-10-16T14:11:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.497+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc04e160 con 0x7f82bc14ad50 2023-10-16T14:11:50.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.498+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f82b4016c70 con 0x7f82bc14ad50 2023-10-16T14:11:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.499+0000 7f82a37fe640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f82a806b0b0 0x7f82a806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.499+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82b408ab70 con 0x7f82bc14ad50 2023-10-16T14:11:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.500+0000 7f82baffd640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f82a806b0b0 0x7f82a806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.501+0000 7f82baffd640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f82a806b0b0 0x7f82a806d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f82ac006fd0 tx=0x7f82ac008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.504+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82b4055710 con 0x7f82bc14ad50 2023-10-16T14:11:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.699+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f82bc0b32f0 con 0x7f82bc14ad50 2023-10-16T14:11:50.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.937+0000 7f82a37fe640 1 -- 172.21.15.60:0/1981136754 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v253) v1 ==== 144+0+0 (secure 0 0 0) 0x7f82b4058d30 con 0x7f82bc14ad50 2023-10-16T14:11:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_metadata' created 2023-10-16T14:11:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.941+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f82a806b0b0 msgr2=0x7f82a806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.941+0000 7f82c13e2640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f82a806b0b0 0x7f82a806d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f82ac006fd0 tx=0x7f82ac008040 comp rx=0 tx=0).stop 2023-10-16T14:11:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.941+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 msgr2=0x7f82bc160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.941+0000 7f82c13e2640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 0x7f82bc160830 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f82bc0b0740 tx=0x7f82b400f520 comp rx=0 tx=0).stop 2023-10-16T14:11:50.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 shutdown_connections 2023-10-16T14:11:50.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f82a806b0b0 0x7f82a806d570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82bc14ad50 0x7f82bc160830 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82bc0c6850 0x7f82bc14a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 --2- 172.21.15.60:0/1981136754 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82a40047c0 0x7f82bc0c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 >> 172.21.15.60:0/1981136754 conn(0x7f82bc0cb360 msgr2=0x7f82bc06bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 shutdown_connections 2023-10-16T14:11:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:50.942+0000 7f82c13e2640 1 -- 172.21.15.60:0/1981136754 wait complete. 2023-10-16T14:11:50.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:11:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.178+0000 7fccf1fb7640 1 Processor -- start 2023-10-16T14:11:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.178+0000 7fccf1fb7640 1 -- start start 2023-10-16T14:11:51.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.180+0000 7fccf1fb7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fccec0ce390 0x7fccec15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:51.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.180+0000 7fccf1fb7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0707e0 con 0x7fccec15ae80 2023-10-16T14:11:51.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.180+0000 7fccf1fb7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec06b7a0 con 0x7fccec15aaa0 2023-10-16T14:11:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.180+0000 7fccf1fb7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccec15ef20 con 0x7fccec0ce390 2023-10-16T14:11:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.180+0000 7fccebfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fccec0ce390 0x7fccec15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.181+0000 7fccebfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fccec0ce390 0x7fccec15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36324/0 (socket says 172.21.15.60:36324) 2023-10-16T14:11:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.181+0000 7fccebfff640 1 -- 172.21.15.60:0/2019457815 learned_addr learned my addr 172.21.15.60:0/2019457815 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.181+0000 7fccebfff640 1 -- 172.21.15.60:0/2019457815 >> v1:172.21.15.144:6789/0 conn(0x7fccec15aaa0 legacy=0x7fccec0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.181+0000 7fccebfff640 1 -- 172.21.15.60:0/2019457815 >> v1:172.21.15.60:6789/0 conn(0x7fccec15ae80 legacy=0x7fccec070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.181+0000 7fccebfff640 1 -- 172.21.15.60:0/2019457815 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec15f200 con 0x7fccec0ce390 2023-10-16T14:11:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.182+0000 7fccebfff640 1 --2- 172.21.15.60:0/2019457815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec0ce390 0x7fccec15e820 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009e50 tx=0x7fcce402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05bfe5a0dc35cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.183+0000 7fccea7fc640 1 -- 172.21.15.60:0/2019457815 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce402cb10 con 0x7fccec0ce390 2023-10-16T14:11:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.183+0000 7fccea7fc640 1 -- 172.21.15.60:0/2019457815 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce402cc70 con 0x7fccec0ce390 2023-10-16T14:11:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.183+0000 7fccea7fc640 1 -- 172.21.15.60:0/2019457815 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce40343f0 con 0x7fccec0ce390 2023-10-16T14:11:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.183+0000 7fccf1fb7640 1 -- 172.21.15.60:0/2019457815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec0ce390 msgr2=0x7fccec15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.183+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/2019457815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec0ce390 0x7fccec15e820 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009e50 tx=0x7fcce402bf70 comp rx=0 tx=0).stop 2023-10-16T14:11:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.184+0000 7fccf1fb7640 1 -- 172.21.15.60:0/2019457815 shutdown_connections 2023-10-16T14:11:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.184+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/2019457815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec0ce390 0x7fccec15e820 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.184+0000 7fccf1fb7640 1 -- 172.21.15.60:0/2019457815 >> 172.21.15.60:0/2019457815 conn(0x7fccec0cb410 msgr2=0x7fccec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.184+0000 7fccf1fb7640 1 -- 172.21.15.60:0/2019457815 shutdown_connections 2023-10-16T14:11:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.184+0000 7fccf1fb7640 1 -- 172.21.15.60:0/2019457815 wait complete. 2023-10-16T14:11:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.186+0000 7fccf1fb7640 1 Processor -- start 2023-10-16T14:11:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.186+0000 7fccf1fb7640 1 -- start start 2023-10-16T14:11:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fccf1fb7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccec0ce390 0x7fccec158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fccf1fb7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 0x7fccec0c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fccf1fb7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccec15ae80 0x7fccec15ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fccf1fb7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec158590 con 0x7fccec15aaa0 2023-10-16T14:11:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fccf1fb7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0c32b0 con 0x7fccec0ce390 2023-10-16T14:11:51.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fccf1fb7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0c3590 con 0x7fccec15ae80 2023-10-16T14:11:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fcceaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 0x7fccec0c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fcceaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 0x7fccec0c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36326/0 (socket says 172.21.15.60:36326) 2023-10-16T14:11:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.187+0000 7fcceaffd640 1 -- 172.21.15.60:0/3641137645 learned_addr learned my addr 172.21.15.60:0/3641137645 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceaffd640 1 -- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccec15ae80 msgr2=0x7fccec15ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceaffd640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccec15ae80 0x7fccec15ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceaffd640 1 -- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccec0ce390 msgr2=0x7fccec158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceb7fe640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccec0ce390 0x7fccec158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceaffd640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccec0ce390 0x7fccec158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceaffd640 1 -- 172.21.15.60:0/3641137645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4009b30 con 0x7fccec15aaa0 2023-10-16T14:11:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fcceaffd640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 0x7fccec0c4310 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fcce000c3c0 tx=0x7fcce000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00052d0 con 0x7fccec15aaa0 2023-10-16T14:11:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0005450 con 0x7fccec15aaa0 2023-10-16T14:11:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0003830 con 0x7fccec15aaa0 2023-10-16T14:11:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec0c3870 con 0x7fccec15aaa0 2023-10-16T14:11:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.188+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec0c3e30 con 0x7fccec15aaa0 2023-10-16T14:11:51.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.189+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec04e160 con 0x7fccec15aaa0 2023-10-16T14:11:51.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.191+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcce0003990 con 0x7fccec15aaa0 2023-10-16T14:11:51.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.192+0000 7fccd3fff640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fccd806b0b0 0x7fccd806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:51.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.192+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fcce00891f0 con 0x7fccec15aaa0 2023-10-16T14:11:51.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.193+0000 7fcceb7fe640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fccd806b0b0 0x7fccd806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:51.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.194+0000 7fcceb7fe640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fccd806b0b0 0x7fccd806d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fccec0715f0 tx=0x7fccdc009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:51.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.196+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcce0053f00 con 0x7fccec15aaa0 2023-10-16T14:11:51.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.395+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fccec0cdc60 con 0x7fccec15aaa0 2023-10-16T14:11:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.948+0000 7fccd3fff640 1 -- 172.21.15.60:0/3641137645 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v254) v1 ==== 176+0+0 (secure 0 0 0) 0x7fcce0057520 con 0x7fccec15aaa0 2023-10-16T14:11:51.953 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_data' created 2023-10-16T14:11:51.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.952+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fccd806b0b0 msgr2=0x7fccd806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.952+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fccd806b0b0 0x7fccd806d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fccec0715f0 tx=0x7fccdc009040 comp rx=0 tx=0).stop 2023-10-16T14:11:51.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.952+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 msgr2=0x7fccec0c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:51.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.952+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 0x7fccec0c4310 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fcce000c3c0 tx=0x7fcce000c880 comp rx=0 tx=0).stop 2023-10-16T14:11:51.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 shutdown_connections 2023-10-16T14:11:51.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fccd806b0b0 0x7fccd806d570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fccec15ae80 0x7fccec15ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccec15aaa0 0x7fccec0c4310 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 --2- 172.21.15.60:0/3641137645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccec0ce390 0x7fccec158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:51.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 >> 172.21.15.60:0/3641137645 conn(0x7fccec0cb410 msgr2=0x7fccec0b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:51.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 shutdown_connections 2023-10-16T14:11:51.959 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:51.953+0000 7fccf1fb7640 1 -- 172.21.15.60:0/3641137645 wait complete. 2023-10-16T14:11:51.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T14:11:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.178+0000 7f2d84d97640 1 Processor -- start 2023-10-16T14:11:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.178+0000 7f2d84d97640 1 -- start start 2023-10-16T14:11:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.179+0000 7f2d84d97640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d80159e90 0x7f2d800ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.179+0000 7f2d84d97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 0x7f2d8006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d84d97640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d80070360 con 0x7f2d80159ab0 2023-10-16T14:11:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d84d97640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8006cb10 con 0x7f2d80159e90 2023-10-16T14:11:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d84d97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d890 con 0x7f2d8006c140 2023-10-16T14:11:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d7ed9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 0x7f2d8006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d7ed9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 0x7f2d8006c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:57266/0 (socket says 172.21.15.60:57266) 2023-10-16T14:11:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2732540461 learned_addr learned my addr 172.21.15.60:0/2732540461 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.180+0000 7f2d77fff640 1 --2- 172.21.15.60:0/2732540461 >> v2:172.21.15.144:3300/0 conn(0x7f2d80159e90 0x7f2d800ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.181+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2732540461 >> v2:172.21.15.144:3300/0 conn(0x7f2d80159e90 msgr2=0x7f2d800ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.181+0000 7f2d7ed9d640 1 --2- 172.21.15.60:0/2732540461 >> v2:172.21.15.144:3300/0 conn(0x7f2d80159e90 0x7f2d800ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.181+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2732540461 >> v1:172.21.15.144:6789/0 conn(0x7f2d80159ab0 legacy=0x7f2d8006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.181+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2732540461 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d800cd1f0 con 0x7f2d8006c140 2023-10-16T14:11:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.183+0000 7f2d7ed9d640 1 --2- 172.21.15.60:0/2732540461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 0x7f2d8006c5d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009e50 tx=0x7f2d7802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da76ccc97125bf53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:52.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.183+0000 7f2d7dd9b640 1 -- 172.21.15.60:0/2732540461 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7802c9e0 con 0x7f2d8006c140 2023-10-16T14:11:52.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.183+0000 7f2d7dd9b640 1 -- 172.21.15.60:0/2732540461 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7802cb40 con 0x7f2d8006c140 2023-10-16T14:11:52.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.183+0000 7f2d7dd9b640 1 -- 172.21.15.60:0/2732540461 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d780343f0 con 0x7f2d8006c140 2023-10-16T14:11:52.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 -- 172.21.15.60:0/2732540461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 msgr2=0x7f2d8006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:52.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2732540461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 0x7f2d8006c5d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009e50 tx=0x7f2d7802be40 comp rx=0 tx=0).stop 2023-10-16T14:11:52.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 -- 172.21.15.60:0/2732540461 shutdown_connections 2023-10-16T14:11:52.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2732540461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d8006c140 0x7f2d8006c5d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2732540461 >> v2:172.21.15.144:3300/0 conn(0x7f2d80159e90 0x7f2d800ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 -- 172.21.15.60:0/2732540461 >> 172.21.15.60:0/2732540461 conn(0x7f2d800ce9b0 msgr2=0x7f2d800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:52.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 -- 172.21.15.60:0/2732540461 shutdown_connections 2023-10-16T14:11:52.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.184+0000 7f2d84d97640 1 -- 172.21.15.60:0/2732540461 wait complete. 2023-10-16T14:11:52.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.186+0000 7f2d84d97640 1 Processor -- start 2023-10-16T14:11:52.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.186+0000 7f2d84d97640 1 -- start start 2023-10-16T14:11:52.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d84d97640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8006c140 0x7f2d80156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:52.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d84d97640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d80159ab0 0x7f2d801f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:52.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d84d97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 0x7f2d800b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:52.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d84d97640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d801f6c10 con 0x7f2d80159e90 2023-10-16T14:11:52.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d84d97640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d800c5320 con 0x7f2d8006c140 2023-10-16T14:11:52.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d84d97640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d800c4710 con 0x7f2d80159ab0 2023-10-16T14:11:52.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d7ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 0x7f2d800b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:52.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d7ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 0x7f2d800b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36332/0 (socket says 172.21.15.60:36332) 2023-10-16T14:11:52.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.187+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2433552275 learned_addr learned my addr 172.21.15.60:0/2433552275 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:52.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d77fff640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d80159ab0 0x7f2d801f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:52.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7e59c640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8006c140 0x7f2d80156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d80159ab0 msgr2=0x7f2d801f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7ed9d640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d80159ab0 0x7f2d801f66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8006c140 msgr2=0x7f2d80156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:52.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7ed9d640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8006c140 0x7f2d80156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7ed9d640 1 -- 172.21.15.60:0/2433552275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78009b30 con 0x7f2d80159e90 2023-10-16T14:11:52.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.188+0000 7f2d7ed9d640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 0x7f2d800b0720 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009e50 tx=0x7f2d78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:52.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.189+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78009510 con 0x7f2d80159e90 2023-10-16T14:11:52.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.189+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7803e450 con 0x7f2d80159e90 2023-10-16T14:11:52.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.189+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7803e800 con 0x7f2d80159e90 2023-10-16T14:11:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.189+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d800c49f0 con 0x7f2d80159e90 2023-10-16T14:11:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.189+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d800c4fb0 con 0x7f2d80159e90 2023-10-16T14:11:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.190+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d800c54b0 con 0x7f2d80159e90 2023-10-16T14:11:52.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.191+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2d7803e960 con 0x7f2d80159e90 2023-10-16T14:11:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.192+0000 7f2d76ffd640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2d6006b0e0 0x7f2d6006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.192+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f2d780ac9c0 con 0x7f2d80159e90 2023-10-16T14:11:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.193+0000 7f2d7e59c640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2d6006b0e0 0x7f2d6006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.194+0000 7f2d7e59c640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2d6006b0e0 0x7f2d6006d5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000f60 tx=0x7f2d70009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:52.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.197+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d78077410 con 0x7f2d80159e90 2023-10-16T14:11:52.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.430+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f2d8004e160 con 0x7f2d80159e90 2023-10-16T14:11:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.975+0000 7f2d76ffd640 1 -- 172.21.15.60:0/2433552275 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v130) v1 ==== 183+0+0 (secure 0 0 0) 0x7f2d7807aa30 con 0x7f2d80159e90 2023-10-16T14:11:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 32 and data pool 33 2023-10-16T14:11:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.983+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2d6006b0e0 msgr2=0x7f2d6006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.983+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2d6006b0e0 0x7f2d6006d5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000f60 tx=0x7f2d70009040 comp rx=0 tx=0).stop 2023-10-16T14:11:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.983+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 msgr2=0x7f2d800b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.983+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 0x7f2d800b0720 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009e50 tx=0x7f2d78004a40 comp rx=0 tx=0).stop 2023-10-16T14:11:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.984+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 shutdown_connections 2023-10-16T14:11:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.984+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f2d6006b0e0 0x7f2d6006d5a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.984+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d80159ab0 0x7f2d801f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.984+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d80159e90 0x7f2d800b0720 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.984+0000 7f2d84d97640 1 --2- 172.21.15.60:0/2433552275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d8006c140 0x7f2d80156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.984+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 >> 172.21.15.60:0/2433552275 conn(0x7f2d800ce9b0 msgr2=0x7f2d800ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.986+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 shutdown_connections 2023-10-16T14:11:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:52.988+0000 7f2d84d97640 1 -- 172.21.15.60:0/2433552275 wait complete. 2023-10-16T14:11:53.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:53.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.208+0000 7f1cf994e640 1 Processor -- start 2023-10-16T14:11:53.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.208+0000 7f1cf994e640 1 -- start start 2023-10-16T14:11:53.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.210+0000 7f1cf994e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cf40b0b20 0x7f1cf41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.210+0000 7f1cf994e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 0x7f1cf415d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.210+0000 7f1cf994e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf415e3e0 con 0x7f1cf40cd0c0 2023-10-16T14:11:53.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.210+0000 7f1cf994e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf40cd4a0 con 0x7f1cf40b0b20 2023-10-16T14:11:53.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.210+0000 7f1cf994e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf415e100 con 0x7f1cf415d3f0 2023-10-16T14:11:53.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.211+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 0x7f1cf415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.211+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 0x7f1cf415d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:57286/0 (socket says 172.21.15.60:57286) 2023-10-16T14:11:53.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.211+0000 7f1cf37fe640 1 -- 172.21.15.60:0/1205658473 learned_addr learned my addr 172.21.15.60:0/1205658473 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:53.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.211+0000 7f1cea7fc640 1 --2- 172.21.15.60:0/1205658473 >> v2:172.21.15.144:3300/0 conn(0x7f1cf40b0b20 0x7f1cf41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.212+0000 7f1cf27fc640 1 -- 172.21.15.60:0/1205658473 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486418358 0 0) 0x7f1cf415e3e0 con 0x7f1cf40cd0c0 2023-10-16T14:11:53.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.212+0000 7f1cf27fc640 1 -- 172.21.15.60:0/1205658473 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce0003520 con 0x7f1cf40cd0c0 2023-10-16T14:11:53.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.212+0000 7f1cf37fe640 1 -- 172.21.15.60:0/1205658473 >> v2:172.21.15.144:3300/0 conn(0x7f1cf40b0b20 msgr2=0x7f1cf41615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:53.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.212+0000 7f1cf37fe640 1 --2- 172.21.15.60:0/1205658473 >> v2:172.21.15.144:3300/0 conn(0x7f1cf40b0b20 0x7f1cf41615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.212+0000 7f1cf37fe640 1 -- 172.21.15.60:0/1205658473 >> v1:172.21.15.144:6789/0 conn(0x7f1cf40cd0c0 legacy=0x7f1cf40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.212+0000 7f1cf37fe640 1 -- 172.21.15.60:0/1205658473 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf4160b10 con 0x7f1cf415d3f0 2023-10-16T14:11:53.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.213+0000 7f1cf37fe640 1 --2- 172.21.15.60:0/1205658473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 0x7f1cf415d820 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009870 tx=0x7f1cec02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ec16b5efddc6050 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:53.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.214+0000 7f1cf27fc640 1 -- 172.21.15.60:0/1205658473 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec031640 con 0x7f1cf415d3f0 2023-10-16T14:11:53.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.214+0000 7f1cf27fc640 1 -- 172.21.15.60:0/1205658473 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec0317a0 con 0x7f1cf415d3f0 2023-10-16T14:11:53.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.214+0000 7f1cf27fc640 1 -- 172.21.15.60:0/1205658473 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec031b30 con 0x7f1cf415d3f0 2023-10-16T14:11:53.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 -- 172.21.15.60:0/1205658473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 msgr2=0x7f1cf415d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 --2- 172.21.15.60:0/1205658473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 0x7f1cf415d820 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009870 tx=0x7f1cec02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 -- 172.21.15.60:0/1205658473 shutdown_connections 2023-10-16T14:11:53.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 --2- 172.21.15.60:0/1205658473 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf415d3f0 0x7f1cf415d820 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 --2- 172.21.15.60:0/1205658473 >> v2:172.21.15.144:3300/0 conn(0x7f1cf40b0b20 0x7f1cf41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 -- 172.21.15.60:0/1205658473 >> 172.21.15.60:0/1205658473 conn(0x7f1cf40cb410 msgr2=0x7f1cf40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:53.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 -- 172.21.15.60:0/1205658473 shutdown_connections 2023-10-16T14:11:53.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.215+0000 7f1cf994e640 1 -- 172.21.15.60:0/1205658473 wait complete. 2023-10-16T14:11:53.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.217+0000 7f1cf994e640 1 Processor -- start 2023-10-16T14:11:53.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.217+0000 7f1cf994e640 1 -- start start 2023-10-16T14:11:53.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf994e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf40b0b20 0x7f1cf4150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf994e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf40cd0c0 0x7f1cf4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf994e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 0x7f1cf41569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf994e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40b4490 con 0x7f1cf415d3f0 2023-10-16T14:11:53.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf994e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf41518a0 con 0x7f1cf40b0b20 2023-10-16T14:11:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf994e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4151b80 con 0x7f1cf40cd0c0 2023-10-16T14:11:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 0x7f1cf41569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.218+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 0x7f1cf41569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36342/0 (socket says 172.21.15.60:36342) 2023-10-16T14:11:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 -- 172.21.15.60:0/317588941 learned_addr learned my addr 172.21.15.60:0/317588941 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 -- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf40cd0c0 msgr2=0x7f1cf4156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf40cd0c0 0x7f1cf4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 -- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf40b0b20 msgr2=0x7f1cf4150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf40b0b20 0x7f1cf4150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 -- 172.21.15.60:0/317588941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0094d0 con 0x7f1cf415d3f0 2023-10-16T14:11:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.219+0000 7f1cf37fe640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 0x7f1cf41569c0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0090b0 tx=0x7f1cec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:53.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.220+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec03a6f0 con 0x7f1cf415d3f0 2023-10-16T14:11:53.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.220+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec03a870 con 0x7f1cf415d3f0 2023-10-16T14:11:53.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.220+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec03ac20 con 0x7f1cf415d3f0 2023-10-16T14:11:53.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.220+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf4151e60 con 0x7f1cf415d3f0 2023-10-16T14:11:53.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.220+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf40b37a0 con 0x7f1cf415d3f0 2023-10-16T14:11:53.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.221+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf4152380 con 0x7f1cf415d3f0 2023-10-16T14:11:53.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.222+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1cec02f050 con 0x7f1cf415d3f0 2023-10-16T14:11:53.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.223+0000 7f1cf0ff9640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f1cd406b0b0 0x7f1cd406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.223+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f1cec0ab460 con 0x7f1cf415d3f0 2023-10-16T14:11:53.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.224+0000 7f1cf2ffd640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f1cd406b0b0 0x7f1cd406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.225+0000 7f1cf2ffd640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f1cd406b0b0 0x7f1cd406d570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40011e0 tx=0x7f1ce40078f0 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:53.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.228+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cf4152380 con 0x7f1cf415d3f0 2023-10-16T14:11:53.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.428+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cf4152380 con 0x7f1cf415d3f0 2023-10-16T14:11:53.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.432+0000 7f1cf0ff9640 1 -- 172.21.15.60:0/317588941 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+20188 (secure 0 0 0) 0x7f1cf4152380 con 0x7f1cf415d3f0 2023-10-16T14:11:53.436 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:53.437 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":255,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:52.432603+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f1cd406b0b0 msgr2=0x7f1cd406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f1cd406b0b0 0x7f1cd406d570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40011e0 tx=0x7f1ce40078f0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 msgr2=0x7f1cf41569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 0x7f1cf41569c0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0090b0 tx=0x7f1cec0046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 shutdown_connections 2023-10-16T14:11:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f1cd406b0b0 0x7f1cd406d570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1cf40cd0c0 0x7f1cf4156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf415d3f0 0x7f1cf41569c0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 --2- 172.21.15.60:0/317588941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cf40b0b20 0x7f1cf4150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 >> 172.21.15.60:0/317588941 conn(0x7f1cf40cb410 msgr2=0x7f1cf40af490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 shutdown_connections 2023-10-16T14:11:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.435+0000 7f1cf994e640 1 -- 172.21.15.60:0/317588941 wait complete. 2023-10-16T14:11:53.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:53.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.681+0000 7f7ddb373640 1 Processor -- start 2023-10-16T14:11:53.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.682+0000 7f7ddb373640 1 -- start start 2023-10-16T14:11:53.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.683+0000 7f7ddb373640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 0x7f7dd415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.683+0000 7f7ddb373640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd415aef0 con 0x7f7dd4070450 2023-10-16T14:11:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.683+0000 7f7ddb373640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd406cc80 con 0x7f7dd406c810 2023-10-16T14:11:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.683+0000 7f7ddb373640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd415e490 con 0x7f7dd4159950 2023-10-16T14:11:53.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.683+0000 7f7dd9b70640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7dd4070450 0x7f7dd406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49002/0 (socket says 172.21.15.60:49002) 2023-10-16T14:11:53.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.683+0000 7f7dd9b70640 1 -- 172.21.15.60:0/2979580410 learned_addr learned my addr 172.21.15.60:0/2979580410 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dda371640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 0x7f7dd415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474178558 0 0) 0x7f7dc8001110 con 0x7f7dd4070450 2023-10-16T14:11:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc0003520 con 0x7f7dd4070450 2023-10-16T14:11:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136322690 0 0) 0x7f7dc0003520 con 0x7f7dd4070450 2023-10-16T14:11:53.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 msgr2=0x7f7dd415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:53.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 0x7f7dd415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 >> v1:172.21.15.144:6789/0 conn(0x7f7dd406c810 legacy=0x7f7dd40c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd415d910 con 0x7f7dd4070450 2023-10-16T14:11:53.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7dc8001800 con 0x7f7dd4070450 2023-10-16T14:11:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dda371640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 0x7f7dd415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:11:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 >> v1:172.21.15.60:6789/0 conn(0x7f7dd4070450 legacy=0x7f7dd406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.684+0000 7f7dd936f640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dc00039a0 0x7f7dc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dd936f640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 0x7f7dc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dd936f640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dc0004d00 0x7f7dc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd415e490 con 0x7f7dc00039a0 2023-10-16T14:11:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc000dd20 con 0x7f7dc0004d00 2023-10-16T14:11:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc000dee0 con 0x7f7dd4159950 2023-10-16T14:11:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dda371640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dc00039a0 0x7f7dc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dda371640 1 -- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 msgr2=0x7f7dc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dda371640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dd4159950 0x7f7dc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dda371640 1 -- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dc0004d00 msgr2=0x7f7dc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dda371640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dc0004d00 0x7f7dc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.685+0000 7f7dda371640 1 -- 172.21.15.60:0/2979580410 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc0003520 con 0x7f7dc00039a0 2023-10-16T14:11:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.686+0000 7f7dda371640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dc00039a0 0x7f7dc0003dc0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00098a0 tx=0x7f7dc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e258378e91f4c8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0004230 con 0x7f7dc00039a0 2023-10-16T14:11:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd0004390 con 0x7f7dc00039a0 2023-10-16T14:11:53.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7dd936f640 1 -- 172.21.15.60:0/2979580410 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0003830 con 0x7f7dc00039a0 2023-10-16T14:11:53.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddb373640 1 -- 172.21.15.60:0/2979580410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dc00039a0 msgr2=0x7f7dc0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2979580410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dc00039a0 0x7f7dc0003dc0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00098a0 tx=0x7f7dc00057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddab72640 1 -- 172.21.15.60:0/2979580410 reap_dead start 2023-10-16T14:11:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddb373640 1 -- 172.21.15.60:0/2979580410 shutdown_connections 2023-10-16T14:11:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddb373640 1 -- 172.21.15.60:0/2979580410 >> 172.21.15.60:0/2979580410 conn(0x7f7dd40ce6a0 msgr2=0x7f7dd40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddb373640 1 -- 172.21.15.60:0/2979580410 shutdown_connections 2023-10-16T14:11:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.687+0000 7f7ddb373640 1 -- 172.21.15.60:0/2979580410 wait complete. 2023-10-16T14:11:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.689+0000 7f7ddb373640 1 Processor -- start 2023-10-16T14:11:53.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.689+0000 7f7ddb373640 1 -- start start 2023-10-16T14:11:53.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7ddb373640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dc00039a0 0x7f7dd40af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7ddb373640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 0x7f7dd40b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7ddb373640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd40af8b0 0x7f7dd40b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7ddb373640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40b2ca0 con 0x7f7dd4157a80 2023-10-16T14:11:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7ddb373640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40b3c80 con 0x7f7dd40af8b0 2023-10-16T14:11:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7ddb373640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40b1d70 con 0x7f7dc00039a0 2023-10-16T14:11:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7dd9b70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 0x7f7dd40b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7dd9b70640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 0x7f7dd40b2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36358/0 (socket says 172.21.15.60:36358) 2023-10-16T14:11:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.690+0000 7f7dd9b70640 1 -- 172.21.15.60:0/2990157660 learned_addr learned my addr 172.21.15.60:0/2990157660 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dd9b70640 1 -- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dc00039a0 msgr2=0x7f7dd40af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dd9b70640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dc00039a0 0x7f7dd40af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dd9b70640 1 -- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd40af8b0 msgr2=0x7f7dd40b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dd9b70640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd40af8b0 0x7f7dd40b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dd9b70640 1 -- 172.21.15.60:0/2990157660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd40c8420 con 0x7f7dd4157a80 2023-10-16T14:11:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dd9b70640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 0x7f7dd40b2220 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8001270 tx=0x7f7dc800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80072d0 con 0x7f7dd4157a80 2023-10-16T14:11:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc8007430 con 0x7f7dd4157a80 2023-10-16T14:11:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80047e0 con 0x7f7dd4157a80 2023-10-16T14:11:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.691+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd40b3ed0 con 0x7f7dd4157a80 2023-10-16T14:11:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.692+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd41586a0 con 0x7f7dd4157a80 2023-10-16T14:11:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.693+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd40b1f40 con 0x7f7dd4157a80 2023-10-16T14:11:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.694+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7dc8004b50 con 0x7f7dd4157a80 2023-10-16T14:11:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.695+0000 7f7dc6ffd640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0e0 0x7f7db806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.695+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f7dc8089110 con 0x7f7dd4157a80 2023-10-16T14:11:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.696+0000 7f7dda371640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0e0 0x7f7db806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.697+0000 7f7dda371640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0e0 0x7f7db806d5a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0035000 tx=0x7f7dd0030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.700+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7dd40b1f40 con 0x7f7dd4157a80 2023-10-16T14:11:53.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.899+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd415c070 con 0x7f7dd4157a80 2023-10-16T14:11:53.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.903+0000 7f7dc6ffd640 1 -- 172.21.15.60:0/2990157660 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+20188 (secure 0 0 0) 0x7f7dd415c070 con 0x7f7dd4157a80 2023-10-16T14:11:53.907 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:53.908 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":255,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:52.432603+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0e0 msgr2=0x7f7db806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0e0 0x7f7db806d5a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0035000 tx=0x7f7dd0030000 comp rx=0 tx=0).stop 2023-10-16T14:11:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 msgr2=0x7f7dd40b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 0x7f7dd40b2220 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8001270 tx=0x7f7dc800cb90 comp rx=0 tx=0).stop 2023-10-16T14:11:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 shutdown_connections 2023-10-16T14:11:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f7db806b0e0 0x7f7db806d5a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7dd40af8b0 0x7f7dd40b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7dd4157a80 0x7f7dd40b2220 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 --2- 172.21.15.60:0/2990157660 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7dc00039a0 0x7f7dd40af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 >> 172.21.15.60:0/2990157660 conn(0x7f7dd40ce6a0 msgr2=0x7f7dd40708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 shutdown_connections 2023-10-16T14:11:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:53.906+0000 7f7ddb373640 1 -- 172.21.15.60:0/2990157660 wait complete. 2023-10-16T14:11:53.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T14:11:54.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.146+0000 7f9dd4ba5640 1 Processor -- start 2023-10-16T14:11:54.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.146+0000 7f9dd4ba5640 1 -- start start 2023-10-16T14:11:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dd4ba5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9dd006d220 0x7f9dd00c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dd4ba5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd0065ab0 0x7f9dd00cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dd4ba5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dd006d600 0x7f9dd00ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dd4ba5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd00acd70 con 0x7f9dd006d600 2023-10-16T14:11:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dd4ba5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd00ac4a0 con 0x7f9dd006d220 2023-10-16T14:11:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dd4ba5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0065f40 con 0x7f9dd0065ab0 2023-10-16T14:11:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.148+0000 7f9dcffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dd006d600 0x7f9dd00ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dd006d600 0x7f9dd00ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36366/0 (socket says 172.21.15.60:36366) 2023-10-16T14:11:54.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 -- 172.21.15.60:0/1124209646 learned_addr learned my addr 172.21.15.60:0/1124209646 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:54.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcf7fe640 1 --2- 172.21.15.60:0/1124209646 >> v2:172.21.15.144:3300/0 conn(0x7f9dd006d220 0x7f9dd00c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:54.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dceffd640 1 --2- 172.21.15.60:0/1124209646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd0065ab0 0x7f9dd00cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:54.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 -- 172.21.15.60:0/1124209646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd0065ab0 msgr2=0x7f9dd00cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:54.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 --2- 172.21.15.60:0/1124209646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd0065ab0 0x7f9dd00cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 -- 172.21.15.60:0/1124209646 >> v2:172.21.15.144:3300/0 conn(0x7f9dd006d220 msgr2=0x7f9dd00c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:54.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 --2- 172.21.15.60:0/1124209646 >> v2:172.21.15.144:3300/0 conn(0x7f9dd006d220 0x7f9dd00c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.149+0000 7f9dcffff640 1 -- 172.21.15.60:0/1124209646 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd00ac720 con 0x7f9dd006d600 2023-10-16T14:11:54.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.150+0000 7f9dcffff640 1 --2- 172.21.15.60:0/1124209646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d600 0x7f9dd00ce0b0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0067320 tx=0x7f9dc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345cf2f033300408 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:54.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dce7fc640 1 -- 172.21.15.60:0/1124209646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc802ca70 con 0x7f9dd006d600 2023-10-16T14:11:54.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dce7fc640 1 -- 172.21.15.60:0/1124209646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc802cbd0 con 0x7f9dd006d600 2023-10-16T14:11:54.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dce7fc640 1 -- 172.21.15.60:0/1124209646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80343f0 con 0x7f9dd006d600 2023-10-16T14:11:54.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/1124209646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d600 msgr2=0x7f9dd00ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:54.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/1124209646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d600 0x7f9dd00ce0b0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0067320 tx=0x7f9dc802bed0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/1124209646 shutdown_connections 2023-10-16T14:11:54.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/1124209646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d600 0x7f9dd00ce0b0 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/1124209646 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd0065ab0 0x7f9dd00cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/1124209646 >> v2:172.21.15.144:3300/0 conn(0x7f9dd006d220 0x7f9dd00c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.151+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/1124209646 >> 172.21.15.60:0/1124209646 conn(0x7f9dd00b0a80 msgr2=0x7f9dd00b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:54.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.152+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/1124209646 shutdown_connections 2023-10-16T14:11:54.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.152+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/1124209646 wait complete. 2023-10-16T14:11:54.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.154+0000 7f9dd4ba5640 1 Processor -- start 2023-10-16T14:11:54.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.154+0000 7f9dd4ba5640 1 -- start start 2023-10-16T14:11:54.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.155+0000 7f9dd4ba5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 0x7f9dd006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.155+0000 7f9dd4ba5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd006d600 0x7f9dd01f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.155+0000 7f9dd4ba5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01f2010 0x7f9dd015f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.155+0000 7f9dd4ba5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd015e940 con 0x7f9dd006d220 2023-10-16T14:11:54.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.155+0000 7f9dd4ba5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd015ec20 con 0x7f9dd01f2010 2023-10-16T14:11:54.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.155+0000 7f9dd4ba5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd01f6400 con 0x7f9dd006d600 2023-10-16T14:11:54.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 0x7f9dd006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:54.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 0x7f9dd006a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36376/0 (socket says 172.21.15.60:36376) 2023-10-16T14:11:54.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 -- 172.21.15.60:0/2056698912 learned_addr learned my addr 172.21.15.60:0/2056698912 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:54.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 -- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd006d600 msgr2=0x7f9dd01f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:54.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd006d600 0x7f9dd01f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcffff640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01f2010 0x7f9dd015f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 -- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01f2010 msgr2=0x7f9dd015f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01f2010 0x7f9dd015f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 -- 172.21.15.60:0/2056698912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8009b30 con 0x7f9dd006d220 2023-10-16T14:11:54.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.156+0000 7f9dcf7fe640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 0x7f9dd006a7b0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000d800 tx=0x7f9dc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:54.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.157+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000b6c0 con 0x7f9dd006d220 2023-10-16T14:11:54.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.157+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc000b820 con 0x7f9dd006d220 2023-10-16T14:11:54.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.157+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000bbb0 con 0x7f9dd006d220 2023-10-16T14:11:54.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.157+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd01f66e0 con 0x7f9dd006d220 2023-10-16T14:11:54.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.157+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd01f24a0 con 0x7f9dd006d220 2023-10-16T14:11:54.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.158+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd004e9b0 con 0x7f9dd006d220 2023-10-16T14:11:54.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.159+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9dc000bd10 con 0x7f9dd006d220 2023-10-16T14:11:54.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.160+0000 7f9db7fff640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9dbc06b0b0 0x7f9dbc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:54.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.160+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f9dc008a2e0 con 0x7f9dd006d220 2023-10-16T14:11:54.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.161+0000 7f9dceffd640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9dbc06b0b0 0x7f9dbc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:54.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.162+0000 7f9dceffd640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9dbc06b0b0 0x7f9dbc06d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4006fd0 tx=0x7f9dc4008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:54.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.164+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9dc0054cd0 con 0x7f9dd006d220 2023-10-16T14:11:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:54.399+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9dd006d000 con 0x7f9dd006d220 2023-10-16T14:11:55.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.000+0000 7f9db7fff640 1 -- 172.21.15.60:0/2056698912 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v133) v1 ==== 121+0+0 (secure 0 0 0) 0x7f9dc00582f0 con 0x7f9dd006d220 2023-10-16T14:11:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9dbc06b0b0 msgr2=0x7f9dbc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9dbc06b0b0 0x7f9dbc06d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4006fd0 tx=0x7f9dc4008040 comp rx=0 tx=0).stop 2023-10-16T14:11:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 msgr2=0x7f9dd006a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 0x7f9dd006a7b0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000d800 tx=0x7f9dc000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 shutdown_connections 2023-10-16T14:11:55.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f9dbc06b0b0 0x7f9dbc06d570 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9dd006d600 0x7f9dd01f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01f2010 0x7f9dd015f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 --2- 172.21.15.60:0/2056698912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd006d220 0x7f9dd006a7b0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 >> 172.21.15.60:0/2056698912 conn(0x7f9dd00b0a80 msgr2=0x7f9dd0070400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:55.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 shutdown_connections 2023-10-16T14:11:55.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.004+0000 7f9dd4ba5640 1 -- 172.21.15.60:0/2056698912 wait complete. 2023-10-16T14:11:55.029 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:11:55.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T14:11:55.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.259+0000 7f6f6b59e640 1 Processor -- start 2023-10-16T14:11:55.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.260+0000 7f6f6b59e640 1 -- start start 2023-10-16T14:11:55.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.261+0000 7f6f6b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f6c070b60 0x7f6f6c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:55.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.261+0000 7f6f6b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f6c06c100 0x7f6f6c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:55.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.261+0000 7f6f6b59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6c15aef0 con 0x7f6f6c159950 2023-10-16T14:11:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.261+0000 7f6f6b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6c15e490 con 0x7f6f6c070b60 2023-10-16T14:11:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.261+0000 7f6f6b59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6c15d910 con 0x7f6f6c06c100 2023-10-16T14:11:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f6c070b60 0x7f6f6c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f6c070b60 0x7f6f6c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36382/0 (socket says 172.21.15.60:36382) 2023-10-16T14:11:55.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 -- 172.21.15.60:0/3882389066 learned_addr learned my addr 172.21.15.60:0/3882389066 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 -- 172.21.15.60:0/3882389066 >> v2:172.21.15.144:3300/0 conn(0x7f6f6c06c100 msgr2=0x7f6f6c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 --2- 172.21.15.60:0/3882389066 >> v2:172.21.15.144:3300/0 conn(0x7f6f6c06c100 0x7f6f6c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 -- 172.21.15.60:0/3882389066 >> v1:172.21.15.144:6789/0 conn(0x7f6f6c159950 legacy=0x7f6f6c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.262+0000 7f6f63fff640 1 -- 172.21.15.60:0/3882389066 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c15dbf0 con 0x7f6f6c070b60 2023-10-16T14:11:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.263+0000 7f6f63fff640 1 --2- 172.21.15.60:0/3882389066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c15a9b0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c0ade70 tx=0x7f6f5402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017281800363118 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f69d9b640 1 -- 172.21.15.60:0/3882389066 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f540317e0 con 0x7f6f6c070b60 2023-10-16T14:11:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f69d9b640 1 -- 172.21.15.60:0/3882389066 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f54031940 con 0x7f6f6c070b60 2023-10-16T14:11:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f69d9b640 1 -- 172.21.15.60:0/3882389066 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54031cd0 con 0x7f6f6c070b60 2023-10-16T14:11:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f6b59e640 1 -- 172.21.15.60:0/3882389066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 msgr2=0x7f6f6c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/3882389066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c15a9b0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c0ade70 tx=0x7f6f5402b880 comp rx=0 tx=0).stop 2023-10-16T14:11:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f6b59e640 1 -- 172.21.15.60:0/3882389066 shutdown_connections 2023-10-16T14:11:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/3882389066 >> v2:172.21.15.144:3300/0 conn(0x7f6f6c06c100 0x7f6f6c06c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/3882389066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c15a9b0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.264+0000 7f6f6b59e640 1 -- 172.21.15.60:0/3882389066 >> 172.21.15.60:0/3882389066 conn(0x7f6f6c0ce6a0 msgr2=0x7f6f6c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.265+0000 7f6f6b59e640 1 -- 172.21.15.60:0/3882389066 shutdown_connections 2023-10-16T14:11:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.265+0000 7f6f6b59e640 1 -- 172.21.15.60:0/3882389066 wait complete. 2023-10-16T14:11:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.266+0000 7f6f6b59e640 1 Processor -- start 2023-10-16T14:11:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.267+0000 7f6f6b59e640 1 -- start start 2023-10-16T14:11:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.267+0000 7f6f6b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f6c06c100 0x7f6f6c14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.267+0000 7f6f6b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.267+0000 7f6f6b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6c159950 0x7f6f6c151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f6b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c14b490 con 0x7f6f6c070b60 2023-10-16T14:11:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f6b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1364f0 con 0x7f6f6c06c100 2023-10-16T14:11:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f6b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1367d0 con 0x7f6f6c159950 2023-10-16T14:11:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36386/0 (socket says 172.21.15.60:36386) 2023-10-16T14:11:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 -- 172.21.15.60:0/1695965720 learned_addr learned my addr 172.21.15.60:0/1695965720 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 -- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6c159950 msgr2=0x7f6f6c151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:55.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6c159950 0x7f6f6c151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 -- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f6c06c100 msgr2=0x7f6f6c14af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f6a59c640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f6c06c100 0x7f6f6c14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f6c06c100 0x7f6f6c14af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f63fff640 1 -- 172.21.15.60:0/1695965720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f540094d0 con 0x7f6f6c070b60 2023-10-16T14:11:55.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.268+0000 7f6f6a59c640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f6c06c100 0x7f6f6c14af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:55.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.269+0000 7f6f63fff640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c154290 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f6f5402b8c0 tx=0x7f6f540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:55.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.269+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54032940 con 0x7f6f6c070b60 2023-10-16T14:11:55.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.269+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f54032aa0 con 0x7f6f6c070b60 2023-10-16T14:11:55.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.269+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54032e50 con 0x7f6f6c070b60 2023-10-16T14:11:55.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.269+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c136ab0 con 0x7f6f6c070b60 2023-10-16T14:11:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.269+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c137070 con 0x7f6f6c070b60 2023-10-16T14:11:55.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.270+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c04e9b0 con 0x7f6f6c070b60 2023-10-16T14:11:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.271+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6f5404e450 con 0x7f6f6c070b60 2023-10-16T14:11:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.272+0000 7f6f62ffd640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6f4c06b0b0 0x7f6f4c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.272+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f6f540abf90 con 0x7f6f6c070b60 2023-10-16T14:11:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.273+0000 7f6f6a59c640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6f4c06b0b0 0x7f6f4c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:55.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.274+0000 7f6f6a59c640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6f4c06b0b0 0x7f6f4c06d570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00a810 tx=0x7f6f5c008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:55.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.276+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f54076860 con 0x7f6f6c070b60 2023-10-16T14:11:55.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:55.511+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6f6c0b4620 con 0x7f6f6c070b60 2023-10-16T14:11:56.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.011+0000 7f6f62ffd640 1 -- 172.21.15.60:0/1695965720 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v134) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6f54079e80 con 0x7f6f6c070b60 2023-10-16T14:11:56.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6f4c06b0b0 msgr2=0x7f6f4c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6f4c06b0b0 0x7f6f4c06d570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00a810 tx=0x7f6f5c008040 comp rx=0 tx=0).stop 2023-10-16T14:11:56.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 msgr2=0x7f6f6c154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c154290 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f6f5402b8c0 tx=0x7f6f540046c0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 shutdown_connections 2023-10-16T14:11:56.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f6f4c06b0b0 0x7f6f4c06d570 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f6c159950 0x7f6f6c151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f6c070b60 0x7f6f6c154290 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 --2- 172.21.15.60:0/1695965720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f6c06c100 0x7f6f6c14af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.014+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 >> 172.21.15.60:0/1695965720 conn(0x7f6f6c0ce6a0 msgr2=0x7f6f6c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:56.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.015+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 shutdown_connections 2023-10-16T14:11:56.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.015+0000 7f6f6b59e640 1 -- 172.21.15.60:0/1695965720 wait complete. 2023-10-16T14:11:56.040 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T14:11:56.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.265+0000 7f52b2bef640 1 Processor -- start 2023-10-16T14:11:56.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.265+0000 7f52b2bef640 1 -- start start 2023-10-16T14:11:56.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.266+0000 7f52b2bef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 0x7f52ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.266+0000 7f52b2bef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f52ac0cce60 0x7f52ac0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b2bef640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac15d120 con 0x7f52ac15cd40 2023-10-16T14:11:56.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b2bef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ac15e180 con 0x7f52ac0cce60 2023-10-16T14:11:56.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b2bef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac06b300 con 0x7f52ac15dcc0 2023-10-16T14:11:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b1bed640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f52ac15cd40 0x7f52ac0cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49014/0 (socket says 172.21.15.60:49014) 2023-10-16T14:11:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b1bed640 1 -- 172.21.15.60:0/3500046659 learned_addr learned my addr 172.21.15.60:0/3500046659 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b13ec640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 0x7f52ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b23ee640 1 --2- 172.21.15.60:0/3500046659 >> v2:172.21.15.144:3300/0 conn(0x7f52ac0cce60 0x7f52ac0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833713826 0 0) 0x7f52ac15d120 con 0x7f52ac15cd40 2023-10-16T14:11:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.267+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5298003520 con 0x7f52ac15cd40 2023-10-16T14:11:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507005002 0 0) 0x7f5298003520 con 0x7f52ac15cd40 2023-10-16T14:11:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 msgr2=0x7f52ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 0x7f52ac0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 >> v2:172.21.15.144:3300/0 conn(0x7f52ac0cce60 msgr2=0x7f52ac0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 --2- 172.21.15.60:0/3500046659 >> v2:172.21.15.144:3300/0 conn(0x7f52ac0cce60 0x7f52ac0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac06b5e0 con 0x7f52ac15cd40 2023-10-16T14:11:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f52a8001670 con 0x7f52ac15cd40 2023-10-16T14:11:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 >> v1:172.21.15.60:6789/0 conn(0x7f52ac15cd40 legacy=0x7f52ac0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5298003970 0x7f5298003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f5298004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 0x7f5298005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac15e180 con 0x7f52980047c0 2023-10-16T14:11:56.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac06b300 con 0x7f5298003970 2023-10-16T14:11:56.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529800ba60 con 0x7f52ac15dcc0 2023-10-16T14:11:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.268+0000 7f52b13ec640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f5298004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b1bed640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5298003970 0x7f5298003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b23ee640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 0x7f5298005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b13ec640 1 -- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 msgr2=0x7f5298005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b13ec640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac15dcc0 0x7f5298005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b13ec640 1 -- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5298003970 msgr2=0x7f5298003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b13ec640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5298003970 0x7f5298003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.269+0000 7f52b13ec640 1 -- 172.21.15.60:0/3500046659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5298003520 con 0x7f52980047c0 2023-10-16T14:11:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.270+0000 7f52b13ec640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f5298004c40 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f529c009a90 tx=0x7f52980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35c9ab07bf071d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.270+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c02c5a0 con 0x7f52980047c0 2023-10-16T14:11:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.270+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c02c700 con 0x7f52980047c0 2023-10-16T14:11:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.270+0000 7f52b0beb640 1 -- 172.21.15.60:0/3500046659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c02ca90 con 0x7f52980047c0 2023-10-16T14:11:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b2bef640 1 -- 172.21.15.60:0/3500046659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 msgr2=0x7f5298004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b2bef640 1 --2- 172.21.15.60:0/3500046659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f5298004c40 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f529c009a90 tx=0x7f52980057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b23ee640 1 -- 172.21.15.60:0/3500046659 reap_dead start 2023-10-16T14:11:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b2bef640 1 -- 172.21.15.60:0/3500046659 shutdown_connections 2023-10-16T14:11:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b2bef640 1 -- 172.21.15.60:0/3500046659 >> 172.21.15.60:0/3500046659 conn(0x7f52ac0ceb80 msgr2=0x7f52ac0cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b2bef640 1 -- 172.21.15.60:0/3500046659 shutdown_connections 2023-10-16T14:11:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.271+0000 7f52b2bef640 1 -- 172.21.15.60:0/3500046659 wait complete. 2023-10-16T14:11:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.273+0000 7f52b2bef640 1 Processor -- start 2023-10-16T14:11:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.273+0000 7f52b2bef640 1 -- start start 2023-10-16T14:11:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b2bef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f52ac0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b2bef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac0ae420 0x7f52ac0c3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b2bef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52ac0b2a00 0x7f52ac0b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b2bef640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac0b33f0 con 0x7f52980047c0 2023-10-16T14:11:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b2bef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac1559c0 con 0x7f52ac0b2a00 2023-10-16T14:11:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b2bef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac0c3c90 con 0x7f52ac0ae420 2023-10-16T14:11:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b1bed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f52ac0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b1bed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f52ac0b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36404/0 (socket says 172.21.15.60:36404) 2023-10-16T14:11:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.274+0000 7f52b1bed640 1 -- 172.21.15.60:0/2967091060 learned_addr learned my addr 172.21.15.60:0/2967091060 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b13ec640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac0ae420 0x7f52ac0c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b23ee640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52ac0b2a00 0x7f52ac0b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b1bed640 1 -- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac0ae420 msgr2=0x7f52ac0c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b1bed640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac0ae420 0x7f52ac0c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b1bed640 1 -- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52ac0b2a00 msgr2=0x7f52ac0b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b1bed640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52ac0b2a00 0x7f52ac0b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b1bed640 1 -- 172.21.15.60:0/2967091060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac0c3f70 con 0x7f52980047c0 2023-10-16T14:11:56.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b23ee640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52ac0b2a00 0x7f52ac0b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:56.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52b1bed640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f52ac0b0f50 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f52a8001cc0 tx=0x7f52a800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:56.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.275+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8005320 con 0x7f52980047c0 2023-10-16T14:11:56.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.276+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a80054a0 con 0x7f52980047c0 2023-10-16T14:11:56.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.276+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80047e0 con 0x7f52980047c0 2023-10-16T14:11:56.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.276+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac15f8f0 con 0x7f52980047c0 2023-10-16T14:11:56.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.276+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ac15feb0 con 0x7f52980047c0 2023-10-16T14:11:56.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.277+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ac15f750 con 0x7f52980047c0 2023-10-16T14:11:56.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.278+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f52a8004940 con 0x7f52980047c0 2023-10-16T14:11:56.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.279+0000 7f52a27fc640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f529006b0e0 0x7f529006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.279+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f52a80895c0 con 0x7f52980047c0 2023-10-16T14:11:56.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.280+0000 7f52b13ec640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f529006b0e0 0x7f529006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.281+0000 7f52b13ec640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f529006b0e0 0x7f529006d5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f529c036000 tx=0x7f529c030000 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:56.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.284+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52ac15f750 con 0x7f52980047c0 2023-10-16T14:11:56.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.518+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52ac0ae800 con 0x7f52980047c0 2023-10-16T14:11:56.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.518+0000 7f52a27fc640 1 -- 172.21.15.60:0/2967091060 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6146 (secure 0 0 0) 0x7f52ac0ae800 con 0x7f52980047c0 2023-10-16T14:11:56.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:56.524 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:11:56.525 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 134 2023-10-16T14:11:56.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f529006b0e0 msgr2=0x7f529006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f529006b0e0 0x7f529006d5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f529c036000 tx=0x7f529c030000 comp rx=0 tx=0).stop 2023-10-16T14:11:56.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 msgr2=0x7f52ac0b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f52ac0b0f50 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f52a8001cc0 tx=0x7f52a800ed50 comp rx=0 tx=0).stop 2023-10-16T14:11:56.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 shutdown_connections 2023-10-16T14:11:56.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f529006b0e0 0x7f529006d5a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52ac0ae420 0x7f52ac0c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52ac0b2a00 0x7f52ac0b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 --2- 172.21.15.60:0/2967091060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52980047c0 0x7f52ac0b0f50 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 >> 172.21.15.60:0/2967091060 conn(0x7f52ac0ceb80 msgr2=0x7f52ac071290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:56.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 shutdown_connections 2023-10-16T14:11:56.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.521+0000 7f52b2bef640 1 -- 172.21.15.60:0/2967091060 wait complete. 2023-10-16T14:11:56.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:11:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.766+0000 7efc4d378640 1 Processor -- start 2023-10-16T14:11:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.766+0000 7efc4d378640 1 -- start start 2023-10-16T14:11:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.768+0000 7efc4d378640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efc4815aaa0 0x7efc4815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.768+0000 7efc4d378640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc4815ae80 0x7efc480cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.768+0000 7efc4d378640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4815ed60 con 0x7efc480ccaa0 2023-10-16T14:11:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.768+0000 7efc4d378640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4806b810 con 0x7efc4815ae80 2023-10-16T14:11:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.768+0000 7efc4d378640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc480707e0 con 0x7efc4815aaa0 2023-10-16T14:11:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc4815ae80 0x7efc480cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc4815ae80 0x7efc480cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36408/0 (socket says 172.21.15.60:36408) 2023-10-16T14:11:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 -- 172.21.15.60:0/2223605523 learned_addr learned my addr 172.21.15.60:0/2223605523 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc46ffd640 1 --2- 172.21.15.60:0/2223605523 >> v2:172.21.15.144:3300/0 conn(0x7efc4815aaa0 0x7efc4815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 -- 172.21.15.60:0/2223605523 >> v2:172.21.15.144:3300/0 conn(0x7efc4815aaa0 msgr2=0x7efc4815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 --2- 172.21.15.60:0/2223605523 >> v2:172.21.15.144:3300/0 conn(0x7efc4815aaa0 0x7efc4815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 -- 172.21.15.60:0/2223605523 >> v1:172.21.15.144:6789/0 conn(0x7efc480ccaa0 legacy=0x7efc48070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc3ffff640 1 -- 172.21.15.60:0/2223605523 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4815ef90 con 0x7efc4815ae80 2023-10-16T14:11:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.769+0000 7efc46ffd640 1 --2- 172.21.15.60:0/2223605523 >> v2:172.21.15.144:3300/0 conn(0x7efc4815aaa0 0x7efc4815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.770+0000 7efc3ffff640 1 --2- 172.21.15.60:0/2223605523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480cc560 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7efc480cd990 tx=0x7efc3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98a486585083d0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc467fc640 1 -- 172.21.15.60:0/2223605523 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30031870 con 0x7efc4815ae80 2023-10-16T14:11:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc467fc640 1 -- 172.21.15.60:0/2223605523 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc300319d0 con 0x7efc4815ae80 2023-10-16T14:11:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc467fc640 1 -- 172.21.15.60:0/2223605523 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30031d60 con 0x7efc4815ae80 2023-10-16T14:11:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc4d378640 1 -- 172.21.15.60:0/2223605523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 msgr2=0x7efc480cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc4d378640 1 --2- 172.21.15.60:0/2223605523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480cc560 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7efc480cd990 tx=0x7efc3002b910 comp rx=0 tx=0).stop 2023-10-16T14:11:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc4d378640 1 -- 172.21.15.60:0/2223605523 shutdown_connections 2023-10-16T14:11:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.771+0000 7efc4d378640 1 --2- 172.21.15.60:0/2223605523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480cc560 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.772+0000 7efc4d378640 1 --2- 172.21.15.60:0/2223605523 >> v2:172.21.15.144:3300/0 conn(0x7efc4815aaa0 0x7efc4815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.772+0000 7efc4d378640 1 -- 172.21.15.60:0/2223605523 >> 172.21.15.60:0/2223605523 conn(0x7efc480cb410 msgr2=0x7efc480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.772+0000 7efc4d378640 1 -- 172.21.15.60:0/2223605523 shutdown_connections 2023-10-16T14:11:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.772+0000 7efc4d378640 1 -- 172.21.15.60:0/2223605523 wait complete. 2023-10-16T14:11:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.773+0000 7efc4d378640 1 Processor -- start 2023-10-16T14:11:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.774+0000 7efc4d378640 1 -- start start 2023-10-16T14:11:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.774+0000 7efc4d378640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc480ccaa0 0x7efc480c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc4d378640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc4815aaa0 0x7efc480c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc4d378640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc4d378640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4815c900 con 0x7efc4815ae80 2023-10-16T14:11:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc4d378640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4815cbe0 con 0x7efc4815aaa0 2023-10-16T14:11:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc4d378640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4815cec0 con 0x7efc480ccaa0 2023-10-16T14:11:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36412/0 (socket says 172.21.15.60:36412) 2023-10-16T14:11:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 -- 172.21.15.60:0/2330056582 learned_addr learned my addr 172.21.15.60:0/2330056582 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc46ffd640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc480ccaa0 0x7efc480c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc3ffff640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc4815aaa0 0x7efc480c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 -- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc480ccaa0 msgr2=0x7efc480c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc480ccaa0 0x7efc480c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 -- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc4815aaa0 msgr2=0x7efc480c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc4815aaa0 0x7efc480c3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc477fe640 1 -- 172.21.15.60:0/2330056582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc300094d0 con 0x7efc4815ae80 2023-10-16T14:11:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.775+0000 7efc3ffff640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc4815aaa0 0x7efc480c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.776+0000 7efc477fe640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480c8620 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7efc4000c780 tx=0x7efc4000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.776+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40004b60 con 0x7efc4815ae80 2023-10-16T14:11:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.776+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4000ce90 con 0x7efc4815ae80 2023-10-16T14:11:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.776+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4000a850 con 0x7efc4815ae80 2023-10-16T14:11:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.776+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4815d1a0 con 0x7efc4815ae80 2023-10-16T14:11:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.776+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4815c590 con 0x7efc4815ae80 2023-10-16T14:11:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.777+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4804e160 con 0x7efc4815ae80 2023-10-16T14:11:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.778+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efc4000a9b0 con 0x7efc4815ae80 2023-10-16T14:11:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.779+0000 7efc3f7fe640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efc2806b0b0 0x7efc2806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.779+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7efc40089f70 con 0x7efc4815ae80 2023-10-16T14:11:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.780+0000 7efc46ffd640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efc2806b0b0 0x7efc2806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:56.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.781+0000 7efc46ffd640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efc2806b0b0 0x7efc2806d570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efc48070c90 tx=0x7efc38008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:56.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.783+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc40054950 con 0x7efc4815ae80 2023-10-16T14:11:56.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.982+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc480cdc60 con 0x7efc4815ae80 2023-10-16T14:11:56.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.985+0000 7efc3f7fe640 1 -- 172.21.15.60:0/2330056582 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+20175 (secure 0 0 0) 0x7efc40057f70 con 0x7efc4815ae80 2023-10-16T14:11:56.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:56.989 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":256,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:11:53.974731+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:11:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.987+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efc2806b0b0 msgr2=0x7efc2806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.987+0000 7efc4d378640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efc2806b0b0 0x7efc2806d570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efc48070c90 tx=0x7efc38008040 comp rx=0 tx=0).stop 2023-10-16T14:11:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 msgr2=0x7efc480c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480c8620 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7efc4000c780 tx=0x7efc4000cc40 comp rx=0 tx=0).stop 2023-10-16T14:11:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 shutdown_connections 2023-10-16T14:11:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7efc2806b0b0 0x7efc2806d570 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4815ae80 0x7efc480c8620 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc4815aaa0 0x7efc480c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 --2- 172.21.15.60:0/2330056582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efc480ccaa0 0x7efc480c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 >> 172.21.15.60:0/2330056582 conn(0x7efc480cb410 msgr2=0x7efc480b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 shutdown_connections 2023-10-16T14:11:56.994 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:56.988+0000 7efc4d378640 1 -- 172.21.15.60:0/2330056582 wait complete. 2023-10-16T14:11:57.010 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:11:57.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:11:57.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.228+0000 7fea8759e640 1 Processor -- start 2023-10-16T14:11:57.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.229+0000 7fea8759e640 1 -- start start 2023-10-16T14:11:57.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.230+0000 7fea8759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fea880ce6b0 0x7fea880cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.230+0000 7fea8759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea880cf4a0 con 0x7fea880cc570 2023-10-16T14:11:57.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.230+0000 7fea8759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea880cb550 con 0x7fea8815ac10 2023-10-16T14:11:57.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.230+0000 7fea8759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea880ceb40 con 0x7fea880ce6b0 2023-10-16T14:11:57.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea86d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fea880cc570 0x7fea880cd320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49026/0 (socket says 172.21.15.60:49026) 2023-10-16T14:11:57.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea86d9d640 1 -- 172.21.15.60:0/3507175280 learned_addr learned my addr 172.21.15.60:0/3507175280 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787277984 0 0) 0x7fea880cf4a0 con 0x7fea880cc570 2023-10-16T14:11:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74003520 con 0x7fea880cc570 2023-10-16T14:11:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea7ffff640 1 --2- 172.21.15.60:0/3507175280 >> v2:172.21.15.144:3300/0 conn(0x7fea880ce6b0 0x7fea880cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2897151511 0 0) 0x7fea74003520 con 0x7fea880cc570 2023-10-16T14:11:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 >> v2:172.21.15.144:3300/0 conn(0x7fea880ce6b0 msgr2=0x7fea880cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 --2- 172.21.15.60:0/3507175280 >> v2:172.21.15.144:3300/0 conn(0x7fea880ce6b0 0x7fea880cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 >> v1:172.21.15.144:6789/0 conn(0x7fea8815ac10 legacy=0x7fea880cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880cb830 con 0x7fea880cc570 2023-10-16T14:11:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fea800019f0 con 0x7fea880cc570 2023-10-16T14:11:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 >> v1:172.21.15.60:6789/0 conn(0x7fea880cc570 legacy=0x7fea880cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea740039f0 0x7fea74003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.231+0000 7fea85d9b640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea740047c0 0x7fea74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea85d9b640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880ceb40 con 0x7fea74005380 2023-10-16T14:11:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7400c200 con 0x7fea740047c0 2023-10-16T14:11:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7400c3c0 con 0x7fea740039f0 2023-10-16T14:11:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea86d9d640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea7ffff640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea740047c0 0x7fea74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea86d9d640 1 -- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea740039f0 msgr2=0x7fea74003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea86d9d640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea740039f0 0x7fea74003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea86d9d640 1 -- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea740047c0 msgr2=0x7fea74004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea86d9d640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea740047c0 0x7fea74004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.232+0000 7fea86d9d640 1 -- 172.21.15.60:0/3507175280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74003520 con 0x7fea74005380 2023-10-16T14:11:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.233+0000 7fea86d9d640 1 -- 172.21.15.60:0/3507175280 reap_dead start 2023-10-16T14:11:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.233+0000 7fea86d9d640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea7400b980 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fea8000a7e0 tx=0x7fea740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53e555829b470b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:57.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80034630 con 0x7fea74005380 2023-10-16T14:11:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8002f070 con 0x7fea74005380 2023-10-16T14:11:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea85d9b640 1 -- 172.21.15.60:0/3507175280 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800349d0 con 0x7fea74005380 2023-10-16T14:11:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 -- 172.21.15.60:0/3507175280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 msgr2=0x7fea7400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea7400b980 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fea8000a7e0 tx=0x7fea740057f0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 -- 172.21.15.60:0/3507175280 shutdown_connections 2023-10-16T14:11:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 --2- 172.21.15.60:0/3507175280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea7400b980 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 -- 172.21.15.60:0/3507175280 >> 172.21.15.60:0/3507175280 conn(0x7fea88099a10 msgr2=0x7fea880d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 -- 172.21.15.60:0/3507175280 shutdown_connections 2023-10-16T14:11:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.234+0000 7fea8759e640 1 -- 172.21.15.60:0/3507175280 wait complete. 2023-10-16T14:11:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.236+0000 7fea8759e640 1 Processor -- start 2023-10-16T14:11:57.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.236+0000 7fea8759e640 1 -- start start 2023-10-16T14:11:57.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.237+0000 7fea8759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea8815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.237+0000 7fea8759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea8815b2f0 0x7fea88155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.237+0000 7fea8759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea8815c0c0 0x7fea8815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.237+0000 7fea8759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8815cab0 con 0x7fea74005380 2023-10-16T14:11:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.237+0000 7fea8759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea881f18d0 con 0x7fea8815b2f0 2023-10-16T14:11:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.237+0000 7fea8759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea881f1b70 con 0x7fea8815c0c0 2023-10-16T14:11:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea8815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea8815e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36434/0 (socket says 172.21.15.60:36434) 2023-10-16T14:11:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 -- 172.21.15.60:0/273154193 learned_addr learned my addr 172.21.15.60:0/273154193 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea7ffff640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea8815b2f0 0x7fea88155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea86d9d640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea8815c0c0 0x7fea8815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 -- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea8815c0c0 msgr2=0x7fea8815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea8815c0c0 0x7fea8815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 -- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea8815b2f0 msgr2=0x7fea88155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea8815b2f0 0x7fea88155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 -- 172.21.15.60:0/273154193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea881f1e50 con 0x7fea74005380 2023-10-16T14:11:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea86d9d640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea8815c0c0 0x7fea8815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.238+0000 7fea8659c640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea8815e0b0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fea7800c480 tx=0x7fea7800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.239+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78012830 con 0x7fea74005380 2023-10-16T14:11:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.239+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea78012990 con 0x7fea74005380 2023-10-16T14:11:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.239+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78012d40 con 0x7fea74005380 2023-10-16T14:11:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.239+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea881f2130 con 0x7fea74005380 2023-10-16T14:11:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.239+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea881557d0 con 0x7fea74005380 2023-10-16T14:11:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.240+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea881f2370 con 0x7fea74005380 2023-10-16T14:11:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.241+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fea7801b400 con 0x7fea74005380 2023-10-16T14:11:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.242+0000 7fea7effd640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fea6806b0e0 0x7fea6806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.242+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fea78089ca0 con 0x7fea74005380 2023-10-16T14:11:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.243+0000 7fea7ffff640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fea6806b0e0 0x7fea6806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.244+0000 7fea7ffff640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fea6806b0e0 0x7fea6806d5a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fea880d09d0 tx=0x7fea70008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:57.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.247+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea881f2370 con 0x7fea74005380 2023-10-16T14:11:57.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.437+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 --> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea880cf780 con 0x7fea6806b0e0 2023-10-16T14:11:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.478+0000 7fea7effd640 1 -- 172.21.15.60:0/273154193 <== mgr.9169 v2:172.21.15.144:6832/4052124783 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526045 (secure 0 0 0) 0x7fea7000c040 con 0x7fea6806b0e0 2023-10-16T14:11:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:11:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fea6806b0e0 msgr2=0x7fea6806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fea6806b0e0 0x7fea6806d5a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fea880d09d0 tx=0x7fea70008040 comp rx=0 tx=0).stop 2023-10-16T14:11:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 msgr2=0x7fea8815e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea8815e0b0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fea7800c480 tx=0x7fea7800c940 comp rx=0 tx=0).stop 2023-10-16T14:11:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 shutdown_connections 2023-10-16T14:11:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fea6806b0e0 0x7fea6806d5a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fea8815c0c0 0x7fea8815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea8815b2f0 0x7fea88155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 --2- 172.21.15.60:0/273154193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea74005380 0x7fea8815e0b0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.489+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 >> 172.21.15.60:0/273154193 conn(0x7fea88099a10 msgr2=0x7fea8806d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.490+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 shutdown_connections 2023-10-16T14:11:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.490+0000 7fea8759e640 1 -- 172.21.15.60:0/273154193 wait complete. 2023-10-16T14:11:57.518 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2023-10-16T14:11:57.067680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":714,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":292,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428472,"kb_used_data":26360,"kb_used_omap":12,"kb_used_meta":401971,"kb_avail":754546248,"statfs":{"total":773094113280,"available":772655357952,"internally_reserved":0,"allocated":26992640,"data_stored":13808094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411618872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002531"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.857803+0000","last_change":"2023-10-16T14:11:52.991222+0000","last_active":"2023-10-16T14:11:54.857803+0000","last_peered":"2023-10-16T14:11:54.857803+0000","last_clean":"2023-10-16T14:11:54.857803+0000","last_became_active":"2023-10-16T14:11:52.990947+0000","last_became_peered":"2023-10-16T14:11:52.990947+0000","last_unstale":"2023-10-16T14:11:54.857803+0000","last_undegraded":"2023-10-16T14:11:54.857803+0000","last_fullsized":"2023-10-16T14:11:54.857803+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:24.644523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986943+0000","last_change":"2023-10-16T14:11:51.988133+0000","last_active":"2023-10-16T14:11:53.986943+0000","last_peered":"2023-10-16T14:11:53.986943+0000","last_clean":"2023-10-16T14:11:53.986943+0000","last_became_active":"2023-10-16T14:11:51.987807+0000","last_became_peered":"2023-10-16T14:11:51.987807+0000","last_unstale":"2023-10-16T14:11:53.986943+0000","last_undegraded":"2023-10-16T14:11:53.986943+0000","last_fullsized":"2023-10-16T14:11:53.986943+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:05.737538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991549+0000","last_change":"2023-10-16T14:11:38.800766+0000","last_active":"2023-10-16T14:11:52.991549+0000","last_peered":"2023-10-16T14:11:52.991549+0000","last_clean":"2023-10-16T14:11:52.991549+0000","last_became_active":"2023-10-16T14:11:38.799147+0000","last_became_peered":"2023-10-16T14:11:38.799147+0000","last_unstale":"2023-10-16T14:11:52.991549+0000","last_undegraded":"2023-10-16T14:11:52.991549+0000","last_fullsized":"2023-10-16T14:11:52.991549+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:09.869393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.008887+0000","last_change":"2023-10-16T14:11:53.005992+0000","last_active":"2023-10-16T14:11:53.008887+0000","last_peered":"2023-10-16T14:11:53.008887+0000","last_clean":"2023-10-16T14:11:53.008887+0000","last_became_active":"2023-10-16T14:11:53.005721+0000","last_became_peered":"2023-10-16T14:11:53.005721+0000","last_unstale":"2023-10-16T14:11:53.008887+0000","last_undegraded":"2023-10-16T14:11:53.008887+0000","last_fullsized":"2023-10-16T14:11:53.008887+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:18.227695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.984497+0000","last_change":"2023-10-16T14:11:51.990577+0000","last_active":"2023-10-16T14:11:52.984497+0000","last_peered":"2023-10-16T14:11:52.984497+0000","last_clean":"2023-10-16T14:11:52.984497+0000","last_became_active":"2023-10-16T14:11:51.990290+0000","last_became_peered":"2023-10-16T14:11:51.990290+0000","last_unstale":"2023-10-16T14:11:52.984497+0000","last_undegraded":"2023-10-16T14:11:52.984497+0000","last_fullsized":"2023-10-16T14:11:52.984497+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:50.097531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993685+0000","last_change":"2023-10-16T14:11:38.811708+0000","last_active":"2023-10-16T14:11:52.993685+0000","last_peered":"2023-10-16T14:11:52.993685+0000","last_clean":"2023-10-16T14:11:52.993685+0000","last_became_active":"2023-10-16T14:11:38.808060+0000","last_became_peered":"2023-10-16T14:11:38.808060+0000","last_unstale":"2023-10-16T14:11:52.993685+0000","last_undegraded":"2023-10-16T14:11:52.993685+0000","last_fullsized":"2023-10-16T14:11:52.993685+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:53.712931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.008555+0000","last_change":"2023-10-16T14:11:53.004533+0000","last_active":"2023-10-16T14:11:53.008555+0000","last_peered":"2023-10-16T14:11:53.008555+0000","last_clean":"2023-10-16T14:11:53.008555+0000","last_became_active":"2023-10-16T14:11:53.004202+0000","last_became_peered":"2023-10-16T14:11:53.004202+0000","last_unstale":"2023-10-16T14:11:53.008555+0000","last_undegraded":"2023-10-16T14:11:53.008555+0000","last_fullsized":"2023-10-16T14:11:53.008555+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:41.964559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.717966+0000","last_change":"2023-10-16T14:11:51.983126+0000","last_active":"2023-10-16T14:11:54.717966+0000","last_peered":"2023-10-16T14:11:54.717966+0000","last_clean":"2023-10-16T14:11:54.717966+0000","last_became_active":"2023-10-16T14:11:51.982846+0000","last_became_peered":"2023-10-16T14:11:51.982846+0000","last_unstale":"2023-10-16T14:11:54.717966+0000","last_undegraded":"2023-10-16T14:11:54.717966+0000","last_fullsized":"2023-10-16T14:11:54.717966+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959501+0000","last_change":"2023-10-16T14:11:38.799851+0000","last_active":"2023-10-16T14:11:52.959501+0000","last_peered":"2023-10-16T14:11:52.959501+0000","last_clean":"2023-10-16T14:11:52.959501+0000","last_became_active":"2023-10-16T14:11:38.799520+0000","last_became_peered":"2023-10-16T14:11:38.799520+0000","last_unstale":"2023-10-16T14:11:52.959501+0000","last_undegraded":"2023-10-16T14:11:52.959501+0000","last_fullsized":"2023-10-16T14:11:52.959501+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:59.480982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993667+0000","last_change":"2023-10-16T14:11:52.992005+0000","last_active":"2023-10-16T14:11:52.993667+0000","last_peered":"2023-10-16T14:11:52.993667+0000","last_clean":"2023-10-16T14:11:52.993667+0000","last_became_active":"2023-10-16T14:11:52.991726+0000","last_became_peered":"2023-10-16T14:11:52.991726+0000","last_unstale":"2023-10-16T14:11:52.993667+0000","last_undegraded":"2023-10-16T14:11:52.993667+0000","last_fullsized":"2023-10-16T14:11:52.993667+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:59.718175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718933+0000","last_change":"2023-10-16T14:11:51.979772+0000","last_active":"2023-10-16T14:11:54.718933+0000","last_peered":"2023-10-16T14:11:54.718933+0000","last_clean":"2023-10-16T14:11:54.718933+0000","last_became_active":"2023-10-16T14:11:51.979485+0000","last_became_peered":"2023-10-16T14:11:51.979485+0000","last_unstale":"2023-10-16T14:11:54.718933+0000","last_undegraded":"2023-10-16T14:11:54.718933+0000","last_fullsized":"2023-10-16T14:11:54.718933+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:07.687465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.987866+0000","last_change":"2023-10-16T14:11:38.811457+0000","last_active":"2023-10-16T14:11:51.987866+0000","last_peered":"2023-10-16T14:11:51.987866+0000","last_clean":"2023-10-16T14:11:51.987866+0000","last_became_active":"2023-10-16T14:11:38.811124+0000","last_became_peered":"2023-10-16T14:11:38.811124+0000","last_unstale":"2023-10-16T14:11:51.987866+0000","last_undegraded":"2023-10-16T14:11:51.987866+0000","last_fullsized":"2023-10-16T14:11:51.987866+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:54.535585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.859198+0000","last_change":"2023-10-16T14:11:52.995293+0000","last_active":"2023-10-16T14:11:54.859198+0000","last_peered":"2023-10-16T14:11:54.859198+0000","last_clean":"2023-10-16T14:11:54.859198+0000","last_became_active":"2023-10-16T14:11:52.995029+0000","last_became_peered":"2023-10-16T14:11:52.995029+0000","last_unstale":"2023-10-16T14:11:54.859198+0000","last_undegraded":"2023-10-16T14:11:54.859198+0000","last_fullsized":"2023-10-16T14:11:54.859198+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:54.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988497+0000","last_change":"2023-10-16T14:11:51.990092+0000","last_active":"2023-10-16T14:11:53.988497+0000","last_peered":"2023-10-16T14:11:53.988497+0000","last_clean":"2023-10-16T14:11:53.988497+0000","last_became_active":"2023-10-16T14:11:51.987479+0000","last_became_peered":"2023-10-16T14:11:51.987479+0000","last_unstale":"2023-10-16T14:11:53.988497+0000","last_undegraded":"2023-10-16T14:11:53.988497+0000","last_fullsized":"2023-10-16T14:11:53.988497+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:06.455649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974822+0000","last_change":"2023-10-16T14:11:38.792747+0000","last_active":"2023-10-16T14:11:52.974822+0000","last_peered":"2023-10-16T14:11:52.974822+0000","last_clean":"2023-10-16T14:11:52.974822+0000","last_became_active":"2023-10-16T14:11:38.792334+0000","last_became_peered":"2023-10-16T14:11:38.792334+0000","last_unstale":"2023-10-16T14:11:52.974822+0000","last_undegraded":"2023-10-16T14:11:52.974822+0000","last_fullsized":"2023-10-16T14:11:52.974822+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:23.689830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858462+0000","last_change":"2023-10-16T14:11:52.983967+0000","last_active":"2023-10-16T14:11:54.858462+0000","last_peered":"2023-10-16T14:11:54.858462+0000","last_clean":"2023-10-16T14:11:54.858462+0000","last_became_active":"2023-10-16T14:11:52.983433+0000","last_became_peered":"2023-10-16T14:11:52.983433+0000","last_unstale":"2023-10-16T14:11:54.858462+0000","last_undegraded":"2023-10-16T14:11:54.858462+0000","last_fullsized":"2023-10-16T14:11:54.858462+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:30.851220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994494+0000","last_change":"2023-10-16T14:11:51.984943+0000","last_active":"2023-10-16T14:11:53.994494+0000","last_peered":"2023-10-16T14:11:53.994494+0000","last_clean":"2023-10-16T14:11:53.994494+0000","last_became_active":"2023-10-16T14:11:51.984081+0000","last_became_peered":"2023-10-16T14:11:51.984081+0000","last_unstale":"2023-10-16T14:11:53.994494+0000","last_undegraded":"2023-10-16T14:11:53.994494+0000","last_fullsized":"2023-10-16T14:11:53.994494+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:50.972815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.983485+0000","last_change":"2023-10-16T14:11:38.814105+0000","last_active":"2023-10-16T14:11:51.983485+0000","last_peered":"2023-10-16T14:11:51.983485+0000","last_clean":"2023-10-16T14:11:51.983485+0000","last_became_active":"2023-10-16T14:11:38.813562+0000","last_became_peered":"2023-10-16T14:11:38.813562+0000","last_unstale":"2023-10-16T14:11:51.983485+0000","last_undegraded":"2023-10-16T14:11:51.983485+0000","last_fullsized":"2023-10-16T14:11:51.983485+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:33.419442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988316+0000","last_change":"2023-10-16T14:11:53.003492+0000","last_active":"2023-10-16T14:11:53.988316+0000","last_peered":"2023-10-16T14:11:53.988316+0000","last_clean":"2023-10-16T14:11:53.988316+0000","last_became_active":"2023-10-16T14:11:53.002918+0000","last_became_peered":"2023-10-16T14:11:53.002918+0000","last_unstale":"2023-10-16T14:11:53.988316+0000","last_undegraded":"2023-10-16T14:11:53.988316+0000","last_fullsized":"2023-10-16T14:11:53.988316+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:29.134870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858931+0000","last_change":"2023-10-16T14:11:51.986183+0000","last_active":"2023-10-16T14:11:54.858931+0000","last_peered":"2023-10-16T14:11:54.858931+0000","last_clean":"2023-10-16T14:11:54.858931+0000","last_became_active":"2023-10-16T14:11:51.985896+0000","last_became_peered":"2023-10-16T14:11:51.985896+0000","last_unstale":"2023-10-16T14:11:54.858931+0000","last_undegraded":"2023-10-16T14:11:54.858931+0000","last_fullsized":"2023-10-16T14:11:54.858931+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:50.638980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974461+0000","last_change":"2023-10-16T14:11:38.807311+0000","last_active":"2023-10-16T14:11:52.974461+0000","last_peered":"2023-10-16T14:11:52.974461+0000","last_clean":"2023-10-16T14:11:52.974461+0000","last_became_active":"2023-10-16T14:11:38.806946+0000","last_became_peered":"2023-10-16T14:11:38.806946+0000","last_unstale":"2023-10-16T14:11:52.974461+0000","last_undegraded":"2023-10-16T14:11:52.974461+0000","last_fullsized":"2023-10-16T14:11:52.974461+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:12.192825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.005131+0000","last_change":"2023-10-16T14:11:52.998297+0000","last_active":"2023-10-16T14:11:53.005131+0000","last_peered":"2023-10-16T14:11:53.005131+0000","last_clean":"2023-10-16T14:11:53.005131+0000","last_became_active":"2023-10-16T14:11:52.997812+0000","last_became_peered":"2023-10-16T14:11:52.997812+0000","last_unstale":"2023-10-16T14:11:53.005131+0000","last_undegraded":"2023-10-16T14:11:53.005131+0000","last_fullsized":"2023-10-16T14:11:53.005131+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:23.097797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"255'2","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.009150+0000","last_change":"2023-10-16T14:11:51.988969+0000","last_active":"2023-10-16T14:11:53.009150+0000","last_peered":"2023-10-16T14:11:53.009150+0000","last_clean":"2023-10-16T14:11:53.009150+0000","last_became_active":"2023-10-16T14:11:51.988313+0000","last_became_peered":"2023-10-16T14:11:51.988313+0000","last_unstale":"2023-10-16T14:11:53.009150+0000","last_undegraded":"2023-10-16T14:11:53.009150+0000","last_fullsized":"2023-10-16T14:11:53.009150+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:41.227136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.987341+0000","last_change":"2023-10-16T14:11:38.796621+0000","last_active":"2023-10-16T14:11:52.987341+0000","last_peered":"2023-10-16T14:11:52.987341+0000","last_clean":"2023-10-16T14:11:52.987341+0000","last_became_active":"2023-10-16T14:11:38.795857+0000","last_became_peered":"2023-10-16T14:11:38.795857+0000","last_unstale":"2023-10-16T14:11:52.987341+0000","last_undegraded":"2023-10-16T14:11:52.987341+0000","last_fullsized":"2023-10-16T14:11:52.987341+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:24.189032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718190+0000","last_change":"2023-10-16T14:11:52.994713+0000","last_active":"2023-10-16T14:11:54.718190+0000","last_peered":"2023-10-16T14:11:54.718190+0000","last_clean":"2023-10-16T14:11:54.718190+0000","last_became_active":"2023-10-16T14:11:52.994427+0000","last_became_peered":"2023-10-16T14:11:52.994427+0000","last_unstale":"2023-10-16T14:11:54.718190+0000","last_undegraded":"2023-10-16T14:11:54.718190+0000","last_fullsized":"2023-10-16T14:11:54.718190+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:11.895561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.068141+0000","last_change":"2023-10-16T14:11:51.977729+0000","last_active":"2023-10-16T14:11:55.068141+0000","last_peered":"2023-10-16T14:11:55.068141+0000","last_clean":"2023-10-16T14:11:55.068141+0000","last_became_active":"2023-10-16T14:11:51.977088+0000","last_became_peered":"2023-10-16T14:11:51.977088+0000","last_unstale":"2023-10-16T14:11:55.068141+0000","last_undegraded":"2023-10-16T14:11:55.068141+0000","last_fullsized":"2023-10-16T14:11:55.068141+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:45.408681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979164+0000","last_change":"2023-10-16T14:11:38.807477+0000","last_active":"2023-10-16T14:11:52.979164+0000","last_peered":"2023-10-16T14:11:52.979164+0000","last_clean":"2023-10-16T14:11:52.979164+0000","last_became_active":"2023-10-16T14:11:38.807149+0000","last_became_peered":"2023-10-16T14:11:38.807149+0000","last_unstale":"2023-10-16T14:11:52.979164+0000","last_undegraded":"2023-10-16T14:11:52.979164+0000","last_fullsized":"2023-10-16T14:11:52.979164+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:16.463192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986793+0000","last_change":"2023-10-16T14:11:52.988356+0000","last_active":"2023-10-16T14:11:53.986793+0000","last_peered":"2023-10-16T14:11:53.986793+0000","last_clean":"2023-10-16T14:11:53.986793+0000","last_became_active":"2023-10-16T14:11:52.986771+0000","last_became_peered":"2023-10-16T14:11:52.986771+0000","last_unstale":"2023-10-16T14:11:53.986793+0000","last_undegraded":"2023-10-16T14:11:53.986793+0000","last_fullsized":"2023-10-16T14:11:53.986793+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:35.593973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.992530+0000","last_change":"2023-10-16T14:11:51.988763+0000","last_active":"2023-10-16T14:11:53.992530+0000","last_peered":"2023-10-16T14:11:53.992530+0000","last_clean":"2023-10-16T14:11:53.992530+0000","last_became_active":"2023-10-16T14:11:51.988375+0000","last_became_peered":"2023-10-16T14:11:51.988375+0000","last_unstale":"2023-10-16T14:11:53.992530+0000","last_undegraded":"2023-10-16T14:11:53.992530+0000","last_fullsized":"2023-10-16T14:11:53.992530+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:33.934070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979076+0000","last_change":"2023-10-16T14:11:38.801109+0000","last_active":"2023-10-16T14:11:52.979076+0000","last_peered":"2023-10-16T14:11:52.979076+0000","last_clean":"2023-10-16T14:11:52.979076+0000","last_became_active":"2023-10-16T14:11:38.800831+0000","last_became_peered":"2023-10-16T14:11:38.800831+0000","last_unstale":"2023-10-16T14:11:52.979076+0000","last_undegraded":"2023-10-16T14:11:52.979076+0000","last_fullsized":"2023-10-16T14:11:52.979076+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:04.386150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646630+0000","last_change":"2023-10-16T14:11:52.989673+0000","last_active":"2023-10-16T14:11:54.646630+0000","last_peered":"2023-10-16T14:11:54.646630+0000","last_clean":"2023-10-16T14:11:54.646630+0000","last_became_active":"2023-10-16T14:11:52.989381+0000","last_became_peered":"2023-10-16T14:11:52.989381+0000","last_unstale":"2023-10-16T14:11:54.646630+0000","last_undegraded":"2023-10-16T14:11:54.646630+0000","last_fullsized":"2023-10-16T14:11:54.646630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:36.641150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.068799+0000","last_change":"2023-10-16T14:11:51.979510+0000","last_active":"2023-10-16T14:11:55.068799+0000","last_peered":"2023-10-16T14:11:55.068799+0000","last_clean":"2023-10-16T14:11:55.068799+0000","last_became_active":"2023-10-16T14:11:51.977340+0000","last_became_peered":"2023-10-16T14:11:51.977340+0000","last_unstale":"2023-10-16T14:11:55.068799+0000","last_undegraded":"2023-10-16T14:11:55.068799+0000","last_fullsized":"2023-10-16T14:11:55.068799+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:31.470827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974499+0000","last_change":"2023-10-16T14:11:38.806641+0000","last_active":"2023-10-16T14:11:52.974499+0000","last_peered":"2023-10-16T14:11:52.974499+0000","last_clean":"2023-10-16T14:11:52.974499+0000","last_became_active":"2023-10-16T14:11:38.806333+0000","last_became_peered":"2023-10-16T14:11:38.806333+0000","last_unstale":"2023-10-16T14:11:52.974499+0000","last_undegraded":"2023-10-16T14:11:52.974499+0000","last_fullsized":"2023-10-16T14:11:52.974499+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:03.262300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645519+0000","last_change":"2023-10-16T14:11:52.990635+0000","last_active":"2023-10-16T14:11:54.645519+0000","last_peered":"2023-10-16T14:11:54.645519+0000","last_clean":"2023-10-16T14:11:54.645519+0000","last_became_active":"2023-10-16T14:11:52.990024+0000","last_became_peered":"2023-10-16T14:11:52.990024+0000","last_unstale":"2023-10-16T14:11:54.645519+0000","last_undegraded":"2023-10-16T14:11:54.645519+0000","last_fullsized":"2023-10-16T14:11:54.645519+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:20.297500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858170+0000","last_change":"2023-10-16T14:11:51.981485+0000","last_active":"2023-10-16T14:11:54.858170+0000","last_peered":"2023-10-16T14:11:54.858170+0000","last_clean":"2023-10-16T14:11:54.858170+0000","last_became_active":"2023-10-16T14:11:51.979549+0000","last_became_peered":"2023-10-16T14:11:51.979549+0000","last_unstale":"2023-10-16T14:11:54.858170+0000","last_undegraded":"2023-10-16T14:11:54.858170+0000","last_fullsized":"2023-10-16T14:11:54.858170+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:31.739211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992402+0000","last_change":"2023-10-16T14:11:38.803389+0000","last_active":"2023-10-16T14:11:52.992402+0000","last_peered":"2023-10-16T14:11:52.992402+0000","last_clean":"2023-10-16T14:11:52.992402+0000","last_became_active":"2023-10-16T14:11:38.803101+0000","last_became_peered":"2023-10-16T14:11:38.803101+0000","last_unstale":"2023-10-16T14:11:52.992402+0000","last_undegraded":"2023-10-16T14:11:52.992402+0000","last_fullsized":"2023-10-16T14:11:52.992402+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:45.651230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.859165+0000","last_change":"2023-10-16T14:11:52.985569+0000","last_active":"2023-10-16T14:11:54.859165+0000","last_peered":"2023-10-16T14:11:54.859165+0000","last_clean":"2023-10-16T14:11:54.859165+0000","last_became_active":"2023-10-16T14:11:52.985244+0000","last_became_peered":"2023-10-16T14:11:52.985244+0000","last_unstale":"2023-10-16T14:11:54.859165+0000","last_undegraded":"2023-10-16T14:11:54.859165+0000","last_fullsized":"2023-10-16T14:11:54.859165+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:45.991538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.984498+0000","last_change":"2023-10-16T14:11:51.988535+0000","last_active":"2023-10-16T14:11:52.984498+0000","last_peered":"2023-10-16T14:11:52.984498+0000","last_clean":"2023-10-16T14:11:52.984498+0000","last_became_active":"2023-10-16T14:11:51.987601+0000","last_became_peered":"2023-10-16T14:11:51.987601+0000","last_unstale":"2023-10-16T14:11:52.984498+0000","last_undegraded":"2023-10-16T14:11:52.984498+0000","last_fullsized":"2023-10-16T14:11:52.984498+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:23.049883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976355+0000","last_change":"2023-10-16T14:11:38.811074+0000","last_active":"2023-10-16T14:11:52.976355+0000","last_peered":"2023-10-16T14:11:52.976355+0000","last_clean":"2023-10-16T14:11:52.976355+0000","last_became_active":"2023-10-16T14:11:38.810772+0000","last_became_peered":"2023-10-16T14:11:38.810772+0000","last_unstale":"2023-10-16T14:11:52.976355+0000","last_undegraded":"2023-10-16T14:11:52.976355+0000","last_fullsized":"2023-10-16T14:11:52.976355+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:03.977528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.003403+0000","last_change":"2023-10-16T14:11:52.997843+0000","last_active":"2023-10-16T14:11:53.003403+0000","last_peered":"2023-10-16T14:11:53.003403+0000","last_clean":"2023-10-16T14:11:53.003403+0000","last_became_active":"2023-10-16T14:11:52.997556+0000","last_became_peered":"2023-10-16T14:11:52.997556+0000","last_unstale":"2023-10-16T14:11:53.003403+0000","last_undegraded":"2023-10-16T14:11:53.003403+0000","last_fullsized":"2023-10-16T14:11:53.003403+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:51.362761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994722+0000","last_change":"2023-10-16T14:11:51.988518+0000","last_active":"2023-10-16T14:11:53.994722+0000","last_peered":"2023-10-16T14:11:53.994722+0000","last_clean":"2023-10-16T14:11:53.994722+0000","last_became_active":"2023-10-16T14:11:51.988105+0000","last_became_peered":"2023-10-16T14:11:51.988105+0000","last_unstale":"2023-10-16T14:11:53.994722+0000","last_undegraded":"2023-10-16T14:11:53.994722+0000","last_fullsized":"2023-10-16T14:11:53.994722+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:12.661473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.985004+0000","last_change":"2023-10-16T14:11:38.806714+0000","last_active":"2023-10-16T14:11:52.985004+0000","last_peered":"2023-10-16T14:11:52.985004+0000","last_clean":"2023-10-16T14:11:52.985004+0000","last_became_active":"2023-10-16T14:11:38.805598+0000","last_became_peered":"2023-10-16T14:11:38.805598+0000","last_unstale":"2023-10-16T14:11:52.985004+0000","last_undegraded":"2023-10-16T14:11:52.985004+0000","last_fullsized":"2023-10-16T14:11:52.985004+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:37.163662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995828+0000","last_change":"2023-10-16T14:11:53.005505+0000","last_active":"2023-10-16T14:11:53.995828+0000","last_peered":"2023-10-16T14:11:53.995828+0000","last_clean":"2023-10-16T14:11:53.995828+0000","last_became_active":"2023-10-16T14:11:53.005195+0000","last_became_peered":"2023-10-16T14:11:53.005195+0000","last_unstale":"2023-10-16T14:11:53.995828+0000","last_undegraded":"2023-10-16T14:11:53.995828+0000","last_fullsized":"2023-10-16T14:11:53.995828+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:21.280292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646874+0000","last_change":"2023-10-16T14:11:51.982174+0000","last_active":"2023-10-16T14:11:54.646874+0000","last_peered":"2023-10-16T14:11:54.646874+0000","last_clean":"2023-10-16T14:11:54.646874+0000","last_became_active":"2023-10-16T14:11:51.981827+0000","last_became_peered":"2023-10-16T14:11:51.981827+0000","last_unstale":"2023-10-16T14:11:54.646874+0000","last_undegraded":"2023-10-16T14:11:54.646874+0000","last_fullsized":"2023-10-16T14:11:54.646874+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:19.718790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982855+0000","last_change":"2023-10-16T14:11:38.808575+0000","last_active":"2023-10-16T14:11:52.982855+0000","last_peered":"2023-10-16T14:11:52.982855+0000","last_clean":"2023-10-16T14:11:52.982855+0000","last_became_active":"2023-10-16T14:11:38.808273+0000","last_became_peered":"2023-10-16T14:11:38.808273+0000","last_unstale":"2023-10-16T14:11:52.982855+0000","last_undegraded":"2023-10-16T14:11:52.982855+0000","last_fullsized":"2023-10-16T14:11:52.982855+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:06.098401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.005109+0000","last_change":"2023-10-16T14:11:52.998086+0000","last_active":"2023-10-16T14:11:53.005109+0000","last_peered":"2023-10-16T14:11:53.005109+0000","last_clean":"2023-10-16T14:11:53.005109+0000","last_became_active":"2023-10-16T14:11:52.997399+0000","last_became_peered":"2023-10-16T14:11:52.997399+0000","last_unstale":"2023-10-16T14:11:53.005109+0000","last_undegraded":"2023-10-16T14:11:53.005109+0000","last_fullsized":"2023-10-16T14:11:53.005109+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:16.114233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069419+0000","last_change":"2023-10-16T14:11:51.981635+0000","last_active":"2023-10-16T14:11:55.069419+0000","last_peered":"2023-10-16T14:11:55.069419+0000","last_clean":"2023-10-16T14:11:55.069419+0000","last_became_active":"2023-10-16T14:11:51.981301+0000","last_became_peered":"2023-10-16T14:11:51.981301+0000","last_unstale":"2023-10-16T14:11:55.069419+0000","last_undegraded":"2023-10-16T14:11:55.069419+0000","last_fullsized":"2023-10-16T14:11:55.069419+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:16.411541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959818+0000","last_change":"2023-10-16T14:11:38.804694+0000","last_active":"2023-10-16T14:11:52.959818+0000","last_peered":"2023-10-16T14:11:52.959818+0000","last_clean":"2023-10-16T14:11:52.959818+0000","last_became_active":"2023-10-16T14:11:38.804373+0000","last_became_peered":"2023-10-16T14:11:38.804373+0000","last_unstale":"2023-10-16T14:11:52.959818+0000","last_undegraded":"2023-10-16T14:11:52.959818+0000","last_fullsized":"2023-10-16T14:11:52.959818+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:42.092693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986689+0000","last_change":"2023-10-16T14:11:52.984083+0000","last_active":"2023-10-16T14:11:53.986689+0000","last_peered":"2023-10-16T14:11:53.986689+0000","last_clean":"2023-10-16T14:11:53.986689+0000","last_became_active":"2023-10-16T14:11:52.981380+0000","last_became_peered":"2023-10-16T14:11:52.981380+0000","last_unstale":"2023-10-16T14:11:53.986689+0000","last_undegraded":"2023-10-16T14:11:53.986689+0000","last_fullsized":"2023-10-16T14:11:53.986689+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:24.467082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"255'8","reported_seq":20,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645058+0000","last_change":"2023-10-16T14:11:51.985634+0000","last_active":"2023-10-16T14:11:54.645058+0000","last_peered":"2023-10-16T14:11:54.645058+0000","last_clean":"2023-10-16T14:11:54.645058+0000","last_became_active":"2023-10-16T14:11:51.985328+0000","last_became_peered":"2023-10-16T14:11:51.985328+0000","last_unstale":"2023-10-16T14:11:54.645058+0000","last_undegraded":"2023-10-16T14:11:54.645058+0000","last_fullsized":"2023-10-16T14:11:54.645058+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:06.367330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993652+0000","last_change":"2023-10-16T14:11:38.799470+0000","last_active":"2023-10-16T14:11:52.993652+0000","last_peered":"2023-10-16T14:11:52.993652+0000","last_clean":"2023-10-16T14:11:52.993652+0000","last_became_active":"2023-10-16T14:11:38.799170+0000","last_became_peered":"2023-10-16T14:11:38.799170+0000","last_unstale":"2023-10-16T14:11:52.993652+0000","last_undegraded":"2023-10-16T14:11:52.993652+0000","last_fullsized":"2023-10-16T14:11:52.993652+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:23.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986664+0000","last_change":"2023-10-16T14:11:52.993056+0000","last_active":"2023-10-16T14:11:53.986664+0000","last_peered":"2023-10-16T14:11:53.986664+0000","last_clean":"2023-10-16T14:11:53.986664+0000","last_became_active":"2023-10-16T14:11:52.992768+0000","last_became_peered":"2023-10-16T14:11:52.992768+0000","last_unstale":"2023-10-16T14:11:53.986664+0000","last_undegraded":"2023-10-16T14:11:53.986664+0000","last_fullsized":"2023-10-16T14:11:53.986664+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:43.826793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.992758+0000","last_change":"2023-10-16T14:11:51.981573+0000","last_active":"2023-10-16T14:11:53.992758+0000","last_peered":"2023-10-16T14:11:53.992758+0000","last_clean":"2023-10-16T14:11:53.992758+0000","last_became_active":"2023-10-16T14:11:51.981268+0000","last_became_peered":"2023-10-16T14:11:51.981268+0000","last_unstale":"2023-10-16T14:11:53.992758+0000","last_undegraded":"2023-10-16T14:11:53.992758+0000","last_fullsized":"2023-10-16T14:11:53.992758+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:55.121470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993774+0000","last_change":"2023-10-16T14:11:38.801518+0000","last_active":"2023-10-16T14:11:52.993774+0000","last_peered":"2023-10-16T14:11:52.993774+0000","last_clean":"2023-10-16T14:11:52.993774+0000","last_became_active":"2023-10-16T14:11:38.801133+0000","last_became_peered":"2023-10-16T14:11:38.801133+0000","last_unstale":"2023-10-16T14:11:52.993774+0000","last_undegraded":"2023-10-16T14:11:52.993774+0000","last_fullsized":"2023-10-16T14:11:52.993774+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:30.423238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646309+0000","last_change":"2023-10-16T14:11:53.002757+0000","last_active":"2023-10-16T14:11:54.646309+0000","last_peered":"2023-10-16T14:11:54.646309+0000","last_clean":"2023-10-16T14:11:54.646309+0000","last_became_active":"2023-10-16T14:11:53.002500+0000","last_became_peered":"2023-10-16T14:11:53.002500+0000","last_unstale":"2023-10-16T14:11:54.646309+0000","last_undegraded":"2023-10-16T14:11:54.646309+0000","last_fullsized":"2023-10-16T14:11:54.646309+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:28.617619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718369+0000","last_change":"2023-10-16T14:11:51.977982+0000","last_active":"2023-10-16T14:11:54.718369+0000","last_peered":"2023-10-16T14:11:54.718369+0000","last_clean":"2023-10-16T14:11:54.718369+0000","last_became_active":"2023-10-16T14:11:51.977655+0000","last_became_peered":"2023-10-16T14:11:51.977655+0000","last_unstale":"2023-10-16T14:11:54.718369+0000","last_undegraded":"2023-10-16T14:11:54.718369+0000","last_fullsized":"2023-10-16T14:11:54.718369+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:44.429107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992657+0000","last_change":"2023-10-16T14:11:38.792372+0000","last_active":"2023-10-16T14:11:52.992657+0000","last_peered":"2023-10-16T14:11:52.992657+0000","last_clean":"2023-10-16T14:11:52.992657+0000","last_became_active":"2023-10-16T14:11:38.789719+0000","last_became_peered":"2023-10-16T14:11:38.789719+0000","last_unstale":"2023-10-16T14:11:52.992657+0000","last_undegraded":"2023-10-16T14:11:52.992657+0000","last_fullsized":"2023-10-16T14:11:52.992657+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:30:12.869949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987669+0000","last_change":"2023-10-16T14:11:52.979953+0000","last_active":"2023-10-16T14:11:53.987669+0000","last_peered":"2023-10-16T14:11:53.987669+0000","last_clean":"2023-10-16T14:11:53.987669+0000","last_became_active":"2023-10-16T14:11:52.979593+0000","last_became_peered":"2023-10-16T14:11:52.979593+0000","last_unstale":"2023-10-16T14:11:53.987669+0000","last_undegraded":"2023-10-16T14:11:53.987669+0000","last_fullsized":"2023-10-16T14:11:53.987669+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:07.480987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.068595+0000","last_change":"2023-10-16T14:11:51.985483+0000","last_active":"2023-10-16T14:11:55.068595+0000","last_peered":"2023-10-16T14:11:55.068595+0000","last_clean":"2023-10-16T14:11:55.068595+0000","last_became_active":"2023-10-16T14:11:51.984905+0000","last_became_peered":"2023-10-16T14:11:51.984905+0000","last_unstale":"2023-10-16T14:11:55.068595+0000","last_undegraded":"2023-10-16T14:11:55.068595+0000","last_fullsized":"2023-10-16T14:11:55.068595+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:06.178911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960933+0000","last_change":"2023-10-16T14:11:38.802052+0000","last_active":"2023-10-16T14:11:52.960933+0000","last_peered":"2023-10-16T14:11:52.960933+0000","last_clean":"2023-10-16T14:11:52.960933+0000","last_became_active":"2023-10-16T14:11:38.801625+0000","last_became_peered":"2023-10-16T14:11:38.801625+0000","last_unstale":"2023-10-16T14:11:52.960933+0000","last_undegraded":"2023-10-16T14:11:52.960933+0000","last_fullsized":"2023-10-16T14:11:52.960933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:27.067528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987192+0000","last_change":"2023-10-16T14:11:52.997042+0000","last_active":"2023-10-16T14:11:53.987192+0000","last_peered":"2023-10-16T14:11:53.987192+0000","last_clean":"2023-10-16T14:11:53.987192+0000","last_became_active":"2023-10-16T14:11:52.996733+0000","last_became_peered":"2023-10-16T14:11:52.996733+0000","last_unstale":"2023-10-16T14:11:53.987192+0000","last_undegraded":"2023-10-16T14:11:53.987192+0000","last_fullsized":"2023-10-16T14:11:53.987192+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:37.268638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"255'6","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988373+0000","last_change":"2023-10-16T14:11:51.987716+0000","last_active":"2023-10-16T14:11:53.988373+0000","last_peered":"2023-10-16T14:11:53.988373+0000","last_clean":"2023-10-16T14:11:53.988373+0000","last_became_active":"2023-10-16T14:11:51.984796+0000","last_became_peered":"2023-10-16T14:11:51.984796+0000","last_unstale":"2023-10-16T14:11:53.988373+0000","last_undegraded":"2023-10-16T14:11:53.988373+0000","last_fullsized":"2023-10-16T14:11:53.988373+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:37.378791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.989049+0000","last_change":"2023-10-16T14:11:38.797163+0000","last_active":"2023-10-16T14:11:52.989049+0000","last_peered":"2023-10-16T14:11:52.989049+0000","last_clean":"2023-10-16T14:11:52.989049+0000","last_became_active":"2023-10-16T14:11:38.796761+0000","last_became_peered":"2023-10-16T14:11:38.796761+0000","last_unstale":"2023-10-16T14:11:52.989049+0000","last_undegraded":"2023-10-16T14:11:52.989049+0000","last_fullsized":"2023-10-16T14:11:52.989049+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:01.946364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.010978+0000","last_change":"2023-10-16T14:11:53.007335+0000","last_active":"2023-10-16T14:11:53.010978+0000","last_peered":"2023-10-16T14:11:53.010978+0000","last_clean":"2023-10-16T14:11:53.010978+0000","last_became_active":"2023-10-16T14:11:53.007036+0000","last_became_peered":"2023-10-16T14:11:53.007036+0000","last_unstale":"2023-10-16T14:11:53.010978+0000","last_undegraded":"2023-10-16T14:11:53.010978+0000","last_fullsized":"2023-10-16T14:11:53.010978+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:10.543714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"255'3","reported_seq":15,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646707+0000","last_change":"2023-10-16T14:11:51.985630+0000","last_active":"2023-10-16T14:11:54.646707+0000","last_peered":"2023-10-16T14:11:54.646707+0000","last_clean":"2023-10-16T14:11:54.646707+0000","last_became_active":"2023-10-16T14:11:51.984856+0000","last_became_peered":"2023-10-16T14:11:51.984856+0000","last_unstale":"2023-10-16T14:11:54.646707+0000","last_undegraded":"2023-10-16T14:11:54.646707+0000","last_fullsized":"2023-10-16T14:11:54.646707+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:56.393116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974570+0000","last_change":"2023-10-16T14:11:38.802974+0000","last_active":"2023-10-16T14:11:52.974570+0000","last_peered":"2023-10-16T14:11:52.974570+0000","last_clean":"2023-10-16T14:11:52.974570+0000","last_became_active":"2023-10-16T14:11:38.802711+0000","last_became_peered":"2023-10-16T14:11:38.802711+0000","last_unstale":"2023-10-16T14:11:52.974570+0000","last_undegraded":"2023-10-16T14:11:52.974570+0000","last_fullsized":"2023-10-16T14:11:52.974570+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:39.092883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988308+0000","last_change":"2023-10-16T14:11:52.998872+0000","last_active":"2023-10-16T14:11:53.988308+0000","last_peered":"2023-10-16T14:11:53.988308+0000","last_clean":"2023-10-16T14:11:53.988308+0000","last_became_active":"2023-10-16T14:11:52.998518+0000","last_became_peered":"2023-10-16T14:11:52.998518+0000","last_unstale":"2023-10-16T14:11:53.988308+0000","last_undegraded":"2023-10-16T14:11:53.988308+0000","last_fullsized":"2023-10-16T14:11:53.988308+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:54.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645834+0000","last_change":"2023-10-16T14:11:51.984957+0000","last_active":"2023-10-16T14:11:54.645834+0000","last_peered":"2023-10-16T14:11:54.645834+0000","last_clean":"2023-10-16T14:11:54.645834+0000","last_became_active":"2023-10-16T14:11:51.984681+0000","last_became_peered":"2023-10-16T14:11:51.984681+0000","last_unstale":"2023-10-16T14:11:54.645834+0000","last_undegraded":"2023-10-16T14:11:54.645834+0000","last_fullsized":"2023-10-16T14:11:54.645834+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:29.591532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991986+0000","last_change":"2023-10-16T14:11:38.800092+0000","last_active":"2023-10-16T14:11:52.991986+0000","last_peered":"2023-10-16T14:11:52.991986+0000","last_clean":"2023-10-16T14:11:52.991986+0000","last_became_active":"2023-10-16T14:11:38.799625+0000","last_became_peered":"2023-10-16T14:11:38.799625+0000","last_unstale":"2023-10-16T14:11:52.991986+0000","last_undegraded":"2023-10-16T14:11:52.991986+0000","last_fullsized":"2023-10-16T14:11:52.991986+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:06.091417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994574+0000","last_change":"2023-10-16T14:11:52.984916+0000","last_active":"2023-10-16T14:11:53.994574+0000","last_peered":"2023-10-16T14:11:53.994574+0000","last_clean":"2023-10-16T14:11:53.994574+0000","last_became_active":"2023-10-16T14:11:52.984211+0000","last_became_peered":"2023-10-16T14:11:52.984211+0000","last_unstale":"2023-10-16T14:11:53.994574+0000","last_undegraded":"2023-10-16T14:11:53.994574+0000","last_fullsized":"2023-10-16T14:11:53.994574+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:42.735397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988479+0000","last_change":"2023-10-16T14:11:51.984778+0000","last_active":"2023-10-16T14:11:53.988479+0000","last_peered":"2023-10-16T14:11:53.988479+0000","last_clean":"2023-10-16T14:11:53.988479+0000","last_became_active":"2023-10-16T14:11:51.984314+0000","last_became_peered":"2023-10-16T14:11:51.984314+0000","last_unstale":"2023-10-16T14:11:53.988479+0000","last_undegraded":"2023-10-16T14:11:53.988479+0000","last_fullsized":"2023-10-16T14:11:53.988479+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:11.025122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982626+0000","last_change":"2023-10-16T14:11:38.809521+0000","last_active":"2023-10-16T14:11:52.982626+0000","last_peered":"2023-10-16T14:11:52.982626+0000","last_clean":"2023-10-16T14:11:52.982626+0000","last_became_active":"2023-10-16T14:11:38.808121+0000","last_became_peered":"2023-10-16T14:11:38.808121+0000","last_unstale":"2023-10-16T14:11:52.982626+0000","last_undegraded":"2023-10-16T14:11:52.982626+0000","last_fullsized":"2023-10-16T14:11:52.982626+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:55.831023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993398+0000","last_change":"2023-10-16T14:11:52.992961+0000","last_active":"2023-10-16T14:11:53.993398+0000","last_peered":"2023-10-16T14:11:53.993398+0000","last_clean":"2023-10-16T14:11:53.993398+0000","last_became_active":"2023-10-16T14:11:52.992670+0000","last_became_peered":"2023-10-16T14:11:52.992670+0000","last_unstale":"2023-10-16T14:11:53.993398+0000","last_undegraded":"2023-10-16T14:11:53.993398+0000","last_fullsized":"2023-10-16T14:11:53.993398+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:20.408918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995259+0000","last_change":"2023-10-16T14:11:51.989138+0000","last_active":"2023-10-16T14:11:53.995259+0000","last_peered":"2023-10-16T14:11:53.995259+0000","last_clean":"2023-10-16T14:11:53.995259+0000","last_became_active":"2023-10-16T14:11:51.988830+0000","last_became_peered":"2023-10-16T14:11:51.988830+0000","last_unstale":"2023-10-16T14:11:53.995259+0000","last_undegraded":"2023-10-16T14:11:53.995259+0000","last_fullsized":"2023-10-16T14:11:53.995259+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:35.140481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.988733+0000","last_change":"2023-10-16T14:11:38.811608+0000","last_active":"2023-10-16T14:11:51.988733+0000","last_peered":"2023-10-16T14:11:51.988733+0000","last_clean":"2023-10-16T14:11:51.988733+0000","last_became_active":"2023-10-16T14:11:38.811328+0000","last_became_peered":"2023-10-16T14:11:38.811328+0000","last_unstale":"2023-10-16T14:11:51.988733+0000","last_undegraded":"2023-10-16T14:11:51.988733+0000","last_fullsized":"2023-10-16T14:11:51.988733+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:39.866362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.644771+0000","last_change":"2023-10-16T14:11:53.001822+0000","last_active":"2023-10-16T14:11:54.644771+0000","last_peered":"2023-10-16T14:11:54.644771+0000","last_clean":"2023-10-16T14:11:54.644771+0000","last_became_active":"2023-10-16T14:11:53.001552+0000","last_became_peered":"2023-10-16T14:11:53.001552+0000","last_unstale":"2023-10-16T14:11:54.644771+0000","last_undegraded":"2023-10-16T14:11:54.644771+0000","last_fullsized":"2023-10-16T14:11:54.644771+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:19.081829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.857803+0000","last_change":"2023-10-16T14:11:51.986975+0000","last_active":"2023-10-16T14:11:54.857803+0000","last_peered":"2023-10-16T14:11:54.857803+0000","last_clean":"2023-10-16T14:11:54.857803+0000","last_became_active":"2023-10-16T14:11:51.986639+0000","last_became_peered":"2023-10-16T14:11:51.986639+0000","last_unstale":"2023-10-16T14:11:54.857803+0000","last_undegraded":"2023-10-16T14:11:54.857803+0000","last_fullsized":"2023-10-16T14:11:54.857803+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:46.717029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979167+0000","last_change":"2023-10-16T14:11:38.811158+0000","last_active":"2023-10-16T14:11:52.979167+0000","last_peered":"2023-10-16T14:11:52.979167+0000","last_clean":"2023-10-16T14:11:52.979167+0000","last_became_active":"2023-10-16T14:11:38.810852+0000","last_became_peered":"2023-10-16T14:11:38.810852+0000","last_unstale":"2023-10-16T14:11:52.979167+0000","last_undegraded":"2023-10-16T14:11:52.979167+0000","last_fullsized":"2023-10-16T14:11:52.979167+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:39.319181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646246+0000","last_change":"2023-10-16T14:11:52.998892+0000","last_active":"2023-10-16T14:11:54.646246+0000","last_peered":"2023-10-16T14:11:54.646246+0000","last_clean":"2023-10-16T14:11:54.646246+0000","last_became_active":"2023-10-16T14:11:52.998383+0000","last_became_peered":"2023-10-16T14:11:52.998383+0000","last_unstale":"2023-10-16T14:11:54.646246+0000","last_undegraded":"2023-10-16T14:11:54.646246+0000","last_fullsized":"2023-10-16T14:11:54.646246+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:30.662291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987327+0000","last_change":"2023-10-16T14:11:51.975784+0000","last_active":"2023-10-16T14:11:53.987327+0000","last_peered":"2023-10-16T14:11:53.987327+0000","last_clean":"2023-10-16T14:11:53.987327+0000","last_became_active":"2023-10-16T14:11:51.975488+0000","last_became_peered":"2023-10-16T14:11:51.975488+0000","last_unstale":"2023-10-16T14:11:53.987327+0000","last_undegraded":"2023-10-16T14:11:53.987327+0000","last_fullsized":"2023-10-16T14:11:53.987327+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:35.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974585+0000","last_change":"2023-10-16T14:11:38.804475+0000","last_active":"2023-10-16T14:11:52.974585+0000","last_peered":"2023-10-16T14:11:52.974585+0000","last_clean":"2023-10-16T14:11:52.974585+0000","last_became_active":"2023-10-16T14:11:38.804187+0000","last_became_peered":"2023-10-16T14:11:38.804187+0000","last_unstale":"2023-10-16T14:11:52.974585+0000","last_undegraded":"2023-10-16T14:11:52.974585+0000","last_fullsized":"2023-10-16T14:11:52.974585+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:59.201892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718944+0000","last_change":"2023-10-16T14:11:52.986688+0000","last_active":"2023-10-16T14:11:54.718944+0000","last_peered":"2023-10-16T14:11:54.718944+0000","last_clean":"2023-10-16T14:11:54.718944+0000","last_became_active":"2023-10-16T14:11:52.985883+0000","last_became_peered":"2023-10-16T14:11:52.985883+0000","last_unstale":"2023-10-16T14:11:54.718944+0000","last_undegraded":"2023-10-16T14:11:54.718944+0000","last_fullsized":"2023-10-16T14:11:54.718944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:51.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.996638+0000","last_change":"2023-10-16T14:11:51.982731+0000","last_active":"2023-10-16T14:11:53.996638+0000","last_peered":"2023-10-16T14:11:53.996638+0000","last_clean":"2023-10-16T14:11:53.996638+0000","last_became_active":"2023-10-16T14:11:51.982235+0000","last_became_peered":"2023-10-16T14:11:51.982235+0000","last_unstale":"2023-10-16T14:11:53.996638+0000","last_undegraded":"2023-10-16T14:11:53.996638+0000","last_fullsized":"2023-10-16T14:11:53.996638+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:44.054294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991197+0000","last_change":"2023-10-16T14:11:38.808466+0000","last_active":"2023-10-16T14:11:52.991197+0000","last_peered":"2023-10-16T14:11:52.991197+0000","last_clean":"2023-10-16T14:11:52.991197+0000","last_became_active":"2023-10-16T14:11:38.808142+0000","last_became_peered":"2023-10-16T14:11:38.808142+0000","last_unstale":"2023-10-16T14:11:52.991197+0000","last_undegraded":"2023-10-16T14:11:52.991197+0000","last_fullsized":"2023-10-16T14:11:52.991197+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:30.136850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069006+0000","last_change":"2023-10-16T14:11:52.995110+0000","last_active":"2023-10-16T14:11:55.069006+0000","last_peered":"2023-10-16T14:11:55.069006+0000","last_clean":"2023-10-16T14:11:55.069006+0000","last_became_active":"2023-10-16T14:11:52.994839+0000","last_became_peered":"2023-10-16T14:11:52.994839+0000","last_unstale":"2023-10-16T14:11:55.069006+0000","last_undegraded":"2023-10-16T14:11:55.069006+0000","last_fullsized":"2023-10-16T14:11:55.069006+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:46.167512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646679+0000","last_change":"2023-10-16T14:11:51.981639+0000","last_active":"2023-10-16T14:11:54.646679+0000","last_peered":"2023-10-16T14:11:54.646679+0000","last_clean":"2023-10-16T14:11:54.646679+0000","last_became_active":"2023-10-16T14:11:51.981366+0000","last_became_peered":"2023-10-16T14:11:51.981366+0000","last_unstale":"2023-10-16T14:11:54.646679+0000","last_undegraded":"2023-10-16T14:11:54.646679+0000","last_fullsized":"2023-10-16T14:11:54.646679+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:11.500203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974639+0000","last_change":"2023-10-16T14:11:38.802531+0000","last_active":"2023-10-16T14:11:52.974639+0000","last_peered":"2023-10-16T14:11:52.974639+0000","last_clean":"2023-10-16T14:11:52.974639+0000","last_became_active":"2023-10-16T14:11:38.802127+0000","last_became_peered":"2023-10-16T14:11:38.802127+0000","last_unstale":"2023-10-16T14:11:52.974639+0000","last_undegraded":"2023-10-16T14:11:52.974639+0000","last_fullsized":"2023-10-16T14:11:52.974639+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:26.191109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994020+0000","last_change":"2023-10-16T14:11:52.996012+0000","last_active":"2023-10-16T14:11:53.994020+0000","last_peered":"2023-10-16T14:11:53.994020+0000","last_clean":"2023-10-16T14:11:53.994020+0000","last_became_active":"2023-10-16T14:11:52.995343+0000","last_became_peered":"2023-10-16T14:11:52.995343+0000","last_unstale":"2023-10-16T14:11:53.994020+0000","last_undegraded":"2023-10-16T14:11:53.994020+0000","last_fullsized":"2023-10-16T14:11:53.994020+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:21.988424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646438+0000","last_change":"2023-10-16T14:11:51.981076+0000","last_active":"2023-10-16T14:11:54.646438+0000","last_peered":"2023-10-16T14:11:54.646438+0000","last_clean":"2023-10-16T14:11:54.646438+0000","last_became_active":"2023-10-16T14:11:51.980733+0000","last_became_peered":"2023-10-16T14:11:51.980733+0000","last_unstale":"2023-10-16T14:11:54.646438+0000","last_undegraded":"2023-10-16T14:11:54.646438+0000","last_fullsized":"2023-10-16T14:11:54.646438+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:47.589072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975151+0000","last_change":"2023-10-16T14:11:38.798475+0000","last_active":"2023-10-16T14:11:52.975151+0000","last_peered":"2023-10-16T14:11:52.975151+0000","last_clean":"2023-10-16T14:11:52.975151+0000","last_became_active":"2023-10-16T14:11:38.797198+0000","last_became_peered":"2023-10-16T14:11:38.797198+0000","last_unstale":"2023-10-16T14:11:52.975151+0000","last_undegraded":"2023-10-16T14:11:52.975151+0000","last_fullsized":"2023-10-16T14:11:52.975151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:37.361885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.005594+0000","last_change":"2023-10-16T14:11:52.999035+0000","last_active":"2023-10-16T14:11:53.005594+0000","last_peered":"2023-10-16T14:11:53.005594+0000","last_clean":"2023-10-16T14:11:53.005594+0000","last_became_active":"2023-10-16T14:11:52.998633+0000","last_became_peered":"2023-10-16T14:11:52.998633+0000","last_unstale":"2023-10-16T14:11:53.005594+0000","last_undegraded":"2023-10-16T14:11:53.005594+0000","last_fullsized":"2023-10-16T14:11:53.005594+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:40.320922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"255'1","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.010355+0000","last_change":"2023-10-16T14:11:51.990728+0000","last_active":"2023-10-16T14:11:53.010355+0000","last_peered":"2023-10-16T14:11:53.010355+0000","last_clean":"2023-10-16T14:11:53.010355+0000","last_became_active":"2023-10-16T14:11:51.990400+0000","last_became_peered":"2023-10-16T14:11:51.990400+0000","last_unstale":"2023-10-16T14:11:53.010355+0000","last_undegraded":"2023-10-16T14:11:53.010355+0000","last_fullsized":"2023-10-16T14:11:53.010355+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:53.060480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.980092+0000","last_change":"2023-10-16T14:11:38.803008+0000","last_active":"2023-10-16T14:11:51.980092+0000","last_peered":"2023-10-16T14:11:51.980092+0000","last_clean":"2023-10-16T14:11:51.980092+0000","last_became_active":"2023-10-16T14:11:38.800876+0000","last_became_peered":"2023-10-16T14:11:38.800876+0000","last_unstale":"2023-10-16T14:11:51.980092+0000","last_undegraded":"2023-10-16T14:11:51.980092+0000","last_fullsized":"2023-10-16T14:11:51.980092+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:03.954367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994944+0000","last_change":"2023-10-16T14:11:52.982991+0000","last_active":"2023-10-16T14:11:53.994944+0000","last_peered":"2023-10-16T14:11:53.994944+0000","last_clean":"2023-10-16T14:11:53.994944+0000","last_became_active":"2023-10-16T14:11:52.982265+0000","last_became_peered":"2023-10-16T14:11:52.982265+0000","last_unstale":"2023-10-16T14:11:53.994944+0000","last_undegraded":"2023-10-16T14:11:53.994944+0000","last_fullsized":"2023-10-16T14:11:53.994944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:42.388825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"255'4","reported_seq":16,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988424+0000","last_change":"2023-10-16T14:11:51.984583+0000","last_active":"2023-10-16T14:11:53.988424+0000","last_peered":"2023-10-16T14:11:53.988424+0000","last_clean":"2023-10-16T14:11:53.988424+0000","last_became_active":"2023-10-16T14:11:51.983925+0000","last_became_peered":"2023-10-16T14:11:51.983925+0000","last_unstale":"2023-10-16T14:11:53.988424+0000","last_undegraded":"2023-10-16T14:11:53.988424+0000","last_fullsized":"2023-10-16T14:11:53.988424+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:42.674017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982725+0000","last_change":"2023-10-16T14:11:38.803699+0000","last_active":"2023-10-16T14:11:52.982725+0000","last_peered":"2023-10-16T14:11:52.982725+0000","last_clean":"2023-10-16T14:11:52.982725+0000","last_became_active":"2023-10-16T14:11:38.801233+0000","last_became_peered":"2023-10-16T14:11:38.801233+0000","last_unstale":"2023-10-16T14:11:52.982725+0000","last_undegraded":"2023-10-16T14:11:52.982725+0000","last_fullsized":"2023-10-16T14:11:52.982725+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:34.743471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.067197+0000","last_change":"2023-10-16T14:11:52.994405+0000","last_active":"2023-10-16T14:11:55.067197+0000","last_peered":"2023-10-16T14:11:55.067197+0000","last_clean":"2023-10-16T14:11:55.067197+0000","last_became_active":"2023-10-16T14:11:52.994065+0000","last_became_peered":"2023-10-16T14:11:52.994065+0000","last_unstale":"2023-10-16T14:11:55.067197+0000","last_undegraded":"2023-10-16T14:11:55.067197+0000","last_fullsized":"2023-10-16T14:11:55.067197+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:55.324917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.965571+0000","last_change":"2023-10-16T14:11:38.800010+0000","last_active":"2023-10-16T14:11:52.965571+0000","last_peered":"2023-10-16T14:11:52.965571+0000","last_clean":"2023-10-16T14:11:52.965571+0000","last_became_active":"2023-10-16T14:11:38.799422+0000","last_became_peered":"2023-10-16T14:11:38.799422+0000","last_unstale":"2023-10-16T14:11:52.965571+0000","last_undegraded":"2023-10-16T14:11:52.965571+0000","last_fullsized":"2023-10-16T14:11:52.965571+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:01:31.181539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979354+0000","last_change":"2023-10-16T14:11:37.778697+0000","last_active":"2023-10-16T14:11:52.979354+0000","last_peered":"2023-10-16T14:11:52.979354+0000","last_clean":"2023-10-16T14:11:52.979354+0000","last_became_active":"2023-10-16T14:11:37.778387+0000","last_became_peered":"2023-10-16T14:11:37.778387+0000","last_unstale":"2023-10-16T14:11:52.979354+0000","last_undegraded":"2023-10-16T14:11:52.979354+0000","last_fullsized":"2023-10-16T14:11:52.979354+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:09.302616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.011033+0000","last_change":"2023-10-16T14:11:53.007479+0000","last_active":"2023-10-16T14:11:53.011033+0000","last_peered":"2023-10-16T14:11:53.011033+0000","last_clean":"2023-10-16T14:11:53.011033+0000","last_became_active":"2023-10-16T14:11:53.006978+0000","last_became_peered":"2023-10-16T14:11:53.006978+0000","last_unstale":"2023-10-16T14:11:53.011033+0000","last_undegraded":"2023-10-16T14:11:53.011033+0000","last_fullsized":"2023-10-16T14:11:53.011033+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:10.030122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.972830+0000","last_change":"2023-10-16T14:11:38.805302+0000","last_active":"2023-10-16T14:11:52.972830+0000","last_peered":"2023-10-16T14:11:52.972830+0000","last_clean":"2023-10-16T14:11:52.972830+0000","last_became_active":"2023-10-16T14:11:38.803432+0000","last_became_peered":"2023-10-16T14:11:38.803432+0000","last_unstale":"2023-10-16T14:11:52.972830+0000","last_undegraded":"2023-10-16T14:11:52.972830+0000","last_fullsized":"2023-10-16T14:11:52.972830+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:42.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959770+0000","last_change":"2023-10-16T14:11:37.781195+0000","last_active":"2023-10-16T14:11:52.959770+0000","last_peered":"2023-10-16T14:11:52.959770+0000","last_clean":"2023-10-16T14:11:52.959770+0000","last_became_active":"2023-10-16T14:11:37.780551+0000","last_became_peered":"2023-10-16T14:11:37.780551+0000","last_unstale":"2023-10-16T14:11:52.959770+0000","last_undegraded":"2023-10-16T14:11:52.959770+0000","last_fullsized":"2023-10-16T14:11:52.959770+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:33.391410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987987+0000","last_change":"2023-10-16T14:11:52.989049+0000","last_active":"2023-10-16T14:11:53.987987+0000","last_peered":"2023-10-16T14:11:53.987987+0000","last_clean":"2023-10-16T14:11:53.987987+0000","last_became_active":"2023-10-16T14:11:52.988719+0000","last_became_peered":"2023-10-16T14:11:52.988719+0000","last_unstale":"2023-10-16T14:11:53.987987+0000","last_undegraded":"2023-10-16T14:11:53.987987+0000","last_fullsized":"2023-10-16T14:11:53.987987+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.795278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992813+0000","last_change":"2023-10-16T14:11:38.802734+0000","last_active":"2023-10-16T14:11:52.992813+0000","last_peered":"2023-10-16T14:11:52.992813+0000","last_clean":"2023-10-16T14:11:52.992813+0000","last_became_active":"2023-10-16T14:11:38.802347+0000","last_became_peered":"2023-10-16T14:11:38.802347+0000","last_unstale":"2023-10-16T14:11:52.992813+0000","last_undegraded":"2023-10-16T14:11:52.992813+0000","last_fullsized":"2023-10-16T14:11:52.992813+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:38.977236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961429+0000","last_change":"2023-10-16T14:11:37.783063+0000","last_active":"2023-10-16T14:11:52.961429+0000","last_peered":"2023-10-16T14:11:52.961429+0000","last_clean":"2023-10-16T14:11:52.961429+0000","last_became_active":"2023-10-16T14:11:37.782768+0000","last_became_peered":"2023-10-16T14:11:37.782768+0000","last_unstale":"2023-10-16T14:11:52.961429+0000","last_undegraded":"2023-10-16T14:11:52.961429+0000","last_fullsized":"2023-10-16T14:11:52.961429+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:36.980155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993428+0000","last_change":"2023-10-16T14:11:52.989636+0000","last_active":"2023-10-16T14:11:52.993428+0000","last_peered":"2023-10-16T14:11:52.993428+0000","last_clean":"2023-10-16T14:11:52.993428+0000","last_became_active":"2023-10-16T14:11:52.989294+0000","last_became_peered":"2023-10-16T14:11:52.989294+0000","last_unstale":"2023-10-16T14:11:52.993428+0000","last_undegraded":"2023-10-16T14:11:52.993428+0000","last_fullsized":"2023-10-16T14:11:52.993428+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:09.306153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973007+0000","last_change":"2023-10-16T14:11:38.786427+0000","last_active":"2023-10-16T14:11:52.973007+0000","last_peered":"2023-10-16T14:11:52.973007+0000","last_clean":"2023-10-16T14:11:52.973007+0000","last_became_active":"2023-10-16T14:11:38.785671+0000","last_became_peered":"2023-10-16T14:11:38.785671+0000","last_unstale":"2023-10-16T14:11:52.973007+0000","last_undegraded":"2023-10-16T14:11:52.973007+0000","last_fullsized":"2023-10-16T14:11:52.973007+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:08.101378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"251'1","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.984174+0000","last_change":"2023-10-16T14:11:37.774757+0000","last_active":"2023-10-16T14:11:51.984174+0000","last_peered":"2023-10-16T14:11:51.984174+0000","last_clean":"2023-10-16T14:11:51.984174+0000","last_became_active":"2023-10-16T14:11:37.774442+0000","last_became_peered":"2023-10-16T14:11:37.774442+0000","last_unstale":"2023-10-16T14:11:51.984174+0000","last_undegraded":"2023-10-16T14:11:51.984174+0000","last_fullsized":"2023-10-16T14:11:51.984174+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:45.874083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994379+0000","last_change":"2023-10-16T14:11:52.981497+0000","last_active":"2023-10-16T14:11:53.994379+0000","last_peered":"2023-10-16T14:11:53.994379+0000","last_clean":"2023-10-16T14:11:53.994379+0000","last_became_active":"2023-10-16T14:11:52.979769+0000","last_became_peered":"2023-10-16T14:11:52.979769+0000","last_unstale":"2023-10-16T14:11:53.994379+0000","last_undegraded":"2023-10-16T14:11:53.994379+0000","last_fullsized":"2023-10-16T14:11:53.994379+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:58:25.884035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975312+0000","last_change":"2023-10-16T14:11:38.809976+0000","last_active":"2023-10-16T14:11:52.975312+0000","last_peered":"2023-10-16T14:11:52.975312+0000","last_clean":"2023-10-16T14:11:52.975312+0000","last_became_active":"2023-10-16T14:11:38.809754+0000","last_became_peered":"2023-10-16T14:11:38.809754+0000","last_unstale":"2023-10-16T14:11:52.975312+0000","last_undegraded":"2023-10-16T14:11:52.975312+0000","last_fullsized":"2023-10-16T14:11:52.975312+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:14.418521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.984993+0000","last_change":"2023-10-16T14:11:37.776847+0000","last_active":"2023-10-16T14:11:52.984993+0000","last_peered":"2023-10-16T14:11:52.984993+0000","last_clean":"2023-10-16T14:11:52.984993+0000","last_became_active":"2023-10-16T14:11:37.775028+0000","last_became_peered":"2023-10-16T14:11:37.775028+0000","last_unstale":"2023-10-16T14:11:52.984993+0000","last_undegraded":"2023-10-16T14:11:52.984993+0000","last_fullsized":"2023-10-16T14:11:52.984993+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:54.089883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858815+0000","last_change":"2023-10-16T14:11:52.992668+0000","last_active":"2023-10-16T14:11:54.858815+0000","last_peered":"2023-10-16T14:11:54.858815+0000","last_clean":"2023-10-16T14:11:54.858815+0000","last_became_active":"2023-10-16T14:11:52.990013+0000","last_became_peered":"2023-10-16T14:11:52.990013+0000","last_unstale":"2023-10-16T14:11:54.858815+0000","last_undegraded":"2023-10-16T14:11:54.858815+0000","last_fullsized":"2023-10-16T14:11:54.858815+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:08.734684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960073+0000","last_change":"2023-10-16T14:11:38.801326+0000","last_active":"2023-10-16T14:11:52.960073+0000","last_peered":"2023-10-16T14:11:52.960073+0000","last_clean":"2023-10-16T14:11:52.960073+0000","last_became_active":"2023-10-16T14:11:38.800581+0000","last_became_peered":"2023-10-16T14:11:38.800581+0000","last_unstale":"2023-10-16T14:11:52.960073+0000","last_undegraded":"2023-10-16T14:11:52.960073+0000","last_fullsized":"2023-10-16T14:11:52.960073+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:09.367409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974977+0000","last_change":"2023-10-16T14:11:37.775158+0000","last_active":"2023-10-16T14:11:52.974977+0000","last_peered":"2023-10-16T14:11:52.974977+0000","last_clean":"2023-10-16T14:11:52.974977+0000","last_became_active":"2023-10-16T14:11:37.773264+0000","last_became_peered":"2023-10-16T14:11:37.773264+0000","last_unstale":"2023-10-16T14:11:52.974977+0000","last_undegraded":"2023-10-16T14:11:52.974977+0000","last_fullsized":"2023-10-16T14:11:52.974977+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:27.581413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993902+0000","last_change":"2023-10-16T14:11:52.988830+0000","last_active":"2023-10-16T14:11:52.993902+0000","last_peered":"2023-10-16T14:11:52.993902+0000","last_clean":"2023-10-16T14:11:52.993902+0000","last_became_active":"2023-10-16T14:11:52.988543+0000","last_became_peered":"2023-10-16T14:11:52.988543+0000","last_unstale":"2023-10-16T14:11:52.993902+0000","last_undegraded":"2023-10-16T14:11:52.993902+0000","last_fullsized":"2023-10-16T14:11:52.993902+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:46.167512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982676+0000","last_change":"2023-10-16T14:11:38.802528+0000","last_active":"2023-10-16T14:11:52.982676+0000","last_peered":"2023-10-16T14:11:52.982676+0000","last_clean":"2023-10-16T14:11:52.982676+0000","last_became_active":"2023-10-16T14:11:38.802212+0000","last_became_peered":"2023-10-16T14:11:38.802212+0000","last_unstale":"2023-10-16T14:11:52.982676+0000","last_undegraded":"2023-10-16T14:11:52.982676+0000","last_fullsized":"2023-10-16T14:11:52.982676+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:04.028597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960624+0000","last_change":"2023-10-16T14:11:37.780327+0000","last_active":"2023-10-16T14:11:52.960624+0000","last_peered":"2023-10-16T14:11:52.960624+0000","last_clean":"2023-10-16T14:11:52.960624+0000","last_became_active":"2023-10-16T14:11:37.779846+0000","last_became_peered":"2023-10-16T14:11:37.779846+0000","last_unstale":"2023-10-16T14:11:52.960624+0000","last_undegraded":"2023-10-16T14:11:52.960624+0000","last_fullsized":"2023-10-16T14:11:52.960624+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:15.446021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986572+0000","last_change":"2023-10-16T14:11:52.987389+0000","last_active":"2023-10-16T14:11:53.986572+0000","last_peered":"2023-10-16T14:11:53.986572+0000","last_clean":"2023-10-16T14:11:53.986572+0000","last_became_active":"2023-10-16T14:11:52.981453+0000","last_became_peered":"2023-10-16T14:11:52.981453+0000","last_unstale":"2023-10-16T14:11:53.986572+0000","last_undegraded":"2023-10-16T14:11:53.986572+0000","last_fullsized":"2023-10-16T14:11:53.986572+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:33.020478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993860+0000","last_change":"2023-10-16T14:11:38.809523+0000","last_active":"2023-10-16T14:11:52.993860+0000","last_peered":"2023-10-16T14:11:52.993860+0000","last_clean":"2023-10-16T14:11:52.993860+0000","last_became_active":"2023-10-16T14:11:38.807663+0000","last_became_peered":"2023-10-16T14:11:38.807663+0000","last_unstale":"2023-10-16T14:11:52.993860+0000","last_undegraded":"2023-10-16T14:11:52.993860+0000","last_fullsized":"2023-10-16T14:11:52.993860+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:55.839639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993751+0000","last_change":"2023-10-16T14:11:37.784204+0000","last_active":"2023-10-16T14:11:52.993751+0000","last_peered":"2023-10-16T14:11:52.993751+0000","last_clean":"2023-10-16T14:11:52.993751+0000","last_became_active":"2023-10-16T14:11:37.783880+0000","last_became_peered":"2023-10-16T14:11:37.783880+0000","last_unstale":"2023-10-16T14:11:52.993751+0000","last_undegraded":"2023-10-16T14:11:52.993751+0000","last_fullsized":"2023-10-16T14:11:52.993751+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:22.672658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993206+0000","last_change":"2023-10-16T14:11:52.985795+0000","last_active":"2023-10-16T14:11:53.993206+0000","last_peered":"2023-10-16T14:11:53.993206+0000","last_clean":"2023-10-16T14:11:53.993206+0000","last_became_active":"2023-10-16T14:11:52.985504+0000","last_became_peered":"2023-10-16T14:11:52.985504+0000","last_unstale":"2023-10-16T14:11:53.993206+0000","last_undegraded":"2023-10-16T14:11:53.993206+0000","last_fullsized":"2023-10-16T14:11:53.993206+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:46.426090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976954+0000","last_change":"2023-10-16T14:11:38.802292+0000","last_active":"2023-10-16T14:11:52.976954+0000","last_peered":"2023-10-16T14:11:52.976954+0000","last_clean":"2023-10-16T14:11:52.976954+0000","last_became_active":"2023-10-16T14:11:38.801904+0000","last_became_peered":"2023-10-16T14:11:38.801904+0000","last_unstale":"2023-10-16T14:11:52.976954+0000","last_undegraded":"2023-10-16T14:11:52.976954+0000","last_fullsized":"2023-10-16T14:11:52.976954+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:44:48.663239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959886+0000","last_change":"2023-10-16T14:11:37.774897+0000","last_active":"2023-10-16T14:11:52.959886+0000","last_peered":"2023-10-16T14:11:52.959886+0000","last_clean":"2023-10-16T14:11:52.959886+0000","last_became_active":"2023-10-16T14:11:37.774465+0000","last_became_peered":"2023-10-16T14:11:37.774465+0000","last_unstale":"2023-10-16T14:11:52.959886+0000","last_undegraded":"2023-10-16T14:11:52.959886+0000","last_fullsized":"2023-10-16T14:11:52.959886+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718522+0000","last_change":"2023-10-16T14:11:52.998125+0000","last_active":"2023-10-16T14:11:54.718522+0000","last_peered":"2023-10-16T14:11:54.718522+0000","last_clean":"2023-10-16T14:11:54.718522+0000","last_became_active":"2023-10-16T14:11:52.997853+0000","last_became_peered":"2023-10-16T14:11:52.997853+0000","last_unstale":"2023-10-16T14:11:54.718522+0000","last_undegraded":"2023-10-16T14:11:54.718522+0000","last_fullsized":"2023-10-16T14:11:54.718522+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:30.650999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961151+0000","last_change":"2023-10-16T14:11:38.809093+0000","last_active":"2023-10-16T14:11:52.961151+0000","last_peered":"2023-10-16T14:11:52.961151+0000","last_clean":"2023-10-16T14:11:52.961151+0000","last_became_active":"2023-10-16T14:11:38.808829+0000","last_became_peered":"2023-10-16T14:11:38.808829+0000","last_unstale":"2023-10-16T14:11:52.961151+0000","last_undegraded":"2023-10-16T14:11:52.961151+0000","last_fullsized":"2023-10-16T14:11:52.961151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:09.522737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"251'3","reported_seq":19,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992384+0000","last_change":"2023-10-16T14:11:37.772650+0000","last_active":"2023-10-16T14:11:52.992384+0000","last_peered":"2023-10-16T14:11:52.992384+0000","last_clean":"2023-10-16T14:11:52.992384+0000","last_became_active":"2023-10-16T14:11:37.772209+0000","last_became_peered":"2023-10-16T14:11:37.772209+0000","last_unstale":"2023-10-16T14:11:52.992384+0000","last_undegraded":"2023-10-16T14:11:52.992384+0000","last_fullsized":"2023-10-16T14:11:52.992384+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:51.981249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718522+0000","last_change":"2023-10-16T14:11:52.983324+0000","last_active":"2023-10-16T14:11:54.718522+0000","last_peered":"2023-10-16T14:11:54.718522+0000","last_clean":"2023-10-16T14:11:54.718522+0000","last_became_active":"2023-10-16T14:11:52.982503+0000","last_became_peered":"2023-10-16T14:11:52.982503+0000","last_unstale":"2023-10-16T14:11:54.718522+0000","last_undegraded":"2023-10-16T14:11:54.718522+0000","last_fullsized":"2023-10-16T14:11:54.718522+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:37.888928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976458+0000","last_change":"2023-10-16T14:11:38.800005+0000","last_active":"2023-10-16T14:11:52.976458+0000","last_peered":"2023-10-16T14:11:52.976458+0000","last_clean":"2023-10-16T14:11:52.976458+0000","last_became_active":"2023-10-16T14:11:38.799654+0000","last_became_peered":"2023-10-16T14:11:38.799654+0000","last_unstale":"2023-10-16T14:11:52.976458+0000","last_undegraded":"2023-10-16T14:11:52.976458+0000","last_fullsized":"2023-10-16T14:11:52.976458+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:59.978269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"254'8","reported_seq":24,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991307+0000","last_change":"2023-10-16T14:11:37.783855+0000","last_active":"2023-10-16T14:11:52.991307+0000","last_peered":"2023-10-16T14:11:52.991307+0000","last_clean":"2023-10-16T14:11:52.991307+0000","last_became_active":"2023-10-16T14:11:37.783474+0000","last_became_peered":"2023-10-16T14:11:37.783474+0000","last_unstale":"2023-10-16T14:11:52.991307+0000","last_undegraded":"2023-10-16T14:11:52.991307+0000","last_fullsized":"2023-10-16T14:11:52.991307+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645849+0000","last_change":"2023-10-16T14:11:52.994061+0000","last_active":"2023-10-16T14:11:54.645849+0000","last_peered":"2023-10-16T14:11:54.645849+0000","last_clean":"2023-10-16T14:11:54.645849+0000","last_became_active":"2023-10-16T14:11:52.993800+0000","last_became_peered":"2023-10-16T14:11:52.993800+0000","last_unstale":"2023-10-16T14:11:54.645849+0000","last_undegraded":"2023-10-16T14:11:54.645849+0000","last_fullsized":"2023-10-16T14:11:54.645849+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:11.479004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974739+0000","last_change":"2023-10-16T14:11:38.796891+0000","last_active":"2023-10-16T14:11:52.974739+0000","last_peered":"2023-10-16T14:11:52.974739+0000","last_clean":"2023-10-16T14:11:52.974739+0000","last_became_active":"2023-10-16T14:11:38.796574+0000","last_became_peered":"2023-10-16T14:11:38.796574+0000","last_unstale":"2023-10-16T14:11:52.974739+0000","last_undegraded":"2023-10-16T14:11:52.974739+0000","last_fullsized":"2023-10-16T14:11:52.974739+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:06.157168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992688+0000","last_change":"2023-10-16T14:11:37.767366+0000","last_active":"2023-10-16T14:11:52.992688+0000","last_peered":"2023-10-16T14:11:52.992688+0000","last_clean":"2023-10-16T14:11:52.992688+0000","last_became_active":"2023-10-16T14:11:37.766979+0000","last_became_peered":"2023-10-16T14:11:37.766979+0000","last_unstale":"2023-10-16T14:11:52.992688+0000","last_undegraded":"2023-10-16T14:11:52.992688+0000","last_fullsized":"2023-10-16T14:11:52.992688+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:25.104652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994154+0000","last_change":"2023-10-16T14:11:52.987377+0000","last_active":"2023-10-16T14:11:53.994154+0000","last_peered":"2023-10-16T14:11:53.994154+0000","last_clean":"2023-10-16T14:11:53.994154+0000","last_became_active":"2023-10-16T14:11:52.987074+0000","last_became_peered":"2023-10-16T14:11:52.987074+0000","last_unstale":"2023-10-16T14:11:53.994154+0000","last_undegraded":"2023-10-16T14:11:53.994154+0000","last_fullsized":"2023-10-16T14:11:53.994154+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:39.310420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975506+0000","last_change":"2023-10-16T14:11:38.801883+0000","last_active":"2023-10-16T14:11:52.975506+0000","last_peered":"2023-10-16T14:11:52.975506+0000","last_clean":"2023-10-16T14:11:52.975506+0000","last_became_active":"2023-10-16T14:11:38.800123+0000","last_became_peered":"2023-10-16T14:11:38.800123+0000","last_unstale":"2023-10-16T14:11:52.975506+0000","last_undegraded":"2023-10-16T14:11:52.975506+0000","last_fullsized":"2023-10-16T14:11:52.975506+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:22.442052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961094+0000","last_change":"2023-10-16T14:11:37.779309+0000","last_active":"2023-10-16T14:11:52.961094+0000","last_peered":"2023-10-16T14:11:52.961094+0000","last_clean":"2023-10-16T14:11:52.961094+0000","last_became_active":"2023-10-16T14:11:37.778681+0000","last_became_peered":"2023-10-16T14:11:37.778681+0000","last_unstale":"2023-10-16T14:11:52.961094+0000","last_undegraded":"2023-10-16T14:11:52.961094+0000","last_fullsized":"2023-10-16T14:11:52.961094+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:41.372800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995674+0000","last_change":"2023-10-16T14:11:52.991729+0000","last_active":"2023-10-16T14:11:53.995674+0000","last_peered":"2023-10-16T14:11:53.995674+0000","last_clean":"2023-10-16T14:11:53.995674+0000","last_became_active":"2023-10-16T14:11:52.991418+0000","last_became_peered":"2023-10-16T14:11:52.991418+0000","last_unstale":"2023-10-16T14:11:53.995674+0000","last_undegraded":"2023-10-16T14:11:53.995674+0000","last_fullsized":"2023-10-16T14:11:53.995674+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:01.672958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.979384+0000","last_change":"2023-10-16T14:11:38.799773+0000","last_active":"2023-10-16T14:11:51.979384+0000","last_peered":"2023-10-16T14:11:51.979384+0000","last_clean":"2023-10-16T14:11:51.979384+0000","last_became_active":"2023-10-16T14:11:38.799446+0000","last_became_peered":"2023-10-16T14:11:38.799446+0000","last_unstale":"2023-10-16T14:11:51.979384+0000","last_undegraded":"2023-10-16T14:11:51.979384+0000","last_fullsized":"2023-10-16T14:11:51.979384+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:16.652122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975194+0000","last_change":"2023-10-16T14:11:37.777163+0000","last_active":"2023-10-16T14:11:52.975194+0000","last_peered":"2023-10-16T14:11:52.975194+0000","last_clean":"2023-10-16T14:11:52.975194+0000","last_became_active":"2023-10-16T14:11:37.776875+0000","last_became_peered":"2023-10-16T14:11:37.776875+0000","last_unstale":"2023-10-16T14:11:52.975194+0000","last_undegraded":"2023-10-16T14:11:52.975194+0000","last_fullsized":"2023-10-16T14:11:52.975194+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:59.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988598+0000","last_change":"2023-10-16T14:11:52.989767+0000","last_active":"2023-10-16T14:11:53.988598+0000","last_peered":"2023-10-16T14:11:53.988598+0000","last_clean":"2023-10-16T14:11:53.988598+0000","last_became_active":"2023-10-16T14:11:52.989398+0000","last_became_peered":"2023-10-16T14:11:52.989398+0000","last_unstale":"2023-10-16T14:11:53.988598+0000","last_undegraded":"2023-10-16T14:11:53.988598+0000","last_fullsized":"2023-10-16T14:11:53.988598+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:09.136777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974747+0000","last_change":"2023-10-16T14:11:38.801468+0000","last_active":"2023-10-16T14:11:52.974747+0000","last_peered":"2023-10-16T14:11:52.974747+0000","last_clean":"2023-10-16T14:11:52.974747+0000","last_became_active":"2023-10-16T14:11:38.801182+0000","last_became_peered":"2023-10-16T14:11:38.801182+0000","last_unstale":"2023-10-16T14:11:52.974747+0000","last_undegraded":"2023-10-16T14:11:52.974747+0000","last_fullsized":"2023-10-16T14:11:52.974747+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:47.695208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"251'8","reported_seq":24,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.967657+0000","last_change":"2023-10-16T14:11:37.784533+0000","last_active":"2023-10-16T14:11:52.967657+0000","last_peered":"2023-10-16T14:11:52.967657+0000","last_clean":"2023-10-16T14:11:52.967657+0000","last_became_active":"2023-10-16T14:11:37.784209+0000","last_became_peered":"2023-10-16T14:11:37.784209+0000","last_unstale":"2023-10-16T14:11:52.967657+0000","last_undegraded":"2023-10-16T14:11:52.967657+0000","last_fullsized":"2023-10-16T14:11:52.967657+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:49.027722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.644770+0000","last_change":"2023-10-16T14:11:52.990437+0000","last_active":"2023-10-16T14:11:54.644770+0000","last_peered":"2023-10-16T14:11:54.644770+0000","last_clean":"2023-10-16T14:11:54.644770+0000","last_became_active":"2023-10-16T14:11:52.989978+0000","last_became_peered":"2023-10-16T14:11:52.989978+0000","last_unstale":"2023-10-16T14:11:54.644770+0000","last_undegraded":"2023-10-16T14:11:54.644770+0000","last_fullsized":"2023-10-16T14:11:54.644770+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:27.613956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.986198+0000","last_change":"2023-10-16T14:11:38.810290+0000","last_active":"2023-10-16T14:11:51.986198+0000","last_peered":"2023-10-16T14:11:51.986198+0000","last_clean":"2023-10-16T14:11:51.986198+0000","last_became_active":"2023-10-16T14:11:38.809830+0000","last_became_peered":"2023-10-16T14:11:38.809830+0000","last_unstale":"2023-10-16T14:11:51.986198+0000","last_undegraded":"2023-10-16T14:11:51.986198+0000","last_fullsized":"2023-10-16T14:11:51.986198+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:31.792439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.978352+0000","last_change":"2023-10-16T14:11:37.786947+0000","last_active":"2023-10-16T14:11:52.978352+0000","last_peered":"2023-10-16T14:11:52.978352+0000","last_clean":"2023-10-16T14:11:52.978352+0000","last_became_active":"2023-10-16T14:11:37.786340+0000","last_became_peered":"2023-10-16T14:11:37.786340+0000","last_unstale":"2023-10-16T14:11:52.978352+0000","last_undegraded":"2023-10-16T14:11:52.978352+0000","last_fullsized":"2023-10-16T14:11:52.978352+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:01.576946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.003171+0000","last_change":"2023-10-16T14:11:53.000467+0000","last_active":"2023-10-16T14:11:53.003171+0000","last_peered":"2023-10-16T14:11:53.003171+0000","last_clean":"2023-10-16T14:11:53.003171+0000","last_became_active":"2023-10-16T14:11:52.998042+0000","last_became_peered":"2023-10-16T14:11:52.998042+0000","last_unstale":"2023-10-16T14:11:53.003171+0000","last_undegraded":"2023-10-16T14:11:53.003171+0000","last_fullsized":"2023-10-16T14:11:53.003171+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:55.324917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.966150+0000","last_change":"2023-10-16T14:11:38.804059+0000","last_active":"2023-10-16T14:11:52.966150+0000","last_peered":"2023-10-16T14:11:52.966150+0000","last_clean":"2023-10-16T14:11:52.966150+0000","last_became_active":"2023-10-16T14:11:38.803768+0000","last_became_peered":"2023-10-16T14:11:38.803768+0000","last_unstale":"2023-10-16T14:11:52.966150+0000","last_undegraded":"2023-10-16T14:11:52.966150+0000","last_fullsized":"2023-10-16T14:11:52.966150+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:40.313169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960155+0000","last_change":"2023-10-16T14:11:37.780012+0000","last_active":"2023-10-16T14:11:52.960155+0000","last_peered":"2023-10-16T14:11:52.960155+0000","last_clean":"2023-10-16T14:11:52.960155+0000","last_became_active":"2023-10-16T14:11:37.779730+0000","last_became_peered":"2023-10-16T14:11:37.779730+0000","last_unstale":"2023-10-16T14:11:52.960155+0000","last_undegraded":"2023-10-16T14:11:52.960155+0000","last_fullsized":"2023-10-16T14:11:52.960155+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.010982+0000","last_change":"2023-10-16T14:11:53.007299+0000","last_active":"2023-10-16T14:11:53.010982+0000","last_peered":"2023-10-16T14:11:53.010982+0000","last_clean":"2023-10-16T14:11:53.010982+0000","last_became_active":"2023-10-16T14:11:53.006618+0000","last_became_peered":"2023-10-16T14:11:53.006618+0000","last_unstale":"2023-10-16T14:11:53.010982+0000","last_undegraded":"2023-10-16T14:11:53.010982+0000","last_fullsized":"2023-10-16T14:11:53.010982+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:22.430811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973640+0000","last_change":"2023-10-16T14:11:38.804003+0000","last_active":"2023-10-16T14:11:52.973640+0000","last_peered":"2023-10-16T14:11:52.973640+0000","last_clean":"2023-10-16T14:11:52.973640+0000","last_became_active":"2023-10-16T14:11:38.803711+0000","last_became_peered":"2023-10-16T14:11:38.803711+0000","last_unstale":"2023-10-16T14:11:52.973640+0000","last_undegraded":"2023-10-16T14:11:52.973640+0000","last_fullsized":"2023-10-16T14:11:52.973640+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:57.279906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976637+0000","last_change":"2023-10-16T14:11:37.784775+0000","last_active":"2023-10-16T14:11:52.976637+0000","last_peered":"2023-10-16T14:11:52.976637+0000","last_clean":"2023-10-16T14:11:52.976637+0000","last_became_active":"2023-10-16T14:11:37.784340+0000","last_became_peered":"2023-10-16T14:11:37.784340+0000","last_unstale":"2023-10-16T14:11:52.976637+0000","last_undegraded":"2023-10-16T14:11:52.976637+0000","last_fullsized":"2023-10-16T14:11:52.976637+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993391+0000","last_change":"2023-10-16T14:11:52.990908+0000","last_active":"2023-10-16T14:11:52.993391+0000","last_peered":"2023-10-16T14:11:52.993391+0000","last_clean":"2023-10-16T14:11:52.993391+0000","last_became_active":"2023-10-16T14:11:52.990599+0000","last_became_peered":"2023-10-16T14:11:52.990599+0000","last_unstale":"2023-10-16T14:11:52.993391+0000","last_undegraded":"2023-10-16T14:11:52.993391+0000","last_fullsized":"2023-10-16T14:11:52.993391+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:15.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992965+0000","last_change":"2023-10-16T14:11:38.797986+0000","last_active":"2023-10-16T14:11:52.992965+0000","last_peered":"2023-10-16T14:11:52.992965+0000","last_clean":"2023-10-16T14:11:52.992965+0000","last_became_active":"2023-10-16T14:11:38.797632+0000","last_became_peered":"2023-10-16T14:11:38.797632+0000","last_unstale":"2023-10-16T14:11:52.992965+0000","last_undegraded":"2023-10-16T14:11:52.992965+0000","last_fullsized":"2023-10-16T14:11:52.992965+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:33.536538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973302+0000","last_change":"2023-10-16T14:11:37.772519+0000","last_active":"2023-10-16T14:11:52.973302+0000","last_peered":"2023-10-16T14:11:52.973302+0000","last_clean":"2023-10-16T14:11:52.973302+0000","last_became_active":"2023-10-16T14:11:37.772241+0000","last_became_peered":"2023-10-16T14:11:37.772241+0000","last_unstale":"2023-10-16T14:11:52.973302+0000","last_undegraded":"2023-10-16T14:11:52.973302+0000","last_fullsized":"2023-10-16T14:11:52.973302+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:47.596657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069291+0000","last_change":"2023-10-16T14:11:52.996241+0000","last_active":"2023-10-16T14:11:55.069291+0000","last_peered":"2023-10-16T14:11:55.069291+0000","last_clean":"2023-10-16T14:11:55.069291+0000","last_became_active":"2023-10-16T14:11:52.995937+0000","last_became_peered":"2023-10-16T14:11:52.995937+0000","last_unstale":"2023-10-16T14:11:55.069291+0000","last_undegraded":"2023-10-16T14:11:55.069291+0000","last_fullsized":"2023-10-16T14:11:55.069291+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:23.097797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.990766+0000","last_change":"2023-10-16T14:11:38.801767+0000","last_active":"2023-10-16T14:11:52.990766+0000","last_peered":"2023-10-16T14:11:52.990766+0000","last_clean":"2023-10-16T14:11:52.990766+0000","last_became_active":"2023-10-16T14:11:38.800797+0000","last_became_peered":"2023-10-16T14:11:38.800797+0000","last_unstale":"2023-10-16T14:11:52.990766+0000","last_undegraded":"2023-10-16T14:11:52.990766+0000","last_fullsized":"2023-10-16T14:11:52.990766+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:05.512554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961589+0000","last_change":"2023-10-16T14:11:37.781065+0000","last_active":"2023-10-16T14:11:52.961589+0000","last_peered":"2023-10-16T14:11:52.961589+0000","last_clean":"2023-10-16T14:11:52.961589+0000","last_became_active":"2023-10-16T14:11:37.780556+0000","last_became_peered":"2023-10-16T14:11:37.780556+0000","last_unstale":"2023-10-16T14:11:52.961589+0000","last_undegraded":"2023-10-16T14:11:52.961589+0000","last_fullsized":"2023-10-16T14:11:52.961589+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:03.865559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993948+0000","last_change":"2023-10-16T14:11:53.001652+0000","last_active":"2023-10-16T14:11:53.993948+0000","last_peered":"2023-10-16T14:11:53.993948+0000","last_clean":"2023-10-16T14:11:53.993948+0000","last_became_active":"2023-10-16T14:11:53.001363+0000","last_became_peered":"2023-10-16T14:11:53.001363+0000","last_unstale":"2023-10-16T14:11:53.993948+0000","last_undegraded":"2023-10-16T14:11:53.993948+0000","last_fullsized":"2023-10-16T14:11:53.993948+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:25.562353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992641+0000","last_change":"2023-10-16T14:11:38.803116+0000","last_active":"2023-10-16T14:11:52.992641+0000","last_peered":"2023-10-16T14:11:52.992641+0000","last_clean":"2023-10-16T14:11:52.992641+0000","last_became_active":"2023-10-16T14:11:38.802518+0000","last_became_peered":"2023-10-16T14:11:38.802518+0000","last_unstale":"2023-10-16T14:11:52.992641+0000","last_undegraded":"2023-10-16T14:11:52.992641+0000","last_fullsized":"2023-10-16T14:11:52.992641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:27.765461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973148+0000","last_change":"2023-10-16T14:11:37.768392+0000","last_active":"2023-10-16T14:11:52.973148+0000","last_peered":"2023-10-16T14:11:52.973148+0000","last_clean":"2023-10-16T14:11:52.973148+0000","last_became_active":"2023-10-16T14:11:37.768081+0000","last_became_peered":"2023-10-16T14:11:37.768081+0000","last_unstale":"2023-10-16T14:11:52.973148+0000","last_undegraded":"2023-10-16T14:11:52.973148+0000","last_fullsized":"2023-10-16T14:11:52.973148+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:13.231020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994164+0000","last_change":"2023-10-16T14:11:52.985092+0000","last_active":"2023-10-16T14:11:53.994164+0000","last_peered":"2023-10-16T14:11:53.994164+0000","last_clean":"2023-10-16T14:11:53.994164+0000","last_became_active":"2023-10-16T14:11:52.984611+0000","last_became_peered":"2023-10-16T14:11:52.984611+0000","last_unstale":"2023-10-16T14:11:53.994164+0000","last_undegraded":"2023-10-16T14:11:53.994164+0000","last_fullsized":"2023-10-16T14:11:53.994164+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:47.800928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.967386+0000","last_change":"2023-10-16T14:11:38.796924+0000","last_active":"2023-10-16T14:11:52.967386+0000","last_peered":"2023-10-16T14:11:52.967386+0000","last_clean":"2023-10-16T14:11:52.967386+0000","last_became_active":"2023-10-16T14:11:38.796610+0000","last_became_peered":"2023-10-16T14:11:38.796610+0000","last_unstale":"2023-10-16T14:11:52.967386+0000","last_undegraded":"2023-10-16T14:11:52.967386+0000","last_fullsized":"2023-10-16T14:11:52.967386+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:29.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.983122+0000","last_change":"2023-10-16T14:11:37.780387+0000","last_active":"2023-10-16T14:11:52.983122+0000","last_peered":"2023-10-16T14:11:52.983122+0000","last_clean":"2023-10-16T14:11:52.983122+0000","last_became_active":"2023-10-16T14:11:37.779362+0000","last_became_peered":"2023-10-16T14:11:37.779362+0000","last_unstale":"2023-10-16T14:11:52.983122+0000","last_undegraded":"2023-10-16T14:11:52.983122+0000","last_fullsized":"2023-10-16T14:11:52.983122+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.719942+0000","last_change":"2023-10-16T14:11:53.000533+0000","last_active":"2023-10-16T14:11:54.719942+0000","last_peered":"2023-10-16T14:11:54.719942+0000","last_clean":"2023-10-16T14:11:54.719942+0000","last_became_active":"2023-10-16T14:11:53.000269+0000","last_became_peered":"2023-10-16T14:11:53.000269+0000","last_unstale":"2023-10-16T14:11:54.719942+0000","last_undegraded":"2023-10-16T14:11:54.719942+0000","last_fullsized":"2023-10-16T14:11:54.719942+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:13.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992076+0000","last_change":"2023-10-16T14:11:38.799901+0000","last_active":"2023-10-16T14:11:52.992076+0000","last_peered":"2023-10-16T14:11:52.992076+0000","last_clean":"2023-10-16T14:11:52.992076+0000","last_became_active":"2023-10-16T14:11:38.799145+0000","last_became_peered":"2023-10-16T14:11:38.799145+0000","last_unstale":"2023-10-16T14:11:52.992076+0000","last_undegraded":"2023-10-16T14:11:52.992076+0000","last_fullsized":"2023-10-16T14:11:52.992076+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.534242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973875+0000","last_change":"2023-10-16T14:11:37.779700+0000","last_active":"2023-10-16T14:11:52.973875+0000","last_peered":"2023-10-16T14:11:52.973875+0000","last_clean":"2023-10-16T14:11:52.973875+0000","last_became_active":"2023-10-16T14:11:37.779082+0000","last_became_peered":"2023-10-16T14:11:37.779082+0000","last_unstale":"2023-10-16T14:11:52.973875+0000","last_undegraded":"2023-10-16T14:11:52.973875+0000","last_fullsized":"2023-10-16T14:11:52.973875+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:12.639268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.067757+0000","last_change":"2023-10-16T14:11:52.998190+0000","last_active":"2023-10-16T14:11:55.067757+0000","last_peered":"2023-10-16T14:11:55.067757+0000","last_clean":"2023-10-16T14:11:55.067757+0000","last_became_active":"2023-10-16T14:11:52.995399+0000","last_became_peered":"2023-10-16T14:11:52.995399+0000","last_unstale":"2023-10-16T14:11:55.067757+0000","last_undegraded":"2023-10-16T14:11:55.067757+0000","last_fullsized":"2023-10-16T14:11:55.067757+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:16.114233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979252+0000","last_change":"2023-10-16T14:11:38.800281+0000","last_active":"2023-10-16T14:11:52.979252+0000","last_peered":"2023-10-16T14:11:52.979252+0000","last_clean":"2023-10-16T14:11:52.979252+0000","last_became_active":"2023-10-16T14:11:38.799813+0000","last_became_peered":"2023-10-16T14:11:38.799813+0000","last_unstale":"2023-10-16T14:11:52.979252+0000","last_undegraded":"2023-10-16T14:11:52.979252+0000","last_fullsized":"2023-10-16T14:11:52.979252+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:41.635286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"251'2","reported_seq":17,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.988830+0000","last_change":"2023-10-16T14:11:37.776293+0000","last_active":"2023-10-16T14:11:51.988830+0000","last_peered":"2023-10-16T14:11:51.988830+0000","last_clean":"2023-10-16T14:11:51.988830+0000","last_became_active":"2023-10-16T14:11:37.775902+0000","last_became_peered":"2023-10-16T14:11:37.775902+0000","last_unstale":"2023-10-16T14:11:51.988830+0000","last_undegraded":"2023-10-16T14:11:51.988830+0000","last_fullsized":"2023-10-16T14:11:51.988830+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:09.428253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993020+0000","last_change":"2023-10-16T14:11:52.990919+0000","last_active":"2023-10-16T14:11:53.993020+0000","last_peered":"2023-10-16T14:11:53.993020+0000","last_clean":"2023-10-16T14:11:53.993020+0000","last_became_active":"2023-10-16T14:11:52.990136+0000","last_became_peered":"2023-10-16T14:11:52.990136+0000","last_unstale":"2023-10-16T14:11:53.993020+0000","last_undegraded":"2023-10-16T14:11:53.993020+0000","last_fullsized":"2023-10-16T14:11:53.993020+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:46.114623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991821+0000","last_change":"2023-10-16T14:11:38.806901+0000","last_active":"2023-10-16T14:11:52.991821+0000","last_peered":"2023-10-16T14:11:52.991821+0000","last_clean":"2023-10-16T14:11:52.991821+0000","last_became_active":"2023-10-16T14:11:38.806614+0000","last_became_peered":"2023-10-16T14:11:38.806614+0000","last_unstale":"2023-10-16T14:11:52.991821+0000","last_undegraded":"2023-10-16T14:11:52.991821+0000","last_fullsized":"2023-10-16T14:11:52.991821+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:36.084105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"252'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.977834+0000","last_change":"2023-10-16T14:11:37.780349+0000","last_active":"2023-10-16T14:11:52.977834+0000","last_peered":"2023-10-16T14:11:52.977834+0000","last_clean":"2023-10-16T14:11:52.977834+0000","last_became_active":"2023-10-16T14:11:37.777254+0000","last_became_peered":"2023-10-16T14:11:37.777254+0000","last_unstale":"2023-10-16T14:11:52.977834+0000","last_undegraded":"2023-10-16T14:11:52.977834+0000","last_fullsized":"2023-10-16T14:11:52.977834+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:22.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.644999+0000","last_change":"2023-10-16T14:11:52.991009+0000","last_active":"2023-10-16T14:11:54.644999+0000","last_peered":"2023-10-16T14:11:54.644999+0000","last_clean":"2023-10-16T14:11:54.644999+0000","last_became_active":"2023-10-16T14:11:52.990420+0000","last_became_peered":"2023-10-16T14:11:52.990420+0000","last_unstale":"2023-10-16T14:11:54.644999+0000","last_undegraded":"2023-10-16T14:11:54.644999+0000","last_fullsized":"2023-10-16T14:11:54.644999+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:12.562346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973061+0000","last_change":"2023-10-16T14:11:38.805448+0000","last_active":"2023-10-16T14:11:52.973061+0000","last_peered":"2023-10-16T14:11:52.973061+0000","last_clean":"2023-10-16T14:11:52.973061+0000","last_became_active":"2023-10-16T14:11:38.805128+0000","last_became_peered":"2023-10-16T14:11:38.805128+0000","last_unstale":"2023-10-16T14:11:52.973061+0000","last_undegraded":"2023-10-16T14:11:52.973061+0000","last_fullsized":"2023-10-16T14:11:52.973061+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:10.898166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976766+0000","last_change":"2023-10-16T14:11:37.782304+0000","last_active":"2023-10-16T14:11:52.976766+0000","last_peered":"2023-10-16T14:11:52.976766+0000","last_clean":"2023-10-16T14:11:52.976766+0000","last_became_active":"2023-10-16T14:11:37.781840+0000","last_became_peered":"2023-10-16T14:11:37.781840+0000","last_unstale":"2023-10-16T14:11:52.976766+0000","last_undegraded":"2023-10-16T14:11:52.976766+0000","last_fullsized":"2023-10-16T14:11:52.976766+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858200+0000","last_change":"2023-10-16T14:11:52.981362+0000","last_active":"2023-10-16T14:11:54.858200+0000","last_peered":"2023-10-16T14:11:54.858200+0000","last_clean":"2023-10-16T14:11:54.858200+0000","last_became_active":"2023-10-16T14:11:52.981029+0000","last_became_peered":"2023-10-16T14:11:52.981029+0000","last_unstale":"2023-10-16T14:11:54.858200+0000","last_undegraded":"2023-10-16T14:11:54.858200+0000","last_fullsized":"2023-10-16T14:11:54.858200+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:01.090353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975057+0000","last_change":"2023-10-16T14:11:38.801623+0000","last_active":"2023-10-16T14:11:52.975057+0000","last_peered":"2023-10-16T14:11:52.975057+0000","last_clean":"2023-10-16T14:11:52.975057+0000","last_became_active":"2023-10-16T14:11:38.801313+0000","last_became_peered":"2023-10-16T14:11:38.801313+0000","last_unstale":"2023-10-16T14:11:52.975057+0000","last_undegraded":"2023-10-16T14:11:52.975057+0000","last_fullsized":"2023-10-16T14:11:52.975057+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:56.175171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973534+0000","last_change":"2023-10-16T14:11:37.775930+0000","last_active":"2023-10-16T14:11:52.973534+0000","last_peered":"2023-10-16T14:11:52.973534+0000","last_clean":"2023-10-16T14:11:52.973534+0000","last_became_active":"2023-10-16T14:11:37.775617+0000","last_became_peered":"2023-10-16T14:11:37.775617+0000","last_unstale":"2023-10-16T14:11:52.973534+0000","last_undegraded":"2023-10-16T14:11:52.973534+0000","last_fullsized":"2023-10-16T14:11:52.973534+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:19.733822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858483+0000","last_change":"2023-10-16T14:11:52.982720+0000","last_active":"2023-10-16T14:11:54.858483+0000","last_peered":"2023-10-16T14:11:54.858483+0000","last_clean":"2023-10-16T14:11:54.858483+0000","last_became_active":"2023-10-16T14:11:52.981925+0000","last_became_peered":"2023-10-16T14:11:52.981925+0000","last_unstale":"2023-10-16T14:11:54.858483+0000","last_undegraded":"2023-10-16T14:11:54.858483+0000","last_fullsized":"2023-10-16T14:11:54.858483+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:18.153466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974888+0000","last_change":"2023-10-16T14:11:38.796749+0000","last_active":"2023-10-16T14:11:52.974888+0000","last_peered":"2023-10-16T14:11:52.974888+0000","last_clean":"2023-10-16T14:11:52.974888+0000","last_became_active":"2023-10-16T14:11:38.796468+0000","last_became_peered":"2023-10-16T14:11:38.796468+0000","last_unstale":"2023-10-16T14:11:52.974888+0000","last_undegraded":"2023-10-16T14:11:52.974888+0000","last_fullsized":"2023-10-16T14:11:52.974888+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:17.187404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976583+0000","last_change":"2023-10-16T14:11:37.785747+0000","last_active":"2023-10-16T14:11:52.976583+0000","last_peered":"2023-10-16T14:11:52.976583+0000","last_clean":"2023-10-16T14:11:52.976583+0000","last_became_active":"2023-10-16T14:11:37.785464+0000","last_became_peered":"2023-10-16T14:11:37.785464+0000","last_unstale":"2023-10-16T14:11:52.976583+0000","last_undegraded":"2023-10-16T14:11:52.976583+0000","last_fullsized":"2023-10-16T14:11:52.976583+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:32.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858945+0000","last_change":"2023-10-16T14:11:52.999461+0000","last_active":"2023-10-16T14:11:54.858945+0000","last_peered":"2023-10-16T14:11:54.858945+0000","last_clean":"2023-10-16T14:11:54.858945+0000","last_became_active":"2023-10-16T14:11:52.999146+0000","last_became_peered":"2023-10-16T14:11:52.999146+0000","last_unstale":"2023-10-16T14:11:54.858945+0000","last_undegraded":"2023-10-16T14:11:54.858945+0000","last_fullsized":"2023-10-16T14:11:54.858945+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:47.618540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.989135+0000","last_change":"2023-10-16T14:11:38.800958+0000","last_active":"2023-10-16T14:11:52.989135+0000","last_peered":"2023-10-16T14:11:52.989135+0000","last_clean":"2023-10-16T14:11:52.989135+0000","last_became_active":"2023-10-16T14:11:38.800677+0000","last_became_peered":"2023-10-16T14:11:38.800677+0000","last_unstale":"2023-10-16T14:11:52.989135+0000","last_undegraded":"2023-10-16T14:11:52.989135+0000","last_fullsized":"2023-10-16T14:11:52.989135+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:18.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.983782+0000","last_change":"2023-10-16T14:11:37.785497+0000","last_active":"2023-10-16T14:11:51.983782+0000","last_peered":"2023-10-16T14:11:51.983782+0000","last_clean":"2023-10-16T14:11:51.983782+0000","last_became_active":"2023-10-16T14:11:37.785162+0000","last_became_peered":"2023-10-16T14:11:37.785162+0000","last_unstale":"2023-10-16T14:11:51.983782+0000","last_undegraded":"2023-10-16T14:11:51.983782+0000","last_fullsized":"2023-10-16T14:11:51.983782+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:28.524950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995213+0000","last_change":"2023-10-16T14:11:52.997961+0000","last_active":"2023-10-16T14:11:53.995213+0000","last_peered":"2023-10-16T14:11:53.995213+0000","last_clean":"2023-10-16T14:11:53.995213+0000","last_became_active":"2023-10-16T14:11:52.997683+0000","last_became_peered":"2023-10-16T14:11:52.997683+0000","last_unstale":"2023-10-16T14:11:53.995213+0000","last_undegraded":"2023-10-16T14:11:53.995213+0000","last_fullsized":"2023-10-16T14:11:53.995213+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:13.592656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.995178+0000","last_change":"2023-10-16T14:11:38.789466+0000","last_active":"2023-10-16T14:11:52.995178+0000","last_peered":"2023-10-16T14:11:52.995178+0000","last_clean":"2023-10-16T14:11:52.995178+0000","last_became_active":"2023-10-16T14:11:38.788914+0000","last_became_peered":"2023-10-16T14:11:38.788914+0000","last_unstale":"2023-10-16T14:11:52.995178+0000","last_undegraded":"2023-10-16T14:11:52.995178+0000","last_fullsized":"2023-10-16T14:11:52.995178+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:36.423758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959110+0000","last_change":"2023-10-16T14:11:37.784108+0000","last_active":"2023-10-16T14:11:52.959110+0000","last_peered":"2023-10-16T14:11:52.959110+0000","last_clean":"2023-10-16T14:11:52.959110+0000","last_became_active":"2023-10-16T14:11:37.783469+0000","last_became_peered":"2023-10-16T14:11:37.783469+0000","last_unstale":"2023-10-16T14:11:52.959110+0000","last_undegraded":"2023-10-16T14:11:52.959110+0000","last_fullsized":"2023-10-16T14:11:52.959110+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069566+0000","last_change":"2023-10-16T14:11:53.000623+0000","last_active":"2023-10-16T14:11:55.069566+0000","last_peered":"2023-10-16T14:11:55.069566+0000","last_clean":"2023-10-16T14:11:55.069566+0000","last_became_active":"2023-10-16T14:11:53.000238+0000","last_became_peered":"2023-10-16T14:11:53.000238+0000","last_unstale":"2023-10-16T14:11:55.069566+0000","last_undegraded":"2023-10-16T14:11:55.069566+0000","last_fullsized":"2023-10-16T14:11:55.069566+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:40.320922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982822+0000","last_change":"2023-10-16T14:11:38.799118+0000","last_active":"2023-10-16T14:11:52.982822+0000","last_peered":"2023-10-16T14:11:52.982822+0000","last_clean":"2023-10-16T14:11:52.982822+0000","last_became_active":"2023-10-16T14:11:38.798708+0000","last_became_peered":"2023-10-16T14:11:38.798708+0000","last_unstale":"2023-10-16T14:11:52.982822+0000","last_undegraded":"2023-10-16T14:11:52.982822+0000","last_fullsized":"2023-10-16T14:11:52.982822+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:36.917040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960867+0000","last_change":"2023-10-16T14:11:37.777958+0000","last_active":"2023-10-16T14:11:52.960867+0000","last_peered":"2023-10-16T14:11:52.960867+0000","last_clean":"2023-10-16T14:11:52.960867+0000","last_became_active":"2023-10-16T14:11:37.777687+0000","last_became_peered":"2023-10-16T14:11:37.777687+0000","last_unstale":"2023-10-16T14:11:52.960867+0000","last_undegraded":"2023-10-16T14:11:52.960867+0000","last_fullsized":"2023-10-16T14:11:52.960867+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:13.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993961+0000","last_change":"2023-10-16T14:11:52.993995+0000","last_active":"2023-10-16T14:11:53.993961+0000","last_peered":"2023-10-16T14:11:53.993961+0000","last_clean":"2023-10-16T14:11:53.993961+0000","last_became_active":"2023-10-16T14:11:52.993712+0000","last_became_peered":"2023-10-16T14:11:52.993712+0000","last_unstale":"2023-10-16T14:11:53.993961+0000","last_undegraded":"2023-10-16T14:11:53.993961+0000","last_fullsized":"2023-10-16T14:11:53.993961+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:51:12.663965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.977663+0000","last_change":"2023-10-16T14:11:38.785931+0000","last_active":"2023-10-16T14:11:52.977663+0000","last_peered":"2023-10-16T14:11:52.977663+0000","last_clean":"2023-10-16T14:11:52.977663+0000","last_became_active":"2023-10-16T14:11:38.785345+0000","last_became_peered":"2023-10-16T14:11:38.785345+0000","last_unstale":"2023-10-16T14:11:52.977663+0000","last_undegraded":"2023-10-16T14:11:52.977663+0000","last_fullsized":"2023-10-16T14:11:52.977663+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:16.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"251'4","reported_seq":20,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.968726+0000","last_change":"2023-10-16T14:11:37.781168+0000","last_active":"2023-10-16T14:11:52.968726+0000","last_peered":"2023-10-16T14:11:52.968726+0000","last_clean":"2023-10-16T14:11:52.968726+0000","last_became_active":"2023-10-16T14:11:37.780834+0000","last_became_peered":"2023-10-16T14:11:37.780834+0000","last_unstale":"2023-10-16T14:11:52.968726+0000","last_undegraded":"2023-10-16T14:11:52.968726+0000","last_fullsized":"2023-10-16T14:11:52.968726+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:21.622808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"237'373","reported_seq":1323,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.977777+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:11:52.977777+0000","last_peered":"2023-10-16T14:11:52.977777+0000","last_clean":"2023-10-16T14:11:52.977777+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:11:52.977777+0000","last_undegraded":"2023-10-16T14:11:52.977777+0000","last_fullsized":"2023-10-16T14:11:52.977777+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543314,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52960,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318880,"statfs":{"total":96636764160,"available":96582533120,"internally_reserved":0,"allocated":3817472,"data_stored":2170995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3460000000000001,"15min":1.3660000000000001},"min":{"1min":0.77900000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.885},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.3480000000000001,"15min":1.3979999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7669999999999999,"5min":1.827,"15min":1.873},"last":6.391}]},{"osd":1,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.2190000000000001,"15min":1.244},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.228,"15min":1.2490000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.72,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":3.4769999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.228,"15min":1.258},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.482,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":4.3250000000000002},{"interface":"front","average":{"1min":1.224,"5min":1.347,"15min":1.3560000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.6930000000000001,"5min":2.028,"15min":2.028},"last":3.4289999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.363,"15min":1.403},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8320000000000001,"5min":1.921,"15min":2.097},"last":2.0790000000000002},{"interface":"front","average":{"1min":1.351,"5min":1.3999999999999999,"15min":1.4259999999999999},"min":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7969999999999999,"5min":1.946,"15min":2.0169999999999999},"last":3.3719999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.23,"15min":1.2390000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.895},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4390000000000001,"15min":1.4119999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.786,"5min":1.982,"15min":2.0009999999999999},"last":2.0230000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.391,"15min":1.3799999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.0590000000000002},"last":4.0860000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.536},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.9550000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":2.5110000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.6259999999999999,"15min":1.633},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.0009999999999999,"5min":2.052,"15min":2.3250000000000002},"last":3.29},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.5249999999999999,"15min":1.48},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.8939999999999999,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":3.524}]}]},{"osd":6,"up_from":14,"seq":60129543315,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64868,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306972,"statfs":{"total":96636764160,"available":96570339328,"internally_reserved":0,"allocated":3231744,"data_stored":1581203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2689999999999999,"15min":1.329},"min":{"1min":0.70299999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.788,"5min":1.887,"15min":1.887},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.3140000000000001,"15min":1.353},"min":{"1min":0.747,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8460000000000001,"5min":1.847,"15min":1.9390000000000001},"last":1.5509999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.206,"15min":1.2250000000000001},"min":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6279999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.21,"15min":1.2270000000000001},"min":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6799999999999999,"5min":1.748,"15min":1.77},"last":0.874}]},{"osd":2,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.24,"15min":1.2549999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.78,"5min":1.823,"15min":1.823},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.282,"15min":1.3169999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.8759999999999999,"5min":1.962,"15min":1.962},"last":1.379}]},{"osd":3,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.327,"15min":1.345},"min":{"1min":0.93400000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":1.075,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.776,"5min":1.911,"15min":1.911},"last":0.93500000000000005}]},{"osd":4,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.393,"15min":1.409},"min":{"1min":1.1259999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5609999999999999,"15min":1.5509999999999999},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.427}]},{"osd":5,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.79800000000000004,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.206},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4990000000000001,"15min":1.5},"min":{"1min":1.2170000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.0990000000000002,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.004}]},{"osd":7,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.512,"15min":1.5049999999999999},"min":{"1min":0.999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0150000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.546,"15min":1.542},"min":{"1min":1.0269999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.157,"5min":2.157,"15min":2.6120000000000001},"last":1.4970000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543316,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30148,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341692,"statfs":{"total":96636764160,"available":96605892608,"internally_reserved":0,"allocated":3203072,"data_stored":1573320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1299999999999999,"15min":1.202},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.4750000000000001,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2150000000000001,"15min":1.2310000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.7110000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.282,"15min":1.298},"min":{"1min":0.77500000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.774,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.395},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1830000000000001,"15min":1.254},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.698,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.97999999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.304,"15min":1.335},"min":{"1min":0.90300000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.587,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.266,"15min":1.3120000000000001},"min":{"1min":0.80200000000000005,"5min":0.746,"15min":0.746},"max":{"1min":1.806,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3859999999999999,"15min":1.407},"min":{"1min":0.89300000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9019999999999999,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.409,"15min":1.431},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8759999999999999,"5min":2.75,"15min":2.75},"last":1.3660000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3160000000000001,"15min":1.2989999999999999},"min":{"1min":0.77300000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.9299999999999999,"5min":2.774,"15min":2.774},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.454,"15min":1.4319999999999999},"min":{"1min":0.93700000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.4930000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4390000000000001,"15min":1.4319999999999999},"min":{"1min":1.1699999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9390000000000001,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.46},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.508,"15min":1.4910000000000001},"min":{"1min":1.048,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9590000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.236}]},{"osd":7,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.593,"15min":1.554},"min":{"1min":1.2170000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1680000000000001,"5min":2.177,"15min":2.4609999999999999},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6200000000000001,"15min":1.603},"min":{"1min":1.113,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1000000000000001,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.5549999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543316,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66228,"kb_used_data":3168,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305612,"statfs":{"total":96636764160,"available":96568946688,"internally_reserved":0,"allocated":3244032,"data_stored":1577937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.145,"15min":1.2050000000000001},"min":{"1min":0.61799999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1799999999999999,"15min":1.1990000000000001},"min":{"1min":0.748,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":0.77500000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1990000000000001,"15min":1.2490000000000001},"min":{"1min":0.80700000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6899999999999999,"5min":1.71,"15min":1.78},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.244,"5min":1.2450000000000001,"15min":1.2869999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7250000000000001,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.2150000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2829999999999999,"15min":1.3160000000000001},"min":{"1min":0.73899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.591,"5min":1.714,"15min":1.865},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3500000000000001,"15min":1.3799999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":1.905,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.0329999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2889999999999999,"15min":1.302},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.45,"15min":1.4279999999999999},"min":{"1min":1.05,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.6439999999999999,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.4199999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.3520000000000001,"15min":1.339},"min":{"1min":1.0980000000000001,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":1.994,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.387,"15min":1.3420000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.258}]},{"osd":6,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.409,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.90400000000000003,"15min":0.872},"max":{"1min":1.931,"5min":1.931,"15min":1.956},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.516,"15min":1.518},"min":{"1min":1.0840000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.0630000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.681}]},{"osd":7,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.5660000000000001,"15min":1.5089999999999999},"min":{"1min":1.0149999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1949999999999998},"last":1.7290000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6479999999999999,"15min":1.6339999999999999},"min":{"1min":1.3620000000000001,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":2.085,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.992}]}]},{"osd":3,"up_from":14,"seq":60129543316,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56556,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315284,"statfs":{"total":96636764160,"available":96578850816,"internally_reserved":0,"allocated":3235840,"data_stored":1583263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.90900000000000003,"15min":0.94799999999999995},"min":{"1min":0.48599999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.879},{"interface":"front","average":{"1min":1.107,"5min":1.1479999999999999,"15min":1.2090000000000001},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.03,"15min":1.075},"min":{"1min":0.55900000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.855},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.133,"5min":1.0980000000000001,"15min":1.123},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.0760000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0960000000000001,"15min":1.1890000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.823,"5min":1.8700000000000001,"15min":2.0539999999999998},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3400000000000001,"15min":1.431},"min":{"1min":0.622,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.909,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.91900000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.482,"15min":1.51},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2160000000000002,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.379,"5min":1.4330000000000001,"15min":1.456},"min":{"1min":0.84099999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.823,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.3460000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.427,"15min":1.4850000000000001},"min":{"1min":0.89900000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8380000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.52,"15min":1.5529999999999999},"min":{"1min":0.871,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9910000000000001,"5min":2.0339999999999998,"15min":2.153},"last":0.95299999999999996}]},{"osd":6,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.5,"15min":1.522},"min":{"1min":0.72899999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.335,"5min":2.335,"15min":2.4329999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4750000000000001,"15min":1.496},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.194,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.6220000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5760000000000001,"15min":1.599},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1859999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.698,"5min":1.6599999999999999,"15min":1.677},"min":{"1min":0.98199999999999998,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.5,"5min":2.569,"15min":2.6070000000000002},"last":1.2350000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543314,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56528,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315312,"statfs":{"total":96636764160,"available":96578879488,"internally_reserved":0,"allocated":3207168,"data_stored":1573410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.93100000000000005,"15min":0.95299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.5029999999999999,"5min":1.597,"15min":2.0089999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.137},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.431,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.105}]},{"osd":1,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.131,"15min":1.171},"min":{"1min":0.56399999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.72,"5min":1.847,"15min":1.847},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.03,"5min":1.2010000000000001,"15min":1.242},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5860000000000001,"5min":1.768,"15min":1.9530000000000001},"last":1.4650000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.107,"15min":1.115},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.038},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.3009999999999999,"15min":1.3700000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.0619999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.67700000000000005}]},{"osd":4,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4710000000000001,"15min":1.5},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.88},{"interface":"front","average":{"1min":1.349,"5min":1.464,"15min":1.4870000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.3140000000000001},"last":1.8320000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.522,"15min":1.5529999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.867,"5min":2.1819999999999999,"15min":2.488},"last":1.7490000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4550000000000001,"15min":1.4710000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.0059999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.3460000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.524,"15min":1.52},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.4079999999999999},"last":1.423},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.5640000000000001,"15min":1.617},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.5569999999999999},"last":1.375}]},{"osd":7,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6319999999999999,"15min":1.655},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.7890000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.619,"15min":1.641},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.2599999999999998,"5min":2.2799999999999998,"15min":2.3319999999999999},"last":1.5149999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543314,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52404,"kb_used_data":3176,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319436,"statfs":{"total":96636764160,"available":96583102464,"internally_reserved":0,"allocated":3252224,"data_stored":1584244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.93000000000000005,"15min":0.93999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.448,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.109,"15min":1.133},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.454}]},{"osd":2,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0129999999999999,"15min":1.0389999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.204,"15min":1.2549999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.98199999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.4369999999999998},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.2789999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.544,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.571}]},{"osd":4,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.258,"15min":1.339},"min":{"1min":0.65700000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.534,"5min":1.776,"15min":2.1259999999999999},"last":1.125},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.488,"15min":1.5309999999999999},"min":{"1min":0.79100000000000004,"5min":0.748,"15min":0.748},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.401}]},{"osd":5,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.446,"15min":1.488},"min":{"1min":0.72899999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.0819999999999999,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.655},{"interface":"front","average":{"1min":1.419,"5min":1.385,"15min":1.4790000000000001},"min":{"1min":0.746,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.371}]},{"osd":6,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5800000000000001,"15min":1.633},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.7200000000000002},"last":1.714},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5409999999999999,"15min":1.603},"min":{"1min":0.85099999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0979999999999999,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.758}]},{"osd":7,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.639,"15min":1.696},"min":{"1min":0.95299999999999996,"5min":0.628,"15min":0.628},"max":{"1min":2.1280000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.8680000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.52,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0630000000000002,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.8109999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543315,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48780,"kb_used_data":3712,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323060,"statfs":{"total":96636764160,"available":96586813440,"internally_reserved":0,"allocated":3801088,"data_stored":2163722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.94299999999999995,"15min":0.94399999999999995},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4299999999999999,"5min":1.704,"15min":1.891},"last":1.3440000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.054,"15min":1.145},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3009999999999999,"5min":1.659,"15min":1.9299999999999999},"last":1.5069999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1299999999999999,"15min":1.169},"min":{"1min":0.73299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.171,"15min":1.2210000000000001},"min":{"1min":0.68799999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.52,"5min":1.893,"15min":1.893},"last":1.0149999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":1.147},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.623,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.29,"15min":1.3660000000000001},"min":{"1min":0.63200000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.681,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.55}]},{"osd":4,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.302,"15min":1.3640000000000001},"min":{"1min":0.84999999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.625,"5min":1.819,"15min":2.0070000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4970000000000001,"15min":1.5189999999999999},"min":{"1min":0.92400000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.919,"5min":2.601,"15min":2.601},"last":1.9330000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4910000000000001,"15min":1.504},"min":{"1min":1.129,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.089,"5min":2.2250000000000001,"15min":2.3279999999999998},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4259999999999999,"15min":1.4790000000000001},"min":{"1min":0.88700000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.401}]},{"osd":6,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.577,"15min":1.613},"min":{"1min":1.1890000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0019999999999998,"5min":2.669,"15min":2.669},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.552,"15min":1.5920000000000001},"min":{"1min":0.94999999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.327,"5min":2.355,"15min":2.3769999999999998},"last":1.637}]},{"osd":7,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.6399999999999999,"15min":1.7010000000000001},"min":{"1min":0.88700000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2170000000000001,"5min":2.484,"15min":2.5270000000000001},"last":1.9910000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.591,"15min":1.6359999999999999},"min":{"1min":1.056,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.9490000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.786}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:11:57.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:11:57.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.789+0000 7f67b26df640 1 Processor -- start 2023-10-16T14:11:57.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.790+0000 7f67b26df640 1 -- start start 2023-10-16T14:11:57.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.791+0000 7f67b26df640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f67ac15cd70 0x7f67ac0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.791+0000 7f67b26df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.791+0000 7f67b26df640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac15d150 con 0x7f67ac0d0be0 2023-10-16T14:11:57.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.791+0000 7f67b26df640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac0d1180 con 0x7f67ac15cd70 2023-10-16T14:11:57.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.791+0000 7f67b26df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac0d1460 con 0x7f67ac0b8480 2023-10-16T14:11:57.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b16dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f67ac15cd70 0x7f67ac0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b1ede640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f67ac0d0be0 0x7f67ac0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49036/0 (socket says 172.21.15.60:49036) 2023-10-16T14:11:57.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b16dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f67ac15cd70 0x7f67ac0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36448/0 (socket says 172.21.15.60:36448) 2023-10-16T14:11:57.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b1ede640 1 -- 172.21.15.60:0/3927407360 learned_addr learned my addr 172.21.15.60:0/3927407360 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:57.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b0edc640 1 --2- 172.21.15.60:0/3927407360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b16dd640 1 -- 172.21.15.60:0/3927407360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0b8480 msgr2=0x7f67ac0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:57.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b16dd640 1 --2- 172.21.15.60:0/3927407360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b16dd640 1 -- 172.21.15.60:0/3927407360 >> v1:172.21.15.60:6789/0 conn(0x7f67ac0d0be0 legacy=0x7f67ac0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67b16dd640 1 -- 172.21.15.60:0/3927407360 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0d1740 con 0x7f67ac15cd70 2023-10-16T14:11:57.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.792+0000 7f67a3fff640 1 -- 172.21.15.60:0/3927407360 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471808775 0 0) 0x7f67ac15d150 con 0x7f67ac0d0be0 2023-10-16T14:11:57.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.794+0000 7f67b16dd640 1 --2- 172.21.15.60:0/3927407360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac0d2220 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f67a80097a0 tx=0x7f67a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b416290f656d92e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:57.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.794+0000 7f67a3fff640 1 -- 172.21.15.60:0/3927407360 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8031830 con 0x7f67ac15cd70 2023-10-16T14:11:57.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.794+0000 7f67a3fff640 1 -- 172.21.15.60:0/3927407360 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8031990 con 0x7f67ac15cd70 2023-10-16T14:11:57.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.794+0000 7f67a3fff640 1 -- 172.21.15.60:0/3927407360 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8031d20 con 0x7f67ac15cd70 2023-10-16T14:11:57.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 -- 172.21.15.60:0/3927407360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 msgr2=0x7f67ac0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 --2- 172.21.15.60:0/3927407360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac0d2220 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f67a80097a0 tx=0x7f67a802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 -- 172.21.15.60:0/3927407360 shutdown_connections 2023-10-16T14:11:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 --2- 172.21.15.60:0/3927407360 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 --2- 172.21.15.60:0/3927407360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac0d2220 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 -- 172.21.15.60:0/3927407360 >> 172.21.15.60:0/3927407360 conn(0x7f67ac0cf570 msgr2=0x7f67ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:57.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 -- 172.21.15.60:0/3927407360 shutdown_connections 2023-10-16T14:11:57.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.795+0000 7f67b26df640 1 -- 172.21.15.60:0/3927407360 wait complete. 2023-10-16T14:11:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.798+0000 7f67b26df640 1 Processor -- start 2023-10-16T14:11:57.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.798+0000 7f67b26df640 1 -- start start 2023-10-16T14:11:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.799+0000 7f67b26df640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67ac0b8480 0x7f67ac155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.799+0000 7f67b26df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0d0be0 0x7f67ac0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.799+0000 7f67b26df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.799+0000 7f67b26df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac1f3110 con 0x7f67ac15cd70 2023-10-16T14:11:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.799+0000 7f67b26df640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac155a80 con 0x7f67ac0b8480 2023-10-16T14:11:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.799+0000 7f67b26df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac1f2e30 con 0x7f67ac0d0be0 2023-10-16T14:11:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac15a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36454/0 (socket says 172.21.15.60:36454) 2023-10-16T14:11:57.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 -- 172.21.15.60:0/1190263645 learned_addr learned my addr 172.21.15.60:0/1190263645 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b0edc640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0d0be0 0x7f67ac0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b16dd640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67ac0b8480 0x7f67ac155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 -- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0d0be0 msgr2=0x7f67ac0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0d0be0 0x7f67ac0cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 -- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67ac0b8480 msgr2=0x7f67ac155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67ac0b8480 0x7f67ac155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.800+0000 7f67b1ede640 1 -- 172.21.15.60:0/1190263645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a8009400 con 0x7f67ac15cd70 2023-10-16T14:11:57.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.801+0000 7f67b1ede640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac15a5b0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c780 tx=0x7f67a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.801+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a400ce30 con 0x7f67ac15cd70 2023-10-16T14:11:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.801+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a400a640 con 0x7f67ac15cd70 2023-10-16T14:11:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.801+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a400a9f0 con 0x7f67ac15cd70 2023-10-16T14:11:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.801+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac0cbbb0 con 0x7f67ac15cd70 2023-10-16T14:11:57.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.801+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac0cc170 con 0x7f67ac15cd70 2023-10-16T14:11:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.802+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac0cc390 con 0x7f67ac15cd70 2023-10-16T14:11:57.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.803+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f67a400ab50 con 0x7f67ac15cd70 2023-10-16T14:11:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.805+0000 7f67a1ffb640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f679006b0b0 0x7f679006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.805+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f67a40892a0 con 0x7f67ac15cd70 2023-10-16T14:11:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.805+0000 7f67b16dd640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f679006b0b0 0x7f679006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:57.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.806+0000 7f67b16dd640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f679006b0b0 0x7f679006d570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f67a8036000 tx=0x7f67a803c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:57.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.808+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67a4053bf0 con 0x7f67ac15cd70 2023-10-16T14:11:57.856 INFO:tasks.ceph.mds.d.smithi144.stderr:2023-10-16T14:11:57.853+0000 7f80fcd30640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:11:58.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:57.999+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 --> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67ac0d1d00 con 0x7f679006b0b0 2023-10-16T14:11:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.039+0000 7f67a1ffb640 1 -- 172.21.15.60:0/1190263645 <== mgr.9169 v2:172.21.15.144:6832/4052124783 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526045 (secure 0 0 0) 0x7f67a8045410 con 0x7f679006b0b0 2023-10-16T14:11:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:11:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f679006b0b0 msgr2=0x7f679006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f679006b0b0 0x7f679006d570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f67a8036000 tx=0x7f67a803c040 comp rx=0 tx=0).stop 2023-10-16T14:11:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 msgr2=0x7f67ac15a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac15a5b0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c780 tx=0x7f67a400cc40 comp rx=0 tx=0).stop 2023-10-16T14:11:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 shutdown_connections 2023-10-16T14:11:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f679006b0b0 0x7f679006d570 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f67ac0d0be0 0x7f67ac0cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ac15cd70 0x7f67ac15a5b0 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 --2- 172.21.15.60:0/1190263645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67ac0b8480 0x7f67ac155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 >> 172.21.15.60:0/1190263645 conn(0x7f67ac0cf570 msgr2=0x7f67ac15fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.050+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 shutdown_connections 2023-10-16T14:11:58.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.051+0000 7f67b26df640 1 -- 172.21.15.60:0/1190263645 wait complete. 2023-10-16T14:11:58.081 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2023-10-16T14:11:57.067680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":714,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":292,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428472,"kb_used_data":26360,"kb_used_omap":12,"kb_used_meta":401971,"kb_avail":754546248,"statfs":{"total":773094113280,"available":772655357952,"internally_reserved":0,"allocated":26992640,"data_stored":13808094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411618872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002531"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.857803+0000","last_change":"2023-10-16T14:11:52.991222+0000","last_active":"2023-10-16T14:11:54.857803+0000","last_peered":"2023-10-16T14:11:54.857803+0000","last_clean":"2023-10-16T14:11:54.857803+0000","last_became_active":"2023-10-16T14:11:52.990947+0000","last_became_peered":"2023-10-16T14:11:52.990947+0000","last_unstale":"2023-10-16T14:11:54.857803+0000","last_undegraded":"2023-10-16T14:11:54.857803+0000","last_fullsized":"2023-10-16T14:11:54.857803+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:24.644523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986943+0000","last_change":"2023-10-16T14:11:51.988133+0000","last_active":"2023-10-16T14:11:53.986943+0000","last_peered":"2023-10-16T14:11:53.986943+0000","last_clean":"2023-10-16T14:11:53.986943+0000","last_became_active":"2023-10-16T14:11:51.987807+0000","last_became_peered":"2023-10-16T14:11:51.987807+0000","last_unstale":"2023-10-16T14:11:53.986943+0000","last_undegraded":"2023-10-16T14:11:53.986943+0000","last_fullsized":"2023-10-16T14:11:53.986943+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:05.737538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991549+0000","last_change":"2023-10-16T14:11:38.800766+0000","last_active":"2023-10-16T14:11:52.991549+0000","last_peered":"2023-10-16T14:11:52.991549+0000","last_clean":"2023-10-16T14:11:52.991549+0000","last_became_active":"2023-10-16T14:11:38.799147+0000","last_became_peered":"2023-10-16T14:11:38.799147+0000","last_unstale":"2023-10-16T14:11:52.991549+0000","last_undegraded":"2023-10-16T14:11:52.991549+0000","last_fullsized":"2023-10-16T14:11:52.991549+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:09.869393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.008887+0000","last_change":"2023-10-16T14:11:53.005992+0000","last_active":"2023-10-16T14:11:53.008887+0000","last_peered":"2023-10-16T14:11:53.008887+0000","last_clean":"2023-10-16T14:11:53.008887+0000","last_became_active":"2023-10-16T14:11:53.005721+0000","last_became_peered":"2023-10-16T14:11:53.005721+0000","last_unstale":"2023-10-16T14:11:53.008887+0000","last_undegraded":"2023-10-16T14:11:53.008887+0000","last_fullsized":"2023-10-16T14:11:53.008887+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:18.227695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.984497+0000","last_change":"2023-10-16T14:11:51.990577+0000","last_active":"2023-10-16T14:11:52.984497+0000","last_peered":"2023-10-16T14:11:52.984497+0000","last_clean":"2023-10-16T14:11:52.984497+0000","last_became_active":"2023-10-16T14:11:51.990290+0000","last_became_peered":"2023-10-16T14:11:51.990290+0000","last_unstale":"2023-10-16T14:11:52.984497+0000","last_undegraded":"2023-10-16T14:11:52.984497+0000","last_fullsized":"2023-10-16T14:11:52.984497+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:50.097531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993685+0000","last_change":"2023-10-16T14:11:38.811708+0000","last_active":"2023-10-16T14:11:52.993685+0000","last_peered":"2023-10-16T14:11:52.993685+0000","last_clean":"2023-10-16T14:11:52.993685+0000","last_became_active":"2023-10-16T14:11:38.808060+0000","last_became_peered":"2023-10-16T14:11:38.808060+0000","last_unstale":"2023-10-16T14:11:52.993685+0000","last_undegraded":"2023-10-16T14:11:52.993685+0000","last_fullsized":"2023-10-16T14:11:52.993685+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:53.712931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.008555+0000","last_change":"2023-10-16T14:11:53.004533+0000","last_active":"2023-10-16T14:11:53.008555+0000","last_peered":"2023-10-16T14:11:53.008555+0000","last_clean":"2023-10-16T14:11:53.008555+0000","last_became_active":"2023-10-16T14:11:53.004202+0000","last_became_peered":"2023-10-16T14:11:53.004202+0000","last_unstale":"2023-10-16T14:11:53.008555+0000","last_undegraded":"2023-10-16T14:11:53.008555+0000","last_fullsized":"2023-10-16T14:11:53.008555+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:41.964559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.717966+0000","last_change":"2023-10-16T14:11:51.983126+0000","last_active":"2023-10-16T14:11:54.717966+0000","last_peered":"2023-10-16T14:11:54.717966+0000","last_clean":"2023-10-16T14:11:54.717966+0000","last_became_active":"2023-10-16T14:11:51.982846+0000","last_became_peered":"2023-10-16T14:11:51.982846+0000","last_unstale":"2023-10-16T14:11:54.717966+0000","last_undegraded":"2023-10-16T14:11:54.717966+0000","last_fullsized":"2023-10-16T14:11:54.717966+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959501+0000","last_change":"2023-10-16T14:11:38.799851+0000","last_active":"2023-10-16T14:11:52.959501+0000","last_peered":"2023-10-16T14:11:52.959501+0000","last_clean":"2023-10-16T14:11:52.959501+0000","last_became_active":"2023-10-16T14:11:38.799520+0000","last_became_peered":"2023-10-16T14:11:38.799520+0000","last_unstale":"2023-10-16T14:11:52.959501+0000","last_undegraded":"2023-10-16T14:11:52.959501+0000","last_fullsized":"2023-10-16T14:11:52.959501+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:59.480982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993667+0000","last_change":"2023-10-16T14:11:52.992005+0000","last_active":"2023-10-16T14:11:52.993667+0000","last_peered":"2023-10-16T14:11:52.993667+0000","last_clean":"2023-10-16T14:11:52.993667+0000","last_became_active":"2023-10-16T14:11:52.991726+0000","last_became_peered":"2023-10-16T14:11:52.991726+0000","last_unstale":"2023-10-16T14:11:52.993667+0000","last_undegraded":"2023-10-16T14:11:52.993667+0000","last_fullsized":"2023-10-16T14:11:52.993667+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:59.718175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718933+0000","last_change":"2023-10-16T14:11:51.979772+0000","last_active":"2023-10-16T14:11:54.718933+0000","last_peered":"2023-10-16T14:11:54.718933+0000","last_clean":"2023-10-16T14:11:54.718933+0000","last_became_active":"2023-10-16T14:11:51.979485+0000","last_became_peered":"2023-10-16T14:11:51.979485+0000","last_unstale":"2023-10-16T14:11:54.718933+0000","last_undegraded":"2023-10-16T14:11:54.718933+0000","last_fullsized":"2023-10-16T14:11:54.718933+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:07.687465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.987866+0000","last_change":"2023-10-16T14:11:38.811457+0000","last_active":"2023-10-16T14:11:51.987866+0000","last_peered":"2023-10-16T14:11:51.987866+0000","last_clean":"2023-10-16T14:11:51.987866+0000","last_became_active":"2023-10-16T14:11:38.811124+0000","last_became_peered":"2023-10-16T14:11:38.811124+0000","last_unstale":"2023-10-16T14:11:51.987866+0000","last_undegraded":"2023-10-16T14:11:51.987866+0000","last_fullsized":"2023-10-16T14:11:51.987866+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:54.535585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.859198+0000","last_change":"2023-10-16T14:11:52.995293+0000","last_active":"2023-10-16T14:11:54.859198+0000","last_peered":"2023-10-16T14:11:54.859198+0000","last_clean":"2023-10-16T14:11:54.859198+0000","last_became_active":"2023-10-16T14:11:52.995029+0000","last_became_peered":"2023-10-16T14:11:52.995029+0000","last_unstale":"2023-10-16T14:11:54.859198+0000","last_undegraded":"2023-10-16T14:11:54.859198+0000","last_fullsized":"2023-10-16T14:11:54.859198+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:54.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988497+0000","last_change":"2023-10-16T14:11:51.990092+0000","last_active":"2023-10-16T14:11:53.988497+0000","last_peered":"2023-10-16T14:11:53.988497+0000","last_clean":"2023-10-16T14:11:53.988497+0000","last_became_active":"2023-10-16T14:11:51.987479+0000","last_became_peered":"2023-10-16T14:11:51.987479+0000","last_unstale":"2023-10-16T14:11:53.988497+0000","last_undegraded":"2023-10-16T14:11:53.988497+0000","last_fullsized":"2023-10-16T14:11:53.988497+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:06.455649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974822+0000","last_change":"2023-10-16T14:11:38.792747+0000","last_active":"2023-10-16T14:11:52.974822+0000","last_peered":"2023-10-16T14:11:52.974822+0000","last_clean":"2023-10-16T14:11:52.974822+0000","last_became_active":"2023-10-16T14:11:38.792334+0000","last_became_peered":"2023-10-16T14:11:38.792334+0000","last_unstale":"2023-10-16T14:11:52.974822+0000","last_undegraded":"2023-10-16T14:11:52.974822+0000","last_fullsized":"2023-10-16T14:11:52.974822+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:23.689830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858462+0000","last_change":"2023-10-16T14:11:52.983967+0000","last_active":"2023-10-16T14:11:54.858462+0000","last_peered":"2023-10-16T14:11:54.858462+0000","last_clean":"2023-10-16T14:11:54.858462+0000","last_became_active":"2023-10-16T14:11:52.983433+0000","last_became_peered":"2023-10-16T14:11:52.983433+0000","last_unstale":"2023-10-16T14:11:54.858462+0000","last_undegraded":"2023-10-16T14:11:54.858462+0000","last_fullsized":"2023-10-16T14:11:54.858462+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:30.851220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994494+0000","last_change":"2023-10-16T14:11:51.984943+0000","last_active":"2023-10-16T14:11:53.994494+0000","last_peered":"2023-10-16T14:11:53.994494+0000","last_clean":"2023-10-16T14:11:53.994494+0000","last_became_active":"2023-10-16T14:11:51.984081+0000","last_became_peered":"2023-10-16T14:11:51.984081+0000","last_unstale":"2023-10-16T14:11:53.994494+0000","last_undegraded":"2023-10-16T14:11:53.994494+0000","last_fullsized":"2023-10-16T14:11:53.994494+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:50.972815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.983485+0000","last_change":"2023-10-16T14:11:38.814105+0000","last_active":"2023-10-16T14:11:51.983485+0000","last_peered":"2023-10-16T14:11:51.983485+0000","last_clean":"2023-10-16T14:11:51.983485+0000","last_became_active":"2023-10-16T14:11:38.813562+0000","last_became_peered":"2023-10-16T14:11:38.813562+0000","last_unstale":"2023-10-16T14:11:51.983485+0000","last_undegraded":"2023-10-16T14:11:51.983485+0000","last_fullsized":"2023-10-16T14:11:51.983485+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:33.419442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988316+0000","last_change":"2023-10-16T14:11:53.003492+0000","last_active":"2023-10-16T14:11:53.988316+0000","last_peered":"2023-10-16T14:11:53.988316+0000","last_clean":"2023-10-16T14:11:53.988316+0000","last_became_active":"2023-10-16T14:11:53.002918+0000","last_became_peered":"2023-10-16T14:11:53.002918+0000","last_unstale":"2023-10-16T14:11:53.988316+0000","last_undegraded":"2023-10-16T14:11:53.988316+0000","last_fullsized":"2023-10-16T14:11:53.988316+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:29.134870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858931+0000","last_change":"2023-10-16T14:11:51.986183+0000","last_active":"2023-10-16T14:11:54.858931+0000","last_peered":"2023-10-16T14:11:54.858931+0000","last_clean":"2023-10-16T14:11:54.858931+0000","last_became_active":"2023-10-16T14:11:51.985896+0000","last_became_peered":"2023-10-16T14:11:51.985896+0000","last_unstale":"2023-10-16T14:11:54.858931+0000","last_undegraded":"2023-10-16T14:11:54.858931+0000","last_fullsized":"2023-10-16T14:11:54.858931+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:50.638980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974461+0000","last_change":"2023-10-16T14:11:38.807311+0000","last_active":"2023-10-16T14:11:52.974461+0000","last_peered":"2023-10-16T14:11:52.974461+0000","last_clean":"2023-10-16T14:11:52.974461+0000","last_became_active":"2023-10-16T14:11:38.806946+0000","last_became_peered":"2023-10-16T14:11:38.806946+0000","last_unstale":"2023-10-16T14:11:52.974461+0000","last_undegraded":"2023-10-16T14:11:52.974461+0000","last_fullsized":"2023-10-16T14:11:52.974461+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:12.192825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.005131+0000","last_change":"2023-10-16T14:11:52.998297+0000","last_active":"2023-10-16T14:11:53.005131+0000","last_peered":"2023-10-16T14:11:53.005131+0000","last_clean":"2023-10-16T14:11:53.005131+0000","last_became_active":"2023-10-16T14:11:52.997812+0000","last_became_peered":"2023-10-16T14:11:52.997812+0000","last_unstale":"2023-10-16T14:11:53.005131+0000","last_undegraded":"2023-10-16T14:11:53.005131+0000","last_fullsized":"2023-10-16T14:11:53.005131+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:23.097797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"255'2","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.009150+0000","last_change":"2023-10-16T14:11:51.988969+0000","last_active":"2023-10-16T14:11:53.009150+0000","last_peered":"2023-10-16T14:11:53.009150+0000","last_clean":"2023-10-16T14:11:53.009150+0000","last_became_active":"2023-10-16T14:11:51.988313+0000","last_became_peered":"2023-10-16T14:11:51.988313+0000","last_unstale":"2023-10-16T14:11:53.009150+0000","last_undegraded":"2023-10-16T14:11:53.009150+0000","last_fullsized":"2023-10-16T14:11:53.009150+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:41.227136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.987341+0000","last_change":"2023-10-16T14:11:38.796621+0000","last_active":"2023-10-16T14:11:52.987341+0000","last_peered":"2023-10-16T14:11:52.987341+0000","last_clean":"2023-10-16T14:11:52.987341+0000","last_became_active":"2023-10-16T14:11:38.795857+0000","last_became_peered":"2023-10-16T14:11:38.795857+0000","last_unstale":"2023-10-16T14:11:52.987341+0000","last_undegraded":"2023-10-16T14:11:52.987341+0000","last_fullsized":"2023-10-16T14:11:52.987341+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:24.189032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718190+0000","last_change":"2023-10-16T14:11:52.994713+0000","last_active":"2023-10-16T14:11:54.718190+0000","last_peered":"2023-10-16T14:11:54.718190+0000","last_clean":"2023-10-16T14:11:54.718190+0000","last_became_active":"2023-10-16T14:11:52.994427+0000","last_became_peered":"2023-10-16T14:11:52.994427+0000","last_unstale":"2023-10-16T14:11:54.718190+0000","last_undegraded":"2023-10-16T14:11:54.718190+0000","last_fullsized":"2023-10-16T14:11:54.718190+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:11.895561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.068141+0000","last_change":"2023-10-16T14:11:51.977729+0000","last_active":"2023-10-16T14:11:55.068141+0000","last_peered":"2023-10-16T14:11:55.068141+0000","last_clean":"2023-10-16T14:11:55.068141+0000","last_became_active":"2023-10-16T14:11:51.977088+0000","last_became_peered":"2023-10-16T14:11:51.977088+0000","last_unstale":"2023-10-16T14:11:55.068141+0000","last_undegraded":"2023-10-16T14:11:55.068141+0000","last_fullsized":"2023-10-16T14:11:55.068141+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:45.408681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979164+0000","last_change":"2023-10-16T14:11:38.807477+0000","last_active":"2023-10-16T14:11:52.979164+0000","last_peered":"2023-10-16T14:11:52.979164+0000","last_clean":"2023-10-16T14:11:52.979164+0000","last_became_active":"2023-10-16T14:11:38.807149+0000","last_became_peered":"2023-10-16T14:11:38.807149+0000","last_unstale":"2023-10-16T14:11:52.979164+0000","last_undegraded":"2023-10-16T14:11:52.979164+0000","last_fullsized":"2023-10-16T14:11:52.979164+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:16.463192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986793+0000","last_change":"2023-10-16T14:11:52.988356+0000","last_active":"2023-10-16T14:11:53.986793+0000","last_peered":"2023-10-16T14:11:53.986793+0000","last_clean":"2023-10-16T14:11:53.986793+0000","last_became_active":"2023-10-16T14:11:52.986771+0000","last_became_peered":"2023-10-16T14:11:52.986771+0000","last_unstale":"2023-10-16T14:11:53.986793+0000","last_undegraded":"2023-10-16T14:11:53.986793+0000","last_fullsized":"2023-10-16T14:11:53.986793+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:35.593973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.992530+0000","last_change":"2023-10-16T14:11:51.988763+0000","last_active":"2023-10-16T14:11:53.992530+0000","last_peered":"2023-10-16T14:11:53.992530+0000","last_clean":"2023-10-16T14:11:53.992530+0000","last_became_active":"2023-10-16T14:11:51.988375+0000","last_became_peered":"2023-10-16T14:11:51.988375+0000","last_unstale":"2023-10-16T14:11:53.992530+0000","last_undegraded":"2023-10-16T14:11:53.992530+0000","last_fullsized":"2023-10-16T14:11:53.992530+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:33.934070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979076+0000","last_change":"2023-10-16T14:11:38.801109+0000","last_active":"2023-10-16T14:11:52.979076+0000","last_peered":"2023-10-16T14:11:52.979076+0000","last_clean":"2023-10-16T14:11:52.979076+0000","last_became_active":"2023-10-16T14:11:38.800831+0000","last_became_peered":"2023-10-16T14:11:38.800831+0000","last_unstale":"2023-10-16T14:11:52.979076+0000","last_undegraded":"2023-10-16T14:11:52.979076+0000","last_fullsized":"2023-10-16T14:11:52.979076+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:04.386150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646630+0000","last_change":"2023-10-16T14:11:52.989673+0000","last_active":"2023-10-16T14:11:54.646630+0000","last_peered":"2023-10-16T14:11:54.646630+0000","last_clean":"2023-10-16T14:11:54.646630+0000","last_became_active":"2023-10-16T14:11:52.989381+0000","last_became_peered":"2023-10-16T14:11:52.989381+0000","last_unstale":"2023-10-16T14:11:54.646630+0000","last_undegraded":"2023-10-16T14:11:54.646630+0000","last_fullsized":"2023-10-16T14:11:54.646630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:36.641150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.068799+0000","last_change":"2023-10-16T14:11:51.979510+0000","last_active":"2023-10-16T14:11:55.068799+0000","last_peered":"2023-10-16T14:11:55.068799+0000","last_clean":"2023-10-16T14:11:55.068799+0000","last_became_active":"2023-10-16T14:11:51.977340+0000","last_became_peered":"2023-10-16T14:11:51.977340+0000","last_unstale":"2023-10-16T14:11:55.068799+0000","last_undegraded":"2023-10-16T14:11:55.068799+0000","last_fullsized":"2023-10-16T14:11:55.068799+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:31.470827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974499+0000","last_change":"2023-10-16T14:11:38.806641+0000","last_active":"2023-10-16T14:11:52.974499+0000","last_peered":"2023-10-16T14:11:52.974499+0000","last_clean":"2023-10-16T14:11:52.974499+0000","last_became_active":"2023-10-16T14:11:38.806333+0000","last_became_peered":"2023-10-16T14:11:38.806333+0000","last_unstale":"2023-10-16T14:11:52.974499+0000","last_undegraded":"2023-10-16T14:11:52.974499+0000","last_fullsized":"2023-10-16T14:11:52.974499+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:03.262300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645519+0000","last_change":"2023-10-16T14:11:52.990635+0000","last_active":"2023-10-16T14:11:54.645519+0000","last_peered":"2023-10-16T14:11:54.645519+0000","last_clean":"2023-10-16T14:11:54.645519+0000","last_became_active":"2023-10-16T14:11:52.990024+0000","last_became_peered":"2023-10-16T14:11:52.990024+0000","last_unstale":"2023-10-16T14:11:54.645519+0000","last_undegraded":"2023-10-16T14:11:54.645519+0000","last_fullsized":"2023-10-16T14:11:54.645519+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:20.297500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858170+0000","last_change":"2023-10-16T14:11:51.981485+0000","last_active":"2023-10-16T14:11:54.858170+0000","last_peered":"2023-10-16T14:11:54.858170+0000","last_clean":"2023-10-16T14:11:54.858170+0000","last_became_active":"2023-10-16T14:11:51.979549+0000","last_became_peered":"2023-10-16T14:11:51.979549+0000","last_unstale":"2023-10-16T14:11:54.858170+0000","last_undegraded":"2023-10-16T14:11:54.858170+0000","last_fullsized":"2023-10-16T14:11:54.858170+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:31.739211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992402+0000","last_change":"2023-10-16T14:11:38.803389+0000","last_active":"2023-10-16T14:11:52.992402+0000","last_peered":"2023-10-16T14:11:52.992402+0000","last_clean":"2023-10-16T14:11:52.992402+0000","last_became_active":"2023-10-16T14:11:38.803101+0000","last_became_peered":"2023-10-16T14:11:38.803101+0000","last_unstale":"2023-10-16T14:11:52.992402+0000","last_undegraded":"2023-10-16T14:11:52.992402+0000","last_fullsized":"2023-10-16T14:11:52.992402+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:45.651230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.859165+0000","last_change":"2023-10-16T14:11:52.985569+0000","last_active":"2023-10-16T14:11:54.859165+0000","last_peered":"2023-10-16T14:11:54.859165+0000","last_clean":"2023-10-16T14:11:54.859165+0000","last_became_active":"2023-10-16T14:11:52.985244+0000","last_became_peered":"2023-10-16T14:11:52.985244+0000","last_unstale":"2023-10-16T14:11:54.859165+0000","last_undegraded":"2023-10-16T14:11:54.859165+0000","last_fullsized":"2023-10-16T14:11:54.859165+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:45.991538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.984498+0000","last_change":"2023-10-16T14:11:51.988535+0000","last_active":"2023-10-16T14:11:52.984498+0000","last_peered":"2023-10-16T14:11:52.984498+0000","last_clean":"2023-10-16T14:11:52.984498+0000","last_became_active":"2023-10-16T14:11:51.987601+0000","last_became_peered":"2023-10-16T14:11:51.987601+0000","last_unstale":"2023-10-16T14:11:52.984498+0000","last_undegraded":"2023-10-16T14:11:52.984498+0000","last_fullsized":"2023-10-16T14:11:52.984498+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:23.049883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976355+0000","last_change":"2023-10-16T14:11:38.811074+0000","last_active":"2023-10-16T14:11:52.976355+0000","last_peered":"2023-10-16T14:11:52.976355+0000","last_clean":"2023-10-16T14:11:52.976355+0000","last_became_active":"2023-10-16T14:11:38.810772+0000","last_became_peered":"2023-10-16T14:11:38.810772+0000","last_unstale":"2023-10-16T14:11:52.976355+0000","last_undegraded":"2023-10-16T14:11:52.976355+0000","last_fullsized":"2023-10-16T14:11:52.976355+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:03.977528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.003403+0000","last_change":"2023-10-16T14:11:52.997843+0000","last_active":"2023-10-16T14:11:53.003403+0000","last_peered":"2023-10-16T14:11:53.003403+0000","last_clean":"2023-10-16T14:11:53.003403+0000","last_became_active":"2023-10-16T14:11:52.997556+0000","last_became_peered":"2023-10-16T14:11:52.997556+0000","last_unstale":"2023-10-16T14:11:53.003403+0000","last_undegraded":"2023-10-16T14:11:53.003403+0000","last_fullsized":"2023-10-16T14:11:53.003403+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:51.362761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994722+0000","last_change":"2023-10-16T14:11:51.988518+0000","last_active":"2023-10-16T14:11:53.994722+0000","last_peered":"2023-10-16T14:11:53.994722+0000","last_clean":"2023-10-16T14:11:53.994722+0000","last_became_active":"2023-10-16T14:11:51.988105+0000","last_became_peered":"2023-10-16T14:11:51.988105+0000","last_unstale":"2023-10-16T14:11:53.994722+0000","last_undegraded":"2023-10-16T14:11:53.994722+0000","last_fullsized":"2023-10-16T14:11:53.994722+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:12.661473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.985004+0000","last_change":"2023-10-16T14:11:38.806714+0000","last_active":"2023-10-16T14:11:52.985004+0000","last_peered":"2023-10-16T14:11:52.985004+0000","last_clean":"2023-10-16T14:11:52.985004+0000","last_became_active":"2023-10-16T14:11:38.805598+0000","last_became_peered":"2023-10-16T14:11:38.805598+0000","last_unstale":"2023-10-16T14:11:52.985004+0000","last_undegraded":"2023-10-16T14:11:52.985004+0000","last_fullsized":"2023-10-16T14:11:52.985004+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:37.163662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995828+0000","last_change":"2023-10-16T14:11:53.005505+0000","last_active":"2023-10-16T14:11:53.995828+0000","last_peered":"2023-10-16T14:11:53.995828+0000","last_clean":"2023-10-16T14:11:53.995828+0000","last_became_active":"2023-10-16T14:11:53.005195+0000","last_became_peered":"2023-10-16T14:11:53.005195+0000","last_unstale":"2023-10-16T14:11:53.995828+0000","last_undegraded":"2023-10-16T14:11:53.995828+0000","last_fullsized":"2023-10-16T14:11:53.995828+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:21.280292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646874+0000","last_change":"2023-10-16T14:11:51.982174+0000","last_active":"2023-10-16T14:11:54.646874+0000","last_peered":"2023-10-16T14:11:54.646874+0000","last_clean":"2023-10-16T14:11:54.646874+0000","last_became_active":"2023-10-16T14:11:51.981827+0000","last_became_peered":"2023-10-16T14:11:51.981827+0000","last_unstale":"2023-10-16T14:11:54.646874+0000","last_undegraded":"2023-10-16T14:11:54.646874+0000","last_fullsized":"2023-10-16T14:11:54.646874+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:19.718790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982855+0000","last_change":"2023-10-16T14:11:38.808575+0000","last_active":"2023-10-16T14:11:52.982855+0000","last_peered":"2023-10-16T14:11:52.982855+0000","last_clean":"2023-10-16T14:11:52.982855+0000","last_became_active":"2023-10-16T14:11:38.808273+0000","last_became_peered":"2023-10-16T14:11:38.808273+0000","last_unstale":"2023-10-16T14:11:52.982855+0000","last_undegraded":"2023-10-16T14:11:52.982855+0000","last_fullsized":"2023-10-16T14:11:52.982855+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:08:06.098401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.005109+0000","last_change":"2023-10-16T14:11:52.998086+0000","last_active":"2023-10-16T14:11:53.005109+0000","last_peered":"2023-10-16T14:11:53.005109+0000","last_clean":"2023-10-16T14:11:53.005109+0000","last_became_active":"2023-10-16T14:11:52.997399+0000","last_became_peered":"2023-10-16T14:11:52.997399+0000","last_unstale":"2023-10-16T14:11:53.005109+0000","last_undegraded":"2023-10-16T14:11:53.005109+0000","last_fullsized":"2023-10-16T14:11:53.005109+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:16.114233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069419+0000","last_change":"2023-10-16T14:11:51.981635+0000","last_active":"2023-10-16T14:11:55.069419+0000","last_peered":"2023-10-16T14:11:55.069419+0000","last_clean":"2023-10-16T14:11:55.069419+0000","last_became_active":"2023-10-16T14:11:51.981301+0000","last_became_peered":"2023-10-16T14:11:51.981301+0000","last_unstale":"2023-10-16T14:11:55.069419+0000","last_undegraded":"2023-10-16T14:11:55.069419+0000","last_fullsized":"2023-10-16T14:11:55.069419+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:16.411541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959818+0000","last_change":"2023-10-16T14:11:38.804694+0000","last_active":"2023-10-16T14:11:52.959818+0000","last_peered":"2023-10-16T14:11:52.959818+0000","last_clean":"2023-10-16T14:11:52.959818+0000","last_became_active":"2023-10-16T14:11:38.804373+0000","last_became_peered":"2023-10-16T14:11:38.804373+0000","last_unstale":"2023-10-16T14:11:52.959818+0000","last_undegraded":"2023-10-16T14:11:52.959818+0000","last_fullsized":"2023-10-16T14:11:52.959818+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:42.092693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986689+0000","last_change":"2023-10-16T14:11:52.984083+0000","last_active":"2023-10-16T14:11:53.986689+0000","last_peered":"2023-10-16T14:11:53.986689+0000","last_clean":"2023-10-16T14:11:53.986689+0000","last_became_active":"2023-10-16T14:11:52.981380+0000","last_became_peered":"2023-10-16T14:11:52.981380+0000","last_unstale":"2023-10-16T14:11:53.986689+0000","last_undegraded":"2023-10-16T14:11:53.986689+0000","last_fullsized":"2023-10-16T14:11:53.986689+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:02:24.467082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"255'8","reported_seq":20,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645058+0000","last_change":"2023-10-16T14:11:51.985634+0000","last_active":"2023-10-16T14:11:54.645058+0000","last_peered":"2023-10-16T14:11:54.645058+0000","last_clean":"2023-10-16T14:11:54.645058+0000","last_became_active":"2023-10-16T14:11:51.985328+0000","last_became_peered":"2023-10-16T14:11:51.985328+0000","last_unstale":"2023-10-16T14:11:54.645058+0000","last_undegraded":"2023-10-16T14:11:54.645058+0000","last_fullsized":"2023-10-16T14:11:54.645058+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:06.367330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993652+0000","last_change":"2023-10-16T14:11:38.799470+0000","last_active":"2023-10-16T14:11:52.993652+0000","last_peered":"2023-10-16T14:11:52.993652+0000","last_clean":"2023-10-16T14:11:52.993652+0000","last_became_active":"2023-10-16T14:11:38.799170+0000","last_became_peered":"2023-10-16T14:11:38.799170+0000","last_unstale":"2023-10-16T14:11:52.993652+0000","last_undegraded":"2023-10-16T14:11:52.993652+0000","last_fullsized":"2023-10-16T14:11:52.993652+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:23.948030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986664+0000","last_change":"2023-10-16T14:11:52.993056+0000","last_active":"2023-10-16T14:11:53.986664+0000","last_peered":"2023-10-16T14:11:53.986664+0000","last_clean":"2023-10-16T14:11:53.986664+0000","last_became_active":"2023-10-16T14:11:52.992768+0000","last_became_peered":"2023-10-16T14:11:52.992768+0000","last_unstale":"2023-10-16T14:11:53.986664+0000","last_undegraded":"2023-10-16T14:11:53.986664+0000","last_fullsized":"2023-10-16T14:11:53.986664+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:43.826793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.992758+0000","last_change":"2023-10-16T14:11:51.981573+0000","last_active":"2023-10-16T14:11:53.992758+0000","last_peered":"2023-10-16T14:11:53.992758+0000","last_clean":"2023-10-16T14:11:53.992758+0000","last_became_active":"2023-10-16T14:11:51.981268+0000","last_became_peered":"2023-10-16T14:11:51.981268+0000","last_unstale":"2023-10-16T14:11:53.992758+0000","last_undegraded":"2023-10-16T14:11:53.992758+0000","last_fullsized":"2023-10-16T14:11:53.992758+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:55.121470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993774+0000","last_change":"2023-10-16T14:11:38.801518+0000","last_active":"2023-10-16T14:11:52.993774+0000","last_peered":"2023-10-16T14:11:52.993774+0000","last_clean":"2023-10-16T14:11:52.993774+0000","last_became_active":"2023-10-16T14:11:38.801133+0000","last_became_peered":"2023-10-16T14:11:38.801133+0000","last_unstale":"2023-10-16T14:11:52.993774+0000","last_undegraded":"2023-10-16T14:11:52.993774+0000","last_fullsized":"2023-10-16T14:11:52.993774+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:30.423238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646309+0000","last_change":"2023-10-16T14:11:53.002757+0000","last_active":"2023-10-16T14:11:54.646309+0000","last_peered":"2023-10-16T14:11:54.646309+0000","last_clean":"2023-10-16T14:11:54.646309+0000","last_became_active":"2023-10-16T14:11:53.002500+0000","last_became_peered":"2023-10-16T14:11:53.002500+0000","last_unstale":"2023-10-16T14:11:54.646309+0000","last_undegraded":"2023-10-16T14:11:54.646309+0000","last_fullsized":"2023-10-16T14:11:54.646309+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:28.617619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718369+0000","last_change":"2023-10-16T14:11:51.977982+0000","last_active":"2023-10-16T14:11:54.718369+0000","last_peered":"2023-10-16T14:11:54.718369+0000","last_clean":"2023-10-16T14:11:54.718369+0000","last_became_active":"2023-10-16T14:11:51.977655+0000","last_became_peered":"2023-10-16T14:11:51.977655+0000","last_unstale":"2023-10-16T14:11:54.718369+0000","last_undegraded":"2023-10-16T14:11:54.718369+0000","last_fullsized":"2023-10-16T14:11:54.718369+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:44.429107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992657+0000","last_change":"2023-10-16T14:11:38.792372+0000","last_active":"2023-10-16T14:11:52.992657+0000","last_peered":"2023-10-16T14:11:52.992657+0000","last_clean":"2023-10-16T14:11:52.992657+0000","last_became_active":"2023-10-16T14:11:38.789719+0000","last_became_peered":"2023-10-16T14:11:38.789719+0000","last_unstale":"2023-10-16T14:11:52.992657+0000","last_undegraded":"2023-10-16T14:11:52.992657+0000","last_fullsized":"2023-10-16T14:11:52.992657+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:30:12.869949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987669+0000","last_change":"2023-10-16T14:11:52.979953+0000","last_active":"2023-10-16T14:11:53.987669+0000","last_peered":"2023-10-16T14:11:53.987669+0000","last_clean":"2023-10-16T14:11:53.987669+0000","last_became_active":"2023-10-16T14:11:52.979593+0000","last_became_peered":"2023-10-16T14:11:52.979593+0000","last_unstale":"2023-10-16T14:11:53.987669+0000","last_undegraded":"2023-10-16T14:11:53.987669+0000","last_fullsized":"2023-10-16T14:11:53.987669+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:07.480987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.068595+0000","last_change":"2023-10-16T14:11:51.985483+0000","last_active":"2023-10-16T14:11:55.068595+0000","last_peered":"2023-10-16T14:11:55.068595+0000","last_clean":"2023-10-16T14:11:55.068595+0000","last_became_active":"2023-10-16T14:11:51.984905+0000","last_became_peered":"2023-10-16T14:11:51.984905+0000","last_unstale":"2023-10-16T14:11:55.068595+0000","last_undegraded":"2023-10-16T14:11:55.068595+0000","last_fullsized":"2023-10-16T14:11:55.068595+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:06.178911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960933+0000","last_change":"2023-10-16T14:11:38.802052+0000","last_active":"2023-10-16T14:11:52.960933+0000","last_peered":"2023-10-16T14:11:52.960933+0000","last_clean":"2023-10-16T14:11:52.960933+0000","last_became_active":"2023-10-16T14:11:38.801625+0000","last_became_peered":"2023-10-16T14:11:38.801625+0000","last_unstale":"2023-10-16T14:11:52.960933+0000","last_undegraded":"2023-10-16T14:11:52.960933+0000","last_fullsized":"2023-10-16T14:11:52.960933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:27.067528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987192+0000","last_change":"2023-10-16T14:11:52.997042+0000","last_active":"2023-10-16T14:11:53.987192+0000","last_peered":"2023-10-16T14:11:53.987192+0000","last_clean":"2023-10-16T14:11:53.987192+0000","last_became_active":"2023-10-16T14:11:52.996733+0000","last_became_peered":"2023-10-16T14:11:52.996733+0000","last_unstale":"2023-10-16T14:11:53.987192+0000","last_undegraded":"2023-10-16T14:11:53.987192+0000","last_fullsized":"2023-10-16T14:11:53.987192+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:37.268638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"255'6","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988373+0000","last_change":"2023-10-16T14:11:51.987716+0000","last_active":"2023-10-16T14:11:53.988373+0000","last_peered":"2023-10-16T14:11:53.988373+0000","last_clean":"2023-10-16T14:11:53.988373+0000","last_became_active":"2023-10-16T14:11:51.984796+0000","last_became_peered":"2023-10-16T14:11:51.984796+0000","last_unstale":"2023-10-16T14:11:53.988373+0000","last_undegraded":"2023-10-16T14:11:53.988373+0000","last_fullsized":"2023-10-16T14:11:53.988373+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:37.378791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.989049+0000","last_change":"2023-10-16T14:11:38.797163+0000","last_active":"2023-10-16T14:11:52.989049+0000","last_peered":"2023-10-16T14:11:52.989049+0000","last_clean":"2023-10-16T14:11:52.989049+0000","last_became_active":"2023-10-16T14:11:38.796761+0000","last_became_peered":"2023-10-16T14:11:38.796761+0000","last_unstale":"2023-10-16T14:11:52.989049+0000","last_undegraded":"2023-10-16T14:11:52.989049+0000","last_fullsized":"2023-10-16T14:11:52.989049+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:01.946364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.010978+0000","last_change":"2023-10-16T14:11:53.007335+0000","last_active":"2023-10-16T14:11:53.010978+0000","last_peered":"2023-10-16T14:11:53.010978+0000","last_clean":"2023-10-16T14:11:53.010978+0000","last_became_active":"2023-10-16T14:11:53.007036+0000","last_became_peered":"2023-10-16T14:11:53.007036+0000","last_unstale":"2023-10-16T14:11:53.010978+0000","last_undegraded":"2023-10-16T14:11:53.010978+0000","last_fullsized":"2023-10-16T14:11:53.010978+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:10.543714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"255'3","reported_seq":15,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646707+0000","last_change":"2023-10-16T14:11:51.985630+0000","last_active":"2023-10-16T14:11:54.646707+0000","last_peered":"2023-10-16T14:11:54.646707+0000","last_clean":"2023-10-16T14:11:54.646707+0000","last_became_active":"2023-10-16T14:11:51.984856+0000","last_became_peered":"2023-10-16T14:11:51.984856+0000","last_unstale":"2023-10-16T14:11:54.646707+0000","last_undegraded":"2023-10-16T14:11:54.646707+0000","last_fullsized":"2023-10-16T14:11:54.646707+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:56.393116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974570+0000","last_change":"2023-10-16T14:11:38.802974+0000","last_active":"2023-10-16T14:11:52.974570+0000","last_peered":"2023-10-16T14:11:52.974570+0000","last_clean":"2023-10-16T14:11:52.974570+0000","last_became_active":"2023-10-16T14:11:38.802711+0000","last_became_peered":"2023-10-16T14:11:38.802711+0000","last_unstale":"2023-10-16T14:11:52.974570+0000","last_undegraded":"2023-10-16T14:11:52.974570+0000","last_fullsized":"2023-10-16T14:11:52.974570+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:39.092883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988308+0000","last_change":"2023-10-16T14:11:52.998872+0000","last_active":"2023-10-16T14:11:53.988308+0000","last_peered":"2023-10-16T14:11:53.988308+0000","last_clean":"2023-10-16T14:11:53.988308+0000","last_became_active":"2023-10-16T14:11:52.998518+0000","last_became_peered":"2023-10-16T14:11:52.998518+0000","last_unstale":"2023-10-16T14:11:53.988308+0000","last_undegraded":"2023-10-16T14:11:53.988308+0000","last_fullsized":"2023-10-16T14:11:53.988308+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:54.509580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645834+0000","last_change":"2023-10-16T14:11:51.984957+0000","last_active":"2023-10-16T14:11:54.645834+0000","last_peered":"2023-10-16T14:11:54.645834+0000","last_clean":"2023-10-16T14:11:54.645834+0000","last_became_active":"2023-10-16T14:11:51.984681+0000","last_became_peered":"2023-10-16T14:11:51.984681+0000","last_unstale":"2023-10-16T14:11:54.645834+0000","last_undegraded":"2023-10-16T14:11:54.645834+0000","last_fullsized":"2023-10-16T14:11:54.645834+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:29.591532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991986+0000","last_change":"2023-10-16T14:11:38.800092+0000","last_active":"2023-10-16T14:11:52.991986+0000","last_peered":"2023-10-16T14:11:52.991986+0000","last_clean":"2023-10-16T14:11:52.991986+0000","last_became_active":"2023-10-16T14:11:38.799625+0000","last_became_peered":"2023-10-16T14:11:38.799625+0000","last_unstale":"2023-10-16T14:11:52.991986+0000","last_undegraded":"2023-10-16T14:11:52.991986+0000","last_fullsized":"2023-10-16T14:11:52.991986+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:06.091417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994574+0000","last_change":"2023-10-16T14:11:52.984916+0000","last_active":"2023-10-16T14:11:53.994574+0000","last_peered":"2023-10-16T14:11:53.994574+0000","last_clean":"2023-10-16T14:11:53.994574+0000","last_became_active":"2023-10-16T14:11:52.984211+0000","last_became_peered":"2023-10-16T14:11:52.984211+0000","last_unstale":"2023-10-16T14:11:53.994574+0000","last_undegraded":"2023-10-16T14:11:53.994574+0000","last_fullsized":"2023-10-16T14:11:53.994574+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:42.735397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988479+0000","last_change":"2023-10-16T14:11:51.984778+0000","last_active":"2023-10-16T14:11:53.988479+0000","last_peered":"2023-10-16T14:11:53.988479+0000","last_clean":"2023-10-16T14:11:53.988479+0000","last_became_active":"2023-10-16T14:11:51.984314+0000","last_became_peered":"2023-10-16T14:11:51.984314+0000","last_unstale":"2023-10-16T14:11:53.988479+0000","last_undegraded":"2023-10-16T14:11:53.988479+0000","last_fullsized":"2023-10-16T14:11:53.988479+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:11.025122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982626+0000","last_change":"2023-10-16T14:11:38.809521+0000","last_active":"2023-10-16T14:11:52.982626+0000","last_peered":"2023-10-16T14:11:52.982626+0000","last_clean":"2023-10-16T14:11:52.982626+0000","last_became_active":"2023-10-16T14:11:38.808121+0000","last_became_peered":"2023-10-16T14:11:38.808121+0000","last_unstale":"2023-10-16T14:11:52.982626+0000","last_undegraded":"2023-10-16T14:11:52.982626+0000","last_fullsized":"2023-10-16T14:11:52.982626+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:55.831023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993398+0000","last_change":"2023-10-16T14:11:52.992961+0000","last_active":"2023-10-16T14:11:53.993398+0000","last_peered":"2023-10-16T14:11:53.993398+0000","last_clean":"2023-10-16T14:11:53.993398+0000","last_became_active":"2023-10-16T14:11:52.992670+0000","last_became_peered":"2023-10-16T14:11:52.992670+0000","last_unstale":"2023-10-16T14:11:53.993398+0000","last_undegraded":"2023-10-16T14:11:53.993398+0000","last_fullsized":"2023-10-16T14:11:53.993398+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:20.408918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995259+0000","last_change":"2023-10-16T14:11:51.989138+0000","last_active":"2023-10-16T14:11:53.995259+0000","last_peered":"2023-10-16T14:11:53.995259+0000","last_clean":"2023-10-16T14:11:53.995259+0000","last_became_active":"2023-10-16T14:11:51.988830+0000","last_became_peered":"2023-10-16T14:11:51.988830+0000","last_unstale":"2023-10-16T14:11:53.995259+0000","last_undegraded":"2023-10-16T14:11:53.995259+0000","last_fullsized":"2023-10-16T14:11:53.995259+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:35.140481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.988733+0000","last_change":"2023-10-16T14:11:38.811608+0000","last_active":"2023-10-16T14:11:51.988733+0000","last_peered":"2023-10-16T14:11:51.988733+0000","last_clean":"2023-10-16T14:11:51.988733+0000","last_became_active":"2023-10-16T14:11:38.811328+0000","last_became_peered":"2023-10-16T14:11:38.811328+0000","last_unstale":"2023-10-16T14:11:51.988733+0000","last_undegraded":"2023-10-16T14:11:51.988733+0000","last_fullsized":"2023-10-16T14:11:51.988733+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:39.866362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.644771+0000","last_change":"2023-10-16T14:11:53.001822+0000","last_active":"2023-10-16T14:11:54.644771+0000","last_peered":"2023-10-16T14:11:54.644771+0000","last_clean":"2023-10-16T14:11:54.644771+0000","last_became_active":"2023-10-16T14:11:53.001552+0000","last_became_peered":"2023-10-16T14:11:53.001552+0000","last_unstale":"2023-10-16T14:11:54.644771+0000","last_undegraded":"2023-10-16T14:11:54.644771+0000","last_fullsized":"2023-10-16T14:11:54.644771+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:19.081829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.857803+0000","last_change":"2023-10-16T14:11:51.986975+0000","last_active":"2023-10-16T14:11:54.857803+0000","last_peered":"2023-10-16T14:11:54.857803+0000","last_clean":"2023-10-16T14:11:54.857803+0000","last_became_active":"2023-10-16T14:11:51.986639+0000","last_became_peered":"2023-10-16T14:11:51.986639+0000","last_unstale":"2023-10-16T14:11:54.857803+0000","last_undegraded":"2023-10-16T14:11:54.857803+0000","last_fullsized":"2023-10-16T14:11:54.857803+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:46.717029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979167+0000","last_change":"2023-10-16T14:11:38.811158+0000","last_active":"2023-10-16T14:11:52.979167+0000","last_peered":"2023-10-16T14:11:52.979167+0000","last_clean":"2023-10-16T14:11:52.979167+0000","last_became_active":"2023-10-16T14:11:38.810852+0000","last_became_peered":"2023-10-16T14:11:38.810852+0000","last_unstale":"2023-10-16T14:11:52.979167+0000","last_undegraded":"2023-10-16T14:11:52.979167+0000","last_fullsized":"2023-10-16T14:11:52.979167+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:09:39.319181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646246+0000","last_change":"2023-10-16T14:11:52.998892+0000","last_active":"2023-10-16T14:11:54.646246+0000","last_peered":"2023-10-16T14:11:54.646246+0000","last_clean":"2023-10-16T14:11:54.646246+0000","last_became_active":"2023-10-16T14:11:52.998383+0000","last_became_peered":"2023-10-16T14:11:52.998383+0000","last_unstale":"2023-10-16T14:11:54.646246+0000","last_undegraded":"2023-10-16T14:11:54.646246+0000","last_fullsized":"2023-10-16T14:11:54.646246+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:30.662291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987327+0000","last_change":"2023-10-16T14:11:51.975784+0000","last_active":"2023-10-16T14:11:53.987327+0000","last_peered":"2023-10-16T14:11:53.987327+0000","last_clean":"2023-10-16T14:11:53.987327+0000","last_became_active":"2023-10-16T14:11:51.975488+0000","last_became_peered":"2023-10-16T14:11:51.975488+0000","last_unstale":"2023-10-16T14:11:53.987327+0000","last_undegraded":"2023-10-16T14:11:53.987327+0000","last_fullsized":"2023-10-16T14:11:53.987327+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:35.286479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974585+0000","last_change":"2023-10-16T14:11:38.804475+0000","last_active":"2023-10-16T14:11:52.974585+0000","last_peered":"2023-10-16T14:11:52.974585+0000","last_clean":"2023-10-16T14:11:52.974585+0000","last_became_active":"2023-10-16T14:11:38.804187+0000","last_became_peered":"2023-10-16T14:11:38.804187+0000","last_unstale":"2023-10-16T14:11:52.974585+0000","last_undegraded":"2023-10-16T14:11:52.974585+0000","last_fullsized":"2023-10-16T14:11:52.974585+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:59.201892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718944+0000","last_change":"2023-10-16T14:11:52.986688+0000","last_active":"2023-10-16T14:11:54.718944+0000","last_peered":"2023-10-16T14:11:54.718944+0000","last_clean":"2023-10-16T14:11:54.718944+0000","last_became_active":"2023-10-16T14:11:52.985883+0000","last_became_peered":"2023-10-16T14:11:52.985883+0000","last_unstale":"2023-10-16T14:11:54.718944+0000","last_undegraded":"2023-10-16T14:11:54.718944+0000","last_fullsized":"2023-10-16T14:11:54.718944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:51.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"255'1","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.996638+0000","last_change":"2023-10-16T14:11:51.982731+0000","last_active":"2023-10-16T14:11:53.996638+0000","last_peered":"2023-10-16T14:11:53.996638+0000","last_clean":"2023-10-16T14:11:53.996638+0000","last_became_active":"2023-10-16T14:11:51.982235+0000","last_became_peered":"2023-10-16T14:11:51.982235+0000","last_unstale":"2023-10-16T14:11:53.996638+0000","last_undegraded":"2023-10-16T14:11:53.996638+0000","last_fullsized":"2023-10-16T14:11:53.996638+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:44.054294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991197+0000","last_change":"2023-10-16T14:11:38.808466+0000","last_active":"2023-10-16T14:11:52.991197+0000","last_peered":"2023-10-16T14:11:52.991197+0000","last_clean":"2023-10-16T14:11:52.991197+0000","last_became_active":"2023-10-16T14:11:38.808142+0000","last_became_peered":"2023-10-16T14:11:38.808142+0000","last_unstale":"2023-10-16T14:11:52.991197+0000","last_undegraded":"2023-10-16T14:11:52.991197+0000","last_fullsized":"2023-10-16T14:11:52.991197+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:30.136850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069006+0000","last_change":"2023-10-16T14:11:52.995110+0000","last_active":"2023-10-16T14:11:55.069006+0000","last_peered":"2023-10-16T14:11:55.069006+0000","last_clean":"2023-10-16T14:11:55.069006+0000","last_became_active":"2023-10-16T14:11:52.994839+0000","last_became_peered":"2023-10-16T14:11:52.994839+0000","last_unstale":"2023-10-16T14:11:55.069006+0000","last_undegraded":"2023-10-16T14:11:55.069006+0000","last_fullsized":"2023-10-16T14:11:55.069006+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:46.167512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646679+0000","last_change":"2023-10-16T14:11:51.981639+0000","last_active":"2023-10-16T14:11:54.646679+0000","last_peered":"2023-10-16T14:11:54.646679+0000","last_clean":"2023-10-16T14:11:54.646679+0000","last_became_active":"2023-10-16T14:11:51.981366+0000","last_became_peered":"2023-10-16T14:11:51.981366+0000","last_unstale":"2023-10-16T14:11:54.646679+0000","last_undegraded":"2023-10-16T14:11:54.646679+0000","last_fullsized":"2023-10-16T14:11:54.646679+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:11.500203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974639+0000","last_change":"2023-10-16T14:11:38.802531+0000","last_active":"2023-10-16T14:11:52.974639+0000","last_peered":"2023-10-16T14:11:52.974639+0000","last_clean":"2023-10-16T14:11:52.974639+0000","last_became_active":"2023-10-16T14:11:38.802127+0000","last_became_peered":"2023-10-16T14:11:38.802127+0000","last_unstale":"2023-10-16T14:11:52.974639+0000","last_undegraded":"2023-10-16T14:11:52.974639+0000","last_fullsized":"2023-10-16T14:11:52.974639+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:26.191109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994020+0000","last_change":"2023-10-16T14:11:52.996012+0000","last_active":"2023-10-16T14:11:53.994020+0000","last_peered":"2023-10-16T14:11:53.994020+0000","last_clean":"2023-10-16T14:11:53.994020+0000","last_became_active":"2023-10-16T14:11:52.995343+0000","last_became_peered":"2023-10-16T14:11:52.995343+0000","last_unstale":"2023-10-16T14:11:53.994020+0000","last_undegraded":"2023-10-16T14:11:53.994020+0000","last_fullsized":"2023-10-16T14:11:53.994020+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:21.988424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"255'2","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.646438+0000","last_change":"2023-10-16T14:11:51.981076+0000","last_active":"2023-10-16T14:11:54.646438+0000","last_peered":"2023-10-16T14:11:54.646438+0000","last_clean":"2023-10-16T14:11:54.646438+0000","last_became_active":"2023-10-16T14:11:51.980733+0000","last_became_peered":"2023-10-16T14:11:51.980733+0000","last_unstale":"2023-10-16T14:11:54.646438+0000","last_undegraded":"2023-10-16T14:11:54.646438+0000","last_fullsized":"2023-10-16T14:11:54.646438+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:47.589072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975151+0000","last_change":"2023-10-16T14:11:38.798475+0000","last_active":"2023-10-16T14:11:52.975151+0000","last_peered":"2023-10-16T14:11:52.975151+0000","last_clean":"2023-10-16T14:11:52.975151+0000","last_became_active":"2023-10-16T14:11:38.797198+0000","last_became_peered":"2023-10-16T14:11:38.797198+0000","last_unstale":"2023-10-16T14:11:52.975151+0000","last_undegraded":"2023-10-16T14:11:52.975151+0000","last_fullsized":"2023-10-16T14:11:52.975151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:37.361885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.005594+0000","last_change":"2023-10-16T14:11:52.999035+0000","last_active":"2023-10-16T14:11:53.005594+0000","last_peered":"2023-10-16T14:11:53.005594+0000","last_clean":"2023-10-16T14:11:53.005594+0000","last_became_active":"2023-10-16T14:11:52.998633+0000","last_became_peered":"2023-10-16T14:11:52.998633+0000","last_unstale":"2023-10-16T14:11:53.005594+0000","last_undegraded":"2023-10-16T14:11:53.005594+0000","last_fullsized":"2023-10-16T14:11:53.005594+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:40.320922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"255'1","reported_seq":12,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.010355+0000","last_change":"2023-10-16T14:11:51.990728+0000","last_active":"2023-10-16T14:11:53.010355+0000","last_peered":"2023-10-16T14:11:53.010355+0000","last_clean":"2023-10-16T14:11:53.010355+0000","last_became_active":"2023-10-16T14:11:51.990400+0000","last_became_peered":"2023-10-16T14:11:51.990400+0000","last_unstale":"2023-10-16T14:11:53.010355+0000","last_undegraded":"2023-10-16T14:11:53.010355+0000","last_fullsized":"2023-10-16T14:11:53.010355+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:53.060480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.980092+0000","last_change":"2023-10-16T14:11:38.803008+0000","last_active":"2023-10-16T14:11:51.980092+0000","last_peered":"2023-10-16T14:11:51.980092+0000","last_clean":"2023-10-16T14:11:51.980092+0000","last_became_active":"2023-10-16T14:11:38.800876+0000","last_became_peered":"2023-10-16T14:11:38.800876+0000","last_unstale":"2023-10-16T14:11:51.980092+0000","last_undegraded":"2023-10-16T14:11:51.980092+0000","last_fullsized":"2023-10-16T14:11:51.980092+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:03.954367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994944+0000","last_change":"2023-10-16T14:11:52.982991+0000","last_active":"2023-10-16T14:11:53.994944+0000","last_peered":"2023-10-16T14:11:53.994944+0000","last_clean":"2023-10-16T14:11:53.994944+0000","last_became_active":"2023-10-16T14:11:52.982265+0000","last_became_peered":"2023-10-16T14:11:52.982265+0000","last_unstale":"2023-10-16T14:11:53.994944+0000","last_undegraded":"2023-10-16T14:11:53.994944+0000","last_fullsized":"2023-10-16T14:11:53.994944+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:42.388825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"255'4","reported_seq":16,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988424+0000","last_change":"2023-10-16T14:11:51.984583+0000","last_active":"2023-10-16T14:11:53.988424+0000","last_peered":"2023-10-16T14:11:53.988424+0000","last_clean":"2023-10-16T14:11:53.988424+0000","last_became_active":"2023-10-16T14:11:51.983925+0000","last_became_peered":"2023-10-16T14:11:51.983925+0000","last_unstale":"2023-10-16T14:11:53.988424+0000","last_undegraded":"2023-10-16T14:11:53.988424+0000","last_fullsized":"2023-10-16T14:11:53.988424+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:50.920165+0000","last_clean_scrub_stamp":"2023-10-16T14:11:50.920165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:42.674017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982725+0000","last_change":"2023-10-16T14:11:38.803699+0000","last_active":"2023-10-16T14:11:52.982725+0000","last_peered":"2023-10-16T14:11:52.982725+0000","last_clean":"2023-10-16T14:11:52.982725+0000","last_became_active":"2023-10-16T14:11:38.801233+0000","last_became_peered":"2023-10-16T14:11:38.801233+0000","last_unstale":"2023-10-16T14:11:52.982725+0000","last_undegraded":"2023-10-16T14:11:52.982725+0000","last_fullsized":"2023-10-16T14:11:52.982725+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:34.743471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.067197+0000","last_change":"2023-10-16T14:11:52.994405+0000","last_active":"2023-10-16T14:11:55.067197+0000","last_peered":"2023-10-16T14:11:55.067197+0000","last_clean":"2023-10-16T14:11:55.067197+0000","last_became_active":"2023-10-16T14:11:52.994065+0000","last_became_peered":"2023-10-16T14:11:52.994065+0000","last_unstale":"2023-10-16T14:11:55.067197+0000","last_undegraded":"2023-10-16T14:11:55.067197+0000","last_fullsized":"2023-10-16T14:11:55.067197+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:55.324917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.965571+0000","last_change":"2023-10-16T14:11:38.800010+0000","last_active":"2023-10-16T14:11:52.965571+0000","last_peered":"2023-10-16T14:11:52.965571+0000","last_clean":"2023-10-16T14:11:52.965571+0000","last_became_active":"2023-10-16T14:11:38.799422+0000","last_became_peered":"2023-10-16T14:11:38.799422+0000","last_unstale":"2023-10-16T14:11:52.965571+0000","last_undegraded":"2023-10-16T14:11:52.965571+0000","last_fullsized":"2023-10-16T14:11:52.965571+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:01:31.181539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979354+0000","last_change":"2023-10-16T14:11:37.778697+0000","last_active":"2023-10-16T14:11:52.979354+0000","last_peered":"2023-10-16T14:11:52.979354+0000","last_clean":"2023-10-16T14:11:52.979354+0000","last_became_active":"2023-10-16T14:11:37.778387+0000","last_became_peered":"2023-10-16T14:11:37.778387+0000","last_unstale":"2023-10-16T14:11:52.979354+0000","last_undegraded":"2023-10-16T14:11:52.979354+0000","last_fullsized":"2023-10-16T14:11:52.979354+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:09.302616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.011033+0000","last_change":"2023-10-16T14:11:53.007479+0000","last_active":"2023-10-16T14:11:53.011033+0000","last_peered":"2023-10-16T14:11:53.011033+0000","last_clean":"2023-10-16T14:11:53.011033+0000","last_became_active":"2023-10-16T14:11:53.006978+0000","last_became_peered":"2023-10-16T14:11:53.006978+0000","last_unstale":"2023-10-16T14:11:53.011033+0000","last_undegraded":"2023-10-16T14:11:53.011033+0000","last_fullsized":"2023-10-16T14:11:53.011033+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:10.030122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.972830+0000","last_change":"2023-10-16T14:11:38.805302+0000","last_active":"2023-10-16T14:11:52.972830+0000","last_peered":"2023-10-16T14:11:52.972830+0000","last_clean":"2023-10-16T14:11:52.972830+0000","last_became_active":"2023-10-16T14:11:38.803432+0000","last_became_peered":"2023-10-16T14:11:38.803432+0000","last_unstale":"2023-10-16T14:11:52.972830+0000","last_undegraded":"2023-10-16T14:11:52.972830+0000","last_fullsized":"2023-10-16T14:11:52.972830+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:42.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959770+0000","last_change":"2023-10-16T14:11:37.781195+0000","last_active":"2023-10-16T14:11:52.959770+0000","last_peered":"2023-10-16T14:11:52.959770+0000","last_clean":"2023-10-16T14:11:52.959770+0000","last_became_active":"2023-10-16T14:11:37.780551+0000","last_became_peered":"2023-10-16T14:11:37.780551+0000","last_unstale":"2023-10-16T14:11:52.959770+0000","last_undegraded":"2023-10-16T14:11:52.959770+0000","last_fullsized":"2023-10-16T14:11:52.959770+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:33.391410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.987987+0000","last_change":"2023-10-16T14:11:52.989049+0000","last_active":"2023-10-16T14:11:53.987987+0000","last_peered":"2023-10-16T14:11:53.987987+0000","last_clean":"2023-10-16T14:11:53.987987+0000","last_became_active":"2023-10-16T14:11:52.988719+0000","last_became_peered":"2023-10-16T14:11:52.988719+0000","last_unstale":"2023-10-16T14:11:53.987987+0000","last_undegraded":"2023-10-16T14:11:53.987987+0000","last_fullsized":"2023-10-16T14:11:53.987987+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:39.795278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992813+0000","last_change":"2023-10-16T14:11:38.802734+0000","last_active":"2023-10-16T14:11:52.992813+0000","last_peered":"2023-10-16T14:11:52.992813+0000","last_clean":"2023-10-16T14:11:52.992813+0000","last_became_active":"2023-10-16T14:11:38.802347+0000","last_became_peered":"2023-10-16T14:11:38.802347+0000","last_unstale":"2023-10-16T14:11:52.992813+0000","last_undegraded":"2023-10-16T14:11:52.992813+0000","last_fullsized":"2023-10-16T14:11:52.992813+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:38.977236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961429+0000","last_change":"2023-10-16T14:11:37.783063+0000","last_active":"2023-10-16T14:11:52.961429+0000","last_peered":"2023-10-16T14:11:52.961429+0000","last_clean":"2023-10-16T14:11:52.961429+0000","last_became_active":"2023-10-16T14:11:37.782768+0000","last_became_peered":"2023-10-16T14:11:37.782768+0000","last_unstale":"2023-10-16T14:11:52.961429+0000","last_undegraded":"2023-10-16T14:11:52.961429+0000","last_fullsized":"2023-10-16T14:11:52.961429+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:06:36.980155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993428+0000","last_change":"2023-10-16T14:11:52.989636+0000","last_active":"2023-10-16T14:11:52.993428+0000","last_peered":"2023-10-16T14:11:52.993428+0000","last_clean":"2023-10-16T14:11:52.993428+0000","last_became_active":"2023-10-16T14:11:52.989294+0000","last_became_peered":"2023-10-16T14:11:52.989294+0000","last_unstale":"2023-10-16T14:11:52.993428+0000","last_undegraded":"2023-10-16T14:11:52.993428+0000","last_fullsized":"2023-10-16T14:11:52.993428+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:09.306153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973007+0000","last_change":"2023-10-16T14:11:38.786427+0000","last_active":"2023-10-16T14:11:52.973007+0000","last_peered":"2023-10-16T14:11:52.973007+0000","last_clean":"2023-10-16T14:11:52.973007+0000","last_became_active":"2023-10-16T14:11:38.785671+0000","last_became_peered":"2023-10-16T14:11:38.785671+0000","last_unstale":"2023-10-16T14:11:52.973007+0000","last_undegraded":"2023-10-16T14:11:52.973007+0000","last_fullsized":"2023-10-16T14:11:52.973007+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:08.101378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"251'1","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.984174+0000","last_change":"2023-10-16T14:11:37.774757+0000","last_active":"2023-10-16T14:11:51.984174+0000","last_peered":"2023-10-16T14:11:51.984174+0000","last_clean":"2023-10-16T14:11:51.984174+0000","last_became_active":"2023-10-16T14:11:37.774442+0000","last_became_peered":"2023-10-16T14:11:37.774442+0000","last_unstale":"2023-10-16T14:11:51.984174+0000","last_undegraded":"2023-10-16T14:11:51.984174+0000","last_fullsized":"2023-10-16T14:11:51.984174+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:45.874083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994379+0000","last_change":"2023-10-16T14:11:52.981497+0000","last_active":"2023-10-16T14:11:53.994379+0000","last_peered":"2023-10-16T14:11:53.994379+0000","last_clean":"2023-10-16T14:11:53.994379+0000","last_became_active":"2023-10-16T14:11:52.979769+0000","last_became_peered":"2023-10-16T14:11:52.979769+0000","last_unstale":"2023-10-16T14:11:53.994379+0000","last_undegraded":"2023-10-16T14:11:53.994379+0000","last_fullsized":"2023-10-16T14:11:53.994379+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:58:25.884035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975312+0000","last_change":"2023-10-16T14:11:38.809976+0000","last_active":"2023-10-16T14:11:52.975312+0000","last_peered":"2023-10-16T14:11:52.975312+0000","last_clean":"2023-10-16T14:11:52.975312+0000","last_became_active":"2023-10-16T14:11:38.809754+0000","last_became_peered":"2023-10-16T14:11:38.809754+0000","last_unstale":"2023-10-16T14:11:52.975312+0000","last_undegraded":"2023-10-16T14:11:52.975312+0000","last_fullsized":"2023-10-16T14:11:52.975312+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:14.418521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.984993+0000","last_change":"2023-10-16T14:11:37.776847+0000","last_active":"2023-10-16T14:11:52.984993+0000","last_peered":"2023-10-16T14:11:52.984993+0000","last_clean":"2023-10-16T14:11:52.984993+0000","last_became_active":"2023-10-16T14:11:37.775028+0000","last_became_peered":"2023-10-16T14:11:37.775028+0000","last_unstale":"2023-10-16T14:11:52.984993+0000","last_undegraded":"2023-10-16T14:11:52.984993+0000","last_fullsized":"2023-10-16T14:11:52.984993+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:54.089883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858815+0000","last_change":"2023-10-16T14:11:52.992668+0000","last_active":"2023-10-16T14:11:54.858815+0000","last_peered":"2023-10-16T14:11:54.858815+0000","last_clean":"2023-10-16T14:11:54.858815+0000","last_became_active":"2023-10-16T14:11:52.990013+0000","last_became_peered":"2023-10-16T14:11:52.990013+0000","last_unstale":"2023-10-16T14:11:54.858815+0000","last_undegraded":"2023-10-16T14:11:54.858815+0000","last_fullsized":"2023-10-16T14:11:54.858815+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:08.734684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960073+0000","last_change":"2023-10-16T14:11:38.801326+0000","last_active":"2023-10-16T14:11:52.960073+0000","last_peered":"2023-10-16T14:11:52.960073+0000","last_clean":"2023-10-16T14:11:52.960073+0000","last_became_active":"2023-10-16T14:11:38.800581+0000","last_became_peered":"2023-10-16T14:11:38.800581+0000","last_unstale":"2023-10-16T14:11:52.960073+0000","last_undegraded":"2023-10-16T14:11:52.960073+0000","last_fullsized":"2023-10-16T14:11:52.960073+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:09.367409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974977+0000","last_change":"2023-10-16T14:11:37.775158+0000","last_active":"2023-10-16T14:11:52.974977+0000","last_peered":"2023-10-16T14:11:52.974977+0000","last_clean":"2023-10-16T14:11:52.974977+0000","last_became_active":"2023-10-16T14:11:37.773264+0000","last_became_peered":"2023-10-16T14:11:37.773264+0000","last_unstale":"2023-10-16T14:11:52.974977+0000","last_undegraded":"2023-10-16T14:11:52.974977+0000","last_fullsized":"2023-10-16T14:11:52.974977+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:27.581413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993902+0000","last_change":"2023-10-16T14:11:52.988830+0000","last_active":"2023-10-16T14:11:52.993902+0000","last_peered":"2023-10-16T14:11:52.993902+0000","last_clean":"2023-10-16T14:11:52.993902+0000","last_became_active":"2023-10-16T14:11:52.988543+0000","last_became_peered":"2023-10-16T14:11:52.988543+0000","last_unstale":"2023-10-16T14:11:52.993902+0000","last_undegraded":"2023-10-16T14:11:52.993902+0000","last_fullsized":"2023-10-16T14:11:52.993902+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:46.167512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982676+0000","last_change":"2023-10-16T14:11:38.802528+0000","last_active":"2023-10-16T14:11:52.982676+0000","last_peered":"2023-10-16T14:11:52.982676+0000","last_clean":"2023-10-16T14:11:52.982676+0000","last_became_active":"2023-10-16T14:11:38.802212+0000","last_became_peered":"2023-10-16T14:11:38.802212+0000","last_unstale":"2023-10-16T14:11:52.982676+0000","last_undegraded":"2023-10-16T14:11:52.982676+0000","last_fullsized":"2023-10-16T14:11:52.982676+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:04.028597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960624+0000","last_change":"2023-10-16T14:11:37.780327+0000","last_active":"2023-10-16T14:11:52.960624+0000","last_peered":"2023-10-16T14:11:52.960624+0000","last_clean":"2023-10-16T14:11:52.960624+0000","last_became_active":"2023-10-16T14:11:37.779846+0000","last_became_peered":"2023-10-16T14:11:37.779846+0000","last_unstale":"2023-10-16T14:11:52.960624+0000","last_undegraded":"2023-10-16T14:11:52.960624+0000","last_fullsized":"2023-10-16T14:11:52.960624+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:15.446021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.986572+0000","last_change":"2023-10-16T14:11:52.987389+0000","last_active":"2023-10-16T14:11:53.986572+0000","last_peered":"2023-10-16T14:11:53.986572+0000","last_clean":"2023-10-16T14:11:53.986572+0000","last_became_active":"2023-10-16T14:11:52.981453+0000","last_became_peered":"2023-10-16T14:11:52.981453+0000","last_unstale":"2023-10-16T14:11:53.986572+0000","last_undegraded":"2023-10-16T14:11:53.986572+0000","last_fullsized":"2023-10-16T14:11:53.986572+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:33.020478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993860+0000","last_change":"2023-10-16T14:11:38.809523+0000","last_active":"2023-10-16T14:11:52.993860+0000","last_peered":"2023-10-16T14:11:52.993860+0000","last_clean":"2023-10-16T14:11:52.993860+0000","last_became_active":"2023-10-16T14:11:38.807663+0000","last_became_peered":"2023-10-16T14:11:38.807663+0000","last_unstale":"2023-10-16T14:11:52.993860+0000","last_undegraded":"2023-10-16T14:11:52.993860+0000","last_fullsized":"2023-10-16T14:11:52.993860+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:55.839639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993751+0000","last_change":"2023-10-16T14:11:37.784204+0000","last_active":"2023-10-16T14:11:52.993751+0000","last_peered":"2023-10-16T14:11:52.993751+0000","last_clean":"2023-10-16T14:11:52.993751+0000","last_became_active":"2023-10-16T14:11:37.783880+0000","last_became_peered":"2023-10-16T14:11:37.783880+0000","last_unstale":"2023-10-16T14:11:52.993751+0000","last_undegraded":"2023-10-16T14:11:52.993751+0000","last_fullsized":"2023-10-16T14:11:52.993751+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:22.672658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993206+0000","last_change":"2023-10-16T14:11:52.985795+0000","last_active":"2023-10-16T14:11:53.993206+0000","last_peered":"2023-10-16T14:11:53.993206+0000","last_clean":"2023-10-16T14:11:53.993206+0000","last_became_active":"2023-10-16T14:11:52.985504+0000","last_became_peered":"2023-10-16T14:11:52.985504+0000","last_unstale":"2023-10-16T14:11:53.993206+0000","last_undegraded":"2023-10-16T14:11:53.993206+0000","last_fullsized":"2023-10-16T14:11:53.993206+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:46.426090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976954+0000","last_change":"2023-10-16T14:11:38.802292+0000","last_active":"2023-10-16T14:11:52.976954+0000","last_peered":"2023-10-16T14:11:52.976954+0000","last_clean":"2023-10-16T14:11:52.976954+0000","last_became_active":"2023-10-16T14:11:38.801904+0000","last_became_peered":"2023-10-16T14:11:38.801904+0000","last_unstale":"2023-10-16T14:11:52.976954+0000","last_undegraded":"2023-10-16T14:11:52.976954+0000","last_fullsized":"2023-10-16T14:11:52.976954+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:44:48.663239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959886+0000","last_change":"2023-10-16T14:11:37.774897+0000","last_active":"2023-10-16T14:11:52.959886+0000","last_peered":"2023-10-16T14:11:52.959886+0000","last_clean":"2023-10-16T14:11:52.959886+0000","last_became_active":"2023-10-16T14:11:37.774465+0000","last_became_peered":"2023-10-16T14:11:37.774465+0000","last_unstale":"2023-10-16T14:11:52.959886+0000","last_undegraded":"2023-10-16T14:11:52.959886+0000","last_fullsized":"2023-10-16T14:11:52.959886+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718522+0000","last_change":"2023-10-16T14:11:52.998125+0000","last_active":"2023-10-16T14:11:54.718522+0000","last_peered":"2023-10-16T14:11:54.718522+0000","last_clean":"2023-10-16T14:11:54.718522+0000","last_became_active":"2023-10-16T14:11:52.997853+0000","last_became_peered":"2023-10-16T14:11:52.997853+0000","last_unstale":"2023-10-16T14:11:54.718522+0000","last_undegraded":"2023-10-16T14:11:54.718522+0000","last_fullsized":"2023-10-16T14:11:54.718522+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:30.650999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961151+0000","last_change":"2023-10-16T14:11:38.809093+0000","last_active":"2023-10-16T14:11:52.961151+0000","last_peered":"2023-10-16T14:11:52.961151+0000","last_clean":"2023-10-16T14:11:52.961151+0000","last_became_active":"2023-10-16T14:11:38.808829+0000","last_became_peered":"2023-10-16T14:11:38.808829+0000","last_unstale":"2023-10-16T14:11:52.961151+0000","last_undegraded":"2023-10-16T14:11:52.961151+0000","last_fullsized":"2023-10-16T14:11:52.961151+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:09.522737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"251'3","reported_seq":19,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992384+0000","last_change":"2023-10-16T14:11:37.772650+0000","last_active":"2023-10-16T14:11:52.992384+0000","last_peered":"2023-10-16T14:11:52.992384+0000","last_clean":"2023-10-16T14:11:52.992384+0000","last_became_active":"2023-10-16T14:11:37.772209+0000","last_became_peered":"2023-10-16T14:11:37.772209+0000","last_unstale":"2023-10-16T14:11:52.992384+0000","last_undegraded":"2023-10-16T14:11:52.992384+0000","last_fullsized":"2023-10-16T14:11:52.992384+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:51.981249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.718522+0000","last_change":"2023-10-16T14:11:52.983324+0000","last_active":"2023-10-16T14:11:54.718522+0000","last_peered":"2023-10-16T14:11:54.718522+0000","last_clean":"2023-10-16T14:11:54.718522+0000","last_became_active":"2023-10-16T14:11:52.982503+0000","last_became_peered":"2023-10-16T14:11:52.982503+0000","last_unstale":"2023-10-16T14:11:54.718522+0000","last_undegraded":"2023-10-16T14:11:54.718522+0000","last_fullsized":"2023-10-16T14:11:54.718522+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:37.888928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976458+0000","last_change":"2023-10-16T14:11:38.800005+0000","last_active":"2023-10-16T14:11:52.976458+0000","last_peered":"2023-10-16T14:11:52.976458+0000","last_clean":"2023-10-16T14:11:52.976458+0000","last_became_active":"2023-10-16T14:11:38.799654+0000","last_became_peered":"2023-10-16T14:11:38.799654+0000","last_unstale":"2023-10-16T14:11:52.976458+0000","last_undegraded":"2023-10-16T14:11:52.976458+0000","last_fullsized":"2023-10-16T14:11:52.976458+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:59.978269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"254'8","reported_seq":24,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991307+0000","last_change":"2023-10-16T14:11:37.783855+0000","last_active":"2023-10-16T14:11:52.991307+0000","last_peered":"2023-10-16T14:11:52.991307+0000","last_clean":"2023-10-16T14:11:52.991307+0000","last_became_active":"2023-10-16T14:11:37.783474+0000","last_became_peered":"2023-10-16T14:11:37.783474+0000","last_unstale":"2023-10-16T14:11:52.991307+0000","last_undegraded":"2023-10-16T14:11:52.991307+0000","last_fullsized":"2023-10-16T14:11:52.991307+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.327444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.645849+0000","last_change":"2023-10-16T14:11:52.994061+0000","last_active":"2023-10-16T14:11:54.645849+0000","last_peered":"2023-10-16T14:11:54.645849+0000","last_clean":"2023-10-16T14:11:54.645849+0000","last_became_active":"2023-10-16T14:11:52.993800+0000","last_became_peered":"2023-10-16T14:11:52.993800+0000","last_unstale":"2023-10-16T14:11:54.645849+0000","last_undegraded":"2023-10-16T14:11:54.645849+0000","last_fullsized":"2023-10-16T14:11:54.645849+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:11.479004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974739+0000","last_change":"2023-10-16T14:11:38.796891+0000","last_active":"2023-10-16T14:11:52.974739+0000","last_peered":"2023-10-16T14:11:52.974739+0000","last_clean":"2023-10-16T14:11:52.974739+0000","last_became_active":"2023-10-16T14:11:38.796574+0000","last_became_peered":"2023-10-16T14:11:38.796574+0000","last_unstale":"2023-10-16T14:11:52.974739+0000","last_undegraded":"2023-10-16T14:11:52.974739+0000","last_fullsized":"2023-10-16T14:11:52.974739+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:06.157168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992688+0000","last_change":"2023-10-16T14:11:37.767366+0000","last_active":"2023-10-16T14:11:52.992688+0000","last_peered":"2023-10-16T14:11:52.992688+0000","last_clean":"2023-10-16T14:11:52.992688+0000","last_became_active":"2023-10-16T14:11:37.766979+0000","last_became_peered":"2023-10-16T14:11:37.766979+0000","last_unstale":"2023-10-16T14:11:52.992688+0000","last_undegraded":"2023-10-16T14:11:52.992688+0000","last_fullsized":"2023-10-16T14:11:52.992688+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:25.104652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994154+0000","last_change":"2023-10-16T14:11:52.987377+0000","last_active":"2023-10-16T14:11:53.994154+0000","last_peered":"2023-10-16T14:11:53.994154+0000","last_clean":"2023-10-16T14:11:53.994154+0000","last_became_active":"2023-10-16T14:11:52.987074+0000","last_became_peered":"2023-10-16T14:11:52.987074+0000","last_unstale":"2023-10-16T14:11:53.994154+0000","last_undegraded":"2023-10-16T14:11:53.994154+0000","last_fullsized":"2023-10-16T14:11:53.994154+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:39.310420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975506+0000","last_change":"2023-10-16T14:11:38.801883+0000","last_active":"2023-10-16T14:11:52.975506+0000","last_peered":"2023-10-16T14:11:52.975506+0000","last_clean":"2023-10-16T14:11:52.975506+0000","last_became_active":"2023-10-16T14:11:38.800123+0000","last_became_peered":"2023-10-16T14:11:38.800123+0000","last_unstale":"2023-10-16T14:11:52.975506+0000","last_undegraded":"2023-10-16T14:11:52.975506+0000","last_fullsized":"2023-10-16T14:11:52.975506+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:22.442052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961094+0000","last_change":"2023-10-16T14:11:37.779309+0000","last_active":"2023-10-16T14:11:52.961094+0000","last_peered":"2023-10-16T14:11:52.961094+0000","last_clean":"2023-10-16T14:11:52.961094+0000","last_became_active":"2023-10-16T14:11:37.778681+0000","last_became_peered":"2023-10-16T14:11:37.778681+0000","last_unstale":"2023-10-16T14:11:52.961094+0000","last_undegraded":"2023-10-16T14:11:52.961094+0000","last_fullsized":"2023-10-16T14:11:52.961094+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:41.372800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995674+0000","last_change":"2023-10-16T14:11:52.991729+0000","last_active":"2023-10-16T14:11:53.995674+0000","last_peered":"2023-10-16T14:11:53.995674+0000","last_clean":"2023-10-16T14:11:53.995674+0000","last_became_active":"2023-10-16T14:11:52.991418+0000","last_became_peered":"2023-10-16T14:11:52.991418+0000","last_unstale":"2023-10-16T14:11:53.995674+0000","last_undegraded":"2023-10-16T14:11:53.995674+0000","last_fullsized":"2023-10-16T14:11:53.995674+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:01.672958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.979384+0000","last_change":"2023-10-16T14:11:38.799773+0000","last_active":"2023-10-16T14:11:51.979384+0000","last_peered":"2023-10-16T14:11:51.979384+0000","last_clean":"2023-10-16T14:11:51.979384+0000","last_became_active":"2023-10-16T14:11:38.799446+0000","last_became_peered":"2023-10-16T14:11:38.799446+0000","last_unstale":"2023-10-16T14:11:51.979384+0000","last_undegraded":"2023-10-16T14:11:51.979384+0000","last_fullsized":"2023-10-16T14:11:51.979384+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:16.652122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975194+0000","last_change":"2023-10-16T14:11:37.777163+0000","last_active":"2023-10-16T14:11:52.975194+0000","last_peered":"2023-10-16T14:11:52.975194+0000","last_clean":"2023-10-16T14:11:52.975194+0000","last_became_active":"2023-10-16T14:11:37.776875+0000","last_became_peered":"2023-10-16T14:11:37.776875+0000","last_unstale":"2023-10-16T14:11:52.975194+0000","last_undegraded":"2023-10-16T14:11:52.975194+0000","last_fullsized":"2023-10-16T14:11:52.975194+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:59.869346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.988598+0000","last_change":"2023-10-16T14:11:52.989767+0000","last_active":"2023-10-16T14:11:53.988598+0000","last_peered":"2023-10-16T14:11:53.988598+0000","last_clean":"2023-10-16T14:11:53.988598+0000","last_became_active":"2023-10-16T14:11:52.989398+0000","last_became_peered":"2023-10-16T14:11:52.989398+0000","last_unstale":"2023-10-16T14:11:53.988598+0000","last_undegraded":"2023-10-16T14:11:53.988598+0000","last_fullsized":"2023-10-16T14:11:53.988598+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:09.136777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974747+0000","last_change":"2023-10-16T14:11:38.801468+0000","last_active":"2023-10-16T14:11:52.974747+0000","last_peered":"2023-10-16T14:11:52.974747+0000","last_clean":"2023-10-16T14:11:52.974747+0000","last_became_active":"2023-10-16T14:11:38.801182+0000","last_became_peered":"2023-10-16T14:11:38.801182+0000","last_unstale":"2023-10-16T14:11:52.974747+0000","last_undegraded":"2023-10-16T14:11:52.974747+0000","last_fullsized":"2023-10-16T14:11:52.974747+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:47.695208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"251'8","reported_seq":24,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.967657+0000","last_change":"2023-10-16T14:11:37.784533+0000","last_active":"2023-10-16T14:11:52.967657+0000","last_peered":"2023-10-16T14:11:52.967657+0000","last_clean":"2023-10-16T14:11:52.967657+0000","last_became_active":"2023-10-16T14:11:37.784209+0000","last_became_peered":"2023-10-16T14:11:37.784209+0000","last_unstale":"2023-10-16T14:11:52.967657+0000","last_undegraded":"2023-10-16T14:11:52.967657+0000","last_fullsized":"2023-10-16T14:11:52.967657+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:49.027722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.644770+0000","last_change":"2023-10-16T14:11:52.990437+0000","last_active":"2023-10-16T14:11:54.644770+0000","last_peered":"2023-10-16T14:11:54.644770+0000","last_clean":"2023-10-16T14:11:54.644770+0000","last_became_active":"2023-10-16T14:11:52.989978+0000","last_became_peered":"2023-10-16T14:11:52.989978+0000","last_unstale":"2023-10-16T14:11:54.644770+0000","last_undegraded":"2023-10-16T14:11:54.644770+0000","last_fullsized":"2023-10-16T14:11:54.644770+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:27.613956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.986198+0000","last_change":"2023-10-16T14:11:38.810290+0000","last_active":"2023-10-16T14:11:51.986198+0000","last_peered":"2023-10-16T14:11:51.986198+0000","last_clean":"2023-10-16T14:11:51.986198+0000","last_became_active":"2023-10-16T14:11:38.809830+0000","last_became_peered":"2023-10-16T14:11:38.809830+0000","last_unstale":"2023-10-16T14:11:51.986198+0000","last_undegraded":"2023-10-16T14:11:51.986198+0000","last_fullsized":"2023-10-16T14:11:51.986198+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:29:31.792439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.978352+0000","last_change":"2023-10-16T14:11:37.786947+0000","last_active":"2023-10-16T14:11:52.978352+0000","last_peered":"2023-10-16T14:11:52.978352+0000","last_clean":"2023-10-16T14:11:52.978352+0000","last_became_active":"2023-10-16T14:11:37.786340+0000","last_became_peered":"2023-10-16T14:11:37.786340+0000","last_unstale":"2023-10-16T14:11:52.978352+0000","last_undegraded":"2023-10-16T14:11:52.978352+0000","last_fullsized":"2023-10-16T14:11:52.978352+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:01.576946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.003171+0000","last_change":"2023-10-16T14:11:53.000467+0000","last_active":"2023-10-16T14:11:53.003171+0000","last_peered":"2023-10-16T14:11:53.003171+0000","last_clean":"2023-10-16T14:11:53.003171+0000","last_became_active":"2023-10-16T14:11:52.998042+0000","last_became_peered":"2023-10-16T14:11:52.998042+0000","last_unstale":"2023-10-16T14:11:53.003171+0000","last_undegraded":"2023-10-16T14:11:53.003171+0000","last_fullsized":"2023-10-16T14:11:53.003171+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:55.324917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.966150+0000","last_change":"2023-10-16T14:11:38.804059+0000","last_active":"2023-10-16T14:11:52.966150+0000","last_peered":"2023-10-16T14:11:52.966150+0000","last_clean":"2023-10-16T14:11:52.966150+0000","last_became_active":"2023-10-16T14:11:38.803768+0000","last_became_peered":"2023-10-16T14:11:38.803768+0000","last_unstale":"2023-10-16T14:11:52.966150+0000","last_undegraded":"2023-10-16T14:11:52.966150+0000","last_fullsized":"2023-10-16T14:11:52.966150+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:40.313169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960155+0000","last_change":"2023-10-16T14:11:37.780012+0000","last_active":"2023-10-16T14:11:52.960155+0000","last_peered":"2023-10-16T14:11:52.960155+0000","last_clean":"2023-10-16T14:11:52.960155+0000","last_became_active":"2023-10-16T14:11:37.779730+0000","last_became_peered":"2023-10-16T14:11:37.779730+0000","last_unstale":"2023-10-16T14:11:52.960155+0000","last_undegraded":"2023-10-16T14:11:52.960155+0000","last_fullsized":"2023-10-16T14:11:52.960155+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.010982+0000","last_change":"2023-10-16T14:11:53.007299+0000","last_active":"2023-10-16T14:11:53.010982+0000","last_peered":"2023-10-16T14:11:53.010982+0000","last_clean":"2023-10-16T14:11:53.010982+0000","last_became_active":"2023-10-16T14:11:53.006618+0000","last_became_peered":"2023-10-16T14:11:53.006618+0000","last_unstale":"2023-10-16T14:11:53.010982+0000","last_undegraded":"2023-10-16T14:11:53.010982+0000","last_fullsized":"2023-10-16T14:11:53.010982+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:19:22.430811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973640+0000","last_change":"2023-10-16T14:11:38.804003+0000","last_active":"2023-10-16T14:11:52.973640+0000","last_peered":"2023-10-16T14:11:52.973640+0000","last_clean":"2023-10-16T14:11:52.973640+0000","last_became_active":"2023-10-16T14:11:38.803711+0000","last_became_peered":"2023-10-16T14:11:38.803711+0000","last_unstale":"2023-10-16T14:11:52.973640+0000","last_undegraded":"2023-10-16T14:11:52.973640+0000","last_fullsized":"2023-10-16T14:11:52.973640+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:57.279906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"251'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976637+0000","last_change":"2023-10-16T14:11:37.784775+0000","last_active":"2023-10-16T14:11:52.976637+0000","last_peered":"2023-10-16T14:11:52.976637+0000","last_clean":"2023-10-16T14:11:52.976637+0000","last_became_active":"2023-10-16T14:11:37.784340+0000","last_became_peered":"2023-10-16T14:11:37.784340+0000","last_unstale":"2023-10-16T14:11:52.976637+0000","last_undegraded":"2023-10-16T14:11:52.976637+0000","last_fullsized":"2023-10-16T14:11:52.976637+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:15.393870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":9,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.993391+0000","last_change":"2023-10-16T14:11:52.990908+0000","last_active":"2023-10-16T14:11:52.993391+0000","last_peered":"2023-10-16T14:11:52.993391+0000","last_clean":"2023-10-16T14:11:52.993391+0000","last_became_active":"2023-10-16T14:11:52.990599+0000","last_became_peered":"2023-10-16T14:11:52.990599+0000","last_unstale":"2023-10-16T14:11:52.993391+0000","last_undegraded":"2023-10-16T14:11:52.993391+0000","last_fullsized":"2023-10-16T14:11:52.993391+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:15.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992965+0000","last_change":"2023-10-16T14:11:38.797986+0000","last_active":"2023-10-16T14:11:52.992965+0000","last_peered":"2023-10-16T14:11:52.992965+0000","last_clean":"2023-10-16T14:11:52.992965+0000","last_became_active":"2023-10-16T14:11:38.797632+0000","last_became_peered":"2023-10-16T14:11:38.797632+0000","last_unstale":"2023-10-16T14:11:52.992965+0000","last_undegraded":"2023-10-16T14:11:52.992965+0000","last_fullsized":"2023-10-16T14:11:52.992965+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:33.536538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973302+0000","last_change":"2023-10-16T14:11:37.772519+0000","last_active":"2023-10-16T14:11:52.973302+0000","last_peered":"2023-10-16T14:11:52.973302+0000","last_clean":"2023-10-16T14:11:52.973302+0000","last_became_active":"2023-10-16T14:11:37.772241+0000","last_became_peered":"2023-10-16T14:11:37.772241+0000","last_unstale":"2023-10-16T14:11:52.973302+0000","last_undegraded":"2023-10-16T14:11:52.973302+0000","last_fullsized":"2023-10-16T14:11:52.973302+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:47.596657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069291+0000","last_change":"2023-10-16T14:11:52.996241+0000","last_active":"2023-10-16T14:11:55.069291+0000","last_peered":"2023-10-16T14:11:55.069291+0000","last_clean":"2023-10-16T14:11:55.069291+0000","last_became_active":"2023-10-16T14:11:52.995937+0000","last_became_peered":"2023-10-16T14:11:52.995937+0000","last_unstale":"2023-10-16T14:11:55.069291+0000","last_undegraded":"2023-10-16T14:11:55.069291+0000","last_fullsized":"2023-10-16T14:11:55.069291+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:23.097797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.990766+0000","last_change":"2023-10-16T14:11:38.801767+0000","last_active":"2023-10-16T14:11:52.990766+0000","last_peered":"2023-10-16T14:11:52.990766+0000","last_clean":"2023-10-16T14:11:52.990766+0000","last_became_active":"2023-10-16T14:11:38.800797+0000","last_became_peered":"2023-10-16T14:11:38.800797+0000","last_unstale":"2023-10-16T14:11:52.990766+0000","last_undegraded":"2023-10-16T14:11:52.990766+0000","last_fullsized":"2023-10-16T14:11:52.990766+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:05.512554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"251'2","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.961589+0000","last_change":"2023-10-16T14:11:37.781065+0000","last_active":"2023-10-16T14:11:52.961589+0000","last_peered":"2023-10-16T14:11:52.961589+0000","last_clean":"2023-10-16T14:11:52.961589+0000","last_became_active":"2023-10-16T14:11:37.780556+0000","last_became_peered":"2023-10-16T14:11:37.780556+0000","last_unstale":"2023-10-16T14:11:52.961589+0000","last_undegraded":"2023-10-16T14:11:52.961589+0000","last_fullsized":"2023-10-16T14:11:52.961589+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:03.865559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993948+0000","last_change":"2023-10-16T14:11:53.001652+0000","last_active":"2023-10-16T14:11:53.993948+0000","last_peered":"2023-10-16T14:11:53.993948+0000","last_clean":"2023-10-16T14:11:53.993948+0000","last_became_active":"2023-10-16T14:11:53.001363+0000","last_became_peered":"2023-10-16T14:11:53.001363+0000","last_unstale":"2023-10-16T14:11:53.993948+0000","last_undegraded":"2023-10-16T14:11:53.993948+0000","last_fullsized":"2023-10-16T14:11:53.993948+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:25.562353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992641+0000","last_change":"2023-10-16T14:11:38.803116+0000","last_active":"2023-10-16T14:11:52.992641+0000","last_peered":"2023-10-16T14:11:52.992641+0000","last_clean":"2023-10-16T14:11:52.992641+0000","last_became_active":"2023-10-16T14:11:38.802518+0000","last_became_peered":"2023-10-16T14:11:38.802518+0000","last_unstale":"2023-10-16T14:11:52.992641+0000","last_undegraded":"2023-10-16T14:11:52.992641+0000","last_fullsized":"2023-10-16T14:11:52.992641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:27.765461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973148+0000","last_change":"2023-10-16T14:11:37.768392+0000","last_active":"2023-10-16T14:11:52.973148+0000","last_peered":"2023-10-16T14:11:52.973148+0000","last_clean":"2023-10-16T14:11:52.973148+0000","last_became_active":"2023-10-16T14:11:37.768081+0000","last_became_peered":"2023-10-16T14:11:37.768081+0000","last_unstale":"2023-10-16T14:11:52.973148+0000","last_undegraded":"2023-10-16T14:11:52.973148+0000","last_fullsized":"2023-10-16T14:11:52.973148+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:13.231020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.994164+0000","last_change":"2023-10-16T14:11:52.985092+0000","last_active":"2023-10-16T14:11:53.994164+0000","last_peered":"2023-10-16T14:11:53.994164+0000","last_clean":"2023-10-16T14:11:53.994164+0000","last_became_active":"2023-10-16T14:11:52.984611+0000","last_became_peered":"2023-10-16T14:11:52.984611+0000","last_unstale":"2023-10-16T14:11:53.994164+0000","last_undegraded":"2023-10-16T14:11:53.994164+0000","last_fullsized":"2023-10-16T14:11:53.994164+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:47.800928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.967386+0000","last_change":"2023-10-16T14:11:38.796924+0000","last_active":"2023-10-16T14:11:52.967386+0000","last_peered":"2023-10-16T14:11:52.967386+0000","last_clean":"2023-10-16T14:11:52.967386+0000","last_became_active":"2023-10-16T14:11:38.796610+0000","last_became_peered":"2023-10-16T14:11:38.796610+0000","last_unstale":"2023-10-16T14:11:52.967386+0000","last_undegraded":"2023-10-16T14:11:52.967386+0000","last_fullsized":"2023-10-16T14:11:52.967386+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:29.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.983122+0000","last_change":"2023-10-16T14:11:37.780387+0000","last_active":"2023-10-16T14:11:52.983122+0000","last_peered":"2023-10-16T14:11:52.983122+0000","last_clean":"2023-10-16T14:11:52.983122+0000","last_became_active":"2023-10-16T14:11:37.779362+0000","last_became_peered":"2023-10-16T14:11:37.779362+0000","last_unstale":"2023-10-16T14:11:52.983122+0000","last_undegraded":"2023-10-16T14:11:52.983122+0000","last_fullsized":"2023-10-16T14:11:52.983122+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.719942+0000","last_change":"2023-10-16T14:11:53.000533+0000","last_active":"2023-10-16T14:11:54.719942+0000","last_peered":"2023-10-16T14:11:54.719942+0000","last_clean":"2023-10-16T14:11:54.719942+0000","last_became_active":"2023-10-16T14:11:53.000269+0000","last_became_peered":"2023-10-16T14:11:53.000269+0000","last_unstale":"2023-10-16T14:11:54.719942+0000","last_undegraded":"2023-10-16T14:11:54.719942+0000","last_fullsized":"2023-10-16T14:11:54.719942+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:13.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.992076+0000","last_change":"2023-10-16T14:11:38.799901+0000","last_active":"2023-10-16T14:11:52.992076+0000","last_peered":"2023-10-16T14:11:52.992076+0000","last_clean":"2023-10-16T14:11:52.992076+0000","last_became_active":"2023-10-16T14:11:38.799145+0000","last_became_peered":"2023-10-16T14:11:38.799145+0000","last_unstale":"2023-10-16T14:11:52.992076+0000","last_undegraded":"2023-10-16T14:11:52.992076+0000","last_fullsized":"2023-10-16T14:11:52.992076+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.534242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973875+0000","last_change":"2023-10-16T14:11:37.779700+0000","last_active":"2023-10-16T14:11:52.973875+0000","last_peered":"2023-10-16T14:11:52.973875+0000","last_clean":"2023-10-16T14:11:52.973875+0000","last_became_active":"2023-10-16T14:11:37.779082+0000","last_became_peered":"2023-10-16T14:11:37.779082+0000","last_unstale":"2023-10-16T14:11:52.973875+0000","last_undegraded":"2023-10-16T14:11:52.973875+0000","last_fullsized":"2023-10-16T14:11:52.973875+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:12.639268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.067757+0000","last_change":"2023-10-16T14:11:52.998190+0000","last_active":"2023-10-16T14:11:55.067757+0000","last_peered":"2023-10-16T14:11:55.067757+0000","last_clean":"2023-10-16T14:11:55.067757+0000","last_became_active":"2023-10-16T14:11:52.995399+0000","last_became_peered":"2023-10-16T14:11:52.995399+0000","last_unstale":"2023-10-16T14:11:55.067757+0000","last_undegraded":"2023-10-16T14:11:55.067757+0000","last_fullsized":"2023-10-16T14:11:55.067757+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:16.114233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.979252+0000","last_change":"2023-10-16T14:11:38.800281+0000","last_active":"2023-10-16T14:11:52.979252+0000","last_peered":"2023-10-16T14:11:52.979252+0000","last_clean":"2023-10-16T14:11:52.979252+0000","last_became_active":"2023-10-16T14:11:38.799813+0000","last_became_peered":"2023-10-16T14:11:38.799813+0000","last_unstale":"2023-10-16T14:11:52.979252+0000","last_undegraded":"2023-10-16T14:11:52.979252+0000","last_fullsized":"2023-10-16T14:11:52.979252+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:23:41.635286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"251'2","reported_seq":17,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.988830+0000","last_change":"2023-10-16T14:11:37.776293+0000","last_active":"2023-10-16T14:11:51.988830+0000","last_peered":"2023-10-16T14:11:51.988830+0000","last_clean":"2023-10-16T14:11:51.988830+0000","last_became_active":"2023-10-16T14:11:37.775902+0000","last_became_peered":"2023-10-16T14:11:37.775902+0000","last_unstale":"2023-10-16T14:11:51.988830+0000","last_undegraded":"2023-10-16T14:11:51.988830+0000","last_fullsized":"2023-10-16T14:11:51.988830+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:09.428253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993020+0000","last_change":"2023-10-16T14:11:52.990919+0000","last_active":"2023-10-16T14:11:53.993020+0000","last_peered":"2023-10-16T14:11:53.993020+0000","last_clean":"2023-10-16T14:11:53.993020+0000","last_became_active":"2023-10-16T14:11:52.990136+0000","last_became_peered":"2023-10-16T14:11:52.990136+0000","last_unstale":"2023-10-16T14:11:53.993020+0000","last_undegraded":"2023-10-16T14:11:53.993020+0000","last_fullsized":"2023-10-16T14:11:53.993020+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:46.114623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.991821+0000","last_change":"2023-10-16T14:11:38.806901+0000","last_active":"2023-10-16T14:11:52.991821+0000","last_peered":"2023-10-16T14:11:52.991821+0000","last_clean":"2023-10-16T14:11:52.991821+0000","last_became_active":"2023-10-16T14:11:38.806614+0000","last_became_peered":"2023-10-16T14:11:38.806614+0000","last_unstale":"2023-10-16T14:11:52.991821+0000","last_undegraded":"2023-10-16T14:11:52.991821+0000","last_fullsized":"2023-10-16T14:11:52.991821+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:36.084105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"252'1","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.977834+0000","last_change":"2023-10-16T14:11:37.780349+0000","last_active":"2023-10-16T14:11:52.977834+0000","last_peered":"2023-10-16T14:11:52.977834+0000","last_clean":"2023-10-16T14:11:52.977834+0000","last_became_active":"2023-10-16T14:11:37.777254+0000","last_became_peered":"2023-10-16T14:11:37.777254+0000","last_unstale":"2023-10-16T14:11:52.977834+0000","last_undegraded":"2023-10-16T14:11:52.977834+0000","last_fullsized":"2023-10-16T14:11:52.977834+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:32:22.053280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.644999+0000","last_change":"2023-10-16T14:11:52.991009+0000","last_active":"2023-10-16T14:11:54.644999+0000","last_peered":"2023-10-16T14:11:54.644999+0000","last_clean":"2023-10-16T14:11:54.644999+0000","last_became_active":"2023-10-16T14:11:52.990420+0000","last_became_peered":"2023-10-16T14:11:52.990420+0000","last_unstale":"2023-10-16T14:11:54.644999+0000","last_undegraded":"2023-10-16T14:11:54.644999+0000","last_fullsized":"2023-10-16T14:11:54.644999+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:12.562346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973061+0000","last_change":"2023-10-16T14:11:38.805448+0000","last_active":"2023-10-16T14:11:52.973061+0000","last_peered":"2023-10-16T14:11:52.973061+0000","last_clean":"2023-10-16T14:11:52.973061+0000","last_became_active":"2023-10-16T14:11:38.805128+0000","last_became_peered":"2023-10-16T14:11:38.805128+0000","last_unstale":"2023-10-16T14:11:52.973061+0000","last_undegraded":"2023-10-16T14:11:52.973061+0000","last_fullsized":"2023-10-16T14:11:52.973061+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:10.898166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976766+0000","last_change":"2023-10-16T14:11:37.782304+0000","last_active":"2023-10-16T14:11:52.976766+0000","last_peered":"2023-10-16T14:11:52.976766+0000","last_clean":"2023-10-16T14:11:52.976766+0000","last_became_active":"2023-10-16T14:11:37.781840+0000","last_became_peered":"2023-10-16T14:11:37.781840+0000","last_unstale":"2023-10-16T14:11:52.976766+0000","last_undegraded":"2023-10-16T14:11:52.976766+0000","last_fullsized":"2023-10-16T14:11:52.976766+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:05.521128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858200+0000","last_change":"2023-10-16T14:11:52.981362+0000","last_active":"2023-10-16T14:11:54.858200+0000","last_peered":"2023-10-16T14:11:54.858200+0000","last_clean":"2023-10-16T14:11:54.858200+0000","last_became_active":"2023-10-16T14:11:52.981029+0000","last_became_peered":"2023-10-16T14:11:52.981029+0000","last_unstale":"2023-10-16T14:11:54.858200+0000","last_undegraded":"2023-10-16T14:11:54.858200+0000","last_fullsized":"2023-10-16T14:11:54.858200+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:01.090353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.975057+0000","last_change":"2023-10-16T14:11:38.801623+0000","last_active":"2023-10-16T14:11:52.975057+0000","last_peered":"2023-10-16T14:11:52.975057+0000","last_clean":"2023-10-16T14:11:52.975057+0000","last_became_active":"2023-10-16T14:11:38.801313+0000","last_became_peered":"2023-10-16T14:11:38.801313+0000","last_unstale":"2023-10-16T14:11:52.975057+0000","last_undegraded":"2023-10-16T14:11:52.975057+0000","last_fullsized":"2023-10-16T14:11:52.975057+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:56.175171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.973534+0000","last_change":"2023-10-16T14:11:37.775930+0000","last_active":"2023-10-16T14:11:52.973534+0000","last_peered":"2023-10-16T14:11:52.973534+0000","last_clean":"2023-10-16T14:11:52.973534+0000","last_became_active":"2023-10-16T14:11:37.775617+0000","last_became_peered":"2023-10-16T14:11:37.775617+0000","last_unstale":"2023-10-16T14:11:52.973534+0000","last_undegraded":"2023-10-16T14:11:52.973534+0000","last_fullsized":"2023-10-16T14:11:52.973534+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:17:19.733822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858483+0000","last_change":"2023-10-16T14:11:52.982720+0000","last_active":"2023-10-16T14:11:54.858483+0000","last_peered":"2023-10-16T14:11:54.858483+0000","last_clean":"2023-10-16T14:11:54.858483+0000","last_became_active":"2023-10-16T14:11:52.981925+0000","last_became_peered":"2023-10-16T14:11:52.981925+0000","last_unstale":"2023-10-16T14:11:54.858483+0000","last_undegraded":"2023-10-16T14:11:54.858483+0000","last_fullsized":"2023-10-16T14:11:54.858483+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:18.153466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.974888+0000","last_change":"2023-10-16T14:11:38.796749+0000","last_active":"2023-10-16T14:11:52.974888+0000","last_peered":"2023-10-16T14:11:52.974888+0000","last_clean":"2023-10-16T14:11:52.974888+0000","last_became_active":"2023-10-16T14:11:38.796468+0000","last_became_peered":"2023-10-16T14:11:38.796468+0000","last_unstale":"2023-10-16T14:11:52.974888+0000","last_undegraded":"2023-10-16T14:11:52.974888+0000","last_fullsized":"2023-10-16T14:11:52.974888+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:17.187404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.976583+0000","last_change":"2023-10-16T14:11:37.785747+0000","last_active":"2023-10-16T14:11:52.976583+0000","last_peered":"2023-10-16T14:11:52.976583+0000","last_clean":"2023-10-16T14:11:52.976583+0000","last_became_active":"2023-10-16T14:11:37.785464+0000","last_became_peered":"2023-10-16T14:11:37.785464+0000","last_unstale":"2023-10-16T14:11:52.976583+0000","last_undegraded":"2023-10-16T14:11:52.976583+0000","last_fullsized":"2023-10-16T14:11:52.976583+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:32.439735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:54.858945+0000","last_change":"2023-10-16T14:11:52.999461+0000","last_active":"2023-10-16T14:11:54.858945+0000","last_peered":"2023-10-16T14:11:54.858945+0000","last_clean":"2023-10-16T14:11:54.858945+0000","last_became_active":"2023-10-16T14:11:52.999146+0000","last_became_peered":"2023-10-16T14:11:52.999146+0000","last_unstale":"2023-10-16T14:11:54.858945+0000","last_undegraded":"2023-10-16T14:11:54.858945+0000","last_fullsized":"2023-10-16T14:11:54.858945+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:47.618540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.989135+0000","last_change":"2023-10-16T14:11:38.800958+0000","last_active":"2023-10-16T14:11:52.989135+0000","last_peered":"2023-10-16T14:11:52.989135+0000","last_clean":"2023-10-16T14:11:52.989135+0000","last_became_active":"2023-10-16T14:11:38.800677+0000","last_became_peered":"2023-10-16T14:11:38.800677+0000","last_unstale":"2023-10-16T14:11:52.989135+0000","last_undegraded":"2023-10-16T14:11:52.989135+0000","last_fullsized":"2023-10-16T14:11:52.989135+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:18.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-16T14:11:51.983782+0000","last_change":"2023-10-16T14:11:37.785497+0000","last_active":"2023-10-16T14:11:51.983782+0000","last_peered":"2023-10-16T14:11:51.983782+0000","last_clean":"2023-10-16T14:11:51.983782+0000","last_became_active":"2023-10-16T14:11:37.785162+0000","last_became_peered":"2023-10-16T14:11:37.785162+0000","last_unstale":"2023-10-16T14:11:51.983782+0000","last_undegraded":"2023-10-16T14:11:51.983782+0000","last_fullsized":"2023-10-16T14:11:51.983782+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:28.524950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.995213+0000","last_change":"2023-10-16T14:11:52.997961+0000","last_active":"2023-10-16T14:11:53.995213+0000","last_peered":"2023-10-16T14:11:53.995213+0000","last_clean":"2023-10-16T14:11:53.995213+0000","last_became_active":"2023-10-16T14:11:52.997683+0000","last_became_peered":"2023-10-16T14:11:52.997683+0000","last_unstale":"2023-10-16T14:11:53.995213+0000","last_undegraded":"2023-10-16T14:11:53.995213+0000","last_fullsized":"2023-10-16T14:11:53.995213+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:13.592656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.995178+0000","last_change":"2023-10-16T14:11:38.789466+0000","last_active":"2023-10-16T14:11:52.995178+0000","last_peered":"2023-10-16T14:11:52.995178+0000","last_clean":"2023-10-16T14:11:52.995178+0000","last_became_active":"2023-10-16T14:11:38.788914+0000","last_became_peered":"2023-10-16T14:11:38.788914+0000","last_unstale":"2023-10-16T14:11:52.995178+0000","last_undegraded":"2023-10-16T14:11:52.995178+0000","last_fullsized":"2023-10-16T14:11:52.995178+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:36.423758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.959110+0000","last_change":"2023-10-16T14:11:37.784108+0000","last_active":"2023-10-16T14:11:52.959110+0000","last_peered":"2023-10-16T14:11:52.959110+0000","last_clean":"2023-10-16T14:11:52.959110+0000","last_became_active":"2023-10-16T14:11:37.783469+0000","last_became_peered":"2023-10-16T14:11:37.783469+0000","last_unstale":"2023-10-16T14:11:52.959110+0000","last_undegraded":"2023-10-16T14:11:52.959110+0000","last_fullsized":"2023-10-16T14:11:52.959110+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:12.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:55.069566+0000","last_change":"2023-10-16T14:11:53.000623+0000","last_active":"2023-10-16T14:11:55.069566+0000","last_peered":"2023-10-16T14:11:55.069566+0000","last_clean":"2023-10-16T14:11:55.069566+0000","last_became_active":"2023-10-16T14:11:53.000238+0000","last_became_peered":"2023-10-16T14:11:53.000238+0000","last_unstale":"2023-10-16T14:11:55.069566+0000","last_undegraded":"2023-10-16T14:11:55.069566+0000","last_fullsized":"2023-10-16T14:11:55.069566+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:40.320922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.982822+0000","last_change":"2023-10-16T14:11:38.799118+0000","last_active":"2023-10-16T14:11:52.982822+0000","last_peered":"2023-10-16T14:11:52.982822+0000","last_clean":"2023-10-16T14:11:52.982822+0000","last_became_active":"2023-10-16T14:11:38.798708+0000","last_became_peered":"2023-10-16T14:11:38.798708+0000","last_unstale":"2023-10-16T14:11:52.982822+0000","last_undegraded":"2023-10-16T14:11:52.982822+0000","last_fullsized":"2023-10-16T14:11:52.982822+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:36.917040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.960867+0000","last_change":"2023-10-16T14:11:37.777958+0000","last_active":"2023-10-16T14:11:52.960867+0000","last_peered":"2023-10-16T14:11:52.960867+0000","last_clean":"2023-10-16T14:11:52.960867+0000","last_became_active":"2023-10-16T14:11:37.777687+0000","last_became_peered":"2023-10-16T14:11:37.777687+0000","last_unstale":"2023-10-16T14:11:52.960867+0000","last_undegraded":"2023-10-16T14:11:52.960867+0000","last_fullsized":"2023-10-16T14:11:52.960867+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:13.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2023-10-16T14:11:53.993961+0000","last_change":"2023-10-16T14:11:52.993995+0000","last_active":"2023-10-16T14:11:53.993961+0000","last_peered":"2023-10-16T14:11:53.993961+0000","last_clean":"2023-10-16T14:11:53.993961+0000","last_became_active":"2023-10-16T14:11:52.993712+0000","last_became_peered":"2023-10-16T14:11:52.993712+0000","last_unstale":"2023-10-16T14:11:53.993961+0000","last_undegraded":"2023-10-16T14:11:53.993961+0000","last_fullsized":"2023-10-16T14:11:53.993961+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:51.938773+0000","last_clean_scrub_stamp":"2023-10-16T14:11:51.938773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:51:12.663965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.977663+0000","last_change":"2023-10-16T14:11:38.785931+0000","last_active":"2023-10-16T14:11:52.977663+0000","last_peered":"2023-10-16T14:11:52.977663+0000","last_clean":"2023-10-16T14:11:52.977663+0000","last_became_active":"2023-10-16T14:11:38.785345+0000","last_became_peered":"2023-10-16T14:11:38.785345+0000","last_unstale":"2023-10-16T14:11:52.977663+0000","last_undegraded":"2023-10-16T14:11:52.977663+0000","last_fullsized":"2023-10-16T14:11:52.977663+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:37.739674+0000","last_clean_scrub_stamp":"2023-10-16T14:11:37.739674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:16.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"251'4","reported_seq":20,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.968726+0000","last_change":"2023-10-16T14:11:37.781168+0000","last_active":"2023-10-16T14:11:52.968726+0000","last_peered":"2023-10-16T14:11:52.968726+0000","last_clean":"2023-10-16T14:11:52.968726+0000","last_became_active":"2023-10-16T14:11:37.780834+0000","last_became_peered":"2023-10-16T14:11:37.780834+0000","last_unstale":"2023-10-16T14:11:52.968726+0000","last_undegraded":"2023-10-16T14:11:52.968726+0000","last_fullsized":"2023-10-16T14:11:52.968726+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":249,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:11:36.722503+0000","last_clean_scrub_stamp":"2023-10-16T14:11:36.722503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:21.622808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"237'373","reported_seq":1323,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-16T14:11:52.977777+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:11:52.977777+0000","last_peered":"2023-10-16T14:11:52.977777+0000","last_clean":"2023-10-16T14:11:52.977777+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:11:52.977777+0000","last_undegraded":"2023-10-16T14:11:52.977777+0000","last_fullsized":"2023-10-16T14:11:52.977777+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2347,"num_read_kb":4920,"num_write":623,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":373,"ondisk_log_size":373,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543314,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52960,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318880,"statfs":{"total":96636764160,"available":96582533120,"internally_reserved":0,"allocated":3817472,"data_stored":2170995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3460000000000001,"15min":1.3660000000000001},"min":{"1min":0.77900000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.885},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.3480000000000001,"15min":1.3979999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7669999999999999,"5min":1.827,"15min":1.873},"last":6.391}]},{"osd":1,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.2190000000000001,"15min":1.244},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.5880000000000001,"5min":1.8560000000000001,"15min":1.8999999999999999},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.228,"15min":1.2490000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.72,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":3.4769999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.228,"15min":1.258},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.482,"5min":1.7689999999999999,"15min":1.8700000000000001},"last":4.3250000000000002},{"interface":"front","average":{"1min":1.224,"5min":1.347,"15min":1.3560000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.6930000000000001,"5min":2.028,"15min":2.028},"last":3.4289999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.363,"15min":1.403},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.8320000000000001,"5min":1.921,"15min":2.097},"last":2.0790000000000002},{"interface":"front","average":{"1min":1.351,"5min":1.3999999999999999,"15min":1.4259999999999999},"min":{"1min":0.90700000000000003,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":1.7969999999999999,"5min":1.946,"15min":2.0169999999999999},"last":3.3719999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.23,"15min":1.2390000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.895},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4390000000000001,"15min":1.4119999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.786,"5min":1.982,"15min":2.0009999999999999},"last":2.0230000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.391,"15min":1.3799999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.0590000000000002},"last":4.0860000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.536},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.9550000000000001,"5min":2.0720000000000001,"15min":2.2229999999999999},"last":2.5110000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.6259999999999999,"15min":1.633},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.0009999999999999,"5min":2.052,"15min":2.3250000000000002},"last":3.29},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.5249999999999999,"15min":1.48},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.8939999999999999,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":3.524}]}]},{"osd":6,"up_from":14,"seq":60129543315,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64868,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306972,"statfs":{"total":96636764160,"available":96570339328,"internally_reserved":0,"allocated":3231744,"data_stored":1581203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2689999999999999,"15min":1.329},"min":{"1min":0.70299999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.788,"5min":1.887,"15min":1.887},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.3140000000000001,"15min":1.353},"min":{"1min":0.747,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8460000000000001,"5min":1.847,"15min":1.9390000000000001},"last":1.5509999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.206,"15min":1.2250000000000001},"min":{"1min":0.69599999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6279999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.21,"15min":1.2270000000000001},"min":{"1min":0.64800000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6799999999999999,"5min":1.748,"15min":1.77},"last":0.874}]},{"osd":2,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.24,"15min":1.2549999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.78,"5min":1.823,"15min":1.823},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.282,"15min":1.3169999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.8759999999999999,"5min":1.962,"15min":1.962},"last":1.379}]},{"osd":3,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.327,"15min":1.345},"min":{"1min":0.93400000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":1.075,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.776,"5min":1.911,"15min":1.911},"last":0.93500000000000005}]},{"osd":4,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.393,"15min":1.409},"min":{"1min":1.1259999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5609999999999999,"15min":1.5509999999999999},"min":{"1min":0.86299999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"max":{"1min":1.9219999999999999,"5min":2.0259999999999998,"15min":2.1179999999999999},"last":1.427}]},{"osd":5,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.79800000000000004,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.206},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.4990000000000001,"15min":1.5},"min":{"1min":1.2170000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.0990000000000002,"5min":2.1909999999999998,"15min":2.1970000000000001},"last":1.004}]},{"osd":7,"last update":"Mon Oct 16 14:11:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.512,"15min":1.5049999999999999},"min":{"1min":0.999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0150000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.546,"15min":1.542},"min":{"1min":1.0269999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.157,"5min":2.157,"15min":2.6120000000000001},"last":1.4970000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543316,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30148,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341692,"statfs":{"total":96636764160,"available":96605892608,"internally_reserved":0,"allocated":3203072,"data_stored":1573320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.1299999999999999,"15min":1.202},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.4750000000000001,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.2150000000000001,"15min":1.2310000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.7110000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.282,"15min":1.298},"min":{"1min":0.77500000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.774,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.395},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1830000000000001,"15min":1.254},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.698,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.97999999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.304,"15min":1.335},"min":{"1min":0.90300000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.587,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.266,"15min":1.3120000000000001},"min":{"1min":0.80200000000000005,"5min":0.746,"15min":0.746},"max":{"1min":1.806,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3859999999999999,"15min":1.407},"min":{"1min":0.89300000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9019999999999999,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.409,"15min":1.431},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8759999999999999,"5min":2.75,"15min":2.75},"last":1.3660000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3160000000000001,"15min":1.2989999999999999},"min":{"1min":0.77300000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.9299999999999999,"5min":2.774,"15min":2.774},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.454,"15min":1.4319999999999999},"min":{"1min":0.93700000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.4930000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4390000000000001,"15min":1.4319999999999999},"min":{"1min":1.1699999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9390000000000001,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.46},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.508,"15min":1.4910000000000001},"min":{"1min":1.048,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9590000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.236}]},{"osd":7,"last update":"Mon Oct 16 14:11:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.593,"15min":1.554},"min":{"1min":1.2170000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.1680000000000001,"5min":2.177,"15min":2.4609999999999999},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6200000000000001,"15min":1.603},"min":{"1min":1.113,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1000000000000001,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.5549999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543316,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66228,"kb_used_data":3168,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305612,"statfs":{"total":96636764160,"available":96568946688,"internally_reserved":0,"allocated":3244032,"data_stored":1577937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.145,"15min":1.2050000000000001},"min":{"1min":0.61799999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1799999999999999,"15min":1.1990000000000001},"min":{"1min":0.748,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":0.77500000000000002}]},{"osd":1,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1990000000000001,"15min":1.2490000000000001},"min":{"1min":0.80700000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6899999999999999,"5min":1.71,"15min":1.78},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.244,"5min":1.2450000000000001,"15min":1.2869999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.7250000000000001,"5min":1.9359999999999999,"15min":2.0499999999999998},"last":1.2150000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2829999999999999,"15min":1.3160000000000001},"min":{"1min":0.73899999999999999,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.591,"5min":1.714,"15min":1.865},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3500000000000001,"15min":1.3799999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":1.905,"5min":1.9219999999999999,"15min":2.0110000000000001},"last":1.0329999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2889999999999999,"15min":1.302},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.45,"15min":1.4279999999999999},"min":{"1min":1.05,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.6439999999999999,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.4199999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.3520000000000001,"15min":1.339},"min":{"1min":1.0980000000000001,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":1.994,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.387,"15min":1.3420000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.258}]},{"osd":6,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.409,"15min":1.387},"min":{"1min":1.0349999999999999,"5min":0.90400000000000003,"15min":0.872},"max":{"1min":1.931,"5min":1.931,"15min":1.956},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.516,"15min":1.518},"min":{"1min":1.0840000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.0630000000000002,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.681}]},{"osd":7,"last update":"Mon Oct 16 14:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.5660000000000001,"15min":1.5089999999999999},"min":{"1min":1.0149999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1949999999999998},"last":1.7290000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6479999999999999,"15min":1.6339999999999999},"min":{"1min":1.3620000000000001,"5min":1.1719999999999999,"15min":1.008},"max":{"1min":2.085,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.992}]}]},{"osd":3,"up_from":14,"seq":60129543316,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56556,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315284,"statfs":{"total":96636764160,"available":96578850816,"internally_reserved":0,"allocated":3235840,"data_stored":1583263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.90900000000000003,"15min":0.94799999999999995},"min":{"1min":0.48599999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.879},{"interface":"front","average":{"1min":1.107,"5min":1.1479999999999999,"15min":1.2090000000000001},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.03,"15min":1.075},"min":{"1min":0.55900000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.855},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.133,"5min":1.0980000000000001,"15min":1.123},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.0760000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0960000000000001,"15min":1.1890000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.823,"5min":1.8700000000000001,"15min":2.0539999999999998},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3400000000000001,"15min":1.431},"min":{"1min":0.622,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.909,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.91900000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.482,"15min":1.51},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2160000000000002,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.379,"5min":1.4330000000000001,"15min":1.456},"min":{"1min":0.84099999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.823,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.3460000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.427,"15min":1.4850000000000001},"min":{"1min":0.89900000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8380000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.52,"15min":1.5529999999999999},"min":{"1min":0.871,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9910000000000001,"5min":2.0339999999999998,"15min":2.153},"last":0.95299999999999996}]},{"osd":6,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.5,"15min":1.522},"min":{"1min":0.72899999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.335,"5min":2.335,"15min":2.4329999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4750000000000001,"15min":1.496},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.194,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.6220000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5760000000000001,"15min":1.599},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1859999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.698,"5min":1.6599999999999999,"15min":1.677},"min":{"1min":0.98199999999999998,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.5,"5min":2.569,"15min":2.6070000000000002},"last":1.2350000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543314,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56528,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315312,"statfs":{"total":96636764160,"available":96578879488,"internally_reserved":0,"allocated":3207168,"data_stored":1573410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.93100000000000005,"15min":0.95299999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.5029999999999999,"5min":1.597,"15min":2.0089999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.137},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.431,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.105}]},{"osd":1,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.131,"15min":1.171},"min":{"1min":0.56399999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.72,"5min":1.847,"15min":1.847},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.03,"5min":1.2010000000000001,"15min":1.242},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.5860000000000001,"5min":1.768,"15min":1.9530000000000001},"last":1.4650000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.107,"15min":1.115},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.038},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.3009999999999999,"15min":1.3700000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.0619999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.67700000000000005}]},{"osd":4,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4710000000000001,"15min":1.5},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.88},{"interface":"front","average":{"1min":1.349,"5min":1.464,"15min":1.4870000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.3140000000000001},"last":1.8320000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.522,"15min":1.5529999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.867,"5min":2.1819999999999999,"15min":2.488},"last":1.7490000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4550000000000001,"15min":1.4710000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.0059999999999998,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.3460000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.524,"15min":1.52},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.4079999999999999},"last":1.423},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.5640000000000001,"15min":1.617},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.5569999999999999},"last":1.375}]},{"osd":7,"last update":"Mon Oct 16 14:11:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.6319999999999999,"15min":1.655},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.7890000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.619,"15min":1.641},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.2599999999999998,"5min":2.2799999999999998,"15min":2.3319999999999999},"last":1.5149999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543314,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52404,"kb_used_data":3176,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319436,"statfs":{"total":96636764160,"available":96583102464,"internally_reserved":0,"allocated":3252224,"data_stored":1584244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.93000000000000005,"15min":0.93999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.448,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.109,"15min":1.133},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.454}]},{"osd":2,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0129999999999999,"15min":1.0389999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.204,"15min":1.2549999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.98199999999999998}]},{"osd":3,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.4369999999999998},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1930000000000001,"15min":1.2789999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.544,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.571}]},{"osd":4,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.258,"15min":1.339},"min":{"1min":0.65700000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.534,"5min":1.776,"15min":2.1259999999999999},"last":1.125},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.488,"15min":1.5309999999999999},"min":{"1min":0.79100000000000004,"5min":0.748,"15min":0.748},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.401}]},{"osd":5,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.446,"15min":1.488},"min":{"1min":0.72899999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.0819999999999999,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.655},{"interface":"front","average":{"1min":1.419,"5min":1.385,"15min":1.4790000000000001},"min":{"1min":0.746,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.371}]},{"osd":6,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5800000000000001,"15min":1.633},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.7200000000000002},"last":1.714},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5409999999999999,"15min":1.603},"min":{"1min":0.85099999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0979999999999999,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.758}]},{"osd":7,"last update":"Mon Oct 16 14:11:19 2023","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.639,"15min":1.696},"min":{"1min":0.95299999999999996,"5min":0.628,"15min":0.628},"max":{"1min":2.1280000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.8680000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.52,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0630000000000002,"5min":2.2029999999999998,"15min":2.2930000000000001},"last":1.8109999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543315,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48780,"kb_used_data":3712,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323060,"statfs":{"total":96636764160,"available":96586813440,"internally_reserved":0,"allocated":3801088,"data_stored":2163722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.94299999999999995,"15min":0.94399999999999995},"min":{"1min":0.38400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4299999999999999,"5min":1.704,"15min":1.891},"last":1.3440000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.054,"15min":1.145},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3009999999999999,"5min":1.659,"15min":1.9299999999999999},"last":1.5069999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1299999999999999,"15min":1.169},"min":{"1min":0.73299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.171,"15min":1.2210000000000001},"min":{"1min":0.68799999999999994,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.52,"5min":1.893,"15min":1.893},"last":1.0149999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1060000000000001,"15min":1.147},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.623,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.29,"15min":1.3660000000000001},"min":{"1min":0.63200000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.681,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":1.55}]},{"osd":4,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.302,"15min":1.3640000000000001},"min":{"1min":0.84999999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.625,"5min":1.819,"15min":2.0070000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4970000000000001,"15min":1.5189999999999999},"min":{"1min":0.92400000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.919,"5min":2.601,"15min":2.601},"last":1.9330000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4910000000000001,"15min":1.504},"min":{"1min":1.129,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.089,"5min":2.2250000000000001,"15min":2.3279999999999998},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4259999999999999,"15min":1.4790000000000001},"min":{"1min":0.88700000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.401}]},{"osd":6,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.577,"15min":1.613},"min":{"1min":1.1890000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0019999999999998,"5min":2.669,"15min":2.669},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.552,"15min":1.5920000000000001},"min":{"1min":0.94999999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.327,"5min":2.355,"15min":2.3769999999999998},"last":1.637}]},{"osd":7,"last update":"Mon Oct 16 14:11:39 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.6399999999999999,"15min":1.7010000000000001},"min":{"1min":0.88700000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2170000000000001,"5min":2.484,"15min":2.5270000000000001},"last":1.9910000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.591,"15min":1.6359999999999999},"min":{"1min":1.056,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.9490000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.786}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:11:58.091 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:11:58.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:11:58.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.354+0000 7f97673a5640 1 Processor -- start 2023-10-16T14:11:58.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.354+0000 7f97673a5640 1 -- start start 2023-10-16T14:11:58.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f97673a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760159950 0x7f976015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f97673a5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9760070450 0x7f97600708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f97673a5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9760070df0 con 0x7f976006c100 2023-10-16T14:11:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f97673a5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976015e490 con 0x7f9760070450 2023-10-16T14:11:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f97673a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976015aef0 con 0x7f9760159950 2023-10-16T14:11:58.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f9765ba2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9760070450 0x7f97600708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f9766ba4640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f976006c100 0x7f976006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49042/0 (socket says 172.21.15.60:49042) 2023-10-16T14:11:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f9766ba4640 1 -- 172.21.15.60:0/1134882693 learned_addr learned my addr 172.21.15.60:0/1134882693 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:11:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.356+0000 7f9765ba2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9760070450 0x7f97600708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36456/0 (socket says 172.21.15.60:36456) 2023-10-16T14:11:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.357+0000 7f97663a3640 1 --2- 172.21.15.60:0/1134882693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760159950 0x7f976015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.357+0000 7f9765ba2640 1 -- 172.21.15.60:0/1134882693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760159950 msgr2=0x7f976015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.357+0000 7f9765ba2640 1 --2- 172.21.15.60:0/1134882693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760159950 0x7f976015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.357+0000 7f9765ba2640 1 -- 172.21.15.60:0/1134882693 >> v1:172.21.15.60:6789/0 conn(0x7f976006c100 legacy=0x7f976006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.357+0000 7f9765ba2640 1 -- 172.21.15.60:0/1134882693 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976006cc80 con 0x7f9760070450 2023-10-16T14:11:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.358+0000 7f9765ba2640 1 --2- 172.21.15.60:0/1134882693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760070450 0x7f97600708b0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f97500097a0 tx=0x7f975002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a060d2c9e960bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.358+0000 7f97653a1640 1 -- 172.21.15.60:0/1134882693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750031780 con 0x7f9760070450 2023-10-16T14:11:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.358+0000 7f97653a1640 1 -- 172.21.15.60:0/1134882693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97500318e0 con 0x7f9760070450 2023-10-16T14:11:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.358+0000 7f97653a1640 1 -- 172.21.15.60:0/1134882693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750031c70 con 0x7f9760070450 2023-10-16T14:11:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 -- 172.21.15.60:0/1134882693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760070450 msgr2=0x7f97600708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 --2- 172.21.15.60:0/1134882693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760070450 0x7f97600708b0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f97500097a0 tx=0x7f975002b820 comp rx=0 tx=0).stop 2023-10-16T14:11:58.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 -- 172.21.15.60:0/1134882693 shutdown_connections 2023-10-16T14:11:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 --2- 172.21.15.60:0/1134882693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760070450 0x7f97600708b0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 --2- 172.21.15.60:0/1134882693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760159950 0x7f976015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 -- 172.21.15.60:0/1134882693 >> 172.21.15.60:0/1134882693 conn(0x7f97600ce6a0 msgr2=0x7f97600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 -- 172.21.15.60:0/1134882693 shutdown_connections 2023-10-16T14:11:58.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.359+0000 7f97673a5640 1 -- 172.21.15.60:0/1134882693 wait complete. 2023-10-16T14:11:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.361+0000 7f97673a5640 1 Processor -- start 2023-10-16T14:11:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.361+0000 7f97673a5640 1 -- start start 2023-10-16T14:11:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f97673a5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f976006c100 0x7f97600c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:58.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f97673a5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760070450 0x7f97600c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f97673a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 0x7f97600b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f97673a5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97600b12c0 con 0x7f9760159950 2023-10-16T14:11:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f97673a5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97600b1560 con 0x7f976006c100 2023-10-16T14:11:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f97673a5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97600b1840 con 0x7f9760070450 2023-10-16T14:11:58.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.362+0000 7f9766ba4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 0x7f97600b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 0x7f97600b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36462/0 (socket says 172.21.15.60:36462) 2023-10-16T14:11:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 -- 172.21.15.60:0/2977746645 learned_addr learned my addr 172.21.15.60:0/2977746645 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9765ba2640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760070450 0x7f97600c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f97663a3640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f976006c100 0x7f97600c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 -- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760070450 msgr2=0x7f97600c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760070450 0x7f97600c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 -- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f976006c100 msgr2=0x7f97600c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f976006c100 0x7f97600c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 -- 172.21.15.60:0/2977746645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750009400 con 0x7f9760159950 2023-10-16T14:11:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f97663a3640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f976006c100 0x7f97600c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:58.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9765ba2640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760070450 0x7f97600c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.363+0000 7f9766ba4640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 0x7f97600b2600 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f975800ca80 tx=0x7f975800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.364+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975800ac30 con 0x7f9760159950 2023-10-16T14:11:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.364+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975800ad90 con 0x7f9760159950 2023-10-16T14:11:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.364+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758012600 con 0x7f9760159950 2023-10-16T14:11:58.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.364+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97600b1b20 con 0x7f9760159950 2023-10-16T14:11:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.364+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97600b20e0 con 0x7f9760159950 2023-10-16T14:11:58.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.365+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97600b22d0 con 0x7f9760159950 2023-10-16T14:11:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.366+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9758007070 con 0x7f9760159950 2023-10-16T14:11:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.367+0000 7f9756ffd640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f974406b0b0 0x7f974406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.367+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f9758089b70 con 0x7f9760159950 2023-10-16T14:11:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.368+0000 7f97663a3640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f974406b0b0 0x7f974406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:58.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.369+0000 7f97663a3640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f974406b0b0 0x7f974406d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f975c006fd0 tx=0x7f975c008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:58.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.372+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9758054610 con 0x7f9760159950 2023-10-16T14:11:58.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:11:58.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:11:58.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.609+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f976004e9b0 con 0x7f9760159950 2023-10-16T14:11:58.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.611+0000 7f9756ffd640 1 -- 172.21.15.60:0/2977746645 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6146 (secure 0 0 0) 0x7f9758017020 con 0x7f9760159950 2023-10-16T14:11:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:11:58.615 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:11:58.615 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 134 2023-10-16T14:11:58.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.613+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f974406b0b0 msgr2=0x7f974406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f974406b0b0 0x7f974406d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f975c006fd0 tx=0x7f975c008040 comp rx=0 tx=0).stop 2023-10-16T14:11:58.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 msgr2=0x7f97600b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:58.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 0x7f97600b2600 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f975800ca80 tx=0x7f975800cf40 comp rx=0 tx=0).stop 2023-10-16T14:11:58.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 shutdown_connections 2023-10-16T14:11:58.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f974406b0b0 0x7f974406d570 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9760070450 0x7f97600c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9760159950 0x7f97600b2600 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 --2- 172.21.15.60:0/2977746645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f976006c100 0x7f97600c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 >> 172.21.15.60:0/2977746645 conn(0x7f97600ce6a0 msgr2=0x7f976006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 shutdown_connections 2023-10-16T14:11:58.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:58.614+0000 7f97673a5640 1 -- 172.21.15.60:0/2977746645 wait complete. 2023-10-16T14:11:59.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 134, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:11:52.433484+0000', 'modified': '2023-10-16T14:11:55.513998+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8186}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8186': {'gid': 8186, 'name': 'd', 'rank': 0, 'incarnation': 130, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.144:6836/521250241', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6834', 'nonce': 521250241}, {'type': 'v1', 'addr': '172.21.15.144:6836', 'nonce': 521250241}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:11:59.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:11:59.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T14:11:59.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.276+0000 7f6a1714e640 1 Processor -- start 2023-10-16T14:11:59.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.277+0000 7f6a1714e640 1 -- start start 2023-10-16T14:11:59.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.278+0000 7f6a1714e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a100c9060 0x7f6a1006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.278+0000 7f6a1714e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a100c9440 0x7f6a1006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.278+0000 7f6a1714e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a100ca560 con 0x7f6a1006c9e0 2023-10-16T14:11:59.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.278+0000 7f6a1714e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a100c9950 con 0x7f6a100c9440 2023-10-16T14:11:59.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.278+0000 7f6a1714e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a100c9c30 con 0x7f6a100c9060 2023-10-16T14:11:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.279+0000 7f6a1694d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a100c9440 0x7f6a1006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.279+0000 7f6a1694d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a100c9440 0x7f6a1006f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38614/0 (socket says 172.21.15.60:38614) 2023-10-16T14:11:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.279+0000 7f6a1694d640 1 -- 172.21.15.60:0/3740335621 learned_addr learned my addr 172.21.15.60:0/3740335621 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.279+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3740335621 >> v2:172.21.15.144:3300/0 conn(0x7f6a100c9060 0x7f6a1006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.280+0000 7f6a1694d640 1 -- 172.21.15.60:0/3740335621 >> v2:172.21.15.144:3300/0 conn(0x7f6a100c9060 msgr2=0x7f6a1006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.280+0000 7f6a1694d640 1 --2- 172.21.15.60:0/3740335621 >> v2:172.21.15.144:3300/0 conn(0x7f6a100c9060 0x7f6a1006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.280+0000 7f6a1694d640 1 -- 172.21.15.60:0/3740335621 >> v1:172.21.15.144:6789/0 conn(0x7f6a1006c9e0 legacy=0x7f6a10164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.280+0000 7f6a1694d640 1 -- 172.21.15.60:0/3740335621 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100c9f10 con 0x7f6a100c9440 2023-10-16T14:11:59.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.281+0000 7f6a1694d640 1 --2- 172.21.15.60:0/3740335621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a100c9440 0x7f6a1006f980 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f6a1006d490 tx=0x7f6a0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b75370fcb113392 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.281+0000 7f6a1514a640 1 -- 172.21.15.60:0/3740335621 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08030860 con 0x7f6a100c9440 2023-10-16T14:11:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1514a640 1 -- 172.21.15.60:0/3740335621 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a080309c0 con 0x7f6a100c9440 2023-10-16T14:11:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1514a640 1 -- 172.21.15.60:0/3740335621 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08030d50 con 0x7f6a100c9440 2023-10-16T14:11:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 -- 172.21.15.60:0/3740335621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a100c9440 msgr2=0x7f6a1006f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3740335621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a100c9440 0x7f6a1006f980 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f6a1006d490 tx=0x7f6a0802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 -- 172.21.15.60:0/3740335621 shutdown_connections 2023-10-16T14:11:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3740335621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a100c9440 0x7f6a1006f980 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3740335621 >> v2:172.21.15.144:3300/0 conn(0x7f6a100c9060 0x7f6a1006f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 -- 172.21.15.60:0/3740335621 >> 172.21.15.60:0/3740335621 conn(0x7f6a100ce6a0 msgr2=0x7f6a100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 -- 172.21.15.60:0/3740335621 shutdown_connections 2023-10-16T14:11:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.282+0000 7f6a1714e640 1 -- 172.21.15.60:0/3740335621 wait complete. 2023-10-16T14:11:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.284+0000 7f6a1714e640 1 Processor -- start 2023-10-16T14:11:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.284+0000 7f6a1714e640 1 -- start start 2023-10-16T14:11:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1714e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 0x7f6a100c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1714e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100c9060 0x7f6a100c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1714e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a100c9440 0x7f6a100c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1714e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100c7250 con 0x7f6a1006c9e0 2023-10-16T14:11:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1714e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100c21c0 con 0x7f6a100c9440 2023-10-16T14:11:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1714e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100c24a0 con 0x7f6a100c9060 2023-10-16T14:11:59.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.285+0000 7f6a1614c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 0x7f6a100c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 0x7f6a100c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38624/0 (socket says 172.21.15.60:38624) 2023-10-16T14:11:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 -- 172.21.15.60:0/3771944916 learned_addr learned my addr 172.21.15.60:0/3771944916 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 -- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100c9060 msgr2=0x7f6a100c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1594b640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100c9060 0x7f6a100c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100c9060 0x7f6a100c67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1694d640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a100c9440 0x7f6a100c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 -- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a100c9440 msgr2=0x7f6a100c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a100c9440 0x7f6a100c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 -- 172.21.15.60:0/3771944916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a080094d0 con 0x7f6a1006c9e0 2023-10-16T14:11:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1694d640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a100c9440 0x7f6a100c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.286+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 0x7f6a100c6260 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c007ec0 tx=0x7f6a0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.287+0000 7f6a06ffd640 1 -- 172.21.15.60:0/3771944916 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c014620 con 0x7f6a1006c9e0 2023-10-16T14:11:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.287+0000 7f6a06ffd640 1 -- 172.21.15.60:0/3771944916 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c0147a0 con 0x7f6a1006c9e0 2023-10-16T14:11:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.287+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a100c2780 con 0x7f6a1006c9e0 2023-10-16T14:11:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.287+0000 7f6a06ffd640 1 -- 172.21.15.60:0/3771944916 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c014b50 con 0x7f6a1006c9e0 2023-10-16T14:11:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.287+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a100c6ce0 con 0x7f6a1006c9e0 2023-10-16T14:11:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.290+0000 7f6a06ffd640 1 -- 172.21.15.60:0/3771944916 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6a0c014cb0 con 0x7f6a1006c9e0 2023-10-16T14:11:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.291+0000 7f6a06ffd640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f69f406b0b0 0x7f69f406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.291+0000 7f6a06ffd640 1 -- 172.21.15.60:0/3771944916 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f6a0c08a740 con 0x7f6a1006c9e0 2023-10-16T14:11:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.292+0000 7f6a1b027740 1 Processor -- start 2023-10-16T14:11:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.292+0000 7f6a1b027740 1 -- start start 2023-10-16T14:11:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.292+0000 7f6a1594b640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f69f406b0b0 0x7f69f406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a550944070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a550973f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a5509445b0 0x55a550974440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1b027740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5508647a0 con 0x55a550971170 2023-10-16T14:11:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5508b6740 con 0x55a5509445b0 2023-10-16T14:11:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5508b3e30 con 0x55a550971550 2023-10-16T14:11:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a550944070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a550944070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38626/0 (socket says 172.21.15.60:38626) 2023-10-16T14:11:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 -- 172.21.15.60:0/3331794324 learned_addr learned my addr 172.21.15.60:0/3331794324 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1594b640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f69f406b0b0 0x7f69f406d570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6a1015b560 tx=0x7f6a00009040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1594b640 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a550973f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1694d640 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a5509445b0 0x55a550974440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 -- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 msgr2=0x55a550973f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a550973f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 -- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a5509445b0 msgr2=0x55a550974440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a5509445b0 0x55a550974440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.293+0000 7f6a1614c640 1 -- 172.21.15.60:0/3331794324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a550873910 con 0x55a550971170 2023-10-16T14:11:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.294+0000 7f6a1594b640 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a550973f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.294+0000 7f6a1614c640 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a550944070 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c058e40 tx=0x7f6a0c0596f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2312b43d35a3f7e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.294+0000 7f69f3fff640 1 -- 172.21.15.60:0/3331794324 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0611b0 con 0x55a550971170 2023-10-16T14:11:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.294+0000 7f69f3fff640 1 -- 172.21.15.60:0/3331794324 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c057250 con 0x55a550971170 2023-10-16T14:11:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.294+0000 7f69f3fff640 1 -- 172.21.15.60:0/3331794324 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c059110 con 0x55a550971170 2023-10-16T14:11:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 -- 172.21.15.60:0/3331794324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 msgr2=0x55a550944070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a550944070 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c058e40 tx=0x7f6a0c0596f0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 -- 172.21.15.60:0/3331794324 shutdown_connections 2023-10-16T14:11:59.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a550973f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a5509445b0 0x55a550974440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 --2- 172.21.15.60:0/3331794324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a550944070 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 -- 172.21.15.60:0/3331794324 >> 172.21.15.60:0/3331794324 conn(0x55a550946380 msgr2=0x55a55098ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 -- 172.21.15.60:0/3331794324 shutdown_connections 2023-10-16T14:11:59.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.295+0000 7f6a1b027740 1 -- 172.21.15.60:0/3331794324 wait complete. 2023-10-16T14:11:59.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.296+0000 7f6a1b027740 1 -- start start 2023-10-16T14:11:59.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.296+0000 7f6a1b027740 1 Processor -- start 2023-10-16T14:11:59.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a5509896d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 0x55a550961ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5508af720 con 0x55a550989c10 2023-10-16T14:11:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55084d970 con 0x55a550971170 2023-10-16T14:11:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a55084d610 con 0x55a550971550 2023-10-16T14:11:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1b027740 10 client.0 _ll_register_callbacks cb 0x55a55086f9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:11:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1694d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 0x55a550961ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1694d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 0x55a550961ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38636/0 (socket says 172.21.15.60:38636) 2023-10-16T14:11:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1694d640 1 -- 172.21.15.60:0/2972902644 learned_addr learned my addr 172.21.15.60:0/2972902644 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.297+0000 7f6a1614c640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1594b640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a5509896d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1694d640 1 -- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 msgr2=0x55a5509896d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1694d640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a5509896d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1694d640 1 -- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 msgr2=0x55a5509768b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1694d640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509768b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1694d640 1 -- 172.21.15.60:0/2972902644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a100c6ce0 con 0x55a550989c10 2023-10-16T14:11:59.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1594b640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a5509896d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:59.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1694d640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 0x55a550961ee0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f6a08009960 tx=0x7f6a0802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a388c8da44ce63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f69f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:11:59.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f69f17fa640 1 -- 172.21.15.60:0/2972902644 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08039860 con 0x55a550989c10 2023-10-16T14:11:59.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f69f17fa640 1 -- 172.21.15.60:0/2972902644 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f6a0803e040 con 0x55a550989c10 2023-10-16T14:11:59.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.298+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5508d6160 con 0x55a550989c10 2023-10-16T14:11:59.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f69f17fa640 1 -- 172.21.15.60:0/2972902644 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a080399c0 con 0x55a550989c10 2023-10-16T14:11:59.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 10 client.12096 fetch_fsmap learned FSMap version 134 2023-10-16T14:11:59.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x55a5508adda0 con 0x55a550989c10 2023-10-16T14:11:59.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f69f17fa640 1 -- 172.21.15.60:0/2972902644 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a5508d6160 con 0x55a550989c10 2023-10-16T14:11:59.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f69f17fa640 1 -- 172.21.15.60:0/2972902644 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 134) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f6a08042050 con 0x55a550989c10 2023-10-16T14:11:59.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 10 client.12096 fetch_fsmap finished waiting for FSMap version 134 2023-10-16T14:11:59.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 10 client.12096 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T14:11:59.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 20 client.12096 populate_metadata read hostname 'smithi060' 2023-10-16T14:11:59.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a31d0 0x55a5509acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 4 client.12096 mds_command: new command op to 8186 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:11:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.299+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 --> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a550881cc0 con 0x55a5509a31d0 2023-10-16T14:11:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.300+0000 7f6a1614c640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a31d0 0x55a5509acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.301+0000 7f6a1614c640 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a31d0 0x55a5509acaa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ec6cbd35fc5efec server_cookie=1460098188a05a7c in_seq=0 out_seq=0 2023-10-16T14:11:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.301+0000 7f69f17fa640 10 client.12096 ms_handle_connect on v2:172.21.15.144:6834/521250241 2023-10-16T14:11:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f69f17fa640 1 -- 172.21.15.60:0/2972902644 <== mds.0 v2:172.21.15.144:6834/521250241 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a550881cc0 con 0x55a5509a31d0 2023-10-16T14:11:59.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f69f17fa640 10 client.12096 handle_command_reply: tid=0 2023-10-16T14:11:59.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 client.12096 shutdown 2023-10-16T14:11:59.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 msgr2=0x55a550961ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 0x55a550961ee0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f6a08009960 tx=0x7f6a0802e000 comp rx=0 tx=0).stop 2023-10-16T14:11:59.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 shutdown_connections 2023-10-16T14:11:59.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a31d0 0x55a5509acaa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550971550 0x55a5509896d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550989c10 0x55a550961ee0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 --2- 172.21.15.60:0/2972902644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f69f17fa640 0 client.12096 ms_handle_reset on v2:172.21.15.144:6834/521250241 2023-10-16T14:11:59.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 >> 172.21.15.60:0/2972902644 conn(0x55a5509445b0 msgr2=0x55a550973ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 shutdown_connections 2023-10-16T14:11:59.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 1 -- 172.21.15.60:0/2972902644 wait complete. 2023-10-16T14:11:59.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.303+0000 7f6a1b027740 20 client.12096 trim_cache size 0 max 16384 2023-10-16T14:11:59.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.321+0000 7f6a1b027740 1 Processor -- start 2023-10-16T14:11:59.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.321+0000 7f6a1b027740 1 -- start start 2023-10-16T14:11:59.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 0x55a550992640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1b027740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a550613f20 con 0x55a550971550 2023-10-16T14:11:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a550987ad0 con 0x55a550971170 2023-10-16T14:11:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a55098fd50 con 0x55a550989c10 2023-10-16T14:11:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1594b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 0x55a550992640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1594b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 0x55a550992640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38652/0 (socket says 172.21.15.60:38652) 2023-10-16T14:11:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.322+0000 7f6a1594b640 1 -- 172.21.15.60:0/1423514346 learned_addr learned my addr 172.21.15.60:0/1423514346 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1614c640 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1694d640 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1594b640 1 -- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 msgr2=0x55a550992b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1594b640 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1594b640 1 -- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 msgr2=0x55a5509b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1594b640 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1594b640 1 -- 172.21.15.60:0/1423514346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a55084cb80 con 0x55a550971550 2023-10-16T14:11:59.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a1594b640 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 0x55a550992640 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f6a00005120 tx=0x7f6a00013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d3f5bec60724d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a04e01640 1 -- 172.21.15.60:0/1423514346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00003810 con 0x55a550971550 2023-10-16T14:11:59.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.323+0000 7f6a04e01640 1 -- 172.21.15.60:0/1423514346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0001b030 con 0x55a550971550 2023-10-16T14:11:59.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a04e01640 1 -- 172.21.15.60:0/1423514346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00003b70 con 0x55a550971550 2023-10-16T14:11:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 -- 172.21.15.60:0/1423514346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 msgr2=0x55a550992640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 0x55a550992640 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f6a00005120 tx=0x7f6a00013000 comp rx=0 tx=0).stop 2023-10-16T14:11:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 -- 172.21.15.60:0/1423514346 shutdown_connections 2023-10-16T14:11:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971550 0x55a550992640 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1423514346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971170 0x55a5509b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 -- 172.21.15.60:0/1423514346 >> 172.21.15.60:0/1423514346 conn(0x55a5509445b0 msgr2=0x55a55093f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 -- 172.21.15.60:0/1423514346 shutdown_connections 2023-10-16T14:11:59.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.324+0000 7f6a1b027740 1 -- 172.21.15.60:0/1423514346 wait complete. 2023-10-16T14:11:59.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.325+0000 7f6a1b027740 1 -- start start 2023-10-16T14:11:59.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.325+0000 7f6a1b027740 1 Processor -- start 2023-10-16T14:11:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a55094a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971550 0x55a5509b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5509f8230 con 0x55a550971170 2023-10-16T14:11:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5509f1840 con 0x55a550971550 2023-10-16T14:11:59.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a550949690 con 0x55a550989c10 2023-10-16T14:11:59.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1b027740 10 client.0 _ll_register_callbacks cb 0x7f6a080096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:11:59.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1614c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a55094a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1614c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a55094a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38662/0 (socket says 172.21.15.60:38662) 2023-10-16T14:11:59.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.326+0000 7f6a1614c640 1 -- 172.21.15.60:0/1127321709 learned_addr learned my addr 172.21.15.60:0/1127321709 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1614c640 1 -- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 msgr2=0x55a550992410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1614c640 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1594b640 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971550 0x55a5509b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1614c640 1 -- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971550 msgr2=0x55a5509b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1614c640 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971550 0x55a5509b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1614c640 1 -- 172.21.15.60:0/1127321709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a100ca560 con 0x55a550971170 2023-10-16T14:11:59.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1614c640 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a55094a170 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c058df0 tx=0x7f6a0c05f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e72ed5ebcb6501 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f69f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:11:59.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f69f17fa640 1 -- 172.21.15.60:0/1127321709 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c05f0f0 con 0x55a550971170 2023-10-16T14:11:59.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f69f17fa640 1 -- 172.21.15.60:0/1127321709 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f6a0c087070 con 0x55a550971170 2023-10-16T14:11:59.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.327+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a0c057250 con 0x55a550971170 2023-10-16T14:11:59.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f69f17fa640 1 -- 172.21.15.60:0/1127321709 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a0c0555c0 con 0x55a550971170 2023-10-16T14:11:59.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 10 client.12102 fetch_fsmap learned FSMap version 134 2023-10-16T14:11:59.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x55a550872170 con 0x55a550971170 2023-10-16T14:11:59.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f69f17fa640 1 -- 172.21.15.60:0/1127321709 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c057250 con 0x55a550971170 2023-10-16T14:11:59.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f69f17fa640 1 -- 172.21.15.60:0/1127321709 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 134) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f6a0c0583a0 con 0x55a550971170 2023-10-16T14:11:59.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 10 client.12102 fetch_fsmap finished waiting for FSMap version 134 2023-10-16T14:11:59.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 10 client.12102 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T14:11:59.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 20 client.12102 populate_metadata read hostname 'smithi060' 2023-10-16T14:11:59.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a1c30 0x55a5509a2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 4 client.12102 mds_command: new command op to 8186 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:11:59.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.328+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 --> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a5508a8700 con 0x55a5509a1c30 2023-10-16T14:11:59.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.329+0000 7f6a1594b640 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a1c30 0x55a5509a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.330+0000 7f6a1594b640 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a1c30 0x55a5509a2010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b79a1dd46fd32672 server_cookie=271eab2380714e30 in_seq=0 out_seq=0 2023-10-16T14:11:59.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.330+0000 7f69f17fa640 10 client.12102 ms_handle_connect on v2:172.21.15.144:6834/521250241 2023-10-16T14:11:59.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f69f17fa640 1 -- 172.21.15.60:0/1127321709 <== mds.0 v2:172.21.15.144:6834/521250241 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a5508a8700 con 0x55a5509a1c30 2023-10-16T14:11:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f69f17fa640 10 client.12102 handle_command_reply: tid=0 2023-10-16T14:11:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 client.12102 shutdown 2023-10-16T14:11:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 msgr2=0x55a55094a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a55094a170 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c058df0 tx=0x7f6a0c05f000 comp rx=0 tx=0).stop 2023-10-16T14:11:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 shutdown_connections 2023-10-16T14:11:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:6834/521250241,v1:172.21.15.144:6836/521250241] conn(0x55a5509a1c30 0x55a5509a2010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a550989c10 0x55a550992410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a550971550 0x55a5509b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 --2- 172.21.15.60:0/1127321709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a550971170 0x55a55094a170 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f69f17fa640 0 client.12102 ms_handle_reset on v2:172.21.15.144:6834/521250241 2023-10-16T14:11:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 >> 172.21.15.60:0/1127321709 conn(0x55a5509445b0 msgr2=0x55a550973ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 shutdown_connections 2023-10-16T14:11:59.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 1 -- 172.21.15.60:0/1127321709 wait complete. 2023-10-16T14:11:59.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.331+0000 7f6a1b027740 20 client.12102 trim_cache size 0 max 16384 2023-10-16T14:11:59.345 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:11:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:11:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:11:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 8186, 2023-10-16T14:11:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:11:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:11:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "backup_fs", 2023-10-16T14:11:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.3525904620000002, 2023-10-16T14:11:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 134, 2023-10-16T14:11:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 256, 2023-10-16T14:11:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 256, 2023-10-16T14:11:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 27.375346516 2023-10-16T14:11:59.348 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:11:59.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f69f406b0b0 msgr2=0x7f69f406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f69f406b0b0 0x7f69f406d570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6a1015b560 tx=0x7f6a00009040 comp rx=0 tx=0).stop 2023-10-16T14:11:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 msgr2=0x7f6a100c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 0x7f6a100c6260 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c007ec0 tx=0x7f6a0c00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 shutdown_connections 2023-10-16T14:11:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f69f406b0b0 0x7f69f406d570 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6a100c9060 0x7f6a100c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a100c9440 0x7f6a100c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 --2- 172.21.15.60:0/3771944916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a1006c9e0 0x7f6a100c6260 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 >> 172.21.15.60:0/3771944916 conn(0x7f6a100ce6a0 msgr2=0x7f6a100af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 shutdown_connections 2023-10-16T14:11:59.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.332+0000 7f6a1714e640 1 -- 172.21.15.60:0/3771944916 wait complete. 2023-10-16T14:11:59.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:11:59.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.570+0000 7f5f912ec640 1 Processor -- start 2023-10-16T14:11:59.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.571+0000 7f5f912ec640 1 -- start start 2023-10-16T14:11:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.572+0000 7f5f912ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f8c15d150 0x7f5f8c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.572+0000 7f5f912ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f8c160b90 0x7f5f8c160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.572+0000 7f5f912ec640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8c0ac890 con 0x7f5f8c069da0 2023-10-16T14:11:59.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.572+0000 7f5f912ec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c15d530 con 0x7f5f8c15d150 2023-10-16T14:11:59.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.572+0000 7f5f912ec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c161650 con 0x7f5f8c160b90 2023-10-16T14:11:59.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f8c15d150 0x7f5f8c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f8c15d150 0x7f5f8c0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38664/0 (socket says 172.21.15.60:38664) 2023-10-16T14:11:59.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 -- 172.21.15.60:0/2738753201 learned_addr learned my addr 172.21.15.60:0/2738753201 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f827fc640 1 --2- 172.21.15.60:0/2738753201 >> v2:172.21.15.144:3300/0 conn(0x7f5f8c160b90 0x7f5f8c160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 -- 172.21.15.60:0/2738753201 >> v2:172.21.15.144:3300/0 conn(0x7f5f8c160b90 msgr2=0x7f5f8c160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:11:59.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 --2- 172.21.15.60:0/2738753201 >> v2:172.21.15.144:3300/0 conn(0x7f5f8c160b90 0x7f5f8c160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 -- 172.21.15.60:0/2738753201 >> v1:172.21.15.60:6789/0 conn(0x7f5f8c069da0 legacy=0x7f5f8c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f8affd640 1 -- 172.21.15.60:0/2738753201 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c161930 con 0x7f5f8c15d150 2023-10-16T14:11:59.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.573+0000 7f5f827fc640 1 --2- 172.21.15.60:0/2738753201 >> v2:172.21.15.144:3300/0 conn(0x7f5f8c160b90 0x7f5f8c160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:11:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.574+0000 7f5f8affd640 1 --2- 172.21.15.60:0/2738753201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c15d150 0x7f5f8c0ad350 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c0cf990 tx=0x7f5f7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e35d936c9ba5cba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.575+0000 7f5f8a7fc640 1 -- 172.21.15.60:0/2738753201 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c031870 con 0x7f5f8c15d150 2023-10-16T14:11:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.575+0000 7f5f8a7fc640 1 -- 172.21.15.60:0/2738753201 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c0319d0 con 0x7f5f8c15d150 2023-10-16T14:11:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.575+0000 7f5f8a7fc640 1 -- 172.21.15.60:0/2738753201 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c031d60 con 0x7f5f8c15d150 2023-10-16T14:11:59.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.575+0000 7f5f912ec640 1 -- 172.21.15.60:0/2738753201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c15d150 msgr2=0x7f5f8c0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.575+0000 7f5f912ec640 1 --2- 172.21.15.60:0/2738753201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c15d150 0x7f5f8c0ad350 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c0cf990 tx=0x7f5f7c02b910 comp rx=0 tx=0).stop 2023-10-16T14:11:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.576+0000 7f5f912ec640 1 -- 172.21.15.60:0/2738753201 shutdown_connections 2023-10-16T14:11:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.576+0000 7f5f912ec640 1 --2- 172.21.15.60:0/2738753201 >> v2:172.21.15.144:3300/0 conn(0x7f5f8c160b90 0x7f5f8c160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.576+0000 7f5f912ec640 1 --2- 172.21.15.60:0/2738753201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c15d150 0x7f5f8c0ad350 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.576+0000 7f5f912ec640 1 -- 172.21.15.60:0/2738753201 >> 172.21.15.60:0/2738753201 conn(0x7f5f8c0ce9b0 msgr2=0x7f5f8c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.576+0000 7f5f912ec640 1 -- 172.21.15.60:0/2738753201 shutdown_connections 2023-10-16T14:11:59.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.576+0000 7f5f912ec640 1 -- 172.21.15.60:0/2738753201 wait complete. 2023-10-16T14:11:59.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.578+0000 7f5f912ec640 1 Processor -- start 2023-10-16T14:11:59.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.578+0000 7f5f912ec640 1 -- start start 2023-10-16T14:11:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f912ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f8c069da0 0x7f5f8c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f912ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f8c15d150 0x7f5f8c0aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f912ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 0x7f5f8c1f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f912ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b0120 con 0x7f5f8c160b90 2023-10-16T14:11:59.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f912ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c15b880 con 0x7f5f8c069da0 2023-10-16T14:11:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f912ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0af4a0 con 0x7f5f8c15d150 2023-10-16T14:11:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f8b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 0x7f5f8c1f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f8b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 0x7f5f8c1f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38670/0 (socket says 172.21.15.60:38670) 2023-10-16T14:11:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f8b7fe640 1 -- 172.21.15.60:0/736743557 learned_addr learned my addr 172.21.15.60:0/736743557 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:11:59.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f8affd640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f8c069da0 0x7f5f8c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.579+0000 7f5f827fc640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f8c15d150 0x7f5f8c0aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f8b7fe640 1 -- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f8c15d150 msgr2=0x7f5f8c0aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f8b7fe640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f8c15d150 0x7f5f8c0aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f8b7fe640 1 -- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f8c069da0 msgr2=0x7f5f8c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f8b7fe640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f8c069da0 0x7f5f8c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f8b7fe640 1 -- 172.21.15.60:0/736743557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0094d0 con 0x7f5f8c160b90 2023-10-16T14:11:59.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f8b7fe640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 0x7f5f8c1f3210 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c6b0 tx=0x7f5f8400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84003b70 con 0x7f5f8c160b90 2023-10-16T14:11:59.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84003cf0 con 0x7f5f8c160b90 2023-10-16T14:11:59.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84005310 con 0x7f5f8c160b90 2023-10-16T14:11:59.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.580+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c0af780 con 0x7f5f8c160b90 2023-10-16T14:11:59.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.581+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c0afd40 con 0x7f5f8c160b90 2023-10-16T14:11:59.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.582+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c0b0260 con 0x7f5f8c160b90 2023-10-16T14:11:59.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.583+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5f84005470 con 0x7f5f8c160b90 2023-10-16T14:11:59.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.584+0000 7f5f83fff640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f5f6c06b0b0 0x7f5f6c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:11:59.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.584+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f5f84089c70 con 0x7f5f8c160b90 2023-10-16T14:11:59.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.584+0000 7f5f8affd640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f5f6c06b0b0 0x7f5f6c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:11:59.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.586+0000 7f5f8affd640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f5f6c06b0b0 0x7f5f6c06d570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c036000 tx=0x7f5f7c03c040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:11:59.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.589+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f8c0b0260 con 0x7f5f8c160b90 2023-10-16T14:11:59.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.812+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 --> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f8c0ace30 con 0x7f5f6c06b0b0 2023-10-16T14:11:59.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.816+0000 7f5f83fff640 1 -- 172.21.15.60:0/736743557 <== mgr.9169 v2:172.21.15.144:6832/4052124783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f5f7c031810 con 0x7f5f6c06b0b0 2023-10-16T14:11:59.819 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:11:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T14:11:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:11:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:11:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "backup_fs" 2023-10-16T14:11:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:11:59.821 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:11:59.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f5f6c06b0b0 msgr2=0x7f5f6c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f5f6c06b0b0 0x7f5f6c06d570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c036000 tx=0x7f5f7c03c040 comp rx=0 tx=0).stop 2023-10-16T14:11:59.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 msgr2=0x7f5f8c1f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:11:59.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 0x7f5f8c1f3210 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c6b0 tx=0x7f5f8400cb70 comp rx=0 tx=0).stop 2023-10-16T14:11:59.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 shutdown_connections 2023-10-16T14:11:59.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f5f6c06b0b0 0x7f5f6c06d570 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5f8c15d150 0x7f5f8c0aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f8c160b90 0x7f5f8c1f3210 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 --2- 172.21.15.60:0/736743557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f8c069da0 0x7f5f8c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:11:59.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 >> 172.21.15.60:0/736743557 conn(0x7f5f8c0ce9b0 msgr2=0x7f5f8c06fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:11:59.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 shutdown_connections 2023-10-16T14:11:59.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:11:59.818+0000 7f5f912ec640 1 -- 172.21.15.60:0/736743557 wait complete. 2023-10-16T14:11:59.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:12:00.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.064+0000 7f31320d7640 1 Processor -- start 2023-10-16T14:12:00.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.064+0000 7f31320d7640 1 -- start start 2023-10-16T14:12:00.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.065+0000 7f31320d7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f312c15ad70 0x7f312c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:00.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.065+0000 7f31320d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0c9ca0 0x7f312c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:00.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f31320d7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312c0cb030 con 0x7f312c0ca3b0 2023-10-16T14:12:00.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f31320d7640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c0c91c0 con 0x7f312c15ad70 2023-10-16T14:12:00.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f31320d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0c94a0 con 0x7f312c0c9ca0 2023-10-16T14:12:00.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f312b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f312c15ad70 0x7f312c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:00.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f3122ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0c9ca0 0x7f312c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f312bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f312c0ca3b0 0x7f312c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:52008/0 (socket says 172.21.15.60:52008) 2023-10-16T14:12:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.066+0000 7f312bfff640 1 -- 172.21.15.60:0/3373357745 learned_addr learned my addr 172.21.15.60:0/3373357745 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.067+0000 7f3122ffd640 1 -- 172.21.15.60:0/3373357745 >> v2:172.21.15.144:3300/0 conn(0x7f312c15ad70 msgr2=0x7f312c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.067+0000 7f3122ffd640 1 --2- 172.21.15.60:0/3373357745 >> v2:172.21.15.144:3300/0 conn(0x7f312c15ad70 0x7f312c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:00.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.067+0000 7f3122ffd640 1 -- 172.21.15.60:0/3373357745 >> v1:172.21.15.144:6789/0 conn(0x7f312c0ca3b0 legacy=0x7f312c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:00.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.067+0000 7f3122ffd640 1 -- 172.21.15.60:0/3373357745 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c0c9780 con 0x7f312c0c9ca0 2023-10-16T14:12:00.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.069+0000 7f3122ffd640 1 --2- 172.21.15.60:0/3373357745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0c9ca0 0x7f312c06a2e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f31140097a0 tx=0x7f311402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35660b10a76d54 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:00.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.070+0000 7f312affd640 1 -- 172.21.15.60:0/3373357745 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114031830 con 0x7f312c0c9ca0 2023-10-16T14:12:00.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.070+0000 7f312affd640 1 -- 172.21.15.60:0/3373357745 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114031990 con 0x7f312c0c9ca0 2023-10-16T14:12:00.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.070+0000 7f312affd640 1 -- 172.21.15.60:0/3373357745 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114031d20 con 0x7f312c0c9ca0 2023-10-16T14:12:00.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.070+0000 7f31320d7640 1 -- 172.21.15.60:0/3373357745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0c9ca0 msgr2=0x7f312c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:00.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.070+0000 7f31320d7640 1 --2- 172.21.15.60:0/3373357745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0c9ca0 0x7f312c06a2e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f31140097a0 tx=0x7f311402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:00.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.071+0000 7f31320d7640 1 -- 172.21.15.60:0/3373357745 shutdown_connections 2023-10-16T14:12:00.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.071+0000 7f31320d7640 1 --2- 172.21.15.60:0/3373357745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0c9ca0 0x7f312c06a2e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:00.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.071+0000 7f31320d7640 1 --2- 172.21.15.60:0/3373357745 >> v2:172.21.15.144:3300/0 conn(0x7f312c15ad70 0x7f312c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.071+0000 7f31320d7640 1 -- 172.21.15.60:0/3373357745 >> 172.21.15.60:0/3373357745 conn(0x7f312c0ce9b0 msgr2=0x7f312c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.071+0000 7f31320d7640 1 -- 172.21.15.60:0/3373357745 shutdown_connections 2023-10-16T14:12:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.071+0000 7f31320d7640 1 -- 172.21.15.60:0/3373357745 wait complete. 2023-10-16T14:12:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.073+0000 7f31320d7640 1 Processor -- start 2023-10-16T14:12:00.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.073+0000 7f31320d7640 1 -- start start 2023-10-16T14:12:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f31320d7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f312c0c9ca0 0x7f312c0c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f31320d7640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0ca3b0 0x7f312c149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f31320d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 0x7f312c149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:00.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f31320d7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0c4090 con 0x7f312c15ad70 2023-10-16T14:12:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f31320d7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c149740 con 0x7f312c0c9ca0 2023-10-16T14:12:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f31320d7640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0c3480 con 0x7f312c0ca3b0 2023-10-16T14:12:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f312bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 0x7f312c149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:00.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f312bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 0x7f312c149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38678/0 (socket says 172.21.15.60:38678) 2023-10-16T14:12:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f312bfff640 1 -- 172.21.15.60:0/1320887647 learned_addr learned my addr 172.21.15.60:0/1320887647 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f312b7fe640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f312c0c9ca0 0x7f312c0c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.074+0000 7f3122ffd640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0ca3b0 0x7f312c149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f312bfff640 1 -- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0ca3b0 msgr2=0x7f312c149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:00.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f312bfff640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0ca3b0 0x7f312c149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f312bfff640 1 -- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f312c0c9ca0 msgr2=0x7f312c0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f312bfff640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f312c0c9ca0 0x7f312c0c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f312bfff640 1 -- 172.21.15.60:0/1320887647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3114009400 con 0x7f312c15ad70 2023-10-16T14:12:00.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f312bfff640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 0x7f312c149f50 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f312400cbf0 tx=0x7f312400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312400ad40 con 0x7f312c15ad70 2023-10-16T14:12:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3124012400 con 0x7f312c15ad70 2023-10-16T14:12:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.075+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c0c3760 con 0x7f312c15ad70 2023-10-16T14:12:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.076+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312400aea0 con 0x7f312c15ad70 2023-10-16T14:12:00.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.076+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c0c3d20 con 0x7f312c15ad70 2023-10-16T14:12:00.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.077+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c0c4240 con 0x7f312c15ad70 2023-10-16T14:12:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.078+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3124007070 con 0x7f312c15ad70 2023-10-16T14:12:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.079+0000 7f3128ff9640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f310c06b0b0 0x7f310c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.079+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f3124089b70 con 0x7f312c15ad70 2023-10-16T14:12:00.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.080+0000 7f312b7fe640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f310c06b0b0 0x7f310c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:00.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.081+0000 7f312b7fe640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f310c06b0b0 0x7f310c06d570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f311c009a30 tx=0x7f311c008040 comp rx=0 tx=0).ready entity=mgr.9169 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:00.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.083+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31240546a0 con 0x7f312c15ad70 2023-10-16T14:12:00.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:00.330+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f312c04e160 con 0x7f312c15ad70 2023-10-16T14:12:01.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.055+0000 7f3128ff9640 1 -- 172.21.15.60:0/1320887647 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v138) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3124057cc0 con 0x7f312c15ad70 2023-10-16T14:12:01.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.059+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f310c06b0b0 msgr2=0x7f310c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.059+0000 7f31320d7640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f310c06b0b0 0x7f310c06d570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f311c009a30 tx=0x7f311c008040 comp rx=0 tx=0).stop 2023-10-16T14:12:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.059+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 msgr2=0x7f312c149f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.059+0000 7f31320d7640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 0x7f312c149f50 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f312400cbf0 tx=0x7f312400a430 comp rx=0 tx=0).stop 2023-10-16T14:12:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.060+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 shutdown_connections 2023-10-16T14:12:01.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.060+0000 7f31320d7640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f310c06b0b0 0x7f310c06d570 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.060+0000 7f31320d7640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f312c0ca3b0 0x7f312c149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.060+0000 7f31320d7640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f312c15ad70 0x7f312c149f50 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.060+0000 7f31320d7640 1 --2- 172.21.15.60:0/1320887647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f312c0c9ca0 0x7f312c0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.060+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 >> 172.21.15.60:0/1320887647 conn(0x7f312c0ce9b0 msgr2=0x7f312c0ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:01.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.061+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 shutdown_connections 2023-10-16T14:12:01.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.061+0000 7f31320d7640 1 -- 172.21.15.60:0/1320887647 wait complete. 2023-10-16T14:12:01.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:12:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.300+0000 7fdc01f48640 1 Processor -- start 2023-10-16T14:12:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.300+0000 7fdc01f48640 1 -- start start 2023-10-16T14:12:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdc01f48640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 0x7fdbfc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdc01f48640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc161740 con 0x7fdbfc0cf5f0 2023-10-16T14:12:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdc01f48640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc06b0f0 con 0x7fdbfc070c10 2023-10-16T14:12:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdc01f48640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1619b0 con 0x7fdbfc070830 2023-10-16T14:12:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdbfaffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdbfc0cf5f0 0x7fdbfc06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47856/0 (socket says 172.21.15.60:47856) 2023-10-16T14:12:01.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdbfaffd640 1 -- 172.21.15.60:0/3366975630 learned_addr learned my addr 172.21.15.60:0/3366975630 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:01.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.302+0000 7fdbfb7fe640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 0x7fdbfc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978002612 0 0) 0x7fdbfc161740 con 0x7fdbfc0cf5f0 2023-10-16T14:12:01.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe4003520 con 0x7fdbfc0cf5f0 2023-10-16T14:12:01.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068103495 0 0) 0x7fdbe4003520 con 0x7fdbfc0cf5f0 2023-10-16T14:12:01.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 msgr2=0x7fdbfc06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:01.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 0x7fdbfc06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 >> v1:172.21.15.144:6789/0 conn(0x7fdbfc070c10 legacy=0x7fdbfc0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc06b3d0 con 0x7fdbfc0cf5f0 2023-10-16T14:12:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fdbf0001670 con 0x7fdbfc0cf5f0 2023-10-16T14:12:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 >> v1:172.21.15.60:6789/0 conn(0x7fdbfc0cf5f0 legacy=0x7fdbfc06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 0x7fdbe40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe4003f20 0x7fdbe40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbe4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1619b0 con 0x7fdbe4004d00 2023-10-16T14:12:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe400bbc0 con 0x7fdbe4003f20 2023-10-16T14:12:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.303+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe400bd80 con 0x7fdbfc070830 2023-10-16T14:12:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfbfff640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbe4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfb7fe640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 0x7fdbe40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfaffd640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe4003f20 0x7fdbe40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfbfff640 1 -- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 msgr2=0x7fdbe40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfbfff640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc070830 0x7fdbe40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfbfff640 1 -- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe4003f20 msgr2=0x7fdbe40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfbfff640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbe4003f20 0x7fdbe40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.304+0000 7fdbfbfff640 1 -- 172.21.15.60:0/3366975630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe4003520 con 0x7fdbe4004d00 2023-10-16T14:12:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.305+0000 7fdbfbfff640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbe4005180 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4001e50 tx=0x7fdbe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eef99552cb98503 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4003b70 con 0x7fdbe4004d00 2023-10-16T14:12:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4003cd0 con 0x7fdbe4004d00 2023-10-16T14:12:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdbfa7fc640 1 -- 172.21.15.60:0/3366975630 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4005310 con 0x7fdbe4004d00 2023-10-16T14:12:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdc01f48640 1 -- 172.21.15.60:0/3366975630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 msgr2=0x7fdbe4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3366975630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbe4005180 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4001e50 tx=0x7fdbe40057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdbfbfff640 1 -- 172.21.15.60:0/3366975630 reap_dead start 2023-10-16T14:12:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdc01f48640 1 -- 172.21.15.60:0/3366975630 shutdown_connections 2023-10-16T14:12:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.306+0000 7fdc01f48640 1 -- 172.21.15.60:0/3366975630 >> 172.21.15.60:0/3366975630 conn(0x7fdbfc0cfac0 msgr2=0x7fdbfc0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.307+0000 7fdc01f48640 1 -- 172.21.15.60:0/3366975630 shutdown_connections 2023-10-16T14:12:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.307+0000 7fdc01f48640 1 -- 172.21.15.60:0/3366975630 wait complete. 2023-10-16T14:12:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.309+0000 7fdc01f48640 1 Processor -- start 2023-10-16T14:12:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.309+0000 7fdc01f48640 1 -- start start 2023-10-16T14:12:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdc01f48640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbfc15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdc01f48640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc0ad7e0 0x7fdbfc0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdc01f48640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbfc15e8d0 0x7fdbfc15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdc01f48640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc15f2c0 con 0x7fdbe4004d00 2023-10-16T14:12:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdc01f48640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0b26a0 con 0x7fdbfc15e8d0 2023-10-16T14:12:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdc01f48640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0b2980 con 0x7fdbfc0ad7e0 2023-10-16T14:12:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdbfb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbfc15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdbfb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbfc15b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38714/0 (socket says 172.21.15.60:38714) 2023-10-16T14:12:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdbfb7fe640 1 -- 172.21.15.60:0/3139962592 learned_addr learned my addr 172.21.15.60:0/3139962592 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdbfaffd640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc0ad7e0 0x7fdbfc0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.310+0000 7fdbfbfff640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbfc15e8d0 0x7fdbfc15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbfb7fe640 1 -- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc0ad7e0 msgr2=0x7fdbfc0c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbfb7fe640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc0ad7e0 0x7fdbfc0c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbfb7fe640 1 -- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbfc15e8d0 msgr2=0x7fdbfc15ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbfb7fe640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbfc15e8d0 0x7fdbfc15ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbfb7fe640 1 -- 172.21.15.60:0/3139962592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0b2c60 con 0x7fdbe4004d00 2023-10-16T14:12:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbfb7fe640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbfc15b3c0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fdbec011600 tx=0x7fdbec011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00f920 con 0x7fdbe4004d00 2023-10-16T14:12:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec00fa80 con 0x7fdbe4004d00 2023-10-16T14:12:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc0b2f40 con 0x7fdbe4004d00 2023-10-16T14:12:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.311+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00fdf0 con 0x7fdbe4004d00 2023-10-16T14:12:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.312+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc15b900 con 0x7fdbe4004d00 2023-10-16T14:12:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.313+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc0cf020 con 0x7fdbe4004d00 2023-10-16T14:12:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.314+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 138) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fdbec015460 con 0x7fdbe4004d00 2023-10-16T14:12:01.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.315+0000 7fdbe3fff640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 0x7fdbe806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.315+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fdbec08a720 con 0x7fdbe4004d00 2023-10-16T14:12:01.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.315+0000 7fdbfaffd640 1 -- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 msgr2=0x7fdbe806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:01.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.315+0000 7fdbfaffd640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 0x7fdbe806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:01.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.323+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbec055080 con 0x7fdbe4004d00 2023-10-16T14:12:01.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.516+0000 7fdbfaffd640 1 -- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 msgr2=0x7fdbe806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:01.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.516+0000 7fdbfaffd640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 0x7fdbe806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:01.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.522+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fdbfc04e160 con 0x7fdbe4004d00 2023-10-16T14:12:01.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.527+0000 7fdbe3fff640 1 -- 172.21.15.60:0/3139962592 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdbec0586a0 con 0x7fdbe4004d00 2023-10-16T14:12:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 msgr2=0x7fdbe806d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 0x7fdbe806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 msgr2=0x7fdbfc15b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.534 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbfc15b3c0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fdbec011600 tx=0x7fdbec011ac0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 shutdown_connections 2023-10-16T14:12:01.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fdbe806b100 0x7fdbe806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbfc0ad7e0 0x7fdbfc0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbfc15e8d0 0x7fdbfc15ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.535 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 --2- 172.21.15.60:0/3139962592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbe4004d00 0x7fdbfc15b3c0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 >> 172.21.15.60:0/3139962592 conn(0x7fdbfc0cfac0 msgr2=0x7fdbfc0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 shutdown_connections 2023-10-16T14:12:01.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.530+0000 7fdc01f48640 1 -- 172.21.15.60:0/3139962592 wait complete. 2023-10-16T14:12:01.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:12:01.593 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:01.591+0000 7f836a62b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:12:01.647 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:01.644+0000 7f836a62b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:12:01.665 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:01.661+0000 7ffa8fa1f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:12:01.723 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:01.720+0000 7ffa8fa1f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:12:01.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.753+0000 7fa2e187e640 1 Processor -- start 2023-10-16T14:12:01.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.753+0000 7fa2e187e640 1 -- start start 2023-10-16T14:12:01.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2e187e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2dc0ad7a0 0x7fa2dc070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2e187e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc0adc30 0x7fa2dc070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2e187e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc0ae1a0 con 0x7fa2dc160c80 2023-10-16T14:12:01.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2e187e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2dc0ae480 con 0x7fa2dc0ad7a0 2023-10-16T14:12:01.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2e187e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc0ae760 con 0x7fa2dc0adc30 2023-10-16T14:12:01.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2d3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2dc0ad7a0 0x7fa2dc070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2d3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2dc0ad7a0 0x7fa2dc070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38728/0 (socket says 172.21.15.60:38728) 2023-10-16T14:12:01.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.755+0000 7fa2d3fff640 1 -- 172.21.15.60:0/2419123192 learned_addr learned my addr 172.21.15.60:0/2419123192 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.756+0000 7fa2d3fff640 1 -- 172.21.15.60:0/2419123192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc0adc30 msgr2=0x7fa2dc070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.756+0000 7fa2e107d640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc0adc30 0x7fa2dc070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.756+0000 7fa2d3fff640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc0adc30 0x7fa2dc070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.756+0000 7fa2d3fff640 1 -- 172.21.15.60:0/2419123192 >> v1:172.21.15.60:6789/0 conn(0x7fa2dc160c80 legacy=0x7fa2dc161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.756+0000 7fa2d3fff640 1 -- 172.21.15.60:0/2419123192 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc0aea40 con 0x7fa2dc0ad7a0 2023-10-16T14:12:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.756+0000 7fa2e107d640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc0adc30 0x7fa2dc070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.757+0000 7fa2d3fff640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc070240 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009870 tx=0x7fa2cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08d75294bda8aba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.757+0000 7fa2d37fe640 1 -- 172.21.15.60:0/2419123192 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc030840 con 0x7fa2dc0ad7a0 2023-10-16T14:12:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.757+0000 7fa2d37fe640 1 -- 172.21.15.60:0/2419123192 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc0309a0 con 0x7fa2dc0ad7a0 2023-10-16T14:12:01.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.757+0000 7fa2d37fe640 1 -- 172.21.15.60:0/2419123192 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc030d30 con 0x7fa2dc0ad7a0 2023-10-16T14:12:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 -- 172.21.15.60:0/2419123192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0ad7a0 msgr2=0x7fa2dc070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc070240 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009870 tx=0x7fa2cc02b980 comp rx=0 tx=0).stop 2023-10-16T14:12:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 -- 172.21.15.60:0/2419123192 shutdown_connections 2023-10-16T14:12:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc0adc30 0x7fa2dc070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 --2- 172.21.15.60:0/2419123192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc070240 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 -- 172.21.15.60:0/2419123192 >> 172.21.15.60:0/2419123192 conn(0x7fa2dc0cc230 msgr2=0x7fa2dc0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 -- 172.21.15.60:0/2419123192 shutdown_connections 2023-10-16T14:12:01.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.758+0000 7fa2e187e640 1 -- 172.21.15.60:0/2419123192 wait complete. 2023-10-16T14:12:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.760+0000 7fa2e187e640 1 Processor -- start 2023-10-16T14:12:01.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.760+0000 7fa2e187e640 1 -- start start 2023-10-16T14:12:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e187e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc0c2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e187e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 0x7fa2dc0c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e187e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc160c80 0x7fa2dc0c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e187e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc1487f0 con 0x7fa2dc0adc30 2023-10-16T14:12:01.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e187e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc148ad0 con 0x7fa2dc0ad7a0 2023-10-16T14:12:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e187e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc148db0 con 0x7fa2dc160c80 2023-10-16T14:12:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 0x7fa2dc0c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 0x7fa2dc0c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38730/0 (socket says 172.21.15.60:38730) 2023-10-16T14:12:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2d3fff640 1 -- 172.21.15.60:0/1404070467 learned_addr learned my addr 172.21.15.60:0/1404070467 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:01.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.761+0000 7fa2e107d640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc160c80 0x7fa2dc0c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2e087c640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc0c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d3fff640 1 -- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc160c80 msgr2=0x7fa2dc0c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d3fff640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc160c80 0x7fa2dc0c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d3fff640 1 -- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2dc0ad7a0 msgr2=0x7fa2dc0c2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d3fff640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc0c2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d3fff640 1 -- 172.21.15.60:0/1404070467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc0094d0 con 0x7fa2dc0adc30 2023-10-16T14:12:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2e087c640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc0c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d3fff640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 0x7fa2dc0c58d0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc000c00 tx=0x7fa2cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc030810 con 0x7fa2dc0adc30 2023-10-16T14:12:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc043400 con 0x7fa2dc0adc30 2023-10-16T14:12:01.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0437d0 con 0x7fa2dc0adc30 2023-10-16T14:12:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.762+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc149090 con 0x7fa2dc0adc30 2023-10-16T14:12:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.763+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc0c2680 con 0x7fa2dc0adc30 2023-10-16T14:12:01.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.764+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc0c2810 con 0x7fa2dc0adc30 2023-10-16T14:12:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.764+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 138) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fa2cc043930 con 0x7fa2dc0adc30 2023-10-16T14:12:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.765+0000 7fa2d17fa640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 0x7fa2c806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.766+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fa2cc03e070 con 0x7fa2dc0adc30 2023-10-16T14:12:01.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.766+0000 7fa2e087c640 1 -- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 msgr2=0x7fa2c806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.766+0000 7fa2e087c640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 0x7fa2c806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:01.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.770+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa2cc0dffe0 con 0x7fa2dc0adc30 2023-10-16T14:12:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.964+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa2dc04e9b0 con 0x7fa2dc0adc30 2023-10-16T14:12:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.967+0000 7fa2e087c640 1 -- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 msgr2=0x7fa2c806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.967+0000 7fa2e087c640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 0x7fa2c806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.970+0000 7fa2d17fa640 1 -- 172.21.15.60:0/1404070467 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa2dc0c2810 con 0x7fa2dc0adc30 2023-10-16T14:12:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.972+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 msgr2=0x7fa2c806d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.972+0000 7fa2e187e640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 0x7fa2c806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.972+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 msgr2=0x7fa2dc0c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.972+0000 7fa2e187e640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 0x7fa2dc0c58d0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc000c00 tx=0x7fa2cc004730 comp rx=0 tx=0).stop 2023-10-16T14:12:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 shutdown_connections 2023-10-16T14:12:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7fa2c806b100 0x7fa2c806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2dc160c80 0x7fa2dc0c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2dc0adc30 0x7fa2dc0c58d0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 --2- 172.21.15.60:0/1404070467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2dc0ad7a0 0x7fa2dc0c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 >> 172.21.15.60:0/1404070467 conn(0x7fa2dc0cc230 msgr2=0x7fa2dc15f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:01.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 shutdown_connections 2023-10-16T14:12:01.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:01.973+0000 7fa2e187e640 1 -- 172.21.15.60:0/1404070467 wait complete. 2023-10-16T14:12:01.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:01.996 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T14:12:02.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2023-10-16T14:12:02.139 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.136+0000 7f836a62b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:12:02.192 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.189+0000 7f836a62b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:12:02.218 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.214+0000 7ffa8fa1f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:12:02.242 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.239+0000 7f836a62b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:12:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.260+0000 7f119d6bb640 1 Processor -- start 2023-10-16T14:12:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.260+0000 7f119d6bb640 1 -- start start 2023-10-16T14:12:02.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.262+0000 7f119d6bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f119815d410 0x7f11980d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f119d6bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11980b5d90 con 0x7f11980cd260 2023-10-16T14:12:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f119d6bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1198160fc0 con 0x7f119815d030 2023-10-16T14:12:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f119d6bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11980d2b50 con 0x7f119815d410 2023-10-16T14:12:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f11977fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f11980cd260 0x7f11980b4720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47868/0 (socket says 172.21.15.60:47868) 2023-10-16T14:12:02.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f11977fe640 1 -- 172.21.15.60:0/2595515981 learned_addr learned my addr 172.21.15.60:0/2595515981 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:02.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f118e7fc640 1 --2- 172.21.15.60:0/2595515981 >> v2:172.21.15.144:3300/0 conn(0x7f119815d410 0x7f11980d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.263+0000 7f11967fc640 1 -- 172.21.15.60:0/2595515981 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850945856 0 0) 0x7f11980b5d90 con 0x7f11980cd260 2023-10-16T14:12:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.264+0000 7f11967fc640 1 -- 172.21.15.60:0/2595515981 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1184003520 con 0x7f11980cd260 2023-10-16T14:12:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.264+0000 7f118e7fc640 1 -- 172.21.15.60:0/2595515981 >> v1:172.21.15.144:6789/0 conn(0x7f119815d030 legacy=0x7f1198161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.264+0000 7f11967fc640 1 -- 172.21.15.60:0/2595515981 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 687683667 0 0) 0x7f1184003520 con 0x7f11980cd260 2023-10-16T14:12:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.264+0000 7f118e7fc640 1 -- 172.21.15.60:0/2595515981 >> v1:172.21.15.60:6789/0 conn(0x7f11980cd260 legacy=0x7f11980b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.264+0000 7f118e7fc640 1 -- 172.21.15.60:0/2595515981 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11980d25d0 con 0x7f119815d410 2023-10-16T14:12:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.265+0000 7f118e7fc640 1 --2- 172.21.15.60:0/2595515981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 0x7f11980d1630 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f11800097a0 tx=0x7f118002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=274ee8bb3de7a2ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.266+0000 7f11967fc640 1 -- 172.21.15.60:0/2595515981 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190001670 con 0x7f119815d410 2023-10-16T14:12:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.266+0000 7f11967fc640 1 -- 172.21.15.60:0/2595515981 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1180031590 con 0x7f119815d410 2023-10-16T14:12:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.266+0000 7f11967fc640 1 -- 172.21.15.60:0/2595515981 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1180031920 con 0x7f119815d410 2023-10-16T14:12:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.266+0000 7f119d6bb640 1 -- 172.21.15.60:0/2595515981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 msgr2=0x7f11980d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.266+0000 7f119d6bb640 1 --2- 172.21.15.60:0/2595515981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 0x7f11980d1630 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f11800097a0 tx=0x7f118002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:12:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.267+0000 7f119d6bb640 1 -- 172.21.15.60:0/2595515981 shutdown_connections 2023-10-16T14:12:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.267+0000 7f119d6bb640 1 --2- 172.21.15.60:0/2595515981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 0x7f11980d1630 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.267+0000 7f119d6bb640 1 -- 172.21.15.60:0/2595515981 >> 172.21.15.60:0/2595515981 conn(0x7f11980cf720 msgr2=0x7f11980cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.267+0000 7f119d6bb640 1 -- 172.21.15.60:0/2595515981 shutdown_connections 2023-10-16T14:12:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.267+0000 7f119d6bb640 1 -- 172.21.15.60:0/2595515981 wait complete. 2023-10-16T14:12:02.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.269+0000 7f119d6bb640 1 Processor -- start 2023-10-16T14:12:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.269+0000 7f119d6bb640 1 -- start start 2023-10-16T14:12:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f119d6bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11980cd260 0x7f119815e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f119d6bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f119815d030 0x7f119814f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:02.279 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.271+0000 7ffa8fa1f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:12:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f119d6bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 0x7f11980b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f119d6bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119814f850 con 0x7f11980cd260 2023-10-16T14:12:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f119d6bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11980b38e0 con 0x7f119815d410 2023-10-16T14:12:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f119d6bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11980b3b80 con 0x7f119815d030 2023-10-16T14:12:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11980cd260 0x7f119815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11980cd260 0x7f119815e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38742/0 (socket says 172.21.15.60:38742) 2023-10-16T14:12:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 -- 172.21.15.60:0/1137757277 learned_addr learned my addr 172.21.15.60:0/1137757277 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f119815d030 0x7f119814f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f11977fe640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 0x7f11980b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f119815d030 msgr2=0x7f119814f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:02.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f119815d030 0x7f119814f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 msgr2=0x7f11980b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f119815d410 0x7f11980b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.271+0000 7f1196ffd640 1 -- 172.21.15.60:0/1137757277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1180009400 con 0x7f11980cd260 2023-10-16T14:12:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.272+0000 7f1196ffd640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11980cd260 0x7f119815e250 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f118800c870 tx=0x7f118800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:02.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.272+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188003b70 con 0x7f11980cd260 2023-10-16T14:12:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.272+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1188003cf0 con 0x7f11980cd260 2023-10-16T14:12:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.272+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118800aa90 con 0x7f11980cd260 2023-10-16T14:12:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.272+0000 7f119d6bb640 1 -- 172.21.15.60:0/1137757277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11980b3e60 con 0x7f11980cd260 2023-10-16T14:12:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.272+0000 7f119d6bb640 1 -- 172.21.15.60:0/1137757277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11980b4420 con 0x7f11980cd260 2023-10-16T14:12:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.274+0000 7f119d6bb640 1 -- 172.21.15.60:0/1137757277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119804e160 con 0x7f11980cd260 2023-10-16T14:12:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.275+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 138) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f118800abf0 con 0x7f11980cd260 2023-10-16T14:12:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.275+0000 7f118ffff640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 0x7f117806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.276+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f1188089d70 con 0x7f11980cd260 2023-10-16T14:12:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.276+0000 7f118e7fc640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 msgr2=0x7f117806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:02.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.276+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 0x7f117806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:02.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.279+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11880546c0 con 0x7f11980cd260 2023-10-16T14:12:02.304 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.301+0000 7f836a62b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:12:02.329 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.325+0000 7ffa8fa1f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:12:02.353 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.351+0000 7f836a62b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:12:02.395 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.391+0000 7ffa8fa1f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:12:02.447 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.443+0000 7ffa8fa1f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:12:02.463 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.460+0000 7f836a62b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:12:02.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.477+0000 7f118e7fc640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 msgr2=0x7f117806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:02.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.477+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 0x7f117806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:02.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.498+0000 7f119d6bb640 1 -- 172.21.15.60:0/1137757277 --> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f11980d2870 con 0x7f117806b100 2023-10-16T14:12:02.524 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.522+0000 7f836a62b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:02.566 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.562+0000 7ffa8fa1f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:12:02.574 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.572+0000 7f836a62b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:12:02.630 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.627+0000 7f836a62b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:12:02.632 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.628+0000 7ffa8fa1f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:02.679 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:02.677+0000 7f836a62b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:12:02.684 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.680+0000 7ffa8fa1f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:12:02.743 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.740+0000 7ffa8fa1f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:12:02.796 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:02.793+0000 7ffa8fa1f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:12:02.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.878+0000 7f118e7fc640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 msgr2=0x7f117806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:02.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:02.878+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 0x7f117806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:12:03.065 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.061+0000 7f836a62b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:12:03.207 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.203+0000 7ffa8fa1f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:12:03.222 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.218+0000 7f836a62b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:12:03.328 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.324+0000 7f836a62b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:12:03.377 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.374+0000 7ffa8fa1f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:12:03.378 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.374+0000 7f836a62b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:12:03.484 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.480+0000 7f836a62b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:03.491 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.488+0000 7ffa8fa1f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:12:03.545 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.541+0000 7ffa8fa1f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:12:03.589 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.585+0000 7f836a62b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:12:03.659 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.656+0000 7ffa8fa1f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:03.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:03.679+0000 7f118e7fc640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 msgr2=0x7f117806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/4052124783 2023-10-16T14:12:03.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:03.679+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 0x7f117806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:12:03.744 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.740+0000 7f836a62b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:03.774 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.770+0000 7ffa8fa1f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:12:03.792 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.789+0000 7f836a62b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:03.911 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:03.907+0000 7f836a62b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:12:03.941 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.937+0000 7ffa8fa1f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:03.994 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:03.990+0000 7ffa8fa1f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:04.120 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:04.116+0000 7ffa8fa1f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:12:04.746 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:04.742+0000 7f836a62b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:12:05.003 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:04.999+0000 7ffa8fa1f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:12:05.041 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.037+0000 7f836a62b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:12:05.099 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.095+0000 7f836a62b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:12:05.211 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.207+0000 7f836a62b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:12:05.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:05.228+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 139) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f1188010dc0 con 0x7f11980cd260 2023-10-16T14:12:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:05.229+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 msgr2=0x7f117806d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:05.229+0000 7f118ffff640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/4052124783,v1:172.21.15.144:6833/4052124783] conn(0x7f117806b100 0x7f117806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:05.329 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:05.325+0000 7ffa8fa1f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:12:05.390 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:05.387+0000 7ffa8fa1f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:12:05.521 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:05.517+0000 7ffa8fa1f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:12:05.647 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.643+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.643+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.643+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.643+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.648 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.643+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.644+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.644+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.644+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.644+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.649 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.644+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.665 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.665 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.665 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e61fd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.666 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.667 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:05.667 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:05.661+0000 7f82e8a02640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.237+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 140) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f11880561a0 con 0x7f11980cd260 2023-10-16T14:12:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.238+0000 7f118ffff640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f1178035800 0x7f11780775f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.238+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 --> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f11780283b0 con 0x7f1178035800 2023-10-16T14:12:06.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.240+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f1178035800 0x7f11780775f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:06.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.259+0000 7f118e7fc640 1 --2- 172.21.15.60:0/1137757277 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f1178035800 0x7f11780775f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1180036000 tx=0x7f118003d040 comp rx=0 tx=0).ready entity=mgr.8312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:06.269 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:06.266+0000 7f82f0251640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2023-10-16T14:12:06.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.266+0000 7f118ffff640 1 -- 172.21.15.60:0/1137757277 <== mgr.8312 v2:172.21.15.144:6832/529676808 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7f11780283b0 con 0x7f1178035800 2023-10-16T14:12:06.270 INFO:teuthology.orchestra.run.smithi060.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2023-10-16T14:12:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T14:12:06.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:06.296 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T14:12:06.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:12:06.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.586+0000 7f52de07c640 1 Processor -- start 2023-10-16T14:12:06.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.586+0000 7f52de07c640 1 -- start start 2023-10-16T14:12:06.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.587+0000 7f52de07c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d8159e90 0x7f52d80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.587+0000 7f52de07c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52d806c140 0x7f52d806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.587+0000 7f52de07c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d8070360 con 0x7f52d8159ab0 2023-10-16T14:12:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.587+0000 7f52de07c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d806cb10 con 0x7f52d806c140 2023-10-16T14:12:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.587+0000 7f52de07c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d815d890 con 0x7f52d8159e90 2023-10-16T14:12:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52d806c140 0x7f52d806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52d806c140 0x7f52d806c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38746/0 (socket says 172.21.15.60:38746) 2023-10-16T14:12:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 -- 172.21.15.60:0/562400426 learned_addr learned my addr 172.21.15.60:0/562400426 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:06.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d6ffd640 1 --2- 172.21.15.60:0/562400426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d8159e90 0x7f52d80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 -- 172.21.15.60:0/562400426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d8159e90 msgr2=0x7f52d80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 --2- 172.21.15.60:0/562400426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d8159e90 0x7f52d80ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 -- 172.21.15.60:0/562400426 >> v1:172.21.15.144:6789/0 conn(0x7f52d8159ab0 legacy=0x7f52d806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.588+0000 7f52d7fff640 1 -- 172.21.15.60:0/562400426 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d80cd1f0 con 0x7f52d806c140 2023-10-16T14:12:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.590+0000 7f52d7fff640 1 --2- 172.21.15.60:0/562400426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d806c140 0x7f52d806c5d0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f52d00098a0 tx=0x7f52d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea068f0b811bfe03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.590+0000 7f52d67fc640 1 -- 172.21.15.60:0/562400426 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00317c0 con 0x7f52d806c140 2023-10-16T14:12:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.590+0000 7f52d67fc640 1 -- 172.21.15.60:0/562400426 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d0031920 con 0x7f52d806c140 2023-10-16T14:12:06.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.590+0000 7f52d67fc640 1 -- 172.21.15.60:0/562400426 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0031cb0 con 0x7f52d806c140 2023-10-16T14:12:06.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 -- 172.21.15.60:0/562400426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d806c140 msgr2=0x7f52d806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:06.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 --2- 172.21.15.60:0/562400426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d806c140 0x7f52d806c5d0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f52d00098a0 tx=0x7f52d002b860 comp rx=0 tx=0).stop 2023-10-16T14:12:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 -- 172.21.15.60:0/562400426 shutdown_connections 2023-10-16T14:12:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 --2- 172.21.15.60:0/562400426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d806c140 0x7f52d806c5d0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 --2- 172.21.15.60:0/562400426 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d8159e90 0x7f52d80ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 -- 172.21.15.60:0/562400426 >> 172.21.15.60:0/562400426 conn(0x7f52d80ce9b0 msgr2=0x7f52d80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:06.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 -- 172.21.15.60:0/562400426 shutdown_connections 2023-10-16T14:12:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.591+0000 7f52de07c640 1 -- 172.21.15.60:0/562400426 wait complete. 2023-10-16T14:12:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.593+0000 7f52de07c640 1 Processor -- start 2023-10-16T14:12:06.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.593+0000 7f52de07c640 1 -- start start 2023-10-16T14:12:06.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52de07c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d806c140 0x7f52d8158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52de07c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 0x7f52d8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52de07c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52d8159e90 0x7f52d81e66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52de07c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d80c2820 con 0x7f52d8159ab0 2023-10-16T14:12:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52de07c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d81e6c30 con 0x7f52d8159e90 2023-10-16T14:12:06.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52de07c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d80c3ca0 con 0x7f52d806c140 2023-10-16T14:12:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 0x7f52d8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 0x7f52d8157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38760/0 (socket says 172.21.15.60:38760) 2023-10-16T14:12:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.594+0000 7f52d6ffd640 1 -- 172.21.15.60:0/1195551417 learned_addr learned my addr 172.21.15.60:0/1195551417 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d7fff640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52d8159e90 0x7f52d81e66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:06.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d6ffd640 1 -- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d806c140 msgr2=0x7f52d8158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d6ffd640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d806c140 0x7f52d8158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d6ffd640 1 -- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52d8159e90 msgr2=0x7f52d81e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d6ffd640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52d8159e90 0x7f52d81e66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d6ffd640 1 -- 172.21.15.60:0/1195551417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00094d0 con 0x7f52d8159ab0 2023-10-16T14:12:06.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52d6ffd640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 0x7f52d8157a90 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f52c000d830 tx=0x7f52c000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:06.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000b6c0 con 0x7f52d8159ab0 2023-10-16T14:12:06.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c000b840 con 0x7f52d8159ab0 2023-10-16T14:12:06.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000bbf0 con 0x7f52d8159ab0 2023-10-16T14:12:06.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.595+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d80c4820 con 0x7f52d8159ab0 2023-10-16T14:12:06.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.596+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d80c41d0 con 0x7f52d8159ab0 2023-10-16T14:12:06.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.597+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d80cd790 con 0x7f52d8159ab0 2023-10-16T14:12:06.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.598+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 140) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f52c001e460 con 0x7f52d8159ab0 2023-10-16T14:12:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.599+0000 7f52cffff640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f52b806ad10 0x7f52b806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.599+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f52c008a090 con 0x7f52d8159ab0 2023-10-16T14:12:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.599+0000 7f52d77fe640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f52b806ad10 0x7f52b806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:06.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.601+0000 7f52d77fe640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f52b806ad10 0x7f52b806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52c80012e0 tx=0x7f52c8007d30 comp rx=0 tx=0).ready entity=mgr.8312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:06.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.603+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52d80cd790 con 0x7f52d8159ab0 2023-10-16T14:12:06.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:06.838+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f52d80cd790 con 0x7f52d8159ab0 2023-10-16T14:12:07.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.286+0000 7f52cffff640 1 -- 172.21.15.60:0/1195551417 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v141) v1 ==== 93+0+0 (secure 0 0 0) 0x7f52d80cd790 con 0x7f52d8159ab0 2023-10-16T14:12:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.292+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f52b806ad10 msgr2=0x7f52b806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.292+0000 7f52de07c640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f52b806ad10 0x7f52b806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52c80012e0 tx=0x7f52c8007d30 comp rx=0 tx=0).stop 2023-10-16T14:12:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.292+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 msgr2=0x7f52d8157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.292+0000 7f52de07c640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 0x7f52d8157a90 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f52c000d830 tx=0x7f52c000dcf0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 shutdown_connections 2023-10-16T14:12:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f52b806ad10 0x7f52b806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52d8159e90 0x7f52d81e66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52d8159ab0 0x7f52d8157a90 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 --2- 172.21.15.60:0/1195551417 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52d806c140 0x7f52d8158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 >> 172.21.15.60:0/1195551417 conn(0x7f52d80ce9b0 msgr2=0x7f52d80b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.293+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 shutdown_connections 2023-10-16T14:12:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.294+0000 7f52de07c640 1 -- 172.21.15.60:0/1195551417 wait complete. 2023-10-16T14:12:07.318 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:12:07.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:12:07.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:12:07.531 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:12:07.532 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:12:07.532 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:07.587 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:12:07.587 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:07.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:07.824 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:07.820+0000 7f37f8c2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:12:07.836 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:07.833+0000 7fec3e83f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:12:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.846+0000 7f79027e2640 1 Processor -- start 2023-10-16T14:12:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.846+0000 7f79027e2640 1 -- start start 2023-10-16T14:12:07.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f79027e2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f78fc0b8480 0x7f78fc0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f79027e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc15d150 con 0x7f78fc0d0be0 2023-10-16T14:12:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f79027e2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc0d1180 con 0x7f78fc15cd70 2023-10-16T14:12:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f79027e2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc0d1460 con 0x7f78fc0b8480 2023-10-16T14:12:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f7901fe1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f78fc0d0be0 0x7f78fc0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47886/0 (socket says 172.21.15.60:47886) 2023-10-16T14:12:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f7901fe1640 1 -- 172.21.15.60:0/2957440265 learned_addr learned my addr 172.21.15.60:0/2957440265 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.848+0000 7f7900fdf640 1 --2- 172.21.15.60:0/2957440265 >> v2:172.21.15.144:3300/0 conn(0x7f78fc0b8480 0x7f78fc0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733142968 0 0) 0x7f78fc15d150 con 0x7f78fc0d0be0 2023-10-16T14:12:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e8003520 con 0x7f78fc0d0be0 2023-10-16T14:12:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3333604381 0 0) 0x7f78e8003520 con 0x7f78fc0d0be0 2023-10-16T14:12:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 >> v2:172.21.15.144:3300/0 conn(0x7f78fc0b8480 msgr2=0x7f78fc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 --2- 172.21.15.60:0/2957440265 >> v2:172.21.15.144:3300/0 conn(0x7f78fc0b8480 0x7f78fc0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 >> v1:172.21.15.144:6789/0 conn(0x7f78fc15cd70 legacy=0x7f78fc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0d1740 con 0x7f78fc0d0be0 2023-10-16T14:12:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f78f40019f0 con 0x7f78fc0d0be0 2023-10-16T14:12:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 >> v1:172.21.15.60:6789/0 conn(0x7f78fc0d0be0 legacy=0x7f78fc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78e80047c0 0x7f78e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78e8005180 0x7f78e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0d1460 con 0x7f78e8003970 2023-10-16T14:12:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e800c120 con 0x7f78e8005180 2023-10-16T14:12:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e800c2e0 con 0x7f78e80047c0 2023-10-16T14:12:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.849+0000 7f79017e0640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.850+0000 7f79017e0640 1 -- 172.21.15.60:0/2957440265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78e80047c0 msgr2=0x7f78e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.850+0000 7f79017e0640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78e80047c0 0x7f78e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.850+0000 7f79017e0640 1 -- 172.21.15.60:0/2957440265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78e8005180 msgr2=0x7f78e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.850+0000 7f79017e0640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78e8005180 0x7f78e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.850+0000 7f79017e0640 1 -- 172.21.15.60:0/2957440265 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e8003520 con 0x7f78e8003970 2023-10-16T14:12:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.850+0000 7f7901fe1640 1 -- 172.21.15.60:0/2957440265 reap_dead start 2023-10-16T14:12:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.851+0000 7f79017e0640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78e8003d90 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f78f8001750 tx=0x7f78e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1122a2afad6f440 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.851+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f80052d0 con 0x7f78e8003970 2023-10-16T14:12:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.851+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f802e050 con 0x7f78e8003970 2023-10-16T14:12:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.851+0000 7f78f3fff640 1 -- 172.21.15.60:0/2957440265 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8003830 con 0x7f78e8003970 2023-10-16T14:12:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 -- 172.21.15.60:0/2957440265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 msgr2=0x7f78e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78e8003d90 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f78f8001750 tx=0x7f78e80057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 -- 172.21.15.60:0/2957440265 shutdown_connections 2023-10-16T14:12:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 --2- 172.21.15.60:0/2957440265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78e8003d90 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 -- 172.21.15.60:0/2957440265 >> 172.21.15.60:0/2957440265 conn(0x7f78fc0cf570 msgr2=0x7f78fc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 -- 172.21.15.60:0/2957440265 shutdown_connections 2023-10-16T14:12:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.852+0000 7f79027e2640 1 -- 172.21.15.60:0/2957440265 wait complete. 2023-10-16T14:12:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.855+0000 7f79027e2640 1 Processor -- start 2023-10-16T14:12:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.855+0000 7f79027e2640 1 -- start start 2023-10-16T14:12:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.856+0000 7f79027e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78fc144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.856+0000 7f79027e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc145440 0x7f78fc15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.856+0000 7f79027e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc15e2d0 0x7f78fc0cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.856+0000 7f79027e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc15f6b0 con 0x7f78e8003970 2023-10-16T14:12:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.856+0000 7f79027e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc1458a0 con 0x7f78fc145440 2023-10-16T14:12:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.856+0000 7f79027e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc15ea30 con 0x7f78fc15e2d0 2023-10-16T14:12:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78fc144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78fc144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38790/0 (socket says 172.21.15.60:38790) 2023-10-16T14:12:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 -- 172.21.15.60:0/3858338364 learned_addr learned my addr 172.21.15.60:0/3858338364 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f7901fe1640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc15e2d0 0x7f78fc0cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f7900fdf640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc145440 0x7f78fc15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 -- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc15e2d0 msgr2=0x7f78fc0cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc15e2d0 0x7f78fc0cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 -- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc145440 msgr2=0x7f78fc15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc145440 0x7f78fc15dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f79017e0640 1 -- 172.21.15.60:0/3858338364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc15ed10 con 0x7f78e8003970 2023-10-16T14:12:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.857+0000 7f7901fe1640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc15e2d0 0x7f78fc0cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.858+0000 7f79017e0640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78fc144f00 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005040 tx=0x7f78f8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.858+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f80076c0 con 0x7f78e8003970 2023-10-16T14:12:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.858+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f8007a30 con 0x7f78e8003970 2023-10-16T14:12:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.858+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8006c20 con 0x7f78e8003970 2023-10-16T14:12:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.858+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc15eff0 con 0x7f78e8003970 2023-10-16T14:12:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.858+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc15e730 con 0x7f78e8003970 2023-10-16T14:12:07.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.859+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc04e9b0 con 0x7f78e8003970 2023-10-16T14:12:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.861+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f78f80035f0 con 0x7f78e8003970 2023-10-16T14:12:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.862+0000 7f78f1ffb640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 0x7f78e006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.862+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f78f80abd90 con 0x7f78e8003970 2023-10-16T14:12:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.862+0000 7f7900fdf640 1 -- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 msgr2=0x7f78e006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.862+0000 7f7900fdf640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 0x7f78e006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:07.866+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78f8076580 con 0x7f78e8003970 2023-10-16T14:12:07.878 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:07.874+0000 7f37f8c2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:12:07.898 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:07.894+0000 7fec3e83f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:12:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.063+0000 7f7900fdf640 1 -- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 msgr2=0x7f78e006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.063+0000 7f7900fdf640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 0x7f78e006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:08.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.066+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78fc0c8ec0 con 0x7f78e8003970 2023-10-16T14:12:08.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.070+0000 7f78f1ffb640 1 -- 172.21.15.60:0/3858338364 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+20809 (secure 0 0 0) 0x7f78f804b070 con 0x7f78e8003970 2023-10-16T14:12:08.075 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:08.075 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":257,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:05.212787+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 msgr2=0x7f78e006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 0x7f78e006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 msgr2=0x7f78fc144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78fc144f00 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005040 tx=0x7f78f8001120 comp rx=0 tx=0).stop 2023-10-16T14:12:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 shutdown_connections 2023-10-16T14:12:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f78e006b0b0 0x7f78e006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f78fc15e2d0 0x7f78fc0cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f78fc145440 0x7f78fc15dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 --2- 172.21.15.60:0/3858338364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e8003970 0x7f78fc144f00 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 >> 172.21.15.60:0/3858338364 conn(0x7f78fc0cf570 msgr2=0x7f78fc0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 shutdown_connections 2023-10-16T14:12:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.073+0000 7f79027e2640 1 -- 172.21.15.60:0/3858338364 wait complete. 2023-10-16T14:12:08.098 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:11:36.185089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:11:37.268518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T14:11:50.763838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T14:11:51.462135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:12:08.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:12:08.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.312+0000 7f311282e640 1 Processor -- start 2023-10-16T14:12:08.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.312+0000 7f311282e640 1 -- start start 2023-10-16T14:12:08.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311282e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b3ec0 0x7f310c0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311282e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f310c0b43c0 0x7f310c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311282e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c06be30 con 0x7f310c0b0ea0 2023-10-16T14:12:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311282e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c06ca00 con 0x7f310c0b43c0 2023-10-16T14:12:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311282e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0ada10 con 0x7f310c0b3ec0 2023-10-16T14:12:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311102b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f310c0b0ea0 0x7f310c15a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47894/0 (socket says 172.21.15.60:47894) 2023-10-16T14:12:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311102b640 1 -- 172.21.15.60:0/1564176249 learned_addr learned my addr 172.21.15.60:0/1564176249 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:08.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311182c640 1 --2- 172.21.15.60:0/1564176249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b3ec0 0x7f310c0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.314+0000 7f311202d640 1 --2- 172.21.15.60:0/1564176249 >> v2:172.21.15.144:3300/0 conn(0x7f310c0b43c0 0x7f310c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311082a640 1 -- 172.21.15.60:0/1564176249 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600089312 0 0) 0x7f310c06be30 con 0x7f310c0b0ea0 2023-10-16T14:12:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311082a640 1 -- 172.21.15.60:0/1564176249 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f8003520 con 0x7f310c0b0ea0 2023-10-16T14:12:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311082a640 1 -- 172.21.15.60:0/1564176249 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911754155 0 0) 0x7f30f8003520 con 0x7f310c0b0ea0 2023-10-16T14:12:08.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311182c640 1 -- 172.21.15.60:0/1564176249 >> v2:172.21.15.144:3300/0 conn(0x7f310c0b43c0 msgr2=0x7f310c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:08.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311182c640 1 --2- 172.21.15.60:0/1564176249 >> v2:172.21.15.144:3300/0 conn(0x7f310c0b43c0 0x7f310c0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311182c640 1 -- 172.21.15.60:0/1564176249 >> v1:172.21.15.60:6789/0 conn(0x7f310c0b0ea0 legacy=0x7f310c15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311182c640 1 -- 172.21.15.60:0/1564176249 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c06c060 con 0x7f310c0b3ec0 2023-10-16T14:12:08.322 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.315+0000 7f311202d640 1 --2- 172.21.15.60:0/1564176249 >> v2:172.21.15.144:3300/0 conn(0x7f310c0b43c0 0x7f310c0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.316+0000 7f311182c640 1 --2- 172.21.15.60:0/1564176249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b3ec0 0x7f310c0cd260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f31080097a0 tx=0x7f310802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f747f92350fa39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.317+0000 7f311082a640 1 -- 172.21.15.60:0/1564176249 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc001670 con 0x7f310c0b3ec0 2023-10-16T14:12:08.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.317+0000 7f311082a640 1 -- 172.21.15.60:0/1564176249 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310802e050 con 0x7f310c0b3ec0 2023-10-16T14:12:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.317+0000 7f311082a640 1 -- 172.21.15.60:0/1564176249 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108030710 con 0x7f310c0b3ec0 2023-10-16T14:12:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 -- 172.21.15.60:0/1564176249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b3ec0 msgr2=0x7f310c0cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 --2- 172.21.15.60:0/1564176249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b3ec0 0x7f310c0cd260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f31080097a0 tx=0x7f310802b710 comp rx=0 tx=0).stop 2023-10-16T14:12:08.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 -- 172.21.15.60:0/1564176249 shutdown_connections 2023-10-16T14:12:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 --2- 172.21.15.60:0/1564176249 >> v2:172.21.15.144:3300/0 conn(0x7f310c0b43c0 0x7f310c0cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 --2- 172.21.15.60:0/1564176249 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b3ec0 0x7f310c0cd260 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 -- 172.21.15.60:0/1564176249 >> 172.21.15.60:0/1564176249 conn(0x7f310c0b0170 msgr2=0x7f310c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 -- 172.21.15.60:0/1564176249 shutdown_connections 2023-10-16T14:12:08.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.318+0000 7f311282e640 1 -- 172.21.15.60:0/1564176249 wait complete. 2023-10-16T14:12:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.320+0000 7f311282e640 1 Processor -- start 2023-10-16T14:12:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.320+0000 7f311282e640 1 -- start start 2023-10-16T14:12:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311282e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b0ea0 0x7f310c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311282e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f310c0b3ec0 0x7f310c0c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311282e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 0x7f310c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311282e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c146bc0 con 0x7f310c0b43c0 2023-10-16T14:12:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311282e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c1462b0 con 0x7f310c0b3ec0 2023-10-16T14:12:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311282e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c1491b0 con 0x7f310c0b0ea0 2023-10-16T14:12:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311202d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 0x7f310c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311202d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 0x7f310c0c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38794/0 (socket says 172.21.15.60:38794) 2023-10-16T14:12:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311202d640 1 -- 172.21.15.60:0/705765256 learned_addr learned my addr 172.21.15.60:0/705765256 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311182c640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b0ea0 0x7f310c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.321+0000 7f311102b640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f310c0b3ec0 0x7f310c0c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311202d640 1 -- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b0ea0 msgr2=0x7f310c148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311202d640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b0ea0 0x7f310c148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311202d640 1 -- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f310c0b3ec0 msgr2=0x7f310c0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311202d640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f310c0b3ec0 0x7f310c0c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311202d640 1 -- 172.21.15.60:0/705765256 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104009a70 con 0x7f310c0b43c0 2023-10-16T14:12:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311102b640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f310c0b3ec0 0x7f310c0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311202d640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 0x7f310c0c9d00 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f310400efc0 tx=0x7f310400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:08.332 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.327+0000 7f37f8c2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:12:08.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104014480 con 0x7f310c0b43c0 2023-10-16T14:12:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400f040 con 0x7f310c0b43c0 2023-10-16T14:12:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3108009400 con 0x7f310c0b43c0 2023-10-16T14:12:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104014840 con 0x7f310c0b43c0 2023-10-16T14:12:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.322+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c146500 con 0x7f310c0b43c0 2023-10-16T14:12:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.323+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c146d00 con 0x7f310c0b43c0 2023-10-16T14:12:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.324+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f31040149a0 con 0x7f310c0b43c0 2023-10-16T14:12:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.325+0000 7f31027fc640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f30f006b080 0x7f30f006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.325+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f310408b0d0 con 0x7f310c0b43c0 2023-10-16T14:12:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.326+0000 7f311182c640 1 -- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f30f006b080 msgr2=0x7f30f006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.326+0000 7f311182c640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f30f006b080 0x7f30f006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.330+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f310c146d00 con 0x7f310c0b43c0 2023-10-16T14:12:08.382 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.379+0000 7fec3e83f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:12:08.383 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.379+0000 7f37f8c2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:12:08.433 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.429+0000 7f37f8c2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:12:08.438 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.434+0000 7fec3e83f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:12:08.491 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.487+0000 7fec3e83f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:12:08.493 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.489+0000 7f37f8c2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:12:08.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.522+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f310c146d00 con 0x7f310c0b43c0 2023-10-16T14:12:08.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.523+0000 7f31027fc640 1 -- 172.21.15.60:0/705765256 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v257) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3104018020 con 0x7f310c0b43c0 2023-10-16T14:12:08.527 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:12:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f30f006b080 msgr2=0x7f30f006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f30f006b080 0x7f30f006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 msgr2=0x7f310c0c9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 0x7f310c0c9d00 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f310400efc0 tx=0x7f310400c490 comp rx=0 tx=0).stop 2023-10-16T14:12:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 shutdown_connections 2023-10-16T14:12:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f30f006b080 0x7f30f006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c0b43c0 0x7f310c0c9d00 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f310c0b3ec0 0x7f310c0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 --2- 172.21.15.60:0/705765256 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f310c0b0ea0 0x7f310c148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.526+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 >> 172.21.15.60:0/705765256 conn(0x7f310c0b0170 msgr2=0x7f310c0af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.527+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 shutdown_connections 2023-10-16T14:12:08.532 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.527+0000 7f311282e640 1 -- 172.21.15.60:0/705765256 wait complete. 2023-10-16T14:12:08.542 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.538+0000 7f37f8c2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:12:08.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:12:08.559 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.555+0000 7fec3e83f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:12:08.611 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.608+0000 7fec3e83f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:12:08.651 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.647+0000 7f37f8c2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:12:08.713 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.709+0000 7f37f8c2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:08.730 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.727+0000 7fec3e83f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:12:08.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.748+0000 7fcba3021640 1 Processor -- start 2023-10-16T14:12:08.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.748+0000 7fcba3021640 1 -- start start 2023-10-16T14:12:08.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.750+0000 7fcba3021640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb9c06d5a0 0x7fcb9c0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.750+0000 7fcba3021640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.750+0000 7fcba3021640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c0ac1c0 con 0x7fcb9c0710e0 2023-10-16T14:12:08.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.750+0000 7fcba3021640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c159fb0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.750+0000 7fcba3021640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0b4d80 con 0x7fcb9c0abd40 2023-10-16T14:12:08.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.751+0000 7fcba201f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb9c06d5a0 0x7fcb9c0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.751+0000 7fcba181e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.751+0000 7fcba181e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c0ae180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:46536/0 (socket says 172.21.15.60:46536) 2023-10-16T14:12:08.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.751+0000 7fcba181e640 1 -- 172.21.15.60:0/1463562631 learned_addr learned my addr 172.21.15.60:0/1463562631 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:08.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.751+0000 7fcba201f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcb9c06d5a0 0x7fcb9c0b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:53146/0 (socket says 172.21.15.60:53146) 2023-10-16T14:12:08.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.751+0000 7fcba201f640 1 -- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 msgr2=0x7fcb9c0ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:08.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.752+0000 7fcba201f640 1 --2- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c0ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.752+0000 7fcba201f640 1 -- 172.21.15.60:0/1463562631 >> v1:172.21.15.144:6789/0 conn(0x7fcb9c0710e0 legacy=0x7fcb9c06fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.752+0000 7fcba201f640 1 -- 172.21.15.60:0/1463562631 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c15e0a0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.753+0000 7fcba201f640 1 --2- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c0b4530 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800ba40 tx=0x7fcb9802e080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42b645b68e5dd548 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:08.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.753+0000 7fcba101d640 1 -- 172.21.15.60:0/1463562631 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9802ee30 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.753+0000 7fcba101d640 1 -- 172.21.15.60:0/1463562631 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb98036440 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.753+0000 7fcba101d640 1 -- 172.21.15.60:0/1463562631 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb980367d0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 -- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c06d5a0 msgr2=0x7fcb9c0b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 --2- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c0b4530 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800ba40 tx=0x7fcb9802e080 comp rx=0 tx=0).stop 2023-10-16T14:12:08.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 -- 172.21.15.60:0/1463562631 shutdown_connections 2023-10-16T14:12:08.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 --2- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c0ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 --2- 172.21.15.60:0/1463562631 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c0b4530 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 -- 172.21.15.60:0/1463562631 >> 172.21.15.60:0/1463562631 conn(0x7fcb9c0b0a80 msgr2=0x7fcb9c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 -- 172.21.15.60:0/1463562631 shutdown_connections 2023-10-16T14:12:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.754+0000 7fcba3021640 1 -- 172.21.15.60:0/1463562631 wait complete. 2023-10-16T14:12:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.756+0000 7fcba3021640 1 Processor -- start 2023-10-16T14:12:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.756+0000 7fcba3021640 1 -- start start 2023-10-16T14:12:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba3021640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c09c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba3021640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c0710e0 0x7fcb9c15a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba3021640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba3021640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c15bb30 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba3021640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c155e00 con 0x7fcb9c0710e0 2023-10-16T14:12:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba3021640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c1f28f0 con 0x7fcb9c0abd40 2023-10-16T14:12:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba201f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c09c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba201f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c09c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38798/0 (socket says 172.21.15.60:38798) 2023-10-16T14:12:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.757+0000 7fcba201f640 1 -- 172.21.15.60:0/3899258202 learned_addr learned my addr 172.21.15.60:0/3899258202 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba181e640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c0710e0 0x7fcb9c15a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba2820640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba201f640 1 -- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 msgr2=0x7fcb9c1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba201f640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba201f640 1 -- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c0710e0 msgr2=0x7fcb9c15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba201f640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c0710e0 0x7fcb9c15a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba201f640 1 -- 172.21.15.60:0/3899258202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9800b6a0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcba201f640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c09c3d0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fcb9802e0c0 tx=0x7fcb980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.758+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9803f850 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.759+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9803f9b0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.759+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9803fd60 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.759+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c1f2bd0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.759+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c15bdf0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.768 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.759+0000 7f37f8c2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:12:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.760+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c0ad960 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.761+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fcb9803d070 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.762+0000 7fcb92ffd640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 0x7fcb8006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.762+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7fcb980ac7b0 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.763+0000 7fcba181e640 1 -- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 msgr2=0x7fcb8006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:08.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.763+0000 7fcba181e640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 0x7fcb8006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:08.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.767+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb9c0ad960 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.799 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.795+0000 7fec3e83f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:08.820 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.816+0000 7f37f8c2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:12:08.852 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.848+0000 7fec3e83f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:12:08.870 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:08.866+0000 7f37f8c2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:12:08.915 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.911+0000 7fec3e83f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:12:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.964+0000 7fcba181e640 1 -- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 msgr2=0x7fcb8006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.964+0000 7fcba181e640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 0x7fcb8006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:08.969 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:08.965+0000 7fec3e83f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:12:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.966+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb9c0ad960 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.967+0000 7fcb92ffd640 1 -- 172.21.15.60:0/3899258202 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v257) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb9c0ad960 con 0x7fcb9c06d5a0 2023-10-16T14:12:08.971 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:12:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 msgr2=0x7fcb8006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 0x7fcb8006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 msgr2=0x7fcb9c09c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c09c3d0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fcb9802e0c0 tx=0x7fcb980046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 shutdown_connections 2023-10-16T14:12:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7fcb8006b0b0 0x7fcb8006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcb9c0abd40 0x7fcb9c1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb9c0710e0 0x7fcb9c15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 --2- 172.21.15.60:0/3899258202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb9c06d5a0 0x7fcb9c09c3d0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.969+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 >> 172.21.15.60:0/3899258202 conn(0x7fcb9c0b0a80 msgr2=0x7fcb9c0ca470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.970+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 shutdown_connections 2023-10-16T14:12:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:08.970+0000 7fcba3021640 1 -- 172.21.15.60:0/3899258202 wait complete. 2023-10-16T14:12:08.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:12:09.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.249+0000 7f99638ec640 1 Processor -- start 2023-10-16T14:12:09.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.249+0000 7f99638ec640 1 -- start start 2023-10-16T14:12:09.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.250+0000 7f99638ec640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f995c0cf710 0x7f995c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.250+0000 7f99638ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f995c0b6cd0 0x7f995c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.250+0000 7f99638ec640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995c0d3980 con 0x7f995c0cfaf0 2023-10-16T14:12:09.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.250+0000 7f99638ec640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c0d3c60 con 0x7f995c0cf710 2023-10-16T14:12:09.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.250+0000 7f99638ec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c0ce630 con 0x7f995c0b6cd0 2023-10-16T14:12:09.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f995c0cf710 0x7f995c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f995c0cf710 0x7f995c0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33154/0 (socket says 172.21.15.60:33154) 2023-10-16T14:12:09.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 -- 172.21.15.60:0/2809492768 learned_addr learned my addr 172.21.15.60:0/2809492768 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:09.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9960e87640 1 --1- 172.21.15.60:0/2809492768 >> v1:172.21.15.60:6789/0 conn(0x7f995c0cfaf0 0x7f995c0d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33522/0 (socket says 172.21.15.60:33522) 2023-10-16T14:12:09.256 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.252+0000 7f37f8c2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:12:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961e89640 1 --2- 172.21.15.60:0/2809492768 >> v2:172.21.15.144:3300/0 conn(0x7f995c0b6cd0 0x7f995c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 -- 172.21.15.60:0/2809492768 >> v2:172.21.15.144:3300/0 conn(0x7f995c0b6cd0 msgr2=0x7f995c0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 --2- 172.21.15.60:0/2809492768 >> v2:172.21.15.144:3300/0 conn(0x7f995c0b6cd0 0x7f995c0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 -- 172.21.15.60:0/2809492768 >> v1:172.21.15.60:6789/0 conn(0x7f995c0cfaf0 legacy=0x7f995c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9961688640 1 -- 172.21.15.60:0/2809492768 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c0ce910 con 0x7f995c0cf710 2023-10-16T14:12:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.251+0000 7f9953fff640 1 -- 172.21.15.60:0/2809492768 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649216314 0 0) 0x7f995c0d3980 con 0x7f995c0cfaf0 2023-10-16T14:12:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.253+0000 7f9961688640 1 --2- 172.21.15.60:0/2809492768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c0b2d10 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f995c0b5890 tx=0x7f995802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1bdcfdcc269a2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.253+0000 7f9953fff640 1 -- 172.21.15.60:0/2809492768 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958004230 con 0x7f995c0cf710 2023-10-16T14:12:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.253+0000 7f9953fff640 1 -- 172.21.15.60:0/2809492768 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9958004390 con 0x7f995c0cf710 2023-10-16T14:12:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.253+0000 7f9953fff640 1 -- 172.21.15.60:0/2809492768 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958003810 con 0x7f995c0cf710 2023-10-16T14:12:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.253+0000 7f99638ec640 1 -- 172.21.15.60:0/2809492768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 msgr2=0x7f995c0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.253+0000 7f99638ec640 1 --2- 172.21.15.60:0/2809492768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c0b2d10 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f995c0b5890 tx=0x7f995802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.254+0000 7f99638ec640 1 -- 172.21.15.60:0/2809492768 shutdown_connections 2023-10-16T14:12:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.254+0000 7f99638ec640 1 --2- 172.21.15.60:0/2809492768 >> v2:172.21.15.144:3300/0 conn(0x7f995c0b6cd0 0x7f995c0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.254+0000 7f99638ec640 1 --2- 172.21.15.60:0/2809492768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c0b2d10 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.254+0000 7f99638ec640 1 -- 172.21.15.60:0/2809492768 >> 172.21.15.60:0/2809492768 conn(0x7f995c0d2cc0 msgr2=0x7f995c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.254+0000 7f99638ec640 1 -- 172.21.15.60:0/2809492768 shutdown_connections 2023-10-16T14:12:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.254+0000 7f99638ec640 1 -- 172.21.15.60:0/2809492768 wait complete. 2023-10-16T14:12:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.255+0000 7f99638ec640 1 Processor -- start 2023-10-16T14:12:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.255+0000 7f99638ec640 1 -- start start 2023-10-16T14:12:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.256+0000 7f99638ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f995c0b6cd0 0x7f995c15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.256+0000 7f99638ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.256+0000 7f99638ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f995c0cfaf0 0x7f995c0cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.256+0000 7f99638ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c15aeb0 con 0x7f995c0cf710 2023-10-16T14:12:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.256+0000 7f99638ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c15bcd0 con 0x7f995c0cfaf0 2023-10-16T14:12:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.256+0000 7f99638ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c15dd50 con 0x7f995c0b6cd0 2023-10-16T14:12:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33156/0 (socket says 172.21.15.60:33156) 2023-10-16T14:12:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 -- 172.21.15.60:0/1316842381 learned_addr learned my addr 172.21.15.60:0/1316842381 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9961688640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f995c0b6cd0 0x7f995c15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 -- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f995c0b6cd0 msgr2=0x7f995c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f995c0b6cd0 0x7f995c15d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 -- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f995c0cfaf0 msgr2=0x7f995c0cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f995c0cfaf0 0x7f995c0cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 -- 172.21.15.60:0/1316842381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958009460 con 0x7f995c0cf710 2023-10-16T14:12:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.257+0000 7f9960e87640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c159cd0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f994c00c4d0 tx=0x7f994c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.258+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c0074e0 con 0x7f995c0cf710 2023-10-16T14:12:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.258+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c007660 con 0x7f995c0cf710 2023-10-16T14:12:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.258+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c004a20 con 0x7f995c0cf710 2023-10-16T14:12:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.258+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995c0b0ba0 con 0x7f995c0cf710 2023-10-16T14:12:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.258+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c15b3a0 con 0x7f995c0cf710 2023-10-16T14:12:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.259+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c15be40 con 0x7f995c0cf710 2023-10-16T14:12:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.260+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f994c004b80 con 0x7f995c0cf710 2023-10-16T14:12:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.261+0000 7f9951ffb640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 0x7f994006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.262+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f994c0891a0 con 0x7f995c0cf710 2023-10-16T14:12:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.262+0000 7f9961688640 1 -- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 msgr2=0x7f994006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.262+0000 7f9961688640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 0x7f994006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.266+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f994c053940 con 0x7f995c0cf710 2023-10-16T14:12:09.383 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:09.379+0000 7fec3e83f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:12:09.416 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.412+0000 7f37f8c2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:12:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.460+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f995c04e160 con 0x7f995c0cf710 2023-10-16T14:12:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.461+0000 7f9951ffb640 1 -- 172.21.15.60:0/1316842381 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v257) v1 ==== 100+0+11 (secure 0 0 0) 0x7f994c056f60 con 0x7f995c0cf710 2023-10-16T14:12:09.465 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:12:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f9961688640 1 -- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 msgr2=0x7f994006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f9961688640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 0x7f994006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 msgr2=0x7f994006d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f99638ec640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 0x7f994006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 msgr2=0x7f995c159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f99638ec640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c159cd0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f994c00c4d0 tx=0x7f994c00c990 comp rx=0 tx=0).stop 2023-10-16T14:12:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 shutdown_connections 2023-10-16T14:12:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.463+0000 7f99638ec640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f994006b0e0 0x7f994006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.464+0000 7f99638ec640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f995c0cfaf0 0x7f995c0cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.464+0000 7f99638ec640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f995c0cf710 0x7f995c159cd0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.464+0000 7f99638ec640 1 --2- 172.21.15.60:0/1316842381 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f995c0b6cd0 0x7f995c15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.464+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 >> 172.21.15.60:0/1316842381 conn(0x7f995c0d2cc0 msgr2=0x7f995c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.464+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 shutdown_connections 2023-10-16T14:12:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.464+0000 7f99638ec640 1 -- 172.21.15.60:0/1316842381 wait complete. 2023-10-16T14:12:09.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T14:12:09.525 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.521+0000 7f37f8c2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:12:09.556 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:09.553+0000 7fec3e83f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:12:09.574 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.570+0000 7f37f8c2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:12:09.673 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:09.669+0000 7fec3e83f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:12:09.680 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.676+0000 7f37f8c2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:09.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.694+0000 7f552d7e9640 1 Processor -- start 2023-10-16T14:12:09.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.694+0000 7f552d7e9640 1 -- start start 2023-10-16T14:12:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552d7e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f552815cd70 0x7f55280d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552d7e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f55280b8480 0x7f55280cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552d7e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552815d150 con 0x7f55280d0be0 2023-10-16T14:12:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552d7e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55280d1180 con 0x7f55280b8480 2023-10-16T14:12:09.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552d7e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55280d1460 con 0x7f552815cd70 2023-10-16T14:12:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552cfe8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f55280d0be0 0x7f55280b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33524/0 (socket says 172.21.15.60:33524) 2023-10-16T14:12:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.696+0000 7f552cfe8640 1 -- 172.21.15.60:0/1340467473 learned_addr learned my addr 172.21.15.60:0/1340467473 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f55277fe640 1 --2- 172.21.15.60:0/1340467473 >> v2:172.21.15.60:3300/0 conn(0x7f55280b8480 0x7f55280cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f5527fff640 1 --2- 172.21.15.60:0/1340467473 >> v2:172.21.15.144:3300/0 conn(0x7f552815cd70 0x7f55280d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f5526ffd640 1 -- 172.21.15.60:0/1340467473 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176057984 0 0) 0x7f552815d150 con 0x7f55280d0be0 2023-10-16T14:12:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f5526ffd640 1 -- 172.21.15.60:0/1340467473 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5510003520 con 0x7f55280d0be0 2023-10-16T14:12:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f55277fe640 1 -- 172.21.15.60:0/1340467473 >> v2:172.21.15.144:3300/0 conn(0x7f552815cd70 msgr2=0x7f55280d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f55277fe640 1 --2- 172.21.15.60:0/1340467473 >> v2:172.21.15.144:3300/0 conn(0x7f552815cd70 0x7f55280d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f55277fe640 1 -- 172.21.15.60:0/1340467473 >> v1:172.21.15.60:6789/0 conn(0x7f55280d0be0 legacy=0x7f55280b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f55277fe640 1 -- 172.21.15.60:0/1340467473 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55280d1740 con 0x7f55280b8480 2023-10-16T14:12:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.697+0000 7f5527fff640 1 --2- 172.21.15.60:0/1340467473 >> v2:172.21.15.144:3300/0 conn(0x7f552815cd70 0x7f55280d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:12:09.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.698+0000 7f55277fe640 1 --2- 172.21.15.60:0/1340467473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55280b8480 0x7f55280cd7c0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f55280b56a0 tx=0x7f551c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9939d611e3457feb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.699+0000 7f5526ffd640 1 -- 172.21.15.60:0/1340467473 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c02e070 con 0x7f55280b8480 2023-10-16T14:12:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.699+0000 7f5526ffd640 1 -- 172.21.15.60:0/1340467473 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c0315a0 con 0x7f55280b8480 2023-10-16T14:12:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.699+0000 7f5526ffd640 1 -- 172.21.15.60:0/1340467473 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c031910 con 0x7f55280b8480 2023-10-16T14:12:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 -- 172.21.15.60:0/1340467473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55280b8480 msgr2=0x7f55280cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 --2- 172.21.15.60:0/1340467473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55280b8480 0x7f55280cd7c0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f55280b56a0 tx=0x7f551c02b840 comp rx=0 tx=0).stop 2023-10-16T14:12:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 -- 172.21.15.60:0/1340467473 shutdown_connections 2023-10-16T14:12:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 --2- 172.21.15.60:0/1340467473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55280b8480 0x7f55280cd7c0 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 --2- 172.21.15.60:0/1340467473 >> v2:172.21.15.144:3300/0 conn(0x7f552815cd70 0x7f55280d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 -- 172.21.15.60:0/1340467473 >> 172.21.15.60:0/1340467473 conn(0x7f55280cf570 msgr2=0x7f55280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 -- 172.21.15.60:0/1340467473 shutdown_connections 2023-10-16T14:12:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.700+0000 7f552d7e9640 1 -- 172.21.15.60:0/1340467473 wait complete. 2023-10-16T14:12:09.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.702+0000 7f552d7e9640 1 Processor -- start 2023-10-16T14:12:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.703+0000 7f552d7e9640 1 -- start start 2023-10-16T14:12:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.704+0000 7f552d7e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55280b8480 0x7f5528159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.704+0000 7f552d7e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55280d0be0 0x7f552815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.704+0000 7f552d7e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 0x7f552815d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.704+0000 7f552d7e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55280b58a0 con 0x7f552815cd70 2023-10-16T14:12:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.704+0000 7f552d7e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55281f54d0 con 0x7f55280d0be0 2023-10-16T14:12:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.704+0000 7f552d7e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552815a4e0 con 0x7f55280b8480 2023-10-16T14:12:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 0x7f552815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 0x7f552815d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33186/0 (socket says 172.21.15.60:33186) 2023-10-16T14:12:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 -- 172.21.15.60:0/584274005 learned_addr learned my addr 172.21.15.60:0/584274005 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f55277fe640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55280d0be0 0x7f552815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f5527fff640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55280b8480 0x7f5528159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 -- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55280b8480 msgr2=0x7f5528159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55280b8480 0x7f5528159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 -- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55280d0be0 msgr2=0x7f552815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55280d0be0 0x7f552815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 -- 172.21.15.60:0/584274005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c009400 con 0x7f552815cd70 2023-10-16T14:12:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.705+0000 7f552cfe8640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 0x7f552815d880 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f552000caa0 tx=0x7f552000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:09.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.706+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552000ac00 con 0x7f552815cd70 2023-10-16T14:12:09.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.706+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552000ad60 con 0x7f552815cd70 2023-10-16T14:12:09.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.706+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520012620 con 0x7f552815cd70 2023-10-16T14:12:09.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.706+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55281f5790 con 0x7f552815cd70 2023-10-16T14:12:09.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.706+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55281f5d50 con 0x7f552815cd70 2023-10-16T14:12:09.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.707+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552804e9b0 con 0x7f552815cd70 2023-10-16T14:12:09.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.708+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f5520007070 con 0x7f552815cd70 2023-10-16T14:12:09.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.709+0000 7f5524ff9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 0x7f551406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:09.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.710+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f5520089fe0 con 0x7f552815cd70 2023-10-16T14:12:09.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.710+0000 7f5527fff640 1 -- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 msgr2=0x7f551406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:09.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.710+0000 7f5527fff640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 0x7f551406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:09.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.714+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5520054780 con 0x7f552815cd70 2023-10-16T14:12:09.731 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:09.727+0000 7fec3e83f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:12:09.790 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.786+0000 7f37f8c2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:12:09.849 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:09.846+0000 7fec3e83f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.910+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f55280c9bc0 con 0x7f552815cd70 2023-10-16T14:12:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.911+0000 7f5527fff640 1 -- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 msgr2=0x7f551406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.911+0000 7f5527fff640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 0x7f551406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.911+0000 7f5524ff9640 1 -- 172.21.15.60:0/584274005 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v257) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5520017020 con 0x7f552815cd70 2023-10-16T14:12:09.916 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:12:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.913+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 msgr2=0x7f551406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.913+0000 7f552d7e9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 0x7f551406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 msgr2=0x7f552815d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 0x7f552815d880 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f552000caa0 tx=0x7f552000cf60 comp rx=0 tx=0).stop 2023-10-16T14:12:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 shutdown_connections 2023-10-16T14:12:09.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f551406b0b0 0x7f551406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f552815cd70 0x7f552815d880 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55280d0be0 0x7f552815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 --2- 172.21.15.60:0/584274005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55280b8480 0x7f5528159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 >> 172.21.15.60:0/584274005 conn(0x7f55280cf570 msgr2=0x7f55280b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:09.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 shutdown_connections 2023-10-16T14:12:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:09.914+0000 7f552d7e9640 1 -- 172.21.15.60:0/584274005 wait complete. 2023-10-16T14:12:09.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T14:12:09.946 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.942+0000 7f37f8c2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:09.965 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:09.961+0000 7fec3e83f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:12:09.996 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:09.992+0000 7f37f8c2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:10.115 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:10.111+0000 7f37f8c2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:12:10.133 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:10.130+0000 7fec3e83f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:10.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.143+0000 7f229759e640 1 Processor -- start 2023-10-16T14:12:10.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.143+0000 7f229759e640 1 -- start start 2023-10-16T14:12:10.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.145+0000 7f229759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f229806c040 0x7f229806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.145+0000 7f229759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22980cda80 0x7f229815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.145+0000 7f229759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229806c960 con 0x7f22980ad970 2023-10-16T14:12:10.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.145+0000 7f229759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229815b9d0 con 0x7f229806c040 2023-10-16T14:12:10.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.145+0000 7f229759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229815bcb0 con 0x7f22980cda80 2023-10-16T14:12:10.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.146+0000 7f229659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f229806c040 0x7f229806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.146+0000 7f2296d9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22980cda80 0x7f229815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.146+0000 7f229659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f229806c040 0x7f229806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55476/0 (socket says 172.21.15.60:55476) 2023-10-16T14:12:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.146+0000 7f229659c640 1 -- 172.21.15.60:0/1386508075 learned_addr learned my addr 172.21.15.60:0/1386508075 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.147+0000 7f229659c640 1 -- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22980cda80 msgr2=0x7f229815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.147+0000 7f229659c640 1 --2- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22980cda80 0x7f229815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.147+0000 7f229659c640 1 -- 172.21.15.60:0/1386508075 >> v1:172.21.15.144:6789/0 conn(0x7f22980ad970 legacy=0x7f229815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.147+0000 7f229659c640 1 -- 172.21.15.60:0/1386508075 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229815bf90 con 0x7f229806c040 2023-10-16T14:12:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.147+0000 7f2296d9d640 1 --2- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22980cda80 0x7f229815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:10.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.148+0000 7f229659c640 1 --2- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f229806c040 0x7f229806c420 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2288009870 tx=0x7f228802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27dc81c6dc26c310 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:10.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.148+0000 7f2295d9b640 1 -- 172.21.15.60:0/1386508075 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288030840 con 0x7f229806c040 2023-10-16T14:12:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.148+0000 7f2295d9b640 1 -- 172.21.15.60:0/1386508075 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22880309a0 con 0x7f229806c040 2023-10-16T14:12:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.148+0000 7f2295d9b640 1 -- 172.21.15.60:0/1386508075 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288030d30 con 0x7f229806c040 2023-10-16T14:12:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 -- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f229806c040 msgr2=0x7f229806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 --2- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f229806c040 0x7f229806c420 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2288009870 tx=0x7f228802b980 comp rx=0 tx=0).stop 2023-10-16T14:12:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 -- 172.21.15.60:0/1386508075 shutdown_connections 2023-10-16T14:12:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 --2- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f22980cda80 0x7f229815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 --2- 172.21.15.60:0/1386508075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f229806c040 0x7f229806c420 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 -- 172.21.15.60:0/1386508075 >> 172.21.15.60:0/1386508075 conn(0x7f22980ce6a0 msgr2=0x7f22980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 -- 172.21.15.60:0/1386508075 shutdown_connections 2023-10-16T14:12:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.149+0000 7f229759e640 1 -- 172.21.15.60:0/1386508075 wait complete. 2023-10-16T14:12:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.151+0000 7f229759e640 1 Processor -- start 2023-10-16T14:12:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.151+0000 7f229759e640 1 -- start start 2023-10-16T14:12:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f229759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f229806c040 0x7f22980c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f229759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22980ad970 0x7f22981f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f229759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 0x7f229815a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f229759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22980c8590 con 0x7f22980cda80 2023-10-16T14:12:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f229759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229814b110 con 0x7f22980ad970 2023-10-16T14:12:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f229759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22980ac930 con 0x7f229806c040 2023-10-16T14:12:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f2296d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 0x7f229815a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f2296d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 0x7f229815a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33188/0 (socket says 172.21.15.60:33188) 2023-10-16T14:12:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.152+0000 7f2296d9d640 1 -- 172.21.15.60:0/2088448450 learned_addr learned my addr 172.21.15.60:0/2088448450 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f228ffff640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22980ad970 0x7f22981f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f229659c640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f229806c040 0x7f22980c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f2296d9d640 1 -- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f229806c040 msgr2=0x7f22980c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f2296d9d640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f229806c040 0x7f22980c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f2296d9d640 1 -- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22980ad970 msgr2=0x7f22981f1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f2296d9d640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22980ad970 0x7f22981f1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f2296d9d640 1 -- 172.21.15.60:0/2088448450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290009a70 con 0x7f22980cda80 2023-10-16T14:12:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.153+0000 7f2296d9d640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 0x7f229815a2b0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f229000efc0 tx=0x7f229000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.154+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290009cd0 con 0x7f22980cda80 2023-10-16T14:12:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.154+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229000f040 con 0x7f22980cda80 2023-10-16T14:12:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.154+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290003990 con 0x7f22980cda80 2023-10-16T14:12:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.154+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22880094d0 con 0x7f22980cda80 2023-10-16T14:12:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.154+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22980acef0 con 0x7f22980cda80 2023-10-16T14:12:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.155+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229806b330 con 0x7f22980cda80 2023-10-16T14:12:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.156+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f2290014480 con 0x7f22980cda80 2023-10-16T14:12:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.157+0000 7f228effd640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 0x7f227806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.157+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f229008acf0 con 0x7f22980cda80 2023-10-16T14:12:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.158+0000 7f229659c640 1 -- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 msgr2=0x7f227806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.158+0000 7f229659c640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 0x7f227806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:10.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.162+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2290055380 con 0x7f22980cda80 2023-10-16T14:12:10.188 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:10.185+0000 7fec3e83f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:10.316 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:10.312+0000 7fec3e83f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:12:10.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.358+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f229804e9b0 con 0x7f22980cda80 2023-10-16T14:12:10.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.359+0000 7f229659c640 1 -- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 msgr2=0x7f227806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:10.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.359+0000 7f229659c640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 0x7f227806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:10.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.359+0000 7f228effd640 1 -- 172.21.15.60:0/2088448450 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v257) v1 ==== 103+0+11 (secure 0 0 0) 0x7f22900589a0 con 0x7f22980cda80 2023-10-16T14:12:10.364 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:12:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 msgr2=0x7f227806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 0x7f227806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 msgr2=0x7f229815a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 0x7f229815a2b0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f229000efc0 tx=0x7f229000c490 comp rx=0 tx=0).stop 2023-10-16T14:12:10.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 shutdown_connections 2023-10-16T14:12:10.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f227806b0b0 0x7f227806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980cda80 0x7f229815a2b0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22980ad970 0x7f22981f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 --2- 172.21.15.60:0/2088448450 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f229806c040 0x7f22980c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 >> 172.21.15.60:0/2088448450 conn(0x7f22980ce6a0 msgr2=0x7f229815e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 shutdown_connections 2023-10-16T14:12:10.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.362+0000 7f229759e640 1 -- 172.21.15.60:0/2088448450 wait complete. 2023-10-16T14:12:10.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:12:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.627+0000 7f003bebb640 1 Processor -- start 2023-10-16T14:12:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.627+0000 7f003bebb640 1 -- start start 2023-10-16T14:12:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f003bebb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f00340c9f10 0x7f003406b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f003bebb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340cdc30 con 0x7f003406ade0 2023-10-16T14:12:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f003bebb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340ccd80 con 0x7f0034071b30 2023-10-16T14:12:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f003bebb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003406b780 con 0x7f00340c9f10 2023-10-16T14:12:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f0039456640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f003406ade0 0x7f00340c9800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33528/0 (socket says 172.21.15.60:33528) 2023-10-16T14:12:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f0039456640 1 -- 172.21.15.60:0/3713634582 learned_addr learned my addr 172.21.15.60:0/3713634582 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.629+0000 7f003a458640 1 --2- 172.21.15.60:0/3713634582 >> v2:172.21.15.60:3300/0 conn(0x7f00340c9f10 0x7f003406b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.630+0000 7f0038c55640 1 -- 172.21.15.60:0/3713634582 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223334016 0 0) 0x7f00340cdc30 con 0x7f003406ade0 2023-10-16T14:12:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.630+0000 7f0038c55640 1 -- 172.21.15.60:0/3713634582 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020003520 con 0x7f003406ade0 2023-10-16T14:12:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.630+0000 7f003a458640 1 -- 172.21.15.60:0/3713634582 >> v1:172.21.15.144:6789/0 conn(0x7f0034071b30 legacy=0x7f003406fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.630+0000 7f003a458640 1 -- 172.21.15.60:0/3713634582 >> v1:172.21.15.60:6789/0 conn(0x7f003406ade0 legacy=0x7f00340c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.630+0000 7f003a458640 1 -- 172.21.15.60:0/3713634582 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340cd020 con 0x7f00340c9f10 2023-10-16T14:12:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.631+0000 7f003a458640 1 --2- 172.21.15.60:0/3713634582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00340c9f10 0x7f003406b240 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f002c00aea0 tx=0x7f002c02d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa1b7b0d3bfbdab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.631+0000 7f0038c55640 1 -- 172.21.15.60:0/3713634582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c02dbe0 con 0x7f00340c9f10 2023-10-16T14:12:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.631+0000 7f0038c55640 1 -- 172.21.15.60:0/3713634582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c02dd40 con 0x7f00340c9f10 2023-10-16T14:12:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.631+0000 7f0038c55640 1 -- 172.21.15.60:0/3713634582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c008640 con 0x7f00340c9f10 2023-10-16T14:12:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 -- 172.21.15.60:0/3713634582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00340c9f10 msgr2=0x7f003406b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 --2- 172.21.15.60:0/3713634582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00340c9f10 0x7f003406b240 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f002c00aea0 tx=0x7f002c02d040 comp rx=0 tx=0).stop 2023-10-16T14:12:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 -- 172.21.15.60:0/3713634582 shutdown_connections 2023-10-16T14:12:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 --2- 172.21.15.60:0/3713634582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00340c9f10 0x7f003406b240 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 -- 172.21.15.60:0/3713634582 >> 172.21.15.60:0/3713634582 conn(0x7f003406bc10 msgr2=0x7f003406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 -- 172.21.15.60:0/3713634582 shutdown_connections 2023-10-16T14:12:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.632+0000 7f003bebb640 1 -- 172.21.15.60:0/3713634582 wait complete. 2023-10-16T14:12:10.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.634+0000 7f003bebb640 1 Processor -- start 2023-10-16T14:12:10.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.634+0000 7f003bebb640 1 -- start start 2023-10-16T14:12:10.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003bebb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f003406ade0 0x7f00341f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003bebb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 0x7f0034146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003bebb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00340c9f10 0x7f0034146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003bebb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c8340 con 0x7f0034071b30 2023-10-16T14:12:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003bebb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c8620 con 0x7f00340c9f10 2023-10-16T14:12:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003bebb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c8900 con 0x7f003406ade0 2023-10-16T14:12:10.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 0x7f0034146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 0x7f0034146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33202/0 (socket says 172.21.15.60:33202) 2023-10-16T14:12:10.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 -- 172.21.15.60:0/885051231 learned_addr learned my addr 172.21.15.60:0/885051231 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:10.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f003a458640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00340c9f10 0x7f0034146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039c57640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f003406ade0 0x7f00341f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:10.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 -- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f003406ade0 msgr2=0x7f00341f61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f003406ade0 0x7f00341f61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 -- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00340c9f10 msgr2=0x7f0034146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00340c9f10 0x7f0034146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.635+0000 7f0039456640 1 -- 172.21.15.60:0/885051231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c00ab80 con 0x7f0034071b30 2023-10-16T14:12:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.636+0000 7f0039456640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 0x7f0034146500 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f002400c410 tx=0x7f002400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.636+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024005ce0 con 0x7f0034071b30 2023-10-16T14:12:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.636+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024005e40 con 0x7f0034071b30 2023-10-16T14:12:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.636+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340c8be0 con 0x7f0034071b30 2023-10-16T14:12:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.636+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240072f0 con 0x7f0034071b30 2023-10-16T14:12:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.636+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00341f6cd0 con 0x7f0034071b30 2023-10-16T14:12:10.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.637+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00340c9100 con 0x7f0034071b30 2023-10-16T14:12:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.639+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f0024007510 con 0x7f0034071b30 2023-10-16T14:12:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.640+0000 7f002a7fc640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 0x7f001806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.640+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f0024089610 con 0x7f0034071b30 2023-10-16T14:12:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.640+0000 7f0039c57640 1 -- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 msgr2=0x7f001806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.640+0000 7f0039c57640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 0x7f001806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:10.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.645+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00340c9100 con 0x7f0034071b30 2023-10-16T14:12:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.842+0000 7f0039c57640 1 -- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 msgr2=0x7f001806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.842+0000 7f0039c57640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 0x7f001806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.853+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f00340c9100 con 0x7f0034071b30 2023-10-16T14:12:10.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.854+0000 7f002a7fc640 1 -- 172.21.15.60:0/885051231 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v257) v1 ==== 81+0+595 (secure 0 0 0) 0x7f00340c9100 con 0x7f0034071b30 2023-10-16T14:12:10.857 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3601797085 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.858 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3692487868 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.858 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2884978111 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.858 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/4230051502 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.858 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2891523992 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.858 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1029476750 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.859 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/4052124783 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.859 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/4052124783 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.859 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2571613028 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.859 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/752392530 2023-10-17T14:12:05.212706+0000 2023-10-16T14:12:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:listed 10 entries 2023-10-16T14:12:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 msgr2=0x7f001806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 0x7f001806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 msgr2=0x7f0034146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 0x7f0034146500 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f002400c410 tx=0x7f002400c8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 shutdown_connections 2023-10-16T14:12:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f001806b0b0 0x7f001806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00340c9f10 0x7f0034146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0034071b30 0x7f0034146500 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 --2- 172.21.15.60:0/885051231 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f003406ade0 0x7f00341f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 >> 172.21.15.60:0/885051231 conn(0x7f003406bc10 msgr2=0x7f00340afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 shutdown_connections 2023-10-16T14:12:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:10.856+0000 7f003bebb640 1 -- 172.21.15.60:0/885051231 wait complete. 2023-10-16T14:12:10.879 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:12:10.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:10.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:12:10.964 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:12:10.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:10.965 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:12:10.971 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:10.967+0000 7f37f8c2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:12:11.010 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:12:11.010 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:11.065 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:12:11.066 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:12:11.066 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:11.124 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:12:11.124 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:12:11.125 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:11.172 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:11.168+0000 7fec3e83f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:12:11.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:12:11.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:11.274 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.270+0000 7f37f8c2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:12:11.332 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.328+0000 7f37f8c2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:12:11.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.392+0000 7f371fe0e640 1 Processor -- start 2023-10-16T14:12:11.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.392+0000 7f371fe0e640 1 -- start start 2023-10-16T14:12:11.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371fe0e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37180ae220 0x7f37180cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371fe0e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371806a4b0 0x7f37180cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371fe0e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37181614f0 con 0x7f3718160f30 2023-10-16T14:12:11.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371fe0e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37180cf760 con 0x7f37180ae220 2023-10-16T14:12:11.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371fe0e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37181617d0 con 0x7f371806a4b0 2023-10-16T14:12:11.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371dbaa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37180ae220 0x7f37180cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371dbaa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37180ae220 0x7f37180cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33210/0 (socket says 172.21.15.60:33210) 2023-10-16T14:12:11.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.394+0000 7f371dbaa640 1 -- 172.21.15.60:0/994152305 learned_addr learned my addr 172.21.15.60:0/994152305 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:11.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.395+0000 7f371dbaa640 1 -- 172.21.15.60:0/994152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371806a4b0 msgr2=0x7f37180cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:11.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.395+0000 7f371e3ab640 1 --2- 172.21.15.60:0/994152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371806a4b0 0x7f37180cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.395+0000 7f371dbaa640 1 --2- 172.21.15.60:0/994152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371806a4b0 0x7f37180cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.395+0000 7f371dbaa640 1 -- 172.21.15.60:0/994152305 >> v1:172.21.15.144:6789/0 conn(0x7f3718160f30 legacy=0x7f3718069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:11.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.395+0000 7f371dbaa640 1 -- 172.21.15.60:0/994152305 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3718161ab0 con 0x7f37180ae220 2023-10-16T14:12:11.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.396+0000 7f371dbaa640 1 --2- 172.21.15.60:0/994152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37180cabd0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f371806ae60 tx=0x7f371402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1664a12d77913846 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:11.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.396+0000 7f371cba8640 1 -- 172.21.15.60:0/994152305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371402e070 con 0x7f37180ae220 2023-10-16T14:12:11.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.396+0000 7f371cba8640 1 -- 172.21.15.60:0/994152305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37140315a0 con 0x7f37180ae220 2023-10-16T14:12:11.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.396+0000 7f371cba8640 1 -- 172.21.15.60:0/994152305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714031930 con 0x7f37180ae220 2023-10-16T14:12:11.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 -- 172.21.15.60:0/994152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 msgr2=0x7f37180cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 --2- 172.21.15.60:0/994152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37180cabd0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f371806ae60 tx=0x7f371402b840 comp rx=0 tx=0).stop 2023-10-16T14:12:11.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 -- 172.21.15.60:0/994152305 shutdown_connections 2023-10-16T14:12:11.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 --2- 172.21.15.60:0/994152305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371806a4b0 0x7f37180cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 --2- 172.21.15.60:0/994152305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37180cabd0 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 -- 172.21.15.60:0/994152305 >> 172.21.15.60:0/994152305 conn(0x7f37180cfb70 msgr2=0x7f37180cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:11.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 -- 172.21.15.60:0/994152305 shutdown_connections 2023-10-16T14:12:11.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.397+0000 7f371fe0e640 1 -- 172.21.15.60:0/994152305 wait complete. 2023-10-16T14:12:11.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.399+0000 7f371fe0e640 1 Processor -- start 2023-10-16T14:12:11.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.399+0000 7f371fe0e640 1 -- start start 2023-10-16T14:12:11.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371fe0e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371806a4b0 0x7f37180b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371fe0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37181f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371fe0e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3718160f30 0x7f371814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371fe0e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37181f6880 con 0x7f37180ae220 2023-10-16T14:12:11.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371fe0e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180b38a0 con 0x7f371806a4b0 2023-10-16T14:12:11.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371fe0e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180b3b80 con 0x7f3718160f30 2023-10-16T14:12:11.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371d3a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37181f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371d3a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37181f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33216/0 (socket says 172.21.15.60:33216) 2023-10-16T14:12:11.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371d3a9640 1 -- 172.21.15.60:0/1949036305 learned_addr learned my addr 172.21.15.60:0/1949036305 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:11.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371dbaa640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371806a4b0 0x7f37180b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.400+0000 7f371e3ab640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3718160f30 0x7f371814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371d3a9640 1 -- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3718160f30 msgr2=0x7f371814ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371d3a9640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3718160f30 0x7f371814ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371d3a9640 1 -- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371806a4b0 msgr2=0x7f37180b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371d3a9640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371806a4b0 0x7f37180b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371d3a9640 1 -- 172.21.15.60:0/1949036305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37140094d0 con 0x7f37180ae220 2023-10-16T14:12:11.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371d3a9640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37181f6340 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f370800c430 tx=0x7f370800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:11.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708011790 con 0x7f37180ae220 2023-10-16T14:12:11.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37080118f0 con 0x7f37180ae220 2023-10-16T14:12:11.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708011ce0 con 0x7f37180ae220 2023-10-16T14:12:11.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37180b3e60 con 0x7f37180ae220 2023-10-16T14:12:11.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.401+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37180b4420 con 0x7f37180ae220 2023-10-16T14:12:11.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.403+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371804e160 con 0x7f37180ae220 2023-10-16T14:12:11.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.404+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f370801c460 con 0x7f37180ae220 2023-10-16T14:12:11.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.405+0000 7f370e7fc640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f36fc06b0b0 0x7f36fc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.405+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7003+0+0 (secure 0 0 0) 0x7f3708089e30 con 0x7f37180ae220 2023-10-16T14:12:11.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.405+0000 7f371dbaa640 1 -- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f36fc06b0b0 msgr2=0x7f36fc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/529676808 2023-10-16T14:12:11.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.405+0000 7f371dbaa640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f36fc06b0b0 0x7f36fc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.413+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3708054730 con 0x7f37180ae220 2023-10-16T14:12:11.445 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.442+0000 7f37f8c2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:12:11.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.462+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 142) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3708054b00 con 0x7f37180ae220 2023-10-16T14:12:11.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.462+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f36fc06b0b0 msgr2=0x7f36fc06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:11.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.462+0000 7f370e7fc640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f36fc06b0b0 0x7f36fc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.504 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:11.500+0000 7fec3e83f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:12:11.546 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.541+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.546 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.541+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.546 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.541+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.546 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.541+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.541+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.542+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.542+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.542+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.542+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.547 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.542+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.554+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.554+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.554+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.558 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.554+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.559 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.554+0000 7f377d7d4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.559 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.555+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.559 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.555+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.560 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.555+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.560 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.555+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.560 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:11.555+0000 7f3779fcd640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:11.569 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:11.566+0000 7fec3e83f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:12:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.653+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37180b5af0 con 0x7f37180ae220 2023-10-16T14:12:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.654+0000 7f370e7fc640 1 -- 172.21.15.60:0/1949036305 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6146 (secure 0 0 0) 0x7f3708070640 con 0x7f37180ae220 2023-10-16T14:12:11.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:11.658 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:11.659 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 134 2023-10-16T14:12:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 msgr2=0x7f37181f6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37181f6340 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f370800c430 tx=0x7f370800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 shutdown_connections 2023-10-16T14:12:11.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:6832/529676808,v1:172.21.15.144:6833/529676808] conn(0x7f36fc06b0b0 0x7f36fc06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3718160f30 0x7f371814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180ae220 0x7f37181f6340 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 --2- 172.21.15.60:0/1949036305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371806a4b0 0x7f37180b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 >> 172.21.15.60:0/1949036305 conn(0x7f37180cfb70 msgr2=0x7f3718160190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 shutdown_connections 2023-10-16T14:12:11.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.657+0000 7f371fe0e640 1 -- 172.21.15.60:0/1949036305 wait complete. 2023-10-16T14:12:11.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:11.690 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:11.687+0000 7fec3e83f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:12:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.872+0000 7f390b59e640 1 Processor -- start 2023-10-16T14:12:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.873+0000 7f390b59e640 1 -- start start 2023-10-16T14:12:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.874+0000 7f390b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c06c040 0x7f390c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.874+0000 7f390b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f390c0ad970 0x7f390c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.874+0000 7f390b59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390c06c960 con 0x7f390c15ebc0 2023-10-16T14:12:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.874+0000 7f390b59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c15b9d0 con 0x7f390c0ad970 2023-10-16T14:12:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.874+0000 7f390b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c15bcb0 con 0x7f390c06c040 2023-10-16T14:12:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f390c0ad970 0x7f390c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f390c0ad970 0x7f390c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33226/0 (socket says 172.21.15.60:33226) 2023-10-16T14:12:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 -- 172.21.15.60:0/4292844373 learned_addr learned my addr 172.21.15.60:0/4292844373 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f390a59c640 1 --2- 172.21.15.60:0/4292844373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c06c040 0x7f390c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 -- 172.21.15.60:0/4292844373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c06c040 msgr2=0x7f390c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 --2- 172.21.15.60:0/4292844373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c06c040 0x7f390c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 -- 172.21.15.60:0/4292844373 >> v1:172.21.15.60:6789/0 conn(0x7f390c15ebc0 legacy=0x7f390c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.875+0000 7f3909d9b640 1 -- 172.21.15.60:0/4292844373 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c15bf90 con 0x7f390c0ad970 2023-10-16T14:12:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.876+0000 7f3909d9b640 1 --2- 172.21.15.60:0/4292844373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c15e680 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f39000098a0 tx=0x7f390002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e2b7d6399da598 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390959a640 1 -- 172.21.15.60:0/4292844373 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900030840 con 0x7f390c0ad970 2023-10-16T14:12:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390959a640 1 -- 172.21.15.60:0/4292844373 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39000309a0 con 0x7f390c0ad970 2023-10-16T14:12:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390959a640 1 -- 172.21.15.60:0/4292844373 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900030d30 con 0x7f390c0ad970 2023-10-16T14:12:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 -- 172.21.15.60:0/4292844373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 msgr2=0x7f390c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 --2- 172.21.15.60:0/4292844373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c15e680 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f39000098a0 tx=0x7f390002b980 comp rx=0 tx=0).stop 2023-10-16T14:12:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 -- 172.21.15.60:0/4292844373 shutdown_connections 2023-10-16T14:12:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 --2- 172.21.15.60:0/4292844373 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c15e680 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 --2- 172.21.15.60:0/4292844373 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c06c040 0x7f390c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 -- 172.21.15.60:0/4292844373 >> 172.21.15.60:0/4292844373 conn(0x7f390c0ce6a0 msgr2=0x7f390c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.877+0000 7f390b59e640 1 -- 172.21.15.60:0/4292844373 shutdown_connections 2023-10-16T14:12:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.878+0000 7f390b59e640 1 -- 172.21.15.60:0/4292844373 wait complete. 2023-10-16T14:12:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.879+0000 7f390b59e640 1 Processor -- start 2023-10-16T14:12:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.879+0000 7f390b59e640 1 -- start start 2023-10-16T14:12:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.880+0000 7f390b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c06c040 0x7f390c1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.880+0000 7f390b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c0b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.880+0000 7f390b59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c15ebc0 0x7f390c0ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.880+0000 7f390b59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0b10f0 con 0x7f390c0ad970 2023-10-16T14:12:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.880+0000 7f390b59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c157a00 con 0x7f390c06c040 2023-10-16T14:12:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.880+0000 7f390b59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0c5910 con 0x7f390c15ebc0 2023-10-16T14:12:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c0b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c0b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33228/0 (socket says 172.21.15.60:33228) 2023-10-16T14:12:11.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 -- 172.21.15.60:0/3230062898 learned_addr learned my addr 172.21.15.60:0/3230062898 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 -- 172.21.15.60:0/3230062898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c15ebc0 msgr2=0x7f390c0ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c15ebc0 0x7f390c0ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 -- 172.21.15.60:0/3230062898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c06c040 msgr2=0x7f390c1574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c06c040 0x7f390c1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:11.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 -- 172.21.15.60:0/3230062898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39000094d0 con 0x7f390c0ad970 2023-10-16T14:12:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.881+0000 7f3909d9b640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c0b0750 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f390002b9c0 tx=0x7f3900004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.882+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900039a90 con 0x7f390c0ad970 2023-10-16T14:12:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.882+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3900039bf0 con 0x7f390c0ad970 2023-10-16T14:12:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.882+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c0c5bf0 con 0x7f390c0ad970 2023-10-16T14:12:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.882+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900043410 con 0x7f390c0ad970 2023-10-16T14:12:11.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.882+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c0c61b0 con 0x7f390c0ad970 2023-10-16T14:12:11.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.883+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c06b330 con 0x7f390c0ad970 2023-10-16T14:12:11.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.884+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 142) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f390004c3b0 con 0x7f390c0ad970 2023-10-16T14:12:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.885+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7f39000abd00 con 0x7f390c0ad970 2023-10-16T14:12:11.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:11.890+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3900076b70 con 0x7f390c0ad970 2023-10-16T14:12:12.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.090+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f390c0c56c0 con 0x7f390c0ad970 2023-10-16T14:12:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.093+0000 7f38faffd640 1 -- 172.21.15.60:0/3230062898 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+21443 (secure 0 0 0) 0x7f390007a190 con 0x7f390c0ad970 2023-10-16T14:12:12.097 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:12.097 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":258,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:11.447519+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:12.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 msgr2=0x7f390c0b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c0b0750 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f390002b9c0 tx=0x7f3900004730 comp rx=0 tx=0).stop 2023-10-16T14:12:12.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 shutdown_connections 2023-10-16T14:12:12.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f390c15ebc0 0x7f390c0ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f390c0ad970 0x7f390c0b0750 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 --2- 172.21.15.60:0/3230062898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c06c040 0x7f390c1574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 >> 172.21.15.60:0/3230062898 conn(0x7f390c0ce6a0 msgr2=0x7f390c069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 shutdown_connections 2023-10-16T14:12:12.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.096+0000 7f390b59e640 1 -- 172.21.15.60:0/3230062898 wait complete. 2023-10-16T14:12:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:11:36.185089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:11:37.268518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T14:11:50.763838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T14:11:51.462135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:12:12.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:12:12.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.390+0000 7fb26fb90640 1 Processor -- start 2023-10-16T14:12:12.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.390+0000 7fb26fb90640 1 -- start start 2023-10-16T14:12:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26fb90640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb268159e90 0x7fb2680ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26fb90640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb26806c140 0x7fb26806c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26fb90640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26806cad0 con 0x7fb268159ab0 2023-10-16T14:12:12.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26fb90640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26815d890 con 0x7fb268159e90 2023-10-16T14:12:12.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26fb90640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb268070360 con 0x7fb26806c140 2023-10-16T14:12:12.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26d12b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb268159e90 0x7fb2680ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26d12b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb268159e90 0x7fb2680ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33230/0 (socket says 172.21.15.60:33230) 2023-10-16T14:12:12.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.391+0000 7fb26d12b640 1 -- 172.21.15.60:0/648106771 learned_addr learned my addr 172.21.15.60:0/648106771 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:12.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.392+0000 7fb26d12b640 1 -- 172.21.15.60:0/648106771 >> v2:172.21.15.144:3300/0 conn(0x7fb26806c140 msgr2=0x7fb26806c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.392+0000 7fb26d12b640 1 --2- 172.21.15.60:0/648106771 >> v2:172.21.15.144:3300/0 conn(0x7fb26806c140 0x7fb26806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.392+0000 7fb26d12b640 1 -- 172.21.15.60:0/648106771 >> v1:172.21.15.144:6789/0 conn(0x7fb268159ab0 legacy=0x7fb26806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.392+0000 7fb26d12b640 1 -- 172.21.15.60:0/648106771 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680cd1f0 con 0x7fb268159e90 2023-10-16T14:12:12.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.393+0000 7fb26d12b640 1 --2- 172.21.15.60:0/648106771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159e90 0x7fb2680ce370 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fb2680adcd0 tx=0x7fb25c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c99fd2f713e651 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.393+0000 7fb26c92a640 1 -- 172.21.15.60:0/648106771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c031750 con 0x7fb268159e90 2023-10-16T14:12:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.393+0000 7fb26c92a640 1 -- 172.21.15.60:0/648106771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c0318b0 con 0x7fb268159e90 2023-10-16T14:12:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.393+0000 7fb26c92a640 1 -- 172.21.15.60:0/648106771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c031c40 con 0x7fb268159e90 2023-10-16T14:12:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 -- 172.21.15.60:0/648106771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159e90 msgr2=0x7fb2680ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 --2- 172.21.15.60:0/648106771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159e90 0x7fb2680ce370 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fb2680adcd0 tx=0x7fb25c02b7f0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 -- 172.21.15.60:0/648106771 shutdown_connections 2023-10-16T14:12:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 --2- 172.21.15.60:0/648106771 >> v2:172.21.15.144:3300/0 conn(0x7fb26806c140 0x7fb26806c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 --2- 172.21.15.60:0/648106771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159e90 0x7fb2680ce370 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 -- 172.21.15.60:0/648106771 >> 172.21.15.60:0/648106771 conn(0x7fb2680ce9b0 msgr2=0x7fb2680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:12.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 -- 172.21.15.60:0/648106771 shutdown_connections 2023-10-16T14:12:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.394+0000 7fb26fb90640 1 -- 172.21.15.60:0/648106771 wait complete. 2023-10-16T14:12:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.396+0000 7fb26fb90640 1 Processor -- start 2023-10-16T14:12:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.396+0000 7fb26fb90640 1 -- start start 2023-10-16T14:12:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26fb90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26806c140 0x7fb268154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26fb90640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 0x7fb2680b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26fb90640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb268159e90 0x7fb2680b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26fb90640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680c4310 con 0x7fb268159ab0 2023-10-16T14:12:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26fb90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680c4560 con 0x7fb26806c140 2023-10-16T14:12:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26fb90640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680c4840 con 0x7fb268159e90 2023-10-16T14:12:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26d12b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 0x7fb2680b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26d12b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 0x7fb2680b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33238/0 (socket says 172.21.15.60:33238) 2023-10-16T14:12:12.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.397+0000 7fb26d12b640 1 -- 172.21.15.60:0/3737074685 learned_addr learned my addr 172.21.15.60:0/3737074685 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:12.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d92c640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26806c140 0x7fb268154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d12b640 1 -- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb268159e90 msgr2=0x7fb2680b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d12b640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb268159e90 0x7fb2680b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d12b640 1 -- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26806c140 msgr2=0x7fb268154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d12b640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26806c140 0x7fb268154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d12b640 1 -- 172.21.15.60:0/3737074685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0094d0 con 0x7fb268159ab0 2023-10-16T14:12:12.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d92c640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26806c140 0x7fb268154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:12.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.398+0000 7fb26d12b640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 0x7fb2680b30e0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fb25c02b830 tx=0x7fb25c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.399+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c033c80 con 0x7fb268159ab0 2023-10-16T14:12:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.399+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c033e00 con 0x7fb268159ab0 2023-10-16T14:12:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.399+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c03c640 con 0x7fb268159ab0 2023-10-16T14:12:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.399+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2680c4b20 con 0x7fb268159ab0 2023-10-16T14:12:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.399+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb268155130 con 0x7fb268159ab0 2023-10-16T14:12:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.400+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2680c4980 con 0x7fb268159ab0 2023-10-16T14:12:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.401+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 142) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb25c03a040 con 0x7fb268159ab0 2023-10-16T14:12:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.402+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7fb25c0ab520 con 0x7fb268159ab0 2023-10-16T14:12:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.406+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2680c4980 con 0x7fb268159ab0 2023-10-16T14:12:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.467+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb25c075d20 con 0x7fb268159ab0 2023-10-16T14:12:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.468+0000 7fb2667fc640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb254072d80 0x7fb254075170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.474+0000 7fb26d92c640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb254072d80 0x7fb254075170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.481+0000 7fb26d92c640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb254072d80 0x7fb254075170 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb2680aed60 tx=0x7fb258009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:12.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.609+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb26806cad0 con 0x7fb268159ab0 2023-10-16T14:12:12.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.609+0000 7fb2667fc640 1 -- 172.21.15.60:0/3737074685 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb26806cad0 con 0x7fb268159ab0 2023-10-16T14:12:12.614 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:12:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb254072d80 msgr2=0x7fb254075170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb254072d80 0x7fb254075170 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb2680aed60 tx=0x7fb258009040 comp rx=0 tx=0).stop 2023-10-16T14:12:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 msgr2=0x7fb2680b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 0x7fb2680b30e0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fb25c02b830 tx=0x7fb25c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 shutdown_connections 2023-10-16T14:12:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb254072d80 0x7fb254075170 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb268159e90 0x7fb2680b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb268159ab0 0x7fb2680b30e0 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 --2- 172.21.15.60:0/3737074685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26806c140 0x7fb268154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 >> 172.21.15.60:0/3737074685 conn(0x7fb2680ce9b0 msgr2=0x7fb2680ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 shutdown_connections 2023-10-16T14:12:12.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.612+0000 7fb26fb90640 1 -- 172.21.15.60:0/3737074685 wait complete. 2023-10-16T14:12:12.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:12:12.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.863+0000 7f28390a8640 1 Processor -- start 2023-10-16T14:12:12.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.863+0000 7f28390a8640 1 -- start start 2023-10-16T14:12:12.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28390a8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f283415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28390a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f283406c810 0x7f28340708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28390a8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2834070df0 con 0x7f2834070450 2023-10-16T14:12:12.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28390a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283415e490 con 0x7f283406c810 2023-10-16T14:12:12.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28390a8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283415aef0 con 0x7f2834159950 2023-10-16T14:12:12.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28337fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2834070450 0x7f283406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33544/0 (socket says 172.21.15.60:33544) 2023-10-16T14:12:12.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.864+0000 7f28337fe640 1 -- 172.21.15.60:0/2711605794 learned_addr learned my addr 172.21.15.60:0/2711605794 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:12.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f28388a7640 1 --2- 172.21.15.60:0/2711605794 >> v2:172.21.15.144:3300/0 conn(0x7f283406c810 0x7f28340708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2833fff640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f283415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725797591 0 0) 0x7f2834070df0 con 0x7f2834070450 2023-10-16T14:12:12.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281c005730 con 0x7f2834070450 2023-10-16T14:12:12.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578085352 0 0) 0x7f281c005730 con 0x7f2834070450 2023-10-16T14:12:12.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 msgr2=0x7f283415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:12.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f283415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 >> v2:172.21.15.144:3300/0 conn(0x7f283406c810 msgr2=0x7f28340708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:12.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 --2- 172.21.15.60:0/2711605794 >> v2:172.21.15.144:3300/0 conn(0x7f283406c810 0x7f28340708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283406cc80 con 0x7f2834070450 2023-10-16T14:12:12.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f2828001670 con 0x7f2834070450 2023-10-16T14:12:12.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.865+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 >> v1:172.21.15.60:6789/0 conn(0x7f2834070450 legacy=0x7f283406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f2832ffd640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c005b80 0x7f281c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f2832ffd640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f281c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f2832ffd640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f281c006f10 0x7f281c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283415e490 con 0x7f281c006f10 2023-10-16T14:12:12.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283415aef0 con 0x7f281c005b80 2023-10-16T14:12:12.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c00dbc0 con 0x7f2834159950 2023-10-16T14:12:12.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28388a7640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f281c006f10 0x7f281c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28337fe640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f281c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28388a7640 1 -- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 msgr2=0x7f281c0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:12.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28388a7640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f281c0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28388a7640 1 -- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c005b80 msgr2=0x7f281c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:12.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28388a7640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f281c005b80 0x7f281c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28388a7640 1 -- 172.21.15.60:0/2711605794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c005730 con 0x7f281c006f10 2023-10-16T14:12:12.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.866+0000 7f28337fe640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2834159950 0x7f281c0069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:12.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f28388a7640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f281c006f10 0x7f281c007390 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f282c009ac0 tx=0x7f281c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b26150f95ca0699 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:12.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c02c710 con 0x7f281c006f10 2023-10-16T14:12:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c02c870 con 0x7f281c006f10 2023-10-16T14:12:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f2832ffd640 1 -- 172.21.15.60:0/2711605794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c02cc00 con 0x7f281c006f10 2023-10-16T14:12:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f28390a8640 1 -- 172.21.15.60:0/2711605794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f281c006f10 msgr2=0x7f281c007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f28390a8640 1 --2- 172.21.15.60:0/2711605794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f281c006f10 0x7f281c007390 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f282c009ac0 tx=0x7f281c007a00 comp rx=0 tx=0).stop 2023-10-16T14:12:12.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f28388a7640 1 -- 172.21.15.60:0/2711605794 reap_dead start 2023-10-16T14:12:12.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.868+0000 7f28390a8640 1 -- 172.21.15.60:0/2711605794 shutdown_connections 2023-10-16T14:12:12.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.869+0000 7f28390a8640 1 -- 172.21.15.60:0/2711605794 >> 172.21.15.60:0/2711605794 conn(0x7f28340ce6a0 msgr2=0x7f28340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:12.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.869+0000 7f28390a8640 1 -- 172.21.15.60:0/2711605794 shutdown_connections 2023-10-16T14:12:12.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.869+0000 7f28390a8640 1 -- 172.21.15.60:0/2711605794 wait complete. 2023-10-16T14:12:12.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.870+0000 7f28390a8640 1 Processor -- start 2023-10-16T14:12:12.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.871+0000 7f28390a8640 1 -- start start 2023-10-16T14:12:12.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.871+0000 7f28390a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 0x7f28340aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f28390a8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28341578a0 0x7f28340c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f28390a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28340c2b40 0x7f28340c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f28390a8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28340af170 con 0x7f28341574c0 2023-10-16T14:12:12.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f28390a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28340c2810 con 0x7f28340c2b40 2023-10-16T14:12:12.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f28390a8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28340c2250 con 0x7f28341578a0 2023-10-16T14:12:12.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 0x7f28340aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 0x7f28340aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33268/0 (socket says 172.21.15.60:33268) 2023-10-16T14:12:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 -- 172.21.15.60:0/2426824582 learned_addr learned my addr 172.21.15.60:0/2426824582 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f28337fe640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28341578a0 0x7f28340c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 -- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28341578a0 msgr2=0x7f28340c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28341578a0 0x7f28340c4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 -- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28340c2b40 msgr2=0x7f28340c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:12.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28340c2b40 0x7f28340c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:12.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.872+0000 7f2833fff640 1 -- 172.21.15.60:0/2426824582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28340c2530 con 0x7f28341574c0 2023-10-16T14:12:12.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.873+0000 7f2833fff640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 0x7f28340aec30 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f282400ef40 tx=0x7f282400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:12.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.873+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824010040 con 0x7f28341574c0 2023-10-16T14:12:12.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.873+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2824009e40 con 0x7f28341574c0 2023-10-16T14:12:12.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.873+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28340af660 con 0x7f28341574c0 2023-10-16T14:12:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.873+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824003bd0 con 0x7f28341574c0 2023-10-16T14:12:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.873+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28340c67c0 con 0x7f28341574c0 2023-10-16T14:12:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.874+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28340ce050 con 0x7f28341574c0 2023-10-16T14:12:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.875+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2824003d30 con 0x7f28341574c0 2023-10-16T14:12:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.876+0000 7f2830ff9640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f282006acc0 0x7f282006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:12.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.876+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7f2824089f80 con 0x7f28341574c0 2023-10-16T14:12:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.877+0000 7f28337fe640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f282006acc0 0x7f282006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.878+0000 7f28337fe640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f282006acc0 0x7f282006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28280017a0 tx=0x7f2828009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:12.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:12.881+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2824054be0 con 0x7f28341574c0 2023-10-16T14:12:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.077+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f283415a4f0 con 0x7f28341574c0 2023-10-16T14:12:13.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.078+0000 7f2830ff9640 1 -- 172.21.15.60:0/2426824582 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2824058200 con 0x7f28341574c0 2023-10-16T14:12:13.082 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:12:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f282006acc0 msgr2=0x7f282006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f282006acc0 0x7f282006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28280017a0 tx=0x7f2828009040 comp rx=0 tx=0).stop 2023-10-16T14:12:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 msgr2=0x7f28340aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 0x7f28340aec30 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f282400ef40 tx=0x7f282400b480 comp rx=0 tx=0).stop 2023-10-16T14:12:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 shutdown_connections 2023-10-16T14:12:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f282006acc0 0x7f282006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f28341578a0 0x7f28340c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28340c2b40 0x7f28340c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 --2- 172.21.15.60:0/2426824582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28341574c0 0x7f28340aec30 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 >> 172.21.15.60:0/2426824582 conn(0x7f28340ce6a0 msgr2=0x7f28340cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 shutdown_connections 2023-10-16T14:12:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.081+0000 7f28390a8640 1 -- 172.21.15.60:0/2426824582 wait complete. 2023-10-16T14:12:13.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:12:13.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.335+0000 7f829507e640 1 Processor -- start 2023-10-16T14:12:13.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.335+0000 7f829507e640 1 -- start start 2023-10-16T14:12:13.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.336+0000 7f829507e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ad970 0x7f82900cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.336+0000 7f829507e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82900ccfb0 0x7f82900cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.336+0000 7f829507e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829015b9d0 con 0x7f829006c040 2023-10-16T14:12:13.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.336+0000 7f829507e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829015bcb0 con 0x7f82900ccfb0 2023-10-16T14:12:13.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829507e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829015bf90 con 0x7f82900ad970 2023-10-16T14:12:13.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82900ccfb0 0x7f82900cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82900ccfb0 0x7f82900cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33278/0 (socket says 172.21.15.60:33278) 2023-10-16T14:12:13.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 -- 172.21.15.60:0/769667138 learned_addr learned my addr 172.21.15.60:0/769667138 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:13.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f8287fff640 1 --2- 172.21.15.60:0/769667138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ad970 0x7f82900cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 -- 172.21.15.60:0/769667138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ad970 msgr2=0x7f82900cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:13.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 --2- 172.21.15.60:0/769667138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ad970 0x7f82900cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 -- 172.21.15.60:0/769667138 >> v1:172.21.15.144:6789/0 conn(0x7f829006c040 legacy=0x7f829006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:13.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.337+0000 7f829487d640 1 -- 172.21.15.60:0/769667138 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82900cc270 con 0x7f82900ccfb0 2023-10-16T14:12:13.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.338+0000 7f829487d640 1 --2- 172.21.15.60:0/769667138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ccfb0 0x7f82900cd440 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f8288009e80 tx=0x7f828802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad32dd9918560ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:13.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.339+0000 7f828f7fe640 1 -- 172.21.15.60:0/769667138 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828802ca50 con 0x7f82900ccfb0 2023-10-16T14:12:13.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.339+0000 7f828f7fe640 1 -- 172.21.15.60:0/769667138 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828802cbb0 con 0x7f82900ccfb0 2023-10-16T14:12:13.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.339+0000 7f828f7fe640 1 -- 172.21.15.60:0/769667138 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82880343f0 con 0x7f82900ccfb0 2023-10-16T14:12:13.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.339+0000 7f829507e640 1 -- 172.21.15.60:0/769667138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ccfb0 msgr2=0x7f82900cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.339+0000 7f829507e640 1 --2- 172.21.15.60:0/769667138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ccfb0 0x7f82900cd440 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f8288009e80 tx=0x7f828802beb0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.340+0000 7f829507e640 1 -- 172.21.15.60:0/769667138 shutdown_connections 2023-10-16T14:12:13.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.340+0000 7f829507e640 1 --2- 172.21.15.60:0/769667138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ccfb0 0x7f82900cd440 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.340+0000 7f829507e640 1 --2- 172.21.15.60:0/769667138 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ad970 0x7f82900cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.340+0000 7f829507e640 1 -- 172.21.15.60:0/769667138 >> 172.21.15.60:0/769667138 conn(0x7f82900ce6a0 msgr2=0x7f82900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:13.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.340+0000 7f829507e640 1 -- 172.21.15.60:0/769667138 shutdown_connections 2023-10-16T14:12:13.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.340+0000 7f829507e640 1 -- 172.21.15.60:0/769667138 wait complete. 2023-10-16T14:12:13.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.341+0000 7f829507e640 1 Processor -- start 2023-10-16T14:12:13.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.342+0000 7f829507e640 1 -- start start 2023-10-16T14:12:13.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829507e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829006c040 0x7f8290156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829507e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 0x7f82900ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829507e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ccfb0 0x7f82900c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829507e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8290156540 con 0x7f82900ad970 2023-10-16T14:12:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829507e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82900b0400 con 0x7f829006c040 2023-10-16T14:12:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829507e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82900b06e0 con 0x7f82900ccfb0 2023-10-16T14:12:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 0x7f82900ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 0x7f82900ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33280/0 (socket says 172.21.15.60:33280) 2023-10-16T14:12:13.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 -- 172.21.15.60:0/1661414332 learned_addr learned my addr 172.21.15.60:0/1661414332 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:13.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f828ffff640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829006c040 0x7f8290156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f829487d640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ccfb0 0x7f82900c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 -- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ccfb0 msgr2=0x7f82900c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ccfb0 0x7f82900c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 -- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829006c040 msgr2=0x7f8290156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829006c040 0x7f8290156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.343+0000 7f8287fff640 1 -- 172.21.15.60:0/1661414332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288009b30 con 0x7f82900ad970 2023-10-16T14:12:13.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.344+0000 7f8287fff640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 0x7f82900ae9f0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f827800c7f0 tx=0x7f827800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.344+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278004220 con 0x7f82900ad970 2023-10-16T14:12:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.344+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82780043a0 con 0x7f82900ad970 2023-10-16T14:12:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.344+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278007860 con 0x7f82900ad970 2023-10-16T14:12:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.344+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82900ae1f0 con 0x7f82900ad970 2023-10-16T14:12:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.344+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82900b09c0 con 0x7f82900ad970 2023-10-16T14:12:13.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.345+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829006b330 con 0x7f82900ad970 2023-10-16T14:12:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.347+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8278007bd0 con 0x7f82900ad970 2023-10-16T14:12:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.348+0000 7f828d7fa640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f827006acc0 0x7f827006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.348+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7f82780899f0 con 0x7f82900ad970 2023-10-16T14:12:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.348+0000 7f828ffff640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f827006acc0 0x7f827006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.350+0000 7f828ffff640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f827006acc0 0x7f827006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f82900ad590 tx=0x7f8280009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:13.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.352+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8278054650 con 0x7f82900ad970 2023-10-16T14:12:13.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.481+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f827808bd50 con 0x7f82900ad970 2023-10-16T14:12:13.525 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:13.520+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.555+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f829004e9b0 con 0x7f82900ad970 2023-10-16T14:12:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.556+0000 7f828d7fa640 1 -- 172.21.15.60:0/1661414332 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v258) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8278075840 con 0x7f82900ad970 2023-10-16T14:12:13.560 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:12:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.558+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f827006acc0 msgr2=0x7f827006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.558+0000 7f829507e640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f827006acc0 0x7f827006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f82900ad590 tx=0x7f8280009040 comp rx=0 tx=0).stop 2023-10-16T14:12:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.558+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 msgr2=0x7f82900ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.558+0000 7f829507e640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 0x7f82900ae9f0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f827800c7f0 tx=0x7f827800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 shutdown_connections 2023-10-16T14:12:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f827006acc0 0x7f827006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f82900ccfb0 0x7f82900c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82900ad970 0x7f82900ae9f0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 --2- 172.21.15.60:0/1661414332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f829006c040 0x7f8290156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 >> 172.21.15.60:0/1661414332 conn(0x7f82900ce6a0 msgr2=0x7f82900cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 shutdown_connections 2023-10-16T14:12:13.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.559+0000 7f829507e640 1 -- 172.21.15.60:0/1661414332 wait complete. 2023-10-16T14:12:13.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T14:12:13.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.809+0000 7f2d7a2e5640 1 Processor -- start 2023-10-16T14:12:13.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.809+0000 7f2d7a2e5640 1 -- start start 2023-10-16T14:12:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.811+0000 7f2d7a2e5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d7406fb50 0x7f2d740cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.811+0000 7f2d7a2e5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d7406c9e0 0x7f2d740ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.811+0000 7f2d7a2e5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d740ca560 con 0x7f2d740c9060 2023-10-16T14:12:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.811+0000 7f2d7a2e5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d740c9950 con 0x7f2d7406c9e0 2023-10-16T14:12:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.811+0000 7f2d7a2e5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d740c9c30 con 0x7f2d7406fb50 2023-10-16T14:12:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d792e3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2d740c9060 0x7f2d7406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33546/0 (socket says 172.21.15.60:33546) 2023-10-16T14:12:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d792e3640 1 -- 172.21.15.60:0/402687909 learned_addr learned my addr 172.21.15.60:0/402687909 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d79ae4640 1 --2- 172.21.15.60:0/402687909 >> v2:172.21.15.60:3300/0 conn(0x7f2d7406c9e0 0x7f2d740ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d78ae2640 1 --2- 172.21.15.60:0/402687909 >> v2:172.21.15.144:3300/0 conn(0x7f2d7406fb50 0x7f2d740cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d67fff640 1 -- 172.21.15.60:0/402687909 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125684515 0 0) 0x7f2d740ca560 con 0x7f2d740c9060 2023-10-16T14:12:13.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d67fff640 1 -- 172.21.15.60:0/402687909 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d60003520 con 0x7f2d740c9060 2023-10-16T14:12:13.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d79ae4640 1 -- 172.21.15.60:0/402687909 >> v2:172.21.15.144:3300/0 conn(0x7f2d7406fb50 msgr2=0x7f2d740cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:13.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d79ae4640 1 --2- 172.21.15.60:0/402687909 >> v2:172.21.15.144:3300/0 conn(0x7f2d7406fb50 0x7f2d740cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d79ae4640 1 -- 172.21.15.60:0/402687909 >> v1:172.21.15.60:6789/0 conn(0x7f2d740c9060 legacy=0x7f2d7406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.812+0000 7f2d79ae4640 1 -- 172.21.15.60:0/402687909 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d740c9f10 con 0x7f2d7406c9e0 2023-10-16T14:12:13.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.814+0000 7f2d79ae4640 1 --2- 172.21.15.60:0/402687909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d7406c9e0 0x7f2d740ccde0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f2d74164cb0 tx=0x7f2d6c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b858bbddb1cbb275 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:13.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.814+0000 7f2d67fff640 1 -- 172.21.15.60:0/402687909 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c02cc70 con 0x7f2d7406c9e0 2023-10-16T14:12:13.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.814+0000 7f2d67fff640 1 -- 172.21.15.60:0/402687909 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c02cdd0 con 0x7f2d7406c9e0 2023-10-16T14:12:13.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.814+0000 7f2d67fff640 1 -- 172.21.15.60:0/402687909 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c034640 con 0x7f2d7406c9e0 2023-10-16T14:12:13.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/402687909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d7406c9e0 msgr2=0x7f2d740ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/402687909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d7406c9e0 0x7f2d740ccde0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f2d74164cb0 tx=0x7f2d6c02bec0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/402687909 shutdown_connections 2023-10-16T14:12:13.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/402687909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d7406c9e0 0x7f2d740ccde0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/402687909 >> v2:172.21.15.144:3300/0 conn(0x7f2d7406fb50 0x7f2d740cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/402687909 >> 172.21.15.60:0/402687909 conn(0x7f2d740ce6a0 msgr2=0x7f2d740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:13.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/402687909 shutdown_connections 2023-10-16T14:12:13.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.815+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/402687909 wait complete. 2023-10-16T14:12:13.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.817+0000 7f2d7a2e5640 1 Processor -- start 2023-10-16T14:12:13.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.817+0000 7f2d7a2e5640 1 -- start start 2023-10-16T14:12:13.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d7a2e5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d7406c9e0 0x7f2d7415dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d7a2e5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d7406fb50 0x7f2d740af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d7a2e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 0x7f2d740af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d7a2e5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740c2090 con 0x7f2d740c9060 2023-10-16T14:12:13.825 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d7a2e5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740c22e0 con 0x7f2d7406c9e0 2023-10-16T14:12:13.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d7a2e5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740c25c0 con 0x7f2d7406fb50 2023-10-16T14:12:13.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d79ae4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 0x7f2d740af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d79ae4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 0x7f2d740af870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33296/0 (socket says 172.21.15.60:33296) 2023-10-16T14:12:13.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.818+0000 7f2d79ae4640 1 -- 172.21.15.60:0/1530680342 learned_addr learned my addr 172.21.15.60:0/1530680342 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:13.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d792e3640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d7406c9e0 0x7f2d7415dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d78ae2640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d7406fb50 0x7f2d740af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d79ae4640 1 -- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d7406fb50 msgr2=0x7f2d740af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d79ae4640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d7406fb50 0x7f2d740af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d79ae4640 1 -- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d7406c9e0 msgr2=0x7f2d7415dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:13.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d79ae4640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d7406c9e0 0x7f2d7415dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:13.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d79ae4640 1 -- 172.21.15.60:0/1530680342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c009a70 con 0x7f2d740c9060 2023-10-16T14:12:13.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d79ae4640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 0x7f2d740af870 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c02c4d0 tx=0x7f2d6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:13.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c03e610 con 0x7f2d740c9060 2023-10-16T14:12:13.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c03e770 con 0x7f2d740c9060 2023-10-16T14:12:13.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d740c28a0 con 0x7f2d740c9060 2023-10-16T14:12:13.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.819+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c03eb60 con 0x7f2d740c9060 2023-10-16T14:12:13.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.820+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d740c30a0 con 0x7f2d740c9060 2023-10-16T14:12:13.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.821+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d740c2dc0 con 0x7f2d740c9060 2023-10-16T14:12:13.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.822+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f2d6c03b050 con 0x7f2d740c9060 2023-10-16T14:12:13.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.823+0000 7f2d65ffb640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:13.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.823+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7f2d6c0ac190 con 0x7f2d740c9060 2023-10-16T14:12:13.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.824+0000 7f2d792e3640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:13.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.825+0000 7f2d792e3640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2d5806b020 0x7f2d5806d4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000f60 tx=0x7f2d70009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:13.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:13.828+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d740c2dc0 con 0x7f2d740c9060 2023-10-16T14:12:14.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.025+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2d740c2dc0 con 0x7f2d740c9060 2023-10-16T14:12:14.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.026+0000 7f2d65ffb640 1 -- 172.21.15.60:0/1530680342 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2d740c2dc0 con 0x7f2d740c9060 2023-10-16T14:12:14.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:12:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.028+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2d5806b020 msgr2=0x7f2d5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.028+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2d5806b020 0x7f2d5806d4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000f60 tx=0x7f2d70009040 comp rx=0 tx=0).stop 2023-10-16T14:12:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.028+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 msgr2=0x7f2d740af870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.028+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 0x7f2d740af870 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c02c4d0 tx=0x7f2d6c004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 shutdown_connections 2023-10-16T14:12:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2d5806b020 0x7f2d5806d4e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2d7406fb50 0x7f2d740af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d740c9060 0x7f2d740af870 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 --2- 172.21.15.60:0/1530680342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d7406c9e0 0x7f2d7415dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 >> 172.21.15.60:0/1530680342 conn(0x7f2d740ce6a0 msgr2=0x7f2d7406b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 shutdown_connections 2023-10-16T14:12:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.029+0000 7f2d7a2e5640 1 -- 172.21.15.60:0/1530680342 wait complete. 2023-10-16T14:12:14.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T14:12:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.277+0000 7f98083e9640 1 Processor -- start 2023-10-16T14:12:14.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.277+0000 7f98083e9640 1 -- start start 2023-10-16T14:12:14.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f98083e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f980015d150 0x7f98000ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f98083e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f980006a4b0 0x7f9800160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f98083e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98000ac890 con 0x7f9800160b90 2023-10-16T14:12:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f98083e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980015d530 con 0x7f980006a4b0 2023-10-16T14:12:14.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f98083e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9800161650 con 0x7f980015d150 2023-10-16T14:12:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f9806986640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f980006a4b0 0x7f9800160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f9806986640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f980006a4b0 0x7f9800160fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33302/0 (socket says 172.21.15.60:33302) 2023-10-16T14:12:14.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.279+0000 7f9806986640 1 -- 172.21.15.60:0/81963439 learned_addr learned my addr 172.21.15.60:0/81963439 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.280+0000 7f9806185640 1 --2- 172.21.15.60:0/81963439 >> v2:172.21.15.144:3300/0 conn(0x7f980015d150 0x7f98000ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.280+0000 7f9806986640 1 -- 172.21.15.60:0/81963439 >> v2:172.21.15.144:3300/0 conn(0x7f980015d150 msgr2=0x7f98000ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.280+0000 7f9806986640 1 --2- 172.21.15.60:0/81963439 >> v2:172.21.15.144:3300/0 conn(0x7f980015d150 0x7f98000ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.280+0000 7f9806986640 1 -- 172.21.15.60:0/81963439 >> v1:172.21.15.144:6789/0 conn(0x7f9800160b90 legacy=0x7f9800069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:14.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.280+0000 7f9806986640 1 -- 172.21.15.60:0/81963439 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9800161930 con 0x7f980006a4b0 2023-10-16T14:12:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.281+0000 7f9806986640 1 --2- 172.21.15.60:0/81963439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980006a4b0 0x7f9800160fe0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f98000b15a0 tx=0x7f97f802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575e593108eb3b61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.281+0000 7f9805183640 1 -- 172.21.15.60:0/81963439 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f802c9f0 con 0x7f980006a4b0 2023-10-16T14:12:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.281+0000 7f9805183640 1 -- 172.21.15.60:0/81963439 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f802cb50 con 0x7f980006a4b0 2023-10-16T14:12:14.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.281+0000 7f9805183640 1 -- 172.21.15.60:0/81963439 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f80343f0 con 0x7f980006a4b0 2023-10-16T14:12:14.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 -- 172.21.15.60:0/81963439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980006a4b0 msgr2=0x7f9800160fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 --2- 172.21.15.60:0/81963439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980006a4b0 0x7f9800160fe0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f98000b15a0 tx=0x7f97f802be50 comp rx=0 tx=0).stop 2023-10-16T14:12:14.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 -- 172.21.15.60:0/81963439 shutdown_connections 2023-10-16T14:12:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 --2- 172.21.15.60:0/81963439 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f980006a4b0 0x7f9800160fe0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 --2- 172.21.15.60:0/81963439 >> v2:172.21.15.144:3300/0 conn(0x7f980015d150 0x7f98000ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 -- 172.21.15.60:0/81963439 >> 172.21.15.60:0/81963439 conn(0x7f98000ce9b0 msgr2=0x7f98000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 -- 172.21.15.60:0/81963439 shutdown_connections 2023-10-16T14:12:14.291 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.282+0000 7f98083e9640 1 -- 172.21.15.60:0/81963439 wait complete. 2023-10-16T14:12:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.284+0000 7f98083e9640 1 Processor -- start 2023-10-16T14:12:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.284+0000 7f98083e9640 1 -- start start 2023-10-16T14:12:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.285+0000 7f98083e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f980006a4b0 0x7f980015f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.285+0000 7f98083e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980015d150 0x7f98000ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.285+0000 7f98083e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 0x7f98000ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.285+0000 7f98083e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98001e98a0 con 0x7f9800160b90 2023-10-16T14:12:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.285+0000 7f98083e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980015e860 con 0x7f980015d150 2023-10-16T14:12:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.285+0000 7f98083e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980015eb40 con 0x7f980006a4b0 2023-10-16T14:12:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 0x7f98000ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 0x7f98000ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33304/0 (socket says 172.21.15.60:33304) 2023-10-16T14:12:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 -- 172.21.15.60:0/4151859104 learned_addr learned my addr 172.21.15.60:0/4151859104 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806185640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f980006a4b0 0x7f980015f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9805984640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980015d150 0x7f98000ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 -- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f980006a4b0 msgr2=0x7f980015f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f980006a4b0 0x7f980015f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 -- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980015d150 msgr2=0x7f98000ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980015d150 0x7f98000ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9806986640 1 -- 172.21.15.60:0/4151859104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8009b30 con 0x7f9800160b90 2023-10-16T14:12:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.286+0000 7f9805984640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980015d150 0x7f98000ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:14.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.287+0000 7f9806986640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 0x7f98000ae780 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f97f8036040 tx=0x7f97f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.287+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8009510 con 0x7f9800160b90 2023-10-16T14:12:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.287+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f803e450 con 0x7f9800160b90 2023-10-16T14:12:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.287+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f803e7e0 con 0x7f9800160b90 2023-10-16T14:12:14.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.287+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980015ee20 con 0x7f9800160b90 2023-10-16T14:12:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.287+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980015f3e0 con 0x7f9800160b90 2023-10-16T14:12:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.288+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980004e160 con 0x7f9800160b90 2023-10-16T14:12:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.290+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f97f803e940 con 0x7f9800160b90 2023-10-16T14:12:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.291+0000 7f97f6ffd640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f97e406b020 0x7f97e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.291+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7f97f80ace60 con 0x7f9800160b90 2023-10-16T14:12:14.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.291+0000 7f9806185640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f97e406b020 0x7f97e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.293+0000 7f9806185640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f97e406b020 0x7f97e406d4e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f97fc006fd0 tx=0x7f97fc008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:14.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.296+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97f8077530 con 0x7f9800160b90 2023-10-16T14:12:14.420 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:14.415+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:14.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.501+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f98000ade10 con 0x7f9800160b90 2023-10-16T14:12:14.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.502+0000 7f97f6ffd640 1 -- 172.21.15.60:0/4151859104 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v258) v1 ==== 103+0+11 (secure 0 0 0) 0x7f97f80430a0 con 0x7f9800160b90 2023-10-16T14:12:14.506 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:12:14.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f97e406b020 msgr2=0x7f97e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f97e406b020 0x7f97e406d4e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f97fc006fd0 tx=0x7f97fc008040 comp rx=0 tx=0).stop 2023-10-16T14:12:14.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 msgr2=0x7f98000ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 0x7f98000ae780 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f97f8036040 tx=0x7f97f8004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:14.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 shutdown_connections 2023-10-16T14:12:14.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f97e406b020 0x7f97e406d4e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9800160b90 0x7f98000ae780 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f980015d150 0x7f98000ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 --2- 172.21.15.60:0/4151859104 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f980006a4b0 0x7f980015f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 >> 172.21.15.60:0/4151859104 conn(0x7f98000ce9b0 msgr2=0x7f98000c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 shutdown_connections 2023-10-16T14:12:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.505+0000 7f98083e9640 1 -- 172.21.15.60:0/4151859104 wait complete. 2023-10-16T14:12:14.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:14.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.754+0000 7f9c11a9b640 1 Processor -- start 2023-10-16T14:12:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.755+0000 7f9c11a9b640 1 -- start start 2023-10-16T14:12:14.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.756+0000 7f9c11a9b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.756+0000 7f9c11a9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c0c15ae80 0x7f9c0c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.756+0000 7f9c11a9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c0c0ccaa0 0x7f9c0c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.756+0000 7f9c11a9b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c06b810 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.756+0000 7f9c11a9b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c15ed60 con 0x7f9c0c15ae80 2023-10-16T14:12:14.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.756+0000 7f9c11a9b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0707e0 con 0x7f9c0c15aaa0 2023-10-16T14:12:14.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c0c0ccaa0 0x7f9c0c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c0c0ccaa0 0x7f9c0c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33318/0 (socket says 172.21.15.60:33318) 2023-10-16T14:12:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 -- 172.21.15.60:0/3163427855 learned_addr learned my addr 172.21.15.60:0/3163427855 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:14.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 -- 172.21.15.60:0/3163427855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 msgr2=0x7f9c0c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:14.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 --2- 172.21.15.60:0/3163427855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 -- 172.21.15.60:0/3163427855 >> v2:172.21.15.144:3300/0 conn(0x7f9c0c15ae80 msgr2=0x7f9c0c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 --2- 172.21.15.60:0/3163427855 >> v2:172.21.15.144:3300/0 conn(0x7f9c0c15ae80 0x7f9c0c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.757+0000 7f9c0bfff640 1 -- 172.21.15.60:0/3163427855 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c15ef90 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.758+0000 7f9c0bfff640 1 --2- 172.21.15.60:0/3163427855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c070f50 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0cb270 tx=0x7f9c0402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99efb4c8a8cf868c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.759+0000 7f9c0a7fc640 1 -- 172.21.15.60:0/3163427855 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0402ca80 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.759+0000 7f9c0a7fc640 1 -- 172.21.15.60:0/3163427855 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0402cbe0 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.759+0000 7f9c0a7fc640 1 -- 172.21.15.60:0/3163427855 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040343f0 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.759+0000 7f9c11a9b640 1 -- 172.21.15.60:0/3163427855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 msgr2=0x7f9c0c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.759+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/3163427855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c070f50 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0cb270 tx=0x7f9c0402bee0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 -- 172.21.15.60:0/3163427855 shutdown_connections 2023-10-16T14:12:14.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/3163427855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c070f50 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/3163427855 >> v2:172.21.15.144:3300/0 conn(0x7f9c0c15ae80 0x7f9c0c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/3163427855 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 -- 172.21.15.60:0/3163427855 >> 172.21.15.60:0/3163427855 conn(0x7f9c0c0cb410 msgr2=0x7f9c0c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:14.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 -- 172.21.15.60:0/3163427855 shutdown_connections 2023-10-16T14:12:14.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.760+0000 7f9c11a9b640 1 -- 172.21.15.60:0/3163427855 wait complete. 2023-10-16T14:12:14.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.762+0000 7f9c11a9b640 1 Processor -- start 2023-10-16T14:12:14.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.762+0000 7f9c11a9b640 1 -- start start 2023-10-16T14:12:14.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c11a9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c11a9b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c0c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c11a9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c0c15ae80 0x7f9c0c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c11a9b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0c91f0 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c11a9b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c1561b0 con 0x7f9c0c15ae80 2023-10-16T14:12:14.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c11a9b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0c5100 con 0x7f9c0c15aaa0 2023-10-16T14:12:14.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c0b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c0b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33330/0 (socket says 172.21.15.60:33330) 2023-10-16T14:12:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c0b7fe640 1 -- 172.21.15.60:0/706050964 learned_addr learned my addr 172.21.15.60:0/706050964 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c0bfff640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c0c15ae80 0x7f9c0c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c0affd640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c0c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.763+0000 7f9c0b7fe640 1 -- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 msgr2=0x7f9c0c0c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0b7fe640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c0c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0b7fe640 1 -- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c0c15ae80 msgr2=0x7f9c0c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:14.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0b7fe640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c0c15ae80 0x7f9c0c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:14.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0b7fe640 1 -- 172.21.15.60:0/706050964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04009b30 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0bfff640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c0c15ae80 0x7f9c0c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:14.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0affd640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c0c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:14.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c0b7fe640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c155c70 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b410 tx=0x7f9bfc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:14.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc011830 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc0119b0 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc011d60 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.764+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c0c93d0 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.765+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c0c9990 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.766+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c04e160 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.767+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f9bfc01a400 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.768+0000 7f9bf3fff640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9bf806b020 0x7f9bf806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:14.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.768+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7f9bfc08a180 con 0x7f9c0c0ccaa0 2023-10-16T14:12:14.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.768+0000 7f9c0affd640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9bf806b020 0x7f9bf806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:14.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.770+0000 7f9c0affd640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9bf806b020 0x7f9bf806d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000a870 tx=0x7f9c00008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:14.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:14.772+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9bfc0548e0 con 0x7f9c0c0ccaa0 2023-10-16T14:12:15.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.017+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c0c070a00 con 0x7f9c0c0ccaa0 2023-10-16T14:12:15.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.018+0000 7f9bf3fff640 1 -- 172.21.15.60:0/706050964 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6146 (secure 0 0 0) 0x7f9bfc015090 con 0x7f9c0c0ccaa0 2023-10-16T14:12:15.022 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:15.023 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:15.023 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 134 2023-10-16T14:12:15.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.021+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9bf806b020 msgr2=0x7f9bf806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.021+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9bf806b020 0x7f9bf806d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000a870 tx=0x7f9c00008040 comp rx=0 tx=0).stop 2023-10-16T14:12:15.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.021+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 msgr2=0x7f9c0c155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.021+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c155c70 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b410 tx=0x7f9bfc00b8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 shutdown_connections 2023-10-16T14:12:15.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9bf806b020 0x7f9bf806d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9c0c15aaa0 0x7f9c0c0c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c0c15ae80 0x7f9c0c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 --2- 172.21.15.60:0/706050964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0ccaa0 0x7f9c0c155c70 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 >> 172.21.15.60:0/706050964 conn(0x7f9c0c0cb410 msgr2=0x7f9c0c0b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:15.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 shutdown_connections 2023-10-16T14:12:15.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.022+0000 7f9c11a9b640 1 -- 172.21.15.60:0/706050964 wait complete. 2023-10-16T14:12:15.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:15.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.266+0000 7efbfd215640 1 Processor -- start 2023-10-16T14:12:15.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.267+0000 7efbfd215640 1 -- start start 2023-10-16T14:12:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbfd215640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efbf8159b20 0x7efbf806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbfd215640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efbf80b0b20 0x7efbf806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbfd215640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf806f7f0 con 0x7efbf8159f00 2023-10-16T14:12:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbfd215640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf806c5e0 con 0x7efbf8159b20 2023-10-16T14:12:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbfd215640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf806c8c0 con 0x7efbf80b0b20 2023-10-16T14:12:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbf6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efbf8159b20 0x7efbf806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbf6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efbf8159b20 0x7efbf806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33334/0 (socket says 172.21.15.60:33334) 2023-10-16T14:12:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.268+0000 7efbf6d9d640 1 -- 172.21.15.60:0/3771975467 learned_addr learned my addr 172.21.15.60:0/3771975467 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:15.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.269+0000 7efbf759e640 1 --2- 172.21.15.60:0/3771975467 >> v2:172.21.15.144:3300/0 conn(0x7efbf80b0b20 0x7efbf806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.269+0000 7efbf6d9d640 1 -- 172.21.15.60:0/3771975467 >> v2:172.21.15.144:3300/0 conn(0x7efbf80b0b20 msgr2=0x7efbf806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:15.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.269+0000 7efbf6d9d640 1 --2- 172.21.15.60:0/3771975467 >> v2:172.21.15.144:3300/0 conn(0x7efbf80b0b20 0x7efbf806fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.269+0000 7efbf6d9d640 1 -- 172.21.15.60:0/3771975467 >> v1:172.21.15.144:6789/0 conn(0x7efbf8159f00 legacy=0x7efbf80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.269+0000 7efbf6d9d640 1 -- 172.21.15.60:0/3771975467 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf806cba0 con 0x7efbf8159b20 2023-10-16T14:12:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.270+0000 7efbf6d9d640 1 --2- 172.21.15.60:0/3771975467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159b20 0x7efbf806bbe0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7efbf80c8bb0 tx=0x7efbe802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815524dc747b8bcc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.270+0000 7efbf659c640 1 -- 172.21.15.60:0/3771975467 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8031890 con 0x7efbf8159b20 2023-10-16T14:12:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbf659c640 1 -- 172.21.15.60:0/3771975467 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe80319f0 con 0x7efbf8159b20 2023-10-16T14:12:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbf659c640 1 -- 172.21.15.60:0/3771975467 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8031d80 con 0x7efbf8159b20 2023-10-16T14:12:15.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 -- 172.21.15.60:0/3771975467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159b20 msgr2=0x7efbf806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 --2- 172.21.15.60:0/3771975467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159b20 0x7efbf806bbe0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7efbf80c8bb0 tx=0x7efbe802b930 comp rx=0 tx=0).stop 2023-10-16T14:12:15.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 -- 172.21.15.60:0/3771975467 shutdown_connections 2023-10-16T14:12:15.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 --2- 172.21.15.60:0/3771975467 >> v2:172.21.15.144:3300/0 conn(0x7efbf80b0b20 0x7efbf806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 --2- 172.21.15.60:0/3771975467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159b20 0x7efbf806bbe0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 -- 172.21.15.60:0/3771975467 >> 172.21.15.60:0/3771975467 conn(0x7efbf80cb410 msgr2=0x7efbf80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:15.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 -- 172.21.15.60:0/3771975467 shutdown_connections 2023-10-16T14:12:15.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.271+0000 7efbfd215640 1 -- 172.21.15.60:0/3771975467 wait complete. 2023-10-16T14:12:15.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.273+0000 7efbfd215640 1 Processor -- start 2023-10-16T14:12:15.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.273+0000 7efbfd215640 1 -- start start 2023-10-16T14:12:15.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.274+0000 7efbfd215640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efbf80b0b20 0x7efbf8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.274+0000 7efbfd215640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efbf8159b20 0x7efbf81467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.274+0000 7efbfd215640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 0x7efbf80ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.274+0000 7efbfd215640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8146d10 con 0x7efbf8159f00 2023-10-16T14:12:15.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.274+0000 7efbfd215640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf815caa0 con 0x7efbf80b0b20 2023-10-16T14:12:15.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.274+0000 7efbfd215640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf815cd40 con 0x7efbf8159b20 2023-10-16T14:12:15.281 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 0x7efbf80ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 0x7efbf80ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33348/0 (socket says 172.21.15.60:33348) 2023-10-16T14:12:15.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 -- 172.21.15.60:0/1603869685 learned_addr learned my addr 172.21.15.60:0/1603869685 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:15.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 -- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efbf8159b20 msgr2=0x7efbf81467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efbf8159b20 0x7efbf81467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.282 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 -- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efbf80b0b20 msgr2=0x7efbf8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efbf80b0b20 0x7efbf8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 -- 172.21.15.60:0/1603869685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe80094d0 con 0x7efbf8159f00 2023-10-16T14:12:15.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbf759e640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 0x7efbf80ae7e0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7efbf000cc30 tx=0x7efbf000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:15.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.275+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000ce40 con 0x7efbf8159f00 2023-10-16T14:12:15.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.276+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf0003990 con 0x7efbf8159f00 2023-10-16T14:12:15.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.276+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf815d020 con 0x7efbf8159f00 2023-10-16T14:12:15.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.276+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0003d20 con 0x7efbf8159f00 2023-10-16T14:12:15.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.276+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf80c77c0 con 0x7efbf8159f00 2023-10-16T14:12:15.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.277+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf80c79e0 con 0x7efbf8159f00 2023-10-16T14:12:15.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.278+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7efbf0003e80 con 0x7efbf8159f00 2023-10-16T14:12:15.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.279+0000 7efbef7fe640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efbd806b020 0x7efbd806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.279+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7efbf008a490 con 0x7efbf8159f00 2023-10-16T14:12:15.285 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.280+0000 7efbf6d9d640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efbd806b020 0x7efbd806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.286 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.281+0000 7efbf6d9d640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efbd806b020 0x7efbd806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efbe8036000 tx=0x7efbe8030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:15.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.283+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efbf0054b80 con 0x7efbf8159f00 2023-10-16T14:12:15.309 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:15.304+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:15.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.486+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efbf804e160 con 0x7efbf8159f00 2023-10-16T14:12:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.493+0000 7efbef7fe640 1 -- 172.21.15.60:0/1603869685 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+21443 (secure 0 0 0) 0x7efbf001d090 con 0x7efbf8159f00 2023-10-16T14:12:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:15.497 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":258,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:11.447519+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efbd806b020 msgr2=0x7efbd806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efbd806b020 0x7efbd806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efbe8036000 tx=0x7efbe8030000 comp rx=0 tx=0).stop 2023-10-16T14:12:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 msgr2=0x7efbf80ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 0x7efbf80ae7e0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7efbf000cc30 tx=0x7efbf000a5c0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 shutdown_connections 2023-10-16T14:12:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efbd806b020 0x7efbd806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efbf8159b20 0x7efbf81467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf8159f00 0x7efbf80ae7e0 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 --2- 172.21.15.60:0/1603869685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efbf80b0b20 0x7efbf8143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 >> 172.21.15.60:0/1603869685 conn(0x7efbf80cb410 msgr2=0x7efbf80ca070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 shutdown_connections 2023-10-16T14:12:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.496+0000 7efbfd215640 1 -- 172.21.15.60:0/1603869685 wait complete. 2023-10-16T14:12:15.521 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:12:15.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.747+0000 7fc5c636c640 1 Processor -- start 2023-10-16T14:12:15.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.747+0000 7fc5c636c640 1 -- start start 2023-10-16T14:12:15.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.748+0000 7fc5c636c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00afef0 0x7fc5c00cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c636c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5c00b2bb0 0x7fc5c006b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c636c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c00d0180 con 0x7fc5c006aeb0 2023-10-16T14:12:15.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c636c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c006b870 con 0x7fc5c00b2bb0 2023-10-16T14:12:15.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c636c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00cfea0 con 0x7fc5c00afef0 2023-10-16T14:12:15.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c4909640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5c00b2bb0 0x7fc5c006b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c4909640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5c00b2bb0 0x7fc5c006b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33360/0 (socket says 172.21.15.60:33360) 2023-10-16T14:12:15.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5c4909640 1 -- 172.21.15.60:0/2600851119 learned_addr learned my addr 172.21.15.60:0/2600851119 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.749+0000 7fc5bffff640 1 --2- 172.21.15.60:0/2600851119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00afef0 0x7fc5c00cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.750+0000 7fc5c4909640 1 -- 172.21.15.60:0/2600851119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00afef0 msgr2=0x7fc5c00cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.750+0000 7fc5c4909640 1 --2- 172.21.15.60:0/2600851119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00afef0 0x7fc5c00cf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.750+0000 7fc5c4909640 1 -- 172.21.15.60:0/2600851119 >> v1:172.21.15.144:6789/0 conn(0x7fc5c006aeb0 legacy=0x7fc5c00b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.750+0000 7fc5c4909640 1 -- 172.21.15.60:0/2600851119 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c00b04d0 con 0x7fc5c00b2bb0 2023-10-16T14:12:15.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.751+0000 7fc5c4909640 1 --2- 172.21.15.60:0/2600851119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b2bb0 0x7fc5c006b330 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009e50 tx=0x7fc5b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab7089c02530d65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.751+0000 7fc5bf7fe640 1 -- 172.21.15.60:0/2600851119 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b802c9e0 con 0x7fc5c00b2bb0 2023-10-16T14:12:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.751+0000 7fc5bf7fe640 1 -- 172.21.15.60:0/2600851119 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b802cb40 con 0x7fc5c00b2bb0 2023-10-16T14:12:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.751+0000 7fc5bf7fe640 1 -- 172.21.15.60:0/2600851119 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80343f0 con 0x7fc5c00b2bb0 2023-10-16T14:12:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 -- 172.21.15.60:0/2600851119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b2bb0 msgr2=0x7fc5c006b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 --2- 172.21.15.60:0/2600851119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b2bb0 0x7fc5c006b330 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009e50 tx=0x7fc5b802be40 comp rx=0 tx=0).stop 2023-10-16T14:12:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 -- 172.21.15.60:0/2600851119 shutdown_connections 2023-10-16T14:12:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 --2- 172.21.15.60:0/2600851119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b2bb0 0x7fc5c006b330 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 --2- 172.21.15.60:0/2600851119 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00afef0 0x7fc5c00cf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 -- 172.21.15.60:0/2600851119 >> 172.21.15.60:0/2600851119 conn(0x7fc5c00d0ad0 msgr2=0x7fc5c00d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 -- 172.21.15.60:0/2600851119 shutdown_connections 2023-10-16T14:12:15.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.752+0000 7fc5c636c640 1 -- 172.21.15.60:0/2600851119 wait complete. 2023-10-16T14:12:15.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.754+0000 7fc5c636c640 1 Processor -- start 2023-10-16T14:12:15.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.754+0000 7fc5c636c640 1 -- start start 2023-10-16T14:12:15.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5c636c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 0x7fc5c00ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5c636c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c00afef0 0x7fc5c01f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5c636c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00b2bb0 0x7fc5c00c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5c636c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00aed20 con 0x7fc5c006aeb0 2023-10-16T14:12:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5c636c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00b3380 con 0x7fc5c00afef0 2023-10-16T14:12:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5c636c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c009ad90 con 0x7fc5c00b2bb0 2023-10-16T14:12:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 0x7fc5c00ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 0x7fc5c00ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33370/0 (socket says 172.21.15.60:33370) 2023-10-16T14:12:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.755+0000 7fc5bffff640 1 -- 172.21.15.60:0/375451431 learned_addr learned my addr 172.21.15.60:0/375451431 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5c4909640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00b2bb0 0x7fc5c00c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5bffff640 1 -- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00b2bb0 msgr2=0x7fc5c00c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5bffff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00b2bb0 0x7fc5c00c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5b7fff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c00afef0 0x7fc5c01f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5bffff640 1 -- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c00afef0 msgr2=0x7fc5c01f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5bffff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c00afef0 0x7fc5c01f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5bffff640 1 -- 172.21.15.60:0/375451431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8009b30 con 0x7fc5c006aeb0 2023-10-16T14:12:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5b7fff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c00afef0 0x7fc5c01f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.756+0000 7fc5bffff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 0x7fc5c00ae7e0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d800 tx=0x7fc5b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.757+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0005990 con 0x7fc5c006aeb0 2023-10-16T14:12:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.757+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b0005af0 con 0x7fc5c006aeb0 2023-10-16T14:12:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.757+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0005ea0 con 0x7fc5c006aeb0 2023-10-16T14:12:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.757+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c009b070 con 0x7fc5c006aeb0 2023-10-16T14:12:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.757+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c00b35d0 con 0x7fc5c006aeb0 2023-10-16T14:12:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.758+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c00b2180 con 0x7fc5c006aeb0 2023-10-16T14:12:15.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.759+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fc5b00096a0 con 0x7fc5c006aeb0 2023-10-16T14:12:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.760+0000 7fc5bd7fa640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc5a006b020 0x7fc5a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.760+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7fc5b008a1b0 con 0x7fc5c006aeb0 2023-10-16T14:12:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.760+0000 7fc5b7fff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc5a006b020 0x7fc5a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.761+0000 7fc5b7fff640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc5a006b020 0x7fc5a006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00caab0 tx=0x7fc5a8009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:15.764+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5b0054910 con 0x7fc5c006aeb0 2023-10-16T14:12:16.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.011+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5c00b37f0 con 0x7fc5c006aeb0 2023-10-16T14:12:16.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.012+0000 7fc5bd7fa640 1 -- 172.21.15.60:0/375451431 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6146 (secure 0 0 0) 0x7fc5b0057f30 con 0x7fc5c006aeb0 2023-10-16T14:12:16.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:16.017 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:16.017 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 134 2023-10-16T14:12:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc5a006b020 msgr2=0x7fc5a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc5a006b020 0x7fc5a006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00caab0 tx=0x7fc5a8009040 comp rx=0 tx=0).stop 2023-10-16T14:12:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 msgr2=0x7fc5c00ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 0x7fc5c00ae7e0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d800 tx=0x7fc5b000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 shutdown_connections 2023-10-16T14:12:16.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc5a006b020 0x7fc5a006d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5c00b2bb0 0x7fc5c00c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c00afef0 0x7fc5c01f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 --2- 172.21.15.60:0/375451431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c006aeb0 0x7fc5c00ae7e0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 >> 172.21.15.60:0/375451431 conn(0x7fc5c00d0ad0 msgr2=0x7fc5c00cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 shutdown_connections 2023-10-16T14:12:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.015+0000 7fc5c636c640 1 -- 172.21.15.60:0/375451431 wait complete. 2023-10-16T14:12:16.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:16.209 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:16.204+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:16.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.257+0000 7fc9709b0640 1 Processor -- start 2023-10-16T14:12:16.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.257+0000 7fc9709b0640 1 -- start start 2023-10-16T14:12:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.258+0000 7fc9709b0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc96c0cd0c0 0x7fc96c1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.258+0000 7fc9709b0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc96c0b0410 0x7fc96c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.258+0000 7fc9709b0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96c15e3e0 con 0x7fc96c15d3f0 2023-10-16T14:12:16.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.258+0000 7fc9709b0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c0b0de0 con 0x7fc96c0b0410 2023-10-16T14:12:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.258+0000 7fc9709b0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c0cd4a0 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc96c0b0410 0x7fc96c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc96c0b0410 0x7fc96c0b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33376/0 (socket says 172.21.15.60:33376) 2023-10-16T14:12:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc96ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc96c15d3f0 0x7fc96c15d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33556/0 (socket says 172.21.15.60:33556) 2023-10-16T14:12:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 -- 172.21.15.60:0/2683614377 learned_addr learned my addr 172.21.15.60:0/2683614377 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc96a59c640 1 --2- 172.21.15.60:0/2683614377 >> v2:172.21.15.144:3300/0 conn(0x7fc96c0cd0c0 0x7fc96c1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 -- 172.21.15.60:0/2683614377 >> v2:172.21.15.144:3300/0 conn(0x7fc96c0cd0c0 msgr2=0x7fc96c1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:16.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 --2- 172.21.15.60:0/2683614377 >> v2:172.21.15.144:3300/0 conn(0x7fc96c0cd0c0 0x7fc96c1615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 -- 172.21.15.60:0/2683614377 >> v1:172.21.15.60:6789/0 conn(0x7fc96c15d3f0 legacy=0x7fc96c15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc969d9b640 1 -- 172.21.15.60:0/2683614377 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c15e100 con 0x7fc96c0b0410 2023-10-16T14:12:16.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.259+0000 7fc96959a640 1 -- 172.21.15.60:0/2683614377 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693115943 0 0) 0x7fc96c15e3e0 con 0x7fc96c15d3f0 2023-10-16T14:12:16.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.261+0000 7fc969d9b640 1 --2- 172.21.15.60:0/2683614377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0b0410 0x7fc96c0b08a0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0cc4e0 tx=0x7fc95402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104a5f6ec455b573 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.261+0000 7fc96959a640 1 -- 172.21.15.60:0/2683614377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954031870 con 0x7fc96c0b0410 2023-10-16T14:12:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.261+0000 7fc96959a640 1 -- 172.21.15.60:0/2683614377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9540319d0 con 0x7fc96c0b0410 2023-10-16T14:12:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.261+0000 7fc96959a640 1 -- 172.21.15.60:0/2683614377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954031d60 con 0x7fc96c0b0410 2023-10-16T14:12:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 -- 172.21.15.60:0/2683614377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0b0410 msgr2=0x7fc96c0b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 --2- 172.21.15.60:0/2683614377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0b0410 0x7fc96c0b08a0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0cc4e0 tx=0x7fc95402b910 comp rx=0 tx=0).stop 2023-10-16T14:12:16.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 -- 172.21.15.60:0/2683614377 shutdown_connections 2023-10-16T14:12:16.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 --2- 172.21.15.60:0/2683614377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0b0410 0x7fc96c0b08a0 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 --2- 172.21.15.60:0/2683614377 >> v2:172.21.15.144:3300/0 conn(0x7fc96c0cd0c0 0x7fc96c1615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 -- 172.21.15.60:0/2683614377 >> 172.21.15.60:0/2683614377 conn(0x7fc96c0cb410 msgr2=0x7fc96c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:16.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 -- 172.21.15.60:0/2683614377 shutdown_connections 2023-10-16T14:12:16.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.262+0000 7fc9709b0640 1 -- 172.21.15.60:0/2683614377 wait complete. 2023-10-16T14:12:16.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.264+0000 7fc9709b0640 1 Processor -- start 2023-10-16T14:12:16.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.264+0000 7fc9709b0640 1 -- start start 2023-10-16T14:12:16.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc9709b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc96c0b0410 0x7fc96c156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc9709b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 0x7fc96c0b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc9709b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc96c15d3f0 0x7fc96c0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc9709b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0b2860 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc9709b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0c22d0 con 0x7fc96c0b0410 2023-10-16T14:12:16.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc9709b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0c25b0 con 0x7fc96c15d3f0 2023-10-16T14:12:16.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 0x7fc96c0b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 0x7fc96c0b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33382/0 (socket says 172.21.15.60:33382) 2023-10-16T14:12:16.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 -- 172.21.15.60:0/1387765218 learned_addr learned my addr 172.21.15.60:0/1387765218 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:16.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc96a59c640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc96c0b0410 0x7fc96c156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 -- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc96c15d3f0 msgr2=0x7fc96c0b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc96c15d3f0 0x7fc96c0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 -- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc96c0b0410 msgr2=0x7fc96c156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc96c0b0410 0x7fc96c156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.266+0000 7fc969d9b640 1 -- 172.21.15.60:0/1387765218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9540094d0 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc96a59c640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc96c0b0410 0x7fc96c156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:16.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc969d9b640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 0x7fc96c0b1f60 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fc954000c00 tx=0x7fc9540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:16.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954004230 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9540043b0 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c0c2890 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954004520 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.267+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c0b24a0 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.269+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c0c2130 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.270+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fc954042070 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.271+0000 7fc962ffd640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc94c06b020 0x7fc94c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.271+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7fc95402d040 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.271+0000 7fc96a59c640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc94c06b020 0x7fc94c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.272+0000 7fc96a59c640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc94c06b020 0x7fc94c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc95c007330 tx=0x7fc95c00b040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:16.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.275+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9540defe0 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.516+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc96c0c2a80 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.517+0000 7fc962ffd640 1 -- 172.21.15.60:0/1387765218 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6146 (secure 0 0 0) 0x7fc954035020 con 0x7fc96c0cd0c0 2023-10-16T14:12:16.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:16.521 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":134,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:38.244116+0000","modified":"2023-10-16T14:11:41.379569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9409":{"gid":9409,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.144:6837/296787071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":296787071},{"type":"v1","addr":"172.21.15.144:6837","nonce":296787071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:16.521 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 134 2023-10-16T14:12:16.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.519+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc94c06b020 msgr2=0x7fc94c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.519+0000 7fc9709b0640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc94c06b020 0x7fc94c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc95c007330 tx=0x7fc95c00b040 comp rx=0 tx=0).stop 2023-10-16T14:12:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 msgr2=0x7fc96c0b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 0x7fc96c0b1f60 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fc954000c00 tx=0x7fc9540046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 shutdown_connections 2023-10-16T14:12:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc94c06b020 0x7fc94c06d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc96c15d3f0 0x7fc96c0b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc96c0cd0c0 0x7fc96c0b1f60 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 --2- 172.21.15.60:0/1387765218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc96c0b0410 0x7fc96c156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 >> 172.21.15.60:0/1387765218 conn(0x7fc96c0cb410 msgr2=0x7fc96c15b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:16.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 shutdown_connections 2023-10-16T14:12:16.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.520+0000 7fc9709b0640 1 -- 172.21.15.60:0/1387765218 wait complete. 2023-10-16T14:12:16.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.764+0000 7fa311e4d640 1 Processor -- start 2023-10-16T14:12:16.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.764+0000 7fa311e4d640 1 -- start start 2023-10-16T14:12:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.766+0000 7fa311e4d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa30c06c040 0x7fa30c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.766+0000 7fa311e4d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa30c0cda80 0x7fa30c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.766+0000 7fa311e4d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30c06c960 con 0x7fa30c0ad970 2023-10-16T14:12:16.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.766+0000 7fa311e4d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30c15b9d0 con 0x7fa30c0cda80 2023-10-16T14:12:16.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.766+0000 7fa311e4d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30c15bcb0 con 0x7fa30c06c040 2023-10-16T14:12:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.766+0000 7fa31164c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa30c0cda80 0x7fa30c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa31164c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa30c0cda80 0x7fa30c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33392/0 (socket says 172.21.15.60:33392) 2023-10-16T14:12:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa31164c640 1 -- 172.21.15.60:0/145556211 learned_addr learned my addr 172.21.15.60:0/145556211 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa310e4b640 1 --2- 172.21.15.60:0/145556211 >> v2:172.21.15.144:3300/0 conn(0x7fa30c06c040 0x7fa30c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa31164c640 1 -- 172.21.15.60:0/145556211 >> v2:172.21.15.144:3300/0 conn(0x7fa30c06c040 msgr2=0x7fa30c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa31164c640 1 --2- 172.21.15.60:0/145556211 >> v2:172.21.15.144:3300/0 conn(0x7fa30c06c040 0x7fa30c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa31164c640 1 -- 172.21.15.60:0/145556211 >> v1:172.21.15.144:6789/0 conn(0x7fa30c0ad970 legacy=0x7fa30c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.767+0000 7fa31164c640 1 -- 172.21.15.60:0/145556211 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c15bf90 con 0x7fa30c0cda80 2023-10-16T14:12:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.769+0000 7fa31164c640 1 --2- 172.21.15.60:0/145556211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0cda80 0x7fa30c15ed90 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0cf350 tx=0x7fa30402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e1232580130c94 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.769+0000 7fa30b7fe640 1 -- 172.21.15.60:0/145556211 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30402ca80 con 0x7fa30c0cda80 2023-10-16T14:12:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.769+0000 7fa30b7fe640 1 -- 172.21.15.60:0/145556211 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30402cbe0 con 0x7fa30c0cda80 2023-10-16T14:12:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.769+0000 7fa30b7fe640 1 -- 172.21.15.60:0/145556211 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3040343f0 con 0x7fa30c0cda80 2023-10-16T14:12:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 -- 172.21.15.60:0/145556211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0cda80 msgr2=0x7fa30c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 --2- 172.21.15.60:0/145556211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0cda80 0x7fa30c15ed90 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0cf350 tx=0x7fa30402bee0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 -- 172.21.15.60:0/145556211 shutdown_connections 2023-10-16T14:12:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 --2- 172.21.15.60:0/145556211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0cda80 0x7fa30c15ed90 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 --2- 172.21.15.60:0/145556211 >> v2:172.21.15.144:3300/0 conn(0x7fa30c06c040 0x7fa30c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 -- 172.21.15.60:0/145556211 >> 172.21.15.60:0/145556211 conn(0x7fa30c0ce6a0 msgr2=0x7fa30c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:16.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 -- 172.21.15.60:0/145556211 shutdown_connections 2023-10-16T14:12:16.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.770+0000 7fa311e4d640 1 -- 172.21.15.60:0/145556211 wait complete. 2023-10-16T14:12:16.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.772+0000 7fa311e4d640 1 Processor -- start 2023-10-16T14:12:16.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.772+0000 7fa311e4d640 1 -- start start 2023-10-16T14:12:16.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa311e4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 0x7fa30c157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa311e4d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c0ad970 0x7fa30c1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa311e4d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0cda80 0x7fa30c0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa311e4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c0c7d40 con 0x7fa30c06c040 2023-10-16T14:12:16.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa311e4d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c0c8020 con 0x7fa30c0ad970 2023-10-16T14:12:16.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa311e4d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c0c8300 con 0x7fa30c0cda80 2023-10-16T14:12:16.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa310e4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 0x7fa30c157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa310e4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 0x7fa30c157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33398/0 (socket says 172.21.15.60:33398) 2023-10-16T14:12:16.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa310e4b640 1 -- 172.21.15.60:0/4149188880 learned_addr learned my addr 172.21.15.60:0/4149188880 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:16.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.773+0000 7fa30bfff640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c0ad970 0x7fa30c1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa31164c640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0cda80 0x7fa30c0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa310e4b640 1 -- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0cda80 msgr2=0x7fa30c0c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa310e4b640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0cda80 0x7fa30c0c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa310e4b640 1 -- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c0ad970 msgr2=0x7fa30c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa310e4b640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c0ad970 0x7fa30c1569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa310e4b640 1 -- 172.21.15.60:0/4149188880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304009b30 con 0x7fa30c06c040 2023-10-16T14:12:16.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa310e4b640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 0x7fa30c157cc0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fa30000d5f0 tx=0x7fa30000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:16.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3000125f0 con 0x7fa30c06c040 2023-10-16T14:12:16.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300012750 con 0x7fa30c06c040 2023-10-16T14:12:16.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300012b00 con 0x7fa30c06c040 2023-10-16T14:12:16.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.774+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30c0ae2a0 con 0x7fa30c06c040 2023-10-16T14:12:16.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.775+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30c1561e0 con 0x7fa30c06c040 2023-10-16T14:12:16.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.776+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30c06b330 con 0x7fa30c06c040 2023-10-16T14:12:16.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.777+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa300012c60 con 0x7fa30c06c040 2023-10-16T14:12:16.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.778+0000 7fa3097fa640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa2f006b020 0x7fa2f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:16.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.778+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7fa30008a3f0 con 0x7fa30c06c040 2023-10-16T14:12:16.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.778+0000 7fa30bfff640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa2f006b020 0x7fa2f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:16.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.780+0000 7fa30bfff640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa2f006b020 0x7fa2f006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0c50b0 tx=0x7fa2f8009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:16.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.783+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa300054bf0 con 0x7fa30c06c040 2023-10-16T14:12:16.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.985+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa30c0ae100 con 0x7fa30c06c040 2023-10-16T14:12:16.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.988+0000 7fa3097fa640 1 -- 172.21.15.60:0/4149188880 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+21443 (secure 0 0 0) 0x7fa30c0ae100 con 0x7fa30c06c040 2023-10-16T14:12:16.992 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:16.993 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":258,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:11.447519+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:11:36.185089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-16T14:11:37.268518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:16.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa2f006b020 msgr2=0x7fa2f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.995 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa2f006b020 0x7fa2f006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0c50b0 tx=0x7fa2f8009040 comp rx=0 tx=0).stop 2023-10-16T14:12:16.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 msgr2=0x7fa30c157cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:16.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 0x7fa30c157cc0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fa30000d5f0 tx=0x7fa30000dab0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 shutdown_connections 2023-10-16T14:12:16.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa2f006b020 0x7fa2f006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.996 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.991+0000 7fa311e4d640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0cda80 0x7fa30c0c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.992+0000 7fa311e4d640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c0ad970 0x7fa30c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.992+0000 7fa311e4d640 1 --2- 172.21.15.60:0/4149188880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c06c040 0x7fa30c157cc0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:16.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.992+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 >> 172.21.15.60:0/4149188880 conn(0x7fa30c0ce6a0 msgr2=0x7fa30c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:16.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.992+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 shutdown_connections 2023-10-16T14:12:16.997 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:16.992+0000 7fa311e4d640 1 -- 172.21.15.60:0/4149188880 wait complete. 2023-10-16T14:12:17.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:12:17.118 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:17.113+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:17.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.238+0000 7fbee4d0f640 1 Processor -- start 2023-10-16T14:12:17.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.239+0000 7fbee4d0f640 1 -- start start 2023-10-16T14:12:17.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.240+0000 7fbee4d0f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 0x7fbee00ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.240+0000 7fbee4d0f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 0x7fbee00ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.240+0000 7fbee4d0f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee00cb030 con 0x7fbee015ad70 2023-10-16T14:12:17.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.240+0000 7fbee4d0f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee00cae30 con 0x7fbee006a4b0 2023-10-16T14:12:17.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.240+0000 7fbee4d0f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee00c91c0 con 0x7fbee00c9ca0 2023-10-16T14:12:17.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.241+0000 7fbeddd9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 0x7fbee00ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.241+0000 7fbeded9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 0x7fbee00ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.241+0000 7fbede59c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fbee015ad70 0x7fbee0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:58494/0 (socket says 172.21.15.60:58494) 2023-10-16T14:12:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.241+0000 7fbeded9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 0x7fbee00ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:35416/0 (socket says 172.21.15.60:35416) 2023-10-16T14:12:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.241+0000 7fbeddd9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 0x7fbee00ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55694/0 (socket says 172.21.15.60:55694) 2023-10-16T14:12:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.241+0000 7fbede59c640 1 -- 172.21.15.60:0/2260445130 learned_addr learned my addr 172.21.15.60:0/2260445130 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.242+0000 7fbeded9d640 1 -- 172.21.15.60:0/2260445130 >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 msgr2=0x7fbee00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.242+0000 7fbeded9d640 1 --2- 172.21.15.60:0/2260445130 >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 0x7fbee00ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.242+0000 7fbeded9d640 1 -- 172.21.15.60:0/2260445130 >> v1:172.21.15.144:6789/0 conn(0x7fbee015ad70 legacy=0x7fbee0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.242+0000 7fbeded9d640 1 -- 172.21.15.60:0/2260445130 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee00c94a0 con 0x7fbee00c9ca0 2023-10-16T14:12:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.242+0000 7fbeddd9b640 1 --2- 172.21.15.60:0/2260445130 >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 0x7fbee00ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:17.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.243+0000 7fbeded9d640 1 --2- 172.21.15.60:0/2260445130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 0x7fbee00ca8f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009d90 tx=0x7fbed802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d14a8a9ba4987b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:17.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.244+0000 7fbedd59a640 1 -- 172.21.15.60:0/2260445130 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed802cc50 con 0x7fbee00c9ca0 2023-10-16T14:12:17.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.244+0000 7fbedd59a640 1 -- 172.21.15.60:0/2260445130 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed802cdb0 con 0x7fbee00c9ca0 2023-10-16T14:12:17.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.244+0000 7fbedd59a640 1 -- 172.21.15.60:0/2260445130 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8034660 con 0x7fbee00c9ca0 2023-10-16T14:12:17.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 -- 172.21.15.60:0/2260445130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 msgr2=0x7fbee00ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/2260445130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 0x7fbee00ca8f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009d90 tx=0x7fbed802bea0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 -- 172.21.15.60:0/2260445130 shutdown_connections 2023-10-16T14:12:17.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/2260445130 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee00c9ca0 0x7fbee00ca8f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/2260445130 >> v2:172.21.15.144:3300/0 conn(0x7fbee006a4b0 0x7fbee00ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 -- 172.21.15.60:0/2260445130 >> 172.21.15.60:0/2260445130 conn(0x7fbee00ce9b0 msgr2=0x7fbee00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 -- 172.21.15.60:0/2260445130 shutdown_connections 2023-10-16T14:12:17.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.245+0000 7fbee4d0f640 1 -- 172.21.15.60:0/2260445130 wait complete. 2023-10-16T14:12:17.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.247+0000 7fbee4d0f640 1 Processor -- start 2023-10-16T14:12:17.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.248+0000 7fbee4d0f640 1 -- start start 2023-10-16T14:12:17.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbee4d0f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee006a4b0 0x7fbee015dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbee4d0f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbee00c9ca0 0x7fbee0151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbee4d0f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 0x7fbee015c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbee4d0f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01523c0 con 0x7fbee015ad70 2023-10-16T14:12:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbee4d0f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee015ca70 con 0x7fbee00c9ca0 2023-10-16T14:12:17.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbee4d0f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee015cd10 con 0x7fbee006a4b0 2023-10-16T14:12:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbeded9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 0x7fbee015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbeded9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 0x7fbee015c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33408/0 (socket says 172.21.15.60:33408) 2023-10-16T14:12:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbeded9d640 1 -- 172.21.15.60:0/1054474182 learned_addr learned my addr 172.21.15.60:0/1054474182 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbeddd9b640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbee00c9ca0 0x7fbee0151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.249+0000 7fbede59c640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee006a4b0 0x7fbee015dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.250+0000 7fbeded9d640 1 -- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee006a4b0 msgr2=0x7fbee015dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.250+0000 7fbeded9d640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee006a4b0 0x7fbee015dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.250+0000 7fbeded9d640 1 -- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbee00c9ca0 msgr2=0x7fbee0151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.250+0000 7fbeded9d640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbee00c9ca0 0x7fbee0151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.250+0000 7fbeded9d640 1 -- 172.21.15.60:0/1054474182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed4009400 con 0x7fbee015ad70 2023-10-16T14:12:17.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.250+0000 7fbeded9d640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 0x7fbee015c530 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009eb0 tx=0x7fbed8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:17.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.251+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed803e610 con 0x7fbee015ad70 2023-10-16T14:12:17.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.251+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed803e770 con 0x7fbee015ad70 2023-10-16T14:12:17.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.251+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed803eb20 con 0x7fbee015ad70 2023-10-16T14:12:17.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.251+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed8009a70 con 0x7fbee015ad70 2023-10-16T14:12:17.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.251+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee015d2d0 con 0x7fbee015ad70 2023-10-16T14:12:17.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.252+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee004e160 con 0x7fbee015ad70 2023-10-16T14:12:17.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.253+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fbed803b050 con 0x7fbee015ad70 2023-10-16T14:12:17.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.255+0000 7fbeceffd640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fbec006b020 0x7fbec006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.255+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7433+0+0 (secure 0 0 0) 0x7fbed80ac910 con 0x7fbee015ad70 2023-10-16T14:12:17.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.256+0000 7fbede59c640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fbec006b020 0x7fbec006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.257+0000 7fbede59c640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fbec006b020 0x7fbec006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbee00c9190 tx=0x7fbed0009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:17.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.260+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbed8076fe0 con 0x7fbee015ad70 2023-10-16T14:12:17.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.510+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbee00c9750 con 0x7fbee015ad70 2023-10-16T14:12:17.514 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:12:17.511+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:12:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.514+0000 7fbeceffd640 1 -- 172.21.15.60:0/1054474182 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v135) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbed807a600 con 0x7fbee015ad70 2023-10-16T14:12:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:12:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.517+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fbec006b020 msgr2=0x7fbec006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.517+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fbec006b020 0x7fbec006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbee00c9190 tx=0x7fbed0009040 comp rx=0 tx=0).stop 2023-10-16T14:12:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.517+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 msgr2=0x7fbee015c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.517+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 0x7fbee015c530 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009eb0 tx=0x7fbed8004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 shutdown_connections 2023-10-16T14:12:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fbec006b020 0x7fbec006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbee015ad70 0x7fbee015c530 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbee00c9ca0 0x7fbee0151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 --2- 172.21.15.60:0/1054474182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbee006a4b0 0x7fbee015dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 >> 172.21.15.60:0/1054474182 conn(0x7fbee00ce9b0 msgr2=0x7fbee00cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 shutdown_connections 2023-10-16T14:12:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.518+0000 7fbee4d0f640 1 -- 172.21.15.60:0/1054474182 wait complete. 2023-10-16T14:12:17.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:12:17.556 INFO:tasks.ceph.mds.b.smithi144.stderr: -4337> 2023-10-16T14:11:42.853+0000 7fb338a3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:12:17.645 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:12:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.758+0000 7fc4b2a59640 1 Processor -- start 2023-10-16T14:12:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.759+0000 7fc4b2a59640 1 -- start start 2023-10-16T14:12:17.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.760+0000 7fc4b2a59640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4ac159950 0x7fc4ac15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.760+0000 7fc4b2a59640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4ac070450 0x7fc4ac0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.760+0000 7fc4b2a59640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac15aef0 con 0x7fc4ac06c100 2023-10-16T14:12:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.760+0000 7fc4b2a59640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac070e20 con 0x7fc4ac070450 2023-10-16T14:12:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.760+0000 7fc4b2a59640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac15e490 con 0x7fc4ac159950 2023-10-16T14:12:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1256640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4ac070450 0x7fc4ac0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b2258640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc4ac06c100 0x7fc4ac06c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33566/0 (socket says 172.21.15.60:33566) 2023-10-16T14:12:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1256640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc4ac070450 0x7fc4ac0708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33424/0 (socket says 172.21.15.60:33424) 2023-10-16T14:12:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b2258640 1 -- 172.21.15.60:0/2563917414 learned_addr learned my addr 172.21.15.60:0/2563917414 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1a57640 1 --2- 172.21.15.60:0/2563917414 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac159950 0x7fc4ac15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1256640 1 -- 172.21.15.60:0/2563917414 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac159950 msgr2=0x7fc4ac15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1256640 1 --2- 172.21.15.60:0/2563917414 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac159950 0x7fc4ac15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1256640 1 -- 172.21.15.60:0/2563917414 >> v1:172.21.15.60:6789/0 conn(0x7fc4ac06c100 legacy=0x7fc4ac06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b1256640 1 -- 172.21.15.60:0/2563917414 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac15d910 con 0x7fc4ac070450 2023-10-16T14:12:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.761+0000 7fc4b0a55640 1 -- 172.21.15.60:0/2563917414 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781270925 0 0) 0x7fc4a4001490 con 0x7fc4ac06c100 2023-10-16T14:12:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.763+0000 7fc4b1256640 1 --2- 172.21.15.60:0/2563917414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac070450 0x7fc4ac0708e0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0ca950 tx=0x7fc49c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9858c7c2f7e4ef2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.763+0000 7fc4b0a55640 1 -- 172.21.15.60:0/2563917414 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c02e070 con 0x7fc4ac070450 2023-10-16T14:12:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.763+0000 7fc4b0a55640 1 -- 172.21.15.60:0/2563917414 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c0315a0 con 0x7fc4ac070450 2023-10-16T14:12:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.763+0000 7fc4b0a55640 1 -- 172.21.15.60:0/2563917414 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c031930 con 0x7fc4ac070450 2023-10-16T14:12:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.763+0000 7fc4b2a59640 1 -- 172.21.15.60:0/2563917414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac070450 msgr2=0x7fc4ac0708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.763+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/2563917414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac070450 0x7fc4ac0708e0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0ca950 tx=0x7fc49c02b840 comp rx=0 tx=0).stop 2023-10-16T14:12:17.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.764+0000 7fc4b2a59640 1 -- 172.21.15.60:0/2563917414 shutdown_connections 2023-10-16T14:12:17.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.764+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/2563917414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac070450 0x7fc4ac0708e0 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.764+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/2563917414 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac159950 0x7fc4ac15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.764+0000 7fc4b2a59640 1 -- 172.21.15.60:0/2563917414 >> 172.21.15.60:0/2563917414 conn(0x7fc4ac0ce6a0 msgr2=0x7fc4ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.764+0000 7fc4b2a59640 1 -- 172.21.15.60:0/2563917414 shutdown_connections 2023-10-16T14:12:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.764+0000 7fc4b2a59640 1 -- 172.21.15.60:0/2563917414 wait complete. 2023-10-16T14:12:17.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.765+0000 7fc4b2a59640 1 Processor -- start 2023-10-16T14:12:17.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.766+0000 7fc4b2a59640 1 -- start start 2023-10-16T14:12:17.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2a59640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac06c100 0x7fc4ac15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2a59640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4ac070450 0x7fc4ac1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2a59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 0x7fc4ac14cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2a59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac14fd60 con 0x7fc4ac159950 2023-10-16T14:12:17.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2a59640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15cf30 con 0x7fc4ac06c100 2023-10-16T14:12:17.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2a59640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15c960 con 0x7fc4ac070450 2023-10-16T14:12:17.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2258640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 0x7fc4ac14cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2258640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 0x7fc4ac14cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33426/0 (socket says 172.21.15.60:33426) 2023-10-16T14:12:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b2258640 1 -- 172.21.15.60:0/902180285 learned_addr learned my addr 172.21.15.60:0/902180285 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b1256640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4ac070450 0x7fc4ac1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.767+0000 7fc4b1a57640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac06c100 0x7fc4ac15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2258640 1 -- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4ac070450 msgr2=0x7fc4ac1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2258640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4ac070450 0x7fc4ac1552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2258640 1 -- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac06c100 msgr2=0x7fc4ac15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2258640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac06c100 0x7fc4ac15b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2258640 1 -- 172.21.15.60:0/902180285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c009400 con 0x7fc4ac159950 2023-10-16T14:12:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b1a57640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac06c100 0x7fc4ac15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2258640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 0x7fc4ac14cb90 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c7d0 tx=0x7fc4a400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400a640 con 0x7fc4ac159950 2023-10-16T14:12:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a400a7c0 con 0x7fc4ac159950 2023-10-16T14:12:17.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400abb0 con 0x7fc4ac159950 2023-10-16T14:12:17.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac15cbe0 con 0x7fc4ac159950 2023-10-16T14:12:17.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.768+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac155c20 con 0x7fc4ac159950 2023-10-16T14:12:17.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.770+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac155e10 con 0x7fc4ac159950 2023-10-16T14:12:17.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.771+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fc4a400ad10 con 0x7fc4ac159950 2023-10-16T14:12:17.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.772+0000 7fc4a27fc640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc49006b020 0x7fc49006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:17.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.772+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7519+0+0 (secure 0 0 0) 0x7fc4a4089440 con 0x7fc4ac159950 2023-10-16T14:12:17.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.773+0000 7fc4b1a57640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc49006b020 0x7fc49006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:17.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.774+0000 7fc4b1a57640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc49006b020 0x7fc49006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8006fd0 tx=0x7fc4a8008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:17.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:17.777+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4a4053b50 con 0x7fc4ac159950 2023-10-16T14:12:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.022+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc4ac15aef0 con 0x7fc4ac159950 2023-10-16T14:12:18.034 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:18.030+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:18.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.518+0000 7fc4a27fc640 1 -- 172.21.15.60:0/902180285 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v136) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc4ac15aef0 con 0x7fc4ac159950 2023-10-16T14:12:18.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc49006b020 msgr2=0x7fc49006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc49006b020 0x7fc49006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8006fd0 tx=0x7fc4a8008040 comp rx=0 tx=0).stop 2023-10-16T14:12:18.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 msgr2=0x7fc4ac14cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 0x7fc4ac14cb90 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c7d0 tx=0x7fc4a400cc90 comp rx=0 tx=0).stop 2023-10-16T14:12:18.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 shutdown_connections 2023-10-16T14:12:18.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fc49006b020 0x7fc49006d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc4ac070450 0x7fc4ac1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac159950 0x7fc4ac14cb90 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 --2- 172.21.15.60:0/902180285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac06c100 0x7fc4ac15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 >> 172.21.15.60:0/902180285 conn(0x7fc4ac0ce6a0 msgr2=0x7fc4ac0af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:18.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 shutdown_connections 2023-10-16T14:12:18.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.522+0000 7fc4b2a59640 1 -- 172.21.15.60:0/902180285 wait complete. 2023-10-16T14:12:18.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:12:18.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.764+0000 7efe2a1b0640 1 Processor -- start 2023-10-16T14:12:18.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.764+0000 7efe2a1b0640 1 -- start start 2023-10-16T14:12:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe2a1b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 0x7efe240ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe2a1b0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240cb890 con 0x7efe2415af90 2023-10-16T14:12:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe2a1b0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240cbac0 con 0x7efe240cda30 2023-10-16T14:12:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe2a1b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240cef80 con 0x7efe240cfae0 2023-10-16T14:12:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe237fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efe2415af90 0x7efe240cd320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33572/0 (socket says 172.21.15.60:33572) 2023-10-16T14:12:18.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe237fe640 1 -- 172.21.15.60:0/2710867477 learned_addr learned my addr 172.21.15.60:0/2710867477 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:18.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe23fff640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 0x7efe240ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:18.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.766+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253928610 0 0) 0x7efe240cb890 con 0x7efe2415af90 2023-10-16T14:12:18.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003520 con 0x7efe2415af90 2023-10-16T14:12:18.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421900266 0 0) 0x7efe0c003520 con 0x7efe2415af90 2023-10-16T14:12:18.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 msgr2=0x7efe240ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:18.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 0x7efe240ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 >> v1:172.21.15.144:6789/0 conn(0x7efe240cda30 legacy=0x7efe240cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240cbda0 con 0x7efe2415af90 2023-10-16T14:12:18.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7efe14001670 con 0x7efe2415af90 2023-10-16T14:12:18.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 >> v1:172.21.15.60:6789/0 conn(0x7efe2415af90 legacy=0x7efe240cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c003a10 0x7efe0c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 0x7efe0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c004d00 0x7efe0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240cef80 con 0x7efe0c003a10 2023-10-16T14:12:18.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00bb10 con 0x7efe0c004d00 2023-10-16T14:12:18.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.767+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00bcd0 con 0x7efe240cfae0 2023-10-16T14:12:18.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.768+0000 7efe237fe640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c003a10 0x7efe0c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:18.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.768+0000 7efe237fe640 1 -- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 msgr2=0x7efe0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:18.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.768+0000 7efe237fe640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe240cfae0 0x7efe0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.768+0000 7efe237fe640 1 -- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c004d00 msgr2=0x7efe0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:18.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.768+0000 7efe237fe640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c004d00 0x7efe0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.768+0000 7efe237fe640 1 -- 172.21.15.60:0/2710867477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c003520 con 0x7efe0c003a10 2023-10-16T14:12:18.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.769+0000 7efe237fe640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c003a10 0x7efe0c003e30 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7efe14001eb0 tx=0x7efe0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de44c56b7105f87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:18.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.769+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14005ad0 con 0x7efe0c003a10 2023-10-16T14:12:18.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.769+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14005c30 con 0x7efe0c003a10 2023-10-16T14:12:18.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.769+0000 7efe227fc640 1 -- 172.21.15.60:0/2710867477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1402c7c0 con 0x7efe0c003a10 2023-10-16T14:12:18.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe2a1b0640 1 -- 172.21.15.60:0/2710867477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c003a10 msgr2=0x7efe0c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/2710867477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe0c003a10 0x7efe0c003e30 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7efe14001eb0 tx=0x7efe0c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe23fff640 1 -- 172.21.15.60:0/2710867477 reap_dead start 2023-10-16T14:12:18.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe2a1b0640 1 -- 172.21.15.60:0/2710867477 shutdown_connections 2023-10-16T14:12:18.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe2a1b0640 1 -- 172.21.15.60:0/2710867477 >> 172.21.15.60:0/2710867477 conn(0x7efe24099e30 msgr2=0x7efe240d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:18.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe2a1b0640 1 -- 172.21.15.60:0/2710867477 shutdown_connections 2023-10-16T14:12:18.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.770+0000 7efe2a1b0640 1 -- 172.21.15.60:0/2710867477 wait complete. 2023-10-16T14:12:18.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.772+0000 7efe2a1b0640 1 Processor -- start 2023-10-16T14:12:18.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.772+0000 7efe2a1b0640 1 -- start start 2023-10-16T14:12:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe2a1b0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003a10 0x7efe2415b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe2a1b0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe24146f40 0x7efe24151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe2a1b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 0x7efe24151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe2a1b0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2406c820 con 0x7efe241473c0 2023-10-16T14:12:18.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe2a1b0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe241520c0 con 0x7efe0c003a10 2023-10-16T14:12:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe2a1b0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe241525d0 con 0x7efe24146f40 2023-10-16T14:12:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 0x7efe24151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe23fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 0x7efe24151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33444/0 (socket says 172.21.15.60:33444) 2023-10-16T14:12:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.773+0000 7efe23fff640 1 -- 172.21.15.60:0/112833085 learned_addr learned my addr 172.21.15.60:0/112833085 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe23fff640 1 -- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe24146f40 msgr2=0x7efe24151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe23fff640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe24146f40 0x7efe24151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe23fff640 1 -- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003a10 msgr2=0x7efe2415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe23fff640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003a10 0x7efe2415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe23fff640 1 -- 172.21.15.60:0/112833085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2406bba0 con 0x7efe241473c0 2023-10-16T14:12:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe23fff640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 0x7efe24151b80 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7efe240d03c0 tx=0x7efe1c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c010040 con 0x7efe241473c0 2023-10-16T14:12:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00bb10 con 0x7efe241473c0 2023-10-16T14:12:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c014400 con 0x7efe241473c0 2023-10-16T14:12:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.774+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2406be80 con 0x7efe241473c0 2023-10-16T14:12:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.775+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2406c440 con 0x7efe241473c0 2023-10-16T14:12:18.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.775+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2404e160 con 0x7efe241473c0 2023-10-16T14:12:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.777+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7efe1c014560 con 0x7efe241473c0 2023-10-16T14:12:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.778+0000 7efe0bfff640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efe1006b020 0x7efe1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.778+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7519+0+0 (secure 0 0 0) 0x7efe1c08a6d0 con 0x7efe241473c0 2023-10-16T14:12:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.778+0000 7efe237fe640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efe1006b020 0x7efe1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.779+0000 7efe237fe640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efe1006b020 0x7efe1006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efe14001eb0 tx=0x7efe14007000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:18.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.782+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe1c054cd0 con 0x7efe241473c0 2023-10-16T14:12:18.970 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:18.965+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:18.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:18.989+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe240ce450 con 0x7efe241473c0 2023-10-16T14:12:19.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.521+0000 7efe0bfff640 1 -- 172.21.15.60:0/112833085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v260) v1 ==== 181+0+0 (secure 0 0 0) 0x7efe1c0582f0 con 0x7efe241473c0 2023-10-16T14:12:19.526 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:12:19.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efe1006b020 msgr2=0x7efe1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:19.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efe1006b020 0x7efe1006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efe14001eb0 tx=0x7efe14007000 comp rx=0 tx=0).stop 2023-10-16T14:12:19.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 msgr2=0x7efe24151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:19.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 0x7efe24151b80 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7efe240d03c0 tx=0x7efe1c00b480 comp rx=0 tx=0).stop 2023-10-16T14:12:19.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 shutdown_connections 2023-10-16T14:12:19.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7efe1006b020 0x7efe1006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efe24146f40 0x7efe24151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe241473c0 0x7efe24151b80 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 --2- 172.21.15.60:0/112833085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe0c003a10 0x7efe2415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.525+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 >> 172.21.15.60:0/112833085 conn(0x7efe24099e30 msgr2=0x7efe240cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:19.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.527+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 shutdown_connections 2023-10-16T14:12:19.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.528+0000 7efe2a1b0640 1 -- 172.21.15.60:0/112833085 wait complete. 2023-10-16T14:12:19.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:12:19.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.769+0000 7fa7c152c640 1 Processor -- start 2023-10-16T14:12:19.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.769+0000 7fa7c152c640 1 -- start start 2023-10-16T14:12:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.771+0000 7fa7c152c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc1599c0 0x7fa7bc0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.771+0000 7fa7c152c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7bc159da0 0x7fa7bc15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.771+0000 7fa7c152c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc15dd10 con 0x7fa7bc0cce20 2023-10-16T14:12:19.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.771+0000 7fa7c152c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc15dfb0 con 0x7fa7bc159da0 2023-10-16T14:12:19.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.771+0000 7fa7c152c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc15e290 con 0x7fa7bc1599c0 2023-10-16T14:12:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7b3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7bc159da0 0x7fa7bc15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7bb7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa7bc0cce20 0x7fa7bc06c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47530/0 (socket says 172.21.15.60:47530) 2023-10-16T14:12:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7bb7fe640 1 -- 172.21.15.60:0/1902336253 learned_addr learned my addr 172.21.15.60:0/1902336253 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7b3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7bc159da0 0x7fa7bc15d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38012/0 (socket says 172.21.15.60:38012) 2023-10-16T14:12:19.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7baffd640 1 --2- 172.21.15.60:0/1902336253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc1599c0 0x7fa7bc0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7b3fff640 1 -- 172.21.15.60:0/1902336253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc1599c0 msgr2=0x7fa7bc0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7b3fff640 1 --2- 172.21.15.60:0/1902336253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc1599c0 0x7fa7bc0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7b3fff640 1 -- 172.21.15.60:0/1902336253 >> v1:172.21.15.60:6789/0 conn(0x7fa7bc0cce20 legacy=0x7fa7bc06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:19.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.772+0000 7fa7b3fff640 1 -- 172.21.15.60:0/1902336253 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc15e570 con 0x7fa7bc159da0 2023-10-16T14:12:19.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.773+0000 7fa7b3fff640 1 --2- 172.21.15.60:0/1902336253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc15d7d0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40097a0 tx=0x7fa7a402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61864e9c6b0e395f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:19.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.774+0000 7fa7ba7fc640 1 -- 172.21.15.60:0/1902336253 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a402d040 con 0x7fa7bc159da0 2023-10-16T14:12:19.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.774+0000 7fa7ba7fc640 1 -- 172.21.15.60:0/1902336253 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4031810 con 0x7fa7bc159da0 2023-10-16T14:12:19.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.774+0000 7fa7ba7fc640 1 -- 172.21.15.60:0/1902336253 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4031ba0 con 0x7fa7bc159da0 2023-10-16T14:12:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.774+0000 7fa7c152c640 1 -- 172.21.15.60:0/1902336253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 msgr2=0x7fa7bc15d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.774+0000 7fa7c152c640 1 --2- 172.21.15.60:0/1902336253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc15d7d0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40097a0 tx=0x7fa7a402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.775+0000 7fa7c152c640 1 -- 172.21.15.60:0/1902336253 shutdown_connections 2023-10-16T14:12:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.775+0000 7fa7c152c640 1 --2- 172.21.15.60:0/1902336253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc15d7d0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.775+0000 7fa7c152c640 1 --2- 172.21.15.60:0/1902336253 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc1599c0 0x7fa7bc0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.775+0000 7fa7c152c640 1 -- 172.21.15.60:0/1902336253 >> 172.21.15.60:0/1902336253 conn(0x7fa7bc0cb360 msgr2=0x7fa7bc0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.775+0000 7fa7c152c640 1 -- 172.21.15.60:0/1902336253 shutdown_connections 2023-10-16T14:12:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.775+0000 7fa7c152c640 1 -- 172.21.15.60:0/1902336253 wait complete. 2023-10-16T14:12:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.776+0000 7fa7c152c640 1 Processor -- start 2023-10-16T14:12:19.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.777+0000 7fa7c152c640 1 -- start start 2023-10-16T14:12:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.777+0000 7fa7c152c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc0cce20 0x7fa7bc1489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.777+0000 7fa7c152c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc1599c0 0x7fa7bc0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.777+0000 7fa7c152c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc0b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.777+0000 7fa7c152c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b2430 con 0x7fa7bc159da0 2023-10-16T14:12:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7c152c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b4350 con 0x7fa7bc1599c0 2023-10-16T14:12:19.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7c152c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b3b10 con 0x7fa7bc0cce20 2023-10-16T14:12:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc0b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc0b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38014/0 (socket says 172.21.15.60:38014) 2023-10-16T14:12:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 -- 172.21.15.60:0/224500700 learned_addr learned my addr 172.21.15.60:0/224500700 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7b3fff640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc1599c0 0x7fa7bc0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7baffd640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc0cce20 0x7fa7bc1489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 -- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc0cce20 msgr2=0x7fa7bc1489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc0cce20 0x7fa7bc1489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 -- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc1599c0 msgr2=0x7fa7bc0c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc1599c0 0x7fa7bc0c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.778+0000 7fa7bb7fe640 1 -- 172.21.15.60:0/224500700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4009400 con 0x7fa7bc159da0 2023-10-16T14:12:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.779+0000 7fa7bb7fe640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc0b1590 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400cbf0 tx=0x7fa7b400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.779+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400ad40 con 0x7fa7bc159da0 2023-10-16T14:12:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.779+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b4012400 con 0x7fa7bc159da0 2023-10-16T14:12:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.779+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400aea0 con 0x7fa7bc159da0 2023-10-16T14:12:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.779+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc0b2090 con 0x7fa7bc159da0 2023-10-16T14:12:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.779+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc0b4070 con 0x7fa7bc159da0 2023-10-16T14:12:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.780+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc0b2570 con 0x7fa7bc159da0 2023-10-16T14:12:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.781+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa7b4007070 con 0x7fa7bc159da0 2023-10-16T14:12:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.782+0000 7fa7b37fe640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79c06b020 0x7fa79c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.782+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 7081+0+0 (secure 0 0 0) 0x7fa7b4089dd0 con 0x7fa7bc159da0 2023-10-16T14:12:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.783+0000 7fa7baffd640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79c06b020 0x7fa79c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.784+0000 7fa7baffd640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79c06b020 0x7fa79c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:19.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.787+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa7bc0b2570 con 0x7fa7bc159da0 2023-10-16T14:12:19.957 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:19.952+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:20.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:19.996+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa7bc0b2570 con 0x7fa7bc159da0 2023-10-16T14:12:20.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.541+0000 7fa7b37fe640 1 -- 172.21.15.60:0/224500700 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v261) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa7bc0b2570 con 0x7fa7bc159da0 2023-10-16T14:12:20.546 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T14:12:20.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79c06b020 msgr2=0x7fa79c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:20.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79c06b020 0x7fa79c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).stop 2023-10-16T14:12:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 msgr2=0x7fa7bc0b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc0b1590 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400cbf0 tx=0x7fa7b400a430 comp rx=0 tx=0).stop 2023-10-16T14:12:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 shutdown_connections 2023-10-16T14:12:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79c06b020 0x7fa79c06d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7bc159da0 0x7fa7bc0b1590 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc1599c0 0x7fa7bc0c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 --2- 172.21.15.60:0/224500700 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7bc0cce20 0x7fa7bc1489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.545+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 >> 172.21.15.60:0/224500700 conn(0x7fa7bc0cb360 msgr2=0x7fa7bc0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:20.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.546+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 shutdown_connections 2023-10-16T14:12:20.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.546+0000 7fa7c152c640 1 -- 172.21.15.60:0/224500700 wait complete. 2023-10-16T14:12:20.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.792+0000 7fb57a9f4640 1 Processor -- start 2023-10-16T14:12:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.792+0000 7fb57a9f4640 1 -- start start 2023-10-16T14:12:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.793+0000 7fb57a9f4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb5740c9060 0x7fb57406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.793+0000 7fb57a9f4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 0x7fb57406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.793+0000 7fb57a9f4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5740ca560 con 0x7fb57406c9e0 2023-10-16T14:12:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.793+0000 7fb57a9f4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5740c9950 con 0x7fb5740c9060 2023-10-16T14:12:20.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.793+0000 7fb57a9f4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c9c30 con 0x7fb5740c9440 2023-10-16T14:12:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.794+0000 7fb57a1f3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 0x7fb57406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.794+0000 7fb5799f2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb5740c9060 0x7fb57406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.794+0000 7fb57a1f3640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 0x7fb57406f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:36320/0 (socket says 172.21.15.60:36320) 2023-10-16T14:12:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.794+0000 7fb57a1f3640 1 -- 172.21.15.60:0/3772846557 learned_addr learned my addr 172.21.15.60:0/3772846557 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.795+0000 7fb57a1f3640 1 -- 172.21.15.60:0/3772846557 >> v2:172.21.15.144:3300/0 conn(0x7fb5740c9060 msgr2=0x7fb57406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.795+0000 7fb57a1f3640 1 --2- 172.21.15.60:0/3772846557 >> v2:172.21.15.144:3300/0 conn(0x7fb5740c9060 0x7fb57406f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.795+0000 7fb57a1f3640 1 -- 172.21.15.60:0/3772846557 >> v1:172.21.15.144:6789/0 conn(0x7fb57406c9e0 legacy=0x7fb574164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.795+0000 7fb57a1f3640 1 -- 172.21.15.60:0/3772846557 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5740c9f10 con 0x7fb5740c9440 2023-10-16T14:12:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.795+0000 7fb5799f2640 1 --2- 172.21.15.60:0/3772846557 >> v2:172.21.15.144:3300/0 conn(0x7fb5740c9060 0x7fb57406f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.797+0000 7fb57a1f3640 1 --2- 172.21.15.60:0/3772846557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 0x7fb57406f980 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009870 tx=0x7fb56c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b9ee67a2be3754 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.797+0000 7fb5789f0640 1 -- 172.21.15.60:0/3772846557 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c030840 con 0x7fb5740c9440 2023-10-16T14:12:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.797+0000 7fb5789f0640 1 -- 172.21.15.60:0/3772846557 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c0309a0 con 0x7fb5740c9440 2023-10-16T14:12:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.797+0000 7fb5789f0640 1 -- 172.21.15.60:0/3772846557 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c030d30 con 0x7fb5740c9440 2023-10-16T14:12:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 -- 172.21.15.60:0/3772846557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 msgr2=0x7fb57406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/3772846557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 0x7fb57406f980 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009870 tx=0x7fb56c02b980 comp rx=0 tx=0).stop 2023-10-16T14:12:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 -- 172.21.15.60:0/3772846557 shutdown_connections 2023-10-16T14:12:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/3772846557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5740c9440 0x7fb57406f980 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/3772846557 >> v2:172.21.15.144:3300/0 conn(0x7fb5740c9060 0x7fb57406f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 -- 172.21.15.60:0/3772846557 >> 172.21.15.60:0/3772846557 conn(0x7fb5740ce6a0 msgr2=0x7fb5740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 -- 172.21.15.60:0/3772846557 shutdown_connections 2023-10-16T14:12:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.798+0000 7fb57a9f4640 1 -- 172.21.15.60:0/3772846557 wait complete. 2023-10-16T14:12:20.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.801+0000 7fb57a9f4640 1 Processor -- start 2023-10-16T14:12:20.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.801+0000 7fb57a9f4640 1 -- start start 2023-10-16T14:12:20.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb57a9f4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb57406c9e0 0x7fb5740bd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:20.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb57a9f4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 0x7fb574152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:20.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb57a9f4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5740c9440 0x7fb5741f9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:20.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb57a9f4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c6b10 con 0x7fb5740c9060 2023-10-16T14:12:20.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb57a9f4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c5c80 con 0x7fb5740c9440 2023-10-16T14:12:20.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb57a9f4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740bd7a0 con 0x7fb57406c9e0 2023-10-16T14:12:20.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb5791f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 0x7fb574152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:20.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb5791f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 0x7fb574152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38018/0 (socket says 172.21.15.60:38018) 2023-10-16T14:12:20.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.802+0000 7fb5791f1640 1 -- 172.21.15.60:0/11101508 learned_addr learned my addr 172.21.15.60:0/11101508 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:20.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5799f2640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb57406c9e0 0x7fb5740bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:20.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb57a1f3640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5740c9440 0x7fb5741f9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5791f1640 1 -- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb57406c9e0 msgr2=0x7fb5740bd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5791f1640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb57406c9e0 0x7fb5740bd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5791f1640 1 -- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5740c9440 msgr2=0x7fb5741f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5791f1640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5740c9440 0x7fb5741f9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5791f1640 1 -- 172.21.15.60:0/11101508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb570009a70 con 0x7fb5740c9060 2023-10-16T14:12:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.803+0000 7fb5791f1640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 0x7fb574152af0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fb56400c400 tx=0x7fb56400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.804+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5640052d0 con 0x7fb5740c9060 2023-10-16T14:12:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.804+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb564005430 con 0x7fb5740c9060 2023-10-16T14:12:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.804+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564003830 con 0x7fb5740c9060 2023-10-16T14:12:20.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.804+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c0094d0 con 0x7fb5740c9060 2023-10-16T14:12:20.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.804+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5740c6170 con 0x7fb5740c9060 2023-10-16T14:12:20.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.805+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57404e9b0 con 0x7fb5740c9060 2023-10-16T14:12:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.806+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fb564003990 con 0x7fb5740c9060 2023-10-16T14:12:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.808+0000 7fb56a7fc640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb55806b020 0x7fb55806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.808+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7fb564089320 con 0x7fb5740c9060 2023-10-16T14:12:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.808+0000 7fb5799f2640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb55806b020 0x7fb55806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.809+0000 7fb5799f2640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb55806b020 0x7fb55806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb5741f9a20 tx=0x7fb57000b040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:20.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:20.812+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb564053d90 con 0x7fb5740c9060 2023-10-16T14:12:20.927 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:20.922+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:21.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.013+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5740b4620 con 0x7fb5740c9060 2023-10-16T14:12:21.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.016+0000 7fb56a7fc640 1 -- 172.21.15.60:0/11101508 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+17921 (secure 0 0 0) 0x7fb5640573b0 con 0x7fb5740c9060 2023-10-16T14:12:21.021 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:21.021 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":261,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:20.538615+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:6835/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:21.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb55806b020 msgr2=0x7fb55806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb55806b020 0x7fb55806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb5741f9a20 tx=0x7fb57000b040 comp rx=0 tx=0).stop 2023-10-16T14:12:21.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 msgr2=0x7fb574152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 0x7fb574152af0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fb56400c400 tx=0x7fb56400c8c0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 shutdown_connections 2023-10-16T14:12:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fb55806b020 0x7fb55806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5740c9440 0x7fb5741f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5740c9060 0x7fb574152af0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 --2- 172.21.15.60:0/11101508 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb57406c9e0 0x7fb5740bd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 >> 172.21.15.60:0/11101508 conn(0x7fb5740ce6a0 msgr2=0x7fb574160bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 shutdown_connections 2023-10-16T14:12:21.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.019+0000 7fb57a9f4640 1 -- 172.21.15.60:0/11101508 wait complete. 2023-10-16T14:12:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T14:11:50.763838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '255', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T14:11:51.462135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:12:21.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:12:21.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.252+0000 7f4430341640 1 Processor -- start 2023-10-16T14:12:21.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.253+0000 7f4430341640 1 -- start start 2023-10-16T14:12:21.258 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f4430341640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44280addb0 0x7f44280caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f4430341640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44280c9000 0x7f4428069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f4430341640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442806a6f0 con 0x7f442806a2e0 2023-10-16T14:12:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f4430341640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4428161a50 con 0x7f44280addb0 2023-10-16T14:12:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f4430341640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44280cb140 con 0x7f44280c9000 2023-10-16T14:12:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f442e0dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44280addb0 0x7f44280caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f442e0dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44280addb0 0x7f44280caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38026/0 (socket says 172.21.15.60:38026) 2023-10-16T14:12:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.254+0000 7f442e0dd640 1 -- 172.21.15.60:0/1732419891 learned_addr learned my addr 172.21.15.60:0/1732419891 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.255+0000 7f442e0dd640 1 -- 172.21.15.60:0/1732419891 >> v2:172.21.15.144:3300/0 conn(0x7f44280c9000 msgr2=0x7f4428069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.255+0000 7f442e0dd640 1 --2- 172.21.15.60:0/1732419891 >> v2:172.21.15.144:3300/0 conn(0x7f44280c9000 0x7f4428069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.255+0000 7f442e0dd640 1 -- 172.21.15.60:0/1732419891 >> v1:172.21.15.144:6789/0 conn(0x7f442806a2e0 legacy=0x7f44280cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.255+0000 7f442e0dd640 1 -- 172.21.15.60:0/1732419891 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442806b030 con 0x7f44280addb0 2023-10-16T14:12:21.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.256+0000 7f442e0dd640 1 --2- 172.21.15.60:0/1732419891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44280addb0 0x7f44280caa80 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f44280cdbe0 tx=0x7f441802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c3922d8857df6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:21.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.256+0000 7f442d0db640 1 -- 172.21.15.60:0/1732419891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441802ca80 con 0x7f44280addb0 2023-10-16T14:12:21.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.256+0000 7f442d0db640 1 -- 172.21.15.60:0/1732419891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441802cbe0 con 0x7f44280addb0 2023-10-16T14:12:21.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f442d0db640 1 -- 172.21.15.60:0/1732419891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180343f0 con 0x7f44280addb0 2023-10-16T14:12:21.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 -- 172.21.15.60:0/1732419891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44280addb0 msgr2=0x7f44280caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 --2- 172.21.15.60:0/1732419891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44280addb0 0x7f44280caa80 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f44280cdbe0 tx=0x7f441802bee0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 -- 172.21.15.60:0/1732419891 shutdown_connections 2023-10-16T14:12:21.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 --2- 172.21.15.60:0/1732419891 >> v2:172.21.15.144:3300/0 conn(0x7f44280c9000 0x7f4428069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 --2- 172.21.15.60:0/1732419891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44280addb0 0x7f44280caa80 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 -- 172.21.15.60:0/1732419891 >> 172.21.15.60:0/1732419891 conn(0x7f44280ce9b0 msgr2=0x7f44280cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:21.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 -- 172.21.15.60:0/1732419891 shutdown_connections 2023-10-16T14:12:21.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.257+0000 7f4430341640 1 -- 172.21.15.60:0/1732419891 wait complete. 2023-10-16T14:12:21.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.259+0000 7f4430341640 1 Processor -- start 2023-10-16T14:12:21.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.259+0000 7f4430341640 1 -- start start 2023-10-16T14:12:21.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f4430341640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 0x7f44280c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f4430341640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44280addb0 0x7f442815ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f4430341640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44280c9000 0x7f44280b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f4430341640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c75a0 con 0x7f442806a2e0 2023-10-16T14:12:21.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f4430341640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c6920 con 0x7f44280c9000 2023-10-16T14:12:21.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f4430341640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c6c00 con 0x7f44280addb0 2023-10-16T14:12:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.260+0000 7f442e0dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 0x7f44280c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 0x7f44280c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38032/0 (socket says 172.21.15.60:38032) 2023-10-16T14:12:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 -- 172.21.15.60:0/1394411337 learned_addr learned my addr 172.21.15.60:0/1394411337 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 -- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44280addb0 msgr2=0x7f442815ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44280addb0 0x7f442815ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 -- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44280c9000 msgr2=0x7f44280b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44280c9000 0x7f44280b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 -- 172.21.15.60:0/1394411337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009b30 con 0x7f442806a2e0 2023-10-16T14:12:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.261+0000 7f442e0dd640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 0x7f44280c79c0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f4418000f80 tx=0x7f4418004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.262+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441803e450 con 0x7f442806a2e0 2023-10-16T14:12:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.262+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441803e5f0 con 0x7f442806a2e0 2023-10-16T14:12:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.262+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441803e9a0 con 0x7f442806a2e0 2023-10-16T14:12:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.262+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44280c6ee0 con 0x7f442806a2e0 2023-10-16T14:12:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.262+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442815f260 con 0x7f442806a2e0 2023-10-16T14:12:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.263+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442804e160 con 0x7f442806a2e0 2023-10-16T14:12:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.264+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f441802cd50 con 0x7f442806a2e0 2023-10-16T14:12:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.265+0000 7f4426ffd640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f441406b020 0x7f441406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.265+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7f44180ac9a0 con 0x7f442806a2e0 2023-10-16T14:12:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.265+0000 7f442d8dc640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f441406b020 0x7f441406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.267+0000 7f442d8dc640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f441406b020 0x7f441406d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4428165c10 tx=0x7f441c006bf0 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:21.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.269+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4418077520 con 0x7f442806a2e0 2023-10-16T14:12:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.473+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44280c7810 con 0x7f442806a2e0 2023-10-16T14:12:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.474+0000 7f4426ffd640 1 -- 172.21.15.60:0/1394411337 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v261) v1 ==== 93+0+10 (secure 0 0 0) 0x7f441807ab40 con 0x7f442806a2e0 2023-10-16T14:12:21.478 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:12:21.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f441406b020 msgr2=0x7f441406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f441406b020 0x7f441406d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4428165c10 tx=0x7f441c006bf0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 msgr2=0x7f44280c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 0x7f44280c79c0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f4418000f80 tx=0x7f4418004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:21.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 shutdown_connections 2023-10-16T14:12:21.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f441406b020 0x7f441406d4e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f44280addb0 0x7f442815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44280c9000 0x7f44280b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 --2- 172.21.15.60:0/1394411337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f442806a2e0 0x7f44280c79c0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 >> 172.21.15.60:0/1394411337 conn(0x7f44280ce9b0 msgr2=0x7f44280aeff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:21.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 shutdown_connections 2023-10-16T14:12:21.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.477+0000 7f4430341640 1 -- 172.21.15.60:0/1394411337 wait complete. 2023-10-16T14:12:21.499 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T14:12:21.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.720+0000 7f832e5cc640 1 Processor -- start 2023-10-16T14:12:21.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.720+0000 7f832e5cc640 1 -- start start 2023-10-16T14:12:21.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f832e5cc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8328065ab0 0x7f8328161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f832e5cc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83280ac890 con 0x7f8328069fa0 2023-10-16T14:12:21.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f832e5cc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83280ae800 con 0x7f8328160530 2023-10-16T14:12:21.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f832e5cc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83280ae1f0 con 0x7f8328065ab0 2023-10-16T14:12:21.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f831ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8328065ab0 0x7f8328161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:21.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f831ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8328065ab0 0x7f8328161100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38040/0 (socket says 172.21.15.60:38040) 2023-10-16T14:12:21.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f831ffff640 1 -- 172.21.15.60:0/3316382868 learned_addr learned my addr 172.21.15.60:0/3316382868 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:21.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.722+0000 7f832cb69640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8328069fa0 0x7f832806a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47532/0 (socket says 172.21.15.60:47532) 2023-10-16T14:12:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.723+0000 7f831ffff640 1 -- 172.21.15.60:0/3316382868 >> v1:172.21.15.144:6789/0 conn(0x7f8328160530 legacy=0x7f83281609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.723+0000 7f831ffff640 1 -- 172.21.15.60:0/3316382868 >> v1:172.21.15.60:6789/0 conn(0x7f8328069fa0 legacy=0x7f832806a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.723+0000 7f831ffff640 1 -- 172.21.15.60:0/3316382868 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83280acae0 con 0x7f8328065ab0 2023-10-16T14:12:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.723+0000 7f83277fe640 1 -- 172.21.15.60:0/3316382868 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681671416 0 0) 0x7f83280ac890 con 0x7f8328069fa0 2023-10-16T14:12:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.724+0000 7f831ffff640 1 --2- 172.21.15.60:0/3316382868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f8328161100 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f8310009870 tx=0x7f831002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5f9849d49147b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.724+0000 7f83277fe640 1 -- 172.21.15.60:0/3316382868 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8310030860 con 0x7f8328065ab0 2023-10-16T14:12:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f83277fe640 1 -- 172.21.15.60:0/3316382868 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83100309c0 con 0x7f8328065ab0 2023-10-16T14:12:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f83277fe640 1 -- 172.21.15.60:0/3316382868 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8310030d50 con 0x7f8328065ab0 2023-10-16T14:12:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 -- 172.21.15.60:0/3316382868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 msgr2=0x7f8328161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 --2- 172.21.15.60:0/3316382868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f8328161100 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f8310009870 tx=0x7f831002b9a0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 -- 172.21.15.60:0/3316382868 shutdown_connections 2023-10-16T14:12:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 --2- 172.21.15.60:0/3316382868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f8328161100 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 -- 172.21.15.60:0/3316382868 >> 172.21.15.60:0/3316382868 conn(0x7f832806bc10 msgr2=0x7f832806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 -- 172.21.15.60:0/3316382868 shutdown_connections 2023-10-16T14:12:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.725+0000 7f832e5cc640 1 -- 172.21.15.60:0/3316382868 wait complete. 2023-10-16T14:12:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.727+0000 7f832e5cc640 1 Processor -- start 2023-10-16T14:12:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.727+0000 7f832e5cc640 1 -- start start 2023-10-16T14:12:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f832e5cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f83280b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f832e5cc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8328069fa0 0x7f832814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f832e5cc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8328160530 0x7f83280afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f832e5cc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83280c3250 con 0x7f8328065ab0 2023-10-16T14:12:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f832e5cc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83280c3530 con 0x7f8328160530 2023-10-16T14:12:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f832e5cc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83280c3810 con 0x7f8328069fa0 2023-10-16T14:12:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.728+0000 7f8327fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f83280b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f83280b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38046/0 (socket says 172.21.15.60:38046) 2023-10-16T14:12:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 -- 172.21.15.60:0/2117456911 learned_addr learned my addr 172.21.15.60:0/2117456911 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 -- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8328069fa0 msgr2=0x7f832814e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8328069fa0 0x7f832814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 -- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8328160530 msgr2=0x7f83280afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8328160530 0x7f83280afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 -- 172.21.15.60:0/2117456911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83100094d0 con 0x7f8328065ab0 2023-10-16T14:12:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f8327fff640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f83280b4840 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f831800c400 tx=0x7f831800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.729+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83180052d0 con 0x7f8328065ab0 2023-10-16T14:12:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.730+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8318005430 con 0x7f8328065ab0 2023-10-16T14:12:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.730+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318003830 con 0x7f8328065ab0 2023-10-16T14:12:21.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.730+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83280c3af0 con 0x7f8328065ab0 2023-10-16T14:12:21.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.730+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83280b4d80 con 0x7f8328065ab0 2023-10-16T14:12:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.731+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83280c4010 con 0x7f8328065ab0 2023-10-16T14:12:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.732+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f8318003990 con 0x7f8328065ab0 2023-10-16T14:12:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.733+0000 7f83257fa640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f830806b020 0x7f830806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.733+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7f8318089310 con 0x7f8328065ab0 2023-10-16T14:12:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.733+0000 7f831ffff640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f830806b020 0x7f830806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.735+0000 7f831ffff640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f830806b020 0x7f830806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8310036000 tx=0x7f831003b040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.737+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83280c4010 con 0x7f8328065ab0 2023-10-16T14:12:21.879 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:21.874+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:21.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.941+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83280c4010 con 0x7f8328065ab0 2023-10-16T14:12:21.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.942+0000 7f83257fa640 1 -- 172.21.15.60:0/2117456911 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v261) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8318053ce0 con 0x7f8328065ab0 2023-10-16T14:12:21.946 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:12:21.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f830806b020 msgr2=0x7f830806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f830806b020 0x7f830806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8310036000 tx=0x7f831003b040 comp rx=0 tx=0).stop 2023-10-16T14:12:21.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 msgr2=0x7f83280b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:21.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f83280b4840 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f831800c400 tx=0x7f831800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 shutdown_connections 2023-10-16T14:12:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f830806b020 0x7f830806d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8328069fa0 0x7f832814e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8328160530 0x7f83280afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 --2- 172.21.15.60:0/2117456911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8328065ab0 0x7f83280b4840 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:21.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 >> 172.21.15.60:0/2117456911 conn(0x7f832806bc10 msgr2=0x7f83280aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:21.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 shutdown_connections 2023-10-16T14:12:21.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:21.945+0000 7f832e5cc640 1 -- 172.21.15.60:0/2117456911 wait complete. 2023-10-16T14:12:21.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T14:12:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.186+0000 7ffa1692c640 1 Processor -- start 2023-10-16T14:12:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.187+0000 7ffa1692c640 1 -- start start 2023-10-16T14:12:22.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa1692c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa100ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa1692c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa1006c140 0x7ffa1006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa1692c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa10070360 con 0x7ffa10159ab0 2023-10-16T14:12:22.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa1692c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1006cb10 con 0x7ffa1006c140 2023-10-16T14:12:22.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa1692c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1015d890 con 0x7ffa10159e90 2023-10-16T14:12:22.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa14ec9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa1006c140 0x7ffa1006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa14ec9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa1006c140 0x7ffa1006c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38058/0 (socket says 172.21.15.60:38058) 2023-10-16T14:12:22.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa14ec9640 1 -- 172.21.15.60:0/2376890885 learned_addr learned my addr 172.21.15.60:0/2376890885 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.189+0000 7ffa07fff640 1 --2- 172.21.15.60:0/2376890885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa100ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.190+0000 7ffa14ec9640 1 -- 172.21.15.60:0/2376890885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 msgr2=0x7ffa100ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.190+0000 7ffa14ec9640 1 --2- 172.21.15.60:0/2376890885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa100ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.190+0000 7ffa14ec9640 1 -- 172.21.15.60:0/2376890885 >> v1:172.21.15.144:6789/0 conn(0x7ffa10159ab0 legacy=0x7ffa1006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.190+0000 7ffa14ec9640 1 -- 172.21.15.60:0/2376890885 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa100cd1f0 con 0x7ffa1006c140 2023-10-16T14:12:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.191+0000 7ffa14ec9640 1 --2- 172.21.15.60:0/2376890885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa1006c140 0x7ffa1006c5d0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009e80 tx=0x7ffa0802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad6a3d397e3001 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.191+0000 7ffa0f7fe640 1 -- 172.21.15.60:0/2376890885 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0802c9c0 con 0x7ffa1006c140 2023-10-16T14:12:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.191+0000 7ffa0f7fe640 1 -- 172.21.15.60:0/2376890885 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0802cb20 con 0x7ffa1006c140 2023-10-16T14:12:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.191+0000 7ffa0f7fe640 1 -- 172.21.15.60:0/2376890885 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa080343f0 con 0x7ffa1006c140 2023-10-16T14:12:22.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 -- 172.21.15.60:0/2376890885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa1006c140 msgr2=0x7ffa1006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 --2- 172.21.15.60:0/2376890885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa1006c140 0x7ffa1006c5d0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009e80 tx=0x7ffa0802be20 comp rx=0 tx=0).stop 2023-10-16T14:12:22.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 -- 172.21.15.60:0/2376890885 shutdown_connections 2023-10-16T14:12:22.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 --2- 172.21.15.60:0/2376890885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa1006c140 0x7ffa1006c5d0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 --2- 172.21.15.60:0/2376890885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa100ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 -- 172.21.15.60:0/2376890885 >> 172.21.15.60:0/2376890885 conn(0x7ffa100ce9b0 msgr2=0x7ffa100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 -- 172.21.15.60:0/2376890885 shutdown_connections 2023-10-16T14:12:22.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.192+0000 7ffa1692c640 1 -- 172.21.15.60:0/2376890885 wait complete. 2023-10-16T14:12:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.194+0000 7ffa1692c640 1 Processor -- start 2023-10-16T14:12:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.194+0000 7ffa1692c640 1 -- start start 2023-10-16T14:12:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa1692c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1006c140 0x7ffa10070e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa1692c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 0x7ffa1014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa1692c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa1015c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa1692c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa100713b0 con 0x7ffa10159ab0 2023-10-16T14:12:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa1692c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1014f390 con 0x7ffa1006c140 2023-10-16T14:12:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa1692c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1015b8c0 con 0x7ffa10159e90 2023-10-16T14:12:22.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa07fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 0x7ffa1014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa07fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 0x7ffa1014ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38068/0 (socket says 172.21.15.60:38068) 2023-10-16T14:12:22.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.195+0000 7ffa07fff640 1 -- 172.21.15.60:0/3637032889 learned_addr learned my addr 172.21.15.60:0/3637032889 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:22.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa14ec9640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa1015c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa0ffff640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1006c140 0x7ffa10070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa07fff640 1 -- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 msgr2=0x7ffa1015c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa07fff640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa1015c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa07fff640 1 -- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1006c140 msgr2=0x7ffa10070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa07fff640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1006c140 0x7ffa10070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa07fff640 1 -- 172.21.15.60:0/3637032889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa08009b30 con 0x7ffa10159ab0 2023-10-16T14:12:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.196+0000 7ffa07fff640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 0x7ffa1014ee50 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800e9b0 tx=0x7ff9f800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.197+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8009670 con 0x7ffa10159ab0 2023-10-16T14:12:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.197+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f800c9a0 con 0x7ffa10159ab0 2023-10-16T14:12:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.197+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f800cd50 con 0x7ffa10159ab0 2023-10-16T14:12:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.197+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1015bba0 con 0x7ffa10159ab0 2023-10-16T14:12:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.197+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1015c160 con 0x7ffa10159ab0 2023-10-16T14:12:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.198+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1004e160 con 0x7ffa10159ab0 2023-10-16T14:12:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.199+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7ff9f8013470 con 0x7ffa10159ab0 2023-10-16T14:12:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.200+0000 7ffa0d7fa640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff9f006b020 0x7ff9f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.201+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7ff9f8089fe0 con 0x7ffa10159ab0 2023-10-16T14:12:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.201+0000 7ffa0ffff640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff9f006b020 0x7ff9f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.202+0000 7ffa0ffff640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff9f006b020 0x7ff9f006d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa00000f60 tx=0x7ffa00009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:22.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.206+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9f80549b0 con 0x7ffa10159ab0 2023-10-16T14:12:22.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.406+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7ffa100ccb60 con 0x7ffa10159ab0 2023-10-16T14:12:22.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.407+0000 7ffa0d7fa640 1 -- 172.21.15.60:0/3637032889 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v261) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff9f80170a0 con 0x7ffa10159ab0 2023-10-16T14:12:22.411 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:12:22.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff9f006b020 msgr2=0x7ff9f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff9f006b020 0x7ff9f006d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa00000f60 tx=0x7ffa00009040 comp rx=0 tx=0).stop 2023-10-16T14:12:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 msgr2=0x7ffa1014ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 0x7ffa1014ee50 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800e9b0 tx=0x7ff9f800ee70 comp rx=0 tx=0).stop 2023-10-16T14:12:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 shutdown_connections 2023-10-16T14:12:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff9f006b020 0x7ff9f006d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffa10159e90 0x7ffa1015c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa10159ab0 0x7ffa1014ee50 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 --2- 172.21.15.60:0/3637032889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1006c140 0x7ffa10070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 >> 172.21.15.60:0/3637032889 conn(0x7ffa100ce9b0 msgr2=0x7ffa1006b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.410+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 shutdown_connections 2023-10-16T14:12:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.411+0000 7ffa1692c640 1 -- 172.21.15.60:0/3637032889 wait complete. 2023-10-16T14:12:22.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.650+0000 7f29ab59e640 1 Processor -- start 2023-10-16T14:12:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.650+0000 7f29ab59e640 1 -- start start 2023-10-16T14:12:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29ab59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f29ac15dde0 0x7f29ac0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29ab59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29ac070f20 0x7f29ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29ab59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac160a90 con 0x7f29ac15c9f0 2023-10-16T14:12:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29ab59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac071350 con 0x7f29ac070f20 2023-10-16T14:12:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29ab59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0cdec0 con 0x7f29ac15dde0 2023-10-16T14:12:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29aad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29ac070f20 0x7f29ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29aad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29ac070f20 0x7f29ac0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38070/0 (socket says 172.21.15.60:38070) 2023-10-16T14:12:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29aad9d640 1 -- 172.21.15.60:0/2951942358 learned_addr learned my addr 172.21.15.60:0/2951942358 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:22.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29a9d9b640 1 --2- 172.21.15.60:0/2951942358 >> v2:172.21.15.144:3300/0 conn(0x7f29ac15dde0 0x7f29ac0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.652+0000 7f29aad9d640 1 -- 172.21.15.60:0/2951942358 >> v2:172.21.15.144:3300/0 conn(0x7f29ac15dde0 msgr2=0x7f29ac0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:22.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.653+0000 7f29aad9d640 1 --2- 172.21.15.60:0/2951942358 >> v2:172.21.15.144:3300/0 conn(0x7f29ac15dde0 0x7f29ac0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.653+0000 7f29aad9d640 1 -- 172.21.15.60:0/2951942358 >> v1:172.21.15.144:6789/0 conn(0x7f29ac15c9f0 legacy=0x7f29ac0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:22.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.653+0000 7f29aad9d640 1 -- 172.21.15.60:0/2951942358 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac0cd900 con 0x7f29ac070f20 2023-10-16T14:12:22.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.654+0000 7f29aad9d640 1 --2- 172.21.15.60:0/2951942358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac070f20 0x7f29ac0ac890 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0b1260 tx=0x7f29a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c192960ff764d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:22.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.654+0000 7f29a959a640 1 -- 172.21.15.60:0/2951942358 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4031870 con 0x7f29ac070f20 2023-10-16T14:12:22.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.654+0000 7f29a959a640 1 -- 172.21.15.60:0/2951942358 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a40319d0 con 0x7f29ac070f20 2023-10-16T14:12:22.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.654+0000 7f29a959a640 1 -- 172.21.15.60:0/2951942358 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4031d60 con 0x7f29ac070f20 2023-10-16T14:12:22.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 -- 172.21.15.60:0/2951942358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac070f20 msgr2=0x7f29ac0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 --2- 172.21.15.60:0/2951942358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac070f20 0x7f29ac0ac890 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0b1260 tx=0x7f29a402b910 comp rx=0 tx=0).stop 2023-10-16T14:12:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 -- 172.21.15.60:0/2951942358 shutdown_connections 2023-10-16T14:12:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 --2- 172.21.15.60:0/2951942358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac070f20 0x7f29ac0ac890 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 --2- 172.21.15.60:0/2951942358 >> v2:172.21.15.144:3300/0 conn(0x7f29ac15dde0 0x7f29ac0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 -- 172.21.15.60:0/2951942358 >> 172.21.15.60:0/2951942358 conn(0x7f29ac0ce6a0 msgr2=0x7f29ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 -- 172.21.15.60:0/2951942358 shutdown_connections 2023-10-16T14:12:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.655+0000 7f29ab59e640 1 -- 172.21.15.60:0/2951942358 wait complete. 2023-10-16T14:12:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.657+0000 7f29ab59e640 1 Processor -- start 2023-10-16T14:12:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.658+0000 7f29ab59e640 1 -- start start 2023-10-16T14:12:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29ab59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ac070f20 0x7f29ac159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29ab59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29ac15c9f0 0x7f29ac159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29ab59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 0x7f29ac06a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29ab59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0c5aa0 con 0x7f29ac15dde0 2023-10-16T14:12:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29ab59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac06b030 con 0x7f29ac15c9f0 2023-10-16T14:12:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29ab59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0c69b0 con 0x7f29ac070f20 2023-10-16T14:12:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.659+0000 7f29aad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 0x7f29ac06a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 0x7f29ac06a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38080/0 (socket says 172.21.15.60:38080) 2023-10-16T14:12:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 -- 172.21.15.60:0/3163057753 learned_addr learned my addr 172.21.15.60:0/3163057753 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aa59c640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ac070f20 0x7f29ac159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29a9d9b640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29ac15c9f0 0x7f29ac159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 -- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ac070f20 msgr2=0x7f29ac159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ac070f20 0x7f29ac159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 -- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29ac15c9f0 msgr2=0x7f29ac159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29ac15c9f0 0x7f29ac159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 -- 172.21.15.60:0/3163057753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a40094d0 con 0x7f29ac15dde0 2023-10-16T14:12:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.660+0000 7f29aad9d640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 0x7f29ac06a3e0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f29a40097c0 tx=0x7f29a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.661+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a403add0 con 0x7f29ac15dde0 2023-10-16T14:12:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.661+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a40443f0 con 0x7f29ac15dde0 2023-10-16T14:12:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.661+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a40447a0 con 0x7f29ac15dde0 2023-10-16T14:12:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.661+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac0c6c90 con 0x7f29ac15dde0 2023-10-16T14:12:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.661+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac06b310 con 0x7f29ac15dde0 2023-10-16T14:12:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.662+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac04e9b0 con 0x7f29ac15dde0 2023-10-16T14:12:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.663+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f29a4044900 con 0x7f29ac15dde0 2023-10-16T14:12:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.664+0000 7f29a2ffd640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f298c06b020 0x7f298c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.664+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7f29a402d040 con 0x7f29ac15dde0 2023-10-16T14:12:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.665+0000 7f29aa59c640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f298c06b020 0x7f298c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.666+0000 7f29aa59c640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f298c06b020 0x7f298c06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f299c007fd0 tx=0x7f299c009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:22.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.669+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29a4077010 con 0x7f29ac15dde0 2023-10-16T14:12:22.833 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:22.828+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:22.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.912+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29ac0adac0 con 0x7f29ac15dde0 2023-10-16T14:12:22.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.913+0000 7f29a2ffd640 1 -- 172.21.15.60:0/3163057753 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4923 (secure 0 0 0) 0x7f29a407a630 con 0x7f29ac15dde0 2023-10-16T14:12:22.917 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:22.917 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":9769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/40626494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":40626494},{"type":"v1","addr":"172.21.15.144:6837","nonce":40626494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:22.918 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 136 2023-10-16T14:12:22.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f298c06b020 msgr2=0x7f298c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f298c06b020 0x7f298c06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f299c007fd0 tx=0x7f299c009040 comp rx=0 tx=0).stop 2023-10-16T14:12:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 msgr2=0x7f29ac06a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 0x7f29ac06a3e0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f29a40097c0 tx=0x7f29a40046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 shutdown_connections 2023-10-16T14:12:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f298c06b020 0x7f298c06d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29ac15dde0 0x7f29ac06a3e0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29ac15c9f0 0x7f29ac159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 --2- 172.21.15.60:0/3163057753 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f29ac070f20 0x7f29ac159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 >> 172.21.15.60:0/3163057753 conn(0x7f29ac0ce6a0 msgr2=0x7f29ac15ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:22.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 shutdown_connections 2023-10-16T14:12:22.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:22.916+0000 7f29ab59e640 1 -- 172.21.15.60:0/3163057753 wait complete. 2023-10-16T14:12:22.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:23.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.144+0000 7fcbdd4bb640 1 Processor -- start 2023-10-16T14:12:23.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.144+0000 7fcbdd4bb640 1 -- start start 2023-10-16T14:12:23.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbdd4bb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbd806c040 0x7fcbd806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbdd4bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd80ad970 0x7fcbd815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbdd4bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcbd815ebc0 0x7fcbd815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbdd4bb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd806c960 con 0x7fcbd806c040 2023-10-16T14:12:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbdd4bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd815b9d0 con 0x7fcbd815ebc0 2023-10-16T14:12:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbdd4bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd815bcb0 con 0x7fcbd80ad970 2023-10-16T14:12:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbd7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbd806c040 0x7fcbd806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbd7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbd806c040 0x7fcbd806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38090/0 (socket says 172.21.15.60:38090) 2023-10-16T14:12:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.146+0000 7fcbd7fff640 1 -- 172.21.15.60:0/3794865164 learned_addr learned my addr 172.21.15.60:0/3794865164 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbcf7fe640 1 --2- 172.21.15.60:0/3794865164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd80ad970 0x7fcbd815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbd7fff640 1 -- 172.21.15.60:0/3794865164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd80ad970 msgr2=0x7fcbd815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3794865164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd80ad970 0x7fcbd815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbdccba640 1 --2- 172.21.15.60:0/3794865164 >> v2:172.21.15.144:3300/0 conn(0x7fcbd815ebc0 0x7fcbd815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbd7fff640 1 -- 172.21.15.60:0/3794865164 >> v2:172.21.15.144:3300/0 conn(0x7fcbd815ebc0 msgr2=0x7fcbd815f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3794865164 >> v2:172.21.15.144:3300/0 conn(0x7fcbd815ebc0 0x7fcbd815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.147+0000 7fcbd7fff640 1 -- 172.21.15.60:0/3794865164 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd815bf90 con 0x7fcbd806c040 2023-10-16T14:12:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.148+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3794865164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd806c040 0x7fcbd806c420 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009e50 tx=0x7fcbc802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a80f6cdc7943564 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.148+0000 7fcbd77fe640 1 -- 172.21.15.60:0/3794865164 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc802ca80 con 0x7fcbd806c040 2023-10-16T14:12:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.148+0000 7fcbd77fe640 1 -- 172.21.15.60:0/3794865164 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc802cbe0 con 0x7fcbd806c040 2023-10-16T14:12:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.148+0000 7fcbd77fe640 1 -- 172.21.15.60:0/3794865164 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc80343f0 con 0x7fcbd806c040 2023-10-16T14:12:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3794865164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd806c040 msgr2=0x7fcbd806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3794865164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd806c040 0x7fcbd806c420 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009e50 tx=0x7fcbc802bee0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3794865164 shutdown_connections 2023-10-16T14:12:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3794865164 >> v2:172.21.15.144:3300/0 conn(0x7fcbd815ebc0 0x7fcbd815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3794865164 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd80ad970 0x7fcbd815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3794865164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd806c040 0x7fcbd806c420 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3794865164 >> 172.21.15.60:0/3794865164 conn(0x7fcbd80ce6a0 msgr2=0x7fcbd80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3794865164 shutdown_connections 2023-10-16T14:12:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.149+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3794865164 wait complete. 2023-10-16T14:12:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.151+0000 7fcbdd4bb640 1 Processor -- start 2023-10-16T14:12:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.151+0000 7fcbdd4bb640 1 -- start start 2023-10-16T14:12:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbdd4bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd806c040 0x7fcbd80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbdd4bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 0x7fcbd80c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbdd4bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbd815ebc0 0x7fcbd81486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbdd4bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8147640 con 0x7fcbd80ad970 2023-10-16T14:12:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbdd4bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8147920 con 0x7fcbd815ebc0 2023-10-16T14:12:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbdd4bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8147c00 con 0x7fcbd806c040 2023-10-16T14:12:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbcf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 0x7fcbd80c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbcf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 0x7fcbd80c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38100/0 (socket says 172.21.15.60:38100) 2023-10-16T14:12:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.152+0000 7fcbcf7fe640 1 -- 172.21.15.60:0/3250904963 learned_addr learned my addr 172.21.15.60:0/3250904963 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbdccba640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbd815ebc0 0x7fcbd81486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd806c040 0x7fcbd80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbcf7fe640 1 -- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd806c040 msgr2=0x7fcbd80ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbcf7fe640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd806c040 0x7fcbd80ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbcf7fe640 1 -- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbd815ebc0 msgr2=0x7fcbd81486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbcf7fe640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbd815ebc0 0x7fcbd81486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbcf7fe640 1 -- 172.21.15.60:0/3250904963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8009b30 con 0x7fcbd80ad970 2023-10-16T14:12:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbdccba640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbd815ebc0 0x7fcbd81486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd806c040 0x7fcbd80ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbcf7fe640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 0x7fcbd80c7c70 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000c570 tx=0x7fcbc000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.153+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0011800 con 0x7fcbd80ad970 2023-10-16T14:12:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.154+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc0011960 con 0x7fcbd80ad970 2023-10-16T14:12:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.154+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0011d10 con 0x7fcbd80ad970 2023-10-16T14:12:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.154+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd8147ee0 con 0x7fcbd80ad970 2023-10-16T14:12:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.154+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd80c81b0 con 0x7fcbd80ad970 2023-10-16T14:12:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.155+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd806b330 con 0x7fcbd80ad970 2023-10-16T14:12:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.156+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fcbc0018360 con 0x7fcbd80ad970 2023-10-16T14:12:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.157+0000 7fcbd57fa640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fcbb806b020 0x7fcbb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.157+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7fcbc0089e70 con 0x7fcbd80ad970 2023-10-16T14:12:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.158+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fcbb806b020 0x7fcbb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.159+0000 7fcbd7fff640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fcbb806b020 0x7fcbb806d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8035000 tx=0x7fcbc8030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.161+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbc0054730 con 0x7fcbd80ad970 2023-10-16T14:12:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.363+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbd804e9b0 con 0x7fcbd80ad970 2023-10-16T14:12:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.367+0000 7fcbd57fa640 1 -- 172.21.15.60:0/3250904963 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+17921 (secure 0 0 0) 0x7fcbc0057d50 con 0x7fcbd80ad970 2023-10-16T14:12:23.371 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:23.371 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":261,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:20.538615+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:6835/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:23.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.369+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fcbb806b020 msgr2=0x7fcbb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.369+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fcbb806b020 0x7fcbb806d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8035000 tx=0x7fcbc8030000 comp rx=0 tx=0).stop 2023-10-16T14:12:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.369+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 msgr2=0x7fcbd80c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.369+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 0x7fcbd80c7c70 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000c570 tx=0x7fcbc000ca30 comp rx=0 tx=0).stop 2023-10-16T14:12:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.369+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 shutdown_connections 2023-10-16T14:12:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.369+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fcbb806b020 0x7fcbb806d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.370+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcbd815ebc0 0x7fcbd81486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.370+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd80ad970 0x7fcbd80c7c70 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.370+0000 7fcbdd4bb640 1 --2- 172.21.15.60:0/3250904963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcbd806c040 0x7fcbd80ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.370+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 >> 172.21.15.60:0/3250904963 conn(0x7fcbd80ce6a0 msgr2=0x7fcbd8070590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:23.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.370+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 shutdown_connections 2023-10-16T14:12:23.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.370+0000 7fcbdd4bb640 1 -- 172.21.15.60:0/3250904963 wait complete. 2023-10-16T14:12:23.393 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T14:12:23.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.618+0000 7fdece50a640 1 Processor -- start 2023-10-16T14:12:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.618+0000 7fdece50a640 1 -- start start 2023-10-16T14:12:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.620+0000 7fdece50a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdec80ad970 0x7fdec80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.620+0000 7fdece50a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec815b9d0 con 0x7fdec80ccfb0 2023-10-16T14:12:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.620+0000 7fdece50a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec815bcb0 con 0x7fdec806c040 2023-10-16T14:12:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.620+0000 7fdece50a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec815bf90 con 0x7fdec80ad970 2023-10-16T14:12:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.620+0000 7fdecdd09640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdec80ccfb0 0x7fdec80cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47542/0 (socket says 172.21.15.60:47542) 2023-10-16T14:12:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdecdd09640 1 -- 172.21.15.60:0/721741382 learned_addr learned my addr 172.21.15.60:0/721741382 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdeccd07640 1 --2- 172.21.15.60:0/721741382 >> v2:172.21.15.144:3300/0 conn(0x7fdec80ad970 0x7fdec80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153796671 0 0) 0x7fdec815b9d0 con 0x7fdec80ccfb0 2023-10-16T14:12:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4003520 con 0x7fdec80ccfb0 2023-10-16T14:12:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271180034 0 0) 0x7fdeb4003520 con 0x7fdec80ccfb0 2023-10-16T14:12:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 >> v2:172.21.15.144:3300/0 conn(0x7fdec80ad970 msgr2=0x7fdec80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 --2- 172.21.15.60:0/721741382 >> v2:172.21.15.144:3300/0 conn(0x7fdec80ad970 0x7fdec80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 >> v1:172.21.15.144:6789/0 conn(0x7fdec806c040 legacy=0x7fdec806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.621+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80cc270 con 0x7fdec80ccfb0 2023-10-16T14:12:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fdec00019f0 con 0x7fdec80ccfb0 2023-10-16T14:12:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 >> v1:172.21.15.60:6789/0 conn(0x7fdec80ccfb0 legacy=0x7fdec80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeb40039c0 0x7fdeb4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdeb40047c0 0x7fdeb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb4005180 0x7fdeb400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec815bf90 con 0x7fdeb40039c0 2023-10-16T14:12:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb400c080 con 0x7fdeb40047c0 2023-10-16T14:12:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb400c240 con 0x7fdeb4005180 2023-10-16T14:12:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdecd508640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeb40039c0 0x7fdeb4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdeccd07640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdeb40047c0 0x7fdeb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdecd508640 1 -- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb4005180 msgr2=0x7fdeb400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdecd508640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb4005180 0x7fdeb400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdecd508640 1 -- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdeb40047c0 msgr2=0x7fdeb4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdecd508640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdeb40047c0 0x7fdeb4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdecd508640 1 -- 172.21.15.60:0/721741382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4003520 con 0x7fdeb40039c0 2023-10-16T14:12:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.622+0000 7fdeccd07640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdeb40047c0 0x7fdeb4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.623+0000 7fdecdd09640 1 -- 172.21.15.60:0/721741382 reap_dead start 2023-10-16T14:12:23.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.624+0000 7fdecd508640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeb40039c0 0x7fdeb4003de0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7fdec4001750 tx=0x7fdeb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2624f061edb04cc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.624+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40052d0 con 0x7fdeb40039c0 2023-10-16T14:12:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.624+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec402e050 con 0x7fdeb40039c0 2023-10-16T14:12:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.624+0000 7fdebffff640 1 -- 172.21.15.60:0/721741382 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4003830 con 0x7fdeb40039c0 2023-10-16T14:12:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.624+0000 7fdece50a640 1 -- 172.21.15.60:0/721741382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeb40039c0 msgr2=0x7fdeb4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.624+0000 7fdece50a640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeb40039c0 0x7fdeb4003de0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7fdec4001750 tx=0x7fdeb40057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.625+0000 7fdece50a640 1 -- 172.21.15.60:0/721741382 shutdown_connections 2023-10-16T14:12:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.625+0000 7fdece50a640 1 --2- 172.21.15.60:0/721741382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdeb40039c0 0x7fdeb4003de0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.625+0000 7fdece50a640 1 -- 172.21.15.60:0/721741382 >> 172.21.15.60:0/721741382 conn(0x7fdec80ce6a0 msgr2=0x7fdec80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.625+0000 7fdece50a640 1 -- 172.21.15.60:0/721741382 shutdown_connections 2023-10-16T14:12:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.625+0000 7fdece50a640 1 -- 172.21.15.60:0/721741382 wait complete. 2023-10-16T14:12:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.626+0000 7fdece50a640 1 Processor -- start 2023-10-16T14:12:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.627+0000 7fdece50a640 1 -- start start 2023-10-16T14:12:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdece50a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb40039c0 0x7fdec8153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdece50a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec80c8cf0 0x7fdec80ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdece50a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 0x7fdec80acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdece50a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80c9e50 con 0x7fdec80ac890 2023-10-16T14:12:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdece50a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80ad280 con 0x7fdec80c8cf0 2023-10-16T14:12:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdece50a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80c91d0 con 0x7fdeb40039c0 2023-10-16T14:12:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 0x7fdec80acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 0x7fdec80acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38122/0 (socket says 172.21.15.60:38122) 2023-10-16T14:12:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 -- 172.21.15.60:0/319216247 learned_addr learned my addr 172.21.15.60:0/319216247 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdeccd07640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec80c8cf0 0x7fdec80ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecd508640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb40039c0 0x7fdec8153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 -- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb40039c0 msgr2=0x7fdec8153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb40039c0 0x7fdec8153840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 -- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec80c8cf0 msgr2=0x7fdec80ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec80c8cf0 0x7fdec80ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.628+0000 7fdecdd09640 1 -- 172.21.15.60:0/319216247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80c94b0 con 0x7fdec80ac890 2023-10-16T14:12:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.629+0000 7fdecdd09640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 0x7fdec80acd40 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fdec0001b20 tx=0x7fdec000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.629+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0004d60 con 0x7fdec80ac890 2023-10-16T14:12:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.629+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec000a5b0 con 0x7fdec80ac890 2023-10-16T14:12:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.629+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec000a960 con 0x7fdec80ac890 2023-10-16T14:12:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.629+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec80c9790 con 0x7fdec80ac890 2023-10-16T14:12:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.629+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec80ca580 con 0x7fdec80ac890 2023-10-16T14:12:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.630+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec804e9b0 con 0x7fdec80ac890 2023-10-16T14:12:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.631+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fdec000aac0 con 0x7fdec80ac890 2023-10-16T14:12:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.632+0000 7fdebdffb640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdeac06b020 0x7fdeac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:23.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.632+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7fdec00893c0 con 0x7fdec80ac890 2023-10-16T14:12:23.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.633+0000 7fdecd508640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdeac06b020 0x7fdeac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:23.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.634+0000 7fdecd508640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdeac06b020 0x7fdeac06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdec4033000 tx=0x7fdec4009030 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:23.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.637+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdec0053e20 con 0x7fdec80ac890 2023-10-16T14:12:23.796 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:23.791+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:23.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.885+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec806b2b0 con 0x7fdec80ac890 2023-10-16T14:12:23.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.885+0000 7fdebdffb640 1 -- 172.21.15.60:0/319216247 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4923 (secure 0 0 0) 0x7fdec0057440 con 0x7fdec80ac890 2023-10-16T14:12:23.889 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:23.890 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":9769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/40626494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":40626494},{"type":"v1","addr":"172.21.15.144:6837","nonce":40626494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 136 2023-10-16T14:12:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdeac06b020 msgr2=0x7fdeac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdeac06b020 0x7fdeac06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdec4033000 tx=0x7fdec4009030 comp rx=0 tx=0).stop 2023-10-16T14:12:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 msgr2=0x7fdec80acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 0x7fdec80acd40 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fdec0001b20 tx=0x7fdec000cba0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 shutdown_connections 2023-10-16T14:12:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdeac06b020 0x7fdeac06d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec80ac890 0x7fdec80acd40 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec80c8cf0 0x7fdec80ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.888+0000 7fdece50a640 1 --2- 172.21.15.60:0/319216247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdeb40039c0 0x7fdec8153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.889+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 >> 172.21.15.60:0/319216247 conn(0x7fdec80ce6a0 msgr2=0x7fdec80af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.889+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 shutdown_connections 2023-10-16T14:12:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:23.889+0000 7fdece50a640 1 -- 172.21.15.60:0/319216247 wait complete. 2023-10-16T14:12:23.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:24.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.137+0000 7faf6f02e640 1 Processor -- start 2023-10-16T14:12:24.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.137+0000 7faf6f02e640 1 -- start start 2023-10-16T14:12:24.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.139+0000 7faf6f02e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf68159950 0x7faf6815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.139+0000 7faf6f02e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf6806c810 0x7faf680708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.139+0000 7faf6f02e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6815aef0 con 0x7faf68070450 2023-10-16T14:12:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.139+0000 7faf6f02e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf68070e20 con 0x7faf68159950 2023-10-16T14:12:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.139+0000 7faf6f02e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6815e490 con 0x7faf6806c810 2023-10-16T14:12:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.139+0000 7faf6e02c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf68159950 0x7faf6815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.140+0000 7faf6e02c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf68159950 0x7faf6815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38128/0 (socket says 172.21.15.60:38128) 2023-10-16T14:12:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.140+0000 7faf6e02c640 1 -- 172.21.15.60:0/63656175 learned_addr learned my addr 172.21.15.60:0/63656175 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.140+0000 7faf6e02c640 1 -- 172.21.15.60:0/63656175 >> v2:172.21.15.144:3300/0 conn(0x7faf6806c810 msgr2=0x7faf680708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.140+0000 7faf6e02c640 1 --2- 172.21.15.60:0/63656175 >> v2:172.21.15.144:3300/0 conn(0x7faf6806c810 0x7faf680708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.140+0000 7faf6e02c640 1 -- 172.21.15.60:0/63656175 >> v1:172.21.15.144:6789/0 conn(0x7faf68070450 legacy=0x7faf6806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.140+0000 7faf6e02c640 1 -- 172.21.15.60:0/63656175 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6815d910 con 0x7faf68159950 2023-10-16T14:12:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.141+0000 7faf6e02c640 1 --2- 172.21.15.60:0/63656175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf6815a9b0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7faf680716a0 tx=0x7faf6402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b09b589a8bea6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.141+0000 7faf6d02a640 1 -- 172.21.15.60:0/63656175 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640317e0 con 0x7faf68159950 2023-10-16T14:12:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.141+0000 7faf6d02a640 1 -- 172.21.15.60:0/63656175 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf64031940 con 0x7faf68159950 2023-10-16T14:12:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.141+0000 7faf6d02a640 1 -- 172.21.15.60:0/63656175 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64031cd0 con 0x7faf68159950 2023-10-16T14:12:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.142+0000 7faf6f02e640 1 -- 172.21.15.60:0/63656175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 msgr2=0x7faf6815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.142+0000 7faf6f02e640 1 --2- 172.21.15.60:0/63656175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf6815a9b0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7faf680716a0 tx=0x7faf6402b880 comp rx=0 tx=0).stop 2023-10-16T14:12:24.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.142+0000 7faf6f02e640 1 -- 172.21.15.60:0/63656175 shutdown_connections 2023-10-16T14:12:24.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.142+0000 7faf6f02e640 1 --2- 172.21.15.60:0/63656175 >> v2:172.21.15.144:3300/0 conn(0x7faf6806c810 0x7faf680708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.142+0000 7faf6f02e640 1 --2- 172.21.15.60:0/63656175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf6815a9b0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.142+0000 7faf6f02e640 1 -- 172.21.15.60:0/63656175 >> 172.21.15.60:0/63656175 conn(0x7faf680ce6a0 msgr2=0x7faf680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.143+0000 7faf6f02e640 1 -- 172.21.15.60:0/63656175 shutdown_connections 2023-10-16T14:12:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.143+0000 7faf6f02e640 1 -- 172.21.15.60:0/63656175 wait complete. 2023-10-16T14:12:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.145+0000 7faf6f02e640 1 Processor -- start 2023-10-16T14:12:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.145+0000 7faf6f02e640 1 -- start start 2023-10-16T14:12:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6f02e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf6806c810 0x7faf681f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6f02e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf68070450 0x7faf680c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6f02e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf68157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6f02e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681435e0 con 0x7faf68159950 2023-10-16T14:12:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6f02e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681f05f0 con 0x7faf6806c810 2023-10-16T14:12:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6f02e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf68142960 con 0x7faf68070450 2023-10-16T14:12:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6e82d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf68157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6e82d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf68157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38134/0 (socket says 172.21.15.60:38134) 2023-10-16T14:12:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6e82d640 1 -- 172.21.15.60:0/3659463441 learned_addr learned my addr 172.21.15.60:0/3659463441 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6e02c640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf6806c810 0x7faf681f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6d82b640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf68070450 0x7faf680c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.147+0000 7faf6e82d640 1 -- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf68070450 msgr2=0x7faf680c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6e82d640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf68070450 0x7faf680c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6e82d640 1 -- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf6806c810 msgr2=0x7faf681f00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6e82d640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf6806c810 0x7faf681f00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6e82d640 1 -- 172.21.15.60:0/3659463441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf640094d0 con 0x7faf68159950 2023-10-16T14:12:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6d82b640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf68070450 0x7faf680c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6e82d640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf68157f00 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7faf6000b970 tx=0x7faf6000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6000cb40 con 0x7faf68159950 2023-10-16T14:12:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6000ccc0 con 0x7faf68159950 2023-10-16T14:12:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60014680 con 0x7faf68159950 2023-10-16T14:12:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf68142c40 con 0x7faf68159950 2023-10-16T14:12:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.148+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf68143200 con 0x7faf68159950 2023-10-16T14:12:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.149+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf68143720 con 0x7faf68159950 2023-10-16T14:12:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.151+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7faf6001d3b0 con 0x7faf68159950 2023-10-16T14:12:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.152+0000 7faf5effd640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf4c06b020 0x7faf4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.152+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7faf60089c10 con 0x7faf68159950 2023-10-16T14:12:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.152+0000 7faf6e02c640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf4c06b020 0x7faf4c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.154+0000 7faf6e02c640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf4c06b020 0x7faf4c06d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf64036000 tx=0x7faf6403d040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.156+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf68143720 con 0x7faf68159950 2023-10-16T14:12:24.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.402+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf68143720 con 0x7faf68159950 2023-10-16T14:12:24.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.403+0000 7faf5effd640 1 -- 172.21.15.60:0/3659463441 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4923 (secure 0 0 0) 0x7faf68143720 con 0x7faf68159950 2023-10-16T14:12:24.407 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:24.407 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":9769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/40626494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":40626494},{"type":"v1","addr":"172.21.15.144:6837","nonce":40626494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:11:52.433484+0000","modified":"2023-10-16T14:11:55.513998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8186":{"gid":8186,"name":"d","rank":0,"incarnation":130,"state":"up:active","state_seq":7,"addr":"172.21.15.144:6836/521250241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":521250241},{"type":"v1","addr":"172.21.15.144:6836","nonce":521250241}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T14:12:24.408 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 136 2023-10-16T14:12:24.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf4c06b020 msgr2=0x7faf4c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf4c06b020 0x7faf4c06d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf64036000 tx=0x7faf6403d040 comp rx=0 tx=0).stop 2023-10-16T14:12:24.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 msgr2=0x7faf68157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf68157f00 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7faf6000b970 tx=0x7faf6000be30 comp rx=0 tx=0).stop 2023-10-16T14:12:24.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 shutdown_connections 2023-10-16T14:12:24.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf4c06b020 0x7faf4c06d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf68070450 0x7faf680c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf68159950 0x7faf68157f00 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 --2- 172.21.15.60:0/3659463441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf6806c810 0x7faf681f00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.406+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 >> 172.21.15.60:0/3659463441 conn(0x7faf680ce6a0 msgr2=0x7faf68159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:24.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.407+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 shutdown_connections 2023-10-16T14:12:24.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.407+0000 7faf6f02e640 1 -- 172.21.15.60:0/3659463441 wait complete. 2023-10-16T14:12:24.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:24.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.649+0000 7ff001f1d640 1 Processor -- start 2023-10-16T14:12:24.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.649+0000 7ff001f1d640 1 -- start start 2023-10-16T14:12:24.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.651+0000 7ff001f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feffc0b0b20 0x7feffc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.651+0000 7ff001f1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc15e3e0 con 0x7feffc15d3f0 2023-10-16T14:12:24.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.651+0000 7ff001f1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0cd4a0 con 0x7feffc0cd0c0 2023-10-16T14:12:24.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.651+0000 7ff001f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc15e100 con 0x7feffc0b0b20 2023-10-16T14:12:24.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.651+0000 7feffbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feffc15d3f0 0x7feffc15d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47554/0 (socket says 172.21.15.60:47554) 2023-10-16T14:12:24.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.651+0000 7feffbfff640 1 -- 172.21.15.60:0/1189943114 learned_addr learned my addr 172.21.15.60:0/1189943114 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:24.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feff2ffd640 1 --2- 172.21.15.60:0/1189943114 >> v2:172.21.15.144:3300/0 conn(0x7feffc0b0b20 0x7feffc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179636978 0 0) 0x7feffc15e3e0 con 0x7feffc15d3f0 2023-10-16T14:12:24.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe8003520 con 0x7feffc15d3f0 2023-10-16T14:12:24.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036803358 0 0) 0x7fefe8003520 con 0x7feffc15d3f0 2023-10-16T14:12:24.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 >> v2:172.21.15.144:3300/0 conn(0x7feffc0b0b20 msgr2=0x7feffc1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:24.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 --2- 172.21.15.60:0/1189943114 >> v2:172.21.15.144:3300/0 conn(0x7feffc0b0b20 0x7feffc1615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 >> v1:172.21.15.144:6789/0 conn(0x7feffc0cd0c0 legacy=0x7feffc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc160b10 con 0x7feffc15d3f0 2023-10-16T14:12:24.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.652+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7feff4001670 con 0x7feffc15d3f0 2023-10-16T14:12:24.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 >> v1:172.21.15.60:6789/0 conn(0x7feffc15d3f0 legacy=0x7feffc15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe80039c0 0x7fefe8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe80051d0 0x7fefe800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15e100 con 0x7fefe80047c0 2023-10-16T14:12:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe800c120 con 0x7fefe80039c0 2023-10-16T14:12:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe800c2e0 con 0x7fefe80051d0 2023-10-16T14:12:24.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feff2ffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feffb7fe640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe80039c0 0x7fefe8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feff2ffd640 1 -- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe80051d0 msgr2=0x7fefe800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:24.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feff2ffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fefe80051d0 0x7fefe800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feff2ffd640 1 -- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe80039c0 msgr2=0x7fefe8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.653+0000 7feff2ffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe80039c0 0x7fefe8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.654+0000 7feff2ffd640 1 -- 172.21.15.60:0/1189943114 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8003520 con 0x7fefe80047c0 2023-10-16T14:12:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.654+0000 7feffb7fe640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefe80039c0 0x7fefe8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.654+0000 7feffbfff640 1 -- 172.21.15.60:0/1189943114 reap_dead start 2023-10-16T14:12:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.655+0000 7feff2ffd640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009f90 tx=0x7fefe80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec9dd9c32c185f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:24.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.655+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe402cbd0 con 0x7fefe80047c0 2023-10-16T14:12:24.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.655+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe400f040 con 0x7fefe80047c0 2023-10-16T14:12:24.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.655+0000 7feffaffd640 1 -- 172.21.15.60:0/1189943114 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4034440 con 0x7fefe80047c0 2023-10-16T14:12:24.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.655+0000 7ff001f1d640 1 -- 172.21.15.60:0/1189943114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 msgr2=0x7fefe8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.655+0000 7ff001f1d640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009f90 tx=0x7fefe80057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.656+0000 7ff001f1d640 1 -- 172.21.15.60:0/1189943114 shutdown_connections 2023-10-16T14:12:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.656+0000 7ff001f1d640 1 --2- 172.21.15.60:0/1189943114 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7fefe8004c40 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.656+0000 7ff001f1d640 1 -- 172.21.15.60:0/1189943114 >> 172.21.15.60:0/1189943114 conn(0x7feffc0cb410 msgr2=0x7feffc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.656+0000 7ff001f1d640 1 -- 172.21.15.60:0/1189943114 shutdown_connections 2023-10-16T14:12:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.656+0000 7ff001f1d640 1 -- 172.21.15.60:0/1189943114 wait complete. 2023-10-16T14:12:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.658+0000 7ff001f1d640 1 Processor -- start 2023-10-16T14:12:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.658+0000 7ff001f1d640 1 -- start start 2023-10-16T14:12:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7ff001f1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7feffc0b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7ff001f1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feffc1517e0 0x7feffc0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7ff001f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feffc151c60 0x7feffc0c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7ff001f1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0af320 con 0x7fefe80047c0 2023-10-16T14:12:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7ff001f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc15b3d0 con 0x7feffc151c60 2023-10-16T14:12:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7ff001f1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0b4730 con 0x7feffc1517e0 2023-10-16T14:12:24.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7feffc0b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7feffc0b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38144/0 (socket says 172.21.15.60:38144) 2023-10-16T14:12:24.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.659+0000 7feffb7fe640 1 -- 172.21.15.60:0/627335448 learned_addr learned my addr 172.21.15.60:0/627335448 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:24.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feff2ffd640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feffc1517e0 0x7feffc0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffb7fe640 1 -- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feffc1517e0 msgr2=0x7feffc0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffb7fe640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feffc1517e0 0x7feffc0c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffbfff640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feffc151c60 0x7feffc0c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffb7fe640 1 -- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feffc151c60 msgr2=0x7feffc0c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffb7fe640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feffc151c60 0x7feffc0c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffb7fe640 1 -- 172.21.15.60:0/627335448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc0ae710 con 0x7fefe80047c0 2023-10-16T14:12:24.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffbfff640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feffc151c60 0x7feffc0c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feffb7fe640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7feffc0b1550 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fefec00d770 tx=0x7fefec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec005270 con 0x7fefe80047c0 2023-10-16T14:12:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec0053d0 con 0x7fefe80047c0 2023-10-16T14:12:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.660+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc0ae980 con 0x7fefe80047c0 2023-10-16T14:12:24.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.661+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec00b8f0 con 0x7fefe80047c0 2023-10-16T14:12:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.661+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc0aef40 con 0x7fefe80047c0 2023-10-16T14:12:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.662+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc04e160 con 0x7fefe80047c0 2023-10-16T14:12:24.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.663+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fefec00ba50 con 0x7fefe80047c0 2023-10-16T14:12:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.664+0000 7feff8ff9640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fefdc06b020 0x7fefdc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.664+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7fefec089d30 con 0x7fefe80047c0 2023-10-16T14:12:24.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.665+0000 7feff2ffd640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fefdc06b020 0x7fefdc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.666+0000 7feff2ffd640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fefdc06b020 0x7fefdc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fefe4036000 tx=0x7fefe402e000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:24.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.669+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefec054690 con 0x7fefe80047c0 2023-10-16T14:12:24.786 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:24.781+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:24.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.873+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feffc0b4910 con 0x7fefe80047c0 2023-10-16T14:12:24.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.876+0000 7feff8ff9640 1 -- 172.21.15.60:0/627335448 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+17921 (secure 0 0 0) 0x7fefec019310 con 0x7fefe80047c0 2023-10-16T14:12:24.881 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:24.881 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":261,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:20.538615+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T14:11:50.763838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"255","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-16T14:11:51.462135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6837/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:6835/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:24.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fefdc06b020 msgr2=0x7fefdc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fefdc06b020 0x7fefdc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fefe4036000 tx=0x7fefe402e000 comp rx=0 tx=0).stop 2023-10-16T14:12:24.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 msgr2=0x7feffc0b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7feffc0b1550 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fefec00d770 tx=0x7fefec00dc30 comp rx=0 tx=0).stop 2023-10-16T14:12:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 shutdown_connections 2023-10-16T14:12:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fefdc06b020 0x7fefdc06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7feffc1517e0 0x7feffc0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feffc151c60 0x7feffc0c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 --2- 172.21.15.60:0/627335448 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80047c0 0x7feffc0b1550 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:24.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 >> 172.21.15.60:0/627335448 conn(0x7feffc0cb410 msgr2=0x7feffc0ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:24.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.879+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 shutdown_connections 2023-10-16T14:12:24.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:24.880+0000 7ff001f1d640 1 -- 172.21.15.60:0/627335448 wait complete. 2023-10-16T14:12:24.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T14:12:25.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.131+0000 7fa5c6116640 1 Processor -- start 2023-10-16T14:12:25.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.132+0000 7fa5c6116640 1 -- start start 2023-10-16T14:12:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.133+0000 7fa5c6116640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c015d150 0x7fa5c00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.133+0000 7fa5c6116640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5c0160b90 0x7fa5c0160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.133+0000 7fa5c6116640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00ac890 con 0x7fa5c0069da0 2023-10-16T14:12:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.133+0000 7fa5c6116640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c015d530 con 0x7fa5c0160b90 2023-10-16T14:12:25.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.133+0000 7fa5c6116640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0161650 con 0x7fa5c015d150 2023-10-16T14:12:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.134+0000 7fa5bffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa5c0069da0 0x7fa5c006a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47566/0 (socket says 172.21.15.60:47566) 2023-10-16T14:12:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.134+0000 7fa5bffff640 1 -- 172.21.15.60:0/2774509479 learned_addr learned my addr 172.21.15.60:0/2774509479 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:25.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.134+0000 7fa5b6ffd640 1 --2- 172.21.15.60:0/2774509479 >> v2:172.21.15.60:3300/0 conn(0x7fa5c0160b90 0x7fa5c0160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.134+0000 7fa5bf7fe640 1 --2- 172.21.15.60:0/2774509479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c015d150 0x7fa5c00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.134+0000 7fa5beffd640 1 -- 172.21.15.60:0/2774509479 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173408422 0 0) 0x7fa5c00ac890 con 0x7fa5c0069da0 2023-10-16T14:12:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.135+0000 7fa5beffd640 1 -- 172.21.15.60:0/2774509479 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac003520 con 0x7fa5c0069da0 2023-10-16T14:12:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.135+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/2774509479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c015d150 msgr2=0x7fa5c00ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:25.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.135+0000 7fa5b6ffd640 1 --2- 172.21.15.60:0/2774509479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c015d150 0x7fa5c00ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.135+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/2774509479 >> v1:172.21.15.60:6789/0 conn(0x7fa5c0069da0 legacy=0x7fa5c006a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.135+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/2774509479 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0161930 con 0x7fa5c0160b90 2023-10-16T14:12:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.136+0000 7fa5b6ffd640 1 --2- 172.21.15.60:0/2774509479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c0160b90 0x7fa5c0160fb0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80097a0 tx=0x7fa5a802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bee1302668e31b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.136+0000 7fa5beffd640 1 -- 172.21.15.60:0/2774509479 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a802d040 con 0x7fa5c0160b90 2023-10-16T14:12:25.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.136+0000 7fa5beffd640 1 -- 172.21.15.60:0/2774509479 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8031810 con 0x7fa5c0160b90 2023-10-16T14:12:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.136+0000 7fa5beffd640 1 -- 172.21.15.60:0/2774509479 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8031b80 con 0x7fa5c0160b90 2023-10-16T14:12:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 -- 172.21.15.60:0/2774509479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c0160b90 msgr2=0x7fa5c0160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 --2- 172.21.15.60:0/2774509479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c0160b90 0x7fa5c0160fb0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80097a0 tx=0x7fa5a802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 -- 172.21.15.60:0/2774509479 shutdown_connections 2023-10-16T14:12:25.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 --2- 172.21.15.60:0/2774509479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c0160b90 0x7fa5c0160fb0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 --2- 172.21.15.60:0/2774509479 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c015d150 0x7fa5c00ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 -- 172.21.15.60:0/2774509479 >> 172.21.15.60:0/2774509479 conn(0x7fa5c00ce9b0 msgr2=0x7fa5c00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:25.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 -- 172.21.15.60:0/2774509479 shutdown_connections 2023-10-16T14:12:25.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.137+0000 7fa5c6116640 1 -- 172.21.15.60:0/2774509479 wait complete. 2023-10-16T14:12:25.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.140+0000 7fa5c6116640 1 Processor -- start 2023-10-16T14:12:25.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.140+0000 7fa5c6116640 1 -- start start 2023-10-16T14:12:25.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.141+0000 7fa5c6116640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c0069da0 0x7fa5c00c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.141+0000 7fa5c6116640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 0x7fa5c0160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.141+0000 7fa5c6116640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5c0160b90 0x7fa5c01f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.141+0000 7fa5c6116640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00c35e0 con 0x7fa5c015d150 2023-10-16T14:12:25.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.141+0000 7fa5c6116640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0160570 con 0x7fa5c0160b90 2023-10-16T14:12:25.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.141+0000 7fa5c6116640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b1630 con 0x7fa5c0069da0 2023-10-16T14:12:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 0x7fa5c0160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 0x7fa5c0160030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38170/0 (socket says 172.21.15.60:38170) 2023-10-16T14:12:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/1028406182 learned_addr learned my addr 172.21.15.60:0/1028406182 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5bf7fe640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c0069da0 0x7fa5c00c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5bffff640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5c0160b90 0x7fa5c01f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c0069da0 msgr2=0x7fa5c00c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c0069da0 0x7fa5c00c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5c0160b90 msgr2=0x7fa5c01f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5c0160b90 0x7fa5c01f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5b6ffd640 1 -- 172.21.15.60:0/1028406182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a8009400 con 0x7fa5c015d150 2023-10-16T14:12:25.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.142+0000 7fa5bf7fe640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c0069da0 0x7fa5c00c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:12:25.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.143+0000 7fa5b6ffd640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 0x7fa5c0160030 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000c00 tx=0x7fa5a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:25.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.143+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8006e10 con 0x7fa5c015d150 2023-10-16T14:12:25.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.143+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a803ad60 con 0x7fa5c015d150 2023-10-16T14:12:25.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.143+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8044620 con 0x7fa5c015d150 2023-10-16T14:12:25.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.143+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c01607a0 con 0x7fa5c015d150 2023-10-16T14:12:25.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.143+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c00c4420 con 0x7fa5c015d150 2023-10-16T14:12:25.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.145+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c004e160 con 0x7fa5c015d150 2023-10-16T14:12:25.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.146+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa5a804d440 con 0x7fa5c015d150 2023-10-16T14:12:25.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.147+0000 7fa5bcff9640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa5a006b020 0x7fa5a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.147+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6667+0+0 (secure 0 0 0) 0x7fa5a80ac0c0 con 0x7fa5c015d150 2023-10-16T14:12:25.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.148+0000 7fa5bf7fe640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa5a006b020 0x7fa5a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.149+0000 7fa5bf7fe640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa5a006b020 0x7fa5a006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0006fd0 tx=0x7fa5b0008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:25.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.151+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5a8076a10 con 0x7fa5c015d150 2023-10-16T14:12:25.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.399+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fa5c00ade10 con 0x7fa5c015d150 2023-10-16T14:12:25.403 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:12:25.400+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:12:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.403+0000 7fa5bcff9640 1 -- 172.21.15.60:0/1028406182 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v137) v1 ==== 167+0+0 (secure 0 0 0) 0x7fa5a807a030 con 0x7fa5c015d150 2023-10-16T14:12:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:12:25.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.406+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa5a006b020 msgr2=0x7fa5a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.406+0000 7fa5c6116640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa5a006b020 0x7fa5a006d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0006fd0 tx=0x7fa5b0008040 comp rx=0 tx=0).stop 2023-10-16T14:12:25.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.406+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 msgr2=0x7fa5c0160030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.406+0000 7fa5c6116640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 0x7fa5c0160030 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000c00 tx=0x7fa5a80046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.407+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 shutdown_connections 2023-10-16T14:12:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.407+0000 7fa5c6116640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa5a006b020 0x7fa5a006d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.407+0000 7fa5c6116640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5c0160b90 0x7fa5c01f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.407+0000 7fa5c6116640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5c015d150 0x7fa5c0160030 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.407+0000 7fa5c6116640 1 --2- 172.21.15.60:0/1028406182 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa5c0069da0 0x7fa5c00c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.407+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 >> 172.21.15.60:0/1028406182 conn(0x7fa5c00ce9b0 msgr2=0x7fa5c006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:25.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.408+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 shutdown_connections 2023-10-16T14:12:25.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.408+0000 7fa5c6116640 1 -- 172.21.15.60:0/1028406182 wait complete. 2023-10-16T14:12:25.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T14:12:25.433 INFO:tasks.ceph.mds.d.smithi144.stderr: -1721> 2023-10-16T14:11:57.853+0000 7f80fcd30640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:12:25.504 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T14:12:25.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.663+0000 7f8280d67640 1 Processor -- start 2023-10-16T14:12:25.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.664+0000 7f8280d67640 1 -- start start 2023-10-16T14:12:25.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.665+0000 7f8280d67640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f827c0cded0 0x7f827c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.665+0000 7f8280d67640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c15d150 con 0x7f827c0b8480 2023-10-16T14:12:25.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.665+0000 7f8280d67640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0d1180 con 0x7f827c15cd70 2023-10-16T14:12:25.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.665+0000 7f8280d67640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c0d1460 con 0x7f827c0cded0 2023-10-16T14:12:25.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.666+0000 7f827bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f827c0cded0 0x7f827c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.666+0000 7f827bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f827c0cded0 0x7f827c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38174/0 (socket says 172.21.15.60:38174) 2023-10-16T14:12:25.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.666+0000 7f827bfff640 1 -- 172.21.15.60:0/3991554625 learned_addr learned my addr 172.21.15.60:0/3991554625 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:25.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.666+0000 7f827bfff640 1 -- 172.21.15.60:0/3991554625 >> v1:172.21.15.144:6789/0 conn(0x7f827c15cd70 legacy=0x7f827c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:25.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.666+0000 7f827bfff640 1 -- 172.21.15.60:0/3991554625 >> v1:172.21.15.60:6789/0 conn(0x7f827c0b8480 legacy=0x7f827c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.666+0000 7f827bfff640 1 -- 172.21.15.60:0/3991554625 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0d1740 con 0x7f827c0cded0 2023-10-16T14:12:25.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.667+0000 7f827bfff640 1 --2- 172.21.15.60:0/3991554625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c0cded0 0x7f827c0b5c90 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f8274009e50 tx=0x7f827402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30629b76724c0f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:25.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f827affd640 1 -- 172.21.15.60:0/3991554625 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827402cb10 con 0x7f827c0cded0 2023-10-16T14:12:25.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f827affd640 1 -- 172.21.15.60:0/3991554625 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827402cc70 con 0x7f827c0cded0 2023-10-16T14:12:25.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f827affd640 1 -- 172.21.15.60:0/3991554625 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82740343f0 con 0x7f827c0cded0 2023-10-16T14:12:25.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 -- 172.21.15.60:0/3991554625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c0cded0 msgr2=0x7f827c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 --2- 172.21.15.60:0/3991554625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c0cded0 0x7f827c0b5c90 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f8274009e50 tx=0x7f827402bf70 comp rx=0 tx=0).stop 2023-10-16T14:12:25.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 -- 172.21.15.60:0/3991554625 shutdown_connections 2023-10-16T14:12:25.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 --2- 172.21.15.60:0/3991554625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c0cded0 0x7f827c0b5c90 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 -- 172.21.15.60:0/3991554625 >> 172.21.15.60:0/3991554625 conn(0x7f827c0cf570 msgr2=0x7f827c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:25.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 -- 172.21.15.60:0/3991554625 shutdown_connections 2023-10-16T14:12:25.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.668+0000 7f8280d67640 1 -- 172.21.15.60:0/3991554625 wait complete. 2023-10-16T14:12:25.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.670+0000 7f8280d67640 1 Processor -- start 2023-10-16T14:12:25.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.670+0000 7f8280d67640 1 -- start start 2023-10-16T14:12:25.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f8280d67640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f827c0b8480 0x7f827c14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f8280d67640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f827c0cded0 0x7f827c1f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f8280d67640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 0x7f827c0b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f8280d67640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c15dff0 con 0x7f827c15cd70 2023-10-16T14:12:25.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f8280d67640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c159890 con 0x7f827c0b8480 2023-10-16T14:12:25.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f8280d67640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c15d3e0 con 0x7f827c0cded0 2023-10-16T14:12:25.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f827bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 0x7f827c0b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f827bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 0x7f827c0b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38176/0 (socket says 172.21.15.60:38176) 2023-10-16T14:12:25.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.671+0000 7f827bfff640 1 -- 172.21.15.60:0/1940735780 learned_addr learned my addr 172.21.15.60:0/1940735780 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:25.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827b7fe640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f827c0b8480 0x7f827c14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827bfff640 1 -- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f827c0cded0 msgr2=0x7f827c1f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827bfff640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f827c0cded0 0x7f827c1f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827bfff640 1 -- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f827c0b8480 msgr2=0x7f827c14ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:25.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827bfff640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f827c0b8480 0x7f827c14ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:25.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827bfff640 1 -- 172.21.15.60:0/1940735780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274009b30 con 0x7f827c15cd70 2023-10-16T14:12:25.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827b7fe640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f827c0b8480 0x7f827c14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:25.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.672+0000 7f827bfff640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 0x7f827c0b2e70 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f8274036040 tx=0x7f8274004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:25.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.673+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274009510 con 0x7f827c15cd70 2023-10-16T14:12:25.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.673+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827403e450 con 0x7f827c15cd70 2023-10-16T14:12:25.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.673+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827403e800 con 0x7f827c15cd70 2023-10-16T14:12:25.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.673+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c15d6c0 con 0x7f827c15cd70 2023-10-16T14:12:25.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.673+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c15dc80 con 0x7f827c15cd70 2023-10-16T14:12:25.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.674+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c15e160 con 0x7f827c15cd70 2023-10-16T14:12:25.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.675+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f827403e960 con 0x7f827c15cd70 2023-10-16T14:12:25.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.676+0000 7f8278ff9640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f825c06b020 0x7f825c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:25.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.676+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6753+0+0 (secure 0 0 0) 0x7f82740acb90 con 0x7f827c15cd70 2023-10-16T14:12:25.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.676+0000 7f827b7fe640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f825c06b020 0x7f825c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:25.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.678+0000 7f827b7fe640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f825c06b020 0x7f825c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a810 tx=0x7f826c008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:25.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.680+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8274077590 con 0x7f827c15cd70 2023-10-16T14:12:25.757 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:25.752+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:25.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:25.917+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f827c04e9b0 con 0x7f827c15cd70 2023-10-16T14:12:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.416+0000 7f8278ff9640 1 -- 172.21.15.60:0/1940735780 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v138) v1 ==== 107+0+0 (secure 0 0 0) 0x7f827407abb0 con 0x7f827c15cd70 2023-10-16T14:12:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f825c06b020 msgr2=0x7f825c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f825c06b020 0x7f825c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a810 tx=0x7f826c008040 comp rx=0 tx=0).stop 2023-10-16T14:12:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 msgr2=0x7f827c0b2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 0x7f827c0b2e70 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f8274036040 tx=0x7f8274004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 shutdown_connections 2023-10-16T14:12:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f825c06b020 0x7f825c06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f827c0cded0 0x7f827c1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f827c15cd70 0x7f827c0b2e70 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 --2- 172.21.15.60:0/1940735780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f827c0b8480 0x7f827c14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.420+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 >> 172.21.15.60:0/1940735780 conn(0x7f827c0cf570 msgr2=0x7f827c15bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.421+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 shutdown_connections 2023-10-16T14:12:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.421+0000 7f8280d67640 1 -- 172.21.15.60:0/1940735780 wait complete. 2023-10-16T14:12:26.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T14:12:26.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.675+0000 7f0227fc9640 1 Processor -- start 2023-10-16T14:12:26.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.675+0000 7f0227fc9640 1 -- start start 2023-10-16T14:12:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0227fc9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02200b8a10 0x7f02200d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0227fc9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02200b4750 0x7f02200b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0227fc9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02200d1420 con 0x7f022015cf50 2023-10-16T14:12:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0227fc9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02200b50b0 con 0x7f02200b8a10 2023-10-16T14:12:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0227fc9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022015d330 con 0x7f02200b4750 2023-10-16T14:12:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0225564640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02200b8a10 0x7f02200d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0225564640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02200b8a10 0x7f02200d2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38192/0 (socket says 172.21.15.60:38192) 2023-10-16T14:12:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.677+0000 7f0225564640 1 -- 172.21.15.60:0/2367289828 learned_addr learned my addr 172.21.15.60:0/2367289828 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:26.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.678+0000 7f0226566640 1 --2- 172.21.15.60:0/2367289828 >> v2:172.21.15.144:3300/0 conn(0x7f02200b4750 0x7f02200b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.678+0000 7f0225564640 1 -- 172.21.15.60:0/2367289828 >> v2:172.21.15.144:3300/0 conn(0x7f02200b4750 msgr2=0x7f02200b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.678+0000 7f0225564640 1 --2- 172.21.15.60:0/2367289828 >> v2:172.21.15.144:3300/0 conn(0x7f02200b4750 0x7f02200b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.678+0000 7f0225564640 1 -- 172.21.15.60:0/2367289828 >> v1:172.21.15.144:6789/0 conn(0x7f022015cf50 legacy=0x7f02200d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.678+0000 7f0225564640 1 -- 172.21.15.60:0/2367289828 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02200d1680 con 0x7f02200b8a10 2023-10-16T14:12:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.679+0000 7f0225564640 1 --2- 172.21.15.60:0/2367289828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b8a10 0x7f02200d2720 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f02200745a0 tx=0x7f021002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d9cb480dfbfa7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.679+0000 7f0224d63640 1 -- 172.21.15.60:0/2367289828 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210031640 con 0x7f02200b8a10 2023-10-16T14:12:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0224d63640 1 -- 172.21.15.60:0/2367289828 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02100317a0 con 0x7f02200b8a10 2023-10-16T14:12:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0224d63640 1 -- 172.21.15.60:0/2367289828 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210031b30 con 0x7f02200b8a10 2023-10-16T14:12:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 -- 172.21.15.60:0/2367289828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b8a10 msgr2=0x7f02200d2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 --2- 172.21.15.60:0/2367289828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b8a10 0x7f02200d2720 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f02200745a0 tx=0x7f021002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 -- 172.21.15.60:0/2367289828 shutdown_connections 2023-10-16T14:12:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 --2- 172.21.15.60:0/2367289828 >> v2:172.21.15.144:3300/0 conn(0x7f02200b4750 0x7f02200b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 --2- 172.21.15.60:0/2367289828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b8a10 0x7f02200d2720 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 -- 172.21.15.60:0/2367289828 >> 172.21.15.60:0/2367289828 conn(0x7f02200cf750 msgr2=0x7f02200cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 -- 172.21.15.60:0/2367289828 shutdown_connections 2023-10-16T14:12:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.680+0000 7f0227fc9640 1 -- 172.21.15.60:0/2367289828 wait complete. 2023-10-16T14:12:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.682+0000 7f0227fc9640 1 Processor -- start 2023-10-16T14:12:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.682+0000 7f0227fc9640 1 -- start start 2023-10-16T14:12:26.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.683+0000 7f0227fc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 0x7f02201f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:26.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.683+0000 7f0227fc9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02200b8a10 0x7f0220159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:26.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.683+0000 7f0227fc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f022015cf50 0x7f022015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:26.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.683+0000 7f0227fc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022014ad30 con 0x7f02200b4750 2023-10-16T14:12:26.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.683+0000 7f0227fc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02200cb850 con 0x7f022015cf50 2023-10-16T14:12:26.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.683+0000 7f0227fc9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02200cac10 con 0x7f02200b8a10 2023-10-16T14:12:26.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 0x7f02201f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:26.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 0x7f02201f5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38204/0 (socket says 172.21.15.60:38204) 2023-10-16T14:12:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 -- 172.21.15.60:0/1366852470 learned_addr learned my addr 172.21.15.60:0/1366852470 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225564640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02200b8a10 0x7f0220159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0226566640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f022015cf50 0x7f022015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 -- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02200b8a10 msgr2=0x7f0220159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02200b8a10 0x7f0220159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 -- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f022015cf50 msgr2=0x7f022015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f022015cf50 0x7f022015a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.684+0000 7f0225d65640 1 -- 172.21.15.60:0/1366852470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100094d0 con 0x7f02200b4750 2023-10-16T14:12:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.685+0000 7f0225d65640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 0x7f02201f5290 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f021c00c430 tx=0x7f021c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.685+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c011790 con 0x7f02200b4750 2023-10-16T14:12:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.685+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c0118f0 con 0x7f02200b4750 2023-10-16T14:12:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.685+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c011ce0 con 0x7f02200b4750 2023-10-16T14:12:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.685+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022014afa0 con 0x7f02200b4750 2023-10-16T14:12:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.685+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02200ca650 con 0x7f02200b4750 2023-10-16T14:12:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.686+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0220072350 con 0x7f02200b4750 2023-10-16T14:12:26.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.687+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f021c01c460 con 0x7f02200b4750 2023-10-16T14:12:26.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.688+0000 7f02167fc640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f020406b020 0x7f020406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:26.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.688+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6753+0+0 (secure 0 0 0) 0x7f021c089d90 con 0x7f02200b4750 2023-10-16T14:12:26.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.689+0000 7f0225564640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f020406b020 0x7f020406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:26.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.690+0000 7f0225564640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f020406b020 0x7f020406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0210036000 tx=0x7f021003c040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:26.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.693+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f021c054700 con 0x7f02200b4750 2023-10-16T14:12:26.774 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:26.770+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:26.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:26.894+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f02200d1ed0 con 0x7f02200b4750 2023-10-16T14:12:27.813 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:27.809+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:27.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.898+0000 7f02167fc640 1 -- 172.21.15.60:0/1366852470 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v263) v1 ==== 190+0+0 (secure 0 0 0) 0x7f021c057d20 con 0x7f02200b4750 2023-10-16T14:12:27.902 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_metadata' removed 2023-10-16T14:12:27.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.904+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f020406b020 msgr2=0x7f020406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:27.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.904+0000 7f0227fc9640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f020406b020 0x7f020406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0210036000 tx=0x7f021003c040 comp rx=0 tx=0).stop 2023-10-16T14:12:27.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.904+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 msgr2=0x7f02201f5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:27.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.904+0000 7f0227fc9640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 0x7f02201f5290 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f021c00c430 tx=0x7f021c00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:12:27.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 shutdown_connections 2023-10-16T14:12:27.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f020406b020 0x7f020406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:27.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f02200b8a10 0x7f0220159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:27.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f022015cf50 0x7f022015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 --2- 172.21.15.60:0/1366852470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02200b4750 0x7f02201f5290 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 >> 172.21.15.60:0/1366852470 conn(0x7f02200cf750 msgr2=0x7f022015bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 shutdown_connections 2023-10-16T14:12:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:27.905+0000 7f0227fc9640 1 -- 172.21.15.60:0/1366852470 wait complete. 2023-10-16T14:12:27.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T14:12:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.162+0000 7f94c5590640 1 Processor -- start 2023-10-16T14:12:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.163+0000 7f94c5590640 1 -- start start 2023-10-16T14:12:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.164+0000 7f94c5590640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c00712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.164+0000 7f94c5590640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f94c00cb600 0x7f94c00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94c5590640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c00cc070 con 0x7f94c0069da0 2023-10-16T14:12:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94c5590640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c0070e90 con 0x7f94c00cb600 2023-10-16T14:12:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94c5590640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c00717f0 con 0x7f94c015ac10 2023-10-16T14:12:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94bf7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f94c0069da0 0x7f94c006c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47594/0 (socket says 172.21.15.60:47594) 2023-10-16T14:12:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94bf7fe640 1 -- 172.21.15.60:0/192339916 learned_addr learned my addr 172.21.15.60:0/192339916 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94beffd640 1 --2- 172.21.15.60:0/192339916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c00712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.165+0000 7f94b67fc640 1 --2- 172.21.15.60:0/192339916 >> v2:172.21.15.144:3300/0 conn(0x7f94c00cb600 0x7f94c00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94be7fc640 1 -- 172.21.15.60:0/192339916 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060097812 0 0) 0x7f94c00cc070 con 0x7f94c0069da0 2023-10-16T14:12:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94be7fc640 1 -- 172.21.15.60:0/192339916 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac003520 con 0x7f94c0069da0 2023-10-16T14:12:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94beffd640 1 -- 172.21.15.60:0/192339916 >> v2:172.21.15.144:3300/0 conn(0x7f94c00cb600 msgr2=0x7f94c00cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94beffd640 1 --2- 172.21.15.60:0/192339916 >> v2:172.21.15.144:3300/0 conn(0x7f94c00cb600 0x7f94c00cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94be7fc640 1 -- 172.21.15.60:0/192339916 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905249008 0 0) 0x7f94ac003520 con 0x7f94c0069da0 2023-10-16T14:12:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94beffd640 1 -- 172.21.15.60:0/192339916 >> v1:172.21.15.60:6789/0 conn(0x7f94c0069da0 legacy=0x7f94c006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.166+0000 7f94beffd640 1 -- 172.21.15.60:0/192339916 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c00707d0 con 0x7f94c015ac10 2023-10-16T14:12:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.168+0000 7f94beffd640 1 --2- 172.21.15.60:0/192339916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c00712b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f94b00097a0 tx=0x7f94b002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f6b2be88c735ea5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:28.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.168+0000 7f94be7fc640 1 -- 172.21.15.60:0/192339916 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b80019f0 con 0x7f94c015ac10 2023-10-16T14:12:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.168+0000 7f94be7fc640 1 -- 172.21.15.60:0/192339916 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b002e050 con 0x7f94c015ac10 2023-10-16T14:12:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.168+0000 7f94be7fc640 1 -- 172.21.15.60:0/192339916 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0030730 con 0x7f94c015ac10 2023-10-16T14:12:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 -- 172.21.15.60:0/192339916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 msgr2=0x7f94c00712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:28.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 --2- 172.21.15.60:0/192339916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c00712b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f94b00097a0 tx=0x7f94b002b710 comp rx=0 tx=0).stop 2023-10-16T14:12:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 -- 172.21.15.60:0/192339916 shutdown_connections 2023-10-16T14:12:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 --2- 172.21.15.60:0/192339916 >> v2:172.21.15.144:3300/0 conn(0x7f94c00cb600 0x7f94c00cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 --2- 172.21.15.60:0/192339916 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c00712b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 -- 172.21.15.60:0/192339916 >> 172.21.15.60:0/192339916 conn(0x7f94c00cc470 msgr2=0x7f94c00cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 -- 172.21.15.60:0/192339916 shutdown_connections 2023-10-16T14:12:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.169+0000 7f94c5590640 1 -- 172.21.15.60:0/192339916 wait complete. 2023-10-16T14:12:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.171+0000 7f94c5590640 1 Processor -- start 2023-10-16T14:12:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.171+0000 7f94c5590640 1 -- start start 2023-10-16T14:12:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94c5590640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 0x7f94c00b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:28.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94c5590640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c00cb600 0x7f94c015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94c5590640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c015b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94c5590640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0158820 con 0x7f94c0069da0 2023-10-16T14:12:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94c5590640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0154810 con 0x7f94c00cb600 2023-10-16T14:12:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94c5590640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c00b2340 con 0x7f94c015ac10 2023-10-16T14:12:28.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 0x7f94c00b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 0x7f94c00b1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38206/0 (socket says 172.21.15.60:38206) 2023-10-16T14:12:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 -- 172.21.15.60:0/3530661568 learned_addr learned my addr 172.21.15.60:0/3530661568 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94b67fc640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c00cb600 0x7f94c015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:28.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c015b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 -- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 msgr2=0x7f94c015b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c015b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 -- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c00cb600 msgr2=0x7f94c015b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c00cb600 0x7f94c015b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94beffd640 1 -- 172.21.15.60:0/3530661568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b0009400 con 0x7f94c0069da0 2023-10-16T14:12:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94b67fc640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c00cb600 0x7f94c015b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.172+0000 7f94bf7fe640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c015b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.173+0000 7f94beffd640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 0x7f94c00b1e00 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f94b0000c00 tx=0x7f94b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.173+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b003c620 con 0x7f94c0069da0 2023-10-16T14:12:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.173+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b003c780 con 0x7f94c0069da0 2023-10-16T14:12:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.173+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b003cb30 con 0x7f94c0069da0 2023-10-16T14:12:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.173+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c00b3270 con 0x7f94c0069da0 2023-10-16T14:12:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.173+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c00c89a0 con 0x7f94c0069da0 2023-10-16T14:12:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.175+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c00c84f0 con 0x7f94c0069da0 2023-10-16T14:12:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.175+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f94b0039070 con 0x7f94c0069da0 2023-10-16T14:12:28.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.176+0000 7f94b7fff640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f94a006b020 0x7f94a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:28.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.177+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6309+0+0 (secure 0 0 0) 0x7f94b00ab520 con 0x7f94c0069da0 2023-10-16T14:12:28.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.177+0000 7f94b67fc640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f94a006b020 0x7f94a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:28.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.178+0000 7f94b67fc640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f94a006b020 0x7f94a006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94c00cefd0 tx=0x7f94a8008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:28.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.181+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94b0075eb0 con 0x7f94c0069da0 2023-10-16T14:12:28.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.386+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94c004e160 con 0x7f94c0069da0 2023-10-16T14:12:28.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:12:28.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:12:28.828 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:28.824+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.902+0000 7f94b7fff640 1 -- 172.21.15.60:0/3530661568 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v264) v1 ==== 178+0+0 (secure 0 0 0) 0x7f94b004a0d0 con 0x7f94c0069da0 2023-10-16T14:12:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'backup_fs_data' removed 2023-10-16T14:12:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f94a006b020 msgr2=0x7f94a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f94a006b020 0x7f94a006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94c00cefd0 tx=0x7f94a8008040 comp rx=0 tx=0).stop 2023-10-16T14:12:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 msgr2=0x7f94c00b1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 0x7f94c00b1e00 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f94b0000c00 tx=0x7f94b00046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 shutdown_connections 2023-10-16T14:12:28.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f94a006b020 0x7f94a006d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f94c015ac10 0x7f94c015b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94c00cb600 0x7f94c015b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.906+0000 7f94c5590640 1 --2- 172.21.15.60:0/3530661568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94c0069da0 0x7f94c00b1e00 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:28.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.907+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 >> 172.21.15.60:0/3530661568 conn(0x7f94c00cc470 msgr2=0x7f94c00cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.909+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 shutdown_connections 2023-10-16T14:12:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:28.909+0000 7f94c5590640 1 -- 172.21.15.60:0/3530661568 wait complete. 2023-10-16T14:12:28.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:12:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.147+0000 7f68209c8640 1 Processor -- start 2023-10-16T14:12:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.148+0000 7f68209c8640 1 -- start start 2023-10-16T14:12:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.149+0000 7f68209c8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f681c0adb30 0x7f681c0cd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.149+0000 7f68209c8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f681c0b0c90 0x7f681c0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.149+0000 7f68209c8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c06f3a0 con 0x7f681c06c630 2023-10-16T14:12:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.149+0000 7f68209c8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0c8e70 con 0x7f681c0b0c90 2023-10-16T14:12:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.149+0000 7f68209c8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0c9150 con 0x7f681c0adb30 2023-10-16T14:12:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f681c06c630 0x7f681c0cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47264/0 (socket says 172.21.15.60:47264) 2023-10-16T14:12:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681ad9d640 1 -- 172.21.15.60:0/3202743812 learned_addr learned my addr 172.21.15.60:0/3202743812 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681b59e640 1 --2- 172.21.15.60:0/3202743812 >> v2:172.21.15.60:3300/0 conn(0x7f681c0b0c90 0x7f681c0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681259c640 1 --2- 172.21.15.60:0/3202743812 >> v2:172.21.15.144:3300/0 conn(0x7f681c0adb30 0x7f681c0cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681b59e640 1 -- 172.21.15.60:0/3202743812 >> v2:172.21.15.144:3300/0 conn(0x7f681c0adb30 msgr2=0x7f681c0cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681b59e640 1 --2- 172.21.15.60:0/3202743812 >> v2:172.21.15.144:3300/0 conn(0x7f681c0adb30 0x7f681c0cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681b59e640 1 -- 172.21.15.60:0/3202743812 >> v1:172.21.15.60:6789/0 conn(0x7f681c06c630 legacy=0x7f681c0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681b59e640 1 -- 172.21.15.60:0/3202743812 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0c9430 con 0x7f681c0b0c90 2023-10-16T14:12:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.150+0000 7f681a59c640 1 -- 172.21.15.60:0/3202743812 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944415136 0 0) 0x7f681c06f3a0 con 0x7f681c06c630 2023-10-16T14:12:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.151+0000 7f681b59e640 1 --2- 172.21.15.60:0/3202743812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0b0c90 0x7f681c0c9950 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f681c073cb0 tx=0x7f681402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f470567424c9cb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f681a59c640 1 -- 172.21.15.60:0/3202743812 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681402e070 con 0x7f681c0b0c90 2023-10-16T14:12:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f681a59c640 1 -- 172.21.15.60:0/3202743812 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68140315a0 con 0x7f681c0b0c90 2023-10-16T14:12:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f681a59c640 1 -- 172.21.15.60:0/3202743812 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814031930 con 0x7f681c0b0c90 2023-10-16T14:12:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f68209c8640 1 -- 172.21.15.60:0/3202743812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0b0c90 msgr2=0x7f681c0c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f68209c8640 1 --2- 172.21.15.60:0/3202743812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0b0c90 0x7f681c0c9950 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f681c073cb0 tx=0x7f681402b840 comp rx=0 tx=0).stop 2023-10-16T14:12:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f68209c8640 1 -- 172.21.15.60:0/3202743812 shutdown_connections 2023-10-16T14:12:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f68209c8640 1 --2- 172.21.15.60:0/3202743812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0b0c90 0x7f681c0c9950 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.152+0000 7f68209c8640 1 --2- 172.21.15.60:0/3202743812 >> v2:172.21.15.144:3300/0 conn(0x7f681c0adb30 0x7f681c0cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.153+0000 7f68209c8640 1 -- 172.21.15.60:0/3202743812 >> 172.21.15.60:0/3202743812 conn(0x7f681c0b0170 msgr2=0x7f681c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.153+0000 7f68209c8640 1 -- 172.21.15.60:0/3202743812 shutdown_connections 2023-10-16T14:12:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.153+0000 7f68209c8640 1 -- 172.21.15.60:0/3202743812 wait complete. 2023-10-16T14:12:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.154+0000 7f68209c8640 1 Processor -- start 2023-10-16T14:12:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.154+0000 7f68209c8640 1 -- start start 2023-10-16T14:12:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.155+0000 7f68209c8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f681c06c630 0x7f681c0c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.155+0000 7f68209c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 0x7f681c0ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.155+0000 7f68209c8640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f681c0b0c90 0x7f681c0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.155+0000 7f68209c8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1fa350 con 0x7f681c0adb30 2023-10-16T14:12:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.155+0000 7f68209c8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1f76a0 con 0x7f681c06c630 2023-10-16T14:12:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.155+0000 7f68209c8640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1f7980 con 0x7f681c0b0c90 2023-10-16T14:12:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 0x7f681c0ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 0x7f681c0ae250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60106/0 (socket says 172.21.15.60:60106) 2023-10-16T14:12:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 -- 172.21.15.60:0/1899929843 learned_addr learned my addr 172.21.15.60:0/1899929843 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681ad9d640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f681c06c630 0x7f681c0c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681b59e640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f681c0b0c90 0x7f681c0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 -- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f681c0b0c90 msgr2=0x7f681c0b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f681c0b0c90 0x7f681c0b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 -- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f681c06c630 msgr2=0x7f681c0c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f681c06c630 0x7f681c0c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681259c640 1 -- 172.21.15.60:0/1899929843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814009400 con 0x7f681c0adb30 2023-10-16T14:12:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.156+0000 7f681b59e640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f681c0b0c90 0x7f681c0b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.157+0000 7f681259c640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 0x7f681c0ae250 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f680400d800 tx=0x7f680400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.157+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680400b6c0 con 0x7f681c0adb30 2023-10-16T14:12:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.157+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680400b820 con 0x7f681c0adb30 2023-10-16T14:12:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.157+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680400bbb0 con 0x7f681c0adb30 2023-10-16T14:12:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.157+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c0c5d40 con 0x7f681c0adb30 2023-10-16T14:12:29.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.157+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c0c6300 con 0x7f681c0adb30 2023-10-16T14:12:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.158+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c0c6490 con 0x7f681c0adb30 2023-10-16T14:12:29.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.159+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f680400bd10 con 0x7f681c0adb30 2023-10-16T14:12:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.161+0000 7f6813fff640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f67fc06b020 0x7f67fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.161+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f6804089ff0 con 0x7f681c0adb30 2023-10-16T14:12:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.162+0000 7f681ad9d640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f67fc06b020 0x7f67fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.163+0000 7f681ad9d640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f67fc06b020 0x7f67fc06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f680c0012e0 tx=0x7f680c007ed0 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:29.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.166+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f68040bcfe0 con 0x7f681c0adb30 2023-10-16T14:12:29.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.369+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f681c04e9b0 con 0x7f681c0adb30 2023-10-16T14:12:29.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.374+0000 7f6813fff640 1 -- 172.21.15.60:0/1899929843 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6804054d00 con 0x7f681c0adb30 2023-10-16T14:12:29.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f67fc06b020 msgr2=0x7f67fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f67fc06b020 0x7f67fc06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f680c0012e0 tx=0x7f680c007ed0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 msgr2=0x7f681c0ae250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 0x7f681c0ae250 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f680400d800 tx=0x7f680400dcc0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 shutdown_connections 2023-10-16T14:12:29.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f67fc06b020 0x7f67fc06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f681c0b0c90 0x7f681c0b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f681c0adb30 0x7f681c0ae250 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 --2- 172.21.15.60:0/1899929843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f681c06c630 0x7f681c0c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 >> 172.21.15.60:0/1899929843 conn(0x7f681c0b0170 msgr2=0x7f681c0b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:29.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 shutdown_connections 2023-10-16T14:12:29.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.378+0000 7f68209c8640 1 -- 172.21.15.60:0/1899929843 wait complete. 2023-10-16T14:12:29.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:12:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.617+0000 7faf48e7c640 1 Processor -- start 2023-10-16T14:12:29.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.617+0000 7faf48e7c640 1 -- start start 2023-10-16T14:12:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.618+0000 7faf48e7c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf44070830 0x7faf4406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.618+0000 7faf48e7c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf44161740 con 0x7faf44070c10 2023-10-16T14:12:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.618+0000 7faf48e7c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4406b0f0 con 0x7faf440cf5f0 2023-10-16T14:12:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.618+0000 7faf48e7c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf441619b0 con 0x7faf44070830 2023-10-16T14:12:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf4259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf44070830 0x7faf4406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf42d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faf44070c10 0x7faf440cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47280/0 (socket says 172.21.15.60:47280) 2023-10-16T14:12:29.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf4259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf44070830 0x7faf4406d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60114/0 (socket says 172.21.15.60:60114) 2023-10-16T14:12:29.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf42d9d640 1 -- 172.21.15.60:0/1479011975 learned_addr learned my addr 172.21.15.60:0/1479011975 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:29.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf4259c640 1 -- 172.21.15.60:0/1479011975 >> v1:172.21.15.144:6789/0 conn(0x7faf440cf5f0 legacy=0x7faf4406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf4259c640 1 -- 172.21.15.60:0/1479011975 >> v1:172.21.15.60:6789/0 conn(0x7faf44070c10 legacy=0x7faf440cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.619+0000 7faf4259c640 1 -- 172.21.15.60:0/1479011975 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4406b3d0 con 0x7faf44070830 2023-10-16T14:12:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.620+0000 7faf4259c640 1 --2- 172.21.15.60:0/1479011975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070830 0x7faf4406d3c0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7faf340097a0 tx=0x7faf3402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46f88263aecdb7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:29.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf41d9b640 1 -- 172.21.15.60:0/1479011975 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3402f050 con 0x7faf44070830 2023-10-16T14:12:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf41d9b640 1 -- 172.21.15.60:0/1479011975 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf34031700 con 0x7faf44070830 2023-10-16T14:12:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf41d9b640 1 -- 172.21.15.60:0/1479011975 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34031a90 con 0x7faf44070830 2023-10-16T14:12:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf48e7c640 1 -- 172.21.15.60:0/1479011975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070830 msgr2=0x7faf4406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf48e7c640 1 --2- 172.21.15.60:0/1479011975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070830 0x7faf4406d3c0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7faf340097a0 tx=0x7faf3402b7a0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf48e7c640 1 -- 172.21.15.60:0/1479011975 shutdown_connections 2023-10-16T14:12:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf48e7c640 1 --2- 172.21.15.60:0/1479011975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070830 0x7faf4406d3c0 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf48e7c640 1 -- 172.21.15.60:0/1479011975 >> 172.21.15.60:0/1479011975 conn(0x7faf440cfac0 msgr2=0x7faf440cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.621+0000 7faf48e7c640 1 -- 172.21.15.60:0/1479011975 shutdown_connections 2023-10-16T14:12:29.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.622+0000 7faf48e7c640 1 -- 172.21.15.60:0/1479011975 wait complete. 2023-10-16T14:12:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.623+0000 7faf48e7c640 1 Processor -- start 2023-10-16T14:12:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.623+0000 7faf48e7c640 1 -- start start 2023-10-16T14:12:29.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.624+0000 7faf48e7c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf44070830 0x7faf44157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.624+0000 7faf48e7c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 0x7faf440c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.624+0000 7faf48e7c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf440cf5f0 0x7faf440c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.624+0000 7faf48e7c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4415aed0 con 0x7faf44070c10 2023-10-16T14:12:29.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.624+0000 7faf48e7c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4415b0e0 con 0x7faf440cf5f0 2023-10-16T14:12:29.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf48e7c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4415b3c0 con 0x7faf44070830 2023-10-16T14:12:29.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 0x7faf440c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 0x7faf440c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60128/0 (socket says 172.21.15.60:60128) 2023-10-16T14:12:29.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 -- 172.21.15.60:0/2886506234 learned_addr learned my addr 172.21.15.60:0/2886506234 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:29.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf4259c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf44070830 0x7faf44157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf42d9d640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf440cf5f0 0x7faf440c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 -- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf44070830 msgr2=0x7faf44157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf44070830 0x7faf44157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 -- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf440cf5f0 msgr2=0x7faf440c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf440cf5f0 0x7faf440c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.625+0000 7faf3bfff640 1 -- 172.21.15.60:0/2886506234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf34009400 con 0x7faf44070c10 2023-10-16T14:12:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.626+0000 7faf3bfff640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 0x7faf440c9810 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009fd0 tx=0x7faf2c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.626+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c011830 con 0x7faf44070c10 2023-10-16T14:12:29.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.626+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c011990 con 0x7faf44070c10 2023-10-16T14:12:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.626+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4415b6a0 con 0x7faf44070c10 2023-10-16T14:12:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.626+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c011d20 con 0x7faf44070c10 2023-10-16T14:12:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.626+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4415bc60 con 0x7faf44070c10 2023-10-16T14:12:29.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.627+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4415be80 con 0x7faf44070c10 2023-10-16T14:12:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.628+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7faf2c01a400 con 0x7faf44070c10 2023-10-16T14:12:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.629+0000 7faf3affd640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf2406b020 0x7faf2406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.629+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7faf2c089b50 con 0x7faf44070c10 2023-10-16T14:12:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.630+0000 7faf4259c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf2406b020 0x7faf2406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.631+0000 7faf4259c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf2406b020 0x7faf2406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf34035000 tx=0x7faf34008030 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.634+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf2c0548b0 con 0x7faf44070c10 2023-10-16T14:12:29.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.835+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7faf4404e160 con 0x7faf44070c10 2023-10-16T14:12:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.841+0000 7faf3affd640 1 -- 172.21.15.60:0/2886506234 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7faf2c057ed0 con 0x7faf44070c10 2023-10-16T14:12:29.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.843+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf2406b020 msgr2=0x7faf2406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.843+0000 7faf48e7c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf2406b020 0x7faf2406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf34035000 tx=0x7faf34008030 comp rx=0 tx=0).stop 2023-10-16T14:12:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.843+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 msgr2=0x7faf440c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.843+0000 7faf48e7c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 0x7faf440c9810 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009fd0 tx=0x7faf2c00b870 comp rx=0 tx=0).stop 2023-10-16T14:12:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 shutdown_connections 2023-10-16T14:12:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7faf2406b020 0x7faf2406d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf440cf5f0 0x7faf440c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf44070c10 0x7faf440c9810 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 --2- 172.21.15.60:0/2886506234 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf44070830 0x7faf44157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 >> 172.21.15.60:0/2886506234 conn(0x7faf440cfac0 msgr2=0x7faf440ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 shutdown_connections 2023-10-16T14:12:29.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:29.844+0000 7faf48e7c640 1 -- 172.21.15.60:0/2886506234 wait complete. 2023-10-16T14:12:29.865 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:29.861+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:29.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:12:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.096+0000 7f5507133640 1 Processor -- start 2023-10-16T14:12:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.096+0000 7f5507133640 1 -- start start 2023-10-16T14:12:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.097+0000 7f5507133640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f55000aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.097+0000 7f5507133640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550006c140 con 0x7f550015e9b0 2023-10-16T14:12:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.097+0000 7f5507133640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550015b2c0 con 0x7f55000cb810 2023-10-16T14:12:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.098+0000 7f5507133640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550006b270 con 0x7f550015aee0 2023-10-16T14:12:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.098+0000 7f55056d0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f550015e9b0 0x7f550015ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47286/0 (socket says 172.21.15.60:47286) 2023-10-16T14:12:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.098+0000 7f55056d0640 1 -- 172.21.15.60:0/3920634601 learned_addr learned my addr 172.21.15.60:0/3920634601 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.098+0000 7f5504ecf640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f55000aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.098+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783188365 0 0) 0x7f550006c140 con 0x7f550015e9b0 2023-10-16T14:12:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.098+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ec003520 con 0x7f550015e9b0 2023-10-16T14:12:30.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1875362599 0 0) 0x7f54ec003520 con 0x7f550015e9b0 2023-10-16T14:12:30.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 msgr2=0x7f55000aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:30.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f55000aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 >> v1:172.21.15.144:6789/0 conn(0x7f55000cb810 legacy=0x7f55000cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550006daf0 con 0x7f550015e9b0 2023-10-16T14:12:30.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f54f8001670 con 0x7f550015e9b0 2023-10-16T14:12:30.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 >> v1:172.21.15.60:6789/0 conn(0x7f550015e9b0 legacy=0x7f550015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f54ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec003f70 0x7f54ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54ec004d00 0x7f54ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550006b270 con 0x7f54ec004d00 2023-10-16T14:12:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec00bb70 con 0x7f54ec003f70 2023-10-16T14:12:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec00bd30 con 0x7f550015aee0 2023-10-16T14:12:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.099+0000 7f55056d0640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54ec004d00 0x7f54ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f54fffff640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec003f70 0x7f54ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f5504ecf640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f54ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f55056d0640 1 -- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 msgr2=0x7f54ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f55056d0640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f54ec0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f55056d0640 1 -- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec003f70 msgr2=0x7f54ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f55056d0640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec003f70 0x7f54ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f55056d0640 1 -- 172.21.15.60:0/3920634601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ec003520 con 0x7f54ec004d00 2023-10-16T14:12:30.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.100+0000 7f5504ecf640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f550015aee0 0x7f54ec0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:30.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.101+0000 7f55056d0640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54ec004d00 0x7f54ec005180 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f54f800c0e0 tx=0x7f54ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58eebaa1debeb28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.101+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8005ad0 con 0x7f54ec004d00 2023-10-16T14:12:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.101+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8005c30 con 0x7f54ec004d00 2023-10-16T14:12:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.101+0000 7f54ff7fe640 1 -- 172.21.15.60:0/3920634601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f802ca50 con 0x7f54ec004d00 2023-10-16T14:12:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f5507133640 1 -- 172.21.15.60:0/3920634601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54ec004d00 msgr2=0x7f54ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f5507133640 1 --2- 172.21.15.60:0/3920634601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54ec004d00 0x7f54ec005180 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f54f800c0e0 tx=0x7f54ec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f55056d0640 1 -- 172.21.15.60:0/3920634601 reap_dead start 2023-10-16T14:12:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f5507133640 1 -- 172.21.15.60:0/3920634601 shutdown_connections 2023-10-16T14:12:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f5507133640 1 -- 172.21.15.60:0/3920634601 >> 172.21.15.60:0/3920634601 conn(0x7f55000cfac0 msgr2=0x7f55000cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f5507133640 1 -- 172.21.15.60:0/3920634601 shutdown_connections 2023-10-16T14:12:30.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.102+0000 7f5507133640 1 -- 172.21.15.60:0/3920634601 wait complete. 2023-10-16T14:12:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.104+0000 7f5507133640 1 Processor -- start 2023-10-16T14:12:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.104+0000 7f5507133640 1 -- start start 2023-10-16T14:12:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f5507133640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec004d00 0x7f55001f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f5507133640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 0x7f55000ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f5507133640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55000aede0 0x7f5500148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f5507133640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5500147e50 con 0x7f55001e5bc0 2023-10-16T14:12:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f5507133640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55000c5f60 con 0x7f54ec004d00 2023-10-16T14:12:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f5507133640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55001f2540 con 0x7f55000aede0 2023-10-16T14:12:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f54fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 0x7f55000ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f54fffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 0x7f55000ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60148/0 (socket says 172.21.15.60:60148) 2023-10-16T14:12:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.105+0000 7f54fffff640 1 -- 172.21.15.60:0/1734985094 learned_addr learned my addr 172.21.15.60:0/1734985094 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f5504ecf640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec004d00 0x7f55001f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fffff640 1 -- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55000aede0 msgr2=0x7f5500148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f55056d0640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55000aede0 0x7f5500148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fffff640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55000aede0 0x7f5500148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fffff640 1 -- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec004d00 msgr2=0x7f55001f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fffff640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec004d00 0x7f55001f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fffff640 1 -- 172.21.15.60:0/1734985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55001471d0 con 0x7f55001e5bc0 2023-10-16T14:12:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f5504ecf640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec004d00 0x7f55001f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:30.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fffff640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 0x7f55000ae8a0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f54e800e600 tx=0x7f54e800eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.106+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e80052e0 con 0x7f55001e5bc0 2023-10-16T14:12:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.107+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e8005440 con 0x7f55001e5bc0 2023-10-16T14:12:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.107+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e8003810 con 0x7f55001e5bc0 2023-10-16T14:12:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.107+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55001474b0 con 0x7f55001e5bc0 2023-10-16T14:12:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.107+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5500147a70 con 0x7f55001e5bc0 2023-10-16T14:12:30.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.108+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550004e160 con 0x7f55001e5bc0 2023-10-16T14:12:30.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.109+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f54e8003970 con 0x7f55001e5bc0 2023-10-16T14:12:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.110+0000 7f54fd7fa640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f54e006b050 0x7f54e006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.110+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f54e80898b0 con 0x7f55001e5bc0 2023-10-16T14:12:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.111+0000 7f5504ecf640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f54e006b050 0x7f54e006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.112+0000 7f5504ecf640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f54e006b050 0x7f54e006d510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54f0007520 tx=0x7f54f000d040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:30.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.114+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54e8054610 con 0x7f55001e5bc0 2023-10-16T14:12:30.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.318+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f55000ce180 con 0x7f55001e5bc0 2023-10-16T14:12:30.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.324+0000 7f54fd7fa640 1 -- 172.21.15.60:0/1734985094 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f54e8057c30 con 0x7f55001e5bc0 2023-10-16T14:12:30.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.326+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f54e006b050 msgr2=0x7f54e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.326+0000 7f5507133640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f54e006b050 0x7f54e006d510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54f0007520 tx=0x7f54f000d040 comp rx=0 tx=0).stop 2023-10-16T14:12:30.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 msgr2=0x7f55000ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 0x7f55000ae8a0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f54e800e600 tx=0x7f54e800eac0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 shutdown_connections 2023-10-16T14:12:30.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f54e006b050 0x7f54e006d510 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55000aede0 0x7f5500148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55001e5bc0 0x7f55000ae8a0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 --2- 172.21.15.60:0/1734985094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54ec004d00 0x7f55001f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 >> 172.21.15.60:0/1734985094 conn(0x7f55000cfac0 msgr2=0x7f55000adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:30.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 shutdown_connections 2023-10-16T14:12:30.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.327+0000 7f5507133640 1 -- 172.21.15.60:0/1734985094 wait complete. 2023-10-16T14:12:30.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T14:12:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.568+0000 7f9a7caf2640 1 Processor -- start 2023-10-16T14:12:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.569+0000 7f9a7caf2640 1 -- start start 2023-10-16T14:12:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.570+0000 7f9a7caf2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a7815aaa0 0x7f9a7815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.570+0000 7f9a7caf2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a78070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.570+0000 7f9a7caf2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7815ed60 con 0x7f9a7815ae80 2023-10-16T14:12:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.570+0000 7f9a7caf2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7806b810 con 0x7f9a7815aaa0 2023-10-16T14:12:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.570+0000 7f9a7caf2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a780707e0 con 0x7f9a780ce390 2023-10-16T14:12:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a7815aaa0 0x7f9a7815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a7815aaa0 0x7f9a7815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60152/0 (socket says 172.21.15.60:60152) 2023-10-16T14:12:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 -- 172.21.15.60:0/2678641789 learned_addr learned my addr 172.21.15.60:0/2678641789 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a76d9d640 1 --2- 172.21.15.60:0/2678641789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a78070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 -- 172.21.15.60:0/2678641789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 msgr2=0x7f9a78070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:30.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 --2- 172.21.15.60:0/2678641789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a78070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 -- 172.21.15.60:0/2678641789 >> v1:172.21.15.144:6789/0 conn(0x7f9a7815ae80 legacy=0x7f9a780cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.571+0000 7f9a7659c640 1 -- 172.21.15.60:0/2678641789 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7815ef90 con 0x7f9a7815aaa0 2023-10-16T14:12:30.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.573+0000 7f9a7659c640 1 --2- 172.21.15.60:0/2678641789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815e820 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009e50 tx=0x7f9a6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9508884ef23a3840 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:30.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.573+0000 7f9a7559a640 1 -- 172.21.15.60:0/2678641789 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6802cb00 con 0x7f9a7815aaa0 2023-10-16T14:12:30.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.573+0000 7f9a7559a640 1 -- 172.21.15.60:0/2678641789 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6802cc60 con 0x7f9a7815aaa0 2023-10-16T14:12:30.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.573+0000 7f9a7559a640 1 -- 172.21.15.60:0/2678641789 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a680343f0 con 0x7f9a7815aaa0 2023-10-16T14:12:30.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 -- 172.21.15.60:0/2678641789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815aaa0 msgr2=0x7f9a7815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/2678641789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815e820 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009e50 tx=0x7f9a6802bf60 comp rx=0 tx=0).stop 2023-10-16T14:12:30.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 -- 172.21.15.60:0/2678641789 shutdown_connections 2023-10-16T14:12:30.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/2678641789 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a78070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/2678641789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815e820 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 -- 172.21.15.60:0/2678641789 >> 172.21.15.60:0/2678641789 conn(0x7f9a780cb410 msgr2=0x7f9a780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:30.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 -- 172.21.15.60:0/2678641789 shutdown_connections 2023-10-16T14:12:30.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.574+0000 7f9a7caf2640 1 -- 172.21.15.60:0/2678641789 wait complete. 2023-10-16T14:12:30.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.576+0000 7f9a7caf2640 1 Processor -- start 2023-10-16T14:12:30.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.576+0000 7f9a7caf2640 1 -- start start 2023-10-16T14:12:30.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7caf2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a781efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7caf2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7caf2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 0x7f9a780b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7caf2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a780c8db0 con 0x7f9a7815ae80 2023-10-16T14:12:30.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7caf2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a781f3db0 con 0x7f9a7815aaa0 2023-10-16T14:12:30.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7caf2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a781f4090 con 0x7f9a780ce390 2023-10-16T14:12:30.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a76d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 0x7f9a780b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a76d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 0x7f9a780b2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60154/0 (socket says 172.21.15.60:60154) 2023-10-16T14:12:30.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a76d9d640 1 -- 172.21.15.60:0/3770142144 learned_addr learned my addr 172.21.15.60:0/3770142144 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:30.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a7659c640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a781efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.577+0000 7f9a75d9b640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a76d9d640 1 -- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 msgr2=0x7f9a781efdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a76d9d640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a781efdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a76d9d640 1 -- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a7815aaa0 msgr2=0x7f9a7815bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a76d9d640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a76d9d640 1 -- 172.21.15.60:0/3770142144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68009b30 con 0x7f9a7815ae80 2023-10-16T14:12:30.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a75d9b640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:30.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a76d9d640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 0x7f9a780b2b40 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000c550 tx=0x7f9a7000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:30.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70004220 con 0x7f9a7815ae80 2023-10-16T14:12:30.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a70004380 con 0x7f9a7815ae80 2023-10-16T14:12:30.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70003830 con 0x7f9a7815ae80 2023-10-16T14:12:30.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.578+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a781f4370 con 0x7f9a7815ae80 2023-10-16T14:12:30.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.579+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a781f02f0 con 0x7f9a7815ae80 2023-10-16T14:12:30.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.580+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7804e160 con 0x7f9a7815ae80 2023-10-16T14:12:30.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.581+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f9a70003990 con 0x7f9a7815ae80 2023-10-16T14:12:30.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.582+0000 7f9a6effd640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9a5806b020 0x7f9a5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:30.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.582+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f9a700897b0 con 0x7f9a7815ae80 2023-10-16T14:12:30.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.582+0000 7f9a7659c640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9a5806b020 0x7f9a5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:30.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.584+0000 7f9a7659c640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9a5806b020 0x7f9a5806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a68035000 tx=0x7f9a68030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:30.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.586+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a70054410 con 0x7f9a7815ae80 2023-10-16T14:12:30.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.857+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f9a780c30c0 con 0x7f9a7815ae80 2023-10-16T14:12:30.906 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:30.902+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:30.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.920+0000 7f9a6effd640 1 -- 172.21.15.60:0/3770142144 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3287) v1 ==== 187+0+0 (secure 0 0 0) 0x7f9a70057a30 con 0x7f9a7815ae80 2023-10-16T14:12:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9a5806b020 msgr2=0x7f9a5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9a5806b020 0x7f9a5806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a68035000 tx=0x7f9a68030000 comp rx=0 tx=0).stop 2023-10-16T14:12:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 msgr2=0x7f9a780b2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 0x7f9a780b2b40 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000c550 tx=0x7f9a7000ca10 comp rx=0 tx=0).stop 2023-10-16T14:12:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 shutdown_connections 2023-10-16T14:12:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9a5806b020 0x7f9a5806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a7815ae80 0x7f9a780b2b40 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a7815aaa0 0x7f9a7815bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 --2- 172.21.15.60:0/3770142144 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a780ce390 0x7f9a781efdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 >> 172.21.15.60:0/3770142144 conn(0x7f9a780cb410 msgr2=0x7f9a780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.924+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 shutdown_connections 2023-10-16T14:12:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:30.925+0000 7f9a7caf2640 1 -- 172.21.15.60:0/3770142144 wait complete. 2023-10-16T14:12:30.949 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2023-10-16T14:12:30.950 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-16T14:12:30.950 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-16T14:12:30.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:31.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.178+0000 7f2ae502b640 1 Processor -- start 2023-10-16T14:12:31.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.178+0000 7f2ae502b640 1 -- start start 2023-10-16T14:12:31.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2ae502b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2ae00ce390 0x7f2ae015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2ae502b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae015ed60 con 0x7f2ae015aaa0 2023-10-16T14:12:31.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2ae502b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae00707e0 con 0x7f2ae015ae80 2023-10-16T14:12:31.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2ae502b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae006b810 con 0x7f2ae00ce390 2023-10-16T14:12:31.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2adf59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2ae00ce390 0x7f2ae015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2adf59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2ae00ce390 0x7f2ae015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60164/0 (socket says 172.21.15.60:60164) 2023-10-16T14:12:31.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2adf59e640 1 -- 172.21.15.60:0/205686458 learned_addr learned my addr 172.21.15.60:0/205686458 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:31.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.180+0000 7f2aded9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2ae015aaa0 0x7f2ae00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47296/0 (socket says 172.21.15.60:47296) 2023-10-16T14:12:31.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.181+0000 7f2adf59e640 1 -- 172.21.15.60:0/205686458 >> v1:172.21.15.144:6789/0 conn(0x7f2ae015ae80 legacy=0x7f2ae0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:31.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.181+0000 7f2adf59e640 1 -- 172.21.15.60:0/205686458 >> v1:172.21.15.60:6789/0 conn(0x7f2ae015aaa0 legacy=0x7f2ae00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.181+0000 7f2adf59e640 1 -- 172.21.15.60:0/205686458 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae015ef90 con 0x7f2ae00ce390 2023-10-16T14:12:31.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.182+0000 7f2adf59e640 1 --2- 172.21.15.60:0/205686458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae015e820 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009870 tx=0x7f2ad802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61df83b5ab039f9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:31.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.182+0000 7f2ade59c640 1 -- 172.21.15.60:0/205686458 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8030860 con 0x7f2ae00ce390 2023-10-16T14:12:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.182+0000 7f2ade59c640 1 -- 172.21.15.60:0/205686458 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad80309c0 con 0x7f2ae00ce390 2023-10-16T14:12:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.182+0000 7f2ade59c640 1 -- 172.21.15.60:0/205686458 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8030d50 con 0x7f2ae00ce390 2023-10-16T14:12:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 -- 172.21.15.60:0/205686458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 msgr2=0x7f2ae015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 --2- 172.21.15.60:0/205686458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae015e820 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009870 tx=0x7f2ad802b9a0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 -- 172.21.15.60:0/205686458 shutdown_connections 2023-10-16T14:12:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 --2- 172.21.15.60:0/205686458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae015e820 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 -- 172.21.15.60:0/205686458 >> 172.21.15.60:0/205686458 conn(0x7f2ae00cb410 msgr2=0x7f2ae00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 -- 172.21.15.60:0/205686458 shutdown_connections 2023-10-16T14:12:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.183+0000 7f2ae502b640 1 -- 172.21.15.60:0/205686458 wait complete. 2023-10-16T14:12:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.185+0000 7f2ae502b640 1 Processor -- start 2023-10-16T14:12:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.185+0000 7f2ae502b640 1 -- start start 2023-10-16T14:12:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2ae502b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae00c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2ae502b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ae015aaa0 0x7f2ae015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2ae502b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae015ae80 0x7f2ae014deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2ae502b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae014e3f0 con 0x7f2ae00ce390 2023-10-16T14:12:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2ae502b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00c9ee0 con 0x7f2ae015ae80 2023-10-16T14:12:31.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2ae502b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00c92d0 con 0x7f2ae015aaa0 2023-10-16T14:12:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2aded9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae00c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2aded9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae00c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60168/0 (socket says 172.21.15.60:60168) 2023-10-16T14:12:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.186+0000 7f2aded9d640 1 -- 172.21.15.60:0/1840467638 learned_addr learned my addr 172.21.15.60:0/1840467638 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2adf59e640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae015ae80 0x7f2ae014deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2ad659c640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ae015aaa0 0x7f2ae015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2aded9d640 1 -- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ae015aaa0 msgr2=0x7f2ae015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2aded9d640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ae015aaa0 0x7f2ae015b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2aded9d640 1 -- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae015ae80 msgr2=0x7f2ae014deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2aded9d640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae015ae80 0x7f2ae014deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2aded9d640 1 -- 172.21.15.60:0/1840467638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad80094d0 con 0x7f2ae00ce390 2023-10-16T14:12:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2adf59e640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae015ae80 0x7f2ae014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.187+0000 7f2aded9d640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae00c6bf0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000c830 tx=0x7f2ad000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.188+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000a640 con 0x7f2ae00ce390 2023-10-16T14:12:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.188+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad000a7a0 con 0x7f2ae00ce390 2023-10-16T14:12:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.188+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae00c95b0 con 0x7f2ae00ce390 2023-10-16T14:12:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.188+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000ab10 con 0x7f2ae00ce390 2023-10-16T14:12:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.188+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae00c9b70 con 0x7f2ae00ce390 2023-10-16T14:12:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.189+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae004e160 con 0x7f2ae00ce390 2023-10-16T14:12:31.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.190+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f2ad000ac70 con 0x7f2ae00ce390 2023-10-16T14:12:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.191+0000 7f2ad7fff640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2ac006b020 0x7f2ac006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.191+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f2ad0089170 con 0x7f2ae00ce390 2023-10-16T14:12:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.192+0000 7f2ad659c640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2ac006b020 0x7f2ac006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.193+0000 7f2ad659c640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2ac006b020 0x7f2ac006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2ae015a500 tx=0x7f2ac8008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:31.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.195+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ad0053f80 con 0x7f2ae00ce390 2023-10-16T14:12:31.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.400+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ae00cdc60 con 0x7f2ae00ce390 2023-10-16T14:12:31.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.402+0000 7f2ad7fff640 1 -- 172.21.15.60:0/1840467638 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+14383 (secure 0 0 0) 0x7f2ad00018c0 con 0x7f2ae00ce390 2023-10-16T14:12:31.406 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:31.406 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":264,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:28.898943+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6836/521250241":"2023-10-17T14:12:25.400911+0000","172.21.15.144:6834/521250241":"2023-10-17T14:12:25.400911+0000","172.21.15.144:6837/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:6835/296787071":"2023-10-17T14:12:17.512189+0000","172.21.15.144:0/2609869736":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2877600084":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1518159871":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6832/529676808":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/3601797085":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3692487868":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1279398314":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2884978111":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/4230051502":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1376067390":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/2891523992":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/1029476750":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3834827725":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/1472951832":"2023-10-17T14:12:11.447458+0000","172.21.15.144:6833/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:6832/4052124783":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/2571613028":"2023-10-17T14:12:05.212706+0000","172.21.15.144:0/3892854062":"2023-10-17T14:12:11.447458+0000","172.21.15.144:0/752392530":"2023-10-17T14:12:05.212706+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:31.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2ac006b020 msgr2=0x7f2ac006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2ac006b020 0x7f2ac006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2ae015a500 tx=0x7f2ac8008040 comp rx=0 tx=0).stop 2023-10-16T14:12:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 msgr2=0x7f2ae00c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae00c6bf0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000c830 tx=0x7f2ad000ccf0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 shutdown_connections 2023-10-16T14:12:31.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2ac006b020 0x7f2ac006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ae015aaa0 0x7f2ae015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae015ae80 0x7f2ae014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 --2- 172.21.15.60:0/1840467638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ae00ce390 0x7f2ae00c6bf0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 >> 172.21.15.60:0/1840467638 conn(0x7f2ae00cb410 msgr2=0x7f2ae006ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 shutdown_connections 2023-10-16T14:12:31.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.405+0000 7f2ae502b640 1 -- 172.21.15.60:0/1840467638 wait complete. 2023-10-16T14:12:31.428 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:12:31.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:12:31.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.645+0000 7f3c8c74c640 1 Processor -- start 2023-10-16T14:12:31.650 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.645+0000 7f3c8c74c640 1 -- start start 2023-10-16T14:12:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c8c74c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3c84159ab0 0x7f3c840ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c8c74c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3c84159e90 0x7f3c8406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c8c74c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c8406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c8c74c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8406cb10 con 0x7f3c84159e90 2023-10-16T14:12:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c8c74c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8415d890 con 0x7f3c84159ab0 2023-10-16T14:12:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c8c74c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c840cd1f0 con 0x7f3c8406c140 2023-10-16T14:12:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c89ce7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3c84159e90 0x7f3c8406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c89ce7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3c84159e90 0x7f3c8406fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60178/0 (socket says 172.21.15.60:60178) 2023-10-16T14:12:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.647+0000 7f3c89ce7640 1 -- 172.21.15.60:0/4269042883 learned_addr learned my addr 172.21.15.60:0/4269042883 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:31.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c8a4e8640 1 --2- 172.21.15.60:0/4269042883 >> v2:172.21.15.144:3300/0 conn(0x7f3c84159ab0 0x7f3c840ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c8ace9640 1 --2- 172.21.15.60:0/4269042883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c8406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c89ce7640 1 -- 172.21.15.60:0/4269042883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 msgr2=0x7f3c8406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:31.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c89ce7640 1 --2- 172.21.15.60:0/4269042883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c8406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c89ce7640 1 -- 172.21.15.60:0/4269042883 >> v2:172.21.15.144:3300/0 conn(0x7f3c84159ab0 msgr2=0x7f3c840ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:31.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c89ce7640 1 --2- 172.21.15.60:0/4269042883 >> v2:172.21.15.144:3300/0 conn(0x7f3c84159ab0 0x7f3c840ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c89ce7640 1 -- 172.21.15.60:0/4269042883 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c840cd4d0 con 0x7f3c84159e90 2023-10-16T14:12:31.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.648+0000 7f3c8a4e8640 1 --2- 172.21.15.60:0/4269042883 >> v2:172.21.15.144:3300/0 conn(0x7f3c84159ab0 0x7f3c840ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:31.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.649+0000 7f3c89ce7640 1 --2- 172.21.15.60:0/4269042883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406fc50 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f3c840cfa00 tx=0x7f3c7402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd98e0ae096222e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:31.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.649+0000 7f3c894e6640 1 -- 172.21.15.60:0/4269042883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74031740 con 0x7f3c84159e90 2023-10-16T14:12:31.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.649+0000 7f3c894e6640 1 -- 172.21.15.60:0/4269042883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c740318a0 con 0x7f3c84159e90 2023-10-16T14:12:31.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.649+0000 7f3c894e6640 1 -- 172.21.15.60:0/4269042883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74031c30 con 0x7f3c84159e90 2023-10-16T14:12:31.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 -- 172.21.15.60:0/4269042883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 msgr2=0x7f3c8406fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/4269042883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406fc50 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f3c840cfa00 tx=0x7f3c7402b7e0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 -- 172.21.15.60:0/4269042883 shutdown_connections 2023-10-16T14:12:31.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/4269042883 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c8406c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/4269042883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406fc50 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/4269042883 >> v2:172.21.15.144:3300/0 conn(0x7f3c84159ab0 0x7f3c840ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 -- 172.21.15.60:0/4269042883 >> 172.21.15.60:0/4269042883 conn(0x7f3c840ce9b0 msgr2=0x7f3c840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:31.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 -- 172.21.15.60:0/4269042883 shutdown_connections 2023-10-16T14:12:31.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.650+0000 7f3c8c74c640 1 -- 172.21.15.60:0/4269042883 wait complete. 2023-10-16T14:12:31.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.652+0000 7f3c8c74c640 1 Processor -- start 2023-10-16T14:12:31.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.652+0000 7f3c8c74c640 1 -- start start 2023-10-16T14:12:31.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.653+0000 7f3c8c74c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c840c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.653+0000 7f3c8c74c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c84159ab0 0x7f3c84151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.653+0000 7f3c8c74c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.653+0000 7f3c8c74c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c840c68c0 con 0x7f3c84159e90 2023-10-16T14:12:31.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.653+0000 7f3c8c74c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8415ba50 con 0x7f3c84159ab0 2023-10-16T14:12:31.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.653+0000 7f3c8c74c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c84151e80 con 0x7f3c8406c140 2023-10-16T14:12:31.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60192/0 (socket says 172.21.15.60:60192) 2023-10-16T14:12:31.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 -- 172.21.15.60:0/1732616806 learned_addr learned my addr 172.21.15.60:0/1732616806 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:31.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8a4e8640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c840c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c89ce7640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c84159ab0 0x7f3c84151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 -- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 msgr2=0x7f3c840c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c840c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 -- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c84159ab0 msgr2=0x7f3c84151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c84159ab0 0x7f3c84151570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8ace9640 1 -- 172.21.15.60:0/1732616806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740094d0 con 0x7f3c84159e90 2023-10-16T14:12:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.654+0000 7f3c8a4e8640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c840c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:31.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.655+0000 7f3c8ace9640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406b2a0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c007d20 tx=0x7f3c7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:31.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.655+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c014420 con 0x7f3c84159e90 2023-10-16T14:12:31.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.655+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c0145a0 con 0x7f3c84159e90 2023-10-16T14:12:31.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.655+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c014950 con 0x7f3c84159e90 2023-10-16T14:12:31.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.655+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8415aca0 con 0x7f3c84159e90 2023-10-16T14:12:31.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.655+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8415b260 con 0x7f3c84159e90 2023-10-16T14:12:31.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.656+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8404e160 con 0x7f3c84159e90 2023-10-16T14:12:31.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.657+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f3c7c014ab0 con 0x7f3c84159e90 2023-10-16T14:12:31.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.658+0000 7f3c7affd640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f3c6806b050 0x7f3c6806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:31.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.658+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f3c7c08a1e0 con 0x7f3c84159e90 2023-10-16T14:12:31.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.659+0000 7f3c8a4e8640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f3c6806b050 0x7f3c6806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:31.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.660+0000 7f3c8a4e8640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f3c6806b050 0x7f3c6806d510 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c80006fd0 tx=0x7f3c80008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:31.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.663+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c7c054fe0 con 0x7f3c84159e90 2023-10-16T14:12:31.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.871+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3c840af8c0 con 0x7f3c84159e90 2023-10-16T14:12:31.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.872+0000 7f3c7affd640 1 -- 172.21.15.60:0/1732616806 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v264) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3c7c058600 con 0x7f3c84159e90 2023-10-16T14:12:31.876 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:12:31.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f3c6806b050 msgr2=0x7f3c6806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f3c6806b050 0x7f3c6806d510 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c80006fd0 tx=0x7f3c80008040 comp rx=0 tx=0).stop 2023-10-16T14:12:31.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 msgr2=0x7f3c8406b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:31.881 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406b2a0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c007d20 tx=0x7f3c7c00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 shutdown_connections 2023-10-16T14:12:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f3c6806b050 0x7f3c6806d510 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.882 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c84159e90 0x7f3c8406b2a0 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c84159ab0 0x7f3c84151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 --2- 172.21.15.60:0/1732616806 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3c8406c140 0x7f3c840c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 >> 172.21.15.60:0/1732616806 conn(0x7f3c840ce9b0 msgr2=0x7f3c84070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 shutdown_connections 2023-10-16T14:12:31.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:31.875+0000 7f3c8c74c640 1 -- 172.21.15.60:0/1732616806 wait complete. 2023-10-16T14:12:31.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T14:12:31.921 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:31.917+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.115+0000 7fdb71aa2640 1 Processor -- start 2023-10-16T14:12:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.115+0000 7fdb71aa2640 1 -- start start 2023-10-16T14:12:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.116+0000 7fdb71aa2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb6c15aaa0 0x7fdb6c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.116+0000 7fdb71aa2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb6c15ae80 0x7fdb6c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.116+0000 7fdb71aa2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c0ccaa0 0x7fdb6c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.116+0000 7fdb71aa2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c06b810 con 0x7fdb6c15aaa0 2023-10-16T14:12:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.116+0000 7fdb71aa2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c15ed60 con 0x7fdb6c15ae80 2023-10-16T14:12:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.116+0000 7fdb71aa2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0707e0 con 0x7fdb6c0ccaa0 2023-10-16T14:12:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb6c15aaa0 0x7fdb6c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb6c15aaa0 0x7fdb6c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60194/0 (socket says 172.21.15.60:60194) 2023-10-16T14:12:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 -- 172.21.15.60:0/3834394240 learned_addr learned my addr 172.21.15.60:0/3834394240 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 -- 172.21.15.60:0/3834394240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c0ccaa0 msgr2=0x7fdb6c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6bfff640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c0ccaa0 0x7fdb6c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c0ccaa0 0x7fdb6c070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 -- 172.21.15.60:0/3834394240 >> v2:172.21.15.144:3300/0 conn(0x7fdb6c15ae80 msgr2=0x7fdb6c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 --2- 172.21.15.60:0/3834394240 >> v2:172.21.15.144:3300/0 conn(0x7fdb6c15ae80 0x7fdb6c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6b7fe640 1 -- 172.21.15.60:0/3834394240 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c15ef90 con 0x7fdb6c15aaa0 2023-10-16T14:12:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.117+0000 7fdb6bfff640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c0ccaa0 0x7fdb6c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.119+0000 7fdb6b7fe640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15aaa0 0x7fdb6c15e820 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0c59c0 tx=0x7fdb5c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dfbdbf46c69164 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.119+0000 7fdb6affd640 1 -- 172.21.15.60:0/3834394240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c031880 con 0x7fdb6c15aaa0 2023-10-16T14:12:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.119+0000 7fdb6affd640 1 -- 172.21.15.60:0/3834394240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c0319e0 con 0x7fdb6c15aaa0 2023-10-16T14:12:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.119+0000 7fdb6affd640 1 -- 172.21.15.60:0/3834394240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c031d70 con 0x7fdb6c15aaa0 2023-10-16T14:12:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 -- 172.21.15.60:0/3834394240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15aaa0 msgr2=0x7fdb6c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15aaa0 0x7fdb6c15e820 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0c59c0 tx=0x7fdb5c02b920 comp rx=0 tx=0).stop 2023-10-16T14:12:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 -- 172.21.15.60:0/3834394240 shutdown_connections 2023-10-16T14:12:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c0ccaa0 0x7fdb6c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/3834394240 >> v2:172.21.15.144:3300/0 conn(0x7fdb6c15ae80 0x7fdb6c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/3834394240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15aaa0 0x7fdb6c15e820 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 -- 172.21.15.60:0/3834394240 >> 172.21.15.60:0/3834394240 conn(0x7fdb6c0cb410 msgr2=0x7fdb6c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 -- 172.21.15.60:0/3834394240 shutdown_connections 2023-10-16T14:12:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.120+0000 7fdb71aa2640 1 -- 172.21.15.60:0/3834394240 wait complete. 2023-10-16T14:12:32.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.122+0000 7fdb71aa2640 1 Processor -- start 2023-10-16T14:12:32.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.122+0000 7fdb71aa2640 1 -- start start 2023-10-16T14:12:32.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb71aa2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb6c0ccaa0 0x7fdb6c1592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb71aa2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c15aaa0 0x7fdb6c1e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb71aa2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 0x7fdb6c1e5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb71aa2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c1e6980 con 0x7fdb6c15ae80 2023-10-16T14:12:32.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb71aa2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c159dd0 con 0x7fdb6c0ccaa0 2023-10-16T14:12:32.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb71aa2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c159830 con 0x7fdb6c15aaa0 2023-10-16T14:12:32.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 0x7fdb6c1e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:32.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb6bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 0x7fdb6c1e5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60196/0 (socket says 172.21.15.60:60196) 2023-10-16T14:12:32.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb6bfff640 1 -- 172.21.15.60:0/864202414 learned_addr learned my addr 172.21.15.60:0/864202414 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:32.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.123+0000 7fdb6b7fe640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb6c0ccaa0 0x7fdb6c1592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:32.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb6bfff640 1 -- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c15aaa0 msgr2=0x7fdb6c1e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:32.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb6bfff640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c15aaa0 0x7fdb6c1e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb6bfff640 1 -- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb6c0ccaa0 msgr2=0x7fdb6c1592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:32.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb6bfff640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb6c0ccaa0 0x7fdb6c1592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb6bfff640 1 -- 172.21.15.60:0/864202414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0094d0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb6bfff640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 0x7fdb6c1e5680 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fdb6400cd50 tx=0x7fdb6400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400f680 con 0x7fdb6c15ae80 2023-10-16T14:12:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6400f800 con 0x7fdb6c15ae80 2023-10-16T14:12:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400fbb0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c1e5d00 con 0x7fdb6c15ae80 2023-10-16T14:12:32.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.124+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c1e62c0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.126+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c1e6ac0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.127+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fdb6400b040 con 0x7fdb6c15ae80 2023-10-16T14:12:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.128+0000 7fdb68ff9640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdb4c06b020 0x7fdb4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.128+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7fdb6408a470 con 0x7fdb6c15ae80 2023-10-16T14:12:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.128+0000 7fdb6b7fe640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdb4c06b020 0x7fdb4c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.129+0000 7fdb6b7fe640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdb4c06b020 0x7fdb4c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c036000 tx=0x7fdb5c03c040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.132+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb6c1e6ac0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.404+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fdb6c1e6ac0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.934+0000 7fdb68ff9640 1 -- 172.21.15.60:0/864202414 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3289) v1 ==== 158+0+0 (secure 0 0 0) 0x7fdb6c1e6ac0 con 0x7fdb6c15ae80 2023-10-16T14:12:32.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdb4c06b020 msgr2=0x7fdb4c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:32.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdb4c06b020 0x7fdb4c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c036000 tx=0x7fdb5c03c040 comp rx=0 tx=0).stop 2023-10-16T14:12:32.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 msgr2=0x7fdb6c1e5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:32.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 0x7fdb6c1e5680 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fdb6400cd50 tx=0x7fdb6400a650 comp rx=0 tx=0).stop 2023-10-16T14:12:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 shutdown_connections 2023-10-16T14:12:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdb4c06b020 0x7fdb4c06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdb6c15aaa0 0x7fdb6c1e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb6c15ae80 0x7fdb6c1e5680 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 --2- 172.21.15.60:0/864202414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb6c0ccaa0 0x7fdb6c1592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 >> 172.21.15.60:0/864202414 conn(0x7fdb6c0cb410 msgr2=0x7fdb6c0ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 shutdown_connections 2023-10-16T14:12:32.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:32.938+0000 7fdb71aa2640 1 -- 172.21.15.60:0/864202414 wait complete. 2023-10-16T14:12:32.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:12:33.003 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:32.999+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:33.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.174+0000 7fab56341640 1 Processor -- start 2023-10-16T14:12:33.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.174+0000 7fab56341640 1 -- start start 2023-10-16T14:12:33.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.175+0000 7fab56341640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab501618b0 0x7fab50065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:33.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.175+0000 7fab56341640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab50065ff0 0x7fab500ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:33.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.175+0000 7fab56341640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab50066420 con 0x7fab5015d330 2023-10-16T14:12:33.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.175+0000 7fab56341640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5006b1e0 con 0x7fab50065ff0 2023-10-16T14:12:33.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.175+0000 7fab56341640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5006cee0 con 0x7fab501618b0 2023-10-16T14:12:33.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab50065ff0 0x7fab500ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:33.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab50065ff0 0x7fab500ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60202/0 (socket says 172.21.15.60:60202) 2023-10-16T14:12:33.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 -- 172.21.15.60:0/3610739391 learned_addr learned my addr 172.21.15.60:0/3610739391 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:33.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab4f7fe640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab501618b0 0x7fab50065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:33.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 -- 172.21.15.60:0/3610739391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab501618b0 msgr2=0x7fab50065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:33.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab501618b0 0x7fab50065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 -- 172.21.15.60:0/3610739391 >> v1:172.21.15.144:6789/0 conn(0x7fab5015d330 legacy=0x7fab5006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:33.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab548de640 1 -- 172.21.15.60:0/3610739391 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500d0890 con 0x7fab50065ff0 2023-10-16T14:12:33.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.176+0000 7fab4f7fe640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab501618b0 0x7fab50065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:33.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.177+0000 7fab548de640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500ce9b0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fab48009870 tx=0x7fab4802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7146c25c73eb40ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:33.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.178+0000 7fab4effd640 1 -- 172.21.15.60:0/3610739391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab480317c0 con 0x7fab50065ff0 2023-10-16T14:12:33.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.178+0000 7fab4effd640 1 -- 172.21.15.60:0/3610739391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48031920 con 0x7fab50065ff0 2023-10-16T14:12:33.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.178+0000 7fab4effd640 1 -- 172.21.15.60:0/3610739391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48031cb0 con 0x7fab50065ff0 2023-10-16T14:12:33.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 -- 172.21.15.60:0/3610739391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 msgr2=0x7fab500ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:33.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500ce9b0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fab48009870 tx=0x7fab4802b860 comp rx=0 tx=0).stop 2023-10-16T14:12:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 -- 172.21.15.60:0/3610739391 shutdown_connections 2023-10-16T14:12:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500ce9b0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 --2- 172.21.15.60:0/3610739391 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab501618b0 0x7fab50065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 -- 172.21.15.60:0/3610739391 >> 172.21.15.60:0/3610739391 conn(0x7fab500d0ab0 msgr2=0x7fab500d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 -- 172.21.15.60:0/3610739391 shutdown_connections 2023-10-16T14:12:33.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.179+0000 7fab56341640 1 -- 172.21.15.60:0/3610739391 wait complete. 2023-10-16T14:12:33.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.181+0000 7fab56341640 1 Processor -- start 2023-10-16T14:12:33.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.181+0000 7fab56341640 1 -- start start 2023-10-16T14:12:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab56341640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab56341640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab5015d330 0x7fab5015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab56341640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab501618b0 0x7fab500c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab56341640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab500b0870 con 0x7fab50065ff0 2023-10-16T14:12:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab56341640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab500b0b50 con 0x7fab501618b0 2023-10-16T14:12:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab56341640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab500b0e30 con 0x7fab5015d330 2023-10-16T14:12:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60204/0 (socket says 172.21.15.60:60204) 2023-10-16T14:12:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 -- 172.21.15.60:0/3933307841 learned_addr learned my addr 172.21.15.60:0/3933307841 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab548de640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab501618b0 0x7fab500c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4f7fe640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab5015d330 0x7fab5015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 -- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab5015d330 msgr2=0x7fab5015dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab5015d330 0x7fab5015dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 -- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab501618b0 msgr2=0x7fab500c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab501618b0 0x7fab500c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab4ffff640 1 -- 172.21.15.60:0/3933307841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480094d0 con 0x7fab50065ff0 2023-10-16T14:12:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.182+0000 7fab548de640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab501618b0 0x7fab500c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.183+0000 7fab4ffff640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500b3080 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fab4000ba50 tx=0x7fab4000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.183+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000cbe0 con 0x7fab50065ff0 2023-10-16T14:12:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.183+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000cd80 con 0x7fab50065ff0 2023-10-16T14:12:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.183+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40014680 con 0x7fab50065ff0 2023-10-16T14:12:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.183+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab500b1110 con 0x7fab50065ff0 2023-10-16T14:12:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.183+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5015e160 con 0x7fab50065ff0 2023-10-16T14:12:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.185+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5015a770 con 0x7fab50065ff0 2023-10-16T14:12:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.185+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fab400147e0 con 0x7fab50065ff0 2023-10-16T14:12:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.187+0000 7fab4cff9640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fab3c06b020 0x7fab3c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.187+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7fab40089860 con 0x7fab50065ff0 2023-10-16T14:12:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.188+0000 7fab4f7fe640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fab3c06b020 0x7fab3c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.189+0000 7fab4f7fe640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fab3c06b020 0x7fab3c06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab501592c0 tx=0x7fab44008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:33.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.192+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab40054670 con 0x7fab50065ff0 2023-10-16T14:12:33.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.388+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fab5004e160 con 0x7fab50065ff0 2023-10-16T14:12:33.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.393+0000 7fab4cff9640 1 -- 172.21.15.60:0/3933307841 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7fab40057c90 con 0x7fab50065ff0 2023-10-16T14:12:33.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fab3c06b020 msgr2=0x7fab3c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:33.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fab3c06b020 0x7fab3c06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab501592c0 tx=0x7fab44008040 comp rx=0 tx=0).stop 2023-10-16T14:12:33.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 msgr2=0x7fab500b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:33.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500b3080 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fab4000ba50 tx=0x7fab4000bf10 comp rx=0 tx=0).stop 2023-10-16T14:12:33.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 shutdown_connections 2023-10-16T14:12:33.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fab3c06b020 0x7fab3c06d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab5015d330 0x7fab5015dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab501618b0 0x7fab500c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 --2- 172.21.15.60:0/3933307841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab50065ff0 0x7fab500b3080 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:33.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 >> 172.21.15.60:0/3933307841 conn(0x7fab500d0ab0 msgr2=0x7fab500cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:33.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 shutdown_connections 2023-10-16T14:12:33.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:33.396+0000 7fab56341640 1 -- 172.21.15.60:0/3933307841 wait complete. 2023-10-16T14:12:33.417 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:12:33.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:12:33.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:12:33.689 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:12:33.689 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:12:33.690 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:33.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:12:33.753 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:12:33.754 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:33.814 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:12:33.814 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:12:33.815 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:33.876 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:12:33.876 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:12:33.876 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:33.936 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:12:33.937 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:12:33.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:34.040 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:34.036+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:34.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.178+0000 7fa684e68640 1 Processor -- start 2023-10-16T14:12:34.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.178+0000 7fa684e68640 1 -- start start 2023-10-16T14:12:34.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.179+0000 7fa684e68640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa68015aaa0 0x7fa68015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.179+0000 7fa684e68640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa68015ae80 0x7fa6800cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa684e68640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68015ed60 con 0x7fa6800ccaa0 2023-10-16T14:12:34.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa684e68640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68006b810 con 0x7fa68015ae80 2023-10-16T14:12:34.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa684e68640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6800707e0 con 0x7fa68015aaa0 2023-10-16T14:12:34.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa677fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa68015ae80 0x7fa6800cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa677fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa68015ae80 0x7fa6800cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60220/0 (socket says 172.21.15.60:60220) 2023-10-16T14:12:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa677fff640 1 -- 172.21.15.60:0/3934656799 learned_addr learned my addr 172.21.15.60:0/3934656799 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa67e59c640 1 --2- 172.21.15.60:0/3934656799 >> v2:172.21.15.144:3300/0 conn(0x7fa68015aaa0 0x7fa68015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa677fff640 1 -- 172.21.15.60:0/3934656799 >> v2:172.21.15.144:3300/0 conn(0x7fa68015aaa0 msgr2=0x7fa68015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.180+0000 7fa677fff640 1 --2- 172.21.15.60:0/3934656799 >> v2:172.21.15.144:3300/0 conn(0x7fa68015aaa0 0x7fa68015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.181+0000 7fa677fff640 1 -- 172.21.15.60:0/3934656799 >> v1:172.21.15.144:6789/0 conn(0x7fa6800ccaa0 legacy=0x7fa680070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.181+0000 7fa677fff640 1 -- 172.21.15.60:0/3934656799 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68015ef90 con 0x7fa68015ae80 2023-10-16T14:12:34.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.182+0000 7fa677fff640 1 --2- 172.21.15.60:0/3934656799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa68015ae80 0x7fa6800cc560 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7fa6800cd990 tx=0x7fa66802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3dbcd3817905394 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.183+0000 7fa67dd9b640 1 -- 172.21.15.60:0/3934656799 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668031870 con 0x7fa68015ae80 2023-10-16T14:12:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.183+0000 7fa67dd9b640 1 -- 172.21.15.60:0/3934656799 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6680319d0 con 0x7fa68015ae80 2023-10-16T14:12:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.183+0000 7fa67dd9b640 1 -- 172.21.15.60:0/3934656799 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668031d60 con 0x7fa68015ae80 2023-10-16T14:12:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.183+0000 7fa684e68640 1 -- 172.21.15.60:0/3934656799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa68015ae80 msgr2=0x7fa6800cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.183+0000 7fa684e68640 1 --2- 172.21.15.60:0/3934656799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa68015ae80 0x7fa6800cc560 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7fa6800cd990 tx=0x7fa66802b910 comp rx=0 tx=0).stop 2023-10-16T14:12:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.184+0000 7fa684e68640 1 -- 172.21.15.60:0/3934656799 shutdown_connections 2023-10-16T14:12:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.184+0000 7fa684e68640 1 --2- 172.21.15.60:0/3934656799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa68015ae80 0x7fa6800cc560 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.184+0000 7fa684e68640 1 --2- 172.21.15.60:0/3934656799 >> v2:172.21.15.144:3300/0 conn(0x7fa68015aaa0 0x7fa68015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.184+0000 7fa684e68640 1 -- 172.21.15.60:0/3934656799 >> 172.21.15.60:0/3934656799 conn(0x7fa6800cb410 msgr2=0x7fa6800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.184+0000 7fa684e68640 1 -- 172.21.15.60:0/3934656799 shutdown_connections 2023-10-16T14:12:34.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.184+0000 7fa684e68640 1 -- 172.21.15.60:0/3934656799 wait complete. 2023-10-16T14:12:34.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.186+0000 7fa684e68640 1 Processor -- start 2023-10-16T14:12:34.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.186+0000 7fa684e68640 1 -- start start 2023-10-16T14:12:34.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa684e68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 0x7fa6801f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa684e68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa68015aaa0 0x7fa6800c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa684e68640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa68015ae80 0x7fa6800ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa684e68640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c6630 con 0x7fa6800ccaa0 2023-10-16T14:12:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa684e68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c6910 con 0x7fa68015aaa0 2023-10-16T14:12:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa684e68640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c6bf0 con 0x7fa68015ae80 2023-10-16T14:12:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 0x7fa6801f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 0x7fa6801f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60224/0 (socket says 172.21.15.60:60224) 2023-10-16T14:12:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 -- 172.21.15.60:0/539318400 learned_addr learned my addr 172.21.15.60:0/539318400 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa677fff640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa68015aaa0 0x7fa6800c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 -- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa68015ae80 msgr2=0x7fa6800ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa68015ae80 0x7fa6800ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 -- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa68015aaa0 msgr2=0x7fa6800c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa68015aaa0 0x7fa6800c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.187+0000 7fa67e59c640 1 -- 172.21.15.60:0/539318400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680094d0 con 0x7fa6800ccaa0 2023-10-16T14:12:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.188+0000 7fa67e59c640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 0x7fa6801f2840 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fa67000d800 tx=0x7fa67000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.188+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67000b6c0 con 0x7fa6800ccaa0 2023-10-16T14:12:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.188+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67000b820 con 0x7fa6800ccaa0 2023-10-16T14:12:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.188+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67000bbb0 con 0x7fa6800ccaa0 2023-10-16T14:12:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.188+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6800c6ed0 con 0x7fa6800ccaa0 2023-10-16T14:12:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.188+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa680156250 con 0x7fa6800ccaa0 2023-10-16T14:12:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.189+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6800af820 con 0x7fa6800ccaa0 2023-10-16T14:12:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.190+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa67000bd10 con 0x7fa6800ccaa0 2023-10-16T14:12:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.192+0000 7fa676ffd640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa66006b020 0x7fa66006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.192+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7fa67008a030 con 0x7fa6800ccaa0 2023-10-16T14:12:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.192+0000 7fa677fff640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa66006b020 0x7fa66006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.194+0000 7fa677fff640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa66006b020 0x7fa66006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa668036000 tx=0x7fa66803c040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:34.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.196+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6700bcfe0 con 0x7fa6800ccaa0 2023-10-16T14:12:34.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.444+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6800c70c0 con 0x7fa6800ccaa0 2023-10-16T14:12:34.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.444+0000 7fa676ffd640 1 -- 172.21.15.60:0/539318400 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+3704 (secure 0 0 0) 0x7fa6800af820 con 0x7fa6800ccaa0 2023-10-16T14:12:34.448 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:34.449 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":138,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1356422846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1356422846},{"type":"v1","addr":"172.21.15.60:6834","nonce":1356422846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8183,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2210505043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2210505043},{"type":"v1","addr":"172.21.15.60:6835","nonce":2210505043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":9769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/40626494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":40626494},{"type":"v1","addr":"172.21.15.144:6837","nonce":40626494}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":9820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1333098019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1333098019},{"type":"v1","addr":"172.21.15.144:6836","nonce":1333098019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":138}],"filesystems":[]} 2023-10-16T14:12:34.449 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 138 2023-10-16T14:12:34.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.447+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa66006b020 msgr2=0x7fa66006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.447+0000 7fa684e68640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa66006b020 0x7fa66006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa668036000 tx=0x7fa66803c040 comp rx=0 tx=0).stop 2023-10-16T14:12:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.447+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 msgr2=0x7fa6801f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.447+0000 7fa684e68640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 0x7fa6801f2840 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fa67000d800 tx=0x7fa67000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 shutdown_connections 2023-10-16T14:12:34.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa66006b020 0x7fa66006d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa68015ae80 0x7fa6800ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa68015aaa0 0x7fa6800c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 --2- 172.21.15.60:0/539318400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6800ccaa0 0x7fa6801f2840 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 >> 172.21.15.60:0/539318400 conn(0x7fa6800cb410 msgr2=0x7fa6800b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:34.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 shutdown_connections 2023-10-16T14:12:34.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.448+0000 7fa684e68640 1 -- 172.21.15.60:0/539318400 wait complete. 2023-10-16T14:12:34.473 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:12:34.473 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:12:34.473 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:12:34.473 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:12:34.474 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:12:34.474 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:12:34.474 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:12:34.475 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:12:34.475 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:12:34.475 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:12:34.475 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:12:34.476 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:12:34.476 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:12:34.476 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:12:34.476 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:12:34.477 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:12:34.493 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:12:34.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:12:34.495 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:12:34.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:12:34.496 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:12:34.497 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:12:34.500 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:12:34.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:12:34.543 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:12:34.543 DEBUG:teuthology.parallel:result is None 2023-10-16T14:12:34.549 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:12:34.550 DEBUG:teuthology.parallel:result is None 2023-10-16T14:12:34.551 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:12:34.551 DEBUG:teuthology.parallel:result is None 2023-10-16T14:12:34.557 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:12:34.557 DEBUG:teuthology.parallel:result is None 2023-10-16T14:12:34.558 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:12:34.716 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:12:34.717 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:12:34.718 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:12:34.718 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T14:12:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.759+0000 7f93d5459640 1 Processor -- start 2023-10-16T14:12:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.760+0000 7f93d5459640 1 -- start start 2023-10-16T14:12:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93d5459640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93d5459640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93d00ad970 0x7f93d015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93d5459640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d006c960 con 0x7f93d015ebc0 2023-10-16T14:12:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93d5459640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d015b9d0 con 0x7f93d00ad970 2023-10-16T14:12:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93d5459640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d015bcb0 con 0x7f93d006c040 2023-10-16T14:12:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93c7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93d00ad970 0x7f93d015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93cffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93cffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:34494/0 (socket says 172.21.15.60:34494) 2023-10-16T14:12:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93c7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93d00ad970 0x7f93d015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60272/0 (socket says 172.21.15.60:60272) 2023-10-16T14:12:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.761+0000 7f93cffff640 1 -- 172.21.15.60:0/1010436081 learned_addr learned my addr 172.21.15.60:0/1010436081 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.762+0000 7f93c7fff640 1 -- 172.21.15.60:0/1010436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 msgr2=0x7f93d006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.762+0000 7f93c7fff640 1 --2- 172.21.15.60:0/1010436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.762+0000 7f93c7fff640 1 -- 172.21.15.60:0/1010436081 >> v1:172.21.15.144:6789/0 conn(0x7f93d015ebc0 legacy=0x7f93d00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.762+0000 7f93c7fff640 1 -- 172.21.15.60:0/1010436081 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d015bf90 con 0x7f93d00ad970 2023-10-16T14:12:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.762+0000 7f93cffff640 1 --2- 172.21.15.60:0/1010436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.763+0000 7f93c7fff640 1 --2- 172.21.15.60:0/1010436081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d00ad970 0x7f93d015e680 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f93b80097d0 tx=0x7f93b802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79f3d3d9061fc69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.763+0000 7f93cf7fe640 1 -- 172.21.15.60:0/1010436081 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b802d040 con 0x7f93d00ad970 2023-10-16T14:12:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.763+0000 7f93cf7fe640 1 -- 172.21.15.60:0/1010436081 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b8031810 con 0x7f93d00ad970 2023-10-16T14:12:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93cf7fe640 1 -- 172.21.15.60:0/1010436081 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b8031ba0 con 0x7f93d00ad970 2023-10-16T14:12:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 -- 172.21.15.60:0/1010436081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d00ad970 msgr2=0x7f93d015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 --2- 172.21.15.60:0/1010436081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d00ad970 0x7f93d015e680 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f93b80097d0 tx=0x7f93b802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 -- 172.21.15.60:0/1010436081 shutdown_connections 2023-10-16T14:12:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 --2- 172.21.15.60:0/1010436081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d00ad970 0x7f93d015e680 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 --2- 172.21.15.60:0/1010436081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 -- 172.21.15.60:0/1010436081 >> 172.21.15.60:0/1010436081 conn(0x7f93d00ce6a0 msgr2=0x7f93d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 -- 172.21.15.60:0/1010436081 shutdown_connections 2023-10-16T14:12:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.764+0000 7f93d5459640 1 -- 172.21.15.60:0/1010436081 wait complete. 2023-10-16T14:12:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.766+0000 7f93d5459640 1 Processor -- start 2023-10-16T14:12:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.766+0000 7f93d5459640 1 -- start start 2023-10-16T14:12:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d5459640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d00aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d5459640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93d00ad970 0x7f93d00b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d5459640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 0x7f93d00ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d5459640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01f25a0 con 0x7f93d015ebc0 2023-10-16T14:12:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d5459640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d00af510 con 0x7f93d00ad970 2023-10-16T14:12:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d5459640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01f16d0 con 0x7f93d006c040 2023-10-16T14:12:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d4c58640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 0x7f93d00ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d4c58640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 0x7f93d00ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60284/0 (socket says 172.21.15.60:60284) 2023-10-16T14:12:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.767+0000 7f93d4c58640 1 -- 172.21.15.60:0/2068063663 learned_addr learned my addr 172.21.15.60:0/2068063663 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93cffff640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d00aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93c7fff640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93d00ad970 0x7f93d00b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93d4c58640 1 -- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 msgr2=0x7f93d00aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93d4c58640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d00aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93d4c58640 1 -- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93d00ad970 msgr2=0x7f93d00b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93d4c58640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93d00ad970 0x7f93d00b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93d4c58640 1 -- 172.21.15.60:0/2068063663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c00094d0 con 0x7f93d015ebc0 2023-10-16T14:12:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.768+0000 7f93d4c58640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 0x7f93d00ca5a0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c7d0 tx=0x7f93c800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.769+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8003b70 con 0x7f93d015ebc0 2023-10-16T14:12:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.769+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8003cf0 con 0x7f93d015ebc0 2023-10-16T14:12:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.769+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b8009400 con 0x7f93d015ebc0 2023-10-16T14:12:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.769+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c800aa90 con 0x7f93d015ebc0 2023-10-16T14:12:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.769+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d01f1c70 con 0x7f93d015ebc0 2023-10-16T14:12:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.770+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d006b330 con 0x7f93d015ebc0 2023-10-16T14:12:34.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.771+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f93c800abf0 con 0x7f93d015ebc0 2023-10-16T14:12:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.772+0000 7f93cd7fa640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f93b006b020 0x7f93b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.772+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f93c8089820 con 0x7f93d015ebc0 2023-10-16T14:12:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.773+0000 7f93cffff640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f93b006b020 0x7f93b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.774+0000 7f93cffff640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f93b006b020 0x7f93b006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93d00b2870 tx=0x7f93c0009240 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:34.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.776+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93c80544f0 con 0x7f93d015ebc0 2023-10-16T14:12:34.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:34.995+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f93d00cc250 con 0x7f93d015ebc0 2023-10-16T14:12:35.036 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:35.032+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:35.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.418+0000 7f93cd7fa640 1 -- 172.21.15.60:0/2068063663 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v266) v1 ==== 97+0+0 (secure 0 0 0) 0x7f93d00cc250 con 0x7f93d015ebc0 2023-10-16T14:12:35.423 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T14:12:35.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f93b006b020 msgr2=0x7f93b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f93b006b020 0x7f93b006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93d00b2870 tx=0x7f93c0009240 comp rx=0 tx=0).stop 2023-10-16T14:12:35.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 msgr2=0x7f93d00ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 0x7f93d00ca5a0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c7d0 tx=0x7f93c800cc90 comp rx=0 tx=0).stop 2023-10-16T14:12:35.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 shutdown_connections 2023-10-16T14:12:35.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f93b006b020 0x7f93b006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d015ebc0 0x7f93d00ca5a0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f93d00ad970 0x7f93d00b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 --2- 172.21.15.60:0/2068063663 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f93d006c040 0x7f93d00aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 >> 172.21.15.60:0/2068063663 conn(0x7f93d00ce6a0 msgr2=0x7f93d015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:35.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 shutdown_connections 2023-10-16T14:12:35.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.423+0000 7f93d5459640 1 -- 172.21.15.60:0/2068063663 wait complete. 2023-10-16T14:12:35.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:12:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.673+0000 7f62bffff640 1 Processor -- start 2023-10-16T14:12:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.674+0000 7f62bffff640 1 -- start start 2023-10-16T14:12:35.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.675+0000 7f62bffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 0x7f62c00cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.675+0000 7f62bffff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00b3730 con 0x7f62c015aa60 2023-10-16T14:12:35.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.675+0000 7f62bffff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00b3980 con 0x7f62c00b4740 2023-10-16T14:12:35.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.675+0000 7f62bffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00b3c60 con 0x7f62c00cc2e0 2023-10-16T14:12:35.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.676+0000 7f62beffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f62c015aa60 0x7f62c00af820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47316/0 (socket says 172.21.15.60:47316) 2023-10-16T14:12:35.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.676+0000 7f62beffd640 1 -- 172.21.15.60:0/1084195213 learned_addr learned my addr 172.21.15.60:0/1084195213 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.676+0000 7f62b7fff640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 0x7f62c00cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.676+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727285356 0 0) 0x7f62c00b3730 con 0x7f62c015aa60 2023-10-16T14:12:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.676+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ac003520 con 0x7f62c015aa60 2023-10-16T14:12:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2945957628 0 0) 0x7f62ac003520 con 0x7f62c015aa60 2023-10-16T14:12:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 msgr2=0x7f62c00cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 0x7f62c00cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 >> v1:172.21.15.144:6789/0 conn(0x7f62c00b4740 legacy=0x7f62c00b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00b3f40 con 0x7f62c015aa60 2023-10-16T14:12:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f62b0001670 con 0x7f62c015aa60 2023-10-16T14:12:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 >> v1:172.21.15.60:6789/0 conn(0x7f62c015aa60 legacy=0x7f62c00af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 0x7f62ac0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac003f70 0x7f62ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac004d00 0x7f62ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00b3c60 con 0x7f62ac004d00 2023-10-16T14:12:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac00bac0 con 0x7f62ac003f70 2023-10-16T14:12:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac00bc80 con 0x7f62c00cc2e0 2023-10-16T14:12:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62bf7fe640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac004d00 0x7f62ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.677+0000 7f62beffd640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 0x7f62ac0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62b7fff640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac003f70 0x7f62ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62bf7fe640 1 -- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 msgr2=0x7f62ac0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62bf7fe640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62c00cc2e0 0x7f62ac0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62bf7fe640 1 -- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac003f70 msgr2=0x7f62ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62bf7fe640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac003f70 0x7f62ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62bf7fe640 1 -- 172.21.15.60:0/1084195213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac003520 con 0x7f62ac004d00 2023-10-16T14:12:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.678+0000 7f62b7fff640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62ac003f70 0x7f62ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.679+0000 7f62bf7fe640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac004d00 0x7f62ac005180 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f62b8001e50 tx=0x7f62ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387d6e0a0391aa22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.679+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8003b70 con 0x7f62ac004d00 2023-10-16T14:12:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.679+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b8003cd0 con 0x7f62ac004d00 2023-10-16T14:12:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.679+0000 7f62be7fc640 1 -- 172.21.15.60:0/1084195213 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8005310 con 0x7f62ac004d00 2023-10-16T14:12:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.679+0000 7f62bffff640 1 -- 172.21.15.60:0/1084195213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac004d00 msgr2=0x7f62ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.679+0000 7f62bffff640 1 --2- 172.21.15.60:0/1084195213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ac004d00 0x7f62ac005180 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f62b8001e50 tx=0x7f62ac0057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.680+0000 7f62bf7fe640 1 -- 172.21.15.60:0/1084195213 reap_dead start 2023-10-16T14:12:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.680+0000 7f62bffff640 1 -- 172.21.15.60:0/1084195213 shutdown_connections 2023-10-16T14:12:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.680+0000 7f62bffff640 1 -- 172.21.15.60:0/1084195213 >> 172.21.15.60:0/1084195213 conn(0x7f62c0065ed0 msgr2=0x7f62c00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.680+0000 7f62bffff640 1 -- 172.21.15.60:0/1084195213 shutdown_connections 2023-10-16T14:12:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.680+0000 7f62bffff640 1 -- 172.21.15.60:0/1084195213 wait complete. 2023-10-16T14:12:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.682+0000 7f62bffff640 1 Processor -- start 2023-10-16T14:12:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.682+0000 7f62bffff640 1 -- start start 2023-10-16T14:12:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62bffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac004d00 0x7f62c0146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 0x7f62c00cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62bffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62c00c5460 0x7f62c00c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62bffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00cf550 con 0x7f62c00ceb90 2023-10-16T14:12:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62bffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00c5e20 con 0x7f62c00c5460 2023-10-16T14:12:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62bffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00c7000 con 0x7f62ac004d00 2023-10-16T14:12:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 0x7f62c00cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 0x7f62c00cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60308/0 (socket says 172.21.15.60:60308) 2023-10-16T14:12:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62b7fff640 1 -- 172.21.15.60:0/1423144227 learned_addr learned my addr 172.21.15.60:0/1423144227 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.683+0000 7f62beffd640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac004d00 0x7f62c0146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62bf7fe640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62c00c5460 0x7f62c00c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b7fff640 1 -- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac004d00 msgr2=0x7f62c0146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b7fff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac004d00 0x7f62c0146b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b7fff640 1 -- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62c00c5460 msgr2=0x7f62c00c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b7fff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62c00c5460 0x7f62c00c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b7fff640 1 -- 172.21.15.60:0/1423144227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00c6040 con 0x7f62c00ceb90 2023-10-16T14:12:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b7fff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 0x7f62c00cf010 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f62a8010910 tx=0x7f62a8010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a800ec20 con 0x7f62c00ceb90 2023-10-16T14:12:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a800ed80 con 0x7f62c00ceb90 2023-10-16T14:12:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c00c6320 con 0x7f62c00ceb90 2023-10-16T14:12:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.684+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a801c600 con 0x7f62c00ceb90 2023-10-16T14:12:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.685+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c00b18a0 con 0x7f62c00ceb90 2023-10-16T14:12:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.686+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c006aa20 con 0x7f62c00ceb90 2023-10-16T14:12:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.687+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f62a801c760 con 0x7f62c00ceb90 2023-10-16T14:12:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.687+0000 7f62b77fe640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f62a006b020 0x7f62a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.688+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f62a8089bd0 con 0x7f62c00ceb90 2023-10-16T14:12:35.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.688+0000 7f62beffd640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f62a006b020 0x7f62a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:35.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.689+0000 7f62beffd640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f62a006b020 0x7f62a006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62b0001050 tx=0x7f62b0009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:35.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.692+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f62a8054d40 con 0x7f62c00ceb90 2023-10-16T14:12:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.967+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f62c00b1420 con 0x7f62c00ceb90 2023-10-16T14:12:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.968+0000 7f62b77fe640 1 -- 172.21.15.60:0/1423144227 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v58) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f62c00b1420 con 0x7f62c00ceb90 2023-10-16T14:12:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:35.972 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:12:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T14:12:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T14:12:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T14:12:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:12:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:12:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T14:12:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T14:12:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:12:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:12:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T14:12:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T14:12:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:12:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:12:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T14:12:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T14:12:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:12:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:12:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T14:12:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T14:12:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T14:12:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T14:12:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T14:12:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T14:12:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T14:12:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T14:12:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T14:12:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T14:12:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T14:12:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T14:12:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T14:12:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T14:12:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T14:12:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T14:12:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:12:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:12:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T14:12:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T14:12:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T14:12:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:12:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T14:12:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T14:12:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:12:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:12:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:12:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T14:12:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T14:12:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:12:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:12:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:12:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T14:12:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T14:12:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:12:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:12:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:12:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:12:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T14:12:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T14:12:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:12:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:12:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:12:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T14:12:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T14:12:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:12:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T14:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T14:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T14:12:36.007 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:12:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f62a006b020 msgr2=0x7f62a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f62a006b020 0x7f62a006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62b0001050 tx=0x7f62b0009040 comp rx=0 tx=0).stop 2023-10-16T14:12:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 msgr2=0x7f62c00cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 0x7f62c00cf010 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f62a8010910 tx=0x7f62a8010dd0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 shutdown_connections 2023-10-16T14:12:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f62a006b020 0x7f62a006d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62c00c5460 0x7f62c00c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62c00ceb90 0x7f62c00cf010 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 --2- 172.21.15.60:0/1423144227 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f62ac004d00 0x7f62c0146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 >> 172.21.15.60:0/1423144227 conn(0x7f62c0065ed0 msgr2=0x7f62c0159390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 shutdown_connections 2023-10-16T14:12:36.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:35.971+0000 7f62bffff640 1 -- 172.21.15.60:0/1423144227 wait complete. 2023-10-16T14:12:36.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:36.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:36.126+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:36.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.243+0000 7fa327b2c640 1 Processor -- start 2023-10-16T14:12:36.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.243+0000 7fa327b2c640 1 -- start start 2023-10-16T14:12:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.244+0000 7fa327b2c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 0x7fa3201615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.244+0000 7fa327b2c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa32015d3f0 0x7fa32015d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.244+0000 7fa327b2c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32015e3e0 con 0x7fa3200cd0c0 2023-10-16T14:12:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.244+0000 7fa327b2c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3200cd4a0 con 0x7fa32015d3f0 2023-10-16T14:12:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.244+0000 7fa327b2c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32015e100 con 0x7fa3200b0b20 2023-10-16T14:12:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.245+0000 7fa3258c8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa3200cd0c0 0x7fa3200b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47326/0 (socket says 172.21.15.60:47326) 2023-10-16T14:12:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.245+0000 7fa3258c8640 1 -- 172.21.15.60:0/584707213 learned_addr learned my addr 172.21.15.60:0/584707213 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.245+0000 7fa3250c7640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 0x7fa3201615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.245+0000 7fa3260c9640 1 --2- 172.21.15.60:0/584707213 >> v2:172.21.15.144:3300/0 conn(0x7fa32015d3f0 0x7fa32015d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.245+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652024707 0 0) 0x7fa32015e3e0 con 0x7fa3200cd0c0 2023-10-16T14:12:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.245+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa30c003520 con 0x7fa3200cd0c0 2023-10-16T14:12:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005308013 0 0) 0x7fa30c003520 con 0x7fa3200cd0c0 2023-10-16T14:12:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 msgr2=0x7fa3201615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 0x7fa3201615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 >> v2:172.21.15.144:3300/0 conn(0x7fa32015d3f0 msgr2=0x7fa32015d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 --2- 172.21.15.60:0/584707213 >> v2:172.21.15.144:3300/0 conn(0x7fa32015d3f0 0x7fa32015d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320160b10 con 0x7fa3200cd0c0 2023-10-16T14:12:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fa31c001670 con 0x7fa3200cd0c0 2023-10-16T14:12:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 >> v1:172.21.15.60:6789/0 conn(0x7fa3200cd0c0 legacy=0x7fa3200b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c003970 0x7fa30c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0047c0 0x7fa30c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 0x7fa30c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200cd4a0 con 0x7fa30c0047c0 2023-10-16T14:12:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32015e100 con 0x7fa30c003970 2023-10-16T14:12:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c00ba00 con 0x7fa3200b0b20 2023-10-16T14:12:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.246+0000 7fa3250c7640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0047c0 0x7fa30c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3250c7640 1 -- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 msgr2=0x7fa30c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3260c9640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 0x7fa30c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3250c7640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa3200b0b20 0x7fa30c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3250c7640 1 -- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c003970 msgr2=0x7fa30c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:36.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3250c7640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa30c003970 0x7fa30c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3250c7640 1 -- 172.21.15.60:0/584707213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c003520 con 0x7fa30c0047c0 2023-10-16T14:12:36.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3250c7640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0047c0 0x7fa30c004c40 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fa310009ac0 tx=0x7fa30c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fc79603517d3b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:36.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31002c6b0 con 0x7fa30c0047c0 2023-10-16T14:12:36.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31002c810 con 0x7fa30c0047c0 2023-10-16T14:12:36.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.248+0000 7fa3248c6640 1 -- 172.21.15.60:0/584707213 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31002cba0 con 0x7fa30c0047c0 2023-10-16T14:12:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa327b2c640 1 -- 172.21.15.60:0/584707213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0047c0 msgr2=0x7fa30c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa327b2c640 1 --2- 172.21.15.60:0/584707213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa30c0047c0 0x7fa30c004c40 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fa310009ac0 tx=0x7fa30c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa3260c9640 1 -- 172.21.15.60:0/584707213 reap_dead start 2023-10-16T14:12:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa327b2c640 1 -- 172.21.15.60:0/584707213 shutdown_connections 2023-10-16T14:12:36.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa327b2c640 1 -- 172.21.15.60:0/584707213 >> 172.21.15.60:0/584707213 conn(0x7fa3200cb410 msgr2=0x7fa3200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa327b2c640 1 -- 172.21.15.60:0/584707213 shutdown_connections 2023-10-16T14:12:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.249+0000 7fa327b2c640 1 -- 172.21.15.60:0/584707213 wait complete. 2023-10-16T14:12:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.251+0000 7fa327b2c640 1 Processor -- start 2023-10-16T14:12:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.251+0000 7fa327b2c640 1 -- start start 2023-10-16T14:12:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa327b2c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0047c0 0x7fa320152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa327b2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 0x7fa3200c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa327b2c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3200c7830 0x7fa32015fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa327b2c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa320152880 con 0x7fa3200c6400 2023-10-16T14:12:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa327b2c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200c8290 con 0x7fa3200c7830 2023-10-16T14:12:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa327b2c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200c7cd0 con 0x7fa30c0047c0 2023-10-16T14:12:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa3250c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 0x7fa3200c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa3250c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 0x7fa3200c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60322/0 (socket says 172.21.15.60:60322) 2023-10-16T14:12:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.252+0000 7fa3250c7640 1 -- 172.21.15.60:0/1052590005 learned_addr learned my addr 172.21.15.60:0/1052590005 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3258c8640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0047c0 0x7fa320152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3250c7640 1 -- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0047c0 msgr2=0x7fa320152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3250c7640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0047c0 0x7fa320152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3250c7640 1 -- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3200c7830 msgr2=0x7fa32015fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3260c9640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3200c7830 0x7fa32015fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3250c7640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3200c7830 0x7fa32015fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3250c7640 1 -- 172.21.15.60:0/1052590005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3200c7fb0 con 0x7fa3200c6400 2023-10-16T14:12:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3260c9640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3200c7830 0x7fa32015fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3250c7640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 0x7fa3200c72f0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fa310037040 tx=0x7fa310004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310003810 con 0x7fa3200c6400 2023-10-16T14:12:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa310003970 con 0x7fa3200c6400 2023-10-16T14:12:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310003d00 con 0x7fa3200c6400 2023-10-16T14:12:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.253+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3200c5c00 con 0x7fa3200c6400 2023-10-16T14:12:36.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.254+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3200c6880 con 0x7fa3200c6400 2023-10-16T14:12:36.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.255+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3200c5a40 con 0x7fa3200c6400 2023-10-16T14:12:36.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.257+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa310047450 con 0x7fa3200c6400 2023-10-16T14:12:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.257+0000 7fa3167fc640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa30406b020 0x7fa30406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.257+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa3100ab7f0 con 0x7fa3200c6400 2023-10-16T14:12:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.257+0000 7fa3258c8640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa30406b020 0x7fa30406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.259+0000 7fa3258c8640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa30406b020 0x7fa30406d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa31c000ff0 tx=0x7fa31c009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:36.268 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.262+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa310076a10 con 0x7fa3200c6400 2023-10-16T14:12:36.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.474+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa32004e160 con 0x7fa3200c6400 2023-10-16T14:12:36.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.476+0000 7fa3167fc640 1 -- 172.21.15.60:0/1052590005 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+12855 (secure 0 0 0) 0x7fa310003e60 con 0x7fa3200c6400 2023-10-16T14:12:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:36.481 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":268,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:35.434741+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":254,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:36.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa30406b020 msgr2=0x7fa30406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa30406b020 0x7fa30406d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa31c000ff0 tx=0x7fa31c009040 comp rx=0 tx=0).stop 2023-10-16T14:12:36.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 msgr2=0x7fa3200c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 0x7fa3200c72f0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fa310037040 tx=0x7fa310004730 comp rx=0 tx=0).stop 2023-10-16T14:12:36.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 shutdown_connections 2023-10-16T14:12:36.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa30406b020 0x7fa30406d4e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3200c7830 0x7fa32015fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3200c6400 0x7fa3200c72f0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 --2- 172.21.15.60:0/1052590005 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa30c0047c0 0x7fa320152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.479+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 >> 172.21.15.60:0/1052590005 conn(0x7fa3200cb410 msgr2=0x7fa3200ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:36.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.480+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 shutdown_connections 2023-10-16T14:12:36.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.480+0000 7fa327b2c640 1 -- 172.21.15.60:0/1052590005 wait complete. 2023-10-16T14:12:36.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:12:36.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:12:36.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.732+0000 7f55199a9640 1 Processor -- start 2023-10-16T14:12:36.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.732+0000 7f55199a9640 1 -- start start 2023-10-16T14:12:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.734+0000 7f55199a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f551415ad70 0x7f5514069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.734+0000 7f55199a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55140c9ca0 0x7f551406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.734+0000 7f55199a9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55140cb030 con 0x7f55140ca3b0 2023-10-16T14:12:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.734+0000 7f55199a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55140c91c0 con 0x7f551415ad70 2023-10-16T14:12:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.734+0000 7f55199a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55140c94a0 con 0x7f55140c9ca0 2023-10-16T14:12:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.734+0000 7f5512ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f551415ad70 0x7f5514069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.735+0000 7f5512ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f551415ad70 0x7f5514069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:51848/0 (socket says 172.21.15.60:51848) 2023-10-16T14:12:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.735+0000 7f5512ffd640 1 -- 172.21.15.60:0/2732825556 learned_addr learned my addr 172.21.15.60:0/2732825556 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:36.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.735+0000 7f55127fc640 1 --2- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55140c9ca0 0x7f551406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.735+0000 7f5512ffd640 1 -- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55140c9ca0 msgr2=0x7f551406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.735+0000 7f5512ffd640 1 --2- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55140c9ca0 0x7f551406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.736+0000 7f5512ffd640 1 -- 172.21.15.60:0/2732825556 >> v1:172.21.15.144:6789/0 conn(0x7f55140ca3b0 legacy=0x7f55140ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.736+0000 7f5512ffd640 1 -- 172.21.15.60:0/2732825556 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55140c9780 con 0x7f551415ad70 2023-10-16T14:12:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.737+0000 7f5512ffd640 1 --2- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f551415ad70 0x7f5514069da0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5504009e50 tx=0x7f550402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90d947eb0e9e6b0d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f5511ffb640 1 -- 172.21.15.60:0/2732825556 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550402cb00 con 0x7f551415ad70 2023-10-16T14:12:36.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f5511ffb640 1 -- 172.21.15.60:0/2732825556 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550402cc60 con 0x7f551415ad70 2023-10-16T14:12:36.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f5511ffb640 1 -- 172.21.15.60:0/2732825556 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55040343f0 con 0x7f551415ad70 2023-10-16T14:12:36.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 -- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f551415ad70 msgr2=0x7f5514069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 --2- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f551415ad70 0x7f5514069da0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5504009e50 tx=0x7f550402bf60 comp rx=0 tx=0).stop 2023-10-16T14:12:36.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 -- 172.21.15.60:0/2732825556 shutdown_connections 2023-10-16T14:12:36.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 --2- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f55140c9ca0 0x7f551406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 --2- 172.21.15.60:0/2732825556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f551415ad70 0x7f5514069da0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 -- 172.21.15.60:0/2732825556 >> 172.21.15.60:0/2732825556 conn(0x7f55140ce9b0 msgr2=0x7f55140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.738+0000 7f55199a9640 1 -- 172.21.15.60:0/2732825556 shutdown_connections 2023-10-16T14:12:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.739+0000 7f55199a9640 1 -- 172.21.15.60:0/2732825556 wait complete. 2023-10-16T14:12:36.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.740+0000 7f55199a9640 1 Processor -- start 2023-10-16T14:12:36.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.740+0000 7f55199a9640 1 -- start start 2023-10-16T14:12:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.741+0000 7f55199a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140c9ca0 0x7f55140c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.741+0000 7f55199a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 0x7f55140c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.741+0000 7f55199a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415ad70 0x7f55140c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.741+0000 7f55199a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55141f41c0 con 0x7f55140ca3b0 2023-10-16T14:12:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.741+0000 7f55199a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514158790 con 0x7f55140c9ca0 2023-10-16T14:12:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.741+0000 7f55199a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55140c4a70 con 0x7f551415ad70 2023-10-16T14:12:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 0x7f55140c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 0x7f55140c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60338/0 (socket says 172.21.15.60:60338) 2023-10-16T14:12:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 -- 172.21.15.60:0/3160715785 learned_addr learned my addr 172.21.15.60:0/3160715785 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f5512ffd640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140c9ca0 0x7f55140c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55137fe640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415ad70 0x7f55140c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 -- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415ad70 msgr2=0x7f55140c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415ad70 0x7f55140c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 -- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140c9ca0 msgr2=0x7f55140c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140c9ca0 0x7f55140c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55127fc640 1 -- 172.21.15.60:0/3160715785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5504009b30 con 0x7f55140ca3b0 2023-10-16T14:12:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.742+0000 7f55137fe640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415ad70 0x7f55140c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:36.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.743+0000 7f55127fc640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 0x7f55140c5a80 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f550800e9b0 tx=0x7f550800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:36.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.743+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508009670 con 0x7f55140ca3b0 2023-10-16T14:12:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.743+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550800c9a0 con 0x7f55140ca3b0 2023-10-16T14:12:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.743+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550800cd50 con 0x7f55140ca3b0 2023-10-16T14:12:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.743+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55141f3540 con 0x7f55140ca3b0 2023-10-16T14:12:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.743+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55141f3b00 con 0x7f55140ca3b0 2023-10-16T14:12:36.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.744+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55141f4300 con 0x7f55140ca3b0 2023-10-16T14:12:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.745+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f5508013470 con 0x7f55140ca3b0 2023-10-16T14:12:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.746+0000 7f54fbfff640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f550006b020 0x7f550006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.746+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5508089930 con 0x7f55140ca3b0 2023-10-16T14:12:36.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.747+0000 7f5512ffd640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f550006b020 0x7f550006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:36.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.748+0000 7f5512ffd640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f550006b020 0x7f550006d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5504035000 tx=0x7f5504030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:36.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.751+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55141f4300 con 0x7f55140ca3b0 2023-10-16T14:12:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.961+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55141f4300 con 0x7f55140ca3b0 2023-10-16T14:12:36.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.962+0000 7f54fbfff640 1 -- 172.21.15.60:0/3160715785 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v268) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55141f4300 con 0x7f55140ca3b0 2023-10-16T14:12:36.966 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:12:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f550006b020 msgr2=0x7f550006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f550006b020 0x7f550006d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5504035000 tx=0x7f5504030000 comp rx=0 tx=0).stop 2023-10-16T14:12:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 msgr2=0x7f55140c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 0x7f55140c5a80 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f550800e9b0 tx=0x7f550800ee70 comp rx=0 tx=0).stop 2023-10-16T14:12:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 shutdown_connections 2023-10-16T14:12:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f550006b020 0x7f550006d4e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f551415ad70 0x7f55140c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55140ca3b0 0x7f55140c5a80 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 --2- 172.21.15.60:0/3160715785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55140c9ca0 0x7f55140c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 >> 172.21.15.60:0/3160715785 conn(0x7f55140ce9b0 msgr2=0x7f551415a040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 shutdown_connections 2023-10-16T14:12:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:36.965+0000 7f55199a9640 1 -- 172.21.15.60:0/3160715785 wait complete. 2023-10-16T14:12:36.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:12:37.208 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:37.204+0000 7f37920bd640 -1 client.9697: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.8336:91553663-e4a9-4d2d-8fe4-cde9a4154018:v1:172.21.15.144:0/1518159871] 2023-10-16T14:12:37.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.216+0000 7f9d3d87c640 1 Processor -- start 2023-10-16T14:12:37.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.216+0000 7f9d3d87c640 1 -- start start 2023-10-16T14:12:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d3d87c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9d380b3070 0x7f9d380cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d3d87c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3e70 0x7f9d380cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d3d87c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d380b3450 0x7f9d380c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d3d87c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d380b3940 con 0x7f9d380b3070 2023-10-16T14:12:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d3d87c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d380b45e0 con 0x7f9d380b3450 2023-10-16T14:12:37.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d3d87c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b3b90 con 0x7f9d380b3e70 2023-10-16T14:12:37.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d36ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9d380b3070 0x7f9d380cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d36ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9d380b3070 0x7f9d380cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60350/0 (socket says 172.21.15.60:60350) 2023-10-16T14:12:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.218+0000 7f9d36ffd640 1 -- 172.21.15.60:0/4136239688 learned_addr learned my addr 172.21.15.60:0/4136239688 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d2e7fc640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3e70 0x7f9d380cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d377fe640 1 --2- 172.21.15.60:0/4136239688 >> v2:172.21.15.144:3300/0 conn(0x7f9d380b3450 0x7f9d380c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d36ffd640 1 -- 172.21.15.60:0/4136239688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3e70 msgr2=0x7f9d380cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d36ffd640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3e70 0x7f9d380cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d36ffd640 1 -- 172.21.15.60:0/4136239688 >> v2:172.21.15.144:3300/0 conn(0x7f9d380b3450 msgr2=0x7f9d380c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d36ffd640 1 --2- 172.21.15.60:0/4136239688 >> v2:172.21.15.144:3300/0 conn(0x7f9d380b3450 0x7f9d380c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d36ffd640 1 -- 172.21.15.60:0/4136239688 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380d3840 con 0x7f9d380b3070 2023-10-16T14:12:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d2e7fc640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3e70 0x7f9d380cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:37.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.219+0000 7f9d377fe640 1 --2- 172.21.15.60:0/4136239688 >> v2:172.21.15.144:3300/0 conn(0x7f9d380b3450 0x7f9d380c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.221+0000 7f9d36ffd640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3070 0x7f9d380cf500 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f9d3808c4f0 tx=0x7f9d2802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc0235997a30d33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:37.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.221+0000 7f9d367fc640 1 -- 172.21.15.60:0/4136239688 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2802cad0 con 0x7f9d380b3070 2023-10-16T14:12:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.221+0000 7f9d367fc640 1 -- 172.21.15.60:0/4136239688 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2802cc30 con 0x7f9d380b3070 2023-10-16T14:12:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.221+0000 7f9d367fc640 1 -- 172.21.15.60:0/4136239688 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280343f0 con 0x7f9d380b3070 2023-10-16T14:12:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.221+0000 7f9d3d87c640 1 -- 172.21.15.60:0/4136239688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3070 msgr2=0x7f9d380cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.221+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3070 0x7f9d380cf500 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f9d3808c4f0 tx=0x7f9d2802bf30 comp rx=0 tx=0).stop 2023-10-16T14:12:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 -- 172.21.15.60:0/4136239688 shutdown_connections 2023-10-16T14:12:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/4136239688 >> v2:172.21.15.144:3300/0 conn(0x7f9d380b3450 0x7f9d380c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3e70 0x7f9d380cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/4136239688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3070 0x7f9d380cf500 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 -- 172.21.15.60:0/4136239688 >> 172.21.15.60:0/4136239688 conn(0x7f9d380d2d20 msgr2=0x7f9d380d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 -- 172.21.15.60:0/4136239688 shutdown_connections 2023-10-16T14:12:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.222+0000 7f9d3d87c640 1 -- 172.21.15.60:0/4136239688 wait complete. 2023-10-16T14:12:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.224+0000 7f9d3d87c640 1 Processor -- start 2023-10-16T14:12:37.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.224+0000 7f9d3d87c640 1 -- start start 2023-10-16T14:12:37.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d3d87c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3070 0x7f9d380cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d3d87c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 0x7f9d381e66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d3d87c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d380b3e70 0x7f9d380ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d3d87c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3815fd20 con 0x7f9d380b3450 2023-10-16T14:12:37.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d3d87c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380cc3e0 con 0x7f9d380b3e70 2023-10-16T14:12:37.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d3d87c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380cc6c0 con 0x7f9d380b3070 2023-10-16T14:12:37.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d2e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 0x7f9d381e66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d2e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 0x7f9d381e66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60352/0 (socket says 172.21.15.60:60352) 2023-10-16T14:12:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.225+0000 7f9d2e7fc640 1 -- 172.21.15.60:0/3946831120 learned_addr learned my addr 172.21.15.60:0/3946831120 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d36ffd640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3070 0x7f9d380cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d377fe640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d380b3e70 0x7f9d380ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d2e7fc640 1 -- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3070 msgr2=0x7f9d380cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d2e7fc640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3070 0x7f9d380cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d2e7fc640 1 -- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d380b3e70 msgr2=0x7f9d380ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d2e7fc640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d380b3e70 0x7f9d380ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d2e7fc640 1 -- 172.21.15.60:0/3946831120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28009b30 con 0x7f9d380b3450 2023-10-16T14:12:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d2e7fc640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 0x7f9d381e66a0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000c7f0 tx=0x7f9d2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:37.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20007820 con 0x7f9d380b3450 2023-10-16T14:12:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20007980 con 0x7f9d380b3450 2023-10-16T14:12:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20007cf0 con 0x7f9d380b3450 2023-10-16T14:12:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.226+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380cc9a0 con 0x7f9d380b3450 2023-10-16T14:12:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.227+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d380ccf60 con 0x7f9d380b3450 2023-10-16T14:12:37.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.229+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d380ce490 con 0x7f9d380b3450 2023-10-16T14:12:37.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.230+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f9d200113f0 con 0x7f9d380b3450 2023-10-16T14:12:37.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.231+0000 7f9d34ff9640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9d1806b020 0x7f9d1806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.231+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9d20089970 con 0x7f9d380b3450 2023-10-16T14:12:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.231+0000 7f9d36ffd640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9d1806b020 0x7f9d1806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.233+0000 7f9d36ffd640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9d1806b020 0x7f9d1806d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9d28035000 tx=0x7f9d28030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:37.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.235+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d20054a70 con 0x7f9d380b3450 2023-10-16T14:12:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.437+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9d3804e160 con 0x7f9d380b3450 2023-10-16T14:12:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.438+0000 7f9d34ff9640 1 -- 172.21.15.60:0/3946831120 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v268) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9d2001f090 con 0x7f9d380b3450 2023-10-16T14:12:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:37.442 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:12:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:37.442 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:12:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:12:37.443 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:37.444 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:12:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9d1806b020 msgr2=0x7f9d1806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9d1806b020 0x7f9d1806d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9d28035000 tx=0x7f9d28030000 comp rx=0 tx=0).stop 2023-10-16T14:12:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 msgr2=0x7f9d381e66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 0x7f9d381e66a0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000c7f0 tx=0x7f9d2000ccb0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 shutdown_connections 2023-10-16T14:12:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9d1806b020 0x7f9d1806d4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d380b3e70 0x7f9d380ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d380b3450 0x7f9d381e66a0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 --2- 172.21.15.60:0/3946831120 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9d380b3070 0x7f9d380cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 >> 172.21.15.60:0/3946831120 conn(0x7f9d380d2d20 msgr2=0x7f9d38160440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 shutdown_connections 2023-10-16T14:12:37.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.441+0000 7f9d3d87c640 1 -- 172.21.15.60:0/3946831120 wait complete. 2023-10-16T14:12:37.463 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:12:37.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:12:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.692+0000 7fa257fff640 1 Processor -- start 2023-10-16T14:12:37.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.692+0000 7fa257fff640 1 -- start start 2023-10-16T14:12:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa257fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25815d0e0 0x7fa2580b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa257fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2580d2040 0x7fa2580cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa257fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2580ce2a0 con 0x7fa2580cd900 2023-10-16T14:12:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa257fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25815d4c0 con 0x7fa2580d2040 2023-10-16T14:12:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa257fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa258160ca0 con 0x7fa25815d0e0 2023-10-16T14:12:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2580d2040 0x7fa2580cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2580d2040 0x7fa2580cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60358/0 (socket says 172.21.15.60:60358) 2023-10-16T14:12:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 -- 172.21.15.60:0/769582956 learned_addr learned my addr 172.21.15.60:0/769582956 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa256ffd640 1 --2- 172.21.15.60:0/769582956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25815d0e0 0x7fa2580b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 -- 172.21.15.60:0/769582956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25815d0e0 msgr2=0x7fa2580b84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 --2- 172.21.15.60:0/769582956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25815d0e0 0x7fa2580b84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 -- 172.21.15.60:0/769582956 >> v1:172.21.15.60:6789/0 conn(0x7fa2580cd900 legacy=0x7fa2580b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.694+0000 7fa2577fe640 1 -- 172.21.15.60:0/769582956 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258160f80 con 0x7fa2580d2040 2023-10-16T14:12:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.696+0000 7fa2577fe640 1 --2- 172.21.15.60:0/769582956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580d2040 0x7fa2580cdd60 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fa250009870 tx=0x7fa25002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847200bd9f5782ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.696+0000 7fa2567fc640 1 -- 172.21.15.60:0/769582956 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250031640 con 0x7fa2580d2040 2023-10-16T14:12:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.696+0000 7fa2567fc640 1 -- 172.21.15.60:0/769582956 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2500317a0 con 0x7fa2580d2040 2023-10-16T14:12:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.696+0000 7fa2567fc640 1 -- 172.21.15.60:0/769582956 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250031b30 con 0x7fa2580d2040 2023-10-16T14:12:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.696+0000 7fa257fff640 1 -- 172.21.15.60:0/769582956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580d2040 msgr2=0x7fa2580cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.696+0000 7fa257fff640 1 --2- 172.21.15.60:0/769582956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580d2040 0x7fa2580cdd60 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fa250009870 tx=0x7fa25002b8f0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.697+0000 7fa257fff640 1 -- 172.21.15.60:0/769582956 shutdown_connections 2023-10-16T14:12:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.697+0000 7fa257fff640 1 --2- 172.21.15.60:0/769582956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580d2040 0x7fa2580cdd60 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.697+0000 7fa257fff640 1 --2- 172.21.15.60:0/769582956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa25815d0e0 0x7fa2580b84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.697+0000 7fa257fff640 1 -- 172.21.15.60:0/769582956 >> 172.21.15.60:0/769582956 conn(0x7fa25806bc60 msgr2=0x7fa25806c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.697+0000 7fa257fff640 1 -- 172.21.15.60:0/769582956 shutdown_connections 2023-10-16T14:12:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.697+0000 7fa257fff640 1 -- 172.21.15.60:0/769582956 wait complete. 2023-10-16T14:12:37.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.698+0000 7fa257fff640 1 Processor -- start 2023-10-16T14:12:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.699+0000 7fa257fff640 1 -- start start 2023-10-16T14:12:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.699+0000 7fa257fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 0x7fa2581514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa257fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2580d2040 0x7fa2581527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa257fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25815d0e0 0x7fa2580ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa257fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580b3340 con 0x7fa2580cd900 2023-10-16T14:12:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa257fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580b2730 con 0x7fa25815d0e0 2023-10-16T14:12:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa257fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580b2a10 con 0x7fa2580d2040 2023-10-16T14:12:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 0x7fa2581514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 0x7fa2581514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60370/0 (socket says 172.21.15.60:60370) 2023-10-16T14:12:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 -- 172.21.15.60:0/3207838003 learned_addr learned my addr 172.21.15.60:0/3207838003 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa24e7fc640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2580d2040 0x7fa2581527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa2577fe640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25815d0e0 0x7fa2580ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 -- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2580d2040 msgr2=0x7fa2581527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2580d2040 0x7fa2581527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 -- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25815d0e0 msgr2=0x7fa2580ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25815d0e0 0x7fa2580ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa256ffd640 1 -- 172.21.15.60:0/3207838003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500094d0 con 0x7fa2580cd900 2023-10-16T14:12:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.700+0000 7fa2577fe640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25815d0e0 0x7fa2580ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.701+0000 7fa256ffd640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 0x7fa2581514e0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fa248007ef0 tx=0x7fa24800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.701+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248014620 con 0x7fa2580cd900 2023-10-16T14:12:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.701+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa248014780 con 0x7fa2580cd900 2023-10-16T14:12:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.701+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248014b30 con 0x7fa2580cd900 2023-10-16T14:12:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.701+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2580b2cf0 con 0x7fa2580cd900 2023-10-16T14:12:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.701+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa258151a20 con 0x7fa2580cd900 2023-10-16T14:12:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.703+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa258072190 con 0x7fa2580cd900 2023-10-16T14:12:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.704+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa248014c90 con 0x7fa2580cd900 2023-10-16T14:12:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.706+0000 7fa24ffff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa23806b020 0x7fa23806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.706+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa248089ff0 con 0x7fa2580cd900 2023-10-16T14:12:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.706+0000 7fa24e7fc640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa23806b020 0x7fa23806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.707+0000 7fa24e7fc640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa23806b020 0x7fa23806d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2581f5d00 tx=0x7fa240009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.711+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa248055160 con 0x7fa2580cd900 2023-10-16T14:12:37.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:37.912+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa25804e9b0 con 0x7fa2580cd900 2023-10-16T14:12:38.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.877+0000 7fa24ffff640 1 -- 172.21.15.60:0/3207838003 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v270) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa248058780 con 0x7fa2580cd900 2023-10-16T14:12:38.880 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T14:12:38.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa23806b020 msgr2=0x7fa23806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa23806b020 0x7fa23806d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2581f5d00 tx=0x7fa240009040 comp rx=0 tx=0).stop 2023-10-16T14:12:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 msgr2=0x7fa2581514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 0x7fa2581514e0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fa248007ef0 tx=0x7fa24800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:12:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 shutdown_connections 2023-10-16T14:12:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa23806b020 0x7fa23806d4e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:38.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2580d2040 0x7fa2581527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:38.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa25815d0e0 0x7fa2580ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:38.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 --2- 172.21.15.60:0/3207838003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2580cd900 0x7fa2581514e0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:38.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 >> 172.21.15.60:0/3207838003 conn(0x7fa25806bc60 msgr2=0x7fa25815d600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:38.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 shutdown_connections 2023-10-16T14:12:38.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:38.881+0000 7fa257fff640 1 -- 172.21.15.60:0/3207838003 wait complete. 2023-10-16T14:12:38.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:12:39.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.130+0000 7f59f27dc640 1 Processor -- start 2023-10-16T14:12:39.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.130+0000 7f59f27dc640 1 -- start start 2023-10-16T14:12:39.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59f27dc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f59ec159f00 0x7f59ec0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:39.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59f27dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:39.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59f27dc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec06f890 con 0x7f59ec159b20 2023-10-16T14:12:39.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59f27dc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ec070d30 con 0x7f59ec159f00 2023-10-16T14:12:39.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59f27dc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec06c650 con 0x7f59ec15d8c0 2023-10-16T14:12:39.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59ebfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f59ec159b20 0x7f59ec06bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50904/0 (socket says 172.21.15.60:50904) 2023-10-16T14:12:39.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.132+0000 7f59ebfff640 1 -- 172.21.15.60:0/415380756 learned_addr learned my addr 172.21.15.60:0/415380756 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59f0d79640 1 --2- 172.21.15.60:0/415380756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59e37fe640 1 --2- 172.21.15.60:0/415380756 >> v2:172.21.15.144:3300/0 conn(0x7f59ec159f00 0x7f59ec0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59eb7fe640 1 -- 172.21.15.60:0/415380756 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900476295 0 0) 0x7f59ec06f890 con 0x7f59ec159b20 2023-10-16T14:12:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59eb7fe640 1 -- 172.21.15.60:0/415380756 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d8003520 con 0x7f59ec159b20 2023-10-16T14:12:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59eb7fe640 1 -- 172.21.15.60:0/415380756 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073535856 0 0) 0x7f59d8003520 con 0x7f59ec159b20 2023-10-16T14:12:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59f0d79640 1 -- 172.21.15.60:0/415380756 >> v2:172.21.15.144:3300/0 conn(0x7f59ec159f00 msgr2=0x7f59ec0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59f0d79640 1 --2- 172.21.15.60:0/415380756 >> v2:172.21.15.144:3300/0 conn(0x7f59ec159f00 0x7f59ec0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59f0d79640 1 -- 172.21.15.60:0/415380756 >> v1:172.21.15.60:6789/0 conn(0x7f59ec159b20 legacy=0x7f59ec06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.133+0000 7f59f0d79640 1 -- 172.21.15.60:0/415380756 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec06c930 con 0x7f59ec15d8c0 2023-10-16T14:12:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.135+0000 7f59f0d79640 1 --2- 172.21.15.60:0/415380756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0b0950 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f59e40097a0 tx=0x7f59e402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f789f5f5a6c1cb29 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.135+0000 7f59eb7fe640 1 -- 172.21.15.60:0/415380756 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc0019f0 con 0x7f59ec15d8c0 2023-10-16T14:12:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.135+0000 7f59eb7fe640 1 -- 172.21.15.60:0/415380756 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e402e050 con 0x7f59ec15d8c0 2023-10-16T14:12:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.135+0000 7f59eb7fe640 1 -- 172.21.15.60:0/415380756 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e4030730 con 0x7f59ec15d8c0 2023-10-16T14:12:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 -- 172.21.15.60:0/415380756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 msgr2=0x7f59ec0b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 --2- 172.21.15.60:0/415380756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0b0950 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f59e40097a0 tx=0x7f59e402b790 comp rx=0 tx=0).stop 2023-10-16T14:12:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 -- 172.21.15.60:0/415380756 shutdown_connections 2023-10-16T14:12:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 --2- 172.21.15.60:0/415380756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0b0950 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 --2- 172.21.15.60:0/415380756 >> v2:172.21.15.144:3300/0 conn(0x7f59ec159f00 0x7f59ec0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 -- 172.21.15.60:0/415380756 >> 172.21.15.60:0/415380756 conn(0x7f59ec0cb410 msgr2=0x7f59ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 -- 172.21.15.60:0/415380756 shutdown_connections 2023-10-16T14:12:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.136+0000 7f59f27dc640 1 -- 172.21.15.60:0/415380756 wait complete. 2023-10-16T14:12:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.138+0000 7f59f27dc640 1 Processor -- start 2023-10-16T14:12:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.139+0000 7f59f27dc640 1 -- start start 2023-10-16T14:12:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f27dc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59ec159b20 0x7f59ec14bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f27dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 0x7f59ec0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f27dc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f27dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0cdf20 con 0x7f59ec159f00 2023-10-16T14:12:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f27dc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0cce80 con 0x7f59ec159b20 2023-10-16T14:12:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f27dc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0cd160 con 0x7f59ec15d8c0 2023-10-16T14:12:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59e37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 0x7f59ec0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59e37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 0x7f59ec0cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48256/0 (socket says 172.21.15.60:48256) 2023-10-16T14:12:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59e37fe640 1 -- 172.21.15.60:0/3086716872 learned_addr learned my addr 172.21.15.60:0/3086716872 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59ebfff640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59ec159b20 0x7f59ec14bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.140+0000 7f59f0d79640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e37fe640 1 -- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 msgr2=0x7f59ec0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e37fe640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e37fe640 1 -- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59ec159b20 msgr2=0x7f59ec14bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:39.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e37fe640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59ec159b20 0x7f59ec14bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e37fe640 1 -- 172.21.15.60:0/3086716872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e4009400 con 0x7f59ec159f00 2023-10-16T14:12:39.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e37fe640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 0x7f59ec0cc3c0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f59d400e9b0 tx=0x7f59d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:39.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4009670 con 0x7f59ec159f00 2023-10-16T14:12:39.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d400c9a0 con 0x7f59ec159f00 2023-10-16T14:12:39.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d400cd50 con 0x7f59ec159f00 2023-10-16T14:12:39.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.141+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec0cd440 con 0x7f59ec159f00 2023-10-16T14:12:39.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.142+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec0cda00 con 0x7f59ec159f00 2023-10-16T14:12:39.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.143+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec04e160 con 0x7f59ec159f00 2023-10-16T14:12:39.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.144+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f59d4013470 con 0x7f59ec159f00 2023-10-16T14:12:39.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.145+0000 7f59e97fa640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f59cc06aff0 0x7f59cc06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:39.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.145+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5270+0+0 (secure 0 0 0) 0x7f59d4089e20 con 0x7f59ec159f00 2023-10-16T14:12:39.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.146+0000 7f59ebfff640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f59cc06aff0 0x7f59cc06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:39.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.147+0000 7f59ebfff640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f59cc06aff0 0x7f59cc06d4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59dc001f10 tx=0x7f59dc009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:39.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.149+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59d4054de0 con 0x7f59ec159f00 2023-10-16T14:12:39.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.347+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f59ec0ad920 con 0x7f59ec159f00 2023-10-16T14:12:39.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.898+0000 7f59e97fa640 1 -- 172.21.15.60:0/3086716872 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v271) v1 ==== 170+0+0 (secure 0 0 0) 0x7f59d40170d0 con 0x7f59ec159f00 2023-10-16T14:12:39.901 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T14:12:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f59cc06aff0 msgr2=0x7f59cc06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:39.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f59cc06aff0 0x7f59cc06d4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59dc001f10 tx=0x7f59dc009040 comp rx=0 tx=0).stop 2023-10-16T14:12:39.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 msgr2=0x7f59ec0cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:39.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 0x7f59ec0cc3c0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f59d400e9b0 tx=0x7f59d400ee70 comp rx=0 tx=0).stop 2023-10-16T14:12:39.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 shutdown_connections 2023-10-16T14:12:39.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f59cc06aff0 0x7f59cc06d4b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f59ec15d8c0 0x7f59ec0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59ec159f00 0x7f59ec0cc3c0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 --2- 172.21.15.60:0/3086716872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59ec159b20 0x7f59ec14bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 >> 172.21.15.60:0/3086716872 conn(0x7f59ec0cb410 msgr2=0x7f59ec0ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 shutdown_connections 2023-10-16T14:12:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:39.902+0000 7f59f27dc640 1 -- 172.21.15.60:0/3086716872 wait complete. 2023-10-16T14:12:39.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:12:40.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.132+0000 7f2c582cf640 1 Processor -- start 2023-10-16T14:12:40.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.133+0000 7f2c582cf640 1 -- start start 2023-10-16T14:12:40.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c582cf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c5015ad70 0x7f2c50069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:40.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c582cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c500cb030 con 0x7f2c500c9ca0 2023-10-16T14:12:40.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c582cf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c500c91c0 con 0x7f2c500caac0 2023-10-16T14:12:40.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c582cf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c500c94a0 con 0x7f2c5015ad70 2023-10-16T14:12:40.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c5606b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c5015ad70 0x7f2c50069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:40.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c5606b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c5015ad70 0x7f2c50069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48264/0 (socket says 172.21.15.60:48264) 2023-10-16T14:12:40.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.134+0000 7f2c5606b640 1 -- 172.21.15.60:0/4015591158 learned_addr learned my addr 172.21.15.60:0/4015591158 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:40.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.135+0000 7f2c5606b640 1 -- 172.21.15.60:0/4015591158 >> v1:172.21.15.144:6789/0 conn(0x7f2c500caac0 legacy=0x7f2c500ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:40.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.135+0000 7f2c5606b640 1 -- 172.21.15.60:0/4015591158 >> v1:172.21.15.60:6789/0 conn(0x7f2c500c9ca0 legacy=0x7f2c500ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:40.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.135+0000 7f2c5606b640 1 -- 172.21.15.60:0/4015591158 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500c9780 con 0x7f2c5015ad70 2023-10-16T14:12:40.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.136+0000 7f2c5606b640 1 --2- 172.21.15.60:0/4015591158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c5015ad70 0x7f2c50069da0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009870 tx=0x7f2c4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f682b3c1a3c7c96d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:40.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.136+0000 7f2c55069640 1 -- 172.21.15.60:0/4015591158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c030860 con 0x7f2c5015ad70 2023-10-16T14:12:40.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.136+0000 7f2c55069640 1 -- 172.21.15.60:0/4015591158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c0309c0 con 0x7f2c5015ad70 2023-10-16T14:12:40.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.136+0000 7f2c55069640 1 -- 172.21.15.60:0/4015591158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c030d50 con 0x7f2c5015ad70 2023-10-16T14:12:40.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 -- 172.21.15.60:0/4015591158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c5015ad70 msgr2=0x7f2c50069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:40.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 --2- 172.21.15.60:0/4015591158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c5015ad70 0x7f2c50069da0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009870 tx=0x7f2c4c02b9a0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 -- 172.21.15.60:0/4015591158 shutdown_connections 2023-10-16T14:12:40.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 --2- 172.21.15.60:0/4015591158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c5015ad70 0x7f2c50069da0 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 -- 172.21.15.60:0/4015591158 >> 172.21.15.60:0/4015591158 conn(0x7f2c500ce9b0 msgr2=0x7f2c500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:40.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 -- 172.21.15.60:0/4015591158 shutdown_connections 2023-10-16T14:12:40.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.137+0000 7f2c582cf640 1 -- 172.21.15.60:0/4015591158 wait complete. 2023-10-16T14:12:40.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.139+0000 7f2c582cf640 1 Processor -- start 2023-10-16T14:12:40.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.140+0000 7f2c582cf640 1 -- start start 2023-10-16T14:12:40.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c582cf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 0x7f2c50147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:40.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c582cf640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c500caac0 0x7f2c5014bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c582cf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c5015ad70 0x7f2c500c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c582cf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500c6730 con 0x7f2c500c9ca0 2023-10-16T14:12:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c582cf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500c5540 con 0x7f2c5015ad70 2023-10-16T14:12:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c582cf640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500c5b20 con 0x7f2c500caac0 2023-10-16T14:12:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 0x7f2c50147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:40.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 0x7f2c50147de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48274/0 (socket says 172.21.15.60:48274) 2023-10-16T14:12:40.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 -- 172.21.15.60:0/1405380794 learned_addr learned my addr 172.21.15.60:0/1405380794 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:40.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 -- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c500caac0 msgr2=0x7f2c5014bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:40.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c500caac0 0x7f2c5014bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 -- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c5015ad70 msgr2=0x7f2c500c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:40.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c5015ad70 0x7f2c500c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.141+0000 7f2c5606b640 1 -- 172.21.15.60:0/1405380794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c0094d0 con 0x7f2c500c9ca0 2023-10-16T14:12:40.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.142+0000 7f2c5606b640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 0x7f2c50147de0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009870 tx=0x7f2c4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:40.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.142+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c039b60 con 0x7f2c500c9ca0 2023-10-16T14:12:40.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.142+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c039cc0 con 0x7f2c500c9ca0 2023-10-16T14:12:40.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.142+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c043630 con 0x7f2c500c9ca0 2023-10-16T14:12:40.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.142+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c500c5e00 con 0x7f2c500c9ca0 2023-10-16T14:12:40.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.142+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c500c63c0 con 0x7f2c500c9ca0 2023-10-16T14:12:40.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.143+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c500c6550 con 0x7f2c500c9ca0 2023-10-16T14:12:40.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.145+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2c4c02f3b0 con 0x7f2c500c9ca0 2023-10-16T14:12:40.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.145+0000 7f2c46ffd640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2c3406b0b0 0x7f2c3406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:40.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.146+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5652+0+0 (secure 0 0 0) 0x7f2c4c0abf30 con 0x7f2c500c9ca0 2023-10-16T14:12:40.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.146+0000 7f2c5586a640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2c3406b0b0 0x7f2c3406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:40.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.147+0000 7f2c5586a640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2c3406b0b0 0x7f2c3406d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c40000f60 tx=0x7f2c400079b0 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:40.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.151+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2c4c076b00 con 0x7f2c500c9ca0 2023-10-16T14:12:40.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.385+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2c5004e160 con 0x7f2c500c9ca0 2023-10-16T14:12:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.902+0000 7f2c46ffd640 1 -- 172.21.15.60:0/1405380794 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v141) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2c4c07a120 con 0x7f2c500c9ca0 2023-10-16T14:12:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 34 and data pool 35 2023-10-16T14:12:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2c3406b0b0 msgr2=0x7f2c3406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2c3406b0b0 0x7f2c3406d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c40000f60 tx=0x7f2c400079b0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 msgr2=0x7f2c50147de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:40.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 0x7f2c50147de0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009870 tx=0x7f2c4c004730 comp rx=0 tx=0).stop 2023-10-16T14:12:40.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 shutdown_connections 2023-10-16T14:12:40.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2c3406b0b0 0x7f2c3406d570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2c500caac0 0x7f2c5014bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c5015ad70 0x7f2c500c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 --2- 172.21.15.60:0/1405380794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c500c9ca0 0x7f2c50147de0 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.907+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 >> 172.21.15.60:0/1405380794 conn(0x7f2c500ce9b0 msgr2=0x7f2c500b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:40.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.908+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 shutdown_connections 2023-10-16T14:12:40.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:40.908+0000 7f2c582cf640 1 -- 172.21.15.60:0/1405380794 wait complete. 2023-10-16T14:12:40.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:41.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.141+0000 7f9fae083640 1 Processor -- start 2023-10-16T14:12:41.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.141+0000 7f9fae083640 1 -- start start 2023-10-16T14:12:41.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.143+0000 7f9fae083640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 0x7f9fa815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.143+0000 7f9fae083640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9fa815ae80 0x7f9fa80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.143+0000 7f9fae083640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa815ed60 con 0x7f9fa80ccaa0 2023-10-16T14:12:41.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.143+0000 7f9fae083640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa806b810 con 0x7f9fa815ae80 2023-10-16T14:12:41.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.143+0000 7f9fae083640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80707e0 con 0x7f9fa815aaa0 2023-10-16T14:12:41.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.144+0000 7f9f9ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9fa815ae80 0x7f9fa80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.144+0000 7f9fa77fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 0x7f9fa815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.144+0000 7f9fa77fe640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 0x7f9fa815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:47074/0 (socket says 172.21.15.60:47074) 2023-10-16T14:12:41.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.144+0000 7f9fa77fe640 1 -- 172.21.15.60:0/465556707 learned_addr learned my addr 172.21.15.60:0/465556707 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:41.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9fa6ffd640 1 -- 172.21.15.60:0/465556707 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817210849 0 0) 0x7f9fa815ed60 con 0x7f9fa80ccaa0 2023-10-16T14:12:41.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9fa6ffd640 1 -- 172.21.15.60:0/465556707 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f94003520 con 0x7f9fa80ccaa0 2023-10-16T14:12:41.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9fa77fe640 1 -- 172.21.15.60:0/465556707 >> v2:172.21.15.144:3300/0 conn(0x7f9fa815ae80 msgr2=0x7f9fa80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:41.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9fa77fe640 1 --2- 172.21.15.60:0/465556707 >> v2:172.21.15.144:3300/0 conn(0x7f9fa815ae80 0x7f9fa80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9fa77fe640 1 -- 172.21.15.60:0/465556707 >> v1:172.21.15.144:6789/0 conn(0x7f9fa80ccaa0 legacy=0x7f9fa8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9fa77fe640 1 -- 172.21.15.60:0/465556707 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa815ef90 con 0x7f9fa815aaa0 2023-10-16T14:12:41.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.145+0000 7f9f9ffff640 1 --2- 172.21.15.60:0/465556707 >> v2:172.21.15.144:3300/0 conn(0x7f9fa815ae80 0x7f9fa80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:41.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.146+0000 7f9fa77fe640 1 --2- 172.21.15.60:0/465556707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 0x7f9fa815e820 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009870 tx=0x7f9f9802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6299f7d0ca530127 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:41.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.147+0000 7f9fa6ffd640 1 -- 172.21.15.60:0/465556707 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98030840 con 0x7f9fa815aaa0 2023-10-16T14:12:41.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.147+0000 7f9fa6ffd640 1 -- 172.21.15.60:0/465556707 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f980309a0 con 0x7f9fa815aaa0 2023-10-16T14:12:41.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.147+0000 7f9fa6ffd640 1 -- 172.21.15.60:0/465556707 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98030d30 con 0x7f9fa815aaa0 2023-10-16T14:12:41.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.147+0000 7f9fae083640 1 -- 172.21.15.60:0/465556707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 msgr2=0x7f9fa815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.147+0000 7f9fae083640 1 --2- 172.21.15.60:0/465556707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 0x7f9fa815e820 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009870 tx=0x7f9f9802b980 comp rx=0 tx=0).stop 2023-10-16T14:12:41.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.148+0000 7f9fae083640 1 -- 172.21.15.60:0/465556707 shutdown_connections 2023-10-16T14:12:41.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.148+0000 7f9fae083640 1 --2- 172.21.15.60:0/465556707 >> v2:172.21.15.144:3300/0 conn(0x7f9fa815ae80 0x7f9fa80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.148+0000 7f9fae083640 1 --2- 172.21.15.60:0/465556707 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa815aaa0 0x7f9fa815e820 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.148+0000 7f9fae083640 1 -- 172.21.15.60:0/465556707 >> 172.21.15.60:0/465556707 conn(0x7f9fa80cb410 msgr2=0x7f9fa80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:41.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.148+0000 7f9fae083640 1 -- 172.21.15.60:0/465556707 shutdown_connections 2023-10-16T14:12:41.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.148+0000 7f9fae083640 1 -- 172.21.15.60:0/465556707 wait complete. 2023-10-16T14:12:41.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.150+0000 7f9fae083640 1 Processor -- start 2023-10-16T14:12:41.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.150+0000 7f9fae083640 1 -- start start 2023-10-16T14:12:41.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fae083640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa80ccaa0 0x7f9fa8155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fae083640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fa815aaa0 0x7f9fa815b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fae083640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 0x7f9fa81590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fae083640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8158320 con 0x7f9fa815ae80 2023-10-16T14:12:41.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fae083640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa8158600 con 0x7f9fa815aaa0 2023-10-16T14:12:41.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fae083640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa81588e0 con 0x7f9fa80ccaa0 2023-10-16T14:12:41.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fa7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 0x7f9fa81590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fa7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 0x7f9fa81590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48290/0 (socket says 172.21.15.60:48290) 2023-10-16T14:12:41.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fa7fff640 1 -- 172.21.15.60:0/4259019535 learned_addr learned my addr 172.21.15.60:0/4259019535 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:41.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9fa77fe640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa80ccaa0 0x7f9fa8155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.151+0000 7f9f9ffff640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fa815aaa0 0x7f9fa815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa7fff640 1 -- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa80ccaa0 msgr2=0x7f9fa8155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa7fff640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa80ccaa0 0x7f9fa8155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa7fff640 1 -- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fa815aaa0 msgr2=0x7f9fa815b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa7fff640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fa815aaa0 0x7f9fa815b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa7fff640 1 -- 172.21.15.60:0/4259019535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f90009400 con 0x7f9fa815ae80 2023-10-16T14:12:41.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa7fff640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 0x7f9fa81590e0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c6e0 tx=0x7f9fa000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:41.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0003b70 con 0x7f9fa815ae80 2023-10-16T14:12:41.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa0003cd0 con 0x7f9fa815ae80 2023-10-16T14:12:41.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa00052d0 con 0x7f9fa815ae80 2023-10-16T14:12:41.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f980094d0 con 0x7f9fa815ae80 2023-10-16T14:12:41.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.152+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa81563e0 con 0x7f9fa815ae80 2023-10-16T14:12:41.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.154+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8158e00 con 0x7f9fa815ae80 2023-10-16T14:12:41.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.155+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9fa0005430 con 0x7f9fa815ae80 2023-10-16T14:12:41.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.155+0000 7f9fa4ff9640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9f8806b0e0 0x7f9f8806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.156+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f9fa0089900 con 0x7f9fa815ae80 2023-10-16T14:12:41.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.156+0000 7f9fa77fe640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9f8806b0e0 0x7f9f8806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.157+0000 7f9fa77fe640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9f8806b0e0 0x7f9f8806d5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9f98036000 tx=0x7f9f98008030 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:41.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.160+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9fa8158e00 con 0x7f9fa815ae80 2023-10-16T14:12:41.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.354+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fa8158e00 con 0x7f9fa815ae80 2023-10-16T14:12:41.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.356+0000 7f9fa4ff9640 1 -- 172.21.15.60:0/4259019535 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) v1 ==== 74+0+16590 (secure 0 0 0) 0x7f9fa8158e00 con 0x7f9fa815ae80 2023-10-16T14:12:41.359 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:41.360 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":272,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:40.386200+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:41.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.359+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9f8806b0e0 msgr2=0x7f9f8806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.359+0000 7f9fae083640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9f8806b0e0 0x7f9f8806d5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9f98036000 tx=0x7f9f98008030 comp rx=0 tx=0).stop 2023-10-16T14:12:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.359+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 msgr2=0x7f9fa81590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.359+0000 7f9fae083640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 0x7f9fa81590e0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000c6e0 tx=0x7f9fa000cba0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 shutdown_connections 2023-10-16T14:12:41.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f9f8806b0e0 0x7f9f8806d5a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa815ae80 0x7f9fa81590e0 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fa815aaa0 0x7f9fa815b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 --2- 172.21.15.60:0/4259019535 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9fa80ccaa0 0x7f9fa8155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 >> 172.21.15.60:0/4259019535 conn(0x7f9fa80cb410 msgr2=0x7f9fa80ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 shutdown_connections 2023-10-16T14:12:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.360+0000 7f9fae083640 1 -- 172.21.15.60:0/4259019535 wait complete. 2023-10-16T14:12:41.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:41.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.601+0000 7f2054abb640 1 Processor -- start 2023-10-16T14:12:41.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.601+0000 7f2054abb640 1 -- start start 2023-10-16T14:12:41.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.602+0000 7f2054abb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f205006c3f0 0x7f205006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.602+0000 7f2054abb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f205015e0c0 0x7f20500b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.602+0000 7f2054abb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20500cd630 con 0x7f205015e540 2023-10-16T14:12:41.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.602+0000 7f2054abb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20500ce1b0 con 0x7f205015e0c0 2023-10-16T14:12:41.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.602+0000 7f2054abb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205006cd10 con 0x7f205006c3f0 2023-10-16T14:12:41.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f205015e0c0 0x7f20500b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f205015e0c0 0x7f20500b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48302/0 (socket says 172.21.15.60:48302) 2023-10-16T14:12:41.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 -- 172.21.15.60:0/491670612 learned_addr learned my addr 172.21.15.60:0/491670612 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:41.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204e59c640 1 --2- 172.21.15.60:0/491670612 >> v2:172.21.15.144:3300/0 conn(0x7f205006c3f0 0x7f205006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 -- 172.21.15.60:0/491670612 >> v2:172.21.15.144:3300/0 conn(0x7f205006c3f0 msgr2=0x7f205006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:41.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 --2- 172.21.15.60:0/491670612 >> v2:172.21.15.144:3300/0 conn(0x7f205006c3f0 0x7f205006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 -- 172.21.15.60:0/491670612 >> v1:172.21.15.144:6789/0 conn(0x7f205015e540 legacy=0x7f20500b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:41.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.603+0000 7f204dd9b640 1 -- 172.21.15.60:0/491670612 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20500d2170 con 0x7f205015e0c0 2023-10-16T14:12:41.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.605+0000 7f204dd9b640 1 --2- 172.21.15.60:0/491670612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205015e0c0 0x7f20500b4730 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f20500d3d00 tx=0x7f203802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d897bff143197ef8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:41.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.605+0000 7f204d59a640 1 -- 172.21.15.60:0/491670612 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038031870 con 0x7f205015e0c0 2023-10-16T14:12:41.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.605+0000 7f204d59a640 1 -- 172.21.15.60:0/491670612 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20380319d0 con 0x7f205015e0c0 2023-10-16T14:12:41.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.605+0000 7f204d59a640 1 -- 172.21.15.60:0/491670612 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038031d60 con 0x7f205015e0c0 2023-10-16T14:12:41.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 -- 172.21.15.60:0/491670612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205015e0c0 msgr2=0x7f20500b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 --2- 172.21.15.60:0/491670612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205015e0c0 0x7f20500b4730 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f20500d3d00 tx=0x7f203802b910 comp rx=0 tx=0).stop 2023-10-16T14:12:41.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 -- 172.21.15.60:0/491670612 shutdown_connections 2023-10-16T14:12:41.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 --2- 172.21.15.60:0/491670612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205015e0c0 0x7f20500b4730 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 --2- 172.21.15.60:0/491670612 >> v2:172.21.15.144:3300/0 conn(0x7f205006c3f0 0x7f205006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 -- 172.21.15.60:0/491670612 >> 172.21.15.60:0/491670612 conn(0x7f20500d2cb0 msgr2=0x7f20500d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:41.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 -- 172.21.15.60:0/491670612 shutdown_connections 2023-10-16T14:12:41.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.606+0000 7f2054abb640 1 -- 172.21.15.60:0/491670612 wait complete. 2023-10-16T14:12:41.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.608+0000 7f2054abb640 1 Processor -- start 2023-10-16T14:12:41.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.608+0000 7f2054abb640 1 -- start start 2023-10-16T14:12:41.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.609+0000 7f2054abb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 0x7f2050154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.609+0000 7f2054abb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f205015e0c0 0x7f2050154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.609+0000 7f2054abb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f205015e540 0x7f20500c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.609+0000 7f2054abb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205015a880 con 0x7f205006c3f0 2023-10-16T14:12:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.609+0000 7f2054abb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2050155740 con 0x7f205015e540 2023-10-16T14:12:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.609+0000 7f2054abb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20501599b0 con 0x7f205015e0c0 2023-10-16T14:12:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 0x7f2050154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 0x7f2050154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48304/0 (socket says 172.21.15.60:48304) 2023-10-16T14:12:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 -- 172.21.15.60:0/1322022425 learned_addr learned my addr 172.21.15.60:0/1322022425 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204dd9b640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f205015e0c0 0x7f2050154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 -- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f205015e0c0 msgr2=0x7f2050154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204ed9d640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f205015e540 0x7f20500c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f205015e0c0 0x7f2050154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 -- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f205015e540 msgr2=0x7f20500c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f205015e540 0x7f20500c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 -- 172.21.15.60:0/1322022425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20380094d0 con 0x7f205006c3f0 2023-10-16T14:12:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.610+0000 7f204e59c640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 0x7f2050154880 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f2040009d10 tx=0x7f204000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.611+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040009b80 con 0x7f205006c3f0 2023-10-16T14:12:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.611+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204000f040 con 0x7f205006c3f0 2023-10-16T14:12:41.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.611+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040014680 con 0x7f205006c3f0 2023-10-16T14:12:41.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.611+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2050159c00 con 0x7f205006c3f0 2023-10-16T14:12:41.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.611+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205015a1c0 con 0x7f205006c3f0 2023-10-16T14:12:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.612+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205004e160 con 0x7f205006c3f0 2023-10-16T14:12:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.613+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f20400147e0 con 0x7f205006c3f0 2023-10-16T14:12:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.614+0000 7f2046ffd640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f203006b0b0 0x7f203006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:41.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.614+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f204008a600 con 0x7f205006c3f0 2023-10-16T14:12:41.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.615+0000 7f204dd9b640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f203006b0b0 0x7f203006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:41.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.616+0000 7f204dd9b640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f203006b0b0 0x7f203006d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2038036000 tx=0x7f203803c040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:41.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.619+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20400552a0 con 0x7f205006c3f0 2023-10-16T14:12:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.818+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20500d0c80 con 0x7f205006c3f0 2023-10-16T14:12:41.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.820+0000 7f2046ffd640 1 -- 172.21.15.60:0/1322022425 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) v1 ==== 74+0+16590 (secure 0 0 0) 0x7f20400588c0 con 0x7f205006c3f0 2023-10-16T14:12:41.823 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:41.823 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":272,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:40.386200+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f203006b0b0 msgr2=0x7f203006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f203006b0b0 0x7f203006d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2038036000 tx=0x7f203803c040 comp rx=0 tx=0).stop 2023-10-16T14:12:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 msgr2=0x7f2050154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 0x7f2050154880 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f2040009d10 tx=0x7f204000c490 comp rx=0 tx=0).stop 2023-10-16T14:12:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 shutdown_connections 2023-10-16T14:12:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f203006b0b0 0x7f203006d570 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f205015e0c0 0x7f2050154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f205015e540 0x7f20500c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 --2- 172.21.15.60:0/1322022425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f205006c3f0 0x7f2050154880 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 >> 172.21.15.60:0/1322022425 conn(0x7f20500d2cb0 msgr2=0x7f205006ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 shutdown_connections 2023-10-16T14:12:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:41.823+0000 7f2054abb640 1 -- 172.21.15.60:0/1322022425 wait complete. 2023-10-16T14:12:41.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:12:42.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.066+0000 7f776da34640 1 Processor -- start 2023-10-16T14:12:42.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.066+0000 7f776da34640 1 -- start start 2023-10-16T14:12:42.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f776da34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f776815ad70 0x7f7768069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:42.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f776da34640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77680c9ca0 0x7f776806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:42.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f776da34640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680cb030 con 0x7f77680ca3b0 2023-10-16T14:12:42.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f776da34640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77680c91c0 con 0x7f77680c9ca0 2023-10-16T14:12:42.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f776da34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77680c94a0 con 0x7f776815ad70 2023-10-16T14:12:42.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f77667fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77680c9ca0 0x7f776806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:42.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f77667fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f77680c9ca0 0x7f776806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48320/0 (socket says 172.21.15.60:48320) 2023-10-16T14:12:42.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f77667fc640 1 -- 172.21.15.60:0/2839016990 learned_addr learned my addr 172.21.15.60:0/2839016990 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:42.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.068+0000 7f7766ffd640 1 --2- 172.21.15.60:0/2839016990 >> v2:172.21.15.144:3300/0 conn(0x7f776815ad70 0x7f7768069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:42.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.069+0000 7f77667fc640 1 -- 172.21.15.60:0/2839016990 >> v2:172.21.15.144:3300/0 conn(0x7f776815ad70 msgr2=0x7f7768069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:42.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.069+0000 7f77667fc640 1 --2- 172.21.15.60:0/2839016990 >> v2:172.21.15.144:3300/0 conn(0x7f776815ad70 0x7f7768069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.069+0000 7f77667fc640 1 -- 172.21.15.60:0/2839016990 >> v1:172.21.15.60:6789/0 conn(0x7f77680ca3b0 legacy=0x7f77680ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:42.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.069+0000 7f77667fc640 1 -- 172.21.15.60:0/2839016990 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77680c9780 con 0x7f77680c9ca0 2023-10-16T14:12:42.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.070+0000 7f77667fc640 1 --2- 172.21.15.60:0/2839016990 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77680c9ca0 0x7f776806a2e0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f77680cf780 tx=0x7f775002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf0522ed78b4ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:42.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.070+0000 7f7765ffb640 1 -- 172.21.15.60:0/2839016990 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750030860 con 0x7f77680c9ca0 2023-10-16T14:12:42.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.070+0000 7f7765ffb640 1 -- 172.21.15.60:0/2839016990 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77500309c0 con 0x7f77680c9ca0 2023-10-16T14:12:42.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.070+0000 7f7765ffb640 1 -- 172.21.15.60:0/2839016990 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750030d50 con 0x7f77680c9ca0 2023-10-16T14:12:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 -- 172.21.15.60:0/2839016990 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77680c9ca0 msgr2=0x7f776806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 --2- 172.21.15.60:0/2839016990 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77680c9ca0 0x7f776806a2e0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f77680cf780 tx=0x7f775002b9a0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 -- 172.21.15.60:0/2839016990 shutdown_connections 2023-10-16T14:12:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 --2- 172.21.15.60:0/2839016990 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77680c9ca0 0x7f776806a2e0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 --2- 172.21.15.60:0/2839016990 >> v2:172.21.15.144:3300/0 conn(0x7f776815ad70 0x7f7768069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 -- 172.21.15.60:0/2839016990 >> 172.21.15.60:0/2839016990 conn(0x7f77680ce9b0 msgr2=0x7f77680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:42.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 -- 172.21.15.60:0/2839016990 shutdown_connections 2023-10-16T14:12:42.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.071+0000 7f776da34640 1 -- 172.21.15.60:0/2839016990 wait complete. 2023-10-16T14:12:42.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.073+0000 7f776da34640 1 Processor -- start 2023-10-16T14:12:42.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.073+0000 7f776da34640 1 -- start start 2023-10-16T14:12:42.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f776da34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77680c9ca0 0x7f776815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:42.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f776da34640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77680ca3b0 0x7f776815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:42.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f776da34640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 0x7f776815c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:42.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f776da34640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776815b7c0 con 0x7f776815ad70 2023-10-16T14:12:42.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f776da34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776815cf50 con 0x7f77680c9ca0 2023-10-16T14:12:42.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f776da34640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77680c5b90 con 0x7f77680ca3b0 2023-10-16T14:12:42.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f77677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 0x7f776815c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:42.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f77677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 0x7f776815c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48326/0 (socket says 172.21.15.60:48326) 2023-10-16T14:12:42.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f77677fe640 1 -- 172.21.15.60:0/336910499 learned_addr learned my addr 172.21.15.60:0/336910499 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:42.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f7766ffd640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77680c9ca0 0x7f776815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:42.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.074+0000 7f77667fc640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77680ca3b0 0x7f776815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:42.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77677fe640 1 -- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77680ca3b0 msgr2=0x7f776815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:42.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77677fe640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77680ca3b0 0x7f776815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77677fe640 1 -- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77680c9ca0 msgr2=0x7f776815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:42.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77677fe640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77680c9ca0 0x7f776815d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77677fe640 1 -- 172.21.15.60:0/336910499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77500094d0 con 0x7f776815ad70 2023-10-16T14:12:42.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77667fc640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77680ca3b0 0x7f776815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:42.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f77677fe640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 0x7f776815c8b0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f776000c7a0 tx=0x7f776000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:42.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760003b70 con 0x7f776815ad70 2023-10-16T14:12:42.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760003cf0 con 0x7f776815ad70 2023-10-16T14:12:42.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760003ea0 con 0x7f776815ad70 2023-10-16T14:12:42.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77680c5e70 con 0x7f776815ad70 2023-10-16T14:12:42.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.075+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77680c6430 con 0x7f776815ad70 2023-10-16T14:12:42.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.077+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77680cb220 con 0x7f776815ad70 2023-10-16T14:12:42.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.077+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f776000aa90 con 0x7f776815ad70 2023-10-16T14:12:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.078+0000 7f775ffff640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f774806b0e0 0x7f774806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.078+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f7760089930 con 0x7f776815ad70 2023-10-16T14:12:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.079+0000 7f7766ffd640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f774806b0e0 0x7f774806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.080+0000 7f7766ffd640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f774806b0e0 0x7f774806d5a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7758006fd0 tx=0x7f7758008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:42.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.083+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f77600545b0 con 0x7f776815ad70 2023-10-16T14:12:42.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.318+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f776804e160 con 0x7f776815ad70 2023-10-16T14:12:42.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.945+0000 7f775ffff640 1 -- 172.21.15.60:0/336910499 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v143) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7760057bd0 con 0x7f776815ad70 2023-10-16T14:12:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f774806b0e0 msgr2=0x7f774806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f774806b0e0 0x7f774806d5a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7758006fd0 tx=0x7f7758008040 comp rx=0 tx=0).stop 2023-10-16T14:12:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 msgr2=0x7f776815c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 0x7f776815c8b0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f776000c7a0 tx=0x7f776000cc60 comp rx=0 tx=0).stop 2023-10-16T14:12:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 shutdown_connections 2023-10-16T14:12:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f774806b0e0 0x7f774806d5a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f77680ca3b0 0x7f776815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f776815ad70 0x7f776815c8b0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 --2- 172.21.15.60:0/336910499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f77680c9ca0 0x7f776815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 >> 172.21.15.60:0/336910499 conn(0x7f77680ce9b0 msgr2=0x7f77680acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.949+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 shutdown_connections 2023-10-16T14:12:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:42.950+0000 7f776da34640 1 -- 172.21.15.60:0/336910499 wait complete. 2023-10-16T14:12:42.973 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:12:42.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:12:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.204+0000 7ff7bf59e640 1 Processor -- start 2023-10-16T14:12:43.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.204+0000 7ff7bf59e640 1 -- start start 2023-10-16T14:12:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.205+0000 7ff7bf59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7c0159950 0x7ff7c015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.205+0000 7ff7bf59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7c006c810 0x7ff7c00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.205+0000 7ff7bf59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c015aef0 con 0x7ff7c0070450 2023-10-16T14:12:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.205+0000 7ff7bf59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c0070e20 con 0x7ff7c0159950 2023-10-16T14:12:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.205+0000 7ff7bf59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c015e490 con 0x7ff7c006c810 2023-10-16T14:12:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7c0159950 0x7ff7c015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7c0159950 0x7ff7c015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48336/0 (socket says 172.21.15.60:48336) 2023-10-16T14:12:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 -- 172.21.15.60:0/1346435568 learned_addr learned my addr 172.21.15.60:0/1346435568 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 -- 172.21.15.60:0/1346435568 >> v2:172.21.15.144:3300/0 conn(0x7ff7c006c810 msgr2=0x7ff7c00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 --2- 172.21.15.60:0/1346435568 >> v2:172.21.15.144:3300/0 conn(0x7ff7c006c810 0x7ff7c00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 -- 172.21.15.60:0/1346435568 >> v1:172.21.15.144:6789/0 conn(0x7ff7c0070450 legacy=0x7ff7c006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:43.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.206+0000 7ff7be59c640 1 -- 172.21.15.60:0/1346435568 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c015d910 con 0x7ff7c0159950 2023-10-16T14:12:43.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.207+0000 7ff7be59c640 1 --2- 172.21.15.60:0/1346435568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0159950 0x7ff7c015a9b0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00716a0 tx=0x7ff7b002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e774efcdc1bb52 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:43.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.208+0000 7ff7bd59a640 1 -- 172.21.15.60:0/1346435568 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b002c9f0 con 0x7ff7c0159950 2023-10-16T14:12:43.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.208+0000 7ff7bd59a640 1 -- 172.21.15.60:0/1346435568 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b002cb50 con 0x7ff7c0159950 2023-10-16T14:12:43.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.208+0000 7ff7bd59a640 1 -- 172.21.15.60:0/1346435568 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b00343f0 con 0x7ff7c0159950 2023-10-16T14:12:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.208+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1346435568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0159950 msgr2=0x7ff7c015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.208+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1346435568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0159950 0x7ff7c015a9b0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00716a0 tx=0x7ff7b002be50 comp rx=0 tx=0).stop 2023-10-16T14:12:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.209+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1346435568 shutdown_connections 2023-10-16T14:12:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.209+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1346435568 >> v2:172.21.15.144:3300/0 conn(0x7ff7c006c810 0x7ff7c00708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.209+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1346435568 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0159950 0x7ff7c015a9b0 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.209+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1346435568 >> 172.21.15.60:0/1346435568 conn(0x7ff7c00ce6a0 msgr2=0x7ff7c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:43.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.209+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1346435568 shutdown_connections 2023-10-16T14:12:43.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.209+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1346435568 wait complete. 2023-10-16T14:12:43.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.210+0000 7ff7bf59e640 1 Processor -- start 2023-10-16T14:12:43.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.211+0000 7ff7bf59e640 1 -- start start 2023-10-16T14:12:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.211+0000 7ff7bf59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c006c810 0x7ff7c0148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bf59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 0x7ff7c00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bf59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c0159950 0x7ff7c015c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bf59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00c2810 con 0x7ff7c0070450 2023-10-16T14:12:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bf59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b1bc0 con 0x7ff7c006c810 2023-10-16T14:12:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bf59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b1ea0 con 0x7ff7c0159950 2023-10-16T14:12:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 0x7ff7c00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 0x7ff7c00b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48340/0 (socket says 172.21.15.60:48340) 2023-10-16T14:12:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 -- 172.21.15.60:0/1277791554 learned_addr learned my addr 172.21.15.60:0/1277791554 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7be59c640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c006c810 0x7ff7c0148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:43.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bed9d640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c0159950 0x7ff7c015c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:43.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 -- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c0159950 msgr2=0x7ff7c015c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:43.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c0159950 0x7ff7c015c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 -- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c006c810 msgr2=0x7ff7c0148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:43.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c006c810 0x7ff7c0148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bdd9b640 1 -- 172.21.15.60:0/1277791554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b0009b30 con 0x7ff7c0070450 2023-10-16T14:12:43.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.212+0000 7ff7bed9d640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c0159950 0x7ff7c015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:43.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.213+0000 7ff7bdd9b640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 0x7ff7c00b12c0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800c410 tx=0x7ff7a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:43.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.213+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80035f0 con 0x7ff7c0070450 2023-10-16T14:12:43.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.213+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a8003750 con 0x7ff7c0070450 2023-10-16T14:12:43.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.213+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8003b40 con 0x7ff7c0070450 2023-10-16T14:12:43.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.213+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c00b2180 con 0x7ff7c0070450 2023-10-16T14:12:43.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.213+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c0148f70 con 0x7ff7c0070450 2023-10-16T14:12:43.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.214+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c00c29e0 con 0x7ff7c0070450 2023-10-16T14:12:43.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.215+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff7a8003ca0 con 0x7ff7c0070450 2023-10-16T14:12:43.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.216+0000 7ff7b6ffd640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff7a006b0e0 0x7ff7a006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:43.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.216+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7ff7a8088c40 con 0x7ff7c0070450 2023-10-16T14:12:43.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.217+0000 7ff7be59c640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff7a006b0e0 0x7ff7a006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:43.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.218+0000 7ff7be59c640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff7a006b0e0 0x7ff7a006d5a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0035000 tx=0x7ff7b0030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:43.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.221+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff7c00c29e0 con 0x7ff7c0070450 2023-10-16T14:12:43.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.459+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff7c00c29e0 con 0x7ff7c0070450 2023-10-16T14:12:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.959+0000 7ff7b6ffd640 1 -- 172.21.15.60:0/1277791554 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v144) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff7c00c29e0 con 0x7ff7c0070450 2023-10-16T14:12:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff7a006b0e0 msgr2=0x7ff7a006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff7a006b0e0 0x7ff7a006d5a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0035000 tx=0x7ff7b0030000 comp rx=0 tx=0).stop 2023-10-16T14:12:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 msgr2=0x7ff7c00b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 0x7ff7c00b12c0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800c410 tx=0x7ff7a800c8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 shutdown_connections 2023-10-16T14:12:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7ff7a006b0e0 0x7ff7a006d5a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff7c0159950 0x7ff7c015c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c0070450 0x7ff7c00b12c0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 --2- 172.21.15.60:0/1277791554 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c006c810 0x7ff7c0148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 >> 172.21.15.60:0/1277791554 conn(0x7ff7c00ce6a0 msgr2=0x7ff7c00c9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 shutdown_connections 2023-10-16T14:12:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:43.963+0000 7ff7bf59e640 1 -- 172.21.15.60:0/1277791554 wait complete. 2023-10-16T14:12:43.990 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:12:43.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:44.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.228+0000 7fdfabe00640 1 Processor -- start 2023-10-16T14:12:44.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.228+0000 7fdfabe00640 1 -- start start 2023-10-16T14:12:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfabe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdfa40ce390 0x7fdfa415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfabe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa415ed60 con 0x7fdfa415aaa0 2023-10-16T14:12:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfabe00640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa40707e0 con 0x7fdfa415ae80 2023-10-16T14:12:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfabe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa406b810 con 0x7fdfa40ce390 2023-10-16T14:12:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfa9b9c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdfa415aaa0 0x7fdfa40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50938/0 (socket says 172.21.15.60:50938) 2023-10-16T14:12:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfa9b9c640 1 -- 172.21.15.60:0/1519587103 learned_addr learned my addr 172.21.15.60:0/1519587103 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.230+0000 7fdfaa39d640 1 --2- 172.21.15.60:0/1519587103 >> v2:172.21.15.60:3300/0 conn(0x7fdfa40ce390 0x7fdfa415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.231+0000 7fdfaa39d640 1 -- 172.21.15.60:0/1519587103 >> v1:172.21.15.144:6789/0 conn(0x7fdfa415ae80 legacy=0x7fdfa4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.231+0000 7fdfaa39d640 1 -- 172.21.15.60:0/1519587103 >> v1:172.21.15.60:6789/0 conn(0x7fdfa415aaa0 legacy=0x7fdfa40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.231+0000 7fdfaa39d640 1 -- 172.21.15.60:0/1519587103 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa415ef90 con 0x7fdfa40ce390 2023-10-16T14:12:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.232+0000 7fdfaa39d640 1 --2- 172.21.15.60:0/1519587103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa40ce390 0x7fdfa415e820 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009d90 tx=0x7fdf9c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c467f109f6887b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.232+0000 7fdfa8b9a640 1 -- 172.21.15.60:0/1519587103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c02cc60 con 0x7fdfa40ce390 2023-10-16T14:12:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.232+0000 7fdfa8b9a640 1 -- 172.21.15.60:0/1519587103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c02cdc0 con 0x7fdfa40ce390 2023-10-16T14:12:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.232+0000 7fdfa8b9a640 1 -- 172.21.15.60:0/1519587103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c034660 con 0x7fdfa40ce390 2023-10-16T14:12:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 -- 172.21.15.60:0/1519587103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa40ce390 msgr2=0x7fdfa415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1519587103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa40ce390 0x7fdfa415e820 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009d90 tx=0x7fdf9c02beb0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 -- 172.21.15.60:0/1519587103 shutdown_connections 2023-10-16T14:12:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1519587103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa40ce390 0x7fdfa415e820 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 -- 172.21.15.60:0/1519587103 >> 172.21.15.60:0/1519587103 conn(0x7fdfa40cb410 msgr2=0x7fdfa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 -- 172.21.15.60:0/1519587103 shutdown_connections 2023-10-16T14:12:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.233+0000 7fdfabe00640 1 -- 172.21.15.60:0/1519587103 wait complete. 2023-10-16T14:12:44.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.235+0000 7fdfabe00640 1 Processor -- start 2023-10-16T14:12:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.235+0000 7fdfabe00640 1 -- start start 2023-10-16T14:12:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfabe00640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfa40ce390 0x7fdfa415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfabe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 0x7fdfa41f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfabe00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfa415ae80 0x7fdfa41f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfabe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa415e200 con 0x7fdfa415aaa0 2023-10-16T14:12:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfabe00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa415d160 con 0x7fdfa415ae80 2023-10-16T14:12:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfabe00640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa415d440 con 0x7fdfa40ce390 2023-10-16T14:12:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfa939b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 0x7fdfa41f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfa939b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 0x7fdfa41f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48360/0 (socket says 172.21.15.60:48360) 2023-10-16T14:12:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.236+0000 7fdfa939b640 1 -- 172.21.15.60:0/1435581977 learned_addr learned my addr 172.21.15.60:0/1435581977 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfaa39d640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfa415ae80 0x7fdfa41f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa9b9c640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfa40ce390 0x7fdfa415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa939b640 1 -- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfa40ce390 msgr2=0x7fdfa415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa939b640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfa40ce390 0x7fdfa415c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa939b640 1 -- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfa415ae80 msgr2=0x7fdfa41f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa939b640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfa415ae80 0x7fdfa41f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa939b640 1 -- 172.21.15.60:0/1435581977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c009a70 con 0x7fdfa415aaa0 2023-10-16T14:12:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa9b9c640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfa40ce390 0x7fdfa415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfaa39d640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfa415ae80 0x7fdfa41f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfa939b640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 0x7fdfa41f2c10 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400b3c0 tx=0x7fdf9400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94011830 con 0x7fdfa415aaa0 2023-10-16T14:12:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf94011990 con 0x7fdfa415aaa0 2023-10-16T14:12:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94011d80 con 0x7fdfa415aaa0 2023-10-16T14:12:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.237+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa415d720 con 0x7fdfa415aaa0 2023-10-16T14:12:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.238+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa415dce0 con 0x7fdfa415aaa0 2023-10-16T14:12:44.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.239+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa415df00 con 0x7fdfa415aaa0 2023-10-16T14:12:44.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.240+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdf9401b460 con 0x7fdfa415aaa0 2023-10-16T14:12:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.241+0000 7fdf9a7fc640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdf8806b0b0 0x7fdf8806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.241+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fdf94089590 con 0x7fdfa415aaa0 2023-10-16T14:12:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.241+0000 7fdfa9b9c640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdf8806b0b0 0x7fdf8806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.243+0000 7fdfa9b9c640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdf8806b0b0 0x7fdf8806d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0005880 tx=0x7fdfa0009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:44.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.245+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf940542a0 con 0x7fdfa415aaa0 2023-10-16T14:12:44.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.480+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa404e160 con 0x7fdfa415aaa0 2023-10-16T14:12:44.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.481+0000 7fdf9a7fc640 1 -- 172.21.15.60:0/1435581977 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4933 (secure 0 0 0) 0x7fdf940578c0 con 0x7fdfa415aaa0 2023-10-16T14:12:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:44.485 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:12:40.387056+0000","modified":"2023-10-16T14:12:43.460423+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12573":{"gid":12573,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2674455624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2674455624},{"type":"v1","addr":"172.21.15.60:6835","nonce":2674455624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T14:12:44.485 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 144 2023-10-16T14:12:44.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdf8806b0b0 msgr2=0x7fdf8806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdf8806b0b0 0x7fdf8806d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0005880 tx=0x7fdfa0009040 comp rx=0 tx=0).stop 2023-10-16T14:12:44.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 msgr2=0x7fdfa41f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 0x7fdfa41f2c10 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400b3c0 tx=0x7fdf9400b880 comp rx=0 tx=0).stop 2023-10-16T14:12:44.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 shutdown_connections 2023-10-16T14:12:44.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fdf8806b0b0 0x7fdf8806d570 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfa415ae80 0x7fdfa41f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfa415aaa0 0x7fdfa41f2c10 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 --2- 172.21.15.60:0/1435581977 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdfa40ce390 0x7fdfa415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 >> 172.21.15.60:0/1435581977 conn(0x7fdfa40cb410 msgr2=0x7fdfa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 shutdown_connections 2023-10-16T14:12:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.484+0000 7fdfabe00640 1 -- 172.21.15.60:0/1435581977 wait complete. 2023-10-16T14:12:44.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:12:44.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.729+0000 7f2fd030c640 1 Processor -- start 2023-10-16T14:12:44.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.729+0000 7f2fd030c640 1 -- start start 2023-10-16T14:12:44.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fd030c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fc8159b20 0x7f2fc806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fd030c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc80b0b20 0x7f2fc806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fd030c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc806f7f0 con 0x7f2fc8159f00 2023-10-16T14:12:44.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fd030c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc806c5e0 con 0x7f2fc8159b20 2023-10-16T14:12:44.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fd030c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc806c8c0 con 0x7f2fc80b0b20 2023-10-16T14:12:44.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fce0a8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fc8159b20 0x7f2fc806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fce0a8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fc8159b20 0x7f2fc806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48374/0 (socket says 172.21.15.60:48374) 2023-10-16T14:12:44.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.730+0000 7f2fce0a8640 1 -- 172.21.15.60:0/1494184014 learned_addr learned my addr 172.21.15.60:0/1494184014 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:44.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.731+0000 7f2fce0a8640 1 -- 172.21.15.60:0/1494184014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc80b0b20 msgr2=0x7f2fc806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:44.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.731+0000 7f2fce8a9640 1 --2- 172.21.15.60:0/1494184014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc80b0b20 0x7f2fc806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.731+0000 7f2fce0a8640 1 --2- 172.21.15.60:0/1494184014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc80b0b20 0x7f2fc806fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.731+0000 7f2fce0a8640 1 -- 172.21.15.60:0/1494184014 >> v1:172.21.15.60:6789/0 conn(0x7f2fc8159f00 legacy=0x7f2fc80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.731+0000 7f2fce0a8640 1 -- 172.21.15.60:0/1494184014 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc806cba0 con 0x7f2fc8159b20 2023-10-16T14:12:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.732+0000 7f2fce0a8640 1 --2- 172.21.15.60:0/1494184014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc8159b20 0x7f2fc806bbe0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009e50 tx=0x7f2fc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c15689032b3a0bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fcd0a6640 1 -- 172.21.15.60:0/1494184014 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc402cb00 con 0x7f2fc8159b20 2023-10-16T14:12:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fcd0a6640 1 -- 172.21.15.60:0/1494184014 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc402cc60 con 0x7f2fc8159b20 2023-10-16T14:12:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fcd0a6640 1 -- 172.21.15.60:0/1494184014 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40343f0 con 0x7f2fc8159b20 2023-10-16T14:12:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 -- 172.21.15.60:0/1494184014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc8159b20 msgr2=0x7f2fc806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 --2- 172.21.15.60:0/1494184014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc8159b20 0x7f2fc806bbe0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009e50 tx=0x7f2fc402bf60 comp rx=0 tx=0).stop 2023-10-16T14:12:44.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 -- 172.21.15.60:0/1494184014 shutdown_connections 2023-10-16T14:12:44.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 --2- 172.21.15.60:0/1494184014 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc80b0b20 0x7f2fc806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 --2- 172.21.15.60:0/1494184014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc8159b20 0x7f2fc806bbe0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 -- 172.21.15.60:0/1494184014 >> 172.21.15.60:0/1494184014 conn(0x7f2fc80cb410 msgr2=0x7f2fc80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:44.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 -- 172.21.15.60:0/1494184014 shutdown_connections 2023-10-16T14:12:44.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.733+0000 7f2fd030c640 1 -- 172.21.15.60:0/1494184014 wait complete. 2023-10-16T14:12:44.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.735+0000 7f2fd030c640 1 Processor -- start 2023-10-16T14:12:44.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.735+0000 7f2fd030c640 1 -- start start 2023-10-16T14:12:44.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.736+0000 7f2fd030c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 0x7f2fc80b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fd030c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc8159b20 0x7f2fc8151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fd030c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fc8159f00 0x7f2fc8152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fd030c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815bec0 con 0x7f2fc80b0b20 2023-10-16T14:12:44.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fd030c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815b240 con 0x7f2fc8159f00 2023-10-16T14:12:44.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fd030c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc815b520 con 0x7f2fc8159b20 2023-10-16T14:12:44.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 0x7f2fc80b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 0x7f2fc80b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48390/0 (socket says 172.21.15.60:48390) 2023-10-16T14:12:44.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 -- 172.21.15.60:0/2255549484 learned_addr learned my addr 172.21.15.60:0/2255549484 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:44.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 -- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc8159b20 msgr2=0x7f2fc8151af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc8159b20 0x7f2fc8151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 -- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fc8159f00 msgr2=0x7f2fc8152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fc8159f00 0x7f2fc8152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.737+0000 7f2fce0a8640 1 -- 172.21.15.60:0/2255549484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4009b30 con 0x7f2fc80b0b20 2023-10-16T14:12:44.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.738+0000 7f2fce0a8640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 0x7f2fc80b3a00 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4036040 tx=0x7f2fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:44.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.738+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc403e450 con 0x7f2fc80b0b20 2023-10-16T14:12:44.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.738+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc403e5f0 con 0x7f2fc80b0b20 2023-10-16T14:12:44.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.738+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc403e9a0 con 0x7f2fc80b0b20 2023-10-16T14:12:44.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.738+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc815b800 con 0x7f2fc80b0b20 2023-10-16T14:12:44.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.738+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc815c2e0 con 0x7f2fc80b0b20 2023-10-16T14:12:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.739+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc815c000 con 0x7f2fc80b0b20 2023-10-16T14:12:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.740+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2fc403eb00 con 0x7f2fc80b0b20 2023-10-16T14:12:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.741+0000 7f2fbeffd640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2fac06b0b0 0x7f2fac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:44.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.741+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f2fc40acbc0 con 0x7f2fc80b0b20 2023-10-16T14:12:44.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.742+0000 7f2fcd8a7640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2fac06b0b0 0x7f2fac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:44.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.743+0000 7f2fcd8a7640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2fac06b0b0 0x7f2fac06d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8000f60 tx=0x7f2fb80079d0 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:44.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.746+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fc815c000 con 0x7f2fc80b0b20 2023-10-16T14:12:44.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.936+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fc815c000 con 0x7f2fc80b0b20 2023-10-16T14:12:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.939+0000 7f2fbeffd640 1 -- 172.21.15.60:0/2255549484 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) v1 ==== 74+0+16577 (secure 0 0 0) 0x7f2fc815c000 con 0x7f2fc80b0b20 2023-10-16T14:12:44.943 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:44.943 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":273,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:41.919465+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:12:44.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.941+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2fac06b0b0 msgr2=0x7f2fac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.941+0000 7f2fd030c640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2fac06b0b0 0x7f2fac06d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8000f60 tx=0x7f2fb80079d0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.941+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 msgr2=0x7f2fc80b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:44.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.941+0000 7f2fd030c640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 0x7f2fc80b3a00 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4036040 tx=0x7f2fc4004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:44.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.941+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 shutdown_connections 2023-10-16T14:12:44.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.941+0000 7f2fd030c640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f2fac06b0b0 0x7f2fac06d570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.942+0000 7f2fd030c640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fc8159b20 0x7f2fc8151af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.942+0000 7f2fd030c640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fc8159f00 0x7f2fc8152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.942+0000 7f2fd030c640 1 --2- 172.21.15.60:0/2255549484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fc80b0b20 0x7f2fc80b3a00 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.942+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 >> 172.21.15.60:0/2255549484 conn(0x7f2fc80cb410 msgr2=0x7f2fc80704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.942+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 shutdown_connections 2023-10-16T14:12:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:44.942+0000 7f2fd030c640 1 -- 172.21.15.60:0/2255549484 wait complete. 2023-10-16T14:12:44.966 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:12:44.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:12:45.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.184+0000 7facfb59e640 1 Processor -- start 2023-10-16T14:12:45.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.184+0000 7facfb59e640 1 -- start start 2023-10-16T14:12:45.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfb59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7facfc06c040 0x7facfc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfb59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facfc0cda80 0x7facfc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfb59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc06c960 con 0x7facfc0ad970 2023-10-16T14:12:45.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfb59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15b9d0 con 0x7facfc0cda80 2023-10-16T14:12:45.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfb59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15bcb0 con 0x7facfc06c040 2023-10-16T14:12:45.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facfc0cda80 0x7facfc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facfc0cda80 0x7facfc15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48400/0 (socket says 172.21.15.60:48400) 2023-10-16T14:12:45.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfad9d640 1 -- 172.21.15.60:0/3731806318 learned_addr learned my addr 172.21.15.60:0/3731806318 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:45.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.186+0000 7facfa59c640 1 --2- 172.21.15.60:0/3731806318 >> v2:172.21.15.144:3300/0 conn(0x7facfc06c040 0x7facfc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.187+0000 7facfad9d640 1 -- 172.21.15.60:0/3731806318 >> v2:172.21.15.144:3300/0 conn(0x7facfc06c040 msgr2=0x7facfc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:45.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.187+0000 7facfad9d640 1 --2- 172.21.15.60:0/3731806318 >> v2:172.21.15.144:3300/0 conn(0x7facfc06c040 0x7facfc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.187+0000 7facfad9d640 1 -- 172.21.15.60:0/3731806318 >> v1:172.21.15.60:6789/0 conn(0x7facfc0ad970 legacy=0x7facfc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.187+0000 7facfad9d640 1 -- 172.21.15.60:0/3731806318 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc15bf90 con 0x7facfc0cda80 2023-10-16T14:12:45.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.188+0000 7facfad9d640 1 --2- 172.21.15.60:0/3731806318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc15ed90 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7facfc0cf350 tx=0x7facf402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b94dc15c5bea4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:45.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.188+0000 7facf9d9b640 1 -- 172.21.15.60:0/3731806318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf402cb10 con 0x7facfc0cda80 2023-10-16T14:12:45.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.188+0000 7facf9d9b640 1 -- 172.21.15.60:0/3731806318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf402cc70 con 0x7facfc0cda80 2023-10-16T14:12:45.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.188+0000 7facf9d9b640 1 -- 172.21.15.60:0/3731806318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf40343f0 con 0x7facfc0cda80 2023-10-16T14:12:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 -- 172.21.15.60:0/3731806318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 msgr2=0x7facfc15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 --2- 172.21.15.60:0/3731806318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc15ed90 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7facfc0cf350 tx=0x7facf402bf70 comp rx=0 tx=0).stop 2023-10-16T14:12:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 -- 172.21.15.60:0/3731806318 shutdown_connections 2023-10-16T14:12:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 --2- 172.21.15.60:0/3731806318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc15ed90 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 --2- 172.21.15.60:0/3731806318 >> v2:172.21.15.144:3300/0 conn(0x7facfc06c040 0x7facfc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 -- 172.21.15.60:0/3731806318 >> 172.21.15.60:0/3731806318 conn(0x7facfc0ce6a0 msgr2=0x7facfc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:45.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 -- 172.21.15.60:0/3731806318 shutdown_connections 2023-10-16T14:12:45.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.189+0000 7facfb59e640 1 -- 172.21.15.60:0/3731806318 wait complete. 2023-10-16T14:12:45.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.191+0000 7facfb59e640 1 Processor -- start 2023-10-16T14:12:45.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.191+0000 7facfb59e640 1 -- start start 2023-10-16T14:12:45.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.192+0000 7facfb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facfc06c040 0x7facfc14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.192+0000 7facfb59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facfc0ad970 0x7facfc0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.192+0000 7facfb59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc06fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.192+0000 7facfb59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc150290 con 0x7facfc0cda80 2023-10-16T14:12:45.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.192+0000 7facfb59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0b2970 con 0x7facfc06c040 2023-10-16T14:12:45.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.192+0000 7facfb59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0b2c50 con 0x7facfc0ad970 2023-10-16T14:12:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc06fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc06fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48402/0 (socket says 172.21.15.60:48402) 2023-10-16T14:12:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 -- 172.21.15.60:0/2147134384 learned_addr learned my addr 172.21.15.60:0/2147134384 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facf3fff640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facfc0ad970 0x7facfc0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfa59c640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facfc06c040 0x7facfc14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 -- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facfc0ad970 msgr2=0x7facfc0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facfc0ad970 0x7facfc0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 -- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facfc06c040 msgr2=0x7facfc14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facfc06c040 0x7facfc14fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 -- 172.21.15.60:0/2147134384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4009b30 con 0x7facfc0cda80 2023-10-16T14:12:45.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facf3fff640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facfc0ad970 0x7facfc0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:45.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.193+0000 7facfad9d640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc06fec0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7facf4036040 tx=0x7facf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:45.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.194+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4009510 con 0x7facfc0cda80 2023-10-16T14:12:45.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.194+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf403e450 con 0x7facfc0cda80 2023-10-16T14:12:45.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.194+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf403e7e0 con 0x7facfc0cda80 2023-10-16T14:12:45.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.194+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facfc0b2f30 con 0x7facfc0cda80 2023-10-16T14:12:45.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.194+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facfc0b34f0 con 0x7facfc0cda80 2023-10-16T14:12:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.195+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facfc06b330 con 0x7facfc0cda80 2023-10-16T14:12:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.196+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7facf403e940 con 0x7facfc0cda80 2023-10-16T14:12:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.197+0000 7facf2ffd640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7facdc06b0e0 0x7facdc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.197+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7facf40ac540 con 0x7facfc0cda80 2023-10-16T14:12:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.198+0000 7facfa59c640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7facdc06b0e0 0x7facdc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.199+0000 7facfa59c640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7facdc06b0e0 0x7facdc06d5a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7facec006fd0 tx=0x7facec008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.201+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facf4077260 con 0x7facfc0cda80 2023-10-16T14:12:45.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.396+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 --> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facfc0cc550 con 0x7facdc06b0e0 2023-10-16T14:12:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.426+0000 7facf2ffd640 1 -- 172.21.15.60:0/2147134384 <== mgr.8393 v2:172.21.15.144:6832/948422863 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286891 (secure 0 0 0) 0x7facec00f860 con 0x7facdc06b0e0 2023-10-16T14:12:45.433 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:12:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7facdc06b0e0 msgr2=0x7facdc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7facdc06b0e0 0x7facdc06d5a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7facec006fd0 tx=0x7facec008040 comp rx=0 tx=0).stop 2023-10-16T14:12:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 msgr2=0x7facfc06fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc06fec0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7facf4036040 tx=0x7facf4004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 shutdown_connections 2023-10-16T14:12:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7facdc06b0e0 0x7facdc06d5a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facfc0ad970 0x7facfc0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facfc0cda80 0x7facfc06fec0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 --2- 172.21.15.60:0/2147134384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facfc06c040 0x7facfc14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 >> 172.21.15.60:0/2147134384 conn(0x7facfc0ce6a0 msgr2=0x7facfc0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 shutdown_connections 2023-10-16T14:12:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.432+0000 7facfb59e640 1 -- 172.21.15.60:0/2147134384 wait complete. 2023-10-16T14:12:45.455 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T14:12:43.499477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2507,"num_read_kb":5248,"num_write":693,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2837,"ondisk_log_size":2837,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":439872,"kb_used_data":28056,"kb_used_omap":12,"kb_used_meta":411699,"kb_avail":754534848,"statfs":{"total":773094113280,"available":772643684352,"internally_reserved":0,"allocated":28729344,"data_stored":14697008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":421580344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":201,"num_write":63,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.617638"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947804+0000","last_change":"2023-10-16T14:12:40.942796+0000","last_active":"2023-10-16T14:12:40.947804+0000","last_peered":"2023-10-16T14:12:40.947804+0000","last_clean":"2023-10-16T14:12:40.947804+0000","last_became_active":"2023-10-16T14:12:40.942519+0000","last_became_peered":"2023-10-16T14:12:40.942519+0000","last_unstale":"2023-10-16T14:12:40.947804+0000","last_undegraded":"2023-10-16T14:12:40.947804+0000","last_fullsized":"2023-10-16T14:12:40.947804+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:34.366625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940966+0000","last_change":"2023-10-16T14:12:40.936194+0000","last_active":"2023-10-16T14:12:40.940966+0000","last_peered":"2023-10-16T14:12:40.940966+0000","last_clean":"2023-10-16T14:12:40.940966+0000","last_became_active":"2023-10-16T14:12:40.934991+0000","last_became_peered":"2023-10-16T14:12:40.934991+0000","last_unstale":"2023-10-16T14:12:40.940966+0000","last_undegraded":"2023-10-16T14:12:40.940966+0000","last_fullsized":"2023-10-16T14:12:40.940966+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:59.307365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943144+0000","last_change":"2023-10-16T14:12:40.940529+0000","last_active":"2023-10-16T14:12:40.943144+0000","last_peered":"2023-10-16T14:12:40.943144+0000","last_clean":"2023-10-16T14:12:40.943144+0000","last_became_active":"2023-10-16T14:12:40.940276+0000","last_became_peered":"2023-10-16T14:12:40.940276+0000","last_unstale":"2023-10-16T14:12:40.943144+0000","last_undegraded":"2023-10-16T14:12:40.943144+0000","last_fullsized":"2023-10-16T14:12:40.943144+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:34.111470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947686+0000","last_change":"2023-10-16T14:12:40.941002+0000","last_active":"2023-10-16T14:12:41.947686+0000","last_peered":"2023-10-16T14:12:41.947686+0000","last_clean":"2023-10-16T14:12:41.947686+0000","last_became_active":"2023-10-16T14:12:40.940579+0000","last_became_peered":"2023-10-16T14:12:40.940579+0000","last_unstale":"2023-10-16T14:12:41.947686+0000","last_undegraded":"2023-10-16T14:12:41.947686+0000","last_fullsized":"2023-10-16T14:12:41.947686+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:55.325274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.944007+0000","last_change":"2023-10-16T14:12:40.940944+0000","last_active":"2023-10-16T14:12:40.944007+0000","last_peered":"2023-10-16T14:12:40.944007+0000","last_clean":"2023-10-16T14:12:40.944007+0000","last_became_active":"2023-10-16T14:12:40.939102+0000","last_became_peered":"2023-10-16T14:12:40.939102+0000","last_unstale":"2023-10-16T14:12:40.944007+0000","last_undegraded":"2023-10-16T14:12:40.944007+0000","last_fullsized":"2023-10-16T14:12:40.944007+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:46.139470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943077+0000","last_change":"2023-10-16T14:12:40.940326+0000","last_active":"2023-10-16T14:12:40.943077+0000","last_peered":"2023-10-16T14:12:40.943077+0000","last_clean":"2023-10-16T14:12:40.943077+0000","last_became_active":"2023-10-16T14:12:40.940031+0000","last_became_peered":"2023-10-16T14:12:40.940031+0000","last_unstale":"2023-10-16T14:12:40.943077+0000","last_undegraded":"2023-10-16T14:12:40.943077+0000","last_fullsized":"2023-10-16T14:12:40.943077+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:33.995995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954502+0000","last_change":"2023-10-16T14:12:40.953749+0000","last_active":"2023-10-16T14:12:40.954502+0000","last_peered":"2023-10-16T14:12:40.954502+0000","last_clean":"2023-10-16T14:12:40.954502+0000","last_became_active":"2023-10-16T14:12:40.953178+0000","last_became_peered":"2023-10-16T14:12:40.953178+0000","last_unstale":"2023-10-16T14:12:40.954502+0000","last_undegraded":"2023-10-16T14:12:40.954502+0000","last_fullsized":"2023-10-16T14:12:40.954502+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:47.225375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.942290+0000","last_change":"2023-10-16T14:12:40.935382+0000","last_active":"2023-10-16T14:12:40.942290+0000","last_peered":"2023-10-16T14:12:40.942290+0000","last_clean":"2023-10-16T14:12:40.942290+0000","last_became_active":"2023-10-16T14:12:40.934479+0000","last_became_peered":"2023-10-16T14:12:40.934479+0000","last_unstale":"2023-10-16T14:12:40.942290+0000","last_undegraded":"2023-10-16T14:12:40.942290+0000","last_fullsized":"2023-10-16T14:12:40.942290+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:02:13.601816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937912+0000","last_change":"2023-10-16T14:12:40.942033+0000","last_active":"2023-10-16T14:12:41.937912+0000","last_peered":"2023-10-16T14:12:41.937912+0000","last_clean":"2023-10-16T14:12:41.937912+0000","last_became_active":"2023-10-16T14:12:40.941750+0000","last_became_peered":"2023-10-16T14:12:40.941750+0000","last_unstale":"2023-10-16T14:12:41.937912+0000","last_undegraded":"2023-10-16T14:12:41.937912+0000","last_fullsized":"2023-10-16T14:12:41.937912+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:40.135762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947417+0000","last_change":"2023-10-16T14:12:40.929102+0000","last_active":"2023-10-16T14:12:41.947417+0000","last_peered":"2023-10-16T14:12:41.947417+0000","last_clean":"2023-10-16T14:12:41.947417+0000","last_became_active":"2023-10-16T14:12:40.926562+0000","last_became_peered":"2023-10-16T14:12:40.926562+0000","last_unstale":"2023-10-16T14:12:41.947417+0000","last_undegraded":"2023-10-16T14:12:41.947417+0000","last_fullsized":"2023-10-16T14:12:41.947417+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:34.169245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406822+0000","last_change":"2023-10-16T14:12:40.947482+0000","last_active":"2023-10-16T14:12:42.406822+0000","last_peered":"2023-10-16T14:12:42.406822+0000","last_clean":"2023-10-16T14:12:42.406822+0000","last_became_active":"2023-10-16T14:12:40.947103+0000","last_became_peered":"2023-10-16T14:12:40.947103+0000","last_unstale":"2023-10-16T14:12:42.406822+0000","last_undegraded":"2023-10-16T14:12:42.406822+0000","last_fullsized":"2023-10-16T14:12:42.406822+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:57.076113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.937575+0000","last_change":"2023-10-16T14:12:40.931890+0000","last_active":"2023-10-16T14:12:40.937575+0000","last_peered":"2023-10-16T14:12:40.937575+0000","last_clean":"2023-10-16T14:12:40.937575+0000","last_became_active":"2023-10-16T14:12:40.931616+0000","last_became_peered":"2023-10-16T14:12:40.931616+0000","last_unstale":"2023-10-16T14:12:40.937575+0000","last_undegraded":"2023-10-16T14:12:40.937575+0000","last_fullsized":"2023-10-16T14:12:40.937575+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:12.643100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.934747+0000","last_change":"2023-10-16T14:12:40.928540+0000","last_active":"2023-10-16T14:12:40.934747+0000","last_peered":"2023-10-16T14:12:40.934747+0000","last_clean":"2023-10-16T14:12:40.934747+0000","last_became_active":"2023-10-16T14:12:40.928261+0000","last_became_peered":"2023-10-16T14:12:40.928261+0000","last_unstale":"2023-10-16T14:12:40.934747+0000","last_undegraded":"2023-10-16T14:12:40.934747+0000","last_fullsized":"2023-10-16T14:12:40.934747+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:55.136856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946892+0000","last_change":"2023-10-16T14:12:40.934086+0000","last_active":"2023-10-16T14:12:41.946892+0000","last_peered":"2023-10-16T14:12:41.946892+0000","last_clean":"2023-10-16T14:12:41.946892+0000","last_became_active":"2023-10-16T14:12:40.933814+0000","last_became_peered":"2023-10-16T14:12:40.933814+0000","last_unstale":"2023-10-16T14:12:41.946892+0000","last_undegraded":"2023-10-16T14:12:41.946892+0000","last_fullsized":"2023-10-16T14:12:41.946892+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:05.018323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.946586+0000","last_change":"2023-10-16T14:12:40.941048+0000","last_active":"2023-10-16T14:12:40.946586+0000","last_peered":"2023-10-16T14:12:40.946586+0000","last_clean":"2023-10-16T14:12:40.946586+0000","last_became_active":"2023-10-16T14:12:40.940603+0000","last_became_peered":"2023-10-16T14:12:40.940603+0000","last_unstale":"2023-10-16T14:12:40.946586+0000","last_undegraded":"2023-10-16T14:12:40.946586+0000","last_fullsized":"2023-10-16T14:12:40.946586+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:29:38.530118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.942269+0000","last_change":"2023-10-16T14:12:40.935075+0000","last_active":"2023-10-16T14:12:40.942269+0000","last_peered":"2023-10-16T14:12:40.942269+0000","last_clean":"2023-10-16T14:12:40.942269+0000","last_became_active":"2023-10-16T14:12:40.933991+0000","last_became_peered":"2023-10-16T14:12:40.933991+0000","last_unstale":"2023-10-16T14:12:40.942269+0000","last_undegraded":"2023-10-16T14:12:40.942269+0000","last_fullsized":"2023-10-16T14:12:40.942269+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:48.321908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.938109+0000","last_change":"2023-10-16T14:12:40.937586+0000","last_active":"2023-10-16T14:12:41.938109+0000","last_peered":"2023-10-16T14:12:41.938109+0000","last_clean":"2023-10-16T14:12:41.938109+0000","last_became_active":"2023-10-16T14:12:40.937158+0000","last_became_peered":"2023-10-16T14:12:40.937158+0000","last_unstale":"2023-10-16T14:12:41.938109+0000","last_undegraded":"2023-10-16T14:12:41.938109+0000","last_fullsized":"2023-10-16T14:12:41.938109+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:29.205963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941296+0000","last_change":"2023-10-16T14:12:40.942201+0000","last_active":"2023-10-16T14:12:41.941296+0000","last_peered":"2023-10-16T14:12:41.941296+0000","last_clean":"2023-10-16T14:12:41.941296+0000","last_became_active":"2023-10-16T14:12:40.941912+0000","last_became_peered":"2023-10-16T14:12:40.941912+0000","last_unstale":"2023-10-16T14:12:41.941296+0000","last_undegraded":"2023-10-16T14:12:41.941296+0000","last_fullsized":"2023-10-16T14:12:41.941296+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:03.456935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947963+0000","last_change":"2023-10-16T14:12:40.945569+0000","last_active":"2023-10-16T14:12:40.947963+0000","last_peered":"2023-10-16T14:12:40.947963+0000","last_clean":"2023-10-16T14:12:40.947963+0000","last_became_active":"2023-10-16T14:12:40.945092+0000","last_became_peered":"2023-10-16T14:12:40.945092+0000","last_unstale":"2023-10-16T14:12:40.947963+0000","last_undegraded":"2023-10-16T14:12:40.947963+0000","last_fullsized":"2023-10-16T14:12:40.947963+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:42.525032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947289+0000","last_change":"2023-10-16T14:12:40.943427+0000","last_active":"2023-10-16T14:12:40.947289+0000","last_peered":"2023-10-16T14:12:40.947289+0000","last_clean":"2023-10-16T14:12:40.947289+0000","last_became_active":"2023-10-16T14:12:40.943169+0000","last_became_peered":"2023-10-16T14:12:40.943169+0000","last_unstale":"2023-10-16T14:12:40.947289+0000","last_undegraded":"2023-10-16T14:12:40.947289+0000","last_fullsized":"2023-10-16T14:12:40.947289+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:28.599827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.928203+0000","last_change":"2023-10-16T14:12:40.923455+0000","last_active":"2023-10-16T14:12:40.928203+0000","last_peered":"2023-10-16T14:12:40.928203+0000","last_clean":"2023-10-16T14:12:40.928203+0000","last_became_active":"2023-10-16T14:12:40.923158+0000","last_became_peered":"2023-10-16T14:12:40.923158+0000","last_unstale":"2023-10-16T14:12:40.928203+0000","last_undegraded":"2023-10-16T14:12:40.928203+0000","last_fullsized":"2023-10-16T14:12:40.928203+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:27.954952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940934+0000","last_change":"2023-10-16T14:12:40.937953+0000","last_active":"2023-10-16T14:12:40.940934+0000","last_peered":"2023-10-16T14:12:40.940934+0000","last_clean":"2023-10-16T14:12:40.940934+0000","last_became_active":"2023-10-16T14:12:40.937007+0000","last_became_peered":"2023-10-16T14:12:40.937007+0000","last_unstale":"2023-10-16T14:12:40.940934+0000","last_undegraded":"2023-10-16T14:12:40.940934+0000","last_fullsized":"2023-10-16T14:12:40.940934+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:31.540683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.946610+0000","last_change":"2023-10-16T14:12:40.945729+0000","last_active":"2023-10-16T14:12:40.946610+0000","last_peered":"2023-10-16T14:12:40.946610+0000","last_clean":"2023-10-16T14:12:40.946610+0000","last_became_active":"2023-10-16T14:12:40.945426+0000","last_became_peered":"2023-10-16T14:12:40.945426+0000","last_unstale":"2023-10-16T14:12:40.946610+0000","last_undegraded":"2023-10-16T14:12:40.946610+0000","last_fullsized":"2023-10-16T14:12:40.946610+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:50.661572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940888+0000","last_change":"2023-10-16T14:12:40.936979+0000","last_active":"2023-10-16T14:12:40.940888+0000","last_peered":"2023-10-16T14:12:40.940888+0000","last_clean":"2023-10-16T14:12:40.940888+0000","last_became_active":"2023-10-16T14:12:40.936692+0000","last_became_peered":"2023-10-16T14:12:40.936692+0000","last_unstale":"2023-10-16T14:12:40.940888+0000","last_undegraded":"2023-10-16T14:12:40.940888+0000","last_fullsized":"2023-10-16T14:12:40.940888+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:21.402564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.937722+0000","last_change":"2023-10-16T14:12:40.936053+0000","last_active":"2023-10-16T14:12:40.937722+0000","last_peered":"2023-10-16T14:12:40.937722+0000","last_clean":"2023-10-16T14:12:40.937722+0000","last_became_active":"2023-10-16T14:12:40.935237+0000","last_became_peered":"2023-10-16T14:12:40.935237+0000","last_unstale":"2023-10-16T14:12:40.937722+0000","last_undegraded":"2023-10-16T14:12:40.937722+0000","last_fullsized":"2023-10-16T14:12:40.937722+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:30.139744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.938095+0000","last_change":"2023-10-16T14:12:40.930421+0000","last_active":"2023-10-16T14:12:40.938095+0000","last_peered":"2023-10-16T14:12:40.938095+0000","last_clean":"2023-10-16T14:12:40.938095+0000","last_became_active":"2023-10-16T14:12:40.926949+0000","last_became_peered":"2023-10-16T14:12:40.926949+0000","last_unstale":"2023-10-16T14:12:40.938095+0000","last_undegraded":"2023-10-16T14:12:40.938095+0000","last_fullsized":"2023-10-16T14:12:40.938095+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:55.514665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947943+0000","last_change":"2023-10-16T14:12:40.945378+0000","last_active":"2023-10-16T14:12:40.947943+0000","last_peered":"2023-10-16T14:12:40.947943+0000","last_clean":"2023-10-16T14:12:40.947943+0000","last_became_active":"2023-10-16T14:12:40.944713+0000","last_became_peered":"2023-10-16T14:12:40.944713+0000","last_unstale":"2023-10-16T14:12:40.947943+0000","last_undegraded":"2023-10-16T14:12:40.947943+0000","last_fullsized":"2023-10-16T14:12:40.947943+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:03.534397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947966+0000","last_change":"2023-10-16T14:12:40.928920+0000","last_active":"2023-10-16T14:12:41.947966+0000","last_peered":"2023-10-16T14:12:41.947966+0000","last_clean":"2023-10-16T14:12:41.947966+0000","last_became_active":"2023-10-16T14:12:40.928644+0000","last_became_peered":"2023-10-16T14:12:40.928644+0000","last_unstale":"2023-10-16T14:12:41.947966+0000","last_undegraded":"2023-10-16T14:12:41.947966+0000","last_fullsized":"2023-10-16T14:12:41.947966+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:00.458147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954420+0000","last_change":"2023-10-16T14:12:40.953157+0000","last_active":"2023-10-16T14:12:40.954420+0000","last_peered":"2023-10-16T14:12:40.954420+0000","last_clean":"2023-10-16T14:12:40.954420+0000","last_became_active":"2023-10-16T14:12:40.952857+0000","last_became_peered":"2023-10-16T14:12:40.952857+0000","last_unstale":"2023-10-16T14:12:40.954420+0000","last_undegraded":"2023-10-16T14:12:40.954420+0000","last_fullsized":"2023-10-16T14:12:40.954420+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:08.579769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954458+0000","last_change":"2023-10-16T14:12:40.953214+0000","last_active":"2023-10-16T14:12:40.954458+0000","last_peered":"2023-10-16T14:12:40.954458+0000","last_clean":"2023-10-16T14:12:40.954458+0000","last_became_active":"2023-10-16T14:12:40.952848+0000","last_became_peered":"2023-10-16T14:12:40.952848+0000","last_unstale":"2023-10-16T14:12:40.954458+0000","last_undegraded":"2023-10-16T14:12:40.954458+0000","last_fullsized":"2023-10-16T14:12:40.954458+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:33.884134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937637+0000","last_change":"2023-10-16T14:12:40.942082+0000","last_active":"2023-10-16T14:12:41.937637+0000","last_peered":"2023-10-16T14:12:41.937637+0000","last_clean":"2023-10-16T14:12:41.937637+0000","last_became_active":"2023-10-16T14:12:40.941383+0000","last_became_peered":"2023-10-16T14:12:40.941383+0000","last_unstale":"2023-10-16T14:12:41.937637+0000","last_undegraded":"2023-10-16T14:12:41.937637+0000","last_fullsized":"2023-10-16T14:12:41.937637+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:24.778415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.936782+0000","last_change":"2023-10-16T14:12:40.928736+0000","last_active":"2023-10-16T14:12:40.936782+0000","last_peered":"2023-10-16T14:12:40.936782+0000","last_clean":"2023-10-16T14:12:40.936782+0000","last_became_active":"2023-10-16T14:12:40.927346+0000","last_became_peered":"2023-10-16T14:12:40.927346+0000","last_unstale":"2023-10-16T14:12:40.936782+0000","last_undegraded":"2023-10-16T14:12:40.936782+0000","last_fullsized":"2023-10-16T14:12:40.936782+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:42.004797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"272'4","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.952022+0000","last_change":"2023-10-16T14:12:39.934680+0000","last_active":"2023-10-16T14:12:40.952022+0000","last_peered":"2023-10-16T14:12:40.952022+0000","last_clean":"2023-10-16T14:12:40.952022+0000","last_became_active":"2023-10-16T14:12:39.934364+0000","last_became_peered":"2023-10-16T14:12:39.934364+0000","last_unstale":"2023-10-16T14:12:40.952022+0000","last_undegraded":"2023-10-16T14:12:40.952022+0000","last_fullsized":"2023-10-16T14:12:40.952022+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:03.047367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407184+0000","last_change":"2023-10-16T14:12:40.949613+0000","last_active":"2023-10-16T14:12:42.407184+0000","last_peered":"2023-10-16T14:12:42.407184+0000","last_clean":"2023-10-16T14:12:42.407184+0000","last_became_active":"2023-10-16T14:12:40.949235+0000","last_became_peered":"2023-10-16T14:12:40.949235+0000","last_unstale":"2023-10-16T14:12:42.407184+0000","last_undegraded":"2023-10-16T14:12:42.407184+0000","last_fullsized":"2023-10-16T14:12:42.407184+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:35.973009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937619+0000","last_change":"2023-10-16T14:12:39.943517+0000","last_active":"2023-10-16T14:12:41.937619+0000","last_peered":"2023-10-16T14:12:41.937619+0000","last_clean":"2023-10-16T14:12:41.937619+0000","last_became_active":"2023-10-16T14:12:39.942151+0000","last_became_peered":"2023-10-16T14:12:39.942151+0000","last_unstale":"2023-10-16T14:12:41.937619+0000","last_undegraded":"2023-10-16T14:12:41.937619+0000","last_fullsized":"2023-10-16T14:12:41.937619+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:07.670532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947155+0000","last_change":"2023-10-16T14:12:40.944633+0000","last_active":"2023-10-16T14:12:40.947155+0000","last_peered":"2023-10-16T14:12:40.947155+0000","last_clean":"2023-10-16T14:12:40.947155+0000","last_became_active":"2023-10-16T14:12:40.944256+0000","last_became_peered":"2023-10-16T14:12:40.944256+0000","last_unstale":"2023-10-16T14:12:40.947155+0000","last_undegraded":"2023-10-16T14:12:40.947155+0000","last_fullsized":"2023-10-16T14:12:40.947155+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:02.484270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"272'2","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937135+0000","last_change":"2023-10-16T14:12:39.938871+0000","last_active":"2023-10-16T14:12:41.937135+0000","last_peered":"2023-10-16T14:12:41.937135+0000","last_clean":"2023-10-16T14:12:41.937135+0000","last_became_active":"2023-10-16T14:12:39.938538+0000","last_became_peered":"2023-10-16T14:12:39.938538+0000","last_unstale":"2023-10-16T14:12:41.937135+0000","last_undegraded":"2023-10-16T14:12:41.937135+0000","last_fullsized":"2023-10-16T14:12:41.937135+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:06.014963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947764+0000","last_change":"2023-10-16T14:12:40.945856+0000","last_active":"2023-10-16T14:12:40.947764+0000","last_peered":"2023-10-16T14:12:40.947764+0000","last_clean":"2023-10-16T14:12:40.947764+0000","last_became_active":"2023-10-16T14:12:40.944531+0000","last_became_peered":"2023-10-16T14:12:40.944531+0000","last_unstale":"2023-10-16T14:12:40.947764+0000","last_undegraded":"2023-10-16T14:12:40.947764+0000","last_fullsized":"2023-10-16T14:12:40.947764+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:14.849580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947198+0000","last_change":"2023-10-16T14:12:39.942390+0000","last_active":"2023-10-16T14:12:40.947198+0000","last_peered":"2023-10-16T14:12:40.947198+0000","last_clean":"2023-10-16T14:12:40.947198+0000","last_became_active":"2023-10-16T14:12:39.941990+0000","last_became_peered":"2023-10-16T14:12:39.941990+0000","last_unstale":"2023-10-16T14:12:40.947198+0000","last_undegraded":"2023-10-16T14:12:40.947198+0000","last_fullsized":"2023-10-16T14:12:40.947198+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:47.156586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943856+0000","last_change":"2023-10-16T14:12:40.940764+0000","last_active":"2023-10-16T14:12:40.943856+0000","last_peered":"2023-10-16T14:12:40.943856+0000","last_clean":"2023-10-16T14:12:40.943856+0000","last_became_active":"2023-10-16T14:12:40.939946+0000","last_became_peered":"2023-10-16T14:12:40.939946+0000","last_unstale":"2023-10-16T14:12:40.943856+0000","last_undegraded":"2023-10-16T14:12:40.943856+0000","last_fullsized":"2023-10-16T14:12:40.943856+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:34.790020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"272'1","reported_seq":12,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.944568+0000","last_change":"2023-10-16T14:12:39.943644+0000","last_active":"2023-10-16T14:12:40.944568+0000","last_peered":"2023-10-16T14:12:40.944568+0000","last_clean":"2023-10-16T14:12:40.944568+0000","last_became_active":"2023-10-16T14:12:39.943150+0000","last_became_peered":"2023-10-16T14:12:39.943150+0000","last_unstale":"2023-10-16T14:12:40.944568+0000","last_undegraded":"2023-10-16T14:12:40.944568+0000","last_fullsized":"2023-10-16T14:12:40.944568+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:50.187777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947306+0000","last_change":"2023-10-16T14:12:40.942714+0000","last_active":"2023-10-16T14:12:40.947306+0000","last_peered":"2023-10-16T14:12:40.947306+0000","last_clean":"2023-10-16T14:12:40.947306+0000","last_became_active":"2023-10-16T14:12:40.942424+0000","last_became_peered":"2023-10-16T14:12:40.942424+0000","last_unstale":"2023-10-16T14:12:40.947306+0000","last_undegraded":"2023-10-16T14:12:40.947306+0000","last_fullsized":"2023-10-16T14:12:40.947306+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:41.356096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940599+0000","last_change":"2023-10-16T14:12:39.936379+0000","last_active":"2023-10-16T14:12:41.940599+0000","last_peered":"2023-10-16T14:12:41.940599+0000","last_clean":"2023-10-16T14:12:41.940599+0000","last_became_active":"2023-10-16T14:12:39.935932+0000","last_became_peered":"2023-10-16T14:12:39.935932+0000","last_unstale":"2023-10-16T14:12:41.940599+0000","last_undegraded":"2023-10-16T14:12:41.940599+0000","last_fullsized":"2023-10-16T14:12:41.940599+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:35.875704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937938+0000","last_change":"2023-10-16T14:12:40.928538+0000","last_active":"2023-10-16T14:12:41.937938+0000","last_peered":"2023-10-16T14:12:41.937938+0000","last_clean":"2023-10-16T14:12:41.937938+0000","last_became_active":"2023-10-16T14:12:40.926549+0000","last_became_peered":"2023-10-16T14:12:40.926549+0000","last_unstale":"2023-10-16T14:12:41.937938+0000","last_undegraded":"2023-10-16T14:12:41.937938+0000","last_fullsized":"2023-10-16T14:12:41.937938+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:44.231128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917618+0000","last_change":"2023-10-16T14:12:39.928718+0000","last_active":"2023-10-16T14:12:40.917618+0000","last_peered":"2023-10-16T14:12:40.917618+0000","last_clean":"2023-10-16T14:12:40.917618+0000","last_became_active":"2023-10-16T14:12:39.928016+0000","last_became_peered":"2023-10-16T14:12:39.928016+0000","last_unstale":"2023-10-16T14:12:40.917618+0000","last_undegraded":"2023-10-16T14:12:40.917618+0000","last_fullsized":"2023-10-16T14:12:40.917618+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:19.422569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940134+0000","last_change":"2023-10-16T14:12:40.937821+0000","last_active":"2023-10-16T14:12:41.940134+0000","last_peered":"2023-10-16T14:12:41.940134+0000","last_clean":"2023-10-16T14:12:41.940134+0000","last_became_active":"2023-10-16T14:12:40.937502+0000","last_became_peered":"2023-10-16T14:12:40.937502+0000","last_unstale":"2023-10-16T14:12:41.940134+0000","last_undegraded":"2023-10-16T14:12:41.940134+0000","last_fullsized":"2023-10-16T14:12:41.940134+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:55.851366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.912696+0000","last_change":"2023-10-16T14:12:39.933667+0000","last_active":"2023-10-16T14:12:40.912696+0000","last_peered":"2023-10-16T14:12:40.912696+0000","last_clean":"2023-10-16T14:12:40.912696+0000","last_became_active":"2023-10-16T14:12:39.932845+0000","last_became_peered":"2023-10-16T14:12:39.932845+0000","last_unstale":"2023-10-16T14:12:40.912696+0000","last_undegraded":"2023-10-16T14:12:40.912696+0000","last_fullsized":"2023-10-16T14:12:40.912696+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:55.667818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941525+0000","last_change":"2023-10-16T14:12:40.942048+0000","last_active":"2023-10-16T14:12:41.941525+0000","last_peered":"2023-10-16T14:12:41.941525+0000","last_clean":"2023-10-16T14:12:41.941525+0000","last_became_active":"2023-10-16T14:12:40.938626+0000","last_became_peered":"2023-10-16T14:12:40.938626+0000","last_unstale":"2023-10-16T14:12:41.941525+0000","last_undegraded":"2023-10-16T14:12:41.941525+0000","last_fullsized":"2023-10-16T14:12:41.941525+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:59.528771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947939+0000","last_change":"2023-10-16T14:12:39.933244+0000","last_active":"2023-10-16T14:12:41.947939+0000","last_peered":"2023-10-16T14:12:41.947939+0000","last_clean":"2023-10-16T14:12:41.947939+0000","last_became_active":"2023-10-16T14:12:39.932799+0000","last_became_peered":"2023-10-16T14:12:39.932799+0000","last_unstale":"2023-10-16T14:12:41.947939+0000","last_undegraded":"2023-10-16T14:12:41.947939+0000","last_fullsized":"2023-10-16T14:12:41.947939+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:40.499697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947995+0000","last_change":"2023-10-16T14:12:40.943903+0000","last_active":"2023-10-16T14:12:41.947995+0000","last_peered":"2023-10-16T14:12:41.947995+0000","last_clean":"2023-10-16T14:12:41.947995+0000","last_became_active":"2023-10-16T14:12:40.943382+0000","last_became_peered":"2023-10-16T14:12:40.943382+0000","last_unstale":"2023-10-16T14:12:41.947995+0000","last_undegraded":"2023-10-16T14:12:41.947995+0000","last_fullsized":"2023-10-16T14:12:41.947995+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:36.547016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407020+0000","last_change":"2023-10-16T14:12:39.940712+0000","last_active":"2023-10-16T14:12:42.407020+0000","last_peered":"2023-10-16T14:12:42.407020+0000","last_clean":"2023-10-16T14:12:42.407020+0000","last_became_active":"2023-10-16T14:12:39.940444+0000","last_became_peered":"2023-10-16T14:12:39.940444+0000","last_unstale":"2023-10-16T14:12:42.407020+0000","last_undegraded":"2023-10-16T14:12:42.407020+0000","last_fullsized":"2023-10-16T14:12:42.407020+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:24.201772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406953+0000","last_change":"2023-10-16T14:12:40.946838+0000","last_active":"2023-10-16T14:12:42.406953+0000","last_peered":"2023-10-16T14:12:42.406953+0000","last_clean":"2023-10-16T14:12:42.406953+0000","last_became_active":"2023-10-16T14:12:40.946501+0000","last_became_peered":"2023-10-16T14:12:40.946501+0000","last_unstale":"2023-10-16T14:12:42.406953+0000","last_undegraded":"2023-10-16T14:12:42.406953+0000","last_fullsized":"2023-10-16T14:12:42.406953+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:53.980955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"272'3","reported_seq":15,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937144+0000","last_change":"2023-10-16T14:12:39.941559+0000","last_active":"2023-10-16T14:12:41.937144+0000","last_peered":"2023-10-16T14:12:41.937144+0000","last_clean":"2023-10-16T14:12:41.937144+0000","last_became_active":"2023-10-16T14:12:39.941204+0000","last_became_peered":"2023-10-16T14:12:39.941204+0000","last_unstale":"2023-10-16T14:12:41.937144+0000","last_undegraded":"2023-10-16T14:12:41.937144+0000","last_fullsized":"2023-10-16T14:12:41.937144+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:59.495481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947664+0000","last_change":"2023-10-16T14:12:40.929769+0000","last_active":"2023-10-16T14:12:41.947664+0000","last_peered":"2023-10-16T14:12:41.947664+0000","last_clean":"2023-10-16T14:12:41.947664+0000","last_became_active":"2023-10-16T14:12:40.929451+0000","last_became_peered":"2023-10-16T14:12:40.929451+0000","last_unstale":"2023-10-16T14:12:41.947664+0000","last_undegraded":"2023-10-16T14:12:41.947664+0000","last_fullsized":"2023-10-16T14:12:41.947664+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:37.123028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"272'6","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954409+0000","last_change":"2023-10-16T14:12:39.941004+0000","last_active":"2023-10-16T14:12:40.954409+0000","last_peered":"2023-10-16T14:12:40.954409+0000","last_clean":"2023-10-16T14:12:40.954409+0000","last_became_active":"2023-10-16T14:12:39.939179+0000","last_became_peered":"2023-10-16T14:12:39.939179+0000","last_unstale":"2023-10-16T14:12:40.954409+0000","last_undegraded":"2023-10-16T14:12:40.954409+0000","last_fullsized":"2023-10-16T14:12:40.954409+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:51.039455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.945759+0000","last_change":"2023-10-16T14:12:40.939835+0000","last_active":"2023-10-16T14:12:40.945759+0000","last_peered":"2023-10-16T14:12:40.945759+0000","last_clean":"2023-10-16T14:12:40.945759+0000","last_became_active":"2023-10-16T14:12:40.939592+0000","last_became_peered":"2023-10-16T14:12:40.939592+0000","last_unstale":"2023-10-16T14:12:40.945759+0000","last_undegraded":"2023-10-16T14:12:40.945759+0000","last_fullsized":"2023-10-16T14:12:40.945759+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:50.906044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940377+0000","last_change":"2023-10-16T14:12:39.927128+0000","last_active":"2023-10-16T14:12:40.940377+0000","last_peered":"2023-10-16T14:12:40.940377+0000","last_clean":"2023-10-16T14:12:40.940377+0000","last_became_active":"2023-10-16T14:12:39.926762+0000","last_became_peered":"2023-10-16T14:12:39.926762+0000","last_unstale":"2023-10-16T14:12:40.940377+0000","last_undegraded":"2023-10-16T14:12:40.940377+0000","last_fullsized":"2023-10-16T14:12:40.940377+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:10.984003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940179+0000","last_change":"2023-10-16T14:12:40.932173+0000","last_active":"2023-10-16T14:12:41.940179+0000","last_peered":"2023-10-16T14:12:41.940179+0000","last_clean":"2023-10-16T14:12:41.940179+0000","last_became_active":"2023-10-16T14:12:40.931555+0000","last_became_peered":"2023-10-16T14:12:40.931555+0000","last_unstale":"2023-10-16T14:12:41.940179+0000","last_undegraded":"2023-10-16T14:12:41.940179+0000","last_fullsized":"2023-10-16T14:12:41.940179+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:32.561673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917340+0000","last_change":"2023-10-16T14:12:39.937752+0000","last_active":"2023-10-16T14:12:40.917340+0000","last_peered":"2023-10-16T14:12:40.917340+0000","last_clean":"2023-10-16T14:12:40.917340+0000","last_became_active":"2023-10-16T14:12:39.937419+0000","last_became_peered":"2023-10-16T14:12:39.937419+0000","last_unstale":"2023-10-16T14:12:40.917340+0000","last_undegraded":"2023-10-16T14:12:40.917340+0000","last_fullsized":"2023-10-16T14:12:40.917340+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:25.330533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406367+0000","last_change":"2023-10-16T14:12:40.951163+0000","last_active":"2023-10-16T14:12:42.406367+0000","last_peered":"2023-10-16T14:12:42.406367+0000","last_clean":"2023-10-16T14:12:42.406367+0000","last_became_active":"2023-10-16T14:12:40.950205+0000","last_became_peered":"2023-10-16T14:12:40.950205+0000","last_unstale":"2023-10-16T14:12:42.406367+0000","last_undegraded":"2023-10-16T14:12:42.406367+0000","last_fullsized":"2023-10-16T14:12:42.406367+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:17.074207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946769+0000","last_change":"2023-10-16T14:12:39.925773+0000","last_active":"2023-10-16T14:12:41.946769+0000","last_peered":"2023-10-16T14:12:41.946769+0000","last_clean":"2023-10-16T14:12:41.946769+0000","last_became_active":"2023-10-16T14:12:39.925301+0000","last_became_peered":"2023-10-16T14:12:39.925301+0000","last_unstale":"2023-10-16T14:12:41.946769+0000","last_undegraded":"2023-10-16T14:12:41.946769+0000","last_fullsized":"2023-10-16T14:12:41.946769+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:28.985481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.945878+0000","last_change":"2023-10-16T14:12:40.940856+0000","last_active":"2023-10-16T14:12:40.945878+0000","last_peered":"2023-10-16T14:12:40.945878+0000","last_clean":"2023-10-16T14:12:40.945878+0000","last_became_active":"2023-10-16T14:12:40.940254+0000","last_became_peered":"2023-10-16T14:12:40.940254+0000","last_unstale":"2023-10-16T14:12:40.945878+0000","last_undegraded":"2023-10-16T14:12:40.945878+0000","last_fullsized":"2023-10-16T14:12:40.945878+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:10.059974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"272'8","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947349+0000","last_change":"2023-10-16T14:12:39.939556+0000","last_active":"2023-10-16T14:12:40.947349+0000","last_peered":"2023-10-16T14:12:40.947349+0000","last_clean":"2023-10-16T14:12:40.947349+0000","last_became_active":"2023-10-16T14:12:39.939244+0000","last_became_peered":"2023-10-16T14:12:39.939244+0000","last_unstale":"2023-10-16T14:12:40.947349+0000","last_undegraded":"2023-10-16T14:12:40.947349+0000","last_fullsized":"2023-10-16T14:12:40.947349+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:54.439596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.932029+0000","last_change":"2023-10-16T14:12:40.928984+0000","last_active":"2023-10-16T14:12:40.932029+0000","last_peered":"2023-10-16T14:12:40.932029+0000","last_clean":"2023-10-16T14:12:40.932029+0000","last_became_active":"2023-10-16T14:12:40.926554+0000","last_became_peered":"2023-10-16T14:12:40.926554+0000","last_unstale":"2023-10-16T14:12:40.932029+0000","last_undegraded":"2023-10-16T14:12:40.932029+0000","last_fullsized":"2023-10-16T14:12:40.932029+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:24.244423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.948110+0000","last_change":"2023-10-16T14:12:39.924031+0000","last_active":"2023-10-16T14:12:41.948110+0000","last_peered":"2023-10-16T14:12:41.948110+0000","last_clean":"2023-10-16T14:12:41.948110+0000","last_became_active":"2023-10-16T14:12:39.923745+0000","last_became_peered":"2023-10-16T14:12:39.923745+0000","last_unstale":"2023-10-16T14:12:41.948110+0000","last_undegraded":"2023-10-16T14:12:41.948110+0000","last_fullsized":"2023-10-16T14:12:41.948110+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:02.671005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940021+0000","last_change":"2023-10-16T14:12:40.936029+0000","last_active":"2023-10-16T14:12:40.940021+0000","last_peered":"2023-10-16T14:12:40.940021+0000","last_clean":"2023-10-16T14:12:40.940021+0000","last_became_active":"2023-10-16T14:12:40.935732+0000","last_became_peered":"2023-10-16T14:12:40.935732+0000","last_unstale":"2023-10-16T14:12:40.940021+0000","last_undegraded":"2023-10-16T14:12:40.940021+0000","last_fullsized":"2023-10-16T14:12:40.940021+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:54.322819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.952260+0000","last_change":"2023-10-16T14:12:39.937298+0000","last_active":"2023-10-16T14:12:40.952260+0000","last_peered":"2023-10-16T14:12:40.952260+0000","last_clean":"2023-10-16T14:12:40.952260+0000","last_became_active":"2023-10-16T14:12:39.937019+0000","last_became_peered":"2023-10-16T14:12:39.937019+0000","last_unstale":"2023-10-16T14:12:40.952260+0000","last_undegraded":"2023-10-16T14:12:40.952260+0000","last_fullsized":"2023-10-16T14:12:40.952260+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:52.174465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941292+0000","last_change":"2023-10-16T14:12:40.950379+0000","last_active":"2023-10-16T14:12:41.941292+0000","last_peered":"2023-10-16T14:12:41.941292+0000","last_clean":"2023-10-16T14:12:41.941292+0000","last_became_active":"2023-10-16T14:12:40.950035+0000","last_became_peered":"2023-10-16T14:12:40.950035+0000","last_unstale":"2023-10-16T14:12:41.941292+0000","last_undegraded":"2023-10-16T14:12:41.941292+0000","last_fullsized":"2023-10-16T14:12:41.941292+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:46.054897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947659+0000","last_change":"2023-10-16T14:12:39.928129+0000","last_active":"2023-10-16T14:12:41.947659+0000","last_peered":"2023-10-16T14:12:41.947659+0000","last_clean":"2023-10-16T14:12:41.947659+0000","last_became_active":"2023-10-16T14:12:39.927848+0000","last_became_peered":"2023-10-16T14:12:39.927848+0000","last_unstale":"2023-10-16T14:12:41.947659+0000","last_undegraded":"2023-10-16T14:12:41.947659+0000","last_fullsized":"2023-10-16T14:12:41.947659+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:00.144505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943162+0000","last_change":"2023-10-16T14:12:40.939787+0000","last_active":"2023-10-16T14:12:40.943162+0000","last_peered":"2023-10-16T14:12:40.943162+0000","last_clean":"2023-10-16T14:12:40.943162+0000","last_became_active":"2023-10-16T14:12:40.939509+0000","last_became_peered":"2023-10-16T14:12:40.939509+0000","last_unstale":"2023-10-16T14:12:40.943162+0000","last_undegraded":"2023-10-16T14:12:40.943162+0000","last_fullsized":"2023-10-16T14:12:40.943162+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:46.210818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947857+0000","last_change":"2023-10-16T14:12:39.933837+0000","last_active":"2023-10-16T14:12:41.947857+0000","last_peered":"2023-10-16T14:12:41.947857+0000","last_clean":"2023-10-16T14:12:41.947857+0000","last_became_active":"2023-10-16T14:12:39.933569+0000","last_became_peered":"2023-10-16T14:12:39.933569+0000","last_unstale":"2023-10-16T14:12:41.947857+0000","last_undegraded":"2023-10-16T14:12:41.947857+0000","last_fullsized":"2023-10-16T14:12:41.947857+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:58.522569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.936950+0000","last_change":"2023-10-16T14:12:40.941796+0000","last_active":"2023-10-16T14:12:41.936950+0000","last_peered":"2023-10-16T14:12:41.936950+0000","last_clean":"2023-10-16T14:12:41.936950+0000","last_became_active":"2023-10-16T14:12:40.941411+0000","last_became_peered":"2023-10-16T14:12:40.941411+0000","last_unstale":"2023-10-16T14:12:41.936950+0000","last_undegraded":"2023-10-16T14:12:41.936950+0000","last_fullsized":"2023-10-16T14:12:41.936950+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:24.580487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947060+0000","last_change":"2023-10-16T14:12:39.938790+0000","last_active":"2023-10-16T14:12:40.947060+0000","last_peered":"2023-10-16T14:12:40.947060+0000","last_clean":"2023-10-16T14:12:40.947060+0000","last_became_active":"2023-10-16T14:12:39.938488+0000","last_became_peered":"2023-10-16T14:12:39.938488+0000","last_unstale":"2023-10-16T14:12:40.947060+0000","last_undegraded":"2023-10-16T14:12:40.947060+0000","last_fullsized":"2023-10-16T14:12:40.947060+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:42.258052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940795+0000","last_change":"2023-10-16T14:12:40.932475+0000","last_active":"2023-10-16T14:12:41.940795+0000","last_peered":"2023-10-16T14:12:41.940795+0000","last_clean":"2023-10-16T14:12:41.940795+0000","last_became_active":"2023-10-16T14:12:40.931912+0000","last_became_peered":"2023-10-16T14:12:40.931912+0000","last_unstale":"2023-10-16T14:12:41.940795+0000","last_undegraded":"2023-10-16T14:12:41.940795+0000","last_fullsized":"2023-10-16T14:12:41.940795+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:48.198209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.925482+0000","last_change":"2023-10-16T14:12:39.939921+0000","last_active":"2023-10-16T14:12:40.925482+0000","last_peered":"2023-10-16T14:12:40.925482+0000","last_clean":"2023-10-16T14:12:40.925482+0000","last_became_active":"2023-10-16T14:12:39.939634+0000","last_became_peered":"2023-10-16T14:12:39.939634+0000","last_unstale":"2023-10-16T14:12:40.925482+0000","last_undegraded":"2023-10-16T14:12:40.925482+0000","last_fullsized":"2023-10-16T14:12:40.925482+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:47.208133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940178+0000","last_change":"2023-10-16T14:12:40.936615+0000","last_active":"2023-10-16T14:12:41.940178+0000","last_peered":"2023-10-16T14:12:41.940178+0000","last_clean":"2023-10-16T14:12:41.940178+0000","last_became_active":"2023-10-16T14:12:40.935677+0000","last_became_peered":"2023-10-16T14:12:40.935677+0000","last_unstale":"2023-10-16T14:12:41.940178+0000","last_undegraded":"2023-10-16T14:12:41.940178+0000","last_fullsized":"2023-10-16T14:12:41.940178+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:32.275285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946984+0000","last_change":"2023-10-16T14:12:39.930125+0000","last_active":"2023-10-16T14:12:41.946984+0000","last_peered":"2023-10-16T14:12:41.946984+0000","last_clean":"2023-10-16T14:12:41.946984+0000","last_became_active":"2023-10-16T14:12:39.929162+0000","last_became_peered":"2023-10-16T14:12:39.929162+0000","last_unstale":"2023-10-16T14:12:41.946984+0000","last_undegraded":"2023-10-16T14:12:41.946984+0000","last_fullsized":"2023-10-16T14:12:41.946984+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:00.530710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.937696+0000","last_change":"2023-10-16T14:12:40.932252+0000","last_active":"2023-10-16T14:12:40.937696+0000","last_peered":"2023-10-16T14:12:40.937696+0000","last_clean":"2023-10-16T14:12:40.937696+0000","last_became_active":"2023-10-16T14:12:40.931966+0000","last_became_peered":"2023-10-16T14:12:40.931966+0000","last_unstale":"2023-10-16T14:12:40.937696+0000","last_undegraded":"2023-10-16T14:12:40.937696+0000","last_fullsized":"2023-10-16T14:12:40.937696+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:33.930874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.912904+0000","last_change":"2023-10-16T14:12:39.933128+0000","last_active":"2023-10-16T14:12:40.912904+0000","last_peered":"2023-10-16T14:12:40.912904+0000","last_clean":"2023-10-16T14:12:40.912904+0000","last_became_active":"2023-10-16T14:12:39.932483+0000","last_became_peered":"2023-10-16T14:12:39.932483+0000","last_unstale":"2023-10-16T14:12:40.912904+0000","last_undegraded":"2023-10-16T14:12:40.912904+0000","last_fullsized":"2023-10-16T14:12:40.912904+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:34.551675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.945748+0000","last_change":"2023-10-16T14:12:40.939835+0000","last_active":"2023-10-16T14:12:40.945748+0000","last_peered":"2023-10-16T14:12:40.945748+0000","last_clean":"2023-10-16T14:12:40.945748+0000","last_became_active":"2023-10-16T14:12:40.939556+0000","last_became_peered":"2023-10-16T14:12:40.939556+0000","last_unstale":"2023-10-16T14:12:40.945748+0000","last_undegraded":"2023-10-16T14:12:40.945748+0000","last_fullsized":"2023-10-16T14:12:40.945748+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:42.809225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"272'2","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407274+0000","last_change":"2023-10-16T14:12:39.938879+0000","last_active":"2023-10-16T14:12:42.407274+0000","last_peered":"2023-10-16T14:12:42.407274+0000","last_clean":"2023-10-16T14:12:42.407274+0000","last_became_active":"2023-10-16T14:12:39.938586+0000","last_became_peered":"2023-10-16T14:12:39.938586+0000","last_unstale":"2023-10-16T14:12:42.407274+0000","last_undegraded":"2023-10-16T14:12:42.407274+0000","last_fullsized":"2023-10-16T14:12:42.407274+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:54.414122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941680+0000","last_change":"2023-10-16T14:12:40.936506+0000","last_active":"2023-10-16T14:12:41.941680+0000","last_peered":"2023-10-16T14:12:41.941680+0000","last_clean":"2023-10-16T14:12:41.941680+0000","last_became_active":"2023-10-16T14:12:40.935931+0000","last_became_peered":"2023-10-16T14:12:40.935931+0000","last_unstale":"2023-10-16T14:12:41.941680+0000","last_undegraded":"2023-10-16T14:12:41.941680+0000","last_fullsized":"2023-10-16T14:12:41.941680+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:26.605559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941572+0000","last_change":"2023-10-16T14:12:39.937705+0000","last_active":"2023-10-16T14:12:41.941572+0000","last_peered":"2023-10-16T14:12:41.941572+0000","last_clean":"2023-10-16T14:12:41.941572+0000","last_became_active":"2023-10-16T14:12:39.937419+0000","last_became_peered":"2023-10-16T14:12:39.937419+0000","last_unstale":"2023-10-16T14:12:41.941572+0000","last_undegraded":"2023-10-16T14:12:41.941572+0000","last_fullsized":"2023-10-16T14:12:41.941572+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:37.216338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947660+0000","last_change":"2023-10-16T14:12:40.944810+0000","last_active":"2023-10-16T14:12:40.947660+0000","last_peered":"2023-10-16T14:12:40.947660+0000","last_clean":"2023-10-16T14:12:40.947660+0000","last_became_active":"2023-10-16T14:12:40.944510+0000","last_became_peered":"2023-10-16T14:12:40.944510+0000","last_unstale":"2023-10-16T14:12:40.947660+0000","last_undegraded":"2023-10-16T14:12:40.947660+0000","last_fullsized":"2023-10-16T14:12:40.947660+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:43.087530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406760+0000","last_change":"2023-10-16T14:12:39.941935+0000","last_active":"2023-10-16T14:12:42.406760+0000","last_peered":"2023-10-16T14:12:42.406760+0000","last_clean":"2023-10-16T14:12:42.406760+0000","last_became_active":"2023-10-16T14:12:39.941555+0000","last_became_peered":"2023-10-16T14:12:39.941555+0000","last_unstale":"2023-10-16T14:12:42.406760+0000","last_undegraded":"2023-10-16T14:12:42.406760+0000","last_fullsized":"2023-10-16T14:12:42.406760+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:22.527108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947249+0000","last_change":"2023-10-16T14:12:40.944756+0000","last_active":"2023-10-16T14:12:40.947249+0000","last_peered":"2023-10-16T14:12:40.947249+0000","last_clean":"2023-10-16T14:12:40.947249+0000","last_became_active":"2023-10-16T14:12:40.944457+0000","last_became_peered":"2023-10-16T14:12:40.944457+0000","last_unstale":"2023-10-16T14:12:40.947249+0000","last_undegraded":"2023-10-16T14:12:40.947249+0000","last_fullsized":"2023-10-16T14:12:40.947249+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:00:26.779056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407112+0000","last_change":"2023-10-16T14:12:39.940002+0000","last_active":"2023-10-16T14:12:42.407112+0000","last_peered":"2023-10-16T14:12:42.407112+0000","last_clean":"2023-10-16T14:12:42.407112+0000","last_became_active":"2023-10-16T14:12:39.939628+0000","last_became_peered":"2023-10-16T14:12:39.939628+0000","last_unstale":"2023-10-16T14:12:42.407112+0000","last_undegraded":"2023-10-16T14:12:42.407112+0000","last_fullsized":"2023-10-16T14:12:42.407112+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:11.400216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.938060+0000","last_change":"2023-10-16T14:12:40.950306+0000","last_active":"2023-10-16T14:12:41.938060+0000","last_peered":"2023-10-16T14:12:41.938060+0000","last_clean":"2023-10-16T14:12:41.938060+0000","last_became_active":"2023-10-16T14:12:40.949894+0000","last_became_peered":"2023-10-16T14:12:40.949894+0000","last_unstale":"2023-10-16T14:12:41.938060+0000","last_undegraded":"2023-10-16T14:12:41.938060+0000","last_fullsized":"2023-10-16T14:12:41.938060+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:11.505844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.935271+0000","last_change":"2023-10-16T14:12:39.943126+0000","last_active":"2023-10-16T14:12:40.935271+0000","last_peered":"2023-10-16T14:12:40.935271+0000","last_clean":"2023-10-16T14:12:40.935271+0000","last_became_active":"2023-10-16T14:12:39.941342+0000","last_became_peered":"2023-10-16T14:12:39.941342+0000","last_unstale":"2023-10-16T14:12:40.935271+0000","last_undegraded":"2023-10-16T14:12:40.935271+0000","last_fullsized":"2023-10-16T14:12:40.935271+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:51.253014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940878+0000","last_change":"2023-10-16T14:12:40.927367+0000","last_active":"2023-10-16T14:12:41.940878+0000","last_peered":"2023-10-16T14:12:41.940878+0000","last_clean":"2023-10-16T14:12:41.940878+0000","last_became_active":"2023-10-16T14:12:40.927058+0000","last_became_peered":"2023-10-16T14:12:40.927058+0000","last_unstale":"2023-10-16T14:12:41.940878+0000","last_undegraded":"2023-10-16T14:12:41.940878+0000","last_fullsized":"2023-10-16T14:12:41.940878+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:05.040275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917961+0000","last_change":"2023-10-16T14:12:39.919948+0000","last_active":"2023-10-16T14:12:40.917961+0000","last_peered":"2023-10-16T14:12:40.917961+0000","last_clean":"2023-10-16T14:12:40.917961+0000","last_became_active":"2023-10-16T14:12:39.919568+0000","last_became_peered":"2023-10-16T14:12:39.919568+0000","last_unstale":"2023-10-16T14:12:40.917961+0000","last_undegraded":"2023-10-16T14:12:40.917961+0000","last_fullsized":"2023-10-16T14:12:40.917961+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:46.651309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940569+0000","last_change":"2023-10-16T14:12:40.929601+0000","last_active":"2023-10-16T14:12:41.940569+0000","last_peered":"2023-10-16T14:12:41.940569+0000","last_clean":"2023-10-16T14:12:41.940569+0000","last_became_active":"2023-10-16T14:12:40.929280+0000","last_became_peered":"2023-10-16T14:12:40.929280+0000","last_unstale":"2023-10-16T14:12:41.940569+0000","last_undegraded":"2023-10-16T14:12:41.940569+0000","last_fullsized":"2023-10-16T14:12:41.940569+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:57.978074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917754+0000","last_change":"2023-10-16T14:12:39.919239+0000","last_active":"2023-10-16T14:12:40.917754+0000","last_peered":"2023-10-16T14:12:40.917754+0000","last_clean":"2023-10-16T14:12:40.917754+0000","last_became_active":"2023-10-16T14:12:39.918787+0000","last_became_peered":"2023-10-16T14:12:39.918787+0000","last_unstale":"2023-10-16T14:12:40.917754+0000","last_undegraded":"2023-10-16T14:12:40.917754+0000","last_fullsized":"2023-10-16T14:12:40.917754+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:04.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946711+0000","last_change":"2023-10-16T14:12:40.945087+0000","last_active":"2023-10-16T14:12:41.946711+0000","last_peered":"2023-10-16T14:12:41.946711+0000","last_clean":"2023-10-16T14:12:41.946711+0000","last_became_active":"2023-10-16T14:12:40.944784+0000","last_became_peered":"2023-10-16T14:12:40.944784+0000","last_unstale":"2023-10-16T14:12:41.946711+0000","last_undegraded":"2023-10-16T14:12:41.946711+0000","last_fullsized":"2023-10-16T14:12:41.946711+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:18.518107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.927058+0000","last_change":"2023-10-16T14:12:39.920707+0000","last_active":"2023-10-16T14:12:40.927058+0000","last_peered":"2023-10-16T14:12:40.927058+0000","last_clean":"2023-10-16T14:12:40.927058+0000","last_became_active":"2023-10-16T14:12:39.920138+0000","last_became_peered":"2023-10-16T14:12:39.920138+0000","last_unstale":"2023-10-16T14:12:40.927058+0000","last_undegraded":"2023-10-16T14:12:40.927058+0000","last_fullsized":"2023-10-16T14:12:40.927058+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:47.134334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407298+0000","last_change":"2023-10-16T14:12:40.950705+0000","last_active":"2023-10-16T14:12:42.407298+0000","last_peered":"2023-10-16T14:12:42.407298+0000","last_clean":"2023-10-16T14:12:42.407298+0000","last_became_active":"2023-10-16T14:12:40.950266+0000","last_became_peered":"2023-10-16T14:12:40.950266+0000","last_unstale":"2023-10-16T14:12:42.407298+0000","last_undegraded":"2023-10-16T14:12:42.407298+0000","last_fullsized":"2023-10-16T14:12:42.407298+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:41.864797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"269'2799","reported_seq":3831,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.933327+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:12:40.933327+0000","last_peered":"2023-10-16T14:12:40.933327+0000","last_clean":"2023-10-16T14:12:40.933327+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:12:40.933327+0000","last_undegraded":"2023-10-16T14:12:40.933327+0000","last_fullsized":"2023-10-16T14:12:40.933327+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2507,"num_read_kb":5248,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2507,"num_read_kb":5248,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2799,"ondisk_log_size":2799,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543323,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49008,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322832,"statfs":{"total":96636764160,"available":96586579968,"internally_reserved":0,"allocated":4034560,"data_stored":2282471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.3140000000000001,"15min":1.3480000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.641,"5min":1.8500000000000001,"15min":1.885},"last":1.51},{"interface":"front","average":{"1min":1.714,"5min":1.405,"15min":1.413},"min":{"1min":0.73699999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.603}]},{"osd":1,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.232,"15min":1.246},"min":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.228,"15min":1.248},"min":{"1min":0.76300000000000001,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.2490000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.252,"15min":1.268},"min":{"1min":0.85299999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.458},{"interface":"front","average":{"1min":1.472,"5min":1.349,"15min":1.361},"min":{"1min":0.95199999999999996,"5min":0.745,"15min":0.745},"max":{"1min":3.4289999999999998,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.4219999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.357,"15min":1.405},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.097},"last":1.302},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4159999999999999,"15min":1.431},"min":{"1min":1.107,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.823}]},{"osd":4,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2589999999999999,"15min":1.242},"min":{"1min":0.94699999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.895,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.345},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4730000000000001,"15min":1.4159999999999999},"min":{"1min":0.98799999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.7470000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.4450000000000001,"15min":1.395},"min":{"1min":1.026,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.546,"15min":1.538},"min":{"1min":1.0800000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.667}]},{"osd":6,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.6599999999999999,"15min":1.6379999999999999},"min":{"1min":1.21,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":3.29,"5min":3.29,"15min":3.29},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5600000000000001,"15min":1.486},"min":{"1min":1.0029999999999999,"5min":0.75,"15min":0.75},"max":{"1min":3.524,"5min":3.524,"15min":3.524},"last":1.7230000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543324,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48420,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323420,"statfs":{"total":96636764160,"available":96587182080,"internally_reserved":0,"allocated":3432448,"data_stored":1685850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.244,"15min":1.3109999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.887,"15min":1.887},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.3149999999999999,"15min":1.345},"min":{"1min":0.84999999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.595,"5min":1.847,"15min":1.9390000000000001},"last":1.2589999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.175,"15min":1.212},"min":{"1min":0.80000000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6220000000000001,"5min":1.8360000000000001,"15min":2.008},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.202},"min":{"1min":0.67900000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.286,"5min":1.706,"15min":1.77},"last":1.2969999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2330000000000001,"15min":1.2470000000000001},"min":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.5960000000000001,"5min":1.823,"15min":1.823},"last":1.002},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2749999999999999,"15min":1.3120000000000001},"min":{"1min":0.98799999999999999,"5min":0.745,"15min":0.745},"max":{"1min":1.619,"5min":1.962,"15min":1.962},"last":1.046}]},{"osd":3,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2829999999999999,"15min":1.333},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4830000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.526},{"interface":"front","average":{"1min":1.294,"5min":1.3919999999999999,"15min":1.405},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.5629999999999999,"5min":1.911,"15min":1.911},"last":1.105}]},{"osd":4,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.389,"15min":1.403},"min":{"1min":0.77300000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.6479999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.456},{"interface":"front","average":{"1min":1.476,"5min":1.546,"15min":1.548},"min":{"1min":1.0680000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.1179999999999999},"last":1.498}]},{"osd":5,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.88300000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0840000000000001,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4710000000000001,"15min":1.486},"min":{"1min":0.91900000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.786,"5min":2.0990000000000002,"15min":2.1970000000000001},"last":1.6910000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.506,"15min":1.5009999999999999},"min":{"1min":0.96799999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0270000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.5289999999999999,"15min":1.536},"min":{"1min":0.998,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.9990000000000001,"5min":2.157,"15min":2.6120000000000001},"last":1.3839999999999999}]}]},{"osd":5,"up_from":14,"seq":60129543325,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48452,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323388,"statfs":{"total":96636764160,"available":96587149312,"internally_reserved":0,"allocated":3461120,"data_stored":1694367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.111,"15min":1.1879999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.546,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.143},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1919999999999999,"15min":1.22},"min":{"1min":0.79400000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.238}]},{"osd":1,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.286,"15min":1.306},"min":{"1min":1.034,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.631,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.095},{"interface":"front","average":{"1min":1.2,"5min":1.1839999999999999,"15min":1.248},"min":{"1min":0.80000000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.931,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.3560000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2909999999999999,"15min":1.333},"min":{"1min":0.82799999999999996,"5min":0.63,"15min":0.63},"max":{"1min":2.024,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.238,"15min":1.3009999999999999},"min":{"1min":0.82599999999999996,"5min":0.746,"15min":0.746},"max":{"1min":1.8600000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":0.93799999999999994}]},{"osd":3,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3899999999999999,"15min":1.403},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.681,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":1.0049999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.827,"5min":2.75,"15min":2.75},"last":1.0189999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3160000000000001,"15min":1.298},"min":{"1min":0.86899999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6839999999999999,"5min":2.774,"15min":2.774},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.4430000000000001},"min":{"1min":1.1879999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.865,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.532}]},{"osd":6,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.464,"15min":1.4390000000000001},"min":{"1min":1.0289999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9399999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.454},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5009999999999999,"15min":1.4870000000000001},"min":{"1min":1.0900000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9790000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.0580000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.5920000000000001,"15min":1.556},"min":{"1min":1.1040000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.153,"5min":2.1680000000000001,"15min":2.4609999999999999},"last":1.403},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6220000000000001,"15min":1.6060000000000001},"min":{"1min":1.272,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.044,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.1859999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543325,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66420,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305420,"statfs":{"total":96636764160,"available":96568750080,"internally_reserved":0,"allocated":3448832,"data_stored":1692195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.151,"15min":1.1899999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.4510000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.575},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1759999999999999,"15min":1.1950000000000001},"min":{"1min":0.76100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6100000000000001,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.083}]},{"osd":1,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.157,"15min":1.2370000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.673,"5min":1.71,"15min":1.78},"last":1.268},{"interface":"front","average":{"1min":1.236,"5min":1.232,"15min":1.28},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.792,"5min":1.792,"15min":2.0499999999999998},"last":1.3149999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.86299999999999999,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.5169999999999999,"5min":1.714,"15min":1.865},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3520000000000001,"15min":1.379},"min":{"1min":1.006,"5min":0.87,"15min":0.87},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.6319999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.5980000000000001,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.4279999999999999},"min":{"1min":0.93000000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.915,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.1459999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.357,"15min":1.3400000000000001},"min":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.0670000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.427},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3899999999999999,"15min":1.349},"min":{"1min":0.89900000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":0.97499999999999998}]},{"osd":6,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.417,"15min":1.3939999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.956},"last":1.466},{"interface":"front","average":{"1min":1.609,"5min":1.538,"15min":1.522},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.3279999999999998,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.512}]},{"osd":7,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.516},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.22},{"interface":"front","average":{"1min":1.665,"5min":1.645,"15min":1.639},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.1200000000000001,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.5509999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543325,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56748,"kb_used_data":3360,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315092,"statfs":{"total":96636764160,"available":96578654208,"internally_reserved":0,"allocated":3440640,"data_stored":1687372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.90900000000000003,"15min":0.94799999999999995},"min":{"1min":0.48599999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.107,"5min":1.1479999999999999,"15min":1.2090000000000001},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.0169999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.03,"15min":1.075},"min":{"1min":0.55900000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.855},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.133,"5min":1.0980000000000001,"15min":1.123},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.169}]},{"osd":2,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0960000000000001,"15min":1.1890000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.823,"5min":1.8700000000000001,"15min":2.0539999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3400000000000001,"15min":1.431},"min":{"1min":0.622,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.909,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.91800000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.482,"15min":1.51},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2160000000000002,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.506},{"interface":"front","average":{"1min":1.379,"5min":1.4330000000000001,"15min":1.456},"min":{"1min":0.84099999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.823,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.6739999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.427,"15min":1.4850000000000001},"min":{"1min":0.89900000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8380000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.226},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.52,"15min":1.5529999999999999},"min":{"1min":0.871,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9910000000000001,"5min":2.0339999999999998,"15min":2.153},"last":1.462}]},{"osd":6,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.5,"15min":1.522},"min":{"1min":0.72899999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.335,"5min":2.335,"15min":2.4329999999999998},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4750000000000001,"15min":1.496},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.194,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.6000000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5760000000000001,"15min":1.599},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1859999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.268},{"interface":"front","average":{"1min":1.698,"5min":1.6599999999999999,"15min":1.677},"min":{"1min":0.98199999999999998,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.5,"5min":2.569,"15min":2.6070000000000002},"last":1.5489999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56752,"kb_used_data":3364,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315088,"statfs":{"total":96636764160,"available":96578650112,"internally_reserved":0,"allocated":3444736,"data_stored":1692193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.5680000000000001,"5min":1.597,"15min":2.0089999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.036,"5min":1.0409999999999999,"15min":1.131},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.3149999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0920000000000001,"15min":1.1519999999999999},"min":{"1min":0.51900000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.6970000000000001,"5min":1.847,"15min":1.847},"last":1.165},{"interface":"front","average":{"1min":1.101,"5min":1.153,"15min":1.2350000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.784,"5min":1.784,"15min":1.9530000000000001},"last":1.0600000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0549999999999999,"15min":1.111},"min":{"1min":0.52900000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3740000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2490000000000001,"15min":1.353},"min":{"1min":0.54800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.3149999999999999,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.4330000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.454,"15min":1.4910000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.5529999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.524},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.444,"15min":1.4890000000000001},"min":{"1min":0.81100000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0409999999999999,"5min":2.1789999999999998,"15min":2.3140000000000001},"last":1.4790000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4910000000000001,"15min":1.548},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.95,"5min":2.1749999999999998,"15min":2.488},"last":1.544},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4530000000000001,"15min":1.4710000000000001},"min":{"1min":0.72899999999999998,"5min":0.627,"15min":0.627},"max":{"1min":2.073,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.8009999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.506,"15min":1.5269999999999999},"min":{"1min":0.95099999999999996,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.4079999999999999},"last":1.643},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5489999999999999,"15min":1.613},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.5569999999999999},"last":1.6799999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.617,"15min":1.661},"min":{"1min":0.98899999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.1960000000000002,"5min":2.4940000000000002,"15min":2.7890000000000001},"last":1.835},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.603,"15min":1.6419999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.161,"5min":2.2799999999999998,"15min":2.3319999999999999},"last":1.8859999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52584,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319256,"statfs":{"total":96636764160,"available":96582918144,"internally_reserved":0,"allocated":3432448,"data_stored":1685794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.88,"15min":0.92600000000000005},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0289999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.876},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0640000000000001,"15min":1.125},"min":{"1min":0.58299999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.647,"5min":1.907,"15min":1.907},"last":1.101}]},{"osd":2,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.97599999999999998,"15min":1.0289999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.601,"5min":1.853,"15min":1.853},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.1519999999999999,"15min":1.2370000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.5289999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.2130000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.2729999999999999,"15min":1.3740000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6559999999999999,"5min":2.1659999999999999,"15min":2.4369999999999998},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.167,"15min":1.274},"min":{"1min":0.63900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7529999999999999,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.228,"15min":1.333},"min":{"1min":0.77500000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.756,"5min":1.756,"15min":2.1259999999999999},"last":1.484},{"interface":"front","average":{"1min":1.335,"5min":1.4390000000000001,"15min":1.516},"min":{"1min":0.90200000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.6459999999999999,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.5189999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4410000000000001,"15min":1.4810000000000001},"min":{"1min":0.85699999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.1070000000000002,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.3740000000000001,"15min":1.4730000000000001},"min":{"1min":0.82599999999999996,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.833,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.696}]},{"osd":6,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.5309999999999999,"15min":1.619},"min":{"1min":0.80400000000000005,"5min":0.748,"15min":0.748},"max":{"1min":1.919,"5min":2.4540000000000002,"15min":2.7200000000000002},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.593},"min":{"1min":0.84599999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2530000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.431}]},{"osd":7,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.6120000000000001,"15min":1.6890000000000001},"min":{"1min":1.109,"5min":0.628,"15min":0.628},"max":{"1min":2.339,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.484,"15min":1.6100000000000001},"min":{"1min":0.95199999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9430000000000001,"5min":2.1720000000000002,"15min":2.2930000000000001},"last":1.7789999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543324,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61488,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310352,"statfs":{"total":96636764160,"available":96573800448,"internally_reserved":0,"allocated":4034560,"data_stored":2276766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.90900000000000003,"15min":0.93300000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3440000000000001,"5min":1.704,"15min":1.891},"last":1.284},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0049999999999999,"15min":1.1279999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.6539999999999999,"15min":1.9299999999999999},"last":0.878}]},{"osd":2,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.0669999999999999,"15min":1.157},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4359999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.123,"15min":1.202},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.657,"5min":1.893,"15min":1.893},"last":1.109}]},{"osd":3,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0640000000000001,"15min":1.141},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7589999999999999,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2270000000000001,"15min":1.351},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.603,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.4330000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.26,"15min":1.3500000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.923,"5min":1.923,"15min":2.0070000000000001},"last":1.206},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.413,"15min":1.5009999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.98,"5min":2.1070000000000002,"15min":2.601},"last":1.6000000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.4430000000000001,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":2.089,"15min":2.3279999999999998},"last":1.54},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.385,"15min":1.468},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.901,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.3420000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.5229999999999999,"15min":1.5960000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.669},"last":1.9630000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5109999999999999,"15min":1.579},"min":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0129999999999999,"5min":2.327,"15min":2.3769999999999998},"last":1.454}]},{"osd":7,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.6080000000000001,"15min":1.6879999999999999},"min":{"1min":0.76700000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0670000000000002,"5min":2.2970000000000002,"15min":2.5270000000000001},"last":1.756},{"interface":"front","average":{"1min":1.482,"5min":1.5509999999999999,"15min":1.6259999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0270000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.7150000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:12:45.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:12:45.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.725+0000 7f371e393640 1 Processor -- start 2023-10-16T14:12:45.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.726+0000 7f371e393640 1 -- start start 2023-10-16T14:12:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.727+0000 7f371e393640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f371815cfc0 0x7f37180b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.727+0000 7f371e393640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37180d2640 0x7f37180b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.727+0000 7f371e393640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180d15a0 con 0x7f371815cbe0 2023-10-16T14:12:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.727+0000 7f371e393640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37180d1880 con 0x7f371815cfc0 2023-10-16T14:12:45.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.727+0000 7f371e393640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37180d1b60 con 0x7f37180d2640 2023-10-16T14:12:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371d391640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f371815cbe0 0x7f37180d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50966/0 (socket says 172.21.15.60:50966) 2023-10-16T14:12:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371d391640 1 -- 172.21.15.60:0/4217232858 learned_addr learned my addr 172.21.15.60:0/4217232858 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371cb90640 1 --2- 172.21.15.60:0/4217232858 >> v2:172.21.15.60:3300/0 conn(0x7f371815cfc0 0x7f37180b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371cb90640 1 -- 172.21.15.60:0/4217232858 >> v2:172.21.15.144:3300/0 conn(0x7f37180d2640 msgr2=0x7f37180b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:45.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371cb90640 1 --2- 172.21.15.60:0/4217232858 >> v2:172.21.15.144:3300/0 conn(0x7f37180d2640 0x7f37180b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371cb90640 1 -- 172.21.15.60:0/4217232858 >> v1:172.21.15.60:6789/0 conn(0x7f371815cbe0 legacy=0x7f37180d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.728+0000 7f371cb90640 1 -- 172.21.15.60:0/4217232858 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180d1e40 con 0x7f371815cfc0 2023-10-16T14:12:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.729+0000 7f371cb90640 1 --2- 172.21.15.60:0/4217232858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f371815cfc0 0x7f37180b4570 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f37180b5b70 tx=0x7f370c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2612386a542336fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f370bfff640 1 -- 172.21.15.60:0/4217232858 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c030610 con 0x7f371815cfc0 2023-10-16T14:12:45.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f370bfff640 1 -- 172.21.15.60:0/4217232858 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c030770 con 0x7f371815cfc0 2023-10-16T14:12:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f370bfff640 1 -- 172.21.15.60:0/4217232858 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c030b00 con 0x7f371815cfc0 2023-10-16T14:12:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f371e393640 1 -- 172.21.15.60:0/4217232858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f371815cfc0 msgr2=0x7f37180b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f371e393640 1 --2- 172.21.15.60:0/4217232858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f371815cfc0 0x7f37180b4570 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f37180b5b70 tx=0x7f370c02b960 comp rx=0 tx=0).stop 2023-10-16T14:12:45.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f371e393640 1 -- 172.21.15.60:0/4217232858 shutdown_connections 2023-10-16T14:12:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f371e393640 1 --2- 172.21.15.60:0/4217232858 >> v2:172.21.15.144:3300/0 conn(0x7f37180d2640 0x7f37180b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f371e393640 1 --2- 172.21.15.60:0/4217232858 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f371815cfc0 0x7f37180b4570 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.730+0000 7f371e393640 1 -- 172.21.15.60:0/4217232858 >> 172.21.15.60:0/4217232858 conn(0x7f37180cf570 msgr2=0x7f37180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.731+0000 7f371e393640 1 -- 172.21.15.60:0/4217232858 shutdown_connections 2023-10-16T14:12:45.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.731+0000 7f371e393640 1 -- 172.21.15.60:0/4217232858 wait complete. 2023-10-16T14:12:45.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.732+0000 7f371e393640 1 Processor -- start 2023-10-16T14:12:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.733+0000 7f371e393640 1 -- start start 2023-10-16T14:12:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.733+0000 7f371e393640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 0x7f37181f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.733+0000 7f371e393640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371815cbe0 0x7f37181f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371e393640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371815cfc0 0x7f37181f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371e393640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180cb4f0 con 0x7f37180d2640 2023-10-16T14:12:45.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371e393640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180c8230 con 0x7f371815cbe0 2023-10-16T14:12:45.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371e393640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180cb210 con 0x7f371815cfc0 2023-10-16T14:12:45.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 0x7f37181f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 0x7f37181f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48434/0 (socket says 172.21.15.60:48434) 2023-10-16T14:12:45.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 -- 172.21.15.60:0/2901004861 learned_addr learned my addr 172.21.15.60:0/2901004861 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:45.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 -- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371815cfc0 msgr2=0x7f37181f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371815cfc0 0x7f37181f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 -- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371815cbe0 msgr2=0x7f37181f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371815cbe0 0x7f37181f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.734+0000 7f371d391640 1 -- 172.21.15.60:0/2901004861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c009400 con 0x7f37180d2640 2023-10-16T14:12:45.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.735+0000 7f371d391640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 0x7f37181f3440 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f371400c4d0 tx=0x7f371400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:45.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.735+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714005ce0 con 0x7f37180d2640 2023-10-16T14:12:45.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.735+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3714005e40 con 0x7f37180d2640 2023-10-16T14:12:45.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.735+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714007270 con 0x7f37180d2640 2023-10-16T14:12:45.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.735+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37180c7be0 con 0x7f37180d2640 2023-10-16T14:12:45.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.735+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37180ca9a0 con 0x7f37180d2640 2023-10-16T14:12:45.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.736+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37180d11e0 con 0x7f37180d2640 2023-10-16T14:12:45.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.737+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f37140073d0 con 0x7f37180d2640 2023-10-16T14:12:45.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.739+0000 7f3709ffb640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f36fc06b0e0 0x7f36fc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:45.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.739+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f3714088ec0 con 0x7f37180d2640 2023-10-16T14:12:45.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.739+0000 7f371cb90640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f36fc06b0e0 0x7f36fc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:45.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.740+0000 7f371cb90640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f36fc06b0e0 0x7f36fc06d5a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f370c036000 tx=0x7f370c030000 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:45.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.744+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3714053c80 con 0x7f37180d2640 2023-10-16T14:12:45.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.945+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 --> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37180d2120 con 0x7f36fc06b0e0 2023-10-16T14:12:45.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.969+0000 7f3709ffb640 1 -- 172.21.15.60:0/2901004861 <== mgr.8393 v2:172.21.15.144:6832/948422863 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286891 (secure 0 0 0) 0x7f370c030410 con 0x7f36fc06b0e0 2023-10-16T14:12:45.974 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:45.978 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:12:45.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f36fc06b0e0 msgr2=0x7f36fc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f36fc06b0e0 0x7f36fc06d5a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f370c036000 tx=0x7f370c030000 comp rx=0 tx=0).stop 2023-10-16T14:12:45.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 msgr2=0x7f37181f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:45.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 0x7f37181f3440 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f371400c4d0 tx=0x7f371400c990 comp rx=0 tx=0).stop 2023-10-16T14:12:45.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 shutdown_connections 2023-10-16T14:12:45.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f36fc06b0e0 0x7f36fc06d5a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f371815cfc0 0x7f37181f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f371815cbe0 0x7f37181f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 --2- 172.21.15.60:0/2901004861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37180d2640 0x7f37181f3440 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:45.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 >> 172.21.15.60:0/2901004861 conn(0x7f37180cf570 msgr2=0x7f37180b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:45.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 shutdown_connections 2023-10-16T14:12:45.983 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:45.978+0000 7f371e393640 1 -- 172.21.15.60:0/2901004861 wait complete. 2023-10-16T14:12:46.004 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T14:12:45.500377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2507,"num_read_kb":5248,"num_write":693,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2837,"ondisk_log_size":2837,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":439872,"kb_used_data":28056,"kb_used_omap":12,"kb_used_meta":411699,"kb_avail":754534848,"statfs":{"total":773094113280,"available":772643684352,"internally_reserved":0,"allocated":28729344,"data_stored":14697008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":421580344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":201,"num_write":63,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002122"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947804+0000","last_change":"2023-10-16T14:12:40.942796+0000","last_active":"2023-10-16T14:12:40.947804+0000","last_peered":"2023-10-16T14:12:40.947804+0000","last_clean":"2023-10-16T14:12:40.947804+0000","last_became_active":"2023-10-16T14:12:40.942519+0000","last_became_peered":"2023-10-16T14:12:40.942519+0000","last_unstale":"2023-10-16T14:12:40.947804+0000","last_undegraded":"2023-10-16T14:12:40.947804+0000","last_fullsized":"2023-10-16T14:12:40.947804+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:15:34.366625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940966+0000","last_change":"2023-10-16T14:12:40.936194+0000","last_active":"2023-10-16T14:12:40.940966+0000","last_peered":"2023-10-16T14:12:40.940966+0000","last_clean":"2023-10-16T14:12:40.940966+0000","last_became_active":"2023-10-16T14:12:40.934991+0000","last_became_peered":"2023-10-16T14:12:40.934991+0000","last_unstale":"2023-10-16T14:12:40.940966+0000","last_undegraded":"2023-10-16T14:12:40.940966+0000","last_fullsized":"2023-10-16T14:12:40.940966+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:50:59.307365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943144+0000","last_change":"2023-10-16T14:12:40.940529+0000","last_active":"2023-10-16T14:12:40.943144+0000","last_peered":"2023-10-16T14:12:40.943144+0000","last_clean":"2023-10-16T14:12:40.943144+0000","last_became_active":"2023-10-16T14:12:40.940276+0000","last_became_peered":"2023-10-16T14:12:40.940276+0000","last_unstale":"2023-10-16T14:12:40.943144+0000","last_undegraded":"2023-10-16T14:12:40.943144+0000","last_fullsized":"2023-10-16T14:12:40.943144+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:51:34.111470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947686+0000","last_change":"2023-10-16T14:12:40.941002+0000","last_active":"2023-10-16T14:12:41.947686+0000","last_peered":"2023-10-16T14:12:41.947686+0000","last_clean":"2023-10-16T14:12:41.947686+0000","last_became_active":"2023-10-16T14:12:40.940579+0000","last_became_peered":"2023-10-16T14:12:40.940579+0000","last_unstale":"2023-10-16T14:12:41.947686+0000","last_undegraded":"2023-10-16T14:12:41.947686+0000","last_fullsized":"2023-10-16T14:12:41.947686+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:49:55.325274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.944007+0000","last_change":"2023-10-16T14:12:40.940944+0000","last_active":"2023-10-16T14:12:40.944007+0000","last_peered":"2023-10-16T14:12:40.944007+0000","last_clean":"2023-10-16T14:12:40.944007+0000","last_became_active":"2023-10-16T14:12:40.939102+0000","last_became_peered":"2023-10-16T14:12:40.939102+0000","last_unstale":"2023-10-16T14:12:40.944007+0000","last_undegraded":"2023-10-16T14:12:40.944007+0000","last_fullsized":"2023-10-16T14:12:40.944007+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:46.139470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943077+0000","last_change":"2023-10-16T14:12:40.940326+0000","last_active":"2023-10-16T14:12:40.943077+0000","last_peered":"2023-10-16T14:12:40.943077+0000","last_clean":"2023-10-16T14:12:40.943077+0000","last_became_active":"2023-10-16T14:12:40.940031+0000","last_became_peered":"2023-10-16T14:12:40.940031+0000","last_unstale":"2023-10-16T14:12:40.943077+0000","last_undegraded":"2023-10-16T14:12:40.943077+0000","last_fullsized":"2023-10-16T14:12:40.943077+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:33.995995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954502+0000","last_change":"2023-10-16T14:12:40.953749+0000","last_active":"2023-10-16T14:12:40.954502+0000","last_peered":"2023-10-16T14:12:40.954502+0000","last_clean":"2023-10-16T14:12:40.954502+0000","last_became_active":"2023-10-16T14:12:40.953178+0000","last_became_peered":"2023-10-16T14:12:40.953178+0000","last_unstale":"2023-10-16T14:12:40.954502+0000","last_undegraded":"2023-10-16T14:12:40.954502+0000","last_fullsized":"2023-10-16T14:12:40.954502+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:47.225375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.942290+0000","last_change":"2023-10-16T14:12:40.935382+0000","last_active":"2023-10-16T14:12:40.942290+0000","last_peered":"2023-10-16T14:12:40.942290+0000","last_clean":"2023-10-16T14:12:40.942290+0000","last_became_active":"2023-10-16T14:12:40.934479+0000","last_became_peered":"2023-10-16T14:12:40.934479+0000","last_unstale":"2023-10-16T14:12:40.942290+0000","last_undegraded":"2023-10-16T14:12:40.942290+0000","last_fullsized":"2023-10-16T14:12:40.942290+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:02:13.601816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937912+0000","last_change":"2023-10-16T14:12:40.942033+0000","last_active":"2023-10-16T14:12:41.937912+0000","last_peered":"2023-10-16T14:12:41.937912+0000","last_clean":"2023-10-16T14:12:41.937912+0000","last_became_active":"2023-10-16T14:12:40.941750+0000","last_became_peered":"2023-10-16T14:12:40.941750+0000","last_unstale":"2023-10-16T14:12:41.937912+0000","last_undegraded":"2023-10-16T14:12:41.937912+0000","last_fullsized":"2023-10-16T14:12:41.937912+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:07:40.135762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947417+0000","last_change":"2023-10-16T14:12:40.929102+0000","last_active":"2023-10-16T14:12:41.947417+0000","last_peered":"2023-10-16T14:12:41.947417+0000","last_clean":"2023-10-16T14:12:41.947417+0000","last_became_active":"2023-10-16T14:12:40.926562+0000","last_became_peered":"2023-10-16T14:12:40.926562+0000","last_unstale":"2023-10-16T14:12:41.947417+0000","last_undegraded":"2023-10-16T14:12:41.947417+0000","last_fullsized":"2023-10-16T14:12:41.947417+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:34.169245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406822+0000","last_change":"2023-10-16T14:12:40.947482+0000","last_active":"2023-10-16T14:12:42.406822+0000","last_peered":"2023-10-16T14:12:42.406822+0000","last_clean":"2023-10-16T14:12:42.406822+0000","last_became_active":"2023-10-16T14:12:40.947103+0000","last_became_peered":"2023-10-16T14:12:40.947103+0000","last_unstale":"2023-10-16T14:12:42.406822+0000","last_undegraded":"2023-10-16T14:12:42.406822+0000","last_fullsized":"2023-10-16T14:12:42.406822+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:12:57.076113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.937575+0000","last_change":"2023-10-16T14:12:40.931890+0000","last_active":"2023-10-16T14:12:40.937575+0000","last_peered":"2023-10-16T14:12:40.937575+0000","last_clean":"2023-10-16T14:12:40.937575+0000","last_became_active":"2023-10-16T14:12:40.931616+0000","last_became_peered":"2023-10-16T14:12:40.931616+0000","last_unstale":"2023-10-16T14:12:40.937575+0000","last_undegraded":"2023-10-16T14:12:40.937575+0000","last_fullsized":"2023-10-16T14:12:40.937575+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:20:12.643100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.934747+0000","last_change":"2023-10-16T14:12:40.928540+0000","last_active":"2023-10-16T14:12:40.934747+0000","last_peered":"2023-10-16T14:12:40.934747+0000","last_clean":"2023-10-16T14:12:40.934747+0000","last_became_active":"2023-10-16T14:12:40.928261+0000","last_became_peered":"2023-10-16T14:12:40.928261+0000","last_unstale":"2023-10-16T14:12:40.934747+0000","last_undegraded":"2023-10-16T14:12:40.934747+0000","last_fullsized":"2023-10-16T14:12:40.934747+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:55.136856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946892+0000","last_change":"2023-10-16T14:12:40.934086+0000","last_active":"2023-10-16T14:12:41.946892+0000","last_peered":"2023-10-16T14:12:41.946892+0000","last_clean":"2023-10-16T14:12:41.946892+0000","last_became_active":"2023-10-16T14:12:40.933814+0000","last_became_peered":"2023-10-16T14:12:40.933814+0000","last_unstale":"2023-10-16T14:12:41.946892+0000","last_undegraded":"2023-10-16T14:12:41.946892+0000","last_fullsized":"2023-10-16T14:12:41.946892+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:05.018323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.946586+0000","last_change":"2023-10-16T14:12:40.941048+0000","last_active":"2023-10-16T14:12:40.946586+0000","last_peered":"2023-10-16T14:12:40.946586+0000","last_clean":"2023-10-16T14:12:40.946586+0000","last_became_active":"2023-10-16T14:12:40.940603+0000","last_became_peered":"2023-10-16T14:12:40.940603+0000","last_unstale":"2023-10-16T14:12:40.946586+0000","last_undegraded":"2023-10-16T14:12:40.946586+0000","last_fullsized":"2023-10-16T14:12:40.946586+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:29:38.530118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.942269+0000","last_change":"2023-10-16T14:12:40.935075+0000","last_active":"2023-10-16T14:12:40.942269+0000","last_peered":"2023-10-16T14:12:40.942269+0000","last_clean":"2023-10-16T14:12:40.942269+0000","last_became_active":"2023-10-16T14:12:40.933991+0000","last_became_peered":"2023-10-16T14:12:40.933991+0000","last_unstale":"2023-10-16T14:12:40.942269+0000","last_undegraded":"2023-10-16T14:12:40.942269+0000","last_fullsized":"2023-10-16T14:12:40.942269+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:56:48.321908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.938109+0000","last_change":"2023-10-16T14:12:40.937586+0000","last_active":"2023-10-16T14:12:41.938109+0000","last_peered":"2023-10-16T14:12:41.938109+0000","last_clean":"2023-10-16T14:12:41.938109+0000","last_became_active":"2023-10-16T14:12:40.937158+0000","last_became_peered":"2023-10-16T14:12:40.937158+0000","last_unstale":"2023-10-16T14:12:41.938109+0000","last_undegraded":"2023-10-16T14:12:41.938109+0000","last_fullsized":"2023-10-16T14:12:41.938109+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:29.205963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941296+0000","last_change":"2023-10-16T14:12:40.942201+0000","last_active":"2023-10-16T14:12:41.941296+0000","last_peered":"2023-10-16T14:12:41.941296+0000","last_clean":"2023-10-16T14:12:41.941296+0000","last_became_active":"2023-10-16T14:12:40.941912+0000","last_became_peered":"2023-10-16T14:12:40.941912+0000","last_unstale":"2023-10-16T14:12:41.941296+0000","last_undegraded":"2023-10-16T14:12:41.941296+0000","last_fullsized":"2023-10-16T14:12:41.941296+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:03.456935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947963+0000","last_change":"2023-10-16T14:12:40.945569+0000","last_active":"2023-10-16T14:12:40.947963+0000","last_peered":"2023-10-16T14:12:40.947963+0000","last_clean":"2023-10-16T14:12:40.947963+0000","last_became_active":"2023-10-16T14:12:40.945092+0000","last_became_peered":"2023-10-16T14:12:40.945092+0000","last_unstale":"2023-10-16T14:12:40.947963+0000","last_undegraded":"2023-10-16T14:12:40.947963+0000","last_fullsized":"2023-10-16T14:12:40.947963+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:42.525032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947289+0000","last_change":"2023-10-16T14:12:40.943427+0000","last_active":"2023-10-16T14:12:40.947289+0000","last_peered":"2023-10-16T14:12:40.947289+0000","last_clean":"2023-10-16T14:12:40.947289+0000","last_became_active":"2023-10-16T14:12:40.943169+0000","last_became_peered":"2023-10-16T14:12:40.943169+0000","last_unstale":"2023-10-16T14:12:40.947289+0000","last_undegraded":"2023-10-16T14:12:40.947289+0000","last_fullsized":"2023-10-16T14:12:40.947289+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:28.599827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.928203+0000","last_change":"2023-10-16T14:12:40.923455+0000","last_active":"2023-10-16T14:12:40.928203+0000","last_peered":"2023-10-16T14:12:40.928203+0000","last_clean":"2023-10-16T14:12:40.928203+0000","last_became_active":"2023-10-16T14:12:40.923158+0000","last_became_peered":"2023-10-16T14:12:40.923158+0000","last_unstale":"2023-10-16T14:12:40.928203+0000","last_undegraded":"2023-10-16T14:12:40.928203+0000","last_fullsized":"2023-10-16T14:12:40.928203+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:27.954952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940934+0000","last_change":"2023-10-16T14:12:40.937953+0000","last_active":"2023-10-16T14:12:40.940934+0000","last_peered":"2023-10-16T14:12:40.940934+0000","last_clean":"2023-10-16T14:12:40.940934+0000","last_became_active":"2023-10-16T14:12:40.937007+0000","last_became_peered":"2023-10-16T14:12:40.937007+0000","last_unstale":"2023-10-16T14:12:40.940934+0000","last_undegraded":"2023-10-16T14:12:40.940934+0000","last_fullsized":"2023-10-16T14:12:40.940934+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:31.540683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.946610+0000","last_change":"2023-10-16T14:12:40.945729+0000","last_active":"2023-10-16T14:12:40.946610+0000","last_peered":"2023-10-16T14:12:40.946610+0000","last_clean":"2023-10-16T14:12:40.946610+0000","last_became_active":"2023-10-16T14:12:40.945426+0000","last_became_peered":"2023-10-16T14:12:40.945426+0000","last_unstale":"2023-10-16T14:12:40.946610+0000","last_undegraded":"2023-10-16T14:12:40.946610+0000","last_fullsized":"2023-10-16T14:12:40.946610+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:53:50.661572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940888+0000","last_change":"2023-10-16T14:12:40.936979+0000","last_active":"2023-10-16T14:12:40.940888+0000","last_peered":"2023-10-16T14:12:40.940888+0000","last_clean":"2023-10-16T14:12:40.940888+0000","last_became_active":"2023-10-16T14:12:40.936692+0000","last_became_peered":"2023-10-16T14:12:40.936692+0000","last_unstale":"2023-10-16T14:12:40.940888+0000","last_undegraded":"2023-10-16T14:12:40.940888+0000","last_fullsized":"2023-10-16T14:12:40.940888+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:21.402564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.937722+0000","last_change":"2023-10-16T14:12:40.936053+0000","last_active":"2023-10-16T14:12:40.937722+0000","last_peered":"2023-10-16T14:12:40.937722+0000","last_clean":"2023-10-16T14:12:40.937722+0000","last_became_active":"2023-10-16T14:12:40.935237+0000","last_became_peered":"2023-10-16T14:12:40.935237+0000","last_unstale":"2023-10-16T14:12:40.937722+0000","last_undegraded":"2023-10-16T14:12:40.937722+0000","last_fullsized":"2023-10-16T14:12:40.937722+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:30.139744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.938095+0000","last_change":"2023-10-16T14:12:40.930421+0000","last_active":"2023-10-16T14:12:40.938095+0000","last_peered":"2023-10-16T14:12:40.938095+0000","last_clean":"2023-10-16T14:12:40.938095+0000","last_became_active":"2023-10-16T14:12:40.926949+0000","last_became_peered":"2023-10-16T14:12:40.926949+0000","last_unstale":"2023-10-16T14:12:40.938095+0000","last_undegraded":"2023-10-16T14:12:40.938095+0000","last_fullsized":"2023-10-16T14:12:40.938095+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:04:55.514665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947943+0000","last_change":"2023-10-16T14:12:40.945378+0000","last_active":"2023-10-16T14:12:40.947943+0000","last_peered":"2023-10-16T14:12:40.947943+0000","last_clean":"2023-10-16T14:12:40.947943+0000","last_became_active":"2023-10-16T14:12:40.944713+0000","last_became_peered":"2023-10-16T14:12:40.944713+0000","last_unstale":"2023-10-16T14:12:40.947943+0000","last_undegraded":"2023-10-16T14:12:40.947943+0000","last_fullsized":"2023-10-16T14:12:40.947943+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:03.534397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947966+0000","last_change":"2023-10-16T14:12:40.928920+0000","last_active":"2023-10-16T14:12:41.947966+0000","last_peered":"2023-10-16T14:12:41.947966+0000","last_clean":"2023-10-16T14:12:41.947966+0000","last_became_active":"2023-10-16T14:12:40.928644+0000","last_became_peered":"2023-10-16T14:12:40.928644+0000","last_unstale":"2023-10-16T14:12:41.947966+0000","last_undegraded":"2023-10-16T14:12:41.947966+0000","last_fullsized":"2023-10-16T14:12:41.947966+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:00.458147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954420+0000","last_change":"2023-10-16T14:12:40.953157+0000","last_active":"2023-10-16T14:12:40.954420+0000","last_peered":"2023-10-16T14:12:40.954420+0000","last_clean":"2023-10-16T14:12:40.954420+0000","last_became_active":"2023-10-16T14:12:40.952857+0000","last_became_peered":"2023-10-16T14:12:40.952857+0000","last_unstale":"2023-10-16T14:12:40.954420+0000","last_undegraded":"2023-10-16T14:12:40.954420+0000","last_fullsized":"2023-10-16T14:12:40.954420+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:08.579769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954458+0000","last_change":"2023-10-16T14:12:40.953214+0000","last_active":"2023-10-16T14:12:40.954458+0000","last_peered":"2023-10-16T14:12:40.954458+0000","last_clean":"2023-10-16T14:12:40.954458+0000","last_became_active":"2023-10-16T14:12:40.952848+0000","last_became_peered":"2023-10-16T14:12:40.952848+0000","last_unstale":"2023-10-16T14:12:40.954458+0000","last_undegraded":"2023-10-16T14:12:40.954458+0000","last_fullsized":"2023-10-16T14:12:40.954458+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:33.884134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937637+0000","last_change":"2023-10-16T14:12:40.942082+0000","last_active":"2023-10-16T14:12:41.937637+0000","last_peered":"2023-10-16T14:12:41.937637+0000","last_clean":"2023-10-16T14:12:41.937637+0000","last_became_active":"2023-10-16T14:12:40.941383+0000","last_became_peered":"2023-10-16T14:12:40.941383+0000","last_unstale":"2023-10-16T14:12:41.937637+0000","last_undegraded":"2023-10-16T14:12:41.937637+0000","last_fullsized":"2023-10-16T14:12:41.937637+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:24.778415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.936782+0000","last_change":"2023-10-16T14:12:40.928736+0000","last_active":"2023-10-16T14:12:40.936782+0000","last_peered":"2023-10-16T14:12:40.936782+0000","last_clean":"2023-10-16T14:12:40.936782+0000","last_became_active":"2023-10-16T14:12:40.927346+0000","last_became_peered":"2023-10-16T14:12:40.927346+0000","last_unstale":"2023-10-16T14:12:40.936782+0000","last_undegraded":"2023-10-16T14:12:40.936782+0000","last_fullsized":"2023-10-16T14:12:40.936782+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:42.004797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"272'4","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.952022+0000","last_change":"2023-10-16T14:12:39.934680+0000","last_active":"2023-10-16T14:12:40.952022+0000","last_peered":"2023-10-16T14:12:40.952022+0000","last_clean":"2023-10-16T14:12:40.952022+0000","last_became_active":"2023-10-16T14:12:39.934364+0000","last_became_peered":"2023-10-16T14:12:39.934364+0000","last_unstale":"2023-10-16T14:12:40.952022+0000","last_undegraded":"2023-10-16T14:12:40.952022+0000","last_fullsized":"2023-10-16T14:12:40.952022+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:03.047367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407184+0000","last_change":"2023-10-16T14:12:40.949613+0000","last_active":"2023-10-16T14:12:42.407184+0000","last_peered":"2023-10-16T14:12:42.407184+0000","last_clean":"2023-10-16T14:12:42.407184+0000","last_became_active":"2023-10-16T14:12:40.949235+0000","last_became_peered":"2023-10-16T14:12:40.949235+0000","last_unstale":"2023-10-16T14:12:42.407184+0000","last_undegraded":"2023-10-16T14:12:42.407184+0000","last_fullsized":"2023-10-16T14:12:42.407184+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:05:35.973009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937619+0000","last_change":"2023-10-16T14:12:39.943517+0000","last_active":"2023-10-16T14:12:41.937619+0000","last_peered":"2023-10-16T14:12:41.937619+0000","last_clean":"2023-10-16T14:12:41.937619+0000","last_became_active":"2023-10-16T14:12:39.942151+0000","last_became_peered":"2023-10-16T14:12:39.942151+0000","last_unstale":"2023-10-16T14:12:41.937619+0000","last_undegraded":"2023-10-16T14:12:41.937619+0000","last_fullsized":"2023-10-16T14:12:41.937619+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:07.670532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947155+0000","last_change":"2023-10-16T14:12:40.944633+0000","last_active":"2023-10-16T14:12:40.947155+0000","last_peered":"2023-10-16T14:12:40.947155+0000","last_clean":"2023-10-16T14:12:40.947155+0000","last_became_active":"2023-10-16T14:12:40.944256+0000","last_became_peered":"2023-10-16T14:12:40.944256+0000","last_unstale":"2023-10-16T14:12:40.947155+0000","last_undegraded":"2023-10-16T14:12:40.947155+0000","last_fullsized":"2023-10-16T14:12:40.947155+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:02.484270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"272'2","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937135+0000","last_change":"2023-10-16T14:12:39.938871+0000","last_active":"2023-10-16T14:12:41.937135+0000","last_peered":"2023-10-16T14:12:41.937135+0000","last_clean":"2023-10-16T14:12:41.937135+0000","last_became_active":"2023-10-16T14:12:39.938538+0000","last_became_peered":"2023-10-16T14:12:39.938538+0000","last_unstale":"2023-10-16T14:12:41.937135+0000","last_undegraded":"2023-10-16T14:12:41.937135+0000","last_fullsized":"2023-10-16T14:12:41.937135+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:06.014963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947764+0000","last_change":"2023-10-16T14:12:40.945856+0000","last_active":"2023-10-16T14:12:40.947764+0000","last_peered":"2023-10-16T14:12:40.947764+0000","last_clean":"2023-10-16T14:12:40.947764+0000","last_became_active":"2023-10-16T14:12:40.944531+0000","last_became_peered":"2023-10-16T14:12:40.944531+0000","last_unstale":"2023-10-16T14:12:40.947764+0000","last_undegraded":"2023-10-16T14:12:40.947764+0000","last_fullsized":"2023-10-16T14:12:40.947764+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:14.849580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947198+0000","last_change":"2023-10-16T14:12:39.942390+0000","last_active":"2023-10-16T14:12:40.947198+0000","last_peered":"2023-10-16T14:12:40.947198+0000","last_clean":"2023-10-16T14:12:40.947198+0000","last_became_active":"2023-10-16T14:12:39.941990+0000","last_became_peered":"2023-10-16T14:12:39.941990+0000","last_unstale":"2023-10-16T14:12:40.947198+0000","last_undegraded":"2023-10-16T14:12:40.947198+0000","last_fullsized":"2023-10-16T14:12:40.947198+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:47.156586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943856+0000","last_change":"2023-10-16T14:12:40.940764+0000","last_active":"2023-10-16T14:12:40.943856+0000","last_peered":"2023-10-16T14:12:40.943856+0000","last_clean":"2023-10-16T14:12:40.943856+0000","last_became_active":"2023-10-16T14:12:40.939946+0000","last_became_peered":"2023-10-16T14:12:40.939946+0000","last_unstale":"2023-10-16T14:12:40.943856+0000","last_undegraded":"2023-10-16T14:12:40.943856+0000","last_fullsized":"2023-10-16T14:12:40.943856+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:34.790020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"272'1","reported_seq":12,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.944568+0000","last_change":"2023-10-16T14:12:39.943644+0000","last_active":"2023-10-16T14:12:40.944568+0000","last_peered":"2023-10-16T14:12:40.944568+0000","last_clean":"2023-10-16T14:12:40.944568+0000","last_became_active":"2023-10-16T14:12:39.943150+0000","last_became_peered":"2023-10-16T14:12:39.943150+0000","last_unstale":"2023-10-16T14:12:40.944568+0000","last_undegraded":"2023-10-16T14:12:40.944568+0000","last_fullsized":"2023-10-16T14:12:40.944568+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:50.187777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947306+0000","last_change":"2023-10-16T14:12:40.942714+0000","last_active":"2023-10-16T14:12:40.947306+0000","last_peered":"2023-10-16T14:12:40.947306+0000","last_clean":"2023-10-16T14:12:40.947306+0000","last_became_active":"2023-10-16T14:12:40.942424+0000","last_became_peered":"2023-10-16T14:12:40.942424+0000","last_unstale":"2023-10-16T14:12:40.947306+0000","last_undegraded":"2023-10-16T14:12:40.947306+0000","last_fullsized":"2023-10-16T14:12:40.947306+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:41.356096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940599+0000","last_change":"2023-10-16T14:12:39.936379+0000","last_active":"2023-10-16T14:12:41.940599+0000","last_peered":"2023-10-16T14:12:41.940599+0000","last_clean":"2023-10-16T14:12:41.940599+0000","last_became_active":"2023-10-16T14:12:39.935932+0000","last_became_peered":"2023-10-16T14:12:39.935932+0000","last_unstale":"2023-10-16T14:12:41.940599+0000","last_undegraded":"2023-10-16T14:12:41.940599+0000","last_fullsized":"2023-10-16T14:12:41.940599+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:35.875704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937938+0000","last_change":"2023-10-16T14:12:40.928538+0000","last_active":"2023-10-16T14:12:41.937938+0000","last_peered":"2023-10-16T14:12:41.937938+0000","last_clean":"2023-10-16T14:12:41.937938+0000","last_became_active":"2023-10-16T14:12:40.926549+0000","last_became_peered":"2023-10-16T14:12:40.926549+0000","last_unstale":"2023-10-16T14:12:41.937938+0000","last_undegraded":"2023-10-16T14:12:41.937938+0000","last_fullsized":"2023-10-16T14:12:41.937938+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:44.231128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917618+0000","last_change":"2023-10-16T14:12:39.928718+0000","last_active":"2023-10-16T14:12:40.917618+0000","last_peered":"2023-10-16T14:12:40.917618+0000","last_clean":"2023-10-16T14:12:40.917618+0000","last_became_active":"2023-10-16T14:12:39.928016+0000","last_became_peered":"2023-10-16T14:12:39.928016+0000","last_unstale":"2023-10-16T14:12:40.917618+0000","last_undegraded":"2023-10-16T14:12:40.917618+0000","last_fullsized":"2023-10-16T14:12:40.917618+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:19.422569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940134+0000","last_change":"2023-10-16T14:12:40.937821+0000","last_active":"2023-10-16T14:12:41.940134+0000","last_peered":"2023-10-16T14:12:41.940134+0000","last_clean":"2023-10-16T14:12:41.940134+0000","last_became_active":"2023-10-16T14:12:40.937502+0000","last_became_peered":"2023-10-16T14:12:40.937502+0000","last_unstale":"2023-10-16T14:12:41.940134+0000","last_undegraded":"2023-10-16T14:12:41.940134+0000","last_fullsized":"2023-10-16T14:12:41.940134+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:55.851366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.912696+0000","last_change":"2023-10-16T14:12:39.933667+0000","last_active":"2023-10-16T14:12:40.912696+0000","last_peered":"2023-10-16T14:12:40.912696+0000","last_clean":"2023-10-16T14:12:40.912696+0000","last_became_active":"2023-10-16T14:12:39.932845+0000","last_became_peered":"2023-10-16T14:12:39.932845+0000","last_unstale":"2023-10-16T14:12:40.912696+0000","last_undegraded":"2023-10-16T14:12:40.912696+0000","last_fullsized":"2023-10-16T14:12:40.912696+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:55.667818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941525+0000","last_change":"2023-10-16T14:12:40.942048+0000","last_active":"2023-10-16T14:12:41.941525+0000","last_peered":"2023-10-16T14:12:41.941525+0000","last_clean":"2023-10-16T14:12:41.941525+0000","last_became_active":"2023-10-16T14:12:40.938626+0000","last_became_peered":"2023-10-16T14:12:40.938626+0000","last_unstale":"2023-10-16T14:12:41.941525+0000","last_undegraded":"2023-10-16T14:12:41.941525+0000","last_fullsized":"2023-10-16T14:12:41.941525+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:59.528771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947939+0000","last_change":"2023-10-16T14:12:39.933244+0000","last_active":"2023-10-16T14:12:41.947939+0000","last_peered":"2023-10-16T14:12:41.947939+0000","last_clean":"2023-10-16T14:12:41.947939+0000","last_became_active":"2023-10-16T14:12:39.932799+0000","last_became_peered":"2023-10-16T14:12:39.932799+0000","last_unstale":"2023-10-16T14:12:41.947939+0000","last_undegraded":"2023-10-16T14:12:41.947939+0000","last_fullsized":"2023-10-16T14:12:41.947939+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:40.499697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947995+0000","last_change":"2023-10-16T14:12:40.943903+0000","last_active":"2023-10-16T14:12:41.947995+0000","last_peered":"2023-10-16T14:12:41.947995+0000","last_clean":"2023-10-16T14:12:41.947995+0000","last_became_active":"2023-10-16T14:12:40.943382+0000","last_became_peered":"2023-10-16T14:12:40.943382+0000","last_unstale":"2023-10-16T14:12:41.947995+0000","last_undegraded":"2023-10-16T14:12:41.947995+0000","last_fullsized":"2023-10-16T14:12:41.947995+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:36.547016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407020+0000","last_change":"2023-10-16T14:12:39.940712+0000","last_active":"2023-10-16T14:12:42.407020+0000","last_peered":"2023-10-16T14:12:42.407020+0000","last_clean":"2023-10-16T14:12:42.407020+0000","last_became_active":"2023-10-16T14:12:39.940444+0000","last_became_peered":"2023-10-16T14:12:39.940444+0000","last_unstale":"2023-10-16T14:12:42.407020+0000","last_undegraded":"2023-10-16T14:12:42.407020+0000","last_fullsized":"2023-10-16T14:12:42.407020+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:24.201772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406953+0000","last_change":"2023-10-16T14:12:40.946838+0000","last_active":"2023-10-16T14:12:42.406953+0000","last_peered":"2023-10-16T14:12:42.406953+0000","last_clean":"2023-10-16T14:12:42.406953+0000","last_became_active":"2023-10-16T14:12:40.946501+0000","last_became_peered":"2023-10-16T14:12:40.946501+0000","last_unstale":"2023-10-16T14:12:42.406953+0000","last_undegraded":"2023-10-16T14:12:42.406953+0000","last_fullsized":"2023-10-16T14:12:42.406953+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:53.980955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"272'3","reported_seq":15,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.937144+0000","last_change":"2023-10-16T14:12:39.941559+0000","last_active":"2023-10-16T14:12:41.937144+0000","last_peered":"2023-10-16T14:12:41.937144+0000","last_clean":"2023-10-16T14:12:41.937144+0000","last_became_active":"2023-10-16T14:12:39.941204+0000","last_became_peered":"2023-10-16T14:12:39.941204+0000","last_unstale":"2023-10-16T14:12:41.937144+0000","last_undegraded":"2023-10-16T14:12:41.937144+0000","last_fullsized":"2023-10-16T14:12:41.937144+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:59.495481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947664+0000","last_change":"2023-10-16T14:12:40.929769+0000","last_active":"2023-10-16T14:12:41.947664+0000","last_peered":"2023-10-16T14:12:41.947664+0000","last_clean":"2023-10-16T14:12:41.947664+0000","last_became_active":"2023-10-16T14:12:40.929451+0000","last_became_peered":"2023-10-16T14:12:40.929451+0000","last_unstale":"2023-10-16T14:12:41.947664+0000","last_undegraded":"2023-10-16T14:12:41.947664+0000","last_fullsized":"2023-10-16T14:12:41.947664+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:37.123028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"272'6","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.954409+0000","last_change":"2023-10-16T14:12:39.941004+0000","last_active":"2023-10-16T14:12:40.954409+0000","last_peered":"2023-10-16T14:12:40.954409+0000","last_clean":"2023-10-16T14:12:40.954409+0000","last_became_active":"2023-10-16T14:12:39.939179+0000","last_became_peered":"2023-10-16T14:12:39.939179+0000","last_unstale":"2023-10-16T14:12:40.954409+0000","last_undegraded":"2023-10-16T14:12:40.954409+0000","last_fullsized":"2023-10-16T14:12:40.954409+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:51.039455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.945759+0000","last_change":"2023-10-16T14:12:40.939835+0000","last_active":"2023-10-16T14:12:40.945759+0000","last_peered":"2023-10-16T14:12:40.945759+0000","last_clean":"2023-10-16T14:12:40.945759+0000","last_became_active":"2023-10-16T14:12:40.939592+0000","last_became_peered":"2023-10-16T14:12:40.939592+0000","last_unstale":"2023-10-16T14:12:40.945759+0000","last_undegraded":"2023-10-16T14:12:40.945759+0000","last_fullsized":"2023-10-16T14:12:40.945759+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:50.906044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940377+0000","last_change":"2023-10-16T14:12:39.927128+0000","last_active":"2023-10-16T14:12:40.940377+0000","last_peered":"2023-10-16T14:12:40.940377+0000","last_clean":"2023-10-16T14:12:40.940377+0000","last_became_active":"2023-10-16T14:12:39.926762+0000","last_became_peered":"2023-10-16T14:12:39.926762+0000","last_unstale":"2023-10-16T14:12:40.940377+0000","last_undegraded":"2023-10-16T14:12:40.940377+0000","last_fullsized":"2023-10-16T14:12:40.940377+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:10.984003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940179+0000","last_change":"2023-10-16T14:12:40.932173+0000","last_active":"2023-10-16T14:12:41.940179+0000","last_peered":"2023-10-16T14:12:41.940179+0000","last_clean":"2023-10-16T14:12:41.940179+0000","last_became_active":"2023-10-16T14:12:40.931555+0000","last_became_peered":"2023-10-16T14:12:40.931555+0000","last_unstale":"2023-10-16T14:12:41.940179+0000","last_undegraded":"2023-10-16T14:12:41.940179+0000","last_fullsized":"2023-10-16T14:12:41.940179+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:32.561673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917340+0000","last_change":"2023-10-16T14:12:39.937752+0000","last_active":"2023-10-16T14:12:40.917340+0000","last_peered":"2023-10-16T14:12:40.917340+0000","last_clean":"2023-10-16T14:12:40.917340+0000","last_became_active":"2023-10-16T14:12:39.937419+0000","last_became_peered":"2023-10-16T14:12:39.937419+0000","last_unstale":"2023-10-16T14:12:40.917340+0000","last_undegraded":"2023-10-16T14:12:40.917340+0000","last_fullsized":"2023-10-16T14:12:40.917340+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:25.330533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406367+0000","last_change":"2023-10-16T14:12:40.951163+0000","last_active":"2023-10-16T14:12:42.406367+0000","last_peered":"2023-10-16T14:12:42.406367+0000","last_clean":"2023-10-16T14:12:42.406367+0000","last_became_active":"2023-10-16T14:12:40.950205+0000","last_became_peered":"2023-10-16T14:12:40.950205+0000","last_unstale":"2023-10-16T14:12:42.406367+0000","last_undegraded":"2023-10-16T14:12:42.406367+0000","last_fullsized":"2023-10-16T14:12:42.406367+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:17.074207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946769+0000","last_change":"2023-10-16T14:12:39.925773+0000","last_active":"2023-10-16T14:12:41.946769+0000","last_peered":"2023-10-16T14:12:41.946769+0000","last_clean":"2023-10-16T14:12:41.946769+0000","last_became_active":"2023-10-16T14:12:39.925301+0000","last_became_peered":"2023-10-16T14:12:39.925301+0000","last_unstale":"2023-10-16T14:12:41.946769+0000","last_undegraded":"2023-10-16T14:12:41.946769+0000","last_fullsized":"2023-10-16T14:12:41.946769+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:04:28.985481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.945878+0000","last_change":"2023-10-16T14:12:40.940856+0000","last_active":"2023-10-16T14:12:40.945878+0000","last_peered":"2023-10-16T14:12:40.945878+0000","last_clean":"2023-10-16T14:12:40.945878+0000","last_became_active":"2023-10-16T14:12:40.940254+0000","last_became_peered":"2023-10-16T14:12:40.940254+0000","last_unstale":"2023-10-16T14:12:40.945878+0000","last_undegraded":"2023-10-16T14:12:40.945878+0000","last_fullsized":"2023-10-16T14:12:40.945878+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:10.059974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"272'8","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947349+0000","last_change":"2023-10-16T14:12:39.939556+0000","last_active":"2023-10-16T14:12:40.947349+0000","last_peered":"2023-10-16T14:12:40.947349+0000","last_clean":"2023-10-16T14:12:40.947349+0000","last_became_active":"2023-10-16T14:12:39.939244+0000","last_became_peered":"2023-10-16T14:12:39.939244+0000","last_unstale":"2023-10-16T14:12:40.947349+0000","last_undegraded":"2023-10-16T14:12:40.947349+0000","last_fullsized":"2023-10-16T14:12:40.947349+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:54.439596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.932029+0000","last_change":"2023-10-16T14:12:40.928984+0000","last_active":"2023-10-16T14:12:40.932029+0000","last_peered":"2023-10-16T14:12:40.932029+0000","last_clean":"2023-10-16T14:12:40.932029+0000","last_became_active":"2023-10-16T14:12:40.926554+0000","last_became_peered":"2023-10-16T14:12:40.926554+0000","last_unstale":"2023-10-16T14:12:40.932029+0000","last_undegraded":"2023-10-16T14:12:40.932029+0000","last_fullsized":"2023-10-16T14:12:40.932029+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:24.244423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.948110+0000","last_change":"2023-10-16T14:12:39.924031+0000","last_active":"2023-10-16T14:12:41.948110+0000","last_peered":"2023-10-16T14:12:41.948110+0000","last_clean":"2023-10-16T14:12:41.948110+0000","last_became_active":"2023-10-16T14:12:39.923745+0000","last_became_peered":"2023-10-16T14:12:39.923745+0000","last_unstale":"2023-10-16T14:12:41.948110+0000","last_undegraded":"2023-10-16T14:12:41.948110+0000","last_fullsized":"2023-10-16T14:12:41.948110+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:02.671005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.940021+0000","last_change":"2023-10-16T14:12:40.936029+0000","last_active":"2023-10-16T14:12:40.940021+0000","last_peered":"2023-10-16T14:12:40.940021+0000","last_clean":"2023-10-16T14:12:40.940021+0000","last_became_active":"2023-10-16T14:12:40.935732+0000","last_became_peered":"2023-10-16T14:12:40.935732+0000","last_unstale":"2023-10-16T14:12:40.940021+0000","last_undegraded":"2023-10-16T14:12:40.940021+0000","last_fullsized":"2023-10-16T14:12:40.940021+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:54.322819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.952260+0000","last_change":"2023-10-16T14:12:39.937298+0000","last_active":"2023-10-16T14:12:40.952260+0000","last_peered":"2023-10-16T14:12:40.952260+0000","last_clean":"2023-10-16T14:12:40.952260+0000","last_became_active":"2023-10-16T14:12:39.937019+0000","last_became_peered":"2023-10-16T14:12:39.937019+0000","last_unstale":"2023-10-16T14:12:40.952260+0000","last_undegraded":"2023-10-16T14:12:40.952260+0000","last_fullsized":"2023-10-16T14:12:40.952260+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:52.174465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941292+0000","last_change":"2023-10-16T14:12:40.950379+0000","last_active":"2023-10-16T14:12:41.941292+0000","last_peered":"2023-10-16T14:12:41.941292+0000","last_clean":"2023-10-16T14:12:41.941292+0000","last_became_active":"2023-10-16T14:12:40.950035+0000","last_became_peered":"2023-10-16T14:12:40.950035+0000","last_unstale":"2023-10-16T14:12:41.941292+0000","last_undegraded":"2023-10-16T14:12:41.941292+0000","last_fullsized":"2023-10-16T14:12:41.941292+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:46.054897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"272'1","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947659+0000","last_change":"2023-10-16T14:12:39.928129+0000","last_active":"2023-10-16T14:12:41.947659+0000","last_peered":"2023-10-16T14:12:41.947659+0000","last_clean":"2023-10-16T14:12:41.947659+0000","last_became_active":"2023-10-16T14:12:39.927848+0000","last_became_peered":"2023-10-16T14:12:39.927848+0000","last_unstale":"2023-10-16T14:12:41.947659+0000","last_undegraded":"2023-10-16T14:12:41.947659+0000","last_fullsized":"2023-10-16T14:12:41.947659+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:00.144505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.943162+0000","last_change":"2023-10-16T14:12:40.939787+0000","last_active":"2023-10-16T14:12:40.943162+0000","last_peered":"2023-10-16T14:12:40.943162+0000","last_clean":"2023-10-16T14:12:40.943162+0000","last_became_active":"2023-10-16T14:12:40.939509+0000","last_became_peered":"2023-10-16T14:12:40.939509+0000","last_unstale":"2023-10-16T14:12:40.943162+0000","last_undegraded":"2023-10-16T14:12:40.943162+0000","last_fullsized":"2023-10-16T14:12:40.943162+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:46.210818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.947857+0000","last_change":"2023-10-16T14:12:39.933837+0000","last_active":"2023-10-16T14:12:41.947857+0000","last_peered":"2023-10-16T14:12:41.947857+0000","last_clean":"2023-10-16T14:12:41.947857+0000","last_became_active":"2023-10-16T14:12:39.933569+0000","last_became_peered":"2023-10-16T14:12:39.933569+0000","last_unstale":"2023-10-16T14:12:41.947857+0000","last_undegraded":"2023-10-16T14:12:41.947857+0000","last_fullsized":"2023-10-16T14:12:41.947857+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:58.522569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.936950+0000","last_change":"2023-10-16T14:12:40.941796+0000","last_active":"2023-10-16T14:12:41.936950+0000","last_peered":"2023-10-16T14:12:41.936950+0000","last_clean":"2023-10-16T14:12:41.936950+0000","last_became_active":"2023-10-16T14:12:40.941411+0000","last_became_peered":"2023-10-16T14:12:40.941411+0000","last_unstale":"2023-10-16T14:12:41.936950+0000","last_undegraded":"2023-10-16T14:12:41.936950+0000","last_fullsized":"2023-10-16T14:12:41.936950+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:24.580487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"272'2","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947060+0000","last_change":"2023-10-16T14:12:39.938790+0000","last_active":"2023-10-16T14:12:40.947060+0000","last_peered":"2023-10-16T14:12:40.947060+0000","last_clean":"2023-10-16T14:12:40.947060+0000","last_became_active":"2023-10-16T14:12:39.938488+0000","last_became_peered":"2023-10-16T14:12:39.938488+0000","last_unstale":"2023-10-16T14:12:40.947060+0000","last_undegraded":"2023-10-16T14:12:40.947060+0000","last_fullsized":"2023-10-16T14:12:40.947060+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:42.258052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940795+0000","last_change":"2023-10-16T14:12:40.932475+0000","last_active":"2023-10-16T14:12:41.940795+0000","last_peered":"2023-10-16T14:12:41.940795+0000","last_clean":"2023-10-16T14:12:41.940795+0000","last_became_active":"2023-10-16T14:12:40.931912+0000","last_became_peered":"2023-10-16T14:12:40.931912+0000","last_unstale":"2023-10-16T14:12:41.940795+0000","last_undegraded":"2023-10-16T14:12:41.940795+0000","last_fullsized":"2023-10-16T14:12:41.940795+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:48.198209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.925482+0000","last_change":"2023-10-16T14:12:39.939921+0000","last_active":"2023-10-16T14:12:40.925482+0000","last_peered":"2023-10-16T14:12:40.925482+0000","last_clean":"2023-10-16T14:12:40.925482+0000","last_became_active":"2023-10-16T14:12:39.939634+0000","last_became_peered":"2023-10-16T14:12:39.939634+0000","last_unstale":"2023-10-16T14:12:40.925482+0000","last_undegraded":"2023-10-16T14:12:40.925482+0000","last_fullsized":"2023-10-16T14:12:40.925482+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:47.208133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940178+0000","last_change":"2023-10-16T14:12:40.936615+0000","last_active":"2023-10-16T14:12:41.940178+0000","last_peered":"2023-10-16T14:12:41.940178+0000","last_clean":"2023-10-16T14:12:41.940178+0000","last_became_active":"2023-10-16T14:12:40.935677+0000","last_became_peered":"2023-10-16T14:12:40.935677+0000","last_unstale":"2023-10-16T14:12:41.940178+0000","last_undegraded":"2023-10-16T14:12:41.940178+0000","last_fullsized":"2023-10-16T14:12:41.940178+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:32.275285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946984+0000","last_change":"2023-10-16T14:12:39.930125+0000","last_active":"2023-10-16T14:12:41.946984+0000","last_peered":"2023-10-16T14:12:41.946984+0000","last_clean":"2023-10-16T14:12:41.946984+0000","last_became_active":"2023-10-16T14:12:39.929162+0000","last_became_peered":"2023-10-16T14:12:39.929162+0000","last_unstale":"2023-10-16T14:12:41.946984+0000","last_undegraded":"2023-10-16T14:12:41.946984+0000","last_fullsized":"2023-10-16T14:12:41.946984+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:00.530710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.937696+0000","last_change":"2023-10-16T14:12:40.932252+0000","last_active":"2023-10-16T14:12:40.937696+0000","last_peered":"2023-10-16T14:12:40.937696+0000","last_clean":"2023-10-16T14:12:40.937696+0000","last_became_active":"2023-10-16T14:12:40.931966+0000","last_became_peered":"2023-10-16T14:12:40.931966+0000","last_unstale":"2023-10-16T14:12:40.937696+0000","last_undegraded":"2023-10-16T14:12:40.937696+0000","last_fullsized":"2023-10-16T14:12:40.937696+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:33.930874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.912904+0000","last_change":"2023-10-16T14:12:39.933128+0000","last_active":"2023-10-16T14:12:40.912904+0000","last_peered":"2023-10-16T14:12:40.912904+0000","last_clean":"2023-10-16T14:12:40.912904+0000","last_became_active":"2023-10-16T14:12:39.932483+0000","last_became_peered":"2023-10-16T14:12:39.932483+0000","last_unstale":"2023-10-16T14:12:40.912904+0000","last_undegraded":"2023-10-16T14:12:40.912904+0000","last_fullsized":"2023-10-16T14:12:40.912904+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:34.551675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.945748+0000","last_change":"2023-10-16T14:12:40.939835+0000","last_active":"2023-10-16T14:12:40.945748+0000","last_peered":"2023-10-16T14:12:40.945748+0000","last_clean":"2023-10-16T14:12:40.945748+0000","last_became_active":"2023-10-16T14:12:40.939556+0000","last_became_peered":"2023-10-16T14:12:40.939556+0000","last_unstale":"2023-10-16T14:12:40.945748+0000","last_undegraded":"2023-10-16T14:12:40.945748+0000","last_fullsized":"2023-10-16T14:12:40.945748+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:42.809225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"272'2","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407274+0000","last_change":"2023-10-16T14:12:39.938879+0000","last_active":"2023-10-16T14:12:42.407274+0000","last_peered":"2023-10-16T14:12:42.407274+0000","last_clean":"2023-10-16T14:12:42.407274+0000","last_became_active":"2023-10-16T14:12:39.938586+0000","last_became_peered":"2023-10-16T14:12:39.938586+0000","last_unstale":"2023-10-16T14:12:42.407274+0000","last_undegraded":"2023-10-16T14:12:42.407274+0000","last_fullsized":"2023-10-16T14:12:42.407274+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:37:54.414122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941680+0000","last_change":"2023-10-16T14:12:40.936506+0000","last_active":"2023-10-16T14:12:41.941680+0000","last_peered":"2023-10-16T14:12:41.941680+0000","last_clean":"2023-10-16T14:12:41.941680+0000","last_became_active":"2023-10-16T14:12:40.935931+0000","last_became_peered":"2023-10-16T14:12:40.935931+0000","last_unstale":"2023-10-16T14:12:41.941680+0000","last_undegraded":"2023-10-16T14:12:41.941680+0000","last_fullsized":"2023-10-16T14:12:41.941680+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:26.605559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.941572+0000","last_change":"2023-10-16T14:12:39.937705+0000","last_active":"2023-10-16T14:12:41.941572+0000","last_peered":"2023-10-16T14:12:41.941572+0000","last_clean":"2023-10-16T14:12:41.941572+0000","last_became_active":"2023-10-16T14:12:39.937419+0000","last_became_peered":"2023-10-16T14:12:39.937419+0000","last_unstale":"2023-10-16T14:12:41.941572+0000","last_undegraded":"2023-10-16T14:12:41.941572+0000","last_fullsized":"2023-10-16T14:12:41.941572+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:37.216338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947660+0000","last_change":"2023-10-16T14:12:40.944810+0000","last_active":"2023-10-16T14:12:40.947660+0000","last_peered":"2023-10-16T14:12:40.947660+0000","last_clean":"2023-10-16T14:12:40.947660+0000","last_became_active":"2023-10-16T14:12:40.944510+0000","last_became_peered":"2023-10-16T14:12:40.944510+0000","last_unstale":"2023-10-16T14:12:40.947660+0000","last_undegraded":"2023-10-16T14:12:40.947660+0000","last_fullsized":"2023-10-16T14:12:40.947660+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:43.087530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.406760+0000","last_change":"2023-10-16T14:12:39.941935+0000","last_active":"2023-10-16T14:12:42.406760+0000","last_peered":"2023-10-16T14:12:42.406760+0000","last_clean":"2023-10-16T14:12:42.406760+0000","last_became_active":"2023-10-16T14:12:39.941555+0000","last_became_peered":"2023-10-16T14:12:39.941555+0000","last_unstale":"2023-10-16T14:12:42.406760+0000","last_undegraded":"2023-10-16T14:12:42.406760+0000","last_fullsized":"2023-10-16T14:12:42.406760+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:22.527108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.947249+0000","last_change":"2023-10-16T14:12:40.944756+0000","last_active":"2023-10-16T14:12:40.947249+0000","last_peered":"2023-10-16T14:12:40.947249+0000","last_clean":"2023-10-16T14:12:40.947249+0000","last_became_active":"2023-10-16T14:12:40.944457+0000","last_became_peered":"2023-10-16T14:12:40.944457+0000","last_unstale":"2023-10-16T14:12:40.947249+0000","last_undegraded":"2023-10-16T14:12:40.947249+0000","last_fullsized":"2023-10-16T14:12:40.947249+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:00:26.779056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407112+0000","last_change":"2023-10-16T14:12:39.940002+0000","last_active":"2023-10-16T14:12:42.407112+0000","last_peered":"2023-10-16T14:12:42.407112+0000","last_clean":"2023-10-16T14:12:42.407112+0000","last_became_active":"2023-10-16T14:12:39.939628+0000","last_became_peered":"2023-10-16T14:12:39.939628+0000","last_unstale":"2023-10-16T14:12:42.407112+0000","last_undegraded":"2023-10-16T14:12:42.407112+0000","last_fullsized":"2023-10-16T14:12:42.407112+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:11.400216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.938060+0000","last_change":"2023-10-16T14:12:40.950306+0000","last_active":"2023-10-16T14:12:41.938060+0000","last_peered":"2023-10-16T14:12:41.938060+0000","last_clean":"2023-10-16T14:12:41.938060+0000","last_became_active":"2023-10-16T14:12:40.949894+0000","last_became_peered":"2023-10-16T14:12:40.949894+0000","last_unstale":"2023-10-16T14:12:41.938060+0000","last_undegraded":"2023-10-16T14:12:41.938060+0000","last_fullsized":"2023-10-16T14:12:41.938060+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:18:11.505844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.935271+0000","last_change":"2023-10-16T14:12:39.943126+0000","last_active":"2023-10-16T14:12:40.935271+0000","last_peered":"2023-10-16T14:12:40.935271+0000","last_clean":"2023-10-16T14:12:40.935271+0000","last_became_active":"2023-10-16T14:12:39.941342+0000","last_became_peered":"2023-10-16T14:12:39.941342+0000","last_unstale":"2023-10-16T14:12:40.935271+0000","last_undegraded":"2023-10-16T14:12:40.935271+0000","last_fullsized":"2023-10-16T14:12:40.935271+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:51.253014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940878+0000","last_change":"2023-10-16T14:12:40.927367+0000","last_active":"2023-10-16T14:12:41.940878+0000","last_peered":"2023-10-16T14:12:41.940878+0000","last_clean":"2023-10-16T14:12:41.940878+0000","last_became_active":"2023-10-16T14:12:40.927058+0000","last_became_peered":"2023-10-16T14:12:40.927058+0000","last_unstale":"2023-10-16T14:12:41.940878+0000","last_undegraded":"2023-10-16T14:12:41.940878+0000","last_fullsized":"2023-10-16T14:12:41.940878+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:05.040275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917961+0000","last_change":"2023-10-16T14:12:39.919948+0000","last_active":"2023-10-16T14:12:40.917961+0000","last_peered":"2023-10-16T14:12:40.917961+0000","last_clean":"2023-10-16T14:12:40.917961+0000","last_became_active":"2023-10-16T14:12:39.919568+0000","last_became_peered":"2023-10-16T14:12:39.919568+0000","last_unstale":"2023-10-16T14:12:40.917961+0000","last_undegraded":"2023-10-16T14:12:40.917961+0000","last_fullsized":"2023-10-16T14:12:40.917961+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:46.651309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.940569+0000","last_change":"2023-10-16T14:12:40.929601+0000","last_active":"2023-10-16T14:12:41.940569+0000","last_peered":"2023-10-16T14:12:41.940569+0000","last_clean":"2023-10-16T14:12:41.940569+0000","last_became_active":"2023-10-16T14:12:40.929280+0000","last_became_peered":"2023-10-16T14:12:40.929280+0000","last_unstale":"2023-10-16T14:12:41.940569+0000","last_undegraded":"2023-10-16T14:12:41.940569+0000","last_fullsized":"2023-10-16T14:12:41.940569+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:57.978074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.917754+0000","last_change":"2023-10-16T14:12:39.919239+0000","last_active":"2023-10-16T14:12:40.917754+0000","last_peered":"2023-10-16T14:12:40.917754+0000","last_clean":"2023-10-16T14:12:40.917754+0000","last_became_active":"2023-10-16T14:12:39.918787+0000","last_became_peered":"2023-10-16T14:12:39.918787+0000","last_unstale":"2023-10-16T14:12:40.917754+0000","last_undegraded":"2023-10-16T14:12:40.917754+0000","last_fullsized":"2023-10-16T14:12:40.917754+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:04.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:41.946711+0000","last_change":"2023-10-16T14:12:40.945087+0000","last_active":"2023-10-16T14:12:41.946711+0000","last_peered":"2023-10-16T14:12:41.946711+0000","last_clean":"2023-10-16T14:12:41.946711+0000","last_became_active":"2023-10-16T14:12:40.944784+0000","last_became_peered":"2023-10-16T14:12:40.944784+0000","last_unstale":"2023-10-16T14:12:41.946711+0000","last_undegraded":"2023-10-16T14:12:41.946711+0000","last_fullsized":"2023-10-16T14:12:41.946711+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:18.518107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.927058+0000","last_change":"2023-10-16T14:12:39.920707+0000","last_active":"2023-10-16T14:12:40.927058+0000","last_peered":"2023-10-16T14:12:40.927058+0000","last_clean":"2023-10-16T14:12:40.927058+0000","last_became_active":"2023-10-16T14:12:39.920138+0000","last_became_peered":"2023-10-16T14:12:39.920138+0000","last_unstale":"2023-10-16T14:12:40.927058+0000","last_undegraded":"2023-10-16T14:12:40.927058+0000","last_fullsized":"2023-10-16T14:12:40.927058+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:38.871907+0000","last_clean_scrub_stamp":"2023-10-16T14:12:38.871907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:47.134334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2023-10-16T14:12:42.407298+0000","last_change":"2023-10-16T14:12:40.950705+0000","last_active":"2023-10-16T14:12:42.407298+0000","last_peered":"2023-10-16T14:12:42.407298+0000","last_clean":"2023-10-16T14:12:42.407298+0000","last_became_active":"2023-10-16T14:12:40.950266+0000","last_became_peered":"2023-10-16T14:12:40.950266+0000","last_unstale":"2023-10-16T14:12:42.407298+0000","last_undegraded":"2023-10-16T14:12:42.407298+0000","last_fullsized":"2023-10-16T14:12:42.407298+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:12:39.878109+0000","last_clean_scrub_stamp":"2023-10-16T14:12:39.878109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:41.864797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"269'2799","reported_seq":3831,"reported_epoch":272,"state":"active+clean","last_fresh":"2023-10-16T14:12:40.933327+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:12:40.933327+0000","last_peered":"2023-10-16T14:12:40.933327+0000","last_clean":"2023-10-16T14:12:40.933327+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:12:40.933327+0000","last_undegraded":"2023-10-16T14:12:40.933327+0000","last_fullsized":"2023-10-16T14:12:40.933327+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2507,"num_read_kb":5248,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2507,"num_read_kb":5248,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2799,"ondisk_log_size":2799,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543323,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49008,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322832,"statfs":{"total":96636764160,"available":96586579968,"internally_reserved":0,"allocated":4034560,"data_stored":2282471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.3140000000000001,"15min":1.3480000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.641,"5min":1.8500000000000001,"15min":1.885},"last":1.51},{"interface":"front","average":{"1min":1.714,"5min":1.405,"15min":1.413},"min":{"1min":0.73699999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.603}]},{"osd":1,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.232,"15min":1.246},"min":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.228,"15min":1.248},"min":{"1min":0.76300000000000001,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.2490000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.252,"15min":1.268},"min":{"1min":0.85299999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.458},{"interface":"front","average":{"1min":1.472,"5min":1.349,"15min":1.361},"min":{"1min":0.95199999999999996,"5min":0.745,"15min":0.745},"max":{"1min":3.4289999999999998,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.4219999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.357,"15min":1.405},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.097},"last":1.302},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4159999999999999,"15min":1.431},"min":{"1min":1.107,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.823}]},{"osd":4,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2589999999999999,"15min":1.242},"min":{"1min":0.94699999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.895,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.345},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4730000000000001,"15min":1.4159999999999999},"min":{"1min":0.98799999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.7470000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.4450000000000001,"15min":1.395},"min":{"1min":1.026,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.546,"15min":1.538},"min":{"1min":1.0800000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.667}]},{"osd":6,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.6599999999999999,"15min":1.6379999999999999},"min":{"1min":1.21,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":3.29,"5min":3.29,"15min":3.29},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5600000000000001,"15min":1.486},"min":{"1min":1.0029999999999999,"5min":0.75,"15min":0.75},"max":{"1min":3.524,"5min":3.524,"15min":3.524},"last":1.7230000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543324,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48420,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323420,"statfs":{"total":96636764160,"available":96587182080,"internally_reserved":0,"allocated":3432448,"data_stored":1685850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.244,"15min":1.3109999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.887,"15min":1.887},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.3149999999999999,"15min":1.345},"min":{"1min":0.84999999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.595,"5min":1.847,"15min":1.9390000000000001},"last":1.2589999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.175,"15min":1.212},"min":{"1min":0.80000000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6220000000000001,"5min":1.8360000000000001,"15min":2.008},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.202},"min":{"1min":0.67900000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.286,"5min":1.706,"15min":1.77},"last":1.2969999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2330000000000001,"15min":1.2470000000000001},"min":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.5960000000000001,"5min":1.823,"15min":1.823},"last":1.002},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2749999999999999,"15min":1.3120000000000001},"min":{"1min":0.98799999999999999,"5min":0.745,"15min":0.745},"max":{"1min":1.619,"5min":1.962,"15min":1.962},"last":1.046}]},{"osd":3,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2829999999999999,"15min":1.333},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4830000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.526},{"interface":"front","average":{"1min":1.294,"5min":1.3919999999999999,"15min":1.405},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.5629999999999999,"5min":1.911,"15min":1.911},"last":1.105}]},{"osd":4,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.389,"15min":1.403},"min":{"1min":0.77300000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.6479999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.456},{"interface":"front","average":{"1min":1.476,"5min":1.546,"15min":1.548},"min":{"1min":1.0680000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.1179999999999999},"last":1.498}]},{"osd":5,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.88300000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0840000000000001,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4710000000000001,"15min":1.486},"min":{"1min":0.91900000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.786,"5min":2.0990000000000002,"15min":2.1970000000000001},"last":1.6910000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.506,"15min":1.5009999999999999},"min":{"1min":0.96799999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0270000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.5289999999999999,"15min":1.536},"min":{"1min":0.998,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.9990000000000001,"5min":2.157,"15min":2.6120000000000001},"last":1.3839999999999999}]}]},{"osd":5,"up_from":14,"seq":60129543325,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48452,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323388,"statfs":{"total":96636764160,"available":96587149312,"internally_reserved":0,"allocated":3461120,"data_stored":1694367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.111,"15min":1.1879999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.546,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.143},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1919999999999999,"15min":1.22},"min":{"1min":0.79400000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.238}]},{"osd":1,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.286,"15min":1.306},"min":{"1min":1.034,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.631,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.095},{"interface":"front","average":{"1min":1.2,"5min":1.1839999999999999,"15min":1.248},"min":{"1min":0.80000000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.931,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.3560000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2909999999999999,"15min":1.333},"min":{"1min":0.82799999999999996,"5min":0.63,"15min":0.63},"max":{"1min":2.024,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.238,"15min":1.3009999999999999},"min":{"1min":0.82599999999999996,"5min":0.746,"15min":0.746},"max":{"1min":1.8600000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":0.93799999999999994}]},{"osd":3,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3899999999999999,"15min":1.403},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.681,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":1.0049999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.827,"5min":2.75,"15min":2.75},"last":1.0189999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3160000000000001,"15min":1.298},"min":{"1min":0.86899999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6839999999999999,"5min":2.774,"15min":2.774},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.4430000000000001},"min":{"1min":1.1879999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.865,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.532}]},{"osd":6,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.464,"15min":1.4390000000000001},"min":{"1min":1.0289999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9399999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.454},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5009999999999999,"15min":1.4870000000000001},"min":{"1min":1.0900000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9790000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.0580000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.5920000000000001,"15min":1.556},"min":{"1min":1.1040000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.153,"5min":2.1680000000000001,"15min":2.4609999999999999},"last":1.403},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6220000000000001,"15min":1.6060000000000001},"min":{"1min":1.272,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.044,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.1859999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543325,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66420,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305420,"statfs":{"total":96636764160,"available":96568750080,"internally_reserved":0,"allocated":3448832,"data_stored":1692195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.151,"15min":1.1899999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.4510000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.575},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1759999999999999,"15min":1.1950000000000001},"min":{"1min":0.76100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6100000000000001,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.083}]},{"osd":1,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.157,"15min":1.2370000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.673,"5min":1.71,"15min":1.78},"last":1.268},{"interface":"front","average":{"1min":1.236,"5min":1.232,"15min":1.28},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.792,"5min":1.792,"15min":2.0499999999999998},"last":1.3149999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.86299999999999999,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.5169999999999999,"5min":1.714,"15min":1.865},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3520000000000001,"15min":1.379},"min":{"1min":1.006,"5min":0.87,"15min":0.87},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.6319999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.5980000000000001,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.4279999999999999},"min":{"1min":0.93000000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.915,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.1459999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.357,"15min":1.3400000000000001},"min":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.0670000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.427},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3899999999999999,"15min":1.349},"min":{"1min":0.89900000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":0.97499999999999998}]},{"osd":6,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.417,"15min":1.3939999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.956},"last":1.466},{"interface":"front","average":{"1min":1.609,"5min":1.538,"15min":1.522},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.3279999999999998,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.512}]},{"osd":7,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.516},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.22},{"interface":"front","average":{"1min":1.665,"5min":1.645,"15min":1.639},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.1200000000000001,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.5509999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543325,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56748,"kb_used_data":3360,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315092,"statfs":{"total":96636764160,"available":96578654208,"internally_reserved":0,"allocated":3440640,"data_stored":1687372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.90900000000000003,"15min":0.94799999999999995},"min":{"1min":0.48599999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.107,"5min":1.1479999999999999,"15min":1.2090000000000001},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.0169999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.03,"15min":1.075},"min":{"1min":0.55900000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.855},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.133,"5min":1.0980000000000001,"15min":1.123},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.169}]},{"osd":2,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0960000000000001,"15min":1.1890000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.823,"5min":1.8700000000000001,"15min":2.0539999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3400000000000001,"15min":1.431},"min":{"1min":0.622,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.909,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.91800000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.482,"15min":1.51},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2160000000000002,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.506},{"interface":"front","average":{"1min":1.379,"5min":1.4330000000000001,"15min":1.456},"min":{"1min":0.84099999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.823,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.6739999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.427,"15min":1.4850000000000001},"min":{"1min":0.89900000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8380000000000001,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.226},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.52,"15min":1.5529999999999999},"min":{"1min":0.871,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9910000000000001,"5min":2.0339999999999998,"15min":2.153},"last":1.462}]},{"osd":6,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.5,"15min":1.522},"min":{"1min":0.72899999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.335,"5min":2.335,"15min":2.4329999999999998},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4750000000000001,"15min":1.496},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.194,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.6000000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:11:49 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5760000000000001,"15min":1.599},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1859999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.268},{"interface":"front","average":{"1min":1.698,"5min":1.6599999999999999,"15min":1.677},"min":{"1min":0.98199999999999998,"5min":0.90600000000000003,"15min":0.88300000000000001},"max":{"1min":2.5,"5min":2.569,"15min":2.6070000000000002},"last":1.5489999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56752,"kb_used_data":3364,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315088,"statfs":{"total":96636764160,"available":96578650112,"internally_reserved":0,"allocated":3444736,"data_stored":1692193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.44600000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.5680000000000001,"5min":1.597,"15min":2.0089999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.036,"5min":1.0409999999999999,"15min":1.131},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.3149999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0920000000000001,"15min":1.1519999999999999},"min":{"1min":0.51900000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.6970000000000001,"5min":1.847,"15min":1.847},"last":1.165},{"interface":"front","average":{"1min":1.101,"5min":1.153,"15min":1.2350000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.784,"5min":1.784,"15min":1.9530000000000001},"last":1.0600000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0549999999999999,"15min":1.111},"min":{"1min":0.52900000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3740000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2490000000000001,"15min":1.353},"min":{"1min":0.54800000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.3149999999999999,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.4330000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.454,"15min":1.4910000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.5529999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.524},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.444,"15min":1.4890000000000001},"min":{"1min":0.81100000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0409999999999999,"5min":2.1789999999999998,"15min":2.3140000000000001},"last":1.4790000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.4910000000000001,"15min":1.548},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.95,"5min":2.1749999999999998,"15min":2.488},"last":1.544},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4530000000000001,"15min":1.4710000000000001},"min":{"1min":0.72899999999999998,"5min":0.627,"15min":0.627},"max":{"1min":2.073,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.8009999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.506,"15min":1.5269999999999999},"min":{"1min":0.95099999999999996,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.4079999999999999},"last":1.643},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5489999999999999,"15min":1.613},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.5569999999999999},"last":1.6799999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:12:19 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.617,"15min":1.661},"min":{"1min":0.98899999999999999,"5min":0.754,"15min":0.754},"max":{"1min":2.1960000000000002,"5min":2.4940000000000002,"15min":2.7890000000000001},"last":1.835},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.603,"15min":1.6419999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.161,"5min":2.2799999999999998,"15min":2.3319999999999999},"last":1.8859999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52584,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319256,"statfs":{"total":96636764160,"available":96582918144,"internally_reserved":0,"allocated":3432448,"data_stored":1685794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.88,"15min":0.92600000000000005},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0289999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.876},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0640000000000001,"15min":1.125},"min":{"1min":0.58299999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.647,"5min":1.907,"15min":1.907},"last":1.101}]},{"osd":2,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.97599999999999998,"15min":1.0289999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.601,"5min":1.853,"15min":1.853},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.1519999999999999,"15min":1.2370000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.5289999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.2130000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.2729999999999999,"15min":1.3740000000000001},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6559999999999999,"5min":2.1659999999999999,"15min":2.4369999999999998},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.167,"15min":1.274},"min":{"1min":0.63900000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7529999999999999,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.2849999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.228,"15min":1.333},"min":{"1min":0.77500000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.756,"5min":1.756,"15min":2.1259999999999999},"last":1.484},{"interface":"front","average":{"1min":1.335,"5min":1.4390000000000001,"15min":1.516},"min":{"1min":0.90200000000000002,"5min":0.748,"15min":0.748},"max":{"1min":1.6459999999999999,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.5189999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4410000000000001,"15min":1.4810000000000001},"min":{"1min":0.85699999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.1070000000000002,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.3740000000000001,"15min":1.4730000000000001},"min":{"1min":0.82599999999999996,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.833,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.696}]},{"osd":6,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.5309999999999999,"15min":1.619},"min":{"1min":0.80400000000000005,"5min":0.748,"15min":0.748},"max":{"1min":1.919,"5min":2.4540000000000002,"15min":2.7200000000000002},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.593},"min":{"1min":0.84599999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2530000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.431}]},{"osd":7,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.6120000000000001,"15min":1.6890000000000001},"min":{"1min":1.109,"5min":0.628,"15min":0.628},"max":{"1min":2.339,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.484,"15min":1.6100000000000001},"min":{"1min":0.95199999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9430000000000001,"5min":2.1720000000000002,"15min":2.2930000000000001},"last":1.7789999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543324,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61488,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310352,"statfs":{"total":96636764160,"available":96573800448,"internally_reserved":0,"allocated":4034560,"data_stored":2276766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.90900000000000003,"15min":0.93300000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3440000000000001,"5min":1.704,"15min":1.891},"last":1.284},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0049999999999999,"15min":1.1279999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.6539999999999999,"15min":1.9299999999999999},"last":0.878}]},{"osd":2,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.0669999999999999,"15min":1.157},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4359999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.123,"15min":1.202},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.657,"5min":1.893,"15min":1.893},"last":1.109}]},{"osd":3,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0640000000000001,"15min":1.141},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7589999999999999,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2270000000000001,"15min":1.351},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.603,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.4330000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.26,"15min":1.3500000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.923,"5min":1.923,"15min":2.0070000000000001},"last":1.206},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.413,"15min":1.5009999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.98,"5min":2.1070000000000002,"15min":2.601},"last":1.6000000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.4430000000000001,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":2.089,"15min":2.3279999999999998},"last":1.54},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.385,"15min":1.468},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.901,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.3420000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.5229999999999999,"15min":1.5960000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.669},"last":1.9630000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5109999999999999,"15min":1.579},"min":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0129999999999999,"5min":2.327,"15min":2.3769999999999998},"last":1.454}]},{"osd":7,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.6080000000000001,"15min":1.6879999999999999},"min":{"1min":0.76700000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0670000000000002,"5min":2.2970000000000002,"15min":2.5270000000000001},"last":1.756},{"interface":"front","average":{"1min":1.482,"5min":1.5509999999999999,"15min":1.6259999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0270000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.7150000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:12:46.010 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:12:46.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:12:46.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.289+0000 7fa7bd3d0640 1 Processor -- start 2023-10-16T14:12:46.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.290+0000 7fa7bd3d0640 1 -- start start 2023-10-16T14:12:46.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.291+0000 7fa7bd3d0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7b8159ac0 0x7fa7b815e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.291+0000 7fa7bd3d0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80cc270 0x7fa7b80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.291+0000 7fa7bd3d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7b80704e0 0x7fa7b8070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.291+0000 7fa7bd3d0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b8070eb0 con 0x7fa7b80704e0 2023-10-16T14:12:46.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.291+0000 7fa7bd3d0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b815e5c0 con 0x7fa7b8159ac0 2023-10-16T14:12:46.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.291+0000 7fa7bd3d0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b806b810 con 0x7fa7b80cc270 2023-10-16T14:12:46.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7b80704e0 0x7fa7b8070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7b80704e0 0x7fa7b8070970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48448/0 (socket says 172.21.15.60:48448) 2023-10-16T14:12:46.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 -- 172.21.15.60:0/4265302408 learned_addr learned my addr 172.21.15.60:0/4265302408 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:46.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7b7fff640 1 --2- 172.21.15.60:0/4265302408 >> v2:172.21.15.144:3300/0 conn(0x7fa7b8159ac0 0x7fa7b815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7affff640 1 --2- 172.21.15.60:0/4265302408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80cc270 0x7fa7b80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 -- 172.21.15.60:0/4265302408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80cc270 msgr2=0x7fa7b80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:46.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 --2- 172.21.15.60:0/4265302408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80cc270 0x7fa7b80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 -- 172.21.15.60:0/4265302408 >> v2:172.21.15.144:3300/0 conn(0x7fa7b8159ac0 msgr2=0x7fa7b815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:46.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 --2- 172.21.15.60:0/4265302408 >> v2:172.21.15.144:3300/0 conn(0x7fa7b8159ac0 0x7fa7b815e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.292+0000 7fa7bcbcf640 1 -- 172.21.15.60:0/4265302408 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80cc810 con 0x7fa7b80704e0 2023-10-16T14:12:46.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.294+0000 7fa7bcbcf640 1 --2- 172.21.15.60:0/4265302408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80704e0 0x7fa7b8070970 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8071b00 tx=0x7fa7b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50969415d94f699f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:46.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.294+0000 7fa7b77fe640 1 -- 172.21.15.60:0/4265302408 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b002ca70 con 0x7fa7b80704e0 2023-10-16T14:12:46.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.294+0000 7fa7b77fe640 1 -- 172.21.15.60:0/4265302408 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b002cbd0 con 0x7fa7b80704e0 2023-10-16T14:12:46.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.294+0000 7fa7b77fe640 1 -- 172.21.15.60:0/4265302408 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b00343f0 con 0x7fa7b80704e0 2023-10-16T14:12:46.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4265302408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80704e0 msgr2=0x7fa7b8070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4265302408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80704e0 0x7fa7b8070970 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8071b00 tx=0x7fa7b002bed0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4265302408 shutdown_connections 2023-10-16T14:12:46.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4265302408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80704e0 0x7fa7b8070970 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4265302408 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80cc270 0x7fa7b80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4265302408 >> v2:172.21.15.144:3300/0 conn(0x7fa7b8159ac0 0x7fa7b815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4265302408 >> 172.21.15.60:0/4265302408 conn(0x7fa7b80ce6a0 msgr2=0x7fa7b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:46.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4265302408 shutdown_connections 2023-10-16T14:12:46.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.295+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4265302408 wait complete. 2023-10-16T14:12:46.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.297+0000 7fa7bd3d0640 1 Processor -- start 2023-10-16T14:12:46.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.297+0000 7fa7bd3d0640 1 -- start start 2023-10-16T14:12:46.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7bd3d0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80704e0 0x7fa7b8151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7bd3d0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 0x7fa7b80af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7bd3d0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7b8159ac0 0x7fa7b814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7bd3d0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8158b00 con 0x7fa7b80cc270 2023-10-16T14:12:46.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7bd3d0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80b3390 con 0x7fa7b8159ac0 2023-10-16T14:12:46.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7bd3d0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81502b0 con 0x7fa7b80704e0 2023-10-16T14:12:46.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 0x7fa7b80af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 0x7fa7b80af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48458/0 (socket says 172.21.15.60:48458) 2023-10-16T14:12:46.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.298+0000 7fa7affff640 1 -- 172.21.15.60:0/4271935769 learned_addr learned my addr 172.21.15.60:0/4271935769 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:46.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7b7fff640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80704e0 0x7fa7b8151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7bcbcf640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7b8159ac0 0x7fa7b814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7affff640 1 -- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80704e0 msgr2=0x7fa7b8151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7affff640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80704e0 0x7fa7b8151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7affff640 1 -- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7b8159ac0 msgr2=0x7fa7b814fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7affff640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7b8159ac0 0x7fa7b814fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7affff640 1 -- 172.21.15.60:0/4271935769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0009b30 con 0x7fa7b80cc270 2023-10-16T14:12:46.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7bcbcf640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7b8159ac0 0x7fa7b814fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:46.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7affff640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 0x7fa7b80af070 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000d800 tx=0x7fa7a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:46.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a00096a0 con 0x7fa7b80cc270 2023-10-16T14:12:46.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0009800 con 0x7fa7b80cc270 2023-10-16T14:12:46.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.299+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b80b3630 con 0x7fa7b80cc270 2023-10-16T14:12:46.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.300+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0009990 con 0x7fa7b80cc270 2023-10-16T14:12:46.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.300+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b80b3bf0 con 0x7fa7b80cc270 2023-10-16T14:12:46.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.301+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b804e9b0 con 0x7fa7b80cc270 2023-10-16T14:12:46.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.302+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa7a0009af0 con 0x7fa7b80cc270 2023-10-16T14:12:46.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.303+0000 7fa7b57fa640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79806b0e0 0x7fa79806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.303+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fa7a0089d10 con 0x7fa7b80cc270 2023-10-16T14:12:46.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.303+0000 7fa7b7fff640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79806b0e0 0x7fa79806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.305+0000 7fa7b7fff640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79806b0e0 0x7fa79806d5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7b815d870 tx=0x7fa7a8008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:46.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.308+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa7a0054c50 con 0x7fa7b80cc270 2023-10-16T14:12:46.429 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T14:12:46.426+0000 7f187850c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:12:46.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.580+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa7b806be90 con 0x7fa7b80cc270 2023-10-16T14:12:46.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.585+0000 7fa7b57fa640 1 -- 172.21.15.60:0/4271935769 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v59) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa7a000b3a0 con 0x7fa7b80cc270 2023-10-16T14:12:46.588 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T14:12:46.588 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T14:12:46.588 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T14:12:46.588 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T14:12:46.589 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:12:46.589 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T14:12:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79806b0e0 msgr2=0x7fa79806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79806b0e0 0x7fa79806d5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7b815d870 tx=0x7fa7a8008040 comp rx=0 tx=0).stop 2023-10-16T14:12:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 msgr2=0x7fa7b80af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 0x7fa7b80af070 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000d800 tx=0x7fa7a000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 shutdown_connections 2023-10-16T14:12:46.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fa79806b0e0 0x7fa79806d5a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7b8159ac0 0x7fa7b814fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7b80cc270 0x7fa7b80af070 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 --2- 172.21.15.60:0/4271935769 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa7b80704e0 0x7fa7b8151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 >> 172.21.15.60:0/4271935769 conn(0x7fa7b80ce6a0 msgr2=0x7fa7b806c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:46.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 shutdown_connections 2023-10-16T14:12:46.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.588+0000 7fa7bd3d0640 1 -- 172.21.15.60:0/4271935769 wait complete. 2023-10-16T14:12:46.611 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:12:46.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.828+0000 7f54a89ac640 1 Processor -- start 2023-10-16T14:12:46.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.828+0000 7f54a89ac640 1 -- start start 2023-10-16T14:12:46.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.829+0000 7f54a89ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 0x7f54a4071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a89ac640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f54a40cdf10 0x7f54a406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a89ac640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a40d0500 con 0x7f54a40720d0 2023-10-16T14:12:46.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a89ac640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a40d07c0 con 0x7f54a40cdf10 2023-10-16T14:12:46.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a89ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40d0aa0 con 0x7f54a415cba0 2023-10-16T14:12:46.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f549affd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f54a40720d0 0x7f54a40cd800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50976/0 (socket says 172.21.15.60:50976) 2023-10-16T14:12:46.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f549affd640 1 -- 172.21.15.60:0/3710993465 learned_addr learned my addr 172.21.15.60:0/3710993465 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:46.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a37fe640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 0x7f54a4071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a3fff640 1 --2- 172.21.15.60:0/3710993465 >> v2:172.21.15.144:3300/0 conn(0x7f54a40cdf10 0x7f54a406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.830+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692674296 0 0) 0x7f54a40d0500 con 0x7f54a40720d0 2023-10-16T14:12:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5490003520 con 0x7f54a40720d0 2023-10-16T14:12:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235295659 0 0) 0x7f5490003520 con 0x7f54a40720d0 2023-10-16T14:12:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 msgr2=0x7f54a4071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 0x7f54a4071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 >> v2:172.21.15.144:3300/0 conn(0x7f54a40cdf10 msgr2=0x7f54a406aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:46.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 --2- 172.21.15.60:0/3710993465 >> v2:172.21.15.144:3300/0 conn(0x7f54a40cdf10 0x7f54a406aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40d0d80 con 0x7f54a40720d0 2023-10-16T14:12:46.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f548c001670 con 0x7f54a40720d0 2023-10-16T14:12:46.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 >> v1:172.21.15.60:6789/0 conn(0x7f54a40720d0 legacy=0x7f54a40cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 0x7f5490003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490003f40 0x7f54900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490004d00 0x7f5490005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40d07c0 con 0x7f5490003f40 2023-10-16T14:12:46.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40d0aa0 con 0x7f5490004d00 2023-10-16T14:12:46.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549000b980 con 0x7f54a415cba0 2023-10-16T14:12:46.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.831+0000 7f549affd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490003f40 0x7f54900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.832+0000 7f54a37fe640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 0x7f5490003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.832+0000 7f549affd640 1 -- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 msgr2=0x7f5490003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:46.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.832+0000 7f549affd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54a415cba0 0x7f5490003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.832+0000 7f549affd640 1 -- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490004d00 msgr2=0x7f5490005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.832+0000 7f549affd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5490004d00 0x7f5490005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.832+0000 7f549affd640 1 -- 172.21.15.60:0/3710993465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490003520 con 0x7f5490003f40 2023-10-16T14:12:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.833+0000 7f549affd640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490003f40 0x7f54900047c0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c190 tx=0x7f54900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b4855e906f9a69 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.833+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c005ad0 con 0x7f5490003f40 2023-10-16T14:12:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.833+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c005c30 con 0x7f5490003f40 2023-10-16T14:12:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.833+0000 7f54a2ffd640 1 -- 172.21.15.60:0/3710993465 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c02c810 con 0x7f5490003f40 2023-10-16T14:12:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a89ac640 1 -- 172.21.15.60:0/3710993465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490003f40 msgr2=0x7f54900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a89ac640 1 --2- 172.21.15.60:0/3710993465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5490003f40 0x7f54900047c0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c190 tx=0x7f54900057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a3fff640 1 -- 172.21.15.60:0/3710993465 reap_dead start 2023-10-16T14:12:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a89ac640 1 -- 172.21.15.60:0/3710993465 shutdown_connections 2023-10-16T14:12:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a89ac640 1 -- 172.21.15.60:0/3710993465 >> 172.21.15.60:0/3710993465 conn(0x7f54a4065ed0 msgr2=0x7f54a40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a89ac640 1 -- 172.21.15.60:0/3710993465 shutdown_connections 2023-10-16T14:12:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.834+0000 7f54a89ac640 1 -- 172.21.15.60:0/3710993465 wait complete. 2023-10-16T14:12:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.836+0000 7f54a89ac640 1 Processor -- start 2023-10-16T14:12:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.836+0000 7f54a89ac640 1 -- start start 2023-10-16T14:12:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.837+0000 7f54a89ac640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490003f40 0x7f54a41f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.837+0000 7f54a89ac640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 0x7f54a41f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.837+0000 7f54a89ac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54a41f5e30 0x7f54a40ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.837+0000 7f54a89ac640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40c83f0 con 0x7f54a4159c50 2023-10-16T14:12:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.837+0000 7f54a89ac640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a41f62b0 con 0x7f54a41f5e30 2023-10-16T14:12:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.837+0000 7f54a89ac640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a41606c0 con 0x7f5490003f40 2023-10-16T14:12:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 0x7f54a41f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 0x7f54a41f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48478/0 (socket says 172.21.15.60:48478) 2023-10-16T14:12:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 -- 172.21.15.60:0/491633186 learned_addr learned my addr 172.21.15.60:0/491633186 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f54a37fe640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490003f40 0x7f54a41f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f54a3fff640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54a41f5e30 0x7f54a40ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 -- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490003f40 msgr2=0x7f54a41f2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490003f40 0x7f54a41f2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 -- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54a41f5e30 msgr2=0x7f54a40ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54a41f5e30 0x7f54a40ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 -- 172.21.15.60:0/491633186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40cb000 con 0x7f54a4159c50 2023-10-16T14:12:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.838+0000 7f549affd640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 0x7f54a41f58f0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c2b0 tx=0x7f548c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.839+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c03c040 con 0x7f54a4159c50 2023-10-16T14:12:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.839+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c007290 con 0x7f54a4159c50 2023-10-16T14:12:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.839+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c007640 con 0x7f54a4159c50 2023-10-16T14:12:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.839+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a40c7a50 con 0x7f54a4159c50 2023-10-16T14:12:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.839+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a40c8010 con 0x7f54a4159c50 2023-10-16T14:12:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.840+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a404e9b0 con 0x7f54a4159c50 2023-10-16T14:12:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.841+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f548c032070 con 0x7f54a4159c50 2023-10-16T14:12:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.842+0000 7f54a0ff9640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f548406b0b0 0x7f548406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:46.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.842+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f548c0ab250 con 0x7f54a4159c50 2023-10-16T14:12:46.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.843+0000 7f54a37fe640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f548406b0b0 0x7f548406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:46.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.844+0000 7f54a37fe640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f548406b0b0 0x7f548406d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5494007440 tx=0x7f549400b040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:46.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:46.846+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f548c075f60 con 0x7f54a4159c50 2023-10-16T14:12:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.082+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54a404e9b0 con 0x7f54a4159c50 2023-10-16T14:12:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.083+0000 7f54a0ff9640 1 -- 172.21.15.60:0/491633186 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f548c02de60 con 0x7f54a4159c50 2023-10-16T14:12:47.086 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:47.086 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:12:40.387056+0000","modified":"2023-10-16T14:12:43.460423+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12573":{"gid":12573,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2674455624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2674455624},{"type":"v1","addr":"172.21.15.60:6835","nonce":2674455624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T14:12:47.086 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 144 2023-10-16T14:12:47.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f548406b0b0 msgr2=0x7f548406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f548406b0b0 0x7f548406d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5494007440 tx=0x7f549400b040 comp rx=0 tx=0).stop 2023-10-16T14:12:47.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 msgr2=0x7f54a41f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 0x7f54a41f58f0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c2b0 tx=0x7f548c000fe0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 shutdown_connections 2023-10-16T14:12:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f548406b0b0 0x7f548406d570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54a41f5e30 0x7f54a40ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a4159c50 0x7f54a41f58f0 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 --2- 172.21.15.60:0/491633186 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5490003f40 0x7f54a41f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 >> 172.21.15.60:0/491633186 conn(0x7f54a4065ed0 msgr2=0x7f54a40cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 shutdown_connections 2023-10-16T14:12:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.086+0000 7f54a89ac640 1 -- 172.21.15.60:0/491633186 wait complete. 2023-10-16T14:12:47.511 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:12:40.387056+0000', 'modified': '2023-10-16T14:12:43.460423+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12573}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12573': {'gid': 12573, 'name': 'a', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/2674455624', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6833', 'nonce': 2674455624}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2674455624}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:12:47.511 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:12:47.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:12:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.748+0000 7fd284b87640 1 Processor -- start 2023-10-16T14:12:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.748+0000 7fd284b87640 1 -- start start 2023-10-16T14:12:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.749+0000 7fd284b87640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd28006c040 0x7fd28006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.749+0000 7fd284b87640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28015f4a0 con 0x7fd2800cda80 2023-10-16T14:12:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.749+0000 7fd284b87640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28006c960 con 0x7fd2800ad970 2023-10-16T14:12:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.750+0000 7fd284b87640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28015b9d0 con 0x7fd28006c040 2023-10-16T14:12:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.750+0000 7fd27ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd2800cda80 0x7fd28015ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50980/0 (socket says 172.21.15.60:50980) 2023-10-16T14:12:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.750+0000 7fd27ffff640 1 -- 172.21.15.60:0/1729542488 learned_addr learned my addr 172.21.15.60:0/1729542488 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.750+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/1729542488 >> v2:172.21.15.144:3300/0 conn(0x7fd28006c040 0x7fd28006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.750+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882275333 0 0) 0x7fd28015f4a0 con 0x7fd2800cda80 2023-10-16T14:12:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.750+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c003520 con 0x7fd2800cda80 2023-10-16T14:12:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127349289 0 0) 0x7fd26c003520 con 0x7fd2800cda80 2023-10-16T14:12:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 >> v2:172.21.15.144:3300/0 conn(0x7fd28006c040 msgr2=0x7fd28006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 --2- 172.21.15.60:0/1729542488 >> v2:172.21.15.144:3300/0 conn(0x7fd28006c040 0x7fd28006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 >> v1:172.21.15.144:6789/0 conn(0x7fd2800ad970 legacy=0x7fd28015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28015bcb0 con 0x7fd2800cda80 2023-10-16T14:12:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd278001670 con 0x7fd2800cda80 2023-10-16T14:12:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 >> v1:172.21.15.60:6789/0 conn(0x7fd2800cda80 legacy=0x7fd28015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd26c003970 0x7fd26c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26c0047c0 0x7fd26c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c005180 0x7fd26c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28015b9d0 con 0x7fd26c0047c0 2023-10-16T14:12:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c00c020 con 0x7fd26c003970 2023-10-16T14:12:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c00c1e0 con 0x7fd26c005180 2023-10-16T14:12:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.751+0000 7fd27effd640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26c0047c0 0x7fd26c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd26c003970 0x7fd26c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27ffff640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c005180 0x7fd26c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27effd640 1 -- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c005180 msgr2=0x7fd26c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27effd640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c005180 0x7fd26c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27effd640 1 -- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd26c003970 msgr2=0x7fd26c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27effd640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd26c003970 0x7fd26c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27effd640 1 -- 172.21.15.60:0/1729542488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c003520 con 0x7fd26c0047c0 2023-10-16T14:12:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.752+0000 7fd27ffff640 1 -- 172.21.15.60:0/1729542488 reap_dead start 2023-10-16T14:12:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.753+0000 7fd27effd640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26c0047c0 0x7fd26c004c40 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fd268001720 tx=0x7fd26c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dca4da18236722b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.753+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680052d0 con 0x7fd26c0047c0 2023-10-16T14:12:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.753+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26802e050 con 0x7fd26c0047c0 2023-10-16T14:12:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.753+0000 7fd27e7fc640 1 -- 172.21.15.60:0/1729542488 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268003830 con 0x7fd26c0047c0 2023-10-16T14:12:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 -- 172.21.15.60:0/1729542488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26c0047c0 msgr2=0x7fd26c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26c0047c0 0x7fd26c004c40 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fd268001720 tx=0x7fd26c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 -- 172.21.15.60:0/1729542488 shutdown_connections 2023-10-16T14:12:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 --2- 172.21.15.60:0/1729542488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd26c0047c0 0x7fd26c004c40 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 -- 172.21.15.60:0/1729542488 >> 172.21.15.60:0/1729542488 conn(0x7fd2800ce6a0 msgr2=0x7fd2800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 -- 172.21.15.60:0/1729542488 shutdown_connections 2023-10-16T14:12:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.754+0000 7fd284b87640 1 -- 172.21.15.60:0/1729542488 wait complete. 2023-10-16T14:12:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.756+0000 7fd284b87640 1 Processor -- start 2023-10-16T14:12:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.756+0000 7fd284b87640 1 -- start start 2023-10-16T14:12:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd284b87640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c0047c0 0x7fd280158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd284b87640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 0x7fd2801504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd284b87640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2800ac890 0x7fd2800acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd284b87640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800c8370 con 0x7fd280153830 2023-10-16T14:12:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd284b87640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800ad250 con 0x7fd2800ac890 2023-10-16T14:12:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd284b87640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd280150a00 con 0x7fd26c0047c0 2023-10-16T14:12:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd27effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 0x7fd2801504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd27effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 0x7fd2801504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48502/0 (socket says 172.21.15.60:48502) 2023-10-16T14:12:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd27effd640 1 -- 172.21.15.60:0/1205080985 learned_addr learned my addr 172.21.15.60:0/1205080985 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.757+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c0047c0 0x7fd280158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27effd640 1 -- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c0047c0 msgr2=0x7fd280158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27effd640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c0047c0 0x7fd280158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27ffff640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2800ac890 0x7fd2800acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27effd640 1 -- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2800ac890 msgr2=0x7fd2800acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27effd640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2800ac890 0x7fd2800acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27effd640 1 -- 172.21.15.60:0/1205080985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd280159060 con 0x7fd280153830 2023-10-16T14:12:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27ffff640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2800ac890 0x7fd2800acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd27effd640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 0x7fd2801504c0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fd26802bbc0 tx=0x7fd268001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd277fff640 1 -- 172.21.15.60:0/1205080985 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680076c0 con 0x7fd280153830 2023-10-16T14:12:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd277fff640 1 -- 172.21.15.60:0/1205080985 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd268007a30 con 0x7fd280153830 2023-10-16T14:12:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd277fff640 1 -- 172.21.15.60:0/1205080985 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268006be0 con 0x7fd280153830 2023-10-16T14:12:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.758+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd280159340 con 0x7fd280153830 2023-10-16T14:12:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.759+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd280159900 con 0x7fd280153830 2023-10-16T14:12:47.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.761+0000 7fd277fff640 1 -- 172.21.15.60:0/1205080985 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd268006d40 con 0x7fd280153830 2023-10-16T14:12:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.762+0000 7fd277fff640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd26006b0e0 0x7fd26006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.762+0000 7fd277fff640 1 -- 172.21.15.60:0/1205080985 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fd2680ab7b0 con 0x7fd280153830 2023-10-16T14:12:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.762+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd26006b0e0 0x7fd26006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.763+0000 7fd28897d740 1 Processor -- start 2023-10-16T14:12:47.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.763+0000 7fd28897d740 1 -- start start 2023-10-16T14:12:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.763+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd26006b0e0 0x7fd26006d5a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd270007520 tx=0x7fd27000b040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa170 0x5600dc67d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd28897d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 0x5600dc6acf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc67d5b0 0x5600dc6ad440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd28897d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc59d7a0 con 0x5600dc6aa550 2023-10-16T14:12:47.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc5ef740 con 0x5600dc6aa170 2023-10-16T14:12:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600dc5ece30 con 0x5600dc67d5b0 2023-10-16T14:12:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 0x5600dc6acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 0x5600dc6acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48504/0 (socket says 172.21.15.60:48504) 2023-10-16T14:12:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27effd640 1 -- 172.21.15.60:0/3984766934 learned_addr learned my addr 172.21.15.60:0/3984766934 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:47.776 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27ffff640 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc67d5b0 0x5600dc6ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa170 0x5600dc67d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27effd640 1 -- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc67d5b0 msgr2=0x5600dc6ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.764+0000 7fd27effd640 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc67d5b0 0x5600dc6ad440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd27effd640 1 -- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa170 msgr2=0x5600dc67d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.777 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd27effd640 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa170 0x5600dc67d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd27effd640 1 -- 172.21.15.60:0/3984766934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600dc5ac910 con 0x5600dc6aa550 2023-10-16T14:12:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa170 0x5600dc67d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd27effd640 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 0x5600dc6acf00 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fd2680050a0 tx=0x7fd26807a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4223ebc20007b685 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd2757fa640 1 -- 172.21.15.60:0/3984766934 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680784e0 con 0x5600dc6aa550 2023-10-16T14:12:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd2757fa640 1 -- 172.21.15.60:0/3984766934 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26808de90 con 0x5600dc6aa550 2023-10-16T14:12:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.765+0000 7fd2757fa640 1 -- 172.21.15.60:0/3984766934 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268076850 con 0x5600dc6aa550 2023-10-16T14:12:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 -- 172.21.15.60:0/3984766934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 msgr2=0x5600dc6acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 0x5600dc6acf00 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fd2680050a0 tx=0x7fd26807a140 comp rx=0 tx=0).stop 2023-10-16T14:12:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 -- 172.21.15.60:0/3984766934 shutdown_connections 2023-10-16T14:12:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc67d5b0 0x5600dc6ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa550 0x5600dc6acf00 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 --2- 172.21.15.60:0/3984766934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa170 0x5600dc67d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 -- 172.21.15.60:0/3984766934 >> 172.21.15.60:0/3984766934 conn(0x5600dc67f380 msgr2=0x5600dc6c7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 -- 172.21.15.60:0/3984766934 shutdown_connections 2023-10-16T14:12:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.766+0000 7fd28897d740 1 -- 172.21.15.60:0/3984766934 wait complete. 2023-10-16T14:12:47.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.767+0000 7fd28897d740 1 -- start start 2023-10-16T14:12:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.767+0000 7fd28897d740 1 Processor -- start 2023-10-16T14:12:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc6af950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc697c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc5e8720 con 0x5600dc6aa170 2023-10-16T14:12:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc586970 con 0x5600dc6c2530 2023-10-16T14:12:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600dc586610 con 0x5600dc6aa550 2023-10-16T14:12:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd28897d740 10 client.0 _ll_register_callbacks cb 0x5600dc5a89d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:12:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd27f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc6af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.782 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd27f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc6af950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48508/0 (socket says 172.21.15.60:48508) 2023-10-16T14:12:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.768+0000 7fd27f7fe640 1 -- 172.21.15.60:0/3657389872 learned_addr learned my addr 172.21.15.60:0/3657389872 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd27f7fe640 1 -- 172.21.15.60:0/3657389872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 msgr2=0x5600dc697c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc697c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd27f7fe640 1 -- 172.21.15.60:0/3657389872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 msgr2=0x5600dc6c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd27f7fe640 1 -- 172.21.15.60:0/3657389872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd280159900 con 0x5600dc6aa170 2023-10-16T14:12:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc6af950 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fd2700045b0 tx=0x7fd270004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4433a8922634ca0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd25e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:12:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd25e7fc640 1 -- 172.21.15.60:0/3657389872 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270028070 con 0x5600dc6aa170 2023-10-16T14:12:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd25e7fc640 1 -- 172.21.15.60:0/3657389872 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fd270019c50 con 0x5600dc6aa170 2023-10-16T14:12:47.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.769+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600dc60f160 con 0x5600dc6aa170 2023-10-16T14:12:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd25e7fc640 1 -- 172.21.15.60:0/3657389872 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd27000e040 con 0x5600dc6aa170 2023-10-16T14:12:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 10 client.12708 fetch_fsmap learned FSMap version 144 2023-10-16T14:12:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x5600dc5e6da0 con 0x5600dc6aa170 2023-10-16T14:12:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd25e7fc640 1 -- 172.21.15.60:0/3657389872 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5600dc60f160 con 0x5600dc6aa170 2023-10-16T14:12:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd25e7fc640 1 -- 172.21.15.60:0/3657389872 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd27002b020 con 0x5600dc6aa170 2023-10-16T14:12:47.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 10 client.12708 fetch_fsmap finished waiting for FSMap version 144 2023-10-16T14:12:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 10 client.12708 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:12:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 20 client.12708 populate_metadata read hostname 'smithi060' 2023-10-16T14:12:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc69e190 0x5600dc69a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 4 client.12708 mds_command: new command op to 12573 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:12:47.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 --> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600dc5bacc0 con 0x5600dc69e190 2023-10-16T14:12:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.770+0000 7fd27effd640 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc69e190 0x5600dc69a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.771+0000 7fd27effd640 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc69e190 0x5600dc69a680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2df062a149f5d99b server_cookie=cf0bd455e716c7ba in_seq=0 out_seq=0 2023-10-16T14:12:47.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.771+0000 7fd25e7fc640 10 client.12708 ms_handle_connect on v2:172.21.15.60:6833/2674455624 2023-10-16T14:12:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd25e7fc640 1 -- 172.21.15.60:0/3657389872 <== mds.0 v2:172.21.15.60:6833/2674455624 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5600dc5bacc0 con 0x5600dc69e190 2023-10-16T14:12:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd25e7fc640 10 client.12708 handle_command_reply: tid=0 2023-10-16T14:12:47.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 client.12708 shutdown 2023-10-16T14:12:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 msgr2=0x5600dc6af950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc6af950 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fd2700045b0 tx=0x7fd270004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 shutdown_connections 2023-10-16T14:12:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc69e190 0x5600dc69a680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc697c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 --2- 172.21.15.60:0/3657389872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc6af950 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 >> 172.21.15.60:0/3657389872 conn(0x5600dc67d5b0 msgr2=0x5600dc6ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.772+0000 7fd25e7fc640 0 client.12708 ms_handle_reset on v2:172.21.15.60:6833/2674455624 2023-10-16T14:12:47.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.773+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 shutdown_connections 2023-10-16T14:12:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.773+0000 7fd28897d740 1 -- 172.21.15.60:0/3657389872 wait complete. 2023-10-16T14:12:47.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.773+0000 7fd28897d740 20 client.12708 trim_cache size 0 max 16384 2023-10-16T14:12:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.789+0000 7fd28897d740 1 Processor -- start 2023-10-16T14:12:47.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.790+0000 7fd28897d740 1 -- start start 2023-10-16T14:12:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.790+0000 7fd28897d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc681240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.790+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc6a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.790+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.790+0000 7fd28897d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc34cf20 con 0x5600dc6aa170 2023-10-16T14:12:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc6b0f50 con 0x5600dc6c2530 2023-10-16T14:12:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600dc6b05e0 con 0x5600dc6aa550 2023-10-16T14:12:47.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc681240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc681240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48524/0 (socket says 172.21.15.60:48524) 2023-10-16T14:12:47.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 -- 172.21.15.60:0/625629635 learned_addr learned my addr 172.21.15.60:0/625629635 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:47.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27effd640 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc6a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.796 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27ffff640 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 -- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 msgr2=0x5600dc6a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc6a86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 -- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 msgr2=0x5600dc6a8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6a8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27f7fe640 1 -- 172.21.15.60:0/625629635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600dc585b80 con 0x5600dc6aa170 2023-10-16T14:12:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.791+0000 7fd27ffff640 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6a8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.792+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc681240 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fd270008ff0 tx=0x7fd270022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a735dc50e5d297 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.792+0000 7fd275ffb640 1 -- 172.21.15.60:0/625629635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27000e070 con 0x5600dc6aa170 2023-10-16T14:12:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.792+0000 7fd275ffb640 1 -- 172.21.15.60:0/625629635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27001f050 con 0x5600dc6aa170 2023-10-16T14:12:47.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.792+0000 7fd275ffb640 1 -- 172.21.15.60:0/625629635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270028040 con 0x5600dc6aa170 2023-10-16T14:12:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 -- 172.21.15.60:0/625629635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 msgr2=0x5600dc681240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc681240 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fd270008ff0 tx=0x7fd270022000 comp rx=0 tx=0).stop 2023-10-16T14:12:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 -- 172.21.15.60:0/625629635 shutdown_connections 2023-10-16T14:12:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa550 0x5600dc6a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6c2530 0x5600dc6a8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 --2- 172.21.15.60:0/625629635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6aa170 0x5600dc681240 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 -- 172.21.15.60:0/625629635 >> 172.21.15.60:0/625629635 conn(0x5600dc67d5b0 msgr2=0x5600dc67c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 -- 172.21.15.60:0/625629635 shutdown_connections 2023-10-16T14:12:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.793+0000 7fd28897d740 1 -- 172.21.15.60:0/625629635 wait complete. 2023-10-16T14:12:47.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.794+0000 7fd28897d740 1 -- start start 2023-10-16T14:12:47.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.794+0000 7fd28897d740 1 Processor -- start 2023-10-16T14:12:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa170 0x5600dc729dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa550 0x5600dc6da5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 0x5600dc6dab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc6c78e0 con 0x5600dc6c2530 2023-10-16T14:12:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600dc697b70 con 0x5600dc6aa550 2023-10-16T14:12:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600dc72a6f0 con 0x5600dc6aa170 2023-10-16T14:12:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.795+0000 7fd28897d740 10 client.0 _ll_register_callbacks cb 0x7fd270009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:12:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 0x5600dc6dab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 0x5600dc6dab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48536/0 (socket says 172.21.15.60:48536) 2023-10-16T14:12:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 -- 172.21.15.60:0/2519708524 learned_addr learned my addr 172.21.15.60:0/2519708524 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 -- 172.21.15.60:0/2519708524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa170 msgr2=0x5600dc729dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa170 0x5600dc729dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 -- 172.21.15.60:0/2519708524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa550 msgr2=0x5600dc6da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:47.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa550 0x5600dc6da5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 -- 172.21.15.60:0/2519708524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600dc5e6da0 con 0x5600dc6c2530 2023-10-16T14:12:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd27ffff640 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 0x5600dc6dab30 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fd28015e5c0 tx=0x7fd278010730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7ddf3f39a16e61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.796+0000 7fd25e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:12:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd25e7fc640 1 -- 172.21.15.60:0/2519708524 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278018070 con 0x5600dc6c2530 2023-10-16T14:12:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd25e7fc640 1 -- 172.21.15.60:0/2519708524 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fd27800c040 con 0x5600dc6c2530 2023-10-16T14:12:47.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd26808de90 con 0x5600dc6c2530 2023-10-16T14:12:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd25e7fc640 1 -- 172.21.15.60:0/2519708524 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd278010e50 con 0x5600dc6c2530 2023-10-16T14:12:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 10 client.12714 fetch_fsmap learned FSMap version 144 2023-10-16T14:12:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x5600dc5ab170 con 0x5600dc6c2530 2023-10-16T14:12:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd25e7fc640 1 -- 172.21.15.60:0/2519708524 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26808de90 con 0x5600dc6c2530 2023-10-16T14:12:47.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd25e7fc640 1 -- 172.21.15.60:0/2519708524 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd27801b020 con 0x5600dc6c2530 2023-10-16T14:12:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 10 client.12714 fetch_fsmap finished waiting for FSMap version 144 2023-10-16T14:12:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 10 client.12714 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:12:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 20 client.12714 populate_metadata read hostname 'smithi060' 2023-10-16T14:12:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc6c7190 0x5600dc696920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:47.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 4 client.12714 mds_command: new command op to 12573 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:12:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.797+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 --> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5600dc5e1700 con 0x5600dc6c7190 2023-10-16T14:12:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.798+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc6c7190 0x5600dc696920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.798+0000 7fd27f7fe640 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc6c7190 0x5600dc696920 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9fe2af41376336 server_cookie=fd331ee15998278a in_seq=0 out_seq=0 2023-10-16T14:12:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.798+0000 7fd25e7fc640 10 client.12714 ms_handle_connect on v2:172.21.15.60:6833/2674455624 2023-10-16T14:12:47.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.798+0000 7fd25e7fc640 1 -- 172.21.15.60:0/2519708524 <== mds.0 v2:172.21.15.60:6833/2674455624 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5600dc5e1700 con 0x5600dc6c7190 2023-10-16T14:12:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.798+0000 7fd25e7fc640 10 client.12714 handle_command_reply: tid=0 2023-10-16T14:12:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.798+0000 7fd28897d740 1 client.12714 shutdown 2023-10-16T14:12:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 msgr2=0x5600dc6dab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 0x5600dc6dab30 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fd28015e5c0 tx=0x7fd278010730 comp rx=0 tx=0).stop 2023-10-16T14:12:47.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 shutdown_connections 2023-10-16T14:12:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:6833/2674455624,v1:172.21.15.60:6835/2674455624] conn(0x5600dc6c7190 0x5600dc696920 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5600dc6c2530 0x5600dc6dab30 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x5600dc6aa550 0x5600dc6da5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 --2- 172.21.15.60:0/2519708524 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x5600dc6aa170 0x5600dc729dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 >> 172.21.15.60:0/2519708524 conn(0x5600dc67d5b0 msgr2=0x5600dc682bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd25e7fc640 0 client.12714 ms_handle_reset on v2:172.21.15.60:6833/2674455624 2023-10-16T14:12:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 shutdown_connections 2023-10-16T14:12:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 1 -- 172.21.15.60:0/2519708524 wait complete. 2023-10-16T14:12:47.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.799+0000 7fd28897d740 20 client.12714 trim_cache size 0 max 16384 2023-10-16T14:12:47.813 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:12:47.814 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:12:47.814 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:12:47.814 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 12573, 2023-10-16T14:12:47.814 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:12:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:12:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T14:12:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8941245459999996, 2023-10-16T14:12:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 144, 2023-10-16T14:12:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 273, 2023-10-16T14:12:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 273, 2023-10-16T14:12:47.816 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.079915274999999 2023-10-16T14:12:47.816 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:12:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd26006b0e0 msgr2=0x7fd26006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd26006b0e0 0x7fd26006d5a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd270007520 tx=0x7fd27000b040 comp rx=0 tx=0).stop 2023-10-16T14:12:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 msgr2=0x7fd2801504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 0x7fd2801504c0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fd26802bbc0 tx=0x7fd268001120 comp rx=0 tx=0).stop 2023-10-16T14:12:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 shutdown_connections 2023-10-16T14:12:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd26006b0e0 0x7fd26006d5a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2800ac890 0x7fd2800acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd280153830 0x7fd2801504c0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 --2- 172.21.15.60:0/1205080985 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd26c0047c0 0x7fd280158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 >> 172.21.15.60:0/1205080985 conn(0x7fd2800ce6a0 msgr2=0x7fd2800cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 shutdown_connections 2023-10-16T14:12:47.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:47.800+0000 7fd284b87640 1 -- 172.21.15.60:0/1205080985 wait complete. 2023-10-16T14:12:47.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:12:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.039+0000 7f384e475640 1 Processor -- start 2023-10-16T14:12:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.040+0000 7f384e475640 1 -- start start 2023-10-16T14:12:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.041+0000 7f384e475640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.041+0000 7f384e475640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3848160830 0x7f3848160c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.041+0000 7f384e475640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38480ca0a0 con 0x7f38480702b0 2023-10-16T14:12:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.041+0000 7f384e475640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3848161840 con 0x7f3848160830 2023-10-16T14:12:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.041+0000 7f384e475640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3848161280 con 0x7f384815cb20 2023-10-16T14:12:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3848160830 0x7f3848160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3848160830 0x7f3848160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48540/0 (socket says 172.21.15.60:48540) 2023-10-16T14:12:48.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 -- 172.21.15.60:0/170750666 learned_addr learned my addr 172.21.15.60:0/170750666 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384d473640 1 --2- 172.21.15.60:0/170750666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 -- 172.21.15.60:0/170750666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 msgr2=0x7f384806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 --2- 172.21.15.60:0/170750666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 -- 172.21.15.60:0/170750666 >> v1:172.21.15.60:6789/0 conn(0x7f38480702b0 legacy=0x7f38480cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f384cc72640 1 -- 172.21.15.60:0/170750666 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848161560 con 0x7f3848160830 2023-10-16T14:12:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.042+0000 7f383ffff640 1 -- 172.21.15.60:0/170750666 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670992155 0 0) 0x7f38480ca0a0 con 0x7f38480702b0 2023-10-16T14:12:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.044+0000 7f384cc72640 1 --2- 172.21.15.60:0/170750666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f3848160c90 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f3838009870 tx=0x7f383802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8583aa9cfc117b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.044+0000 7f383ffff640 1 -- 172.21.15.60:0/170750666 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838030840 con 0x7f3848160830 2023-10-16T14:12:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.044+0000 7f383ffff640 1 -- 172.21.15.60:0/170750666 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38380309a0 con 0x7f3848160830 2023-10-16T14:12:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.044+0000 7f383ffff640 1 -- 172.21.15.60:0/170750666 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838030d30 con 0x7f3848160830 2023-10-16T14:12:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.044+0000 7f384e475640 1 -- 172.21.15.60:0/170750666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 msgr2=0x7f3848160c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.044+0000 7f384e475640 1 --2- 172.21.15.60:0/170750666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f3848160c90 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f3838009870 tx=0x7f383802b980 comp rx=0 tx=0).stop 2023-10-16T14:12:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.045+0000 7f384e475640 1 -- 172.21.15.60:0/170750666 shutdown_connections 2023-10-16T14:12:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.045+0000 7f384e475640 1 --2- 172.21.15.60:0/170750666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f3848160c90 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.045+0000 7f384e475640 1 --2- 172.21.15.60:0/170750666 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.045+0000 7f384e475640 1 -- 172.21.15.60:0/170750666 >> 172.21.15.60:0/170750666 conn(0x7f38480cf7c0 msgr2=0x7f38480cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.045+0000 7f384e475640 1 -- 172.21.15.60:0/170750666 shutdown_connections 2023-10-16T14:12:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.045+0000 7f384e475640 1 -- 172.21.15.60:0/170750666 wait complete. 2023-10-16T14:12:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.046+0000 7f384e475640 1 Processor -- start 2023-10-16T14:12:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.047+0000 7f384e475640 1 -- start start 2023-10-16T14:12:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.047+0000 7f384e475640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38480702b0 0x7f38481f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384e475640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384e475640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f384815e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384e475640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384815df50 con 0x7f3848160830 2023-10-16T14:12:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384e475640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38481f3010 con 0x7f38480702b0 2023-10-16T14:12:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384e475640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38481f32f0 con 0x7f384815cb20 2023-10-16T14:12:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f384815e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f384815e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48554/0 (socket says 172.21.15.60:48554) 2023-10-16T14:12:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 -- 172.21.15.60:0/3573126482 learned_addr learned my addr 172.21.15.60:0/3573126482 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384cc72640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 -- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 msgr2=0x7f384815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384815baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 -- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38480702b0 msgr2=0x7f38481f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38480702b0 0x7f38481f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.048+0000 7f384dc74640 1 -- 172.21.15.60:0/3573126482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38380094d0 con 0x7f3848160830 2023-10-16T14:12:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.049+0000 7f384dc74640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f384815e730 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f384000c710 tx=0x7f384000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.049+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840003da0 con 0x7f3848160830 2023-10-16T14:12:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.049+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384000ce20 con 0x7f3848160830 2023-10-16T14:12:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.049+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38481f35d0 con 0x7f3848160830 2023-10-16T14:12:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.049+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840005290 con 0x7f3848160830 2023-10-16T14:12:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.049+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38481f2c20 con 0x7f3848160830 2023-10-16T14:12:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.050+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384815e450 con 0x7f3848160830 2023-10-16T14:12:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.051+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f38400053f0 con 0x7f3848160830 2023-10-16T14:12:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.052+0000 7f383dffb640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f382c06b0b0 0x7f382c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:48.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.052+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f3840089240 con 0x7f3848160830 2023-10-16T14:12:48.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.053+0000 7f384d473640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f382c06b0b0 0x7f382c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:48.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.054+0000 7f384d473640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f382c06b0b0 0x7f382c06d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f38440098f0 tx=0x7f3844008040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.057+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3840053ff0 con 0x7f3848160830 2023-10-16T14:12:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.291+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f384804e9b0 con 0x7f3848160830 2023-10-16T14:12:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.293+0000 7f383dffb640 1 -- 172.21.15.60:0/3573126482 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v144) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3840017060 con 0x7f3848160830 2023-10-16T14:12:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:12:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f382c06b0b0 msgr2=0x7f382c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f382c06b0b0 0x7f382c06d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f38440098f0 tx=0x7f3844008040 comp rx=0 tx=0).stop 2023-10-16T14:12:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 msgr2=0x7f384815e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f384815e730 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f384000c710 tx=0x7f384000cbd0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 shutdown_connections 2023-10-16T14:12:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f382c06b0b0 0x7f382c06d570 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f384815cb20 0x7f384815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3848160830 0x7f384815e730 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.295+0000 7f384e475640 1 --2- 172.21.15.60:0/3573126482 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f38480702b0 0x7f38481f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.296+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 >> 172.21.15.60:0/3573126482 conn(0x7f38480cf7c0 msgr2=0x7f38480ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.296+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 shutdown_connections 2023-10-16T14:12:48.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:48.296+0000 7f384e475640 1 -- 172.21.15.60:0/3573126482 wait complete. 2023-10-16T14:12:48.318 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:12:48.319 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:12:48.319 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T14:12:48.319 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:12:48.320 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:12:48.320 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:12:48.320 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:12:48.321 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:12:48.321 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:12:48.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:48.322 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T14:12:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:12:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:12:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:12:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T14:12:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:12:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T14:12:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T14:12:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:12:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T14:12:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T14:12:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:12:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:12:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T14:12:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:12:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 36419sec preferred_lft 36419sec 2023-10-16T14:12:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T14:12:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:12:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:12:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T14:12:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:12:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T14:12:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:12:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T14:12:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:12:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:12:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T14:12:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:12:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:12:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T14:12:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:12:48.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:48.387 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:12:48.485 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T14:12:48.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:48.490 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T14:12:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T14:12:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:12:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:12:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:12:48.548 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:12:48.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:48.549 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T14:12:48.550 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T14:12:48.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:12:48.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:12:48.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:12:48.551 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T14:12:48.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:48.758 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:12:48.814 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:12:48.814 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:12:48.815 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:12:48.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:48.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T14:12:48.911 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:12:48.912 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:12:48.970 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:12:48.971 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:12:48.971 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:12:49.025 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T14:12:49.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:49.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:12:49.161 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:12:49.162 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:12:49.193 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:12:49.193 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:12:49.194 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T14:12:49.194 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T14:12:49.194 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T14:12:49.195 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:12:49.195 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:12:49.195 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:12:49.195 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:12:49.196 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:12:49.199 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:12:49.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:49.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:12:49.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:49.281 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:12:49.350 INFO:teuthology.orchestra.run.smithi060.stdout:client8633 2023-10-16T14:12:49.351 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:12:49.351 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:12:49.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:12:49.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.638+0000 7f83c9f01640 1 Processor -- start 2023-10-16T14:12:49.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.638+0000 7f83c9f01640 1 -- start start 2023-10-16T14:12:49.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.639+0000 7f83c9f01640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f83c40ad350 0x7f83c4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:49.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.639+0000 7f83c9f01640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83c406a2e0 0x7f83c40c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:49.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.639+0000 7f83c9f01640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c406a730 con 0x7f83c415d150 2023-10-16T14:12:49.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.639+0000 7f83c9f01640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c415d530 con 0x7f83c406a2e0 2023-10-16T14:12:49.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.639+0000 7f83c9f01640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c40ac890 con 0x7f83c40ad350 2023-10-16T14:12:49.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83c406a2e0 0x7f83c40c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:49.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83c406a2e0 0x7f83c40c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51834/0 (socket says 172.21.15.60:51834) 2023-10-16T14:12:49.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 -- 172.21.15.60:0/3975227758 learned_addr learned my addr 172.21.15.60:0/3975227758 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:49.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c2ffd640 1 --2- 172.21.15.60:0/3975227758 >> v2:172.21.15.144:3300/0 conn(0x7f83c40ad350 0x7f83c4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:49.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 -- 172.21.15.60:0/3975227758 >> v2:172.21.15.144:3300/0 conn(0x7f83c40ad350 msgr2=0x7f83c4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:49.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 --2- 172.21.15.60:0/3975227758 >> v2:172.21.15.144:3300/0 conn(0x7f83c40ad350 0x7f83c4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 -- 172.21.15.60:0/3975227758 >> v1:172.21.15.144:6789/0 conn(0x7f83c415d150 legacy=0x7f83c4160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:49.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.640+0000 7f83c3fff640 1 -- 172.21.15.60:0/3975227758 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c40ad7a0 con 0x7f83c406a2e0 2023-10-16T14:12:49.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.642+0000 7f83c3fff640 1 --2- 172.21.15.60:0/3975227758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c406a2e0 0x7f83c40c9000 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f83c40cf990 tx=0x7f83bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f7cfaf74ec30bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:49.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.642+0000 7f83c27fc640 1 -- 172.21.15.60:0/3975227758 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc02ca80 con 0x7f83c406a2e0 2023-10-16T14:12:49.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.642+0000 7f83c27fc640 1 -- 172.21.15.60:0/3975227758 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83bc02cbe0 con 0x7f83c406a2e0 2023-10-16T14:12:49.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.642+0000 7f83c27fc640 1 -- 172.21.15.60:0/3975227758 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0343f0 con 0x7f83c406a2e0 2023-10-16T14:12:49.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 -- 172.21.15.60:0/3975227758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c406a2e0 msgr2=0x7f83c40c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:49.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 --2- 172.21.15.60:0/3975227758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c406a2e0 0x7f83c40c9000 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f83c40cf990 tx=0x7f83bc02bee0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 -- 172.21.15.60:0/3975227758 shutdown_connections 2023-10-16T14:12:49.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 --2- 172.21.15.60:0/3975227758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c406a2e0 0x7f83c40c9000 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 --2- 172.21.15.60:0/3975227758 >> v2:172.21.15.144:3300/0 conn(0x7f83c40ad350 0x7f83c4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.648 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 -- 172.21.15.60:0/3975227758 >> 172.21.15.60:0/3975227758 conn(0x7f83c40ce9b0 msgr2=0x7f83c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:49.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 -- 172.21.15.60:0/3975227758 shutdown_connections 2023-10-16T14:12:49.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.643+0000 7f83c9f01640 1 -- 172.21.15.60:0/3975227758 wait complete. 2023-10-16T14:12:49.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.645+0000 7f83c9f01640 1 Processor -- start 2023-10-16T14:12:49.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.645+0000 7f83c9f01640 1 -- start start 2023-10-16T14:12:49.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c9f01640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83c406a2e0 0x7f83c40c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:49.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c9f01640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 0x7f83c40b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:49.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c9f01640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83c415d150 0x7f83c40c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:49.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c9f01640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c41561f0 con 0x7f83c40ad350 2023-10-16T14:12:49.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c9f01640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40c83b0 con 0x7f83c415d150 2023-10-16T14:12:49.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c9f01640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c40ae590 con 0x7f83c406a2e0 2023-10-16T14:12:49.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 0x7f83c40b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:49.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 0x7f83c40b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51850/0 (socket says 172.21.15.60:51850) 2023-10-16T14:12:49.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.646+0000 7f83c2ffd640 1 -- 172.21.15.60:0/4261596502 learned_addr learned my addr 172.21.15.60:0/4261596502 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:49.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c37fe640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83c406a2e0 0x7f83c40c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:49.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c3fff640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83c415d150 0x7f83c40c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:49.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c2ffd640 1 -- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83c406a2e0 msgr2=0x7f83c40c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:49.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c2ffd640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83c406a2e0 0x7f83c40c7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c2ffd640 1 -- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83c415d150 msgr2=0x7f83c40c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:49.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c2ffd640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83c415d150 0x7f83c40c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c2ffd640 1 -- 172.21.15.60:0/4261596502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc009b30 con 0x7f83c40ad350 2023-10-16T14:12:49.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c2ffd640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 0x7f83c40b36c0 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f83b800d800 tx=0x7f83b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:49.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b800b6c0 con 0x7f83c40ad350 2023-10-16T14:12:49.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b800b820 con 0x7f83c40ad350 2023-10-16T14:12:49.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b800bbb0 con 0x7f83c40ad350 2023-10-16T14:12:49.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.647+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c40ae870 con 0x7f83c40ad350 2023-10-16T14:12:49.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.648+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c40aee30 con 0x7f83c40ad350 2023-10-16T14:12:49.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.649+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c404e160 con 0x7f83c40ad350 2023-10-16T14:12:49.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.650+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f83b800bd10 con 0x7f83c40ad350 2023-10-16T14:12:49.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.650+0000 7f83abfff640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f83b006b100 0x7f83b006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:49.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.651+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f83b808a080 con 0x7f83c40ad350 2023-10-16T14:12:49.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.651+0000 7f83c37fe640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f83b006b100 0x7f83b006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:49.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.652+0000 7f83c37fe640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f83b006b100 0x7f83b006d5c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83b40015d0 tx=0x7f83b4009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:49.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.655+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83b80bcfe0 con 0x7f83c40ad350 2023-10-16T14:12:49.883 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.880+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 --> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f83c4161930 con 0x7f83b006b100 2023-10-16T14:12:49.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.884+0000 7f83abfff640 1 -- 172.21.15.60:0/4261596502 <== mgr.8393 v2:172.21.15.144:6832/948422863 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f83b40050d0 con 0x7f83b006b100 2023-10-16T14:12:49.887 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:12:49.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:12:49.888 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T14:12:49.888 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:12:49.888 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:12:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f83b006b100 msgr2=0x7f83b006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:49.890 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f83b006b100 0x7f83b006d5c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83b40015d0 tx=0x7f83b4009040 comp rx=0 tx=0).stop 2023-10-16T14:12:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 msgr2=0x7f83c40b36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 0x7f83c40b36c0 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f83b800d800 tx=0x7f83b800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 shutdown_connections 2023-10-16T14:12:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f83b006b100 0x7f83b006d5c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83c415d150 0x7f83c40c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83c40ad350 0x7f83c40b36c0 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 --2- 172.21.15.60:0/4261596502 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f83c406a2e0 0x7f83c40c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.887+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 >> 172.21.15.60:0/4261596502 conn(0x7f83c40ce9b0 msgr2=0x7f83c40ca290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:49.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.888+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 shutdown_connections 2023-10-16T14:12:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:49.888+0000 7f83c9f01640 1 -- 172.21.15.60:0/4261596502 wait complete. 2023-10-16T14:12:49.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:12:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.140+0000 7f84c3fff640 1 Processor -- start 2023-10-16T14:12:50.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.140+0000 7f84c3fff640 1 -- start start 2023-10-16T14:12:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f84c40ad970 0x7f84c40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c40ccfb0 0x7f84c40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c3fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c415b9d0 con 0x7f84c406c040 2023-10-16T14:12:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c3fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c415bcb0 con 0x7f84c40ad970 2023-10-16T14:12:50.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c415bf90 con 0x7f84c40ccfb0 2023-10-16T14:12:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f84c406c040 0x7f84c406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49624/0 (socket says 172.21.15.60:49624) 2023-10-16T14:12:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.142+0000 7f84c2ffd640 1 -- 172.21.15.60:0/2333664557 learned_addr learned my addr 172.21.15.60:0/2333664557 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:50.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c27fc640 1 --2- 172.21.15.60:0/2333664557 >> v2:172.21.15.144:3300/0 conn(0x7f84c40ad970 0x7f84c40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:50.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c37fe640 1 --2- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c40ccfb0 0x7f84c40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:50.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c1ffb640 1 -- 172.21.15.60:0/2333664557 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619826648 0 0) 0x7f84c415b9d0 con 0x7f84c406c040 2023-10-16T14:12:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c1ffb640 1 -- 172.21.15.60:0/2333664557 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b0003520 con 0x7f84c406c040 2023-10-16T14:12:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c27fc640 1 -- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c40ccfb0 msgr2=0x7f84c40cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c27fc640 1 --2- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c40ccfb0 0x7f84c40cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c27fc640 1 -- 172.21.15.60:0/2333664557 >> v1:172.21.15.60:6789/0 conn(0x7f84c406c040 legacy=0x7f84c406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.143+0000 7f84c27fc640 1 -- 172.21.15.60:0/2333664557 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40cc270 con 0x7f84c40ad970 2023-10-16T14:12:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.145+0000 7f84c27fc640 1 --2- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c40cca70 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f84b40097a0 tx=0x7f84b402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e4b5b25117ebcb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.145+0000 7f84c1ffb640 1 -- 172.21.15.60:0/2333664557 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b402d040 con 0x7f84c40ad970 2023-10-16T14:12:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c1ffb640 1 -- 172.21.15.60:0/2333664557 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b4031810 con 0x7f84c40ad970 2023-10-16T14:12:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c1ffb640 1 -- 172.21.15.60:0/2333664557 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4031ba0 con 0x7f84c40ad970 2023-10-16T14:12:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c3fff640 1 -- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 msgr2=0x7f84c40cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c3fff640 1 --2- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c40cca70 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f84b40097a0 tx=0x7f84b402b8b0 comp rx=0 tx=0).stop 2023-10-16T14:12:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c3fff640 1 -- 172.21.15.60:0/2333664557 shutdown_connections 2023-10-16T14:12:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c3fff640 1 --2- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c40ccfb0 0x7f84c40cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c3fff640 1 --2- 172.21.15.60:0/2333664557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c40cca70 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.146+0000 7f84c3fff640 1 -- 172.21.15.60:0/2333664557 >> 172.21.15.60:0/2333664557 conn(0x7f84c40ce6a0 msgr2=0x7f84c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.147+0000 7f84c3fff640 1 -- 172.21.15.60:0/2333664557 shutdown_connections 2023-10-16T14:12:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.147+0000 7f84c3fff640 1 -- 172.21.15.60:0/2333664557 wait complete. 2023-10-16T14:12:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.148+0000 7f84c3fff640 1 Processor -- start 2023-10-16T14:12:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.148+0000 7f84c3fff640 1 -- start start 2023-10-16T14:12:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.149+0000 7f84c3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c406c040 0x7f84c4152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.149+0000 7f84c3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.149+0000 7f84c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 0x7f84c415de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.149+0000 7f84c3fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40caca0 con 0x7f84c40ccfb0 2023-10-16T14:12:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.149+0000 7f84c3fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40c5910 con 0x7f84c40ad970 2023-10-16T14:12:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.149+0000 7f84c3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40c5bf0 con 0x7f84c406c040 2023-10-16T14:12:50.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 0x7f84c415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 0x7f84c415de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51854/0 (socket says 172.21.15.60:51854) 2023-10-16T14:12:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 -- 172.21.15.60:0/4043669380 learned_addr learned my addr 172.21.15.60:0/4043669380 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c2ffd640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c406c040 0x7f84c4152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 -- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c406c040 msgr2=0x7f84c4152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c406c040 0x7f84c4152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c27fc640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 -- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 msgr2=0x7f84c415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c37fe640 1 -- 172.21.15.60:0/4043669380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b4009400 con 0x7f84c40ccfb0 2023-10-16T14:12:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.150+0000 7f84c27fc640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.151+0000 7f84c37fe640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 0x7f84c415de40 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00efc0 tx=0x7f84bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.151+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc009ba0 con 0x7f84c40ccfb0 2023-10-16T14:12:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.151+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc00f040 con 0x7f84c40ccfb0 2023-10-16T14:12:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.151+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0146c0 con 0x7f84c40ccfb0 2023-10-16T14:12:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.151+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c40c5ed0 con 0x7f84c40ccfb0 2023-10-16T14:12:50.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.151+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c4152b50 con 0x7f84c40ccfb0 2023-10-16T14:12:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.152+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c406b330 con 0x7f84c40ccfb0 2023-10-16T14:12:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.153+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f84bc014820 con 0x7f84c40ccfb0 2023-10-16T14:12:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.154+0000 7f84bb7fe640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f84a406b0b0 0x7f84a406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.154+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f84bc08a780 con 0x7f84c40ccfb0 2023-10-16T14:12:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.155+0000 7f84c2ffd640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f84a406b0b0 0x7f84a406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:50.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.156+0000 7f84c2ffd640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f84a406b0b0 0x7f84a406d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0011e0 tx=0x7f84ac009040 comp rx=0 tx=0).ready entity=mgr.8393 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:50.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.159+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84bc0552f0 con 0x7f84c40ccfb0 2023-10-16T14:12:50.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:50.405+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f84c40c63f0 con 0x7f84c40ccfb0 2023-10-16T14:12:51.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.011+0000 7f84bb7fe640 1 -- 172.21.15.60:0/4043669380 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v146) v1 ==== 92+0+0 (secure 0 0 0) 0x7f84bc058910 con 0x7f84c40ccfb0 2023-10-16T14:12:51.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f84a406b0b0 msgr2=0x7f84a406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f84a406b0b0 0x7f84a406d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0011e0 tx=0x7f84ac009040 comp rx=0 tx=0).stop 2023-10-16T14:12:51.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 msgr2=0x7f84c415de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 0x7f84c415de40 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00efc0 tx=0x7f84bc00c490 comp rx=0 tx=0).stop 2023-10-16T14:12:51.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 shutdown_connections 2023-10-16T14:12:51.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f84a406b0b0 0x7f84a406d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84c40ccfb0 0x7f84c415de40 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84c40ad970 0x7f84c415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 --2- 172.21.15.60:0/4043669380 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84c406c040 0x7f84c4152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.015+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 >> 172.21.15.60:0/4043669380 conn(0x7f84c40ce6a0 msgr2=0x7f84c4159620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:51.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.016+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 shutdown_connections 2023-10-16T14:12:51.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.016+0000 7f84c3fff640 1 -- 172.21.15.60:0/4043669380 wait complete. 2023-10-16T14:12:51.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:12:51.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.254+0000 7fd7a4324640 1 Processor -- start 2023-10-16T14:12:51.257 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.254+0000 7fd7a4324640 1 -- start start 2023-10-16T14:12:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a4324640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 0x7fd79c0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a4324640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c15eff0 con 0x7fd79c065ab0 2023-10-16T14:12:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a4324640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c15ecc0 con 0x7fd79c06faf0 2023-10-16T14:12:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a4324640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0d0930 con 0x7fd79c15af80 2023-10-16T14:12:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a28c1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd79c065ab0 0x7fd79c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49630/0 (socket says 172.21.15.60:49630) 2023-10-16T14:12:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a28c1640 1 -- 172.21.15.60:0/1105714693 learned_addr learned my addr 172.21.15.60:0/1105714693 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.256+0000 7fd7a20c0640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 0x7fd79c0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429456382 0 0) 0x7fd79c15eff0 con 0x7fd79c065ab0 2023-10-16T14:12:51.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003520 con 0x7fd79c065ab0 2023-10-16T14:12:51.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587805493 0 0) 0x7fd788003520 con 0x7fd79c065ab0 2023-10-16T14:12:51.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 msgr2=0x7fd79c0cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:51.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 0x7fd79c0cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 >> v1:172.21.15.144:6789/0 conn(0x7fd79c06faf0 legacy=0x7fd79c15f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.263 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c15b360 con 0x7fd79c065ab0 2023-10-16T14:12:51.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd7940019f0 con 0x7fd79c065ab0 2023-10-16T14:12:51.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 >> v1:172.21.15.60:6789/0 conn(0x7fd79c065ab0 legacy=0x7fd79c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 0x7fd7880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd788003f20 0x7fd7880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.264 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd788004d00 0x7fd788005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0d0930 con 0x7fd788003f20 2023-10-16T14:12:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800bb10 con 0x7fd788004d00 2023-10-16T14:12:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.257+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800bcd0 con 0x7fd79c15af80 2023-10-16T14:12:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a18bf640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd788003f20 0x7fd7880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a20c0640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 0x7fd7880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.265 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a28c1640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd788004d00 0x7fd788005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a18bf640 1 -- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 msgr2=0x7fd7880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:51.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a18bf640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd79c15af80 0x7fd7880039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a18bf640 1 -- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd788004d00 msgr2=0x7fd788005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:51.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a18bf640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd788004d00 0x7fd788005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.266 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.258+0000 7fd7a18bf640 1 -- 172.21.15.60:0/1105714693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788003520 con 0x7fd788003f20 2023-10-16T14:12:51.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.259+0000 7fd7a18bf640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd788003f20 0x7fd7880047c0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fd78c001750 tx=0x7fd7880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e37f12fbcbcc73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:51.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0052d0 con 0x7fd788003f20 2023-10-16T14:12:51.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c02e050 con 0x7fd788003f20 2023-10-16T14:12:51.267 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a10be640 1 -- 172.21.15.60:0/1105714693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0035f0 con 0x7fd788003f20 2023-10-16T14:12:51.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a4324640 1 -- 172.21.15.60:0/1105714693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd788003f20 msgr2=0x7fd7880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a4324640 1 --2- 172.21.15.60:0/1105714693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd788003f20 0x7fd7880047c0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fd78c001750 tx=0x7fd7880057f0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a28c1640 1 -- 172.21.15.60:0/1105714693 reap_dead start 2023-10-16T14:12:51.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a4324640 1 -- 172.21.15.60:0/1105714693 shutdown_connections 2023-10-16T14:12:51.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a4324640 1 -- 172.21.15.60:0/1105714693 >> 172.21.15.60:0/1105714693 conn(0x7fd79c0d0ab0 msgr2=0x7fd79c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:51.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a4324640 1 -- 172.21.15.60:0/1105714693 shutdown_connections 2023-10-16T14:12:51.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.260+0000 7fd7a4324640 1 -- 172.21.15.60:0/1105714693 wait complete. 2023-10-16T14:12:51.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.262+0000 7fd7a4324640 1 Processor -- start 2023-10-16T14:12:51.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.262+0000 7fd7a4324640 1 -- start start 2023-10-16T14:12:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.263+0000 7fd7a4324640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd788003f20 0x7fd79c159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.263+0000 7fd7a4324640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd79c0c8950 0x7fd79c0b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.263+0000 7fd7a4324640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 0x7fd79c1f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.263+0000 7fd7a4324640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0c8530 con 0x7fd79c0b1e30 2023-10-16T14:12:51.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.263+0000 7fd7a4324640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c159ea0 con 0x7fd79c0c8950 2023-10-16T14:12:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.263+0000 7fd7a4324640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0b22b0 con 0x7fd788003f20 2023-10-16T14:12:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 0x7fd79c1f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 0x7fd79c1f0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51874/0 (socket says 172.21.15.60:51874) 2023-10-16T14:12:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 -- 172.21.15.60:0/2896604557 learned_addr learned my addr 172.21.15.60:0/2896604557 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a20c0640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd788003f20 0x7fd79c159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a18bf640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd79c0c8950 0x7fd79c0b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 -- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd788003f20 msgr2=0x7fd79c159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd788003f20 0x7fd79c159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 -- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd79c0c8950 msgr2=0x7fd79c0b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd79c0c8950 0x7fd79c0b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 -- 172.21.15.60:0/2896604557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c1f1b00 con 0x7fd79c0b1e30 2023-10-16T14:12:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a18bf640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd79c0c8950 0x7fd79c0b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.264+0000 7fd7a28c1640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 0x7fd79c1f0d80 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0744f0 tx=0x7fd79400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.265+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79400ec10 con 0x7fd79c0b1e30 2023-10-16T14:12:51.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.265+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79400ed70 con 0x7fd79c0b1e30 2023-10-16T14:12:51.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.265+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79400eee0 con 0x7fd79c0b1e30 2023-10-16T14:12:51.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.265+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c0c7b90 con 0x7fd79c0b1e30 2023-10-16T14:12:51.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.265+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c0c8150 con 0x7fd79c0b1e30 2023-10-16T14:12:51.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.266+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79c0c8670 con 0x7fd79c0b1e30 2023-10-16T14:12:51.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.267+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 146) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fd794013660 con 0x7fd79c0b1e30 2023-10-16T14:12:51.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.268+0000 7fd792ffd640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 0x7fd78006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.268+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fd794089e10 con 0x7fd79c0b1e30 2023-10-16T14:12:51.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.268+0000 7fd7a20c0640 1 -- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 msgr2=0x7fd78006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:51.279 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.268+0000 7fd7a20c0640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 0x7fd78006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:51.280 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.273+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd79c0c8670 con 0x7fd79c0b1e30 2023-10-16T14:12:51.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.469+0000 7fd7a20c0640 1 -- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 msgr2=0x7fd78006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:51.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.469+0000 7fd7a20c0640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 0x7fd78006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:51.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.471+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd79c0c8670 con 0x7fd79c0b1e30 2023-10-16T14:12:51.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.476+0000 7fd792ffd640 1 -- 172.21.15.60:0/2896604557 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd79c0c8670 con 0x7fd79c0b1e30 2023-10-16T14:12:51.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 msgr2=0x7fd78006d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:51.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 0x7fd78006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 msgr2=0x7fd79c1f0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 0x7fd79c1f0d80 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0744f0 tx=0x7fd79400e530 comp rx=0 tx=0).stop 2023-10-16T14:12:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 shutdown_connections 2023-10-16T14:12:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fd78006b130 0x7fd78006d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd79c0b1e30 0x7fd79c1f0d80 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd79c0c8950 0x7fd79c0b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 --2- 172.21.15.60:0/2896604557 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd788003f20 0x7fd79c159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 >> 172.21.15.60:0/2896604557 conn(0x7fd79c0d0ab0 msgr2=0x7fd79c15d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:51.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 shutdown_connections 2023-10-16T14:12:51.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.479+0000 7fd7a4324640 1 -- 172.21.15.60:0/2896604557 wait complete. 2023-10-16T14:12:51.499 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:12:51.543 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:51.540+0000 7fdb60432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:12:51.596 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:51.593+0000 7fdb60432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:12:51.611 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:51.609+0000 7fe9eee44040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:12:51.669 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:51.667+0000 7fe9eee44040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:12:51.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.706+0000 7fe6da890640 1 Processor -- start 2023-10-16T14:12:51.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.707+0000 7fe6da890640 1 -- start start 2023-10-16T14:12:51.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6da890640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6d406a4b0 0x7fe6d40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6da890640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6d40c9ca0 0x7fe6d40ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6da890640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d40cb030 con 0x7fe6d415ad70 2023-10-16T14:12:51.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6da890640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d40cae30 con 0x7fe6d406a4b0 2023-10-16T14:12:51.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6da890640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d40c91c0 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6cbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6d406a4b0 0x7fe6d40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6d3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe6d415ad70 0x7fe6d4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49634/0 (socket says 172.21.15.60:49634) 2023-10-16T14:12:51.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.708+0000 7fe6d3fff640 1 -- 172.21.15.60:0/4076262017 learned_addr learned my addr 172.21.15.60:0/4076262017 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:51.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.709+0000 7fe6d8e2d640 1 --2- 172.21.15.60:0/4076262017 >> v2:172.21.15.144:3300/0 conn(0x7fe6d40c9ca0 0x7fe6d40ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.709+0000 7fe6cbfff640 1 -- 172.21.15.60:0/4076262017 >> v2:172.21.15.144:3300/0 conn(0x7fe6d40c9ca0 msgr2=0x7fe6d40ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:51.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.709+0000 7fe6cbfff640 1 --2- 172.21.15.60:0/4076262017 >> v2:172.21.15.144:3300/0 conn(0x7fe6d40c9ca0 0x7fe6d40ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.709+0000 7fe6cbfff640 1 -- 172.21.15.60:0/4076262017 >> v1:172.21.15.60:6789/0 conn(0x7fe6d415ad70 legacy=0x7fe6d4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.709+0000 7fe6cbfff640 1 -- 172.21.15.60:0/4076262017 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40c94a0 con 0x7fe6d406a4b0 2023-10-16T14:12:51.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.709+0000 7fe6d37fe640 1 -- 172.21.15.60:0/4076262017 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631809257 0 0) 0x7fe6d40cb030 con 0x7fe6d415ad70 2023-10-16T14:12:51.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.710+0000 7fe6cbfff640 1 --2- 172.21.15.60:0/4076262017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d406a4b0 0x7fe6d40ca3b0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fe6d406f270 tx=0x7fe6bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538747c45ed5a894 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:51.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.710+0000 7fe6d37fe640 1 -- 172.21.15.60:0/4076262017 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc031830 con 0x7fe6d406a4b0 2023-10-16T14:12:51.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.710+0000 7fe6d37fe640 1 -- 172.21.15.60:0/4076262017 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc031990 con 0x7fe6d406a4b0 2023-10-16T14:12:51.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.710+0000 7fe6d37fe640 1 -- 172.21.15.60:0/4076262017 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc031d20 con 0x7fe6d406a4b0 2023-10-16T14:12:51.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 -- 172.21.15.60:0/4076262017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d406a4b0 msgr2=0x7fe6d40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 --2- 172.21.15.60:0/4076262017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d406a4b0 0x7fe6d40ca3b0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fe6d406f270 tx=0x7fe6bc02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 -- 172.21.15.60:0/4076262017 shutdown_connections 2023-10-16T14:12:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 --2- 172.21.15.60:0/4076262017 >> v2:172.21.15.144:3300/0 conn(0x7fe6d40c9ca0 0x7fe6d40ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 --2- 172.21.15.60:0/4076262017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d406a4b0 0x7fe6d40ca3b0 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 -- 172.21.15.60:0/4076262017 >> 172.21.15.60:0/4076262017 conn(0x7fe6d40ce9b0 msgr2=0x7fe6d40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:51.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 -- 172.21.15.60:0/4076262017 shutdown_connections 2023-10-16T14:12:51.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.711+0000 7fe6da890640 1 -- 172.21.15.60:0/4076262017 wait complete. 2023-10-16T14:12:51.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.713+0000 7fe6da890640 1 Processor -- start 2023-10-16T14:12:51.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.713+0000 7fe6da890640 1 -- start start 2023-10-16T14:12:51.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6da890640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6d406a4b0 0x7fe6d4151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6da890640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 0x7fe6d40c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6da890640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6d415ad70 0x7fe6d40c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6da890640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40c52e0 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6da890640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40ac890 con 0x7fe6d415ad70 2023-10-16T14:12:51.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6da890640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40ad130 con 0x7fe6d406a4b0 2023-10-16T14:12:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.714+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 0x7fe6d40c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 0x7fe6d40c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51880/0 (socket says 172.21.15.60:51880) 2023-10-16T14:12:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 -- 172.21.15.60:0/3043894345 learned_addr learned my addr 172.21.15.60:0/3043894345 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6d3fff640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6d406a4b0 0x7fe6d4151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6d8e2d640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6d415ad70 0x7fe6d40c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 -- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6d406a4b0 msgr2=0x7fe6d4151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6d406a4b0 0x7fe6d4151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 -- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6d415ad70 msgr2=0x7fe6d40c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6d415ad70 0x7fe6d40c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 -- 172.21.15.60:0/3043894345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc009400 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.715+0000 7fe6cbfff640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 0x7fe6d40c6ac0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc02bee0 tx=0x7fe6bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.716+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc03ba70 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.716+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc03bbf0 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.716+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0433f0 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.716+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d40c3ae0 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.716+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d40acdc0 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.717+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d404e160 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.718+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 146) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fe6bc043550 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.719+0000 7fe6d17fa640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 0x7fe6b406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.719+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7fe6bc0ac410 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.719+0000 7fe6d3fff640 1 -- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 msgr2=0x7fe6b406d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:51.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.719+0000 7fe6d3fff640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 0x7fe6b406d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:51.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.723+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6bc077120 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.915+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe6d406fb20 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.920+0000 7fe6d3fff640 1 -- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 msgr2=0x7fe6b406d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:51.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.920+0000 7fe6d3fff640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 0x7fe6b406d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:51.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.922+0000 7fe6d17fa640 1 -- 172.21.15.60:0/3043894345 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe6bc07a740 con 0x7fe6d40c9ca0 2023-10-16T14:12:51.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 msgr2=0x7fe6b406d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:51.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 0x7fe6b406d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 msgr2=0x7fe6d40c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:51.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 0x7fe6d40c6ac0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc02bee0 tx=0x7fe6bc0046c0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 shutdown_connections 2023-10-16T14:12:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7fe6b406b130 0x7fe6b406d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6d415ad70 0x7fe6d40c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d40c9ca0 0x7fe6d40c6ac0 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 --2- 172.21.15.60:0/3043894345 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6d406a4b0 0x7fe6d4151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:51.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 >> 172.21.15.60:0/3043894345 conn(0x7fe6d40ce9b0 msgr2=0x7fe6d40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:51.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 shutdown_connections 2023-10-16T14:12:51.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:51.924+0000 7fe6da890640 1 -- 172.21.15.60:0/3043894345 wait complete. 2023-10-16T14:12:51.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:51.945 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T14:12:52.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T14:12:52.047 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.044+0000 7fdb60432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:12:52.098 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.095+0000 7fdb60432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:12:52.148 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.145+0000 7fdb60432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:12:52.154 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.151+0000 7fe9eee44040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:12:52.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.203+0000 7f61f3fff640 1 Processor -- start 2023-10-16T14:12:52.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.203+0000 7f61f3fff640 1 -- start start 2023-10-16T14:12:52.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.204+0000 7f61f3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61f415b320 0x7f61f40cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:52.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.204+0000 7f61f3fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f415b9d0 con 0x7f61f406c040 2023-10-16T14:12:52.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.204+0000 7f61f3fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f415bcb0 con 0x7f61f40ad970 2023-10-16T14:12:52.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.204+0000 7f61f3fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f415b7d0 con 0x7f61f415b320 2023-10-16T14:12:52.208 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.204+0000 7fdb60432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:12:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f2ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61f406c040 0x7f61f406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49646/0 (socket says 172.21.15.60:49646) 2023-10-16T14:12:52.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f2ffd640 1 -- 172.21.15.60:0/510100261 learned_addr learned my addr 172.21.15.60:0/510100261 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f37fe640 1 --2- 172.21.15.60:0/510100261 >> v2:172.21.15.60:3300/0 conn(0x7f61f415b320 0x7f61f40cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f37fe640 1 -- 172.21.15.60:0/510100261 >> v1:172.21.15.144:6789/0 conn(0x7f61f40ad970 legacy=0x7f61f40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f37fe640 1 -- 172.21.15.60:0/510100261 >> v1:172.21.15.60:6789/0 conn(0x7f61f406c040 legacy=0x7f61f406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:52.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f37fe640 1 -- 172.21.15.60:0/510100261 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f415bf90 con 0x7f61f415b320 2023-10-16T14:12:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.205+0000 7f61f1ffb640 1 -- 172.21.15.60:0/510100261 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297165137 0 0) 0x7f61f415b9d0 con 0x7f61f406c040 2023-10-16T14:12:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.207+0000 7f61f37fe640 1 --2- 172.21.15.60:0/510100261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f415b320 0x7f61f40cd180 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0097d0 tx=0x7f61ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e674b1aa54cf4d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:52.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.207+0000 7f61f1ffb640 1 -- 172.21.15.60:0/510100261 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec031830 con 0x7f61f415b320 2023-10-16T14:12:52.211 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.207+0000 7fe9eee44040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:12:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.207+0000 7f61f1ffb640 1 -- 172.21.15.60:0/510100261 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec031990 con 0x7f61f415b320 2023-10-16T14:12:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.207+0000 7f61f1ffb640 1 -- 172.21.15.60:0/510100261 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec031d20 con 0x7f61f415b320 2023-10-16T14:12:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.207+0000 7f61f3fff640 1 -- 172.21.15.60:0/510100261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f415b320 msgr2=0x7f61f40cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.207+0000 7f61f3fff640 1 --2- 172.21.15.60:0/510100261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f415b320 0x7f61f40cd180 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0097d0 tx=0x7f61ec02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.208+0000 7f61f3fff640 1 -- 172.21.15.60:0/510100261 shutdown_connections 2023-10-16T14:12:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.208+0000 7f61f3fff640 1 --2- 172.21.15.60:0/510100261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f415b320 0x7f61f40cd180 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:52.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.208+0000 7f61f3fff640 1 -- 172.21.15.60:0/510100261 >> 172.21.15.60:0/510100261 conn(0x7f61f40ce6a0 msgr2=0x7f61f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.208+0000 7f61f3fff640 1 -- 172.21.15.60:0/510100261 shutdown_connections 2023-10-16T14:12:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.208+0000 7f61f3fff640 1 -- 172.21.15.60:0/510100261 wait complete. 2023-10-16T14:12:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.209+0000 7f61f3fff640 1 Processor -- start 2023-10-16T14:12:52.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.209+0000 7f61f3fff640 1 -- start start 2023-10-16T14:12:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.210+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61f406c040 0x7f61f40b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.210+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f40ad970 0x7f61f40acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:52.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.210+0000 7f61f3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f415b320 0x7f61f41587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.210+0000 7f61f3fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f40b21c0 con 0x7f61f40ad970 2023-10-16T14:12:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.210+0000 7f61f3fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f40c4f70 con 0x7f61f415b320 2023-10-16T14:12:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.210+0000 7f61f3fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f41f25e0 con 0x7f61f406c040 2023-10-16T14:12:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f40ad970 0x7f61f40acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:52.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f40ad970 0x7f61f40acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51892/0 (socket says 172.21.15.60:51892) 2023-10-16T14:12:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 -- 172.21.15.60:0/2714908195 learned_addr learned my addr 172.21.15.60:0/2714908195 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f37fe640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f415b320 0x7f61f41587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61f406c040 msgr2=0x7f61f40b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f61f406c040 0x7f61f40b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:52.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f415b320 msgr2=0x7f61f41587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f415b320 0x7f61f41587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 -- 172.21.15.60:0/2714908195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec009400 con 0x7f61f40ad970 2023-10-16T14:12:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f37fe640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f415b320 0x7f61f41587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.211+0000 7f61f27fc640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f40ad970 0x7f61f40acb00 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c430 tx=0x7f61dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:52.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.212+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc011790 con 0x7f61f40ad970 2023-10-16T14:12:52.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.212+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc0118f0 con 0x7f61f40ad970 2023-10-16T14:12:52.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.212+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc011ca0 con 0x7f61f40ad970 2023-10-16T14:12:52.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.212+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f41f2850 con 0x7f61f40ad970 2023-10-16T14:12:52.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.212+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f41f2e10 con 0x7f61f40ad970 2023-10-16T14:12:52.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.213+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f406b330 con 0x7f61f40ad970 2023-10-16T14:12:52.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.214+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 146) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f61dc01c460 con 0x7f61f40ad970 2023-10-16T14:12:52.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.215+0000 7f61ebfff640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 0x7f61d406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:52.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.215+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f61dc089a00 con 0x7f61f40ad970 2023-10-16T14:12:52.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.215+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 msgr2=0x7f61d406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:52.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.215+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 0x7f61d406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:12:52.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.220+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61dc054750 con 0x7f61f40ad970 2023-10-16T14:12:52.257 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.254+0000 7fdb60432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:12:52.293 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.290+0000 7fe9eee44040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:12:52.365 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.362+0000 7fdb60432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:12:52.401 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.398+0000 7fe9eee44040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:12:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.416+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 msgr2=0x7f61d406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.416+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 0x7f61d406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:12:52.427 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.424+0000 7fdb60432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:52.469 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.466+0000 7fe9eee44040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:12:52.476 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.473+0000 7fdb60432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:12:52.533 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.530+0000 7fdb60432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:12:52.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.550+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 --> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f61f40cc550 con 0x7f61d406b100 2023-10-16T14:12:52.582 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.579+0000 7fdb60432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:12:52.586 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.583+0000 7fe9eee44040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:12:52.651 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.648+0000 7fe9eee44040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:52.704 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.701+0000 7fe9eee44040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:12:52.763 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.760+0000 7fe9eee44040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:12:52.816 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:52.813+0000 7fe9eee44040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:12:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.817+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 msgr2=0x7f61d406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:52.817+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 0x7f61d406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:12:52.960 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:52.957+0000 7fdb60432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:12:53.118 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.115+0000 7fdb60432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:12:53.216 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.214+0000 7fe9eee44040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:12:53.224 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.222+0000 7fdb60432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:12:53.274 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.272+0000 7fdb60432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:12:53.381 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.378+0000 7fdb60432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:53.384 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.381+0000 7fe9eee44040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:12:53.487 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.484+0000 7fdb60432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:12:53.498 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.495+0000 7fe9eee44040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:12:53.551 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.549+0000 7fe9eee44040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:12:53.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:53.618+0000 7f61f2ffd640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 msgr2=0x7f61d406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/948422863 2023-10-16T14:12:53.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:53.618+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 0x7f61d406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:12:53.643 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.640+0000 7fdb60432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:53.664 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.662+0000 7fe9eee44040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:53.692 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.689+0000 7fdb60432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:53.776 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.773+0000 7fe9eee44040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:12:53.811 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:53.807+0000 7fdb60432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:12:53.940 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.937+0000 7fe9eee44040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:12:53.992 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:53.989+0000 7fe9eee44040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:12:54.117 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:54.114+0000 7fe9eee44040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:12:54.647 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:54.644+0000 7fdb60432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:12:54.940 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:54.937+0000 7fdb60432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:12:54.997 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:54.993+0000 7fdb60432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:12:55.010 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:55.007+0000 7fe9eee44040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:12:55.109 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.106+0000 7fdb60432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:12:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:55.128+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 147) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f61dc054b20 con 0x7f61f40ad970 2023-10-16T14:12:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:55.129+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 msgr2=0x7f61d406d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:12:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:55.129+0000 7f61ebfff640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/948422863,v1:172.21.15.144:6833/948422863] conn(0x7f61d406b100 0x7f61d406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:55.341 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:55.338+0000 7fe9eee44040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:12:55.345 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadc778640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.345 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadc778640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.345 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadc778640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.345 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadc778640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadc778640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadf7be640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadf7be640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadf7be640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadf7be640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.346 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:12:55.342+0000 7fdadf7be640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:12:55.404 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:55.401+0000 7fe9eee44040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:12:55.522 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:12:55.519+0000 7fe9eee44040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:12:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.140+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f61dc00e280 con 0x7f61f40ad970 2023-10-16T14:12:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.141+0000 7f61ebfff640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f61d40356c0 0x7f61d4035aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.141+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 --> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d4035fe0 con 0x7f61d40356c0 2023-10-16T14:12:56.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.153+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f61d40356c0 0x7f61d4035aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.162+0000 7f61f2ffd640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f61d40356c0 0x7f61d4035aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f61e4007fd0 tx=0x7f61e4007da0 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.331+0000 7f61ebfff640 1 -- 172.21.15.60:0/2714908195 <== mgr.9958 v2:172.21.15.144:6832/2906320450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f61d4035fe0 con 0x7f61d40356c0 2023-10-16T14:12:56.491 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path snap_test_dir1 2023-10-16T14:12:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.334+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f61d40356c0 msgr2=0x7f61d4035aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.334+0000 7f61f3fff640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f61d40356c0 0x7f61d4035aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f61e4007fd0 tx=0x7f61e4007da0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.334+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f40ad970 msgr2=0x7f61f40acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.334+0000 7f61f3fff640 1 --2- 172.21.15.60:0/2714908195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f40ad970 0x7f61f40acb00 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c430 tx=0x7f61dc00c8f0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.334+0000 7f61f37fe640 1 -- 172.21.15.60:0/2714908195 reap_dead start 2023-10-16T14:12:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.334+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 shutdown_connections 2023-10-16T14:12:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.335+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 >> 172.21.15.60:0/2714908195 conn(0x7f61f40ce6a0 msgr2=0x7f61f40c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.335+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 shutdown_connections 2023-10-16T14:12:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.335+0000 7f61f3fff640 1 -- 172.21.15.60:0/2714908195 wait complete. 2023-10-16T14:12:56.495 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T14:12:56.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T14:12:56.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.732+0000 7fc71ab1d640 1 Processor -- start 2023-10-16T14:12:56.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.732+0000 7fc71ab1d640 1 -- start start 2023-10-16T14:12:56.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc71ab1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140b0360 0x7fc7140b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc71ab1d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7140ca510 0x7fc7140ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc71ab1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140b0d00 con 0x7fc71415ac10 2023-10-16T14:12:56.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc71ab1d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71415f800 con 0x7fc7140ca510 2023-10-16T14:12:56.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc71ab1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71406ce40 con 0x7fc7140b0360 2023-10-16T14:12:56.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc7190ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7140ca510 0x7fc7140ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc7190ba640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7140ca510 0x7fc7140ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51906/0 (socket says 172.21.15.60:51906) 2023-10-16T14:12:56.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc7190ba640 1 -- 172.21.15.60:0/982086517 learned_addr learned my addr 172.21.15.60:0/982086517 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:56.739 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.734+0000 7fc713fff640 1 --2- 172.21.15.60:0/982086517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140b0360 0x7fc7140b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.735+0000 7fc7190ba640 1 -- 172.21.15.60:0/982086517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140b0360 msgr2=0x7fc7140b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.735+0000 7fc7190ba640 1 --2- 172.21.15.60:0/982086517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140b0360 0x7fc7140b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.735+0000 7fc7190ba640 1 -- 172.21.15.60:0/982086517 >> v1:172.21.15.144:6789/0 conn(0x7fc71415ac10 legacy=0x7fc71406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.735+0000 7fc7190ba640 1 -- 172.21.15.60:0/982086517 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71415b310 con 0x7fc7140ca510 2023-10-16T14:12:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.736+0000 7fc7190ba640 1 --2- 172.21.15.60:0/982086517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7140ca510 0x7fc7140ca940 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009e50 tx=0x7fc70c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2733418f0f5dcbd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.736+0000 7fc7137fe640 1 -- 172.21.15.60:0/982086517 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c02c9e0 con 0x7fc7140ca510 2023-10-16T14:12:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.736+0000 7fc7137fe640 1 -- 172.21.15.60:0/982086517 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c02cb40 con 0x7fc7140ca510 2023-10-16T14:12:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.736+0000 7fc7137fe640 1 -- 172.21.15.60:0/982086517 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0343f0 con 0x7fc7140ca510 2023-10-16T14:12:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 -- 172.21.15.60:0/982086517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7140ca510 msgr2=0x7fc7140ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/982086517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7140ca510 0x7fc7140ca940 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009e50 tx=0x7fc70c02be40 comp rx=0 tx=0).stop 2023-10-16T14:12:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 -- 172.21.15.60:0/982086517 shutdown_connections 2023-10-16T14:12:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/982086517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7140ca510 0x7fc7140ca940 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/982086517 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140b0360 0x7fc7140b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 -- 172.21.15.60:0/982086517 >> 172.21.15.60:0/982086517 conn(0x7fc7140cb360 msgr2=0x7fc7140cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 -- 172.21.15.60:0/982086517 shutdown_connections 2023-10-16T14:12:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.737+0000 7fc71ab1d640 1 -- 172.21.15.60:0/982086517 wait complete. 2023-10-16T14:12:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.738+0000 7fc71ab1d640 1 Processor -- start 2023-10-16T14:12:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.739+0000 7fc71ab1d640 1 -- start start 2023-10-16T14:12:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc71ab1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7140b0360 0x7fc7140acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc71ab1d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140ca510 0x7fc7141f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc71ab1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 0x7fc7141f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc71ab1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140cd680 con 0x7fc71415ac10 2023-10-16T14:12:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc71ab1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140cd960 con 0x7fc7140b0360 2023-10-16T14:12:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc71ab1d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140cdc40 con 0x7fc7140ca510 2023-10-16T14:12:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 0x7fc7141f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 0x7fc7141f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51908/0 (socket says 172.21.15.60:51908) 2023-10-16T14:12:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 -- 172.21.15.60:0/3366290247 learned_addr learned my addr 172.21.15.60:0/3366290247 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc713fff640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140ca510 0x7fc7141f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 -- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140ca510 msgr2=0x7fc7141f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140ca510 0x7fc7141f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 -- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7140b0360 msgr2=0x7fc7140acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7188b9640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7140b0360 0x7fc7140acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7140b0360 0x7fc7140acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.740+0000 7fc7190ba640 1 -- 172.21.15.60:0/3366290247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c009b30 con 0x7fc71415ac10 2023-10-16T14:12:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.741+0000 7fc7190ba640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 0x7fc7141f65d0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7fc70c036040 tx=0x7fc70c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.741+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c009510 con 0x7fc71415ac10 2023-10-16T14:12:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.741+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c03e450 con 0x7fc71415ac10 2023-10-16T14:12:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.741+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c03e7e0 con 0x7fc71415ac10 2023-10-16T14:12:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.741+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71415c410 con 0x7fc71415ac10 2023-10-16T14:12:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.741+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7140ad040 con 0x7fc71415ac10 2023-10-16T14:12:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.742+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71415c270 con 0x7fc71415ac10 2023-10-16T14:12:56.750 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.743+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc70c03e940 con 0x7fc71415ac10 2023-10-16T14:12:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.744+0000 7fc7117fa640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fc6f806ad10 0x7fc6f806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.744+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7fc70c0ac320 con 0x7fc71415ac10 2023-10-16T14:12:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.745+0000 7fc7188b9640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fc6f806ad10 0x7fc6f806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.746+0000 7fc7188b9640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fc6f806ad10 0x7fc6f806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc708003d90 tx=0x7fc708009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.749+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc71415c270 con 0x7fc71415ac10 2023-10-16T14:12:56.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:56.976+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 --> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc71415b5b0 con 0x7fc6f806ad10 2023-10-16T14:12:57.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.143+0000 7fc7117fa640 1 -- 172.21.15.60:0/3366290247 <== mgr.9958 v2:172.21.15.144:6832/2906320450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fc708003980 con 0x7fc6f806ad10 2023-10-16T14:12:57.146 INFO:teuthology.orchestra.run.smithi060.stdout:Retention added to path snap_test_dir1 2023-10-16T14:12:57.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.146+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fc6f806ad10 msgr2=0x7fc6f806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.146+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fc6f806ad10 0x7fc6f806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc708003d90 tx=0x7fc708009040 comp rx=0 tx=0).stop 2023-10-16T14:12:57.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.146+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 msgr2=0x7fc7141f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.146+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 0x7fc7141f65d0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7fc70c036040 tx=0x7fc70c004a40 comp rx=0 tx=0).stop 2023-10-16T14:12:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 shutdown_connections 2023-10-16T14:12:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fc6f806ad10 0x7fc6f806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc7140ca510 0x7fc7141f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71415ac10 0x7fc7141f65d0 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 --2- 172.21.15.60:0/3366290247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7140b0360 0x7fc7140acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 >> 172.21.15.60:0/3366290247 conn(0x7fc7140cb360 msgr2=0x7fc7140c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:57.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 shutdown_connections 2023-10-16T14:12:57.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.147+0000 7fc71ab1d640 1 -- 172.21.15.60:0/3366290247 wait complete. 2023-10-16T14:12:57.173 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T14:12:57.173 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-14_13_00 in ~3s... 2023-10-16T14:12:57.173 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T14:12:57.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T14:12:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.403+0000 7fba8e675640 1 Processor -- start 2023-10-16T14:12:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.403+0000 7fba8e675640 1 -- start start 2023-10-16T14:12:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8e675640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba880ad970 0x7fba880cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8e675640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8e675640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8815b9d0 con 0x7fba8806c040 2023-10-16T14:12:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8e675640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8815bcb0 con 0x7fba880ad970 2023-10-16T14:12:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8e675640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8815bf90 con 0x7fba880ccfb0 2023-10-16T14:12:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8ce72640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba880ad970 0x7fba880cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8d673640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fba8806c040 0x7fba8806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49658/0 (socket says 172.21.15.60:49658) 2023-10-16T14:12:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8d673640 1 -- 172.21.15.60:0/703196074 learned_addr learned my addr 172.21.15.60:0/703196074 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:57.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.405+0000 7fba8de74640 1 --2- 172.21.15.60:0/703196074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.406+0000 7fba8ce72640 1 -- 172.21.15.60:0/703196074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 msgr2=0x7fba880cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.406+0000 7fba8ce72640 1 --2- 172.21.15.60:0/703196074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.406+0000 7fba8ce72640 1 -- 172.21.15.60:0/703196074 >> v1:172.21.15.60:6789/0 conn(0x7fba8806c040 legacy=0x7fba8806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.406+0000 7fba8ce72640 1 -- 172.21.15.60:0/703196074 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880cc270 con 0x7fba880ad970 2023-10-16T14:12:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.406+0000 7fba7ffff640 1 -- 172.21.15.60:0/703196074 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143855605 0 0) 0x7fba8815b9d0 con 0x7fba8806c040 2023-10-16T14:12:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.407+0000 7fba8ce72640 1 --2- 172.21.15.60:0/703196074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880ad970 0x7fba880cca70 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7fba780097a0 tx=0x7fba7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f90ec6368dcf9f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.407+0000 7fba7ffff640 1 -- 172.21.15.60:0/703196074 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78031830 con 0x7fba880ad970 2023-10-16T14:12:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.407+0000 7fba7ffff640 1 -- 172.21.15.60:0/703196074 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba78031990 con 0x7fba880ad970 2023-10-16T14:12:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.407+0000 7fba7ffff640 1 -- 172.21.15.60:0/703196074 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78031d20 con 0x7fba880ad970 2023-10-16T14:12:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 -- 172.21.15.60:0/703196074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880ad970 msgr2=0x7fba880cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 --2- 172.21.15.60:0/703196074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880ad970 0x7fba880cca70 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7fba780097a0 tx=0x7fba7802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 -- 172.21.15.60:0/703196074 shutdown_connections 2023-10-16T14:12:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 --2- 172.21.15.60:0/703196074 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 --2- 172.21.15.60:0/703196074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba880ad970 0x7fba880cca70 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 -- 172.21.15.60:0/703196074 >> 172.21.15.60:0/703196074 conn(0x7fba880ce6a0 msgr2=0x7fba880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 -- 172.21.15.60:0/703196074 shutdown_connections 2023-10-16T14:12:57.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.408+0000 7fba8e675640 1 -- 172.21.15.60:0/703196074 wait complete. 2023-10-16T14:12:57.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.410+0000 7fba8e675640 1 Processor -- start 2023-10-16T14:12:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.410+0000 7fba8e675640 1 -- start start 2023-10-16T14:12:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8e675640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 0x7fba880c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8e675640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba880ad970 0x7fba880c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8e675640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8e675640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba880cdfc0 con 0x7fba8806c040 2023-10-16T14:12:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8e675640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba880cda00 con 0x7fba880ad970 2023-10-16T14:12:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8e675640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba880cdce0 con 0x7fba880ccfb0 2023-10-16T14:12:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8d673640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 0x7fba880c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8d673640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 0x7fba880c2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51918/0 (socket says 172.21.15.60:51918) 2023-10-16T14:12:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.411+0000 7fba8d673640 1 -- 172.21.15.60:0/2756788191 learned_addr learned my addr 172.21.15.60:0/2756788191 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8d673640 1 -- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 msgr2=0x7fba880c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8d673640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8d673640 1 -- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba880ad970 msgr2=0x7fba880c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8d673640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba880ad970 0x7fba880c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8d673640 1 -- 172.21.15.60:0/2756788191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba78009400 con 0x7fba8806c040 2023-10-16T14:12:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8d673640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 0x7fba880c2420 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fba8400c4d0 tx=0x7fba8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba840074e0 con 0x7fba8806c040 2023-10-16T14:12:57.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba84007640 con 0x7fba8806c040 2023-10-16T14:12:57.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84004a20 con 0x7fba8806c040 2023-10-16T14:12:57.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.412+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba880afa10 con 0x7fba8806c040 2023-10-16T14:12:57.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.413+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba880affd0 con 0x7fba8806c040 2023-10-16T14:12:57.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.414+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8806b330 con 0x7fba8806c040 2023-10-16T14:12:57.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.415+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fba84004b80 con 0x7fba8806c040 2023-10-16T14:12:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.416+0000 7fba7dffb640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fba6c06ad40 0x7fba6c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.416+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7fba840885c0 con 0x7fba8806c040 2023-10-16T14:12:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.416+0000 7fba8ce72640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fba6c06ad40 0x7fba6c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.418+0000 7fba8ce72640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fba6c06ad40 0x7fba6c06d200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fba78036000 tx=0x7fba78030000 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:57.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.420+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba840537d0 con 0x7fba8806c040 2023-10-16T14:12:57.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.636+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 --> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba880cc550 con 0x7fba6c06ad40 2023-10-16T14:12:57.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.653+0000 7fba7dffb640 1 -- 172.21.15.60:0/2756788191 <== mgr.9958 v2:172.21.15.144:6832/2906320450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fba78031800 con 0x7fba6c06ad40 2023-10-16T14:12:57.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:12:57.658 INFO:teuthology.orchestra.run.smithi060.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T14:12:57.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fba6c06ad40 msgr2=0x7fba6c06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fba6c06ad40 0x7fba6c06d200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fba78036000 tx=0x7fba78030000 comp rx=0 tx=0).stop 2023-10-16T14:12:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 msgr2=0x7fba880c2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 0x7fba880c2420 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fba8400c4d0 tx=0x7fba8400c990 comp rx=0 tx=0).stop 2023-10-16T14:12:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 shutdown_connections 2023-10-16T14:12:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7fba6c06ad40 0x7fba6c06d200 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fba880ccfb0 0x7fba880c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fba880ad970 0x7fba880c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 --2- 172.21.15.60:0/2756788191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba8806c040 0x7fba880c2420 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 >> 172.21.15.60:0/2756788191 conn(0x7fba880ce6a0 msgr2=0x7fba880c89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:57.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 shutdown_connections 2023-10-16T14:12:57.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.656+0000 7fba8e675640 1 -- 172.21.15.60:0/2756788191 wait complete. 2023-10-16T14:12:57.679 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T14:12:57.679 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T14:12:57.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T14:12:57.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.912+0000 7f3046ec9640 1 Processor -- start 2023-10-16T14:12:57.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.912+0000 7f3046ec9640 1 -- start start 2023-10-16T14:12:57.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.914+0000 7f3046ec9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3040159e90 0x7f30400ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.914+0000 7f3046ec9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f304006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.914+0000 7f3046ec9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3040070360 con 0x7f3040159ab0 2023-10-16T14:12:57.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.914+0000 7f3046ec9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304006cb10 con 0x7f3040159e90 2023-10-16T14:12:57.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.914+0000 7f3046ec9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304015d890 con 0x7f304006c140 2023-10-16T14:12:57.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.915+0000 7f3037fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3040159e90 0x7f30400ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.915+0000 7f3045466640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f304006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.915+0000 7f3044c65640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3040159ab0 0x7f304006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:42128/0 (socket says 172.21.15.60:42128) 2023-10-16T14:12:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.915+0000 7f3044c65640 1 -- 172.21.15.60:0/3104985271 learned_addr learned my addr 172.21.15.60:0/3104985271 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.916+0000 7f3045466640 1 -- 172.21.15.60:0/3104985271 >> v2:172.21.15.144:3300/0 conn(0x7f3040159e90 msgr2=0x7f30400ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.916+0000 7f3045466640 1 --2- 172.21.15.60:0/3104985271 >> v2:172.21.15.144:3300/0 conn(0x7f3040159e90 0x7f30400ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.916+0000 7f3045466640 1 -- 172.21.15.60:0/3104985271 >> v1:172.21.15.144:6789/0 conn(0x7f3040159ab0 legacy=0x7f304006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.916+0000 7f3045466640 1 -- 172.21.15.60:0/3104985271 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400cd1f0 con 0x7f304006c140 2023-10-16T14:12:57.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.916+0000 7f3037fff640 1 --2- 172.21.15.60:0/3104985271 >> v2:172.21.15.144:3300/0 conn(0x7f3040159e90 0x7f30400ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:12:57.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.917+0000 7f3045466640 1 --2- 172.21.15.60:0/3104985271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f304006c5d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f30380097a0 tx=0x7f303802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da1f51d51404942 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:57.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f30377fe640 1 -- 172.21.15.60:0/3104985271 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303802f050 con 0x7f304006c140 2023-10-16T14:12:57.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f30377fe640 1 -- 172.21.15.60:0/3104985271 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038031700 con 0x7f304006c140 2023-10-16T14:12:57.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f30377fe640 1 -- 172.21.15.60:0/3104985271 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038031a90 con 0x7f304006c140 2023-10-16T14:12:57.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f3046ec9640 1 -- 172.21.15.60:0/3104985271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 msgr2=0x7f304006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f3046ec9640 1 --2- 172.21.15.60:0/3104985271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f304006c5d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f30380097a0 tx=0x7f303802b7a0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f3046ec9640 1 -- 172.21.15.60:0/3104985271 shutdown_connections 2023-10-16T14:12:57.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f3046ec9640 1 --2- 172.21.15.60:0/3104985271 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f304006c5d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f3046ec9640 1 --2- 172.21.15.60:0/3104985271 >> v2:172.21.15.144:3300/0 conn(0x7f3040159e90 0x7f30400ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.918+0000 7f3046ec9640 1 -- 172.21.15.60:0/3104985271 >> 172.21.15.60:0/3104985271 conn(0x7f30400ce9b0 msgr2=0x7f30400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:57.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.919+0000 7f3046ec9640 1 -- 172.21.15.60:0/3104985271 shutdown_connections 2023-10-16T14:12:57.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.919+0000 7f3046ec9640 1 -- 172.21.15.60:0/3104985271 wait complete. 2023-10-16T14:12:57.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.921+0000 7f3046ec9640 1 Processor -- start 2023-10-16T14:12:57.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.921+0000 7f3046ec9640 1 -- start start 2023-10-16T14:12:57.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3046ec9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f30401f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3046ec9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 0x7f30400acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3046ec9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3040159e90 0x7f30400ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3046ec9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400c88e0 con 0x7f3040159ab0 2023-10-16T14:12:57.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3046ec9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30401f2900 con 0x7f3040159e90 2023-10-16T14:12:57.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3046ec9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400c7c60 con 0x7f304006c140 2023-10-16T14:12:57.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.922+0000 7f3037fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 0x7f30400acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 0x7f30400acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51926/0 (socket says 172.21.15.60:51926) 2023-10-16T14:12:57.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 -- 172.21.15.60:0/1178398032 learned_addr learned my addr 172.21.15.60:0/1178398032 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:57.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3045466640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3040159e90 0x7f30400ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3044c65640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f30401f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 -- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 msgr2=0x7f30401f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f30401f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 -- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3040159e90 msgr2=0x7f30400ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:57.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3040159e90 0x7f30400ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:57.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 -- 172.21.15.60:0/1178398032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030009400 con 0x7f3040159ab0 2023-10-16T14:12:57.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.923+0000 7f3037fff640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 0x7f30400acc60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f303000ebe0 tx=0x7f303000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:57.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.924+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030009760 con 0x7f3040159ab0 2023-10-16T14:12:57.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.924+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303000cc30 con 0x7f3040159ab0 2023-10-16T14:12:57.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.924+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3038009400 con 0x7f3040159ab0 2023-10-16T14:12:57.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.924+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303000eee0 con 0x7f3040159ab0 2023-10-16T14:12:57.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.924+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30400c8220 con 0x7f3040159ab0 2023-10-16T14:12:57.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.925+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304004e160 con 0x7f3040159ab0 2023-10-16T14:12:57.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.926+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f3030013470 con 0x7f3040159ab0 2023-10-16T14:12:57.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.927+0000 7f30357fa640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f302c06afe0 0x7f302c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:57.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.927+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f3030089c90 con 0x7f3040159ab0 2023-10-16T14:12:57.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.928+0000 7f3044c65640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f302c06afe0 0x7f302c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:57.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.929+0000 7f3044c65640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f302c06afe0 0x7f302c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f303c005c00 tx=0x7f303c009040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:57.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:57.932+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3030054ae0 con 0x7f3040159ab0 2023-10-16T14:12:58.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.152+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 --> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f30400cd7b0 con 0x7f302c06afe0 2023-10-16T14:12:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.339+0000 7f30357fa640 1 -- 172.21.15.60:0/1178398032 <== mgr.9958 v2:172.21.15.144:6832/2906320450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f303c00ec40 con 0x7f302c06afe0 2023-10-16T14:12:58.342 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path snap_test_dir1 2023-10-16T14:12:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f302c06afe0 msgr2=0x7f302c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f302c06afe0 0x7f302c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f303c005c00 tx=0x7f303c009040 comp rx=0 tx=0).stop 2023-10-16T14:12:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 msgr2=0x7f30400acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 0x7f30400acc60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f303000ebe0 tx=0x7f303000c520 comp rx=0 tx=0).stop 2023-10-16T14:12:58.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 shutdown_connections 2023-10-16T14:12:58.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f302c06afe0 0x7f302c06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3040159e90 0x7f30400ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3040159ab0 0x7f30400acc60 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 --2- 172.21.15.60:0/1178398032 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f304006c140 0x7f30401f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 >> 172.21.15.60:0/1178398032 conn(0x7f30400ce9b0 msgr2=0x7f30400ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 shutdown_connections 2023-10-16T14:12:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.343+0000 7f3046ec9640 1 -- 172.21.15.60:0/1178398032 wait complete. 2023-10-16T14:12:58.369 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T14:12:58.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:58.370 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T14:12:58.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:12:58.430 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T14:12:58.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:12:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.717+0000 7f40992ff640 1 Processor -- start 2023-10-16T14:12:58.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.717+0000 7f40992ff640 1 -- start start 2023-10-16T14:12:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.718+0000 7f40992ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f409406c040 0x7f409406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f40992ff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40940ad970 0x7f409415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f40992ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409406c960 con 0x7f409415ebc0 2023-10-16T14:12:58.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f40992ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409415b9d0 con 0x7f409406c040 2023-10-16T14:12:58.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f40992ff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409415bcb0 con 0x7f40940ad970 2023-10-16T14:12:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f4093fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f409406c040 0x7f409406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f4098afe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f409415ebc0 0x7f40940cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:49670/0 (socket says 172.21.15.60:49670) 2023-10-16T14:12:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f4098afe640 1 -- 172.21.15.60:0/2406704288 learned_addr learned my addr 172.21.15.60:0/2406704288 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:12:58.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f408bfff640 1 --2- 172.21.15.60:0/2406704288 >> v2:172.21.15.144:3300/0 conn(0x7f40940ad970 0x7f409415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.719+0000 7f4093fff640 1 -- 172.21.15.60:0/2406704288 >> v2:172.21.15.144:3300/0 conn(0x7f40940ad970 msgr2=0x7f409415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:12:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.720+0000 7f4093fff640 1 --2- 172.21.15.60:0/2406704288 >> v2:172.21.15.144:3300/0 conn(0x7f40940ad970 0x7f409415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.720+0000 7f4093fff640 1 -- 172.21.15.60:0/2406704288 >> v1:172.21.15.60:6789/0 conn(0x7f409415ebc0 legacy=0x7f40940cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.720+0000 7f4093fff640 1 -- 172.21.15.60:0/2406704288 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409415bf90 con 0x7f409406c040 2023-10-16T14:12:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.720+0000 7f408bfff640 1 --2- 172.21.15.60:0/2406704288 >> v2:172.21.15.144:3300/0 conn(0x7f40940ad970 0x7f409415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:58.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.720+0000 7f40937fe640 1 -- 172.21.15.60:0/2406704288 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614203873 0 0) 0x7f409406c960 con 0x7f409415ebc0 2023-10-16T14:12:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.721+0000 7f4093fff640 1 --2- 172.21.15.60:0/2406704288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409406c040 0x7f409406c420 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f40940cb060 tx=0x7f408402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fe0530001672c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.721+0000 7f40937fe640 1 -- 172.21.15.60:0/2406704288 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084031830 con 0x7f409406c040 2023-10-16T14:12:58.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.721+0000 7f40937fe640 1 -- 172.21.15.60:0/2406704288 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4084031990 con 0x7f409406c040 2023-10-16T14:12:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.721+0000 7f40937fe640 1 -- 172.21.15.60:0/2406704288 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084031d20 con 0x7f409406c040 2023-10-16T14:12:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 -- 172.21.15.60:0/2406704288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409406c040 msgr2=0x7f409406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 --2- 172.21.15.60:0/2406704288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409406c040 0x7f409406c420 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f40940cb060 tx=0x7f408402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 -- 172.21.15.60:0/2406704288 shutdown_connections 2023-10-16T14:12:58.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 --2- 172.21.15.60:0/2406704288 >> v2:172.21.15.144:3300/0 conn(0x7f40940ad970 0x7f409415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 --2- 172.21.15.60:0/2406704288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409406c040 0x7f409406c420 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 -- 172.21.15.60:0/2406704288 >> 172.21.15.60:0/2406704288 conn(0x7f40940ce6a0 msgr2=0x7f40940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 -- 172.21.15.60:0/2406704288 shutdown_connections 2023-10-16T14:12:58.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.722+0000 7f40992ff640 1 -- 172.21.15.60:0/2406704288 wait complete. 2023-10-16T14:12:58.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.724+0000 7f40992ff640 1 Processor -- start 2023-10-16T14:12:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.724+0000 7f40992ff640 1 -- start start 2023-10-16T14:12:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f40992ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f409406c040 0x7f4094155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f40992ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40940ad970 0x7f40940b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f40992ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 0x7f40940c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f40992ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940b12c0 con 0x7f409415ebc0 2023-10-16T14:12:58.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f40992ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940b1b20 con 0x7f40940ad970 2023-10-16T14:12:58.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f40992ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40940b0ad0 con 0x7f409406c040 2023-10-16T14:12:58.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f4098afe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 0x7f40940c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:58.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f4098afe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 0x7f40940c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51946/0 (socket says 172.21.15.60:51946) 2023-10-16T14:12:58.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.725+0000 7f4098afe640 1 -- 172.21.15.60:0/3641256266 learned_addr learned my addr 172.21.15.60:0/3641256266 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:12:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f408bfff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40940ad970 0x7f40940b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4093fff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f409406c040 0x7f4094155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4098afe640 1 -- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f409406c040 msgr2=0x7f4094155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4098afe640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f409406c040 0x7f4094155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4098afe640 1 -- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40940ad970 msgr2=0x7f40940b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4098afe640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40940ad970 0x7f40940b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4098afe640 1 -- 172.21.15.60:0/3641256266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084009400 con 0x7f409415ebc0 2023-10-16T14:12:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4093fff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f409406c040 0x7f4094155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:12:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f4098afe640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 0x7f40940c9c10 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c7e0 tx=0x7f408c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.726+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00a640 con 0x7f409415ebc0 2023-10-16T14:12:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.727+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c00a7c0 con 0x7f409415ebc0 2023-10-16T14:12:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.727+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00ab70 con 0x7f409415ebc0 2023-10-16T14:12:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.727+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40940b14d0 con 0x7f409415ebc0 2023-10-16T14:12:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.727+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4094156510 con 0x7f409415ebc0 2023-10-16T14:12:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.728+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409406b330 con 0x7f409415ebc0 2023-10-16T14:12:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.729+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f408c00acd0 con 0x7f409415ebc0 2023-10-16T14:12:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.730+0000 7f40917fa640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f407406afe0 0x7f407406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:12:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.730+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f408c0890e0 con 0x7f409415ebc0 2023-10-16T14:12:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.731+0000 7f4093fff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f407406afe0 0x7f407406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:12:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.732+0000 7f4093fff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f407406afe0 0x7f407406d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4084036000 tx=0x7f408403c040 comp rx=0 tx=0).ready entity=mgr.9958 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:12:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.734+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f408c053d70 con 0x7f409415ebc0 2023-10-16T14:12:58.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:12:58.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:12:58.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:58.986+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f409404e9b0 con 0x7f409415ebc0 2023-10-16T14:12:59.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.909+0000 7f40917fa640 1 -- 172.21.15.60:0/3641256266 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v150) v1 ==== 93+0+0 (secure 0 0 0) 0x7f408c018020 con 0x7f409415ebc0 2023-10-16T14:12:59.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f407406afe0 msgr2=0x7f407406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:59.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f407406afe0 0x7f407406d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4084036000 tx=0x7f408403c040 comp rx=0 tx=0).stop 2023-10-16T14:12:59.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 msgr2=0x7f40940c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:12:59.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 0x7f40940c9c10 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c7e0 tx=0x7f408c00cca0 comp rx=0 tx=0).stop 2023-10-16T14:12:59.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 shutdown_connections 2023-10-16T14:12:59.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f407406afe0 0x7f407406d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:59.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f409415ebc0 0x7f40940c9c10 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:59.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40940ad970 0x7f40940b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:59.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 --2- 172.21.15.60:0/3641256266 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f409406c040 0x7f4094155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:12:59.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.913+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 >> 172.21.15.60:0/3641256266 conn(0x7f40940ce6a0 msgr2=0x7f409415c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:12:59.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.915+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 shutdown_connections 2023-10-16T14:12:59.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:12:59.916+0000 7f40992ff640 1 -- 172.21.15.60:0/3641256266 wait complete. 2023-10-16T14:12:59.949 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:12:59.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:13:00.081 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:13:00.180 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:13:00.181 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:00.181 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:00.236 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:00.237 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:00.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:00.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:00.443+0000 7f5e45632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:13:00.471 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:00.468+0000 7fb3b184f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:13:00.501 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:00.497+0000 7f5e45632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:13:00.529 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:00.526+0000 7fb3b184f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:13:00.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.545+0000 7f45b2a1f640 1 Processor -- start 2023-10-16T14:13:00.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.545+0000 7f45b2a1f640 1 -- start start 2023-10-16T14:13:00.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.546+0000 7f45b2a1f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f45ac06b4c0 0x7f45ac0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45b2a1f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac06e170 con 0x7f45ac160420 2023-10-16T14:13:00.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45b2a1f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac0cd460 con 0x7f45ac160800 2023-10-16T14:13:00.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45b2a1f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac06e3c0 con 0x7f45ac06b4c0 2023-10-16T14:13:00.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45abfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f45ac160420 0x7f45ac06adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56894/0 (socket says 172.21.15.60:56894) 2023-10-16T14:13:00.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45abfff640 1 -- 172.21.15.60:0/3184530076 learned_addr learned my addr 172.21.15.60:0/3184530076 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478439483 0 0) 0x7f45ac06e170 con 0x7f45ac160420 2023-10-16T14:13:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4598003520 con 0x7f45ac160420 2023-10-16T14:13:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3184530076 >> v2:172.21.15.144:3300/0 conn(0x7f45ac06b4c0 0x7f45ac0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.547+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4045204204 0 0) 0x7f4598003520 con 0x7f45ac160420 2023-10-16T14:13:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 >> v2:172.21.15.144:3300/0 conn(0x7f45ac06b4c0 msgr2=0x7f45ac0ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 --2- 172.21.15.60:0/3184530076 >> v2:172.21.15.144:3300/0 conn(0x7f45ac06b4c0 0x7f45ac0ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 >> v1:172.21.15.144:6789/0 conn(0x7f45ac160800 legacy=0x7f45ac070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac06e6a0 con 0x7f45ac160420 2023-10-16T14:13:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f459c001670 con 0x7f45ac160420 2023-10-16T14:13:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 >> v1:172.21.15.60:6789/0 conn(0x7f45ac160420 legacy=0x7f45ac06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4598003970 0x7f4598003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f4598004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4598005180 0x7f459800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac06e3c0 con 0x7f45980047c0 2023-10-16T14:13:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459800c0b0 con 0x7f4598005180 2023-10-16T14:13:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459800c270 con 0x7f4598003970 2023-10-16T14:13:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f4598004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45abfff640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4598003970 0x7f4598003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45a3fff640 1 -- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4598003970 msgr2=0x7f4598003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4598003970 0x7f4598003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.548+0000 7f45a3fff640 1 -- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4598005180 msgr2=0x7f459800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.549+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4598005180 0x7f459800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.549+0000 7f45a3fff640 1 -- 172.21.15.60:0/3184530076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4598003520 con 0x7f45980047c0 2023-10-16T14:13:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.549+0000 7f45b0fbc640 1 -- 172.21.15.60:0/3184530076 reap_dead start 2023-10-16T14:13:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.550+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f4598004c40 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f459400ba00 tx=0x7f45980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670b1ce7f902c501 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.550+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594032630 con 0x7f45980047c0 2023-10-16T14:13:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.550+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594032790 con 0x7f45980047c0 2023-10-16T14:13:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.550+0000 7f45ab7fe640 1 -- 172.21.15.60:0/3184530076 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594032b20 con 0x7f45980047c0 2023-10-16T14:13:00.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3184530076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 msgr2=0x7f4598004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f4598004c40 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f459400ba00 tx=0x7f45980057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3184530076 shutdown_connections 2023-10-16T14:13:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3184530076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f4598004c40 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3184530076 >> 172.21.15.60:0/3184530076 conn(0x7f45ac0b0d30 msgr2=0x7f45ac0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:00.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3184530076 shutdown_connections 2023-10-16T14:13:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.551+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3184530076 wait complete. 2023-10-16T14:13:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.553+0000 7f45b2a1f640 1 Processor -- start 2023-10-16T14:13:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.553+0000 7f45b2a1f640 1 -- start start 2023-10-16T14:13:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45b2a1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f45ac0c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45b2a1f640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45ac0c76d0 0x7f45ac0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45b2a1f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45ac0c8090 0x7f45ac0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45b2a1f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac0c8510 con 0x7f45980047c0 2023-10-16T14:13:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45b2a1f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac15a660 con 0x7f45ac0c8090 2023-10-16T14:13:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45b2a1f640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac09a5a0 con 0x7f45ac0c76d0 2023-10-16T14:13:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f45ac0c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f45ac0c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52960/0 (socket says 172.21.15.60:52960) 2023-10-16T14:13:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.554+0000 7f45abfff640 1 -- 172.21.15.60:0/3826115402 learned_addr learned my addr 172.21.15.60:0/3826115402 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45b0fbc640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45ac0c8090 0x7f45ac0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45ac0c76d0 0x7f45ac0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45abfff640 1 -- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45ac0c76d0 msgr2=0x7f45ac0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45abfff640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45ac0c76d0 0x7f45ac0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45abfff640 1 -- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45ac0c8090 msgr2=0x7f45ac0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45abfff640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45ac0c8090 0x7f45ac0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45abfff640 1 -- 172.21.15.60:0/3826115402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac09a880 con 0x7f45980047c0 2023-10-16T14:13:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45abfff640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f45ac0c8ef0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f459c001200 tx=0x7f459c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c00ee80 con 0x7f45980047c0 2023-10-16T14:13:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c00c8d0 con 0x7f45980047c0 2023-10-16T14:13:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c00cc40 con 0x7f45980047c0 2023-10-16T14:13:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.555+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac09ab60 con 0x7f45980047c0 2023-10-16T14:13:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.556+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac09b120 con 0x7f45980047c0 2023-10-16T14:13:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.557+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac04e160 con 0x7f45980047c0 2023-10-16T14:13:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.558+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f459c01d460 con 0x7f45980047c0 2023-10-16T14:13:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.559+0000 7f45a97fa640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 0x7f458c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.559+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f459c089b20 con 0x7f45980047c0 2023-10-16T14:13:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.559+0000 7f45a3fff640 1 -- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 msgr2=0x7f458c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.559+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 0x7f458c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.564+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f459c054760 con 0x7f45980047c0 2023-10-16T14:13:00.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.760+0000 7f45a3fff640 1 -- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 msgr2=0x7f458c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:00.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.760+0000 7f45a3fff640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 0x7f458c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:00.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.761+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45ac0b51c0 con 0x7f45980047c0 2023-10-16T14:13:00.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.763+0000 7f45a97fa640 1 -- 172.21.15.60:0/3826115402 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) v1 ==== 74+0+17209 (secure 0 0 0) 0x7f459c057d80 con 0x7f45980047c0 2023-10-16T14:13:00.766 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:00.766 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":274,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:55.110971+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1984005880":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2098656759":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6833/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/307080156":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6832/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2842352837":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2156516012":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2942868570":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/786228305":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1402597431":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:00.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 msgr2=0x7f458c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 0x7f458c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 msgr2=0x7f45ac0c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f45ac0c8ef0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f459c001200 tx=0x7f459c00ec30 comp rx=0 tx=0).stop 2023-10-16T14:13:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 shutdown_connections 2023-10-16T14:13:00.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f458c06b020 0x7f458c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f45ac0c76d0 0x7f45ac0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45ac0c8090 0x7f45ac0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.765+0000 7f45b2a1f640 1 --2- 172.21.15.60:0/3826115402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45980047c0 0x7f45ac0c8ef0 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.766+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 >> 172.21.15.60:0/3826115402 conn(0x7f45ac0b0d30 msgr2=0x7f45ac0c9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:00.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.766+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 shutdown_connections 2023-10-16T14:13:00.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:00.766+0000 7f45b2a1f640 1 -- 172.21.15.60:0/3826115402 wait complete. 2023-10-16T14:13:00.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:12:37.972576+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:12:39.411671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '273', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:13:00.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:13:00.948 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:00.944+0000 7f5e45632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:13:01.000 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:00.996+0000 7f5e45632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:13:01.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.004+0000 7f2392ddd640 1 Processor -- start 2023-10-16T14:13:01.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.005+0000 7f2392ddd640 1 -- start start 2023-10-16T14:13:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.006+0000 7f2392ddd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f238c15e350 0x7f238c0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.006+0000 7f2392ddd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c070970 0x7f238c0ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.006+0000 7f2392ddd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c06fbc0 con 0x7f238c15e980 2023-10-16T14:13:01.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.006+0000 7f2392ddd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c15e730 con 0x7f238c15e350 2023-10-16T14:13:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.006+0000 7f2392ddd640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0c9920 con 0x7f238c070970 2023-10-16T14:13:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.006+0000 7f23925dc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f238c15e980 0x7f238c15edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56898/0 (socket says 172.21.15.60:56898) 2023-10-16T14:13:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f23925dc640 1 -- 172.21.15.60:0/2622880685 learned_addr learned my addr 172.21.15.60:0/2622880685 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2391ddb640 1 --2- 172.21.15.60:0/2622880685 >> v2:172.21.15.144:3300/0 conn(0x7f238c15e350 0x7f238c0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f23915da640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c070970 0x7f238c0ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2390dd9640 1 -- 172.21.15.60:0/2622880685 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713117603 0 0) 0x7f238c06fbc0 con 0x7f238c15e980 2023-10-16T14:13:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2390dd9640 1 -- 172.21.15.60:0/2622880685 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378003520 con 0x7f238c15e980 2023-10-16T14:13:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2391ddb640 1 -- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c070970 msgr2=0x7f238c0ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2391ddb640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c070970 0x7f238c0ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2391ddb640 1 -- 172.21.15.60:0/2622880685 >> v1:172.21.15.60:6789/0 conn(0x7f238c15e980 legacy=0x7f238c15edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f2391ddb640 1 -- 172.21.15.60:0/2622880685 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0c9c00 con 0x7f238c15e350 2023-10-16T14:13:01.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.007+0000 7f23915da640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c070970 0x7f238c0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.009+0000 7f2391ddb640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c15e350 0x7f238c0cdb40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2388009d90 tx=0x7f238802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e8c1170816b0399 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2390dd9640 1 -- 172.21.15.60:0/2622880685 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238802cb30 con 0x7f238c15e350 2023-10-16T14:13:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2390dd9640 1 -- 172.21.15.60:0/2622880685 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238802cc90 con 0x7f238c15e350 2023-10-16T14:13:01.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2390dd9640 1 -- 172.21.15.60:0/2622880685 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388034440 con 0x7f238c15e350 2023-10-16T14:13:01.016 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.010+0000 7fb3b184f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:13:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 -- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c15e350 msgr2=0x7f238c0cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c15e350 0x7f238c0cdb40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2388009d90 tx=0x7f238802bd80 comp rx=0 tx=0).stop 2023-10-16T14:13:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 -- 172.21.15.60:0/2622880685 shutdown_connections 2023-10-16T14:13:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c070970 0x7f238c0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 --2- 172.21.15.60:0/2622880685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c15e350 0x7f238c0cdb40 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 -- 172.21.15.60:0/2622880685 >> 172.21.15.60:0/2622880685 conn(0x7f238c0b0a50 msgr2=0x7f238c0b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:01.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 -- 172.21.15.60:0/2622880685 shutdown_connections 2023-10-16T14:13:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.010+0000 7f2392ddd640 1 -- 172.21.15.60:0/2622880685 wait complete. 2023-10-16T14:13:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.012+0000 7f2392ddd640 1 Processor -- start 2023-10-16T14:13:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.012+0000 7f2392ddd640 1 -- start start 2023-10-16T14:13:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f2392ddd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c070970 0x7f238c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f2392ddd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 0x7f238c0c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f2392ddd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c15e980 0x7f238c0c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f2392ddd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0c92b0 con 0x7f238c15e350 2023-10-16T14:13:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f2392ddd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0c8730 con 0x7f238c070970 2023-10-16T14:13:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f2392ddd640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0c8a10 con 0x7f238c15e980 2023-10-16T14:13:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f23915da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 0x7f238c0c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f23915da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 0x7f238c0c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52968/0 (socket says 172.21.15.60:52968) 2023-10-16T14:13:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.013+0000 7f23915da640 1 -- 172.21.15.60:0/3821054595 learned_addr learned my addr 172.21.15.60:0/3821054595 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f2391ddb640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c070970 0x7f238c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23925dc640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c15e980 0x7f238c0c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23915da640 1 -- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c15e980 msgr2=0x7f238c0c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23915da640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c15e980 0x7f238c0c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23915da640 1 -- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c070970 msgr2=0x7f238c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23915da640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c070970 0x7f238c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23915da640 1 -- 172.21.15.60:0/3821054595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388009a70 con 0x7f238c15e350 2023-10-16T14:13:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.014+0000 7f23915da640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 0x7f238c0c8100 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f237c00d7d0 tx=0x7f237c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.015+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c004030 con 0x7f238c15e350 2023-10-16T14:13:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.015+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c004190 con 0x7f238c15e350 2023-10-16T14:13:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.015+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c0c8cf0 con 0x7f238c15e350 2023-10-16T14:13:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.015+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c004520 con 0x7f238c15e350 2023-10-16T14:13:01.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.015+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c0c57d0 con 0x7f238c15e350 2023-10-16T14:13:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.016+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c06e500 con 0x7f238c15e350 2023-10-16T14:13:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.017+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f237c0096a0 con 0x7f238c15e350 2023-10-16T14:13:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.018+0000 7f23827fc640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 0x7f237006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.018+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f237c089cd0 con 0x7f238c15e350 2023-10-16T14:13:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.018+0000 7f2391ddb640 1 -- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 msgr2=0x7f237006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.018+0000 7f2391ddb640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 0x7f237006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.022+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f237c054890 con 0x7f238c15e350 2023-10-16T14:13:01.052 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.048+0000 7f5e45632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:13:01.098 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.094+0000 7fb3b184f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:13:01.113 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.109+0000 7f5e45632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:13:01.153 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.149+0000 7fb3b184f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:13:01.161 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.157+0000 7f5e45632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:13:01.219 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.215+0000 7fb3b184f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:13:01.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.219+0000 7f2391ddb640 1 -- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 msgr2=0x7f237006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:01.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.219+0000 7f2391ddb640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 0x7f237006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:01.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.252+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f238c0716e0 con 0x7f238c15e350 2023-10-16T14:13:01.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.253+0000 7f23827fc640 1 -- 172.21.15.60:0/3821054595 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v274) v1 ==== 93+0+10 (secure 0 0 0) 0x7f237c057eb0 con 0x7f238c15e350 2023-10-16T14:13:01.257 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:13:01.259 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.255+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 msgr2=0x7f237006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:01.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.255+0000 7f2392ddd640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 0x7f237006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.255+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 msgr2=0x7f238c0c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.255+0000 7f2392ddd640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 0x7f238c0c8100 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f237c00d7d0 tx=0x7f237c00dc90 comp rx=0 tx=0).stop 2023-10-16T14:13:01.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 shutdown_connections 2023-10-16T14:13:01.260 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f237006b020 0x7f237006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f238c15e980 0x7f238c0c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f238c15e350 0x7f238c0c8100 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 --2- 172.21.15.60:0/3821054595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f238c070970 0x7f238c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 >> 172.21.15.60:0/3821054595 conn(0x7f238c0b0a50 msgr2=0x7f238c15edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:01.261 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 shutdown_connections 2023-10-16T14:13:01.262 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.256+0000 7f2392ddd640 1 -- 172.21.15.60:0/3821054595 wait complete. 2023-10-16T14:13:01.270 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.267+0000 7f5e45632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:13:01.272 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.269+0000 7fb3b184f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:13:01.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:13:01.333 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.329+0000 7f5e45632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:01.381 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.377+0000 7f5e45632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:13:01.395 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.392+0000 7fb3b184f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:13:01.438 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.434+0000 7f5e45632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:13:01.467 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.463+0000 7fb3b184f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:01.488 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.484+0000 7f5e45632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:13:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.485+0000 7f192a1ff640 1 Processor -- start 2023-10-16T14:13:01.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.486+0000 7f192a1ff640 1 -- start start 2023-10-16T14:13:01.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.487+0000 7f192a1ff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f192415cd70 0x7f19240d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.487+0000 7f192a1ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19240b8480 0x7f19240cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.487+0000 7f192a1ff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19240d0be0 0x7f19240cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.487+0000 7f192a1ff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192415d150 con 0x7f19240d0be0 2023-10-16T14:13:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.487+0000 7f192a1ff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19240d1180 con 0x7f192415cd70 2023-10-16T14:13:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.487+0000 7f192a1ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19240d1460 con 0x7f19240b8480 2023-10-16T14:13:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19240d0be0 0x7f19240cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19240d0be0 0x7f19240cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52976/0 (socket says 172.21.15.60:52976) 2023-10-16T14:13:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 -- 172.21.15.60:0/430348662 learned_addr learned my addr 172.21.15.60:0/430348662 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19291fd640 1 --2- 172.21.15.60:0/430348662 >> v2:172.21.15.144:3300/0 conn(0x7f192415cd70 0x7f19240d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19289fc640 1 --2- 172.21.15.60:0/430348662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19240b8480 0x7f19240cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 -- 172.21.15.60:0/430348662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19240b8480 msgr2=0x7f19240cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:01.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 --2- 172.21.15.60:0/430348662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19240b8480 0x7f19240cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 -- 172.21.15.60:0/430348662 >> v2:172.21.15.144:3300/0 conn(0x7f192415cd70 msgr2=0x7f19240d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:01.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 --2- 172.21.15.60:0/430348662 >> v2:172.21.15.144:3300/0 conn(0x7f192415cd70 0x7f19240d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.488+0000 7f19299fe640 1 -- 172.21.15.60:0/430348662 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19240d1740 con 0x7f19240d0be0 2023-10-16T14:13:01.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.490+0000 7f19299fe640 1 --2- 172.21.15.60:0/430348662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240d0be0 0x7f19240cdd00 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f19240b56a0 tx=0x7f191c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2ce2bb51023c8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:01.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.490+0000 7f191bfff640 1 -- 172.21.15.60:0/430348662 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c02ca80 con 0x7f19240d0be0 2023-10-16T14:13:01.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.490+0000 7f191bfff640 1 -- 172.21.15.60:0/430348662 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c02cbe0 con 0x7f19240d0be0 2023-10-16T14:13:01.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.490+0000 7f191bfff640 1 -- 172.21.15.60:0/430348662 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c0343f0 con 0x7f19240d0be0 2023-10-16T14:13:01.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 -- 172.21.15.60:0/430348662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240d0be0 msgr2=0x7f19240cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 --2- 172.21.15.60:0/430348662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240d0be0 0x7f19240cdd00 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f19240b56a0 tx=0x7f191c02bee0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 -- 172.21.15.60:0/430348662 shutdown_connections 2023-10-16T14:13:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 --2- 172.21.15.60:0/430348662 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240d0be0 0x7f19240cdd00 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 --2- 172.21.15.60:0/430348662 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f19240b8480 0x7f19240cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 --2- 172.21.15.60:0/430348662 >> v2:172.21.15.144:3300/0 conn(0x7f192415cd70 0x7f19240d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 -- 172.21.15.60:0/430348662 >> 172.21.15.60:0/430348662 conn(0x7f19240cf570 msgr2=0x7f19240cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 -- 172.21.15.60:0/430348662 shutdown_connections 2023-10-16T14:13:01.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.491+0000 7f192a1ff640 1 -- 172.21.15.60:0/430348662 wait complete. 2023-10-16T14:13:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.493+0000 7f192a1ff640 1 Processor -- start 2023-10-16T14:13:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.493+0000 7f192a1ff640 1 -- start start 2023-10-16T14:13:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.494+0000 7f192a1ff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 0x7f1924153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.494+0000 7f192a1ff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19240d0be0 0x7f19240b74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.494+0000 7f192a1ff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192415cd70 0x7f19241f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.494+0000 7f192a1ff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19241602f0 con 0x7f19240b8480 2023-10-16T14:13:01.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.494+0000 7f192a1ff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1924154760 con 0x7f19240d0be0 2023-10-16T14:13:01.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.494+0000 7f192a1ff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1924154a40 con 0x7f192415cd70 2023-10-16T14:13:01.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 0x7f1924153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 0x7f1924153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52990/0 (socket says 172.21.15.60:52990) 2023-10-16T14:13:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 -- 172.21.15.60:0/2303458864 learned_addr learned my addr 172.21.15.60:0/2303458864 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19299fe640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192415cd70 0x7f19241f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19289fc640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19240d0be0 0x7f19240b74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 -- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192415cd70 msgr2=0x7f19241f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192415cd70 0x7f19241f50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 -- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19240d0be0 msgr2=0x7f19240b74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19240d0be0 0x7f19240b74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 -- 172.21.15.60:0/2303458864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c009b30 con 0x7f19240b8480 2023-10-16T14:13:01.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.495+0000 7f19291fd640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 0x7f1924153a50 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f192000d800 tx=0x7f192000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.496+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192000b6c0 con 0x7f19240b8480 2023-10-16T14:13:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.496+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192000b820 con 0x7f19240b8480 2023-10-16T14:13:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.496+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1924154d20 con 0x7f19240b8480 2023-10-16T14:13:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.496+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192000bbb0 con 0x7f19240b8480 2023-10-16T14:13:01.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.496+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1924153f90 con 0x7f19240b8480 2023-10-16T14:13:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.497+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192404e9b0 con 0x7f19240b8480 2023-10-16T14:13:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.499+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f192000bd10 con 0x7f19240b8480 2023-10-16T14:13:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.500+0000 7f1919ffb640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f190806b020 0x7f190806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.500+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f192008a0b0 con 0x7f19240b8480 2023-10-16T14:13:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.500+0000 7f19289fc640 1 -- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f190806b020 msgr2=0x7f190806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:01.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.500+0000 7f19289fc640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f190806b020 0x7f190806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:01.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.504+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1920054c60 con 0x7f19240b8480 2023-10-16T14:13:01.525 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.521+0000 7fb3b184f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:13:01.586 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.582+0000 7fb3b184f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:13:01.639 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:01.636+0000 7fb3b184f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:13:01.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.693+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19240c85d0 con 0x7f19240b8480 2023-10-16T14:13:01.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.694+0000 7f1919ffb640 1 -- 172.21.15.60:0/2303458864 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v274) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1920058280 con 0x7f19240b8480 2023-10-16T14:13:01.698 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:13:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f190806b020 msgr2=0x7f190806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f190806b020 0x7f190806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 msgr2=0x7f1924153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 0x7f1924153a50 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f192000d800 tx=0x7f192000dcc0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 shutdown_connections 2023-10-16T14:13:01.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f190806b020 0x7f190806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f192415cd70 0x7f19241f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19240d0be0 0x7f19240b74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 --2- 172.21.15.60:0/2303458864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19240b8480 0x7f1924153a50 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 >> 172.21.15.60:0/2303458864 conn(0x7f19240cf570 msgr2=0x7f19240d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:01.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 shutdown_connections 2023-10-16T14:13:01.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.697+0000 7f192a1ff640 1 -- 172.21.15.60:0/2303458864 wait complete. 2023-10-16T14:13:01.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:13:01.866 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:01.863+0000 7f5e45632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:13:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.935+0000 7f6b9874e640 1 Processor -- start 2023-10-16T14:13:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.935+0000 7f6b9874e640 1 -- start start 2023-10-16T14:13:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.936+0000 7f6b9874e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b900cf710 0x7f6b900b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.936+0000 7f6b9874e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b900b6cd0 0x7f6b900b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.936+0000 7f6b9874e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b900d3980 con 0x7f6b900cfaf0 2023-10-16T14:13:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.936+0000 7f6b9874e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b900d3c60 con 0x7f6b900cf710 2023-10-16T14:13:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.936+0000 7f6b9874e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b900ce630 con 0x7f6b900b6cd0 2023-10-16T14:13:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b964ea640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b900cf710 0x7f6b900b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b95ce9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b900cfaf0 0x7f6b900d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56904/0 (socket says 172.21.15.60:56904) 2023-10-16T14:13:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b95ce9640 1 -- 172.21.15.60:0/912687718 learned_addr learned my addr 172.21.15.60:0/912687718 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b96ceb640 1 --2- 172.21.15.60:0/912687718 >> v2:172.21.15.144:3300/0 conn(0x7f6b900b6cd0 0x7f6b900b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b964ea640 1 -- 172.21.15.60:0/912687718 >> v2:172.21.15.144:3300/0 conn(0x7f6b900b6cd0 msgr2=0x7f6b900b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b964ea640 1 --2- 172.21.15.60:0/912687718 >> v2:172.21.15.144:3300/0 conn(0x7f6b900b6cd0 0x7f6b900b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b964ea640 1 -- 172.21.15.60:0/912687718 >> v1:172.21.15.60:6789/0 conn(0x7f6b900cfaf0 legacy=0x7f6b900d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b964ea640 1 -- 172.21.15.60:0/912687718 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b900ce910 con 0x7f6b900cf710 2023-10-16T14:13:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.937+0000 7f6b954e8640 1 -- 172.21.15.60:0/912687718 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372727380 0 0) 0x7f6b900d3980 con 0x7f6b900cfaf0 2023-10-16T14:13:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b964ea640 1 --2- 172.21.15.60:0/912687718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cf710 0x7f6b900b2d10 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f6b900b5890 tx=0x7f6b8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3912a1e70c1eb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b954e8640 1 -- 172.21.15.60:0/912687718 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c02e070 con 0x7f6b900cf710 2023-10-16T14:13:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b954e8640 1 -- 172.21.15.60:0/912687718 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c0315a0 con 0x7f6b900cf710 2023-10-16T14:13:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b954e8640 1 -- 172.21.15.60:0/912687718 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c031930 con 0x7f6b900cf710 2023-10-16T14:13:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b9874e640 1 -- 172.21.15.60:0/912687718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cf710 msgr2=0x7f6b900b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b9874e640 1 --2- 172.21.15.60:0/912687718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cf710 0x7f6b900b2d10 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f6b900b5890 tx=0x7f6b8c02b840 comp rx=0 tx=0).stop 2023-10-16T14:13:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b9874e640 1 -- 172.21.15.60:0/912687718 shutdown_connections 2023-10-16T14:13:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b9874e640 1 --2- 172.21.15.60:0/912687718 >> v2:172.21.15.144:3300/0 conn(0x7f6b900b6cd0 0x7f6b900b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b9874e640 1 --2- 172.21.15.60:0/912687718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cf710 0x7f6b900b2d10 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.939+0000 7f6b9874e640 1 -- 172.21.15.60:0/912687718 >> 172.21.15.60:0/912687718 conn(0x7f6b900d2cc0 msgr2=0x7f6b900d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.940+0000 7f6b9874e640 1 -- 172.21.15.60:0/912687718 shutdown_connections 2023-10-16T14:13:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.940+0000 7f6b9874e640 1 -- 172.21.15.60:0/912687718 wait complete. 2023-10-16T14:13:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.942+0000 7f6b9874e640 1 Processor -- start 2023-10-16T14:13:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.942+0000 7f6b9874e640 1 -- start start 2023-10-16T14:13:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.945+0000 7f6b9874e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b900b6cd0 0x7f6b900d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.945+0000 7f6b9874e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b900cf710 0x7f6b900d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.945+0000 7f6b9874e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 0x7f6b900c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.945+0000 7f6b9874e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cbea0 con 0x7f6b900cfaf0 2023-10-16T14:13:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.945+0000 7f6b9874e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cc0b0 con 0x7f6b900cf710 2023-10-16T14:13:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.945+0000 7f6b9874e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cc390 con 0x7f6b900b6cd0 2023-10-16T14:13:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 0x7f6b900c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 0x7f6b900c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53006/0 (socket says 172.21.15.60:53006) 2023-10-16T14:13:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 -- 172.21.15.60:0/3102712337 learned_addr learned my addr 172.21.15.60:0/3102712337 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b95ce9640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b900cf710 0x7f6b900d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 -- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b900b6cd0 msgr2=0x7f6b900d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b900b6cd0 0x7f6b900d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 -- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b900cf710 msgr2=0x7f6b900d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b900cf710 0x7f6b900d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.946+0000 7f6b96ceb640 1 -- 172.21.15.60:0/3102712337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c0093a0 con 0x7f6b900cfaf0 2023-10-16T14:13:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.947+0000 7f6b96ceb640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 0x7f6b900c9e90 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f6b88007e60 tx=0x7f6b8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.948+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88014420 con 0x7f6b900cfaf0 2023-10-16T14:13:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.948+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b880145a0 con 0x7f6b900cfaf0 2023-10-16T14:13:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.948+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b900cc670 con 0x7f6b900cfaf0 2023-10-16T14:13:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.948+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88014950 con 0x7f6b900cfaf0 2023-10-16T14:13:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.948+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b900ccc30 con 0x7f6b900cfaf0 2023-10-16T14:13:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.949+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b900cce50 con 0x7f6b900cfaf0 2023-10-16T14:13:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.951+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f6b88014ab0 con 0x7f6b900cfaf0 2023-10-16T14:13:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.952+0000 7f6b86ffd640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f6b7406b020 0x7f6b7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.952+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f6b8808a470 con 0x7f6b900cfaf0 2023-10-16T14:13:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.952+0000 7f6b964ea640 1 -- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f6b7406b020 msgr2=0x7f6b7406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.952+0000 7f6b964ea640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f6b7406b020 0x7f6b7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:01.957+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b880550c0 con 0x7f6b900cfaf0 2023-10-16T14:13:02.029 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.025+0000 7f5e45632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:13:02.059 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.055+0000 7fb3b184f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:13:02.136 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.132+0000 7f5e45632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:13:02.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.146+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6b9008c490 con 0x7f6b900cfaf0 2023-10-16T14:13:02.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.147+0000 7f6b86ffd640 1 -- 172.21.15.60:0/3102712337 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v274) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6b880586e0 con 0x7f6b900cfaf0 2023-10-16T14:13:02.151 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:13:02.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f6b7406b020 msgr2=0x7f6b7406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:02.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f6b7406b020 0x7f6b7406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 msgr2=0x7f6b900c9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:02.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 0x7f6b900c9e90 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f6b88007e60 tx=0x7f6b8800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 shutdown_connections 2023-10-16T14:13:02.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f6b7406b020 0x7f6b7406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b900cfaf0 0x7f6b900c9e90 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b900cf710 0x7f6b900d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 --2- 172.21.15.60:0/3102712337 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b900b6cd0 0x7f6b900d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 >> 172.21.15.60:0/3102712337 conn(0x7f6b900d2cc0 msgr2=0x7f6b900cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:02.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 shutdown_connections 2023-10-16T14:13:02.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.150+0000 7f6b9874e640 1 -- 172.21.15.60:0/3102712337 wait complete. 2023-10-16T14:13:02.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:13:02.185 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.181+0000 7f5e45632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:13:02.230 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.226+0000 7fb3b184f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:13:02.292 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.288+0000 7f5e45632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:02.346 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.342+0000 7fb3b184f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:13:02.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.371+0000 7ff043fff640 1 Processor -- start 2023-10-16T14:13:02.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.371+0000 7ff043fff640 1 -- start start 2023-10-16T14:13:02.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff043fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 0x7ff0440d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:02.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff043fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0440cded0 0x7ff0440b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff043fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04415d150 con 0x7ff0440b8480 2023-10-16T14:13:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff043fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0440d1180 con 0x7ff0440cded0 2023-10-16T14:13:02.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff043fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440d1460 con 0x7ff04415cd70 2023-10-16T14:13:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff042ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 0x7ff0440d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff042ffd640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 0x7ff0440d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:45252/0 (socket says 172.21.15.60:45252) 2023-10-16T14:13:02.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.373+0000 7ff042ffd640 1 -- 172.21.15.60:0/764828339 learned_addr learned my addr 172.21.15.60:0/764828339 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.374+0000 7ff0437fe640 1 --2- 172.21.15.60:0/764828339 >> v2:172.21.15.144:3300/0 conn(0x7ff0440cded0 0x7ff0440b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.374+0000 7ff042ffd640 1 -- 172.21.15.60:0/764828339 >> v2:172.21.15.144:3300/0 conn(0x7ff0440cded0 msgr2=0x7ff0440b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.374+0000 7ff042ffd640 1 --2- 172.21.15.60:0/764828339 >> v2:172.21.15.144:3300/0 conn(0x7ff0440cded0 0x7ff0440b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.374+0000 7ff042ffd640 1 -- 172.21.15.60:0/764828339 >> v1:172.21.15.144:6789/0 conn(0x7ff0440b8480 legacy=0x7ff0440cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:02.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.374+0000 7ff042ffd640 1 -- 172.21.15.60:0/764828339 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440d1740 con 0x7ff04415cd70 2023-10-16T14:13:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.375+0000 7ff042ffd640 1 --2- 172.21.15.60:0/764828339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 0x7ff0440d2220 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51253ef783fe33e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff0427fc640 1 -- 172.21.15.60:0/764828339 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034030840 con 0x7ff04415cd70 2023-10-16T14:13:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff0427fc640 1 -- 172.21.15.60:0/764828339 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0340309a0 con 0x7ff04415cd70 2023-10-16T14:13:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff0427fc640 1 -- 172.21.15.60:0/764828339 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034030d30 con 0x7ff04415cd70 2023-10-16T14:13:02.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff043fff640 1 -- 172.21.15.60:0/764828339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 msgr2=0x7ff0440d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:02.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff043fff640 1 --2- 172.21.15.60:0/764828339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 0x7ff0440d2220 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03402b980 comp rx=0 tx=0).stop 2023-10-16T14:13:02.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff043fff640 1 -- 172.21.15.60:0/764828339 shutdown_connections 2023-10-16T14:13:02.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff043fff640 1 --2- 172.21.15.60:0/764828339 >> v2:172.21.15.144:3300/0 conn(0x7ff0440cded0 0x7ff0440b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff043fff640 1 --2- 172.21.15.60:0/764828339 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff04415cd70 0x7ff0440d2220 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.376+0000 7ff043fff640 1 -- 172.21.15.60:0/764828339 >> 172.21.15.60:0/764828339 conn(0x7ff0440cf570 msgr2=0x7ff0440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:02.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.377+0000 7ff043fff640 1 -- 172.21.15.60:0/764828339 shutdown_connections 2023-10-16T14:13:02.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.377+0000 7ff043fff640 1 -- 172.21.15.60:0/764828339 wait complete. 2023-10-16T14:13:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.378+0000 7ff043fff640 1 Processor -- start 2023-10-16T14:13:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.378+0000 7ff043fff640 1 -- start start 2023-10-16T14:13:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.379+0000 7ff043fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440b8480 0x7ff0440b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.379+0000 7ff043fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0440cded0 0x7ff0440b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.379+0000 7ff043fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 0x7ff0440b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.379+0000 7ff043fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440b1870 con 0x7ff04415cd70 2023-10-16T14:13:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.379+0000 7ff043fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440b23f0 con 0x7ff0440b8480 2023-10-16T14:13:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.379+0000 7ff043fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440b1ac0 con 0x7ff0440cded0 2023-10-16T14:13:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 0x7ff0440b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 0x7ff0440b2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53018/0 (socket says 172.21.15.60:53018) 2023-10-16T14:13:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 -- 172.21.15.60:0/1254406745 learned_addr learned my addr 172.21.15.60:0/1254406745 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff042ffd640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440b8480 0x7ff0440b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff03a7fc640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0440cded0 0x7ff0440b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 -- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0440cded0 msgr2=0x7ff0440b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0440cded0 0x7ff0440b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 -- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440b8480 msgr2=0x7ff0440b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440b8480 0x7ff0440b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff0437fe640 1 -- 172.21.15.60:0/1254406745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340094d0 con 0x7ff04415cd70 2023-10-16T14:13:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.380+0000 7ff03a7fc640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0440cded0 0x7ff0440b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.381+0000 7ff0437fe640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 0x7ff0440b2880 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7ff03c007dc0 tx=0x7ff03c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.381+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c014420 con 0x7ff04415cd70 2023-10-16T14:13:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.381+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c0145a0 con 0x7ff04415cd70 2023-10-16T14:13:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.381+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c014950 con 0x7ff04415cd70 2023-10-16T14:13:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.381+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0440b1da0 con 0x7ff04415cd70 2023-10-16T14:13:02.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.381+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0440b2dc0 con 0x7ff04415cd70 2023-10-16T14:13:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.383+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0440d1ce0 con 0x7ff04415cd70 2023-10-16T14:13:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.383+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7ff03c014ab0 con 0x7ff04415cd70 2023-10-16T14:13:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.384+0000 7ff040ff9640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 0x7ff02806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.384+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7ff03c08a330 con 0x7ff04415cd70 2023-10-16T14:13:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.385+0000 7ff042ffd640 1 -- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 msgr2=0x7ff02806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.385+0000 7ff042ffd640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 0x7ff02806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.389+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff0440d1ce0 con 0x7ff04415cd70 2023-10-16T14:13:02.400 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.397+0000 7f5e45632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:13:02.402 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.398+0000 7fb3b184f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:13:02.517 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.513+0000 7fb3b184f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:02.556 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.552+0000 7f5e45632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.586+0000 7ff042ffd640 1 -- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 msgr2=0x7ff02806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.586+0000 7ff042ffd640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 0x7ff02806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:02.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.597+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff0440d1ce0 con 0x7ff04415cd70 2023-10-16T14:13:02.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.598+0000 7ff040ff9640 1 -- 172.21.15.60:0/1254406745 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 11 entries v274) v1 ==== 81+0+651 (secure 0 0 0) 0x7ff0440d1ce0 con 0x7ff04415cd70 2023-10-16T14:13:02.602 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1984005880 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.603 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2098656759 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.603 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/948422863 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.603 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/307080156 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.603 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/948422863 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.604 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2842352837 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.604 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2156516012 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.604 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2942868570 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.604 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/786228305 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.604 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1402597431 2023-10-17T14:12:55.110891+0000 2023-10-16T14:13:02.605 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1518159871 2023-10-16T15:12:37.308329+0000 2023-10-16T14:13:02.605 INFO:teuthology.orchestra.run.smithi060.stderr:listed 11 entries 2023-10-16T14:13:02.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 msgr2=0x7ff02806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:02.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 0x7ff02806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 msgr2=0x7ff0440b2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:02.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 0x7ff0440b2880 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7ff03c007dc0 tx=0x7ff03c00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 shutdown_connections 2023-10-16T14:13:02.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7ff02806b050 0x7ff02806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff0440cded0 0x7ff0440b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff04415cd70 0x7ff0440b2880 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 --2- 172.21.15.60:0/1254406745 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440b8480 0x7ff0440b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:02.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 >> 172.21.15.60:0/1254406745 conn(0x7ff0440cf570 msgr2=0x7ff0440d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:02.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 shutdown_connections 2023-10-16T14:13:02.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:02.601+0000 7ff043fff640 1 -- 172.21.15.60:0/1254406745 wait complete. 2023-10-16T14:13:02.608 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.601+0000 7f5e45632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:02.622 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:13:02.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:02.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:13:02.633 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.630+0000 7fb3b184f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:13:02.721 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:13:02.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:02.722 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:13:02.724 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:02.720+0000 7f5e45632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:13:02.807 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.803+0000 7fb3b184f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:02.816 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:02.816 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:02.859 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.855+0000 7fb3b184f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:02.873 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:02.873 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:02.874 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:02.931 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:13:02.931 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:02.932 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:02.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:02.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:02.997 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:02.987+0000 7fb3b184f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:13:03.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.195+0000 7faf0359e640 1 Processor -- start 2023-10-16T14:13:03.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.196+0000 7faf0359e640 1 -- start start 2023-10-16T14:13:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.197+0000 7faf0359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf040b8480 0x7faf040cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.197+0000 7faf0359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.197+0000 7faf0359e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0415d150 con 0x7faf0415cd70 2023-10-16T14:13:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.197+0000 7faf0359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf040d1180 con 0x7faf040b8480 2023-10-16T14:13:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.197+0000 7faf0359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf040d1460 con 0x7faf040d0be0 2023-10-16T14:13:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf040b8480 0x7faf040cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf040b8480 0x7faf040cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53026/0 (socket says 172.21.15.60:53026) 2023-10-16T14:13:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 -- 172.21.15.60:0/3505702134 learned_addr learned my addr 172.21.15.60:0/3505702134 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faf02d9d640 1 --2- 172.21.15.60:0/3505702134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 -- 172.21.15.60:0/3505702134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 msgr2=0x7faf040cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 --2- 172.21.15.60:0/3505702134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 -- 172.21.15.60:0/3505702134 >> v1:172.21.15.144:6789/0 conn(0x7faf0415cd70 legacy=0x7faf040d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faefbfff640 1 -- 172.21.15.60:0/3505702134 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf040d1740 con 0x7faf040b8480 2023-10-16T14:13:03.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.198+0000 7faf01d9b640 1 -- 172.21.15.60:0/3505702134 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666050863 0 0) 0x7faf0415d150 con 0x7faf0415cd70 2023-10-16T14:13:03.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.200+0000 7faefbfff640 1 --2- 172.21.15.60:0/3505702134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf040b8480 0x7faf040cd7c0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7faeec009870 tx=0x7faeec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4629213353e38a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:03.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.200+0000 7faf01d9b640 1 -- 172.21.15.60:0/3505702134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec030840 con 0x7faf040b8480 2023-10-16T14:13:03.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf01d9b640 1 -- 172.21.15.60:0/3505702134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec0309a0 con 0x7faf040b8480 2023-10-16T14:13:03.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf01d9b640 1 -- 172.21.15.60:0/3505702134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec030d30 con 0x7faf040b8480 2023-10-16T14:13:03.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 -- 172.21.15.60:0/3505702134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf040b8480 msgr2=0x7faf040cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 --2- 172.21.15.60:0/3505702134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf040b8480 0x7faf040cd7c0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7faeec009870 tx=0x7faeec02b980 comp rx=0 tx=0).stop 2023-10-16T14:13:03.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 -- 172.21.15.60:0/3505702134 shutdown_connections 2023-10-16T14:13:03.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 --2- 172.21.15.60:0/3505702134 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 --2- 172.21.15.60:0/3505702134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf040b8480 0x7faf040cd7c0 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 -- 172.21.15.60:0/3505702134 >> 172.21.15.60:0/3505702134 conn(0x7faf040cf570 msgr2=0x7faf040cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:03.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 -- 172.21.15.60:0/3505702134 shutdown_connections 2023-10-16T14:13:03.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.201+0000 7faf0359e640 1 -- 172.21.15.60:0/3505702134 wait complete. 2023-10-16T14:13:03.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.203+0000 7faf0359e640 1 Processor -- start 2023-10-16T14:13:03.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.203+0000 7faf0359e640 1 -- start start 2023-10-16T14:13:03.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf0359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf040b8480 0x7faf0415fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf0359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf0359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 0x7faf040cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf0359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf040ca4c0 con 0x7faf0415cd70 2023-10-16T14:13:03.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf0359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf040ca7a0 con 0x7faf040b8480 2023-10-16T14:13:03.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf0359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf040caa80 con 0x7faf040d0be0 2023-10-16T14:13:03.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf02d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 0x7faf040cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faefbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.204+0000 7faf02d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 0x7faf040cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53028/0 (socket says 172.21.15.60:53028) 2023-10-16T14:13:03.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 -- 172.21.15.60:0/315262884 learned_addr learned my addr 172.21.15.60:0/315262884 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:03.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf0259c640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf040b8480 0x7faf0415fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 -- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 msgr2=0x7faf040c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 -- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf040b8480 msgr2=0x7faf0415fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf040b8480 0x7faf0415fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 -- 172.21.15.60:0/315262884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0094d0 con 0x7faf0415cd70 2023-10-16T14:13:03.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf0259c640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf040b8480 0x7faf0415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:03.213 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.205+0000 7faf02d9d640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 0x7faf040cb670 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7faefc007d90 tx=0x7faefc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:03.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.206+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc014420 con 0x7faf0415cd70 2023-10-16T14:13:03.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.206+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faefc0145a0 con 0x7faf0415cd70 2023-10-16T14:13:03.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.206+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc014950 con 0x7faf0415cd70 2023-10-16T14:13:03.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.206+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf040cad60 con 0x7faf0415cd70 2023-10-16T14:13:03.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.206+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf04160270 con 0x7faf0415cd70 2023-10-16T14:13:03.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.207+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf040cb280 con 0x7faf0415cd70 2023-10-16T14:13:03.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.208+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7faefc014ab0 con 0x7faf0415cd70 2023-10-16T14:13:03.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.209+0000 7faefaffd640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 0x7faee406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.209+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7faefc08a3a0 con 0x7faf0415cd70 2023-10-16T14:13:03.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.209+0000 7faf0259c640 1 -- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 msgr2=0x7faee406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:03.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.209+0000 7faf0259c640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 0x7faee406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:03.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.214+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf040cb280 con 0x7faf0415cd70 2023-10-16T14:13:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.410+0000 7faf0259c640 1 -- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 msgr2=0x7faee406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:03.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.410+0000 7faf0259c640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 0x7faee406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:03.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.446+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf040cb280 con 0x7faf0415cd70 2023-10-16T14:13:03.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.447+0000 7faefaffd640 1 -- 172.21.15.60:0/315262884 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4933 (secure 0 0 0) 0x7faf040cb280 con 0x7faf0415cd70 2023-10-16T14:13:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:03.451 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:12:40.387056+0000","modified":"2023-10-16T14:12:43.460423+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12573":{"gid":12573,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2674455624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2674455624},{"type":"v1","addr":"172.21.15.60:6835","nonce":2674455624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T14:13:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 144 2023-10-16T14:13:03.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 msgr2=0x7faee406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:03.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 0x7faee406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 msgr2=0x7faf040cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 0x7faf040cb670 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7faefc007d90 tx=0x7faefc00e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 shutdown_connections 2023-10-16T14:13:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7faee406b020 0x7faee406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faf040d0be0 0x7faf040c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf0415cd70 0x7faf040cb670 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 --2- 172.21.15.60:0/315262884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf040b8480 0x7faf0415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.449+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 >> 172.21.15.60:0/315262884 conn(0x7faf040cf570 msgr2=0x7faf040b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:03.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.450+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 shutdown_connections 2023-10-16T14:13:03.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.450+0000 7faf0359e640 1 -- 172.21.15.60:0/315262884 wait complete. 2023-10-16T14:13:03.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:03.563 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:03.559+0000 7f5e45632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:13:03.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.660+0000 7f1f6359e640 1 Processor -- start 2023-10-16T14:13:03.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.660+0000 7f1f6359e640 1 -- start start 2023-10-16T14:13:03.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f640d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6359e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f640b8480 0x7f1f640cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1f640d0be0 0x7f1f640cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6359e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6415d150 con 0x7f1f640b8480 2023-10-16T14:13:03.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f640d1180 con 0x7f1f640d0be0 2023-10-16T14:13:03.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640d1460 con 0x7f1f6415cd70 2023-10-16T14:13:03.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f5bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f640b8480 0x7f1f640cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f5bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f640b8480 0x7f1f640cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53040/0 (socket says 172.21.15.60:53040) 2023-10-16T14:13:03.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f5bfff640 1 -- 172.21.15.60:0/1354593481 learned_addr learned my addr 172.21.15.60:0/1354593481 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:03.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f6259c640 1 --2- 172.21.15.60:0/1354593481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f640d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.662+0000 7f1f62d9d640 1 --2- 172.21.15.60:0/1354593481 >> v2:172.21.15.144:3300/0 conn(0x7f1f640d0be0 0x7f1f640cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.663+0000 7f1f5bfff640 1 -- 172.21.15.60:0/1354593481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 msgr2=0x7f1f640d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:03.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.663+0000 7f1f5bfff640 1 --2- 172.21.15.60:0/1354593481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f640d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.663+0000 7f1f5bfff640 1 -- 172.21.15.60:0/1354593481 >> v2:172.21.15.144:3300/0 conn(0x7f1f640d0be0 msgr2=0x7f1f640cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:03.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.663+0000 7f1f5bfff640 1 --2- 172.21.15.60:0/1354593481 >> v2:172.21.15.144:3300/0 conn(0x7f1f640d0be0 0x7f1f640cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.663+0000 7f1f5bfff640 1 -- 172.21.15.60:0/1354593481 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640d1740 con 0x7f1f640b8480 2023-10-16T14:13:03.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.664+0000 7f1f5bfff640 1 --2- 172.21.15.60:0/1354593481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640b8480 0x7f1f640cd7c0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f1f640b2520 tx=0x7f1f4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5d862762afc130 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:03.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.664+0000 7f1f61d9b640 1 -- 172.21.15.60:0/1354593481 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c031870 con 0x7f1f640b8480 2023-10-16T14:13:03.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f61d9b640 1 -- 172.21.15.60:0/1354593481 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c0319d0 con 0x7f1f640b8480 2023-10-16T14:13:03.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f61d9b640 1 -- 172.21.15.60:0/1354593481 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c031d60 con 0x7f1f640b8480 2023-10-16T14:13:03.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 -- 172.21.15.60:0/1354593481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640b8480 msgr2=0x7f1f640cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 --2- 172.21.15.60:0/1354593481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640b8480 0x7f1f640cd7c0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f1f640b2520 tx=0x7f1f4c02b910 comp rx=0 tx=0).stop 2023-10-16T14:13:03.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 -- 172.21.15.60:0/1354593481 shutdown_connections 2023-10-16T14:13:03.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 --2- 172.21.15.60:0/1354593481 >> v2:172.21.15.144:3300/0 conn(0x7f1f640d0be0 0x7f1f640cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 --2- 172.21.15.60:0/1354593481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640b8480 0x7f1f640cd7c0 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 --2- 172.21.15.60:0/1354593481 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f640d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 -- 172.21.15.60:0/1354593481 >> 172.21.15.60:0/1354593481 conn(0x7f1f640cf570 msgr2=0x7f1f640cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:03.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 -- 172.21.15.60:0/1354593481 shutdown_connections 2023-10-16T14:13:03.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.665+0000 7f1f6359e640 1 -- 172.21.15.60:0/1354593481 wait complete. 2023-10-16T14:13:03.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.667+0000 7f1f6359e640 1 Processor -- start 2023-10-16T14:13:03.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.667+0000 7f1f6359e640 1 -- start start 2023-10-16T14:13:03.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.668+0000 7f1f6359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f640b8480 0x7f1f64158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.668+0000 7f1f6359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640d0be0 0x7f1f640c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.668+0000 7f1f6359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f6415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.668+0000 7f1f6359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640b4200 con 0x7f1f640d0be0 2023-10-16T14:13:03.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.668+0000 7f1f6359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f641603f0 con 0x7f1f640b8480 2023-10-16T14:13:03.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.668+0000 7f1f6359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f641f00d0 con 0x7f1f6415cd70 2023-10-16T14:13:03.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f6415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f6415f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:45278/0 (socket says 172.21.15.60:45278) 2023-10-16T14:13:03.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 -- 172.21.15.60:0/775738452 learned_addr learned my addr 172.21.15.60:0/775738452 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:03.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f5bfff640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640d0be0 0x7f1f640c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f6259c640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f640b8480 0x7f1f64158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:03.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 -- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f640b8480 msgr2=0x7f1f64158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f640b8480 0x7f1f64158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 -- 172.21.15.60:0/775738452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640d0be0 msgr2=0x7f1f640c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640d0be0 0x7f1f640c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.669+0000 7f1f62d9d640 1 -- 172.21.15.60:0/775738452 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c0094d0 con 0x7f1f6415cd70 2023-10-16T14:13:03.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.670+0000 7f1f62d9d640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f6415f9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00efc0 tx=0x7f1f5c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:03.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.670+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c009b10 con 0x7f1f6415cd70 2023-10-16T14:13:03.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.670+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5c00f040 con 0x7f1f6415cd70 2023-10-16T14:13:03.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.670+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0146c0 con 0x7f1f6415cd70 2023-10-16T14:13:03.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.670+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f640b3860 con 0x7f1f6415cd70 2023-10-16T14:13:03.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.670+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f64159370 con 0x7f1f6415cd70 2023-10-16T14:13:03.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.671+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f640b4340 con 0x7f1f6415cd70 2023-10-16T14:13:03.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.673+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 4 ==== mgrmap(e 150) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f1f5c014820 con 0x7f1f6415cd70 2023-10-16T14:13:03.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.673+0000 7f1f5affd640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:03.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.674+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f1f5c08aea0 con 0x7f1f6415cd70 2023-10-16T14:13:03.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.674+0000 7f1f6259c640 1 -- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 msgr2=0x7f1f4406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:03.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.674+0000 7f1f6259c640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:03.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.686+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f5c055a50 con 0x7f1f6415cd70 2023-10-16T14:13:03.852 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:03.848+0000 7fb3b184f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:13:03.860 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:03.856+0000 7f5e45632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:13:03.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.875+0000 7f1f6259c640 1 -- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 msgr2=0x7f1f4406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/2906320450 2023-10-16T14:13:03.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.875+0000 7f1f6259c640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.881+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f640b4340 con 0x7f1f6415cd70 2023-10-16T14:13:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.884+0000 7f1f5affd640 1 -- 172.21.15.60:0/775738452 <== mon.2 v2:172.21.15.144:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) v1 ==== 74+0+17209 (secure 0 0 0) 0x7f1f640b4340 con 0x7f1f6415cd70 2023-10-16T14:13:03.889 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:03.889 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":274,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:12:55.110971+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/1984005880":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2098656759":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6833/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/307080156":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6832/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2842352837":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2156516012":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2942868570":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/786228305":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1402597431":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 msgr2=0x7f1f4406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 msgr2=0x7f1f6415f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:03.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f6415f9a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00efc0 tx=0x7f1f5c00c490 comp rx=0 tx=0).stop 2023-10-16T14:13:03.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 shutdown_connections 2023-10-16T14:13:03.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:6832/2906320450,v1:172.21.15.144:6833/2906320450] conn(0x7f1f4406b020 0x7f1f4406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1f6415cd70 0x7f1f6415f9a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f640d0be0 0x7f1f640c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 --2- 172.21.15.60:0/775738452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f640b8480 0x7f1f64158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:03.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 >> 172.21.15.60:0/775738452 conn(0x7f1f640cf570 msgr2=0x7f1f640b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:03.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.887+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 shutdown_connections 2023-10-16T14:13:03.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:03.888+0000 7f1f6359e640 1 -- 172.21.15.60:0/775738452 wait complete. 2023-10-16T14:13:03.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:12:37.972576+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:12:39.411671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '273', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:13:03.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:13:03.917 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:03.913+0000 7f5e45632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:13:04.036 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.032+0000 7f5e45632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:13:04.128 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.124+0000 7f5dc7258640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.128 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.124+0000 7f5dc7258640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.129 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.124+0000 7f5dc7258640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.129 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.124+0000 7f5dc7258640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.129 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.124+0000 7f5dc7258640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.125+0000 7f5dc9a5d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.125+0000 7f5dc9a5d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.125+0000 7f5dc9a5d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.125+0000 7f5dc9a5d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.130 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:04.125+0000 7f5dc9a5d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:04.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.132+0000 7f285759e640 1 Processor -- start 2023-10-16T14:13:04.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.133+0000 7f285759e640 1 -- start start 2023-10-16T14:13:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f285806c040 0x7f285806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f28580cda80 0x7f285815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285806c960 con 0x7f28580ad970 2023-10-16T14:13:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285815b9d0 con 0x7f285806c040 2023-10-16T14:13:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285759e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285815bcb0 con 0x7f28580cda80 2023-10-16T14:13:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f285806c040 0x7f285806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f285806c040 0x7f285806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53056/0 (socket says 172.21.15.60:53056) 2023-10-16T14:13:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f285659c640 1 -- 172.21.15.60:0/2032609997 learned_addr learned my addr 172.21.15.60:0/2032609997 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f2855d9b640 1 --1- 172.21.15.60:0/2032609997 >> v1:172.21.15.60:6789/0 conn(0x7f28580ad970 0x7f285815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56906/0 (socket says 172.21.15.60:56906) 2023-10-16T14:13:04.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.134+0000 7f2856d9d640 1 --2- 172.21.15.60:0/2032609997 >> v2:172.21.15.144:3300/0 conn(0x7f28580cda80 0x7f285815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.135+0000 7f285659c640 1 -- 172.21.15.60:0/2032609997 >> v2:172.21.15.144:3300/0 conn(0x7f28580cda80 msgr2=0x7f285815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:04.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.135+0000 7f285659c640 1 --2- 172.21.15.60:0/2032609997 >> v2:172.21.15.144:3300/0 conn(0x7f28580cda80 0x7f285815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.135+0000 7f285659c640 1 -- 172.21.15.60:0/2032609997 >> v1:172.21.15.60:6789/0 conn(0x7f28580ad970 legacy=0x7f285815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.135+0000 7f285659c640 1 -- 172.21.15.60:0/2032609997 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285815bf90 con 0x7f285806c040 2023-10-16T14:13:04.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.136+0000 7f285659c640 1 --2- 172.21.15.60:0/2032609997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285806c040 0x7f285806c420 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f28580cf300 tx=0x7f285002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2737ac24d361a06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:04.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.136+0000 7f285559a640 1 -- 172.21.15.60:0/2032609997 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850030870 con 0x7f285806c040 2023-10-16T14:13:04.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.136+0000 7f285559a640 1 -- 172.21.15.60:0/2032609997 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28500309d0 con 0x7f285806c040 2023-10-16T14:13:04.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.136+0000 7f285559a640 1 -- 172.21.15.60:0/2032609997 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850030d60 con 0x7f285806c040 2023-10-16T14:13:04.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 -- 172.21.15.60:0/2032609997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285806c040 msgr2=0x7f285806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 --2- 172.21.15.60:0/2032609997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285806c040 0x7f285806c420 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f28580cf300 tx=0x7f285002b9b0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 -- 172.21.15.60:0/2032609997 shutdown_connections 2023-10-16T14:13:04.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 --2- 172.21.15.60:0/2032609997 >> v2:172.21.15.144:3300/0 conn(0x7f28580cda80 0x7f285815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 --2- 172.21.15.60:0/2032609997 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285806c040 0x7f285806c420 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 -- 172.21.15.60:0/2032609997 >> 172.21.15.60:0/2032609997 conn(0x7f28580ce6a0 msgr2=0x7f28580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:04.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 -- 172.21.15.60:0/2032609997 shutdown_connections 2023-10-16T14:13:04.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.137+0000 7f285759e640 1 -- 172.21.15.60:0/2032609997 wait complete. 2023-10-16T14:13:04.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.139+0000 7f285759e640 1 Processor -- start 2023-10-16T14:13:04.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.139+0000 7f285759e640 1 -- start start 2023-10-16T14:13:04.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f285806c040 0x7f285815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 0x7f28581f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28580cda80 0x7f28581f5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28581f4d70 con 0x7f28580ad970 2023-10-16T14:13:04.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28581f40f0 con 0x7f28580cda80 2023-10-16T14:13:04.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28581f43d0 con 0x7f285806c040 2023-10-16T14:13:04.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f2855d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 0x7f28581f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f285659c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f285806c040 0x7f285815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f2855d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 0x7f28581f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53068/0 (socket says 172.21.15.60:53068) 2023-10-16T14:13:04.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f2855d9b640 1 -- 172.21.15.60:0/4246880756 learned_addr learned my addr 172.21.15.60:0/4246880756 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:04.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f2856d9d640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28580cda80 0x7f28581f5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f2855d9b640 1 -- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f285806c040 msgr2=0x7f285815c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.140+0000 7f2855d9b640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f285806c040 0x7f285815c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f2855d9b640 1 -- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28580cda80 msgr2=0x7f28581f5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f2855d9b640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28580cda80 0x7f28581f5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f2855d9b640 1 -- 172.21.15.60:0/4246880756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500094d0 con 0x7f28580ad970 2023-10-16T14:13:04.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f2856d9d640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28580cda80 0x7f28581f5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:04.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f2855d9b640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 0x7f28581f3750 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f284000c870 tx=0x7f284000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:04.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284000a640 con 0x7f28580ad970 2023-10-16T14:13:04.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284000a7a0 con 0x7f28580ad970 2023-10-16T14:13:04.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284000ab10 con 0x7f28580ad970 2023-10-16T14:13:04.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28581f46b0 con 0x7f28580ad970 2023-10-16T14:13:04.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.141+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285815cf20 con 0x7f28580ad970 2023-10-16T14:13:04.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.143+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285806b330 con 0x7f28580ad970 2023-10-16T14:13:04.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.143+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 151) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f284000ac70 con 0x7f28580ad970 2023-10-16T14:13:04.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.144+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f2840088c10 con 0x7f28580ad970 2023-10-16T14:13:04.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.148+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2840053d70 con 0x7f28580ad970 2023-10-16T14:13:04.186 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:04.183+0000 7fb3b184f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:13:04.246 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:04.243+0000 7fb3b184f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:13:04.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.340+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28581f4eb0 con 0x7f28580ad970 2023-10-16T14:13:04.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.342+0000 7f284effd640 1 -- 172.21.15.60:0/4246880756 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v275) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28581f4eb0 con 0x7f28580ad970 2023-10-16T14:13:04.346 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:13:04.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 msgr2=0x7f28581f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 0x7f28581f3750 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f284000c870 tx=0x7f284000cd30 comp rx=0 tx=0).stop 2023-10-16T14:13:04.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 shutdown_connections 2023-10-16T14:13:04.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28580cda80 0x7f28581f5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28580ad970 0x7f28581f3750 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 --2- 172.21.15.60:0/4246880756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f285806c040 0x7f285815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.344+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 >> 172.21.15.60:0/4246880756 conn(0x7f28580ce6a0 msgr2=0x7f28580ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:04.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.345+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 shutdown_connections 2023-10-16T14:13:04.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.345+0000 7f285759e640 1 -- 172.21.15.60:0/4246880756 wait complete. 2023-10-16T14:13:04.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:13:04.367 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:04.363+0000 7fb3b184f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:13:04.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.579+0000 7fdee4ae5640 1 Processor -- start 2023-10-16T14:13:04.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.579+0000 7fdee4ae5640 1 -- start start 2023-10-16T14:13:04.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdee4ae5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 0x7fdee0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdee4ae5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdee00caac0 0x7fdee006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdee4ae5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00cb030 con 0x7fdee00c9ca0 2023-10-16T14:13:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdee4ae5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee00c91c0 con 0x7fdee00caac0 2023-10-16T14:13:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdee4ae5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c94a0 con 0x7fdee015ad70 2023-10-16T14:13:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdeddd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdee00c9ca0 0x7fdee00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56908/0 (socket says 172.21.15.60:56908) 2023-10-16T14:13:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdeddd9b640 1 -- 172.21.15.60:0/2028057837 learned_addr learned my addr 172.21.15.60:0/2028057837 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdeded9d640 1 --2- 172.21.15.60:0/2028057837 >> v2:172.21.15.144:3300/0 conn(0x7fdee00caac0 0x7fdee006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.581+0000 7fdede59c640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 0x7fdee0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636405827 0 0) 0x7fdee00cb030 con 0x7fdee00c9ca0 2023-10-16T14:13:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec8003520 con 0x7fdee00c9ca0 2023-10-16T14:13:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1433702902 0 0) 0x7fdec8003520 con 0x7fdee00c9ca0 2023-10-16T14:13:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 msgr2=0x7fdee0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 0x7fdee0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 >> v2:172.21.15.144:3300/0 conn(0x7fdee00caac0 msgr2=0x7fdee006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:04.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 --2- 172.21.15.60:0/2028057837 >> v2:172.21.15.144:3300/0 conn(0x7fdee00caac0 0x7fdee006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00c9780 con 0x7fdee00c9ca0 2023-10-16T14:13:04.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fded4001670 con 0x7fdee00c9ca0 2023-10-16T14:13:04.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 >> v1:172.21.15.60:6789/0 conn(0x7fdee00c9ca0 legacy=0x7fdee00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec8003970 0x7fdec8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 0x7fdec80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdec8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c91c0 con 0x7fdec8004d00 2023-10-16T14:13:04.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c94a0 con 0x7fdec8003970 2023-10-16T14:13:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec800b980 con 0x7fdee015ad70 2023-10-16T14:13:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.582+0000 7fdeded9d640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdec8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdeded9d640 1 -- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 msgr2=0x7fdec80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdeddd9b640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 0x7fdec80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdede59c640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec8003970 0x7fdec8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdeded9d640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee015ad70 0x7fdec80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdeded9d640 1 -- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec8003970 msgr2=0x7fdec8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdeded9d640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec8003970 0x7fdec8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdeded9d640 1 -- 172.21.15.60:0/2028057837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8003520 con 0x7fdec8004d00 2023-10-16T14:13:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.583+0000 7fdede59c640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec8003970 0x7fdec8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.584+0000 7fdeded9d640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdec8005180 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fded8009fc0 tx=0x7fdec80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb1afb3ee6cfe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.584+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded802cc60 con 0x7fdec8004d00 2023-10-16T14:13:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.584+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded800f040 con 0x7fdec8004d00 2023-10-16T14:13:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.584+0000 7fdedd59a640 1 -- 172.21.15.60:0/2028057837 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8034440 con 0x7fdec8004d00 2023-10-16T14:13:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdee4ae5640 1 -- 172.21.15.60:0/2028057837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 msgr2=0x7fdec8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdee4ae5640 1 --2- 172.21.15.60:0/2028057837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdec8005180 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fded8009fc0 tx=0x7fdec80057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdeded9d640 1 -- 172.21.15.60:0/2028057837 reap_dead start 2023-10-16T14:13:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdee4ae5640 1 -- 172.21.15.60:0/2028057837 shutdown_connections 2023-10-16T14:13:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdee4ae5640 1 -- 172.21.15.60:0/2028057837 >> 172.21.15.60:0/2028057837 conn(0x7fdee00ce9b0 msgr2=0x7fdee00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdee4ae5640 1 -- 172.21.15.60:0/2028057837 shutdown_connections 2023-10-16T14:13:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.585+0000 7fdee4ae5640 1 -- 172.21.15.60:0/2028057837 wait complete. 2023-10-16T14:13:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.587+0000 7fdee4ae5640 1 Processor -- start 2023-10-16T14:13:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.587+0000 7fdee4ae5640 1 -- start start 2023-10-16T14:13:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdee4ae5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdee00c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdee4ae5640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee0071440 0x7fdee0157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdee4ae5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee00aec00 0x7fdee00af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdee4ae5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00af5f0 con 0x7fdec8004d00 2023-10-16T14:13:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdee4ae5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01f2810 con 0x7fdee00aec00 2023-10-16T14:13:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdee4ae5640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01f2af0 con 0x7fdee0071440 2023-10-16T14:13:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdede59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdee00c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdede59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdee00c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53082/0 (socket says 172.21.15.60:53082) 2023-10-16T14:13:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.588+0000 7fdede59c640 1 -- 172.21.15.60:0/1719047685 learned_addr learned my addr 172.21.15.60:0/1719047685 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdeded9d640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee00aec00 0x7fdee00af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdeddd9b640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee0071440 0x7fdee0157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdede59c640 1 -- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee0071440 msgr2=0x7fdee0157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdede59c640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee0071440 0x7fdee0157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdede59c640 1 -- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee00aec00 msgr2=0x7fdee00af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdede59c640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee00aec00 0x7fdee00af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdede59c640 1 -- 172.21.15.60:0/1719047685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee01f2dd0 con 0x7fdec8004d00 2023-10-16T14:13:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdeddd9b640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee0071440 0x7fdee0157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdeded9d640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee00aec00 0x7fdee00af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.589+0000 7fdede59c640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdee00c8ab0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7fded0010890 tx=0x7fded0010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.590+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded000eb00 con 0x7fdec8004d00 2023-10-16T14:13:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.590+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded000ec60 con 0x7fdec8004d00 2023-10-16T14:13:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.590+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded001c400 con 0x7fdec8004d00 2023-10-16T14:13:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.590+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee01f30b0 con 0x7fdec8004d00 2023-10-16T14:13:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.590+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee01f2580 con 0x7fdec8004d00 2023-10-16T14:13:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.591+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee01f32f0 con 0x7fdec8004d00 2023-10-16T14:13:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.592+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 151) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fded001c560 con 0x7fdec8004d00 2023-10-16T14:13:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.594+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7fded0089870 con 0x7fdec8004d00 2023-10-16T14:13:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.598+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdee01f32f0 con 0x7fdec8004d00 2023-10-16T14:13:04.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.795+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdee01f32f0 con 0x7fdec8004d00 2023-10-16T14:13:04.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.797+0000 7fdeceffd640 1 -- 172.21.15.60:0/1719047685 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v275) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdee01f32f0 con 0x7fdec8004d00 2023-10-16T14:13:04.800 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:13:04.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 msgr2=0x7fdee00c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:04.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdee00c8ab0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7fded0010890 tx=0x7fded0010d50 comp rx=0 tx=0).stop 2023-10-16T14:13:04.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 shutdown_connections 2023-10-16T14:13:04.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdee0071440 0x7fdee0157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee00aec00 0x7fdee00af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 --2- 172.21.15.60:0/1719047685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdec8004d00 0x7fdee00c8ab0 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 >> 172.21.15.60:0/1719047685 conn(0x7fdee00ce9b0 msgr2=0x7fdee00b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 shutdown_connections 2023-10-16T14:13:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:04.799+0000 7fdee4ae5640 1 -- 172.21.15.60:0/1719047685 wait complete. 2023-10-16T14:13:04.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:13:05.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.045+0000 7f95c3303640 1 Processor -- start 2023-10-16T14:13:05.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.045+0000 7f95c3303640 1 -- start start 2023-10-16T14:13:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c3303640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f95bc15b320 0x7f95bc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c3303640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc15b9d0 con 0x7f95bc06c040 2023-10-16T14:13:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c3303640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc15bcb0 con 0x7f95bc0ad970 2023-10-16T14:13:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c3303640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc15b7d0 con 0x7f95bc15b320 2023-10-16T14:13:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c2301640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f95bc06c040 0x7f95bc06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56920/0 (socket says 172.21.15.60:56920) 2023-10-16T14:13:05.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c2301640 1 -- 172.21.15.60:0/2899803762 learned_addr learned my addr 172.21.15.60:0/2899803762 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:05.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.047+0000 7f95c2b02640 1 --2- 172.21.15.60:0/2899803762 >> v2:172.21.15.144:3300/0 conn(0x7f95bc15b320 0x7f95bc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.048+0000 7f95c2b02640 1 -- 172.21.15.60:0/2899803762 >> v1:172.21.15.144:6789/0 conn(0x7f95bc0ad970 legacy=0x7f95bc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.048+0000 7f95c2b02640 1 -- 172.21.15.60:0/2899803762 >> v1:172.21.15.60:6789/0 conn(0x7f95bc06c040 legacy=0x7f95bc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.048+0000 7f95c2b02640 1 -- 172.21.15.60:0/2899803762 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc15bf90 con 0x7f95bc15b320 2023-10-16T14:13:05.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.050+0000 7f95c2b02640 1 --2- 172.21.15.60:0/2899803762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc15b320 0x7f95bc0cd180 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009dc0 tx=0x7f95b402bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=765f807f165de360 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.067+0000 7f95c12ff640 1 -- 172.21.15.60:0/2899803762 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b402ccd0 con 0x7f95bc15b320 2023-10-16T14:13:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.067+0000 7f95c12ff640 1 -- 172.21.15.60:0/2899803762 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b402ce30 con 0x7f95bc15b320 2023-10-16T14:13:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.067+0000 7f95c12ff640 1 -- 172.21.15.60:0/2899803762 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4034660 con 0x7f95bc15b320 2023-10-16T14:13:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.067+0000 7f95c3303640 1 -- 172.21.15.60:0/2899803762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc15b320 msgr2=0x7f95bc0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.067+0000 7f95c3303640 1 --2- 172.21.15.60:0/2899803762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc15b320 0x7f95bc0cd180 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009dc0 tx=0x7f95b402bf20 comp rx=0 tx=0).stop 2023-10-16T14:13:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.068+0000 7f95c3303640 1 -- 172.21.15.60:0/2899803762 shutdown_connections 2023-10-16T14:13:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.068+0000 7f95c3303640 1 --2- 172.21.15.60:0/2899803762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc15b320 0x7f95bc0cd180 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.068+0000 7f95c3303640 1 -- 172.21.15.60:0/2899803762 >> 172.21.15.60:0/2899803762 conn(0x7f95bc0ce6a0 msgr2=0x7f95bc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.068+0000 7f95c3303640 1 -- 172.21.15.60:0/2899803762 shutdown_connections 2023-10-16T14:13:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.068+0000 7f95c3303640 1 -- 172.21.15.60:0/2899803762 wait complete. 2023-10-16T14:13:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.070+0000 7f95c3303640 1 Processor -- start 2023-10-16T14:13:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.070+0000 7f95c3303640 1 -- start start 2023-10-16T14:13:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c3303640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06c040 0x7f95bc0ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c3303640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ad970 0x7f95bc0b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c3303640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 0x7f95bc1f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c3303640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc14d620 con 0x7f95bc15b320 2023-10-16T14:13:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c3303640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc0b06f0 con 0x7f95bc06c040 2023-10-16T14:13:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c3303640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1f6880 con 0x7f95bc0ad970 2023-10-16T14:13:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c2b02640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 0x7f95bc1f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c2b02640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 0x7f95bc1f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53094/0 (socket says 172.21.15.60:53094) 2023-10-16T14:13:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.071+0000 7f95c2b02640 1 -- 172.21.15.60:0/1190349369 learned_addr learned my addr 172.21.15.60:0/1190349369 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c1b00640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ad970 0x7f95bc0b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c2b02640 1 -- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ad970 msgr2=0x7f95bc0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c2b02640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ad970 0x7f95bc0b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c2b02640 1 -- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06c040 msgr2=0x7f95bc0ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c2b02640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06c040 0x7f95bc0ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c2b02640 1 -- 172.21.15.60:0/1190349369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b4009a70 con 0x7f95bc15b320 2023-10-16T14:13:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c2b02640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 0x7f95bc1f6340 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f95b402dfd0 tx=0x7f95b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b403d600 con 0x7f95bc15b320 2023-10-16T14:13:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b403d760 con 0x7f95bc15b320 2023-10-16T14:13:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b403db10 con 0x7f95bc15b320 2023-10-16T14:13:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.072+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc0afdc0 con 0x7f95bc15b320 2023-10-16T14:13:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.073+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc0b0380 con 0x7f95bc15b320 2023-10-16T14:13:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.074+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc06b330 con 0x7f95bc15b320 2023-10-16T14:13:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.075+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 152) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f95b403b040 con 0x7f95bc15b320 2023-10-16T14:13:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.075+0000 7f95b2ffd640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f95a006acc0 0x7f95a006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.076+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f95b40abce0 con 0x7f95bc15b320 2023-10-16T14:13:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.077+0000 7f95c2301640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f95a006acc0 0x7f95a006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.080+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95b4076ca0 con 0x7f95bc15b320 2023-10-16T14:13:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.080+0000 7f95c2301640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f95a006acc0 0x7f95a006d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0cdf20 tx=0x7f95b8007a40 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:05.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.269+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95bc0cc270 con 0x7f95bc15b320 2023-10-16T14:13:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.270+0000 7f95b2ffd640 1 -- 172.21.15.60:0/1190349369 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v275) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95b407a2c0 con 0x7f95bc15b320 2023-10-16T14:13:05.274 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:13:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f95a006acc0 msgr2=0x7f95a006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f95a006acc0 0x7f95a006d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0cdf20 tx=0x7f95b8007a40 comp rx=0 tx=0).stop 2023-10-16T14:13:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 msgr2=0x7f95bc1f6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 0x7f95bc1f6340 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f95b402dfd0 tx=0x7f95b4004a40 comp rx=0 tx=0).stop 2023-10-16T14:13:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 shutdown_connections 2023-10-16T14:13:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f95a006acc0 0x7f95a006d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f95bc0ad970 0x7f95bc0b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95bc15b320 0x7f95bc1f6340 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 --2- 172.21.15.60:0/1190349369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95bc06c040 0x7f95bc0ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.272+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 >> 172.21.15.60:0/1190349369 conn(0x7f95bc0ce6a0 msgr2=0x7f95bc0ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.273+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 shutdown_connections 2023-10-16T14:13:05.278 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.273+0000 7f95c3303640 1 -- 172.21.15.60:0/1190349369 wait complete. 2023-10-16T14:13:05.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.502+0000 7fb99ffff640 1 Processor -- start 2023-10-16T14:13:05.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.502+0000 7fb99ffff640 1 -- start start 2023-10-16T14:13:05.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.504+0000 7fb99ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a006c040 0x7fb9a006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.504+0000 7fb99ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9a00ad970 0x7fb9a015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.504+0000 7fb99ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9a015ebc0 0x7fb9a015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.504+0000 7fb99ffff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a006c960 con 0x7fb9a00ad970 2023-10-16T14:13:05.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.504+0000 7fb99ffff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a015b9d0 con 0x7fb9a015ebc0 2023-10-16T14:13:05.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.504+0000 7fb99ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a015bcb0 con 0x7fb9a006c040 2023-10-16T14:13:05.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9a00ad970 0x7fb9a015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9a00ad970 0x7fb9a015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53100/0 (socket says 172.21.15.60:53100) 2023-10-16T14:13:05.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 -- 172.21.15.60:0/3996200060 learned_addr learned my addr 172.21.15.60:0/3996200060 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb99effd640 1 --2- 172.21.15.60:0/3996200060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a006c040 0x7fb9a006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb99f7fe640 1 --2- 172.21.15.60:0/3996200060 >> v2:172.21.15.144:3300/0 conn(0x7fb9a015ebc0 0x7fb9a015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 -- 172.21.15.60:0/3996200060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a006c040 msgr2=0x7fb9a006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 --2- 172.21.15.60:0/3996200060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a006c040 0x7fb9a006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 -- 172.21.15.60:0/3996200060 >> v2:172.21.15.144:3300/0 conn(0x7fb9a015ebc0 msgr2=0x7fb9a015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:05.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 --2- 172.21.15.60:0/3996200060 >> v2:172.21.15.144:3300/0 conn(0x7fb9a015ebc0 0x7fb9a015f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.505+0000 7fb997fff640 1 -- 172.21.15.60:0/3996200060 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a015bf90 con 0x7fb9a00ad970 2023-10-16T14:13:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.506+0000 7fb997fff640 1 --2- 172.21.15.60:0/3996200060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a00ad970 0x7fb9a015e680 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fb988009870 tx=0x7fb98802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef1d782d3310852 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99e7fc640 1 -- 172.21.15.60:0/3996200060 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988031640 con 0x7fb9a00ad970 2023-10-16T14:13:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99e7fc640 1 -- 172.21.15.60:0/3996200060 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9880317a0 con 0x7fb9a00ad970 2023-10-16T14:13:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99e7fc640 1 -- 172.21.15.60:0/3996200060 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988031b30 con 0x7fb9a00ad970 2023-10-16T14:13:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 -- 172.21.15.60:0/3996200060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a00ad970 msgr2=0x7fb9a015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3996200060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a00ad970 0x7fb9a015e680 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fb988009870 tx=0x7fb98802b8f0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 -- 172.21.15.60:0/3996200060 shutdown_connections 2023-10-16T14:13:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3996200060 >> v2:172.21.15.144:3300/0 conn(0x7fb9a015ebc0 0x7fb9a015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3996200060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a00ad970 0x7fb9a015e680 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3996200060 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a006c040 0x7fb9a006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.507+0000 7fb99ffff640 1 -- 172.21.15.60:0/3996200060 >> 172.21.15.60:0/3996200060 conn(0x7fb9a00ce6a0 msgr2=0x7fb9a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.508+0000 7fb99ffff640 1 -- 172.21.15.60:0/3996200060 shutdown_connections 2023-10-16T14:13:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.508+0000 7fb99ffff640 1 -- 172.21.15.60:0/3996200060 wait complete. 2023-10-16T14:13:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.509+0000 7fb99ffff640 1 Processor -- start 2023-10-16T14:13:05.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.510+0000 7fb99ffff640 1 -- start start 2023-10-16T14:13:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.510+0000 7fb99ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9a006c040 0x7fb9a01e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.510+0000 7fb99ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a00ad970 0x7fb9a00c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.510+0000 7fb99ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 0x7fb9a00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99ffff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01e5870 con 0x7fb9a015ebc0 2023-10-16T14:13:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99ffff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01e5b50 con 0x7fb9a006c040 2023-10-16T14:13:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99ffff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a01e5e30 con 0x7fb9a00ad970 2023-10-16T14:13:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 0x7fb9a00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 0x7fb9a00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53116/0 (socket says 172.21.15.60:53116) 2023-10-16T14:13:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 -- 172.21.15.60:0/3870680279 learned_addr learned my addr 172.21.15.60:0/3870680279 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb997fff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a00ad970 0x7fb9a00c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99effd640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9a006c040 0x7fb9a01e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 -- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a00ad970 msgr2=0x7fb9a00c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a00ad970 0x7fb9a00c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 -- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9a006c040 msgr2=0x7fb9a01e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9a006c040 0x7fb9a01e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.511+0000 7fb99f7fe640 1 -- 172.21.15.60:0/3870680279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9880094d0 con 0x7fb9a015ebc0 2023-10-16T14:13:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.512+0000 7fb99f7fe640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 0x7fb9a00ac890 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fb998007e40 tx=0x7fb99800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.512+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998014620 con 0x7fb9a015ebc0 2023-10-16T14:13:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.512+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99800ee90 con 0x7fb9a015ebc0 2023-10-16T14:13:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.512+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9980149e0 con 0x7fb9a015ebc0 2023-10-16T14:13:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.512+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a01e6110 con 0x7fb9a015ebc0 2023-10-16T14:13:05.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.512+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a0147d80 con 0x7fb9a015ebc0 2023-10-16T14:13:05.522 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.513+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a006b330 con 0x7fb9a015ebc0 2023-10-16T14:13:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.515+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 152) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb998014b40 con 0x7fb9a015ebc0 2023-10-16T14:13:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.516+0000 7fb9977fe640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb98006acf0 0x7fb98006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.516+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7fb998089ff0 con 0x7fb9a015ebc0 2023-10-16T14:13:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.517+0000 7fb99effd640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb98006acf0 0x7fb98006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.518+0000 7fb99effd640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb98006acf0 0x7fb98006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb990006fd0 tx=0x7fb990008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.521+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9980bcfe0 con 0x7fb9a015ebc0 2023-10-16T14:13:05.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.756+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9a00c8d80 con 0x7fb9a015ebc0 2023-10-16T14:13:05.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.757+0000 7fb9977fe640 1 -- 172.21.15.60:0/3870680279 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4933 (secure 0 0 0) 0x7fb9a00c8d80 con 0x7fb9a015ebc0 2023-10-16T14:13:05.761 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:05.762 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:12:40.387056+0000","modified":"2023-10-16T14:12:43.460423+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12573":{"gid":12573,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2674455624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2674455624},{"type":"v1","addr":"172.21.15.60:6835","nonce":2674455624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T14:13:05.762 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 144 2023-10-16T14:13:05.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb98006acf0 msgr2=0x7fb98006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb98006acf0 0x7fb98006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb990006fd0 tx=0x7fb990008040 comp rx=0 tx=0).stop 2023-10-16T14:13:05.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 msgr2=0x7fb9a00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:05.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 0x7fb9a00ac890 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fb998007e40 tx=0x7fb99800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 shutdown_connections 2023-10-16T14:13:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb98006acf0 0x7fb98006d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb9a00ad970 0x7fb9a00c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a015ebc0 0x7fb9a00ac890 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 --2- 172.21.15.60:0/3870680279 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9a006c040 0x7fb9a01e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 >> 172.21.15.60:0/3870680279 conn(0x7fb9a00ce6a0 msgr2=0x7fb9a00cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 shutdown_connections 2023-10-16T14:13:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:05.760+0000 7fb99ffff640 1 -- 172.21.15.60:0/3870680279 wait complete. 2023-10-16T14:13:05.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:06.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.005+0000 7f2e43fff640 1 Processor -- start 2023-10-16T14:13:06.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.005+0000 7f2e43fff640 1 -- start start 2023-10-16T14:13:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.007+0000 7f2e43fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2e440ad970 0x7f2e440cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.007+0000 7f2e43fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4415b9d0 con 0x7f2e440ccfb0 2023-10-16T14:13:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.007+0000 7f2e43fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4415bcb0 con 0x7f2e4406c040 2023-10-16T14:13:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.007+0000 7f2e43fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4415bf90 con 0x7f2e440ad970 2023-10-16T14:13:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.007+0000 7f2e437fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2e440ccfb0 0x7f2e440cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56932/0 (socket says 172.21.15.60:56932) 2023-10-16T14:13:06.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.007+0000 7f2e437fe640 1 -- 172.21.15.60:0/1190969131 learned_addr learned my addr 172.21.15.60:0/1190969131 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.008+0000 7f2e427fc640 1 --2- 172.21.15.60:0/1190969131 >> v2:172.21.15.144:3300/0 conn(0x7f2e440ad970 0x7f2e440cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.008+0000 7f2e41ffb640 1 -- 172.21.15.60:0/1190969131 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804297503 0 0) 0x7f2e4415b9d0 con 0x7f2e440ccfb0 2023-10-16T14:13:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.008+0000 7f2e41ffb640 1 -- 172.21.15.60:0/1190969131 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2c003520 con 0x7f2e440ccfb0 2023-10-16T14:13:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.008+0000 7f2e427fc640 1 -- 172.21.15.60:0/1190969131 >> v1:172.21.15.144:6789/0 conn(0x7f2e4406c040 legacy=0x7f2e4406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.008+0000 7f2e427fc640 1 -- 172.21.15.60:0/1190969131 >> v1:172.21.15.60:6789/0 conn(0x7f2e440ccfb0 legacy=0x7f2e440cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.009+0000 7f2e427fc640 1 -- 172.21.15.60:0/1190969131 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e440cc270 con 0x7f2e440ad970 2023-10-16T14:13:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.010+0000 7f2e427fc640 1 --2- 172.21.15.60:0/1190969131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ad970 0x7f2e440cca70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2e380097d0 tx=0x7f2e3802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19cf13ae6cee5811 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e41ffb640 1 -- 172.21.15.60:0/1190969131 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38030800 con 0x7f2e440ad970 2023-10-16T14:13:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e41ffb640 1 -- 172.21.15.60:0/1190969131 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e38030960 con 0x7f2e440ad970 2023-10-16T14:13:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e41ffb640 1 -- 172.21.15.60:0/1190969131 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38030cd0 con 0x7f2e440ad970 2023-10-16T14:13:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e43fff640 1 -- 172.21.15.60:0/1190969131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ad970 msgr2=0x7f2e440cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e43fff640 1 --2- 172.21.15.60:0/1190969131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ad970 0x7f2e440cca70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2e380097d0 tx=0x7f2e3802b940 comp rx=0 tx=0).stop 2023-10-16T14:13:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e43fff640 1 -- 172.21.15.60:0/1190969131 shutdown_connections 2023-10-16T14:13:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e43fff640 1 --2- 172.21.15.60:0/1190969131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ad970 0x7f2e440cca70 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e43fff640 1 -- 172.21.15.60:0/1190969131 >> 172.21.15.60:0/1190969131 conn(0x7f2e440ce6a0 msgr2=0x7f2e440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.011+0000 7f2e43fff640 1 -- 172.21.15.60:0/1190969131 shutdown_connections 2023-10-16T14:13:06.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.012+0000 7f2e43fff640 1 -- 172.21.15.60:0/1190969131 wait complete. 2023-10-16T14:13:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.013+0000 7f2e43fff640 1 Processor -- start 2023-10-16T14:13:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.013+0000 7f2e43fff640 1 -- start start 2023-10-16T14:13:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.014+0000 7f2e43fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 0x7f2e440afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.014+0000 7f2e43fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e440ad970 0x7f2e4414e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.014+0000 7f2e43fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ccfb0 0x7f2e44152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.014+0000 7f2e43fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440b04d0 con 0x7f2e4406c040 2023-10-16T14:13:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.014+0000 7f2e43fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4414efd0 con 0x7f2e440ccfb0 2023-10-16T14:13:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.014+0000 7f2e43fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4414f2b0 con 0x7f2e440ad970 2023-10-16T14:13:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 0x7f2e440afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 0x7f2e440afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53128/0 (socket says 172.21.15.60:53128) 2023-10-16T14:13:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 -- 172.21.15.60:0/3477879624 learned_addr learned my addr 172.21.15.60:0/3477879624 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 -- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e440ad970 msgr2=0x7f2e4414e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e440ad970 0x7f2e4414e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 -- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ccfb0 msgr2=0x7f2e44152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e437fe640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ccfb0 0x7f2e44152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ccfb0 0x7f2e44152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 -- 172.21.15.60:0/3477879624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38009400 con 0x7f2e4406c040 2023-10-16T14:13:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e437fe640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ccfb0 0x7f2e44152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.015+0000 7f2e42ffd640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 0x7f2e440afb80 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c400 tx=0x7f2e3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.016+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e340052d0 con 0x7f2e4406c040 2023-10-16T14:13:06.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.016+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e34005430 con 0x7f2e4406c040 2023-10-16T14:13:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.016+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4414f590 con 0x7f2e4406c040 2023-10-16T14:13:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.016+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34003830 con 0x7f2e4406c040 2023-10-16T14:13:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.016+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e440b02a0 con 0x7f2e4406c040 2023-10-16T14:13:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.017+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4406b330 con 0x7f2e4406c040 2023-10-16T14:13:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.018+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 152) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2e34003990 con 0x7f2e4406c040 2023-10-16T14:13:06.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.019+0000 7f2e2bfff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2e3006acc0 0x7f2e3006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.019+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f2e34088c20 con 0x7f2e4406c040 2023-10-16T14:13:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.020+0000 7f2e427fc640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2e3006acc0 0x7f2e3006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.021+0000 7f2e427fc640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2e3006acc0 0x7f2e3006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e38036000 tx=0x7f2e38030000 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.023+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e34053be0 con 0x7f2e4406c040 2023-10-16T14:13:06.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.068+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 153) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2e34053fb0 con 0x7f2e4406c040 2023-10-16T14:13:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.232+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e4404e9b0 con 0x7f2e4406c040 2023-10-16T14:13:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.235+0000 7f2e2bfff640 1 -- 172.21.15.60:0/3477879624 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+17780 (secure 0 0 0) 0x7f2e34073270 con 0x7f2e4406c040 2023-10-16T14:13:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:06.239 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":275,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:04.037380+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3492229757":"2023-10-17T14:13:04.037340+0000","172.21.15.144:6833/2906320450":"2023-10-17T14:13:04.037340+0000","172.21.15.144:6832/2906320450":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1668072974":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/2110570505":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/405908562":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/2193300484":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1434044163":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/983873377":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1984005880":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2098656759":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6833/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/307080156":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6832/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2842352837":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2156516012":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2942868570":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/786228305":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1402597431":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2e3006acc0 msgr2=0x7f2e3006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2e3006acc0 0x7f2e3006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e38036000 tx=0x7f2e38030000 comp rx=0 tx=0).stop 2023-10-16T14:13:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 msgr2=0x7f2e440afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 0x7f2e440afb80 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c400 tx=0x7f2e3400c8c0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 shutdown_connections 2023-10-16T14:13:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2e3006acc0 0x7f2e3006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e440ad970 0x7f2e4414e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e440ccfb0 0x7f2e44152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 --2- 172.21.15.60:0/3477879624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e4406c040 0x7f2e440afb80 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.237+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 >> 172.21.15.60:0/3477879624 conn(0x7f2e440ce6a0 msgr2=0x7f2e440cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:06.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.238+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 shutdown_connections 2023-10-16T14:13:06.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.238+0000 7f2e43fff640 1 -- 172.21.15.60:0/3477879624 wait complete. 2023-10-16T14:13:06.261 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:13:06.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.459+0000 7f37e8421640 1 Processor -- start 2023-10-16T14:13:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.459+0000 7f37e8421640 1 -- start start 2023-10-16T14:13:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e8421640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37e015ae80 0x7f37e015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e8421640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e015ed60 con 0x7f37e015aaa0 2023-10-16T14:13:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e8421640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e00707e0 con 0x7f37e00ce390 2023-10-16T14:13:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e8421640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e006b810 con 0x7f37e015ae80 2023-10-16T14:13:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e61bd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f37e015aaa0 0x7f37e00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56942/0 (socket says 172.21.15.60:56942) 2023-10-16T14:13:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e61bd640 1 -- 172.21.15.60:0/3434134194 learned_addr learned my addr 172.21.15.60:0/3434134194 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.461+0000 7f37e59bc640 1 --2- 172.21.15.60:0/3434134194 >> v2:172.21.15.60:3300/0 conn(0x7f37e015ae80 0x7f37e015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.462+0000 7f37e51bb640 1 -- 172.21.15.60:0/3434134194 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371200053 0 0) 0x7f37e015ed60 con 0x7f37e015aaa0 2023-10-16T14:13:06.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.462+0000 7f37e51bb640 1 -- 172.21.15.60:0/3434134194 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc003520 con 0x7f37e015aaa0 2023-10-16T14:13:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.462+0000 7f37e59bc640 1 -- 172.21.15.60:0/3434134194 >> v1:172.21.15.144:6789/0 conn(0x7f37e00ce390 legacy=0x7f37e0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.462+0000 7f37e59bc640 1 -- 172.21.15.60:0/3434134194 >> v1:172.21.15.60:6789/0 conn(0x7f37e015aaa0 legacy=0x7f37e00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.462+0000 7f37e59bc640 1 -- 172.21.15.60:0/3434134194 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e015ef90 con 0x7f37e015ae80 2023-10-16T14:13:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.463+0000 7f37e59bc640 1 --2- 172.21.15.60:0/3434134194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e015e820 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f37d00097a0 tx=0x7f37d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1b0792c0b17a75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.463+0000 7f37e51bb640 1 -- 172.21.15.60:0/3434134194 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d0031830 con 0x7f37e015ae80 2023-10-16T14:13:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.463+0000 7f37e51bb640 1 -- 172.21.15.60:0/3434134194 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d0031990 con 0x7f37e015ae80 2023-10-16T14:13:06.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.463+0000 7f37e51bb640 1 -- 172.21.15.60:0/3434134194 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d0031d20 con 0x7f37e015ae80 2023-10-16T14:13:06.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 -- 172.21.15.60:0/3434134194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 msgr2=0x7f37e015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 --2- 172.21.15.60:0/3434134194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e015e820 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f37d00097a0 tx=0x7f37d002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 -- 172.21.15.60:0/3434134194 shutdown_connections 2023-10-16T14:13:06.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 --2- 172.21.15.60:0/3434134194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e015e820 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 -- 172.21.15.60:0/3434134194 >> 172.21.15.60:0/3434134194 conn(0x7f37e00cb410 msgr2=0x7f37e00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 -- 172.21.15.60:0/3434134194 shutdown_connections 2023-10-16T14:13:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.464+0000 7f37e8421640 1 -- 172.21.15.60:0/3434134194 wait complete. 2023-10-16T14:13:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.466+0000 7f37e8421640 1 Processor -- start 2023-10-16T14:13:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.466+0000 7f37e8421640 1 -- start start 2023-10-16T14:13:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.467+0000 7f37e8421640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e00ce390 0x7f37e01429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.467+0000 7f37e8421640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37e015aaa0 0x7f37e00c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.467+0000 7f37e8421640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e01440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.467+0000 7f37e8421640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0143c20 con 0x7f37e015ae80 2023-10-16T14:13:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.467+0000 7f37e8421640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0143010 con 0x7f37e00ce390 2023-10-16T14:13:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.467+0000 7f37e8421640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01432f0 con 0x7f37e015aaa0 2023-10-16T14:13:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e01440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e01440b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53152/0 (socket says 172.21.15.60:53152) 2023-10-16T14:13:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 -- 172.21.15.60:0/1159375385 learned_addr learned my addr 172.21.15.60:0/1159375385 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e59bc640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37e015aaa0 0x7f37e00c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e61bd640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e00ce390 0x7f37e01429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 -- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37e015aaa0 msgr2=0x7f37e00c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37e015aaa0 0x7f37e00c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 -- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e00ce390 msgr2=0x7f37e01429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e00ce390 0x7f37e01429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 -- 172.21.15.60:0/1159375385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d0009400 con 0x7f37e015ae80 2023-10-16T14:13:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e61bd640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e00ce390 0x7f37e01429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.468+0000 7f37e69be640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e01440b0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f37d800c780 tx=0x7f37d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.469+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d8004b60 con 0x7f37e015ae80 2023-10-16T14:13:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.469+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d800ce90 con 0x7f37e015ae80 2023-10-16T14:13:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.469+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d800a850 con 0x7f37e015ae80 2023-10-16T14:13:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.469+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e01435d0 con 0x7f37e015ae80 2023-10-16T14:13:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.469+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e00c43e0 con 0x7f37e015ae80 2023-10-16T14:13:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.470+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e00ade70 con 0x7f37e015ae80 2023-10-16T14:13:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.471+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 153) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f37d800a9b0 con 0x7f37e015ae80 2023-10-16T14:13:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.472+0000 7f37d6ffd640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f37c406af30 0x7f37c406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.472+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f37d8089bb0 con 0x7f37e015ae80 2023-10-16T14:13:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.473+0000 7f37e61bd640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f37c406af30 0x7f37c406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.474+0000 7f37e61bd640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f37c406af30 0x7f37c406d3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0011e0 tx=0x7f37dc009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.477+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f37d8054810 con 0x7f37e015ae80 2023-10-16T14:13:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.714+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37e004e160 con 0x7f37e015ae80 2023-10-16T14:13:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.715+0000 7f37d6ffd640 1 -- 172.21.15.60:0/1159375385 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f37d8057e30 con 0x7f37e015ae80 2023-10-16T14:13:06.719 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:06.719 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:12:40.387056+0000","modified":"2023-10-16T14:12:43.460423+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12573":{"gid":12573,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2674455624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2674455624},{"type":"v1","addr":"172.21.15.60:6835","nonce":2674455624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T14:13:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 144 2023-10-16T14:13:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f37c406af30 msgr2=0x7f37c406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f37c406af30 0x7f37c406d3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0011e0 tx=0x7f37dc009040 comp rx=0 tx=0).stop 2023-10-16T14:13:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 msgr2=0x7f37e01440b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e01440b0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f37d800c780 tx=0x7f37d800cc40 comp rx=0 tx=0).stop 2023-10-16T14:13:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 shutdown_connections 2023-10-16T14:13:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f37c406af30 0x7f37c406d3f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f37e015aaa0 0x7f37e00c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37e015ae80 0x7f37e01440b0 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 --2- 172.21.15.60:0/1159375385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e00ce390 0x7f37e01429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 >> 172.21.15.60:0/1159375385 conn(0x7f37e00cb410 msgr2=0x7f37e00b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 shutdown_connections 2023-10-16T14:13:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.718+0000 7f37e8421640 1 -- 172.21.15.60:0/1159375385 wait complete. 2023-10-16T14:13:06.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:06.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.959+0000 7fe1fa1e2640 1 Processor -- start 2023-10-16T14:13:06.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.960+0000 7fe1fa1e2640 1 -- start start 2023-10-16T14:13:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.961+0000 7fe1fa1e2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe1f40b4c80 0x7fe1f4167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.961+0000 7fe1fa1e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f40d15a0 con 0x7fe1f415cbe0 2023-10-16T14:13:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.961+0000 7fe1fa1e2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f40d1880 con 0x7fe1f415cfc0 2023-10-16T14:13:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.961+0000 7fe1fa1e2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f40d1b60 con 0x7fe1f40b4c80 2023-10-16T14:13:06.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.962+0000 7fe1f91e0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe1f415cbe0 0x7fe1f40d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56954/0 (socket says 172.21.15.60:56954) 2023-10-16T14:13:06.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.962+0000 7fe1f91e0640 1 -- 172.21.15.60:0/1625217907 learned_addr learned my addr 172.21.15.60:0/1625217907 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:06.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.962+0000 7fe1f99e1640 1 --2- 172.21.15.60:0/1625217907 >> v2:172.21.15.60:3300/0 conn(0x7fe1f40b4c80 0x7fe1f4167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.962+0000 7fe1f99e1640 1 -- 172.21.15.60:0/1625217907 >> v1:172.21.15.144:6789/0 conn(0x7fe1f415cfc0 legacy=0x7fe1f40b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.962+0000 7fe1f99e1640 1 -- 172.21.15.60:0/1625217907 >> v1:172.21.15.60:6789/0 conn(0x7fe1f415cbe0 legacy=0x7fe1f40d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.962+0000 7fe1f99e1640 1 -- 172.21.15.60:0/1625217907 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f40d1e40 con 0x7fe1f40b4c80 2023-10-16T14:13:06.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.964+0000 7fe1f99e1640 1 --2- 172.21.15.60:0/1625217907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f40b4c80 0x7fe1f4167910 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0097a0 tx=0x7fe1ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6461940b72f85d09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.964+0000 7fe1ebfff640 1 -- 172.21.15.60:0/1625217907 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec030610 con 0x7fe1f40b4c80 2023-10-16T14:13:06.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.964+0000 7fe1ebfff640 1 -- 172.21.15.60:0/1625217907 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec030770 con 0x7fe1f40b4c80 2023-10-16T14:13:06.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.964+0000 7fe1ebfff640 1 -- 172.21.15.60:0/1625217907 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec030b00 con 0x7fe1f40b4c80 2023-10-16T14:13:06.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1625217907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f40b4c80 msgr2=0x7fe1f4167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1625217907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f40b4c80 0x7fe1f4167910 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0097a0 tx=0x7fe1ec02b960 comp rx=0 tx=0).stop 2023-10-16T14:13:06.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1625217907 shutdown_connections 2023-10-16T14:13:06.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1625217907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f40b4c80 0x7fe1f4167910 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1625217907 >> 172.21.15.60:0/1625217907 conn(0x7fe1f40cf570 msgr2=0x7fe1f40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:06.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1625217907 shutdown_connections 2023-10-16T14:13:06.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.965+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1625217907 wait complete. 2023-10-16T14:13:06.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.967+0000 7fe1fa1e2640 1 Processor -- start 2023-10-16T14:13:06.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.967+0000 7fe1fa1e2640 1 -- start start 2023-10-16T14:13:06.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1fa1e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1f40b4c80 0x7fe1f415d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1fa1e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 0x7fe1f415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1fa1e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f415cfc0 0x7fe1f40c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1fa1e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415e9a0 con 0x7fe1f415cbe0 2023-10-16T14:13:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1fa1e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415dd20 con 0x7fe1f415cfc0 2023-10-16T14:13:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1fa1e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f415e000 con 0x7fe1f40b4c80 2023-10-16T14:13:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1f89df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 0x7fe1f415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1f89df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 0x7fe1f415edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53166/0 (socket says 172.21.15.60:53166) 2023-10-16T14:13:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1f89df640 1 -- 172.21.15.60:0/1979534795 learned_addr learned my addr 172.21.15.60:0/1979534795 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:06.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1f91e0640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1f40b4c80 0x7fe1f415d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1f99e1640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f415cfc0 0x7fe1f40c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.968+0000 7fe1f89df640 1 -- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1f40b4c80 msgr2=0x7fe1f415d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1f89df640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1f40b4c80 0x7fe1f415d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1f89df640 1 -- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f415cfc0 msgr2=0x7fe1f40c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:06.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1f89df640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f415cfc0 0x7fe1f40c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:06.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1f89df640 1 -- 172.21.15.60:0/1979534795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec009400 con 0x7fe1f415cbe0 2023-10-16T14:13:06.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1f99e1640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f415cfc0 0x7fe1f40c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:06.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1f89df640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 0x7fe1f415edc0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400c3c0 tx=0x7fe1e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40052d0 con 0x7fe1f415cbe0 2023-10-16T14:13:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e4005450 con 0x7fe1f415cbe0 2023-10-16T14:13:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4003830 con 0x7fe1f415cbe0 2023-10-16T14:13:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f415e2e0 con 0x7fe1f415cbe0 2023-10-16T14:13:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.969+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f41541c0 con 0x7fe1f415cbe0 2023-10-16T14:13:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.970+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f415eae0 con 0x7fe1f415cbe0 2023-10-16T14:13:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.971+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 153) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe1e4003990 con 0x7fe1f415cbe0 2023-10-16T14:13:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.972+0000 7fe1e9ffb640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe1d806af00 0x7fe1d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.972+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7fe1e4088f00 con 0x7fe1f415cbe0 2023-10-16T14:13:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.973+0000 7fe1f91e0640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe1d806af00 0x7fe1d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.975+0000 7fe1f91e0640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe1d806af00 0x7fe1d806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0005c00 tx=0x7fe1f0009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:06.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:06.978+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1f415eae0 con 0x7fe1f415cbe0 2023-10-16T14:13:07.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.214+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1f415eae0 con 0x7fe1f415cbe0 2023-10-16T14:13:07.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.215+0000 7fe1e9ffb640 1 -- 172.21.15.60:0/1979534795 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4933 (secure 0 0 0) 0x7fe1f415eae0 con 0x7fe1f415cbe0 2023-10-16T14:13:07.219 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:07.219 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":144,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:12:40.387056+0000","modified":"2023-10-16T14:12:43.460423+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12573":{"gid":12573,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2674455624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2674455624},{"type":"v1","addr":"172.21.15.60:6835","nonce":2674455624}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T14:13:07.220 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 144 2023-10-16T14:13:07.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.217+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe1d806af00 msgr2=0x7fe1d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.217+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe1d806af00 0x7fe1d806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0005c00 tx=0x7fe1f0009040 comp rx=0 tx=0).stop 2023-10-16T14:13:07.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 msgr2=0x7fe1f415edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 0x7fe1f415edc0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400c3c0 tx=0x7fe1e400c880 comp rx=0 tx=0).stop 2023-10-16T14:13:07.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 shutdown_connections 2023-10-16T14:13:07.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe1d806af00 0x7fe1d806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1f415cfc0 0x7fe1f40c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1f415cbe0 0x7fe1f415edc0 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 --2- 172.21.15.60:0/1979534795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe1f40b4c80 0x7fe1f415d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 >> 172.21.15.60:0/1979534795 conn(0x7fe1f40cf570 msgr2=0x7fe1f40b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 shutdown_connections 2023-10-16T14:13:07.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.218+0000 7fe1fa1e2640 1 -- 172.21.15.60:0/1979534795 wait complete. 2023-10-16T14:13:07.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.448+0000 7f2fd8863640 1 Processor -- start 2023-10-16T14:13:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.448+0000 7f2fd8863640 1 -- start start 2023-10-16T14:13:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd8863640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fd415cd70 0x7f2fd40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd8863640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2fd40cded0 0x7f2fd40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd8863640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd415d150 con 0x7f2fd40b8480 2023-10-16T14:13:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd8863640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd40d1180 con 0x7f2fd415cd70 2023-10-16T14:13:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd8863640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd40d1460 con 0x7f2fd40cded0 2023-10-16T14:13:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fd415cd70 0x7f2fd40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2fd415cd70 0x7f2fd40d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53182/0 (socket says 172.21.15.60:53182) 2023-10-16T14:13:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2fd40b8480 0x7f2fd40cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56968/0 (socket says 172.21.15.60:56968) 2023-10-16T14:13:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1730456143 learned_addr learned my addr 172.21.15.60:0/1730456143 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.450+0000 7f2fd359e640 1 --2- 172.21.15.60:0/1730456143 >> v2:172.21.15.144:3300/0 conn(0x7f2fd40cded0 0x7f2fd40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.451+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1730456143 >> v2:172.21.15.144:3300/0 conn(0x7f2fd40cded0 msgr2=0x7f2fd40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:07.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.451+0000 7f2fd2d9d640 1 --2- 172.21.15.60:0/1730456143 >> v2:172.21.15.144:3300/0 conn(0x7f2fd40cded0 0x7f2fd40b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.451+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1730456143 >> v1:172.21.15.60:6789/0 conn(0x7f2fd40b8480 legacy=0x7f2fd40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.451+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1730456143 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40d1740 con 0x7f2fd415cd70 2023-10-16T14:13:07.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.452+0000 7f2fd2d9d640 1 --2- 172.21.15.60:0/1730456143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd415cd70 0x7f2fd40d2220 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40b8910 tx=0x7f2fc402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7508073ef0091580 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:07.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.452+0000 7f2fd1d9b640 1 -- 172.21.15.60:0/1730456143 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4030860 con 0x7f2fd415cd70 2023-10-16T14:13:07.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.452+0000 7f2fd1d9b640 1 -- 172.21.15.60:0/1730456143 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc40309c0 con 0x7f2fd415cd70 2023-10-16T14:13:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.452+0000 7f2fd1d9b640 1 -- 172.21.15.60:0/1730456143 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4030d50 con 0x7f2fd415cd70 2023-10-16T14:13:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 -- 172.21.15.60:0/1730456143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd415cd70 msgr2=0x7f2fd40d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1730456143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd415cd70 0x7f2fd40d2220 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40b8910 tx=0x7f2fc402b9a0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 -- 172.21.15.60:0/1730456143 shutdown_connections 2023-10-16T14:13:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1730456143 >> v2:172.21.15.144:3300/0 conn(0x7f2fd40cded0 0x7f2fd40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1730456143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd415cd70 0x7f2fd40d2220 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 -- 172.21.15.60:0/1730456143 >> 172.21.15.60:0/1730456143 conn(0x7f2fd40cf570 msgr2=0x7f2fd40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:07.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 -- 172.21.15.60:0/1730456143 shutdown_connections 2023-10-16T14:13:07.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.453+0000 7f2fd8863640 1 -- 172.21.15.60:0/1730456143 wait complete. 2023-10-16T14:13:07.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.455+0000 7f2fd8863640 1 Processor -- start 2023-10-16T14:13:07.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.455+0000 7f2fd8863640 1 -- start start 2023-10-16T14:13:07.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd8863640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 0x7f2fd41f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd8863640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fd40cded0 0x7f2fd4158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd8863640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd415cd70 0x7f2fd41568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd8863640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd41f2030 con 0x7f2fd40b8480 2023-10-16T14:13:07.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd8863640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4159c10 con 0x7f2fd415cd70 2023-10-16T14:13:07.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd8863640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4159370 con 0x7f2fd40cded0 2023-10-16T14:13:07.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 0x7f2fd41f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 0x7f2fd41f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53190/0 (socket says 172.21.15.60:53190) 2023-10-16T14:13:07.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1975333355 learned_addr learned my addr 172.21.15.60:0/1975333355 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:07.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd359e640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd415cd70 0x7f2fd41568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fd40cded0 msgr2=0x7f2fd4158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fd40cded0 0x7f2fd4158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd415cd70 msgr2=0x7f2fd41568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd415cd70 0x7f2fd41568c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.456+0000 7f2fd2d9d640 1 -- 172.21.15.60:0/1975333355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40094d0 con 0x7f2fd40b8480 2023-10-16T14:13:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.457+0000 7f2fd2d9d640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 0x7f2fd41f1af0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f2fc402b9e0 tx=0x7f2fc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.457+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4030850 con 0x7f2fd40b8480 2023-10-16T14:13:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.457+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc4039d90 con 0x7f2fd40b8480 2023-10-16T14:13:07.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.457+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4043650 con 0x7f2fd40b8480 2023-10-16T14:13:07.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.457+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd4159650 con 0x7f2fd40b8480 2023-10-16T14:13:07.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.457+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd40cad40 con 0x7f2fd40b8480 2023-10-16T14:13:07.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.459+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd40caf80 con 0x7f2fd40b8480 2023-10-16T14:13:07.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.460+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 153) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2fc402f3b0 con 0x7f2fd40b8480 2023-10-16T14:13:07.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.460+0000 7f2fbb7fe640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2fc006af00 0x7f2fc006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.460+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f2fc40abf60 con 0x7f2fd40b8480 2023-10-16T14:13:07.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.461+0000 7f2fd259c640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2fc006af00 0x7f2fc006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.462+0000 7f2fd259c640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2fc006af00 0x7f2fc006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40ccc70 tx=0x7f2fc80079c0 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:07.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.465+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fc4076b40 con 0x7f2fd40b8480 2023-10-16T14:13:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.662+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fd404e9b0 con 0x7f2fd40b8480 2023-10-16T14:13:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.663+0000 7f2fbb7fe640 1 -- 172.21.15.60:0/1975333355 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+17780 (secure 0 0 0) 0x7f2fc4034090 con 0x7f2fd40b8480 2023-10-16T14:13:07.668 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:07.668 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":275,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:04.037380+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:12:37.972576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-16T14:12:39.411671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3492229757":"2023-10-17T14:13:04.037340+0000","172.21.15.144:6833/2906320450":"2023-10-17T14:13:04.037340+0000","172.21.15.144:6832/2906320450":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1668072974":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/2110570505":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/405908562":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/2193300484":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1434044163":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/983873377":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1984005880":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2098656759":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6833/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/307080156":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6832/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2842352837":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2156516012":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2942868570":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/786228305":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1402597431":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2fc006af00 msgr2=0x7f2fc006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2fc006af00 0x7f2fc006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40ccc70 tx=0x7f2fc80079c0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 msgr2=0x7f2fd41f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 0x7f2fd41f1af0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f2fc402b9e0 tx=0x7f2fc4004730 comp rx=0 tx=0).stop 2023-10-16T14:13:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 shutdown_connections 2023-10-16T14:13:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2fc006af00 0x7f2fc006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2fd40cded0 0x7f2fd4158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd415cd70 0x7f2fd41568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 --2- 172.21.15.60:0/1975333355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fd40b8480 0x7f2fd41f1af0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 >> 172.21.15.60:0/1975333355 conn(0x7f2fd40cf570 msgr2=0x7f2fd40b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 shutdown_connections 2023-10-16T14:13:07.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.666+0000 7f2fd8863640 1 -- 172.21.15.60:0/1975333355 wait complete. 2023-10-16T14:13:07.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:13:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.912+0000 7f693f301640 1 Processor -- start 2023-10-16T14:13:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.912+0000 7f693f301640 1 -- start start 2023-10-16T14:13:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.913+0000 7f693f301640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 0x7f69380caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.913+0000 7f693f301640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380c9000 0x7f6938069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.913+0000 7f693f301640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380cb140 con 0x7f693806a2e0 2023-10-16T14:13:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.913+0000 7f693f301640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6938161a50 con 0x7f69380addb0 2023-10-16T14:13:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.913+0000 7f693f301640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693806a750 con 0x7f69380c9000 2023-10-16T14:13:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.914+0000 7f693d09d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 0x7f69380caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.914+0000 7f693d09d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 0x7f69380caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:36084/0 (socket says 172.21.15.60:36084) 2023-10-16T14:13:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.914+0000 7f693d09d640 1 -- 172.21.15.60:0/3760326762 learned_addr learned my addr 172.21.15.60:0/3760326762 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.914+0000 7f693c89c640 1 --2- 172.21.15.60:0/3760326762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380c9000 0x7f6938069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.915+0000 7f693c89c640 1 -- 172.21.15.60:0/3760326762 >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 msgr2=0x7f69380caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.915+0000 7f693c89c640 1 --2- 172.21.15.60:0/3760326762 >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 0x7f69380caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.915+0000 7f693c89c640 1 -- 172.21.15.60:0/3760326762 >> v1:172.21.15.144:6789/0 conn(0x7f693806a2e0 legacy=0x7f69380cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.915+0000 7f693c89c640 1 -- 172.21.15.60:0/3760326762 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693806b030 con 0x7f69380c9000 2023-10-16T14:13:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.915+0000 7f693d09d640 1 --2- 172.21.15.60:0/3760326762 >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 0x7f69380caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.916+0000 7f693c89c640 1 --2- 172.21.15.60:0/3760326762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380c9000 0x7f6938069da0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f69280097a0 tx=0x7f692802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41dd2f14390821f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f692ffff640 1 -- 172.21.15.60:0/3760326762 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692802d040 con 0x7f69380c9000 2023-10-16T14:13:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f692ffff640 1 -- 172.21.15.60:0/3760326762 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6928031810 con 0x7f69380c9000 2023-10-16T14:13:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f692ffff640 1 -- 172.21.15.60:0/3760326762 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928031ba0 con 0x7f69380c9000 2023-10-16T14:13:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 -- 172.21.15.60:0/3760326762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380c9000 msgr2=0x7f6938069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 --2- 172.21.15.60:0/3760326762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380c9000 0x7f6938069da0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f69280097a0 tx=0x7f692802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 -- 172.21.15.60:0/3760326762 shutdown_connections 2023-10-16T14:13:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 --2- 172.21.15.60:0/3760326762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380c9000 0x7f6938069da0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 --2- 172.21.15.60:0/3760326762 >> v2:172.21.15.144:3300/0 conn(0x7f69380addb0 0x7f69380caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 -- 172.21.15.60:0/3760326762 >> 172.21.15.60:0/3760326762 conn(0x7f69380ce9b0 msgr2=0x7f69380cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 -- 172.21.15.60:0/3760326762 shutdown_connections 2023-10-16T14:13:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.917+0000 7f693f301640 1 -- 172.21.15.60:0/3760326762 wait complete. 2023-10-16T14:13:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.919+0000 7f693f301640 1 Processor -- start 2023-10-16T14:13:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.919+0000 7f693f301640 1 -- start start 2023-10-16T14:13:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.920+0000 7f693f301640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 0x7f6938164860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.920+0000 7f693f301640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380addb0 0x7f693815fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.920+0000 7f693f301640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69380c9000 0x7f69380afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.920+0000 7f693f301640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6938163db0 con 0x7f693806a2e0 2023-10-16T14:13:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.920+0000 7f693f301640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6938162d10 con 0x7f69380c9000 2023-10-16T14:13:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.920+0000 7f693f301640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6938162ff0 con 0x7f69380addb0 2023-10-16T14:13:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 0x7f6938164860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 0x7f6938164860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53206/0 (socket says 172.21.15.60:53206) 2023-10-16T14:13:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 -- 172.21.15.60:0/3301100572 learned_addr learned my addr 172.21.15.60:0/3301100572 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d89e640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69380c9000 0x7f69380afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 -- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380addb0 msgr2=0x7f693815fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380addb0 0x7f693815fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 -- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69380c9000 msgr2=0x7f69380afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69380c9000 0x7f69380afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 -- 172.21.15.60:0/3301100572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69340094d0 con 0x7f693806a2e0 2023-10-16T14:13:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d89e640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69380c9000 0x7f69380afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.921+0000 7f693d09d640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 0x7f6938164860 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f693400e9d0 tx=0x7f693400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:07.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.922+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693400ca60 con 0x7f693806a2e0 2023-10-16T14:13:07.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.922+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693400cbc0 con 0x7f693806a2e0 2023-10-16T14:13:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.922+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6928009400 con 0x7f693806a2e0 2023-10-16T14:13:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.922+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934013470 con 0x7f693806a2e0 2023-10-16T14:13:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.922+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69381635b0 con 0x7f693806a2e0 2023-10-16T14:13:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.923+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69380ae190 con 0x7f693806a2e0 2023-10-16T14:13:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.924+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 153) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f69340135d0 con 0x7f693806a2e0 2023-10-16T14:13:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.925+0000 7f692dffb640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f691c06af30 0x7f691c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.925+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6569+0+0 (secure 0 0 0) 0x7f6934089d90 con 0x7f693806a2e0 2023-10-16T14:13:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.926+0000 7f693c89c640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f691c06af30 0x7f691c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.927+0000 7f693c89c640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f691c06af30 0x7f691c06d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f69280097a0 tx=0x7f6928030000 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:07.929+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6934054a80 con 0x7f693806a2e0 2023-10-16T14:13:08.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.092+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6934054e50 con 0x7f693806a2e0 2023-10-16T14:13:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.175+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f693804e160 con 0x7f693806a2e0 2023-10-16T14:13:08.180 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:13:08.177+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:13:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.180+0000 7f692dffb640 1 -- 172.21.15.60:0/3301100572 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v145) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6934087b50 con 0x7f693806a2e0 2023-10-16T14:13:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:13:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f691c06af30 msgr2=0x7f691c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f691c06af30 0x7f691c06d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f69280097a0 tx=0x7f6928030000 comp rx=0 tx=0).stop 2023-10-16T14:13:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 msgr2=0x7f6938164860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 0x7f6938164860 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f693400e9d0 tx=0x7f693400ee90 comp rx=0 tx=0).stop 2023-10-16T14:13:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 shutdown_connections 2023-10-16T14:13:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f691c06af30 0x7f691c06d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f69380addb0 0x7f693815fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69380c9000 0x7f69380afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 --2- 172.21.15.60:0/3301100572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693806a2e0 0x7f6938164860 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.184+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 >> 172.21.15.60:0/3301100572 conn(0x7f69380ce9b0 msgr2=0x7f69380ae640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.185+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 shutdown_connections 2023-10-16T14:13:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.185+0000 7f693f301640 1 -- 172.21.15.60:0/3301100572 wait complete. 2023-10-16T14:13:08.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:13:08.218 INFO:tasks.ceph.mds.a.smithi060.stderr: -3978> 2023-10-16T14:12:46.426+0000 7f187850c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:13:08.282 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:13:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.406+0000 7fa61d305640 1 Processor -- start 2023-10-16T14:13:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.407+0000 7fa61d305640 1 -- start start 2023-10-16T14:13:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.408+0000 7fa61d305640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa6180caac0 0x7fa6180ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.408+0000 7fa61d305640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180cb030 con 0x7fa61806a4b0 2023-10-16T14:13:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.408+0000 7fa61d305640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180c91c0 con 0x7fa61815ad70 2023-10-16T14:13:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.408+0000 7fa61d305640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180c94a0 con 0x7fa6180caac0 2023-10-16T14:13:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.408+0000 7fa60ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa61806a4b0 0x7fa6180ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56978/0 (socket says 172.21.15.60:56978) 2023-10-16T14:13:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa60ffff640 1 -- 172.21.15.60:0/569367262 learned_addr learned my addr 172.21.15.60:0/569367262 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa6177fe640 1 --2- 172.21.15.60:0/569367262 >> v2:172.21.15.60:3300/0 conn(0x7fa6180caac0 0x7fa6180ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa6167fc640 1 -- 172.21.15.60:0/569367262 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134775411 0 0) 0x7fa6180cb030 con 0x7fa61806a4b0 2023-10-16T14:13:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa6167fc640 1 -- 172.21.15.60:0/569367262 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa604003520 con 0x7fa61806a4b0 2023-10-16T14:13:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa6177fe640 1 -- 172.21.15.60:0/569367262 >> v1:172.21.15.144:6789/0 conn(0x7fa61815ad70 legacy=0x7fa618069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa6177fe640 1 -- 172.21.15.60:0/569367262 >> v1:172.21.15.60:6789/0 conn(0x7fa61806a4b0 legacy=0x7fa6180ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.409+0000 7fa6177fe640 1 -- 172.21.15.60:0/569367262 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180c9780 con 0x7fa6180caac0 2023-10-16T14:13:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.410+0000 7fa6177fe640 1 --2- 172.21.15.60:0/569367262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6180caac0 0x7fa6180ad4d0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fa610009d90 tx=0x7fa61002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23406cc9a84b3634 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa6167fc640 1 -- 172.21.15.60:0/569367262 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61002cd00 con 0x7fa6180caac0 2023-10-16T14:13:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa6167fc640 1 -- 172.21.15.60:0/569367262 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61002ce60 con 0x7fa6180caac0 2023-10-16T14:13:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa6167fc640 1 -- 172.21.15.60:0/569367262 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610034660 con 0x7fa6180caac0 2023-10-16T14:13:08.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa61d305640 1 -- 172.21.15.60:0/569367262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6180caac0 msgr2=0x7fa6180ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa61d305640 1 --2- 172.21.15.60:0/569367262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6180caac0 0x7fa6180ad4d0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fa610009d90 tx=0x7fa61002bf50 comp rx=0 tx=0).stop 2023-10-16T14:13:08.417 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa61d305640 1 -- 172.21.15.60:0/569367262 shutdown_connections 2023-10-16T14:13:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa61d305640 1 --2- 172.21.15.60:0/569367262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6180caac0 0x7fa6180ad4d0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.411+0000 7fa61d305640 1 -- 172.21.15.60:0/569367262 >> 172.21.15.60:0/569367262 conn(0x7fa6180ce9b0 msgr2=0x7fa6180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.412+0000 7fa61d305640 1 -- 172.21.15.60:0/569367262 shutdown_connections 2023-10-16T14:13:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.412+0000 7fa61d305640 1 -- 172.21.15.60:0/569367262 wait complete. 2023-10-16T14:13:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.413+0000 7fa61d305640 1 Processor -- start 2023-10-16T14:13:08.419 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.413+0000 7fa61d305640 1 -- start start 2023-10-16T14:13:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.414+0000 7fa61d305640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 0x7fa618159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.414+0000 7fa61d305640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6180caac0 0x7fa6180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.414+0000 7fa61d305640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa61815ad70 0x7fa6180c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.414+0000 7fa61d305640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618159b50 con 0x7fa61806a4b0 2023-10-16T14:13:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa61d305640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c8120 con 0x7fa6180caac0 2023-10-16T14:13:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa61d305640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180ad190 con 0x7fa61815ad70 2023-10-16T14:13:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 0x7fa618159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 0x7fa618159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53234/0 (socket says 172.21.15.60:53234) 2023-10-16T14:13:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 -- 172.21.15.60:0/3592283359 learned_addr learned my addr 172.21.15.60:0/3592283359 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa60ffff640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6180caac0 0x7fa6180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa6177fe640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa61815ad70 0x7fa6180c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 -- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa61815ad70 msgr2=0x7fa6180c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa61815ad70 0x7fa6180c4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 -- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6180caac0 msgr2=0x7fa6180ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6180caac0 0x7fa6180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.415+0000 7fa616ffd640 1 -- 172.21.15.60:0/3592283359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610009a70 con 0x7fa61806a4b0 2023-10-16T14:13:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.416+0000 7fa616ffd640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 0x7fa618159610 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fa60800c400 tx=0x7fa60800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.416+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6080052d0 con 0x7fa61806a4b0 2023-10-16T14:13:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.416+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa608005430 con 0x7fa61806a4b0 2023-10-16T14:13:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.416+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608003830 con 0x7fa61806a4b0 2023-10-16T14:13:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.416+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6180ad470 con 0x7fa61806a4b0 2023-10-16T14:13:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.416+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6180c6950 con 0x7fa61806a4b0 2023-10-16T14:13:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.417+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61804e160 con 0x7fa61806a4b0 2023-10-16T14:13:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.418+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa608003990 con 0x7fa61806a4b0 2023-10-16T14:13:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.419+0000 7fa60f7fe640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa5f806af90 0x7fa5f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.419+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6655+0+0 (secure 0 0 0) 0x7fa608089140 con 0x7fa61806a4b0 2023-10-16T14:13:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.420+0000 7fa60ffff640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa5f806af90 0x7fa5f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.421+0000 7fa60ffff640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa5f806af90 0x7fa5f806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa61806f690 tx=0x7fa600009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:08.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.424+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa608053c90 con 0x7fa61806a4b0 2023-10-16T14:13:08.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:08.657+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa618069ff0 con 0x7fa61806a4b0 2023-10-16T14:13:09.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.197+0000 7fa60f7fe640 1 -- 172.21.15.60:0/3592283359 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v146) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6080572b0 con 0x7fa61806a4b0 2023-10-16T14:13:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.200+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa5f806af90 msgr2=0x7fa5f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.200+0000 7fa61d305640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa5f806af90 0x7fa5f806d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa61806f690 tx=0x7fa600009040 comp rx=0 tx=0).stop 2023-10-16T14:13:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.200+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 msgr2=0x7fa618159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.200+0000 7fa61d305640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 0x7fa618159610 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fa60800c400 tx=0x7fa60800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 shutdown_connections 2023-10-16T14:13:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa5f806af90 0x7fa5f806d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa61815ad70 0x7fa6180c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6180caac0 0x7fa6180ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 --2- 172.21.15.60:0/3592283359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa61806a4b0 0x7fa618159610 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 >> 172.21.15.60:0/3592283359 conn(0x7fa6180ce9b0 msgr2=0x7fa61815d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 shutdown_connections 2023-10-16T14:13:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.201+0000 7fa61d305640 1 -- 172.21.15.60:0/3592283359 wait complete. 2023-10-16T14:13:09.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:13:09.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.459+0000 7f934759e640 1 Processor -- start 2023-10-16T14:13:09.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.459+0000 7f934759e640 1 -- start start 2023-10-16T14:13:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f934759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9348159950 0x7f934815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f934759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934815aef0 con 0x7f934806c810 2023-10-16T14:13:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f934759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934815d910 con 0x7f9348070450 2023-10-16T14:13:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f934759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934815e490 con 0x7f9348159950 2023-10-16T14:13:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f9346d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f934806c810 0x7f93480c9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34322/0 (socket says 172.21.15.60:34322) 2023-10-16T14:13:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f9346d9d640 1 -- 172.21.15.60:0/1024362717 learned_addr learned my addr 172.21.15.60:0/1024362717 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.461+0000 7f934659c640 1 --2- 172.21.15.60:0/1024362717 >> v2:172.21.15.60:3300/0 conn(0x7f9348159950 0x7f934815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.462+0000 7f9345d9b640 1 -- 172.21.15.60:0/1024362717 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095023205 0 0) 0x7f9340001490 con 0x7f934806c810 2023-10-16T14:13:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.462+0000 7f9345d9b640 1 -- 172.21.15.60:0/1024362717 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334005730 con 0x7f934806c810 2023-10-16T14:13:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.462+0000 7f934659c640 1 -- 172.21.15.60:0/1024362717 >> v1:172.21.15.144:6789/0 conn(0x7f9348070450 legacy=0x7f934806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.462+0000 7f934659c640 1 -- 172.21.15.60:0/1024362717 >> v1:172.21.15.60:6789/0 conn(0x7f934806c810 legacy=0x7f93480c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.462+0000 7f934659c640 1 -- 172.21.15.60:0/1024362717 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934815dbf0 con 0x7f9348159950 2023-10-16T14:13:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.463+0000 7f934659c640 1 --2- 172.21.15.60:0/1024362717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348159950 0x7f934815a9b0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f93380097a0 tx=0x7f933802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d751e6412912b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.463+0000 7f9345d9b640 1 -- 172.21.15.60:0/1024362717 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338031830 con 0x7f9348159950 2023-10-16T14:13:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.463+0000 7f9345d9b640 1 -- 172.21.15.60:0/1024362717 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9338031990 con 0x7f9348159950 2023-10-16T14:13:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.463+0000 7f9345d9b640 1 -- 172.21.15.60:0/1024362717 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338031d20 con 0x7f9348159950 2023-10-16T14:13:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 -- 172.21.15.60:0/1024362717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348159950 msgr2=0x7f934815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 --2- 172.21.15.60:0/1024362717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348159950 0x7f934815a9b0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f93380097a0 tx=0x7f933802b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 -- 172.21.15.60:0/1024362717 shutdown_connections 2023-10-16T14:13:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 --2- 172.21.15.60:0/1024362717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348159950 0x7f934815a9b0 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 -- 172.21.15.60:0/1024362717 >> 172.21.15.60:0/1024362717 conn(0x7f93480ce6a0 msgr2=0x7f93480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 -- 172.21.15.60:0/1024362717 shutdown_connections 2023-10-16T14:13:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.464+0000 7f934759e640 1 -- 172.21.15.60:0/1024362717 wait complete. 2023-10-16T14:13:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.466+0000 7f934759e640 1 Processor -- start 2023-10-16T14:13:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.466+0000 7f934759e640 1 -- start start 2023-10-16T14:13:09.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.467+0000 7f934759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f934806c810 0x7f93481edb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:09.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.467+0000 7f934759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 0x7f93480c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.467+0000 7f934759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9348159950 0x7f93480c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.467+0000 7f934759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c2c30 con 0x7f9348070450 2023-10-16T14:13:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.467+0000 7f934759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c2f10 con 0x7f934806c810 2023-10-16T14:13:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.467+0000 7f934759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c31f0 con 0x7f9348159950 2023-10-16T14:13:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 0x7f93480c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 0x7f93480c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46458/0 (socket says 172.21.15.60:46458) 2023-10-16T14:13:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 -- 172.21.15.60:0/3246364125 learned_addr learned my addr 172.21.15.60:0/3246364125 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f934659c640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f934806c810 0x7f93481edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f9346d9d640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9348159950 0x7f93480c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 -- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9348159950 msgr2=0x7f93480c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9348159950 0x7f93480c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 -- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f934806c810 msgr2=0x7f93481edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f934806c810 0x7f93481edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.468+0000 7f933ffff640 1 -- 172.21.15.60:0/3246364125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338009400 con 0x7f9348070450 2023-10-16T14:13:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.469+0000 7f933ffff640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 0x7f93480c21b0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f933000c450 tx=0x7f933000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.469+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300052d0 con 0x7f9348070450 2023-10-16T14:13:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.469+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330005430 con 0x7f9348070450 2023-10-16T14:13:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.469+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93480c34d0 con 0x7f9348070450 2023-10-16T14:13:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.469+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330012870 con 0x7f9348070450 2023-10-16T14:13:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.469+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934814be40 con 0x7f9348070450 2023-10-16T14:13:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.470+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934804e9b0 con 0x7f9348070450 2023-10-16T14:13:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.471+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93300129d0 con 0x7f9348070450 2023-10-16T14:13:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.472+0000 7f933effd640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f932806af90 0x7f932806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.472+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6655+0+0 (secure 0 0 0) 0x7f9330089660 con 0x7f9348070450 2023-10-16T14:13:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.473+0000 7f934659c640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f932806af90 0x7f932806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.474+0000 7f934659c640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f932806af90 0x7f932806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9338036000 tx=0x7f933803c040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:09.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.477+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9330054250 con 0x7f9348070450 2023-10-16T14:13:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:09.672+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f93480c36c0 con 0x7f9348070450 2023-10-16T14:13:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.200+0000 7f933effd640 1 -- 172.21.15.60:0/3246364125 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v277) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9330057870 con 0x7f9348070450 2023-10-16T14:13:10.205 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:13:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.204+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f932806af90 msgr2=0x7f932806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.204+0000 7f934759e640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f932806af90 0x7f932806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9338036000 tx=0x7f933803c040 comp rx=0 tx=0).stop 2023-10-16T14:13:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.204+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 msgr2=0x7f93480c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.204+0000 7f934759e640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 0x7f93480c21b0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f933000c450 tx=0x7f933000c910 comp rx=0 tx=0).stop 2023-10-16T14:13:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.205+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 shutdown_connections 2023-10-16T14:13:10.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.205+0000 7f934759e640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f932806af90 0x7f932806d450 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9338036000 tx=0x7f933803c040 comp rx=0 tx=0).stop 2023-10-16T14:13:10.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.205+0000 7f934759e640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9348159950 0x7f93480c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.205+0000 7f934759e640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9348070450 0x7f93480c21b0 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.205+0000 7f934759e640 1 --2- 172.21.15.60:0/3246364125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f934806c810 0x7f93481edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.205+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 >> 172.21.15.60:0/3246364125 conn(0x7f93480ce6a0 msgr2=0x7f934815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:10.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.206+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 shutdown_connections 2023-10-16T14:13:10.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.207+0000 7f934759e640 1 -- 172.21.15.60:0/3246364125 wait complete. 2023-10-16T14:13:10.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:13:10.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.460+0000 7f6365cb4640 1 Processor -- start 2023-10-16T14:13:10.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.460+0000 7f6365cb4640 1 -- start start 2023-10-16T14:13:10.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.461+0000 7f6365cb4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 0x7f6360165aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:10.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.461+0000 7f6365cb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600b0360 0x7f63600b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:10.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.461+0000 7f6365cb4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6360165610 con 0x7f63600c1d30 2023-10-16T14:13:10.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.461+0000 7f6365cb4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600b0cb0 con 0x7f63600c8fa0 2023-10-16T14:13:10.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.461+0000 7f6365cb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600ce370 con 0x7f63600b0360 2023-10-16T14:13:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.462+0000 7f635effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 0x7f6360165aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.462+0000 7f635ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600b0360 0x7f63600b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.462+0000 7f635effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 0x7f6360165aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55390/0 (socket says 172.21.15.60:55390) 2023-10-16T14:13:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.462+0000 7f635effd640 1 -- 172.21.15.60:0/2511456826 learned_addr learned my addr 172.21.15.60:0/2511456826 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:10.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.463+0000 7f635ffff640 1 -- 172.21.15.60:0/2511456826 >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 msgr2=0x7f6360165aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:10.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.463+0000 7f635ffff640 1 --2- 172.21.15.60:0/2511456826 >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 0x7f6360165aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.463+0000 7f635ffff640 1 -- 172.21.15.60:0/2511456826 >> v1:172.21.15.144:6789/0 conn(0x7f63600c1d30 legacy=0x7f63600cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:10.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.463+0000 7f635ffff640 1 -- 172.21.15.60:0/2511456826 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600ce650 con 0x7f63600b0360 2023-10-16T14:13:10.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.463+0000 7f635effd640 1 --2- 172.21.15.60:0/2511456826 >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 0x7f6360165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:10.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.464+0000 7f635ffff640 1 --2- 172.21.15.60:0/2511456826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600b0360 0x7f63600b0770 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f63580097a0 tx=0x7f635802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d539eb8fc64c32d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:10.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.465+0000 7f635e7fc640 1 -- 172.21.15.60:0/2511456826 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635802d040 con 0x7f63600b0360 2023-10-16T14:13:10.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.465+0000 7f635e7fc640 1 -- 172.21.15.60:0/2511456826 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6358031810 con 0x7f63600b0360 2023-10-16T14:13:10.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.465+0000 7f635e7fc640 1 -- 172.21.15.60:0/2511456826 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358031ba0 con 0x7f63600b0360 2023-10-16T14:13:10.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 -- 172.21.15.60:0/2511456826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600b0360 msgr2=0x7f63600b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:10.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 --2- 172.21.15.60:0/2511456826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600b0360 0x7f63600b0770 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f63580097a0 tx=0x7f635802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 -- 172.21.15.60:0/2511456826 shutdown_connections 2023-10-16T14:13:10.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 --2- 172.21.15.60:0/2511456826 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600b0360 0x7f63600b0770 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 --2- 172.21.15.60:0/2511456826 >> v2:172.21.15.144:3300/0 conn(0x7f63600c8fa0 0x7f6360165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 -- 172.21.15.60:0/2511456826 >> 172.21.15.60:0/2511456826 conn(0x7f63600cb360 msgr2=0x7f63600cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:10.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 -- 172.21.15.60:0/2511456826 shutdown_connections 2023-10-16T14:13:10.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.466+0000 7f6365cb4640 1 -- 172.21.15.60:0/2511456826 wait complete. 2023-10-16T14:13:10.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.468+0000 7f6365cb4640 1 Processor -- start 2023-10-16T14:13:10.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.468+0000 7f6365cb4640 1 -- start start 2023-10-16T14:13:10.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.469+0000 7f6365cb4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600b0360 0x7f63600c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:10.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.469+0000 7f6365cb4640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600c1d30 0x7f6360161a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:10.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.469+0000 7f6365cb4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 0x7f6360161f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:10.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.469+0000 7f6365cb4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600c48d0 con 0x7f63600c8fa0 2023-10-16T14:13:10.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.469+0000 7f6365cb4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600c8800 con 0x7f63600b0360 2023-10-16T14:13:10.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.469+0000 7f6365cb4640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63601624a0 con 0x7f63600c1d30 2023-10-16T14:13:10.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 0x7f6360161f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:10.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 0x7f6360161f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46460/0 (socket says 172.21.15.60:46460) 2023-10-16T14:13:10.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 -- 172.21.15.60:0/1883315153 learned_addr learned my addr 172.21.15.60:0/1883315153 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:10.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635f7fe640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600b0360 0x7f63600c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:10.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635effd640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600c1d30 0x7f6360161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:10.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 -- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600c1d30 msgr2=0x7f6360161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:10.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600c1d30 0x7f6360161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 -- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600b0360 msgr2=0x7f63600c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:10.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600b0360 0x7f63600c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:10.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635ffff640 1 -- 172.21.15.60:0/1883315153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63540094d0 con 0x7f63600c8fa0 2023-10-16T14:13:10.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635f7fe640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600b0360 0x7f63600c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:10.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.470+0000 7f635effd640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600c1d30 0x7f6360161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:10.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.471+0000 7f635ffff640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 0x7f6360161f60 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f63580097a0 tx=0x7f63580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:10.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.471+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635803ac60 con 0x7f63600c8fa0 2023-10-16T14:13:10.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.471+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635803adc0 con 0x7f63600c8fa0 2023-10-16T14:13:10.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.471+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358044640 con 0x7f63600c8fa0 2023-10-16T14:13:10.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.471+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6358009400 con 0x7f63600c8fa0 2023-10-16T14:13:10.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.471+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63600c8520 con 0x7f63600c8fa0 2023-10-16T14:13:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.472+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63600c4a10 con 0x7f63600c8fa0 2023-10-16T14:13:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.473+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f635804d440 con 0x7f63600c8fa0 2023-10-16T14:13:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.474+0000 7f6347fff640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f634c06af90 0x7f634c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.474+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f63580abc80 con 0x7f63600c8fa0 2023-10-16T14:13:10.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.475+0000 7f635f7fe640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f634c06af90 0x7f634c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:10.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.476+0000 7f635f7fe640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f634c06af90 0x7f634c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6360163f90 tx=0x7f6350009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:10.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.479+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63600c4a10 con 0x7f63600c8fa0 2023-10-16T14:13:10.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:10.689+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f63600c4a10 con 0x7f63600c8fa0 2023-10-16T14:13:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.217+0000 7f6347fff640 1 -- 172.21.15.60:0/1883315153 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v278) v1 ==== 169+0+0 (secure 0 0 0) 0x7f63600c4a10 con 0x7f63600c8fa0 2023-10-16T14:13:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T14:13:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f634c06af90 msgr2=0x7f634c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f634c06af90 0x7f634c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6360163f90 tx=0x7f6350009040 comp rx=0 tx=0).stop 2023-10-16T14:13:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 msgr2=0x7f6360161f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 0x7f6360161f60 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f63580097a0 tx=0x7f63580046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 shutdown_connections 2023-10-16T14:13:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f634c06af90 0x7f634c06d450 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6360163f90 tx=0x7f6350009040 comp rx=0 tx=0).stop 2023-10-16T14:13:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f63600c1d30 0x7f6360161a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63600c8fa0 0x7f6360161f60 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 --2- 172.21.15.60:0/1883315153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63600b0360 0x7f63600c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.222+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 >> 172.21.15.60:0/1883315153 conn(0x7f63600cb360 msgr2=0x7f63600af850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.225+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 shutdown_connections 2023-10-16T14:13:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.225+0000 7f6365cb4640 1 -- 172.21.15.60:0/1883315153 wait complete. 2023-10-16T14:13:11.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:13:11.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.478+0000 7f53d221a640 1 Processor -- start 2023-10-16T14:13:11.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.478+0000 7f53d221a640 1 -- start start 2023-10-16T14:13:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.479+0000 7f53d221a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f53cc159ad0 0x7f53cc070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d221a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc0c9de0 0x7f53cc0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d221a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc0cd260 con 0x7f53cc15d650 2023-10-16T14:13:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d221a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53cc071470 con 0x7f53cc159ad0 2023-10-16T14:13:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d221a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc0ca7b0 con 0x7f53cc0c9de0 2023-10-16T14:13:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d1218640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f53cc159ad0 0x7f53cc070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d0a17640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f53cc15d650 0x7f53cc06aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34324/0 (socket says 172.21.15.60:34324) 2023-10-16T14:13:11.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d0a17640 1 -- 172.21.15.60:0/2906535477 learned_addr learned my addr 172.21.15.60:0/2906535477 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:11.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d1218640 1 -- 172.21.15.60:0/2906535477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc0c9de0 msgr2=0x7f53cc0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:11.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d1218640 1 --2- 172.21.15.60:0/2906535477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc0c9de0 0x7f53cc0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d1218640 1 -- 172.21.15.60:0/2906535477 >> v1:172.21.15.60:6789/0 conn(0x7f53cc15d650 legacy=0x7f53cc06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.480+0000 7f53d1218640 1 -- 172.21.15.60:0/2906535477 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc15de30 con 0x7f53cc159ad0 2023-10-16T14:13:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.482+0000 7f53d1218640 1 --2- 172.21.15.60:0/2906535477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc070f30 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f53c80097a0 tx=0x7f53c802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3432c8ab135e3b45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.482+0000 7f53c3fff640 1 -- 172.21.15.60:0/2906535477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c802d040 con 0x7f53cc159ad0 2023-10-16T14:13:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.482+0000 7f53c3fff640 1 -- 172.21.15.60:0/2906535477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c8031810 con 0x7f53cc159ad0 2023-10-16T14:13:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.482+0000 7f53c3fff640 1 -- 172.21.15.60:0/2906535477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8031ba0 con 0x7f53cc159ad0 2023-10-16T14:13:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 -- 172.21.15.60:0/2906535477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 msgr2=0x7f53cc070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 --2- 172.21.15.60:0/2906535477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc070f30 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f53c80097a0 tx=0x7f53c802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 -- 172.21.15.60:0/2906535477 shutdown_connections 2023-10-16T14:13:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 --2- 172.21.15.60:0/2906535477 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc0c9de0 0x7f53cc0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 --2- 172.21.15.60:0/2906535477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc070f30 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 -- 172.21.15.60:0/2906535477 >> 172.21.15.60:0/2906535477 conn(0x7f53cc0b0170 msgr2=0x7f53cc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:11.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 -- 172.21.15.60:0/2906535477 shutdown_connections 2023-10-16T14:13:11.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.483+0000 7f53d221a640 1 -- 172.21.15.60:0/2906535477 wait complete. 2023-10-16T14:13:11.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.486+0000 7f53d221a640 1 Processor -- start 2023-10-16T14:13:11.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.487+0000 7f53d221a640 1 -- start start 2023-10-16T14:13:11.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.488+0000 7f53d221a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53cc0c9de0 0x7f53cc167070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.488+0000 7f53d221a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.488+0000 7f53d221a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc15d650 0x7f53cc153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.488+0000 7f53d221a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc14c280 con 0x7f53cc159ad0 2023-10-16T14:13:11.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.488+0000 7f53d221a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc14ab30 con 0x7f53cc0c9de0 2023-10-16T14:13:11.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.488+0000 7f53d221a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc14ad40 con 0x7f53cc15d650 2023-10-16T14:13:11.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.489+0000 7f53d1a19640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc15d650 0x7f53cc153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.489+0000 7f53d0a17640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.489+0000 7f53d1a19640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc15d650 0x7f53cc153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:37380/0 (socket says 172.21.15.60:37380) 2023-10-16T14:13:11.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.489+0000 7f53d1a19640 1 -- 172.21.15.60:0/2177175548 learned_addr learned my addr 172.21.15.60:0/2177175548 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:11.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.489+0000 7f53d1218640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53cc0c9de0 0x7f53cc167070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.491+0000 7f53d0a17640 1 -- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc15d650 msgr2=0x7f53cc153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.491+0000 7f53d0a17640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc15d650 0x7f53cc153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.491+0000 7f53d0a17640 1 -- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53cc0c9de0 msgr2=0x7f53cc167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.491+0000 7f53d0a17640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53cc0c9de0 0x7f53cc167070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.491+0000 7f53d0a17640 1 -- 172.21.15.60:0/2177175548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c8009400 con 0x7f53cc159ad0 2023-10-16T14:13:11.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.492+0000 7f53d0a17640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc153910 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00c440 tx=0x7f53bc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:11.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.493+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc005ce0 con 0x7f53cc159ad0 2023-10-16T14:13:11.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.493+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53bc005e40 con 0x7f53cc159ad0 2023-10-16T14:13:11.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.493+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc0072d0 con 0x7f53cc159ad0 2023-10-16T14:13:11.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.493+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc14b020 con 0x7f53cc159ad0 2023-10-16T14:13:11.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.493+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc14b5e0 con 0x7f53cc159ad0 2023-10-16T14:13:11.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.496+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc14c3c0 con 0x7f53cc159ad0 2023-10-16T14:13:11.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.497+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f53bc0074f0 con 0x7f53cc159ad0 2023-10-16T14:13:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.500+0000 7f53c1ffb640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53b006b060 0x7f53b006d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.500+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f53bc088cc0 con 0x7f53cc159ad0 2023-10-16T14:13:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.501+0000 7f53d1218640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53b006b060 0x7f53b006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.503+0000 7f53d1218640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53b006b060 0x7f53b006d520 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f53c8036000 tx=0x7f53c803c040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:11.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.504+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53cc14c3c0 con 0x7f53cc159ad0 2023-10-16T14:13:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.700+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f53cc14c3c0 con 0x7f53cc159ad0 2023-10-16T14:13:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.705+0000 7f53c1ffb640 1 -- 172.21.15.60:0/2177175548 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7f53cc14c3c0 con 0x7f53cc159ad0 2023-10-16T14:13:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53b006b060 msgr2=0x7f53b006d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53b006b060 0x7f53b006d520 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f53c8036000 tx=0x7f53c803c040 comp rx=0 tx=0).stop 2023-10-16T14:13:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 msgr2=0x7f53cc153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc153910 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00c440 tx=0x7f53bc001e70 comp rx=0 tx=0).stop 2023-10-16T14:13:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 shutdown_connections 2023-10-16T14:13:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53b006b060 0x7f53b006d520 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f53cc15d650 0x7f53cc153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53cc159ad0 0x7f53cc153910 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 --2- 172.21.15.60:0/2177175548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f53cc0c9de0 0x7f53cc167070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 >> 172.21.15.60:0/2177175548 conn(0x7f53cc0b0170 msgr2=0x7f53cc0af230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 shutdown_connections 2023-10-16T14:13:11.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.709+0000 7f53d221a640 1 -- 172.21.15.60:0/2177175548 wait complete. 2023-10-16T14:13:11.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:13:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.957+0000 7fa0e6516640 1 Processor -- start 2023-10-16T14:13:11.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.957+0000 7fa0e6516640 1 -- start start 2023-10-16T14:13:11.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e6516640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa0e006a4b0 0x7fa0e00ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e6516640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa0e00c9ca0 0x7fa0e00ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e6516640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00cb030 con 0x7fa0e015ad70 2023-10-16T14:13:11.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e6516640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e00cae30 con 0x7fa0e00c9ca0 2023-10-16T14:13:11.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e6516640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e00c91c0 con 0x7fa0e006a4b0 2023-10-16T14:13:11.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e4ab3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa0e00c9ca0 0x7fa0e00ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.959+0000 7fa0e4ab3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa0e00c9ca0 0x7fa0e00ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46480/0 (socket says 172.21.15.60:46480) 2023-10-16T14:13:11.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.960+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/464954083 learned_addr learned my addr 172.21.15.60:0/464954083 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:11.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.960+0000 7fa0d77fe640 1 --2- 172.21.15.60:0/464954083 >> v2:172.21.15.144:3300/0 conn(0x7fa0e006a4b0 0x7fa0e00ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.960+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/464954083 >> v2:172.21.15.144:3300/0 conn(0x7fa0e006a4b0 msgr2=0x7fa0e00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.960+0000 7fa0e4ab3640 1 --2- 172.21.15.60:0/464954083 >> v2:172.21.15.144:3300/0 conn(0x7fa0e006a4b0 0x7fa0e00ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.960+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/464954083 >> v1:172.21.15.144:6789/0 conn(0x7fa0e015ad70 legacy=0x7fa0e0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.960+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/464954083 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00c94a0 con 0x7fa0e00c9ca0 2023-10-16T14:13:11.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.961+0000 7fa0e4ab3640 1 --2- 172.21.15.60:0/464954083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e00c9ca0 0x7fa0e00ca8f0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00ce870 tx=0x7fa0d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a7ba6e7ee43479 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.961+0000 7fa0df7fe640 1 -- 172.21.15.60:0/464954083 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8031870 con 0x7fa0e00c9ca0 2023-10-16T14:13:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0df7fe640 1 -- 172.21.15.60:0/464954083 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d80319d0 con 0x7fa0e00c9ca0 2023-10-16T14:13:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0df7fe640 1 -- 172.21.15.60:0/464954083 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8031d60 con 0x7fa0e00c9ca0 2023-10-16T14:13:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 -- 172.21.15.60:0/464954083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e00c9ca0 msgr2=0x7fa0e00ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 --2- 172.21.15.60:0/464954083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e00c9ca0 0x7fa0e00ca8f0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00ce870 tx=0x7fa0d802b910 comp rx=0 tx=0).stop 2023-10-16T14:13:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 -- 172.21.15.60:0/464954083 shutdown_connections 2023-10-16T14:13:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 --2- 172.21.15.60:0/464954083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e00c9ca0 0x7fa0e00ca8f0 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 --2- 172.21.15.60:0/464954083 >> v2:172.21.15.144:3300/0 conn(0x7fa0e006a4b0 0x7fa0e00ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 -- 172.21.15.60:0/464954083 >> 172.21.15.60:0/464954083 conn(0x7fa0e00ce9b0 msgr2=0x7fa0e00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.962+0000 7fa0e6516640 1 -- 172.21.15.60:0/464954083 shutdown_connections 2023-10-16T14:13:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.963+0000 7fa0e6516640 1 -- 172.21.15.60:0/464954083 wait complete. 2023-10-16T14:13:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.964+0000 7fa0e6516640 1 Processor -- start 2023-10-16T14:13:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.964+0000 7fa0e6516640 1 -- start start 2023-10-16T14:13:11.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.965+0000 7fa0e6516640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0e006a4b0 0x7fa0e0152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e6516640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0e00c9ca0 0x7fa0e00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e6516640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 0x7fa0e00c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e6516640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e00ae230 con 0x7fa0e015ad70 2023-10-16T14:13:11.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e6516640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e00ad190 con 0x7fa0e006a4b0 2023-10-16T14:13:11.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e6516640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e00ad470 con 0x7fa0e00c9ca0 2023-10-16T14:13:11.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 0x7fa0e00c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 0x7fa0e00c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46496/0 (socket says 172.21.15.60:46496) 2023-10-16T14:13:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/386333556 learned_addr learned my addr 172.21.15.60:0/386333556 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0d77fe640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0e00c9ca0 0x7fa0e00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0dffff640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0e006a4b0 0x7fa0e0152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0e00c9ca0 msgr2=0x7fa0e00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0e00c9ca0 0x7fa0e00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0e006a4b0 msgr2=0x7fa0e0152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0e006a4b0 0x7fa0e0152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.966+0000 7fa0e4ab3640 1 -- 172.21.15.60:0/386333556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d80094d0 con 0x7fa0e015ad70 2023-10-16T14:13:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.967+0000 7fa0e4ab3640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 0x7fa0e00c4940 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fa0d802b950 tx=0x7fa0d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.967+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d80443f0 con 0x7fa0e015ad70 2023-10-16T14:13:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.967+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d8044570 con 0x7fa0e015ad70 2023-10-16T14:13:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.967+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8044920 con 0x7fa0e015ad70 2023-10-16T14:13:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.967+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e00ad750 con 0x7fa0e015ad70 2023-10-16T14:13:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.967+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e00add10 con 0x7fa0e015ad70 2023-10-16T14:13:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.969+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e004e160 con 0x7fa0e015ad70 2023-10-16T14:13:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.969+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0d8044a80 con 0x7fa0e015ad70 2023-10-16T14:13:11.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.970+0000 7fa0dd7fa640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa0c006afc0 0x7fa0c006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:11.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.970+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7fa0d802d040 con 0x7fa0e015ad70 2023-10-16T14:13:11.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.971+0000 7fa0dffff640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa0c006afc0 0x7fa0c006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:11.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.972+0000 7fa0dffff640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa0c006afc0 0x7fa0c006d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009f60 tx=0x7fa0d00094f0 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:11.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:11.975+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0d8077050 con 0x7fa0e015ad70 2023-10-16T14:13:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.173+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa0e00af950 con 0x7fa0e015ad70 2023-10-16T14:13:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.181+0000 7fa0dd7fa640 1 -- 172.21.15.60:0/386333556 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v92) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa0d807a670 con 0x7fa0e015ad70 2023-10-16T14:13:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.183+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa0c006afc0 msgr2=0x7fa0c006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.183+0000 7fa0e6516640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa0c006afc0 0x7fa0c006d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009f60 tx=0x7fa0d00094f0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.183+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 msgr2=0x7fa0e00c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.183+0000 7fa0e6516640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 0x7fa0e00c4940 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fa0d802b950 tx=0x7fa0d80046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 shutdown_connections 2023-10-16T14:13:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa0c006afc0 0x7fa0c006d480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa0e00c9ca0 0x7fa0e00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0e015ad70 0x7fa0e00c4940 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 --2- 172.21.15.60:0/386333556 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0e006a4b0 0x7fa0e0152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 >> 172.21.15.60:0/386333556 conn(0x7fa0e00ce9b0 msgr2=0x7fa0e00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 shutdown_connections 2023-10-16T14:13:12.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.184+0000 7fa0e6516640 1 -- 172.21.15.60:0/386333556 wait complete. 2023-10-16T14:13:12.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:13:12.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.427+0000 7f360f59e640 1 Processor -- start 2023-10-16T14:13:12.431 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.427+0000 7f360f59e640 1 -- start start 2023-10-16T14:13:12.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f360f59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36100ad970 0x7f36100cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f360f59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f36100ccfb0 0x7f36100cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f360f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361015b9d0 con 0x7f361006c040 2023-10-16T14:13:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f360f59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361015bcb0 con 0x7f36100ad970 2023-10-16T14:13:12.433 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f360f59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361015bf90 con 0x7f36100ccfb0 2023-10-16T14:13:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f3607fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36100ad970 0x7f36100cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f3607fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36100ad970 0x7f36100cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46500/0 (socket says 172.21.15.60:46500) 2023-10-16T14:13:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.429+0000 7f3607fff640 1 -- 172.21.15.60:0/41656423 learned_addr learned my addr 172.21.15.60:0/41656423 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.430+0000 7f360ed9d640 1 --2- 172.21.15.60:0/41656423 >> v2:172.21.15.144:3300/0 conn(0x7f36100ccfb0 0x7f36100cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.435 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.430+0000 7f3607fff640 1 -- 172.21.15.60:0/41656423 >> v2:172.21.15.144:3300/0 conn(0x7f36100ccfb0 msgr2=0x7f36100cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.430+0000 7f3607fff640 1 --2- 172.21.15.60:0/41656423 >> v2:172.21.15.144:3300/0 conn(0x7f36100ccfb0 0x7f36100cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.430+0000 7f3607fff640 1 -- 172.21.15.60:0/41656423 >> v1:172.21.15.144:6789/0 conn(0x7f361006c040 legacy=0x7f361006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.430+0000 7f3607fff640 1 -- 172.21.15.60:0/41656423 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100cc270 con 0x7f36100ad970 2023-10-16T14:13:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.431+0000 7f3607fff640 1 --2- 172.21.15.60:0/41656423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100cca70 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009870 tx=0x7f35f802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b427f5608628088 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:12.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.431+0000 7f360dd9b640 1 -- 172.21.15.60:0/41656423 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8031860 con 0x7f36100ad970 2023-10-16T14:13:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.431+0000 7f360dd9b640 1 -- 172.21.15.60:0/41656423 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f80319c0 con 0x7f36100ad970 2023-10-16T14:13:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.431+0000 7f360dd9b640 1 -- 172.21.15.60:0/41656423 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8031d50 con 0x7f36100ad970 2023-10-16T14:13:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 -- 172.21.15.60:0/41656423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 msgr2=0x7f36100cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 --2- 172.21.15.60:0/41656423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100cca70 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009870 tx=0x7f35f802b900 comp rx=0 tx=0).stop 2023-10-16T14:13:12.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 -- 172.21.15.60:0/41656423 shutdown_connections 2023-10-16T14:13:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 --2- 172.21.15.60:0/41656423 >> v2:172.21.15.144:3300/0 conn(0x7f36100ccfb0 0x7f36100cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 --2- 172.21.15.60:0/41656423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100cca70 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 -- 172.21.15.60:0/41656423 >> 172.21.15.60:0/41656423 conn(0x7f36100ce6a0 msgr2=0x7f36100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 -- 172.21.15.60:0/41656423 shutdown_connections 2023-10-16T14:13:12.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.432+0000 7f360f59e640 1 -- 172.21.15.60:0/41656423 wait complete. 2023-10-16T14:13:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.434+0000 7f360f59e640 1 Processor -- start 2023-10-16T14:13:12.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.434+0000 7f360f59e640 1 -- start start 2023-10-16T14:13:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f360f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f361006c040 0x7f361014ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f360f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f360f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36100ccfb0 0x7f36101567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f360f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36100b2b80 con 0x7f36100ad970 2023-10-16T14:13:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f360f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36100c5b30 con 0x7f361006c040 2023-10-16T14:13:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f360f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36100c5e10 con 0x7f36100ccfb0 2023-10-16T14:13:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f3607fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f3607fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46504/0 (socket says 172.21.15.60:46504) 2023-10-16T14:13:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.435+0000 7f3607fff640 1 -- 172.21.15.60:0/883038724 learned_addr learned my addr 172.21.15.60:0/883038724 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f360e59c640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f361006c040 0x7f361014ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f360ed9d640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36100ccfb0 0x7f36101567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3607fff640 1 -- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36100ccfb0 msgr2=0x7f36101567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3607fff640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36100ccfb0 0x7f36101567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3607fff640 1 -- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f361006c040 msgr2=0x7f361014ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3607fff640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f361006c040 0x7f361014ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3607fff640 1 -- 172.21.15.60:0/883038724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80094d0 con 0x7f36100ad970 2023-10-16T14:13:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f360e59c640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f361006c040 0x7f361014ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:12.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f360ed9d640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36100ccfb0 0x7f36101567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3607fff640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100c7290 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009870 tx=0x7f35f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f803ac60 con 0x7f36100ad970 2023-10-16T14:13:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f803adc0 con 0x7f36100ad970 2023-10-16T14:13:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8044610 con 0x7f36100ad970 2023-10-16T14:13:12.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.436+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36100c60f0 con 0x7f36100ad970 2023-10-16T14:13:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.437+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3610151f40 con 0x7f36100ad970 2023-10-16T14:13:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.438+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361006b330 con 0x7f36100ad970 2023-10-16T14:13:12.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.439+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35f804d450 con 0x7f36100ad970 2023-10-16T14:13:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.440+0000 7f3606ffd640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f35f006afc0 0x7f35f006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.440+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f35f80abea0 con 0x7f36100ad970 2023-10-16T14:13:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.440+0000 7f360e59c640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f35f006afc0 0x7f35f006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.441+0000 7f360e59c640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f35f006afc0 0x7f35f006d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3600003eb0 tx=0x7f3600009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:12.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.444+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35f8076cd0 con 0x7f36100ad970 2023-10-16T14:13:12.649 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.645+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f36100c6610 con 0x7f36100ad970 2023-10-16T14:13:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.653+0000 7f3606ffd640 1 -- 172.21.15.60:0/883038724 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) v1 ==== 120+0+0 (secure 0 0 0) 0x7f36100c6610 con 0x7f36100ad970 2023-10-16T14:13:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f35f006afc0 msgr2=0x7f35f006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f35f006afc0 0x7f35f006d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3600003eb0 tx=0x7f3600009040 comp rx=0 tx=0).stop 2023-10-16T14:13:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 msgr2=0x7f36100c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100c7290 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009870 tx=0x7f35f80046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 shutdown_connections 2023-10-16T14:13:12.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f35f006afc0 0x7f35f006d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f36100ccfb0 0x7f36101567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36100ad970 0x7f36100c7290 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 --2- 172.21.15.60:0/883038724 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f361006c040 0x7f361014ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 >> 172.21.15.60:0/883038724 conn(0x7f36100ce6a0 msgr2=0x7f361015dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:12.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 shutdown_connections 2023-10-16T14:13:12.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.656+0000 7f360f59e640 1 -- 172.21.15.60:0/883038724 wait complete. 2023-10-16T14:13:12.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T14:13:12.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.902+0000 7f4d13fff640 1 Processor -- start 2023-10-16T14:13:12.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.902+0000 7f4d13fff640 1 -- start start 2023-10-16T14:13:12.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.904+0000 7f4d13fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d14070d70 0x7f4d1406bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.904+0000 7f4d13fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 0x7f4d1406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.904+0000 7f4d13fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1406c9c0 con 0x7f4d1415aad0 2023-10-16T14:13:12.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.904+0000 7f4d13fff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d140b4620 con 0x7f4d140712c0 2023-10-16T14:13:12.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.904+0000 7f4d13fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d140b4340 con 0x7f4d14070d70 2023-10-16T14:13:12.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.905+0000 7f4d137fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 0x7f4d1406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.905+0000 7f4d137fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 0x7f4d1406c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:55458/0 (socket says 172.21.15.60:55458) 2023-10-16T14:13:12.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.905+0000 7f4d137fe640 1 -- 172.21.15.60:0/1039779842 learned_addr learned my addr 172.21.15.60:0/1039779842 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:12.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.905+0000 7f4d0bfff640 1 --2- 172.21.15.60:0/1039779842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d14070d70 0x7f4d1406bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d127fc640 1 -- 172.21.15.60:0/1039779842 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181372001 0 0) 0x7f4d1406c9c0 con 0x7f4d1415aad0 2023-10-16T14:13:12.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d127fc640 1 -- 172.21.15.60:0/1039779842 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d00003520 con 0x7f4d1415aad0 2023-10-16T14:13:12.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d0bfff640 1 -- 172.21.15.60:0/1039779842 >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 msgr2=0x7f4d1406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:12.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d0bfff640 1 --2- 172.21.15.60:0/1039779842 >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 0x7f4d1406c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d0bfff640 1 -- 172.21.15.60:0/1039779842 >> v1:172.21.15.144:6789/0 conn(0x7f4d1415aad0 legacy=0x7f4d140cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d0bfff640 1 -- 172.21.15.60:0/1039779842 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1406f520 con 0x7f4d14070d70 2023-10-16T14:13:12.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.906+0000 7f4d137fe640 1 --2- 172.21.15.60:0/1039779842 >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 0x7f4d1406c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.907+0000 7f4d0bfff640 1 --2- 172.21.15.60:0/1039779842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d14070d70 0x7f4d1406bf40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0097a0 tx=0x7f4cfc02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4d0103b1c769ee5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d127fc640 1 -- 172.21.15.60:0/1039779842 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc030800 con 0x7f4d14070d70 2023-10-16T14:13:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d127fc640 1 -- 172.21.15.60:0/1039779842 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cfc030960 con 0x7f4d14070d70 2023-10-16T14:13:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d127fc640 1 -- 172.21.15.60:0/1039779842 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc030cf0 con 0x7f4d14070d70 2023-10-16T14:13:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 -- 172.21.15.60:0/1039779842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d14070d70 msgr2=0x7f4d1406bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1039779842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d14070d70 0x7f4d1406bf40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0097a0 tx=0x7f4cfc02b940 comp rx=0 tx=0).stop 2023-10-16T14:13:12.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 -- 172.21.15.60:0/1039779842 shutdown_connections 2023-10-16T14:13:12.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1039779842 >> v2:172.21.15.144:3300/0 conn(0x7f4d140712c0 0x7f4d1406c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1039779842 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d14070d70 0x7f4d1406bf40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 -- 172.21.15.60:0/1039779842 >> 172.21.15.60:0/1039779842 conn(0x7f4d140b0a90 msgr2=0x7f4d140ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:12.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 -- 172.21.15.60:0/1039779842 shutdown_connections 2023-10-16T14:13:12.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.908+0000 7f4d13fff640 1 -- 172.21.15.60:0/1039779842 wait complete. 2023-10-16T14:13:12.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.910+0000 7f4d13fff640 1 Processor -- start 2023-10-16T14:13:12.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.910+0000 7f4d13fff640 1 -- start start 2023-10-16T14:13:12.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.911+0000 7f4d13fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 0x7f4d140c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.911+0000 7f4d13fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d140712c0 0x7f4d141f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.911+0000 7f4d13fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d1415aad0 0x7f4d141585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.911+0000 7f4d13fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d14152420 con 0x7f4d14070d70 2023-10-16T14:13:12.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.911+0000 7f4d13fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d141f1160 con 0x7f4d1415aad0 2023-10-16T14:13:12.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.911+0000 7f4d13fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d140c2cd0 con 0x7f4d140712c0 2023-10-16T14:13:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 0x7f4d140c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 0x7f4d140c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46514/0 (socket says 172.21.15.60:46514) 2023-10-16T14:13:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 -- 172.21.15.60:0/1606927949 learned_addr learned my addr 172.21.15.60:0/1606927949 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d137fe640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d1415aad0 0x7f4d141585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d0bfff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d140712c0 0x7f4d141f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 -- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d140712c0 msgr2=0x7f4d141f0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d140712c0 0x7f4d141f0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 -- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d1415aad0 msgr2=0x7f4d141585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d1415aad0 0x7f4d141585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 -- 172.21.15.60:0/1606927949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c009b30 con 0x7f4d14070d70 2023-10-16T14:13:12.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d137fe640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d1415aad0 0x7f4d141585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.912+0000 7f4d12ffd640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 0x7f4d140c48c0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400c490 tx=0x7f4d0400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.913+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d040052d0 con 0x7f4d14070d70 2023-10-16T14:13:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.913+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d04005430 con 0x7f4d14070d70 2023-10-16T14:13:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.913+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc009400 con 0x7f4d14070d70 2023-10-16T14:13:12.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.913+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04012870 con 0x7f4d14070d70 2023-10-16T14:13:12.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.913+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d140c3250 con 0x7f4d14070d70 2023-10-16T14:13:12.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.914+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1404e9b0 con 0x7f4d14070d70 2023-10-16T14:13:12.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.915+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4d040129d0 con 0x7f4d14070d70 2023-10-16T14:13:12.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.917+0000 7f4d0b7fe640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f4cf406afc0 0x7f4cf406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:12.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.917+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f4d040893f0 con 0x7f4d14070d70 2023-10-16T14:13:12.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.918+0000 7f4d0bfff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f4cf406afc0 0x7f4cf406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:12.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.919+0000 7f4d0bfff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f4cf406afc0 0x7f4cf406d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc036000 tx=0x7f4cfc03b040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:12.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:12.920+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d04054320 con 0x7f4d14070d70 2023-10-16T14:13:13.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.186+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f4d140c33e0 con 0x7f4d14070d70 2023-10-16T14:13:13.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.222+0000 7f4d0b7fe640 1 -- 172.21.15.60:0/1606927949 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3324) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4d04057940 con 0x7f4d14070d70 2023-10-16T14:13:13.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f4cf406afc0 msgr2=0x7f4cf406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f4cf406afc0 0x7f4cf406d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc036000 tx=0x7f4cfc03b040 comp rx=0 tx=0).stop 2023-10-16T14:13:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 msgr2=0x7f4d140c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 0x7f4d140c48c0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400c490 tx=0x7f4d0400c950 comp rx=0 tx=0).stop 2023-10-16T14:13:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 shutdown_connections 2023-10-16T14:13:13.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f4cf406afc0 0x7f4cf406d480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4d140712c0 0x7f4d141f0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d1415aad0 0x7f4d141585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 --2- 172.21.15.60:0/1606927949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d14070d70 0x7f4d140c48c0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 >> 172.21.15.60:0/1606927949 conn(0x7f4d140b0a90 msgr2=0x7f4d140cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 shutdown_connections 2023-10-16T14:13:13.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.226+0000 7f4d13fff640 1 -- 172.21.15.60:0/1606927949 wait complete. 2023-10-16T14:13:13.250 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-16T14:13:13.251 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol and group arguments to snap-schedule cli. 2023-10-16T14:13:13.251 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T14:13:13.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:13.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.476+0000 7f8b4d0ec640 1 Processor -- start 2023-10-16T14:13:13.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.477+0000 7f8b4d0ec640 1 -- start start 2023-10-16T14:13:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.478+0000 7f8b4d0ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b4815ae80 0x7f8b4815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.478+0000 7f8b4d0ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b480ce390 0x7f8b48070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.478+0000 7f8b4d0ec640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4815ed60 con 0x7f8b4815aaa0 2023-10-16T14:13:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.478+0000 7f8b4d0ec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4806b810 con 0x7f8b4815ae80 2023-10-16T14:13:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.478+0000 7f8b4d0ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480707e0 con 0x7f8b480ce390 2023-10-16T14:13:13.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.479+0000 7f8b3ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b4815ae80 0x7f8b4815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.479+0000 7f8b4759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b480ce390 0x7f8b48070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.479+0000 7f8b46d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f8b4815aaa0 0x7f8b480cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:51456/0 (socket says 172.21.15.60:51456) 2023-10-16T14:13:13.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.479+0000 7f8b46d9d640 1 -- 172.21.15.60:0/2975144885 learned_addr learned my addr 172.21.15.60:0/2975144885 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:13.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b4659c640 1 -- 172.21.15.60:0/2975144885 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270317572 0 0) 0x7f8b4815ed60 con 0x7f8b4815aaa0 2023-10-16T14:13:13.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b4659c640 1 -- 172.21.15.60:0/2975144885 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b34003520 con 0x7f8b4815aaa0 2023-10-16T14:13:13.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b3ffff640 1 -- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b480ce390 msgr2=0x7f8b48070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:13.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b3ffff640 1 --2- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b480ce390 0x7f8b48070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b3ffff640 1 -- 172.21.15.60:0/2975144885 >> v1:172.21.15.144:6789/0 conn(0x7f8b4815aaa0 legacy=0x7f8b480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b3ffff640 1 -- 172.21.15.60:0/2975144885 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4815ef90 con 0x7f8b4815ae80 2023-10-16T14:13:13.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.480+0000 7f8b4759e640 1 --2- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b480ce390 0x7f8b48070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:13.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.482+0000 7f8b3ffff640 1 --2- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815ae80 0x7f8b4815e820 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bd83e31f45757bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:13.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.482+0000 7f8b4659c640 1 -- 172.21.15.60:0/2975144885 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30031830 con 0x7f8b4815ae80 2023-10-16T14:13:13.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4659c640 1 -- 172.21.15.60:0/2975144885 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30031990 con 0x7f8b4815ae80 2023-10-16T14:13:13.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4659c640 1 -- 172.21.15.60:0/2975144885 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30031d20 con 0x7f8b4815ae80 2023-10-16T14:13:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815ae80 msgr2=0x7f8b4815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815ae80 0x7f8b4815e820 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b3002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2975144885 shutdown_connections 2023-10-16T14:13:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b480ce390 0x7f8b48070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2975144885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815ae80 0x7f8b4815e820 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2975144885 >> 172.21.15.60:0/2975144885 conn(0x7f8b480cb410 msgr2=0x7f8b480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:13.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2975144885 shutdown_connections 2023-10-16T14:13:13.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.483+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2975144885 wait complete. 2023-10-16T14:13:13.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.485+0000 7f8b4d0ec640 1 Processor -- start 2023-10-16T14:13:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.485+0000 7f8b4d0ec640 1 -- start start 2023-10-16T14:13:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.486+0000 7f8b4d0ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 0x7f8b481590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.486+0000 7f8b4d0ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815aaa0 0x7f8b480a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.486+0000 7f8b4d0ec640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b4815ae80 0x7f8b480a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b4d0ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b48159630 con 0x7f8b480ce390 2023-10-16T14:13:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b4d0ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4815c430 con 0x7f8b4815aaa0 2023-10-16T14:13:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b4d0ec640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c45f0 con 0x7f8b4815ae80 2023-10-16T14:13:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 0x7f8b481590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 0x7f8b481590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46516/0 (socket says 172.21.15.60:46516) 2023-10-16T14:13:13.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 -- 172.21.15.60:0/2877770283 learned_addr learned my addr 172.21.15.60:0/2877770283 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:13.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b3ffff640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815aaa0 0x7f8b480a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 -- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b4815ae80 msgr2=0x7f8b480a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b4815ae80 0x7f8b480a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 -- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815aaa0 msgr2=0x7f8b480a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815aaa0 0x7f8b480a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b46d9d640 1 -- 172.21.15.60:0/2877770283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b40009a70 con 0x7f8b480ce390 2023-10-16T14:13:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.487+0000 7f8b3ffff640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815aaa0 0x7f8b480a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.488+0000 7f8b46d9d640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 0x7f8b481590f0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f8b38001eb0 tx=0x7f8b3800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.488+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38005ce0 con 0x7f8b480ce390 2023-10-16T14:13:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.488+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38005e40 con 0x7f8b480ce390 2023-10-16T14:13:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.488+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380072d0 con 0x7f8b480ce390 2023-10-16T14:13:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.488+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b30009400 con 0x7f8b480ce390 2023-10-16T14:13:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.488+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b480c4030 con 0x7f8b480ce390 2023-10-16T14:13:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.489+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4804e160 con 0x7f8b480ce390 2023-10-16T14:13:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.490+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b380074b0 con 0x7f8b480ce390 2023-10-16T14:13:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.491+0000 7f8b3f7fe640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8b2806af90 0x7f8b2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.491+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f8b38088d30 con 0x7f8b480ce390 2023-10-16T14:13:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.492+0000 7f8b3ffff640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8b2806af90 0x7f8b2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.493+0000 7f8b3ffff640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8b2806af90 0x7f8b2806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8b30036000 tx=0x7f8b3003c040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.495+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b38053c80 con 0x7f8b480ce390 2023-10-16T14:13:13.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.695+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b480cdc60 con 0x7f8b480ce390 2023-10-16T14:13:13.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.698+0000 7f8b3f7fe640 1 -- 172.21.15.60:0/2877770283 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+14250 (secure 0 0 0) 0x7f8b38017020 con 0x7f8b480ce390 2023-10-16T14:13:13.702 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:13.702 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":278,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:11.214527+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6833/2674455624":"2023-10-17T14:13:08.177713+0000","172.21.15.144:0/3492229757":"2023-10-17T14:13:04.037340+0000","172.21.15.144:6833/2906320450":"2023-10-17T14:13:04.037340+0000","172.21.15.144:6832/2906320450":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1668072974":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/2110570505":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/405908562":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/2193300484":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1434044163":"2023-10-17T14:13:04.037340+0000","172.21.15.60:6835/2674455624":"2023-10-17T14:13:08.177713+0000","172.21.15.144:0/983873377":"2023-10-17T14:13:04.037340+0000","172.21.15.144:0/1984005880":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2098656759":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6833/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/307080156":"2023-10-17T14:12:55.110891+0000","172.21.15.144:6832/948422863":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2842352837":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2156516012":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/2942868570":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/786228305":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1402597431":"2023-10-17T14:12:55.110891+0000","172.21.15.144:0/1518159871":"2023-10-16T15:12:37.308329+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:13.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.700+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8b2806af90 msgr2=0x7f8b2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.700+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8b2806af90 0x7f8b2806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8b30036000 tx=0x7f8b3003c040 comp rx=0 tx=0).stop 2023-10-16T14:13:13.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.700+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 msgr2=0x7f8b481590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.700+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 0x7f8b481590f0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f8b38001eb0 tx=0x7f8b3800c960 comp rx=0 tx=0).stop 2023-10-16T14:13:13.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 shutdown_connections 2023-10-16T14:13:13.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8b2806af90 0x7f8b2806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8b4815ae80 0x7f8b480a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b4815aaa0 0x7f8b480a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 --2- 172.21.15.60:0/2877770283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b480ce390 0x7f8b481590f0 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 >> 172.21.15.60:0/2877770283 conn(0x7f8b480cb410 msgr2=0x7f8b480acfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:13.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 shutdown_connections 2023-10-16T14:13:13.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.701+0000 7f8b4d0ec640 1 -- 172.21.15.60:0/2877770283 wait complete. 2023-10-16T14:13:13.725 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:13:13.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:13:13.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.958+0000 7f84d891d640 1 Processor -- start 2023-10-16T14:13:13.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.959+0000 7f84d891d640 1 -- start start 2023-10-16T14:13:13.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.960+0000 7f84d891d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f84d415ad70 0x7f84d4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.960+0000 7f84d891d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40c9ca0 0x7f84d406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.960+0000 7f84d891d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d40cb030 con 0x7f84d40ca3b0 2023-10-16T14:13:13.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.960+0000 7f84d891d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d40c91c0 con 0x7f84d415ad70 2023-10-16T14:13:13.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.960+0000 7f84d891d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40c94a0 con 0x7f84d40c9ca0 2023-10-16T14:13:13.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f84d415ad70 0x7f84d4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d2d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f84d40ca3b0 0x7f84d40ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34334/0 (socket says 172.21.15.60:34334) 2023-10-16T14:13:13.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d2d9d640 1 -- 172.21.15.60:0/3242637676 learned_addr learned my addr 172.21.15.60:0/3242637676 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:13.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d1d9b640 1 --2- 172.21.15.60:0/3242637676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40c9ca0 0x7f84d406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d259c640 1 -- 172.21.15.60:0/3242637676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40c9ca0 msgr2=0x7f84d406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d259c640 1 --2- 172.21.15.60:0/3242637676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40c9ca0 0x7f84d406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d259c640 1 -- 172.21.15.60:0/3242637676 >> v1:172.21.15.60:6789/0 conn(0x7f84d40ca3b0 legacy=0x7f84d40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d259c640 1 -- 172.21.15.60:0/3242637676 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d40c9780 con 0x7f84d415ad70 2023-10-16T14:13:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.961+0000 7f84d159a640 1 -- 172.21.15.60:0/3242637676 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101277306 0 0) 0x7f84d40cb030 con 0x7f84d40ca3b0 2023-10-16T14:13:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.962+0000 7f84d259c640 1 --2- 172.21.15.60:0/3242637676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d4069da0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f84c40097a0 tx=0x7f84c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8c319cd3ea5e5a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.963+0000 7f84d159a640 1 -- 172.21.15.60:0/3242637676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4031830 con 0x7f84d415ad70 2023-10-16T14:13:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.963+0000 7f84d159a640 1 -- 172.21.15.60:0/3242637676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4031990 con 0x7f84d415ad70 2023-10-16T14:13:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.963+0000 7f84d159a640 1 -- 172.21.15.60:0/3242637676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4031d20 con 0x7f84d415ad70 2023-10-16T14:13:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.963+0000 7f84d891d640 1 -- 172.21.15.60:0/3242637676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 msgr2=0x7f84d4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.963+0000 7f84d891d640 1 --2- 172.21.15.60:0/3242637676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d4069da0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f84c40097a0 tx=0x7f84c402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.964+0000 7f84d891d640 1 -- 172.21.15.60:0/3242637676 shutdown_connections 2023-10-16T14:13:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.964+0000 7f84d891d640 1 --2- 172.21.15.60:0/3242637676 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40c9ca0 0x7f84d406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.964+0000 7f84d891d640 1 --2- 172.21.15.60:0/3242637676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d4069da0 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.964+0000 7f84d891d640 1 -- 172.21.15.60:0/3242637676 >> 172.21.15.60:0/3242637676 conn(0x7f84d40ce9b0 msgr2=0x7f84d40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.964+0000 7f84d891d640 1 -- 172.21.15.60:0/3242637676 shutdown_connections 2023-10-16T14:13:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.964+0000 7f84d891d640 1 -- 172.21.15.60:0/3242637676 wait complete. 2023-10-16T14:13:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.965+0000 7f84d891d640 1 Processor -- start 2023-10-16T14:13:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.966+0000 7f84d891d640 1 -- start start 2023-10-16T14:13:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.966+0000 7f84d891d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84d40c9ca0 0x7f84d40a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d891d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40ca3b0 0x7f84d40ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d891d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d40a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d891d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40ab160 con 0x7f84d415ad70 2023-10-16T14:13:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d891d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40a8d40 con 0x7f84d40c9ca0 2023-10-16T14:13:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d891d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40ab370 con 0x7f84d40ca3b0 2023-10-16T14:13:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d40a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d40a9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46532/0 (socket says 172.21.15.60:46532) 2023-10-16T14:13:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 -- 172.21.15.60:0/1388340680 learned_addr learned my addr 172.21.15.60:0/1388340680 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d259c640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84d40c9ca0 0x7f84d40a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d1d9b640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40ca3b0 0x7f84d40ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 -- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40ca3b0 msgr2=0x7f84d40ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40ca3b0 0x7f84d40ac130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 -- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84d40c9ca0 msgr2=0x7f84d40a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84d40c9ca0 0x7f84d40a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.967+0000 7f84d2d9d640 1 -- 172.21.15.60:0/1388340680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4009400 con 0x7f84d415ad70 2023-10-16T14:13:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.968+0000 7f84d2d9d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d40a9320 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00c7a0 tx=0x7f84cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.968+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc00a640 con 0x7f84d415ad70 2023-10-16T14:13:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.968+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc00a7a0 con 0x7f84d415ad70 2023-10-16T14:13:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.968+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d40ab650 con 0x7f84d415ad70 2023-10-16T14:13:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.968+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc00abd0 con 0x7f84d415ad70 2023-10-16T14:13:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.968+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d40ab9e0 con 0x7f84d415ad70 2023-10-16T14:13:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.969+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d40cb030 con 0x7f84d415ad70 2023-10-16T14:13:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.970+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f84cc00ad30 con 0x7f84d415ad70 2023-10-16T14:13:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.971+0000 7f84caffd640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f84b406af90 0x7f84b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.971+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f84cc088ff0 con 0x7f84d415ad70 2023-10-16T14:13:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.972+0000 7f84d259c640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f84b406af90 0x7f84b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.973+0000 7f84d259c640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f84b406af90 0x7f84b406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f84c4036000 tx=0x7f84c4008030 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:13.975+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84cc053f20 con 0x7f84d415ad70 2023-10-16T14:13:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.172+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84d404e160 con 0x7f84d415ad70 2023-10-16T14:13:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.173+0000 7f84caffd640 1 -- 172.21.15.60:0/1388340680 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v278) v1 ==== 93+0+10 (secure 0 0 0) 0x7f84cc018020 con 0x7f84d415ad70 2023-10-16T14:13:14.177 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:13:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f84b406af90 msgr2=0x7f84b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f84b406af90 0x7f84b406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f84c4036000 tx=0x7f84c4008030 comp rx=0 tx=0).stop 2023-10-16T14:13:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 msgr2=0x7f84d40a9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d40a9320 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00c7a0 tx=0x7f84cc00cc60 comp rx=0 tx=0).stop 2023-10-16T14:13:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 shutdown_connections 2023-10-16T14:13:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f84b406af90 0x7f84b406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f84d40ca3b0 0x7f84d40ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84d415ad70 0x7f84d40a9320 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 --2- 172.21.15.60:0/1388340680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84d40c9ca0 0x7f84d40a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 >> 172.21.15.60:0/1388340680 conn(0x7f84d40ce9b0 msgr2=0x7f84d415b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 shutdown_connections 2023-10-16T14:13:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.176+0000 7f84d891d640 1 -- 172.21.15.60:0/1388340680 wait complete. 2023-10-16T14:13:14.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments' 2023-10-16T14:13:14.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.428+0000 7f3ddb2bb640 1 Processor -- start 2023-10-16T14:13:14.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.428+0000 7f3ddb2bb640 1 -- start start 2023-10-16T14:13:14.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.430+0000 7f3ddb2bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3dd415aaa0 0x7f3dd415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.430+0000 7f3ddb2bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd415ed60 con 0x7f3dd40ce390 2023-10-16T14:13:14.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.430+0000 7f3ddb2bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40707e0 con 0x7f3dd415ae80 2023-10-16T14:13:14.434 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.430+0000 7f3ddb2bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd406b810 con 0x7f3dd415aaa0 2023-10-16T14:13:14.436 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dd9858640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3dd40ce390 0x7f3dd4070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34346/0 (socket says 172.21.15.60:34346) 2023-10-16T14:13:14.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dd9858640 1 -- 172.21.15.60:0/9654474 learned_addr learned my addr 172.21.15.60:0/9654474 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:14.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dd9057640 1 --2- 172.21.15.60:0/9654474 >> v2:172.21.15.144:3300/0 conn(0x7f3dd415aaa0 0x7f3dd415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202389250 0 0) 0x7f3dd415ed60 con 0x7f3dd40ce390 2023-10-16T14:13:14.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc0003520 con 0x7f3dd40ce390 2023-10-16T14:13:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762075201 0 0) 0x7f3dc0003520 con 0x7f3dd40ce390 2023-10-16T14:13:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 >> v2:172.21.15.144:3300/0 conn(0x7f3dd415aaa0 msgr2=0x7f3dd415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 --2- 172.21.15.60:0/9654474 >> v2:172.21.15.144:3300/0 conn(0x7f3dd415aaa0 0x7f3dd415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 >> v1:172.21.15.144:6789/0 conn(0x7f3dd415ae80 legacy=0x7f3dd40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.431+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd415ef90 con 0x7f3dd40ce390 2023-10-16T14:13:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f3dcc0019f0 con 0x7f3dd40ce390 2023-10-16T14:13:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9057640 1 --2- 172.21.15.60:0/9654474 >> v2:172.21.15.144:3300/0 conn(0x7f3dd415aaa0 0x7f3dd415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 >> v1:172.21.15.60:6789/0 conn(0x7f3dd40ce390 legacy=0x7f3dd4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dc00039c0 0x7f3dc0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc00047c0 0x7f3dc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc0005180 0x7f3dc000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd406b810 con 0x7f3dc0005180 2023-10-16T14:13:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc000c080 con 0x7f3dc00047c0 2023-10-16T14:13:14.440 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc000c240 con 0x7f3dc00039c0 2023-10-16T14:13:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9858640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc0005180 0x7f3dc000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9858640 1 -- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dc00039c0 msgr2=0x7f3dc0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9858640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dc00039c0 0x7f3dc0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9858640 1 -- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc00047c0 msgr2=0x7f3dc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:14.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd8856640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc00047c0 0x7f3dc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9858640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc00047c0 0x7f3dc0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.432+0000 7f3dd9858640 1 -- 172.21.15.60:0/9654474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0003520 con 0x7f3dc0005180 2023-10-16T14:13:14.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.433+0000 7f3dd8856640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc00047c0 0x7f3dc0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.433+0000 7f3dd9858640 1 -- 172.21.15.60:0/9654474 reap_dead start 2023-10-16T14:13:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.434+0000 7f3dd9858640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc0005180 0x7f3dc000b980 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00a750 tx=0x7f3dc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de4d9d4bfa05d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:14.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.434+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc034630 con 0x7f3dc0005180 2023-10-16T14:13:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.434+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc02f070 con 0x7f3dc0005180 2023-10-16T14:13:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.434+0000 7f3dcbfff640 1 -- 172.21.15.60:0/9654474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc0349f0 con 0x7f3dc0005180 2023-10-16T14:13:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/9654474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc0005180 msgr2=0x7f3dc000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc0005180 0x7f3dc000b980 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00a750 tx=0x7f3dc00057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/9654474 shutdown_connections 2023-10-16T14:13:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/9654474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc0005180 0x7f3dc000b980 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/9654474 >> 172.21.15.60:0/9654474 conn(0x7f3dd40cb410 msgr2=0x7f3dd40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/9654474 shutdown_connections 2023-10-16T14:13:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.435+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/9654474 wait complete. 2023-10-16T14:13:14.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.437+0000 7f3ddb2bb640 1 Processor -- start 2023-10-16T14:13:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.437+0000 7f3ddb2bb640 1 -- start start 2023-10-16T14:13:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3ddb2bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc0005180 0x7f3dd4155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3ddb2bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dd40c8610 0x7f3dd4154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3ddb2bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 0x7f3dd40acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3ddb2bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd415e1d0 con 0x7f3dd40ac890 2023-10-16T14:13:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3ddb2bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40c8a90 con 0x7f3dc0005180 2023-10-16T14:13:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3ddb2bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40ad250 con 0x7f3dd40c8610 2023-10-16T14:13:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3dd9858640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 0x7f3dd40acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3dd9858640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 0x7f3dd40acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46548/0 (socket says 172.21.15.60:46548) 2023-10-16T14:13:14.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3dd9858640 1 -- 172.21.15.60:0/2393422902 learned_addr learned my addr 172.21.15.60:0/2393422902 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.438+0000 7f3dd9057640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc0005180 0x7f3dd4155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd8856640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dd40c8610 0x7f3dd4154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd9858640 1 -- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dd40c8610 msgr2=0x7f3dd4154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd9858640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dd40c8610 0x7f3dd4154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd9858640 1 -- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc0005180 msgr2=0x7f3dd4155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd9858640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc0005180 0x7f3dd4155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd9858640 1 -- 172.21.15.60:0/2393422902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4155180 con 0x7f3dd40ac890 2023-10-16T14:13:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd8856640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dd40c8610 0x7f3dd4154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:14.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dd9858640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 0x7f3dd40acd10 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc000f80 tx=0x7f3dd415f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc034840 con 0x7f3dd40ac890 2023-10-16T14:13:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc0349a0 con 0x7f3dd40ac890 2023-10-16T14:13:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc034d10 con 0x7f3dd40ac890 2023-10-16T14:13:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.439+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd40ca330 con 0x7f3dd40ac890 2023-10-16T14:13:14.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.440+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd415db10 con 0x7f3dd40ac890 2023-10-16T14:13:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.441+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd415e310 con 0x7f3dd40ac890 2023-10-16T14:13:14.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.441+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3dcc031030 con 0x7f3dd40ac890 2023-10-16T14:13:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.442+0000 7f3dc9ffb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f3db806b010 0x7f3db806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.442+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f3dcc0abb20 con 0x7f3dd40ac890 2023-10-16T14:13:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.443+0000 7f3dd9057640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f3db806b010 0x7f3db806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:14.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.444+0000 7f3dd9057640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f3db806b010 0x7f3db806d4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009be0 tx=0x7f3dd0008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:14.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.447+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3dd415e310 con 0x7f3dd40ac890 2023-10-16T14:13:14.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:14.715+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]} v 0) v1 -- 0x7f3dd415e310 con 0x7f3dd40ac890 2023-10-16T14:13:15.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.234+0000 7f3dc9ffb640 1 -- 172.21.15.60:0/2393422902 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]}]=0 v3326) v1 ==== 201+0+0 (secure 0 0 0) 0x7f3dd415e310 con 0x7f3dd40ac890 2023-10-16T14:13:15.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.237+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f3db806b010 msgr2=0x7f3db806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.237+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f3db806b010 0x7f3db806d4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0009be0 tx=0x7f3dd0008040 comp rx=0 tx=0).stop 2023-10-16T14:13:15.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 msgr2=0x7f3dd40acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 0x7f3dd40acd10 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc000f80 tx=0x7f3dd415f830 comp rx=0 tx=0).stop 2023-10-16T14:13:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 shutdown_connections 2023-10-16T14:13:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f3db806b010 0x7f3db806d4d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3dd40c8610 0x7f3dd4154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dd40ac890 0x7f3dd40acd10 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 --2- 172.21.15.60:0/2393422902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3dc0005180 0x7f3dd4155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 >> 172.21.15.60:0/2393422902 conn(0x7f3dd40cb410 msgr2=0x7f3dd4158880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:15.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 shutdown_connections 2023-10-16T14:13:15.244 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.238+0000 7f3ddb2bb640 1 -- 172.21.15.60:0/2393422902 wait complete. 2023-10-16T14:13:15.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T14:13:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.495+0000 7f541b051640 1 Processor -- start 2023-10-16T14:13:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.495+0000 7f541b051640 1 -- start start 2023-10-16T14:13:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f541b051640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f541415ad70 0x7f5414069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f541b051640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140cb030 con 0x7f54140c9ca0 2023-10-16T14:13:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f541b051640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140c91c0 con 0x7f54140caac0 2023-10-16T14:13:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f541b051640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140c94a0 con 0x7f541415ad70 2023-10-16T14:13:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f5413fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f54140c9ca0 0x7f54140ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34354/0 (socket says 172.21.15.60:34354) 2023-10-16T14:13:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f5413fff640 1 -- 172.21.15.60:0/1941423682 learned_addr learned my addr 172.21.15.60:0/1941423682 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.497+0000 7f5418ded640 1 --2- 172.21.15.60:0/1941423682 >> v2:172.21.15.60:3300/0 conn(0x7f541415ad70 0x7f5414069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.498+0000 7f54137fe640 1 -- 172.21.15.60:0/1941423682 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471481351 0 0) 0x7f54140cb030 con 0x7f54140c9ca0 2023-10-16T14:13:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.498+0000 7f54137fe640 1 -- 172.21.15.60:0/1941423682 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5400003520 con 0x7f54140c9ca0 2023-10-16T14:13:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.498+0000 7f5418ded640 1 -- 172.21.15.60:0/1941423682 >> v1:172.21.15.144:6789/0 conn(0x7f54140caac0 legacy=0x7f54140ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.498+0000 7f5418ded640 1 -- 172.21.15.60:0/1941423682 >> v1:172.21.15.60:6789/0 conn(0x7f54140c9ca0 legacy=0x7f54140ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.498+0000 7f5418ded640 1 -- 172.21.15.60:0/1941423682 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140c9780 con 0x7f541415ad70 2023-10-16T14:13:15.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.499+0000 7f5418ded640 1 --2- 172.21.15.60:0/1941423682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f5414069da0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f5404009d90 tx=0x7f540402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f46e04fef478183 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:15.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.500+0000 7f54137fe640 1 -- 172.21.15.60:0/1941423682 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540402cd00 con 0x7f541415ad70 2023-10-16T14:13:15.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.500+0000 7f54137fe640 1 -- 172.21.15.60:0/1941423682 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540402ce60 con 0x7f541415ad70 2023-10-16T14:13:15.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.500+0000 7f54137fe640 1 -- 172.21.15.60:0/1941423682 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404034640 con 0x7f541415ad70 2023-10-16T14:13:15.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.500+0000 7f541b051640 1 -- 172.21.15.60:0/1941423682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 msgr2=0x7f5414069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.500+0000 7f541b051640 1 --2- 172.21.15.60:0/1941423682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f5414069da0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f5404009d90 tx=0x7f540402bf50 comp rx=0 tx=0).stop 2023-10-16T14:13:15.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.501+0000 7f541b051640 1 -- 172.21.15.60:0/1941423682 shutdown_connections 2023-10-16T14:13:15.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.501+0000 7f541b051640 1 --2- 172.21.15.60:0/1941423682 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f5414069da0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.501+0000 7f541b051640 1 -- 172.21.15.60:0/1941423682 >> 172.21.15.60:0/1941423682 conn(0x7f54140ce9b0 msgr2=0x7f54140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:15.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.501+0000 7f541b051640 1 -- 172.21.15.60:0/1941423682 shutdown_connections 2023-10-16T14:13:15.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.501+0000 7f541b051640 1 -- 172.21.15.60:0/1941423682 wait complete. 2023-10-16T14:13:15.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.502+0000 7f541b051640 1 Processor -- start 2023-10-16T14:13:15.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.503+0000 7f541b051640 1 -- start start 2023-10-16T14:13:15.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.503+0000 7f541b051640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54140c9ca0 0x7f54140ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:15.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.503+0000 7f541b051640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54140caac0 0x7f541414e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:15.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f541b051640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f54140c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:15.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f541b051640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140c82a0 con 0x7f541415ad70 2023-10-16T14:13:15.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f541b051640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140c8c70 con 0x7f54140c9ca0 2023-10-16T14:13:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f541b051640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54140c7970 con 0x7f54140caac0 2023-10-16T14:13:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f54140c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f54140c8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46554/0 (socket says 172.21.15.60:46554) 2023-10-16T14:13:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 -- 172.21.15.60:0/1200354018 learned_addr learned my addr 172.21.15.60:0/1200354018 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:15.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f5418ded640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54140c9ca0 0x7f54140ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:15.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f5413fff640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54140caac0 0x7f541414e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:15.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 -- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54140caac0 msgr2=0x7f541414e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54140caac0 0x7f541414e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 -- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54140c9ca0 msgr2=0x7f54140ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54140c9ca0 0x7f54140ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.504+0000 7f54195ee640 1 -- 172.21.15.60:0/1200354018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404009a70 con 0x7f541415ad70 2023-10-16T14:13:15.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.505+0000 7f54195ee640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f54140c8730 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f540c001eb0 tx=0x7f540c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:15.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.505+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c0052d0 con 0x7f541415ad70 2023-10-16T14:13:15.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.505+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540c005430 con 0x7f541415ad70 2023-10-16T14:13:15.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.505+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c003810 con 0x7f541415ad70 2023-10-16T14:13:15.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.505+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54140c7c50 con 0x7f541415ad70 2023-10-16T14:13:15.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.505+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54140ad9a0 con 0x7f541415ad70 2023-10-16T14:13:15.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.506+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541404e160 con 0x7f541415ad70 2023-10-16T14:13:15.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.507+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f540c003970 con 0x7f541415ad70 2023-10-16T14:13:15.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.508+0000 7f54117fa640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53f406afc0 0x7f53f406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:15.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.508+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7f540c088d80 con 0x7f541415ad70 2023-10-16T14:13:15.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.509+0000 7f5418ded640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53f406afc0 0x7f53f406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:15.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.510+0000 7f5418ded640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53f406afc0 0x7f53f406d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5404036000 tx=0x7f5404030000 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:15.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.513+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f540c053cb0 con 0x7f541415ad70 2023-10-16T14:13:15.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.710+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f54140cefa0 con 0x7f541415ad70 2023-10-16T14:13:15.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.717+0000 7f54117fa640 1 -- 172.21.15.60:0/1200354018 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f540c0170c0 con 0x7f541415ad70 2023-10-16T14:13:15.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53f406afc0 msgr2=0x7f53f406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53f406afc0 0x7f53f406d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5404036000 tx=0x7f5404030000 comp rx=0 tx=0).stop 2023-10-16T14:13:15.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 msgr2=0x7f54140c8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:15.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f54140c8730 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f540c001eb0 tx=0x7f540c00c890 comp rx=0 tx=0).stop 2023-10-16T14:13:15.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 shutdown_connections 2023-10-16T14:13:15.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f53f406afc0 0x7f53f406d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f54140caac0 0x7f541414e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f541415ad70 0x7f54140c8730 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 --2- 172.21.15.60:0/1200354018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54140c9ca0 0x7f54140ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:15.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 >> 172.21.15.60:0/1200354018 conn(0x7f54140ce9b0 msgr2=0x7f54140cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:15.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.720+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 shutdown_connections 2023-10-16T14:13:15.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:15.721+0000 7f541b051640 1 -- 172.21.15.60:0/1200354018 wait complete. 2023-10-16T14:13:15.743 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:13:15.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:13:15.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:13:15.995 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:13:15.996 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:15.996 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:16.059 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:13:16.060 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:16.060 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:16.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:16.120 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:16.120 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:16.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:13:16.183 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:16.183 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:16.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:16.242 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T14:13:16.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:16.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.477+0000 7fcabd5e2640 1 Processor -- start 2023-10-16T14:13:16.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.477+0000 7fcabd5e2640 1 -- start start 2023-10-16T14:13:16.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcabd5e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab806c040 0x7fcab806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcabd5e2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab815f4a0 con 0x7fcab80cda80 2023-10-16T14:13:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcabd5e2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab806c960 con 0x7fcab80ad970 2023-10-16T14:13:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcabd5e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab815b9d0 con 0x7fcab806c040 2023-10-16T14:13:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcabcde1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcab80cda80 0x7fcab815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34362/0 (socket says 172.21.15.60:34362) 2023-10-16T14:13:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcabcde1640 1 -- 172.21.15.60:0/1839197061 learned_addr learned my addr 172.21.15.60:0/1839197061 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.479+0000 7fcab7fff640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab806c040 0x7fcab806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492126185 0 0) 0x7fcab815f4a0 con 0x7fcab80cda80 2023-10-16T14:13:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa4003520 con 0x7fcab80cda80 2023-10-16T14:13:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2296864090 0 0) 0x7fcaa4003520 con 0x7fcab80cda80 2023-10-16T14:13:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab806c040 msgr2=0x7fcab806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab806c040 0x7fcab806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 >> v1:172.21.15.144:6789/0 conn(0x7fcab80ad970 legacy=0x7fcab815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab815bcb0 con 0x7fcab80cda80 2023-10-16T14:13:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fcab0001670 con 0x7fcab80cda80 2023-10-16T14:13:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 >> v1:172.21.15.60:6789/0 conn(0x7fcab80cda80 legacy=0x7fcab815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcaa40039a0 0x7fcaa4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa40047c0 0x7fcaa4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa4005180 0x7fcaa400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab815b9d0 con 0x7fcaa4005180 2023-10-16T14:13:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa400c080 con 0x7fcaa40047c0 2023-10-16T14:13:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa400c240 con 0x7fcaa40039a0 2023-10-16T14:13:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcab7fff640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab806c040 0x7fcab806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.480+0000 7fcabcde1640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa4005180 0x7fcaa400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcabcde1640 1 -- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcaa40039a0 msgr2=0x7fcaa4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcab77fe640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa40047c0 0x7fcaa4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcabcde1640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcaa40039a0 0x7fcaa4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcabcde1640 1 -- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa40047c0 msgr2=0x7fcaa4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:16.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcabcde1640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa40047c0 0x7fcaa4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcabcde1640 1 -- 172.21.15.60:0/1839197061 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4003520 con 0x7fcaa4005180 2023-10-16T14:13:16.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.481+0000 7fcab77fe640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa40047c0 0x7fcaa4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.482+0000 7fcabcde1640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa4005180 0x7fcaa400b980 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fcab000c0e0 tx=0x7fcaa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2720da3f21674b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.482+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0005ad0 con 0x7fcaa4005180 2023-10-16T14:13:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab0005c30 con 0x7fcaa4005180 2023-10-16T14:13:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcab6ffd640 1 -- 172.21.15.60:0/1839197061 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab002ca50 con 0x7fcaa4005180 2023-10-16T14:13:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabd5e2640 1 -- 172.21.15.60:0/1839197061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa4005180 msgr2=0x7fcaa400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/1839197061 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa4005180 0x7fcaa400b980 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fcab000c0e0 tx=0x7fcaa40057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabcde1640 1 -- 172.21.15.60:0/1839197061 reap_dead start 2023-10-16T14:13:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabd5e2640 1 -- 172.21.15.60:0/1839197061 shutdown_connections 2023-10-16T14:13:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabd5e2640 1 -- 172.21.15.60:0/1839197061 >> 172.21.15.60:0/1839197061 conn(0x7fcab80ce6a0 msgr2=0x7fcab80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabd5e2640 1 -- 172.21.15.60:0/1839197061 shutdown_connections 2023-10-16T14:13:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.483+0000 7fcabd5e2640 1 -- 172.21.15.60:0/1839197061 wait complete. 2023-10-16T14:13:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.485+0000 7fcabd5e2640 1 Processor -- start 2023-10-16T14:13:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.485+0000 7fcabd5e2640 1 -- start start 2023-10-16T14:13:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabd5e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa4005180 0x7fcab81547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabd5e2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab8155780 0x7fcab80c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabd5e2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 0x7fcab80c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabd5e2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8156d70 con 0x7fcab8155c00 2023-10-16T14:13:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabd5e2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80c84c0 con 0x7fcaa4005180 2023-10-16T14:13:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabd5e2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8156160 con 0x7fcab8155780 2023-10-16T14:13:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.486+0000 7fcabcde1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 0x7fcab80c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 0x7fcab80c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46570/0 (socket says 172.21.15.60:46570) 2023-10-16T14:13:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 -- 172.21.15.60:0/2793897617 learned_addr learned my addr 172.21.15.60:0/2793897617 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcab7fff640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa4005180 0x7fcab81547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcab77fe640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab8155780 0x7fcab80c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 -- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab8155780 msgr2=0x7fcab80c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab8155780 0x7fcab80c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 -- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa4005180 msgr2=0x7fcab81547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa4005180 0x7fcab81547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 -- 172.21.15.60:0/2793897617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab8156440 con 0x7fcab8155c00 2023-10-16T14:13:16.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.487+0000 7fcabcde1640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 0x7fcab80c7f80 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fcab002c480 tx=0x7fcab002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:16.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.488+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab003c040 con 0x7fcab8155c00 2023-10-16T14:13:16.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.488+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab002ca10 con 0x7fcab8155c00 2023-10-16T14:13:16.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.488+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab002cd80 con 0x7fcab8155c00 2023-10-16T14:13:16.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.488+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab8156720 con 0x7fcab8155c00 2023-10-16T14:13:16.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.488+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab8154fb0 con 0x7fcab8155c00 2023-10-16T14:13:16.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.489+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab806b330 con 0x7fcab8155c00 2023-10-16T14:13:16.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.490+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcab0032070 con 0x7fcab8155c00 2023-10-16T14:13:16.500 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.492+0000 7fcab4ff9640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fca9806afc0 0x7fca9806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:16.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.492+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7fcab00aa290 con 0x7fcab8155c00 2023-10-16T14:13:16.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.492+0000 7fcab7fff640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fca9806afc0 0x7fca9806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:16.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.494+0000 7fcab7fff640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fca9806afc0 0x7fca9806d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcab80cf350 tx=0x7fcaa800b040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:16.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.496+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcab0075130 con 0x7fcab8155c00 2023-10-16T14:13:16.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.737+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcab804e9b0 con 0x7fcab8155c00 2023-10-16T14:13:16.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.739+0000 7fcab4ff9640 1 -- 172.21.15.60:0/2793897617 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+3709 (secure 0 0 0) 0x7fcab0078750 con 0x7fcab8155c00 2023-10-16T14:13:16.743 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:16.743 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":146,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6836/1211606217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":1211606217},{"type":"v1","addr":"172.21.15.144:6836","nonce":1211606217}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9865,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1320083367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6835","nonce":1320083367},{"type":"v1","addr":"172.21.15.144:6837","nonce":1320083367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12570,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/2050062800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2050062800},{"type":"v1","addr":"172.21.15.60:6834","nonce":2050062800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":12957,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/141507267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":141507267},{"type":"v1","addr":"172.21.15.60:6835","nonce":141507267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146}],"filesystems":[]} 2023-10-16T14:13:16.744 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 146 2023-10-16T14:13:16.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fca9806afc0 msgr2=0x7fca9806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fca9806afc0 0x7fca9806d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcab80cf350 tx=0x7fcaa800b040 comp rx=0 tx=0).stop 2023-10-16T14:13:16.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 msgr2=0x7fcab80c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:16.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 0x7fcab80c7f80 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fcab002c480 tx=0x7fcab002e070 comp rx=0 tx=0).stop 2023-10-16T14:13:16.746 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 shutdown_connections 2023-10-16T14:13:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fca9806afc0 0x7fca9806d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcab8155780 0x7fcab80c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcab8155c00 0x7fcab80c7f80 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 --2- 172.21.15.60:0/2793897617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcaa4005180 0x7fcab81547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 >> 172.21.15.60:0/2793897617 conn(0x7fcab80ce6a0 msgr2=0x7fcab80ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 shutdown_connections 2023-10-16T14:13:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:16.742+0000 7fcabd5e2640 1 -- 172.21.15.60:0/2793897617 wait complete. 2023-10-16T14:13:16.765 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T14:13:16.765 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T14:13:16.766 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:13:16.766 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:13:16.766 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T14:13:16.766 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T14:13:16.767 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:13:16.767 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:13:16.767 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T14:13:16.768 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T14:13:16.768 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:13:16.768 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:13:16.769 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T14:13:16.769 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T14:13:16.770 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:13:16.770 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:13:16.785 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:13:16.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T14:13:16.786 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:13:16.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T14:13:16.791 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:13:16.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T14:13:16.794 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:13:16.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T14:13:16.861 INFO:tasks.ceph.mds.c:Started 2023-10-16T14:13:16.862 INFO:tasks.ceph.mds.d:Started 2023-10-16T14:13:16.862 DEBUG:teuthology.parallel:result is None 2023-10-16T14:13:16.863 DEBUG:teuthology.parallel:result is None 2023-10-16T14:13:16.867 INFO:tasks.ceph.mds.b:Started 2023-10-16T14:13:16.867 DEBUG:teuthology.parallel:result is None 2023-10-16T14:13:16.869 INFO:tasks.ceph.mds.a:Started 2023-10-16T14:13:16.869 DEBUG:teuthology.parallel:result is None 2023-10-16T14:13:16.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T14:13:17.009 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2023-10-16T14:13:17.009 INFO:tasks.ceph.mds.d.smithi144.stdout:starting mds.d at 2023-10-16T14:13:17.034 INFO:tasks.ceph.mds.b.smithi144.stdout:starting mds.b at 2023-10-16T14:13:17.039 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:13:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.065+0000 7fd278533640 1 Processor -- start 2023-10-16T14:13:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.066+0000 7fd278533640 1 -- start start 2023-10-16T14:13:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd278533640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd270070830 0x7fd27006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd278533640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2700cf5f0 0x7fd27006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd278533640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd270070c10 0x7fd2700cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd278533640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd270161740 con 0x7fd270070c10 2023-10-16T14:13:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd278533640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27006c680 con 0x7fd270070830 2023-10-16T14:13:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd278533640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2700cdbd0 con 0x7fd2700cf5f0 2023-10-16T14:13:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd276ad0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd270070c10 0x7fd2700cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd276ad0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd270070c10 0x7fd2700cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46596/0 (socket says 172.21.15.60:46596) 2023-10-16T14:13:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd276ad0640 1 -- 172.21.15.60:0/1151332326 learned_addr learned my addr 172.21.15.60:0/1151332326 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.067+0000 7fd2762cf640 1 --2- 172.21.15.60:0/1151332326 >> v2:172.21.15.144:3300/0 conn(0x7fd270070830 0x7fd27006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.068+0000 7fd275ace640 1 --2- 172.21.15.60:0/1151332326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2700cf5f0 0x7fd27006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.068+0000 7fd276ad0640 1 -- 172.21.15.60:0/1151332326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2700cf5f0 msgr2=0x7fd27006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:17.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.068+0000 7fd276ad0640 1 --2- 172.21.15.60:0/1151332326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2700cf5f0 0x7fd27006c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.068+0000 7fd276ad0640 1 -- 172.21.15.60:0/1151332326 >> v2:172.21.15.144:3300/0 conn(0x7fd270070830 msgr2=0x7fd27006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:17.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.068+0000 7fd276ad0640 1 --2- 172.21.15.60:0/1151332326 >> v2:172.21.15.144:3300/0 conn(0x7fd270070830 0x7fd27006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.068+0000 7fd276ad0640 1 -- 172.21.15.60:0/1151332326 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27006b0f0 con 0x7fd270070c10 2023-10-16T14:13:17.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.069+0000 7fd276ad0640 1 --2- 172.21.15.60:0/1151332326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270070c10 0x7fd2700cd690 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7fd27006c010 tx=0x7fd26802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1d53394c13de02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:17.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.069+0000 7fd2752cd640 1 -- 172.21.15.60:0/1151332326 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26802c950 con 0x7fd270070c10 2023-10-16T14:13:17.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.069+0000 7fd2752cd640 1 -- 172.21.15.60:0/1151332326 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26802cab0 con 0x7fd270070c10 2023-10-16T14:13:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.069+0000 7fd2752cd640 1 -- 172.21.15.60:0/1151332326 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26802ce40 con 0x7fd270070c10 2023-10-16T14:13:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 -- 172.21.15.60:0/1151332326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270070c10 msgr2=0x7fd2700cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 --2- 172.21.15.60:0/1151332326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270070c10 0x7fd2700cd690 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7fd27006c010 tx=0x7fd26802bdb0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 -- 172.21.15.60:0/1151332326 shutdown_connections 2023-10-16T14:13:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 --2- 172.21.15.60:0/1151332326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270070c10 0x7fd2700cd690 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 --2- 172.21.15.60:0/1151332326 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd2700cf5f0 0x7fd27006c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 --2- 172.21.15.60:0/1151332326 >> v2:172.21.15.144:3300/0 conn(0x7fd270070830 0x7fd27006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 -- 172.21.15.60:0/1151332326 >> 172.21.15.60:0/1151332326 conn(0x7fd2700cfac0 msgr2=0x7fd2700cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 -- 172.21.15.60:0/1151332326 shutdown_connections 2023-10-16T14:13:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.070+0000 7fd278533640 1 -- 172.21.15.60:0/1151332326 wait complete. 2023-10-16T14:13:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.072+0000 7fd278533640 1 Processor -- start 2023-10-16T14:13:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.072+0000 7fd278533640 1 -- start start 2023-10-16T14:13:17.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd278533640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd270070830 0x7fd27014c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd278533640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd270070c10 0x7fd2700aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd278533640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 0x7fd2700b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd278533640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27014df90 con 0x7fd2700cf5f0 2023-10-16T14:13:17.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd278533640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27014d380 con 0x7fd270070830 2023-10-16T14:13:17.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd278533640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27014d660 con 0x7fd270070c10 2023-10-16T14:13:17.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd276ad0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 0x7fd2700b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd276ad0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 0x7fd2700b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46612/0 (socket says 172.21.15.60:46612) 2023-10-16T14:13:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.073+0000 7fd276ad0640 1 -- 172.21.15.60:0/4024322224 learned_addr learned my addr 172.21.15.60:0/4024322224 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd2762cf640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd270070830 0x7fd27014c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd275ace640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd270070c10 0x7fd2700aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd276ad0640 1 -- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd270070c10 msgr2=0x7fd2700aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd276ad0640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd270070c10 0x7fd2700aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd276ad0640 1 -- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd270070830 msgr2=0x7fd27014c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd276ad0640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd270070830 0x7fd27014c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd276ad0640 1 -- 172.21.15.60:0/4024322224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268009b30 con 0x7fd2700cf5f0 2023-10-16T14:13:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.074+0000 7fd276ad0640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 0x7fd2700b0ce0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7fd268036040 tx=0x7fd268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.075+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26803e450 con 0x7fd2700cf5f0 2023-10-16T14:13:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.075+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26803e5b0 con 0x7fd2700cf5f0 2023-10-16T14:13:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.075+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26803e960 con 0x7fd2700cf5f0 2023-10-16T14:13:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.075+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27014d940 con 0x7fd2700cf5f0 2023-10-16T14:13:17.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.075+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27014cb80 con 0x7fd2700cf5f0 2023-10-16T14:13:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.076+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27004e160 con 0x7fd2700cf5f0 2023-10-16T14:13:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.077+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd26803eac0 con 0x7fd2700cf5f0 2023-10-16T14:13:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.078+0000 7fd266ffd640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd25406af90 0x7fd25406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.078+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5803+0+0 (secure 0 0 0) 0x7fd2680ac5f0 con 0x7fd2700cf5f0 2023-10-16T14:13:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.078+0000 7fd2762cf640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd25406af90 0x7fd25406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.079+0000 7fd2762cf640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd25406af90 0x7fd25406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd26c006fd0 tx=0x7fd26c008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.083+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2680775b0 con 0x7fd2700cf5f0 2023-10-16T14:13:17.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:17.290+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd2700b5a40 con 0x7fd2700cf5f0 2023-10-16T14:13:18.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.755+0000 7fd266ffd640 1 -- 172.21.15.60:0/4024322224 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v283) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd26807abd0 con 0x7fd2700cf5f0 2023-10-16T14:13:18.759 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2023-10-16T14:13:18.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd25406af90 msgr2=0x7fd25406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:18.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd25406af90 0x7fd25406d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd26c006fd0 tx=0x7fd26c008040 comp rx=0 tx=0).stop 2023-10-16T14:13:18.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 msgr2=0x7fd2700b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:18.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 0x7fd2700b0ce0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7fd268036040 tx=0x7fd268004a40 comp rx=0 tx=0).stop 2023-10-16T14:13:18.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 shutdown_connections 2023-10-16T14:13:18.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd25406af90 0x7fd25406d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:18.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd270070c10 0x7fd2700aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:18.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700cf5f0 0x7fd2700b0ce0 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:18.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 --2- 172.21.15.60:0/4024322224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd270070830 0x7fd27014c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:18.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.758+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 >> 172.21.15.60:0/4024322224 conn(0x7fd2700cfac0 msgr2=0x7fd2700ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:18.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.759+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 shutdown_connections 2023-10-16T14:13:18.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:18.759+0000 7fd278533640 1 -- 172.21.15.60:0/4024322224 wait complete. 2023-10-16T14:13:18.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T14:13:19.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.002+0000 7f9535ae6640 1 Processor -- start 2023-10-16T14:13:19.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.002+0000 7f9535ae6640 1 -- start start 2023-10-16T14:13:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9535ae6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f953006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9535ae6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f95300ad970 0x7f953015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9535ae6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f953015ebc0 0x7f953015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9535ae6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953006c960 con 0x7f953015ebc0 2023-10-16T14:13:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9535ae6640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953015b9d0 con 0x7f95300ad970 2023-10-16T14:13:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9535ae6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953015bcb0 con 0x7f953006c040 2023-10-16T14:13:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f95352e5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f953015ebc0 0x7f953015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f95352e5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f953015ebc0 0x7f953015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47414/0 (socket says 172.21.15.60:47414) 2023-10-16T14:13:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f95352e5640 1 -- 172.21.15.60:0/3105387528 learned_addr learned my addr 172.21.15.60:0/3105387528 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.004+0000 7f9534ae4640 1 --2- 172.21.15.60:0/3105387528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f953006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.005+0000 7f952ffff640 1 --2- 172.21.15.60:0/3105387528 >> v2:172.21.15.144:3300/0 conn(0x7f95300ad970 0x7f953015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.005+0000 7f95352e5640 1 -- 172.21.15.60:0/3105387528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 msgr2=0x7f953006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.005+0000 7f95352e5640 1 --2- 172.21.15.60:0/3105387528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f953006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.005+0000 7f95352e5640 1 -- 172.21.15.60:0/3105387528 >> v2:172.21.15.144:3300/0 conn(0x7f95300ad970 msgr2=0x7f953015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.005+0000 7f95352e5640 1 --2- 172.21.15.60:0/3105387528 >> v2:172.21.15.144:3300/0 conn(0x7f95300ad970 0x7f953015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.005+0000 7f95352e5640 1 -- 172.21.15.60:0/3105387528 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953015bf90 con 0x7f953015ebc0 2023-10-16T14:13:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.006+0000 7f95352e5640 1 --2- 172.21.15.60:0/3105387528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f953015ebc0 0x7f953015f050 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f9528009e50 tx=0x7f952802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c503b1fbd73dc8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:19.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.006+0000 7f952f7fe640 1 -- 172.21.15.60:0/3105387528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952802ca70 con 0x7f953015ebc0 2023-10-16T14:13:19.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.006+0000 7f952f7fe640 1 -- 172.21.15.60:0/3105387528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952802cbd0 con 0x7f953015ebc0 2023-10-16T14:13:19.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.006+0000 7f952f7fe640 1 -- 172.21.15.60:0/3105387528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95280343f0 con 0x7f953015ebc0 2023-10-16T14:13:19.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 -- 172.21.15.60:0/3105387528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f953015ebc0 msgr2=0x7f953015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 --2- 172.21.15.60:0/3105387528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f953015ebc0 0x7f953015f050 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f9528009e50 tx=0x7f952802bed0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 -- 172.21.15.60:0/3105387528 shutdown_connections 2023-10-16T14:13:19.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 --2- 172.21.15.60:0/3105387528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f953015ebc0 0x7f953015f050 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 --2- 172.21.15.60:0/3105387528 >> v2:172.21.15.144:3300/0 conn(0x7f95300ad970 0x7f953015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 --2- 172.21.15.60:0/3105387528 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f953006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 -- 172.21.15.60:0/3105387528 >> 172.21.15.60:0/3105387528 conn(0x7f95300ce6a0 msgr2=0x7f95300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:19.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.007+0000 7f9535ae6640 1 -- 172.21.15.60:0/3105387528 shutdown_connections 2023-10-16T14:13:19.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.008+0000 7f9535ae6640 1 -- 172.21.15.60:0/3105387528 wait complete. 2023-10-16T14:13:19.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.010+0000 7f9535ae6640 1 Processor -- start 2023-10-16T14:13:19.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.010+0000 7f9535ae6640 1 -- start start 2023-10-16T14:13:19.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.011+0000 7f9535ae6640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f95300c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f9535ae6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 0x7f9530156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f9535ae6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f953015ebc0 0x7f95300afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f9535ae6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530158600 con 0x7f95300ad970 2023-10-16T14:13:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f9535ae6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530157420 con 0x7f953015ebc0 2023-10-16T14:13:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f9535ae6640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95300c77d0 con 0x7f953006c040 2023-10-16T14:13:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 0x7f9530156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 0x7f9530156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47424/0 (socket says 172.21.15.60:47424) 2023-10-16T14:13:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 -- 172.21.15.60:0/217423782 learned_addr learned my addr 172.21.15.60:0/217423782 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f9534ae4640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f95300c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f95352e5640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f953015ebc0 0x7f95300afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 -- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 msgr2=0x7f95300c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f95300c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 -- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f953015ebc0 msgr2=0x7f95300afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f953015ebc0 0x7f95300afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.012+0000 7f952ffff640 1 -- 172.21.15.60:0/217423782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528009b30 con 0x7f95300ad970 2023-10-16T14:13:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.013+0000 7f952ffff640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 0x7f9530156ee0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f951c00d5f0 tx=0x7f951c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.013+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c0125f0 con 0x7f95300ad970 2023-10-16T14:13:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.013+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c012750 con 0x7f95300ad970 2023-10-16T14:13:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.013+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95300c7ab0 con 0x7f95300ad970 2023-10-16T14:13:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.013+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c012b00 con 0x7f95300ad970 2023-10-16T14:13:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.013+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95300c8070 con 0x7f95300ad970 2023-10-16T14:13:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.015+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9530158740 con 0x7f95300ad970 2023-10-16T14:13:19.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.015+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f951c012c60 con 0x7f95300ad970 2023-10-16T14:13:19.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.016+0000 7f952d7fa640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f951406af90 0x7f951406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.017+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f951c089920 con 0x7f95300ad970 2023-10-16T14:13:19.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.017+0000 7f9534ae4640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f951406af90 0x7f951406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.018+0000 7f9534ae4640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f951406af90 0x7f951406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9524006fd0 tx=0x7f9524008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.021+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9530158740 con 0x7f95300ad970 2023-10-16T14:13:19.290 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.287+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9530158740 con 0x7f95300ad970 2023-10-16T14:13:19.292 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.288+0000 7f952d7fa640 1 -- 172.21.15.60:0/217423782 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v60) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9530158740 con 0x7f95300ad970 2023-10-16T14:13:19.292 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:19.292 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:13:19.293 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2023-10-16T14:13:19.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.293 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2023-10-16T14:13:19.293 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lFtiKARAArnbqjKfTTeVExgQ6e1ia+A==", 2023-10-16T14:13:19.293 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.294 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.294 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:13:19.294 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:13:19.294 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.295 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2023-10-16T14:13:19.295 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lNgukJRAANhJ/vDcmFSv5qwfi/dnOaA==", 2023-10-16T14:13:19.296 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.296 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.296 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:13:19.296 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:13:19.296 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.297 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2023-10-16T14:13:19.297 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lYc5+ExAAR8RzOTYBjNbC+RnWyBjuyA==", 2023-10-16T14:13:19.298 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.298 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.298 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:13:19.298 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:13:19.299 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.299 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.299 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2023-10-16T14:13:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBYLS1lK+CFNhAAjU0a87gRy901EmKWNAyLKQ==", 2023-10-16T14:13:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:13:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:13:19.301 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.302 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2023-10-16T14:13:19.302 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBbLS1lU1XpMRAA6RzCIa7951Wxyb4dtQAiXQ==", 2023-10-16T14:13:19.302 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.302 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.302 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.303 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.304 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2023-10-16T14:13:19.304 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBfLS1luQ1kEBAAcJDjGSMKpjdLoipc4fZEhw==", 2023-10-16T14:13:19.304 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.304 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.304 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.305 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.305 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2023-10-16T14:13:19.305 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBiLS1lMoXeGhAAU7VHoS+GrmspJt7DBjLkow==", 2023-10-16T14:13:19.306 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.306 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.307 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2023-10-16T14:13:19.307 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBlLS1l1RtJNRAAxppk8IX+2lRC0pRBPhEhPQ==", 2023-10-16T14:13:19.307 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.307 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.308 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.308 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.308 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2023-10-16T14:13:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBtLS1lHe9zBhAAHqpE4A6sKPrvDV9WuaR6Ig==", 2023-10-16T14:13:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.310 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2023-10-16T14:13:19.311 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBwLS1lln9TIBAAHBLcrKnFz1n38VGw4Y6raQ==", 2023-10-16T14:13:19.311 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.311 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.311 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.311 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2023-10-16T14:13:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBzLS1lS/0UKhAARhDOAfYBUwWYu7WZtpwLng==", 2023-10-16T14:13:19.312 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2023-10-16T14:13:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQB3LS1lbNLyBhAA5XF/WRp8Zjw0gIandC1VGA==", 2023-10-16T14:13:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.314 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2023-10-16T14:13:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2023-10-16T14:13:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2023-10-16T14:13:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw==", 2023-10-16T14:13:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.316 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2023-10-16T14:13:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T14:13:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.317 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2023-10-16T14:13:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lQchNIRAAW+YAntMTF1tAnDxpoVqIWQ==", 2023-10-16T14:13:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:13:19.318 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:13:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:13:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.319 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2023-10-16T14:13:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1lZnckFRAAdT9FphJ7kBYAZ/cOct3zKg==", 2023-10-16T14:13:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:13:19.320 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:13:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:13:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2023-10-16T14:13:19.321 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBZLS1l94XmKxAAqo3RTE55uK0rpLK3YrmEyA==", 2023-10-16T14:13:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2023-10-16T14:13:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2023-10-16T14:13:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2023-10-16T14:13:19.322 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2023-10-16T14:13:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2023-10-16T14:13:19.323 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBWLS1l/svOLhAAlCTuWsVHglKO+bqabl/Q2A==", 2023-10-16T14:13:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:13:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2023-10-16T14:13:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2023-10-16T14:13:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2023-10-16T14:13:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1l5+aEMBAAJQrNq5z/Lp7KMdJpplNdXA==", 2023-10-16T14:13:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.325 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:13:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:13:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2023-10-16T14:13:19.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:19.327 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2023-10-16T14:13:19.327 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBXLS1lIQE8JBAA9pV2LrXH91Uzswa37dAEgA==", 2023-10-16T14:13:19.327 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2023-10-16T14:13:19.327 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2023-10-16T14:13:19.327 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2023-10-16T14:13:19.327 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2023-10-16T14:13:19.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:19.328 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-10-16T14:13:19.328 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:13:19.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f951406af90 msgr2=0x7f951406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f951406af90 0x7f951406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9524006fd0 tx=0x7f9524008040 comp rx=0 tx=0).stop 2023-10-16T14:13:19.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 msgr2=0x7f9530156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 0x7f9530156ee0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f951c00d5f0 tx=0x7f951c00dab0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 shutdown_connections 2023-10-16T14:13:19.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f951406af90 0x7f951406d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f953015ebc0 0x7f95300afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95300ad970 0x7f9530156ee0 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 --2- 172.21.15.60:0/217423782 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f953006c040 0x7f95300c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 >> 172.21.15.60:0/217423782 conn(0x7f95300ce6a0 msgr2=0x7f95300ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:19.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 shutdown_connections 2023-10-16T14:13:19.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.291+0000 7f9535ae6640 1 -- 172.21.15.60:0/217423782 wait complete. 2023-10-16T14:13:19.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:19.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.561+0000 7faaaa2bb640 1 Processor -- start 2023-10-16T14:13:19.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.562+0000 7faaaa2bb640 1 -- start start 2023-10-16T14:13:19.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.563+0000 7faaaa2bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 0x7faaa415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.563+0000 7faaaa2bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa415ed60 con 0x7faaa40ce390 2023-10-16T14:13:19.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.563+0000 7faaaa2bb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa40707e0 con 0x7faaa415ae80 2023-10-16T14:13:19.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.563+0000 7faaaa2bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa406b810 con 0x7faaa415aaa0 2023-10-16T14:13:19.569 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.564+0000 7faaa8858640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faaa40ce390 0x7faaa4070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37568/0 (socket says 172.21.15.60:37568) 2023-10-16T14:13:19.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.564+0000 7faaa8858640 1 -- 172.21.15.60:0/3874312819 learned_addr learned my addr 172.21.15.60:0/3874312819 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:19.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.564+0000 7faaa3fff640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 0x7faaa415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.564+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137610122 0 0) 0x7faaa415ed60 con 0x7faaa40ce390 2023-10-16T14:13:19.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.564+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c003520 con 0x7faaa40ce390 2023-10-16T14:13:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754937710 0 0) 0x7faa8c003520 con 0x7faaa40ce390 2023-10-16T14:13:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 msgr2=0x7faaa415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 0x7faaa415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 >> v1:172.21.15.144:6789/0 conn(0x7faaa415ae80 legacy=0x7faaa40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa415ef90 con 0x7faaa40ce390 2023-10-16T14:13:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7faa9c0019f0 con 0x7faaa40ce390 2023-10-16T14:13:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 >> v1:172.21.15.60:6789/0 conn(0x7faaa40ce390 legacy=0x7faaa4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 0x7faa8c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa8c003f70 0x7faa8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c004d00 0x7faa8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa406b810 con 0x7faa8c003f70 2023-10-16T14:13:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c00bac0 con 0x7faa8c004d00 2023-10-16T14:13:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c00bc80 con 0x7faaa415aaa0 2023-10-16T14:13:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.565+0000 7faaa37fe640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa8c003f70 0x7faa8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa3fff640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 0x7faa8c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa8858640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c004d00 0x7faa8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa37fe640 1 -- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 msgr2=0x7faa8c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa37fe640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa415aaa0 0x7faa8c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa37fe640 1 -- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c004d00 msgr2=0x7faa8c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:19.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa37fe640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c004d00 0x7faa8c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.566+0000 7faaa37fe640 1 -- 172.21.15.60:0/3874312819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c003520 con 0x7faa8c003f70 2023-10-16T14:13:19.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.567+0000 7faaa37fe640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa8c003f70 0x7faa8c0047c0 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7faa98001750 tx=0x7faa8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202a1347d407cd75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:19.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.567+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa980052d0 con 0x7faa8c003f70 2023-10-16T14:13:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.567+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9802e050 con 0x7faa8c003f70 2023-10-16T14:13:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.567+0000 7faaa2ffd640 1 -- 172.21.15.60:0/3874312819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98003830 con 0x7faa8c003f70 2023-10-16T14:13:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaaa2bb640 1 -- 172.21.15.60:0/3874312819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa8c003f70 msgr2=0x7faa8c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/3874312819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa8c003f70 0x7faa8c0047c0 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7faa98001750 tx=0x7faa8c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaa8858640 1 -- 172.21.15.60:0/3874312819 reap_dead start 2023-10-16T14:13:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaaa2bb640 1 -- 172.21.15.60:0/3874312819 shutdown_connections 2023-10-16T14:13:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaaa2bb640 1 -- 172.21.15.60:0/3874312819 >> 172.21.15.60:0/3874312819 conn(0x7faaa40cb410 msgr2=0x7faaa40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaaa2bb640 1 -- 172.21.15.60:0/3874312819 shutdown_connections 2023-10-16T14:13:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.568+0000 7faaaa2bb640 1 -- 172.21.15.60:0/3874312819 wait complete. 2023-10-16T14:13:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.570+0000 7faaaa2bb640 1 Processor -- start 2023-10-16T14:13:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.570+0000 7faaaa2bb640 1 -- start start 2023-10-16T14:13:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaaa2bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c003f70 0x7faaa4150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaaa2bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 0x7faaa40c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaaa2bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa40c4120 0x7faaa415ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaaa2bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa40c2250 con 0x7faaa40c3780 2023-10-16T14:13:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaaa2bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa4150790 con 0x7faa8c003f70 2023-10-16T14:13:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaaa2bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa40c4a70 con 0x7faaa40c4120 2023-10-16T14:13:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaa37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 0x7faaa40c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaa37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 0x7faaa40c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47442/0 (socket says 172.21.15.60:47442) 2023-10-16T14:13:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.571+0000 7faaa37fe640 1 -- 172.21.15.60:0/120209162 learned_addr learned my addr 172.21.15.60:0/120209162 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa3fff640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c003f70 0x7faaa4150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa37fe640 1 -- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa40c4120 msgr2=0x7faaa415ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa37fe640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa40c4120 0x7faaa415ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa37fe640 1 -- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c003f70 msgr2=0x7faaa4150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa37fe640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c003f70 0x7faaa4150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa37fe640 1 -- 172.21.15.60:0/120209162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa40c4d50 con 0x7faaa40c3780 2023-10-16T14:13:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa3fff640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c003f70 0x7faaa4150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa37fe640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 0x7faaa40c3be0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7faa98008fd0 tx=0x7faa98001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.572+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa980069e0 con 0x7faaa40c3780 2023-10-16T14:13:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.573+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98006b40 con 0x7faaa40c3780 2023-10-16T14:13:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.573+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa415c280 con 0x7faaa40c3780 2023-10-16T14:13:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.573+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa980035f0 con 0x7faaa40c3780 2023-10-16T14:13:19.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.573+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa415bf40 con 0x7faaa40c3780 2023-10-16T14:13:19.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.574+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa415c0d0 con 0x7faaa40c3780 2023-10-16T14:13:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.575+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa98003750 con 0x7faaa40c3780 2023-10-16T14:13:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.576+0000 7faaa0ff9640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7faa9006af90 0x7faa9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.576+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faa980ab500 con 0x7faaa40c3780 2023-10-16T14:13:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.577+0000 7faaa3fff640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7faa9006af90 0x7faa9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.578+0000 7faaa3fff640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7faa9006af90 0x7faa9006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa94009a00 tx=0x7faa94008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:19.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.580+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa98076760 con 0x7faaa40c3780 2023-10-16T14:13:19.781 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.778+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaa404e160 con 0x7faaa40c3780 2023-10-16T14:13:19.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.780+0000 7faaa0ff9640 1 -- 172.21.15.60:0/120209162 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v283) v1 ==== 74+0+12855 (secure 0 0 0) 0x7faa98079d80 con 0x7faaa40c3780 2023-10-16T14:13:19.784 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:19.784 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":283,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:18.752476+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:19.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.782+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7faa9006af90 msgr2=0x7faa9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.782+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7faa9006af90 0x7faa9006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa94009a00 tx=0x7faa94008040 comp rx=0 tx=0).stop 2023-10-16T14:13:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.782+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 msgr2=0x7faaa40c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.782+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 0x7faaa40c3be0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7faa98008fd0 tx=0x7faa98001120 comp rx=0 tx=0).stop 2023-10-16T14:13:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 shutdown_connections 2023-10-16T14:13:19.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7faa9006af90 0x7faa9006d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7faaa40c4120 0x7faaa415ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faaa40c3780 0x7faaa40c3be0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 --2- 172.21.15.60:0/120209162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa8c003f70 0x7faaa4150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 >> 172.21.15.60:0/120209162 conn(0x7faaa40cb410 msgr2=0x7faaa40b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:19.788 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 shutdown_connections 2023-10-16T14:13:19.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:19.783+0000 7faaaa2bb640 1 -- 172.21.15.60:0/120209162 wait complete. 2023-10-16T14:13:19.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:13:19.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:13:20.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.032+0000 7f14ee247640 1 Processor -- start 2023-10-16T14:13:20.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.032+0000 7f14ee247640 1 -- start start 2023-10-16T14:13:20.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.034+0000 7f14ee247640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14e80b3ec0 0x7f14e80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.034+0000 7f14ee247640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e806be30 con 0x7f14e80b43c0 2023-10-16T14:13:20.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.034+0000 7f14ee247640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80ada10 con 0x7f14e80b0ea0 2023-10-16T14:13:20.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.034+0000 7f14ee247640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e806ca00 con 0x7f14e80b3ec0 2023-10-16T14:13:20.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.034+0000 7f14eda46640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f14e80b43c0 0x7f14e815aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37572/0 (socket says 172.21.15.60:37572) 2023-10-16T14:13:20.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.034+0000 7f14eda46640 1 -- 172.21.15.60:0/2243094762 learned_addr learned my addr 172.21.15.60:0/2243094762 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:20.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478976376 0 0) 0x7f14e806be30 con 0x7f14e80b43c0 2023-10-16T14:13:20.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d4003520 con 0x7f14e80b43c0 2023-10-16T14:13:20.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14ed245640 1 --2- 172.21.15.60:0/2243094762 >> v2:172.21.15.144:3300/0 conn(0x7f14e80b3ec0 0x7f14e80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575697240 0 0) 0x7f14d4003520 con 0x7f14e80b43c0 2023-10-16T14:13:20.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 >> v2:172.21.15.144:3300/0 conn(0x7f14e80b3ec0 msgr2=0x7f14e80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:20.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 --2- 172.21.15.60:0/2243094762 >> v2:172.21.15.144:3300/0 conn(0x7f14e80b3ec0 0x7f14e80cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 >> v1:172.21.15.144:6789/0 conn(0x7f14e80b0ea0 legacy=0x7f14e815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.035+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e806c060 con 0x7f14e80b43c0 2023-10-16T14:13:20.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f14e00019f0 con 0x7f14e80b43c0 2023-10-16T14:13:20.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 >> v1:172.21.15.60:6789/0 conn(0x7f14e80b43c0 legacy=0x7f14e815aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14d4003970 0x7f14d4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d40047c0 0x7f14d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14d4005380 0x7f14d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e806ca00 con 0x7f14d4003970 2023-10-16T14:13:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d400c210 con 0x7f14d4005380 2023-10-16T14:13:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d400c3d0 con 0x7f14d40047c0 2023-10-16T14:13:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14ed245640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14d4003970 0x7f14d4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14eda46640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14d4005380 0x7f14d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14ed245640 1 -- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d40047c0 msgr2=0x7f14d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:20.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14ed245640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d40047c0 0x7f14d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14ed245640 1 -- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14d4005380 msgr2=0x7f14d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:20.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.036+0000 7f14ed245640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14d4005380 0x7f14d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.037+0000 7f14ed245640 1 -- 172.21.15.60:0/2243094762 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4003520 con 0x7f14d4003970 2023-10-16T14:13:20.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.037+0000 7f14eda46640 1 -- 172.21.15.60:0/2243094762 reap_dead start 2023-10-16T14:13:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.038+0000 7f14ed245640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14d4003970 0x7f14d4003d90 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f14e400ba30 tx=0x7f14d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa80b60b2664a09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.038+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4032640 con 0x7f14d4003970 2023-10-16T14:13:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.038+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e40327a0 con 0x7f14d4003970 2023-10-16T14:13:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.038+0000 7f14dffff640 1 -- 172.21.15.60:0/2243094762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4032b30 con 0x7f14d4003970 2023-10-16T14:13:20.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 -- 172.21.15.60:0/2243094762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14d4003970 msgr2=0x7f14d4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14d4003970 0x7f14d4003d90 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f14e400ba30 tx=0x7f14d40057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 -- 172.21.15.60:0/2243094762 shutdown_connections 2023-10-16T14:13:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 --2- 172.21.15.60:0/2243094762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14d4003970 0x7f14d4003d90 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 -- 172.21.15.60:0/2243094762 >> 172.21.15.60:0/2243094762 conn(0x7f14e80b0170 msgr2=0x7f14e80b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:20.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 -- 172.21.15.60:0/2243094762 shutdown_connections 2023-10-16T14:13:20.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.039+0000 7f14ee247640 1 -- 172.21.15.60:0/2243094762 wait complete. 2023-10-16T14:13:20.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.042+0000 7f14ee247640 1 Processor -- start 2023-10-16T14:13:20.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.042+0000 7f14ee247640 1 -- start start 2023-10-16T14:13:20.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14ee247640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d4003970 0x7f14e8150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14ee247640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14e80702c0 0x7f14e8070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14ee247640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 0x7f14e815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14ee247640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8070c80 con 0x7f14e815c310 2023-10-16T14:13:20.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14ee247640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e815ccd0 con 0x7f14e80702c0 2023-10-16T14:13:20.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14ee247640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c85a0 con 0x7f14d4003970 2023-10-16T14:13:20.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14eda46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 0x7f14e815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14eda46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 0x7f14e815c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47456/0 (socket says 172.21.15.60:47456) 2023-10-16T14:13:20.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.043+0000 7f14eda46640 1 -- 172.21.15.60:0/1763877454 learned_addr learned my addr 172.21.15.60:0/1763877454 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:20.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14ed245640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d4003970 0x7f14e8150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eda46640 1 -- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d4003970 msgr2=0x7f14e8150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eca44640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14e80702c0 0x7f14e8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eda46640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d4003970 0x7f14e8150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eda46640 1 -- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14e80702c0 msgr2=0x7f14e8070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eda46640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14e80702c0 0x7f14e8070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eda46640 1 -- 172.21.15.60:0/1763877454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e815df90 con 0x7f14e815c310 2023-10-16T14:13:20.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.044+0000 7f14eda46640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 0x7f14e815c790 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f14e0007730 tx=0x7f14e0001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.045+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e000ea50 con 0x7f14e815c310 2023-10-16T14:13:20.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.045+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e000ebb0 con 0x7f14e815c310 2023-10-16T14:13:20.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.045+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e815cef0 con 0x7f14e815c310 2023-10-16T14:13:20.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.045+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0014430 con 0x7f14e815c310 2023-10-16T14:13:20.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.045+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e815d4b0 con 0x7f14e815c310 2023-10-16T14:13:20.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.046+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e804e9b0 con 0x7f14e815c310 2023-10-16T14:13:20.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.047+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f14e0014590 con 0x7f14e815c310 2023-10-16T14:13:20.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.048+0000 7f14ddffb640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f14cc06afc0 0x7f14cc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.048+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f14e0089080 con 0x7f14e815c310 2023-10-16T14:13:20.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.049+0000 7f14ed245640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f14cc06afc0 0x7f14cc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.050+0000 7f14ed245640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f14cc06afc0 0x7f14cc06d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14e400ba00 tx=0x7f14e403d040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.053+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14e0054400 con 0x7f14e815c310 2023-10-16T14:13:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.246+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14e81510b0 con 0x7f14e815c310 2023-10-16T14:13:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.247+0000 7f14ddffb640 1 -- 172.21.15.60:0/1763877454 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v283) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14e0057a20 con 0x7f14e815c310 2023-10-16T14:13:20.251 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:13:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f14cc06afc0 msgr2=0x7f14cc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f14cc06afc0 0x7f14cc06d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f14e400ba00 tx=0x7f14e403d040 comp rx=0 tx=0).stop 2023-10-16T14:13:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 msgr2=0x7f14e815c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.254 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 0x7f14e815c790 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f14e0007730 tx=0x7f14e0001ad0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 shutdown_connections 2023-10-16T14:13:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f14cc06afc0 0x7f14cc06d480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14e815c310 0x7f14e815c790 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14e80702c0 0x7f14e8070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.255 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 --2- 172.21.15.60:0/1763877454 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f14d4003970 0x7f14e8150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 >> 172.21.15.60:0/1763877454 conn(0x7f14e80b0170 msgr2=0x7f14e80b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.250+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 shutdown_connections 2023-10-16T14:13:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.251+0000 7f14ee247640 1 -- 172.21.15.60:0/1763877454 wait complete. 2023-10-16T14:13:20.272 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T14:13:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.495+0000 7f791eb63640 1 Processor -- start 2023-10-16T14:13:20.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.495+0000 7f791eb63640 1 -- start start 2023-10-16T14:13:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791eb63640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7918159890 0x7f7918070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791eb63640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f79180710b0 0x7f79180afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791eb63640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918071510 con 0x7f79180b01e0 2023-10-16T14:13:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791eb63640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79180b1720 con 0x7f79180710b0 2023-10-16T14:13:20.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791eb63640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79180b1a00 con 0x7f7918159890 2023-10-16T14:13:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791e362640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f79180b01e0 0x7f791815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37588/0 (socket says 172.21.15.60:37588) 2023-10-16T14:13:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791e362640 1 -- 172.21.15.60:0/3342971552 learned_addr learned my addr 172.21.15.60:0/3342971552 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791d360640 1 --2- 172.21.15.60:0/3342971552 >> v2:172.21.15.60:3300/0 conn(0x7f79180710b0 0x7f79180afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.497+0000 7f791db61640 1 --2- 172.21.15.60:0/3342971552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7918159890 0x7f7918070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.498+0000 7f791cb5f640 1 -- 172.21.15.60:0/3342971552 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999222755 0 0) 0x7f7910001490 con 0x7f79180b01e0 2023-10-16T14:13:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.498+0000 7f791cb5f640 1 -- 172.21.15.60:0/3342971552 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7904003520 con 0x7f79180b01e0 2023-10-16T14:13:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.498+0000 7f791d360640 1 -- 172.21.15.60:0/3342971552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7918159890 msgr2=0x7f7918070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.498+0000 7f791d360640 1 --2- 172.21.15.60:0/3342971552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7918159890 0x7f7918070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.498+0000 7f791d360640 1 -- 172.21.15.60:0/3342971552 >> v1:172.21.15.60:6789/0 conn(0x7f79180b01e0 legacy=0x7f791815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.498+0000 7f791d360640 1 -- 172.21.15.60:0/3342971552 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180b1ce0 con 0x7f79180710b0 2023-10-16T14:13:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.499+0000 7f791d360640 1 --2- 172.21.15.60:0/3342971552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79180710b0 0x7f79180afca0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f79080097a0 tx=0x7f790802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842640c1c6340f87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.499+0000 7f791cb5f640 1 -- 172.21.15.60:0/3342971552 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790802d040 con 0x7f79180710b0 2023-10-16T14:13:20.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.499+0000 7f791cb5f640 1 -- 172.21.15.60:0/3342971552 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7908031810 con 0x7f79180710b0 2023-10-16T14:13:20.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.499+0000 7f791cb5f640 1 -- 172.21.15.60:0/3342971552 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908031ba0 con 0x7f79180710b0 2023-10-16T14:13:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 -- 172.21.15.60:0/3342971552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79180710b0 msgr2=0x7f79180afca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 --2- 172.21.15.60:0/3342971552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79180710b0 0x7f79180afca0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f79080097a0 tx=0x7f790802b8b0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 -- 172.21.15.60:0/3342971552 shutdown_connections 2023-10-16T14:13:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 --2- 172.21.15.60:0/3342971552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79180710b0 0x7f79180afca0 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 --2- 172.21.15.60:0/3342971552 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7918159890 0x7f7918070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 -- 172.21.15.60:0/3342971552 >> 172.21.15.60:0/3342971552 conn(0x7f7918069da0 msgr2=0x7f791806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 -- 172.21.15.60:0/3342971552 shutdown_connections 2023-10-16T14:13:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.500+0000 7f791eb63640 1 -- 172.21.15.60:0/3342971552 wait complete. 2023-10-16T14:13:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.502+0000 7f791eb63640 1 Processor -- start 2023-10-16T14:13:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.502+0000 7f791eb63640 1 -- start start 2023-10-16T14:13:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791eb63640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79180710b0 0x7f7918146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791eb63640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79180b01e0 0x7f79181502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791eb63640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 0x7f7918148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791eb63640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7918145480 con 0x7f7918159890 2023-10-16T14:13:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791eb63640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7918145760 con 0x7f79180b01e0 2023-10-16T14:13:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791eb63640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7918145a40 con 0x7f79180710b0 2023-10-16T14:13:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.503+0000 7f791e362640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 0x7f7918148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 0x7f7918148be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47474/0 (socket says 172.21.15.60:47474) 2023-10-16T14:13:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 -- 172.21.15.60:0/1177046790 learned_addr learned my addr 172.21.15.60:0/1177046790 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791db61640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79180710b0 0x7f7918146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791d360640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79180b01e0 0x7f79181502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 -- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79180710b0 msgr2=0x7f7918146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79180710b0 0x7f7918146520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 -- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79180b01e0 msgr2=0x7f79181502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79180b01e0 0x7f79181502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 -- 172.21.15.60:0/1177046790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7908009400 con 0x7f7918159890 2023-10-16T14:13:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791d360640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79180b01e0 0x7f79181502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791db61640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79180710b0 0x7f7918146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.504+0000 7f791e362640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 0x7f7918148be0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f791000cb40 tx=0x7f791000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.505+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791000aca0 con 0x7f7918159890 2023-10-16T14:13:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.505+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791000ae00 con 0x7f7918159890 2023-10-16T14:13:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.505+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910012620 con 0x7f7918159890 2023-10-16T14:13:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.505+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7918145d20 con 0x7f7918159890 2023-10-16T14:13:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.505+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7918150800 con 0x7f7918159890 2023-10-16T14:13:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.506+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79181509f0 con 0x7f7918159890 2023-10-16T14:13:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.507+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7910007070 con 0x7f7918159890 2023-10-16T14:13:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.509+0000 7f790e7fc640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f78fc06af90 0x7f78fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.509+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7910089320 con 0x7f7918159890 2023-10-16T14:13:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.510+0000 7f791db61640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f78fc06af90 0x7f78fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.511+0000 7f791db61640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f78fc06af90 0x7f78fc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7918150d30 tx=0x7f7914008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.514+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7910054600 con 0x7f7918159890 2023-10-16T14:13:20.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.710+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7918071510 con 0x7f7918159890 2023-10-16T14:13:20.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.711+0000 7f790e7fc640 1 -- 172.21.15.60:0/1177046790 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v283) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7910017060 con 0x7f7918159890 2023-10-16T14:13:20.715 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:20.715 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:13:20.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:20.716 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-10-16T14:13:20.716 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-10-16T14:13:20.716 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:20.716 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:13:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f78fc06af90 msgr2=0x7f78fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f78fc06af90 0x7f78fc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7918150d30 tx=0x7f7914008040 comp rx=0 tx=0).stop 2023-10-16T14:13:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 msgr2=0x7f7918148be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 0x7f7918148be0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f791000cb40 tx=0x7f791000a060 comp rx=0 tx=0).stop 2023-10-16T14:13:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 shutdown_connections 2023-10-16T14:13:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f78fc06af90 0x7f78fc06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7918159890 0x7f7918148be0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79180b01e0 0x7f79181502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 --2- 172.21.15.60:0/1177046790 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79180710b0 0x7f7918146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 >> 172.21.15.60:0/1177046790 conn(0x7f7918069da0 msgr2=0x7f79180cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:20.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 shutdown_connections 2023-10-16T14:13:20.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.714+0000 7f791eb63640 1 -- 172.21.15.60:0/1177046790 wait complete. 2023-10-16T14:13:20.738 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T14:13:20.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T14:13:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.959+0000 7fd776743640 1 Processor -- start 2023-10-16T14:13:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.960+0000 7fd776743640 1 -- start start 2023-10-16T14:13:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.961+0000 7fd776743640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7700b8480 0x7fd7700cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.961+0000 7fd776743640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77015d150 con 0x7fd77015cd70 2023-10-16T14:13:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.961+0000 7fd776743640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7700d1180 con 0x7fd7700d0be0 2023-10-16T14:13:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.961+0000 7fd776743640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7700d1460 con 0x7fd7700b8480 2023-10-16T14:13:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd774f40640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7700b8480 0x7fd7700cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd775741640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd77015cd70 0x7fd7700d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37602/0 (socket says 172.21.15.60:37602) 2023-10-16T14:13:20.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd775741640 1 -- 172.21.15.60:0/2539264318 learned_addr learned my addr 172.21.15.60:0/2539264318 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:20.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd774f40640 1 -- 172.21.15.60:0/2539264318 >> v1:172.21.15.144:6789/0 conn(0x7fd7700d0be0 legacy=0x7fd7700b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd767fff640 1 -- 172.21.15.60:0/2539264318 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097868343 0 0) 0x7fd77015d150 con 0x7fd77015cd70 2023-10-16T14:13:20.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd774f40640 1 -- 172.21.15.60:0/2539264318 >> v1:172.21.15.60:6789/0 conn(0x7fd77015cd70 legacy=0x7fd7700d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.962+0000 7fd774f40640 1 -- 172.21.15.60:0/2539264318 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7700d1740 con 0x7fd7700b8480 2023-10-16T14:13:20.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.964+0000 7fd774f40640 1 --2- 172.21.15.60:0/2539264318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd7700cd7c0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7fd7600097a0 tx=0x7fd76002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beeb56f39d88cf8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.964+0000 7fd767fff640 1 -- 172.21.15.60:0/2539264318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760031830 con 0x7fd7700b8480 2023-10-16T14:13:20.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.964+0000 7fd767fff640 1 -- 172.21.15.60:0/2539264318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760031990 con 0x7fd7700b8480 2023-10-16T14:13:20.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.964+0000 7fd767fff640 1 -- 172.21.15.60:0/2539264318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760031d20 con 0x7fd7700b8480 2023-10-16T14:13:20.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 -- 172.21.15.60:0/2539264318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 msgr2=0x7fd7700cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 --2- 172.21.15.60:0/2539264318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd7700cd7c0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7fd7600097a0 tx=0x7fd76002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 -- 172.21.15.60:0/2539264318 shutdown_connections 2023-10-16T14:13:20.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 --2- 172.21.15.60:0/2539264318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd7700cd7c0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 -- 172.21.15.60:0/2539264318 >> 172.21.15.60:0/2539264318 conn(0x7fd7700cf570 msgr2=0x7fd7700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:20.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 -- 172.21.15.60:0/2539264318 shutdown_connections 2023-10-16T14:13:20.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.965+0000 7fd776743640 1 -- 172.21.15.60:0/2539264318 wait complete. 2023-10-16T14:13:20.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.967+0000 7fd776743640 1 Processor -- start 2023-10-16T14:13:20.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.967+0000 7fd776743640 1 -- start start 2023-10-16T14:13:20.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd776743640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd77014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd776743640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7700d0be0 0x7fd7700cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd776743640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd77015cd70 0x7fd77015f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd776743640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77015d450 con 0x7fd7700b8480 2023-10-16T14:13:20.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd776743640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7701f5a60 con 0x7fd77015cd70 2023-10-16T14:13:20.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd776743640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7701f5d40 con 0x7fd7700d0be0 2023-10-16T14:13:20.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.968+0000 7fd775741640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd77014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd77014eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47492/0 (socket says 172.21.15.60:47492) 2023-10-16T14:13:20.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 -- 172.21.15.60:0/3354942420 learned_addr learned my addr 172.21.15.60:0/3354942420 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:20.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775f42640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd77015cd70 0x7fd77015f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd774f40640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7700d0be0 0x7fd7700cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 -- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7700d0be0 msgr2=0x7fd7700cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7700d0be0 0x7fd7700cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 -- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd77015cd70 msgr2=0x7fd77015f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:20.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd77015cd70 0x7fd77015f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:20.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 -- 172.21.15.60:0/3354942420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd760009400 con 0x7fd7700b8480 2023-10-16T14:13:20.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775f42640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd77015cd70 0x7fd77015f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:20.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.969+0000 7fd775741640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd77014eb00 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c8c0 tx=0x7fd76c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.970+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00a640 con 0x7fd7700b8480 2023-10-16T14:13:20.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.970+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c00a7a0 con 0x7fd7700b8480 2023-10-16T14:13:20.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.970+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00ab90 con 0x7fd7700b8480 2023-10-16T14:13:20.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.970+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7701f6020 con 0x7fd7700b8480 2023-10-16T14:13:20.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.970+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7701517f0 con 0x7fd7700b8480 2023-10-16T14:13:20.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.971+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77004e9b0 con 0x7fd7700b8480 2023-10-16T14:13:20.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.972+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd76c00acf0 con 0x7fd7700b8480 2023-10-16T14:13:20.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.973+0000 7fd765ffb640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd75406af90 0x7fd75406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:20.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.973+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd76c088a70 con 0x7fd7700b8480 2023-10-16T14:13:20.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.974+0000 7fd774f40640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd75406af90 0x7fd75406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:20.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.975+0000 7fd774f40640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd75406af90 0x7fd75406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd760036000 tx=0x7fd76003c040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:20.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:20.977+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd76c053d50 con 0x7fd7700b8480 2023-10-16T14:13:21.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.172+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd7700b5460 con 0x7fd7700b8480 2023-10-16T14:13:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.775+0000 7fd765ffb640 1 -- 172.21.15.60:0/3354942420 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v284) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd76c018070 con 0x7fd7700b8480 2023-10-16T14:13:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-10-16T14:13:21.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd75406af90 msgr2=0x7fd75406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:21.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd75406af90 0x7fd75406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd760036000 tx=0x7fd76003c040 comp rx=0 tx=0).stop 2023-10-16T14:13:21.783 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 msgr2=0x7fd77014eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd77014eb00 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00c8c0 tx=0x7fd76c00cd80 comp rx=0 tx=0).stop 2023-10-16T14:13:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 shutdown_connections 2023-10-16T14:13:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd75406af90 0x7fd75406d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:21.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd7700d0be0 0x7fd7700cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd77015cd70 0x7fd77015f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 --2- 172.21.15.60:0/3354942420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7700b8480 0x7fd77014eb00 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.779+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 >> 172.21.15.60:0/3354942420 conn(0x7fd7700cf570 msgr2=0x7fd77006bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.780+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 shutdown_connections 2023-10-16T14:13:21.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:21.780+0000 7fd776743640 1 -- 172.21.15.60:0/3354942420 wait complete. 2023-10-16T14:13:21.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T14:13:22.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.021+0000 7f7727815640 1 Processor -- start 2023-10-16T14:13:22.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.021+0000 7f7727815640 1 -- start start 2023-10-16T14:13:22.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.023+0000 7f7727815640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7720159b20 0x7f772006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.023+0000 7f7727815640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7720159f00 0x7f77200b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.023+0000 7f7727815640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772006f890 con 0x7f772015d8c0 2023-10-16T14:13:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.023+0000 7f7727815640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7720070d30 con 0x7f7720159b20 2023-10-16T14:13:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.023+0000 7f7727815640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772006c650 con 0x7f7720159f00 2023-10-16T14:13:22.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7720159b20 0x7f772006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:22.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7720159b20 0x7f772006bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47494/0 (socket says 172.21.15.60:47494) 2023-10-16T14:13:22.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 -- 172.21.15.60:0/347273651 learned_addr learned my addr 172.21.15.60:0/347273651 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:22.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f7724db0640 1 --2- 172.21.15.60:0/347273651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7720159f00 0x7f77200b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:22.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 -- 172.21.15.60:0/347273651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7720159f00 msgr2=0x7f77200b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:22.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 --2- 172.21.15.60:0/347273651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7720159f00 0x7f77200b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 -- 172.21.15.60:0/347273651 >> v1:172.21.15.144:6789/0 conn(0x7f772015d8c0 legacy=0x7f77200cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:22.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.024+0000 7f77255b1640 1 -- 172.21.15.60:0/347273651 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772006c930 con 0x7f7720159b20 2023-10-16T14:13:22.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.025+0000 7f77255b1640 1 --2- 172.21.15.60:0/347273651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f772006bbe0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f771c009870 tx=0x7f771c02db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528bb1795833d4e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:22.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.026+0000 7f7717fff640 1 -- 172.21.15.60:0/347273651 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c02f8c0 con 0x7f7720159b20 2023-10-16T14:13:22.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.026+0000 7f7717fff640 1 -- 172.21.15.60:0/347273651 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c02fa20 con 0x7f7720159b20 2023-10-16T14:13:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.026+0000 7f7717fff640 1 -- 172.21.15.60:0/347273651 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c02fdb0 con 0x7f7720159b20 2023-10-16T14:13:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.026+0000 7f7727815640 1 -- 172.21.15.60:0/347273651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 msgr2=0x7f772006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.026+0000 7f7727815640 1 --2- 172.21.15.60:0/347273651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f772006bbe0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f771c009870 tx=0x7f771c02db00 comp rx=0 tx=0).stop 2023-10-16T14:13:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.027+0000 7f7727815640 1 -- 172.21.15.60:0/347273651 shutdown_connections 2023-10-16T14:13:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.027+0000 7f7727815640 1 --2- 172.21.15.60:0/347273651 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7720159f00 0x7f77200b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.027+0000 7f7727815640 1 --2- 172.21.15.60:0/347273651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f772006bbe0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.027+0000 7f7727815640 1 -- 172.21.15.60:0/347273651 >> 172.21.15.60:0/347273651 conn(0x7f77200cb410 msgr2=0x7f77200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:22.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.027+0000 7f7727815640 1 -- 172.21.15.60:0/347273651 shutdown_connections 2023-10-16T14:13:22.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.027+0000 7f7727815640 1 -- 172.21.15.60:0/347273651 wait complete. 2023-10-16T14:13:22.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.029+0000 7f7727815640 1 Processor -- start 2023-10-16T14:13:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.029+0000 7f7727815640 1 -- start start 2023-10-16T14:13:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.030+0000 7f7727815640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f77201f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.030+0000 7f7727815640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7720159f00 0x7f77200b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.030+0000 7f7727815640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d8c0 0x7f77200b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.030+0000 7f7727815640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7720150c80 con 0x7f7720159b20 2023-10-16T14:13:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.030+0000 7f7727815640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200c5420 con 0x7f7720159f00 2023-10-16T14:13:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.030+0000 7f7727815640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77201518b0 con 0x7f772015d8c0 2023-10-16T14:13:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f77201f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f77201f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47496/0 (socket says 172.21.15.60:47496) 2023-10-16T14:13:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 -- 172.21.15.60:0/3097723436 learned_addr learned my addr 172.21.15.60:0/3097723436 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 -- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d8c0 msgr2=0x7f77200b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d8c0 0x7f77200b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 -- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7720159f00 msgr2=0x7f77200b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7720159f00 0x7f77200b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.031+0000 7f77255b1640 1 -- 172.21.15.60:0/3097723436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c0094d0 con 0x7f7720159b20 2023-10-16T14:13:22.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.032+0000 7f77255b1640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f77201f2200 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f771c000c00 tx=0x7f771c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.032+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c042880 con 0x7f7720159b20 2023-10-16T14:13:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.032+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c0429e0 con 0x7f7720159b20 2023-10-16T14:13:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.032+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c042d90 con 0x7f7720159b20 2023-10-16T14:13:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.032+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7720150ed0 con 0x7f7720159b20 2023-10-16T14:13:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.032+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77200c4c20 con 0x7f7720159b20 2023-10-16T14:13:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.033+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772004e160 con 0x7f7720159b20 2023-10-16T14:13:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.035+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f771c040440 con 0x7f7720159b20 2023-10-16T14:13:22.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.036+0000 7f7715ffb640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f770406af90 0x7f770406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:22.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.036+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f771c033070 con 0x7f7720159b20 2023-10-16T14:13:22.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.036+0000 7f7724db0640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f770406af90 0x7f770406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:22.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.037+0000 7f7724db0640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f770406af90 0x7f770406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77200ad590 tx=0x7f7710008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:22.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.040+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f771c077850 con 0x7f7720159b20 2023-10-16T14:13:22.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.238+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f772006cbe0 con 0x7f7720159b20 2023-10-16T14:13:22.786 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.782+0000 7f7715ffb640 1 -- 172.21.15.60:0/3097723436 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v285) v1 ==== 170+0+0 (secure 0 0 0) 0x7f771c07ae70 con 0x7f7720159b20 2023-10-16T14:13:22.786 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-10-16T14:13:22.789 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.786+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f770406af90 msgr2=0x7f770406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:22.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.786+0000 7f7727815640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f770406af90 0x7f770406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77200ad590 tx=0x7f7710008040 comp rx=0 tx=0).stop 2023-10-16T14:13:22.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.786+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 msgr2=0x7f77201f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:22.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.786+0000 7f7727815640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f77201f2200 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f771c000c00 tx=0x7f771c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 shutdown_connections 2023-10-16T14:13:22.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f770406af90 0x7f770406d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f772015d8c0 0x7f77200b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7720159f00 0x7f77200b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 --2- 172.21.15.60:0/3097723436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7720159b20 0x7f77201f2200 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 >> 172.21.15.60:0/3097723436 conn(0x7f77200cb410 msgr2=0x7f772015dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.787+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 shutdown_connections 2023-10-16T14:13:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:22.788+0000 7f7727815640 1 -- 172.21.15.60:0/3097723436 wait complete. 2023-10-16T14:13:22.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T14:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.015+0000 7f8ae7d41640 1 Processor -- start 2023-10-16T14:13:23.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.015+0000 7f8ae7d41640 1 -- start start 2023-10-16T14:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.016+0000 7f8ae7d41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae015d150 0x7f8ae00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.016+0000 7f8ae7d41640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ae0160b90 0x7f8ae0160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.016+0000 7f8ae7d41640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8ae0069da0 0x7f8ae006a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.016+0000 7f8ae7d41640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae015d530 con 0x7f8ae0160b90 2023-10-16T14:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.016+0000 7f8ae7d41640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae006a6f0 con 0x7f8ae0069da0 2023-10-16T14:13:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.016+0000 7f8ae7d41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00ac890 con 0x7f8ae015d150 2023-10-16T14:13:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.017+0000 7f8ae52dc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ae0160b90 0x7f8ae0160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.017+0000 7f8ae52dc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ae0160b90 0x7f8ae0160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47500/0 (socket says 172.21.15.60:47500) 2023-10-16T14:13:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.017+0000 7f8ae52dc640 1 -- 172.21.15.60:0/3536225461 learned_addr learned my addr 172.21.15.60:0/3536225461 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.017+0000 7f8ae5add640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae015d150 0x7f8ae00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.017+0000 7f8ae62de640 1 --2- 172.21.15.60:0/3536225461 >> v2:172.21.15.144:3300/0 conn(0x7f8ae0069da0 0x7f8ae006a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae52dc640 1 -- 172.21.15.60:0/3536225461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae015d150 msgr2=0x7f8ae00ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae52dc640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae015d150 0x7f8ae00ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae52dc640 1 -- 172.21.15.60:0/3536225461 >> v2:172.21.15.144:3300/0 conn(0x7f8ae0069da0 msgr2=0x7f8ae006a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae52dc640 1 --2- 172.21.15.60:0/3536225461 >> v2:172.21.15.144:3300/0 conn(0x7f8ae0069da0 0x7f8ae006a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae52dc640 1 -- 172.21.15.60:0/3536225461 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0161650 con 0x7f8ae0160b90 2023-10-16T14:13:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae62de640 1 --2- 172.21.15.60:0/3536225461 >> v2:172.21.15.144:3300/0 conn(0x7f8ae0069da0 0x7f8ae006a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.018+0000 7f8ae5add640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae015d150 0x7f8ae00ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.019+0000 7f8ae52dc640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae0160fb0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00cf990 tx=0x7f8ad002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c43107f9cfaa81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.019+0000 7f8ae4adb640 1 -- 172.21.15.60:0/3536225461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0031870 con 0x7f8ae0160b90 2023-10-16T14:13:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.019+0000 7f8ae4adb640 1 -- 172.21.15.60:0/3536225461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad00319d0 con 0x7f8ae0160b90 2023-10-16T14:13:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.019+0000 7f8ae4adb640 1 -- 172.21.15.60:0/3536225461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0031d60 con 0x7f8ae0160b90 2023-10-16T14:13:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 -- 172.21.15.60:0/3536225461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 msgr2=0x7f8ae0160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae0160fb0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00cf990 tx=0x7f8ad002b910 comp rx=0 tx=0).stop 2023-10-16T14:13:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 -- 172.21.15.60:0/3536225461 shutdown_connections 2023-10-16T14:13:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/3536225461 >> v2:172.21.15.144:3300/0 conn(0x7f8ae0069da0 0x7f8ae006a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae0160fb0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/3536225461 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae015d150 0x7f8ae00ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 -- 172.21.15.60:0/3536225461 >> 172.21.15.60:0/3536225461 conn(0x7f8ae00ce9b0 msgr2=0x7f8ae00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 -- 172.21.15.60:0/3536225461 shutdown_connections 2023-10-16T14:13:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.020+0000 7f8ae7d41640 1 -- 172.21.15.60:0/3536225461 wait complete. 2023-10-16T14:13:23.029 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.022+0000 7f8ae7d41640 1 Processor -- start 2023-10-16T14:13:23.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.022+0000 7f8ae7d41640 1 -- start start 2023-10-16T14:13:23.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae7d41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae0069da0 0x7f8ae00b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae7d41640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ae015d150 0x7f8ae01f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae7d41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae00c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae7d41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01f6e20 con 0x7f8ae0160b90 2023-10-16T14:13:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae7d41640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00af6b0 con 0x7f8ae015d150 2023-10-16T14:13:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae7d41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00c2a20 con 0x7f8ae0069da0 2023-10-16T14:13:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae62de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae00c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae62de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae00c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47514/0 (socket says 172.21.15.60:47514) 2023-10-16T14:13:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.023+0000 7f8ae62de640 1 -- 172.21.15.60:0/1125169022 learned_addr learned my addr 172.21.15.60:0/1125169022 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:23.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae5add640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae0069da0 0x7f8ae00b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae52dc640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ae015d150 0x7f8ae01f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae62de640 1 -- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae0069da0 msgr2=0x7f8ae00b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:23.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae62de640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae0069da0 0x7f8ae00b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae62de640 1 -- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ae015d150 msgr2=0x7f8ae01f5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:23.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae62de640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ae015d150 0x7f8ae01f5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae62de640 1 -- 172.21.15.60:0/1125169022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00094d0 con 0x7f8ae0160b90 2023-10-16T14:13:23.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.024+0000 7f8ae62de640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae00c24e0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8007e10 tx=0x7f8ad800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:23.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.025+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8003b90 con 0x7f8ae0160b90 2023-10-16T14:13:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.025+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad800f040 con 0x7f8ae0160b90 2023-10-16T14:13:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.025+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800de80 con 0x7f8ae0160b90 2023-10-16T14:13:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.025+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01e77a0 con 0x7f8ae0160b90 2023-10-16T14:13:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.025+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01f6860 con 0x7f8ae0160b90 2023-10-16T14:13:23.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.026+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae004e160 con 0x7f8ae0160b90 2023-10-16T14:13:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.027+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8ad8004030 con 0x7f8ae0160b90 2023-10-16T14:13:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.028+0000 7f8ad67fc640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8ac406af90 0x7f8ac406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.028+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(285..285 src has 1..285) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8ad8089ad0 con 0x7f8ae0160b90 2023-10-16T14:13:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.029+0000 7f8ae5add640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8ac406af90 0x7f8ac406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:23.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.030+0000 7f8ae5add640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8ac406af90 0x7f8ac406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8adc006fd0 tx=0x7f8adc008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:23.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.033+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8ad80549b0 con 0x7f8ae0160b90 2023-10-16T14:13:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.268+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8ae00ace00 con 0x7f8ae0160b90 2023-10-16T14:13:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.893+0000 7f8ad67fc640 1 -- 172.21.15.60:0/1125169022 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 36 and data pool 37 v148) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8ad801a0c0 con 0x7f8ae0160b90 2023-10-16T14:13:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 36 and data pool 37 2023-10-16T14:13:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8ac406af90 msgr2=0x7f8ac406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8ac406af90 0x7f8ac406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8adc006fd0 tx=0x7f8adc008040 comp rx=0 tx=0).stop 2023-10-16T14:13:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 msgr2=0x7f8ae00c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae00c24e0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8007e10 tx=0x7f8ad800d4a0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 shutdown_connections 2023-10-16T14:13:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8ac406af90 0x7f8ac406d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ae0160b90 0x7f8ae00c24e0 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ae015d150 0x7f8ae01f5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 --2- 172.21.15.60:0/1125169022 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ae0069da0 0x7f8ae00b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 >> 172.21.15.60:0/1125169022 conn(0x7f8ae00ce9b0 msgr2=0x7f8ae00b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.897+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 shutdown_connections 2023-10-16T14:13:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:23.898+0000 7f8ae7d41640 1 -- 172.21.15.60:0/1125169022 wait complete. 2023-10-16T14:13:23.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:24.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.113+0000 7fb646d41640 1 Processor -- start 2023-10-16T14:13:24.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.114+0000 7fb646d41640 1 -- start start 2023-10-16T14:13:24.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.115+0000 7fb646d41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6400b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.115+0000 7fb646d41640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb64015d880 0x7fb6400b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.115+0000 7fb646d41640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64006f7f0 con 0x7fb640159b20 2023-10-16T14:13:24.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.115+0000 7fb646d41640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64006c5e0 con 0x7fb64015d880 2023-10-16T14:13:24.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.115+0000 7fb646d41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64006c8c0 con 0x7fb640159f00 2023-10-16T14:13:24.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.116+0000 7fb63ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6400b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.116+0000 7fb63ffff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6400b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:35662/0 (socket says 172.21.15.60:35662) 2023-10-16T14:13:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.116+0000 7fb63ffff640 1 -- 172.21.15.60:0/2501942001 learned_addr learned my addr 172.21.15.60:0/2501942001 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.116+0000 7fb6452de640 1 --2- 172.21.15.60:0/2501942001 >> v2:172.21.15.144:3300/0 conn(0x7fb64015d880 0x7fb6400b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.117+0000 7fb63ffff640 1 -- 172.21.15.60:0/2501942001 >> v2:172.21.15.144:3300/0 conn(0x7fb64015d880 msgr2=0x7fb6400b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:24.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.117+0000 7fb63ffff640 1 --2- 172.21.15.60:0/2501942001 >> v2:172.21.15.144:3300/0 conn(0x7fb64015d880 0x7fb6400b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.117+0000 7fb63ffff640 1 -- 172.21.15.60:0/2501942001 >> v1:172.21.15.144:6789/0 conn(0x7fb640159b20 legacy=0x7fb64006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.117+0000 7fb63ffff640 1 -- 172.21.15.60:0/2501942001 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64006cba0 con 0x7fb640159f00 2023-10-16T14:13:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.118+0000 7fb63ffff640 1 --2- 172.21.15.60:0/2501942001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6400b0410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009870 tx=0x7fb62c02db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b60d5079702210e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.119+0000 7fb63f7fe640 1 -- 172.21.15.60:0/2501942001 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c02e940 con 0x7fb640159f00 2023-10-16T14:13:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.119+0000 7fb63f7fe640 1 -- 172.21.15.60:0/2501942001 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c02eaa0 con 0x7fb640159f00 2023-10-16T14:13:24.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.119+0000 7fb63f7fe640 1 -- 172.21.15.60:0/2501942001 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c02ee30 con 0x7fb640159f00 2023-10-16T14:13:24.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 -- 172.21.15.60:0/2501942001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 msgr2=0x7fb6400b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 --2- 172.21.15.60:0/2501942001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6400b0410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009870 tx=0x7fb62c02db90 comp rx=0 tx=0).stop 2023-10-16T14:13:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 -- 172.21.15.60:0/2501942001 shutdown_connections 2023-10-16T14:13:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 --2- 172.21.15.60:0/2501942001 >> v2:172.21.15.144:3300/0 conn(0x7fb64015d880 0x7fb6400b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 --2- 172.21.15.60:0/2501942001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6400b0410 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 -- 172.21.15.60:0/2501942001 >> 172.21.15.60:0/2501942001 conn(0x7fb6400cb410 msgr2=0x7fb6400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 -- 172.21.15.60:0/2501942001 shutdown_connections 2023-10-16T14:13:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.120+0000 7fb646d41640 1 -- 172.21.15.60:0/2501942001 wait complete. 2023-10-16T14:13:24.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.122+0000 7fb646d41640 1 Processor -- start 2023-10-16T14:13:24.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.123+0000 7fb646d41640 1 -- start start 2023-10-16T14:13:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb646d41640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 0x7fb6400b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb646d41640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6401f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb646d41640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb64015d880 0x7fb640158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb646d41640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400b4200 con 0x7fb640159b20 2023-10-16T14:13:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb646d41640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400b3160 con 0x7fb64015d880 2023-10-16T14:13:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb646d41640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400b3440 con 0x7fb640159f00 2023-10-16T14:13:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb644add640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 0x7fb6400b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb644add640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 0x7fb6400b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47520/0 (socket says 172.21.15.60:47520) 2023-10-16T14:13:24.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.124+0000 7fb644add640 1 -- 172.21.15.60:0/526690160 learned_addr learned my addr 172.21.15.60:0/526690160 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb6452de640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb64015d880 0x7fb640158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb63ffff640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6401f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb644add640 1 -- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 msgr2=0x7fb6401f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb644add640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6401f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb644add640 1 -- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb64015d880 msgr2=0x7fb640158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb644add640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb64015d880 0x7fb640158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb644add640 1 -- 172.21.15.60:0/526690160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0094d0 con 0x7fb640159b20 2023-10-16T14:13:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.125+0000 7fb644add640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 0x7fb6400b12d0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c400 tx=0x7fb63400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:24.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.126+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6340052d0 con 0x7fb640159b20 2023-10-16T14:13:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.126+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634005430 con 0x7fb640159b20 2023-10-16T14:13:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.126+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634003830 con 0x7fb640159b20 2023-10-16T14:13:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.126+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6400b3720 con 0x7fb640159b20 2023-10-16T14:13:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.126+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6400b3ab0 con 0x7fb640159b20 2023-10-16T14:13:24.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.127+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6400b3f20 con 0x7fb640159b20 2023-10-16T14:13:24.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.128+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb634003990 con 0x7fb640159b20 2023-10-16T14:13:24.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.129+0000 7fb63d7fa640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb62406af90 0x7fb62406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.130+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb634088db0 con 0x7fb640159b20 2023-10-16T14:13:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.130+0000 7fb63ffff640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb62406af90 0x7fb62406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.131+0000 7fb63ffff640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb62406af90 0x7fb62406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb62c038000 tx=0x7fb62c032000 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:24.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.134+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb6400b3f20 con 0x7fb640159b20 2023-10-16T14:13:24.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.332+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6400b3f20 con 0x7fb640159b20 2023-10-16T14:13:24.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.336+0000 7fb63d7fa640 1 -- 172.21.15.60:0/526690160 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+16528 (secure 0 0 0) 0x7fb6400b3f20 con 0x7fb640159b20 2023-10-16T14:13:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:24.340 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":286,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:23.270947+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb62406af90 msgr2=0x7fb62406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb62406af90 0x7fb62406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb62c038000 tx=0x7fb62c032000 comp rx=0 tx=0).stop 2023-10-16T14:13:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 msgr2=0x7fb6400b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 0x7fb6400b12d0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c400 tx=0x7fb63400c8c0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 shutdown_connections 2023-10-16T14:13:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fb62406af90 0x7fb62406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb640159f00 0x7fb6401f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb64015d880 0x7fb640158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 --2- 172.21.15.60:0/526690160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb640159b20 0x7fb6400b12d0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 >> 172.21.15.60:0/526690160 conn(0x7fb6400cb410 msgr2=0x7fb6400cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:24.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 shutdown_connections 2023-10-16T14:13:24.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.339+0000 7fb646d41640 1 -- 172.21.15.60:0/526690160 wait complete. 2023-10-16T14:13:24.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:24.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.573+0000 7f016627e640 1 Processor -- start 2023-10-16T14:13:24.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.573+0000 7f016627e640 1 -- start start 2023-10-16T14:13:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016627e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0160144440 0x7f016015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016627e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f016015f1b0 0x7f016015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016627e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016006f1d0 con 0x7f0160144060 2023-10-16T14:13:24.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016627e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600cd430 con 0x7f0160144440 2023-10-16T14:13:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016627e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600cd610 con 0x7f016015f1b0 2023-10-16T14:13:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016527c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0160144060 0x7f01600b0a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37618/0 (socket says 172.21.15.60:37618) 2023-10-16T14:13:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f0164a7b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0160144440 0x7f016015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.575+0000 7f016527c640 1 -- 172.21.15.60:0/835591680 learned_addr learned my addr 172.21.15.60:0/835591680 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.576+0000 7f0165a7d640 1 --2- 172.21.15.60:0/835591680 >> v2:172.21.15.144:3300/0 conn(0x7f016015f1b0 0x7f016015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.576+0000 7f0164a7b640 1 -- 172.21.15.60:0/835591680 >> v2:172.21.15.144:3300/0 conn(0x7f016015f1b0 msgr2=0x7f016015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.576+0000 7f0164a7b640 1 --2- 172.21.15.60:0/835591680 >> v2:172.21.15.144:3300/0 conn(0x7f016015f1b0 0x7f016015eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.576+0000 7f0164a7b640 1 -- 172.21.15.60:0/835591680 >> v1:172.21.15.60:6789/0 conn(0x7f0160144060 legacy=0x7f01600b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.576+0000 7f0164a7b640 1 -- 172.21.15.60:0/835591680 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600cd8f0 con 0x7f0160144440 2023-10-16T14:13:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.577+0000 7f0164a7b640 1 --2- 172.21.15.60:0/835591680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144440 0x7f016015e5a0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f01600b1150 tx=0x7f014c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad3188d4697ebc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.577+0000 7f015ffff640 1 -- 172.21.15.60:0/835591680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c02e070 con 0x7f0160144440 2023-10-16T14:13:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.577+0000 7f015ffff640 1 -- 172.21.15.60:0/835591680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0315a0 con 0x7f0160144440 2023-10-16T14:13:24.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.577+0000 7f015ffff640 1 -- 172.21.15.60:0/835591680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c031930 con 0x7f0160144440 2023-10-16T14:13:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 -- 172.21.15.60:0/835591680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144440 msgr2=0x7f016015e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 --2- 172.21.15.60:0/835591680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144440 0x7f016015e5a0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f01600b1150 tx=0x7f014c02b840 comp rx=0 tx=0).stop 2023-10-16T14:13:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 -- 172.21.15.60:0/835591680 shutdown_connections 2023-10-16T14:13:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 --2- 172.21.15.60:0/835591680 >> v2:172.21.15.144:3300/0 conn(0x7f016015f1b0 0x7f016015eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 --2- 172.21.15.60:0/835591680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144440 0x7f016015e5a0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 -- 172.21.15.60:0/835591680 >> 172.21.15.60:0/835591680 conn(0x7f0160069da0 msgr2=0x7f016006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:24.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 -- 172.21.15.60:0/835591680 shutdown_connections 2023-10-16T14:13:24.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.578+0000 7f016627e640 1 -- 172.21.15.60:0/835591680 wait complete. 2023-10-16T14:13:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.580+0000 7f016627e640 1 Processor -- start 2023-10-16T14:13:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.580+0000 7f016627e640 1 -- start start 2023-10-16T14:13:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016627e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 0x7f016013f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016627e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0160144440 0x7f0160158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016627e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f016015f1b0 0x7f0160159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016627e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01601f5eb0 con 0x7f0160144060 2023-10-16T14:13:24.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016627e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01601f6190 con 0x7f0160144440 2023-10-16T14:13:24.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016627e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01601f6470 con 0x7f016015f1b0 2023-10-16T14:13:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016527c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 0x7f016013f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016527c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 0x7f016013f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47538/0 (socket says 172.21.15.60:47538) 2023-10-16T14:13:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.581+0000 7f016527c640 1 -- 172.21.15.60:0/24512661 learned_addr learned my addr 172.21.15.60:0/24512661 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f0164a7b640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0160144440 0x7f0160158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016527c640 1 -- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f016015f1b0 msgr2=0x7f0160159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016527c640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f016015f1b0 0x7f0160159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016527c640 1 -- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0160144440 msgr2=0x7f0160158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016527c640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0160144440 0x7f0160158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016527c640 1 -- 172.21.15.60:0/24512661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c009400 con 0x7f0160144060 2023-10-16T14:13:24.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016527c640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 0x7f016013f8a0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f015400cba0 tx=0x7f015400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:24.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015400ad40 con 0x7f0160144060 2023-10-16T14:13:24.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154012400 con 0x7f0160144060 2023-10-16T14:13:24.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015400aea0 con 0x7f0160144060 2023-10-16T14:13:24.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.582+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01601f6750 con 0x7f0160144060 2023-10-16T14:13:24.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.583+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0160141e90 con 0x7f0160144060 2023-10-16T14:13:24.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.584+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01600af380 con 0x7f0160144060 2023-10-16T14:13:24.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.585+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0154007070 con 0x7f0160144060 2023-10-16T14:13:24.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.586+0000 7f015dffb640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f014406af90 0x7f014406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:24.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.586+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01540895e0 con 0x7f0160144060 2023-10-16T14:13:24.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.586+0000 7f0164a7b640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f014406af90 0x7f014406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:24.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.588+0000 7f0164a7b640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f014406af90 0x7f014406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f014c036000 tx=0x7f014c03d040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:24.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.591+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f01540546c0 con 0x7f0160144060 2023-10-16T14:13:24.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.788+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01600674f0 con 0x7f0160144060 2023-10-16T14:13:24.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.791+0000 7f015dffb640 1 -- 172.21.15.60:0/24512661 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) v1 ==== 74+0+16528 (secure 0 0 0) 0x7f0154057ce0 con 0x7f0160144060 2023-10-16T14:13:24.795 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:24.795 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":286,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:23.270947+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:24.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f014406af90 msgr2=0x7f014406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f014406af90 0x7f014406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f014c036000 tx=0x7f014c03d040 comp rx=0 tx=0).stop 2023-10-16T14:13:24.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 msgr2=0x7f016013f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:24.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 0x7f016013f8a0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f015400cba0 tx=0x7f015400a430 comp rx=0 tx=0).stop 2023-10-16T14:13:24.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 shutdown_connections 2023-10-16T14:13:24.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f014406af90 0x7f014406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f016015f1b0 0x7f0160159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0160144440 0x7f0160158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 --2- 172.21.15.60:0/24512661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0160144060 0x7f016013f8a0 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:24.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 >> 172.21.15.60:0/24512661 conn(0x7f0160069da0 msgr2=0x7f016006a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:24.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 shutdown_connections 2023-10-16T14:13:24.800 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:24.794+0000 7f016627e640 1 -- 172.21.15.60:0/24512661 wait complete. 2023-10-16T14:13:24.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T14:13:25.043 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.040+0000 7f7b7cbfb640 1 Processor -- start 2023-10-16T14:13:25.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.040+0000 7f7b7cbfb640 1 -- start start 2023-10-16T14:13:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.041+0000 7f7b7cbfb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b7815ad70 0x7f7b78069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b7cbfb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780cb030 con 0x7f7b780c9ca0 2023-10-16T14:13:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b7cbfb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780c91c0 con 0x7f7b780caac0 2023-10-16T14:13:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b7cbfb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b780c94a0 con 0x7f7b7815ad70 2023-10-16T14:13:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b6ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7b780c9ca0 0x7f7b780ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37624/0 (socket says 172.21.15.60:37624) 2023-10-16T14:13:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b6ffff640 1 -- 172.21.15.60:0/4219025325 learned_addr learned my addr 172.21.15.60:0/4219025325 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b7659c640 1 --2- 172.21.15.60:0/4219025325 >> v2:172.21.15.144:3300/0 conn(0x7f7b7815ad70 0x7f7b78069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.042+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732235508 0 0) 0x7f7b780cb030 con 0x7f7b780c9ca0 2023-10-16T14:13:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64003520 con 0x7f7b780c9ca0 2023-10-16T14:13:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1222773123 0 0) 0x7f7b64003520 con 0x7f7b780c9ca0 2023-10-16T14:13:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 >> v2:172.21.15.144:3300/0 conn(0x7f7b7815ad70 msgr2=0x7f7b78069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 --2- 172.21.15.60:0/4219025325 >> v2:172.21.15.144:3300/0 conn(0x7f7b7815ad70 0x7f7b78069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 >> v1:172.21.15.144:6789/0 conn(0x7f7b780caac0 legacy=0x7f7b780ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780c9780 con 0x7f7b780c9ca0 2023-10-16T14:13:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7b60001670 con 0x7f7b780c9ca0 2023-10-16T14:13:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 >> v1:172.21.15.60:6789/0 conn(0x7f7b780c9ca0 legacy=0x7f7b780ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b640039c0 0x7f7b64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b640047c0 0x7f7b64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b64005180 0x7f7b6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780c94a0 con 0x7f7b64005180 2023-10-16T14:13:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400c080 con 0x7f7b640039c0 2023-10-16T14:13:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.043+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400c240 con 0x7f7b640047c0 2023-10-16T14:13:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b64005180 0x7f7b6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b7659c640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b640039c0 0x7f7b64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b6ffff640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b640047c0 0x7f7b64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b76d9d640 1 -- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b640047c0 msgr2=0x7f7b64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b640047c0 0x7f7b64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b76d9d640 1 -- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b640039c0 msgr2=0x7f7b64003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b640039c0 0x7f7b64003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.044+0000 7f7b76d9d640 1 -- 172.21.15.60:0/4219025325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64003520 con 0x7f7b64005180 2023-10-16T14:13:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.045+0000 7f7b76d9d640 1 -- 172.21.15.60:0/4219025325 reap_dead start 2023-10-16T14:13:25.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.046+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b64005180 0x7f7b6400b980 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f7b70001e50 tx=0x7f7b640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f69b63434cba0b1b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.046+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70003b70 con 0x7f7b64005180 2023-10-16T14:13:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.046+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b70003d10 con 0x7f7b64005180 2023-10-16T14:13:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.046+0000 7f7b75d9b640 1 -- 172.21.15.60:0/4219025325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70005310 con 0x7f7b64005180 2023-10-16T14:13:25.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/4219025325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b64005180 msgr2=0x7f7b6400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b64005180 0x7f7b6400b980 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f7b70001e50 tx=0x7f7b640057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/4219025325 shutdown_connections 2023-10-16T14:13:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/4219025325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b64005180 0x7f7b6400b980 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/4219025325 >> 172.21.15.60:0/4219025325 conn(0x7f7b780ce9b0 msgr2=0x7f7b780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/4219025325 shutdown_connections 2023-10-16T14:13:25.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.047+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/4219025325 wait complete. 2023-10-16T14:13:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.049+0000 7f7b7cbfb640 1 Processor -- start 2023-10-16T14:13:25.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.049+0000 7f7b7cbfb640 1 -- start start 2023-10-16T14:13:25.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b7cbfb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b64005180 0x7f7b780af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b7cbfb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815d6e0 0x7f7b78153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b7cbfb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 0x7f7b7806a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b7cbfb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7806a760 con 0x7f7b78069da0 2023-10-16T14:13:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b7cbfb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78153560 con 0x7f7b64005180 2023-10-16T14:13:25.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b7cbfb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7815e120 con 0x7f7b7815d6e0 2023-10-16T14:13:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b76d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 0x7f7b7806a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b76d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 0x7f7b7806a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47546/0 (socket says 172.21.15.60:47546) 2023-10-16T14:13:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.050+0000 7f7b76d9d640 1 -- 172.21.15.60:0/3723674874 learned_addr learned my addr 172.21.15.60:0/3723674874 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b76d9d640 1 -- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815d6e0 msgr2=0x7f7b78153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815d6e0 0x7f7b78153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b76d9d640 1 -- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b64005180 msgr2=0x7f7b780af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b64005180 0x7f7b780af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b76d9d640 1 -- 172.21.15.60:0/3723674874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780cdd70 con 0x7f7b78069da0 2023-10-16T14:13:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b76d9d640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 0x7f7b7806a220 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f7b70001f70 tx=0x7f7b700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70006d50 con 0x7f7b78069da0 2023-10-16T14:13:25.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b70007db0 con 0x7f7b78069da0 2023-10-16T14:13:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70007270 con 0x7f7b78069da0 2023-10-16T14:13:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7815de40 con 0x7f7b78069da0 2023-10-16T14:13:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.051+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b780cd570 con 0x7f7b78069da0 2023-10-16T14:13:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.053+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7804e160 con 0x7f7b78069da0 2023-10-16T14:13:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.054+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b700073d0 con 0x7f7b78069da0 2023-10-16T14:13:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.055+0000 7f7b6effd640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7b5806af90 0x7f7b5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:25.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.055+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b700ab6e0 con 0x7f7b78069da0 2023-10-16T14:13:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.055+0000 7f7b7659c640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7b5806af90 0x7f7b5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.056+0000 7f7b7659c640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7b5806af90 0x7f7b5806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7b780b3610 tx=0x7f7b68009320 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.059+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b700764d0 con 0x7f7b78069da0 2023-10-16T14:13:25.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.297+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7b7806ab10 con 0x7f7b78069da0 2023-10-16T14:13:25.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.907+0000 7f7b6effd640 1 -- 172.21.15.60:0/3723674874 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v151) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7b70079af0 con 0x7f7b78069da0 2023-10-16T14:13:25.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.911+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7b5806af90 msgr2=0x7f7b5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.911+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7b5806af90 0x7f7b5806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7b780b3610 tx=0x7f7b68009320 comp rx=0 tx=0).stop 2023-10-16T14:13:25.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.911+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 msgr2=0x7f7b7806a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:25.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.911+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 0x7f7b7806a220 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f7b70001f70 tx=0x7f7b700014a0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.911+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 shutdown_connections 2023-10-16T14:13:25.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.911+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7b5806af90 0x7f7b5806d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.912+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7b7815d6e0 0x7f7b78153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.912+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b78069da0 0x7f7b7806a220 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.912+0000 7f7b7cbfb640 1 --2- 172.21.15.60:0/3723674874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b64005180 0x7f7b780af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:25.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.912+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 >> 172.21.15.60:0/3723674874 conn(0x7f7b780ce9b0 msgr2=0x7f7b780c8980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:25.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.912+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 shutdown_connections 2023-10-16T14:13:25.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:25.912+0000 7f7b7cbfb640 1 -- 172.21.15.60:0/3723674874 wait complete. 2023-10-16T14:13:25.939 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T14:13:25.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T14:13:26.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.173+0000 7f8fb0d33640 1 Processor -- start 2023-10-16T14:13:26.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.174+0000 7f8fb0d33640 1 -- start start 2023-10-16T14:13:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.175+0000 7f8fb0d33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac159ab0 0x7f8fac0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.175+0000 7f8fb0d33640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8fac159e90 0x7f8fac06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.175+0000 7f8fb0d33640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8fac06c140 0x7f8fac06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.175+0000 7f8fb0d33640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fac06cb10 con 0x7f8fac159e90 2023-10-16T14:13:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.175+0000 7f8fb0d33640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fac15d890 con 0x7f8fac06c140 2023-10-16T14:13:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.175+0000 7f8fb0d33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0cd1f0 con 0x7f8fac159ab0 2023-10-16T14:13:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8fac159e90 0x7f8fac06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8fac159e90 0x7f8fac06fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47560/0 (socket says 172.21.15.60:47560) 2023-10-16T14:13:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 -- 172.21.15.60:0/2979166202 learned_addr learned my addr 172.21.15.60:0/2979166202 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8faa59c640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac159ab0 0x7f8fac0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 -- 172.21.15.60:0/2979166202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac159ab0 msgr2=0x7f8fac0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8faad9d640 1 --2- 172.21.15.60:0/2979166202 >> v2:172.21.15.144:3300/0 conn(0x7f8fac06c140 0x7f8fac06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac159ab0 0x7f8fac0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 -- 172.21.15.60:0/2979166202 >> v2:172.21.15.144:3300/0 conn(0x7f8fac06c140 msgr2=0x7f8fac06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:26.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 --2- 172.21.15.60:0/2979166202 >> v2:172.21.15.144:3300/0 conn(0x7f8fac06c140 0x7f8fac06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8fa9d9b640 1 -- 172.21.15.60:0/2979166202 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac0cd4d0 con 0x7f8fac159e90 2023-10-16T14:13:26.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8faa59c640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac159ab0 0x7f8fac0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:26.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.176+0000 7f8faad9d640 1 --2- 172.21.15.60:0/2979166202 >> v2:172.21.15.144:3300/0 conn(0x7f8fac06c140 0x7f8fac06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:26.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.177+0000 7f8fa9d9b640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac06fc50 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000bb10 tx=0x7f8fa002dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98f2981c5a1ff0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:26.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.178+0000 7f8fa959a640 1 -- 172.21.15.60:0/2979166202 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa002ed50 con 0x7f8fac159e90 2023-10-16T14:13:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.178+0000 7f8fa959a640 1 -- 172.21.15.60:0/2979166202 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa0036440 con 0x7f8fac159e90 2023-10-16T14:13:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.178+0000 7f8fa959a640 1 -- 172.21.15.60:0/2979166202 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00367d0 con 0x7f8fac159e90 2023-10-16T14:13:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.178+0000 7f8fb0d33640 1 -- 172.21.15.60:0/2979166202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 msgr2=0x7f8fac06fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.178+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac06fc50 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000bb10 tx=0x7f8fa002dfa0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 -- 172.21.15.60:0/2979166202 shutdown_connections 2023-10-16T14:13:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/2979166202 >> v2:172.21.15.144:3300/0 conn(0x7f8fac06c140 0x7f8fac06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac06fc50 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/2979166202 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac159ab0 0x7f8fac0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 -- 172.21.15.60:0/2979166202 >> 172.21.15.60:0/2979166202 conn(0x7f8fac0ce9b0 msgr2=0x7f8fac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 -- 172.21.15.60:0/2979166202 shutdown_connections 2023-10-16T14:13:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.179+0000 7f8fb0d33640 1 -- 172.21.15.60:0/2979166202 wait complete. 2023-10-16T14:13:26.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.181+0000 7f8fb0d33640 1 Processor -- start 2023-10-16T14:13:26.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.181+0000 7f8fb0d33640 1 -- start start 2023-10-16T14:13:26.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8fb0d33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac06c140 0x7f8fac155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8fb0d33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac159ab0 0x7f8fac15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8fb0d33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac15b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8fb0d33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0c8860 con 0x7f8fac159e90 2023-10-16T14:13:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8fb0d33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac15bc40 con 0x7f8fac159ab0 2023-10-16T14:13:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8fb0d33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac156100 con 0x7f8fac06c140 2023-10-16T14:13:26.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8faad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac15b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8faad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac15b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47576/0 (socket says 172.21.15.60:47576) 2023-10-16T14:13:26.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.182+0000 7f8faad9d640 1 -- 172.21.15.60:0/3485591235 learned_addr learned my addr 172.21.15.60:0/3485591235 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:26.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faa59c640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac06c140 0x7f8fac155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8fa9d9b640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac159ab0 0x7f8fac15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faad9d640 1 -- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac06c140 msgr2=0x7f8fac155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:26.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faad9d640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac06c140 0x7f8fac155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faad9d640 1 -- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac159ab0 msgr2=0x7f8fac15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:26.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faad9d640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac159ab0 0x7f8fac15aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faad9d640 1 -- 172.21.15.60:0/3485591235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa000b770 con 0x7f8fac159e90 2023-10-16T14:13:26.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faa59c640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac06c140 0x7f8fac155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:26.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8faad9d640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac15b400 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400c7f0 tx=0x7f8fa400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:26.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4007820 con 0x7f8fac159e90 2023-10-16T14:13:26.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa4007980 con 0x7f8fac159e90 2023-10-16T14:13:26.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4007d30 con 0x7f8fac159e90 2023-10-16T14:13:26.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.183+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac1563e0 con 0x7f8fac159e90 2023-10-16T14:13:26.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.184+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac1569a0 con 0x7f8fac159e90 2023-10-16T14:13:26.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.185+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac156b90 con 0x7f8fac159e90 2023-10-16T14:13:26.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.186+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8fa40113f0 con 0x7f8fac159e90 2023-10-16T14:13:26.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.187+0000 7f8f9affd640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8f8c06af90 0x7f8f8c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:26.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.187+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fa4089c70 con 0x7f8fac159e90 2023-10-16T14:13:26.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.187+0000 7f8faa59c640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8f8c06af90 0x7f8f8c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:26.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.188+0000 7f8faa59c640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8f8c06af90 0x7f8f8c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00d430 tx=0x7f8f9c007530 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:26.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.191+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fa4054b20 con 0x7f8fac159e90 2023-10-16T14:13:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.428+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8fac04e160 con 0x7f8fac159e90 2023-10-16T14:13:26.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.917+0000 7f8f9affd640 1 -- 172.21.15.60:0/3485591235 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v152) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8fa401f0c0 con 0x7f8fac159e90 2023-10-16T14:13:26.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8f8c06af90 msgr2=0x7f8f8c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:26.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8f8c06af90 0x7f8f8c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00d430 tx=0x7f8f9c007530 comp rx=0 tx=0).stop 2023-10-16T14:13:26.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 msgr2=0x7f8fac15b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:26.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac15b400 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400c7f0 tx=0x7f8fa400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 shutdown_connections 2023-10-16T14:13:26.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f8f8c06af90 0x7f8f8c06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fac159e90 0x7f8fac15b400 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8fac159ab0 0x7f8fac15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 --2- 172.21.15.60:0/3485591235 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8fac06c140 0x7f8fac155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:26.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 >> 172.21.15.60:0/3485591235 conn(0x7f8fac0ce9b0 msgr2=0x7f8fac158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:26.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 shutdown_connections 2023-10-16T14:13:26.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:26.921+0000 7f8fb0d33640 1 -- 172.21.15.60:0/3485591235 wait complete. 2023-10-16T14:13:26.948 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T14:13:26.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:27.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.174+0000 7fe93e98e640 1 Processor -- start 2023-10-16T14:13:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.174+0000 7fe93e98e640 1 -- start start 2023-10-16T14:13:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.175+0000 7fe93e98e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe93815cd70 0x7fe9380d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.175+0000 7fe93e98e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe9380b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.175+0000 7fe93e98e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93815d150 con 0x7fe9380b8480 2023-10-16T14:13:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.175+0000 7fe93e98e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9380d1180 con 0x7fe93815cd70 2023-10-16T14:13:27.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.175+0000 7fe93e98e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380d1460 con 0x7fe9380cded0 2023-10-16T14:13:27.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.176+0000 7fe93e18d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe9380b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.176+0000 7fe93e18d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe9380b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:35732/0 (socket says 172.21.15.60:35732) 2023-10-16T14:13:27.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.176+0000 7fe93e18d640 1 -- 172.21.15.60:0/1747523781 learned_addr learned my addr 172.21.15.60:0/1747523781 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:27.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.176+0000 7fe93d98c640 1 --2- 172.21.15.60:0/1747523781 >> v2:172.21.15.144:3300/0 conn(0x7fe93815cd70 0x7fe9380d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.177+0000 7fe93e18d640 1 -- 172.21.15.60:0/1747523781 >> v2:172.21.15.144:3300/0 conn(0x7fe93815cd70 msgr2=0x7fe9380d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.177+0000 7fe93e18d640 1 --2- 172.21.15.60:0/1747523781 >> v2:172.21.15.144:3300/0 conn(0x7fe93815cd70 0x7fe9380d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.177+0000 7fe93e18d640 1 -- 172.21.15.60:0/1747523781 >> v1:172.21.15.144:6789/0 conn(0x7fe9380b8480 legacy=0x7fe9380cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.177+0000 7fe93e18d640 1 -- 172.21.15.60:0/1747523781 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9380d1740 con 0x7fe9380cded0 2023-10-16T14:13:27.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.177+0000 7fe93c98a640 1 -- 172.21.15.60:0/1747523781 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759402694 0 0) 0x7fe93815d150 con 0x7fe9380b8480 2023-10-16T14:13:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.179+0000 7fe93e18d640 1 --2- 172.21.15.60:0/1747523781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe9380b5c90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe930009e50 tx=0x7fe93002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646dc0b6315885de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.179+0000 7fe93c98a640 1 -- 172.21.15.60:0/1747523781 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93002cb00 con 0x7fe9380cded0 2023-10-16T14:13:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.179+0000 7fe93c98a640 1 -- 172.21.15.60:0/1747523781 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93002cc60 con 0x7fe9380cded0 2023-10-16T14:13:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.179+0000 7fe93c98a640 1 -- 172.21.15.60:0/1747523781 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9300343f0 con 0x7fe9380cded0 2023-10-16T14:13:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 -- 172.21.15.60:0/1747523781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 msgr2=0x7fe9380b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 --2- 172.21.15.60:0/1747523781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe9380b5c90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe930009e50 tx=0x7fe93002bf60 comp rx=0 tx=0).stop 2023-10-16T14:13:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 -- 172.21.15.60:0/1747523781 shutdown_connections 2023-10-16T14:13:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 --2- 172.21.15.60:0/1747523781 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe9380b5c90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 --2- 172.21.15.60:0/1747523781 >> v2:172.21.15.144:3300/0 conn(0x7fe93815cd70 0x7fe9380d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 -- 172.21.15.60:0/1747523781 >> 172.21.15.60:0/1747523781 conn(0x7fe9380cf570 msgr2=0x7fe9380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 -- 172.21.15.60:0/1747523781 shutdown_connections 2023-10-16T14:13:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.180+0000 7fe93e98e640 1 -- 172.21.15.60:0/1747523781 wait complete. 2023-10-16T14:13:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.182+0000 7fe93e98e640 1 Processor -- start 2023-10-16T14:13:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.182+0000 7fe93e98e640 1 -- start start 2023-10-16T14:13:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.183+0000 7fe93e98e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 0x7fe93815a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.183+0000 7fe93e98e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe93815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.183+0000 7fe93e98e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93815cd70 0x7fe93815ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.183+0000 7fe93e98e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93815b580 con 0x7fe9380b8480 2023-10-16T14:13:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.183+0000 7fe93e98e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9381591c0 con 0x7fe93815cd70 2023-10-16T14:13:27.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.183+0000 7fe93e98e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe938159460 con 0x7fe9380cded0 2023-10-16T14:13:27.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 0x7fe93815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 0x7fe93815a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47592/0 (socket says 172.21.15.60:47592) 2023-10-16T14:13:27.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 -- 172.21.15.60:0/2688451649 learned_addr learned my addr 172.21.15.60:0/2688451649 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:27.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93e18d640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93815cd70 0x7fe93815ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d18b640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe93815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 -- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 msgr2=0x7fe93815fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe93815fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 -- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93815cd70 msgr2=0x7fe93815ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93815cd70 0x7fe93815ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d98c640 1 -- 172.21.15.60:0/2688451649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930009b30 con 0x7fe9380b8480 2023-10-16T14:13:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.184+0000 7fe93d18b640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe93815fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:27.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.185+0000 7fe93d98c640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 0x7fe93815a220 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fe93400e870 tx=0x7fe93400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.185+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340094f0 con 0x7fe9380b8480 2023-10-16T14:13:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.185+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93400c940 con 0x7fe9380b8480 2023-10-16T14:13:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.185+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400ccf0 con 0x7fe9380b8480 2023-10-16T14:13:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.185+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe938159740 con 0x7fe9380b8480 2023-10-16T14:13:27.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.185+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe938159d00 con 0x7fe9380b8480 2023-10-16T14:13:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.187+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93804e9b0 con 0x7fe9380b8480 2023-10-16T14:13:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.188+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe934013470 con 0x7fe9380b8480 2023-10-16T14:13:27.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.189+0000 7fe92e7fc640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe91c06af90 0x7fe91c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.189+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe934089950 con 0x7fe9380b8480 2023-10-16T14:13:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.189+0000 7fe93d18b640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe91c06af90 0x7fe91c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.191+0000 7fe93d18b640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe91c06af90 0x7fe91c06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe928007fd0 tx=0x7fe928009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:27.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.194+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe934054850 con 0x7fe9380b8480 2023-10-16T14:13:27.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.434+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe938067e30 con 0x7fe9380b8480 2023-10-16T14:13:27.439 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.435+0000 7fe92e7fc640 1 -- 172.21.15.60:0/2688451649 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fe934021090 con 0x7fe9380b8480 2023-10-16T14:13:27.439 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:27.440 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:13:23.271856+0000","modified":"2023-10-16T14:13:26.430490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13065":{"gid":13065,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2549968691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2549968691},{"type":"v1","addr":"172.21.15.60:6835","nonce":2549968691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T14:13:27.440 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 152 2023-10-16T14:13:27.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe91c06af90 msgr2=0x7fe91c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe91c06af90 0x7fe91c06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe928007fd0 tx=0x7fe928009040 comp rx=0 tx=0).stop 2023-10-16T14:13:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 msgr2=0x7fe93815a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 0x7fe93815a220 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fe93400e870 tx=0x7fe93400ed30 comp rx=0 tx=0).stop 2023-10-16T14:13:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 shutdown_connections 2023-10-16T14:13:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fe91c06af90 0x7fe91c06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe9380cded0 0x7fe93815fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe93815cd70 0x7fe93815ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 --2- 172.21.15.60:0/2688451649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9380b8480 0x7fe93815a220 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.438+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 >> 172.21.15.60:0/2688451649 conn(0x7fe9380cf570 msgr2=0x7fe9380b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.439+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 shutdown_connections 2023-10-16T14:13:27.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.439+0000 7fe93e98e640 1 -- 172.21.15.60:0/2688451649 wait complete. 2023-10-16T14:13:27.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.683+0000 7f74c9d43640 1 Processor -- start 2023-10-16T14:13:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.683+0000 7f74c9d43640 1 -- start start 2023-10-16T14:13:27.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.685+0000 7f74c9d43640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f74c415dde0 0x7f74c40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.685+0000 7f74c9d43640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f74c4070f20 0x7f74c40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.685+0000 7f74c9d43640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c4160a90 con 0x7f74c415c9f0 2023-10-16T14:13:27.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.685+0000 7f74c9d43640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c4071350 con 0x7f74c415dde0 2023-10-16T14:13:27.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.685+0000 7f74c9d43640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c40cdec0 con 0x7f74c4070f20 2023-10-16T14:13:27.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f74c415dde0 0x7f74c40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f74c415dde0 0x7f74c40709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47600/0 (socket says 172.21.15.60:47600) 2023-10-16T14:13:27.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 -- 172.21.15.60:0/3728045854 learned_addr learned my addr 172.21.15.60:0/3728045854 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:27.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c8d41640 1 --1- 172.21.15.60:0/3728045854 >> v1:172.21.15.60:6789/0 conn(0x7f74c415c9f0 0x7f74c40c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37640/0 (socket says 172.21.15.60:37640) 2023-10-16T14:13:27.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c9542640 1 --2- 172.21.15.60:0/3728045854 >> v2:172.21.15.144:3300/0 conn(0x7f74c4070f20 0x7f74c40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 -- 172.21.15.60:0/3728045854 >> v2:172.21.15.144:3300/0 conn(0x7f74c4070f20 msgr2=0x7f74c40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:27.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 --2- 172.21.15.60:0/3728045854 >> v2:172.21.15.144:3300/0 conn(0x7f74c4070f20 0x7f74c40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 -- 172.21.15.60:0/3728045854 >> v1:172.21.15.60:6789/0 conn(0x7f74c415c9f0 legacy=0x7f74c40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c3fff640 1 -- 172.21.15.60:0/3728045854 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40cd900 con 0x7f74c415dde0 2023-10-16T14:13:27.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c37fe640 1 -- 172.21.15.60:0/3728045854 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385951637 0 0) 0x7f74c4160a90 con 0x7f74c415c9f0 2023-10-16T14:13:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.686+0000 7f74c9542640 1 --2- 172.21.15.60:0/3728045854 >> v2:172.21.15.144:3300/0 conn(0x7f74c4070f20 0x7f74c40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.688+0000 7f74c3fff640 1 --2- 172.21.15.60:0/3728045854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415dde0 0x7f74c40709e0 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f74c406fcc0 tx=0x7f74b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f907ae66ea9d8ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.688+0000 7f74c37fe640 1 -- 172.21.15.60:0/3728045854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0030860 con 0x7f74c415dde0 2023-10-16T14:13:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.688+0000 7f74c37fe640 1 -- 172.21.15.60:0/3728045854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b00309c0 con 0x7f74c415dde0 2023-10-16T14:13:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.688+0000 7f74c37fe640 1 -- 172.21.15.60:0/3728045854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0030d50 con 0x7f74c415dde0 2023-10-16T14:13:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 -- 172.21.15.60:0/3728045854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415dde0 msgr2=0x7f74c40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3728045854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415dde0 0x7f74c40709e0 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f74c406fcc0 tx=0x7f74b002b9a0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 -- 172.21.15.60:0/3728045854 shutdown_connections 2023-10-16T14:13:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3728045854 >> v2:172.21.15.144:3300/0 conn(0x7f74c4070f20 0x7f74c40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3728045854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415dde0 0x7f74c40709e0 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 -- 172.21.15.60:0/3728045854 >> 172.21.15.60:0/3728045854 conn(0x7f74c40ce6a0 msgr2=0x7f74c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 -- 172.21.15.60:0/3728045854 shutdown_connections 2023-10-16T14:13:27.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.689+0000 7f74c9d43640 1 -- 172.21.15.60:0/3728045854 wait complete. 2023-10-16T14:13:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.691+0000 7f74c9d43640 1 Processor -- start 2023-10-16T14:13:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.691+0000 7f74c9d43640 1 -- start start 2023-10-16T14:13:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c9d43640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c4070f20 0x7f74c4150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c9d43640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 0x7f74c4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c9d43640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74c415dde0 0x7f74c40c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c9d43640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40c3170 con 0x7f74c415c9f0 2023-10-16T14:13:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c9d43640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40c2bb0 con 0x7f74c4070f20 2023-10-16T14:13:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c9d43640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40c2e90 con 0x7f74c415dde0 2023-10-16T14:13:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 0x7f74c4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 0x7f74c4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47616/0 (socket says 172.21.15.60:47616) 2023-10-16T14:13:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.692+0000 7f74c3fff640 1 -- 172.21.15.60:0/3414418675 learned_addr learned my addr 172.21.15.60:0/3414418675 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c9542640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74c415dde0 0x7f74c40c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c8d41640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c4070f20 0x7f74c4150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c3fff640 1 -- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74c415dde0 msgr2=0x7f74c40c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c3fff640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74c415dde0 0x7f74c40c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c3fff640 1 -- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c4070f20 msgr2=0x7f74c4150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c3fff640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c4070f20 0x7f74c4150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c3fff640 1 -- 172.21.15.60:0/3414418675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00094d0 con 0x7f74c415c9f0 2023-10-16T14:13:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c8d41640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c4070f20 0x7f74c4150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.693+0000 7f74c3fff640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 0x7f74c4069da0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f74b00097c0 tx=0x7f74b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.694+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0039a90 con 0x7f74c415c9f0 2023-10-16T14:13:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.694+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0039bf0 con 0x7f74c415c9f0 2023-10-16T14:13:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.694+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c40c3da0 con 0x7f74c415c9f0 2023-10-16T14:13:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.694+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0043410 con 0x7f74c415c9f0 2023-10-16T14:13:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.694+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c41510a0 con 0x7f74c415c9f0 2023-10-16T14:13:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.695+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c40c42c0 con 0x7f74c415c9f0 2023-10-16T14:13:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.696+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74b004c3b0 con 0x7f74c415c9f0 2023-10-16T14:13:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.697+0000 7f74c17fa640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f74a806afc0 0x7f74a806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.697+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74b00abd40 con 0x7f74c415c9f0 2023-10-16T14:13:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.698+0000 7f74c8d41640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f74a806afc0 0x7f74a806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.699+0000 7f74c8d41640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f74a806afc0 0x7f74a806d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74b8000f60 tx=0x7f74b8009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.701+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74c40c42c0 con 0x7f74c415c9f0 2023-10-16T14:13:27.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.899+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74c40c42c0 con 0x7f74c415c9f0 2023-10-16T14:13:27.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.903+0000 7f74c17fa640 1 -- 172.21.15.60:0/3414418675 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v287) v1 ==== 74+0+16515 (secure 0 0 0) 0x7f74c40c42c0 con 0x7f74c415c9f0 2023-10-16T14:13:27.907 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:27.907 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":287,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:24.811741+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:27.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f74a806afc0 msgr2=0x7f74a806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f74a806afc0 0x7f74a806d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74b8000f60 tx=0x7f74b8009040 comp rx=0 tx=0).stop 2023-10-16T14:13:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 msgr2=0x7f74c4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 0x7f74c4069da0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f74b00097c0 tx=0x7f74b0004730 comp rx=0 tx=0).stop 2023-10-16T14:13:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 shutdown_connections 2023-10-16T14:13:27.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f74a806afc0 0x7f74a806d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f74c415dde0 0x7f74c40c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74c415c9f0 0x7f74c4069da0 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 --2- 172.21.15.60:0/3414418675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74c4070f20 0x7f74c4150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:27.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.905+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 >> 172.21.15.60:0/3414418675 conn(0x7f74c40ce6a0 msgr2=0x7f74c40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:27.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.906+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 shutdown_connections 2023-10-16T14:13:27.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:27.906+0000 7f74c9d43640 1 -- 172.21.15.60:0/3414418675 wait complete. 2023-10-16T14:13:27.912 INFO:tasks.ceph.mds.a.smithi060.stderr:2023-10-16T14:13:27.906+0000 7f09aa277640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T14:13:27.930 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T14:13:27.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:13:28.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.150+0000 7fdbd79e9640 1 Processor -- start 2023-10-16T14:13:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.150+0000 7fdbd79e9640 1 -- start start 2023-10-16T14:13:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd79e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdbd0159b20 0x7fdbd006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd79e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdbd0159f00 0x7fdbd00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd79e9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd006f7f0 con 0x7fdbd006c120 2023-10-16T14:13:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd79e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd006c5e0 con 0x7fdbd0159b20 2023-10-16T14:13:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd79e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd006c8c0 con 0x7fdbd0159f00 2023-10-16T14:13:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd5785640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdbd0159b20 0x7fdbd006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd5785640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdbd0159b20 0x7fdbd006bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47632/0 (socket says 172.21.15.60:47632) 2023-10-16T14:13:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.152+0000 7fdbd5785640 1 -- 172.21.15.60:0/676120509 learned_addr learned my addr 172.21.15.60:0/676120509 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.153+0000 7fdbd4f84640 1 --2- 172.21.15.60:0/676120509 >> v2:172.21.15.144:3300/0 conn(0x7fdbd0159f00 0x7fdbd00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.153+0000 7fdbd5785640 1 -- 172.21.15.60:0/676120509 >> v2:172.21.15.144:3300/0 conn(0x7fdbd0159f00 msgr2=0x7fdbd00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.153+0000 7fdbd5785640 1 --2- 172.21.15.60:0/676120509 >> v2:172.21.15.144:3300/0 conn(0x7fdbd0159f00 0x7fdbd00b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.153+0000 7fdbd5785640 1 -- 172.21.15.60:0/676120509 >> v1:172.21.15.144:6789/0 conn(0x7fdbd006c120 legacy=0x7fdbd00cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:28.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.153+0000 7fdbd5785640 1 -- 172.21.15.60:0/676120509 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd006cba0 con 0x7fdbd0159b20 2023-10-16T14:13:28.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.153+0000 7fdbd4f84640 1 --2- 172.21.15.60:0/676120509 >> v2:172.21.15.144:3300/0 conn(0x7fdbd0159f00 0x7fdbd00b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:28.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.154+0000 7fdbd5785640 1 --2- 172.21.15.60:0/676120509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159b20 0x7fdbd006bbe0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7fdbd015d770 tx=0x7fdbcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1244c62079824b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:28.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.154+0000 7fdbc3fff640 1 -- 172.21.15.60:0/676120509 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc031870 con 0x7fdbd0159b20 2023-10-16T14:13:28.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbc3fff640 1 -- 172.21.15.60:0/676120509 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc0319d0 con 0x7fdbd0159b20 2023-10-16T14:13:28.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbc3fff640 1 -- 172.21.15.60:0/676120509 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc031d60 con 0x7fdbd0159b20 2023-10-16T14:13:28.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 -- 172.21.15.60:0/676120509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159b20 msgr2=0x7fdbd006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/676120509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159b20 0x7fdbd006bbe0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7fdbd015d770 tx=0x7fdbcc02b910 comp rx=0 tx=0).stop 2023-10-16T14:13:28.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 -- 172.21.15.60:0/676120509 shutdown_connections 2023-10-16T14:13:28.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/676120509 >> v2:172.21.15.144:3300/0 conn(0x7fdbd0159f00 0x7fdbd00b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/676120509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159b20 0x7fdbd006bbe0 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 -- 172.21.15.60:0/676120509 >> 172.21.15.60:0/676120509 conn(0x7fdbd00cb410 msgr2=0x7fdbd00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:28.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 -- 172.21.15.60:0/676120509 shutdown_connections 2023-10-16T14:13:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.155+0000 7fdbd79e9640 1 -- 172.21.15.60:0/676120509 wait complete. 2023-10-16T14:13:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.157+0000 7fdbd79e9640 1 Processor -- start 2023-10-16T14:13:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.157+0000 7fdbd79e9640 1 -- start start 2023-10-16T14:13:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.158+0000 7fdbd79e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbd006c120 0x7fdbd0151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.158+0000 7fdbd79e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbd0159b20 0x7fdbd00ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.158+0000 7fdbd79e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 0x7fdbd0158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.158+0000 7fdbd79e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd00b0630 con 0x7fdbd0159f00 2023-10-16T14:13:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.158+0000 7fdbd79e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd00ae460 con 0x7fdbd006c120 2023-10-16T14:13:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.158+0000 7fdbd79e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd00ae740 con 0x7fdbd0159b20 2023-10-16T14:13:28.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 0x7fdbd0158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 0x7fdbd0158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47640/0 (socket says 172.21.15.60:47640) 2023-10-16T14:13:28.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 -- 172.21.15.60:0/1528009843 learned_addr learned my addr 172.21.15.60:0/1528009843 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:28.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd4f84640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbd0159b20 0x7fdbd00ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5785640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbd006c120 0x7fdbd0151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 -- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbd0159b20 msgr2=0x7fdbd00ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbd0159b20 0x7fdbd00ad480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 -- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbd006c120 msgr2=0x7fdbd0151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbd006c120 0x7fdbd0151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 -- 172.21.15.60:0/1528009843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc0094d0 con 0x7fdbd0159f00 2023-10-16T14:13:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.159+0000 7fdbd5f86640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 0x7fdbd0158140 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800b970 tx=0x7fdbc800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.160+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc800cb40 con 0x7fdbd0159f00 2023-10-16T14:13:28.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.160+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc800cca0 con 0x7fdbd0159f00 2023-10-16T14:13:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.160+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8014680 con 0x7fdbd0159f00 2023-10-16T14:13:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.160+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd00aea20 con 0x7fdbd0159f00 2023-10-16T14:13:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.160+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd00aefe0 con 0x7fdbd0159f00 2023-10-16T14:13:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.161+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd00af200 con 0x7fdbd0159f00 2023-10-16T14:13:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.162+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdbc801d3b0 con 0x7fdbd0159f00 2023-10-16T14:13:28.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.163+0000 7fdbc1ffb640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fdbb406af90 0x7fdbb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.163+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbc8089670 con 0x7fdbd0159f00 2023-10-16T14:13:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.164+0000 7fdbd5785640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fdbb406af90 0x7fdbb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.165+0000 7fdbd5785640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fdbb406af90 0x7fdbb406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc036000 tx=0x7fdbcc03c040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.168+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbc8054570 con 0x7fdbd0159f00 2023-10-16T14:13:28.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.361+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 --> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbd006ce80 con 0x7fdbb406af90 2023-10-16T14:13:28.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.391+0000 7fdbc1ffb640 1 -- 172.21.15.60:0/1528009843 <== mgr.10042 v2:172.21.15.144:6832/3623959149 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287180 (secure 0 0 0) 0x7fdbcc031cc0 con 0x7fdbb406af90 2023-10-16T14:13:28.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:13:28.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fdbb406af90 msgr2=0x7fdbb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fdbb406af90 0x7fdbb406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc036000 tx=0x7fdbcc03c040 comp rx=0 tx=0).stop 2023-10-16T14:13:28.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 msgr2=0x7fdbd0158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 0x7fdbd0158140 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800b970 tx=0x7fdbc800be30 comp rx=0 tx=0).stop 2023-10-16T14:13:28.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 shutdown_connections 2023-10-16T14:13:28.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fdbb406af90 0x7fdbb406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fdbd0159b20 0x7fdbd00ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdbd0159f00 0x7fdbd0158140 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 --2- 172.21.15.60:0/1528009843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdbd006c120 0x7fdbd0151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 >> 172.21.15.60:0/1528009843 conn(0x7fdbd00cb410 msgr2=0x7fdbd00cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:28.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 shutdown_connections 2023-10-16T14:13:28.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.398+0000 7fdbd79e9640 1 -- 172.21.15.60:0/1528009843 wait complete. 2023-10-16T14:13:28.424 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T14:13:28.081991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2699,"num_read_kb":5660,"num_write":733,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2859,"ondisk_log_size":2859,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":463476,"kb_used_data":29416,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754511244,"statfs":{"total":773094113280,"available":772619513856,"internally_reserved":0,"allocated":30121984,"data_stored":15385476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002216"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823007+0000","last_change":"2023-10-16T14:13:23.852963+0000","last_active":"2023-10-16T14:13:24.823007+0000","last_peered":"2023-10-16T14:13:24.823007+0000","last_clean":"2023-10-16T14:13:24.823007+0000","last_became_active":"2023-10-16T14:13:23.851773+0000","last_became_peered":"2023-10-16T14:13:23.851773+0000","last_unstale":"2023-10-16T14:13:24.823007+0000","last_undegraded":"2023-10-16T14:13:24.823007+0000","last_fullsized":"2023-10-16T14:13:24.823007+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:59.454611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.819166+0000","last_change":"2023-10-16T14:13:23.834193+0000","last_active":"2023-10-16T14:13:24.819166+0000","last_peered":"2023-10-16T14:13:24.819166+0000","last_clean":"2023-10-16T14:13:24.819166+0000","last_became_active":"2023-10-16T14:13:23.833658+0000","last_became_peered":"2023-10-16T14:13:23.833658+0000","last_unstale":"2023-10-16T14:13:24.819166+0000","last_undegraded":"2023-10-16T14:13:24.819166+0000","last_fullsized":"2023-10-16T14:13:24.819166+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:13.217594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857262+0000","last_change":"2023-10-16T14:13:23.846821+0000","last_active":"2023-10-16T14:13:23.857262+0000","last_peered":"2023-10-16T14:13:23.857262+0000","last_clean":"2023-10-16T14:13:23.857262+0000","last_became_active":"2023-10-16T14:13:23.846530+0000","last_became_peered":"2023-10-16T14:13:23.846530+0000","last_unstale":"2023-10-16T14:13:23.857262+0000","last_undegraded":"2023-10-16T14:13:23.857262+0000","last_fullsized":"2023-10-16T14:13:23.857262+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:14.529616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859972+0000","last_change":"2023-10-16T14:13:23.854726+0000","last_active":"2023-10-16T14:13:23.859972+0000","last_peered":"2023-10-16T14:13:23.859972+0000","last_clean":"2023-10-16T14:13:23.859972+0000","last_became_active":"2023-10-16T14:13:23.849834+0000","last_became_peered":"2023-10-16T14:13:23.849834+0000","last_unstale":"2023-10-16T14:13:23.859972+0000","last_undegraded":"2023-10-16T14:13:23.859972+0000","last_fullsized":"2023-10-16T14:13:23.859972+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:08.105629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859856+0000","last_change":"2023-10-16T14:13:23.851466+0000","last_active":"2023-10-16T14:13:23.859856+0000","last_peered":"2023-10-16T14:13:23.859856+0000","last_clean":"2023-10-16T14:13:23.859856+0000","last_became_active":"2023-10-16T14:13:23.851129+0000","last_became_peered":"2023-10-16T14:13:23.851129+0000","last_unstale":"2023-10-16T14:13:23.859856+0000","last_undegraded":"2023-10-16T14:13:23.859856+0000","last_fullsized":"2023-10-16T14:13:23.859856+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:17.034746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858145+0000","last_change":"2023-10-16T14:13:23.855253+0000","last_active":"2023-10-16T14:13:23.858145+0000","last_peered":"2023-10-16T14:13:23.858145+0000","last_clean":"2023-10-16T14:13:23.858145+0000","last_became_active":"2023-10-16T14:13:23.854947+0000","last_became_peered":"2023-10-16T14:13:23.854947+0000","last_unstale":"2023-10-16T14:13:23.858145+0000","last_undegraded":"2023-10-16T14:13:23.858145+0000","last_fullsized":"2023-10-16T14:13:23.858145+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:44.048362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821052+0000","last_change":"2023-10-16T14:13:23.843499+0000","last_active":"2023-10-16T14:13:24.821052+0000","last_peered":"2023-10-16T14:13:24.821052+0000","last_clean":"2023-10-16T14:13:24.821052+0000","last_became_active":"2023-10-16T14:13:23.841968+0000","last_became_peered":"2023-10-16T14:13:23.841968+0000","last_unstale":"2023-10-16T14:13:24.821052+0000","last_undegraded":"2023-10-16T14:13:24.821052+0000","last_fullsized":"2023-10-16T14:13:24.821052+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:49.918820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.860004+0000","last_change":"2023-10-16T14:13:23.858865+0000","last_active":"2023-10-16T14:13:23.860004+0000","last_peered":"2023-10-16T14:13:23.860004+0000","last_clean":"2023-10-16T14:13:23.860004+0000","last_became_active":"2023-10-16T14:13:23.858549+0000","last_became_peered":"2023-10-16T14:13:23.858549+0000","last_unstale":"2023-10-16T14:13:23.860004+0000","last_undegraded":"2023-10-16T14:13:23.860004+0000","last_fullsized":"2023-10-16T14:13:23.860004+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:06.232795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818945+0000","last_change":"2023-10-16T14:13:23.851913+0000","last_active":"2023-10-16T14:13:24.818945+0000","last_peered":"2023-10-16T14:13:24.818945+0000","last_clean":"2023-10-16T14:13:24.818945+0000","last_became_active":"2023-10-16T14:13:23.851624+0000","last_became_peered":"2023-10-16T14:13:23.851624+0000","last_unstale":"2023-10-16T14:13:24.818945+0000","last_undegraded":"2023-10-16T14:13:24.818945+0000","last_fullsized":"2023-10-16T14:13:24.818945+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:28.861779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.824149+0000","last_change":"2023-10-16T14:13:23.852441+0000","last_active":"2023-10-16T14:13:24.824149+0000","last_peered":"2023-10-16T14:13:24.824149+0000","last_clean":"2023-10-16T14:13:24.824149+0000","last_became_active":"2023-10-16T14:13:23.850746+0000","last_became_peered":"2023-10-16T14:13:23.850746+0000","last_unstale":"2023-10-16T14:13:24.824149+0000","last_undegraded":"2023-10-16T14:13:24.824149+0000","last_fullsized":"2023-10-16T14:13:24.824149+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:07.478142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.854460+0000","last_change":"2023-10-16T14:13:23.849283+0000","last_active":"2023-10-16T14:13:23.854460+0000","last_peered":"2023-10-16T14:13:23.854460+0000","last_clean":"2023-10-16T14:13:23.854460+0000","last_became_active":"2023-10-16T14:13:23.848884+0000","last_became_peered":"2023-10-16T14:13:23.848884+0000","last_unstale":"2023-10-16T14:13:23.854460+0000","last_undegraded":"2023-10-16T14:13:23.854460+0000","last_fullsized":"2023-10-16T14:13:23.854460+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:48:34.438353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.860975+0000","last_change":"2023-10-16T14:13:23.860570+0000","last_active":"2023-10-16T14:13:23.860975+0000","last_peered":"2023-10-16T14:13:23.860975+0000","last_clean":"2023-10-16T14:13:23.860975+0000","last_became_active":"2023-10-16T14:13:23.860271+0000","last_became_peered":"2023-10-16T14:13:23.860271+0000","last_unstale":"2023-10-16T14:13:23.860975+0000","last_undegraded":"2023-10-16T14:13:23.860975+0000","last_fullsized":"2023-10-16T14:13:23.860975+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:17.009125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823797+0000","last_change":"2023-10-16T14:13:23.847281+0000","last_active":"2023-10-16T14:13:24.823797+0000","last_peered":"2023-10-16T14:13:24.823797+0000","last_clean":"2023-10-16T14:13:24.823797+0000","last_became_active":"2023-10-16T14:13:23.846819+0000","last_became_peered":"2023-10-16T14:13:23.846819+0000","last_unstale":"2023-10-16T14:13:24.823797+0000","last_undegraded":"2023-10-16T14:13:24.823797+0000","last_fullsized":"2023-10-16T14:13:24.823797+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:28.376935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.831759+0000","last_change":"2023-10-16T14:13:23.829028+0000","last_active":"2023-10-16T14:13:23.831759+0000","last_peered":"2023-10-16T14:13:23.831759+0000","last_clean":"2023-10-16T14:13:23.831759+0000","last_became_active":"2023-10-16T14:13:23.828238+0000","last_became_peered":"2023-10-16T14:13:23.828238+0000","last_unstale":"2023-10-16T14:13:23.831759+0000","last_undegraded":"2023-10-16T14:13:23.831759+0000","last_fullsized":"2023-10-16T14:13:23.831759+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:26.977069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859699+0000","last_change":"2023-10-16T14:13:23.850901+0000","last_active":"2023-10-16T14:13:23.859699+0000","last_peered":"2023-10-16T14:13:23.859699+0000","last_clean":"2023-10-16T14:13:23.859699+0000","last_became_active":"2023-10-16T14:13:23.850577+0000","last_became_peered":"2023-10-16T14:13:23.850577+0000","last_unstale":"2023-10-16T14:13:23.859699+0000","last_undegraded":"2023-10-16T14:13:23.859699+0000","last_fullsized":"2023-10-16T14:13:23.859699+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:38.527703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.856205+0000","last_change":"2023-10-16T14:13:23.852934+0000","last_active":"2023-10-16T14:13:23.856205+0000","last_peered":"2023-10-16T14:13:23.856205+0000","last_clean":"2023-10-16T14:13:23.856205+0000","last_became_active":"2023-10-16T14:13:23.852647+0000","last_became_peered":"2023-10-16T14:13:23.852647+0000","last_unstale":"2023-10-16T14:13:23.856205+0000","last_undegraded":"2023-10-16T14:13:23.856205+0000","last_fullsized":"2023-10-16T14:13:23.856205+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:37.220473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822767+0000","last_change":"2023-10-16T14:13:23.859347+0000","last_active":"2023-10-16T14:13:24.822767+0000","last_peered":"2023-10-16T14:13:24.822767+0000","last_clean":"2023-10-16T14:13:24.822767+0000","last_became_active":"2023-10-16T14:13:23.858892+0000","last_became_peered":"2023-10-16T14:13:23.858892+0000","last_unstale":"2023-10-16T14:13:24.822767+0000","last_undegraded":"2023-10-16T14:13:24.822767+0000","last_fullsized":"2023-10-16T14:13:24.822767+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:44.195288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821795+0000","last_change":"2023-10-16T14:13:23.853942+0000","last_active":"2023-10-16T14:13:24.821795+0000","last_peered":"2023-10-16T14:13:24.821795+0000","last_clean":"2023-10-16T14:13:24.821795+0000","last_became_active":"2023-10-16T14:13:23.853579+0000","last_became_peered":"2023-10-16T14:13:23.853579+0000","last_unstale":"2023-10-16T14:13:24.821795+0000","last_undegraded":"2023-10-16T14:13:24.821795+0000","last_fullsized":"2023-10-16T14:13:24.821795+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:48.130917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.840340+0000","last_change":"2023-10-16T14:13:23.834012+0000","last_active":"2023-10-16T14:13:23.840340+0000","last_peered":"2023-10-16T14:13:23.840340+0000","last_clean":"2023-10-16T14:13:23.840340+0000","last_became_active":"2023-10-16T14:13:23.833222+0000","last_became_peered":"2023-10-16T14:13:23.833222+0000","last_unstale":"2023-10-16T14:13:23.840340+0000","last_undegraded":"2023-10-16T14:13:23.840340+0000","last_fullsized":"2023-10-16T14:13:23.840340+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:40.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818909+0000","last_change":"2023-10-16T14:13:23.850058+0000","last_active":"2023-10-16T14:13:24.818909+0000","last_peered":"2023-10-16T14:13:24.818909+0000","last_clean":"2023-10-16T14:13:24.818909+0000","last_became_active":"2023-10-16T14:13:23.849739+0000","last_became_peered":"2023-10-16T14:13:23.849739+0000","last_unstale":"2023-10-16T14:13:24.818909+0000","last_undegraded":"2023-10-16T14:13:24.818909+0000","last_fullsized":"2023-10-16T14:13:24.818909+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:20.305797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.853257+0000","last_change":"2023-10-16T14:13:23.844321+0000","last_active":"2023-10-16T14:13:23.853257+0000","last_peered":"2023-10-16T14:13:23.853257+0000","last_clean":"2023-10-16T14:13:23.853257+0000","last_became_active":"2023-10-16T14:13:23.844030+0000","last_became_peered":"2023-10-16T14:13:23.844030+0000","last_unstale":"2023-10-16T14:13:23.853257+0000","last_undegraded":"2023-10-16T14:13:23.853257+0000","last_fullsized":"2023-10-16T14:13:23.853257+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:36.878610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.842092+0000","last_change":"2023-10-16T14:13:23.836814+0000","last_active":"2023-10-16T14:13:23.842092+0000","last_peered":"2023-10-16T14:13:23.842092+0000","last_clean":"2023-10-16T14:13:23.842092+0000","last_became_active":"2023-10-16T14:13:23.836118+0000","last_became_peered":"2023-10-16T14:13:23.836118+0000","last_unstale":"2023-10-16T14:13:23.842092+0000","last_undegraded":"2023-10-16T14:13:23.842092+0000","last_fullsized":"2023-10-16T14:13:23.842092+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:38.311249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823414+0000","last_change":"2023-10-16T14:13:23.848122+0000","last_active":"2023-10-16T14:13:24.823414+0000","last_peered":"2023-10-16T14:13:24.823414+0000","last_clean":"2023-10-16T14:13:24.823414+0000","last_became_active":"2023-10-16T14:13:23.846297+0000","last_became_peered":"2023-10-16T14:13:23.846297+0000","last_unstale":"2023-10-16T14:13:24.823414+0000","last_undegraded":"2023-10-16T14:13:24.823414+0000","last_fullsized":"2023-10-16T14:13:24.823414+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:06.788358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857734+0000","last_change":"2023-10-16T14:13:23.857062+0000","last_active":"2023-10-16T14:13:23.857734+0000","last_peered":"2023-10-16T14:13:23.857734+0000","last_clean":"2023-10-16T14:13:23.857734+0000","last_became_active":"2023-10-16T14:13:23.856691+0000","last_became_peered":"2023-10-16T14:13:23.856691+0000","last_unstale":"2023-10-16T14:13:23.857734+0000","last_undegraded":"2023-10-16T14:13:23.857734+0000","last_fullsized":"2023-10-16T14:13:23.857734+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:44.517072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823116+0000","last_change":"2023-10-16T14:13:23.856591+0000","last_active":"2023-10-16T14:13:24.823116+0000","last_peered":"2023-10-16T14:13:24.823116+0000","last_clean":"2023-10-16T14:13:24.823116+0000","last_became_active":"2023-10-16T14:13:23.856101+0000","last_became_peered":"2023-10-16T14:13:23.856101+0000","last_unstale":"2023-10-16T14:13:24.823116+0000","last_undegraded":"2023-10-16T14:13:24.823116+0000","last_fullsized":"2023-10-16T14:13:24.823116+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:59.837730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.824284+0000","last_change":"2023-10-16T14:13:23.841073+0000","last_active":"2023-10-16T14:13:24.824284+0000","last_peered":"2023-10-16T14:13:24.824284+0000","last_clean":"2023-10-16T14:13:24.824284+0000","last_became_active":"2023-10-16T14:13:23.840723+0000","last_became_peered":"2023-10-16T14:13:23.840723+0000","last_unstale":"2023-10-16T14:13:24.824284+0000","last_undegraded":"2023-10-16T14:13:24.824284+0000","last_fullsized":"2023-10-16T14:13:24.824284+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:07.676069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.853432+0000","last_change":"2023-10-16T14:13:23.846346+0000","last_active":"2023-10-16T14:13:23.853432+0000","last_peered":"2023-10-16T14:13:23.853432+0000","last_clean":"2023-10-16T14:13:23.853432+0000","last_became_active":"2023-10-16T14:13:23.846023+0000","last_became_peered":"2023-10-16T14:13:23.846023+0000","last_unstale":"2023-10-16T14:13:23.853432+0000","last_undegraded":"2023-10-16T14:13:23.853432+0000","last_fullsized":"2023-10-16T14:13:23.853432+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:29.108473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850512+0000","last_change":"2023-10-16T14:13:23.845286+0000","last_active":"2023-10-16T14:13:23.850512+0000","last_peered":"2023-10-16T14:13:23.850512+0000","last_clean":"2023-10-16T14:13:23.850512+0000","last_became_active":"2023-10-16T14:13:23.844831+0000","last_became_peered":"2023-10-16T14:13:23.844831+0000","last_unstale":"2023-10-16T14:13:23.850512+0000","last_undegraded":"2023-10-16T14:13:23.850512+0000","last_fullsized":"2023-10-16T14:13:23.850512+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:12.889338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823425+0000","last_change":"2023-10-16T14:13:23.855638+0000","last_active":"2023-10-16T14:13:24.823425+0000","last_peered":"2023-10-16T14:13:24.823425+0000","last_clean":"2023-10-16T14:13:24.823425+0000","last_became_active":"2023-10-16T14:13:23.855345+0000","last_became_peered":"2023-10-16T14:13:23.855345+0000","last_unstale":"2023-10-16T14:13:24.823425+0000","last_undegraded":"2023-10-16T14:13:24.823425+0000","last_fullsized":"2023-10-16T14:13:24.823425+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:21.983883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850867+0000","last_change":"2023-10-16T14:13:23.848508+0000","last_active":"2023-10-16T14:13:23.850867+0000","last_peered":"2023-10-16T14:13:23.850867+0000","last_clean":"2023-10-16T14:13:23.850867+0000","last_became_active":"2023-10-16T14:13:23.848208+0000","last_became_peered":"2023-10-16T14:13:23.848208+0000","last_unstale":"2023-10-16T14:13:23.850867+0000","last_undegraded":"2023-10-16T14:13:23.850867+0000","last_fullsized":"2023-10-16T14:13:23.850867+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:16.217629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858346+0000","last_change":"2023-10-16T14:13:23.848976+0000","last_active":"2023-10-16T14:13:23.858346+0000","last_peered":"2023-10-16T14:13:23.858346+0000","last_clean":"2023-10-16T14:13:23.858346+0000","last_became_active":"2023-10-16T14:13:23.848703+0000","last_became_peered":"2023-10-16T14:13:23.848703+0000","last_unstale":"2023-10-16T14:13:23.858346+0000","last_undegraded":"2023-10-16T14:13:23.858346+0000","last_fullsized":"2023-10-16T14:13:23.858346+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:50:53.825346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.856246+0000","last_change":"2023-10-16T14:13:23.851765+0000","last_active":"2023-10-16T14:13:23.856246+0000","last_peered":"2023-10-16T14:13:23.856246+0000","last_clean":"2023-10-16T14:13:23.856246+0000","last_became_active":"2023-10-16T14:13:23.849363+0000","last_became_peered":"2023-10-16T14:13:23.849363+0000","last_unstale":"2023-10-16T14:13:23.856246+0000","last_undegraded":"2023-10-16T14:13:23.856246+0000","last_fullsized":"2023-10-16T14:13:23.856246+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:42.205020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"286'4","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911069+0000","last_change":"2023-10-16T14:13:22.809923+0000","last_active":"2023-10-16T14:13:23.911069+0000","last_peered":"2023-10-16T14:13:23.911069+0000","last_clean":"2023-10-16T14:13:23.911069+0000","last_became_active":"2023-10-16T14:13:22.809574+0000","last_became_peered":"2023-10-16T14:13:22.809574+0000","last_unstale":"2023-10-16T14:13:23.911069+0000","last_undegraded":"2023-10-16T14:13:23.911069+0000","last_fullsized":"2023-10-16T14:13:23.911069+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:37.337590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.852619+0000","last_change":"2023-10-16T14:13:23.850634+0000","last_active":"2023-10-16T14:13:23.852619+0000","last_peered":"2023-10-16T14:13:23.852619+0000","last_clean":"2023-10-16T14:13:23.852619+0000","last_became_active":"2023-10-16T14:13:23.850341+0000","last_became_peered":"2023-10-16T14:13:23.850341+0000","last_unstale":"2023-10-16T14:13:23.852619+0000","last_undegraded":"2023-10-16T14:13:23.852619+0000","last_fullsized":"2023-10-16T14:13:23.852619+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:52.117283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909694+0000","last_change":"2023-10-16T14:13:22.822269+0000","last_active":"2023-10-16T14:13:23.909694+0000","last_peered":"2023-10-16T14:13:23.909694+0000","last_clean":"2023-10-16T14:13:23.909694+0000","last_became_active":"2023-10-16T14:13:22.821906+0000","last_became_peered":"2023-10-16T14:13:22.821906+0000","last_unstale":"2023-10-16T14:13:23.909694+0000","last_undegraded":"2023-10-16T14:13:23.909694+0000","last_fullsized":"2023-10-16T14:13:23.909694+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:23:16.061037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858862+0000","last_change":"2023-10-16T14:13:23.857247+0000","last_active":"2023-10-16T14:13:23.858862+0000","last_peered":"2023-10-16T14:13:23.858862+0000","last_clean":"2023-10-16T14:13:23.858862+0000","last_became_active":"2023-10-16T14:13:23.853524+0000","last_became_peered":"2023-10-16T14:13:23.853524+0000","last_unstale":"2023-10-16T14:13:23.858862+0000","last_undegraded":"2023-10-16T14:13:23.858862+0000","last_fullsized":"2023-10-16T14:13:23.858862+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:11.845966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909008+0000","last_change":"2023-10-16T14:13:22.818448+0000","last_active":"2023-10-16T14:13:23.909008+0000","last_peered":"2023-10-16T14:13:23.909008+0000","last_clean":"2023-10-16T14:13:23.909008+0000","last_became_active":"2023-10-16T14:13:22.818166+0000","last_became_peered":"2023-10-16T14:13:22.818166+0000","last_unstale":"2023-10-16T14:13:23.909008+0000","last_undegraded":"2023-10-16T14:13:23.909008+0000","last_fullsized":"2023-10-16T14:13:23.909008+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:45.568999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.854237+0000","last_change":"2023-10-16T14:13:23.848075+0000","last_active":"2023-10-16T14:13:23.854237+0000","last_peered":"2023-10-16T14:13:23.854237+0000","last_clean":"2023-10-16T14:13:23.854237+0000","last_became_active":"2023-10-16T14:13:23.847755+0000","last_became_peered":"2023-10-16T14:13:23.847755+0000","last_unstale":"2023-10-16T14:13:23.854237+0000","last_undegraded":"2023-10-16T14:13:23.854237+0000","last_fullsized":"2023-10-16T14:13:23.854237+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:15.024086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.910961+0000","last_change":"2023-10-16T14:13:22.817045+0000","last_active":"2023-10-16T14:13:23.910961+0000","last_peered":"2023-10-16T14:13:23.910961+0000","last_clean":"2023-10-16T14:13:23.910961+0000","last_became_active":"2023-10-16T14:13:22.816708+0000","last_became_peered":"2023-10-16T14:13:22.816708+0000","last_unstale":"2023-10-16T14:13:23.910961+0000","last_undegraded":"2023-10-16T14:13:23.910961+0000","last_fullsized":"2023-10-16T14:13:23.910961+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:58.965998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.854203+0000","last_change":"2023-10-16T14:13:23.847100+0000","last_active":"2023-10-16T14:13:23.854203+0000","last_peered":"2023-10-16T14:13:23.854203+0000","last_clean":"2023-10-16T14:13:23.854203+0000","last_became_active":"2023-10-16T14:13:23.846016+0000","last_became_peered":"2023-10-16T14:13:23.846016+0000","last_unstale":"2023-10-16T14:13:23.854203+0000","last_undegraded":"2023-10-16T14:13:23.854203+0000","last_fullsized":"2023-10-16T14:13:23.854203+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:24.762452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909972+0000","last_change":"2023-10-16T14:13:22.820319+0000","last_active":"2023-10-16T14:13:23.909972+0000","last_peered":"2023-10-16T14:13:23.909972+0000","last_clean":"2023-10-16T14:13:23.909972+0000","last_became_active":"2023-10-16T14:13:22.820006+0000","last_became_peered":"2023-10-16T14:13:22.820006+0000","last_unstale":"2023-10-16T14:13:23.909972+0000","last_undegraded":"2023-10-16T14:13:23.909972+0000","last_fullsized":"2023-10-16T14:13:23.909972+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:13.432474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818538+0000","last_change":"2023-10-16T14:13:23.839287+0000","last_active":"2023-10-16T14:13:24.818538+0000","last_peered":"2023-10-16T14:13:24.818538+0000","last_clean":"2023-10-16T14:13:24.818538+0000","last_became_active":"2023-10-16T14:13:23.837957+0000","last_became_peered":"2023-10-16T14:13:23.837957+0000","last_unstale":"2023-10-16T14:13:24.818538+0000","last_undegraded":"2023-10-16T14:13:24.818538+0000","last_fullsized":"2023-10-16T14:13:24.818538+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:12.907881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"286'1","reported_seq":13,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823276+0000","last_change":"2023-10-16T14:13:22.817425+0000","last_active":"2023-10-16T14:13:24.823276+0000","last_peered":"2023-10-16T14:13:24.823276+0000","last_clean":"2023-10-16T14:13:24.823276+0000","last_became_active":"2023-10-16T14:13:22.817145+0000","last_became_peered":"2023-10-16T14:13:22.817145+0000","last_unstale":"2023-10-16T14:13:24.823276+0000","last_undegraded":"2023-10-16T14:13:24.823276+0000","last_fullsized":"2023-10-16T14:13:24.823276+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:36.911391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821697+0000","last_change":"2023-10-16T14:13:23.850835+0000","last_active":"2023-10-16T14:13:24.821697+0000","last_peered":"2023-10-16T14:13:24.821697+0000","last_clean":"2023-10-16T14:13:24.821697+0000","last_became_active":"2023-10-16T14:13:23.849064+0000","last_became_peered":"2023-10-16T14:13:23.849064+0000","last_unstale":"2023-10-16T14:13:24.821697+0000","last_undegraded":"2023-10-16T14:13:24.821697+0000","last_fullsized":"2023-10-16T14:13:24.821697+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:27.218505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.839504+0000","last_change":"2023-10-16T14:13:22.822837+0000","last_active":"2023-10-16T14:13:23.839504+0000","last_peered":"2023-10-16T14:13:23.839504+0000","last_clean":"2023-10-16T14:13:23.839504+0000","last_became_active":"2023-10-16T14:13:22.822513+0000","last_became_peered":"2023-10-16T14:13:22.822513+0000","last_unstale":"2023-10-16T14:13:23.839504+0000","last_undegraded":"2023-10-16T14:13:23.839504+0000","last_fullsized":"2023-10-16T14:13:23.839504+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:08.670848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858124+0000","last_change":"2023-10-16T14:13:23.854575+0000","last_active":"2023-10-16T14:13:23.858124+0000","last_peered":"2023-10-16T14:13:23.858124+0000","last_clean":"2023-10-16T14:13:23.858124+0000","last_became_active":"2023-10-16T14:13:23.854306+0000","last_became_peered":"2023-10-16T14:13:23.854306+0000","last_unstale":"2023-10-16T14:13:23.858124+0000","last_undegraded":"2023-10-16T14:13:23.858124+0000","last_fullsized":"2023-10-16T14:13:23.858124+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:09.503214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.838392+0000","last_change":"2023-10-16T14:13:22.819537+0000","last_active":"2023-10-16T14:13:23.838392+0000","last_peered":"2023-10-16T14:13:23.838392+0000","last_clean":"2023-10-16T14:13:23.838392+0000","last_became_active":"2023-10-16T14:13:22.819191+0000","last_became_peered":"2023-10-16T14:13:22.819191+0000","last_unstale":"2023-10-16T14:13:23.838392+0000","last_undegraded":"2023-10-16T14:13:23.838392+0000","last_fullsized":"2023-10-16T14:13:23.838392+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:23.397691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850242+0000","last_change":"2023-10-16T14:13:23.843647+0000","last_active":"2023-10-16T14:13:23.850242+0000","last_peered":"2023-10-16T14:13:23.850242+0000","last_clean":"2023-10-16T14:13:23.850242+0000","last_became_active":"2023-10-16T14:13:23.843366+0000","last_became_peered":"2023-10-16T14:13:23.843366+0000","last_unstale":"2023-10-16T14:13:23.850242+0000","last_undegraded":"2023-10-16T14:13:23.850242+0000","last_fullsized":"2023-10-16T14:13:23.850242+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:50.205808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.831272+0000","last_change":"2023-10-16T14:13:22.818654+0000","last_active":"2023-10-16T14:13:23.831272+0000","last_peered":"2023-10-16T14:13:23.831272+0000","last_clean":"2023-10-16T14:13:23.831272+0000","last_became_active":"2023-10-16T14:13:22.818192+0000","last_became_peered":"2023-10-16T14:13:22.818192+0000","last_unstale":"2023-10-16T14:13:23.831272+0000","last_undegraded":"2023-10-16T14:13:23.831272+0000","last_fullsized":"2023-10-16T14:13:23.831272+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:12.031535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.847093+0000","last_change":"2023-10-16T14:13:23.841994+0000","last_active":"2023-10-16T14:13:23.847093+0000","last_peered":"2023-10-16T14:13:23.847093+0000","last_clean":"2023-10-16T14:13:23.847093+0000","last_became_active":"2023-10-16T14:13:23.838024+0000","last_became_peered":"2023-10-16T14:13:23.838024+0000","last_unstale":"2023-10-16T14:13:23.847093+0000","last_undegraded":"2023-10-16T14:13:23.847093+0000","last_fullsized":"2023-10-16T14:13:23.847093+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:34.307823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.829632+0000","last_change":"2023-10-16T14:13:22.818527+0000","last_active":"2023-10-16T14:13:23.829632+0000","last_peered":"2023-10-16T14:13:23.829632+0000","last_clean":"2023-10-16T14:13:23.829632+0000","last_became_active":"2023-10-16T14:13:22.818226+0000","last_became_peered":"2023-10-16T14:13:22.818226+0000","last_unstale":"2023-10-16T14:13:23.829632+0000","last_undegraded":"2023-10-16T14:13:23.829632+0000","last_fullsized":"2023-10-16T14:13:23.829632+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:06.558564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859993+0000","last_change":"2023-10-16T14:13:23.855752+0000","last_active":"2023-10-16T14:13:23.859993+0000","last_peered":"2023-10-16T14:13:23.859993+0000","last_clean":"2023-10-16T14:13:23.859993+0000","last_became_active":"2023-10-16T14:13:23.855472+0000","last_became_peered":"2023-10-16T14:13:23.855472+0000","last_unstale":"2023-10-16T14:13:23.859993+0000","last_undegraded":"2023-10-16T14:13:23.859993+0000","last_fullsized":"2023-10-16T14:13:23.859993+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:14.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"286'3","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911606+0000","last_change":"2023-10-16T14:13:22.820168+0000","last_active":"2023-10-16T14:13:23.911606+0000","last_peered":"2023-10-16T14:13:23.911606+0000","last_clean":"2023-10-16T14:13:23.911606+0000","last_became_active":"2023-10-16T14:13:22.819844+0000","last_became_peered":"2023-10-16T14:13:22.819844+0000","last_unstale":"2023-10-16T14:13:23.911606+0000","last_undegraded":"2023-10-16T14:13:23.911606+0000","last_fullsized":"2023-10-16T14:13:23.911606+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:09.626406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.852487+0000","last_change":"2023-10-16T14:13:23.848598+0000","last_active":"2023-10-16T14:13:23.852487+0000","last_peered":"2023-10-16T14:13:23.852487+0000","last_clean":"2023-10-16T14:13:23.852487+0000","last_became_active":"2023-10-16T14:13:23.848254+0000","last_became_peered":"2023-10-16T14:13:23.848254+0000","last_unstale":"2023-10-16T14:13:23.852487+0000","last_undegraded":"2023-10-16T14:13:23.852487+0000","last_fullsized":"2023-10-16T14:13:23.852487+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:46.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"286'6","reported_seq":18,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822147+0000","last_change":"2023-10-16T14:13:22.820212+0000","last_active":"2023-10-16T14:13:24.822147+0000","last_peered":"2023-10-16T14:13:24.822147+0000","last_clean":"2023-10-16T14:13:24.822147+0000","last_became_active":"2023-10-16T14:13:22.818337+0000","last_became_peered":"2023-10-16T14:13:22.818337+0000","last_unstale":"2023-10-16T14:13:24.822147+0000","last_undegraded":"2023-10-16T14:13:24.822147+0000","last_fullsized":"2023-10-16T14:13:24.822147+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.157456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.847485+0000","last_change":"2023-10-16T14:13:23.844272+0000","last_active":"2023-10-16T14:13:23.847485+0000","last_peered":"2023-10-16T14:13:23.847485+0000","last_clean":"2023-10-16T14:13:23.847485+0000","last_became_active":"2023-10-16T14:13:23.843965+0000","last_became_peered":"2023-10-16T14:13:23.843965+0000","last_unstale":"2023-10-16T14:13:23.847485+0000","last_undegraded":"2023-10-16T14:13:23.847485+0000","last_fullsized":"2023-10-16T14:13:23.847485+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:25.349259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911400+0000","last_change":"2023-10-16T14:13:22.825587+0000","last_active":"2023-10-16T14:13:23.911400+0000","last_peered":"2023-10-16T14:13:23.911400+0000","last_clean":"2023-10-16T14:13:23.911400+0000","last_became_active":"2023-10-16T14:13:22.825278+0000","last_became_peered":"2023-10-16T14:13:22.825278+0000","last_unstale":"2023-10-16T14:13:23.911400+0000","last_undegraded":"2023-10-16T14:13:23.911400+0000","last_fullsized":"2023-10-16T14:13:23.911400+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:18.768306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821656+0000","last_change":"2023-10-16T14:13:23.845312+0000","last_active":"2023-10-16T14:13:24.821656+0000","last_peered":"2023-10-16T14:13:24.821656+0000","last_clean":"2023-10-16T14:13:24.821656+0000","last_became_active":"2023-10-16T14:13:23.844839+0000","last_became_peered":"2023-10-16T14:13:23.844839+0000","last_unstale":"2023-10-16T14:13:24.821656+0000","last_undegraded":"2023-10-16T14:13:24.821656+0000","last_fullsized":"2023-10-16T14:13:24.821656+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:51.134491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.841867+0000","last_change":"2023-10-16T14:13:22.823813+0000","last_active":"2023-10-16T14:13:23.841867+0000","last_peered":"2023-10-16T14:13:23.841867+0000","last_clean":"2023-10-16T14:13:23.841867+0000","last_became_active":"2023-10-16T14:13:22.823110+0000","last_became_peered":"2023-10-16T14:13:22.823110+0000","last_unstale":"2023-10-16T14:13:23.841867+0000","last_undegraded":"2023-10-16T14:13:23.841867+0000","last_fullsized":"2023-10-16T14:13:23.841867+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:54.298210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855191+0000","last_change":"2023-10-16T14:13:23.851045+0000","last_active":"2023-10-16T14:13:23.855191+0000","last_peered":"2023-10-16T14:13:23.855191+0000","last_clean":"2023-10-16T14:13:23.855191+0000","last_became_active":"2023-10-16T14:13:23.850278+0000","last_became_peered":"2023-10-16T14:13:23.850278+0000","last_unstale":"2023-10-16T14:13:23.855191+0000","last_undegraded":"2023-10-16T14:13:23.855191+0000","last_fullsized":"2023-10-16T14:13:23.855191+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:45.381924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":12,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822834+0000","last_change":"2023-10-16T14:13:22.821018+0000","last_active":"2023-10-16T14:13:24.822834+0000","last_peered":"2023-10-16T14:13:24.822834+0000","last_clean":"2023-10-16T14:13:24.822834+0000","last_became_active":"2023-10-16T14:13:22.820028+0000","last_became_peered":"2023-10-16T14:13:22.820028+0000","last_unstale":"2023-10-16T14:13:24.822834+0000","last_undegraded":"2023-10-16T14:13:24.822834+0000","last_fullsized":"2023-10-16T14:13:24.822834+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:44.760671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821410+0000","last_change":"2023-10-16T14:13:23.836980+0000","last_active":"2023-10-16T14:13:24.821410+0000","last_peered":"2023-10-16T14:13:24.821410+0000","last_clean":"2023-10-16T14:13:24.821410+0000","last_became_active":"2023-10-16T14:13:23.836568+0000","last_became_peered":"2023-10-16T14:13:23.836568+0000","last_unstale":"2023-10-16T14:13:24.821410+0000","last_undegraded":"2023-10-16T14:13:24.821410+0000","last_fullsized":"2023-10-16T14:13:24.821410+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:43.399338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"286'8","reported_seq":19,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911379+0000","last_change":"2023-10-16T14:13:22.822079+0000","last_active":"2023-10-16T14:13:23.911379+0000","last_peered":"2023-10-16T14:13:23.911379+0000","last_clean":"2023-10-16T14:13:23.911379+0000","last_became_active":"2023-10-16T14:13:22.821034+0000","last_became_peered":"2023-10-16T14:13:22.821034+0000","last_unstale":"2023-10-16T14:13:23.911379+0000","last_undegraded":"2023-10-16T14:13:23.911379+0000","last_fullsized":"2023-10-16T14:13:23.911379+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:32.556902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859937+0000","last_change":"2023-10-16T14:13:23.852080+0000","last_active":"2023-10-16T14:13:23.859937+0000","last_peered":"2023-10-16T14:13:23.859937+0000","last_clean":"2023-10-16T14:13:23.859937+0000","last_became_active":"2023-10-16T14:13:23.850555+0000","last_became_peered":"2023-10-16T14:13:23.850555+0000","last_unstale":"2023-10-16T14:13:23.859937+0000","last_undegraded":"2023-10-16T14:13:23.859937+0000","last_fullsized":"2023-10-16T14:13:23.859937+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:37.593137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911379+0000","last_change":"2023-10-16T14:13:22.823211+0000","last_active":"2023-10-16T14:13:23.911379+0000","last_peered":"2023-10-16T14:13:23.911379+0000","last_clean":"2023-10-16T14:13:23.911379+0000","last_became_active":"2023-10-16T14:13:22.821905+0000","last_became_peered":"2023-10-16T14:13:22.821905+0000","last_unstale":"2023-10-16T14:13:23.911379+0000","last_undegraded":"2023-10-16T14:13:23.911379+0000","last_fullsized":"2023-10-16T14:13:23.911379+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:24.465598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.856263+0000","last_change":"2023-10-16T14:13:23.853353+0000","last_active":"2023-10-16T14:13:23.856263+0000","last_peered":"2023-10-16T14:13:23.856263+0000","last_clean":"2023-10-16T14:13:23.856263+0000","last_became_active":"2023-10-16T14:13:23.853007+0000","last_became_peered":"2023-10-16T14:13:23.853007+0000","last_unstale":"2023-10-16T14:13:23.856263+0000","last_undegraded":"2023-10-16T14:13:23.856263+0000","last_fullsized":"2023-10-16T14:13:23.856263+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:34.943312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.907368+0000","last_change":"2023-10-16T14:13:22.816241+0000","last_active":"2023-10-16T14:13:23.907368+0000","last_peered":"2023-10-16T14:13:23.907368+0000","last_clean":"2023-10-16T14:13:23.907368+0000","last_became_active":"2023-10-16T14:13:22.815939+0000","last_became_peered":"2023-10-16T14:13:22.815939+0000","last_unstale":"2023-10-16T14:13:23.907368+0000","last_undegraded":"2023-10-16T14:13:23.907368+0000","last_fullsized":"2023-10-16T14:13:23.907368+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:53.881997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859792+0000","last_change":"2023-10-16T14:13:23.858751+0000","last_active":"2023-10-16T14:13:23.859792+0000","last_peered":"2023-10-16T14:13:23.859792+0000","last_clean":"2023-10-16T14:13:23.859792+0000","last_became_active":"2023-10-16T14:13:23.857906+0000","last_became_peered":"2023-10-16T14:13:23.857906+0000","last_unstale":"2023-10-16T14:13:23.859792+0000","last_undegraded":"2023-10-16T14:13:23.859792+0000","last_fullsized":"2023-10-16T14:13:23.859792+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:42.426426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.908983+0000","last_change":"2023-10-16T14:13:22.804262+0000","last_active":"2023-10-16T14:13:23.908983+0000","last_peered":"2023-10-16T14:13:23.908983+0000","last_clean":"2023-10-16T14:13:23.908983+0000","last_became_active":"2023-10-16T14:13:22.803845+0000","last_became_peered":"2023-10-16T14:13:22.803845+0000","last_unstale":"2023-10-16T14:13:23.908983+0000","last_undegraded":"2023-10-16T14:13:23.908983+0000","last_fullsized":"2023-10-16T14:13:23.908983+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:00.409898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850000+0000","last_change":"2023-10-16T14:13:23.847627+0000","last_active":"2023-10-16T14:13:23.850000+0000","last_peered":"2023-10-16T14:13:23.850000+0000","last_clean":"2023-10-16T14:13:23.850000+0000","last_became_active":"2023-10-16T14:13:23.847259+0000","last_became_peered":"2023-10-16T14:13:23.847259+0000","last_unstale":"2023-10-16T14:13:23.850000+0000","last_undegraded":"2023-10-16T14:13:23.850000+0000","last_fullsized":"2023-10-16T14:13:23.850000+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:44.429647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.839301+0000","last_change":"2023-10-16T14:13:22.823755+0000","last_active":"2023-10-16T14:13:23.839301+0000","last_peered":"2023-10-16T14:13:23.839301+0000","last_clean":"2023-10-16T14:13:23.839301+0000","last_became_active":"2023-10-16T14:13:22.823064+0000","last_became_peered":"2023-10-16T14:13:22.823064+0000","last_unstale":"2023-10-16T14:13:23.839301+0000","last_undegraded":"2023-10-16T14:13:23.839301+0000","last_fullsized":"2023-10-16T14:13:23.839301+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:46.910114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855132+0000","last_change":"2023-10-16T14:13:23.850839+0000","last_active":"2023-10-16T14:13:23.855132+0000","last_peered":"2023-10-16T14:13:23.855132+0000","last_clean":"2023-10-16T14:13:23.855132+0000","last_became_active":"2023-10-16T14:13:23.849992+0000","last_became_peered":"2023-10-16T14:13:23.849992+0000","last_unstale":"2023-10-16T14:13:23.855132+0000","last_undegraded":"2023-10-16T14:13:23.855132+0000","last_fullsized":"2023-10-16T14:13:23.855132+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:29.037125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909664+0000","last_change":"2023-10-16T14:13:22.822943+0000","last_active":"2023-10-16T14:13:23.909664+0000","last_peered":"2023-10-16T14:13:23.909664+0000","last_clean":"2023-10-16T14:13:23.909664+0000","last_became_active":"2023-10-16T14:13:22.821419+0000","last_became_peered":"2023-10-16T14:13:22.821419+0000","last_unstale":"2023-10-16T14:13:23.909664+0000","last_undegraded":"2023-10-16T14:13:23.909664+0000","last_fullsized":"2023-10-16T14:13:23.909664+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:13.657920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818707+0000","last_change":"2023-10-16T14:13:23.834838+0000","last_active":"2023-10-16T14:13:24.818707+0000","last_peered":"2023-10-16T14:13:24.818707+0000","last_clean":"2023-10-16T14:13:24.818707+0000","last_became_active":"2023-10-16T14:13:23.831788+0000","last_became_peered":"2023-10-16T14:13:23.831788+0000","last_unstale":"2023-10-16T14:13:24.818707+0000","last_undegraded":"2023-10-16T14:13:24.818707+0000","last_fullsized":"2023-10-16T14:13:24.818707+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:20.506502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.832821+0000","last_change":"2023-10-16T14:13:22.817112+0000","last_active":"2023-10-16T14:13:23.832821+0000","last_peered":"2023-10-16T14:13:23.832821+0000","last_clean":"2023-10-16T14:13:23.832821+0000","last_became_active":"2023-10-16T14:13:22.816728+0000","last_became_peered":"2023-10-16T14:13:22.816728+0000","last_unstale":"2023-10-16T14:13:23.832821+0000","last_undegraded":"2023-10-16T14:13:23.832821+0000","last_fullsized":"2023-10-16T14:13:23.832821+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:07.984843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857805+0000","last_change":"2023-10-16T14:13:23.853614+0000","last_active":"2023-10-16T14:13:23.857805+0000","last_peered":"2023-10-16T14:13:23.857805+0000","last_clean":"2023-10-16T14:13:23.857805+0000","last_became_active":"2023-10-16T14:13:23.853022+0000","last_became_peered":"2023-10-16T14:13:23.853022+0000","last_unstale":"2023-10-16T14:13:23.857805+0000","last_undegraded":"2023-10-16T14:13:23.857805+0000","last_fullsized":"2023-10-16T14:13:23.857805+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:09.935977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.837038+0000","last_change":"2023-10-16T14:13:22.820743+0000","last_active":"2023-10-16T14:13:23.837038+0000","last_peered":"2023-10-16T14:13:23.837038+0000","last_clean":"2023-10-16T14:13:23.837038+0000","last_became_active":"2023-10-16T14:13:22.820466+0000","last_became_peered":"2023-10-16T14:13:22.820466+0000","last_unstale":"2023-10-16T14:13:23.837038+0000","last_undegraded":"2023-10-16T14:13:23.837038+0000","last_fullsized":"2023-10-16T14:13:23.837038+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:43.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855176+0000","last_change":"2023-10-16T14:13:23.850572+0000","last_active":"2023-10-16T14:13:23.855176+0000","last_peered":"2023-10-16T14:13:23.855176+0000","last_clean":"2023-10-16T14:13:23.855176+0000","last_became_active":"2023-10-16T14:13:23.849531+0000","last_became_peered":"2023-10-16T14:13:23.849531+0000","last_unstale":"2023-10-16T14:13:23.855176+0000","last_undegraded":"2023-10-16T14:13:23.855176+0000","last_fullsized":"2023-10-16T14:13:23.855176+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:18.870664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.846211+0000","last_change":"2023-10-16T14:13:22.819882+0000","last_active":"2023-10-16T14:13:23.846211+0000","last_peered":"2023-10-16T14:13:23.846211+0000","last_clean":"2023-10-16T14:13:23.846211+0000","last_became_active":"2023-10-16T14:13:22.819623+0000","last_became_peered":"2023-10-16T14:13:22.819623+0000","last_unstale":"2023-10-16T14:13:23.846211+0000","last_undegraded":"2023-10-16T14:13:23.846211+0000","last_fullsized":"2023-10-16T14:13:23.846211+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.819575+0000","last_change":"2023-10-16T14:13:23.840883+0000","last_active":"2023-10-16T14:13:24.819575+0000","last_peered":"2023-10-16T14:13:24.819575+0000","last_clean":"2023-10-16T14:13:24.819575+0000","last_became_active":"2023-10-16T14:13:23.840509+0000","last_became_peered":"2023-10-16T14:13:23.840509+0000","last_unstale":"2023-10-16T14:13:24.819575+0000","last_undegraded":"2023-10-16T14:13:24.819575+0000","last_fullsized":"2023-10-16T14:13:24.819575+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:55.312400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.910961+0000","last_change":"2023-10-16T14:13:22.811428+0000","last_active":"2023-10-16T14:13:23.910961+0000","last_peered":"2023-10-16T14:13:23.910961+0000","last_clean":"2023-10-16T14:13:23.910961+0000","last_became_active":"2023-10-16T14:13:22.811112+0000","last_became_peered":"2023-10-16T14:13:22.811112+0000","last_unstale":"2023-10-16T14:13:23.910961+0000","last_undegraded":"2023-10-16T14:13:23.910961+0000","last_fullsized":"2023-10-16T14:13:23.910961+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:37.312115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823452+0000","last_change":"2023-10-16T14:13:23.855446+0000","last_active":"2023-10-16T14:13:24.823452+0000","last_peered":"2023-10-16T14:13:24.823452+0000","last_clean":"2023-10-16T14:13:24.823452+0000","last_became_active":"2023-10-16T14:13:23.854283+0000","last_became_peered":"2023-10-16T14:13:23.854283+0000","last_unstale":"2023-10-16T14:13:24.823452+0000","last_undegraded":"2023-10-16T14:13:24.823452+0000","last_fullsized":"2023-10-16T14:13:24.823452+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:06.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.845190+0000","last_change":"2023-10-16T14:13:22.818518+0000","last_active":"2023-10-16T14:13:23.845190+0000","last_peered":"2023-10-16T14:13:23.845190+0000","last_clean":"2023-10-16T14:13:23.845190+0000","last_became_active":"2023-10-16T14:13:22.818237+0000","last_became_peered":"2023-10-16T14:13:22.818237+0000","last_unstale":"2023-10-16T14:13:23.845190+0000","last_undegraded":"2023-10-16T14:13:23.845190+0000","last_fullsized":"2023-10-16T14:13:23.845190+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:29.308153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.851025+0000","last_change":"2023-10-16T14:13:23.849087+0000","last_active":"2023-10-16T14:13:23.851025+0000","last_peered":"2023-10-16T14:13:23.851025+0000","last_clean":"2023-10-16T14:13:23.851025+0000","last_became_active":"2023-10-16T14:13:23.848811+0000","last_became_peered":"2023-10-16T14:13:23.848811+0000","last_unstale":"2023-10-16T14:13:23.851025+0000","last_undegraded":"2023-10-16T14:13:23.851025+0000","last_fullsized":"2023-10-16T14:13:23.851025+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:05.789670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.843494+0000","last_change":"2023-10-16T14:13:22.825845+0000","last_active":"2023-10-16T14:13:23.843494+0000","last_peered":"2023-10-16T14:13:23.843494+0000","last_clean":"2023-10-16T14:13:23.843494+0000","last_became_active":"2023-10-16T14:13:22.825550+0000","last_became_peered":"2023-10-16T14:13:22.825550+0000","last_unstale":"2023-10-16T14:13:23.843494+0000","last_undegraded":"2023-10-16T14:13:23.843494+0000","last_fullsized":"2023-10-16T14:13:23.843494+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:24.340708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822663+0000","last_change":"2023-10-16T14:13:23.839666+0000","last_active":"2023-10-16T14:13:24.822663+0000","last_peered":"2023-10-16T14:13:24.822663+0000","last_clean":"2023-10-16T14:13:24.822663+0000","last_became_active":"2023-10-16T14:13:23.839307+0000","last_became_peered":"2023-10-16T14:13:23.839307+0000","last_unstale":"2023-10-16T14:13:24.822663+0000","last_undegraded":"2023-10-16T14:13:24.822663+0000","last_fullsized":"2023-10-16T14:13:24.822663+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:01.499282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.842143+0000","last_change":"2023-10-16T14:13:22.822052+0000","last_active":"2023-10-16T14:13:23.842143+0000","last_peered":"2023-10-16T14:13:23.842143+0000","last_clean":"2023-10-16T14:13:23.842143+0000","last_became_active":"2023-10-16T14:13:22.821755+0000","last_became_peered":"2023-10-16T14:13:22.821755+0000","last_unstale":"2023-10-16T14:13:23.842143+0000","last_undegraded":"2023-10-16T14:13:23.842143+0000","last_fullsized":"2023-10-16T14:13:23.842143+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:15.887787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857504+0000","last_change":"2023-10-16T14:13:23.853934+0000","last_active":"2023-10-16T14:13:23.857504+0000","last_peered":"2023-10-16T14:13:23.857504+0000","last_clean":"2023-10-16T14:13:23.857504+0000","last_became_active":"2023-10-16T14:13:23.852820+0000","last_became_peered":"2023-10-16T14:13:23.852820+0000","last_unstale":"2023-10-16T14:13:23.857504+0000","last_undegraded":"2023-10-16T14:13:23.857504+0000","last_fullsized":"2023-10-16T14:13:23.857504+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:36.021425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.841521+0000","last_change":"2023-10-16T14:13:22.822079+0000","last_active":"2023-10-16T14:13:23.841521+0000","last_peered":"2023-10-16T14:13:23.841521+0000","last_clean":"2023-10-16T14:13:23.841521+0000","last_became_active":"2023-10-16T14:13:22.821789+0000","last_became_peered":"2023-10-16T14:13:22.821789+0000","last_unstale":"2023-10-16T14:13:23.841521+0000","last_undegraded":"2023-10-16T14:13:23.841521+0000","last_fullsized":"2023-10-16T14:13:23.841521+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:56.741371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859874+0000","last_change":"2023-10-16T14:13:23.853960+0000","last_active":"2023-10-16T14:13:23.859874+0000","last_peered":"2023-10-16T14:13:23.859874+0000","last_clean":"2023-10-16T14:13:23.859874+0000","last_became_active":"2023-10-16T14:13:23.853087+0000","last_became_peered":"2023-10-16T14:13:23.853087+0000","last_unstale":"2023-10-16T14:13:23.859874+0000","last_undegraded":"2023-10-16T14:13:23.859874+0000","last_fullsized":"2023-10-16T14:13:23.859874+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:42.880722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.842053+0000","last_change":"2023-10-16T14:13:22.823982+0000","last_active":"2023-10-16T14:13:23.842053+0000","last_peered":"2023-10-16T14:13:23.842053+0000","last_clean":"2023-10-16T14:13:23.842053+0000","last_became_active":"2023-10-16T14:13:22.823555+0000","last_became_peered":"2023-10-16T14:13:22.823555+0000","last_unstale":"2023-10-16T14:13:23.842053+0000","last_undegraded":"2023-10-16T14:13:23.842053+0000","last_fullsized":"2023-10-16T14:13:23.842053+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:05.425101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855902+0000","last_change":"2023-10-16T14:13:23.853370+0000","last_active":"2023-10-16T14:13:23.855902+0000","last_peered":"2023-10-16T14:13:23.855902+0000","last_clean":"2023-10-16T14:13:23.855902+0000","last_became_active":"2023-10-16T14:13:23.852714+0000","last_became_peered":"2023-10-16T14:13:23.852714+0000","last_unstale":"2023-10-16T14:13:23.855902+0000","last_undegraded":"2023-10-16T14:13:23.855902+0000","last_fullsized":"2023-10-16T14:13:23.855902+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:18.804667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.827954+0000","last_change":"2023-10-16T14:13:22.809610+0000","last_active":"2023-10-16T14:13:23.827954+0000","last_peered":"2023-10-16T14:13:23.827954+0000","last_clean":"2023-10-16T14:13:23.827954+0000","last_became_active":"2023-10-16T14:13:22.809204+0000","last_became_peered":"2023-10-16T14:13:22.809204+0000","last_unstale":"2023-10-16T14:13:23.827954+0000","last_undegraded":"2023-10-16T14:13:23.827954+0000","last_fullsized":"2023-10-16T14:13:23.827954+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:40.668703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821744+0000","last_change":"2023-10-16T14:13:23.834888+0000","last_active":"2023-10-16T14:13:24.821744+0000","last_peered":"2023-10-16T14:13:24.821744+0000","last_clean":"2023-10-16T14:13:24.821744+0000","last_became_active":"2023-10-16T14:13:23.834444+0000","last_became_peered":"2023-10-16T14:13:23.834444+0000","last_unstale":"2023-10-16T14:13:24.821744+0000","last_undegraded":"2023-10-16T14:13:24.821744+0000","last_fullsized":"2023-10-16T14:13:24.821744+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:42.315996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.838995+0000","last_change":"2023-10-16T14:13:22.819988+0000","last_active":"2023-10-16T14:13:23.838995+0000","last_peered":"2023-10-16T14:13:23.838995+0000","last_clean":"2023-10-16T14:13:23.838995+0000","last_became_active":"2023-10-16T14:13:22.819675+0000","last_became_peered":"2023-10-16T14:13:22.819675+0000","last_unstale":"2023-10-16T14:13:23.838995+0000","last_undegraded":"2023-10-16T14:13:23.838995+0000","last_fullsized":"2023-10-16T14:13:23.838995+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:02.851606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.860140+0000","last_change":"2023-10-16T14:13:23.856456+0000","last_active":"2023-10-16T14:13:23.860140+0000","last_peered":"2023-10-16T14:13:23.860140+0000","last_clean":"2023-10-16T14:13:23.860140+0000","last_became_active":"2023-10-16T14:13:23.856104+0000","last_became_peered":"2023-10-16T14:13:23.856104+0000","last_unstale":"2023-10-16T14:13:23.860140+0000","last_undegraded":"2023-10-16T14:13:23.860140+0000","last_fullsized":"2023-10-16T14:13:23.860140+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:01.935031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"275'2821","reported_seq":3920,"reported_epoch":285,"state":"active+clean","last_fresh":"2023-10-16T14:13:22.812309+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:13:22.812309+0000","last_peered":"2023-10-16T14:13:22.812309+0000","last_clean":"2023-10-16T14:13:22.812309+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:13:22.812309+0000","last_undegraded":"2023-10-16T14:13:22.812309+0000","last_fullsized":"2023-10-16T14:13:22.812309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2699,"num_read_kb":5660,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2699,"num_read_kb":5660,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2821,"ondisk_log_size":2821,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543330,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49180,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322660,"statfs":{"total":96636764160,"available":96586403840,"internally_reserved":0,"allocated":4202496,"data_stored":2365358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.3140000000000001,"15min":1.3480000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.641,"5min":1.8500000000000001,"15min":1.885},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.714,"5min":1.405,"15min":1.413},"min":{"1min":0.73699999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.675}]},{"osd":1,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.232,"15min":1.246},"min":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.228,"15min":1.248},"min":{"1min":0.76300000000000001,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.482}]},{"osd":2,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.252,"15min":1.268},"min":{"1min":0.85299999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.349,"15min":1.361},"min":{"1min":0.95199999999999996,"5min":0.745,"15min":0.745},"max":{"1min":3.4289999999999998,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.27}]},{"osd":3,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.357,"15min":1.405},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.097},"last":1.415},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4159999999999999,"15min":1.431},"min":{"1min":1.107,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.5920000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2589999999999999,"15min":1.242},"min":{"1min":0.94699999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.895,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4730000000000001,"15min":1.4159999999999999},"min":{"1min":0.98799999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.4450000000000001,"15min":1.395},"min":{"1min":1.026,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.546,"15min":1.538},"min":{"1min":1.0800000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.5309999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.6599999999999999,"15min":1.6379999999999999},"min":{"1min":1.21,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":3.29,"5min":3.29,"15min":3.29},"last":1.627},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5600000000000001,"15min":1.486},"min":{"1min":1.0029999999999999,"5min":0.75,"15min":0.75},"max":{"1min":3.524,"5min":3.524,"15min":3.524},"last":1.3779999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543331,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48608,"kb_used_data":3536,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323232,"statfs":{"total":96636764160,"available":96586989568,"internally_reserved":0,"allocated":3620864,"data_stored":1776602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.244,"15min":1.3109999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.887,"15min":1.887},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.3149999999999999,"15min":1.345},"min":{"1min":0.84999999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.595,"5min":1.847,"15min":1.9390000000000001},"last":1.345}]},{"osd":1,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.175,"15min":1.212},"min":{"1min":0.80000000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6220000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.306},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.202},"min":{"1min":0.67900000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.286,"5min":1.706,"15min":1.77},"last":1.264}]},{"osd":2,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2330000000000001,"15min":1.2470000000000001},"min":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.5960000000000001,"5min":1.823,"15min":1.823},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2749999999999999,"15min":1.3120000000000001},"min":{"1min":0.98799999999999999,"5min":0.745,"15min":0.745},"max":{"1min":1.619,"5min":1.962,"15min":1.962},"last":1.131}]},{"osd":3,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2829999999999999,"15min":1.333},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4830000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3919999999999999,"15min":1.405},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.5629999999999999,"5min":1.911,"15min":1.911},"last":1.2170000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.389,"15min":1.403},"min":{"1min":0.77300000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.6479999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.456},{"interface":"front","average":{"1min":1.476,"5min":1.546,"15min":1.548},"min":{"1min":1.0680000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.1179999999999999},"last":1.4330000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.88300000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0840000000000001,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.034},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4710000000000001,"15min":1.486},"min":{"1min":0.91900000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.786,"5min":2.0990000000000002,"15min":2.1970000000000001},"last":0.99399999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.506,"15min":1.5009999999999999},"min":{"1min":0.96799999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0270000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.5289999999999999,"15min":1.536},"min":{"1min":0.998,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.9990000000000001,"5min":2.157,"15min":2.6120000000000001},"last":1.194}]}]},{"osd":5,"up_from":14,"seq":60129543332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48612,"kb_used_data":3540,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323228,"statfs":{"total":96636764160,"available":96586985472,"internally_reserved":0,"allocated":3624960,"data_stored":1776692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.111,"15min":1.1879999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.546,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.369},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1919999999999999,"15min":1.22},"min":{"1min":0.79400000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.4119999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.286,"15min":1.306},"min":{"1min":1.034,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.631,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.171},{"interface":"front","average":{"1min":1.2,"5min":1.1839999999999999,"15min":1.248},"min":{"1min":0.80000000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.931,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.0429999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2909999999999999,"15min":1.333},"min":{"1min":0.82799999999999996,"5min":0.63,"15min":0.63},"max":{"1min":2.024,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.238,"15min":1.3009999999999999},"min":{"1min":0.82599999999999996,"5min":0.746,"15min":0.746},"max":{"1min":1.8600000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.482}]},{"osd":3,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3899999999999999,"15min":1.403},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.681,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":1.0049999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.827,"5min":2.75,"15min":2.75},"last":1.5289999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3160000000000001,"15min":1.298},"min":{"1min":0.86899999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6839999999999999,"5min":2.774,"15min":2.774},"last":1.254},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.4430000000000001},"min":{"1min":1.1879999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.865,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.655}]},{"osd":6,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.464,"15min":1.4390000000000001},"min":{"1min":1.0289999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9399999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.454},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5009999999999999,"15min":1.4870000000000001},"min":{"1min":1.0900000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9790000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.6200000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.5920000000000001,"15min":1.556},"min":{"1min":1.1040000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.153,"5min":2.1680000000000001,"15min":2.4609999999999999},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6220000000000001,"15min":1.6060000000000001},"min":{"1min":1.272,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.044,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.5880000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84636,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287204,"statfs":{"total":96636764160,"available":96550096896,"internally_reserved":0,"allocated":3616768,"data_stored":1775642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.151,"15min":1.1899999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.4510000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.52},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1759999999999999,"15min":1.1950000000000001},"min":{"1min":0.76100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6100000000000001,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.2989999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.157,"15min":1.2370000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.673,"5min":1.71,"15min":1.78},"last":1.079},{"interface":"front","average":{"1min":1.236,"5min":1.232,"15min":1.28},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.792,"5min":1.792,"15min":2.0499999999999998},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.86299999999999999,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.5169999999999999,"5min":1.714,"15min":1.865},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3520000000000001,"15min":1.379},"min":{"1min":1.006,"5min":0.87,"15min":0.87},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.77}]},{"osd":3,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.5980000000000001,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.395},{"interface":"front","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.4279999999999999},"min":{"1min":0.93000000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.915,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.248}]},{"osd":5,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.357,"15min":1.3400000000000001},"min":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.0670000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3899999999999999,"15min":1.349},"min":{"1min":0.89900000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.647}]},{"osd":6,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.417,"15min":1.3939999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.956},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.538,"15min":1.522},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.3279999999999998,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.8660000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.516},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.645,"15min":1.639},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.1200000000000001,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.5920000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543332,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56920,"kb_used_data":3528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314920,"statfs":{"total":96636764160,"available":96578478080,"internally_reserved":0,"allocated":3612672,"data_stored":1775572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.90700000000000003,"15min":0.94199999999999995},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":1.077,"5min":1.595,"15min":1.595},"last":1.151},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1499999999999999,"15min":1.204},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.8100000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.2070000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.04,"15min":1.0669999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.524,"5min":1.6870000000000001,"15min":1.855},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.0980000000000001,"15min":1.1140000000000001},"min":{"1min":0.52600000000000002,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.4570000000000001,"5min":2.125,"15min":2.125},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1100000000000001,"15min":1.1839999999999999},"min":{"1min":0.42999999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3160000000000001,"15min":1.4119999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.8069999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.3300000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.458,"15min":1.4970000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.67000000000000004},"max":{"1min":1.8120000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.109},{"interface":"front","average":{"1min":1.345,"5min":1.4330000000000001,"15min":1.4530000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.925,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.413,"15min":1.482},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.69699999999999995},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.508,"15min":1.5449999999999999},"min":{"1min":1.024,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":2.0209999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.4490000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.4890000000000001,"15min":1.5129999999999999},"min":{"1min":0.85899999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0739999999999998,"5min":2.335,"15min":2.4329999999999998},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.4650000000000001,"15min":1.488},"min":{"1min":0.78100000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.1699999999999999,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.86199999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5680000000000001,"15min":1.589},"min":{"1min":0.98899999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1080000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.542},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.6479999999999999,"15min":1.6679999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.2610000000000001,"5min":2.569,"15min":2.6070000000000002},"last":1.3839999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543330,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56928,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314912,"statfs":{"total":96636764160,"available":96578469888,"internally_reserved":0,"allocated":3616768,"data_stored":1775114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.93200000000000005,"15min":0.94999999999999996},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":1.2},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.47599999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.7090000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.0369999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5580000000000001,"5min":1.847,"15min":1.847},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.125,"15min":1.2210000000000001},"min":{"1min":0.67100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.415,"5min":1.784,"15min":1.9530000000000001},"last":1.415}]},{"osd":3,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.05,"15min":1.0980000000000001},"min":{"1min":0.56100000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3759999999999999,"5min":1.9179999999999999,"15min":2.1299999999999999},"last":1.264},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.252,"15min":1.3460000000000001},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.1480000000000001,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":1.4690000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.4890000000000001},"min":{"1min":0.78200000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.298,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.732},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.4790000000000001},"min":{"1min":0.88900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.659}]},{"osd":5,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.51,"15min":1.548},"min":{"1min":0.89000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.923},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.462,"15min":1.4650000000000001},"min":{"1min":0.80600000000000005,"5min":0.627,"15min":0.627},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.53}]},{"osd":6,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.524,"15min":1.52},"min":{"1min":0.91300000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.613},{"interface":"front","average":{"1min":1.524,"5min":1.5580000000000001,"15min":1.605},"min":{"1min":0.77900000000000003,"5min":0.752,"15min":0.752},"max":{"1min":2.52,"5min":2.52,"15min":2.5569999999999999},"last":2.1669999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.641,"15min":1.6599999999999999},"min":{"1min":0.996,"5min":0.754,"15min":0.754},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":2.093},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.613,"15min":1.637},"min":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.8720000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543330,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56928,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314912,"statfs":{"total":96636764160,"available":96578469888,"internally_reserved":0,"allocated":3616768,"data_stored":1775046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":1.244},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.0609999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.113},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":0.93700000000000006}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":1.3129999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61664,"kb_used_data":4112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310176,"statfs":{"total":96636764160,"available":96573620224,"internally_reserved":0,"allocated":4210688,"data_stored":2365450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.90900000000000003,"15min":0.93300000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3440000000000001,"5min":1.704,"15min":1.891},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0049999999999999,"15min":1.1279999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.6539999999999999,"15min":1.9299999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.0669999999999999,"15min":1.157},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4359999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.123,"15min":1.202},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.657,"5min":1.893,"15min":1.893},"last":0.89500000000000002}]},{"osd":3,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0640000000000001,"15min":1.141},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7589999999999999,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":0.749},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2270000000000001,"15min":1.351},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.603,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.1319999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.26,"15min":1.3500000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.923,"5min":1.923,"15min":2.0070000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.413,"15min":1.5009999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.98,"5min":2.1070000000000002,"15min":2.601},"last":1.3859999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.4430000000000001,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":2.089,"15min":2.3279999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.385,"15min":1.468},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.901,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.0349999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.5229999999999999,"15min":1.5960000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.669},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5109999999999999,"15min":1.579},"min":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0129999999999999,"5min":2.327,"15min":2.3769999999999998},"last":1.262}]},{"osd":7,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.6080000000000001,"15min":1.6879999999999999},"min":{"1min":0.76700000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0670000000000002,"5min":2.2970000000000002,"15min":2.5270000000000001},"last":1.369},{"interface":"front","average":{"1min":1.482,"5min":1.5509999999999999,"15min":1.6259999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0270000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.1839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:13:28.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:13:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.681+0000 7f7c7d793640 1 Processor -- start 2023-10-16T14:13:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.681+0000 7f7c7d793640 1 -- start start 2023-10-16T14:13:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c7d793640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c78070b60 0x7f7c7815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c7d793640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c7806c100 0x7f7c7806c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c7d793640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7815aef0 con 0x7f7c78159950 2023-10-16T14:13:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c7d793640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7815e490 con 0x7f7c7806c100 2023-10-16T14:13:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c7d793640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7815d910 con 0x7f7c78070b60 2023-10-16T14:13:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c77fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7c78159950 0x7f7c78070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37656/0 (socket says 172.21.15.60:37656) 2023-10-16T14:13:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c77fff640 1 -- 172.21.15.60:0/204047304 learned_addr learned my addr 172.21.15.60:0/204047304 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.683+0000 7f7c7cf92640 1 --2- 172.21.15.60:0/204047304 >> v2:172.21.15.60:3300/0 conn(0x7f7c7806c100 0x7f7c7806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c777fe640 1 -- 172.21.15.60:0/204047304 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434500105 0 0) 0x7f7c68001110 con 0x7f7c78159950 2023-10-16T14:13:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c6f7fe640 1 --2- 172.21.15.60:0/204047304 >> v2:172.21.15.144:3300/0 conn(0x7f7c78070b60 0x7f7c7815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c777fe640 1 -- 172.21.15.60:0/204047304 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c64003520 con 0x7f7c78159950 2023-10-16T14:13:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c7cf92640 1 -- 172.21.15.60:0/204047304 >> v2:172.21.15.144:3300/0 conn(0x7f7c78070b60 msgr2=0x7f7c7815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c7cf92640 1 --2- 172.21.15.60:0/204047304 >> v2:172.21.15.144:3300/0 conn(0x7f7c78070b60 0x7f7c7815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c7cf92640 1 -- 172.21.15.60:0/204047304 >> v1:172.21.15.60:6789/0 conn(0x7f7c78159950 legacy=0x7f7c78070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.684+0000 7f7c7cf92640 1 -- 172.21.15.60:0/204047304 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7815dbf0 con 0x7f7c7806c100 2023-10-16T14:13:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.685+0000 7f7c7cf92640 1 --2- 172.21.15.60:0/204047304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c7806c100 0x7f7c7806c510 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f7c780716a0 tx=0x7f7c7002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8ee4f44c7abcda server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.685+0000 7f7c777fe640 1 -- 172.21.15.60:0/204047304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7002cbf0 con 0x7f7c7806c100 2023-10-16T14:13:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.685+0000 7f7c777fe640 1 -- 172.21.15.60:0/204047304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7002cd50 con 0x7f7c7806c100 2023-10-16T14:13:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.685+0000 7f7c777fe640 1 -- 172.21.15.60:0/204047304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70034660 con 0x7f7c7806c100 2023-10-16T14:13:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 -- 172.21.15.60:0/204047304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c7806c100 msgr2=0x7f7c7806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 --2- 172.21.15.60:0/204047304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c7806c100 0x7f7c7806c510 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f7c780716a0 tx=0x7f7c7002be40 comp rx=0 tx=0).stop 2023-10-16T14:13:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 -- 172.21.15.60:0/204047304 shutdown_connections 2023-10-16T14:13:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 --2- 172.21.15.60:0/204047304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c7806c100 0x7f7c7806c510 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 --2- 172.21.15.60:0/204047304 >> v2:172.21.15.144:3300/0 conn(0x7f7c78070b60 0x7f7c7815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 -- 172.21.15.60:0/204047304 >> 172.21.15.60:0/204047304 conn(0x7f7c780ce6a0 msgr2=0x7f7c780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 -- 172.21.15.60:0/204047304 shutdown_connections 2023-10-16T14:13:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.686+0000 7f7c7d793640 1 -- 172.21.15.60:0/204047304 wait complete. 2023-10-16T14:13:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.688+0000 7f7c7d793640 1 Processor -- start 2023-10-16T14:13:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.688+0000 7f7c7d793640 1 -- start start 2023-10-16T14:13:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.689+0000 7f7c7d793640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c7806c100 0x7f7c780c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.689+0000 7f7c7d793640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c78070b60 0x7f7c780c2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.689+0000 7f7c7d793640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 0x7f7c7813d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.689+0000 7f7c7d793640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c78152300 con 0x7f7c78159950 2023-10-16T14:13:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.689+0000 7f7c7d793640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7815d290 con 0x7f7c78070b60 2023-10-16T14:13:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.689+0000 7f7c7d793640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7815c710 con 0x7f7c7806c100 2023-10-16T14:13:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 0x7f7c7813d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 0x7f7c7813d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47648/0 (socket says 172.21.15.60:47648) 2023-10-16T14:13:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 -- 172.21.15.60:0/271318609 learned_addr learned my addr 172.21.15.60:0/271318609 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c77fff640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c7806c100 0x7f7c780c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c6f7fe640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c78070b60 0x7f7c780c2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 -- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c7806c100 msgr2=0x7f7c780c1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c7806c100 0x7f7c780c1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 -- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c78070b60 msgr2=0x7f7c780c2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c78070b60 0x7f7c780c2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 -- 172.21.15.60:0/271318609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c70009a70 con 0x7f7c78159950 2023-10-16T14:13:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.690+0000 7f7c7cf92640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 0x7f7c7813d910 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009e80 tx=0x7f7c70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.691+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70040650 con 0x7f7c78159950 2023-10-16T14:13:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.691+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c700407b0 con 0x7f7c78159950 2023-10-16T14:13:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.691+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70040b60 con 0x7f7c78159950 2023-10-16T14:13:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.691+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7815c9f0 con 0x7f7c78159950 2023-10-16T14:13:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.691+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7815cfb0 con 0x7f7c78159950 2023-10-16T14:13:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.692+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7813b8c0 con 0x7f7c78159950 2023-10-16T14:13:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.693+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7c70040cc0 con 0x7f7c78159950 2023-10-16T14:13:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.694+0000 7f7c757fa640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7c5806afc0 0x7f7c5806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.694+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c700ac3a0 con 0x7f7c78159950 2023-10-16T14:13:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.695+0000 7f7c77fff640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7c5806afc0 0x7f7c5806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.696+0000 7f7c77fff640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7c5806afc0 0x7f7c5806d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c68007fd0 tx=0x7f7c68009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.698+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c7813b8c0 con 0x7f7c78159950 2023-10-16T14:13:28.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.890+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 --> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c7815e1b0 con 0x7f7c5806afc0 2023-10-16T14:13:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.913+0000 7f7c757fa640 1 -- 172.21.15.60:0/271318609 <== mgr.10042 v2:172.21.15.144:6832/3623959149 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287180 (secure 0 0 0) 0x7f7c68001c70 con 0x7f7c5806afc0 2023-10-16T14:13:28.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:13:28.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7c5806afc0 msgr2=0x7f7c5806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7c5806afc0 0x7f7c5806d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c68007fd0 tx=0x7f7c68009040 comp rx=0 tx=0).stop 2023-10-16T14:13:28.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 msgr2=0x7f7c7813d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:28.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 0x7f7c7813d910 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009e80 tx=0x7f7c70004a40 comp rx=0 tx=0).stop 2023-10-16T14:13:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 shutdown_connections 2023-10-16T14:13:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f7c5806afc0 0x7f7c5806d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c78159950 0x7f7c7813d910 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c78070b60 0x7f7c780c2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 --2- 172.21.15.60:0/271318609 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7c7806c100 0x7f7c780c1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 >> 172.21.15.60:0/271318609 conn(0x7f7c780ce6a0 msgr2=0x7f7c780c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 shutdown_connections 2023-10-16T14:13:28.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:28.918+0000 7f7c7d793640 1 -- 172.21.15.60:0/271318609 wait complete. 2023-10-16T14:13:28.941 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-16T14:13:28.081991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2699,"num_read_kb":5660,"num_write":733,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2859,"ondisk_log_size":2859,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":463476,"kb_used_data":29416,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754511244,"statfs":{"total":773094113280,"available":772619513856,"internally_reserved":0,"allocated":30121984,"data_stored":15385476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002216"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823007+0000","last_change":"2023-10-16T14:13:23.852963+0000","last_active":"2023-10-16T14:13:24.823007+0000","last_peered":"2023-10-16T14:13:24.823007+0000","last_clean":"2023-10-16T14:13:24.823007+0000","last_became_active":"2023-10-16T14:13:23.851773+0000","last_became_peered":"2023-10-16T14:13:23.851773+0000","last_unstale":"2023-10-16T14:13:24.823007+0000","last_undegraded":"2023-10-16T14:13:24.823007+0000","last_fullsized":"2023-10-16T14:13:24.823007+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:59.454611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.819166+0000","last_change":"2023-10-16T14:13:23.834193+0000","last_active":"2023-10-16T14:13:24.819166+0000","last_peered":"2023-10-16T14:13:24.819166+0000","last_clean":"2023-10-16T14:13:24.819166+0000","last_became_active":"2023-10-16T14:13:23.833658+0000","last_became_peered":"2023-10-16T14:13:23.833658+0000","last_unstale":"2023-10-16T14:13:24.819166+0000","last_undegraded":"2023-10-16T14:13:24.819166+0000","last_fullsized":"2023-10-16T14:13:24.819166+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:13.217594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857262+0000","last_change":"2023-10-16T14:13:23.846821+0000","last_active":"2023-10-16T14:13:23.857262+0000","last_peered":"2023-10-16T14:13:23.857262+0000","last_clean":"2023-10-16T14:13:23.857262+0000","last_became_active":"2023-10-16T14:13:23.846530+0000","last_became_peered":"2023-10-16T14:13:23.846530+0000","last_unstale":"2023-10-16T14:13:23.857262+0000","last_undegraded":"2023-10-16T14:13:23.857262+0000","last_fullsized":"2023-10-16T14:13:23.857262+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:14.529616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859972+0000","last_change":"2023-10-16T14:13:23.854726+0000","last_active":"2023-10-16T14:13:23.859972+0000","last_peered":"2023-10-16T14:13:23.859972+0000","last_clean":"2023-10-16T14:13:23.859972+0000","last_became_active":"2023-10-16T14:13:23.849834+0000","last_became_peered":"2023-10-16T14:13:23.849834+0000","last_unstale":"2023-10-16T14:13:23.859972+0000","last_undegraded":"2023-10-16T14:13:23.859972+0000","last_fullsized":"2023-10-16T14:13:23.859972+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:08.105629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859856+0000","last_change":"2023-10-16T14:13:23.851466+0000","last_active":"2023-10-16T14:13:23.859856+0000","last_peered":"2023-10-16T14:13:23.859856+0000","last_clean":"2023-10-16T14:13:23.859856+0000","last_became_active":"2023-10-16T14:13:23.851129+0000","last_became_peered":"2023-10-16T14:13:23.851129+0000","last_unstale":"2023-10-16T14:13:23.859856+0000","last_undegraded":"2023-10-16T14:13:23.859856+0000","last_fullsized":"2023-10-16T14:13:23.859856+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:17.034746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858145+0000","last_change":"2023-10-16T14:13:23.855253+0000","last_active":"2023-10-16T14:13:23.858145+0000","last_peered":"2023-10-16T14:13:23.858145+0000","last_clean":"2023-10-16T14:13:23.858145+0000","last_became_active":"2023-10-16T14:13:23.854947+0000","last_became_peered":"2023-10-16T14:13:23.854947+0000","last_unstale":"2023-10-16T14:13:23.858145+0000","last_undegraded":"2023-10-16T14:13:23.858145+0000","last_fullsized":"2023-10-16T14:13:23.858145+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:50:44.048362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821052+0000","last_change":"2023-10-16T14:13:23.843499+0000","last_active":"2023-10-16T14:13:24.821052+0000","last_peered":"2023-10-16T14:13:24.821052+0000","last_clean":"2023-10-16T14:13:24.821052+0000","last_became_active":"2023-10-16T14:13:23.841968+0000","last_became_peered":"2023-10-16T14:13:23.841968+0000","last_unstale":"2023-10-16T14:13:24.821052+0000","last_undegraded":"2023-10-16T14:13:24.821052+0000","last_fullsized":"2023-10-16T14:13:24.821052+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:49.918820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.860004+0000","last_change":"2023-10-16T14:13:23.858865+0000","last_active":"2023-10-16T14:13:23.860004+0000","last_peered":"2023-10-16T14:13:23.860004+0000","last_clean":"2023-10-16T14:13:23.860004+0000","last_became_active":"2023-10-16T14:13:23.858549+0000","last_became_peered":"2023-10-16T14:13:23.858549+0000","last_unstale":"2023-10-16T14:13:23.860004+0000","last_undegraded":"2023-10-16T14:13:23.860004+0000","last_fullsized":"2023-10-16T14:13:23.860004+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:06.232795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818945+0000","last_change":"2023-10-16T14:13:23.851913+0000","last_active":"2023-10-16T14:13:24.818945+0000","last_peered":"2023-10-16T14:13:24.818945+0000","last_clean":"2023-10-16T14:13:24.818945+0000","last_became_active":"2023-10-16T14:13:23.851624+0000","last_became_peered":"2023-10-16T14:13:23.851624+0000","last_unstale":"2023-10-16T14:13:24.818945+0000","last_undegraded":"2023-10-16T14:13:24.818945+0000","last_fullsized":"2023-10-16T14:13:24.818945+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:37:28.861779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.824149+0000","last_change":"2023-10-16T14:13:23.852441+0000","last_active":"2023-10-16T14:13:24.824149+0000","last_peered":"2023-10-16T14:13:24.824149+0000","last_clean":"2023-10-16T14:13:24.824149+0000","last_became_active":"2023-10-16T14:13:23.850746+0000","last_became_peered":"2023-10-16T14:13:23.850746+0000","last_unstale":"2023-10-16T14:13:24.824149+0000","last_undegraded":"2023-10-16T14:13:24.824149+0000","last_fullsized":"2023-10-16T14:13:24.824149+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:07.478142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.854460+0000","last_change":"2023-10-16T14:13:23.849283+0000","last_active":"2023-10-16T14:13:23.854460+0000","last_peered":"2023-10-16T14:13:23.854460+0000","last_clean":"2023-10-16T14:13:23.854460+0000","last_became_active":"2023-10-16T14:13:23.848884+0000","last_became_peered":"2023-10-16T14:13:23.848884+0000","last_unstale":"2023-10-16T14:13:23.854460+0000","last_undegraded":"2023-10-16T14:13:23.854460+0000","last_fullsized":"2023-10-16T14:13:23.854460+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:48:34.438353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.860975+0000","last_change":"2023-10-16T14:13:23.860570+0000","last_active":"2023-10-16T14:13:23.860975+0000","last_peered":"2023-10-16T14:13:23.860975+0000","last_clean":"2023-10-16T14:13:23.860975+0000","last_became_active":"2023-10-16T14:13:23.860271+0000","last_became_peered":"2023-10-16T14:13:23.860271+0000","last_unstale":"2023-10-16T14:13:23.860975+0000","last_undegraded":"2023-10-16T14:13:23.860975+0000","last_fullsized":"2023-10-16T14:13:23.860975+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:17.009125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823797+0000","last_change":"2023-10-16T14:13:23.847281+0000","last_active":"2023-10-16T14:13:24.823797+0000","last_peered":"2023-10-16T14:13:24.823797+0000","last_clean":"2023-10-16T14:13:24.823797+0000","last_became_active":"2023-10-16T14:13:23.846819+0000","last_became_peered":"2023-10-16T14:13:23.846819+0000","last_unstale":"2023-10-16T14:13:24.823797+0000","last_undegraded":"2023-10-16T14:13:24.823797+0000","last_fullsized":"2023-10-16T14:13:24.823797+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:28.376935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.831759+0000","last_change":"2023-10-16T14:13:23.829028+0000","last_active":"2023-10-16T14:13:23.831759+0000","last_peered":"2023-10-16T14:13:23.831759+0000","last_clean":"2023-10-16T14:13:23.831759+0000","last_became_active":"2023-10-16T14:13:23.828238+0000","last_became_peered":"2023-10-16T14:13:23.828238+0000","last_unstale":"2023-10-16T14:13:23.831759+0000","last_undegraded":"2023-10-16T14:13:23.831759+0000","last_fullsized":"2023-10-16T14:13:23.831759+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:26.977069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859699+0000","last_change":"2023-10-16T14:13:23.850901+0000","last_active":"2023-10-16T14:13:23.859699+0000","last_peered":"2023-10-16T14:13:23.859699+0000","last_clean":"2023-10-16T14:13:23.859699+0000","last_became_active":"2023-10-16T14:13:23.850577+0000","last_became_peered":"2023-10-16T14:13:23.850577+0000","last_unstale":"2023-10-16T14:13:23.859699+0000","last_undegraded":"2023-10-16T14:13:23.859699+0000","last_fullsized":"2023-10-16T14:13:23.859699+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:38.527703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.856205+0000","last_change":"2023-10-16T14:13:23.852934+0000","last_active":"2023-10-16T14:13:23.856205+0000","last_peered":"2023-10-16T14:13:23.856205+0000","last_clean":"2023-10-16T14:13:23.856205+0000","last_became_active":"2023-10-16T14:13:23.852647+0000","last_became_peered":"2023-10-16T14:13:23.852647+0000","last_unstale":"2023-10-16T14:13:23.856205+0000","last_undegraded":"2023-10-16T14:13:23.856205+0000","last_fullsized":"2023-10-16T14:13:23.856205+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:37.220473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822767+0000","last_change":"2023-10-16T14:13:23.859347+0000","last_active":"2023-10-16T14:13:24.822767+0000","last_peered":"2023-10-16T14:13:24.822767+0000","last_clean":"2023-10-16T14:13:24.822767+0000","last_became_active":"2023-10-16T14:13:23.858892+0000","last_became_peered":"2023-10-16T14:13:23.858892+0000","last_unstale":"2023-10-16T14:13:24.822767+0000","last_undegraded":"2023-10-16T14:13:24.822767+0000","last_fullsized":"2023-10-16T14:13:24.822767+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:44.195288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821795+0000","last_change":"2023-10-16T14:13:23.853942+0000","last_active":"2023-10-16T14:13:24.821795+0000","last_peered":"2023-10-16T14:13:24.821795+0000","last_clean":"2023-10-16T14:13:24.821795+0000","last_became_active":"2023-10-16T14:13:23.853579+0000","last_became_peered":"2023-10-16T14:13:23.853579+0000","last_unstale":"2023-10-16T14:13:24.821795+0000","last_undegraded":"2023-10-16T14:13:24.821795+0000","last_fullsized":"2023-10-16T14:13:24.821795+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:48.130917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.840340+0000","last_change":"2023-10-16T14:13:23.834012+0000","last_active":"2023-10-16T14:13:23.840340+0000","last_peered":"2023-10-16T14:13:23.840340+0000","last_clean":"2023-10-16T14:13:23.840340+0000","last_became_active":"2023-10-16T14:13:23.833222+0000","last_became_peered":"2023-10-16T14:13:23.833222+0000","last_unstale":"2023-10-16T14:13:23.840340+0000","last_undegraded":"2023-10-16T14:13:23.840340+0000","last_fullsized":"2023-10-16T14:13:23.840340+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:40.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818909+0000","last_change":"2023-10-16T14:13:23.850058+0000","last_active":"2023-10-16T14:13:24.818909+0000","last_peered":"2023-10-16T14:13:24.818909+0000","last_clean":"2023-10-16T14:13:24.818909+0000","last_became_active":"2023-10-16T14:13:23.849739+0000","last_became_peered":"2023-10-16T14:13:23.849739+0000","last_unstale":"2023-10-16T14:13:24.818909+0000","last_undegraded":"2023-10-16T14:13:24.818909+0000","last_fullsized":"2023-10-16T14:13:24.818909+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:57:20.305797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.853257+0000","last_change":"2023-10-16T14:13:23.844321+0000","last_active":"2023-10-16T14:13:23.853257+0000","last_peered":"2023-10-16T14:13:23.853257+0000","last_clean":"2023-10-16T14:13:23.853257+0000","last_became_active":"2023-10-16T14:13:23.844030+0000","last_became_peered":"2023-10-16T14:13:23.844030+0000","last_unstale":"2023-10-16T14:13:23.853257+0000","last_undegraded":"2023-10-16T14:13:23.853257+0000","last_fullsized":"2023-10-16T14:13:23.853257+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:36.878610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.842092+0000","last_change":"2023-10-16T14:13:23.836814+0000","last_active":"2023-10-16T14:13:23.842092+0000","last_peered":"2023-10-16T14:13:23.842092+0000","last_clean":"2023-10-16T14:13:23.842092+0000","last_became_active":"2023-10-16T14:13:23.836118+0000","last_became_peered":"2023-10-16T14:13:23.836118+0000","last_unstale":"2023-10-16T14:13:23.842092+0000","last_undegraded":"2023-10-16T14:13:23.842092+0000","last_fullsized":"2023-10-16T14:13:23.842092+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:38.311249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823414+0000","last_change":"2023-10-16T14:13:23.848122+0000","last_active":"2023-10-16T14:13:24.823414+0000","last_peered":"2023-10-16T14:13:24.823414+0000","last_clean":"2023-10-16T14:13:24.823414+0000","last_became_active":"2023-10-16T14:13:23.846297+0000","last_became_peered":"2023-10-16T14:13:23.846297+0000","last_unstale":"2023-10-16T14:13:24.823414+0000","last_undegraded":"2023-10-16T14:13:24.823414+0000","last_fullsized":"2023-10-16T14:13:24.823414+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:06.788358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857734+0000","last_change":"2023-10-16T14:13:23.857062+0000","last_active":"2023-10-16T14:13:23.857734+0000","last_peered":"2023-10-16T14:13:23.857734+0000","last_clean":"2023-10-16T14:13:23.857734+0000","last_became_active":"2023-10-16T14:13:23.856691+0000","last_became_peered":"2023-10-16T14:13:23.856691+0000","last_unstale":"2023-10-16T14:13:23.857734+0000","last_undegraded":"2023-10-16T14:13:23.857734+0000","last_fullsized":"2023-10-16T14:13:23.857734+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:44.517072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823116+0000","last_change":"2023-10-16T14:13:23.856591+0000","last_active":"2023-10-16T14:13:24.823116+0000","last_peered":"2023-10-16T14:13:24.823116+0000","last_clean":"2023-10-16T14:13:24.823116+0000","last_became_active":"2023-10-16T14:13:23.856101+0000","last_became_peered":"2023-10-16T14:13:23.856101+0000","last_unstale":"2023-10-16T14:13:24.823116+0000","last_undegraded":"2023-10-16T14:13:24.823116+0000","last_fullsized":"2023-10-16T14:13:24.823116+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:59.837730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.824284+0000","last_change":"2023-10-16T14:13:23.841073+0000","last_active":"2023-10-16T14:13:24.824284+0000","last_peered":"2023-10-16T14:13:24.824284+0000","last_clean":"2023-10-16T14:13:24.824284+0000","last_became_active":"2023-10-16T14:13:23.840723+0000","last_became_peered":"2023-10-16T14:13:23.840723+0000","last_unstale":"2023-10-16T14:13:24.824284+0000","last_undegraded":"2023-10-16T14:13:24.824284+0000","last_fullsized":"2023-10-16T14:13:24.824284+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:07.676069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.853432+0000","last_change":"2023-10-16T14:13:23.846346+0000","last_active":"2023-10-16T14:13:23.853432+0000","last_peered":"2023-10-16T14:13:23.853432+0000","last_clean":"2023-10-16T14:13:23.853432+0000","last_became_active":"2023-10-16T14:13:23.846023+0000","last_became_peered":"2023-10-16T14:13:23.846023+0000","last_unstale":"2023-10-16T14:13:23.853432+0000","last_undegraded":"2023-10-16T14:13:23.853432+0000","last_fullsized":"2023-10-16T14:13:23.853432+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:29.108473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850512+0000","last_change":"2023-10-16T14:13:23.845286+0000","last_active":"2023-10-16T14:13:23.850512+0000","last_peered":"2023-10-16T14:13:23.850512+0000","last_clean":"2023-10-16T14:13:23.850512+0000","last_became_active":"2023-10-16T14:13:23.844831+0000","last_became_peered":"2023-10-16T14:13:23.844831+0000","last_unstale":"2023-10-16T14:13:23.850512+0000","last_undegraded":"2023-10-16T14:13:23.850512+0000","last_fullsized":"2023-10-16T14:13:23.850512+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:05:12.889338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823425+0000","last_change":"2023-10-16T14:13:23.855638+0000","last_active":"2023-10-16T14:13:24.823425+0000","last_peered":"2023-10-16T14:13:24.823425+0000","last_clean":"2023-10-16T14:13:24.823425+0000","last_became_active":"2023-10-16T14:13:23.855345+0000","last_became_peered":"2023-10-16T14:13:23.855345+0000","last_unstale":"2023-10-16T14:13:24.823425+0000","last_undegraded":"2023-10-16T14:13:24.823425+0000","last_fullsized":"2023-10-16T14:13:24.823425+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:21.983883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850867+0000","last_change":"2023-10-16T14:13:23.848508+0000","last_active":"2023-10-16T14:13:23.850867+0000","last_peered":"2023-10-16T14:13:23.850867+0000","last_clean":"2023-10-16T14:13:23.850867+0000","last_became_active":"2023-10-16T14:13:23.848208+0000","last_became_peered":"2023-10-16T14:13:23.848208+0000","last_unstale":"2023-10-16T14:13:23.850867+0000","last_undegraded":"2023-10-16T14:13:23.850867+0000","last_fullsized":"2023-10-16T14:13:23.850867+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:16.217629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858346+0000","last_change":"2023-10-16T14:13:23.848976+0000","last_active":"2023-10-16T14:13:23.858346+0000","last_peered":"2023-10-16T14:13:23.858346+0000","last_clean":"2023-10-16T14:13:23.858346+0000","last_became_active":"2023-10-16T14:13:23.848703+0000","last_became_peered":"2023-10-16T14:13:23.848703+0000","last_unstale":"2023-10-16T14:13:23.858346+0000","last_undegraded":"2023-10-16T14:13:23.858346+0000","last_fullsized":"2023-10-16T14:13:23.858346+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:50:53.825346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.856246+0000","last_change":"2023-10-16T14:13:23.851765+0000","last_active":"2023-10-16T14:13:23.856246+0000","last_peered":"2023-10-16T14:13:23.856246+0000","last_clean":"2023-10-16T14:13:23.856246+0000","last_became_active":"2023-10-16T14:13:23.849363+0000","last_became_peered":"2023-10-16T14:13:23.849363+0000","last_unstale":"2023-10-16T14:13:23.856246+0000","last_undegraded":"2023-10-16T14:13:23.856246+0000","last_fullsized":"2023-10-16T14:13:23.856246+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:51:42.205020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"286'4","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911069+0000","last_change":"2023-10-16T14:13:22.809923+0000","last_active":"2023-10-16T14:13:23.911069+0000","last_peered":"2023-10-16T14:13:23.911069+0000","last_clean":"2023-10-16T14:13:23.911069+0000","last_became_active":"2023-10-16T14:13:22.809574+0000","last_became_peered":"2023-10-16T14:13:22.809574+0000","last_unstale":"2023-10-16T14:13:23.911069+0000","last_undegraded":"2023-10-16T14:13:23.911069+0000","last_fullsized":"2023-10-16T14:13:23.911069+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:37.337590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.852619+0000","last_change":"2023-10-16T14:13:23.850634+0000","last_active":"2023-10-16T14:13:23.852619+0000","last_peered":"2023-10-16T14:13:23.852619+0000","last_clean":"2023-10-16T14:13:23.852619+0000","last_became_active":"2023-10-16T14:13:23.850341+0000","last_became_peered":"2023-10-16T14:13:23.850341+0000","last_unstale":"2023-10-16T14:13:23.852619+0000","last_undegraded":"2023-10-16T14:13:23.852619+0000","last_fullsized":"2023-10-16T14:13:23.852619+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:52.117283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909694+0000","last_change":"2023-10-16T14:13:22.822269+0000","last_active":"2023-10-16T14:13:23.909694+0000","last_peered":"2023-10-16T14:13:23.909694+0000","last_clean":"2023-10-16T14:13:23.909694+0000","last_became_active":"2023-10-16T14:13:22.821906+0000","last_became_peered":"2023-10-16T14:13:22.821906+0000","last_unstale":"2023-10-16T14:13:23.909694+0000","last_undegraded":"2023-10-16T14:13:23.909694+0000","last_fullsized":"2023-10-16T14:13:23.909694+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:23:16.061037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858862+0000","last_change":"2023-10-16T14:13:23.857247+0000","last_active":"2023-10-16T14:13:23.858862+0000","last_peered":"2023-10-16T14:13:23.858862+0000","last_clean":"2023-10-16T14:13:23.858862+0000","last_became_active":"2023-10-16T14:13:23.853524+0000","last_became_peered":"2023-10-16T14:13:23.853524+0000","last_unstale":"2023-10-16T14:13:23.858862+0000","last_undegraded":"2023-10-16T14:13:23.858862+0000","last_fullsized":"2023-10-16T14:13:23.858862+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:11.845966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909008+0000","last_change":"2023-10-16T14:13:22.818448+0000","last_active":"2023-10-16T14:13:23.909008+0000","last_peered":"2023-10-16T14:13:23.909008+0000","last_clean":"2023-10-16T14:13:23.909008+0000","last_became_active":"2023-10-16T14:13:22.818166+0000","last_became_peered":"2023-10-16T14:13:22.818166+0000","last_unstale":"2023-10-16T14:13:23.909008+0000","last_undegraded":"2023-10-16T14:13:23.909008+0000","last_fullsized":"2023-10-16T14:13:23.909008+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:45.568999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.854237+0000","last_change":"2023-10-16T14:13:23.848075+0000","last_active":"2023-10-16T14:13:23.854237+0000","last_peered":"2023-10-16T14:13:23.854237+0000","last_clean":"2023-10-16T14:13:23.854237+0000","last_became_active":"2023-10-16T14:13:23.847755+0000","last_became_peered":"2023-10-16T14:13:23.847755+0000","last_unstale":"2023-10-16T14:13:23.854237+0000","last_undegraded":"2023-10-16T14:13:23.854237+0000","last_fullsized":"2023-10-16T14:13:23.854237+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:15.024086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.910961+0000","last_change":"2023-10-16T14:13:22.817045+0000","last_active":"2023-10-16T14:13:23.910961+0000","last_peered":"2023-10-16T14:13:23.910961+0000","last_clean":"2023-10-16T14:13:23.910961+0000","last_became_active":"2023-10-16T14:13:22.816708+0000","last_became_peered":"2023-10-16T14:13:22.816708+0000","last_unstale":"2023-10-16T14:13:23.910961+0000","last_undegraded":"2023-10-16T14:13:23.910961+0000","last_fullsized":"2023-10-16T14:13:23.910961+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:58.965998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.854203+0000","last_change":"2023-10-16T14:13:23.847100+0000","last_active":"2023-10-16T14:13:23.854203+0000","last_peered":"2023-10-16T14:13:23.854203+0000","last_clean":"2023-10-16T14:13:23.854203+0000","last_became_active":"2023-10-16T14:13:23.846016+0000","last_became_peered":"2023-10-16T14:13:23.846016+0000","last_unstale":"2023-10-16T14:13:23.854203+0000","last_undegraded":"2023-10-16T14:13:23.854203+0000","last_fullsized":"2023-10-16T14:13:23.854203+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:28:24.762452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909972+0000","last_change":"2023-10-16T14:13:22.820319+0000","last_active":"2023-10-16T14:13:23.909972+0000","last_peered":"2023-10-16T14:13:23.909972+0000","last_clean":"2023-10-16T14:13:23.909972+0000","last_became_active":"2023-10-16T14:13:22.820006+0000","last_became_peered":"2023-10-16T14:13:22.820006+0000","last_unstale":"2023-10-16T14:13:23.909972+0000","last_undegraded":"2023-10-16T14:13:23.909972+0000","last_fullsized":"2023-10-16T14:13:23.909972+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:13.432474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818538+0000","last_change":"2023-10-16T14:13:23.839287+0000","last_active":"2023-10-16T14:13:24.818538+0000","last_peered":"2023-10-16T14:13:24.818538+0000","last_clean":"2023-10-16T14:13:24.818538+0000","last_became_active":"2023-10-16T14:13:23.837957+0000","last_became_peered":"2023-10-16T14:13:23.837957+0000","last_unstale":"2023-10-16T14:13:24.818538+0000","last_undegraded":"2023-10-16T14:13:24.818538+0000","last_fullsized":"2023-10-16T14:13:24.818538+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:12.907881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"286'1","reported_seq":13,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823276+0000","last_change":"2023-10-16T14:13:22.817425+0000","last_active":"2023-10-16T14:13:24.823276+0000","last_peered":"2023-10-16T14:13:24.823276+0000","last_clean":"2023-10-16T14:13:24.823276+0000","last_became_active":"2023-10-16T14:13:22.817145+0000","last_became_peered":"2023-10-16T14:13:22.817145+0000","last_unstale":"2023-10-16T14:13:24.823276+0000","last_undegraded":"2023-10-16T14:13:24.823276+0000","last_fullsized":"2023-10-16T14:13:24.823276+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:36.911391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821697+0000","last_change":"2023-10-16T14:13:23.850835+0000","last_active":"2023-10-16T14:13:24.821697+0000","last_peered":"2023-10-16T14:13:24.821697+0000","last_clean":"2023-10-16T14:13:24.821697+0000","last_became_active":"2023-10-16T14:13:23.849064+0000","last_became_peered":"2023-10-16T14:13:23.849064+0000","last_unstale":"2023-10-16T14:13:24.821697+0000","last_undegraded":"2023-10-16T14:13:24.821697+0000","last_fullsized":"2023-10-16T14:13:24.821697+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:27.218505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.839504+0000","last_change":"2023-10-16T14:13:22.822837+0000","last_active":"2023-10-16T14:13:23.839504+0000","last_peered":"2023-10-16T14:13:23.839504+0000","last_clean":"2023-10-16T14:13:23.839504+0000","last_became_active":"2023-10-16T14:13:22.822513+0000","last_became_peered":"2023-10-16T14:13:22.822513+0000","last_unstale":"2023-10-16T14:13:23.839504+0000","last_undegraded":"2023-10-16T14:13:23.839504+0000","last_fullsized":"2023-10-16T14:13:23.839504+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:08.670848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.858124+0000","last_change":"2023-10-16T14:13:23.854575+0000","last_active":"2023-10-16T14:13:23.858124+0000","last_peered":"2023-10-16T14:13:23.858124+0000","last_clean":"2023-10-16T14:13:23.858124+0000","last_became_active":"2023-10-16T14:13:23.854306+0000","last_became_peered":"2023-10-16T14:13:23.854306+0000","last_unstale":"2023-10-16T14:13:23.858124+0000","last_undegraded":"2023-10-16T14:13:23.858124+0000","last_fullsized":"2023-10-16T14:13:23.858124+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:09.503214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.838392+0000","last_change":"2023-10-16T14:13:22.819537+0000","last_active":"2023-10-16T14:13:23.838392+0000","last_peered":"2023-10-16T14:13:23.838392+0000","last_clean":"2023-10-16T14:13:23.838392+0000","last_became_active":"2023-10-16T14:13:22.819191+0000","last_became_peered":"2023-10-16T14:13:22.819191+0000","last_unstale":"2023-10-16T14:13:23.838392+0000","last_undegraded":"2023-10-16T14:13:23.838392+0000","last_fullsized":"2023-10-16T14:13:23.838392+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:16:23.397691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850242+0000","last_change":"2023-10-16T14:13:23.843647+0000","last_active":"2023-10-16T14:13:23.850242+0000","last_peered":"2023-10-16T14:13:23.850242+0000","last_clean":"2023-10-16T14:13:23.850242+0000","last_became_active":"2023-10-16T14:13:23.843366+0000","last_became_peered":"2023-10-16T14:13:23.843366+0000","last_unstale":"2023-10-16T14:13:23.850242+0000","last_undegraded":"2023-10-16T14:13:23.850242+0000","last_fullsized":"2023-10-16T14:13:23.850242+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:50.205808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.831272+0000","last_change":"2023-10-16T14:13:22.818654+0000","last_active":"2023-10-16T14:13:23.831272+0000","last_peered":"2023-10-16T14:13:23.831272+0000","last_clean":"2023-10-16T14:13:23.831272+0000","last_became_active":"2023-10-16T14:13:22.818192+0000","last_became_peered":"2023-10-16T14:13:22.818192+0000","last_unstale":"2023-10-16T14:13:23.831272+0000","last_undegraded":"2023-10-16T14:13:23.831272+0000","last_fullsized":"2023-10-16T14:13:23.831272+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:12.031535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.847093+0000","last_change":"2023-10-16T14:13:23.841994+0000","last_active":"2023-10-16T14:13:23.847093+0000","last_peered":"2023-10-16T14:13:23.847093+0000","last_clean":"2023-10-16T14:13:23.847093+0000","last_became_active":"2023-10-16T14:13:23.838024+0000","last_became_peered":"2023-10-16T14:13:23.838024+0000","last_unstale":"2023-10-16T14:13:23.847093+0000","last_undegraded":"2023-10-16T14:13:23.847093+0000","last_fullsized":"2023-10-16T14:13:23.847093+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:34.307823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.829632+0000","last_change":"2023-10-16T14:13:22.818527+0000","last_active":"2023-10-16T14:13:23.829632+0000","last_peered":"2023-10-16T14:13:23.829632+0000","last_clean":"2023-10-16T14:13:23.829632+0000","last_became_active":"2023-10-16T14:13:22.818226+0000","last_became_peered":"2023-10-16T14:13:22.818226+0000","last_unstale":"2023-10-16T14:13:23.829632+0000","last_undegraded":"2023-10-16T14:13:23.829632+0000","last_fullsized":"2023-10-16T14:13:23.829632+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:06.558564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859993+0000","last_change":"2023-10-16T14:13:23.855752+0000","last_active":"2023-10-16T14:13:23.859993+0000","last_peered":"2023-10-16T14:13:23.859993+0000","last_clean":"2023-10-16T14:13:23.859993+0000","last_became_active":"2023-10-16T14:13:23.855472+0000","last_became_peered":"2023-10-16T14:13:23.855472+0000","last_unstale":"2023-10-16T14:13:23.859993+0000","last_undegraded":"2023-10-16T14:13:23.859993+0000","last_fullsized":"2023-10-16T14:13:23.859993+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:14.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"286'3","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911606+0000","last_change":"2023-10-16T14:13:22.820168+0000","last_active":"2023-10-16T14:13:23.911606+0000","last_peered":"2023-10-16T14:13:23.911606+0000","last_clean":"2023-10-16T14:13:23.911606+0000","last_became_active":"2023-10-16T14:13:22.819844+0000","last_became_peered":"2023-10-16T14:13:22.819844+0000","last_unstale":"2023-10-16T14:13:23.911606+0000","last_undegraded":"2023-10-16T14:13:23.911606+0000","last_fullsized":"2023-10-16T14:13:23.911606+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:09.626406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.852487+0000","last_change":"2023-10-16T14:13:23.848598+0000","last_active":"2023-10-16T14:13:23.852487+0000","last_peered":"2023-10-16T14:13:23.852487+0000","last_clean":"2023-10-16T14:13:23.852487+0000","last_became_active":"2023-10-16T14:13:23.848254+0000","last_became_peered":"2023-10-16T14:13:23.848254+0000","last_unstale":"2023-10-16T14:13:23.852487+0000","last_undegraded":"2023-10-16T14:13:23.852487+0000","last_fullsized":"2023-10-16T14:13:23.852487+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:00:46.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"286'6","reported_seq":18,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822147+0000","last_change":"2023-10-16T14:13:22.820212+0000","last_active":"2023-10-16T14:13:24.822147+0000","last_peered":"2023-10-16T14:13:24.822147+0000","last_clean":"2023-10-16T14:13:24.822147+0000","last_became_active":"2023-10-16T14:13:22.818337+0000","last_became_peered":"2023-10-16T14:13:22.818337+0000","last_unstale":"2023-10-16T14:13:24.822147+0000","last_undegraded":"2023-10-16T14:13:24.822147+0000","last_fullsized":"2023-10-16T14:13:24.822147+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:33:25.157456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.847485+0000","last_change":"2023-10-16T14:13:23.844272+0000","last_active":"2023-10-16T14:13:23.847485+0000","last_peered":"2023-10-16T14:13:23.847485+0000","last_clean":"2023-10-16T14:13:23.847485+0000","last_became_active":"2023-10-16T14:13:23.843965+0000","last_became_peered":"2023-10-16T14:13:23.843965+0000","last_unstale":"2023-10-16T14:13:23.847485+0000","last_undegraded":"2023-10-16T14:13:23.847485+0000","last_fullsized":"2023-10-16T14:13:23.847485+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:25.349259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911400+0000","last_change":"2023-10-16T14:13:22.825587+0000","last_active":"2023-10-16T14:13:23.911400+0000","last_peered":"2023-10-16T14:13:23.911400+0000","last_clean":"2023-10-16T14:13:23.911400+0000","last_became_active":"2023-10-16T14:13:22.825278+0000","last_became_peered":"2023-10-16T14:13:22.825278+0000","last_unstale":"2023-10-16T14:13:23.911400+0000","last_undegraded":"2023-10-16T14:13:23.911400+0000","last_fullsized":"2023-10-16T14:13:23.911400+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:18.768306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821656+0000","last_change":"2023-10-16T14:13:23.845312+0000","last_active":"2023-10-16T14:13:24.821656+0000","last_peered":"2023-10-16T14:13:24.821656+0000","last_clean":"2023-10-16T14:13:24.821656+0000","last_became_active":"2023-10-16T14:13:23.844839+0000","last_became_peered":"2023-10-16T14:13:23.844839+0000","last_unstale":"2023-10-16T14:13:24.821656+0000","last_undegraded":"2023-10-16T14:13:24.821656+0000","last_fullsized":"2023-10-16T14:13:24.821656+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:51.134491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.841867+0000","last_change":"2023-10-16T14:13:22.823813+0000","last_active":"2023-10-16T14:13:23.841867+0000","last_peered":"2023-10-16T14:13:23.841867+0000","last_clean":"2023-10-16T14:13:23.841867+0000","last_became_active":"2023-10-16T14:13:22.823110+0000","last_became_peered":"2023-10-16T14:13:22.823110+0000","last_unstale":"2023-10-16T14:13:23.841867+0000","last_undegraded":"2023-10-16T14:13:23.841867+0000","last_fullsized":"2023-10-16T14:13:23.841867+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:54.298210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855191+0000","last_change":"2023-10-16T14:13:23.851045+0000","last_active":"2023-10-16T14:13:23.855191+0000","last_peered":"2023-10-16T14:13:23.855191+0000","last_clean":"2023-10-16T14:13:23.855191+0000","last_became_active":"2023-10-16T14:13:23.850278+0000","last_became_peered":"2023-10-16T14:13:23.850278+0000","last_unstale":"2023-10-16T14:13:23.855191+0000","last_undegraded":"2023-10-16T14:13:23.855191+0000","last_fullsized":"2023-10-16T14:13:23.855191+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:49:45.381924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":12,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822834+0000","last_change":"2023-10-16T14:13:22.821018+0000","last_active":"2023-10-16T14:13:24.822834+0000","last_peered":"2023-10-16T14:13:24.822834+0000","last_clean":"2023-10-16T14:13:24.822834+0000","last_became_active":"2023-10-16T14:13:22.820028+0000","last_became_peered":"2023-10-16T14:13:22.820028+0000","last_unstale":"2023-10-16T14:13:24.822834+0000","last_undegraded":"2023-10-16T14:13:24.822834+0000","last_fullsized":"2023-10-16T14:13:24.822834+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:44.760671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821410+0000","last_change":"2023-10-16T14:13:23.836980+0000","last_active":"2023-10-16T14:13:24.821410+0000","last_peered":"2023-10-16T14:13:24.821410+0000","last_clean":"2023-10-16T14:13:24.821410+0000","last_became_active":"2023-10-16T14:13:23.836568+0000","last_became_peered":"2023-10-16T14:13:23.836568+0000","last_unstale":"2023-10-16T14:13:24.821410+0000","last_undegraded":"2023-10-16T14:13:24.821410+0000","last_fullsized":"2023-10-16T14:13:24.821410+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:43.399338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"286'8","reported_seq":19,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911379+0000","last_change":"2023-10-16T14:13:22.822079+0000","last_active":"2023-10-16T14:13:23.911379+0000","last_peered":"2023-10-16T14:13:23.911379+0000","last_clean":"2023-10-16T14:13:23.911379+0000","last_became_active":"2023-10-16T14:13:22.821034+0000","last_became_peered":"2023-10-16T14:13:22.821034+0000","last_unstale":"2023-10-16T14:13:23.911379+0000","last_undegraded":"2023-10-16T14:13:23.911379+0000","last_fullsized":"2023-10-16T14:13:23.911379+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:32.556902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859937+0000","last_change":"2023-10-16T14:13:23.852080+0000","last_active":"2023-10-16T14:13:23.859937+0000","last_peered":"2023-10-16T14:13:23.859937+0000","last_clean":"2023-10-16T14:13:23.859937+0000","last_became_active":"2023-10-16T14:13:23.850555+0000","last_became_peered":"2023-10-16T14:13:23.850555+0000","last_unstale":"2023-10-16T14:13:23.859937+0000","last_undegraded":"2023-10-16T14:13:23.859937+0000","last_fullsized":"2023-10-16T14:13:23.859937+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:37.593137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.911379+0000","last_change":"2023-10-16T14:13:22.823211+0000","last_active":"2023-10-16T14:13:23.911379+0000","last_peered":"2023-10-16T14:13:23.911379+0000","last_clean":"2023-10-16T14:13:23.911379+0000","last_became_active":"2023-10-16T14:13:22.821905+0000","last_became_peered":"2023-10-16T14:13:22.821905+0000","last_unstale":"2023-10-16T14:13:23.911379+0000","last_undegraded":"2023-10-16T14:13:23.911379+0000","last_fullsized":"2023-10-16T14:13:23.911379+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:31:24.465598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.856263+0000","last_change":"2023-10-16T14:13:23.853353+0000","last_active":"2023-10-16T14:13:23.856263+0000","last_peered":"2023-10-16T14:13:23.856263+0000","last_clean":"2023-10-16T14:13:23.856263+0000","last_became_active":"2023-10-16T14:13:23.853007+0000","last_became_peered":"2023-10-16T14:13:23.853007+0000","last_unstale":"2023-10-16T14:13:23.856263+0000","last_undegraded":"2023-10-16T14:13:23.856263+0000","last_fullsized":"2023-10-16T14:13:23.856263+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:34.943312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.907368+0000","last_change":"2023-10-16T14:13:22.816241+0000","last_active":"2023-10-16T14:13:23.907368+0000","last_peered":"2023-10-16T14:13:23.907368+0000","last_clean":"2023-10-16T14:13:23.907368+0000","last_became_active":"2023-10-16T14:13:22.815939+0000","last_became_peered":"2023-10-16T14:13:22.815939+0000","last_unstale":"2023-10-16T14:13:23.907368+0000","last_undegraded":"2023-10-16T14:13:23.907368+0000","last_fullsized":"2023-10-16T14:13:23.907368+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:53.881997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859792+0000","last_change":"2023-10-16T14:13:23.858751+0000","last_active":"2023-10-16T14:13:23.859792+0000","last_peered":"2023-10-16T14:13:23.859792+0000","last_clean":"2023-10-16T14:13:23.859792+0000","last_became_active":"2023-10-16T14:13:23.857906+0000","last_became_peered":"2023-10-16T14:13:23.857906+0000","last_unstale":"2023-10-16T14:13:23.859792+0000","last_undegraded":"2023-10-16T14:13:23.859792+0000","last_fullsized":"2023-10-16T14:13:23.859792+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:42.426426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"286'1","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.908983+0000","last_change":"2023-10-16T14:13:22.804262+0000","last_active":"2023-10-16T14:13:23.908983+0000","last_peered":"2023-10-16T14:13:23.908983+0000","last_clean":"2023-10-16T14:13:23.908983+0000","last_became_active":"2023-10-16T14:13:22.803845+0000","last_became_peered":"2023-10-16T14:13:22.803845+0000","last_unstale":"2023-10-16T14:13:23.908983+0000","last_undegraded":"2023-10-16T14:13:23.908983+0000","last_fullsized":"2023-10-16T14:13:23.908983+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:00.409898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.850000+0000","last_change":"2023-10-16T14:13:23.847627+0000","last_active":"2023-10-16T14:13:23.850000+0000","last_peered":"2023-10-16T14:13:23.850000+0000","last_clean":"2023-10-16T14:13:23.850000+0000","last_became_active":"2023-10-16T14:13:23.847259+0000","last_became_peered":"2023-10-16T14:13:23.847259+0000","last_unstale":"2023-10-16T14:13:23.850000+0000","last_undegraded":"2023-10-16T14:13:23.850000+0000","last_fullsized":"2023-10-16T14:13:23.850000+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:44.429647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.839301+0000","last_change":"2023-10-16T14:13:22.823755+0000","last_active":"2023-10-16T14:13:23.839301+0000","last_peered":"2023-10-16T14:13:23.839301+0000","last_clean":"2023-10-16T14:13:23.839301+0000","last_became_active":"2023-10-16T14:13:22.823064+0000","last_became_peered":"2023-10-16T14:13:22.823064+0000","last_unstale":"2023-10-16T14:13:23.839301+0000","last_undegraded":"2023-10-16T14:13:23.839301+0000","last_fullsized":"2023-10-16T14:13:23.839301+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:46.910114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855132+0000","last_change":"2023-10-16T14:13:23.850839+0000","last_active":"2023-10-16T14:13:23.855132+0000","last_peered":"2023-10-16T14:13:23.855132+0000","last_clean":"2023-10-16T14:13:23.855132+0000","last_became_active":"2023-10-16T14:13:23.849992+0000","last_became_peered":"2023-10-16T14:13:23.849992+0000","last_unstale":"2023-10-16T14:13:23.855132+0000","last_undegraded":"2023-10-16T14:13:23.855132+0000","last_fullsized":"2023-10-16T14:13:23.855132+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:29.037125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.909664+0000","last_change":"2023-10-16T14:13:22.822943+0000","last_active":"2023-10-16T14:13:23.909664+0000","last_peered":"2023-10-16T14:13:23.909664+0000","last_clean":"2023-10-16T14:13:23.909664+0000","last_became_active":"2023-10-16T14:13:22.821419+0000","last_became_peered":"2023-10-16T14:13:22.821419+0000","last_unstale":"2023-10-16T14:13:23.909664+0000","last_undegraded":"2023-10-16T14:13:23.909664+0000","last_fullsized":"2023-10-16T14:13:23.909664+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:13.657920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.818707+0000","last_change":"2023-10-16T14:13:23.834838+0000","last_active":"2023-10-16T14:13:24.818707+0000","last_peered":"2023-10-16T14:13:24.818707+0000","last_clean":"2023-10-16T14:13:24.818707+0000","last_became_active":"2023-10-16T14:13:23.831788+0000","last_became_peered":"2023-10-16T14:13:23.831788+0000","last_unstale":"2023-10-16T14:13:24.818707+0000","last_undegraded":"2023-10-16T14:13:24.818707+0000","last_fullsized":"2023-10-16T14:13:24.818707+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:20.506502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.832821+0000","last_change":"2023-10-16T14:13:22.817112+0000","last_active":"2023-10-16T14:13:23.832821+0000","last_peered":"2023-10-16T14:13:23.832821+0000","last_clean":"2023-10-16T14:13:23.832821+0000","last_became_active":"2023-10-16T14:13:22.816728+0000","last_became_peered":"2023-10-16T14:13:22.816728+0000","last_unstale":"2023-10-16T14:13:23.832821+0000","last_undegraded":"2023-10-16T14:13:23.832821+0000","last_fullsized":"2023-10-16T14:13:23.832821+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:07.984843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857805+0000","last_change":"2023-10-16T14:13:23.853614+0000","last_active":"2023-10-16T14:13:23.857805+0000","last_peered":"2023-10-16T14:13:23.857805+0000","last_clean":"2023-10-16T14:13:23.857805+0000","last_became_active":"2023-10-16T14:13:23.853022+0000","last_became_peered":"2023-10-16T14:13:23.853022+0000","last_unstale":"2023-10-16T14:13:23.857805+0000","last_undegraded":"2023-10-16T14:13:23.857805+0000","last_fullsized":"2023-10-16T14:13:23.857805+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:09.935977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.837038+0000","last_change":"2023-10-16T14:13:22.820743+0000","last_active":"2023-10-16T14:13:23.837038+0000","last_peered":"2023-10-16T14:13:23.837038+0000","last_clean":"2023-10-16T14:13:23.837038+0000","last_became_active":"2023-10-16T14:13:22.820466+0000","last_became_peered":"2023-10-16T14:13:22.820466+0000","last_unstale":"2023-10-16T14:13:23.837038+0000","last_undegraded":"2023-10-16T14:13:23.837038+0000","last_fullsized":"2023-10-16T14:13:23.837038+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:43.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855176+0000","last_change":"2023-10-16T14:13:23.850572+0000","last_active":"2023-10-16T14:13:23.855176+0000","last_peered":"2023-10-16T14:13:23.855176+0000","last_clean":"2023-10-16T14:13:23.855176+0000","last_became_active":"2023-10-16T14:13:23.849531+0000","last_became_peered":"2023-10-16T14:13:23.849531+0000","last_unstale":"2023-10-16T14:13:23.855176+0000","last_undegraded":"2023-10-16T14:13:23.855176+0000","last_fullsized":"2023-10-16T14:13:23.855176+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:18.870664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.846211+0000","last_change":"2023-10-16T14:13:22.819882+0000","last_active":"2023-10-16T14:13:23.846211+0000","last_peered":"2023-10-16T14:13:23.846211+0000","last_clean":"2023-10-16T14:13:23.846211+0000","last_became_active":"2023-10-16T14:13:22.819623+0000","last_became_peered":"2023-10-16T14:13:22.819623+0000","last_unstale":"2023-10-16T14:13:23.846211+0000","last_undegraded":"2023-10-16T14:13:23.846211+0000","last_fullsized":"2023-10-16T14:13:23.846211+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:20.044305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.819575+0000","last_change":"2023-10-16T14:13:23.840883+0000","last_active":"2023-10-16T14:13:24.819575+0000","last_peered":"2023-10-16T14:13:24.819575+0000","last_clean":"2023-10-16T14:13:24.819575+0000","last_became_active":"2023-10-16T14:13:23.840509+0000","last_became_peered":"2023-10-16T14:13:23.840509+0000","last_unstale":"2023-10-16T14:13:24.819575+0000","last_undegraded":"2023-10-16T14:13:24.819575+0000","last_fullsized":"2023-10-16T14:13:24.819575+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:55.312400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"286'2","reported_seq":13,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.910961+0000","last_change":"2023-10-16T14:13:22.811428+0000","last_active":"2023-10-16T14:13:23.910961+0000","last_peered":"2023-10-16T14:13:23.910961+0000","last_clean":"2023-10-16T14:13:23.910961+0000","last_became_active":"2023-10-16T14:13:22.811112+0000","last_became_peered":"2023-10-16T14:13:22.811112+0000","last_unstale":"2023-10-16T14:13:23.910961+0000","last_undegraded":"2023-10-16T14:13:23.910961+0000","last_fullsized":"2023-10-16T14:13:23.910961+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:37.312115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.823452+0000","last_change":"2023-10-16T14:13:23.855446+0000","last_active":"2023-10-16T14:13:24.823452+0000","last_peered":"2023-10-16T14:13:24.823452+0000","last_clean":"2023-10-16T14:13:24.823452+0000","last_became_active":"2023-10-16T14:13:23.854283+0000","last_became_peered":"2023-10-16T14:13:23.854283+0000","last_unstale":"2023-10-16T14:13:24.823452+0000","last_undegraded":"2023-10-16T14:13:24.823452+0000","last_fullsized":"2023-10-16T14:13:24.823452+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:06.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.845190+0000","last_change":"2023-10-16T14:13:22.818518+0000","last_active":"2023-10-16T14:13:23.845190+0000","last_peered":"2023-10-16T14:13:23.845190+0000","last_clean":"2023-10-16T14:13:23.845190+0000","last_became_active":"2023-10-16T14:13:22.818237+0000","last_became_peered":"2023-10-16T14:13:22.818237+0000","last_unstale":"2023-10-16T14:13:23.845190+0000","last_undegraded":"2023-10-16T14:13:23.845190+0000","last_fullsized":"2023-10-16T14:13:23.845190+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:29.308153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.851025+0000","last_change":"2023-10-16T14:13:23.849087+0000","last_active":"2023-10-16T14:13:23.851025+0000","last_peered":"2023-10-16T14:13:23.851025+0000","last_clean":"2023-10-16T14:13:23.851025+0000","last_became_active":"2023-10-16T14:13:23.848811+0000","last_became_peered":"2023-10-16T14:13:23.848811+0000","last_unstale":"2023-10-16T14:13:23.851025+0000","last_undegraded":"2023-10-16T14:13:23.851025+0000","last_fullsized":"2023-10-16T14:13:23.851025+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:05.789670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.843494+0000","last_change":"2023-10-16T14:13:22.825845+0000","last_active":"2023-10-16T14:13:23.843494+0000","last_peered":"2023-10-16T14:13:23.843494+0000","last_clean":"2023-10-16T14:13:23.843494+0000","last_became_active":"2023-10-16T14:13:22.825550+0000","last_became_peered":"2023-10-16T14:13:22.825550+0000","last_unstale":"2023-10-16T14:13:23.843494+0000","last_undegraded":"2023-10-16T14:13:23.843494+0000","last_fullsized":"2023-10-16T14:13:23.843494+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:24.340708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.822663+0000","last_change":"2023-10-16T14:13:23.839666+0000","last_active":"2023-10-16T14:13:24.822663+0000","last_peered":"2023-10-16T14:13:24.822663+0000","last_clean":"2023-10-16T14:13:24.822663+0000","last_became_active":"2023-10-16T14:13:23.839307+0000","last_became_peered":"2023-10-16T14:13:23.839307+0000","last_unstale":"2023-10-16T14:13:24.822663+0000","last_undegraded":"2023-10-16T14:13:24.822663+0000","last_fullsized":"2023-10-16T14:13:24.822663+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:01.499282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.842143+0000","last_change":"2023-10-16T14:13:22.822052+0000","last_active":"2023-10-16T14:13:23.842143+0000","last_peered":"2023-10-16T14:13:23.842143+0000","last_clean":"2023-10-16T14:13:23.842143+0000","last_became_active":"2023-10-16T14:13:22.821755+0000","last_became_peered":"2023-10-16T14:13:22.821755+0000","last_unstale":"2023-10-16T14:13:23.842143+0000","last_undegraded":"2023-10-16T14:13:23.842143+0000","last_fullsized":"2023-10-16T14:13:23.842143+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:15.887787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.857504+0000","last_change":"2023-10-16T14:13:23.853934+0000","last_active":"2023-10-16T14:13:23.857504+0000","last_peered":"2023-10-16T14:13:23.857504+0000","last_clean":"2023-10-16T14:13:23.857504+0000","last_became_active":"2023-10-16T14:13:23.852820+0000","last_became_peered":"2023-10-16T14:13:23.852820+0000","last_unstale":"2023-10-16T14:13:23.857504+0000","last_undegraded":"2023-10-16T14:13:23.857504+0000","last_fullsized":"2023-10-16T14:13:23.857504+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:36.021425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.841521+0000","last_change":"2023-10-16T14:13:22.822079+0000","last_active":"2023-10-16T14:13:23.841521+0000","last_peered":"2023-10-16T14:13:23.841521+0000","last_clean":"2023-10-16T14:13:23.841521+0000","last_became_active":"2023-10-16T14:13:22.821789+0000","last_became_peered":"2023-10-16T14:13:22.821789+0000","last_unstale":"2023-10-16T14:13:23.841521+0000","last_undegraded":"2023-10-16T14:13:23.841521+0000","last_fullsized":"2023-10-16T14:13:23.841521+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:56.741371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.859874+0000","last_change":"2023-10-16T14:13:23.853960+0000","last_active":"2023-10-16T14:13:23.859874+0000","last_peered":"2023-10-16T14:13:23.859874+0000","last_clean":"2023-10-16T14:13:23.859874+0000","last_became_active":"2023-10-16T14:13:23.853087+0000","last_became_peered":"2023-10-16T14:13:23.853087+0000","last_unstale":"2023-10-16T14:13:23.859874+0000","last_undegraded":"2023-10-16T14:13:23.859874+0000","last_fullsized":"2023-10-16T14:13:23.859874+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:42.880722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.842053+0000","last_change":"2023-10-16T14:13:22.823982+0000","last_active":"2023-10-16T14:13:23.842053+0000","last_peered":"2023-10-16T14:13:23.842053+0000","last_clean":"2023-10-16T14:13:23.842053+0000","last_became_active":"2023-10-16T14:13:22.823555+0000","last_became_peered":"2023-10-16T14:13:22.823555+0000","last_unstale":"2023-10-16T14:13:23.842053+0000","last_undegraded":"2023-10-16T14:13:23.842053+0000","last_fullsized":"2023-10-16T14:13:23.842053+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:37:05.425101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.855902+0000","last_change":"2023-10-16T14:13:23.853370+0000","last_active":"2023-10-16T14:13:23.855902+0000","last_peered":"2023-10-16T14:13:23.855902+0000","last_clean":"2023-10-16T14:13:23.855902+0000","last_became_active":"2023-10-16T14:13:23.852714+0000","last_became_peered":"2023-10-16T14:13:23.852714+0000","last_unstale":"2023-10-16T14:13:23.855902+0000","last_undegraded":"2023-10-16T14:13:23.855902+0000","last_fullsized":"2023-10-16T14:13:23.855902+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:18.804667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.827954+0000","last_change":"2023-10-16T14:13:22.809610+0000","last_active":"2023-10-16T14:13:23.827954+0000","last_peered":"2023-10-16T14:13:23.827954+0000","last_clean":"2023-10-16T14:13:23.827954+0000","last_became_active":"2023-10-16T14:13:22.809204+0000","last_became_peered":"2023-10-16T14:13:22.809204+0000","last_unstale":"2023-10-16T14:13:23.827954+0000","last_undegraded":"2023-10-16T14:13:23.827954+0000","last_fullsized":"2023-10-16T14:13:23.827954+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:40.668703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":287,"state":"active+clean","last_fresh":"2023-10-16T14:13:24.821744+0000","last_change":"2023-10-16T14:13:23.834888+0000","last_active":"2023-10-16T14:13:24.821744+0000","last_peered":"2023-10-16T14:13:24.821744+0000","last_clean":"2023-10-16T14:13:24.821744+0000","last_became_active":"2023-10-16T14:13:23.834444+0000","last_became_peered":"2023-10-16T14:13:23.834444+0000","last_unstale":"2023-10-16T14:13:24.821744+0000","last_undegraded":"2023-10-16T14:13:24.821744+0000","last_fullsized":"2023-10-16T14:13:24.821744+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:42.315996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.838995+0000","last_change":"2023-10-16T14:13:22.819988+0000","last_active":"2023-10-16T14:13:23.838995+0000","last_peered":"2023-10-16T14:13:23.838995+0000","last_clean":"2023-10-16T14:13:23.838995+0000","last_became_active":"2023-10-16T14:13:22.819675+0000","last_became_peered":"2023-10-16T14:13:22.819675+0000","last_unstale":"2023-10-16T14:13:23.838995+0000","last_undegraded":"2023-10-16T14:13:23.838995+0000","last_fullsized":"2023-10-16T14:13:23.838995+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:21.769901+0000","last_clean_scrub_stamp":"2023-10-16T14:13:21.769901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:02.851606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":9,"reported_epoch":286,"state":"active+clean","last_fresh":"2023-10-16T14:13:23.860140+0000","last_change":"2023-10-16T14:13:23.856456+0000","last_active":"2023-10-16T14:13:23.860140+0000","last_peered":"2023-10-16T14:13:23.860140+0000","last_clean":"2023-10-16T14:13:23.860140+0000","last_became_active":"2023-10-16T14:13:23.856104+0000","last_became_peered":"2023-10-16T14:13:23.856104+0000","last_unstale":"2023-10-16T14:13:23.860140+0000","last_undegraded":"2023-10-16T14:13:23.860140+0000","last_fullsized":"2023-10-16T14:13:23.860140+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T14:13:22.775764+0000","last_clean_scrub_stamp":"2023-10-16T14:13:22.775764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:01.935031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"275'2821","reported_seq":3920,"reported_epoch":285,"state":"active+clean","last_fresh":"2023-10-16T14:13:22.812309+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:13:22.812309+0000","last_peered":"2023-10-16T14:13:22.812309+0000","last_clean":"2023-10-16T14:13:22.812309+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:13:22.812309+0000","last_undegraded":"2023-10-16T14:13:22.812309+0000","last_fullsized":"2023-10-16T14:13:22.812309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2699,"num_read_kb":5660,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2699,"num_read_kb":5660,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2821,"ondisk_log_size":2821,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543330,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49180,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322660,"statfs":{"total":96636764160,"available":96586403840,"internally_reserved":0,"allocated":4202496,"data_stored":2365358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.3140000000000001,"15min":1.3480000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.641,"5min":1.8500000000000001,"15min":1.885},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.714,"5min":1.405,"15min":1.413},"min":{"1min":0.73699999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.675}]},{"osd":1,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.232,"15min":1.246},"min":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.228,"15min":1.248},"min":{"1min":0.76300000000000001,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.482}]},{"osd":2,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.252,"15min":1.268},"min":{"1min":0.85299999999999998,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.349,"15min":1.361},"min":{"1min":0.95199999999999996,"5min":0.745,"15min":0.745},"max":{"1min":3.4289999999999998,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.27}]},{"osd":3,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.357,"15min":1.405},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.097},"last":1.415},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4159999999999999,"15min":1.431},"min":{"1min":1.107,"5min":0.90100000000000002,"15min":0.86599999999999999},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.5920000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2589999999999999,"15min":1.242},"min":{"1min":0.94699999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.895,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4730000000000001,"15min":1.4159999999999999},"min":{"1min":0.98799999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.4450000000000001,"15min":1.395},"min":{"1min":1.026,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.546,"15min":1.538},"min":{"1min":1.0800000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.5309999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.6599999999999999,"15min":1.6379999999999999},"min":{"1min":1.21,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":3.29,"5min":3.29,"15min":3.29},"last":1.627},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5600000000000001,"15min":1.486},"min":{"1min":1.0029999999999999,"5min":0.75,"15min":0.75},"max":{"1min":3.524,"5min":3.524,"15min":3.524},"last":1.3779999999999999}]}]},{"osd":6,"up_from":14,"seq":60129543331,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48608,"kb_used_data":3536,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323232,"statfs":{"total":96636764160,"available":96586989568,"internally_reserved":0,"allocated":3620864,"data_stored":1776602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.244,"15min":1.3109999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.887,"15min":1.887},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.3149999999999999,"15min":1.345},"min":{"1min":0.84999999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.595,"5min":1.847,"15min":1.9390000000000001},"last":1.345}]},{"osd":1,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.175,"15min":1.212},"min":{"1min":0.80000000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6220000000000001,"5min":1.8360000000000001,"15min":2.008},"last":1.306},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.202},"min":{"1min":0.67900000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.286,"5min":1.706,"15min":1.77},"last":1.264}]},{"osd":2,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2330000000000001,"15min":1.2470000000000001},"min":{"1min":0.72899999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.5960000000000001,"5min":1.823,"15min":1.823},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2749999999999999,"15min":1.3120000000000001},"min":{"1min":0.98799999999999999,"5min":0.745,"15min":0.745},"max":{"1min":1.619,"5min":1.962,"15min":1.962},"last":1.131}]},{"osd":3,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2829999999999999,"15min":1.333},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4830000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3919999999999999,"15min":1.405},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.5629999999999999,"5min":1.911,"15min":1.911},"last":1.2170000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.389,"15min":1.403},"min":{"1min":0.77300000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.6479999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.456},{"interface":"front","average":{"1min":1.476,"5min":1.546,"15min":1.548},"min":{"1min":1.0680000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.1179999999999999},"last":1.4330000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.405,"15min":1.379},"min":{"1min":0.88300000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0840000000000001,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.034},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4710000000000001,"15min":1.486},"min":{"1min":0.91900000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.786,"5min":2.0990000000000002,"15min":2.1970000000000001},"last":0.99399999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:12:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.506,"15min":1.5009999999999999},"min":{"1min":0.96799999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0270000000000001,"5min":2.0960000000000001,"15min":2.2730000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.5289999999999999,"15min":1.536},"min":{"1min":0.998,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.9990000000000001,"5min":2.157,"15min":2.6120000000000001},"last":1.194}]}]},{"osd":5,"up_from":14,"seq":60129543332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48612,"kb_used_data":3540,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323228,"statfs":{"total":96636764160,"available":96586985472,"internally_reserved":0,"allocated":3624960,"data_stored":1776692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.111,"15min":1.1879999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.546,"5min":2.8330000000000002,"15min":2.8330000000000002},"last":1.369},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1919999999999999,"15min":1.22},"min":{"1min":0.79400000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.4119999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.286,"15min":1.306},"min":{"1min":1.034,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.631,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.171},{"interface":"front","average":{"1min":1.2,"5min":1.1839999999999999,"15min":1.248},"min":{"1min":0.80000000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.931,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.0429999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2909999999999999,"15min":1.333},"min":{"1min":0.82799999999999996,"5min":0.63,"15min":0.63},"max":{"1min":2.024,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.238,"15min":1.3009999999999999},"min":{"1min":0.82599999999999996,"5min":0.746,"15min":0.746},"max":{"1min":1.8600000000000001,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.482}]},{"osd":3,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3899999999999999,"15min":1.403},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.681,"5min":1.9590000000000001,"15min":2.0499999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":1.0049999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.827,"5min":2.75,"15min":2.75},"last":1.5289999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3160000000000001,"15min":1.298},"min":{"1min":0.86899999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6839999999999999,"5min":2.774,"15min":2.774},"last":1.254},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.4430000000000001},"min":{"1min":1.1879999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.865,"5min":2.0910000000000002,"15min":2.3330000000000002},"last":1.655}]},{"osd":6,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.464,"15min":1.4390000000000001},"min":{"1min":1.0289999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.9399999999999999,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.454},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.5009999999999999,"15min":1.4870000000000001},"min":{"1min":1.0900000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.9790000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":1.6200000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.5920000000000001,"15min":1.556},"min":{"1min":1.1040000000000001,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.153,"5min":2.1680000000000001,"15min":2.4609999999999999},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6220000000000001,"15min":1.6060000000000001},"min":{"1min":1.272,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.044,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.5880000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84636,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287204,"statfs":{"total":96636764160,"available":96550096896,"internally_reserved":0,"allocated":3616768,"data_stored":1775642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.151,"15min":1.1899999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.4510000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.52},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1759999999999999,"15min":1.1950000000000001},"min":{"1min":0.76100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.6100000000000001,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.2989999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.157,"15min":1.2370000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.673,"5min":1.71,"15min":1.78},"last":1.079},{"interface":"front","average":{"1min":1.236,"5min":1.232,"15min":1.28},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.792,"5min":1.792,"15min":2.0499999999999998},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2749999999999999,"15min":1.3140000000000001},"min":{"1min":0.86299999999999999,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.5169999999999999,"5min":1.714,"15min":1.865},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3520000000000001,"15min":1.379},"min":{"1min":1.006,"5min":0.87,"15min":0.87},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.77}]},{"osd":3,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.5980000000000001,"5min":1.8939999999999999,"15min":1.9650000000000001},"last":1.395},{"interface":"front","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.4279999999999999},"min":{"1min":0.93000000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.915,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.248}]},{"osd":5,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.357,"15min":1.3400000000000001},"min":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.74099999999999999},"max":{"1min":2.0670000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3899999999999999,"15min":1.349},"min":{"1min":0.89900000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.647}]},{"osd":6,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.417,"15min":1.3939999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.956},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.538,"15min":1.522},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":2.3279999999999998,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.8660000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:12:34 2023","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.516},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.645,"15min":1.639},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.1200000000000001,"5min":2.1280000000000001,"15min":2.3660000000000001},"last":1.5920000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543332,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56920,"kb_used_data":3528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314920,"statfs":{"total":96636764160,"available":96578478080,"internally_reserved":0,"allocated":3612672,"data_stored":1775572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.90700000000000003,"15min":0.94199999999999995},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":1.077,"5min":1.595,"15min":1.595},"last":1.151},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1499999999999999,"15min":1.204},"min":{"1min":0.47099999999999997,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.8100000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.2070000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.04,"15min":1.0669999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.524,"5min":1.6870000000000001,"15min":1.855},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.0980000000000001,"15min":1.1140000000000001},"min":{"1min":0.52600000000000002,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.4570000000000001,"5min":2.125,"15min":2.125},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1100000000000001,"15min":1.1839999999999999},"min":{"1min":0.42999999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3160000000000001,"15min":1.4119999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.8069999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.3300000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.458,"15min":1.4970000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.67000000000000004},"max":{"1min":1.8120000000000001,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.109},{"interface":"front","average":{"1min":1.345,"5min":1.4330000000000001,"15min":1.4530000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.925,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.413,"15min":1.482},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.69699999999999995},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.508,"15min":1.5449999999999999},"min":{"1min":1.024,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":2.0209999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.4490000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.4890000000000001,"15min":1.5129999999999999},"min":{"1min":0.85899999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0739999999999998,"5min":2.335,"15min":2.4329999999999998},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.4650000000000001,"15min":1.488},"min":{"1min":0.78100000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.1699999999999999,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":0.86199999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:12:50 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5680000000000001,"15min":1.589},"min":{"1min":0.98899999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1080000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.542},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.6479999999999999,"15min":1.6679999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.2610000000000001,"5min":2.569,"15min":2.6070000000000002},"last":1.3839999999999999}]}]},{"osd":2,"up_from":14,"seq":60129543330,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56928,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314912,"statfs":{"total":96636764160,"available":96578469888,"internally_reserved":0,"allocated":3616768,"data_stored":1775114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.93200000000000005,"15min":0.94999999999999996},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":1.2},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.47599999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.7090000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.0369999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5580000000000001,"5min":1.847,"15min":1.847},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.125,"15min":1.2210000000000001},"min":{"1min":0.67100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.415,"5min":1.784,"15min":1.9530000000000001},"last":1.415}]},{"osd":3,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.05,"15min":1.0980000000000001},"min":{"1min":0.56100000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3759999999999999,"5min":1.9179999999999999,"15min":2.1299999999999999},"last":1.264},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.252,"15min":1.3460000000000001},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.1480000000000001,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":1.4690000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.4890000000000001},"min":{"1min":0.78200000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.298,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.732},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.4790000000000001},"min":{"1min":0.88900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.659}]},{"osd":5,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.51,"15min":1.548},"min":{"1min":0.89000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.923},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.462,"15min":1.4650000000000001},"min":{"1min":0.80600000000000005,"5min":0.627,"15min":0.627},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.53}]},{"osd":6,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.524,"15min":1.52},"min":{"1min":0.91300000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.613},{"interface":"front","average":{"1min":1.524,"5min":1.5580000000000001,"15min":1.605},"min":{"1min":0.77900000000000003,"5min":0.752,"15min":0.752},"max":{"1min":2.52,"5min":2.52,"15min":2.5569999999999999},"last":2.1669999999999998}]},{"osd":7,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.641,"15min":1.6599999999999999},"min":{"1min":0.996,"5min":0.754,"15min":0.754},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":2.093},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.613,"15min":1.637},"min":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.8720000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543330,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56928,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314912,"statfs":{"total":96636764160,"available":96578469888,"internally_reserved":0,"allocated":3616768,"data_stored":1775046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":1.244},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.0609999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.113},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":0.93700000000000006}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":1.3129999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61664,"kb_used_data":4112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310176,"statfs":{"total":96636764160,"available":96573620224,"internally_reserved":0,"allocated":4210688,"data_stored":2365450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.90900000000000003,"15min":0.93300000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3440000000000001,"5min":1.704,"15min":1.891},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0049999999999999,"15min":1.1279999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.6539999999999999,"15min":1.9299999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.0669999999999999,"15min":1.157},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4359999999999999,"5min":1.6679999999999999,"15min":2.0569999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.123,"15min":1.202},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.657,"5min":1.893,"15min":1.893},"last":0.89500000000000002}]},{"osd":3,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0640000000000001,"15min":1.141},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7589999999999999,"5min":1.7869999999999999,"15min":1.8600000000000001},"last":0.749},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.2270000000000001,"15min":1.351},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.603,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.1319999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.26,"15min":1.3500000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.923,"5min":1.923,"15min":2.0070000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.413,"15min":1.5009999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.98,"5min":2.1070000000000002,"15min":2.601},"last":1.3859999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.4430000000000001,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9359999999999999,"5min":2.089,"15min":2.3279999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.385,"15min":1.468},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.901,"5min":2.1259999999999999,"15min":2.2869999999999999},"last":1.0349999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.5229999999999999,"15min":1.5960000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.669},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5109999999999999,"15min":1.579},"min":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0129999999999999,"5min":2.327,"15min":2.3769999999999998},"last":1.262}]},{"osd":7,"last update":"Mon Oct 16 14:12:41 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.6080000000000001,"15min":1.6879999999999999},"min":{"1min":0.76700000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0670000000000002,"5min":2.2970000000000002,"15min":2.5270000000000001},"last":1.369},{"interface":"front","average":{"1min":1.482,"5min":1.5509999999999999,"15min":1.6259999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0270000000000001,"5min":2.3679999999999999,"15min":2.464},"last":1.1839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:13:28.947 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T14:13:28.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T14:13:28.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:13:29.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:13:29.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.214+0000 7f9a6b02e640 1 Processor -- start 2023-10-16T14:13:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.214+0000 7f9a6b02e640 1 -- start start 2023-10-16T14:13:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6b02e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a640acfa0 0x7f9a64071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6b02e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64070760 con 0x7f9a6415dc90 2023-10-16T14:13:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6b02e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640cdac0 con 0x7f9a64070bf0 2023-10-16T14:13:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6b02e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640715c0 con 0x7f9a640acfa0 2023-10-16T14:13:29.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6a02c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9a6415dc90 0x7f9a640c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51102/0 (socket says 172.21.15.60:51102) 2023-10-16T14:13:29.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6a02c640 1 -- 172.21.15.60:0/3909978865 learned_addr learned my addr 172.21.15.60:0/3909978865 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:29.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.216+0000 7f9a6a82d640 1 --2- 172.21.15.60:0/3909978865 >> v2:172.21.15.60:3300/0 conn(0x7f9a640acfa0 0x7f9a64071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.217+0000 7f9a6902a640 1 -- 172.21.15.60:0/3909978865 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155725866 0 0) 0x7f9a64070760 con 0x7f9a6415dc90 2023-10-16T14:13:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.217+0000 7f9a6902a640 1 -- 172.21.15.60:0/3909978865 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a50003520 con 0x7f9a6415dc90 2023-10-16T14:13:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.217+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3909978865 >> v1:172.21.15.144:6789/0 conn(0x7f9a64070bf0 legacy=0x7f9a640ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.217+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3909978865 >> v1:172.21.15.60:6789/0 conn(0x7f9a6415dc90 legacy=0x7f9a640c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.217+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3909978865 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64070110 con 0x7f9a640acfa0 2023-10-16T14:13:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.218+0000 7f9a6a82d640 1 --2- 172.21.15.60:0/3909978865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a640acfa0 0x7f9a64071080 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009d90 tx=0x7f9a5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10118d35a087c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.218+0000 7f9a6902a640 1 -- 172.21.15.60:0/3909978865 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c02cc60 con 0x7f9a640acfa0 2023-10-16T14:13:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.218+0000 7f9a6902a640 1 -- 172.21.15.60:0/3909978865 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c02cdc0 con 0x7f9a640acfa0 2023-10-16T14:13:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.218+0000 7f9a6902a640 1 -- 172.21.15.60:0/3909978865 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c034660 con 0x7f9a640acfa0 2023-10-16T14:13:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3909978865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a640acfa0 msgr2=0x7f9a64071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3909978865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a640acfa0 0x7f9a64071080 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009d90 tx=0x7f9a5c02beb0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3909978865 shutdown_connections 2023-10-16T14:13:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3909978865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a640acfa0 0x7f9a64071080 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3909978865 >> 172.21.15.60:0/3909978865 conn(0x7f9a640ce6a0 msgr2=0x7f9a640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3909978865 shutdown_connections 2023-10-16T14:13:29.225 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.219+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3909978865 wait complete. 2023-10-16T14:13:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.222+0000 7f9a6b02e640 1 Processor -- start 2023-10-16T14:13:29.226 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.222+0000 7f9a6b02e640 1 -- start start 2023-10-16T14:13:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.223+0000 7f9a6b02e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a64070bf0 0x7f9a64156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.223+0000 7f9a6b02e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a640acfa0 0x7f9a640b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.223+0000 7f9a6b02e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 0x7f9a6415e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.223+0000 7f9a6b02e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641f2450 con 0x7f9a6415dc90 2023-10-16T14:13:29.228 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.223+0000 7f9a6b02e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640b4220 con 0x7f9a64070bf0 2023-10-16T14:13:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.223+0000 7f9a6b02e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641f5c50 con 0x7f9a640acfa0 2023-10-16T14:13:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 0x7f9a6415e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 0x7f9a6415e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33632/0 (socket says 172.21.15.60:33632) 2023-10-16T14:13:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3811478690 learned_addr learned my addr 172.21.15.60:0/3811478690 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a02c640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a64070bf0 0x7f9a64156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a640acfa0 msgr2=0x7f9a640b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a640acfa0 0x7f9a640b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a64070bf0 msgr2=0x7f9a64156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a64070bf0 0x7f9a64156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.230 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a82d640 1 -- 172.21.15.60:0/3811478690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c009a70 con 0x7f9a6415dc90 2023-10-16T14:13:29.231 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.224+0000 7f9a6a02c640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a64070bf0 0x7f9a64156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:29.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.225+0000 7f9a6a82d640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 0x7f9a6415e070 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c037040 tx=0x7f9a5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:29.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.225+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c040650 con 0x7f9a6415dc90 2023-10-16T14:13:29.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.225+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c0407b0 con 0x7f9a6415dc90 2023-10-16T14:13:29.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.225+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a641f5f30 con 0x7f9a6415dc90 2023-10-16T14:13:29.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.225+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c040b60 con 0x7f9a6415dc90 2023-10-16T14:13:29.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.225+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a641f64f0 con 0x7f9a6415dc90 2023-10-16T14:13:29.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.226+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6404e9b0 con 0x7f9a6415dc90 2023-10-16T14:13:29.233 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.228+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a5c040cc0 con 0x7f9a6415dc90 2023-10-16T14:13:29.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.229+0000 7f9a5affd640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f9a4806afc0 0x7f9a4806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.229+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a5c0ac500 con 0x7f9a6415dc90 2023-10-16T14:13:29.234 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.230+0000 7f9a6a02c640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f9a4806afc0 0x7f9a4806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.235 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.231+0000 7f9a6a02c640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f9a4806afc0 0x7f9a4806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a600011e0 tx=0x7f9a60009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:29.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.234+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a5c077550 con 0x7f9a6415dc90 2023-10-16T14:13:29.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.501+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9a640ae940 con 0x7f9a6415dc90 2023-10-16T14:13:29.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.505+0000 7f9a5affd640 1 -- 172.21.15.60:0/3811478690 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v61) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9a5c009450 con 0x7f9a6415dc90 2023-10-16T14:13:29.509 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2023-10-16T14:13:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBZLS1lTyw3CRAA0C27KAzFtMJO3vg8eF4Pnw== 2023-10-16T14:13:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2023-10-16T14:13:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2023-10-16T14:13:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T14:13:29.510 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2023-10-16T14:13:29.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f9a4806afc0 msgr2=0x7f9a4806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f9a4806afc0 0x7f9a4806d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a600011e0 tx=0x7f9a60009040 comp rx=0 tx=0).stop 2023-10-16T14:13:29.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 msgr2=0x7f9a6415e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.512 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 0x7f9a6415e070 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c037040 tx=0x7f9a5c004a40 comp rx=0 tx=0).stop 2023-10-16T14:13:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 shutdown_connections 2023-10-16T14:13:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f9a4806afc0 0x7f9a4806d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a640acfa0 0x7f9a640b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6415dc90 0x7f9a6415e070 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 --2- 172.21.15.60:0/3811478690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a64070bf0 0x7f9a64156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 >> 172.21.15.60:0/3811478690 conn(0x7f9a640ce6a0 msgr2=0x7f9a640af740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:29.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 shutdown_connections 2023-10-16T14:13:29.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.508+0000 7f9a6b02e640 1 -- 172.21.15.60:0/3811478690 wait complete. 2023-10-16T14:13:29.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:29.751 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.748+0000 7efef592e640 1 Processor -- start 2023-10-16T14:13:29.752 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.748+0000 7efef592e640 1 -- start start 2023-10-16T14:13:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.750+0000 7efef592e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.750+0000 7efef592e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.750+0000 7efef592e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef006c960 con 0x7efef015ebc0 2023-10-16T14:13:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.750+0000 7efef592e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef015b9d0 con 0x7efef00ad970 2023-10-16T14:13:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.750+0000 7efef592e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef015bcb0 con 0x7efef006c040 2023-10-16T14:13:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.750+0000 7efef492c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efeeffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef492c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:49620/0 (socket says 172.21.15.60:49620) 2023-10-16T14:13:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef492c640 1 -- 172.21.15.60:0/3217860 learned_addr learned my addr 172.21.15.60:0/3217860 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef512d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7efef015ebc0 0x7efef00cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:50858/0 (socket says 172.21.15.60:50858) 2023-10-16T14:13:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef492c640 1 -- 172.21.15.60:0/3217860 >> v2:172.21.15.144:3300/0 conn(0x7efef00ad970 msgr2=0x7efef015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef492c640 1 --2- 172.21.15.60:0/3217860 >> v2:172.21.15.144:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef492c640 1 -- 172.21.15.60:0/3217860 >> v1:172.21.15.144:6789/0 conn(0x7efef015ebc0 legacy=0x7efef00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efeef7fe640 1 -- 172.21.15.60:0/3217860 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481955010 0 0) 0x7efef006c960 con 0x7efef015ebc0 2023-10-16T14:13:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.751+0000 7efef492c640 1 -- 172.21.15.60:0/3217860 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef015bf90 con 0x7efef006c040 2023-10-16T14:13:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.753+0000 7efef492c640 1 --2- 172.21.15.60:0/3217860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 0x7efef006c420 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7efee40098a0 tx=0x7efee402b970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc5fac10a8dfe3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.753+0000 7efeef7fe640 1 -- 172.21.15.60:0/3217860 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4030830 con 0x7efef006c040 2023-10-16T14:13:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.753+0000 7efeef7fe640 1 -- 172.21.15.60:0/3217860 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee4030990 con 0x7efef006c040 2023-10-16T14:13:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.753+0000 7efeef7fe640 1 -- 172.21.15.60:0/3217860 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4030d20 con 0x7efef006c040 2023-10-16T14:13:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 -- 172.21.15.60:0/3217860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 msgr2=0x7efef006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 --2- 172.21.15.60:0/3217860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 0x7efef006c420 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7efee40098a0 tx=0x7efee402b970 comp rx=0 tx=0).stop 2023-10-16T14:13:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 -- 172.21.15.60:0/3217860 shutdown_connections 2023-10-16T14:13:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 --2- 172.21.15.60:0/3217860 >> v2:172.21.15.144:3300/0 conn(0x7efef00ad970 0x7efef015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 --2- 172.21.15.60:0/3217860 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef006c040 0x7efef006c420 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 -- 172.21.15.60:0/3217860 >> 172.21.15.60:0/3217860 conn(0x7efef00ce6a0 msgr2=0x7efef00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 -- 172.21.15.60:0/3217860 shutdown_connections 2023-10-16T14:13:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.754+0000 7efef592e640 1 -- 172.21.15.60:0/3217860 wait complete. 2023-10-16T14:13:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.756+0000 7efef592e640 1 Processor -- start 2023-10-16T14:13:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.756+0000 7efef592e640 1 -- start start 2023-10-16T14:13:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.757+0000 7efef592e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efef006c040 0x7efef0146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.757+0000 7efef592e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef00ad970 0x7efef0149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.757+0000 7efef592e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 0x7efef015d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.757+0000 7efef592e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef014a310 con 0x7efef015ebc0 2023-10-16T14:13:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.757+0000 7efef592e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef0147000 con 0x7efef006c040 2023-10-16T14:13:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.757+0000 7efef592e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef0149d50 con 0x7efef00ad970 2023-10-16T14:13:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 0x7efef015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 0x7efef015d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33634/0 (socket says 172.21.15.60:33634) 2023-10-16T14:13:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 -- 172.21.15.60:0/829944857 learned_addr learned my addr 172.21.15.60:0/829944857 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efeeffff640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef00ad970 0x7efef0149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef492c640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efef006c040 0x7efef0146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 -- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef00ad970 msgr2=0x7efef0149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef00ad970 0x7efef0149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 -- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efef006c040 msgr2=0x7efef0146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efef006c040 0x7efef0146ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efef512d640 1 -- 172.21.15.60:0/829944857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40094d0 con 0x7efef015ebc0 2023-10-16T14:13:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.758+0000 7efeeffff640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef00ad970 0x7efef0149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:29.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.759+0000 7efef512d640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 0x7efef015d330 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7efee800c7e0 tx=0x7efee800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:29.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.759+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee800a640 con 0x7efef015ebc0 2023-10-16T14:13:29.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.759+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee800a7c0 con 0x7efef015ebc0 2023-10-16T14:13:29.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.759+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef014a030 con 0x7efef015ebc0 2023-10-16T14:13:29.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.759+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee800ab70 con 0x7efef015ebc0 2023-10-16T14:13:29.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.759+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0147760 con 0x7efef015ebc0 2023-10-16T14:13:29.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.760+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef006b330 con 0x7efef015ebc0 2023-10-16T14:13:29.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.761+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efee800acd0 con 0x7efef015ebc0 2023-10-16T14:13:29.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.762+0000 7efeed7fa640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7efed406af90 0x7efed406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:29.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.762+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efee8088dc0 con 0x7efef015ebc0 2023-10-16T14:13:29.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.763+0000 7efef492c640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7efed406af90 0x7efed406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:29.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.764+0000 7efef492c640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7efed406af90 0x7efed406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efee4036000 tx=0x7efee403b040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:29.767+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efee8053d50 con 0x7efef015ebc0 2023-10-16T14:13:30.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.000+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef01472e0 con 0x7efef015ebc0 2023-10-16T14:13:30.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.001+0000 7efeed7fa640 1 -- 172.21.15.60:0/829944857 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4929 (secure 0 0 0) 0x7efef01472e0 con 0x7efef015ebc0 2023-10-16T14:13:30.005 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:30.005 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:13:23.271856+0000","modified":"2023-10-16T14:13:26.430490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13065":{"gid":13065,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2549968691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2549968691},{"type":"v1","addr":"172.21.15.60:6835","nonce":2549968691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T14:13:30.006 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 152 2023-10-16T14:13:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7efed406af90 msgr2=0x7efed406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7efed406af90 0x7efed406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efee4036000 tx=0x7efee403b040 comp rx=0 tx=0).stop 2023-10-16T14:13:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 msgr2=0x7efef015d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 0x7efef015d330 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7efee800c7e0 tx=0x7efee800cca0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 shutdown_connections 2023-10-16T14:13:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7efed406af90 0x7efed406d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7efef00ad970 0x7efef0149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efef015ebc0 0x7efef015d330 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 --2- 172.21.15.60:0/829944857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efef006c040 0x7efef0146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 >> 172.21.15.60:0/829944857 conn(0x7efef00ce6a0 msgr2=0x7efef00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.004+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 shutdown_connections 2023-10-16T14:13:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.005+0000 7efef592e640 1 -- 172.21.15.60:0/829944857 wait complete. 2023-10-16T14:13:30.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 152, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T14:13:23.271856+0000', 'modified': '2023-10-16T14:13:26.430490+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13065}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13065': {'gid': 13065, 'name': 'a', 'rank': 0, 'incarnation': 148, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/2549968691', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 2549968691}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2549968691}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T14:13:30.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T14:13:30.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T14:13:30.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.666+0000 7f57c759e640 1 Processor -- start 2023-10-16T14:13:30.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.666+0000 7f57c759e640 1 -- start start 2023-10-16T14:13:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57c759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57c80ad970 0x7f57c80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57c759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c815b9d0 con 0x7f57c80ccfb0 2023-10-16T14:13:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57c759e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c815bcb0 con 0x7f57c806c040 2023-10-16T14:13:30.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57c759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c815bf90 con 0x7f57c80ad970 2023-10-16T14:13:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57bdd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57c80ad970 0x7f57c80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57bdd9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57c80ad970 0x7f57c80cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33638/0 (socket says 172.21.15.60:33638) 2023-10-16T14:13:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4105195019 learned_addr learned my addr 172.21.15.60:0/4105195019 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.667+0000 7f57c6d9d640 1 --1- 172.21.15.60:0/4105195019 >> v1:172.21.15.60:6789/0 conn(0x7f57c80ccfb0 0x7f57c80cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51106/0 (socket says 172.21.15.60:51106) 2023-10-16T14:13:30.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.668+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4105195019 >> v1:172.21.15.144:6789/0 conn(0x7f57c806c040 legacy=0x7f57c806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.668+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4105195019 >> v1:172.21.15.60:6789/0 conn(0x7f57c80ccfb0 legacy=0x7f57c80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.668+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4105195019 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c80cc270 con 0x7f57c80ad970 2023-10-16T14:13:30.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.669+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/4105195019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c80ad970 0x7f57c80cca70 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f57b00098a0 tx=0x7f57b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f86e6f1c551b62 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.669+0000 7f57c5d9b640 1 -- 172.21.15.60:0/4105195019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0030840 con 0x7f57c80ad970 2023-10-16T14:13:30.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.669+0000 7f57c5d9b640 1 -- 172.21.15.60:0/4105195019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b00309a0 con 0x7f57c80ad970 2023-10-16T14:13:30.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.669+0000 7f57c5d9b640 1 -- 172.21.15.60:0/4105195019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0030d30 con 0x7f57c80ad970 2023-10-16T14:13:30.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 -- 172.21.15.60:0/4105195019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c80ad970 msgr2=0x7f57c80cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 --2- 172.21.15.60:0/4105195019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c80ad970 0x7f57c80cca70 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f57b00098a0 tx=0x7f57b002b980 comp rx=0 tx=0).stop 2023-10-16T14:13:30.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 -- 172.21.15.60:0/4105195019 shutdown_connections 2023-10-16T14:13:30.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 --2- 172.21.15.60:0/4105195019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c80ad970 0x7f57c80cca70 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 -- 172.21.15.60:0/4105195019 >> 172.21.15.60:0/4105195019 conn(0x7f57c80ce6a0 msgr2=0x7f57c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 -- 172.21.15.60:0/4105195019 shutdown_connections 2023-10-16T14:13:30.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.670+0000 7f57c759e640 1 -- 172.21.15.60:0/4105195019 wait complete. 2023-10-16T14:13:30.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.672+0000 7f57c759e640 1 Processor -- start 2023-10-16T14:13:30.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.672+0000 7f57c759e640 1 -- start start 2023-10-16T14:13:30.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 0x7f57c80b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57c80ad970 0x7f57c80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57c80ccfb0 0x7f57c815aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c80b24b0 con 0x7f57c806c040 2023-10-16T14:13:30.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c80b2750 con 0x7f57c80ccfb0 2023-10-16T14:13:30.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c80b2a30 con 0x7f57c80ad970 2023-10-16T14:13:30.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 0x7f57c80b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 0x7f57c80b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33654/0 (socket says 172.21.15.60:33654) 2023-10-16T14:13:30.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.673+0000 7f57c659c640 1 -- 172.21.15.60:0/963803201 learned_addr learned my addr 172.21.15.60:0/963803201 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57c80ad970 0x7f57c80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c6d9d640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57c80ccfb0 0x7f57c815aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c659c640 1 -- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57c80ad970 msgr2=0x7f57c80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c659c640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57c80ad970 0x7f57c80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c659c640 1 -- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57c80ccfb0 msgr2=0x7f57c815aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c659c640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57c80ccfb0 0x7f57c815aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c659c640 1 -- 172.21.15.60:0/963803201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b00094d0 con 0x7f57c806c040 2023-10-16T14:13:30.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c6d9d640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57c80ccfb0 0x7f57c815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:30.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.674+0000 7f57c659c640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 0x7f57c80b0850 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f57b800ba50 tx=0x7f57b800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.675+0000 7f57bf7fe640 1 -- 172.21.15.60:0/963803201 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b800cbe0 con 0x7f57c806c040 2023-10-16T14:13:30.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.675+0000 7f57bf7fe640 1 -- 172.21.15.60:0/963803201 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b800cd80 con 0x7f57c806c040 2023-10-16T14:13:30.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.675+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c80b2d10 con 0x7f57c806c040 2023-10-16T14:13:30.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.675+0000 7f57bf7fe640 1 -- 172.21.15.60:0/963803201 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8014680 con 0x7f57c806c040 2023-10-16T14:13:30.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.675+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c80b0d90 con 0x7f57c806c040 2023-10-16T14:13:30.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.678+0000 7f57bf7fe640 1 -- 172.21.15.60:0/963803201 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f57b80147e0 con 0x7f57c806c040 2023-10-16T14:13:30.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.679+0000 7f57bf7fe640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f57a806af90 0x7f57a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.679+0000 7f57bf7fe640 1 -- 172.21.15.60:0/963803201 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57b8089790 con 0x7f57c806c040 2023-10-16T14:13:30.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.679+0000 7f57cf79c740 1 Processor -- start 2023-10-16T14:13:30.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 -- start start 2023-10-16T14:13:30.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f57a806af90 0x7f57a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6170 0x55a9fd089070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0b8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0895b0 0x55a9fd0b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcfa97a0 con 0x55a9fd0b6550 2023-10-16T14:13:30.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcffb740 con 0x55a9fd0b6170 2023-10-16T14:13:30.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.680+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcff8e30 con 0x55a9fd0895b0 2023-10-16T14:13:30.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0b8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0b8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33666/0 (socket says 172.21.15.60:33666) 2023-10-16T14:13:30.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4211780750 learned_addr learned my addr 172.21.15.60:0/4211780750 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57c659c640 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6170 0x55a9fd089070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57c6d9d640 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0895b0 0x55a9fd0b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0895b0 msgr2=0x55a9fd0b9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0895b0 0x55a9fd0b9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6170 msgr2=0x55a9fd089070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6170 0x55a9fd089070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.681+0000 7f57bdd9b640 1 -- 172.21.15.60:0/4211780750 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9fcfb8910 con 0x55a9fd0b6550 2023-10-16T14:13:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.682+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f57a806af90 0x7f57a806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57b0036000 tx=0x7f57b0008030 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.682+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0b8f00 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5920237b35b1a38 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.682+0000 7f57a7fff640 1 -- 172.21.15.60:0/4211780750 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0039ea0 con 0x55a9fd0b6550 2023-10-16T14:13:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.682+0000 7f57a7fff640 1 -- 172.21.15.60:0/4211780750 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b0044450 con 0x55a9fd0b6550 2023-10-16T14:13:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.682+0000 7f57a7fff640 1 -- 172.21.15.60:0/4211780750 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00447c0 con 0x55a9fd0b6550 2023-10-16T14:13:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 -- 172.21.15.60:0/4211780750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 msgr2=0x55a9fd0b8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0b8f00 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0046000 comp rx=0 tx=0).stop 2023-10-16T14:13:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 -- 172.21.15.60:0/4211780750 shutdown_connections 2023-10-16T14:13:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0895b0 0x55a9fd0b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0b8f00 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 --2- 172.21.15.60:0/4211780750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6170 0x55a9fd089070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 -- 172.21.15.60:0/4211780750 >> 172.21.15.60:0/4211780750 conn(0x55a9fd08b380 msgr2=0x55a9fd0d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 -- 172.21.15.60:0/4211780750 shutdown_connections 2023-10-16T14:13:30.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.683+0000 7f57cf79c740 1 -- 172.21.15.60:0/4211780750 wait complete. 2023-10-16T14:13:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.684+0000 7f57cf79c740 1 -- start start 2023-10-16T14:13:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.684+0000 7f57cf79c740 1 Processor -- start 2023-10-16T14:13:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd0baf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 0x55a9fd0bb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcff4720 con 0x55a9fd0cde80 2023-10-16T14:13:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcf92970 con 0x55a9fd0b6550 2023-10-16T14:13:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcf92610 con 0x55a9fd0b6170 2023-10-16T14:13:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57cf79c740 10 client.0 _ll_register_callbacks cb 0x55a9fcfb49d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:13:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57c6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 0x55a9fd0bb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57c6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 0x55a9fd0bb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33672/0 (socket says 172.21.15.60:33672) 2023-10-16T14:13:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57c6d9d640 1 -- 172.21.15.60:0/2806611136 learned_addr learned my addr 172.21.15.60:0/2806611136 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.685+0000 7f57c659c640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd0baf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57c6d9d640 1 -- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 msgr2=0x55a9fd0ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57c6d9d640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57c6d9d640 1 -- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 msgr2=0x55a9fd0baf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57c6d9d640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd0baf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57c6d9d640 1 -- 172.21.15.60:0/2806611136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c80cc270 con 0x55a9fd0cde80 2023-10-16T14:13:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd0baf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57c6d9d640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 0x55a9fd0bb480 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f57c00076a0 tx=0x7f57c0001560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866ac367fcdd097c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:13:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57a57fa640 1 -- 172.21.15.60:0/2806611136 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c00133f0 con 0x55a9fd0cde80 2023-10-16T14:13:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57a57fa640 1 -- 172.21.15.60:0/2806611136 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57c0013550 con 0x55a9fd0cde80 2023-10-16T14:13:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.686+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9fd01b160 con 0x55a9fd0cde80 2023-10-16T14:13:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57a57fa640 1 -- 172.21.15.60:0/2806611136 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57c0013710 con 0x55a9fd0cde80 2023-10-16T14:13:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 10 client.13197 fetch_fsmap learned FSMap version 152 2023-10-16T14:13:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=2+}) v3 -- 0x55a9fcff2da0 con 0x55a9fd0cde80 2023-10-16T14:13:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57a57fa640 1 -- 172.21.15.60:0/2806611136 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a9fd01b160 con 0x55a9fd0cde80 2023-10-16T14:13:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57a57fa640 1 -- 172.21.15.60:0/2806611136 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 152) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f57c0012070 con 0x55a9fd0cde80 2023-10-16T14:13:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 10 client.13197 fetch_fsmap finished waiting for FSMap version 152 2023-10-16T14:13:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 10 client.13197 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:13:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 20 client.13197 populate_metadata read hostname 'smithi060' 2023-10-16T14:13:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd0ed4e0 0x55a9fd0f1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 4 client.13197 mds_command: new command op to 13065 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T14:13:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 --> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9fcfc6cc0 con 0x55a9fd0ed4e0 2023-10-16T14:13:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.687+0000 7f57c659c640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd0ed4e0 0x55a9fd0f1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.688+0000 7f57c659c640 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd0ed4e0 0x55a9fd0f1220 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3051e602e060babb server_cookie=15bc0a0a817e9e7b in_seq=0 out_seq=0 2023-10-16T14:13:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.688+0000 7f57a57fa640 10 client.13197 ms_handle_connect on v2:172.21.15.60:6834/2549968691 2023-10-16T14:13:30.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57a57fa640 1 -- 172.21.15.60:0/2806611136 <== mds.0 v2:172.21.15.60:6834/2549968691 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a9fcfc6cc0 con 0x55a9fd0ed4e0 2023-10-16T14:13:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57a57fa640 10 client.13197 handle_command_reply: tid=0 2023-10-16T14:13:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 client.13197 shutdown 2023-10-16T14:13:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 msgr2=0x55a9fd0bb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 0x55a9fd0bb480 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f57c00076a0 tx=0x7f57c0001560 comp rx=0 tx=0).stop 2023-10-16T14:13:30.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 shutdown_connections 2023-10-16T14:13:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd0ed4e0 0x55a9fd0f1220 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0cde80 0x55a9fd0bb480 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd0baf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 --2- 172.21.15.60:0/2806611136 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57a57fa640 0 client.13197 ms_handle_reset on v2:172.21.15.60:6834/2549968691 2023-10-16T14:13:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 >> 172.21.15.60:0/2806611136 conn(0x55a9fd0895b0 msgr2=0x55a9fd08d130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 shutdown_connections 2023-10-16T14:13:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 1 -- 172.21.15.60:0/2806611136 wait complete. 2023-10-16T14:13:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.689+0000 7f57cf79c740 20 client.13197 trim_cache size 0 max 16384 2023-10-16T14:13:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.706+0000 7f57cf79c740 1 Processor -- start 2023-10-16T14:13:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.706+0000 7f57cf79c740 1 -- start start 2023-10-16T14:13:30.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.707+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.707+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.711 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.707+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0cde80 0x55a9fd0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.707+0000 7f57cf79c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fcd58f20 con 0x55a9fd0b6550 2023-10-16T14:13:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.707+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fd089b60 con 0x55a9fd0cde80 2023-10-16T14:13:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.707+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9fd088980 con 0x55a9fd0b6170 2023-10-16T14:13:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0a9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33678/0 (socket says 172.21.15.60:33678) 2023-10-16T14:13:30.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 -- 172.21.15.60:0/1296571308 learned_addr learned my addr 172.21.15.60:0/1296571308 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 -- 172.21.15.60:0/1296571308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 msgr2=0x55a9fd0e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 -- 172.21.15.60:0/1296571308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0cde80 msgr2=0x55a9fd0aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0cde80 0x55a9fd0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 -- 172.21.15.60:0/1296571308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9fcf91b80 con 0x55a9fd0b6550 2023-10-16T14:13:30.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.708+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0a9f70 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f57b00098a0 tx=0x7f57b0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3036193ba12d10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57bcd99640 1 -- 172.21.15.60:0/1296571308 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0039ea0 con 0x55a9fd0b6550 2023-10-16T14:13:30.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57bcd99640 1 -- 172.21.15.60:0/1296571308 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b00447c0 con 0x55a9fd0b6550 2023-10-16T14:13:30.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57bcd99640 1 -- 172.21.15.60:0/1296571308 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0044b30 con 0x55a9fd0b6550 2023-10-16T14:13:30.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 -- 172.21.15.60:0/1296571308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 msgr2=0x55a9fd0a9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0a9f70 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f57b00098a0 tx=0x7f57b0009240 comp rx=0 tx=0).stop 2023-10-16T14:13:30.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 -- 172.21.15.60:0/1296571308 shutdown_connections 2023-10-16T14:13:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0cde80 0x55a9fd0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6550 0x55a9fd0a9f70 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 --2- 172.21.15.60:0/1296571308 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0b6170 0x55a9fd0e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 -- 172.21.15.60:0/1296571308 >> 172.21.15.60:0/1296571308 conn(0x55a9fd0895b0 msgr2=0x55a9fd090130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 -- 172.21.15.60:0/1296571308 shutdown_connections 2023-10-16T14:13:30.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.709+0000 7f57cf79c740 1 -- 172.21.15.60:0/1296571308 wait complete. 2023-10-16T14:13:30.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.710+0000 7f57cf79c740 1 -- start start 2023-10-16T14:13:30.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.710+0000 7f57cf79c740 1 Processor -- start 2023-10-16T14:13:30.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 0x55a9fd0d76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd136370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0cde80 0x55a9fd0a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fd1368b0 con 0x55a9fd0b6170 2023-10-16T14:13:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9fd13c360 con 0x55a9fd0b6550 2023-10-16T14:13:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9fd13db10 con 0x55a9fd0cde80 2023-10-16T14:13:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.711+0000 7f57cf79c740 10 client.0 _ll_register_callbacks cb 0x7f57c0007e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T14:13:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 0x55a9fd0d76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 0x55a9fd0d76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33690/0 (socket says 172.21.15.60:33690) 2023-10-16T14:13:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 -- 172.21.15.60:0/374886952 learned_addr learned my addr 172.21.15.60:0/374886952 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 -- 172.21.15.60:0/374886952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0cde80 msgr2=0x55a9fd0a9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0cde80 0x55a9fd0a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 -- 172.21.15.60:0/374886952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 msgr2=0x55a9fd136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd136370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 -- 172.21.15.60:0/374886952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a9fcfb8910 con 0x55a9fd0b6170 2023-10-16T14:13:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57c659c640 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 0x55a9fd0d76a0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f57b8058310 tx=0x7f57b80564c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c879bdda3f60717 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2023-10-16T14:13:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57a57fa640 1 -- 172.21.15.60:0/374886952 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8087c10 con 0x55a9fd0b6170 2023-10-16T14:13:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.712+0000 7f57a57fa640 1 -- 172.21.15.60:0/374886952 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57b8087d70 con 0x55a9fd0b6170 2023-10-16T14:13:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57b0044450 con 0x55a9fd0b6170 2023-10-16T14:13:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57a57fa640 1 -- 172.21.15.60:0/374886952 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57b8085030 con 0x55a9fd0b6170 2023-10-16T14:13:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 10 client.13203 fetch_fsmap learned FSMap version 152 2023-10-16T14:13:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=2+}) v3 -- 0x55a9fcfb7170 con 0x55a9fd0b6170 2023-10-16T14:13:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57a57fa640 1 -- 172.21.15.60:0/374886952 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b807a080 con 0x55a9fd0b6170 2023-10-16T14:13:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57a57fa640 1 -- 172.21.15.60:0/374886952 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 152) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f57b8057870 con 0x55a9fd0b6170 2023-10-16T14:13:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 10 client.13203 fetch_fsmap finished waiting for FSMap version 152 2023-10-16T14:13:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 10 client.13203 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T14:13:30.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 20 client.13203 populate_metadata read hostname 'smithi060' 2023-10-16T14:13:30.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd13c970 0x55a9fd13cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 4 client.13203 mds_command: new command op to 13065 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T14:13:30.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 --> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a9fcfed700 con 0x55a9fd13c970 2023-10-16T14:13:30.727 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.713+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd13c970 0x55a9fd13cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57bdd9b640 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd13c970 0x55a9fd13cd50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d107f3117ff7ff server_cookie=e16d1f04006dbc86 in_seq=0 out_seq=0 2023-10-16T14:13:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57a57fa640 10 client.13203 ms_handle_connect on v2:172.21.15.60:6834/2549968691 2023-10-16T14:13:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57a57fa640 1 -- 172.21.15.60:0/374886952 <== mds.0 v2:172.21.15.60:6834/2549968691 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55a9fcfed700 con 0x55a9fd13c970 2023-10-16T14:13:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57a57fa640 10 client.13203 handle_command_reply: tid=0 2023-10-16T14:13:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57cf79c740 1 client.13203 shutdown 2023-10-16T14:13:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 msgr2=0x55a9fd0d76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.714+0000 7f57cf79c740 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 0x55a9fd0d76a0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f57b8058310 tx=0x7f57b80564c0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 shutdown_connections 2023-10-16T14:13:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:6834/2549968691,v1:172.21.15.60:6835/2549968691] conn(0x55a9fd13c970 0x55a9fd13cd50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x55a9fd0cde80 0x55a9fd0a9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55a9fd0b6550 0x55a9fd136370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57a57fa640 0 client.13203 ms_handle_reset on v2:172.21.15.60:6834/2549968691 2023-10-16T14:13:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 --2- 172.21.15.60:0/374886952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a9fd0b6170 0x55a9fd0d76a0 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 >> 172.21.15.60:0/374886952 conn(0x55a9fd0895b0 msgr2=0x55a9fd0caac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 shutdown_connections 2023-10-16T14:13:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 1 -- 172.21.15.60:0/374886952 wait complete. 2023-10-16T14:13:30.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.715+0000 7f57cf79c740 20 client.13203 trim_cache size 0 max 16384 2023-10-16T14:13:30.731 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:13:30.731 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "d49fcbeb-3f16-4e9a-bc57-c0252aab0f68", 2023-10-16T14:13:30.731 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2023-10-16T14:13:30.731 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 13065, 2023-10-16T14:13:30.732 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2023-10-16T14:13:30.732 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2023-10-16T14:13:30.732 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2023-10-16T14:13:30.732 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8201423239999999, 2023-10-16T14:13:30.732 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 152, 2023-10-16T14:13:30.733 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 287, 2023-10-16T14:13:30.733 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 287, 2023-10-16T14:13:30.733 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.674723262000001 2023-10-16T14:13:30.733 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:13:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f57a806af90 msgr2=0x7f57a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f57a806af90 0x7f57a806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57b0036000 tx=0x7f57b0008030 comp rx=0 tx=0).stop 2023-10-16T14:13:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 msgr2=0x7f57c80b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 0x7f57c80b0850 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f57b800ba50 tx=0x7f57b800bf10 comp rx=0 tx=0).stop 2023-10-16T14:13:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 shutdown_connections 2023-10-16T14:13:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f57a806af90 0x7f57a806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f57c80ad970 0x7f57c80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57c80ccfb0 0x7f57c815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 --2- 172.21.15.60:0/963803201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57c806c040 0x7f57c80b0850 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 >> 172.21.15.60:0/963803201 conn(0x7f57c80ce6a0 msgr2=0x7f57c806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 shutdown_connections 2023-10-16T14:13:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.716+0000 7f57c759e640 1 -- 172.21.15.60:0/963803201 wait complete. 2023-10-16T14:13:30.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T14:13:30.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.959+0000 7f4311e21640 1 Processor -- start 2023-10-16T14:13:30.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.959+0000 7f4311e21640 1 -- start start 2023-10-16T14:13:30.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311e21640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f430c15aad0 0x7f430c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311e21640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f430c0712c0 0x7f430c070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311e21640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c0b4620 con 0x7f430c070d70 2023-10-16T14:13:30.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311e21640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430c0b4340 con 0x7f430c0712c0 2023-10-16T14:13:30.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311e21640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430c06f520 con 0x7f430c15aad0 2023-10-16T14:13:30.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311620640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f430c0712c0 0x7f430c070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311620640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f430c0712c0 0x7f430c070000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33698/0 (socket says 172.21.15.60:33698) 2023-10-16T14:13:30.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.961+0000 7f4311620640 1 -- 172.21.15.60:0/1930564218 learned_addr learned my addr 172.21.15.60:0/1930564218 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.962+0000 7f4311620640 1 -- 172.21.15.60:0/1930564218 >> v2:172.21.15.144:3300/0 conn(0x7f430c15aad0 msgr2=0x7f430c0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.962+0000 7f4311620640 1 --2- 172.21.15.60:0/1930564218 >> v2:172.21.15.144:3300/0 conn(0x7f430c15aad0 0x7f430c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.962+0000 7f4311620640 1 -- 172.21.15.60:0/1930564218 >> v1:172.21.15.144:6789/0 conn(0x7f430c070d70 legacy=0x7f430c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:30.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.962+0000 7f4311620640 1 -- 172.21.15.60:0/1930564218 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c06f800 con 0x7f430c0712c0 2023-10-16T14:13:30.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.963+0000 7f4311620640 1 --2- 172.21.15.60:0/1930564218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c0712c0 0x7f430c070000 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f430c0cf4f0 tx=0x7f430402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed29f265852dc73f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.963+0000 7f430b7fe640 1 -- 172.21.15.60:0/1930564218 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430402cb20 con 0x7f430c0712c0 2023-10-16T14:13:30.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.963+0000 7f430b7fe640 1 -- 172.21.15.60:0/1930564218 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430402cc80 con 0x7f430c0712c0 2023-10-16T14:13:30.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.963+0000 7f430b7fe640 1 -- 172.21.15.60:0/1930564218 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43040343f0 con 0x7f430c0712c0 2023-10-16T14:13:30.969 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 -- 172.21.15.60:0/1930564218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c0712c0 msgr2=0x7f430c070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 --2- 172.21.15.60:0/1930564218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c0712c0 0x7f430c070000 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f430c0cf4f0 tx=0x7f430402bf80 comp rx=0 tx=0).stop 2023-10-16T14:13:30.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 -- 172.21.15.60:0/1930564218 shutdown_connections 2023-10-16T14:13:30.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 --2- 172.21.15.60:0/1930564218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c0712c0 0x7f430c070000 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 --2- 172.21.15.60:0/1930564218 >> v2:172.21.15.144:3300/0 conn(0x7f430c15aad0 0x7f430c0cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 -- 172.21.15.60:0/1930564218 >> 172.21.15.60:0/1930564218 conn(0x7f430c0b0a90 msgr2=0x7f430c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:30.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 -- 172.21.15.60:0/1930564218 shutdown_connections 2023-10-16T14:13:30.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.964+0000 7f4311e21640 1 -- 172.21.15.60:0/1930564218 wait complete. 2023-10-16T14:13:30.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.965+0000 7f4311e21640 1 Processor -- start 2023-10-16T14:13:30.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.966+0000 7f4311e21640 1 -- start start 2023-10-16T14:13:30.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311e21640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 0x7f430c156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.973 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311e21640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f430c0712c0 0x7f430c15d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311e21640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f430c15aad0 0x7f430c15d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311e21640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0c21b0 con 0x7f430c070d70 2023-10-16T14:13:30.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311e21640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0af560 con 0x7f430c15aad0 2023-10-16T14:13:30.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311e21640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0c2470 con 0x7f430c0712c0 2023-10-16T14:13:30.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 0x7f430c156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 0x7f430c156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33714/0 (socket says 172.21.15.60:33714) 2023-10-16T14:13:30.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 -- 172.21.15.60:0/3588134029 learned_addr learned my addr 172.21.15.60:0/3588134029 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:30.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f430bfff640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f430c0712c0 0x7f430c15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 -- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f430c0712c0 msgr2=0x7f430c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f430c0712c0 0x7f430c15d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4311620640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f430c15aad0 0x7f430c15d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 -- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f430c15aad0 msgr2=0x7f430c15d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:30.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f430c15aad0 0x7f430c15d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:30.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f4310e1f640 1 -- 172.21.15.60:0/3588134029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304009b30 con 0x7f430c070d70 2023-10-16T14:13:30.976 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.967+0000 7f430bfff640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f430c0712c0 0x7f430c15d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.968+0000 7f4310e1f640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 0x7f430c156bc0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7f430000b410 tx=0x7f430000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.968+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300011830 con 0x7f430c070d70 2023-10-16T14:13:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.968+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43000119b0 con 0x7f430c070d70 2023-10-16T14:13:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.968+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300011d60 con 0x7f430c070d70 2023-10-16T14:13:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.968+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c0c2750 con 0x7f430c070d70 2023-10-16T14:13:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.968+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c157100 con 0x7f430c070d70 2023-10-16T14:13:30.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.969+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c0c2c70 con 0x7f430c070d70 2023-10-16T14:13:30.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.970+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f430001a400 con 0x7f430c070d70 2023-10-16T14:13:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.971+0000 7f43097fa640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f42f006af90 0x7f42f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.971+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4300089910 con 0x7f430c070d70 2023-10-16T14:13:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.972+0000 7f430bfff640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f42f006af90 0x7f42f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.973+0000 7f430bfff640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f42f006af90 0x7f42f006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42f8007fd0 tx=0x7f42f8009040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:30.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:30.976+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f430c0c2c70 con 0x7f430c070d70 2023-10-16T14:13:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.212+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f430c0c2c70 con 0x7f430c070d70 2023-10-16T14:13:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.213+0000 7f43097fa640 1 -- 172.21.15.60:0/3588134029 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v152) v1 ==== 53+0+73 (secure 0 0 0) 0x7f430c0c2c70 con 0x7f430c070d70 2023-10-16T14:13:31.217 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T14:13:31.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f42f006af90 msgr2=0x7f42f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:31.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f42f006af90 0x7f42f006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42f8007fd0 tx=0x7f42f8009040 comp rx=0 tx=0).stop 2023-10-16T14:13:31.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 msgr2=0x7f430c156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:31.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 0x7f430c156bc0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7f430000b410 tx=0x7f430000b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 shutdown_connections 2023-10-16T14:13:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f42f006af90 0x7f42f006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f430c0712c0 0x7f430c15d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f430c15aad0 0x7f430c15d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:31.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 --2- 172.21.15.60:0/3588134029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f430c070d70 0x7f430c156bc0 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 >> 172.21.15.60:0/3588134029 conn(0x7f430c0b0a90 msgr2=0x7f430c0ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 shutdown_connections 2023-10-16T14:13:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:31.216+0000 7f4311e21640 1 -- 172.21.15.60:0/3588134029 wait complete. 2023-10-16T14:13:31.240 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T14:13:31.240 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T14:13:31.241 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2023-10-16T14:13:31.241 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T14:13:31.241 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T14:13:31.242 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T14:13:31.242 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T14:13:31.242 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T14:13:31.243 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:13:31.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.243 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2023-10-16T14:13:31.301 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T14:13:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T14:13:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T14:13:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2023-10-16T14:13:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.302 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:13:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2023-10-16T14:13:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2023-10-16T14:13:31.303 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:13:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2023-10-16T14:13:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2023-10-16T14:13:31.303 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T14:13:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:13:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2023-10-16T14:13:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T14:13:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 36376sec preferred_lft 36376sec 2023-10-16T14:13:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2023-10-16T14:13:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.305 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T14:13:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2023-10-16T14:13:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2023-10-16T14:13:31.305 INFO:teuthology.orchestra.run.smithi060.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T14:13:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff 2023-10-16T14:13:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T14:13:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::9878:11ff:fe1d:2ab4/64 scope link 2023-10-16T14:13:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.307 INFO:teuthology.orchestra.run.smithi060.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:13:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether e2:d0:86:cb:0c:a6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:13:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::e0d0:86ff:fecb:ca6/64 scope link 2023-10-16T14:13:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.307 INFO:teuthology.orchestra.run.smithi060.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T14:13:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 76:b0:b5:e8:17:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:13:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::74b0:b5ff:fee8:17cb/64 scope link 2023-10-16T14:13:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2023-10-16T14:13:31.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.310 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T14:13:31.407 INFO:teuthology.orchestra.run.smithi060.stdout:1 2023-10-16T14:13:31.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.412 DEBUG:teuthology.orchestra.run.smithi060:> route 2023-10-16T14:13:31.470 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2023-10-16T14:13:31.471 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T14:13:31.471 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T14:13:31.472 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T14:13:31.472 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T14:13:31.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.473 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2023-10-16T14:13:31.473 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2023-10-16T14:13:31.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T14:13:31.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T14:13:31.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T14:13:31.475 DEBUG:teuthology.orchestra.run.smithi060:> ') 2023-10-16T14:13:31.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.674 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:13:31.729 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:13:31.729 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:13:31.730 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:13:31.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2023-10-16T14:13:31.832 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:13:31.833 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T14:13:31.895 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T14:13:31.895 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T14:13:31.896 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:13:31.947 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-16T14:13:31.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:31.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:13:32.089 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:13:32.089 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:13:32.125 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-16T14:13:32.125 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-16T14:13:32.125 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.144:3300,172.21.15.144:3301" 2023-10-16T14:13:32.126 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@d49fcbeb-3f16-4e9a-bc57-c0252aab0f68.cephfs=/ 2023-10-16T14:13:32.126 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.144:3300/172.21.15.144:3301" will pass to kernel 2023-10-16T14:13:32.128 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-16T14:13:32.128 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2023-10-16T14:13:32.128 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-16T14:13:32.128 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2023-10-16T14:13:32.129 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-16T14:13:32.132 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T14:13:32.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:32.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T14:13:32.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:32.234 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T14:13:32.308 INFO:teuthology.orchestra.run.smithi060.stdout:client10315 2023-10-16T14:13:32.308 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:32.309 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:32.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T14:13:32.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.579+0000 7fc3833e9640 1 Processor -- start 2023-10-16T14:13:32.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.579+0000 7fc3833e9640 1 -- start start 2023-10-16T14:13:32.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc3833e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc37c06c040 0x7fc37c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:32.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc3833e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:32.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc3833e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c06c960 con 0x7fc37c0ad970 2023-10-16T14:13:32.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc3833e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37c15b9d0 con 0x7fc37c06c040 2023-10-16T14:13:32.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc3833e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c15bcb0 con 0x7fc37c0cda80 2023-10-16T14:13:32.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc3823e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc37c06c040 0x7fc37c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc381be6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc37c0ad970 0x7fc37c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51122/0 (socket says 172.21.15.60:51122) 2023-10-16T14:13:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.581+0000 7fc381be6640 1 -- 172.21.15.60:0/349159520 learned_addr learned my addr 172.21.15.60:0/349159520 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.582+0000 7fc382be8640 1 --2- 172.21.15.60:0/349159520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.582+0000 7fc3823e7640 1 -- 172.21.15.60:0/349159520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 msgr2=0x7fc37c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.582+0000 7fc3823e7640 1 --2- 172.21.15.60:0/349159520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.582+0000 7fc3823e7640 1 -- 172.21.15.60:0/349159520 >> v1:172.21.15.60:6789/0 conn(0x7fc37c0ad970 legacy=0x7fc37c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.582+0000 7fc3823e7640 1 -- 172.21.15.60:0/349159520 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c15bf90 con 0x7fc37c06c040 2023-10-16T14:13:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.582+0000 7fc3813e5640 1 -- 172.21.15.60:0/349159520 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854579588 0 0) 0x7fc37c06c960 con 0x7fc37c0ad970 2023-10-16T14:13:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.583+0000 7fc3823e7640 1 --2- 172.21.15.60:0/349159520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c06c040 0x7fc37c06c420 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7fc370009d90 tx=0x7fc37002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead5165e6c0e95d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.583+0000 7fc3813e5640 1 -- 172.21.15.60:0/349159520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37002cc50 con 0x7fc37c06c040 2023-10-16T14:13:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.583+0000 7fc3813e5640 1 -- 172.21.15.60:0/349159520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37002cdb0 con 0x7fc37c06c040 2023-10-16T14:13:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.583+0000 7fc3813e5640 1 -- 172.21.15.60:0/349159520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc370034660 con 0x7fc37c06c040 2023-10-16T14:13:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 -- 172.21.15.60:0/349159520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c06c040 msgr2=0x7fc37c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 --2- 172.21.15.60:0/349159520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c06c040 0x7fc37c06c420 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7fc370009d90 tx=0x7fc37002bea0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 -- 172.21.15.60:0/349159520 shutdown_connections 2023-10-16T14:13:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 --2- 172.21.15.60:0/349159520 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 --2- 172.21.15.60:0/349159520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c06c040 0x7fc37c06c420 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 -- 172.21.15.60:0/349159520 >> 172.21.15.60:0/349159520 conn(0x7fc37c0ce6a0 msgr2=0x7fc37c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 -- 172.21.15.60:0/349159520 shutdown_connections 2023-10-16T14:13:32.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.584+0000 7fc3833e9640 1 -- 172.21.15.60:0/349159520 wait complete. 2023-10-16T14:13:32.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.586+0000 7fc3833e9640 1 Processor -- start 2023-10-16T14:13:32.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.586+0000 7fc3833e9640 1 -- start start 2023-10-16T14:13:32.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc3833e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc37c06c040 0x7fc37c0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:32.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc3833e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 0x7fc37c0c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:32.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc3833e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c0c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc3833e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c150040 con 0x7fc37c0ad970 2023-10-16T14:13:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc3833e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c1543c0 con 0x7fc37c06c040 2023-10-16T14:13:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc3833e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c14f7a0 con 0x7fc37c0cda80 2023-10-16T14:13:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.587+0000 7fc381be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 0x7fc37c0c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:32.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 0x7fc37c0c5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33730/0 (socket says 172.21.15.60:33730) 2023-10-16T14:13:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 -- 172.21.15.60:0/1684043706 learned_addr learned my addr 172.21.15.60:0/1684043706 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc3823e7640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc37c06c040 0x7fc37c0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc382be8640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c0c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 -- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 msgr2=0x7fc37c0c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c0c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 -- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc37c06c040 msgr2=0x7fc37c0c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc37c06c040 0x7fc37c0c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 -- 172.21.15.60:0/1684043706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370009a70 con 0x7fc37c0ad970 2023-10-16T14:13:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.588+0000 7fc381be6640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 0x7fc37c0c5f60 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c4b0 tx=0x7fc36800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.589+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3680074e0 con 0x7fc37c0ad970 2023-10-16T14:13:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.589+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc368007660 con 0x7fc37c0ad970 2023-10-16T14:13:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.589+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368004a20 con 0x7fc37c0ad970 2023-10-16T14:13:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.589+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c14fa80 con 0x7fc37c0ad970 2023-10-16T14:13:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.589+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c0cac10 con 0x7fc37c0ad970 2023-10-16T14:13:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.590+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c06b330 con 0x7fc37c0ad970 2023-10-16T14:13:32.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.591+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc368004b80 con 0x7fc37c0ad970 2023-10-16T14:13:32.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.592+0000 7fc37affd640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fc36006af90 0x7fc36006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.593+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc368088950 con 0x7fc37c0ad970 2023-10-16T14:13:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.593+0000 7fc3823e7640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fc36006af90 0x7fc36006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.594+0000 7fc3823e7640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fc36006af90 0x7fc36006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc370036000 tx=0x7fc370030000 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.597+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3680539a0 con 0x7fc37c0ad970 2023-10-16T14:13:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.816+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 --> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc37c0cc550 con 0x7fc36006af90 2023-10-16T14:13:32.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.820+0000 7fc37affd640 1 -- 172.21.15.60:0/1684043706 <== mgr.10042 v2:172.21.15.144:6832/3623959149 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc3700344c0 con 0x7fc36006af90 2023-10-16T14:13:32.824 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-10-16T14:13:32.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-10-16T14:13:32.824 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs" 2023-10-16T14:13:32.824 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-10-16T14:13:32.825 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-10-16T14:13:32.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fc36006af90 msgr2=0x7fc36006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:32.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fc36006af90 0x7fc36006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc370036000 tx=0x7fc370030000 comp rx=0 tx=0).stop 2023-10-16T14:13:32.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 msgr2=0x7fc37c0c5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:32.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 0x7fc37c0c5f60 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c4b0 tx=0x7fc36800c970 comp rx=0 tx=0).stop 2023-10-16T14:13:32.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 shutdown_connections 2023-10-16T14:13:32.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fc36006af90 0x7fc36006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc37c0cda80 0x7fc37c0c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc37c0ad970 0x7fc37c0c5f60 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.822+0000 7fc3833e9640 1 --2- 172.21.15.60:0/1684043706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc37c06c040 0x7fc37c0c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:32.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.823+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 >> 172.21.15.60:0/1684043706 conn(0x7fc37c0ce6a0 msgr2=0x7fc37c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:32.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.823+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 shutdown_connections 2023-10-16T14:13:32.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:32.823+0000 7fc3833e9640 1 -- 172.21.15.60:0/1684043706 wait complete. 2023-10-16T14:13:32.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T14:13:33.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.071+0000 7f6b0d2f1640 1 Processor -- start 2023-10-16T14:13:33.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.071+0000 7f6b0d2f1640 1 -- start start 2023-10-16T14:13:33.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b0d2f1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b08159950 0x7f6b0815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:33.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b0d2f1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b08070450 0x7f6b080708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:33.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b0d2f1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b08070df0 con 0x7f6b0806c100 2023-10-16T14:13:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b0d2f1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0815e490 con 0x7f6b08070450 2023-10-16T14:13:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b0d2f1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0815aef0 con 0x7f6b08159950 2023-10-16T14:13:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b077fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b08070450 0x7f6b080708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b077fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b08070450 0x7f6b080708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33744/0 (socket says 172.21.15.60:33744) 2023-10-16T14:13:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b077fe640 1 -- 172.21.15.60:0/1594277601 learned_addr learned my addr 172.21.15.60:0/1594277601 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:33.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.073+0000 7f6b07fff640 1 --2- 172.21.15.60:0/1594277601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b08159950 0x7f6b0815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:33.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.074+0000 7f6b077fe640 1 -- 172.21.15.60:0/1594277601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b08159950 msgr2=0x7f6b0815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.074+0000 7f6b077fe640 1 --2- 172.21.15.60:0/1594277601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b08159950 0x7f6b0815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.074+0000 7f6b077fe640 1 -- 172.21.15.60:0/1594277601 >> v1:172.21.15.144:6789/0 conn(0x7f6b0806c100 legacy=0x7f6b0806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.074+0000 7f6b077fe640 1 -- 172.21.15.60:0/1594277601 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0806cc80 con 0x7f6b08070450 2023-10-16T14:13:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.075+0000 7f6b077fe640 1 --2- 172.21.15.60:0/1594277601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08070450 0x7f6b080708b0 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009870 tx=0x7f6afc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8ecd99f27f079a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:33.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.075+0000 7f6b06ffd640 1 -- 172.21.15.60:0/1594277601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0317c0 con 0x7f6b08070450 2023-10-16T14:13:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b06ffd640 1 -- 172.21.15.60:0/1594277601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc031920 con 0x7f6b08070450 2023-10-16T14:13:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b06ffd640 1 -- 172.21.15.60:0/1594277601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc031cb0 con 0x7f6b08070450 2023-10-16T14:13:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/1594277601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08070450 msgr2=0x7f6b080708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/1594277601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08070450 0x7f6b080708b0 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009870 tx=0x7f6afc02b860 comp rx=0 tx=0).stop 2023-10-16T14:13:33.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/1594277601 shutdown_connections 2023-10-16T14:13:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/1594277601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08070450 0x7f6b080708b0 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/1594277601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b08159950 0x7f6b0815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.076+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/1594277601 >> 172.21.15.60:0/1594277601 conn(0x7f6b080ce6a0 msgr2=0x7f6b080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.077+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/1594277601 shutdown_connections 2023-10-16T14:13:33.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.077+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/1594277601 wait complete. 2023-10-16T14:13:33.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.078+0000 7f6b0d2f1640 1 Processor -- start 2023-10-16T14:13:33.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.078+0000 7f6b0d2f1640 1 -- start start 2023-10-16T14:13:33.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.079+0000 7f6b0d2f1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b0806c100 0x7f6b0814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:33.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.079+0000 7f6b0d2f1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b08070450 0x7f6b080c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:33.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.079+0000 7f6b0d2f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 0x7f6b080c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.079+0000 7f6b0d2f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080c27e0 con 0x7f6b08159950 2023-10-16T14:13:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.079+0000 7f6b0d2f1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080c21b0 con 0x7f6b08070450 2023-10-16T14:13:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.079+0000 7f6b0d2f1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080b34c0 con 0x7f6b0806c100 2023-10-16T14:13:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 0x7f6b080c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:33.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 0x7f6b080c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33756/0 (socket says 172.21.15.60:33756) 2023-10-16T14:13:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 -- 172.21.15.60:0/3851415546 learned_addr learned my addr 172.21.15.60:0/3851415546 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b07fff640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b0806c100 0x7f6b0814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b077fe640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b08070450 0x7f6b080c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:33.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 -- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b0806c100 msgr2=0x7f6b0814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b0806c100 0x7f6b0814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 -- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b08070450 msgr2=0x7f6b080c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b08070450 0x7f6b080c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b0caf0640 1 -- 172.21.15.60:0/3851415546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc0094d0 con 0x7f6b08159950 2023-10-16T14:13:33.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.080+0000 7f6b077fe640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b08070450 0x7f6b080c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.081+0000 7f6b0caf0640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 0x7f6b080c4900 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000b970 tx=0x7f6b0000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.081+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0000cb40 con 0x7f6b08159950 2023-10-16T14:13:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.081+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0000cca0 con 0x7f6b08159950 2023-10-16T14:13:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.081+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00014680 con 0x7f6b08159950 2023-10-16T14:13:33.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.081+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b080b37a0 con 0x7f6b08159950 2023-10-16T14:13:33.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.081+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b08150120 con 0x7f6b08159950 2023-10-16T14:13:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.082+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b080c2920 con 0x7f6b08159950 2023-10-16T14:13:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.083+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 154) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b0001d3b0 con 0x7f6b08159950 2023-10-16T14:13:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.085+0000 7f6b04ff9640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f6af406afe0 0x7f6af406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:33.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.085+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b00089620 con 0x7f6b08159950 2023-10-16T14:13:33.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.085+0000 7f6b07fff640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f6af406afe0 0x7f6af406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:33.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.086+0000 7f6b07fff640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f6af406afe0 0x7f6af406d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6af8006fd0 tx=0x7f6af8008040 comp rx=0 tx=0).ready entity=mgr.10042 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:33.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.089+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b080c2920 con 0x7f6b08159950 2023-10-16T14:13:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.340+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6b080c2920 con 0x7f6b08159950 2023-10-16T14:13:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.948+0000 7f6b04ff9640 1 -- 172.21.15.60:0/3851415546 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v155) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6b080c2920 con 0x7f6b08159950 2023-10-16T14:13:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.951+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f6af406afe0 msgr2=0x7f6af406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.951+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f6af406afe0 0x7f6af406d4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6af8006fd0 tx=0x7f6af8008040 comp rx=0 tx=0).stop 2023-10-16T14:13:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.951+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 msgr2=0x7f6b080c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 0x7f6b080c4900 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000b970 tx=0x7f6b0000be30 comp rx=0 tx=0).stop 2023-10-16T14:13:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 shutdown_connections 2023-10-16T14:13:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f6af406afe0 0x7f6af406d4a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b08159950 0x7f6b080c4900 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b08070450 0x7f6b080c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 --2- 172.21.15.60:0/3851415546 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6b0806c100 0x7f6b0814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.952+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 >> 172.21.15.60:0/3851415546 conn(0x7f6b080ce6a0 msgr2=0x7f6b08072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.953+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 shutdown_connections 2023-10-16T14:13:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:33.953+0000 7f6b0d2f1640 1 -- 172.21.15.60:0/3851415546 wait complete. 2023-10-16T14:13:33.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T14:13:34.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.179+0000 7fd741bfc640 1 Processor -- start 2023-10-16T14:13:34.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.179+0000 7fd741bfc640 1 -- start start 2023-10-16T14:13:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd741bfc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd73c06b8c0 0x7fd73c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd741bfc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd73c15d130 0x7fd73c0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd741bfc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c0cd390 con 0x7fd73c06b1a0 2023-10-16T14:13:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd741bfc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c06c1e0 con 0x7fd73c06b8c0 2023-10-16T14:13:34.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd741bfc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c0cc2f0 con 0x7fd73c15d130 2023-10-16T14:13:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd73b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd73c06b8c0 0x7fd73c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd73b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd73c06b8c0 0x7fd73c06bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33776/0 (socket says 172.21.15.60:33776) 2023-10-16T14:13:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd73b7fe640 1 -- 172.21.15.60:0/3138236364 learned_addr learned my addr 172.21.15.60:0/3138236364 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd73b7fe640 1 -- 172.21.15.60:0/3138236364 >> v2:172.21.15.144:3300/0 conn(0x7fd73c15d130 msgr2=0x7fd73c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.181+0000 7fd733fff640 1 --2- 172.21.15.60:0/3138236364 >> v2:172.21.15.144:3300/0 conn(0x7fd73c15d130 0x7fd73c0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.182+0000 7fd73b7fe640 1 --2- 172.21.15.60:0/3138236364 >> v2:172.21.15.144:3300/0 conn(0x7fd73c15d130 0x7fd73c0cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.182+0000 7fd73b7fe640 1 -- 172.21.15.60:0/3138236364 >> v1:172.21.15.144:6789/0 conn(0x7fd73c06b1a0 legacy=0x7fd73c0cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.182+0000 7fd73b7fe640 1 -- 172.21.15.60:0/3138236364 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0cc5d0 con 0x7fd73c06b8c0 2023-10-16T14:13:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.182+0000 7fd733fff640 1 --2- 172.21.15.60:0/3138236364 >> v2:172.21.15.144:3300/0 conn(0x7fd73c15d130 0x7fd73c0cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.183+0000 7fd73b7fe640 1 --2- 172.21.15.60:0/3138236364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c06b8c0 0x7fd73c06bca0 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7fd73c073d50 tx=0x7fd72c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ffbbd3030cb104 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.183+0000 7fd73affd640 1 -- 172.21.15.60:0/3138236364 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c004230 con 0x7fd73c06b8c0 2023-10-16T14:13:34.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.183+0000 7fd73affd640 1 -- 172.21.15.60:0/3138236364 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c004390 con 0x7fd73c06b8c0 2023-10-16T14:13:34.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.183+0000 7fd73affd640 1 -- 172.21.15.60:0/3138236364 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c003810 con 0x7fd73c06b8c0 2023-10-16T14:13:34.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 -- 172.21.15.60:0/3138236364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c06b8c0 msgr2=0x7fd73c06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3138236364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c06b8c0 0x7fd73c06bca0 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7fd73c073d50 tx=0x7fd72c02b8f0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 -- 172.21.15.60:0/3138236364 shutdown_connections 2023-10-16T14:13:34.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3138236364 >> v2:172.21.15.144:3300/0 conn(0x7fd73c15d130 0x7fd73c0cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3138236364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c06b8c0 0x7fd73c06bca0 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 -- 172.21.15.60:0/3138236364 >> 172.21.15.60:0/3138236364 conn(0x7fd73c0d0ad0 msgr2=0x7fd73c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:34.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 -- 172.21.15.60:0/3138236364 shutdown_connections 2023-10-16T14:13:34.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.184+0000 7fd741bfc640 1 -- 172.21.15.60:0/3138236364 wait complete. 2023-10-16T14:13:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.186+0000 7fd741bfc640 1 Processor -- start 2023-10-16T14:13:34.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.186+0000 7fd741bfc640 1 -- start start 2023-10-16T14:13:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd741bfc640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd73c06b1a0 0x7fd73c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd741bfc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c06b8c0 0x7fd73c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd741bfc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 0x7fd73c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd741bfc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0cab80 con 0x7fd73c15d130 2023-10-16T14:13:34.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd741bfc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c1f6b10 con 0x7fd73c06b8c0 2023-10-16T14:13:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd741bfc640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0c7ff0 con 0x7fd73c06b1a0 2023-10-16T14:13:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd73bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 0x7fd73c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd73bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 0x7fd73c155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33780/0 (socket says 172.21.15.60:33780) 2023-10-16T14:13:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd73bfff640 1 -- 172.21.15.60:0/3458113838 learned_addr learned my addr 172.21.15.60:0/3458113838 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:34.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd73b7fe640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd73c06b1a0 0x7fd73c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.187+0000 7fd733fff640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c06b8c0 0x7fd73c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd73bfff640 1 -- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd73c06b1a0 msgr2=0x7fd73c1f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd73bfff640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd73c06b1a0 0x7fd73c1f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd73bfff640 1 -- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c06b8c0 msgr2=0x7fd73c15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd73bfff640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c06b8c0 0x7fd73c15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd73bfff640 1 -- 172.21.15.60:0/3458113838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c009460 con 0x7fd73c15d130 2023-10-16T14:13:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd73bfff640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 0x7fd73c155b90 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fd73400c780 tx=0x7fd73400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.188+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734003d80 con 0x7fd73c15d130 2023-10-16T14:13:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.189+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73400ce90 con 0x7fd73c15d130 2023-10-16T14:13:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.189+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c0c82d0 con 0x7fd73c15d130 2023-10-16T14:13:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.189+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7340052b0 con 0x7fd73c15d130 2023-10-16T14:13:34.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.189+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c0c8890 con 0x7fd73c15d130 2023-10-16T14:13:34.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.190+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c04e160 con 0x7fd73c15d130 2023-10-16T14:13:34.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.191+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 155) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fd734005410 con 0x7fd73c15d130 2023-10-16T14:13:34.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.192+0000 7fd738ff9640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 0x7fd71c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.192+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd734088f90 con 0x7fd73c15d130 2023-10-16T14:13:34.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.192+0000 7fd73b7fe640 1 -- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 msgr2=0x7fd71c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:34.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.192+0000 7fd73b7fe640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 0x7fd71c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:34.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.201+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd734053fd0 con 0x7fd73c15d130 2023-10-16T14:13:34.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.393+0000 7fd73b7fe640 1 -- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 msgr2=0x7fd71c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:34.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.393+0000 7fd73b7fe640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 0x7fd71c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:34.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.395+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd73c06b6a0 con 0x7fd73c15d130 2023-10-16T14:13:34.406 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.402+0000 7fd738ff9640 1 -- 172.21.15.60:0/3458113838 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd734017070 con 0x7fd73c15d130 2023-10-16T14:13:34.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 msgr2=0x7fd71c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:34.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 0x7fd71c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 msgr2=0x7fd73c155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 0x7fd73c155b90 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fd73400c780 tx=0x7fd73400cc40 comp rx=0 tx=0).stop 2023-10-16T14:13:34.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 shutdown_connections 2023-10-16T14:13:34.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fd71c06afe0 0x7fd71c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd73c15d130 0x7fd73c155b90 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd73c06b8c0 0x7fd73c15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 --2- 172.21.15.60:0/3458113838 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd73c06b1a0 0x7fd73c1f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 >> 172.21.15.60:0/3458113838 conn(0x7fd73c0d0ad0 msgr2=0x7fd73c06ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:34.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 shutdown_connections 2023-10-16T14:13:34.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.405+0000 7fd741bfc640 1 -- 172.21.15.60:0/3458113838 wait complete. 2023-10-16T14:13:34.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T14:13:34.489 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:34.486+0000 7f869dcda040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:13:34.502 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:34.499+0000 7f7707e3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:13:34.542 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:34.540+0000 7f869dcda040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:13:34.560 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:34.557+0000 7f7707e3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:13:34.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.625+0000 7f2cee989640 1 Processor -- start 2023-10-16T14:13:34.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.625+0000 7f2cee989640 1 -- start start 2023-10-16T14:13:34.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.626+0000 7f2cee989640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070830 0x7f2ce806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.626+0000 7f2cee989640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ce8070c10 0x7f2ce80cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.626+0000 7f2cee989640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce8161740 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.626+0000 7f2cee989640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80cdbd0 con 0x7f2ce8070c10 2023-10-16T14:13:34.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.626+0000 7f2cee989640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce806b0f0 con 0x7f2ce8070830 2023-10-16T14:13:34.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.627+0000 7f2cdf7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2ce80cf5f0 0x7f2ce806c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51134/0 (socket says 172.21.15.60:51134) 2023-10-16T14:13:34.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.627+0000 7f2cdf7fe640 1 -- 172.21.15.60:0/3469830349 learned_addr learned my addr 172.21.15.60:0/3469830349 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:34.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.627+0000 7f2ce7fff640 1 --2- 172.21.15.60:0/3469830349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070830 0x7f2ce806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.627+0000 7f2cecf26640 1 --2- 172.21.15.60:0/3469830349 >> v2:172.21.15.144:3300/0 conn(0x7f2ce8070c10 0x7f2ce80cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.627+0000 7f2ce77fe640 1 -- 172.21.15.60:0/3469830349 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379711963 0 0) 0x7f2ce8161740 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.627+0000 7f2ce77fe640 1 -- 172.21.15.60:0/3469830349 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4003520 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.628+0000 7f2ce7fff640 1 -- 172.21.15.60:0/3469830349 >> v2:172.21.15.144:3300/0 conn(0x7f2ce8070c10 msgr2=0x7f2ce80cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.628+0000 7f2ce7fff640 1 --2- 172.21.15.60:0/3469830349 >> v2:172.21.15.144:3300/0 conn(0x7f2ce8070c10 0x7f2ce80cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.628+0000 7f2ce7fff640 1 -- 172.21.15.60:0/3469830349 >> v1:172.21.15.60:6789/0 conn(0x7f2ce80cf5f0 legacy=0x7f2ce806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.628+0000 7f2ce77fe640 1 -- 172.21.15.60:0/3469830349 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 409850475 0 0) 0x7f2cd4003520 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.628+0000 7f2ce7fff640 1 -- 172.21.15.60:0/3469830349 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce81619b0 con 0x7f2ce8070830 2023-10-16T14:13:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.629+0000 7f2ce7fff640 1 --2- 172.21.15.60:0/3469830349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070830 0x7f2ce806d3c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009d90 tx=0x7f2cd802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da166e888520eef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2ce77fe640 1 -- 172.21.15.60:0/3469830349 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0001670 con 0x7f2ce8070830 2023-10-16T14:13:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2ce77fe640 1 -- 172.21.15.60:0/3469830349 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd802c780 con 0x7f2ce8070830 2023-10-16T14:13:34.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2ce77fe640 1 -- 172.21.15.60:0/3469830349 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd802caf0 con 0x7f2ce8070830 2023-10-16T14:13:34.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 -- 172.21.15.60:0/3469830349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070830 msgr2=0x7f2ce806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 --2- 172.21.15.60:0/3469830349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070830 0x7f2ce806d3c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009d90 tx=0x7f2cd802bc50 comp rx=0 tx=0).stop 2023-10-16T14:13:34.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 -- 172.21.15.60:0/3469830349 shutdown_connections 2023-10-16T14:13:34.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 --2- 172.21.15.60:0/3469830349 >> v2:172.21.15.144:3300/0 conn(0x7f2ce8070c10 0x7f2ce80cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 --2- 172.21.15.60:0/3469830349 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070830 0x7f2ce806d3c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 -- 172.21.15.60:0/3469830349 >> 172.21.15.60:0/3469830349 conn(0x7f2ce80cfac0 msgr2=0x7f2ce80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:34.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.630+0000 7f2cee989640 1 -- 172.21.15.60:0/3469830349 shutdown_connections 2023-10-16T14:13:34.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.631+0000 7f2cee989640 1 -- 172.21.15.60:0/3469830349 wait complete. 2023-10-16T14:13:34.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.632+0000 7f2cee989640 1 Processor -- start 2023-10-16T14:13:34.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.632+0000 7f2cee989640 1 -- start start 2023-10-16T14:13:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.633+0000 7f2cee989640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce8070830 0x7f2ce815eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.633+0000 7f2cee989640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070c10 0x7f2ce80b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.633+0000 7f2cee989640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 0x7f2ce80b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.633+0000 7f2cee989640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b2ec0 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.633+0000 7f2cee989640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b2340 con 0x7f2ce8070830 2023-10-16T14:13:34.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.633+0000 7f2cee989640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b2620 con 0x7f2ce8070c10 2023-10-16T14:13:34.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 0x7f2ce80b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 0x7f2ce80b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33794/0 (socket says 172.21.15.60:33794) 2023-10-16T14:13:34.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 -- 172.21.15.60:0/3712063865 learned_addr learned my addr 172.21.15.60:0/3712063865 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:34.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2ce7fff640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce8070830 0x7f2ce815eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cdf7fe640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070c10 0x7f2ce80b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:34.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 -- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070c10 msgr2=0x7f2ce80b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070c10 0x7f2ce80b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 -- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce8070830 msgr2=0x7f2ce815eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce8070830 0x7f2ce815eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.634+0000 7f2cecf26640 1 -- 172.21.15.60:0/3712063865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8009a70 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.635+0000 7f2cecf26640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 0x7f2ce80b3eb0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e930 tx=0x7f2ce000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:34.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.635+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0009690 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.635+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce000c980 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.635+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce80b2900 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.635+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000cd30 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.644 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.635+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce815f080 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.645 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.636+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce804e160 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.637+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 155) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2ce0013470 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.638+0000 7f2ce57fa640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 0x7f2cc806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.638+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ce0089a20 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.638+0000 7f2ce7fff640 1 -- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 msgr2=0x7f2cc806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:34.646 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.638+0000 7f2ce7fff640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 0x7f2cc806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:34.647 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.642+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ce0054b20 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.836+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f2ce806d060 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.839+0000 7f2ce7fff640 1 -- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 msgr2=0x7f2cc806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:34.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.839+0000 7f2ce7fff640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 0x7f2cc806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:34.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.841+0000 7f2ce57fa640 1 -- 172.21.15.60:0/3712063865 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2ce00170c0 con 0x7f2ce80cf5f0 2023-10-16T14:13:34.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 msgr2=0x7f2cc806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 0x7f2cc806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 msgr2=0x7f2ce80b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 0x7f2ce80b3eb0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e930 tx=0x7f2ce000edf0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 shutdown_connections 2023-10-16T14:13:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7f2cc806afe0 0x7f2cc806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ce8070c10 0x7f2ce80b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ce80cf5f0 0x7f2ce80b3eb0 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 --2- 172.21.15.60:0/3712063865 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce8070830 0x7f2ce815eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.844+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 >> 172.21.15.60:0/3712063865 conn(0x7f2ce80cfac0 msgr2=0x7f2ce815ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.845+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 shutdown_connections 2023-10-16T14:13:34.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:34.845+0000 7f2cee989640 1 -- 172.21.15.60:0/3712063865 wait complete. 2023-10-16T14:13:34.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 2023-10-16T14:13:34.990 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:34.987+0000 7f869dcda040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:13:35.041 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.038+0000 7f869dcda040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:13:35.045 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.041+0000 7f7707e3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:13:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.078+0000 7fa8c9d73640 1 Processor -- start 2023-10-16T14:13:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.078+0000 7fa8c9d73640 1 -- start start 2023-10-16T14:13:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.079+0000 7fa8c9d73640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070830 0x7fa8c406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.079+0000 7fa8c9d73640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c4161740 con 0x7fa8c4070c10 2023-10-16T14:13:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.079+0000 7fa8c9d73640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c406b0f0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.079+0000 7fa8c9d73640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41619b0 con 0x7fa8c4070830 2023-10-16T14:13:35.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.079+0000 7fa8c3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa8c4070c10 0x7fa8c40cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51136/0 (socket says 172.21.15.60:51136) 2023-10-16T14:13:35.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.079+0000 7fa8c3fff640 1 -- 172.21.15.60:0/1667705795 learned_addr learned my addr 172.21.15.60:0/1667705795 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:35.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/1667705795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070830 0x7fa8c406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163931430 0 0) 0x7fa8c4161740 con 0x7fa8c4070c10 2023-10-16T14:13:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ac003520 con 0x7fa8c4070c10 2023-10-16T14:13:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235969178 0 0) 0x7fa8c406b0f0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c4161740 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c37fe640 1 -- 172.21.15.60:0/1667705795 >> v1:172.21.15.144:6789/0 conn(0x7fa8c40cf5f0 legacy=0x7fa8c406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c37fe640 1 -- 172.21.15.60:0/1667705795 >> v1:172.21.15.60:6789/0 conn(0x7fa8c4070c10 legacy=0x7fa8c40cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2452993320 0 0) 0x7fa8ac003520 con 0x7fa8c4070c10 2023-10-16T14:13:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.080+0000 7fa8c37fe640 1 -- 172.21.15.60:0/1667705795 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c406b3d0 con 0x7fa8c4070830 2023-10-16T14:13:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.082+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/1667705795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070830 0x7fa8c406d3c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009dc0 tx=0x7fa8b402bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e365325d840a22ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.082+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc001670 con 0x7fa8c4070830 2023-10-16T14:13:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.082+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b402c7f0 con 0x7fa8c4070830 2023-10-16T14:13:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.082+0000 7fa8c27fc640 1 -- 172.21.15.60:0/1667705795 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b402cb60 con 0x7fa8c4070830 2023-10-16T14:13:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 -- 172.21.15.60:0/1667705795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070830 msgr2=0x7fa8c406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 --2- 172.21.15.60:0/1667705795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070830 0x7fa8c406d3c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009dc0 tx=0x7fa8b402bcc0 comp rx=0 tx=0).stop 2023-10-16T14:13:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 -- 172.21.15.60:0/1667705795 shutdown_connections 2023-10-16T14:13:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 --2- 172.21.15.60:0/1667705795 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070830 0x7fa8c406d3c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 -- 172.21.15.60:0/1667705795 >> 172.21.15.60:0/1667705795 conn(0x7fa8c40cfac0 msgr2=0x7fa8c40cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 -- 172.21.15.60:0/1667705795 shutdown_connections 2023-10-16T14:13:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.083+0000 7fa8c9d73640 1 -- 172.21.15.60:0/1667705795 wait complete. 2023-10-16T14:13:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.085+0000 7fa8c9d73640 1 Processor -- start 2023-10-16T14:13:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.085+0000 7fa8c9d73640 1 -- start start 2023-10-16T14:13:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c9d73640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c4070830 0x7fa8c415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c9d73640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070c10 0x7fa8c40b15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c9d73640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40cf5f0 0x7fa8c40b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c9d73640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41f35e0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c9d73640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40b3360 con 0x7fa8c4070830 2023-10-16T14:13:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c9d73640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41f29d0 con 0x7fa8c4070c10 2023-10-16T14:13:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40cf5f0 0x7fa8c40b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40cf5f0 0x7fa8c40b1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33800/0 (socket says 172.21.15.60:33800) 2023-10-16T14:13:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c3fff640 1 -- 172.21.15.60:0/51254217 learned_addr learned my addr 172.21.15.60:0/51254217 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c4070830 0x7fa8c415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.086+0000 7fa8c2ffd640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070c10 0x7fa8c40b15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:35.094 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.089+0000 7f869dcda040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:13:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c3fff640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070c10 msgr2=0x7fa8c40b15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c3fff640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070c10 0x7fa8c40b15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c3fff640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c4070830 msgr2=0x7fa8c415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c3fff640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8c4070830 0x7fa8c415b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c3fff640 1 -- 172.21.15.60:0/51254217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b4009a70 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c2ffd640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa8c4070c10 0x7fa8c40b15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c3fff640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40cf5f0 0x7fa8c40b1b30 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c680 tx=0x7fa8bc00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc005ad0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc005c30 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c41f2cb0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.087+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc007070 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.088+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c41f3270 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.089+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c40b34d0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.090+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 155) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fa8bc0071d0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.091+0000 7fa8abfff640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.091+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8bc089420 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.092+0000 7fa8c37fe640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 msgr2=0x7fa8b006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.092+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.096+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8c40b34d0 con 0x7fa8c40cf5f0 2023-10-16T14:13:35.102 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.099+0000 7f7707e3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:13:35.153 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.150+0000 7f869dcda040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:13:35.156 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.152+0000 7f7707e3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:13:35.201 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.198+0000 7f869dcda040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:13:35.220 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.216+0000 7f7707e3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:13:35.271 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.268+0000 7f7707e3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:13:35.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.293+0000 7fa8c37fe640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 msgr2=0x7fa8b006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.293+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:35.313 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.308+0000 7f869dcda040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:13:35.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.320+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 --> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8c40cf060 con 0x7fa8b006afe0 2023-10-16T14:13:35.373 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.370+0000 7f869dcda040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:35.387 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.384+0000 7f7707e3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:13:35.423 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.421+0000 7f869dcda040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:13:35.454 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.450+0000 7f7707e3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:35.481 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.478+0000 7f869dcda040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:13:35.505 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.501+0000 7f7707e3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:13:35.530 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.528+0000 7f869dcda040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:13:35.564 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.560+0000 7f7707e3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:13:35.616 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:35.613+0000 7f7707e3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:13:35.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.694+0000 7fa8c37fe640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 msgr2=0x7fa8b006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:35.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:35.694+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T14:13:35.908 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:35.905+0000 7f869dcda040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:13:36.015 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.011+0000 7f7707e3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:13:36.066 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.063+0000 7f869dcda040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:13:36.174 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.171+0000 7f869dcda040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:13:36.183 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.179+0000 7f7707e3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:13:36.223 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.220+0000 7f869dcda040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:13:36.295 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.292+0000 7f7707e3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:13:36.329 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.326+0000 7f869dcda040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:36.348 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.344+0000 7f7707e3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:13:36.434 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.431+0000 7f869dcda040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:13:36.461 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.457+0000 7f7707e3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:36.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:36.495+0000 7fa8c37fe640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 msgr2=0x7fa8b006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/3623959149 2023-10-16T14:13:36.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:36.495+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T14:13:36.572 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.568+0000 7f7707e3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:13:36.591 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.587+0000 7f869dcda040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:36.639 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.636+0000 7f869dcda040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:36.736 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.732+0000 7f7707e3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:36.757 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:36.754+0000 7f869dcda040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:13:36.787 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.783+0000 7f7707e3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:36.913 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:36.909+0000 7f7707e3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:13:37.589 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:37.586+0000 7f869dcda040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:13:37.777 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:37.773+0000 7f7707e3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:13:37.880 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:37.877+0000 7f869dcda040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:13:37.936 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:37.934+0000 7f869dcda040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:13:38.049 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.046+0000 7f869dcda040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:13:38.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:38.067+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 156) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fa8bc087940 con 0x7fa8c40cf5f0 2023-10-16T14:13:38.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:38.068+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 msgr2=0x7fa8b006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:38.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:38.068+0000 7fa8abfff640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/3623959149,v1:172.21.15.144:6833/3623959149] conn(0x7fa8b006afe0 0x7fa8b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:38.097 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:38.094+0000 7f7707e3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:13:38.164 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:38.161+0000 7f7707e3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:13:38.316 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:38.312+0000 7f7707e3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:13:38.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861d75e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861d75e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861d75e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861d75e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861d75e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861a718640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861a718640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.372 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861a718640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.373 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861a718640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:38.373 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:38.368+0000 7f861a718640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.078+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 157) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa8bc054790 con 0x7fa8c40cf5f0 2023-10-16T14:13:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.079+0000 7fa8abfff640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fa8b0035880 0x7fa8b0074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.079+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8b0035c60 con 0x7fa8b0035880 2023-10-16T14:13:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.081+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fa8b0035880 0x7fa8b0074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.096+0000 7fa8c37fe640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fa8b0035880 0x7fa8b0074ea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4036000 tx=0x7fa8b4030000 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:39.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:39.118+0000 7f8617f13640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:39.120 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:39.118+0000 7f8617f13640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:39.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:39.118+0000 7f8617f13640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:39.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:39.118+0000 7f8617f13640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:39.121 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:39.118+0000 7f8617f13640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:39.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.179+0000 7fa8abfff640 1 -- 172.21.15.60:0/51254217 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa8b0035c60 con 0x7fa8b0035880 2023-10-16T14:13:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fa8b0035880 msgr2=0x7fa8b0074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fa8b0035880 0x7fa8b0074ea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4036000 tx=0x7fa8b4030000 comp rx=0 tx=0).stop 2023-10-16T14:13:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40cf5f0 msgr2=0x7fa8c40b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 --2- 172.21.15.60:0/51254217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40cf5f0 0x7fa8c40b1b30 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c680 tx=0x7fa8bc00cb40 comp rx=0 tx=0).stop 2023-10-16T14:13:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c3fff640 1 -- 172.21.15.60:0/51254217 reap_dead start 2023-10-16T14:13:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 shutdown_connections 2023-10-16T14:13:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 >> 172.21.15.60:0/51254217 conn(0x7fa8c40cfac0 msgr2=0x7fa8c406c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 shutdown_connections 2023-10-16T14:13:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.183+0000 7fa8c9d73640 1 -- 172.21.15.60:0/51254217 wait complete. 2023-10-16T14:13:39.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv1 . 1m 2023-10-16T14:13:39.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.442+0000 7f169bdc0640 1 Processor -- start 2023-10-16T14:13:39.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.443+0000 7f169bdc0640 1 -- start start 2023-10-16T14:13:39.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f169bdc0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16941599c0 0x7f16940c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f169bdc0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1694159da0 0x7f169415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f169bdc0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169415dd10 con 0x7f16940cce20 2023-10-16T14:13:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f169bdc0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169415dfb0 con 0x7f16941599c0 2023-10-16T14:13:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f169bdc0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169415e290 con 0x7f1694159da0 2023-10-16T14:13:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f1699b5c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16941599c0 0x7f16940c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:39.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f1699b5c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16941599c0 0x7f16940c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55128/0 (socket says 172.21.15.60:55128) 2023-10-16T14:13:39.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f1699b5c640 1 -- 172.21.15.60:0/56639429 learned_addr learned my addr 172.21.15.60:0/56639429 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:39.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.444+0000 7f169a35d640 1 --1- 172.21.15.60:0/56639429 >> v1:172.21.15.60:6789/0 conn(0x7f16940cce20 0x7f169406c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43336/0 (socket says 172.21.15.60:43336) 2023-10-16T14:13:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f169935b640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1694159da0 0x7f169415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f1699b5c640 1 -- 172.21.15.60:0/56639429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1694159da0 msgr2=0x7f169415d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f1699b5c640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1694159da0 0x7f169415d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f1699b5c640 1 -- 172.21.15.60:0/56639429 >> v1:172.21.15.60:6789/0 conn(0x7f16940cce20 legacy=0x7f169406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f1699b5c640 1 -- 172.21.15.60:0/56639429 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169415e570 con 0x7f16941599c0 2023-10-16T14:13:39.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f1698b5a640 1 -- 172.21.15.60:0/56639429 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983421331 0 0) 0x7f169415dd10 con 0x7f16940cce20 2023-10-16T14:13:39.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.445+0000 7f169935b640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1694159da0 0x7f169415d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:39.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.446+0000 7f1699b5c640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16941599c0 0x7f16940c9300 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f1690009870 tx=0x7f169002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed79ffb41345fad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:39.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.446+0000 7f1698b5a640 1 -- 172.21.15.60:0/56639429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690030840 con 0x7f16941599c0 2023-10-16T14:13:39.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.446+0000 7f1698b5a640 1 -- 172.21.15.60:0/56639429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16900309a0 con 0x7f16941599c0 2023-10-16T14:13:39.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.446+0000 7f1698b5a640 1 -- 172.21.15.60:0/56639429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690030d30 con 0x7f16941599c0 2023-10-16T14:13:39.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 -- 172.21.15.60:0/56639429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16941599c0 msgr2=0x7f16940c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16941599c0 0x7f16940c9300 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f1690009870 tx=0x7f169002b980 comp rx=0 tx=0).stop 2023-10-16T14:13:39.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 -- 172.21.15.60:0/56639429 shutdown_connections 2023-10-16T14:13:39.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1694159da0 0x7f169415d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 --2- 172.21.15.60:0/56639429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16941599c0 0x7f16940c9300 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 -- 172.21.15.60:0/56639429 >> 172.21.15.60:0/56639429 conn(0x7f16940cb360 msgr2=0x7f16940cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:39.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 -- 172.21.15.60:0/56639429 shutdown_connections 2023-10-16T14:13:39.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.447+0000 7f169bdc0640 1 -- 172.21.15.60:0/56639429 wait complete. 2023-10-16T14:13:39.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.449+0000 7f169bdc0640 1 Processor -- start 2023-10-16T14:13:39.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.449+0000 7f169bdc0640 1 -- start start 2023-10-16T14:13:39.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f169bdc0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 0x7f16941f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f169bdc0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16941599c0 0x7f169414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f169bdc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1694159da0 0x7f16941f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f169bdc0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940b2b40 con 0x7f16940cce20 2023-10-16T14:13:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f169bdc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16941f34f0 con 0x7f1694159da0 2023-10-16T14:13:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f169bdc0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16941f2570 con 0x7f16941599c0 2023-10-16T14:13:39.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f1699b5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 0x7f16941f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:39.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f1699b5c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 0x7f16941f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55142/0 (socket says 172.21.15.60:55142) 2023-10-16T14:13:39.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.450+0000 7f1699b5c640 1 -- 172.21.15.60:0/377137910 learned_addr learned my addr 172.21.15.60:0/377137910 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:39.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f1699b5c640 1 -- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16941599c0 msgr2=0x7f169414dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f1699b5c640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16941599c0 0x7f169414dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f1699b5c640 1 -- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1694159da0 msgr2=0x7f16941f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f1699b5c640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1694159da0 0x7f16941f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f1699b5c640 1 -- 172.21.15.60:0/377137910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900094d0 con 0x7f16940cce20 2023-10-16T14:13:39.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f1699b5c640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 0x7f16941f2fb0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f169002b9c0 tx=0x7f1690004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690030810 con 0x7f16940cce20 2023-10-16T14:13:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690044480 con 0x7f16940cce20 2023-10-16T14:13:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690044850 con 0x7f16940cce20 2023-10-16T14:13:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.451+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169414e2b0 con 0x7f16940cce20 2023-10-16T14:13:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.452+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16941f6850 con 0x7f16940cce20 2023-10-16T14:13:39.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.453+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169404e160 con 0x7f16940cce20 2023-10-16T14:13:39.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.454+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 157) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f169002f390 con 0x7f16940cce20 2023-10-16T14:13:39.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.454+0000 7f168a7fc640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f167806ad10 0x7f167806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:39.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.455+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f169003e070 con 0x7f16940cce20 2023-10-16T14:13:39.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.455+0000 7f169935b640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f167806ad10 0x7f167806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:39.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.456+0000 7f169935b640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f167806ad10 0x7f167806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1684006fd0 tx=0x7f1684008040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:39.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.459+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16900770e0 con 0x7f16940cce20 2023-10-16T14:13:39.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.677+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv1", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f169406fa50 con 0x7f167806ad10 2023-10-16T14:13:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.903+0000 7f168a7fc640 1 -- 172.21.15.60:0/377137910 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+84 (secure 0 0 0) 0x7f168400ae00 con 0x7f167806ad10 2023-10-16T14:13:39.907 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path /volumes/_nogroup/sv1/140d4b27-ec8c-4bfc-b3ad-4c0bb1ce025b/.. 2023-10-16T14:13:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.906+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f167806ad10 msgr2=0x7f167806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.906+0000 7f169bdc0640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f167806ad10 0x7f167806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1684006fd0 tx=0x7f1684008040 comp rx=0 tx=0).stop 2023-10-16T14:13:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.906+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 msgr2=0x7f16941f2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 0x7f16941f2fb0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f169002b9c0 tx=0x7f1690004730 comp rx=0 tx=0).stop 2023-10-16T14:13:39.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 shutdown_connections 2023-10-16T14:13:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f167806ad10 0x7f167806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f16941599c0 0x7f169414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1694159da0 0x7f16941f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 --2- 172.21.15.60:0/377137910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16940cce20 0x7f16941f2fb0 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 >> 172.21.15.60:0/377137910 conn(0x7f16940cb360 msgr2=0x7f16940c4420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 shutdown_connections 2023-10-16T14:13:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:39.907+0000 7f169bdc0640 1 -- 172.21.15.60:0/377137910 wait complete. 2023-10-16T14:13:39.934 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv1/140d4b27-ec8c-4bfc-b3ad-4c0bb1ce025b/.. 2023-10-16T14:13:39.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv1 --group mygrp . 1m 2023-10-16T14:13:40.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.171+0000 7f0bc007c640 1 Processor -- start 2023-10-16T14:13:40.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.171+0000 7f0bc007c640 1 -- start start 2023-10-16T14:13:40.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bc007c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0bb80b3070 0x7f0bb80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bc007c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0bb80d1b10 0x7f0bb80cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bc007c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb80b3940 con 0x7f0bb80b3e70 2023-10-16T14:13:40.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bc007c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb80b45e0 con 0x7f0bb80d1b10 2023-10-16T14:13:40.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bc007c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb80b3b90 con 0x7f0bb80b3070 2023-10-16T14:13:40.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bbd617640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0bb80b3e70 0x7f0bb80c6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43338/0 (socket says 172.21.15.60:43338) 2023-10-16T14:13:40.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bbd617640 1 -- 172.21.15.60:0/956108979 learned_addr learned my addr 172.21.15.60:0/956108979 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:40.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.172+0000 7f0bbe619640 1 --2- 172.21.15.60:0/956108979 >> v2:172.21.15.60:3300/0 conn(0x7f0bb80d1b10 0x7f0bb80cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbde18640 1 --2- 172.21.15.60:0/956108979 >> v2:172.21.15.144:3300/0 conn(0x7f0bb80b3070 0x7f0bb80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbce16640 1 -- 172.21.15.60:0/956108979 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076837090 0 0) 0x7f0bb80b3940 con 0x7f0bb80b3e70 2023-10-16T14:13:40.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbce16640 1 -- 172.21.15.60:0/956108979 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba4003520 con 0x7f0bb80b3e70 2023-10-16T14:13:40.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbe619640 1 -- 172.21.15.60:0/956108979 >> v2:172.21.15.144:3300/0 conn(0x7f0bb80b3070 msgr2=0x7f0bb80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:40.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbe619640 1 --2- 172.21.15.60:0/956108979 >> v2:172.21.15.144:3300/0 conn(0x7f0bb80b3070 0x7f0bb80cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbe619640 1 -- 172.21.15.60:0/956108979 >> v1:172.21.15.60:6789/0 conn(0x7f0bb80b3e70 legacy=0x7f0bb80c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.173+0000 7f0bbe619640 1 -- 172.21.15.60:0/956108979 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80d3840 con 0x7f0bb80d1b10 2023-10-16T14:13:40.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.174+0000 7f0bbe619640 1 --2- 172.21.15.60:0/956108979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80d1b10 0x7f0bb80cfa40 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f0bb808ba60 tx=0x7f0bb002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a95768daadf4bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.174+0000 7f0bbce16640 1 -- 172.21.15.60:0/956108979 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb002cc80 con 0x7f0bb80d1b10 2023-10-16T14:13:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bbce16640 1 -- 172.21.15.60:0/956108979 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb002cde0 con 0x7f0bb80d1b10 2023-10-16T14:13:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bbce16640 1 -- 172.21.15.60:0/956108979 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0034660 con 0x7f0bb80d1b10 2023-10-16T14:13:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bc007c640 1 -- 172.21.15.60:0/956108979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80d1b10 msgr2=0x7f0bb80cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bc007c640 1 --2- 172.21.15.60:0/956108979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80d1b10 0x7f0bb80cfa40 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f0bb808ba60 tx=0x7f0bb002bed0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bc007c640 1 -- 172.21.15.60:0/956108979 shutdown_connections 2023-10-16T14:13:40.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bc007c640 1 --2- 172.21.15.60:0/956108979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80d1b10 0x7f0bb80cfa40 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bc007c640 1 --2- 172.21.15.60:0/956108979 >> v2:172.21.15.144:3300/0 conn(0x7f0bb80b3070 0x7f0bb80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.175+0000 7f0bc007c640 1 -- 172.21.15.60:0/956108979 >> 172.21.15.60:0/956108979 conn(0x7f0bb80d2d20 msgr2=0x7f0bb80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:40.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.176+0000 7f0bc007c640 1 -- 172.21.15.60:0/956108979 shutdown_connections 2023-10-16T14:13:40.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.176+0000 7f0bc007c640 1 -- 172.21.15.60:0/956108979 wait complete. 2023-10-16T14:13:40.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.177+0000 7f0bc007c640 1 Processor -- start 2023-10-16T14:13:40.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.177+0000 7f0bc007c640 1 -- start start 2023-10-16T14:13:40.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.178+0000 7f0bc007c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b3070 0x7f0bb8153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.178+0000 7f0bc007c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb80b3e70 0x7f0bb80b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.178+0000 7f0bc007c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb80d1b10 0x7f0bb80cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.178+0000 7f0bc007c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81e6ad0 con 0x7f0bb80b3070 2023-10-16T14:13:40.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.178+0000 7f0bc007c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81e5be0 con 0x7f0bb80b3e70 2023-10-16T14:13:40.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.178+0000 7f0bc007c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81e5ec0 con 0x7f0bb80d1b10 2023-10-16T14:13:40.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b3070 0x7f0bb8153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b3070 0x7f0bb8153130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55164/0 (socket says 172.21.15.60:55164) 2023-10-16T14:13:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 -- 172.21.15.60:0/2857036638 learned_addr learned my addr 172.21.15.60:0/2857036638 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbd617640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb80b3e70 0x7f0bb80b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 -- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb80d1b10 msgr2=0x7f0bb80cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0bb80d1b10 0x7f0bb80cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 -- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb80b3e70 msgr2=0x7f0bb80b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb80b3e70 0x7f0bb80b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbde18640 1 -- 172.21.15.60:0/2857036638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb0009a70 con 0x7f0bb80b3070 2023-10-16T14:13:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.179+0000 7f0bbd617640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0bb80b3e70 0x7f0bb80b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.180+0000 7f0bbde18640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b3070 0x7f0bb8153130 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400d800 tx=0x7f0bb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.180+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400b6c0 con 0x7f0bb80b3070 2023-10-16T14:13:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.180+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb400b820 con 0x7f0bb80b3070 2023-10-16T14:13:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.180+0000 7f0bc007c640 1 -- 172.21.15.60:0/2857036638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb81e61a0 con 0x7f0bb80b3070 2023-10-16T14:13:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.180+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400bbd0 con 0x7f0bb80b3070 2023-10-16T14:13:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.180+0000 7f0bc007c640 1 -- 172.21.15.60:0/2857036638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb81e6760 con 0x7f0bb80b3070 2023-10-16T14:13:40.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.182+0000 7f0bc007c640 1 -- 172.21.15.60:0/2857036638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb804e160 con 0x7f0bb80b3070 2023-10-16T14:13:40.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.182+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 158) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f0bb400bd30 con 0x7f0bb80b3070 2023-10-16T14:13:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.184+0000 7f0bae7fc640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f0b9c06b070 0x7f0b9c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.184+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0bb408a0f0 con 0x7f0bb80b3070 2023-10-16T14:13:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.184+0000 7f0bbd617640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f0b9c06b070 0x7f0b9c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.185+0000 7f0bbd617640 1 --2- 172.21.15.60:0/2857036638 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f0b9c06b070 0x7f0b9c06d530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80c67c0 tx=0x7f0ba8009040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.189+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bb4054d30 con 0x7f0bb80b3070 2023-10-16T14:13:40.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.406+0000 7f0bc007c640 1 -- 172.21.15.60:0/2857036638 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv1", "group": "mygrp", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bb80d3b20 con 0x7f0b9c06b070 2023-10-16T14:13:40.414 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:40.411+0000 7f8624f6d640 -1 mgr.server reply reply (22) Invalid argument No such subvol: mygrp::sv1 2023-10-16T14:13:40.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.411+0000 7f0bae7fc640 1 -- 172.21.15.60:0/2857036638 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: -22 No such subvol: mygrp::sv1) v1 ==== 34+0+0 (secure 0 0 0) 0x7f0ba8001d20 con 0x7f0b9c06b070 2023-10-16T14:13:40.415 INFO:teuthology.orchestra.run.smithi060.stderr:Error EINVAL: No such subvol: mygrp::sv1 2023-10-16T14:13:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T14:13:40.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --subvol sv1 . 1m 2023-10-16T14:13:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.652+0000 7f1e24ad1640 1 Processor -- start 2023-10-16T14:13:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.652+0000 7f1e24ad1640 1 -- start start 2023-10-16T14:13:40.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.653+0000 7f1e24ad1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200b8480 0x7f1e200cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.653+0000 7f1e24ad1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e200d0be0 0x7f1e200cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.653+0000 7f1e24ad1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2015d150 con 0x7f1e2015cd70 2023-10-16T14:13:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.653+0000 7f1e24ad1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e200d1180 con 0x7f1e200d0be0 2023-10-16T14:13:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.653+0000 7f1e24ad1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200d1460 con 0x7f1e200b8480 2023-10-16T14:13:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.653+0000 7f1e1ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e200d0be0 0x7f1e200cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e200d0be0 0x7f1e200cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55172/0 (socket says 172.21.15.60:55172) 2023-10-16T14:13:40.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1ffff640 1 -- 172.21.15.60:0/617024870 learned_addr learned my addr 172.21.15.60:0/617024870 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:40.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1effd640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200b8480 0x7f1e200cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1ffff640 1 -- 172.21.15.60:0/617024870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200b8480 msgr2=0x7f1e200cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:40.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1ffff640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200b8480 0x7f1e200cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1ffff640 1 -- 172.21.15.60:0/617024870 >> v1:172.21.15.144:6789/0 conn(0x7f1e2015cd70 legacy=0x7f1e200d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1ffff640 1 -- 172.21.15.60:0/617024870 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200d1740 con 0x7f1e200d0be0 2023-10-16T14:13:40.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.654+0000 7f1e1effd640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200b8480 0x7f1e200cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:40.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.656+0000 7f1e1ffff640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e200d0be0 0x7f1e200cdd00 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009870 tx=0x7f1e1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc99f2c08bc4965 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:40.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.656+0000 7f1e1e7fc640 1 -- 172.21.15.60:0/617024870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18030840 con 0x7f1e200d0be0 2023-10-16T14:13:40.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.656+0000 7f1e1e7fc640 1 -- 172.21.15.60:0/617024870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e180309a0 con 0x7f1e200d0be0 2023-10-16T14:13:40.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.656+0000 7f1e1e7fc640 1 -- 172.21.15.60:0/617024870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18030d30 con 0x7f1e200d0be0 2023-10-16T14:13:40.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 -- 172.21.15.60:0/617024870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e200d0be0 msgr2=0x7f1e200cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e200d0be0 0x7f1e200cdd00 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009870 tx=0x7f1e1802b980 comp rx=0 tx=0).stop 2023-10-16T14:13:40.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 -- 172.21.15.60:0/617024870 shutdown_connections 2023-10-16T14:13:40.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e200d0be0 0x7f1e200cdd00 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/617024870 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200b8480 0x7f1e200cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 -- 172.21.15.60:0/617024870 >> 172.21.15.60:0/617024870 conn(0x7f1e200cf570 msgr2=0x7f1e200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 -- 172.21.15.60:0/617024870 shutdown_connections 2023-10-16T14:13:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.657+0000 7f1e24ad1640 1 -- 172.21.15.60:0/617024870 wait complete. 2023-10-16T14:13:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.659+0000 7f1e24ad1640 1 Processor -- start 2023-10-16T14:13:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.659+0000 7f1e24ad1640 1 -- start start 2023-10-16T14:13:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e24ad1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e200b8480 0x7f1e201f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e24ad1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200d0be0 0x7f1e200b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e24ad1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 0x7f1e200c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e24ad1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e201f6bc0 con 0x7f1e2015cd70 2023-10-16T14:13:40.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e24ad1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200b2cb0 con 0x7f1e200b8480 2023-10-16T14:13:40.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e24ad1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200b2f50 con 0x7f1e200d0be0 2023-10-16T14:13:40.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 0x7f1e200c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 0x7f1e200c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55188/0 (socket says 172.21.15.60:55188) 2023-10-16T14:13:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.660+0000 7f1e1ffff640 1 -- 172.21.15.60:0/164992204 learned_addr learned my addr 172.21.15.60:0/164992204 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1f7fe640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e200b8480 0x7f1e201f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1effd640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200d0be0 0x7f1e200b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1ffff640 1 -- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200d0be0 msgr2=0x7f1e200b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1ffff640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200d0be0 0x7f1e200b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1ffff640 1 -- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e200b8480 msgr2=0x7f1e201f6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:40.674 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1ffff640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e200b8480 0x7f1e201f6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:40.675 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1ffff640 1 -- 172.21.15.60:0/164992204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180094d0 con 0x7f1e2015cd70 2023-10-16T14:13:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1f7fe640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e200b8480 0x7f1e201f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.661+0000 7f1e1ffff640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 0x7f1e200c8b00 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009870 tx=0x7f1e18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.662+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18039c90 con 0x7f1e2015cd70 2023-10-16T14:13:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.662+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e18039e10 con 0x7f1e2015cd70 2023-10-16T14:13:40.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.662+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e200b3230 con 0x7f1e2015cd70 2023-10-16T14:13:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.662+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18043640 con 0x7f1e2015cd70 2023-10-16T14:13:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.662+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e200b37f0 con 0x7f1e2015cd70 2023-10-16T14:13:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.663+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2004e9b0 con 0x7f1e2015cd70 2023-10-16T14:13:40.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.664+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 158) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f1e180437a0 con 0x7f1e2015cd70 2023-10-16T14:13:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.665+0000 7f1e17fff640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f1e0006b070 0x7f1e0006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:40.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.665+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f1e1803e070 con 0x7f1e2015cd70 2023-10-16T14:13:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.666+0000 7f1e1f7fe640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f1e0006b070 0x7f1e0006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.667+0000 7f1e1f7fe640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f1e0006b070 0x7f1e0006d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1e10001d70 tx=0x7f1e10009040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:40.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.670+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e180777b0 con 0x7f1e2015cd70 2023-10-16T14:13:40.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:40.892+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "subvol": "sv1", "path": ".", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e200d1d00 con 0x7f1e0006b070 2023-10-16T14:13:41.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.121+0000 7f1e17fff640 1 -- 172.21.15.60:0/164992204 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f1e10006b50 con 0x7f1e0006b070 2023-10-16T14:13:41.125 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule removed for path /volumes/_nogroup/sv1/140d4b27-ec8c-4bfc-b3ad-4c0bb1ce025b/.. 2023-10-16T14:13:41.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.124+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f1e0006b070 msgr2=0x7f1e0006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.124+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f1e0006b070 0x7f1e0006d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1e10001d70 tx=0x7f1e10009040 comp rx=0 tx=0).stop 2023-10-16T14:13:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.124+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 msgr2=0x7f1e200c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.124+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 0x7f1e200c8b00 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009870 tx=0x7f1e18004730 comp rx=0 tx=0).stop 2023-10-16T14:13:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 shutdown_connections 2023-10-16T14:13:41.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f1e0006b070 0x7f1e0006d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e200d0be0 0x7f1e200b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2015cd70 0x7f1e200c8b00 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 --2- 172.21.15.60:0/164992204 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e200b8480 0x7f1e201f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 >> 172.21.15.60:0/164992204 conn(0x7f1e200cf570 msgr2=0x7f1e200b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:41.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 shutdown_connections 2023-10-16T14:13:41.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.125+0000 7f1e24ad1640 1 -- 172.21.15.60:0/164992204 wait complete. 2023-10-16T14:13:41.153 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /volumes/_nogroup/sv1/140d4b27-ec8c-4bfc-b3ad-4c0bb1ce025b/.. 2023-10-16T14:13:41.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2023-10-16T14:13:41.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.380+0000 7f6f8d8a2640 1 Processor -- start 2023-10-16T14:13:41.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.381+0000 7f6f8d8a2640 1 -- start start 2023-10-16T14:13:41.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f8d8a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f880addb0 0x7f6f880caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f8d8a2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f880c9000 0x7f6f88069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f8d8a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8806a6f0 con 0x7f6f8806a2e0 2023-10-16T14:13:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f8d8a2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f88161a50 con 0x7f6f880c9000 2023-10-16T14:13:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f8d8a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f880cb140 con 0x7f6f880addb0 2023-10-16T14:13:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f867fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f880c9000 0x7f6f88069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f867fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f880c9000 0x7f6f88069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55198/0 (socket says 172.21.15.60:55198) 2023-10-16T14:13:41.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.383+0000 7f6f867fc640 1 -- 172.21.15.60:0/2756378044 learned_addr learned my addr 172.21.15.60:0/2756378044 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:41.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.384+0000 7f6f86ffd640 1 --2- 172.21.15.60:0/2756378044 >> v2:172.21.15.144:3300/0 conn(0x7f6f880addb0 0x7f6f880caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.384+0000 7f6f867fc640 1 -- 172.21.15.60:0/2756378044 >> v2:172.21.15.144:3300/0 conn(0x7f6f880addb0 msgr2=0x7f6f880caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:41.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.384+0000 7f6f867fc640 1 --2- 172.21.15.60:0/2756378044 >> v2:172.21.15.144:3300/0 conn(0x7f6f880addb0 0x7f6f880caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.384+0000 7f6f867fc640 1 -- 172.21.15.60:0/2756378044 >> v1:172.21.15.144:6789/0 conn(0x7f6f8806a2e0 legacy=0x7f6f880cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:41.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.384+0000 7f6f867fc640 1 -- 172.21.15.60:0/2756378044 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8806b030 con 0x7f6f880c9000 2023-10-16T14:13:41.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.385+0000 7f6f867fc640 1 --2- 172.21.15.60:0/2756378044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f880c9000 0x7f6f88069da0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f6f880b15a0 tx=0x7f6f7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70163f6fc8e66198 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:41.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.385+0000 7f6f85ffb640 1 -- 172.21.15.60:0/2756378044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c031870 con 0x7f6f880c9000 2023-10-16T14:13:41.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f85ffb640 1 -- 172.21.15.60:0/2756378044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c0319d0 con 0x7f6f880c9000 2023-10-16T14:13:41.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f85ffb640 1 -- 172.21.15.60:0/2756378044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c031d60 con 0x7f6f880c9000 2023-10-16T14:13:41.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/2756378044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f880c9000 msgr2=0x7f6f88069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/2756378044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f880c9000 0x7f6f88069da0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f6f880b15a0 tx=0x7f6f7c02b910 comp rx=0 tx=0).stop 2023-10-16T14:13:41.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/2756378044 shutdown_connections 2023-10-16T14:13:41.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/2756378044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f880c9000 0x7f6f88069da0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/2756378044 >> v2:172.21.15.144:3300/0 conn(0x7f6f880addb0 0x7f6f880caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/2756378044 >> 172.21.15.60:0/2756378044 conn(0x7f6f880ce9b0 msgr2=0x7f6f880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:41.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/2756378044 shutdown_connections 2023-10-16T14:13:41.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.386+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/2756378044 wait complete. 2023-10-16T14:13:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.388+0000 7f6f8d8a2640 1 Processor -- start 2023-10-16T14:13:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.388+0000 7f6f8d8a2640 1 -- start start 2023-10-16T14:13:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.389+0000 7f6f8d8a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 0x7f6f88158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.389+0000 7f6f8d8a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f880addb0 0x7f6f880b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.389+0000 7f6f8d8a2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f880c9000 0x7f6f880c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.389+0000 7f6f8d8a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c6d10 con 0x7f6f8806a2e0 2023-10-16T14:13:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.389+0000 7f6f8d8a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c6ff0 con 0x7f6f880addb0 2023-10-16T14:13:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.389+0000 7f6f8d8a2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c72d0 con 0x7f6f880c9000 2023-10-16T14:13:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 0x7f6f88158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 0x7f6f88158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55208/0 (socket says 172.21.15.60:55208) 2023-10-16T14:13:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 -- 172.21.15.60:0/3637527299 learned_addr learned my addr 172.21.15.60:0/3637527299 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 -- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f880c9000 msgr2=0x7f6f880c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f880c9000 0x7f6f880c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 -- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f880addb0 msgr2=0x7f6f880b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f880addb0 0x7f6f880b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 -- 172.21.15.60:0/3637527299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0094d0 con 0x7f6f8806a2e0 2023-10-16T14:13:41.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.390+0000 7f6f86ffd640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 0x7f6f88158c30 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f6f78007ef0 tx=0x7f6f7800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:41.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.391+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78014620 con 0x7f6f8806a2e0 2023-10-16T14:13:41.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.391+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f78014780 con 0x7f6f8806a2e0 2023-10-16T14:13:41.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.391+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78014b10 con 0x7f6f8806a2e0 2023-10-16T14:13:41.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.391+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f880c75b0 con 0x7f6f8806a2e0 2023-10-16T14:13:41.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.391+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f880b29b0 con 0x7f6f8806a2e0 2023-10-16T14:13:41.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.392+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8804e160 con 0x7f6f8806a2e0 2023-10-16T14:13:41.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.393+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 158) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f6f78014c70 con 0x7f6f8806a2e0 2023-10-16T14:13:41.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.394+0000 7f6f6ffff640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6f7406b070 0x7f6f7406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.394+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6f7808a5c0 con 0x7f6f8806a2e0 2023-10-16T14:13:41.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.394+0000 7f6f867fc640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6f7406b070 0x7f6f7406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.396+0000 7f6f867fc640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6f7406b070 0x7f6f7406d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c036000 tx=0x7f6f7c030000 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:41.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.398+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f780551f0 con 0x7f6f8806a2e0 2023-10-16T14:13:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.628+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f880c7160 con 0x7f6f7406b070 2023-10-16T14:13:41.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.661+0000 7f6f6ffff640 1 -- 172.21.15.60:0/3637527299 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f880c7160 con 0x7f6f7406b070 2023-10-16T14:13:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6f7406b070 msgr2=0x7f6f7406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6f7406b070 0x7f6f7406d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c036000 tx=0x7f6f7c030000 comp rx=0 tx=0).stop 2023-10-16T14:13:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 msgr2=0x7f6f88158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 0x7f6f88158c30 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f6f78007ef0 tx=0x7f6f7800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 shutdown_connections 2023-10-16T14:13:41.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6f7406b070 0x7f6f7406d530 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6f880c9000 0x7f6f880c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f880addb0 0x7f6f880b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 --2- 172.21.15.60:0/3637527299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f8806a2e0 0x7f6f88158c30 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 >> 172.21.15.60:0/3637527299 conn(0x7f6f880ce9b0 msgr2=0x7f6f880cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:41.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 shutdown_connections 2023-10-16T14:13:41.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.664+0000 7f6f8d8a2640 1 -- 172.21.15.60:0/3637527299 wait complete. 2023-10-16T14:13:41.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs mygrp 2023-10-16T14:13:41.693 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:41.690+0000 7f8619716640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:41.694 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:41.690+0000 7f8619716640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:41.694 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:41.690+0000 7f8619716640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:41.694 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:41.690+0000 7f8619716640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:41.694 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:41.690+0000 7f8619716640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:41.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.903+0000 7f66bdca2640 1 Processor -- start 2023-10-16T14:13:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.903+0000 7f66bdca2640 1 -- start start 2023-10-16T14:13:41.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.904+0000 7f66bdca2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b815ad70 0x7f66b8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66bdca2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f66b80caac0 0x7f66b806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66bdca2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b80cb030 con 0x7f66b80c9ca0 2023-10-16T14:13:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66bdca2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b80c91c0 con 0x7f66b80caac0 2023-10-16T14:13:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66bdca2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80c94a0 con 0x7f66b815ad70 2023-10-16T14:13:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66b6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f66b80c9ca0 0x7f66b80ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43348/0 (socket says 172.21.15.60:43348) 2023-10-16T14:13:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66b6ffd640 1 -- 172.21.15.60:0/153865539 learned_addr learned my addr 172.21.15.60:0/153865539 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:41.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66b7fff640 1 --2- 172.21.15.60:0/153865539 >> v2:172.21.15.60:3300/0 conn(0x7f66b80caac0 0x7f66b806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.905+0000 7f66b77fe640 1 --2- 172.21.15.60:0/153865539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b815ad70 0x7f66b8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.906+0000 7f66b67fc640 1 -- 172.21.15.60:0/153865539 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302343571 0 0) 0x7f66b80cb030 con 0x7f66b80c9ca0 2023-10-16T14:13:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.906+0000 7f66b7fff640 1 -- 172.21.15.60:0/153865539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b815ad70 msgr2=0x7f66b8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:41.911 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.906+0000 7f66b7fff640 1 --2- 172.21.15.60:0/153865539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b815ad70 0x7f66b8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.906+0000 7f66b7fff640 1 -- 172.21.15.60:0/153865539 >> v1:172.21.15.60:6789/0 conn(0x7f66b80c9ca0 legacy=0x7f66b80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.912 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.906+0000 7f66b7fff640 1 -- 172.21.15.60:0/153865539 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b80c9780 con 0x7f66b80caac0 2023-10-16T14:13:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.907+0000 7f66b7fff640 1 --2- 172.21.15.60:0/153865539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b80caac0 0x7f66b806a2e0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009d90 tx=0x7f66b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de78cf0a212082 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.907+0000 7f66b67fc640 1 -- 172.21.15.60:0/153865539 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b002cc60 con 0x7f66b80caac0 2023-10-16T14:13:41.913 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.907+0000 7f66b67fc640 1 -- 172.21.15.60:0/153865539 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b002cdc0 con 0x7f66b80caac0 2023-10-16T14:13:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.907+0000 7f66b67fc640 1 -- 172.21.15.60:0/153865539 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0034660 con 0x7f66b80caac0 2023-10-16T14:13:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 -- 172.21.15.60:0/153865539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b80caac0 msgr2=0x7f66b806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 --2- 172.21.15.60:0/153865539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b80caac0 0x7f66b806a2e0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009d90 tx=0x7f66b002beb0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.914 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 -- 172.21.15.60:0/153865539 shutdown_connections 2023-10-16T14:13:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 --2- 172.21.15.60:0/153865539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b80caac0 0x7f66b806a2e0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 --2- 172.21.15.60:0/153865539 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b815ad70 0x7f66b8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 -- 172.21.15.60:0/153865539 >> 172.21.15.60:0/153865539 conn(0x7f66b80ce9b0 msgr2=0x7f66b80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 -- 172.21.15.60:0/153865539 shutdown_connections 2023-10-16T14:13:41.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.908+0000 7f66bdca2640 1 -- 172.21.15.60:0/153865539 wait complete. 2023-10-16T14:13:41.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.910+0000 7f66bdca2640 1 Processor -- start 2023-10-16T14:13:41.916 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.910+0000 7f66bdca2640 1 -- start start 2023-10-16T14:13:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66bdca2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66b80c9ca0 0x7f66b8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66bdca2640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b80caac0 0x7f66b80cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66bdca2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 0x7f66b814ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66bdca2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815a610 con 0x7f66b815ad70 2023-10-16T14:13:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66bdca2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b80cd870 con 0x7f66b80c9ca0 2023-10-16T14:13:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66bdca2640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b814d370 con 0x7f66b80caac0 2023-10-16T14:13:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 0x7f66b814ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 0x7f66b814ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55236/0 (socket says 172.21.15.60:55236) 2023-10-16T14:13:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.911+0000 7f66b7fff640 1 -- 172.21.15.60:0/3515807655 learned_addr learned my addr 172.21.15.60:0/3515807655 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b6ffd640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b80caac0 0x7f66b80cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b77fe640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66b80c9ca0 0x7f66b8159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b7fff640 1 -- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b80caac0 msgr2=0x7f66b80cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b7fff640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b80caac0 0x7f66b80cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b7fff640 1 -- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66b80c9ca0 msgr2=0x7f66b8159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b7fff640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66b80c9ca0 0x7f66b8159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b7fff640 1 -- 172.21.15.60:0/3515807655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b0009a70 con 0x7f66b815ad70 2023-10-16T14:13:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b77fe640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66b80c9ca0 0x7f66b8159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66b7fff640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 0x7f66b814ce30 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009eb0 tx=0x7f66b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:41.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0040650 con 0x7f66b815ad70 2023-10-16T14:13:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00407d0 con 0x7f66b815ad70 2023-10-16T14:13:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0040b80 con 0x7f66b815ad70 2023-10-16T14:13:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.912+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b81f2de0 con 0x7f66b815ad70 2023-10-16T14:13:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.913+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b81f33a0 con 0x7f66b815ad70 2023-10-16T14:13:41.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.914+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b80cb030 con 0x7f66b815ad70 2023-10-16T14:13:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.915+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 158) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f66b0040ce0 con 0x7f66b815ad70 2023-10-16T14:13:41.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.916+0000 7f669ffff640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f66a406b070 0x7f66a406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:41.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.916+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f66b00ac9b0 con 0x7f66b815ad70 2023-10-16T14:13:41.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.917+0000 7f66b77fe640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f66a406b070 0x7f66a406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:41.924 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.918+0000 7f66b77fe640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f66a406b070 0x7f66a406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f66a8006fd0 tx=0x7f66a8008040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:41.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:41.921+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66b00775e0 con 0x7f66b815ad70 2023-10-16T14:13:42.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.147+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "mygrp", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b80ad4d0 con 0x7f66a406b070 2023-10-16T14:13:42.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.174+0000 7f669ffff640 1 -- 172.21.15.60:0/3515807655 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f66a8009530 con 0x7f66a406b070 2023-10-16T14:13:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f66a406b070 msgr2=0x7f66a406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f66a406b070 0x7f66a406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f66a8006fd0 tx=0x7f66a8008040 comp rx=0 tx=0).stop 2023-10-16T14:13:42.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 msgr2=0x7f66b814ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 0x7f66b814ce30 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009eb0 tx=0x7f66b0004a40 comp rx=0 tx=0).stop 2023-10-16T14:13:42.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 shutdown_connections 2023-10-16T14:13:42.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f66a406b070 0x7f66a406d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f66b80caac0 0x7f66b80cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66b815ad70 0x7f66b814ce30 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 --2- 172.21.15.60:0/3515807655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66b80c9ca0 0x7f66b8159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.176+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 >> 172.21.15.60:0/3515807655 conn(0x7f66b80ce9b0 msgr2=0x7f66b815d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:42.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.177+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 shutdown_connections 2023-10-16T14:13:42.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.177+0000 7f66bdca2640 1 -- 172.21.15.60:0/3515807655 wait complete. 2023-10-16T14:13:42.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --group_name mygrp 2023-10-16T14:13:42.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.426+0000 7f059d173640 1 Processor -- start 2023-10-16T14:13:42.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.426+0000 7f059d173640 1 -- start start 2023-10-16T14:13:42.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.428+0000 7f059d173640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 0x7f0598070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.428+0000 7f059d173640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980ca3a0 con 0x7f0598070780 2023-10-16T14:13:42.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.428+0000 7f059d173640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980c9d70 con 0x7f05981573e0 2023-10-16T14:13:42.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.428+0000 7f059d173640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0598070c10 con 0x7f05980ca830 2023-10-16T14:13:42.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.428+0000 7f059c972640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0598070780 0x7f059806bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43360/0 (socket says 172.21.15.60:43360) 2023-10-16T14:13:42.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.428+0000 7f059c972640 1 -- 172.21.15.60:0/1698224802 learned_addr learned my addr 172.21.15.60:0/1698224802 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:42.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f05977fe640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 0x7f0598070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925118638 0 0) 0x7f05980ca3a0 con 0x7f0598070780 2023-10-16T14:13:42.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0580005730 con 0x7f0598070780 2023-10-16T14:13:42.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985011010 0 0) 0x7f0580005730 con 0x7f0598070780 2023-10-16T14:13:42.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 msgr2=0x7f0598070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:42.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 0x7f0598070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 >> v1:172.21.15.144:6789/0 conn(0x7f05981573e0 legacy=0x7f0598159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05981577c0 con 0x7f0598070780 2023-10-16T14:13:42.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f0590001670 con 0x7f0598070780 2023-10-16T14:13:42.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 >> v1:172.21.15.60:6789/0 conn(0x7f0598070780 legacy=0x7f059806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.429+0000 7f0596ffd640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 0x7f0580005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f0596ffd640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006180 0x7f05800069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f0596ffd640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0580006f10 0x7f0580007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0598070c10 con 0x7f0580006f10 2023-10-16T14:13:42.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058000dcd0 con 0x7f0580006180 2023-10-16T14:13:42.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058000de90 con 0x7f05980ca830 2023-10-16T14:13:42.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f059c972640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0580006f10 0x7f0580007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f0597fff640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 0x7f0580005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f05977fe640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006180 0x7f05800069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f059c972640 1 -- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 msgr2=0x7f0580005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:42.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f059c972640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05980ca830 0x7f0580005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f059c972640 1 -- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006180 msgr2=0x7f05800069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f059c972640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006180 0x7f05800069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f059c972640 1 -- 172.21.15.60:0/1698224802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580005730 con 0x7f0580006f10 2023-10-16T14:13:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.430+0000 7f05977fe640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006180 0x7f05800069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f059c972640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0580006f10 0x7f0580007390 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f059000c0e0 tx=0x7f0580007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c019717a8c8bc62 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590005ad0 con 0x7f0580006f10 2023-10-16T14:13:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0590005c30 con 0x7f0580006f10 2023-10-16T14:13:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f0596ffd640 1 -- 172.21.15.60:0/1698224802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059002c7d0 con 0x7f0580006f10 2023-10-16T14:13:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f059d173640 1 -- 172.21.15.60:0/1698224802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0580006f10 msgr2=0x7f0580007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f059d173640 1 --2- 172.21.15.60:0/1698224802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0580006f10 0x7f0580007390 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f059000c0e0 tx=0x7f0580007a00 comp rx=0 tx=0).stop 2023-10-16T14:13:42.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.432+0000 7f059c972640 1 -- 172.21.15.60:0/1698224802 reap_dead start 2023-10-16T14:13:42.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.433+0000 7f059d173640 1 -- 172.21.15.60:0/1698224802 shutdown_connections 2023-10-16T14:13:42.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.433+0000 7f059d173640 1 -- 172.21.15.60:0/1698224802 >> 172.21.15.60:0/1698224802 conn(0x7f05980cc230 msgr2=0x7f05980cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:42.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.433+0000 7f059d173640 1 -- 172.21.15.60:0/1698224802 shutdown_connections 2023-10-16T14:13:42.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.433+0000 7f059d173640 1 -- 172.21.15.60:0/1698224802 wait complete. 2023-10-16T14:13:42.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.434+0000 7f059d173640 1 Processor -- start 2023-10-16T14:13:42.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.435+0000 7f059d173640 1 -- start start 2023-10-16T14:13:42.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.435+0000 7f059d173640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006f10 0x7f05980ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f059d173640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 0x7f0598155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f059d173640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05981f5880 0x7f05981f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f059d173640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05981f65b0 con 0x7f0598155880 2023-10-16T14:13:42.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f059d173640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05981f1f90 con 0x7f0580006f10 2023-10-16T14:13:42.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f059d173640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0598156240 con 0x7f05981f5880 2023-10-16T14:13:42.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 0x7f0598155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 0x7f0598155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55256/0 (socket says 172.21.15.60:55256) 2023-10-16T14:13:42.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 -- 172.21.15.60:0/779854316 learned_addr learned my addr 172.21.15.60:0/779854316 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:42.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f0597fff640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006f10 0x7f05980ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f059c972640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05981f5880 0x7f05981f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 -- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05981f5880 msgr2=0x7f05981f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05981f5880 0x7f05981f5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.669 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 -- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006f10 msgr2=0x7f05980ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006f10 0x7f05980ad3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f05977fe640 1 -- 172.21.15.60:0/779854316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05981f2fa0 con 0x7f0598155880 2023-10-16T14:13:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.436+0000 7f0597fff640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006f10 0x7f05980ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.437+0000 7f05977fe640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 0x7f0598155d00 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f058c0108c0 tx=0x7f058c010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.437+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00eb00 con 0x7f0598155880 2023-10-16T14:13:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.437+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c00ec60 con 0x7f0598155880 2023-10-16T14:13:42.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.437+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c01c400 con 0x7f0598155880 2023-10-16T14:13:42.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.437+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05981f21e0 con 0x7f0598155880 2023-10-16T14:13:42.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.437+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05981f27a0 con 0x7f0598155880 2023-10-16T14:13:42.671 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.438+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059804e9b0 con 0x7f0598155880 2023-10-16T14:13:42.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.439+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 158) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f058c01c560 con 0x7f0598155880 2023-10-16T14:13:42.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.440+0000 7f0594ff9640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f058406b070 0x7f058406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:42.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.440+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f058c089ff0 con 0x7f0598155880 2023-10-16T14:13:42.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.441+0000 7f0597fff640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f058406b070 0x7f058406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:42.672 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.442+0000 7f0597fff640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f058406b070 0x7f058406d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0588007fd0 tx=0x7f0588009040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:42.673 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.445+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f058c0bcfe0 con 0x7f0598155880 2023-10-16T14:13:42.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.673+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "mygrp", "target": ["mon-mgr", ""]}) v1 -- 0x7f059815ac80 con 0x7f058406b070 2023-10-16T14:13:42.738 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.734+0000 7f0594ff9640 1 -- 172.21.15.60:0/779854316 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f058800d050 con 0x7f058406b070 2023-10-16T14:13:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.737+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f058406b070 msgr2=0x7f058406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.737+0000 7f059d173640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f058406b070 0x7f058406d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0588007fd0 tx=0x7f0588009040 comp rx=0 tx=0).stop 2023-10-16T14:13:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 msgr2=0x7f0598155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 0x7f0598155d00 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f058c0108c0 tx=0x7f058c010d80 comp rx=0 tx=0).stop 2023-10-16T14:13:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 shutdown_connections 2023-10-16T14:13:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f058406b070 0x7f058406d530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f05981f5880 0x7f05981f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0598155880 0x7f0598155d00 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 --2- 172.21.15.60:0/779854316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0580006f10 0x7f05980ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 >> 172.21.15.60:0/779854316 conn(0x7f05980cc230 msgr2=0x7f059806d190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 shutdown_connections 2023-10-16T14:13:42.744 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:42.738+0000 7f059d173640 1 -- 172.21.15.60:0/779854316 wait complete. 2023-10-16T14:13:42.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv1 . 1m 2023-10-16T14:13:43.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.000+0000 7fb5c4b68640 1 Processor -- start 2023-10-16T14:13:43.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.000+0000 7fb5c4b68640 1 -- start start 2023-10-16T14:13:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5c4b68640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5c00c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5c4b68640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c00c9620 con 0x7fb5c00b0360 2023-10-16T14:13:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5c4b68640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c00ad4a0 con 0x7fb5c015cf60 2023-10-16T14:13:43.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5c4b68640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c015d340 con 0x7fb5c00c9ab0 2023-10-16T14:13:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5bed9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb5c00b0360 0x7fb5c00ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43372/0 (socket says 172.21.15.60:43372) 2023-10-16T14:13:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5bed9d640 1 -- 172.21.15.60:0/3529599800 learned_addr learned my addr 172.21.15.60:0/3529599800 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.002+0000 7fb5bdd9b640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5c00c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961898145 0 0) 0x7fb5c00c9620 con 0x7fb5c00b0360 2023-10-16T14:13:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a8003520 con 0x7fb5c00b0360 2023-10-16T14:13:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790507237 0 0) 0x7fb5a8003520 con 0x7fb5c00b0360 2023-10-16T14:13:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 msgr2=0x7fb5c00c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5c00c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 >> v1:172.21.15.144:6789/0 conn(0x7fb5c015cf60 legacy=0x7fb5c006f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c00c8fd0 con 0x7fb5c00b0360 2023-10-16T14:13:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fb5b80019f0 con 0x7fb5c00b0360 2023-10-16T14:13:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 >> v1:172.21.15.60:6789/0 conn(0x7fb5c00b0360 legacy=0x7fb5c00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5a8003a10 0x7fb5a8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a80047c0 0x7fb5a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a800ba80 con 0x7fb5a8003a10 2023-10-16T14:13:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a800bbf0 con 0x7fb5a80047c0 2023-10-16T14:13:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.003+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a800bdb0 con 0x7fb5c00c9ab0 2023-10-16T14:13:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5be59c640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5a8003a10 0x7fb5a8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5bed9d640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5bdd9b640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a80047c0 0x7fb5a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5be59c640 1 -- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 msgr2=0x7fb5a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5be59c640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5a8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5be59c640 1 -- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a80047c0 msgr2=0x7fb5a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5be59c640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5a80047c0 0x7fb5a8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5be59c640 1 -- 172.21.15.60:0/3529599800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a8003520 con 0x7fb5a8003a10 2023-10-16T14:13:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.004+0000 7fb5bed9d640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5c00c9ab0 0x7fb5a8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.005+0000 7fb5be59c640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5a8003a10 0x7fb5a8003e30 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0001750 tx=0x7fb5a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d94793a871d623 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00052d0 con 0x7fb5a8003a10 2023-10-16T14:13:43.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b002e050 con 0x7fb5a8003a10 2023-10-16T14:13:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5bd59a640 1 -- 172.21.15.60:0/3529599800 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00035f0 con 0x7fb5a8003a10 2023-10-16T14:13:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5c4b68640 1 -- 172.21.15.60:0/3529599800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5a8003a10 msgr2=0x7fb5a8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5c4b68640 1 --2- 172.21.15.60:0/3529599800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5a8003a10 0x7fb5a8003e30 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0001750 tx=0x7fb5a80057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5bed9d640 1 -- 172.21.15.60:0/3529599800 reap_dead start 2023-10-16T14:13:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.006+0000 7fb5c4b68640 1 -- 172.21.15.60:0/3529599800 shutdown_connections 2023-10-16T14:13:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.007+0000 7fb5c4b68640 1 -- 172.21.15.60:0/3529599800 >> 172.21.15.60:0/3529599800 conn(0x7fb5c00cb360 msgr2=0x7fb5c00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.007+0000 7fb5c4b68640 1 -- 172.21.15.60:0/3529599800 shutdown_connections 2023-10-16T14:13:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.007+0000 7fb5c4b68640 1 -- 172.21.15.60:0/3529599800 wait complete. 2023-10-16T14:13:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.008+0000 7fb5c4b68640 1 Processor -- start 2023-10-16T14:13:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.009+0000 7fb5c4b68640 1 -- start start 2023-10-16T14:13:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5c4b68640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a8003a10 0x7fb5c0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5c4b68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c00c83e0 0x7fb5c01f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5c4b68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5c00c4590 0x7fb5c00c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5c4b68640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c00c3e70 con 0x7fb5c00c83e0 2023-10-16T14:13:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5c4b68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c00c4f50 con 0x7fb5c00c4590 2023-10-16T14:13:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5c4b68640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c00c32f0 con 0x7fb5a8003a10 2023-10-16T14:13:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c00c83e0 0x7fb5c01f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c00c83e0 0x7fb5c01f2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55268/0 (socket says 172.21.15.60:55268) 2023-10-16T14:13:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 -- 172.21.15.60:0/338880825 learned_addr learned my addr 172.21.15.60:0/338880825 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bed9d640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5c00c4590 0x7fb5c00c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5be59c640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a8003a10 0x7fb5c0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 -- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a8003a10 msgr2=0x7fb5c0150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a8003a10 0x7fb5c0150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 -- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5c00c4590 msgr2=0x7fb5c00c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.010+0000 7fb5bdd9b640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5c00c4590 0x7fb5c00c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5bdd9b640 1 -- 172.21.15.60:0/338880825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c00c3540 con 0x7fb5c00c83e0 2023-10-16T14:13:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5be59c640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb5a8003a10 0x7fb5c0150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5bdd9b640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5c00c83e0 0x7fb5c01f2050 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4010860 tx=0x7fb5b4010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4009610 con 0x7fb5c00c83e0 2023-10-16T14:13:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b400eb60 con 0x7fb5c00c83e0 2023-10-16T14:13:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5c4b68640 1 -- 172.21.15.60:0/338880825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c00c3820 con 0x7fb5c00c83e0 2023-10-16T14:13:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b401c400 con 0x7fb5c00c83e0 2023-10-16T14:13:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.011+0000 7fb5c4b68640 1 -- 172.21.15.60:0/338880825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c0150fd0 con 0x7fb5c00c83e0 2023-10-16T14:13:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.012+0000 7fb5c4b68640 1 -- 172.21.15.60:0/338880825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c004e160 con 0x7fb5c00c83e0 2023-10-16T14:13:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.014+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 159) v1 ==== 89507+0+0 (secure 0 0 0) 0x7fb5b401c560 con 0x7fb5c00c83e0 2023-10-16T14:13:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.014+0000 7fb5aeffd640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fb5a006b1c0 0x7fb5a006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.014+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fb5b408a060 con 0x7fb5c00c83e0 2023-10-16T14:13:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.015+0000 7fb5be59c640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fb5a006b1c0 0x7fb5a006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.016+0000 7fb5be59c640 1 --2- 172.21.15.60:0/338880825 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fb5a006b1c0 0x7fb5a006d680 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0033000 tx=0x7fb5b0009030 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:43.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.019+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5b40bcfe0 con 0x7fb5c00c83e0 2023-10-16T14:13:43.239 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.236+0000 7fb5c4b68640 1 -- 172.21.15.60:0/338880825 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv1", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5c01511c0 con 0x7fb5a006b1c0 2023-10-16T14:13:43.252 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:43.249+0000 7f8624f6d640 -1 mgr.server reply reply (22) Invalid argument No such subvol: None::sv1 2023-10-16T14:13:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.249+0000 7fb5aeffd640 1 -- 172.21.15.60:0/338880825 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: -22 No such subvol: None::sv1) v1 ==== 33+0+0 (secure 0 0 0) 0x7fb5c01511c0 con 0x7fb5a006b1c0 2023-10-16T14:13:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:Error EINVAL: No such subvol: None::sv1 2023-10-16T14:13:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T14:13:43.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --group mygrp . 1m 2023-10-16T14:13:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.487+0000 7fe66b4d1640 1 Processor -- start 2023-10-16T14:13:43.491 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.487+0000 7fe66b4d1640 1 -- start start 2023-10-16T14:13:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.488+0000 7fe66b4d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640b8480 0x7fe6640cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.488+0000 7fe66b4d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6640d0be0 0x7fe6640cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.488+0000 7fe66b4d1640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66415d150 con 0x7fe66415cd70 2023-10-16T14:13:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.488+0000 7fe66b4d1640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6640d1180 con 0x7fe6640d0be0 2023-10-16T14:13:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.488+0000 7fe66b4d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640d1460 con 0x7fe6640b8480 2023-10-16T14:13:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.489+0000 7fe66acd0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6640d0be0 0x7fe6640cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.489+0000 7fe669cce640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640b8480 0x7fe6640cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.489+0000 7fe66acd0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6640d0be0 0x7fe6640cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.60:36926/0 (socket says 172.21.15.60:36926) 2023-10-16T14:13:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.489+0000 7fe66acd0640 1 -- 172.21.15.60:0/1115507904 learned_addr learned my addr 172.21.15.60:0/1115507904 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.490+0000 7fe66acd0640 1 -- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640b8480 msgr2=0x7fe6640cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.490+0000 7fe66acd0640 1 --2- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640b8480 0x7fe6640cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.490+0000 7fe66acd0640 1 -- 172.21.15.60:0/1115507904 >> v1:172.21.15.144:6789/0 conn(0x7fe66415cd70 legacy=0x7fe6640d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.490+0000 7fe66acd0640 1 -- 172.21.15.60:0/1115507904 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640d1740 con 0x7fe6640d0be0 2023-10-16T14:13:43.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.492+0000 7fe66acd0640 1 --2- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640d0be0 0x7fe6640cdd00 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009870 tx=0x7fe65c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e6f78fd772364b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:43.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.492+0000 7fe6694cd640 1 -- 172.21.15.60:0/1115507904 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c030840 con 0x7fe6640d0be0 2023-10-16T14:13:43.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.492+0000 7fe6694cd640 1 -- 172.21.15.60:0/1115507904 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c0309a0 con 0x7fe6640d0be0 2023-10-16T14:13:43.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.492+0000 7fe6694cd640 1 -- 172.21.15.60:0/1115507904 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c030d30 con 0x7fe6640d0be0 2023-10-16T14:13:43.501 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 -- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640d0be0 msgr2=0x7fe6640cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640d0be0 0x7fe6640cdd00 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009870 tx=0x7fe65c02b980 comp rx=0 tx=0).stop 2023-10-16T14:13:43.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 -- 172.21.15.60:0/1115507904 shutdown_connections 2023-10-16T14:13:43.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640d0be0 0x7fe6640cdd00 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/1115507904 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640b8480 0x7fe6640cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.502 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 -- 172.21.15.60:0/1115507904 >> 172.21.15.60:0/1115507904 conn(0x7fe6640cf570 msgr2=0x7fe6640cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 -- 172.21.15.60:0/1115507904 shutdown_connections 2023-10-16T14:13:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.493+0000 7fe66b4d1640 1 -- 172.21.15.60:0/1115507904 wait complete. 2023-10-16T14:13:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.495+0000 7fe66b4d1640 1 Processor -- start 2023-10-16T14:13:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.495+0000 7fe66b4d1640 1 -- start start 2023-10-16T14:13:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66b4d1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640b8480 0x7fe6641f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66b4d1640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640d0be0 0x7fe6641f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66b4d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 0x7fe6641f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66b4d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640c6700 con 0x7fe66415cd70 2023-10-16T14:13:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66b4d1640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6641f61a0 con 0x7fe6640b8480 2023-10-16T14:13:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66b4d1640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6641f6480 con 0x7fe6640d0be0 2023-10-16T14:13:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66acd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 0x7fe6641f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.504 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66acd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 0x7fe6641f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55276/0 (socket says 172.21.15.60:55276) 2023-10-16T14:13:43.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.496+0000 7fe66acd0640 1 -- 172.21.15.60:0/837855796 learned_addr learned my addr 172.21.15.60:0/837855796 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:43.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66a4cf640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640b8480 0x7fe6641f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe669cce640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640d0be0 0x7fe6641f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66acd0640 1 -- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640d0be0 msgr2=0x7fe6641f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.505 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66acd0640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640d0be0 0x7fe6641f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66acd0640 1 -- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640b8480 msgr2=0x7fe6641f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66acd0640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640b8480 0x7fe6641f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66acd0640 1 -- 172.21.15.60:0/837855796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0094d0 con 0x7fe66415cd70 2023-10-16T14:13:43.506 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.497+0000 7fe66acd0640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 0x7fe6641f5b70 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009870 tx=0x7fe65c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:43.507 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.498+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c039a90 con 0x7fe66415cd70 2023-10-16T14:13:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.498+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c039c10 con 0x7fe66415cd70 2023-10-16T14:13:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.498+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c043410 con 0x7fe66415cd70 2023-10-16T14:13:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.498+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66415e430 con 0x7fe66415cd70 2023-10-16T14:13:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.498+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66415e9f0 con 0x7fe66415cd70 2023-10-16T14:13:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.499+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66404e9b0 con 0x7fe66415cd70 2023-10-16T14:13:43.508 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.500+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 159) v1 ==== 89507+0+0 (secure 0 0 0) 0x7fe65c02f390 con 0x7fe66415cd70 2023-10-16T14:13:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.501+0000 7fe662ffd640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fe65006b190 0x7fe65006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.501+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe65c0abe40 con 0x7fe66415cd70 2023-10-16T14:13:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.502+0000 7fe66a4cf640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fe65006b190 0x7fe65006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:43.509 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.503+0000 7fe66a4cf640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fe65006b190 0x7fe65006d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe6640ca690 tx=0x7fe654009040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:43.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.506+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe65c076990 con 0x7fe66415cd70 2023-10-16T14:13:43.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.722+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 --> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "group": "mygrp", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6640d1d00 con 0x7fe65006b190 2023-10-16T14:13:43.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.911+0000 7fe662ffd640 1 -- 172.21.15.60:0/837855796 <== mgr.8915 v2:172.21.15.144:6832/1784354286 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe654001a20 con 0x7fe65006b190 2023-10-16T14:13:43.915 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path . 2023-10-16T14:13:43.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fe65006b190 msgr2=0x7fe65006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.918 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fe65006b190 0x7fe65006d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe6640ca690 tx=0x7fe654009040 comp rx=0 tx=0).stop 2023-10-16T14:13:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 msgr2=0x7fe6641f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 0x7fe6641f5b70 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009870 tx=0x7fe65c004730 comp rx=0 tx=0).stop 2023-10-16T14:13:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 shutdown_connections 2023-10-16T14:13:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7fe65006b190 0x7fe65006d650 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe6640d0be0 0x7fe6641f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe66415cd70 0x7fe6641f5b70 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 --2- 172.21.15.60:0/837855796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6640b8480 0x7fe6641f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:43.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.914+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 >> 172.21.15.60:0/837855796 conn(0x7fe6640cf570 msgr2=0x7fe6640d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:43.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.915+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 shutdown_connections 2023-10-16T14:13:43.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:43.915+0000 7fe66b4d1640 1 -- 172.21.15.60:0/837855796 wait complete. 2023-10-16T14:13:43.940 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path . 2023-10-16T14:13:43.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T14:13:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.158+0000 7f6dc467c640 1 Processor -- start 2023-10-16T14:13:44.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.158+0000 7f6dc467c640 1 -- start start 2023-10-16T14:13:44.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc467c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc467c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6dbc160b90 0x7f6dbc160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc467c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0ac890 con 0x7f6dbc069da0 2023-10-16T14:13:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc467c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dbc15d530 con 0x7f6dbc160b90 2023-10-16T14:13:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc467c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc161650 con 0x7f6dbc15d150 2023-10-16T14:13:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc1c17640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6dbc160b90 0x7f6dbc160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc2c19640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6dbc069da0 0x7f6dbc06a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43380/0 (socket says 172.21.15.60:43380) 2023-10-16T14:13:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc1c17640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6dbc160b90 0x7f6dbc160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55282/0 (socket says 172.21.15.60:55282) 2023-10-16T14:13:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.160+0000 7f6dc2c19640 1 -- 172.21.15.60:0/2094178710 learned_addr learned my addr 172.21.15.60:0/2094178710 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.161+0000 7f6dc2418640 1 --2- 172.21.15.60:0/2094178710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.161+0000 7f6dc1c17640 1 -- 172.21.15.60:0/2094178710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 msgr2=0x7f6dbc0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.161+0000 7f6dc1c17640 1 --2- 172.21.15.60:0/2094178710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.161+0000 7f6dc1c17640 1 -- 172.21.15.60:0/2094178710 >> v1:172.21.15.60:6789/0 conn(0x7f6dbc069da0 legacy=0x7f6dbc06a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.161+0000 7f6dc1c17640 1 -- 172.21.15.60:0/2094178710 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc161930 con 0x7f6dbc160b90 2023-10-16T14:13:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.162+0000 7f6dc1c17640 1 --2- 172.21.15.60:0/2094178710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc160b90 0x7f6dbc160fb0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0097a0 tx=0x7f6dac02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972180be54931922 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.162+0000 7f6dc1416640 1 -- 172.21.15.60:0/2094178710 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac02d040 con 0x7f6dbc160b90 2023-10-16T14:13:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.162+0000 7f6dc1416640 1 -- 172.21.15.60:0/2094178710 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac031810 con 0x7f6dbc160b90 2023-10-16T14:13:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc1416640 1 -- 172.21.15.60:0/2094178710 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac031ba0 con 0x7f6dbc160b90 2023-10-16T14:13:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 -- 172.21.15.60:0/2094178710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc160b90 msgr2=0x7f6dbc160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 --2- 172.21.15.60:0/2094178710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc160b90 0x7f6dbc160fb0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0097a0 tx=0x7f6dac02b8b0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 -- 172.21.15.60:0/2094178710 shutdown_connections 2023-10-16T14:13:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 --2- 172.21.15.60:0/2094178710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc160b90 0x7f6dbc160fb0 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 --2- 172.21.15.60:0/2094178710 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 -- 172.21.15.60:0/2094178710 >> 172.21.15.60:0/2094178710 conn(0x7f6dbc0ce9b0 msgr2=0x7f6dbc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 -- 172.21.15.60:0/2094178710 shutdown_connections 2023-10-16T14:13:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.163+0000 7f6dc467c640 1 -- 172.21.15.60:0/2094178710 wait complete. 2023-10-16T14:13:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.165+0000 7f6dc467c640 1 Processor -- start 2023-10-16T14:13:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.165+0000 7f6dc467c640 1 -- start start 2023-10-16T14:13:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc467c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 0x7f6dbc1f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc467c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc467c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dbc160b90 0x7f6dbc1f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc467c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0c4410 con 0x7f6dbc069da0 2023-10-16T14:13:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc467c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc1f2920 con 0x7f6dbc160b90 2023-10-16T14:13:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc467c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0c24e0 con 0x7f6dbc15d150 2023-10-16T14:13:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc2418640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 0x7f6dbc1f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc2418640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 0x7f6dbc1f6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55296/0 (socket says 172.21.15.60:55296) 2023-10-16T14:13:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.166+0000 7f6dc2418640 1 -- 172.21.15.60:0/3514814597 learned_addr learned my addr 172.21.15.60:0/3514814597 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc1c17640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2418640 1 -- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 msgr2=0x7f6dbc0c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2c19640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dbc160b90 0x7f6dbc1f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2418640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2418640 1 -- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dbc160b90 msgr2=0x7f6dbc1f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2418640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dbc160b90 0x7f6dbc1f23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2418640 1 -- 172.21.15.60:0/3514814597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac009400 con 0x7f6dbc069da0 2023-10-16T14:13:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6dc2418640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 0x7f6dbc1f6a50 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f6db800d800 tx=0x7f6db800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.167+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800b6c0 con 0x7f6dbc069da0 2023-10-16T14:13:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.168+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800b820 con 0x7f6dbc069da0 2023-10-16T14:13:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.168+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800bbb0 con 0x7f6dbc069da0 2023-10-16T14:13:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.168+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc0c4600 con 0x7f6dbc069da0 2023-10-16T14:13:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.168+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc0c2780 con 0x7f6dbc069da0 2023-10-16T14:13:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.169+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc04e160 con 0x7f6dbc069da0 2023-10-16T14:13:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.170+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 159) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f6db800bd10 con 0x7f6dbc069da0 2023-10-16T14:13:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.170+0000 7f6db2ffd640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6da006b190 0x7f6da006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.171+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6db808a140 con 0x7f6dbc069da0 2023-10-16T14:13:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.171+0000 7f6dc1c17640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6da006b190 0x7f6da006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.172+0000 7f6dc1c17640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6da006b190 0x7f6da006d650 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6dac036000 tx=0x7f6dac03c040 comp rx=0 tx=0).ready entity=mgr.8915 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.175+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6db8054dd0 con 0x7f6dbc069da0 2023-10-16T14:13:44.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.420+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f6dbc0ade10 con 0x7f6dbc069da0 2023-10-16T14:13:44.930 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.926+0000 7f6db2ffd640 1 -- 172.21.15.60:0/3514814597 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v160) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6db80583f0 con 0x7f6dbc069da0 2023-10-16T14:13:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6da006b190 msgr2=0x7f6da006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6da006b190 0x7f6da006d650 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6dac036000 tx=0x7f6dac03c040 comp rx=0 tx=0).stop 2023-10-16T14:13:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 msgr2=0x7f6dbc1f6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 0x7f6dbc1f6a50 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f6db800d800 tx=0x7f6db800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 shutdown_connections 2023-10-16T14:13:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f6da006b190 0x7f6da006d650 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6dbc15d150 0x7f6dbc0c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6dbc160b90 0x7f6dbc1f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 --2- 172.21.15.60:0/3514814597 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dbc069da0 0x7f6dbc1f6a50 secure :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f6db800d800 tx=0x7f6db800dcc0 comp rx=0 tx=0).stop 2023-10-16T14:13:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.930+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 >> 172.21.15.60:0/3514814597 conn(0x7f6dbc0ce9b0 msgr2=0x7f6dbc1581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.931+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 shutdown_connections 2023-10-16T14:13:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:44.931+0000 7f6dc467c640 1 -- 172.21.15.60:0/3514814597 wait complete. 2023-10-16T14:13:44.962 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T14:13:44.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:13:45.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T14:13:45.175 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T14:13:45.175 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:45.176 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:45.229 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:45.229 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:45.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:45.464 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:45.461+0000 7fbbcb42f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:13:45.518 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:45.515+0000 7fbbcb42f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:13:45.524 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:45.520+0000 7f0004e2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T14:13:45.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.538+0000 7f52f52aa640 1 Processor -- start 2023-10-16T14:13:45.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.538+0000 7f52f52aa640 1 -- start start 2023-10-16T14:13:45.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52f52aa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52f015cbe0 0x7f52f00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:45.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52f52aa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f52f015cfc0 0x7f52f00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:45.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52f52aa640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00d15a0 con 0x7f52f00d1090 2023-10-16T14:13:45.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52f52aa640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00d1880 con 0x7f52f015cbe0 2023-10-16T14:13:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52f52aa640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00d1b60 con 0x7f52f015cfc0 2023-10-16T14:13:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52effff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52f015cbe0 0x7f52f00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52effff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52f015cbe0 0x7f52f00d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55338/0 (socket says 172.21.15.60:55338) 2023-10-16T14:13:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52effff640 1 -- 172.21.15.60:0/3382868721 learned_addr learned my addr 172.21.15.60:0/3382868721 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.540+0000 7f52ef7fe640 1 --2- 172.21.15.60:0/3382868721 >> v2:172.21.15.144:3300/0 conn(0x7f52f015cfc0 0x7f52f00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.541+0000 7f52effff640 1 -- 172.21.15.60:0/3382868721 >> v2:172.21.15.144:3300/0 conn(0x7f52f015cfc0 msgr2=0x7f52f00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.541+0000 7f52effff640 1 --2- 172.21.15.60:0/3382868721 >> v2:172.21.15.144:3300/0 conn(0x7f52f015cfc0 0x7f52f00b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.541+0000 7f52effff640 1 -- 172.21.15.60:0/3382868721 >> v1:172.21.15.60:6789/0 conn(0x7f52f00d1090 legacy=0x7f52f006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.541+0000 7f52effff640 1 -- 172.21.15.60:0/3382868721 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00d1e40 con 0x7f52f015cbe0 2023-10-16T14:13:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.542+0000 7f52effff640 1 --2- 172.21.15.60:0/3382868721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f00d0b50 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f52f00cf4b0 tx=0x7f52e002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc12431fb5a49bf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.542+0000 7f52eeffd640 1 -- 172.21.15.60:0/3382868721 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e002f850 con 0x7f52f015cbe0 2023-10-16T14:13:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.542+0000 7f52eeffd640 1 -- 172.21.15.60:0/3382868721 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e002f9b0 con 0x7f52f015cbe0 2023-10-16T14:13:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.542+0000 7f52eeffd640 1 -- 172.21.15.60:0/3382868721 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e002fd40 con 0x7f52f015cbe0 2023-10-16T14:13:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 -- 172.21.15.60:0/3382868721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 msgr2=0x7f52f00d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3382868721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f00d0b50 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f52f00cf4b0 tx=0x7f52e002ba30 comp rx=0 tx=0).stop 2023-10-16T14:13:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 -- 172.21.15.60:0/3382868721 shutdown_connections 2023-10-16T14:13:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3382868721 >> v2:172.21.15.144:3300/0 conn(0x7f52f015cfc0 0x7f52f00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3382868721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f00d0b50 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 -- 172.21.15.60:0/3382868721 >> 172.21.15.60:0/3382868721 conn(0x7f52f00cf570 msgr2=0x7f52f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 -- 172.21.15.60:0/3382868721 shutdown_connections 2023-10-16T14:13:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.543+0000 7f52f52aa640 1 -- 172.21.15.60:0/3382868721 wait complete. 2023-10-16T14:13:45.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.545+0000 7f52f52aa640 1 Processor -- start 2023-10-16T14:13:45.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.545+0000 7f52f52aa640 1 -- start start 2023-10-16T14:13:45.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.546+0000 7f52f52aa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52f00d1090 0x7f52f01523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:45.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.546+0000 7f52f52aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:45.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.546+0000 7f52f52aa640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52f015cfc0 0x7f52f01f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:45.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.546+0000 7f52f52aa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0157130 con 0x7f52f015cbe0 2023-10-16T14:13:45.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.546+0000 7f52f52aa640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0154f60 con 0x7f52f00d1090 2023-10-16T14:13:45.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.546+0000 7f52f52aa640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0155240 con 0x7f52f015cfc0 2023-10-16T14:13:45.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:45.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f015fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55352/0 (socket says 172.21.15.60:55352) 2023-10-16T14:13:45.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 -- 172.21.15.60:0/3413093071 learned_addr learned my addr 172.21.15.60:0/3413093071 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:45.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52effff640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52f00d1090 0x7f52f01523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:45.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52f4aa9640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52f015cfc0 0x7f52f01f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:45.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 -- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52f015cfc0 msgr2=0x7f52f01f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:45.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52f015cfc0 0x7f52f01f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 -- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52f00d1090 msgr2=0x7f52f01523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:45.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52f00d1090 0x7f52f01523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 -- 172.21.15.60:0/3413093071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00094d0 con 0x7f52f015cbe0 2023-10-16T14:13:45.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52f4aa9640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52f015cfc0 0x7f52f01f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:45.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.547+0000 7f52ef7fe640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f015fb60 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f52e400e840 tx=0x7f52e400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:45.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.548+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e40094f0 con 0x7f52f015cbe0 2023-10-16T14:13:45.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.548+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e400c940 con 0x7f52f015cbe0 2023-10-16T14:13:45.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.548+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e400ccd0 con 0x7f52f015cbe0 2023-10-16T14:13:45.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.548+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f0155520 con 0x7f52f015cbe0 2023-10-16T14:13:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.548+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f01f3230 con 0x7f52f015cbe0 2023-10-16T14:13:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.549+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f004e9b0 con 0x7f52f015cbe0 2023-10-16T14:13:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.550+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f52e4013470 con 0x7f52f015cbe0 2023-10-16T14:13:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.552+0000 7f52ecff9640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 0x7f52dc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.552+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f52e4089ba0 con 0x7f52f015cbe0 2023-10-16T14:13:45.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.552+0000 7f52effff640 1 -- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 msgr2=0x7f52dc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:45.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.552+0000 7f52effff640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 0x7f52dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:45.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.556+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52e40548a0 con 0x7f52f015cbe0 2023-10-16T14:13:45.588 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:45.584+0000 7f0004e2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T14:13:45.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.753+0000 7f52effff640 1 -- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 msgr2=0x7f52dc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:45.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.753+0000 7f52effff640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 0x7f52dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:45.757 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.754+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f00cefb0 con 0x7f52f015cbe0 2023-10-16T14:13:45.760 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.757+0000 7f52ecff9640 1 -- 172.21.15.60:0/3413093071 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v288) v1 ==== 74+0+17023 (secure 0 0 0) 0x7f52e4021090 con 0x7f52f015cbe0 2023-10-16T14:13:45.761 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:45.761 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":288,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:38.050803+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3615094779":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/1408575976":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/2718528188":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/2109842343":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/306505374":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6833/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/3276093492":"2023-10-17T14:13:38.050732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:45.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.759+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 msgr2=0x7f52dc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:45.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.759+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 0x7f52dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.759+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 msgr2=0x7f52f015fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:45.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.759+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f015fb60 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f52e400e840 tx=0x7f52e400ed00 comp rx=0 tx=0).stop 2023-10-16T14:13:45.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 shutdown_connections 2023-10-16T14:13:45.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f52dc06b140 0x7f52dc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f52f015cfc0 0x7f52f01f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52f015cbe0 0x7f52f015fb60 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 --2- 172.21.15.60:0/3413093071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52f00d1090 0x7f52f01523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:45.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 >> 172.21.15.60:0/3413093071 conn(0x7f52f00cf570 msgr2=0x7f52f00b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:45.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 shutdown_connections 2023-10-16T14:13:45.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.760+0000 7f52f52aa640 1 -- 172.21.15.60:0/3413093071 wait complete. 2023-10-16T14:13:45.782 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:13:21.237677+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '286', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:13:22.301769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:13:45.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:13:45.963 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:45.960+0000 7fbbcb42f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:13:45.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.995+0000 7f7a2f59e640 1 Processor -- start 2023-10-16T14:13:45.999 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.995+0000 7f7a2f59e640 1 -- start start 2023-10-16T14:13:46.000 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a3015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3006be30 con 0x7f7a300b3ec0 2023-10-16T14:13:46.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a300ada10 con 0x7f7a300cd970 2023-10-16T14:13:46.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3006ca00 con 0x7f7a300b0ea0 2023-10-16T14:13:46.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2e59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7a300b3ec0 0x7f7a300cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43410/0 (socket says 172.21.15.60:43410) 2023-10-16T14:13:46.001 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2e59c640 1 -- 172.21.15.60:0/1273920632 learned_addr learned my addr 172.21.15.60:0/1273920632 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:46.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a3015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.003 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.997+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679097515 0 0) 0x7f7a3006be30 con 0x7f7a300b3ec0 2023-10-16T14:13:46.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a18003520 con 0x7f7a300b3ec0 2023-10-16T14:13:46.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3733080043 0 0) 0x7f7a18003520 con 0x7f7a300b3ec0 2023-10-16T14:13:46.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 msgr2=0x7f7a3015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:46.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a3015a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.004 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 >> v1:172.21.15.144:6789/0 conn(0x7f7a300cd970 legacy=0x7f7a3015a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3006c060 con 0x7f7a300b3ec0 2023-10-16T14:13:46.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f7a20001670 con 0x7f7a300b3ec0 2023-10-16T14:13:46.005 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 >> v1:172.21.15.60:6789/0 conn(0x7f7a300b3ec0 legacy=0x7f7a300cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a180039a0 0x7f7a18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a180047c0 0x7f7a18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3006ca00 con 0x7f7a180047c0 2023-10-16T14:13:46.006 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18005510 con 0x7f7a180039a0 2023-10-16T14:13:46.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1800bec0 con 0x7f7a300b0ea0 2023-10-16T14:13:46.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.998+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a180047c0 0x7f7a18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2e59c640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a180039a0 0x7f7a18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2ed9d640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 msgr2=0x7f7a1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:46.007 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a180039a0 msgr2=0x7f7a18003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:46.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a180039a0 0x7f7a18003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/1273920632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a18003520 con 0x7f7a180047c0 2023-10-16T14:13:46.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:45.999+0000 7f7a2ed9d640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a300b0ea0 0x7f7a1800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:46.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.001+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a180047c0 0x7f7a18004c40 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400ba30 tx=0x7f7a180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ce14b10a293c58 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:46.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.001+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24032630 con 0x7f7a180047c0 2023-10-16T14:13:46.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.001+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24032790 con 0x7f7a180047c0 2023-10-16T14:13:46.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.001+0000 7f7a2d59a640 1 -- 172.21.15.60:0/1273920632 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24032b20 con 0x7f7a180047c0 2023-10-16T14:13:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2f59e640 1 -- 172.21.15.60:0/1273920632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a180047c0 msgr2=0x7f7a18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/1273920632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a180047c0 0x7f7a18004c40 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400ba30 tx=0x7f7a180057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2ed9d640 1 -- 172.21.15.60:0/1273920632 reap_dead start 2023-10-16T14:13:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2f59e640 1 -- 172.21.15.60:0/1273920632 shutdown_connections 2023-10-16T14:13:46.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2f59e640 1 -- 172.21.15.60:0/1273920632 >> 172.21.15.60:0/1273920632 conn(0x7f7a300b0170 msgr2=0x7f7a300b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2f59e640 1 -- 172.21.15.60:0/1273920632 shutdown_connections 2023-10-16T14:13:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.002+0000 7f7a2f59e640 1 -- 172.21.15.60:0/1273920632 wait complete. 2023-10-16T14:13:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.004+0000 7f7a2f59e640 1 Processor -- start 2023-10-16T14:13:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.004+0000 7f7a2f59e640 1 -- start start 2023-10-16T14:13:46.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a180047c0 0x7f7a30154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 0x7f7a300c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a300c5900 0x7f7a3015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30154590 con 0x7f7a300c2c80 2023-10-16T14:13:46.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300c6550 con 0x7f7a300c5900 2023-10-16T14:13:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300c6270 con 0x7f7a180047c0 2023-10-16T14:13:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 0x7f7a300c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 0x7f7a300c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55364/0 (socket says 172.21.15.60:55364) 2023-10-16T14:13:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/2275534720 learned_addr learned my addr 172.21.15.60:0/2275534720 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:46.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2e59c640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a180047c0 0x7f7a30154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.005+0000 7f7a2ed9d640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a300c5900 0x7f7a3015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a180047c0 msgr2=0x7f7a30154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a180047c0 0x7f7a30154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a300c5900 msgr2=0x7f7a3015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a300c5900 0x7f7a3015c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2dd9b640 1 -- 172.21.15.60:0/2275534720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300c3560 con 0x7f7a300c2c80 2023-10-16T14:13:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2dd9b640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 0x7f7a300c4040 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402beb0 tx=0x7f7a24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24004220 con 0x7f7a300c2c80 2023-10-16T14:13:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24004380 con 0x7f7a300c2c80 2023-10-16T14:13:46.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2403caa0 con 0x7f7a300c2c80 2023-10-16T14:13:46.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.006+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a300c3840 con 0x7f7a300c2c80 2023-10-16T14:13:46.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.007+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a300c4580 con 0x7f7a300c2c80 2023-10-16T14:13:46.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.008+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3004e9b0 con 0x7f7a300c2c80 2023-10-16T14:13:46.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.009+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f7a2403cc00 con 0x7f7a300c2c80 2023-10-16T14:13:46.021 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.013+0000 7fbbcb42f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:13:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.009+0000 7f7a1effd640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.010+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7a240ac5c0 con 0x7f7a300c2c80 2023-10-16T14:13:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.010+0000 7f7a2e59c640 1 -- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f7a1006b140 msgr2=0x7f7a1006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:46.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.010+0000 7f7a2e59c640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:46.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.014+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a240771b0 con 0x7f7a300c2c80 2023-10-16T14:13:46.066 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.063+0000 7fbbcb42f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:13:46.076 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.073+0000 7f0004e2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T14:13:46.126 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.123+0000 7fbbcb42f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:13:46.132 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.128+0000 7f0004e2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T14:13:46.175 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.172+0000 7fbbcb42f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:13:46.184 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.181+0000 7f0004e2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T14:13:46.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.207+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a3004e9b0 con 0x7f7a300c2c80 2023-10-16T14:13:46.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.208+0000 7f7a1effd640 1 -- 172.21.15.60:0/2275534720 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v288) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a2407a7d0 con 0x7f7a300c2c80 2023-10-16T14:13:46.212 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:13:46.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f7a1006b140 msgr2=0x7f7a1006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:46.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 msgr2=0x7f7a300c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 0x7f7a300c4040 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f7a2402beb0 tx=0x7f7a24004730 comp rx=0 tx=0).stop 2023-10-16T14:13:46.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 shutdown_connections 2023-10-16T14:13:46.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f7a1006b140 0x7f7a1006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a300c5900 0x7f7a3015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a300c2c80 0x7f7a300c4040 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 --2- 172.21.15.60:0/2275534720 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f7a180047c0 0x7f7a30154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 >> 172.21.15.60:0/2275534720 conn(0x7f7a300b0170 msgr2=0x7f7a300b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.210+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 shutdown_connections 2023-10-16T14:13:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.211+0000 7f7a2f59e640 1 -- 172.21.15.60:0/2275534720 wait complete. 2023-10-16T14:13:46.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:13:46.250 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.247+0000 7f0004e2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T14:13:46.285 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.282+0000 7fbbcb42f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:13:46.302 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.299+0000 7f0004e2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T14:13:46.347 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.345+0000 7fbbcb42f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:46.397 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.394+0000 7fbbcb42f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:13:46.421 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.418+0000 7f0004e2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T14:13:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.443+0000 7f47955e9640 1 Processor -- start 2023-10-16T14:13:46.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.444+0000 7f47955e9640 1 -- start start 2023-10-16T14:13:46.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f47955e9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f47900b8480 0x7f47900cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f47955e9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f47900d0be0 0x7f47900cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f47955e9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479015d150 con 0x7f479015cd70 2023-10-16T14:13:46.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f47955e9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47900d1180 con 0x7f47900d0be0 2023-10-16T14:13:46.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f47955e9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47900d1460 con 0x7f47900b8480 2023-10-16T14:13:46.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f478ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f479015cd70 0x7f47900d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43420/0 (socket says 172.21.15.60:43420) 2023-10-16T14:13:46.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f478ffff640 1 -- 172.21.15.60:0/3702667665 learned_addr learned my addr 172.21.15.60:0/3702667665 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:46.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.445+0000 7f4794de8640 1 --2- 172.21.15.60:0/3702667665 >> v2:172.21.15.60:3300/0 conn(0x7f47900d0be0 0x7f47900cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.446+0000 7f4787fff640 1 --2- 172.21.15.60:0/3702667665 >> v2:172.21.15.144:3300/0 conn(0x7f47900b8480 0x7f47900cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.446+0000 7f478f7fe640 1 -- 172.21.15.60:0/3702667665 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511977756 0 0) 0x7f479015d150 con 0x7f479015cd70 2023-10-16T14:13:46.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.446+0000 7f4794de8640 1 -- 172.21.15.60:0/3702667665 >> v2:172.21.15.144:3300/0 conn(0x7f47900b8480 msgr2=0x7f47900cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:46.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.446+0000 7f4794de8640 1 --2- 172.21.15.60:0/3702667665 >> v2:172.21.15.144:3300/0 conn(0x7f47900b8480 0x7f47900cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.446+0000 7f4794de8640 1 -- 172.21.15.60:0/3702667665 >> v1:172.21.15.60:6789/0 conn(0x7f479015cd70 legacy=0x7f47900d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.446+0000 7f4794de8640 1 -- 172.21.15.60:0/3702667665 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900d1740 con 0x7f47900d0be0 2023-10-16T14:13:46.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.447+0000 7f4794de8640 1 --2- 172.21.15.60:0/3702667665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f47900cdd00 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f4790066a00 tx=0x7f478802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf0b2ddd31ec251 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:46.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.447+0000 7f478f7fe640 1 -- 172.21.15.60:0/3702667665 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478802cbe0 con 0x7f47900d0be0 2023-10-16T14:13:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.447+0000 7f478f7fe640 1 -- 172.21.15.60:0/3702667665 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478802cd40 con 0x7f47900d0be0 2023-10-16T14:13:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.447+0000 7f478f7fe640 1 -- 172.21.15.60:0/3702667665 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788034660 con 0x7f47900d0be0 2023-10-16T14:13:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 -- 172.21.15.60:0/3702667665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 msgr2=0x7f47900cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 --2- 172.21.15.60:0/3702667665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f47900cdd00 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f4790066a00 tx=0x7f478802be30 comp rx=0 tx=0).stop 2023-10-16T14:13:46.455 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 -- 172.21.15.60:0/3702667665 shutdown_connections 2023-10-16T14:13:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 --2- 172.21.15.60:0/3702667665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f47900cdd00 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 --2- 172.21.15.60:0/3702667665 >> v2:172.21.15.144:3300/0 conn(0x7f47900b8480 0x7f47900cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 -- 172.21.15.60:0/3702667665 >> 172.21.15.60:0/3702667665 conn(0x7f47900cf570 msgr2=0x7f47900cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 -- 172.21.15.60:0/3702667665 shutdown_connections 2023-10-16T14:13:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.448+0000 7f47955e9640 1 -- 172.21.15.60:0/3702667665 wait complete. 2023-10-16T14:13:46.457 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.451+0000 7fbbcb42f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:13:46.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.450+0000 7f47955e9640 1 Processor -- start 2023-10-16T14:13:46.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.450+0000 7f47955e9640 1 -- start start 2023-10-16T14:13:46.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f47955e9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47900b8480 0x7f4790156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.460 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f47955e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f479015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f47955e9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479015cd70 0x7f47901f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f47955e9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900cba30 con 0x7f47900d0be0 2023-10-16T14:13:46.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f47955e9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47901569d0 con 0x7f47900b8480 2023-10-16T14:13:46.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f47955e9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47901591c0 con 0x7f479015cd70 2023-10-16T14:13:46.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f4787fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f479015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f4787fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f479015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55386/0 (socket says 172.21.15.60:55386) 2023-10-16T14:13:46.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.451+0000 7f4787fff640 1 -- 172.21.15.60:0/4136466879 learned_addr learned my addr 172.21.15.60:0/4136466879 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:46.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f478ffff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47900b8480 0x7f4790156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f4787fff640 1 -- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479015cd70 msgr2=0x7f47901f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f4787fff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479015cd70 0x7f47901f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f4787fff640 1 -- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47900b8480 msgr2=0x7f4790156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f4787fff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47900b8480 0x7f4790156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f4787fff640 1 -- 172.21.15.60:0/4136466879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4788009a70 con 0x7f47900d0be0 2023-10-16T14:13:46.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f478ffff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47900b8480 0x7f4790156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:46.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f4787fff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f479015d5f0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f477800d5f0 tx=0x7f477800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:46.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.452+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780125f0 con 0x7f47900d0be0 2023-10-16T14:13:46.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.453+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778012750 con 0x7f47900d0be0 2023-10-16T14:13:46.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.453+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47901594a0 con 0x7f47900d0be0 2023-10-16T14:13:46.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.453+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778012b00 con 0x7f47900d0be0 2023-10-16T14:13:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.453+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47900cb6c0 con 0x7f47900d0be0 2023-10-16T14:13:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.454+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47900cb850 con 0x7f47900d0be0 2023-10-16T14:13:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.455+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f4778012c60 con 0x7f47900d0be0 2023-10-16T14:13:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.456+0000 7f478d7fa640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 0x7f477006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.456+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f477808a0e0 con 0x7f47900d0be0 2023-10-16T14:13:46.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.456+0000 7f478ffff640 1 -- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 msgr2=0x7f477006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:46.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.456+0000 7f478ffff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 0x7f477006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:46.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.461+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4778054ca0 con 0x7f47900d0be0 2023-10-16T14:13:46.490 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.486+0000 7f0004e2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:46.504 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.501+0000 7fbbcb42f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:13:46.543 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.539+0000 7f0004e2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T14:13:46.603 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.599+0000 7f0004e2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T14:13:46.655 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:46.652+0000 7f0004e2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T14:13:46.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.657+0000 7f478ffff640 1 -- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 msgr2=0x7f477006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:46.661 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.657+0000 7f478ffff640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 0x7f477006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:46.662 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.658+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f479004e9b0 con 0x7f47900d0be0 2023-10-16T14:13:46.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.659+0000 7f478d7fa640 1 -- 172.21.15.60:0/4136466879 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v288) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47780582c0 con 0x7f47900d0be0 2023-10-16T14:13:46.664 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:13:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 msgr2=0x7f477006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 0x7f477006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 msgr2=0x7f479015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f479015d5f0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f477800d5f0 tx=0x7f477800dab0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 shutdown_connections 2023-10-16T14:13:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f477006b140 0x7f477006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f479015cd70 0x7f47901f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47900d0be0 0x7f479015d5f0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 --2- 172.21.15.60:0/4136466879 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47900b8480 0x7f4790156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 >> 172.21.15.60:0/4136466879 conn(0x7f47900cf570 msgr2=0x7f47900cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:46.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 shutdown_connections 2023-10-16T14:13:46.668 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.662+0000 7f47955e9640 1 -- 172.21.15.60:0/4136466879 wait complete. 2023-10-16T14:13:46.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:13:46.884 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:46.881+0000 7fbbcb42f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:13:46.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.947+0000 7f2e67039640 1 Processor -- start 2023-10-16T14:13:46.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.947+0000 7f2e67039640 1 -- start start 2023-10-16T14:13:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.948+0000 7f2e67039640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.948+0000 7f2e67039640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e60159e90 0x7f2e6006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.948+0000 7f2e67039640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e600cd1f0 con 0x7f2e6006c140 2023-10-16T14:13:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.948+0000 7f2e67039640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6015d890 con 0x7f2e60159e90 2023-10-16T14:13:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.948+0000 7f2e67039640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e600cd4d0 con 0x7f2e60159ab0 2023-10-16T14:13:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e60159e90 0x7f2e6006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.953 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e60159e90 0x7f2e6006fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55390/0 (socket says 172.21.15.60:55390) 2023-10-16T14:13:46.954 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 -- 172.21.15.60:0/2174383576 learned_addr learned my addr 172.21.15.60:0/2174383576 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:46.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 -- 172.21.15.60:0/2174383576 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 msgr2=0x7f2e600ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:46.955 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 --2- 172.21.15.60:0/2174383576 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 -- 172.21.15.60:0/2174383576 >> v1:172.21.15.144:6789/0 conn(0x7f2e6006c140 legacy=0x7f2e6015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:46.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.949+0000 7f2e5ffff640 1 -- 172.21.15.60:0/2174383576 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e600cd7b0 con 0x7f2e60159e90 2023-10-16T14:13:46.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.950+0000 7f2e5ffff640 1 --2- 172.21.15.60:0/2174383576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e60159e90 0x7f2e6006fc50 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0098a0 tx=0x7f2e4c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158e90c02f48b0e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:46.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.950+0000 7f2e5f7fe640 1 -- 172.21.15.60:0/2174383576 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c031520 con 0x7f2e60159e90 2023-10-16T14:13:46.956 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.950+0000 7f2e5f7fe640 1 -- 172.21.15.60:0/2174383576 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c031680 con 0x7f2e60159e90 2023-10-16T14:13:46.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.950+0000 7f2e5f7fe640 1 -- 172.21.15.60:0/2174383576 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c031a10 con 0x7f2e60159e90 2023-10-16T14:13:46.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 -- 172.21.15.60:0/2174383576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e60159e90 msgr2=0x7f2e6006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 --2- 172.21.15.60:0/2174383576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e60159e90 0x7f2e6006fc50 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0098a0 tx=0x7f2e4c02b7c0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.957 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 -- 172.21.15.60:0/2174383576 shutdown_connections 2023-10-16T14:13:46.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 --2- 172.21.15.60:0/2174383576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e60159e90 0x7f2e6006fc50 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 --2- 172.21.15.60:0/2174383576 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 -- 172.21.15.60:0/2174383576 >> 172.21.15.60:0/2174383576 conn(0x7f2e600ce9b0 msgr2=0x7f2e600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:46.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 -- 172.21.15.60:0/2174383576 shutdown_connections 2023-10-16T14:13:46.958 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.951+0000 7f2e67039640 1 -- 172.21.15.60:0/2174383576 wait complete. 2023-10-16T14:13:46.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.953+0000 7f2e67039640 1 Processor -- start 2023-10-16T14:13:46.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.953+0000 7f2e67039640 1 -- start start 2023-10-16T14:13:46.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.954+0000 7f2e67039640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 0x7f2e6015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.961 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.954+0000 7f2e67039640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.954+0000 7f2e67039640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e60159e90 0x7f2e6014ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.954+0000 7f2e67039640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6015cd60 con 0x7f2e6006c140 2023-10-16T14:13:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.954+0000 7f2e67039640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6015c2b0 con 0x7f2e60159e90 2023-10-16T14:13:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.954+0000 7f2e67039640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6014c990 con 0x7f2e60159ab0 2023-10-16T14:13:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 0x7f2e6015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 0x7f2e6015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55406/0 (socket says 172.21.15.60:55406) 2023-10-16T14:13:46.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 -- 172.21.15.60:0/4263692601 learned_addr learned my addr 172.21.15.60:0/4263692601 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:46.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e5ffff640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.963 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e655d6640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e60159e90 0x7f2e6014ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:46.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 -- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 msgr2=0x7f2e600c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600c1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 -- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e60159e90 msgr2=0x7f2e6014ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:46.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e60159e90 0x7f2e6014ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:46.964 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 -- 172.21.15.60:0/4263692601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0094d0 con 0x7f2e6006c140 2023-10-16T14:13:46.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.955+0000 7f2e64dd5640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 0x7f2e6015bd70 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b4a0 tx=0x7f2e5400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:46.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.956+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54010610 con 0x7f2e6006c140 2023-10-16T14:13:46.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.956+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54010770 con 0x7f2e6006c140 2023-10-16T14:13:46.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.956+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54010b20 con 0x7f2e6006c140 2023-10-16T14:13:46.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.956+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6015c520 con 0x7f2e6006c140 2023-10-16T14:13:46.965 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.956+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6014cdc0 con 0x7f2e6006c140 2023-10-16T14:13:46.966 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.957+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6014cfe0 con 0x7f2e6006c140 2023-10-16T14:13:46.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.958+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f2e54010c80 con 0x7f2e6006c140 2023-10-16T14:13:46.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.959+0000 7f2e5d7fa640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 0x7f2e4406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:46.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.959+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2e540897f0 con 0x7f2e6006c140 2023-10-16T14:13:46.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.959+0000 7f2e5ffff640 1 -- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 msgr2=0x7f2e4406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:46.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.959+0000 7f2e5ffff640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 0x7f2e4406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:46.968 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:46.963+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e540543f0 con 0x7f2e6006c140 2023-10-16T14:13:47.045 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.043+0000 7fbbcb42f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:13:47.070 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.067+0000 7f0004e2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T14:13:47.160 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.156+0000 7fbbcb42f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:13:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.160+0000 7f2e5ffff640 1 -- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 msgr2=0x7f2e4406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.160+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2e6004e160 con 0x7f2e6006c140 2023-10-16T14:13:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.160+0000 7f2e5ffff640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 0x7f2e4406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:47.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.161+0000 7f2e5d7fa640 1 -- 172.21.15.60:0/4263692601 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v288) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2e54057a10 con 0x7f2e6006c140 2023-10-16T14:13:47.165 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:13:47.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 msgr2=0x7f2e4406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:47.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 0x7f2e4406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 msgr2=0x7f2e6015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:47.168 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 0x7f2e6015bd70 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b4a0 tx=0x7f2e5400b960 comp rx=0 tx=0).stop 2023-10-16T14:13:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 shutdown_connections 2023-10-16T14:13:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f2e4406b140 0x7f2e4406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2e60159ab0 0x7f2e600c1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e60159e90 0x7f2e6014ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.169 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 --2- 172.21.15.60:0/4263692601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e6006c140 0x7f2e6015bd70 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 >> 172.21.15.60:0/4263692601 conn(0x7f2e600ce9b0 msgr2=0x7f2e600c44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 shutdown_connections 2023-10-16T14:13:47.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.164+0000 7f2e67039640 1 -- 172.21.15.60:0/4263692601 wait complete. 2023-10-16T14:13:47.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T14:13:47.210 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.207+0000 7fbbcb42f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:13:47.246 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.243+0000 7f0004e2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T14:13:47.317 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.314+0000 7fbbcb42f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:47.362 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.358+0000 7f0004e2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T14:13:47.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.394+0000 7ffb56b47640 1 Processor -- start 2023-10-16T14:13:47.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.394+0000 7ffb56b47640 1 -- start start 2023-10-16T14:13:47.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.395+0000 7ffb56b47640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffb500cbf20 0x7ffb500aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:47.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.395+0000 7ffb56b47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffb5015e9b0 0x7ffb5015ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:47.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.395+0000 7ffb56b47640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5006c140 con 0x7ffb5015aee0 2023-10-16T14:13:47.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.395+0000 7ffb56b47640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5015b2c0 con 0x7ffb500cbf20 2023-10-16T14:13:47.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.395+0000 7ffb56b47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5006b270 con 0x7ffb5015e9b0 2023-10-16T14:13:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.395+0000 7ffb47fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffb500cbf20 0x7ffb500aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb47fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffb500cbf20 0x7ffb500aeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55416/0 (socket says 172.21.15.60:55416) 2023-10-16T14:13:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb47fff640 1 -- 172.21.15.60:0/2630596354 learned_addr learned my addr 172.21.15.60:0/2630596354 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb550e4640 1 --2- 172.21.15.60:0/2630596354 >> v2:172.21.15.144:3300/0 conn(0x7ffb5015e9b0 0x7ffb5015ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb47fff640 1 -- 172.21.15.60:0/2630596354 >> v2:172.21.15.144:3300/0 conn(0x7ffb5015e9b0 msgr2=0x7ffb5015ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:47.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb47fff640 1 --2- 172.21.15.60:0/2630596354 >> v2:172.21.15.144:3300/0 conn(0x7ffb5015e9b0 0x7ffb5015ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb47fff640 1 -- 172.21.15.60:0/2630596354 >> v1:172.21.15.144:6789/0 conn(0x7ffb5015aee0 legacy=0x7ffb500cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:47.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.396+0000 7ffb47fff640 1 -- 172.21.15.60:0/2630596354 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5006daf0 con 0x7ffb500cbf20 2023-10-16T14:13:47.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.397+0000 7ffb47fff640 1 --2- 172.21.15.60:0/2630596354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb500cbf20 0x7ffb500aeef0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7ffb50070550 tx=0x7ffb4002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba5e6a923013dc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:47.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.398+0000 7ffb477fe640 1 -- 172.21.15.60:0/2630596354 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb400317e0 con 0x7ffb500cbf20 2023-10-16T14:13:47.403 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.398+0000 7ffb477fe640 1 -- 172.21.15.60:0/2630596354 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb40031940 con 0x7ffb500cbf20 2023-10-16T14:13:47.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.398+0000 7ffb477fe640 1 -- 172.21.15.60:0/2630596354 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40031cd0 con 0x7ffb500cbf20 2023-10-16T14:13:47.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.398+0000 7ffb56b47640 1 -- 172.21.15.60:0/2630596354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb500cbf20 msgr2=0x7ffb500aeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:47.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.398+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2630596354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb500cbf20 0x7ffb500aeef0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7ffb50070550 tx=0x7ffb4002b880 comp rx=0 tx=0).stop 2023-10-16T14:13:47.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.399+0000 7ffb56b47640 1 -- 172.21.15.60:0/2630596354 shutdown_connections 2023-10-16T14:13:47.404 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.399+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2630596354 >> v2:172.21.15.144:3300/0 conn(0x7ffb5015e9b0 0x7ffb5015ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.399+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2630596354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb500cbf20 0x7ffb500aeef0 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.399+0000 7ffb56b47640 1 -- 172.21.15.60:0/2630596354 >> 172.21.15.60:0/2630596354 conn(0x7ffb500cfac0 msgr2=0x7ffb500cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:47.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.399+0000 7ffb56b47640 1 -- 172.21.15.60:0/2630596354 shutdown_connections 2023-10-16T14:13:47.405 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.399+0000 7ffb56b47640 1 -- 172.21.15.60:0/2630596354 wait complete. 2023-10-16T14:13:47.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.400+0000 7ffb56b47640 1 Processor -- start 2023-10-16T14:13:47.407 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.401+0000 7ffb56b47640 1 -- start start 2023-10-16T14:13:47.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.401+0000 7ffb56b47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb500cbf20 0x7ffb500c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:47.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb56b47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb5015aee0 0x7ffb500cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:47.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb56b47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 0x7ffb500ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:47.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb56b47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500ce710 con 0x7ffb5015e9b0 2023-10-16T14:13:47.408 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb56b47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5015b730 con 0x7ffb5015aee0 2023-10-16T14:13:47.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb56b47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500c71d0 con 0x7ffb500cbf20 2023-10-16T14:13:47.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 0x7ffb500ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:47.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 0x7ffb500ceeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55422/0 (socket says 172.21.15.60:55422) 2023-10-16T14:13:47.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 -- 172.21.15.60:0/2829505443 learned_addr learned my addr 172.21.15.60:0/2829505443 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:47.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb548e3640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb500cbf20 0x7ffb500c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:47.409 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb47fff640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb5015aee0 0x7ffb500cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:47.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 -- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb500cbf20 msgr2=0x7ffb500c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:47.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb500cbf20 0x7ffb500c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 -- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb5015aee0 msgr2=0x7ffb500cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:47.410 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb5015aee0 0x7ffb500cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.402+0000 7ffb550e4640 1 -- 172.21.15.60:0/2829505443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb400094d0 con 0x7ffb5015e9b0 2023-10-16T14:13:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.403+0000 7ffb550e4640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 0x7ffb500ceeb0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7ffb48007e90 tx=0x7ffb4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.403+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48014420 con 0x7ffb5015e9b0 2023-10-16T14:13:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.403+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb48014580 con 0x7ffb5015e9b0 2023-10-16T14:13:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.403+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48014930 con 0x7ffb5015e9b0 2023-10-16T14:13:47.411 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.403+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb500cdbd0 con 0x7ffb5015e9b0 2023-10-16T14:13:47.412 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.403+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb500ce150 con 0x7ffb5015e9b0 2023-10-16T14:13:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.405+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5004e160 con 0x7ffb5015e9b0 2023-10-16T14:13:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.405+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7ffb48014a90 con 0x7ffb5015e9b0 2023-10-16T14:13:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.407+0000 7ffb457fa640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 0x7ffb3c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.407+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ffb4808a450 con 0x7ffb5015e9b0 2023-10-16T14:13:47.413 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.407+0000 7ffb548e3640 1 -- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 msgr2=0x7ffb3c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:47.414 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.407+0000 7ffb548e3640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 0x7ffb3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:47.415 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.411+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb480550e0 con 0x7ffb5015e9b0 2023-10-16T14:13:47.416 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.413+0000 7f0004e2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T14:13:47.424 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.421+0000 7fbbcb42f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:13:47.531 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.528+0000 7f0004e2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:47.580 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.577+0000 7fbbcb42f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.608+0000 7ffb548e3640 1 -- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 msgr2=0x7ffb3c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.608+0000 7ffb548e3640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 0x7ffb3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.619+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffb5006d210 con 0x7ffb5015e9b0 2023-10-16T14:13:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.619+0000 7ffb457fa640 1 -- 172.21.15.60:0/2829505443 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v288) v1 ==== 80+0+477 (secure 0 0 0) 0x7ffb48058700 con 0x7ffb5015e9b0 2023-10-16T14:13:47.623 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3615094779 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.623 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/1408575976 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.624 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2718528188 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.624 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/2109842343 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.624 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/306505374 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.624 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6833/3623959149 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.624 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:6832/3623959149 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.625 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.144:0/3276093492 2023-10-17T14:13:38.050732+0000 2023-10-16T14:13:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:listed 8 entries 2023-10-16T14:13:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 msgr2=0x7ffb3c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 0x7ffb3c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 msgr2=0x7ffb500ceeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 0x7ffb500ceeb0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7ffb48007e90 tx=0x7ffb4800e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 shutdown_connections 2023-10-16T14:13:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7ffb3c06b140 0x7ffb3c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb5015e9b0 0x7ffb500ceeb0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb5015aee0 0x7ffb500cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 --2- 172.21.15.60:0/2829505443 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ffb500cbf20 0x7ffb500c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.622+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 >> 172.21.15.60:0/2829505443 conn(0x7ffb500cfac0 msgr2=0x7ffb50158900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.623+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 shutdown_connections 2023-10-16T14:13:47.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:47.623+0000 7ffb56b47640 1 -- 172.21.15.60:0/2829505443 wait complete. 2023-10-16T14:13:47.629 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.625+0000 7fbbcb42f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:47.645 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T14:13:47.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:47.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T14:13:47.647 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.642+0000 7f0004e2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T14:13:47.746 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:47.743+0000 7fbbcb42f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:13:47.753 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:13:47.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:47.754 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T14:13:47.808 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:47.808 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:47.816 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.812+0000 7f0004e2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T14:13:47.865 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:47.865 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:47.866 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:47.876 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.869+0000 7f0004e2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T14:13:47.919 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:13:47.920 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:47.920 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:47.973 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:47.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:48.000 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:47.996+0000 7f0004e2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T14:13:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.184+0000 7f37391a9640 1 Processor -- start 2023-10-16T14:13:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.185+0000 7f37391a9640 1 -- start start 2023-10-16T14:13:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.186+0000 7f37391a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37340cd0c0 0x7f37341615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.186+0000 7f37391a9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37340b0410 0x7f37340b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.186+0000 7f37391a9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373415e3e0 con 0x7f373415d3f0 2023-10-16T14:13:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.186+0000 7f37391a9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37340b0de0 con 0x7f37340b0410 2023-10-16T14:13:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.186+0000 7f37391a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37340cd4a0 con 0x7f37340cd0c0 2023-10-16T14:13:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f372bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37340b0410 0x7f37340b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f373359e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f373415d3f0 0x7f373415d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43426/0 (socket says 172.21.15.60:43426) 2023-10-16T14:13:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f373359e640 1 -- 172.21.15.60:0/576862556 learned_addr learned my addr 172.21.15.60:0/576862556 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:48.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f3732d9d640 1 --2- 172.21.15.60:0/576862556 >> v2:172.21.15.144:3300/0 conn(0x7f37340cd0c0 0x7f37341615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f372bfff640 1 -- 172.21.15.60:0/576862556 >> v2:172.21.15.144:3300/0 conn(0x7f37340cd0c0 msgr2=0x7f37341615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f372bfff640 1 --2- 172.21.15.60:0/576862556 >> v2:172.21.15.144:3300/0 conn(0x7f37340cd0c0 0x7f37341615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f372bfff640 1 -- 172.21.15.60:0/576862556 >> v1:172.21.15.60:6789/0 conn(0x7f373415d3f0 legacy=0x7f373415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f372bfff640 1 -- 172.21.15.60:0/576862556 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373415e100 con 0x7f37340b0410 2023-10-16T14:13:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.187+0000 7f373259c640 1 -- 172.21.15.60:0/576862556 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891349627 0 0) 0x7f373415e3e0 con 0x7f373415d3f0 2023-10-16T14:13:48.194 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.188+0000 7f372bfff640 1 --2- 172.21.15.60:0/576862556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340b0410 0x7f37340b08a0 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f37340cc4e0 tx=0x7f371c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5510df763b9dac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f373259c640 1 -- 172.21.15.60:0/576862556 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c02e070 con 0x7f37340b0410 2023-10-16T14:13:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f373259c640 1 -- 172.21.15.60:0/576862556 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c0315a0 con 0x7f37340b0410 2023-10-16T14:13:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f373259c640 1 -- 172.21.15.60:0/576862556 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c031930 con 0x7f37340b0410 2023-10-16T14:13:48.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 -- 172.21.15.60:0/576862556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340b0410 msgr2=0x7f37340b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 --2- 172.21.15.60:0/576862556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340b0410 0x7f37340b08a0 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f37340cc4e0 tx=0x7f371c02b840 comp rx=0 tx=0).stop 2023-10-16T14:13:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 -- 172.21.15.60:0/576862556 shutdown_connections 2023-10-16T14:13:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 --2- 172.21.15.60:0/576862556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340b0410 0x7f37340b08a0 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 --2- 172.21.15.60:0/576862556 >> v2:172.21.15.144:3300/0 conn(0x7f37340cd0c0 0x7f37341615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 -- 172.21.15.60:0/576862556 >> 172.21.15.60:0/576862556 conn(0x7f37340cb410 msgr2=0x7f37340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 -- 172.21.15.60:0/576862556 shutdown_connections 2023-10-16T14:13:48.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.189+0000 7f37391a9640 1 -- 172.21.15.60:0/576862556 wait complete. 2023-10-16T14:13:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.191+0000 7f37391a9640 1 Processor -- start 2023-10-16T14:13:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.191+0000 7f37391a9640 1 -- start start 2023-10-16T14:13:48.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f37391a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37340b0410 0x7f373406b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f37391a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 0x7f37341f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f37391a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f373415d3f0 0x7f37341f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f37391a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373406bc30 con 0x7f37340cd0c0 2023-10-16T14:13:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f37391a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373406bf10 con 0x7f37340b0410 2023-10-16T14:13:48.200 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f37391a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373406c1f0 con 0x7f373415d3f0 2023-10-16T14:13:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f372bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 0x7f37341f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f372bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 0x7f37341f5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55440/0 (socket says 172.21.15.60:55440) 2023-10-16T14:13:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.192+0000 7f372bfff640 1 -- 172.21.15.60:0/3625344322 learned_addr learned my addr 172.21.15.60:0/3625344322 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f3732d9d640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37340b0410 0x7f373406b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.201 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f373359e640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f373415d3f0 0x7f37341f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372bfff640 1 -- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f373415d3f0 msgr2=0x7f37341f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372bfff640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f373415d3f0 0x7f37341f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372bfff640 1 -- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37340b0410 msgr2=0x7f373406b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372bfff640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37340b0410 0x7f373406b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372bfff640 1 -- 172.21.15.60:0/3625344322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c009400 con 0x7f37340cd0c0 2023-10-16T14:13:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f373359e640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f373415d3f0 0x7f37341f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372bfff640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 0x7f37341f5830 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f371c000c00 tx=0x7f371c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.193+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c032900 con 0x7f37340cd0c0 2023-10-16T14:13:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.194+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c032a80 con 0x7f37340cd0c0 2023-10-16T14:13:48.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.194+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c032e30 con 0x7f37340cd0c0 2023-10-16T14:13:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.194+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373406c4d0 con 0x7f37340cd0c0 2023-10-16T14:13:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.194+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373406ccd0 con 0x7f37340cd0c0 2023-10-16T14:13:48.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.195+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373404e160 con 0x7f37340cd0c0 2023-10-16T14:13:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.196+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f371c04d3f0 con 0x7f37340cd0c0 2023-10-16T14:13:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.197+0000 7f372b7fe640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.197+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f371c0ab930 con 0x7f37340cd0c0 2023-10-16T14:13:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.197+0000 7f3732d9d640 1 -- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 msgr2=0x7f371406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.197+0000 7f3732d9d640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.201+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f371c0763e0 con 0x7f37340cd0c0 2023-10-16T14:13:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.398+0000 7f3732d9d640 1 -- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 msgr2=0x7f371406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:48.402 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.398+0000 7f3732d9d640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:48.437 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.433+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37340b4910 con 0x7f37340cd0c0 2023-10-16T14:13:48.438 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.435+0000 7f372b7fe640 1 -- 172.21.15.60:0/3625344322 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f371c079a00 con 0x7f37340cd0c0 2023-10-16T14:13:48.439 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:48.439 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:13:23.271856+0000","modified":"2023-10-16T14:13:26.430490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13065":{"gid":13065,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2549968691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2549968691},{"type":"v1","addr":"172.21.15.60:6835","nonce":2549968691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T14:13:48.439 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 152 2023-10-16T14:13:48.441 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.437+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 msgr2=0x7f371406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:48.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.437+0000 7f37391a9640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.437+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 msgr2=0x7f37341f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 0x7f37341f5830 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f371c000c00 tx=0x7f371c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.442 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 shutdown_connections 2023-10-16T14:13:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f371406b140 0x7f371406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f373415d3f0 0x7f37341f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37340cd0c0 0x7f37341f5830 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 --2- 172.21.15.60:0/3625344322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37340b0410 0x7f373406b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 >> 172.21.15.60:0/3625344322 conn(0x7f37340cb410 msgr2=0x7f373415b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:48.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 shutdown_connections 2023-10-16T14:13:48.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.438+0000 7f37391a9640 1 -- 172.21.15.60:0/3625344322 wait complete. 2023-10-16T14:13:48.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:48.589 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:48.586+0000 7fbbcb42f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:13:48.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.697+0000 7f8cb0044640 1 Processor -- start 2023-10-16T14:13:48.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.697+0000 7f8cb0044640 1 -- start start 2023-10-16T14:13:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.698+0000 7f8cb0044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80b3070 0x7f8ca80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.698+0000 7f8cb0044640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca80d1b10 0x7f8ca80cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.698+0000 7f8cb0044640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca80b3940 con 0x7f8ca80b3e70 2023-10-16T14:13:48.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.698+0000 7f8cb0044640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca80b45e0 con 0x7f8ca80d1b10 2023-10-16T14:13:48.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.698+0000 7f8cb0044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b3b90 con 0x7f8ca80b3070 2023-10-16T14:13:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cae5e1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca80d1b10 0x7f8ca80cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cae5e1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8ca80d1b10 0x7f8ca80cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55446/0 (socket says 172.21.15.60:55446) 2023-10-16T14:13:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cae5e1640 1 -- 172.21.15.60:0/983939001 learned_addr learned my addr 172.21.15.60:0/983939001 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cadde0640 1 --2- 172.21.15.60:0/983939001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80b3070 0x7f8ca80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cae5e1640 1 -- 172.21.15.60:0/983939001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80b3070 msgr2=0x7f8ca80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cae5e1640 1 --2- 172.21.15.60:0/983939001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80b3070 0x7f8ca80cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cae5e1640 1 -- 172.21.15.60:0/983939001 >> v1:172.21.15.60:6789/0 conn(0x7f8ca80b3e70 legacy=0x7f8ca80c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.700+0000 7f8cae5e1640 1 -- 172.21.15.60:0/983939001 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca80d3840 con 0x7f8ca80d1b10 2023-10-16T14:13:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.699+0000 7f8cacdde640 1 -- 172.21.15.60:0/983939001 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806519948 0 0) 0x7f8ca80b3940 con 0x7f8ca80b3e70 2023-10-16T14:13:48.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.701+0000 7f8cae5e1640 1 --2- 172.21.15.60:0/983939001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80d1b10 0x7f8ca80cfa40 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009e50 tx=0x7f8ca002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a7474ba9d8bf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:48.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.701+0000 7f8cacdde640 1 -- 172.21.15.60:0/983939001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca002cb00 con 0x7f8ca80d1b10 2023-10-16T14:13:48.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.701+0000 7f8cacdde640 1 -- 172.21.15.60:0/983939001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca002cc60 con 0x7f8ca80d1b10 2023-10-16T14:13:48.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.701+0000 7f8cacdde640 1 -- 172.21.15.60:0/983939001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca00343f0 con 0x7f8ca80d1b10 2023-10-16T14:13:48.708 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 -- 172.21.15.60:0/983939001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80d1b10 msgr2=0x7f8ca80cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 --2- 172.21.15.60:0/983939001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80d1b10 0x7f8ca80cfa40 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009e50 tx=0x7f8ca002bf60 comp rx=0 tx=0).stop 2023-10-16T14:13:48.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 -- 172.21.15.60:0/983939001 shutdown_connections 2023-10-16T14:13:48.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 --2- 172.21.15.60:0/983939001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80d1b10 0x7f8ca80cfa40 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.709 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 --2- 172.21.15.60:0/983939001 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80b3070 0x7f8ca80cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 -- 172.21.15.60:0/983939001 >> 172.21.15.60:0/983939001 conn(0x7f8ca80d2d20 msgr2=0x7f8ca80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 -- 172.21.15.60:0/983939001 shutdown_connections 2023-10-16T14:13:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.702+0000 7f8cb0044640 1 -- 172.21.15.60:0/983939001 wait complete. 2023-10-16T14:13:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.704+0000 7f8cb0044640 1 Processor -- start 2023-10-16T14:13:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.704+0000 7f8cb0044640 1 -- start start 2023-10-16T14:13:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.705+0000 7f8cb0044640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80b3070 0x7f8ca8156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.705+0000 7f8cb0044640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 0x7f8ca80b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.705+0000 7f8cb0044640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80d1b10 0x7f8ca81f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.705+0000 7f8cb0044640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b6b90 con 0x7f8ca80b3e70 2023-10-16T14:13:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.705+0000 7f8cb0044640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b23e0 con 0x7f8ca80b3070 2023-10-16T14:13:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.705+0000 7f8cb0044640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b26c0 con 0x7f8ca80d1b10 2023-10-16T14:13:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 0x7f8ca80b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 0x7f8ca80b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55456/0 (socket says 172.21.15.60:55456) 2023-10-16T14:13:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 -- 172.21.15.60:0/2775385563 learned_addr learned my addr 172.21.15.60:0/2775385563 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cadde0640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80b3070 0x7f8ca8156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cae5e1640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80d1b10 0x7f8ca81f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 -- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80d1b10 msgr2=0x7f8ca81f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80d1b10 0x7f8ca81f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 -- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80b3070 msgr2=0x7f8ca8156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80b3070 0x7f8ca8156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 -- 172.21.15.60:0/2775385563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0009b30 con 0x7f8ca80b3e70 2023-10-16T14:13:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.706+0000 7f8cad5df640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 0x7f8ca80b1a00 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800c4b0 tx=0x7f8c9800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.707+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c98005480 con 0x7f8ca80b3e70 2023-10-16T14:13:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.707+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c980035f0 con 0x7f8ca80b3e70 2023-10-16T14:13:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.707+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c980039e0 con 0x7f8ca80b3e70 2023-10-16T14:13:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.707+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca80b6de0 con 0x7f8ca80b3e70 2023-10-16T14:13:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.707+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca80c9df0 con 0x7f8ca80b3e70 2023-10-16T14:13:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.708+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca804e160 con 0x7f8ca80b3e70 2023-10-16T14:13:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.709+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 160) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f8c98003b40 con 0x7f8ca80b3e70 2023-10-16T14:13:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.710+0000 7f8c9e7fc640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 0x7f8c8c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.710+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8c98089140 con 0x7f8ca80b3e70 2023-10-16T14:13:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.710+0000 7f8cadde0640 1 -- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 msgr2=0x7f8c8c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:48.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.710+0000 7f8cadde0640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 0x7f8c8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T14:13:48.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.719+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c98053db0 con 0x7f8ca80b3e70 2023-10-16T14:13:48.884 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:48.881+0000 7fbbcb42f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:13:48.898 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:48.894+0000 7f0004e2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T14:13:48.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.911+0000 7f8cadde0640 1 -- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 msgr2=0x7f8c8c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:6832/1784354286 2023-10-16T14:13:48.915 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.911+0000 7f8cadde0640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 0x7f8c8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T14:13:48.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.917+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ca80b2880 con 0x7f8ca80b3e70 2023-10-16T14:13:48.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.920+0000 7f8c9e7fc640 1 -- 172.21.15.60:0/2775385563 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v288) v1 ==== 74+0+17023 (secure 0 0 0) 0x7f8c98017090 con 0x7f8ca80b3e70 2023-10-16T14:13:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:48.924 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":288,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:38.050803+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:0/3615094779":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/1408575976":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/2718528188":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/2109842343":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/306505374":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6833/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/3276093492":"2023-10-17T14:13:38.050732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.922+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 msgr2=0x7f8c8c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T14:13:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.922+0000 7f8cb0044640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 0x7f8c8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 msgr2=0x7f8ca80b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 0x7f8ca80b1a00 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800c4b0 tx=0x7f8c9800c970 comp rx=0 tx=0).stop 2023-10-16T14:13:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 shutdown_connections 2023-10-16T14:13:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:6832/1784354286,v1:172.21.15.144:6833/1784354286] conn(0x7f8c8c06b140 0x7f8c8c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f8ca80d1b10 0x7f8ca81f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ca80b3e70 0x7f8ca80b1a00 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.928 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 --2- 172.21.15.60:0/2775385563 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ca80b3070 0x7f8ca8156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:48.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 >> 172.21.15.60:0/2775385563 conn(0x7f8ca80d2d20 msgr2=0x7f8ca80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:48.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 shutdown_connections 2023-10-16T14:13:48.929 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:48.923+0000 7f8cb0044640 1 -- 172.21.15.60:0/2775385563 wait complete. 2023-10-16T14:13:48.941 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:48.938+0000 7fbbcb42f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:13:48.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:33:17.807385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T14:13:21.237677+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '286', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T14:13:22.301769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T14:13:48.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T14:13:49.056 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.053+0000 7fbbcb42f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:13:49.145 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.145 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.145 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.146 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.146 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4cef4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.146 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4fefa640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.146 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4fefa640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.147 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4fefa640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.147 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4fefa640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.147 INFO:tasks.ceph.mgr.x.smithi144.stderr:2023-10-16T14:13:49.142+0000 7fbb4fefa640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T14:13:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.167+0000 7fbb8d07c640 1 Processor -- start 2023-10-16T14:13:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.167+0000 7fbb8d07c640 1 -- start start 2023-10-16T14:13:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8d07c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb88159ab0 0x7fbb880ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8d07c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb8806c140 0x7fbb8806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8d07c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb88070360 con 0x7fbb88159e90 2023-10-16T14:13:49.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8d07c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb8806cb10 con 0x7fbb88159ab0 2023-10-16T14:13:49.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8d07c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8815d890 con 0x7fbb8806c140 2023-10-16T14:13:49.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb86d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb88159ab0 0x7fbb880ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb86d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb88159ab0 0x7fbb880ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51438/0 (socket says 172.21.15.60:51438) 2023-10-16T14:13:49.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8659c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fbb88159e90 0x7fbb8806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.60:47498/0 (socket says 172.21.15.60:47498) 2023-10-16T14:13:49.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb86d9d640 1 -- 172.21.15.60:0/3909266402 learned_addr learned my addr 172.21.15.60:0/3909266402 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:49.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.168+0000 7fbb8759e640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb8806c140 0x7fbb8806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.169+0000 7fbb86d9d640 1 -- 172.21.15.60:0/3909266402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb8806c140 msgr2=0x7fbb8806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:49.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.169+0000 7fbb86d9d640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb8806c140 0x7fbb8806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.169+0000 7fbb86d9d640 1 -- 172.21.15.60:0/3909266402 >> v1:172.21.15.144:6789/0 conn(0x7fbb88159e90 legacy=0x7fbb8806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.169+0000 7fbb86d9d640 1 -- 172.21.15.60:0/3909266402 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb880cd1f0 con 0x7fbb88159ab0 2023-10-16T14:13:49.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.169+0000 7fbb85d9b640 1 -- 172.21.15.60:0/3909266402 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947340834 0 0) 0x7fbb88070360 con 0x7fbb88159e90 2023-10-16T14:13:49.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.169+0000 7fbb8759e640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb8806c140 0x7fbb8806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.170+0000 7fbb86d9d640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880ce370 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009870 tx=0x7fbb7802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7445954f378696fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.170+0000 7fbb85d9b640 1 -- 172.21.15.60:0/3909266402 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb780317c0 con 0x7fbb88159ab0 2023-10-16T14:13:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.170+0000 7fbb85d9b640 1 -- 172.21.15.60:0/3909266402 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb78031920 con 0x7fbb88159ab0 2023-10-16T14:13:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.170+0000 7fbb85d9b640 1 -- 172.21.15.60:0/3909266402 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78031cb0 con 0x7fbb88159ab0 2023-10-16T14:13:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 -- 172.21.15.60:0/3909266402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 msgr2=0x7fbb880ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880ce370 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009870 tx=0x7fbb7802b860 comp rx=0 tx=0).stop 2023-10-16T14:13:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 -- 172.21.15.60:0/3909266402 shutdown_connections 2023-10-16T14:13:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb8806c140 0x7fbb8806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/3909266402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880ce370 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 -- 172.21.15.60:0/3909266402 >> 172.21.15.60:0/3909266402 conn(0x7fbb880ce9b0 msgr2=0x7fbb880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 -- 172.21.15.60:0/3909266402 shutdown_connections 2023-10-16T14:13:49.179 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.171+0000 7fbb8d07c640 1 -- 172.21.15.60:0/3909266402 wait complete. 2023-10-16T14:13:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.173+0000 7fbb8d07c640 1 Processor -- start 2023-10-16T14:13:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.173+0000 7fbb8d07c640 1 -- start start 2023-10-16T14:13:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8d07c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb8806c140 0x7fbb880ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8d07c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.182 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8d07c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb88159e90 0x7fbb880c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8d07c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8814a4c0 con 0x7fbb88159ab0 2023-10-16T14:13:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8d07c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880c30c0 con 0x7fbb8806c140 2023-10-16T14:13:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8d07c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881f5b40 con 0x7fbb88159e90 2023-10-16T14:13:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51448/0 (socket says 172.21.15.60:51448) 2023-10-16T14:13:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8659c640 1 -- 172.21.15.60:0/396394792 learned_addr learned my addr 172.21.15.60:0/396394792 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb86d9d640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb8806c140 0x7fbb880ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.174+0000 7fbb8759e640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb88159e90 0x7fbb880c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb8659c640 1 -- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb88159e90 msgr2=0x7fbb880c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb8659c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb88159e90 0x7fbb880c24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb8659c640 1 -- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb8806c140 msgr2=0x7fbb880ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb8659c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb8806c140 0x7fbb880ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb8659c640 1 -- 172.21.15.60:0/396394792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80009a70 con 0x7fbb88159ab0 2023-10-16T14:13:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb86d9d640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb8806c140 0x7fbb880ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T14:13:49.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb8659c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880acdd0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00c4d0 tx=0x7fbb7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:49.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.175+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c0074e0 con 0x7fbb88159ab0 2023-10-16T14:13:49.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.176+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7c007640 con 0x7fbb88159ab0 2023-10-16T14:13:49.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.176+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c004a20 con 0x7fbb88159ab0 2023-10-16T14:13:49.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.176+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb780094d0 con 0x7fbb88159ab0 2023-10-16T14:13:49.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.176+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881f6100 con 0x7fbb88159ab0 2023-10-16T14:13:49.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.177+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8804e160 con 0x7fbb88159ab0 2023-10-16T14:13:49.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.179+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 161) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbb7c004b80 con 0x7fbb88159ab0 2023-10-16T14:13:49.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.180+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7fbb7c089130 con 0x7fbb88159ab0 2023-10-16T14:13:49.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.184+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb7c054300 con 0x7fbb88159ab0 2023-10-16T14:13:49.255 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:49.252+0000 7f0004e2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T14:13:49.321 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:49.317+0000 7f0004e2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T14:13:49.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.384+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb880ccb60 con 0x7fbb88159ab0 2023-10-16T14:13:49.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.385+0000 7fbb6f7fe640 1 -- 172.21.15.60:0/396394792 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v289) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb7c057920 con 0x7fbb88159ab0 2023-10-16T14:13:49.389 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-10-16T14:13:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.387+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 msgr2=0x7fbb880acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.387+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880acdd0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00c4d0 tx=0x7fbb7c00c8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 shutdown_connections 2023-10-16T14:13:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbb88159e90 0x7fbb880c24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb88159ab0 0x7fbb880acdd0 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 --2- 172.21.15.60:0/396394792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb8806c140 0x7fbb880ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 >> 172.21.15.60:0/396394792 conn(0x7fbb880ce9b0 msgr2=0x7fbb88159010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 shutdown_connections 2023-10-16T14:13:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.388+0000 7fbb8d07c640 1 -- 172.21.15.60:0/396394792 wait complete. 2023-10-16T14:13:49.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T14:13:49.447 INFO:tasks.ceph.mgr.y.smithi060.stderr:2023-10-16T14:13:49.444+0000 7f0004e2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T14:13:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.619+0000 7facc759e640 1 Processor -- start 2023-10-16T14:13:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.620+0000 7facc759e640 1 -- start start 2023-10-16T14:13:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.621+0000 7facc759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc806c040 0x7facc806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.621+0000 7facc759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facc80cda80 0x7facc815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.621+0000 7facc759e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc806c960 con 0x7facc80ad970 2023-10-16T14:13:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.621+0000 7facc759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc815b9d0 con 0x7facc80cda80 2023-10-16T14:13:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.621+0000 7facc759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc815bcb0 con 0x7facc806c040 2023-10-16T14:13:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.621+0000 7facc6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facc80cda80 0x7facc815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facc80cda80 0x7facc815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51452/0 (socket says 172.21.15.60:51452) 2023-10-16T14:13:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc6d9d640 1 -- 172.21.15.60:0/757726819 learned_addr learned my addr 172.21.15.60:0/757726819 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc5d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7facc80ad970 0x7facc815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60288/0 (socket says 172.21.15.60:60288) 2023-10-16T14:13:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc659c640 1 --2- 172.21.15.60:0/757726819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc806c040 0x7facc806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc6d9d640 1 -- 172.21.15.60:0/757726819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc806c040 msgr2=0x7facc806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc6d9d640 1 --2- 172.21.15.60:0/757726819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc806c040 0x7facc806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc6d9d640 1 -- 172.21.15.60:0/757726819 >> v1:172.21.15.60:6789/0 conn(0x7facc80ad970 legacy=0x7facc815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.622+0000 7facc6d9d640 1 -- 172.21.15.60:0/757726819 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc815bf90 con 0x7facc80cda80 2023-10-16T14:13:49.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.623+0000 7facc6d9d640 1 --2- 172.21.15.60:0/757726819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc80cda80 0x7facc815ed90 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7facc0009e50 tx=0x7facc002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369626169ce5830 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:49.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.623+0000 7facc559a640 1 -- 172.21.15.60:0/757726819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc002cb00 con 0x7facc80cda80 2023-10-16T14:13:49.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.623+0000 7facc559a640 1 -- 172.21.15.60:0/757726819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc002cc60 con 0x7facc80cda80 2023-10-16T14:13:49.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc559a640 1 -- 172.21.15.60:0/757726819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc00343f0 con 0x7facc80cda80 2023-10-16T14:13:49.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 -- 172.21.15.60:0/757726819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc80cda80 msgr2=0x7facc815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 --2- 172.21.15.60:0/757726819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc80cda80 0x7facc815ed90 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7facc0009e50 tx=0x7facc002bf60 comp rx=0 tx=0).stop 2023-10-16T14:13:49.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 -- 172.21.15.60:0/757726819 shutdown_connections 2023-10-16T14:13:49.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 --2- 172.21.15.60:0/757726819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc80cda80 0x7facc815ed90 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 --2- 172.21.15.60:0/757726819 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc806c040 0x7facc806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 -- 172.21.15.60:0/757726819 >> 172.21.15.60:0/757726819 conn(0x7facc80ce6a0 msgr2=0x7facc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:49.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 -- 172.21.15.60:0/757726819 shutdown_connections 2023-10-16T14:13:49.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.624+0000 7facc759e640 1 -- 172.21.15.60:0/757726819 wait complete. 2023-10-16T14:13:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.626+0000 7facc759e640 1 Processor -- start 2023-10-16T14:13:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.626+0000 7facc759e640 1 -- start start 2023-10-16T14:13:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 0x7facc80c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc759e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc80ad970 0x7facc80c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc80cda80 0x7facc8150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc80c7fe0 con 0x7facc806c040 2023-10-16T14:13:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc759e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc80c7610 con 0x7facc80cda80 2023-10-16T14:13:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc759e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc8151980 con 0x7facc80ad970 2023-10-16T14:13:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.627+0000 7facc659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 0x7facc80c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 0x7facc80c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51464/0 (socket says 172.21.15.60:51464) 2023-10-16T14:13:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 -- 172.21.15.60:0/3628408025 learned_addr learned my addr 172.21.15.60:0/3628408025 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc5d9b640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc80ad970 0x7facc80c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 -- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc80ad970 msgr2=0x7facc80c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc80ad970 0x7facc80c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 -- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc80cda80 msgr2=0x7facc8150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc80cda80 0x7facc8150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 -- 172.21.15.60:0/3628408025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0009b30 con 0x7facc806c040 2023-10-16T14:13:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.628+0000 7facc659c640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 0x7facc80c43b0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7facb800c820 tx=0x7facb800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.629+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8004020 con 0x7facc806c040 2023-10-16T14:13:49.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.629+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb8004180 con 0x7facc806c040 2023-10-16T14:13:49.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.629+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8007830 con 0x7facc806c040 2023-10-16T14:13:49.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.629+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc80c6fc0 con 0x7facc806c040 2023-10-16T14:13:49.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.629+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc8150130 con 0x7facc806c040 2023-10-16T14:13:49.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.630+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc806b330 con 0x7facc806c040 2023-10-16T14:13:49.642 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.631+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 161) v1 ==== 88940+0+0 (secure 0 0 0) 0x7facb8007990 con 0x7facc806c040 2023-10-16T14:13:49.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.632+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7facb8089160 con 0x7facc806c040 2023-10-16T14:13:49.643 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.636+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facb8054240 con 0x7facc806c040 2023-10-16T14:13:49.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.839+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7facc806c630 con 0x7facc806c040 2023-10-16T14:13:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.839+0000 7facb6ffd640 1 -- 172.21.15.60:0/3628408025 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v289) v1 ==== 104+0+11 (secure 0 0 0) 0x7facb8057860 con 0x7facc806c040 2023-10-16T14:13:49.846 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2023-10-16T14:13:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 msgr2=0x7facc80c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 0x7facc80c43b0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7facb800c820 tx=0x7facb800cce0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 shutdown_connections 2023-10-16T14:13:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7facc80ad970 0x7facc80c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc80cda80 0x7facc8150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 --2- 172.21.15.60:0/3628408025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc806c040 0x7facc80c43b0 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 >> 172.21.15.60:0/3628408025 conn(0x7facc80ce6a0 msgr2=0x7facc815de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 shutdown_connections 2023-10-16T14:13:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:49.842+0000 7facc759e640 1 -- 172.21.15.60:0/3628408025 wait complete. 2023-10-16T14:13:49.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T14:13:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.142+0000 7fe4b9d0a640 1 Processor -- start 2023-10-16T14:13:50.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.142+0000 7fe4b9d0a640 1 -- start start 2023-10-16T14:13:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.144+0000 7fe4b9d0a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b406a4b0 0x7fe4b40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.144+0000 7fe4b9d0a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40cb030 con 0x7fe4b415ad70 2023-10-16T14:13:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.144+0000 7fe4b9d0a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40c91c0 con 0x7fe4b40c9ca0 2023-10-16T14:13:50.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.144+0000 7fe4b9d0a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40c94a0 con 0x7fe4b406a4b0 2023-10-16T14:13:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.144+0000 7fe4b37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe4b415ad70 0x7fe4b4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60304/0 (socket says 172.21.15.60:60304) 2023-10-16T14:13:50.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.144+0000 7fe4b37fe640 1 -- 172.21.15.60:0/305689963 learned_addr learned my addr 172.21.15.60:0/305689963 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b2ffd640 1 --2- 172.21.15.60:0/305689963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b406a4b0 0x7fe4b40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b27fc640 1 -- 172.21.15.60:0/305689963 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243757222 0 0) 0x7fe4b40cb030 con 0x7fe4b415ad70 2023-10-16T14:13:50.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b27fc640 1 -- 172.21.15.60:0/305689963 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c003520 con 0x7fe4b415ad70 2023-10-16T14:13:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b2ffd640 1 -- 172.21.15.60:0/305689963 >> v1:172.21.15.144:6789/0 conn(0x7fe4b40c9ca0 legacy=0x7fe4b40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b2ffd640 1 -- 172.21.15.60:0/305689963 >> v1:172.21.15.60:6789/0 conn(0x7fe4b415ad70 legacy=0x7fe4b4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b27fc640 1 -- 172.21.15.60:0/305689963 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088787190 0 0) 0x7fe49c003520 con 0x7fe4b415ad70 2023-10-16T14:13:50.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.145+0000 7fe4b2ffd640 1 -- 172.21.15.60:0/305689963 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40c9780 con 0x7fe4b406a4b0 2023-10-16T14:13:50.152 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.147+0000 7fe4b2ffd640 1 --2- 172.21.15.60:0/305689963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b406a4b0 0x7fe4b40ca3b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80097d0 tx=0x7fe4a802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba5fd75b6aeadf50 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.147+0000 7fe4b27fc640 1 -- 172.21.15.60:0/305689963 <== mon.2 v2:172.21.15.144:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4001670 con 0x7fe4b406a4b0 2023-10-16T14:13:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.147+0000 7fe4b27fc640 1 -- 172.21.15.60:0/305689963 <== mon.2 v2:172.21.15.144:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a802e050 con 0x7fe4b406a4b0 2023-10-16T14:13:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.147+0000 7fe4b27fc640 1 -- 172.21.15.60:0/305689963 <== mon.2 v2:172.21.15.144:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8030710 con 0x7fe4b406a4b0 2023-10-16T14:13:50.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/305689963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b406a4b0 msgr2=0x7fe4b40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/305689963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b406a4b0 0x7fe4b40ca3b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80097d0 tx=0x7fe4a802b800 comp rx=0 tx=0).stop 2023-10-16T14:13:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/305689963 shutdown_connections 2023-10-16T14:13:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/305689963 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b406a4b0 0x7fe4b40ca3b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/305689963 >> 172.21.15.60:0/305689963 conn(0x7fe4b40ce9b0 msgr2=0x7fe4b40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/305689963 shutdown_connections 2023-10-16T14:13:50.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.148+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/305689963 wait complete. 2023-10-16T14:13:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.150+0000 7fe4b9d0a640 1 Processor -- start 2023-10-16T14:13:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.150+0000 7fe4b9d0a640 1 -- start start 2023-10-16T14:13:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b9d0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 0x7fe4b41f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b9d0a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4b40c9ca0 0x7fe4b40ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b9d0a640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b415ad70 0x7fe4b414da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b9d0a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40adc10 con 0x7fe4b406a4b0 2023-10-16T14:13:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b9d0a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40adef0 con 0x7fe4b40c9ca0 2023-10-16T14:13:50.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b9d0a640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40ae1d0 con 0x7fe4b415ad70 2023-10-16T14:13:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.151+0000 7fe4b37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 0x7fe4b41f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 0x7fe4b41f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51478/0 (socket says 172.21.15.60:51478) 2023-10-16T14:13:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 -- 172.21.15.60:0/1016172395 learned_addr learned my addr 172.21.15.60:0/1016172395 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b2ffd640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4b40c9ca0 0x7fe4b40ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b3fff640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b415ad70 0x7fe4b414da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 -- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b415ad70 msgr2=0x7fe4b414da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b415ad70 0x7fe4b414da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 -- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4b40c9ca0 msgr2=0x7fe4b40ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4b40c9ca0 0x7fe4b40ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 -- 172.21.15.60:0/1016172395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a8009400 con 0x7fe4b406a4b0 2023-10-16T14:13:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b2ffd640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4b40c9ca0 0x7fe4b40ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.152+0000 7fe4b37fe640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 0x7fe4b41f1f30 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400c780 tx=0x7fe4a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.153+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4005ad0 con 0x7fe4b406a4b0 2023-10-16T14:13:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.153+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4005c30 con 0x7fe4b406a4b0 2023-10-16T14:13:50.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.153+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b414bb90 con 0x7fe4b406a4b0 2023-10-16T14:13:50.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.153+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4007070 con 0x7fe4b406a4b0 2023-10-16T14:13:50.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.153+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b40ad9c0 con 0x7fe4b406a4b0 2023-10-16T14:13:50.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.154+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b404e160 con 0x7fe4b406a4b0 2023-10-16T14:13:50.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.155+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 162) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe4a40071d0 con 0x7fe4b406a4b0 2023-10-16T14:13:50.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.156+0000 7fe49bfff640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fe4a006acc0 0x7fe4a006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.156+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7fe4a4089380 con 0x7fe4b406a4b0 2023-10-16T14:13:50.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.157+0000 7fe4b2ffd640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fe4a006acc0 0x7fe4a006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.158+0000 7fe4b2ffd640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fe4a006acc0 0x7fe4a006d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8034000 tx=0x7fe4a8008030 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:50.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.161+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4a40542f0 con 0x7fe4b406a4b0 2023-10-16T14:13:50.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.358+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4b415a470 con 0x7fe4b406a4b0 2023-10-16T14:13:50.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.359+0000 7fe49bfff640 1 -- 172.21.15.60:0/1016172395 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v289) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe4a4057910 con 0x7fe4b406a4b0 2023-10-16T14:13:50.363 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2023-10-16T14:13:50.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fe4a006acc0 msgr2=0x7fe4a006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fe4a006acc0 0x7fe4a006d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8034000 tx=0x7fe4a8008030 comp rx=0 tx=0).stop 2023-10-16T14:13:50.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 msgr2=0x7fe4b41f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 0x7fe4b41f1f30 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400c780 tx=0x7fe4a400cc40 comp rx=0 tx=0).stop 2023-10-16T14:13:50.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 shutdown_connections 2023-10-16T14:13:50.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fe4a006acc0 0x7fe4a006d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fe4b415ad70 0x7fe4b414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4b40c9ca0 0x7fe4b40ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 --2- 172.21.15.60:0/1016172395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4b406a4b0 0x7fe4b41f1f30 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 >> 172.21.15.60:0/1016172395 conn(0x7fe4b40ce9b0 msgr2=0x7fe4b40cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:50.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 shutdown_connections 2023-10-16T14:13:50.368 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.362+0000 7fe4b9d0a640 1 -- 172.21.15.60:0/1016172395 wait complete. 2023-10-16T14:13:50.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:50.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.614+0000 7f76d5f89640 1 Processor -- start 2023-10-16T14:13:50.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.614+0000 7f76d5f89640 1 -- start start 2023-10-16T14:13:50.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.615+0000 7f76d5f89640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76d006c3f0 0x7f76d006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.615+0000 7f76d5f89640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f76d015e540 0x7f76d00b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76d5f89640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d00cd630 con 0x7f76d015e0c0 2023-10-16T14:13:50.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76d5f89640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00ce1b0 con 0x7f76d015e540 2023-10-16T14:13:50.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76d5f89640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d006cd10 con 0x7f76d006c3f0 2023-10-16T14:13:50.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76cffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f76d015e540 0x7f76d00b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76c6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f76d015e0c0 0x7f76d00b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60316/0 (socket says 172.21.15.60:60316) 2023-10-16T14:13:50.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76c6ffd640 1 -- 172.21.15.60:0/680473084 learned_addr learned my addr 172.21.15.60:0/680473084 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:50.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76cf7fe640 1 --2- 172.21.15.60:0/680473084 >> v2:172.21.15.144:3300/0 conn(0x7f76d006c3f0 0x7f76d006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76cffff640 1 -- 172.21.15.60:0/680473084 >> v2:172.21.15.144:3300/0 conn(0x7f76d006c3f0 msgr2=0x7f76d006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:50.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.616+0000 7f76cffff640 1 --2- 172.21.15.60:0/680473084 >> v2:172.21.15.144:3300/0 conn(0x7f76d006c3f0 0x7f76d006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.617+0000 7f76cffff640 1 -- 172.21.15.60:0/680473084 >> v1:172.21.15.60:6789/0 conn(0x7f76d015e0c0 legacy=0x7f76d00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.617+0000 7f76cffff640 1 -- 172.21.15.60:0/680473084 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00d2170 con 0x7f76d015e540 2023-10-16T14:13:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.618+0000 7f76cffff640 1 --2- 172.21.15.60:0/680473084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e540 0x7f76d00b1710 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f76d00d3d00 tx=0x7f76c802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7488d6d9b5bb9f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.618+0000 7f76ceffd640 1 -- 172.21.15.60:0/680473084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c802cc70 con 0x7f76d015e540 2023-10-16T14:13:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.618+0000 7f76ceffd640 1 -- 172.21.15.60:0/680473084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c802cdd0 con 0x7f76d015e540 2023-10-16T14:13:50.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.618+0000 7f76ceffd640 1 -- 172.21.15.60:0/680473084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8034660 con 0x7f76d015e540 2023-10-16T14:13:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.618+0000 7f76d5f89640 1 -- 172.21.15.60:0/680473084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e540 msgr2=0x7f76d00b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.618+0000 7f76d5f89640 1 --2- 172.21.15.60:0/680473084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e540 0x7f76d00b1710 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f76d00d3d00 tx=0x7f76c802bec0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.619+0000 7f76d5f89640 1 -- 172.21.15.60:0/680473084 shutdown_connections 2023-10-16T14:13:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.619+0000 7f76d5f89640 1 --2- 172.21.15.60:0/680473084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e540 0x7f76d00b1710 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.619+0000 7f76d5f89640 1 --2- 172.21.15.60:0/680473084 >> v2:172.21.15.144:3300/0 conn(0x7f76d006c3f0 0x7f76d006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.619+0000 7f76d5f89640 1 -- 172.21.15.60:0/680473084 >> 172.21.15.60:0/680473084 conn(0x7f76d00d2cb0 msgr2=0x7f76d00d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:50.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.619+0000 7f76d5f89640 1 -- 172.21.15.60:0/680473084 shutdown_connections 2023-10-16T14:13:50.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.619+0000 7f76d5f89640 1 -- 172.21.15.60:0/680473084 wait complete. 2023-10-16T14:13:50.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.621+0000 7f76d5f89640 1 Processor -- start 2023-10-16T14:13:50.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.621+0000 7f76d5f89640 1 -- start start 2023-10-16T14:13:50.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76d5f89640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76d006c3f0 0x7f76d0154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76d5f89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 0x7f76d015fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76d5f89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76d015e540 0x7f76d0160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76d5f89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0159f70 con 0x7f76d015e0c0 2023-10-16T14:13:50.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76d5f89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00cb470 con 0x7f76d015e540 2023-10-16T14:13:50.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76d5f89640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00cb750 con 0x7f76d006c3f0 2023-10-16T14:13:50.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76c6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 0x7f76d015fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76c6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 0x7f76d015fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51490/0 (socket says 172.21.15.60:51490) 2023-10-16T14:13:50.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76c6ffd640 1 -- 172.21.15.60:0/3321702116 learned_addr learned my addr 172.21.15.60:0/3321702116 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:50.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76cffff640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76d015e540 0x7f76d0160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76cf7fe640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76d006c3f0 0x7f76d0154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76c6ffd640 1 -- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76d006c3f0 msgr2=0x7f76d0154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76c6ffd640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76d006c3f0 0x7f76d0154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.622+0000 7f76c6ffd640 1 -- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76d015e540 msgr2=0x7f76d0160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76c6ffd640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76d015e540 0x7f76d0160280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76c6ffd640 1 -- 172.21.15.60:0/3321702116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009a70 con 0x7f76d015e0c0 2023-10-16T14:13:50.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76c6ffd640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 0x7f76d015fd40 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f76b800c4a0 tx=0x7f76b800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:50.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8005ce0 con 0x7f76d015e0c0 2023-10-16T14:13:50.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8005e40 con 0x7f76d015e0c0 2023-10-16T14:13:50.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8007270 con 0x7f76d015e0c0 2023-10-16T14:13:50.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d00cba30 con 0x7f76d015e0c0 2023-10-16T14:13:50.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.623+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d0155030 con 0x7f76d015e0c0 2023-10-16T14:13:50.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.625+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d00cbf50 con 0x7f76d015e0c0 2023-10-16T14:13:50.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.625+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 162) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f76b8007490 con 0x7f76d015e0c0 2023-10-16T14:13:50.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.626+0000 7f76ccff9640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f76b006acc0 0x7f76b006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:50.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.627+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f76b8088b00 con 0x7f76d015e0c0 2023-10-16T14:13:50.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.627+0000 7f76cf7fe640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f76b006acc0 0x7f76b006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:50.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.628+0000 7f76cf7fe640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f76b006acc0 0x7f76b006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:50.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.631+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76d00cbf50 con 0x7f76d015e0c0 2023-10-16T14:13:50.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.869+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d00cbf50 con 0x7f76d015e0c0 2023-10-16T14:13:50.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.870+0000 7f76ccff9640 1 -- 172.21.15.60:0/3321702116 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f76d00cbf50 con 0x7f76d015e0c0 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:13:23.271856+0000","modified":"2023-10-16T14:13:26.430490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13065":{"gid":13065,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2549968691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2549968691},{"type":"v1","addr":"172.21.15.60:6835","nonce":2549968691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T14:13:50.875 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 152 2023-10-16T14:13:50.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.873+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f76b006acc0 msgr2=0x7f76b006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.873+0000 7f76d5f89640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f76b006acc0 0x7f76b006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0008040 comp rx=0 tx=0).stop 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.873+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 msgr2=0x7f76d015fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.873+0000 7f76d5f89640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 0x7f76d015fd40 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f76b800c4a0 tx=0x7f76b800c960 comp rx=0 tx=0).stop 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 shutdown_connections 2023-10-16T14:13:50.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f76b006acc0 0x7f76b006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76d015e540 0x7f76d0160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015e0c0 0x7f76d015fd40 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 --2- 172.21.15.60:0/3321702116 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f76d006c3f0 0x7f76d0154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 >> 172.21.15.60:0/3321702116 conn(0x7f76d00d2cb0 msgr2=0x7f76d00d1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:50.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 shutdown_connections 2023-10-16T14:13:50.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:50.874+0000 7f76d5f89640 1 -- 172.21.15.60:0/3321702116 wait complete. 2023-10-16T14:13:50.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.116+0000 7f90c624b640 1 Processor -- start 2023-10-16T14:13:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.116+0000 7f90c624b640 1 -- start start 2023-10-16T14:13:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90c624b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f90c006c7d0 0x7f90c00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90c624b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90c00b4c70 0x7f90c00b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90c624b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c00cd630 con 0x7f90c006c3f0 2023-10-16T14:13:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90c624b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c015e7d0 con 0x7f90c00b4c70 2023-10-16T14:13:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90c624b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c00ce1b0 con 0x7f90c006c7d0 2023-10-16T14:13:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90bffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90c00b4c70 0x7f90c00b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90bffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90c00b4c70 0x7f90c00b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51504/0 (socket says 172.21.15.60:51504) 2023-10-16T14:13:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.118+0000 7f90bffff640 1 -- 172.21.15.60:0/3292385517 learned_addr learned my addr 172.21.15.60:0/3292385517 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.119+0000 7f90beffd640 1 --2- 172.21.15.60:0/3292385517 >> v2:172.21.15.144:3300/0 conn(0x7f90c006c7d0 0x7f90c00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.119+0000 7f90bffff640 1 -- 172.21.15.60:0/3292385517 >> v2:172.21.15.144:3300/0 conn(0x7f90c006c7d0 msgr2=0x7f90c00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.119+0000 7f90bffff640 1 --2- 172.21.15.60:0/3292385517 >> v2:172.21.15.144:3300/0 conn(0x7f90c006c7d0 0x7f90c00b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.119+0000 7f90bffff640 1 -- 172.21.15.60:0/3292385517 >> v1:172.21.15.144:6789/0 conn(0x7f90c006c3f0 legacy=0x7f90c015e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.119+0000 7f90bffff640 1 -- 172.21.15.60:0/3292385517 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c00b50a0 con 0x7f90c00b4c70 2023-10-16T14:13:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.120+0000 7f90bffff640 1 --2- 172.21.15.60:0/3292385517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00b4c70 0x7f90c00b1710 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f90c00d3a50 tx=0x7f90b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac1750c5ad1ea45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.120+0000 7f90be7fc640 1 -- 172.21.15.60:0/3292385517 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b802ca80 con 0x7f90c00b4c70 2023-10-16T14:13:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.120+0000 7f90be7fc640 1 -- 172.21.15.60:0/3292385517 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b802cbe0 con 0x7f90c00b4c70 2023-10-16T14:13:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.120+0000 7f90be7fc640 1 -- 172.21.15.60:0/3292385517 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b80343f0 con 0x7f90c00b4c70 2023-10-16T14:13:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 -- 172.21.15.60:0/3292385517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00b4c70 msgr2=0x7f90c00b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 --2- 172.21.15.60:0/3292385517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00b4c70 0x7f90c00b1710 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f90c00d3a50 tx=0x7f90b802bee0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 -- 172.21.15.60:0/3292385517 shutdown_connections 2023-10-16T14:13:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 --2- 172.21.15.60:0/3292385517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c00b4c70 0x7f90c00b1710 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 --2- 172.21.15.60:0/3292385517 >> v2:172.21.15.144:3300/0 conn(0x7f90c006c7d0 0x7f90c00b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 -- 172.21.15.60:0/3292385517 >> 172.21.15.60:0/3292385517 conn(0x7f90c00d2cb0 msgr2=0x7f90c00d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 -- 172.21.15.60:0/3292385517 shutdown_connections 2023-10-16T14:13:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.121+0000 7f90c624b640 1 -- 172.21.15.60:0/3292385517 wait complete. 2023-10-16T14:13:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.123+0000 7f90c624b640 1 Processor -- start 2023-10-16T14:13:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.123+0000 7f90c624b640 1 -- start start 2023-10-16T14:13:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.124+0000 7f90c624b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c006c3f0 0x7f90c01f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.124+0000 7f90c624b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 0x7f90c0153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.124+0000 7f90c624b640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c00b4c70 0x7f90c0159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.124+0000 7f90c624b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00c9cf0 con 0x7f90c006c7d0 2023-10-16T14:13:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.124+0000 7f90c624b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00c9fd0 con 0x7f90c006c3f0 2023-10-16T14:13:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.124+0000 7f90c624b640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00ca2b0 con 0x7f90c00b4c70 2023-10-16T14:13:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 0x7f90c0153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 0x7f90c0153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51516/0 (socket says 172.21.15.60:51516) 2023-10-16T14:13:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 -- 172.21.15.60:0/2354136185 learned_addr learned my addr 172.21.15.60:0/2354136185 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90bf7fe640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c006c3f0 0x7f90c01f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90bffff640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c00b4c70 0x7f90c0159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 -- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c00b4c70 msgr2=0x7f90c0159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c00b4c70 0x7f90c0159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 -- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c006c3f0 msgr2=0x7f90c01f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c006c3f0 0x7f90c01f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90beffd640 1 -- 172.21.15.60:0/2354136185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b8009b30 con 0x7f90c006c7d0 2023-10-16T14:13:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90bffff640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c00b4c70 0x7f90c0159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.125+0000 7f90bf7fe640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c006c3f0 0x7f90c01f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.126+0000 7f90beffd640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 0x7f90c0153d10 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f90b400d800 tx=0x7f90b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.126+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400b6c0 con 0x7f90c006c7d0 2023-10-16T14:13:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.126+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b400b820 con 0x7f90c006c7d0 2023-10-16T14:13:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.126+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400bbb0 con 0x7f90c006c7d0 2023-10-16T14:13:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.126+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c00ca590 con 0x7f90c006c7d0 2023-10-16T14:13:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.126+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c00cab50 con 0x7f90c006c7d0 2023-10-16T14:13:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.127+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c00cad10 con 0x7f90c006c7d0 2023-10-16T14:13:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.128+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f90b400bd10 con 0x7f90c006c7d0 2023-10-16T14:13:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.129+0000 7f90a7fff640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f90ac06af80 0x7f90ac06d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.129+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f90b4089d70 con 0x7f90c006c7d0 2023-10-16T14:13:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.130+0000 7f90bf7fe640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f90ac06af80 0x7f90ac06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.131+0000 7f90bf7fe640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f90ac06af80 0x7f90ac06d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f90b0006fd0 tx=0x7f90b0008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.134+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f90b4054c10 con 0x7f90c006c7d0 2023-10-16T14:13:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.334+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90c004e160 con 0x7f90c006c7d0 2023-10-16T14:13:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.337+0000 7f90a7fff640 1 -- 172.21.15.60:0/2354136185 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v289) v1 ==== 74+0+17719 (secure 0 0 0) 0x7f90b4058230 con 0x7f90c006c7d0 2023-10-16T14:13:51.341 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:51.341 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":289,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:49.056966+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/1784354286":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/758300120":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/1291769788":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/1290683266":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3884644447":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3738039093":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/579488795":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3615094779":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/1408575976":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/277289582":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/2718528188":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/4157805917":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/2109842343":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/306505374":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6833/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/1784354286":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3309380401":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3276093492":"2023-10-17T14:13:38.050732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f90ac06af80 msgr2=0x7f90ac06d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f90ac06af80 0x7f90ac06d440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f90b0006fd0 tx=0x7f90b0008040 comp rx=0 tx=0).stop 2023-10-16T14:13:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 msgr2=0x7f90c0153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 0x7f90c0153d10 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f90b400d800 tx=0x7f90b400dcc0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 shutdown_connections 2023-10-16T14:13:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f90ac06af80 0x7f90ac06d440 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f90c00b4c70 0x7f90c0159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90c006c7d0 0x7f90c0153d10 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 --2- 172.21.15.60:0/2354136185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90c006c3f0 0x7f90c01f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.339+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 >> 172.21.15.60:0/2354136185 conn(0x7f90c00d2cb0 msgr2=0x7f90c00b8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.340+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 shutdown_connections 2023-10-16T14:13:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.340+0000 7f90c624b640 1 -- 172.21.15.60:0/2354136185 wait complete. 2023-10-16T14:13:51.363 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:13:51.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.580+0000 7f2ca359e640 1 Processor -- start 2023-10-16T14:13:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.580+0000 7f2ca359e640 1 -- start start 2023-10-16T14:13:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.582+0000 7f2ca359e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ca40ca0a0 0x7f2ca406c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.582+0000 7f2ca359e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca415d910 con 0x7f2ca4070b60 2023-10-16T14:13:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.582+0000 7f2ca359e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca415dbf0 con 0x7f2ca4159950 2023-10-16T14:13:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.582+0000 7f2ca359e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca415e490 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.583+0000 7f2c9bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2ca4070b60 0x7f2ca406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60332/0 (socket says 172.21.15.60:60332) 2023-10-16T14:13:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.583+0000 7f2c9bfff640 1 -- 172.21.15.60:0/615184188 learned_addr learned my addr 172.21.15.60:0/615184188 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.583+0000 7f2ca2d9d640 1 --2- 172.21.15.60:0/615184188 >> v2:172.21.15.144:3300/0 conn(0x7f2ca40ca0a0 0x7f2ca406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.583+0000 7f2ca1d9b640 1 -- 172.21.15.60:0/615184188 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086997797 0 0) 0x7f2ca415d910 con 0x7f2ca4070b60 2023-10-16T14:13:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.583+0000 7f2ca1d9b640 1 -- 172.21.15.60:0/615184188 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c90005730 con 0x7f2ca4070b60 2023-10-16T14:13:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.584+0000 7f2ca2d9d640 1 -- 172.21.15.60:0/615184188 >> v1:172.21.15.144:6789/0 conn(0x7f2ca4159950 legacy=0x7f2ca4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.584+0000 7f2ca2d9d640 1 -- 172.21.15.60:0/615184188 >> v1:172.21.15.60:6789/0 conn(0x7f2ca4070b60 legacy=0x7f2ca406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.584+0000 7f2ca1d9b640 1 -- 172.21.15.60:0/615184188 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3041770169 0 0) 0x7f2c90005730 con 0x7f2ca4070b60 2023-10-16T14:13:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.584+0000 7f2ca2d9d640 1 -- 172.21.15.60:0/615184188 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca415ded0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.585+0000 7f2ca2d9d640 1 --2- 172.21.15.60:0/615184188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca406c810 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009d90 tx=0x7f2c9c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=456253ae5f3f8810 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca1d9b640 1 -- 172.21.15.60:0/615184188 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c001670 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca1d9b640 1 -- 172.21.15.60:0/615184188 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9c02c8b0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca1d9b640 1 -- 172.21.15.60:0/615184188 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c02cc40 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca359e640 1 -- 172.21.15.60:0/615184188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca40ca0a0 msgr2=0x7f2ca406c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca359e640 1 --2- 172.21.15.60:0/615184188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca406c810 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009d90 tx=0x7f2c9c02bd80 comp rx=0 tx=0).stop 2023-10-16T14:13:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca359e640 1 -- 172.21.15.60:0/615184188 shutdown_connections 2023-10-16T14:13:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca359e640 1 --2- 172.21.15.60:0/615184188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca406c810 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.586+0000 7f2ca359e640 1 -- 172.21.15.60:0/615184188 >> 172.21.15.60:0/615184188 conn(0x7f2ca40ce6a0 msgr2=0x7f2ca40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.587+0000 7f2ca359e640 1 -- 172.21.15.60:0/615184188 shutdown_connections 2023-10-16T14:13:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.587+0000 7f2ca359e640 1 -- 172.21.15.60:0/615184188 wait complete. 2023-10-16T14:13:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.589+0000 7f2ca359e640 1 Processor -- start 2023-10-16T14:13:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.589+0000 7f2ca359e640 1 -- start start 2023-10-16T14:13:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.590+0000 7f2ca359e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ca4070b60 0x7f2ca40c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca4154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca4159950 0x7f2ca415b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca359e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40b1750 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca359e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40b1a30 con 0x7f2ca4159950 2023-10-16T14:13:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca359e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40b1d10 con 0x7f2ca4070b60 2023-10-16T14:13:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca4154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca4154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51524/0 (socket says 172.21.15.60:51524) 2023-10-16T14:13:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 -- 172.21.15.60:0/3178209081 learned_addr learned my addr 172.21.15.60:0/3178209081 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca2d9d640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca4159950 0x7f2ca415b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2ca259c640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ca4070b60 0x7f2ca40c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 -- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ca4070b60 msgr2=0x7f2ca40c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ca4070b60 0x7f2ca40c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 -- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca4159950 msgr2=0x7f2ca415b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca4159950 0x7f2ca415b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.591+0000 7f2c9bfff640 1 -- 172.21.15.60:0/3178209081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c009a70 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.592+0000 7f2c9bfff640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca4154ec0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00c760 tx=0x7f2c8c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.592+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c005ad0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.592+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c005c30 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.592+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c007070 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.592+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca40b1ff0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.592+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca40c53b0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.594+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca40af180 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.595+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2c8c0071d0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.597+0000 7f2c9affd640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2c8406af00 0x7f2c8406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.597+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f2c8c0897b0 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.598+0000 7f2ca259c640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2c8406af00 0x7f2c8406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.599+0000 7f2ca259c640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2c8406af00 0x7f2c8406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c94000f60 tx=0x7f2c94009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.602+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ca40af180 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.840+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ca40af180 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.841+0000 7f2c9affd640 1 -- 172.21.15.60:0/3178209081 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f2ca40af180 con 0x7f2ca40ca0a0 2023-10-16T14:13:51.845 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:51.846 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:13:23.271856+0000","modified":"2023-10-16T14:13:26.430490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13065":{"gid":13065,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2549968691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2549968691},{"type":"v1","addr":"172.21.15.60:6835","nonce":2549968691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T14:13:51.846 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 152 2023-10-16T14:13:51.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2c8406af00 msgr2=0x7f2c8406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2c8406af00 0x7f2c8406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c94000f60 tx=0x7f2c94009040 comp rx=0 tx=0).stop 2023-10-16T14:13:51.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 msgr2=0x7f2ca4154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:51.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca4154ec0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00c760 tx=0x7f2c8c00cc20 comp rx=0 tx=0).stop 2023-10-16T14:13:51.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 shutdown_connections 2023-10-16T14:13:51.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2c8406af00 0x7f2c8406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ca4159950 0x7f2ca415b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca40ca0a0 0x7f2ca4154ec0 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 --2- 172.21.15.60:0/3178209081 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2ca4070b60 0x7f2ca40c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:51.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.844+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 >> 172.21.15.60:0/3178209081 conn(0x7f2ca40ce6a0 msgr2=0x7f2ca4070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:51.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.845+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 shutdown_connections 2023-10-16T14:13:51.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:51.845+0000 7f2ca359e640 1 -- 172.21.15.60:0/3178209081 wait complete. 2023-10-16T14:13:51.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:13:52.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.083+0000 7faba3213640 1 Processor -- start 2023-10-16T14:13:52.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.083+0000 7faba3213640 1 -- start start 2023-10-16T14:13:52.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba3213640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab9c06c040 0x7fab9c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba3213640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fab9c0cda80 0x7fab9c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba3213640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c06c960 con 0x7fab9c0ad970 2023-10-16T14:13:52.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba3213640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c15b9d0 con 0x7fab9c06c040 2023-10-16T14:13:52.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba3213640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c15bcb0 con 0x7fab9c0cda80 2023-10-16T14:13:52.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2211640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab9c06c040 0x7fab9c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2211640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab9c06c040 0x7fab9c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51536/0 (socket says 172.21.15.60:51536) 2023-10-16T14:13:52.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2211640 1 -- 172.21.15.60:0/1138369042 learned_addr learned my addr 172.21.15.60:0/1138369042 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:52.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2a12640 1 --2- 172.21.15.60:0/1138369042 >> v2:172.21.15.144:3300/0 conn(0x7fab9c0cda80 0x7fab9c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2211640 1 -- 172.21.15.60:0/1138369042 >> v2:172.21.15.144:3300/0 conn(0x7fab9c0cda80 msgr2=0x7fab9c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2211640 1 --2- 172.21.15.60:0/1138369042 >> v2:172.21.15.144:3300/0 conn(0x7fab9c0cda80 0x7fab9c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.085+0000 7faba2211640 1 -- 172.21.15.60:0/1138369042 >> v1:172.21.15.144:6789/0 conn(0x7fab9c0ad970 legacy=0x7fab9c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:52.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.086+0000 7faba2211640 1 -- 172.21.15.60:0/1138369042 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c15bf90 con 0x7fab9c06c040 2023-10-16T14:13:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.087+0000 7faba2211640 1 --2- 172.21.15.60:0/1138369042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c06c420 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0cf300 tx=0x7fab9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2999821493f0c565 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba120f640 1 -- 172.21.15.60:0/1138369042 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98031870 con 0x7fab9c06c040 2023-10-16T14:13:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba120f640 1 -- 172.21.15.60:0/1138369042 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab980319d0 con 0x7fab9c06c040 2023-10-16T14:13:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba120f640 1 -- 172.21.15.60:0/1138369042 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98031d60 con 0x7fab9c06c040 2023-10-16T14:13:52.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 -- 172.21.15.60:0/1138369042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 msgr2=0x7fab9c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 --2- 172.21.15.60:0/1138369042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c06c420 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0cf300 tx=0x7fab9802b910 comp rx=0 tx=0).stop 2023-10-16T14:13:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 -- 172.21.15.60:0/1138369042 shutdown_connections 2023-10-16T14:13:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 --2- 172.21.15.60:0/1138369042 >> v2:172.21.15.144:3300/0 conn(0x7fab9c0cda80 0x7fab9c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 --2- 172.21.15.60:0/1138369042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c06c420 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 -- 172.21.15.60:0/1138369042 >> 172.21.15.60:0/1138369042 conn(0x7fab9c0ce6a0 msgr2=0x7fab9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:52.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.089+0000 7faba3213640 1 -- 172.21.15.60:0/1138369042 shutdown_connections 2023-10-16T14:13:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.090+0000 7faba3213640 1 -- 172.21.15.60:0/1138369042 wait complete. 2023-10-16T14:13:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.091+0000 7faba3213640 1 Processor -- start 2023-10-16T14:13:52.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.091+0000 7faba3213640 1 -- start start 2023-10-16T14:13:52.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.092+0000 7faba3213640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c0c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.092+0000 7faba3213640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab9c0ad970 0x7fab9c0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.092+0000 7faba3213640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab9c0cda80 0x7fab9c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.092+0000 7faba3213640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0c8310 con 0x7fab9c06c040 2023-10-16T14:13:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.092+0000 7faba3213640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0afa40 con 0x7fab9c0cda80 2023-10-16T14:13:52.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.092+0000 7faba3213640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c0c8030 con 0x7fab9c0ad970 2023-10-16T14:13:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c0c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c0c76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51542/0 (socket says 172.21.15.60:51542) 2023-10-16T14:13:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 -- 172.21.15.60:0/2143277702 learned_addr learned my addr 172.21.15.60:0/2143277702 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:52.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 -- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab9c0ad970 msgr2=0x7fab9c0c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab9c0ad970 0x7fab9c0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 -- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab9c0cda80 msgr2=0x7fab9c0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab9c0cda80 0x7fab9c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 -- 172.21.15.60:0/2143277702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab980094d0 con 0x7fab9c06c040 2023-10-16T14:13:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.093+0000 7faba2211640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c0c76e0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fab9802b950 tx=0x7fab980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.095+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9803ad30 con 0x7fab9c06c040 2023-10-16T14:13:52.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.095+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab980443f0 con 0x7fab9c06c040 2023-10-16T14:13:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.095+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab980447a0 con 0x7fab9c06c040 2023-10-16T14:13:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.095+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c0af3f0 con 0x7fab9c06c040 2023-10-16T14:13:52.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.096+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c0af0a0 con 0x7fab9c06c040 2023-10-16T14:13:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.097+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c06b330 con 0x7fab9c06c040 2023-10-16T14:13:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.098+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fab98044900 con 0x7fab9c06c040 2023-10-16T14:13:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.099+0000 7fab92ffd640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fab8006af00 0x7fab8006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.099+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7fab9802d040 con 0x7fab9c06c040 2023-10-16T14:13:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.100+0000 7faba1a10640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fab8006af00 0x7fab8006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.101+0000 7faba1a10640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fab8006af00 0x7fab8006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0c7690 tx=0x7fab8c0079b0 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:52.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.104+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab98076dc0 con 0x7fab9c06c040 2023-10-16T14:13:52.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.338+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab9c0c98c0 con 0x7fab9c06c040 2023-10-16T14:13:52.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.339+0000 7fab92ffd640 1 -- 172.21.15.60:0/2143277702 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fab9807a3e0 con 0x7fab9c06c040 2023-10-16T14:13:52.343 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:52.343 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":152,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T14:13:23.271856+0000","modified":"2023-10-16T14:13:26.430490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13065":{"gid":13065,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2549968691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2549968691},{"type":"v1","addr":"172.21.15.60:6835","nonce":2549968691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T14:13:52.344 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 152 2023-10-16T14:13:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fab8006af00 msgr2=0x7fab8006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fab8006af00 0x7fab8006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0c7690 tx=0x7fab8c0079b0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 msgr2=0x7fab9c0c76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c0c76e0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fab9802b950 tx=0x7fab980046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 shutdown_connections 2023-10-16T14:13:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fab8006af00 0x7fab8006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fab9c0ad970 0x7fab9c0c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fab9c0cda80 0x7fab9c0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 --2- 172.21.15.60:0/2143277702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab9c06c040 0x7fab9c0c76e0 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 >> 172.21.15.60:0/2143277702 conn(0x7fab9c0ce6a0 msgr2=0x7fab9c0c91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 shutdown_connections 2023-10-16T14:13:52.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.342+0000 7faba3213640 1 -- 172.21.15.60:0/2143277702 wait complete. 2023-10-16T14:13:52.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:13:52.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.575+0000 7f339e4af640 1 Processor -- start 2023-10-16T14:13:52.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.575+0000 7f339e4af640 1 -- start start 2023-10-16T14:13:52.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.577+0000 7f339e4af640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f339815ae80 0x7f339815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.577+0000 7f339e4af640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339815ed60 con 0x7f33980ce390 2023-10-16T14:13:52.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.577+0000 7f339e4af640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33980707e0 con 0x7f339815aaa0 2023-10-16T14:13:52.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.577+0000 7f339e4af640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339806b810 con 0x7f339815ae80 2023-10-16T14:13:52.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.577+0000 7f338ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f339815ae80 0x7f339815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.577+0000 7f338ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f339815ae80 0x7f339815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51554/0 (socket says 172.21.15.60:51554) 2023-10-16T14:13:52.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.578+0000 7f338ffff640 1 -- 172.21.15.60:0/1937934920 learned_addr learned my addr 172.21.15.60:0/1937934920 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:52.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.578+0000 7f338ffff640 1 -- 172.21.15.60:0/1937934920 >> v1:172.21.15.144:6789/0 conn(0x7f339815aaa0 legacy=0x7f33980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:52.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.578+0000 7f338ffff640 1 -- 172.21.15.60:0/1937934920 >> v1:172.21.15.60:6789/0 conn(0x7f33980ce390 legacy=0x7f3398070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.578+0000 7f338ffff640 1 -- 172.21.15.60:0/1937934920 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339815ef90 con 0x7f339815ae80 2023-10-16T14:13:52.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.579+0000 7f338ffff640 1 --2- 172.21.15.60:0/1937934920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f339815ae80 0x7f339815e820 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f3380009870 tx=0x7f338002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b6f316994a87d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:52.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.579+0000 7f33977fe640 1 -- 172.21.15.60:0/1937934920 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380030860 con 0x7f339815ae80 2023-10-16T14:13:52.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f33977fe640 1 -- 172.21.15.60:0/1937934920 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33800309c0 con 0x7f339815ae80 2023-10-16T14:13:52.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f33977fe640 1 -- 172.21.15.60:0/1937934920 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380030d50 con 0x7f339815ae80 2023-10-16T14:13:52.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 -- 172.21.15.60:0/1937934920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f339815ae80 msgr2=0x7f339815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 --2- 172.21.15.60:0/1937934920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f339815ae80 0x7f339815e820 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f3380009870 tx=0x7f338002b9a0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 -- 172.21.15.60:0/1937934920 shutdown_connections 2023-10-16T14:13:52.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 --2- 172.21.15.60:0/1937934920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f339815ae80 0x7f339815e820 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 -- 172.21.15.60:0/1937934920 >> 172.21.15.60:0/1937934920 conn(0x7f33980cb410 msgr2=0x7f33980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:52.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 -- 172.21.15.60:0/1937934920 shutdown_connections 2023-10-16T14:13:52.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.580+0000 7f339e4af640 1 -- 172.21.15.60:0/1937934920 wait complete. 2023-10-16T14:13:52.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.582+0000 7f339e4af640 1 Processor -- start 2023-10-16T14:13:52.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.582+0000 7f339e4af640 1 -- start start 2023-10-16T14:13:52.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f339e4af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 0x7f3398145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f339e4af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339815aaa0 0x7f33980c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f339e4af640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f339815ae80 0x7f33980c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f339e4af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3398147830 con 0x7f33980ce390 2023-10-16T14:13:52.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f339e4af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3398146c20 con 0x7f339815aaa0 2023-10-16T14:13:52.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f339e4af640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3398146f00 con 0x7f339815ae80 2023-10-16T14:13:52.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f3397fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 0x7f3398145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f3397fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 0x7f3398145ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51564/0 (socket says 172.21.15.60:51564) 2023-10-16T14:13:52.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.583+0000 7f3397fff640 1 -- 172.21.15.60:0/2760646311 learned_addr learned my addr 172.21.15.60:0/2760646311 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:52.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f3397fff640 1 -- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f339815ae80 msgr2=0x7f33980c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f3397fff640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f339815ae80 0x7f33980c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f3397fff640 1 -- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339815aaa0 msgr2=0x7f33980c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f3397fff640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339815aaa0 0x7f33980c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f3397fff640 1 -- 172.21.15.60:0/2760646311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800094d0 con 0x7f33980ce390 2023-10-16T14:13:52.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f3397fff640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 0x7f3398145ea0 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f338800c430 tx=0x7f338800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:52.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388011790 con 0x7f33980ce390 2023-10-16T14:13:52.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33880118f0 con 0x7f33980ce390 2023-10-16T14:13:52.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388011ce0 con 0x7f33980ce390 2023-10-16T14:13:52.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.584+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33981471e0 con 0x7f33980ce390 2023-10-16T14:13:52.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.585+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3398146860 con 0x7f33980ce390 2023-10-16T14:13:52.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.586+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33981469f0 con 0x7f33980ce390 2023-10-16T14:13:52.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.587+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f338801c460 con 0x7f33980ce390 2023-10-16T14:13:52.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.588+0000 7f33957fa640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f337806af30 0x7f337806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:52.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.588+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f3388089960 con 0x7f33980ce390 2023-10-16T14:13:52.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.589+0000 7f338ffff640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f337806af30 0x7f337806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:52.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.590+0000 7f338ffff640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f337806af30 0x7f337806d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3380036000 tx=0x7f338003b040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:52.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.593+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3388054680 con 0x7f33980ce390 2023-10-16T14:13:52.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.793+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f339804e160 con 0x7f33980ce390 2023-10-16T14:13:52.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.796+0000 7f33957fa640 1 -- 172.21.15.60:0/2760646311 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v289) v1 ==== 74+0+17719 (secure 0 0 0) 0x7f3388057ca0 con 0x7f33980ce390 2023-10-16T14:13:52.800 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:13:52.800 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":289,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:49.056966+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2023-10-16T14:13:21.237677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2023-10-16T14:13:22.301769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/1784354286":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/758300120":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/1291769788":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/1290683266":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3884644447":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3738039093":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/579488795":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3615094779":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/1408575976":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/277289582":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/2718528188":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/4157805917":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/2109842343":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/306505374":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6833/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/1784354286":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3309380401":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3276093492":"2023-10-17T14:13:38.050732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:13:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f337806af30 msgr2=0x7f337806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f337806af30 0x7f337806d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3380036000 tx=0x7f338003b040 comp rx=0 tx=0).stop 2023-10-16T14:13:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 msgr2=0x7f3398145ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 0x7f3398145ea0 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f338800c430 tx=0x7f338800c8f0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 shutdown_connections 2023-10-16T14:13:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f337806af30 0x7f337806d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f339815ae80 0x7f33980c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339815aaa0 0x7f33980c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 --2- 172.21.15.60:0/2760646311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33980ce390 0x7f3398145ea0 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 >> 172.21.15.60:0/2760646311 conn(0x7f33980cb410 msgr2=0x7f33980b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:52.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 shutdown_connections 2023-10-16T14:13:52.805 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:52.799+0000 7f339e4af640 1 -- 172.21.15.60:0/2760646311 wait complete. 2023-10-16T14:13:52.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T14:13:53.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.041+0000 7f96b8378640 1 Processor -- start 2023-10-16T14:13:53.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.041+0000 7f96b8378640 1 -- start start 2023-10-16T14:13:53.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b8378640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b00caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b8378640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96b00c9000 0x7f96b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b8378640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00cb140 con 0x7f96b006a2e0 2023-10-16T14:13:53.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b8378640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b0161a50 con 0x7f96b00c9000 2023-10-16T14:13:53.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b8378640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b006a750 con 0x7f96b00addb0 2023-10-16T14:13:53.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b6114640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b00caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b6114640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b00caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3301/0 says I am v2:172.21.15.60:56830/0 (socket says 172.21.15.60:56830) 2023-10-16T14:13:53.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b6114640 1 -- 172.21.15.60:0/2920157976 learned_addr learned my addr 172.21.15.60:0/2920157976 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:53.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.043+0000 7f96b5913640 1 --2- 172.21.15.60:0/2920157976 >> v2:172.21.15.144:3300/0 conn(0x7f96b00c9000 0x7f96b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.044+0000 7f96b5913640 1 -- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 msgr2=0x7f96b00caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:53.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.044+0000 7f96b5913640 1 --2- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b00caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.044+0000 7f96b5913640 1 -- 172.21.15.60:0/2920157976 >> v1:172.21.15.144:6789/0 conn(0x7f96b006a2e0 legacy=0x7f96b00cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.044+0000 7f96b5913640 1 -- 172.21.15.60:0/2920157976 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b006b030 con 0x7f96b00c9000 2023-10-16T14:13:53.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.046+0000 7f96b5913640 1 --2- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00c9000 0x7f96b0069da0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f96a00097d0 tx=0x7f96a002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbaada2d2d0c2df0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:53.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.046+0000 7f96b5112640 1 -- 172.21.15.60:0/2920157976 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a002d040 con 0x7f96b00c9000 2023-10-16T14:13:53.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.046+0000 7f96b5112640 1 -- 172.21.15.60:0/2920157976 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a0031810 con 0x7f96b00c9000 2023-10-16T14:13:53.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.046+0000 7f96b5112640 1 -- 172.21.15.60:0/2920157976 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0031ba0 con 0x7f96b00c9000 2023-10-16T14:13:53.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 -- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00c9000 msgr2=0x7f96b0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 --2- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00c9000 0x7f96b0069da0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f96a00097d0 tx=0x7f96a002b8b0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 -- 172.21.15.60:0/2920157976 shutdown_connections 2023-10-16T14:13:53.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 --2- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b00c9000 0x7f96b0069da0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 --2- 172.21.15.60:0/2920157976 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b00caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 -- 172.21.15.60:0/2920157976 >> 172.21.15.60:0/2920157976 conn(0x7f96b00ce9b0 msgr2=0x7f96b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:53.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 -- 172.21.15.60:0/2920157976 shutdown_connections 2023-10-16T14:13:53.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.047+0000 7f96b8378640 1 -- 172.21.15.60:0/2920157976 wait complete. 2023-10-16T14:13:53.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.049+0000 7f96b8378640 1 Processor -- start 2023-10-16T14:13:53.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.049+0000 7f96b8378640 1 -- start start 2023-10-16T14:13:53.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b8378640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b006a2e0 0x7f96b00c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b8378640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b01fa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b8378640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 0x7f96b00b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b8378640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00c6ad0 con 0x7f96b00c9000 2023-10-16T14:13:53.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b8378640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b01fa860 con 0x7f96b006a2e0 2023-10-16T14:13:53.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b8378640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b01faa90 con 0x7f96b00addb0 2023-10-16T14:13:53.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.050+0000 7f96b6915640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 0x7f96b00b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 0x7f96b00b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51576/0 (socket says 172.21.15.60:51576) 2023-10-16T14:13:53.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 -- 172.21.15.60:0/1039442491 learned_addr learned my addr 172.21.15.60:0/1039442491 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:53.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6114640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b006a2e0 0x7f96b00c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b5913640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b01fa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 -- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 msgr2=0x7f96b01fa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b01fa320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 -- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b006a2e0 msgr2=0x7f96b00c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b006a2e0 0x7f96b00c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 -- 172.21.15.60:0/1039442491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0009400 con 0x7f96b00c9000 2023-10-16T14:13:53.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6114640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b006a2e0 0x7f96b00c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:53.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96b6915640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 0x7f96b00b15a0 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f96a800c7a0 tx=0x7f96a800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:53.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.051+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8003b70 con 0x7f96b00c9000 2023-10-16T14:13:53.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.052+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8003cf0 con 0x7f96b00c9000 2023-10-16T14:13:53.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.052+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b01fad70 con 0x7f96b00c9000 2023-10-16T14:13:53.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.052+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8003ea0 con 0x7f96b00c9000 2023-10-16T14:13:53.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.052+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b00c6ce0 con 0x7f96b00c9000 2023-10-16T14:13:53.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.053+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b00c6ed0 con 0x7f96b00c9000 2023-10-16T14:13:53.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.054+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f96a800aa90 con 0x7f96b00c9000 2023-10-16T14:13:53.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.056+0000 7f96a6ffd640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f969406af00 0x7f969406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.056+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f96a8089870 con 0x7f96b00c9000 2023-10-16T14:13:53.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.057+0000 7f96b6114640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f969406af00 0x7f969406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.058+0000 7f96b6114640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f969406af00 0x7f969406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009ac0 tx=0x7f96ac008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:53.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.061+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96a8054580 con 0x7f96b00c9000 2023-10-16T14:13:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.106+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f96a800ad40 con 0x7f96b00c9000 2023-10-16T14:13:53.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.299+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f96b01fb290 con 0x7f96b00c9000 2023-10-16T14:13:53.304 INFO:tasks.ceph.mon.a.smithi060.stderr:2023-10-16T14:13:53.300+0000 7fcf618fe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T14:13:53.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.304+0000 7f96a6ffd640 1 -- 172.21.15.60:0/1039442491 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v153) v1 ==== 161+0+0 (secure 0 0 0) 0x7f96b01fb290 con 0x7f96b00c9000 2023-10-16T14:13:53.308 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T14:13:53.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f969406af00 msgr2=0x7f969406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f969406af00 0x7f969406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009ac0 tx=0x7f96ac008040 comp rx=0 tx=0).stop 2023-10-16T14:13:53.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 msgr2=0x7f96b00b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 0x7f96b00b15a0 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f96a800c7a0 tx=0x7f96a800cc60 comp rx=0 tx=0).stop 2023-10-16T14:13:53.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 shutdown_connections 2023-10-16T14:13:53.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f969406af00 0x7f969406d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f96b00addb0 0x7f96b01fa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b00c9000 0x7f96b00b15a0 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 --2- 172.21.15.60:0/1039442491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96b006a2e0 0x7f96b00c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 >> 172.21.15.60:0/1039442491 conn(0x7f96b00ce9b0 msgr2=0x7f96b0071040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:53.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 shutdown_connections 2023-10-16T14:13:53.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.307+0000 7f96b8378640 1 -- 172.21.15.60:0/1039442491 wait complete. 2023-10-16T14:13:53.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T14:13:53.389 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2023-10-16T14:13:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.538+0000 7fda370ad640 1 Processor -- start 2023-10-16T14:13:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.538+0000 7fda370ad640 1 -- start start 2023-10-16T14:13:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda370ad640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fda300cded0 0x7fda300b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda370ad640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3015d150 con 0x7fda3015cd70 2023-10-16T14:13:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda370ad640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda300d1180 con 0x7fda300b8480 2023-10-16T14:13:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda370ad640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda300d1460 con 0x7fda300cded0 2023-10-16T14:13:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda360ab640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fda3015cd70 0x7fda300d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60360/0 (socket says 172.21.15.60:60360) 2023-10-16T14:13:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda360ab640 1 -- 172.21.15.60:0/3239540743 learned_addr learned my addr 172.21.15.60:0/3239540743 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806171598 0 0) 0x7fda3015d150 con 0x7fda3015cd70 2023-10-16T14:13:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda368ac640 1 --2- 172.21.15.60:0/3239540743 >> v2:172.21.15.144:3300/0 conn(0x7fda300cded0 0x7fda300b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.540+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda18003520 con 0x7fda3015cd70 2023-10-16T14:13:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979947584 0 0) 0x7fda18003520 con 0x7fda3015cd70 2023-10-16T14:13:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 >> v2:172.21.15.144:3300/0 conn(0x7fda300cded0 msgr2=0x7fda300b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 --2- 172.21.15.60:0/3239540743 >> v2:172.21.15.144:3300/0 conn(0x7fda300cded0 0x7fda300b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 >> v1:172.21.15.144:6789/0 conn(0x7fda300b8480 legacy=0x7fda300cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300d1740 con 0x7fda3015cd70 2023-10-16T14:13:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fda200019f0 con 0x7fda3015cd70 2023-10-16T14:13:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 >> v1:172.21.15.60:6789/0 conn(0x7fda3015cd70 legacy=0x7fda300d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda18003970 0x7fda18003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda18005180 0x7fda1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300d1460 con 0x7fda18005180 2023-10-16T14:13:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1800c0b0 con 0x7fda180047c0 2023-10-16T14:13:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1800c270 con 0x7fda18003970 2023-10-16T14:13:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.541+0000 7fda368ac640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda18005180 0x7fda1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda358aa640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda360ab640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda18003970 0x7fda18003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda368ac640 1 -- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda18003970 msgr2=0x7fda18003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda368ac640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda18003970 0x7fda18003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda368ac640 1 -- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda180047c0 msgr2=0x7fda18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda368ac640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda368ac640 1 -- 172.21.15.60:0/3239540743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18003520 con 0x7fda18005180 2023-10-16T14:13:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda358aa640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda180047c0 0x7fda18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.542+0000 7fda360ab640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda18003970 0x7fda18003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.543+0000 7fda368ac640 1 -- 172.21.15.60:0/3239540743 reap_dead start 2023-10-16T14:13:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.543+0000 7fda368ac640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda18005180 0x7fda1800b980 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fda2800b9e0 tx=0x7fda180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7187a99a3d02d2dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.543+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28032620 con 0x7fda18005180 2023-10-16T14:13:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.543+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28032780 con 0x7fda18005180 2023-10-16T14:13:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.543+0000 7fda350a9640 1 -- 172.21.15.60:0/3239540743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28032b30 con 0x7fda18005180 2023-10-16T14:13:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 -- 172.21.15.60:0/3239540743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda18005180 msgr2=0x7fda1800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda18005180 0x7fda1800b980 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fda2800b9e0 tx=0x7fda180057f0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 -- 172.21.15.60:0/3239540743 shutdown_connections 2023-10-16T14:13:53.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 --2- 172.21.15.60:0/3239540743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda18005180 0x7fda1800b980 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 -- 172.21.15.60:0/3239540743 >> 172.21.15.60:0/3239540743 conn(0x7fda300cf570 msgr2=0x7fda300cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:53.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 -- 172.21.15.60:0/3239540743 shutdown_connections 2023-10-16T14:13:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.544+0000 7fda370ad640 1 -- 172.21.15.60:0/3239540743 wait complete. 2023-10-16T14:13:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.545+0000 7fda370ad640 1 Processor -- start 2023-10-16T14:13:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.546+0000 7fda370ad640 1 -- start start 2023-10-16T14:13:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda370ad640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda18005180 0x7fda300cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda370ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 0x7fda300d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda370ad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda300d2fc0 0x7fda3015e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda370ad640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300d3470 con 0x7fda300cc7d0 2023-10-16T14:13:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda370ad640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300c9bc0 con 0x7fda18005180 2023-10-16T14:13:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda370ad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3015f6e0 con 0x7fda300d2fc0 2023-10-16T14:13:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 0x7fda300d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 0x7fda300d2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51606/0 (socket says 172.21.15.60:51606) 2023-10-16T14:13:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 -- 172.21.15.60:0/2887979822 learned_addr learned my addr 172.21.15.60:0/2887979822 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda360ab640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda18005180 0x7fda300cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 -- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda300d2fc0 msgr2=0x7fda3015e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda300d2fc0 0x7fda3015e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 -- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda18005180 msgr2=0x7fda300cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda18005180 0x7fda300cc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda358aa640 1 -- 172.21.15.60:0/2887979822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300c9e80 con 0x7fda300cc7d0 2023-10-16T14:13:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.547+0000 7fda360ab640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda18005180 0x7fda300cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.548+0000 7fda358aa640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 0x7fda300d2a80 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7fda2400d590 tx=0x7fda2400da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:53.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.548+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24007220 con 0x7fda300cc7d0 2023-10-16T14:13:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.548+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda240073a0 con 0x7fda300cc7d0 2023-10-16T14:13:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.548+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda240067d0 con 0x7fda300cc7d0 2023-10-16T14:13:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.548+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda300ca160 con 0x7fda300cc7d0 2023-10-16T14:13:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.548+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda300ca960 con 0x7fda300cc7d0 2023-10-16T14:13:53.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.549+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda300ca680 con 0x7fda300cc7d0 2023-10-16T14:13:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.550+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fda24006930 con 0x7fda300cc7d0 2023-10-16T14:13:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.551+0000 7fda2effd640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fda1c06afc0 0x7fda1c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.551+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6612+0+0 (secure 0 0 0) 0x7fda240899e0 con 0x7fda300cc7d0 2023-10-16T14:13:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.552+0000 7fda360ab640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fda1c06afc0 0x7fda1c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.553+0000 7fda360ab640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fda1c06afc0 0x7fda1c06d480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda20001350 tx=0x7fda20009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:53.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.555+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda300ca680 con 0x7fda300cc7d0 2023-10-16T14:13:53.785 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:53.781+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fda300ca680 con 0x7fda300cc7d0 2023-10-16T14:13:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.307+0000 7fda2effd640 1 -- 172.21.15.60:0/2887979822 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v154) v1 ==== 104+0+0 (secure 0 0 0) 0x7fda300ca680 con 0x7fda300cc7d0 2023-10-16T14:13:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.310+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fda1c06afc0 msgr2=0x7fda1c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.310+0000 7fda370ad640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fda1c06afc0 0x7fda1c06d480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda20001350 tx=0x7fda20009040 comp rx=0 tx=0).stop 2023-10-16T14:13:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.310+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 msgr2=0x7fda300d2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.310+0000 7fda370ad640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 0x7fda300d2a80 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7fda2400d590 tx=0x7fda2400da50 comp rx=0 tx=0).stop 2023-10-16T14:13:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 shutdown_connections 2023-10-16T14:13:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fda1c06afc0 0x7fda1c06d480 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fda300d2fc0 0x7fda3015e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda300cc7d0 0x7fda300d2a80 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 --2- 172.21.15.60:0/2887979822 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda18005180 0x7fda300cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 >> 172.21.15.60:0/2887979822 conn(0x7fda300cf570 msgr2=0x7fda300b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 shutdown_connections 2023-10-16T14:13:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.311+0000 7fda370ad640 1 -- 172.21.15.60:0/2887979822 wait complete. 2023-10-16T14:13:54.339 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T14:13:54.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.569+0000 7f2dd83fb640 1 Processor -- start 2023-10-16T14:13:54.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.569+0000 7f2dd83fb640 1 -- start start 2023-10-16T14:13:54.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.570+0000 7f2dd83fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dd01592e0 0x7f2dd006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:54.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd83fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:54.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd83fb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd01596c0 con 0x7f2dd006afc0 2023-10-16T14:13:54.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd83fb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd006c6c0 con 0x7f2dd01592e0 2023-10-16T14:13:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd83fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd00cc860 con 0x7f2dd00c9640 2023-10-16T14:13:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dd01592e0 0x7f2dd006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dd01592e0 0x7f2dd006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51618/0 (socket says 172.21.15.60:51618) 2023-10-16T14:13:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 -- 172.21.15.60:0/3535909691 learned_addr learned my addr 172.21.15.60:0/3535909691 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6998640 1 --2- 172.21.15.60:0/3535909691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 -- 172.21.15.60:0/3535909691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 msgr2=0x7f2dd00cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 --2- 172.21.15.60:0/3535909691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 -- 172.21.15.60:0/3535909691 >> v1:172.21.15.144:6789/0 conn(0x7f2dd006afc0 legacy=0x7f2dd00c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.571+0000 7f2dd6197640 1 -- 172.21.15.60:0/3535909691 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00ccb40 con 0x7f2dd01592e0 2023-10-16T14:13:54.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.573+0000 7f2dd6197640 1 --2- 172.21.15.60:0/3535909691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd01592e0 0x7f2dd006bea0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009e50 tx=0x7f2dcc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c06c3580ff6266 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:54.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.573+0000 7f2dd5195640 1 -- 172.21.15.60:0/3535909691 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc02ca70 con 0x7f2dd01592e0 2023-10-16T14:13:54.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.573+0000 7f2dd5195640 1 -- 172.21.15.60:0/3535909691 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc02cbd0 con 0x7f2dd01592e0 2023-10-16T14:13:54.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.573+0000 7f2dd5195640 1 -- 172.21.15.60:0/3535909691 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0343f0 con 0x7f2dd01592e0 2023-10-16T14:13:54.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.573+0000 7f2dd83fb640 1 -- 172.21.15.60:0/3535909691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd01592e0 msgr2=0x7f2dd006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:54.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.573+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/3535909691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd01592e0 0x7f2dd006bea0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009e50 tx=0x7f2dcc02bed0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.574+0000 7f2dd83fb640 1 -- 172.21.15.60:0/3535909691 shutdown_connections 2023-10-16T14:13:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.574+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/3535909691 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.574+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/3535909691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd01592e0 0x7f2dd006bea0 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.574+0000 7f2dd83fb640 1 -- 172.21.15.60:0/3535909691 >> 172.21.15.60:0/3535909691 conn(0x7f2dd00cb360 msgr2=0x7f2dd00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.574+0000 7f2dd83fb640 1 -- 172.21.15.60:0/3535909691 shutdown_connections 2023-10-16T14:13:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.574+0000 7f2dd83fb640 1 -- 172.21.15.60:0/3535909691 wait complete. 2023-10-16T14:13:54.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.576+0000 7f2dd83fb640 1 Processor -- start 2023-10-16T14:13:54.582 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.576+0000 7f2dd83fb640 1 -- start start 2023-10-16T14:13:54.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd83fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 0x7f2dd01486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:54.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd83fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:54.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd83fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2dd0148e10 0x7f2dd014ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:54.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd83fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd0148bf0 con 0x7f2dd006afc0 2023-10-16T14:13:54.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd83fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd015acd0 con 0x7f2dd0148e10 2023-10-16T14:13:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd83fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd015afb0 con 0x7f2dd00c9640 2023-10-16T14:13:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 0x7f2dd01486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 0x7f2dd01486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51632/0 (socket says 172.21.15.60:51632) 2023-10-16T14:13:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 -- 172.21.15.60:0/2643359590 learned_addr learned my addr 172.21.15.60:0/2643359590 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd5996640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6998640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2dd0148e10 0x7f2dd014ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 -- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 msgr2=0x7f2dd00c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 -- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2dd0148e10 msgr2=0x7f2dd014ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2dd0148e10 0x7f2dd014ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.577+0000 7f2dd6197640 1 -- 172.21.15.60:0/2643359590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc009b30 con 0x7f2dd006afc0 2023-10-16T14:13:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.578+0000 7f2dd6998640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2dd0148e10 0x7f2dd014ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.578+0000 7f2dd6197640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 0x7f2dd01486b0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc02c460 tx=0x7f2dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.578+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc009510 con 0x7f2dd006afc0 2023-10-16T14:13:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.578+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc03e450 con 0x7f2dd006afc0 2023-10-16T14:13:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.578+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd015b290 con 0x7f2dd006afc0 2023-10-16T14:13:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.578+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc03e820 con 0x7f2dd006afc0 2023-10-16T14:13:54.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.579+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd015b850 con 0x7f2dd006afc0 2023-10-16T14:13:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.579+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd004e160 con 0x7f2dd006afc0 2023-10-16T14:13:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.581+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2dcc03e980 con 0x7f2dd006afc0 2023-10-16T14:13:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.581+0000 7f2dc6ffd640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2db406af90 0x7f2db406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.582+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(290..290 src has 1..290) v4 ==== 6612+0+0 (secure 0 0 0) 0x7f2dcc0ac9b0 con 0x7f2dd006afc0 2023-10-16T14:13:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.582+0000 7f2dd5996640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2db406af90 0x7f2db406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.583+0000 7f2dd5996640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2db406af90 0x7f2db406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00c2d20 tx=0x7f2dc0009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:54.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.586+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2dcc077520 con 0x7f2dd006afc0 2023-10-16T14:13:54.787 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:54.783+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2dd00c1aa0 con 0x7f2dd006afc0 2023-10-16T14:13:55.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.317+0000 7f2dc6ffd640 1 -- 172.21.15.60:0/2643359590 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v291) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2dcc07ab40 con 0x7f2dd006afc0 2023-10-16T14:13:55.321 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2023-10-16T14:13:55.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2db406af90 msgr2=0x7f2db406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:55.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2db406af90 0x7f2db406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00c2d20 tx=0x7f2dc0009040 comp rx=0 tx=0).stop 2023-10-16T14:13:55.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 msgr2=0x7f2dd01486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:55.326 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 0x7f2dd01486b0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc02c460 tx=0x7f2dcc004a40 comp rx=0 tx=0).stop 2023-10-16T14:13:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 shutdown_connections 2023-10-16T14:13:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f2db406af90 0x7f2db406d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f2dd00c9640 0x7f2dd00c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2dd0148e10 0x7f2dd014ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 --2- 172.21.15.60:0/2643359590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dd006afc0 0x7f2dd01486b0 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.322+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 >> 172.21.15.60:0/2643359590 conn(0x7f2dd00cb360 msgr2=0x7f2dd00aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:55.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.323+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 shutdown_connections 2023-10-16T14:13:55.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.323+0000 7f2dd83fb640 1 -- 172.21.15.60:0/2643359590 wait complete. 2023-10-16T14:13:55.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T14:13:55.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.556+0000 7f1e475fb640 1 Processor -- start 2023-10-16T14:13:55.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.556+0000 7f1e475fb640 1 -- start start 2023-10-16T14:13:55.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.557+0000 7f1e475fb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1e4015cf50 0x7f1e400d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.557+0000 7f1e475fb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e400b8a10 0x7f1e400d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.557+0000 7f1e475fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e400b4750 0x7f1e400b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.557+0000 7f1e475fb640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e400b5100 con 0x7f1e400b8a10 2023-10-16T14:13:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.557+0000 7f1e475fb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4015d330 con 0x7f1e4015cf50 2023-10-16T14:13:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.557+0000 7f1e475fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e400d1680 con 0x7f1e400b4750 2023-10-16T14:13:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e400b8a10 0x7f1e400d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:55.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e400b8a10 0x7f1e400d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51640/0 (socket says 172.21.15.60:51640) 2023-10-16T14:13:55.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 -- 172.21.15.60:0/334956462 learned_addr learned my addr 172.21.15.60:0/334956462 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:55.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e45397640 1 --2- 172.21.15.60:0/334956462 >> v2:172.21.15.144:3300/0 conn(0x7f1e4015cf50 0x7f1e400d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:55.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e45b98640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e400b4750 0x7f1e400b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:55.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 -- 172.21.15.60:0/334956462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e400b4750 msgr2=0x7f1e400b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:55.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e400b4750 0x7f1e400b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 -- 172.21.15.60:0/334956462 >> v2:172.21.15.144:3300/0 conn(0x7f1e4015cf50 msgr2=0x7f1e400d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:55.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 --2- 172.21.15.60:0/334956462 >> v2:172.21.15.144:3300/0 conn(0x7f1e4015cf50 0x7f1e400d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e44b96640 1 -- 172.21.15.60:0/334956462 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e400d1960 con 0x7f1e400b8a10 2023-10-16T14:13:55.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.558+0000 7f1e45b98640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e400b4750 0x7f1e400b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:55.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.559+0000 7f1e44b96640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b8a10 0x7f1e400d0d10 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f1e400cd520 tx=0x7f1e2c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2909be47593002ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:55.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e3ffff640 1 -- 172.21.15.60:0/334956462 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c031880 con 0x7f1e400b8a10 2023-10-16T14:13:55.566 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e3ffff640 1 -- 172.21.15.60:0/334956462 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c0319e0 con 0x7f1e400b8a10 2023-10-16T14:13:55.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e3ffff640 1 -- 172.21.15.60:0/334956462 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c031d70 con 0x7f1e400b8a10 2023-10-16T14:13:55.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 -- 172.21.15.60:0/334956462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b8a10 msgr2=0x7f1e400d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:55.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b8a10 0x7f1e400d0d10 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f1e400cd520 tx=0x7f1e2c02b920 comp rx=0 tx=0).stop 2023-10-16T14:13:55.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 -- 172.21.15.60:0/334956462 shutdown_connections 2023-10-16T14:13:55.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e400b4750 0x7f1e400b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.567 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 --2- 172.21.15.60:0/334956462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b8a10 0x7f1e400d0d10 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 --2- 172.21.15.60:0/334956462 >> v2:172.21.15.144:3300/0 conn(0x7f1e4015cf50 0x7f1e400d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.560+0000 7f1e475fb640 1 -- 172.21.15.60:0/334956462 >> 172.21.15.60:0/334956462 conn(0x7f1e400cf750 msgr2=0x7f1e400cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:55.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.561+0000 7f1e475fb640 1 -- 172.21.15.60:0/334956462 shutdown_connections 2023-10-16T14:13:55.568 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.561+0000 7f1e475fb640 1 -- 172.21.15.60:0/334956462 wait complete. 2023-10-16T14:13:55.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.562+0000 7f1e475fb640 1 Processor -- start 2023-10-16T14:13:55.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.562+0000 7f1e475fb640 1 -- start start 2023-10-16T14:13:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.563+0000 7f1e475fb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 0x7f1e4014a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.563+0000 7f1e475fb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e400b8a10 0x7f1e400b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.563+0000 7f1e475fb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e4015cf50 0x7f1e400b7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.563+0000 7f1e475fb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e400ccbd0 con 0x7f1e400b4750 2023-10-16T14:13:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.563+0000 7f1e475fb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4014a9b0 con 0x7f1e400b8a10 2023-10-16T14:13:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.563+0000 7f1e475fb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e40156880 con 0x7f1e4015cf50 2023-10-16T14:13:55.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 0x7f1e4014a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:55.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 0x7f1e4014a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51656/0 (socket says 172.21.15.60:51656) 2023-10-16T14:13:55.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 -- 172.21.15.60:0/3367145302 learned_addr learned my addr 172.21.15.60:0/3367145302 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:55.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 -- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e4015cf50 msgr2=0x7f1e400b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:55.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e4015cf50 0x7f1e400b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 -- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e400b8a10 msgr2=0x7f1e400b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:55.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e400b8a10 0x7f1e400b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:55.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 -- 172.21.15.60:0/3367145302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c0094d0 con 0x7f1e400b4750 2023-10-16T14:13:55.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.564+0000 7f1e45397640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 0x7f1e4014a470 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f1e34007e40 tx=0x7f1e3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:55.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.565+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e34014620 con 0x7f1e400b4750 2023-10-16T14:13:55.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.565+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3400ee90 con 0x7f1e400b4750 2023-10-16T14:13:55.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.565+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e40156b60 con 0x7f1e400b4750 2023-10-16T14:13:55.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.565+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e340149c0 con 0x7f1e400b4750 2023-10-16T14:13:55.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.565+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e40157120 con 0x7f1e400b4750 2023-10-16T14:13:55.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.566+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e40072350 con 0x7f1e400b4750 2023-10-16T14:13:55.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.567+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1e34014b20 con 0x7f1e400b4750 2023-10-16T14:13:55.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.568+0000 7f1e3dffb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1e2406af90 0x7f1e2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:55.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.568+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f1e3408a320 con 0x7f1e400b4750 2023-10-16T14:13:55.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.569+0000 7f1e44b96640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1e2406af90 0x7f1e2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:55.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.570+0000 7f1e44b96640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1e2406af90 0x7f1e2406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c036000 tx=0x7f1e2c030000 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:55.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.572+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e34055050 con 0x7f1e400b4750 2023-10-16T14:13:55.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:55.771+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1e4004e160 con 0x7f1e400b4750 2023-10-16T14:13:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.320+0000 7f1e3dffb640 1 -- 172.21.15.60:0/3367145302 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v292) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1e34058670 con 0x7f1e400b4750 2023-10-16T14:13:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2023-10-16T14:13:56.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.324+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1e2406af90 msgr2=0x7f1e2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.324+0000 7f1e475fb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1e2406af90 0x7f1e2406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c036000 tx=0x7f1e2c030000 comp rx=0 tx=0).stop 2023-10-16T14:13:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.324+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 msgr2=0x7f1e4014a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.324+0000 7f1e475fb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 0x7f1e4014a470 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f1e34007e40 tx=0x7f1e3400e4b0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.326+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 shutdown_connections 2023-10-16T14:13:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.326+0000 7f1e475fb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1e2406af90 0x7f1e2406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.326+0000 7f1e475fb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1e4015cf50 0x7f1e400b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.326+0000 7f1e475fb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e400b8a10 0x7f1e400b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.326+0000 7f1e475fb640 1 --2- 172.21.15.60:0/3367145302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e400b4750 0x7f1e4014a470 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.326+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 >> 172.21.15.60:0/3367145302 conn(0x7f1e400cf750 msgr2=0x7f1e400d0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.327+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 shutdown_connections 2023-10-16T14:13:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.327+0000 7f1e475fb640 1 -- 172.21.15.60:0/3367145302 wait complete. 2023-10-16T14:13:56.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T14:13:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.569+0000 7f1c28fef640 1 Processor -- start 2023-10-16T14:13:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.569+0000 7f1c28fef640 1 -- start start 2023-10-16T14:13:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c28fef640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c24159ad0 0x7f1c24070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c28fef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2415d650 0x7f1c2406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c28fef640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240cd260 con 0x7f1c2406b3e0 2023-10-16T14:13:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c28fef640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c24071470 con 0x7f1c24159ad0 2023-10-16T14:13:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c28fef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415de30 con 0x7f1c2415d650 2023-10-16T14:13:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c237fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c24159ad0 0x7f1c24070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c237fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c24159ad0 0x7f1c24070f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51666/0 (socket says 172.21.15.60:51666) 2023-10-16T14:13:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c237fe640 1 -- 172.21.15.60:0/3088347098 learned_addr learned my addr 172.21.15.60:0/3088347098 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c237fe640 1 -- 172.21.15.60:0/3088347098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2415d650 msgr2=0x7f1c2406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c237fe640 1 --2- 172.21.15.60:0/3088347098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2415d650 0x7f1c2406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.571+0000 7f1c237fe640 1 -- 172.21.15.60:0/3088347098 >> v1:172.21.15.60:6789/0 conn(0x7f1c2406b3e0 legacy=0x7f1c240c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.572+0000 7f1c237fe640 1 -- 172.21.15.60:0/3088347098 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2415e110 con 0x7f1c24159ad0 2023-10-16T14:13:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.573+0000 7f1c237fe640 1 --2- 172.21.15.60:0/3088347098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24070f30 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009e50 tx=0x7f1c1402e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb5e4dfe0b63ba2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.573+0000 7f1c227fc640 1 -- 172.21.15.60:0/3088347098 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1402ed00 con 0x7f1c24159ad0 2023-10-16T14:13:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.573+0000 7f1c227fc640 1 -- 172.21.15.60:0/3088347098 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1402ee60 con 0x7f1c24159ad0 2023-10-16T14:13:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.573+0000 7f1c227fc640 1 -- 172.21.15.60:0/3088347098 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14036610 con 0x7f1c24159ad0 2023-10-16T14:13:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.573+0000 7f1c28fef640 1 -- 172.21.15.60:0/3088347098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 msgr2=0x7f1c24070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.573+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3088347098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24070f30 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009e50 tx=0x7f1c1402e160 comp rx=0 tx=0).stop 2023-10-16T14:13:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.574+0000 7f1c28fef640 1 -- 172.21.15.60:0/3088347098 shutdown_connections 2023-10-16T14:13:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.574+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3088347098 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2415d650 0x7f1c2406aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.574+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3088347098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24070f30 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.574+0000 7f1c28fef640 1 -- 172.21.15.60:0/3088347098 >> 172.21.15.60:0/3088347098 conn(0x7f1c240b0170 msgr2=0x7f1c240b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.574+0000 7f1c28fef640 1 -- 172.21.15.60:0/3088347098 shutdown_connections 2023-10-16T14:13:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.574+0000 7f1c28fef640 1 -- 172.21.15.60:0/3088347098 wait complete. 2023-10-16T14:13:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.575+0000 7f1c28fef640 1 Processor -- start 2023-10-16T14:13:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.576+0000 7f1c28fef640 1 -- start start 2023-10-16T14:13:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c28fef640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2406b3e0 0x7f1c240c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c28fef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c28fef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c2415d650 0x7f1c2415c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c28fef640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240c58e0 con 0x7f1c24159ad0 2023-10-16T14:13:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c28fef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241540b0 con 0x7f1c2415d650 2023-10-16T14:13:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c28fef640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415cc90 con 0x7f1c2406b3e0 2023-10-16T14:13:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51680/0 (socket says 172.21.15.60:51680) 2023-10-16T14:13:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 -- 172.21.15.60:0/3174695166 learned_addr learned my addr 172.21.15.60:0/3174695166 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c237fe640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2406b3e0 0x7f1c240c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c23fff640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c2415d650 0x7f1c2415c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 -- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2406b3e0 msgr2=0x7f1c240c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2406b3e0 0x7f1c240c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 -- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c2415d650 msgr2=0x7f1c2415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c2415d650 0x7f1c2415c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.577+0000 7f1c22ffd640 1 -- 172.21.15.60:0/3174695166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14009b30 con 0x7f1c24159ad0 2023-10-16T14:13:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.578+0000 7f1c22ffd640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24153b70 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00b4d0 tx=0x7f1c0c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.578+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c010870 con 0x7f1c24159ad0 2023-10-16T14:13:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.578+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0109d0 con 0x7f1c24159ad0 2023-10-16T14:13:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.578+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c010d80 con 0x7f1c24159ad0 2023-10-16T14:13:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.578+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2415cef0 con 0x7f1c24159ad0 2023-10-16T14:13:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.578+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c240c5dd0 con 0x7f1c24159ad0 2023-10-16T14:13:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.579+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2404e9b0 con 0x7f1c24159ad0 2023-10-16T14:13:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.580+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c0c00f3c0 con 0x7f1c24159ad0 2023-10-16T14:13:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.581+0000 7f1c1bfff640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1c0406af90 0x7f1c0406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.581+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f1c0c0897a0 con 0x7f1c24159ad0 2023-10-16T14:13:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.582+0000 7f1c237fe640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1c0406af90 0x7f1c0406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.583+0000 7f1c237fe640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1c0406af90 0x7f1c0406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1c14037000 tx=0x7f1c1400d040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.586+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c0c054690 con 0x7f1c24159ad0 2023-10-16T14:13:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.780+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1c240b4600 con 0x7f1c24159ad0 2023-10-16T14:13:56.791 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.787+0000 7f1c1bfff640 1 -- 172.21.15.60:0/3174695166 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1c0c057cb0 con 0x7f1c24159ad0 2023-10-16T14:13:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1c0406af90 msgr2=0x7f1c0406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1c0406af90 0x7f1c0406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1c14037000 tx=0x7f1c1400d040 comp rx=0 tx=0).stop 2023-10-16T14:13:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 msgr2=0x7f1c24153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24153b70 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00b4d0 tx=0x7f1c0c00b990 comp rx=0 tx=0).stop 2023-10-16T14:13:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 shutdown_connections 2023-10-16T14:13:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1c0406af90 0x7f1c0406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c2415d650 0x7f1c2415c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c24159ad0 0x7f1c24153b70 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 --2- 172.21.15.60:0/3174695166 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1c2406b3e0 0x7f1c240c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.790+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 >> 172.21.15.60:0/3174695166 conn(0x7f1c240b0170 msgr2=0x7f1c2406f170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.791+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 shutdown_connections 2023-10-16T14:13:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:56.791+0000 7f1c28fef640 1 -- 172.21.15.60:0/3174695166 wait complete. 2023-10-16T14:13:56.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T14:13:57.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.015+0000 7f12cbffb640 1 Processor -- start 2023-10-16T14:13:57.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.015+0000 7f12cbffb640 1 -- start start 2023-10-16T14:13:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.016+0000 7f12cbffb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f12c406c850 0x7f12c40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12cbffb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c4070360 con 0x7f12c4159e90 2023-10-16T14:13:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12cbffb640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c40cd1f0 con 0x7f12c4159ab0 2023-10-16T14:13:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12cbffb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c415d890 con 0x7f12c406c850 2023-10-16T14:13:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12c9596640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f12c4159e90 0x7f12c406c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60372/0 (socket says 172.21.15.60:60372) 2023-10-16T14:13:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12c9596640 1 -- 172.21.15.60:0/2436507805 learned_addr learned my addr 172.21.15.60:0/2436507805 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12ca598640 1 --2- 172.21.15.60:0/2436507805 >> v2:172.21.15.144:3300/0 conn(0x7f12c406c850 0x7f12c40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710072655 0 0) 0x7f12c4070360 con 0x7f12c4159e90 2023-10-16T14:13:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.017+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b0005730 con 0x7f12c4159e90 2023-10-16T14:13:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921364834 0 0) 0x7f12b0005730 con 0x7f12c4159e90 2023-10-16T14:13:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 >> v2:172.21.15.144:3300/0 conn(0x7f12c406c850 msgr2=0x7f12c40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 --2- 172.21.15.60:0/2436507805 >> v2:172.21.15.144:3300/0 conn(0x7f12c406c850 0x7f12c40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 >> v1:172.21.15.144:6789/0 conn(0x7f12c4159ab0 legacy=0x7f12c406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40cd4d0 con 0x7f12c4159e90 2023-10-16T14:13:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f12b8001a10 con 0x7f12c4159e90 2023-10-16T14:13:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 >> v1:172.21.15.60:6789/0 conn(0x7f12c4159e90 legacy=0x7f12c406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12b0005b80 0x7f12b0005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b00069d0 0x7f12b0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b0007400 0x7f12b000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c415d890 con 0x7f12b0005b80 2023-10-16T14:13:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b000e360 con 0x7f12b0007400 2023-10-16T14:13:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b000e520 con 0x7f12b00069d0 2023-10-16T14:13:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.018+0000 7f12c9d97640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12b0005b80 0x7f12b0005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12c9596640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b00069d0 0x7f12b0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12c9d97640 1 -- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b00069d0 msgr2=0x7f12b0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12c9d97640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b00069d0 0x7f12b0006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12c9d97640 1 -- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b0007400 msgr2=0x7f12b000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12ca598640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b0007400 0x7f12b000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12c9d97640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b0007400 0x7f12b000db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12ca598640 1 -- 172.21.15.60:0/2436507805 reap_dead start 2023-10-16T14:13:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.019+0000 7f12c9d97640 1 -- 172.21.15.60:0/2436507805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0005730 con 0x7f12b0005b80 2023-10-16T14:13:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.020+0000 7f12c9d97640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12b0005b80 0x7f12b0005f80 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f12c0003db0 tx=0x7f12b0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25a5aa06a816a18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.020+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0032600 con 0x7f12b0005b80 2023-10-16T14:13:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.020+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c002e050 con 0x7f12b0005b80 2023-10-16T14:13:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.020+0000 7f12c8d95640 1 -- 172.21.15.60:0/2436507805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0032a20 con 0x7f12b0005b80 2023-10-16T14:13:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 -- 172.21.15.60:0/2436507805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12b0005b80 msgr2=0x7f12b0005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12b0005b80 0x7f12b0005f80 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f12c0003db0 tx=0x7f12b0007a00 comp rx=0 tx=0).stop 2023-10-16T14:13:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 -- 172.21.15.60:0/2436507805 shutdown_connections 2023-10-16T14:13:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 --2- 172.21.15.60:0/2436507805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12b0005b80 0x7f12b0005f80 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 -- 172.21.15.60:0/2436507805 >> 172.21.15.60:0/2436507805 conn(0x7f12c40ce9b0 msgr2=0x7f12c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 -- 172.21.15.60:0/2436507805 shutdown_connections 2023-10-16T14:13:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.021+0000 7f12cbffb640 1 -- 172.21.15.60:0/2436507805 wait complete. 2023-10-16T14:13:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.023+0000 7f12cbffb640 1 Processor -- start 2023-10-16T14:13:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.023+0000 7f12cbffb640 1 -- start start 2023-10-16T14:13:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12cbffb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b0005b80 0x7f12c414bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12cbffb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 0x7f12c4158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12cbffb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12c41ee930 0x7f12c41587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12cbffb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c71d0 con 0x7f12c41ee4b0 2023-10-16T14:13:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12cbffb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c3960 con 0x7f12c41ee930 2023-10-16T14:13:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12cbffb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c40c3c40 con 0x7f12b0005b80 2023-10-16T14:13:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12c9596640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 0x7f12c4158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12c9596640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 0x7f12c4158260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51686/0 (socket says 172.21.15.60:51686) 2023-10-16T14:13:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.024+0000 7f12c9596640 1 -- 172.21.15.60:0/3781551686 learned_addr learned my addr 172.21.15.60:0/3781551686 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9d97640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b0005b80 0x7f12c414bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9596640 1 -- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b0005b80 msgr2=0x7f12c414bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9596640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b0005b80 0x7f12c414bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9596640 1 -- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12c41ee930 msgr2=0x7f12c41587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12ca598640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12c41ee930 0x7f12c41587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9596640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12c41ee930 0x7f12c41587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9596640 1 -- 172.21.15.60:0/3781551686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40c3f20 con 0x7f12c41ee4b0 2023-10-16T14:13:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12c9596640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 0x7f12c4158260 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f12b80020c0 tx=0x7f12b800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8004d60 con 0x7f12c41ee4b0 2023-10-16T14:13:57.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b800be80 con 0x7f12c41ee4b0 2023-10-16T14:13:57.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b80072f0 con 0x7f12c41ee4b0 2023-10-16T14:13:57.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.025+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c40c4200 con 0x7f12c41ee4b0 2023-10-16T14:13:57.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.026+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c414c130 con 0x7f12c41ee4b0 2023-10-16T14:13:57.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.027+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c40c4720 con 0x7f12c41ee4b0 2023-10-16T14:13:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.028+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f12b8007450 con 0x7f12c41ee4b0 2023-10-16T14:13:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.029+0000 7f12b67fc640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f12a806af90 0x7f12a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.029+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f12b80898f0 con 0x7f12c41ee4b0 2023-10-16T14:13:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.029+0000 7f12c9d97640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f12a806af90 0x7f12a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.031+0000 7f12c9d97640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f12a806af90 0x7f12a806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f12c0034000 tx=0x7f12c0009030 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.033+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12c40c4720 con 0x7f12c41ee4b0 2023-10-16T14:13:57.232 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.228+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f12c40c4720 con 0x7f12c41ee4b0 2023-10-16T14:13:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.234+0000 7f12b67fc640 1 -- 172.21.15.60:0/3781551686 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v98) v1 ==== 104+0+0 (secure 0 0 0) 0x7f12c40c4720 con 0x7f12c41ee4b0 2023-10-16T14:13:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.236+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f12a806af90 msgr2=0x7f12a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.236+0000 7f12cbffb640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f12a806af90 0x7f12a806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f12c0034000 tx=0x7f12c0009030 comp rx=0 tx=0).stop 2023-10-16T14:13:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 msgr2=0x7f12c4158260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 0x7f12c4158260 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f12b80020c0 tx=0x7f12b800b4a0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 shutdown_connections 2023-10-16T14:13:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f12a806af90 0x7f12a806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12c41ee930 0x7f12c41587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c41ee4b0 0x7f12c4158260 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 --2- 172.21.15.60:0/3781551686 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f12b0005b80 0x7f12c414bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 >> 172.21.15.60:0/3781551686 conn(0x7f12c40ce9b0 msgr2=0x7f12c40af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 shutdown_connections 2023-10-16T14:13:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.237+0000 7f12cbffb640 1 -- 172.21.15.60:0/3781551686 wait complete. 2023-10-16T14:13:57.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T14:13:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.479+0000 7f5073fff640 1 Processor -- start 2023-10-16T14:13:57.483 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.479+0000 7f5073fff640 1 -- start start 2023-10-16T14:13:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.480+0000 7f5073fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5074071b90 0x7f50740cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f5073fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740cafd0 con 0x7f50740cd800 2023-10-16T14:13:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f5073fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740cb2b0 con 0x7f5074161b60 2023-10-16T14:13:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f5073fff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50740cbc50 con 0x7f5074071b90 2023-10-16T14:13:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f50737fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f50740cd800 0x7f50740d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60376/0 (socket says 172.21.15.60:60376) 2023-10-16T14:13:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f50737fe640 1 -- 172.21.15.60:0/3063604219 learned_addr learned my addr 172.21.15.60:0/3063604219 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:13:57.486 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f50727fc640 1 --2- 172.21.15.60:0/3063604219 >> v2:172.21.15.60:3300/0 conn(0x7f5074071b90 0x7f50740cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f5071ffb640 1 -- 172.21.15.60:0/3063604219 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557118867 0 0) 0x7f50740cafd0 con 0x7f50740cd800 2023-10-16T14:13:57.487 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f5071ffb640 1 -- 172.21.15.60:0/3063604219 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5060003520 con 0x7f50740cd800 2023-10-16T14:13:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.481+0000 7f50727fc640 1 -- 172.21.15.60:0/3063604219 >> v1:172.21.15.144:6789/0 conn(0x7f5074161b60 legacy=0x7f5074161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.482+0000 7f50727fc640 1 -- 172.21.15.60:0/3063604219 >> v1:172.21.15.60:6789/0 conn(0x7f50740cd800 legacy=0x7f50740d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.482+0000 7f50727fc640 1 -- 172.21.15.60:0/3063604219 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740cb590 con 0x7f5074071b90 2023-10-16T14:13:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.483+0000 7f50727fc640 1 --2- 172.21.15.60:0/3063604219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5074071b90 0x7f50740cc070 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f505c0097a0 tx=0x7f505c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa5aa706c1e4bea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.483+0000 7f5071ffb640 1 -- 172.21.15.60:0/3063604219 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c031830 con 0x7f5074071b90 2023-10-16T14:13:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.483+0000 7f5071ffb640 1 -- 172.21.15.60:0/3063604219 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c031990 con 0x7f5074071b90 2023-10-16T14:13:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.483+0000 7f5071ffb640 1 -- 172.21.15.60:0/3063604219 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c031d20 con 0x7f5074071b90 2023-10-16T14:13:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.483+0000 7f5073fff640 1 -- 172.21.15.60:0/3063604219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5074071b90 msgr2=0x7f50740cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.483+0000 7f5073fff640 1 --2- 172.21.15.60:0/3063604219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5074071b90 0x7f50740cc070 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f505c0097a0 tx=0x7f505c02b8d0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.484+0000 7f5073fff640 1 -- 172.21.15.60:0/3063604219 shutdown_connections 2023-10-16T14:13:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.484+0000 7f5073fff640 1 --2- 172.21.15.60:0/3063604219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5074071b90 0x7f50740cc070 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.484+0000 7f5073fff640 1 -- 172.21.15.60:0/3063604219 >> 172.21.15.60:0/3063604219 conn(0x7f5074065ed0 msgr2=0x7f50740662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.484+0000 7f5073fff640 1 -- 172.21.15.60:0/3063604219 shutdown_connections 2023-10-16T14:13:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.484+0000 7f5073fff640 1 -- 172.21.15.60:0/3063604219 wait complete. 2023-10-16T14:13:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.485+0000 7f5073fff640 1 Processor -- start 2023-10-16T14:13:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.486+0000 7f5073fff640 1 -- start start 2023-10-16T14:13:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.486+0000 7f5073fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5074071b90 0x7f50740c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.486+0000 7f5073fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 0x7f50740c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.486+0000 7f5073fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5074161b60 0x7f50740c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f5073fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50740cab40 con 0x7f50740cd800 2023-10-16T14:13:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f5073fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50740c6270 con 0x7f5074071b90 2023-10-16T14:13:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f5073fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50740c1d20 con 0x7f5074161b60 2023-10-16T14:13:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 0x7f50740c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 0x7f50740c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51704/0 (socket says 172.21.15.60:51704) 2023-10-16T14:13:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 -- 172.21.15.60:0/1998696456 learned_addr learned my addr 172.21.15.60:0/1998696456 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f5072ffd640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5074071b90 0x7f50740c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50737fe640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5074161b60 0x7f50740c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 -- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5074161b60 msgr2=0x7f50740c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5074161b60 0x7f50740c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 -- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5074071b90 msgr2=0x7f50740c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5074071b90 0x7f50740c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.487+0000 7f50727fc640 1 -- 172.21.15.60:0/1998696456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c009400 con 0x7f50740cd800 2023-10-16T14:13:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.488+0000 7f50727fc640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 0x7f50740c12a0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098c0 tx=0x7f505c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.488+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c03a6f0 con 0x7f50740cd800 2023-10-16T14:13:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.488+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c03a870 con 0x7f50740cd800 2023-10-16T14:13:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.488+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50740c5cb0 con 0x7f50740cd800 2023-10-16T14:13:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.488+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c03ac20 con 0x7f50740cd800 2023-10-16T14:13:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.488+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50740c8240 con 0x7f50740cd800 2023-10-16T14:13:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.489+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507404e9b0 con 0x7f50740cd800 2023-10-16T14:13:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.490+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f505c04c3b0 con 0x7f50740cd800 2023-10-16T14:13:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.491+0000 7f506b7fe640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f505406b010 0x7f505406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.491+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f505c0ab640 con 0x7f50740cd800 2023-10-16T14:13:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.492+0000 7f5072ffd640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f505406b010 0x7f505406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.493+0000 7f5072ffd640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f505406b010 0x7f505406d4d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5064009f20 tx=0x7f50640094d0 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.495+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f505c076650 con 0x7f50740cd800 2023-10-16T14:13:57.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.687+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f50740d01e0 con 0x7f50740cd800 2023-10-16T14:13:57.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.694+0000 7f506b7fe640 1 -- 172.21.15.60:0/1998696456 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v99) v1 ==== 120+0+0 (secure 0 0 0) 0x7f505c079c70 con 0x7f50740cd800 2023-10-16T14:13:57.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f505406b010 msgr2=0x7f505406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f505406b010 0x7f505406d4d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5064009f20 tx=0x7f50640094d0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 msgr2=0x7f50740c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 0x7f50740c12a0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098c0 tx=0x7f505c0046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 shutdown_connections 2023-10-16T14:13:57.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f505406b010 0x7f505406d4d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5074161b60 0x7f50740c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50740cd800 0x7f50740c12a0 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 --2- 172.21.15.60:0/1998696456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5074071b90 0x7f50740c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 >> 172.21.15.60:0/1998696456 conn(0x7f5074065ed0 msgr2=0x7f507406c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:57.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 shutdown_connections 2023-10-16T14:13:57.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.697+0000 7f5073fff640 1 -- 172.21.15.60:0/1998696456 wait complete. 2023-10-16T14:13:57.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments' 2023-10-16T14:13:57.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.930+0000 7fa24f1c9640 1 Processor -- start 2023-10-16T14:13:57.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.930+0000 7fa24f1c9640 1 -- start start 2023-10-16T14:13:57.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24f1c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2481615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24f1c9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa24815d3f0 0x7fa24815d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24f1c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24815e3e0 con 0x7fa2480cd0c0 2023-10-16T14:13:57.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24f1c9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480cd4a0 con 0x7fa24815d3f0 2023-10-16T14:13:57.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24f1c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24815e100 con 0x7fa2480b0b20 2023-10-16T14:13:57.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24d766640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa24815d3f0 0x7fa24815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24d766640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa24815d3f0 0x7fa24815d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51710/0 (socket says 172.21.15.60:51710) 2023-10-16T14:13:57.937 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.932+0000 7fa24d766640 1 -- 172.21.15.60:0/90112425 learned_addr learned my addr 172.21.15.60:0/90112425 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:57.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.933+0000 7fa23ffff640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2481615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.933+0000 7fa24d766640 1 -- 172.21.15.60:0/90112425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 msgr2=0x7fa2481615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.933+0000 7fa24d766640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2481615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.938 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.933+0000 7fa24d766640 1 -- 172.21.15.60:0/90112425 >> v1:172.21.15.144:6789/0 conn(0x7fa2480cd0c0 legacy=0x7fa2480b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:13:57.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.933+0000 7fa24d766640 1 -- 172.21.15.60:0/90112425 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248160b10 con 0x7fa24815d3f0 2023-10-16T14:13:57.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.933+0000 7fa23ffff640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2481615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:13:57.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.934+0000 7fa24d766640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa24815d860 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fa240009870 tx=0x7fa24002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9776380146ffe05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.934+0000 7fa23f7fe640 1 -- 172.21.15.60:0/90112425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2400317c0 con 0x7fa24815d3f0 2023-10-16T14:13:57.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.934+0000 7fa23f7fe640 1 -- 172.21.15.60:0/90112425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa240031920 con 0x7fa24815d3f0 2023-10-16T14:13:57.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.934+0000 7fa23f7fe640 1 -- 172.21.15.60:0/90112425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240031cb0 con 0x7fa24815d3f0 2023-10-16T14:13:57.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 -- 172.21.15.60:0/90112425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 msgr2=0x7fa24815d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa24815d860 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fa240009870 tx=0x7fa24002b860 comp rx=0 tx=0).stop 2023-10-16T14:13:57.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 -- 172.21.15.60:0/90112425 shutdown_connections 2023-10-16T14:13:57.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa24815d860 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/90112425 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2481615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 -- 172.21.15.60:0/90112425 >> 172.21.15.60:0/90112425 conn(0x7fa2480cb410 msgr2=0x7fa2480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:57.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 -- 172.21.15.60:0/90112425 shutdown_connections 2023-10-16T14:13:57.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.935+0000 7fa24f1c9640 1 -- 172.21.15.60:0/90112425 wait complete. 2023-10-16T14:13:57.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.937+0000 7fa24f1c9640 1 Processor -- start 2023-10-16T14:13:57.942 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.937+0000 7fa24f1c9640 1 -- start start 2023-10-16T14:13:57.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24f1c9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2480aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24f1c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480cd0c0 0x7fa24815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24f1c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa2480ce970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24f1c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24815f920 con 0x7fa24815d3f0 2023-10-16T14:13:57.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24f1c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480b00d0 con 0x7fa2480cd0c0 2023-10-16T14:13:57.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24f1c9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480af4c0 con 0x7fa2480b0b20 2023-10-16T14:13:57.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24d766640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa2480ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24d766640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa2480ce970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51714/0 (socket says 172.21.15.60:51714) 2023-10-16T14:13:57.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24d766640 1 -- 172.21.15.60:0/850594490 learned_addr learned my addr 172.21.15.60:0/850594490 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:13:57.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa24cf65640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2480aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.938+0000 7fa23ffff640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480cd0c0 0x7fa24815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24d766640 1 -- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 msgr2=0x7fa2480aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24d766640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2480aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24d766640 1 -- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480cd0c0 msgr2=0x7fa24815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:57.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24d766640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480cd0c0 0x7fa24815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:57.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24d766640 1 -- 172.21.15.60:0/850594490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2400094d0 con 0x7fa24815d3f0 2023-10-16T14:13:57.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24cf65640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2480aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:13:57.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24d766640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa2480ce970 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fa24002b8a0 tx=0x7fa2400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240032920 con 0x7fa24815d3f0 2023-10-16T14:13:57.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa240032ac0 con 0x7fa24815d3f0 2023-10-16T14:13:57.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240032e70 con 0x7fa24815d3f0 2023-10-16T14:13:57.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2480af7a0 con 0x7fa24815d3f0 2023-10-16T14:13:57.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.939+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2480afd60 con 0x7fa24815d3f0 2023-10-16T14:13:57.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.941+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2480c3000 con 0x7fa24815d3f0 2023-10-16T14:13:57.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.942+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa2400453f0 con 0x7fa24815d3f0 2023-10-16T14:13:57.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.942+0000 7fa23d7fa640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa23406af90 0x7fa23406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:13:57.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.943+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fa2400abbb0 con 0x7fa24815d3f0 2023-10-16T14:13:57.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.943+0000 7fa24cf65640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa23406af90 0x7fa23406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:13:57.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.944+0000 7fa24cf65640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa23406af90 0x7fa23406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2440045b0 tx=0x7fa244009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:13:57.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:57.947+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa240076c40 con 0x7fa24815d3f0 2023-10-16T14:13:58.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.212+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]} v 0) v1 -- 0x7fa24804e160 con 0x7fa24815d3f0 2023-10-16T14:13:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.342+0000 7fa23d7fa640 1 -- 172.21.15.60:0/850594490 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]}]=0 v3363) v1 ==== 198+0+0 (secure 0 0 0) 0x7fa24007a260 con 0x7fa24815d3f0 2023-10-16T14:13:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa23406af90 msgr2=0x7fa23406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa23406af90 0x7fa23406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2440045b0 tx=0x7fa244009040 comp rx=0 tx=0).stop 2023-10-16T14:13:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 msgr2=0x7fa2480ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:13:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa2480ce970 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fa24002b8a0 tx=0x7fa2400046c0 comp rx=0 tx=0).stop 2023-10-16T14:13:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 shutdown_connections 2023-10-16T14:13:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa23406af90 0x7fa23406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa24815d3f0 0x7fa2480ce970 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2480cd0c0 0x7fa24815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 --2- 172.21.15.60:0/850594490 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa2480b0b20 0x7fa2480aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:13:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 >> 172.21.15.60:0/850594490 conn(0x7fa2480cb410 msgr2=0x7fa24806d030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:13:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.346+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 shutdown_connections 2023-10-16T14:13:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:13:58.347+0000 7fa24f1c9640 1 -- 172.21.15.60:0/850594490 wait complete. 2023-10-16T14:13:58.371 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. ... FAIL 2023-10-16T14:13:58.372 INFO:tasks.cephfs_test_runner: 2023-10-16T14:13:58.372 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-16T14:13:58.372 INFO:tasks.cephfs_test_runner:FAIL: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T14:13:58.372 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. 2023-10-16T14:13:58.373 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T14:13:58.373 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-16T14:13:58.373 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs/test_snap_schedules.py", line 606, in test_snap_schedule_subvol_and_group_arguments 2023-10-16T14:13:58.373 INFO:tasks.cephfs_test_runner: self.fs_snap_schedule_cmd('add', '--group', 'mygrp', path='.', snap_schedule='1m') 2023-10-16T14:13:58.374 INFO:tasks.cephfs_test_runner:AssertionError: CommandFailedError not raised 2023-10-16T14:13:58.374 INFO:tasks.cephfs_test_runner: 2023-10-16T14:13:58.374 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T14:13:58.374 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5993.812s 2023-10-16T14:13:58.375 INFO:tasks.cephfs_test_runner: 2023-10-16T14:13:58.375 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2023-10-16T14:13:58.375 INFO:tasks.cephfs_test_runner: 2023-10-16T14:13:58.375 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-16T14:13:58.375 INFO:tasks.cephfs_test_runner:FAIL: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T14:13:58.376 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. 2023-10-16T14:13:58.376 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T14:13:58.376 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-16T14:13:58.376 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs/test_snap_schedules.py", line 606, in test_snap_schedule_subvol_and_group_arguments 2023-10-16T14:13:58.377 INFO:tasks.cephfs_test_runner: self.fs_snap_schedule_cmd('add', '--group', 'mygrp', path='.', snap_schedule='1m') 2023-10-16T14:13:58.377 INFO:tasks.cephfs_test_runner:AssertionError: CommandFailedError not raised 2023-10-16T14:13:58.377 INFO:tasks.cephfs_test_runner: 2023-10-16T14:13:58.377 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T14:13:58.440 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1d383f6274cf4dca8b51a83c05f39862 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T14:13:58.442 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-16T14:13:58.452 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-16T14:13:58.489 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-16T14:13:58.489 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:58.489 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:58.565 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:13:58.566 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-10-16T14:13:58.566 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:58.625 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-16T14:13:58.626 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:58.626 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:58.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:58.687 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-10-16T14:13:58.687 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T14:13:58.740 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054289,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249400k,nr_inodes=812350,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498808k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-16T14:13:58.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:58.741 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:13:58.804 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T14:13:58.805 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T14:13:58.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:58.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T14:13:58.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:58.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T14:13:59.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2023-10-16T14:13:59.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.150 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2023-10-16T14:13:59.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:13:59.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2023-10-16T14:13:59.246 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:13:59.332 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2023-10-16T14:13:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:13:59.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.336 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-16T14:13:59.388 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T14:13:59.389 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-16T14:13:59.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T14:13:59.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T14:13:59.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.620 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-16T14:13:59.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.755 DEBUG:teuthology.orchestra.run.smithi144:> ip netns list 2023-10-16T14:13:59.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T14:13:59.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo ip link delete ceph-brx 2023-10-16T14:13:59.946 INFO:teuthology.orchestra.run.smithi144.stderr:Cannot find device "ceph-brx" 2023-10-16T14:13:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:13:59.950 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-16T14:13:59.960 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T14:13:59.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:14:00.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.198+0000 7f950a0ad640 1 Processor -- start 2023-10-16T14:14:00.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.198+0000 7f950a0ad640 1 -- start start 2023-10-16T14:14:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f950a0ad640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f950406fb50 0x7f95040cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f950a0ad640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f950406c9e0 0x7f95040ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f950a0ad640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95040ca560 con 0x7f95040c9060 2023-10-16T14:14:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f950a0ad640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95040c9950 con 0x7f950406c9e0 2023-10-16T14:14:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f950a0ad640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95040c9c30 con 0x7f950406fb50 2023-10-16T14:14:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f95098ac640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f950406c9e0 0x7f95040ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f95098ac640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f950406c9e0 0x7f95040ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47910/0 (socket says 172.21.15.60:47910) 2023-10-16T14:14:00.206 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.200+0000 7f95098ac640 1 -- 172.21.15.60:0/3898985226 learned_addr learned my addr 172.21.15.60:0/3898985226 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:00.207 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.201+0000 7f95088aa640 1 --2- 172.21.15.60:0/3898985226 >> v2:172.21.15.144:3300/0 conn(0x7f950406fb50 0x7f95040cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.201+0000 7f95098ac640 1 -- 172.21.15.60:0/3898985226 >> v2:172.21.15.144:3300/0 conn(0x7f950406fb50 msgr2=0x7f95040cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.201+0000 7f95098ac640 1 --2- 172.21.15.60:0/3898985226 >> v2:172.21.15.144:3300/0 conn(0x7f950406fb50 0x7f95040cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.201+0000 7f95098ac640 1 -- 172.21.15.60:0/3898985226 >> v1:172.21.15.144:6789/0 conn(0x7f95040c9060 legacy=0x7f950406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:00.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.201+0000 7f95098ac640 1 -- 172.21.15.60:0/3898985226 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95040c9f10 con 0x7f950406c9e0 2023-10-16T14:14:00.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.202+0000 7f95098ac640 1 --2- 172.21.15.60:0/3898985226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f95040ccde0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f9504164cb0 tx=0x7f94fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94d5cba06d3bd36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:00.209 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.202+0000 7f94fbfff640 1 -- 172.21.15.60:0/3898985226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc02ca80 con 0x7f950406c9e0 2023-10-16T14:14:00.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.202+0000 7f94fbfff640 1 -- 172.21.15.60:0/3898985226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94fc02cbe0 con 0x7f950406c9e0 2023-10-16T14:14:00.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.202+0000 7f94fbfff640 1 -- 172.21.15.60:0/3898985226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0343f0 con 0x7f950406c9e0 2023-10-16T14:14:00.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 -- 172.21.15.60:0/3898985226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 msgr2=0x7f95040ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.210 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 --2- 172.21.15.60:0/3898985226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f95040ccde0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f9504164cb0 tx=0x7f94fc02bee0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 -- 172.21.15.60:0/3898985226 shutdown_connections 2023-10-16T14:14:00.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 --2- 172.21.15.60:0/3898985226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f95040ccde0 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.211 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 --2- 172.21.15.60:0/3898985226 >> v2:172.21.15.144:3300/0 conn(0x7f950406fb50 0x7f95040cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 -- 172.21.15.60:0/3898985226 >> 172.21.15.60:0/3898985226 conn(0x7f95040ce6a0 msgr2=0x7f95040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:00.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 -- 172.21.15.60:0/3898985226 shutdown_connections 2023-10-16T14:14:00.212 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.203+0000 7f950a0ad640 1 -- 172.21.15.60:0/3898985226 wait complete. 2023-10-16T14:14:00.214 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.205+0000 7f950a0ad640 1 Processor -- start 2023-10-16T14:14:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.205+0000 7f950a0ad640 1 -- start start 2023-10-16T14:14:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.206+0000 7f950a0ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f9504161e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.206+0000 7f950a0ad640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f950406fb50 0x7f95040c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.206+0000 7f950a0ad640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95040c9060 0x7f95041f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.206+0000 7f950a0ad640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95040c6750 con 0x7f950406c9e0 2023-10-16T14:14:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.206+0000 7f950a0ad640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504162490 con 0x7f95040c9060 2023-10-16T14:14:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.206+0000 7f950a0ad640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9504162770 con 0x7f950406fb50 2023-10-16T14:14:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f9504161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f9504161e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47916/0 (socket says 172.21.15.60:47916) 2023-10-16T14:14:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 -- 172.21.15.60:0/2661182979 learned_addr learned my addr 172.21.15.60:0/2661182979 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95088aa640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f950406fb50 0x7f95040c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95098ac640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95040c9060 0x7f95041f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 -- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f950406fb50 msgr2=0x7f95040c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f950406fb50 0x7f95040c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 -- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95040c9060 msgr2=0x7f95041f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95040c9060 0x7f95041f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95090ab640 1 -- 172.21.15.60:0/2661182979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc009b30 con 0x7f950406c9e0 2023-10-16T14:14:00.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.207+0000 7f95088aa640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f950406fb50 0x7f95040c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:00.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.208+0000 7f95090ab640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f9504161e00 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f950000b3e0 tx=0x7f950000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:00.220 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.208+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500011830 con 0x7f950406c9e0 2023-10-16T14:14:00.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.208+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95000119b0 con 0x7f950406c9e0 2023-10-16T14:14:00.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.208+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500011d60 con 0x7f950406c9e0 2023-10-16T14:14:00.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.208+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9504162a50 con 0x7f950406c9e0 2023-10-16T14:14:00.221 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.208+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95040c7030 con 0x7f950406c9e0 2023-10-16T14:14:00.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.209+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950404e9b0 con 0x7f950406c9e0 2023-10-16T14:14:00.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.210+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f950001a400 con 0x7f950406c9e0 2023-10-16T14:14:00.222 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.211+0000 7f94f9ffb640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f94e806af90 0x7f94e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.211+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f95000898a0 con 0x7f950406c9e0 2023-10-16T14:14:00.223 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.212+0000 7f95088aa640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f94e806af90 0x7f94e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.213+0000 7f95088aa640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f94e806af90 0x7f94e806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95041fafb0 tx=0x7f94f4008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:00.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.216+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95000548a0 con 0x7f950406c9e0 2023-10-16T14:14:00.416 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.413+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95040ca1f0 con 0x7f94e806af90 2023-10-16T14:14:00.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.420+0000 7f94f9ffb640 1 -- 172.21.15.60:0/2661182979 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48182 (secure 0 0 0) 0x7f94f400c040 con 0x7f94e806af90 2023-10-16T14:14:00.424 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:00.425 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:14:00.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f94e806af90 msgr2=0x7f94e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f94e806af90 0x7f94e806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95041fafb0 tx=0x7f94f4008040 comp rx=0 tx=0).stop 2023-10-16T14:14:00.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 msgr2=0x7f9504161e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f9504161e00 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f950000b3e0 tx=0x7f950000b8a0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 shutdown_connections 2023-10-16T14:14:00.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f94e806af90 0x7f94e806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f950406fb50 0x7f95040c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95040c9060 0x7f95041f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 --2- 172.21.15.60:0/2661182979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f950406c9e0 0x7f9504161e00 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 >> 172.21.15.60:0/2661182979 conn(0x7f95040ce6a0 msgr2=0x7f950406b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:00.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 shutdown_connections 2023-10-16T14:14:00.430 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.423+0000 7f950a0ad640 1 -- 172.21.15.60:0/2661182979 wait complete. 2023-10-16T14:14:00.444 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-16T14:13:59.095045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":473744,"kb_used_data":31408,"kb_used_omap":12,"kb_used_meta":442227,"kb_avail":754500976,"statfs":{"total":773094113280,"available":772608999424,"internally_reserved":0,"allocated":32161792,"data_stored":17224140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":452841016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-810930,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-312,"num_read_kb":-156,"num_write":-368,"num_write_kb":-3126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002025"},"pg_stats":[{"pgid":"1.0","version":"289'2843","reported_seq":4002,"reported_epoch":289,"state":"active+clean","last_fresh":"2023-10-16T14:13:51.197818+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:13:51.197818+0000","last_peered":"2023-10-16T14:13:51.197818+0000","last_clean":"2023-10-16T14:13:51.197818+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:13:51.197818+0000","last_undegraded":"2023-10-16T14:13:51.197818+0000","last_fullsized":"2023-10-16T14:13:51.197818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543335,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49608,"kb_used_data":4540,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322232,"statfs":{"total":96636764160,"available":96585965568,"internally_reserved":0,"allocated":4648960,"data_stored":2785687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2889999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":1.381},{"interface":"front","average":{"1min":1.331,"5min":1.4139999999999999,"15min":1.4079999999999999},"min":{"1min":0.76400000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.133,"5min":6.391,"15min":6.391},"last":1.431}]},{"osd":1,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.212,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.482},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.238,"15min":1.252},"min":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.982,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.228,"15min":1.258},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.736,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.242},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.745,"15min":0.745},"max":{"1min":2.0529999999999999,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.181}]},{"osd":3,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3660000000000001,"15min":1.403},"min":{"1min":0.86799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.853,"5min":2.0790000000000002,"15min":2.097},"last":1.3},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.823,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.276}]},{"osd":4,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.256,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.593,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.133},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4630000000000001,"15min":1.4239999999999999},"min":{"1min":1.0089999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8620000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.448,"15min":1.397},"min":{"1min":0.879,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.0880000000000001,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.349},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.5589999999999999,"15min":1.5429999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3159999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.6339999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.6579999999999999,"15min":1.641},"min":{"1min":1.0680000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.4500000000000002,"5min":3.29,"15min":3.29},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5529999999999999,"15min":1.496},"min":{"1min":0.85999999999999999,"5min":0.75,"15min":0.75},"max":{"1min":2.0419999999999998,"5min":3.524,"15min":3.524},"last":1.548}]}]},{"osd":6,"up_from":14,"seq":60129543336,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48744,"kb_used_data":3676,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323096,"statfs":{"total":96636764160,"available":96586850304,"internally_reserved":0,"allocated":3764224,"data_stored":1893201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2330000000000001,"15min":1.2989999999999999},"min":{"1min":0.92300000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6200000000000001,"5min":1.788,"15min":1.887},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.266,"5min":1.2909999999999999,"15min":1.3420000000000001},"min":{"1min":0.85499999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7509999999999999,"5min":1.847,"15min":1.9390000000000001},"last":1.252}]},{"osd":1,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.157,"15min":1.204},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6819999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.032},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.1970000000000001},"min":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.64200000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.77},"last":1.202}]},{"osd":2,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.86899999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.24,"5min":1.294,"15min":1.3029999999999999},"min":{"1min":0.76400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.5549999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3240000000000001,"15min":1.339},"min":{"1min":0.95199999999999996,"5min":0.79300000000000004,"15min":0.70499999999999996},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.409},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.1519999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4219999999999999,"15min":1.4119999999999999},"min":{"1min":0.90700000000000003,"5min":0.77300000000000002,"15min":0.51400000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5580000000000001,"15min":1.548},"min":{"1min":0.88900000000000001,"5min":0.86299999999999999,"15min":0.78400000000000003},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.1179999999999999},"last":1.355}]},{"osd":5,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.427,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.964,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.675},{"interface":"front","average":{"1min":1.617,"5min":1.526,"15min":1.496},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.64400000000000002},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.514}]},{"osd":7,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.5309999999999999,"15min":1.512},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.68600000000000005},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.468},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5589999999999999,"15min":1.54},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.6120000000000001},"last":1.774}]}]},{"osd":5,"up_from":14,"seq":60129543337,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52908,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318932,"statfs":{"total":96636764160,"available":96582586368,"internally_reserved":0,"allocated":3768320,"data_stored":1893475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0900000000000001,"15min":1.181},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":2.8330000000000002},"last":0.874},{"interface":"front","average":{"1min":1.216,"5min":1.2070000000000001,"15min":1.2170000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9830000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.95199999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.804,"5min":1.871,"15min":1.8779999999999999},"last":1.593},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.153,"15min":1.23},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.3799999999999999,"5min":1.931,"15min":2.5840000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.2569999999999999,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.65400000000000003,"15min":0.63},"max":{"1min":1.591,"5min":2.024,"15min":2.8119999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.1950000000000001,"15min":1.2849999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.492,"5min":1.8600000000000001,"15min":2.6259999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.387,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.821},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.361,"15min":1.419},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.794,"5min":1.8759999999999999,"15min":2.75},"last":1.1950000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2869999999999999,"15min":1.302},"min":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.982,"5min":1.982,"15min":2.774},"last":2.149},{"interface":"front","average":{"1min":1.504,"5min":1.478,"15min":1.4470000000000001},"min":{"1min":1.2490000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.3330000000000002},"last":2.0609999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4550000000000001,"15min":1.4399999999999999},"min":{"1min":1.028,"5min":0.59399999999999997,"15min":0.55900000000000005},"max":{"1min":2.081,"5min":2.081,"15min":2.6720000000000002},"last":1.732},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.89100000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7010000000000001},"last":2.242}]},{"osd":7,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.593,"15min":1.5580000000000001},"min":{"1min":1.278,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.4609999999999999},"last":1.97},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":1.1579999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.2010000000000001,"5min":2.238,"15min":2.8719999999999999},"last":1.9279999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543337,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89168,"kb_used_data":3908,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282672,"statfs":{"total":96636764160,"available":96545456128,"internally_reserved":0,"allocated":4001792,"data_stored":2131171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.1930000000000001},"min":{"1min":0.85899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.575,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.105},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.159,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5629999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.179}]},{"osd":1,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1439999999999999,"15min":1.2270000000000001},"min":{"1min":0.73099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.27},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7210000000000001,"5min":1.792,"15min":2.0499999999999998},"last":1.2210000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2529999999999999,"15min":1.302},"min":{"1min":0.81599999999999995,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.4299999999999999,"5min":1.714,"15min":1.865},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.365,"15min":1.3779999999999999},"min":{"1min":1.105,"5min":0.87,"15min":0.87},"max":{"1min":1.77,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.4199999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.292,"15min":1.2989999999999999},"min":{"1min":0.996,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9650000000000001},"last":1.075},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.425},"min":{"1min":1.1459999999999999,"5min":0.92800000000000005,"15min":0.86799999999999999},"max":{"1min":1.887,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.4670000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4019999999999999,"15min":1.3540000000000001},"min":{"1min":0.997,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.738,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.415,"15min":1.3600000000000001},"min":{"1min":0.97499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0659999999999998,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.5129999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4390000000000001,"15min":1.3999999999999999},"min":{"1min":1.091,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.972},{"interface":"front","average":{"1min":1.649,"5min":1.5760000000000001,"15min":1.5289999999999999},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":1.9790000000000001,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.581}]},{"osd":7,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.591,"15min":1.5269999999999999},"min":{"1min":1.22,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.1219999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.399},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.657,"15min":1.645},"min":{"1min":1.335,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.3660000000000001},"last":1.6639999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543337,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56964,"kb_used_data":3576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314876,"statfs":{"total":96636764160,"available":96578433024,"internally_reserved":0,"allocated":3661824,"data_stored":1802566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91000000000000003,"15min":0.93799999999999994},"min":{"1min":0.53900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.54,"5min":1.595,"15min":1.595},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.105,"15min":1.1899999999999999},"min":{"1min":0.57399999999999995,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.5529999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.98599999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0229999999999999,"15min":1.0700000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.6319999999999999,"5min":1.6870000000000001,"15min":1.855},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0620000000000001,"15min":1.105},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.9890000000000001,"5min":2.125,"15min":2.125},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0569999999999999,"15min":1.1739999999999999},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.544,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":1.073},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.4039999999999999},"min":{"1min":0.93200000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6739999999999999,"5min":1.909,"15min":1.9319999999999999},"last":1.1319999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4390000000000001,"15min":1.4870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.67000000000000004},"max":{"1min":1.891,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.27},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3859999999999999,"15min":1.4470000000000001},"min":{"1min":0.80500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6599999999999999,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.51}]},{"osd":5,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4019999999999999,"15min":1.4730000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0579999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4950000000000001,"15min":1.5409999999999999},"min":{"1min":0.88700000000000001,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9339999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.288}]},{"osd":6,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.508},"min":{"1min":0.96599999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.8300000000000001,"5min":2.335,"15min":2.4329999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.4279999999999999,"15min":1.486},"min":{"1min":0.752,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9279999999999999,"5min":2.194,"15min":2.4180000000000001},"last":1.3420000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5289999999999999,"15min":1.5840000000000001},"min":{"1min":0.875,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1320000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.611,"15min":1.665},"min":{"1min":0.91300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.089,"5min":2.569,"15min":2.6070000000000002},"last":1.4750000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543335,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56968,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314872,"statfs":{"total":96636764160,"available":96578428928,"internally_reserved":0,"allocated":3665920,"data_stored":1802108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.93200000000000005,"15min":0.94999999999999996},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.47599999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.7090000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.073}]},{"osd":1,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5580000000000001,"5min":1.847,"15min":1.847},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.125,"15min":1.2210000000000001},"min":{"1min":0.67100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.415,"5min":1.784,"15min":1.9530000000000001},"last":0.93999999999999995}]},{"osd":3,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.05,"15min":1.0980000000000001},"min":{"1min":0.56100000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3759999999999999,"5min":1.9179999999999999,"15min":2.1299999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.252,"15min":1.3460000000000001},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.1480000000000001,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":0.79600000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.4890000000000001},"min":{"1min":0.78200000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.298,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.115},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.4790000000000001},"min":{"1min":0.88900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.155}]},{"osd":5,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.51,"15min":1.548},"min":{"1min":0.89000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.462,"15min":1.4650000000000001},"min":{"1min":0.80600000000000005,"5min":0.627,"15min":0.627},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.4910000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.524,"15min":1.52},"min":{"1min":0.91300000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.681},{"interface":"front","average":{"1min":1.524,"5min":1.5580000000000001,"15min":1.605},"min":{"1min":0.77900000000000003,"5min":0.752,"15min":0.752},"max":{"1min":2.52,"5min":2.52,"15min":2.5569999999999999},"last":1.5309999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.641,"15min":1.6599999999999999},"min":{"1min":0.996,"5min":0.754,"15min":0.754},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.613,"15min":1.637},"min":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.605}]}]},{"osd":1,"up_from":14,"seq":60129543335,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57352,"kb_used_data":3964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314488,"statfs":{"total":96636764160,"available":96578035712,"internally_reserved":0,"allocated":4059136,"data_stored":2195256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":1.2330000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.008},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.4239999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.1739999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.29},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.591}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.651},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":0.80900000000000005}]}]},{"osd":0,"up_from":14,"seq":60129543336,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62032,"kb_used_data":4484,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94309808,"statfs":{"total":96636764160,"available":96573243392,"internally_reserved":0,"allocated":4591616,"data_stored":2720676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91600000000000004,"15min":0.93999999999999995},"min":{"1min":0.46200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.98999999999999999,"15min":1.1140000000000001},"min":{"1min":0.60599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9299999999999999},"last":0.81200000000000006}]},{"osd":2,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0609999999999999,"15min":1.149},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.768,"5min":1.768,"15min":2.0569999999999999},"last":1.181},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0880000000000001,"15min":1.1890000000000001},"min":{"1min":0.72299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":1.3260000000000001,"5min":1.78,"15min":1.893},"last":0.93400000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0269999999999999,"15min":1.125},"min":{"1min":0.66700000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2290000000000001,"5min":1.7869999999999999,"15min":1.8520000000000001},"last":1.008},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2350000000000001,"15min":1.341},"min":{"1min":0.79500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5880000000000001,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.052}]},{"osd":4,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.343},"min":{"1min":0.95399999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5900000000000001,"5min":1.923,"15min":2.0070000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3979999999999999,"15min":1.4910000000000001},"min":{"1min":0.996,"5min":0.748,"15min":0.748},"max":{"1min":2.0169999999999999,"5min":2.1070000000000002,"15min":2.601},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.4810000000000001},"min":{"1min":0.80100000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.081,"5min":2.089,"15min":2.3279999999999998},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.3720000000000001,"15min":1.45},"min":{"1min":0.89700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.754,"5min":2.0169999999999999,"15min":2.2829999999999999},"last":1.226}]},{"osd":6,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5049999999999999,"15min":1.5880000000000001},"min":{"1min":0.94099999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0990000000000002,"5min":2.2730000000000001,"15min":2.669},"last":1.395},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.494,"15min":1.5720000000000001},"min":{"1min":0.86099999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.893,"5min":2.327,"15min":2.3769999999999998},"last":1.7470000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5840000000000001,"15min":1.677},"min":{"1min":0.93899999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.5270000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.5249999999999999,"15min":1.6120000000000001},"min":{"1min":1.119,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.893,"5min":2.3679999999999999,"15min":2.464},"last":1.3380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:14:00.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:14:00.676 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.673+0000 7fa119875640 1 Processor -- start 2023-10-16T14:14:00.677 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.673+0000 7fa119875640 1 -- start start 2023-10-16T14:14:00.678 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa119875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1141615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa119875640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1140b0410 0x7fa1140b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa119875640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa11415d3f0 0x7fa11415d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.679 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa119875640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1140b0db0 con 0x7fa1140b0410 2023-10-16T14:14:00.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa119875640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11415e3e0 con 0x7fa11415d3f0 2023-10-16T14:14:00.680 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa119875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140cd4a0 con 0x7fa1140cd0c0 2023-10-16T14:14:00.681 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa10bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1140b0410 0x7fa1140b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa10bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1140b0410 0x7fa1140b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47932/0 (socket says 172.21.15.60:47932) 2023-10-16T14:14:00.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.675+0000 7fa10bfff640 1 -- 172.21.15.60:0/4064313788 learned_addr learned my addr 172.21.15.60:0/4064313788 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:00.682 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa1137fe640 1 --2- 172.21.15.60:0/4064313788 >> v2:172.21.15.144:3300/0 conn(0x7fa11415d3f0 0x7fa11415d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa112ffd640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1141615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa10bfff640 1 -- 172.21.15.60:0/4064313788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 msgr2=0x7fa1141615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:00.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa10bfff640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1141615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.683 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa10bfff640 1 -- 172.21.15.60:0/4064313788 >> v2:172.21.15.144:3300/0 conn(0x7fa11415d3f0 msgr2=0x7fa11415d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:00.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa10bfff640 1 --2- 172.21.15.60:0/4064313788 >> v2:172.21.15.144:3300/0 conn(0x7fa11415d3f0 0x7fa11415d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa10bfff640 1 -- 172.21.15.60:0/4064313788 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11415e100 con 0x7fa1140b0410 2023-10-16T14:14:00.684 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.676+0000 7fa112ffd640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1141615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:00.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.677+0000 7fa10bfff640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa1140b0870 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fa11406da30 tx=0x7fa0fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e593620ec7ef682 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:00.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.678+0000 7fa1127fc640 1 -- 172.21.15.60:0/4064313788 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc0317e0 con 0x7fa1140b0410 2023-10-16T14:14:00.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.678+0000 7fa1127fc640 1 -- 172.21.15.60:0/4064313788 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc031940 con 0x7fa1140b0410 2023-10-16T14:14:00.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.678+0000 7fa1127fc640 1 -- 172.21.15.60:0/4064313788 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc031cd0 con 0x7fa1140b0410 2023-10-16T14:14:00.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.678+0000 7fa119875640 1 -- 172.21.15.60:0/4064313788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 msgr2=0x7fa1140b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.678+0000 7fa119875640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa1140b0870 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fa11406da30 tx=0x7fa0fc02b880 comp rx=0 tx=0).stop 2023-10-16T14:14:00.688 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 -- 172.21.15.60:0/4064313788 shutdown_connections 2023-10-16T14:14:00.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 --2- 172.21.15.60:0/4064313788 >> v2:172.21.15.144:3300/0 conn(0x7fa11415d3f0 0x7fa11415d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa1140b0870 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 --2- 172.21.15.60:0/4064313788 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1141615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 -- 172.21.15.60:0/4064313788 >> 172.21.15.60:0/4064313788 conn(0x7fa1140cb410 msgr2=0x7fa1140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:00.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 -- 172.21.15.60:0/4064313788 shutdown_connections 2023-10-16T14:14:00.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.679+0000 7fa119875640 1 -- 172.21.15.60:0/4064313788 wait complete. 2023-10-16T14:14:00.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.681+0000 7fa119875640 1 Processor -- start 2023-10-16T14:14:00.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.681+0000 7fa119875640 1 -- start start 2023-10-16T14:14:00.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa119875640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa114149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa119875640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1140c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa119875640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa11415d3f0 0x7fa11415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa119875640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11414a290 con 0x7fa1140b0410 2023-10-16T14:14:00.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa119875640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140aef70 con 0x7fa11415d3f0 2023-10-16T14:14:00.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa119875640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140af210 con 0x7fa1140cd0c0 2023-10-16T14:14:00.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa114149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa114149d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47936/0 (socket says 172.21.15.60:47936) 2023-10-16T14:14:00.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 -- 172.21.15.60:0/2708051363 learned_addr learned my addr 172.21.15.60:0/2708051363 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:00.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa10bfff640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1140c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa1137fe640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa11415d3f0 0x7fa11415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 -- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 msgr2=0x7fa1140c2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1140c2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 -- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa11415d3f0 msgr2=0x7fa11415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa11415d3f0 0x7fa11415aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.682+0000 7fa112ffd640 1 -- 172.21.15.60:0/2708051363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0094d0 con 0x7fa1140b0410 2023-10-16T14:14:00.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.683+0000 7fa112ffd640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa114149d50 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c7f0 tx=0x7fa10400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:00.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.683+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104007820 con 0x7fa1140b0410 2023-10-16T14:14:00.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.683+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104007980 con 0x7fa1140b0410 2023-10-16T14:14:00.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.683+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104007d30 con 0x7fa1140b0410 2023-10-16T14:14:00.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.683+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140af4f0 con 0x7fa1140b0410 2023-10-16T14:14:00.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.683+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1140afab0 con 0x7fa1140b0410 2023-10-16T14:14:00.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.685+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1140adb60 con 0x7fa1140b0410 2023-10-16T14:14:00.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.686+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa1040113f0 con 0x7fa1140b0410 2023-10-16T14:14:00.698 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.686+0000 7fa110ff9640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa0f406af90 0x7fa0f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:00.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.687+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fa104089bf0 con 0x7fa1140b0410 2023-10-16T14:14:00.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.687+0000 7fa10bfff640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa0f406af90 0x7fa0f406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:00.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.688+0000 7fa10bfff640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa0f406af90 0x7fa0f406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc036000 tx=0x7fa0fc03d040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:00.699 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.691+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1040549d0 con 0x7fa1140b0410 2023-10-16T14:14:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.884+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa114160b10 con 0x7fa0f406af90 2023-10-16T14:14:00.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.891+0000 7fa110ff9640 1 -- 172.21.15.60:0/2708051363 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48182 (secure 0 0 0) 0x7fa0fc031b30 con 0x7fa0f406af90 2023-10-16T14:14:00.895 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:00.895 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:14:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa0f406af90 msgr2=0x7fa0f406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa0f406af90 0x7fa0f406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc036000 tx=0x7fa0fc03d040 comp rx=0 tx=0).stop 2023-10-16T14:14:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 msgr2=0x7fa114149d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa114149d50 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c7f0 tx=0x7fa10400ccb0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 shutdown_connections 2023-10-16T14:14:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fa0f406af90 0x7fa0f406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fa1140cd0c0 0x7fa1140c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa11415d3f0 0x7fa11415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 --2- 172.21.15.60:0/2708051363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1140b0410 0x7fa114149d50 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.893+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 >> 172.21.15.60:0/2708051363 conn(0x7fa1140cb410 msgr2=0x7fa1140cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.894+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 shutdown_connections 2023-10-16T14:14:00.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:00.894+0000 7fa119875640 1 -- 172.21.15.60:0/2708051363 wait complete. 2023-10-16T14:14:00.916 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-16T14:13:59.095045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":473744,"kb_used_data":31408,"kb_used_omap":12,"kb_used_meta":442227,"kb_avail":754500976,"statfs":{"total":773094113280,"available":772608999424,"internally_reserved":0,"allocated":32161792,"data_stored":17224140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":452841016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-810930,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-312,"num_read_kb":-156,"num_write":-368,"num_write_kb":-3126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002025"},"pg_stats":[{"pgid":"1.0","version":"289'2843","reported_seq":4002,"reported_epoch":289,"state":"active+clean","last_fresh":"2023-10-16T14:13:51.197818+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:13:51.197818+0000","last_peered":"2023-10-16T14:13:51.197818+0000","last_clean":"2023-10-16T14:13:51.197818+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:13:51.197818+0000","last_undegraded":"2023-10-16T14:13:51.197818+0000","last_fullsized":"2023-10-16T14:13:51.197818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543335,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49608,"kb_used_data":4540,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322232,"statfs":{"total":96636764160,"available":96585965568,"internally_reserved":0,"allocated":4648960,"data_stored":2785687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2889999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":1.381},{"interface":"front","average":{"1min":1.331,"5min":1.4139999999999999,"15min":1.4079999999999999},"min":{"1min":0.76400000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.133,"5min":6.391,"15min":6.391},"last":1.431}]},{"osd":1,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.212,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.482},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.238,"15min":1.252},"min":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.982,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.228,"15min":1.258},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.736,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.242},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.745,"15min":0.745},"max":{"1min":2.0529999999999999,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.181}]},{"osd":3,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3660000000000001,"15min":1.403},"min":{"1min":0.86799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.853,"5min":2.0790000000000002,"15min":2.097},"last":1.3},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.823,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.276}]},{"osd":4,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.256,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.593,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.133},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4630000000000001,"15min":1.4239999999999999},"min":{"1min":1.0089999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8620000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.448,"15min":1.397},"min":{"1min":0.879,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.0880000000000001,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.349},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.5589999999999999,"15min":1.5429999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3159999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.6339999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.6579999999999999,"15min":1.641},"min":{"1min":1.0680000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.4500000000000002,"5min":3.29,"15min":3.29},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5529999999999999,"15min":1.496},"min":{"1min":0.85999999999999999,"5min":0.75,"15min":0.75},"max":{"1min":2.0419999999999998,"5min":3.524,"15min":3.524},"last":1.548}]}]},{"osd":6,"up_from":14,"seq":60129543336,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48744,"kb_used_data":3676,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323096,"statfs":{"total":96636764160,"available":96586850304,"internally_reserved":0,"allocated":3764224,"data_stored":1893201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2330000000000001,"15min":1.2989999999999999},"min":{"1min":0.92300000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6200000000000001,"5min":1.788,"15min":1.887},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.266,"5min":1.2909999999999999,"15min":1.3420000000000001},"min":{"1min":0.85499999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7509999999999999,"5min":1.847,"15min":1.9390000000000001},"last":1.252}]},{"osd":1,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.157,"15min":1.204},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6819999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.032},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.1970000000000001},"min":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.64200000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.77},"last":1.202}]},{"osd":2,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.86899999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.24,"5min":1.294,"15min":1.3029999999999999},"min":{"1min":0.76400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.5549999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3240000000000001,"15min":1.339},"min":{"1min":0.95199999999999996,"5min":0.79300000000000004,"15min":0.70499999999999996},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.409},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.1519999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4219999999999999,"15min":1.4119999999999999},"min":{"1min":0.90700000000000003,"5min":0.77300000000000002,"15min":0.51400000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5580000000000001,"15min":1.548},"min":{"1min":0.88900000000000001,"5min":0.86299999999999999,"15min":0.78400000000000003},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.1179999999999999},"last":1.355}]},{"osd":5,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.427,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.964,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.675},{"interface":"front","average":{"1min":1.617,"5min":1.526,"15min":1.496},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.64400000000000002},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.514}]},{"osd":7,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.5309999999999999,"15min":1.512},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.68600000000000005},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.468},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5589999999999999,"15min":1.54},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.6120000000000001},"last":1.774}]}]},{"osd":5,"up_from":14,"seq":60129543337,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52908,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318932,"statfs":{"total":96636764160,"available":96582586368,"internally_reserved":0,"allocated":3768320,"data_stored":1893475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0900000000000001,"15min":1.181},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":2.8330000000000002},"last":0.874},{"interface":"front","average":{"1min":1.216,"5min":1.2070000000000001,"15min":1.2170000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9830000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.95199999999999996}]},{"osd":1,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.804,"5min":1.871,"15min":1.8779999999999999},"last":1.593},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.153,"15min":1.23},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.3799999999999999,"5min":1.931,"15min":2.5840000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.2569999999999999,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.65400000000000003,"15min":0.63},"max":{"1min":1.591,"5min":2.024,"15min":2.8119999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.1950000000000001,"15min":1.2849999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.492,"5min":1.8600000000000001,"15min":2.6259999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.387,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.821},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.361,"15min":1.419},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.794,"5min":1.8759999999999999,"15min":2.75},"last":1.1950000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2869999999999999,"15min":1.302},"min":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.982,"5min":1.982,"15min":2.774},"last":2.149},{"interface":"front","average":{"1min":1.504,"5min":1.478,"15min":1.4470000000000001},"min":{"1min":1.2490000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.3330000000000002},"last":2.0609999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4550000000000001,"15min":1.4399999999999999},"min":{"1min":1.028,"5min":0.59399999999999997,"15min":0.55900000000000005},"max":{"1min":2.081,"5min":2.081,"15min":2.6720000000000002},"last":1.732},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.89100000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7010000000000001},"last":2.242}]},{"osd":7,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.593,"15min":1.5580000000000001},"min":{"1min":1.278,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.4609999999999999},"last":1.97},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":1.1579999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.2010000000000001,"5min":2.238,"15min":2.8719999999999999},"last":1.9279999999999999}]}]},{"osd":4,"up_from":14,"seq":60129543337,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89168,"kb_used_data":3908,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282672,"statfs":{"total":96636764160,"available":96545456128,"internally_reserved":0,"allocated":4001792,"data_stored":2131171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.1930000000000001},"min":{"1min":0.85899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.575,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.105},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.159,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5629999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.179}]},{"osd":1,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1439999999999999,"15min":1.2270000000000001},"min":{"1min":0.73099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.27},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7210000000000001,"5min":1.792,"15min":2.0499999999999998},"last":1.2210000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2529999999999999,"15min":1.302},"min":{"1min":0.81599999999999995,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.4299999999999999,"5min":1.714,"15min":1.865},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.365,"15min":1.3779999999999999},"min":{"1min":1.105,"5min":0.87,"15min":0.87},"max":{"1min":1.77,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.4199999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.292,"15min":1.2989999999999999},"min":{"1min":0.996,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9650000000000001},"last":1.075},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.425},"min":{"1min":1.1459999999999999,"5min":0.92800000000000005,"15min":0.86799999999999999},"max":{"1min":1.887,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.4670000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4019999999999999,"15min":1.3540000000000001},"min":{"1min":0.997,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.738,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.415,"15min":1.3600000000000001},"min":{"1min":0.97499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0659999999999998,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.5129999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4390000000000001,"15min":1.3999999999999999},"min":{"1min":1.091,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.972},{"interface":"front","average":{"1min":1.649,"5min":1.5760000000000001,"15min":1.5289999999999999},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":1.9790000000000001,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.581}]},{"osd":7,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.591,"15min":1.5269999999999999},"min":{"1min":1.22,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.1219999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.399},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.657,"15min":1.645},"min":{"1min":1.335,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.3660000000000001},"last":1.6639999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543337,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56964,"kb_used_data":3576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314876,"statfs":{"total":96636764160,"available":96578433024,"internally_reserved":0,"allocated":3661824,"data_stored":1802566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91000000000000003,"15min":0.93799999999999994},"min":{"1min":0.53900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.54,"5min":1.595,"15min":1.595},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.105,"15min":1.1899999999999999},"min":{"1min":0.57399999999999995,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.5529999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.98599999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0229999999999999,"15min":1.0700000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.6319999999999999,"5min":1.6870000000000001,"15min":1.855},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0620000000000001,"15min":1.105},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.9890000000000001,"5min":2.125,"15min":2.125},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0569999999999999,"15min":1.1739999999999999},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.544,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":1.073},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.4039999999999999},"min":{"1min":0.93200000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6739999999999999,"5min":1.909,"15min":1.9319999999999999},"last":1.1319999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4390000000000001,"15min":1.4870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.67000000000000004},"max":{"1min":1.891,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.27},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3859999999999999,"15min":1.4470000000000001},"min":{"1min":0.80500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6599999999999999,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.51}]},{"osd":5,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4019999999999999,"15min":1.4730000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0579999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4950000000000001,"15min":1.5409999999999999},"min":{"1min":0.88700000000000001,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9339999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.288}]},{"osd":6,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.508},"min":{"1min":0.96599999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.8300000000000001,"5min":2.335,"15min":2.4329999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.4279999999999999,"15min":1.486},"min":{"1min":0.752,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9279999999999999,"5min":2.194,"15min":2.4180000000000001},"last":1.3420000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5289999999999999,"15min":1.5840000000000001},"min":{"1min":0.875,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1320000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.611,"15min":1.665},"min":{"1min":0.91300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.089,"5min":2.569,"15min":2.6070000000000002},"last":1.4750000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543335,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56968,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314872,"statfs":{"total":96636764160,"available":96578428928,"internally_reserved":0,"allocated":3665920,"data_stored":1802108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.93200000000000005,"15min":0.94999999999999996},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.47599999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.7090000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.073}]},{"osd":1,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5580000000000001,"5min":1.847,"15min":1.847},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.125,"15min":1.2210000000000001},"min":{"1min":0.67100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.415,"5min":1.784,"15min":1.9530000000000001},"last":0.93999999999999995}]},{"osd":3,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.05,"15min":1.0980000000000001},"min":{"1min":0.56100000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3759999999999999,"5min":1.9179999999999999,"15min":2.1299999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.252,"15min":1.3460000000000001},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.1480000000000001,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":0.79600000000000004}]},{"osd":4,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.4890000000000001},"min":{"1min":0.78200000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.298,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.115},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.4790000000000001},"min":{"1min":0.88900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.155}]},{"osd":5,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.51,"15min":1.548},"min":{"1min":0.89000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.462,"15min":1.4650000000000001},"min":{"1min":0.80600000000000005,"5min":0.627,"15min":0.627},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.4910000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.524,"15min":1.52},"min":{"1min":0.91300000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.681},{"interface":"front","average":{"1min":1.524,"5min":1.5580000000000001,"15min":1.605},"min":{"1min":0.77900000000000003,"5min":0.752,"15min":0.752},"max":{"1min":2.52,"5min":2.52,"15min":2.5569999999999999},"last":1.5309999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.641,"15min":1.6599999999999999},"min":{"1min":0.996,"5min":0.754,"15min":0.754},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.613,"15min":1.637},"min":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.605}]}]},{"osd":1,"up_from":14,"seq":60129543335,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57352,"kb_used_data":3964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314488,"statfs":{"total":96636764160,"available":96578035712,"internally_reserved":0,"allocated":4059136,"data_stored":2195256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":1.2330000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.008},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.4239999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.1739999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.78400000000000003}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.29},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.591}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.651},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":0.80900000000000005}]}]},{"osd":0,"up_from":14,"seq":60129543336,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62032,"kb_used_data":4484,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94309808,"statfs":{"total":96636764160,"available":96573243392,"internally_reserved":0,"allocated":4591616,"data_stored":2720676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91600000000000004,"15min":0.93999999999999995},"min":{"1min":0.46200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.98999999999999999,"15min":1.1140000000000001},"min":{"1min":0.60599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9299999999999999},"last":0.81200000000000006}]},{"osd":2,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0609999999999999,"15min":1.149},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.768,"5min":1.768,"15min":2.0569999999999999},"last":1.181},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0880000000000001,"15min":1.1890000000000001},"min":{"1min":0.72299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":1.3260000000000001,"5min":1.78,"15min":1.893},"last":0.93400000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0269999999999999,"15min":1.125},"min":{"1min":0.66700000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2290000000000001,"5min":1.7869999999999999,"15min":1.8520000000000001},"last":1.008},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2350000000000001,"15min":1.341},"min":{"1min":0.79500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5880000000000001,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.052}]},{"osd":4,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.343},"min":{"1min":0.95399999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5900000000000001,"5min":1.923,"15min":2.0070000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3979999999999999,"15min":1.4910000000000001},"min":{"1min":0.996,"5min":0.748,"15min":0.748},"max":{"1min":2.0169999999999999,"5min":2.1070000000000002,"15min":2.601},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.4810000000000001},"min":{"1min":0.80100000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.081,"5min":2.089,"15min":2.3279999999999998},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.3720000000000001,"15min":1.45},"min":{"1min":0.89700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.754,"5min":2.0169999999999999,"15min":2.2829999999999999},"last":1.226}]},{"osd":6,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5049999999999999,"15min":1.5880000000000001},"min":{"1min":0.94099999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0990000000000002,"5min":2.2730000000000001,"15min":2.669},"last":1.395},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.494,"15min":1.5720000000000001},"min":{"1min":0.86099999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.893,"5min":2.327,"15min":2.3769999999999998},"last":1.7470000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5840000000000001,"15min":1.677},"min":{"1min":0.93899999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.5270000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.5249999999999999,"15min":1.6120000000000001},"min":{"1min":1.119,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.893,"5min":2.3679999999999999,"15min":2.464},"last":1.3380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:14:00.918 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T14:14:00.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:14:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.136+0000 7f50d70bb640 1 Processor -- start 2023-10-16T14:14:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.136+0000 7f50d70bb640 1 -- start start 2023-10-16T14:14:01.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d70bb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50d015ae80 0x7f50d015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d70bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 0x7f50d0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d70bb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d015ed60 con 0x7f50d015aaa0 2023-10-16T14:14:01.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d70bb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d006b810 con 0x7f50d015ae80 2023-10-16T14:14:01.142 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d70bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00707e0 con 0x7f50d00ce390 2023-10-16T14:14:01.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d4e57640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f50d015aaa0 0x7f50d00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53414/0 (socket says 172.21.15.60:53414) 2023-10-16T14:14:01.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.138+0000 7f50d4e57640 1 -- 172.21.15.60:0/343434083 learned_addr learned my addr 172.21.15.60:0/343434083 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cffff640 1 --2- 172.21.15.60:0/343434083 >> v2:172.21.15.144:3300/0 conn(0x7f50d015ae80 0x7f50d015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50d5658640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 0x7f50d0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252482623 0 0) 0x7f50d015ed60 con 0x7f50d015aaa0 2023-10-16T14:14:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c0003520 con 0x7f50d015aaa0 2023-10-16T14:14:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3932803628 0 0) 0x7f50c0003520 con 0x7f50d015aaa0 2023-10-16T14:14:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 msgr2=0x7f50d0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 0x7f50d0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 >> v2:172.21.15.144:3300/0 conn(0x7f50d015ae80 msgr2=0x7f50d015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 --2- 172.21.15.60:0/343434083 >> v2:172.21.15.144:3300/0 conn(0x7f50d015ae80 0x7f50d015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d015ef90 con 0x7f50d015aaa0 2023-10-16T14:14:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f50c4001690 con 0x7f50d015aaa0 2023-10-16T14:14:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 >> v1:172.21.15.60:6789/0 conn(0x7f50d015aaa0 legacy=0x7f50d00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 0x7f50c0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.139+0000 7f50cf7fe640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50c0004d00 0x7f50c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d006b810 con 0x7f50c0003f40 2023-10-16T14:14:01.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00707e0 con 0x7f50c0004d00 2023-10-16T14:14:01.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c000b980 con 0x7f50d00ce390 2023-10-16T14:14:01.149 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cffff640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50d4e57640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 0x7f50c0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50d5658640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50c0004d00 0x7f50c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cffff640 1 -- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 msgr2=0x7f50c0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:01.150 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cffff640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00ce390 0x7f50c0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cffff640 1 -- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50c0004d00 msgr2=0x7f50c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:01.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cffff640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50c0004d00 0x7f50c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50cffff640 1 -- 172.21.15.60:0/343434083 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0003520 con 0x7f50c0003f40 2023-10-16T14:14:01.151 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.140+0000 7f50d5658640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50c0004d00 0x7f50c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.142+0000 7f50cffff640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50c00047c0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f50bc009ac0 tx=0x7f50c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586b4df677eb0e0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.142+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc02c6b0 con 0x7f50c0003f40 2023-10-16T14:14:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.142+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc02c810 con 0x7f50c0003f40 2023-10-16T14:14:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.142+0000 7f50cf7fe640 1 -- 172.21.15.60:0/343434083 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc02cba0 con 0x7f50c0003f40 2023-10-16T14:14:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d70bb640 1 -- 172.21.15.60:0/343434083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 msgr2=0x7f50c00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d70bb640 1 --2- 172.21.15.60:0/343434083 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50c00047c0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f50bc009ac0 tx=0x7f50c00057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d5658640 1 -- 172.21.15.60:0/343434083 reap_dead start 2023-10-16T14:14:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d70bb640 1 -- 172.21.15.60:0/343434083 shutdown_connections 2023-10-16T14:14:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d70bb640 1 -- 172.21.15.60:0/343434083 >> 172.21.15.60:0/343434083 conn(0x7f50d00cb410 msgr2=0x7f50d00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d70bb640 1 -- 172.21.15.60:0/343434083 shutdown_connections 2023-10-16T14:14:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.143+0000 7f50d70bb640 1 -- 172.21.15.60:0/343434083 wait complete. 2023-10-16T14:14:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.145+0000 7f50d70bb640 1 Processor -- start 2023-10-16T14:14:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.145+0000 7f50d70bb640 1 -- start start 2023-10-16T14:14:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d70bb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50d014c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d70bb640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00c3710 0x7f50d00afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d70bb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d00c2d60 0x7f50d00b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d70bb640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00c3c10 con 0x7f50c0003f40 2023-10-16T14:14:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d70bb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00c3ef0 con 0x7f50d00c2d60 2023-10-16T14:14:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d70bb640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d00c2560 con 0x7f50d00c3710 2023-10-16T14:14:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d4e57640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50d014c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d4e57640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50d014c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47956/0 (socket says 172.21.15.60:47956) 2023-10-16T14:14:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50d4e57640 1 -- 172.21.15.60:0/2394662442 learned_addr learned my addr 172.21.15.60:0/2394662442 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.146+0000 7f50cffff640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00c3710 0x7f50d00afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d5658640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d00c2d60 0x7f50d00b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d4e57640 1 -- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00c3710 msgr2=0x7f50d00afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d4e57640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00c3710 0x7f50d00afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d4e57640 1 -- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d00c2d60 msgr2=0x7f50d00b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d4e57640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d00c2d60 0x7f50d00b0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d4e57640 1 -- 172.21.15.60:0/2394662442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d00c2840 con 0x7f50c0003f40 2023-10-16T14:14:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d4e57640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50d014c790 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f50c400e700 tx=0x7f50c400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400c6c0 con 0x7f50c0003f40 2023-10-16T14:14:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c400c820 con 0x7f50c0003f40 2023-10-16T14:14:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400cbd0 con 0x7f50c0003f40 2023-10-16T14:14:01.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.147+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d014ccd0 con 0x7f50c0003f40 2023-10-16T14:14:01.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.148+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d00b00f0 con 0x7f50c0003f40 2023-10-16T14:14:01.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.149+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d004e160 con 0x7f50c0003f40 2023-10-16T14:14:01.163 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.150+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f50c400cd30 con 0x7f50c0003f40 2023-10-16T14:14:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.151+0000 7f50cd7fa640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f50b406afc0 0x7f50b406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.164 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.151+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f50c4089690 con 0x7f50c0003f40 2023-10-16T14:14:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.151+0000 7f50cffff640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f50b406afc0 0x7f50b406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.152+0000 7f50cffff640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f50b406afc0 0x7f50b406d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f50bc036000 tx=0x7f50bc030000 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.155+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50c4054600 con 0x7f50c0003f40 2023-10-16T14:14:01.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.350+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50d015f550 con 0x7f50b406afc0 2023-10-16T14:14:01.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.357+0000 7f50cd7fa640 1 -- 172.21.15.60:0/2394662442 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48379 (secure 0 0 0) 0x7f50bc004140 con 0x7f50b406afc0 2023-10-16T14:14:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:01.362 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:14:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.359+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f50b406afc0 msgr2=0x7f50b406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.359+0000 7f50d70bb640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f50b406afc0 0x7f50b406d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f50bc036000 tx=0x7f50bc030000 comp rx=0 tx=0).stop 2023-10-16T14:14:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.359+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 msgr2=0x7f50d014c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.359+0000 7f50d70bb640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50d014c790 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f50c400e700 tx=0x7f50c400ebc0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 shutdown_connections 2023-10-16T14:14:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f50b406afc0 0x7f50b406d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f50d00c3710 0x7f50d00afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50d00c2d60 0x7f50d00b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 --2- 172.21.15.60:0/2394662442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50c0003f40 0x7f50d014c790 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 >> 172.21.15.60:0/2394662442 conn(0x7f50d00cb410 msgr2=0x7f50d015a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:01.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 shutdown_connections 2023-10-16T14:14:01.367 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.360+0000 7f50d70bb640 1 -- 172.21.15.60:0/2394662442 wait complete. 2023-10-16T14:14:01.383 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-16T14:14:01.095703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":472344,"kb_used_data":29992,"kb_used_omap":12,"kb_used_meta":442227,"kb_avail":754502376,"statfs":{"total":773094113280,"available":772610433024,"internally_reserved":0,"allocated":30711808,"data_stored":15702224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":452841016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-810930,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-312,"num_read_kb":-156,"num_write":-368,"num_write_kb":-3126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.787210"},"pg_stats":[{"pgid":"1.0","version":"289'2843","reported_seq":4005,"reported_epoch":292,"state":"active+clean","last_fresh":"2023-10-16T14:13:56.826895+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:13:56.826895+0000","last_peered":"2023-10-16T14:13:56.826895+0000","last_clean":"2023-10-16T14:13:56.826895+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:13:56.826895+0000","last_undegraded":"2023-10-16T14:13:56.826895+0000","last_fullsized":"2023-10-16T14:13:56.826895+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49256,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322584,"statfs":{"total":96636764160,"available":96586326016,"internally_reserved":0,"allocated":4284416,"data_stored":2405554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2889999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":1.03},{"interface":"front","average":{"1min":1.331,"5min":1.4139999999999999,"15min":1.4079999999999999},"min":{"1min":0.76400000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.133,"5min":6.391,"15min":6.391},"last":0.92200000000000004}]},{"osd":1,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.212,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.238,"15min":1.252},"min":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.982,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.228,"15min":1.258},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.736,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.745,"15min":0.745},"max":{"1min":2.0529999999999999,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.0509999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3660000000000001,"15min":1.403},"min":{"1min":0.86799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.853,"5min":2.0790000000000002,"15min":2.097},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.823,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.113}]},{"osd":4,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.256,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.593,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4630000000000001,"15min":1.4239999999999999},"min":{"1min":1.0089999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8620000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.448,"15min":1.397},"min":{"1min":0.879,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.0880000000000001,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.5589999999999999,"15min":1.5429999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3159999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.6579999999999999,"15min":1.641},"min":{"1min":1.0680000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.4500000000000002,"5min":3.29,"15min":3.29},"last":0.625},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5529999999999999,"15min":1.496},"min":{"1min":0.85999999999999999,"5min":0.75,"15min":0.75},"max":{"1min":2.0419999999999998,"5min":3.524,"15min":3.524},"last":0.88300000000000001}]}]},{"osd":6,"up_from":14,"seq":60129543337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48672,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323168,"statfs":{"total":96636764160,"available":96586924032,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2330000000000001,"15min":1.2989999999999999},"min":{"1min":0.92300000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6200000000000001,"5min":1.788,"15min":1.887},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.2909999999999999,"15min":1.3420000000000001},"min":{"1min":0.85499999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7509999999999999,"5min":1.847,"15min":1.9390000000000001},"last":1.1899999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.157,"15min":1.204},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6819999999999999,"5min":1.8360000000000001,"15min":2.008},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.1970000000000001},"min":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.64200000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.77},"last":1.4970000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.86899999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.389},{"interface":"front","average":{"1min":1.24,"5min":1.294,"15min":1.3029999999999999},"min":{"1min":0.76400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.4490000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3240000000000001,"15min":1.339},"min":{"1min":0.95199999999999996,"5min":0.79300000000000004,"15min":0.70499999999999996},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.071},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.651}]},{"osd":4,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4219999999999999,"15min":1.4119999999999999},"min":{"1min":0.90700000000000003,"5min":0.77300000000000002,"15min":0.51400000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5580000000000001,"15min":1.548},"min":{"1min":0.88900000000000001,"5min":0.86299999999999999,"15min":0.78400000000000003},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.1179999999999999},"last":1.5389999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.427,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.964,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.617,"5min":1.526,"15min":1.496},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.64400000000000002},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.7030000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.5309999999999999,"15min":1.512},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.68600000000000005},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.766},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5589999999999999,"15min":1.54},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.6120000000000001},"last":1.2490000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52832,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319008,"statfs":{"total":96636764160,"available":96582664192,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0900000000000001,"15min":1.181},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":2.8330000000000002},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.216,"5min":1.2070000000000001,"15min":1.2170000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9830000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.91800000000000004}]},{"osd":1,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.804,"5min":1.871,"15min":1.8779999999999999},"last":1.278},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.153,"15min":1.23},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.3799999999999999,"5min":1.931,"15min":2.5840000000000001},"last":1.02}]},{"osd":2,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.2569999999999999,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.65400000000000003,"15min":0.63},"max":{"1min":1.591,"5min":2.024,"15min":2.8119999999999998},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.1950000000000001,"15min":1.2849999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.492,"5min":1.8600000000000001,"15min":2.6259999999999999},"last":1.103}]},{"osd":3,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.387,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.361,"15min":1.419},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.794,"5min":1.8759999999999999,"15min":2.75},"last":1.1279999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2869999999999999,"15min":1.302},"min":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.982,"5min":1.982,"15min":2.774},"last":0.755},{"interface":"front","average":{"1min":1.504,"5min":1.478,"15min":1.4470000000000001},"min":{"1min":1.2490000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.3330000000000002},"last":0.79700000000000004}]},{"osd":6,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4550000000000001,"15min":1.4399999999999999},"min":{"1min":1.028,"5min":0.59399999999999997,"15min":0.55900000000000005},"max":{"1min":2.081,"5min":2.081,"15min":2.6720000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.89100000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7010000000000001},"last":0.88400000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.593,"15min":1.5580000000000001},"min":{"1min":1.278,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.4609999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":1.1579999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.2010000000000001,"5min":2.238,"15min":2.8719999999999999},"last":0.96399999999999997}]}]},{"osd":4,"up_from":14,"seq":60129543338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88864,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282976,"statfs":{"total":96636764160,"available":96545767424,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.1930000000000001},"min":{"1min":0.85899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.575,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.159,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5629999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":0.995}]},{"osd":1,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1439999999999999,"15min":1.2270000000000001},"min":{"1min":0.73099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.171},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.27},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7210000000000001,"5min":1.792,"15min":2.0499999999999998},"last":1.256}]},{"osd":2,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2529999999999999,"15min":1.302},"min":{"1min":0.81599999999999995,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.4299999999999999,"5min":1.714,"15min":1.865},"last":1.48},{"interface":"front","average":{"1min":1.381,"5min":1.365,"15min":1.3779999999999999},"min":{"1min":1.105,"5min":0.87,"15min":0.87},"max":{"1min":1.77,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.073}]},{"osd":3,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.292,"15min":1.2989999999999999},"min":{"1min":0.996,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9650000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.425},"min":{"1min":1.1459999999999999,"5min":0.92800000000000005,"15min":0.86799999999999999},"max":{"1min":1.887,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.3109999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4019999999999999,"15min":1.3540000000000001},"min":{"1min":0.997,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.738,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.415,"15min":1.3600000000000001},"min":{"1min":0.97499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0659999999999998,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.21}]},{"osd":6,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4390000000000001,"15min":1.3999999999999999},"min":{"1min":1.091,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.649,"5min":1.5760000000000001,"15min":1.5289999999999999},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":1.9790000000000001,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.379}]},{"osd":7,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.591,"15min":1.5269999999999999},"min":{"1min":1.22,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.1219999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.141},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.657,"15min":1.645},"min":{"1min":1.335,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.3660000000000001},"last":1.4219999999999999}]}]},{"osd":3,"up_from":14,"seq":60129543338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91000000000000003,"15min":0.93799999999999994},"min":{"1min":0.53900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.54,"5min":1.595,"15min":1.595},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.105,"15min":1.1899999999999999},"min":{"1min":0.57399999999999995,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.5529999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.876}]},{"osd":1,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0229999999999999,"15min":1.0700000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.6319999999999999,"5min":1.6870000000000001,"15min":1.855},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0620000000000001,"15min":1.105},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.9890000000000001,"5min":2.125,"15min":2.125},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0569999999999999,"15min":1.1739999999999999},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.544,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":0.316},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.4039999999999999},"min":{"1min":0.93200000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6739999999999999,"5min":1.909,"15min":1.9319999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4390000000000001,"15min":1.4870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.67000000000000004},"max":{"1min":1.891,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3859999999999999,"15min":1.4470000000000001},"min":{"1min":0.80500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6599999999999999,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.5840000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4019999999999999,"15min":1.4730000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0579999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4950000000000001,"15min":1.5409999999999999},"min":{"1min":0.88700000000000001,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9339999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.244}]},{"osd":6,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.508},"min":{"1min":0.96599999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.8300000000000001,"5min":2.335,"15min":2.4329999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.417,"5min":1.4279999999999999,"15min":1.486},"min":{"1min":0.752,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9279999999999999,"5min":2.194,"15min":2.4180000000000001},"last":1.5109999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5289999999999999,"15min":1.5840000000000001},"min":{"1min":0.875,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1320000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.611,"15min":1.665},"min":{"1min":0.91300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.089,"5min":2.569,"15min":2.6070000000000002},"last":1.1180000000000001}]}]},{"osd":2,"up_from":14,"seq":60129543336,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.93200000000000005,"15min":0.94999999999999996},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.47599999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.7090000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.0069999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5580000000000001,"5min":1.847,"15min":1.847},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.125,"15min":1.2210000000000001},"min":{"1min":0.67100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.415,"5min":1.784,"15min":1.9530000000000001},"last":1.133}]},{"osd":3,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.05,"15min":1.0980000000000001},"min":{"1min":0.56100000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3759999999999999,"5min":1.9179999999999999,"15min":2.1299999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.252,"15min":1.3460000000000001},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.1480000000000001,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":1.5}]},{"osd":4,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.4890000000000001},"min":{"1min":0.78200000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.298,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.375},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.4790000000000001},"min":{"1min":0.88900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.323}]},{"osd":5,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.51,"15min":1.548},"min":{"1min":0.89000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.427},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.462,"15min":1.4650000000000001},"min":{"1min":0.80600000000000005,"5min":0.627,"15min":0.627},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.5700000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.524,"15min":1.52},"min":{"1min":0.91300000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.454},{"interface":"front","average":{"1min":1.524,"5min":1.5580000000000001,"15min":1.605},"min":{"1min":0.77900000000000003,"5min":0.752,"15min":0.752},"max":{"1min":2.52,"5min":2.52,"15min":2.5569999999999999},"last":1.1990000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.641,"15min":1.6599999999999999},"min":{"1min":0.996,"5min":0.754,"15min":0.754},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":1.524},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.613,"15min":1.637},"min":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.2669999999999999}]}]},{"osd":1,"up_from":14,"seq":60129543336,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":1.319},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":0.70599999999999996}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.0129999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.96399999999999997}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":1.371},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.494}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.603},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.405}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":1.5649999999999999}]}]},{"osd":0,"up_from":14,"seq":60129543337,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61732,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310108,"statfs":{"total":96636764160,"available":96573550592,"internally_reserved":0,"allocated":4284416,"data_stored":2405554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91600000000000004,"15min":0.93999999999999995},"min":{"1min":0.46200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.98999999999999999,"15min":1.1140000000000001},"min":{"1min":0.60599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9299999999999999},"last":0.81200000000000006}]},{"osd":2,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0609999999999999,"15min":1.149},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.768,"5min":1.768,"15min":2.0569999999999999},"last":1.181},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0880000000000001,"15min":1.1890000000000001},"min":{"1min":0.72299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":1.3260000000000001,"5min":1.78,"15min":1.893},"last":0.93400000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0269999999999999,"15min":1.125},"min":{"1min":0.66700000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2290000000000001,"5min":1.7869999999999999,"15min":1.8520000000000001},"last":1.008},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2350000000000001,"15min":1.341},"min":{"1min":0.79500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5880000000000001,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.052}]},{"osd":4,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.343},"min":{"1min":0.95399999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5900000000000001,"5min":1.923,"15min":2.0070000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3979999999999999,"15min":1.4910000000000001},"min":{"1min":0.996,"5min":0.748,"15min":0.748},"max":{"1min":2.0169999999999999,"5min":2.1070000000000002,"15min":2.601},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.4810000000000001},"min":{"1min":0.80100000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.081,"5min":2.089,"15min":2.3279999999999998},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.3720000000000001,"15min":1.45},"min":{"1min":0.89700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.754,"5min":2.0169999999999999,"15min":2.2829999999999999},"last":1.226}]},{"osd":6,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5049999999999999,"15min":1.5880000000000001},"min":{"1min":0.94099999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0990000000000002,"5min":2.2730000000000001,"15min":2.669},"last":1.395},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.494,"15min":1.5720000000000001},"min":{"1min":0.86099999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.893,"5min":2.327,"15min":2.3769999999999998},"last":1.7470000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5840000000000001,"15min":1.677},"min":{"1min":0.93899999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.5270000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.5249999999999999,"15min":1.6120000000000001},"min":{"1min":1.119,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.893,"5min":2.3679999999999999,"15min":2.464},"last":1.3380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:14:01.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T14:14:01.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.612+0000 7f504f421640 1 Processor -- start 2023-10-16T14:14:01.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.612+0000 7f504f421640 1 -- start start 2023-10-16T14:14:01.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504f421640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f504806c040 0x7f504806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504f421640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f50480ad970 0x7f504815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504f421640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f504815f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504f421640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504806c960 con 0x7f504806c040 2023-10-16T14:14:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504f421640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504815b9d0 con 0x7f50480ad970 2023-10-16T14:14:01.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504f421640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504815bcb0 con 0x7f504815ebc0 2023-10-16T14:14:01.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504e41f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f504806c040 0x7f504806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504e41f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f504806c040 0x7f504806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47970/0 (socket says 172.21.15.60:47970) 2023-10-16T14:14:01.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.614+0000 7f504e41f640 1 -- 172.21.15.60:0/2884081971 learned_addr learned my addr 172.21.15.60:0/2884081971 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:01.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504ec20640 1 --2- 172.21.15.60:0/2884081971 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f504815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504dc1e640 1 --2- 172.21.15.60:0/2884081971 >> v2:172.21.15.144:3300/0 conn(0x7f50480ad970 0x7f504815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504e41f640 1 -- 172.21.15.60:0/2884081971 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 msgr2=0x7f504815f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:01.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504e41f640 1 --2- 172.21.15.60:0/2884081971 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f504815f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504e41f640 1 -- 172.21.15.60:0/2884081971 >> v2:172.21.15.144:3300/0 conn(0x7f50480ad970 msgr2=0x7f504815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:01.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504e41f640 1 --2- 172.21.15.60:0/2884081971 >> v2:172.21.15.144:3300/0 conn(0x7f50480ad970 0x7f504815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.615+0000 7f504e41f640 1 -- 172.21.15.60:0/2884081971 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504815bf90 con 0x7f504806c040 2023-10-16T14:14:01.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.616+0000 7f504e41f640 1 --2- 172.21.15.60:0/2884081971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f504806c420 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f50480cb060 tx=0x7f504402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fcaef946970b90 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:01.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.616+0000 7f504d41d640 1 -- 172.21.15.60:0/2884081971 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044031860 con 0x7f504806c040 2023-10-16T14:14:01.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.616+0000 7f504d41d640 1 -- 172.21.15.60:0/2884081971 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50440319c0 con 0x7f504806c040 2023-10-16T14:14:01.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.616+0000 7f504d41d640 1 -- 172.21.15.60:0/2884081971 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044031d50 con 0x7f504806c040 2023-10-16T14:14:01.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 -- 172.21.15.60:0/2884081971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 msgr2=0x7f504806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 --2- 172.21.15.60:0/2884081971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f504806c420 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f50480cb060 tx=0x7f504402b900 comp rx=0 tx=0).stop 2023-10-16T14:14:01.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 -- 172.21.15.60:0/2884081971 shutdown_connections 2023-10-16T14:14:01.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 --2- 172.21.15.60:0/2884081971 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f504815f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 --2- 172.21.15.60:0/2884081971 >> v2:172.21.15.144:3300/0 conn(0x7f50480ad970 0x7f504815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 --2- 172.21.15.60:0/2884081971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f504806c420 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 -- 172.21.15.60:0/2884081971 >> 172.21.15.60:0/2884081971 conn(0x7f50480ce6a0 msgr2=0x7f50480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:01.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 -- 172.21.15.60:0/2884081971 shutdown_connections 2023-10-16T14:14:01.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.617+0000 7f504f421640 1 -- 172.21.15.60:0/2884081971 wait complete. 2023-10-16T14:14:01.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.619+0000 7f504f421640 1 Processor -- start 2023-10-16T14:14:01.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.619+0000 7f504f421640 1 -- start start 2023-10-16T14:14:01.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504f421640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f50480ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504f421640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50480ad970 0x7f50480c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504f421640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f50480ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504f421640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cadb0 con 0x7f504806c040 2023-10-16T14:14:01.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504f421640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480aec80 con 0x7f50480ad970 2023-10-16T14:14:01.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504f421640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50480c7ef0 con 0x7f504815ebc0 2023-10-16T14:14:01.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.620+0000 7f504e41f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f50480ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f50480ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47974/0 (socket says 172.21.15.60:47974) 2023-10-16T14:14:01.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 -- 172.21.15.60:0/3580983770 learned_addr learned my addr 172.21.15.60:0/3580983770 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:01.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504ec20640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f50480ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504dc1e640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50480ad970 0x7f50480c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 -- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 msgr2=0x7f50480ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f50480ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 -- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50480ad970 msgr2=0x7f50480c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50480ad970 0x7f50480c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.632 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 -- 172.21.15.60:0/3580983770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440094d0 con 0x7f504806c040 2023-10-16T14:14:01.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.621+0000 7f504e41f640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f50480ca870 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f50440097c0 tx=0x7f50440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:01.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.622+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504403ac60 con 0x7f504806c040 2023-10-16T14:14:01.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.622+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504403adc0 con 0x7f504806c040 2023-10-16T14:14:01.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.622+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044044610 con 0x7f504806c040 2023-10-16T14:14:01.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.622+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50480b03a0 con 0x7f504806c040 2023-10-16T14:14:01.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.622+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50480c9440 con 0x7f504806c040 2023-10-16T14:14:01.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.623+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504806b330 con 0x7f504806c040 2023-10-16T14:14:01.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.624+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f504404d450 con 0x7f504806c040 2023-10-16T14:14:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.625+0000 7f503effd640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f502c06afe0 0x7f502c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.625+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f50440abd10 con 0x7f504806c040 2023-10-16T14:14:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.626+0000 7f504dc1e640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f502c06afe0 0x7f502c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.627+0000 7f504dc1e640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f502c06afe0 0x7f502c06d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50480c68a0 tx=0x7f5038008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:01.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.629+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5044076d00 con 0x7f504806c040 2023-10-16T14:14:01.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.828+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f504804e9b0 con 0x7f504806c040 2023-10-16T14:14:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.830+0000 7f503effd640 1 -- 172.21.15.60:0/3580983770 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v292) v1 ==== 74+0+14189 (secure 0 0 0) 0x7f5044035070 con 0x7f504806c040 2023-10-16T14:14:01.834 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:01.834 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":292,"fsid":"d49fcbeb-3f16-4e9a-bc57-c0252aab0f68","created":"2023-10-16T12:33:04.597099+0000","modified":"2023-10-16T14:13:56.317948+0000","last_up_change":"2023-10-16T12:33:16.511383+0000","last_in_change":"2023-10-16T12:33:08.137168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:33:17.807385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95d94d30-9650-41d5-b171-abf1ef7481ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6825","nonce":132166300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6827","nonce":132166300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6831","nonce":132166300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":132166300},{"type":"v1","addr":"172.21.15.60:6829","nonce":132166300}]},"public_addr":"172.21.15.60:6825/132166300","cluster_addr":"172.21.15.60:6827/132166300","heartbeat_back_addr":"172.21.15.60:6831/132166300","heartbeat_front_addr":"172.21.15.60:6829/132166300","state":["exists","up"]},{"osd":1,"uuid":"573b73e3-922c-4699-a735-2278cc9fa140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6815","nonce":4168261604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6819","nonce":4168261604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6824","nonce":4168261604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":4168261604},{"type":"v1","addr":"172.21.15.60:6821","nonce":4168261604}]},"public_addr":"172.21.15.60:6815/4168261604","cluster_addr":"172.21.15.60:6819/4168261604","heartbeat_back_addr":"172.21.15.60:6824/4168261604","heartbeat_front_addr":"172.21.15.60:6821/4168261604","state":["exists","up"]},{"osd":2,"uuid":"67f1d1ee-c766-4cf0-a83d-60d628c75c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6801","nonce":1201910913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6803","nonce":1201910913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6807","nonce":1201910913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1201910913},{"type":"v1","addr":"172.21.15.60:6805","nonce":1201910913}]},"public_addr":"172.21.15.60:6801/1201910913","cluster_addr":"172.21.15.60:6803/1201910913","heartbeat_back_addr":"172.21.15.60:6807/1201910913","heartbeat_front_addr":"172.21.15.60:6805/1201910913","state":["exists","up"]},{"osd":3,"uuid":"b4006289-df86-422f-889e-1221600060e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6809","nonce":302373198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6811","nonce":302373198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6817","nonce":302373198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":302373198},{"type":"v1","addr":"172.21.15.60:6814","nonce":302373198}]},"public_addr":"172.21.15.60:6809/302373198","cluster_addr":"172.21.15.60:6811/302373198","heartbeat_back_addr":"172.21.15.60:6817/302373198","heartbeat_front_addr":"172.21.15.60:6814/302373198","state":["exists","up"]},{"osd":4,"uuid":"e11a5f19-1b14-45dd-8749-b082b4ddb0c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6801","nonce":2038056415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6803","nonce":2038056415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6807","nonce":2038056415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2038056415},{"type":"v1","addr":"172.21.15.144:6805","nonce":2038056415}]},"public_addr":"172.21.15.144:6801/2038056415","cluster_addr":"172.21.15.144:6803/2038056415","heartbeat_back_addr":"172.21.15.144:6807/2038056415","heartbeat_front_addr":"172.21.15.144:6805/2038056415","state":["exists","up"]},{"osd":5,"uuid":"e19e3b6a-c82e-47b4-bbc1-2542f699dd39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6815","nonce":1961528470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6819","nonce":1961528470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6828","nonce":1961528470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":1961528470},{"type":"v1","addr":"172.21.15.144:6824","nonce":1961528470}]},"public_addr":"172.21.15.144:6815/1961528470","cluster_addr":"172.21.15.144:6819/1961528470","heartbeat_back_addr":"172.21.15.144:6828/1961528470","heartbeat_front_addr":"172.21.15.144:6824/1961528470","state":["exists","up"]},{"osd":6,"uuid":"b23bca8c-e831-4949-b3da-3e333307e029","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6818","nonce":3196639449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6823","nonce":3196639449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6831","nonce":3196639449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6825","nonce":3196639449},{"type":"v1","addr":"172.21.15.144:6827","nonce":3196639449}]},"public_addr":"172.21.15.144:6818/3196639449","cluster_addr":"172.21.15.144:6823/3196639449","heartbeat_back_addr":"172.21.15.144:6831/3196639449","heartbeat_front_addr":"172.21.15.144:6827/3196639449","state":["exists","up"]},{"osd":7,"uuid":"4062aa4b-4de1-479c-ad22-40a6624130b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":285,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6809","nonce":3315484258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6811","nonce":3315484258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6829","nonce":3315484258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3315484258},{"type":"v1","addr":"172.21.15.144:6816","nonce":3315484258}]},"public_addr":"172.21.15.144:6809/3315484258","cluster_addr":"172.21.15.144:6811/3315484258","heartbeat_back_addr":"172.21.15.144:6829/3315484258","heartbeat_front_addr":"172.21.15.144:6816/3315484258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.482746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.167448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.504341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.700997+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.477007+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.827939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:33:13.483487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.144:6833/1784354286":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/758300120":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/1291769788":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/1290683266":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3884644447":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3738039093":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/579488795":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3615094779":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/1408575976":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/277289582":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/2718528188":"2023-10-17T14:13:38.050732+0000","172.21.15.144:0/4157805917":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/2109842343":"2023-10-17T14:13:38.050732+0000","172.21.15.60:6834/2549968691":"2023-10-17T14:13:53.301342+0000","172.21.15.144:0/306505374":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6833/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.144:6832/3623959149":"2023-10-17T14:13:38.050732+0000","172.21.15.60:6835/2549968691":"2023-10-17T14:13:53.301342+0000","172.21.15.144:6832/1784354286":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3309380401":"2023-10-17T14:13:49.056900+0000","172.21.15.144:0/3276093492":"2023-10-17T14:13:38.050732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T14:14:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f502c06afe0 msgr2=0x7f502c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f502c06afe0 0x7f502c06d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50480c68a0 tx=0x7f5038008040 comp rx=0 tx=0).stop 2023-10-16T14:14:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 msgr2=0x7f50480ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f50480ca870 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f50440097c0 tx=0x7f50440046c0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 shutdown_connections 2023-10-16T14:14:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f502c06afe0 0x7f502c06d4a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f504815ebc0 0x7f50480ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f50480ad970 0x7f50480c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 --2- 172.21.15.60:0/3580983770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f504806c040 0x7f50480ca870 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 >> 172.21.15.60:0/3580983770 conn(0x7f50480ce6a0 msgr2=0x7f5048159570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 shutdown_connections 2023-10-16T14:14:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:01.833+0000 7f504f421640 1 -- 172.21.15.60:0/3580983770 wait complete. 2023-10-16T14:14:02.856 INFO:tasks.ceph:Scrubbing osd.0 2023-10-16T14:14:02.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:03.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.091+0000 7f4ff5f33640 1 Processor -- start 2023-10-16T14:14:03.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.091+0000 7f4ff5f33640 1 -- start start 2023-10-16T14:14:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.092+0000 7f4ff5f33640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4ff0159da0 0x7f4ff00c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.092+0000 7f4ff5f33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 0x7f4ff006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4ff5f33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff015dfb0 con 0x7f4ff01599c0 2023-10-16T14:14:03.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4ff5f33640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff015e290 con 0x7f4ff0159da0 2023-10-16T14:14:03.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4ff5f33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff015e570 con 0x7f4ff00cce20 2023-10-16T14:14:03.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4fef7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ff01599c0 0x7f4ff015d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53420/0 (socket says 172.21.15.60:53420) 2023-10-16T14:14:03.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4fef7fe640 1 -- 172.21.15.60:0/3602958203 learned_addr learned my addr 172.21.15.60:0/3602958203 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:03.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4feffff640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 0x7f4ff006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3602958203 >> v2:172.21.15.144:3300/0 conn(0x7f4ff0159da0 0x7f4ff00c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567097053 0 0) 0x7f4ff015dfb0 con 0x7f4ff01599c0 2023-10-16T14:14:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.093+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003520 con 0x7f4ff01599c0 2023-10-16T14:14:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282158322 0 0) 0x7f4fdc003520 con 0x7f4ff01599c0 2023-10-16T14:14:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 msgr2=0x7f4ff006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 0x7f4ff006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 >> v2:172.21.15.144:3300/0 conn(0x7f4ff0159da0 msgr2=0x7f4ff00c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3602958203 >> v2:172.21.15.144:3300/0 conn(0x7f4ff0159da0 0x7f4ff00c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff006fda0 con 0x7f4ff01599c0 2023-10-16T14:14:03.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f4fe0001690 con 0x7f4ff01599c0 2023-10-16T14:14:03.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 >> v1:172.21.15.60:6789/0 conn(0x7f4ff01599c0 legacy=0x7f4ff015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 0x7f4fdc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fdc003f60 0x7f4fdc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc004d00 0x7f4fdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff015e290 con 0x7f4fdc003f60 2023-10-16T14:14:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff015e570 con 0x7f4fdc004d00 2023-10-16T14:14:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00b9b0 con 0x7f4ff00cce20 2023-10-16T14:14:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.094+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fdc003f60 0x7f4fdc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4fef7fe640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 0x7f4fdc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4feffff640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc004d00 0x7f4fdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4fe7fff640 1 -- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 msgr2=0x7f4fdc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff00cce20 0x7f4fdc0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4fe7fff640 1 -- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc004d00 msgr2=0x7f4fdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc004d00 0x7f4fdc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.095+0000 7f4fe7fff640 1 -- 172.21.15.60:0/3602958203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc003520 con 0x7f4fdc003f60 2023-10-16T14:14:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.096+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fdc003f60 0x7f4fdc0047c0 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009ac0 tx=0x7f4fdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b87c5c50f9aa48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.096+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd802c710 con 0x7f4fdc003f60 2023-10-16T14:14:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.096+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd802c870 con 0x7f4fdc003f60 2023-10-16T14:14:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.096+0000 7f4feeffd640 1 -- 172.21.15.60:0/3602958203 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd802cc00 con 0x7f4fdc003f60 2023-10-16T14:14:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3602958203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fdc003f60 msgr2=0x7f4fdc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4ff5f33640 1 --2- 172.21.15.60:0/3602958203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fdc003f60 0x7f4fdc0047c0 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009ac0 tx=0x7f4fdc0057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4feffff640 1 -- 172.21.15.60:0/3602958203 reap_dead start 2023-10-16T14:14:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3602958203 shutdown_connections 2023-10-16T14:14:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3602958203 >> 172.21.15.60:0/3602958203 conn(0x7f4ff00cb360 msgr2=0x7f4ff00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3602958203 shutdown_connections 2023-10-16T14:14:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.097+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3602958203 wait complete. 2023-10-16T14:14:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.099+0000 7f4ff5f33640 1 Processor -- start 2023-10-16T14:14:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.100+0000 7f4ff5f33640 1 -- start start 2023-10-16T14:14:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4ff5f33640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc003f60 0x7f4ff0163e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4ff5f33640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff0164380 0x7f4ff00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4ff5f33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00c1d30 0x7f4ff00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4ff5f33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0164b70 con 0x7f4ff00c1d30 2023-10-16T14:14:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4ff5f33640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0143720 con 0x7f4fdc003f60 2023-10-16T14:14:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4ff5f33640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0164890 con 0x7f4ff0164380 2023-10-16T14:14:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4feffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00c1d30 0x7f4ff00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4feffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00c1d30 0x7f4ff00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47990/0 (socket says 172.21.15.60:47990) 2023-10-16T14:14:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4feffff640 1 -- 172.21.15.60:0/3315318921 learned_addr learned my addr 172.21.15.60:0/3315318921 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff0164380 0x7f4ff00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.101+0000 7f4fef7fe640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc003f60 0x7f4ff0163e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4feffff640 1 -- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff0164380 msgr2=0x7f4ff00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4feffff640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f4ff0164380 0x7f4ff00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4feffff640 1 -- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc003f60 msgr2=0x7f4ff0163e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4feffff640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4fdc003f60 0x7f4ff0163e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4feffff640 1 -- 172.21.15.60:0/3315318921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0156db0 con 0x7f4ff00c1d30 2023-10-16T14:14:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4feffff640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00c1d30 0x7f4ff00acdd0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00c9840 tx=0x7f4fe800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8017400 con 0x7f4ff00c1d30 2023-10-16T14:14:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8011040 con 0x7f4ff00c1d30 2023-10-16T14:14:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff0157090 con 0x7f4ff00c1d30 2023-10-16T14:14:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80177c0 con 0x7f4ff00c1d30 2023-10-16T14:14:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.102+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff0143af0 con 0x7f4ff00c1d30 2023-10-16T14:14:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.103+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ff006fa50 con 0x7f4ff00c1d30 2023-10-16T14:14:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.105+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4fe8017920 con 0x7f4ff00c1d30 2023-10-16T14:14:03.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fecff9640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f4fd006af90 0x7f4fd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f4fe8089d10 con 0x7f4ff00c1d30 2023-10-16T14:14:03.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fecff9640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7f4fd00714d0 0x7f4fd00738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 --> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fd0073fa0 con 0x7f4fd00714d0 2023-10-16T14:14:03.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fe808d020 con 0x7f4ff00c1d30 2023-10-16T14:14:03.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7f4fd00714d0 0x7f4fd00738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.106+0000 7f4fef7fe640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f4fd006af90 0x7f4fd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.107+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7f4fd00714d0 0x7f4fd00738f0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.108+0000 7f4fef7fe640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f4fd006af90 0x7f4fd006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00011e0 tx=0x7f4fe0009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.108+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== osd.0 v2:172.21.15.60:6823/132166300 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f4fd0073fa0 con 0x7f4fd00714d0 2023-10-16T14:14:03.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.122+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 --> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4ff0156c10 con 0x7f4fd00714d0 2023-10-16T14:14:03.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4fecff9640 1 -- 172.21.15.60:0/3315318921 <== osd.0 v2:172.21.15.60:6823/132166300 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4ff0156c10 con 0x7f4fd00714d0 2023-10-16T14:14:03.128 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:03.129 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:03.129 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:03.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7f4fd00714d0 msgr2=0x7f4fd00738f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4ff5f33640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:6823/132166300,v1:172.21.15.60:6825/132166300] conn(0x7f4fd00714d0 0x7f4fd00738f0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f4fd006af90 msgr2=0x7f4fd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4ff5f33640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f4fd006af90 0x7f4fd006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00011e0 tx=0x7f4fe0009040 comp rx=0 tx=0).stop 2023-10-16T14:14:03.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00c1d30 msgr2=0x7f4ff00acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4ff5f33640 1 --2- 172.21.15.60:0/3315318921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff00c1d30 0x7f4ff00acdd0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00c9840 tx=0x7f4fe800f4a0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.123+0000 7f4feffff640 1 -- 172.21.15.60:0/3315318921 reap_dead start 2023-10-16T14:14:03.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.124+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 shutdown_connections 2023-10-16T14:14:03.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.124+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 >> 172.21.15.60:0/3315318921 conn(0x7f4ff00cb360 msgr2=0x7f4ff006b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:03.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.124+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 shutdown_connections 2023-10-16T14:14:03.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.124+0000 7f4ff5f33640 1 -- 172.21.15.60:0/3315318921 wait complete. 2023-10-16T14:14:03.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-16T14:14:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.366+0000 7f3f6bfff640 1 Processor -- start 2023-10-16T14:14:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.366+0000 7f3f6bfff640 1 -- start start 2023-10-16T14:14:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.367+0000 7f3f6bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3f6c0cded0 0x7f3f6c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6bfff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c15d150 con 0x7f3f6c0b8480 2023-10-16T14:14:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6bfff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c0d1180 con 0x7f3f6c15cd70 2023-10-16T14:14:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6bfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c0d1460 con 0x7f3f6c0cded0 2023-10-16T14:14:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6a7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3f6c0b8480 0x7f3f6c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53424/0 (socket says 172.21.15.60:53424) 2023-10-16T14:14:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6a7fc640 1 -- 172.21.15.60:0/755916435 learned_addr learned my addr 172.21.15.60:0/755916435 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6b7fe640 1 --2- 172.21.15.60:0/755916435 >> v2:172.21.15.60:3300/0 conn(0x7f3f6c0cded0 0x7f3f6c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.368+0000 7f3f6b7fe640 1 -- 172.21.15.60:0/755916435 >> v1:172.21.15.144:6789/0 conn(0x7f3f6c15cd70 legacy=0x7f3f6c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.369+0000 7f3f6b7fe640 1 -- 172.21.15.60:0/755916435 >> v1:172.21.15.60:6789/0 conn(0x7f3f6c0b8480 legacy=0x7f3f6c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.369+0000 7f3f6b7fe640 1 -- 172.21.15.60:0/755916435 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c0d1740 con 0x7f3f6c0cded0 2023-10-16T14:14:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.369+0000 7f3f69ffb640 1 -- 172.21.15.60:0/755916435 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517453574 0 0) 0x7f3f6c15d150 con 0x7f3f6c0b8480 2023-10-16T14:14:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.370+0000 7f3f6b7fe640 1 --2- 172.21.15.60:0/755916435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0cded0 0x7f3f6c0b5c90 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f3f640097a0 tx=0x7f3f6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46adde906727965e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f69ffb640 1 -- 172.21.15.60:0/755916435 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64031830 con 0x7f3f6c0cded0 2023-10-16T14:14:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f69ffb640 1 -- 172.21.15.60:0/755916435 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f64031990 con 0x7f3f6c0cded0 2023-10-16T14:14:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f69ffb640 1 -- 172.21.15.60:0/755916435 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64031d20 con 0x7f3f6c0cded0 2023-10-16T14:14:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f6bfff640 1 -- 172.21.15.60:0/755916435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0cded0 msgr2=0x7f3f6c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/755916435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0cded0 0x7f3f6c0b5c90 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f3f640097a0 tx=0x7f3f6402b8d0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f6bfff640 1 -- 172.21.15.60:0/755916435 shutdown_connections 2023-10-16T14:14:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.372+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/755916435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0cded0 0x7f3f6c0b5c90 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.373+0000 7f3f6bfff640 1 -- 172.21.15.60:0/755916435 >> 172.21.15.60:0/755916435 conn(0x7f3f6c0cf570 msgr2=0x7f3f6c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.373+0000 7f3f6bfff640 1 -- 172.21.15.60:0/755916435 shutdown_connections 2023-10-16T14:14:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.373+0000 7f3f6bfff640 1 -- 172.21.15.60:0/755916435 wait complete. 2023-10-16T14:14:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.374+0000 7f3f6bfff640 1 Processor -- start 2023-10-16T14:14:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.374+0000 7f3f6bfff640 1 -- start start 2023-10-16T14:14:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.375+0000 7f3f6bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 0x7f3f6c15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.375+0000 7f3f6bfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f6c0cded0 0x7f3f6c0b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f6c15cd70 0x7f3f6c0b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6bfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cc210 con 0x7f3f6c0b8480 2023-10-16T14:14:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6bfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cc4f0 con 0x7f3f6c15cd70 2023-10-16T14:14:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6bfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cc7d0 con 0x7f3f6c0cded0 2023-10-16T14:14:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 0x7f3f6c15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 0x7f3f6c15a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47996/0 (socket says 172.21.15.60:47996) 2023-10-16T14:14:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 -- 172.21.15.60:0/2381639583 learned_addr learned my addr 172.21.15.60:0/2381639583 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6a7fc640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f6c0cded0 0x7f3f6c0b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6b7fe640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f6c15cd70 0x7f3f6c0b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 -- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f6c0cded0 msgr2=0x7f3f6c0b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f6c0cded0 0x7f3f6c0b6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 -- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f6c15cd70 msgr2=0x7f3f6c0b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f6c15cd70 0x7f3f6c0b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6affd640 1 -- 172.21.15.60:0/2381639583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64009400 con 0x7f3f6c0b8480 2023-10-16T14:14:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.376+0000 7f3f6b7fe640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f6c15cd70 0x7f3f6c0b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.377+0000 7f3f6affd640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 0x7f3f6c15a550 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00ba50 tx=0x7f3f5c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.386 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.377+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c00cbe0 con 0x7f3f6c0b8480 2023-10-16T14:14:03.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.377+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c00cd80 con 0x7f3f6c0b8480 2023-10-16T14:14:03.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.377+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c014680 con 0x7f3f6c0b8480 2023-10-16T14:14:03.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.377+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c0ccab0 con 0x7f3f6c0b8480 2023-10-16T14:14:03.387 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.377+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c0cb680 con 0x7f3f6c0b8480 2023-10-16T14:14:03.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.378+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c0ccca0 con 0x7f3f6c0b8480 2023-10-16T14:14:03.388 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.380+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f5c0147e0 con 0x7f3f6c0b8480 2023-10-16T14:14:03.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.381+0000 7f3f537fe640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f3f5806af90 0x7f3f5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.381+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f3f5c089610 con 0x7f3f6c0b8480 2023-10-16T14:14:03.389 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.381+0000 7f3f6a7fc640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f3f5806af90 0x7f3f5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.382+0000 7f3f6a7fc640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f3f5806af90 0x7f3f5806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3f60007fd0 tx=0x7f3f60009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.390 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.385+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f5c054640 con 0x7f3f6c0b8480 2023-10-16T14:14:03.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.581+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f6c0cc660 con 0x7f3f5806af90 2023-10-16T14:14:03.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.583+0000 7f3f537fe640 1 -- 172.21.15.60:0/2381639583 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3f6c0cc660 con 0x7f3f5806af90 2023-10-16T14:14:03.586 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2023-10-16T14:14:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f3f5806af90 msgr2=0x7f3f5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f3f5806af90 0x7f3f5806d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3f60007fd0 tx=0x7f3f60009040 comp rx=0 tx=0).stop 2023-10-16T14:14:03.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 msgr2=0x7f3f6c15a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 0x7f3f6c15a550 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00ba50 tx=0x7f3f5c00bf10 comp rx=0 tx=0).stop 2023-10-16T14:14:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 shutdown_connections 2023-10-16T14:14:03.590 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f3f5806af90 0x7f3f5806d450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f3f6c0cded0 0x7f3f6c0b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f6c15cd70 0x7f3f6c0b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 --2- 172.21.15.60:0/2381639583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f6c0b8480 0x7f3f6c15a550 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 >> 172.21.15.60:0/2381639583 conn(0x7f3f6c0cf570 msgr2=0x7f3f6c0b5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 shutdown_connections 2023-10-16T14:14:03.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.585+0000 7f3f6bfff640 1 -- 172.21.15.60:0/2381639583 wait complete. 2023-10-16T14:14:03.609 INFO:tasks.ceph:Scrubbing osd.1 2023-10-16T14:14:03.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:03.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.841+0000 7fc5caf47640 1 Processor -- start 2023-10-16T14:14:03.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.841+0000 7fc5caf47640 1 -- start start 2023-10-16T14:14:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.842+0000 7fc5caf47640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc5c406c040 0x7fc5c406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5caf47640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c415f4a0 con 0x7fc5c40cda80 2023-10-16T14:14:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5caf47640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c406c960 con 0x7fc5c40ad970 2023-10-16T14:14:03.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5caf47640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c415b9d0 con 0x7fc5c406c040 2023-10-16T14:14:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5ca746640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc5c40cda80 0x7fc5c415ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53432/0 (socket says 172.21.15.60:53432) 2023-10-16T14:14:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5ca746640 1 -- 172.21.15.60:0/3674327956 learned_addr learned my addr 172.21.15.60:0/3674327956 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:03.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5c9f45640 1 --2- 172.21.15.60:0/3674327956 >> v2:172.21.15.144:3300/0 conn(0x7fc5c406c040 0x7fc5c406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.843+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780745187 0 0) 0x7fc5c415f4a0 con 0x7fc5c40cda80 2023-10-16T14:14:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003520 con 0x7fc5c40cda80 2023-10-16T14:14:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1838990321 0 0) 0x7fc5b0003520 con 0x7fc5c40cda80 2023-10-16T14:14:03.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 >> v2:172.21.15.144:3300/0 conn(0x7fc5c406c040 msgr2=0x7fc5c406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 --2- 172.21.15.60:0/3674327956 >> v2:172.21.15.144:3300/0 conn(0x7fc5c406c040 0x7fc5c406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 >> v1:172.21.15.144:6789/0 conn(0x7fc5c40ad970 legacy=0x7fc5c415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c415bcb0 con 0x7fc5c40cda80 2023-10-16T14:14:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fc5bc0019f0 con 0x7fc5c40cda80 2023-10-16T14:14:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 >> v1:172.21.15.60:6789/0 conn(0x7fc5c40cda80 legacy=0x7fc5c415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b00039c0 0x7fc5b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.852 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b0005180 0x7fc5b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c415b9d0 con 0x7fc5b00047c0 2023-10-16T14:14:03.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000c080 con 0x7fc5b00039c0 2023-10-16T14:14:03.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000c240 con 0x7fc5b0005180 2023-10-16T14:14:03.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.844+0000 7fc5c9744640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.845+0000 7fc5c9744640 1 -- 172.21.15.60:0/3674327956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b0005180 msgr2=0x7fc5b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.845+0000 7fc5c9744640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b0005180 0x7fc5b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.845+0000 7fc5c9744640 1 -- 172.21.15.60:0/3674327956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b00039c0 msgr2=0x7fc5b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:03.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.845+0000 7fc5c9744640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5b00039c0 0x7fc5b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.845+0000 7fc5c9744640 1 -- 172.21.15.60:0/3674327956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0003520 con 0x7fc5b00047c0 2023-10-16T14:14:03.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.845+0000 7fc5ca746640 1 -- 172.21.15.60:0/3674327956 reap_dead start 2023-10-16T14:14:03.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.846+0000 7fc5c9744640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4001720 tx=0x7fc5b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fbf87ba2ffa93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.846+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40052d0 con 0x7fc5b00047c0 2023-10-16T14:14:03.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.846+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b402e050 con 0x7fc5b00047c0 2023-10-16T14:14:03.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.846+0000 7fc5c8f43640 1 -- 172.21.15.60:0/3674327956 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4003830 con 0x7fc5b00047c0 2023-10-16T14:14:03.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 -- 172.21.15.60:0/3674327956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5b00047c0 msgr2=0x7fc5b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4001720 tx=0x7fc5b00057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 -- 172.21.15.60:0/3674327956 shutdown_connections 2023-10-16T14:14:03.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 --2- 172.21.15.60:0/3674327956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 -- 172.21.15.60:0/3674327956 >> 172.21.15.60:0/3674327956 conn(0x7fc5c40ce6a0 msgr2=0x7fc5c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 -- 172.21.15.60:0/3674327956 shutdown_connections 2023-10-16T14:14:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.847+0000 7fc5caf47640 1 -- 172.21.15.60:0/3674327956 wait complete. 2023-10-16T14:14:03.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.849+0000 7fc5caf47640 1 Processor -- start 2023-10-16T14:14:03.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.849+0000 7fc5caf47640 1 -- start start 2023-10-16T14:14:03.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.850+0000 7fc5caf47640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b00047c0 0x7fc5c41e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.850+0000 7fc5caf47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c4156340 0x7fc5c40b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.850+0000 7fc5caf47640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c41557f0 0x7fc5c4155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.850+0000 7fc5caf47640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c3ab0 con 0x7fc5c4156340 2023-10-16T14:14:03.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.850+0000 7fc5caf47640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c6ae0 con 0x7fc5c41557f0 2023-10-16T14:14:03.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.850+0000 7fc5caf47640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c41561b0 con 0x7fc5b00047c0 2023-10-16T14:14:03.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c4156340 0x7fc5c40b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c4156340 0x7fc5c40b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48022/0 (socket says 172.21.15.60:48022) 2023-10-16T14:14:03.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 -- 172.21.15.60:0/45476377 learned_addr learned my addr 172.21.15.60:0/45476377 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:03.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9f45640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b00047c0 0x7fc5c41e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5ca746640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c41557f0 0x7fc5c4155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 -- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b00047c0 msgr2=0x7fc5c41e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fc5b00047c0 0x7fc5c41e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 -- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c41557f0 msgr2=0x7fc5c4155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c41557f0 0x7fc5c4155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 -- 172.21.15.60:0/45476377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4156c10 con 0x7fc5c4156340 2023-10-16T14:14:03.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5ca746640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5c41557f0 0x7fc5c4155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.851+0000 7fc5c9744640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c4156340 0x7fc5c40b2be0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4034040 tx=0x7fc5b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.852+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40076c0 con 0x7fc5c4156340 2023-10-16T14:14:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.852+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b40069e0 con 0x7fc5c4156340 2023-10-16T14:14:03.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.852+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4006d90 con 0x7fc5c4156340 2023-10-16T14:14:03.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.852+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c40c2f30 con 0x7fc5c4156340 2023-10-16T14:14:03.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.852+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c40c34f0 con 0x7fc5c4156340 2023-10-16T14:14:03.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.853+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc5c40cc270 con 0x7fc5c4156340 2023-10-16T14:14:03.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.854+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc5b40035f0 con 0x7fc5c4156340 2023-10-16T14:14:03.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.855+0000 7fc5ba7fc640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fc5a806af90 0x7fc5a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.855+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fc5b40ab560 con 0x7fc5c4156340 2023-10-16T14:14:03.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.856+0000 7fc5ba7fc640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc5a80714d0 0x7fc5a80738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:03.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.856+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 --> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5a8073fa0 con 0x7fc5a80714d0 2023-10-16T14:14:03.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.856+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5b40adc00 con 0x7fc5c4156340 2023-10-16T14:14:03.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.856+0000 7fc5ca746640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc5a80714d0 0x7fc5a80738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.856+0000 7fc5c9f45640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fc5a806af90 0x7fc5a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:03.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.857+0000 7fc5ca746640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc5a80714d0 0x7fc5a80738f0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.857+0000 7fc5c9f45640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fc5a806af90 0x7fc5a806d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4063ab0 tx=0x7fc5c0008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:03.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.857+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== osd.1 v2:172.21.15.60:6813/4168261604 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fc5a8073fa0 con 0x7fc5a80714d0 2023-10-16T14:14:03.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.872+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 --> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc5c415bf90 con 0x7fc5a80714d0 2023-10-16T14:14:03.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.872+0000 7fc5ba7fc640 1 -- 172.21.15.60:0/45476377 <== osd.1 v2:172.21.15.60:6813/4168261604 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc5bc001e30 con 0x7fc5a80714d0 2023-10-16T14:14:03.876 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:03.877 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:03.877 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:03.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc5a80714d0 msgr2=0x7fc5a80738f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:6813/4168261604,v1:172.21.15.60:6815/4168261604] conn(0x7fc5a80714d0 0x7fc5a80738f0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:03.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fc5a806af90 msgr2=0x7fc5a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fc5a806af90 0x7fc5a806d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4063ab0 tx=0x7fc5c0008040 comp rx=0 tx=0).stop 2023-10-16T14:14:03.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c4156340 msgr2=0x7fc5c40b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:03.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 --2- 172.21.15.60:0/45476377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c4156340 0x7fc5c40b2be0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4034040 tx=0x7fc5b4001120 comp rx=0 tx=0).stop 2023-10-16T14:14:03.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5ca746640 1 -- 172.21.15.60:0/45476377 reap_dead start 2023-10-16T14:14:03.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 shutdown_connections 2023-10-16T14:14:03.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 >> 172.21.15.60:0/45476377 conn(0x7fc5c40ce6a0 msgr2=0x7fc5c4069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:03.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.873+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 shutdown_connections 2023-10-16T14:14:03.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:03.874+0000 7fc5caf47640 1 -- 172.21.15.60:0/45476377 wait complete. 2023-10-16T14:14:03.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-16T14:14:04.118 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.115+0000 7f5ef69a9640 1 Processor -- start 2023-10-16T14:14:04.119 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.115+0000 7f5ef69a9640 1 -- start start 2023-10-16T14:14:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.116+0000 7f5ef69a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5ef0159e90 0x7f5ef00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.116+0000 7f5ef69a9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ef006c140 0x7f5ef006c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef69a9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef006cad0 con 0x7f5ef0159ab0 2023-10-16T14:14:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef69a9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef015d890 con 0x7f5ef006c140 2023-10-16T14:14:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef69a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef0070360 con 0x7f5ef0159e90 2023-10-16T14:14:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5eeffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5ef0159ab0 0x7f5ef006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53434/0 (socket says 172.21.15.60:53434) 2023-10-16T14:14:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5eeffff640 1 -- 172.21.15.60:0/3096757227 learned_addr learned my addr 172.21.15.60:0/3096757227 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef4f46640 1 --2- 172.21.15.60:0/3096757227 >> v2:172.21.15.60:3300/0 conn(0x7f5ef006c140 0x7f5ef006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5eef7fe640 1 --2- 172.21.15.60:0/3096757227 >> v2:172.21.15.144:3300/0 conn(0x7f5ef0159e90 0x7f5ef00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5eeeffd640 1 -- 172.21.15.60:0/3096757227 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524713710 0 0) 0x7f5ee0001110 con 0x7f5ef0159ab0 2023-10-16T14:14:04.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5eeeffd640 1 -- 172.21.15.60:0/3096757227 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed8003520 con 0x7f5ef0159ab0 2023-10-16T14:14:04.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef4f46640 1 -- 172.21.15.60:0/3096757227 >> v2:172.21.15.144:3300/0 conn(0x7f5ef0159e90 msgr2=0x7f5ef00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:04.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef4f46640 1 --2- 172.21.15.60:0/3096757227 >> v2:172.21.15.144:3300/0 conn(0x7f5ef0159e90 0x7f5ef00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.117+0000 7f5ef4f46640 1 -- 172.21.15.60:0/3096757227 >> v1:172.21.15.60:6789/0 conn(0x7f5ef0159ab0 legacy=0x7f5ef006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.118+0000 7f5ef4f46640 1 -- 172.21.15.60:0/3096757227 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00cd1f0 con 0x7f5ef006c140 2023-10-16T14:14:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.119+0000 7f5ef4f46640 1 --2- 172.21.15.60:0/3096757227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef006c590 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f5ef006d900 tx=0x7f5ee802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8c783e8e231a73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.119+0000 7f5eeeffd640 1 -- 172.21.15.60:0/3096757227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee802cb60 con 0x7f5ef006c140 2023-10-16T14:14:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.119+0000 7f5eeeffd640 1 -- 172.21.15.60:0/3096757227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee802ccc0 con 0x7f5ef006c140 2023-10-16T14:14:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.119+0000 7f5eeeffd640 1 -- 172.21.15.60:0/3096757227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8034660 con 0x7f5ef006c140 2023-10-16T14:14:04.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 -- 172.21.15.60:0/3096757227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 msgr2=0x7f5ef006c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/3096757227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef006c590 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f5ef006d900 tx=0x7f5ee802bdb0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 -- 172.21.15.60:0/3096757227 shutdown_connections 2023-10-16T14:14:04.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/3096757227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef006c590 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/3096757227 >> v2:172.21.15.144:3300/0 conn(0x7f5ef0159e90 0x7f5ef00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 -- 172.21.15.60:0/3096757227 >> 172.21.15.60:0/3096757227 conn(0x7f5ef00ce9b0 msgr2=0x7f5ef00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:04.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 -- 172.21.15.60:0/3096757227 shutdown_connections 2023-10-16T14:14:04.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.120+0000 7f5ef69a9640 1 -- 172.21.15.60:0/3096757227 wait complete. 2023-10-16T14:14:04.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.122+0000 7f5ef69a9640 1 Processor -- start 2023-10-16T14:14:04.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.122+0000 7f5ef69a9640 1 -- start start 2023-10-16T14:14:04.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5ef69a9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef00c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5ef69a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef0159ab0 0x7f5ef01f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5ef69a9640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef0159e90 0x7f5ef0157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.133 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5ef69a9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef015a4d0 con 0x7f5ef006c140 2023-10-16T14:14:04.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5ef69a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef015a7b0 con 0x7f5ef0159ab0 2023-10-16T14:14:04.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5ef69a9640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef015aa90 con 0x7f5ef0159e90 2023-10-16T14:14:04.134 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5eeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef00c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5eeffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef00c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48038/0 (socket says 172.21.15.60:48038) 2023-10-16T14:14:04.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.123+0000 7f5eeffff640 1 -- 172.21.15.60:0/1575180799 learned_addr learned my addr 172.21.15.60:0/1575180799 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:04.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5ef4f46640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef0159e90 0x7f5ef0157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.135 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eef7fe640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef0159ab0 0x7f5ef01f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eeffff640 1 -- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef0159e90 msgr2=0x7f5ef0157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eeffff640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef0159e90 0x7f5ef0157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eeffff640 1 -- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef0159ab0 msgr2=0x7f5ef01f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.136 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eeffff640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef0159ab0 0x7f5ef01f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eeffff640 1 -- 172.21.15.60:0/1575180799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009a70 con 0x7f5ef006c140 2023-10-16T14:14:04.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eef7fe640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef0159ab0 0x7f5ef01f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:04.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eeffff640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef00c7db0 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c700 tx=0x7f5ee000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.137 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.124+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0005ce0 con 0x7f5ef006c140 2023-10-16T14:14:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.125+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0005e40 con 0x7f5ef006c140 2023-10-16T14:14:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.125+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0007290 con 0x7f5ef006c140 2023-10-16T14:14:04.138 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.125+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef015ad70 con 0x7f5ef006c140 2023-10-16T14:14:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.125+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef00c84d0 con 0x7f5ef006c140 2023-10-16T14:14:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.126+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef004e160 con 0x7f5ef006c140 2023-10-16T14:14:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.127+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ee00074b0 con 0x7f5ef006c140 2023-10-16T14:14:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.128+0000 7f5eecff9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f5edc06af90 0x7f5edc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.139 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.128+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f5ee0088aa0 con 0x7f5ef006c140 2023-10-16T14:14:04.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.128+0000 7f5eef7fe640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f5edc06af90 0x7f5edc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.140 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.130+0000 7f5eef7fe640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f5edc06af90 0x7f5edc06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5ef015c5b0 tx=0x7f5ee4008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.141 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.132+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ee00574b0 con 0x7f5ef006c140 2023-10-16T14:14:04.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.330+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef00cd7b0 con 0x7f5edc06af90 2023-10-16T14:14:04.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.331+0000 7f5eecff9640 1 -- 172.21.15.60:0/1575180799 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5ee4009e10 con 0x7f5edc06af90 2023-10-16T14:14:04.335 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2023-10-16T14:14:04.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f5edc06af90 msgr2=0x7f5edc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f5edc06af90 0x7f5edc06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5ef015c5b0 tx=0x7f5ee4008040 comp rx=0 tx=0).stop 2023-10-16T14:14:04.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 msgr2=0x7f5ef00c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef00c7db0 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000c700 tx=0x7f5ee000cb00 comp rx=0 tx=0).stop 2023-10-16T14:14:04.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 shutdown_connections 2023-10-16T14:14:04.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f5edc06af90 0x7f5edc06d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f5ef0159e90 0x7f5ef0157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef0159ab0 0x7f5ef01f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 --2- 172.21.15.60:0/1575180799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ef006c140 0x7f5ef00c7db0 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 >> 172.21.15.60:0/1575180799 conn(0x7f5ef00ce9b0 msgr2=0x7f5ef00aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:04.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 shutdown_connections 2023-10-16T14:14:04.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.334+0000 7f5ef69a9640 1 -- 172.21.15.60:0/1575180799 wait complete. 2023-10-16T14:14:04.358 INFO:tasks.ceph:Scrubbing osd.2 2023-10-16T14:14:04.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:04.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.581+0000 7fce97fff640 1 Processor -- start 2023-10-16T14:14:04.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.582+0000 7fce97fff640 1 -- start start 2023-10-16T14:14:04.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.584+0000 7fce97fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce980ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.584+0000 7fce97fff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980cd4a0 con 0x7fce980ad4d0 2023-10-16T14:14:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.584+0000 7fce97fff640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9815ac60 con 0x7fce98153740 2023-10-16T14:14:04.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.584+0000 7fce97fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce98070030 con 0x7fce980ccb00 2023-10-16T14:14:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.584+0000 7fce977fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fce980ad4d0 0x7fce980ad960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53438/0 (socket says 172.21.15.60:53438) 2023-10-16T14:14:04.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.584+0000 7fce977fe640 1 -- 172.21.15.60:0/2570282284 learned_addr learned my addr 172.21.15.60:0/2570282284 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce8ffff640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce980ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319758480 0 0) 0x7fce980cd4a0 con 0x7fce980ad4d0 2023-10-16T14:14:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84003520 con 0x7fce980ad4d0 2023-10-16T14:14:04.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 231728556 0 0) 0x7fce84003520 con 0x7fce980ad4d0 2023-10-16T14:14:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 msgr2=0x7fce980ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce980ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 >> v1:172.21.15.144:6789/0 conn(0x7fce98153740 legacy=0x7fce9815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98070310 con 0x7fce980ad4d0 2023-10-16T14:14:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fce900019f0 con 0x7fce980ad4d0 2023-10-16T14:14:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 >> v1:172.21.15.60:6789/0 conn(0x7fce980ad4d0 legacy=0x7fce980ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce840039a0 0x7fce84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce98070030 con 0x7fce840047c0 2023-10-16T14:14:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8400bab0 con 0x7fce840039a0 2023-10-16T14:14:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.585+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8400bc70 con 0x7fce980ccb00 2023-10-16T14:14:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce8ffff640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce8ffff640 1 -- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 msgr2=0x7fce840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce96ffd640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce840039a0 0x7fce84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce977fe640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce8ffff640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce8ffff640 1 -- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce840039a0 msgr2=0x7fce84003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce8ffff640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce840039a0 0x7fce84003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce8ffff640 1 -- 172.21.15.60:0/2570282284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84003520 con 0x7fce840047c0 2023-10-16T14:14:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce977fe640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980ccb00 0x7fce840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.586+0000 7fce96ffd640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce840039a0 0x7fce84003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.587+0000 7fce8ffff640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce84004c40 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7fce8000aac0 tx=0x7fce840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ceec2902a95a59b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.587+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8002cdf0 con 0x7fce840047c0 2023-10-16T14:14:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce80034440 con 0x7fce840047c0 2023-10-16T14:14:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce967fc640 1 -- 172.21.15.60:0/2570282284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800347d0 con 0x7fce840047c0 2023-10-16T14:14:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce97fff640 1 -- 172.21.15.60:0/2570282284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 msgr2=0x7fce84004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce97fff640 1 --2- 172.21.15.60:0/2570282284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce84004c40 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7fce8000aac0 tx=0x7fce840057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce977fe640 1 -- 172.21.15.60:0/2570282284 reap_dead start 2023-10-16T14:14:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce97fff640 1 -- 172.21.15.60:0/2570282284 shutdown_connections 2023-10-16T14:14:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce97fff640 1 -- 172.21.15.60:0/2570282284 >> 172.21.15.60:0/2570282284 conn(0x7fce980ce6a0 msgr2=0x7fce980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce97fff640 1 -- 172.21.15.60:0/2570282284 shutdown_connections 2023-10-16T14:14:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.588+0000 7fce97fff640 1 -- 172.21.15.60:0/2570282284 wait complete. 2023-10-16T14:14:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.590+0000 7fce97fff640 1 Processor -- start 2023-10-16T14:14:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.591+0000 7fce97fff640 1 -- start start 2023-10-16T14:14:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce97fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce980c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce97fff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980b12c0 0x7fce98152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce97fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce980b3ed0 0x7fce981583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce97fff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b3a40 con 0x7fce840047c0 2023-10-16T14:14:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce97fff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b1740 con 0x7fce980b3ed0 2023-10-16T14:14:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce97fff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b2e30 con 0x7fce980b12c0 2023-10-16T14:14:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce980c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce980c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48042/0 (socket says 172.21.15.60:48042) 2023-10-16T14:14:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 -- 172.21.15.60:0/3769974950 learned_addr learned my addr 172.21.15.60:0/3769974950 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce8ffff640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980b12c0 0x7fce98152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce977fe640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce980b3ed0 0x7fce981583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 -- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980b12c0 msgr2=0x7fce98152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fce980b12c0 0x7fce98152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 -- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce980b3ed0 msgr2=0x7fce981583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce980b3ed0 0x7fce981583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.592+0000 7fce96ffd640 1 -- 172.21.15.60:0/3769974950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980b3110 con 0x7fce840047c0 2023-10-16T14:14:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.593+0000 7fce96ffd640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce980c4b80 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7fce8800d7d0 tx=0x7fce8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.593+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88017070 con 0x7fce840047c0 2023-10-16T14:14:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.593+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce88005070 con 0x7fce840047c0 2023-10-16T14:14:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.593+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88005440 con 0x7fce840047c0 2023-10-16T14:14:04.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.593+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce980b33f0 con 0x7fce840047c0 2023-10-16T14:14:04.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.593+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce980b4350 con 0x7fce840047c0 2023-10-16T14:14:04.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.594+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fce980708d0 con 0x7fce840047c0 2023-10-16T14:14:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.595+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fce88007030 con 0x7fce840047c0 2023-10-16T14:14:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.596+0000 7fce8f7fe640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fce7806afc0 0x7fce7806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.596+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fce88089210 con 0x7fce840047c0 2023-10-16T14:14:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.597+0000 7fce8f7fe640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7fce78071500 0x7fce78073920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.597+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 --> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce78073fd0 con 0x7fce78071500 2023-10-16T14:14:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.597+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce8808b7f0 con 0x7fce840047c0 2023-10-16T14:14:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.597+0000 7fce977fe640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7fce78071500 0x7fce78073920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.597+0000 7fce8ffff640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fce7806afc0 0x7fce7806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.598+0000 7fce977fe640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7fce78071500 0x7fce78073920 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.598+0000 7fce8ffff640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fce7806afc0 0x7fce7806d480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fce80036000 tx=0x7fce80030000 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.598+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== osd.2 v2:172.21.15.60:6800/1201910913 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fce78073fd0 con 0x7fce78071500 2023-10-16T14:14:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.614+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 --> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fce980705f0 con 0x7fce78071500 2023-10-16T14:14:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.614+0000 7fce8f7fe640 1 -- 172.21.15.60:0/3769974950 <== osd.2 v2:172.21.15.60:6800/1201910913 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fce9000d040 con 0x7fce78071500 2023-10-16T14:14:04.619 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:04.619 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:04.619 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:04.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7fce78071500 msgr2=0x7fce78073920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:6800/1201910913,v1:172.21.15.60:6801/1201910913] conn(0x7fce78071500 0x7fce78073920 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fce7806afc0 msgr2=0x7fce7806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fce7806afc0 0x7fce7806d480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fce80036000 tx=0x7fce80030000 comp rx=0 tx=0).stop 2023-10-16T14:14:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 msgr2=0x7fce980c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 --2- 172.21.15.60:0/3769974950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce840047c0 0x7fce980c4b80 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7fce8800d7d0 tx=0x7fce8800dc90 comp rx=0 tx=0).stop 2023-10-16T14:14:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce977fe640 1 -- 172.21.15.60:0/3769974950 reap_dead start 2023-10-16T14:14:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 shutdown_connections 2023-10-16T14:14:04.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.615+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 >> 172.21.15.60:0/3769974950 conn(0x7fce980ce6a0 msgr2=0x7fce980af700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:04.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.616+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 shutdown_connections 2023-10-16T14:14:04.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.616+0000 7fce97fff640 1 -- 172.21.15.60:0/3769974950 wait complete. 2023-10-16T14:14:04.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-16T14:14:04.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.852+0000 7f0e8d4df640 1 Processor -- start 2023-10-16T14:14:04.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.852+0000 7f0e8d4df640 1 -- start start 2023-10-16T14:14:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.854+0000 7f0e8d4df640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e880ad970 0x7f0e880cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.854+0000 7f0e8d4df640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8815b9d0 con 0x7f0e880ccfb0 2023-10-16T14:14:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.854+0000 7f0e8d4df640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8815bcb0 con 0x7f0e8806c040 2023-10-16T14:14:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.854+0000 7f0e8d4df640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8815bf90 con 0x7f0e880ad970 2023-10-16T14:14:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.854+0000 7f0e877fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e880ad970 0x7f0e880cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.855+0000 7f0e877fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e880ad970 0x7f0e880cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48058/0 (socket says 172.21.15.60:48058) 2023-10-16T14:14:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.855+0000 7f0e877fe640 1 -- 172.21.15.60:0/3073891945 learned_addr learned my addr 172.21.15.60:0/3073891945 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.855+0000 7f0e8ccde640 1 --1- 172.21.15.60:0/3073891945 >> v1:172.21.15.60:6789/0 conn(0x7f0e880ccfb0 0x7f0e880cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53446/0 (socket says 172.21.15.60:53446) 2023-10-16T14:14:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.855+0000 7f0e877fe640 1 -- 172.21.15.60:0/3073891945 >> v1:172.21.15.144:6789/0 conn(0x7f0e8806c040 legacy=0x7f0e8806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.855+0000 7f0e877fe640 1 -- 172.21.15.60:0/3073891945 >> v1:172.21.15.60:6789/0 conn(0x7f0e880ccfb0 legacy=0x7f0e880cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.855+0000 7f0e877fe640 1 -- 172.21.15.60:0/3073891945 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e880cc270 con 0x7f0e880ad970 2023-10-16T14:14:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.856+0000 7f0e877fe640 1 --2- 172.21.15.60:0/3073891945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e880ad970 0x7f0e880cca70 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0098a0 tx=0x7f0e7c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9677d18acedf1911 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.856+0000 7f0e86ffd640 1 -- 172.21.15.60:0/3073891945 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c02f830 con 0x7f0e880ad970 2023-10-16T14:14:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e86ffd640 1 -- 172.21.15.60:0/3073891945 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c02f990 con 0x7f0e880ad970 2023-10-16T14:14:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e86ffd640 1 -- 172.21.15.60:0/3073891945 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c02fd20 con 0x7f0e880ad970 2023-10-16T14:14:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 -- 172.21.15.60:0/3073891945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e880ad970 msgr2=0x7f0e880cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/3073891945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e880ad970 0x7f0e880cca70 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0098a0 tx=0x7f0e7c02ba10 comp rx=0 tx=0).stop 2023-10-16T14:14:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 -- 172.21.15.60:0/3073891945 shutdown_connections 2023-10-16T14:14:04.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/3073891945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e880ad970 0x7f0e880cca70 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 -- 172.21.15.60:0/3073891945 >> 172.21.15.60:0/3073891945 conn(0x7f0e880ce6a0 msgr2=0x7f0e880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:04.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 -- 172.21.15.60:0/3073891945 shutdown_connections 2023-10-16T14:14:04.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.857+0000 7f0e8d4df640 1 -- 172.21.15.60:0/3073891945 wait complete. 2023-10-16T14:14:04.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.859+0000 7f0e8d4df640 1 Processor -- start 2023-10-16T14:14:04.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.859+0000 7f0e8d4df640 1 -- start start 2023-10-16T14:14:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.860+0000 7f0e8d4df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 0x7f0e881e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.860+0000 7f0e8d4df640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e880ad970 0x7f0e880b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.860+0000 7f0e8d4df640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e880ccfb0 0x7f0e880b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.860+0000 7f0e8d4df640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880af150 con 0x7f0e8806c040 2023-10-16T14:14:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.860+0000 7f0e8d4df640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880af430 con 0x7f0e880ad970 2023-10-16T14:14:04.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.860+0000 7f0e8d4df640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e88151c90 con 0x7f0e880ccfb0 2023-10-16T14:14:04.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 0x7f0e881e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 0x7f0e881e4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48062/0 (socket says 172.21.15.60:48062) 2023-10-16T14:14:04.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 -- 172.21.15.60:0/2063324677 learned_addr learned my addr 172.21.15.60:0/2063324677 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:04.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e877fe640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e880ad970 0x7f0e880b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e8ccde640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e880ccfb0 0x7f0e880b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 -- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e880ccfb0 msgr2=0x7f0e880b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e880ccfb0 0x7f0e880b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 -- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e880ad970 msgr2=0x7f0e880b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e880ad970 0x7f0e880b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e87fff640 1 -- 172.21.15.60:0/2063324677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0094d0 con 0x7f0e8806c040 2023-10-16T14:14:04.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e877fe640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e880ad970 0x7f0e880b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.861+0000 7f0e8ccde640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e880ccfb0 0x7f0e880b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.862+0000 7f0e87fff640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 0x7f0e881e4f40 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c400 tx=0x7f0e7800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.862+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e780052d0 con 0x7f0e8806c040 2023-10-16T14:14:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.862+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78005430 con 0x7f0e8806c040 2023-10-16T14:14:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.862+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e78003830 con 0x7f0e8806c040 2023-10-16T14:14:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.862+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e881e58b0 con 0x7f0e8806c040 2023-10-16T14:14:04.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.862+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e881e5e70 con 0x7f0e8806c040 2023-10-16T14:14:04.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.863+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8806b330 con 0x7f0e8806c040 2023-10-16T14:14:04.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.864+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0e78003990 con 0x7f0e8806c040 2023-10-16T14:14:04.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.865+0000 7f0e84ff9640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f0e7406afc0 0x7f0e7406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:04.876 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.865+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f0e78088c50 con 0x7f0e8806c040 2023-10-16T14:14:04.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.866+0000 7f0e877fe640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f0e7406afc0 0x7f0e7406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:04.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.867+0000 7f0e877fe640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f0e7406afc0 0x7f0e7406d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c039040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:04.877 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:04.870+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e7808e020 con 0x7f0e8806c040 2023-10-16T14:14:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.066+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e880cdb70 con 0x7f0e7406afc0 2023-10-16T14:14:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.067+0000 7f0e84ff9640 1 -- 172.21.15.60:0/2063324677 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0e7c02fb80 con 0x7f0e7406afc0 2023-10-16T14:14:05.071 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2023-10-16T14:14:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f0e7406afc0 msgr2=0x7f0e7406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f0e7406afc0 0x7f0e7406d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c039040 comp rx=0 tx=0).stop 2023-10-16T14:14:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 msgr2=0x7f0e881e4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 0x7f0e881e4f40 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c400 tx=0x7f0e7800c8c0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 shutdown_connections 2023-10-16T14:14:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f0e7406afc0 0x7f0e7406d480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f0e880ccfb0 0x7f0e880b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e880ad970 0x7f0e880b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 --2- 172.21.15.60:0/2063324677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e8806c040 0x7f0e881e4f40 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.069+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 >> 172.21.15.60:0/2063324677 conn(0x7f0e880ce6a0 msgr2=0x7f0e8806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.070+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 shutdown_connections 2023-10-16T14:14:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.070+0000 7f0e8d4df640 1 -- 172.21.15.60:0/2063324677 wait complete. 2023-10-16T14:14:05.092 INFO:tasks.ceph:Scrubbing osd.3 2023-10-16T14:14:05.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:05.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.328+0000 7f1bdbfff640 1 Processor -- start 2023-10-16T14:14:05.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.328+0000 7f1bdbfff640 1 -- start start 2023-10-16T14:14:05.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdbfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1bdc15cd70 0x7f1bdc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0b8480 0x7f1bdc0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bdc0d0be0 0x7f1bdc0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdbfff640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc15d150 con 0x7f1bdc0d0be0 2023-10-16T14:14:05.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdbfff640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc0d1180 con 0x7f1bdc15cd70 2023-10-16T14:14:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdbfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0d1460 con 0x7f1bdc0b8480 2023-10-16T14:14:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bdc0d0be0 0x7f1bdc0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bdc0d0be0 0x7f1bdc0cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48074/0 (socket says 172.21.15.60:48074) 2023-10-16T14:14:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bdb7fe640 1 -- 172.21.15.60:0/485084217 learned_addr learned my addr 172.21.15.60:0/485084217 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.330+0000 7f1bda7fc640 1 --2- 172.21.15.60:0/485084217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0b8480 0x7f1bdc0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdb7fe640 1 -- 172.21.15.60:0/485084217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0b8480 msgr2=0x7f1bdc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:05.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/485084217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0b8480 0x7f1bdc0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdb7fe640 1 -- 172.21.15.60:0/485084217 >> v2:172.21.15.144:3300/0 conn(0x7f1bdc15cd70 msgr2=0x7f1bdc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdaffd640 1 --2- 172.21.15.60:0/485084217 >> v2:172.21.15.144:3300/0 conn(0x7f1bdc15cd70 0x7f1bdc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/485084217 >> v2:172.21.15.144:3300/0 conn(0x7f1bdc15cd70 0x7f1bdc0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdb7fe640 1 -- 172.21.15.60:0/485084217 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc0d1740 con 0x7f1bdc0d0be0 2023-10-16T14:14:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.331+0000 7f1bdaffd640 1 --2- 172.21.15.60:0/485084217 >> v2:172.21.15.144:3300/0 conn(0x7f1bdc15cd70 0x7f1bdc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.332+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/485084217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0d0be0 0x7f1bdc0cdd00 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0b56a0 tx=0x7f1bd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb80167ebec7466 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.332+0000 7f1bd9ffb640 1 -- 172.21.15.60:0/485084217 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4031870 con 0x7f1bdc0d0be0 2023-10-16T14:14:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.332+0000 7f1bd9ffb640 1 -- 172.21.15.60:0/485084217 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd40319d0 con 0x7f1bdc0d0be0 2023-10-16T14:14:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.332+0000 7f1bd9ffb640 1 -- 172.21.15.60:0/485084217 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4031d60 con 0x7f1bdc0d0be0 2023-10-16T14:14:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 -- 172.21.15.60:0/485084217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0d0be0 msgr2=0x7f1bdc0cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/485084217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0d0be0 0x7f1bdc0cdd00 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0b56a0 tx=0x7f1bd402b910 comp rx=0 tx=0).stop 2023-10-16T14:14:05.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 -- 172.21.15.60:0/485084217 shutdown_connections 2023-10-16T14:14:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/485084217 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0d0be0 0x7f1bdc0cdd00 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/485084217 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0b8480 0x7f1bdc0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/485084217 >> v2:172.21.15.144:3300/0 conn(0x7f1bdc15cd70 0x7f1bdc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 -- 172.21.15.60:0/485084217 >> 172.21.15.60:0/485084217 conn(0x7f1bdc0cf570 msgr2=0x7f1bdc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 -- 172.21.15.60:0/485084217 shutdown_connections 2023-10-16T14:14:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.333+0000 7f1bdbfff640 1 -- 172.21.15.60:0/485084217 wait complete. 2023-10-16T14:14:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.335+0000 7f1bdbfff640 1 Processor -- start 2023-10-16T14:14:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.335+0000 7f1bdbfff640 1 -- start start 2023-10-16T14:14:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0b8480 0x7f1bdc0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdbfff640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0d0be0 0x7f1bdc0c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bdc15cd70 0x7f1bdc0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdbfff640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc145d70 con 0x7f1bdc0b8480 2023-10-16T14:14:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdbfff640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc146050 con 0x7f1bdc15cd70 2023-10-16T14:14:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdbfff640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc146330 con 0x7f1bdc0d0be0 2023-10-16T14:14:05.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0b8480 0x7f1bdc0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0b8480 0x7f1bdc0c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48076/0 (socket says 172.21.15.60:48076) 2023-10-16T14:14:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.336+0000 7f1bdaffd640 1 -- 172.21.15.60:0/2743544614 learned_addr learned my addr 172.21.15.60:0/2743544614 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bda7fc640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0d0be0 0x7f1bdc0c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bdc15cd70 0x7f1bdc0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdaffd640 1 -- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0d0be0 msgr2=0x7f1bdc0c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdaffd640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f1bdc0d0be0 0x7f1bdc0c6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdaffd640 1 -- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bdc15cd70 msgr2=0x7f1bdc0b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdaffd640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bdc15cd70 0x7f1bdc0b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdaffd640 1 -- 172.21.15.60:0/2743544614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40094d0 con 0x7f1bdc0b8480 2023-10-16T14:14:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bdc15cd70 0x7f1bdc0b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.337+0000 7f1bdaffd640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0b8480 0x7f1bdc0c6450 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00cf60 tx=0x7f1bcc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.338+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc00d840 con 0x7f1bdc0b8480 2023-10-16T14:14:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.338+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc00d9a0 con 0x7f1bdc0b8480 2023-10-16T14:14:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.338+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc00dd30 con 0x7f1bdc0b8480 2023-10-16T14:14:05.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.338+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bdc146610 con 0x7f1bdc0b8480 2023-10-16T14:14:05.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.338+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc145900 con 0x7f1bdc0b8480 2023-10-16T14:14:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.339+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1bdc0d1d00 con 0x7f1bdc0b8480 2023-10-16T14:14:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.340+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1bcc015440 con 0x7f1bdc0b8480 2023-10-16T14:14:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.341+0000 7f1bd37fe640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1bbc06af90 0x7f1bbc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.341+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f1bcc08a1d0 con 0x7f1bdc0b8480 2023-10-16T14:14:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.341+0000 7f1bd37fe640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f1bbc0714d0 0x7f1bbc0738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.341+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 --> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bbc073fa0 con 0x7f1bbc0714d0 2023-10-16T14:14:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.341+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bcc08c7b0 con 0x7f1bdc0b8480 2023-10-16T14:14:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.342+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f1bbc0714d0 0x7f1bbc0738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.342+0000 7f1bda7fc640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1bbc06af90 0x7f1bbc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.342+0000 7f1bdb7fe640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f1bbc0714d0 0x7f1bbc0738f0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.343+0000 7f1bda7fc640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1bbc06af90 0x7f1bbc06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4006fd0 tx=0x7f1bc4008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.343+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== osd.3 v2:172.21.15.60:6808/302373198 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1bbc073fa0 con 0x7f1bbc0714d0 2023-10-16T14:14:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.357+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 --> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1bdc0d1a20 con 0x7f1bbc0714d0 2023-10-16T14:14:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.358+0000 7f1bd37fe640 1 -- 172.21.15.60:0/2743544614 <== osd.3 v2:172.21.15.60:6808/302373198 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1bd4031bc0 con 0x7f1bbc0714d0 2023-10-16T14:14:05.362 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:05.362 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:05.363 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f1bbc0714d0 msgr2=0x7f1bbc0738f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:6808/302373198,v1:172.21.15.60:6809/302373198] conn(0x7f1bbc0714d0 0x7f1bbc0738f0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1bbc06af90 msgr2=0x7f1bbc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f1bbc06af90 0x7f1bbc06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4006fd0 tx=0x7f1bc4008040 comp rx=0 tx=0).stop 2023-10-16T14:14:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0b8480 msgr2=0x7f1bdc0c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 --2- 172.21.15.60:0/2743544614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bdc0b8480 0x7f1bdc0c6450 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00cf60 tx=0x7f1bcc00cf90 comp rx=0 tx=0).stop 2023-10-16T14:14:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdb7fe640 1 -- 172.21.15.60:0/2743544614 reap_dead start 2023-10-16T14:14:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 shutdown_connections 2023-10-16T14:14:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 >> 172.21.15.60:0/2743544614 conn(0x7f1bdc0cf570 msgr2=0x7f1bdc0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 shutdown_connections 2023-10-16T14:14:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.359+0000 7f1bdbfff640 1 -- 172.21.15.60:0/2743544614 wait complete. 2023-10-16T14:14:05.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-16T14:14:05.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.600+0000 7f9901128640 1 Processor -- start 2023-10-16T14:14:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.600+0000 7f9901128640 1 -- start start 2023-10-16T14:14:05.605 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.601+0000 7f9901128640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98fc06c040 0x7f98fc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f9901128640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f98fc0ad970 0x7f98fc15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f9901128640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc15ebc0 0x7f98fc15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f9901128640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc06c960 con 0x7f98fc06c040 2023-10-16T14:14:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f9901128640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc15b9d0 con 0x7f98fc0ad970 2023-10-16T14:14:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f9901128640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc15bcb0 con 0x7f98fc15ebc0 2023-10-16T14:14:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98fc06c040 0x7f98fc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98fc06c040 0x7f98fc06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48082/0 (socket says 172.21.15.60:48082) 2023-10-16T14:14:05.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 -- 172.21.15.60:0/1518058787 learned_addr learned my addr 172.21.15.60:0/1518058787 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98f37fe640 1 --2- 172.21.15.60:0/1518058787 >> v2:172.21.15.144:3300/0 conn(0x7f98fc0ad970 0x7f98fc15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 -- 172.21.15.60:0/1518058787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc15ebc0 msgr2=0x7f98fc15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 --2- 172.21.15.60:0/1518058787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc15ebc0 0x7f98fc15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 -- 172.21.15.60:0/1518058787 >> v2:172.21.15.144:3300/0 conn(0x7f98fc0ad970 msgr2=0x7f98fc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.602+0000 7f98fbfff640 1 --2- 172.21.15.60:0/1518058787 >> v2:172.21.15.144:3300/0 conn(0x7f98fc0ad970 0x7f98fc15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.603+0000 7f98fbfff640 1 -- 172.21.15.60:0/1518058787 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc15bf90 con 0x7f98fc06c040 2023-10-16T14:14:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.603+0000 7f98f37fe640 1 --2- 172.21.15.60:0/1518058787 >> v2:172.21.15.144:3300/0 conn(0x7f98fc0ad970 0x7f98fc15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.604+0000 7f98fbfff640 1 --2- 172.21.15.60:0/1518058787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc06c420 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0cb060 tx=0x7f98ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaaa2d4e6a630071 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.604+0000 7f98fb7fe640 1 -- 172.21.15.60:0/1518058787 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec02ca80 con 0x7f98fc06c040 2023-10-16T14:14:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.604+0000 7f98fb7fe640 1 -- 172.21.15.60:0/1518058787 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec02cbe0 con 0x7f98fc06c040 2023-10-16T14:14:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.604+0000 7f98fb7fe640 1 -- 172.21.15.60:0/1518058787 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec0343f0 con 0x7f98fc06c040 2023-10-16T14:14:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 -- 172.21.15.60:0/1518058787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 msgr2=0x7f98fc06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 --2- 172.21.15.60:0/1518058787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc06c420 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0cb060 tx=0x7f98ec02bee0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 -- 172.21.15.60:0/1518058787 shutdown_connections 2023-10-16T14:14:05.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 --2- 172.21.15.60:0/1518058787 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc15ebc0 0x7f98fc15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 --2- 172.21.15.60:0/1518058787 >> v2:172.21.15.144:3300/0 conn(0x7f98fc0ad970 0x7f98fc15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 --2- 172.21.15.60:0/1518058787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc06c420 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 -- 172.21.15.60:0/1518058787 >> 172.21.15.60:0/1518058787 conn(0x7f98fc0ce6a0 msgr2=0x7f98fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:05.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 -- 172.21.15.60:0/1518058787 shutdown_connections 2023-10-16T14:14:05.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.605+0000 7f9901128640 1 -- 172.21.15.60:0/1518058787 wait complete. 2023-10-16T14:14:05.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.607+0000 7f9901128640 1 Processor -- start 2023-10-16T14:14:05.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.607+0000 7f9901128640 1 -- start start 2023-10-16T14:14:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f9901128640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f9901128640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc0ad970 0x7f98fc0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f9901128640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98fc15ebc0 0x7f98fc0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f9901128640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc152e50 con 0x7f98fc06c040 2023-10-16T14:14:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f9901128640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1539d0 con 0x7f98fc15ebc0 2023-10-16T14:14:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f9901128640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1530a0 con 0x7f98fc0ad970 2023-10-16T14:14:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.608+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48090/0 (socket says 172.21.15.60:48090) 2023-10-16T14:14:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 -- 172.21.15.60:0/2852941445 learned_addr learned my addr 172.21.15.60:0/2852941445 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f9900927640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98fc15ebc0 0x7f98fc0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98f37fe640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc0ad970 0x7f98fc0c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 -- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc0ad970 msgr2=0x7f98fc0c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc0ad970 0x7f98fc0c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 -- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98fc15ebc0 msgr2=0x7f98fc0c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98fc15ebc0 0x7f98fc0c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 -- 172.21.15.60:0/2852941445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec009b30 con 0x7f98fc06c040 2023-10-16T14:14:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f9900927640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98fc15ebc0 0x7f98fc0c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:14:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.609+0000 7f98fbfff640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc0c5770 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009e20 tx=0x7f98ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.610+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec03e450 con 0x7f98fc06c040 2023-10-16T14:14:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.610+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec03e5d0 con 0x7f98fc06c040 2023-10-16T14:14:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.610+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec03e980 con 0x7f98fc06c040 2023-10-16T14:14:05.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.610+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc153380 con 0x7f98fc06c040 2023-10-16T14:14:05.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.610+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc154f60 con 0x7f98fc06c040 2023-10-16T14:14:05.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.611+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc06b330 con 0x7f98fc06c040 2023-10-16T14:14:05.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.612+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f98ec02cd50 con 0x7f98fc06c040 2023-10-16T14:14:05.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.613+0000 7f98f97fa640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f98dc06af90 0x7f98dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:05.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.614+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f98ec0ac4e0 con 0x7f98fc06c040 2023-10-16T14:14:05.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.614+0000 7f98f37fe640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f98dc06af90 0x7f98dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:05.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.616+0000 7f98f37fe640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f98dc06af90 0x7f98dc06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98fc14c300 tx=0x7f98e4008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:05.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.618+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98ec0773d0 con 0x7f98fc06c040 2023-10-16T14:14:05.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.813+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f98fc0cc550 con 0x7f98dc06af90 2023-10-16T14:14:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.814+0000 7f98f97fa640 1 -- 172.21.15.60:0/2852941445 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f98e400a1f0 con 0x7f98dc06af90 2023-10-16T14:14:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 3 to deep-scrub 2023-10-16T14:14:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f98dc06af90 msgr2=0x7f98dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f98dc06af90 0x7f98dc06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98fc14c300 tx=0x7f98e4008040 comp rx=0 tx=0).stop 2023-10-16T14:14:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 msgr2=0x7f98fc0c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc0c5770 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009e20 tx=0x7f98ec004a40 comp rx=0 tx=0).stop 2023-10-16T14:14:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 shutdown_connections 2023-10-16T14:14:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f98dc06af90 0x7f98dc06d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f98fc0ad970 0x7f98fc0c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f98fc15ebc0 0x7f98fc0c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 --2- 172.21.15.60:0/2852941445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98fc06c040 0x7f98fc0c5770 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 >> 172.21.15.60:0/2852941445 conn(0x7f98fc0ce6a0 msgr2=0x7f98fc15d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 shutdown_connections 2023-10-16T14:14:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:05.817+0000 7f9901128640 1 -- 172.21.15.60:0/2852941445 wait complete. 2023-10-16T14:14:05.840 INFO:tasks.ceph:Scrubbing osd.4 2023-10-16T14:14:05.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.067+0000 7f9a6cd28640 1 Processor -- start 2023-10-16T14:14:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.067+0000 7f9a6cd28640 1 -- start start 2023-10-16T14:14:06.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.068+0000 7f9a6cd28640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a6815c100 0x7f9a680ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.068+0000 7f9a6cd28640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a680cd4a0 con 0x7f9a68153740 2023-10-16T14:14:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.068+0000 7f9a6cd28640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6815ac60 con 0x7f9a680ccb00 2023-10-16T14:14:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.068+0000 7f9a6cd28640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a68070030 con 0x7f9a6815c100 2023-10-16T14:14:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a677fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9a68153740 0x7f9a6815a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53450/0 (socket says 172.21.15.60:53450) 2023-10-16T14:14:06.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a677fe640 1 -- 172.21.15.60:0/1159906520 learned_addr learned my addr 172.21.15.60:0/1159906520 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:06.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a67fff640 1 --2- 172.21.15.60:0/1159906520 >> v2:172.21.15.60:3300/0 conn(0x7f9a6815c100 0x7f9a680ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a667fc640 1 -- 172.21.15.60:0/1159906520 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255813989 0 0) 0x7f9a680cd4a0 con 0x7f9a68153740 2023-10-16T14:14:06.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a67fff640 1 -- 172.21.15.60:0/1159906520 >> v1:172.21.15.144:6789/0 conn(0x7f9a680ccb00 legacy=0x7f9a680ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a67fff640 1 -- 172.21.15.60:0/1159906520 >> v1:172.21.15.60:6789/0 conn(0x7f9a68153740 legacy=0x7f9a6815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.069+0000 7f9a67fff640 1 -- 172.21.15.60:0/1159906520 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68070310 con 0x7f9a6815c100 2023-10-16T14:14:06.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.070+0000 7f9a67fff640 1 --2- 172.21.15.60:0/1159906520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6815c100 0x7f9a680ccf60 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009d90 tx=0x7f9a6002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0c0ef327260d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a667fc640 1 -- 172.21.15.60:0/1159906520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6002cc60 con 0x7f9a6815c100 2023-10-16T14:14:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a667fc640 1 -- 172.21.15.60:0/1159906520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6002cdc0 con 0x7f9a6815c100 2023-10-16T14:14:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a667fc640 1 -- 172.21.15.60:0/1159906520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60034660 con 0x7f9a6815c100 2023-10-16T14:14:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a6cd28640 1 -- 172.21.15.60:0/1159906520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6815c100 msgr2=0x7f9a680ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a6cd28640 1 --2- 172.21.15.60:0/1159906520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6815c100 0x7f9a680ccf60 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009d90 tx=0x7f9a6002beb0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a6cd28640 1 -- 172.21.15.60:0/1159906520 shutdown_connections 2023-10-16T14:14:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a6cd28640 1 --2- 172.21.15.60:0/1159906520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a6815c100 0x7f9a680ccf60 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.071+0000 7f9a6cd28640 1 -- 172.21.15.60:0/1159906520 >> 172.21.15.60:0/1159906520 conn(0x7f9a680ce6a0 msgr2=0x7f9a680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.072+0000 7f9a6cd28640 1 -- 172.21.15.60:0/1159906520 shutdown_connections 2023-10-16T14:14:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.072+0000 7f9a6cd28640 1 -- 172.21.15.60:0/1159906520 wait complete. 2023-10-16T14:14:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.073+0000 7f9a6cd28640 1 Processor -- start 2023-10-16T14:14:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.073+0000 7f9a6cd28640 1 -- start start 2023-10-16T14:14:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.074+0000 7f9a6cd28640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a680ccb00 0x7f9a680c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.074+0000 7f9a6cd28640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a68153740 0x7f9a680c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.074+0000 7f9a6cd28640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a6815c100 0x7f9a681f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.074+0000 7f9a6cd28640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680c6a60 con 0x7f9a680ccb00 2023-10-16T14:14:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.074+0000 7f9a6cd28640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68156550 con 0x7f9a68153740 2023-10-16T14:14:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.074+0000 7f9a6cd28640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680c60c0 con 0x7f9a6815c100 2023-10-16T14:14:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a680ccb00 0x7f9a680c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a680ccb00 0x7f9a680c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48114/0 (socket says 172.21.15.60:48114) 2023-10-16T14:14:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 -- 172.21.15.60:0/2158466812 learned_addr learned my addr 172.21.15.60:0/2158466812 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 -- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a6815c100 msgr2=0x7f9a681f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f9a6815c100 0x7f9a681f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 -- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a68153740 msgr2=0x7f9a680c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a68153740 0x7f9a680c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 -- 172.21.15.60:0/2158466812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60009a70 con 0x7f9a680ccb00 2023-10-16T14:14:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.075+0000 7f9a677fe640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a680ccb00 0x7f9a680c8850 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c4d0 tx=0x7f9a5800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.076+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580074e0 con 0x7f9a680ccb00 2023-10-16T14:14:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.076+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58007660 con 0x7f9a680ccb00 2023-10-16T14:14:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.076+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58004a20 con 0x7f9a680ccb00 2023-10-16T14:14:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.076+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a680c63a0 con 0x7f9a680ccb00 2023-10-16T14:14:06.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.076+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a681f6320 con 0x7f9a680ccb00 2023-10-16T14:14:06.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.077+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9a680708d0 con 0x7f9a680ccb00 2023-10-16T14:14:06.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.078+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a58004b80 con 0x7f9a680ccb00 2023-10-16T14:14:06.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.079+0000 7f9a4ffff640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f9a5406af90 0x7f9a5406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.079+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f9a580889d0 con 0x7f9a680ccb00 2023-10-16T14:14:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.079+0000 7f9a4ffff640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7f9a540714d0 0x7f9a540738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.079+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 --> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a54073fa0 con 0x7f9a540714d0 2023-10-16T14:14:06.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.079+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a5808d2a0 con 0x7f9a680ccb00 2023-10-16T14:14:06.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.080+0000 7f9a66ffd640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f9a5406af90 0x7f9a5406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.080+0000 7f9a67fff640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7f9a540714d0 0x7f9a540738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.081+0000 7f9a66ffd640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f9a5406af90 0x7f9a5406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a680c8490 tx=0x7f9a5c0079b0 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.082+0000 7f9a67fff640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7f9a540714d0 0x7f9a540738f0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.084+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== osd.4 v2:172.21.15.144:6800/2038056415 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f9a54073fa0 con 0x7f9a540714d0 2023-10-16T14:14:06.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.098+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 --> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9a680705f0 con 0x7f9a540714d0 2023-10-16T14:14:06.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.099+0000 7f9a4ffff640 1 -- 172.21.15.60:0/2158466812 <== osd.4 v2:172.21.15.144:6800/2038056415 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9a60034440 con 0x7f9a540714d0 2023-10-16T14:14:06.103 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:06.104 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:06.104 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:06.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7f9a540714d0 msgr2=0x7f9a540738f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6800/2038056415,v1:172.21.15.144:6801/2038056415] conn(0x7f9a540714d0 0x7f9a540738f0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f9a5406af90 msgr2=0x7f9a5406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f9a5406af90 0x7f9a5406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a680c8490 tx=0x7f9a5c0079b0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a680ccb00 msgr2=0x7f9a680c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 --2- 172.21.15.60:0/2158466812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a680ccb00 0x7f9a680c8850 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c4d0 tx=0x7f9a5800c990 comp rx=0 tx=0).stop 2023-10-16T14:14:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a67fff640 1 -- 172.21.15.60:0/2158466812 reap_dead start 2023-10-16T14:14:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 shutdown_connections 2023-10-16T14:14:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 >> 172.21.15.60:0/2158466812 conn(0x7f9a680ce6a0 msgr2=0x7f9a680c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.100+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 shutdown_connections 2023-10-16T14:14:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.101+0000 7f9a6cd28640 1 -- 172.21.15.60:0/2158466812 wait complete. 2023-10-16T14:14:06.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-16T14:14:06.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.330+0000 7fbf1f59e640 1 Processor -- start 2023-10-16T14:14:06.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.330+0000 7fbf1f59e640 1 -- start start 2023-10-16T14:14:06.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.331+0000 7fbf1f59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbf2015dde0 0x7fbf200709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.331+0000 7fbf1f59e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf20160a90 con 0x7fbf2015c9f0 2023-10-16T14:14:06.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.332+0000 7fbf1f59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf200cd900 con 0x7fbf20070f20 2023-10-16T14:14:06.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.332+0000 7fbf1f59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf20071350 con 0x7fbf2015dde0 2023-10-16T14:14:06.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.332+0000 7fbf1e59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbf2015c9f0 0x7fbf200c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53462/0 (socket says 172.21.15.60:53462) 2023-10-16T14:14:06.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.332+0000 7fbf1e59c640 1 -- 172.21.15.60:0/572101419 learned_addr learned my addr 172.21.15.60:0/572101419 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:06.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.332+0000 7fbf17fff640 1 --2- 172.21.15.60:0/572101419 >> v2:172.21.15.144:3300/0 conn(0x7fbf2015dde0 0x7fbf200709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148108852 0 0) 0x7fbf20160a90 con 0x7fbf2015c9f0 2023-10-16T14:14:06.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c003520 con 0x7fbf2015c9f0 2023-10-16T14:14:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2390988364 0 0) 0x7fbf0c003520 con 0x7fbf2015c9f0 2023-10-16T14:14:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 >> v2:172.21.15.144:3300/0 conn(0x7fbf2015dde0 msgr2=0x7fbf200709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 --2- 172.21.15.60:0/572101419 >> v2:172.21.15.144:3300/0 conn(0x7fbf2015dde0 0x7fbf200709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 >> v1:172.21.15.144:6789/0 conn(0x7fbf20070f20 legacy=0x7fbf200ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf200cdbe0 con 0x7fbf2015c9f0 2023-10-16T14:14:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fbf10001670 con 0x7fbf2015c9f0 2023-10-16T14:14:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 >> v1:172.21.15.60:6789/0 conn(0x7fbf2015c9f0 legacy=0x7fbf200c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf0c003970 0x7fbf0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf0c0047c0 0x7fbf0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c005180 0x7fbf0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf20071350 con 0x7fbf0c0047c0 2023-10-16T14:14:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c00c020 con 0x7fbf0c003970 2023-10-16T14:14:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.333+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c00c1e0 con 0x7fbf0c005180 2023-10-16T14:14:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf17fff640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf0c0047c0 0x7fbf0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf1e59c640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf0c003970 0x7fbf0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf17fff640 1 -- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c005180 msgr2=0x7fbf0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf17fff640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c005180 0x7fbf0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf17fff640 1 -- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf0c003970 msgr2=0x7fbf0c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf17fff640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf0c003970 0x7fbf0c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf17fff640 1 -- 172.21.15.60:0/572101419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c003520 con 0x7fbf0c0047c0 2023-10-16T14:14:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf1e59c640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf0c003970 0x7fbf0c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.334+0000 7fbf1ed9d640 1 -- 172.21.15.60:0/572101419 reap_dead start 2023-10-16T14:14:06.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.335+0000 7fbf17fff640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf0c0047c0 0x7fbf0c004c40 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009ac0 tx=0x7fbf0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab922ca965570f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0802c710 con 0x7fbf0c0047c0 2023-10-16T14:14:06.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0802c870 con 0x7fbf0c0047c0 2023-10-16T14:14:06.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1dd9b640 1 -- 172.21.15.60:0/572101419 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0802cc00 con 0x7fbf0c0047c0 2023-10-16T14:14:06.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 -- 172.21.15.60:0/572101419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf0c0047c0 msgr2=0x7fbf0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf0c0047c0 0x7fbf0c004c40 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009ac0 tx=0x7fbf0c0057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 -- 172.21.15.60:0/572101419 shutdown_connections 2023-10-16T14:14:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/572101419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf0c0047c0 0x7fbf0c004c40 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 -- 172.21.15.60:0/572101419 >> 172.21.15.60:0/572101419 conn(0x7fbf200ce6a0 msgr2=0x7fbf200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 -- 172.21.15.60:0/572101419 shutdown_connections 2023-10-16T14:14:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.336+0000 7fbf1f59e640 1 -- 172.21.15.60:0/572101419 wait complete. 2023-10-16T14:14:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.338+0000 7fbf1f59e640 1 Processor -- start 2023-10-16T14:14:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.338+0000 7fbf1f59e640 1 -- start start 2023-10-16T14:14:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.339+0000 7fbf1f59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c0047c0 0x7fbf200c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.339+0000 7fbf1f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf201f2100 0x7fbf200c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.339+0000 7fbf1f59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 0x7fbf200c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.339+0000 7fbf1f59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf20158480 con 0x7fbf200c21d0 2023-10-16T14:14:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.339+0000 7fbf1f59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf200c50e0 con 0x7fbf201f2100 2023-10-16T14:14:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.339+0000 7fbf1f59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf200c30c0 con 0x7fbf0c0047c0 2023-10-16T14:14:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 0x7fbf200c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 0x7fbf200c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48136/0 (socket says 172.21.15.60:48136) 2023-10-16T14:14:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 -- 172.21.15.60:0/592974756 learned_addr learned my addr 172.21.15.60:0/592974756 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1e59c640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c0047c0 0x7fbf200c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 -- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c0047c0 msgr2=0x7fbf200c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c0047c0 0x7fbf200c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 -- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf201f2100 msgr2=0x7fbf200c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf201f2100 0x7fbf200c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.340+0000 7fbf1ed9d640 1 -- 172.21.15.60:0/592974756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf200c2790 con 0x7fbf200c21d0 2023-10-16T14:14:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.341+0000 7fbf1ed9d640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 0x7fbf200c4ba0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7fbf2006d350 tx=0x7fbf1800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.341+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18003970 con 0x7fbf200c21d0 2023-10-16T14:14:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.341+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18003ad0 con 0x7fbf200c21d0 2023-10-16T14:14:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.341+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18003e80 con 0x7fbf200c21d0 2023-10-16T14:14:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.341+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2006bfd0 con 0x7fbf200c21d0 2023-10-16T14:14:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.341+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2006c590 con 0x7fbf200c21d0 2023-10-16T14:14:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.342+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2006c7b0 con 0x7fbf200c21d0 2023-10-16T14:14:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.343+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf18003fe0 con 0x7fbf200c21d0 2023-10-16T14:14:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.344+0000 7fbf16ffd640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fbf0006afc0 0x7fbf0006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.344+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fbf18088a20 con 0x7fbf200c21d0 2023-10-16T14:14:06.358 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.345+0000 7fbf1e59c640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fbf0006afc0 0x7fbf0006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.346+0000 7fbf1e59c640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fbf0006afc0 0x7fbf0006d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf10001200 tx=0x7fbf10009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.349+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf180539d0 con 0x7fbf200c21d0 2023-10-16T14:14:06.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.544+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf200704c0 con 0x7fbf0006afc0 2023-10-16T14:14:06.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.545+0000 7fbf16ffd640 1 -- 172.21.15.60:0/592974756 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbf1000d050 con 0x7fbf0006afc0 2023-10-16T14:14:06.549 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 4 to deep-scrub 2023-10-16T14:14:06.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fbf0006afc0 msgr2=0x7fbf0006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fbf0006afc0 0x7fbf0006d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf10001200 tx=0x7fbf10009040 comp rx=0 tx=0).stop 2023-10-16T14:14:06.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 msgr2=0x7fbf200c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 0x7fbf200c4ba0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7fbf2006d350 tx=0x7fbf1800c8b0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 shutdown_connections 2023-10-16T14:14:06.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fbf0006afc0 0x7fbf0006d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf200c21d0 0x7fbf200c4ba0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf201f2100 0x7fbf200c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.554 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 --2- 172.21.15.60:0/592974756 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fbf0c0047c0 0x7fbf200c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 >> 172.21.15.60:0/592974756 conn(0x7fbf200ce6a0 msgr2=0x7fbf200c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:06.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.548+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 shutdown_connections 2023-10-16T14:14:06.555 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.549+0000 7fbf1f59e640 1 -- 172.21.15.60:0/592974756 wait complete. 2023-10-16T14:14:06.572 INFO:tasks.ceph:Scrubbing osd.5 2023-10-16T14:14:06.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:06.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.802+0000 7ff407f2c640 1 Processor -- start 2023-10-16T14:14:06.806 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.802+0000 7ff407f2c640 1 -- start start 2023-10-16T14:14:06.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.803+0000 7ff407f2c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff400159da0 0x7ff4000c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.807 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.803+0000 7ff407f2c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40015dfb0 con 0x7ff4001599c0 2023-10-16T14:14:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.803+0000 7ff407f2c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40015e290 con 0x7ff4000cce20 2023-10-16T14:14:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.804+0000 7ff407f2c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40015e570 con 0x7ff400159da0 2023-10-16T14:14:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.804+0000 7ff405cc8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff4001599c0 0x7ff40015d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53478/0 (socket says 172.21.15.60:53478) 2023-10-16T14:14:06.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.804+0000 7ff405cc8640 1 -- 172.21.15.60:0/4170770981 learned_addr learned my addr 172.21.15.60:0/4170770981 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:06.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.804+0000 7ff4054c7640 1 --2- 172.21.15.60:0/4170770981 >> v2:172.21.15.144:3300/0 conn(0x7ff400159da0 0x7ff4000c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.804+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545155208 0 0) 0x7ff40015dfb0 con 0x7ff4001599c0 2023-10-16T14:14:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.804+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec003520 con 0x7ff4001599c0 2023-10-16T14:14:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3249473938 0 0) 0x7ff3ec003520 con 0x7ff4001599c0 2023-10-16T14:14:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 >> v2:172.21.15.144:3300/0 conn(0x7ff400159da0 msgr2=0x7ff4000c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 --2- 172.21.15.60:0/4170770981 >> v2:172.21.15.144:3300/0 conn(0x7ff400159da0 0x7ff4000c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 >> v1:172.21.15.144:6789/0 conn(0x7ff4000cce20 legacy=0x7ff40006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40006fda0 con 0x7ff4001599c0 2023-10-16T14:14:06.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7ff3fc001670 con 0x7ff4001599c0 2023-10-16T14:14:06.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 >> v1:172.21.15.60:6789/0 conn(0x7ff4001599c0 legacy=0x7ff40015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0039e0 0x7ff3ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3ec0047c0 0x7ff3ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3ec005180 0x7ff3ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40015e570 con 0x7ff3ec0047c0 2023-10-16T14:14:06.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec00c0b0 con 0x7ff3ec005180 2023-10-16T14:14:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec00c270 con 0x7ff3ec0039e0 2023-10-16T14:14:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff4054c7640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3ec0047c0 0x7ff3ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.805+0000 7ff405cc8640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0039e0 0x7ff3ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4054c7640 1 -- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0039e0 msgr2=0x7ff3ec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4054c7640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0039e0 0x7ff3ec003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4054c7640 1 -- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3ec005180 msgr2=0x7ff3ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:06.818 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4064c9640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3ec005180 0x7ff3ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4054c7640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3ec005180 0x7ff3ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4054c7640 1 -- 172.21.15.60:0/4170770981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec003520 con 0x7ff3ec0047c0 2023-10-16T14:14:06.819 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4064c9640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3ec005180 0x7ff3ec00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:06.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.806+0000 7ff4064c9640 1 -- 172.21.15.60:0/4170770981 reap_dead start 2023-10-16T14:14:06.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.807+0000 7ff4054c7640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3ec0047c0 0x7ff3ec004c40 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009ac0 tx=0x7ff3ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92d23a6e04e3288 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.820 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.807+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f002c6b0 con 0x7ff3ec0047c0 2023-10-16T14:14:06.821 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.807+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f002c810 con 0x7ff3ec0047c0 2023-10-16T14:14:06.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.807+0000 7ff404cc6640 1 -- 172.21.15.60:0/4170770981 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f002cba0 con 0x7ff3ec0047c0 2023-10-16T14:14:06.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 -- 172.21.15.60:0/4170770981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3ec0047c0 msgr2=0x7ff3ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3ec0047c0 0x7ff3ec004c40 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009ac0 tx=0x7ff3ec0057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 -- 172.21.15.60:0/4170770981 shutdown_connections 2023-10-16T14:14:06.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 --2- 172.21.15.60:0/4170770981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3ec0047c0 0x7ff3ec004c40 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 -- 172.21.15.60:0/4170770981 >> 172.21.15.60:0/4170770981 conn(0x7ff4000cb360 msgr2=0x7ff4000cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:06.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 -- 172.21.15.60:0/4170770981 shutdown_connections 2023-10-16T14:14:06.824 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.808+0000 7ff407f2c640 1 -- 172.21.15.60:0/4170770981 wait complete. 2023-10-16T14:14:06.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.810+0000 7ff407f2c640 1 Processor -- start 2023-10-16T14:14:06.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.810+0000 7ff407f2c640 1 -- start start 2023-10-16T14:14:06.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff407f2c640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0047c0 0x7ff40014acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff407f2c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff400151ec0 0x7ff4000c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff407f2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4000b1500 0x7ff4000ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff407f2c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000b0270 con 0x7ff4000b1500 2023-10-16T14:14:06.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff407f2c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40014b1f0 con 0x7ff400151ec0 2023-10-16T14:14:06.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff407f2c640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000c8840 con 0x7ff3ec0047c0 2023-10-16T14:14:06.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff4064c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4000b1500 0x7ff4000ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff4064c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4000b1500 0x7ff4000ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48150/0 (socket says 172.21.15.60:48150) 2023-10-16T14:14:06.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff4064c9640 1 -- 172.21.15.60:0/3888050617 learned_addr learned my addr 172.21.15.60:0/3888050617 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:06.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.811+0000 7ff405cc8640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0047c0 0x7ff40014acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4054c7640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff400151ec0 0x7ff4000c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4064c9640 1 -- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0047c0 msgr2=0x7ff40014acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.831 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4064c9640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff3ec0047c0 0x7ff40014acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4064c9640 1 -- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff400151ec0 msgr2=0x7ff4000c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4064c9640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff400151ec0 0x7ff4000c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4064c9640 1 -- 172.21.15.60:0/3888050617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000af5f0 con 0x7ff4000b1500 2023-10-16T14:14:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff4064c9640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4000b1500 0x7ff4000ace30 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800d540 tx=0x7ff3f800da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8003d80 con 0x7ff4000b1500 2023-10-16T14:14:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f80050f0 con 0x7ff4000b1500 2023-10-16T14:14:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80054a0 con 0x7ff4000b1500 2023-10-16T14:14:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.812+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4000af8d0 con 0x7ff4000b1500 2023-10-16T14:14:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.813+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4000afe90 con 0x7ff4000b1500 2023-10-16T14:14:06.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.814+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff40006fa50 con 0x7ff4000b1500 2023-10-16T14:14:06.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.815+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff3f8005600 con 0x7ff4000b1500 2023-10-16T14:14:06.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.816+0000 7ff3f67fc640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff3e406af80 0x7ff3e406d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.816+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7ff3f80891c0 con 0x7ff4000b1500 2023-10-16T14:14:06.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.816+0000 7ff3f67fc640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7ff3e40714c0 0x7ff3e40738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:06.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.816+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 --> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3e4073f90 con 0x7ff3e40714c0 2023-10-16T14:14:06.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.816+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3f808b7a0 con 0x7ff4000b1500 2023-10-16T14:14:06.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.816+0000 7ff405cc8640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff3e406af80 0x7ff3e406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.817+0000 7ff4054c7640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7ff3e40714c0 0x7ff3e40738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:06.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.818+0000 7ff405cc8640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff3e406af80 0x7ff3e406d440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0017a0 tx=0x7ff3fc001630 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.818+0000 7ff4054c7640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7ff3e40714c0 0x7ff3e40738e0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:06.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.820+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== osd.5 v2:172.21.15.144:6814/1961528470 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff3e4073f90 con 0x7ff3e40714c0 2023-10-16T14:14:06.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.834+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 --> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff4000b03b0 con 0x7ff3e40714c0 2023-10-16T14:14:06.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.835+0000 7ff3f67fc640 1 -- 172.21.15.60:0/3888050617 <== osd.5 v2:172.21.15.144:6814/1961528470 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff4000b03b0 con 0x7ff3e40714c0 2023-10-16T14:14:06.840 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:06.841 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:06.841 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:06.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7ff3e40714c0 msgr2=0x7ff3e40738e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.842 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6814/1961528470,v1:172.21.15.144:6815/1961528470] conn(0x7ff3e40714c0 0x7ff3e40738e0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:06.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff3e406af80 msgr2=0x7ff3e406d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff3e406af80 0x7ff3e406d440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0017a0 tx=0x7ff3fc001630 comp rx=0 tx=0).stop 2023-10-16T14:14:06.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4000b1500 msgr2=0x7ff4000ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:06.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 --2- 172.21.15.60:0/3888050617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4000b1500 0x7ff4000ace30 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800d540 tx=0x7ff3f800da00 comp rx=0 tx=0).stop 2023-10-16T14:14:06.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff4064c9640 1 -- 172.21.15.60:0/3888050617 reap_dead start 2023-10-16T14:14:06.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 shutdown_connections 2023-10-16T14:14:06.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 >> 172.21.15.60:0/3888050617 conn(0x7ff4000cb360 msgr2=0x7ff40006c190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:06.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 shutdown_connections 2023-10-16T14:14:06.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:06.836+0000 7ff407f2c640 1 -- 172.21.15.60:0/3888050617 wait complete. 2023-10-16T14:14:06.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-16T14:14:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.087+0000 7fd565f11640 1 Processor -- start 2023-10-16T14:14:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.087+0000 7fd565f11640 1 -- start start 2023-10-16T14:14:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd565f11640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd56006d220 0x7fd5600c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd565f11640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 0x7fd5600ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd565f11640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600acd70 con 0x7fd560065ab0 2023-10-16T14:14:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd565f11640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5600ac4a0 con 0x7fd56006d220 2023-10-16T14:14:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd565f11640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd560065f40 con 0x7fd5600ce280 2023-10-16T14:14:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd55ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd560065ab0 0x7fd5600cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53490/0 (socket says 172.21.15.60:53490) 2023-10-16T14:14:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd55ffff640 1 -- 172.21.15.60:0/1908037297 learned_addr learned my addr 172.21.15.60:0/1908037297 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.089+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287419081 0 0) 0x7fd5600acd70 con 0x7fd560065ab0 2023-10-16T14:14:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd548003520 con 0x7fd560065ab0 2023-10-16T14:14:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd564f0f640 1 --2- 172.21.15.60:0/1908037297 >> v2:172.21.15.144:3300/0 conn(0x7fd56006d220 0x7fd5600c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd565710640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 0x7fd5600ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3769509850 0 0) 0x7fd548003520 con 0x7fd560065ab0 2023-10-16T14:14:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 msgr2=0x7fd5600ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 0x7fd5600ad200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 >> v2:172.21.15.144:3300/0 conn(0x7fd56006d220 msgr2=0x7fd5600c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 --2- 172.21.15.60:0/1908037297 >> v2:172.21.15.144:3300/0 conn(0x7fd56006d220 0x7fd5600c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600ac720 con 0x7fd560065ab0 2023-10-16T14:14:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7fd554001670 con 0x7fd560065ab0 2023-10-16T14:14:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 >> v1:172.21.15.60:6789/0 conn(0x7fd560065ab0 legacy=0x7fd5600cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd548003970 0x7fd548003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 0x7fd5480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd548004d00 0x7fd548005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5600ac4a0 con 0x7fd548003970 2023-10-16T14:14:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd560065f40 con 0x7fd548004d00 2023-10-16T14:14:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.090+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54800bae0 con 0x7fd5600ce280 2023-10-16T14:14:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd564f0f640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd548003970 0x7fd548003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd55ffff640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 0x7fd5480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd565710640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd548004d00 0x7fd548005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd564f0f640 1 -- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 msgr2=0x7fd5480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd564f0f640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd5600ce280 0x7fd5480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd564f0f640 1 -- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd548004d00 msgr2=0x7fd548005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd564f0f640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd548004d00 0x7fd548005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.091+0000 7fd564f0f640 1 -- 172.21.15.60:0/1908037297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd548003520 con 0x7fd548003970 2023-10-16T14:14:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.093+0000 7fd564f0f640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd548003970 0x7fd548003d90 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fd550007360 tx=0x7fd5480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c86749c4cc231e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.093+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55002ccf0 con 0x7fd548003970 2023-10-16T14:14:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.093+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55002ce50 con 0x7fd548003970 2023-10-16T14:14:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.093+0000 7fd55f7fe640 1 -- 172.21.15.60:0/1908037297 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550034660 con 0x7fd548003970 2023-10-16T14:14:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565f11640 1 -- 172.21.15.60:0/1908037297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd548003970 msgr2=0x7fd548003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565f11640 1 --2- 172.21.15.60:0/1908037297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd548003970 0x7fd548003d90 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fd550007360 tx=0x7fd5480057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565710640 1 -- 172.21.15.60:0/1908037297 reap_dead start 2023-10-16T14:14:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565f11640 1 -- 172.21.15.60:0/1908037297 shutdown_connections 2023-10-16T14:14:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565f11640 1 -- 172.21.15.60:0/1908037297 >> 172.21.15.60:0/1908037297 conn(0x7fd5600b0a80 msgr2=0x7fd5600b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565f11640 1 -- 172.21.15.60:0/1908037297 shutdown_connections 2023-10-16T14:14:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.094+0000 7fd565f11640 1 -- 172.21.15.60:0/1908037297 wait complete. 2023-10-16T14:14:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.096+0000 7fd565f11640 1 Processor -- start 2023-10-16T14:14:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.096+0000 7fd565f11640 1 -- start start 2023-10-16T14:14:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565f11640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd548003970 0x7fd560099ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565f11640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5601f5ef0 0x7fd5601f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565f11640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 0x7fd5600b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565f11640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5600b1c50 con 0x7fd5600b3bb0 2023-10-16T14:14:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565f11640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601f68b0 con 0x7fd5601f5ef0 2023-10-16T14:14:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565f11640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5601f5a10 con 0x7fd548003970 2023-10-16T14:14:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565710640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 0x7fd5600b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565710640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 0x7fd5600b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48170/0 (socket says 172.21.15.60:48170) 2023-10-16T14:14:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.097+0000 7fd565710640 1 -- 172.21.15.60:0/2107434791 learned_addr learned my addr 172.21.15.60:0/2107434791 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd564f0f640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd548003970 0x7fd560099ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd55ffff640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5601f5ef0 0x7fd5601f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd565710640 1 -- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd548003970 msgr2=0x7fd560099ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd565710640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd548003970 0x7fd560099ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd565710640 1 -- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5601f5ef0 msgr2=0x7fd5601f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd565710640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5601f5ef0 0x7fd5601f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd565710640 1 -- 172.21.15.60:0/2107434791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600b29d0 con 0x7fd5600b3bb0 2023-10-16T14:14:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd55ffff640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5601f5ef0 0x7fd5601f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd565710640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 0x7fd5600b4030 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fd5600671f0 tx=0x7fd55800f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.098+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558004440 con 0x7fd5600b3bb0 2023-10-16T14:14:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.099+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558003610 con 0x7fd5600b3bb0 2023-10-16T14:14:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.099+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5600b1ef0 con 0x7fd5600b3bb0 2023-10-16T14:14:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.099+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580039c0 con 0x7fd5600b3bb0 2023-10-16T14:14:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.099+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5600b24b0 con 0x7fd5600b3bb0 2023-10-16T14:14:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.100+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5600b26a0 con 0x7fd5600b3bb0 2023-10-16T14:14:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.101+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd558003b20 con 0x7fd5600b3bb0 2023-10-16T14:14:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.102+0000 7fd55d7fa640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd54c06af90 0x7fd54c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.103+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fd558089770 con 0x7fd5600b3bb0 2023-10-16T14:14:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.103+0000 7fd564f0f640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd54c06af90 0x7fd54c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.105+0000 7fd564f0f640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd54c06af90 0x7fd54c06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd550036000 tx=0x7fd550030000 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.107+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5580546f0 con 0x7fd5600b3bb0 2023-10-16T14:14:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.306+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5600aca00 con 0x7fd54c06af90 2023-10-16T14:14:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.307+0000 7fd55d7fa640 1 -- 172.21.15.60:0/2107434791 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd550003990 con 0x7fd54c06af90 2023-10-16T14:14:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 5 to deep-scrub 2023-10-16T14:14:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.309+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd54c06af90 msgr2=0x7fd54c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.309+0000 7fd565f11640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd54c06af90 0x7fd54c06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd550036000 tx=0x7fd550030000 comp rx=0 tx=0).stop 2023-10-16T14:14:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.309+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 msgr2=0x7fd5600b4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.309+0000 7fd565f11640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 0x7fd5600b4030 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fd5600671f0 tx=0x7fd55800f8a0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 shutdown_connections 2023-10-16T14:14:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd54c06af90 0x7fd54c06d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5600b3bb0 0x7fd5600b4030 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5601f5ef0 0x7fd5601f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 --2- 172.21.15.60:0/2107434791 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd548003970 0x7fd560099ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 >> 172.21.15.60:0/2107434791 conn(0x7fd5600b0a80 msgr2=0x7fd56009aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 shutdown_connections 2023-10-16T14:14:07.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.310+0000 7fd565f11640 1 -- 172.21.15.60:0/2107434791 wait complete. 2023-10-16T14:14:07.334 INFO:tasks.ceph:Scrubbing osd.6 2023-10-16T14:14:07.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.566+0000 7fb3cb59e640 1 Processor -- start 2023-10-16T14:14:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.567+0000 7fb3cb59e640 1 -- start start 2023-10-16T14:14:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.568+0000 7fb3cb59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3cc15cd70 0x7fb3cc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.568+0000 7fb3cb59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0b8480 0x7fb3cc0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.568+0000 7fb3cb59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3cc15d150 con 0x7fb3cc0d0be0 2023-10-16T14:14:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.568+0000 7fb3cb59e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc0d1180 con 0x7fb3cc15cd70 2023-10-16T14:14:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.568+0000 7fb3cb59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0d1460 con 0x7fb3cc0b8480 2023-10-16T14:14:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3cc15cd70 0x7fb3cc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3cc15cd70 0x7fb3cc0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48180/0 (socket says 172.21.15.60:48180) 2023-10-16T14:14:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 -- 172.21.15.60:0/3944831561 learned_addr learned my addr 172.21.15.60:0/3944831561 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3c3fff640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0b8480 0x7fb3cc0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 -- 172.21.15.60:0/3944831561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0b8480 msgr2=0x7fb3cc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0b8480 0x7fb3cc0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 -- 172.21.15.60:0/3944831561 >> v1:172.21.15.144:6789/0 conn(0x7fb3cc0d0be0 legacy=0x7fb3cc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3ca59c640 1 -- 172.21.15.60:0/3944831561 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0d1740 con 0x7fb3cc15cd70 2023-10-16T14:14:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.569+0000 7fb3c3fff640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0b8480 0x7fb3cc0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:14:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.571+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0d2220 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009870 tx=0x7fb3bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c56984760f6f6aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.572+0000 7fb3c9d9b640 1 -- 172.21.15.60:0/3944831561 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc030840 con 0x7fb3cc15cd70 2023-10-16T14:14:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.572+0000 7fb3c9d9b640 1 -- 172.21.15.60:0/3944831561 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc0309a0 con 0x7fb3cc15cd70 2023-10-16T14:14:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.572+0000 7fb3c9d9b640 1 -- 172.21.15.60:0/3944831561 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc030d30 con 0x7fb3cc15cd70 2023-10-16T14:14:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.572+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3944831561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 msgr2=0x7fb3cc0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.572+0000 7fb3cb59e640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0d2220 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009870 tx=0x7fb3bc02b980 comp rx=0 tx=0).stop 2023-10-16T14:14:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.573+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3944831561 shutdown_connections 2023-10-16T14:14:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.573+0000 7fb3cb59e640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0b8480 0x7fb3cc0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.573+0000 7fb3cb59e640 1 --2- 172.21.15.60:0/3944831561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0d2220 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.573+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3944831561 >> 172.21.15.60:0/3944831561 conn(0x7fb3cc0cf570 msgr2=0x7fb3cc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.573+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3944831561 shutdown_connections 2023-10-16T14:14:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.573+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3944831561 wait complete. 2023-10-16T14:14:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.574+0000 7fb3cb59e640 1 Processor -- start 2023-10-16T14:14:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.575+0000 7fb3cb59e640 1 -- start start 2023-10-16T14:14:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3cc0b8480 0x7fb3cc151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cb59e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0d0be0 0x7fb3cc156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cb59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cb59e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc152510 con 0x7fb3cc15cd70 2023-10-16T14:14:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cb59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc0c6de0 con 0x7fb3cc0b8480 2023-10-16T14:14:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cb59e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc158d30 con 0x7fb3cc0d0be0 2023-10-16T14:14:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48182/0 (socket says 172.21.15.60:48182) 2023-10-16T14:14:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 -- 172.21.15.60:0/3504950129 learned_addr learned my addr 172.21.15.60:0/3504950129 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3cc0b8480 0x7fb3cc151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3c3fff640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0d0be0 0x7fb3cc156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 -- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0d0be0 msgr2=0x7fb3cc156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fb3cc0d0be0 0x7fb3cc156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 -- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3cc0b8480 msgr2=0x7fb3cc151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.576+0000 7fb3cad9d640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3cc0b8480 0x7fb3cc151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3cad9d640 1 -- 172.21.15.60:0/3504950129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0094d0 con 0x7fb3cc15cd70 2023-10-16T14:14:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3cad9d640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0cac10 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c730 tx=0x7fb3c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4003b70 con 0x7fb3cc15cd70 2023-10-16T14:14:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c4003cf0 con 0x7fb3cc15cd70 2023-10-16T14:14:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4003ea0 con 0x7fb3cc15cd70 2023-10-16T14:14:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3cc159010 con 0x7fb3cc15cd70 2023-10-16T14:14:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.577+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc1f2660 con 0x7fb3cc15cd70 2023-10-16T14:14:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.578+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3cc0c68f0 con 0x7fb3cc15cd70 2023-10-16T14:14:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.580+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3c4005310 con 0x7fb3cc15cd70 2023-10-16T14:14:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.580+0000 7fb3c2ffd640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fb3ac06af90 0x7fb3ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.581+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fb3c4089fd0 con 0x7fb3cc15cd70 2023-10-16T14:14:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.581+0000 7fb3c2ffd640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7fb3ac0714d0 0x7fb3ac0738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.581+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 --> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3ac073fa0 con 0x7fb3ac0714d0 2023-10-16T14:14:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.581+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3c408a400 con 0x7fb3cc15cd70 2023-10-16T14:14:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.581+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fb3ac06af90 0x7fb3ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.582+0000 7fb3c3fff640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7fb3ac0714d0 0x7fb3ac0738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.583+0000 7fb3ca59c640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fb3ac06af90 0x7fb3ac06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc036000 tx=0x7fb3bc008030 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.583+0000 7fb3c3fff640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7fb3ac0714d0 0x7fb3ac0738f0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.585+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== osd.6 v2:172.21.15.144:6813/3196639449 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fb3ac073fa0 con 0x7fb3ac0714d0 2023-10-16T14:14:07.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.599+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 --> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb3cc0d1d00 con 0x7fb3ac0714d0 2023-10-16T14:14:07.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.600+0000 7fb3c2ffd640 1 -- 172.21.15.60:0/3504950129 <== osd.6 v2:172.21.15.144:6813/3196639449 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb3b400ba60 con 0x7fb3ac0714d0 2023-10-16T14:14:07.604 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:07.604 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:07.605 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.600+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7fb3ac0714d0 msgr2=0x7fb3ac0738f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6813/3196639449,v1:172.21.15.144:6818/3196639449] conn(0x7fb3ac0714d0 0x7fb3ac0738f0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.606 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fb3ac06af90 msgr2=0x7fb3ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fb3ac06af90 0x7fb3ac06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc036000 tx=0x7fb3bc008030 comp rx=0 tx=0).stop 2023-10-16T14:14:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 msgr2=0x7fb3cc0cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 --2- 172.21.15.60:0/3504950129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3cc15cd70 0x7fb3cc0cac10 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c730 tx=0x7fb3c400cbf0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.607 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cad9d640 1 -- 172.21.15.60:0/3504950129 reap_dead start 2023-10-16T14:14:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 shutdown_connections 2023-10-16T14:14:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 >> 172.21.15.60:0/3504950129 conn(0x7fb3cc0cf570 msgr2=0x7fb3cc0b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 shutdown_connections 2023-10-16T14:14:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.601+0000 7fb3cb59e640 1 -- 172.21.15.60:0/3504950129 wait complete. 2023-10-16T14:14:07.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-16T14:14:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.846+0000 7f6c6298e640 1 Processor -- start 2023-10-16T14:14:07.851 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.847+0000 7f6c6298e640 1 -- start start 2023-10-16T14:14:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.848+0000 7f6c6298e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6c5c15c9f0 0x7f6c5c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.848+0000 7f6c6298e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6c5c0c9230 0x7f6c5c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.848+0000 7f6c6298e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c070f20 0x7f6c5c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.848+0000 7f6c6298e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0cdec0 con 0x7f6c5c0c9230 2023-10-16T14:14:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.848+0000 7f6c6298e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c160a90 con 0x7f6c5c15c9f0 2023-10-16T14:14:07.854 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.848+0000 7f6c6298e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0cd900 con 0x7f6c5c070f20 2023-10-16T14:14:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6c5c0c9230 0x7f6c5c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6c5c0c9230 0x7f6c5c0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48186/0 (socket says 172.21.15.60:48186) 2023-10-16T14:14:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 -- 172.21.15.60:0/2460573574 learned_addr learned my addr 172.21.15.60:0/2460573574 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:07.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6218d640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c070f20 0x7f6c5c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6198c640 1 --2- 172.21.15.60:0/2460573574 >> v2:172.21.15.144:3300/0 conn(0x7f6c5c15c9f0 0x7f6c5c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 -- 172.21.15.60:0/2460573574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c070f20 msgr2=0x7f6c5c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c070f20 0x7f6c5c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 -- 172.21.15.60:0/2460573574 >> v2:172.21.15.144:3300/0 conn(0x7f6c5c15c9f0 msgr2=0x7f6c5c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 --2- 172.21.15.60:0/2460573574 >> v2:172.21.15.144:3300/0 conn(0x7f6c5c15c9f0 0x7f6c5c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6118b640 1 -- 172.21.15.60:0/2460573574 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0cdbe0 con 0x7f6c5c0c9230 2023-10-16T14:14:07.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6218d640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c070f20 0x7f6c5c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:14:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.849+0000 7f6c6198c640 1 --2- 172.21.15.60:0/2460573574 >> v2:172.21.15.144:3300/0 conn(0x7f6c5c15c9f0 0x7f6c5c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:14:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.851+0000 7f6c6118b640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c0709e0 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0ad4e0 tx=0x7f6c4c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9d3a69da59dee4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.851+0000 7f6c6098a640 1 -- 172.21.15.60:0/2460573574 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c031880 con 0x7f6c5c0c9230 2023-10-16T14:14:07.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.851+0000 7f6c6098a640 1 -- 172.21.15.60:0/2460573574 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c0319e0 con 0x7f6c5c0c9230 2023-10-16T14:14:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.851+0000 7f6c6098a640 1 -- 172.21.15.60:0/2460573574 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c031d70 con 0x7f6c5c0c9230 2023-10-16T14:14:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 -- 172.21.15.60:0/2460573574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c0c9230 msgr2=0x7f6c5c0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c0709e0 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0ad4e0 tx=0x7f6c4c02b920 comp rx=0 tx=0).stop 2023-10-16T14:14:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 -- 172.21.15.60:0/2460573574 shutdown_connections 2023-10-16T14:14:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c070f20 0x7f6c5c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 --2- 172.21.15.60:0/2460573574 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c0709e0 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 --2- 172.21.15.60:0/2460573574 >> v2:172.21.15.144:3300/0 conn(0x7f6c5c15c9f0 0x7f6c5c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 -- 172.21.15.60:0/2460573574 >> 172.21.15.60:0/2460573574 conn(0x7f6c5c0ce6a0 msgr2=0x7f6c5c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 -- 172.21.15.60:0/2460573574 shutdown_connections 2023-10-16T14:14:07.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.852+0000 7f6c6298e640 1 -- 172.21.15.60:0/2460573574 wait complete. 2023-10-16T14:14:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.854+0000 7f6c6298e640 1 Processor -- start 2023-10-16T14:14:07.866 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.854+0000 7f6c6298e640 1 -- start start 2023-10-16T14:14:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6298e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 0x7f6c5c0afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6298e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6298e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c15c9f0 0x7f6c5c0c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6298e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c159340 con 0x7f6c5c070f20 2023-10-16T14:14:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6298e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0c6d00 con 0x7f6c5c0c9230 2023-10-16T14:14:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6298e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c15b2f0 con 0x7f6c5c15c9f0 2023-10-16T14:14:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6198c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 0x7f6c5c0afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6198c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 0x7f6c5c0afbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48192/0 (socket says 172.21.15.60:48192) 2023-10-16T14:14:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.855+0000 7f6c6198c640 1 -- 172.21.15.60:0/585756899 learned_addr learned my addr 172.21.15.60:0/585756899 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6218d640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c15c9f0 0x7f6c5c0c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6118b640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6198c640 1 -- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c15c9f0 msgr2=0x7f6c5c0c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6198c640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c15c9f0 0x7f6c5c0c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6198c640 1 -- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c5c0c9230 msgr2=0x7f6c5c158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:07.870 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6198c640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6198c640 1 -- 172.21.15.60:0/585756899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c0094d0 con 0x7f6c5c070f20 2023-10-16T14:14:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.856+0000 7f6c6198c640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 0x7f6c5c0afbe0 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800d7d0 tx=0x7f6c5800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.871 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.857+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004030 con 0x7f6c5c070f20 2023-10-16T14:14:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.857+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c58004190 con 0x7f6c5c070f20 2023-10-16T14:14:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.857+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004520 con 0x7f6c5c070f20 2023-10-16T14:14:07.872 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.857+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c15b010 con 0x7f6c5c070f20 2023-10-16T14:14:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.857+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c0c6500 con 0x7f6c5c070f20 2023-10-16T14:14:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.858+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c04e9b0 con 0x7f6c5c070f20 2023-10-16T14:14:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.859+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c5800b6c0 con 0x7f6c5c070f20 2023-10-16T14:14:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.860+0000 7f6c527fc640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f6c4006afc0 0x7f6c4006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.861+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f6c580898e0 con 0x7f6c5c070f20 2023-10-16T14:14:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.861+0000 7f6c6118b640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f6c4006afc0 0x7f6c4006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.862+0000 7f6c6118b640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f6c4006afc0 0x7f6c4006d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c036000 tx=0x7f6c4c03c040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:07.875 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:07.865+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c58054850 con 0x7f6c5c070f20 2023-10-16T14:14:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.064+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c5c0704c0 con 0x7f6c4006afc0 2023-10-16T14:14:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.066+0000 7f6c527fc640 1 -- 172.21.15.60:0/585756899 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6c4c031c50 con 0x7f6c4006afc0 2023-10-16T14:14:08.069 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 6 to deep-scrub 2023-10-16T14:14:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f6c4006afc0 msgr2=0x7f6c4006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f6c4006afc0 0x7f6c4006d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c036000 tx=0x7f6c4c03c040 comp rx=0 tx=0).stop 2023-10-16T14:14:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 msgr2=0x7f6c5c0afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 0x7f6c5c0afbe0 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800d7d0 tx=0x7f6c5800dc90 comp rx=0 tx=0).stop 2023-10-16T14:14:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 shutdown_connections 2023-10-16T14:14:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f6c4006afc0 0x7f6c4006d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f6c5c15c9f0 0x7f6c5c0c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c5c0c9230 0x7f6c5c158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 --2- 172.21.15.60:0/585756899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c5c070f20 0x7f6c5c0afbe0 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 >> 172.21.15.60:0/585756899 conn(0x7f6c5c0ce6a0 msgr2=0x7f6c5c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:08.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 shutdown_connections 2023-10-16T14:14:08.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.068+0000 7f6c6298e640 1 -- 172.21.15.60:0/585756899 wait complete. 2023-10-16T14:14:08.092 INFO:tasks.ceph:Scrubbing osd.7 2023-10-16T14:14:08.092 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-16T14:14:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.324+0000 7f4575444640 1 Processor -- start 2023-10-16T14:14:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.324+0000 7f4575444640 1 -- start start 2023-10-16T14:14:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f4575444640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f45700b0360 0x7f45700b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f4575444640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45700ca510 0x7f45700ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.330 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f4575444640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457015b310 con 0x7f457015ac10 2023-10-16T14:14:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f4575444640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457015f800 con 0x7f45700ca510 2023-10-16T14:14:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f4575444640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45700b0d30 con 0x7f45700b0360 2023-10-16T14:14:08.331 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f456f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45700ca510 0x7f45700ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f456f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45700ca510 0x7f45700ca920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48206/0 (socket says 172.21.15.60:48206) 2023-10-16T14:14:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.326+0000 7f456f7fe640 1 -- 172.21.15.60:0/894052901 learned_addr learned my addr 172.21.15.60:0/894052901 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f456f7fe640 1 -- 172.21.15.60:0/894052901 >> v2:172.21.15.144:3300/0 conn(0x7f45700b0360 msgr2=0x7f45700b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f4567fff640 1 --2- 172.21.15.60:0/894052901 >> v2:172.21.15.144:3300/0 conn(0x7f45700b0360 0x7f45700b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.334 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f456f7fe640 1 --2- 172.21.15.60:0/894052901 >> v2:172.21.15.144:3300/0 conn(0x7f45700b0360 0x7f45700b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f456f7fe640 1 -- 172.21.15.60:0/894052901 >> v1:172.21.15.60:6789/0 conn(0x7f457015ac10 legacy=0x7f457006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f456e7fc640 1 -- 172.21.15.60:0/894052901 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21775874 0 0) 0x7f457015b310 con 0x7f457015ac10 2023-10-16T14:14:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f456f7fe640 1 -- 172.21.15.60:0/894052901 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457006f640 con 0x7f45700ca510 2023-10-16T14:14:08.335 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.327+0000 7f4567fff640 1 --2- 172.21.15.60:0/894052901 >> v2:172.21.15.144:3300/0 conn(0x7f45700b0360 0x7f45700b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.328+0000 7f456f7fe640 1 --2- 172.21.15.60:0/894052901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f45700ca920 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f45700ae2b0 tx=0x7f456802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d68d1e503b737ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.329+0000 7f456e7fc640 1 -- 172.21.15.60:0/894052901 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456802ca80 con 0x7f45700ca510 2023-10-16T14:14:08.336 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.329+0000 7f456e7fc640 1 -- 172.21.15.60:0/894052901 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456802cbe0 con 0x7f45700ca510 2023-10-16T14:14:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.329+0000 7f456e7fc640 1 -- 172.21.15.60:0/894052901 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45680343f0 con 0x7f45700ca510 2023-10-16T14:14:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.329+0000 7f4575444640 1 -- 172.21.15.60:0/894052901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 msgr2=0x7f45700ca920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.329+0000 7f4575444640 1 --2- 172.21.15.60:0/894052901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f45700ca920 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f45700ae2b0 tx=0x7f456802bee0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.337 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.330+0000 7f4575444640 1 -- 172.21.15.60:0/894052901 shutdown_connections 2023-10-16T14:14:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.330+0000 7f4575444640 1 --2- 172.21.15.60:0/894052901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f45700ca920 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.330+0000 7f4575444640 1 --2- 172.21.15.60:0/894052901 >> v2:172.21.15.144:3300/0 conn(0x7f45700b0360 0x7f45700b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.330+0000 7f4575444640 1 -- 172.21.15.60:0/894052901 >> 172.21.15.60:0/894052901 conn(0x7f45700cb360 msgr2=0x7f45700cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:08.338 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.330+0000 7f4575444640 1 -- 172.21.15.60:0/894052901 shutdown_connections 2023-10-16T14:14:08.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.330+0000 7f4575444640 1 -- 172.21.15.60:0/894052901 wait complete. 2023-10-16T14:14:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.332+0000 7f4575444640 1 Processor -- start 2023-10-16T14:14:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.332+0000 7f4575444640 1 -- start start 2023-10-16T14:14:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4575444640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45700b0360 0x7f45700cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4575444640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f457006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4575444640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f457015ac10 0x7f457006b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4575444640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701f2340 con 0x7f45700ca510 2023-10-16T14:14:08.342 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4575444640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701f2600 con 0x7f45700b0360 2023-10-16T14:14:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4575444640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45701f28e0 con 0x7f457015ac10 2023-10-16T14:14:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f457006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f457006afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48222/0 (socket says 172.21.15.60:48222) 2023-10-16T14:14:08.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 -- 172.21.15.60:0/3738843416 learned_addr learned my addr 172.21.15.60:0/3738843416 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 -- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f457015ac10 msgr2=0x7f457006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f457015ac10 0x7f457006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 -- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45700b0360 msgr2=0x7f45700cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45700b0360 0x7f45700cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.333+0000 7f4567fff640 1 -- 172.21.15.60:0/3738843416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4568009b30 con 0x7f45700ca510 2023-10-16T14:14:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.334+0000 7f4567fff640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f457006afc0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f455800c7f0 tx=0x7f455800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.334+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558007820 con 0x7f45700ca510 2023-10-16T14:14:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.334+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558007980 con 0x7f45700ca510 2023-10-16T14:14:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.334+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558007cf0 con 0x7f45700ca510 2023-10-16T14:14:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.334+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45701f2bc0 con 0x7f45700ca510 2023-10-16T14:14:08.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.334+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45700cc4b0 con 0x7f45700ca510 2023-10-16T14:14:08.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.335+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45701f2770 con 0x7f45700ca510 2023-10-16T14:14:08.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.337+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f45580113f0 con 0x7f45700ca510 2023-10-16T14:14:08.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.337+0000 7f45677fe640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f455006af90 0x7f455006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.337+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f4558089b50 con 0x7f45700ca510 2023-10-16T14:14:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.338+0000 7f45677fe640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f45500714d0 0x7f45500738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.338+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 --> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4550073fa0 con 0x7f45500714d0 2023-10-16T14:14:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.338+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=292) v2 ==== 24+0+0 (secure 0 0 0) 0x7f455808e2a0 con 0x7f45700ca510 2023-10-16T14:14:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.338+0000 7f456effd640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f455006af90 0x7f455006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.338+0000 7f456f7fe640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f45500714d0 0x7f45500738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.350 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.339+0000 7f456f7fe640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f45500714d0 0x7f45500738f0 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.339+0000 7f456effd640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f455006af90 0x7f455006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f457006bdf0 tx=0x7f4560007910 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.351 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.342+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== osd.7 v2:172.21.15.144:6808/3315484258 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f4550073fa0 con 0x7f45500714d0 2023-10-16T14:14:08.359 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.356+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 --> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f45701f2a50 con 0x7f45500714d0 2023-10-16T14:14:08.360 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.356+0000 7f45677fe640 1 -- 172.21.15.60:0/3738843416 <== osd.7 v2:172.21.15.144:6808/3315484258 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f45701f2a50 con 0x7f45500714d0 2023-10-16T14:14:08.360 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-10-16T14:14:08.361 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T14:14:08.361 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-10-16T14:14:08.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f45500714d0 msgr2=0x7f45500738f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.362 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6808/3315484258,v1:172.21.15.144:6809/3315484258] conn(0x7f45500714d0 0x7f45500738f0 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f455006af90 msgr2=0x7f455006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f455006af90 0x7f455006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f457006bdf0 tx=0x7f4560007910 comp rx=0 tx=0).stop 2023-10-16T14:14:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 msgr2=0x7f457006afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.363 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 --2- 172.21.15.60:0/3738843416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45700ca510 0x7f457006afc0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f455800c7f0 tx=0x7f455800ccb0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f456f7fe640 1 -- 172.21.15.60:0/3738843416 reap_dead start 2023-10-16T14:14:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 shutdown_connections 2023-10-16T14:14:08.364 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 >> 172.21.15.60:0/3738843416 conn(0x7f45700cb360 msgr2=0x7f45700ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:08.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.357+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 shutdown_connections 2023-10-16T14:14:08.365 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.358+0000 7f4575444640 1 -- 172.21.15.60:0/3738843416 wait complete. 2023-10-16T14:14:08.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-16T14:14:08.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.605+0000 7fd07694e640 1 Processor -- start 2023-10-16T14:14:08.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.605+0000 7fd07694e640 1 -- start start 2023-10-16T14:14:08.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07694e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd070065ab0 0x7fd0700c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07694e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700acd70 con 0x7fd0700ce280 2023-10-16T14:14:08.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07694e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700ac4a0 con 0x7fd07006d220 2023-10-16T14:14:08.611 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07694e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd070065f40 con 0x7fd070065ab0 2023-10-16T14:14:08.612 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07514b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd070065ab0 0x7fd0700c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07514b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd070065ab0 0x7fd0700c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48232/0 (socket says 172.21.15.60:48232) 2023-10-16T14:14:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.607+0000 7fd07514b640 1 -- 172.21.15.60:0/2198913631 learned_addr learned my addr 172.21.15.60:0/2198913631 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.608+0000 7fd07514b640 1 -- 172.21.15.60:0/2198913631 >> v1:172.21.15.144:6789/0 conn(0x7fd07006d220 legacy=0x7fd0700cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:08.613 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.608+0000 7fd07514b640 1 -- 172.21.15.60:0/2198913631 >> v1:172.21.15.60:6789/0 conn(0x7fd0700ce280 legacy=0x7fd0700ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.608+0000 7fd07514b640 1 -- 172.21.15.60:0/2198913631 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0700ac720 con 0x7fd070065ab0 2023-10-16T14:14:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.609+0000 7fd07514b640 1 --2- 172.21.15.60:0/2198913631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd070065ab0 0x7fd0700c95e0 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fd060009870 tx=0x7fd06002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d076203f34a8f587 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07494a640 1 -- 172.21.15.60:0/2198913631 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060031870 con 0x7fd070065ab0 2023-10-16T14:14:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07494a640 1 -- 172.21.15.60:0/2198913631 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0600319d0 con 0x7fd070065ab0 2023-10-16T14:14:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07494a640 1 -- 172.21.15.60:0/2198913631 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060031d60 con 0x7fd070065ab0 2023-10-16T14:14:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 -- 172.21.15.60:0/2198913631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd070065ab0 msgr2=0x7fd0700c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 --2- 172.21.15.60:0/2198913631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd070065ab0 0x7fd0700c95e0 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fd060009870 tx=0x7fd06002b910 comp rx=0 tx=0).stop 2023-10-16T14:14:08.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 -- 172.21.15.60:0/2198913631 shutdown_connections 2023-10-16T14:14:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 --2- 172.21.15.60:0/2198913631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd070065ab0 0x7fd0700c95e0 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 -- 172.21.15.60:0/2198913631 >> 172.21.15.60:0/2198913631 conn(0x7fd0700b0a80 msgr2=0x7fd0700b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:08.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 -- 172.21.15.60:0/2198913631 shutdown_connections 2023-10-16T14:14:08.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.610+0000 7fd07694e640 1 -- 172.21.15.60:0/2198913631 wait complete. 2023-10-16T14:14:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.613+0000 7fd07694e640 1 Processor -- start 2023-10-16T14:14:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.613+0000 7fd07694e640 1 -- start start 2023-10-16T14:14:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07694e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 0x7fd0700c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07694e640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0700ce280 0x7fd0701f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07694e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0700c75f0 0x7fd0701f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.622 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07694e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700c8710 con 0x7fd07006d220 2023-10-16T14:14:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07694e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07009c900 con 0x7fd0700c75f0 2023-10-16T14:14:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07694e640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700c7b00 con 0x7fd0700ce280 2023-10-16T14:14:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07594c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 0x7fd0700c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.614+0000 7fd07594c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 0x7fd0700c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48240/0 (socket says 172.21.15.60:48240) 2023-10-16T14:14:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 -- 172.21.15.60:0/781661378 learned_addr learned my addr 172.21.15.60:0/781661378 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 -- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0700ce280 msgr2=0x7fd0701f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07514b640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0700ce280 0x7fd0701f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07614d640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0700c75f0 0x7fd0701f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0700ce280 0x7fd0701f5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 -- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0700c75f0 msgr2=0x7fd0701f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0700c75f0 0x7fd0701f61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 -- 172.21.15.60:0/781661378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0600094d0 con 0x7fd07006d220 2023-10-16T14:14:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.615+0000 7fd07594c640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 0x7fd0700c70b0 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c7c0 tx=0x7fd06c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.616+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c00aa50 con 0x7fd07006d220 2023-10-16T14:14:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.616+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c00abb0 con 0x7fd07006d220 2023-10-16T14:14:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.616+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c012420 con 0x7fd07006d220 2023-10-16T14:14:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.616+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0700c7de0 con 0x7fd07006d220 2023-10-16T14:14:08.628 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.616+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0700c83a0 con 0x7fd07006d220 2023-10-16T14:14:08.629 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.617+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd070161850 con 0x7fd07006d220 2023-10-16T14:14:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.618+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd06c012580 con 0x7fd07006d220 2023-10-16T14:14:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.619+0000 7fd0667fc640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd05406af90 0x7fd05406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.619+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fd06c089730 con 0x7fd07006d220 2023-10-16T14:14:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.620+0000 7fd07514b640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd05406af90 0x7fd05406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.621+0000 7fd07514b640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd05406af90 0x7fd05406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd060036000 tx=0x7fd06003c040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.623+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd06c0546a0 con 0x7fd07006d220 2023-10-16T14:14:08.822 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.818+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0700aca00 con 0x7fd05406af90 2023-10-16T14:14:08.823 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.820+0000 7fd0667fc640 1 -- 172.21.15.60:0/781661378 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd060031cc0 con 0x7fd05406af90 2023-10-16T14:14:08.823 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 7 to deep-scrub 2023-10-16T14:14:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd05406af90 msgr2=0x7fd05406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd05406af90 0x7fd05406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd060036000 tx=0x7fd06003c040 comp rx=0 tx=0).stop 2023-10-16T14:14:08.826 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 msgr2=0x7fd0700c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:08.827 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 0x7fd0700c70b0 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c7c0 tx=0x7fd06c00cc80 comp rx=0 tx=0).stop 2023-10-16T14:14:08.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 shutdown_connections 2023-10-16T14:14:08.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fd05406af90 0x7fd05406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fd0700ce280 0x7fd0701f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.828 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0700c75f0 0x7fd0701f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.822+0000 7fd07694e640 1 --2- 172.21.15.60:0/781661378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd07006d220 0x7fd0700c70b0 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:08.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.823+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 >> 172.21.15.60:0/781661378 conn(0x7fd0700b0a80 msgr2=0x7fd07015c390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:08.829 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.823+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 shutdown_connections 2023-10-16T14:14:08.830 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:08.823+0000 7fd07694e640 1 -- 172.21.15.60:0/781661378 wait complete. 2023-10-16T14:14:08.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:14:09.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.072+0000 7fcc8f116640 1 Processor -- start 2023-10-16T14:14:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.072+0000 7fcc8f116640 1 -- start start 2023-10-16T14:14:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.073+0000 7fcc8f116640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcc88160800 0x7fcc880ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.073+0000 7fcc8f116640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8806e170 con 0x7fcc88160420 2023-10-16T14:14:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8f116640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc880cd460 con 0x7fcc8806b4c0 2023-10-16T14:14:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8f116640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8806e3c0 con 0x7fcc88160800 2023-10-16T14:14:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8ceb2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcc88160420 0x7fcc8806adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54246/0 (socket says 172.21.15.60:54246) 2023-10-16T14:14:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8ceb2640 1 -- 172.21.15.60:0/3781969977 learned_addr learned my addr 172.21.15.60:0/3781969977 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8d6b3640 1 --2- 172.21.15.60:0/3781969977 >> v2:172.21.15.60:3300/0 conn(0x7fcc88160800 0x7fcc880ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc877fe640 1 -- 172.21.15.60:0/3781969977 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984155130 0 0) 0x7fcc8806e170 con 0x7fcc88160420 2023-10-16T14:14:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3781969977 >> v1:172.21.15.144:6789/0 conn(0x7fcc8806b4c0 legacy=0x7fcc88070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3781969977 >> v1:172.21.15.60:6789/0 conn(0x7fcc88160420 legacy=0x7fcc8806adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.074+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3781969977 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8806e6a0 con 0x7fcc88160800 2023-10-16T14:14:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.075+0000 7fcc8d6b3640 1 --2- 172.21.15.60:0/3781969977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880ae170 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc8002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5649eb5fc4d188dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.076+0000 7fcc877fe640 1 -- 172.21.15.60:0/3781969977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8002cc60 con 0x7fcc88160800 2023-10-16T14:14:09.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.076+0000 7fcc877fe640 1 -- 172.21.15.60:0/3781969977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8002cdc0 con 0x7fcc88160800 2023-10-16T14:14:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.076+0000 7fcc877fe640 1 -- 172.21.15.60:0/3781969977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80034660 con 0x7fcc88160800 2023-10-16T14:14:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.076+0000 7fcc8f116640 1 -- 172.21.15.60:0/3781969977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 msgr2=0x7fcc880ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:09.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.076+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3781969977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880ae170 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc8002beb0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.077+0000 7fcc8f116640 1 -- 172.21.15.60:0/3781969977 shutdown_connections 2023-10-16T14:14:09.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.077+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3781969977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880ae170 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.077+0000 7fcc8f116640 1 -- 172.21.15.60:0/3781969977 >> 172.21.15.60:0/3781969977 conn(0x7fcc880b0d30 msgr2=0x7fcc880b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:09.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.077+0000 7fcc8f116640 1 -- 172.21.15.60:0/3781969977 shutdown_connections 2023-10-16T14:14:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.077+0000 7fcc8f116640 1 -- 172.21.15.60:0/3781969977 wait complete. 2023-10-16T14:14:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.078+0000 7fcc8f116640 1 Processor -- start 2023-10-16T14:14:09.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.079+0000 7fcc8f116640 1 -- start start 2023-10-16T14:14:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8f116640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc8806b4c0 0x7fcc881f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8f116640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc88160420 0x7fcc881f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8f116640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:09.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8f116640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881f5810 con 0x7fcc88160800 2023-10-16T14:14:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8f116640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881f33f0 con 0x7fcc88160420 2023-10-16T14:14:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8f116640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8809d7e0 con 0x7fcc8806b4c0 2023-10-16T14:14:09.090 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8d6b3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8d6b3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880af5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51254/0 (socket says 172.21.15.60:51254) 2023-10-16T14:14:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3584037632 learned_addr learned my addr 172.21.15.60:0/3584037632 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc87fff640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc88160420 0x7fcc881f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:09.091 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.080+0000 7fcc8ceb2640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc8806b4c0 0x7fcc881f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc8806b4c0 msgr2=0x7fcc881f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8d6b3640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc8806b4c0 0x7fcc881f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc88160420 msgr2=0x7fcc881f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:09.092 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8d6b3640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc88160420 0x7fcc881f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8d6b3640 1 -- 172.21.15.60:0/3584037632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc80009a70 con 0x7fcc88160800 2023-10-16T14:14:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc87fff640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc88160420 0x7fcc881f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T14:14:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8d6b3640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880af5d0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:09.093 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8003e610 con 0x7fcc88160800 2023-10-16T14:14:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8003e790 con 0x7fcc88160800 2023-10-16T14:14:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8809dac0 con 0x7fcc88160800 2023-10-16T14:14:09.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.081+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8003eb40 con 0x7fcc88160800 2023-10-16T14:14:09.095 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.082+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8809e080 con 0x7fcc88160800 2023-10-16T14:14:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.083+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8804e160 con 0x7fcc88160800 2023-10-16T14:14:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.084+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc8003b050 con 0x7fcc88160800 2023-10-16T14:14:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.085+0000 7fcc857fa640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fcc6c06afc0 0x7fcc6c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.085+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7fcc800abf40 con 0x7fcc88160800 2023-10-16T14:14:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.085+0000 7fcc8ceb2640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fcc6c06afc0 0x7fcc6c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.087+0000 7fcc8ceb2640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fcc6c06afc0 0x7fcc6c06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000f60 tx=0x7fcc7c007b50 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.089+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc80076ec0 con 0x7fcc88160800 2023-10-16T14:14:09.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.282+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc880cd0f0 con 0x7fcc6c06afc0 2023-10-16T14:14:09.293 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.289+0000 7fcc857fa640 1 -- 172.21.15.60:0/3584037632 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48138 (secure 0 0 0) 0x7fcc7c007170 con 0x7fcc6c06afc0 2023-10-16T14:14:09.294 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:09.295 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:14:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fcc6c06afc0 msgr2=0x7fcc6c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fcc6c06afc0 0x7fcc6c06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c000f60 tx=0x7fcc7c007b50 comp rx=0 tx=0).stop 2023-10-16T14:14:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 msgr2=0x7fcc880af5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880af5d0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc80004a40 comp rx=0 tx=0).stop 2023-10-16T14:14:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 shutdown_connections 2023-10-16T14:14:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7fcc6c06afc0 0x7fcc6c06d480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc88160800 0x7fcc880af5d0 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc88160420 0x7fcc881f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 --2- 172.21.15.60:0/3584037632 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7fcc8806b4c0 0x7fcc881f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 >> 172.21.15.60:0/3584037632 conn(0x7fcc880b0d30 msgr2=0x7fcc880c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 shutdown_connections 2023-10-16T14:14:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:09.292+0000 7fcc8f116640 1 -- 172.21.15.60:0/3584037632 wait complete. 2023-10-16T14:14:09.314 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-16T14:14:09.097972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":472344,"kb_used_data":29992,"kb_used_omap":12,"kb_used_meta":442227,"kb_avail":754502376,"statfs":{"total":773094113280,"available":772610433024,"internally_reserved":0,"allocated":30711808,"data_stored":15702224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":452841016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"1.0","version":"289'2843","reported_seq":4005,"reported_epoch":292,"state":"active+clean","last_fresh":"2023-10-16T14:13:56.826895+0000","last_change":"2023-10-16T12:33:20.069065+0000","last_active":"2023-10-16T14:13:56.826895+0000","last_peered":"2023-10-16T14:13:56.826895+0000","last_clean":"2023-10-16T14:13:56.826895+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:13:56.826895+0000","last_undegraded":"2023-10-16T14:13:56.826895+0000","last_fullsized":"2023-10-16T14:13:56.826895+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:33:18.519540+0000","last_clean_scrub_stamp":"2023-10-16T12:33:18.519540+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:41.729354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2843,"ondisk_log_size":2843,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49256,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322584,"statfs":{"total":96636764160,"available":96586326016,"internally_reserved":0,"allocated":4284416,"data_stored":2405554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2889999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.331,"5min":1.4139999999999999,"15min":1.4079999999999999},"min":{"1min":0.76400000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.133,"5min":6.391,"15min":6.391},"last":0.88100000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.212,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.629},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.238,"15min":1.252},"min":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.982,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.282}]},{"osd":2,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.228,"15min":1.258},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.736,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.745,"15min":0.745},"max":{"1min":2.0529999999999999,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.147}]},{"osd":3,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3660000000000001,"15min":1.403},"min":{"1min":0.86799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.853,"5min":2.0790000000000002,"15min":2.097},"last":1.458},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.823,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.375}]},{"osd":4,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.256,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.593,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4630000000000001,"15min":1.4239999999999999},"min":{"1min":1.0089999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8620000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.6870000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.448,"15min":1.397},"min":{"1min":0.879,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.0880000000000001,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.206},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.5589999999999999,"15min":1.5429999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3159999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.75}]},{"osd":6,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.6579999999999999,"15min":1.641},"min":{"1min":1.0680000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.4500000000000002,"5min":3.29,"15min":3.29},"last":1.8029999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5529999999999999,"15min":1.496},"min":{"1min":0.85999999999999999,"5min":0.75,"15min":0.75},"max":{"1min":2.0419999999999998,"5min":3.524,"15min":3.524},"last":1.542}]}]},{"osd":6,"up_from":14,"seq":60129543338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48672,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323168,"statfs":{"total":96636764160,"available":96586924032,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2330000000000001,"15min":1.2989999999999999},"min":{"1min":0.92300000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6200000000000001,"5min":1.788,"15min":1.887},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.266,"5min":1.2909999999999999,"15min":1.3420000000000001},"min":{"1min":0.85499999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7509999999999999,"5min":1.847,"15min":1.9390000000000001},"last":0.93000000000000005}]},{"osd":1,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.157,"15min":1.204},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6819999999999999,"5min":1.8360000000000001,"15min":2.008},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.1970000000000001},"min":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.64200000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.77},"last":1.2310000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.86899999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.294,"15min":1.3029999999999999},"min":{"1min":0.76400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.3049999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3240000000000001,"15min":1.339},"min":{"1min":0.95199999999999996,"5min":0.79300000000000004,"15min":0.70499999999999996},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.625},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.1120000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4219999999999999,"15min":1.4119999999999999},"min":{"1min":0.90700000000000003,"5min":0.77300000000000002,"15min":0.51400000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.96},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5580000000000001,"15min":1.548},"min":{"1min":0.88900000000000001,"5min":0.86299999999999999,"15min":0.78400000000000003},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.1179999999999999},"last":1.4139999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.427,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.964,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.502},{"interface":"front","average":{"1min":1.617,"5min":1.526,"15min":1.496},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.64400000000000002},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.8720000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.5309999999999999,"15min":1.512},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.68600000000000005},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.716},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5589999999999999,"15min":1.54},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.6120000000000001},"last":1.6020000000000001}]}]},{"osd":5,"up_from":14,"seq":60129543339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52832,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319008,"statfs":{"total":96636764160,"available":96582664192,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0900000000000001,"15min":1.181},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":2.8330000000000002},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.2070000000000001,"15min":1.2170000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9830000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.2549999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.804,"5min":1.871,"15min":1.8779999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.153,"15min":1.23},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.3799999999999999,"5min":1.931,"15min":2.5840000000000001},"last":0.98899999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.2569999999999999,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.65400000000000003,"15min":0.63},"max":{"1min":1.591,"5min":2.024,"15min":2.8119999999999998},"last":1.708},{"interface":"front","average":{"1min":1.119,"5min":1.1950000000000001,"15min":1.2849999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.492,"5min":1.8600000000000001,"15min":2.6259999999999999},"last":1.163}]},{"osd":3,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.387,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.365},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.361,"15min":1.419},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.794,"5min":1.8759999999999999,"15min":2.75},"last":1.5920000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2869999999999999,"15min":1.302},"min":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.982,"5min":1.982,"15min":2.774},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.478,"15min":1.4470000000000001},"min":{"1min":1.2490000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.3330000000000002},"last":1.661}]},{"osd":6,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4550000000000001,"15min":1.4399999999999999},"min":{"1min":1.028,"5min":0.59399999999999997,"15min":0.55900000000000005},"max":{"1min":2.081,"5min":2.081,"15min":2.6720000000000002},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.89100000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7010000000000001},"last":1.6399999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.593,"15min":1.5580000000000001},"min":{"1min":1.278,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.4609999999999999},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":1.1579999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.2010000000000001,"5min":2.238,"15min":2.8719999999999999},"last":1.853}]}]},{"osd":4,"up_from":14,"seq":60129543339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88864,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282976,"statfs":{"total":96636764160,"available":96545767424,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.1930000000000001},"min":{"1min":0.85899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.575,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.159,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5629999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":1.367}]},{"osd":1,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1439999999999999,"15min":1.2270000000000001},"min":{"1min":0.73099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.27},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7210000000000001,"5min":1.792,"15min":2.0499999999999998},"last":1.196}]},{"osd":2,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2529999999999999,"15min":1.302},"min":{"1min":0.81599999999999995,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.4299999999999999,"5min":1.714,"15min":1.865},"last":1.097},{"interface":"front","average":{"1min":1.381,"5min":1.365,"15min":1.3779999999999999},"min":{"1min":1.105,"5min":0.87,"15min":0.87},"max":{"1min":1.77,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.5289999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.292,"15min":1.2989999999999999},"min":{"1min":0.996,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9650000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.425},"min":{"1min":1.1459999999999999,"5min":0.92800000000000005,"15min":0.86799999999999999},"max":{"1min":1.887,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.284}]},{"osd":5,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4019999999999999,"15min":1.3540000000000001},"min":{"1min":0.997,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.738,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.583},{"interface":"front","average":{"1min":1.427,"5min":1.415,"15min":1.3600000000000001},"min":{"1min":0.97499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0659999999999998,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.4379999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4390000000000001,"15min":1.3999999999999999},"min":{"1min":1.091,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.649,"5min":1.5760000000000001,"15min":1.5289999999999999},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":1.9790000000000001,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.333}]},{"osd":7,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.591,"15min":1.5269999999999999},"min":{"1min":1.22,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.1219999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.657,"15min":1.645},"min":{"1min":1.335,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.3660000000000001},"last":1.411}]}]},{"osd":3,"up_from":14,"seq":60129543339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91000000000000003,"15min":0.93799999999999994},"min":{"1min":0.53900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.54,"5min":1.595,"15min":1.595},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.05,"5min":1.105,"15min":1.1899999999999999},"min":{"1min":0.57399999999999995,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.5529999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.5720000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0229999999999999,"15min":1.0700000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.6319999999999999,"5min":1.6870000000000001,"15min":1.855},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0620000000000001,"15min":1.105},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.9890000000000001,"5min":2.125,"15min":2.125},"last":1.472}]},{"osd":2,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0569999999999999,"15min":1.1739999999999999},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.544,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.4039999999999999},"min":{"1min":0.93200000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6739999999999999,"5min":1.909,"15min":1.9319999999999999},"last":1.393}]},{"osd":4,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4390000000000001,"15min":1.4870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.67000000000000004},"max":{"1min":1.891,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.7030000000000001},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3859999999999999,"15min":1.4470000000000001},"min":{"1min":0.80500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6599999999999999,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.778}]},{"osd":5,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4019999999999999,"15min":1.4730000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0579999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4950000000000001,"15min":1.5409999999999999},"min":{"1min":0.88700000000000001,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9339999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.526}]},{"osd":6,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.508},"min":{"1min":0.96599999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.8300000000000001,"5min":2.335,"15min":2.4329999999999998},"last":1.8440000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.4279999999999999,"15min":1.486},"min":{"1min":0.752,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9279999999999999,"5min":2.194,"15min":2.4180000000000001},"last":1.885}]},{"osd":7,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5289999999999999,"15min":1.5840000000000001},"min":{"1min":0.875,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1320000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.605},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.611,"15min":1.665},"min":{"1min":0.91300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.089,"5min":2.569,"15min":2.6070000000000002},"last":2.2080000000000002}]}]},{"osd":2,"up_from":14,"seq":60129543337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.93200000000000005,"15min":0.94999999999999996},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.47599999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.7090000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1799999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0840000000000001,"15min":1.137},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5580000000000001,"5min":1.847,"15min":1.847},"last":1.2270000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.125,"15min":1.2210000000000001},"min":{"1min":0.67100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.415,"5min":1.784,"15min":1.9530000000000001},"last":1.3640000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.05,"15min":1.0980000000000001},"min":{"1min":0.56100000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3759999999999999,"5min":1.9179999999999999,"15min":2.1299999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.252,"15min":1.3460000000000001},"min":{"1min":0.623,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.1480000000000001,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":1.3160000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.4890000000000001},"min":{"1min":0.78200000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.298,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.466},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.4790000000000001},"min":{"1min":0.88900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.4970000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.51,"15min":1.548},"min":{"1min":0.89000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.462,"15min":1.4650000000000001},"min":{"1min":0.80600000000000005,"5min":0.627,"15min":0.627},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.127}]},{"osd":6,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.524,"15min":1.52},"min":{"1min":0.91300000000000003,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.5580000000000001,"15min":1.605},"min":{"1min":0.77900000000000003,"5min":0.752,"15min":0.752},"max":{"1min":2.52,"5min":2.52,"15min":2.5569999999999999},"last":1.6519999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:13:22 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.641,"15min":1.6599999999999999},"min":{"1min":0.996,"5min":0.754,"15min":0.754},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":1.71},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.613,"15min":1.637},"min":{"1min":0.86299999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.5840000000000001}]}]},{"osd":1,"up_from":14,"seq":60129543337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":1.0109999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.4299999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.202}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":1.113},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":2.0339999999999998}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.474},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.367}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":2.105},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.835}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.752},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":1.415}]}]},{"osd":0,"up_from":14,"seq":60129543338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61732,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310108,"statfs":{"total":96636764160,"available":96573550592,"internally_reserved":0,"allocated":4284416,"data_stored":2405554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91600000000000004,"15min":0.93999999999999995},"min":{"1min":0.46200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.98999999999999999,"15min":1.1140000000000001},"min":{"1min":0.60599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9299999999999999},"last":0.73899999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0609999999999999,"15min":1.149},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.768,"5min":1.768,"15min":2.0569999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0880000000000001,"15min":1.1890000000000001},"min":{"1min":0.72299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":1.3260000000000001,"5min":1.78,"15min":1.893},"last":0.876}]},{"osd":3,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0269999999999999,"15min":1.125},"min":{"1min":0.66700000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2290000000000001,"5min":1.7869999999999999,"15min":1.8520000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2350000000000001,"15min":1.341},"min":{"1min":0.79500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5880000000000001,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.1919999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.343},"min":{"1min":0.95399999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5900000000000001,"5min":1.923,"15min":2.0070000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3979999999999999,"15min":1.4910000000000001},"min":{"1min":0.996,"5min":0.748,"15min":0.748},"max":{"1min":2.0169999999999999,"5min":2.1070000000000002,"15min":2.601},"last":1.474}]},{"osd":5,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.4810000000000001},"min":{"1min":0.80100000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.081,"5min":2.089,"15min":2.3279999999999998},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.337,"5min":1.3720000000000001,"15min":1.45},"min":{"1min":0.89700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.754,"5min":2.0169999999999999,"15min":2.2829999999999999},"last":1.1279999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5049999999999999,"15min":1.5880000000000001},"min":{"1min":0.94099999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0990000000000002,"5min":2.2730000000000001,"15min":2.669},"last":1.538},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.494,"15min":1.5720000000000001},"min":{"1min":0.86099999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.893,"5min":2.327,"15min":2.3769999999999998},"last":1.2829999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5840000000000001,"15min":1.677},"min":{"1min":0.93899999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.5270000000000001},"last":1.444},{"interface":"front","average":{"1min":1.478,"5min":1.5249999999999999,"15min":1.6120000000000001},"min":{"1min":1.119,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.893,"5min":2.3679999999999999,"15min":2.464},"last":1.407}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:14:09.318 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-16T12:33:18.519540+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=12, tm_min=33, tm_sec=18, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=14, tm_min=14, tm_sec=1, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T14:14:09.318 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T14:14:29.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T14:14:29.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:14:29.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:14:29.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.541+0000 7f325c9c0640 1 Processor -- start 2023-10-16T14:14:29.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.541+0000 7f325c9c0640 1 -- start start 2023-10-16T14:14:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f325c9c0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f325815ae80 0x7f325815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f325c9c0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325815ed60 con 0x7f32580ce390 2023-10-16T14:14:29.546 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f325c9c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32580707e0 con 0x7f325815aaa0 2023-10-16T14:14:29.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f325c9c0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325806b810 con 0x7f325815ae80 2023-10-16T14:14:29.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f324ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f325815ae80 0x7f325815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:29.547 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f3256d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f32580ce390 0x7f3258070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45214/0 (socket says 172.21.15.60:45214) 2023-10-16T14:14:29.548 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.543+0000 7f3256d9d640 1 -- 172.21.15.60:0/35322099 learned_addr learned my addr 172.21.15.60:0/35322099 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:29.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.544+0000 7f324ffff640 1 -- 172.21.15.60:0/35322099 >> v1:172.21.15.144:6789/0 conn(0x7f325815aaa0 legacy=0x7f32580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:29.549 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.544+0000 7f324ffff640 1 -- 172.21.15.60:0/35322099 >> v1:172.21.15.60:6789/0 conn(0x7f32580ce390 legacy=0x7f3258070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.544+0000 7f324ffff640 1 -- 172.21.15.60:0/35322099 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325815ef90 con 0x7f325815ae80 2023-10-16T14:14:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.544+0000 7f3255d9b640 1 -- 172.21.15.60:0/35322099 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118104735 0 0) 0x7f325815ed60 con 0x7f32580ce390 2023-10-16T14:14:29.550 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.545+0000 7f324ffff640 1 --2- 172.21.15.60:0/35322099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815ae80 0x7f325815e820 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f32400097a0 tx=0x7f324002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5953b91629eb663b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.545+0000 7f3255d9b640 1 -- 172.21.15.60:0/35322099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240031830 con 0x7f325815ae80 2023-10-16T14:14:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.545+0000 7f3255d9b640 1 -- 172.21.15.60:0/35322099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3240031990 con 0x7f325815ae80 2023-10-16T14:14:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.545+0000 7f3255d9b640 1 -- 172.21.15.60:0/35322099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240031d20 con 0x7f325815ae80 2023-10-16T14:14:29.551 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 -- 172.21.15.60:0/35322099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815ae80 msgr2=0x7f325815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 --2- 172.21.15.60:0/35322099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815ae80 0x7f325815e820 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f32400097a0 tx=0x7f324002b8d0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 -- 172.21.15.60:0/35322099 shutdown_connections 2023-10-16T14:14:29.552 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 --2- 172.21.15.60:0/35322099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815ae80 0x7f325815e820 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 -- 172.21.15.60:0/35322099 >> 172.21.15.60:0/35322099 conn(0x7f32580cb410 msgr2=0x7f32580cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 -- 172.21.15.60:0/35322099 shutdown_connections 2023-10-16T14:14:29.553 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.546+0000 7f325c9c0640 1 -- 172.21.15.60:0/35322099 wait complete. 2023-10-16T14:14:29.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.548+0000 7f325c9c0640 1 Processor -- start 2023-10-16T14:14:29.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.548+0000 7f325c9c0640 1 -- start start 2023-10-16T14:14:29.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f325c9c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32580ce390 0x7f32581f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:29.556 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f325c9c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 0x7f32581f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f325c9c0640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325815ae80 0x7f32580c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f325c9c0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32580afcc0 con 0x7f325815aaa0 2023-10-16T14:14:29.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f325c9c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581f3ca0 con 0x7f32580ce390 2023-10-16T14:14:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f325c9c0640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32581f1830 con 0x7f325815ae80 2023-10-16T14:14:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f324ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 0x7f32581f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:29.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f324ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 0x7f32581f47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33986/0 (socket says 172.21.15.60:33986) 2023-10-16T14:14:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.549+0000 7f324ffff640 1 -- 172.21.15.60:0/3139337483 learned_addr learned my addr 172.21.15.60:0/3139337483 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f325659c640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32580ce390 0x7f32581f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324ffff640 1 -- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325815ae80 msgr2=0x7f32580c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324ffff640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325815ae80 0x7f32580c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324ffff640 1 -- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32580ce390 msgr2=0x7f32581f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324ffff640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32580ce390 0x7f32581f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324ffff640 1 -- 172.21.15.60:0/3139337483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3240009400 con 0x7f325815aaa0 2023-10-16T14:14:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324ffff640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 0x7f32581f47e0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f32400098c0 tx=0x7f32400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324003aa60 con 0x7f325815aaa0 2023-10-16T14:14:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324003abc0 con 0x7f325815aaa0 2023-10-16T14:14:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32400423f0 con 0x7f325815aaa0 2023-10-16T14:14:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.550+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32580af320 con 0x7f325815aaa0 2023-10-16T14:14:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.551+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32580af8e0 con 0x7f325815aaa0 2023-10-16T14:14:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.552+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32580afe00 con 0x7f325815aaa0 2023-10-16T14:14:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.553+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3240030380 con 0x7f325815aaa0 2023-10-16T14:14:29.563 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.553+0000 7f324effd640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f323806af90 0x7f323806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.554+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f32400ab4e0 con 0x7f325815aaa0 2023-10-16T14:14:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.554+0000 7f325659c640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f323806af90 0x7f323806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.555+0000 7f325659c640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f323806af90 0x7f323806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f32580ad9a0 tx=0x7f3248009040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:29.565 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.558+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32580afe00 con 0x7f325815aaa0 2023-10-16T14:14:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.750+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 --> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f325815f550 con 0x7f323806af90 2023-10-16T14:14:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.756+0000 7f324effd640 1 -- 172.21.15.60:0/3139337483 <== mgr.9014 v2:172.21.15.144:6832/3626350367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48162 (secure 0 0 0) 0x7f324800e7e0 con 0x7f323806af90 2023-10-16T14:14:29.760 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2023-10-16T14:14:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f323806af90 msgr2=0x7f323806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f323806af90 0x7f323806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f32580ad9a0 tx=0x7f3248009040 comp rx=0 tx=0).stop 2023-10-16T14:14:29.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 msgr2=0x7f32581f47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 0x7f32581f47e0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f32400098c0 tx=0x7f32400046c0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 shutdown_connections 2023-10-16T14:14:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f323806af90 0x7f323806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f325815ae80 0x7f32580c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f325815aaa0 0x7f32581f47e0 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 --2- 172.21.15.60:0/3139337483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32580ce390 0x7f32581f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 >> 172.21.15.60:0/3139337483 conn(0x7f32580cb410 msgr2=0x7f32580b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:29.764 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 shutdown_connections 2023-10-16T14:14:29.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:29.759+0000 7f325c9c0640 1 -- 172.21.15.60:0/3139337483 wait complete. 2023-10-16T14:14:29.781 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-16T14:14:29.104638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2844,"ondisk_log_size":2844,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":472332,"kb_used_data":29992,"kb_used_omap":12,"kb_used_meta":442227,"kb_avail":754502388,"statfs":{"total":773094113280,"available":772610445312,"internally_reserved":0,"allocated":30711808,"data_stored":15702224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":452841016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004218"},"pg_stats":[{"pgid":"1.0","version":"292'2844","reported_seq":4015,"reported_epoch":292,"state":"active+clean","last_fresh":"2023-10-16T14:14:09.795770+0000","last_change":"2023-10-16T14:14:09.795896+0000","last_active":"2023-10-16T14:14:09.795770+0000","last_peered":"2023-10-16T14:14:09.795770+0000","last_clean":"2023-10-16T14:14:09.795770+0000","last_became_active":"2023-10-16T12:33:20.067831+0000","last_became_peered":"2023-10-16T12:33:20.067831+0000","last_unstale":"2023-10-16T14:14:09.795770+0000","last_undegraded":"2023-10-16T14:14:09.795770+0000","last_fullsized":"2023-10-16T14:14:09.795770+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2844","last_scrub_stamp":"2023-10-16T14:14:09.795598+0000","last_deep_scrub":"292'2844","last_deep_scrub_stamp":"2023-10-16T14:14:09.795598+0000","last_clean_scrub_stamp":"2023-10-16T14:14:09.795598+0000","objects_scrubbed":2,"log_size":2844,"log_dups_size":0,"ondisk_log_size":2844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:26.992791+0000","scrub_duration":0.013482312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2891,"num_read_kb":6072,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2844,"ondisk_log_size":2844,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49248,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322592,"statfs":{"total":96636764160,"available":96586334208,"internally_reserved":0,"allocated":4284416,"data_stored":2405554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2889999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.4139999999999999,"15min":1.4079999999999999},"min":{"1min":0.76400000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.133,"5min":6.391,"15min":6.391},"last":1.756}]},{"osd":1,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.212,"15min":1.2350000000000001},"min":{"1min":0.81799999999999995,"5min":0.70999999999999996,"15min":0.69599999999999995},"max":{"1min":1.7809999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.238,"15min":1.252},"min":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.64400000000000002},"max":{"1min":1.982,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.294}]},{"osd":2,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.228,"15min":1.258},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.736,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.01},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.361},"min":{"1min":0.84999999999999998,"5min":0.745,"15min":0.745},"max":{"1min":2.0529999999999999,"5min":3.4289999999999998,"15min":3.4289999999999998},"last":1.425}]},{"osd":3,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3660000000000001,"15min":1.403},"min":{"1min":0.86799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.853,"5min":2.0790000000000002,"15min":2.097},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.823,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.5660000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.256,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.593,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.129},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4630000000000001,"15min":1.4239999999999999},"min":{"1min":1.0089999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8620000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.5049999999999999}]},{"osd":5,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.448,"15min":1.397},"min":{"1min":0.879,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.0880000000000001,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.5589999999999999,"15min":1.5429999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3159999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.359}]},{"osd":6,"last update":"Mon Oct 16 14:13:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.6579999999999999,"15min":1.641},"min":{"1min":1.0680000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.4500000000000002,"5min":3.29,"15min":3.29},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5529999999999999,"15min":1.496},"min":{"1min":0.85999999999999999,"5min":0.75,"15min":0.75},"max":{"1min":2.0419999999999998,"5min":3.524,"15min":3.524},"last":1.659}]}]},{"osd":6,"up_from":14,"seq":60129543342,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48672,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323168,"statfs":{"total":96636764160,"available":96586924032,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2330000000000001,"15min":1.2989999999999999},"min":{"1min":0.92300000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6200000000000001,"5min":1.788,"15min":1.887},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.2909999999999999,"15min":1.3420000000000001},"min":{"1min":0.85499999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.7509999999999999,"5min":1.847,"15min":1.9390000000000001},"last":1.288}]},{"osd":1,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.157,"15min":1.204},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6819999999999999,"5min":1.8360000000000001,"15min":2.008},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1719999999999999,"15min":1.1970000000000001},"min":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.64200000000000002},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.77},"last":1.0660000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.86899999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.294,"15min":1.3029999999999999},"min":{"1min":0.76400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.2130000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3240000000000001,"15min":1.339},"min":{"1min":0.95199999999999996,"5min":0.79300000000000004,"15min":0.70499999999999996},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.681},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.472}]},{"osd":4,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4219999999999999,"15min":1.4119999999999999},"min":{"1min":0.90700000000000003,"5min":0.77300000000000002,"15min":0.51400000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5580000000000001,"15min":1.548},"min":{"1min":0.88900000000000001,"5min":0.86299999999999999,"15min":0.78400000000000003},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.1179999999999999},"last":1.587}]},{"osd":5,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.427,"15min":1.3799999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.964,"5min":2.1019999999999999,"15min":2.1549999999999998},"last":1.625},{"interface":"front","average":{"1min":1.617,"5min":1.526,"15min":1.496},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.64400000000000002},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.8500000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.5309999999999999,"15min":1.512},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.68600000000000005},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.8859999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5589999999999999,"15min":1.54},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.6120000000000001},"last":1.728}]}]},{"osd":5,"up_from":14,"seq":60129543343,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52832,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319008,"statfs":{"total":96636764160,"available":96582664192,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0900000000000001,"15min":1.181},"min":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":2.8330000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.2070000000000001,"15min":1.2170000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9830000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.97699999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.804,"5min":1.871,"15min":1.8779999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.153,"15min":1.23},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.3799999999999999,"5min":1.931,"15min":2.5840000000000001},"last":1.1100000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.2569999999999999,"15min":1.321},"min":{"1min":0.76600000000000001,"5min":0.65400000000000003,"15min":0.63},"max":{"1min":1.591,"5min":2.024,"15min":2.8119999999999998},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.1950000000000001,"15min":1.2849999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.492,"5min":1.8600000000000001,"15min":2.6259999999999999},"last":1.7829999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.387,"15min":1.3999999999999999},"min":{"1min":0.99299999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.361,"15min":1.419},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.794,"5min":1.8759999999999999,"15min":2.75},"last":1.6970000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.2869999999999999,"15min":1.302},"min":{"1min":0.81299999999999994,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.982,"5min":1.982,"15min":2.774},"last":1.9570000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.478,"15min":1.4470000000000001},"min":{"1min":1.2490000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.3330000000000002},"last":1.2549999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4550000000000001,"15min":1.4399999999999999},"min":{"1min":1.028,"5min":0.59399999999999997,"15min":0.55900000000000005},"max":{"1min":2.081,"5min":2.081,"15min":2.6720000000000002},"last":1.841},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.89100000000000001,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7010000000000001},"last":1.893}]},{"osd":7,"last update":"Mon Oct 16 14:13:26 2023","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.593,"15min":1.5580000000000001},"min":{"1min":1.278,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.4609999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":1.1579999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.2010000000000001,"5min":2.238,"15min":2.8719999999999999},"last":1.9330000000000001}]}]},{"osd":4,"up_from":14,"seq":60129543343,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88864,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282976,"statfs":{"total":96636764160,"available":96545767424,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.1930000000000001},"min":{"1min":0.85899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.575,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.159,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.5629999999999999,"5min":1.8859999999999999,"15min":1.9019999999999999},"last":0.97999999999999998}]},{"osd":1,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1439999999999999,"15min":1.2270000000000001},"min":{"1min":0.73099999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.27},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.7210000000000001,"5min":1.792,"15min":2.0499999999999998},"last":1.387}]},{"osd":2,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2529999999999999,"15min":1.302},"min":{"1min":0.81599999999999995,"5min":0.73899999999999999,"15min":0.55500000000000005},"max":{"1min":1.4299999999999999,"5min":1.714,"15min":1.865},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.381,"5min":1.365,"15min":1.3779999999999999},"min":{"1min":1.105,"5min":0.87,"15min":0.87},"max":{"1min":1.77,"5min":1.9370000000000001,"15min":2.0110000000000001},"last":1.5069999999999999}]},{"osd":3,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.292,"15min":1.2989999999999999},"min":{"1min":0.996,"5min":0.77000000000000002,"15min":0.65100000000000002},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9650000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4299999999999999,"15min":1.425},"min":{"1min":1.1459999999999999,"5min":0.92800000000000005,"15min":0.86799999999999999},"max":{"1min":1.887,"5min":5.2350000000000003,"15min":5.2350000000000003},"last":1.454}]},{"osd":5,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4019999999999999,"15min":1.3540000000000001},"min":{"1min":0.997,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.738,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.569},{"interface":"front","average":{"1min":1.427,"5min":1.415,"15min":1.3600000000000001},"min":{"1min":0.97499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0659999999999998,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.49}]},{"osd":6,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4390000000000001,"15min":1.3999999999999999},"min":{"1min":1.091,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.1200000000000001},"last":1.323},{"interface":"front","average":{"1min":1.649,"5min":1.5760000000000001,"15min":1.5289999999999999},"min":{"1min":1.2090000000000001,"5min":0.98599999999999999,"15min":0.94199999999999995},"max":{"1min":1.9790000000000001,"5min":2.5830000000000002,"15min":2.6429999999999998},"last":1.681}]},{"osd":7,"last update":"Mon Oct 16 14:13:36 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.591,"15min":1.5269999999999999},"min":{"1min":1.22,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.1219999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.276},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.657,"15min":1.645},"min":{"1min":1.335,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.3660000000000001},"last":1.5980000000000001}]}]},{"osd":3,"up_from":14,"seq":60129543343,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91000000000000003,"15min":0.93799999999999994},"min":{"1min":0.53900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.54,"5min":1.595,"15min":1.595},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.105,"15min":1.1899999999999999},"min":{"1min":0.57399999999999995,"5min":0.44900000000000001,"15min":0.42399999999999999},"max":{"1min":1.5529999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.0229999999999999}]},{"osd":1,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0229999999999999,"15min":1.0700000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.441},"max":{"1min":1.6319999999999999,"5min":1.6870000000000001,"15min":1.855},"last":1.056},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0620000000000001,"15min":1.105},"min":{"1min":0.56999999999999995,"5min":0.495,"15min":0.46000000000000002},"max":{"1min":1.9890000000000001,"5min":2.125,"15min":2.125},"last":0.88400000000000001}]},{"osd":2,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0569999999999999,"15min":1.1739999999999999},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.544,"5min":2.0179999999999998,"15min":2.0539999999999998},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.272,"15min":1.4039999999999999},"min":{"1min":0.93200000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6739999999999999,"5min":1.909,"15min":1.9319999999999999},"last":1.6259999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4390000000000001,"15min":1.4870000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.67000000000000004},"max":{"1min":1.891,"5min":4.1299999999999999,"15min":4.1299999999999999},"last":1.458},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3859999999999999,"15min":1.4470000000000001},"min":{"1min":0.80500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6599999999999999,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.4930000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4019999999999999,"15min":1.4730000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0579999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.417},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.4950000000000001,"15min":1.5409999999999999},"min":{"1min":0.88700000000000001,"5min":0.871,"15min":0.77500000000000002},"max":{"1min":1.9339999999999999,"5min":2.0339999999999998,"15min":2.153},"last":1.532}]},{"osd":6,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4410000000000001,"15min":1.508},"min":{"1min":0.96599999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.8300000000000001,"5min":2.335,"15min":2.4329999999999998},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.4279999999999999,"15min":1.486},"min":{"1min":0.752,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.9279999999999999,"5min":2.194,"15min":2.4180000000000001},"last":1.5660000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:51 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5289999999999999,"15min":1.5840000000000001},"min":{"1min":0.875,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1320000000000001,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.611,"15min":1.665},"min":{"1min":0.91300000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.089,"5min":2.569,"15min":2.6070000000000002},"last":1.329}]}]},{"osd":2,"up_from":14,"seq":60129543341,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.94099999999999995},"min":{"1min":0.55400000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.1879999999999999,"5min":1.7589999999999999,"15min":2.0089999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.008,"15min":1.113},"min":{"1min":0.47799999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.8280000000000001,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.4810000000000001}]},{"osd":1,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.022,"15min":1.119},"min":{"1min":0.57399999999999995,"5min":0.46700000000000003,"15min":0.44500000000000001},"max":{"1min":1.5189999999999999,"5min":1.72,"15min":1.847},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0760000000000001,"15min":1.2110000000000001},"min":{"1min":0.51200000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.5249999999999999,"5min":1.784,"15min":1.9530000000000001},"last":0.93300000000000005}]},{"osd":3,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.012,"15min":1.087},"min":{"1min":0.55700000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":2.1299999999999999},"last":1.056},{"interface":"front","average":{"1min":1.153,"5min":1.212,"15min":1.327},"min":{"1min":0.65500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.665,"5min":2.3149999999999999,"15min":2.4329999999999998},"last":1.3320000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.425,"15min":1.4750000000000001},"min":{"1min":0.90000000000000002,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6759999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3899999999999999,"15min":1.462},"min":{"1min":0.80800000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5660000000000001,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.268}]},{"osd":5,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.462,"15min":1.5269999999999999},"min":{"1min":0.84799999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.694,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.407,"15min":1.454},"min":{"1min":0.754,"5min":0.627,"15min":0.627},"max":{"1min":1.9239999999999999,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":1.226}]},{"osd":6,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.472,"15min":1.5089999999999999},"min":{"1min":0.874,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.8839999999999999,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.524,"15min":1.5880000000000001},"min":{"1min":1.0449999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.7909999999999999,"5min":2.52,"15min":2.5569999999999999},"last":1.6899999999999999}]},{"osd":7,"last update":"Mon Oct 16 14:14:24 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.5840000000000001,"15min":1.6459999999999999},"min":{"1min":0.93600000000000005,"5min":0.754,"15min":0.754},"max":{"1min":1.8520000000000001,"5min":2.7149999999999999,"15min":2.7890000000000001},"last":1.732},{"interface":"front","average":{"1min":1.387,"5min":1.54,"15min":1.6160000000000001},"min":{"1min":0.87,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.8140000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.667}]}]},{"osd":1,"up_from":14,"seq":60129543341,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56996,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314844,"statfs":{"total":96636764160,"available":96578400256,"internally_reserved":0,"allocated":3690496,"data_stored":1815186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.878,"15min":0.92000000000000004},"min":{"1min":0.48899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2609999999999999,"5min":1.7669999999999999,"15min":1.8520000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0620000000000001,"15min":1.1160000000000001},"min":{"1min":0.501,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7070000000000001,"5min":1.907,"15min":1.907},"last":1.0009999999999999}]},{"osd":2,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.499,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4470000000000001,"5min":1.853,"15min":1.853},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.226},"min":{"1min":0.61299999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.2070000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2649999999999999,"15min":1.357},"min":{"1min":0.66400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0289999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.1639999999999999,"15min":1.2549999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7470000000000001,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":1.3009999999999999}]},{"osd":4,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.242,"15min":1.3240000000000001},"min":{"1min":0.90400000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.79,"5min":1.79,"15min":2.1259999999999999},"last":1.423},{"interface":"front","average":{"1min":1.339,"5min":1.4119999999999999,"15min":1.498},"min":{"1min":0.84499999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.825,"5min":2.3399999999999999,"15min":2.3410000000000002},"last":1.373}]},{"osd":5,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4490000000000001,"15min":1.472},"min":{"1min":0.625,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.02,"5min":2.9140000000000001,"15min":2.9140000000000001},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.407,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.5189999999999999}]},{"osd":6,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.548,"15min":1.607},"min":{"1min":1.113,"5min":0.748,"15min":0.748},"max":{"1min":2.0800000000000001,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.623},{"interface":"front","average":{"1min":1.411,"5min":1.522,"15min":1.579},"min":{"1min":0.88600000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.2690000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.7050000000000001}]},{"osd":7,"last update":"Mon Oct 16 14:13:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.619,"15min":1.6759999999999999},"min":{"1min":1.0840000000000001,"5min":0.628,"15min":0.628},"max":{"1min":2.3540000000000001,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.5980000000000001},"min":{"1min":0.94399999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.2610000000000001},"last":1.897}]}]},{"osd":0,"up_from":14,"seq":60129543342,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61728,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310112,"statfs":{"total":96636764160,"available":96573554688,"internally_reserved":0,"allocated":4284416,"data_stored":2405554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91600000000000004,"15min":0.93999999999999995},"min":{"1min":0.46200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.9379999999999999,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.98999999999999999,"15min":1.1140000000000001},"min":{"1min":0.60599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.9299999999999999},"last":1.052}]},{"osd":2,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0609999999999999,"15min":1.149},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.768,"5min":1.768,"15min":2.0569999999999999},"last":0.996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0880000000000001,"15min":1.1890000000000001},"min":{"1min":0.72299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":1.3260000000000001,"5min":1.78,"15min":1.893},"last":1.3360000000000001}]},{"osd":3,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0269999999999999,"15min":1.125},"min":{"1min":0.66700000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2290000000000001,"5min":1.7869999999999999,"15min":1.8520000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2350000000000001,"15min":1.341},"min":{"1min":0.79500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5880000000000001,"5min":1.9630000000000001,"15min":2.4009999999999998},"last":1.2350000000000001}]},{"osd":4,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2569999999999999,"15min":1.343},"min":{"1min":0.95399999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5900000000000001,"5min":1.923,"15min":2.0070000000000001},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3979999999999999,"15min":1.4910000000000001},"min":{"1min":0.996,"5min":0.748,"15min":0.748},"max":{"1min":2.0169999999999999,"5min":2.1070000000000002,"15min":2.601},"last":1.4530000000000001}]},{"osd":5,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4219999999999999,"15min":1.4810000000000001},"min":{"1min":0.80100000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.081,"5min":2.089,"15min":2.3279999999999998},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.3720000000000001,"15min":1.45},"min":{"1min":0.89700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.754,"5min":2.0169999999999999,"15min":2.2829999999999999},"last":1.4770000000000001}]},{"osd":6,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5049999999999999,"15min":1.5880000000000001},"min":{"1min":0.94099999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0990000000000002,"5min":2.2730000000000001,"15min":2.669},"last":1.526},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.494,"15min":1.5720000000000001},"min":{"1min":0.86099999999999999,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.893,"5min":2.327,"15min":2.3769999999999998},"last":1.411}]},{"osd":7,"last update":"Mon Oct 16 14:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5840000000000001,"15min":1.677},"min":{"1min":0.93899999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.5270000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.5249999999999999,"15min":1.6120000000000001},"min":{"1min":1.119,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.893,"5min":2.3679999999999999,"15min":2.464},"last":1.706}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T14:14:29.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-16T14:14:30.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.033+0000 7ff601840640 1 Processor -- start 2023-10-16T14:14:30.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.033+0000 7ff601840640 1 -- start start 2023-10-16T14:14:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.034+0000 7ff601840640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff5fc15cd70 0x7ff5fc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff601840640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5fc0cded0 0x7ff5fc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff601840640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc15d150 con 0x7ff5fc0b8480 2023-10-16T14:14:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff601840640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0d1180 con 0x7ff5fc0cded0 2023-10-16T14:14:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff601840640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0d1460 con 0x7ff5fc15cd70 2023-10-16T14:14:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5fc0cded0 0x7ff5fc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5fc0cded0 0x7ff5fc0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33992/0 (socket says 172.21.15.60:33992) 2023-10-16T14:14:30.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 -- 172.21.15.60:0/2548095494 learned_addr learned my addr 172.21.15.60:0/2548095494 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:30.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60083e640 1 --2- 172.21.15.60:0/2548095494 >> v2:172.21.15.144:3300/0 conn(0x7ff5fc15cd70 0x7ff5fc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 -- 172.21.15.60:0/2548095494 >> v2:172.21.15.144:3300/0 conn(0x7ff5fc15cd70 msgr2=0x7ff5fc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 --2- 172.21.15.60:0/2548095494 >> v2:172.21.15.144:3300/0 conn(0x7ff5fc15cd70 0x7ff5fc0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 -- 172.21.15.60:0/2548095494 >> v1:172.21.15.144:6789/0 conn(0x7ff5fc0b8480 legacy=0x7ff5fc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60103f640 1 -- 172.21.15.60:0/2548095494 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0d1740 con 0x7ff5fc0cded0 2023-10-16T14:14:30.042 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.035+0000 7ff60083e640 1 --2- 172.21.15.60:0/2548095494 >> v2:172.21.15.144:3300/0 conn(0x7ff5fc15cd70 0x7ff5fc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.036+0000 7ff60103f640 1 --2- 172.21.15.60:0/2548095494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0b5c90 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0b56a0 tx=0x7ff5f402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873159d2553cc2eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.037+0000 7ff5fb7fe640 1 -- 172.21.15.60:0/2548095494 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f402ca80 con 0x7ff5fc0cded0 2023-10-16T14:14:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.037+0000 7ff5fb7fe640 1 -- 172.21.15.60:0/2548095494 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f402cbe0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.037+0000 7ff5fb7fe640 1 -- 172.21.15.60:0/2548095494 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40343f0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.037+0000 7ff601840640 1 -- 172.21.15.60:0/2548095494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 msgr2=0x7ff5fc0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.037+0000 7ff601840640 1 --2- 172.21.15.60:0/2548095494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0b5c90 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0b56a0 tx=0x7ff5f402bee0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.038+0000 7ff601840640 1 -- 172.21.15.60:0/2548095494 shutdown_connections 2023-10-16T14:14:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.038+0000 7ff601840640 1 --2- 172.21.15.60:0/2548095494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0b5c90 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.038+0000 7ff601840640 1 --2- 172.21.15.60:0/2548095494 >> v2:172.21.15.144:3300/0 conn(0x7ff5fc15cd70 0x7ff5fc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.038+0000 7ff601840640 1 -- 172.21.15.60:0/2548095494 >> 172.21.15.60:0/2548095494 conn(0x7ff5fc0cf570 msgr2=0x7ff5fc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.038+0000 7ff601840640 1 -- 172.21.15.60:0/2548095494 shutdown_connections 2023-10-16T14:14:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.038+0000 7ff601840640 1 -- 172.21.15.60:0/2548095494 wait complete. 2023-10-16T14:14:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.039+0000 7ff601840640 1 Processor -- start 2023-10-16T14:14:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.039+0000 7ff601840640 1 -- start start 2023-10-16T14:14:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.040+0000 7ff601840640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5fc0b8480 0x7ff5fc1f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.040+0000 7ff601840640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.040+0000 7ff601840640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5fc15cd70 0x7ff5fc0c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.040+0000 7ff601840640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0a9820 con 0x7ff5fc0cded0 2023-10-16T14:14:30.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.040+0000 7ff601840640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14c2a0 con 0x7ff5fc15cd70 2023-10-16T14:14:30.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.040+0000 7ff601840640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14c580 con 0x7ff5fc0b8480 2023-10-16T14:14:30.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0d2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34002/0 (socket says 172.21.15.60:34002) 2023-10-16T14:14:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 -- 172.21.15.60:0/3874432431 learned_addr learned my addr 172.21.15.60:0/3874432431 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff60083e640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5fc0b8480 0x7ff5fc1f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 -- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5fc0b8480 msgr2=0x7ff5fc1f2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff60103f640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5fc15cd70 0x7ff5fc0c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5fc0b8480 0x7ff5fc1f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 -- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5fc15cd70 msgr2=0x7ff5fc0c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5fc15cd70 0x7ff5fc0c81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 -- 172.21.15.60:0/3874432431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4009b30 con 0x7ff5fc0cded0 2023-10-16T14:14:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.041+0000 7ff5fbfff640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0d2cf0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800c410 tx=0x7ff5e800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.042+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80035f0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.042+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e8003750 con 0x7ff5fc0cded0 2023-10-16T14:14:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.042+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8003b40 con 0x7ff5fc0cded0 2023-10-16T14:14:30.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.042+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc1f23a0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.042+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc1f2960 con 0x7ff5fc0cded0 2023-10-16T14:14:30.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.043+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc04e9b0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.044+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff5e8003ca0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.045+0000 7ff5f97fa640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff5e006af90 0x7ff5e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.045+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7ff5e80889c0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.046+0000 7ff60083e640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff5e006af90 0x7ff5e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.047+0000 7ff60083e640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff5e006af90 0x7ff5e006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000b820 tx=0x7ff5f0008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:30.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.049+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5e8053940 con 0x7ff5fc0cded0 2023-10-16T14:14:30.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.238+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff5fc0b17a0 con 0x7ff5fc0cded0 2023-10-16T14:14:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.244+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff5e8056f60 con 0x7ff5fc0cded0 2023-10-16T14:14:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.245+0000 7ff5f97fa640 1 -- 172.21.15.60:0/3874432431 <== mon.0 v2:172.21.15.60:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff5e8056f60 con 0x7ff5fc0cded0 2023-10-16T14:14:30.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff5e006af90 msgr2=0x7ff5e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff5e006af90 0x7ff5e006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000b820 tx=0x7ff5f0008040 comp rx=0 tx=0).stop 2023-10-16T14:14:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 msgr2=0x7ff5fc0d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0d2cf0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800c410 tx=0x7ff5e800c8d0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 shutdown_connections 2023-10-16T14:14:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7ff5e006af90 0x7ff5e006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5fc15cd70 0x7ff5fc0c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5fc0cded0 0x7ff5fc0d2cf0 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 --2- 172.21.15.60:0/3874432431 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7ff5fc0b8480 0x7ff5fc1f2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 >> 172.21.15.60:0/3874432431 conn(0x7ff5fc0cf570 msgr2=0x7ff5fc0b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 shutdown_connections 2023-10-16T14:14:30.252 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.247+0000 7ff601840640 1 -- 172.21.15.60:0/3874432431 wait complete. 2023-10-16T14:14:30.266 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T14:14:30.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T14:14:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.448+0000 7f79d2a5d640 1 Processor -- start 2023-10-16T14:14:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.448+0000 7f79d2a5d640 1 -- start start 2023-10-16T14:14:30.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.449+0000 7f79d2a5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79cc159b20 0x7f79cc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.449+0000 7f79d2a5d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 0x7f79cc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.449+0000 7f79d2a5d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79cc06f7f0 con 0x7f79cc159f00 2023-10-16T14:14:30.453 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.449+0000 7f79d2a5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79cc06c5e0 con 0x7f79cc159b20 2023-10-16T14:14:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.449+0000 7f79d2a5d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06c8c0 con 0x7f79cc0b0b20 2023-10-16T14:14:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.450+0000 7f79c3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f79cc159f00 0x7f79cc0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45226/0 (socket says 172.21.15.60:45226) 2023-10-16T14:14:30.454 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.450+0000 7f79c3fff640 1 -- 172.21.15.60:0/3805009099 learned_addr learned my addr 172.21.15.60:0/3805009099 (peer_addr_for_me v1:172.21.15.60:0/0) 2023-10-16T14:14:30.457 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.450+0000 7f79cbfff640 1 --2- 172.21.15.60:0/3805009099 >> v2:172.21.15.144:3300/0 conn(0x7f79cc159b20 0x7f79cc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.450+0000 7f79d0ffa640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 0x7f79cc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.450+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139648945 0 0) 0x7f79cc06f7f0 con 0x7f79cc159f00 2023-10-16T14:14:30.458 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.450+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b8003520 con 0x7f79cc159f00 2023-10-16T14:14:30.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 105050366 0 0) 0x7f79b8003520 con 0x7f79cc159f00 2023-10-16T14:14:30.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 msgr2=0x7f79cc06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:30.459 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 0x7f79cc06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 >> v2:172.21.15.144:3300/0 conn(0x7f79cc159b20 msgr2=0x7f79cc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:30.461 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 --2- 172.21.15.60:0/3805009099 >> v2:172.21.15.144:3300/0 conn(0x7f79cc159b20 0x7f79cc06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc06cba0 con 0x7f79cc159f00 2023-10-16T14:14:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1058587258 0 0) 0x7f79b4001670 con 0x7f79cc159f00 2023-10-16T14:14:30.462 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 >> v1:172.21.15.60:6789/0 conn(0x7f79cc159f00 legacy=0x7f79cc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 0x7f79b8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79b8003f40 0x7f79b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.463 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79b8004d00 0x7f79b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06c5e0 con 0x7f79b8003f40 2023-10-16T14:14:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06c8c0 con 0x7f79b8004d00 2023-10-16T14:14:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b800b980 con 0x7f79cc0b0b20 2023-10-16T14:14:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79c3fff640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79b8003f40 0x7f79b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79c3fff640 1 -- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 msgr2=0x7f79b8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79c3fff640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79cc0b0b20 0x7f79b8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.465 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79c3fff640 1 -- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79b8004d00 msgr2=0x7f79b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T14:14:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79c3fff640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79b8004d00 0x7f79b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.451+0000 7f79c3fff640 1 -- 172.21.15.60:0/3805009099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8003520 con 0x7f79b8003f40 2023-10-16T14:14:30.466 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.453+0000 7f79c3fff640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79b8003f40 0x7f79b80047c0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f79b400c110 tx=0x7f79b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf85cf94d5ea11b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.453+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4005ad0 con 0x7f79b8003f40 2023-10-16T14:14:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.453+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f79b4005c50 con 0x7f79b8003f40 2023-10-16T14:14:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.453+0000 7f79cb7fe640 1 -- 172.21.15.60:0/3805009099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b402c7e0 con 0x7f79b8003f40 2023-10-16T14:14:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d2a5d640 1 -- 172.21.15.60:0/3805009099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79b8003f40 msgr2=0x7f79b80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/3805009099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79b8003f40 0x7f79b80047c0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f79b400c110 tx=0x7f79b80057f0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.468 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d0ffa640 1 -- 172.21.15.60:0/3805009099 reap_dead start 2023-10-16T14:14:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d2a5d640 1 -- 172.21.15.60:0/3805009099 shutdown_connections 2023-10-16T14:14:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d2a5d640 1 -- 172.21.15.60:0/3805009099 >> 172.21.15.60:0/3805009099 conn(0x7f79cc0cb410 msgr2=0x7f79cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:30.469 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d2a5d640 1 -- 172.21.15.60:0/3805009099 shutdown_connections 2023-10-16T14:14:30.470 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.454+0000 7f79d2a5d640 1 -- 172.21.15.60:0/3805009099 wait complete. 2023-10-16T14:14:30.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.457+0000 7f79d2a5d640 1 Processor -- start 2023-10-16T14:14:30.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.457+0000 7f79d2a5d640 1 -- start start 2023-10-16T14:14:30.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79d2a5d640 1 --2- >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79b8003f40 0x7f79cc1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79d2a5d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 0x7f79cc0b3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79d2a5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79cc0b3dd0 0x7f79cc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79d2a5d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc0b4310 con 0x7f79cc0b1f90 2023-10-16T14:14:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79d2a5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc1f6520 con 0x7f79cc0b3dd0 2023-10-16T14:14:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79d2a5d640 1 -- --> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc1f6800 con 0x7f79b8003f40 2023-10-16T14:14:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 0x7f79cc0b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 0x7f79cc0b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34012/0 (socket says 172.21.15.60:34012) 2023-10-16T14:14:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.459+0000 7f79c3fff640 1 -- 172.21.15.60:0/2443115314 learned_addr learned my addr 172.21.15.60:0/2443115314 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-10-16T14:14:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79cbfff640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79b8003f40 0x7f79cc1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79d0ffa640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79cc0b3dd0 0x7f79cc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c3fff640 1 -- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79b8003f40 msgr2=0x7f79cc1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c3fff640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79b8003f40 0x7f79cc1560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c3fff640 1 -- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79cc0b3dd0 msgr2=0x7f79cc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c3fff640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79cc0b3dd0 0x7f79cc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c3fff640 1 -- 172.21.15.60:0/2443115314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc1f6ae0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79d0ffa640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79cc0b3dd0 0x7f79cc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T14:14:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c3fff640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 0x7f79cc0b3890 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7f79b402c380 tx=0x7f79b4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:30.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.460+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b403c040 con 0x7f79cc0b1f90 2023-10-16T14:14:30.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.461+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f79b402dd00 con 0x7f79cc0b1f90 2023-10-16T14:14:30.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.461+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4007290 con 0x7f79cc0b1f90 2023-10-16T14:14:30.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.461+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79cc1f6dc0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.461+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79cc0acdd0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.462+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79cc04e160 con 0x7f79cc0b1f90 2023-10-16T14:14:30.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.463+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79b4032070 con 0x7f79cc0b1f90 2023-10-16T14:14:30.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.464+0000 7f79c97fa640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f79ac06afc0 0x7f79ac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T14:14:30.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.464+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 5760+0+0 (secure 0 0 0) 0x7f79b40aaff0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.481 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.465+0000 7f79cbfff640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f79ac06afc0 0x7f79ac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T14:14:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.466+0000 7f79cbfff640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f79ac06afc0 0x7f79ac06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f79cc1579e0 tx=0x7f79bc008040 comp rx=0 tx=0).ready entity=mgr.9014 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T14:14:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.468+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79b4075ed0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.694+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79cc0b10b0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.697 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.694+0000 7f79c97fa640 1 -- 172.21.15.60:0/2443115314 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+3708 (secure 0 0 0) 0x7f79b40794f0 con 0x7f79cc0b1f90 2023-10-16T14:14:30.698 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-10-16T14:14:30.698 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":154,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10222,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/940706561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":940706561},{"type":"v1","addr":"172.21.15.144:6835","nonce":940706561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10225,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3706515123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3706515123},{"type":"v1","addr":"172.21.15.60:6833","nonce":3706515123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":10234,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/652070224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":652070224},{"type":"v1","addr":"172.21.15.144:6837","nonce":652070224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":13527,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/3253622174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":3253622174},{"type":"v1","addr":"172.21.15.60:6835","nonce":3253622174}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[]} 2023-10-16T14:14:30.699 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 154 2023-10-16T14:14:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f79ac06afc0 msgr2=0x7f79ac06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f79ac06afc0 0x7f79ac06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f79cc1579e0 tx=0x7f79bc008040 comp rx=0 tx=0).stop 2023-10-16T14:14:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 msgr2=0x7f79cc0b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T14:14:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 0x7f79cc0b3890 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7f79b402c380 tx=0x7f79b4001930 comp rx=0 tx=0).stop 2023-10-16T14:14:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 shutdown_connections 2023-10-16T14:14:30.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:6832/3626350367,v1:172.21.15.144:6833/3626350367] conn(0x7f79ac06afc0 0x7f79ac06d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79cc0b3dd0 0x7f79cc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79cc0b1f90 0x7f79cc0b3890 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 --2- 172.21.15.60:0/2443115314 >> [v2:172.21.15.144:3301/0,v1:172.21.15.144:6790/0] conn(0x7f79b8003f40 0x7f79cc1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T14:14:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 >> 172.21.15.60:0/2443115314 conn(0x7f79cc0cb410 msgr2=0x7f79cc158480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T14:14:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 shutdown_connections 2023-10-16T14:14:30.703 INFO:teuthology.orchestra.run.smithi060.stderr:2023-10-16T14:14:30.697+0000 7f79d2a5d640 1 -- 172.21.15.60:0/2443115314 wait complete. 2023-10-16T14:14:30.716 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-16T14:14:30.717 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-16T14:14:30.717 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T14:14:30.718 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:30.731 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T14:14:30.732 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T14:14:30.732 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:30.798 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T14:14:30.799 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T14:14:30.800 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:30.833 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T14:14:30.833 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T14:14:30.833 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:30.902 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T14:14:30.902 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-16T14:14:30.903 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-16T14:14:30.903 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:30.996 INFO:tasks.ceph.osd.0:Stopped 2023-10-16T14:14:30.996 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-16T14:14:30.997 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.087 INFO:tasks.ceph.osd.1:Stopped 2023-10-16T14:14:31.087 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-16T14:14:31.088 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.181 INFO:tasks.ceph.osd.2:Stopped 2023-10-16T14:14:31.181 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-16T14:14:31.182 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.261 INFO:tasks.ceph.osd.3:Stopped 2023-10-16T14:14:31.262 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-16T14:14:31.262 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.309 INFO:tasks.ceph.osd.4:Stopped 2023-10-16T14:14:31.310 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-16T14:14:31.310 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.364 INFO:tasks.ceph.osd.5:Stopped 2023-10-16T14:14:31.364 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-16T14:14:31.365 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.418 INFO:tasks.ceph.osd.6:Stopped 2023-10-16T14:14:31.419 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-16T14:14:31.419 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.479 INFO:tasks.ceph.osd.7:Stopped 2023-10-16T14:14:31.479 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-16T14:14:31.480 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T14:14:31.480 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.565 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T14:14:31.566 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T14:14:31.566 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.638 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T14:14:31.639 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-16T14:14:31.639 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-16T14:14:31.639 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.683 INFO:tasks.ceph.mon.a:Stopped 2023-10-16T14:14:31.684 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-16T14:14:31.684 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.738 INFO:tasks.ceph.mon.b:Stopped 2023-10-16T14:14:31.739 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-16T14:14:31.739 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T14:14:31.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-16T14:14:31.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-16T14:14:31.781 INFO:tasks.ceph.mon.c:Stopped 2023-10-16T14:14:31.782 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-16T14:14:31.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T14:14:31.895 INFO:teuthology.orchestra.run.smithi060.stdout:2023-10-16T12:48:32.760464+0000 mon.a (mon.0) 512 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-10-16T14:14:31.896 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-16T14:14:31.897 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:14:31.897 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-16T14:14:32.172 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:14:32.172 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-16T14:14:32.341 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:14:32.342 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-16T14:14:32.507 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:14:32.507 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-16T14:14:32.691 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T14:14:32.691 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-16T14:14:33.007 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T14:14:33.007 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-16T14:14:33.173 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T14:14:33.173 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-16T14:14:33.324 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T14:14:33.324 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-16T14:14:33.505 INFO:tasks.ceph:Archiving mon data... 2023-10-16T14:14:33.507 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/data/mon.a.tgz 2023-10-16T14:14:33.508 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2023-10-16T14:14:33.570 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.3s7Hn13PSW 2023-10-16T14:14:33.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3s7Hn13PSW 2023-10-16T14:14:34.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.3s7Hn13PSW 2023-10-16T14:14:34.509 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.3s7Hn13PSW is 4MB 2023-10-16T14:14:34.690 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.3s7Hn13PSW 2023-10-16T14:14:34.756 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/data/mon.b.tgz 2023-10-16T14:14:34.757 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2023-10-16T14:14:34.817 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.zAtjbe9vle 2023-10-16T14:14:34.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zAtjbe9vle 2023-10-16T14:14:35.850 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.zAtjbe9vle 2023-10-16T14:14:35.998 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.zAtjbe9vle is 4MB 2023-10-16T14:14:36.169 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.zAtjbe9vle 2023-10-16T14:14:36.231 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/data/mon.c.tgz 2023-10-16T14:14:36.232 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2023-10-16T14:14:36.284 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.LavDJ7anTt 2023-10-16T14:14:36.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.LavDJ7anTt 2023-10-16T14:14:36.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-16T14:14:36.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-16T14:14:37.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.LavDJ7anTt 2023-10-16T14:14:37.204 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.LavDJ7anTt is 4MB 2023-10-16T14:14:37.373 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.LavDJ7anTt 2023-10-16T14:14:37.435 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-16T14:14:37.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T14:14:37.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T14:14:37.551 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-16T14:14:37.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T14:14:37.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T14:14:37.660 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T14:14:37.668 INFO:teuthology.orchestra.run.smithi144.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T14:14:37.681 INFO:tasks.ceph:Archiving crash dumps... 2023-10-16T14:14:37.683 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/remote/smithi060/crash 2023-10-16T14:14:37.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T14:14:37.772 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/remote/smithi144/crash 2023-10-16T14:14:37.773 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T14:14:37.871 INFO:tasks.ceph:Shutting down logrotate 2023-10-16T14:14:37.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:14:37.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T14:14:38.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T14:14:38.118 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T14:14:38.220 INFO:tasks.ceph:Compressing logs... 2023-10-16T14:14:38.221 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T14:14:38.230 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T14:14:38.324 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-16T14:14:38.324 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87383.log 2023-10-16T14:14:38.326 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87343.log 2023-10-16T14:14:38.327 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87383.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87383.log.gz 2023-10-16T14:14:38.327 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87583.log 2023-10-16T14:14:38.328 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87343.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T14:14:38.329 INFO:teuthology.orchestra.run.smithi144.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87343.log.gz 2023-10-16T14:14:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143787.log 2023-10-16T14:14:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132583.log 2023-10-16T14:14:38.331 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87583.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87583.log.gz 2023-10-16T14:14:38.331 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-16T14:14:38.332 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87463.log 2023-10-16T14:14:38.333 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87303.log 2023-10-16T14:14:38.334 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87463.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87463.log.gz 2023-10-16T14:14:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106417.log 2023-10-16T14:14:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132583.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143787.log.gz 2023-10-16T14:14:38.335 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.132583.log.gz 2023-10-16T14:14:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135980.log 2023-10-16T14:14:38.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151236.log 2023-10-16T14:14:38.336 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106417.log.gz 2023-10-16T14:14:38.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144302.log 2023-10-16T14:14:38.338 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.135980.log.gz 2023-10-16T14:14:38.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116355.log 2023-10-16T14:14:38.339 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.151236.log.gz 2023-10-16T14:14:38.340 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87503.log 2023-10-16T14:14:38.340 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.87303.log: -- /var/log/ceph/ceph-osd.4.log 2023-10-16T14:14:38.340 INFO:teuthology.orchestra.run.smithi144.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87303.log.gz 2023-10-16T14:14:38.341 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87503.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87503.log.gz 2023-10-16T14:14:38.341 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.93104.log 2023-10-16T14:14:38.341 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-16T14:14:38.341 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.3.93104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.93104.log.gz 2023-10-16T14:14:38.342 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-16T14:14:38.343 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87543.log 2023-10-16T14:14:38.344 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.92711.log 2023-10-16T14:14:38.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119713.log 2023-10-16T14:14:38.346 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.144302.log.gz 2023-10-16T14:14:38.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116355.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116355.log.gz 2023-10-16T14:14:38.347 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107006.log 2023-10-16T14:14:38.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119713.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119713.log.gz 2023-10-16T14:14:38.347 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131589.log 2023-10-16T14:14:38.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.107006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107006.log.gz 2023-10-16T14:14:38.348 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134977.log 2023-10-16T14:14:38.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131589.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.131589.log.gz 2023-10-16T14:14:38.348 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105873.log 2023-10-16T14:14:38.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134977.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134977.log.gz 2023-10-16T14:14:38.349 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95371.log 2023-10-16T14:14:38.353 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87543.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87543.log.gz 2023-10-16T14:14:38.353 INFO:teuthology.orchestra.run.smithi144.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T14:14:38.353 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.1.92711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.92711.log.gz 2023-10-16T14:14:38.354 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-16T14:14:38.354 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T14:14:38.354 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-16T14:14:38.354 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-16T14:14:38.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114989.log 2023-10-16T14:14:38.356 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105873.log.gz 2023-10-16T14:14:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95371.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.95371.log.gz 2023-10-16T14:14:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106291.log 2023-10-16T14:14:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109145.log 2023-10-16T14:14:38.357 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114989.log.gz 2023-10-16T14:14:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106291.log: gzip -5 89.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.106291.log.gz 2023-10-16T14:14:38.358 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.105736.log 2023-10-16T14:14:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109145.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.160136.log -- replaced with /var/log/ceph/ceph-client.admin.109145.log.gz 2023-10-16T14:14:38.358 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:38.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148065.log 2023-10-16T14:14:38.359 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105736.log.gz 2023-10-16T14:14:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131379.log 2023-10-16T14:14:38.359 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.160136.log.gz 2023-10-16T14:14:38.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148065.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148065.log.gz 2023-10-16T14:14:38.360 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148233.log 2023-10-16T14:14:38.362 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87423.log 2023-10-16T14:14:38.363 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.87423.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-16T14:14:38.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110813.log 2023-10-16T14:14:38.365 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.131379.log.gz 2023-10-16T14:14:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150882.log 2023-10-16T14:14:38.365 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148233.log.gz 2023-10-16T14:14:38.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110813.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.110813.log.gz 2023-10-16T14:14:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124287.log 2023-10-16T14:14:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95875.log 2023-10-16T14:14:38.366 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.150882.log.gz 2023-10-16T14:14:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124287.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.124287.log.gz 2023-10-16T14:14:38.367 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.117540.log 2023-10-16T14:14:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95875.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95875.log.gz 2023-10-16T14:14:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158256.log 2023-10-16T14:14:38.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91294.log 2023-10-16T14:14:38.368 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117540.log.gz 2023-10-16T14:14:38.369 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87423.log.gz 2023-10-16T14:14:38.370 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mon.c.log: 94.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T14:14:38.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158256.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.158256.log.gz 2023-10-16T14:14:38.372 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154277.log 2023-10-16T14:14:38.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91294.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91294.log.gz 2023-10-16T14:14:38.372 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126738.log 2023-10-16T14:14:38.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154277.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-16T14:14:38.373 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154277.log.gz 2023-10-16T14:14:38.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105045.log 2023-10-16T14:14:38.373 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126738.log.gz 2023-10-16T14:14:38.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153190.log 2023-10-16T14:14:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132761.log 2023-10-16T14:14:38.375 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105045.log.gz 2023-10-16T14:14:38.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92802.log 2023-10-16T14:14:38.376 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153190.log.gz 2023-10-16T14:14:38.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132761.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.132761.log.gz -5 2023-10-16T14:14:38.376 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.111261.log 2023-10-16T14:14:38.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.145453.log 2023-10-16T14:14:38.377 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.92802.log.gz 2023-10-16T14:14:38.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112158.log 2023-10-16T14:14:38.378 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111261.log.gz 2023-10-16T14:14:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.145453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.145453.log.gz 2023-10-16T14:14:38.379 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106819.log 2023-10-16T14:14:38.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160826.log 2023-10-16T14:14:38.381 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112158.log.gz 2023-10-16T14:14:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106819.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106819.log.gz 2023-10-16T14:14:38.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91734.log 2023-10-16T14:14:38.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146217.log 2023-10-16T14:14:38.387 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.160826.log.gz 2023-10-16T14:14:38.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91734.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91734.log.gz 2023-10-16T14:14:38.388 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106543.log 2023-10-16T14:14:38.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114905.log 2023-10-16T14:14:38.388 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.146217.log.gz 2023-10-16T14:14:38.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106543.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106543.log.gz 2023-10-16T14:14:38.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152056.log 2023-10-16T14:14:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99850.log 2023-10-16T14:14:38.390 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114905.log.gz 2023-10-16T14:14:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152056.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152056.log.gz 2023-10-16T14:14:38.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117474.log 2023-10-16T14:14:38.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131253.log 2023-10-16T14:14:38.392 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99850.log.gz 2023-10-16T14:14:38.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97480.log 2023-10-16T14:14:38.393 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117474.log.gz 2023-10-16T14:14:38.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131253.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.131253.log.gz 2023-10-16T14:14:38.394 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91872.log 2023-10-16T14:14:38.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143997.log 2023-10-16T14:14:38.395 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97480.log.gz 2023-10-16T14:14:38.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151783.log 2023-10-16T14:14:38.396 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91872.log.gz 2023-10-16T14:14:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143997.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.143997.log.gz 2023-10-16T14:14:38.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152308.log 2023-10-16T14:14:38.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114277.log 2023-10-16T14:14:38.398 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151783.log.gz 2023-10-16T14:14:38.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152308.log: gzip -5 88.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.152308.log.gz -- 2023-10-16T14:14:38.399 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.96862.log 2023-10-16T14:14:38.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133404.log 2023-10-16T14:14:38.401 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114277.log.gz 2023-10-16T14:14:38.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145116.log 88.3% 2023-10-16T14:14:38.402 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96862.log.gz 2023-10-16T14:14:38.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119755.log 2023-10-16T14:14:38.403 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.133404.log.gz 2023-10-16T14:14:38.404 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97353.log 2023-10-16T14:14:38.404 INFO:teuthology.orchestra.run.smithi060.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.145116.log.gz 2023-10-16T14:14:38.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148359.log 2023-10-16T14:14:38.405 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119755.log.gz 2023-10-16T14:14:38.406 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97353.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97353.log.gz 2023-10-16T14:14:38.406 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112785.log 2023-10-16T14:14:38.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148359.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.148359.log.gz 2023-10-16T14:14:38.408 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132453.log 2023-10-16T14:14:38.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112785.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112785.log.gz 2023-10-16T14:14:38.409 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112828.log 2023-10-16T14:14:38.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132453.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.132453.log.gz 2023-10-16T14:14:38.410 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98022.log 2023-10-16T14:14:38.411 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112828.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112828.log.gz 2023-10-16T14:14:38.411 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125669.log 2023-10-16T14:14:38.412 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143829.log 2023-10-16T14:14:38.413 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98022.log.gz 2023-10-16T14:14:38.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125669.log: gzip -5 88.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.125669.log.gz 2023-10-16T14:14:38.414 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.149327.log 2023-10-16T14:14:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143829.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143829.log.gz 2023-10-16T14:14:38.415 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160262.log 2023-10-16T14:14:38.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149327.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149327.log.gz 2023-10-16T14:14:38.417 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142838.log 2023-10-16T14:14:38.419 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160262.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160262.log.gz 2023-10-16T14:14:38.419 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97980.log 2023-10-16T14:14:38.419 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.142838.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142838.log.gz 2023-10-16T14:14:38.420 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136621.log 2023-10-16T14:14:38.420 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97980.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97980.log.gz 2023-10-16T14:14:38.421 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158793.log 2023-10-16T14:14:38.421 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136621.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.136621.log.gz --verbose 2023-10-16T14:14:38.422 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.93930.log 2023-10-16T14:14:38.423 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158793.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158793.log.gz 2023-10-16T14:14:38.423 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.127201.log 2023-10-16T14:14:38.424 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93930.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93930.log.gz 2023-10-16T14:14:38.424 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105568.log 2023-10-16T14:14:38.425 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.127201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.127201.log.gz 2023-10-16T14:14:38.425 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106501.log 2023-10-16T14:14:38.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109026.log 2023-10-16T14:14:38.427 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105568.log.gz 2023-10-16T14:14:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106501.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106501.log.gz 2023-10-16T14:14:38.428 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123044.log 2023-10-16T14:14:38.429 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109026.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109026.log.gz 2023-10-16T14:14:38.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125254.log 2023-10-16T14:14:38.431 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132028.log 2023-10-16T14:14:38.431 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123044.log.gz 2023-10-16T14:14:38.432 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119533.log 2023-10-16T14:14:38.432 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125254.log.gz 2023-10-16T14:14:38.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121331.log 2023-10-16T14:14:38.433 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.132028.log.gz 2023-10-16T14:14:38.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119533.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119533.log.gz 2023-10-16T14:14:38.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113917.log 2023-10-16T14:14:38.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121331.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.121331.log.gz 2023-10-16T14:14:38.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131776.log 2023-10-16T14:14:38.437 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103415.log 2023-10-16T14:14:38.437 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113917.log.gz 2023-10-16T14:14:38.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114127.log 2023-10-16T14:14:38.438 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131776.log.gz 2023-10-16T14:14:38.439 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.103415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120466.log 2023-10-16T14:14:38.439 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103415.log.gz 2023-10-16T14:14:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114127.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114127.log.gz 2023-10-16T14:14:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95203.log 2023-10-16T14:14:38.442 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154445.log 2023-10-16T14:14:38.442 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120466.log.gz 2023-10-16T14:14:38.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95203.log: gzip -5 --verbose 88.8% -- /var/log/ceph/ceph-client.admin.92809.log -- replaced with /var/log/ceph/ceph-client.admin.95203.log.gz 2023-10-16T14:14:38.443 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:38.444 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154445.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.154445.log.gz 2023-10-16T14:14:38.444 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125879.log 2023-10-16T14:14:38.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92809.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92809.log.gz 2023-10-16T14:14:38.446 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124652.log 2023-10-16T14:14:38.446 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93468.log 2023-10-16T14:14:38.447 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.125879.log.gz 2023-10-16T14:14:38.448 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134851.log 2023-10-16T14:14:38.448 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124652.log.gz 2023-10-16T14:14:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156341.log 2023-10-16T14:14:38.449 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93468.log.gz 2023-10-16T14:14:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121688.log 2023-10-16T14:14:38.451 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134851.log.gz 2023-10-16T14:14:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156341.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.156341.log.gz 2023-10-16T14:14:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148713.log 2023-10-16T14:14:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157494.log 2023-10-16T14:14:38.453 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121688.log.gz 2023-10-16T14:14:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149577.log 2023-10-16T14:14:38.454 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.148713.log.gz 2023-10-16T14:14:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106161.log 2023-10-16T14:14:38.455 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157494.log.gz 2023-10-16T14:14:38.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99732.log 2023-10-16T14:14:38.456 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.149577.log.gz 2023-10-16T14:14:38.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106161.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106161.log.gz 2023-10-16T14:14:38.458 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104628.log 2023-10-16T14:14:38.458 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122085.log 2023-10-16T14:14:38.459 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99732.log.gz 2023-10-16T14:14:38.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124328.log 2023-10-16T14:14:38.460 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104628.log.gz 2023-10-16T14:14:38.461 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87633.log 2023-10-16T14:14:38.462 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122085.log.gz 2023-10-16T14:14:38.462 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138478.log 2023-10-16T14:14:38.463 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124328.log.gz 2023-10-16T14:14:38.464 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.87633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155629.log 2023-10-16T14:14:38.464 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87633.log.gz 2023-10-16T14:14:38.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112389.log 2023-10-16T14:14:38.465 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138478.log.gz 2023-10-16T14:14:38.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.140398.log 2023-10-16T14:14:38.466 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.155629.log.gz 2023-10-16T14:14:38.467 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125338.log 2023-10-16T14:14:38.467 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112389.log.gz 2023-10-16T14:14:38.468 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.140398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.140398.log.gz 2023-10-16T14:14:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132327.log 2023-10-16T14:14:38.470 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125837.log 2023-10-16T14:14:38.470 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125338.log.gz 2023-10-16T14:14:38.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135462.log 2023-10-16T14:14:38.472 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.132327.log.gz 2023-10-16T14:14:38.472 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144632.log 2023-10-16T14:14:38.473 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.125837.log.gz 2023-10-16T14:14:38.473 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135462.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135462.log.gz 2023-10-16T14:14:38.474 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97030.log 2023-10-16T14:14:38.474 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160178.log 2023-10-16T14:14:38.475 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144632.log.gz 2023-10-16T14:14:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97030.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97030.log.gz 2023-10-16T14:14:38.476 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115167.log 2023-10-16T14:14:38.477 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145158.log 2023-10-16T14:14:38.477 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160178.log.gz 2023-10-16T14:14:38.478 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151452.log 2023-10-16T14:14:38.479 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115167.log.gz 2023-10-16T14:14:38.479 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138436.log 2023-10-16T14:14:38.480 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.145158.log.gz 2023-10-16T14:14:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151452.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151452.log.gz 2023-10-16T14:14:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159505.log 2023-10-16T14:14:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95329.log 2023-10-16T14:14:38.482 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138436.log.gz 2023-10-16T14:14:38.483 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121604.log 2023-10-16T14:14:38.483 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.159505.log.gz 2023-10-16T14:14:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113609.log 2023-10-16T14:14:38.484 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95329.log.gz 2023-10-16T14:14:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152140.log 2023-10-16T14:14:38.485 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121604.log.gz 2023-10-16T14:14:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111303.log 2023-10-16T14:14:38.487 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113609.log.gz 2023-10-16T14:14:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152140.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152140.log.gz 2023-10-16T14:14:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113182.log 2023-10-16T14:14:38.489 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153334.log 2023-10-16T14:14:38.490 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.111303.log.gz 2023-10-16T14:14:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126780.log 2023-10-16T14:14:38.491 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113182.log.gz 2023-10-16T14:14:38.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147456.log 2023-10-16T14:14:38.492 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.153334.log.gz 2023-10-16T14:14:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126780.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.126780.log.gz 2023-10-16T14:14:38.493 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123214.log 2023-10-16T14:14:38.494 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92797.log 2023-10-16T14:14:38.495 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.147456.log.gz 2023-10-16T14:14:38.495 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124855.log 2023-10-16T14:14:38.496 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123214.log.gz 2023-10-16T14:14:38.497 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92797.log: gzip -5 88.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.92797.log.gz -- 2023-10-16T14:14:38.497 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.137468.log 2023-10-16T14:14:38.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104586.log 2023-10-16T14:14:38.498 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.124855.log.gz 2023-10-16T14:14:38.499 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140626.log 2023-10-16T14:14:38.500 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.137468.log.gz 2023-10-16T14:14:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96946.log 2023-10-16T14:14:38.501 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104586.log.gz 2023-10-16T14:14:38.502 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148023.log 2023-10-16T14:14:38.502 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140626.log.gz 2023-10-16T14:14:38.503 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143913.log 2023-10-16T14:14:38.503 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96946.log.gz 2023-10-16T14:14:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93134.log 2023-10-16T14:14:38.504 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.148023.log.gz 2023-10-16T14:14:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96742.log 2023-10-16T14:14:38.505 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143913.log.gz 2023-10-16T14:14:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117017.log 2023-10-16T14:14:38.507 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93134.log.gz 2023-10-16T14:14:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96742.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96742.log.gz 2023-10-16T14:14:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118880.log 2023-10-16T14:14:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144720.log 2023-10-16T14:14:38.509 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.117017.log.gz 2023-10-16T14:14:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96238.log 2023-10-16T14:14:38.510 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118880.log.gz 2023-10-16T14:14:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155324.log 2023-10-16T14:14:38.512 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.144720.log.gz 2023-10-16T14:14:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96238.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96238.log.gz 2023-10-16T14:14:38.513 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120201.log 2023-10-16T14:14:38.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97522.log 2023-10-16T14:14:38.514 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.155324.log.gz 2023-10-16T14:14:38.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150710.log 2023-10-16T14:14:38.515 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.120201.log.gz 2023-10-16T14:14:38.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97522.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.126058.log -- replaced with /var/log/ceph/ceph-client.admin.97522.log.gz 2023-10-16T14:14:38.516 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:38.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135378.log 2023-10-16T14:14:38.518 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.150710.log.gz 2023-10-16T14:14:38.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126058.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126058.log.gz 2023-10-16T14:14:38.519 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145963.log 2023-10-16T14:14:38.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123448.log 2023-10-16T14:14:38.520 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135378.log.gz 2023-10-16T14:14:38.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133139.log 2023-10-16T14:14:38.521 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145963.log.gz 2023-10-16T14:14:38.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93972.log 2023-10-16T14:14:38.522 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123448.log.gz 2023-10-16T14:14:38.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131158.log 2023-10-16T14:14:38.525 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133139.log.gz 2023-10-16T14:14:38.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120550.log 2023-10-16T14:14:38.527 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.93972.log.gz 2023-10-16T14:14:38.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91998.log 2023-10-16T14:14:38.527 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131158.log.gz 2023-10-16T14:14:38.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118754.log 2023-10-16T14:14:38.529 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.120550.log.gz 2023-10-16T14:14:38.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154403.log 2023-10-16T14:14:38.530 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91998.log.gz 2023-10-16T14:14:38.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138833.log 2023-10-16T14:14:38.531 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118754.log.gz 2023-10-16T14:14:38.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113789.log 2023-10-16T14:14:38.532 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154403.log.gz 2023-10-16T14:14:38.533 INFO:teuthology.orchestra.run.smithi144.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-16T14:14:38.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126432.log 2023-10-16T14:14:38.534 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138833.log.gz 2023-10-16T14:14:38.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97199.log 2023-10-16T14:14:38.534 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113789.log.gz 2023-10-16T14:14:38.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151008.log 2023-10-16T14:14:38.536 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126432.log.gz 2023-10-16T14:14:38.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97199.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97199.log.gz 2023-10-16T14:14:38.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139169.log 2023-10-16T14:14:38.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151008.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.151008.log.gz --verbose 2023-10-16T14:14:38.538 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.131944.log 2023-10-16T14:14:38.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149959.log 2023-10-16T14:14:38.539 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139169.log.gz 2023-10-16T14:14:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131944.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.131944.log.gz 2023-10-16T14:14:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145531.log 2023-10-16T14:14:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103941.log 2023-10-16T14:14:38.541 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.149959.log.gz 2023-10-16T14:14:38.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145531.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145531.log.gz 2023-10-16T14:14:38.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150966.log 2023-10-16T14:14:38.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.103941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115304.log 2023-10-16T14:14:38.543 INFO:teuthology.orchestra.run.smithi060.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103941.log.gz 2023-10-16T14:14:38.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144804.log 2023-10-16T14:14:38.545 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150966.log.gz 2023-10-16T14:14:38.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144990.log 2023-10-16T14:14:38.546 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115304.log.gz 2023-10-16T14:14:38.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149535.log 2023-10-16T14:14:38.547 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.144804.log.gz 2023-10-16T14:14:38.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118922.log 2023-10-16T14:14:38.548 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144990.log.gz 2023-10-16T14:14:38.549 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99614.log 2023-10-16T14:14:38.549 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.149535.log.gz 2023-10-16T14:14:38.550 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133668.log 2023-10-16T14:14:38.550 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118922.log.gz 2023-10-16T14:14:38.551 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118000.log 2023-10-16T14:14:38.551 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99614.log.gz 2023-10-16T14:14:38.552 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125921.log 2023-10-16T14:14:38.552 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133668.log.gz 2023-10-16T14:14:38.553 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91956.log 2023-10-16T14:14:38.553 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.118000.log.gz 2023-10-16T14:14:38.554 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98582.log 2023-10-16T14:14:38.555 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125921.log.gz 2023-10-16T14:14:38.555 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124492.log 2023-10-16T14:14:38.556 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91956.log.gz 2023-10-16T14:14:38.557 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139345.log 2023-10-16T14:14:38.557 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98582.log.gz 2023-10-16T14:14:38.558 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124492.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124492.log.gz 2023-10-16T14:14:38.558 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98370.log 2023-10-16T14:14:38.559 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139345.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139345.log.gz 2023-10-16T14:14:38.559 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98666.log 2023-10-16T14:14:38.560 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149665.log 2023-10-16T14:14:38.561 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98370.log.gz 2023-10-16T14:14:38.561 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98666.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98666.log.gz 2023-10-16T14:14:38.562 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147372.log 2023-10-16T14:14:38.563 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107312.log 2023-10-16T14:14:38.563 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149665.log.gz 2023-10-16T14:14:38.564 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147372.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147372.log.gz 2023-10-16T14:14:38.564 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145074.log 2023-10-16T14:14:38.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.107312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107211.log 2023-10-16T14:14:38.566 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107312.log.gz 2023-10-16T14:14:38.566 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113140.log 2023-10-16T14:14:38.567 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.145074.log.gz 2023-10-16T14:14:38.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.107211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136022.log 2023-10-16T14:14:38.568 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107211.log.gz 2023-10-16T14:14:38.569 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113140.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113140.log.gz 2023-10-16T14:14:38.569 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132245.log 2023-10-16T14:14:38.570 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136022.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.136022.log.gz 2023-10-16T14:14:38.570 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140756.log 2023-10-16T14:14:38.571 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155503.log 2023-10-16T14:14:38.572 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.132245.log.gz 2023-10-16T14:14:38.573 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158297.log 2023-10-16T14:14:38.573 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.140756.log.gz 2023-10-16T14:14:38.574 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155503.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.155503.log.gz -- 2023-10-16T14:14:38.574 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.155052.log 2023-10-16T14:14:38.575 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158297.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.158297.log.gz 2023-10-16T14:14:38.576 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145200.log 2023-10-16T14:14:38.577 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99291.log 2023-10-16T14:14:38.577 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.155052.log.gz 2023-10-16T14:14:38.578 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.150380.log 2023-10-16T14:14:38.579 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99291.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.145200.log.gz 2023-10-16T14:14:38.579 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110683.log 2023-10-16T14:14:38.580 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99291.log.gz 2023-10-16T14:14:38.580 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.150380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.150380.log.gz 2023-10-16T14:14:38.581 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153022.log 2023-10-16T14:14:38.582 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101643.log 2023-10-16T14:14:38.582 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110683.log.gz 2023-10-16T14:14:38.583 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153376.log 2023-10-16T14:14:38.583 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.153022.log.gz 2023-10-16T14:14:38.584 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139265.log 2023-10-16T14:14:38.585 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101643.log.gz 2023-10-16T14:14:38.585 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109964.log 2023-10-16T14:14:38.586 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.153376.log.gz 2023-10-16T14:14:38.587 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118964.log 2023-10-16T14:14:38.587 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.139265.log.gz 2023-10-16T14:14:38.588 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122554.log 2023-10-16T14:14:38.588 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109964.log.gz 2023-10-16T14:14:38.589 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118964.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118964.log.gz 2023-10-16T14:14:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97072.log 2023-10-16T14:14:38.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146685.log 2023-10-16T14:14:38.591 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122554.log.gz 2023-10-16T14:14:38.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97072.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97072.log.gz -5 2023-10-16T14:14:38.592 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.112515.log 2023-10-16T14:14:38.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146685.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146685.log.gz 2023-10-16T14:14:38.593 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156730.log 2023-10-16T14:14:38.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112515.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112515.log.gz 2023-10-16T14:14:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154099.log 2023-10-16T14:14:38.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150193.log 2023-10-16T14:14:38.596 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.156730.log.gz 2023-10-16T14:14:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154099.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154099.log.gz 2023-10-16T14:14:38.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119132.log 2023-10-16T14:14:38.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153883.log 2023-10-16T14:14:38.598 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.150193.log.gz 2023-10-16T14:14:38.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92474.log 2023-10-16T14:14:38.600 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119132.log.gz 2023-10-16T14:14:38.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111844.log 2023-10-16T14:14:38.601 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.153883.log.gz 2023-10-16T14:14:38.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125795.log 2023-10-16T14:14:38.602 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92474.log.gz 2023-10-16T14:14:38.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144039.log 2023-10-16T14:14:38.604 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111844.log.gz 2023-10-16T14:14:38.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138646.log 2023-10-16T14:14:38.605 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.125795.log.gz 2023-10-16T14:14:38.606 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123130.log 2023-10-16T14:14:38.606 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144039.log.gz 2023-10-16T14:14:38.607 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138646.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.138646.log.gz 2023-10-16T14:14:38.608 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121086.log 2023-10-16T14:14:38.608 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112557.log 2023-10-16T14:14:38.609 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.123130.log.gz 2023-10-16T14:14:38.610 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145615.log 2023-10-16T14:14:38.610 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121086.log.gz 2023-10-16T14:14:38.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113651.log 2023-10-16T14:14:38.612 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112557.log.gz 2023-10-16T14:14:38.612 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92793.log 2023-10-16T14:14:38.613 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145615.log.gz 2023-10-16T14:14:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117755.log 2023-10-16T14:14:38.614 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113651.log.gz 2023-10-16T14:14:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160556.log 2023-10-16T14:14:38.615 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92793.log.gz 2023-10-16T14:14:38.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114542.log 2023-10-16T14:14:38.617 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117755.log.gz 2023-10-16T14:14:38.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125380.log 2023-10-16T14:14:38.618 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.160556.log.gz 2023-10-16T14:14:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114542.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114542.log.gz 2023-10-16T14:14:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105357.log 2023-10-16T14:14:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122678.log 2023-10-16T14:14:38.620 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125380.log.gz 2023-10-16T14:14:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105357.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105357.log.gz 2023-10-16T14:14:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92166.log 2023-10-16T14:14:38.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92250.log 2023-10-16T14:14:38.623 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.122678.log.gz 2023-10-16T14:14:38.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92166.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92166.log.gz 2023-10-16T14:14:38.624 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150085.log 2023-10-16T14:14:38.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92250.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.92250.log.gz 2023-10-16T14:14:38.626 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135019.log 2023-10-16T14:14:38.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112345.log 2023-10-16T14:14:38.627 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150085.log.gz 2023-10-16T14:14:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135019.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.135019.log.gz 2023-10-16T14:14:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149833.log 2023-10-16T14:14:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99063.log 2023-10-16T14:14:38.630 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112345.log.gz 2023-10-16T14:14:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149833.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149833.log.gz 2023-10-16T14:14:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158835.log 2023-10-16T14:14:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.99063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.99063.log.gz 2023-10-16T14:14:38.633 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95032.log 2023-10-16T14:14:38.633 INFO:teuthology.orchestra.run.smithi144.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-16T14:14:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158835.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158835.log.gz 2023-10-16T14:14:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143493.log 2023-10-16T14:14:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155986.log 2023-10-16T14:14:38.635 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.95032.log.gz 2023-10-16T14:14:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143493.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143493.log.gz 2023-10-16T14:14:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94440.log 2023-10-16T14:14:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137144.log 2023-10-16T14:14:38.638 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.155986.log.gz 2023-10-16T14:14:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.94440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94440.log.gz 2023-10-16T14:14:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93594.log 2023-10-16T14:14:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154191.log 2023-10-16T14:14:38.640 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137144.log.gz 2023-10-16T14:14:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135812.log 2023-10-16T14:14:38.642 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93594.log.gz 2023-10-16T14:14:38.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148443.log 2023-10-16T14:14:38.643 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154191.log.gz 2023-10-16T14:14:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145871.log 2023-10-16T14:14:38.644 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135812.log.gz 2023-10-16T14:14:38.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126948.log 2023-10-16T14:14:38.645 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148443.log.gz 2023-10-16T14:14:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101584.log 2023-10-16T14:14:38.647 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145871.log.gz 2023-10-16T14:14:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115073.log 2023-10-16T14:14:38.648 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.126948.log.gz 2023-10-16T14:14:38.649 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97262.log 2023-10-16T14:14:38.649 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101584.log.gz 2023-10-16T14:14:38.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115073.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.115073.log.gz 2023-10-16T14:14:38.651 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138917.log 2023-10-16T14:14:38.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138376.log 2023-10-16T14:14:38.652 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97262.log.gz 2023-10-16T14:14:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138917.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138917.log.gz 2023-10-16T14:14:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122804.log 2023-10-16T14:14:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104670.log 2023-10-16T14:14:38.655 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138376.log.gz 2023-10-16T14:14:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122804.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122804.log.gzgzip 2023-10-16T14:14:38.656 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.116014.log 2023-10-16T14:14:38.657 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152686.log 2023-10-16T14:14:38.658 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104670.log.gz 2023-10-16T14:14:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113319.log 2023-10-16T14:14:38.659 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116014.log.gz 2023-10-16T14:14:38.660 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146385.log 2023-10-16T14:14:38.660 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152686.log.gz 2023-10-16T14:14:38.661 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153586.log 2023-10-16T14:14:38.661 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113319.log.gz 2023-10-16T14:14:38.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152224.log 2023-10-16T14:14:38.663 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146385.log.gz 2023-10-16T14:14:38.664 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138688.log 2023-10-16T14:14:38.664 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153586.log.gz 2023-10-16T14:14:38.665 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111677.log 2023-10-16T14:14:38.665 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152224.log.gz 2023-10-16T14:14:38.666 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92208.log 2023-10-16T14:14:38.666 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138688.log.gz 2023-10-16T14:14:38.667 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127014.log 2023-10-16T14:14:38.668 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111677.log.gz 2023-10-16T14:14:38.669 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98232.log 2023-10-16T14:14:38.669 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92208.log.gz 2023-10-16T14:14:38.670 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115972.log 2023-10-16T14:14:38.670 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127014.log.gz 2023-10-16T14:14:38.671 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99141.log 2023-10-16T14:14:38.671 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98232.log.gz 2023-10-16T14:14:38.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99951.log 2023-10-16T14:14:38.673 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115972.log.gz 2023-10-16T14:14:38.674 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100245.log 2023-10-16T14:14:38.674 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99141.log.gz 2023-10-16T14:14:38.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159463.log 2023-10-16T14:14:38.675 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99951.log.gz 2023-10-16T14:14:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100245.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100245.log.gz 2023-10-16T14:14:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109407.log 2023-10-16T14:14:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134809.log 2023-10-16T14:14:38.678 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159463.log.gz 2023-10-16T14:14:38.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146259.log 2023-10-16T14:14:38.679 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109407.log.gz 2023-10-16T14:14:38.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125296.log 2023-10-16T14:14:38.681 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134809.log.gz 2023-10-16T14:14:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146259.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.146259.log.gz 2023-10-16T14:14:38.682 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.136317.log 2023-10-16T14:14:38.682 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146899.log 2023-10-16T14:14:38.683 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125296.log.gz 2023-10-16T14:14:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.136317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136317.log.gz 2023-10-16T14:14:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123088.log 2023-10-16T14:14:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135642.log 2023-10-16T14:14:38.686 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146899.log.gz 2023-10-16T14:14:38.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123088.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123088.log.gz 2023-10-16T14:14:38.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157104.log 2023-10-16T14:14:38.688 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146469.log 2023-10-16T14:14:38.688 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135642.log.gz 2023-10-16T14:14:38.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157104.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157104.log.gz 2023-10-16T14:14:38.689 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139085.log 2023-10-16T14:14:38.690 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122942.log 2023-10-16T14:14:38.691 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.146469.log.gz 2023-10-16T14:14:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139085.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.139085.log.gz 2023-10-16T14:14:38.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120093.log 2023-10-16T14:14:38.693 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98106.log 2023-10-16T14:14:38.694 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.122942.log.gz 2023-10-16T14:14:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120093.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.120093.log.gz 2023-10-16T14:14:38.695 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147771.log 2023-10-16T14:14:38.696 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158981.log 2023-10-16T14:14:38.696 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98106.log.gz 2023-10-16T14:14:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98414.log 2023-10-16T14:14:38.697 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147771.log.gz 2023-10-16T14:14:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117167.log 2023-10-16T14:14:38.699 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158981.log.gz 2023-10-16T14:14:38.700 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98414.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.98414.log.gz 2023-10-16T14:14:38.700 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121646.log 2023-10-16T14:14:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158119.log 2023-10-16T14:14:38.701 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.117167.log.gz 2023-10-16T14:14:38.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121646.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121646.log.gz 2023-10-16T14:14:38.702 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139683.log 2023-10-16T14:14:38.703 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144207.log 2023-10-16T14:14:38.703 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158119.log.gz 2023-10-16T14:14:38.704 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159318.log 88.9% 2023-10-16T14:14:38.705 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.139683.log.gz 2023-10-16T14:14:38.706 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149068.log 2023-10-16T14:14:38.706 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.144207.log.gz 2023-10-16T14:14:38.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159318.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159318.log.gz 2023-10-16T14:14:38.707 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92453.log 2023-10-16T14:14:38.708 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146535.log 2023-10-16T14:14:38.708 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149068.log.gz 2023-10-16T14:14:38.709 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133554.log 2023-10-16T14:14:38.709 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92453.log.gz 2023-10-16T14:14:38.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122984.log 2023-10-16T14:14:38.711 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146535.log.gz 2023-10-16T14:14:38.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105610.log 2023-10-16T14:14:38.712 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.133554.log.gz 2023-10-16T14:14:38.712 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96526.log 2023-10-16T14:14:38.713 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122984.log.gz 2023-10-16T14:14:38.714 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105610.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105610.log.gz 2023-10-16T14:14:38.714 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96400.log 2023-10-16T14:14:38.715 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103678.log 2023-10-16T14:14:38.715 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96526.log.gz 2023-10-16T14:14:38.716 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96400.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.96400.log.gz -5 2023-10-16T14:14:38.716 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.123340.log 2023-10-16T14:14:38.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.103678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109187.log 2023-10-16T14:14:38.717 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103678.log.gz 2023-10-16T14:14:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123340.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123340.log.gz 2023-10-16T14:14:38.719 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104961.log 2023-10-16T14:14:38.719 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152392.log 2023-10-16T14:14:38.720 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109187.log.gz 2023-10-16T14:14:38.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104961.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.104961.log.gz 2023-10-16T14:14:38.721 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122211.log 2023-10-16T14:14:38.722 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151867.log 2023-10-16T14:14:38.722 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152392.log.gz 2023-10-16T14:14:38.723 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122211.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122211.log.gz 2023-10-16T14:14:38.723 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96196.log 2023-10-16T14:14:38.724 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153064.log 2023-10-16T14:14:38.725 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151867.log.gz 2023-10-16T14:14:38.725 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96196.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.96196.log.gz 2023-10-16T14:14:38.726 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156551.log 2023-10-16T14:14:38.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112954.log 2023-10-16T14:14:38.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156551.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153064.log.gz 2023-10-16T14:14:38.728 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.156551.log.gz 2023-10-16T14:14:38.729 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131692.log 2023-10-16T14:14:38.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112954.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112954.log.gz 2023-10-16T14:14:38.731 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100775.log 2023-10-16T14:14:38.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131692.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.131692.log.gz 2023-10-16T14:14:38.732 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119307.log 2023-10-16T14:14:38.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109502.log 2023-10-16T14:14:38.733 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100775.log.gz 2023-10-16T14:14:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119307.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119307.log.gz 2023-10-16T14:14:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137510.log 2023-10-16T14:14:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92460.log 2023-10-16T14:14:38.736 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109502.log.gz 2023-10-16T14:14:38.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137510.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137510.log.gz 2023-10-16T14:14:38.737 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134580.log 2023-10-16T14:14:38.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153544.log 2023-10-16T14:14:38.738 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92460.log.gz 2023-10-16T14:14:38.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134580.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134580.log.gz 2023-10-16T14:14:38.739 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160430.log 2023-10-16T14:14:38.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114085.log 2023-10-16T14:14:38.740 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153544.log.gz 2023-10-16T14:14:38.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160430.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.100077.log 2023-10-16T14:14:38.742 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.160430.log.gz 2023-10-16T14:14:38.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149749.log 2023-10-16T14:14:38.743 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114085.log.gz 2023-10-16T14:14:38.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100077.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100077.log.gz 2023-10-16T14:14:38.744 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118796.log 2023-10-16T14:14:38.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.114464.log 2023-10-16T14:14:38.745 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149749.log.gz 2023-10-16T14:14:38.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118796.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.118796.log.gz 2023-10-16T14:14:38.747 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113038.log 2023-10-16T14:14:38.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.114464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.114464.log.gz 2023-10-16T14:14:38.748 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109229.log 2023-10-16T14:14:38.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150584.log 2023-10-16T14:14:38.749 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113038.log.gz 2023-10-16T14:14:38.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147939.log 2023-10-16T14:14:38.750 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109229.log.gz 2023-10-16T14:14:38.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122848.log 2023-10-16T14:14:38.751 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150584.log.gz 2023-10-16T14:14:38.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147939.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147939.log.gz 2023-10-16T14:14:38.753 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117432.log 2023-10-16T14:14:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122848.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.122848.log.gz 2023-10-16T14:14:38.754 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97438.log 2023-10-16T14:14:38.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104918.log 2023-10-16T14:14:38.755 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117432.log.gz 2023-10-16T14:14:38.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97438.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97438.log.gz 2023-10-16T14:14:38.756 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132905.log 2023-10-16T14:14:38.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158548.log 2023-10-16T14:14:38.758 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104918.log.gz 2023-10-16T14:14:38.758 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132905.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.132905.log.gz 2023-10-16T14:14:38.759 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156892.log 2023-10-16T14:14:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158548.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158548.log.gz 2023-10-16T14:14:38.760 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126138.log 2023-10-16T14:14:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156892.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.156892.log.gz 2023-10-16T14:14:38.761 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140476.log 2023-10-16T14:14:38.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119663.log 2023-10-16T14:14:38.763 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126138.log.gz 2023-10-16T14:14:38.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140476.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.140476.log.gz 2023-10-16T14:14:38.764 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134683.log 2023-10-16T14:14:38.765 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115532.log 2023-10-16T14:14:38.765 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119663.log.gz 2023-10-16T14:14:38.766 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92810.log 2023-10-16T14:14:38.766 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.134683.log.gz 2023-10-16T14:14:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115532.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115532.log.gz 2023-10-16T14:14:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105778.log 2023-10-16T14:14:38.768 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97812.log 2023-10-16T14:14:38.769 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92810.log.gz 2023-10-16T14:14:38.769 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105778.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105778.log.gz 2023-10-16T14:14:38.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135254.log 2023-10-16T14:14:38.771 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97812.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97812.log.gz 2023-10-16T14:14:38.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117839.log 2023-10-16T14:14:38.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123821.log 2023-10-16T14:14:38.772 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135254.log.gz 2023-10-16T14:14:38.773 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117839.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117839.log.gz 2023-10-16T14:14:38.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159420.log 2023-10-16T14:14:38.774 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123821.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123821.log.gz 2023-10-16T14:14:38.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139043.log 2023-10-16T14:14:38.775 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130777.log 2023-10-16T14:14:38.776 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.159420.log.gz 2023-10-16T14:14:38.777 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139043.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.139043.log.gz 2023-10-16T14:14:38.777 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.115262.log 2023-10-16T14:14:38.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.130777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111177.log 2023-10-16T14:14:38.778 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130777.log.gz 2023-10-16T14:14:38.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115262.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.115262.log.gz 2023-10-16T14:14:38.779 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126864.log 2023-10-16T14:14:38.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92292.log 2023-10-16T14:14:38.781 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111177.log.gz 2023-10-16T14:14:38.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160388.log 2023-10-16T14:14:38.782 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126864.log.gz 2023-10-16T14:14:38.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92292.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.92292.log.gz 2023-10-16T14:14:38.783 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.95665.log 2023-10-16T14:14:38.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112996.log 2023-10-16T14:14:38.784 INFO:teuthology.orchestra.run.smithi060.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.160388.log.gz 2023-10-16T14:14:38.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95665.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95665.log.gz 2023-10-16T14:14:38.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115218.log 2023-10-16T14:14:38.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112996.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112996.log.gz -5 2023-10-16T14:14:38.787 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.152728.log 2023-10-16T14:14:38.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134149.log 2023-10-16T14:14:38.788 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115218.log.gz 2023-10-16T14:14:38.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152728.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.153841.log -- replaced with /var/log/ceph/ceph-client.admin.152728.log.gz 2023-10-16T14:14:38.789 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:38.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134149.log: gzip 89.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.134149.log.gz --verbose 2023-10-16T14:14:38.790 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.97114.log 2023-10-16T14:14:38.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102083.log 2023-10-16T14:14:38.792 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.153841.log.gz 2023-10-16T14:14:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97114.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97114.log.gz 2023-10-16T14:14:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.120388.log 2023-10-16T14:14:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.102083.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102083.log.gz 2023-10-16T14:14:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97156.log 2023-10-16T14:14:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.120388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.120388.log.gz 2023-10-16T14:14:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157284.log 2023-10-16T14:14:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97156.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.97156.log.gz 2023-10-16T14:14:38.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117101.log 2023-10-16T14:14:38.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156112.log 2023-10-16T14:14:38.798 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157284.log.gz 2023-10-16T14:14:38.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125963.log 2023-10-16T14:14:38.799 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.117101.log.gz 2023-10-16T14:14:38.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115719.log 2023-10-16T14:14:38.800 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.156112.log.gz 2023-10-16T14:14:38.801 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125963.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125963.log.gz 2023-10-16T14:14:38.801 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112116.log 2023-10-16T14:14:38.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100161.log 2023-10-16T14:14:38.803 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115719.log.gz 2023-10-16T14:14:38.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112116.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112116.log.gz 2023-10-16T14:14:38.804 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151176.log 2023-10-16T14:14:38.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93510.log 2023-10-16T14:14:38.806 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100161.log.gz 2023-10-16T14:14:38.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151176.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151176.log.gz 2023-10-16T14:14:38.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104754.log 2023-10-16T14:14:38.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150542.log 2023-10-16T14:14:38.808 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93510.log.gz 2023-10-16T14:14:38.809 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104754.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T14:14:38.809 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104754.log.gz 2023-10-16T14:14:38.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148191.log 2023-10-16T14:14:38.810 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.150542.log.gz 2023-10-16T14:14:38.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135420.log 2023-10-16T14:14:38.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115803.log 2023-10-16T14:14:38.813 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148191.log.gz 2023-10-16T14:14:38.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135420.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.135420.log.gz 2023-10-16T14:14:38.814 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121247.log 2023-10-16T14:14:38.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154011.log 2023-10-16T14:14:38.815 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115803.log.gz 2023-10-16T14:14:38.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121247.log: gzip 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T14:14:38.817 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.146133.log 2023-10-16T14:14:38.817 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121247.log.gz 2023-10-16T14:14:38.818 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154011.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154011.log.gz 2023-10-16T14:14:38.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95959.log 2023-10-16T14:14:38.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146133.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146133.log.gz 2023-10-16T14:14:38.819 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142933.log 2023-10-16T14:14:38.820 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116567.log 2023-10-16T14:14:38.821 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95959.log.gz 2023-10-16T14:14:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.142933.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142933.log.gz 2023-10-16T14:14:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154319.log 2023-10-16T14:14:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116873.log 2023-10-16T14:14:38.822 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116567.log.gz 2023-10-16T14:14:38.823 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154319.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154319.log.gz 2023-10-16T14:14:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93144.log 2023-10-16T14:14:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139001.log 2023-10-16T14:14:38.825 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116873.log.gz 2023-10-16T14:14:38.826 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93144.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93144.log.gz 2023-10-16T14:14:38.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157198.log 2023-10-16T14:14:38.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144948.log 2023-10-16T14:14:38.827 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139001.log.gz 2023-10-16T14:14:38.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157198.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.157198.log.gz 2023-10-16T14:14:38.828 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114650.log 2023-10-16T14:14:38.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105171.log 2023-10-16T14:14:38.829 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.144948.log.gz 2023-10-16T14:14:38.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114650.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114650.log.gz 2023-10-16T14:14:38.831 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116056.log 2023-10-16T14:14:38.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153502.log 2023-10-16T14:14:38.832 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105171.log.gz 2023-10-16T14:14:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116056.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116056.log.gz 2023-10-16T14:14:38.832 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121814.log 2023-10-16T14:14:38.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105484.log 2023-10-16T14:14:38.833 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.153502.log.gz 2023-10-16T14:14:38.834 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121814.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121814.log.gz 2023-10-16T14:14:38.835 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127387.log 2023-10-16T14:14:38.835 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105484.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105484.log.gz 2023-10-16T14:14:38.836 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113831.log 2023-10-16T14:14:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127387.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.127387.log.gz 2023-10-16T14:14:38.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105694.log 2023-10-16T14:14:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113831.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113831.log.gz 2023-10-16T14:14:38.838 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95623.log 2023-10-16T14:14:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105694.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105694.log.gz 2023-10-16T14:14:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145921.log 2023-10-16T14:14:38.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122337.log 2023-10-16T14:14:38.840 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.95623.log.gz 2023-10-16T14:14:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145921.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145921.log.gz 2023-10-16T14:14:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132989.log 2023-10-16T14:14:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110967.log 2023-10-16T14:14:38.843 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122337.log.gz 2023-10-16T14:14:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132989.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.132989.log.gz 2023-10-16T14:14:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115888.log 2023-10-16T14:14:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155111.log 2023-10-16T14:14:38.845 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110967.log.gz 2023-10-16T14:14:38.846 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121898.log 2023-10-16T14:14:38.846 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.115888.log.gz 2023-10-16T14:14:38.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118042.log 2023-10-16T14:14:38.847 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.155111.log.gz 2023-10-16T14:14:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124939.log 2023-10-16T14:14:38.848 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121898.log.gz 2023-10-16T14:14:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118042.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118042.log.gz 2023-10-16T14:14:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99909.log 2023-10-16T14:14:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140798.log 2023-10-16T14:14:38.850 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124939.log.gz 2023-10-16T14:14:38.851 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99909.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99909.log.gz 2023-10-16T14:14:38.852 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125422.log 2023-10-16T14:14:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138730.log 2023-10-16T14:14:38.853 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.140798.log.gz 2023-10-16T14:14:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125422.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.125422.log.gz 2023-10-16T14:14:38.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158709.log 2023-10-16T14:14:38.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106711.log 2023-10-16T14:14:38.855 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138730.log.gz 2023-10-16T14:14:38.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106669.log 2023-10-16T14:14:38.856 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158709.log.gz 2023-10-16T14:14:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123992.log 2023-10-16T14:14:38.857 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106711.log.gz 2023-10-16T14:14:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106669.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106669.log.gz 2023-10-16T14:14:38.858 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148527.log 2023-10-16T14:14:38.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135548.log 2023-10-16T14:14:38.859 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123992.log.gz 2023-10-16T14:14:38.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148527.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.148527.log.gz 2023-10-16T14:14:38.860 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135061.log 2023-10-16T14:14:38.861 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135548.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.135548.log.gz 2023-10-16T14:14:38.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148798.log 2023-10-16T14:14:38.862 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125170.log 2023-10-16T14:14:38.862 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135061.log.gz 2023-10-16T14:14:38.863 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148798.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148798.log.gz 2023-10-16T14:14:38.863 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123172.log 2023-10-16T14:14:38.864 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125170.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.125170.log.gz 2023-10-16T14:14:38.865 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92040.log 2023-10-16T14:14:38.865 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123172.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.123172.log.gz 2023-10-16T14:14:38.866 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136130.log 2023-10-16T14:14:38.866 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99367.log 2023-10-16T14:14:38.866 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92040.log.gz 2023-10-16T14:14:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136130.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136130.log.gz 2023-10-16T14:14:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132821.log 2023-10-16T14:14:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136064.log 2023-10-16T14:14:38.868 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99367.log.gz 2023-10-16T14:14:38.870 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132821.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.132821.log.gz 2023-10-16T14:14:38.870 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105213.log 2023-10-16T14:14:38.870 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96988.log 2023-10-16T14:14:38.871 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105213.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.136064.log.gz 2023-10-16T14:14:38.871 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105213.log.gz 2023-10-16T14:14:38.872 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145266.log 2023-10-16T14:14:38.873 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112870.log 2023-10-16T14:14:38.873 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96988.log.gz 2023-10-16T14:14:38.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145266.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145266.log.gz 2023-10-16T14:14:38.874 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124813.log 2023-10-16T14:14:38.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92808.log 2023-10-16T14:14:38.875 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112870.log.gz 2023-10-16T14:14:38.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119799.log 2023-10-16T14:14:38.877 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124813.log.gz 2023-10-16T14:14:38.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92808.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92808.log.gz 2023-10-16T14:14:38.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97311.log 2023-10-16T14:14:38.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95455.log 2023-10-16T14:14:38.879 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119799.log.gz 2023-10-16T14:14:38.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105953.log 2023-10-16T14:14:38.880 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97311.log.gz 2023-10-16T14:14:38.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130896.log 2023-10-16T14:14:38.881 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95455.log.gz 2023-10-16T14:14:38.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105953.log.gz 2023-10-16T14:14:38.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159024.log 2023-10-16T14:14:38.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.130896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96112.log 2023-10-16T14:14:38.883 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130896.log.gz 2023-10-16T14:14:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159024.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159024.log.gz 2023-10-16T14:14:38.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93636.log 2023-10-16T14:14:38.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152476.log 2023-10-16T14:14:38.886 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96112.log.gz 2023-10-16T14:14:38.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93636.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93636.log.gz 2023-10-16T14:14:38.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105129.log 2023-10-16T14:14:38.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105255.log 2023-10-16T14:14:38.888 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.152476.log.gz 2023-10-16T14:14:38.889 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140584.log 2023-10-16T14:14:38.889 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105129.log.gz 2023-10-16T14:14:38.890 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105255.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105255.log.gz 2023-10-16T14:14:38.890 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119841.log 2023-10-16T14:14:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136395.log 2023-10-16T14:14:38.892 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140584.log.gz 2023-10-16T14:14:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144382.log 2023-10-16T14:14:38.893 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119841.log.gz 2023-10-16T14:14:38.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105442.log 2023-10-16T14:14:38.894 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.136395.log.gz 2023-10-16T14:14:38.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144382.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144382.log.gz 2023-10-16T14:14:38.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91688.log 2023-10-16T14:14:38.897 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124410.log 2023-10-16T14:14:38.897 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105442.log.gz 2023-10-16T14:14:38.898 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91786.log 2023-10-16T14:14:38.898 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91688.log.gz 2023-10-16T14:14:38.899 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.124410.log: /var/log/ceph/ceph-client.admin.122720.log 2023-10-16T14:14:38.900 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124410.log.gz 2023-10-16T14:14:38.900 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136437.log 2023-10-16T14:14:38.901 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91786.log.gz 2023-10-16T14:14:38.902 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122720.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.122720.log.gz 2023-10-16T14:14:38.902 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152182.log 2023-10-16T14:14:38.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110641.log 2023-10-16T14:14:38.903 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.136437.log.gz 2023-10-16T14:14:38.904 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152182.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.152182.log.gz -5 2023-10-16T14:14:38.904 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.125033.log 2023-10-16T14:14:38.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110641.log: gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.110641.log.gz /var/log/ceph/ceph-client.admin.106119.log 2023-10-16T14:14:38.906 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:38.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114584.log 2023-10-16T14:14:38.907 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125033.log.gz 2023-10-16T14:14:38.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106119.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106119.log.gz 2023-10-16T14:14:38.909 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109670.log 2023-10-16T14:14:38.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117059.log 2023-10-16T14:14:38.910 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114584.log.gz 2023-10-16T14:14:38.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154149.log 2023-10-16T14:14:38.911 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109670.log.gz 2023-10-16T14:14:38.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117059.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.117059.log.gz 2023-10-16T14:14:38.912 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.104866.log 2023-10-16T14:14:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111135.log 2023-10-16T14:14:38.914 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154149.log.gz 2023-10-16T14:14:38.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104866.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104866.log.gz 2023-10-16T14:14:38.915 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121772.log 2023-10-16T14:14:38.916 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132719.log 2023-10-16T14:14:38.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111135.log.gz 2023-10-16T14:14:38.917 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121772.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.121772.log.gz 2023-10-16T14:14:38.917 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.117354.log 2023-10-16T14:14:38.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160346.log 2023-10-16T14:14:38.918 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.132719.log.gz 2023-10-16T14:14:38.919 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.117354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.117354.log.gz 2023-10-16T14:14:38.920 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98708.log 2023-10-16T14:14:38.921 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96484.log 2023-10-16T14:14:38.921 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.160346.log.gz 2023-10-16T14:14:38.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98708.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98708.log.gz 2023-10-16T14:14:38.922 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110923.log 2023-10-16T14:14:38.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116523.log 2023-10-16T14:14:38.923 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96484.log.gz 2023-10-16T14:14:38.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110923.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110923.log.gz 2023-10-16T14:14:38.924 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153148.log 2023-10-16T14:14:38.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112032.log 2023-10-16T14:14:38.926 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116523.log.gz 2023-10-16T14:14:38.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153148.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153148.log.gz 2023-10-16T14:14:38.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114786.log 2023-10-16T14:14:38.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143703.log 2023-10-16T14:14:38.928 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112032.log.gz 2023-10-16T14:14:38.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114786.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114786.log.gz 2023-10-16T14:14:38.929 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148924.log 2023-10-16T14:14:38.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143703.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143703.log.gz 2023-10-16T14:14:38.931 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159589.log 2023-10-16T14:14:38.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148924.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148924.log.gz 2023-10-16T14:14:38.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150127.log 2023-10-16T14:14:38.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.157891.log 2023-10-16T14:14:38.933 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.159589.log.gz 2023-10-16T14:14:38.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120675.log 2023-10-16T14:14:38.934 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.150127.log.gz 2023-10-16T14:14:38.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.157891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.157891.log.gz 2023-10-16T14:14:38.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158929.log 2023-10-16T14:14:38.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134412.log 2023-10-16T14:14:38.937 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120675.log.gz 2023-10-16T14:14:38.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149008.log 2023-10-16T14:14:38.938 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158929.log.gz 2023-10-16T14:14:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134412.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134412.log.gz 2023-10-16T14:14:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124051.log 2023-10-16T14:14:38.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114692.log 2023-10-16T14:14:38.940 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.149008.log.gz 2023-10-16T14:14:38.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122892.log 2023-10-16T14:14:38.941 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.124051.log.gz 2023-10-16T14:14:38.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114692.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114692.log.gz 2023-10-16T14:14:38.942 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96001.log 2023-10-16T14:14:38.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110725.log 2023-10-16T14:14:38.944 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122892.log.gz 2023-10-16T14:14:38.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.133326.log 2023-10-16T14:14:38.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.96001.log.gz 2023-10-16T14:14:38.946 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110725.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110725.log.gz 2023-10-16T14:14:38.946 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151302.log 2023-10-16T14:14:38.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.133326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.133326.log.gz 2023-10-16T14:14:38.947 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101466.log 2023-10-16T14:14:38.948 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151302.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151302.log.gz 2023-10-16T14:14:38.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157969.log 2023-10-16T14:14:38.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137858.log 2023-10-16T14:14:38.950 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101466.log.gz 2023-10-16T14:14:38.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157969.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.157969.log.gz 2023-10-16T14:14:38.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97649.log 2023-10-16T14:14:38.952 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157368.log 2023-10-16T14:14:38.952 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.137858.log.gz 2023-10-16T14:14:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115846.log 2023-10-16T14:14:38.953 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97649.log.gz 2023-10-16T14:14:38.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157368.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157368.log.gz 2023-10-16T14:14:38.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123863.log 2023-10-16T14:14:38.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147067.log 2023-10-16T14:14:38.956 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.115846.log.gz 2023-10-16T14:14:38.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123863.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123863.log.gz 2023-10-16T14:14:38.957 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.156299.log 2023-10-16T14:14:38.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147067.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147067.log.gz 2023-10-16T14:14:38.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143619.log 2023-10-16T14:14:38.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98750.log 2023-10-16T14:14:38.959 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156299.log.gz 2023-10-16T14:14:38.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143619.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.143619.log.gz 2023-10-16T14:14:38.960 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115031.log 2023-10-16T14:14:38.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98750.log: gzip -5 87.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.98750.log.gz -- 2023-10-16T14:14:38.962 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.143451.log 2023-10-16T14:14:38.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93678.log 2023-10-16T14:14:38.963 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115031.log.gz 2023-10-16T14:14:38.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143451.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.143451.log.gz 2023-10-16T14:14:38.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114211.log 2023-10-16T14:14:38.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93678.log: gzip -5 88.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.93678.log.gz -- 2023-10-16T14:14:38.965 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.125585.log 2023-10-16T14:14:38.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102626.log 2023-10-16T14:14:38.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125585.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.114211.log.gz 2023-10-16T14:14:38.968 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.125585.log.gz 2023-10-16T14:14:38.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143535.log 2023-10-16T14:14:38.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.102626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126654.log 2023-10-16T14:14:38.969 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102626.log.gz 2023-10-16T14:14:38.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95791.log 2023-10-16T14:14:38.971 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143535.log.gz 2023-10-16T14:14:38.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126654.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126654.log.gz 2023-10-16T14:14:38.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110174.log 2023-10-16T14:14:38.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156509.log 88.5% 2023-10-16T14:14:38.973 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95791.log.gz 2023-10-16T14:14:38.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138562.log 2023-10-16T14:14:38.974 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110174.log.gz 2023-10-16T14:14:38.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156509.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156509.log.gz 2023-10-16T14:14:38.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92473.log 2023-10-16T14:14:38.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96658.log 2023-10-16T14:14:38.977 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138562.log.gz 2023-10-16T14:14:38.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92473.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92473.log.gz 2023-10-16T14:14:38.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153681.log 2023-10-16T14:14:38.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139977.log 2023-10-16T14:14:38.980 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96658.log.gz 2023-10-16T14:14:38.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153681.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153681.log.gz 2023-10-16T14:14:38.981 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98498.log 2023-10-16T14:14:38.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143745.log 2023-10-16T14:14:38.983 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.139977.log.gz 2023-10-16T14:14:38.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98498.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98498.log.gz 2023-10-16T14:14:38.983 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-16T14:14:38.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143745.log: gzip 87.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.143745.log.gz -- 2023-10-16T14:14:38.985 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.91914.log 2023-10-16T14:14:38.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153106.log 2023-10-16T14:14:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91914.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91914.log.gz 2023-10-16T14:14:38.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102363.log 2023-10-16T14:14:38.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106585.log 2023-10-16T14:14:38.989 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.153106.log.gz 2023-10-16T14:14:38.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.102363.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102363.log.gz 2023-10-16T14:14:38.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159150.log 2023-10-16T14:14:38.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137977.log 2023-10-16T14:14:38.991 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106585.log.gz 2023-10-16T14:14:38.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159150.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159150.log.gz 2023-10-16T14:14:38.992 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148840.log 2023-10-16T14:14:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143052.log 2023-10-16T14:14:38.994 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137977.log.gz 2023-10-16T14:14:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148840.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148840.log.gz 2023-10-16T14:14:38.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116747.log 2023-10-16T14:14:38.997 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158423.log 2023-10-16T14:14:38.997 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143052.log.gz 2023-10-16T14:14:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116747.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116747.log.gz 2023-10-16T14:14:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126261.log 2023-10-16T14:14:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96154.log 2023-10-16T14:14:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126261.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158423.log.gz 2023-10-16T14:14:39.001 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126261.log.gz 2023-10-16T14:14:39.001 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126568.log 2023-10-16T14:14:39.006 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100035.log 2023-10-16T14:14:39.007 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96154.log.gz 2023-10-16T14:14:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126568.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.126568.log.gz 2023-10-16T14:14:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132070.log 2023-10-16T14:14:39.008 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95161.log 2023-10-16T14:14:39.008 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100035.log.gz 2023-10-16T14:14:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132070.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.132070.log.gz 2023-10-16T14:14:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94575.log 2023-10-16T14:14:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152938.log 2023-10-16T14:14:39.010 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95161.log.gz 2023-10-16T14:14:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.94575.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94575.log.gz 2023-10-16T14:14:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131421.log 2023-10-16T14:14:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105003.log 2023-10-16T14:14:39.012 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.152938.log.gz 2023-10-16T14:14:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131421.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.131421.log.gz 2023-10-16T14:14:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135938.log 2023-10-16T14:14:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145657.log 2023-10-16T14:14:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135938.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105003.log.gz 2023-10-16T14:14:39.015 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135938.log.gz 2023-10-16T14:14:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124169.log 2023-10-16T14:14:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120051.log 2023-10-16T14:14:39.017 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145657.log.gz 2023-10-16T14:14:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124169.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124169.log.gz 2023-10-16T14:14:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127279.log 2023-10-16T14:14:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126526.log 2023-10-16T14:14:39.020 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120051.log.gz 2023-10-16T14:14:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127279.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127279.log.gz 2023-10-16T14:14:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139851.log 2023-10-16T14:14:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113695.log 2023-10-16T14:14:39.022 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126526.log.gz 2023-10-16T14:14:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139851.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139851.log.gz 2023-10-16T14:14:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115490.log 2023-10-16T14:14:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136503.log 2023-10-16T14:14:39.024 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113695.log.gz 2023-10-16T14:14:39.025 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115490.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115490.log.gz 2023-10-16T14:14:39.026 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106333.log 2023-10-16T14:14:39.027 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127488.log 2023-10-16T14:14:39.027 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136503.log.gz 2023-10-16T14:14:39.028 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106333.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106333.log.gz 2023-10-16T14:14:39.028 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99555.log 2023-10-16T14:14:39.029 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97896.log 2023-10-16T14:14:39.029 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127488.log.gz 2023-10-16T14:14:39.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99555.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99555.log.gz 2023-10-16T14:14:39.030 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118711.log 2023-10-16T14:14:39.031 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152560.log 2023-10-16T14:14:39.032 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97896.log.gz 2023-10-16T14:14:39.032 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118711.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118711.log.gz 2023-10-16T14:14:39.033 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156070.log 2023-10-16T14:14:39.034 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93846.log 2023-10-16T14:14:39.034 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152560.log.gz 2023-10-16T14:14:39.035 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156070.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.156070.log.gz 2023-10-16T14:14:39.036 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101230.log 2023-10-16T14:14:39.037 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113400.log 2023-10-16T14:14:39.037 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93846.log.gz 2023-10-16T14:14:39.038 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101230.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.101230.log.gz 2023-10-16T14:14:39.038 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132495.log 2023-10-16T14:14:39.039 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113400.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113400.log.gz 2023-10-16T14:14:39.039 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139809.log 2023-10-16T14:14:39.040 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132495.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.132495.log.gz 2023-10-16T14:14:39.040 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.81319.log 2023-10-16T14:14:39.041 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116193.log 2023-10-16T14:14:39.041 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.139809.log.gz 2023-10-16T14:14:39.042 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.81319.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.81319.log.gz 2023-10-16T14:14:39.043 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130980.log 2023-10-16T14:14:39.043 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116193.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116193.log.gz 2023-10-16T14:14:39.044 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99249.log 2023-10-16T14:14:39.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.130980.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.130980.log.gz 2023-10-16T14:14:39.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95749.log 2023-10-16T14:14:39.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99249.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.99249.log.gz 2023-10-16T14:14:39.047 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.152980.log 2023-10-16T14:14:39.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95749.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95749.log.gz 2023-10-16T14:14:39.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138145.log 2023-10-16T14:14:39.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134055.log 2023-10-16T14:14:39.049 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152980.log.gz 2023-10-16T14:14:39.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138145.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138145.log.gz 2023-10-16T14:14:39.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148882.log 2023-10-16T14:14:39.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134055.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134055.log.gz 2023-10-16T14:14:39.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149152.log 2023-10-16T14:14:39.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148882.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148882.log.gz 2023-10-16T14:14:39.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155723.log 2023-10-16T14:14:39.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149152.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.149152.log.gz 2023-10-16T14:14:39.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131295.log 2023-10-16T14:14:39.055 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155723.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155723.log.gz 2023-10-16T14:14:39.055 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157704.log 2023-10-16T14:14:39.056 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131295.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131295.log.gz 2023-10-16T14:14:39.056 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147126.log 2023-10-16T14:14:39.057 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157704.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157704.log.gz 2023-10-16T14:14:39.057 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98456.log 2023-10-16T14:14:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156257.log 2023-10-16T14:14:39.059 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147126.log.gz 2023-10-16T14:14:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98456.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98456.log.gz 2023-10-16T14:14:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160783.log 2023-10-16T14:14:39.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156257.log: gzip -5 87.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.156257.log.gz 2023-10-16T14:14:39.061 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.119925.log 2023-10-16T14:14:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160783.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.160783.log.gz 2023-10-16T14:14:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98276.log 2023-10-16T14:14:39.063 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137409.log 2023-10-16T14:14:39.063 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.119925.log.gz 2023-10-16T14:14:39.064 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98276.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.98276.log.gz 2023-10-16T14:14:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109838.log 2023-10-16T14:14:39.065 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137409.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.137409.log.gz 2023-10-16T14:14:39.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147603.log 2023-10-16T14:14:39.067 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109838.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109838.log.gz 2023-10-16T14:14:39.067 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111051.log 2023-10-16T14:14:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147603.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147603.log.gz 2023-10-16T14:14:39.068 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109754.log 2023-10-16T14:14:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111051.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111051.log.gz 2023-10-16T14:14:39.069 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150043.log 2023-10-16T14:14:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132165.log 2023-10-16T14:14:39.071 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109754.log.gz 2023-10-16T14:14:39.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150043.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150043.log.gz 2023-10-16T14:14:39.072 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138875.log 2023-10-16T14:14:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132165.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.132165.log.gz 2023-10-16T14:14:39.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148107.log 2023-10-16T14:14:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138875.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138875.log.gz 2023-10-16T14:14:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147498.log 2023-10-16T14:14:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148107.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148107.log.gz 2023-10-16T14:14:39.075 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110258.log 2023-10-16T14:14:39.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147498.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147498.log.gz 2023-10-16T14:14:39.077 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118220.log 2023-10-16T14:14:39.077 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110258.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110258.log.gz 2023-10-16T14:14:39.078 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131734.log 2023-10-16T14:14:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118220.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.118220.log.gz 2023-10-16T14:14:39.079 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113739.log 2023-10-16T14:14:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131734.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131734.log.gz 2023-10-16T14:14:39.080 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156383.log 2023-10-16T14:14:39.081 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113739.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113739.log.gz 2023-10-16T14:14:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109880.log 2023-10-16T14:14:39.082 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153460.log 2023-10-16T14:14:39.083 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156383.log.gz 2023-10-16T14:14:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109880.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109880.log.gz 2023-10-16T14:14:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122762.log 2023-10-16T14:14:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153460.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153460.log.gz 2023-10-16T14:14:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133929.log 2023-10-16T14:14:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122762.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122762.log.gz 2023-10-16T14:14:39.086 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125128.log 2023-10-16T14:14:39.087 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150840.log 2023-10-16T14:14:39.087 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133929.log.gz 2023-10-16T14:14:39.088 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125128.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.125128.log.gz 2023-10-16T14:14:39.089 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120135.log 2023-10-16T14:14:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133710.log 2023-10-16T14:14:39.090 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150840.log.gz 2023-10-16T14:14:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101761.log 2023-10-16T14:14:39.091 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.120135.log.gz 2023-10-16T14:14:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133710.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.133710.log.gz 2023-10-16T14:14:39.092 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.112200.log 2023-10-16T14:14:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101761.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101761.log.gz 2023-10-16T14:14:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95833.log 2023-10-16T14:14:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150626.log 2023-10-16T14:14:39.095 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112200.log.gz 2023-10-16T14:14:39.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95833.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95833.log.gz 2023-10-16T14:14:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127601.log 2023-10-16T14:14:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150626.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150626.log.gz 2023-10-16T14:14:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116439.log 2023-10-16T14:14:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127601.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.127601.log.gz 2023-10-16T14:14:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158011.log 2023-10-16T14:14:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133971.log 2023-10-16T14:14:39.099 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116439.log.gz 2023-10-16T14:14:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158011.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158011.log.gz 2023-10-16T14:14:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98540.log 2023-10-16T14:14:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156976.log 2023-10-16T14:14:39.102 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.133971.log.gz 2023-10-16T14:14:39.102 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98540.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98540.log.gz 2023-10-16T14:14:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96280.log 2023-10-16T14:14:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156976.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156976.log.gz 2023-10-16T14:14:39.104 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135684.log 2023-10-16T14:14:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96280.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96280.log.gz 2023-10-16T14:14:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106203.log 2023-10-16T14:14:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135684.log: gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.135684.log.gz /var/log/ceph/ceph-client.admin.150924.log 2023-10-16T14:14:39.106 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106203.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106203.log.gz 2023-10-16T14:14:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147288.log 2023-10-16T14:14:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148611.log 2023-10-16T14:14:39.109 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.150924.log.gz 2023-10-16T14:14:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147288.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147288.log.gz 2023-10-16T14:14:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159066.log 2023-10-16T14:14:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117797.log 2023-10-16T14:14:39.111 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148611.log.gz 2023-10-16T14:14:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159066.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159066.log.gz 2023-10-16T14:14:39.112 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135770.log 2023-10-16T14:14:39.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119967.log 2023-10-16T14:14:39.113 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.117797.log.gz 2023-10-16T14:14:39.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135770.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135770.log.gz 2023-10-16T14:14:39.115 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138103.log 2023-10-16T14:14:39.116 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154637.log 2023-10-16T14:14:39.116 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119967.log.gz 2023-10-16T14:14:39.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138103.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138103.log.gz 2023-10-16T14:14:39.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119406.log 2023-10-16T14:14:39.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112912.log 2023-10-16T14:14:39.118 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154637.log.gz 2023-10-16T14:14:39.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119406.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119406.log.gz 2023-10-16T14:14:39.120 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93137.log 2023-10-16T14:14:39.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114001.log 2023-10-16T14:14:39.121 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112912.log.gz 2023-10-16T14:14:39.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93137.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93137.log.gz 2023-10-16T14:14:39.122 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148401.log 2023-10-16T14:14:39.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140685.log 2023-10-16T14:14:39.124 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114001.log.gz 2023-10-16T14:14:39.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148401.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148401.log.gz 2023-10-16T14:14:39.125 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139468.log 2023-10-16T14:14:39.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146175.log 2023-10-16T14:14:39.126 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.140685.log.gz 2023-10-16T14:14:39.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139468.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.139468.log.gz 2023-10-16T14:14:39.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120616.log 2023-10-16T14:14:39.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146983.log 2023-10-16T14:14:39.129 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146175.log.gz 2023-10-16T14:14:39.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120616.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.120616.log.gz --verbose 2023-10-16T14:14:39.130 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.131986.log 2023-10-16T14:14:39.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122001.log 2023-10-16T14:14:39.131 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146983.log.gz 2023-10-16T14:14:39.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131986.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.131986.log.gz 2023-10-16T14:14:39.133 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111219.log 2023-10-16T14:14:39.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124369.log 2023-10-16T14:14:39.134 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122001.log.gz 2023-10-16T14:14:39.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111219.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.111219.log.gz 2023-10-16T14:14:39.135 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.118567.log 2023-10-16T14:14:39.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101171.log 2023-10-16T14:14:39.137 INFO:teuthology.orchestra.run.smithi060.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.124369.log.gz 2023-10-16T14:14:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143136.log 2023-10-16T14:14:39.138 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118567.log.gz 2023-10-16T14:14:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101171.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101171.log.gz -5 2023-10-16T14:14:39.139 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.149707.log 2023-10-16T14:14:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114947.log 2023-10-16T14:14:39.141 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.143136.log.gz 2023-10-16T14:14:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157552.log 2023-10-16T14:14:39.142 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149707.log.gz 2023-10-16T14:14:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114947.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.114947.log.gz 2023-10-16T14:14:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140145.log 2023-10-16T14:14:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101803.log 2023-10-16T14:14:39.145 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.157552.log.gz 2023-10-16T14:14:39.146 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148653.log 2023-10-16T14:14:39.146 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.140145.log.gz 2023-10-16T14:14:39.147 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101803.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101803.log.gz 2023-10-16T14:14:39.148 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-16T14:14:39.149 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130938.log 2023-10-16T14:14:39.149 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148653.log.gz 2023-10-16T14:14:39.150 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135854.log 2023-10-16T14:14:39.151 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.130938.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130938.log.gz 2023-10-16T14:14:39.152 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152266.log 2023-10-16T14:14:39.153 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135728.log 2023-10-16T14:14:39.154 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135854.log: /var/log/ceph/ceph-client.admin.152266.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135854.log.gz 2023-10-16T14:14:39.155 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.152266.log.gz 2023-10-16T14:14:39.155 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139595.log 2023-10-16T14:14:39.156 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117624.log 2023-10-16T14:14:39.156 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135728.log.gz 2023-10-16T14:14:39.157 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139595.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139595.log.gz 2023-10-16T14:14:39.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95119.log 2023-10-16T14:14:39.159 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132411.log 2023-10-16T14:14:39.159 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117624.log: /var/log/ceph/ceph-client.admin.95119.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117624.log.gz 2023-10-16T14:14:39.160 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95119.log.gz 2023-10-16T14:14:39.161 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111635.log 2023-10-16T14:14:39.161 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98624.log 2023-10-16T14:14:39.162 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.132411.log.gz 2023-10-16T14:14:39.163 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111635.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111635.log.gz 2023-10-16T14:14:39.163 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93804.log 2023-10-16T14:14:39.164 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155248.log 2023-10-16T14:14:39.165 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98624.log.gz 2023-10-16T14:14:39.166 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93804.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93804.log.gz 2023-10-16T14:14:39.166 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153232.log 2023-10-16T14:14:39.167 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143178.log 2023-10-16T14:14:39.167 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155248.log.gz 2023-10-16T14:14:39.168 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153232.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.153232.log.gz 2023-10-16T14:14:39.169 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143955.log 2023-10-16T14:14:39.170 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92082.log 2023-10-16T14:14:39.170 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.143178.log.gz 2023-10-16T14:14:39.171 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143955.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143955.log.gz 2023-10-16T14:14:39.171 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146643.log 2023-10-16T14:14:39.172 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93552.log 2023-10-16T14:14:39.173 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92082.log.gz 2023-10-16T14:14:39.173 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146643.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146643.log.gz 2023-10-16T14:14:39.174 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145699.log 2023-10-16T14:14:39.175 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146343.log 2023-10-16T14:14:39.176 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93552.log.gz 2023-10-16T14:14:39.176 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145699.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145699.log.gz 2023-10-16T14:14:39.177 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138061.log 2023-10-16T14:14:39.178 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95287.log 2023-10-16T14:14:39.178 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.146343.log.gz 2023-10-16T14:14:39.179 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138061.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138061.log.gz 2023-10-16T14:14:39.180 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148485.log 2023-10-16T14:14:39.180 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151580.log 2023-10-16T14:14:39.181 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95287.log.gz 2023-10-16T14:14:39.182 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148485.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.148485.log.gz 2023-10-16T14:14:39.182 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109922.log 2023-10-16T14:14:39.183 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118651.log 2023-10-16T14:14:39.183 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151580.log.gz 2023-10-16T14:14:39.184 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109922.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109922.log.gz 2023-10-16T14:14:39.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101003.log 2023-10-16T14:14:39.186 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152434.log 2023-10-16T14:14:39.186 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.118651.log.gz 2023-10-16T14:14:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101003.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101003.log.gz 2023-10-16T14:14:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118609.log 2023-10-16T14:14:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91532.log 2023-10-16T14:14:39.189 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152434.log.gz 2023-10-16T14:14:39.190 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118609.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118609.log.gz 2023-10-16T14:14:39.190 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146301.log 2023-10-16T14:14:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131547.log 2023-10-16T14:14:39.192 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91532.log.gz 2023-10-16T14:14:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146301.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.146301.log.gz 2023-10-16T14:14:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116915.log 2023-10-16T14:14:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101045.log 2023-10-16T14:14:39.194 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131547.log.gz 2023-10-16T14:14:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116915.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116915.log.gz 2023-10-16T14:14:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120009.log 2023-10-16T14:14:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151344.log 2023-10-16T14:14:39.197 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101045.log.gz 2023-10-16T14:14:39.198 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92468.log 2023-10-16T14:14:39.199 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120009.log.gz 2023-10-16T14:14:39.199 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111009.log 2023-10-16T14:14:39.200 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151344.log.gz 2023-10-16T14:14:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92468.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92468.log.gz 2023-10-16T14:14:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151386.log 2023-10-16T14:14:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114734.log 2023-10-16T14:14:39.202 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111009.log.gz 2023-10-16T14:14:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91632.log 2023-10-16T14:14:39.204 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151386.log.gz 2023-10-16T14:14:39.205 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145741.log 2023-10-16T14:14:39.205 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114734.log.gz 2023-10-16T14:14:39.206 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91632.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91632.log.gz 2023-10-16T14:14:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134328.log 2023-10-16T14:14:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125212.log 2023-10-16T14:14:39.208 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145741.log.gz 2023-10-16T14:14:39.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134328.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134328.log.gz 2023-10-16T14:14:39.209 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105297.log 2023-10-16T14:14:39.210 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143409.log 2023-10-16T14:14:39.211 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.125212.log.gz 2023-10-16T14:14:39.212 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105297.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105297.log.gz 2023-10-16T14:14:39.212 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160010.log 2023-10-16T14:14:39.213 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134370.log 2023-10-16T14:14:39.214 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143409.log.gz 2023-10-16T14:14:39.215 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160010.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.160010.log.gz 2023-10-16T14:14:39.215 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151699.log 2023-10-16T14:14:39.216 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135504.log 2023-10-16T14:14:39.216 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.134370.log.gz 2023-10-16T14:14:39.217 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151699.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.151699.log.gz --verbose 2023-10-16T14:14:39.217 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.115406.log 2023-10-16T14:14:39.218 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159968.log 2023-10-16T14:14:39.219 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.135504.log.gz 2023-10-16T14:14:39.220 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115406.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.115406.log.gz 2023-10-16T14:14:39.220 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93146.log 2023-10-16T14:14:39.221 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97854.log 2023-10-16T14:14:39.222 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.159968.log.gz 2023-10-16T14:14:39.223 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120508.log 2023-10-16T14:14:39.223 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93146.log.gz 2023-10-16T14:14:39.224 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97854.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97854.log.gz 2023-10-16T14:14:39.225 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143220.log 2023-10-16T14:14:39.225 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153761.log 2023-10-16T14:14:39.226 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.120508.log.gz 2023-10-16T14:14:39.227 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143220.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143220.log.gz 2023-10-16T14:14:39.227 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149917.log 2023-10-16T14:14:39.228 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153761.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153761.log.gz 2023-10-16T14:14:39.229 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.154824.log 2023-10-16T14:14:39.230 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138239.log 2023-10-16T14:14:39.230 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.149917.log.gz 2023-10-16T14:14:39.231 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.154824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.154824.log.gz 2023-10-16T14:14:39.231 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116661.log 2023-10-16T14:14:39.232 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138239.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.138239.log.gz 2023-10-16T14:14:39.233 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110048.log 2023-10-16T14:14:39.234 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122295.log 2023-10-16T14:14:39.234 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116661.log.gz 2023-10-16T14:14:39.235 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110048.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110048.log.gz 2023-10-16T14:14:39.236 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132369.log 2023-10-16T14:14:39.237 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107127.log 2023-10-16T14:14:39.238 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122295.log.gz 2023-10-16T14:14:39.238 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132369.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.132369.log.gz 2023-10-16T14:14:39.239 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144548.log 2023-10-16T14:14:39.240 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.107127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105526.log 2023-10-16T14:14:39.240 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.107127.log.gz 2023-10-16T14:14:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144548.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144548.log.gz 2023-10-16T14:14:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122253.log 2023-10-16T14:14:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105526.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105526.log.gz 2023-10-16T14:14:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138520.log 2023-10-16T14:14:39.244 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112074.log 2023-10-16T14:14:39.245 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122253.log.gz 2023-10-16T14:14:39.246 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138520.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.138520.log.gz 2023-10-16T14:14:39.246 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131463.log 2023-10-16T14:14:39.247 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112074.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112074.log.gz 2023-10-16T14:14:39.247 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156593.log 2023-10-16T14:14:39.248 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138959.log 2023-10-16T14:14:39.249 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.131463.log.gz 2023-10-16T14:14:39.250 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156593.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.156593.log.gz 2023-10-16T14:14:39.250 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87766.log 2023-10-16T14:14:39.251 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138959.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138959.log.gz 2023-10-16T14:14:39.251 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147025.log 2023-10-16T14:14:39.252 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.87766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149409.log 2023-10-16T14:14:39.253 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87766.log.gz 2023-10-16T14:14:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147025.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147025.log.gz 2023-10-16T14:14:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110132.log 2023-10-16T14:14:39.255 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149409.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149409.log.gz 2023-10-16T14:14:39.256 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95581.log 2023-10-16T14:14:39.257 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110132.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110132.log.gz 2023-10-16T14:14:39.258 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138019.log 2023-10-16T14:14:39.258 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95581.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95581.log.gz 2023-10-16T14:14:39.259 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104467.log 2023-10-16T14:14:39.260 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160871.log 2023-10-16T14:14:39.261 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138019.log.gz 2023-10-16T14:14:39.261 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104467.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104467.log.gz 2023-10-16T14:14:39.262 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146744.log 2023-10-16T14:14:39.263 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107407.log 2023-10-16T14:14:39.264 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.160871.log.gz 2023-10-16T14:14:39.265 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146744.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146744.log.gz 2023-10-16T14:14:39.265 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101129.log 2023-10-16T14:14:39.266 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.107407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95245.log 2023-10-16T14:14:39.266 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107407.log.gz 2023-10-16T14:14:39.267 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149621.log 2023-10-16T14:14:39.268 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101129.log.gz 2023-10-16T14:14:39.269 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154986.log 2023-10-16T14:14:39.269 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95245.log.gz 2023-10-16T14:14:39.270 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149621.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.149621.log.gz 2023-10-16T14:14:39.271 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100817.log 2023-10-16T14:14:39.271 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133446.log 2023-10-16T14:14:39.272 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154986.log.gz 2023-10-16T14:14:39.273 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100817.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100817.log.gz 2023-10-16T14:14:39.273 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93426.log 2023-10-16T14:14:39.274 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133446.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133446.log.gz 2023-10-16T14:14:39.275 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134454.log 2023-10-16T14:14:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138334.log 2023-10-16T14:14:39.276 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93426.log.gz 2023-10-16T14:14:39.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134454.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134454.log.gz 2023-10-16T14:14:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150754.log 2023-10-16T14:14:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138334.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138334.log.gz 2023-10-16T14:14:39.279 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93720.log 2023-10-16T14:14:39.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112294.log 2023-10-16T14:14:39.281 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150754.log.gz 2023-10-16T14:14:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93720.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93720.log.gz 2023-10-16T14:14:39.282 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92807.log 2023-10-16T14:14:39.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112294.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.112294.log.gz 2023-10-16T14:14:39.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157018.log 2023-10-16T14:14:39.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93384.log 2023-10-16T14:14:39.285 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92807.log.gz 2023-10-16T14:14:39.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157018.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.157018.log.gz 2023-10-16T14:14:39.286 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139553.log 2023-10-16T14:14:39.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93384.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93384.log.gz 2023-10-16T14:14:39.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143094.log 2023-10-16T14:14:39.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139553.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139553.log.gz 2023-10-16T14:14:39.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122636.log 2023-10-16T14:14:39.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96700.log 2023-10-16T14:14:39.291 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.143094.log.gz 2023-10-16T14:14:39.292 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122636.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122636.log.gz 2023-10-16T14:14:39.292 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138791.log 2023-10-16T14:14:39.293 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96700.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.96700.log.gz --verbose 2023-10-16T14:14:39.294 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.96358.log 2023-10-16T14:14:39.295 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138791.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138791.log.gz 2023-10-16T14:14:39.295 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133768.log 2023-10-16T14:14:39.296 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96358.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.96358.log.gz 2023-10-16T14:14:39.297 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147897.log 2023-10-16T14:14:39.297 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121373.log 2023-10-16T14:14:39.298 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.133768.log.gz 2023-10-16T14:14:39.299 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147897.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147897.log.gz 2023-10-16T14:14:39.299 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100287.log 2023-10-16T14:14:39.301 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121373.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.121373.log.gz -5 2023-10-16T14:14:39.301 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.154529.log 2023-10-16T14:14:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91830.log 2023-10-16T14:14:39.303 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100287.log.gz 2023-10-16T14:14:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154529.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.154529.log.gz 2023-10-16T14:14:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160598.log 2023-10-16T14:14:39.305 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150458.log 2023-10-16T14:14:39.305 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.91830.log.gz 2023-10-16T14:14:39.306 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160598.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.160598.log.gz 2023-10-16T14:14:39.307 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157326.log 2023-10-16T14:14:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148756.log 2023-10-16T14:14:39.308 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150458.log.gz 2023-10-16T14:14:39.310 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111913.log 2023-10-16T14:14:39.310 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.157326.log.gz 2023-10-16T14:14:39.311 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148756.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148756.log.gz 2023-10-16T14:14:39.311 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134496.log 2023-10-16T14:14:39.312 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147687.log 2023-10-16T14:14:39.313 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111913.log.gz 2023-10-16T14:14:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134496.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.134496.log.gz 2023-10-16T14:14:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138604.log 2023-10-16T14:14:39.315 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147687.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147687.log.gz 2023-10-16T14:14:39.315 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116397.log 2023-10-16T14:14:39.317 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131205.log 2023-10-16T14:14:39.317 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138604.log.gz 2023-10-16T14:14:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116397.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116397.log.gz 2023-10-16T14:14:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156635.log 2023-10-16T14:14:39.319 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131205.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.131205.log.gz 2023-10-16T14:14:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110395.log 2023-10-16T14:14:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137186.log 2023-10-16T14:14:39.321 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156635.log.gz 2023-10-16T14:14:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110395.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110395.log.gz 2023-10-16T14:14:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112641.log 2023-10-16T14:14:39.324 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.137186.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137186.log.gz 2023-10-16T14:14:39.324 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111743.log 2023-10-16T14:14:39.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112641.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112641.log.gz 2023-10-16T14:14:39.326 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93026.log 2023-10-16T14:14:39.327 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147330.log 2023-10-16T14:14:39.327 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111743.log.gz 2023-10-16T14:14:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93026.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93026.log.gz 2023-10-16T14:14:39.328 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150798.log 2023-10-16T14:14:39.329 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147330.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147330.log.gz 2023-10-16T14:14:39.330 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148149.log 2023-10-16T14:14:39.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146941.log 2023-10-16T14:14:39.331 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150798.log.gz 2023-10-16T14:14:39.332 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148149.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.148149.log.gz 2023-10-16T14:14:39.333 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100859.log 2023-10-16T14:14:39.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104204.log 2023-10-16T14:14:39.335 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146941.log.gz 2023-10-16T14:14:39.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151825.log 2023-10-16T14:14:39.336 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.100859.log.gz 2023-10-16T14:14:39.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104204.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.104204.log.gz 2023-10-16T14:14:39.337 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158667.log 2023-10-16T14:14:39.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155944.log 2023-10-16T14:14:39.339 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151825.log.gz 2023-10-16T14:14:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158667.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.158667.log.gz 2023-10-16T14:14:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109628.log 2023-10-16T14:14:39.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155944.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.155944.log.gz 2023-10-16T14:14:39.342 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-16T14:14:39.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150668.log 2023-10-16T14:14:39.343 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109628.log.gz 2023-10-16T14:14:39.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113959.log 2023-10-16T14:14:39.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119448.log 2023-10-16T14:14:39.346 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150668.log.gz 2023-10-16T14:14:39.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158751.log 2023-10-16T14:14:39.347 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113959.log.gz 2023-10-16T14:14:39.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144676.log 2023-10-16T14:14:39.349 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119448.log.gz 2023-10-16T14:14:39.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111369.log 2023-10-16T14:14:39.350 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.158751.log.gz 2023-10-16T14:14:39.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98320.log 2023-10-16T14:14:39.352 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144676.log.gz 2023-10-16T14:14:39.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157148.log 2023-10-16T14:14:39.354 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111369.log.gz 2023-10-16T14:14:39.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109586.log 2023-10-16T14:14:39.355 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98320.log.gz 2023-10-16T14:14:39.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100119.log 2023-10-16T14:14:39.356 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.157148.log.gz 2023-10-16T14:14:39.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112725.log 2023-10-16T14:14:39.358 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109586.log.gz 2023-10-16T14:14:39.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118126.log 2023-10-16T14:14:39.359 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100119.log.gz 2023-10-16T14:14:39.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159192.log 2023-10-16T14:14:39.360 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112725.log.gz 2023-10-16T14:14:39.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110300.log 2023-10-16T14:14:39.362 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118126.log.gz 2023-10-16T14:14:39.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151741.log 2023-10-16T14:14:39.364 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.159192.log.gz 2023-10-16T14:14:39.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118315.log 2023-10-16T14:14:39.365 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110300.log.gz 2023-10-16T14:14:39.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149791.log 2023-10-16T14:14:39.366 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151741.log.gz 2023-10-16T14:14:39.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95917.log 2023-10-16T14:14:39.368 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118315.log.gz 2023-10-16T14:14:39.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126476.log 2023-10-16T14:14:39.370 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149791.log.gz 2023-10-16T14:14:39.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126219.log 2023-10-16T14:14:39.371 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95917.log.gz 2023-10-16T14:14:39.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126476.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.126476.log.gz -5 2023-10-16T14:14:39.372 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.118084.log 2023-10-16T14:14:39.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113523.log 2023-10-16T14:14:39.374 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.126219.log.gz 2023-10-16T14:14:39.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133887.log 2023-10-16T14:14:39.375 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118084.log.gz 2023-10-16T14:14:39.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155164.log 2023-10-16T14:14:39.376 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113523.log.gz 2023-10-16T14:14:39.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132539.log 2023-10-16T14:14:39.378 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.133887.log.gz 2023-10-16T14:14:39.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118838.log 2023-10-16T14:14:39.380 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155164.log.gz 2023-10-16T14:14:39.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150500.log 2023-10-16T14:14:39.381 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.132539.log.gz 2023-10-16T14:14:39.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135174.log 2023-10-16T14:14:39.383 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118838.log.gz 2023-10-16T14:14:39.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146049.log 2023-10-16T14:14:39.384 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150500.log.gz 2023-10-16T14:14:39.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157638.log 2023-10-16T14:14:39.386 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135174.log.gz 2023-10-16T14:14:39.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140103.log 2023-10-16T14:14:39.387 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146049.log.gz 2023-10-16T14:14:39.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155206.log 2023-10-16T14:14:39.389 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.157638.log.gz 2023-10-16T14:14:39.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120993.log 2023-10-16T14:14:39.390 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.140103.log.gz 2023-10-16T14:14:39.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112683.log 2023-10-16T14:14:39.391 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.155206.log.gz 2023-10-16T14:14:39.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155461.log 2023-10-16T14:14:39.393 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.120993.log.gz 2023-10-16T14:14:39.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123298.log 2023-10-16T14:14:39.395 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112683.log.gz 2023-10-16T14:14:39.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149451.log 2023-10-16T14:14:39.396 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.155461.log.gz 2023-10-16T14:14:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144762.log 2023-10-16T14:14:39.397 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.123298.log.gz 2023-10-16T14:14:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157452.log 2023-10-16T14:14:39.399 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149451.log.gz 2023-10-16T14:14:39.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133073.log 2023-10-16T14:14:39.400 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.144762.log.gz 2023-10-16T14:14:39.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139511.log 2023-10-16T14:14:39.402 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157452.log.gz 2023-10-16T14:14:39.403 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105087.log 2023-10-16T14:14:39.403 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.133073.log.gz 2023-10-16T14:14:39.404 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160220.log 88.7% 2023-10-16T14:14:39.405 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.139511.log.gz 2023-10-16T14:14:39.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129706.log 2023-10-16T14:14:39.406 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105087.log.gz 2023-10-16T14:14:39.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139725.log 2023-10-16T14:14:39.407 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160220.log.gz 2023-10-16T14:14:39.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.129706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124110.log 2023-10-16T14:14:39.409 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129706.log.gz 2023-10-16T14:14:39.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147855.log 2023-10-16T14:14:39.410 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139725.log.gz 2023-10-16T14:14:39.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146857.log 2023-10-16T14:14:39.414 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124110.log.gz 2023-10-16T14:14:39.414 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139935.log 2023-10-16T14:14:39.414 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.147855.log.gz 2023-10-16T14:14:39.415 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140019.log 2023-10-16T14:14:39.415 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146857.log.gz 2023-10-16T14:14:39.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134767.log 2023-10-16T14:14:39.416 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139935.log.gz 2023-10-16T14:14:39.417 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149493.log 2023-10-16T14:14:39.418 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140019.log.gz 2023-10-16T14:14:39.419 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139639.log 2023-10-16T14:14:39.419 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134767.log.gz 2023-10-16T14:14:39.420 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105652.log 2023-10-16T14:14:39.421 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149493.log.gz 2023-10-16T14:14:39.422 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118399.log 2023-10-16T14:14:39.422 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139639.log.gz 2023-10-16T14:14:39.423 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147813.log 2023-10-16T14:14:39.424 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105652.log.gz 2023-10-16T14:14:39.425 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160052.log 2023-10-16T14:14:39.425 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118399.log.gz 2023-10-16T14:14:39.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160640.log 2023-10-16T14:14:39.426 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147813.log.gz 2023-10-16T14:14:39.428 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123256.log 2023-10-16T14:14:39.428 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160052.log.gz 2023-10-16T14:14:39.429 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106076.log 2023-10-16T14:14:39.430 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.160640.log.gz 2023-10-16T14:14:39.431 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95077.log 2023-10-16T14:14:39.431 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123256.log.gz 2023-10-16T14:14:39.432 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100943.log 2023-10-16T14:14:39.433 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106076.log.gz 2023-10-16T14:14:39.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110769.log 2023-10-16T14:14:39.434 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95077.log.gz 2023-10-16T14:14:39.435 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.94876.log 2023-10-16T14:14:39.435 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100943.log.gz 2023-10-16T14:14:39.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98148.log 2023-10-16T14:14:39.437 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110769.log.gz 2023-10-16T14:14:39.437 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.2.94876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.94876.log.gz 2023-10-16T14:14:39.438 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108916.log 2023-10-16T14:14:39.440 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121730.log 88.5% 2023-10-16T14:14:39.440 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98148.log.gz 2023-10-16T14:14:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.108916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157060.log 2023-10-16T14:14:39.441 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108916.log.gz 2023-10-16T14:14:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98190.log 2023-10-16T14:14:39.443 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.121730.log.gz 2023-10-16T14:14:39.444 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158381.log 2023-10-16T14:14:39.444 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157060.log.gz 2023-10-16T14:14:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97396.log 2023-10-16T14:14:39.446 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98190.log.gz 2023-10-16T14:14:39.447 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97770.log 2023-10-16T14:14:39.447 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158381.log.gz 2023-10-16T14:14:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96904.log 2023-10-16T14:14:39.449 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97396.log.gz 2023-10-16T14:14:39.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99437.log 2023-10-16T14:14:39.450 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97770.log.gz 2023-10-16T14:14:39.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101702.log 2023-10-16T14:14:39.451 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96904.log.gz 2023-10-16T14:14:39.453 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115930.log 2023-10-16T14:14:39.453 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99437.log.gz 2023-10-16T14:14:39.454 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152854.log 2023-10-16T14:14:39.455 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101702.log.gz 2023-10-16T14:14:39.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97564.log 2023-10-16T14:14:39.456 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.115930.log.gz 2023-10-16T14:14:39.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136545.log 2023-10-16T14:14:39.457 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152854.log.gz 2023-10-16T14:14:39.458 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155860.log 2023-10-16T14:14:39.459 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97564.log.gz 2023-10-16T14:14:39.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96616.log 2023-10-16T14:14:39.460 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.136545.log.gz 2023-10-16T14:14:39.461 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115616.log 2023-10-16T14:14:39.462 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.155860.log.gz 2023-10-16T14:14:39.463 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126906.log 2023-10-16T14:14:39.463 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96616.log.gz 2023-10-16T14:14:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124771.log 2023-10-16T14:14:39.465 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.115616.log.gz 2023-10-16T14:14:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126906.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.126906.log.gz 2023-10-16T14:14:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116957.log 2023-10-16T14:14:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124771.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.124771.log.gz 2023-10-16T14:14:39.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155818.log 2023-10-16T14:14:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159108.log 2023-10-16T14:14:39.470 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116957.log.gz 2023-10-16T14:14:39.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112599.log 2023-10-16T14:14:39.471 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.155818.log.gz 2023-10-16T14:14:39.472 INFO:teuthology.orchestra.run.smithi060.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-16T14:14:39.473 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159108.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159108.log.gz 2023-10-16T14:14:39.473 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132947.log 2023-10-16T14:14:39.474 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112599.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112599.log.gz 2023-10-16T14:14:39.474 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106459.log 2023-10-16T14:14:39.475 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153292.log 2023-10-16T14:14:39.476 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.132947.log.gz 2023-10-16T14:14:39.477 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106459.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106459.log.gz 2023-10-16T14:14:39.477 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156934.log 2023-10-16T14:14:39.478 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134893.log 2023-10-16T14:14:39.478 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.153292.log.gz 2023-10-16T14:14:39.479 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156934.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156934.log.gz 2023-10-16T14:14:39.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154235.log 2023-10-16T14:14:39.480 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99496.log 2023-10-16T14:14:39.481 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134893.log.gz 2023-10-16T14:14:39.481 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154235.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154235.log.gz 2023-10-16T14:14:39.482 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152644.log 2023-10-16T14:14:39.483 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134244.log 2023-10-16T14:14:39.483 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99496.log.gz 2023-10-16T14:14:39.484 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152644.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152644.log.gz 2023-10-16T14:14:39.484 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158339.log 2023-10-16T14:14:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133512.log 2023-10-16T14:14:39.486 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.134244.log.gz 2023-10-16T14:14:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158339.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158339.log.gz 2023-10-16T14:14:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159276.log 2023-10-16T14:14:39.488 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133512.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-16T14:14:39.488 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.133512.log.gz 2023-10-16T14:14:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159276.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.159276.log.gz 2023-10-16T14:14:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92124.log 2023-10-16T14:14:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117677.log 2023-10-16T14:14:39.491 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92124.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92124.log.gz 2023-10-16T14:14:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92800.log 2023-10-16T14:14:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136674.log 2023-10-16T14:14:39.493 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.117677.log.gz 2023-10-16T14:14:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92800.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92800.log.gz 2023-10-16T14:14:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147414.log 2023-10-16T14:14:39.495 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.136674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157240.log 2023-10-16T14:14:39.496 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136674.log.gz 2023-10-16T14:14:39.497 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134640.log 2023-10-16T14:14:39.497 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.147414.log.gz 2023-10-16T14:14:39.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115761.log 2023-10-16T14:14:39.499 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.157240.log.gz 2023-10-16T14:14:39.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126388.log 2023-10-16T14:14:39.500 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134640.log.gz 2023-10-16T14:14:39.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115761.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115761.log.gz 2023-10-16T14:14:39.501 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147981.log 2023-10-16T14:14:39.502 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113875.log 2023-10-16T14:14:39.503 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126388.log.gz 2023-10-16T14:14:39.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147981.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147981.log.gz 2023-10-16T14:14:39.504 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106034.log 2023-10-16T14:14:39.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113875.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113875.log.gz 2023-10-16T14:14:39.506 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106247.log 2023-10-16T14:14:39.507 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152896.log 2023-10-16T14:14:39.508 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106034.log.gz 2023-10-16T14:14:39.509 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152812.log 2023-10-16T14:14:39.509 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106247.log.gz 2023-10-16T14:14:39.510 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113481.log 2023-10-16T14:14:39.511 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152896.log.gz 2023-10-16T14:14:39.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134196.log 2023-10-16T14:14:39.512 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152812.log.gz 2023-10-16T14:14:39.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113481.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113481.log.gzgzip 2023-10-16T14:14:39.514 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.152518.log 2023-10-16T14:14:39.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152770.log 2023-10-16T14:14:39.515 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134196.log.gz 2023-10-16T14:14:39.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122043.log 2023-10-16T14:14:39.517 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152518.log.gz 2023-10-16T14:14:39.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119090.log 2023-10-16T14:14:39.518 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152770.log.gz 2023-10-16T14:14:39.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127429.log 2023-10-16T14:14:39.520 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.122043.log.gz 2023-10-16T14:14:39.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119090.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119090.log.gz 2023-10-16T14:14:39.522 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98064.log 2023-10-16T14:14:39.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127429.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127429.log.gz 2023-10-16T14:14:39.523 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120728.log 2023-10-16T14:14:39.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.123635.log 2023-10-16T14:14:39.524 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98064.log.gz 2023-10-16T14:14:39.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.120728.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120728.log.gz 2023-10-16T14:14:39.526 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121519.log 2023-10-16T14:14:39.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.123635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.123635.log.gz 2023-10-16T14:14:39.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154571.log 2023-10-16T14:14:39.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111785.log 2023-10-16T14:14:39.529 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121519.log.gz 2023-10-16T14:14:39.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154571.log: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143871.log 2023-10-16T14:14:39.530 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.154571.log.gz 2023-10-16T14:14:39.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145783.log 2023-10-16T14:14:39.532 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111785.log.gz 2023-10-16T14:14:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143871.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143871.log.gz 2023-10-16T14:14:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91471.log 2023-10-16T14:14:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144123.log 2023-10-16T14:14:39.534 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145783.log.gz 2023-10-16T14:14:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144463.log 2023-10-16T14:14:39.536 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91471.log.gz 2023-10-16T14:14:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144123.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.144123.log.gz 2023-10-16T14:14:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119575.log 2023-10-16T14:14:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93762.log 2023-10-16T14:14:39.538 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144463.log.gz 2023-10-16T14:14:39.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119575.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119575.log.gz 2023-10-16T14:14:39.539 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144906.log 2023-10-16T14:14:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113567.log 2023-10-16T14:14:39.541 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93762.log.gz 2023-10-16T14:14:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144906.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144906.log.gz 2023-10-16T14:14:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132633.log 2023-10-16T14:14:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125711.log 2023-10-16T14:14:39.543 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113567.log.gz 2023-10-16T14:14:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132633.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.132633.log.gz 2023-10-16T14:14:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123922.log 2023-10-16T14:14:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159234.log 2023-10-16T14:14:39.546 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125711.log.gz 2023-10-16T14:14:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123922.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.123922.log.gz -5 2023-10-16T14:14:39.547 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.131902.log 2023-10-16T14:14:39.548 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143577.log 2023-10-16T14:14:39.548 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159234.log.gz 2023-10-16T14:14:39.549 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109712.log 2023-10-16T14:14:39.550 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.131902.log.gz 2023-10-16T14:14:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143577.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143577.log.gz 2023-10-16T14:14:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125085.log 2023-10-16T14:14:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95539.log 2023-10-16T14:14:39.552 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109712.log.gz 2023-10-16T14:14:39.553 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146577.log 88.7% 2023-10-16T14:14:39.554 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.125085.log.gz 2023-10-16T14:14:39.555 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95539.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95539.log.gz 2023-10-16T14:14:39.555 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132675.log 2023-10-16T14:14:39.557 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146577.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.146007.log 2023-10-16T14:14:39.557 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.146577.log.gz 2023-10-16T14:14:39.558 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132675.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.132675.log.gz 2023-10-16T14:14:39.558 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109458.log 2023-10-16T14:14:39.560 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96442.log 2023-10-16T14:14:39.560 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146007.log.gz 2023-10-16T14:14:39.561 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109458.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.109458.log.gz 2023-10-16T14:14:39.561 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122474.log 2023-10-16T14:14:39.562 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93342.log 2023-10-16T14:14:39.563 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96442.log.gz 2023-10-16T14:14:39.563 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122474.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122474.log.gz 2023-10-16T14:14:39.564 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122127.log 2023-10-16T14:14:39.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93342.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93342.log.gz 2023-10-16T14:14:39.565 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121958.log 2023-10-16T14:14:39.566 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122127.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T14:14:39.567 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.122127.log.gz 2023-10-16T14:14:39.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121958.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121958.log.gz 2023-10-16T14:14:39.568 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134935.log 2023-10-16T14:14:39.570 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132863.log 2023-10-16T14:14:39.571 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160682.log 2023-10-16T14:14:39.571 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134935.log.gz 2023-10-16T14:14:39.573 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.132863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147246.log 2023-10-16T14:14:39.573 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.132863.log.gz 2023-10-16T14:14:39.574 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154055.log 2023-10-16T14:14:39.574 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160682.log.gz 2023-10-16T14:14:39.576 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147246.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.147246.log.gz 2023-10-16T14:14:39.576 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.115676.log 2023-10-16T14:14:39.577 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154055.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.154055.log.gz 2023-10-16T14:14:39.577 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159547.log 2023-10-16T14:14:39.578 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115676.log: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T14:14:39.579 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118357.log 2023-10-16T14:14:39.579 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.115676.log.gz 2023-10-16T14:14:39.580 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102889.log 2023-10-16T14:14:39.581 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159547.log.gz 2023-10-16T14:14:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118357.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.118357.log.gz 2023-10-16T14:14:39.582 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94110.log 2023-10-16T14:14:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.102889.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102889.log.gz 2023-10-16T14:14:39.583 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145032.log 2023-10-16T14:14:39.584 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.94110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124228.log 2023-10-16T14:14:39.584 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94110.log.gz 2023-10-16T14:14:39.585 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145032.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145032.log.gz 2023-10-16T14:14:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134538.log 2023-10-16T14:14:39.586 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124228.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124228.log.gz 2023-10-16T14:14:39.587 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111093.log 2023-10-16T14:14:39.588 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125482.log 2023-10-16T14:14:39.588 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134538.log.gz 2023-10-16T14:14:39.589 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.111093.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111093.log.gz 2023-10-16T14:14:39.589 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118273.log 2023-10-16T14:14:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144849.log 2023-10-16T14:14:39.591 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.125482.log.gz 2023-10-16T14:14:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118273.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118273.log.gz 2023-10-16T14:14:39.592 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104712.log 2023-10-16T14:14:39.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99791.log 2023-10-16T14:14:39.593 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.144849.log.gz 2023-10-16T14:14:39.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.104712.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104712.log.gz 2023-10-16T14:14:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115448.log 2023-10-16T14:14:39.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144590.log 2023-10-16T14:14:39.596 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99791.log.gz 2023-10-16T14:14:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115448.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115448.log.gz 2023-10-16T14:14:39.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131631.log 2023-10-16T14:14:39.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144590.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-16T14:14:39.599 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.144590.log.gz 2023-10-16T14:14:39.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131631.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.131631.log.gz 2023-10-16T14:14:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106375.log 2023-10-16T14:14:39.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119619.log 2023-10-16T14:14:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106375.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106375.log.gz 2023-10-16T14:14:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109544.log 2023-10-16T14:14:39.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98792.log 2023-10-16T14:14:39.604 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119619.log.gz/var/log/ceph/ceph-client.admin.109544.log: 2023-10-16T14:14:39.605 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109544.log.gz 2023-10-16T14:14:39.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131064.log 2023-10-16T14:14:39.607 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133031.log 2023-10-16T14:14:39.607 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.98792.log.gz 2023-10-16T14:14:39.608 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148275.log 2023-10-16T14:14:39.609 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.131064.log.gz 2023-10-16T14:14:39.609 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133031.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.133031.log.gz 2023-10-16T14:14:39.610 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131818.log 2023-10-16T14:14:39.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160304.log 2023-10-16T14:14:39.611 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148275.log.gz 2023-10-16T14:14:39.612 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131818.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.131818.log.gz 2023-10-16T14:14:39.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-16T14:14:39.614 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160304.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.160304.log.gz 2023-10-16T14:14:39.614 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135896.log 2023-10-16T14:14:39.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139426.log 2023-10-16T14:14:39.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135896.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.135896.log.gz 2023-10-16T14:14:39.618 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101407.log 2023-10-16T14:14:39.619 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139426.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139426.log.gz 2023-10-16T14:14:39.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109271.log 2023-10-16T14:14:39.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101407.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101407.log.gz 2023-10-16T14:14:39.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159631.log 2023-10-16T14:14:39.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109271.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109271.log.gz 2023-10-16T14:14:39.625 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154944.log 2023-10-16T14:14:39.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159631.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159631.log.gz 2023-10-16T14:14:39.627 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113080.log 2023-10-16T14:14:39.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154944.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.154944.log.gz 2023-10-16T14:14:39.628 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107253.log 2023-10-16T14:14:39.630 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113080.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.113080.log.gz 2023-10-16T14:14:39.631 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115364.log 2023-10-16T14:14:39.631 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.107253.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107253.log.gz 2023-10-16T14:14:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153927.log 2023-10-16T14:14:39.634 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115364.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.115364.log.gz 2023-10-16T14:14:39.635 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109796.log 2023-10-16T14:14:39.636 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153927.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153927.log.gz 2023-10-16T14:14:39.637 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158161.log 2023-10-16T14:14:39.638 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109796.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109796.log.gz 2023-10-16T14:14:39.640 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129748.log 2023-10-16T14:14:39.640 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158161.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158161.log.gz 2023-10-16T14:14:39.641 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143314.log 2023-10-16T14:14:39.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.129748.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.129748.log.gz 2023-10-16T14:14:39.644 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114169.log 2023-10-16T14:14:39.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143314.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143314.log.gz 2023-10-16T14:14:39.646 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151951.log 2023-10-16T14:14:39.647 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114169.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114169.log.gz 2023-10-16T14:14:39.648 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93140.log 2023-10-16T14:14:39.649 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151951.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151951.log.gz 2023-10-16T14:14:39.650 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121205.log 2023-10-16T14:14:39.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93140.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93140.log.gz 2023-10-16T14:14:39.652 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143661.log 2023-10-16T14:14:39.653 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121205.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121205.log.gz 2023-10-16T14:14:39.655 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110599.log 2023-10-16T14:14:39.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143661.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143661.log.gz 2023-10-16T14:14:39.656 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126304.log 2023-10-16T14:14:39.658 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110599.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110599.log.gz 2023-10-16T14:14:39.659 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118441.log 2023-10-16T14:14:39.660 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126304.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.126304.log.gz 2023-10-16T14:14:39.661 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151909.log 2023-10-16T14:14:39.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118441.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118441.log.gz 2023-10-16T14:14:39.663 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91580.log 2023-10-16T14:14:39.664 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151909.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151909.log.gz 2023-10-16T14:14:39.665 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91390.log 2023-10-16T14:14:39.666 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91580.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91580.log.gz 2023-10-16T14:14:39.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139127.log 2023-10-16T14:14:39.668 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.91390.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91390.log.gz 2023-10-16T14:14:39.669 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99183.log 2023-10-16T14:14:39.671 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139127.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.139127.log.gz 2023-10-16T14:14:39.672 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153418.log 2023-10-16T14:14:39.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99183.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99183.log.gz 2023-10-16T14:14:39.673 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100203.log 2023-10-16T14:14:39.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153418.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153418.log.gz 2023-10-16T14:14:39.676 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158214.log 2023-10-16T14:14:39.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100203.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100203.log.gz 2023-10-16T14:14:39.678 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96784.log 2023-10-16T14:14:39.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158214.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.158214.log.gz 2023-10-16T14:14:39.680 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153969.log 2023-10-16T14:14:39.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.96784.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96784.log.gz 2023-10-16T14:14:39.682 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93142.log 2023-10-16T14:14:39.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.153969.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.153969.log.gz 2023-10-16T14:14:39.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116703.log 2023-10-16T14:14:39.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93142.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93142.log.gz 2023-10-16T14:14:39.687 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149875.log/var/log/ceph/ceph-client.admin.116703.log: 2023-10-16T14:14:39.688 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116703.log.gz 2023-10-16T14:14:39.688 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146091.log 2023-10-16T14:14:39.691 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149875.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.149875.log.gz 2023-10-16T14:14:39.692 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156810.log 2023-10-16T14:14:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146091.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146091.log.gz 2023-10-16T14:14:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134725.log 2023-10-16T14:14:39.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156810.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.156810.log.gz 2023-10-16T14:14:39.695 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123382.log 2023-10-16T14:14:39.696 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134725.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.134725.log.gz 2023-10-16T14:14:39.697 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134286.log 2023-10-16T14:14:39.699 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123382.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.123382.log.gz 2023-10-16T14:14:39.700 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/ceph-client.admin.134286.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.145573.log 2023-10-16T14:14:39.701 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134286.log.gz 2023-10-16T14:14:39.702 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156154.log 2023-10-16T14:14:39.703 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145573.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145573.log.gz 2023-10-16T14:14:39.704 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127321.log 2023-10-16T14:14:39.705 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156154.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.156154.log.gz 2023-10-16T14:14:39.706 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103152.log 2023-10-16T14:14:39.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127321.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.127321.log.gz 2023-10-16T14:14:39.708 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126612.log 2023-10-16T14:14:39.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.103152.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103152.log.gz 2023-10-16T14:14:39.711 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156214.log 2023-10-16T14:14:39.712 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126612.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.126612.log.gz 2023-10-16T14:14:39.713 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98876.log 2023-10-16T14:14:39.714 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156214.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.156214.log.gz 2023-10-16T14:14:39.715 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155545.log 2023-10-16T14:14:39.716 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.98876.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.98876.log.gz 2023-10-16T14:14:39.717 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160472.log 2023-10-16T14:14:39.719 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155545.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.155545.log.gz 2023-10-16T14:14:39.719 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121856.log 2023-10-16T14:14:39.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160472.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.160472.log.gz 2023-10-16T14:14:39.721 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126346.log 2023-10-16T14:14:39.722 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121856.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121856.log.gz 2023-10-16T14:14:39.724 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126822.log 2023-10-16T14:14:39.725 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126346.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.126346.log.gz 2023-10-16T14:14:39.726 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125753.log 2023-10-16T14:14:39.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126822.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.126822.log.gz 2023-10-16T14:14:39.728 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124451.log 2023-10-16T14:14:39.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125753.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.125753.log.gz 2023-10-16T14:14:39.730 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101348.log 2023-10-16T14:14:39.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124451.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.124451.log.gz 2023-10-16T14:14:39.732 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156425.log 2023-10-16T14:14:39.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101348.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101348.log.gz 2023-10-16T14:14:39.734 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149110.log 2023-10-16T14:14:39.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156425.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156425.log.gz 2023-10-16T14:14:39.736 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109313.log 2023-10-16T14:14:39.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149110.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149110.log.gz 2023-10-16T14:14:39.739 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150001.log 2023-10-16T14:14:39.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.109313.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109313.log.gz 2023-10-16T14:14:39.741 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121289.log 2023-10-16T14:14:39.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.150001.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150001.log.gz 2023-10-16T14:14:39.743 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160724.log 2023-10-16T14:14:39.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121289.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.121289.log.gz 2023-10-16T14:14:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115574.log 2023-10-16T14:14:39.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160724.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160724.log.gz 2023-10-16T14:14:39.747 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101289.log 2023-10-16T14:14:39.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.115574.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.115574.log.gz 2023-10-16T14:14:39.749 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119006.log 2023-10-16T14:14:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101289.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101289.log.gz 2023-10-16T14:14:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147645.log 2023-10-16T14:14:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119006.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119006.log.gz 2023-10-16T14:14:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116481.log 2023-10-16T14:14:39.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147645.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147645.log.gz 2023-10-16T14:14:39.756 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154902.log 2023-10-16T14:14:39.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116481.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116481.log.gz 2023-10-16T14:14:39.758 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155902.log 2023-10-16T14:14:39.759 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154902.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154902.log.gz 2023-10-16T14:14:39.760 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99673.log 2023-10-16T14:14:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155902.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.155902.log.gz 2023-10-16T14:14:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155587.log 2023-10-16T14:14:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99673.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99673.log.gz 2023-10-16T14:14:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139767.log 2023-10-16T14:14:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155587.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.155587.log.gz 2023-10-16T14:14:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157410.log 2023-10-16T14:14:39.768 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139767.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.139767.log.gz 2023-10-16T14:14:39.769 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118525.log 2023-10-16T14:14:39.770 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157410.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.157410.log.gz 2023-10-16T14:14:39.770 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101087.log 2023-10-16T14:14:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118525.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118525.log.gz 2023-10-16T14:14:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118483.log 2023-10-16T14:14:39.774 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101087.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101087.log.gz 2023-10-16T14:14:39.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100901.log 2023-10-16T14:14:39.776 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.118483.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118483.log.gz 2023-10-16T14:14:39.777 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95497.log 2023-10-16T14:14:39.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100901.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100901.log.gz 2023-10-16T14:14:39.779 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106753.log 2023-10-16T14:14:39.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95497.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95497.log.gz 2023-10-16T14:14:39.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159360.log 2023-10-16T14:14:39.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106753.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.106753.log.gz 2023-10-16T14:14:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152350.log 2023-10-16T14:14:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159360.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.159360.log.gz 2023-10-16T14:14:39.786 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93888.log 2023-10-16T14:14:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152350.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.152350.log.gz 2023-10-16T14:14:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123713.log 2023-10-16T14:14:39.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93888.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93888.log.gz 2023-10-16T14:14:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134013.log 2023-10-16T14:14:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123713.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123713.log.gz 2023-10-16T14:14:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87379.log 2023-10-16T14:14:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.134013.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134013.log.gz 2023-10-16T14:14:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140518.log 2023-10-16T14:14:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.87379.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.87379.log.gz 2023-10-16T14:14:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.140518.log: -- /var/log/ceph/ceph-client.admin.125542.log 2023-10-16T14:14:39.798 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.140518.log.gz 2023-10-16T14:14:39.799 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101525.log 2023-10-16T14:14:39.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.125542.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.125542.log.gz 2023-10-16T14:14:39.801 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158465.log 2023-10-16T14:14:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.101525.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101525.log.gz 2023-10-16T14:14:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92464.log 2023-10-16T14:14:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158465.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158465.log.gz 2023-10-16T14:14:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143366.log 2023-10-16T14:14:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92464.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92464.log.gz 2023-10-16T14:14:39.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151050.log 2023-10-16T14:14:39.809 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.143366.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143366.log.gz 2023-10-16T14:14:39.810 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160514.log 2023-10-16T14:14:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151050.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151050.log.gz 2023-10-16T14:14:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119048.log 2023-10-16T14:14:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160514.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160514.log.gz 2023-10-16T14:14:39.814 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97606.log 2023-10-16T14:14:39.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119048.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.119048.log.gz 2023-10-16T14:14:39.816 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99993.log 2023-10-16T14:14:39.817 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97606.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97606.log.gz 2023-10-16T14:14:39.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151134.log 2023-10-16T14:14:39.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.99993.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99993.log.gz 2023-10-16T14:14:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111556.log 2023-10-16T14:14:39.821 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151134.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151134.log.gz 2023-10-16T14:14:39.823 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157596.log 2023-10-16T14:14:39.824 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.111556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.111556.log.gz 2023-10-16T14:14:39.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114043.log 2023-10-16T14:14:39.826 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.157596.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157596.log.gz 2023-10-16T14:14:39.827 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131505.log 2023-10-16T14:14:39.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.114043.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114043.log.gz 2023-10-16T14:14:39.829 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95413.log 2023-10-16T14:14:39.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131505.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.131505.log.gz 2023-10-16T14:14:39.831 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126696.log 2023-10-16T14:14:39.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95413.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95413.log.gz 2023-10-16T14:14:39.833 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152602.log 2023-10-16T14:14:39.835 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.126696.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126696.log.gz 2023-10-16T14:14:39.835 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131337.log 2023-10-16T14:14:39.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152602.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.152602.log.gz 2023-10-16T14:14:39.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149247.log 2023-10-16T14:14:39.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131337.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.131337.log.gz 2023-10-16T14:14:39.840 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148569.log 2023-10-16T14:14:39.841 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.149247.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149247.log.gz 2023-10-16T14:14:39.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159926.log 2023-10-16T14:14:39.843 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148569.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148569.log.gz 2023-10-16T14:14:39.844 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116098.log 2023-10-16T14:14:39.845 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159926.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.159926.log.gz 2023-10-16T14:14:39.846 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144505.log 2023-10-16T14:14:39.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116098.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116098.log.gz 2023-10-16T14:14:39.848 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154487.log 2023-10-16T14:14:39.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144505.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144505.log.gz 2023-10-16T14:14:39.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131860.log 2023-10-16T14:14:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154487.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.154487.log.gz 2023-10-16T14:14:39.853 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146427.log 2023-10-16T14:14:39.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131860.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.131860.log.gz 2023-10-16T14:14:39.855 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93105.log 2023-10-16T14:14:39.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146427.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146427.log.gz 2023-10-16T14:14:39.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117582.log 2023-10-16T14:14:39.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.93105.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93105.log.gz 2023-10-16T14:14:39.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140061.log 2023-10-16T14:14:39.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117582.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117582.log.gz 2023-10-16T14:14:39.861 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116789.log 2023-10-16T14:14:39.862 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140061.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.140061.log.gz 2023-10-16T14:14:39.863 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124897.log 2023-10-16T14:14:39.864 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116789.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.116789.log.gz 2023-10-16T14:14:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152098.log 2023-10-16T14:14:39.866 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.124897.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124897.log.gz 2023-10-16T14:14:39.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145827.log 2023-10-16T14:14:39.869 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.152098.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.152098.log.gz 2023-10-16T14:14:39.870 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92472.log 2023-10-16T14:14:39.871 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.145827.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145827.log.gz 2023-10-16T14:14:39.872 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123755.log 2023-10-16T14:14:39.873 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92472.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92472.log.gz 2023-10-16T14:14:39.874 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122169.log 2023-10-16T14:14:39.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.123755.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.123755.log.gz 2023-10-16T14:14:39.876 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127559.log 2023-10-16T14:14:39.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122169.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122169.log.gz 2023-10-16T14:14:39.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121562.log 2023-10-16T14:14:39.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.127559.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127559.log.gz 2023-10-16T14:14:39.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154361.log 2023-10-16T14:14:39.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121562.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121562.log.gz 2023-10-16T14:14:39.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112431.log 2023-10-16T14:14:39.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.154361.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154361.log.gz 2023-10-16T14:14:39.885 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131022.log 2023-10-16T14:14:39.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112431.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112431.log.gz 2023-10-16T14:14:39.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117958.log 2023-10-16T14:14:39.888 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.131022.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.131022.log.gz 2023-10-16T14:14:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159884.log 2023-10-16T14:14:39.890 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.117958.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117958.log.gz 2023-10-16T14:14:39.891 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105400.log 2023-10-16T14:14:39.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159884.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.159884.log.gz 2023-10-16T14:14:39.893 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113224.log 2023-10-16T14:14:39.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.105400.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105400.log.gz 2023-10-16T14:14:39.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119883.log 2023-10-16T14:14:39.897 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.113224.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113224.log.gz 2023-10-16T14:14:39.898 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116831.log 2023-10-16T14:14:39.899 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119883.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119883.log.gz 2023-10-16T14:14:39.900 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116611.log 2023-10-16T14:14:39.901 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116831.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.116831.log.gz 2023-10-16T14:14:39.902 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95707.log 2023-10-16T14:14:39.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116611.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116611.log.gz 2023-10-16T14:14:39.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.95707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158077.log 2023-10-16T14:14:39.905 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95707.log.gz 2023-10-16T14:14:39.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146815.log 2023-10-16T14:14:39.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158077.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158077.log.gz 2023-10-16T14:14:39.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159673.log 2023-10-16T14:14:39.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.146815.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146815.log.gz 2023-10-16T14:14:39.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110557.log 2023-10-16T14:14:39.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.159673.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159673.log.gz 2023-10-16T14:14:39.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121467.log 2023-10-16T14:14:39.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110557.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110557.log.gz 2023-10-16T14:14:39.915 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156028.log 2023-10-16T14:14:39.916 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.121467.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.121467.log.gz 2023-10-16T14:14:39.917 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110090.log 2023-10-16T14:14:39.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156028.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156028.log.gz 2023-10-16T14:14:39.919 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144165.log 2023-10-16T14:14:39.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110090.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110090.log.gz 2023-10-16T14:14:39.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92466.log 2023-10-16T14:14:39.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144165.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144165.log.gz 2023-10-16T14:14:39.923 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122379.log 2023-10-16T14:14:39.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.92466.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92466.log.gz 2023-10-16T14:14:39.926 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106627.log 2023-10-16T14:14:39.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.122379.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122379.log.gz 2023-10-16T14:14:39.928 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100329.log 2023-10-16T14:14:39.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.106627.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106627.log.gz 2023-10-16T14:14:39.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110006.log 2023-10-16T14:14:39.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.100329.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100329.log.gz 2023-10-16T14:14:39.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147729.log 2023-10-16T14:14:39.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110006.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110006.log.gz 2023-10-16T14:14:39.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-16T14:14:39.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.147729.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147729.log.gz 2023-10-16T14:14:39.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135592.log 2023-10-16T14:14:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.135592.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135592.log.gz 2023-10-16T14:14:39.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116313.log 2023-10-16T14:14:39.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.116313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155774.log 2023-10-16T14:14:39.944 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116313.log.gz 2023-10-16T14:14:39.946 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160094.log 2023-10-16T14:14:39.948 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.155774.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.155774.log.gz 2023-10-16T14:14:39.951 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.160094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156467.log 2023-10-16T14:14:39.953 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160094.log.gz 2023-10-16T14:14:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107085.log 2023-10-16T14:14:39.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.156467.log: /var/log/ceph/ceph-client.admin.107085.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107085.log.gz 2023-10-16T14:14:39.960 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158506.log 2023-10-16T14:14:39.960 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.156467.log.gz 2023-10-16T14:14:39.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.158506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151511.log 2023-10-16T14:14:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151511.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.158506.log.gz 2023-10-16T14:14:39.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148966.log 2023-10-16T14:14:39.969 INFO:teuthology.orchestra.run.smithi060.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.151511.log.gz 2023-10-16T14:14:39.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.148966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148317.log 2023-10-16T14:14:39.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151092.log 2023-10-16T14:14:39.974 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4%/var/log/ceph/ceph-client.admin.148317.log: -- replaced with /var/log/ceph/ceph-client.admin.148966.log.gz 2023-10-16T14:14:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119227.log 2023-10-16T14:14:39.975 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.148317.log.gz 2023-10-16T14:14:39.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.151092.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151092.log.gz 2023-10-16T14:14:39.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.119227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138282.log 2023-10-16T14:14:39.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.138282.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119227.log.gz 2023-10-16T14:14:39.984 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135336.log 2023-10-16T14:14:39.985 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.138282.log.gz 2023-10-16T14:14:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110881.log 2023-10-16T14:14:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110881.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110881.log.gz 2023-10-16T14:14:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.135336.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135336.log.gz 2023-10-16T14:14:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133613.log 2023-10-16T14:14:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.133613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119491.log 2023-10-16T14:14:39.996 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.133613.log.gz 2023-10-16T14:14:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144081.log 2023-10-16T14:14:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.144081.log: /var/log/ceph/ceph-client.admin.119491.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107365.log 2023-10-16T14:14:40.003 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.119491.log.gz 2023-10-16T14:14:40.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144081.log.gz 2023-10-16T14:14:40.006 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125627.log 2023-10-16T14:14:40.010 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.107365.log: /var/log/ceph/ceph-client.admin.125627.log: 88.3%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.107365.log.gz -- /var/log/ceph/ceph-client.admin.110216.log 2023-10-16T14:14:40.011 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:40.012 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125627.log.gz 2023-10-16T14:14:40.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110216.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110216.log.gz 2023-10-16T14:14:40.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112473.log 2023-10-16T14:14:40.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.112473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110475.log 2023-10-16T14:14:40.020 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112473.log.gz 2023-10-16T14:14:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.110475.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110475.log.gz 2023-10-16T14:14:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97938.log 2023-10-16T14:14:40.026 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.97938.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97938.log.gz 2023-10-16T14:14:40.026 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139893.log 2023-10-16T14:14:40.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.139893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140211.log 2023-10-16T14:14:40.031 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.139893.log.gz 2023-10-16T14:14:40.033 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.140211.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.140211.log.gz 2023-10-16T14:14:40.484 INFO:teuthology.orchestra.run.smithi144.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-16T14:14:40.763 INFO:teuthology.orchestra.run.smithi060.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-16T14:14:40.795 INFO:teuthology.orchestra.run.smithi144.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-16T14:14:40.827 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-16T14:14:40.860 INFO:teuthology.orchestra.run.smithi144.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-16T14:14:41.842 INFO:teuthology.orchestra.run.smithi144.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-16T14:14:41.861 INFO:teuthology.orchestra.run.smithi144.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-16T14:14:41.890 INFO:teuthology.orchestra.run.smithi144.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-16T14:14:42.693 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-16T14:14:42.694 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-16T14:14:42.774 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-16T14:14:43.062 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-16T14:14:43.086 INFO:teuthology.orchestra.run.smithi144.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-16T14:14:43.090 INFO:teuthology.orchestra.run.smithi144.stderr: 2023-10-16T14:14:43.090 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m4.806s 2023-10-16T14:14:43.090 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m20.993s 2023-10-16T14:14:43.091 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m1.873s 2023-10-16T14:14:44.608 INFO:teuthology.orchestra.run.smithi060.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-16T14:14:44.612 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-10-16T14:14:44.613 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m6.332s 2023-10-16T14:14:44.613 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m21.256s 2023-10-16T14:14:44.613 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m4.809s 2023-10-16T14:14:44.614 INFO:tasks.ceph:Archiving logs... 2023-10-16T14:14:44.615 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/remote/smithi060/log 2023-10-16T14:14:44.616 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T14:14:48.199 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/remote/smithi144/log 2023-10-16T14:14:48.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T14:14:50.703 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-16T14:14:50.712 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-16T14:14:50.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T14:14:50.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T14:14:50.876 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-16T14:14:50.885 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-16T14:14:50.886 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T14:14:50.895 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T14:14:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2023-10-16T14:14:50.951 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2023-10-16T14:14:50.957 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T14:14:50.957 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2023-10-16T14:14:50.958 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 1 -383us[ -383us] +/- 79ms 2023-10-16T14:14:50.958 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 +1567us[+1569us] +/- 57ms 2023-10-16T14:14:50.958 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -1769us[-1769us] +/- 77ms 2023-10-16T14:14:50.959 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:14:50.961 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T14:14:50.961 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2023-10-16T14:14:50.961 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 58 -442us[ -397us] +/- 78ms 2023-10-16T14:14:50.962 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 +1559us[+1604us] +/- 57ms 2023-10-16T14:14:50.962 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv03.front.sepia.ceph.com 3 7 377 53 -1788us[-1744us] +/- 77ms 2023-10-16T14:14:50.962 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T14:14:50.963 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-16T14:14:50.973 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-16T14:14:50.974 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-16T14:14:51.021 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T14:14:51.212 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T14:14:51.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T14:14:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:14:51.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T14:14:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:14:51.677 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-16T14:14:51.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-16T14:14:51.721 INFO:teuthology.task.internal:Duration was 6886.835087 seconds 2023-10-16T14:14:51.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-16T14:14:51.822 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-16T14:14:51.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T14:14:51.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T14:14:52.004 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T14:14:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T14:14:52.413 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-16T14:14:52.413 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2023-10-16T14:14:52.414 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T14:14:52.493 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2023-10-16T14:14:52.494 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T14:14:52.575 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-16T14:14:52.576 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T14:14:52.585 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T14:14:52.684 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-16T14:14:52.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T14:14:52.692 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T14:14:52.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-16T14:14:52.979 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-16T14:14:52.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T14:14:53.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T14:14:53.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-16T14:14:53.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T14:14:53.209 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T14:14:53.320 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2023-10-16T14:14:53.324 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2023-10-16T14:14:53.376 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T14:14:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:14:53.428 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T14:14:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T14:14:53.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-16T14:14:53.497 INFO:teuthology.task.internal:Transferring archived files... 2023-10-16T14:14:53.498 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/remote/smithi060 2023-10-16T14:14:53.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T14:14:53.616 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986/remote/smithi144 2023-10-16T14:14:53.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T14:14:53.726 INFO:teuthology.task.internal:Removing archive directory... 2023-10-16T14:14:53.726 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T14:14:53.736 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T14:14:53.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-16T14:14:53.803 INFO:teuthology.task.internal:Not uploading archives. 2023-10-16T14:14:53.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-16T14:14:53.881 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-16T14:14:53.881 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T14:14:53.890 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T14:14:53.944 INFO:teuthology.orchestra.run.smithi060.stdout: 525007 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 14:14 /home/ubuntu/cephtest 2023-10-16T14:14:53.951 INFO:teuthology.orchestra.run.smithi144.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 14:14 /home/ubuntu/cephtest 2023-10-16T14:14:53.954 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-16T14:14:54.022 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-16T14:14:54.181 INFO:teuthology.nuke:Checking targets against current locks 2023-10-16T14:14:54.209 DEBUG:teuthology.nuke:shortname: smithi060 2023-10-16T14:14:54.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T14:14:54.235 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 12:10:19.113882', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH9fQ0gtr9jZfHA6EZvk7I44etDngq/AlmCWvEkCDjt0JPtwM4iUwHk9OZCL+D1oK6jGFM14qRQXlIlT4mXQIDs='} 2023-10-16T14:14:54.239 DEBUG:teuthology.nuke:shortname: smithi144 2023-10-16T14:14:54.239 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T14:14:54.258 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428986', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 12:10:19.112006', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfNhnIlKlzJjnyFNggcHy/KuWjP3yzVYBJoBRaR4nIeuYgvBKYWmC5RzfBA9XWR9KibzaLap60SCOgnwgtX6Bg='} 2023-10-16T14:14:54.286 INFO:teuthology.orchestra.console.smithi060:Power off 2023-10-16T14:14:54.287 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T14:14:54.309 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2023-10-16T14:14:54.310 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:14:54.330 INFO:teuthology.orchestra.console.smithi144:Power off 2023-10-16T14:14:54.331 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T14:14:54.341 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2023-10-16T14:14:54.355 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off 2023-10-16T14:14:54.355 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:14:54.379 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2023-10-16T14:14:58.342 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:14:58.381 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:14:58.454 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2023-10-16T14:14:58.491 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2023-10-16T14:15:02.456 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:15:02.493 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:15:02.567 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2023-10-16T14:15:02.605 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2023-10-16T14:15:02.669 INFO:teuthology.orchestra.console.smithi060:Power off completed 2023-10-16T14:15:06.606 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T14:15:06.717 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2023-10-16T14:15:06.818 INFO:teuthology.orchestra.console.smithi144:Power off completed 2023-10-16T14:15:06.923 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 client.1-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6886.83508682251 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1d383f6274cf4dca8b51a83c05f39862 status: fail success: false 2023-10-16T14:15:06.924 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T14:15:07.010 INFO:teuthology.run:FAIL